Skip to contentSkip to conversation

KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

PATCH Applied Marc Zyngier 6 participants 140 messages Nov 29, 2021 – Jan 27, 2022 Page 1 of 2

Subsystems (4)

v5 of 10 Applied Latest is v10 →

lore
Share & tools
lore

Marc Zyngier

author maintainer
#0 lore raw

v5 cover KVM: arm64: ARMv8.3/8.4 Nested Virtualization support

Here the bi-annual drop of the KVM/arm64 NV support code, the least loved series in the history of KVM!

Not a lot has changed since [1]: a number of bug fixes (wrong MMU context being selected leading to failing TLB invalidations, fixes around the handling of nested faults), a complete rewrite of the early exit handling, a change in the way the NV support is enabled ("kvm-arm.mode=nested"), and a rebase on top of 5.16-rc1.

The first 2 patches are already on their way upstream as fixes, and the third is a likely candidate too.

As usual, blame me for any bug, and nobody else. It has been tested on my usual zoo, and nothing caught fire. Which means nothing, of course. Obviously, it isn't feature complete, and it is quite easy to write a guest that will not behave as intended. The current goal is to make sure that non-NV KVM is unaffected by the NV stuff.

It is massively painful to run on the FVP, but if you have a Neoverse V1 or N2 system (or anything else with ARMv8.4-NV) that is collecting dust, I have the right stuff to keep it busy.

	M.

[1] https://lore.kernel.org/r/[email protected]

Andre Przywara (1):
KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ

Christoffer Dall (15):
KVM: arm64: nv: Introduce nested virtualization VCPU feature
KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set
KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x
KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state
KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values
KVM: arm64: nv: Handle trapped ERET from virtual EL2
KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor
KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2
KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2
changes
KVM: arm64: nv: Implement nested Stage-2 page table walk logic
KVM: arm64: nv: Unmap/flush shadow stage 2 page tables
KVM: arm64: nv: arch_timer: Support hyp timer emulation
KVM: arm64: nv: vgic: Emulate the HW bit in software
KVM: arm64: nv: Add nested GICv3 tracepoints
KVM: arm64: nv: Sync nested timer state with ARMv8.4

Jintack Lim (18):
arm64: Add ARM64_HAS_NESTED_VIRT cpufeature
KVM: arm64: nv: Handle HCR_EL2.NV system register traps
KVM: arm64: nv: Support virtual EL2 exceptions
KVM: arm64: nv: Inject HVC exceptions to the virtual EL2
KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2
KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2
KVM: arm64: nv: Handle PSCI call via smc from the guest
KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting
KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings
KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting
KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings
KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting
KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2
KVM: arm64: nv: Configure HCR_EL2 for nested virtualization
KVM: arm64: nv: Introduce sys_reg_desc.forward_trap
KVM: arm64: nv: Set a handler for the system instruction traps
KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2
KVM: arm64: nv: Nested GICv3 Support

Marc Zyngier (35):
KVM: arm64: Save PSTATE early on exit
KVM: arm64: Move pkvm's special 32bit handling into a generic
infrastructure
KVM: arm64: Add minimal handling for the ARMv8.7 PMU
KVM: arm64: Rework kvm_pgtable initialisation
KVM: arm64: Allow preservation of the S2 SW bits
KVM: arm64: nv: Add EL2 system registers to vcpu context
KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers
KVM: arm64: nv: Handle virtual EL2 registers in
vcpu_read/write_sys_reg()
KVM: arm64: nv: Handle SPSR_EL2 specially
KVM: arm64: nv: Handle HCR_EL2.E2H specially
KVM: arm64: nv: Save/Restore vEL2 sysregs
KVM: arm64: nv: Forward debug traps to the nested guest
KVM: arm64: nv: Filter out unsupported features from ID regs
KVM: arm64: nv: Hide RAS from nested guests
KVM: arm64: nv: Support multiple nested Stage-2 mmu structures
KVM: arm64: nv: Handle shadow stage 2 page faults
KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's
KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2
KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's
KVM: arm64: nv: Add handling of EL2-specific timer registers
KVM: arm64: nv: Load timer before the GIC
KVM: arm64: nv: Don't load the GICv4 context on entering a nested
guest
KVM: arm64: nv: Implement maintenance interrupt forwarding
KVM: arm64: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT
KVM: arm64: nv: Add handling of ARMv8.4-TTL TLB invalidation
KVM: arm64: nv: Invalidate TLBs based on shadow S2 TTL-like
information
KVM: arm64: nv: Tag shadow S2 entries with nested level
KVM: arm64: nv: Add include containing the VNCR_EL2 offsets
KVM: arm64: nv: Map VNCR-capable registers to a separate page
KVM: arm64: nv: Move nested vgic state into the sysreg file
KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature
KVM: arm64: nv: Allocate VNCR page when required
KVM: arm64: nv: Enable ARMv8.4-NV support
KVM: arm64: nv: Fast-track 'InHost' exception returns
KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests

 .../admin-guide/kernel-parameters.txt         |    7 +-
 .../virt/kvm/devices/arm-vgic-v3.rst          |   12 +-
 arch/arm64/include/asm/esr.h                  |    6 +
 arch/arm64/include/asm/kvm_arm.h              |   29 +-
 arch/arm64/include/asm/kvm_asm.h              |    4 +
 arch/arm64/include/asm/kvm_emulate.h          |  145 +-
 arch/arm64/include/asm/kvm_host.h             |  185 ++-
 arch/arm64/include/asm/kvm_hyp.h              |    2 +
 arch/arm64/include/asm/kvm_mmu.h              |   18 +-
 arch/arm64/include/asm/kvm_nested.h           |  152 +++
 arch/arm64/include/asm/kvm_pgtable.h          |    9 +-
 arch/arm64/include/asm/sysreg.h               |  102 +-
 arch/arm64/include/asm/vncr_mapping.h         |   74 +
 arch/arm64/include/uapi/asm/kvm.h             |    2 +
 arch/arm64/kernel/cpufeature.c                |   34 +
 arch/arm64/kvm/Makefile                       |    4 +-
 arch/arm64/kvm/arch_timer.c                   |  202 ++-
 arch/arm64/kvm/arm.c                          |   42 +-
 arch/arm64/kvm/at.c                           |  231 ++++
 arch/arm64/kvm/emulate-nested.c               |  186 +++
 arch/arm64/kvm/guest.c                        |    6 +
 arch/arm64/kvm/handle_exit.c                  |   81 +-
 arch/arm64/kvm/hyp/exception.c                |   45 +-
 arch/arm64/kvm/hyp/include/hyp/switch.h       |   26 +-
 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h    |   31 +-
 arch/arm64/kvm/hyp/nvhe/mem_protect.c         |    4 +-
 arch/arm64/kvm/hyp/nvhe/switch.c              |   10 +-
 arch/arm64/kvm/hyp/nvhe/sysreg-sr.c           |    2 +-
 arch/arm64/kvm/hyp/pgtable.c                  |   13 +-
 arch/arm64/kvm/hyp/vgic-v3-sr.c               |    2 +-
 arch/arm64/kvm/hyp/vhe/switch.c               |  185 ++-
 arch/arm64/kvm/hyp/vhe/sysreg-sr.c            |  125 +-
 arch/arm64/kvm/hyp/vhe/tlb.c                  |   83 ++
 arch/arm64/kvm/inject_fault.c                 |   68 +-
 arch/arm64/kvm/mmu.c                          |  205 ++-
 arch/arm64/kvm/nested.c                       |  922 +++++++++++++
 arch/arm64/kvm/pmu-emul.c                     |    1 +
 arch/arm64/kvm/reset.c                        |   11 +-
 arch/arm64/kvm/sys_regs.c                     | 1213 ++++++++++++++++-
 arch/arm64/kvm/sys_regs.h                     |    8 +
 arch/arm64/kvm/trace_arm.h                    |   65 +-
 arch/arm64/kvm/vgic/vgic-init.c               |   30 +
 arch/arm64/kvm/vgic/vgic-kvm-device.c         |   22 +
 arch/arm64/kvm/vgic/vgic-nested-trace.h       |  137 ++
 arch/arm64/kvm/vgic/vgic-v3-nested.c          |  240 ++++
 arch/arm64/kvm/vgic/vgic-v3.c                 |   39 +-
 arch/arm64/kvm/vgic/vgic.c                    |   44 +
 arch/arm64/kvm/vgic/vgic.h                    |   10 +
 arch/arm64/tools/cpucaps                      |    2 +
 include/kvm/arm_arch_timer.h                  |    9 +-
 include/kvm/arm_vgic.h                        |   16 +
 include/uapi/linux/kvm.h                      |    1 +
 tools/arch/arm/include/uapi/asm/kvm.h         |    1 +
 53 files changed, 4918 insertions(+), 185 deletions(-)
 create mode 100644 arch/arm64/include/asm/kvm_nested.h
 create mode 100644 arch/arm64/include/asm/vncr_mapping.h
 create mode 100644 arch/arm64/kvm/at.c
 create mode 100644 arch/arm64/kvm/emulate-nested.c
 create mode 100644 arch/arm64/kvm/nested.c
 create mode 100644 arch/arm64/kvm/vgic/vgic-nested-trace.h
 create mode 100644 arch/arm64/kvm/vgic/vgic-v3-nested.c
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#1 lore raw

PATCH v5 1/69 KVM: arm64: Save PSTATE early on exit

In order to be able to use primitives such as vcpu_mode_is_32bit(), we need to synchronize the guest PSTATE. However, this is currently done deep into the bowels of the world-switch code, and we do have helpers evaluating this much earlier (__vgic_v3_perform_cpuif_access and handle_aarch32_guest, for example).

Move the saving of the guest pstate into the early fixups, which cures the first issue. The second one will be addressed separately.

---
 arch/arm64/kvm/hyp/include/hyp/switch.h    | 6 ++++++
 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 7 ++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/include/hyp/switch.h +6 -0
index 7a0af1d39303..d79fd101615f 100644
@@ -429,6 +429,12 @@ static inline bool kvm_hyp_handle_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
  */
 static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
 {
+	/*
+	 * Save PSTATE early so that we can evaluate the vcpu mode
+	 * early on.
+	 */
+	vcpu->arch.ctxt.regs.pstate = read_sysreg_el2(SYS_SPSR);
+
 	if (ARM_EXCEPTION_CODE(*exit_code) != ARM_EXCEPTION_IRQ)
 		vcpu->arch.fault.esr_el2 = read_sysreg_el2(SYS_ESR);
 
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +6 -1
index de7e14c862e6..7ecca8b07851 100644
@@ -70,7 +70,12 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt)
 static inline void __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt)
 {
 	ctxt->regs.pc			= read_sysreg_el2(SYS_ELR);
-	ctxt->regs.pstate		= read_sysreg_el2(SYS_SPSR);
+	/*
+	 * Guest PSTATE gets saved at guest fixup time in all
+	 * cases. We still need to handle the nVHE host side here.
+	 */
+	if (!has_vhe() && ctxt->__hyp_running_vcpu)
+		ctxt->regs.pstate	= read_sysreg_el2(SYS_SPSR);
 
 	if (cpus_have_final_cap(ARM64_HAS_RAS_EXTN))
 		ctxt_sys_reg(ctxt, DISR_EL1) = read_sysreg_s(SYS_VDISR_EL2);
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 1/69
#2 lore raw

Re: [PATCH v5 01/69] KVM: arm64: Save PSTATE early on exit

Quoted 6 lines · Marc Zyngier

In order to be able to use primitives such as vcpu_mode_is_32bit(), we need to synchronize the guest PSTATE. However, this is currently done deep into the bowels of the world-switch code, and we do have helpers evaluating this much earlier (__vgic_v3_perform_cpuif_access and handle_aarch32_guest, for example).

Quoted 6 lines

Move the saving of the guest pstate into the early fixups, which cures the first issue. The second one will be addressed separately.

Signature · 6 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#3 lore raw

PATCH v5 2/69 KVM: arm64: Move pkvm's special 32bit handling into a generic infrastructure

In reply to Marc Zyngier

Protected KVM is trying to turn AArch32 exceptions into an illegal exception entry. Unfortunately, it does that in a way that is a bit abrupt, and too early for PSTATE to be available.

Instead, move it to the fixup code, which is a more reasonable place for it. This will also be useful for the NV code.

---
 arch/arm64/kvm/hyp/include/hyp/switch.h | 8 ++++++++
 arch/arm64/kvm/hyp/nvhe/switch.c        | 8 +-------
 arch/arm64/kvm/hyp/vhe/switch.c         | 4 ++++
 3 files changed, 13 insertions(+), 7 deletions(-)
arch/arm64/kvm/hyp/include/hyp/switch.h +8 -0
index d79fd101615f..96c5f3fb7838 100644
@@ -403,6 +403,8 @@ typedef bool (*exit_handler_fn)(struct kvm_vcpu *, u64 *);
 
 static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu);
 
+static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code);
+
 /*
  * Allow the hypervisor to handle the exit with an exit handler if it has one.
  *
@@ -435,6 +437,12 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code)
 	 */
 	vcpu->arch.ctxt.regs.pstate = read_sysreg_el2(SYS_SPSR);
 
+	/*
+	 * Check whether we want to repaint the state one way or
+	 * another.
+	 */
+	early_exit_filter(vcpu, exit_code);
+
 	if (ARM_EXCEPTION_CODE(*exit_code) != ARM_EXCEPTION_IRQ)
 		vcpu->arch.fault.esr_el2 = read_sysreg_el2(SYS_ESR);
 
arch/arm64/kvm/hyp/nvhe/switch.c +1 -7
index c0e3fed26d93..d13115a12434 100644
@@ -233,7 +233,7 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)
  * Returns false if the guest ran in AArch32 when it shouldn't have, and
  * thus should exit to the host, or true if a the guest run loop can continue.
  */
-static bool handle_aarch32_guest(struct kvm_vcpu *vcpu, u64 *exit_code)
+static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code)
 {
 	struct kvm *kvm = kern_hyp_va(vcpu->kvm);
 
@@ -248,10 +248,7 @@ static bool handle_aarch32_guest(struct kvm_vcpu *vcpu, u64 *exit_code)
 		vcpu->arch.target = -1;
 		*exit_code &= BIT(ARM_EXIT_WITH_SERROR_BIT);
 		*exit_code |= ARM_EXCEPTION_IL;
-		return false;
 	}
-
-	return true;
 }
 
 /* Switch to the guest for legacy non-VHE systems */
@@ -316,9 +313,6 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
 		/* Jump in the fire! */
 		exit_code = __guest_enter(vcpu);
 
-		if (unlikely(!handle_aarch32_guest(vcpu, &exit_code)))
-			break;
-
 		/* And we're baaack! */
 	} while (fixup_guest_exit(vcpu, &exit_code));
 
arch/arm64/kvm/hyp/vhe/switch.c +4 -0
index 5a2cb5d9bc4b..fbb26b93c347 100644
@@ -112,6 +112,10 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)
 	return hyp_exit_handlers;
 }
 
+static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code)
+{
+}
+
 /* Switch to the guest for VHE systems running in EL2 */
 static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
 {
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#4 lore raw

Re: [PATCH v5 02/69] KVM: arm64: Move pkvm's special 32bit handling into a generic infrastructure

arch/arm64/kvm/hyp/nvhe/switch.c static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu) · line 236

Quoted 10 lines · Marc Zyngier
arch/arm64/kvm/hyp/nvhe/switch.c +1 -1
index c0e3fed26d93..d13115a12434 100644
@@ -233,7 +233,7 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)
  * Returns false if the guest ran in AArch32 when it shouldn't have, and
  * thus should exit to the host, or true if a the guest run loop can continue.
  */
-static bool handle_aarch32_guest(struct kvm_vcpu *vcpu, u64 *exit_code)
+static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code)

The comment above this function needs fixing up - it talks about the return value, but this patch changes the return type to be void.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
author maintainer
#5 lore raw

PATCH v5 3/69 KVM: arm64: Add minimal handling for the ARMv8.7 PMU

In reply to Marc Zyngier

When running a KVM guest hosted on an ARMv8.7 machine, the host kernel complains that it doesn't know about the architected number of events.

Fix it by adding the PMUver code corresponding to PMUv3 for ARMv8.7.

---
 arch/arm64/include/asm/sysreg.h | 1 +
 arch/arm64/kvm/pmu-emul.c       | 1 +
 2 files changed, 2 insertions(+)
arch/arm64/include/asm/sysreg.h +1 -0
index 16b3f1a1d468..615dd6278f8b 100644
@@ -937,6 +937,7 @@
 #define ID_AA64DFR0_PMUVER_8_1		0x4
 #define ID_AA64DFR0_PMUVER_8_4		0x5
 #define ID_AA64DFR0_PMUVER_8_5		0x6
+#define ID_AA64DFR0_PMUVER_8_7		0x7
 #define ID_AA64DFR0_PMUVER_IMP_DEF	0xf
 
 #define ID_AA64DFR0_PMSVER_8_2		0x1
arch/arm64/kvm/pmu-emul.c +1 -0
index a5e4bbf5e68f..ca92cc5c71c6 100644
@@ -28,6 +28,7 @@ static u32 kvm_pmu_event_mask(struct kvm *kvm)
 	case ID_AA64DFR0_PMUVER_8_1:
 	case ID_AA64DFR0_PMUVER_8_4:
 	case ID_AA64DFR0_PMUVER_8_5:
+	case ID_AA64DFR0_PMUVER_8_7:
 		return GENMASK(15, 0);
 	default:		/* Shouldn't be here, just for sanity */
 		WARN_ONCE(1, "Unknown PMU version %d\n", kvm->arch.pmuver);
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 3/69
#6 lore raw

Re: [PATCH v5 03/69] KVM: arm64: Add minimal handling for the ARMv8.7 PMU

Quoted 10 lines · Marc Zyngier

When running a KVM guest hosted on an ARMv8.7 machine, the host kernel complains that it doesn't know about the architected number of events.

Fix it by adding the PMUver code corresponding to PMUv3 for ARMv8.7.

Signature · 6 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#7 lore raw

PATCH v5 4/69 KVM: arm64: Rework kvm_pgtable initialisation

In reply to Marc Zyngier

Ganapatrao reported that the kvm_pgtable->mmu pointer is more or less hardcoded to the main S2 mmu structure, while the nested code needs it to point to other instances (as we have one instance per nested context).

Rework the initialisation of the kvm_pgtable structure so that this assumtion doesn't hold true anymore. This requires some minor changes to the order in which things are initialised (the mmu->arch pointer being the critical one).

---
 arch/arm64/include/asm/kvm_pgtable.h  | 9 ++++-----
 arch/arm64/kvm/hyp/nvhe/mem_protect.c | 4 ++--
 arch/arm64/kvm/hyp/pgtable.c          | 6 +++---
 arch/arm64/kvm/mmu.c                  | 4 ++--
 4 files changed, 11 insertions(+), 12 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +4 -5
index 027783829584..8b808e70c2b2 100644
@@ -270,8 +270,7 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift);
 /**
  * __kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table.
  * @pgt:	Uninitialised page-table structure to initialise.
- * @arch:	Arch-specific KVM structure representing the guest virtual
- *		machine.
+ * @mmu:	S2 MMU context for this S2 translation
  * @mm_ops:	Memory management callbacks.
  * @flags:	Stage-2 configuration flags.
  * @force_pte_cb: Function that returns true if page level mappings must
@@ -279,13 +278,13 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift);
  *
  * Return: 0 on success, negative error code on failure.
  */
-int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch,
+int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
 			      struct kvm_pgtable_mm_ops *mm_ops,
 			      enum kvm_pgtable_stage2_flags flags,
 			      kvm_pgtable_force_pte_cb_t force_pte_cb);
 
-#define kvm_pgtable_stage2_init(pgt, arch, mm_ops) \
-	__kvm_pgtable_stage2_init(pgt, arch, mm_ops, 0, NULL)
+#define kvm_pgtable_stage2_init(pgt, mmu, mm_ops) \
+	__kvm_pgtable_stage2_init(pgt, mmu, mm_ops, 0, NULL)
 
 /**
  * kvm_pgtable_stage2_destroy() - Destroy an unused guest stage-2 page-table.
arch/arm64/kvm/hyp/nvhe/mem_protect.c +2 -2
index c1a90dd022b8..7d7b7037dc68 100644
@@ -103,19 +103,19 @@ int kvm_host_prepare_stage2(void *pgt_pool_base)
 
 	prepare_host_vtcr();
 	hyp_spin_lock_init(&host_kvm.lock);
+	mmu->arch = &host_kvm.arch;
 
 	ret = prepare_s2_pool(pgt_pool_base);
 	if (ret)
 		return ret;
 
-	ret = __kvm_pgtable_stage2_init(&host_kvm.pgt, &host_kvm.arch,
+	ret = __kvm_pgtable_stage2_init(&host_kvm.pgt, mmu,
 					&host_kvm.mm_ops, KVM_HOST_S2_FLAGS,
 					host_stage2_force_pte_cb);
 	if (ret)
 		return ret;
 
 	mmu->pgd_phys = __hyp_pa(host_kvm.pgt.pgd);
-	mmu->arch = &host_kvm.arch;
 	mmu->pgt = &host_kvm.pgt;
 	WRITE_ONCE(mmu->vmid.vmid_gen, 0);
 	WRITE_ONCE(mmu->vmid.vmid, 0);
arch/arm64/kvm/hyp/pgtable.c +3 -3
index f8ceebe4982e..8cdbc43fa651 100644
@@ -1116,13 +1116,13 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size)
 }
 
 
-int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch,
+int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
 			      struct kvm_pgtable_mm_ops *mm_ops,
 			      enum kvm_pgtable_stage2_flags flags,
 			      kvm_pgtable_force_pte_cb_t force_pte_cb)
 {
 	size_t pgd_sz;
-	u64 vtcr = arch->vtcr;
+	u64 vtcr = mmu->arch->vtcr;
 	u32 ia_bits = VTCR_EL2_IPA(vtcr);
 	u32 sl0 = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr);
 	u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0;
@@ -1135,7 +1135,7 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch,
 	pgt->ia_bits		= ia_bits;
 	pgt->start_level	= start_level;
 	pgt->mm_ops		= mm_ops;
-	pgt->mmu		= &arch->mmu;
+	pgt->mmu		= mmu;
 	pgt->flags		= flags;
 	pgt->force_pte_cb	= force_pte_cb;
 
arch/arm64/kvm/mmu.c +2 -2
index 326cdfec74a1..9eec548fccd1 100644
@@ -516,7 +516,8 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
 	if (!pgt)
 		return -ENOMEM;
 
-	err = kvm_pgtable_stage2_init(pgt, &kvm->arch, &kvm_s2_mm_ops);
+	mmu->arch = &kvm->arch;
+	err = kvm_pgtable_stage2_init(pgt, mmu, &kvm_s2_mm_ops);
 	if (err)
 		goto out_free_pgtable;
 
@@ -529,7 +530,6 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
 	for_each_possible_cpu(cpu)
 		*per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1;
 
-	mmu->arch = &kvm->arch;
 	mmu->pgt = pgt;
 	mmu->pgd_phys = __pa(pgt->pgd);
 	WRITE_ONCE(mmu->vmid.vmid_gen, 0);
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#8 lore raw

[irqchip: irq/irqchip-next] KVM: arm64: Rework kvm_pgtable initialisation

The following commit has been merged into the irq/irqchip-next branch of irqchip:

The following commit has been merged into the irq/irqchip-next branch of irqchip:

Commit-ID:     9d8604b28575ccab3afd8d6f56cab9a6c0d281ef
Gitweb:        https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/9d8604b28575ccab3afd8d6f56cab9a6c0d281ef
Author:        Marc Zyngier <[email protected]>
AuthorDate:    Mon, 29 Nov 2021 20:00:45 
Committer:     Marc Zyngier <[email protected]>
CommitterDate: Thu, 16 Dec 2021 17:01:05 

KVM: arm64: Rework kvm_pgtable initialisation

Ganapatrao reported that the kvm_pgtable->mmu pointer is more or less hardcoded to the main S2 mmu structure, while the nested code needs it to point to other instances (as we have one instance per nested context).

Rework the initialisation of the kvm_pgtable structure so that this assumtion doesn't hold true anymore. This requires some minor changes to the order in which things are initialised (the mmu->arch pointer being the critical one).

---
 arch/arm64/include/asm/kvm_pgtable.h  |  9 ++++-----
 arch/arm64/kvm/hyp/nvhe/mem_protect.c |  4 ++--
 arch/arm64/kvm/hyp/pgtable.c          |  6 +++---
 arch/arm64/kvm/mmu.c                  |  4 ++--
 4 files changed, 11 insertions(+), 12 deletions(-)
arch/arm64/include/asm/kvm_pgtable.h +4 -5
index 0277838..8b808e7 100644
@@ -270,8 +270,7 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift);
 /**
  * __kvm_pgtable_stage2_init() - Initialise a guest stage-2 page-table.
  * @pgt:	Uninitialised page-table structure to initialise.
- * @arch:	Arch-specific KVM structure representing the guest virtual
- *		machine.
+ * @mmu:	S2 MMU context for this S2 translation
  * @mm_ops:	Memory management callbacks.
  * @flags:	Stage-2 configuration flags.
  * @force_pte_cb: Function that returns true if page level mappings must
@@ -279,13 +278,13 @@ u64 kvm_get_vtcr(u64 mmfr0, u64 mmfr1, u32 phys_shift);
  *
  * Return: 0 on success, negative error code on failure.
  */
-int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch,
+int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
 			      struct kvm_pgtable_mm_ops *mm_ops,
 			      enum kvm_pgtable_stage2_flags flags,
 			      kvm_pgtable_force_pte_cb_t force_pte_cb);
 
-#define kvm_pgtable_stage2_init(pgt, arch, mm_ops) \
-	__kvm_pgtable_stage2_init(pgt, arch, mm_ops, 0, NULL)
+#define kvm_pgtable_stage2_init(pgt, mmu, mm_ops) \
+	__kvm_pgtable_stage2_init(pgt, mmu, mm_ops, 0, NULL)
 
 /**
  * kvm_pgtable_stage2_destroy() - Destroy an unused guest stage-2 page-table.
arch/arm64/kvm/hyp/nvhe/mem_protect.c +2 -2
index c1a90dd..7d7b703 100644
@@ -103,19 +103,19 @@ int kvm_host_prepare_stage2(void *pgt_pool_base)
 
 	prepare_host_vtcr();
 	hyp_spin_lock_init(&host_kvm.lock);
+	mmu->arch = &host_kvm.arch;
 
 	ret = prepare_s2_pool(pgt_pool_base);
 	if (ret)
 		return ret;
 
-	ret = __kvm_pgtable_stage2_init(&host_kvm.pgt, &host_kvm.arch,
+	ret = __kvm_pgtable_stage2_init(&host_kvm.pgt, mmu,
 					&host_kvm.mm_ops, KVM_HOST_S2_FLAGS,
 					host_stage2_force_pte_cb);
 	if (ret)
 		return ret;
 
 	mmu->pgd_phys = __hyp_pa(host_kvm.pgt.pgd);
-	mmu->arch = &host_kvm.arch;
 	mmu->pgt = &host_kvm.pgt;
 	WRITE_ONCE(mmu->vmid.vmid_gen, 0);
 	WRITE_ONCE(mmu->vmid.vmid, 0);
arch/arm64/kvm/hyp/pgtable.c +3 -3
index f8ceebe..8cdbc43 100644
@@ -1116,13 +1116,13 @@ int kvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size)
 }
 
 
-int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch,
+int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_s2_mmu *mmu,
 			      struct kvm_pgtable_mm_ops *mm_ops,
 			      enum kvm_pgtable_stage2_flags flags,
 			      kvm_pgtable_force_pte_cb_t force_pte_cb)
 {
 	size_t pgd_sz;
-	u64 vtcr = arch->vtcr;
+	u64 vtcr = mmu->arch->vtcr;
 	u32 ia_bits = VTCR_EL2_IPA(vtcr);
 	u32 sl0 = FIELD_GET(VTCR_EL2_SL0_MASK, vtcr);
 	u32 start_level = VTCR_EL2_TGRAN_SL0_BASE - sl0;
@@ -1135,7 +1135,7 @@ int __kvm_pgtable_stage2_init(struct kvm_pgtable *pgt, struct kvm_arch *arch,
 	pgt->ia_bits		= ia_bits;
 	pgt->start_level	= start_level;
 	pgt->mm_ops		= mm_ops;
-	pgt->mmu		= &arch->mmu;
+	pgt->mmu		= mmu;
 	pgt->flags		= flags;
 	pgt->force_pte_cb	= force_pte_cb;
 
arch/arm64/kvm/mmu.c +2 -2
index 326cdfe..9eec548 100644
@@ -516,7 +516,8 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
 	if (!pgt)
 		return -ENOMEM;
 
-	err = kvm_pgtable_stage2_init(pgt, &kvm->arch, &kvm_s2_mm_ops);
+	mmu->arch = &kvm->arch;
+	err = kvm_pgtable_stage2_init(pgt, mmu, &kvm_s2_mm_ops);
 	if (err)
 		goto out_free_pgtable;
 
@@ -529,7 +530,6 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
 	for_each_possible_cpu(cpu)
 		*per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1;
 
-	mmu->arch = &kvm->arch;
 	mmu->pgt = pgt;
 	mmu->pgd_phys = __pa(pgt->pgd);
 	WRITE_ONCE(mmu->vmid.vmid_gen, 0);
Reviewed-by on v5 4/69
#9 lore raw

Re: [PATCH v5 04/69] KVM: arm64: Rework kvm_pgtable initialisation

In reply to Marc Zyngier
Quoted 5 lines · Marc Zyngier

Ganapatrao reported that the kvm_pgtable->mmu pointer is more or less hardcoded to the main S2 mmu structure, while the nested code needs it to point to other instances (as we have one instance per nested context).

Quoted 8 lines

Rework the initialisation of the kvm_pgtable structure so that this assumtion doesn't hold true anymore. This requires some minor changes to the order in which things are initialised (the mmu->arch pointer being the critical one).

Looks fairly simple.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
author maintainer
#10 lore raw

PATCH v5 5/69 KVM: arm64: Allow preservation of the S2 SW bits

In reply to Marc Zyngier

The S2 page table code has a limited use the SW bits, but we are about to need them to encode some guest Stage-2 information (its mapping size in the form of the TTL encoding).

Propagate the SW bits specified by the caller, and store them into the corresponding entry.

---
 arch/arm64/kvm/hyp/pgtable.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +4 -3
index 8cdbc43fa651..d69e400b2de6 100644
@@ -1064,9 +1064,6 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
 	u32 level;
 	kvm_pte_t set = 0, clr = 0;
 
-	if (prot & KVM_PTE_LEAF_ATTR_HI_SW)
-		return -EINVAL;
-
 	if (prot & KVM_PGTABLE_PROT_R)
 		set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
 
@@ -1076,6 +1073,10 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
 	if (prot & KVM_PGTABLE_PROT_X)
 		clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
 
+	/* Always propagate the SW bits */
+	clr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_SW, 0xf);
+	set |= prot & KVM_PTE_LEAF_ATTR_HI_SW;
+
 	ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level);
 	if (!ret)
 		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, level);
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 5/69
#11 lore raw

Re: [PATCH v5 05/69] KVM: arm64: Allow preservation of the S2 SW bits

Hi Marc,

Quoted 27 lines · Marc Zyngier

The S2 page table code has a limited use the SW bits, but we are about to need them to encode some guest Stage-2 information (its mapping size in the form of the TTL encoding).

Propagate the SW bits specified by the caller, and store them into the corresponding entry.

---
 arch/arm64/kvm/hyp/pgtable.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +0 -3
index 8cdbc43fa651..d69e400b2de6 100644
@@ -1064,9 +1064,6 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
 	u32 level;
 	kvm_pte_t set = 0, clr = 0;
 
-	if (prot & KVM_PTE_LEAF_ATTR_HI_SW)
-		return -EINVAL;
-
 	if (prot & KVM_PGTABLE_PROT_R)
 		set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
 

arch/arm64/kvm/hyp/pgtable.c int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, · line 1077

Quoted 5 lines
arch/arm64/kvm/hyp/pgtable.c +2 -0
@@ -1076,6 +1073,10 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
 	if (prot & KVM_PGTABLE_PROT_X)
 		clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
 
+	/* Always propagate the SW bits */
+	clr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_SW, 0xf);

Nitpick: isn't that the same as:

	clr |= KVM_PTE_LEAF_ATTR_HI_SW;

which looks more readable to me.

arch/arm64/kvm/hyp/pgtable.c int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, · line 1078

Quoted 1 line
arch/arm64/kvm/hyp/pgtable.c +1 -0
+	set |= prot & KVM_PTE_LEAF_ATTR_HI_SW;

Checked stage2_attr_walker() callbak, first it clears the bits in clr, then sets the bits in set, so this looks correct to me:

Thanks,
Alex

Quoted 7 lines
arch/arm64/kvm/hyp/pgtable.c +1 -0
+
 	ret = stage2_update_leaf_attrs(pgt, addr, 1, set, clr, NULL, &level);
 	if (!ret)
 		kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, pgt->mmu, addr, level);
Signature · 1 line
2.30.2
author maintainer
#12 lore raw

Re: [PATCH v5 05/69] KVM: arm64: Allow preservation of the S2 SW bits

Hi Alex,

Quoted 41 lines · Alexandru Elisei

Hi Marc,

Quoted 27 lines · Marc Zyngier

The S2 page table code has a limited use the SW bits, but we are about to need them to encode some guest Stage-2 information (its mapping size in the form of the TTL encoding).

Propagate the SW bits specified by the caller, and store them into the corresponding entry.

---
 arch/arm64/kvm/hyp/pgtable.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
arch/arm64/kvm/hyp/pgtable.c +0 -3
index 8cdbc43fa651..d69e400b2de6 100644
@@ -1064,9 +1064,6 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
 	u32 level;
 	kvm_pte_t set = 0, clr = 0;
 
-	if (prot & KVM_PTE_LEAF_ATTR_HI_SW)
-		return -EINVAL;
-
 	if (prot & KVM_PGTABLE_PROT_R)
 		set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_R;
 

arch/arm64/kvm/hyp/pgtable.c int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, · line 1077

Quoted 5 lines
arch/arm64/kvm/hyp/pgtable.c +2 -0
@@ -1076,6 +1073,10 @@ int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr,
 	if (prot & KVM_PGTABLE_PROT_X)
 		clr |= KVM_PTE_LEAF_ATTR_HI_S2_XN;
 
+	/* Always propagate the SW bits */
+	clr |= FIELD_PREP(KVM_PTE_LEAF_ATTR_HI_SW, 0xf);

Nitpick: isn't that the same as:

	clr |= KVM_PTE_LEAF_ATTR_HI_SW;
Quoted 8 lines

which looks more readable to me.

arch/arm64/kvm/hyp/pgtable.c int kvm_pgtable_stage2_relax_perms(struct kvm_pgtable *pgt, u64 addr, · line 1078

Quoted 1 line
arch/arm64/kvm/hyp/pgtable.c +1 -0
+	set |= prot & KVM_PTE_LEAF_ATTR_HI_SW;

Checked stage2_attr_walker() callbak, first it clears the bits in clr, then sets the bits in set, so this looks correct to me:

Thanks. However, I have now dropped this patch since as it turns out, the PTE update does preserve pre-existing SW bits. I am now carrying this:

@@ -1212,6 +1218,11 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,

         * kvm_pgtable_stage2_map() should be called to change block size.
         */
        if (fault_status == FSC_PERM && vma_pagesize == fault_granule) {
+               /*
+                * Drop the SW bits in favour of those stored in the
+                * PTE, which will be preserved.
+                */
+               prot &= ~KVM_NV_GUEST_MAP_SZ;
                ret = kvm_pgtable_stage2_relax_perms(pgt, fault_ipa, prot);
        } else {
                ret = kvm_pgtable_stage2_map(pgt, fault_ipa, vma_pagesize,

as part of the patch that tags the shadow S2 with the guest's S2 mapping size (or level, which amounts to the same thing).

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Reviewed-by on v5 5/69
#13 lore raw

Re: [PATCH v5 05/69] KVM: arm64: Allow preservation of the S2 SW bits

In reply to Marc Zyngier
Quoted 1 line · Marc Zyngier

The S2 page table code has a limited use the SW bits, but we are about

I think the opening clause needs to be rewritten.

Quoted 7 lines

to need them to encode some guest Stage-2 information (its mapping size in the form of the TTL encoding).

Propagate the SW bits specified by the caller, and store them into the corresponding entry.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
author maintainer
#14 lore raw

PATCH v5 6/69 arm64: Add ARM64_HAS_NESTED_VIRT cpufeature

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

Add a new ARM64_HAS_NESTED_VIRT feature to indicate that the CPU has the ARMv8.3 nested virtualization capability, together with the 'kvm-arm.mode=nested' command line option.

This will be used to support nested virtualization in KVM.

[maz: moved the command-line option to kvm-arm.mode]
---
 .../admin-guide/kernel-parameters.txt         |  7 +++++-
 arch/arm64/include/asm/kvm_host.h             |  5 ++++
 arch/arm64/kernel/cpufeature.c                | 24 +++++++++++++++++++
 arch/arm64/kvm/arm.c                          |  5 ++++
 arch/arm64/tools/cpucaps                      |  1 +
 5 files changed, 41 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +6 -1
index 9725c546a0d4..008d567b042f 100644
@@ -2380,9 +2380,14 @@
 				   state is kept private from the host.
 				   Not valid if the kernel is running in EL2.
 
+			nested: VHE-based mode with support for nested
+				virtualization. Requires at least ARMv8.3
+				hardware.
+
 			Defaults to VHE/nVHE based on hardware support. Setting
 			mode to "protected" will disable kexec and hibernation
-			for the host.
+			for the host. "nested" is experimental and should be
+			used with extreme caution.
 
 	kvm-arm.vgic_v3_group0_trap=
 			[KVM,ARM] Trap guest accesses to GICv3 group-0
arch/arm64/include/asm/kvm_host.h +5 -0
index 2a5f7f38006f..20d7cb450ef3 100644
@@ -58,9 +58,14 @@
 enum kvm_mode {
 	KVM_MODE_DEFAULT,
 	KVM_MODE_PROTECTED,
+	KVM_MODE_NV,
 	KVM_MODE_NONE,
 };
+#ifdef CONFIG_KVM
 enum kvm_mode kvm_get_mode(void);
+#else
+static inline enum kvm_mode kvm_get_mode(void) { return KVM_MODE_NONE; };
+#endif
 
 DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
 
arch/arm64/kernel/cpufeature.c +24 -0
index 6f3e677d88f1..4d57e98cdde5 100644
@@ -1761,6 +1761,20 @@ static void cpu_copy_el2regs(const struct arm64_cpu_capabilities *__unused)
 		write_sysreg(read_sysreg(tpidr_el1), tpidr_el2);
 }
 
+static bool has_nested_virt_support(const struct arm64_cpu_capabilities *cap,
+				    int scope)
+{
+	if (kvm_get_mode() != KVM_MODE_NV)
+		return false;
+
+	if (!has_cpuid_feature(cap, scope)) {
+		pr_warn("unavailable: %s\n", cap->desc);
+		return false;
+	}
+
+	return true;
+}
+
 static void cpu_has_fwb(const struct arm64_cpu_capabilities *__unused)
 {
 	u64 val = read_sysreg_s(SYS_CLIDR_EL1);
@@ -1994,6 +2008,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.matches = runs_at_el2,
 		.cpu_enable = cpu_copy_el2regs,
 	},
+	{
+		.desc = "Nested Virtualization Support",
+		.capability = ARM64_HAS_NESTED_VIRT,
+		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
+		.matches = has_nested_virt_support,
+		.sys_reg = SYS_ID_AA64MMFR2_EL1,
+		.sign = FTR_UNSIGNED,
+		.field_pos = ID_AA64MMFR2_NV_SHIFT,
+		.min_field_value = 1,
+	},
 	{
 		.capability = ARM64_HAS_32BIT_EL0_DO_NOT_USE,
 		.type = ARM64_CPUCAP_SYSTEM_FEATURE,
arch/arm64/kvm/arm.c +5 -0
index 2f03cbfefe67..68da54d58cd0 100644
@@ -2174,6 +2174,11 @@ static int __init early_kvm_mode_cfg(char *arg)
 		return 0;
 	}
 
+	if (strcmp(arg, "nested") == 0 && !WARN_ON(!is_kernel_in_hyp_mode())) {
+		kvm_mode = KVM_MODE_NV;
+		return 0;
+	}
+
 	if (strcmp(arg, "none") == 0) {
 		kvm_mode = KVM_MODE_NONE;
 		return 0;
arch/arm64/tools/cpucaps +1 -0
index 870c39537dd0..a49864b56a07 100644
@@ -26,6 +26,7 @@ HAS_GENERIC_AUTH_IMP_DEF
 HAS_IRQ_PRIO_MASKING
 HAS_LDAPR
 HAS_LSE_ATOMICS
+HAS_NESTED_VIRT
 HAS_NO_FPSIMD
 HAS_NO_HW_PREFETCH
 HAS_PAN
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#15 lore raw

PATCH v5 7/69 KVM: arm64: nv: Introduce nested virtualization VCPU feature

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

Introduce the feature bit and a primitive that checks if the feature is set behind a static key check based on the cpus_have_const_cap check.

Checking nested_virt_in_use() on systems without nested virt enabled should have neglgible overhead.

We don't yet allow userspace to actually set this feature.

---
 arch/arm64/include/asm/kvm_nested.h | 14 ++++++++++++++
 arch/arm64/include/uapi/asm/kvm.h   |  1 +
 2 files changed, 15 insertions(+)
 create mode 100644 arch/arm64/include/asm/kvm_nested.h
arch/arm64/include/asm/kvm_nested.h new file +14 -0
new file mode 100644
index 000000000000..1028ac65a897
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ARM64_KVM_NESTED_H
+#define __ARM64_KVM_NESTED_H
+
+#include <linux/kvm_host.h>
+
+static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu)
+{
+	return (!__is_defined(__KVM_NVHE_HYPERVISOR__) &&
+		cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) &&
+		test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));
+}
+
+#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/include/uapi/asm/kvm.h +1 -0
index b3edde68bc3e..395a4c039bcc 100644
@@ -106,6 +106,7 @@ struct kvm_regs {
 #define KVM_ARM_VCPU_SVE		4 /* enable SVE for this CPU */
 #define KVM_ARM_VCPU_PTRAUTH_ADDRESS	5 /* VCPU uses address authentication */
 #define KVM_ARM_VCPU_PTRAUTH_GENERIC	6 /* VCPU uses generic authentication */
+#define KVM_ARM_VCPU_HAS_EL2		7 /* Support nested virtualization */
 
 struct kvm_vcpu_init {
 	__u32 target;
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 7/69
#16 lore raw

Re: [PATCH v5 07/69] KVM: arm64: nv: Introduce nested virtualization VCPU feature

Hi Marc,

Quoted 7 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

Introduce the feature bit and a primitive that checks if the feature is set behind a static key check based on the cpus_have_const_cap check.

Checking nested_virt_in_use() on systems without nested virt enabled should have neglgible overhead.

Typo: negligible

Quoted 36 lines

We don't yet allow userspace to actually set this feature.

---
  arch/arm64/include/asm/kvm_nested.h | 14 ++++++++++++++
  arch/arm64/include/uapi/asm/kvm.h   |  1 +
  2 files changed, 15 insertions(+)

create mode 100644 arch/arm64/include/asm/kvm_nested.h

arch/arm64/include/asm/kvm_nested.h new file +14 -0
new file mode 100644
index 000000000000..1028ac65a897
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ARM64_KVM_NESTED_H
+#define __ARM64_KVM_NESTED_H
+
+#include <linux/kvm_host.h>
+
+static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu)
+{
+	return (!__is_defined(__KVM_NVHE_HYPERVISOR__) &&
+		cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) &&
+		test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));
+}
+
+#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/include/uapi/asm/kvm.h +0 -0
index b3edde68bc3e..395a4c039bcc 100644

arch/arm64/include/uapi/asm/kvm.h struct kvm_regs { · line 112

Quoted 7 lines
arch/arm64/include/uapi/asm/kvm.h +1 -0
@@ -106,6 +106,7 @@ struct kvm_regs {
  #define KVM_ARM_VCPU_SVE		4 /* enable SVE for this CPU */
  #define KVM_ARM_VCPU_PTRAUTH_ADDRESS	5 /* VCPU uses address authentication */
  #define KVM_ARM_VCPU_PTRAUTH_GENERIC	6 /* VCPU uses generic authentication */
+#define KVM_ARM_VCPU_HAS_EL2		7 /* Support nested virtualization */
 
  struct kvm_vcpu_init {
  	__u32 target;

It looks good to me.
Please feel free to add.

Thanks,
Ganapat

#17 lore raw

Re: [PATCH v5 07/69] KVM: arm64: nv: Introduce nested virtualization VCPU feature

In reply to Marc Zyngier

Hi Marc,

Quoted 25 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

Introduce the feature bit and a primitive that checks if the feature is set behind a static key check based on the cpus_have_const_cap check.

Checking nested_virt_in_use() on systems without nested virt enabled should have neglgible overhead.

We don't yet allow userspace to actually set this feature.

---
 arch/arm64/include/asm/kvm_nested.h | 14 ++++++++++++++
 arch/arm64/include/uapi/asm/kvm.h   |  1 +
 2 files changed, 15 insertions(+)
 create mode 100644 arch/arm64/include/asm/kvm_nested.h
arch/arm64/include/asm/kvm_nested.h new file +1 -0
new file mode 100644
index 000000000000..1028ac65a897
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */

arch/arm64/include/asm/kvm_nested.h line 11

Quoted 10 lines
arch/arm64/include/asm/kvm_nested.h new file +10 -0
+#ifndef __ARM64_KVM_NESTED_H
+#define __ARM64_KVM_NESTED_H
+
+#include <linux/kvm_host.h>
+
+static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu)
+{
+	return (!__is_defined(__KVM_NVHE_HYPERVISOR__) &&
+		cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) &&
+		test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));

kvm_vcpu_init_nested() checks the ARM64_HAS_NESTED_VIRT cap before setting the features bit, so I guess you can drop this check here if you're interested in correctness.

But the reason the cap check done is performance, right? Same with the NVHE define.

Thanks,
Alex

Quoted 18 lines
arch/arm64/include/asm/kvm_nested.h new file +3 -0
+}
+
+#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/include/uapi/asm/kvm.h +1 -0
index b3edde68bc3e..395a4c039bcc 100644
@@ -106,6 +106,7 @@ struct kvm_regs {
 #define KVM_ARM_VCPU_SVE		4 /* enable SVE for this CPU */
 #define KVM_ARM_VCPU_PTRAUTH_ADDRESS	5 /* VCPU uses address authentication */
 #define KVM_ARM_VCPU_PTRAUTH_GENERIC	6 /* VCPU uses generic authentication */
+#define KVM_ARM_VCPU_HAS_EL2		7 /* Support nested virtualization */
 
 struct kvm_vcpu_init {
 	__u32 target;
Signature · 1 line
2.30.2
author maintainer
#18 lore raw

Re: [PATCH v5 07/69] KVM: arm64: nv: Introduce nested virtualization VCPU feature

Quoted 44 lines · Alexandru Elisei

Hi Marc,

Quoted 25 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

Introduce the feature bit and a primitive that checks if the feature is set behind a static key check based on the cpus_have_const_cap check.

Checking nested_virt_in_use() on systems without nested virt enabled should have neglgible overhead.

We don't yet allow userspace to actually set this feature.

---
 arch/arm64/include/asm/kvm_nested.h | 14 ++++++++++++++
 arch/arm64/include/uapi/asm/kvm.h   |  1 +
 2 files changed, 15 insertions(+)
 create mode 100644 arch/arm64/include/asm/kvm_nested.h
arch/arm64/include/asm/kvm_nested.h new file +1 -0
new file mode 100644
index 000000000000..1028ac65a897
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: GPL-2.0 */

arch/arm64/include/asm/kvm_nested.h line 11

Quoted 10 lines
arch/arm64/include/asm/kvm_nested.h new file +10 -0
+#ifndef __ARM64_KVM_NESTED_H
+#define __ARM64_KVM_NESTED_H
+
+#include <linux/kvm_host.h>
+
+static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu)
+{
+	return (!__is_defined(__KVM_NVHE_HYPERVISOR__) &&
+		cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) &&
+		test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));

kvm_vcpu_init_nested() checks the ARM64_HAS_NESTED_VIRT cap before setting the features bit, so I guess you can drop this check here if you're interested in correctness.

Quoted 2 lines

But the reason the cap check done is performance, right? Same with the NVHE define.

Exactly. The capability check allows us to use a static key to bypass the test_bit() when NV isn't enable, which is 100% of our use cases. Given that this checked fairly often on some of the fast paths, it is worth doing it.

The NVHE check goes even further by allowing some dead code removal in the nVHE-specific code, which cannot use NV by construction.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Reviewed-by on v5 7/69
#19 lore raw

Re: [PATCH v5 07/69] KVM: arm64: nv: Introduce nested virtualization VCPU feature

In reply to Marc Zyngier
Quoted 5 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

Introduce the feature bit and a primitive that checks if the feature is set behind a static key check based on the cpus_have_const_cap check.

Quoted 7 lines

Checking nested_virt_in_use() on systems without nested virt enabled should have neglgible overhead.

We don't yet allow userspace to actually set this feature.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
author maintainer
#20 lore raw

PATCH v5 8/69 KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

Reset the VCPU with PSTATE.M = EL2h when the nested virtualization feature is enabled on the VCPU.

[maz: rework register reset not to use empty data structures]
---
 arch/arm64/kvm/reset.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/arm64/kvm/reset.c +8 -2
index 426bd7fbc3fd..38a7182819fb 100644
@@ -27,6 +27,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
 #include <asm/virt.h>
 
 /* Maximum phys_shift supported for any VM on this host */
@@ -38,6 +39,9 @@ static u32 kvm_ipa_limit;
 #define VCPU_RESET_PSTATE_EL1	(PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
 				 PSR_F_BIT | PSR_D_BIT)
 
+#define VCPU_RESET_PSTATE_EL2	(PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
+				 PSR_F_BIT | PSR_D_BIT)
+
 #define VCPU_RESET_PSTATE_SVC	(PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
 				 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
 
@@ -176,8 +180,8 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 	if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)
 		return false;
 
-	/* MTE is incompatible with AArch32 */
-	if (kvm_has_mte(vcpu->kvm) && is32bit)
+	/* MTE and NV are incompatible with AArch32 */
+	if ((kvm_has_mte(vcpu->kvm) || nested_virt_in_use(vcpu)) && is32bit)
 		return false;
 
 	/* Check that the vcpus are either all 32bit or all 64bit */
@@ -255,6 +259,8 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 	default:
 		if (test_bit(KVM_ARM_VCPU_EL1_32BIT, vcpu->arch.features)) {
 			pstate = VCPU_RESET_PSTATE_SVC;
+		} else if (nested_virt_in_use(vcpu)) {
+			pstate = VCPU_RESET_PSTATE_EL2;
 		} else {
 			pstate = VCPU_RESET_PSTATE_EL1;
 		}
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#21 lore raw

Re: [PATCH v5 08/69] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set

Hi Marc,

Quoted 35 lines · Marc Zyngier

From: Christoffer Dall <christoffer.dall at arm.com>

Reset the VCPU with PSTATE.M = EL2h when the nested virtualization feature is enabled on the VCPU.

Signed-off-by Christoffer Dall <christoffer.dall at arm.com>
[maz: rework register reset not to use empty data structures]
Signed-off-by Marc Zyngier <maz at kernel.org>
---
 arch/arm64/kvm/reset.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/arm64/kvm/reset.c +4 -0
index 426bd7fbc3fd..38a7182819fb 100644
@@ -27,6 +27,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
 #include <asm/virt.h>
 
 /* Maximum phys_shift supported for any VM on this host */
@@ -38,6 +39,9 @@ static u32 kvm_ipa_limit;
 #define VCPU_RESET_PSTATE_EL1	(PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
 				 PSR_F_BIT | PSR_D_BIT)
 
+#define VCPU_RESET_PSTATE_EL2	(PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
+				 PSR_F_BIT | PSR_D_BIT)
+
 #define VCPU_RESET_PSTATE_SVC	(PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
 				 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
 

arch/arm64/kvm/reset.c static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu) · line 185

Quoted 8 lines
arch/arm64/kvm/reset.c +2 -2
@@ -176,8 +180,8 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 	if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)
 		return false;
 
-	/* MTE is incompatible with AArch32 */
-	if (kvm_has_mte(vcpu->kvm) && is32bit)
+	/* MTE and NV are incompatible with AArch32 */
+	if ((kvm_has_mte(vcpu->kvm) || nested_virt_in_use(vcpu)) && is32bit)
 		return false;

Should something similar be done for SVE? I see from the ID register emulation that SVE is hidden from the guest but there isn't anything in kvm_vcpu_enable_sve() that checks if NV is in use. That means it's possible to have both nested_virt_in_use(vcpu) and vcpu_has_sve(vcpu) be true simultaneously. If that happens, the FPSIMD fixup can get confused

	/*
	 * Don't handle SVE traps for non-SVE vcpus here. This
	 * includes NV guests for the time being.
	 */
	if (!sve_guest && (esr_ec != ESR_ELx_EC_FP_ASIMD ||
			   guest_hyp_fpsimd_traps_enabled(vcpu)))
		return false;

and incorrectly restore the wrong context instead of forwarding a FPSIMD trap to the guest hypervisor.

Thanks,
Chase

Quoted 12 lines
arch/arm64/kvm/reset.c +2 -0
 	/* Check that the vcpus are either all 32bit or all 64bit */
@@ -255,6 +259,8 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 	default:
 		if (test_bit(KVM_ARM_VCPU_EL1_32BIT, vcpu->arch.features)) {
 			pstate = VCPU_RESET_PSTATE_SVC;
+		} else if (nested_virt_in_use(vcpu)) {
+			pstate = VCPU_RESET_PSTATE_EL2;
 		} else {
 			pstate = VCPU_RESET_PSTATE_EL1;
 		}
Signature · 1 line
2.30.2
author maintainer
#22 lore raw

Re: [PATCH v5 08/69] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set

Quoted 62 lines · Chase Conklin

Hi Marc,

Quoted 35 lines · Marc Zyngier

From: Christoffer Dall <christoffer.dall at arm.com>

Reset the VCPU with PSTATE.M = EL2h when the nested virtualization feature is enabled on the VCPU.

Signed-off-by Christoffer Dall <christoffer.dall at arm.com>
[maz: rework register reset not to use empty data structures]
Signed-off-by Marc Zyngier <maz at kernel.org>
---
 arch/arm64/kvm/reset.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/arm64/kvm/reset.c +4 -0
index 426bd7fbc3fd..38a7182819fb 100644
@@ -27,6 +27,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
 #include <asm/virt.h>
 
 /* Maximum phys_shift supported for any VM on this host */
@@ -38,6 +39,9 @@ static u32 kvm_ipa_limit;
 #define VCPU_RESET_PSTATE_EL1	(PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
 				 PSR_F_BIT | PSR_D_BIT)
 
+#define VCPU_RESET_PSTATE_EL2	(PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
+				 PSR_F_BIT | PSR_D_BIT)
+
 #define VCPU_RESET_PSTATE_SVC	(PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
 				 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
 

arch/arm64/kvm/reset.c static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu) · line 185

Quoted 8 lines
arch/arm64/kvm/reset.c +2 -2
@@ -176,8 +180,8 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 	if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)
 		return false;
 
-	/* MTE is incompatible with AArch32 */
-	if (kvm_has_mte(vcpu->kvm) && is32bit)
+	/* MTE and NV are incompatible with AArch32 */
+	if ((kvm_has_mte(vcpu->kvm) || nested_virt_in_use(vcpu)) && is32bit)
 		return false;

Should something similar be done for SVE? I see from the ID register emulation that SVE is hidden from the guest but there isn't anything in kvm_vcpu_enable_sve() that checks if NV is in use. That means it's possible to have both nested_virt_in_use(vcpu) and vcpu_has_sve(vcpu) be true simultaneously. If that happens, the FPSIMD fixup can get confused

	/*
	 * Don't handle SVE traps for non-SVE vcpus here. This
	 * includes NV guests for the time being.
	 */
	if (!sve_guest && (esr_ec != ESR_ELx_EC_FP_ASIMD ||
			   guest_hyp_fpsimd_traps_enabled(vcpu)))
		return false;
Quoted 2 lines

and incorrectly restore the wrong context instead of forwarding a FPSIMD trap to the guest hypervisor.

Yes, nice catch. I have added this to kvm_reset_vcpu() to prevent the issue.

	if (nested_virt_in_use(vcpu) &&
	    vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) {
		ret = -EINVAL;
		goto out;
	}

I may also rename nested_virt_in_use() to vcpu_has_nv(), which would fit the rest of the code a bit better.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Reviewed-by on v5 8/69
#23 lore raw

Re: [PATCH v5 08/69] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set

In reply to Marc Zyngier
Quoted 8 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

Reset the VCPU with PSTATE.M = EL2h when the nested virtualization feature is enabled on the VCPU.

[maz: rework register reset not to use empty data structures]

However, a couple of comments below.

Quoted 27 lines
---
 arch/arm64/kvm/reset.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/arm64/kvm/reset.c +4 -0
index 426bd7fbc3fd..38a7182819fb 100644
@@ -27,6 +27,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
 #include <asm/virt.h>
 
 /* Maximum phys_shift supported for any VM on this host */
@@ -38,6 +39,9 @@ static u32 kvm_ipa_limit;
 #define VCPU_RESET_PSTATE_EL1	(PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
 				 PSR_F_BIT | PSR_D_BIT)
 
+#define VCPU_RESET_PSTATE_EL2	(PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
+				 PSR_F_BIT | PSR_D_BIT)
+
 #define VCPU_RESET_PSTATE_SVC	(PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
 				 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
 

arch/arm64/kvm/reset.c static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu) · line 185

Quoted 8 lines
arch/arm64/kvm/reset.c +2 -2
@@ -176,8 +180,8 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 	if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)
 		return false;
 
-	/* MTE is incompatible with AArch32 */
-	if (kvm_has_mte(vcpu->kvm) && is32bit)
+	/* MTE and NV are incompatible with AArch32 */
+	if ((kvm_has_mte(vcpu->kvm) || nested_virt_in_use(vcpu)) && is32bit)
 		return false;

It seems we have a bunch of:

	if (something && is32bit)
		return false;

tests here - would it make sense to do:

	if (is32bit) {
		if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1))
			return false;

		/* MTE is incompatible with AArch32 */
		if (kvm_has_mte(vcpu->kvm))
			return false;

		/* NV is incompatible with AArch32 */
		if (nested_virt_in_use(vcpu))
			return false;
	}

in terms of improved readability?

arch/arm64/kvm/reset.c int kvm_reset_vcpu(struct kvm_vcpu *vcpu) · line 266

Quoted 9 lines
arch/arm64/kvm/reset.c +2 -0
@@ -255,6 +259,8 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
 	default:
 		if (test_bit(KVM_ARM_VCPU_EL1_32BIT, vcpu->arch.features)) {
 			pstate = VCPU_RESET_PSTATE_SVC;
+		} else if (nested_virt_in_use(vcpu)) {
+			pstate = VCPU_RESET_PSTATE_EL2;
 		} else {
 			pstate = VCPU_RESET_PSTATE_EL1;
 		}

Not an issue with your patch, but the switch around this looks useless. The only case is this default case, so it's entirely a no-op.

Signature · 6 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#24 lore raw

Re: [PATCH v5 08/69] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set

Quoted 71 lines · Russell King (Oracle)
Quoted 8 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

Reset the VCPU with PSTATE.M = EL2h when the nested virtualization feature is enabled on the VCPU.

[maz: rework register reset not to use empty data structures]

However, a couple of comments below.

Quoted 27 lines
---
 arch/arm64/kvm/reset.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
arch/arm64/kvm/reset.c +4 -0
index 426bd7fbc3fd..38a7182819fb 100644
@@ -27,6 +27,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
 #include <asm/virt.h>
 
 /* Maximum phys_shift supported for any VM on this host */
@@ -38,6 +39,9 @@ static u32 kvm_ipa_limit;
 #define VCPU_RESET_PSTATE_EL1	(PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \
 				 PSR_F_BIT | PSR_D_BIT)
 
+#define VCPU_RESET_PSTATE_EL2	(PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \
+				 PSR_F_BIT | PSR_D_BIT)
+
 #define VCPU_RESET_PSTATE_SVC	(PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \
 				 PSR_AA32_I_BIT | PSR_AA32_F_BIT)
 

arch/arm64/kvm/reset.c static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu) · line 185

Quoted 8 lines
arch/arm64/kvm/reset.c +2 -2
@@ -176,8 +180,8 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 	if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)
 		return false;
 
-	/* MTE is incompatible with AArch32 */
-	if (kvm_has_mte(vcpu->kvm) && is32bit)
+	/* MTE and NV are incompatible with AArch32 */
+	if ((kvm_has_mte(vcpu->kvm) || nested_virt_in_use(vcpu)) && is32bit)
 		return false;

It seems we have a bunch of:

	if (something && is32bit)
		return false;

tests here - would it make sense to do:

	if (is32bit) {
		if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1))
			return false;

		/* MTE is incompatible with AArch32 */
		if (kvm_has_mte(vcpu->kvm))
			return false;

		/* NV is incompatible with AArch32 */
		if (nested_virt_in_use(vcpu))
			return false;
	}
Quoted 1 line

in terms of improved readability?

Agreed. I've now reworked to follow this pattern.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
author maintainer
#25 lore raw

PATCH v5 9/69 KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

We were not allowing userspace to set a more privileged mode for the VCPU than EL1, but we should allow this when nested virtualization is enabled for the VCPU.

---
 arch/arm64/kvm/guest.c | 6 ++++++
 1 file changed, 6 insertions(+)
arch/arm64/kvm/guest.c +6 -0
index e116c7767730..84d97f0394cb 100644
@@ -24,6 +24,7 @@
 #include <asm/fpsimd.h>
 #include <asm/kvm.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
 #include <asm/sigcontext.h>
 
 #include "trace.h"
@@ -259,6 +260,11 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
 			if (vcpu_el1_is_32bit(vcpu))
 				return -EINVAL;
 			break;
+		case PSR_MODE_EL2h:
+		case PSR_MODE_EL2t:
+			if (vcpu_el1_is_32bit(vcpu) || !nested_virt_in_use(vcpu))
+				return -EINVAL;
+			break;
 		default:
 			err = -EINVAL;
 			goto out;
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 9/69
#26 lore raw

Re: [PATCH v5 09/69] KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x

Quoted 8 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

We were not allowing userspace to set a more privileged mode for the VCPU than EL1, but we should allow this when nested virtualization is enabled for the VCPU.

Signature · 6 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#27 lore raw

PATCH v5 10/69 KVM: arm64: nv: Add EL2 system registers to vcpu context

In reply to Marc Zyngier

Add the minimal set of EL2 system registers to the vcpu context. Nothing uses them just yet.

---
 arch/arm64/include/asm/kvm_host.h | 33 ++++++++++++++++++++++++++++++-
 1 file changed, 32 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_host.h +32 -1
index 20d7cb450ef3..4f642a2e9c34 100644
@@ -221,12 +221,43 @@ enum vcpu_sysreg {
 	TFSR_EL1,	/* Tag Fault Status Register (EL1) */
 	TFSRE0_EL1,	/* Tag Fault Status Register (EL0) */
 
-	/* 32bit specific registers. Keep them at the end of the range */
+	/* 32bit specific registers. */
 	DACR32_EL2,	/* Domain Access Control Register */
 	IFSR32_EL2,	/* Instruction Fault Status Register */
 	FPEXC32_EL2,	/* Floating-Point Exception Control Register */
 	DBGVCR32_EL2,	/* Debug Vector Catch Register */
 
+	/* EL2 registers */
+	VPIDR_EL2,	/* Virtualization Processor ID Register */
+	VMPIDR_EL2,	/* Virtualization Multiprocessor ID Register */
+	SCTLR_EL2,	/* System Control Register (EL2) */
+	ACTLR_EL2,	/* Auxiliary Control Register (EL2) */
+	HCR_EL2,	/* Hypervisor Configuration Register */
+	MDCR_EL2,	/* Monitor Debug Configuration Register (EL2) */
+	CPTR_EL2,	/* Architectural Feature Trap Register (EL2) */
+	HSTR_EL2,	/* Hypervisor System Trap Register */
+	HACR_EL2,	/* Hypervisor Auxiliary Control Register */
+	TTBR0_EL2,	/* Translation Table Base Register 0 (EL2) */
+	TTBR1_EL2,	/* Translation Table Base Register 1 (EL2) */
+	TCR_EL2,	/* Translation Control Register (EL2) */
+	VTTBR_EL2,	/* Virtualization Translation Table Base Register */
+	VTCR_EL2,	/* Virtualization Translation Control Register */
+	SPSR_EL2,	/* EL2 saved program status register */
+	ELR_EL2,	/* EL2 exception link register */
+	AFSR0_EL2,	/* Auxiliary Fault Status Register 0 (EL2) */
+	AFSR1_EL2,	/* Auxiliary Fault Status Register 1 (EL2) */
+	ESR_EL2,	/* Exception Syndrome Register (EL2) */
+	FAR_EL2,	/* Hypervisor IPA Fault Address Register */
+	HPFAR_EL2,	/* Hypervisor IPA Fault Address Register */
+	MAIR_EL2,	/* Memory Attribute Indirection Register (EL2) */
+	AMAIR_EL2,	/* Auxiliary Memory Attribute Indirection Register (EL2) */
+	VBAR_EL2,	/* Vector Base Address Register (EL2) */
+	RVBAR_EL2,	/* Reset Vector Base Address Register */
+	CONTEXTIDR_EL2,	/* Context ID Register (EL2) */
+	TPIDR_EL2,	/* EL2 Software Thread ID Register */
+	CNTHCTL_EL2,	/* Counter-timer Hypervisor Control register */
+	SP_EL2,		/* EL2 Stack Pointer */
+
 	NR_SYS_REGS	/* Nothing after this line! */
 };
 
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 10/69
#28 lore raw

Re: [PATCH v5 10/69] KVM: arm64: nv: Add EL2 system registers to vcpu context

Quoted 5 lines · Marc Zyngier

Add the minimal set of EL2 system registers to the vcpu context. Nothing uses them just yet.

Signature · 6 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#29 lore raw

PATCH v5 11/69 KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

When running a nested hypervisor we commonly have to figure out if the VCPU mode is running in the context of a guest hypervisor or guest guest, or just a normal guest.

Add convenient primitives for this.

---
 arch/arm64/include/asm/kvm_emulate.h | 55 ++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +55 -0
index f4871e47b2d0..f4b079945d0f 100644
@@ -176,6 +176,61 @@ static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num,
 		vcpu_gp_regs(vcpu)->regs[reg_num] = val;
 }
 
+static inline bool vcpu_mode_el2_ctxt(const struct kvm_cpu_context *ctxt)
+{
+	unsigned long cpsr = ctxt->regs.pstate;
+
+	switch (cpsr & (PSR_MODE32_BIT | PSR_MODE_MASK)) {
+	case PSR_MODE_EL2h:
+	case PSR_MODE_EL2t:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static inline bool vcpu_mode_el2(const struct kvm_vcpu *vcpu)
+{
+	return vcpu_mode_el2_ctxt(&vcpu->arch.ctxt);
+}
+
+static inline bool __vcpu_el2_e2h_is_set(const struct kvm_cpu_context *ctxt)
+{
+	return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_E2H;
+}
+
+static inline bool vcpu_el2_e2h_is_set(const struct kvm_vcpu *vcpu)
+{
+	return __vcpu_el2_e2h_is_set(&vcpu->arch.ctxt);
+}
+
+static inline bool __vcpu_el2_tge_is_set(const struct kvm_cpu_context *ctxt)
+{
+	return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_TGE;
+}
+
+static inline bool vcpu_el2_tge_is_set(const struct kvm_vcpu *vcpu)
+{
+	return __vcpu_el2_tge_is_set(&vcpu->arch.ctxt);
+}
+
+static inline bool __is_hyp_ctxt(const struct kvm_cpu_context *ctxt)
+{
+	/*
+	 * We are in a hypervisor context if the vcpu mode is EL2 or
+	 * E2H and TGE bits are set. The latter means we are in the user space
+	 * of the VHE kernel. ARMv8.1 ARM describes this as 'InHost'
+	 */
+	return vcpu_mode_el2_ctxt(ctxt) ||
+		(__vcpu_el2_e2h_is_set(ctxt) && __vcpu_el2_tge_is_set(ctxt)) ||
+		WARN_ON(__vcpu_el2_tge_is_set(ctxt));
+}
+
+static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu)
+{
+	return __is_hyp_ctxt(&vcpu->arch.ctxt);
+}
+
 /*
  * The layout of SPSR for an AArch32 state is different when observed from an
  * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#30 lore raw

Re: [PATCH v5 11/69] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state

Hi Marc,

Bunch of bikeshedding regarding names below, which can be safely ignored.

Quoted 19 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

When running a nested hypervisor we commonly have to figure out if the VCPU mode is running in the context of a guest hypervisor or guest guest, or just a normal guest.

Add convenient primitives for this.

---
 arch/arm64/include/asm/kvm_emulate.h | 55 ++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +0 -0
index f4871e47b2d0..f4b079945d0f 100644

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 179

Quoted 4 lines
arch/arm64/include/asm/kvm_emulate.h +1 -0
@@ -176,6 +176,61 @@ static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num,
 		vcpu_gp_regs(vcpu)->regs[reg_num] = val;
 }
 
+static inline bool vcpu_mode_el2_ctxt(const struct kvm_cpu_context *ctxt)

"The Armv8-A architecture defines a set of Exception levels, EL0 to EL3" (ARM DDI 0487G.a, page G1-5941).

"AArch64 state does not support modes. Modes are a concept that is specific to AArch32 state." (ARM DDI 0487G.a, page G1-5944).

Wouldn't it be better to use the same terminology as the architecture?

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 181

Quoted 2 lines
arch/arm64/include/asm/kvm_emulate.h +2 -0
+{
+	unsigned long cpsr = ctxt->regs.pstate;

CPSR is an AArch32 register. Why not name the variable pstate?

Quoted 21 lines
arch/arm64/include/asm/kvm_emulate.h +21 -0
+
+	switch (cpsr & (PSR_MODE32_BIT | PSR_MODE_MASK)) {
+	case PSR_MODE_EL2h:
+	case PSR_MODE_EL2t:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static inline bool vcpu_mode_el2(const struct kvm_vcpu *vcpu)
+{
+	return vcpu_mode_el2_ctxt(&vcpu->arch.ctxt);
+}
+
+static inline bool __vcpu_el2_e2h_is_set(const struct kvm_cpu_context *ctxt)
+{
+	return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_E2H;
+}
+
+static inline bool vcpu_el2_e2h_is_set(const struct kvm_vcpu *vcpu)

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 212

Quoted 10 lines
arch/arm64/include/asm/kvm_emulate.h +10 -0
+{
+	return __vcpu_el2_e2h_is_set(&vcpu->arch.ctxt);
+}
+
+static inline bool __vcpu_el2_tge_is_set(const struct kvm_cpu_context *ctxt)
+{
+	return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_TGE;
+}
+
+static inline bool vcpu_el2_tge_is_set(const struct kvm_vcpu *vcpu)

This is confusing. What does the exception level have to do with the HCR_EL2.TGE bit being set? Wouldn't vcpu_hcr_tge_is_set() be better?

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 222

Quoted 10 lines
arch/arm64/include/asm/kvm_emulate.h +10 -0
+{
+	return __vcpu_el2_tge_is_set(&vcpu->arch.ctxt);
+}
+
+static inline bool __is_hyp_ctxt(const struct kvm_cpu_context *ctxt)
+{
+	/*
+	 * We are in a hypervisor context if the vcpu mode is EL2 or
+	 * E2H and TGE bits are set. The latter means we are in the user space
+	 * of the VHE kernel. ARMv8.1 ARM describes this as 'InHost'

So why not call the function vcpu_is_inhost() or something along those lines to match the architecture?

Thanks,
Alex

Quoted 17 lines
arch/arm64/include/asm/kvm_emulate.h +11 -0
+	 */
+	return vcpu_mode_el2_ctxt(ctxt) ||
+		(__vcpu_el2_e2h_is_set(ctxt) && __vcpu_el2_tge_is_set(ctxt)) ||
+		WARN_ON(__vcpu_el2_tge_is_set(ctxt));
+}
+
+static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu)
+{
+	return __is_hyp_ctxt(&vcpu->arch.ctxt);
+}
+
 /*
  * The layout of SPSR for an AArch32 state is different when observed from an
  * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32
Signature · 1 line
2.30.2
author maintainer
#31 lore raw

Re: [PATCH v5 11/69] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state

Quoted 33 lines · Alexandru Elisei

Hi Marc,

Bunch of bikeshedding regarding names below, which can be safely ignored.

Quoted 19 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

When running a nested hypervisor we commonly have to figure out if the VCPU mode is running in the context of a guest hypervisor or guest guest, or just a normal guest.

Add convenient primitives for this.

---
 arch/arm64/include/asm/kvm_emulate.h | 55 ++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +0 -0
index f4871e47b2d0..f4b079945d0f 100644

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 179

Quoted 4 lines
arch/arm64/include/asm/kvm_emulate.h +1 -0
@@ -176,6 +176,61 @@ static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num,
 		vcpu_gp_regs(vcpu)->regs[reg_num] = val;
 }
 
+static inline bool vcpu_mode_el2_ctxt(const struct kvm_cpu_context *ctxt)

"The Armv8-A architecture defines a set of Exception levels, EL0 to EL3" (ARM DDI 0487G.a, page G1-5941).

Quoted 4 lines

"AArch64 state does not support modes. Modes are a concept that is specific to AArch32 state." (ARM DDI 0487G.a, page G1-5944).

Wouldn't it be better to use the same terminology as the architecture?

Probably. I'll see how invasive it is to repaint this. It still remains that the 'mode' term is used all over the shop (for example, PSR_MODE_*).

Quoted 4 lines

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 181

Quoted 2 lines
arch/arm64/include/asm/kvm_emulate.h +2 -0
+{
+	unsigned long cpsr = ctxt->regs.pstate;

CPSR is an AArch32 register. Why not name the variable pstate?

Because we have *a ton* of existing CPSR references all over the shop (more than references to pstate, actually), owing to the AArch32 heritage of KVM/arm64. Yes, I can change this locally without any damage. But repainting the whole of KVM would be fairly pointless (same with hsr/esr, hfar/far_el2...).

Quoted 27 lines
Quoted 27 lines
arch/arm64/include/asm/kvm_emulate.h +26 -0
+
+	switch (cpsr & (PSR_MODE32_BIT | PSR_MODE_MASK)) {
+	case PSR_MODE_EL2h:
+	case PSR_MODE_EL2t:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static inline bool vcpu_mode_el2(const struct kvm_vcpu *vcpu)
+{
+	return vcpu_mode_el2_ctxt(&vcpu->arch.ctxt);
+}
+
+static inline bool __vcpu_el2_e2h_is_set(const struct kvm_cpu_context *ctxt)
+{
+	return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_E2H;
+}
+
+static inline bool vcpu_el2_e2h_is_set(const struct kvm_vcpu *vcpu)
+{
+	return __vcpu_el2_e2h_is_set(&vcpu->arch.ctxt);
+}
+
+static inline bool __vcpu_el2_tge_is_set(const struct kvm_cpu_context *ctxt)
Quoted 8 lines

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 212

Quoted 5 lines
arch/arm64/include/asm/kvm_emulate.h +5 -0
+{
+	return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_TGE;
+}
+
+static inline bool vcpu_el2_tge_is_set(const struct kvm_vcpu *vcpu)

This is confusing. What does the exception level have to do with the HCR_EL2.TGE bit being set? Wouldn't vcpu_hcr_tge_is_set() be better?

Sure, why not. Again, I'll see how invasive such a repainting is across 70 patches.

Quoted 14 lines

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 222

Quoted 11 lines
arch/arm64/include/asm/kvm_emulate.h +10 -0
+{
+	return __vcpu_el2_tge_is_set(&vcpu->arch.ctxt);
+}
+
+static inline bool __is_hyp_ctxt(const struct kvm_cpu_context *ctxt)
+{
+	/*
+	 * We are in a hypervisor context if the vcpu mode is EL2 or
+	 * E2H and TGE bits are set. The latter means we are in the user space
+	 * of the VHE kernel. ARMv8.1 ARM describes this as 'InHost'

So why not call the function vcpu_is_inhost() or something along those lines to match the architecture?

Because this is not the architectural 'InHost' primitive, which returns 'false' if HCR_EL2.E2H==0. The second term of the expression could be written in terms of an InHost primitive, but that's about it.

Quoted 4 lines

Thanks,
Alex

Quoted 14 lines
Quoted 4 lines
arch/arm64/include/asm/kvm_emulate.h +4 -0
+	 */
+	return vcpu_mode_el2_ctxt(ctxt) ||
+		(__vcpu_el2_e2h_is_set(ctxt) && __vcpu_el2_tge_is_set(ctxt)) ||
+		WARN_ON(__vcpu_el2_tge_is_set(ctxt));

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 236

Quoted 10 lines
arch/arm64/include/asm/kvm_emulate.h +7 -0
+}
+
+static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu)
+{
+	return __is_hyp_ctxt(&vcpu->arch.ctxt);
+}
+
 /*
  * The layout of SPSR for an AArch32 state is different when observed from an
  * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#32 lore raw

Re: [PATCH v5 11/69] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state

Hi Marc,

Quoted 35 lines · Marc Zyngier
Quoted 33 lines · Alexandru Elisei

Hi Marc,

Bunch of bikeshedding regarding names below, which can be safely ignored.

Quoted 19 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

When running a nested hypervisor we commonly have to figure out if the VCPU mode is running in the context of a guest hypervisor or guest guest, or just a normal guest.

Add convenient primitives for this.

---
 arch/arm64/include/asm/kvm_emulate.h | 55 ++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +0 -0
index f4871e47b2d0..f4b079945d0f 100644

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 179

Quoted 4 lines
arch/arm64/include/asm/kvm_emulate.h +1 -0
@@ -176,6 +176,61 @@ static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num,
 		vcpu_gp_regs(vcpu)->regs[reg_num] = val;
 }
 
+static inline bool vcpu_mode_el2_ctxt(const struct kvm_cpu_context *ctxt)

"The Armv8-A architecture defines a set of Exception levels, EL0 to EL3" (ARM DDI 0487G.a, page G1-5941).

Quoted 8 lines
Quoted 4 lines

"AArch64 state does not support modes. Modes are a concept that is specific to AArch32 state." (ARM DDI 0487G.a, page G1-5944).

Wouldn't it be better to use the same terminology as the architecture?

Probably. I'll see how invasive it is to repaint this. It still remains that the 'mode' term is used all over the shop (for example, PSR_MODE_*).

It might be easier if it's a separate patch on top of this series, instead of renaming the function in (potentially) every patch that follows. I can write that patch, if it helps.

Quoted 11 lines
Quoted 5 lines

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 181

Quoted 3 lines
arch/arm64/include/asm/kvm_emulate.h +2 -0
+{
+	unsigned long cpsr = ctxt->regs.pstate;

CPSR is an AArch32 register. Why not name the variable pstate?

Because we have *a ton* of existing CPSR references all over the shop (more than references to pstate, actually), owing to the AArch32 heritage of KVM/arm64. Yes, I can change this locally without any damage. But repainting the whole of KVM would be fairly pointless (same with hsr/esr, hfar/far_el2...).

That's fine, I wasn't suggesting replacing it everywhere, that's too invasive.

Quoted 55 lines
Quoted 28 lines
Quoted 28 lines
arch/arm64/include/asm/kvm_emulate.h +26 -0
+
+	switch (cpsr & (PSR_MODE32_BIT | PSR_MODE_MASK)) {
+	case PSR_MODE_EL2h:
+	case PSR_MODE_EL2t:
+		return true;
+	default:
+		return false;
+	}
+}
+
+static inline bool vcpu_mode_el2(const struct kvm_vcpu *vcpu)
+{
+	return vcpu_mode_el2_ctxt(&vcpu->arch.ctxt);
+}
+
+static inline bool __vcpu_el2_e2h_is_set(const struct kvm_cpu_context *ctxt)
+{
+	return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_E2H;
+}
+
+static inline bool vcpu_el2_e2h_is_set(const struct kvm_vcpu *vcpu)
+{
+	return __vcpu_el2_e2h_is_set(&vcpu->arch.ctxt);
+}
+
+static inline bool __vcpu_el2_tge_is_set(const struct kvm_cpu_context *ctxt)
Quoted 8 lines

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 212

Quoted 5 lines
arch/arm64/include/asm/kvm_emulate.h +5 -0
+{
+	return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_TGE;
+}
+
+static inline bool vcpu_el2_tge_is_set(const struct kvm_vcpu *vcpu)

This is confusing. What does the exception level have to do with the HCR_EL2.TGE bit being set? Wouldn't vcpu_hcr_tge_is_set() be better?

Sure, why not. Again, I'll see how invasive such a repainting is across 70 patches.

Quoted 14 lines

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 222

Quoted 11 lines
arch/arm64/include/asm/kvm_emulate.h +10 -0
+{
+	return __vcpu_el2_tge_is_set(&vcpu->arch.ctxt);
+}
+
+static inline bool __is_hyp_ctxt(const struct kvm_cpu_context *ctxt)
+{
+	/*
+	 * We are in a hypervisor context if the vcpu mode is EL2 or
+	 * E2H and TGE bits are set. The latter means we are in the user space
+	 * of the VHE kernel. ARMv8.1 ARM describes this as 'InHost'

So why not call the function vcpu_is_inhost() or something along those lines to match the architecture?

Quoted 3 lines

Because this is not the architectural 'InHost' primitive, which returns 'false' if HCR_EL2.E2H==0. The second term of the expression could be written in terms of an InHost primitive, but that's about it.

That's true, my mistake.

Thanks,
Alex

Quoted 26 lines
Quoted 5 lines

Thanks,
Alex

Quoted 14 lines
Quoted 4 lines
arch/arm64/include/asm/kvm_emulate.h +4 -0
+	 */
+	return vcpu_mode_el2_ctxt(ctxt) ||
+		(__vcpu_el2_e2h_is_set(ctxt) && __vcpu_el2_tge_is_set(ctxt)) ||
+		WARN_ON(__vcpu_el2_tge_is_set(ctxt));

arch/arm64/include/asm/kvm_emulate.h static __always_inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, · line 236

Quoted 10 lines
arch/arm64/include/asm/kvm_emulate.h +7 -0
+}
+
+static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu)
+{
+	return __is_hyp_ctxt(&vcpu->arch.ctxt);
+}
+
 /*
  * The layout of SPSR for an AArch32 state is different when observed from an
  * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Reviewed-by on v5 11/69
#33 lore raw

Re: [PATCH v5 11/69] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state

In reply to Marc Zyngier
Quoted 10 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

When running a nested hypervisor we commonly have to figure out if the VCPU mode is running in the context of a guest hypervisor or guest guest, or just a normal guest.

Add convenient primitives for this.

Signature · 6 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#34 lore raw

PATCH v5 12/69 KVM: arm64: nv: Handle HCR_EL2.NV system register traps

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

ARM v8.3 introduces a new bit in the HCR_EL2, which is the NV bit. When this bit is set, accessing EL2 registers in EL1 traps to EL2. In addition, executing the following instructions in EL1 will trap to EL2:
tlbi, at, eret, and msr/mrs instructions to access SP_EL1. Most of the instructions that trap to EL2 with the NV bit were undef at EL1 prior to ARM v8.3. The only instruction that was not undef is eret.

This patch sets up a handler for EL2 registers and SP_EL1 register accesses at EL1. The host hypervisor keeps those register values in memory, and will emulate their behavior.

This patch doesn't set the NV bit yet. It will be set in a later patch once nested virtualization support is completed.

[maz: added SCTLR_EL2 RES0/RES1 handling]
---
 arch/arm64/include/asm/sysreg.h |  41 +++++++++++-
 arch/arm64/kvm/sys_regs.c       | 109 ++++++++++++++++++++++++++++++--
 2 files changed, 144 insertions(+), 6 deletions(-)
arch/arm64/include/asm/sysreg.h +40 -1
index 615dd6278f8b..c77fe5401826 100644
@@ -531,10 +531,26 @@
 
 #define SYS_PMCCFILTR_EL0		sys_reg(3, 3, 14, 15, 7)
 
+#define SYS_VPIDR_EL2			sys_reg(3, 4, 0, 0, 0)
+#define SYS_VMPIDR_EL2			sys_reg(3, 4, 0, 0, 5)
+
 #define SYS_SCTLR_EL2			sys_reg(3, 4, 1, 0, 0)
+#define SYS_ACTLR_EL2			sys_reg(3, 4, 1, 0, 1)
+#define SYS_HCR_EL2			sys_reg(3, 4, 1, 1, 0)
+#define SYS_MDCR_EL2			sys_reg(3, 4, 1, 1, 1)
+#define SYS_CPTR_EL2			sys_reg(3, 4, 1, 1, 2)
+#define SYS_HSTR_EL2			sys_reg(3, 4, 1, 1, 3)
 #define SYS_HFGRTR_EL2			sys_reg(3, 4, 1, 1, 4)
 #define SYS_HFGWTR_EL2			sys_reg(3, 4, 1, 1, 5)
 #define SYS_HFGITR_EL2			sys_reg(3, 4, 1, 1, 6)
+#define SYS_HACR_EL2			sys_reg(3, 4, 1, 1, 7)
+
+#define SYS_TTBR0_EL2			sys_reg(3, 4, 2, 0, 0)
+#define SYS_TTBR1_EL2			sys_reg(3, 4, 2, 0, 1)
+#define SYS_TCR_EL2			sys_reg(3, 4, 2, 0, 2)
+#define SYS_VTTBR_EL2			sys_reg(3, 4, 2, 1, 0)
+#define SYS_VTCR_EL2			sys_reg(3, 4, 2, 1, 2)
+
 #define SYS_ZCR_EL2			sys_reg(3, 4, 1, 2, 0)
 #define SYS_TRFCR_EL2			sys_reg(3, 4, 1, 2, 1)
 #define SYS_DACR32_EL2			sys_reg(3, 4, 3, 0, 0)
@@ -543,14 +559,26 @@
 #define SYS_HAFGRTR_EL2			sys_reg(3, 4, 3, 1, 6)
 #define SYS_SPSR_EL2			sys_reg(3, 4, 4, 0, 0)
 #define SYS_ELR_EL2			sys_reg(3, 4, 4, 0, 1)
+#define SYS_SP_EL1			sys_reg(3, 4, 4, 1, 0)
 #define SYS_IFSR32_EL2			sys_reg(3, 4, 5, 0, 1)
+#define SYS_AFSR0_EL2			sys_reg(3, 4, 5, 1, 0)
+#define SYS_AFSR1_EL2			sys_reg(3, 4, 5, 1, 1)
 #define SYS_ESR_EL2			sys_reg(3, 4, 5, 2, 0)
 #define SYS_VSESR_EL2			sys_reg(3, 4, 5, 2, 3)
 #define SYS_FPEXC32_EL2			sys_reg(3, 4, 5, 3, 0)
 #define SYS_TFSR_EL2			sys_reg(3, 4, 5, 6, 0)
 #define SYS_FAR_EL2			sys_reg(3, 4, 6, 0, 0)
 
-#define SYS_VDISR_EL2			sys_reg(3, 4, 12, 1,  1)
+#define SYS_FAR_EL2			sys_reg(3, 4, 6, 0, 0)
+#define SYS_HPFAR_EL2			sys_reg(3, 4, 6, 0, 4)
+
+#define SYS_MAIR_EL2			sys_reg(3, 4, 10, 2, 0)
+#define SYS_AMAIR_EL2			sys_reg(3, 4, 10, 3, 0)
+
+#define SYS_VBAR_EL2			sys_reg(3, 4, 12, 0, 0)
+#define SYS_RVBAR_EL2			sys_reg(3, 4, 12, 0, 1)
+#define SYS_RMR_EL2			sys_reg(3, 4, 12, 0, 2)
+#define SYS_VDISR_EL2			sys_reg(3, 4, 12, 1, 1)
 #define __SYS__AP0Rx_EL2(x)		sys_reg(3, 4, 12, 8, x)
 #define SYS_ICH_AP0R0_EL2		__SYS__AP0Rx_EL2(0)
 #define SYS_ICH_AP0R1_EL2		__SYS__AP0Rx_EL2(1)
@@ -592,15 +620,24 @@
 #define SYS_ICH_LR14_EL2		__SYS__LR8_EL2(6)
 #define SYS_ICH_LR15_EL2		__SYS__LR8_EL2(7)
 
+#define SYS_CONTEXTIDR_EL2		sys_reg(3, 4, 13, 0, 1)
+#define SYS_TPIDR_EL2			sys_reg(3, 4, 13, 0, 2)
+
+#define SYS_CNTVOFF_EL2			sys_reg(3, 4, 14, 0, 3)
+#define SYS_CNTHCTL_EL2			sys_reg(3, 4, 14, 1, 0)
+
 /* VHE encodings for architectural EL0/1 system registers */
 #define SYS_SCTLR_EL12			sys_reg(3, 5, 1, 0, 0)
 #define SYS_CPACR_EL12			sys_reg(3, 5, 1, 0, 2)
 #define SYS_ZCR_EL12			sys_reg(3, 5, 1, 2, 0)
+
 #define SYS_TTBR0_EL12			sys_reg(3, 5, 2, 0, 0)
 #define SYS_TTBR1_EL12			sys_reg(3, 5, 2, 0, 1)
 #define SYS_TCR_EL12			sys_reg(3, 5, 2, 0, 2)
+
 #define SYS_SPSR_EL12			sys_reg(3, 5, 4, 0, 0)
 #define SYS_ELR_EL12			sys_reg(3, 5, 4, 0, 1)
+
 #define SYS_AFSR0_EL12			sys_reg(3, 5, 5, 1, 0)
 #define SYS_AFSR1_EL12			sys_reg(3, 5, 5, 1, 1)
 #define SYS_ESR_EL12			sys_reg(3, 5, 5, 2, 0)
@@ -618,6 +655,8 @@
 #define SYS_CNTV_CTL_EL02		sys_reg(3, 5, 14, 3, 1)
 #define SYS_CNTV_CVAL_EL02		sys_reg(3, 5, 14, 3, 2)
 
+#define SYS_SP_EL2			sys_reg(3, 6,  4, 1, 0)
+
 /* Common SCTLR_ELx flags. */
 #define SCTLR_ELx_DSSBS	(BIT(44))
 #define SCTLR_ELx_ATA	(BIT(43))
arch/arm64/kvm/sys_regs.c +104 -5

143 lines not on this page · raw message

Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#35 lore raw

Re: [PATCH v5 12/69] KVM: arm64: nv: Handle HCR_EL2.NV system register traps

Hi Marc,

Nitpick, but HCR_EL2.NV also traps accesses to *_EL02 and *_EL12 registers, according to ARM DDI 0487G.a, page D5-2770. The subject could be changed to "Handle HCR_EL2.NV *_EL2 system register traps" to better match the content, but doesn't make much a difference overall.

Quoted 109 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

ARM v8.3 introduces a new bit in the HCR_EL2, which is the NV bit. When this bit is set, accessing EL2 registers in EL1 traps to EL2. In addition, executing the following instructions in EL1 will trap to EL2:
tlbi, at, eret, and msr/mrs instructions to access SP_EL1. Most of the instructions that trap to EL2 with the NV bit were undef at EL1 prior to ARM v8.3. The only instruction that was not undef is eret.

This patch sets up a handler for EL2 registers and SP_EL1 register accesses at EL1. The host hypervisor keeps those register values in memory, and will emulate their behavior.

This patch doesn't set the NV bit yet. It will be set in a later patch once nested virtualization support is completed.

[maz: added SCTLR_EL2 RES0/RES1 handling]
---
 arch/arm64/include/asm/sysreg.h |  41 +++++++++++-
 arch/arm64/kvm/sys_regs.c       | 109 ++++++++++++++++++++++++++++++--
 2 files changed, 144 insertions(+), 6 deletions(-)
arch/arm64/include/asm/sysreg.h +38 -1
index 615dd6278f8b..c77fe5401826 100644
@@ -531,10 +531,26 @@
 
 #define SYS_PMCCFILTR_EL0		sys_reg(3, 3, 14, 15, 7)
 
+#define SYS_VPIDR_EL2			sys_reg(3, 4, 0, 0, 0)
+#define SYS_VMPIDR_EL2			sys_reg(3, 4, 0, 0, 5)
+
 #define SYS_SCTLR_EL2			sys_reg(3, 4, 1, 0, 0)
+#define SYS_ACTLR_EL2			sys_reg(3, 4, 1, 0, 1)
+#define SYS_HCR_EL2			sys_reg(3, 4, 1, 1, 0)
+#define SYS_MDCR_EL2			sys_reg(3, 4, 1, 1, 1)
+#define SYS_CPTR_EL2			sys_reg(3, 4, 1, 1, 2)
+#define SYS_HSTR_EL2			sys_reg(3, 4, 1, 1, 3)
 #define SYS_HFGRTR_EL2			sys_reg(3, 4, 1, 1, 4)
 #define SYS_HFGWTR_EL2			sys_reg(3, 4, 1, 1, 5)
 #define SYS_HFGITR_EL2			sys_reg(3, 4, 1, 1, 6)
+#define SYS_HACR_EL2			sys_reg(3, 4, 1, 1, 7)
+
+#define SYS_TTBR0_EL2			sys_reg(3, 4, 2, 0, 0)
+#define SYS_TTBR1_EL2			sys_reg(3, 4, 2, 0, 1)
+#define SYS_TCR_EL2			sys_reg(3, 4, 2, 0, 2)
+#define SYS_VTTBR_EL2			sys_reg(3, 4, 2, 1, 0)
+#define SYS_VTCR_EL2			sys_reg(3, 4, 2, 1, 2)
+
 #define SYS_ZCR_EL2			sys_reg(3, 4, 1, 2, 0)
 #define SYS_TRFCR_EL2			sys_reg(3, 4, 1, 2, 1)
 #define SYS_DACR32_EL2			sys_reg(3, 4, 3, 0, 0)
@@ -543,14 +559,26 @@
 #define SYS_HAFGRTR_EL2			sys_reg(3, 4, 3, 1, 6)
 #define SYS_SPSR_EL2			sys_reg(3, 4, 4, 0, 0)
 #define SYS_ELR_EL2			sys_reg(3, 4, 4, 0, 1)
+#define SYS_SP_EL1			sys_reg(3, 4, 4, 1, 0)
 #define SYS_IFSR32_EL2			sys_reg(3, 4, 5, 0, 1)
+#define SYS_AFSR0_EL2			sys_reg(3, 4, 5, 1, 0)
+#define SYS_AFSR1_EL2			sys_reg(3, 4, 5, 1, 1)
 #define SYS_ESR_EL2			sys_reg(3, 4, 5, 2, 0)
 #define SYS_VSESR_EL2			sys_reg(3, 4, 5, 2, 3)
 #define SYS_FPEXC32_EL2			sys_reg(3, 4, 5, 3, 0)
 #define SYS_TFSR_EL2			sys_reg(3, 4, 5, 6, 0)
 #define SYS_FAR_EL2			sys_reg(3, 4, 6, 0, 0)
 
-#define SYS_VDISR_EL2			sys_reg(3, 4, 12, 1,  1)
+#define SYS_FAR_EL2			sys_reg(3, 4, 6, 0, 0)
+#define SYS_HPFAR_EL2			sys_reg(3, 4, 6, 0, 4)
+
+#define SYS_MAIR_EL2			sys_reg(3, 4, 10, 2, 0)
+#define SYS_AMAIR_EL2			sys_reg(3, 4, 10, 3, 0)
+
+#define SYS_VBAR_EL2			sys_reg(3, 4, 12, 0, 0)
+#define SYS_RVBAR_EL2			sys_reg(3, 4, 12, 0, 1)
+#define SYS_RMR_EL2			sys_reg(3, 4, 12, 0, 2)
+#define SYS_VDISR_EL2			sys_reg(3, 4, 12, 1, 1)
 #define __SYS__AP0Rx_EL2(x)		sys_reg(3, 4, 12, 8, x)
 #define SYS_ICH_AP0R0_EL2		__SYS__AP0Rx_EL2(0)
 #define SYS_ICH_AP0R1_EL2		__SYS__AP0Rx_EL2(1)
@@ -592,15 +620,24 @@
 #define SYS_ICH_LR14_EL2		__SYS__LR8_EL2(6)
 #define SYS_ICH_LR15_EL2		__SYS__LR8_EL2(7)
 
+#define SYS_CONTEXTIDR_EL2		sys_reg(3, 4, 13, 0, 1)
+#define SYS_TPIDR_EL2			sys_reg(3, 4, 13, 0, 2)
+
+#define SYS_CNTVOFF_EL2			sys_reg(3, 4, 14, 0, 3)
+#define SYS_CNTHCTL_EL2			sys_reg(3, 4, 14, 1, 0)
+
 /* VHE encodings for architectural EL0/1 system registers */
 #define SYS_SCTLR_EL12			sys_reg(3, 5, 1, 0, 0)
 #define SYS_CPACR_EL12			sys_reg(3, 5, 1, 0, 2)
 #define SYS_ZCR_EL12			sys_reg(3, 5, 1, 2, 0)
+
 #define SYS_TTBR0_EL12			sys_reg(3, 5, 2, 0, 0)
 #define SYS_TTBR1_EL12			sys_reg(3, 5, 2, 0, 1)
 #define SYS_TCR_EL12			sys_reg(3, 5, 2, 0, 2)
+
 #define SYS_SPSR_EL12			sys_reg(3, 5, 4, 0, 0)
 #define SYS_ELR_EL12			sys_reg(3, 5, 4, 0, 1)
+
 #define SYS_AFSR0_EL12			sys_reg(3, 5, 5, 1, 0)
 #define SYS_AFSR1_EL12			sys_reg(3, 5, 5, 1, 1)
 #define SYS_ESR_EL12			sys_reg(3, 5, 5, 2, 0)

arch/arm64/include/asm/sysreg.h line 658

Quoted 4 lines
arch/arm64/include/asm/sysreg.h +1 -0
@@ -618,6 +655,8 @@
 #define SYS_CNTV_CTL_EL02		sys_reg(3, 5, 14, 3, 1)
 #define SYS_CNTV_CVAL_EL02		sys_reg(3, 5, 14, 3, 2)
 
+#define SYS_SP_EL2			sys_reg(3, 6,  4, 1, 0)

Checked the encoding for the newly added registers, they match.

Quoted 33 lines
arch/arm64/include/asm/sysreg.h +1 -0
+
 /* Common SCTLR_ELx flags. */
 #define SCTLR_ELx_DSSBS	(BIT(44))
 #define SCTLR_ELx_ATA	(BIT(43))
arch/arm64/kvm/sys_regs.c +21 -0
index e3ec1a44f94d..a23701f29858 100644
@@ -105,6 +105,46 @@ static u32 get_ccsidr(u32 csselr)
 	return ccsidr;
 }
 
+static bool access_rw(struct kvm_vcpu *vcpu,
+		      struct sys_reg_params *p,
+		      const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		vcpu_write_sys_reg(vcpu, p->regval, r->reg);
+	else
+		p->regval = vcpu_read_sys_reg(vcpu, r->reg);
+
+	return true;
+}
+
+static bool access_sctlr_el2(struct kvm_vcpu *vcpu,
+			     struct sys_reg_params *p,
+			     const struct sys_reg_desc *r)
+{
+	if (p->is_write) {
+		u64 val = p->regval;
+
+		if (vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu)) {
+			val &= ~(GENMASK_ULL(63,45) | GENMASK_ULL(34, 32) |

arch/arm64/kvm/sys_regs.c static u32 get_ccsidr(u32 csselr) · line 138

Quoted 10 lines
arch/arm64/kvm/sys_regs.c +10 -0
+				 BIT_ULL(17) | BIT_ULL(9));
+			val |=  SCTLR_EL1_RES1;
+		} else {
+			val &= ~(GENMASK_ULL(63,45) | BIT_ULL(42) |
+				 GENMASK_ULL(39, 38) | GENMASK_ULL(35, 32) |
+				 BIT_ULL(26) | BIT_ULL(24) | BIT_ULL(20) |
+				 BIT_ULL(17) | GENMASK_ULL(15, 14) |
+				 GENMASK(10, 7));
+			val |=  SCTLR_EL2_RES1;
+		}

Some bits in SCTLR_EL2 are functional bits when {E2H, TGE} = {1, 1}, otherwise they are RES0. This is how ARM DDI 0487G.a describes the behaviour of bits which are RES0 only in some contexts (page Glossary-8529, emphasis added by me):

"For a bit in a read/write register, when the bit is described as RES0:

- An indirect write to the register sets the bit to 0.

- ** A read of the bit must return the value last successfully written to the bit, by either a direct or an indirect write, regardless of the use of the register when the bit was written **

If the bit has not been successfully written since reset, then the read of the bit returns the reset value if there is one, or otherwise returns an UNKNOWN value.

- A direct write to the bit must update a storage location associated with the bit.

- While the use of the register is such that the bit is described as RES0, the value of the bit must have no effect on the operation of the PE, other than determining the value read back from that bit, unless this Manual explicitly defines additional properties for the bit."

Let's take bit 24, E0E, as an example. When E2H,TGE != {1,1} (which means the bit is now RES0), KVM clears the bit on a write, when according to the above definition, it should save the value.

Thanks,
Alex

Quoted 116 lines
arch/arm64/kvm/sys_regs.c +73 -5
+
+		vcpu_write_sys_reg(vcpu, val, r->reg);
+	} else {
+		p->regval = vcpu_read_sys_reg(vcpu, r->reg);
+	}
+
+	return true;
+}
+
 /*
  * See note at ARMv7 ARM B1.14.4 (TL;DR: S/W ops are not easily virtualized).
  */
@@ -263,6 +303,14 @@ static bool trap_raz_wi(struct kvm_vcpu *vcpu,
 		return read_zero(vcpu, p);
 }
 
+static bool trap_undef(struct kvm_vcpu *vcpu,
+		       struct sys_reg_params *p,
+		       const struct sys_reg_desc *r)
+{
+	kvm_inject_undefined(vcpu);
+	return false;
+}
+
 /*
  * ARMv8.1 mandates at least a trivial LORegion implementation, where all the
  * RW registers are RES0 (which we can implement as RAZ/WI). On an ARMv8.0
@@ -342,12 +390,9 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu,
 			    struct sys_reg_params *p,
 			    const struct sys_reg_desc *r)
 {
-	if (p->is_write) {
-		vcpu_write_sys_reg(vcpu, p->regval, r->reg);
+	access_rw(vcpu, p, r);
+	if (p->is_write)
 		vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY;
-	} else {
-		p->regval = vcpu_read_sys_reg(vcpu, r->reg);
-	}
 
 	trace_trap_reg(__func__, r->reg, p->is_write, p->regval);
 
@@ -1411,6 +1456,18 @@ static unsigned int mte_visibility(const struct kvm_vcpu *vcpu,
 	.set_user = set_raz_id_reg,		\
 }
 
+static bool access_sp_el1(struct kvm_vcpu *vcpu,
+			  struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		__vcpu_sys_reg(vcpu, SP_EL1) = p->regval;
+	else
+		p->regval = __vcpu_sys_reg(vcpu, SP_EL1);
+
+	return true;
+}
+
 /*
  * Architected system registers.
  * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
@@ -1825,9 +1882,51 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ PMU_SYS_REG(SYS_PMCCFILTR_EL0), .access = access_pmu_evtyper,
 	  .reset = reset_val, .reg = PMCCFILTR_EL0, .val = 0 },
 
+	{ SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_val, VPIDR_EL2, 0 },
+	{ SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_val, VMPIDR_EL2, 0 },
+
+	{ SYS_DESC(SYS_SCTLR_EL2), access_sctlr_el2, reset_val, SCTLR_EL2, SCTLR_EL2_RES1 },
+	{ SYS_DESC(SYS_ACTLR_EL2), access_rw, reset_val, ACTLR_EL2, 0 },
+	{ SYS_DESC(SYS_HCR_EL2), access_rw, reset_val, HCR_EL2, 0 },
+	{ SYS_DESC(SYS_MDCR_EL2), access_rw, reset_val, MDCR_EL2, 0 },
+	{ SYS_DESC(SYS_CPTR_EL2), access_rw, reset_val, CPTR_EL2, CPTR_EL2_DEFAULT },
+	{ SYS_DESC(SYS_HSTR_EL2), access_rw, reset_val, HSTR_EL2, 0 },
+	{ SYS_DESC(SYS_HACR_EL2), access_rw, reset_val, HACR_EL2, 0 },
+
+	{ SYS_DESC(SYS_TTBR0_EL2), access_rw, reset_val, TTBR0_EL2, 0 },
+	{ SYS_DESC(SYS_TTBR1_EL2), access_rw, reset_val, TTBR1_EL2, 0 },
+	{ SYS_DESC(SYS_TCR_EL2), access_rw, reset_val, TCR_EL2, TCR_EL2_RES1 },
+	{ SYS_DESC(SYS_VTTBR_EL2), access_rw, reset_val, VTTBR_EL2, 0 },
+	{ SYS_DESC(SYS_VTCR_EL2), access_rw, reset_val, VTCR_EL2, 0 },
+
 	{ SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 },
+	{ SYS_DESC(SYS_SPSR_EL2), access_rw, reset_val, SPSR_EL2, 0 },
+	{ SYS_DESC(SYS_ELR_EL2), access_rw, reset_val, ELR_EL2, 0 },
+	{ SYS_DESC(SYS_SP_EL1), access_sp_el1},
+
 	{ SYS_DESC(SYS_IFSR32_EL2), NULL, reset_unknown, IFSR32_EL2 },
+	{ SYS_DESC(SYS_AFSR0_EL2), access_rw, reset_val, AFSR0_EL2, 0 },
+	{ SYS_DESC(SYS_AFSR1_EL2), access_rw, reset_val, AFSR1_EL2, 0 },
+	{ SYS_DESC(SYS_ESR_EL2), access_rw, reset_val, ESR_EL2, 0 },
 	{ SYS_DESC(SYS_FPEXC32_EL2), NULL, reset_val, FPEXC32_EL2, 0x700 },
+
+	{ SYS_DESC(SYS_FAR_EL2), access_rw, reset_val, FAR_EL2, 0 },
+	{ SYS_DESC(SYS_HPFAR_EL2), access_rw, reset_val, HPFAR_EL2, 0 },
+
+	{ SYS_DESC(SYS_MAIR_EL2), access_rw, reset_val, MAIR_EL2, 0 },
+	{ SYS_DESC(SYS_AMAIR_EL2), access_rw, reset_val, AMAIR_EL2, 0 },
+
+	{ SYS_DESC(SYS_VBAR_EL2), access_rw, reset_val, VBAR_EL2, 0 },
+	{ SYS_DESC(SYS_RVBAR_EL2), access_rw, reset_val, RVBAR_EL2, 0 },
+	{ SYS_DESC(SYS_RMR_EL2), trap_undef },
+
+	{ SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 },
+	{ SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 },
+
+	{ SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 },
+	{ SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 },
+
+	{ SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 },
 };
 
 static bool trap_dbgdidr(struct kvm_vcpu *vcpu,
Signature · 1 line
2.30.2
author maintainer
#36 lore raw

Re: [PATCH v5 12/69] KVM: arm64: nv: Handle HCR_EL2.NV system register traps

Quoted 186 lines · Alexandru Elisei

Hi Marc,

Nitpick, but HCR_EL2.NV also traps accesses to *_EL02 and *_EL12 registers, according to ARM DDI 0487G.a, page D5-2770. The subject could be changed to "Handle HCR_EL2.NV *_EL2 system register traps" to better match the content, but doesn't make much a difference overall.

Quoted 109 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

ARM v8.3 introduces a new bit in the HCR_EL2, which is the NV bit. When this bit is set, accessing EL2 registers in EL1 traps to EL2. In addition, executing the following instructions in EL1 will trap to EL2:
tlbi, at, eret, and msr/mrs instructions to access SP_EL1. Most of the instructions that trap to EL2 with the NV bit were undef at EL1 prior to ARM v8.3. The only instruction that was not undef is eret.

This patch sets up a handler for EL2 registers and SP_EL1 register accesses at EL1. The host hypervisor keeps those register values in memory, and will emulate their behavior.

This patch doesn't set the NV bit yet. It will be set in a later patch once nested virtualization support is completed.

[maz: added SCTLR_EL2 RES0/RES1 handling]
---
 arch/arm64/include/asm/sysreg.h |  41 +++++++++++-
 arch/arm64/kvm/sys_regs.c       | 109 ++++++++++++++++++++++++++++++--
 2 files changed, 144 insertions(+), 6 deletions(-)
arch/arm64/include/asm/sysreg.h +38 -1
index 615dd6278f8b..c77fe5401826 100644
@@ -531,10 +531,26 @@
 
 #define SYS_PMCCFILTR_EL0		sys_reg(3, 3, 14, 15, 7)
 
+#define SYS_VPIDR_EL2			sys_reg(3, 4, 0, 0, 0)
+#define SYS_VMPIDR_EL2			sys_reg(3, 4, 0, 0, 5)
+
 #define SYS_SCTLR_EL2			sys_reg(3, 4, 1, 0, 0)
+#define SYS_ACTLR_EL2			sys_reg(3, 4, 1, 0, 1)
+#define SYS_HCR_EL2			sys_reg(3, 4, 1, 1, 0)
+#define SYS_MDCR_EL2			sys_reg(3, 4, 1, 1, 1)
+#define SYS_CPTR_EL2			sys_reg(3, 4, 1, 1, 2)
+#define SYS_HSTR_EL2			sys_reg(3, 4, 1, 1, 3)
 #define SYS_HFGRTR_EL2			sys_reg(3, 4, 1, 1, 4)
 #define SYS_HFGWTR_EL2			sys_reg(3, 4, 1, 1, 5)
 #define SYS_HFGITR_EL2			sys_reg(3, 4, 1, 1, 6)
+#define SYS_HACR_EL2			sys_reg(3, 4, 1, 1, 7)
+
+#define SYS_TTBR0_EL2			sys_reg(3, 4, 2, 0, 0)
+#define SYS_TTBR1_EL2			sys_reg(3, 4, 2, 0, 1)
+#define SYS_TCR_EL2			sys_reg(3, 4, 2, 0, 2)
+#define SYS_VTTBR_EL2			sys_reg(3, 4, 2, 1, 0)
+#define SYS_VTCR_EL2			sys_reg(3, 4, 2, 1, 2)
+
 #define SYS_ZCR_EL2			sys_reg(3, 4, 1, 2, 0)
 #define SYS_TRFCR_EL2			sys_reg(3, 4, 1, 2, 1)
 #define SYS_DACR32_EL2			sys_reg(3, 4, 3, 0, 0)
@@ -543,14 +559,26 @@
 #define SYS_HAFGRTR_EL2			sys_reg(3, 4, 3, 1, 6)
 #define SYS_SPSR_EL2			sys_reg(3, 4, 4, 0, 0)
 #define SYS_ELR_EL2			sys_reg(3, 4, 4, 0, 1)
+#define SYS_SP_EL1			sys_reg(3, 4, 4, 1, 0)
 #define SYS_IFSR32_EL2			sys_reg(3, 4, 5, 0, 1)
+#define SYS_AFSR0_EL2			sys_reg(3, 4, 5, 1, 0)
+#define SYS_AFSR1_EL2			sys_reg(3, 4, 5, 1, 1)
 #define SYS_ESR_EL2			sys_reg(3, 4, 5, 2, 0)
 #define SYS_VSESR_EL2			sys_reg(3, 4, 5, 2, 3)
 #define SYS_FPEXC32_EL2			sys_reg(3, 4, 5, 3, 0)
 #define SYS_TFSR_EL2			sys_reg(3, 4, 5, 6, 0)
 #define SYS_FAR_EL2			sys_reg(3, 4, 6, 0, 0)
 
-#define SYS_VDISR_EL2			sys_reg(3, 4, 12, 1,  1)
+#define SYS_FAR_EL2			sys_reg(3, 4, 6, 0, 0)
+#define SYS_HPFAR_EL2			sys_reg(3, 4, 6, 0, 4)
+
+#define SYS_MAIR_EL2			sys_reg(3, 4, 10, 2, 0)
+#define SYS_AMAIR_EL2			sys_reg(3, 4, 10, 3, 0)
+
+#define SYS_VBAR_EL2			sys_reg(3, 4, 12, 0, 0)
+#define SYS_RVBAR_EL2			sys_reg(3, 4, 12, 0, 1)
+#define SYS_RMR_EL2			sys_reg(3, 4, 12, 0, 2)
+#define SYS_VDISR_EL2			sys_reg(3, 4, 12, 1, 1)
 #define __SYS__AP0Rx_EL2(x)		sys_reg(3, 4, 12, 8, x)
 #define SYS_ICH_AP0R0_EL2		__SYS__AP0Rx_EL2(0)
 #define SYS_ICH_AP0R1_EL2		__SYS__AP0Rx_EL2(1)
@@ -592,15 +620,24 @@
 #define SYS_ICH_LR14_EL2		__SYS__LR8_EL2(6)
 #define SYS_ICH_LR15_EL2		__SYS__LR8_EL2(7)
 
+#define SYS_CONTEXTIDR_EL2		sys_reg(3, 4, 13, 0, 1)
+#define SYS_TPIDR_EL2			sys_reg(3, 4, 13, 0, 2)
+
+#define SYS_CNTVOFF_EL2			sys_reg(3, 4, 14, 0, 3)
+#define SYS_CNTHCTL_EL2			sys_reg(3, 4, 14, 1, 0)
+
 /* VHE encodings for architectural EL0/1 system registers */
 #define SYS_SCTLR_EL12			sys_reg(3, 5, 1, 0, 0)
 #define SYS_CPACR_EL12			sys_reg(3, 5, 1, 0, 2)
 #define SYS_ZCR_EL12			sys_reg(3, 5, 1, 2, 0)
+
 #define SYS_TTBR0_EL12			sys_reg(3, 5, 2, 0, 0)
 #define SYS_TTBR1_EL12			sys_reg(3, 5, 2, 0, 1)
 #define SYS_TCR_EL12			sys_reg(3, 5, 2, 0, 2)
+
 #define SYS_SPSR_EL12			sys_reg(3, 5, 4, 0, 0)
 #define SYS_ELR_EL12			sys_reg(3, 5, 4, 0, 1)
+
 #define SYS_AFSR0_EL12			sys_reg(3, 5, 5, 1, 0)
 #define SYS_AFSR1_EL12			sys_reg(3, 5, 5, 1, 1)
 #define SYS_ESR_EL12			sys_reg(3, 5, 5, 2, 0)

arch/arm64/include/asm/sysreg.h line 658

Quoted 4 lines
arch/arm64/include/asm/sysreg.h +1 -0
@@ -618,6 +655,8 @@
 #define SYS_CNTV_CTL_EL02		sys_reg(3, 5, 14, 3, 1)
 #define SYS_CNTV_CVAL_EL02		sys_reg(3, 5, 14, 3, 2)
 
+#define SYS_SP_EL2			sys_reg(3, 6,  4, 1, 0)

Checked the encoding for the newly added registers, they match.

Quoted 33 lines
arch/arm64/include/asm/sysreg.h +1 -0
+
 /* Common SCTLR_ELx flags. */
 #define SCTLR_ELx_DSSBS	(BIT(44))
 #define SCTLR_ELx_ATA	(BIT(43))
arch/arm64/kvm/sys_regs.c +21 -0
index e3ec1a44f94d..a23701f29858 100644
@@ -105,6 +105,46 @@ static u32 get_ccsidr(u32 csselr)
 	return ccsidr;
 }
 
+static bool access_rw(struct kvm_vcpu *vcpu,
+		      struct sys_reg_params *p,
+		      const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		vcpu_write_sys_reg(vcpu, p->regval, r->reg);
+	else
+		p->regval = vcpu_read_sys_reg(vcpu, r->reg);
+
+	return true;
+}
+
+static bool access_sctlr_el2(struct kvm_vcpu *vcpu,
+			     struct sys_reg_params *p,
+			     const struct sys_reg_desc *r)
+{
+	if (p->is_write) {
+		u64 val = p->regval;
+
+		if (vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu)) {
+			val &= ~(GENMASK_ULL(63,45) | GENMASK_ULL(34, 32) |

arch/arm64/kvm/sys_regs.c static u32 get_ccsidr(u32 csselr) · line 138

Quoted 10 lines
arch/arm64/kvm/sys_regs.c +10 -0
+				 BIT_ULL(17) | BIT_ULL(9));
+			val |=  SCTLR_EL1_RES1;
+		} else {
+			val &= ~(GENMASK_ULL(63,45) | BIT_ULL(42) |
+				 GENMASK_ULL(39, 38) | GENMASK_ULL(35, 32) |
+				 BIT_ULL(26) | BIT_ULL(24) | BIT_ULL(20) |
+				 BIT_ULL(17) | GENMASK_ULL(15, 14) |
+				 GENMASK(10, 7));
+			val |=  SCTLR_EL2_RES1;
+		}

Some bits in SCTLR_EL2 are functional bits when {E2H, TGE} = {1, 1}, otherwise they are RES0. This is how ARM DDI 0487G.a describes the behaviour of bits which are RES0 only in some contexts (page Glossary-8529, emphasis added by me):

"For a bit in a read/write register, when the bit is described as RES0:

- An indirect write to the register sets the bit to 0.

- ** A read of the bit must return the value last successfully written to the bit, by either a direct or an indirect write, regardless of the use of the register when the bit was written **

If the bit has not been successfully written since reset, then the read of the bit returns the reset value if there is one, or otherwise returns an UNKNOWN value.

- A direct write to the bit must update a storage location associated with the bit.

Quoted 8 lines

- While the use of the register is such that the bit is described as RES0, the value of the bit must have no effect on the operation of the PE, other than determining the value read back from that bit, unless this Manual explicitly defines additional properties for the bit."

Let's take bit 24, E0E, as an example. When E2H,TGE != {1,1} (which means the bit is now RES0), KVM clears the bit on a write, when according to the above definition, it should save the value.

That's an interesting observation. It means we should munge these bits at save/restore time (specially in the nVHE case where we have to translate SCTLR_EL2 to the SCTLR_EL1 format), rather than at access time.

This would also solve the discrepancy we have between the ARMv8.3 trapping and the ARMv8.4 memory write (we can sanitise the former, but not the latter).

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#37 lore raw

Re: [PATCH v5 12/69] KVM: arm64: nv: Handle HCR_EL2.NV system register traps

In reply to Marc Zyngier
Quoted 20 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

ARM v8.3 introduces a new bit in the HCR_EL2, which is the NV bit. When this bit is set, accessing EL2 registers in EL1 traps to EL2. In addition, executing the following instructions in EL1 will trap to EL2:
tlbi, at, eret, and msr/mrs instructions to access SP_EL1. Most of the instructions that trap to EL2 with the NV bit were undef at EL1 prior to ARM v8.3. The only instruction that was not undef is eret.

This patch sets up a handler for EL2 registers and SP_EL1 register accesses at EL1. The host hypervisor keeps those register values in memory, and will emulate their behavior.

This patch doesn't set the NV bit yet. It will be set in a later patch once nested virtualization support is completed.

[maz: added SCTLR_EL2 RES0/RES1 handling]

---

...

Quoted 39 lines
arch/arm64/kvm/sys_regs.c +32 -0
@@ -1825,9 +1882,51 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ PMU_SYS_REG(SYS_PMCCFILTR_EL0), .access = access_pmu_evtyper,
 	  .reset = reset_val, .reg = PMCCFILTR_EL0, .val = 0 },
 
+	{ SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_val, VPIDR_EL2, 0 },
+	{ SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_val, VMPIDR_EL2, 0 },
+
+	{ SYS_DESC(SYS_SCTLR_EL2), access_sctlr_el2, reset_val, SCTLR_EL2, SCTLR_EL2_RES1 },
+	{ SYS_DESC(SYS_ACTLR_EL2), access_rw, reset_val, ACTLR_EL2, 0 },
+	{ SYS_DESC(SYS_HCR_EL2), access_rw, reset_val, HCR_EL2, 0 },
+	{ SYS_DESC(SYS_MDCR_EL2), access_rw, reset_val, MDCR_EL2, 0 },
+	{ SYS_DESC(SYS_CPTR_EL2), access_rw, reset_val, CPTR_EL2, CPTR_EL2_DEFAULT },
+	{ SYS_DESC(SYS_HSTR_EL2), access_rw, reset_val, HSTR_EL2, 0 },
+	{ SYS_DESC(SYS_HACR_EL2), access_rw, reset_val, HACR_EL2, 0 },
+
+	{ SYS_DESC(SYS_TTBR0_EL2), access_rw, reset_val, TTBR0_EL2, 0 },
+	{ SYS_DESC(SYS_TTBR1_EL2), access_rw, reset_val, TTBR1_EL2, 0 },
+	{ SYS_DESC(SYS_TCR_EL2), access_rw, reset_val, TCR_EL2, TCR_EL2_RES1 },
+	{ SYS_DESC(SYS_VTTBR_EL2), access_rw, reset_val, VTTBR_EL2, 0 },
+	{ SYS_DESC(SYS_VTCR_EL2), access_rw, reset_val, VTCR_EL2, 0 },
+
 	{ SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 },
+	{ SYS_DESC(SYS_SPSR_EL2), access_rw, reset_val, SPSR_EL2, 0 },
+	{ SYS_DESC(SYS_ELR_EL2), access_rw, reset_val, ELR_EL2, 0 },
+	{ SYS_DESC(SYS_SP_EL1), access_sp_el1},
+
 	{ SYS_DESC(SYS_IFSR32_EL2), NULL, reset_unknown, IFSR32_EL2 },
+	{ SYS_DESC(SYS_AFSR0_EL2), access_rw, reset_val, AFSR0_EL2, 0 },
+	{ SYS_DESC(SYS_AFSR1_EL2), access_rw, reset_val, AFSR1_EL2, 0 },
+	{ SYS_DESC(SYS_ESR_EL2), access_rw, reset_val, ESR_EL2, 0 },
 	{ SYS_DESC(SYS_FPEXC32_EL2), NULL, reset_val, FPEXC32_EL2, 0x700 },
+
+	{ SYS_DESC(SYS_FAR_EL2), access_rw, reset_val, FAR_EL2, 0 },
+	{ SYS_DESC(SYS_HPFAR_EL2), access_rw, reset_val, HPFAR_EL2, 0 },
+
+	{ SYS_DESC(SYS_MAIR_EL2), access_rw, reset_val, MAIR_EL2, 0 },
+	{ SYS_DESC(SYS_AMAIR_EL2), access_rw, reset_val, AMAIR_EL2, 0 },
+
+	{ SYS_DESC(SYS_VBAR_EL2), access_rw, reset_val, VBAR_EL2, 0 },

arch/arm64/kvm/sys_regs.c static const struct sys_reg_desc sys_reg_descs[] = { · line 1929

Quoted 10 lines
arch/arm64/kvm/sys_regs.c +10 -0
+	{ SYS_DESC(SYS_RVBAR_EL2), access_rw, reset_val, RVBAR_EL2, 0 },
+	{ SYS_DESC(SYS_RMR_EL2), trap_undef },
+
+	{ SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 },
+	{ SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 },
+
+	{ SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 },
+	{ SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 },
+
+	{ SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 },

Doesn't this have an effect on the ability to migrate guests between identical hardware but running kernels with vs without this patch? From what I remember, QEMU fails a migration if the migration target has less system registers than the migration source.

If so, this should at the very least be spelt out in the commit message - it's a user experience breaking change. Maybe also preventing the exposure of these when NV is disabled would be a good idea?

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
author maintainer
#38 lore raw

Re: [PATCH v5 12/69] KVM: arm64: nv: Handle HCR_EL2.NV system register traps

Quoted 78 lines · Russell King (Oracle)
Quoted 20 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

ARM v8.3 introduces a new bit in the HCR_EL2, which is the NV bit. When this bit is set, accessing EL2 registers in EL1 traps to EL2. In addition, executing the following instructions in EL1 will trap to EL2:
tlbi, at, eret, and msr/mrs instructions to access SP_EL1. Most of the instructions that trap to EL2 with the NV bit were undef at EL1 prior to ARM v8.3. The only instruction that was not undef is eret.

This patch sets up a handler for EL2 registers and SP_EL1 register accesses at EL1. The host hypervisor keeps those register values in memory, and will emulate their behavior.

This patch doesn't set the NV bit yet. It will be set in a later patch once nested virtualization support is completed.

[maz: added SCTLR_EL2 RES0/RES1 handling]

---

...

Quoted 39 lines
arch/arm64/kvm/sys_regs.c +32 -0
@@ -1825,9 +1882,51 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ PMU_SYS_REG(SYS_PMCCFILTR_EL0), .access = access_pmu_evtyper,
 	  .reset = reset_val, .reg = PMCCFILTR_EL0, .val = 0 },
 
+	{ SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_val, VPIDR_EL2, 0 },
+	{ SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_val, VMPIDR_EL2, 0 },
+
+	{ SYS_DESC(SYS_SCTLR_EL2), access_sctlr_el2, reset_val, SCTLR_EL2, SCTLR_EL2_RES1 },
+	{ SYS_DESC(SYS_ACTLR_EL2), access_rw, reset_val, ACTLR_EL2, 0 },
+	{ SYS_DESC(SYS_HCR_EL2), access_rw, reset_val, HCR_EL2, 0 },
+	{ SYS_DESC(SYS_MDCR_EL2), access_rw, reset_val, MDCR_EL2, 0 },
+	{ SYS_DESC(SYS_CPTR_EL2), access_rw, reset_val, CPTR_EL2, CPTR_EL2_DEFAULT },
+	{ SYS_DESC(SYS_HSTR_EL2), access_rw, reset_val, HSTR_EL2, 0 },
+	{ SYS_DESC(SYS_HACR_EL2), access_rw, reset_val, HACR_EL2, 0 },
+
+	{ SYS_DESC(SYS_TTBR0_EL2), access_rw, reset_val, TTBR0_EL2, 0 },
+	{ SYS_DESC(SYS_TTBR1_EL2), access_rw, reset_val, TTBR1_EL2, 0 },
+	{ SYS_DESC(SYS_TCR_EL2), access_rw, reset_val, TCR_EL2, TCR_EL2_RES1 },
+	{ SYS_DESC(SYS_VTTBR_EL2), access_rw, reset_val, VTTBR_EL2, 0 },
+	{ SYS_DESC(SYS_VTCR_EL2), access_rw, reset_val, VTCR_EL2, 0 },
+
 	{ SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 },
+	{ SYS_DESC(SYS_SPSR_EL2), access_rw, reset_val, SPSR_EL2, 0 },
+	{ SYS_DESC(SYS_ELR_EL2), access_rw, reset_val, ELR_EL2, 0 },
+	{ SYS_DESC(SYS_SP_EL1), access_sp_el1},
+
 	{ SYS_DESC(SYS_IFSR32_EL2), NULL, reset_unknown, IFSR32_EL2 },
+	{ SYS_DESC(SYS_AFSR0_EL2), access_rw, reset_val, AFSR0_EL2, 0 },
+	{ SYS_DESC(SYS_AFSR1_EL2), access_rw, reset_val, AFSR1_EL2, 0 },
+	{ SYS_DESC(SYS_ESR_EL2), access_rw, reset_val, ESR_EL2, 0 },
 	{ SYS_DESC(SYS_FPEXC32_EL2), NULL, reset_val, FPEXC32_EL2, 0x700 },
+
+	{ SYS_DESC(SYS_FAR_EL2), access_rw, reset_val, FAR_EL2, 0 },
+	{ SYS_DESC(SYS_HPFAR_EL2), access_rw, reset_val, HPFAR_EL2, 0 },
+
+	{ SYS_DESC(SYS_MAIR_EL2), access_rw, reset_val, MAIR_EL2, 0 },
+	{ SYS_DESC(SYS_AMAIR_EL2), access_rw, reset_val, AMAIR_EL2, 0 },
+
+	{ SYS_DESC(SYS_VBAR_EL2), access_rw, reset_val, VBAR_EL2, 0 },

arch/arm64/kvm/sys_regs.c static const struct sys_reg_desc sys_reg_descs[] = { · line 1929

Quoted 10 lines
arch/arm64/kvm/sys_regs.c +10 -0
+	{ SYS_DESC(SYS_RVBAR_EL2), access_rw, reset_val, RVBAR_EL2, 0 },
+	{ SYS_DESC(SYS_RMR_EL2), trap_undef },
+
+	{ SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 },
+	{ SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 },
+
+	{ SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 },
+	{ SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 },
+
+	{ SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 },

Doesn't this have an effect on the ability to migrate guests between identical hardware but running kernels with vs without this patch? From what I remember, QEMU fails a migration if the migration target has less system registers than the migration source.

Quoted 3 lines

If so, this should at the very least be spelt out in the commit message - it's a user experience breaking change. Maybe also preventing the exposure of these when NV is disabled would be a good idea?

Yes, that's a known issue. I have taken steps to avoid exposing these registers (the get-reg-list test now passes on a !NV system, while it didn't before).

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
author maintainer
#39 lore raw

PATCH v5 13/69 KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

The VMPIDR_EL2 and VPIDR_EL2 are architecturally UNKNOWN at reset, but let's be nice to a guest hypervisor behaving foolishly and reset these to something reasonable anyway.

---
 arch/arm64/kvm/sys_regs.c | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)
arch/arm64/kvm/sys_regs.c +21 -4
index a23701f29858..730a24468915 100644
@@ -618,7 +618,7 @@ static void reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 	vcpu_write_sys_reg(vcpu, actlr, ACTLR_EL1);
 }
 
-static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+static u64 compute_reset_mpidr(struct kvm_vcpu *vcpu)
 {
 	u64 mpidr;
 
@@ -632,7 +632,24 @@ static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
 	mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0);
 	mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1);
 	mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2);
-	vcpu_write_sys_reg(vcpu, (1ULL << 31) | mpidr, MPIDR_EL1);
+	mpidr |= (1ULL << 31);
+
+	return mpidr;
+}
+
+static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	vcpu_write_sys_reg(vcpu, compute_reset_mpidr(vcpu), MPIDR_EL1);
+}
+
+static void reset_vmpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	vcpu_write_sys_reg(vcpu, compute_reset_mpidr(vcpu), VMPIDR_EL2);
+}
+
+static void reset_vpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)
+{
+	vcpu_write_sys_reg(vcpu, read_cpuid_id(), VPIDR_EL2);
 }
 
 static unsigned int pmu_visibility(const struct kvm_vcpu *vcpu,
@@ -1882,8 +1899,8 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ PMU_SYS_REG(SYS_PMCCFILTR_EL0), .access = access_pmu_evtyper,
 	  .reset = reset_val, .reg = PMCCFILTR_EL0, .val = 0 },
 
-	{ SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_val, VPIDR_EL2, 0 },
-	{ SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_val, VMPIDR_EL2, 0 },
+	{ SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_vpidr, VPIDR_EL2 },
+	{ SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_vmpidr, VMPIDR_EL2 },
 
 	{ SYS_DESC(SYS_SCTLR_EL2), access_sctlr_el2, reset_val, SCTLR_EL2, SCTLR_EL2_RES1 },
 	{ SYS_DESC(SYS_ACTLR_EL2), access_rw, reset_val, ACTLR_EL2, 0 },
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 13/69
#40 lore raw

Re: [PATCH v5 13/69] KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values

Quoted 8 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

The VMPIDR_EL2 and VPIDR_EL2 are architecturally UNKNOWN at reset, but let's be nice to a guest hypervisor behaving foolishly and reset these to something reasonable anyway.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
author maintainer
#41 lore raw

PATCH v5 14/69 KVM: arm64: nv: Support virtual EL2 exceptions

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor).

[maz: switch to common exception injection framework]
---
 .mailmap                             |   1 +
 arch/arm64/include/asm/kvm_arm.h     |  17 +++
 arch/arm64/include/asm/kvm_emulate.h |  10 ++
 arch/arm64/include/asm/kvm_host.h    |   1 +
 arch/arm64/kvm/Makefile              |   2 +-
 arch/arm64/kvm/emulate-nested.c      | 176 +++++++++++++++++++++++++++
 arch/arm64/kvm/hyp/exception.c       |  49 ++++++--
 arch/arm64/kvm/inject_fault.c        |  68 +++++++++--
 arch/arm64/kvm/trace_arm.h           |  59 +++++++++
 9 files changed, 362 insertions(+), 21 deletions(-)
 create mode 100644 arch/arm64/kvm/emulate-nested.c
.mailmap +1 -0
index 14314e3c5d5e..491238a888cb 100644
@@ -167,6 +167,7 @@ Jeff Layton <[email protected]> <[email protected]>
 Jens Axboe <[email protected]>
 Jens Osterkamp <[email protected]>
 Jernej Skrabec <[email protected]> <[email protected]>
+<[email protected]> <[email protected]>
 Jiri Slaby <[email protected]> <[email protected]>
 Jiri Slaby <[email protected]> <[email protected]>
 Jiri Slaby <[email protected]> <[email protected]>
arch/arm64/include/asm/kvm_arm.h +17 -0
index a39fcf318c77..589a6b92d741 100644
@@ -359,4 +359,21 @@
 #define CPACR_EL1_TTA		(1 << 28)
 #define CPACR_EL1_DEFAULT	(CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN)
 
+#define kvm_mode_names				\
+	{ PSR_MODE_EL0t,	"EL0t" },	\
+	{ PSR_MODE_EL1t,	"EL1t" },	\
+	{ PSR_MODE_EL1h,	"EL1h" },	\
+	{ PSR_MODE_EL2t,	"EL2t" },	\
+	{ PSR_MODE_EL2h,	"EL2h" },	\
+	{ PSR_MODE_EL3t,	"EL3t" },	\
+	{ PSR_MODE_EL3h,	"EL3h" },	\
+	{ PSR_AA32_MODE_USR,	"32-bit USR" },	\
+	{ PSR_AA32_MODE_FIQ,	"32-bit FIQ" },	\
+	{ PSR_AA32_MODE_IRQ,	"32-bit IRQ" },	\
+	{ PSR_AA32_MODE_SVC,	"32-bit SVC" },	\
+	{ PSR_AA32_MODE_ABT,	"32-bit ABT" },	\
+	{ PSR_AA32_MODE_HYP,	"32-bit HYP" },	\
+	{ PSR_AA32_MODE_UND,	"32-bit UND" },	\
+	{ PSR_AA32_MODE_SYS,	"32-bit SYS" }
+
 #endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/include/asm/kvm_emulate.h +10 -0
index f4b079945d0f..46c849ba281e 100644
@@ -33,6 +33,12 @@ enum exception_type {
 	except_type_serror	= 0x180,
 };
 
+#define kvm_exception_type_names		\
+	{ except_type_sync,	"SYNC"   },	\
+	{ except_type_irq,	"IRQ"    },	\
+	{ except_type_fiq,	"FIQ"    },	\
+	{ except_type_serror,	"SERROR" }
+
 bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);
 void kvm_skip_instr32(struct kvm_vcpu *vcpu);
 
@@ -41,6 +47,10 @@ void kvm_inject_vabt(struct kvm_vcpu *vcpu);
 void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr);
 void kvm_inject_pabt(struct kvm_vcpu *vcpu, unsigned long addr);
 
+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu);
+int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2);
+int kvm_inject_nested_irq(struct kvm_vcpu *vcpu);
+
 static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
 {
 	return !(vcpu->arch.hcr_el2 & HCR_RW);
arch/arm64/include/asm/kvm_host.h +1 -0
index 4f642a2e9c34..7b6fe18ee450 100644
@@ -477,6 +477,7 @@ struct kvm_vcpu_arch {
 #define KVM_ARM64_EXCEPT_AA64_ELx_SERR	(3 << 9)
 #define KVM_ARM64_EXCEPT_AA64_EL1	(0 << 11)
 #define KVM_ARM64_EXCEPT_AA64_EL2	(1 << 11)
+#define KVM_ARM64_EXCEPT_AA64_EL_MASK	(1 << 11)
 
 /*
  * Overlaps with KVM_ARM64_EXCEPT_MASK on purpose so that it can't be
arch/arm64/kvm/Makefile +1 -1
index 989bb5dad2c8..1f602526e9a2 100644
@@ -16,7 +16,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \
 	 inject_fault.o va_layout.o handle_exit.o \
 	 guest.o debug.o reset.o sys_regs.o \
 	 vgic-sys-reg-v3.o fpsimd.o pmu.o \
-	 arch_timer.o trng.o\
+	 arch_timer.o trng.o emulate-nested.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
 	 vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/emulate-nested.c new file +176 -0

176 lines not on this page · raw message

arch/arm64/kvm/hyp/exception.c +39 -10

90 lines not on this page · raw message

arch/arm64/kvm/inject_fault.c +58 -10

100 lines not on this page · raw message

arch/arm64/kvm/trace_arm.h +59 -0

71 lines not on this page · raw message

Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 14/69
#42 lore raw

Re: [PATCH v5 14/69] KVM: arm64: nv: Support virtual EL2 exceptions

Hi Marc,

Quoted 426 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor).

[maz: switch to common exception injection framework]
---
  .mailmap                             |   1 +
  arch/arm64/include/asm/kvm_arm.h     |  17 +++
  arch/arm64/include/asm/kvm_emulate.h |  10 ++
  arch/arm64/include/asm/kvm_host.h    |   1 +
  arch/arm64/kvm/Makefile              |   2 +-
  arch/arm64/kvm/emulate-nested.c      | 176 +++++++++++++++++++++++++++
  arch/arm64/kvm/hyp/exception.c       |  49 ++++++--
  arch/arm64/kvm/inject_fault.c        |  68 +++++++++--
  arch/arm64/kvm/trace_arm.h           |  59 +++++++++
  9 files changed, 362 insertions(+), 21 deletions(-)

create mode 100644 arch/arm64/kvm/emulate-nested.c

.mailmap +1 -0
index 14314e3c5d5e..491238a888cb 100644
@@ -167,6 +167,7 @@ Jeff Layton <[email protected]> <[email protected]>
  Jens Axboe <[email protected]>
  Jens Osterkamp <[email protected]>
  Jernej Skrabec <[email protected]> <[email protected]>
+<[email protected]> <[email protected]>
  Jiri Slaby <[email protected]> <[email protected]>
  Jiri Slaby <[email protected]> <[email protected]>
  Jiri Slaby <[email protected]> <[email protected]>
arch/arm64/include/asm/kvm_arm.h +17 -0
index a39fcf318c77..589a6b92d741 100644
@@ -359,4 +359,21 @@
  #define CPACR_EL1_TTA		(1 << 28)
  #define CPACR_EL1_DEFAULT	(CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN)
 
+#define kvm_mode_names				\
+	{ PSR_MODE_EL0t,	"EL0t" },	\
+	{ PSR_MODE_EL1t,	"EL1t" },	\
+	{ PSR_MODE_EL1h,	"EL1h" },	\
+	{ PSR_MODE_EL2t,	"EL2t" },	\
+	{ PSR_MODE_EL2h,	"EL2h" },	\
+	{ PSR_MODE_EL3t,	"EL3t" },	\
+	{ PSR_MODE_EL3h,	"EL3h" },	\
+	{ PSR_AA32_MODE_USR,	"32-bit USR" },	\
+	{ PSR_AA32_MODE_FIQ,	"32-bit FIQ" },	\
+	{ PSR_AA32_MODE_IRQ,	"32-bit IRQ" },	\
+	{ PSR_AA32_MODE_SVC,	"32-bit SVC" },	\
+	{ PSR_AA32_MODE_ABT,	"32-bit ABT" },	\
+	{ PSR_AA32_MODE_HYP,	"32-bit HYP" },	\
+	{ PSR_AA32_MODE_UND,	"32-bit UND" },	\
+	{ PSR_AA32_MODE_SYS,	"32-bit SYS" }
+
  #endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/include/asm/kvm_emulate.h +10 -0
index f4b079945d0f..46c849ba281e 100644
@@ -33,6 +33,12 @@ enum exception_type {
  	except_type_serror	= 0x180,
  };
 
+#define kvm_exception_type_names		\
+	{ except_type_sync,	"SYNC"   },	\
+	{ except_type_irq,	"IRQ"    },	\
+	{ except_type_fiq,	"FIQ"    },	\
+	{ except_type_serror,	"SERROR" }
+
  bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);
  void kvm_skip_instr32(struct kvm_vcpu *vcpu);
 
@@ -41,6 +47,10 @@ void kvm_inject_vabt(struct kvm_vcpu *vcpu);
  void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr);
  void kvm_inject_pabt(struct kvm_vcpu *vcpu, unsigned long addr);
 
+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu);
+int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2);
+int kvm_inject_nested_irq(struct kvm_vcpu *vcpu);
+
  static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
  {
  	return !(vcpu->arch.hcr_el2 & HCR_RW);
arch/arm64/include/asm/kvm_host.h +1 -0
index 4f642a2e9c34..7b6fe18ee450 100644
@@ -477,6 +477,7 @@ struct kvm_vcpu_arch {
  #define KVM_ARM64_EXCEPT_AA64_ELx_SERR	(3 << 9)
  #define KVM_ARM64_EXCEPT_AA64_EL1	(0 << 11)
  #define KVM_ARM64_EXCEPT_AA64_EL2	(1 << 11)
+#define KVM_ARM64_EXCEPT_AA64_EL_MASK	(1 << 11)
 
  /*
   * Overlaps with KVM_ARM64_EXCEPT_MASK on purpose so that it can't be
arch/arm64/kvm/Makefile +1 -1
index 989bb5dad2c8..1f602526e9a2 100644
@@ -16,7 +16,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \
  	 inject_fault.o va_layout.o handle_exit.o \
  	 guest.o debug.o reset.o sys_regs.o \
  	 vgic-sys-reg-v3.o fpsimd.o pmu.o \
-	 arch_timer.o trng.o\
+	 arch_timer.o trng.o emulate-nested.o \
  	 vgic/vgic.o vgic/vgic-init.o \
  	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
  	 vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/emulate-nested.c new file +176 -0
new file mode 100644
index 000000000000..339e8272b01e
@@ -0,0 +1,176 @@
+/*
+ * Copyright (C) 2016 - Linaro and Columbia University
+ * Author: Jintack Lim <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/kvm.h>
+#include <linux/kvm_host.h>
+
+#include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+
+#include "hyp/include/hyp/adjust_pc.h"
+
+#include "trace.h"
+
+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
+{
+	u64 spsr, elr, mode;
+	bool direct_eret;
+
+	/*
+	 * Going through the whole put/load motions is a waste of time
+	 * if this is a VHE guest hypervisor returning to its own
+	 * userspace, or the hypervisor performing a local exception
+	 * return. No need to save/restore registers, no need to
+	 * switch S2 MMU. Just do the canonical ERET.
+	 */
+	spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2);
+	mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	direct_eret  = (mode == PSR_MODE_EL0t &&
+			vcpu_el2_e2h_is_set(vcpu) &&
+			vcpu_el2_tge_is_set(vcpu));
+	direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
+
+	if (direct_eret) {
+		*vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2);
+		*vcpu_cpsr(vcpu) = spsr;
+		trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr);
+		return;
+	}
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);
+
+	elr = __vcpu_sys_reg(vcpu, ELR_EL2);
+
+	trace_kvm_nested_eret(vcpu, elr, spsr);
+
+	/*
+	 * Note that the current exception level is always the virtual EL2,
+	 * since we set HCR_EL2.NV bit only when entering the virtual EL2.
+	 */
+	*vcpu_pc(vcpu) = elr;
+	*vcpu_cpsr(vcpu) = spsr;
+
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();
+}
+
+static void kvm_inject_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2,
+				     enum exception_type type)
+{
+	trace_kvm_inject_nested_exception(vcpu, esr_el2, type);
+
+	switch (type) {
+	case except_type_sync:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_SYNC;
+		break;
+	case except_type_irq:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_IRQ;
+		break;
+	default:
+		WARN_ONCE(1, "Unsupported EL2 exception injection %d\n", type);
+	}
+
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL2		|
+			     KVM_ARM64_PENDING_EXCEPTION);
+
+	vcpu_write_sys_reg(vcpu, esr_el2, ESR_EL2);
+}
+
+/*
+ * Emulate taking an exception to EL2.
+ * See ARM ARM J8.1.2 AArch64.TakeException()
+ */
+static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2,
+			     enum exception_type type)
+{
+	u64 pstate, mode;
+	bool direct_inject;
+
+	if (!nested_virt_in_use(vcpu)) {
+		kvm_err("Unexpected call to %s for the non-nesting configuration\n",
+				__func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * As for ERET, we can avoid doing too much on the injection path by
+	 * checking that we either took the exception from a VHE host
+	 * userspace or from vEL2. In these cases, there is no change in
+	 * translation regime (or anything else), so let's do as little as
+	 * possible.
+	 */
+	pstate = *vcpu_cpsr(vcpu);
+	mode = pstate & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	direct_inject  = (mode == PSR_MODE_EL0t &&
+			  vcpu_el2_e2h_is_set(vcpu) &&
+			  vcpu_el2_tge_is_set(vcpu));
+	direct_inject |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
+
+	if (direct_inject) {
+		kvm_inject_el2_exception(vcpu, esr_el2, type);
+		return 1;
+	}
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);
+
+	kvm_inject_el2_exception(vcpu, esr_el2, type);
+
+	/*
+	 * A hard requirement is that a switch between EL1 and EL2
+	 * contexts has to happen between a put/load, so that we can
+	 * pick the correct timer and interrupt configuration, among
+	 * other things.
+	 *
+	 * Make sure the exception actually took place before we load
+	 * the new context.
+	 */
+	__kvm_adjust_pc(vcpu);
+
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();
+
+	return 1;
+}
+
+int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2)
+{
+	return kvm_inject_nested(vcpu, esr_el2, except_type_sync);
+}
+
+int kvm_inject_nested_irq(struct kvm_vcpu *vcpu)
+{
+	/*
+	 * Do not inject an irq if the:
+	 *  - Current exception level is EL2, and
+	 *  - virtual HCR_EL2.TGE == 0
+	 *  - virtual HCR_EL2.IMO == 0
+	 *
+	 * See Table D1-17 "Physical interrupt target and masking when EL3 is
+	 * not implemented and EL2 is implemented" in ARM DDI 0487C.a.
+	 */
+
+	if (vcpu_mode_el2(vcpu) && !vcpu_el2_tge_is_set(vcpu) &&
+	    !(__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_IMO))
+		return 1;
+
+	/* esr_el2 value doesn't matter for exits due to irqs. */
+	return kvm_inject_nested(vcpu, 0, except_type_irq);
+}
arch/arm64/kvm/hyp/exception.c +39 -10
index 0418399e0a20..4ef5e86efd8b 100644
@@ -13,6 +13,7 @@
  #include <hyp/adjust_pc.h>
  #include <linux/kvm_host.h>
  #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
 
  #if !defined (__KVM_NVHE_HYPERVISOR__) && !defined (__KVM_VHE_HYPERVISOR__)
  #error Hypervisor code only!
@@ -22,7 +23,9 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
  {
  	u64 val;
 
-	if (__vcpu_read_sys_reg_from_cpu(reg, &val))
+	if (unlikely(nested_virt_in_use(vcpu)))
+		return vcpu_read_sys_reg(vcpu, reg);
+	else if (__vcpu_read_sys_reg_from_cpu(reg, &val))
  		return val;
 
  	return __vcpu_sys_reg(vcpu, reg);
@@ -30,14 +33,24 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
 
  static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
  {
-	if (__vcpu_write_sys_reg_to_cpu(val, reg))
-		return;
-
-	 __vcpu_sys_reg(vcpu, reg) = val;
+	if (unlikely(nested_virt_in_use(vcpu)))
+		vcpu_write_sys_reg(vcpu, val, reg);
+	else if (!__vcpu_write_sys_reg_to_cpu(val, reg))
+		__vcpu_sys_reg(vcpu, reg) = val;
  }
 
-static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, u64 val)
+static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode,
+			      u64 val)
  {
+	if (unlikely(nested_virt_in_use(vcpu))) {
+		if (target_mode == PSR_MODE_EL1h)
+			vcpu_write_sys_reg(vcpu, val, SPSR_EL1);
+		else
+			vcpu_write_sys_reg(vcpu, val, SPSR_EL2);
+
+		return;
+	}
+
  	write_sysreg_el1(val, SYS_SPSR);
  }
 
@@ -97,6 +110,11 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
  		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
  		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
  		break;
+	case PSR_MODE_EL2h:
+		vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2);
+		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2);
+		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
+		break;
  	default:
  		/* Don't do that */
  		BUG();
@@ -149,7 +167,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
  	new |= target_mode;
 
  	*vcpu_cpsr(vcpu) = new;
-	__vcpu_write_spsr(vcpu, old);
+	__vcpu_write_spsr(vcpu, target_mode, old);
  }
 
  /*
@@ -320,11 +338,22 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu)
  		      KVM_ARM64_EXCEPT_AA64_EL1):
  			enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync);
  			break;
+
+		case (KVM_ARM64_EXCEPT_AA64_ELx_SYNC |
+		      KVM_ARM64_EXCEPT_AA64_EL2):
+			enter_exception64(vcpu, PSR_MODE_EL2h, except_type_sync);
+			break;
+
+		case (KVM_ARM64_EXCEPT_AA64_ELx_IRQ |
+		      KVM_ARM64_EXCEPT_AA64_EL2):
+			enter_exception64(vcpu, PSR_MODE_EL2h, except_type_irq);
+			break;
+
  		default:
  			/*
-			 * Only EL1_SYNC makes sense so far, EL2_{SYNC,IRQ}
-			 * will be implemented at some point. Everything
-			 * else gets silently ignored.
+			 * Only EL1_SYNC and EL2_{SYNC,IRQ} makes
+			 * sense so far. Everything else gets silently
+			 * ignored.
  			 */
  			break;
  		}
arch/arm64/kvm/inject_fault.c +19 -0
index b47df73e98d7..5dcf3f8b08b8 100644
@@ -12,19 +12,58 @@
 
  #include <linux/kvm_host.h>
  #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
  #include <asm/esr.h>
 
+static void pend_sync_exception(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_ELx_SYNC	|
+			     KVM_ARM64_PENDING_EXCEPTION);
+
+	/* If not nesting, EL1 is the only possible exception target */
+	if (likely(!nested_virt_in_use(vcpu))) {
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		return;
+	}
+
+	/*
+	 * With NV, we need to pick between EL1 and EL2. Note that we
+	 * never deal with a nesting exception here, hence never
+	 * changing context, and the exception itself can be delayed
+	 * until the next entry.
+	 */
+	switch(*vcpu_cpsr(vcpu) & PSR_MODE_MASK) {

arch/arm64/kvm/inject_fault.c line 45

Quoted 10 lines
arch/arm64/kvm/inject_fault.c +10 -0
+	case PSR_MODE_EL2h:
+	case PSR_MODE_EL2t:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL2;
+		break;
+	case PSR_MODE_EL1h:
+	case PSR_MODE_EL1t:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		break;
+	case PSR_MODE_EL0t:
+		if (vcpu_el2_tge_is_set(vcpu) & HCR_TGE)

IMO, AND with HCR_TGE is not needed.

Quoted 135 lines
arch/arm64/kvm/inject_fault.c +29 -10
+			vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL2;
+		else
+			vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		break;
+	default:
+		BUG();
+	}
+}
+
+static bool match_target_el(struct kvm_vcpu *vcpu, unsigned long target)
+{
+	return (vcpu->arch.flags & KVM_ARM64_EXCEPT_AA64_EL_MASK) == target;
+}
+
  static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr)
  {
  	unsigned long cpsr = *vcpu_cpsr(vcpu);
  	bool is_aarch32 = vcpu_mode_is_32bit(vcpu);
  	u32 esr = 0;
 
-	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1		|
-			     KVM_ARM64_EXCEPT_AA64_ELx_SYNC	|
-			     KVM_ARM64_PENDING_EXCEPTION);
-
-	vcpu_write_sys_reg(vcpu, addr, FAR_EL1);
+	pend_sync_exception(vcpu);
 
  	/*
  	 * Build an {i,d}abort, depending on the level and the
@@ -45,16 +84,22 @@ static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr
  	if (!is_iabt)
  		esr |= ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT;
 
-	vcpu_write_sys_reg(vcpu, esr | ESR_ELx_FSC_EXTABT, ESR_EL1);
+	esr |= ESR_ELx_FSC_EXTABT;
+
+	if (match_target_el(vcpu, KVM_ARM64_EXCEPT_AA64_EL1)) {
+		vcpu_write_sys_reg(vcpu, addr, FAR_EL1);
+		vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
+	} else {
+		vcpu_write_sys_reg(vcpu, addr, FAR_EL2);
+		vcpu_write_sys_reg(vcpu, esr, ESR_EL2);
+	}
  }
 
  static void inject_undef64(struct kvm_vcpu *vcpu)
  {
  	u32 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT);
 
-	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1		|
-			     KVM_ARM64_EXCEPT_AA64_ELx_SYNC	|
-			     KVM_ARM64_PENDING_EXCEPTION);
+	pend_sync_exception(vcpu);
 
  	/*
  	 * Build an unknown exception, depending on the instruction
@@ -63,7 +108,10 @@ static void inject_undef64(struct kvm_vcpu *vcpu)
  	if (kvm_vcpu_trap_il_is32bit(vcpu))
  		esr |= ESR_ELx_IL;
 
-	vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
+	if (match_target_el(vcpu, KVM_ARM64_EXCEPT_AA64_EL1))
+		vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
+	else
+		vcpu_write_sys_reg(vcpu, esr, ESR_EL2);
  }
 
  #define DFSR_FSC_EXTABT_LPAE	0x10
arch/arm64/kvm/trace_arm.h +52 -0
index 33e4e7dd2719..f3e46a976125 100644
@@ -2,6 +2,7 @@
  #if !defined(_TRACE_ARM_ARM64_KVM_H) || defined(TRACE_HEADER_MULTI_READ)
  #define _TRACE_ARM_ARM64_KVM_H
 
+#include <asm/kvm_emulate.h>
  #include <kvm/arm_arch_timer.h>
  #include <linux/tracepoint.h>
 
@@ -301,6 +302,64 @@ TRACE_EVENT(kvm_timer_emulate,
  		  __entry->timer_idx, __entry->should_fire)
  );
 
+TRACE_EVENT(kvm_nested_eret,
+	TP_PROTO(struct kvm_vcpu *vcpu, unsigned long elr_el2,
+		 unsigned long spsr_el2),
+	TP_ARGS(vcpu, elr_el2, spsr_el2),
+
+	TP_STRUCT__entry(
+		__field(struct kvm_vcpu *,	vcpu)
+		__field(unsigned long,		elr_el2)
+		__field(unsigned long,		spsr_el2)
+		__field(unsigned long,		target_mode)
+		__field(unsigned long,		hcr_el2)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu = vcpu;
+		__entry->elr_el2 = elr_el2;
+		__entry->spsr_el2 = spsr_el2;
+		__entry->target_mode = spsr_el2 & (PSR_MODE_MASK | PSR_MODE32_BIT);
+		__entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
+	),
+
+	TP_printk("elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx",
+		  __entry->elr_el2, __entry->spsr_el2,
+		  __print_symbolic(__entry->target_mode, kvm_mode_names),
+		  __entry->hcr_el2)
+);
+
+TRACE_EVENT(kvm_inject_nested_exception,
+	TP_PROTO(struct kvm_vcpu *vcpu, u64 esr_el2, int type),
+	TP_ARGS(vcpu, esr_el2, type),
+
+	TP_STRUCT__entry(
+		__field(struct kvm_vcpu *,		vcpu)
+		__field(unsigned long,			esr_el2)
+		__field(int,				type)
+		__field(unsigned long,			spsr_el2)
+		__field(unsigned long,			pc)
+		__field(unsigned long,			source_mode)
+		__field(unsigned long,			hcr_el2)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu = vcpu;
+		__entry->esr_el2 = esr_el2;
+		__entry->type = type;
+		__entry->spsr_el2 = *vcpu_cpsr(vcpu);
+		__entry->pc = *vcpu_pc(vcpu);
+		__entry->source_mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT);
+		__entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
+	),
+

arch/arm64/kvm/trace_arm.h TRACE_EVENT(kvm_timer_emulate, · line 365

Quoted 10 lines
arch/arm64/kvm/trace_arm.h +7 -0
+	TP_printk("%s: esr_el2 0x%lx elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx",
+		  __print_symbolic(__entry->type, kvm_exception_type_names),
+		  __entry->esr_el2, __entry->pc, __entry->spsr_el2,
+		  __print_symbolic(__entry->source_mode, kvm_mode_names),
+		  __entry->hcr_el2)
+);
+
  #endif /* _TRACE_ARM_ARM64_KVM_H */
 
  #undef TRACE_INCLUDE_PATH

It looks good to me, please feel free to add.

Thanks,
Ganapat

#43 lore raw

Re: [PATCH v5 14/69] KVM: arm64: nv: Support virtual EL2 exceptions

In reply to Marc Zyngier

Hi Marc,

Quoted 6 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor).

Might be useful to explain why emulation shouldn't (or can't) be done in software in that particular case.

It looks to me like kvm_inject_nested() and kvm_emulate_nested_eret() handles the above case, so why does does it say that emulation must not be done entirely in software?

Quoted 120 lines
[maz: switch to common exception injection framework]
---
 .mailmap                             |   1 +
 arch/arm64/include/asm/kvm_arm.h     |  17 +++
 arch/arm64/include/asm/kvm_emulate.h |  10 ++
 arch/arm64/include/asm/kvm_host.h    |   1 +
 arch/arm64/kvm/Makefile              |   2 +-
 arch/arm64/kvm/emulate-nested.c      | 176 +++++++++++++++++++++++++++
 arch/arm64/kvm/hyp/exception.c       |  49 ++++++--
 arch/arm64/kvm/inject_fault.c        |  68 +++++++++--
 arch/arm64/kvm/trace_arm.h           |  59 +++++++++
 9 files changed, 362 insertions(+), 21 deletions(-)
 create mode 100644 arch/arm64/kvm/emulate-nested.c
.mailmap +1 -0
index 14314e3c5d5e..491238a888cb 100644
@@ -167,6 +167,7 @@ Jeff Layton <[email protected]> <[email protected]>
 Jens Axboe <[email protected]>
 Jens Osterkamp <[email protected]>
 Jernej Skrabec <[email protected]> <[email protected]>
+<[email protected]> <[email protected]>
 Jiri Slaby <[email protected]> <[email protected]>
 Jiri Slaby <[email protected]> <[email protected]>
 Jiri Slaby <[email protected]> <[email protected]>
arch/arm64/include/asm/kvm_arm.h +17 -0
index a39fcf318c77..589a6b92d741 100644
@@ -359,4 +359,21 @@
 #define CPACR_EL1_TTA		(1 << 28)
 #define CPACR_EL1_DEFAULT	(CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN)
 
+#define kvm_mode_names				\
+	{ PSR_MODE_EL0t,	"EL0t" },	\
+	{ PSR_MODE_EL1t,	"EL1t" },	\
+	{ PSR_MODE_EL1h,	"EL1h" },	\
+	{ PSR_MODE_EL2t,	"EL2t" },	\
+	{ PSR_MODE_EL2h,	"EL2h" },	\
+	{ PSR_MODE_EL3t,	"EL3t" },	\
+	{ PSR_MODE_EL3h,	"EL3h" },	\
+	{ PSR_AA32_MODE_USR,	"32-bit USR" },	\
+	{ PSR_AA32_MODE_FIQ,	"32-bit FIQ" },	\
+	{ PSR_AA32_MODE_IRQ,	"32-bit IRQ" },	\
+	{ PSR_AA32_MODE_SVC,	"32-bit SVC" },	\
+	{ PSR_AA32_MODE_ABT,	"32-bit ABT" },	\
+	{ PSR_AA32_MODE_HYP,	"32-bit HYP" },	\
+	{ PSR_AA32_MODE_UND,	"32-bit UND" },	\
+	{ PSR_AA32_MODE_SYS,	"32-bit SYS" }
+
 #endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/include/asm/kvm_emulate.h +10 -0
index f4b079945d0f..46c849ba281e 100644
@@ -33,6 +33,12 @@ enum exception_type {
 	except_type_serror	= 0x180,
 };
 
+#define kvm_exception_type_names		\
+	{ except_type_sync,	"SYNC"   },	\
+	{ except_type_irq,	"IRQ"    },	\
+	{ except_type_fiq,	"FIQ"    },	\
+	{ except_type_serror,	"SERROR" }
+
 bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);
 void kvm_skip_instr32(struct kvm_vcpu *vcpu);
 
@@ -41,6 +47,10 @@ void kvm_inject_vabt(struct kvm_vcpu *vcpu);
 void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr);
 void kvm_inject_pabt(struct kvm_vcpu *vcpu, unsigned long addr);
 
+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu);
+int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2);
+int kvm_inject_nested_irq(struct kvm_vcpu *vcpu);
+
 static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
 {
 	return !(vcpu->arch.hcr_el2 & HCR_RW);
arch/arm64/include/asm/kvm_host.h +1 -0
index 4f642a2e9c34..7b6fe18ee450 100644
@@ -477,6 +477,7 @@ struct kvm_vcpu_arch {
 #define KVM_ARM64_EXCEPT_AA64_ELx_SERR	(3 << 9)
 #define KVM_ARM64_EXCEPT_AA64_EL1	(0 << 11)
 #define KVM_ARM64_EXCEPT_AA64_EL2	(1 << 11)
+#define KVM_ARM64_EXCEPT_AA64_EL_MASK	(1 << 11)
 
 /*
  * Overlaps with KVM_ARM64_EXCEPT_MASK on purpose so that it can't be
arch/arm64/kvm/Makefile +1 -1
index 989bb5dad2c8..1f602526e9a2 100644
@@ -16,7 +16,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \
 	 inject_fault.o va_layout.o handle_exit.o \
 	 guest.o debug.o reset.o sys_regs.o \
 	 vgic-sys-reg-v3.o fpsimd.o pmu.o \
-	 arch_timer.o trng.o\
+	 arch_timer.o trng.o emulate-nested.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
 	 vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/emulate-nested.c new file +5 -0
new file mode 100644
index 000000000000..339e8272b01e
@@ -0,0 +1,176 @@
+/*
+ * Copyright (C) 2016 - Linaro and Columbia University
+ * Author: Jintack Lim <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify

arch/arm64/kvm/emulate-nested.c line 15

Quoted 10 lines
arch/arm64/kvm/emulate-nested.c new file +10 -0
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.

Wouldn't an SPDX license identifier be better here?

Quoted 133 lines
arch/arm64/kvm/emulate-nested.c new file +133 -0
+ */
+
+#include <linux/kvm.h>
+#include <linux/kvm_host.h>
+
+#include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+
+#include "hyp/include/hyp/adjust_pc.h"
+
+#include "trace.h"
+
+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
+{
+	u64 spsr, elr, mode;
+	bool direct_eret;
+
+	/*
+	 * Going through the whole put/load motions is a waste of time
+	 * if this is a VHE guest hypervisor returning to its own
+	 * userspace, or the hypervisor performing a local exception
+	 * return. No need to save/restore registers, no need to
+	 * switch S2 MMU. Just do the canonical ERET.
+	 */
+	spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2);
+	mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	direct_eret  = (mode == PSR_MODE_EL0t &&
+			vcpu_el2_e2h_is_set(vcpu) &&
+			vcpu_el2_tge_is_set(vcpu));
+	direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
+
+	if (direct_eret) {
+		*vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2);
+		*vcpu_cpsr(vcpu) = spsr;
+		trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr);
+		return;
+	}
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);
+
+	elr = __vcpu_sys_reg(vcpu, ELR_EL2);
+
+	trace_kvm_nested_eret(vcpu, elr, spsr);
+
+	/*
+	 * Note that the current exception level is always the virtual EL2,
+	 * since we set HCR_EL2.NV bit only when entering the virtual EL2.
+	 */
+	*vcpu_pc(vcpu) = elr;
+	*vcpu_cpsr(vcpu) = spsr;
+
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();
+}
+
+static void kvm_inject_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2,
+				     enum exception_type type)
+{
+	trace_kvm_inject_nested_exception(vcpu, esr_el2, type);
+
+	switch (type) {
+	case except_type_sync:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_SYNC;
+		break;
+	case except_type_irq:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_IRQ;
+		break;
+	default:
+		WARN_ONCE(1, "Unsupported EL2 exception injection %d\n", type);
+	}
+
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL2		|
+			     KVM_ARM64_PENDING_EXCEPTION);
+
+	vcpu_write_sys_reg(vcpu, esr_el2, ESR_EL2);
+}
+
+/*
+ * Emulate taking an exception to EL2.
+ * See ARM ARM J8.1.2 AArch64.TakeException()
+ */
+static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2,
+			     enum exception_type type)
+{
+	u64 pstate, mode;
+	bool direct_inject;
+
+	if (!nested_virt_in_use(vcpu)) {
+		kvm_err("Unexpected call to %s for the non-nesting configuration\n",
+				__func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * As for ERET, we can avoid doing too much on the injection path by
+	 * checking that we either took the exception from a VHE host
+	 * userspace or from vEL2. In these cases, there is no change in
+	 * translation regime (or anything else), so let's do as little as
+	 * possible.
+	 */
+	pstate = *vcpu_cpsr(vcpu);
+	mode = pstate & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	direct_inject  = (mode == PSR_MODE_EL0t &&
+			  vcpu_el2_e2h_is_set(vcpu) &&
+			  vcpu_el2_tge_is_set(vcpu));
+	direct_inject |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
+
+	if (direct_inject) {
+		kvm_inject_el2_exception(vcpu, esr_el2, type);
+		return 1;
+	}
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);
+
+	kvm_inject_el2_exception(vcpu, esr_el2, type);
+
+	/*
+	 * A hard requirement is that a switch between EL1 and EL2
+	 * contexts has to happen between a put/load, so that we can
+	 * pick the correct timer and interrupt configuration, among
+	 * other things.
+	 *
+	 * Make sure the exception actually took place before we load
+	 * the new context.
+	 */
+	__kvm_adjust_pc(vcpu);
+
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();

arch/arm64/kvm/emulate-nested.c line 158

Quoted 10 lines
arch/arm64/kvm/emulate-nested.c new file +10 -0
+
+	return 1;
+}
+
+int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2)
+{
+	return kvm_inject_nested(vcpu, esr_el2, except_type_sync);
+}
+
+int kvm_inject_nested_irq(struct kvm_vcpu *vcpu)

I assume this is actually injecting an IRQ to the guest's virtual EL2, not a nested interrupt, right?

arch/arm64/kvm/emulate-nested.c line 167

Quoted 9 lines
arch/arm64/kvm/emulate-nested.c new file +9 -0
+{
+	/*
+	 * Do not inject an irq if the:
+	 *  - Current exception level is EL2, and
+	 *  - virtual HCR_EL2.TGE == 0
+	 *  - virtual HCR_EL2.IMO == 0
+	 *
+	 * See Table D1-17 "Physical interrupt target and masking when EL3 is
+	 * not implemented and EL2 is implemented" in ARM DDI 0487C.a.

That's now Table D1-13 in ARM DDI 0487G.a, the latest Arm ARM as of now, in case you want to update the comment. The table name isn't unchanged, so that can be used to find it in newer versions, so I don't think the update is strictly necessary.

I assume the comment refers to the first line of the table, as that's the only case when an interrupt is unconditionally not taken when asserted. But according to the table, the interrupt is not taken when the *target* EL is EL1 and the CPU is executing at EL2. Which makes me very confused. Isn't this function supposed to inject an interrupt to virtual EL2? If this function also injects interrupts to virtual EL1, why doesn't it also check that the target EL is EL1 before returning?

Quoted 22 lines
arch/arm64/kvm/emulate-nested.c new file +9 -0
+	 */
+
+	if (vcpu_mode_el2(vcpu) && !vcpu_el2_tge_is_set(vcpu) &&
+	    !(__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_IMO))
+		return 1;
+
+	/* esr_el2 value doesn't matter for exits due to irqs. */
+	return kvm_inject_nested(vcpu, 0, except_type_irq);
+}
arch/arm64/kvm/hyp/exception.c +1 -0
index 0418399e0a20..4ef5e86efd8b 100644
@@ -13,6 +13,7 @@
 #include <hyp/adjust_pc.h>
 #include <linux/kvm_host.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
 
 #if !defined (__KVM_NVHE_HYPERVISOR__) && !defined (__KVM_VHE_HYPERVISOR__)
 #error Hypervisor code only!

arch/arm64/kvm/hyp/exception.c static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) · line 29

Quoted 8 lines
arch/arm64/kvm/hyp/exception.c +3 -1
@@ -22,7 +23,9 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
 {
 	u64 val;
 
-	if (__vcpu_read_sys_reg_from_cpu(reg, &val))
+	if (unlikely(nested_virt_in_use(vcpu)))
+		return vcpu_read_sys_reg(vcpu, reg);
+	else if (__vcpu_read_sys_reg_from_cpu(reg, &val))
 		return val;

I find the above unintuitive. The original code duplicated the functionality from sys_regs.c::vcpu_read_sys_reg() in order to be able to run the code at EL2 in the protected KVM case. So why are we now adding back the dependency to the EL1 code? Yes, the call to vcpu_read_sys_reg() is never compiled because of how nested_virt_in_use() is defined, but still looks unconsistent with the purpose of the function (and this file).

After careful digging, I realized why this change is needed:
kvm_inject_nested() calls __kvm_adjust_pc() after a vcpu_put(), so the function needs to check vcpu->arch.sysregs_loaded_on_cpu, which now can be false. Why not copy the body of vcpu_read_sys_reg(), which is what __vcpu_read_sys_reg is supposed to do? With a comment explaining why sysregs_loaded_on_cpu can be false, I think the code would be a lot more understandable.

Or is it that you want all VCPU register accesses to be done with vcpu_read_sys_reg() when the VCPU has the EL2 feature?

Quoted 94 lines
arch/arm64/kvm/hyp/exception.c +35 -9
 	return __vcpu_sys_reg(vcpu, reg);
@@ -30,14 +33,24 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
 
 static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
 {
-	if (__vcpu_write_sys_reg_to_cpu(val, reg))
-		return;
-
-	 __vcpu_sys_reg(vcpu, reg) = val;
+	if (unlikely(nested_virt_in_use(vcpu)))
+		vcpu_write_sys_reg(vcpu, val, reg);
+	else if (!__vcpu_write_sys_reg_to_cpu(val, reg))
+		__vcpu_sys_reg(vcpu, reg) = val;
 }
 
-static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, u64 val)
+static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode,
+			      u64 val)
 {
+	if (unlikely(nested_virt_in_use(vcpu))) {
+		if (target_mode == PSR_MODE_EL1h)
+			vcpu_write_sys_reg(vcpu, val, SPSR_EL1);
+		else
+			vcpu_write_sys_reg(vcpu, val, SPSR_EL2);
+
+		return;
+	}
+
 	write_sysreg_el1(val, SYS_SPSR);
 }
 
@@ -97,6 +110,11 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
 		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
 		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
 		break;
+	case PSR_MODE_EL2h:
+		vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2);
+		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2);
+		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
+		break;
 	default:
 		/* Don't do that */
 		BUG();
@@ -149,7 +167,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
 	new |= target_mode;
 
 	*vcpu_cpsr(vcpu) = new;
-	__vcpu_write_spsr(vcpu, old);
+	__vcpu_write_spsr(vcpu, target_mode, old);
 }
 
 /*
@@ -320,11 +338,22 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu)
 		      KVM_ARM64_EXCEPT_AA64_EL1):
 			enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync);
 			break;
+
+		case (KVM_ARM64_EXCEPT_AA64_ELx_SYNC |
+		      KVM_ARM64_EXCEPT_AA64_EL2):
+			enter_exception64(vcpu, PSR_MODE_EL2h, except_type_sync);
+			break;
+
+		case (KVM_ARM64_EXCEPT_AA64_ELx_IRQ |
+		      KVM_ARM64_EXCEPT_AA64_EL2):
+			enter_exception64(vcpu, PSR_MODE_EL2h, except_type_irq);
+			break;
+
 		default:
 			/*
-			 * Only EL1_SYNC makes sense so far, EL2_{SYNC,IRQ}
-			 * will be implemented at some point. Everything
-			 * else gets silently ignored.
+			 * Only EL1_SYNC and EL2_{SYNC,IRQ} makes
+			 * sense so far. Everything else gets silently
+			 * ignored.
 			 */
 			break;
 		}
arch/arm64/kvm/inject_fault.c +5 -0
index b47df73e98d7..5dcf3f8b08b8 100644
@@ -12,19 +12,58 @@
 
 #include <linux/kvm_host.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
 #include <asm/esr.h>
 
+static void pend_sync_exception(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_ELx_SYNC	|
+			     KVM_ARM64_PENDING_EXCEPTION);

arch/arm64/kvm/inject_fault.c line 31

Quoted 10 lines
arch/arm64/kvm/inject_fault.c +10 -0
+
+	/* If not nesting, EL1 is the only possible exception target */
+	if (likely(!nested_virt_in_use(vcpu))) {
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		return;
+	}
+
+	/*
+	 * With NV, we need to pick between EL1 and EL2. Note that we
+	 * never deal with a nesting exception here, hence never

I don't understand what "a nesting exception" refers to. In emulate_nested.c, the comment for kvm_inject_nested() says that it emulates taking an exception to EL2, so I assume a *nested* exception is an exception injected to guest virtual EL2. But what is a *nesting* exception? Am I misunderstanding something?

Quoted 4 lines
arch/arm64/kvm/inject_fault.c +4 -0
+	 * changing context, and the exception itself can be delayed
+	 * until the next entry.
+	 */
+	switch(*vcpu_cpsr(vcpu) & PSR_MODE_MASK) {

arch/arm64/kvm/inject_fault.c line 45

Quoted 10 lines
arch/arm64/kvm/inject_fault.c +10 -0
+	case PSR_MODE_EL2h:
+	case PSR_MODE_EL2t:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL2;
+		break;
+	case PSR_MODE_EL1h:
+	case PSR_MODE_EL1t:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		break;
+	case PSR_MODE_EL0t:
+		if (vcpu_el2_tge_is_set(vcpu) & HCR_TGE)

Doesn't vcpu_el2_tge_is_set() already return true if HCR_TGE is set?

Quoted 148 lines
arch/arm64/kvm/inject_fault.c +29 -10
+			vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL2;
+		else
+			vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		break;
+	default:
+		BUG();
+	}
+}
+
+static bool match_target_el(struct kvm_vcpu *vcpu, unsigned long target)
+{
+	return (vcpu->arch.flags & KVM_ARM64_EXCEPT_AA64_EL_MASK) == target;
+}
+
 static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr)
 {
 	unsigned long cpsr = *vcpu_cpsr(vcpu);
 	bool is_aarch32 = vcpu_mode_is_32bit(vcpu);
 	u32 esr = 0;
 
-	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1		|
-			     KVM_ARM64_EXCEPT_AA64_ELx_SYNC	|
-			     KVM_ARM64_PENDING_EXCEPTION);
-
-	vcpu_write_sys_reg(vcpu, addr, FAR_EL1);
+	pend_sync_exception(vcpu);
 
 	/*
 	 * Build an {i,d}abort, depending on the level and the
@@ -45,16 +84,22 @@ static void inject_abt64(struct kvm_vcpu *vcpu, bool is_iabt, unsigned long addr
 	if (!is_iabt)
 		esr |= ESR_ELx_EC_DABT_LOW << ESR_ELx_EC_SHIFT;
 
-	vcpu_write_sys_reg(vcpu, esr | ESR_ELx_FSC_EXTABT, ESR_EL1);
+	esr |= ESR_ELx_FSC_EXTABT;
+
+	if (match_target_el(vcpu, KVM_ARM64_EXCEPT_AA64_EL1)) {
+		vcpu_write_sys_reg(vcpu, addr, FAR_EL1);
+		vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
+	} else {
+		vcpu_write_sys_reg(vcpu, addr, FAR_EL2);
+		vcpu_write_sys_reg(vcpu, esr, ESR_EL2);
+	}
 }
 
 static void inject_undef64(struct kvm_vcpu *vcpu)
 {
 	u32 esr = (ESR_ELx_EC_UNKNOWN << ESR_ELx_EC_SHIFT);
 
-	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1		|
-			     KVM_ARM64_EXCEPT_AA64_ELx_SYNC	|
-			     KVM_ARM64_PENDING_EXCEPTION);
+	pend_sync_exception(vcpu);
 
 	/*
 	 * Build an unknown exception, depending on the instruction
@@ -63,7 +108,10 @@ static void inject_undef64(struct kvm_vcpu *vcpu)
 	if (kvm_vcpu_trap_il_is32bit(vcpu))
 		esr |= ESR_ELx_IL;
 
-	vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
+	if (match_target_el(vcpu, KVM_ARM64_EXCEPT_AA64_EL1))
+		vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
+	else
+		vcpu_write_sys_reg(vcpu, esr, ESR_EL2);
 }
 
 #define DFSR_FSC_EXTABT_LPAE	0x10
arch/arm64/kvm/trace_arm.h +59 -0
index 33e4e7dd2719..f3e46a976125 100644
@@ -2,6 +2,7 @@
 #if !defined(_TRACE_ARM_ARM64_KVM_H) || defined(TRACE_HEADER_MULTI_READ)
 #define _TRACE_ARM_ARM64_KVM_H
 
+#include <asm/kvm_emulate.h>
 #include <kvm/arm_arch_timer.h>
 #include <linux/tracepoint.h>
 
@@ -301,6 +302,64 @@ TRACE_EVENT(kvm_timer_emulate,
 		  __entry->timer_idx, __entry->should_fire)
 );
 
+TRACE_EVENT(kvm_nested_eret,
+	TP_PROTO(struct kvm_vcpu *vcpu, unsigned long elr_el2,
+		 unsigned long spsr_el2),
+	TP_ARGS(vcpu, elr_el2, spsr_el2),
+
+	TP_STRUCT__entry(
+		__field(struct kvm_vcpu *,	vcpu)
+		__field(unsigned long,		elr_el2)
+		__field(unsigned long,		spsr_el2)
+		__field(unsigned long,		target_mode)
+		__field(unsigned long,		hcr_el2)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu = vcpu;
+		__entry->elr_el2 = elr_el2;
+		__entry->spsr_el2 = spsr_el2;
+		__entry->target_mode = spsr_el2 & (PSR_MODE_MASK | PSR_MODE32_BIT);
+		__entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
+	),
+
+	TP_printk("elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx",
+		  __entry->elr_el2, __entry->spsr_el2,
+		  __print_symbolic(__entry->target_mode, kvm_mode_names),
+		  __entry->hcr_el2)
+);
+
+TRACE_EVENT(kvm_inject_nested_exception,
+	TP_PROTO(struct kvm_vcpu *vcpu, u64 esr_el2, int type),
+	TP_ARGS(vcpu, esr_el2, type),
+
+	TP_STRUCT__entry(
+		__field(struct kvm_vcpu *,		vcpu)
+		__field(unsigned long,			esr_el2)
+		__field(int,				type)
+		__field(unsigned long,			spsr_el2)
+		__field(unsigned long,			pc)
+		__field(unsigned long,			source_mode)
+		__field(unsigned long,			hcr_el2)
+	),
+
+	TP_fast_assign(
+		__entry->vcpu = vcpu;
+		__entry->esr_el2 = esr_el2;
+		__entry->type = type;
+		__entry->spsr_el2 = *vcpu_cpsr(vcpu);
+		__entry->pc = *vcpu_pc(vcpu);
+		__entry->source_mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT);
+		__entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
+	),
+
+	TP_printk("%s: esr_el2 0x%lx elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx",
+		  __print_symbolic(__entry->type, kvm_exception_type_names),
+		  __entry->esr_el2, __entry->pc, __entry->spsr_el2,
+		  __print_symbolic(__entry->source_mode, kvm_mode_names),
+		  __entry->hcr_el2)
+);
+
 #endif /* _TRACE_ARM_ARM64_KVM_H */
 
 #undef TRACE_INCLUDE_PATH
Signature · 1 line
2.30.2
author maintainer
#44 lore raw

Re: [PATCH v5 14/69] KVM: arm64: nv: Support virtual EL2 exceptions

Quoted 11 lines · Alexandru Elisei

Hi Marc,

Quoted 6 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor).

Quoted 6 lines

Might be useful to explain why emulation shouldn't (or can't) be done in software in that particular case.

It looks to me like kvm_inject_nested() and kvm_emulate_nested_eret() handles the above case, so why does does it say that emulation must not be done entirely in software?

Not quite. What can be done in HW is the transition between EL0 and vEL2 when the guest is VHE (where EL2 looks just like EL1). Everything else has to be mediated one way or the other.

Quoted 125 lines
Quoted 125 lines
[maz: switch to common exception injection framework]
---
 .mailmap                             |   1 +
 arch/arm64/include/asm/kvm_arm.h     |  17 +++
 arch/arm64/include/asm/kvm_emulate.h |  10 ++
 arch/arm64/include/asm/kvm_host.h    |   1 +
 arch/arm64/kvm/Makefile              |   2 +-
 arch/arm64/kvm/emulate-nested.c      | 176 +++++++++++++++++++++++++++
 arch/arm64/kvm/hyp/exception.c       |  49 ++++++--
 arch/arm64/kvm/inject_fault.c        |  68 +++++++++--
 arch/arm64/kvm/trace_arm.h           |  59 +++++++++
 9 files changed, 362 insertions(+), 21 deletions(-)
 create mode 100644 arch/arm64/kvm/emulate-nested.c
.mailmap +1 -0
index 14314e3c5d5e..491238a888cb 100644
@@ -167,6 +167,7 @@ Jeff Layton <[email protected]> <[email protected]>
 Jens Axboe <[email protected]>
 Jens Osterkamp <[email protected]>
 Jernej Skrabec <[email protected]> <[email protected]>
+<[email protected]> <[email protected]>
 Jiri Slaby <[email protected]> <[email protected]>
 Jiri Slaby <[email protected]> <[email protected]>
 Jiri Slaby <[email protected]> <[email protected]>
arch/arm64/include/asm/kvm_arm.h +17 -0
index a39fcf318c77..589a6b92d741 100644
@@ -359,4 +359,21 @@
 #define CPACR_EL1_TTA		(1 << 28)
 #define CPACR_EL1_DEFAULT	(CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN)
 
+#define kvm_mode_names				\
+	{ PSR_MODE_EL0t,	"EL0t" },	\
+	{ PSR_MODE_EL1t,	"EL1t" },	\
+	{ PSR_MODE_EL1h,	"EL1h" },	\
+	{ PSR_MODE_EL2t,	"EL2t" },	\
+	{ PSR_MODE_EL2h,	"EL2h" },	\
+	{ PSR_MODE_EL3t,	"EL3t" },	\
+	{ PSR_MODE_EL3h,	"EL3h" },	\
+	{ PSR_AA32_MODE_USR,	"32-bit USR" },	\
+	{ PSR_AA32_MODE_FIQ,	"32-bit FIQ" },	\
+	{ PSR_AA32_MODE_IRQ,	"32-bit IRQ" },	\
+	{ PSR_AA32_MODE_SVC,	"32-bit SVC" },	\
+	{ PSR_AA32_MODE_ABT,	"32-bit ABT" },	\
+	{ PSR_AA32_MODE_HYP,	"32-bit HYP" },	\
+	{ PSR_AA32_MODE_UND,	"32-bit UND" },	\
+	{ PSR_AA32_MODE_SYS,	"32-bit SYS" }
+
 #endif /* __ARM64_KVM_ARM_H__ */
arch/arm64/include/asm/kvm_emulate.h +10 -0
index f4b079945d0f..46c849ba281e 100644
@@ -33,6 +33,12 @@ enum exception_type {
 	except_type_serror	= 0x180,
 };
 
+#define kvm_exception_type_names		\
+	{ except_type_sync,	"SYNC"   },	\
+	{ except_type_irq,	"IRQ"    },	\
+	{ except_type_fiq,	"FIQ"    },	\
+	{ except_type_serror,	"SERROR" }
+
 bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);
 void kvm_skip_instr32(struct kvm_vcpu *vcpu);
 
@@ -41,6 +47,10 @@ void kvm_inject_vabt(struct kvm_vcpu *vcpu);
 void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr);
 void kvm_inject_pabt(struct kvm_vcpu *vcpu, unsigned long addr);
 
+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu);
+int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2);
+int kvm_inject_nested_irq(struct kvm_vcpu *vcpu);
+
 static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
 {
 	return !(vcpu->arch.hcr_el2 & HCR_RW);
arch/arm64/include/asm/kvm_host.h +1 -0
index 4f642a2e9c34..7b6fe18ee450 100644
@@ -477,6 +477,7 @@ struct kvm_vcpu_arch {
 #define KVM_ARM64_EXCEPT_AA64_ELx_SERR	(3 << 9)
 #define KVM_ARM64_EXCEPT_AA64_EL1	(0 << 11)
 #define KVM_ARM64_EXCEPT_AA64_EL2	(1 << 11)
+#define KVM_ARM64_EXCEPT_AA64_EL_MASK	(1 << 11)
 
 /*
  * Overlaps with KVM_ARM64_EXCEPT_MASK on purpose so that it can't be
arch/arm64/kvm/Makefile +1 -1
index 989bb5dad2c8..1f602526e9a2 100644
@@ -16,7 +16,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \
 	 inject_fault.o va_layout.o handle_exit.o \
 	 guest.o debug.o reset.o sys_regs.o \
 	 vgic-sys-reg-v3.o fpsimd.o pmu.o \
-	 arch_timer.o trng.o\
+	 arch_timer.o trng.o emulate-nested.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
 	 vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/emulate-nested.c new file +9 -0
new file mode 100644
index 000000000000..339e8272b01e
@@ -0,0 +1,176 @@
+/*
+ * Copyright (C) 2016 - Linaro and Columbia University
+ * Author: Jintack Lim <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
Quoted 8 lines

arch/arm64/kvm/emulate-nested.c line 15

Quoted 6 lines
arch/arm64/kvm/emulate-nested.c new file +6 -0
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.

Wouldn't an SPDX license identifier be better here?

Yup, done.

Quoted 139 lines
Quoted 139 lines
arch/arm64/kvm/emulate-nested.c new file +138 -0
+ */
+
+#include <linux/kvm.h>
+#include <linux/kvm_host.h>
+
+#include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+
+#include "hyp/include/hyp/adjust_pc.h"
+
+#include "trace.h"
+
+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
+{
+	u64 spsr, elr, mode;
+	bool direct_eret;
+
+	/*
+	 * Going through the whole put/load motions is a waste of time
+	 * if this is a VHE guest hypervisor returning to its own
+	 * userspace, or the hypervisor performing a local exception
+	 * return. No need to save/restore registers, no need to
+	 * switch S2 MMU. Just do the canonical ERET.
+	 */
+	spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2);
+	mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	direct_eret  = (mode == PSR_MODE_EL0t &&
+			vcpu_el2_e2h_is_set(vcpu) &&
+			vcpu_el2_tge_is_set(vcpu));
+	direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
+
+	if (direct_eret) {
+		*vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2);
+		*vcpu_cpsr(vcpu) = spsr;
+		trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr);
+		return;
+	}
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);
+
+	elr = __vcpu_sys_reg(vcpu, ELR_EL2);
+
+	trace_kvm_nested_eret(vcpu, elr, spsr);
+
+	/*
+	 * Note that the current exception level is always the virtual EL2,
+	 * since we set HCR_EL2.NV bit only when entering the virtual EL2.
+	 */
+	*vcpu_pc(vcpu) = elr;
+	*vcpu_cpsr(vcpu) = spsr;
+
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();
+}
+
+static void kvm_inject_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2,
+				     enum exception_type type)
+{
+	trace_kvm_inject_nested_exception(vcpu, esr_el2, type);
+
+	switch (type) {
+	case except_type_sync:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_SYNC;
+		break;
+	case except_type_irq:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_IRQ;
+		break;
+	default:
+		WARN_ONCE(1, "Unsupported EL2 exception injection %d\n", type);
+	}
+
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL2		|
+			     KVM_ARM64_PENDING_EXCEPTION);
+
+	vcpu_write_sys_reg(vcpu, esr_el2, ESR_EL2);
+}
+
+/*
+ * Emulate taking an exception to EL2.
+ * See ARM ARM J8.1.2 AArch64.TakeException()
+ */
+static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2,
+			     enum exception_type type)
+{
+	u64 pstate, mode;
+	bool direct_inject;
+
+	if (!nested_virt_in_use(vcpu)) {
+		kvm_err("Unexpected call to %s for the non-nesting configuration\n",
+				__func__);
+		return -EINVAL;
+	}
+
+	/*
+	 * As for ERET, we can avoid doing too much on the injection path by
+	 * checking that we either took the exception from a VHE host
+	 * userspace or from vEL2. In these cases, there is no change in
+	 * translation regime (or anything else), so let's do as little as
+	 * possible.
+	 */
+	pstate = *vcpu_cpsr(vcpu);
+	mode = pstate & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	direct_inject  = (mode == PSR_MODE_EL0t &&
+			  vcpu_el2_e2h_is_set(vcpu) &&
+			  vcpu_el2_tge_is_set(vcpu));
+	direct_inject |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
+
+	if (direct_inject) {
+		kvm_inject_el2_exception(vcpu, esr_el2, type);
+		return 1;
+	}
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);
+
+	kvm_inject_el2_exception(vcpu, esr_el2, type);
+
+	/*
+	 * A hard requirement is that a switch between EL1 and EL2
+	 * contexts has to happen between a put/load, so that we can
+	 * pick the correct timer and interrupt configuration, among
+	 * other things.
+	 *
+	 * Make sure the exception actually took place before we load
+	 * the new context.
+	 */
+	__kvm_adjust_pc(vcpu);
+
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();
+
+	return 1;
+}
+
+int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2)
Quoted 8 lines

arch/arm64/kvm/emulate-nested.c line 158

Quoted 5 lines
arch/arm64/kvm/emulate-nested.c new file +5 -0
+{
+	return kvm_inject_nested(vcpu, esr_el2, except_type_sync);
+}
+
+int kvm_inject_nested_irq(struct kvm_vcpu *vcpu)

I assume this is actually injecting an IRQ to the guest's virtual EL2, not a nested interrupt, right?

Neither, really. This is injecting an IRQ *exception* into the guest. The actual interrupt delivery is done via the vgic. If nested interrupts (in the interrupt preemption sense) have to happen, the vgic will take care of it.

Quoted 16 lines

arch/arm64/kvm/emulate-nested.c line 167

Quoted 10 lines
arch/arm64/kvm/emulate-nested.c new file +9 -0
+{
+	/*
+	 * Do not inject an irq if the:
+	 *  - Current exception level is EL2, and
+	 *  - virtual HCR_EL2.TGE == 0
+	 *  - virtual HCR_EL2.IMO == 0
+	 *
+	 * See Table D1-17 "Physical interrupt target and masking when EL3 is
+	 * not implemented and EL2 is implemented" in ARM DDI 0487C.a.

That's now Table D1-13 in ARM DDI 0487G.a, the latest Arm ARM as of now, in case you want to update the comment. The table name isn't unchanged, so that can be used to find it in newer versions, so I don't think the update is strictly necessary.

Quoted 7 lines

I assume the comment refers to the first line of the table, as that's the only case when an interrupt is unconditionally not taken when asserted. But according to the table, the interrupt is not taken when the *target* EL is EL1 and the CPU is executing at EL2. Which makes me very confused. Isn't this function supposed to inject an interrupt to virtual EL2? If this function also injects interrupts to virtual EL1, why doesn't it also check that the target EL is EL1 before returning?

This is already checked by the caller: see kvm_vgic_flush_hwstate() which has the following sequence:

	if (vgic_state_is_nested(vcpu)) {
		if (kvm_vgic_vcpu_pending_irq(vcpu))
			kvm_make_request(KVM_REQ_GUEST_HYP_IRQ_PENDING, vcpu);

which ends up triggering the exception injection.

Quoted 47 lines
Quoted 23 lines
arch/arm64/kvm/emulate-nested.c new file +9 -0
+	 */
+
+	if (vcpu_mode_el2(vcpu) && !vcpu_el2_tge_is_set(vcpu) &&
+	    !(__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_IMO))
+		return 1;
+
+	/* esr_el2 value doesn't matter for exits due to irqs. */
+	return kvm_inject_nested(vcpu, 0, except_type_irq);
+}
arch/arm64/kvm/hyp/exception.c +1 -0
index 0418399e0a20..4ef5e86efd8b 100644
@@ -13,6 +13,7 @@
 #include <hyp/adjust_pc.h>
 #include <linux/kvm_host.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
 
 #if !defined (__KVM_NVHE_HYPERVISOR__) && !defined (__KVM_VHE_HYPERVISOR__)
 #error Hypervisor code only!

arch/arm64/kvm/hyp/exception.c static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) · line 29

Quoted 8 lines
arch/arm64/kvm/hyp/exception.c +3 -1
@@ -22,7 +23,9 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
 {
 	u64 val;
 
-	if (__vcpu_read_sys_reg_from_cpu(reg, &val))
+	if (unlikely(nested_virt_in_use(vcpu)))
+		return vcpu_read_sys_reg(vcpu, reg);
+	else if (__vcpu_read_sys_reg_from_cpu(reg, &val))
 		return val;

I find the above unintuitive. The original code duplicated the functionality from sys_regs.c::vcpu_read_sys_reg() in order to be able to run the code at EL2 in the protected KVM case. So why are we now adding back the dependency to the EL1 code? Yes, the call to vcpu_read_sys_reg() is never compiled because of how nested_virt_in_use() is defined, but still looks unconsistent with the purpose of the function (and this file).

After careful digging, I realized why this change is needed:
kvm_inject_nested() calls __kvm_adjust_pc() after a vcpu_put(), so the function needs to check vcpu->arch.sysregs_loaded_on_cpu, which now can be false. Why not copy the body of vcpu_read_sys_reg(), which is what __vcpu_read_sys_reg is supposed to do? With a comment explaining why sysregs_loaded_on_cpu can be false, I think the code would be a lot more understandable.

Quoted 2 lines

Or is it that you want all VCPU register accesses to be done with vcpu_read_sys_reg() when the VCPU has the EL2 feature?

Both. The whole NV support is based on this "put/load" sequences when we alternate between EL{0,1} and EL2, and tracking whether you're loaded or not is really difficult. I agree that this is pretty ugly, but I also don't want to spread the crap everywhere (and I have trouble keeping both the NV and pKVM models in my head at the same time!).

Quoted 103 lines
Quoted 103 lines
arch/arm64/kvm/hyp/exception.c +35 -9
 	return __vcpu_sys_reg(vcpu, reg);
@@ -30,14 +33,24 @@ static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
 
 static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
 {
-	if (__vcpu_write_sys_reg_to_cpu(val, reg))
-		return;
-
-	 __vcpu_sys_reg(vcpu, reg) = val;
+	if (unlikely(nested_virt_in_use(vcpu)))
+		vcpu_write_sys_reg(vcpu, val, reg);
+	else if (!__vcpu_write_sys_reg_to_cpu(val, reg))
+		__vcpu_sys_reg(vcpu, reg) = val;
 }
 
-static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, u64 val)
+static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode,
+			      u64 val)
 {
+	if (unlikely(nested_virt_in_use(vcpu))) {
+		if (target_mode == PSR_MODE_EL1h)
+			vcpu_write_sys_reg(vcpu, val, SPSR_EL1);
+		else
+			vcpu_write_sys_reg(vcpu, val, SPSR_EL2);
+
+		return;
+	}
+
 	write_sysreg_el1(val, SYS_SPSR);
 }
 
@@ -97,6 +110,11 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
 		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1);
 		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1);
 		break;
+	case PSR_MODE_EL2h:
+		vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2);
+		sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2);
+		__vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2);
+		break;
 	default:
 		/* Don't do that */
 		BUG();
@@ -149,7 +167,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
 	new |= target_mode;
 
 	*vcpu_cpsr(vcpu) = new;
-	__vcpu_write_spsr(vcpu, old);
+	__vcpu_write_spsr(vcpu, target_mode, old);
 }
 
 /*
@@ -320,11 +338,22 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu)
 		      KVM_ARM64_EXCEPT_AA64_EL1):
 			enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync);
 			break;
+
+		case (KVM_ARM64_EXCEPT_AA64_ELx_SYNC |
+		      KVM_ARM64_EXCEPT_AA64_EL2):
+			enter_exception64(vcpu, PSR_MODE_EL2h, except_type_sync);
+			break;
+
+		case (KVM_ARM64_EXCEPT_AA64_ELx_IRQ |
+		      KVM_ARM64_EXCEPT_AA64_EL2):
+			enter_exception64(vcpu, PSR_MODE_EL2h, except_type_irq);
+			break;
+
 		default:
 			/*
-			 * Only EL1_SYNC makes sense so far, EL2_{SYNC,IRQ}
-			 * will be implemented at some point. Everything
-			 * else gets silently ignored.
+			 * Only EL1_SYNC and EL2_{SYNC,IRQ} makes
+			 * sense so far. Everything else gets silently
+			 * ignored.
 			 */
 			break;
 		}
arch/arm64/kvm/inject_fault.c +13 -0
index b47df73e98d7..5dcf3f8b08b8 100644
@@ -12,19 +12,58 @@
 
 #include <linux/kvm_host.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
 #include <asm/esr.h>
 
+static void pend_sync_exception(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_ELx_SYNC	|
+			     KVM_ARM64_PENDING_EXCEPTION);
+
+	/* If not nesting, EL1 is the only possible exception target */
+	if (likely(!nested_virt_in_use(vcpu))) {
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		return;
+	}
+
+	/*
Quoted 8 lines

arch/arm64/kvm/inject_fault.c line 31

Quoted 2 lines
arch/arm64/kvm/inject_fault.c +2 -0
+	 * With NV, we need to pick between EL1 and EL2. Note that we
+	 * never deal with a nesting exception here, hence never

I don't understand what "a nesting exception" refers to. In emulate_nested.c, the comment for kvm_inject_nested() says that it emulates taking an exception to EL2, so I assume a *nested* exception is an exception injected to guest virtual EL2. But what is a *nesting* exception? Am I misunderstanding something?

Same thing. It is just that the code and the comments have been written by 3 different people, none of whom being native English speakers, and this is bound to be inconsistent.

Just get over it. ;-)

Quoted 17 lines
Quoted 5 lines
arch/arm64/kvm/inject_fault.c +4 -0
+	 * changing context, and the exception itself can be delayed
+	 * until the next entry.
+	 */
+	switch(*vcpu_cpsr(vcpu) & PSR_MODE_MASK) {

arch/arm64/kvm/inject_fault.c line 45

Quoted 10 lines
arch/arm64/kvm/inject_fault.c +10 -0
+	case PSR_MODE_EL2h:
+	case PSR_MODE_EL2t:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL2;
+		break;
+	case PSR_MODE_EL1h:
+	case PSR_MODE_EL1t:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		break;
+	case PSR_MODE_EL0t:
+		if (vcpu_el2_tge_is_set(vcpu) & HCR_TGE)

Doesn't vcpu_el2_tge_is_set() already return true if HCR_TGE is set?

Weee!!! Nice catch! Good thing we almost never inject any of those (we mostly run wel;l behaved SW), as we'd target the wrong exception level...

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#45 lore raw

Re: [PATCH v5 14/69] KVM: arm64: nv: Support virtual EL2 exceptions

In reply to Marc Zyngier
Quoted 12 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor).

[maz: switch to common exception injection framework]

---

...

Quoted 9 lines
arch/arm64/kvm/emulate-nested.c new file +9 -0
+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
+{
+	u64 spsr, elr, mode;
+	bool direct_eret;
+
+	/*
+	 * Going through the whole put/load motions is a waste of time
+	 * if this is a VHE guest hypervisor returning to its own
+	 * userspace, or the hypervisor performing a local exception

arch/arm64/kvm/emulate-nested.c line 46

Quoted 10 lines
arch/arm64/kvm/emulate-nested.c new file +10 -0
+	 * return. No need to save/restore registers, no need to
+	 * switch S2 MMU. Just do the canonical ERET.
+	 */
+	spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2);
+	mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	direct_eret  = (mode == PSR_MODE_EL0t &&
+			vcpu_el2_e2h_is_set(vcpu) &&
+			vcpu_el2_tge_is_set(vcpu));
+	direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);

There are excessive parens on the RHS of the above two.

Quoted 8 lines
arch/arm64/kvm/emulate-nested.c new file +8 -0
+static void kvm_inject_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2,
+				     enum exception_type type)
+{
+	trace_kvm_inject_nested_exception(vcpu, esr_el2, type);
+
+	switch (type) {
+	case except_type_sync:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_SYNC;

arch/arm64/kvm/emulate-nested.c line 90

Quoted 10 lines
arch/arm64/kvm/emulate-nested.c new file +10 -0
+		break;
+	case except_type_irq:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_ELx_IRQ;
+		break;
+	default:
+		WARN_ONCE(1, "Unsupported EL2 exception injection %d\n", type);
+	}
+
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL2		|
+			     KVM_ARM64_PENDING_EXCEPTION);

Ditto, and the "|" has too much white space before it with no good reason.

arch/arm64/kvm/emulate-nested.c line 107

Quoted 13 lines
arch/arm64/kvm/emulate-nested.c new file +13 -0
+/*
+ * Emulate taking an exception to EL2.
+ * See ARM ARM J8.1.2 AArch64.TakeException()
+ */
+static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2,
+			     enum exception_type type)
+{
+	u64 pstate, mode;
+	bool direct_inject;
+
+	if (!nested_virt_in_use(vcpu)) {
+		kvm_err("Unexpected call to %s for the non-nesting configuration\n",
+				__func__);

Too much indentation. I'm guessing this "unexpected" condition isn't something that can be caused by a rogue guest? If it can, doesn't this need to be rate limited?

Quoted 7 lines
arch/arm64/kvm/emulate-nested.c new file +7 -0
+		return -EINVAL;
+	}
+
+	/*
+	 * As for ERET, we can avoid doing too much on the injection path by
+	 * checking that we either took the exception from a VHE host
+	 * userspace or from vEL2. In these cases, there is no change in

arch/arm64/kvm/emulate-nested.c line 124

Quoted 10 lines
arch/arm64/kvm/emulate-nested.c new file +10 -0
+	 * translation regime (or anything else), so let's do as little as
+	 * possible.
+	 */
+	pstate = *vcpu_cpsr(vcpu);
+	mode = pstate & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	direct_inject  = (mode == PSR_MODE_EL0t &&
+			  vcpu_el2_e2h_is_set(vcpu) &&
+			  vcpu_el2_tge_is_set(vcpu));
+	direct_inject |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);

Too many parens again...

Quoted 4 lines
arch/arm64/kvm/hyp/exception.c +0 -0
index 0418399e0a20..4ef5e86efd8b 100644

...

Quoted 10 lines
arch/arm64/kvm/hyp/exception.c +1 -1
@@ -149,7 +167,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode,
 	new |= target_mode;
 
 	*vcpu_cpsr(vcpu) = new;
-	__vcpu_write_spsr(vcpu, old);
+	__vcpu_write_spsr(vcpu, target_mode, old);
 }
 
 /*

arch/arm64/kvm/hyp/exception.c static void kvm_inject_exception(struct kvm_vcpu *vcpu) · line 343

Quoted 6 lines
arch/arm64/kvm/hyp/exception.c +3 -0
@@ -320,11 +338,22 @@ static void kvm_inject_exception(struct kvm_vcpu *vcpu)
 		      KVM_ARM64_EXCEPT_AA64_EL1):
 			enter_exception64(vcpu, PSR_MODE_EL1h, except_type_sync);
 			break;
+
+		case (KVM_ARM64_EXCEPT_AA64_ELx_SYNC |
+		      KVM_ARM64_EXCEPT_AA64_EL2):

Too many parens...

arch/arm64/kvm/hyp/exception.c static void kvm_inject_exception(struct kvm_vcpu *vcpu) · line 348

Quoted 5 lines
arch/arm64/kvm/hyp/exception.c +5 -0
+			enter_exception64(vcpu, PSR_MODE_EL2h, except_type_sync);
+			break;
+
+		case (KVM_ARM64_EXCEPT_AA64_ELx_IRQ |
+		      KVM_ARM64_EXCEPT_AA64_EL2):

Too many parens...

Quoted 5 lines
arch/arm64/kvm/inject_fault.c +0 -0
index b47df73e98d7..5dcf3f8b08b8 100644

arch/arm64/kvm/inject_fault.c line 21

Quoted 10 lines
arch/arm64/kvm/inject_fault.c +5 -0
@@ -12,19 +12,58 @@
 
 #include <linux/kvm_host.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
 #include <asm/esr.h>
 
+static void pend_sync_exception(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_ELx_SYNC	|
+			     KVM_ARM64_PENDING_EXCEPTION);

Too many parens...

Quoted 20 lines
arch/arm64/kvm/inject_fault.c +20 -0
+
+	/* If not nesting, EL1 is the only possible exception target */
+	if (likely(!nested_virt_in_use(vcpu))) {
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		return;
+	}
+
+	/*
+	 * With NV, we need to pick between EL1 and EL2. Note that we
+	 * never deal with a nesting exception here, hence never
+	 * changing context, and the exception itself can be delayed
+	 * until the next entry.
+	 */
+	switch(*vcpu_cpsr(vcpu) & PSR_MODE_MASK) {
+	case PSR_MODE_EL2h:
+	case PSR_MODE_EL2t:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL2;
+		break;
+	case PSR_MODE_EL1h:
+	case PSR_MODE_EL1t:

arch/arm64/kvm/inject_fault.c line 51

Quoted 10 lines
arch/arm64/kvm/inject_fault.c +10 -0
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		break;
+	case PSR_MODE_EL0t:
+		if (vcpu_el2_tge_is_set(vcpu) & HCR_TGE)
+			vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL2;
+		else
+			vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		break;
+	default:
+		BUG();

Is taking out the host really appropriate here? Is this something a rogue guest could trigger?

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
author maintainer
#46 lore raw

Re: [PATCH v5 14/69] KVM: arm64: nv: Support virtual EL2 exceptions

Quoted 28 lines · Russell King (Oracle)
Quoted 12 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor).

[maz: switch to common exception injection framework]

---

...

Quoted 13 lines
arch/arm64/kvm/emulate-nested.c new file +13 -0
+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
+{
+	u64 spsr, elr, mode;
+	bool direct_eret;
+
+	/*
+	 * Going through the whole put/load motions is a waste of time
+	 * if this is a VHE guest hypervisor returning to its own
+	 * userspace, or the hypervisor performing a local exception
+	 * return. No need to save/restore registers, no need to
+	 * switch S2 MMU. Just do the canonical ERET.
+	 */
+	spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2);
Quoted 8 lines

arch/arm64/kvm/emulate-nested.c line 46

Quoted 6 lines
arch/arm64/kvm/emulate-nested.c new file +6 -0
+	mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	direct_eret  = (mode == PSR_MODE_EL0t &&
+			vcpu_el2_e2h_is_set(vcpu) &&
+			vcpu_el2_tge_is_set(vcpu));
+	direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);

There are excessive parens on the RHS of the above two.

I guess this is my personal taste, and this is the kind of cosmetic things that help me reason about the code. Some people use syntax highlighting, I use bracketing. I don't think this really matters in the grand scheme of things.

[...]

Quoted 9 lines
Quoted 9 lines
arch/arm64/kvm/emulate-nested.c new file +9 -0
+/*
+ * Emulate taking an exception to EL2.
+ * See ARM ARM J8.1.2 AArch64.TakeException()
+ */
+static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2,
+			     enum exception_type type)
+{
+	u64 pstate, mode;
+	bool direct_inject;
Quoted 8 lines

arch/arm64/kvm/emulate-nested.c line 107

Quoted 4 lines
arch/arm64/kvm/emulate-nested.c new file +4 -0
+
+	if (!nested_virt_in_use(vcpu)) {
+		kvm_err("Unexpected call to %s for the non-nesting configuration\n",
+				__func__);

Too much indentation. I'm guessing this "unexpected" condition isn't something that can be caused by a rogue guest? If it can, doesn't this need to be rate limited?

If we end-up here, this is very much a hypervisor logic bug.

[...]

Quoted 25 lines
Quoted 25 lines
arch/arm64/kvm/inject_fault.c +25 -0
+
+	/* If not nesting, EL1 is the only possible exception target */
+	if (likely(!nested_virt_in_use(vcpu))) {
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		return;
+	}
+
+	/*
+	 * With NV, we need to pick between EL1 and EL2. Note that we
+	 * never deal with a nesting exception here, hence never
+	 * changing context, and the exception itself can be delayed
+	 * until the next entry.
+	 */
+	switch(*vcpu_cpsr(vcpu) & PSR_MODE_MASK) {
+	case PSR_MODE_EL2h:
+	case PSR_MODE_EL2t:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL2;
+		break;
+	case PSR_MODE_EL1h:
+	case PSR_MODE_EL1t:
+		vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		break;
+	case PSR_MODE_EL0t:
+		if (vcpu_el2_tge_is_set(vcpu) & HCR_TGE)
+			vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL2;
Quoted 8 lines

arch/arm64/kvm/inject_fault.c line 51

Quoted 5 lines
arch/arm64/kvm/inject_fault.c +5 -0
+		else
+			vcpu->arch.flags |= KVM_ARM64_EXCEPT_AA64_EL1;
+		break;
+	default:
+		BUG();

Is taking out the host really appropriate here? Is this something a rogue guest could trigger?

This switch is supposed to cover all the NS exception levels, in either stack configuration. If we suddenly find ourselves with a non-architectural state, we have horribly messed up. And no, a guest shouldn't be able to affect this. If it can, that's even more of a reason to take everything down ASAP.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#47 lore raw

Re: [PATCH v5 14/69] KVM: arm64: nv: Support virtual EL2 exceptions

In reply to Marc Zyngier

Hi Marc,

Quoted 81 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor).

[maz: switch to common exception injection framework]

---

[..]

arch/arm64/kvm/emulate-nested.c new file +60 -0
new file mode 100644
index 000000000000..339e8272b01e
@@ -0,0 +1,176 @@
+/*
+ * Copyright (C) 2016 - Linaro and Columbia University
+ * Author: Jintack Lim <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/kvm.h>
+#include <linux/kvm_host.h>
+
+#include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+
+#include "hyp/include/hyp/adjust_pc.h"
+
+#include "trace.h"
+
+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
+{
+	u64 spsr, elr, mode;
+	bool direct_eret;
+
+	/*
+	 * Going through the whole put/load motions is a waste of time
+	 * if this is a VHE guest hypervisor returning to its own
+	 * userspace, or the hypervisor performing a local exception
+	 * return. No need to save/restore registers, no need to
+	 * switch S2 MMU. Just do the canonical ERET.
+	 */
+	spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2);
+	mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	direct_eret  = (mode == PSR_MODE_EL0t &&
+			vcpu_el2_e2h_is_set(vcpu) &&
+			vcpu_el2_tge_is_set(vcpu));
+	direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
+
+	if (direct_eret) {
+		*vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2);
+		*vcpu_cpsr(vcpu) = spsr;
+		trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr);
+		return;
+	}
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);
+
+	elr = __vcpu_sys_reg(vcpu, ELR_EL2);
+
+	trace_kvm_nested_eret(vcpu, elr, spsr);

arch/arm64/kvm/emulate-nested.c line 70

Quoted 10 lines
arch/arm64/kvm/emulate-nested.c new file +10 -0
+
+	/*
+	 * Note that the current exception level is always the virtual EL2,
+	 * since we set HCR_EL2.NV bit only when entering the virtual EL2.
+	 */
+	*vcpu_pc(vcpu) = elr;
+	*vcpu_cpsr(vcpu) = spsr;
+
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();

According to ARM DDI 0487G.a, page D13-3289, ERET'ing to EL1 when HCR_EL2.TGE is set is an illegal exception return. I don't see this case treated here.

Thanks,
Alex

author maintainer
#48 lore raw

Re: [PATCH v5 14/69] KVM: arm64: nv: Support virtual EL2 exceptions

Quoted 43 lines · Alexandru Elisei

Hi Marc,

Quoted 39 lines · Marc Zyngier
arch/arm64/kvm/emulate-nested.c new file +39 -0
+void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
+{
+	u64 spsr, elr, mode;
+	bool direct_eret;
+
+	/*
+	 * Going through the whole put/load motions is a waste of time
+	 * if this is a VHE guest hypervisor returning to its own
+	 * userspace, or the hypervisor performing a local exception
+	 * return. No need to save/restore registers, no need to
+	 * switch S2 MMU. Just do the canonical ERET.
+	 */
+	spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2);
+	mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	direct_eret  = (mode == PSR_MODE_EL0t &&
+			vcpu_el2_e2h_is_set(vcpu) &&
+			vcpu_el2_tge_is_set(vcpu));
+	direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t);
+
+	if (direct_eret) {
+		*vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2);
+		*vcpu_cpsr(vcpu) = spsr;
+		trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr);
+		return;
+	}
+
+	preempt_disable();
+	kvm_arch_vcpu_put(vcpu);
+
+	elr = __vcpu_sys_reg(vcpu, ELR_EL2);
+
+	trace_kvm_nested_eret(vcpu, elr, spsr);
+
+	/*
+	 * Note that the current exception level is always the virtual EL2,
+	 * since we set HCR_EL2.NV bit only when entering the virtual EL2.
+	 */
+	*vcpu_pc(vcpu) = elr;
Quoted 8 lines

arch/arm64/kvm/emulate-nested.c line 70

Quoted 4 lines
arch/arm64/kvm/emulate-nested.c new file +4 -0
+	*vcpu_cpsr(vcpu) = spsr;
+
+	kvm_arch_vcpu_load(vcpu, smp_processor_id());
+	preempt_enable();

According to ARM DDI 0487G.a, page D13-3289, ERET'ing to EL1 when HCR_EL2.TGE is set is an illegal exception return. I don't see this case treated here.

Yes, good call.

I've now added handling for both the return to EL1 with TGE set as well as return to a 32bit mode. The return to EL3 case will directly be handled by the HW, and the return from AArch32 to AArch64 cannot happen by construction.

Thanks for spotting it.

	M.
Signature · 5 lines
Without deviation from the norm, progress is not possible.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#49 lore raw

PATCH v5 15/69 KVM: arm64: nv: Inject HVC exceptions to the virtual EL2

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

As we expect all PSCI calls from the L1 hypervisor to be performed using SMC when nested virtualization is enabled, it is clear that all HVC instruction from the VM (including from the virtual EL2) are supposed to handled in the virtual EL2.

Forward these to EL2 as required.

[maz: add handling of HCR_EL2.HCD]
---
 arch/arm64/kvm/handle_exit.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/arm64/kvm/handle_exit.c +11 -0
index 275a27368a04..1fd1c6dfd6a0 100644
@@ -16,6 +16,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
 #include <asm/debug-monitors.h>
 #include <asm/traps.h>
 
@@ -40,6 +41,16 @@ static int handle_hvc(struct kvm_vcpu *vcpu)
 			    kvm_vcpu_hvc_get_imm(vcpu));
 	vcpu->stat.hvc_exit_stat++;
 
+	/* Forward hvc instructions to the virtual EL2 if the guest has EL2. */
+	if (nested_virt_in_use(vcpu)) {
+		if (vcpu_read_sys_reg(vcpu, HCR_EL2) & HCR_HCD)
+			kvm_inject_undefined(vcpu);
+		else
+			kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu));
+
+		return 1;
+	}
+
 	ret = kvm_hvc_call_handler(vcpu);
 	if (ret < 0) {
 		vcpu_set_reg(vcpu, 0, ~0UL);
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 15/69
#50 lore raw

Re: [PATCH v5 15/69] KVM: arm64: nv: Inject HVC exceptions to the virtual EL2

Quoted 7 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

As we expect all PSCI calls from the L1 hypervisor to be performed using SMC when nested virtualization is enabled, it is clear that all HVC instruction from the VM (including from the virtual EL2) are supposed to handled in the virtual EL2.

Quoted 7 lines

Forward these to EL2 as required.

[maz: add handling of HCR_EL2.HCD]
---
 arch/arm64/kvm/handle_exit.c | 11 +++++++++++
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
Reviewed-by on v5 15/69
#51 lore raw

Re: [PATCH v5 15/69] KVM: arm64: nv: Inject HVC exceptions to the virtual EL2

In reply to Marc Zyngier

Hi Marc,

Quoted 31 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

As we expect all PSCI calls from the L1 hypervisor to be performed using SMC when nested virtualization is enabled, it is clear that all HVC instruction from the VM (including from the virtual EL2) are supposed to handled in the virtual EL2.

Forward these to EL2 as required.

[maz: add handling of HCR_EL2.HCD]
---
 arch/arm64/kvm/handle_exit.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/arm64/kvm/handle_exit.c +1 -0
index 275a27368a04..1fd1c6dfd6a0 100644
@@ -16,6 +16,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
 #include <asm/debug-monitors.h>
 #include <asm/traps.h>
 
@@ -40,6 +41,16 @@ static int handle_hvc(struct kvm_vcpu *vcpu)
 			    kvm_vcpu_hvc_get_imm(vcpu));
 	vcpu->stat.hvc_exit_stat++;

arch/arm64/kvm/handle_exit.c static int handle_hvc(struct kvm_vcpu *vcpu) · line 52

Quoted 10 lines
arch/arm64/kvm/handle_exit.c +9 -0
 
+	/* Forward hvc instructions to the virtual EL2 if the guest has EL2. */
+	if (nested_virt_in_use(vcpu)) {
+		if (vcpu_read_sys_reg(vcpu, HCR_EL2) & HCR_HCD)
+			kvm_inject_undefined(vcpu);
+		else
+			kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu));
+
+		return 1;
+	}

Looks good to me:

Thanks,
Alex

Quoted 7 lines
arch/arm64/kvm/handle_exit.c +1 -0
+
 	ret = kvm_hvc_call_handler(vcpu);
 	if (ret < 0) {
 		vcpu_set_reg(vcpu, 0, ~0UL);
Signature · 1 line
2.30.2
author maintainer
#52 lore raw

PATCH v5 16/69 KVM: arm64: nv: Handle trapped ERET from virtual EL2

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

When a guest hypervisor running virtual EL2 in EL1 executes an ERET instruction, we will have set HCR_EL2.NV which traps ERET to EL2, so that we can emulate the exception return in software.

---
 arch/arm64/include/asm/esr.h     |  5 +++++
 arch/arm64/include/asm/kvm_arm.h |  2 +-
 arch/arm64/kvm/handle_exit.c     | 10 ++++++++++
 3 files changed, 16 insertions(+), 1 deletion(-)
arch/arm64/include/asm/esr.h +5 -0
index d52a0b269ee8..6835e4231119 100644
@@ -257,6 +257,11 @@
 		(((e) & ESR_ELx_SYS64_ISS_OP2_MASK) >>		\
 		 ESR_ELx_SYS64_ISS_OP2_SHIFT))
 
+/* ISS field definitions for ERET/ERETAA/ERETAB trapping */
+
+#define ESR_ELx_ERET_ISS_ERET_ERETAx	0x2
+#define ESR_ELx_ERET_ISS_ERETA_ERATAB	0x1
+
 /*
  * ISS field definitions for floating-point exception traps
  * (FP_EXC_32/FP_EXC_64).
arch/arm64/include/asm/kvm_arm.h +1 -1
index 589a6b92d741..0a0ee998ec5a 100644
@@ -353,7 +353,7 @@
 	ECN(SP_ALIGN), ECN(FP_EXC32), ECN(FP_EXC64), ECN(SERROR), \
 	ECN(BREAKPT_LOW), ECN(BREAKPT_CUR), ECN(SOFTSTP_LOW), \
 	ECN(SOFTSTP_CUR), ECN(WATCHPT_LOW), ECN(WATCHPT_CUR), \
-	ECN(BKPT32), ECN(VECTOR32), ECN(BRK64)
+	ECN(BKPT32), ECN(VECTOR32), ECN(BRK64), ECN(ERET)
 
 #define CPACR_EL1_FPEN		(3 << 20)
 #define CPACR_EL1_TTA		(1 << 28)
arch/arm64/kvm/handle_exit.c +10 -0
index 1fd1c6dfd6a0..95ae624d6aa8 100644
@@ -169,6 +169,15 @@ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+static int kvm_handle_eret(struct kvm_vcpu *vcpu)
+{
+	if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERET_ERETAx)
+		return kvm_handle_ptrauth(vcpu);
+
+	kvm_emulate_nested_eret(vcpu);
+	return 1;
+}
+
 static exit_handle_fn arm_exit_handlers[] = {
 	[0 ... ESR_ELx_EC_MAX]	= kvm_handle_unknown_ec,
 	[ESR_ELx_EC_WFx]	= kvm_handle_wfx,
@@ -183,6 +192,7 @@ static exit_handle_fn arm_exit_handlers[] = {
 	[ESR_ELx_EC_SMC64]	= handle_smc,
 	[ESR_ELx_EC_SYS64]	= kvm_handle_sys_reg,
 	[ESR_ELx_EC_SVE]	= handle_sve,
+	[ESR_ELx_EC_ERET]	= kvm_handle_eret,
 	[ESR_ELx_EC_IABT_LOW]	= kvm_handle_guest_abort,
 	[ESR_ELx_EC_DABT_LOW]	= kvm_handle_guest_abort,
 	[ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug,
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 16/69
#53 lore raw

Re: [PATCH v5 16/69] KVM: arm64: nv: Handle trapped ERET from virtual EL2

Quoted 8 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

When a guest hypervisor running virtual EL2 in EL1 executes an ERET instruction, we will have set HCR_EL2.NV which traps ERET to EL2, so that we can emulate the exception return in software.

Signature · 6 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 16/69
#54 lore raw

Re: [PATCH v5 16/69] KVM: arm64: nv: Handle trapped ERET from virtual EL2

In reply to Marc Zyngier

Hi Marc,

Quoted 19 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

When a guest hypervisor running virtual EL2 in EL1 executes an ERET instruction, we will have set HCR_EL2.NV which traps ERET to EL2, so that we can emulate the exception return in software.

---
 arch/arm64/include/asm/esr.h     |  5 +++++
 arch/arm64/include/asm/kvm_arm.h |  2 +-
 arch/arm64/kvm/handle_exit.c     | 10 ++++++++++
 3 files changed, 16 insertions(+), 1 deletion(-)
arch/arm64/include/asm/esr.h +0 -0
index d52a0b269ee8..6835e4231119 100644

arch/arm64/include/asm/esr.h line 263

Quoted 7 lines
arch/arm64/include/asm/esr.h +4 -0
@@ -257,6 +257,11 @@
 		(((e) & ESR_ELx_SYS64_ISS_OP2_MASK) >>		\
 		 ESR_ELx_SYS64_ISS_OP2_SHIFT))
 
+/* ISS field definitions for ERET/ERETAA/ERETAB trapping */
+
+#define ESR_ELx_ERET_ISS_ERET_ERETAx	0x2
+#define ESR_ELx_ERET_ISS_ERETA_ERATAB	0x1
                            ^^^^^

Shouldn't that be ERETAA?

Other than that, the patch looks good to me, although I'm unfamiliar with PAuth:

Thanks,
Alex

Quoted 48 lines
arch/arm64/include/asm/esr.h +1 -0
+
 /*
  * ISS field definitions for floating-point exception traps
  * (FP_EXC_32/FP_EXC_64).
arch/arm64/include/asm/kvm_arm.h +1 -1
index 589a6b92d741..0a0ee998ec5a 100644
@@ -353,7 +353,7 @@
 	ECN(SP_ALIGN), ECN(FP_EXC32), ECN(FP_EXC64), ECN(SERROR), \
 	ECN(BREAKPT_LOW), ECN(BREAKPT_CUR), ECN(SOFTSTP_LOW), \
 	ECN(SOFTSTP_CUR), ECN(WATCHPT_LOW), ECN(WATCHPT_CUR), \
-	ECN(BKPT32), ECN(VECTOR32), ECN(BRK64)
+	ECN(BKPT32), ECN(VECTOR32), ECN(BRK64), ECN(ERET)
 
 #define CPACR_EL1_FPEN		(3 << 20)
 #define CPACR_EL1_TTA		(1 << 28)
arch/arm64/kvm/handle_exit.c +10 -0
index 1fd1c6dfd6a0..95ae624d6aa8 100644
@@ -169,6 +169,15 @@ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+static int kvm_handle_eret(struct kvm_vcpu *vcpu)
+{
+	if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERET_ERETAx)
+		return kvm_handle_ptrauth(vcpu);
+
+	kvm_emulate_nested_eret(vcpu);
+	return 1;
+}
+
 static exit_handle_fn arm_exit_handlers[] = {
 	[0 ... ESR_ELx_EC_MAX]	= kvm_handle_unknown_ec,
 	[ESR_ELx_EC_WFx]	= kvm_handle_wfx,
@@ -183,6 +192,7 @@ static exit_handle_fn arm_exit_handlers[] = {
 	[ESR_ELx_EC_SMC64]	= handle_smc,
 	[ESR_ELx_EC_SYS64]	= kvm_handle_sys_reg,
 	[ESR_ELx_EC_SVE]	= handle_sve,
+	[ESR_ELx_EC_ERET]	= kvm_handle_eret,
 	[ESR_ELx_EC_IABT_LOW]	= kvm_handle_guest_abort,
 	[ESR_ELx_EC_DABT_LOW]	= kvm_handle_guest_abort,
 	[ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug,
Signature · 1 line
2.30.2
author maintainer
#55 lore raw

Re: [PATCH v5 16/69] KVM: arm64: nv: Handle trapped ERET from virtual EL2

Quoted 26 lines · Alexandru Elisei

Hi Marc,

Quoted 22 lines · Marc Zyngier

From: Christoffer Dall <[email protected]>

When a guest hypervisor running virtual EL2 in EL1 executes an ERET instruction, we will have set HCR_EL2.NV which traps ERET to EL2, so that we can emulate the exception return in software.

---
 arch/arm64/include/asm/esr.h     |  5 +++++
 arch/arm64/include/asm/kvm_arm.h |  2 +-
 arch/arm64/kvm/handle_exit.c     | 10 ++++++++++
 3 files changed, 16 insertions(+), 1 deletion(-)
arch/arm64/include/asm/esr.h +0 -0
index d52a0b269ee8..6835e4231119 100644
@@ -257,6 +257,11 @@
 		(((e) & ESR_ELx_SYS64_ISS_OP2_MASK) >>		\
 		 ESR_ELx_SYS64_ISS_OP2_SHIFT))
 
Quoted 6 lines

arch/arm64/include/asm/esr.h line 263

Quoted 4 lines
arch/arm64/include/asm/esr.h +4 -0
+/* ISS field definitions for ERET/ERETAA/ERETAB trapping */
+
+#define ESR_ELx_ERET_ISS_ERET_ERETAx	0x2
+#define ESR_ELx_ERET_ISS_ERETA_ERATAB	0x1
                            ^^^^^

Shouldn't that be ERETAA?

Hmm, this is rather ambiguous indeed. I'm going to repaint it as:

#define ESR_ELx_ERET_ISS_ERET 0x2
#define ESR_ELx_ERET_ISS_ERETA 0x1

which matches the ARM ARM wording as of DDI0487G.b.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
author maintainer
#56 lore raw

PATCH v5 17/69 KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers

In reply to Marc Zyngier

Some EL2 system registers immediately affect the current execution of the system, so we need to use their respective EL1 counterparts. For this we need to define a mapping between the two. In general, this only affects non-VHE guest hypervisors, as VHE system registers are compatible with the EL1 counterparts.

These helpers will get used in subsequent patches.

---
 arch/arm64/include/asm/kvm_nested.h | 50 +++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
arch/arm64/include/asm/kvm_nested.h +50 -0
index 1028ac65a897..67a2c0d05233 100644
@@ -2,6 +2,7 @@
 #ifndef __ARM64_KVM_NESTED_H
 #define __ARM64_KVM_NESTED_H
 
+#include <linux/bitfield.h>
 #include <linux/kvm_host.h>
 
 static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu)
@@ -11,4 +12,53 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu)
 		test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));
 }
 
+/* Translation helpers from non-VHE EL2 to EL1 */
+static inline u64 tcr_el2_ips_to_tcr_el1_ps(u64 tcr_el2)
+{
+	return (u64)FIELD_GET(TCR_EL2_PS_MASK, tcr_el2) << TCR_IPS_SHIFT;
+}
+
+static inline u64 translate_tcr_el2_to_tcr_el1(u64 tcr)
+{
+	return TCR_EPD1_MASK |				/* disable TTBR1_EL1 */
+	       ((tcr & TCR_EL2_TBI) ? TCR_TBI0 : 0) |
+	       tcr_el2_ips_to_tcr_el1_ps(tcr) |
+	       (tcr & TCR_EL2_TG0_MASK) |
+	       (tcr & TCR_EL2_ORGN0_MASK) |
+	       (tcr & TCR_EL2_IRGN0_MASK) |
+	       (tcr & TCR_EL2_T0SZ_MASK);
+}
+
+static inline u64 translate_cptr_el2_to_cpacr_el1(u64 cptr_el2)
+{
+	u64 cpacr_el1 = 0;
+
+	if (!(cptr_el2 & CPTR_EL2_TFP))
+		cpacr_el1 |= CPACR_EL1_FPEN;
+	if (cptr_el2 & CPTR_EL2_TTA)
+		cpacr_el1 |= CPACR_EL1_TTA;
+	if (!(cptr_el2 & CPTR_EL2_TZ))
+		cpacr_el1 |= CPACR_EL1_ZEN;
+
+	return cpacr_el1;
+}
+
+static inline u64 translate_sctlr_el2_to_sctlr_el1(u64 sctlr)
+{
+	/* Bit 20 is RES1 in SCTLR_EL1, but RES0 in SCTLR_EL2 */
+	return sctlr | BIT(20);
+}
+
+static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0)
+{
+	/* Force ASID to 0 (ASID 0 or RES0) */
+	return ttbr0 & ~GENMASK_ULL(63, 48);
+}
+
+static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl)
+{
+	return ((FIELD_GET(CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN, cnthctl) << 10) |
+		(cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN)));
+}
+
 #endif /* __ARM64_KVM_NESTED_H */
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#57 lore raw

Re: [PATCH v5 17/69] KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers

Hi Marc,

Quoted 28 lines · Marc Zyngier

Some EL2 system registers immediately affect the current execution of the system, so we need to use their respective EL1 counterparts. For this we need to define a mapping between the two. In general, this only affects non-VHE guest hypervisors, as VHE system registers are compatible with the EL1 counterparts.

These helpers will get used in subsequent patches.

---
 arch/arm64/include/asm/kvm_nested.h | 50 +++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
arch/arm64/include/asm/kvm_nested.h +1 -0
index 1028ac65a897..67a2c0d05233 100644
@@ -2,6 +2,7 @@
 #ifndef __ARM64_KVM_NESTED_H
 #define __ARM64_KVM_NESTED_H
 
+#include <linux/bitfield.h>
 #include <linux/kvm_host.h>
 
 static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu)

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 16

Quoted 5 lines
arch/arm64/include/asm/kvm_nested.h +2 -0
@@ -11,4 +12,53 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu)
 		test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));
 }
 
+/* Translation helpers from non-VHE EL2 to EL1 */
+static inline u64 tcr_el2_ips_to_tcr_el1_ps(u64 tcr_el2)

When E2H = 0, there is no IPS field in TCR_EL2, but there is a PS field. And for TCR_EL1, there is no PS field, but there is an IPS field. Maybe tcr_el2_ps_to_tcr_el1_ips() would be more precise, and would also match the field defines used by the function?

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 29

Quoted 13 lines
arch/arm64/include/asm/kvm_nested.h +13 -0
+{
+	return (u64)FIELD_GET(TCR_EL2_PS_MASK, tcr_el2) << TCR_IPS_SHIFT;
+}
+
+static inline u64 translate_tcr_el2_to_tcr_el1(u64 tcr)
+{
+	return TCR_EPD1_MASK |				/* disable TTBR1_EL1 */
+	       ((tcr & TCR_EL2_TBI) ? TCR_TBI0 : 0) |
+	       tcr_el2_ips_to_tcr_el1_ps(tcr) |
+	       (tcr & TCR_EL2_TG0_MASK) |
+	       (tcr & TCR_EL2_ORGN0_MASK) |
+	       (tcr & TCR_EL2_IRGN0_MASK) |
+	       (tcr & TCR_EL2_T0SZ_MASK);

There are a few fields in TCR_EL2 which have a corresponding field in TCR_EL1, when E2H = 0: HPD -> HPD0 (hierarchical permissions toggle), HA and HD (hardware management of dirty bit and access flag), DS (when FEAT_LPA2), and probably others. Why do we not also translate them? Is it because we hide the feature they depend on (FEAT_HPDS, FEAT_HAFBDS, etc) in the guest ID registers? Is it something else?

Quoted 4 lines
arch/arm64/include/asm/kvm_nested.h +4 -0
+}
+
+static inline u64 translate_cptr_el2_to_cpacr_el1(u64 cptr_el2)
+{

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 43

Quoted 10 lines
arch/arm64/include/asm/kvm_nested.h +10 -0
+	u64 cpacr_el1 = 0;
+
+	if (!(cptr_el2 & CPTR_EL2_TFP))
+		cpacr_el1 |= CPACR_EL1_FPEN;
+	if (cptr_el2 & CPTR_EL2_TTA)
+		cpacr_el1 |= CPACR_EL1_TTA;
+	if (!(cptr_el2 & CPTR_EL2_TZ))
+		cpacr_el1 |= CPACR_EL1_ZEN;
+
+	return cpacr_el1;

Nitpick: it would make comparing against the architecture easier if the fields were checked in the order they were definied in the architecture. So first check the TTA bit, then TFP and lastly TZ.

I checked the field definitions for CPTR_EL2 and the above looks correct to me, as TFP, TTA and TZ were the only fields which affect EL2; I also checked that the values in CPACR_EL1 are set correctly to mirror the CPTR_EL2 settings.

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 49

Quoted 6 lines
arch/arm64/include/asm/kvm_nested.h +6 -0
+}
+
+static inline u64 translate_sctlr_el2_to_sctlr_el1(u64 sctlr)
+{
+	/* Bit 20 is RES1 in SCTLR_EL1, but RES0 in SCTLR_EL2 */
+	return sctlr | BIT(20);

Bits 8 and 7 in SCTLR_EL2 are RES0 when E2H,TGE != {1,1}, but they are RES1 in SCTLR_EL1 if EL0 is not capable of using AArch32. Shouldn't we also set them?

Bit 5 in SCTLR_EL2 is RES1 when E2H,TGE != {1,1}, but it is RES0 in SCTLR_EL1 if EL0 is not capable of using AArch32. Shouldn't we clear it?

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 54

Quoted 5 lines
arch/arm64/include/asm/kvm_nested.h +5 -0
+}
+
+static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0)
+{
+	/* Force ASID to 0 (ASID 0 or RES0) */

That got me confused at first, until I realized that the first ASID refers to the ASID field of the register, and the second ASID to the translation table property. Might be more helpful if the comment was simply "Clear the ASID field" or something like that.

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 60

Quoted 6 lines
arch/arm64/include/asm/kvm_nested.h +6 -0
+	return ttbr0 & ~GENMASK_ULL(63, 48);
+}
+
+static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl)
+{
+	return ((FIELD_GET(CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN, cnthctl) << 10) |

I don't understand why those two bits are left shifted by 10, the result is 0x3 << 10 and CNTKCTL_EL[16:10] is RES0.

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 61

Quoted 1 line
arch/arm64/include/asm/kvm_nested.h +1 -0
+		(cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN)));

CNTKCTL_EL1.{EVNTI,EVNTDIR,EVNTEN} refer to CNT*V*CT_EL0, CNTHCTL_EL2.{EVNTI,EVNTDIR,EVNTEN} refer to CNT*P*CT_EL0. I don't understand why they are treated as equivalent.

I get the feeling I'm misunderstanding something about this function.

Thanks,
Alex

Quoted 6 lines
arch/arm64/include/asm/kvm_nested.h +2 -0
+}
+
 #endif /* __ARM64_KVM_NESTED_H */
Signature · 1 line
2.30.2
author maintainer
#58 lore raw

Re: [PATCH v5 17/69] KVM: arm64: nv: Add non-VHE-EL2->EL1 translation helpers

Quoted 35 lines · Alexandru Elisei

Hi Marc,

Quoted 31 lines · Marc Zyngier

Some EL2 system registers immediately affect the current execution of the system, so we need to use their respective EL1 counterparts. For this we need to define a mapping between the two. In general, this only affects non-VHE guest hypervisors, as VHE system registers are compatible with the EL1 counterparts.

These helpers will get used in subsequent patches.

---
 arch/arm64/include/asm/kvm_nested.h | 50 +++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
arch/arm64/include/asm/kvm_nested.h +1 -0
index 1028ac65a897..67a2c0d05233 100644
@@ -2,6 +2,7 @@
 #ifndef __ARM64_KVM_NESTED_H
 #define __ARM64_KVM_NESTED_H
 
+#include <linux/bitfield.h>
 #include <linux/kvm_host.h>
 
 static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu)
@@ -11,4 +12,53 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu)
 		test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features));
 }
 
Quoted 7 lines

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 16

Quoted 2 lines
arch/arm64/include/asm/kvm_nested.h +2 -0
+/* Translation helpers from non-VHE EL2 to EL1 */
+static inline u64 tcr_el2_ips_to_tcr_el1_ps(u64 tcr_el2)

When E2H = 0, there is no IPS field in TCR_EL2, but there is a PS field. And for TCR_EL1, there is no PS field, but there is an IPS field. Maybe tcr_el2_ps_to_tcr_el1_ips() would be more precise, and would also match the field defines used by the function?

Yes, good point. Done.

Quoted 13 lines
Quoted 13 lines
arch/arm64/include/asm/kvm_nested.h +12 -0
+{
+	return (u64)FIELD_GET(TCR_EL2_PS_MASK, tcr_el2) << TCR_IPS_SHIFT;
+}
+
+static inline u64 translate_tcr_el2_to_tcr_el1(u64 tcr)
+{
+	return TCR_EPD1_MASK |				/* disable TTBR1_EL1 */
+	       ((tcr & TCR_EL2_TBI) ? TCR_TBI0 : 0) |
+	       tcr_el2_ips_to_tcr_el1_ps(tcr) |
+	       (tcr & TCR_EL2_TG0_MASK) |
+	       (tcr & TCR_EL2_ORGN0_MASK) |
+	       (tcr & TCR_EL2_IRGN0_MASK) |
Quoted 8 lines

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 29

Quoted 1 line
arch/arm64/include/asm/kvm_nested.h +1 -0
+	       (tcr & TCR_EL2_T0SZ_MASK);

There are a few fields in TCR_EL2 which have a corresponding field in TCR_EL1, when E2H = 0: HPD -> HPD0 (hierarchical permissions toggle), HA and HD (hardware management of dirty bit and access flag), DS (when FEAT_LPA2), and probably others. Why do we not also translate them? Is it because we hide the feature they depend on (FEAT_HPDS, FEAT_HAFBDS, etc) in the guest ID registers? Is it something else?

We do hide the features, and for some of them they simply cannot be implemented. HA/HD are pretty good example of how the architecture is broken in this respect (the PT update happens in the shadow, while the guest can only look at its own copy).

Quoted 11 lines
Quoted 11 lines
arch/arm64/include/asm/kvm_nested.h +10 -0
+}
+
+static inline u64 translate_cptr_el2_to_cpacr_el1(u64 cptr_el2)
+{
+	u64 cpacr_el1 = 0;
+
+	if (!(cptr_el2 & CPTR_EL2_TFP))
+		cpacr_el1 |= CPACR_EL1_FPEN;
+	if (cptr_el2 & CPTR_EL2_TTA)
+		cpacr_el1 |= CPACR_EL1_TTA;
Quoted 8 lines

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 43

Quoted 4 lines
arch/arm64/include/asm/kvm_nested.h +4 -0
+	if (!(cptr_el2 & CPTR_EL2_TZ))
+		cpacr_el1 |= CPACR_EL1_ZEN;
+
+	return cpacr_el1;

Nitpick: it would make comparing against the architecture easier if the fields were checked in the order they were definied in the architecture. So first check the TTA bit, then TFP and lastly TZ.

You are easy to please! ;-) Done.

Quoted 17 lines

I checked the field definitions for CPTR_EL2 and the above looks correct to me, as TFP, TTA and TZ were the only fields which affect EL2; I also checked that the values in CPACR_EL1 are set correctly to mirror the CPTR_EL2 settings.

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 49

Quoted 6 lines
arch/arm64/include/asm/kvm_nested.h +6 -0
+}
+
+static inline u64 translate_sctlr_el2_to_sctlr_el1(u64 sctlr)
+{
+	/* Bit 20 is RES1 in SCTLR_EL1, but RES0 in SCTLR_EL2 */
+	return sctlr | BIT(20);

Bits 8 and 7 in SCTLR_EL2 are RES0 when E2H,TGE != {1,1}, but they are RES1 in SCTLR_EL1 if EL0 is not capable of using AArch32. Shouldn't we also set them?

Quoted 3 lines

Bit 5 in SCTLR_EL2 is RES1 when E2H,TGE != {1,1}, but it is RES0 in SCTLR_EL1 if EL0 is not capable of using AArch32. Shouldn't we clear it?

I actually reworked all of that as such:

static inline u64 translate_sctlr_el2_to_sctlr_el1(u64 val)

{
	/* Only preserve the minimal set of bits we support */
	val &= (SCTLR_ELx_M | SCTLR_ELx_A | SCTLR_ELx_C | SCTLR_ELx_SA |
		SCTLR_ELx_I | SCTLR_ELx_IESB | SCTLR_ELx_WXN | SCTLR_ELx_EE);
	val |= SCTLR_EL1_RES1;

	return val;
}
Quoted 10 lines

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 54

Quoted 5 lines
arch/arm64/include/asm/kvm_nested.h +5 -0
+}
+
+static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0)
+{
+	/* Force ASID to 0 (ASID 0 or RES0) */

That got me confused at first, until I realized that the first ASID refers to the ASID field of the register, and the second ASID to the translation table property. Might be more helpful if the comment was simply "Clear the ASID field" or something like that.

Done.

Quoted 11 lines

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 60

Quoted 7 lines
arch/arm64/include/asm/kvm_nested.h +6 -0
+	return ttbr0 & ~GENMASK_ULL(63, 48);
+}
+
+static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl)
+{
+	return ((FIELD_GET(CNTHCTL_EL1PCTEN | CNTHCTL_EL1PCEN, cnthctl) << 10) |

I don't understand why those two bits are left shifted by 10, the result is 0x3 << 10 and CNTKCTL_EL[16:10] is RES0.

Quoted 8 lines

arch/arm64/include/asm/kvm_nested.h static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) · line 61

Quoted 1 line
arch/arm64/include/asm/kvm_nested.h +1 -0
+		(cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN)));

CNTKCTL_EL1.{EVNTI,EVNTDIR,EVNTEN} refer to CNT*V*CT_EL0, CNTHCTL_EL2.{EVNTI,EVNTDIR,EVNTEN} refer to CNT*P*CT_EL0. I don't understand why they are treated as equivalent.

I get the feeling I'm misunderstanding something about this function.

It's a classic one. Remember that we are running VHE, and remapping a nVHE view of CNTHCTL_EL2 into the VHE view *for the guest*, and that these things are completely shifted around (it has the CNTKCTL_EL1 format).

For example, on nVHE, CNTHCTL_EL2.EL1PCTEN is bit 0. On nVHE, this is bit 10. That's why we have this shift, and that you now need some paracetamol.

You can also look at the way we deal with the same stuff in kvm_timer_init_vhe().

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
author maintainer
#59 lore raw

PATCH v5 18/69 KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg()

In reply to Marc Zyngier

KVM internally uses accessor functions when reading or writing the guest's system registers. This takes care of accessing either the stored copy or using the "live" EL1 system registers when the host uses VHE.

With the introduction of virtual EL2 we add a bunch of EL2 system registers, which now must also be taken care of:
- If the guest is running in vEL2, and we access an EL1 sysreg, we must revert to the stored version of that, and not use the CPU's copy.
- If the guest is running in vEL1, and we access an EL2 sysreg, we must also use the stored version, since the CPU carries the EL1 copy.
- Some EL2 system registers are supposed to affect the current execution of the system, so we need to put them into their respective EL1 counterparts. For this we need to define a mapping between the two. This is done using the newly introduced struct el2_sysreg_map.
- Some EL2 system registers have a different format than their EL1 counterpart, so we need to translate them before writing them to the CPU. This is done using an (optional) translate function in the map.
- There are the three special registers SP_EL2, SPSR_EL2 and ELR_EL2, which need some separate handling (SPSR_EL2 is being handled in a separate patch).

All of these cases are now wrapped into the existing accessor functions, so KVM users wouldn't need to care whether they access EL2 or EL1 registers and also which state the guest is in.

This handles what was formerly known as the "shadow state" dynamically, without requiring a separate copy for each vCPU EL.

---
 arch/arm64/kvm/sys_regs.c | 143 ++++++++++++++++++++++++++++++++++++--
 1 file changed, 139 insertions(+), 4 deletions(-)
arch/arm64/kvm/sys_regs.c +139 -4
index 730a24468915..61596355e42d 100644
@@ -24,6 +24,7 @@
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_hyp.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
 #include <asm/perf_event.h>
 #include <asm/sysreg.h>
 
@@ -64,23 +65,157 @@ static bool write_to_read_only(struct kvm_vcpu *vcpu,
 	return false;
 }
 
+#define PURE_EL2_SYSREG(el2)						\
+	case el2: {							\
+		*el1r = el2;						\
+		return true;						\
+	}
+
+#define MAPPED_EL2_SYSREG(el2, el1, fn)					\
+	case el2: {							\
+		*xlate = fn;						\
+		*el1r = el1;						\
+		return true;						\
+	}
+
+static bool get_el2_mapping(unsigned int reg,
+			    unsigned int *el1r, u64 (**xlate)(u64))
+{
+	switch (reg) {
+		PURE_EL2_SYSREG(  VPIDR_EL2	);
+		PURE_EL2_SYSREG(  VMPIDR_EL2	);
+		PURE_EL2_SYSREG(  ACTLR_EL2	);
+		PURE_EL2_SYSREG(  HCR_EL2	);
+		PURE_EL2_SYSREG(  MDCR_EL2	);
+		PURE_EL2_SYSREG(  HSTR_EL2	);
+		PURE_EL2_SYSREG(  HACR_EL2	);
+		PURE_EL2_SYSREG(  VTTBR_EL2	);
+		PURE_EL2_SYSREG(  VTCR_EL2	);
+		PURE_EL2_SYSREG(  RVBAR_EL2	);
+		PURE_EL2_SYSREG(  TPIDR_EL2	);
+		PURE_EL2_SYSREG(  HPFAR_EL2	);
+		PURE_EL2_SYSREG(  ELR_EL2	);
+		PURE_EL2_SYSREG(  SPSR_EL2	);
+		MAPPED_EL2_SYSREG(SCTLR_EL2,   SCTLR_EL1,
+				  translate_sctlr_el2_to_sctlr_el1	     );
+		MAPPED_EL2_SYSREG(CPTR_EL2,    CPACR_EL1,
+				  translate_cptr_el2_to_cpacr_el1	     );
+		MAPPED_EL2_SYSREG(TTBR0_EL2,   TTBR0_EL1,
+				  translate_ttbr0_el2_to_ttbr0_el1	     );
+		MAPPED_EL2_SYSREG(TTBR1_EL2,   TTBR1_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(TCR_EL2,     TCR_EL1,
+				  translate_tcr_el2_to_tcr_el1		     );
+		MAPPED_EL2_SYSREG(VBAR_EL2,    VBAR_EL1,    NULL	     );
+		MAPPED_EL2_SYSREG(AFSR0_EL2,   AFSR0_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(AFSR1_EL2,   AFSR1_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(ESR_EL2,     ESR_EL1,     NULL	     );
+		MAPPED_EL2_SYSREG(FAR_EL2,     FAR_EL1,     NULL	     );
+		MAPPED_EL2_SYSREG(MAIR_EL2,    MAIR_EL1,    NULL	     );
+		MAPPED_EL2_SYSREG(AMAIR_EL2,   AMAIR_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(CNTHCTL_EL2, CNTKCTL_EL1,
+				  translate_cnthctl_el2_to_cntkctl_el1	     );
+	default:
+		return false;
+	}
+}
+
 u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
 {
 	u64 val = 0x8badf00d8badf00d;
+	u64 (*xlate)(u64) = NULL;
+	unsigned int el1r;
+
+	if (!vcpu->arch.sysregs_loaded_on_cpu)
+		goto memory_read;
+
+	if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) {
+		if (!is_hyp_ctxt(vcpu))
+			goto memory_read;
+
+		/*
+		 * ELR_EL2 is special cased for now.
+		 */
+		switch (reg) {
+		case ELR_EL2:
+			return read_sysreg_el1(SYS_ELR);
+		}
+
+		/*
+		 * If this register does not have an EL1 counterpart,
+		 * then read the stored EL2 version.
+		 */
+		if (reg == el1r)
+			goto memory_read;
+
+		/*
+		 * If we have a non-VHE guest and that the sysreg
+		 * requires translation to be used at EL1, use the
+		 * in-memory copy instead.
+		 */
+		if (!vcpu_el2_e2h_is_set(vcpu) && xlate)
+			goto memory_read;
+
+		/* Get the current version of the EL1 counterpart. */
+		WARN_ON(!__vcpu_read_sys_reg_from_cpu(el1r, &val));
+		return val;
+	}
+
+	/* EL1 register can't be on the CPU if the guest is in vEL2. */
+	if (unlikely(is_hyp_ctxt(vcpu)))
+		goto memory_read;
 
-	if (vcpu->arch.sysregs_loaded_on_cpu &&
-	    __vcpu_read_sys_reg_from_cpu(reg, &val))
+	if (__vcpu_read_sys_reg_from_cpu(reg, &val))
 		return val;
 
+memory_read:
 	return __vcpu_sys_reg(vcpu, reg);
 }
 
 void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
 {
-	if (vcpu->arch.sysregs_loaded_on_cpu &&
-	    __vcpu_write_sys_reg_to_cpu(val, reg))
+	u64 (*xlate)(u64) = NULL;
+	unsigned int el1r;
+
+	if (!vcpu->arch.sysregs_loaded_on_cpu)
+		goto memory_write;
+
+	if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) {
+		if (!is_hyp_ctxt(vcpu))
+			goto memory_write;
+
+		/*
+		 * Always store a copy of the write to memory to avoid having
+		 * to reverse-translate virtual EL2 system registers for a
+		 * non-VHE guest hypervisor.
+		 */
+		__vcpu_sys_reg(vcpu, reg) = val;
+
+		switch (reg) {
+		case ELR_EL2:
+			write_sysreg_el1(val, SYS_ELR);
+			return;
+		}
+
+		/* No EL1 counterpart? We're done here.? */
+		if (reg == el1r)
+			return;
+
+		if (!vcpu_el2_e2h_is_set(vcpu) && xlate)
+			val = xlate(val);
+
+		/* Redirect this to the EL1 version of the register. */
+		WARN_ON(!__vcpu_write_sys_reg_to_cpu(val, el1r));
+		return;
+	}
+
+	/* EL1 register can't be on the CPU if the guest is in vEL2. */
+	if (unlikely(is_hyp_ctxt(vcpu)))
+		goto memory_write;
+
+	if (__vcpu_write_sys_reg_to_cpu(val, reg))
 		return;
 
+memory_write:
 	 __vcpu_sys_reg(vcpu, reg) = val;
 }
 
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#60 lore raw

Re: [PATCH v5 18/69] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg()

Quoted 9 lines · Marc Zyngier

KVM internally uses accessor functions when reading or writing the guest's system registers. This takes care of accessing either the stored copy or using the "live" EL1 system registers when the host uses VHE.

With the introduction of virtual EL2 we add a bunch of EL2 system registers, which now must also be taken care of:
- If the guest is running in vEL2, and we access an EL1 sysreg, we must revert to the stored version of that, and not use the CPU's copy.
- If the guest is running in vEL1, and we access an EL2 sysreg, we must

Do we have vEL1? or is it a typo?

Quoted 190 lines

also use the stored version, since the CPU carries the EL1 copy.
- Some EL2 system registers are supposed to affect the current execution of the system, so we need to put them into their respective EL1 counterparts. For this we need to define a mapping between the two. This is done using the newly introduced struct el2_sysreg_map.
- Some EL2 system registers have a different format than their EL1 counterpart, so we need to translate them before writing them to the CPU. This is done using an (optional) translate function in the map.
- There are the three special registers SP_EL2, SPSR_EL2 and ELR_EL2, which need some separate handling (SPSR_EL2 is being handled in a separate patch).

All of these cases are now wrapped into the existing accessor functions, so KVM users wouldn't need to care whether they access EL2 or EL1 registers and also which state the guest is in.

This handles what was formerly known as the "shadow state" dynamically, without requiring a separate copy for each vCPU EL.

---
  arch/arm64/kvm/sys_regs.c | 143 ++++++++++++++++++++++++++++++++++++--
  1 file changed, 139 insertions(+), 4 deletions(-)
arch/arm64/kvm/sys_regs.c +134 -4
index 730a24468915..61596355e42d 100644
@@ -24,6 +24,7 @@
  #include <asm/kvm_emulate.h>
  #include <asm/kvm_hyp.h>
  #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
  #include <asm/perf_event.h>
  #include <asm/sysreg.h>
 
@@ -64,23 +65,157 @@ static bool write_to_read_only(struct kvm_vcpu *vcpu,
  	return false;
  }
 
+#define PURE_EL2_SYSREG(el2)						\
+	case el2: {							\
+		*el1r = el2;						\
+		return true;						\
+	}
+
+#define MAPPED_EL2_SYSREG(el2, el1, fn)					\
+	case el2: {							\
+		*xlate = fn;						\
+		*el1r = el1;						\
+		return true;						\
+	}
+
+static bool get_el2_mapping(unsigned int reg,
+			    unsigned int *el1r, u64 (**xlate)(u64))
+{
+	switch (reg) {
+		PURE_EL2_SYSREG(  VPIDR_EL2	);
+		PURE_EL2_SYSREG(  VMPIDR_EL2	);
+		PURE_EL2_SYSREG(  ACTLR_EL2	);
+		PURE_EL2_SYSREG(  HCR_EL2	);
+		PURE_EL2_SYSREG(  MDCR_EL2	);
+		PURE_EL2_SYSREG(  HSTR_EL2	);
+		PURE_EL2_SYSREG(  HACR_EL2	);
+		PURE_EL2_SYSREG(  VTTBR_EL2	);
+		PURE_EL2_SYSREG(  VTCR_EL2	);
+		PURE_EL2_SYSREG(  RVBAR_EL2	);
+		PURE_EL2_SYSREG(  TPIDR_EL2	);
+		PURE_EL2_SYSREG(  HPFAR_EL2	);
+		PURE_EL2_SYSREG(  ELR_EL2	);
+		PURE_EL2_SYSREG(  SPSR_EL2	);
+		MAPPED_EL2_SYSREG(SCTLR_EL2,   SCTLR_EL1,
+				  translate_sctlr_el2_to_sctlr_el1	     );
+		MAPPED_EL2_SYSREG(CPTR_EL2,    CPACR_EL1,
+				  translate_cptr_el2_to_cpacr_el1	     );
+		MAPPED_EL2_SYSREG(TTBR0_EL2,   TTBR0_EL1,
+				  translate_ttbr0_el2_to_ttbr0_el1	     );
+		MAPPED_EL2_SYSREG(TTBR1_EL2,   TTBR1_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(TCR_EL2,     TCR_EL1,
+				  translate_tcr_el2_to_tcr_el1		     );
+		MAPPED_EL2_SYSREG(VBAR_EL2,    VBAR_EL1,    NULL	     );
+		MAPPED_EL2_SYSREG(AFSR0_EL2,   AFSR0_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(AFSR1_EL2,   AFSR1_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(ESR_EL2,     ESR_EL1,     NULL	     );
+		MAPPED_EL2_SYSREG(FAR_EL2,     FAR_EL1,     NULL	     );
+		MAPPED_EL2_SYSREG(MAIR_EL2,    MAIR_EL1,    NULL	     );
+		MAPPED_EL2_SYSREG(AMAIR_EL2,   AMAIR_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(CNTHCTL_EL2, CNTKCTL_EL1,
+				  translate_cnthctl_el2_to_cntkctl_el1	     );
+	default:
+		return false;
+	}
+}
+
  u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
  {
  	u64 val = 0x8badf00d8badf00d;
+	u64 (*xlate)(u64) = NULL;
+	unsigned int el1r;
+
+	if (!vcpu->arch.sysregs_loaded_on_cpu)
+		goto memory_read;
+
+	if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) {
+		if (!is_hyp_ctxt(vcpu))
+			goto memory_read;
+
+		/*
+		 * ELR_EL2 is special cased for now.
+		 */
+		switch (reg) {
+		case ELR_EL2:
+			return read_sysreg_el1(SYS_ELR);
+		}
+
+		/*
+		 * If this register does not have an EL1 counterpart,
+		 * then read the stored EL2 version.
+		 */
+		if (reg == el1r)
+			goto memory_read;
+
+		/*
+		 * If we have a non-VHE guest and that the sysreg
+		 * requires translation to be used at EL1, use the
+		 * in-memory copy instead.
+		 */
+		if (!vcpu_el2_e2h_is_set(vcpu) && xlate)
+			goto memory_read;
+
+		/* Get the current version of the EL1 counterpart. */
+		WARN_ON(!__vcpu_read_sys_reg_from_cpu(el1r, &val));
+		return val;
+	}
+
+	/* EL1 register can't be on the CPU if the guest is in vEL2. */
+	if (unlikely(is_hyp_ctxt(vcpu)))
+		goto memory_read;
 
-	if (vcpu->arch.sysregs_loaded_on_cpu &&
-	    __vcpu_read_sys_reg_from_cpu(reg, &val))
+	if (__vcpu_read_sys_reg_from_cpu(reg, &val))
  		return val;
 
+memory_read:
  	return __vcpu_sys_reg(vcpu, reg);
  }
 
  void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
  {
-	if (vcpu->arch.sysregs_loaded_on_cpu &&
-	    __vcpu_write_sys_reg_to_cpu(val, reg))
+	u64 (*xlate)(u64) = NULL;
+	unsigned int el1r;
+
+	if (!vcpu->arch.sysregs_loaded_on_cpu)
+		goto memory_write;
+
+	if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) {
+		if (!is_hyp_ctxt(vcpu))
+			goto memory_write;
+
+		/*
+		 * Always store a copy of the write to memory to avoid having
+		 * to reverse-translate virtual EL2 system registers for a
+		 * non-VHE guest hypervisor.
+		 */
+		__vcpu_sys_reg(vcpu, reg) = val;
+
+		switch (reg) {
+		case ELR_EL2:
+			write_sysreg_el1(val, SYS_ELR);
+			return;
+		}
+
+		/* No EL1 counterpart? We're done here.? */
+		if (reg == el1r)
+			return;
+
+		if (!vcpu_el2_e2h_is_set(vcpu) && xlate)
+			val = xlate(val);
+
+		/* Redirect this to the EL1 version of the register. */
+		WARN_ON(!__vcpu_write_sys_reg_to_cpu(val, el1r));
+		return;
+	}
+
+	/* EL1 register can't be on the CPU if the guest is in vEL2. */

arch/arm64/kvm/sys_regs.c static bool write_to_read_only(struct kvm_vcpu *vcpu, · line 221

Quoted 10 lines
arch/arm64/kvm/sys_regs.c +5 -0
+	if (unlikely(is_hyp_ctxt(vcpu)))
+		goto memory_write;
+
+	if (__vcpu_write_sys_reg_to_cpu(val, reg))
  		return;
 
+memory_write:
  	 __vcpu_sys_reg(vcpu, reg) = val;
  }
 

Thanks,
Ganapat

author maintainer
#61 lore raw

Re: [PATCH v5 18/69] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg()

Quoted 7 lines · Ganapatrao Kulkarni
Quoted 4 lines · Marc Zyngier

KVM internally uses accessor functions when reading or writing the guest's system registers. This takes care of accessing either the stored copy or using the "live" EL1 system registers when the host uses VHE.

Quoted 7 lines
Quoted 5 lines

With the introduction of virtual EL2 we add a bunch of EL2 system registers, which now must also be taken care of:
- If the guest is running in vEL2, and we access an EL1 sysreg, we must revert to the stored version of that, and not use the CPU's copy.
- If the guest is running in vEL1, and we access an EL2 sysreg, we must

Do we have vEL1? or is it a typo?

Not a typo, but only a convention (there is no such concept in the architecture). vELx denotes the exception level the guest thinks it is running at while running at EL1 (as it is the case for both vEL1 and vEL2).

Depending on the exception level and the running mode (VHE or not) you emulate at any given time, you access the sysregs differently: they can be either live in the CPU, stored in memory, with or without translation. That's why I'm using these 'parallel' exception levels to denote which is which...

HTH,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#62 lore raw

Re: [PATCH v5 18/69] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg()

Quoted 15 lines · Marc Zyngier
Quoted 13 lines · Ganapatrao Kulkarni
Quoted 9 lines · Marc Zyngier

KVM internally uses accessor functions when reading or writing the guest's system registers. This takes care of accessing either the stored copy or using the "live" EL1 system registers when the host uses VHE.

With the introduction of virtual EL2 we add a bunch of EL2 system registers, which now must also be taken care of:
- If the guest is running in vEL2, and we access an EL1 sysreg, we must revert to the stored version of that, and not use the CPU's copy.
- If the guest is running in vEL1, and we access an EL2 sysreg, we must

Quoted 7 lines
Quoted 1 line

Do we have vEL1? or is it a typo?

Not a typo, but only a convention (there is no such concept in the architecture). vELx denotes the exception level the guest thinks it is running at while running at EL1 (as it is the case for both vEL1 and vEL2).

OK got it, this is to deal with Non-VHE case.

Quoted 7 lines

Depending on the exception level and the running mode (VHE or not) you emulate at any given time, you access the sysregs differently: they can be either live in the CPU, stored in memory, with or without translation. That's why I'm using these 'parallel' exception levels to denote which is which...

HTH,

Thanks.

Quoted 3 lines
	M.

Looks good to me, please feel free to add,

Thanks,
Ganapat

author maintainer
#63 lore raw

Re: [PATCH v5 18/69] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg()

Quoted 21 lines · Ganapatrao Kulkarni
Quoted 17 lines · Marc Zyngier
Quoted 14 lines · Ganapatrao Kulkarni
Quoted 9 lines · Marc Zyngier

KVM internally uses accessor functions when reading or writing the guest's system registers. This takes care of accessing either the stored copy or using the "live" EL1 system registers when the host uses VHE.

With the introduction of virtual EL2 we add a bunch of EL2 system registers, which now must also be taken care of:
- If the guest is running in vEL2, and we access an EL1 sysreg, we must revert to the stored version of that, and not use the CPU's copy.
- If the guest is running in vEL1, and we access an EL2 sysreg, we must

Do we have vEL1? or is it a typo?

Quoted 7 lines
Quoted 5 lines

Not a typo, but only a convention (there is no such concept in the architecture). vELx denotes the exception level the guest thinks it is running at while running at EL1 (as it is the case for both vEL1 and vEL2).

OK got it, this is to deal with Non-VHE case.

No, you'd have the exact same thing with a VHE guest itself running an EL1 guest. You really cannot distinguish the two cases.

In general, you can't really think the NV support in terms of VHE or nVHE, or even in terms of guest level. You need to think in terms of a single machine with three exception levels, and follow the rules of the architecture to the letter.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#64 lore raw

Re: [PATCH v5 18/69] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg()

Quoted 29 lines · Marc Zyngier
Quoted 27 lines · Ganapatrao Kulkarni
Quoted 22 lines · Marc Zyngier
Quoted 7 lines · Ganapatrao Kulkarni
Quoted 4 lines · Marc Zyngier

KVM internally uses accessor functions when reading or writing the guest's system registers. This takes care of accessing either the stored copy or using the "live" EL1 system registers when the host uses VHE.

Quoted 7 lines
Quoted 5 lines

With the introduction of virtual EL2 we add a bunch of EL2 system registers, which now must also be taken care of:
- If the guest is running in vEL2, and we access an EL1 sysreg, we must revert to the stored version of that, and not use the CPU's copy.
- If the guest is running in vEL1, and we access an EL2 sysreg, we must

Do we have vEL1? or is it a typo?

Not a typo, but only a convention (there is no such concept in the architecture). vELx denotes the exception level the guest thinks it is running at while running at EL1 (as it is the case for both vEL1 and vEL2).

Quoted 5 lines
Quoted 1 line

OK got it, this is to deal with Non-VHE case.

No, you'd have the exact same thing with a VHE guest itself running an EL1 guest. You really cannot distinguish the two cases.

Okay understood, thanks.

Quoted 4 lines

In general, you can't really think the NV support in terms of VHE or nVHE, or even in terms of guest level. You need to think in terms of a single machine with three exception levels, and follow the rules of the architecture to the letter.

OK.

Quoted 5 lines

Thanks,

	M.

Thanks,
Ganapat

Reviewed-by on v5 18/69
#65 lore raw

Re: [PATCH v5 18/69] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg()

In reply to Marc Zyngier

Hi Marc,

Quoted 56 lines · Marc Zyngier

KVM internally uses accessor functions when reading or writing the guest's system registers. This takes care of accessing either the stored copy or using the "live" EL1 system registers when the host uses VHE.

With the introduction of virtual EL2 we add a bunch of EL2 system registers, which now must also be taken care of:
- If the guest is running in vEL2, and we access an EL1 sysreg, we must revert to the stored version of that, and not use the CPU's copy.
- If the guest is running in vEL1, and we access an EL2 sysreg, we must also use the stored version, since the CPU carries the EL1 copy.
- Some EL2 system registers are supposed to affect the current execution of the system, so we need to put them into their respective EL1 counterparts. For this we need to define a mapping between the two. This is done using the newly introduced struct el2_sysreg_map.
- Some EL2 system registers have a different format than their EL1 counterpart, so we need to translate them before writing them to the CPU. This is done using an (optional) translate function in the map.
- There are the three special registers SP_EL2, SPSR_EL2 and ELR_EL2, which need some separate handling (SPSR_EL2 is being handled in a separate patch).

All of these cases are now wrapped into the existing accessor functions, so KVM users wouldn't need to care whether they access EL2 or EL1 registers and also which state the guest is in.

This handles what was formerly known as the "shadow state" dynamically, without requiring a separate copy for each vCPU EL.

---
 arch/arm64/kvm/sys_regs.c | 143 ++++++++++++++++++++++++++++++++++++--
 1 file changed, 139 insertions(+), 4 deletions(-)
arch/arm64/kvm/sys_regs.c +6 -0
index 730a24468915..61596355e42d 100644
@@ -24,6 +24,7 @@
 #include <asm/kvm_emulate.h>
 #include <asm/kvm_hyp.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
 #include <asm/perf_event.h>
 #include <asm/sysreg.h>
 
@@ -64,23 +65,157 @@ static bool write_to_read_only(struct kvm_vcpu *vcpu,
 	return false;
 }
 
+#define PURE_EL2_SYSREG(el2)						\
+	case el2: {							\
+		*el1r = el2;						\
+		return true;						\
+	}

arch/arm64/kvm/sys_regs.c static bool write_to_read_only(struct kvm_vcpu *vcpu, · line 82

Quoted 10 lines
arch/arm64/kvm/sys_regs.c +10 -0
+
+#define MAPPED_EL2_SYSREG(el2, el1, fn)					\
+	case el2: {							\
+		*xlate = fn;						\
+		*el1r = el1;						\
+		return true;						\
+	}
+
+static bool get_el2_mapping(unsigned int reg,
+			    unsigned int *el1r, u64 (**xlate)(u64))

Nitpick: this could be renamed to get_el2_to_el1_mapping() to remove any ambiguities regarding what is being mapped to what (and to match the translate_* functions). With our without this change, the patch looks correct to me:

Thanks,
Alex

Quoted 146 lines
arch/arm64/kvm/sys_regs.c +123 -4
+{
+	switch (reg) {
+		PURE_EL2_SYSREG(  VPIDR_EL2	);
+		PURE_EL2_SYSREG(  VMPIDR_EL2	);
+		PURE_EL2_SYSREG(  ACTLR_EL2	);
+		PURE_EL2_SYSREG(  HCR_EL2	);
+		PURE_EL2_SYSREG(  MDCR_EL2	);
+		PURE_EL2_SYSREG(  HSTR_EL2	);
+		PURE_EL2_SYSREG(  HACR_EL2	);
+		PURE_EL2_SYSREG(  VTTBR_EL2	);
+		PURE_EL2_SYSREG(  VTCR_EL2	);
+		PURE_EL2_SYSREG(  RVBAR_EL2	);
+		PURE_EL2_SYSREG(  TPIDR_EL2	);
+		PURE_EL2_SYSREG(  HPFAR_EL2	);
+		PURE_EL2_SYSREG(  ELR_EL2	);
+		PURE_EL2_SYSREG(  SPSR_EL2	);
+		MAPPED_EL2_SYSREG(SCTLR_EL2,   SCTLR_EL1,
+				  translate_sctlr_el2_to_sctlr_el1	     );
+		MAPPED_EL2_SYSREG(CPTR_EL2,    CPACR_EL1,
+				  translate_cptr_el2_to_cpacr_el1	     );
+		MAPPED_EL2_SYSREG(TTBR0_EL2,   TTBR0_EL1,
+				  translate_ttbr0_el2_to_ttbr0_el1	     );
+		MAPPED_EL2_SYSREG(TTBR1_EL2,   TTBR1_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(TCR_EL2,     TCR_EL1,
+				  translate_tcr_el2_to_tcr_el1		     );
+		MAPPED_EL2_SYSREG(VBAR_EL2,    VBAR_EL1,    NULL	     );
+		MAPPED_EL2_SYSREG(AFSR0_EL2,   AFSR0_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(AFSR1_EL2,   AFSR1_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(ESR_EL2,     ESR_EL1,     NULL	     );
+		MAPPED_EL2_SYSREG(FAR_EL2,     FAR_EL1,     NULL	     );
+		MAPPED_EL2_SYSREG(MAIR_EL2,    MAIR_EL1,    NULL	     );
+		MAPPED_EL2_SYSREG(AMAIR_EL2,   AMAIR_EL1,   NULL	     );
+		MAPPED_EL2_SYSREG(CNTHCTL_EL2, CNTKCTL_EL1,
+				  translate_cnthctl_el2_to_cntkctl_el1	     );
+	default:
+		return false;
+	}
+}
+
 u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
 {
 	u64 val = 0x8badf00d8badf00d;
+	u64 (*xlate)(u64) = NULL;
+	unsigned int el1r;
+
+	if (!vcpu->arch.sysregs_loaded_on_cpu)
+		goto memory_read;
+
+	if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) {
+		if (!is_hyp_ctxt(vcpu))
+			goto memory_read;
+
+		/*
+		 * ELR_EL2 is special cased for now.
+		 */
+		switch (reg) {
+		case ELR_EL2:
+			return read_sysreg_el1(SYS_ELR);
+		}
+
+		/*
+		 * If this register does not have an EL1 counterpart,
+		 * then read the stored EL2 version.
+		 */
+		if (reg == el1r)
+			goto memory_read;
+
+		/*
+		 * If we have a non-VHE guest and that the sysreg
+		 * requires translation to be used at EL1, use the
+		 * in-memory copy instead.
+		 */
+		if (!vcpu_el2_e2h_is_set(vcpu) && xlate)
+			goto memory_read;
+
+		/* Get the current version of the EL1 counterpart. */
+		WARN_ON(!__vcpu_read_sys_reg_from_cpu(el1r, &val));
+		return val;
+	}
+
+	/* EL1 register can't be on the CPU if the guest is in vEL2. */
+	if (unlikely(is_hyp_ctxt(vcpu)))
+		goto memory_read;
 
-	if (vcpu->arch.sysregs_loaded_on_cpu &&
-	    __vcpu_read_sys_reg_from_cpu(reg, &val))
+	if (__vcpu_read_sys_reg_from_cpu(reg, &val))
 		return val;
 
+memory_read:
 	return __vcpu_sys_reg(vcpu, reg);
 }
 
 void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
 {
-	if (vcpu->arch.sysregs_loaded_on_cpu &&
-	    __vcpu_write_sys_reg_to_cpu(val, reg))
+	u64 (*xlate)(u64) = NULL;
+	unsigned int el1r;
+
+	if (!vcpu->arch.sysregs_loaded_on_cpu)
+		goto memory_write;
+
+	if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) {
+		if (!is_hyp_ctxt(vcpu))
+			goto memory_write;
+
+		/*
+		 * Always store a copy of the write to memory to avoid having
+		 * to reverse-translate virtual EL2 system registers for a
+		 * non-VHE guest hypervisor.
+		 */
+		__vcpu_sys_reg(vcpu, reg) = val;
+
+		switch (reg) {
+		case ELR_EL2:
+			write_sysreg_el1(val, SYS_ELR);
+			return;
+		}
+
+		/* No EL1 counterpart? We're done here.? */
+		if (reg == el1r)
+			return;
+
+		if (!vcpu_el2_e2h_is_set(vcpu) && xlate)
+			val = xlate(val);
+
+		/* Redirect this to the EL1 version of the register. */
+		WARN_ON(!__vcpu_write_sys_reg_to_cpu(val, el1r));
+		return;
+	}
+
+	/* EL1 register can't be on the CPU if the guest is in vEL2. */
+	if (unlikely(is_hyp_ctxt(vcpu)))
+		goto memory_write;
+
+	if (__vcpu_write_sys_reg_to_cpu(val, reg))
 		return;
 
+memory_write:
 	 __vcpu_sys_reg(vcpu, reg) = val;
 }
 
Signature · 1 line
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author maintainer
#66 lore raw

PATCH v5 19/69 KVM: arm64: nv: Handle SPSR_EL2 specially

In reply to Marc Zyngier

SPSR_EL2 needs special attention when running nested on ARMv8.3:

If taking an exception while running at vEL2 (actually EL1), the HW will update the SPSR_EL1 register with the EL1 mode. We need to track this in order to make sure that accesses to the virtual view of SPSR_EL2 is correct.

To do so, we place an illegal value in SPSR_EL1.M, and patch it accordingly if required when accessing it.

---
 arch/arm64/include/asm/kvm_emulate.h | 37 ++++++++++++++++++++++++++++
 arch/arm64/kvm/sys_regs.c            | 23 +++++++++++++++--
 2 files changed, 58 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +37 -0
index 46c849ba281e..2db11ff3fdad 100644
@@ -241,6 +241,43 @@ static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu)
 	return __is_hyp_ctxt(&vcpu->arch.ctxt);
 }
 
+static inline u64 __fixup_spsr_el2_write(struct kvm_cpu_context *ctxt, u64 val)
+{
+	if (!__vcpu_el2_e2h_is_set(ctxt)) {
+		/*
+		 * Clear the .M field when writing SPSR to the CPU, so that we
+		 * can detect when the CPU clobbered our SPSR copy during a
+		 * local exception.
+		 */
+		val &= ~0xc;
+	}
+
+	return val;
+}
+
+static inline u64 __fixup_spsr_el2_read(const struct kvm_cpu_context *ctxt, u64 val)
+{
+	if (__vcpu_el2_e2h_is_set(ctxt))
+		return val;
+
+	/*
+	 * SPSR.M == 0 means the CPU has not touched the SPSR, so the
+	 * register has still the value we saved on the last write.
+	 */
+	if ((val & 0xc) == 0)
+		return ctxt_sys_reg(ctxt, SPSR_EL2);
+
+	/*
+	 * Otherwise there was a "local" exception on the CPU,
+	 * which from the guest's point of view was being taken from
+	 * EL2 to EL2, although it actually happened to be from
+	 * EL1 to EL1.
+	 * So we need to fix the .M field in SPSR, to make it look
+	 * like EL2, which is what the guest would expect.
+	 */
+	return (val & ~0x0c) | CurrentEL_EL2;
+}
+
 /*
  * The layout of SPSR for an AArch32 state is different when observed from an
  * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32
arch/arm64/kvm/sys_regs.c +21 -2
index 61596355e42d..875040bcfbe1 100644
@@ -133,11 +133,14 @@ u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
 			goto memory_read;
 
 		/*
-		 * ELR_EL2 is special cased for now.
+		 * ELR_EL2 and SPSR_EL2 are special cased for now.
 		 */
 		switch (reg) {
 		case ELR_EL2:
 			return read_sysreg_el1(SYS_ELR);
+		case SPSR_EL2:
+			val = read_sysreg_el1(SYS_SPSR);
+			return __fixup_spsr_el2_read(&vcpu->arch.ctxt, val);
 		}
 
 		/*
@@ -194,6 +197,10 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
 		case ELR_EL2:
 			write_sysreg_el1(val, SYS_ELR);
 			return;
+		case SPSR_EL2:
+			val = __fixup_spsr_el2_write(&vcpu->arch.ctxt, val);
+			write_sysreg_el1(val, SYS_SPSR);
+			return;
 		}
 
 		/* No EL1 counterpart? We're done here.? */
@@ -1620,6 +1627,18 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu,
 	return true;
 }
 
+static bool access_spsr_el2(struct kvm_vcpu *vcpu,
+			    struct sys_reg_params *p,
+			    const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2);
+	else
+		p->regval = vcpu_read_sys_reg(vcpu, SPSR_EL2);
+
+	return true;
+}
+
 /*
  * Architected system registers.
  * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
@@ -2052,7 +2071,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_VTCR_EL2), access_rw, reset_val, VTCR_EL2, 0 },
 
 	{ SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 },
-	{ SYS_DESC(SYS_SPSR_EL2), access_rw, reset_val, SPSR_EL2, 0 },
+	{ SYS_DESC(SYS_SPSR_EL2), access_spsr_el2, reset_val, SPSR_EL2, 0 },
 	{ SYS_DESC(SYS_ELR_EL2), access_rw, reset_val, ELR_EL2, 0 },
 	{ SYS_DESC(SYS_SP_EL1), access_sp_el1},
 
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 19/69
#67 lore raw

Re: [PATCH v5 19/69] KVM: arm64: nv: Handle SPSR_EL2 specially

Hi Marc,

Quoted 9 lines · Marc Zyngier

SPSR_EL2 needs special attention when running nested on ARMv8.3:

If taking an exception while running at vEL2 (actually EL1), the HW will update the SPSR_EL1 register with the EL1 mode. We need to track this in order to make sure that accesses to the virtual view of SPSR_EL2 is correct.

To do so, we place an illegal value in SPSR_EL1.M, and patch it accordingly if required when accessing it.

When KVM runs a L1 guest hypervisor with hardware HCR_EL2.NV1 clear (virtual HCR_EL2.E2H set), when an exception is taken from vEL2 to vEL2, SPSR_EL1.M is automatically set to CurrentEL_EL2. But this doesn't happen when hardware NV1 is set (virtual HCR_EL2.E2H unset) and as far as I can tell this patch does the right thing to emulate it in software:

Thanks,
Alex

Quoted 117 lines
---
 arch/arm64/include/asm/kvm_emulate.h | 37 ++++++++++++++++++++++++++++
 arch/arm64/kvm/sys_regs.c            | 23 +++++++++++++++--
 2 files changed, 58 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +37 -0
index 46c849ba281e..2db11ff3fdad 100644
@@ -241,6 +241,43 @@ static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu)
 	return __is_hyp_ctxt(&vcpu->arch.ctxt);
 }
 
+static inline u64 __fixup_spsr_el2_write(struct kvm_cpu_context *ctxt, u64 val)
+{
+	if (!__vcpu_el2_e2h_is_set(ctxt)) {
+		/*
+		 * Clear the .M field when writing SPSR to the CPU, so that we
+		 * can detect when the CPU clobbered our SPSR copy during a
+		 * local exception.
+		 */
+		val &= ~0xc;
+	}
+
+	return val;
+}
+
+static inline u64 __fixup_spsr_el2_read(const struct kvm_cpu_context *ctxt, u64 val)
+{
+	if (__vcpu_el2_e2h_is_set(ctxt))
+		return val;
+
+	/*
+	 * SPSR.M == 0 means the CPU has not touched the SPSR, so the
+	 * register has still the value we saved on the last write.
+	 */
+	if ((val & 0xc) == 0)
+		return ctxt_sys_reg(ctxt, SPSR_EL2);
+
+	/*
+	 * Otherwise there was a "local" exception on the CPU,
+	 * which from the guest's point of view was being taken from
+	 * EL2 to EL2, although it actually happened to be from
+	 * EL1 to EL1.
+	 * So we need to fix the .M field in SPSR, to make it look
+	 * like EL2, which is what the guest would expect.
+	 */
+	return (val & ~0x0c) | CurrentEL_EL2;
+}
+
 /*
  * The layout of SPSR for an AArch32 state is different when observed from an
  * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32
arch/arm64/kvm/sys_regs.c +21 -2
index 61596355e42d..875040bcfbe1 100644
@@ -133,11 +133,14 @@ u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg)
 			goto memory_read;
 
 		/*
-		 * ELR_EL2 is special cased for now.
+		 * ELR_EL2 and SPSR_EL2 are special cased for now.
 		 */
 		switch (reg) {
 		case ELR_EL2:
 			return read_sysreg_el1(SYS_ELR);
+		case SPSR_EL2:
+			val = read_sysreg_el1(SYS_SPSR);
+			return __fixup_spsr_el2_read(&vcpu->arch.ctxt, val);
 		}
 
 		/*
@@ -194,6 +197,10 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
 		case ELR_EL2:
 			write_sysreg_el1(val, SYS_ELR);
 			return;
+		case SPSR_EL2:
+			val = __fixup_spsr_el2_write(&vcpu->arch.ctxt, val);
+			write_sysreg_el1(val, SYS_SPSR);
+			return;
 		}
 
 		/* No EL1 counterpart? We're done here.? */
@@ -1620,6 +1627,18 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu,
 	return true;
 }
 
+static bool access_spsr_el2(struct kvm_vcpu *vcpu,
+			    struct sys_reg_params *p,
+			    const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2);
+	else
+		p->regval = vcpu_read_sys_reg(vcpu, SPSR_EL2);
+
+	return true;
+}
+
 /*
  * Architected system registers.
  * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
@@ -2052,7 +2071,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_VTCR_EL2), access_rw, reset_val, VTCR_EL2, 0 },
 
 	{ SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 },
-	{ SYS_DESC(SYS_SPSR_EL2), access_rw, reset_val, SPSR_EL2, 0 },
+	{ SYS_DESC(SYS_SPSR_EL2), access_spsr_el2, reset_val, SPSR_EL2, 0 },
 	{ SYS_DESC(SYS_ELR_EL2), access_rw, reset_val, ELR_EL2, 0 },
 	{ SYS_DESC(SYS_SP_EL1), access_sp_el1},
 
Signature · 1 line
2.30.2
author maintainer
#68 lore raw

PATCH v5 20/69 KVM: arm64: nv: Handle HCR_EL2.E2H specially

In reply to Marc Zyngier

HCR_EL2.E2H is nasty, as a flip of this bit completely changes the way we deal with a lot of the state. So when the guest flips this bit (sysregs are live), do the put/load dance so that we have a consistent state.

Yes, this is slow. Don't do it.

---
 arch/arm64/kvm/sys_regs.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
arch/arm64/kvm/sys_regs.c +20 -0
index 875040bcfbe1..c1408dff58fa 100644
@@ -183,9 +183,24 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
 		goto memory_write;
 
 	if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) {
+		bool need_put_load;
+
 		if (!is_hyp_ctxt(vcpu))
 			goto memory_write;
 
+		/*
+		 * HCR_EL2.E2H is nasty: it changes the way we interpret a
+		 * lot of the EL2 state, so treat is as a full state
+		 * transition.
+		 */
+		need_put_load = ((reg == HCR_EL2) &&
+				 vcpu_el2_e2h_is_set(vcpu) != !!(val & HCR_E2H));
+
+		if (need_put_load) {
+			preempt_disable();
+			kvm_arch_vcpu_put(vcpu);
+		}
+
 		/*
 		 * Always store a copy of the write to memory to avoid having
 		 * to reverse-translate virtual EL2 system registers for a
@@ -193,6 +208,11 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg)
 		 */
 		__vcpu_sys_reg(vcpu, reg) = val;
 
+		if (need_put_load) {
+			kvm_arch_vcpu_load(vcpu, smp_processor_id());
+			preempt_enable();
+		}
+
 		switch (reg) {
 		case ELR_EL2:
 			write_sysreg_el1(val, SYS_ELR);
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#69 lore raw

PATCH v5 21/69 KVM: arm64: nv: Save/Restore vEL2 sysregs

In reply to Marc Zyngier

Whenever we need to restore the guest's system registers to the CPU, we now need to take care of the EL2 system registers as well. Most of them are accessed via traps only, but some have an immediate effect and also a guest running in VHE mode would expect them to be accessible via their EL1 encoding, which we do not trap.

For vEL2 we write the virtual EL2 registers with an identical format directly into their EL1 counterpart, and translate the few registers that have a different format for the same effect on the execution when running a non-VHE guest guest hypervisor.

Based on an initial patch from Andre Przywara, rewritten many times since.

---
 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h |   5 +-
 arch/arm64/kvm/hyp/nvhe/sysreg-sr.c        |   2 +-
 arch/arm64/kvm/hyp/vhe/sysreg-sr.c         | 125 ++++++++++++++++++++-
 3 files changed, 127 insertions(+), 5 deletions(-)
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +3 -2
index 7ecca8b07851..283f780f5f56 100644
@@ -92,9 +92,10 @@ static inline void __sysreg_restore_user_state(struct kvm_cpu_context *ctxt)
 	write_sysreg(ctxt_sys_reg(ctxt, TPIDRRO_EL0),	tpidrro_el0);
 }
 
-static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt)
+static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt,
+					      u64 mpidr)
 {
-	write_sysreg(ctxt_sys_reg(ctxt, MPIDR_EL1),	vmpidr_el2);
+	write_sysreg(mpidr,				vmpidr_el2);
 	write_sysreg(ctxt_sys_reg(ctxt, CSSELR_EL1),	csselr_el1);
 
 	if (has_vhe() ||
arch/arm64/kvm/hyp/nvhe/sysreg-sr.c +1 -1
index 29305022bc04..dba101565de3 100644
@@ -28,7 +28,7 @@ void __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt)
 
 void __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
 {
-	__sysreg_restore_el1_state(ctxt);
+	__sysreg_restore_el1_state(ctxt, ctxt_sys_reg(ctxt, MPIDR_EL1));
 	__sysreg_restore_common_state(ctxt);
 	__sysreg_restore_user_state(ctxt);
 	__sysreg_restore_el2_return_state(ctxt);
arch/arm64/kvm/hyp/vhe/sysreg-sr.c +123 -2

154 lines not on this page · raw message

Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#70 lore raw

PATCH v5 22/69 KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

We can no longer blindly copy the VCPU's PSTATE into SPSR_EL2 and return to the guest and vice versa when taking an exception to the hypervisor, because we emulate virtual EL2 in EL1 and therefore have to translate the mode field from EL2 to EL1 and vice versa.

This requires keeping track of the state we enter the guest, for which we transiently use a dedicated flag.

---
 arch/arm64/include/asm/kvm_host.h          |  1 +
 arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 19 ++++++++++++++++-
 arch/arm64/kvm/hyp/vhe/switch.c            | 24 ++++++++++++++++++++++
 3 files changed, 43 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_host.h +1 -0
index 7b6fe18ee450..6a7b13edc5cb 100644
@@ -456,6 +456,7 @@ struct kvm_vcpu_arch {
 #define KVM_ARM64_EXCEPT_MASK		(7 << 9) /* Target EL/MODE */
 #define KVM_ARM64_DEBUG_STATE_SAVE_SPE	(1 << 12) /* Save SPE context if active  */
 #define KVM_ARM64_DEBUG_STATE_SAVE_TRBE	(1 << 13) /* Save TRBE context if active  */
+#define KVM_ARM64_IN_HYP_CONTEXT	(1 << 14) /* Guest running in HYP context */
 
 #define KVM_GUESTDBG_VALID_MASK (KVM_GUESTDBG_ENABLE | \
 				 KVM_GUESTDBG_USE_SW_BP | \
arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +18 -1
index 283f780f5f56..e3689c6ce4cc 100644
@@ -157,9 +157,26 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt,
 	write_sysreg_el1(ctxt_sys_reg(ctxt, SPSR_EL1),	SYS_SPSR);
 }
 
+/* Read the VCPU state's PSTATE, but translate (v)EL2 to EL1. */
+static inline u64 to_hw_pstate(const struct kvm_cpu_context *ctxt)
+{
+	u64 mode = ctxt->regs.pstate & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+	switch (mode) {
+	case PSR_MODE_EL2t:
+		mode = PSR_MODE_EL1t;
+		break;
+	case PSR_MODE_EL2h:
+		mode = PSR_MODE_EL1h;
+		break;
+	}
+
+	return (ctxt->regs.pstate & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode;
+}
+
 static inline void __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt)
 {
-	u64 pstate = ctxt->regs.pstate;
+	u64 pstate = to_hw_pstate(ctxt);
 	u64 mode = pstate & PSR_AA32_MODE_MASK;
 
 	/*
arch/arm64/kvm/hyp/vhe/switch.c +24 -0
index fbb26b93c347..9d5572ceb001 100644
@@ -114,6 +114,25 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)
 
 static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code)
 {
+	/*
+	 * If we were in HYP context on entry, adjust the PSTATE view
+	 * so that the usual helpers work correctly.
+	 */
+	if (unlikely(vcpu->arch.flags & KVM_ARM64_IN_HYP_CONTEXT)) {
+		u64 mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT);
+
+		switch (mode) {
+		case PSR_MODE_EL1t:
+			mode = PSR_MODE_EL2t;
+			break;
+		case PSR_MODE_EL1h:
+			mode = PSR_MODE_EL2h;
+			break;
+		}
+
+		*vcpu_cpsr(vcpu) &= ~(PSR_MODE_MASK | PSR_MODE32_BIT);
+		*vcpu_cpsr(vcpu) |= mode;
+	}
 }
 
 /* Switch to the guest for VHE systems running in EL2 */
@@ -148,6 +167,11 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
 	sysreg_restore_guest_state_vhe(guest_ctxt);
 	__debug_switch_to_guest(vcpu);
 
+	if (is_hyp_ctxt(vcpu))
+		vcpu->arch.flags |= KVM_ARM64_IN_HYP_CONTEXT;
+	else
+		vcpu->arch.flags &= ~KVM_ARM64_IN_HYP_CONTEXT;
+
 	do {
 		/* Jump in the fire! */
 		exit_code = __guest_enter(vcpu);
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#71 lore raw

PATCH v5 23/69 KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

When running in virtual EL2 mode, we actually run the hardware in EL1 and therefore have to use the EL1 registers to ensure correct operation.

By setting the HCR.TVM and HCR.TVRM we ensure that the virtual EL2 mode doesn't shoot itself in the foot when setting up what it believes to be a different mode's system register state (for example when preparing to switch to a VM).

We can leverage the existing sysregs infrastructure to support trapped accesses to these registers.

---
 arch/arm64/kvm/hyp/include/hyp/switch.h |  4 +---
 arch/arm64/kvm/hyp/nvhe/switch.c        |  2 +-
 arch/arm64/kvm/hyp/vhe/switch.c         |  7 ++++++-
 arch/arm64/kvm/sys_regs.c               | 19 ++++++++++++++++---
 4 files changed, 24 insertions(+), 8 deletions(-)
arch/arm64/kvm/hyp/include/hyp/switch.h +1 -3
index 96c5f3fb7838..0987abd4d57d 100644
@@ -111,10 +111,8 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu)
 		write_sysreg(0, pmuserenr_el0);
 }
 
-static inline void ___activate_traps(struct kvm_vcpu *vcpu)
+static inline void ___activate_traps(struct kvm_vcpu *vcpu, u64 hcr)
 {
-	u64 hcr = vcpu->arch.hcr_el2;
-
 	if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM))
 		hcr |= HCR_TVM;
 
arch/arm64/kvm/hyp/nvhe/switch.c +1 -1
index d13115a12434..b005eabcaddc 100644
@@ -39,7 +39,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
 {
 	u64 val;
 
-	___activate_traps(vcpu);
+	___activate_traps(vcpu, vcpu->arch.hcr_el2);
 	__activate_traps_common(vcpu);
 
 	val = vcpu->arch.cptr_el2;
arch/arm64/kvm/hyp/vhe/switch.c +6 -1
index 9d5572ceb001..87a68496073a 100644
@@ -33,9 +33,14 @@ DEFINE_PER_CPU(unsigned long, kvm_hyp_vector);
 
 static void __activate_traps(struct kvm_vcpu *vcpu)
 {
+	u64 hcr = vcpu->arch.hcr_el2;
 	u64 val;
 
-	___activate_traps(vcpu);
+	/* Trap VM sysreg accesses if an EL2 guest is not using VHE. */
+	if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu))
+		hcr |= HCR_TVM | HCR_TRVM;
+
+	___activate_traps(vcpu, hcr);
 
 	val = read_sysreg(cpacr_el1);
 	val |= CPACR_EL1_TTA;
arch/arm64/kvm/sys_regs.c +16 -3
index c1408dff58fa..19f7e2260462 100644
@@ -350,8 +350,15 @@ static void get_access_mask(const struct sys_reg_desc *r, u64 *mask, u64 *shift)
 
 /*
  * Generic accessor for VM registers. Only called as long as HCR_TVM
- * is set. If the guest enables the MMU, we stop trapping the VM
- * sys_regs and leave it in complete control of the caches.
+ * is set.
+ *
+ * This is set in two cases: either (1) we're running at vEL2, or (2)
+ * we're running at EL1 and the guest has its MMU off.
+ *
+ * (1) TVM/TRVM is set, as we need to virtualise some of the VM
+ * registers for the guest hypervisor
+ * (2) Once the guest enables the MMU, we stop trapping the VM sys_regs
+ * and leave it in complete control of the caches.
  */
 static bool access_vm_reg(struct kvm_vcpu *vcpu,
 			  struct sys_reg_params *p,
@@ -360,7 +367,13 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
 	bool was_enabled = vcpu_has_cache_enabled(vcpu);
 	u64 val, mask, shift;
 
-	BUG_ON(!p->is_write);
+	/* We don't expect TRVM on the host */
+	BUG_ON(!vcpu_mode_el2(vcpu) && !p->is_write);
+
+	if (!p->is_write) {
+		p->regval = vcpu_read_sys_reg(vcpu, r->reg);
+		return true;
+	}
 
 	get_access_mask(r, &mask, &shift);
 
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#72 lore raw

PATCH v5 24/69 KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

For the same reason we trap virtual memory register accesses at virtual EL2, we need to trap SPSR_EL1, ELR_EL1 and VBAR_EL1 accesses. ARM v8.3 introduces the HCR_EL2.NV1 bit to be able to trap on those register accesses in EL1. Do not set this bit until the whole nesting support is completed.

---
 arch/arm64/kvm/sys_regs.c | 29 ++++++++++++++++++++++++++++-
 1 file changed, 28 insertions(+), 1 deletion(-)
arch/arm64/kvm/sys_regs.c +28 -1
index 19f7e2260462..5960178b7d7c 100644
@@ -1660,6 +1660,30 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu,
 	return true;
 }
 
+static bool access_elr(struct kvm_vcpu *vcpu,
+		       struct sys_reg_params *p,
+		       const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1);
+	else
+		p->regval = vcpu_read_sys_reg(vcpu, ELR_EL1);
+
+	return true;
+}
+
+static bool access_spsr(struct kvm_vcpu *vcpu,
+			struct sys_reg_params *p,
+			const struct sys_reg_desc *r)
+{
+	if (p->is_write)
+		__vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval;
+	else
+		p->regval = __vcpu_sys_reg(vcpu, SPSR_EL1);
+
+	return true;
+}
+
 static bool access_spsr_el2(struct kvm_vcpu *vcpu,
 			    struct sys_reg_params *p,
 			    const struct sys_reg_desc *r)
@@ -1822,6 +1846,9 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	PTRAUTH_KEY(APDB),
 	PTRAUTH_KEY(APGA),
 
+	{ SYS_DESC(SYS_SPSR_EL1), access_spsr},
+	{ SYS_DESC(SYS_ELR_EL1), access_elr},
+
 	{ SYS_DESC(SYS_AFSR0_EL1), access_vm_reg, reset_unknown, AFSR0_EL1 },
 	{ SYS_DESC(SYS_AFSR1_EL1), access_vm_reg, reset_unknown, AFSR1_EL1 },
 	{ SYS_DESC(SYS_ESR_EL1), access_vm_reg, reset_unknown, ESR_EL1 },
@@ -1869,7 +1896,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_LORC_EL1), trap_loregion },
 	{ SYS_DESC(SYS_LORID_EL1), trap_loregion },
 
-	{ SYS_DESC(SYS_VBAR_EL1), NULL, reset_val, VBAR_EL1, 0 },
+	{ SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 },
 	{ SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 },
 
 	{ SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only },
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#73 lore raw

PATCH v5 25/69 KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

For the same reason we trap virtual memory register accesses in virtual EL2, we trap CPACR_EL1 access too; We allow the virtual EL2 mode to access EL1 system register state instead of the virtual EL2 one.

---
 arch/arm64/include/asm/kvm_arm.h | 2 +-
 arch/arm64/kvm/hyp/vhe/switch.c  | 3 +++
 arch/arm64/kvm/sys_regs.c        | 2 +-
 3 files changed, 5 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_arm.h +1 -1
index 0a0ee998ec5a..2eccf883e8fe 100644
@@ -276,7 +276,7 @@
 #define CPTR_EL2_TFP_SHIFT 10
 
 /* Hyp Coprocessor Trap Register */
-#define CPTR_EL2_TCPAC	(1 << 31)
+#define CPTR_EL2_TCPAC	(1U << 31)
 #define CPTR_EL2_TAM	(1 << 30)
 #define CPTR_EL2_TTA	(1 << 20)
 #define CPTR_EL2_TFP	(1 << CPTR_EL2_TFP_SHIFT)
arch/arm64/kvm/hyp/vhe/switch.c +3 -0
index 87a68496073a..57f43e607819 100644
@@ -65,6 +65,9 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
 		__activate_traps_fpsimd32(vcpu);
 	}
 
+	if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu))
+		val |= CPTR_EL2_TCPAC;
+	
 	write_sysreg(val, cpacr_el1);
 
 	write_sysreg(__this_cpu_read(kvm_hyp_vector), vbar_el1);
arch/arm64/kvm/sys_regs.c +1 -1
index 5960178b7d7c..6490b0e3dcaf 100644
@@ -1829,7 +1829,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 
 	{ SYS_DESC(SYS_SCTLR_EL1), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 },
 	{ SYS_DESC(SYS_ACTLR_EL1), access_actlr, reset_actlr, ACTLR_EL1 },
-	{ SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 },
+	{ SYS_DESC(SYS_CPACR_EL1), access_rw, reset_val, CPACR_EL1, 0 },
 
 	MTE_REG(RGSR_EL1),
 	MTE_REG(GCR_EL1),
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#74 lore raw

PATCH v5 26/69 KVM: arm64: nv: Handle PSCI call via smc from the guest

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

VMs used to execute hvc #0 for the psci call if EL3 is not implemented. However, when we come to provide the virtual EL2 mode to the VM, the host OS inside the VM calls kvm_call_hyp() which is also hvc #0. So, it's hard to differentiate between them from the host hypervisor's point of view.

So, let the VM execute smc instruction for the psci call. On ARMv8.3, even if EL3 is not implemented, a smc instruction executed at non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than being treated as UNDEFINED. So, the host hypervisor can handle this psci call without any confusion.

---
 arch/arm64/kvm/handle_exit.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
arch/arm64/kvm/handle_exit.c +22 -2
index 95ae624d6aa8..72b981004b1d 100644
@@ -62,6 +62,8 @@ static int handle_hvc(struct kvm_vcpu *vcpu)
 
 static int handle_smc(struct kvm_vcpu *vcpu)
 {
+	int ret;
+
 	/*
 	 * "If an SMC instruction executed at Non-secure EL1 is
 	 * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a
@@ -69,10 +71,28 @@ static int handle_smc(struct kvm_vcpu *vcpu)
 	 *
 	 * We need to advance the PC after the trap, as it would
 	 * otherwise return to the same address...
+	 *
+	 * If imm is non-zero, it's not defined, so just skip it.
+	 */
+	if (kvm_vcpu_hvc_get_imm(vcpu)) {
+		vcpu_set_reg(vcpu, 0, ~0UL);
+		kvm_incr_pc(vcpu);
+		return 1;
+	}
+
+	/*
+	 * If imm is zero, it's a psci call.
+	 * Note that on ARMv8.3, even if EL3 is not implemented, SMC executed
+	 * at Non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than
+	 * being treated as UNDEFINED.
 	 */
-	vcpu_set_reg(vcpu, 0, ~0UL);
+	ret = kvm_hvc_call_handler(vcpu);
+	if (ret < 0)
+		vcpu_set_reg(vcpu, 0, ~0UL);
+
 	kvm_incr_pc(vcpu);
-	return 1;
+
+	return ret;
 }
 
 /*
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#75 lore raw

PATCH v5 27/69 KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

Forward exceptions due to WFI or WFE instructions to the virtual EL2 if they are not coming from the virtual EL2 and virtual HCR_EL2.TWX is set.

---
 arch/arm64/include/asm/kvm_nested.h |  2 ++
 arch/arm64/kvm/Makefile             |  2 +-
 arch/arm64/kvm/handle_exit.c        | 11 +++++++-
 arch/arm64/kvm/nested.c             | 40 +++++++++++++++++++++++++++++
 4 files changed, 53 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm64/kvm/nested.c
arch/arm64/include/asm/kvm_nested.h +2 -0
index 67a2c0d05233..4c2ac9650a3e 100644
@@ -61,4 +61,6 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl)
 		(cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN)));
 }
 
+int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe);
+
 #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/Makefile +1 -1
index 1f602526e9a2..923ee13728b4 100644
@@ -16,7 +16,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \
 	 inject_fault.o va_layout.o handle_exit.o \
 	 guest.o debug.o reset.o sys_regs.o \
 	 vgic-sys-reg-v3.o fpsimd.o pmu.o \
-	 arch_timer.o trng.o emulate-nested.o \
+	 arch_timer.o trng.o emulate-nested.o nested.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
 	 vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/handle_exit.c +10 -1
index 72b981004b1d..68a9579aa13e 100644
@@ -119,7 +119,16 @@ static int handle_no_fpsimd(struct kvm_vcpu *vcpu)
  */
 static int kvm_handle_wfx(struct kvm_vcpu *vcpu)
 {
-	if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_WFx_ISS_WFE) {
+	bool is_wfe = !!(kvm_vcpu_get_esr(vcpu) & ESR_ELx_WFx_ISS_WFE);
+
+	if (nested_virt_in_use(vcpu)) {
+		int ret = handle_wfx_nested(vcpu, is_wfe);
+
+		if (ret != -EINVAL)
+			return ret;
+	}
+
+	if (is_wfe) {
 		trace_kvm_wfx_arm64(*vcpu_pc(vcpu), true);
 		vcpu->stat.wfe_exit_stat++;
 		kvm_vcpu_on_spin(vcpu, vcpu_mode_priv(vcpu));
arch/arm64/kvm/nested.c new file +40 -0
new file mode 100644
index 000000000000..42a96c8d2adc
@@ -0,0 +1,40 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2017 - Columbia University and Linaro Ltd.
+ * Author: Jintack Lim <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/kvm.h>
+#include <linux/kvm_host.h>
+
+#include <asm/kvm_emulate.h>
+
+/*
+ * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and
+ * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor
+ * handle this.
+ */
+int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe)
+{
+	u64 hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
+
+	if (vcpu_mode_el2(vcpu))
+		return -EINVAL;
+
+	if ((is_wfe && (hcr_el2 & HCR_TWE)) || (!is_wfe && (hcr_el2 & HCR_TWI)))
+		return kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu));
+
+	return -EINVAL;
+}
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#76 lore raw

PATCH v5 28/69 KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

Forward traps due to FP/ASIMD register accesses to the virtual EL2 if virtual CPTR_EL2.TFP is set (with HCR_EL2.E2H == 0) or CPTR_EL2.FPEN is configure to do so (with HCR_EL2.E2h == 1).

[maz: account for HCR_EL2.E2H when testing for TFP/FPEN, with
all the hard work actually being done by Chase Conklin]

---
 arch/arm64/include/asm/kvm_emulate.h    | 26 +++++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c            | 16 +++++++++++----
 arch/arm64/kvm/hyp/include/hyp/switch.h |  8 ++++++--
 3 files changed, 44 insertions(+), 6 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +26 -0
index 2db11ff3fdad..105fb0d4a366 100644
@@ -11,12 +11,14 @@
 #ifndef __ARM64_KVM_EMULATE_H__
 #define __ARM64_KVM_EMULATE_H__
 
+#include <linux/bitfield.h>
 #include <linux/kvm_host.h>
 
 #include <asm/debug-monitors.h>
 #include <asm/esr.h>
 #include <asm/kvm_arm.h>
 #include <asm/kvm_hyp.h>
+#include <asm/kvm_nested.h>
 #include <asm/ptrace.h>
 #include <asm/cputype.h>
 #include <asm/virt.h>
@@ -324,6 +326,30 @@ static inline bool vcpu_mode_priv(const struct kvm_vcpu *vcpu)
 	return mode != PSR_MODE_EL0t;
 }
 
+static inline bool guest_hyp_fpsimd_traps_enabled(const struct kvm_vcpu *vcpu)
+{
+	u64 val;
+
+	if (!nested_virt_in_use(vcpu))
+		return false;
+
+	val = vcpu_read_sys_reg(vcpu, CPTR_EL2);
+
+	if (!vcpu_el2_e2h_is_set(vcpu))
+		return (val & CPTR_EL2_TFP);
+
+	switch (FIELD_GET(CPACR_EL1_FPEN, val)) {
+	case 0b00:
+	case 0b10:
+		return true;
+	case 0b01:
+		return vcpu_el2_tge_is_set(vcpu) && !vcpu_mode_el2(vcpu);
+	case 0b11:
+	default:		/* GCC is dumb */
+		return false;
+	}
+}
+
 static __always_inline u32 kvm_vcpu_get_esr(const struct kvm_vcpu *vcpu)
 {
 	return vcpu->arch.fault.esr_el2;
arch/arm64/kvm/handle_exit.c +12 -4
index 68a9579aa13e..7721c7c36137 100644
@@ -96,11 +96,19 @@ static int handle_smc(struct kvm_vcpu *vcpu)
 }
 
 /*
- * Guest access to FP/ASIMD registers are routed to this handler only
- * when the system doesn't support FP/ASIMD.
+ * This handles the cases where the system does not support FP/ASIMD or when
+ * we are running nested virtualization and the guest hypervisor is trapping
+ * FP/ASIMD accesses by its guest guest.
+ *
+ * All other handling of guest vs. host FP/ASIMD register state is handled in
+ * fixup_guest_exit().
  */
-static int handle_no_fpsimd(struct kvm_vcpu *vcpu)
+static int kvm_handle_fpasimd(struct kvm_vcpu *vcpu)
 {
+	if (guest_hyp_fpsimd_traps_enabled(vcpu))
+		return kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu));
+
+	/* This is the case when the system doesn't support FP/ASIMD. */
 	kvm_inject_undefined(vcpu);
 	return 1;
 }
@@ -229,7 +237,7 @@ static exit_handle_fn arm_exit_handlers[] = {
 	[ESR_ELx_EC_BREAKPT_LOW]= kvm_handle_guest_debug,
 	[ESR_ELx_EC_BKPT32]	= kvm_handle_guest_debug,
 	[ESR_ELx_EC_BRK64]	= kvm_handle_guest_debug,
-	[ESR_ELx_EC_FP_ASIMD]	= handle_no_fpsimd,
+	[ESR_ELx_EC_FP_ASIMD]	= kvm_handle_fpasimd,
 	[ESR_ELx_EC_PAC]	= kvm_handle_ptrauth,
 };
 
arch/arm64/kvm/hyp/include/hyp/switch.h +6 -2
index 0987abd4d57d..110b347f887a 100644
@@ -184,8 +184,12 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)
 
 	esr_ec = kvm_vcpu_trap_get_class(vcpu);
 
-	/* Don't handle SVE traps for non-SVE vcpus here: */
-	if (!sve_guest && esr_ec != ESR_ELx_EC_FP_ASIMD)
+	/*
+	 * Don't handle SVE traps for non-SVE vcpus here. This
+	 * includes NV guests for the time being.
+	 */
+	if (!sve_guest && (esr_ec != ESR_ELx_EC_FP_ASIMD ||
+			   guest_hyp_fpsimd_traps_enabled(vcpu)))
 		return false;
 
 	/* Valid trap.  Switch the context: */
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#77 lore raw

PATCH v5 29/69 KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

Forward traps due to HCR_EL2.NV bit to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.NV bit is set.

In addition to EL2 register accesses, setting NV bit will also make EL12 register accesses trap to EL2. To emulate this for the virtual EL2, forword traps due to EL12 register accessses to the virtual EL2 if the virtual HCR_EL2.NV bit is set.

This is for recursive nested virtualization.

[Moved code to emulate-nested.c]
---
 arch/arm64/include/asm/kvm_arm.h    |  1 +
 arch/arm64/include/asm/kvm_nested.h |  2 ++
 arch/arm64/kvm/emulate-nested.c     | 27 +++++++++++++++++++++++++++
 arch/arm64/kvm/handle_exit.c        |  7 +++++++
 arch/arm64/kvm/sys_regs.c           | 24 ++++++++++++++++++++++++
 5 files changed, 61 insertions(+)
arch/arm64/include/asm/kvm_arm.h +1 -0
index 2eccf883e8fe..9759bc893a51 100644
@@ -20,6 +20,7 @@
 #define HCR_AMVOFFEN	(UL(1) << 51)
 #define HCR_FIEN	(UL(1) << 47)
 #define HCR_FWB		(UL(1) << 46)
+#define HCR_NV		(UL(1) << 42)
 #define HCR_API		(UL(1) << 41)
 #define HCR_APK		(UL(1) << 40)
 #define HCR_TEA		(UL(1) << 37)
arch/arm64/include/asm/kvm_nested.h +2 -0
index 4c2ac9650a3e..26cba7b4d743 100644
@@ -62,5 +62,7 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl)
 }
 
 int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe);
+extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
+extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
 
 #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/emulate-nested.c +27 -0
index 339e8272b01e..8c7f2fe24bc6 100644
@@ -25,11 +25,38 @@
 
 #include "trace.h"
 
+bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit)
+{
+	bool control_bit_set;
+
+	if (!nested_virt_in_use(vcpu))
+		return false;
+
+	control_bit_set = __vcpu_sys_reg(vcpu, HCR_EL2) & control_bit;
+	if (!vcpu_mode_el2(vcpu) && control_bit_set) {
+		kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu));
+		return true;
+	}
+	return false;
+}
+
+bool forward_nv_traps(struct kvm_vcpu *vcpu)
+{
+	return forward_traps(vcpu, HCR_NV);
+}
+
 void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
 {
 	u64 spsr, elr, mode;
 	bool direct_eret;
 
+	/*
+	 * Forward this trap to the virtual EL2 if the virtual
+	 * HCR_EL2.NV bit is set and this is coming from !EL2.
+	 */
+	if (forward_nv_traps(vcpu))
+		return;
+
 	/*
 	 * Going through the whole put/load motions is a waste of time
 	 * if this is a VHE guest hypervisor returning to its own
arch/arm64/kvm/handle_exit.c +7 -0
index 7721c7c36137..6ff709c124d0 100644
@@ -64,6 +64,13 @@ static int handle_smc(struct kvm_vcpu *vcpu)
 {
 	int ret;
 
+	/*
+	 * Forward this trapped smc instruction to the virtual EL2 if
+	 * the guest has asked for it.
+	 */
+	if (forward_traps(vcpu, HCR_TSC))
+		return 1;
+
 	/*
 	 * "If an SMC instruction executed at Non-secure EL1 is
 	 * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a
arch/arm64/kvm/sys_regs.c +24 -0
index 6490b0e3dcaf..3468b8df8661 100644
@@ -267,10 +267,19 @@ static u32 get_ccsidr(u32 csselr)
 	return ccsidr;
 }
 
+static bool el12_reg(struct sys_reg_params *p)
+{
+	/* All *_EL12 registers have Op1=5. */
+	return (p->Op1 == 5);
+}
+
 static bool access_rw(struct kvm_vcpu *vcpu,
 		      struct sys_reg_params *p,
 		      const struct sys_reg_desc *r)
 {
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
 	if (p->is_write)
 		vcpu_write_sys_reg(vcpu, p->regval, r->reg);
 	else
@@ -283,6 +292,9 @@ static bool access_sctlr_el2(struct kvm_vcpu *vcpu,
 			     struct sys_reg_params *p,
 			     const struct sys_reg_desc *r)
 {
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
 	if (p->is_write) {
 		u64 val = p->regval;
 
@@ -367,6 +379,9 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
 	bool was_enabled = vcpu_has_cache_enabled(vcpu);
 	u64 val, mask, shift;
 
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
 	/* We don't expect TRVM on the host */
 	BUG_ON(!vcpu_mode_el2(vcpu) && !p->is_write);
 
@@ -1664,6 +1679,9 @@ static bool access_elr(struct kvm_vcpu *vcpu,
 		       struct sys_reg_params *p,
 		       const struct sys_reg_desc *r)
 {
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
 	if (p->is_write)
 		vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1);
 	else
@@ -1676,6 +1694,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu,
 			struct sys_reg_params *p,
 			const struct sys_reg_desc *r)
 {
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
 	if (p->is_write)
 		__vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval;
 	else
@@ -1688,6 +1709,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu,
 			    struct sys_reg_params *p,
 			    const struct sys_reg_desc *r)
 {
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
 	if (p->is_write)
 		vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2);
 	else
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#78 lore raw

Re: [PATCH v5 29/69] KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting

Hi Marc,

Quoted 11 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

Forward traps due to HCR_EL2.NV bit to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.NV bit is set.

In addition to EL2 register accesses, setting NV bit will also make EL12 register accesses trap to EL2. To emulate this for the virtual EL2, forword traps due to EL12 register accessses to the virtual EL2 if the virtual HCR_EL2.NV bit is set.

This is for recursive nested virtualization.

What is recursive nested virtualization means?
Are we going to set NV/NV1/NV2 bits of ID_AA64MMFR2_EL1 of Guest-Hypervisor to support NV in Guest-Hypervisor?

Quoted 163 lines
[Moved code to emulate-nested.c]
---
  arch/arm64/include/asm/kvm_arm.h    |  1 +
  arch/arm64/include/asm/kvm_nested.h |  2 ++
  arch/arm64/kvm/emulate-nested.c     | 27 +++++++++++++++++++++++++++
  arch/arm64/kvm/handle_exit.c        |  7 +++++++
  arch/arm64/kvm/sys_regs.c           | 24 ++++++++++++++++++++++++
  5 files changed, 61 insertions(+)
arch/arm64/include/asm/kvm_arm.h +1 -0
index 2eccf883e8fe..9759bc893a51 100644
@@ -20,6 +20,7 @@
  #define HCR_AMVOFFEN	(UL(1) << 51)
  #define HCR_FIEN	(UL(1) << 47)
  #define HCR_FWB		(UL(1) << 46)
+#define HCR_NV		(UL(1) << 42)
  #define HCR_API		(UL(1) << 41)
  #define HCR_APK		(UL(1) << 40)
  #define HCR_TEA		(UL(1) << 37)
arch/arm64/include/asm/kvm_nested.h +2 -0
index 4c2ac9650a3e..26cba7b4d743 100644
@@ -62,5 +62,7 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl)
  }
 
  int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe);
+extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
+extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
 
  #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/emulate-nested.c +27 -0
index 339e8272b01e..8c7f2fe24bc6 100644
@@ -25,11 +25,38 @@
 
  #include "trace.h"
 
+bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit)
+{
+	bool control_bit_set;
+
+	if (!nested_virt_in_use(vcpu))
+		return false;
+
+	control_bit_set = __vcpu_sys_reg(vcpu, HCR_EL2) & control_bit;
+	if (!vcpu_mode_el2(vcpu) && control_bit_set) {
+		kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu));
+		return true;
+	}
+	return false;
+}
+
+bool forward_nv_traps(struct kvm_vcpu *vcpu)
+{
+	return forward_traps(vcpu, HCR_NV);
+}
+
  void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu)
  {
  	u64 spsr, elr, mode;
  	bool direct_eret;
 
+	/*
+	 * Forward this trap to the virtual EL2 if the virtual
+	 * HCR_EL2.NV bit is set and this is coming from !EL2.
+	 */
+	if (forward_nv_traps(vcpu))
+		return;
+
  	/*
  	 * Going through the whole put/load motions is a waste of time
  	 * if this is a VHE guest hypervisor returning to its own
arch/arm64/kvm/handle_exit.c +7 -0
index 7721c7c36137..6ff709c124d0 100644
@@ -64,6 +64,13 @@ static int handle_smc(struct kvm_vcpu *vcpu)
  {
  	int ret;
 
+	/*
+	 * Forward this trapped smc instruction to the virtual EL2 if
+	 * the guest has asked for it.
+	 */
+	if (forward_traps(vcpu, HCR_TSC))
+		return 1;
+
  	/*
  	 * "If an SMC instruction executed at Non-secure EL1 is
  	 * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a
arch/arm64/kvm/sys_regs.c +21 -0
index 6490b0e3dcaf..3468b8df8661 100644
@@ -267,10 +267,19 @@ static u32 get_ccsidr(u32 csselr)
  	return ccsidr;
  }
 
+static bool el12_reg(struct sys_reg_params *p)
+{
+	/* All *_EL12 registers have Op1=5. */
+	return (p->Op1 == 5);
+}
+
  static bool access_rw(struct kvm_vcpu *vcpu,
  		      struct sys_reg_params *p,
  		      const struct sys_reg_desc *r)
  {
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
  	if (p->is_write)
  		vcpu_write_sys_reg(vcpu, p->regval, r->reg);
  	else
@@ -283,6 +292,9 @@ static bool access_sctlr_el2(struct kvm_vcpu *vcpu,
  			     struct sys_reg_params *p,
  			     const struct sys_reg_desc *r)
  {
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
  	if (p->is_write) {
  		u64 val = p->regval;
 
@@ -367,6 +379,9 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
  	bool was_enabled = vcpu_has_cache_enabled(vcpu);
  	u64 val, mask, shift;
 
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
  	/* We don't expect TRVM on the host */
  	BUG_ON(!vcpu_mode_el2(vcpu) && !p->is_write);
 
@@ -1664,6 +1679,9 @@ static bool access_elr(struct kvm_vcpu *vcpu,
  		       struct sys_reg_params *p,
  		       const struct sys_reg_desc *r)
  {
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
  	if (p->is_write)
  		vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1);
  	else
@@ -1676,6 +1694,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu,
  			struct sys_reg_params *p,
  			const struct sys_reg_desc *r)
  {
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
  	if (p->is_write)
  		__vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval;
  	else

arch/arm64/kvm/sys_regs.c static bool access_spsr_el2(struct kvm_vcpu *vcpu, · line 1717

Quoted 9 lines
arch/arm64/kvm/sys_regs.c +3 -0
@@ -1688,6 +1709,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu,
  			    struct sys_reg_params *p,
  			    const struct sys_reg_desc *r)
  {
+	if (el12_reg(p) && forward_nv_traps(vcpu))
+		return false;
+
  	if (p->is_write)
  		vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2);
  	else

Thanks,
Ganapat

author maintainer
#79 lore raw

Re: [PATCH v5 29/69] KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting

Quoted 14 lines · Ganapatrao Kulkarni

Hi Marc,

Quoted 10 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

Forward traps due to HCR_EL2.NV bit to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.NV bit is set.

In addition to EL2 register accesses, setting NV bit will also make EL12 register accesses trap to EL2. To emulate this for the virtual EL2, forword traps due to EL12 register accessses to the virtual EL2 if the virtual HCR_EL2.NV bit is set.

Quoted 5 lines
Quoted 1 line

This is for recursive nested virtualization.

What is recursive nested virtualization means? Are we going to set NV/NV1/NV2 bits of ID_AA64MMFR2_EL1 of Guest-Hypervisor to support NV in Guest-Hypervisor?

Of course. An implementation of nested virtualisation that would stop at L1 would be pretty crap and fail to live up to the 'turtles all the way down' paradigm.

Note that the recursive support is still a work in progress, as making it work for real in a software model is an exercise in futility (for example, we make no effort to make the VNCR_EL2 mapping work past L1).

Once someone sends me a NV-capable box, I'll get it working.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
author maintainer
#80 lore raw

PATCH v5 30/69 KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

Forward the EL1 virtual memory register traps to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.TVM or TRVM bit is set.

This is for recursive nested virtualization.

---
 arch/arm64/kvm/sys_regs.c | 7 +++++++
 1 file changed, 7 insertions(+)
arch/arm64/kvm/sys_regs.c +7 -0
index 3468b8df8661..e96877fc3b2a 100644
@@ -382,6 +382,13 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu,
 	if (el12_reg(p) && forward_nv_traps(vcpu))
 		return false;
 
+	if (!el12_reg(p)) {
+		u64 bit = p->is_write ? HCR_TVM : HCR_TRVM;
+
+		if (forward_traps(vcpu, bit))
+			return false;
+	}
+
 	/* We don't expect TRVM on the host */
 	BUG_ON(!vcpu_mode_el2(vcpu) && !p->is_write);
 
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#81 lore raw

PATCH v5 31/69 KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

Forward ELR_EL1, SPSR_EL1 and VBAR_EL1 traps to the virtual EL2 if the virtual HCR_EL2.NV bit is set.

This is for recursive nested virtualization.

---
 arch/arm64/include/asm/kvm_arm.h |  1 +
 arch/arm64/kvm/sys_regs.c        | 28 +++++++++++++++++++++++++++-
 2 files changed, 28 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_arm.h +1 -0
index 9759bc893a51..68af5509e4b0 100644
@@ -20,6 +20,7 @@
 #define HCR_AMVOFFEN	(UL(1) << 51)
 #define HCR_FIEN	(UL(1) << 47)
 #define HCR_FWB		(UL(1) << 46)
+#define HCR_NV1		(UL(1) << 43)
 #define HCR_NV		(UL(1) << 42)
 #define HCR_API		(UL(1) << 41)
 #define HCR_APK		(UL(1) << 40)
arch/arm64/kvm/sys_regs.c +27 -1
index e96877fc3b2a..511e06b6f603 100644
@@ -288,6 +288,22 @@ static bool access_rw(struct kvm_vcpu *vcpu,
 	return true;
 }
 
+/* This function is to support the recursive nested virtualization */
+static bool forward_nv1_traps(struct kvm_vcpu *vcpu, struct sys_reg_params *p)
+{
+	return forward_traps(vcpu, HCR_NV1);
+}
+
+static bool access_vbar_el1(struct kvm_vcpu *vcpu,
+			    struct sys_reg_params *p,
+			    const struct sys_reg_desc *r)
+{
+	if (forward_nv1_traps(vcpu, p))
+		return false;
+
+	return access_rw(vcpu, p, r);
+}
+
 static bool access_sctlr_el2(struct kvm_vcpu *vcpu,
 			     struct sys_reg_params *p,
 			     const struct sys_reg_desc *r)
@@ -1682,6 +1698,7 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu,
 	return true;
 }
 
+
 static bool access_elr(struct kvm_vcpu *vcpu,
 		       struct sys_reg_params *p,
 		       const struct sys_reg_desc *r)
@@ -1689,6 +1706,9 @@ static bool access_elr(struct kvm_vcpu *vcpu,
 	if (el12_reg(p) && forward_nv_traps(vcpu))
 		return false;
 
+	if (!el12_reg(p) && forward_nv1_traps(vcpu, p))
+		return false;
+
 	if (p->is_write)
 		vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1);
 	else
@@ -1704,6 +1724,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu,
 	if (el12_reg(p) && forward_nv_traps(vcpu))
 		return false;
 
+	if (!el12_reg(p) && forward_nv1_traps(vcpu, p))
+		return false;
+
 	if (p->is_write)
 		__vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval;
 	else
@@ -1719,6 +1742,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu,
 	if (el12_reg(p) && forward_nv_traps(vcpu))
 		return false;
 
+	if (!el12_reg(p) && forward_nv1_traps(vcpu, p))
+		return false;
+
 	if (p->is_write)
 		vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2);
 	else
@@ -1927,7 +1953,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_LORC_EL1), trap_loregion },
 	{ SYS_DESC(SYS_LORID_EL1), trap_loregion },
 
-	{ SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 },
+	{ SYS_DESC(SYS_VBAR_EL1), access_vbar_el1, reset_val, VBAR_EL1, 0 },
 	{ SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 },
 
 	{ SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only },
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#82 lore raw

Re: [PATCH v5 31/69] KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting

Quoted 31 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

Forward ELR_EL1, SPSR_EL1 and VBAR_EL1 traps to the virtual EL2 if the virtual HCR_EL2.NV bit is set.

This is for recursive nested virtualization.

---
  arch/arm64/include/asm/kvm_arm.h |  1 +
  arch/arm64/kvm/sys_regs.c        | 28 +++++++++++++++++++++++++++-
  2 files changed, 28 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_arm.h +1 -0
index 9759bc893a51..68af5509e4b0 100644
@@ -20,6 +20,7 @@
  #define HCR_AMVOFFEN	(UL(1) << 51)
  #define HCR_FIEN	(UL(1) << 47)
  #define HCR_FWB		(UL(1) << 46)
+#define HCR_NV1		(UL(1) << 43)
  #define HCR_NV		(UL(1) << 42)
  #define HCR_API		(UL(1) << 41)
  #define HCR_APK		(UL(1) << 40)
arch/arm64/kvm/sys_regs.c +0 -0
index e96877fc3b2a..511e06b6f603 100644

arch/arm64/kvm/sys_regs.c static bool access_rw(struct kvm_vcpu *vcpu, · line 296

Quoted 9 lines
arch/arm64/kvm/sys_regs.c +6 -0
@@ -288,6 +288,22 @@ static bool access_rw(struct kvm_vcpu *vcpu,
  	return true;
  }
 
+/* This function is to support the recursive nested virtualization */
+static bool forward_nv1_traps(struct kvm_vcpu *vcpu, struct sys_reg_params *p)
+{
+	return forward_traps(vcpu, HCR_NV1);
+}
+

Shall we move this helper to emulate-nested.c?

Quoted 52 lines
arch/arm64/kvm/sys_regs.c +20 -0
+static bool access_vbar_el1(struct kvm_vcpu *vcpu,
+			    struct sys_reg_params *p,
+			    const struct sys_reg_desc *r)
+{
+	if (forward_nv1_traps(vcpu, p))
+		return false;
+
+	return access_rw(vcpu, p, r);
+}
+
  static bool access_sctlr_el2(struct kvm_vcpu *vcpu,
  			     struct sys_reg_params *p,
  			     const struct sys_reg_desc *r)
@@ -1682,6 +1698,7 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu,
  	return true;
  }
 
+
  static bool access_elr(struct kvm_vcpu *vcpu,
  		       struct sys_reg_params *p,
  		       const struct sys_reg_desc *r)
@@ -1689,6 +1706,9 @@ static bool access_elr(struct kvm_vcpu *vcpu,
  	if (el12_reg(p) && forward_nv_traps(vcpu))
  		return false;
 
+	if (!el12_reg(p) && forward_nv1_traps(vcpu, p))
+		return false;
+
  	if (p->is_write)
  		vcpu_write_sys_reg(vcpu, p->regval, ELR_EL1);
  	else
@@ -1704,6 +1724,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu,
  	if (el12_reg(p) && forward_nv_traps(vcpu))
  		return false;
 
+	if (!el12_reg(p) && forward_nv1_traps(vcpu, p))
+		return false;
+
  	if (p->is_write)
  		__vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval;
  	else
@@ -1719,6 +1742,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu,
  	if (el12_reg(p) && forward_nv_traps(vcpu))
  		return false;
 
+	if (!el12_reg(p) && forward_nv1_traps(vcpu, p))
+		return false;
+
  	if (p->is_write)
  		vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2);
  	else

arch/arm64/kvm/sys_regs.c static const struct sys_reg_desc sys_reg_descs[] = { · line 1959

Quoted 8 lines
arch/arm64/kvm/sys_regs.c +1 -1
@@ -1927,7 +1953,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {
  	{ SYS_DESC(SYS_LORC_EL1), trap_loregion },
  	{ SYS_DESC(SYS_LORID_EL1), trap_loregion },
 
-	{ SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 },
+	{ SYS_DESC(SYS_VBAR_EL1), access_vbar_el1, reset_val, VBAR_EL1, 0 },
  	{ SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 },
 
  	{ SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only },

Thanks,
Ganapat

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

author maintainer
#83 lore raw

Re: [PATCH v5 31/69] KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting

Quoted 38 lines · Ganapatrao Kulkarni
Quoted 34 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

Forward ELR_EL1, SPSR_EL1 and VBAR_EL1 traps to the virtual EL2 if the virtual HCR_EL2.NV bit is set.

This is for recursive nested virtualization.

---
  arch/arm64/include/asm/kvm_arm.h |  1 +
  arch/arm64/kvm/sys_regs.c        | 28 +++++++++++++++++++++++++++-
  2 files changed, 28 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_arm.h +1 -0
index 9759bc893a51..68af5509e4b0 100644
@@ -20,6 +20,7 @@
  #define HCR_AMVOFFEN	(UL(1) << 51)
  #define HCR_FIEN	(UL(1) << 47)
  #define HCR_FWB		(UL(1) << 46)
+#define HCR_NV1		(UL(1) << 43)
  #define HCR_NV		(UL(1) << 42)
  #define HCR_API		(UL(1) << 41)
  #define HCR_APK		(UL(1) << 40)
arch/arm64/kvm/sys_regs.c +0 -0
index e96877fc3b2a..511e06b6f603 100644
@@ -288,6 +288,22 @@ static bool access_rw(struct kvm_vcpu *vcpu,
  	return true;
  }
  +/* This function is to support the recursive nested
Quoted 8 lines

arch/arm64/kvm/sys_regs.c static bool access_rw(struct kvm_vcpu *vcpu, · line 296

Quoted 6 lines

virtualization */

arch/arm64/kvm/sys_regs.c +5 -0
+static bool forward_nv1_traps(struct kvm_vcpu *vcpu, struct sys_reg_params *p)
+{
+	return forward_traps(vcpu, HCR_NV1);
+}
+

Shall we move this helper to emulate-nested.c?

Sure, that shouldn't be a problem.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
author maintainer
#84 lore raw

PATCH v5 32/69 KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

With HCR_EL2.NV bit set, accesses to EL12 registers in the virtual EL2 trap to EL2. Handle those traps just like we do for EL1 registers.

One exception is CNTKCTL_EL12. We don't trap on CNTKCTL_EL1 for non-VHE virtual EL2 because we don't have to. However, accessing CNTKCTL_EL12 will trap since it's one of the EL12 registers controlled by HCR_EL2.NV bit. Therefore, add a handler for it and don't treat it as a non-trap-registers when preparing a shadow context.

---
 arch/arm64/kvm/sys_regs.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
arch/arm64/kvm/sys_regs.c +17 -0
index 511e06b6f603..692cade54caf 100644
@@ -2214,6 +2214,23 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 },
 	{ SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 },
 
+	{ SYS_DESC(SYS_SCTLR_EL12), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 },
+	{ SYS_DESC(SYS_CPACR_EL12), access_rw, reset_val, CPACR_EL1, 0 },
+	{ SYS_DESC(SYS_TTBR0_EL12), access_vm_reg, reset_unknown, TTBR0_EL1 },
+	{ SYS_DESC(SYS_TTBR1_EL12), access_vm_reg, reset_unknown, TTBR1_EL1 },
+	{ SYS_DESC(SYS_TCR_EL12), access_vm_reg, reset_val, TCR_EL1, 0 },
+	{ SYS_DESC(SYS_SPSR_EL12), access_spsr},
+	{ SYS_DESC(SYS_ELR_EL12), access_elr},
+	{ SYS_DESC(SYS_AFSR0_EL12), access_vm_reg, reset_unknown, AFSR0_EL1 },
+	{ SYS_DESC(SYS_AFSR1_EL12), access_vm_reg, reset_unknown, AFSR1_EL1 },
+	{ SYS_DESC(SYS_ESR_EL12), access_vm_reg, reset_unknown, ESR_EL1 },
+	{ SYS_DESC(SYS_FAR_EL12), access_vm_reg, reset_unknown, FAR_EL1 },
+	{ SYS_DESC(SYS_MAIR_EL12), access_vm_reg, reset_unknown, MAIR_EL1 },
+	{ SYS_DESC(SYS_AMAIR_EL12), access_vm_reg, reset_amair_el1, AMAIR_EL1 },
+	{ SYS_DESC(SYS_VBAR_EL12), access_rw, reset_val, VBAR_EL1, 0 },
+	{ SYS_DESC(SYS_CONTEXTIDR_EL12), access_vm_reg, reset_val, CONTEXTIDR_EL1, 0 },
+	{ SYS_DESC(SYS_CNTKCTL_EL12), access_rw, reset_val, CNTKCTL_EL1, 0 },
+
 	{ SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 },
 };
 
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#85 lore raw

PATCH v5 33/69 KVM: arm64: nv: Forward debug traps to the nested guest

In reply to Marc Zyngier

On handling a debug trap, check whether we need to forward it to the guest before handling it.

---
 arch/arm64/include/asm/kvm_nested.h | 2 ++
 arch/arm64/kvm/emulate-nested.c     | 9 +++++++--
 arch/arm64/kvm/sys_regs.c           | 3 +++
 3 files changed, 12 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_nested.h +2 -0
index 26cba7b4d743..07c15f51cf86 100644
@@ -62,6 +62,8 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl)
 }
 
 int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe);
+extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
+			    u64 control_bit);
 extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
 extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
 
arch/arm64/kvm/emulate-nested.c +7 -2
index 8c7f2fe24bc6..5fbf8978acf2 100644
@@ -25,14 +25,14 @@
 
 #include "trace.h"
 
-bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit)
+bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit)
 {
 	bool control_bit_set;
 
 	if (!nested_virt_in_use(vcpu))
 		return false;
 
-	control_bit_set = __vcpu_sys_reg(vcpu, HCR_EL2) & control_bit;
+	control_bit_set = __vcpu_sys_reg(vcpu, reg) & control_bit;
 	if (!vcpu_mode_el2(vcpu) && control_bit_set) {
 		kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu));
 		return true;
@@ -40,6 +40,11 @@ bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit)
 	return false;
 }
 
+bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit)
+{
+	return __forward_traps(vcpu, HCR_EL2, control_bit);
+}
+
 bool forward_nv_traps(struct kvm_vcpu *vcpu)
 {
 	return forward_traps(vcpu, HCR_NV);
arch/arm64/kvm/sys_regs.c +3 -0
index 692cade54caf..9deedd5a058f 100644
@@ -603,6 +603,9 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu,
 			    struct sys_reg_params *p,
 			    const struct sys_reg_desc *r)
 {
+	if (__forward_traps(vcpu, MDCR_EL2, MDCR_EL2_TDA | MDCR_EL2_TDE))
+		return false;
+
 	access_rw(vcpu, p, r);
 	if (p->is_write)
 		vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY;
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#86 lore raw

PATCH v5 34/69 KVM: arm64: nv: Configure HCR_EL2 for nested virtualization

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

We enable nested virtualization by setting the HCR NV and NV1 bit.

When the virtual E2H bit is set, we can support EL2 register accesses via EL1 registers from the virtual EL2 by doing trap-and-emulate. A better alternative, however, is to allow the virtual EL2 to access EL2 register states without trap. This can be easily achieved by not traping EL1 registers since those registers already have EL2 register states.

---
 arch/arm64/include/asm/kvm_arm.h |  1 +
 arch/arm64/kvm/hyp/vhe/switch.c  | 38 +++++++++++++++++++++++++++++---
 2 files changed, 36 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_arm.h +1 -0
index 68af5509e4b0..b8a0d410035b 100644
@@ -87,6 +87,7 @@
 			 HCR_BSU_IS | HCR_FB | HCR_TACR | \
 			 HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \
 			 HCR_FMO | HCR_IMO | HCR_PTW )
+#define HCR_GUEST_NV_FILTER_FLAGS (HCR_ATA | HCR_API | HCR_APK | HCR_RW)
 #define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF)
 #define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA)
 #define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC)
arch/arm64/kvm/hyp/vhe/switch.c +35 -3
index 57f43e607819..da80c969e623 100644
@@ -36,9 +36,41 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
 	u64 hcr = vcpu->arch.hcr_el2;
 	u64 val;
 
-	/* Trap VM sysreg accesses if an EL2 guest is not using VHE. */
-	if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu))
-		hcr |= HCR_TVM | HCR_TRVM;
+	if (is_hyp_ctxt(vcpu)) {
+		hcr |= HCR_NV;
+
+		if (!vcpu_el2_e2h_is_set(vcpu)) {
+			/*
+			 * For a guest hypervisor on v8.0, trap and emulate
+			 * the EL1 virtual memory control register accesses.
+			 */
+			hcr |= HCR_TVM | HCR_TRVM | HCR_NV1;
+		} else {
+			/*
+			 * For a guest hypervisor on v8.1 (VHE), allow to
+			 * access the EL1 virtual memory control registers
+			 * natively. These accesses are to access EL2 register
+			 * states.
+			 * Note that we still need to respect the virtual
+			 * HCR_EL2 state.
+			 */
+			u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
+
+			vhcr_el2 &= ~HCR_GUEST_NV_FILTER_FLAGS;
+
+			/*
+			 * We already set TVM to handle set/way cache maint
+			 * ops traps, this somewhat collides with the nested
+			 * virt trapping for nVHE. So turn this off for now
+			 * here, in the hope that VHE guests won't ever do this.
+			 * TODO: find out whether it's worth to support both
+			 * cases at the same time.
+			 */
+			hcr &= ~HCR_TVM;
+
+			hcr |= vhcr_el2 & (HCR_TVM | HCR_TRVM);
+		}
+	}
 
 	___activate_traps(vcpu, hcr);
 
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#87 lore raw

Re: [PATCH v5 34/69] KVM: arm64: nv: Configure HCR_EL2 for nested virtualization

Quoted 51 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

We enable nested virtualization by setting the HCR NV and NV1 bit.

When the virtual E2H bit is set, we can support EL2 register accesses via EL1 registers from the virtual EL2 by doing trap-and-emulate. A better alternative, however, is to allow the virtual EL2 to access EL2 register states without trap. This can be easily achieved by not traping EL1 registers since those registers already have EL2 register states.

---
  arch/arm64/include/asm/kvm_arm.h |  1 +
  arch/arm64/kvm/hyp/vhe/switch.c  | 38 +++++++++++++++++++++++++++++---
  2 files changed, 36 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_arm.h +1 -0
index 68af5509e4b0..b8a0d410035b 100644
@@ -87,6 +87,7 @@
  			 HCR_BSU_IS | HCR_FB | HCR_TACR | \
  			 HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \
  			 HCR_FMO | HCR_IMO | HCR_PTW )
+#define HCR_GUEST_NV_FILTER_FLAGS (HCR_ATA | HCR_API | HCR_APK | HCR_RW)
  #define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF)
  #define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA)
  #define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC)
arch/arm64/kvm/hyp/vhe/switch.c +11 -3
index 57f43e607819..da80c969e623 100644
@@ -36,9 +36,41 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
  	u64 hcr = vcpu->arch.hcr_el2;
  	u64 val;
 
-	/* Trap VM sysreg accesses if an EL2 guest is not using VHE. */
-	if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu))
-		hcr |= HCR_TVM | HCR_TRVM;
+	if (is_hyp_ctxt(vcpu)) {
+		hcr |= HCR_NV;
+
+		if (!vcpu_el2_e2h_is_set(vcpu)) {
+			/*
+			 * For a guest hypervisor on v8.0, trap and emulate
+			 * the EL1 virtual memory control register accesses.
+			 */
+			hcr |= HCR_TVM | HCR_TRVM | HCR_NV1;
+		} else {
+			/*

arch/arm64/kvm/hyp/vhe/switch.c static void __activate_traps(struct kvm_vcpu *vcpu) · line 59

Quoted 10 lines
arch/arm64/kvm/hyp/vhe/switch.c +10 -0
+			 * For a guest hypervisor on v8.1 (VHE), allow to
+			 * access the EL1 virtual memory control registers
+			 * natively. These accesses are to access EL2 register
+			 * states.
+			 * Note that we still need to respect the virtual
+			 * HCR_EL2 state.
+			 */
+			u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
+
+			vhcr_el2 &= ~HCR_GUEST_NV_FILTER_FLAGS;

Why HCR_RW is cleared here, May I know please?

Quoted 7 lines
arch/arm64/kvm/hyp/vhe/switch.c +7 -0
+
+			/*
+			 * We already set TVM to handle set/way cache maint
+			 * ops traps, this somewhat collides with the nested
+			 * virt trapping for nVHE. So turn this off for now
+			 * here, in the hope that VHE guests won't ever do this.
+			 * TODO: find out whether it's worth to support both

arch/arm64/kvm/hyp/vhe/switch.c static void __activate_traps(struct kvm_vcpu *vcpu) · line 76

Quoted 10 lines
arch/arm64/kvm/hyp/vhe/switch.c +7 -0
+			 * cases at the same time.
+			 */
+			hcr &= ~HCR_TVM;
+
+			hcr |= vhcr_el2 & (HCR_TVM | HCR_TRVM);
+		}
+	}
 
  	___activate_traps(vcpu, hcr);
 

Thanks,
Ganapat

author maintainer
#88 lore raw

Re: [PATCH v5 34/69] KVM: arm64: nv: Configure HCR_EL2 for nested virtualization

Quoted 59 lines · Ganapatrao Kulkarni
Quoted 55 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

We enable nested virtualization by setting the HCR NV and NV1 bit.

When the virtual E2H bit is set, we can support EL2 register accesses via EL1 registers from the virtual EL2 by doing trap-and-emulate. A better alternative, however, is to allow the virtual EL2 to access EL2 register states without trap. This can be easily achieved by not traping EL1 registers since those registers already have EL2 register states.

---
  arch/arm64/include/asm/kvm_arm.h |  1 +
  arch/arm64/kvm/hyp/vhe/switch.c  | 38 +++++++++++++++++++++++++++++---
  2 files changed, 36 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_arm.h +1 -0
index 68af5509e4b0..b8a0d410035b 100644
@@ -87,6 +87,7 @@
  			 HCR_BSU_IS | HCR_FB | HCR_TACR | \
  			 HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \
  			 HCR_FMO | HCR_IMO | HCR_PTW )
+#define HCR_GUEST_NV_FILTER_FLAGS (HCR_ATA | HCR_API | HCR_APK | HCR_RW)
  #define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF)
  #define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA)
  #define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC)
arch/arm64/kvm/hyp/vhe/switch.c +0 -0
index 57f43e607819..da80c969e623 100644
@@ -36,9 +36,41 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
  	u64 hcr = vcpu->arch.hcr_el2;
  	u64 val;
  -	/* Trap VM sysreg accesses if an EL2 guest is not using

VHE. */

arch/arm64/kvm/hyp/vhe/switch.c +15 -2
-	if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu))
-		hcr |= HCR_TVM | HCR_TRVM;
+	if (is_hyp_ctxt(vcpu)) {
+		hcr |= HCR_NV;
+
+		if (!vcpu_el2_e2h_is_set(vcpu)) {
+			/*
+			 * For a guest hypervisor on v8.0, trap and emulate
+			 * the EL1 virtual memory control register accesses.
+			 */
+			hcr |= HCR_TVM | HCR_TRVM | HCR_NV1;
+		} else {
+			/*
+			 * For a guest hypervisor on v8.1 (VHE), allow to
+			 * access the EL1 virtual memory control registers
+			 * natively. These accesses are to access EL2 register
+			 * states.
Quoted 8 lines

arch/arm64/kvm/hyp/vhe/switch.c static void __activate_traps(struct kvm_vcpu *vcpu) · line 59

Quoted 6 lines
arch/arm64/kvm/hyp/vhe/switch.c +6 -0
+			 * Note that we still need to respect the virtual
+			 * HCR_EL2 state.
+			 */
+			u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
+
+			vhcr_el2 &= ~HCR_GUEST_NV_FILTER_FLAGS;

Why HCR_RW is cleared here, May I know please?

Good question. That's clearly a leftover from an early rework. It really doesn't matter, as we are merging the guest's configuration into the host's, and the host already has HCR_EL2.RW set.

What HCR_GUEST_NV_FILTER_FLAGS should contain is only the bits we don't want to deal with at this stage of the NV support. I'll fix that for the next round.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#89 lore raw

Re: [PATCH v5 34/69] KVM: arm64: nv: Configure HCR_EL2 for nested virtualization

Quoted 70 lines · Marc Zyngier
Quoted 59 lines · Ganapatrao Kulkarni
Quoted 55 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

We enable nested virtualization by setting the HCR NV and NV1 bit.

When the virtual E2H bit is set, we can support EL2 register accesses via EL1 registers from the virtual EL2 by doing trap-and-emulate. A better alternative, however, is to allow the virtual EL2 to access EL2 register states without trap. This can be easily achieved by not traping EL1 registers since those registers already have EL2 register states.

---
   arch/arm64/include/asm/kvm_arm.h |  1 +
   arch/arm64/kvm/hyp/vhe/switch.c  | 38 +++++++++++++++++++++++++++++---
   2 files changed, 36 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_arm.h +1 -0
index 68af5509e4b0..b8a0d410035b 100644
@@ -87,6 +87,7 @@
   			 HCR_BSU_IS | HCR_FB | HCR_TACR | \
   			 HCR_AMO | HCR_SWIO | HCR_TIDCP | HCR_RW | HCR_TLOR | \
   			 HCR_FMO | HCR_IMO | HCR_PTW )
+#define HCR_GUEST_NV_FILTER_FLAGS (HCR_ATA | HCR_API | HCR_APK | HCR_RW)
   #define HCR_VIRT_EXCP_MASK (HCR_VSE | HCR_VI | HCR_VF)
   #define HCR_HOST_NVHE_FLAGS (HCR_RW | HCR_API | HCR_APK | HCR_ATA)
   #define HCR_HOST_NVHE_PROTECTED_FLAGS (HCR_HOST_NVHE_FLAGS | HCR_TSC)
arch/arm64/kvm/hyp/vhe/switch.c +0 -0
index 57f43e607819..da80c969e623 100644
@@ -36,9 +36,41 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
   	u64 hcr = vcpu->arch.hcr_el2;
   	u64 val;
   -	/* Trap VM sysreg accesses if an EL2 guest is not using

VHE. */

arch/arm64/kvm/hyp/vhe/switch.c +15 -2
-	if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu))
-		hcr |= HCR_TVM | HCR_TRVM;
+	if (is_hyp_ctxt(vcpu)) {
+		hcr |= HCR_NV;
+
+		if (!vcpu_el2_e2h_is_set(vcpu)) {
+			/*
+			 * For a guest hypervisor on v8.0, trap and emulate
+			 * the EL1 virtual memory control register accesses.
+			 */
+			hcr |= HCR_TVM | HCR_TRVM | HCR_NV1;
+		} else {
+			/*
+			 * For a guest hypervisor on v8.1 (VHE), allow to
+			 * access the EL1 virtual memory control registers
+			 * natively. These accesses are to access EL2 register
+			 * states.
Quoted 8 lines

arch/arm64/kvm/hyp/vhe/switch.c static void __activate_traps(struct kvm_vcpu *vcpu) · line 59

Quoted 6 lines
arch/arm64/kvm/hyp/vhe/switch.c +6 -0
+			 * Note that we still need to respect the virtual
+			 * HCR_EL2 state.
+			 */
+			u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
+
+			vhcr_el2 &= ~HCR_GUEST_NV_FILTER_FLAGS;

Why HCR_RW is cleared here, May I know please?

Quoted 3 lines

Good question. That's clearly a leftover from an early rework. It really doesn't matter, as we are merging the guest's configuration into the host's, and the host already has HCR_EL2.RW set.

Thanks, I too felt the same.

x>

Quoted 4 lines

What HCR_GUEST_NV_FILTER_FLAGS should contain is only the bits we don't want to deal with at this stage of the NV support. I'll fix that for the next round.

sure, thanks.

Quoted 4 lines

Thanks,

	M.

Thanks,
Ganapat

author maintainer
#90 lore raw

PATCH v5 35/69 KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

So far we were flushing almost the entire universe whenever a VM would load/unload the SCTLR_EL1 and the two versions of that register had different MMU enabled settings. This turned out to be so slow that it prevented forward progress for a nested VM, because a scheduler timer tick interrupt would always be pending when we reached the nested VM.

To avoid this problem, we consider the SCTLR_EL2 when evaluating if caches are on or off when entering virtual EL2 (because this is the value that we end up shadowing onto the hardware EL1 register).

---
 arch/arm64/include/asm/kvm_mmu.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_mmu.h +5 -1
index 02d378887743..c018c7b40761 100644
@@ -115,6 +115,7 @@ alternative_cb_end
 #include <asm/cache.h>
 #include <asm/cacheflush.h>
 #include <asm/mmu_context.h>
+#include <asm/kvm_emulate.h>
 
 void kvm_update_va_mask(struct alt_instr *alt,
 			__le32 *origptr, __le32 *updptr, int nr_inst);
@@ -185,7 +186,10 @@ struct kvm;
 
 static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu)
 {
-	return (vcpu_read_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101;
+	if (vcpu_mode_el2(vcpu))
+		return (__vcpu_sys_reg(vcpu, SCTLR_EL2) & 0b101) == 0b101;
+	else
+		return (vcpu_read_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101;
 }
 
 static inline void __clean_dcache_guest_page(void *va, size_t size)
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#91 lore raw

PATCH v5 36/69 KVM: arm64: nv: Filter out unsupported features from ID regs

In reply to Marc Zyngier

As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest.

Whilst we are at it, avertize ARMv8.4-TTL as well as ARMv8.5-GTG.

---
 arch/arm64/include/asm/kvm_nested.h |   6 ++
 arch/arm64/kvm/nested.c             | 152 ++++++++++++++++++++++++++++
 arch/arm64/kvm/sys_regs.c           |   4 +-
 arch/arm64/kvm/sys_regs.h           |   2 +
 4 files changed, 163 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_nested.h +6 -0
index 07c15f51cf86..026ddaad972c 100644
@@ -67,4 +67,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
 extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
 extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
 
+struct sys_reg_params;
+struct sys_reg_desc;
+
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r);
+
 #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +152 -0

161 lines not on this page · raw message

arch/arm64/kvm/sys_regs.c +3 -1
index 9deedd5a058f..19b33ccb61b8 100644
@@ -1431,8 +1431,10 @@ static bool access_id_reg(struct kvm_vcpu *vcpu,
 			  const struct sys_reg_desc *r)
 {
 	bool raz = sysreg_visible_as_raz(vcpu, r);
+	bool ret = __access_id_reg(vcpu, p, r, raz);
 
-	return __access_id_reg(vcpu, p, r, raz);
+	access_nested_id_reg(vcpu, p, r);
+	return ret;
 }
 
 static bool access_raz_id_reg(struct kvm_vcpu *vcpu,
arch/arm64/kvm/sys_regs.h +2 -0
index cc0cc95a0280..d260c26b1834 100644
@@ -201,4 +201,6 @@ const struct sys_reg_desc *find_reg_by_id(u64 id,
 	CRn(sys_reg_CRn(reg)), CRm(sys_reg_CRm(reg)),	\
 	Op2(sys_reg_Op2(reg))
 
+#define FEATURE(x)	(GENMASK_ULL(x##_SHIFT + 3, x##_SHIFT))
+
 #endif /* __ARM64_KVM_SYS_REGS_LOCAL_H__ */
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
#92 lore raw

Re: [PATCH v5 36/69] KVM: arm64: nv: Filter out unsupported features from ID regs

Hi Marc,

Quoted 117 lines · Marc Zyngier

As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest.

Whilst we are at it, avertize ARMv8.4-TTL as well as ARMv8.5-GTG.

---
  arch/arm64/include/asm/kvm_nested.h |   6 ++
  arch/arm64/kvm/nested.c             | 152 ++++++++++++++++++++++++++++
  arch/arm64/kvm/sys_regs.c           |   4 +-
  arch/arm64/kvm/sys_regs.h           |   2 +
  4 files changed, 163 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_nested.h +6 -0
index 07c15f51cf86..026ddaad972c 100644
@@ -67,4 +67,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
  extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
  extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
 
+struct sys_reg_params;
+struct sys_reg_desc;
+
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r);
+
  #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +72 -0
index 42a96c8d2adc..19b674983e13 100644
@@ -20,6 +20,10 @@
  #include <linux/kvm_host.h>
 
  #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+#include <asm/sysreg.h>
+
+#include "sys_regs.h"
 
  /*
   * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and
@@ -38,3 +42,151 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe)
 
  	return -EINVAL;
  }
+
+/*
+ * Our emulated CPU doesn't support all the possible features. For the
+ * sake of simplicity (and probably mental sanity), wipe out a number
+ * of feature bits we don't intend to support for the time being.
+ * This list should get updated as new features get added to the NV
+ * support, and new extension to the architecture.
+ */
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
+			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
+	u64 val, tmp;
+
+	if (!nested_virt_in_use(v))
+		return;
+
+	val = p->regval;
+
+	switch (id) {
+	case SYS_ID_AA64ISAR0_EL1:
+		/* Support everything but O.S. and Range TLBIs */
+		val &= ~(FEATURE(ID_AA64ISAR0_TLB)	|
+			 GENMASK_ULL(27, 24)		|
+			 GENMASK_ULL(3, 0));
+		break;
+
+	case SYS_ID_AA64ISAR1_EL1:
+		/* Support everything but PtrAuth and Spec Invalidation */
+		val &= ~(GENMASK_ULL(63, 56)		|
+			 FEATURE(ID_AA64ISAR1_SPECRES)	|
+			 FEATURE(ID_AA64ISAR1_GPI)	|
+			 FEATURE(ID_AA64ISAR1_GPA)	|
+			 FEATURE(ID_AA64ISAR1_API)	|
+			 FEATURE(ID_AA64ISAR1_APA));
+		break;
+
+	case SYS_ID_AA64PFR0_EL1:
+		/* No AMU, MPAM, S-EL2, RAS or SVE */
+		val &= ~(GENMASK_ULL(55, 52)		|
+			 FEATURE(ID_AA64PFR0_AMU)	|
+			 FEATURE(ID_AA64PFR0_MPAM)	|
+			 FEATURE(ID_AA64PFR0_SEL2)	|
+			 FEATURE(ID_AA64PFR0_RAS)	|
+			 FEATURE(ID_AA64PFR0_SVE)	|
+			 FEATURE(ID_AA64PFR0_EL3)	|
+			 FEATURE(ID_AA64PFR0_EL2));
+		/* 64bit EL2/EL3 only */
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL2), 0b0001);
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL3), 0b0001);
+		break;
+
+	case SYS_ID_AA64PFR1_EL1:
+		/* Only support SSBS */
+		val &= FEATURE(ID_AA64PFR1_SSBS);
+		break;
+
+	case SYS_ID_AA64MMFR0_EL1:
+		/* Hide ECV, FGT, ExS, Secure Memory */
+		val &= ~(GENMASK_ULL(63, 43)			|
+			 FEATURE(ID_AA64MMFR0_TGRAN4_2)		|
+			 FEATURE(ID_AA64MMFR0_TGRAN16_2)	|
+			 FEATURE(ID_AA64MMFR0_TGRAN64_2)	|
+			 FEATURE(ID_AA64MMFR0_SNSMEM));
+
+		/* Disallow unsupported S2 page sizes */
+		switch (PAGE_SIZE) {

arch/arm64/kvm/nested.c int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) · line 122

Quoted 10 lines
arch/arm64/kvm/nested.c +10 -0
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0001);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0001);
+			fallthrough;
+		case SZ_4K:
+			/* Support everything */
+			break;
+		}

It seems to me that Host hypervisor(L0) has to boot with 4KB page size to support all (4, 16 and 64KB) page sizes at L1, any specific reason for this restriction?

arch/arm64/kvm/nested.c int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) · line 135

Quoted 13 lines
arch/arm64/kvm/nested.c +13 -0
+		/* Advertize supported S2 page sizes */
+		switch (PAGE_SIZE) {
+		case SZ_4K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0010);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0010);
+			fallthrough;
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN64_2), 0b0010);
+			break;
+		}
+		/* Cap PARange to 40bits */

Any specific reasons for the 40 bit cap?

Quoted 78 lines
arch/arm64/kvm/nested.c +57 -0
+		tmp = FIELD_GET(FEATURE(ID_AA64MMFR0_PARANGE), val);
+		if (tmp > 0b0010) {
+			val &= ~FEATURE(ID_AA64MMFR0_PARANGE);
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_PARANGE), 0b0010);
+		}
+		break;
+
+	case SYS_ID_AA64MMFR1_EL1:
+		val &= (FEATURE(ID_AA64MMFR1_PAN)	|
+			FEATURE(ID_AA64MMFR1_LOR)	|
+			FEATURE(ID_AA64MMFR1_HPD)	|
+			FEATURE(ID_AA64MMFR1_VHE)	|
+			FEATURE(ID_AA64MMFR1_VMIDBITS));
+		break;
+
+	case SYS_ID_AA64MMFR2_EL1:
+		val &= ~(FEATURE(ID_AA64MMFR2_EVT)	|
+			 FEATURE(ID_AA64MMFR2_BBM)	|
+			 FEATURE(ID_AA64MMFR2_TTL)	|
+			 GENMASK_ULL(47, 44)		|
+			 FEATURE(ID_AA64MMFR2_ST)	|
+			 FEATURE(ID_AA64MMFR2_CCIDX)	|
+			 FEATURE(ID_AA64MMFR2_LVA));
+
+		/* Force TTL support */
+		val |= FIELD_PREP(FEATURE(ID_AA64MMFR2_TTL), 0b0001);
+		break;
+
+	case SYS_ID_AA64DFR0_EL1:
+		/* Only limited support for PMU, Debug, BPs and WPs */
+		val &= (FEATURE(ID_AA64DFR0_PMSVER)	|
+			FEATURE(ID_AA64DFR0_WRPS)	|
+			FEATURE(ID_AA64DFR0_BRPS)	|
+			FEATURE(ID_AA64DFR0_DEBUGVER));
+
+		/* Cap PMU to ARMv8.1 */
+		tmp = FIELD_GET(FEATURE(ID_AA64DFR0_PMUVER), val);
+		if (tmp > 0b0100) {
+			val &= ~FEATURE(ID_AA64DFR0_PMUVER);
+			val |= FIELD_PREP(FEATURE(ID_AA64DFR0_PMUVER), 0b0100);
+		}
+		/* Cap Debug to ARMv8.1 */
+		tmp = FIELD_GET(FEATURE(ID_AA64DFR0_DEBUGVER), val);
+		if (tmp > 0b0111) {
+			val &= ~FEATURE(ID_AA64DFR0_DEBUGVER);
+			val |= FIELD_PREP(FEATURE(ID_AA64DFR0_DEBUGVER), 0b0111);
+		}
+		break;
+
+	default:
+		/* Unknown register, just wipe it clean */
+		val = 0;
+		break;
+	}
+
+	p->regval = val;
+}
arch/arm64/kvm/sys_regs.c +3 -1
index 9deedd5a058f..19b33ccb61b8 100644
@@ -1431,8 +1431,10 @@ static bool access_id_reg(struct kvm_vcpu *vcpu,
  			  const struct sys_reg_desc *r)
  {
  	bool raz = sysreg_visible_as_raz(vcpu, r);
+	bool ret = __access_id_reg(vcpu, p, r, raz);
 
-	return __access_id_reg(vcpu, p, r, raz);
+	access_nested_id_reg(vcpu, p, r);
+	return ret;
  }
 
  static bool access_raz_id_reg(struct kvm_vcpu *vcpu,
arch/arm64/kvm/sys_regs.h +0 -0
index cc0cc95a0280..d260c26b1834 100644

arch/arm64/kvm/sys_regs.h const struct sys_reg_desc *find_reg_by_id(u64 id, · line 206

Quoted 6 lines
arch/arm64/kvm/sys_regs.h +2 -0
@@ -201,4 +201,6 @@ const struct sys_reg_desc *find_reg_by_id(u64 id,
  	CRn(sys_reg_CRn(reg)), CRm(sys_reg_CRm(reg)),	\
  	Op2(sys_reg_Op2(reg))
 
+#define FEATURE(x)	(GENMASK_ULL(x##_SHIFT + 3, x##_SHIFT))
+
  #endif /* __ARM64_KVM_SYS_REGS_LOCAL_H__ */

Thanks,
Ganapat

author maintainer
#93 lore raw

Re: [PATCH v5 36/69] KVM: arm64: nv: Filter out unsupported features from ID regs

Quoted 124 lines · Ganapatrao Kulkarni

Hi Marc,

Quoted 119 lines · Marc Zyngier

As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest.

Whilst we are at it, avertize ARMv8.4-TTL as well as ARMv8.5-GTG.

---
  arch/arm64/include/asm/kvm_nested.h |   6 ++
  arch/arm64/kvm/nested.c             | 152 ++++++++++++++++++++++++++++
  arch/arm64/kvm/sys_regs.c           |   4 +-
  arch/arm64/kvm/sys_regs.h           |   2 +
  4 files changed, 163 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_nested.h +5 -0
index 07c15f51cf86..026ddaad972c 100644
@@ -67,4 +67,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
  extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
  extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
  +struct sys_reg_params;
+struct sys_reg_desc;
+
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r);
+
  #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +78 -0
index 42a96c8d2adc..19b674983e13 100644
@@ -20,6 +20,10 @@
  #include <linux/kvm_host.h>
    #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+#include <asm/sysreg.h>
+
+#include "sys_regs.h"
    /*
   * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and
@@ -38,3 +42,151 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe)
    	return -EINVAL;
  }
+
+/*
+ * Our emulated CPU doesn't support all the possible features. For the
+ * sake of simplicity (and probably mental sanity), wipe out a number
+ * of feature bits we don't intend to support for the time being.
+ * This list should get updated as new features get added to the NV
+ * support, and new extension to the architecture.
+ */
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
+			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
+	u64 val, tmp;
+
+	if (!nested_virt_in_use(v))
+		return;
+
+	val = p->regval;
+
+	switch (id) {
+	case SYS_ID_AA64ISAR0_EL1:
+		/* Support everything but O.S. and Range TLBIs */
+		val &= ~(FEATURE(ID_AA64ISAR0_TLB)	|
+			 GENMASK_ULL(27, 24)		|
+			 GENMASK_ULL(3, 0));
+		break;
+
+	case SYS_ID_AA64ISAR1_EL1:
+		/* Support everything but PtrAuth and Spec Invalidation */
+		val &= ~(GENMASK_ULL(63, 56)		|
+			 FEATURE(ID_AA64ISAR1_SPECRES)	|
+			 FEATURE(ID_AA64ISAR1_GPI)	|
+			 FEATURE(ID_AA64ISAR1_GPA)	|
+			 FEATURE(ID_AA64ISAR1_API)	|
+			 FEATURE(ID_AA64ISAR1_APA));
+		break;
+
+	case SYS_ID_AA64PFR0_EL1:
+		/* No AMU, MPAM, S-EL2, RAS or SVE */
+		val &= ~(GENMASK_ULL(55, 52)		|
+			 FEATURE(ID_AA64PFR0_AMU)	|
+			 FEATURE(ID_AA64PFR0_MPAM)	|
+			 FEATURE(ID_AA64PFR0_SEL2)	|
+			 FEATURE(ID_AA64PFR0_RAS)	|
+			 FEATURE(ID_AA64PFR0_SVE)	|
+			 FEATURE(ID_AA64PFR0_EL3)	|
+			 FEATURE(ID_AA64PFR0_EL2));
+		/* 64bit EL2/EL3 only */
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL2), 0b0001);
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL3), 0b0001);
+		break;
+
+	case SYS_ID_AA64PFR1_EL1:
+		/* Only support SSBS */
+		val &= FEATURE(ID_AA64PFR1_SSBS);
+		break;
+
+	case SYS_ID_AA64MMFR0_EL1:
+		/* Hide ECV, FGT, ExS, Secure Memory */
+		val &= ~(GENMASK_ULL(63, 43)			|
+			 FEATURE(ID_AA64MMFR0_TGRAN4_2)		|
+			 FEATURE(ID_AA64MMFR0_TGRAN16_2)	|
+			 FEATURE(ID_AA64MMFR0_TGRAN64_2)	|
+			 FEATURE(ID_AA64MMFR0_SNSMEM));
+
+		/* Disallow unsupported S2 page sizes */
+		switch (PAGE_SIZE) {
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0001);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0001);
+			fallthrough;
Quoted 8 lines

arch/arm64/kvm/nested.c int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) · line 122

Quoted 4 lines
arch/arm64/kvm/nested.c +4 -0
+		case SZ_4K:
+			/* Support everything */
+			break;
+		}

It seems to me that Host hypervisor(L0) has to boot with 4KB page size to support all (4, 16 and 64KB) page sizes at L1, any specific reason for this restriction?

Well, yes.

If you have a L0 that has booted with (let's say) 64kB page size, how do you provide S2 mappings with 4kB granularity so that you can implement the permissions that a L1 guest hypervisor can impose on its own guest, given that KVM currently mandates S1 and S2 to use the same page sizes?

You can't. That's why we tell the guest hypervisor how much we support, and the guest hypervisor can decide to go ahead or not depending on what it does.

If one day we can support S2 mappings that are smaller than the host page sizes, then we'll be able to allow to advertise all page sizes. But I wouldn't hold my breath for this to happen.

Quoted 16 lines
Quoted 4 lines
arch/arm64/kvm/nested.c +3 -0
+		/* Advertize supported S2 page sizes */
+		switch (PAGE_SIZE) {
+		case SZ_4K:

arch/arm64/kvm/nested.c int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) · line 135

Quoted 10 lines
arch/arm64/kvm/nested.c +10 -0
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0010);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0010);
+			fallthrough;
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN64_2), 0b0010);
+			break;
+		}
+		/* Cap PARange to 40bits */

Any specific reasons for the 40 bit cap?

The only platform this currently runs on is a model, and 1TB of address space is what it supports. At some point, this will require userspace involvement to set it up, but we're not quite ready for that either. And given that there is no HW, the urge for changing this is extremely limited.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#94 lore raw

Re: [PATCH v5 36/69] KVM: arm64: nv: Filter out unsupported features from ID regs

Quoted 143 lines · Marc Zyngier
Quoted 124 lines · Ganapatrao Kulkarni

Hi Marc,

Quoted 119 lines · Marc Zyngier

As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest.

Whilst we are at it, avertize ARMv8.4-TTL as well as ARMv8.5-GTG.

---
   arch/arm64/include/asm/kvm_nested.h |   6 ++
   arch/arm64/kvm/nested.c             | 152 ++++++++++++++++++++++++++++
   arch/arm64/kvm/sys_regs.c           |   4 +-
   arch/arm64/kvm/sys_regs.h           |   2 +
   4 files changed, 163 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_nested.h +5 -0
index 07c15f51cf86..026ddaad972c 100644
@@ -67,4 +67,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
   extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
   extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
   +struct sys_reg_params;
+struct sys_reg_desc;
+
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r);
+
   #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +78 -0
index 42a96c8d2adc..19b674983e13 100644
@@ -20,6 +20,10 @@
   #include <linux/kvm_host.h>
     #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+#include <asm/sysreg.h>
+
+#include "sys_regs.h"
     /*
    * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and
@@ -38,3 +42,151 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe)
     	return -EINVAL;
   }
+
+/*
+ * Our emulated CPU doesn't support all the possible features. For the
+ * sake of simplicity (and probably mental sanity), wipe out a number
+ * of feature bits we don't intend to support for the time being.
+ * This list should get updated as new features get added to the NV
+ * support, and new extension to the architecture.
+ */
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
+			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
+	u64 val, tmp;
+
+	if (!nested_virt_in_use(v))
+		return;
+
+	val = p->regval;
+
+	switch (id) {
+	case SYS_ID_AA64ISAR0_EL1:
+		/* Support everything but O.S. and Range TLBIs */
+		val &= ~(FEATURE(ID_AA64ISAR0_TLB)	|
+			 GENMASK_ULL(27, 24)		|
+			 GENMASK_ULL(3, 0));
+		break;
+
+	case SYS_ID_AA64ISAR1_EL1:
+		/* Support everything but PtrAuth and Spec Invalidation */
+		val &= ~(GENMASK_ULL(63, 56)		|
+			 FEATURE(ID_AA64ISAR1_SPECRES)	|
+			 FEATURE(ID_AA64ISAR1_GPI)	|
+			 FEATURE(ID_AA64ISAR1_GPA)	|
+			 FEATURE(ID_AA64ISAR1_API)	|
+			 FEATURE(ID_AA64ISAR1_APA));
+		break;
+
+	case SYS_ID_AA64PFR0_EL1:
+		/* No AMU, MPAM, S-EL2, RAS or SVE */
+		val &= ~(GENMASK_ULL(55, 52)		|
+			 FEATURE(ID_AA64PFR0_AMU)	|
+			 FEATURE(ID_AA64PFR0_MPAM)	|
+			 FEATURE(ID_AA64PFR0_SEL2)	|
+			 FEATURE(ID_AA64PFR0_RAS)	|
+			 FEATURE(ID_AA64PFR0_SVE)	|
+			 FEATURE(ID_AA64PFR0_EL3)	|
+			 FEATURE(ID_AA64PFR0_EL2));
+		/* 64bit EL2/EL3 only */
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL2), 0b0001);
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL3), 0b0001);
+		break;
+
+	case SYS_ID_AA64PFR1_EL1:
+		/* Only support SSBS */
+		val &= FEATURE(ID_AA64PFR1_SSBS);
+		break;
+
+	case SYS_ID_AA64MMFR0_EL1:
+		/* Hide ECV, FGT, ExS, Secure Memory */
+		val &= ~(GENMASK_ULL(63, 43)			|
+			 FEATURE(ID_AA64MMFR0_TGRAN4_2)		|
+			 FEATURE(ID_AA64MMFR0_TGRAN16_2)	|
+			 FEATURE(ID_AA64MMFR0_TGRAN64_2)	|
+			 FEATURE(ID_AA64MMFR0_SNSMEM));
+
+		/* Disallow unsupported S2 page sizes */
+		switch (PAGE_SIZE) {
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0001);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0001);
+			fallthrough;
Quoted 8 lines

arch/arm64/kvm/nested.c int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) · line 122

Quoted 4 lines
arch/arm64/kvm/nested.c +4 -0
+		case SZ_4K:
+			/* Support everything */
+			break;
+		}

It seems to me that Host hypervisor(L0) has to boot with 4KB page size to support all (4, 16 and 64KB) page sizes at L1, any specific reason for this restriction?

Well, yes.

If you have a L0 that has booted with (let's say) 64kB page size, how do you provide S2 mappings with 4kB granularity so that you can implement the permissions that a L1 guest hypervisor can impose on its own guest, given that KVM currently mandates S1 and S2 to use the same page sizes?

Quoted 7 lines

You can't. That's why we tell the guest hypervisor how much we support, and the guest hypervisor can decide to go ahead or not depending on what it does.

If one day we can support S2 mappings that are smaller than the host page sizes, then we'll be able to allow to advertise all page sizes. But I wouldn't hold my breath for this to happen.

Thanks for the detailed explanation!.
Can we put one line comment that explains why this manipulation? It would be helpful to see a comment like S2 PAGE_SIZE should be at-least the size of Host PAGE_SIZE?

Quoted 18 lines
Quoted 17 lines
Quoted 5 lines
arch/arm64/kvm/nested.c +3 -0
+		/* Advertize supported S2 page sizes */
+		switch (PAGE_SIZE) {
+		case SZ_4K:

arch/arm64/kvm/nested.c int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) · line 135

Quoted 10 lines
arch/arm64/kvm/nested.c +10 -0
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0010);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0010);
+			fallthrough;
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN64_2), 0b0010);
+			break;
+		}
+		/* Cap PARange to 40bits */

Any specific reasons for the 40 bit cap?

Quoted 5 lines

The only platform this currently runs on is a model, and 1TB of address space is what it supports. At some point, this will require userspace involvement to set it up, but we're not quite ready for that either. And given that there is no HW, the urge for changing this is extremely limited.

Makes sense, thanks.

Please feel free to add.

Quoted 3 lines
	M.

Thanks,
Ganapat

author maintainer
#95 lore raw

Re: [PATCH v5 36/69] KVM: arm64: nv: Filter out unsupported features from ID regs

Quoted 151 lines · Ganapatrao Kulkarni
Quoted 147 lines · Marc Zyngier
Quoted 124 lines · Ganapatrao Kulkarni

Hi Marc,

Quoted 119 lines · Marc Zyngier

As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest.

Whilst we are at it, avertize ARMv8.4-TTL as well as ARMv8.5-GTG.

---
   arch/arm64/include/asm/kvm_nested.h |   6 ++
   arch/arm64/kvm/nested.c             | 152 ++++++++++++++++++++++++++++
   arch/arm64/kvm/sys_regs.c           |   4 +-
   arch/arm64/kvm/sys_regs.h           |   2 +
   4 files changed, 163 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_nested.h +5 -0
index 07c15f51cf86..026ddaad972c 100644
@@ -67,4 +67,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
   extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
   extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
   +struct sys_reg_params;
+struct sys_reg_desc;
+
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r);
+
   #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +78 -0
index 42a96c8d2adc..19b674983e13 100644
@@ -20,6 +20,10 @@
   #include <linux/kvm_host.h>
     #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+#include <asm/sysreg.h>
+
+#include "sys_regs.h"
     /*
    * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and
@@ -38,3 +42,151 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe)
     	return -EINVAL;
   }
+
+/*
+ * Our emulated CPU doesn't support all the possible features. For the
+ * sake of simplicity (and probably mental sanity), wipe out a number
+ * of feature bits we don't intend to support for the time being.
+ * This list should get updated as new features get added to the NV
+ * support, and new extension to the architecture.
+ */
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
+			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
+	u64 val, tmp;
+
+	if (!nested_virt_in_use(v))
+		return;
+
+	val = p->regval;
+
+	switch (id) {
+	case SYS_ID_AA64ISAR0_EL1:
+		/* Support everything but O.S. and Range TLBIs */
+		val &= ~(FEATURE(ID_AA64ISAR0_TLB)	|
+			 GENMASK_ULL(27, 24)		|
+			 GENMASK_ULL(3, 0));
+		break;
+
+	case SYS_ID_AA64ISAR1_EL1:
+		/* Support everything but PtrAuth and Spec Invalidation */
+		val &= ~(GENMASK_ULL(63, 56)		|
+			 FEATURE(ID_AA64ISAR1_SPECRES)	|
+			 FEATURE(ID_AA64ISAR1_GPI)	|
+			 FEATURE(ID_AA64ISAR1_GPA)	|
+			 FEATURE(ID_AA64ISAR1_API)	|
+			 FEATURE(ID_AA64ISAR1_APA));
+		break;
+
+	case SYS_ID_AA64PFR0_EL1:
+		/* No AMU, MPAM, S-EL2, RAS or SVE */
+		val &= ~(GENMASK_ULL(55, 52)		|
+			 FEATURE(ID_AA64PFR0_AMU)	|
+			 FEATURE(ID_AA64PFR0_MPAM)	|
+			 FEATURE(ID_AA64PFR0_SEL2)	|
+			 FEATURE(ID_AA64PFR0_RAS)	|
+			 FEATURE(ID_AA64PFR0_SVE)	|
+			 FEATURE(ID_AA64PFR0_EL3)	|
+			 FEATURE(ID_AA64PFR0_EL2));
+		/* 64bit EL2/EL3 only */
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL2), 0b0001);
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL3), 0b0001);
+		break;
+
+	case SYS_ID_AA64PFR1_EL1:
+		/* Only support SSBS */
+		val &= FEATURE(ID_AA64PFR1_SSBS);
+		break;
+
+	case SYS_ID_AA64MMFR0_EL1:
+		/* Hide ECV, FGT, ExS, Secure Memory */
+		val &= ~(GENMASK_ULL(63, 43)			|
+			 FEATURE(ID_AA64MMFR0_TGRAN4_2)		|
+			 FEATURE(ID_AA64MMFR0_TGRAN16_2)	|
+			 FEATURE(ID_AA64MMFR0_TGRAN64_2)	|
+			 FEATURE(ID_AA64MMFR0_SNSMEM));
+
+		/* Disallow unsupported S2 page sizes */
+		switch (PAGE_SIZE) {
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0001);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0001);
+			fallthrough;
Quoted 8 lines

arch/arm64/kvm/nested.c int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) · line 121

Quoted 4 lines
arch/arm64/kvm/nested.c +4 -0
+		case SZ_4K:
+			/* Support everything */
+			break;
+		}

It seems to me that Host hypervisor(L0) has to boot with 4KB page size to support all (4, 16 and 64KB) page sizes at L1, any specific reason for this restriction?

Well, yes.

If you have a L0 that has booted with (let's say) 64kB page size, how do you provide S2 mappings with 4kB granularity so that you can implement the permissions that a L1 guest hypervisor can impose on its own guest, given that KVM currently mandates S1 and S2 to use the same page sizes?

You can't. That's why we tell the guest hypervisor how much we support, and the guest hypervisor can decide to go ahead or not depending on what it does.

Quoted 8 lines
Quoted 3 lines

If one day we can support S2 mappings that are smaller than the host page sizes, then we'll be able to allow to advertise all page sizes. But I wouldn't hold my breath for this to happen.

Thanks for the detailed explanation!.
Can we put one line comment that explains why this manipulation? It would be helpful to see a comment like S2 PAGE_SIZE should be at-least the size of Host PAGE_SIZE?

Can do, but we need to get the terminology straight, because this is very quickly becoming confusing. Something like:

arch/arm64/kvm/nested.c +6 -1
index 7c9dd1edf011..d35a947f5679 100644
@@ -850,7 +850,12 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
 			/* Support everything */
 			break;
 		}
-		/* Advertize supported S2 page sizes */
+		/*
+		 * Since we can't support a guest S2 page size smaller than
+		 * the host's own page size (due to KVM only populating its
+		 * own S2 using the kernel's page size), advertise the
+		 * limitation using FEAT_GTG.
+		 */
 		switch (PAGE_SIZE) {
 		case SZ_4K:
 			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0010);

So not quite a one line comment! ;-)

Ultimately, all there is to know is in the description of FEAT_GTG in the ARMv8 ARM.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#96 lore raw

Re: [PATCH v5 36/69] KVM: arm64: nv: Filter out unsupported features from ID regs

Quoted 170 lines · Marc Zyngier
Quoted 151 lines · Ganapatrao Kulkarni
Quoted 147 lines · Marc Zyngier
Quoted 124 lines · Ganapatrao Kulkarni

Hi Marc,

Quoted 119 lines · Marc Zyngier

As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest.

Whilst we are at it, avertize ARMv8.4-TTL as well as ARMv8.5-GTG.

---
    arch/arm64/include/asm/kvm_nested.h |   6 ++
    arch/arm64/kvm/nested.c             | 152 ++++++++++++++++++++++++++++
    arch/arm64/kvm/sys_regs.c           |   4 +-
    arch/arm64/kvm/sys_regs.h           |   2 +
    4 files changed, 163 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_nested.h +5 -0
index 07c15f51cf86..026ddaad972c 100644
@@ -67,4 +67,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
    extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
    extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
    +struct sys_reg_params;
+struct sys_reg_desc;
+
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r);
+
    #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +78 -0
index 42a96c8d2adc..19b674983e13 100644
@@ -20,6 +20,10 @@
    #include <linux/kvm_host.h>
      #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+#include <asm/sysreg.h>
+
+#include "sys_regs.h"
      /*
     * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and
@@ -38,3 +42,151 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe)
      	return -EINVAL;
    }
+
+/*
+ * Our emulated CPU doesn't support all the possible features. For the
+ * sake of simplicity (and probably mental sanity), wipe out a number
+ * of feature bits we don't intend to support for the time being.
+ * This list should get updated as new features get added to the NV
+ * support, and new extension to the architecture.
+ */
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
+			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
+	u64 val, tmp;
+
+	if (!nested_virt_in_use(v))
+		return;
+
+	val = p->regval;
+
+	switch (id) {
+	case SYS_ID_AA64ISAR0_EL1:
+		/* Support everything but O.S. and Range TLBIs */
+		val &= ~(FEATURE(ID_AA64ISAR0_TLB)	|
+			 GENMASK_ULL(27, 24)		|
+			 GENMASK_ULL(3, 0));
+		break;
+
+	case SYS_ID_AA64ISAR1_EL1:
+		/* Support everything but PtrAuth and Spec Invalidation */
+		val &= ~(GENMASK_ULL(63, 56)		|
+			 FEATURE(ID_AA64ISAR1_SPECRES)	|
+			 FEATURE(ID_AA64ISAR1_GPI)	|
+			 FEATURE(ID_AA64ISAR1_GPA)	|
+			 FEATURE(ID_AA64ISAR1_API)	|
+			 FEATURE(ID_AA64ISAR1_APA));
+		break;
+
+	case SYS_ID_AA64PFR0_EL1:
+		/* No AMU, MPAM, S-EL2, RAS or SVE */
+		val &= ~(GENMASK_ULL(55, 52)		|
+			 FEATURE(ID_AA64PFR0_AMU)	|
+			 FEATURE(ID_AA64PFR0_MPAM)	|
+			 FEATURE(ID_AA64PFR0_SEL2)	|
+			 FEATURE(ID_AA64PFR0_RAS)	|
+			 FEATURE(ID_AA64PFR0_SVE)	|
+			 FEATURE(ID_AA64PFR0_EL3)	|
+			 FEATURE(ID_AA64PFR0_EL2));
+		/* 64bit EL2/EL3 only */
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL2), 0b0001);
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL3), 0b0001);
+		break;
+
+	case SYS_ID_AA64PFR1_EL1:
+		/* Only support SSBS */
+		val &= FEATURE(ID_AA64PFR1_SSBS);
+		break;
+
+	case SYS_ID_AA64MMFR0_EL1:
+		/* Hide ECV, FGT, ExS, Secure Memory */
+		val &= ~(GENMASK_ULL(63, 43)			|
+			 FEATURE(ID_AA64MMFR0_TGRAN4_2)		|
+			 FEATURE(ID_AA64MMFR0_TGRAN16_2)	|
+			 FEATURE(ID_AA64MMFR0_TGRAN64_2)	|
+			 FEATURE(ID_AA64MMFR0_SNSMEM));
+
+		/* Disallow unsupported S2 page sizes */
+		switch (PAGE_SIZE) {
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0001);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0001);
+			fallthrough;
Quoted 8 lines

arch/arm64/kvm/nested.c int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) · line 121

Quoted 4 lines
arch/arm64/kvm/nested.c +4 -0
+		case SZ_4K:
+			/* Support everything */
+			break;
+		}

It seems to me that Host hypervisor(L0) has to boot with 4KB page size to support all (4, 16 and 64KB) page sizes at L1, any specific reason for this restriction?

Well, yes.

If you have a L0 that has booted with (let's say) 64kB page size, how do you provide S2 mappings with 4kB granularity so that you can implement the permissions that a L1 guest hypervisor can impose on its own guest, given that KVM currently mandates S1 and S2 to use the same page sizes?

You can't. That's why we tell the guest hypervisor how much we support, and the guest hypervisor can decide to go ahead or not depending on what it does.

Quoted 8 lines
Quoted 3 lines

If one day we can support S2 mappings that are smaller than the host page sizes, then we'll be able to allow to advertise all page sizes. But I wouldn't hold my breath for this to happen.

Thanks for the detailed explanation!.
Can we put one line comment that explains why this manipulation? It would be helpful to see a comment like S2 PAGE_SIZE should be at-least the size of Host PAGE_SIZE?

Can do, but we need to get the terminology straight, because this is very quickly becoming confusing. Something like:

arch/arm64/kvm/nested.c +0 -0
index 7c9dd1edf011..d35a947f5679 100644

arch/arm64/kvm/nested.c void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, · line 857

Quoted 9 lines
arch/arm64/kvm/nested.c +5 -1
@@ -850,7 +850,12 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
  			/* Support everything */
  			break;
  		}
-		/* Advertize supported S2 page sizes */
+		/*
+		 * Since we can't support a guest S2 page size smaller than
+		 * the host's own page size (due to KVM only populating its
+		 * own S2 using the kernel's page size), advertise the
+		 * limitation using FEAT_GTG.

Thanks this helps!.

Quoted 6 lines
arch/arm64/kvm/nested.c +1 -0
+		 */
  		switch (PAGE_SIZE) {
  		case SZ_4K:
  			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0010);

So not quite a one line comment! ;-)

Indeed, thanks.

Quoted 8 lines

Ultimately, all there is to know is in the description of FEAT_GTG in the ARMv8 ARM.

Thanks,

	M.

Thanks,
Ganapat

#97 lore raw

Re: [PATCH v5 36/69] KVM: arm64: nv: Filter out unsupported features from ID regs

In reply to Marc Zyngier

Hi Marc,

Quoted 170 lines · Marc Zyngier
Quoted 151 lines · Ganapatrao Kulkarni
Quoted 147 lines · Marc Zyngier
Quoted 124 lines · Ganapatrao Kulkarni

Hi Marc,

Quoted 119 lines · Marc Zyngier

As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest.

Whilst we are at it, avertize ARMv8.4-TTL as well as ARMv8.5-GTG.

---
    arch/arm64/include/asm/kvm_nested.h |   6 ++
    arch/arm64/kvm/nested.c             | 152 ++++++++++++++++++++++++++++
    arch/arm64/kvm/sys_regs.c           |   4 +-
    arch/arm64/kvm/sys_regs.h           |   2 +
    4 files changed, 163 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_nested.h +5 -0
index 07c15f51cf86..026ddaad972c 100644
@@ -67,4 +67,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
    extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
    extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
    +struct sys_reg_params;
+struct sys_reg_desc;
+
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r);
+
    #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +78 -0
index 42a96c8d2adc..19b674983e13 100644
@@ -20,6 +20,10 @@
    #include <linux/kvm_host.h>
      #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+#include <asm/sysreg.h>
+
+#include "sys_regs.h"
      /*
     * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and
@@ -38,3 +42,151 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe)
      	return -EINVAL;
    }
+
+/*
+ * Our emulated CPU doesn't support all the possible features. For the
+ * sake of simplicity (and probably mental sanity), wipe out a number
+ * of feature bits we don't intend to support for the time being.
+ * This list should get updated as new features get added to the NV
+ * support, and new extension to the architecture.
+ */
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
+			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
+	u64 val, tmp;
+
+	if (!nested_virt_in_use(v))
+		return;
+
+	val = p->regval;
+
+	switch (id) {
+	case SYS_ID_AA64ISAR0_EL1:
+		/* Support everything but O.S. and Range TLBIs */
+		val &= ~(FEATURE(ID_AA64ISAR0_TLB)	|
+			 GENMASK_ULL(27, 24)		|
+			 GENMASK_ULL(3, 0));
+		break;
+
+	case SYS_ID_AA64ISAR1_EL1:
+		/* Support everything but PtrAuth and Spec Invalidation */
+		val &= ~(GENMASK_ULL(63, 56)		|
+			 FEATURE(ID_AA64ISAR1_SPECRES)	|
+			 FEATURE(ID_AA64ISAR1_GPI)	|
+			 FEATURE(ID_AA64ISAR1_GPA)	|
+			 FEATURE(ID_AA64ISAR1_API)	|
+			 FEATURE(ID_AA64ISAR1_APA));
+		break;
+
+	case SYS_ID_AA64PFR0_EL1:
+		/* No AMU, MPAM, S-EL2, RAS or SVE */
+		val &= ~(GENMASK_ULL(55, 52)		|
+			 FEATURE(ID_AA64PFR0_AMU)	|
+			 FEATURE(ID_AA64PFR0_MPAM)	|
+			 FEATURE(ID_AA64PFR0_SEL2)	|
+			 FEATURE(ID_AA64PFR0_RAS)	|
+			 FEATURE(ID_AA64PFR0_SVE)	|
+			 FEATURE(ID_AA64PFR0_EL3)	|
+			 FEATURE(ID_AA64PFR0_EL2));
+		/* 64bit EL2/EL3 only */
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL2), 0b0001);
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL3), 0b0001);
+		break;
+
+	case SYS_ID_AA64PFR1_EL1:
+		/* Only support SSBS */
+		val &= FEATURE(ID_AA64PFR1_SSBS);
+		break;
+
+	case SYS_ID_AA64MMFR0_EL1:
+		/* Hide ECV, FGT, ExS, Secure Memory */
+		val &= ~(GENMASK_ULL(63, 43)			|
+			 FEATURE(ID_AA64MMFR0_TGRAN4_2)		|
+			 FEATURE(ID_AA64MMFR0_TGRAN16_2)	|
+			 FEATURE(ID_AA64MMFR0_TGRAN64_2)	|
+			 FEATURE(ID_AA64MMFR0_SNSMEM));
+
+		/* Disallow unsupported S2 page sizes */
+		switch (PAGE_SIZE) {
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0001);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0001);
+			fallthrough;
Quoted 8 lines

arch/arm64/kvm/nested.c int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) · line 121

Quoted 4 lines
arch/arm64/kvm/nested.c +4 -0
+		case SZ_4K:
+			/* Support everything */
+			break;
+		}

It seems to me that Host hypervisor(L0) has to boot with 4KB page size to support all (4, 16 and 64KB) page sizes at L1, any specific reason for this restriction?

Well, yes.

If you have a L0 that has booted with (let's say) 64kB page size, how do you provide S2 mappings with 4kB granularity so that you can implement the permissions that a L1 guest hypervisor can impose on its own guest, given that KVM currently mandates S1 and S2 to use the same page sizes?

You can't. That's why we tell the guest hypervisor how much we support, and the guest hypervisor can decide to go ahead or not depending on what it does.

Quoted 8 lines
Quoted 3 lines

If one day we can support S2 mappings that are smaller than the host page sizes, then we'll be able to allow to advertise all page sizes. But I wouldn't hold my breath for this to happen.

Thanks for the detailed explanation!.
Can we put one line comment that explains why this manipulation? It would be helpful to see a comment like S2 PAGE_SIZE should be at-least the size of Host PAGE_SIZE?

Can do, but we need to get the terminology straight, because this is very quickly becoming confusing. Something like:

arch/arm64/kvm/nested.c +0 -0
index 7c9dd1edf011..d35a947f5679 100644

arch/arm64/kvm/nested.c void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, · line 858

Quoted 10 lines
arch/arm64/kvm/nested.c +6 -1
@@ -850,7 +850,12 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
  			/* Support everything */
  			break;
  		}
-		/* Advertize supported S2 page sizes */
+		/*
+		 * Since we can't support a guest S2 page size smaller than
+		 * the host's own page size (due to KVM only populating its
+		 * own S2 using the kernel's page size), advertise the
+		 * limitation using FEAT_GTG.
+		 */

I have tried booting L0 with 4K page-size and L1 with 64K and with this config the L2/NestedVM boot hangs. I have tried L2 with page-sizes 4K and 64K(though S1 page size of L2 should not matter?).

Quoted 4 lines
arch/arm64/kvm/nested.c +0 -0
  		switch (PAGE_SIZE) {
  		case SZ_4K:
  			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0010);
Quoted 9 lines

So not quite a one line comment! ;-)

Ultimately, all there is to know is in the description of FEAT_GTG in the ARMv8 ARM.

Thanks,

	M.

Thanks,
Ganapat

author maintainer
#98 lore raw

Re: [PATCH v5 36/69] KVM: arm64: nv: Filter out unsupported features from ID regs

Quoted 180 lines · Ganapatrao Kulkarni

Hi Marc,

Quoted 176 lines · Marc Zyngier
Quoted 151 lines · Ganapatrao Kulkarni
Quoted 147 lines · Marc Zyngier
Quoted 124 lines · Ganapatrao Kulkarni

Hi Marc,

Quoted 119 lines · Marc Zyngier

As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest.

Whilst we are at it, avertize ARMv8.4-TTL as well as ARMv8.5-GTG.

---
    arch/arm64/include/asm/kvm_nested.h |   6 ++
    arch/arm64/kvm/nested.c             | 152 ++++++++++++++++++++++++++++
    arch/arm64/kvm/sys_regs.c           |   4 +-
    arch/arm64/kvm/sys_regs.h           |   2 +
    4 files changed, 163 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_nested.h +5 -0
index 07c15f51cf86..026ddaad972c 100644
@@ -67,4 +67,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
    extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
    extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
    +struct sys_reg_params;
+struct sys_reg_desc;
+
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r);
+
    #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +78 -0
index 42a96c8d2adc..19b674983e13 100644
@@ -20,6 +20,10 @@
    #include <linux/kvm_host.h>
      #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+#include <asm/sysreg.h>
+
+#include "sys_regs.h"
      /*
     * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and
@@ -38,3 +42,151 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe)
      	return -EINVAL;
    }
+
+/*
+ * Our emulated CPU doesn't support all the possible features. For the
+ * sake of simplicity (and probably mental sanity), wipe out a number
+ * of feature bits we don't intend to support for the time being.
+ * This list should get updated as new features get added to the NV
+ * support, and new extension to the architecture.
+ */
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
+			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
+	u64 val, tmp;
+
+	if (!nested_virt_in_use(v))
+		return;
+
+	val = p->regval;
+
+	switch (id) {
+	case SYS_ID_AA64ISAR0_EL1:
+		/* Support everything but O.S. and Range TLBIs */
+		val &= ~(FEATURE(ID_AA64ISAR0_TLB)	|
+			 GENMASK_ULL(27, 24)		|
+			 GENMASK_ULL(3, 0));
+		break;
+
+	case SYS_ID_AA64ISAR1_EL1:
+		/* Support everything but PtrAuth and Spec Invalidation */
+		val &= ~(GENMASK_ULL(63, 56)		|
+			 FEATURE(ID_AA64ISAR1_SPECRES)	|
+			 FEATURE(ID_AA64ISAR1_GPI)	|
+			 FEATURE(ID_AA64ISAR1_GPA)	|
+			 FEATURE(ID_AA64ISAR1_API)	|
+			 FEATURE(ID_AA64ISAR1_APA));
+		break;
+
+	case SYS_ID_AA64PFR0_EL1:
+		/* No AMU, MPAM, S-EL2, RAS or SVE */
+		val &= ~(GENMASK_ULL(55, 52)		|
+			 FEATURE(ID_AA64PFR0_AMU)	|
+			 FEATURE(ID_AA64PFR0_MPAM)	|
+			 FEATURE(ID_AA64PFR0_SEL2)	|
+			 FEATURE(ID_AA64PFR0_RAS)	|
+			 FEATURE(ID_AA64PFR0_SVE)	|
+			 FEATURE(ID_AA64PFR0_EL3)	|
+			 FEATURE(ID_AA64PFR0_EL2));
+		/* 64bit EL2/EL3 only */
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL2), 0b0001);
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL3), 0b0001);
+		break;
+
+	case SYS_ID_AA64PFR1_EL1:
+		/* Only support SSBS */
+		val &= FEATURE(ID_AA64PFR1_SSBS);
+		break;
+
+	case SYS_ID_AA64MMFR0_EL1:
+		/* Hide ECV, FGT, ExS, Secure Memory */
+		val &= ~(GENMASK_ULL(63, 43)			|
+			 FEATURE(ID_AA64MMFR0_TGRAN4_2)		|
+			 FEATURE(ID_AA64MMFR0_TGRAN16_2)	|
+			 FEATURE(ID_AA64MMFR0_TGRAN64_2)	|
+			 FEATURE(ID_AA64MMFR0_SNSMEM));
+
+		/* Disallow unsupported S2 page sizes */
+		switch (PAGE_SIZE) {
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0001);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0001);
+			fallthrough;
Quoted 8 lines

arch/arm64/kvm/nested.c int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) · line 121

Quoted 4 lines
arch/arm64/kvm/nested.c +4 -0
+		case SZ_4K:
+			/* Support everything */
+			break;
+		}

It seems to me that Host hypervisor(L0) has to boot with 4KB page size to support all (4, 16 and 64KB) page sizes at L1, any specific reason for this restriction?

Well, yes.

If you have a L0 that has booted with (let's say) 64kB page size, how do you provide S2 mappings with 4kB granularity so that you can implement the permissions that a L1 guest hypervisor can impose on its own guest, given that KVM currently mandates S1 and S2 to use the same page sizes?

You can't. That's why we tell the guest hypervisor how much we support, and the guest hypervisor can decide to go ahead or not depending on what it does.

Quoted 8 lines
Quoted 3 lines

If one day we can support S2 mappings that are smaller than the host page sizes, then we'll be able to allow to advertise all page sizes. But I wouldn't hold my breath for this to happen.

Thanks for the detailed explanation!.
Can we put one line comment that explains why this manipulation? It would be helpful to see a comment like S2 PAGE_SIZE should be at-least the size of Host PAGE_SIZE?

Can do, but we need to get the terminology straight, because this is very quickly becoming confusing. Something like:

arch/arm64/kvm/nested.c +2 -1
index 7c9dd1edf011..d35a947f5679 100644
@@ -850,7 +850,12 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
  			/* Support everything */
  			break;
  		}
-		/* Advertize supported S2 page sizes */
+		/*
+		 * Since we can't support a guest S2 page size smaller than
Quoted 8 lines

arch/arm64/kvm/nested.c void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, · line 858

Quoted 4 lines
arch/arm64/kvm/nested.c +4 -0
+		 * the host's own page size (due to KVM only populating its
+		 * own S2 using the kernel's page size), advertise the
+		 * limitation using FEAT_GTG.
+		 */

I have tried booting L0 with 4K page-size and L1 with 64K and with this config the L2/NestedVM boot hangs. I have tried L2 with page-sizes 4K and 64K(though S1 page size of L2 should not matter?).

S1 shouldn't matter, but if that's what you are seeing, there is obviously an issue trying to satisfy a translation fault in this configuration, and we get stuck.

Could you please add some tracing and work out whether this is the case? I bet this is an issue trying to combine the L1 S2 translation (which will be 64kB aligned) with the faulting IPA address, but I can't immediately pinpoint the bug.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
#99 lore raw

Re: [PATCH v5 36/69] KVM: arm64: nv: Filter out unsupported features from ID regs

In reply to Marc Zyngier
Quoted 7 lines · Marc Zyngier

As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest.

Whilst we are at it, avertize ARMv8.4-TTL as well as ARMv8.5-GTG.

Typo: advertize.

Quoted 211 lines
---
  arch/arm64/include/asm/kvm_nested.h |   6 ++
  arch/arm64/kvm/nested.c             | 152 ++++++++++++++++++++++++++++
  arch/arm64/kvm/sys_regs.c           |   4 +-
  arch/arm64/kvm/sys_regs.h           |   2 +
  4 files changed, 163 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_nested.h +6 -0
index 07c15f51cf86..026ddaad972c 100644
@@ -67,4 +67,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
  extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit);
  extern bool forward_nv_traps(struct kvm_vcpu *vcpu);
 
+struct sys_reg_params;
+struct sys_reg_desc;
+
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r);
+
  #endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/nested.c +152 -0
index 42a96c8d2adc..19b674983e13 100644
@@ -20,6 +20,10 @@
  #include <linux/kvm_host.h>
 
  #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
+#include <asm/sysreg.h>
+
+#include "sys_regs.h"
 
  /*
   * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and
@@ -38,3 +42,151 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe)
 
  	return -EINVAL;
  }
+
+/*
+ * Our emulated CPU doesn't support all the possible features. For the
+ * sake of simplicity (and probably mental sanity), wipe out a number
+ * of feature bits we don't intend to support for the time being.
+ * This list should get updated as new features get added to the NV
+ * support, and new extension to the architecture.
+ */
+void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p,
+			  const struct sys_reg_desc *r)
+{
+	u32 id = sys_reg((u32)r->Op0, (u32)r->Op1,
+			 (u32)r->CRn, (u32)r->CRm, (u32)r->Op2);
+	u64 val, tmp;
+
+	if (!nested_virt_in_use(v))
+		return;
+
+	val = p->regval;
+
+	switch (id) {
+	case SYS_ID_AA64ISAR0_EL1:
+		/* Support everything but O.S. and Range TLBIs */
+		val &= ~(FEATURE(ID_AA64ISAR0_TLB)	|
+			 GENMASK_ULL(27, 24)		|
+			 GENMASK_ULL(3, 0));
+		break;
+
+	case SYS_ID_AA64ISAR1_EL1:
+		/* Support everything but PtrAuth and Spec Invalidation */
+		val &= ~(GENMASK_ULL(63, 56)		|
+			 FEATURE(ID_AA64ISAR1_SPECRES)	|
+			 FEATURE(ID_AA64ISAR1_GPI)	|
+			 FEATURE(ID_AA64ISAR1_GPA)	|
+			 FEATURE(ID_AA64ISAR1_API)	|
+			 FEATURE(ID_AA64ISAR1_APA));
+		break;
+
+	case SYS_ID_AA64PFR0_EL1:
+		/* No AMU, MPAM, S-EL2, RAS or SVE */
+		val &= ~(GENMASK_ULL(55, 52)		|
+			 FEATURE(ID_AA64PFR0_AMU)	|
+			 FEATURE(ID_AA64PFR0_MPAM)	|
+			 FEATURE(ID_AA64PFR0_SEL2)	|
+			 FEATURE(ID_AA64PFR0_RAS)	|
+			 FEATURE(ID_AA64PFR0_SVE)	|
+			 FEATURE(ID_AA64PFR0_EL3)	|
+			 FEATURE(ID_AA64PFR0_EL2));
+		/* 64bit EL2/EL3 only */
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL2), 0b0001);
+		val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL3), 0b0001);
+		break;
+
+	case SYS_ID_AA64PFR1_EL1:
+		/* Only support SSBS */
+		val &= FEATURE(ID_AA64PFR1_SSBS);
+		break;
+
+	case SYS_ID_AA64MMFR0_EL1:
+		/* Hide ECV, FGT, ExS, Secure Memory */
+		val &= ~(GENMASK_ULL(63, 43)			|
+			 FEATURE(ID_AA64MMFR0_TGRAN4_2)		|
+			 FEATURE(ID_AA64MMFR0_TGRAN16_2)	|
+			 FEATURE(ID_AA64MMFR0_TGRAN64_2)	|
+			 FEATURE(ID_AA64MMFR0_SNSMEM));
+
+		/* Disallow unsupported S2 page sizes */
+		switch (PAGE_SIZE) {
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0001);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0001);
+			fallthrough;
+		case SZ_4K:
+			/* Support everything */
+			break;
+		}
+		/* Advertize supported S2 page sizes */
+		switch (PAGE_SIZE) {
+		case SZ_4K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0010);
+			fallthrough;
+		case SZ_16K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0010);
+			fallthrough;
+		case SZ_64K:
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN64_2), 0b0010);
+			break;
+		}
+		/* Cap PARange to 40bits */
+		tmp = FIELD_GET(FEATURE(ID_AA64MMFR0_PARANGE), val);
+		if (tmp > 0b0010) {
+			val &= ~FEATURE(ID_AA64MMFR0_PARANGE);
+			val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_PARANGE), 0b0010);
+		}
+		break;
+
+	case SYS_ID_AA64MMFR1_EL1:
+		val &= (FEATURE(ID_AA64MMFR1_PAN)	|
+			FEATURE(ID_AA64MMFR1_LOR)	|
+			FEATURE(ID_AA64MMFR1_HPD)	|
+			FEATURE(ID_AA64MMFR1_VHE)	|
+			FEATURE(ID_AA64MMFR1_VMIDBITS));
+		break;
+
+	case SYS_ID_AA64MMFR2_EL1:
+		val &= ~(FEATURE(ID_AA64MMFR2_EVT)	|
+			 FEATURE(ID_AA64MMFR2_BBM)	|
+			 FEATURE(ID_AA64MMFR2_TTL)	|
+			 GENMASK_ULL(47, 44)		|
+			 FEATURE(ID_AA64MMFR2_ST)	|
+			 FEATURE(ID_AA64MMFR2_CCIDX)	|
+			 FEATURE(ID_AA64MMFR2_LVA));
+
+		/* Force TTL support */
+		val |= FIELD_PREP(FEATURE(ID_AA64MMFR2_TTL), 0b0001);
+		break;
+
+	case SYS_ID_AA64DFR0_EL1:
+		/* Only limited support for PMU, Debug, BPs and WPs */
+		val &= (FEATURE(ID_AA64DFR0_PMSVER)	|
+			FEATURE(ID_AA64DFR0_WRPS)	|
+			FEATURE(ID_AA64DFR0_BRPS)	|
+			FEATURE(ID_AA64DFR0_DEBUGVER));
+
+		/* Cap PMU to ARMv8.1 */
+		tmp = FIELD_GET(FEATURE(ID_AA64DFR0_PMUVER), val);
+		if (tmp > 0b0100) {
+			val &= ~FEATURE(ID_AA64DFR0_PMUVER);
+			val |= FIELD_PREP(FEATURE(ID_AA64DFR0_PMUVER), 0b0100);
+		}
+		/* Cap Debug to ARMv8.1 */
+		tmp = FIELD_GET(FEATURE(ID_AA64DFR0_DEBUGVER), val);
+		if (tmp > 0b0111) {
+			val &= ~FEATURE(ID_AA64DFR0_DEBUGVER);
+			val |= FIELD_PREP(FEATURE(ID_AA64DFR0_DEBUGVER), 0b0111);
+		}
+		break;
+
+	default:
+		/* Unknown register, just wipe it clean */
+		val = 0;
+		break;
+	}
+
+	p->regval = val;
+}
arch/arm64/kvm/sys_regs.c +3 -1
index 9deedd5a058f..19b33ccb61b8 100644
@@ -1431,8 +1431,10 @@ static bool access_id_reg(struct kvm_vcpu *vcpu,
  			  const struct sys_reg_desc *r)
  {
  	bool raz = sysreg_visible_as_raz(vcpu, r);
+	bool ret = __access_id_reg(vcpu, p, r, raz);
 
-	return __access_id_reg(vcpu, p, r, raz);
+	access_nested_id_reg(vcpu, p, r);
+	return ret;
  }
 
  static bool access_raz_id_reg(struct kvm_vcpu *vcpu,
arch/arm64/kvm/sys_regs.h +0 -0
index cc0cc95a0280..d260c26b1834 100644

arch/arm64/kvm/sys_regs.h const struct sys_reg_desc *find_reg_by_id(u64 id, · line 206

Quoted 6 lines
arch/arm64/kvm/sys_regs.h +2 -0
@@ -201,4 +201,6 @@ const struct sys_reg_desc *find_reg_by_id(u64 id,
  	CRn(sys_reg_CRn(reg)), CRm(sys_reg_CRm(reg)),	\
  	Op2(sys_reg_Op2(reg))
 
+#define FEATURE(x)	(GENMASK_ULL(x##_SHIFT + 3, x##_SHIFT))
+
  #endif /* __ARM64_KVM_SYS_REGS_LOCAL_H__ */

Thanks,
Ganapat

author maintainer
#100 lore raw

PATCH v5 37/69 KVM: arm64: nv: Hide RAS from nested guests

In reply to Marc Zyngier

We don't want to expose complicated features to guests until we have a good grasp on the basic CPU emulation. So let's pretend that RAS, doesn't exist in a nested guest. We already hide the feature bits, let's now make sure VDISR_EL1 will UNDEF.

---
 arch/arm64/kvm/sys_regs.c | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/kvm/sys_regs.c +1 -0
index 19b33ccb61b8..3922ecc00869 100644
@@ -2212,6 +2212,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_VBAR_EL2), access_rw, reset_val, VBAR_EL2, 0 },
 	{ SYS_DESC(SYS_RVBAR_EL2), access_rw, reset_val, RVBAR_EL2, 0 },
 	{ SYS_DESC(SYS_RMR_EL2), trap_undef },
+	{ SYS_DESC(SYS_VDISR_EL2), trap_undef },
 
 	{ SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 },
 	{ SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 },
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#101 lore raw

PATCH v5 38/69 KVM: arm64: nv: Support multiple nested Stage-2 mmu structures

In reply to Marc Zyngier

Add Stage-2 mmu data structures for virtual EL2 and for nested guests. We don't yet populate shadow Stage-2 page tables, but we now have a framework for getting to a shadow Stage-2 pgd.

We allocate twice the number of vcpus as Stage-2 mmu structures because that's sufficient for each vcpu running two translation regimes without having to flush the Stage-2 page tables.

---
 arch/arm64/include/asm/kvm_host.h   |  29 +++++
 arch/arm64/include/asm/kvm_mmu.h    |   9 ++
 arch/arm64/include/asm/kvm_nested.h |   7 +
 arch/arm64/kvm/arm.c                |  16 ++-
 arch/arm64/kvm/mmu.c                |  29 +++--
 arch/arm64/kvm/nested.c             | 195 ++++++++++++++++++++++++++++
 6 files changed, 275 insertions(+), 10 deletions(-)
arch/arm64/include/asm/kvm_host.h +29 -0
index 6a7b13edc5cb..00c3366129b8 100644
@@ -102,14 +102,43 @@ struct kvm_s2_mmu {
 	int __percpu *last_vcpu_ran;
 
 	struct kvm_arch *arch;
+
+	/*
+	 * For a shadow stage-2 MMU, the virtual vttbr programmed by the guest
+	 * hypervisor.  Unused for kvm_arch->mmu. Set to 1 when the structure
+	 * contains no valid information.
+	 */
+	u64	vttbr;
+
+	/* true when this represents a nested context where virtual HCR_EL2.VM == 1 */
+	bool	nested_stage2_enabled;
+
+	/*
+	 *  0: Nobody is currently using this, check vttbr for validity
+	 * >0: Somebody is actively using this.
+	 */
+	atomic_t refcnt;
 };
 
+static inline bool kvm_s2_mmu_valid(struct kvm_s2_mmu *mmu)
+{
+	return !(mmu->vttbr & 1);
+}
+
 struct kvm_arch_memory_slot {
 };
 
 struct kvm_arch {
 	struct kvm_s2_mmu mmu;
 
+	/*
+	 * Stage 2 paging stage for VMs with nested virtual using a virtual
+	 * VMID.
+	 */
+	struct kvm_s2_mmu *nested_mmus;
+	size_t nested_mmus_size;
+	int nested_mmus_next;
+
 	/* VTCR_EL2 value for this VM */
 	u64    vtcr;
 
arch/arm64/include/asm/kvm_mmu.h +9 -0
index c018c7b40761..7250594e3d68 100644
@@ -116,6 +116,7 @@ alternative_cb_end
 #include <asm/cacheflush.h>
 #include <asm/mmu_context.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
 
 void kvm_update_va_mask(struct alt_instr *alt,
 			__le32 *origptr, __le32 *updptr, int nr_inst);
@@ -159,6 +160,7 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
 			     void **haddr);
 void free_hyp_pgds(void);
 
+void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size);
 void stage2_unmap_vm(struct kvm *kvm);
 int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu);
 void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu);
@@ -294,5 +296,12 @@ static inline struct kvm *kvm_s2_mmu_to_kvm(struct kvm_s2_mmu *mmu)
 {
 	return container_of(mmu->arch, struct kvm, arch);
 }
+
+static inline u64 get_vmid(u64 vttbr)
+{
+	return (vttbr & VTTBR_VMID_MASK(kvm_get_vmid_bits())) >>
+		VTTBR_VMID_SHIFT;
+}
+
 #endif /* __ASSEMBLY__ */
 #endif /* __ARM64_KVM_MMU_H__ */
arch/arm64/include/asm/kvm_nested.h +7 -0
index 026ddaad972c..473ecd1d60d0 100644
@@ -61,6 +61,13 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl)
 		(cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN)));
 }
 
+extern void kvm_init_nested(struct kvm *kvm);
+extern int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu);
+extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu);
+extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr);
+extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu);
+extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu);
+
 int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe);
 extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
 			    u64 control_bit);
arch/arm64/kvm/arm.c +15 -1
index 68da54d58cd0..1dbf63319b99 100644
@@ -37,6 +37,7 @@
 #include <asm/kvm_arm.h>
 #include <asm/kvm_asm.h>
 #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
 #include <asm/kvm_emulate.h>
 #include <asm/sections.h>
 
@@ -146,6 +147,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 	if (ret)
 		return ret;
 
+	kvm_init_nested(kvm);
+
 	ret = create_hyp_mappings(kvm, kvm + 1, PAGE_HYP);
 	if (ret)
 		goto out_free_stage2_pgd;
@@ -389,6 +392,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 	struct kvm_s2_mmu *mmu;
 	int *last_ran;
 
+	if (nested_virt_in_use(vcpu))
+		kvm_vcpu_load_hw_mmu(vcpu);
+
 	mmu = vcpu->arch.hw_mmu;
 	last_ran = this_cpu_ptr(mmu->last_vcpu_ran);
 
@@ -437,6 +443,9 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
 	kvm_vgic_put(vcpu);
 	kvm_vcpu_pmu_restore_host(vcpu);
 
+	if (nested_virt_in_use(vcpu))
+		kvm_vcpu_put_hw_mmu(vcpu);
+
 	vcpu->cpu = -1;
 }
 
@@ -1088,8 +1097,13 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 
 	vcpu->arch.target = phys_target;
 
+	/* Prepare for nested if required */
+	ret = kvm_vcpu_init_nested(vcpu);
+
 	/* Now we know what it is, we can reset it. */
-	ret = kvm_reset_vcpu(vcpu);
+	if (!ret)
+		ret = kvm_reset_vcpu(vcpu);
+
 	if (ret) {
 		vcpu->arch.target = -1;
 		bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
arch/arm64/kvm/mmu.c +20 -9
index 9eec548fccd1..ab1653b8e601 100644
@@ -162,7 +162,7 @@ static void invalidate_icache_guest_page(void *va, size_t size)
  * does.
  */
 /**
- * unmap_stage2_range -- Clear stage2 page table entries to unmap a range
+ * __unmap_stage2_range -- Clear stage2 page table entries to unmap a range
  * @mmu:   The KVM stage-2 MMU pointer
  * @start: The intermediate physical base address of the range to unmap
  * @size:  The size of the area to unmap
@@ -185,7 +185,7 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64
 				   may_block));
 }
 
-static void unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size)
+void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size)
 {
 	__unmap_stage2_range(mmu, start, size, true);
 }
@@ -507,7 +507,20 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
 	int cpu, err;
 	struct kvm_pgtable *pgt;
 
+	/*
+	 * If we already have our page tables in place, and that the
+	 * MMU context is the canonical one, we have a bug somewhere,
+	 * as this is only supposed to ever happen once per VM.
+	 *
+	 * Otherwise, we're building nested page tables, and that's
+	 * probably because userspace called KVM_ARM_VCPU_INIT more
+	 * than once on the same vcpu. Since that's actually legal,
+	 * don't kick a fuss and leave gracefully.
+	 */
 	if (mmu->pgt != NULL) {
+		if (&kvm->arch.mmu != mmu)
+			return 0;
+
 		kvm_err("kvm_arch already initialized?\n");
 		return -EINVAL;
 	}
@@ -533,6 +546,9 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
 	mmu->pgt = pgt;
 	mmu->pgd_phys = __pa(pgt->pgd);
 	WRITE_ONCE(mmu->vmid.vmid_gen, 0);
+
+	kvm_init_nested_s2_mmu(mmu);
+
 	return 0;
 
 out_destroy_pgtable:
@@ -578,7 +594,7 @@ static void stage2_unmap_memslot(struct kvm *kvm,
 
 		if (!(vma->vm_flags & VM_PFNMAP)) {
 			gpa_t gpa = addr + (vm_start - memslot->userspace_addr);
-			unmap_stage2_range(&kvm->arch.mmu, gpa, vm_end - vm_start);
+			kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, vm_end - vm_start);
 		}
 		hva = vm_end;
 	} while (hva < reg_end);
@@ -1556,11 +1572,6 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen)
 {
 }
 
-void kvm_arch_flush_shadow_all(struct kvm *kvm)
-{
-	kvm_free_stage2_pgd(&kvm->arch.mmu);
-}
-
 void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 				   struct kvm_memory_slot *slot)
 {
@@ -1568,7 +1579,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 	phys_addr_t size = slot->npages << PAGE_SHIFT;
 
 	spin_lock(&kvm->mmu_lock);
-	unmap_stage2_range(&kvm->arch.mmu, gpa, size);
+	kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, size);
 	spin_unlock(&kvm->mmu_lock);
 }
 
arch/arm64/kvm/nested.c +195 -0

213 lines not on this page · raw message

Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 38/69
#102 lore raw

Re: [PATCH v5 38/69] KVM: arm64: nv: Support multiple nested Stage-2 mmu structures

Quoted 344 lines · Marc Zyngier

Add Stage-2 mmu data structures for virtual EL2 and for nested guests. We don't yet populate shadow Stage-2 page tables, but we now have a framework for getting to a shadow Stage-2 pgd.

We allocate twice the number of vcpus as Stage-2 mmu structures because that's sufficient for each vcpu running two translation regimes without having to flush the Stage-2 page tables.

---
  arch/arm64/include/asm/kvm_host.h   |  29 +++++
  arch/arm64/include/asm/kvm_mmu.h    |   9 ++
  arch/arm64/include/asm/kvm_nested.h |   7 +
  arch/arm64/kvm/arm.c                |  16 ++-
  arch/arm64/kvm/mmu.c                |  29 +++--
  arch/arm64/kvm/nested.c             | 195 ++++++++++++++++++++++++++++
  6 files changed, 275 insertions(+), 10 deletions(-)
arch/arm64/include/asm/kvm_host.h +29 -0
index 6a7b13edc5cb..00c3366129b8 100644
@@ -102,14 +102,43 @@ struct kvm_s2_mmu {
  	int __percpu *last_vcpu_ran;
 
  	struct kvm_arch *arch;
+
+	/*
+	 * For a shadow stage-2 MMU, the virtual vttbr programmed by the guest
+	 * hypervisor.  Unused for kvm_arch->mmu. Set to 1 when the structure
+	 * contains no valid information.
+	 */
+	u64	vttbr;
+
+	/* true when this represents a nested context where virtual HCR_EL2.VM == 1 */
+	bool	nested_stage2_enabled;
+
+	/*
+	 *  0: Nobody is currently using this, check vttbr for validity
+	 * >0: Somebody is actively using this.
+	 */
+	atomic_t refcnt;
  };
 
+static inline bool kvm_s2_mmu_valid(struct kvm_s2_mmu *mmu)
+{
+	return !(mmu->vttbr & 1);
+}
+
  struct kvm_arch_memory_slot {
  };
 
  struct kvm_arch {
  	struct kvm_s2_mmu mmu;
 
+	/*
+	 * Stage 2 paging stage for VMs with nested virtual using a virtual
+	 * VMID.
+	 */
+	struct kvm_s2_mmu *nested_mmus;
+	size_t nested_mmus_size;
+	int nested_mmus_next;
+
  	/* VTCR_EL2 value for this VM */
  	u64    vtcr;
 
arch/arm64/include/asm/kvm_mmu.h +9 -0
index c018c7b40761..7250594e3d68 100644
@@ -116,6 +116,7 @@ alternative_cb_end
  #include <asm/cacheflush.h>
  #include <asm/mmu_context.h>
  #include <asm/kvm_emulate.h>
+#include <asm/kvm_nested.h>
 
  void kvm_update_va_mask(struct alt_instr *alt,
  			__le32 *origptr, __le32 *updptr, int nr_inst);
@@ -159,6 +160,7 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
  			     void **haddr);
  void free_hyp_pgds(void);
 
+void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size);
  void stage2_unmap_vm(struct kvm *kvm);
  int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu);
  void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu);
@@ -294,5 +296,12 @@ static inline struct kvm *kvm_s2_mmu_to_kvm(struct kvm_s2_mmu *mmu)
  {
  	return container_of(mmu->arch, struct kvm, arch);
  }
+
+static inline u64 get_vmid(u64 vttbr)
+{
+	return (vttbr & VTTBR_VMID_MASK(kvm_get_vmid_bits())) >>
+		VTTBR_VMID_SHIFT;
+}
+
  #endif /* __ASSEMBLY__ */
  #endif /* __ARM64_KVM_MMU_H__ */
arch/arm64/include/asm/kvm_nested.h +7 -0
index 026ddaad972c..473ecd1d60d0 100644
@@ -61,6 +61,13 @@ static inline u64 translate_cnthctl_el2_to_cntkctl_el1(u64 cnthctl)
  		(cnthctl & (CNTHCTL_EVNTI | CNTHCTL_EVNTDIR | CNTHCTL_EVNTEN)));
  }
 
+extern void kvm_init_nested(struct kvm *kvm);
+extern int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu);
+extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu);
+extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr);
+extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu);
+extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu);
+
  int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe);
  extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
  			    u64 control_bit);
arch/arm64/kvm/arm.c +15 -1
index 68da54d58cd0..1dbf63319b99 100644
@@ -37,6 +37,7 @@
  #include <asm/kvm_arm.h>
  #include <asm/kvm_asm.h>
  #include <asm/kvm_mmu.h>
+#include <asm/kvm_nested.h>
  #include <asm/kvm_emulate.h>
  #include <asm/sections.h>
 
@@ -146,6 +147,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
  	if (ret)
  		return ret;
 
+	kvm_init_nested(kvm);
+
  	ret = create_hyp_mappings(kvm, kvm + 1, PAGE_HYP);
  	if (ret)
  		goto out_free_stage2_pgd;
@@ -389,6 +392,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
  	struct kvm_s2_mmu *mmu;
  	int *last_ran;
 
+	if (nested_virt_in_use(vcpu))
+		kvm_vcpu_load_hw_mmu(vcpu);
+
  	mmu = vcpu->arch.hw_mmu;
  	last_ran = this_cpu_ptr(mmu->last_vcpu_ran);
 
@@ -437,6 +443,9 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
  	kvm_vgic_put(vcpu);
  	kvm_vcpu_pmu_restore_host(vcpu);
 
+	if (nested_virt_in_use(vcpu))
+		kvm_vcpu_put_hw_mmu(vcpu);
+
  	vcpu->cpu = -1;
  }
 
@@ -1088,8 +1097,13 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 
  	vcpu->arch.target = phys_target;
 
+	/* Prepare for nested if required */
+	ret = kvm_vcpu_init_nested(vcpu);
+
  	/* Now we know what it is, we can reset it. */
-	ret = kvm_reset_vcpu(vcpu);
+	if (!ret)
+		ret = kvm_reset_vcpu(vcpu);
+
  	if (ret) {
  		vcpu->arch.target = -1;
  		bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
arch/arm64/kvm/mmu.c +20 -9
index 9eec548fccd1..ab1653b8e601 100644
@@ -162,7 +162,7 @@ static void invalidate_icache_guest_page(void *va, size_t size)
   * does.
   */
  /**
- * unmap_stage2_range -- Clear stage2 page table entries to unmap a range
+ * __unmap_stage2_range -- Clear stage2 page table entries to unmap a range
   * @mmu:   The KVM stage-2 MMU pointer
   * @start: The intermediate physical base address of the range to unmap
   * @size:  The size of the area to unmap
@@ -185,7 +185,7 @@ static void __unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64
  				   may_block));
  }
 
-static void unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size)
+void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size)
  {
  	__unmap_stage2_range(mmu, start, size, true);
  }
@@ -507,7 +507,20 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
  	int cpu, err;
  	struct kvm_pgtable *pgt;
 
+	/*
+	 * If we already have our page tables in place, and that the
+	 * MMU context is the canonical one, we have a bug somewhere,
+	 * as this is only supposed to ever happen once per VM.
+	 *
+	 * Otherwise, we're building nested page tables, and that's
+	 * probably because userspace called KVM_ARM_VCPU_INIT more
+	 * than once on the same vcpu. Since that's actually legal,
+	 * don't kick a fuss and leave gracefully.
+	 */
  	if (mmu->pgt != NULL) {
+		if (&kvm->arch.mmu != mmu)
+			return 0;
+
  		kvm_err("kvm_arch already initialized?\n");
  		return -EINVAL;
  	}
@@ -533,6 +546,9 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu)
  	mmu->pgt = pgt;
  	mmu->pgd_phys = __pa(pgt->pgd);
  	WRITE_ONCE(mmu->vmid.vmid_gen, 0);
+
+	kvm_init_nested_s2_mmu(mmu);
+
  	return 0;
 
  out_destroy_pgtable:
@@ -578,7 +594,7 @@ static void stage2_unmap_memslot(struct kvm *kvm,
 
  		if (!(vma->vm_flags & VM_PFNMAP)) {
  			gpa_t gpa = addr + (vm_start - memslot->userspace_addr);
-			unmap_stage2_range(&kvm->arch.mmu, gpa, vm_end - vm_start);
+			kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, vm_end - vm_start);
  		}
  		hva = vm_end;
  	} while (hva < reg_end);
@@ -1556,11 +1572,6 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen)
  {
  }
 
-void kvm_arch_flush_shadow_all(struct kvm *kvm)
-{
-	kvm_free_stage2_pgd(&kvm->arch.mmu);
-}
-
  void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
  				   struct kvm_memory_slot *slot)
  {
@@ -1568,7 +1579,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
  	phys_addr_t size = slot->npages << PAGE_SHIFT;
 
  	spin_lock(&kvm->mmu_lock);
-	unmap_stage2_range(&kvm->arch.mmu, gpa, size);
+	kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, size);
  	spin_unlock(&kvm->mmu_lock);
  }
 
arch/arm64/kvm/nested.c +72 -0
index 19b674983e13..b034a2343374 100644
@@ -19,12 +19,189 @@
  #include <linux/kvm.h>
  #include <linux/kvm_host.h>
 
+#include <asm/kvm_arm.h>
  #include <asm/kvm_emulate.h>
+#include <asm/kvm_mmu.h>
  #include <asm/kvm_nested.h>
  #include <asm/sysreg.h>
 
  #include "sys_regs.h"
 
+void kvm_init_nested(struct kvm *kvm)
+{
+	kvm->arch.nested_mmus = NULL;
+	kvm->arch.nested_mmus_size = 0;
+}
+
+int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_s2_mmu *tmp;
+	int num_mmus;
+	int ret = -ENOMEM;
+
+	if (!test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features))
+		return 0;
+
+	if (!cpus_have_final_cap(ARM64_HAS_NESTED_VIRT))
+		return -EINVAL;
+
+	mutex_lock(&kvm->lock);
+
+	/*
+	 * Let's treat memory allocation failures as benign: If we fail to
+	 * allocate anything, return an error and keep the allocated array
+	 * alive. Userspace may try to recover by intializing the vcpu
+	 * again, and there is no reason to affect the whole VM for this.
+	 */
+	num_mmus = atomic_read(&kvm->online_vcpus) * 2;
+	tmp = krealloc(kvm->arch.nested_mmus,
+		       num_mmus * sizeof(*kvm->arch.nested_mmus),
+		       GFP_KERNEL_ACCOUNT | __GFP_ZERO);
+	if (tmp) {
+		/*
+		 * If we went through a realocation, adjust the MMU
+		 * back-pointers in the previously initialised
+		 * pg_table structures.
+		 */
+		if (kvm->arch.nested_mmus != tmp) {
+			int i;
+
+			for (i = 0; i < num_mmus - 2; i++)
+				tmp[i].pgt->mmu = &tmp[i];
+		}
+
+		if (kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 1]) ||
+		    kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 2])) {
+			kvm_free_stage2_pgd(&tmp[num_mmus - 1]);
+			kvm_free_stage2_pgd(&tmp[num_mmus - 2]);
+		} else {
+			kvm->arch.nested_mmus_size = num_mmus;
+			ret = 0;
+		}
+
+		kvm->arch.nested_mmus = tmp;
+	}
+
+	mutex_unlock(&kvm->lock);
+	return ret;
+}
+
+/* Must be called with kvm->lock held */
+struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr)
+{
+	bool nested_stage2_enabled = hcr & HCR_VM;
+	int i;
+
+	/* Don't consider the CnP bit for the vttbr match */
+	vttbr = vttbr & ~VTTBR_CNP_BIT;
+
+	/*

arch/arm64/kvm/nested.c line 109

Quoted 10 lines
arch/arm64/kvm/nested.c +10 -0
+	 * Two possibilities when looking up a S2 MMU context:
+	 *
+	 * - either S2 is enabled in the guest, and we need a context that
+         *   is S2-enabled and matches the full VTTBR (VMID+BADDR), which
+         *   makes it safe from a TLB conflict perspective (a broken guest
+         *   won't be able to generate them),
+	 *
+	 * - or S2 is disabled, and we need a context that is S2-disabled
+         *   and matches the VMID only, as all TLBs are tagged by VMID even
+         *   if S2 translation is enabled.

I think you were intended to say "if S2 translation is disabled".

Quoted 113 lines
arch/arm64/kvm/nested.c +106 -0
+	 */
+	for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
+		struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
+
+		if (!kvm_s2_mmu_valid(mmu))
+			continue;
+
+		if (nested_stage2_enabled &&
+		    mmu->nested_stage2_enabled &&
+		    vttbr == mmu->vttbr)
+			return mmu;
+
+		if (!nested_stage2_enabled &&
+		    !mmu->nested_stage2_enabled &&
+		    get_vmid(vttbr) == get_vmid(mmu->vttbr))
+			return mmu;
+	}
+	return NULL;
+}
+
+static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2);
+	u64 hcr= vcpu_read_sys_reg(vcpu, HCR_EL2);
+	struct kvm_s2_mmu *s2_mmu;
+	int i;
+
+	s2_mmu = lookup_s2_mmu(kvm, vttbr, hcr);
+	if (s2_mmu)
+		goto out;
+
+	/*
+	 * Make sure we don't always search from the same point, or we
+	 * will always reuse a potentially active context, leaving
+	 * free contexts unused.
+	 */
+	for (i = kvm->arch.nested_mmus_next;
+	     i < (kvm->arch.nested_mmus_size + kvm->arch.nested_mmus_next);
+	     i++) {
+		s2_mmu = &kvm->arch.nested_mmus[i % kvm->arch.nested_mmus_size];
+
+		if (atomic_read(&s2_mmu->refcnt) == 0)
+			break;
+	}
+	BUG_ON(atomic_read(&s2_mmu->refcnt)); /* We have struct MMUs to spare */
+
+	/* Set the scene for the next search */
+	kvm->arch.nested_mmus_next = (i + 1) % kvm->arch.nested_mmus_size;
+
+	if (kvm_s2_mmu_valid(s2_mmu)) {
+		/* Clear the old state */
+		kvm_unmap_stage2_range(s2_mmu, 0, kvm_phys_size(kvm));
+		if (s2_mmu->vmid.vmid_gen)
+			kvm_call_hyp(__kvm_tlb_flush_vmid, s2_mmu);
+	}
+
+	/*
+	 * The virtual VMID (modulo CnP) will be used as a key when matching
+	 * an existing kvm_s2_mmu.
+	 */
+	s2_mmu->vttbr = vttbr & ~VTTBR_CNP_BIT;
+	s2_mmu->nested_stage2_enabled = hcr & HCR_VM;
+
+out:
+	atomic_inc(&s2_mmu->refcnt);
+	return s2_mmu;
+}
+
+void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu)
+{
+	mmu->vttbr = 1;
+	mmu->nested_stage2_enabled = false;
+	atomic_set(&mmu->refcnt, 0);
+}
+
+void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu)
+{
+	if (is_hyp_ctxt(vcpu)) {
+		vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
+	} else {
+		spin_lock(&vcpu->kvm->mmu_lock);
+		vcpu->arch.hw_mmu = get_s2_mmu_nested(vcpu);
+		spin_unlock(&vcpu->kvm->mmu_lock);
+	}
+}
+
+void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu)
+{
+	if (vcpu->arch.hw_mmu != &vcpu->kvm->arch.mmu) {
+		atomic_dec(&vcpu->arch.hw_mmu->refcnt);
+		vcpu->arch.hw_mmu = NULL;
+	}
+}
+
  /*
   * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and
   * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor
@@ -43,6 +220,24 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe)
  	return -EINVAL;
  }
 
+void kvm_arch_flush_shadow_all(struct kvm *kvm)
+{
+	int i;
+
+	for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
+		struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
+
+		WARN_ON(atomic_read(&mmu->refcnt));
+
+		if (!atomic_read(&mmu->refcnt))
+			kvm_free_stage2_pgd(mmu);

arch/arm64/kvm/nested.c int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) · line 243

Quoted 10 lines
arch/arm64/kvm/nested.c +7 -0
+	}
+	kfree(kvm->arch.nested_mmus);
+	kvm->arch.nested_mmus = NULL;
+	kvm->arch.nested_mmus_size = 0;
+	kvm_free_stage2_pgd(&kvm->arch.mmu);
+}
+
  /*
   * Our emulated CPU doesn't support all the possible features. For the
   * sake of simplicity (and probably mental sanity), wipe out a number

It looks good to me, please feel free to add.

Thanks,
Ganapat

author maintainer
#103 lore raw

Re: [PATCH v5 38/69] KVM: arm64: nv: Support multiple nested Stage-2 mmu structures

Quoted 17 lines · Ganapatrao Kulkarni
Quoted 13 lines · Marc Zyngier
arch/arm64/kvm/nested.c +13 -0
+struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr)
+{
+	bool nested_stage2_enabled = hcr & HCR_VM;
+	int i;
+
+	/* Don't consider the CnP bit for the vttbr match */
+	vttbr = vttbr & ~VTTBR_CNP_BIT;
+
+	/*
+	 * Two possibilities when looking up a S2 MMU context:
+	 *
+	 * - either S2 is enabled in the guest, and we need a context that
+         *   is S2-enabled and matches the full VTTBR (VMID+BADDR), which
Quoted 8 lines

arch/arm64/kvm/nested.c line 109

Quoted 6 lines
arch/arm64/kvm/nested.c +6 -0
+         *   makes it safe from a TLB conflict perspective (a broken guest
+         *   won't be able to generate them),
+	 *
+	 * - or S2 is disabled, and we need a context that is S2-disabled
+         *   and matches the VMID only, as all TLBs are tagged by VMID even
+         *   if S2 translation is enabled.

I think you were intended to say "if S2 translation is disabled".

Yes, absolutely. Good catch.

Thanks,

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
author maintainer
#104 lore raw

PATCH v5 39/69 KVM: arm64: nv: Implement nested Stage-2 page table walk logic

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

Based on the pseudo-code in the ARM ARM, implement a stage 2 software page table walker.

[maz: heavily reworked for future ARMv8.4-TTL support]
---
 arch/arm64/include/asm/esr.h        |   1 +
 arch/arm64/include/asm/kvm_arm.h    |   2 +
 arch/arm64/include/asm/kvm_nested.h |  13 ++
 arch/arm64/kvm/nested.c             | 267 ++++++++++++++++++++++++++++
 4 files changed, 283 insertions(+)
arch/arm64/include/asm/esr.h +1 -0
index 6835e4231119..206bfb701791 100644
@@ -130,6 +130,7 @@
 #define ESR_ELx_CM 		(UL(1) << ESR_ELx_CM_SHIFT)
 
 /* ISS field definitions for exceptions taken in to Hyp */
+#define ESR_ELx_FSC_ADDRSZ	(0x00)
 #define ESR_ELx_CV		(UL(1) << 24)
 #define ESR_ELx_COND_SHIFT	(20)
 #define ESR_ELx_COND_MASK	(UL(0xF) << ESR_ELx_COND_SHIFT)
arch/arm64/include/asm/kvm_arm.h +2 -0
index b8a0d410035b..12f4bc8a805b 100644
@@ -272,6 +272,8 @@
 #define VTTBR_VMID_SHIFT  (UL(48))
 #define VTTBR_VMID_MASK(size) (_AT(u64, (1 << size) - 1) << VTTBR_VMID_SHIFT)
 
+#define SCTLR_EE	(UL(1) << 25)
+
 /* Hyp System Trap Register */
 #define HSTR_EL2_T(x)	(1 << x)
 
arch/arm64/include/asm/kvm_nested.h +13 -0
index 473ecd1d60d0..b784d7891851 100644
@@ -68,6 +68,19 @@ extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr);
 extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu);
 extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu);
 
+struct kvm_s2_trans {
+	phys_addr_t output;
+	unsigned long block_size;
+	bool writable;
+	bool readable;
+	int level;
+	u32 esr;
+	u64 upper_attr;
+};
+
+extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
+			      struct kvm_s2_trans *result);
+
 int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe);
 extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
 			    u64 control_bit);
arch/arm64/kvm/nested.c +267 -0

273 lines not on this page · raw message

Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#105 lore raw

PATCH v5 40/69 KVM: arm64: nv: Handle shadow stage 2 page faults

In reply to Marc Zyngier

If we are faulting on a shadow stage 2 translation, we first walk the guest hypervisor's stage 2 page table to see if it has a mapping. If not, we inject a stage 2 page fault to the virtual EL2. Otherwise, we create a mapping in the shadow stage 2 page table.

Note that we have to deal with two IPAs when we got a shadow stage 2 page fault. One is the address we faulted on, and is in the L2 guest phys space. The other is from the guest stage-2 page table walk, and is in the L1 guest phys space. To differentiate them, we rename variables so that fault_ipa is used for the former and ipa is used for the latter.

---
 arch/arm64/include/asm/kvm_emulate.h |   6 ++
 arch/arm64/include/asm/kvm_nested.h  |  18 +++++
 arch/arm64/kvm/mmu.c                 | 102 +++++++++++++++++++++++----
 arch/arm64/kvm/nested.c              |  48 +++++++++++++
 4 files changed, 159 insertions(+), 15 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +6 -0
index 105fb0d4a366..94d8814b7c8b 100644
@@ -602,4 +602,10 @@ static inline bool vcpu_has_feature(struct kvm_vcpu *vcpu, int feature)
 	return test_bit(feature, vcpu->arch.features);
 }
 
+static inline bool kvm_is_shadow_s2_fault(struct kvm_vcpu *vcpu)
+{
+	return (vcpu->arch.hw_mmu != &vcpu->kvm->arch.mmu &&
+		vcpu->arch.hw_mmu->nested_stage2_enabled);
+}
+
 #endif /* __ARM64_KVM_EMULATE_H__ */
arch/arm64/include/asm/kvm_nested.h +18 -0
index b784d7891851..4f93a5dab183 100644
@@ -78,9 +78,27 @@ struct kvm_s2_trans {
 	u64 upper_attr;
 };
 
+static inline phys_addr_t kvm_s2_trans_output(struct kvm_s2_trans *trans)
+{
+	return trans->output;
+}
+
+static inline unsigned long kvm_s2_trans_size(struct kvm_s2_trans *trans)
+{
+	return trans->block_size;
+}
+
+static inline u32 kvm_s2_trans_esr(struct kvm_s2_trans *trans)
+{
+	return trans->esr;
+}
+
 extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
 			      struct kvm_s2_trans *result);
 
+extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu,
+				    struct kvm_s2_trans *trans);
+extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2);
 int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe);
 extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
 			    u64 control_bit);
arch/arm64/kvm/mmu.c +87 -15

205 lines not on this page · raw message

arch/arm64/kvm/nested.c +48 -0

60 lines not on this page · raw message

Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#106 lore raw

PATCH v5 41/69 KVM: arm64: nv: Restrict S2 RD/WR permissions to match the guest's

In reply to Marc Zyngier

When mapping a page in a shadow stage-2, special care must be taken not to be more permissive than the guest is (writable or readable page when the guest hasn't set that permission).

---
 arch/arm64/include/asm/kvm_nested.h | 15 +++++++++++++++
 arch/arm64/kvm/mmu.c                | 14 +++++++++++++-
 arch/arm64/kvm/nested.c             |  2 +-
 3 files changed, 29 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_nested.h +15 -0
index 4f93a5dab183..3f3d8e10bd99 100644
@@ -93,6 +93,21 @@ static inline u32 kvm_s2_trans_esr(struct kvm_s2_trans *trans)
 	return trans->esr;
 }
 
+static inline bool kvm_s2_trans_readable(struct kvm_s2_trans *trans)
+{
+	return trans->readable;
+}
+
+static inline bool kvm_s2_trans_writable(struct kvm_s2_trans *trans)
+{
+	return trans->writable;
+}
+
+static inline bool kvm_s2_trans_executable(struct kvm_s2_trans *trans)
+{
+	return !(trans->upper_attr & BIT(54));
+}
+
 extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
 			      struct kvm_s2_trans *result);
 
arch/arm64/kvm/mmu.c +13 -1
index fc3613d3f503..d63f60f95abc 100644
@@ -1123,6 +1123,17 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 	if (exec_fault && device)
 		return -ENOEXEC;
 
+	/*
+	 * Potentially reduce shadow S2 permissions to match the guest's own
+	 * S2. For exec faults, we'd only reach this point if the guest
+	 * actually allowed it (see kvm_s2_handle_perm_fault).
+	 */
+	if (kvm_is_shadow_s2_fault(vcpu)) {
+		writable &= kvm_s2_trans_writable(nested);
+		if (!kvm_s2_trans_readable(nested))
+			prot &= ~KVM_PGTABLE_PROT_R;
+	}
+
 	spin_lock(&kvm->mmu_lock);
 	pgt = vcpu->arch.hw_mmu->pgt;
 	if (mmu_notifier_retry(kvm, mmu_seq))
@@ -1161,7 +1172,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
 
 	if (device)
 		prot |= KVM_PGTABLE_PROT_DEVICE;
-	else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC))
+	else if (cpus_have_const_cap(ARM64_HAS_CACHE_DIC) &&
+		 kvm_s2_trans_executable(nested))
 		prot |= KVM_PGTABLE_PROT_X;
 
 	/*
arch/arm64/kvm/nested.c +1 -1
index c34e96955e04..fb748f2b9e87 100644
@@ -493,7 +493,7 @@ int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans)
 		return 0;
 
 	if (kvm_vcpu_trap_is_iabt(vcpu)) {
-		forward_fault = (trans->upper_attr & BIT(54));
+		forward_fault = !kvm_s2_trans_executable(trans);
 	} else {
 		bool write_fault = kvm_is_write_fault(vcpu);
 
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#107 lore raw

PATCH v5 42/69 KVM: arm64: nv: Unmap/flush shadow stage 2 page tables

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

Unmap/flush shadow stage 2 page tables for the nested VMs as well as the stage 2 page table for the guest hypervisor.

Note: A bunch of the code in mmu.c relating to MMU notifiers is currently dealt with in an extremely abrupt way, for example by clearing out an entire shadow stage-2 table. This will be handled in a more efficient way using the reverse mapping feature in a later version of the patch series.

---
 arch/arm64/include/asm/kvm_mmu.h    |  3 +++
 arch/arm64/include/asm/kvm_nested.h |  3 +++
 arch/arm64/kvm/mmu.c                | 31 +++++++++++++++++++----
 arch/arm64/kvm/nested.c             | 39 +++++++++++++++++++++++++++++
 4 files changed, 71 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_mmu.h +3 -0
index 7250594e3d68..63b2eb5f4bcb 100644
@@ -158,6 +158,8 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 			   void __iomem **haddr);
 int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
 			     void **haddr);
+void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu,
+			    phys_addr_t addr, phys_addr_t end);
 void free_hyp_pgds(void);
 
 void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size);
@@ -166,6 +168,7 @@ int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu);
 void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu);
 int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
 			  phys_addr_t pa, unsigned long size, bool writable);
+void kvm_stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end);
 
 int kvm_handle_guest_abort(struct kvm_vcpu *vcpu);
 
arch/arm64/include/asm/kvm_nested.h +3 -0
index 3f3d8e10bd99..2987806850f0 100644
@@ -114,6 +114,9 @@ extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa,
 extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu,
 				    struct kvm_s2_trans *trans);
 extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2);
+extern void kvm_nested_s2_wp(struct kvm *kvm);
+extern void kvm_nested_s2_clear(struct kvm *kvm);
+extern void kvm_nested_s2_flush(struct kvm *kvm);
 int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe);
 extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg,
 			    u64 control_bit);
arch/arm64/kvm/mmu.c +26 -5
index d63f60f95abc..a9e743f39bfd 100644
@@ -190,13 +190,20 @@ void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size)
 	__unmap_stage2_range(mmu, start, size, true);
 }
 
+void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu,
+			    phys_addr_t addr, phys_addr_t end)
+{
+	stage2_apply_range_resched(kvm_s2_mmu_to_kvm(mmu), addr, end, kvm_pgtable_stage2_flush);
+}
+
 static void stage2_flush_memslot(struct kvm *kvm,
 				 struct kvm_memory_slot *memslot)
 {
 	phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT;
 	phys_addr_t end = addr + PAGE_SIZE * memslot->npages;
+	struct kvm_s2_mmu *mmu = &kvm->arch.mmu;
 
-	stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_flush);
+	kvm_stage2_flush_range(mmu, addr, end);
 }
 
 /**
@@ -219,6 +226,8 @@ static void stage2_flush_vm(struct kvm *kvm)
 	kvm_for_each_memslot(memslot, slots)
 		stage2_flush_memslot(kvm, memslot);
 
+	kvm_nested_s2_flush(kvm);
+
 	spin_unlock(&kvm->mmu_lock);
 	srcu_read_unlock(&kvm->srcu, idx);
 }
@@ -621,6 +630,8 @@ void stage2_unmap_vm(struct kvm *kvm)
 	kvm_for_each_memslot(memslot, slots)
 		stage2_unmap_memslot(kvm, memslot);
 
+	kvm_nested_s2_clear(kvm);
+
 	spin_unlock(&kvm->mmu_lock);
 	mmap_read_unlock(current->mm);
 	srcu_read_unlock(&kvm->srcu, idx);
@@ -690,12 +701,12 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa,
 }
 
 /**
- * stage2_wp_range() - write protect stage2 memory region range
+ * kvm_stage2_wp_range() - write protect stage2 memory region range
  * @mmu:        The KVM stage-2 MMU pointer
  * @addr:	Start address of range
  * @end:	End address of range
  */
-static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end)
+void kvm_stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end)
 {
 	struct kvm *kvm = kvm_s2_mmu_to_kvm(mmu);
 	stage2_apply_range_resched(kvm, addr, end, kvm_pgtable_stage2_wrprotect);
@@ -727,7 +738,8 @@ static void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot)
 	end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT;
 
 	spin_lock(&kvm->mmu_lock);
-	stage2_wp_range(&kvm->arch.mmu, start, end);
+	kvm_stage2_wp_range(&kvm->arch.mmu, start, end);
+	kvm_nested_s2_wp(kvm);
 	spin_unlock(&kvm->mmu_lock);
 	kvm_flush_remote_tlbs(kvm);
 }
@@ -751,7 +763,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm,
 	phys_addr_t start = (base_gfn +  __ffs(mask)) << PAGE_SHIFT;
 	phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT;
 
-	stage2_wp_range(&kvm->arch.mmu, start, end);
+	kvm_stage2_wp_range(&kvm->arch.mmu, start, end);
 }
 
 /*
@@ -766,6 +778,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm,
 		gfn_t gfn_offset, unsigned long mask)
 {
 	kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask);
+	kvm_nested_s2_wp(kvm);
 }
 
 static void kvm_send_hwpoison_signal(unsigned long address, short lsb)
@@ -1405,6 +1418,7 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range)
 			     (range->end - range->start) << PAGE_SHIFT,
 			     range->may_block);
 
+	kvm_nested_s2_clear(kvm);
 	return false;
 }
 
@@ -1436,6 +1450,7 @@ bool kvm_set_spte_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 			       PAGE_SIZE, __pfn_to_phys(pfn),
 			       KVM_PGTABLE_PROT_R, NULL);
 
+	kvm_nested_s2_clear(kvm);
 	return false;
 }
 
@@ -1454,6 +1469,11 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
 					range->start << PAGE_SHIFT);
 	pte = __pte(kpte);
 	return pte_valid(pte) && pte_young(pte);
+
+	/*
+	 * TODO: Handle nested_mmu structures here using the reverse mapping in
+	 * a later version of patch series.
+	 */
 }
 
 bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
@@ -1664,6 +1684,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 
 	spin_lock(&kvm->mmu_lock);
 	kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, size);
+	kvm_nested_s2_clear(kvm);
 	spin_unlock(&kvm->mmu_lock);
 }
 
arch/arm64/kvm/nested.c +39 -0
index fb748f2b9e87..b7e10f6178b9 100644
@@ -517,6 +517,45 @@ int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2)
 	return kvm_inject_nested_sync(vcpu, esr_el2);
 }
 
+/* expects kvm->mmu_lock to be held */
+void kvm_nested_s2_wp(struct kvm *kvm)
+{
+	int i;
+
+	for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
+		struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
+
+		if (kvm_s2_mmu_valid(mmu))
+			kvm_stage2_wp_range(mmu, 0, kvm_phys_size(kvm));
+	}
+}
+
+/* expects kvm->mmu_lock to be held */
+void kvm_nested_s2_clear(struct kvm *kvm)
+{
+	int i;
+
+	for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
+		struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
+
+		if (kvm_s2_mmu_valid(mmu))
+			kvm_unmap_stage2_range(mmu, 0, kvm_phys_size(kvm));
+	}
+}
+
+/* expects kvm->mmu_lock to be held */
+void kvm_nested_s2_flush(struct kvm *kvm)
+{
+	int i;
+
+	for (i = 0; i < kvm->arch.nested_mmus_size; i++) {
+		struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i];
+
+		if (kvm_s2_mmu_valid(mmu))
+			kvm_stage2_flush_range(mmu, 0, kvm_phys_size(kvm));
+	}
+}
+
 /*
  * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and
  * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#108 lore raw

PATCH v5 43/69 KVM: arm64: nv: Introduce sys_reg_desc.forward_trap

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

This introduces a function prototype to determine if we need to forward system instruction traps to the virtual EL2. The implementation of forward_trap functions for each system instruction will be added in later patches.

---
 arch/arm64/kvm/sys_regs.c | 8 ++++++++
 arch/arm64/kvm/sys_regs.h | 6 ++++++
 2 files changed, 14 insertions(+)
arch/arm64/kvm/sys_regs.c +8 -0
index 3922ecc00869..77f07f960d84 100644
@@ -2598,6 +2598,14 @@ static void perform_access(struct kvm_vcpu *vcpu,
 	 */
 	BUG_ON(!r->access);
 
+	/*
+	 * Forward this trap to the virtual EL2 if the guest hypervisor has
+	 * configured to trap the current instruction.
+	 */
+	if (nested_virt_in_use(vcpu) && r->forward_trap
+	    && unlikely(r->forward_trap(vcpu)))
+		return;
+
 	/* Skip instruction if instructed so */
 	if (likely(r->access(vcpu, params, r)))
 		kvm_incr_pc(vcpu);
arch/arm64/kvm/sys_regs.h +6 -0
index d260c26b1834..267b2f37444c 100644
@@ -72,6 +72,12 @@ struct sys_reg_desc {
 	int (*set_user)(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd,
 			const struct kvm_one_reg *reg, void __user *uaddr);
 
+	/*
+	 * Forward the trap to the virtual EL2 if the guest hypervisor has
+	 * configured to trap the current instruction.
+	 */
+	bool (*forward_trap)(struct kvm_vcpu *vcpu);
+
 	/* Return mask of REG_* runtime visibility overrides */
 	unsigned int (*visibility)(const struct kvm_vcpu *vcpu,
 				   const struct sys_reg_desc *rd);
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#109 lore raw

PATCH v5 44/69 KVM: arm64: nv: Set a handler for the system instruction traps

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

When HCR.NV bit is set, execution of the EL2 translation regime address aranslation instructions and TLB maintenance instructions are trapped to EL2. In addition, execution of the EL1 translation regime address aranslation instructions and TLB maintenance instructions that are only accessible from EL2 and above are trapped to EL2. In these cases, ESR_EL2.EC will be set to 0x18.

Rework the system instruction emulation framework to handle potentially all system instruction traps other than MSR/MRS instructions. Those system instructions would be AT and TLBI instructions controlled by HCR_EL2.NV, AT, and TTLB bits.

[maz: squashed two patches together, redispatched various bits around]
---
 arch/arm64/include/asm/kvm_host.h |  4 +--
 arch/arm64/kvm/handle_exit.c      |  2 +-
 arch/arm64/kvm/sys_regs.c         | 48 +++++++++++++++++++++++++------
 3 files changed, 42 insertions(+), 12 deletions(-)
arch/arm64/include/asm/kvm_host.h +2 -2
index 00c3366129b8..35f3d7939484 100644
@@ -411,7 +411,7 @@ struct kvm_vcpu_arch {
 	/*
 	 * Guest registers we preserve during guest debugging.
 	 *
-	 * These shadow registers are updated by the kvm_handle_sys_reg
+	 * These shadow registers are updated by the kvm_handle_sys
 	 * trap handler if the guest accesses or updates them while we
 	 * are using guest debug.
 	 */
@@ -729,7 +729,7 @@ int kvm_handle_cp14_32(struct kvm_vcpu *vcpu);
 int kvm_handle_cp14_64(struct kvm_vcpu *vcpu);
 int kvm_handle_cp15_32(struct kvm_vcpu *vcpu);
 int kvm_handle_cp15_64(struct kvm_vcpu *vcpu);
-int kvm_handle_sys_reg(struct kvm_vcpu *vcpu);
+int kvm_handle_sys(struct kvm_vcpu *vcpu);
 
 void kvm_reset_sys_regs(struct kvm_vcpu *vcpu);
 
arch/arm64/kvm/handle_exit.c +1 -1
index 6ff709c124d0..8b3b758e674e 100644
@@ -234,7 +234,7 @@ static exit_handle_fn arm_exit_handlers[] = {
 	[ESR_ELx_EC_SMC32]	= handle_smc,
 	[ESR_ELx_EC_HVC64]	= handle_hvc,
 	[ESR_ELx_EC_SMC64]	= handle_smc,
-	[ESR_ELx_EC_SYS64]	= kvm_handle_sys_reg,
+	[ESR_ELx_EC_SYS64]	= kvm_handle_sys,
 	[ESR_ELx_EC_SVE]	= handle_sve,
 	[ESR_ELx_EC_ERET]	= kvm_handle_eret,
 	[ESR_ELx_EC_IABT_LOW]	= kvm_handle_guest_abort,
arch/arm64/kvm/sys_regs.c +39 -9

87 lines not on this page · raw message

Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Reviewed-by on v5 44/69
#110 lore raw

Re: [PATCH v5 44/69] KVM: arm64: nv: Set a handler for the system instruction traps

Quoted 4 lines · Marc Zyngier

From: Jintack Lim <[email protected]>

When HCR.NV bit is set, execution of the EL2 translation regime address aranslation instructions and TLB maintenance instructions are trapped to

Typo: translation

Quoted 2 lines

EL2. In addition, execution of the EL1 translation regime address aranslation instructions and TLB maintenance instructions that are only

Typo: translation

Quoted 142 lines

accessible from EL2 and above are trapped to EL2. In these cases, ESR_EL2.EC will be set to 0x18.

Rework the system instruction emulation framework to handle potentially all system instruction traps other than MSR/MRS instructions. Those system instructions would be AT and TLBI instructions controlled by HCR_EL2.NV, AT, and TTLB bits.

[maz: squashed two patches together, redispatched various bits around]
---
  arch/arm64/include/asm/kvm_host.h |  4 +--
  arch/arm64/kvm/handle_exit.c      |  2 +-
  arch/arm64/kvm/sys_regs.c         | 48 +++++++++++++++++++++++++------
  3 files changed, 42 insertions(+), 12 deletions(-)
arch/arm64/include/asm/kvm_host.h +2 -2
index 00c3366129b8..35f3d7939484 100644
@@ -411,7 +411,7 @@ struct kvm_vcpu_arch {
  	/*
  	 * Guest registers we preserve during guest debugging.
  	 *
-	 * These shadow registers are updated by the kvm_handle_sys_reg
+	 * These shadow registers are updated by the kvm_handle_sys
  	 * trap handler if the guest accesses or updates them while we
  	 * are using guest debug.
  	 */
@@ -729,7 +729,7 @@ int kvm_handle_cp14_32(struct kvm_vcpu *vcpu);
  int kvm_handle_cp14_64(struct kvm_vcpu *vcpu);
  int kvm_handle_cp15_32(struct kvm_vcpu *vcpu);
  int kvm_handle_cp15_64(struct kvm_vcpu *vcpu);
-int kvm_handle_sys_reg(struct kvm_vcpu *vcpu);
+int kvm_handle_sys(struct kvm_vcpu *vcpu);
 
  void kvm_reset_sys_regs(struct kvm_vcpu *vcpu);
 
arch/arm64/kvm/handle_exit.c +1 -1
index 6ff709c124d0..8b3b758e674e 100644
@@ -234,7 +234,7 @@ static exit_handle_fn arm_exit_handlers[] = {
  	[ESR_ELx_EC_SMC32]	= handle_smc,
  	[ESR_ELx_EC_HVC64]	= handle_hvc,
  	[ESR_ELx_EC_SMC64]	= handle_smc,
-	[ESR_ELx_EC_SYS64]	= kvm_handle_sys_reg,
+	[ESR_ELx_EC_SYS64]	= kvm_handle_sys,
  	[ESR_ELx_EC_SVE]	= handle_sve,
  	[ESR_ELx_EC_ERET]	= kvm_handle_eret,
  	[ESR_ELx_EC_IABT_LOW]	= kvm_handle_guest_abort,
arch/arm64/kvm/sys_regs.c +38 -9
index 77f07f960d84..222f8c6f1d7e 100644
@@ -1770,10 +1770,6 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu,
   * more demanding guest...
   */
  static const struct sys_reg_desc sys_reg_descs[] = {
-	{ SYS_DESC(SYS_DC_ISW), access_dcsw },
-	{ SYS_DESC(SYS_DC_CSW), access_dcsw },
-	{ SYS_DESC(SYS_DC_CISW), access_dcsw },
-
  	DBG_BCR_BVR_WCR_WVR_EL1(0),
  	DBG_BCR_BVR_WCR_WVR_EL1(1),
  	{ SYS_DESC(SYS_MDCCINT_EL1), trap_debug_regs, reset_val, MDCCINT_EL1, 0 },
@@ -2240,6 +2236,14 @@ static const struct sys_reg_desc sys_reg_descs[] = {
  	{ SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 },
  };
 
+#define SYS_INSN_TO_DESC(insn, access_fn, forward_fn)	\
+	{ SYS_DESC((insn)), (access_fn), NULL, 0, 0, NULL, NULL, (forward_fn) }
+static struct sys_reg_desc sys_insn_descs[] = {
+	{ SYS_DESC(SYS_DC_ISW), access_dcsw },
+	{ SYS_DESC(SYS_DC_CSW), access_dcsw },
+	{ SYS_DESC(SYS_DC_CISW), access_dcsw },
+};
+
  static bool trap_dbgdidr(struct kvm_vcpu *vcpu,
  			struct sys_reg_params *p,
  			const struct sys_reg_desc *r)
@@ -2794,6 +2798,24 @@ static int emulate_sys_reg(struct kvm_vcpu *vcpu,
  	return 1;
  }
 
+static int emulate_sys_instr(struct kvm_vcpu *vcpu, struct sys_reg_params *p)
+{
+	const struct sys_reg_desc *r;
+
+	/* Search from the system instruction table. */
+	r = find_reg(p, sys_insn_descs, ARRAY_SIZE(sys_insn_descs));
+
+	if (likely(r)) {
+		perform_access(vcpu, p, r);
+	} else {
+		kvm_err("Unsupported guest sys instruction at: %lx\n",
+			*vcpu_pc(vcpu));
+		print_sys_reg_instr(p);
+		kvm_inject_undefined(vcpu);
+	}
+	return 1;
+}
+
  /**
   * kvm_reset_sys_regs - sets system registers to reset value
   * @vcpu: The VCPU pointer
@@ -2811,10 +2833,11 @@ void kvm_reset_sys_regs(struct kvm_vcpu *vcpu)
  }
 
  /**
- * kvm_handle_sys_reg -- handles a mrs/msr trap on a guest sys_reg access
+ * kvm_handle_sys-- handles a system instruction or mrs/msr instruction trap
+		    on a guest execution
   * @vcpu: The VCPU pointer
   */
-int kvm_handle_sys_reg(struct kvm_vcpu *vcpu)
+int kvm_handle_sys(struct kvm_vcpu *vcpu)
  {
  	struct sys_reg_params params;
  	unsigned long esr = kvm_vcpu_get_esr(vcpu);
@@ -2826,10 +2849,16 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu)
  	params = esr_sys64_to_params(esr);
  	params.regval = vcpu_get_reg(vcpu, Rt);
 
-	ret = emulate_sys_reg(vcpu, &params);
+	if (params.Op0 == 1) {
+		/* System instructions */
+		ret = emulate_sys_instr(vcpu, &params);
+	} else {
+		/* MRS/MSR instructions */
+		ret = emulate_sys_reg(vcpu, &params);
+		if (!params.is_write)
+			vcpu_set_reg(vcpu, Rt, params.regval);
+	}
 
-	if (!params.is_write)
-		vcpu_set_reg(vcpu, Rt, params.regval);
  	return ret;
  }
 

arch/arm64/kvm/sys_regs.c void kvm_sys_reg_table_init(void) · line 3280

Quoted 7 lines
arch/arm64/kvm/sys_regs.c +1 -0
@@ -3245,6 +3274,7 @@ void kvm_sys_reg_table_init(void)
  	BUG_ON(check_sysreg_table(cp15_regs, ARRAY_SIZE(cp15_regs), true));
  	BUG_ON(check_sysreg_table(cp15_64_regs, ARRAY_SIZE(cp15_64_regs), true));
  	BUG_ON(check_sysreg_table(invariant_sys_regs, ARRAY_SIZE(invariant_sys_regs), false));
+	BUG_ON(check_sysreg_table(sys_insn_descs, ARRAY_SIZE(sys_insn_descs), false));
 
  	/* We abuse the reset function to overwrite the table itself. */
  	for (i = 0; i < ARRAY_SIZE(invariant_sys_regs); i++)

It looks good to me, please feel free to add.

Thanks,
Ganapat

author maintainer
#111 lore raw

PATCH v5 45/69 KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2

In reply to Marc Zyngier

From: Jintack Lim <[email protected]>

When supporting nested virtualization a guest hypervisor executing AT instructions must be trapped and emulated by the host hypervisor, because untrapped AT instructions operating on S1E1 will use the wrong translation regieme (the one used to emulate virtual EL2 in EL1 instead of virtual EL1) and AT instructions operating on S12 will not work from EL1.

This patch does several things.

1. List and define all AT system instructions to emulate and document the emulation design.

2. Implement AT instruction handling logic in EL2. This will be used to emulate AT instructions executed in the virtual EL2.

AT instruction emulation works by loading the proper processor context, which depends on the trapped instruction and the virtual HCR_EL2, to the EL1 virtual memory control registers and executing AT instructions. Note that ctxt->hw_sys_regs is expected to have the proper processor context before calling the handling function(__kvm_at_insn) implemented in this patch.

4. Emulate AT S1E[01] instructions by issuing the same instructions in EL2. We set the physical EL1 registers, NV and NV1 bits as described in the AT instruction emulation overview.

5. Emulate AT A12E[01] instructions in two steps: First, do the stage-1 translation by reusing the existing AT emulation functions. Second, do the stage-2 translation by walking the guest hypervisor's stage-2 page table in software. Record the translation result to PAR_EL1.

6. Emulate AT S1E2 instructions by issuing the corresponding S1E1 instructions in EL2. We set the physical EL1 registers and the HCR_EL2 register as described in the AT instruction emulation overview.

7. Forward system instruction traps to the virtual EL2 if the corresponding virtual AT bit is set in the virtual HCR_EL2.

[ Much logic above has been reworked by Marc Zyngier ]

---
 arch/arm64/include/asm/kvm_arm.h |   2 +
 arch/arm64/include/asm/kvm_asm.h |   2 +
 arch/arm64/include/asm/sysreg.h  |  17 +++
 arch/arm64/kvm/Makefile          |   2 +-
 arch/arm64/kvm/at.c              | 231 +++++++++++++++++++++++++++++++
 arch/arm64/kvm/hyp/vhe/switch.c  |  13 +-
 arch/arm64/kvm/sys_regs.c        | 201 ++++++++++++++++++++++++++-
 7 files changed, 463 insertions(+), 5 deletions(-)
 create mode 100644 arch/arm64/kvm/at.c
arch/arm64/include/asm/kvm_arm.h +2 -0
index 12f4bc8a805b..b603466803d2 100644
@@ -20,6 +20,7 @@
 #define HCR_AMVOFFEN	(UL(1) << 51)
 #define HCR_FIEN	(UL(1) << 47)
 #define HCR_FWB		(UL(1) << 46)
+#define HCR_AT		(UL(1) << 44)
 #define HCR_NV1		(UL(1) << 43)
 #define HCR_NV		(UL(1) << 42)
 #define HCR_API		(UL(1) << 41)
@@ -118,6 +119,7 @@
 #define VTCR_EL2_TG0_16K	TCR_TG0_16K
 #define VTCR_EL2_TG0_64K	TCR_TG0_64K
 #define VTCR_EL2_SH0_MASK	TCR_SH0_MASK
+#define VTCR_EL2_SH0_SHIFT	TCR_SH0_SHIFT
 #define VTCR_EL2_SH0_INNER	TCR_SH0_INNER
 #define VTCR_EL2_ORGN0_MASK	TCR_ORGN0_MASK
 #define VTCR_EL2_ORGN0_WBWA	TCR_ORGN0_WBWA
arch/arm64/include/asm/kvm_asm.h +2 -0
index 50d5e4de244c..abea7e164400 100644
@@ -207,6 +207,8 @@ extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa,
 extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);
 
 extern void __kvm_timer_set_cntvoff(u64 cntvoff);
+extern void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
+extern void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
 
 extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
 
arch/arm64/include/asm/sysreg.h +17 -0
index c77fe5401826..b673adeb190f 100644
@@ -657,6 +657,23 @@
 
 #define SYS_SP_EL2			sys_reg(3, 6,  4, 1, 0)
 
+/* AT instructions */
+#define AT_Op0 1
+#define AT_CRn 7
+
+#define OP_AT_S1E1R	sys_insn(AT_Op0, 0, AT_CRn, 8, 0)
+#define OP_AT_S1E1W	sys_insn(AT_Op0, 0, AT_CRn, 8, 1)
+#define OP_AT_S1E0R	sys_insn(AT_Op0, 0, AT_CRn, 8, 2)
+#define OP_AT_S1E0W	sys_insn(AT_Op0, 0, AT_CRn, 8, 3)
+#define OP_AT_S1E1RP	sys_insn(AT_Op0, 0, AT_CRn, 9, 0)
+#define OP_AT_S1E1WP	sys_insn(AT_Op0, 0, AT_CRn, 9, 1)
+#define OP_AT_S1E2R	sys_insn(AT_Op0, 4, AT_CRn, 8, 0)
+#define OP_AT_S1E2W	sys_insn(AT_Op0, 4, AT_CRn, 8, 1)
+#define OP_AT_S12E1R	sys_insn(AT_Op0, 4, AT_CRn, 8, 4)
+#define OP_AT_S12E1W	sys_insn(AT_Op0, 4, AT_CRn, 8, 5)
+#define OP_AT_S12E0R	sys_insn(AT_Op0, 4, AT_CRn, 8, 6)
+#define OP_AT_S12E0W	sys_insn(AT_Op0, 4, AT_CRn, 8, 7)
+
 /* Common SCTLR_ELx flags. */
 #define SCTLR_ELx_DSSBS	(BIT(44))
 #define SCTLR_ELx_ATA	(BIT(43))
arch/arm64/kvm/Makefile +1 -1
index 923ee13728b4..8e19da1a07e1 100644
@@ -16,7 +16,7 @@ kvm-y := $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o \
 	 inject_fault.o va_layout.o handle_exit.o \
 	 guest.o debug.o reset.o sys_regs.o \
 	 vgic-sys-reg-v3.o fpsimd.o pmu.o \
-	 arch_timer.o trng.o emulate-nested.o nested.o \
+	 arch_timer.o trng.o emulate-nested.o nested.o at.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
 	 vgic/vgic-v3.o vgic/vgic-v4.o \
arch/arm64/kvm/at.c new file +231 -0

231 lines not on this page · raw message

arch/arm64/kvm/hyp/vhe/switch.c +11 -2
index da80c969e623..657a74bab73c 100644
@@ -42,9 +42,10 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
 		if (!vcpu_el2_e2h_is_set(vcpu)) {
 			/*
 			 * For a guest hypervisor on v8.0, trap and emulate
-			 * the EL1 virtual memory control register accesses.
+			 * the EL1 virtual memory control register accesses
+			 * as well as the AT S1 operations.
 			 */
-			hcr |= HCR_TVM | HCR_TRVM | HCR_NV1;
+			hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_NV1;
 		} else {
 			/*
 			 * For a guest hypervisor on v8.1 (VHE), allow to
@@ -69,6 +70,14 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
 			hcr &= ~HCR_TVM;
 
 			hcr |= vhcr_el2 & (HCR_TVM | HCR_TRVM);
+
+			/*
+			 * If we're using the EL1 translation regime
+			 * (TGE clear), then ensure that AT S1 ops are
+			 * trapped too.
+			 */
+			if (!vcpu_el2_tge_is_set(vcpu))
+				hcr |= HCR_AT;
 		}
 	}
 
arch/arm64/kvm/sys_regs.c +199 -2

217 lines not on this page · raw message

Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#112 lore raw

PATCH v5 46/69 KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2

In reply to Marc Zyngier

When supporting nested virtualization a guest hypervisor executing TLBI instructions must be trapped and emulated by the host hypervisor, because the guest hypervisor can only affect physical TLB entries relating to its own execution environment (virtual EL2 in EL1) but not to the nested guests as required by the semantics of the instructions and TLBI instructions might also result in updates (invalidations) to shadow page tables.

This patch does several things.

1. List and define all TLBI system instructions to emulate.

2. Emulate TLBI ALLE2(IS) instruction executed in the virtual EL2. Since we emulate the virtual EL2 in the EL1, we invalidate EL1&0 regime stage 1 TLB entries with setting vttbr_el2 having the VMID of the virtual EL2.

3. Emulate TLBI VAE2* instruction executed in the virtual EL2. Based on the same principle as TLBI ALLE2 instruction, we can simply emulate those instructions by executing corresponding VAE1* instructions with the virtual EL2's VMID assigned by the host hypervisor.

Note that we are able to emulate TLBI ALLE2IS precisely by only invalidating stage 1 TLB entries via TLBI VMALL1IS instruction, but to make it simeple, we reuse the existing function, __kvm_tlb_flush_vmid(), which invalidates both of stage 1 and 2 TLB entries.

4. TLBI ALLE1(IS) instruction invalidates all EL1&0 regime stage 1 and 2 TLB entries (on all PEs in the same Inner Shareable domain). To emulate these instructions, we first need to clear all the mappings in the shadow page tables since executing those instructions implies the change of mappings in the stage 2 page tables maintained by the guest hypervisor. We then need to invalidate all EL1&0 regime stage 1 and 2 TLB entries of all VMIDs, which are assigned by the host hypervisor, for this VM.

5. Based on the same principle as TLBI ALLE1(IS) emulation, we clear the mappings in the shadow stage-2 page tables and invalidate TLB entries. But this time we do it only for the current VMID from the guest hypervisor's perspective, not for all VMIDs.

6. Based on the same principle as TLBI ALLE1(IS) and TLBI VMALLS12E1(IS) emulation, we clear the mappings in the shadow stage-2 page tables and invalidate TLB entries. We do it only for one mapping for the current VMID from the guest hypervisor's view.

7. Forward system instruction traps to the virtual EL2 if a corresponding bit in the virtual HCR_EL2 is set.

8. Even though a guest hypervisor can execute TLBI instructions that are accesible at EL1 without trap, it's wrong; All those TLBI instructions work based on current VMID, and when running a guest hypervisor current VMID is the one for itself, not the one from the virtual vttbr_el2. So letting a guest hypervisor execute those TLBI instructions results in invalidating its own TLB entries and leaving invalid TLB entries unhandled.

Therefore we trap and emulate those TLBI instructions. The emulation is simple; we find a shadow VMID mapped to the virtual vttbr_el2, set it in the physical vttbr_el2, then execute the same instruction in EL2.

We don't set HCR_EL2.TTLB bit yet.

[ Changes performed by Marc Zynger:

The TLBI handling code more or less directly execute the same instruction that has been trapped (with an EL2->EL1 conversion in the case of an EL2 TLBI), but that's unfortunately not enough:

- TLBIs must be upgraded to the Inner Shareable domain to account for vcpu migration, just like we already have with HCR_EL2.FB.

- The DSB instruction that synchronises these must thus be on the Inner Shareable domain as well.

- Prior to executing the TLBI, we need another DSB ISHST to make sure that the update to the page tables is now visible.

Ordering of system instructions fixed

- The current TLB invalidation code is pretty buggy, as it assume a page mapping. On the contrary, it is likely that TLB invalidation will cover more than a single page, and the size should be decided by the guests configuration (and not the host's).

Since we don't cache the guest mapping sizes in the shadow PT yet, let's assume the worse case (a block mapping) and invalidate that.

Take this opportunity to fix the decoding of the parameter (it isn't a straight IPA).

- In general, we always emulate local TBL invalidations as being as upgraded to the Inner Shareable domain so that we can easily deal with vcpu migration. This is consistent with the fact that we set HCR_EL2.FB when running non-nested VMs.

So let's emulate TLBI ALLE2 as ALLE2IS.
]

[ Changes performed by Christoffer Dall:

Sometimes when we are invalidating the TLB for a certain S2 MMU context, this context can also have EL2 context associated with it and we have to invalidate this too.
]

---
 arch/arm64/include/asm/kvm_asm.h |   2 +
 arch/arm64/include/asm/sysreg.h  |  36 ++++++
 arch/arm64/kvm/hyp/vhe/switch.c  |   8 +-
 arch/arm64/kvm/hyp/vhe/tlb.c     |  81 ++++++++++++
 arch/arm64/kvm/mmu.c             |  19 ++-
 arch/arm64/kvm/sys_regs.c        | 212 +++++++++++++++++++++++++++++++
 6 files changed, 353 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_asm.h +2 -0
index abea7e164400..7f56f1b14445 100644
@@ -205,6 +205,8 @@ extern void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu);
 extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa,
 				     int level);
 extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu);
+extern void __kvm_tlb_vae2is(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding);
+extern void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding);
 
 extern void __kvm_timer_set_cntvoff(u64 cntvoff);
 extern void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr);
arch/arm64/include/asm/sysreg.h +36 -0
index b673adeb190f..199333b02ece 100644
@@ -674,6 +674,42 @@
 #define OP_AT_S12E0R	sys_insn(AT_Op0, 4, AT_CRn, 8, 6)
 #define OP_AT_S12E0W	sys_insn(AT_Op0, 4, AT_CRn, 8, 7)
 
+/* TLBI instructions */
+#define TLBI_Op0	1
+#define TLBI_Op1_EL1	0	/* Accessible from EL1 or higher */
+#define TLBI_Op1_EL2	4	/* Accessible from EL2 or higher */
+#define TLBI_CRn	8
+#define tlbi_insn_el1(CRm, Op2)	sys_insn(TLBI_Op0, TLBI_Op1_EL1, TLBI_CRn, (CRm), (Op2))
+#define tlbi_insn_el2(CRm, Op2)	sys_insn(TLBI_Op0, TLBI_Op1_EL2, TLBI_CRn, (CRm), (Op2))
+
+#define OP_TLBI_VMALLE1IS	tlbi_insn_el1(3, 0)
+#define OP_TLBI_VAE1IS		tlbi_insn_el1(3, 1)
+#define OP_TLBI_ASIDE1IS	tlbi_insn_el1(3, 2)
+#define OP_TLBI_VAAE1IS		tlbi_insn_el1(3, 3)
+#define OP_TLBI_VALE1IS		tlbi_insn_el1(3, 5)
+#define OP_TLBI_VAALE1IS	tlbi_insn_el1(3, 7)
+#define OP_TLBI_VMALLE1		tlbi_insn_el1(7, 0)
+#define OP_TLBI_VAE1		tlbi_insn_el1(7, 1)
+#define OP_TLBI_ASIDE1		tlbi_insn_el1(7, 2)
+#define OP_TLBI_VAAE1		tlbi_insn_el1(7, 3)
+#define OP_TLBI_VALE1		tlbi_insn_el1(7, 5)
+#define OP_TLBI_VAALE1		tlbi_insn_el1(7, 7)
+
+#define OP_TLBI_IPAS2E1IS	tlbi_insn_el2(0, 1)
+#define OP_TLBI_IPAS2LE1IS	tlbi_insn_el2(0, 5)
+#define OP_TLBI_ALLE2IS		tlbi_insn_el2(3, 0)
+#define OP_TLBI_VAE2IS		tlbi_insn_el2(3, 1)
+#define OP_TLBI_ALLE1IS		tlbi_insn_el2(3, 4)
+#define OP_TLBI_VALE2IS		tlbi_insn_el2(3, 5)
+#define OP_TLBI_VMALLS12E1IS	tlbi_insn_el2(3, 6)
+#define OP_TLBI_IPAS2E1		tlbi_insn_el2(4, 1)
+#define OP_TLBI_IPAS2LE1	tlbi_insn_el2(4, 5)
+#define OP_TLBI_ALLE2		tlbi_insn_el2(7, 0)
+#define OP_TLBI_VAE2		tlbi_insn_el2(7, 1)
+#define OP_TLBI_ALLE1		tlbi_insn_el2(7, 4)
+#define OP_TLBI_VALE2		tlbi_insn_el2(7, 5)
+#define OP_TLBI_VMALLS12E1	tlbi_insn_el2(7, 6)
+
 /* Common SCTLR_ELx flags. */
 #define SCTLR_ELx_DSSBS	(BIT(44))
 #define SCTLR_ELx_ATA	(BIT(43))
arch/arm64/kvm/hyp/vhe/switch.c +4 -4
index 657a74bab73c..71a4914761d0 100644
@@ -45,7 +45,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
 			 * the EL1 virtual memory control register accesses
 			 * as well as the AT S1 operations.
 			 */
-			hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_NV1;
+			hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_TTLB | HCR_NV1;
 		} else {
 			/*
 			 * For a guest hypervisor on v8.1 (VHE), allow to
@@ -73,11 +73,11 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
 
 			/*
 			 * If we're using the EL1 translation regime
-			 * (TGE clear), then ensure that AT S1 ops are
-			 * trapped too.
+			 * (TGE clear), then ensure that AT S1 and
+			 * TLBI E1 ops are trapped too.
 			 */
 			if (!vcpu_el2_tge_is_set(vcpu))
-				hcr |= HCR_AT;
+				hcr |= HCR_AT | HCR_TTLB;
 		}
 	}
 
arch/arm64/kvm/hyp/vhe/tlb.c +81 -0

84 lines not on this page · raw message

arch/arm64/kvm/mmu.c +18 -1
index a9e743f39bfd..7e640cae732d 100644
@@ -80,8 +80,25 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot)
  */
 void kvm_flush_remote_tlbs(struct kvm *kvm)
 {
+	struct kvm_s2_mmu *mmu = &kvm->arch.mmu;
+
 	++kvm->stat.generic.remote_tlb_flush_requests;
-	kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu);
+
+	if (mmu == &kvm->arch.mmu) {
+		/*
+		 * For a normal (i.e. non-nested) guest, flush entries for the
+		 * given VMID *
+		 */
+		kvm_call_hyp(__kvm_tlb_flush_vmid, mmu);
+	} else {
+		/*
+		 * When supporting nested virtualization, we can have multiple
+		 * VMIDs in play for each VCPU in the VM, so it's really not
+		 * worth it to try to quiesce the system and flush all the
+		 * VMIDs that may be in use, instead just nuke the whole thing.
+		 */
+		kvm_call_hyp(__kvm_flush_vm_context);
+	}
 }
 
 static bool kvm_is_device_pfn(unsigned long pfn)
arch/arm64/kvm/sys_regs.c +212 -0

236 lines not on this page · raw message

Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#113 lore raw

PATCH v5 47/69 KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's

In reply to Marc Zyngier

When entering a L2 guest (nested virt enabled, but not in hypervisor context), we need to honor the traps the L1 guest has asked enabled.

For now, just OR the guest's HCR_EL2 into the host's. We may have to do some filtering in the future though.

---
 arch/arm64/kvm/hyp/vhe/switch.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/arm64/kvm/hyp/vhe/switch.c +5 -0
index 71a4914761d0..ef4488db6dc1 100644
@@ -79,6 +79,11 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
 			if (!vcpu_el2_tge_is_set(vcpu))
 				hcr |= HCR_AT | HCR_TTLB;
 		}
+	} else if (nested_virt_in_use(vcpu)) {
+		u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2);
+
+		vhcr_el2 &= ~HCR_GUEST_NV_FILTER_FLAGS;
+		hcr |= vhcr_el2;
 	}
 
 	___activate_traps(vcpu, hcr);
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
author maintainer
#114 lore raw

PATCH v5 48/69 KVM: arm64: nv: arch_timer: Support hyp timer emulation

In reply to Marc Zyngier

From: Christoffer Dall <[email protected]>

Emulating EL2 also means emulating the EL2 timers. To do so, we expand our timer framework to deal with at most 4 timers. At any given time, two timers are using the HW timers, and the two others are purely emulated.

The role of deciding which is which at any given time is left to a mapping function which is called every time we need to make such a decision.

[maz: added CNTVOFF support, general reworking for v4.8]
---
 arch/arm64/include/asm/kvm_host.h |   4 +
 arch/arm64/kvm/arch_timer.c       | 165 ++++++++++++++++++++++++++++--
 arch/arm64/kvm/sys_regs.c         |   7 +-
 arch/arm64/kvm/trace_arm.h        |   6 +-
 arch/arm64/kvm/vgic/vgic.c        |  15 +++
 include/kvm/arm_arch_timer.h      |   8 +-
 include/kvm/arm_vgic.h            |   1 +
 7 files changed, 194 insertions(+), 12 deletions(-)
arch/arm64/include/asm/kvm_host.h +4 -0
index 35f3d7939484..96edd6348bd1 100644
@@ -286,6 +286,10 @@ enum vcpu_sysreg {
 	TPIDR_EL2,	/* EL2 Software Thread ID Register */
 	CNTHCTL_EL2,	/* Counter-timer Hypervisor Control register */
 	SP_EL2,		/* EL2 Stack Pointer */
+	CNTHP_CTL_EL2,
+	CNTHP_CVAL_EL2,
+	CNTHV_CTL_EL2,
+	CNTHV_CVAL_EL2,
 
 	NR_SYS_REGS	/* Nothing after this line! */
 };
arch/arm64/kvm/arch_timer.c +157 -8

395 lines not on this page · raw message

arch/arm64/kvm/sys_regs.c +6 -1
index 5bd1ea835cf0..e6c9a01670c9 100644
@@ -1325,6 +1325,11 @@ static bool access_arch_timer(struct kvm_vcpu *vcpu,
 		tmr = TIMER_PTIMER;
 		treg = TIMER_REG_CVAL;
 		break;
+	case SYS_CNTVOFF_EL2:
+		tmr = TIMER_VTIMER;
+		treg = TIMER_REG_VOFF;
+		break;
+
 	default:
 		BUG();
 	}
@@ -2222,7 +2227,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {
 	{ SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 },
 	{ SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 },
 
-	{ SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 },
+	{ SYS_DESC(SYS_CNTVOFF_EL2), access_arch_timer },
 	{ SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 },
 
 	{ SYS_DESC(SYS_SCTLR_EL12), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 },
arch/arm64/kvm/trace_arm.h +5 -1
index f3e46a976125..6ce5c025218d 100644
@@ -206,6 +206,7 @@ TRACE_EVENT(kvm_get_timer_map,
 		__field(	unsigned long,		vcpu_id	)
 		__field(	int,			direct_vtimer	)
 		__field(	int,			direct_ptimer	)
+		__field(	int,			emul_vtimer	)
 		__field(	int,			emul_ptimer	)
 	),
 
@@ -214,14 +215,17 @@ TRACE_EVENT(kvm_get_timer_map,
 		__entry->direct_vtimer		= arch_timer_ctx_index(map->direct_vtimer);
 		__entry->direct_ptimer =
 			(map->direct_ptimer) ? arch_timer_ctx_index(map->direct_ptimer) : -1;
+		__entry->emul_vtimer =
+			(map->emul_vtimer) ? arch_timer_ctx_index(map->emul_vtimer) : -1;
 		__entry->emul_ptimer =
 			(map->emul_ptimer) ? arch_timer_ctx_index(map->emul_ptimer) : -1;
 	),
 
-	TP_printk("VCPU: %ld, dv: %d, dp: %d, ep: %d",
+	TP_printk("VCPU: %ld, dv: %d, dp: %d, ev: %d, ep: %d",
 		  __entry->vcpu_id,
 		  __entry->direct_vtimer,
 		  __entry->direct_ptimer,
+		  __entry->emul_vtimer,
 		  __entry->emul_ptimer)
 );
 
arch/arm64/kvm/vgic/vgic.c +15 -0
index 5dad4996cfb2..21d088e8597d 100644
@@ -573,6 +573,21 @@ int kvm_vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, unsigned int vintid)
 	return 0;
 }
 
+int kvm_vgic_get_map(struct kvm_vcpu *vcpu, unsigned int vintid)
+{
+	struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, vintid);
+	unsigned long flags;
+	int ret = -1;
+
+	raw_spin_lock_irqsave(&irq->irq_lock, flags);
+	if (irq->hw)
+		ret = irq->hwintid;
+	raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+	vgic_put_irq(vcpu->kvm, irq);
+	return ret;
+}
+
 /**
  * kvm_vgic_set_owner - Set the owner of an interrupt for a VM
  *
include/kvm/arm_arch_timer.h +6 -2
index 51c19381108c..0a76dac8cb6a 100644
@@ -13,6 +13,8 @@
 enum kvm_arch_timers {
 	TIMER_PTIMER,
 	TIMER_VTIMER,
+	TIMER_HVTIMER,
+	TIMER_HPTIMER,
 	NR_KVM_TIMERS
 };
 
@@ -21,6 +23,7 @@ enum kvm_arch_timer_regs {
 	TIMER_REG_CVAL,
 	TIMER_REG_TVAL,
 	TIMER_REG_CTL,
+	TIMER_REG_VOFF,
 };
 
 struct arch_timer_context {
@@ -47,6 +50,7 @@ struct arch_timer_context {
 struct timer_map {
 	struct arch_timer_context *direct_vtimer;
 	struct arch_timer_context *direct_ptimer;
+	struct arch_timer_context *emul_vtimer;
 	struct arch_timer_context *emul_ptimer;
 };
 
@@ -85,12 +89,12 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu);
 
 void kvm_timer_init_vhe(void);
 
-bool kvm_arch_timer_get_input_level(int vintid);
-
 #define vcpu_timer(v)	(&(v)->arch.timer_cpu)
 #define vcpu_get_timer(v,t)	(&vcpu_timer(v)->timers[(t)])
 #define vcpu_vtimer(v)	(&(v)->arch.timer_cpu.timers[TIMER_VTIMER])
 #define vcpu_ptimer(v)	(&(v)->arch.timer_cpu.timers[TIMER_PTIMER])
+#define vcpu_hvtimer(v)	(&(v)->arch.timer_cpu.timers[TIMER_HVTIMER])
+#define vcpu_hptimer(v)	(&(v)->arch.timer_cpu.timers[TIMER_HPTIMER])
 
 #define arch_timer_ctx_index(ctx)	((ctx) - vcpu_timer((ctx)->vcpu)->timers)
 
include/kvm/arm_vgic.h +1 -0
index e602d848fc1a..510122b68d8a 100644
@@ -373,6 +373,7 @@ int kvm_vgic_inject_irq(struct kvm *kvm, int cpuid, unsigned int intid,
 int kvm_vgic_map_phys_irq(struct kvm_vcpu *vcpu, unsigned int host_irq,
 			  u32 vintid, struct irq_ops *ops);
 int kvm_vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, unsigned int vintid);
+int kvm_vgic_get_map(struct kvm_vcpu *vcpu, unsigned int vintid);
 bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int vintid);
 
 int kvm_vgic_vcpu_pending_irq(struct kvm_vcpu *vcpu);
Signature · 6 lines
2.30.2

_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm