Skip to contentSkip to conversation

KVM: arm64: Host EL2 entry improvements

PATCH In review Marc Zyngier 2 participants 8 messages Nov 9, 2020 – Nov 20, 2020

Subsystems (3)
KVM arm64 KVM arm64
lore
Share & tools
lore

Marc Zyngier

author maintainer
#0 lore raw

v2 cover KVM: arm64: Host EL2 entry improvements

This small series reworks various bits of the host EL2 entry after Andrew's extensive rework to move from direct function calls to a SMCCC implementation.

The first 2 patches allow the use of direct function pointers at EL2, something that we can't do at the moment (other than PC-relative addressing). This requires a helper to translate pointers at runtime, but the result is neat enough. This allows the rewrite of the host HVC handling in a more maintainable way.

Note that this version now includes the result of a discussion with Nick, providing some funky pointer mangling in order to make the use of these pointers vaguely safer (no, they are not safe at all).

Another patch removes the direct use of kimage_voffset, which we won't be able to trust for much longer.

The last two patches are just cleanups and optimisations.

* From v1 [1]:
- Merged the 3 first patches as fixes
- Added pointer mangling for function calls
- Moved EL2 entry ldp a couple of instructions later (as suggested by Alex)
- Rebased on top of -rc3

[1] https://lore.kernel.org/r/[email protected]

Marc Zyngier (5):
KVM: arm64: Add kimg_hyp_va() helper
KVM: arm64: Turn host HVC handling into a dispatch table
KVM: arm64: Patch kimage_voffset instead of loading the EL1 value
KVM: arm64: Simplify __kvm_enable_ssbs()
KVM: arm64: Avoid repetitive stack access on host EL1 to EL2 exception

 arch/arm64/include/asm/kvm_asm.h    |   2 -
 arch/arm64/include/asm/kvm_mmu.h    |  40 +++++
 arch/arm64/include/asm/sysreg.h     |   1 +
 arch/arm64/kernel/image-vars.h      |   5 +-
 arch/arm64/kvm/hyp/nvhe/host.S      |  11 +-
 arch/arm64/kvm/hyp/nvhe/hyp-main.c  | 232 +++++++++++++++++-----------
 arch/arm64/kvm/hyp/nvhe/sysreg-sr.c |  11 --
 arch/arm64/kvm/va_layout.c          |  56 +++++++
 8 files changed, 241 insertions(+), 117 deletions(-)
Signature · 1 line
2.28.0
author maintainer
#1 lore raw

PATCH v2 1/5 KVM: arm64: Add kimg_hyp_va() helper

KVM/arm64 is so far unable to deal with function pointers, as the compiler will generate the kernel's runtime VA, and not the linear mapping address, meaning that kern_hyp_va() will give the wrong result.

We so far have been able to use PC-relative addressing, but that's not always easy to use, and prevents the implementation of things like the mapping of an index to a pointer.

To allow this, provide a new helper that computes the required translation from the kernel image to the HYP VA space.

Signed-off-by Marc Zyngier <[email protected]>
---
 arch/arm64/include/asm/kvm_mmu.h | 18 ++++++++++++
 arch/arm64/kvm/va_layout.c       | 50 ++++++++++++++++++++++++++++++++
 2 files changed, 68 insertions(+)
arch/arm64/include/asm/kvm_mmu.h +18 -0
index 331394306cce..608c3a83e740 100644
@@ -98,6 +98,24 @@ static __always_inline unsigned long __kern_hyp_va(unsigned long v)
 
 #define kern_hyp_va(v) 	((typeof(v))(__kern_hyp_va((unsigned long)(v))))
 
+static __always_inline unsigned long __kimg_hyp_va(unsigned long v)
+{
+	unsigned long offset;
+
+	asm volatile(ALTERNATIVE_CB("movz %0, #0\n"
+				    "movk %0, #0, lsl #16\n"
+				    "movk %0, #0, lsl #32\n"
+				    "movk %0, #0, lsl #48\n",
+				    kvm_update_kimg_phys_offset)
+		     : "=r" (offset));
+
+	return __kern_hyp_va((v - offset) | PAGE_OFFSET);
+}
+
+#define kimg_fn_hyp_va(v) 	((typeof(*v))(__kimg_hyp_va((unsigned long)(v))))
+
+#define kimg_fn_ptr(x)	(typeof(x) **)(x)
+
 /*
  * We currently support using a VM-specified IPA size. For backward
  * compatibility, the default IPA size is fixed to 40bits.
arch/arm64/kvm/va_layout.c +50 -0

59 lines not on this page · raw message

Signature · 1 line
2.28.0
author maintainer
#2 lore raw

PATCH v2 2/5 KVM: arm64: Turn host HVC handling into a dispatch table

In reply to Marc Zyngier

Now that we can use function pointer, use a dispatch table to call the individual HVC handlers, leading to more maintainable code.

Further improvements include helpers to declare the mapping of local variables to values passed in the host context.

Reviewed-by Alexandru Elisei <[email protected]>
Signed-off-by Marc Zyngier <[email protected]>
---
 arch/arm64/kernel/image-vars.h     |   1 +
 arch/arm64/kvm/hyp/nvhe/hyp-main.c | 228 +++++++++++++++++------------
 2 files changed, 135 insertions(+), 94 deletions(-)
arch/arm64/kernel/image-vars.h +1 -0
index c615b285ff5b..e8c194f8de88 100644
@@ -64,6 +64,7 @@ __efistub__ctype		= _ctype;
 /* Alternative callbacks for init-time patching of nVHE hyp code. */
 KVM_NVHE_ALIAS(kvm_patch_vector_branch);
 KVM_NVHE_ALIAS(kvm_update_va_mask);
+KVM_NVHE_ALIAS(kvm_update_kimg_phys_offset);
 
 /* Global kernel state accessed by nVHE hyp code. */
 KVM_NVHE_ALIAS(kvm_vgic_global_state);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +134 -94

240 lines not on this page · raw message

Signature · 1 line
2.28.0
author maintainer
#3 lore raw

PATCH v2 3/5 KVM: arm64: Patch kimage_voffset instead of loading the EL1 value

In reply to Marc Zyngier

Directly using the kimage_voffset variable is fine for now, but will become more problematic as we start distrusting EL1.

Instead, patch the kimage_voffset into the HYP text, ensuring we don't have to load an untrusted value later on.

Signed-off-by Marc Zyngier <[email protected]>
---
 arch/arm64/include/asm/kvm_mmu.h | 22 ++++++++++++++++++++++
 arch/arm64/kernel/image-vars.h   |  4 +---
 arch/arm64/kvm/hyp/nvhe/host.S   |  5 +----
 arch/arm64/kvm/va_layout.c       |  6 ++++++
 4 files changed, 30 insertions(+), 7 deletions(-)
arch/arm64/include/asm/kvm_mmu.h +22 -0
index 608c3a83e740..400d2d73ce12 100644
@@ -72,6 +72,28 @@ alternative_cb kvm_update_va_mask
 alternative_cb_end
 .endm
 
+/*
+ * Convert a kernel image address to a PA
+ * reg: kernel address to be converted in place
+ * tmp: temporary register
+ *
+ * The actual code generation takes place in kvm_get_kimage_voffset, and
+ * the instructions below are only there to reserve the space and
+ * perform the register allocation (kvm_get_kimage_voffset uses the
+ * specific registers encoded in the instructions).
+ */
+.macro kimg_pa reg, tmp
+alternative_cb kvm_get_kimage_voffset
+       movz    \tmp, #0
+       movk    \tmp, #0, lsl #16
+       movk    \tmp, #0, lsl #32
+       movk    \tmp, #0, lsl #48
+alternative_cb_end
+
+       /* reg = __pa(reg) */
+       sub     \reg, \reg, \tmp
+.endm
+	 
 #else
 
 #include <linux/pgtable.h>
arch/arm64/kernel/image-vars.h +1 -3
index e8c194f8de88..4b32588918d9 100644
@@ -65,13 +65,11 @@ __efistub__ctype		= _ctype;
 KVM_NVHE_ALIAS(kvm_patch_vector_branch);
 KVM_NVHE_ALIAS(kvm_update_va_mask);
 KVM_NVHE_ALIAS(kvm_update_kimg_phys_offset);
+KVM_NVHE_ALIAS(kvm_get_kimage_voffset);
 
 /* Global kernel state accessed by nVHE hyp code. */
 KVM_NVHE_ALIAS(kvm_vgic_global_state);
 
-/* Kernel constant needed to compute idmap addresses. */
-KVM_NVHE_ALIAS(kimage_voffset);
-
 /* Kernel symbols used to call panic() from nVHE hyp code (via ERET). */
 KVM_NVHE_ALIAS(__hyp_panic_string);
 KVM_NVHE_ALIAS(panic);
arch/arm64/kvm/hyp/nvhe/host.S +1 -4
index ed27f06a31ba..4e207c1c5126 100644
@@ -115,10 +115,7 @@ SYM_FUNC_END(__hyp_do_panic)
 	 * Preserve x0-x4, which may contain stub parameters.
 	 */
 	ldr	x5, =__kvm_handle_stub_hvc
-	ldr_l	x6, kimage_voffset
-
-	/* x5 = __pa(x5) */
-	sub	x5, x5, x6
+	kimg_pa x5, x6
 	br	x5
 .L__vect_end\@:
 .if ((.L__vect_end\@ - .L__vect_start\@) > 0x80)
arch/arm64/kvm/va_layout.c +6 -0
index 1d00d2cb93fd..d61117805de0 100644
@@ -251,3 +251,9 @@ void kvm_update_kimg_phys_offset(struct alt_instr *alt,
 {
 	generate_mov_q(kimage_voffset + PHYS_OFFSET, origptr, updptr, nr_inst);
 }
+
+void kvm_get_kimage_voffset(struct alt_instr *alt,
+			    __le32 *origptr, __le32 *updptr, int nr_inst)
+{
+	generate_mov_q(kimage_voffset, origptr, updptr, nr_inst);
+}
Signature · 1 line
2.28.0
#4 lore raw

Re: [PATCH v2 3/5] KVM: arm64: Patch kimage_voffset instead of loading the EL1 value

Hey Marc,

Just noticed in kvmarm/queue that the whitespacing in this patch is off.

arch/arm64/include/asm/kvm_mmu.h alternative_cb kvm_update_va_mask · line 95

Quoted 11 lines

+.macro kimg_pa reg, tmp

arch/arm64/include/asm/kvm_mmu.h +10 -0
+alternative_cb kvm_get_kimage_voffset
+       movz    \tmp, #0
+       movk    \tmp, #0, lsl #16
+       movk    \tmp, #0, lsl #32
+       movk    \tmp, #0, lsl #48
+alternative_cb_end
+
+       /* reg = __pa(reg) */
+       sub     \reg, \reg, \tmp
+.endm

This uses spaces instead of tabs.

Review

Quoted 2 lines
(quoted diff) +1 -0
+	 
 #else

This added empty line actually has a tab in it.

-David
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

author maintainer
#5 lore raw

Re: [PATCH v2 3/5] KVM: arm64: Patch kimage_voffset instead of loading the EL1 value

Quoted 17 lines · David Brazdil

Hey Marc,

Just noticed in kvmarm/queue that the whitespacing in this patch is off.

arch/arm64/include/asm/kvm_mmu.h alternative_cb kvm_update_va_mask · line 95

Quoted 11 lines

+.macro kimg_pa reg, tmp

arch/arm64/include/asm/kvm_mmu.h +10 -0
+alternative_cb kvm_get_kimage_voffset
+       movz    \tmp, #0
+       movk    \tmp, #0, lsl #16
+       movk    \tmp, #0, lsl #32
+       movk    \tmp, #0, lsl #48
+alternative_cb_end
+
+       /* reg = __pa(reg) */
+       sub     \reg, \reg, \tmp
+.endm

This uses spaces instead of tabs.

Quoted 3 lines
Quoted 2 lines

+
#else

This added empty line actually has a tab in it.

Well spotted. Now fixed.

Thanks,

        M.
Signature · 1 line
Jazz is not dead. It just smells funny...
author maintainer
#6 lore raw

PATCH v2 4/5 KVM: arm64: Simplify __kvm_enable_ssbs()

In reply to Marc Zyngier

Move the setting of SSBS directly into the HVC handler, using the C helpers rather than the inline asssembly code.

Reviewed-by Alexandru Elisei <[email protected]>
Signed-off-by Marc Zyngier <[email protected]>
---
 arch/arm64/include/asm/kvm_asm.h    |  2 --
 arch/arm64/include/asm/sysreg.h     |  1 +
 arch/arm64/kvm/hyp/nvhe/hyp-main.c  |  6 +++++-
 arch/arm64/kvm/hyp/nvhe/sysreg-sr.c | 11 -----------
 4 files changed, 6 insertions(+), 14 deletions(-)
arch/arm64/include/asm/kvm_asm.h +0 -2
index 54387ccd1ab2..a542c422a036 100644
@@ -189,8 +189,6 @@ extern void __kvm_timer_set_cntvoff(u64 cntvoff);
 
 extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
 
-extern void __kvm_enable_ssbs(void);
-
 extern u64 __vgic_v3_get_ich_vtr_el2(void);
 extern u64 __vgic_v3_read_vmcr(void);
 extern void __vgic_v3_write_vmcr(u32 vmcr);
arch/arm64/include/asm/sysreg.h +1 -0
index 174817ba119c..153faa15d490 100644
@@ -461,6 +461,7 @@
 
 #define SYS_PMCCFILTR_EL0		sys_reg(3, 3, 14, 15, 7)
 
+#define SYS_SCTLR_EL2			sys_reg(3, 4, 1, 0, 0)
 #define SYS_ZCR_EL2			sys_reg(3, 4, 1, 2, 0)
 #define SYS_DACR32_EL2			sys_reg(3, 4, 3, 0, 0)
 #define SYS_SPSR_EL2			sys_reg(3, 4, 4, 0, 0)
arch/arm64/kvm/hyp/nvhe/hyp-main.c +5 -1
index c0543b2e760e..82df7fc24760 100644
@@ -58,7 +58,11 @@ static void handle___kvm_timer_set_cntvoff(struct kvm_cpu_context *host_ctxt)
 
 static void handle___kvm_enable_ssbs(struct kvm_cpu_context *host_ctxt)
 {
-	__kvm_enable_ssbs();
+	u64 tmp;
+
+	tmp = read_sysreg_el2(SYS_SCTLR);
+	tmp |= SCTLR_ELx_DSSBS;
+	write_sysreg_el2(tmp, SYS_SCTLR);
 }
 
 static void handle___vgic_v3_get_ich_vtr_el2(struct kvm_cpu_context *host_ctxt)
arch/arm64/kvm/hyp/nvhe/sysreg-sr.c +0 -11
index 88a25fc8fcd3..29305022bc04 100644
@@ -33,14 +33,3 @@ void __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt)
 	__sysreg_restore_user_state(ctxt);
 	__sysreg_restore_el2_return_state(ctxt);
 }
-
-void __kvm_enable_ssbs(void)
-{
-	u64 tmp;
-
-	asm volatile(
-	"mrs	%0, sctlr_el2\n"
-	"orr	%0, %0, %1\n"
-	"msr	sctlr_el2, %0"
-	: "=&r" (tmp) : "L" (SCTLR_ELx_DSSBS));
-}
Signature · 1 line
2.28.0
author maintainer
#7 lore raw

PATCH v2 5/5 KVM: arm64: Avoid repetitive stack access on host EL1 to EL2 exception

In reply to Marc Zyngier

Registers x0/x1 get repeateadly pushed and poped during a host HVC call. Instead, leave the registers on the stack, trading a store instruction on the fast path for an add on the slow path.

Reviewed-by Alexandru Elisei <[email protected]>
Signed-off-by Marc Zyngier <[email protected]>
---
 arch/arm64/kvm/hyp/nvhe/host.S | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm64/kvm/hyp/nvhe/host.S +3 -3
index 4e207c1c5126..fe2740b224cf 100644
@@ -13,8 +13,6 @@
 	.text
 
 SYM_FUNC_START(__host_exit)
-	stp	x0, x1, [sp, #-16]!
-
 	get_host_ctxt	x0, x1
 
 	/* Store the host regs x2 and x3 */

@@ -99,13 +97,15 @@ SYM_FUNC_END(__hyp_do_panic)
 	mrs	x0, esr_el2
 	lsr	x0, x0, #ESR_ELx_EC_SHIFT
 	cmp	x0, #ESR_ELx_EC_HVC64
-	ldp	x0, x1, [sp], #16
 	b.ne	__host_exit
 
+	ldp	x0, x1, [sp]		// Don't fixup the stack yet
+
 	/* Check for a stub HVC call */
 	cmp	x0, #HVC_STUB_HCALL_NR
 	b.hs	__host_exit
 
+	add	sp, sp, #16
 	/*
 	 * Compute the idmap address of __kvm_handle_stub_hvc and
 	 * jump there. Since we use kimage_voffset, do not use the
Signature · 1 line
2.28.0