Skip to contentSkip to conversation

Fix setting SVE and SME traps in (h)VHE

PATCH Superseded Fuad Tabba 3 participants 11 messages Jul 21, 2023 – Jul 22, 2023

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Fuad Tabba

author
#0 lore raw

v2 cover Fix setting SVE and SME traps in (h)VHE

Hi,

Changes from V1:
- Expanded the cover letter to clarify the reasoning behind being consistent in writing to the architectural trap register based on the KVM mode (Marc)
- Factored out the code for checking E2H into a macro (Oliver)
- Factored out the code that selects which register to write to into a function (Oliver)

The (re)setting and disabling of SVE/SME trap handling (mostly) done for the hVHE work [*] misses a couple of cases.

This patch series ensures that these traps are disabled on setup and reset. Moreover, it makes the code consistent in using CPACR_EL1 or CPTR_EL2, depending on the mode.

CPACR_EL1 aliases to CPTR_EL2 when HCR_EL2.E2H == 1, but by being consistent we don't need to issue a synchronisation when alternating between one or the other accessor. Moreover, when running hVHE under NV, we don't trap unnecessarily on accessing CPTR_EL2, while CPACR_EL1 can be used directly without any trap.

Based on Linux 6.5-rc2.

Cheers,
/fuad

[*] https://lore.kernel.org/all/[email protected]/

Fuad Tabba (7):
KVM: arm64: Factor out code for checking E2H into a macro
KVM: arm64: Use the appropriate feature trap register for SVE at EL2
setup
KVM: arm64: Disable SME traps for (h)VHE at setup
KVM: arm64: Helper to write to appropriate feature trap register based
on mode
KVM: arm64: Use the appropriate feature trap register when activating
traps
KVM: arm64: Fix resetting SVE trap values on reset for hVHE
KVM: arm64: Fix resetting SME trap values on reset for (h)VHE

 arch/arm64/include/asm/el2_setup.h   | 44 ++++++++++++++++++----------
 arch/arm64/include/asm/kvm_emulate.h | 21 ++++++++++---
 arch/arm64/kvm/hyp/nvhe/switch.c     |  2 +-
 3 files changed, 47 insertions(+), 20 deletions(-)

base-commit: fdf0eaf11452d72945af31804e2a1048ee1b574c

Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#1 lore raw

PATCH v2 1/7 KVM: arm64: Factor out code for checking E2H into a macro

The code for checking whether the kernel is in (h)VHE mode is repeated, and will be needed again in future patches. Factor it out in a macro.

No functional change intended.
No change in emitted assembly code intended.

---
 arch/arm64/include/asm/el2_setup.h | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
arch/arm64/include/asm/el2_setup.h +10 -9
index 8e5ffb58f83e..383f6be66ed6 100644
@@ -31,6 +31,13 @@
 .Lskip_hcrx_\@:
 .endm
 
+/* Check if running in host at EL2 mode, i.e., (h)VHE. Jumps to fail if not. */
+.macro __check_e2h fail, tmp
+	mrs	\tmp, hcr_el2
+	and	\tmp, \tmp, #HCR_E2H
+	cbz	\tmp, \fail
+.endm
+
 /*
  * Allow Non-secure EL1 and EL0 to access physical timer and counter.
  * This is not necessary for VHE, since the host kernel runs in EL2,
@@ -43,9 +50,7 @@
  */
 .macro __init_el2_timers
 	mov	x0, #3				// Enable EL1 physical timers
-	mrs	x1, hcr_el2
-	and	x1, x1, #HCR_E2H
-	cbz	x1, .LnVHE_\@
+	__check_e2h .LnVHE_\@, x1
 	lsl	x0, x0, #10
 .LnVHE_\@:
 	msr	cnthctl_el2, x0
@@ -139,9 +144,7 @@
 
 /* Coprocessor traps */
 .macro __init_el2_cptr
-	mrs	x1, hcr_el2
-	and	x1, x1, #HCR_E2H
-	cbz	x1, .LnVHE_\@
+	__check_e2h .LnVHE_\@, x1
 	mov	x0, #(CPACR_EL1_FPEN_EL1EN | CPACR_EL1_FPEN_EL0EN)
 	b	.Lset_cptr_\@
 .LnVHE_\@:
@@ -269,9 +272,7 @@
 
 .Linit_sve_\@:	/* SVE register access */
 	mrs	x0, cptr_el2			// Disable SVE traps
-	mrs	x1, hcr_el2
-	and	x1, x1, #HCR_E2H
-	cbz	x1, .Lcptr_nvhe_\@
+	__check_e2h .Lcptr_nvhe_\@, x1
 
 	// VHE case
 	orr	x0, x0, #(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN)
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
#2 lore raw

Re: [PATCH v2 1/7] KVM: arm64: Factor out code for checking E2H into a macro

Hi Fuad,

Quoted 8 lines · Fuad Tabba

The code for checking whether the kernel is in (h)VHE mode is repeated, and will be needed again in future patches. Factor it out in a macro.

No functional change intended.
No change in emitted assembly code intended.

Looks good, one nitpick below:

Quoted 9 lines
---
 arch/arm64/include/asm/el2_setup.h | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
arch/arm64/include/asm/el2_setup.h +0 -0
index 8e5ffb58f83e..383f6be66ed6 100644

arch/arm64/include/asm/el2_setup.h line 40

Quoted 10 lines
arch/arm64/include/asm/el2_setup.h +7 -0
@@ -31,6 +31,13 @@
 .Lskip_hcrx_\@:
 .endm
 
+/* Check if running in host at EL2 mode, i.e., (h)VHE. Jumps to fail if not. */
+.macro __check_e2h fail, tmp
+	mrs	\tmp, hcr_el2
+	and	\tmp, \tmp, #HCR_E2H
+	cbz	\tmp, \fail
+.endm
+

I was really hoping for the macro to have hVHE somewhere in the name, since it may not be immediately obvious to folks why we're testing E2H in this context.

So maybe: s/__check_e2h/__check_hvhe/ ?

I can fix it up when applying the series if you agree.

Signature · 2 lines
Thanks,
Oliver
author
#3 lore raw

Re: [PATCH v2 1/7] KVM: arm64: Factor out code for checking E2H into a macro

Hi Oliver,

Quoted 39 lines · Oliver Upton

Hi Fuad,

Quoted 8 lines · Fuad Tabba

The code for checking whether the kernel is in (h)VHE mode is repeated, and will be needed again in future patches. Factor it out in a macro.

No functional change intended.
No change in emitted assembly code intended.

Looks good, one nitpick below:

Quoted 9 lines
---
 arch/arm64/include/asm/el2_setup.h | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
arch/arm64/include/asm/el2_setup.h +0 -0
index 8e5ffb58f83e..383f6be66ed6 100644

arch/arm64/include/asm/el2_setup.h line 40

Quoted 10 lines
arch/arm64/include/asm/el2_setup.h +7 -0
@@ -31,6 +31,13 @@
 .Lskip_hcrx_\@:
 .endm
 
+/* Check if running in host at EL2 mode, i.e., (h)VHE. Jumps to fail if not. */
+.macro __check_e2h fail, tmp
+     mrs     \tmp, hcr_el2
+     and     \tmp, \tmp, #HCR_E2H
+     cbz     \tmp, \fail
+.endm
+

I was really hoping for the macro to have hVHE somewhere in the name, since it may not be immediately obvious to folks why we're testing E2H in this context.

Quoted 3 lines

So maybe: s/__check_e2h/__check_hvhe/ ?

I can fix it up when applying the series if you agree.

I agree. Thanks for doing that!

Cheers,
/fuad

Quoted 4 lines

--
Thanks,
Oliver

author
#4 lore raw

PATCH v2 2/7 KVM: arm64: Use the appropriate feature trap register for SVE at EL2 setup

In reply to Fuad Tabba

Use the architectural feature trap/control register that corresponds to the current KVM mode, i.e., CPTR_EL2 or CPACR_EL1, when setting up SVE feature traps.

---
 arch/arm64/include/asm/el2_setup.h | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
arch/arm64/include/asm/el2_setup.h +9 -6
index 383f6be66ed6..1f3d0aa6d5f5 100644
@@ -146,11 +146,12 @@
 .macro __init_el2_cptr
 	__check_e2h .LnVHE_\@, x1
 	mov	x0, #(CPACR_EL1_FPEN_EL1EN | CPACR_EL1_FPEN_EL0EN)
-	b	.Lset_cptr_\@
+	msr	cpacr_el1, x0
+	b	.Lskip_set_cptr_\@
 .LnVHE_\@:
 	mov	x0, #0x33ff
-.Lset_cptr_\@:
 	msr	cptr_el2, x0			// Disable copro. traps to EL2
+.Lskip_set_cptr_\@:
 .endm
 
 /* Disable any fine grained traps */
@@ -271,17 +272,19 @@
 	check_override id_aa64pfr0, ID_AA64PFR0_EL1_SVE_SHIFT, .Linit_sve_\@, .Lskip_sve_\@, x1, x2
 
 .Linit_sve_\@:	/* SVE register access */
-	mrs	x0, cptr_el2			// Disable SVE traps
 	__check_e2h .Lcptr_nvhe_\@, x1
 
-	// VHE case
+	// (h)VHE case
+	mrs	x0, cpacr_el1			// Disable SVE traps
 	orr	x0, x0, #(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN)
-	b	.Lset_cptr_\@
+	msr	cpacr_el1, x0
+	b	.Lskip_set_cptr_\@
 
 .Lcptr_nvhe_\@: // nVHE case
+	mrs	x0, cptr_el2			// Disable SVE traps
 	bic	x0, x0, #CPTR_EL2_TZ
-.Lset_cptr_\@:
 	msr	cptr_el2, x0
+.Lskip_set_cptr_\@:
 	isb
 	mov	x1, #ZCR_ELx_LEN_MASK		// SVE: Enable full vector
 	msr_s	SYS_ZCR_EL2, x1			// length for EL1.
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#5 lore raw

PATCH v2 3/7 KVM: arm64: Disable SME traps for (h)VHE at setup

In reply to Fuad Tabba

Ensure that SME traps are disabled for (h)VHE when setting up EL2, as they are for nVHE.

---
 arch/arm64/include/asm/el2_setup.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/arm64/include/asm/el2_setup.h +10 -0
index 1f3d0aa6d5f5..3bd2e84daeef 100644
@@ -293,9 +293,19 @@
 	check_override id_aa64pfr1, ID_AA64PFR1_EL1_SME_SHIFT, .Linit_sme_\@, .Lskip_sme_\@, x1, x2
 
 .Linit_sme_\@:	/* SME register access and priority mapping */
+	__check_e2h .Lcptr_nvhe_sme_\@, x1
+
+	// (h)VHE case
+	mrs	x0, cpacr_el1			// Disable SME traps
+	orr	x0, x0, #(CPACR_EL1_SMEN_EL0EN | CPACR_EL1_SMEN_EL1EN)
+	msr	cpacr_el1, x0
+	b	.Lskip_set_cptr_sme_\@
+
+.Lcptr_nvhe_sme_\@: // nVHE case
 	mrs	x0, cptr_el2			// Disable SME traps
 	bic	x0, x0, #CPTR_EL2_TSM
 	msr	cptr_el2, x0
+.Lskip_set_cptr_sme_\@:
 	isb
 
 	mrs	x1, sctlr_el2
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#6 lore raw

PATCH v2 4/7 KVM: arm64: Helper to write to appropriate feature trap register based on mode

In reply to Fuad Tabba

Factor out the code that decides whether to write to the feature trap registers, CPTR_EL2 or CPACR_EL1, based on the KVM mode, i.e., (h)VHE or nVHE.

This function will be used in the subsequent patch.

No functional change intended.

---
 arch/arm64/include/asm/kvm_emulate.h | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +9 -4
index efc0b45d79c3..f5941f6dce49 100644
@@ -571,6 +571,14 @@ static inline bool vcpu_has_feature(struct kvm_vcpu *vcpu, int feature)
 	return test_bit(feature, vcpu->arch.features);
 }
 
+static __always_inline void kvm_write_cptr_el2(u64 val)
+{
+	if (has_vhe() || has_hvhe())
+		write_sysreg(val, cpacr_el1);
+	else
+		write_sysreg(val, cptr_el2);
+}
+
 static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu)
 {
 	u64 val;
@@ -597,9 +605,6 @@ static __always_inline void kvm_reset_cptr_el2(struct kvm_vcpu *vcpu)
 {
 	u64 val = kvm_get_reset_cptr_el2(vcpu);
 
-	if (has_vhe() || has_hvhe())
-		write_sysreg(val, cpacr_el1);
-	else
-		write_sysreg(val, cptr_el2);
+	kvm_write_cptr_el2(val);
 }
 #endif /* __ARM64_KVM_EMULATE_H__ */
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#7 lore raw

PATCH v2 5/7 KVM: arm64: Use the appropriate feature trap register when activating traps

In reply to Fuad Tabba

Instead of writing directly to cptr_el2, use the helper that selects which feature trap register to write to based on the KVM mode.

Fixes 75c76ab5a641 ("KVM: arm64: Rework CPTR_EL2 programming for HVHE configuration")
---
 arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/hyp/nvhe/switch.c +1 -1
index 0a6271052def..e89a23153e85 100644
@@ -63,7 +63,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
 		__activate_traps_fpsimd32(vcpu);
 	}
 
-	write_sysreg(val, cptr_el2);
+	kvm_write_cptr_el2(val);
 	write_sysreg(__this_cpu_read(kvm_hyp_vector), vbar_el2);
 
 	if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#8 lore raw

PATCH v2 6/7 KVM: arm64: Fix resetting SVE trap values on reset for hVHE

In reply to Fuad Tabba

Ensure that SVE traps are disabled for hVHE, if the FPSIMD state isn't owned by the guest, when getting the reset value for the architectural feature control register.

Fixes 75c76ab5a641 ("KVM: arm64: Rework CPTR_EL2 programming for HVHE configuration")
---
 arch/arm64/include/asm/kvm_emulate.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +4 -0
index f5941f6dce49..c04ff63fedab 100644
@@ -588,6 +588,10 @@ static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu)
 		       CPACR_EL1_ZEN_EL1EN);
 	} else if (has_hvhe()) {
 		val = (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN);
+
+		if (vcpu_has_sve(vcpu) &&
+		    (vcpu->arch.fp_state != FP_STATE_GUEST_OWNED))
+			val |= CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN;
 	} else {
 		val = CPTR_NVHE_EL2_RES1;
 
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#9 lore raw

PATCH v2 7/7 KVM: arm64: Fix resetting SME trap values on reset for (h)VHE

In reply to Fuad Tabba

Ensure that SME traps are disabled for (h)VHE when getting the reset value for the architectural feature control register.

Fixes 75c76ab5a641 ("KVM: arm64: Rework CPTR_EL2 programming for HVHE configuration")
---
 arch/arm64/include/asm/kvm_emulate.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +4 -0
index c04ff63fedab..8feba4446f49 100644
@@ -586,12 +586,16 @@ static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu)
 	if (has_vhe()) {
 		val = (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN |
 		       CPACR_EL1_ZEN_EL1EN);
+		if (cpus_have_final_cap(ARM64_SME))
+			val |= CPACR_EL1_SMEN_EL1EN;
 	} else if (has_hvhe()) {
 		val = (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN);
 
 		if (vcpu_has_sve(vcpu) &&
 		    (vcpu->arch.fp_state != FP_STATE_GUEST_OWNED))
 			val |= CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN;
+		if (cpus_have_final_cap(ARM64_SME))
+			val |= CPACR_EL1_SMEN_EL1EN | CPACR_EL1_SMEN_EL0EN;
 	} else {
 		val = CPTR_NVHE_EL2_RES1;
 
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
maintainer Reviewed-by on the v2 series
#10 lore raw
In reply to Fuad Tabba

Hey Fuad,

Quoted 43 lines · Fuad Tabba

Hi,

Changes from V1:
- Expanded the cover letter to clarify the reasoning behind being consistent in writing to the architectural trap register based on the KVM mode (Marc)
- Factored out the code for checking E2H into a macro (Oliver)
- Factored out the code that selects which register to write to into a function (Oliver)

The (re)setting and disabling of SVE/SME trap handling (mostly) done for the hVHE work [*] misses a couple of cases.

This patch series ensures that these traps are disabled on setup and reset. Moreover, it makes the code consistent in using CPACR_EL1 or CPTR_EL2, depending on the mode.

CPACR_EL1 aliases to CPTR_EL2 when HCR_EL2.E2H == 1, but by being consistent we don't need to issue a synchronisation when alternating between one or the other accessor. Moreover, when running hVHE under NV, we don't trap unnecessarily on accessing CPTR_EL2, while CPACR_EL1 can be used directly without any trap.

Based on Linux 6.5-rc2.

Cheers,
/fuad

[*] https://lore.kernel.org/all/[email protected]/

Fuad Tabba (7):
KVM: arm64: Factor out code for checking E2H into a macro
KVM: arm64: Use the appropriate feature trap register for SVE at EL2
setup
KVM: arm64: Disable SME traps for (h)VHE at setup
KVM: arm64: Helper to write to appropriate feature trap register based
on mode
KVM: arm64: Use the appropriate feature trap register when activating
traps
KVM: arm64: Fix resetting SVE trap values on reset for hVHE
KVM: arm64: Fix resetting SME trap values on reset for (h)VHE

Quoted 4 lines
 arch/arm64/include/asm/el2_setup.h   | 44 ++++++++++++++++++----------
 arch/arm64/include/asm/kvm_emulate.h | 21 ++++++++++---
 arch/arm64/kvm/hyp/nvhe/switch.c     |  2 +-
 3 files changed, 47 insertions(+), 20 deletions(-)

Thanks again for fixing this, and going the extra mile on the cleanup front. FWIW,

	     M.
Signature · 1 line
Without deviation from the norm, progress is not possible.