Skip to contentSkip to conversation

Fix setting SVE and SME traps in (h)VHE

PATCH Applied Fuad Tabba 3 participants 12 messages Jul 24, 2023 – Jul 26, 2023

Subsystems (2)
lore
Share & tools
lore

Fuad Tabba

author
#0 lore raw

v3 cover Fix setting SVE and SME traps in (h)VHE

Hi,

Changes from V2:
- Rebased onto Linux 6.5-rc3
- s/__check_e2h/__check_hvhe/g (Oliver)
- Fixed bug in calculating CPACR_EL1 reset value for hVHE in trapping SVE

Changes from V1:
- Expanded the cover letter to clarify the reasoning behind being consistent in writing to the architectural trap register based on the KVM mode (Marc)
- Factored out the code for checking E2H into a macro (Oliver)
- Factored out the code that selects which register to write to into a function (Oliver)

The (re)setting and disabling of SVE/SME trap handling (mostly) done for the hVHE work [*] misses a couple of cases.

This patch series ensures that these traps are disabled on setup and reset. Moreover, it makes the code consistent in using CPACR_EL1 or CPTR_EL2, depending on the mode.

CPACR_EL1 aliases to CPTR_EL2 when HCR_EL2.E2H == 1, but by being consistent we don't need to issue a synchronisation when alternating between one or the other accessor. Moreover, when running hVHE under NV, we don't trap unnecessarily on accessing CPTR_EL2, while CPACR_EL1 can be used directly without any trap.

Based on Linux 6.5-rc3.

Cheers,
/fuad

Fuad Tabba (7):
KVM: arm64: Factor out code for checking (h)VHE mode into a macro
KVM: arm64: Use the appropriate feature trap register for SVE at EL2
setup
KVM: arm64: Disable SME traps for (h)VHE at setup
KVM: arm64: Helper to write to appropriate feature trap register based
on mode
KVM: arm64: Use the appropriate feature trap register when activating
traps
KVM: arm64: Fix resetting SVE trap values on reset for hVHE
KVM: arm64: Fix resetting SME trap values on reset for (h)VHE

 arch/arm64/include/asm/el2_setup.h   | 44 ++++++++++++++++++----------
 arch/arm64/include/asm/kvm_emulate.h | 21 ++++++++++---
 arch/arm64/kvm/hyp/nvhe/switch.c     |  2 +-
 3 files changed, 47 insertions(+), 20 deletions(-)

base-commit: 6eaae198076080886b9e7d57f4ae06fa782f90ef

Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#1 lore raw

PATCH v3 1/7 KVM: arm64: Factor out code for checking E2H into a macro

The code for checking whether the kernel is in (h)VHE mode is repeated, and will be needed again in future patches. Factor it out in a macro.

No functional change intended.
No change in emitted assembly code intended.

---
 arch/arm64/include/asm/el2_setup.h | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
arch/arm64/include/asm/el2_setup.h +10 -9
index 8e5ffb58f83e..16d3bafa715d 100644
@@ -31,6 +31,13 @@
 .Lskip_hcrx_\@:
 .endm
 
+/* Check if running in host at EL2 mode, i.e., (h)VHE. Jump to fail if not. */
+.macro __check_hvhe fail, tmp
+	mrs	\tmp, hcr_el2
+	and	\tmp, \tmp, #HCR_E2H
+	cbz	\tmp, \fail
+.endm
+
 /*
  * Allow Non-secure EL1 and EL0 to access physical timer and counter.
  * This is not necessary for VHE, since the host kernel runs in EL2,
@@ -43,9 +50,7 @@
  */
 .macro __init_el2_timers
 	mov	x0, #3				// Enable EL1 physical timers
-	mrs	x1, hcr_el2
-	and	x1, x1, #HCR_E2H
-	cbz	x1, .LnVHE_\@
+	__check_hvhe .LnVHE_\@, x1
 	lsl	x0, x0, #10
 .LnVHE_\@:
 	msr	cnthctl_el2, x0
@@ -139,9 +144,7 @@
 
 /* Coprocessor traps */
 .macro __init_el2_cptr
-	mrs	x1, hcr_el2
-	and	x1, x1, #HCR_E2H
-	cbz	x1, .LnVHE_\@
+	__check_hvhe .LnVHE_\@, x1
 	mov	x0, #(CPACR_EL1_FPEN_EL1EN | CPACR_EL1_FPEN_EL0EN)
 	b	.Lset_cptr_\@
 .LnVHE_\@:
@@ -269,9 +272,7 @@
 
 .Linit_sve_\@:	/* SVE register access */
 	mrs	x0, cptr_el2			// Disable SVE traps
-	mrs	x1, hcr_el2
-	and	x1, x1, #HCR_E2H
-	cbz	x1, .Lcptr_nvhe_\@
+	__check_hvhe .Lcptr_nvhe_\@, x1
 
 	// VHE case
 	orr	x0, x0, #(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN)
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#2 lore raw

Re: [PATCH v3 1/7] KVM: arm64: Factor out code for checking E2H into a macro

Apologies for this one. I changed the subject of this patch without deleting it from my patches folder before sending, hence the duplication.

/fuad

Quoted 67 lines · Fuad Tabba

The code for checking whether the kernel is in (h)VHE mode is repeated, and will be needed again in future patches. Factor it out in a macro.

No functional change intended.
No change in emitted assembly code intended.

---
 arch/arm64/include/asm/el2_setup.h | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
arch/arm64/include/asm/el2_setup.h +10 -10
index 8e5ffb58f83e..16d3bafa715d 100644
@@ -31,6 +31,13 @@
 .Lskip_hcrx_\@:
 .endm
 
+/* Check if running in host at EL2 mode, i.e., (h)VHE. Jump to fail if not. */
+.macro __check_hvhe fail, tmp
+       mrs     \tmp, hcr_el2
+       and     \tmp, \tmp, #HCR_E2H
+       cbz     \tmp, \fail
+.endm
+
 /*
  * Allow Non-secure EL1 and EL0 to access physical timer and counter.
  * This is not necessary for VHE, since the host kernel runs in EL2,
@@ -43,9 +50,7 @@
  */
 .macro __init_el2_timers
        mov     x0, #3                          // Enable EL1 physical timers
-       mrs     x1, hcr_el2
-       and     x1, x1, #HCR_E2H
-       cbz     x1, .LnVHE_\@
+       __check_hvhe .LnVHE_\@, x1
        lsl     x0, x0, #10
 .LnVHE_\@:
        msr     cnthctl_el2, x0
@@ -139,9 +144,7 @@
 
 /* Coprocessor traps */
 .macro __init_el2_cptr
-       mrs     x1, hcr_el2
-       and     x1, x1, #HCR_E2H
-       cbz     x1, .LnVHE_\@
+       __check_hvhe .LnVHE_\@, x1
        mov     x0, #(CPACR_EL1_FPEN_EL1EN | CPACR_EL1_FPEN_EL0EN)
        b       .Lset_cptr_\@
 .LnVHE_\@:
@@ -269,9 +272,7 @@
 
 .Linit_sve_\@: /* SVE register access */
        mrs     x0, cptr_el2                    // Disable SVE traps
-       mrs     x1, hcr_el2
-       and     x1, x1, #HCR_E2H
-       cbz     x1, .Lcptr_nvhe_\@
+       __check_hvhe .Lcptr_nvhe_\@, x1
 
        // VHE case
        orr     x0, x0, #(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN)
--

2.41.0.487.g6d72f3e995-goog

author
#3 lore raw

PATCH v3 1/7 KVM: arm64: Factor out code for checking (h)VHE mode into a macro

In reply to Fuad Tabba

The code for checking whether the kernel is in (h)VHE mode is repeated, and will be needed again in future patches. Factor it out in a macro.

No functional change intended.
No change in emitted assembly code intended.

---
 arch/arm64/include/asm/el2_setup.h | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
arch/arm64/include/asm/el2_setup.h +10 -9
index 8e5ffb58f83e..16d3bafa715d 100644
@@ -31,6 +31,13 @@
 .Lskip_hcrx_\@:
 .endm
 
+/* Check if running in host at EL2 mode, i.e., (h)VHE. Jump to fail if not. */
+.macro __check_hvhe fail, tmp
+	mrs	\tmp, hcr_el2
+	and	\tmp, \tmp, #HCR_E2H
+	cbz	\tmp, \fail
+.endm
+
 /*
  * Allow Non-secure EL1 and EL0 to access physical timer and counter.
  * This is not necessary for VHE, since the host kernel runs in EL2,
@@ -43,9 +50,7 @@
  */
 .macro __init_el2_timers
 	mov	x0, #3				// Enable EL1 physical timers
-	mrs	x1, hcr_el2
-	and	x1, x1, #HCR_E2H
-	cbz	x1, .LnVHE_\@
+	__check_hvhe .LnVHE_\@, x1
 	lsl	x0, x0, #10
 .LnVHE_\@:
 	msr	cnthctl_el2, x0
@@ -139,9 +144,7 @@
 
 /* Coprocessor traps */
 .macro __init_el2_cptr
-	mrs	x1, hcr_el2
-	and	x1, x1, #HCR_E2H
-	cbz	x1, .LnVHE_\@
+	__check_hvhe .LnVHE_\@, x1
 	mov	x0, #(CPACR_EL1_FPEN_EL1EN | CPACR_EL1_FPEN_EL0EN)
 	b	.Lset_cptr_\@
 .LnVHE_\@:
@@ -269,9 +272,7 @@
 
 .Linit_sve_\@:	/* SVE register access */
 	mrs	x0, cptr_el2			// Disable SVE traps
-	mrs	x1, hcr_el2
-	and	x1, x1, #HCR_E2H
-	cbz	x1, .Lcptr_nvhe_\@
+	__check_hvhe .Lcptr_nvhe_\@, x1
 
 	// VHE case
 	orr	x0, x0, #(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN)
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#4 lore raw

PATCH v3 2/7 KVM: arm64: Use the appropriate feature trap register for SVE at EL2 setup

In reply to Fuad Tabba

Use the architectural feature trap/control register that corresponds to the current KVM mode, i.e., CPTR_EL2 or CPACR_EL1, when setting up SVE feature traps.

---
 arch/arm64/include/asm/el2_setup.h | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
arch/arm64/include/asm/el2_setup.h +9 -6
index 16d3bafa715d..41c5b02f38c5 100644
@@ -146,11 +146,12 @@
 .macro __init_el2_cptr
 	__check_hvhe .LnVHE_\@, x1
 	mov	x0, #(CPACR_EL1_FPEN_EL1EN | CPACR_EL1_FPEN_EL0EN)
-	b	.Lset_cptr_\@
+	msr	cpacr_el1, x0
+	b	.Lskip_set_cptr_\@
 .LnVHE_\@:
 	mov	x0, #0x33ff
-.Lset_cptr_\@:
 	msr	cptr_el2, x0			// Disable copro. traps to EL2
+.Lskip_set_cptr_\@:
 .endm
 
 /* Disable any fine grained traps */
@@ -271,17 +272,19 @@
 	check_override id_aa64pfr0, ID_AA64PFR0_EL1_SVE_SHIFT, .Linit_sve_\@, .Lskip_sve_\@, x1, x2
 
 .Linit_sve_\@:	/* SVE register access */
-	mrs	x0, cptr_el2			// Disable SVE traps
 	__check_hvhe .Lcptr_nvhe_\@, x1
 
-	// VHE case
+	// (h)VHE case
+	mrs	x0, cpacr_el1			// Disable SVE traps
 	orr	x0, x0, #(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN)
-	b	.Lset_cptr_\@
+	msr	cpacr_el1, x0
+	b	.Lskip_set_cptr_\@
 
 .Lcptr_nvhe_\@: // nVHE case
+	mrs	x0, cptr_el2			// Disable SVE traps
 	bic	x0, x0, #CPTR_EL2_TZ
-.Lset_cptr_\@:
 	msr	cptr_el2, x0
+.Lskip_set_cptr_\@:
 	isb
 	mov	x1, #ZCR_ELx_LEN_MASK		// SVE: Enable full vector
 	msr_s	SYS_ZCR_EL2, x1			// length for EL1.
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#5 lore raw

PATCH v3 3/7 KVM: arm64: Disable SME traps for (h)VHE at setup

In reply to Fuad Tabba

Ensure that SME traps are disabled for (h)VHE when setting up EL2, as they are for nVHE.

---
 arch/arm64/include/asm/el2_setup.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/arm64/include/asm/el2_setup.h +10 -0
index 41c5b02f38c5..b7afaa026842 100644
@@ -293,9 +293,19 @@
 	check_override id_aa64pfr1, ID_AA64PFR1_EL1_SME_SHIFT, .Linit_sme_\@, .Lskip_sme_\@, x1, x2
 
 .Linit_sme_\@:	/* SME register access and priority mapping */
+	__check_hvhe .Lcptr_nvhe_sme_\@, x1
+
+	// (h)VHE case
+	mrs	x0, cpacr_el1			// Disable SME traps
+	orr	x0, x0, #(CPACR_EL1_SMEN_EL0EN | CPACR_EL1_SMEN_EL1EN)
+	msr	cpacr_el1, x0
+	b	.Lskip_set_cptr_sme_\@
+
+.Lcptr_nvhe_sme_\@: // nVHE case
 	mrs	x0, cptr_el2			// Disable SME traps
 	bic	x0, x0, #CPTR_EL2_TSM
 	msr	cptr_el2, x0
+.Lskip_set_cptr_sme_\@:
 	isb
 
 	mrs	x1, sctlr_el2
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#6 lore raw

PATCH v3 4/7 KVM: arm64: Helper to write to appropriate feature trap register based on mode

In reply to Fuad Tabba

Factor out the code that decides whether to write to the feature trap registers, CPTR_EL2 or CPACR_EL1, based on the KVM mode, i.e., (h)VHE or nVHE.

This function will be used in the subsequent patch.

No functional change intended.

---
 arch/arm64/include/asm/kvm_emulate.h | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +9 -4
index efc0b45d79c3..f5941f6dce49 100644
@@ -571,6 +571,14 @@ static inline bool vcpu_has_feature(struct kvm_vcpu *vcpu, int feature)
 	return test_bit(feature, vcpu->arch.features);
 }
 
+static __always_inline void kvm_write_cptr_el2(u64 val)
+{
+	if (has_vhe() || has_hvhe())
+		write_sysreg(val, cpacr_el1);
+	else
+		write_sysreg(val, cptr_el2);
+}
+
 static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu)
 {
 	u64 val;
@@ -597,9 +605,6 @@ static __always_inline void kvm_reset_cptr_el2(struct kvm_vcpu *vcpu)
 {
 	u64 val = kvm_get_reset_cptr_el2(vcpu);
 
-	if (has_vhe() || has_hvhe())
-		write_sysreg(val, cpacr_el1);
-	else
-		write_sysreg(val, cptr_el2);
+	kvm_write_cptr_el2(val);
 }
 #endif /* __ARM64_KVM_EMULATE_H__ */
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#7 lore raw

PATCH v3 5/7 KVM: arm64: Use the appropriate feature trap register when activating traps

In reply to Fuad Tabba

Instead of writing directly to cptr_el2, use the helper that selects which feature trap register to write to based on the KVM mode.

Fixes 75c76ab5a641 ("KVM: arm64: Rework CPTR_EL2 programming for HVHE configuration")
---
 arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/hyp/nvhe/switch.c +1 -1
index 0a6271052def..e89a23153e85 100644
@@ -63,7 +63,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
 		__activate_traps_fpsimd32(vcpu);
 	}
 
-	write_sysreg(val, cptr_el2);
+	kvm_write_cptr_el2(val);
 	write_sysreg(__this_cpu_read(kvm_hyp_vector), vbar_el2);
 
 	if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#8 lore raw

PATCH v3 6/7 KVM: arm64: Fix resetting SVE trap values on reset for hVHE

In reply to Fuad Tabba

Ensure that SVE traps are disabled for hVHE, if the FPSIMD state isn't owned by the guest, when getting the reset value for the architectural feature control register.

Fixes 75c76ab5a641 ("KVM: arm64: Rework CPTR_EL2 programming for HVHE configuration")
---
 arch/arm64/include/asm/kvm_emulate.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +4 -0
index f5941f6dce49..adfb7d0ac55b 100644
@@ -588,6 +588,10 @@ static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu)
 		       CPACR_EL1_ZEN_EL1EN);
 	} else if (has_hvhe()) {
 		val = (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN);
+
+		if (!vcpu_has_sve(vcpu) ||
+		    (vcpu->arch.fp_state != FP_STATE_GUEST_OWNED))
+			val |= CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN;
 	} else {
 		val = CPTR_NVHE_EL2_RES1;
 
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
author
#9 lore raw

PATCH v3 7/7 KVM: arm64: Fix resetting SME trap values on reset for (h)VHE

In reply to Fuad Tabba

Ensure that SME traps are disabled for (h)VHE when getting the reset value for the architectural feature control register.

Fixes 75c76ab5a641 ("KVM: arm64: Rework CPTR_EL2 programming for HVHE configuration")
---
 arch/arm64/include/asm/kvm_emulate.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +4 -0
index adfb7d0ac55b..3d6725ff0bf6 100644
@@ -586,12 +586,16 @@ static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu)
 	if (has_vhe()) {
 		val = (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN |
 		       CPACR_EL1_ZEN_EL1EN);
+		if (cpus_have_final_cap(ARM64_SME))
+			val |= CPACR_EL1_SMEN_EL1EN;
 	} else if (has_hvhe()) {
 		val = (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN);
 
 		if (!vcpu_has_sve(vcpu) ||
 		    (vcpu->arch.fp_state != FP_STATE_GUEST_OWNED))
 			val |= CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN;
+		if (cpus_have_final_cap(ARM64_SME))
+			val |= CPACR_EL1_SMEN_EL1EN | CPACR_EL1_SMEN_EL0EN;
 	} else {
 		val = CPTR_NVHE_EL2_RES1;
 
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
maintainer Reviewed-by on the v3 series
#10 lore raw
In reply to Fuad Tabba
Quoted 24 lines · Fuad Tabba

Hi,

Changes from V2:
- Rebased onto Linux 6.5-rc3
- s/__check_e2h/__check_hvhe/g (Oliver)
- Fixed bug in calculating CPACR_EL1 reset value for hVHE in trapping SVE

Changes from V1:
- Expanded the cover letter to clarify the reasoning behind being consistent in writing to the architectural trap register based on the KVM mode (Marc)
- Factored out the code for checking E2H into a macro (Oliver)
- Factored out the code that selects which register to write to into a function (Oliver)

The (re)setting and disabling of SVE/SME trap handling (mostly) done for the hVHE work [*] misses a couple of cases.

This patch series ensures that these traps are disabled on setup and reset. Moreover, it makes the code consistent in using CPACR_EL1 or CPTR_EL2, depending on the mode.

Quoted 7 lines

CPACR_EL1 aliases to CPTR_EL2 when HCR_EL2.E2H == 1, but by being consistent we don't need to issue a synchronisation when alternating between one or the other accessor. Moreover, when running hVHE under NV, we don't trap unnecessarily on accessing CPTR_EL2, while CPACR_EL1 can be used directly without any trap.

Based on Linux 6.5-rc3.

	M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Applied
#11 lore raw

Re: (subset) [PATCH v3 0/7] Fix setting SVE and SME traps in (h)VHE

In reply to Fuad Tabba
Quoted 14 lines · Fuad Tabba

Changes from V2:
- Rebased onto Linux 6.5-rc3
- s/__check_e2h/__check_hvhe/g (Oliver)
- Fixed bug in calculating CPACR_EL1 reset value for hVHE in trapping SVE

Changes from V1:
- Expanded the cover letter to clarify the reasoning behind being consistent in writing to the architectural trap register based on the KVM mode (Marc)
- Factored out the code for checking E2H into a macro (Oliver)
- Factored out the code that selects which register to write to into a function (Oliver)

Quoted 1 line

[...]

Applied to kvmarm/fixes, thanks!

[1/7] KVM: arm64: Factor out code for checking (h)VHE mode into a macro

      https://git.kernel.org/kvmarm/kvmarm/c/ce92232614a5
[2/7] KVM: arm64: Use the appropriate feature trap register for SVE at EL2 setup
      https://git.kernel.org/kvmarm/kvmarm/c/45a3681a10ff
[3/7] KVM: arm64: Disable SME traps for (h)VHE at setup
      https://git.kernel.org/kvmarm/kvmarm/c/380624d4358b
[4/7] KVM: arm64: Helper to write to appropriate feature trap register based on mode
      https://git.kernel.org/kvmarm/kvmarm/c/90ae31c65d5a
[5/7] KVM: arm64: Use the appropriate feature trap register when activating traps
      https://git.kernel.org/kvmarm/kvmarm/c/a9626099a51f
[6/7] KVM: arm64: Fix resetting SVE trap values on reset for hVHE
      https://git.kernel.org/kvmarm/kvmarm/c/7af0d5e50006
[7/7] KVM: arm64: Fix resetting SME trap values on reset for (h)VHE
      https://git.kernel.org/kvmarm/kvmarm/c/375110ab51de

--
Best,
Oliver