Fix setting SVE and SME traps in (h)VHE
v3 cover Fix setting SVE and SME traps in (h)VHE
Hi,
Changes from V2:
- Rebased onto Linux 6.5-rc3
- s/__check_e2h/__check_hvhe/g (Oliver)
- Fixed bug in calculating CPACR_EL1 reset value for hVHE in trapping SVE
Changes from V1:
- Expanded the cover letter to clarify the reasoning behind being consistent in writing to the architectural trap register based on the KVM mode (Marc)
- Factored out the code for checking E2H into a macro (Oliver)
- Factored out the code that selects which register to write to into a function (Oliver)
The (re)setting and disabling of SVE/SME trap handling (mostly) done for the hVHE work [*] misses a couple of cases.
This patch series ensures that these traps are disabled on setup and reset. Moreover, it makes the code consistent in using CPACR_EL1 or CPTR_EL2, depending on the mode.
CPACR_EL1 aliases to CPTR_EL2 when HCR_EL2.E2H == 1, but by being consistent we don't need to issue a synchronisation when alternating between one or the other accessor. Moreover, when running hVHE under NV, we don't trap unnecessarily on accessing CPTR_EL2, while CPACR_EL1 can be used directly without any trap.
Based on Linux 6.5-rc3.
Cheers,
/fuad
Fuad Tabba (7):
KVM: arm64: Factor out code for checking (h)VHE mode into a macro
KVM: arm64: Use the appropriate feature trap register for SVE at EL2
setup
KVM: arm64: Disable SME traps for (h)VHE at setup
KVM: arm64: Helper to write to appropriate feature trap register based
on mode
KVM: arm64: Use the appropriate feature trap register when activating
traps
KVM: arm64: Fix resetting SVE trap values on reset for hVHE
KVM: arm64: Fix resetting SME trap values on reset for (h)VHE
arch/arm64/include/asm/el2_setup.h | 44 ++++++++++++++++++---------- arch/arm64/include/asm/kvm_emulate.h | 21 ++++++++++--- arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- 3 files changed, 47 insertions(+), 20 deletions(-)
base-commit: 6eaae198076080886b9e7d57f4ae06fa782f90ef
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
PATCH v3 1/7 KVM: arm64: Factor out code for checking E2H into a macro
The code for checking whether the kernel is in (h)VHE mode is repeated, and will be needed again in future patches. Factor it out in a macro.
No functional change intended.
No change in emitted assembly code intended.
--- arch/arm64/include/asm/el2_setup.h | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)
arch/arm64/include/asm/el2_setup.h +10 -9
@@ -31,6 +31,13 @@ .Lskip_hcrx_\@: .endm +/* Check if running in host at EL2 mode, i.e., (h)VHE. Jump to fail if not. */ +.macro __check_hvhe fail, tmp + mrs \tmp, hcr_el2 + and \tmp, \tmp, #HCR_E2H + cbz \tmp, \fail +.endm +/* * Allow Non-secure EL1 and EL0 to access physical timer and counter. * This is not necessary for VHE, since the host kernel runs in EL2,
@@ -43,9 +50,7 @@*/ .macro __init_el2_timers mov x0, #3 // Enable EL1 physical timers -mrs x1, hcr_el2 - and x1, x1, #HCR_E2H - cbz x1, .LnVHE_\@ + __check_hvhe .LnVHE_\@, x1 lsl x0, x0, #10 .LnVHE_\@: msr cnthctl_el2, x0
@@ -139,9 +144,7 @@/* Coprocessor traps */ .macro __init_el2_cptr -mrs x1, hcr_el2 - and x1, x1, #HCR_E2H - cbz x1, .LnVHE_\@ + __check_hvhe .LnVHE_\@, x1 mov x0, #(CPACR_EL1_FPEN_EL1EN | CPACR_EL1_FPEN_EL0EN) b .Lset_cptr_\@ .LnVHE_\@:
@@ -269,9 +272,7 @@ .Linit_sve_\@: /* SVE register access */ mrs x0, cptr_el2 // Disable SVE traps - mrs x1, hcr_el2 - and x1, x1, #HCR_E2H - cbz x1, .Lcptr_nvhe_\@ + __check_hvhe .Lcptr_nvhe_\@, x1// VHE case orr x0, x0, #(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN)
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
Re: [PATCH v3 1/7] KVM: arm64: Factor out code for checking E2H into a macro
Apologies for this one. I changed the subject of this patch without deleting it from my patches folder before sending, hence the duplication.
/fuad
Quoted 67 lines · Fuad Tabba
The code for checking whether the kernel is in (h)VHE mode is repeated, and will be needed again in future patches. Factor it out in a macro.
No functional change intended.
No change in emitted assembly code intended.Signed-off-by Fuad Tabba <[email protected]>--- arch/arm64/include/asm/el2_setup.h | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)arch/arm64/include/asm/el2_setup.h +10
-10@@ -31,6 +31,13 @@ .Lskip_hcrx_\@: .endm +/* Check if running in host at EL2 mode, i.e., (h)VHE. Jump to fail if not. */ +.macro __check_hvhe fail, tmp + mrs \tmp, hcr_el2 + and \tmp, \tmp, #HCR_E2H + cbz \tmp, \fail +.endm +/* * Allow Non-secure EL1 and EL0 to access physical timer and counter. * This is not necessary for VHE, since the host kernel runs in EL2, @@ -43,9 +50,7 @@*/ .macro __init_el2_timers mov x0, #3 // Enable EL1 physical timers -mrs x1, hcr_el2 - and x1, x1, #HCR_E2H - cbz x1, .LnVHE_\@ + __check_hvhe .LnVHE_\@, x1 lsl x0, x0, #10 .LnVHE_\@: msr cnthctl_el2, x0@@ -139,9 +144,7 @@/* Coprocessor traps */ .macro __init_el2_cptr -mrs x1, hcr_el2 - and x1, x1, #HCR_E2H - cbz x1, .LnVHE_\@ + __check_hvhe .LnVHE_\@, x1 mov x0, #(CPACR_EL1_FPEN_EL1EN | CPACR_EL1_FPEN_EL0EN) b .Lset_cptr_\@ .LnVHE_\@:@@ -269,9 +272,7 @@ .Linit_sve_\@: /* SVE register access */ mrs x0, cptr_el2 // Disable SVE traps - mrs x1, hcr_el2 - and x1, x1, #HCR_E2H - cbz x1, .Lcptr_nvhe_\@ + __check_hvhe .Lcptr_nvhe_\@, x1// VHE case orr x0, x0, #(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN) --2.41.0.487.g6d72f3e995-goog
PATCH v3 1/7 KVM: arm64: Factor out code for checking (h)VHE mode into a macro
In reply to Fuad TabbaThe code for checking whether the kernel is in (h)VHE mode is repeated, and will be needed again in future patches. Factor it out in a macro.
No functional change intended.
No change in emitted assembly code intended.
--- arch/arm64/include/asm/el2_setup.h | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)
arch/arm64/include/asm/el2_setup.h +10 -9
@@ -31,6 +31,13 @@ .Lskip_hcrx_\@: .endm +/* Check if running in host at EL2 mode, i.e., (h)VHE. Jump to fail if not. */ +.macro __check_hvhe fail, tmp + mrs \tmp, hcr_el2 + and \tmp, \tmp, #HCR_E2H + cbz \tmp, \fail +.endm +/* * Allow Non-secure EL1 and EL0 to access physical timer and counter. * This is not necessary for VHE, since the host kernel runs in EL2,
@@ -43,9 +50,7 @@*/ .macro __init_el2_timers mov x0, #3 // Enable EL1 physical timers -mrs x1, hcr_el2 - and x1, x1, #HCR_E2H - cbz x1, .LnVHE_\@ + __check_hvhe .LnVHE_\@, x1 lsl x0, x0, #10 .LnVHE_\@: msr cnthctl_el2, x0
@@ -139,9 +144,7 @@/* Coprocessor traps */ .macro __init_el2_cptr -mrs x1, hcr_el2 - and x1, x1, #HCR_E2H - cbz x1, .LnVHE_\@ + __check_hvhe .LnVHE_\@, x1 mov x0, #(CPACR_EL1_FPEN_EL1EN | CPACR_EL1_FPEN_EL0EN) b .Lset_cptr_\@ .LnVHE_\@:
@@ -269,9 +272,7 @@ .Linit_sve_\@: /* SVE register access */ mrs x0, cptr_el2 // Disable SVE traps - mrs x1, hcr_el2 - and x1, x1, #HCR_E2H - cbz x1, .Lcptr_nvhe_\@ + __check_hvhe .Lcptr_nvhe_\@, x1// VHE case orr x0, x0, #(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN)
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
PATCH v3 2/7 KVM: arm64: Use the appropriate feature trap register for SVE at EL2 setup
In reply to Fuad TabbaUse the architectural feature trap/control register that corresponds to the current KVM mode, i.e., CPTR_EL2 or CPACR_EL1, when setting up SVE feature traps.
--- arch/arm64/include/asm/el2_setup.h | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-)
arch/arm64/include/asm/el2_setup.h +9 -6
@@ -146,11 +146,12 @@ .macro __init_el2_cptr __check_hvhe .LnVHE_\@, x1 mov x0, #(CPACR_EL1_FPEN_EL1EN | CPACR_EL1_FPEN_EL0EN) - b .Lset_cptr_\@ + msr cpacr_el1, x0 + b .Lskip_set_cptr_\@ .LnVHE_\@: mov x0, #0x33ff -.Lset_cptr_\@: msr cptr_el2, x0 // Disable copro. traps to EL2 +.Lskip_set_cptr_\@: .endm/* Disable any fine grained traps */
@@ -271,17 +272,19 @@ check_override id_aa64pfr0, ID_AA64PFR0_EL1_SVE_SHIFT, .Linit_sve_\@, .Lskip_sve_\@, x1, x2 .Linit_sve_\@: /* SVE register access */ - mrs x0, cptr_el2 // Disable SVE traps __check_hvhe .Lcptr_nvhe_\@, x1 -// VHE case +// (h)VHE case + mrs x0, cpacr_el1 // Disable SVE traps orr x0, x0, #(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN) - b .Lset_cptr_\@ + msr cpacr_el1, x0 + b .Lskip_set_cptr_\@ .Lcptr_nvhe_\@: // nVHE case + mrs x0, cptr_el2 // Disable SVE traps bic x0, x0, #CPTR_EL2_TZ -.Lset_cptr_\@: msr cptr_el2, x0 +.Lskip_set_cptr_\@: isb mov x1, #ZCR_ELx_LEN_MASK // SVE: Enable full vector msr_s SYS_ZCR_EL2, x1 // length for EL1.
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
PATCH v3 3/7 KVM: arm64: Disable SME traps for (h)VHE at setup
In reply to Fuad TabbaEnsure that SME traps are disabled for (h)VHE when setting up EL2, as they are for nVHE.
--- arch/arm64/include/asm/el2_setup.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/arm64/include/asm/el2_setup.h +10 -0
@@ -293,9 +293,19 @@ check_override id_aa64pfr1, ID_AA64PFR1_EL1_SME_SHIFT, .Linit_sme_\@, .Lskip_sme_\@, x1, x2 .Linit_sme_\@: /* SME register access and priority mapping */ + __check_hvhe .Lcptr_nvhe_sme_\@, x1 + +// (h)VHE case + mrs x0, cpacr_el1 // Disable SME traps + orr x0, x0, #(CPACR_EL1_SMEN_EL0EN | CPACR_EL1_SMEN_EL1EN) + msr cpacr_el1, x0 + b .Lskip_set_cptr_sme_\@ + +.Lcptr_nvhe_sme_\@: // nVHE case mrs x0, cptr_el2 // Disable SME traps bic x0, x0, #CPTR_EL2_TSM msr cptr_el2, x0 +.Lskip_set_cptr_sme_\@: isb mrs x1, sctlr_el2
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
PATCH v3 4/7 KVM: arm64: Helper to write to appropriate feature trap register based on mode
In reply to Fuad TabbaFactor out the code that decides whether to write to the feature trap registers, CPTR_EL2 or CPACR_EL1, based on the KVM mode, i.e., (h)VHE or nVHE.
This function will be used in the subsequent patch.
No functional change intended.
--- arch/arm64/include/asm/kvm_emulate.h | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +9 -4
@@ -571,6 +571,14 @@ static inline bool vcpu_has_feature(struct kvm_vcpu *vcpu, int feature)return test_bit (feature, vcpu->arch.features); } +static __always_inline void kvm_write_cptr_el2(u64 val) +{ +if (has_vhe () ||has_hvhe ()) +write_sysreg (val, cpacr_el1); +else +write_sysreg (val, cptr_el2); +} + static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu) {u64 val;
@@ -597,9 +605,6 @@ static __always_inline void kvm_reset_cptr_el2(struct kvm_vcpu *vcpu) {u64 val =kvm_get_reset_cptr_el2 (vcpu); -if (has_vhe () ||has_hvhe ()) -write_sysreg (val, cpacr_el1); -else -write_sysreg (val, cptr_el2); +kvm_write_cptr_el2 (val); }#endif /* __ARM64_KVM_EMULATE_H__ */
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
PATCH v3 5/7 KVM: arm64: Use the appropriate feature trap register when activating traps
In reply to Fuad TabbaInstead of writing directly to cptr_el2, use the helper that selects which feature trap register to write to based on the KVM mode.
--- arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/hyp/nvhe/switch.c +1 -1
@@ -63,7 +63,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu)__activate_traps_fpsimd32 (vcpu); } -write_sysreg (val, cptr_el2); +kvm_write_cptr_el2 (val);write_sysreg (__this_cpu_read (kvm_hyp_vector), vbar_el2);if (cpus_have_final_cap (ARM64_WORKAROUND_SPECULATIVE_AT )) {
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
PATCH v3 6/7 KVM: arm64: Fix resetting SVE trap values on reset for hVHE
In reply to Fuad TabbaEnsure that SVE traps are disabled for hVHE, if the FPSIMD state isn't owned by the guest, when getting the reset value for the architectural feature control register.
--- arch/arm64/include/asm/kvm_emulate.h | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +4 -0
@@ -588,6 +588,10 @@ static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu) CPACR_EL1_ZEN_EL1EN); }else if (has_hvhe ()) { val = (CPACR_EL1_FPEN_EL0EN |CPACR_EL1_FPEN_EL1EN ); + +if (!vcpu_has_sve (vcpu) || + (vcpu->arch.fp_state !=FP_STATE_GUEST_OWNED )) + val |=CPACR_EL1_ZEN_EL1EN |CPACR_EL1_ZEN_EL0EN ; }else { val =CPTR_NVHE_EL2_RES1 ;
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
PATCH v3 7/7 KVM: arm64: Fix resetting SME trap values on reset for (h)VHE
In reply to Fuad TabbaEnsure that SME traps are disabled for (h)VHE when getting the reset value for the architectural feature control register.
--- arch/arm64/include/asm/kvm_emulate.h | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +4 -0
@@ -586,12 +586,16 @@ static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu)if (has_vhe ()) { val = (CPACR_EL1_FPEN_EL0EN |CPACR_EL1_FPEN_EL1EN |CPACR_EL1_ZEN_EL1EN ); +if (cpus_have_final_cap (ARM64_SME )) + val |=CPACR_EL1_SMEN_EL1EN ; }else if (has_hvhe ()) { val = (CPACR_EL1_FPEN_EL0EN |CPACR_EL1_FPEN_EL1EN );if (!vcpu_has_sve (vcpu) || (vcpu->arch.fp_state !=FP_STATE_GUEST_OWNED )) val |=CPACR_EL1_ZEN_EL1EN |CPACR_EL1_ZEN_EL0EN ; +if (cpus_have_final_cap (ARM64_SME )) + val |=CPACR_EL1_SMEN_EL1EN |CPACR_EL1_SMEN_EL0EN ; }else { val =CPTR_NVHE_EL2_RES1 ;
Signature · 1 line
2.41.0.487.g6d72f3e995-goog
Quoted 24 lines · Fuad Tabba
Hi,
Changes from V2:
- Rebased onto Linux 6.5-rc3
- s/__check_e2h/__check_hvhe/g (Oliver)
- Fixed bug in calculatingCPACR_EL1reset value for hVHE in trapping SVEChanges from V1:
- Expanded the cover letter to clarify the reasoning behind being consistent in writing to the architectural trap register based on the KVM mode (Marc)
- Factored out the code for checking E2H into a macro (Oliver)
- Factored out the code that selects which register to write to into a function (Oliver)The (re)setting and disabling of SVE/SME trap handling (mostly) done for the hVHE work [*] misses a couple of cases.
This patch series ensures that these traps are disabled on setup and reset. Moreover, it makes the code consistent in using
CPACR_EL1orCPTR_EL2, depending on the mode.
Quoted 7 lines
CPACR_EL1aliases toCPTR_EL2when HCR_EL2.E2H == 1, but by being consistent we don't need to issue a synchronisation when alternating between one or the other accessor. Moreover, when running hVHE under NV, we don't trap unnecessarily on accessingCPTR_EL2, whileCPACR_EL1can be used directly without any trap.Based on Linux 6.5-rc3.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Oliver Upton
AppliedRe: (subset) [PATCH v3 0/7] Fix setting SVE and SME traps in (h)VHE
In reply to Fuad TabbaQuoted 14 lines · Fuad Tabba
Changes from V2:
- Rebased onto Linux 6.5-rc3
- s/__check_e2h/__check_hvhe/g (Oliver)
- Fixed bug in calculatingCPACR_EL1reset value for hVHE in trapping SVEChanges from V1:
- Expanded the cover letter to clarify the reasoning behind being consistent in writing to the architectural trap register based on the KVM mode (Marc)
- Factored out the code for checking E2H into a macro (Oliver)
- Factored out the code that selects which register to write to into a function (Oliver)
Quoted 1 line
[...]
Applied to kvmarm/fixes, thanks!
[1/7] KVM: arm64: Factor out code for checking (h)VHE mode into a macro
https://git.kernel.org/kvmarm/kvmarm/c/ce92232614a5 [2/7] KVM: arm64: Use the appropriate feature trap register for SVE at EL2 setup https://git.kernel.org/kvmarm/kvmarm/c/45a3681a10ff [3/7] KVM: arm64: Disable SME traps for (h)VHE at setup https://git.kernel.org/kvmarm/kvmarm/c/380624d4358b [4/7] KVM: arm64: Helper to write to appropriate feature trap register based on mode https://git.kernel.org/kvmarm/kvmarm/c/90ae31c65d5a [5/7] KVM: arm64: Use the appropriate feature trap register when activating traps https://git.kernel.org/kvmarm/kvmarm/c/a9626099a51f [6/7] KVM: arm64: Fix resetting SVE trap values on reset for hVHE https://git.kernel.org/kvmarm/kvmarm/c/7af0d5e50006 [7/7] KVM: arm64: Fix resetting SME trap values on reset for (h)VHE https://git.kernel.org/kvmarm/kvmarm/c/375110ab51de
--
Best,
Oliver