KVM: arm64: nv: FPSIMD/SVE, plus some other CPTR goodies
v2 cover KVM: arm64: nv: FPSIMD/SVE, plus some other CPTR goodies
As discussed, here's the combined series of Marc + I's patches that implement all the trap forwarding / merging logic required to observe the L1 CPTR configuration when running an L2 guest.
Like before, this was tested on Neoverse-V2 with L0, L1, and L2 running fpsimd-test and sve-test on top of one another.
v1 [1] -> v2:
- Grab Marc's CPTR trap patches [2]
- Avoid taking two traps when L1 accesses ZCR_EL2 while the host owns FP regs. The fast path will now load the FP regs and forward the sysreg trap to the slow path for complete handling.
- Add a comment describing the above behavior (Marc)
- Avoid the use of guest_hyp_*_traps_enabled() in __activate_cptr_traps() for better codegen (Marc)
- Document the reason for only testing bit[0] of CPTR_EL2.xEN when folding L1 traps into the hardware CPTR value.
- Add a helper for synthesizing SVE traps, rather than open-coding the ESR value in access_zcr_el2()
[1]: https://lore.kernel.org/kvmarm/[email protected]/
[2]: https://lore.kernel.org/kvmarm/[email protected]/
Jintack Lim (1):
KVM: arm64: nv: Forward FP/ASIMD traps to guest hypervisor
Marc Zyngier (4):
KVM: arm64: nv: Handle CPACR_EL1 traps
KVM: arm64: nv: Add TCPAC/TTA to CPTR->CPACR conversion helper
KVM: arm64: nv: Add trap description for CPTR_EL2
KVM: arm64: nv: Add additional trap setup for CPTR_EL2
Oliver Upton (10):
KVM: arm64: nv: Forward SVE traps to guest hypervisor
KVM: arm64: nv: Load guest FP state for ZCR_EL2 trap
KVM: arm64: nv: Load guest hyp's ZCR into EL1 state
KVM: arm64: nv: Handle ZCR_EL2 traps
KVM: arm64: nv: Save guest's ZCR_EL2 when in hyp context
KVM: arm64: nv: Use guest hypervisor's max VL when running nested
guest
KVM: arm64: nv: Ensure correct VL is loaded before saving SVE state
KVM: arm64: Spin off helper for programming CPTR traps
KVM: arm64: nv: Honor guest hypervisor's FP/SVE traps in CPTR_EL2
KVM: arm64: Allow the use of SVE+NV
arch/arm64/include/asm/kvm_emulate.h | 55 +++++++++ arch/arm64/include/asm/kvm_host.h | 7 ++ arch/arm64/include/asm/kvm_nested.h | 5 +- arch/arm64/kvm/arm.c | 5 - arch/arm64/kvm/emulate-nested.c | 91 ++++++++++++++ arch/arm64/kvm/fpsimd.c | 22 +++- arch/arm64/kvm/handle_exit.c | 19 ++- arch/arm64/kvm/hyp/include/hyp/switch.h | 24 +++- arch/arm64/kvm/hyp/vhe/switch.c | 156 ++++++++++++++++++++---- arch/arm64/kvm/nested.c | 3 +- arch/arm64/kvm/sys_regs.c | 38 ++++++ 11 files changed, 380 insertions(+), 45 deletions(-)
base-commit: c3f38fa61af77b49866b006939479069cd451173
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 1/15 KVM: arm64: nv: Forward FP/ASIMD traps to guest hypervisor
From: Jintack Lim <[email protected]>
Give precedence to the guest hypervisor's trap configuration when routing an FP/ASIMD trap taken to EL2. Take advantage of the infrastructure for translating CPTR_EL2 into the VHE (i.e. EL1) format and base the trap decision solely on the VHE view of the register. The in-memory value of CPTR_EL2 will always be up to date for the guest hypervisor (more on that later), so just read it directly from memory.
Bury all of this behind a macro keyed off of the CPTR bitfield in anticipation of supporting other traps (e.g. SVE).
[maz: account for HCR_EL2.E2H when testing for TFP/FPEN, with
all the hard work actually being done by Chase Conklin]
[ oliver: translate nVHE->VHE format for testing traps; macro for reuse
in other CPTR_EL2.xEN fields ]
--- arch/arm64/include/asm/kvm_emulate.h | 43 +++++++++++++++++++++++++ arch/arm64/include/asm/kvm_nested.h | 1 - arch/arm64/kvm/handle_exit.c | 16 ++++++--- arch/arm64/kvm/hyp/include/hyp/switch.h | 3 ++ 4 files changed, 58 insertions(+), 5 deletions(-)
arch/arm64/include/asm/kvm_emulate.h +43 -0
@@ -11,6 +11,7 @@#ifndef __ARM64_KVM_EMULATE_H__#define __ARM64_KVM_EMULATE_H__ +#include <linux/bitfield.h> #include <linux/kvm_host.h> #include <asm/debug-monitors.h>
@@ -599,4 +600,46 @@ static __always_inline void kvm_reset_cptr_el2(struct kvm_vcpu *vcpu) kvm_write_cptr_el2(val); } + +/* +* Returns a 'sanitised' view of CPTR_EL2, translating from nVHE to the VHE +* format if E2H isn't set. +*/ +static inline u64 vcpu_sanitised_cptr_el2 (const struct kvm_vcpu *vcpu) +{ +u64 cptr =__vcpu_sys_reg (vcpu,CPTR_EL2 ); + +if (!vcpu_el2_e2h_is_set (vcpu)) + cptr =translate_cptr_el2_to_cpacr_el1 (cptr); + +return cptr; +} + +static inline bool ____cptr_xen_trap_enabled (const struct kvm_vcpu *vcpu, +unsigned int xen) +{ +switch (xen) { +case 0b00 : +case 0b10 : +return true; +case 0b01 : +return vcpu_el2_tge_is_set (vcpu) && !vcpu_is_el2 (vcpu); +case 0b11 : +default : +return false; + } +} + +#define __guest_hyp_cptr_xen_trap_enabled (vcpu, xen) \ + (!vcpu_has_nv(vcpu) ? false : \ + ____cptr_xen_trap_enabled(vcpu, \ + SYS_FIELD_GET(CPACR_ELx, xen, \ + vcpu_sanitised_cptr_el2(vcpu)))) + +static inline bool guest_hyp_fpsimd_traps_enabled (const struct kvm_vcpu *vcpu) +{ +return __guest_hyp_cptr_xen_trap_enabled (vcpu,FPEN ); +} + +#endif /* __ARM64_KVM_EMULATE_H__ */
arch/arm64/include/asm/kvm_nested.h +0 -1
@@ -75,5 +75,4 @@ static inline bool kvm_auth_eretax(struct kvm_vcpu *vcpu, u64 *elr)return false; }#endif -#endif /* __ARM64_KVM_NESTED_H */
arch/arm64/kvm/handle_exit.c +12 -4
@@ -94,11 +94,19 @@ static int handle_smc(struct kvm_vcpu *vcpu) }/* -* Guest access to FP/ASIMD registers are routed to this handler only -* when the system doesn't support FP/ASIMD. +* This handles the cases where the system does not support FP/ASIMD or when +* we are running nested virtualization and the guest hypervisor is trapping +* FP/ASIMD accesses by its guest guest. +* +* All other handling of guest vs. host FP/ASIMD register state is handled in +* fixup_guest_exit(). */ -static int handle_no_fpsimd (struct kvm_vcpu *vcpu) +static int kvm_handle_fpasimd (struct kvm_vcpu *vcpu) { +if (guest_hyp_fpsimd_traps_enabled (vcpu)) +return kvm_inject_nested_sync (vcpu,kvm_vcpu_get_esr (vcpu)); + +/* This is the case when the system doesn't support FP/ASIMD. */ kvm_inject_undefined (vcpu);return 1 ; }
@@ -304,7 +312,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_BREAKPT_LOW]= kvm_handle_guest_debug, [ESR_ELx_EC_BKPT32] = kvm_handle_guest_debug, [ESR_ELx_EC_BRK64] = kvm_handle_guest_debug, - [ESR_ELx_EC_FP_ASIMD] = handle_no_fpsimd, + [ESR_ELx_EC_FP_ASIMD] = kvm_handle_fpasimd, [ESR_ELx_EC_PAC] = kvm_handle_ptrauth, };
arch/arm64/kvm/hyp/include/hyp/switch.h +3 -0
@@ -341,6 +341,9 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)/* Only handle traps the vCPU can support here: */ switch (esr_ec) {case ESR_ELx_EC_FP_ASIMD: +/* Forward traps to the guest hypervisor as required */ +if (guest_hyp_fpsimd_traps_enabled (vcpu)) +return false;break ;case ESR_ELx_EC_SVE:if (!sve_guest)
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
Re: [PATCH v2 01/15] KVM: arm64: nv: Forward FP/ASIMD traps to guest hypervisor
Quoted 92 lines · Oliver Upton
From: Jintack Lim <[email protected]>
Give precedence to the guest hypervisor's trap configuration when routing an FP/ASIMD trap taken to EL2. Take advantage of the infrastructure for translating
CPTR_EL2into the VHE (i.e. EL1) format and base the trap decision solely on the VHE view of the register. The in-memory value ofCPTR_EL2will always be up to date for the guest hypervisor (more on that later), so just read it directly from memory.Bury all of this behind a macro keyed off of the CPTR bitfield in anticipation of supporting other traps (e.g. SVE).
Signed-off-by Jintack Lim <[email protected]>Signed-off-by Christoffer Dall <[email protected]>[maz: account for HCR_EL2.E2H when testing for TFP/FPEN, with
all the hard work actually being done by Chase Conklin]Signed-off-by Marc Zyngier <[email protected]>[ oliver: translate
nVHE->VHEformat for testing traps; macro for reuse
in other CPTR_EL2.xEN fields ]Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/include/asm/kvm_emulate.h | 43 +++++++++++++++++++++++++ arch/arm64/include/asm/kvm_nested.h | 1 - arch/arm64/kvm/handle_exit.c | 16 ++++++--- arch/arm64/kvm/hyp/include/hyp/switch.h | 3 ++ 4 files changed, 58 insertions(+), 5 deletions(-)arch/arm64/include/asm/kvm_emulate.h +43
-0@@ -11,6 +11,7 @@#ifndef __ARM64_KVM_EMULATE_H__#define __ARM64_KVM_EMULATE_H__ +#include <linux/bitfield.h> #include <linux/kvm_host.h> #include <asm/debug-monitors.h> @@ -599,4 +600,46 @@ static __always_inline void kvm_reset_cptr_el2(struct kvm_vcpu *vcpu) kvm_write_cptr_el2(val); } + +/* +* Returns a 'sanitised' view of CPTR_EL2, translating from nVHE to the VHE +* format if E2H isn't set. +*/ +static inline u64 vcpu_sanitised_cptr_el2 (const struct kvm_vcpu *vcpu) +{ +u64 cptr =__vcpu_sys_reg (vcpu,CPTR_EL2 ); + +if (!vcpu_el2_e2h_is_set (vcpu)) + cptr =translate_cptr_el2_to_cpacr_el1 (cptr); + +return cptr; +} + +static inline bool ____cptr_xen_trap_enabled (const struct kvm_vcpu *vcpu, +unsigned int xen) +{ +switch (xen) { +case 0b00 : +case 0b10 : +return true; +case 0b01 : +return vcpu_el2_tge_is_set (vcpu) && !vcpu_is_el2 (vcpu); +case 0b11 : +default : +return false; + } +} + +#define __guest_hyp_cptr_xen_trap_enabled (vcpu, xen) \ + (!vcpu_has_nv(vcpu) ? false : \ + ____cptr_xen_trap_enabled(vcpu, \ + SYS_FIELD_GET(CPACR_ELx, xen, \ + vcpu_sanitised_cptr_el2(vcpu)))) + +static inline bool guest_hyp_fpsimd_traps_enabled (const struct kvm_vcpu *vcpu) +{ +return __guest_hyp_cptr_xen_trap_enabled (vcpu,FPEN ); +} + +#endif /* __ARM64_KVM_EMULATE_H__ */ arch/arm64/include/asm/kvm_nested.h +0
-0
arch/arm64/include/asm/kvm_nested.h static inline bool kvm_auth_eretax(struct kvm_vcpu *vcpu, u64 *elr) · line 78
Quoted 5 lines
arch/arm64/include/asm/kvm_nested.h +0
-1@@ -75,5 +75,4 @@ static inline bool kvm_auth_eretax(struct kvm_vcpu *vcpu, u64 *elr)return false; }#endif -#endif /* __ARM64_KVM_NESTED_H */
nit: spurious change.
Aside from that:
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
PATCH v2 2/15 KVM: arm64: nv: Forward SVE traps to guest hypervisor
In reply to Oliver UptonSimilar to FPSIMD traps, don't load SVE state if the guest hypervisor has SVE traps enabled and forward the trap instead. Note that ZCR_EL2 will require some special handling, as it takes a sysreg trap to EL2 when HCR_EL2.NV = 1.
--- arch/arm64/include/asm/kvm_emulate.h | 4 ++++ arch/arm64/kvm/handle_exit.c | 3 +++ arch/arm64/kvm/hyp/include/hyp/switch.h | 2 ++ 3 files changed, 9 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +4 -0
@@ -641,5 +641,9 @@ static inline bool guest_hyp_fpsimd_traps_enabled(const struct kvm_vcpu *vcpu) return __guest_hyp_cptr_xen_trap_enabled(vcpu, FPEN); } +static inline bool guest_hyp_sve_traps_enabled (const struct kvm_vcpu *vcpu) +{ +return __guest_hyp_cptr_xen_trap_enabled (vcpu,ZEN ); +}#endif /* __ARM64_KVM_EMULATE_H__ */
arch/arm64/kvm/handle_exit.c +3 -0
@@ -217,6 +217,9 @@ static int kvm_handle_unknown_ec(struct kvm_vcpu *vcpu)*/ static int handle_sve (struct kvm_vcpu *vcpu) { +if (guest_hyp_sve_traps_enabled (vcpu)) +return kvm_inject_nested_sync (vcpu,kvm_vcpu_get_esr (vcpu)); +kvm_inject_undefined (vcpu);return 1 ; }
arch/arm64/kvm/hyp/include/hyp/switch.h +2 -0
@@ -348,6 +348,8 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)case ESR_ELx_EC_SVE:if (!sve_guest)return false; +if (guest_hyp_sve_traps_enabled (vcpu)) +return false;break ;default :return false;
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 3/15 KVM: arm64: nv: Handle CPACR_EL1 traps
In reply to Oliver UptonFrom: Marc Zyngier <[email protected]>
Handle CPACR_EL1 accesses when running a VHE guest. In order to limit the cost of the emulation, implement it ass a shallow exit.
In the other cases:
- this is a nVHE L1 which will write to memory, and we don't trap
- this is a L2 guest:
* the L1 has CPTR_EL2.TCPAC==0, and the L2 has direct register access
* the L1 has CPTR_EL2.TCPAC==1, and the L2 will trap, but the handling is defered to the general handling for forwarding
--- arch/arm64/kvm/hyp/vhe/switch.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/vhe/switch.c +31 -1
@@ -262,10 +262,40 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code)return true; } +static bool kvm_hyp_handle_cpacr_el1 (struct kvm_vcpu *vcpu,u64 *exit_code) +{ +u64 esr =kvm_vcpu_get_esr (vcpu); +int rt; + +if (!is_hyp_ctxt (vcpu) ||esr_sys64_to_sysreg (esr) !=SYS_CPACR_EL1 ) +return false; + + rt =kvm_vcpu_sys_get_rt (vcpu); + +if ((esr & ESR_ELx_SYS64_ISS_DIR_MASK) == ESR_ELx_SYS64_ISS_DIR_READ) { +vcpu_set_reg (vcpu, rt,__vcpu_sys_reg (vcpu,CPTR_EL2 )); + }else { +vcpu_write_sys_reg (vcpu,vcpu_get_reg (vcpu, rt),CPTR_EL2 ); +__activate_cptr_traps (vcpu); + } + +__kvm_skip_instr (vcpu); + +return true; +} + +static bool kvm_hyp_handle_sysreg_vhe (struct kvm_vcpu *vcpu,u64 *exit_code) +{ +if (kvm_hyp_handle_cpacr_el1 (vcpu, exit_code)) +return true; + +return kvm_hyp_handle_sysreg (vcpu, exit_code); +} +static const exit_handler_fn hyp_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] =NULL , [ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32, - [ESR_ELx_EC_SYS64] = kvm_hyp_handle_sysreg, + [ESR_ELx_EC_SYS64] = kvm_hyp_handle_sysreg_vhe, [ESR_ELx_EC_SVE] = kvm_hyp_handle_fpsimd, [ESR_ELx_EC_FP_ASIMD] = kvm_hyp_handle_fpsimd, [ESR_ELx_EC_IABT_LOW] = kvm_hyp_handle_iabt_low,
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
Marc Zyngier
maintainerRe: [PATCH v2 03/15] KVM: arm64: nv: Handle CPACR_EL1 traps
Quoted 37 lines · Oliver Upton
From: Marc Zyngier <[email protected]>
Handle
CPACR_EL1accesses when running a VHE guest. In order to limit the cost of the emulation, implement it ass a shallow exit.In the other cases:
- this is a nVHE L1 which will write to memory, and we don't trap
- this is a L2 guest:
* the L1 has CPTR_EL2.TCPAC==0, and the L2 has direct register access
* the L1 has CPTR_EL2.TCPAC==1, and the L2 will trap, but the handling is defered to the general handling for forwarding
Signed-off-by Marc Zyngier <[email protected]>Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/kvm/hyp/vhe/switch.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-)arch/arm64/kvm/hyp/vhe/switch.c +5
-0@@ -262,10 +262,40 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code)return true; } +static bool kvm_hyp_handle_cpacr_el1 (struct kvm_vcpu *vcpu,u64 *exit_code) +{ +u64 esr =kvm_vcpu_get_esr (vcpu); +int rt; +
arch/arm64/kvm/hyp/vhe/switch.c static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) · line 279
Quoted 10 lines
arch/arm64/kvm/hyp/vhe/switch.c +10
-0+if (!is_hyp_ctxt (vcpu) ||esr_sys64_to_sysreg (esr) !=SYS_CPACR_EL1 ) +return false; + + rt =kvm_vcpu_sys_get_rt (vcpu); + +if ((esr & ESR_ELx_SYS64_ISS_DIR_MASK) == ESR_ELx_SYS64_ISS_DIR_READ) { +vcpu_set_reg (vcpu, rt,__vcpu_sys_reg (vcpu,CPTR_EL2 )); + }else { +vcpu_write_sys_reg (vcpu,vcpu_get_reg (vcpu, rt),CPTR_EL2 ); +__activate_cptr_traps (vcpu);
This doesn't bisect, as this helper is only introduced in patch #10. You probably want to keep it towards the end of the series.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH v2 03/15] KVM: arm64: nv: Handle CPACR_EL1 traps
Quoted 44 lines · Marc Zyngier
Quoted 42 lines · Oliver Upton
From: Marc Zyngier <[email protected]>
Handle
CPACR_EL1accesses when running a VHE guest. In order to limit the cost of the emulation, implement it ass a shallow exit.In the other cases:
- this is a nVHE L1 which will write to memory, and we don't trap
- this is a L2 guest:
* the L1 has CPTR_EL2.TCPAC==0, and the L2 has direct register access
* the L1 has CPTR_EL2.TCPAC==1, and the L2 will trap, but the handling is defered to the general handling for forwarding
Signed-off-by Marc Zyngier <[email protected]>Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/kvm/hyp/vhe/switch.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-)arch/arm64/kvm/hyp/vhe/switch.c +10
-0@@ -262,10 +262,40 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code)return true; } +static bool kvm_hyp_handle_cpacr_el1 (struct kvm_vcpu *vcpu,u64 *exit_code) +{ +u64 esr =kvm_vcpu_get_esr (vcpu); +int rt; + +if (!is_hyp_ctxt (vcpu) ||esr_sys64_to_sysreg (esr) !=SYS_CPACR_EL1 ) +return false; + + rt =kvm_vcpu_sys_get_rt (vcpu); +
Quoted 8 lines
arch/arm64/kvm/hyp/vhe/switch.c static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) · line 279
Quoted 5 lines
arch/arm64/kvm/hyp/vhe/switch.c +5
-0+if ((esr & ESR_ELx_SYS64_ISS_DIR_MASK) == ESR_ELx_SYS64_ISS_DIR_READ) { +vcpu_set_reg (vcpu, rt,__vcpu_sys_reg (vcpu,CPTR_EL2 )); + }else { +vcpu_write_sys_reg (vcpu,vcpu_get_reg (vcpu, rt),CPTR_EL2 ); +__activate_cptr_traps (vcpu);This doesn't bisect, as this helper is only introduced in patch #10. You probably want to keep it towards the end of the series.
Ah, derp, I wanted to use the kvm_hyp_handle_sysreg_vhe() you introduced for the subsequent patch. I'll just move them both.
Signature · 2 lines
Thanks, Oliver
PATCH v2 4/15 KVM: arm64: nv: Load guest FP state for ZCR_EL2 trap
In reply to Oliver UptonRound out the ZCR_EL2 gymnastics by loading SVE state in the fast path when the guest hypervisor tries to access SVE state.
--- arch/arm64/kvm/hyp/include/hyp/switch.h | 4 ++++ arch/arm64/kvm/hyp/vhe/switch.c | 27 +++++++++++++++++++++++++ 2 files changed, 31 insertions(+)
arch/arm64/kvm/hyp/include/hyp/switch.h +4 -0
@@ -345,6 +345,10 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)if (guest_hyp_fpsimd_traps_enabled (vcpu))return false;break ; +case ESR_ELx_EC_SYS64: +if (WARN_ON_ONCE (!is_hyp_ctxt (vcpu))) +return false; + fallthrough;case ESR_ELx_EC_SVE:if (!sve_guest)return false;
arch/arm64/kvm/hyp/vhe/switch.c +27 -0
@@ -284,11 +284,38 @@ static bool kvm_hyp_handle_cpacr_el1(struct kvm_vcpu *vcpu, u64 *exit_code)return true; } +static bool kvm_hyp_handle_zcr_el2 (struct kvm_vcpu *vcpu,u64 *exit_code) +{ +u32 sysreg =esr_sys64_to_sysreg (kvm_vcpu_get_esr (vcpu)); + +if (!vcpu_has_nv (vcpu)) +return false; + +if (sysreg !=SYS_ZCR_EL2 ) +return false; + +if (guest_owns_fp_regs ()) +return false; + +/* +* ZCR_EL2 traps are handled in the slow path, with the expectation +* that the guest's FP context has already been loaded onto the CPU. +* +* Load the guest's FP context and unconditionally forward to the +* slow path for handling (i.e. return false). +*/ +kvm_hyp_handle_fpsimd (vcpu, exit_code); +return false; +} +static bool kvm_hyp_handle_sysreg_vhe (struct kvm_vcpu *vcpu,u64 *exit_code) {if (kvm_hyp_handle_cpacr_el1 (vcpu, exit_code))return true; +if (kvm_hyp_handle_zcr_el2 (vcpu, exit_code)) +return true; +return kvm_hyp_handle_sysreg (vcpu, exit_code); }
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 5/15 KVM: arm64: nv: Load guest hyp's ZCR into EL1 state
In reply to Oliver UptonLoad the guest hypervisor's ZCR_EL2 into the corresponding EL1 register when restoring SVE state, as ZCR_EL2 affects the VL in the hypervisor context.
--- arch/arm64/include/asm/kvm_host.h | 4 ++++ arch/arm64/kvm/hyp/include/hyp/switch.h | 3 ++- 2 files changed, 6 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_host.h +4 -0
@@ -844,6 +844,10 @@ struct kvm_vcpu_arch {#define vcpu_sve_max_vq (vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl) +#define vcpu_sve_zcr_el1 (vcpu) \ + (unlikely(is_hyp_ctxt(vcpu)) ? __vcpu_sys_reg(vcpu, ZCR_EL2) : \ + __vcpu_sys_reg(vcpu, ZCR_EL1)) +#define vcpu_sve_state_size (vcpu) ({ \ size_t __size_ret; \ unsigned int __vcpu_vq; \
arch/arm64/kvm/hyp/include/hyp/switch.h +2 -1
@@ -317,7 +317,8 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu)sve_cond_update_zcr_vq (vcpu_sve_max_vq (vcpu) -1 ,SYS_ZCR_EL2 );__sve_restore_state (vcpu_sve_pffr (vcpu), &vcpu->arch.ctxt.fp_regs.fpsr); -write_sysreg_el1 (__vcpu_sys_reg (vcpu,ZCR_EL1 ),SYS_ZCR ); + +write_sysreg_el1 (vcpu_sve_zcr_el1 (vcpu),SYS_ZCR ); }/*
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
Marc Zyngier
maintainerRe: [PATCH v2 05/15] KVM: arm64: nv: Load guest hyp's ZCR into EL1 state
Quoted 16 lines · Oliver Upton
Load the guest hypervisor's
ZCR_EL2into the corresponding EL1 register when restoring SVE state, asZCR_EL2affects the VL in the hypervisor context.Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 4 ++++ arch/arm64/kvm/hyp/include/hyp/switch.h | 3 ++- 2 files changed, 6 insertions(+), 1 deletion(-)arch/arm64/include/asm/kvm_host.h +0
-0
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 850
Quoted 7 lines
arch/arm64/include/asm/kvm_host.h +4
-0@@ -844,6 +844,10 @@ struct kvm_vcpu_arch {#define vcpu_sve_max_vq (vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl) +#define vcpu_sve_zcr_el1 (vcpu) \ + (unlikely(is_hyp_ctxt(vcpu)) ? __vcpu_sys_reg(vcpu, ZCR_EL2) : \ + __vcpu_sys_reg(vcpu, ZCR_EL1)) +
I have the feeling this abstracts the access at the wrong level. It's not that it gives the wrong result, but it hides the register and is only concerned with the *value*.
In turn, it makes the helper unusable with the *write* side, as shown in patch 7.
Quoted 8 lines
arch/arm64/include/asm/kvm_host.h +0
-0#define vcpu_sve_state_size (vcpu) ({ \ size_t __size_ret; \ unsigned int __vcpu_vq; \arch/arm64/kvm/hyp/include/hyp/switch.h +0
-0
arch/arm64/kvm/hyp/include/hyp/switch.h static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu) · line 324
Quoted 9 lines
arch/arm64/kvm/hyp/include/hyp/switch.h +2
-1@@ -317,7 +317,8 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu)sve_cond_update_zcr_vq (vcpu_sve_max_vq (vcpu) -1 ,SYS_ZCR_EL2 );__sve_restore_state (vcpu_sve_pffr (vcpu), &vcpu->arch.ctxt.fp_regs.fpsr); -write_sysreg_el1 (__vcpu_sys_reg (vcpu,ZCR_EL1 ),SYS_ZCR ); + +write_sysreg_el1 (vcpu_sve_zcr_el1 (vcpu),SYS_ZCR ); }/*
I would instead propose the following:
arch/arm64/include/asm/kvm_host.h +2 -3
@@ -845,9 +845,8 @@ struct kvm_vcpu_arch {#define vcpu_sve_max_vq (vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl) -#define vcpu_sve_zcr_el1 (vcpu) \ - (unlikely(is_hyp_ctxt(vcpu)) ? __vcpu_sys_reg(vcpu, ZCR_EL2) : \ - __vcpu_sys_reg(vcpu, ZCR_EL1)) +#define vcpu_sve_zcr_elx (vcpu) \ + (unlikely(is_hyp_ctxt(vcpu)) ? ZCR_EL2 : ZCR_EL1)#define vcpu_sve_state_size (vcpu) ({ \ size_t __size_ret; \
arch/arm64/kvm/fpsimd.c +1 -4
@@ -179,10 +179,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)* If the vCPU is in the hyp context then ZCR_EL1 is * loaded with its vEL2 counterpart. */ -if (is_hyp_ctxt (vcpu)) -__vcpu_sys_reg (vcpu,ZCR_EL2 ) = zcr; -else -__vcpu_sys_reg (vcpu,ZCR_EL1 ) = zcr; +__vcpu_sys_reg (vcpu,vcpu_sve_zcr_elx (vcpu)) = zcr;/* * Restore the VL that was saved when bound to the CPU,
arch/arm64/kvm/hyp/include/hyp/switch.h +1 -1
@@ -330,7 +330,7 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu)if (vcpu_has_nv (vcpu) && !is_hyp_ctxt (vcpu))sve_cond_update_zcr_vq (__vcpu_sys_reg (vcpu,ZCR_EL2 ),SYS_ZCR_EL2 ); -write_sysreg_el1 (vcpu_sve_zcr_el1 (vcpu),SYS_ZCR ); +write_sysreg_el1 (__vcpu_sys_reg (vcpu,vcpu_sve_zcr_elx (vcpu)),SYS_ZCR ); }/*
which makes the helper select the correct guest register for the context, and only that. In turn, the write side is much cleaner and symmetry is restored.
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH v2 05/15] KVM: arm64: nv: Load guest hyp's ZCR into EL1 state
Quoted 21 lines · Marc Zyngier
Quoted 19 lines · Oliver Upton
Load the guest hypervisor's
ZCR_EL2into the corresponding EL1 register when restoring SVE state, asZCR_EL2affects the VL in the hypervisor context.Signed-off-by Oliver Upton <[email protected]>--- arch/arm64/include/asm/kvm_host.h | 4 ++++ arch/arm64/kvm/hyp/include/hyp/switch.h | 3 ++- 2 files changed, 6 insertions(+), 1 deletion(-)arch/arm64/include/asm/kvm_host.h +0
-0@@ -844,6 +844,10 @@ struct kvm_vcpu_arch {#define vcpu_sve_max_vq (vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl)
Quoted 8 lines
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 850
Quoted 4 lines
arch/arm64/include/asm/kvm_host.h +4
-0+#define vcpu_sve_zcr_el1 (vcpu) \ + (unlikely(is_hyp_ctxt(vcpu)) ? __vcpu_sys_reg(vcpu, ZCR_EL2) : \ + __vcpu_sys_reg(vcpu, ZCR_EL1)) +I have the feeling this abstracts the access at the wrong level. It's not that it gives the wrong result, but it hides the register and is only concerned with the *value*.
Agreed, this was done out of hacky convenience for myself and I didn't revisit.
Quoted 46 lines
arch/arm64/include/asm/kvm_host.h +2
-3@@ -845,9 +845,8 @@ struct kvm_vcpu_arch {#define vcpu_sve_max_vq (vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl) -#define vcpu_sve_zcr_el1 (vcpu) \ - (unlikely(is_hyp_ctxt(vcpu)) ? __vcpu_sys_reg(vcpu, ZCR_EL2) : \ - __vcpu_sys_reg(vcpu, ZCR_EL1)) +#define vcpu_sve_zcr_elx (vcpu) \ + (unlikely(is_hyp_ctxt(vcpu)) ? ZCR_EL2 : ZCR_EL1)#define vcpu_sve_state_size (vcpu) ({ \ size_t __size_ret; \arch/arm64/kvm/fpsimd.c +1
-4@@ -179,10 +179,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)* If the vCPU is in the hyp context then ZCR_EL1 is * loaded with its vEL2 counterpart. */ -if (is_hyp_ctxt (vcpu)) -__vcpu_sys_reg (vcpu,ZCR_EL2 ) = zcr; -else -__vcpu_sys_reg (vcpu,ZCR_EL1 ) = zcr; +__vcpu_sys_reg (vcpu,vcpu_sve_zcr_elx (vcpu)) = zcr;/* * Restore the VL that was saved when bound to the CPU, arch/arm64/kvm/hyp/include/hyp/switch.h +1
-1@@ -330,7 +330,7 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu)if (vcpu_has_nv (vcpu) && !is_hyp_ctxt (vcpu))sve_cond_update_zcr_vq (__vcpu_sys_reg (vcpu,ZCR_EL2 ),SYS_ZCR_EL2 ); -write_sysreg_el1 (vcpu_sve_zcr_el1 (vcpu),SYS_ZCR ); +write_sysreg_el1 (__vcpu_sys_reg (vcpu,vcpu_sve_zcr_elx (vcpu)),SYS_ZCR ); }/*
Quoted 3 lines
which makes the helper select the correct guest register for the context, and only that. In turn, the write side is much cleaner and symmetry is restored.
LGTM, I'll squash it in.
Signature · 2 lines
Thanks, Oliver
PATCH v2 6/15 KVM: arm64: nv: Handle ZCR_EL2 traps
In reply to Oliver UptonUnlike other SVE-related registers, ZCR_EL2 takes a sysreg trap to EL2 when HCR_EL2.NV = 1. KVM still needs to honor the guest hypervisor's trap configuration, which expects an SVE trap (i.e. ESR_EL2.EC = 0x19) when CPTR traps are enabled for the vCPU's current context.
Otherwise, if the guest hypervisor has traps disabled, emulate the access by mapping the requested VL into ZCR_EL1.
--- arch/arm64/include/asm/kvm_emulate.h | 8 ++++++ arch/arm64/include/asm/kvm_host.h | 3 +++ arch/arm64/kvm/sys_regs.c | 38 ++++++++++++++++++++++++++++ 3 files changed, 49 insertions(+)
arch/arm64/include/asm/kvm_emulate.h +8 -0
@@ -56,6 +56,14 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu);int kvm_inject_nested_sync (struct kvm_vcpu *vcpu,u64 esr_el2);int kvm_inject_nested_irq (struct kvm_vcpu *vcpu); +static inline void kvm_inject_nested_sve_trap (struct kvm_vcpu *vcpu) +{ +u64 esr =FIELD_PREP (ESR_ELx_EC_MASK, ESR_ELx_EC_SVE) | + ESR_ELx_IL; + +kvm_inject_nested_sync (vcpu, esr); +} +#if defined(__KVM_VHE_HYPERVISOR__) || defined(__KVM_NVHE_HYPERVISOR__) static __always_inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu) {
arch/arm64/include/asm/kvm_host.h +3 -0
@@ -422,6 +422,7 @@ enum vcpu_sysreg {MDCR_EL2 ,/* Monitor Debug Configuration Register (EL2) */ CPTR_EL2 ,/* Architectural Feature Trap Register (EL2) */ HACR_EL2 ,/* Hypervisor Auxiliary Control Register */ +ZCR_EL2 ,/* SVE Control Register (EL2) */ TTBR0_EL2 ,/* Translation Table Base Register 0 (EL2) */ TTBR1_EL2 ,/* Translation Table Base Register 1 (EL2) */ TCR_EL2, /* Translation Control Register (EL2) */
@@ -972,6 +973,7 @@ static inline bool __vcpu_read_sys_reg_from_cpu(int reg, u64 *val)case DACR32_EL2 : *val =read_sysreg_s (SYS_DACR32_EL2 );break ;case IFSR32_EL2 : *val =read_sysreg_s (SYS_IFSR32_EL2 );break ;case DBGVCR32_EL2 : *val =read_sysreg_s (SYS_DBGVCR32_EL2 );break ; +case ZCR_EL1 : *val =read_sysreg_s (SYS_ZCR_EL12 );break ;default :return false; }
@@ -1017,6 +1019,7 @@ static inline bool __vcpu_write_sys_reg_to_cpu(u64 val, int reg)case DACR32_EL2 :write_sysreg_s (val,SYS_DACR32_EL2 );break ;case IFSR32_EL2 :write_sysreg_s (val,SYS_IFSR32_EL2 );break ;case DBGVCR32_EL2 :write_sysreg_s (val,SYS_DBGVCR32_EL2 );break ; +case ZCR_EL1 :write_sysreg_s (val,SYS_ZCR_EL12 );break ;default :return false; }
arch/arm64/kvm/sys_regs.c +38 -0
@@ -121,6 +121,7 @@ static bool get_el2_to_el1_mapping(unsigned int reg,MAPPED_EL2_SYSREG (AMAIR_EL2 ,AMAIR_EL1 ,NULL );MAPPED_EL2_SYSREG (ELR_EL2 ,ELR_EL1 ,NULL );MAPPED_EL2_SYSREG (SPSR_EL2 ,SPSR_EL1 ,NULL ); +MAPPED_EL2_SYSREG (ZCR_EL2 ,ZCR_EL1 ,NULL );default :return false; }
@@ -2199,6 +2200,40 @@ static u64 reset_hcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r)return __vcpu_sys_reg (vcpu, r->reg) = val; } +static unsigned int sve_el2_visibility (const struct kvm_vcpu *vcpu, +const struct sys_reg_desc *rd) +{ +unsigned int r; + + r =el2_visibility (vcpu, rd); +if (r) +return r; + +return sve_visibility (vcpu, rd); +} + +static bool access_zcr_el2 (struct kvm_vcpu *vcpu, +struct sys_reg_params *p, +const struct sys_reg_desc *r) +{ +unsigned int vq; + +if (guest_hyp_sve_traps_enabled (vcpu)) { +kvm_inject_nested_sve_trap (vcpu); +return true; + } + +if (!p->is_write) { + p->regval =vcpu_read_sys_reg (vcpu,ZCR_EL2 ); +return true; + } + + vq =SYS_FIELD_GET (ZCR_ELx,LEN , p->regval) +1 ; + vq =min (vq,vcpu_sve_max_vq (vcpu)); +vcpu_write_sys_reg (vcpu, vq -1 ,ZCR_EL2 ); +return true; +} +/* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2
@@ -2688,6 +2723,9 @@ static const struct sys_reg_desc sys_reg_descs[] = {EL2_REG_VNCR (HFGITR_EL2 , reset_val,0 ),EL2_REG_VNCR (HACR_EL2 , reset_val,0 ), + { SYS_DESC(SYS_ZCR_EL2), .access = access_zcr_el2, .reset = reset_val, + .visibility = sve_el2_visibility, .reg = ZCR_EL2 }, +EL2_REG_VNCR (HCRX_EL2 , reset_val,0 ), EL2_REG(TTBR0_EL2, access_rw, reset_val, 0),
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 7/15 KVM: arm64: nv: Save guest's ZCR_EL2 when in hyp context
In reply to Oliver UptonWhen running a guest hypervisor, ZCR_EL2 is an alias for the counterpart EL1 state.
--- arch/arm64/kvm/fpsimd.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/arm64/kvm/fpsimd.c +10 -1
@@ -173,7 +173,16 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)if (guest_owns_fp_regs ()) {if (vcpu_has_sve (vcpu)) { -__vcpu_sys_reg (vcpu,ZCR_EL1 ) =read_sysreg_el1 (SYS_ZCR ); +u64 zcr =read_sysreg_el1 (SYS_ZCR ); + +/* +* If the vCPU is in the hyp context then ZCR_EL1 is +* loaded with its vEL2 counterpart. +*/ +if (is_hyp_ctxt (vcpu)) +__vcpu_sys_reg (vcpu,ZCR_EL2 ) = zcr; +else +__vcpu_sys_reg (vcpu,ZCR_EL1 ) = zcr;/* * Restore the VL that was saved when bound to the CPU,
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 8/15 KVM: arm64: nv: Use guest hypervisor's max VL when running nested guest
In reply to Oliver UptonThe max VL for nested guests is additionally constrained by the max VL selected by the guest hypervisor. Use that instead of KVM's max VL when running a nested guest.
Note that the guest hypervisor's ZCR_EL2 is sanitised against the VM's max VL at the time of access, so there's no additional handling required at the time of use.
--- arch/arm64/kvm/hyp/include/hyp/switch.h | 12 ++++++++++++ 1 file changed, 12 insertions(+)
arch/arm64/kvm/hyp/include/hyp/switch.h +12 -0
@@ -314,10 +314,22 @@ static bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_code)static inline void __hyp_sve_restore_guest (struct kvm_vcpu *vcpu) { +/* +* The vCPU's saved SVE state layout always matches the max VL of the +* vCPU. Start off with the max VL so we can load the SVE state. +*/ sve_cond_update_zcr_vq (vcpu_sve_max_vq (vcpu) -1 ,SYS_ZCR_EL2 );__sve_restore_state (vcpu_sve_pffr (vcpu), &vcpu->arch.ctxt.fp_regs.fpsr); +/* +* The effective VL for a VM could differ from the max VL when running a +* nested guest, as the guest hypervisor could select a smaller VL. Slap +* that into hardware before wrapping up. +*/ +if (vcpu_has_nv (vcpu) && !is_hyp_ctxt (vcpu)) +sve_cond_update_zcr_vq (__vcpu_sys_reg (vcpu,ZCR_EL2 ),SYS_ZCR_EL2 ); +write_sysreg_el1 (vcpu_sve_zcr_el1 (vcpu),SYS_ZCR ); }
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 9/15 KVM: arm64: nv: Ensure correct VL is loaded before saving SVE state
In reply to Oliver UptonIt is possible that the guest hypervisor has selected a smaller VL than the maximum for its nested guest. As such, ZCR_EL2 may be configured for a different VL when exiting a nested guest.
Set ZCR_EL2 (via the EL1 alias) to the maximum VL for the VM before saving SVE state as the SVE save area is dimensioned by the max VL.
--- arch/arm64/kvm/fpsimd.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
arch/arm64/kvm/fpsimd.c +7 -4
@@ -193,11 +193,14 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)* Note that this means that at guest exit ZCR_EL1 is * not necessarily the same as on guest entry. * -* Restoring the VL isn't needed in VHE mode since -* ZCR_EL2 (accessed via ZCR_EL1) would fulfill the same -* role when doing the save from EL2. +* ZCR_EL2 holds the guest hypervisor's VL when running +* a nested guest, which could be smaller than the +* max for the vCPU. Similar to above, we first need to +* switch to a VL consistent with the layout of the +* vCPU's SVE state. KVM support for NV implies VHE, so +* using the ZCR_EL1 alias is safe. */ -if (!has_vhe ()) +if (!has_vhe () || (vcpu_has_nv (vcpu) && !is_hyp_ctxt (vcpu)))sve_cond_update_zcr_vq (vcpu_sve_max_vq (vcpu) -1 ,SYS_ZCR_EL1 ); }
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 10/15 KVM: arm64: Spin off helper for programming CPTR traps
In reply to Oliver UptonA subsequent change to KVM will add preliminary support for merging a guest hypervisor's CPTR traps with that of KVM. Prepare by spinning off a new helper for managing CPTR traps.
Avoid reading CPACR_EL1 for the baseline trap config, and start off with the most restrictive set of traps that is subsequently relaxed.
--- arch/arm64/kvm/hyp/vhe/switch.c | 49 ++++++++++++++++----------------- 1 file changed, 24 insertions(+), 25 deletions(-)
arch/arm64/kvm/hyp/vhe/switch.c +24 -25
@@ -65,6 +65,29 @@ static u64 __compute_hcr(struct kvm_vcpu *vcpu)return hcr | (__vcpu_sys_reg (vcpu,HCR_EL2 ) & ~NV_HCR_GUEST_EXCLUDE ); } +static void __activate_cptr_traps (struct kvm_vcpu *vcpu) +{ +/* +* With VHE (HCR.E2H == 1), accesses to CPACR_EL1 are routed to +* CPTR_EL2. In general, CPACR_EL1 has the same layout as CPTR_EL2, +* except for some missing controls, such as TAM. +* In this case, CPTR_EL2.TAM has the same position with or without +* VHE (HCR.E2H == 1) which allows us to use here the CPTR_EL2.TAM +* shift value for trapping the AMU accesses. +*/ +u64 val = CPACR_ELx_TTA |CPTR_EL2_TAM ; + +if (guest_owns_fp_regs ()) { + val |= CPACR_ELx_FPEN; +if (vcpu_has_sve (vcpu)) + val |= CPACR_ELx_ZEN; + }else { +__activate_traps_fpsimd32 (vcpu); + } + +write_sysreg (val, cpacr_el1); +} +static void __activate_traps (struct kvm_vcpu *vcpu) {u64 val;
@@ -91,31 +114,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) } } - val =read_sysreg (cpacr_el1); - val |= CPACR_ELx_TTA; - val &= ~(CPACR_EL1_ZEN_EL0EN |CPACR_EL1_ZEN_EL1EN | -CPACR_EL1_SMEN_EL0EN |CPACR_EL1_SMEN_EL1EN ); - -/* -* With VHE (HCR.E2H == 1), accesses to CPACR_EL1 are routed to -* CPTR_EL2. In general, CPACR_EL1 has the same layout as CPTR_EL2, -* except for some missing controls, such as TAM. -* In this case, CPTR_EL2.TAM has the same position with or without -* VHE (HCR.E2H == 1) which allows us to use here the CPTR_EL2.TAM -* shift value for trapping the AMU accesses. -*/ - - val |=CPTR_EL2_TAM ; - -if (guest_owns_fp_regs ()) { -if (vcpu_has_sve (vcpu)) - val |=CPACR_EL1_ZEN_EL0EN |CPACR_EL1_ZEN_EL1EN ; - }else { - val &= ~(CPACR_EL1_FPEN_EL0EN |CPACR_EL1_FPEN_EL1EN ); -__activate_traps_fpsimd32 (vcpu); - } - -write_sysreg (val, cpacr_el1); +__activate_cptr_traps (vcpu);write_sysreg (__this_cpu_read (kvm_hyp_vector), vbar_el1); }
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 11/15 KVM: arm64: nv: Honor guest hypervisor's FP/SVE traps in CPTR_EL2
In reply to Oliver UptonStart folding the guest hypervisor's FP/SVE traps into the value programmed in hardware. Note that as of writing this is dead code, since KVM does a full put() / load() for every nested exception boundary which saves + flushes the FP/SVE state.
However, this will become useful when we can keep the guest's FP/SVE state alive across a nested exception boundary and the host no longer needs to conservatively program traps.
--- arch/arm64/kvm/hyp/vhe/switch.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
arch/arm64/kvm/hyp/vhe/switch.c +31 -0
@@ -67,6 +67,8 @@ static u64 __compute_hcr(struct kvm_vcpu *vcpu)static void __activate_cptr_traps (struct kvm_vcpu *vcpu) { +u64 cptr; +/* * With VHE (HCR.E2H == 1), accesses to CPACR_EL1 are routed to * CPTR_EL2. In general, CPACR_EL1 has the same layout as CPTR_EL2,
@@ -85,6 +87,35 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu)__activate_traps_fpsimd32 (vcpu); } +/* +* Layer the guest hypervisor's trap configuration on top of our own if +* we're in a nested context. +*/ +if (!vcpu_has_nv (vcpu) ||is_hyp_ctxt (vcpu)) + goto write; + + cptr =vcpu_sanitised_cptr_el2 (vcpu); + +/* +* Pay attention, there's some interesting detail here. +* +* The CPTR_EL2.xEN fields are 2 bits wide, although there are only two +* meaningful trap states when HCR_EL2.TGE = 0 (running a nested guest): +* +* - CPTR_EL2.xEN = x0, traps are enabled +* - CPTR_EL2.xEN = x1, traps are disabled +* +* In other words, bit[0] determines if guest accesses trap or not. In +* the interest of simplicity, clear the entire field if the guest +* hypervisor has traps enabled to dispel any illusion of something more +* complicated taking place. +*/ +if (!(SYS_FIELD_GET (CPACR_ELx,FPEN , cptr) &BIT (0 ))) + val &= ~CPACR_ELx_FPEN; +if (!(SYS_FIELD_GET (CPACR_ELx,ZEN , cptr) &BIT (0 ))) + val &= ~CPACR_ELx_ZEN; + +write:write_sysreg (val, cpacr_el1); }
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 12/15 KVM: arm64: nv: Add TCPAC/TTA to CPTR->CPACR conversion helper
In reply to Oliver UptonFrom: Marc Zyngier <[email protected]>
We are missing the propagation of CPTR_EL2.{TCPAC,TTA} into the CPACR format. Make sure we preserve these bits.
--- arch/arm64/include/asm/kvm_nested.h | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_nested.h +3 -1
@@ -32,7 +32,7 @@ static inline u64 translate_tcr_el2_to_tcr_el1(u64 tcr)static inline u64 translate_cptr_el2_to_cpacr_el1 (u64 cptr_el2) { -u64 cpacr_el1 =0 ; +u64 cpacr_el1 = CPACR_ELx_RES1;if (cptr_el2 &CPTR_EL2_TTA ) cpacr_el1 |= CPACR_ELx_TTA;
@@ -41,6 +41,8 @@ static inline u64 translate_cptr_el2_to_cpacr_el1(u64 cptr_el2)if (!(cptr_el2 &CPTR_EL2_TZ )) cpacr_el1 |= CPACR_ELx_ZEN; + cpacr_el1 |= cptr_el2 & (CPTR_EL2_TCPAC |CPTR_EL2_TAM ); +return cpacr_el1; }
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 13/15 KVM: arm64: nv: Add trap description for CPTR_EL2
In reply to Oliver UptonFrom: Marc Zyngier <[email protected]>
Add trap description for CPTR_EL2.{TCPAC,TAM,E0POE,TTA}.
TTA is a bit annoying as it changes location depending on E2H. This forces us to add yet another "complex" trap condition.
--- arch/arm64/kvm/emulate-nested.c | 91 +++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+)
arch/arm64/kvm/emulate-nested.c +91 -0
@@ -79,6 +79,10 @@ enum cgt_group_id {CGT_MDCR_E2TB ,CGT_MDCR_TDCC , +CGT_CPACR_E0POE , +CGT_CPTR_TAM , +CGT_CPTR_TCPAC , +/* * Anything after this point is a combination of coarse trap * controls, which must all be evaluated to decide what to do.
@@ -106,6 +110,8 @@ enum cgt_group_id {CGT_CNTHCTL_EL1PCTEN = __COMPLEX_CONDITIONS__,CGT_CNTHCTL_EL1PTEN , +CGT_CPTR_TTA , +/* Must be last */ __NR_CGT_GROUP_IDS__ };
@@ -345,6 +351,24 @@ static const struct trap_bits coarse_trap_bits[] = { .mask = MDCR_EL2_TDCC, .behaviour = BEHAVE_FORWARD_ANY, }, + [CGT_CPACR_E0POE] = { + .index = CPTR_EL2, + .value = CPACR_ELx_E0POE, + .mask = CPACR_ELx_E0POE, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_CPTR_TAM] = { + .index = CPTR_EL2, + .value = CPTR_EL2_TAM, + .mask = CPTR_EL2_TAM, + .behaviour = BEHAVE_FORWARD_ANY, + }, + [CGT_CPTR_TCPAC] = { + .index = CPTR_EL2, + .value = CPTR_EL2_TCPAC, + .mask = CPTR_EL2_TCPAC, + .behaviour = BEHAVE_FORWARD_ANY, + }, };#define MCB (id, ...) \
@@ -410,12 +434,26 @@ static enum trap_behaviour check_cnthctl_el1pten(struct kvm_vcpu *vcpu)return BEHAVE_FORWARD_ANY ; } +static enum trap_behaviour check_cptr_tta (struct kvm_vcpu *vcpu) +{ +u64 val =__vcpu_sys_reg (vcpu,CPTR_EL2 ); + +if (!vcpu_el2_e2h_is_set (vcpu)) + val =translate_cptr_el2_to_cpacr_el1 (val); + +if (val & CPACR_ELx_TTA) +return BEHAVE_FORWARD_ANY ; + +return BEHAVE_HANDLE_LOCALLY ; +} +#define CCC (id, fn) \ [id - __COMPLEX_CONDITIONS__] = fnstatic const complex_condition_check ccc[] = {CCC (CGT_CNTHCTL_EL1PCTEN , check_cnthctl_el1pcten),CCC (CGT_CNTHCTL_EL1PTEN , check_cnthctl_el1pten), +CCC (CGT_CPTR_TTA , check_cptr_tta), };/*
@@ -1000,6 +1038,59 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initconst = {SR_TRAP (SYS_TRBPTR_EL1 ,CGT_MDCR_E2TB ),SR_TRAP (SYS_TRBSR_EL1 ,CGT_MDCR_E2TB ),SR_TRAP (SYS_TRBTRG_EL1 ,CGT_MDCR_E2TB ), +SR_TRAP (SYS_CPACR_EL1 ,CGT_CPTR_TCPAC ), +SR_TRAP (SYS_AMUSERENR_EL0 ,CGT_CPTR_TAM ), +SR_TRAP (SYS_AMCFGR_EL0 ,CGT_CPTR_TAM ), +SR_TRAP (SYS_AMCGCR_EL0 ,CGT_CPTR_TAM ), +SR_TRAP (SYS_AMCNTENCLR0_EL0 ,CGT_CPTR_TAM ), +SR_TRAP (SYS_AMCNTENCLR1_EL0 ,CGT_CPTR_TAM ), +SR_TRAP (SYS_AMCNTENSET0_EL0 ,CGT_CPTR_TAM ), +SR_TRAP (SYS_AMCNTENSET1_EL0 ,CGT_CPTR_TAM ), +SR_TRAP (SYS_AMCR_EL0 ,CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR0_EL0 (0 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR0_EL0 (1 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR0_EL0 (2 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR0_EL0 (3 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (0 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (1 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (2 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (3 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (4 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (5 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (6 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (7 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (8 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (9 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (10 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (11 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (12 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (13 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (14 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVCNTR1_EL0 (15 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER0_EL0 (0 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER0_EL0 (1 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER0_EL0 (2 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER0_EL0 (3 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (0 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (1 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (2 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (3 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (4 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (5 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (6 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (7 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (8 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (9 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (10 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (11 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (12 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (13 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (14 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_AMEVTYPER1_EL0 (15 ),CGT_CPTR_TAM ), +SR_TRAP (SYS_POR_EL0 ,CGT_CPACR_E0POE ), +/* op0=2, op1=1, and CRn<0b1000 */ +SR_RANGE_TRAP (sys_reg (2 ,1 ,0 ,0 ,0 ), +sys_reg (2 ,1 ,7 ,15 ,7 ),CGT_CPTR_TTA ),SR_TRAP (SYS_CNTP_TVAL_EL0 ,CGT_CNTHCTL_EL1PTEN ),SR_TRAP (SYS_CNTP_CVAL_EL0 ,CGT_CNTHCTL_EL1PTEN ), SR_TRAP(SYS_CNTP_CTL_EL0, CGT_CNTHCTL_EL1PTEN),
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 14/15 KVM: arm64: nv: Add additional trap setup for CPTR_EL2
In reply to Oliver UptonFrom: Marc Zyngier <[email protected]>
We need to teach KVM a couple of new tricks. CPTR_EL2 and its VHE accessor CPACR_EL1 need to be handled specially:
- CPACR_EL1 is trapped on VHE so that we can track the TCPAC and TTA bits
- CPTR_EL2.{TCPAC,E0POE} are propagated from L1 to L2
--- arch/arm64/kvm/hyp/vhe/switch.c | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/vhe/switch.c +18 -1
@@ -87,11 +87,23 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu)__activate_traps_fpsimd32 (vcpu); } +if (!vcpu_has_nv (vcpu)) + goto write; + +/* +* The architecture is a bit crap (what a surprise): an EL2 guest +* writing to CPTR_EL2 via CPACR_EL1 can't set any of TCPAC or TTA, +* as they are RES0 in the guest's view. To work around it, trap the +* sucker using the very same bit it can't set... +*/ +if (vcpu_el2_e2h_is_set (vcpu) &&is_hyp_ctxt (vcpu)) + val |=CPTR_EL2_TCPAC ; +/* * Layer the guest hypervisor's trap configuration on top of our own if * we're in a nested context. */ -if (!vcpu_has_nv (vcpu) ||is_hyp_ctxt (vcpu)) +if (is_hyp_ctxt (vcpu)) goto write; cptr =vcpu_sanitised_cptr_el2 (vcpu);
@@ -115,6 +127,11 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu)if (!(SYS_FIELD_GET (CPACR_ELx,ZEN , cptr) &BIT (0 ))) val &= ~CPACR_ELx_ZEN; +if (kvm_has_feat (vcpu->kvm,ID_AA64MMFR3_EL1 ,S2POE ,IMP )) + val |= cptr & CPACR_ELx_E0POE; + + val |= cptr &CPTR_EL2_TCPAC ; + write:write_sysreg (val, cpacr_el1); }
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
PATCH v2 15/15 KVM: arm64: Allow the use of SVE+NV
In reply to Oliver UptonAllow SVE and NV to mix now that everything is in place to handle it correctly.
--- arch/arm64/kvm/arm.c | 5 ----- arch/arm64/kvm/nested.c | 3 +-- 2 files changed, 1 insertion(+), 7 deletions(-)
arch/arm64/kvm/arm.c +0 -5
@@ -1419,11 +1419,6 @@ static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu, test_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, &features))return -EINVAL ; -/* Disallow NV+SVE for the time being */ -if (test_bit (KVM_ARM_VCPU_HAS_EL2 , &features) && -test_bit (KVM_ARM_VCPU_SVE , &features)) -return -EINVAL ; -if (!test_bit (KVM_ARM_VCPU_EL1_32BIT , &features))return 0 ;
arch/arm64/kvm/nested.c +1 -2
@@ -41,13 +41,12 @@ static u64 limit_nv_id_reg(u32 id, u64 val)break ;case SYS_ID_AA64PFR0_EL1 : -/* No AMU, MPAM, S-EL2, RAS or SVE */ +/* No AMU, MPAM, S-EL2, or RAS */ val &= ~(GENMASK_ULL (55 ,52 ) |NV_FTR (PFR0 ,AMU ) |NV_FTR (PFR0 ,MPAM ) |NV_FTR (PFR0 ,SEL2 ) |NV_FTR (PFR0 ,RAS ) | -NV_FTR (PFR0 ,SVE ) |NV_FTR (PFR0 ,EL3 ) |NV_FTR (PFR0 ,EL2 ) |NV_FTR (PFR0 ,EL1 ));
Signature · 1 line
2.45.2.627.g7a2c4fd464-goog
Quoted 46 lines · Oliver Upton
As discussed, here's the combined series of Marc + I's patches that implement all the trap forwarding / merging logic required to observe the L1 CPTR configuration when running an L2 guest.
Like before, this was tested on Neoverse-V2 with L0, L1, and L2 running fpsimd-test and sve-test on top of one another.
v1 [1] -> v2:
- Grab Marc's CPTR trap patches [2]
- Avoid taking two traps when L1 accessesZCR_EL2while the host owns FP regs. The fast path will now load the FP regs and forward the sysreg trap to the slow path for complete handling.
- Add a comment describing the above behavior (Marc)
- Avoid the use of guest_hyp_*_traps_enabled() in__activate_cptr_traps()for better codegen (Marc)
- Document the reason for only testing bit[0] of CPTR_EL2.xEN when folding L1 traps into the hardware CPTR value.
- Add a helper for synthesizing SVE traps, rather than open-coding the ESR value inaccess_zcr_el2()[1]: https://lore.kernel.org/kvmarm/[email protected]/
[2]: https://lore.kernel.org/kvmarm/[email protected]/Jintack Lim (1):
KVM: arm64: nv: Forward FP/ASIMD traps to guest hypervisorMarc Zyngier (4):
KVM: arm64: nv: HandleCPACR_EL1traps
KVM: arm64: nv: Add TCPAC/TTA toCPTR->CPACRconversion helper
KVM: arm64: nv: Add trap description forCPTR_EL2
KVM: arm64: nv: Add additional trap setup forCPTR_EL2Oliver Upton (10):
KVM: arm64: nv: Forward SVE traps to guest hypervisor
KVM: arm64: nv: Load guest FP state forZCR_EL2trap
KVM: arm64: nv: Load guest hyp's ZCR into EL1 state
KVM: arm64: nv: HandleZCR_EL2traps
KVM: arm64: nv: Save guest'sZCR_EL2when in hyp context
KVM: arm64: nv: Use guest hypervisor's max VL when running nested
guest
KVM: arm64: nv: Ensure correct VL is loaded before saving SVE state
KVM: arm64: Spin off helper for programming CPTR traps
KVM: arm64: nv: Honor guest hypervisor's FP/SVE traps inCPTR_EL2
KVM: arm64: Allow the use of SVE+NV
Quoted 12 lines
arch/arm64/include/asm/kvm_emulate.h | 55 +++++++++ arch/arm64/include/asm/kvm_host.h | 7 ++ arch/arm64/include/asm/kvm_nested.h | 5 +- arch/arm64/kvm/arm.c | 5 - arch/arm64/kvm/emulate-nested.c | 91 ++++++++++++++ arch/arm64/kvm/fpsimd.c | 22 +++- arch/arm64/kvm/handle_exit.c | 19 ++- arch/arm64/kvm/hyp/include/hyp/switch.h | 24 +++- arch/arm64/kvm/hyp/vhe/switch.c | 156 ++++++++++++++++++++---- arch/arm64/kvm/nested.c | 3 +- arch/arm64/kvm/sys_regs.c | 38 ++++++ 11 files changed, 380 insertions(+), 45 deletions(-)
Patches 1,5,7 may require some very minor rework, but they are functionally correct AFAICT. Patches 3,12-14 do not warrant my RB.
For patches 1-2,4-11,15, and with the bisection issue fixed:
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.