KVM: arm64: Add support for FP8
Marc Zyngier
maintainerv4 cover KVM: arm64: Add support for FP8
Although FP8 support was merged in 6.9, the KVM side was dropped, with no sign of it being picked up again. Given that its absence is getting in the way of NV upstreaming (HCRX_EL2 needs fleshing out), here's a small series addressing it.
The support is following the save/restore model established for the rest of the FP code, with FPMR being tied to it. The sole additions are the handling of traps in a nested context, and the corresponding ID registers being made writable. As an extra cleanup, SVCR and FPMR are moved into the sysreg array.
Patches are on top of v6.11-rc1. Note that this is compile-tested only, as I have no access to FP8 HW or model (and running NV in a model is not something I wish to entertain ever again).
* From v3 [3]
- Simplify handling of SYS_ID_AA64PFR2_EL1
* From v2 [2]
- Add missing kern_hyp_va() when dereferencing vcpu->kvm on nVHE setups
* From v1 [1]
- Correctly save/restore the guest state (duh), including pKVM (double duh)
- Add a predicate for FPMR support in a VM, as this gets used more than twice...
[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]
[3] https://lore.kernel.org/r/[email protected]
Marc Zyngier (8):
KVM: arm64: Move SVCR into the sysreg array
KVM: arm64: Add predicate for FPMR support in a VM
KVM: arm64: Move FPMR into the sysreg array
KVM: arm64: Add save/restore support for FPMR
KVM: arm64: Honor trap routing for FPMR
KVM: arm64: Expose ID_AA64FPFR0_EL1 as a writable ID reg
KVM: arm64: Enable FP8 support when available and configured
KVM: arm64: Expose ID_AA64PFR2_EL1 to userspace and guests
arch/arm64/include/asm/kvm_host.h | 20 ++++++++++++++-- arch/arm64/kvm/emulate-nested.c | 8 +++++++ arch/arm64/kvm/fpsimd.c | 5 ++-- arch/arm64/kvm/hyp/include/hyp/switch.h | 3 +++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 9 +++++++ arch/arm64/kvm/hyp/nvhe/switch.c | 9 +++++++ arch/arm64/kvm/hyp/vhe/switch.c | 3 +++ arch/arm64/kvm/sys_regs.c | 32 ++++++++++++++++++++++--- 8 files changed, 82 insertions(+), 7 deletions(-)
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 1/8 KVM: arm64: Move SVCR into the sysreg array
SVCR is just a system register, and has no purpose being outside of the sysreg array. If anything, it only makes it more difficult to eventually support SME one day. If ever.
Move it into the array with its little friends, and associate it with a visibility predicate.
Although this is dead code, it at least paves the way for the next set of FP-related extensions.
--- arch/arm64/include/asm/kvm_host.h | 4 +++- arch/arm64/kvm/fpsimd.c | 2 +- arch/arm64/kvm/sys_regs.c | 11 ++++++++++- 3 files changed, 14 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_host.h +3 -1
@@ -446,6 +446,9 @@ enum vcpu_sysreg {GCR_EL1 ,/* Tag Control Register */ TFSRE0_EL1 ,/* Tag Fault Status Register (EL0) */ +/* FP/SIMD/SVE */ +SVCR , +/* 32bit specific registers. */ DACR32_EL2 ,/* Domain Access Control Register */ IFSR32_EL2, /* Instruction Fault Status Register */
@@ -664,7 +667,6 @@ struct kvm_vcpu_arch {void *sve_state;enum fp_type fp_type;unsigned int sve_max_vl; -u64 svcr;u64 fpmr;/* Stage 2 paging state used by the hardware on next switch */
arch/arm64/kvm/fpsimd.c +1 -1
@@ -134,7 +134,7 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) fp_state.sve_state = vcpu->arch.sve_state; fp_state.sve_vl = vcpu->arch.sve_max_vl; fp_state.sme_state =NULL ; - fp_state.svcr = &vcpu->arch.svcr; + fp_state.svcr = &__vcpu_sys_reg (vcpu,SVCR ); fp_state.fpmr = &vcpu->arch.fpmr; fp_state.fp_type = &vcpu->arch.fp_type;
arch/arm64/kvm/sys_regs.c +10 -1
@@ -1669,6 +1669,15 @@ static unsigned int sve_visibility(const struct kvm_vcpu *vcpu,return REG_HIDDEN ; } +static unsigned int sme_visibility (const struct kvm_vcpu *vcpu, +const struct sys_reg_desc *rd) +{ +if (kvm_has_feat (vcpu->kvm,ID_AA64PFR1_EL1 ,SME ,IMP )) +return 0 ; + +return REG_HIDDEN ; +} +static u64 read_sanitised_id_aa64pfr0_el1 (struct kvm_vcpu *vcpu,const struct sys_reg_desc *rd) {
@@ -2535,7 +2544,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {CTR_EL0_IDC_MASK | CTR_EL0_DminLine_MASK | CTR_EL0_IminLine_MASK), - { SYS_DESC(SYS_SVCR), undef_access }, + { SYS_DESC(SYS_SVCR), undef_access, reset_val, SVCR, 0, .visibility = sme_visibility }, { PMU_SYS_REG(PMCR_EL0), .access = access_pmcr, .reset = reset_pmcr, .reg = PMCR_EL0, .get_user = get_pmcr, .set_user = set_pmcr },
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 2/8 KVM: arm64: Add predicate for FPMR support in a VM
In reply to Marc ZyngierAs we are about to check for the advertisement of FPMR support to a guest in a number of places, add a predicate that will gate most of the support code for FPMR.
--- arch/arm64/include/asm/kvm_host.h | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/include/asm/kvm_host.h +4 -0
@@ -1475,4 +1475,8 @@ void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val); (pa + pi + pa3) ==1 ; \ }) +#define kvm_has_fpmr (k) \ + (system_supports_fpmr() && \ + kvm_has_feat((k), ID_AA64PFR2_EL1, FPMR, IMP)) +#endif /* __ARM64_KVM_HOST_H__ */
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 3/8 KVM: arm64: Move FPMR into the sysreg array
In reply to Marc ZyngierJust like SVCR, FPMR is currently stored at the wrong location.
Let's move it where it belongs.
--- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kvm/fpsimd.c | 2 +- arch/arm64/kvm/sys_regs.c | 10 ++++++++++ 3 files changed, 12 insertions(+), 2 deletions(-)
arch/arm64/include/asm/kvm_host.h +1 -1
@@ -448,6 +448,7 @@ enum vcpu_sysreg {/* FP/SIMD/SVE */ SVCR , +FPMR ,/* 32bit specific registers. */ DACR32_EL2, /* Domain Access Control Register */
@@ -667,7 +668,6 @@ struct kvm_vcpu_arch {void *sve_state;enum fp_type fp_type;unsigned int sve_max_vl; -u64 fpmr;/* Stage 2 paging state used by the hardware on next switch */ struct kvm_s2_mmu *hw_mmu;
arch/arm64/kvm/fpsimd.c +1 -1
@@ -135,7 +135,7 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) fp_state.sve_vl = vcpu->arch.sve_max_vl; fp_state.sme_state =NULL ; fp_state.svcr = &__vcpu_sys_reg (vcpu,SVCR ); - fp_state.fpmr = &vcpu->arch.fpmr; + fp_state.fpmr = &__vcpu_sys_reg (vcpu,FPMR ); fp_state.fp_type = &vcpu->arch.fp_type;if (vcpu_has_sve (vcpu))
arch/arm64/kvm/sys_regs.c +10 -0
@@ -1678,6 +1678,15 @@ static unsigned int sme_visibility(const struct kvm_vcpu *vcpu,return REG_HIDDEN ; } +static unsigned int fp8_visibility (const struct kvm_vcpu *vcpu, +const struct sys_reg_desc *rd) +{ +if (kvm_has_fpmr (vcpu->kvm)) +return 0 ; + +return REG_HIDDEN ; +} +static u64 read_sanitised_id_aa64pfr0_el1 (struct kvm_vcpu *vcpu,const struct sys_reg_desc *rd) {
@@ -2545,6 +2554,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { CTR_EL0_DminLine_MASK | CTR_EL0_IminLine_MASK), { SYS_DESC(SYS_SVCR), undef_access, reset_val, SVCR, 0, .visibility = sme_visibility }, + { SYS_DESC(SYS_FPMR), undef_access, reset_val, FPMR, 0, .visibility = fp8_visibility }, { PMU_SYS_REG(PMCR_EL0), .access = access_pmcr, .reset = reset_pmcr, .reg = PMCR_EL0, .get_user = get_pmcr, .set_user = set_pmcr },
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 4/8 KVM: arm64: Add save/restore support for FPMR
In reply to Marc ZyngierJust like the rest of the FP/SIMD state, FPMR needs to be context switched.
The only interesting thing here is that we need to treat the pKVM part a bit differently, as the host FP state is never written back to the vcpu thread, but instead stored locally and eagerly restored.
--- arch/arm64/include/asm/kvm_host.h | 10 ++++++++++ arch/arm64/kvm/fpsimd.c | 1 + arch/arm64/kvm/hyp/include/hyp/switch.h | 3 +++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 9 +++++++++ arch/arm64/kvm/hyp/nvhe/switch.c | 9 +++++++++ arch/arm64/kvm/hyp/vhe/switch.c | 3 +++ 6 files changed, 35 insertions(+)
arch/arm64/include/asm/kvm_host.h +10 -0
@@ -599,6 +599,16 @@ struct kvm_host_data {struct cpu_sve_state *sve_state; }; +union { +/* HYP VA pointer to the host storage for FPMR */ +u64 *fpmr_ptr; +/* +* Used by pKVM only, as it needs to provide storage +* for the host +*/ +u64 fpmr; + }; +/* Ownership of the FP regs */ enum {FP_STATE_FREE ,
arch/arm64/kvm/fpsimd.c +1 -0
@@ -63,6 +63,7 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu)*/ *host_data_ptr (fp_owner) =FP_STATE_HOST_OWNED ; *host_data_ptr (fpsimd_state) =kern_hyp_va (¤t->thread.uw.fpsimd_state); + *host_data_ptr (fpmr_ptr) =kern_hyp_va (¤t->thread.uw.fpmr);vcpu_clear_flag (vcpu,HOST_SVE_ENABLED );if (read_sysreg (cpacr_el1) &CPACR_EL1_ZEN_EL0EN )
arch/arm64/kvm/hyp/include/hyp/switch.h +3 -0
@@ -404,6 +404,9 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)else __fpsimd_restore_state (&vcpu->arch.ctxt.fp_regs); +if (kvm_has_fpmr (kern_hyp_va (vcpu->kvm))) +write_sysreg_s (__vcpu_sys_reg (vcpu,FPMR ),SYS_FPMR ); +/* Skip restoring fpexc32 for AArch64 guests */ if (!(read_sysreg (hcr_el2) &HCR_RW ))write_sysreg (__vcpu_sys_reg (vcpu,FPEXC32_EL2 ), fpexc32_el2);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +9 -0
@@ -62,6 +62,8 @@ static void fpsimd_sve_flush(void)static void fpsimd_sve_sync (struct kvm_vcpu *vcpu) { +bool has_fpmr; +if (!guest_owns_fp_regs ())return ;
@@ -73,11 +75,18 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu)else __fpsimd_save_state (&vcpu->arch.ctxt.fp_regs); + has_fpmr =kvm_has_fpmr (kern_hyp_va (vcpu->kvm)); +if (has_fpmr) +__vcpu_sys_reg (vcpu,FPMR ) =read_sysreg_s (SYS_FPMR ); +if (system_supports_sve ())__hyp_sve_restore_host ();else __fpsimd_restore_state (*host_data_ptr (fpsimd_state)); +if (has_fpmr) +write_sysreg_s (*host_data_ptr (fpmr),SYS_FPMR ); + *host_data_ptr (fp_owner) =FP_STATE_HOST_OWNED ; }
arch/arm64/kvm/hyp/nvhe/switch.c +9 -0
@@ -198,6 +198,15 @@ static void kvm_hyp_save_fpsimd_host(struct kvm_vcpu *vcpu) }else {__fpsimd_save_state (*host_data_ptr (fpsimd_state)); } + +if (kvm_has_fpmr (kern_hyp_va (vcpu->kvm))) { +u64 val =read_sysreg_s (SYS_FPMR ); + +if (unlikely (is_protected_kvm_enabled ())) + *host_data_ptr (fpmr) = val; +else + **host_data_ptr (fpmr_ptr) = val; + } }static const exit_handler_fn hyp_exit_handlers[] = {
arch/arm64/kvm/hyp/vhe/switch.c +3 -0
@@ -312,6 +312,9 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code)static void kvm_hyp_save_fpsimd_host (struct kvm_vcpu *vcpu) {__fpsimd_save_state (*host_data_ptr (fpsimd_state)); + +if (kvm_has_fpmr (vcpu->kvm)) + **host_data_ptr (fpmr_ptr) =read_sysreg_s (SYS_FPMR ); }static bool kvm_hyp_handle_tlbi_el2 (struct kvm_vcpu *vcpu,u64 *exit_code)
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 5/8 KVM: arm64: Honor trap routing for FPMR
In reply to Marc ZyngierHCRX_EL2.EnFPM controls the trapping of FPMR (as well as the validity of any FP8 instruction, but we don't really care about this last part).
Describe the trap bit so that the exception can be reinjected in a NV guest.
--- arch/arm64/kvm/emulate-nested.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/arm64/kvm/emulate-nested.c +8 -0
@@ -83,6 +83,7 @@ enum cgt_group_id {CGT_CPTR_TAM ,CGT_CPTR_TCPAC , + CGT_HCRX_EnFPM, CGT_HCRX_TCR2En,/*
@@ -372,6 +373,12 @@ static const struct trap_bits coarse_trap_bits[] = { .mask = CPTR_EL2_TCPAC, .behaviour = BEHAVE_FORWARD_ANY, }, + [CGT_HCRX_EnFPM] = { + .index = HCRX_EL2, + .value = 0, + .mask = HCRX_EL2_EnFPM, + .behaviour = BEHAVE_FORWARD_ANY, + }, [CGT_HCRX_TCR2En] = { .index = HCRX_EL2, .value = 0,
@@ -1108,6 +1115,7 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initconst = {SR_TRAP (SYS_CNTP_CTL_EL0 ,CGT_CNTHCTL_EL1PTEN ),SR_TRAP (SYS_CNTPCT_EL0 ,CGT_CNTHCTL_EL1PCTEN ),SR_TRAP (SYS_CNTPCTSS_EL0 ,CGT_CNTHCTL_EL1PCTEN ), + SR_TRAP(SYS_FPMR, CGT_HCRX_EnFPM), };static DEFINE_XARRAY (sr_forward_xa);
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 6/8 KVM: arm64: Expose ID_AA64FPFR0_EL1 as a writable ID reg
In reply to Marc ZyngierID_AA64FPFR0_EL1 contains all sort of bits that contain a description of which FP8 subfeatures are implemented.
We don't really care about them, so let's just expose that register and allow userspace to disable subfeatures at will.
--- arch/arm64/kvm/sys_regs.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/sys_regs.c +1 -1
@@ -2386,7 +2386,7 @@ static const struct sys_reg_desc sys_reg_descs[] = {ID_WRITABLE (ID_AA64ZFR0_EL1 , ~ID_AA64ZFR0_EL1_RES0 ),ID_HIDDEN (ID_AA64SMFR0_EL1 ),ID_UNALLOCATED (4 ,6 ), -ID_UNALLOCATED (4 ,7 ), +ID_WRITABLE (ID_AA64FPFR0_EL1 , ~ID_AA64FPFR0_EL1_RES0 ),/* CRm=5 */ { SYS_DESC(SYS_ID_AA64DFR0_EL1),
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 7/8 KVM: arm64: Enable FP8 support when available and configured
In reply to Marc ZyngierIf userspace has enabled FP8 support (by setting ID_AA64PFR2_EL1.FPMR to 1), let's enable the feature by setting HCRX_EL2.EnFPM for the vcpu.
--- arch/arm64/kvm/sys_regs.c | 3 +++ 1 file changed, 3 insertions(+)
arch/arm64/kvm/sys_regs.c +3 -0
@@ -4579,6 +4579,9 @@ void kvm_calculate_traps(struct kvm_vcpu *vcpu)if (kvm_has_feat (kvm,ID_AA64MMFR3_EL1 ,TCRX ,IMP )) vcpu->arch.hcrx_el2 |= HCRX_EL2_TCR2En; + +if (kvm_has_fpmr (kvm)) + vcpu->arch.hcrx_el2 |= HCRX_EL2_EnFPM; }if (test_bit (KVM_ARCH_FLAG_FGU_INITIALIZED , &kvm->arch.flags))
Signature · 1 line
2.39.2
Marc Zyngier
maintainerPATCH v4 8/8 KVM: arm64: Expose ID_AA64PFR2_EL1 to userspace and guests
In reply to Marc ZyngierEverything is now in place for a guest to "enjoy" FP8 support. Expose ID_AA64PFR2_EL1 to both userspace and guests, with the explicit restriction of only being able to clear FPMR.
All other features (MTE* at the time of writing) are hidden and not writable.
--- arch/arm64/kvm/sys_regs.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/kvm/sys_regs.c +5 -1
@@ -1539,6 +1539,10 @@ static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu, val &= ~ARM64_FEATURE_MASK (ID_AA64PFR1_EL1_SME );break ; +case SYS_ID_AA64PFR2_EL1 : +/* We only expose FPMR */ + val &=ID_AA64PFR2_EL1_FPMR ; +break ;case SYS_ID_AA64ISAR1_EL1 :if (!vcpu_has_ptrauth (vcpu)) val &= ~(ARM64_FEATURE_MASK(ID_AA64ISAR1_EL1_APA) |
@@ -2381,7 +2385,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { ID_AA64PFR0_EL1_AdvSIMD | ID_AA64PFR0_EL1_FP), },ID_SANITISED (ID_AA64PFR1_EL1 ), -ID_UNALLOCATED (4 ,2 ), +ID_WRITABLE (ID_AA64PFR2_EL1 ,ID_AA64PFR2_EL1_FPMR ),ID_UNALLOCATED (4 ,3 ),ID_WRITABLE (ID_AA64ZFR0_EL1 , ~ID_AA64ZFR0_EL1_RES0 ), ID_HIDDEN(ID_AA64SMFR0_EL1),
Signature · 1 line
2.39.2
Quoted 4 lines · Marc Zyngier
Although FP8 support was merged in 6.9, the KVM side was dropped, with no sign of it being picked up again. Given that its absence is getting in the way of NV upstreaming (
HCRX_EL2needs fleshing out), here's a small series addressing it.
The code looks good to me and I didn't spot any problems in testing so:
Quoted 5 lines · Marc Zyngier
Although FP8 support was merged in 6.9, the KVM side was dropped, with no sign of it being picked up again. Given that its absence is getting in the way of NV upstreaming (
HCRX_EL2needs fleshing out), here's a small series addressing it.
Quoted 7 lines
The support is following the save/restore model established for the rest of the FP code, with FPMR being tied to it. The sole additions are the handling of traps in a nested context, and the corresponding ID registers being made writable. As an extra cleanup, SVCR and FPMR are moved into the sysreg array.
[...]
Applied to next, thanks!
[1/8] KVM: arm64: Move SVCR into the sysreg array
commit: b55688943597df06f202c67341da5b9b0ec54e93
[2/8] KVM: arm64: Add predicate for FPMR support in a VM
commit: d4db98791aa5316677a1da9bfa0788068c9863dc
[3/8] KVM: arm64: Move FPMR into the sysreg array
commit: 7d9c1ed6f4bfa8d5fcafad847ac64e2839a04301
[4/8] KVM: arm64: Add save/restore support for FPMR
commit: ef3be86021c3bdf384c36d9d4aa1ee9fe65b95af
[5/8] KVM: arm64: Honor trap routing for FPMR
commit: b8f669b491ec4693d07126b20db0fbe747556d11
[6/8] KVM: arm64: Expose ID_AA64FPFR0_EL1 as a writable ID reg
commit: 6d7307651a8a021e7286e90264676b893cb6032d
[7/8] KVM: arm64: Enable FP8 support when available and configured
commit: c9150a8ad9cdb69584d4ec5af61481df41498eb8
[8/8] KVM: arm64: Expose ID_AA64PFR2_EL1 to userspace and guests
commit: 13c7a51eeb747ec315485ac7b13d4ea03707f53eCheers,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.