KVM: arm64: Fix handling of host fpsimd/sve state in protected mode
v4 cover KVM: arm64: Fix handling of host fpsimd/sve state in protected mode
Changes since v3 [1]:
- Rebased on Linux 6.10-rc2 (c3f38fa61af7)
- Dropped v3 patches 8--11 (Oliver)
- Removed unnecessary isb()s (Oliver)
- Formatting/comments (Mark)
- Fix __sve_save_state()/__sve_restore_state() prototypes (Mark)
- Save/restore ffr with the sve state
- Added a patch that checks at hyp that SME features aren't enabled on guest entry, to ensure it's not in streaming mode
With the KVM host data rework [2], handling of fpsimd and sve state in protected mode is done at hyp. For protected VMs, we don't want to leak any guest state to the host, including whether a guest has used fpsimd/sve.
To complete the work started with the host data rework, in regards to protected mode, ensure that the host's fpsimd context and its sve context are restored on guest exit, since the rework has hidden the fpsimd/sve state from the host.
This patch series eagerly restores the host fpsimd/sve state on guest exit when running in protected mode, which happens only if the guest has used fpsimd/sve. This means that the saving of the state is lazy, similar to the behavior of KVM in other modes, but the restoration of the host state is eager.
This series is based on Linux 6.10-rc2 (c3f38fa61af7).
Tested on qemu, with the kernel sve stress tests.
Cheers,
/fuad
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
Fuad Tabba (9):
KVM: arm64: Reintroduce __sve_save_state
KVM: arm64: Fix prototype for __sve_save_state/__sve_restore_state
KVM: arm64: Abstract set/clear of CPTR_EL2 bits behind helper
KVM: arm64: Specialize handling of host fpsimd state on trap
KVM: arm64: Allocate memory mapped at hyp for host sve state in pKVM
KVM: arm64: Eagerly restore host fpsimd/sve state in pKVM
KVM: arm64: Consolidate initializing the host data's fpsimd_state/sve
in pKVM
KVM: arm64: Refactor CPACR trap bit setting/clearing to use ELx format
KVM: arm64: Ensure that SME controls are disabled in protected mode
arch/arm64/include/asm/el2_setup.h | 6 +- arch/arm64/include/asm/kvm_arm.h | 6 ++ arch/arm64/include/asm/kvm_emulate.h | 71 +++++++++++++++++++-- arch/arm64/include/asm/kvm_host.h | 25 +++++++- arch/arm64/include/asm/kvm_hyp.h | 4 +- arch/arm64/include/asm/kvm_pkvm.h | 9 +++ arch/arm64/kvm/arm.c | 76 ++++++++++++++++++++++ arch/arm64/kvm/fpsimd.c | 11 +++- arch/arm64/kvm/hyp/fpsimd.S | 6 ++ arch/arm64/kvm/hyp/include/hyp/switch.h | 36 ++++++----- arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 1 - arch/arm64/kvm/hyp/nvhe/hyp-main.c | 84 ++++++++++++++++++++++--- arch/arm64/kvm/hyp/nvhe/pkvm.c | 17 ++--- arch/arm64/kvm/hyp/nvhe/setup.c | 25 +++++++- arch/arm64/kvm/hyp/nvhe/switch.c | 24 ++++++- arch/arm64/kvm/hyp/vhe/switch.c | 12 ++-- arch/arm64/kvm/reset.c | 3 + 17 files changed, 358 insertions(+), 58 deletions(-)
base-commit: c3f38fa61af77b49866b006939479069cd451173
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
PATCH v4 1/9 KVM: arm64: Reintroduce __sve_save_state
Now that the hypervisor is handling the host sve state in protected mode, it needs to be able to save it.
This reverts commit e66425fc9ba3 ("KVM: arm64: Remove unused __sve_save_state").
--- arch/arm64/include/asm/kvm_hyp.h | 1 + arch/arm64/kvm/hyp/fpsimd.S | 6 ++++++ 2 files changed, 7 insertions(+)
arch/arm64/include/asm/kvm_hyp.h +1 -0
@@ -111,6 +111,7 @@ void __debug_restore_host_buffers_nvhe(struct kvm_vcpu *vcpu);void __fpsimd_save_state (struct user_fpsimd_state *fp_regs);void __fpsimd_restore_state (struct user_fpsimd_state *fp_regs); +void __sve_save_state (void *sve_pffr,u32 *fpsr);void __sve_restore_state (void *sve_pffr,u32 *fpsr);u64 __guest_enter (struct kvm_vcpu *vcpu);
arch/arm64/kvm/hyp/fpsimd.S +6 -0
@@ -25,3 +25,9 @@ SYM_FUNC_START(__sve_restore_state) sve_load 0, x1, x2, 3 ret SYM_FUNC_END(__sve_restore_state) + +SYM_FUNC_START(__sve_save_state) + mov x2, #1 + sve_save 0, x1, x2, 3 + ret +SYM_FUNC_END(__sve_save_state)
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
Re: [PATCH v4 1/9] KVM: arm64: Reintroduce __sve_save_state
arch/arm64/kvm/hyp/fpsimd.S SYM_FUNC_START(__sve_restore_state) · line 33
Quoted 7 lines · Fuad Tabba
arch/arm64/kvm/hyp/fpsimd.S +6
-0+void __sve_save_state(void *sve_pffr, u32 *fpsr); +SYM_FUNC_START(__sve_save_state) + mov x2, #1 + sve_save 0, x1, x2, 3 + ret +SYM_FUNC_END(__sve_save_state)
This is still trying to use x2 as an argument here?
Re: [PATCH v4 1/9] KVM: arm64: Reintroduce __sve_save_state
Hi Mark,
Quoted 5 lines · Mark Brown
Quoted 1 line · Fuad Tabba
+void
__sve_save_state(void *sve_pffr, u32 *fpsr);
Quoted 7 lines
arch/arm64/kvm/hyp/fpsimd.S SYM_FUNC_START(__sve_restore_state) · line 33
Quoted 5 lines
arch/arm64/kvm/hyp/fpsimd.S +5
-0+SYM_FUNC_START(__sve_save_state) + mov x2, #1 + sve_save 0, x1, x2, 3 + ret +SYM_FUNC_END(__sve_save_state)This is still trying to use x2 as an argument here?
Since __sve_restore_state also has the same problem, I fixed both in the following patch:
KVM: arm64: Fix prototype for __sve_save_state/__sve_restore_state
https://lore.kernel.org/all/[email protected]/
Cheers,
/fuad
Re: [PATCH v4 1/9] KVM: arm64: Reintroduce __sve_save_state
Quoted 10 lines · Fuad Tabba
Quoted 3 lines · Mark Brown
On Mon, Jun 03, 2024 at 01:28:43PM +0100, Fuad Tabba wrote:
This is still trying to use x2 as an argument here?
Since
__sve_restore_statealso has the same problem, I fixed both in the following patch:KVM: arm64: Fix prototype for __sve_save_state/__sve_restore_state
https://lore.kernel.org/all/[email protected]/
Ah, I see - it'd have been good to note that in the changelog.
PATCH v4 2/9 KVM: arm64: Fix prototype for __sve_save_state/__sve_restore_state
In reply to Fuad TabbaSince the prototypes for __sve_save_state/__sve_restore_state at hyp were added, the underlying macro has acquired a third parameter for saving/restoring ffr.
Fix the prototypes to account for the third parameter, and restore the ffr for the guest since it is saved.
--- arch/arm64/include/asm/kvm_hyp.h | 4 ++-- arch/arm64/kvm/hyp/include/hyp/switch.h | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-)
arch/arm64/include/asm/kvm_hyp.h +2 -2
@@ -111,8 +111,8 @@ void __debug_restore_host_buffers_nvhe(struct kvm_vcpu *vcpu);void __fpsimd_save_state (struct user_fpsimd_state *fp_regs);void __fpsimd_restore_state (struct user_fpsimd_state *fp_regs); -void __sve_save_state (void *sve_pffr,u32 *fpsr); -void __sve_restore_state (void *sve_pffr,u32 *fpsr); +void __sve_save_state (void *sve_pffr,u32 *fpsr,int save_ffr); +void __sve_restore_state (void *sve_pffr,u32 *fpsr,int restore_ffr);u64 __guest_enter (struct kvm_vcpu *vcpu);
arch/arm64/kvm/hyp/include/hyp/switch.h +2 -1
@@ -316,7 +316,8 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu) {sve_cond_update_zcr_vq (vcpu_sve_max_vq (vcpu) -1 ,SYS_ZCR_EL2 );__sve_restore_state (vcpu_sve_pffr (vcpu), - &vcpu->arch.ctxt.fp_regs.fpsr); + &vcpu->arch.ctxt.fp_regs.fpsr, + true);write_sysreg_el1 (__vcpu_sys_reg (vcpu,ZCR_EL1 ),SYS_ZCR ); }
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
Re: [PATCH v4 2/9] KVM: arm64: Fix prototype for __sve_save_state/__sve_restore_state
Quoted 6 lines · Fuad Tabba
Since the prototypes for __sve_save_state/__sve_restore_state at hyp were added, the underlying macro has acquired a third parameter for saving/restoring ffr.
Fix the prototypes to account for the third parameter, and restore the ffr for the guest since it is saved.
The change to fix restore should be sent as a bug fix.
PATCH v4 3/9 KVM: arm64: Abstract set/clear of CPTR_EL2 bits behind helper
In reply to Fuad TabbaThe same traps controlled by CPTR_EL2 or CPACR_EL1 need to be toggled in different parts of the code, but the exact bits and their polarity differ between these two formats and the mode (vhe/nvhe/hvhe).
To reduce the amount of duplicated code and the chance of getting the wrong bit/polarity or missing a field, abstract the set/clear of CPTR_EL2 bits behind a helper.
Since (h)VHE is the way of the future, use the CPACR_EL1 format, which is a subset of the VHE CPTR_EL2, as a reference.
No functional change intended.
--- arch/arm64/include/asm/kvm_arm.h | 6 +++ arch/arm64/include/asm/kvm_emulate.h | 62 +++++++++++++++++++++++++ arch/arm64/kvm/hyp/include/hyp/switch.h | 18 ++----- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 6 +-- 4 files changed, 73 insertions(+), 19 deletions(-)
arch/arm64/include/asm/kvm_arm.h +6 -0
@@ -305,6 +305,12 @@ GENMASK(19, 14) | \ BIT(11)) +#define CPTR_VHE_EL2_RES0 (GENMASK(63, 32) | \ + GENMASK(27, 26) | \ + GENMASK(23, 22) | \ + GENMASK(19, 18) | \ + GENMASK(15, 0)) +/* Hyp Debug Configuration Register bits */ #define MDCR_EL2_E2TB_MASK (UL(0x3))#define MDCR_EL2_E2TB_SHIFT (UL(24))
arch/arm64/include/asm/kvm_emulate.h +62 -0
68 lines not on this page · raw message
arch/arm64/kvm/hyp/include/hyp/switch.h +4 -14
@@ -331,7 +331,6 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) {bool sve_guest;u8 esr_ec; -u64 reg;if (!system_supports_fpsimd ())return false;
@@ -354,19 +353,10 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)/* Valid trap. Switch the context: */ /* First disable enough traps to allow us to update the registers */ -if (has_vhe () ||has_hvhe ()) { - reg =CPACR_EL1_FPEN_EL0EN |CPACR_EL1_FPEN_EL1EN ; -if (sve_guest) - reg |=CPACR_EL1_ZEN_EL0EN |CPACR_EL1_ZEN_EL1EN ; - -sysreg_clear_set (cpacr_el1,0 , reg); - }else { - reg =CPTR_EL2_TFP ; -if (sve_guest) - reg |=CPTR_EL2_TZ ; - -sysreg_clear_set (cptr_el2, reg,0 ); - } +if (sve_guest) +cpacr_clear_set (0 , CPACR_ELx_FPEN | CPACR_ELx_ZEN); +else +cpacr_clear_set (0 , CPACR_ELx_FPEN);isb ();/* Write out the host state if it's in the registers */
arch/arm64/kvm/hyp/nvhe/hyp-main.c +1 -5
@@ -405,11 +405,7 @@ void handle_trap(struct kvm_cpu_context *host_ctxt)handle_host_smc (host_ctxt);break ;case ESR_ELx_EC_SVE: -if (has_hvhe ()) -sysreg_clear_set (cpacr_el1,0 , (CPACR_EL1_ZEN_EL1EN | -CPACR_EL1_ZEN_EL0EN )); -else -sysreg_clear_set (cptr_el2,CPTR_EL2_TZ ,0 ); +cpacr_clear_set (0 , CPACR_ELx_ZEN);isb ();sve_cond_update_zcr_vq (ZCR_ELx_LEN_MASK,SYS_ZCR_EL2 );break ;
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
PATCH v4 4/9 KVM: arm64: Specialize handling of host fpsimd state on trap
In reply to Fuad TabbaIn subsequent patches, n/vhe will diverge on saving the host fpsimd/sve state when taking a guest fpsimd/sve trap. Add a specialized helper to handle it.
No functional change intended.
--- arch/arm64/kvm/hyp/include/hyp/switch.h | 4 +++- arch/arm64/kvm/hyp/nvhe/switch.c | 5 +++++ arch/arm64/kvm/hyp/vhe/switch.c | 5 +++++ 3 files changed, 13 insertions(+), 1 deletion(-)
arch/arm64/kvm/hyp/include/hyp/switch.h +3 -1
@@ -321,6 +321,8 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu)write_sysreg_el1 (__vcpu_sys_reg (vcpu,ZCR_EL1 ),SYS_ZCR ); } +static void kvm_hyp_save_fpsimd_host (struct kvm_vcpu *vcpu); +/* * We trap the first access to the FP/SIMD to save the host context and * restore the guest context lazily.
@@ -361,7 +363,7 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)/* Write out the host state if it's in the registers */ if (host_owns_fp_regs ()) -__fpsimd_save_state (*host_data_ptr (fpsimd_state)); +kvm_hyp_save_fpsimd_host (vcpu);/* Restore the guest state */ if (sve_guest)
arch/arm64/kvm/hyp/nvhe/switch.c +5 -0
@@ -182,6 +182,11 @@ static bool kvm_handle_pvm_sys64(struct kvm_vcpu *vcpu, u64 *exit_code) kvm_handle_pvm_sysreg(vcpu, exit_code)); } +static void kvm_hyp_save_fpsimd_host (struct kvm_vcpu *vcpu) +{ +__fpsimd_save_state (*host_data_ptr (fpsimd_state)); +} +static const exit_handler_fn hyp_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] =NULL , [ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32,
arch/arm64/kvm/hyp/vhe/switch.c +5 -0
@@ -262,6 +262,11 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code)return true; } +static void kvm_hyp_save_fpsimd_host (struct kvm_vcpu *vcpu) +{ +__fpsimd_save_state (*host_data_ptr (fpsimd_state)); +} +static const exit_handler_fn hyp_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] =NULL , [ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32,
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
PATCH v4 5/9 KVM: arm64: Allocate memory mapped at hyp for host sve state in pKVM
In reply to Fuad TabbaProtected mode needs to maintain (save/restore) the host's sve state, rather than relying on the host kernel to do that. This is to avoid leaking information to the host about guests and the type of operations they are performing.
As a first step towards that, allocate memory mapped at hyp, per cpu, for the host sve state. The following patch will use this memory to save/restore the host state.
--- arch/arm64/include/asm/kvm_host.h | 17 ++++++++ arch/arm64/include/asm/kvm_hyp.h | 1 + arch/arm64/include/asm/kvm_pkvm.h | 9 ++++ arch/arm64/kvm/arm.c | 68 +++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/pkvm.c | 2 + arch/arm64/kvm/hyp/nvhe/setup.c | 24 +++++++++++ arch/arm64/kvm/reset.c | 3 ++ 7 files changed, 124 insertions(+)
arch/arm64/include/asm/kvm_host.h +17 -0
@@ -76,6 +76,7 @@ static inline enum kvm_mode kvm_get_mode(void) { return KVM_MODE_NONE; };DECLARE_STATIC_KEY_FALSE (userspace_irqchip_in_use);extern unsigned int __ro_after_init kvm_sve_max_vl; +extern unsigned int __ro_after_init kvm_host_sve_max_vl;int __init kvm_arm_init_sve (void ); u32 __attribute_const__ kvm_target_cpu(void);
@@ -521,6 +522,20 @@ struct kvm_cpu_context {u64 *vncr_array; }; +struct cpu_sve_state { +__u64 zcr_el1; + +/* +* Ordering is important since __sve_save_state/__sve_restore_state +* relies on it. +*/ +__u32 fpsr; +__u32 fpcr; + +/* Must be SVE_VQ_BYTES (128 bit) aligned. */ +__u8 sve_regs[]; +}; +/* * This structure is instantiated on a per-CPU basis, and contains * data that is:
@@ -534,7 +549,9 @@ struct kvm_cpu_context {*/ struct kvm_host_data {struct kvm_cpu_context host_ctxt; +struct user_fpsimd_state *fpsimd_state;/* hyp VA */ +struct cpu_sve_state *sve_state;/* hyp VA */ /* Ownership of the FP regs */ enum {
arch/arm64/include/asm/kvm_hyp.h +1 -0
@@ -143,5 +143,6 @@ extern u64 kvm_nvhe_sym(id_aa64smfr0_el1_sys_val);extern unsigned long kvm_nvhe_sym (__icache_flags );extern unsigned int kvm_nvhe_sym (kvm_arm_vmid_bits ); +extern unsigned int kvm_nvhe_sym (kvm_host_sve_max_vl );#endif /* __ARM64_KVM_HYP_H__ */
arch/arm64/include/asm/kvm_pkvm.h +9 -0
@@ -128,4 +128,13 @@ static inline unsigned long hyp_ffa_proxy_pages(void) return (2 * KVM_FFA_MBOX_NR_PAGES) + DIV_ROUND_UP(desc_max, PAGE_SIZE); } +static inline size_t pkvm_host_sve_state_size (void ) +{ +if (!system_supports_sve ()) +return 0 ; + +return size_add (sizeof (struct cpu_sve_state ), +SVE_SIG_REGS_SIZE (sve_vq_from_vl (kvm_host_sve_max_vl))); +} +#endif /* __ARM64_KVM_PKVM_H__ */
arch/arm64/kvm/arm.c +68 -0
104 lines not on this page · raw message
arch/arm64/kvm/hyp/nvhe/pkvm.c +2 -0
@@ -18,6 +18,8 @@ unsigned long __icache_flags;/* Used by kvm_get_vttbr(). */ unsigned int kvm_arm_vmid_bits; +unsigned int kvm_host_sve_max_vl; +/* * Set trap register values based on features in ID_AA64PFR0. */
arch/arm64/kvm/hyp/nvhe/setup.c +24 -0
@@ -67,6 +67,28 @@ static int divide_memory_pool(void *virt, unsigned long size)return 0 ; } +static int pkvm_create_host_sve_mappings (void ) +{ +void *start, *end; +int ret, i; + +if (!system_supports_sve ()) +return 0 ; + +for (i =0 ; i < hyp_nr_cpus; i++) { +struct kvm_host_data *host_data =per_cpu_ptr (&kvm_host_data, i); +struct cpu_sve_state *sve_state = host_data->sve_state; + + start =kern_hyp_va (sve_state); + end = start +PAGE_ALIGN (pkvm_host_sve_state_size ()); + ret =pkvm_create_mappings (start, end,PAGE_HYP ); +if (ret) +return ret; + } + +return 0 ; +} +static int recreate_hyp_mappings (phys_addr_t phys,unsigned long size,unsigned long *per_cpu_base,u32 hyp_va_bits)
@@ -125,6 +147,8 @@ static int recreate_hyp_mappings(phys_addr_t phys, unsigned long size,return ret; } +pkvm_create_host_sve_mappings (); +/* * Map the host sections RO in the hypervisor, but transfer the * ownership from the host to the hypervisor itself to make sure they
arch/arm64/kvm/reset.c +3 -0
@@ -32,6 +32,7 @@/* Maximum phys_shift supported for any VM on this host */ static u32 __ro_after_init kvm_ipa_limit; +unsigned int __ro_after_init kvm_host_sve_max_vl;/* * ARMv8 Reset Values
@@ -51,6 +52,8 @@ int __init kvm_arm_init_sve(void) {if (system_supports_sve ()) { kvm_sve_max_vl =sve_max_virtualisable_vl (); + kvm_host_sve_max_vl =sve_max_vl (); +kvm_nvhe_sym (kvm_host_sve_max_vl) = kvm_host_sve_max_vl;/* * The get_sve_reg()/set_sve_reg() ioctl interface will need
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
Re: [PATCH v4 5/9] KVM: arm64: Allocate memory mapped at hyp for host sve state in pKVM
arch/arm64/kvm/reset.c int __init kvm_arm_init_sve(void) · line 56
Quoted 5 lines · Fuad Tabba
{
arch/arm64/kvm/reset.c +2
-0if (system_supports_sve ()) { kvm_sve_max_vl =sve_max_virtualisable_vl (); + kvm_host_sve_max_vl =sve_max_vl (); +kvm_nvhe_sym (kvm_host_sve_max_vl) = kvm_host_sve_max_vl;
As discussed on the prior version this either needs to be the maximum available physical VL on any individual PE or the hypervisor needs to use this VL explicitly when saving/restoring host state.
Re: [PATCH v4 5/9] KVM: arm64: Allocate memory mapped at hyp for host sve state in pKVM
Hi Mark,
Quoted 9 lines · Mark Brown
arch/arm64/kvm/reset.c int __init kvm_arm_init_sve(void) · line 56
Quoted 5 lines · Fuad Tabba
{ if (system_supports_sve()) { kvm_sve_max_vl = sve_max_virtualisable_vl();arch/arm64/kvm/reset.c +2
-0+ kvm_host_sve_max_vl =sve_max_vl (); +kvm_nvhe_sym (kvm_host_sve_max_vl) = kvm_host_sve_max_vl;
Quoted 3 lines
As discussed on the prior version this either needs to be the maximum available physical VL on any individual PE or the hypervisor needs to use this VL explicitly when saving/restoring host state.
Sorry it took me a while, but I understand what the issue is now. I'll send a patch to fix this.
Thanks for being patient with me,
/fuad
PATCH v4 6/9 KVM: arm64: Eagerly restore host fpsimd/sve state in pKVM
In reply to Fuad TabbaWhen running in protected mode we don't want to leak protected guest state to the host, including whether a guest has used fpsimd/sve. Therefore, eagerly restore the host state on guest exit when running in protected mode, which happens only if the guest has used fpsimd/sve.
--- arch/arm64/kvm/hyp/include/hyp/switch.h | 13 ++++- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 67 +++++++++++++++++++++++-- arch/arm64/kvm/hyp/nvhe/pkvm.c | 2 + arch/arm64/kvm/hyp/nvhe/switch.c | 16 +++++- 4 files changed, 93 insertions(+), 5 deletions(-)
arch/arm64/kvm/hyp/include/hyp/switch.h +12 -1
@@ -321,6 +321,17 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu)write_sysreg_el1 (__vcpu_sys_reg (vcpu,ZCR_EL1 ),SYS_ZCR ); } +static inline void __hyp_sve_save_host (void ) +{ +struct cpu_sve_state *sve_state = *host_data_ptr (sve_state); + + sve_state->zcr_el1 =read_sysreg_el1 (SYS_ZCR ); +write_sysreg_s (ZCR_ELx_LEN_MASK,SYS_ZCR_EL2 ); +__sve_save_state (sve_state->sve_regs +sve_ffr_offset (kvm_host_sve_max_vl), + &sve_state->fpsr, + true); +} +static void kvm_hyp_save_fpsimd_host (struct kvm_vcpu *vcpu);/*
@@ -355,7 +366,7 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)/* Valid trap. Switch the context: */ /* First disable enough traps to allow us to update the registers */ -if (sve_guest) +if (sve_guest || (is_protected_kvm_enabled () &&system_supports_sve ()))cpacr_clear_set (0 , CPACR_ELx_FPEN | CPACR_ELx_ZEN);else cpacr_clear_set (0 , CPACR_ELx_FPEN);
arch/arm64/kvm/hyp/nvhe/hyp-main.c +64 -3
94 lines not on this page · raw message
arch/arm64/kvm/hyp/nvhe/pkvm.c +2 -0
@@ -588,6 +588,8 @@ int __pkvm_init_vcpu(pkvm_handle_t handle, struct kvm_vcpu *host_vcpu,if (ret)unmap_donated_memory (hyp_vcpu,sizeof (*hyp_vcpu)); + hyp_vcpu->vcpu.arch.cptr_el2 =kvm_get_reset_cptr_el2 (&hyp_vcpu->vcpu); +return ret; }
arch/arm64/kvm/hyp/nvhe/switch.c +15 -1
@@ -184,7 +184,21 @@ static bool kvm_handle_pvm_sys64(struct kvm_vcpu *vcpu, u64 *exit_code)static void kvm_hyp_save_fpsimd_host (struct kvm_vcpu *vcpu) { -__fpsimd_save_state (*host_data_ptr (fpsimd_state)); +/* +* Non-protected kvm relies on the host restoring its sve state. +* Protected kvm restores the host's sve state as not to reveal that +* fpsimd was used by a guest nor leak upper sve bits. +*/ +if (unlikely (is_protected_kvm_enabled () &&system_supports_sve ())) { +__hyp_sve_save_host (); + +/* Re-enable SVE traps if not supported for the guest vcpu. */ +if (!vcpu_has_sve (vcpu)) +cpacr_clear_set (CPACR_ELx_ZEN,0 ); + + }else { +__fpsimd_save_state (*host_data_ptr (fpsimd_state)); + } }static const exit_handler_fn hyp_exit_handlers[] = {
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
Re: [PATCH v4 6/9] KVM: arm64: Eagerly restore host fpsimd/sve state in pKVM
arch/arm64/kvm/hyp/nvhe/hyp-main.c DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); · line 61
Quoted 4 lines · Fuad Tabba
arch/arm64/kvm/hyp/nvhe/hyp-main.c +4
-0+static void fpsimd_sve_flush (void ) +{ + *host_data_ptr (fp_owner) =FP_STATE_HOST_OWNED ; +}
My previous comments about this being confusing still stand.
Re: [PATCH v4 6/9] KVM: arm64: Eagerly restore host fpsimd/sve state in pKVM
Hi Mark,
Quoted 9 lines · Mark Brown
arch/arm64/kvm/hyp/nvhe/hyp-main.c DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); · line 61
Quoted 4 lines · Fuad Tabba
arch/arm64/kvm/hyp/nvhe/hyp-main.c +4
-0+static void fpsimd_sve_flush (void ) +{ + *host_data_ptr (fp_owner) =FP_STATE_HOST_OWNED ; +}My previous comments about this being confusing still stand.
Sorry, I missed this in my reply to v3.
This follows the convention for save/flush in hyp-main.c. Since the act of flushing the fpsimd/sve state is lazy, i.e., only takes place if the guest were to use fpsimd/sve, then the only thing that we need to do to flush is to mark the state as owned by the host.
You suggested inlining this, but since this is static, I think the compiler would do that. Even though it's only one line, it maintains symmetry with fpsimd_sve_sync().
Cheers,
/fuad
Re: [PATCH v4 6/9] KVM: arm64: Eagerly restore host fpsimd/sve state in pKVM
Quoted 10 lines · Fuad Tabba
Quoted 8 lines · Mark Brown
arch/arm64/kvm/hyp/nvhe/hyp-main.c DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); · line 61
Quoted 4 lines · Fuad Tabba
arch/arm64/kvm/hyp/nvhe/hyp-main.c +4
-0+static void fpsimd_sve_flush (void ) +{ + *host_data_ptr (fp_owner) =FP_STATE_HOST_OWNED ; +}My previous comments about this being confusing still stand.
Quoted 6 lines
Sorry, I missed this in my reply to v3.
This follows the convention for save/flush in
hyp-main.c. Since the act of flushing the fpsimd/sve state is lazy, i.e., only takes place if the guest were to use fpsimd/sve, then the only thing that we need to do to flush is to mark the state as owned by the host.
I think this needs a comment mentioning what's going on here.
Quoted 3 lines
You suggested inlining this, but since this is static, I think the compiler would do that. Even though it's only one line, it maintains symmetry with
fpsimd_sve_sync().
The reason I was suggesting inlining was that it removes the need to name the function.
Marc Zyngier
maintainerRe: [PATCH v4 6/9] KVM: arm64: Eagerly restore host fpsimd/sve state in pKVM
Quoted 20 lines · Mark Brown
[1 <text/plain; utf-8 (quoted-printable)>]
Quoted 10 lines · Fuad Tabba
Quoted 8 lines · Mark Brown
arch/arm64/kvm/hyp/nvhe/hyp-main.c DEFINE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); · line 61
Quoted 4 lines · Fuad Tabba
arch/arm64/kvm/hyp/nvhe/hyp-main.c +4
-0+static void fpsimd_sve_flush (void ) +{ + *host_data_ptr (fp_owner) =FP_STATE_HOST_OWNED ; +}My previous comments about this being confusing still stand.
Quoted 6 lines
Sorry, I missed this in my reply to v3.
This follows the convention for save/flush in
hyp-main.c. Since the act of flushing the fpsimd/sve state is lazy, i.e., only takes place if the guest were to use fpsimd/sve, then the only thing that we need to do to flush is to mark the state as owned by the host.
Quoted 8 lines
I think this needs a comment mentioning what's going on here.
Quoted 3 lines
You suggested inlining this, but since this is static, I think the compiler would do that. Even though it's only one line, it maintains symmetry with
fpsimd_sve_sync().The reason I was suggesting inlining was that it removes the need to name the function.
You're missing the point.
The name is important, and the current name is correct. We use *flush (resp. *sync) for operations that need to happen before (resp. after) the entry into the guest. This is consistent with the rest of the code base for most of the subsystems KVM/arm64 deals with.
So the current form of this function, as a standalone function with its current name, stays.
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH v4 6/9] KVM: arm64: Eagerly restore host fpsimd/sve state in pKVM
Quoted 11 lines · Marc Zyngier
Quoted 8 lines · Mark Brown
Quoted 3 lines · Fuad Tabba
You suggested inlining this, but since this is static, I think the compiler would do that. Even though it's only one line, it maintains symmetry with
fpsimd_sve_sync().The reason I was suggesting inlining was that it removes the need to name the function.
You're missing the point.
I think not.
Quoted 4 lines
The name is important, and the current name is correct. We use *flush (resp. *sync) for operations that need to happen before (resp. after) the entry into the guest. This is consistent with the rest of the code base for most of the subsystems KVM/arm64 deals with.
Hence my suggestion to add documentation to the function to avoid the risk of surprises, the above text explains the motivation for the suggestion that Faud is responding to. Either approach would work.
PATCH v4 7/9 KVM: arm64: Consolidate initializing the host data's fpsimd_state/sve in pKVM
In reply to Fuad TabbaNow that we have introduced finalize_init_hyp_mode(), lets consolidate the initializing of the host_data fpsimd_state and sve state.
--- arch/arm64/include/asm/kvm_host.h | 10 ++++++++-- arch/arm64/kvm/arm.c | 20 ++++++++++++++------ arch/arm64/kvm/hyp/include/nvhe/pkvm.h | 1 - arch/arm64/kvm/hyp/nvhe/pkvm.c | 11 ----------- arch/arm64/kvm/hyp/nvhe/setup.c | 1 - 5 files changed, 22 insertions(+), 21 deletions(-)
arch/arm64/include/asm/kvm_host.h +8 -2
@@ -550,8 +550,14 @@ struct cpu_sve_state {struct kvm_host_data {struct kvm_cpu_context host_ctxt; -struct user_fpsimd_state *fpsimd_state;/* hyp VA */ -struct cpu_sve_state *sve_state;/* hyp VA */ +/* +* All pointers in this union are hyp VA. +* sve_state is only used in pKVM and if system_supports_sve(). +*/ +union { +struct user_fpsimd_state *fpsimd_state; +struct cpu_sve_state *sve_state; + };/* Ownership of the FP regs */ enum {
arch/arm64/kvm/arm.c +14 -6
@@ -2444,14 +2444,22 @@ static void finalize_init_hyp_mode(void) {int cpu; -if (!is_protected_kvm_enabled () || !system_supports_sve ()) -return ; +if (system_supports_sve () &&is_protected_kvm_enabled ()) { +for_each_possible_cpu (cpu) { +struct cpu_sve_state *sve_state; -for_each_possible_cpu (cpu) { -struct cpu_sve_state *sve_state; + sve_state =per_cpu_ptr_nvhe_sym (kvm_host_data, cpu)->sve_state; +per_cpu_ptr_nvhe_sym (kvm_host_data, cpu)->sve_state = +kern_hyp_va (sve_state); + } + }else { +for_each_possible_cpu (cpu) { +struct user_fpsimd_state *fpsimd_state; - sve_state =per_cpu_ptr_nvhe_sym (kvm_host_data, cpu)->sve_state; -per_cpu_ptr_nvhe_sym (kvm_host_data, cpu)->sve_state =kern_hyp_va (sve_state); + fpsimd_state = &per_cpu_ptr_nvhe_sym (kvm_host_data, cpu)->host_ctxt.fp_regs; +per_cpu_ptr_nvhe_sym (kvm_host_data, cpu)->fpsimd_state = +kern_hyp_va (fpsimd_state); + } } }
arch/arm64/kvm/hyp/include/nvhe/pkvm.h +0 -1
@@ -59,7 +59,6 @@ static inline bool pkvm_hyp_vcpu_is_protected(struct pkvm_hyp_vcpu *hyp_vcpu) }void pkvm_hyp_vm_table_init (void *tbl); -void pkvm_host_fpsimd_state_init (void );int __pkvm_init_vm (struct kvm *host_kvm,unsigned long vm_hva,unsigned long pgd_hva);
arch/arm64/kvm/hyp/nvhe/pkvm.c +0 -11
@@ -249,17 +249,6 @@ void pkvm_hyp_vm_table_init(void *tbl) vm_table = tbl; } -void pkvm_host_fpsimd_state_init (void ) -{ -unsigned long i; - -for (i =0 ; i < hyp_nr_cpus; i++) { -struct kvm_host_data *host_data =per_cpu_ptr (&kvm_host_data, i); - - host_data->fpsimd_state = &host_data->host_ctxt.fp_regs; - } -} -/* * Return the hyp vm structure corresponding to the handle. */
arch/arm64/kvm/hyp/nvhe/setup.c +0 -1
@@ -324,7 +324,6 @@ void __noreturn __pkvm_init_finalise(void) goto out;pkvm_hyp_vm_table_init (vm_table_base); -pkvm_host_fpsimd_state_init (); out:/* * We tail-called to here from handle___pkvm_init() and will not return,
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
Re: [PATCH v4 7/9] KVM: arm64: Consolidate initializing the host data's fpsimd_state/sve in pKVM
Quoted 3 lines · Fuad Tabba
Now that we have introduced
finalize_init_hyp_mode(), lets consolidate the initializing of thehost_datafpsimd_stateand sve state.
PATCH v4 8/9 KVM: arm64: Refactor CPACR trap bit setting/clearing to use ELx format
In reply to Fuad TabbaWhen setting/clearing CPACR bits for EL0 and EL1, use the ELx format of the bits, which covers both. This makes the code clearer, and reduces the chances of accidentally missing a bit.
No functional change intended.
--- arch/arm64/include/asm/el2_setup.h | 6 +++--- arch/arm64/include/asm/kvm_emulate.h | 9 ++++----- arch/arm64/kvm/fpsimd.c | 4 +--- arch/arm64/kvm/hyp/nvhe/pkvm.c | 2 +- arch/arm64/kvm/hyp/nvhe/switch.c | 5 ++--- arch/arm64/kvm/hyp/vhe/switch.c | 7 +++---- 6 files changed, 14 insertions(+), 19 deletions(-)
arch/arm64/include/asm/el2_setup.h +3 -3
@@ -146,7 +146,7 @@/* Coprocessor traps */ .macro __init_el2_cptr __check_hvhe .LnVHE_\@, x1 - mov x0, #(CPACR_EL1_FPEN_EL1EN | CPACR_EL1_FPEN_EL0EN) + mov x0, #CPACR_ELx_FPENmsr cpacr_el1, x0 b .Lskip_set_cptr_\@ .LnVHE_\@:
@@ -277,7 +277,7 @@// (h)VHE case mrs x0, cpacr_el1// Disable SVE traps - orr x0, x0, #(CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN) + orr x0, x0, #CPACR_ELx_ZEN msr cpacr_el1, x0 b .Lskip_set_cptr_\@
@@ -298,7 +298,7 @@// (h)VHE case mrs x0, cpacr_el1// Disable SME traps - orr x0, x0, #(CPACR_EL1_SMEN_EL0EN | CPACR_EL1_SMEN_EL1EN) + orr x0, x0, #CPACR_ELx_SMEN msr cpacr_el1, x0 b .Lskip_set_cptr_sme_\@
arch/arm64/include/asm/kvm_emulate.h +4 -5
@@ -632,17 +632,16 @@ static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu)u64 val;if (has_vhe ()) { - val = (CPACR_EL1_FPEN_EL0EN |CPACR_EL1_FPEN_EL1EN | -CPACR_EL1_ZEN_EL1EN ); + val = (CPACR_ELx_FPEN |CPACR_EL1_ZEN_EL1EN );if (cpus_have_final_cap (ARM64_SME )) val |=CPACR_EL1_SMEN_EL1EN ; }else if (has_hvhe ()) { - val = (CPACR_EL1_FPEN_EL0EN |CPACR_EL1_FPEN_EL1EN ); + val = CPACR_ELx_FPEN;if (!vcpu_has_sve (vcpu) || !guest_owns_fp_regs ()) - val |=CPACR_EL1_ZEN_EL1EN |CPACR_EL1_ZEN_EL0EN ; + val |= CPACR_ELx_ZEN;if (cpus_have_final_cap (ARM64_SME )) - val |=CPACR_EL1_SMEN_EL1EN |CPACR_EL1_SMEN_EL0EN ; + val |= CPACR_ELx_SMEN; }else { val =CPTR_NVHE_EL2_RES1 ;
arch/arm64/kvm/fpsimd.c +1 -3
@@ -161,9 +161,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)if (has_vhe () &&system_supports_sme ()) {/* Also restore EL0 state seen on entry */ if (vcpu_get_flag (vcpu,HOST_SME_ENABLED )) -sysreg_clear_set (CPACR_EL1 ,0 , -CPACR_EL1_SMEN_EL0EN | -CPACR_EL1_SMEN_EL1EN ); +sysreg_clear_set (CPACR_EL1 ,0 , CPACR_ELx_SMEN);else sysreg_clear_set(CPACR_EL1, CPACR_EL1_SMEN_EL0EN,
arch/arm64/kvm/hyp/nvhe/pkvm.c +1 -1
@@ -65,7 +65,7 @@ static void pvm_init_traps_aa64pfr0(struct kvm_vcpu *vcpu)/* Trap SVE */ if (!FIELD_GET (ARM64_FEATURE_MASK (ID_AA64PFR0_EL1_SVE ), feature_ids)) {if (has_hvhe ()) - cptr_clear |=CPACR_EL1_ZEN_EL0EN |CPACR_EL1_ZEN_EL1EN ; + cptr_clear |= CPACR_ELx_ZEN;else cptr_set |=CPTR_EL2_TZ ; }
arch/arm64/kvm/hyp/nvhe/switch.c +2 -3
@@ -48,15 +48,14 @@ static void __activate_traps(struct kvm_vcpu *vcpu) val |=has_hvhe () ?CPACR_EL1_TTA :CPTR_EL2_TTA ;if (cpus_have_final_cap (ARM64_SME )) {if (has_hvhe ()) - val &= ~(CPACR_EL1_SMEN_EL1EN |CPACR_EL1_SMEN_EL0EN ); + val &= ~CPACR_ELx_SMEN;else val |=CPTR_EL2_TSM ; }if (!guest_owns_fp_regs ()) {if (has_hvhe ()) - val &= ~(CPACR_EL1_FPEN_EL0EN |CPACR_EL1_FPEN_EL1EN | -CPACR_EL1_ZEN_EL0EN |CPACR_EL1_ZEN_EL1EN ); + val &= ~(CPACR_ELx_FPEN | CPACR_ELx_ZEN);else val |=CPTR_EL2_TFP |CPTR_EL2_TZ ;
arch/arm64/kvm/hyp/vhe/switch.c +3 -4
@@ -93,8 +93,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) val =read_sysreg (cpacr_el1); val |= CPACR_ELx_TTA; - val &= ~(CPACR_EL1_ZEN_EL0EN |CPACR_EL1_ZEN_EL1EN | -CPACR_EL1_SMEN_EL0EN |CPACR_EL1_SMEN_EL1EN ); + val &= ~(CPACR_ELx_ZEN | CPACR_ELx_SMEN);/* * With VHE (HCR.E2H == 1), accesses to CPACR_EL1 are routed to
@@ -109,9 +108,9 @@ static void __activate_traps(struct kvm_vcpu *vcpu)if (guest_owns_fp_regs ()) {if (vcpu_has_sve (vcpu)) - val |=CPACR_EL1_ZEN_EL0EN |CPACR_EL1_ZEN_EL1EN ; + val |= CPACR_ELx_ZEN; }else { - val &= ~(CPACR_EL1_FPEN_EL0EN |CPACR_EL1_FPEN_EL1EN ); + val &= ~CPACR_ELx_FPEN;__activate_traps_fpsimd32 (vcpu); }
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
PATCH v4 9/9 KVM: arm64: Ensure that SME controls are disabled in protected mode
In reply to Fuad TabbaKVM (and pKVM) do not support SME guests. Therefore KVM ensures that the host's SME state is flushed and that SME controls for enabling access to ZA storage and for streaming are disabled.
pKVM needs to protect against a buggy/malicious host. Ensure that it wouldn't run a guest when protected mode is enabled should any of the SME controls be enabled.
--- arch/arm64/kvm/fpsimd.c | 7 +++++++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 +++++++++++ 2 files changed, 18 insertions(+)
arch/arm64/kvm/fpsimd.c +7 -0
@@ -90,6 +90,13 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu)fpsimd_save_and_flush_cpu_state (); } } + +/* +* If normal guests gain SME support, maintain this behavior for pKVM +* guests, which don't support SME. +*/ +WARN_ON (is_protected_kvm_enabled () &&system_supports_sme () && +read_sysreg_s (SYS_SVCR )); }/*
arch/arm64/kvm/hyp/nvhe/hyp-main.c +11 -0
@@ -140,6 +140,17 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt)struct pkvm_hyp_vcpu *hyp_vcpu;struct kvm *host_kvm; +/* +* KVM (and pKVM) doesn't support SME guests for now, and +* ensures that SME features aren't enabled in pstate when +* loading a vcpu. Therefore, if SME features enabled the host +* is misbehaving. +*/ +if (unlikely (system_supports_sme () &&read_sysreg_s (SYS_SVCR ))) { + ret = -EINVAL ; + goto out; + } + host_kvm =kern_hyp_va (host_vcpu->kvm); hyp_vcpu =pkvm_load_hyp_vcpu (host_kvm->arch.pkvm.handle, host_vcpu->vcpu_idx);
Signature · 1 line
2.45.1.288.g0e0cd299f1-goog
Re: [PATCH v4 9/9] KVM: arm64: Ensure that SME controls are disabled in protected mode
arch/arm64/kvm/fpsimd.c void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) · line 100
Quoted 13 lines · Fuad Tabba
+++
b/arch/arm64/kvm/fpsimd.carch/arm64/kvm/fpsimd.c +7
-0@@ -90,6 +90,13 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu)fpsimd_save_and_flush_cpu_state (); } } + +/* +* If normal guests gain SME support, maintain this behavior for pKVM +* guests, which don't support SME. +*/ +WARN_ON (is_protected_kvm_enabled () &&system_supports_sme () && +read_sysreg_s (SYS_SVCR )); }
The comment doesn't line up clearly with the check here, we're checking if the hypervisor supports protected guests not if the current guest is protected. At this point we're dealing with the host state rather than the guest state so the guest being protected or not doesn't matter when we check SVCR. I'm not sure this assert really makes sense, or perhaps it should be hoisted into the above system_support_sme() section?
Quoted 11 lines · Fuad Tabba
Changes since v3 [1]:
- Rebased on Linux 6.10-rc2 (c3f38fa61af7)
- Dropped v3 patches 8--11 (Oliver)
- Removed unnecessary isb()s (Oliver)
- Formatting/comments (Mark)
- Fix __sve_save_state()/__sve_restore_state() prototypes (Mark)
- Save/restore ffr with the sve state
- Added a patch that checks at hyp that SME features aren't enabled on guest entry, to ensure it's not in streaming mode[...]
Applied to fixes, thanks!
[1/9] KVM: arm64: Reintroduce __sve_save_state
commit: 87bb39ed40bdf1596b8820e800226e24eb642677
[2/9] KVM: arm64: Fix prototype for __sve_save_state/__sve_restore_state
commit: 45f4ea9bcfe909b3461059990b1e232e55dde809
[3/9] KVM: arm64: Abstract set/clear of CPTR_EL2 bits behind helper
commit: 6d8fb3cbf7e06431a607c30c1bc4cd53a62c220a
[4/9] KVM: arm64: Specialize handling of host fpsimd state on trap
commit: e511e08a9f496948b13aac50610f2d17335f56c3
[5/9] KVM: arm64: Allocate memory mapped at hyp for host sve state in pKVM
commit: 66d5b53e20a6e00b7ce3b652a3e2db967f7b33d0
[6/9] KVM: arm64: Eagerly restore host fpsimd/sve state in pKVM
commit: b5b9955617bc0b41546f2fa7c3dbcc048b43dc82
[7/9] KVM: arm64: Consolidate initializing the host data's fpsimd_state/sve in pKVM
commit: 1696fc2174dbab12228ea9ec4c213d6aeea348f8
[8/9] KVM: arm64: Refactor CPACR trap bit setting/clearing to use ELx format
commit: a69283ae1db8dd416870d931caa9e2d3d2c1cd8b
[9/9] KVM: arm64: Ensure that SME controls are disabled in protected mode
commit: afb91f5f8ad7af172d993a34fde1947892408f53Cheers,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.