KVM: arm64: Rework FPSIMD/SVE tracking
Marc Zyngier
maintainerv3 cover KVM: arm64: Rework FPSIMD/SVE tracking
This is v3 of this series aiming at simplifying the FP handling.
It recently became apparent that we are mapping each vcpu thread's thread_info structure at EL2 for the sole purpose of checking on the TIF_FOREIGN_FPSTATE flag.
Given that this looks like a slightly over-engineered way of sharing a single bit of information, let's move to a slightly more obvious implementation by maintaining a vcpu-private shadow flag that represents the same state.
In the same vein, it appears that the code that deals with saving the host SVE state when used by the guest can never run, and that's by construction. This is actually a good thing, because it be guaranteed to explode on nVHE. Let's get rid of it.
I also take this opportunity to add what looks like a missing, and nonetheless crucial piece of information to the FPSIMD code regarding the way KVM (ab)uses the TIF_FOREIGN_FPSTATE.
Lightly tested on an A53 box with a bunch of paranoia instances running in both host and guests, and more heavily on a FVP to check the SVE behaviour (using the sve-test selftest running in both host and guest at the same time).
I plan to take this series for a spin in -next immediately.
* From v2 [2]:
- Removed obsolete comments
- Removed __sve_save_state which is now unused
* From v1 [1]:
- New patch getting rid of the host SVE save code
- Reworded the documentation update patch
[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]
Marc Zyngier (6):
KVM: arm64: Reorder vcpu flag definitions
KVM: arm64: Get rid of host SVE tracking/saving
KVM: arm64: Remove unused __sve_save_state
KVM: arm64: Introduce flag shadowing TIF_FOREIGN_FPSTATE
KVM: arm64: Stop mapping current thread_info at EL2 arm64/fpsimd: Document the use of TIF_FOREIGN_FPSTATE by KVM
arch/arm64/include/asm/kvm_host.h | 29 +++++++++--------- arch/arm64/include/asm/kvm_hyp.h | 1 - arch/arm64/kernel/fpsimd.c | 6 +++- arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/fpsimd.c | 40 +++++++++---------------- arch/arm64/kvm/hyp/fpsimd.S | 6 ---- arch/arm64/kvm/hyp/include/hyp/switch.h | 30 +++---------------- arch/arm64/kvm/hyp/nvhe/switch.c | 1 - arch/arm64/kvm/hyp/vhe/switch.c | 1 - 9 files changed, 38 insertions(+), 77 deletions(-)
Signature · 1 line
2.30.2
Marc Zyngier
maintainerPATCH v3 1/6 KVM: arm64: Reorder vcpu flag definitions
The vcpu arch flags are in an interesting, semi random order. As I have made the mistake of reusing a flag once, let's rework this in an order that I find a bit less confusing.
--- arch/arm64/include/asm/kvm_host.h | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-)
arch/arm64/include/asm/kvm_host.h +12 -12
@@ -417,14 +417,12 @@ struct kvm_vcpu_arch {#define KVM_ARM64_VCPU_SVE_FINALIZED (1 << 6)/* SVE config completed */ #define KVM_ARM64_GUEST_HAS_PTRAUTH (1 << 7)/* PTRAUTH exposed to guest */ #define KVM_ARM64_PENDING_EXCEPTION (1 << 8)/* Exception pending */ +/* +* Overlaps with KVM_ARM64_EXCEPT_MASK on purpose so that it can't be +* set together with an exception... +*/ +#define KVM_ARM64_INCREMENT_PC (1 << 9)/* Increment PC */ #define KVM_ARM64_EXCEPT_MASK (7 << 9)/* Target EL/MODE */ -#define KVM_ARM64_DEBUG_STATE_SAVE_SPE (1 << 12)/* Save SPE context if active */ -#define KVM_ARM64_DEBUG_STATE_SAVE_TRBE (1 << 13)/* Save TRBE context if active */ - -#define KVM_GUESTDBG_VALID_MASK (KVM_GUESTDBG_ENABLE | \ - KVM_GUESTDBG_USE_SW_BP | \ - KVM_GUESTDBG_USE_HW | \ - KVM_GUESTDBG_SINGLESTEP)/* * When KVM_ARM64_PENDING_EXCEPTION is set, KVM_ARM64_EXCEPT_MASK can * take the following values:
@@ -442,11 +440,13 @@ struct kvm_vcpu_arch {#define KVM_ARM64_EXCEPT_AA64_EL1 (0 << 11)#define KVM_ARM64_EXCEPT_AA64_EL2 (1 << 11) -/* -* Overlaps with KVM_ARM64_EXCEPT_MASK on purpose so that it can't be -* set together with an exception... -*/ -#define KVM_ARM64_INCREMENT_PC (1 << 9)/* Increment PC */ +#define KVM_ARM64_DEBUG_STATE_SAVE_SPE (1 << 12)/* Save SPE context if active */ +#define KVM_ARM64_DEBUG_STATE_SAVE_TRBE (1 << 13)/* Save TRBE context if active */ + +#define KVM_GUESTDBG_VALID_MASK (KVM_GUESTDBG_ENABLE | \ + KVM_GUESTDBG_USE_SW_BP | \ + KVM_GUESTDBG_USE_HW | \ + KVM_GUESTDBG_SINGLESTEP)#define vcpu_has_sve (vcpu) (system_supports_sve() && \ ((vcpu)->arch.flags & KVM_ARM64_GUEST_HAS_SVE))
Signature · 1 line
2.30.2
Marc Zyngier
maintainerPATCH v3 2/6 KVM: arm64: Get rid of host SVE tracking/saving
In reply to Marc ZyngierThe SVE host tracking in KVM is pretty involved. It relies on a set of flags tracking the ownership of the SVE register, as well as that of the EL0 access.
It is also pretty scary: __hyp_sve_save_host() computes a thread_struct pointer and obtains a sve_state which gets directly accessed without further ado, even on nVHE. How can this even work?
The answer to that is that it doesn't, and that this is mostly dead code. Closer examination shows that on executing a syscall, userspace loses its SVE state entirely. This is part of the ABI. Another thing to notice is that although the kernel provides helpers such as kernel_neon_begin()/end(), they only deal with the FP/NEON state, and not SVE.
Given that you can only execute a guest as the result of a syscall, and that the kernel cannot use SVE by itself, it becomes pretty obvious that there is never any host SVE state to save, and that this code is only there to increase confusion.
Get rid of the TIF_SVE tracking and host save infrastructure altogether.
--- arch/arm64/include/asm/kvm_host.h | 1 - arch/arm64/kvm/fpsimd.c | 20 +++++------------- arch/arm64/kvm/hyp/include/hyp/switch.h | 27 +++---------------------- 3 files changed, 8 insertions(+), 40 deletions(-)
arch/arm64/include/asm/kvm_host.h +0 -1
@@ -411,7 +411,6 @@ struct kvm_vcpu_arch {#define KVM_ARM64_DEBUG_DIRTY (1 << 0)#define KVM_ARM64_FP_ENABLED (1 << 1)/* guest FP regs loaded */ #define KVM_ARM64_FP_HOST (1 << 2)/* host FP regs loaded */ -#define KVM_ARM64_HOST_SVE_IN_USE (1 << 3)/* backup for host TIF_SVE */ #define KVM_ARM64_HOST_SVE_ENABLED (1 << 4)/* SVE enabled for EL0 */ #define KVM_ARM64_GUEST_HAS_SVE (1 << 5)/* SVE exposed to guest */ #define KVM_ARM64_VCPU_SVE_FINALIZED (1 << 6)/* SVE config completed */
arch/arm64/kvm/fpsimd.c +5 -15
@@ -66,22 +66,15 @@ int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu)* * Here, we just set the correct metadata to indicate that the FPSIMD * state in the cpu regs (if any) belongs to current on the host. -* -* TIF_SVE is backed up here, since it may get clobbered with guest state. -* This flag is restored by kvm_arch_vcpu_put_fp(vcpu). */ void kvm_arch_vcpu_load_fp (struct kvm_vcpu *vcpu) {BUG_ON (!current->mm); +BUG_ON (test_thread_flag (TIF_SVE )); - vcpu->arch.flags &= ~(KVM_ARM64_FP_ENABLED | -KVM_ARM64_HOST_SVE_IN_USE | -KVM_ARM64_HOST_SVE_ENABLED ); + vcpu->arch.flags &= ~KVM_ARM64_FP_ENABLED ; vcpu->arch.flags |=KVM_ARM64_FP_HOST ; -if (test_thread_flag (TIF_SVE )) - vcpu->arch.flags |=KVM_ARM64_HOST_SVE_IN_USE ; -if (read_sysreg (cpacr_el1) &CPACR_EL1_ZEN_EL0EN ) vcpu->arch.flags |=KVM_ARM64_HOST_SVE_ENABLED ; }
@@ -115,13 +108,11 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu)void kvm_arch_vcpu_put_fp (struct kvm_vcpu *vcpu) {unsigned long flags; -bool host_has_sve =system_supports_sve (); -bool guest_has_sve =vcpu_has_sve (vcpu);local_irq_save (flags);if (vcpu->arch.flags &KVM_ARM64_FP_ENABLED ) { -if (guest_has_sve) { +if (vcpu_has_sve (vcpu)) {__vcpu_sys_reg (vcpu,ZCR_EL1 ) =read_sysreg_el1 (SYS_ZCR );/* Restore the VL that was saved when bound to the CPU */
@@ -131,7 +122,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) }fpsimd_save_and_flush_cpu_state (); - }else if (has_vhe () && host_has_sve) { + }else if (has_vhe () &&system_supports_sve ()) {/* * The FPSIMD/SVE state in the CPU has not been touched, and we * have SVE (and VHE): CPACR_EL1 (alias CPTR_EL2) has been
@@ -145,8 +136,7 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)sysreg_clear_set (CPACR_EL1 ,CPACR_EL1_ZEN_EL0EN ,0 ); } -update_thread_flag (TIF_SVE , - vcpu->arch.flags &KVM_ARM64_HOST_SVE_IN_USE ); +update_thread_flag (TIF_SVE ,0 );local_irq_restore (flags); }
arch/arm64/kvm/hyp/include/hyp/switch.h +3 -24
@@ -143,16 +143,6 @@ static inline bool __populate_fault_info(struct kvm_vcpu *vcpu)return __get_fault_info (vcpu->arch.fault.esr_el2, &vcpu->arch.fault); } -static inline void __hyp_sve_save_host (struct kvm_vcpu *vcpu) -{ -struct thread_struct *thread; - - thread = container_of(vcpu->arch.host_fpsimd_state, struct thread_struct, - uw.fpsimd_state); - -__sve_save_state (sve_pffr (thread), &vcpu->arch.host_fpsimd_state->fpsr); -} -static inline void __hyp_sve_restore_guest (struct kvm_vcpu *vcpu) {sve_cond_update_zcr_vq (vcpu_sve_max_vq (vcpu) -1 ,SYS_ZCR_EL2 );
@@ -169,21 +159,14 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu)*/ static bool kvm_hyp_handle_fpsimd (struct kvm_vcpu *vcpu,u64 *exit_code) { -bool sve_guest, sve_host; +bool sve_guest;u8 esr_ec;u64 reg;if (!system_supports_fpsimd ())return false; -if (system_supports_sve ()) { - sve_guest =vcpu_has_sve (vcpu); - sve_host = vcpu->arch.flags &KVM_ARM64_HOST_SVE_IN_USE ; - }else { - sve_guest = false; - sve_host = false; - } - + sve_guest =vcpu_has_sve (vcpu); esr_ec =kvm_vcpu_trap_get_class (vcpu);/* Don't handle SVE traps for non-SVE vcpus here: */
@@ -207,11 +190,7 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)isb ();if (vcpu->arch.flags &KVM_ARM64_FP_HOST ) { -if (sve_host) -__hyp_sve_save_host (vcpu); -else -__fpsimd_save_state (vcpu->arch.host_fpsimd_state); - +__fpsimd_save_state (vcpu->arch.host_fpsimd_state); vcpu->arch.flags &= ~KVM_ARM64_FP_HOST ; }
Signature · 1 line
2.30.2
Marc Zyngier
maintainerPATCH v3 3/6 KVM: arm64: Remove unused __sve_save_state
In reply to Marc ZyngierNow that we don't have any users left for __sve_save_state, remove it altogether. Should we ever need to save the SVE state from the hypervisor again, we can always re-introduce it.
--- arch/arm64/include/asm/kvm_hyp.h | 1 - arch/arm64/kvm/hyp/fpsimd.S | 6 ------ 2 files changed, 7 deletions(-)
arch/arm64/include/asm/kvm_hyp.h +0 -1
@@ -90,7 +90,6 @@ void __debug_restore_host_buffers_nvhe(struct kvm_vcpu *vcpu);void __fpsimd_save_state (struct user_fpsimd_state *fp_regs);void __fpsimd_restore_state (struct user_fpsimd_state *fp_regs); -void __sve_save_state (void *sve_pffr,u32 *fpsr);void __sve_restore_state (void *sve_pffr,u32 *fpsr);#ifndef __KVM_NVHE_HYPERVISOR__
arch/arm64/kvm/hyp/fpsimd.S +0 -6
@@ -25,9 +25,3 @@ SYM_FUNC_START(__sve_restore_state) sve_load 0, x1, x2, 3 ret SYM_FUNC_END(__sve_restore_state) - -SYM_FUNC_START(__sve_save_state) - mov x2, #1 - sve_save 0, x1, x2, 3 - ret -SYM_FUNC_END(__sve_save_state)
Signature · 1 line
2.30.2
Re: [PATCH v3 3/6] KVM: arm64: Remove unused __sve_save_state
Quoted 3 lines · Marc Zyngier
Now that we don't have any users left for
__sve_save_state, remove it altogether. Should we ever need to save the SVE state from the hypervisor again, we can always re-introduce it.
Marc Zyngier
maintainerPATCH v3 4/6 KVM: arm64: Introduce flag shadowing TIF_FOREIGN_FPSTATE
In reply to Marc ZyngierWe currently have to maintain a mapping the thread_info structure at EL2 in order to be able to check the TIF_FOREIGN_FPSTATE flag.
In order to eventually get rid of this, start with a vcpu flag that shadows the thread flag on each entry into the hypervisor.
--- arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/kvm/arm.c | 1 + arch/arm64/kvm/fpsimd.c | 8 ++++++++ arch/arm64/kvm/hyp/include/hyp/switch.h | 2 +- 4 files changed, 12 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kvm_host.h +2 -0
@@ -441,6 +441,7 @@ struct kvm_vcpu_arch {#define KVM_ARM64_DEBUG_STATE_SAVE_SPE (1 << 12)/* Save SPE context if active */ #define KVM_ARM64_DEBUG_STATE_SAVE_TRBE (1 << 13)/* Save TRBE context if active */ +#define KVM_ARM64_FP_FOREIGN_FPSTATE (1 << 14)#define KVM_GUESTDBG_VALID_MASK (KVM_GUESTDBG_ENABLE | \ KVM_GUESTDBG_USE_SW_BP | \
@@ -736,6 +737,7 @@ long kvm_vm_ioctl_mte_copy_tags(struct kvm *kvm,/* Guest/host FPSIMD coordination helpers */ int kvm_arch_vcpu_run_map_fp (struct kvm_vcpu *vcpu);void kvm_arch_vcpu_load_fp (struct kvm_vcpu *vcpu); +void kvm_arch_vcpu_ctxflush_fp (struct kvm_vcpu *vcpu);void kvm_arch_vcpu_ctxsync_fp (struct kvm_vcpu *vcpu);void kvm_arch_vcpu_put_fp (struct kvm_vcpu *vcpu);
arch/arm64/kvm/arm.c +1 -0
@@ -849,6 +849,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) }kvm_arm_setup_debug (vcpu); +kvm_arch_vcpu_ctxflush_fp (vcpu);/************************************************************** * Enter the guest
arch/arm64/kvm/fpsimd.c +8 -0
@@ -79,6 +79,14 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu) vcpu->arch.flags |=KVM_ARM64_HOST_SVE_ENABLED ; } +void kvm_arch_vcpu_ctxflush_fp (struct kvm_vcpu *vcpu) +{ +if (test_thread_flag (TIF_FOREIGN_FPSTATE )) + vcpu->arch.flags |=KVM_ARM64_FP_FOREIGN_FPSTATE ; +else + vcpu->arch.flags &= ~KVM_ARM64_FP_FOREIGN_FPSTATE ; +} +/* * If the guest FPSIMD state was loaded, update the host's context * tracking data mark the CPU FPSIMD regs as dirty and belonging to vcpu
arch/arm64/kvm/hyp/include/hyp/switch.h +1 -1
@@ -49,7 +49,7 @@ static inline bool update_fp_enabled(struct kvm_vcpu *vcpu)* trap the accesses. */ if (!system_supports_fpsimd () || - vcpu->arch.host_thread_info->flags & _TIF_FOREIGN_FPSTATE) + vcpu->arch.flags &KVM_ARM64_FP_FOREIGN_FPSTATE ) vcpu->arch.flags &= ~(KVM_ARM64_FP_ENABLED |KVM_ARM64_FP_HOST );
Signature · 1 line
2.30.2
Marc Zyngier
maintainerPATCH v3 5/6 KVM: arm64: Stop mapping current thread_info at EL2
In reply to Marc ZyngierNow that we can track an equivalent of TIF_FOREIGN_FPSTATE, drop the mapping of current's thread_info at EL2.
--- arch/arm64/include/asm/kvm_host.h | 2 -- arch/arm64/kvm/fpsimd.c | 12 +----------- arch/arm64/kvm/hyp/include/hyp/switch.h | 1 - arch/arm64/kvm/hyp/nvhe/switch.c | 1 - arch/arm64/kvm/hyp/vhe/switch.c | 1 - 5 files changed, 1 insertion(+), 16 deletions(-)
arch/arm64/include/asm/kvm_host.h +0 -2
@@ -26,7 +26,6 @@#include <asm/fpsimd.h> #include <asm/kvm.h> #include <asm/kvm_asm.h> -#include <asm/thread_info.h> #define __KVM_HAVE_ARCH_INTC_INITIALIZED
@@ -321,7 +320,6 @@ struct kvm_vcpu_arch {struct kvm_guest_debug_arch vcpu_debug_state;struct kvm_guest_debug_arch external_debug_state; -struct thread_info *host_thread_info;/* hyp VA */ struct user_fpsimd_state *host_fpsimd_state;/* hyp VA */ struct {
arch/arm64/kvm/fpsimd.c +1 -11
@@ -7,7 +7,6 @@*/ #include <linux/irqflags.h> #include <linux/sched.h> -#include <linux/thread_info.h> #include <linux/kvm_host.h> #include <asm/fpsimd.h> #include <asm/kvm_asm.h>
@@ -28,17 +27,9 @@ int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) {int ret; -struct thread_info *ti = ¤t->thread_info;struct user_fpsimd_state *fpsimd = ¤t->thread.uw.fpsimd_state; -/* -* Make sure the host task thread flags and fpsimd state are -* visible to hyp: -*/ - ret =create_hyp_mappings (ti, ti +1 ,PAGE_HYP ); -if (ret) - goto error; - +/* Make sure the host task fpsimd state is visible to hyp: */ ret =create_hyp_mappings (fpsimd, fpsimd +1 ,PAGE_HYP );if (ret) goto error;
@@ -54,7 +45,6 @@ int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) goto error; } - vcpu->arch.host_thread_info =kern_hyp_va (ti); vcpu->arch.host_fpsimd_state =kern_hyp_va (fpsimd); error:return ret;
arch/arm64/kvm/hyp/include/hyp/switch.h +0 -1
@@ -29,7 +29,6 @@#include <asm/fpsimd.h> #include <asm/debug-monitors.h> #include <asm/processor.h> -#include <asm/thread_info.h> struct kvm_exception_table_entry {int insn, fixup;
arch/arm64/kvm/hyp/nvhe/switch.c +0 -1
@@ -25,7 +25,6 @@#include <asm/fpsimd.h> #include <asm/debug-monitors.h> #include <asm/processor.h> -#include <asm/thread_info.h> #include <nvhe/fixed_config.h> #include <nvhe/mem_protect.h>
arch/arm64/kvm/hyp/vhe/switch.c +0 -1
@@ -24,7 +24,6 @@#include <asm/fpsimd.h> #include <asm/debug-monitors.h> #include <asm/processor.h> -#include <asm/thread_info.h> /* VHE specific context */ DEFINE_PER_CPU(struct kvm_host_data, kvm_host_data);
Signature · 1 line
2.30.2
Re: [PATCH v3 5/6] KVM: arm64: Stop mapping current thread_info at EL2
Quoted 2 lines · Marc Zyngier
Now that we can track an equivalent of
TIF_FOREIGN_FPSTATE, drop the mapping of current'sthread_infoat EL2.
Marc Zyngier
maintainerPATCH v3 6/6 arm64/fpsimd: Document the use of TIF_FOREIGN_FPSTATE by KVM
In reply to Marc ZyngierThe bit of documentation that talks about TIF_FOREIGN_FPSTATE does not mention the ungodly tricks that KVM plays with this flag.
Try and document this for the posterity.
--- arch/arm64/kernel/fpsimd.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/arm64/kernel/fpsimd.c +5 -1
@@ -78,7 +78,11 @@* indicate whether or not the userland FPSIMD state of the current task is * present in the registers. The flag is set unless the FPSIMD registers of this * CPU currently contain the most recent userland FPSIMD state of the current -* task. +* task. If the task is behaving as a VMM, then this is will be managed by +* KVM which will clear it to indicate that the vcpu FPSIMD state is currently +* loaded on the CPU, allowing the state to be saved if a FPSIMD-aware +* softirq kicks in. Upon vcpu_put(), KVM will save the vcpu FP state and +* flag the register state as invalid. * * In order to allow softirq handlers to use FPSIMD, kernel_neon_begin() may * save the task's FPSIMD context back to task_struct from softirq context.
Signature · 1 line
2.30.2