arm64/sve: Clean up KVM integration and optimise syscalls
Subsystems (3)
v3 cover arm64/sve: Clean up KVM integration and optimise syscalls
This patch series attempts to clarify the tracking of which set of floating point registers we save on systems supporting SVE, particularly with reference to KVM, and then uses the results of this clarification to improve the performance of simple syscalls where we return directly to userspace in cases where userspace is using SVE.
At present we track which register state is active by using the TIF_SVE flag for the current task which also controls if userspace is able to use SVE, this is reasonably straightforward if limiting but for KVM it gets a bit hairy since we may have guest state loaded in registers. This results in KVM modifying TIF_SVE for the VMM task while the guest is running which doesn't entirely help make things easy to follow. To help make things clearer the series changes things so that in addition to TIF_SVE we explicitly track both the type of registers that are currently saved in the task struct and the type of registers that we should save when we do so. TIF_SVE then solely controls if userspace can use SVE without trapping, it has no function for KVM guests and we can remove the code for managing it from KVM.
The refactoring to add the separate tracking is initially done by adding the new state together with checks that the state corresponds to expectations when we look at it before subsequent patches make use of the separated state, the goal being to both split out the more repetitive bits of tha change and make it easier to debug any problems that might arise.
With the state tracked separately we then start to optimise the performance of syscalls when the process is using SVE. Currently every syscall disables SVE for userspace which means that we need to trap to EL1 again on the next SVE instruction, flush the SVE registers, and reenable SVE for EL0, creating overhead for tasks that mix SVE and syscalls. We build on the above refactoring to eliminate this overhead for simple syscalls which return directly to userspace by keeping SVE enabled unless we need to reload the state from memory, meaning that if syscalls do not block we avoid the overhead of trapping to EL1 again on next use of SVE.
v3:
- Rebase onto my series "arm64/sme: SME related fixes" since there is a direct dependency on the signal fix and testing is much easier with the bug fixes rolled in.
- s/type/fp_type/ in struct fpsimd_last_state_struct.
- Add comment about the V register storage being ignored when data is stored in SVE format.
- Move dropping of special casing for FPSIMD register state in SME into a separate patch later in the series.
- Simplify logic in task_fpsimd_load().
- Remove support for leaving the SVE state not shared with FPSIMD untouched, keep the unconditional flush.
v2:
- Rebase onto v5.19-rc3.
- Don't warn when restoring streaming mode SVE without TIF_SVE.
Mark Brown (7):
KVM: arm64: Discard any SVE state when entering KVM guests arm64/fpsimd: Track the saved FPSIMD state type separately to TIF_SVE arm64/fpsimd: Have KVM explicitly say which FP registers to save arm64/fpsimd: Stop using TIF_SVE to manage register saving in KVM arm64/fpsimd: Load FP state based on recorded data type arm64/fpsimd: SME no longer requires SVE register state arm64/sve: Leave SVE enabled on syscall if we don't context switch
arch/arm64/include/asm/fpsimd.h | 4 +- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/include/asm/processor.h | 7 ++ arch/arm64/kernel/fpsimd.c | 137 +++++++++++++++++++++++------ arch/arm64/kernel/process.c | 2 + arch/arm64/kernel/ptrace.c | 5 +- arch/arm64/kernel/signal.c | 7 +- arch/arm64/kernel/syscall.c | 19 ++-- arch/arm64/kvm/fpsimd.c | 16 ++-- 9 files changed, 148 insertions(+), 50 deletions(-)
base-commit: bb357a5e4232401e587da41329d8de5b42acd10e
Signature · 6 lines
2.30.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 1/7 KVM: arm64: Discard any SVE state when entering KVM guests
Since 8383741ab2e773a99 (KVM: arm64: Get rid of host SVE tracking/saving) KVM has not tracked the host SVE state, relying on the fact that we currently disable SVE whenever we perform a syscall. This may not be true in future since performance optimisation may result in us keeping SVE enabled in order to avoid needing to take access traps to reenable it. Handle this by clearing TIF_SVE and converting the stored task state to FPSIMD format when preparing to run the guest. This is done with a new call fpsimd_kvm_prepare() to keep the direct state manipulation functions internal to fpsimd.c.
--- arch/arm64/include/asm/fpsimd.h | 1 + arch/arm64/kernel/fpsimd.c | 23 +++++++++++++++++++++++ arch/arm64/kvm/fpsimd.c | 3 ++- 3 files changed, 26 insertions(+), 1 deletion(-)
arch/arm64/include/asm/fpsimd.h +1 -0
@@ -56,6 +56,7 @@ extern void fpsimd_signal_preserve_current_state(void);extern void fpsimd_preserve_current_state (void );extern void fpsimd_restore_current_state (void );extern void fpsimd_update_current_state (struct user_fpsimd_state const *state); +extern void fpsimd_kvm_prepare (void ); extern void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *state, void *sve_state, unsigned int sve_vl,
arch/arm64/kernel/fpsimd.c +23 -0
@@ -1627,6 +1627,29 @@ void fpsimd_signal_preserve_current_state(void)sve_to_fpsimd (current); } +/* +* Called by KVM when entering the guest. +*/ +void fpsimd_kvm_prepare (void ) +{ +if (!system_supports_sve ()) +return ; + +/* +* KVM does not save host SVE state since we can only enter +* the guest from a syscall so the ABI means that only the +* non-saved SVE state needs to be saved. If we have left +* SVE enabled for performance reasons then update the task +* state to be FPSIMD only. +*/ +get_cpu_fpsimd_context (); + +if (test_and_clear_thread_flag (TIF_SVE )) +sve_to_fpsimd (current); + +put_cpu_fpsimd_context (); +} +/* * Associate current's FPSIMD context with this cpu * The caller must have ownership of the cpu FPSIMD context before calling
arch/arm64/kvm/fpsimd.c +2 -1
@@ -75,7 +75,8 @@ int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu)void kvm_arch_vcpu_load_fp (struct kvm_vcpu *vcpu) {BUG_ON (!current->mm); -BUG_ON (test_thread_flag (TIF_SVE )); + +fpsimd_kvm_prepare ();if (!system_supports_fpsimd ())return ;
Signature · 6 lines
2.30.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 1/7] KVM: arm64: Discard any SVE state when entering KVM guests
Quoted 69 lines · Mark Brown
Since
8383741ab2e773a99(KVM: arm64: Get rid of host SVE tracking/saving) KVM has not tracked the host SVE state, relying on the fact that we currently disable SVE whenever we perform a syscall. This may not be true in future since performance optimisation may result in us keeping SVE enabled in order to avoid needing to take access traps to reenable it. Handle this by clearingTIF_SVEand converting the stored task state to FPSIMD format when preparing to run the guest. This is done with a new callfpsimd_kvm_prepare()to keep the direct state manipulation functions internal tofpsimd.c.Signed-off-by Mark Brown <[email protected]>--- arch/arm64/include/asm/fpsimd.h | 1 + arch/arm64/kernel/fpsimd.c | 23 +++++++++++++++++++++++ arch/arm64/kvm/fpsimd.c | 3 ++- 3 files changed, 26 insertions(+), 1 deletion(-)arch/arm64/include/asm/fpsimd.h +1
-0@@ -56,6 +56,7 @@ extern void fpsimd_signal_preserve_current_state(void);extern void fpsimd_preserve_current_state (void );extern void fpsimd_restore_current_state (void );extern void fpsimd_update_current_state (struct user_fpsimd_state const *state); +extern void fpsimd_kvm_prepare (void ); extern void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *state, void *sve_state, unsigned int sve_vl,arch/arm64/kernel/fpsimd.c +23
-0@@ -1627,6 +1627,29 @@ void fpsimd_signal_preserve_current_state(void)sve_to_fpsimd (current); } +/* +* Called by KVM when entering the guest. +*/ +void fpsimd_kvm_prepare (void ) +{ +if (!system_supports_sve ()) +return ; + +/* +* KVM does not save host SVE state since we can only enter +* the guest from a syscall so the ABI means that only the +* non-saved SVE state needs to be saved. If we have left +* SVE enabled for performance reasons then update the task +* state to be FPSIMD only. +*/ +get_cpu_fpsimd_context (); + +if (test_and_clear_thread_flag (TIF_SVE )) +sve_to_fpsimd (current); + +put_cpu_fpsimd_context (); +} +/* * Associate current's FPSIMD context with this cpu * The caller must have ownership of the cpu FPSIMD context before calling arch/arm64/kvm/fpsimd.c +0
-0
arch/arm64/kvm/fpsimd.c int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) · line 79
Quoted 6 lines
arch/arm64/kvm/fpsimd.c +2
-1@@ -75,7 +75,8 @@ int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu)void kvm_arch_vcpu_load_fp (struct kvm_vcpu *vcpu) {BUG_ON (!current->mm); -BUG_ON (test_thread_flag (TIF_SVE )); + +fpsimd_kvm_prepare ();
Why is this *before* the check against system_supports_fpsimd()? I don't think the architecture allows SVE without FP, for obvious reasons...
arch/arm64/kvm/fpsimd.c int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) · line 82
Quoted 3 lines
arch/arm64/kvm/fpsimd.c +0
-0if (!system_supports_fpsimd ())return ;
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 1/7] KVM: arm64: Discard any SVE state when entering KVM guests
Quoted 12 lines · Marc Zyngier
arch/arm64/kvm/fpsimd.c int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) · line 79
Quoted 6 lines · Mark Brown
void
kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu){arch/arm64/kvm/fpsimd.c +2
-1BUG_ON (!current->mm); -BUG_ON (test_thread_flag (TIF_SVE )); + +fpsimd_kvm_prepare ();Why is this *before* the check against
system_supports_fpsimd()? I don't think the architecture allows SVE without FP, for obvious reasons...
Good point, though now that I think about it I can't think of a requirement for FP when implementing SME (there's certainly not one for SVE). There's no use for that hook now though.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 1/7] KVM: arm64: Discard any SVE state when entering KVM guests
Quoted 16 lines · Mark Brown
[1 <text/plain; us-ascii (quoted-printable)>]
Quoted 12 lines · Marc Zyngier
arch/arm64/kvm/fpsimd.c int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu) · line 79
Quoted 6 lines · Mark Brown
void
kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu){arch/arm64/kvm/fpsimd.c +2
-1BUG_ON (!current->mm); -BUG_ON (test_thread_flag (TIF_SVE )); + +fpsimd_kvm_prepare ();Why is this *before* the check against
system_supports_fpsimd()? I don't think the architecture allows SVE without FP, for obvious reasons...
Quoted 3 lines
Good point, though now that I think about it I can't think of a requirement for FP when implementing SME (there's certainly not one for SVE).
Even if the architecture was allowing this madness, KVM doesn't allow SVE if FP is not available, just like the rest of the kernel.
Quoted 1 line
There's no use for that hook now though.
Care to clarify?
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 1/7] KVM: arm64: Discard any SVE state when entering KVM guests
Quoted 5 lines · Marc Zyngier
Quoted 1 line · Mark Brown
There's no use for that hook now though.
Care to clarify?
We don't do anything for SME even if we were to support SME with no FP properly.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 2/7 arm64/fpsimd: Track the saved FPSIMD state type separately to TIF_SVE
In reply to Mark BrownWhen we save the state for the floating point registers this can be done in the form visible through either the FPSIMD V registers or the SVE Z and P registers. At present we track which format is currently used based on TIF_SVE and the SME streaming mode state but particularly in the SVE case this limits our options for optimising things, especially around syscalls. Introduce a new enum in thread_struct which explicitly states which format is active and keep it up to date when we change it.
At present we do not use this state except to verify that it has the expected value when loading the state, future patches will introduce functional changes.
--- arch/arm64/include/asm/fpsimd.h | 2 +- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/include/asm/processor.h | 6 ++++ arch/arm64/kernel/fpsimd.c | 58 ++++++++++++++++++++++-------- arch/arm64/kernel/process.c | 2 ++ arch/arm64/kernel/ptrace.c | 3 ++ arch/arm64/kernel/signal.c | 7 +++- arch/arm64/kvm/fpsimd.c | 3 +- 8 files changed, 64 insertions(+), 18 deletions(-)
arch/arm64/include/asm/fpsimd.h +1 -1
@@ -61,7 +61,7 @@ extern void fpsimd_kvm_prepare(void);extern void fpsimd_bind_state_to_cpu (struct user_fpsimd_state *state,void *sve_state,unsigned int sve_vl,void *za_state,unsigned int sme_vl, -u64 *svcr); +u64 *svcr,enum fp_state *type);extern void fpsimd_flush_task_state (struct task_struct *target);extern void fpsimd_save_and_flush_cpu_state (void );
arch/arm64/include/asm/kvm_host.h +1 -0
@@ -310,6 +310,7 @@ struct kvm_vcpu_arch {void *sve_state;unsigned int sve_max_vl;u64 svcr; +enum fp_state fp_type;/* Stage 2 paging state used by the hardware on next switch */ struct kvm_s2_mmu *hw_mmu;
arch/arm64/include/asm/processor.h +6 -0
@@ -122,6 +122,11 @@ enum vec_type { ARM64_VEC_MAX, }; +enum fp_state { +FP_STATE_FPSIMD , +FP_STATE_SVE , +}; +struct cpu_context {unsigned long x19;unsigned long x20;
@@ -152,6 +157,7 @@ struct thread_struct {struct user_fpsimd_state fpsimd_state; } uw; +enum fp_state fp_type;/* registers FPSIMD or SVE? */ unsigned int fpsimd_cpu;void *sve_state;/* SVE registers, if any */ void *za_state;/* ZA register, if any */
arch/arm64/kernel/fpsimd.c +43 -15
144 lines not on this page · raw message
arch/arm64/kernel/process.c +2 -0
@@ -335,6 +335,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)clear_tsk_thread_flag (dst,TIF_SME ); } + dst->thread.fp_type =FP_STATE_FPSIMD ; +/* clear any pending asynchronous tag fault raised by the parent */ clear_tsk_thread_flag (dst,TIF_MTE_ASYNC_FAULT );
arch/arm64/kernel/ptrace.c +3 -0
@@ -894,6 +894,7 @@ static int sve_set_common(struct task_struct *target,clear_tsk_thread_flag (target,TIF_SVE );if (type ==ARM64_VEC_SME )fpsimd_force_sync_to_sve (target); + target->thread.fp_type =FP_STATE_FPSIMD ; goto out; }
@@ -916,6 +917,7 @@ static int sve_set_common(struct task_struct *target,if (!target->thread.sve_state) { ret = -ENOMEM ;clear_tsk_thread_flag (target,TIF_SVE ); + target->thread.fp_type =FP_STATE_FPSIMD ; goto out; }
@@ -927,6 +929,7 @@ static int sve_set_common(struct task_struct *target,*/ fpsimd_sync_to_sve (target);set_tsk_thread_flag (target,TIF_SVE ); + target->thread.fp_type =FP_STATE_SVE ;BUILD_BUG_ON (SVE_PT_SVE_OFFSET !=sizeof (header)); start =SVE_PT_SVE_OFFSET ;
arch/arm64/kernel/signal.c +6 -1
@@ -207,6 +207,7 @@ static int restore_fpsimd_context(struct fpsimd_context __user *ctx)__get_user_error (fpsimd.fpcr, &ctx->fpcr, err);clear_thread_flag (TIF_SVE ); + current->thread.fp_type =FP_STATE_FPSIMD ;/* load the hardware registers from the fpsimd_state structure */ if (!err)
@@ -292,6 +293,7 @@ static int restore_sve_fpsimd_context(struct user_ctxs *user)if (sve.head.size <=sizeof (*user->sve)) {clear_thread_flag (TIF_SVE ); current->thread.svcr &= ~SVCR_SM_MASK ; + current->thread.fp_type =FP_STATE_FPSIMD ; goto fpsimd_only; }
@@ -327,6 +329,7 @@ static int restore_sve_fpsimd_context(struct user_ctxs *user) current->thread.svcr |=SVCR_SM_MASK ; elseset_thread_flag (TIF_SVE ); + current->thread.fp_type =FP_STATE_SVE ; fpsimd_only:/* copy the FP and status/control registers */
@@ -932,9 +935,11 @@ static void setup_return(struct pt_regs *regs, struct k_sigaction *ka,* FPSIMD register state - flush the saved FPSIMD * register state in case it gets loaded. */ -if (current->thread.svcr &SVCR_SM_MASK ) +if (current->thread.svcr &SVCR_SM_MASK ) {memset (¤t->thread.uw.fpsimd_state,0 ,sizeof (current->thread.uw.fpsimd_state)); + current->thread.fp_type =FP_STATE_FPSIMD ; + } current->thread.svcr &= ~(SVCR_ZA_MASK |SVCR_SM_MASK );
arch/arm64/kvm/fpsimd.c +2 -1
@@ -140,7 +140,8 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu)fpsimd_bind_state_to_cpu (&vcpu->arch.ctxt.fp_regs, vcpu->arch.sve_state, vcpu->arch.sve_max_vl, -NULL ,0 , &vcpu->arch.svcr); +NULL ,0 , &vcpu->arch.svcr, + &vcpu->arch.fp_type);clear_thread_flag (TIF_FOREIGN_FPSTATE );update_thread_flag (TIF_SVE ,vcpu_has_sve (vcpu));
Signature · 6 lines
2.30.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 2/7] arm64/fpsimd: Track the saved FPSIMD state type separately to TIF_SVE
Quoted 43 lines · Mark Brown
When we save the state for the floating point registers this can be done in the form visible through either the FPSIMD V registers or the SVE Z and P registers. At present we track which format is currently used based on
TIF_SVEand the SME streaming mode state but particularly in the SVE case this limits our options for optimising things, especially around syscalls. Introduce a new enum inthread_structwhich explicitly states which format is active and keep it up to date when we change it.At present we do not use this state except to verify that it has the expected value when loading the state, future patches will introduce functional changes.
Signed-off-by Mark Brown <[email protected]>--- arch/arm64/include/asm/fpsimd.h | 2 +- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/include/asm/processor.h | 6 ++++ arch/arm64/kernel/fpsimd.c | 58 ++++++++++++++++++++++-------- arch/arm64/kernel/process.c | 2 ++ arch/arm64/kernel/ptrace.c | 3 ++ arch/arm64/kernel/signal.c | 7 +++- arch/arm64/kvm/fpsimd.c | 3 +- 8 files changed, 64 insertions(+), 18 deletions(-)arch/arm64/include/asm/fpsimd.h +1
-1@@ -61,7 +61,7 @@ extern void fpsimd_kvm_prepare(void);extern void fpsimd_bind_state_to_cpu (struct user_fpsimd_state *state,void *sve_state,unsigned int sve_vl,void *za_state,unsigned int sme_vl, -u64 *svcr); +u64 *svcr,enum fp_state *type);extern void fpsimd_flush_task_state (struct task_struct *target);extern void fpsimd_save_and_flush_cpu_state (void );arch/arm64/include/asm/kvm_host.h +0
-0
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 313
Quoted 4 lines
arch/arm64/include/asm/kvm_host.h +1
-0@@ -310,6 +310,7 @@ struct kvm_vcpu_arch {void *sve_state;unsigned int sve_max_vl;u64 svcr; +enum fp_state fp_type;
Is it a state or a type? Some consistency would help. Also, what does this represent? Your commit message keeps talking about the FP/SVE state for the host, but this is obviously a guest-related structure. How do the two relate?
Finally, before this patch, pahole shows this:
struct kvm_vcpu_arch {
struct kvm_cpu_context ctxt; /* 0 1824 */
/* --- cacheline 28 boundary (1792 bytes) was 32 bytes ago --- */
void * sve_state; /* 1824 8 */
unsigned int sve_max_vl; /* 1832 4 */
/* XXX 4 bytes hole, try to pack */
u64 svcr; /* 1840 8 */
struct kvm_s2_mmu * hw_mmu; /* 1848 8 */
[...]
};
After it, we gain an extra hole:
struct kvm_vcpu_arch {
struct kvm_cpu_context ctxt; /* 0 1824 */
/* --- cacheline 28 boundary (1792 bytes) was 32 bytes ago --- */
void * sve_state; /* 1824 8 */
unsigned int sve_max_vl; /* 1832 4 */
/* XXX 4 bytes hole, try to pack */
u64 svcr; /* 1840 8 */
enum fp_state fp_type; /* 1848 4 */
/* XXX 4 bytes hole, try to pack */
/* --- cacheline 29 boundary (1856 bytes) --- */
struct kvm_s2_mmu * hw_mmu; /* 1856 8 */
[...]
};
Packing things wouldn't hurt.
Quoted 20 lines
arch/arm64/include/asm/kvm_host.h +0
-0/* Stage 2 paging state used by the hardware on next switch */ struct kvm_s2_mmu *hw_mmu;arch/arm64/include/asm/processor.h +5
-0@@ -122,6 +122,11 @@ enum vec_type { ARM64_VEC_MAX, }; +enum fp_state { +FP_STATE_FPSIMD , +FP_STATE_SVE , +}; +struct cpu_context {unsigned long x19;unsigned long x20;
arch/arm64/include/asm/processor.h struct thread_struct { · line 160
Quoted 4 lines
arch/arm64/include/asm/processor.h +1
-0@@ -152,6 +157,7 @@ struct thread_struct {struct user_fpsimd_state fpsimd_state; } uw; +enum fp_state fp_type;/* registers FPSIMD or SVE? */
Same comment about the state vs type.
arch/arm64/kernel/fpsimd.c struct fpsimd_last_state_struct { · line 128
Quoted 12 lines
arch/arm64/include/asm/processor.h +0
-0unsigned int fpsimd_cpu;void *sve_state;/* SVE registers, if any */ void *za_state;/* ZA register, if any */ arch/arm64/kernel/fpsimd.c +1
-0@@ -125,6 +125,7 @@ struct fpsimd_last_state_struct {u64 *svcr;unsigned int sve_vl;unsigned int sme_vl; +enum fp_state *fp_type;
Same thing. Grouping the pointer together would probably help readability as well.
Quoted 239 lines
arch/arm64/kernel/fpsimd.c +42
-15}; static DEFINE_PER_CPU(struct fpsimd_last_state_struct, fpsimd_last_state);@@ -330,15 +331,6 @@ void task_set_vl_onexec(struct task_struct *task, enum vec_type type,* The task can execute SVE instructions while in userspace without * trapping to the kernel. * -* When stored, Z0-Z31 (incorporating Vn in bits[127:0] or the -* corresponding Zn), P0-P15 and FFR are encoded in -* task->thread.sve_state, formatted appropriately for vector -* length task->thread.sve_vl or, if SVCR.SM is set, -* task->thread.sme_vl. -* -* task->thread.sve_state must point to a valid buffer at least -* sve_state_size(task) bytes in size. -* * During any syscall, the kernel may optionally clear TIF_SVE and * discard the vector state except for the FPSIMD subset. * @@ -348,7 +340,15 @@ void task_set_vl_onexec(struct task_struct *task, enum vec_type type,* do_sve_acc() to be called, which does some preparation and then * sets TIF_SVE. * -* When stored, FPSIMD registers V0-V31 are encoded in +* During any syscall, the kernel may optionally clear TIF_SVE and +* discard the vector state except for the FPSIMD subset. +* +* The data will be stored in one of two formats: +* +* * FPSIMD only - FP_STATE_FPSIMD: +* +* When the FPSIMD only state stored task->thread.fp_type is set to +* FP_STATE_FPSIMD, the FPSIMD registers V0-V31 are encoded in * task->thread.uw.fpsimd_state; bits [max : 128] for each of Z0-Z31 are * logically zero but not stored anywhere; P0-P15 and FFR are not * stored and have unspecified values from userspace's point of @@ -358,6 +358,19 @@ void task_set_vl_onexec(struct task_struct *task, enum vec_type type,* task->thread.sve_state does not need to be non-NULL, valid or any * particular size: it must not be dereferenced. * +* * SVE state - FP_STATE_SVE: +* +* When the full SVE state is stored task->thread.fp_type is set to +* FP_STATE_SVE and Z0-Z31 (incorporating Vn in bits[127:0] or the +* corresponding Zn), P0-P15 and FFR are encoded in in +* task->thread.sve_state, formatted appropriately for vector +* length task->thread.sve_vl or, if SVCR.SM is set, +* task->thread.sme_vl. The storage for the vector registers in +* task->thread.uw.fpsimd_state should be ignored. +* +* task->thread.sve_state must point to a valid buffer at least +* sve_state_size(task) bytes in size. +* * * FPSR and FPCR are always stored in task->thread.uw.fpsimd_state * irrespective of whether TIF_SVE is clear or set, since these are * not vector length dependent. @@ -404,12 +417,15 @@ static void task_fpsimd_load(void) } } -if (restore_sve_regs) +if (restore_sve_regs) { +WARN_ON_ONCE (current->thread.fp_type !=FP_STATE_SVE );sve_load_state (sve_pffr (¤t->thread), ¤t->thread.uw.fpsimd_state.fpsr, restore_ffr); -else + }else { +WARN_ON_ONCE (current->thread.fp_type !=FP_STATE_FPSIMD );fpsimd_load_state (¤t->thread.uw.fpsimd_state); + } }/* @@ -474,8 +490,10 @@ static void fpsimd_save(void)sve_save_state ((char *)last->sve_state +sve_ffr_offset (vl), &last->st->fpsr, save_ffr); + *last->fp_type =FP_STATE_SVE ; }else {fpsimd_save_state (last->st); + *last->fp_type =FP_STATE_FPSIMD ; } }@@ -848,8 +866,10 @@ int vec_set_vector_length(struct task_struct *task, enum vec_type type,fpsimd_flush_task_state (task);if (test_and_clear_tsk_thread_flag (task,TIF_SVE ) || -thread_sm_enabled (&task->thread)) +thread_sm_enabled (&task->thread)) {sve_to_fpsimd (task); + task->thread.fp_type =FP_STATE_FPSIMD ; + }if (system_supports_sme () && type ==ARM64_VEC_SME ) { task->thread.svcr &= ~(SVCR_SM_MASK |@@ -1368,6 +1388,7 @@ static void sve_init_regs(void)fpsimd_bind_task_to_cpu (); }else {fpsimd_to_sve (current); + current->thread.fp_type =FP_STATE_SVE ; } }@@ -1596,6 +1617,8 @@ void fpsimd_flush_thread(void) current->thread.svcr =0 ; } + current->thread.fp_type =FP_STATE_FPSIMD ; +put_cpu_fpsimd_context ();kfree (sve_state);kfree (za_state);@@ -1644,8 +1667,10 @@ void fpsimd_kvm_prepare(void)*/ get_cpu_fpsimd_context (); -if (test_and_clear_thread_flag (TIF_SVE )) +if (test_and_clear_thread_flag (TIF_SVE )) {sve_to_fpsimd (current); + current->thread.fp_type =FP_STATE_FPSIMD ; + }put_cpu_fpsimd_context (); }@@ -1667,6 +1692,7 @@ static void fpsimd_bind_task_to_cpu(void) last->sve_vl =task_get_sve_vl (current); last->sme_vl =task_get_sme_vl (current); last->svcr = ¤t->thread.svcr; + last->fp_type = ¤t->thread.fp_type; current->thread.fpsimd_cpu =smp_processor_id ();/* @@ -1690,7 +1716,8 @@ static void fpsimd_bind_task_to_cpu(void)void fpsimd_bind_state_to_cpu (struct user_fpsimd_state *st,void *sve_state,unsigned int sve_vl,void *za_state, -unsigned int sme_vl,u64 *svcr) +unsigned int sme_vl,u64 *svcr, +enum fp_state *type) {struct fpsimd_last_state_struct *last =this_cpu_ptr (&fpsimd_last_state);@@ -1704,6 +1731,7 @@ void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *st, void *sve_state, last->za_state = za_state; last->sve_vl = sve_vl; last->sme_vl = sme_vl; + last->fp_type = type; }/* arch/arm64/kernel/process.c +2
-0@@ -335,6 +335,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)clear_tsk_thread_flag (dst,TIF_SME ); } + dst->thread.fp_type =FP_STATE_FPSIMD ; +/* clear any pending asynchronous tag fault raised by the parent */ clear_tsk_thread_flag (dst,TIF_MTE_ASYNC_FAULT );arch/arm64/kernel/ptrace.c +3
-0@@ -894,6 +894,7 @@ static int sve_set_common(struct task_struct *target,clear_tsk_thread_flag (target,TIF_SVE );if (type ==ARM64_VEC_SME )fpsimd_force_sync_to_sve (target); + target->thread.fp_type =FP_STATE_FPSIMD ; goto out; }@@ -916,6 +917,7 @@ static int sve_set_common(struct task_struct *target,if (!target->thread.sve_state) { ret = -ENOMEM ;clear_tsk_thread_flag (target,TIF_SVE ); + target->thread.fp_type =FP_STATE_FPSIMD ; goto out; }@@ -927,6 +929,7 @@ static int sve_set_common(struct task_struct *target,*/ fpsimd_sync_to_sve (target);set_tsk_thread_flag (target,TIF_SVE ); + target->thread.fp_type =FP_STATE_SVE ;BUILD_BUG_ON (SVE_PT_SVE_OFFSET !=sizeof (header)); start =SVE_PT_SVE_OFFSET ;arch/arm64/kernel/signal.c +6
-1@@ -207,6 +207,7 @@ static int restore_fpsimd_context(struct fpsimd_context __user *ctx)__get_user_error (fpsimd.fpcr, &ctx->fpcr, err);clear_thread_flag (TIF_SVE ); + current->thread.fp_type =FP_STATE_FPSIMD ;/* load the hardware registers from the fpsimd_state structure */ if (!err)@@ -292,6 +293,7 @@ static int restore_sve_fpsimd_context(struct user_ctxs *user)if (sve.head.size <=sizeof (*user->sve)) {clear_thread_flag (TIF_SVE ); current->thread.svcr &= ~SVCR_SM_MASK ; + current->thread.fp_type =FP_STATE_FPSIMD ; goto fpsimd_only; }@@ -327,6 +329,7 @@ static int restore_sve_fpsimd_context(struct user_ctxs *user) current->thread.svcr |=SVCR_SM_MASK ; elseset_thread_flag (TIF_SVE ); + current->thread.fp_type =FP_STATE_SVE ; fpsimd_only:/* copy the FP and status/control registers */ @@ -932,9 +935,11 @@ static void setup_return(struct pt_regs *regs, struct k_sigaction *ka,* FPSIMD register state - flush the saved FPSIMD * register state in case it gets loaded. */ -if (current->thread.svcr &SVCR_SM_MASK ) +if (current->thread.svcr &SVCR_SM_MASK ) {memset (¤t->thread.uw.fpsimd_state,0 ,sizeof (current->thread.uw.fpsimd_state)); + current->thread.fp_type =FP_STATE_FPSIMD ; + } current->thread.svcr &= ~(SVCR_ZA_MASK |SVCR_SM_MASK );arch/arm64/kvm/fpsimd.c +0
-0
arch/arm64/kvm/fpsimd.c void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) · line 147
Quoted 9 lines
arch/arm64/kvm/fpsimd.c +2
-1@@ -140,7 +140,8 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu)fpsimd_bind_state_to_cpu (&vcpu->arch.ctxt.fp_regs, vcpu->arch.sve_state, vcpu->arch.sve_max_vl, -NULL ,0 , &vcpu->arch.svcr); +NULL ,0 , &vcpu->arch.svcr, + &vcpu->arch.fp_type);clear_thread_flag (TIF_FOREIGN_FPSTATE );update_thread_flag (TIF_SVE ,vcpu_has_sve (vcpu));
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 2/7] arm64/fpsimd: Track the saved FPSIMD state type separately to TIF_SVE
Quoted 10 lines · Marc Zyngier
Quoted 7 lines · Mark Brown
When we save the state for the floating point registers this can be done in the form visible through either the FPSIMD V registers or the SVE Z and P registers. At present we track which format is currently used based on
TIF_SVEand the SME streaming mode state but particularly in the SVE case this limits our options for optimising things, especially around syscalls. Introduce a new enum inthread_structwhich explicitly states which format is active and keep it up to date when we change it.
Quoted 7 lines
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 313
Quoted 5 lines
At present we do not use this state except to verify that it has the expected value when loading the state, future patches will introduce functional changes.
arch/arm64/include/asm/kvm_host.h +1
-0+enum fp_state fp_type;Is it a state or a type? Some consistency would help. Also, what does
We can bikeshed this either way - the state currently stored is of a particular type. I'll probably go for type.
Quoted 3 lines
this represent? Your commit message keeps talking about the FP/SVE state for the host, but this is obviously a guest-related structure. How do the two relate?
The commit message talks about saving the floating point state in general which is something we do for both the host and the guest. The optimisation cases I am focusing on right now are more on host usage but the complexity with tracking that currently blocks them crosses both host and guest, indeed the biggest improvement overall is probably that tracking the guest state stops requiring us to fiddle with the host task's state which to me at least makes things clearer.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 2/7] arm64/fpsimd: Track the saved FPSIMD state type separately to TIF_SVE
Quoted 17 lines · Mark Brown
[1 <text/plain; us-ascii (7bit)>]
Quoted 13 lines · Marc Zyngier
Quoted 11 lines · Mark Brown
When we save the state for the floating point registers this can be done in the form visible through either the FPSIMD V registers or the SVE Z and P registers. At present we track which format is currently used based on
TIF_SVEand the SME streaming mode state but particularly in the SVE case this limits our options for optimising things, especially around syscalls. Introduce a new enum inthread_structwhich explicitly states which format is active and keep it up to date when we change it.At present we do not use this state except to verify that it has the expected value when loading the state, future patches will introduce functional changes.
Quoted 6 lines
Quoted 3 lines
arch/arm64/include/asm/kvm_host.h struct kvm_vcpu_arch { · line 313
Quoted 1 line
arch/arm64/include/asm/kvm_host.h +1
-0+enum fp_state fp_type;Is it a state or a type? Some consistency would help. Also, what does
We can bikeshed this either way - the state currently stored is of a particular type. I'll probably go for type.
Then please do it consistently. At the moment, this is a bizarre mix of the two, and this is already hard enough to reason about this that we don't need extra complexity!
Quoted 5 lines
Quoted 4 lines
this represent? Your commit message keeps talking about the FP/SVE state for the host, but this is obviously a guest-related structure. How do the two relate?
Quoted 8 lines
The commit message talks about saving the floating point state in general which is something we do for both the host and the guest. The optimisation cases I am focusing on right now are more on host usage but the complexity with tracking that currently blocks them crosses both host and guest, indeed the biggest improvement overall is probably that tracking the guest state stops requiring us to fiddle with the host task's state which to me at least makes things clearer.
At least for the KVM part, I want a clear comment explaining what this tracks and how this is used, because at the moment, I'm only guessing. And I've had enough guessing with this code...
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 3/7 arm64/fpsimd: Have KVM explicitly say which FP registers to save
In reply to Mark BrownIn order to avoid needlessly saving and restoring the guest registers KVM relies on the host FPSMID code to save the guest registers when we context switch away from the guest. This is done by binding the KVM guest state to the CPU on top of the task state that was originally there, then carefully managing the TIF_SVE flag for the task to cause the host to save the full SVE state when needed regardless of the needs of the host task. This works well enough but isn't terribly direct about what is going on and makes it much more complicated to try to optimise what we're doing with the SVE register state.
Let's instead have KVM pass in the register state it wants saving when it binds to the CPU. We introduce a new FP_TYPE_TASK for use during normal task binding to indicate that we should base our decisions on the current task. In order to ease any future debugging that might be required this patch does not actually update any of the decision making about what to save, it merely starts tracking the new information and warns if the requested state is not what we would otherwise have decided to save.
--- arch/arm64/include/asm/fpsimd.h | 3 ++- arch/arm64/include/asm/processor.h | 1 + arch/arm64/kernel/fpsimd.c | 20 +++++++++++++++++++- arch/arm64/kvm/fpsimd.c | 9 ++++++++- 4 files changed, 30 insertions(+), 3 deletions(-)
arch/arm64/include/asm/fpsimd.h +2 -1
@@ -61,7 +61,8 @@ extern void fpsimd_kvm_prepare(void);extern void fpsimd_bind_state_to_cpu (struct user_fpsimd_state *state,void *sve_state,unsigned int sve_vl,void *za_state,unsigned int sme_vl, -u64 *svcr,enum fp_state *type); +u64 *svcr,enum fp_state *type, +enum fp_state to_save);extern void fpsimd_flush_task_state (struct task_struct *target);extern void fpsimd_save_and_flush_cpu_state (void );
arch/arm64/include/asm/processor.h +1 -0
@@ -123,6 +123,7 @@ enum vec_type { };enum fp_state { +FP_STATE_TASK ,/* Save based on current, invalid as fp_type */ FP_STATE_FPSIMD ,FP_STATE_SVE , };
arch/arm64/kernel/fpsimd.c +19 -1
@@ -126,6 +126,7 @@ struct fpsimd_last_state_struct {unsigned int sve_vl;unsigned int sme_vl;enum fp_state *fp_type; +enum fp_state to_save; }; static DEFINE_PER_CPU(struct fpsimd_last_state_struct, fpsimd_last_state);
@@ -459,6 +460,21 @@ static void fpsimd_save(void) vl = last->sve_vl; } +/* +* For now we're just validating that the requested state is +* consistent with what we'd otherwise work out. +*/ +switch (last->to_save) { +case FP_STATE_TASK : +break ; +case FP_STATE_FPSIMD : +WARN_ON_ONCE (save_sve_regs); +break ; +case FP_STATE_SVE : +WARN_ON_ONCE (!save_sve_regs); +break ; + } +if (system_supports_sme ()) {u64 *svcr = last->svcr;
@@ -1693,6 +1709,7 @@ static void fpsimd_bind_task_to_cpu(void) last->sme_vl =task_get_sme_vl (current); last->svcr = ¤t->thread.svcr; last->fp_type = ¤t->thread.fp_type; + last->to_save =FP_STATE_TASK ; current->thread.fpsimd_cpu =smp_processor_id ();/*
@@ -1717,7 +1734,7 @@ static void fpsimd_bind_task_to_cpu(void)void fpsimd_bind_state_to_cpu (struct user_fpsimd_state *st,void *sve_state,unsigned int sve_vl,void *za_state,unsigned int sme_vl,u64 *svcr, -enum fp_state *type) +enum fp_state *type,enum fp_state to_save) {struct fpsimd_last_state_struct *last =this_cpu_ptr (&fpsimd_last_state);
@@ -1732,6 +1749,7 @@ void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *st, void *sve_state, last->sve_vl = sve_vl; last->sme_vl = sme_vl; last->fp_type = type; + last->to_save = to_save; }/*
arch/arm64/kvm/fpsimd.c +8 -1
@@ -130,9 +130,16 @@ void kvm_arch_vcpu_ctxflush_fp(struct kvm_vcpu *vcpu)*/ void kvm_arch_vcpu_ctxsync_fp (struct kvm_vcpu *vcpu) { +enum fp_state fp_type; +WARN_ON_ONCE (!irqs_disabled ());if (vcpu->arch.fp_state ==FP_STATE_GUEST_OWNED ) { +if (vcpu_has_sve (vcpu)) + fp_type =FP_STATE_SVE ; +else + fp_type =FP_STATE_FPSIMD ; +/* * Currently we do not support SME guests so SVCR is * always 0 and we just need a variable to point to.
@@ -141,7 +148,7 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) vcpu->arch.sve_state, vcpu->arch.sve_max_vl,NULL ,0 , &vcpu->arch.svcr, - &vcpu->arch.fp_type); + &vcpu->arch.fp_type, fp_type);clear_thread_flag (TIF_FOREIGN_FPSTATE );update_thread_flag (TIF_SVE ,vcpu_has_sve (vcpu));
Signature · 6 lines
2.30.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 3/7] arm64/fpsimd: Have KVM explicitly say which FP registers to save
Quoted 46 lines · Mark Brown
In order to avoid needlessly saving and restoring the guest registers KVM relies on the host FPSMID code to save the guest registers when we context switch away from the guest. This is done by binding the KVM guest state to the CPU on top of the task state that was originally there, then carefully managing the
TIF_SVEflag for the task to cause the host to save the full SVE state when needed regardless of the needs of the host task. This works well enough but isn't terribly direct about what is going on and makes it much more complicated to try to optimise what we're doing with the SVE register state.Let's instead have KVM pass in the register state it wants saving when it binds to the CPU. We introduce a new
FP_TYPE_TASKfor use during normal task binding to indicate that we should base our decisions on the current task. In order to ease any future debugging that might be required this patch does not actually update any of the decision making about what to save, it merely starts tracking the new information and warns if the requested state is not what we would otherwise have decided to save.Signed-off-by Mark Brown <[email protected]>--- arch/arm64/include/asm/fpsimd.h | 3 ++- arch/arm64/include/asm/processor.h | 1 + arch/arm64/kernel/fpsimd.c | 20 +++++++++++++++++++- arch/arm64/kvm/fpsimd.c | 9 ++++++++- 4 files changed, 30 insertions(+), 3 deletions(-)arch/arm64/include/asm/fpsimd.h +2
-1@@ -61,7 +61,8 @@ extern void fpsimd_kvm_prepare(void);extern void fpsimd_bind_state_to_cpu (struct user_fpsimd_state *state,void *sve_state,unsigned int sve_vl,void *za_state,unsigned int sme_vl, -u64 *svcr,enum fp_state *type); +u64 *svcr,enum fp_state *type, +enum fp_state to_save);extern void fpsimd_flush_task_state (struct task_struct *target);extern void fpsimd_save_and_flush_cpu_state (void );arch/arm64/include/asm/processor.h +0
-0
arch/arm64/include/asm/processor.h enum vec_type { · line 126
Quoted 4 lines
arch/arm64/include/asm/processor.h +1
-0@@ -123,6 +123,7 @@ enum vec_type { };enum fp_state { +FP_STATE_TASK ,/* Save based on current, invalid as fp_type */
How is that related to the FP_TYPE_TASK in the commit message? What does this 'invalid as fp_type' mean?
Quoted 16 lines
arch/arm64/include/asm/processor.h +0
-0FP_STATE_FPSIMD , FP_STATE_SVE, };arch/arm64/kernel/fpsimd.c +1
-0@@ -126,6 +126,7 @@ struct fpsimd_last_state_struct {unsigned int sve_vl;unsigned int sme_vl;enum fp_state *fp_type; +enum fp_state to_save; }; static DEFINE_PER_CPU(struct fpsimd_last_state_struct, fpsimd_last_state);
arch/arm64/kernel/fpsimd.c static void fpsimd_save(void) · line 465
Quoted 6 lines
arch/arm64/kernel/fpsimd.c +3
-0@@ -459,6 +460,21 @@ static void fpsimd_save(void) vl = last->sve_vl; } +/* +* For now we're just validating that the requested state is +* consistent with what we'd otherwise work out.
Nit: work out? or worked out? the "we'd" doesn't help disambiguate it for a non-native speaker.
Quoted 24 lines
arch/arm64/kernel/fpsimd.c +13
-0+*/ +switch (last->to_save) { +case FP_STATE_TASK : +break ; +case FP_STATE_FPSIMD : +WARN_ON_ONCE (save_sve_regs); +break ; +case FP_STATE_SVE : +WARN_ON_ONCE (!save_sve_regs); +break ; + } +if (system_supports_sme ()) {u64 *svcr = last->svcr;@@ -1693,6 +1709,7 @@ static void fpsimd_bind_task_to_cpu(void) last->sme_vl =task_get_sme_vl (current); last->svcr = ¤t->thread.svcr; last->fp_type = ¤t->thread.fp_type; + last->to_save =FP_STATE_TASK ; current->thread.fpsimd_cpu =smp_processor_id ();/*
arch/arm64/kernel/fpsimd.c static void fpsimd_bind_task_to_cpu(void) · line 1737
Quoted 5 lines
arch/arm64/kernel/fpsimd.c +1
-1@@ -1717,7 +1734,7 @@ static void fpsimd_bind_task_to_cpu(void)void fpsimd_bind_state_to_cpu (struct user_fpsimd_state *st,void *sve_state,unsigned int sve_vl,void *za_state,unsigned int sme_vl,u64 *svcr, -enum fp_state *type) +enum fp_state *type,enum fp_state to_save)
OK, how many discrete arguments are we going to pass to this function, which most of them are part the vcpu structure? It really feels like what you want is a getter for the per-cpu structure, and let the KVM code do the actual business. If this function was supposed to provide some level of abstraction, well, it's a fail.
Quoted 16 lines
arch/arm64/kernel/fpsimd.c +1
-0{struct fpsimd_last_state_struct *last =this_cpu_ptr (&fpsimd_last_state);@@ -1732,6 +1749,7 @@ void fpsimd_bind_state_to_cpu(struct user_fpsimd_state *st, void *sve_state, last->sve_vl = sve_vl; last->sme_vl = sme_vl; last->fp_type = type; + last->to_save = to_save; }/* arch/arm64/kvm/fpsimd.c +0
-0
arch/arm64/kvm/fpsimd.c void kvm_arch_vcpu_ctxflush_fp(struct kvm_vcpu *vcpu) · line 139
Quoted 10 lines
arch/arm64/kvm/fpsimd.c +4
-0@@ -130,9 +130,16 @@ void kvm_arch_vcpu_ctxflush_fp(struct kvm_vcpu *vcpu)*/ void kvm_arch_vcpu_ctxsync_fp (struct kvm_vcpu *vcpu) { +enum fp_state fp_type; +WARN_ON_ONCE (!irqs_disabled ());if (vcpu->arch.fp_state ==FP_STATE_GUEST_OWNED ) { +if (vcpu_has_sve (vcpu)) + fp_type =FP_STATE_SVE ;
Eventually, I'd like to relax this, and start tracking the actual use of the guest rather than assuming that SVE guest use SVE at all times (odds are they won't).
I hope this series still leaves us with this option.
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 3/7] arm64/fpsimd: Have KVM explicitly say which FP registers to save
Quoted 7 lines · Marc Zyngier
arch/arm64/include/asm/processor.h enum vec_type { · line 126
Quoted 2 lines · Mark Brown
enum
fp_state{arch/arm64/include/asm/processor.h +1
-0+ FP_STATE_TASK, /* Save based on current, invalid as fp_type */How is that related to the
FP_TYPE_TASKin the commit message? What
TYPE in the commit message should be STATE.
Quoted 1 line
does this 'invalid as
fp_type' mean?
It means that using FP_STATE_TASK as a value for the fp_type member of the task struck recording what type of state is currently stored for the task is not valid, one of the other two values representing what was actually saved must be chosen.
Quoted 6 lines
arch/arm64/kernel/fpsimd.c static void fpsimd_save(void) · line 465
Quoted 3 lines
arch/arm64/kernel/fpsimd.c +3
-0+/* +* For now we're just validating that the requested state is +* consistent with what we'd otherwise work out. Nit: work out? or worked out? the "we'd" doesn't help disambiguate it for a non-native speaker.
we'd == we would so work out to match the tense.
Quoted 11 lines
arch/arm64/kernel/fpsimd.c static void fpsimd_bind_task_to_cpu(void) · line 1737
Quoted 5 lines
void fpsimd_bind_state_to_cpu(struct
user_fpsimd_state*st, void*sve_state,arch/arm64/kernel/fpsimd.c +1
-1unsigned int sve_vl, void *za_state, unsigned int sme_vl, u64 *svcr, - enum fp_state *type) + enum fp_state *type, enum fp_state to_save)OK, how many discrete arguments are we going to pass to this function, which most of them are part the vcpu structure? It really feels like what you want is a getter for the per-cpu structure, and let the KVM code do the actual business. If this function was supposed to provide some level of abstraction, well, it's a fail.
I agree that this is not an ideal interface, I am merely following the previously chosen idiom since I haven't been able to figure out why we were doing it in the first place and with a lot of these things it turns out that there's some actual reason.
It's not even like fpsimd_bind_task_to_cpu() has ever been written in terms of this function, there's two parallel implementations. My best guess was that it was some combination of not peering at KVM internals and keeping struct fpsimd_last_state_struct internal to fpsimd.c (since we're effectively just passing one of those in in a more verbose form) but never anything solid enough to be sure.
Quoted 14 lines
arch/arm64/kvm/fpsimd.c void kvm_arch_vcpu_ctxflush_fp(struct kvm_vcpu *vcpu) · line 139
Quoted 9 lines
void
kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu){arch/arm64/kvm/fpsimd.c +4
-0+enum fp_state fp_type; +WARN_ON_ONCE (!irqs_disabled ());if (vcpu->arch.fp_state ==FP_STATE_GUEST_OWNED ) { +if (vcpu_has_sve (vcpu)) + fp_type =FP_STATE_SVE ;Eventually, I'd like to relax this, and start tracking the actual use of the guest rather than assuming that SVE guest use SVE at all times (odds are they won't).
Quoted 1 line
I hope this series still leaves us with this option.
Yes, it probably makes it more tractable with KVM being able to just say what type of state it wants to save so there's less to take care of syncing with the host task so the code is a lot more direct - it will just be a case of setting the desired fp_type whenever a decision is made about what state type to save.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 3/7] arm64/fpsimd: Have KVM explicitly say which FP registers to save
Quoted 11 lines · Mark Brown
[1 <text/plain; us-ascii (quoted-printable)>]
Quoted 7 lines · Marc Zyngier
arch/arm64/include/asm/processor.h enum vec_type { · line 126
Quoted 2 lines · Mark Brown
enum
fp_state{arch/arm64/include/asm/processor.h +1
-0+ FP_STATE_TASK, /* Save based on current, invalid as fp_type */How is that related to the
FP_TYPE_TASKin the commit message? What
Quoted 8 lines
TYPE in the commit message should be STATE.
Quoted 1 line
does this 'invalid as
fp_type' mean?It means that using
FP_STATE_TASKas a value for thefp_typemember of the task struck recording what type of state is currently stored for the task is not valid, one of the other two values representing what was actually saved must be chosen.
Then this definitely represents something else, and shouldn't be a state or a type, whatever you decide to call it in the end. There is the state of the FP/SVE unit, and what some piece of SW wants to save. They match in some cases, and differ in other (the TASK value). I'd rather you encode them as them as different types to lift the ambiguity.
Quoted 22 lines
Quoted 7 lines
arch/arm64/kernel/fpsimd.c static void fpsimd_save(void) · line 465
Quoted 4 lines
arch/arm64/kernel/fpsimd.c +3
-0+/* +* For now we're just validating that the requested state is +* consistent with what we'd otherwise work out. Nit: work out? or worked out? the "we'd" doesn't help disambiguate it for a non-native speaker.
we'd == we would so work out to match the tense.
Quoted 11 lines
arch/arm64/kernel/fpsimd.c static void fpsimd_bind_task_to_cpu(void) · line 1737
Quoted 5 lines
void fpsimd_bind_state_to_cpu(struct
user_fpsimd_state*st, void*sve_state,arch/arm64/kernel/fpsimd.c +1
-1unsigned int sve_vl, void *za_state, unsigned int sme_vl, u64 *svcr, - enum fp_state *type) + enum fp_state *type, enum fp_state to_save)OK, how many discrete arguments are we going to pass to this function, which most of them are part the vcpu structure? It really feels like what you want is a getter for the per-cpu structure, and let the KVM code do the actual business. If this function was supposed to provide some level of abstraction, well, it's a fail.
Quoted 4 lines
I agree that this is not an ideal interface, I am merely following the previously chosen idiom since I haven't been able to figure out why we were doing it in the first place and with a lot of these things it turns out that there's some actual reason.
Huh. If we're changing anything around this code, we'd better understand what we are doing...
Quoted 7 lines
It's not even like
fpsimd_bind_task_to_cpu()has ever been written in terms of this function, there's two parallel implementations. My best guess was that it was some combination of not peering at KVM internals and keeping structfpsimd_last_state_structinternal tofpsimd.c(since we're effectively just passing one of those in in a more verbose form) but never anything solid enough to be sure.
Up to you, but adding extra parameters to this function really feels like the wrong thing to do.
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 3/7] arm64/fpsimd: Have KVM explicitly say which FP registers to save
Quoted 7 lines · Marc Zyngier
Quoted 4 lines · Mark Brown
It means that using
FP_STATE_TASKas a value for thefp_typemember of the task struck recording what type of state is currently stored for the task is not valid, one of the other two values representing what was actually saved must be chosen.
Quoted 6 lines
Then this definitely represents something else, and shouldn't be a state or a type, whatever you decide to call it in the end. There is the state of the FP/SVE unit, and what some piece of SW wants to save. They match in some cases, and differ in other (the TASK value). I'd rather you encode them as them as different types to lift the ambiguity.
I did try the other way as well and found it was creating it's own problems - you end up with two almost identical enums which need to be separately named in some meaningful yet not excessively verbose fashion and have to remember which one to use in which context.
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
OK, how many discrete arguments are we going to pass to this function, which most of them are part the vcpu structure? It really feels like what you want is a getter for the per-cpu structure, and let the KVM code do the actual business. If this function was supposed to provide some level of abstraction, well, it's a fail.
Quoted 7 lines
Quoted 4 lines
I agree that this is not an ideal interface, I am merely following the previously chosen idiom since I haven't been able to figure out why we were doing it in the first place and with a lot of these things it turns out that there's some actual reason.
Huh. If we're changing anything around this code, we'd better understand what we are doing...
I'm confident I understand well enough what it's doing, I'm just really unclear on why it's written the way it is.
Quoted 10 lines
Quoted 7 lines
It's not even like
fpsimd_bind_task_to_cpu()has ever been written in terms of this function, there's two parallel implementations. My best guess was that it was some combination of not peering at KVM internals and keeping structfpsimd_last_state_structinternal tofpsimd.c(since we're effectively just passing one of those in in a more verbose form) but never anything solid enough to be sure.Up to you, but adding extra parameters to this function really feels like the wrong thing to do.
Indeed, I'm not a big fan myself. I also don't want to get this and some other work I'm doing tied up on stylistic improvements to what's already there which seem likely introduce additional iterations - I'd rather loop back on this as a separate thing later.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 4/7 arm64/fpsimd: Stop using TIF_SVE to manage register saving in KVM
In reply to Mark BrownNow that we are explicitly telling the host FP code which register state it needs to save we can remove the manipulation of TIF_SVE from the KVM code, simplifying it and allowing us to optimise our handling of normal tasks. Remove the manipulation of TIF_SVE from KVM and instead rely on to_save to ensure we save the correct data for it.
--- arch/arm64/kernel/fpsimd.c | 22 ++++------------------ arch/arm64/kvm/fpsimd.c | 3 --- 2 files changed, 4 insertions(+), 21 deletions(-)
arch/arm64/kernel/fpsimd.c +4 -18
@@ -436,8 +436,8 @@ static void task_fpsimd_load(void)* last, if KVM is involved this may be the guest VM context rather * than the host thread for the VM pointed to by current. This means * that we must always reference the state storage via last rather -* than via current, other than the TIF_ flags which KVM will -* carefully maintain for us. +* than via current, if we are saving KVM state then it will have +* ensured that the type of registers to save is set in last->to_save. */ static void fpsimd_save (void ) {
@@ -454,27 +454,13 @@ static void fpsimd_save(void)if (test_thread_flag (TIF_FOREIGN_FPSTATE ))return ; -if (test_thread_flag (TIF_SVE )) { +if ((last->to_save ==FP_STATE_TASK &&test_thread_flag (TIF_SVE )) || + last->to_save ==FP_STATE_SVE ) { save_sve_regs = true; save_ffr = true; vl = last->sve_vl; } -/* -* For now we're just validating that the requested state is -* consistent with what we'd otherwise work out. -*/ -switch (last->to_save) { -case FP_STATE_TASK : -break ; -case FP_STATE_FPSIMD : -WARN_ON_ONCE (save_sve_regs); -break ; -case FP_STATE_SVE : -WARN_ON_ONCE (!save_sve_regs); -break ; - } -if (system_supports_sme ()) {u64 *svcr = last->svcr;
arch/arm64/kvm/fpsimd.c +0 -3
@@ -151,7 +151,6 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) &vcpu->arch.fp_type, fp_type);clear_thread_flag (TIF_FOREIGN_FPSTATE ); -update_thread_flag (TIF_SVE ,vcpu_has_sve (vcpu)); } }
@@ -208,7 +207,5 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)sysreg_clear_set (CPACR_EL1 ,CPACR_EL1_ZEN_EL0EN ,0 ); } -update_thread_flag (TIF_SVE ,0 ); -local_irq_restore (flags); }
Signature · 6 lines
2.30.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 4/7] arm64/fpsimd: Stop using TIF_SVE to manage register saving in KVM
Quoted 45 lines · Mark Brown
Now that we are explicitly telling the host FP code which register state it needs to save we can remove the manipulation of
TIF_SVEfrom the KVM code, simplifying it and allowing us to optimise our handling of normal tasks. Remove the manipulation ofTIF_SVEfrom KVM and instead rely onto_saveto ensure we save the correct data for it.Signed-off-by Mark Brown <[email protected]>--- arch/arm64/kernel/fpsimd.c | 22 ++++------------------ arch/arm64/kvm/fpsimd.c | 3 --- 2 files changed, 4 insertions(+), 21 deletions(-)arch/arm64/kernel/fpsimd.c +4
-8@@ -436,8 +436,8 @@ static void task_fpsimd_load(void)* last, if KVM is involved this may be the guest VM context rather * than the host thread for the VM pointed to by current. This means * that we must always reference the state storage via last rather -* than via current, other than the TIF_ flags which KVM will -* carefully maintain for us. +* than via current, if we are saving KVM state then it will have +* ensured that the type of registers to save is set in last->to_save. */ static void fpsimd_save (void ) {@@ -454,27 +454,13 @@ static void fpsimd_save(void)if (test_thread_flag (TIF_FOREIGN_FPSTATE ))return ; -if (test_thread_flag (TIF_SVE )) { +if ((last->to_save ==FP_STATE_TASK &&test_thread_flag (TIF_SVE )) || + last->to_save ==FP_STATE_SVE ) { save_sve_regs = true; save_ffr = true; vl = last->sve_vl; } -/* -* For now we're just validating that the requested state is -* consistent with what we'd otherwise work out. -*/ -switch (last->to_save) {
arch/arm64/kernel/fpsimd.c static void fpsimd_save(void) · line 477
Quoted 10 lines
arch/arm64/kernel/fpsimd.c +0
-10-case FP_STATE_TASK : -break ; -case FP_STATE_FPSIMD : -WARN_ON_ONCE (save_sve_regs); -break ; -case FP_STATE_SVE : -WARN_ON_ONCE (!save_sve_regs); -break ; - } -
Given how short-lived this code is, consider dropping it altogether. Actually, the previous patch would make a lot more sense if it was merged with this one.
Quoted 16 lines
arch/arm64/kernel/fpsimd.c +0
-0if (system_supports_sme ()) {u64 *svcr = last->svcr;arch/arm64/kvm/fpsimd.c +0
-1@@ -151,7 +151,6 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) &vcpu->arch.fp_type, fp_type);clear_thread_flag (TIF_FOREIGN_FPSTATE ); -update_thread_flag (TIF_SVE ,vcpu_has_sve (vcpu)); } }
arch/arm64/kvm/fpsimd.c void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) · line 211
Quoted 7 lines
arch/arm64/kvm/fpsimd.c +0
-2@@ -208,7 +207,5 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)sysreg_clear_set (CPACR_EL1 ,CPACR_EL1_ZEN_EL0EN ,0 ); } -update_thread_flag (TIF_SVE ,0 ); -local_irq_restore (flags); }
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 4/7] arm64/fpsimd: Stop using TIF_SVE to manage register saving in KVM
Quoted 16 lines · Marc Zyngier
arch/arm64/kernel/fpsimd.c static void fpsimd_save(void) · line 476
Quoted 10 lines · Mark Brown
arch/arm64/kernel/fpsimd.c +0
-10-switch (last->to_save) { -case FP_STATE_TASK : -break ; -case FP_STATE_FPSIMD : -WARN_ON_ONCE (save_sve_regs); -break ; -case FP_STATE_SVE : -WARN_ON_ONCE (!save_sve_regs); -break ; - }Given how short-lived this code is, consider dropping it altogether. Actually, the previous patch would make a lot more sense if it was merged with this one.
My thinking here is to introduce the state tracking and the behaviour change separately to make it easier to unpick things if anything goes wrong, it means that the behaviour change is in clearly isolated patches separate to the more wide spread changes to behaviour. The early patches make it more explicit what we are currently doing, the later ones do new things.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 5/7 arm64/fpsimd: Load FP state based on recorded data type
In reply to Mark BrownNow that we are recording the type of floating point register state we are saving when we save it we can use that information when we load to decide which register state is required and bring the TIF_SVE state into sync with the loaded register state.
The SME state detauls are already recorded directly in the saved SVCR and handled based on the information there.
Since we are not changing any of the save paths there should be no functional change from this patch, further patches will make use of this to optimise and clarify the code.
--- arch/arm64/kernel/fpsimd.c | 39 ++++++++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 8 deletions(-)
arch/arm64/kernel/fpsimd.c +31 -8
@@ -392,11 +392,36 @@ static void task_fpsimd_load(void)WARN_ON (!system_supports_fpsimd ());WARN_ON (!have_cpu_fpsimd_context ()); -/* Check if we should restore SVE first */ -if (IS_ENABLED (CONFIG_ARM64_SVE ) &&test_thread_flag (TIF_SVE )) { -sve_set_vq (sve_vq_from_vl (task_get_sve_vl (current)) -1 ); - restore_sve_regs = true; - restore_ffr = true; +if (system_supports_sve ()) { +switch (current->thread.fp_type) { +case FP_STATE_FPSIMD : +/* Stop tracking SVE for this task until next use. */ +if (test_and_clear_thread_flag (TIF_SVE )) +sve_user_disable (); +break ; +case FP_STATE_SVE : +if (!thread_sm_enabled (¤t->thread) && + !WARN_ON_ONCE (!test_and_set_thread_flag (TIF_SVE ))) +sve_user_enable (); + +if (test_thread_flag (TIF_SVE )) +sve_set_vq (sve_vq_from_vl (task_get_sve_vl (current)) -1 ); + + restore_sve_regs = true; + restore_ffr = true; +break ; +default : +/* +* This should never happen, we should always +* record what we saved when we save. We +* always at least have the memory allocated +* for FPSMID registers so try that and hope +* for the best. +*/ +WARN_ON_ONCE (1 ); +clear_thread_flag (TIF_SVE ); +break ; + } }/* Restore SME, override SVE register configuration if needed */
@@ -412,10 +437,8 @@ static void task_fpsimd_load(void)if (thread_za_enabled (¤t->thread))za_load_state (current->thread.za_state); -if (thread_sm_enabled (¤t->thread)) { - restore_sve_regs = true; +if (thread_sm_enabled (¤t->thread)) restore_ffr =system_supports_fa64 (); - } }if (restore_sve_regs) {
Signature · 6 lines
2.30.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Marc Zyngier
maintainerRe: [PATCH v3 5/7] arm64/fpsimd: Load FP state based on recorded data type
Quoted 5 lines · Mark Brown
Now that we are recording the type of floating point register state we are saving when we save it we can use that information when we load to decide which register state is required and bring the
TIF_SVEstate into sync with the loaded register state.
Really, this sentence makes zero sense to me. Please at least add some punctuation, because the only words that spring to mind here are "DOES NOT COMPUTE".
Quoted 45 lines
The SME state detauls are already recorded directly in the saved SVCR and handled based on the information there.
Since we are not changing any of the save paths there should be no functional change from this patch, further patches will make use of this to optimise and clarify the code.
Signed-off-by Mark Brown <[email protected]>--- arch/arm64/kernel/fpsimd.c | 39 ++++++++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 8 deletions(-)arch/arm64/kernel/fpsimd.c +19
-5@@ -392,11 +392,36 @@ static void task_fpsimd_load(void)WARN_ON (!system_supports_fpsimd ());WARN_ON (!have_cpu_fpsimd_context ()); -/* Check if we should restore SVE first */ -if (IS_ENABLED (CONFIG_ARM64_SVE ) &&test_thread_flag (TIF_SVE )) { -sve_set_vq (sve_vq_from_vl (task_get_sve_vl (current)) -1 ); - restore_sve_regs = true; - restore_ffr = true; +if (system_supports_sve ()) { +switch (current->thread.fp_type) { +case FP_STATE_FPSIMD : +/* Stop tracking SVE for this task until next use. */ +if (test_and_clear_thread_flag (TIF_SVE )) +sve_user_disable (); +break ; +case FP_STATE_SVE : +if (!thread_sm_enabled (¤t->thread) && + !WARN_ON_ONCE (!test_and_set_thread_flag (TIF_SVE ))) +sve_user_enable (); + +if (test_thread_flag (TIF_SVE )) +sve_set_vq (sve_vq_from_vl (task_get_sve_vl (current)) -1 ); + + restore_sve_regs = true; + restore_ffr = true; +break ; +default :
arch/arm64/kernel/fpsimd.c static void task_fpsimd_load(void) · line 423
Quoted 10 lines
arch/arm64/kernel/fpsimd.c +10
-0+/* +* This should never happen, we should always +* record what we saved when we save. We +* always at least have the memory allocated +* for FPSMID registers so try that and hope +* for the best. +*/ +WARN_ON_ONCE (1 ); +clear_thread_flag (TIF_SVE ); +break ;
What makes it impossible for FP_STATE_TASK to reach this point? If that's indeed an impossible case, please document it.
Quoted 6 lines
arch/arm64/kernel/fpsimd.c +1
-0+ } }/* Restore SME, override SVE register configuration if needed */ @@ -412,10 +437,8 @@ static void task_fpsimd_load(void)if (thread_za_enabled (¤t->thread))
arch/arm64/kernel/fpsimd.c static void task_fpsimd_load(void) · line 444
Quoted 10 lines
arch/arm64/kernel/fpsimd.c +1
-3za_load_state (current->thread.za_state); -if (thread_sm_enabled (¤t->thread)) { - restore_sve_regs = true; +if (thread_sm_enabled (¤t->thread)) restore_ffr =system_supports_fa64 (); - } }if (restore_sve_regs) {
Thanks,
M.
Signature · 5 lines
Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
Re: [PATCH v3 5/7] arm64/fpsimd: Load FP state based on recorded data type
Quoted 10 lines · Marc Zyngier
Quoted 4 lines · Mark Brown
Now that we are recording the type of floating point register state we are saving when we save it we can use that information when we load to decide which register state is required and bring the
TIF_SVEstate into sync with the loaded register state.Really, this sentence makes zero sense to me. Please at least add some punctuation, because the only words that spring to mind here are "DOES NOT COMPUTE".
I'll try to come up with something...
Quoted 6 lines
Quoted 6 lines
arch/arm64/kernel/fpsimd.c +6
-0+default : +/* +* This should never happen, we should always +* record what we saved when we save. We +* always at least have the memory allocated +* for FPSMID registers so try that and hope
Quoted 8 lines
arch/arm64/kernel/fpsimd.c static void task_fpsimd_load(void) · line 423
Quoted 5 lines
arch/arm64/kernel/fpsimd.c +5
-0+* for the best. +*/ +WARN_ON_ONCE (1 ); +clear_thread_flag (TIF_SVE ); +break ;What makes it impossible for
FP_STATE_TASKto reach this point? If that's indeed an impossible case, please document it.
That's what the "we should always record what we saved when we saved" is doing, and the comment in the header about it not being valid to record _TASK as a saved state. When we write the register state to memory we must always write either FPSIMD or SVE register values depending on which registers we saved state for. _TASK is not a meaningful state for stored register values.
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 6/7 arm64/fpsimd: SME no longer requires SVE register state
In reply to Mark BrownNow that we track the type of register state stored separately to tracking what is active in the task it is valid to have FPSIMD register state stored while in streaming mode so remove the special case handling for SME when setting FPSIMD register state.
--- arch/arm64/kernel/fpsimd.c | 3 +-- arch/arm64/kernel/ptrace.c | 2 -- 2 files changed, 1 insertion(+), 4 deletions(-)
arch/arm64/kernel/fpsimd.c +1 -2
@@ -811,8 +811,7 @@ void fpsimd_sync_to_sve(struct task_struct *task)*/ void sve_sync_to_fpsimd (struct task_struct *task) { -if (test_tsk_thread_flag (task,TIF_SVE ) || -thread_sm_enabled (&task->thread)) +if (task->thread.fp_type ==FP_STATE_SVE )sve_to_fpsimd (task); }
arch/arm64/kernel/ptrace.c +0 -2
@@ -892,8 +892,6 @@ static int sve_set_common(struct task_struct *target, ret =__fpr_set (target, regset, pos, count, kbuf, ubuf,SVE_PT_FPSIMD_OFFSET );clear_tsk_thread_flag (target,TIF_SVE ); -if (type ==ARM64_VEC_SME ) -fpsimd_force_sync_to_sve (target); target->thread.fp_type =FP_STATE_FPSIMD ; goto out; }
Signature · 6 lines
2.30.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
PATCH v3 7/7 arm64/sve: Leave SVE enabled on syscall if we don't context switch
In reply to Mark BrownThe syscall ABI says that the SVE register state not shared with FPSIMD may not be preserved on syscall, and this is the only mechanism we have in the ABI to stop tracking the extra SVE state for a process. Currently we do this unconditionally by means of disabling SVE for the process on syscall, causing userspace to take a trap to EL1 if it uses SVE again. These extra traps result in a noticeable overhead for using SVE instead of FPSIMD in some workloads, especially for simple syscalls where we can return directly to userspace and would not otherwise need to update the floating point registers. Tests with fp-pidbench show an approximately 70% overhead on a range of implementations when SVE is in use - while this is an extreme and entirely artificial benchmark it is clear that there is some useful room for improvement here.
Now that we have the ability to track the decision about what to save seprately to TIF_SVE we can improve things by leaving TIF_SVE enabled on syscall but only saving the FPSIMD registers if we are in a syscall. This means that if we need to restore the register state from memory (eg, after a context switch or kernel mode NEON) we will drop TIF_SVE and reenable traps for userspace but if we can just return to userspace then traps will remain disabled.
Since our current implementation has the effect of zeroing all the SVE register state not shared with FPSIMD on syscall we replace the disabling of TIF_SVE with a flush of the non-shared register state, this means that there is still some overhead for syscalls when SVE is in use but it is much reduced.
--- arch/arm64/kernel/fpsimd.c | 8 +++++++- arch/arm64/kernel/syscall.c | 19 +++++-------------- 2 files changed, 12 insertions(+), 15 deletions(-)
arch/arm64/kernel/fpsimd.c +7 -1
@@ -477,7 +477,13 @@ static void fpsimd_save(void)if (test_thread_flag (TIF_FOREIGN_FPSTATE ))return ; -if ((last->to_save ==FP_STATE_TASK &&test_thread_flag (TIF_SVE )) || +/* +* If a task is in a syscall the ABI allows us to only +* preserve the state shared with FPSIMD so don't bother +* saving the full SVE state in that case. +*/ +if ((last->to_save ==FP_STATE_TASK &&test_thread_flag (TIF_SVE ) && + !in_syscall (current_pt_regs ())) || last->to_save ==FP_STATE_SVE ) { save_sve_regs = true; save_ffr = true;
arch/arm64/kernel/syscall.c +5 -14
@@ -183,21 +183,12 @@ static inline void fp_user_discard(void)if (!system_supports_sve ())return ; -/* -* If SME is not active then disable SVE, the registers will -* be cleared when userspace next attempts to access them and -* we do not need to track the SVE register state until then. -*/ -clear_thread_flag (TIF_SVE ); +if (test_thread_flag (TIF_SVE )) { +unsigned int sve_vq_minus_one; -/* -* task_fpsimd_load() won't be called to update CPACR_EL1 in -* ret_to_user unless TIF_FOREIGN_FPSTATE is still set, which only -* happens if a context switch or kernel_neon_begin() or context -* modification (sigreturn, ptrace) intervenes. -* So, ensure that CPACR_EL1 is already correct for the fast-path case. -*/ -sve_user_disable (); + sve_vq_minus_one =sve_vq_from_vl (task_get_sve_vl (current)) -1 ; +sve_flush_live (true, sve_vq_minus_one); + } }void do_el0_svc (struct pt_regs *regs)
Signature · 6 lines
2.30.2 _______________________________________________ kvmarm mailing list [email protected] https://lists.cs.columbia.edu/mailman/listinfo/kvmarm