arm64: Run kernel mode NEON with preemption enabled
Subsystems (2)
v2 cover arm64: Run kernel mode NEON with preemption enabled
From: Ard Biesheuvel <[email protected]>
Currently, kernel mode NEON (SIMD) support is implemented in a way that requires preemption to be disabled while the SIMD registers are live. The reason for this is that those registers are not in the set that is preserved/restored on exception entry/exit and context switch, as this would impact performance generally, even for workloads where kernel mode SIMD is not the bottleneck.
However, doing substantial work with preemption disabled is not great, as it affects scheduling latency, which is especially problematic for real-time use cases. So ideally, we should keep preemption enabled when we can, and find another way to ensure that this does not corrupt the NEON register state of in-kernel SIMD users.
This series implements a suggestion by Mark Rutland, and introduces a thread_info flag TIF_USING_KMODE_NEON, which indicates to the thread switch machinery that the task in question has live kernel mode SIMD state which needs to be preserved and restored. The space needed for this is allocated in thread_struct. (*)
Given that currently, we run kernel mode NEON with softirqs disabled (to avoid the need for preserving kernel mode NEON context belonging to task context while the SIMD unit is being used by code running in softirq context), just removing the preempt_disable/enable calls is not sufficient, and we also need to leave softirqs enabled. This means that we may need to preserve kernel mode NEON state not only on a context switch, but also when code running in softirq context takes ownership of the SIMD unit, but this is straight-forward once we add the scratch space to thread_struct.
(*) We might decide to allocate this space (~512 bytes) dynamically, if the thread_struct memory footprint causes issues. However, we should also explore doing the same for the user space FPSIMD state, as kernel threads never return to user space and have no need for this allocation.
v2:
- tweak some commit logs for clarity
- integrate with the existing lazy restore logic
- add Mark's R-b to patch #1
Ard Biesheuvel (4):
arm64: fpsimd: Drop unneeded 'busy' flag
arm64: fpsimd: Preserve/restore kernel mode NEON at context switch
arm64: fpsimd: Implement lazy restore for kernel mode FPSIMD
arm64: crypto: Remove conditional yield logic
arch/arm64/crypto/aes-glue.c | 21 +-- arch/arm64/crypto/aes-modes.S | 2 - arch/arm64/crypto/sha1-ce-core.S | 2 - arch/arm64/crypto/sha1-ce-glue.c | 19 +-- arch/arm64/crypto/sha2-ce-core.S | 2 - arch/arm64/crypto/sha2-ce-glue.c | 19 +-- arch/arm64/crypto/sha3-ce-core.S | 4 +- arch/arm64/crypto/sha3-ce-glue.c | 14 +- arch/arm64/crypto/sha512-ce-core.S | 2 - arch/arm64/crypto/sha512-ce-glue.c | 16 +- arch/arm64/include/asm/assembler.h | 29 ---- arch/arm64/include/asm/processor.h | 3 + arch/arm64/include/asm/simd.h | 11 +- arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/kernel/asm-offsets.c | 4 - arch/arm64/kernel/fpsimd.c | 162 +++++++++++++------- 16 files changed, 138 insertions(+), 173 deletions(-)
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 1/4 arm64: fpsimd: Drop unneeded 'busy' flag
From: Ard Biesheuvel <[email protected]>
Kernel mode NEON will preserve the user mode FPSIMD state by saving it into the task struct before clobbering the registers. In order to avoid the need for preserving kernel mode state too, we disallow nested use of kernel mode NEON, i..e, use in softirq context while the interrupted task context was using kernel mode NEON too.
Originally, this policy was implemented using a per-CPU flag which was exposed via may_use_simd(), requiring the users of the kernel mode NEON to deal with the possibility that it might return false, and having NEON and non-NEON code paths. This policy was changed by commit 13150149aa6ded1 ("arm64: fpsimd: run kernel mode NEON with softirqs disabled"), and now, softirq processing is disabled entirely instead, and so may_use_simd() can never fail when called from task or softirq context.
This means we can drop the fpsimd_context_busy flag entirely, and instead, ensure that we disable softirq processing in places where we formerly relied on the flag for preventing races in the FPSIMD preserve routines.
--- arch/arm64/include/asm/simd.h | 11 +--- arch/arm64/kernel/fpsimd.c | 53 +++++--------------- 2 files changed, 13 insertions(+), 51 deletions(-)
arch/arm64/include/asm/simd.h +1 -10
@@ -12,8 +12,6 @@#include <linux/preempt.h> #include <linux/types.h> -DECLARE_PER_CPU (bool, fpsimd_context_busy); -#ifdef CONFIG_KERNEL_MODE_NEON /*
@@ -28,17 +26,10 @@ static __must_check inline bool may_use_simd(void)/* * We must make sure that the SVE has been initialized properly * before using the SIMD in kernel. -* fpsimd_context_busy is only set while preemption is disabled, -* and is clear whenever preemption is enabled. Since -* this_cpu_read() is atomic w.r.t. preemption, fpsimd_context_busy -* cannot change under our feet -- if it's set we cannot be -* migrated, and if it's clear we cannot be migrated to a CPU -* where it is set. */ return !WARN_ON (!system_capabilities_finalized ()) &&system_supports_fpsimd () && - !in_hardirq () && !irqs_disabled () && !in_nmi () && - !this_cpu_read(fpsimd_context_busy); + !in_hardirq() && !irqs_disabled() && !in_nmi(); }#else /* ! CONFIG_KERNEL_MODE_NEON */
arch/arm64/kernel/fpsimd.c +12 -41
123 lines not on this page · raw message
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 2/4 arm64: fpsimd: Preserve/restore kernel mode NEON at context switch
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Currently, the FPSIMD register file is not preserved and restored along with the general registers on exception entry/exit or context switch. For this reason, we disable preemption when enabling FPSIMD for kernel mode use in task context, and suspend the processing of softirqs so that there are no concurrent uses in the kernel. (Kernel mode FPSIMD may not be used at all in other contexts).
Disabling preemption while doing CPU intensive work on inputs of potentially unbounded size is bad for real-time performance, which is why we try and ensure that SIMD crypto code does not operate on more than ~4k at a time, which is an arbitrary limit and requires assembler code to implement efficiently.
We can avoid the need for disabling preemption if we can ensure that any in-kernel users of the NEON will not lose the FPSIMD register state across a context switch. And given that disabling softirqs implicitly disables preemption as well, we will also have to ensure that a softirq that runs code using FPSIMD can safely interrupt an in-kernel user.
So introduce a thread_info flag TIF_USING_KMODE_FPSIMD, and modify the context switch hook for FPSIMD to preserve and restore the kernel mode FPSIMD to/from struct thread_struct when it is set. This avoids any scheduling blackouts due to prolonged use of FPSIMD in kernel mode, without the need for manual yielding.
In order to support softirq processing while FPSIMD is being used in kernel task context, use the same flag to decide whether the kernel mode FPSIMD state needs to be preserved and restored before allowing FPSIMD to be used in softirq context.
--- arch/arm64/include/asm/processor.h | 2 + arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/kernel/fpsimd.c | 91 ++++++++++++++++---- 3 files changed, 76 insertions(+), 18 deletions(-)
arch/arm64/include/asm/processor.h +2 -0
@@ -167,6 +167,8 @@ struct thread_struct {unsigned long fault_address;/* fault info */ unsigned long fault_code;/* ESR_EL1 value */ struct debug_info debug;/* debugging */ + +struct user_fpsimd_state kmode_fpsimd_state;#ifdef CONFIG_ARM64_PTR_AUTH struct ptrauth_keys_user keys_user;#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL
arch/arm64/include/asm/thread_info.h +1 -0
@@ -80,6 +80,7 @@ void arch_setup_new_exec(void);#define TIF_TAGGED_ADDR 26/* Allow tagged user addresses */ #define TIF_SME 27/* SME in use */ #define TIF_SME_VL_INHERIT 28/* Inherit SME vl_onexec across exec */ +#define TIF_USING_KMODE_FPSIMD 29/* Task is in a kernel mode FPSIMD section */ #define _TIF_SIGPENDING (1 << TIF_SIGPENDING)#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)
arch/arm64/kernel/fpsimd.c +73 -18
156 lines not on this page · raw message
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 2/4] arm64: fpsimd: Preserve/restore kernel mode NEON at context switch
Quoted 10 lines · Ard Biesheuvel
arch/arm64/kernel/fpsimd.c +2
-8-/* -* Fix up TIF_FOREIGN_FPSTATE to correctly describe next's -* state. For kernel threads, FPSIMD registers are never loaded -* and wrong_task and wrong_cpu will always be true. -*/ - wrong_task =__this_cpu_read (fpsimd_last_state.st) != - &next->thread.uw.fpsimd_state; - wrong_cpu = next->thread.fpsimd_cpu !=smp_processor_id (); +if (test_tsk_thread_flag (next,TIF_USING_KMODE_FPSIMD )) { +fpsimd_load_kernel_state (next);
arch/arm64/kernel/fpsimd.c void fpsimd_thread_switch(struct task_struct *next) · line 1513
Quoted 10 lines
arch/arm64/kernel/fpsimd.c +10
-0+set_tsk_thread_flag (next,TIF_FOREIGN_FPSTATE ); + }else { +/* +* Fix up TIF_FOREIGN_FPSTATE to correctly describe next's +* state. For kernel threads, FPSIMD registers are never +* loaded and wrong_task and wrong_cpu will always be true. +*/ + wrong_task =__this_cpu_read (fpsimd_last_state.st) != + &next->thread.uw.fpsimd_state; + wrong_cpu = next->thread.fpsimd_cpu !=smp_processor_id ();
The "For kernel threads, FPSIMD registers are never loaded" part is outdated now, right?
- Eric
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 2/4] arm64: fpsimd: Preserve/restore kernel mode NEON at context switch
Quoted 23 lines · Eric Biggers
Quoted 10 lines · Ard Biesheuvel
arch/arm64/kernel/fpsimd.c +2
-8-/* -* Fix up TIF_FOREIGN_FPSTATE to correctly describe next's -* state. For kernel threads, FPSIMD registers are never loaded -* and wrong_task and wrong_cpu will always be true. -*/ - wrong_task =__this_cpu_read (fpsimd_last_state.st) != - &next->thread.uw.fpsimd_state; - wrong_cpu = next->thread.fpsimd_cpu !=smp_processor_id (); +if (test_tsk_thread_flag (next,TIF_USING_KMODE_FPSIMD )) { +fpsimd_load_kernel_state (next);arch/arm64/kernel/fpsimd.c void fpsimd_thread_switch(struct task_struct *next) · line 1513
Quoted 10 lines
arch/arm64/kernel/fpsimd.c +10
-0+set_tsk_thread_flag (next,TIF_FOREIGN_FPSTATE ); + }else { +/* +* Fix up TIF_FOREIGN_FPSTATE to correctly describe next's +* state. For kernel threads, FPSIMD registers are never +* loaded and wrong_task and wrong_cpu will always be true. +*/ + wrong_task =__this_cpu_read (fpsimd_last_state.st) != + &next->thread.uw.fpsimd_state; + wrong_cpu = next->thread.fpsimd_cpu !=smp_processor_id ();
Quoted 3 lines
The "For kernel threads, FPSIMD registers are never loaded" part is outdated now, right?
Yeah, this needs a tweak to clarify that it refers to user mode FPSIMD state, which does not exist for kernel threads.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 2/4] arm64: fpsimd: Preserve/restore kernel mode NEON at context switch
In reply to Ard BiesheuvelQuoted 8 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Currently, the FPSIMD register file is not preserved and restored along with the general registers on exception entry/exit or context switch. For this reason, we disable preemption when enabling FPSIMD for kernel mode use in task context, and suspend the processing of softirqs so that there are no concurrent uses in the kernel. (Kernel mode FPSIMD may not be used at all in other contexts).
With Eric's comment addressed:
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 3/4 arm64: fpsimd: Implement lazy restore for kernel mode FPSIMD
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Now that kernel mode FPSIMD state is context switched along with other task state, we can enable the existing logic that keeps track of which task's FPSIMD state the CPU is holding in its registers. If it is the context of the task that we are switching to, we can elide the reload of the FPSIMD state from memory.
Note that we also need to check whether the FPSIMD state on this CPU is the most recent: if a task gets migrated away and back again, the state in memory may be more recent than the state in the CPU. So add another CPU id field to task_struct to keep track of this. (We could reuse the existing CPU id field used for user mode context, but that might result in user state to be discarded unnecessarily, given that two distinct CPUs could be holding the most recent user mode state and the most recent kernel mode state)
--- arch/arm64/include/asm/processor.h | 1 + arch/arm64/kernel/fpsimd.c | 18 ++++++++++++++++++ 2 files changed, 19 insertions(+)
arch/arm64/include/asm/processor.h +1 -0
@@ -169,6 +169,7 @@ struct thread_struct {struct debug_info debug;/* debugging */ struct user_fpsimd_state kmode_fpsimd_state; +unsigned int kmode_fpsimd_cpu;#ifdef CONFIG_ARM64_PTR_AUTH struct ptrauth_keys_user keys_user;#ifdef CONFIG_ARM64_PTR_AUTH_KERNEL
arch/arm64/kernel/fpsimd.c +18 -0
@@ -1476,12 +1476,30 @@ void do_fpsimd_exc(unsigned long esr, struct pt_regs *regs)static void fpsimd_load_kernel_state (struct task_struct *task) { +struct cpu_fp_state *last =this_cpu_ptr (&fpsimd_last_state); + +/* +* Elide the load if this CPU holds the most recent kernel mode +* FPSIMD context of the current task. +*/ +if (last->st == &task->thread.kmode_fpsimd_state && + task->thread.kmode_fpsimd_cpu ==smp_processor_id ()) +return ; +fpsimd_load_state (&task->thread.kmode_fpsimd_state); }static void fpsimd_save_kernel_state (struct task_struct *task) { +struct cpu_fp_state cpu_fp_state = { + .st = &task->thread.kmode_fpsimd_state, + .to_save =FP_STATE_FPSIMD , + }; +fpsimd_save_state (&task->thread.kmode_fpsimd_state); +fpsimd_bind_state_to_cpu (&cpu_fp_state); + + task->thread.kmode_fpsimd_cpu =smp_processor_id (); }void fpsimd_thread_switch (struct task_struct *next)
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v2 3/4] arm64: fpsimd: Implement lazy restore for kernel mode FPSIMD
Quoted 7 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Now that kernel mode FPSIMD state is context switched along with other task state, we can enable the existing logic that keeps track of which task's FPSIMD state the CPU is holding in its registers. If it is the context of the task that we are switching to, we can elide the reload of the FPSIMD state from memory.
Thanks for doing this:
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v2 4/4 arm64: crypto: Remove conditional yield logic
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Some classes of crypto algorithms (such as skciphers or aeads) have natural yield points, but SIMD based shashes yield the NEON unit manually to avoid causing scheduling blackouts when operating on large inputs.
This is no longer necessary now that kernel mode NEON runs with preemption enabled, so remove this logic from the crypto code.
--- arch/arm64/crypto/aes-glue.c | 21 +++++--------- arch/arm64/crypto/aes-modes.S | 2 -- arch/arm64/crypto/sha1-ce-core.S | 2 -- arch/arm64/crypto/sha1-ce-glue.c | 19 ++++--------- arch/arm64/crypto/sha2-ce-core.S | 2 -- arch/arm64/crypto/sha2-ce-glue.c | 19 ++++--------- arch/arm64/crypto/sha3-ce-core.S | 4 +-- arch/arm64/crypto/sha3-ce-glue.c | 14 ++++------ arch/arm64/crypto/sha512-ce-core.S | 2 -- arch/arm64/crypto/sha512-ce-glue.c | 16 ++++------- arch/arm64/include/asm/assembler.h | 29 -------------------- arch/arm64/kernel/asm-offsets.c | 4 --- 12 files changed, 30 insertions(+), 104 deletions(-)
arch/arm64/crypto/aes-glue.c +7 -14
@@ -109,9 +109,9 @@ asmlinkage void aes_essiv_cbc_decrypt(u8 out[], u8 const in[], u32 const rk1[], int rounds, int blocks, u8 iv[], u32 const rk2[]); -asmlinkage int aes_mac_update(u8 const in[], u32 const rk[], int rounds, -int blocks,u8 dg[],int enc_before, -int enc_after); +asmlinkage void aes_mac_update(u8 const in[], u32 const rk[], int rounds, +int blocks,u8 dg[],int enc_before, +int enc_after);struct crypto_aes_xts_ctx {struct crypto_aes_ctx key1;
@@ -880,17 +880,10 @@ static void mac_do_update(struct crypto_aes_ctx *ctx, u8 const in[], int blocks,int rounds =6 + ctx->key_length /4 ;if (crypto_simd_usable ()) { -int rem; - -do { -kernel_neon_begin (); - rem =aes_mac_update (in, ctx->key_enc, rounds, blocks, - dg, enc_before, enc_after); -kernel_neon_end (); - in += (blocks - rem) *AES_BLOCK_SIZE ; - blocks = rem; - enc_before =0 ; - }while (blocks); +kernel_neon_begin (); +aes_mac_update (in, ctx->key_enc, rounds, blocks, dg, + enc_before, enc_after); +kernel_neon_end (); }else {if (enc_before)aes_encrypt (ctx, dg, dg);
arch/arm64/crypto/aes-modes.S +0 -2
@@ -842,7 +842,6 @@ AES_FUNC_START(aes_mac_update) cbz w5, .Lmacout encrypt_block v0, w2, x1, x7, w8 st1 {v0.16b}, [x4] /* return dg */ - cond_yield .Lmacout, x7, x8 b .Lmacloop4x .Lmac1x: add w3, w3, #4
@@ -861,6 +860,5 @@ AES_FUNC_START(aes_mac_update) .Lmacout: st1 {v0.16b}, [x4] /* return dg */ - mov w0, w3 ret AES_FUNC_END(aes_mac_update)
arch/arm64/crypto/sha1-ce-core.S +0 -2
@@ -121,7 +121,6 @@ CPU_LE( rev32 v11.16b, v11.16b ) add dgav.4s, dgav.4s, dg0v.4s cbz w2, 2f - cond_yield 3f, x5, x6 b 0b /*
@@ -145,6 +144,5 @@ CPU_LE( rev32 v11.16b, v11.16b ) /* store new state */ 3: st1 {dgav.4s}, [x0] str dgb, [x0, #16] - mov w0, w2 ret SYM_FUNC_END(__sha1_ce_transform)
arch/arm64/crypto/sha1-ce-glue.c +6 -13
@@ -29,23 +29,16 @@ struct sha1_ce_state {extern const u32 sha1_ce_offsetof_count;extern const u32 sha1_ce_offsetof_finalize; -asmlinkage int __sha1_ce_transform(struct sha1_ce_state *sst, u8 const *src, -int blocks); +asmlinkage void __sha1_ce_transform(struct sha1_ce_state *sst, u8 const *src, +int blocks);static void sha1_ce_transform (struct sha1_state *sst,u8 const *src,int blocks) { -while (blocks) { -int rem; - -kernel_neon_begin (); - rem =__sha1_ce_transform (container_of (sst, - struct sha1_ce_state, - sst), src, blocks); -kernel_neon_end (); - src += (blocks - rem) *SHA1_BLOCK_SIZE ; - blocks = rem; - } +kernel_neon_begin (); + __sha1_ce_transform(container_of(sst, struct sha1_ce_state, sst), src, + blocks); +kernel_neon_end (); } const u32 sha1_ce_offsetof_count = offsetof(struct sha1_ce_state, sst.count);
arch/arm64/crypto/sha2-ce-core.S +0 -2
@@ -129,7 +129,6 @@ CPU_LE( rev32 v19.16b, v19.16b ) /* handled all input blocks? */ cbz w2, 2f - cond_yield 3f, x5, x6 b 0b /*
@@ -152,6 +151,5 @@ CPU_LE( rev32 v19.16b, v19.16b ) /* store new state */ 3: st1 {dgav.4s, dgbv.4s}, [x0] - mov w0, w2 ret SYM_FUNC_END(__sha256_ce_transform)
arch/arm64/crypto/sha2-ce-glue.c +6 -13
@@ -30,23 +30,16 @@ struct sha256_ce_state {extern const u32 sha256_ce_offsetof_count;extern const u32 sha256_ce_offsetof_finalize; -asmlinkage int __sha256_ce_transform(struct sha256_ce_state *sst, u8 const *src, -int blocks); +asmlinkage void __sha256_ce_transform(struct sha256_ce_state *sst, u8 const *src, +int blocks);static void sha256_ce_transform (struct sha256_state *sst,u8 const *src,int blocks) { -while (blocks) { -int rem; - -kernel_neon_begin (); - rem =__sha256_ce_transform (container_of (sst, - struct sha256_ce_state, - sst), src, blocks); -kernel_neon_end (); - src += (blocks - rem) *SHA256_BLOCK_SIZE ; - blocks = rem; - } +kernel_neon_begin (); + __sha256_ce_transform(container_of(sst, struct sha256_ce_state, sst), + src, blocks); +kernel_neon_end (); } const u32 sha256_ce_offsetof_count = offsetof(struct sha256_ce_state,
arch/arm64/crypto/sha3-ce-core.S +1 -3
@@ -184,18 +184,16 @@ SYM_FUNC_START(sha3_ce_transform) eor v0.16b, v0.16b, v31.16b cbnz w8, 3b - cond_yield 4f, x8, x9 cbnz w2, 0b /* save state */ -4: st1 { v0.1d- v3.1d}, [x0], #32 + st1 { v0.1d- v3.1d}, [x0], #32 st1 { v4.1d- v7.1d}, [x0], #32 st1 { v8.1d-v11.1d}, [x0], #32 st1 {v12.1d-v15.1d}, [x0], #32 st1 {v16.1d-v19.1d}, [x0], #32 st1 {v20.1d-v23.1d}, [x0], #32 st1 {v24.1d}, [x0] - mov w0, w2 ret SYM_FUNC_END(sha3_ce_transform)
arch/arm64/crypto/sha3-ce-glue.c +5 -9
@@ -28,8 +28,8 @@ MODULE_ALIAS_CRYPTO("sha3-256");MODULE_ALIAS_CRYPTO ("sha3-384" );MODULE_ALIAS_CRYPTO ("sha3-512" ); -asmlinkage int sha3_ce_transform(u64 *st, const u8 *data, int blocks, -int md_len); +asmlinkage void sha3_ce_transform(u64 *st, const u8 *data, int blocks, +int md_len);static int sha3_update (struct shash_desc *desc,const u8 *data,unsigned int len)
@@ -59,15 +59,11 @@ static int sha3_update(struct shash_desc *desc, const u8 *data, blocks = len / sctx->rsiz; len %= sctx->rsiz; -while (blocks) { -int rem; - +if (blocks) {kernel_neon_begin (); - rem =sha3_ce_transform (sctx->st, data, blocks, - digest_size); +sha3_ce_transform (sctx->st, data, blocks, digest_size);kernel_neon_end (); - data += (blocks - rem) * sctx->rsiz; - blocks = rem; + data += blocks * sctx->rsiz; } }
arch/arm64/crypto/sha512-ce-core.S +0 -2
@@ -195,12 +195,10 @@ CPU_LE( rev64 v19.16b, v19.16b ) add v10.2d, v10.2d, v2.2d add v11.2d, v11.2d, v3.2d - cond_yield 3f, x4, x5 /* handled all input blocks? */ cbnz w2, 0b /* store new state */ 3: st1 {v8.2d-v11.2d}, [x0] - mov w0, w2 ret SYM_FUNC_END(__sha512_ce_transform)
arch/arm64/crypto/sha512-ce-glue.c +5 -11
@@ -26,23 +26,17 @@ MODULE_LICENSE("GPL v2");MODULE_ALIAS_CRYPTO ("sha384" );MODULE_ALIAS_CRYPTO ("sha512" ); -asmlinkage int __sha512_ce_transform(struct sha512_state *sst, u8 const *src, -int blocks); +asmlinkage void __sha512_ce_transform(struct sha512_state *sst, u8 const *src, +int blocks); asmlinkage void sha512_block_data_order(u64 *digest, u8 const *src, int blocks);static void sha512_ce_transform (struct sha512_state *sst,u8 const *src,int blocks) { -while (blocks) { -int rem; - -kernel_neon_begin (); - rem =__sha512_ce_transform (sst, src, blocks); -kernel_neon_end (); - src += (blocks - rem) *SHA512_BLOCK_SIZE ; - blocks = rem; - } +kernel_neon_begin (); +__sha512_ce_transform (sst, src, blocks); +kernel_neon_end (); } static void sha512_arm64_transform(struct sha512_state *sst, u8 const *src,
arch/arm64/include/asm/assembler.h +0 -29
@@ -759,35 +759,6 @@ alternative_endif set_sctlr sctlr_el2, \reg .endm -/* -* Check whether preempt/bh-disabled asm code should yield as soon as -* it is able. This is the case if we are currently running in task -* context, and either a softirq is pending, or the TIF_NEED_RESCHED -* flag is set and re-enabling preemption a single time would result in -* a preempt count of zero. (Note that the TIF_NEED_RESCHED flag is -* stored negated in the top word of the thread_info::preempt_count -* field) -*/ - .macro cond_yield, lbl:req, tmp:req, tmp2:req - get_current_task \tmp - ldr \tmp, [\tmp, #TSK_TI_PREEMPT] -/* -* If we are serving a softirq, there is no point in yielding: the -* softirq will not be preempted no matter what we do, so we should -* run to completion as quickly as we can. -*/ - tbnz \tmp, #SOFTIRQ_SHIFT, .Lnoyield_\@ -#ifdef CONFIG_PREEMPTION - sub \tmp, \tmp, #PREEMPT_DISABLE_OFFSET - cbz \tmp, \lbl -#endif - adr_l \tmp, irq_stat + IRQ_CPUSTAT_SOFTIRQ_PENDING - get_this_cpu_offset \tmp2 - ldr w\tmp, [\tmp, \tmp2] - cbnz w\tmp, \lbl // yield on pending softirq in task context -.Lnoyield_\@: - .endm -/* * Branch Target Identifier (BTI) */
arch/arm64/kernel/asm-offsets.c +0 -4
@@ -116,10 +116,6 @@ int main(void)DEFINE (DMA_TO_DEVICE ,DMA_TO_DEVICE );DEFINE (DMA_FROM_DEVICE ,DMA_FROM_DEVICE );BLANK (); -DEFINE (PREEMPT_DISABLE_OFFSET ,PREEMPT_DISABLE_OFFSET ); -DEFINE (SOFTIRQ_SHIFT ,SOFTIRQ_SHIFT ); -DEFINE (IRQ_CPUSTAT_SOFTIRQ_PENDING , offsetof(irq_cpustat_t , __softirq_pending)); -BLANK ();DEFINE (CPU_BOOT_TASK , offsetof(struct secondary_data , task));BLANK ();DEFINE (FTR_OVR_VAL_OFFSET , offsetof(struct arm64_ftr_override , val));
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi Ard,
Quoted 63 lines · Ard Biesheuvel
Currently, kernel mode NEON (SIMD) support is implemented in a way that requires preemption to be disabled while the SIMD registers are live. The reason for this is that those registers are not in the set that is preserved/restored on exception entry/exit and context switch, as this would impact performance generally, even for workloads where kernel mode SIMD is not the bottleneck.
However, doing substantial work with preemption disabled is not great, as it affects scheduling latency, which is especially problematic for real-time use cases. So ideally, we should keep preemption enabled when we can, and find another way to ensure that this does not corrupt the NEON register state of in-kernel SIMD users.
This series implements a suggestion by Mark Rutland, and introduces a
thread_infoflagTIF_USING_KMODE_NEON, which indicates to the thread switch machinery that the task in question has live kernel mode SIMD state which needs to be preserved and restored. The space needed for this is allocated inthread_struct. (*)Given that currently, we run kernel mode NEON with softirqs disabled (to avoid the need for preserving kernel mode NEON context belonging to task context while the SIMD unit is being used by code running in softirq context), just removing the preempt_disable/enable calls is not sufficient, and we also need to leave softirqs enabled. This means that we may need to preserve kernel mode NEON state not only on a context switch, but also when code running in softirq context takes ownership of the SIMD unit, but this is straight-forward once we add the scratch space to
thread_struct.(*) We might decide to allocate this space (~512 bytes) dynamically, if the
thread_structmemory footprint causes issues. However, we should also explore doing the same for the user space FPSIMD state, as kernel threads never return to user space and have no need for this allocation.v2:
- tweak some commit logs for clarity
- integrate with the existing lazy restore logic
- add Mark's R-b to patch #1Cc Marc Zyngier <[email protected]>Cc Will Deacon <[email protected]>Cc Mark Rutland <[email protected]>Cc Kees Cook <[email protected]>Cc Catalin Marinas <[email protected]>Cc Mark Brown <[email protected]>Cc Eric Biggers <[email protected]>Ard Biesheuvel (4):
arm64: fpsimd: Drop unneeded 'busy' flag
arm64: fpsimd: Preserve/restore kernel mode NEON at context switch
arm64: fpsimd: Implement lazy restore for kernel mode FPSIMD
arm64: crypto: Remove conditional yield logicarch/arm64/crypto/aes-glue.c | 21 +-- arch/arm64/crypto/aes-modes.S | 2 - arch/arm64/crypto/sha1-ce-core.S | 2 - arch/arm64/crypto/sha1-ce-glue.c | 19 +-- arch/arm64/crypto/sha2-ce-core.S | 2 - arch/arm64/crypto/sha2-ce-glue.c | 19 +-- arch/arm64/crypto/sha3-ce-core.S | 4 +- arch/arm64/crypto/sha3-ce-glue.c | 14 +- arch/arm64/crypto/sha512-ce-core.S | 2 -
Quoted 8 lines
arch/arm64/crypto/sha512-ce-glue.c | 16 +- arch/arm64/include/asm/assembler.h | 29 ---- arch/arm64/include/asm/processor.h | 3 + arch/arm64/include/asm/simd.h | 11 +- arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/kernel/asm-offsets.c | 4 - arch/arm64/kernel/fpsimd.c | 162 +++++++++++++------- 16 files changed, 138 insertions(+), 173 deletions(-)
Thanks for doing this! This seems like a very nice improvement, as long as the extra 528 bytes in thread_struct is not too much of an issue.
The arch/arm64/crypto/ changes seem to miss removing some of the places that do the "yield every 4096 bytes" thing:
git grep '\<4096\|SZ_4K\>' arch/arm64/crypto/
arch/arm64/crypto/aes-ce-ccm-glue.c: n = min_t(u32, n, SZ_4K); /* yield NEON at least every 4k */
arch/arm64/crypto/aes-ce-ccm-glue.c: if (len / SZ_4K > (len - n) / SZ_4K) {
arch/arm64/crypto/chacha-neon-glue.c: unsigned int todo = min_t(unsigned int, bytes, SZ_4K);
arch/arm64/crypto/crct10dif-ce-glue.c: if (chunk > SZ_4K + CRC_T10DIF_PMULL_CHUNK_SIZE)
arch/arm64/crypto/crct10dif-ce-glue.c: chunk = SZ_4K;
arch/arm64/crypto/crct10dif-ce-glue.c: if (chunk > SZ_4K + CRC_T10DIF_PMULL_CHUNK_SIZE)
arch/arm64/crypto/crct10dif-ce-glue.c: chunk = SZ_4K;
arch/arm64/crypto/nhpoly1305-neon-glue.c: unsigned int n = min_t(unsigned int, srclen, SZ_4K);
arch/arm64/crypto/poly1305-glue.c: unsigned int todo = min_t(unsigned int, len, SZ_4K);
arch/arm64/crypto/polyval-ce-glue.c: nblocks = min(srclen, 4096U) / POLYVAL_BLOCK_SIZE;Those can all be cleaned up too, right?
Please make sure to also update the comments above the assembly functions as needed. For example, in the following, the return type changed to void:
/*
* int __sha1_ce_transform(struct sha1_ce_state *sst, u8 const *src,
* int blocks)
*/
SYM_FUNC_START(__sha1_ce_transform)Can you also Cc linux-crypto on future versions of this patch series?
Thanks!
- Eric
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 76 lines · Eric Biggers
Hi Ard,
Quoted 63 lines · Ard Biesheuvel
Currently, kernel mode NEON (SIMD) support is implemented in a way that requires preemption to be disabled while the SIMD registers are live. The reason for this is that those registers are not in the set that is preserved/restored on exception entry/exit and context switch, as this would impact performance generally, even for workloads where kernel mode SIMD is not the bottleneck.
However, doing substantial work with preemption disabled is not great, as it affects scheduling latency, which is especially problematic for real-time use cases. So ideally, we should keep preemption enabled when we can, and find another way to ensure that this does not corrupt the NEON register state of in-kernel SIMD users.
This series implements a suggestion by Mark Rutland, and introduces a
thread_infoflagTIF_USING_KMODE_NEON, which indicates to the thread switch machinery that the task in question has live kernel mode SIMD state which needs to be preserved and restored. The space needed for this is allocated inthread_struct. (*)Given that currently, we run kernel mode NEON with softirqs disabled (to avoid the need for preserving kernel mode NEON context belonging to task context while the SIMD unit is being used by code running in softirq context), just removing the preempt_disable/enable calls is not sufficient, and we also need to leave softirqs enabled. This means that we may need to preserve kernel mode NEON state not only on a context switch, but also when code running in softirq context takes ownership of the SIMD unit, but this is straight-forward once we add the scratch space to
thread_struct.(*) We might decide to allocate this space (~512 bytes) dynamically, if the
thread_structmemory footprint causes issues. However, we should also explore doing the same for the user space FPSIMD state, as kernel threads never return to user space and have no need for this allocation.v2:
- tweak some commit logs for clarity
- integrate with the existing lazy restore logic
- add Mark's R-b to patch #1Cc Marc Zyngier <[email protected]>Cc Will Deacon <[email protected]>Cc Mark Rutland <[email protected]>Cc Kees Cook <[email protected]>Cc Catalin Marinas <[email protected]>Cc Mark Brown <[email protected]>Cc Eric Biggers <[email protected]>Ard Biesheuvel (4):
arm64: fpsimd: Drop unneeded 'busy' flag
arm64: fpsimd: Preserve/restore kernel mode NEON at context switch
arm64: fpsimd: Implement lazy restore for kernel mode FPSIMD
arm64: crypto: Remove conditional yield logicarch/arm64/crypto/aes-glue.c | 21 +-- arch/arm64/crypto/aes-modes.S | 2 - arch/arm64/crypto/sha1-ce-core.S | 2 - arch/arm64/crypto/sha1-ce-glue.c | 19 +-- arch/arm64/crypto/sha2-ce-core.S | 2 - arch/arm64/crypto/sha2-ce-glue.c | 19 +-- arch/arm64/crypto/sha3-ce-core.S | 4 +- arch/arm64/crypto/sha3-ce-glue.c | 14 +- arch/arm64/crypto/sha512-ce-core.S | 2 -Quoted 8 lines
arch/arm64/crypto/sha512-ce-glue.c | 16 +- arch/arm64/include/asm/assembler.h | 29 ---- arch/arm64/include/asm/processor.h | 3 + arch/arm64/include/asm/simd.h | 11 +- arch/arm64/include/asm/thread_info.h | 1 + arch/arm64/kernel/asm-offsets.c | 4 - arch/arm64/kernel/fpsimd.c | 162 +++++++++++++------- 16 files changed, 138 insertions(+), 173 deletions(-)
Quoted 3 lines
Thanks for doing this! This seems like a very nice improvement, as long as the extra 528 bytes in
thread_structis not too much of an issue.
Indeed. And we can always wire up a kmem_cache if it is, and (as I mentioned in the cover letter) we might actually do the same for the user mode state in that case, given that kernel threads have no use for it.
Quoted 3 lines
The
arch/arm64/crypto/changes seem to miss removing some of the places that do the "yield every 4096 bytes" thing:
...
Quoted 3 lines
Those can all be cleaned up too, right?
Yes. I'll add a patch for that. I missed those because the reason I looked into this was the desire to get rid of the cond_yield macro, but all of this becomes unnecessary as well as a result.
Quoted 9 lines
Please make sure to also update the comments above the assembly functions as needed. For example, in the following, the return type changed to void:
/* * int __sha1_ce_transform(struct sha1_ce_state *sst, u8 const *src, * int blocks) */ SYM_FUNC_START(__sha1_ce_transform)
OK
Quoted 2 lines
Can you also Cc linux-crypto on future versions of this patch series?
Will do.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel