ARM[64]: kernel mode NEON in atomic contexts
Subsystems (4)
v3 cover ARM[64]: kernel mode NEON in atomic contexts
Take #3 of this RFC series.
Instead of having additional separate versions of kernel_neon_begin/end, the existing ones now have been modified to always take a preallocated stack area as an argument.
The stack area is allocated by DEFINE_NEON_REGSTACK[_PARTIAL](varname), where the partial version takes an additional int num_regs indicating how many registers need to be freed up.
In the !in_interrupt() case, these functions operate as before, and the regstack is defined to minimal size in this case as it will remain unused anyway. In the in_interrupt() case, 'num_regs' (or all) NEON registers are stacked/unstacked using the allocated stack region.
Patches #1 and #4 implement the above for ARM and ARM64, respectively. Patch #3 implements the optimization for ARM64 suggested by Catalin, which has no lazy restore, potentially resulting in lots of unnecessary stack/unstack sequences otherwise.
The remaining patches are existing or new users of this API, for reference.
Ard Biesheuvel (7):
ARM: add support for kernel mode NEON in atomic context
ARM: port NEON version of xor_blocks() to new kmode NEON api
ARM64: defer reloading a task's FPSIMD state to userland resume
ARM64: add support for kernel mode NEON in atomic context
ARM64: add Crypto Extensions based synchronous core AES cipher
ARM64: add Crypto Extensions based synchronous AES in CCM mode lib/raid6: port NEON implementation to updated kmode NEON api
arch/arm/include/asm/fpstate.h | 12 + arch/arm/include/asm/neon.h | 32 ++- arch/arm/include/asm/xor.h | 48 ++-- arch/arm/vfp/vfphw.S | 45 ++++ arch/arm/vfp/vfpmodule.c | 55 +++-- arch/arm64/Makefile | 11 +- arch/arm64/crypto/Makefile | 14 ++ arch/arm64/crypto/aes-sync.c | 453 ++++++++++++++++++++++++++++++++++ arch/arm64/crypto/aesce-ccm.S | 186 ++++++++++++++ arch/arm64/include/asm/fpsimd.h | 17 ++ arch/arm64/include/asm/fpsimdmacros.h | 35 +++ arch/arm64/include/asm/neon.h | 31 ++- arch/arm64/include/asm/thread_info.h | 4 +- arch/arm64/kernel/entry-fpsimd.S | 24 ++ arch/arm64/kernel/entry.S | 2 +- arch/arm64/kernel/fpsimd.c | 34 +-- arch/arm64/kernel/signal.c | 2 + lib/raid6/neon.c | 9 +- 18 files changed, 932 insertions(+), 82 deletions(-) create mode 100644 arch/arm64/crypto/Makefile create mode 100644 arch/arm64/crypto/aes-sync.c create mode 100644 arch/arm64/crypto/aesce-ccm.S
Signature · 1 line
1.8.1.2
RFC PATCH v3 1/7 ARM: add support for kernel mode NEON in atomic context
Some applications, such as WPA CCMP encryption, do substantial amounts of work in non-process context. In order to support accelerated NEON implementations under these circumstances, we need a way to preserve the NEON context that may
(a) belong to a completely unrelated userland process (if the
NEON unit is turned off atm);
(b) belong to current userland;
(c) belong to current kernel mode in process context.
The best way to deal with this is to just stack whatever registers we are going to use, and unstack them when we are done.
This patch modifies kernel_neon_begin() and kernel_neon_end(), so they may be called from any context. To address the in_interrupt() case, they now both take a parameter defined by DEFINE_NEON_REGSTACK() or DEFINE_NEON_REGSTACK_PARTIAL() [in case only a few NEON registers are in fact used]. The !in_interrupt() case is unchanged from before.
--- arch/arm/include/asm/fpstate.h | 12 +++++++++ arch/arm/include/asm/neon.h | 32 +++++++++++++++++++++--- arch/arm/vfp/vfphw.S | 45 ++++++++++++++++++++++++++++++++++ arch/arm/vfp/vfpmodule.c | 55 ++++++++++++++++++++++++------------------ 4 files changed, 118 insertions(+), 26 deletions(-)
arch/arm/include/asm/fpstate.h +12 -0
@@ -52,6 +52,18 @@ union vfp_state {extern void vfp_flush_thread (union vfp_state *);extern void vfp_release_thread (union vfp_state *); +/* +* Variable sized struct for stacking the bottom 'n' NEON registers. +*/ +struct vfp_partial_state { +__u32 fpexc; +__u32 fpscr; +__u8 qregs[]__aligned(16) ; +}__aligned(16) ; + +extern void vfp_load_partial_state (struct vfp_partial_state *,u32 num_regs); +extern void vfp_save_partial_state (struct vfp_partial_state *,u32 num_regs); +#define FP_HARD_SIZE 35struct fp_hard_struct {
arch/arm/include/asm/neon.h +29 -3
@@ -8,10 +8,30 @@* published by the Free Software Foundation. */ +#include <linux/types.h> +#include <linux/hardirq.h> +#include <asm/fpstate.h> #include <asm/hwcap.h> #define cpu_has_neon () (!!(elf_hwcap & HWCAP_NEON)) +/* +* Avoid wasting stack space by making the size of the allocated area depend on +* whether we are currently running in process context. (If this is the case, we +* will use the normal preserve/restore mechanism, leaving the allocated stack +* space unused.) +*/ +#define __QREG_SIZE (num) \ + ((!in_interrupt()) ? 0 : (num) > 16 ? 256 : 16 * (((num) + 1) & ~1U)) + +#define DEFINE_NEON_REGSTACK_PARTIAL (v, num) \ + struct { \ + struct vfp_partial_state regs; \ + u8 qregs[__QREG_SIZE(num)]; \ + } v + +#define DEFINE_NEON_REGSTACK (name) DEFINE_NEON_REGSTACK_PARTIAL(name, 16) +#ifdef __ARM_NEON__/*
@@ -27,10 +47,16 @@* -mpfu=neon is set. */ -#define kernel_neon_begin () \ +#define kernel_neon_begin (p) \ BUILD_BUG_ON_MSG(1, "kernel_neon_begin() called from NEON code")#else -void kernel_neon_begin (void ); +#define kernel_neon_begin (p) \ + __kernel_neon_begin(&(p).regs, sizeof((p).qregs)/16)#endif -void kernel_neon_end (void ); + +#define kernel_neon_end (p) \ + __kernel_neon_end(&(p).regs, sizeof((p).qregs)/16) + +void __kernel_neon_begin (struct vfp_partial_state *regs,u32 num_regs); +void __kernel_neon_end (struct vfp_partial_state *regs,u32 num_regs);
arch/arm/vfp/vfphw.S +45 -0
48 lines not on this page · raw message
arch/arm/vfp/vfpmodule.c +32 -23
76 lines not on this page · raw message
Signature · 1 line
1.8.1.2
Catalin Marinas
maintainerRFC PATCH v3 1/7 ARM: add support for kernel mode NEON in atomic context
Quoted 19 lines · Ard Biesheuvel
arch/arm/include/asm/neon.h +7
-0@@ -8,10 +8,30 @@* published by the Free Software Foundation. */ +#include <linux/types.h> +#include <linux/hardirq.h> +#include <asm/fpstate.h> #include <asm/hwcap.h> #define cpu_has_neon () (!!(elf_hwcap & HWCAP_NEON)) +/* +* Avoid wasting stack space by making the size of the allocated area depend on +* whether we are currently running in process context. (If this is the case, we +* will use the normal preserve/restore mechanism, leaving the allocated stack
arch/arm/include/asm/neon.h line 31
Quoted 10 lines
arch/arm/include/asm/neon.h +10
-0+* space unused.) +*/ +#define __QREG_SIZE (num) \ + ((!in_interrupt()) ? 0 : (num) > 16 ? 256 : 16 * (((num) + 1) & ~1U)) + +#define DEFINE_NEON_REGSTACK_PARTIAL (v, num) \ + struct { \ + struct vfp_partial_state regs; \ + u8 qregs[__QREG_SIZE(num)]; \ + } v
Oh, interesting gcc feature. What does it generate?
Signature · 1 line
Catalin
RFC PATCH v3 1/7 ARM: add support for kernel mode NEON in atomic context
Quoted 16 lines · Catalin Marinas
Quoted 15 lines · Ard Biesheuvel
arch/arm/include/asm/neon.h +3
-0@@ -8,10 +8,30 @@* published by the Free Software Foundation. */ +#include <linux/types.h> +#include <linux/hardirq.h> +#include <asm/fpstate.h> #include <asm/hwcap.h> #define cpu_has_neon () (!!(elf_hwcap & HWCAP_NEON))
Quoted 17 lines
Quoted 4 lines
arch/arm/include/asm/neon.h +4
-0+/* +* Avoid wasting stack space by making the size of the allocated area depend on +* whether we are currently running in process context. (If this is the case, we +* will use the normal preserve/restore mechanism, leaving the allocated stack arch/arm/include/asm/neon.h line 31
Quoted 10 lines
arch/arm/include/asm/neon.h +10
-0+* space unused.) +*/ +#define __QREG_SIZE (num) \ + ((!in_interrupt()) ? 0 : (num) > 16 ? 256 : 16 * (((num) + 1) & ~1U)) + +#define DEFINE_NEON_REGSTACK_PARTIAL (v, num) \ + struct { \ + struct vfp_partial_state regs; \ + u8 qregs[__QREG_SIZE(num)]; \ + } vOh, interesting gcc feature. What does it generate?
Well, it's not a feature particular to GCC, as far as I am aware. The anonymous struct is just runtime variably sized depending on in_interrupt() and the requested number of registers.
Signature · 1 line
Ard.
Catalin Marinas
maintainerRFC PATCH v3 1/7 ARM: add support for kernel mode NEON in atomic context
Quoted 35 lines · Ard Biesheuvel
Quoted 16 lines · Catalin Marinas
Quoted 15 lines · Ard Biesheuvel
arch/arm/include/asm/neon.h +3
-0@@ -8,10 +8,30 @@* published by the Free Software Foundation. */ +#include <linux/types.h> +#include <linux/hardirq.h> +#include <asm/fpstate.h> #include <asm/hwcap.h> #define cpu_has_neon () (!!(elf_hwcap & HWCAP_NEON))Quoted 17 lines
Quoted 4 lines
arch/arm/include/asm/neon.h +4
-0+/* +* Avoid wasting stack space by making the size of the allocated area depend on +* whether we are currently running in process context. (If this is the case, we +* will use the normal preserve/restore mechanism, leaving the allocated stack arch/arm/include/asm/neon.h line 31
Quoted 10 lines
arch/arm/include/asm/neon.h +10
-0+* space unused.) +*/ +#define __QREG_SIZE (num) \ + ((!in_interrupt()) ? 0 : (num) > 16 ? 256 : 16 * (((num) + 1) & ~1U)) + +#define DEFINE_NEON_REGSTACK_PARTIAL (v, num) \ + struct { \ + struct vfp_partial_state regs; \ + u8 qregs[__QREG_SIZE(num)]; \ + } vOh, interesting gcc feature. What does it generate?
Quoted 3 lines
Well, it's not a feature particular to GCC, as far as I am aware. The anonymous struct is just runtime variably sized depending on
in_interrupt()and the requested number of registers.
OK, it looks like it's valid C99. I was worried the compiler may generate something like an alloca() library call.
Signature · 1 line
Catalin
RFC PATCH v3 2/7 ARM: port NEON version of xor_blocks() to new kmode NEON api
In reply to Ard BiesheuvelIt is now permissible to use the NEON in non-process context, so update the XOR code so it uses the NEON version even in non-process context.
--- arch/arm/include/asm/xor.h | 48 +++++++++++++++++++--------------------------- 1 file changed, 20 insertions(+), 28 deletions(-)
arch/arm/include/asm/xor.h +20 -28
@@ -151,52 +151,44 @@ extern struct xor_block_template const xor_block_neon_inner;static void xor_neon_2 (unsigned long bytes,unsigned long *p1,unsigned long *p2) { -if (in_interrupt ()) { -xor_arm4regs_2 (bytes, p1, p2); - }else { -kernel_neon_begin (); - xor_block_neon_inner.do_2 (bytes, p1, p2); -kernel_neon_end (); - } +DEFINE_NEON_REGSTACK (s); + +kernel_neon_begin (s); + xor_block_neon_inner.do_2 (bytes, p1, p2); +kernel_neon_end (s); }static void xor_neon_3 (unsigned long bytes,unsigned long *p1,unsigned long *p2,unsigned long *p3) { -if (in_interrupt ()) { -xor_arm4regs_3 (bytes, p1, p2, p3); - }else { -kernel_neon_begin (); - xor_block_neon_inner.do_3 (bytes, p1, p2, p3); -kernel_neon_end (); - } +DEFINE_NEON_REGSTACK (s); + +kernel_neon_begin (s); + xor_block_neon_inner.do_3 (bytes, p1, p2, p3); +kernel_neon_end (s); }static void xor_neon_4 (unsigned long bytes,unsigned long *p1,unsigned long *p2,unsigned long *p3,unsigned long *p4) { -if (in_interrupt ()) { -xor_arm4regs_4 (bytes, p1, p2, p3, p4); - }else { -kernel_neon_begin (); - xor_block_neon_inner.do_4 (bytes, p1, p2, p3, p4); -kernel_neon_end (); - } +DEFINE_NEON_REGSTACK (s); + +kernel_neon_begin (s); + xor_block_neon_inner.do_4 (bytes, p1, p2, p3, p4); +kernel_neon_end (s); }static void xor_neon_5 (unsigned long bytes,unsigned long *p1,unsigned long *p2,unsigned long *p3,unsigned long *p4,unsigned long *p5) { -if (in_interrupt ()) { -xor_arm4regs_5 (bytes, p1, p2, p3, p4, p5); - }else { -kernel_neon_begin (); - xor_block_neon_inner.do_5 (bytes, p1, p2, p3, p4, p5); -kernel_neon_end (); - } +DEFINE_NEON_REGSTACK (s); + +kernel_neon_begin (s); + xor_block_neon_inner.do_5 (bytes, p1, p2, p3, p4, p5); +kernel_neon_end (s); }static struct xor_block_template xor_block_neon = {
Signature · 1 line
1.8.1.2
RFC PATCH v3 3/7 ARM64: defer reloading a task's FPSIMD state to userland resume
In reply to Ard BiesheuvelModify kernel_neon_begin() and kernel_neon_end() so subsequent calls don't need to preserve/restore the userland FPSIMD state if the task has not entered userland in the mean time.
--- arch/arm64/include/asm/thread_info.h | 4 +++- arch/arm64/kernel/entry.S | 2 +- arch/arm64/kernel/fpsimd.c | 7 ++----- arch/arm64/kernel/signal.c | 2 ++ 4 files changed, 8 insertions(+), 7 deletions(-)
arch/arm64/include/asm/thread_info.h +3 -1
@@ -106,6 +106,7 @@ static inline struct thread_info *current_thread_info(void)#define TIF_SIGPENDING 0#define TIF_NEED_RESCHED 1#define TIF_NOTIFY_RESUME 2/* callback before returning to user */ +#define TIF_RELOAD_FPSTATE 3/* user FPSIMD context saved to mem */ #define TIF_SYSCALL_TRACE 8#define TIF_POLLING_NRFLAG 16#define TIF_MEMDIE 18/* is terminating due to OOM killer */
@@ -118,10 +119,11 @@ static inline struct thread_info *current_thread_info(void)#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) +#define _TIF_RELOAD_FPSTATE (1 << TIF_RELOAD_FPSTATE)#define _TIF_32BIT (1 << TIF_32BIT)#define _TIF_WORK_MASK (_TIF_NEED_RESCHED | _TIF_SIGPENDING | \ - _TIF_NOTIFY_RESUME) + _TIF_NOTIFY_RESUME | _TIF_RELOAD_FPSTATE)#endif /* __KERNEL__ */ #endif /* __ASM_THREAD_INFO_H */
arch/arm64/kernel/entry.S +1 -1
@@ -589,7 +589,7 @@ fast_work_pending: str x0, [sp, #S_X0] // returned x0 work_pending: tbnz x1, #TIF_NEED_RESCHED, work_resched - /* TIF_SIGPENDING or TIF_NOTIFY_RESUME case */ + /* TIF_SIGPENDING/TIF_NOTIFY_RESUME/TIF_RELOAD_FPSTATE case */ ldr x2, [sp, #S_PSTATE] mov x0, sp // 'regs' tst x2, #PSR_MODE_MASK // user mode regs?
arch/arm64/kernel/fpsimd.c +2 -5
@@ -72,7 +72,7 @@ void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs)void fpsimd_thread_switch (struct task_struct *next) {/* check if not kernel threads */ -if (current->mm) +if (current->mm && !test_and_clear_thread_flag (TIF_RELOAD_FPSTATE ))fpsimd_save_state (¤t->thread.fpsimd_state);if (next->mm)fpsimd_load_state (&next->thread.fpsimd_state);
@@ -95,16 +95,13 @@ void kernel_neon_begin(void)BUG_ON (in_interrupt ());preempt_disable (); -if (current->mm) +if (current->mm && !test_and_set_thread_flag (TIF_RELOAD_FPSTATE ))fpsimd_save_state (¤t->thread.fpsimd_state); }EXPORT_SYMBOL (kernel_neon_begin);void kernel_neon_end (void ) { -if (current->mm) -fpsimd_load_state (¤t->thread.fpsimd_state); -preempt_enable (); }EXPORT_SYMBOL (kernel_neon_end);
arch/arm64/kernel/signal.c +2 -0
@@ -416,4 +416,6 @@ asmlinkage void do_notify_resume(struct pt_regs *regs,clear_thread_flag (TIF_NOTIFY_RESUME );tracehook_notify_resume (regs); } +if (test_and_clear_thread_flag (TIF_RELOAD_FPSTATE )) +fpsimd_load_state (¤t->thread.fpsimd_state); }
Signature · 1 line
1.8.1.2
Catalin Marinas
maintainerRFC PATCH v3 3/7 ARM64: defer reloading a task's FPSIMD state to userland resume
arch/arm64/kernel/fpsimd.c void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs) · line 76
Quoted 9 lines · Ard Biesheuvel
arch/arm64/kernel/fpsimd.c +1
-1@@ -72,7 +72,7 @@ void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs)void fpsimd_thread_switch (struct task_struct *next) {/* check if not kernel threads */ -if (current->mm) +if (current->mm && !test_and_clear_thread_flag (TIF_RELOAD_FPSTATE ))fpsimd_save_state (¤t->thread.fpsimd_state);
Why does it need test_and_set_thread_flag() here? Some comments would be useful as it looks strange to check a reload flag to decide whether to save a state. Or change the name to something like 'dirty'.
arch/arm64/kernel/fpsimd.c void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs) · line 78
Quoted 2 lines
arch/arm64/kernel/fpsimd.c +0
-0if (next->mm)fpsimd_load_state (&next->thread.fpsimd_state);
This function could be optimised a bit more to avoid saving/restoring if the switch only happened between a user thread and a kernel one (and back again) since the FP state may not have been dirtied. But what I had in mind was per-CPU fpstate (possibly pointer or some flag) rather than per-thread.
arch/arm64/kernel/signal.c asmlinkage void do_notify_resume(struct pt_regs *regs, · line 420
Quoted 8 lines
arch/arm64/kernel/signal.c +2
-0@@ -416,4 +416,6 @@ asmlinkage void do_notify_resume(struct pt_regs *regs,clear_thread_flag (TIF_NOTIFY_RESUME );tracehook_notify_resume (regs); } +if (test_and_clear_thread_flag (TIF_RELOAD_FPSTATE )) +fpsimd_load_state (¤t->thread.fpsimd_state);
I think this code can be preempted, it is run with IRQs enabled. And there is a small window where we cleared the flag but haven't loaded the state.
Signature · 1 line
Catalin
RFC PATCH v3 3/7 ARM64: defer reloading a task's FPSIMD state to userland resume
Quoted 15 lines · Catalin Marinas
arch/arm64/kernel/fpsimd.c void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs) · line 76
Quoted 9 lines · Ard Biesheuvel
arch/arm64/kernel/fpsimd.c +1
-1@@ -72,7 +72,7 @@ void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs)void fpsimd_thread_switch (struct task_struct *next) {/* check if not kernel threads */ -if (current->mm) +if (current->mm && !test_and_clear_thread_flag (TIF_RELOAD_FPSTATE ))fpsimd_save_state (¤t->thread.fpsimd_state);Why does it need
test_and_set_thread_flag()here? Some comments would be useful as it looks strange to check a reload flag to decide whether to save a state. Or change the name to something like 'dirty'.
Actually, it's test and clear. If the userland register file has already been preserved for the purpose of performing kernel mode NEON, it should not be saved again when the task gets scheduled out. The clearing could also be deferred to the time when the task gets scheduled in again. Or perhaps, it would be even better to always defer loading the userland state for the next task when that task in fact enters userland.
Quoted 9 lines
arch/arm64/kernel/fpsimd.c void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs) · line 78
Quoted 2 lines
arch/arm64/kernel/fpsimd.c +0
-0if (next->mm)fpsimd_load_state (&next->thread.fpsimd_state);This function could be optimised a bit more to avoid saving/restoring if the switch only happened between a user thread and a kernel one (and back again) since the FP state may not have been dirtied. But what I had in mind was per-CPU fpstate (possibly pointer or some flag) rather than per-thread.
Well, then we are entering the realm of lazy restore, imo. There were some patches proposed for that already, I think? But I do agree that at his point, there is no need to restore the userland register contents yet, it can be deferred to the point when the task reenters userland (as mentioned above).
Quoted 4 lines
Quoted 4 lines
arch/arm64/kernel/signal.c +0
-0@@ -416,4 +416,6 @@ asmlinkage void do_noti fy_resume(struct pt_regs *regs,clear_thread_flag (TIF_NOTIFY_RESUME );
Quoted 9 lines
arch/arm64/kernel/signal.c asmlinkage void do_notify_resume(struct pt_regs *regs, · line 420
Quoted 4 lines
arch/arm64/kernel/signal.c +2
-0tracehook_notify_resume (regs); } +if (test_and_clear_thread_flag (TIF_RELOAD_FPSTATE )) +fpsimd_load_state (¤t->thread.fpsimd_state);I think this code can be preempted, it is run with IRQs enabled. And there is a small window where we cleared the flag but haven't loaded the state.
If we are preempted at this point, the fpstate will be loaded in the normal way the next time this task runs, so I think this is harmless. Although I guess we may be restoring the fp state twice in that case?
So in summary, what I need to do is:
- rework to use a per_cpu flag rather than a TIF;
- preserve the userland state (if it has one) when a task gets scheduled out;
- restore the userland state when a task enters userland;
I will propose an updated patch after I wrap up the work on the in_interrupt() kernel mode NEON, as these topics are really orthogonal and there is no reason to keep them combined in a single series.
Signature · 1 line
Ard.
Catalin Marinas
maintainerRFC PATCH v3 3/7 ARM64: defer reloading a task's FPSIMD state to userland resume
Quoted 16 lines · Ard Biesheuvel
Quoted 14 lines · Catalin Marinas
arch/arm64/kernel/fpsimd.c void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs) · line 76
Quoted 9 lines · Ard Biesheuvel
arch/arm64/kernel/fpsimd.c +1
-1@@ -72,7 +72,7 @@ void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs)void fpsimd_thread_switch (struct task_struct *next) {/* check if not kernel threads */ -if (current->mm) +if (current->mm && !test_and_clear_thread_flag (TIF_RELOAD_FPSTATE ))fpsimd_save_state (¤t->thread.fpsimd_state);Why does it need
test_and_set_thread_flag()here? Some comments would be useful as it looks strange to check a reload flag to decide whether to save a state. Or change the name to something like 'dirty'.
Quoted 1 line
Actually, it's test and clear.
Yes, just a typo.
Quoted 5 lines
If the userland register file has
already been preserved for the purpose of performing kernel mode NEON, it should not be saved again when the task gets scheduled out. The clearing could also be deferred to the time when the task gets scheduled in again.
The above should be turned into a comment in the code.
Quoted 3 lines
Or perhaps, it would be even better to always
defer loading the userland state for the next task when that task in fact enters userland.
It needs some more thinking, its usually cleaner in the context switching code.
Quoted 9 lines
Quoted 8 lines
arch/arm64/kernel/fpsimd.c void do_fpsimd_exc(unsigned int esr, struct pt_regs *regs) · line 78
Quoted 2 lines
arch/arm64/kernel/fpsimd.c +0
-0if (next->mm)fpsimd_load_state (&next->thread.fpsimd_state);This function could be optimised a bit more to avoid saving/restoring if the switch only happened between a user thread and a kernel one (and back again) since the FP state may not have been dirtied. But what I had in mind was per-CPU fpstate (possibly pointer or some flag) rather than per-thread.
Quoted 5 lines
Well, then we are entering the realm of lazy restore, imo. There were some patches proposed for that already, I think? But I do agree that at his point, there is no need to restore the userland register contents yet, it can be deferred to the point when the task reenters userland (as mentioned above).
Not entirely lazy. What I dont want to see (without proper benchmarks) is disabling the FP at context switch and restoring the registers lazily via the fault mechanism. What Im proposing above is not lazy, just an optimisation for a clear case where the FP is not used in a kernel thread.
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
arch/arm64/kernel/signal.c +0
-0@@ -416,4 +416,6 @@ asmlinkage void do_noti fy_resume(struct pt_regs *regs,clear_thread_flag (TIF_NOTIFY_RESUME );Quoted 8 lines
arch/arm64/kernel/signal.c asmlinkage void do_notify_resume(struct pt_regs *regs, · line 420
Quoted 4 lines
arch/arm64/kernel/signal.c +2
-0tracehook_notify_resume (regs); } +if (test_and_clear_thread_flag (TIF_RELOAD_FPSTATE )) +fpsimd_load_state (¤t->thread.fpsimd_state);I think this code can be preempted, it is run with IRQs enabled. And there is a small window where we cleared the flag but haven't loaded the state.
Quoted 3 lines
If we are preempted at this point, the fpstate will be loaded in the normal way the next time this task runs, so I think this is harmless. Although I guess we may be restoring the fp state twice in that case?
Lets say task A does an svc and gets into kernel mode followed by kernel_neon_begin/end(). Before returning to user, the kernel runs the above test_and_clear_thread_flag(). Immediately after, an interrupt happens the task A is preempted. The fpsimd_thread_switch() function finds that TIF_RELOAD_FPSTATE is cleared and save the current FP state. But the FP regs contain whatever the kernel neon code did, so you corrupt the existing data.
Quoted 4 lines
So in summary, what I need to do is:
- rework to use aper_cpuflag rather than a TIF;
- preserve the userland state (if it has one) when a task gets scheduled out;
- restore the userland state when a task enters userland;
Happy to discuss the algorithm before you code it (unless you prefer to write the code quickly).
Quoted 3 lines
I will propose an updated patch after I wrap up the work on the
in_interrupt()kernel mode NEON, as these topics are really orthogonal and there is no reason to keep them combined in a single series.
Sounds fine.
Catalin
RFC PATCH v3 4/7 ARM64: add support for kernel mode NEON in atomic context
In reply to Ard BiesheuvelThis patch modifies kernel_neon_begin() and kernel_neon_end(), so they may be called from any context. To address the in_interrupt() case, they now both take a parameter defined by DEFINE_NEON_REGSTACK() or DEFINE_NEON_REGSTACK_PARTIAL() [in case only a few NEON registers are in fact used]. The !in_interrupt() case is unchanged from before.
--- arch/arm64/include/asm/fpsimd.h | 17 +++++++++++++++++ arch/arm64/include/asm/fpsimdmacros.h | 35 +++++++++++++++++++++++++++++++++++ arch/arm64/include/asm/neon.h | 31 +++++++++++++++++++++++++++++-- arch/arm64/kernel/entry-fpsimd.S | 24 ++++++++++++++++++++++++ arch/arm64/kernel/fpsimd.c | 29 ++++++++++++++++++----------- 5 files changed, 123 insertions(+), 13 deletions(-)
arch/arm64/include/asm/fpsimd.h +17 -0
@@ -39,6 +39,18 @@ struct fpsimd_state { }; }; +/* +* Variable sized struct for stacking the bottom 'n' FP/SIMD registers. +* Mainly intended for kernel use of v8 Crypto Extensions which only +* needs a few registers and may need to execute in atomic context. +*/ +struct fpsimd_partial_state { +u32 fpsr; +u32 fpcr; +__uint128_t vregs[]__aligned(16) ; +}__aligned(16) ; + +#if defined(__KERNEL__) && defined(CONFIG_COMPAT )/* Masks for extracting the FPSR and FPCR from the FPSCR */ #define VFP_FPSCR_STAT_MASK 0xf800009f
@@ -55,6 +67,11 @@ struct task_struct;extern void fpsimd_save_state (struct fpsimd_state *state);extern void fpsimd_load_state (struct fpsimd_state *state); +extern void fpsimd_save_partial_state (struct fpsimd_partial_state *state, +u32 num_regs); +extern void fpsimd_load_partial_state (struct fpsimd_partial_state *state, +u32 num_regs); +extern void fpsimd_thread_switch (struct task_struct *next);extern void fpsimd_flush_thread (void );
arch/arm64/include/asm/fpsimdmacros.h +35 -0
@@ -62,3 +62,38 @@ ldr w\tmpnr, [\state, #16 * 2 + 4] msr fpcr, x\tmpnr .endm + +.altmacro +.macro q2op, op, q1, q2, state + \op q\q1, q\q2, [\state, #-(16 * \q1) - 16] +.endm + +.macro fpsimd_save_partial state, num, tmpnr1, tmpnr2 + mrs x\tmpnr1, fpsr + mrs x\tmpnr2, fpcr + stp w\tmpnr1, w\tmpnr2, [\state] + adr x\tmpnr1, 0f + add \state, \state, \num, lsl #4 + sub x\tmpnr1, x\tmpnr1, \num, lsl #1 + br x\tmpnr1 + .irp qa, 30, 28, 26, 24, 22, 20, 18, 16, 14, 12, 10, 8, 6, 4, 2, 0 + qb = \qa + 1 + q2op stp, \qa, %qb, \state + .endr +0: +.endm + +.macro fpsimd_restore_partial state, num, tmpnr1, tmpnr2 + ldp w\tmpnr1, w\tmpnr2, [\state] + msr fpsr, x\tmpnr1 + msr fpcr, x\tmpnr2 + adr x\tmpnr1, 0f + add \state, \state, \num, lsl #4 + sub x\tmpnr1, x\tmpnr1, \num, lsl #1 + br x\tmpnr1 + .irp qa, 30, 28, 26, 24, 22, 20, 18, 16, 14, 12, 10, 8, 6, 4, 2, 0 + qb = \qa + 1 + q2op ldp, \qa, %qb, \state + .endr +0: +.endm
arch/arm64/include/asm/neon.h +29 -2
@@ -8,7 +8,34 @@* published by the Free Software Foundation. */ +#include <linux/hardirq.h> +#include <linux/types.h> +#include <asm/fpsimd.h> +#define cpu_has_neon () (1) -void kernel_neon_begin (void ); -void kernel_neon_end (void ); +/* +* Avoid wasting stack space by making the size of the allocated area depend on +* whether we are currently running in process context. (If this is the case, we +* will use the normal preserve/restore mechanism, leaving the allocated stack +* space unused.) +*/ +#define __VREG_SIZE (num) \ + ((!in_interrupt()) ? 0 : (num) > 32 ? 512 : 32 * (((num) + 1) & ~1U)) + +#define DEFINE_NEON_REGSTACK_PARTIAL (v, num) \ + struct { \ + struct fpsimd_partial_state regs; \ + u8 vregs[__VREG_SIZE(num)]; \ + } v + +#define DEFINE_NEON_REGSTACK (name) DEFINE_NEON_REGSTACK_PARTIAL(name, 32) + +#define kernel_neon_begin (p) \ + __kernel_neon_begin(&(p).regs, sizeof((p).vregs)/16) + +#define kernel_neon_end (p) \ + __kernel_neon_end(&(p).regs, sizeof((p).vregs)/16) + +void __kernel_neon_begin (struct fpsimd_partial_state *regs,u32 num_regs); +void __kernel_neon_end (struct fpsimd_partial_state *regs,u32 num_regs);
arch/arm64/kernel/entry-fpsimd.S +24 -0
@@ -41,3 +41,27 @@ ENTRY(fpsimd_load_state) fpsimd_restore x0, 8 ret ENDPROC(fpsimd_load_state) + +#ifdef CONFIG_KERNEL_MODE_NEON + +/* + * Save the bottom n FP registers. + * + * x0 - pointer to struct fpsimd_partial_state + */ +ENTRY(fpsimd_save_partial_state) + fpsimd_save_partial x0, x1, 8, 9 + ret +ENDPROC(fpsimd_load_partial_state) + +/* + * Load the bottom n FP registers. + * + * x0 - pointer to struct fpsimd_partial_state + */ +ENTRY(fpsimd_load_partial_state) + fpsimd_restore_partial x0, x1, 8, 9 + ret +ENDPROC(fpsimd_load_partial_state) + +#endif
arch/arm64/kernel/fpsimd.c +18 -11
@@ -89,22 +89,29 @@ void fpsimd_flush_thread(void)/* * Kernel-side NEON support functions */ -void kernel_neon_begin (void ) +void __kernel_neon_begin (struct fpsimd_partial_state *regs,u32 num_regs) { -/* Avoid using the NEON in interrupt context */ -BUG_ON (in_interrupt ()); -preempt_disable (); - -if (current->mm && !test_and_set_thread_flag (TIF_RELOAD_FPSTATE )) -fpsimd_save_state (¤t->thread.fpsimd_state); +if (in_interrupt ()) { +BUG_ON (!num_regs); +fpsimd_save_partial_state (regs, num_regs); + }else { +preempt_disable (); +if (current->mm && + !test_and_set_thread_flag (TIF_RELOAD_FPSTATE )) +fpsimd_save_state (¤t->thread.fpsimd_state); + } } -EXPORT_SYMBOL (kernel_neon_begin); +EXPORT_SYMBOL (__kernel_neon_begin); -void kernel_neon_end (void ) +void __kernel_neon_end (struct fpsimd_partial_state *regs,u32 num_regs) { -preempt_enable (); +if (in_interrupt ()) { +BUG_ON (!num_regs); +fpsimd_load_partial_state (regs, num_regs); + }else +preempt_enable (); } -EXPORT_SYMBOL (kernel_neon_end); +EXPORT_SYMBOL (__kernel_neon_end);#endif /* CONFIG_KERNEL_MODE_NEON */
Signature · 1 line
1.8.1.2
RFC PATCH v3 5/7 ARM64: add Crypto Extensions based synchronous core AES cipher
In reply to Ard BiesheuvelThis implements the core AES cipher using the Crypto Extensions, using only NEON register q0 and q1.
--- arch/arm64/Makefile | 11 +++-- arch/arm64/crypto/Makefile | 14 ++++++ arch/arm64/crypto/aes-sync.c | 106 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 126 insertions(+), 5 deletions(-) create mode 100644 arch/arm64/crypto/Makefile create mode 100644 arch/arm64/crypto/aes-sync.c
arch/arm64/Makefile +6 -5
@@ -36,11 +36,12 @@ TEXT_OFFSET := 0x00080000 export TEXT_OFFSET GZFLAGS -core-y += arch/arm64/kernel/ arch/arm64/mm/ -core-$(CONFIG_KVM) += arch/arm64/kvm/ -core-$(CONFIG_XEN) += arch/arm64/xen/ -libs-y := arch/arm64/lib/ $(libs-y) -libs-y += $(LIBGCC) +core-y += arch/arm64/kernel/ arch/arm64/mm/ +core-$(CONFIG_KVM) += arch/arm64/kvm/ +core-$(CONFIG_XEN) += arch/arm64/xen/ +core-$(CONFIG_CRYPTO) += arch/arm64/crypto/ +libs-y := arch/arm64/lib/ $(libs-y) +libs-y += $(LIBGCC) # Default target when executing plain make KBUILD_IMAGE := Image.gz
arch/arm64/crypto/Makefile new file +14 -0
@@ -0,0 +1,14 @@ +# +# linux/arch/arm64/crypto/Makefile +# +# Copyright (C) 2013 Linaro Ltd <[email protected]> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License version 2 as +# published by the Free Software Foundation. +# + +aesce-sync-y := aes-sync.o +obj-m += aesce-sync.o + +CFLAGS_aes-sync.o += -march=armv8-a+crypto
arch/arm64/crypto/aes-sync.c new file +106 -0
106 lines not on this page · raw message
Signature · 1 line
1.8.1.2
RFC PATCH v3 6/7 ARM64: add Crypto Extensions based synchronous AES in CCM mode
In reply to Ard BiesheuvelThis implements the CCM AEAD chaining mode for AES using Crypto Extensions instructions.
--- arch/arm64/crypto/Makefile | 2 +- arch/arm64/crypto/aes-sync.c | 355 +++++++++++++++++++++++++++++++++++++++++- arch/arm64/crypto/aesce-ccm.S | 186 ++++++++++++++++++++++ 3 files changed, 538 insertions(+), 5 deletions(-) create mode 100644 arch/arm64/crypto/aesce-ccm.S
arch/arm64/crypto/Makefile +1 -1
@@ -8,7 +8,7 @@ # published by the Free Software Foundation. # -aesce-sync-y := aes-sync.o +aesce-sync-y := aes-sync.o aesce-ccm.o obj-m += aesce-sync.o CFLAGS_aes-sync.o += -march=armv8-a+crypto
arch/arm64/crypto/aes-sync.c +351 -4
383 lines not on this page · raw message
arch/arm64/crypto/aesce-ccm.S new file +186 -0
186 lines not on this page · raw message
Signature · 1 line
1.8.1.2
RFC PATCH v3 7/7 lib/raid6: port NEON implementation to updated kmode NEON api
In reply to Ard Biesheuvel--- lib/raid6/neon.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
lib/raid6/neon.c +5 -4
@@ -13,8 +13,8 @@#ifdef __KERNEL__#include <asm/neon.h> #else -#define kernel_neon_begin () -#define kernel_neon_end () +#define kernel_neon_begin (s) +#define kernel_neon_end (s)#define cpu_has_neon () (1)#endif
@@ -33,12 +33,13 @@ static void raid6_neon ## _n ## _gen_syndrome(int disks, \size_t bytes,void **ptrs) \ { \ +DEFINE_NEON_REGSTACK (s); \ void raid6_neon ## _n ## _gen_syndrome_real(int, \unsigned long ,void **); \ -kernel_neon_begin (); \ +kernel_neon_begin (s); \ raid6_neon ## _n ## _gen_syndrome_real(disks, \ (unsigned long )bytes,ptrs ); \ -kernel_neon_end (); \ +kernel_neon_end (s); \ } \ struct raid6_calls const raid6_neonx ## _n = { \ raid6_neon ## _n ## _gen_syndrome, \
Signature · 1 line
1.8.1.2
v3 cover ARM[64]: kernel mode NEON in atomic contexts
In reply to Ard BiesheuvelQuoted 5 lines · Ard Biesheuvel
Take #3 of this RFC series.
Instead of having additional separate versions of kernel_neon_begin/end, the existing ones now have been modified to always take a preallocated stack area as an argument.
The problem with this approach is that you break git bisect by making the kernel unbuildable when this series is partially applied. Either you make kernel_neon_begin/end into wrappers with no argument around the new interface, or you change all users at the same time as the interface. One big principle is not to break the kernel build in the middle of a patch series when altering an existing interface.
Quoted 8 lines
The stack area is allocated by DEFINE_NEON_REGSTACK[_PARTIAL](varname), where the partial version takes an additional int
num_regsindicating how many registers need to be freed up.In the !in_interrupt() case, these functions operate as before, and the regstack is defined to minimal size in this case as it will remain unused anyway. In the
in_interrupt()case, 'num_regs' (or all) NEON registers are stacked/unstacked using the allocated stack region.
Would have been nice to have the stack simply be a NULL pointer when !in_interrupt() or when the number of regs is 0. This would remove the need for a runtime check on !num_regs. I don't see an obvious way to accomplish that right now though.
Nicolas
v3 cover ARM[64]: kernel mode NEON in atomic contexts
Quoted 6 lines · Nicolas Pitre
Quoted 3 lines · Ard Biesheuvel
Instead of having additional separate versions of kernel_neon_begin/end, the existing ones now have been modified to always take a preallocated stack area as an argument.
Quoted 7 lines
The problem with this approach is that you break git bisect by making the kernel unbuildable when this series is partially applied. Either you make kernel_neon_begin/end into wrappers with no argument around the new interface, or you change all users at the same time as the interface. One big principle is not to break the kernel build in the middle of a patch series when altering an existing interface.
I see.
Quoted 9 lines
Quoted 8 lines
The stack area is allocated by DEFINE_NEON_REGSTACK[_PARTIAL](varname), where the partial version takes an additional int
num_regsindicating how many registers need to be freed up.In the !in_interrupt() case, these functions operate as before, and the regstack is defined to minimal size in this case as it will remain unused anyway. In the
in_interrupt()case, 'num_regs' (or all) NEON registers are stacked/unstacked using the allocated stack region.
Quoted 5 lines
Would have been nice to have the stack simply be a NULL pointer when !in_interrupt() or when the number of regs is 0. This would remove the need for a runtime check on !num_regs. I don't see an obvious way to accomplish that right now though.
We could address both of these issues by implementing Catalin's suggestion to reserve per-process vfp_states[] for both irq and softirq context in addition to the ordinary one, but it would waste a lot of space imo. What is your take on that?
Signature · 1 line
Ard.
v3 cover ARM[64]: kernel mode NEON in atomic contexts
Quoted 32 lines · Ard Biesheuvel
Quoted 6 lines · Nicolas Pitre
Quoted 3 lines · Ard Biesheuvel
Instead of having additional separate versions of kernel_neon_begin/end, the existing ones now have been modified to always take a preallocated stack area as an argument.
Quoted 7 lines
The problem with this approach is that you break git bisect by making the kernel unbuildable when this series is partially applied. Either you make kernel_neon_begin/end into wrappers with no argument around the new interface, or you change all users at the same time as the interface. One big principle is not to break the kernel build in the middle of a patch series when altering an existing interface.
I see.
Quoted 9 lines
Quoted 8 lines
The stack area is allocated by DEFINE_NEON_REGSTACK[_PARTIAL](varname), where the partial version takes an additional int
num_regsindicating how many registers need to be freed up.In the !in_interrupt() case, these functions operate as before, and the regstack is defined to minimal size in this case as it will remain unused anyway. In the
in_interrupt()case, 'num_regs' (or all) NEON registers are stacked/unstacked using the allocated stack region.Quoted 5 lines
Would have been nice to have the stack simply be a NULL pointer when !in_interrupt() or when the number of regs is 0. This would remove the need for a runtime check on !num_regs. I don't see an obvious way to accomplish that right now though.
Quoted 5 lines
We could address both of these issues by implementing Catalin's suggestion to reserve per-process vfp_states[] for both irq and softirq context in addition to the ordinary one, but it would waste a lot of space imo. What is your take on that?
Replying to self: two per-cpu vfp_states, one for irq and one for softirq, is probably the best approach here. I still need to add kernel_neon_begin_partial() in this case, but the existing users can remain unmodified.
I will do a v4 by end of next week.
Regards,
Ard.
v3 cover ARM[64]: kernel mode NEON in atomic contexts
In reply to Ard BiesheuvelQuoted 18 lines · Ard Biesheuvel
Quoted 11 lines · Nicolas Pitre
Quoted 8 lines · Ard Biesheuvel
The stack area is allocated by DEFINE_NEON_REGSTACK[_PARTIAL](varname), where the partial version takes an additional int
num_regsindicating how many registers need to be freed up.In the !in_interrupt() case, these functions operate as before, and the regstack is defined to minimal size in this case as it will remain unused anyway. In the
in_interrupt()case, 'num_regs' (or all) NEON registers are stacked/unstacked using the allocated stack region.Quoted 5 lines
Would have been nice to have the stack simply be a NULL pointer when !in_interrupt() or when the number of regs is 0. This would remove the need for a runtime check on !num_regs. I don't see an obvious way to accomplish that right now though.
Quoted 4 lines
We could address both of these issues by implementing Catalin's suggestion to reserve per-process vfp_states[] for both irq and softirq context in addition to the ordinary one, but it would waste a lot of space imo. What is your take on that?
I agree that this would be rather wasteful. I really like your current approach of dynamically allocating just the right amount of space on the stack. I'm not a big fan of statically allocated memory which is seldomly used.
What I meant by my suggestion was something like this:
#define kernel_neon_begin(p) \ __kernel_neon_begin(sizeof((p).qregs) ? &(p).regs : NULL, \ sizeof((p).qregs)/16)
However it seems gcc is not clever enough to optimize the stack usage away at all in that case which is worse than your current version. So better forget about this suggestion.
Nicolas
Catalin Marinas
maintainerv3 cover ARM[64]: kernel mode NEON in atomic contexts
Quoted 25 lines · Nicolas Pitre
Quoted 18 lines · Ard Biesheuvel
Quoted 11 lines · Nicolas Pitre
Quoted 8 lines · Ard Biesheuvel
The stack area is allocated by DEFINE_NEON_REGSTACK[_PARTIAL](varname), where the partial version takes an additional int
num_regsindicating how many registers need to be freed up.In the !in_interrupt() case, these functions operate as before, and the regstack is defined to minimal size in this case as it will remain unused anyway. In the
in_interrupt()case, 'num_regs' (or all) NEON registers are stacked/unstacked using the allocated stack region.Quoted 5 lines
Would have been nice to have the stack simply be a NULL pointer when !in_interrupt() or when the number of regs is 0. This would remove the need for a runtime check on !num_regs. I don't see an obvious way to accomplish that right now though.
Quoted 4 lines
We could address both of these issues by implementing Catalin's suggestion to reserve per-process vfp_states[] for both irq and softirq context in addition to the ordinary one, but it would waste a lot of space imo. What is your take on that?
Quoted 4 lines
I agree that this would be rather wasteful. I really like your current approach of dynamically allocating just the right amount of space on the stack. I'm not a big fan of statically allocated memory which is seldomly used.
I agree here, especially since we need to cover both soft and hard irqs. It would be about 1KB per CPU, not noticeable even on big systems but still looks like it's only going to be used rarely.
Signature · 1 line
Catalin