ARM: add support for kernel mode NEON/VFP
Subsystems (2)
PATCH 1/2 ARM: add support for kernel mode NEON/VFP
In order to safely support the use of NEON or VFP instructions in kernel mode, some precautions need to be taken:
- the userland context that may be present in the registers (even if the NEON/VFP is currently disabled) must be stored under the correct task (which may not be 'current' in the UP case),
- to avoid having to keep track of additional vfpstates for the kernel side, disallow the use of NEON/VFP in interrupt context and run with preemption disabled,
- after use, re-enable preemption and re-enable the lazy restore machinery by disabling the NEON/VFP unit.
This patch adds the functions kernel_vfp_begin() and kernel_vfp_end() which take care of the above.
--- arch/arm/include/asm/vfp.h | 5 +++++ arch/arm/vfp/vfpmodule.c | 39 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+)
arch/arm/include/asm/vfp.h +5 -0
@@ -82,3 +82,8 @@#define VFPOPDESC_UNUSED_BIT (24)#define VFPOPDESC_UNUSED_MASK (0xFF << VFPOPDESC_UNUSED_BIT)#define VFPOPDESC_OPDESC_MASK (~(VFPOPDESC_LENGTH_MASK | VFPOPDESC_UNUSED_MASK)) + +#ifndef __ASSEMBLY__ +void kernel_vfp_begin (void ); +void kernel_vfp_end (void ); +#endif
arch/arm/vfp/vfpmodule.c +39 -0
@@ -20,6 +20,7 @@#include <linux/init.h> #include <linux/uaccess.h> #include <linux/user.h> +#include <linux/export.h> #include <asm/cp15.h> #include <asm/cputype.h>
@@ -649,6 +650,44 @@ static int vfp_hotplug(struct notifier_block *b, unsigned long action, }/* +* Kernel-side NEON/VFP support functions +*/ +void kernel_vfp_begin (void ) +{ +struct thread_info *thread =current_thread_info (); +unsigned int cpu; +u32 fpexc; + +/* Avoid using the NEON/VFP in interrupt context */ +might_sleep (); + cpu =get_cpu (); + + fpexc =fmrx (FPEXC ) |FPEXC_EN ; +fmxr (FPEXC , fpexc); + +/* +* Save the userland NEON/VFP state. Under UP, the owner could be a task +* other than 'current' +*/ +if (vfp_state_in_hw (cpu, thread)) +vfp_save_state (&thread->vfpstate, fpexc); +#ifndef CONFIG_SMP + else if (vfp_current_hw_state[cpu] != NULL) + vfp_save_state(vfp_current_hw_state[cpu], fpexc); +#endif + vfp_current_hw_state[cpu] =NULL ; +} +EXPORT_SYMBOL (kernel_vfp_begin); + +void kernel_vfp_end (void ) +{ +/* Disable the NEON/VFP unit. */ +fmxr (FPEXC ,fmrx (FPEXC ) & ~FPEXC_EN ); +put_cpu (); +} +EXPORT_SYMBOL (kernel_vfp_end); + +/* * VFP support code initialisation. */ static int __init vfp_init (void )
Signature · 1 line
1.8.1.2
PATCH 2/2 ARM: move VFP init to an earlier boot stage
In order to use the NEON/VFP unit in the kernel, we should initialize it a bit earlier in the boot process so NEON users that like to do a quick benchmark at load time (like the xor_blocks or RAID-6 code) find the NEON/VFP unit already enabled.
Replaced late_initcall() with core_initcall().
--- arch/arm/vfp/vfpmodule.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/vfp/vfpmodule.c +1 -1
@@ -770,4 +770,4 @@ static int __init vfp_init(void)return 0 ; } -late_initcall (vfp_init); +core_initcall (vfp_init);
Signature · 1 line
1.8.1.2
PATCH 2/2 ARM: move VFP init to an earlier boot stage
Quoted 5 lines · Ard Biesheuvel
In order to use the NEON/VFP unit in the kernel, we should initialize it a bit earlier in the boot process so NEON users that like to do a quick benchmark at load time (like the
xor_blocksor RAID-6 code) find the NEON/VFP unit already enabled.
So what are the results of those benchmarks that the RAID-6 code does?
This isn't something which will get accepted without there being a very strong case for it; firstly, we've always said "no FP in the kernel". Secondly, it makes it too easy for people to start thinking that FP is safe in the kernel. It isn't - this will go horribly wrong if the VFP hardware bounces an instruction to the support code.
Lastly, what happens if you sleep between kernel_vfp_start() and kernel_vfp_end() ? That's a context switch which will mess up your VFP state. I think you need to take a spinlock (not _bh nor _irqsave versions) in here to ensure that people don't get the idea that they _can_ sleep between these two.
So, good attempt, but I think you need to think about this more, and also justify the need for it.
PATCH 2/2 ARM: move VFP init to an earlier boot stage
Quoted 9 lines · Russell King - ARM Linux
Quoted 5 lines · Ard Biesheuvel
In order to use the NEON/VFP unit in the kernel, we should initialize it a bit earlier in the boot process so NEON users that like to do a quick benchmark at load time (like the
xor_blocksor RAID-6 code) find the NEON/VFP unit already enabled.So what are the results of those benchmarks that the RAID-6 code does?
Output captured from a Cortex-A15 @ 1.7 GHz:
xor: measuring software checksum speed
arm4regs : 2261.600 MB/sec
8regs : 1771.600 MB/sec
32regs : 1441.600 MB/sec
neon : 3619.600 MB/sec
xor: using function: neon (3619.600 MB/sec)Output captured from a Cortex-A15 @ 1.7 GHz:
raid6: int32x1 200 MB/s
raid6: int32x2 304 MB/s
raid6: int32x4 388 MB/s
raid6: int32x8 423 MB/s
raid6: neonx1 799 MB/s
raid6: neonx2 1364 MB/s
raid6: neonx4 1731 MB/s
raid6: neonx8 1676 MB/s raid6: using algorithm neonx4 (1731 MB/s)
raid6: using intx1 recovery algorithm
I can put up both patches for RFC if you like.
Quoted 6 lines
This isn't something which will get accepted without there being a very strong case for it; firstly, we've always said "no FP in the kernel". Secondly, it makes it too easy for people to start thinking that FP is safe in the kernel. It isn't - this will go horribly wrong if the VFP hardware bounces an instruction to the support code.
That is clear. In the two cases I have been playing with, the begin/end calls are in distinct compilation units from the actual functions containing the NEON code. Perhaps we should enforce this by conditionally turning kernel_vfp_begin() into a macro which errors out (i.e., if __ARM_NEON__ is defined)?
Quoted 6 lines
Lastly, what happens if you sleep between
kernel_vfp_start()andkernel_vfp_end()? That's a context switch which will mess up your VFP state. I think you need to take a spinlock (not_bhnor_irqsaveversions) in here to ensure that people don't get the idea that they _can_ sleep between these two.
So generally, sleeping under preempt_disable() (or get cpu()) in this case) is not currently handled in that way?
That is a misunderstanding on my part, I will address it in the code.
Regards,
Signature · 1 line
Ard.
Quoted 2 lines
So, good attempt, but I think you need to think about this more, and also justify the need for it.
PATCH 2/2 ARM: move VFP init to an earlier boot stage
In reply to Russell King - ARM LinuxQuoted 11 lines · Russell King - ARM Linux
Quoted 5 lines · Ard Biesheuvel
In order to use the NEON/VFP unit in the kernel, we should initialize it a bit earlier in the boot process so NEON users that like to do a quick benchmark at load time (like the
xor_blocksor RAID-6 code) find the NEON/VFP unit already enabled.So what are the results of those benchmarks that the RAID-6 code does?
This isn't something which will get accepted without there being a very strong case for it; firstly, we've always said "no FP in the kernel".
That was my opinion as well. However the numbers are compelling.
And X86 and PPC do use FP in the kernel already for very specific tasks because of that.
Quoted 3 lines
Secondly, it makes it too easy for people to start thinking that FP is safe in the kernel. It isn't - this will go horribly wrong if the VFP hardware bounces an instruction to the support code.
Hmmm. good point. We probably should set a flag in kernel_vfp_begin() and BUG() if the support code is invoked while that flag is set.
Quoted 5 lines
Lastly, what happens if you sleep between
kernel_vfp_start()andkernel_vfp_end()? That's a context switch which will mess up your VFP state. I think you need to take a spinlock (not_bhnor_irqsaveversions) in here to ensure that people don't get the idea that they _can_ sleep between these two.
Isn't preemption off sufficient to flag that case? The scheduler is very noisy when schedule() is called with preemption disabled.
Nicolas
PATCH 2/2 ARM: move VFP init to an earlier boot stage
Quoted 9 lines · Nicolas Pitre
Quoted 3 lines · Russell King - ARM Linux
Secondly, it makes it too easy for people to start thinking that FP is safe in the kernel. It isn't - this will go horribly wrong if the VFP hardware bounces an instruction to the support code.
Hmmm. good point. We probably should set a flag in
kernel_vfp_begin()andBUG()if the support code is invoked while that flag is set.
Currently, the lazy restore is armed even in kernel mode, so it pulls in the userland context if you issue NEON/VFP instructions in kernel mode with the FP unit off. This is probably a side effect of the VFP detection code relying on this code path as well.
As kernel_vfp_begin() disarms the lazy restore and takes care of preserving the userland context itself, entering __und_svc because of NEON/VFP instructions should only be allowed in this particular case, i.e., VFP detection and all other instances should BUG() imho (regardless of if we end up doing NEON in the kernel or not). Doing it that way, I don't think there is a reason for keeping additional flags.
Quoted 9 lines
Quoted 5 lines
Lastly, what happens if you sleep between
kernel_vfp_start()andkernel_vfp_end()? That's a context switch which will mess up your VFP state. I think you need to take a spinlock (not_bhnor_irqsaveversions) in here to ensure that people don't get the idea that they _can_ sleep between these two.Isn't preemption off sufficient to flag that case? The scheduler is very noisy when
schedule()is called with preemption disabled.
That was my understanding as well.
Signature · 1 line
Ard.
PATCH 2/2 ARM: move VFP init to an earlier boot stage
Quoted 11 lines · Ard Biesheuvel
Quoted 9 lines · Nicolas Pitre
Quoted 3 lines · Russell King - ARM Linux
Secondly, it makes it too easy for people to start thinking that FP is safe in the kernel. It isn't - this will go horribly wrong if the VFP hardware bounces an instruction to the support code.
Hmmm. good point. We probably should set a flag in
kernel_vfp_begin()andBUG()if the support code is invoked while that flag is set.
Quoted 11 lines
Currently, the lazy restore is armed even in kernel mode, so it pulls in the userland context if you issue NEON/VFP instructions in kernel mode with the FP unit off. This is probably a side effect of the VFP detection code relying on this code path as well.
Askernel_vfp_begin()disarms the lazy restore and takes care of preserving the userland context itself, entering__und_svcbecause of NEON/VFP instructions should only be allowed in this particular case, i.e., VFP detection and all other instances shouldBUG()imho (regardless of if we end up doing NEON in the kernel or not). Doing it that way, I don't think there is a reason for keeping additional flags.
The concern is more about VFP instructions that could trigger an exception where support code is invoked to emulate that instruction.
Nicolas
PATCH 2/2 ARM: move VFP init to an earlier boot stage
Quoted 13 lines · Nicolas Pitre
Quoted 11 lines · Ard Biesheuvel
Currently, the lazy restore is armed even in kernel mode, so it pulls in the userland context if you issue NEON/VFP instructions in kernel mode with the FP unit off. This is probably a side effect of the VFP detection code relying on this code path as well.
Askernel_vfp_begin()disarms the lazy restore and takes care of preserving the userland context itself, entering__und_svcbecause of NEON/VFP instructions should only be allowed in this particular case, i.e., VFP detection and all other instances shouldBUG()imho (regardless of if we end up doing NEON in the kernel or not). Doing it that way, I don't think there is a reason for keeping additional flags.
Quoted 3 lines
The concern is more about VFP instructions that could trigger an exception where support code is invoked to emulate that instruction.
So perhaps that is where we should draw the line: I am not proposing supporting arbitrary VFP code in the kernel, just NEON (which does not bounce and hence does not require the support code). Maybe I should rename to kernel_neon_begin/kernel_neon_end to reflect that?
Regards,
Ard.