ARM crc64 and XOR using NEON intrinsics
Subsystems (5)
cover ARM crc64 and XOR using NEON intrinsics
From: Ard Biesheuvel <[email protected]>
This is a follow-up to both [0] and [1], both of which included patch #1 of this series, which introduces the asm/neon-intrinsics.h header on 32-bit ARM. The remaining changes rely on this.
The purpose of this series is to streamline / clean up the use of NEON intrinsics on 32-bit ARM, by sharing more code, clean up Make rules and finally, getting rid of the hacked up types.h header, which does some nasty things that are only needed when building NEON intrinsics code.
Patches #2 and #3 replace the ARM autovectorized XOR implementation with the NEON intrinsics version used by arm64.
Patches #4 and #5 enable the arm64 NEON intrinsics implementation of crc64 on 32-bit ARM.
Patches #6 and #7 drop the direct includes of <arm_neon.h> and perform some additional cleanup to reduce the delta between ARM and arm64 code and Make rules.
It would probably be easiest to take all these changes through a single tree, and the CRC tree seems like a suitable candidate, if Eric agrees.
[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/
Ard Biesheuvel (8):
ARM: Add a neon-intrinsics.h header like on arm64 xor/arm: Replace vectorized implementation with arm64's intrinsics xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM lib/crc: Turn NEON intrinsics crc64 implementation into common code lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64
crypto: aegis128 - Use neon-intrinsics.h on ARM too lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
ARM: Remove hacked-up asm/types.h header
Documentation/arch/arm/kernel_mode_neon.rst | 4 +-
arch/arm/include/asm/neon-intrinsics.h | 60 ++++++++
arch/arm/include/uapi/asm/types.h | 41 ------
crypto/Makefile | 10 +-
crypto/aegis128-neon-inner.c | 4 +-
lib/crc/Kconfig | 1 +
lib/crc/Makefile | 9 +-
lib/crc/arm/crc64-neon.h | 34 +++++
lib/crc/arm/crc64.h | 36 +++++
lib/crc/arm64/crc64-neon.h | 21 +++
lib/crc/arm64/crc64.h | 4 +-
lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} | 26 +---
lib/raid/xor/Makefile | 13 +-
lib/raid/xor/arm/xor-neon.c | 26 ----
lib/raid/xor/arm/xor-neon.h | 7 +
lib/raid/xor/arm/xor_arch.h | 7 +-
lib/raid/xor/arm64/xor-eor3.c | 146 ++++++++++++++++++++
lib/raid/xor/xor-8regs.c | 2 -
lib/raid/xor/{arm64 => }/xor-neon.c | 143 +------------------
lib/raid6/neon.uc | 2 +-
lib/raid6/recov_neon_inner.c | 2 +-
21 files changed, 340 insertions(+), 258 deletions(-)
create mode 100644 arch/arm/include/asm/neon-intrinsics.h
delete mode 100644 arch/arm/include/uapi/asm/types.h
create mode 100644 lib/crc/arm/crc64-neon.h
create mode 100644 lib/crc/arm/crc64.h
create mode 100644 lib/crc/arm64/crc64-neon.h
rename lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} (62%)
delete mode 100644 lib/raid/xor/arm/xor-neon.c
create mode 100644 lib/raid/xor/arm/xor-neon.h
create mode 100644 lib/raid/xor/arm64/xor-eor3.c
rename lib/raid/xor/{arm64 => }/xor-neon.c (56%)base-commit: 6596a02b207886e9e00bb0161c7fd59fea53c081
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
PATCH 1/8 ARM: Add a neon-intrinsics.h header like on arm64
From: Ard Biesheuvel <[email protected]>
Add a header asm/neon-intrinsics.h similar to the one that arm64 has. This makes it possible for NEON intrinsics code to be shared seamlessly between ARM and arm64.
--- Documentation/arch/arm/kernel_mode_neon.rst | 4 +- arch/arm/include/asm/neon-intrinsics.h | 60 ++++++++++++++++++++ 2 files changed, 63 insertions(+), 1 deletion(-)
Documentation/arch/arm/kernel_mode_neon.rst +3 -1
@@ -121,4 +121,6 @@ observe the following in addition to the rules above: * Compile the unit containing the NEON intrinsics with '-ffreestanding' so GCC uses its builtin version of <stdint.h> (this is a C99 header which the kernel does not supply); -* Include <arm_neon.h> last, or at least after <linux/types.h> +* Do not include <arm_neon.h> directly: instead, include <asm/neon-intrinsics.h>, + which tweaks some macro definitions so that system headers can be included + safely.
arch/arm/include/asm/neon-intrinsics.h new file +60 -0
60 lines not on this page · raw message
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
PATCH 2/8 xor/arm: Replace vectorized implementation with arm64's intrinsics
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Drop the XOR implementation generated by the vectorizer: this has always been a bit of a hack, and now that arm64 has an intrinsics version that works on ARM too, let's use that instead.
So copy the part of the arm64 code that can be shared (so not the EOR3 version). The arm64 code will be updated in a subsequent patch to share this implementation.
Performance (QEMU mach-virt VM running on Synquacer [Cortex-A53 @ 1 GHz]
Before:
[ 3.519687] xor: measuring software checksum speed [ 3.521725] neon : 1660 MB/sec [ 3.524733] 32regs : 1105 MB/sec [ 3.527751] 8regs : 1098 MB/sec [ 3.529911] arm4regs : 1540 MB/sec
After:
[ 3.517654] xor: measuring software checksum speed [ 3.519454] neon : 1896 MB/sec [ 3.522499] 32regs : 1090 MB/sec [ 3.525560] 8regs : 1083 MB/sec [ 3.527700] arm4regs : 1556 MB/sec
--- lib/raid/xor/Makefile | 6 +- lib/raid/xor/arm/xor-neon.c | 26 --- lib/raid/xor/arm/xor-neon.h | 7 + lib/raid/xor/arm/xor_arch.h | 7 +- lib/raid/xor/xor-8regs.c | 2 - lib/raid/xor/xor-neon.c | 175 ++++++++++++++++++++ 6 files changed, 186 insertions(+), 37 deletions(-)
lib/raid/xor/Makefile +3 -3
@@ -17,7 +17,7 @@ endif xor-$(CONFIG_ALPHA) += alpha/xor.o xor-$(CONFIG_ARM) += arm/xor.o ifeq ($(CONFIG_ARM),y) -xor-$(CONFIG_KERNEL_MODE_NEON) += arm/xor-neon.o arm/xor-neon-glue.o +xor-$(CONFIG_KERNEL_MODE_NEON) += xor-neon.o arm/xor-neon-glue.o endif xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o
@@ -31,8 +31,8 @@ xor-$(CONFIG_X86_32) += x86/xor-avx.o x86/xor-sse.o x86/xor-mmx.o xor-$(CONFIG_X86_64) += x86/xor-avx.o x86/xor-sse.o obj-y += tests/ -CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU) -CFLAGS_REMOVE_arm/xor-neon.o += $(CC_FLAGS_NO_FPU) +CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) +CFLAGS_REMOVE_xor-neon.o += $(CC_FLAGS_NO_FPU) CFLAGS_arm64/xor-neon.o += $(CC_FLAGS_FPU) CFLAGS_REMOVE_arm64/xor-neon.o += $(CC_FLAGS_NO_FPU)
lib/raid/xor/arm/xor-neon.c deleted +0 -26
@@ -1,26 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* -* Copyright (C) 2013 Linaro Ltd <[email protected]> -*/ - -#include "xor_impl.h" -#include "xor_arch.h" - -#ifndef __ARM_NEON__ -#error You should compile this file with '-march=armv7-a -mfloat-abi=softfp -mfpu=neon' -#endif - -/* -* Pull in the reference implementations while instructing GCC (through -* -ftree-vectorize) to attempt to exploit implicit parallelism and emit -* NEON instructions. Clang does this by default at O2 so no pragma is -* needed. -*/ -#ifdef CONFIG_CC_IS_GCC -#pragma GCC optimize "tree-vectorize" -#endif - -#define NO_TEMPLATE -#include "../xor-8regs.c" - -__DO_XOR_BLOCKS (neon_inner, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);
lib/raid/xor/arm/xor-neon.h new file +7 -0
@@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +extern struct xor_block_template xor_block_arm4regs; +extern struct xor_block_template xor_block_neon; + +void xor_gen_neon_inner (void *dest,void **srcs,unsigned int src_cnt, +unsigned int bytes);
lib/raid/xor/arm/xor_arch.h +1 -6
@@ -3,12 +3,7 @@* Copyright (C) 2001 Russell King */ #include <asm/neon.h> - -extern struct xor_block_template xor_block_arm4regs; -extern struct xor_block_template xor_block_neon; - -void xor_gen_neon_inner (void *dest,void **srcs,unsigned int src_cnt, -unsigned int bytes); +#include "xor-neon.h" static __always_inline void __init arch_xor_init(void) {
lib/raid/xor/xor-8regs.c +0 -2
@@ -93,11 +93,9 @@ xor_8regs_5(unsigned long bytes, unsigned long * __restrict p1, }while (--lines >0 ); } -#ifndef NO_TEMPLATE DO_XOR_BLOCKS(8regs, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);struct xor_block_template xor_block_8regs = { .name ="8regs" , .xor_gen = xor_gen_8regs, }; -#endif /* NO_TEMPLATE */
lib/raid/xor/xor-neon.c new file +175 -0
175 lines not on this page · raw message
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
Re: [PATCH 2/8] xor/arm: Replace vectorized implementation with arm64's intrinsics
Hi ard.
I like this patch.
So, I'd be crazy not to say what I love here.
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0)); + v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2)); + v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4)); + v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6)); + + /* p1 ^= p3 */ + v0 = veorq_u64(v0, vld1q_u64(dp3 + 0)); + v1 = veorq_u64(v1, vld1q_u64(dp3 + 2)); + v2 = veorq_u64(v2, vld1q_u64(dp3 + 4)); + v3 = veorq_u64(v3, vld1q_u64(dp3 + 6)); + + /* store */ + vst1q_u64(dp1 + 0, v0); + vst1q_u64(dp1 + 2, v1); + vst1q_u64(dp1 + 4, v2); + vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ } while (--lines > 0);
+}
I really like how clean this is, I'm personally nodding my head here
Taking the "bad" guesswork of the compiler here is also amazing, it also guarantees we won't get stupid regressions in the future.
Also, that performance boost is even better ;)
I'm not the biggest expert of this subdirectory, but I understand it well.
So well,
Thanks! (I will review your lib patches) :)
Re: [PATCH 2/8] xor/arm: Replace vectorized implementation with arm64's intrinsics
In reply to Ard BiesheuvelNice!
PATCH 3/8 xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Tweak the arm64 code so that the pure NEON intrinsics implementation of XOR is shared between arm64 and ARM. While at it, rename the arm64 specific piece xor-eor3.c to reflect that only the version based on the EOR3 instruction is kept there.
--- lib/raid/xor/Makefile | 7 +- lib/raid/xor/arm64/xor-eor3.c | 146 +++++++++ lib/raid/xor/arm64/xor-neon.c | 312 -------------------- lib/raid/xor/xor-neon.c | 4 + 4 files changed, 154 insertions(+), 315 deletions(-)
lib/raid/xor/Makefile +4 -3
@@ -19,7 +19,8 @@ xor-$(CONFIG_ARM) += arm/xor.o ifeq ($(CONFIG_ARM),y) xor-$(CONFIG_KERNEL_MODE_NEON) += xor-neon.o arm/xor-neon-glue.o endif -xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o +xor-$(CONFIG_ARM64) += xor-neon.o arm64/xor-eor3.o \ + arm64/xor-neon-glue.o xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd_glue.o xor-$(CONFIG_ALTIVEC) += powerpc/xor_vmx.o powerpc/xor_vmx_glue.o
@@ -34,8 +35,8 @@ obj-y += tests/ CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) CFLAGS_REMOVE_xor-neon.o += $(CC_FLAGS_NO_FPU) -CFLAGS_arm64/xor-neon.o += $(CC_FLAGS_FPU) -CFLAGS_REMOVE_arm64/xor-neon.o += $(CC_FLAGS_NO_FPU) +CFLAGS_arm64/xor-eor3.o += $(CC_FLAGS_FPU) +CFLAGS_REMOVE_arm64/xor-eor3.o += $(CC_FLAGS_NO_FPU) CFLAGS_powerpc/xor_vmx.o += -mhard-float -maltivec \ $(call cc-option,-mabi=altivec) \
lib/raid/xor/arm64/xor-eor3.c new file +146 -0
146 lines not on this page · raw message
lib/raid/xor/arm64/xor-neon.c deleted +0 -312
312 lines not on this page · raw message
lib/raid/xor/xor-neon.c +4 -0
@@ -173,3 +173,7 @@ static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,__DO_XOR_BLOCKS (neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4, __xor_neon_5); + +#ifdef CONFIG_ARM64 +extern typeof(__xor_neon_2) __xor_eor3_2 __alias(__xor_neon_2); +#endif
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
Re: [PATCH 3/8] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
Hi ard.
lib/raid/xor/xor-neon.c static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1, · line 179
Quoted 3 lines
lib/raid/xor/xor-neon.c +3
-0+#ifdef CONFIG_ARM64 +extern typeof(__xor_neon_2) >__xor_eor3_2 >__alias(__xor_neon_2); +#endif
Creative. A reduction of about 150 lines of duplicate code while maintaining
the __alias for the 2 input case is great.
Thanks!
Re: [PATCH 3/8] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
In reply to Ard Biesheuvellib/raid/xor/arm64/xor-eor3.c line 10
Quoted 2 lines
lib/raid/xor/arm64/xor-eor3.c new file +2
-0+extern void __xor_eor3_2 (unsigned long bytes,unsigned long * __restrict p1, +const unsigned long * __restrict p2);
Does the alias magic prevent this from being in a header? If so a comment would be nice, otherwise moving it to a header would be even better.
Ard Biesheuvel
reviewerRe: [PATCH 3/8] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
Quoted 4 lines · Christoph Hellwig
lib/raid/xor/arm64/xor-eor3.c line 10
Quoted 2 lines
lib/raid/xor/arm64/xor-eor3.c new file +2
-0+extern void __xor_eor3_2 (unsigned long bytes,unsigned long * __restrict p1, +const unsigned long * __restrict p2);Does the alias magic prevent this from being in a header?
Yes, it emits the ELF symbol for the alias, and this is only permitted in the compilation unit that defines the original.
Quoted 2 lines
If so a comment
would be nice, otherwise moving it to a header would be even better.
Ack.
PATCH 4/8 lib/crc: Turn NEON intrinsics crc64 implementation into common code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Move and rename the CRC64 NEON intrinsics implementation source file and rename the function name to reflect that it is NEON code that can be shared. This will be wired up for 32-bit ARM in a subsequent patch.
---
lib/crc/Makefile | 6 ++---
lib/crc/arm64/crc64-neon.h | 21 ++++++++++++++++
lib/crc/arm64/crc64.h | 4 +--
lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} | 26 +++-----------------
4 files changed, 30 insertions(+), 27 deletions(-)lib/crc/Makefile +3 -3
@@ -39,9 +39,9 @@ crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) CFLAGS_crc64-main.o += -I$(src)/$(SRCARCH) -CFLAGS_REMOVE_arm64/crc64-neon-inner.o += $(CC_FLAGS_NO_FPU) -CFLAGS_arm64/crc64-neon-inner.o += $(CC_FLAGS_FPU) -march=armv8-a+crypto -crc64-$(CONFIG_ARM64) += arm64/crc64-neon-inner.o +CFLAGS_REMOVE_crc64-neon.o += $(CC_FLAGS_NO_FPU) +CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) -march=armv8-a+crypto +crc64-$(CONFIG_ARM64) += crc64-neon.o crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o crc64-$(CONFIG_X86) += x86/crc64-pclmul.o
lib/crc/arm64/crc64-neon.h new file +21 -0
@@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0-only + +static inline uint64x2_t pmull64 (uint64x2_t a,uint64x2_t b) +{ +return vreinterpretq_u64_p128 (vmull_p64 (vgetq_lane_u64 (a,0 ), +vgetq_lane_u64 (b,0 ))); +} + +static inline uint64x2_t pmull64_high (uint64x2_t a,uint64x2_t b) +{ +poly64x2_t l =vreinterpretq_p64_u64 (a); +poly64x2_t m =vreinterpretq_p64_u64 (b); + +return vreinterpretq_u64_p128 (vmull_high_p64 (l, m)); +} + +static inline uint64x2_t pmull64_hi_lo (uint64x2_t a,uint64x2_t b) +{ +return vreinterpretq_u64_p128 (vmull_p64 (vgetq_lane_u64 (a,1 ), +vgetq_lane_u64 (b,0 ))); +}
lib/crc/arm64/crc64.h +2 -2
@@ -8,7 +8,7 @@#include <linux/minmax.h> #include <linux/sizes.h> -u64 crc64_nvme_arm64_c (u64 crc,const u8 *p,size_t len); +u64 crc64_nvme_neon (u64 crc,const u8 *p,size_t len);#define crc64_be_arch crc64_be_generic
@@ -19,7 +19,7 @@ static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)size_t chunk = len & ~15 ;scoped_ksimd () - crc = crc64_nvme_arm64_c(crc, p, chunk); + crc = crc64_nvme_neon(crc, p, chunk); p += chunk; len &=15 ;
lib/crc/crc64-neon.c renamed from lib/crc/arm64/crc64-neon-inner.c +4 -22
@@ -6,7 +6,9 @@#include <linux/types.h> #include <asm/neon-intrinsics.h> -u64 crc64_nvme_arm64_c (u64 crc,const u8 *p,size_t len); +#include "crc64-neon.h" + +u64 crc64_nvme_neon (u64 crc,const u8 *p,size_t len);/* x^191 mod G, x^127 mod G */ static const u64 fold_consts_val[2 ] = {0xeadc41fd2ba3d420ULL ,
@@ -15,27 +17,7 @@ static const u64 fold_consts_val[2] = { 0xeadc41fd2ba3d420ULL,static const u64 bconsts_val[2 ] = {0x27ecfa329aef9f77ULL ,0x34d926535897936aULL }; -static inline uint64x2_t pmull64 (uint64x2_t a,uint64x2_t b) -{ -return vreinterpretq_u64_p128 (vmull_p64 (vgetq_lane_u64 (a,0 ), -vgetq_lane_u64 (b,0 ))); -} - -static inline uint64x2_t pmull64_high (uint64x2_t a,uint64x2_t b) -{ -poly64x2_t l =vreinterpretq_p64_u64 (a); -poly64x2_t m =vreinterpretq_p64_u64 (b); - -return vreinterpretq_u64_p128 (vmull_high_p64 (l, m)); -} - -static inline uint64x2_t pmull64_hi_lo (uint64x2_t a,uint64x2_t b) -{ -return vreinterpretq_u64_p128 (vmull_p64 (vgetq_lane_u64 (a,1 ), -vgetq_lane_u64 (b,0 ))); -} - -u64 crc64_nvme_arm64_c (u64 crc,const u8 *p,size_t len) +u64 crc64_nvme_neon (u64 crc,const u8 *p,size_t len) {uint64x2_t fold_consts =vld1q_u64 (fold_consts_val);uint64x2_t v0 = { crc,0 };
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
Re: [PATCH 4/8] lib/crc: Turn NEON intrinsics crc64 implementation into common code
Hi ard.
lib/crc/arm64/crc64-neon.h new file +4 -0
@@ -0,0 +1,21 @@ +// SPDX-License-Identifier: GPL-2.0-only + +static inline uint64x2_t pmull64 (uint64x2_t a,uint64x2_t b) +{
return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 0), vgetq_lane_u64(b, 0)));
+}
+static inline uint64x2_t pmull64_high(uint64x2_t a, uint64x2_t b) +{poly64x2_t l = vreinterpretq_p64_u64(a);poly64x2_t m = vreinterpretq_p64_u64(b);
return vreinterpretq_u64_p128(vmull_high_p64(l, m));
+}
Makes sense.
Moving these polynomial multiplication wrappers into their own header is good.
Thanks!
PATCH 5/8 lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Tweak the NEON intrinsics crc64 code written for arm64 so it can be built for 32-bit ARM as well. The only workaround needed is to provide alternatives for vmull_p64() and vmull_high_p64() on Clang, which only defines those when building for the AArch64 or arm64ec ISA. Use the same helpers for GCC too, to avoid doubling the size of the test/validation matrix.
KUnit benchmark results (Cortex-A53 @ 1 Ghz)
Before:
# crc64_nvme_benchmark: len=1: 35 MB/s
# crc64_nvme_benchmark: len=16: 78 MB/s
# crc64_nvme_benchmark: len=64: 87 MB/s
# crc64_nvme_benchmark: len=127: 88 MB/s
# crc64_nvme_benchmark: len=128: 88 MB/s
# crc64_nvme_benchmark: len=200: 89 MB/s
# crc64_nvme_benchmark: len=256: 89 MB/s
# crc64_nvme_benchmark: len=511: 89 MB/s
# crc64_nvme_benchmark: len=512: 89 MB/s
# crc64_nvme_benchmark: len=1024: 90 MB/s
# crc64_nvme_benchmark: len=3173: 90 MB/s
# crc64_nvme_benchmark: len=4096: 90 MB/s
# crc64_nvme_benchmark: len=16384: 90 MB/s
After:
# crc64_nvme_benchmark: len=1: 32 MB/s
# crc64_nvme_benchmark: len=16: 76 MB/s
# crc64_nvme_benchmark: len=64: 71 MB/s
# crc64_nvme_benchmark: len=127: 88 MB/s
# crc64_nvme_benchmark: len=128: 618 MB/s
# crc64_nvme_benchmark: len=200: 542 MB/s
# crc64_nvme_benchmark: len=256: 920 MB/s
# crc64_nvme_benchmark: len=511: 836 MB/s
# crc64_nvme_benchmark: len=512: 1261 MB/s
# crc64_nvme_benchmark: len=1024: 1531 MB/s
# crc64_nvme_benchmark: len=3173: 1731 MB/s
# crc64_nvme_benchmark: len=4096: 1851 MB/s
# crc64_nvme_benchmark: len=16384: 1858 MB/s
Don't bother with big-endian, as it doesn't work correctly on Clang, and is barely used these days.
Note that ARM disables preemption and softirq processing when using kernel mode SIMD, so take care not to hog the CPU for too long.
--- lib/crc/Kconfig | 1 + lib/crc/Makefile | 5 ++- lib/crc/arm/crc64-neon.h | 34 ++++++++++++++++++ lib/crc/arm/crc64.h | 36 ++++++++++++++++++++ 4 files changed, 75 insertions(+), 1 deletion(-)
lib/crc/Kconfig +1 -0
@@ -82,6 +82,7 @@ config CRC64 config CRC64_ARCH bool depends on CRC64 && CRC_OPTIMIZATIONS + default y if ARM && KERNEL_MODE_NEON && !CPU_BIG_ENDIAN default y if ARM64 default y if RISCV && RISCV_ISA_ZBC && 64BIT default y if X86_64
lib/crc/Makefile +4 -1
@@ -39,8 +39,11 @@ crc64-y := crc64-main.o ifeq ($(CONFIG_CRC64_ARCH),y) CFLAGS_crc64-main.o += -I$(src)/$(SRCARCH) +crc64-cflags-$(CONFIG_ARM) += -march=armv8-a -mfpu=crypto-neon-fp-armv8 +crc64-cflags-$(CONFIG_ARM64) += -march=armv8-a+crypto CFLAGS_REMOVE_crc64-neon.o += $(CC_FLAGS_NO_FPU) -CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) -march=armv8-a+crypto +CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) $(crc64-cflags-y) +crc64-$(CONFIG_ARM) += crc64-neon.o crc64-$(CONFIG_ARM64) += crc64-neon.o crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o
lib/crc/arm/crc64-neon.h new file +34 -0
@@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0-only + +static inline uint64x2_t pmull64 (uint64x2_t a,uint64x2_t b) +{ +uint64_t l =vgetq_lane_u64 (a,0 ); +uint64_t m =vgetq_lane_u64 (b,0 ); +uint64x2_t result; + + asm("vmull.p64 %q0, %P1, %P2" :"=w" (result) :"w" (l),"w" (m)); + +return result; +} + +static inline uint64x2_t pmull64_high (uint64x2_t a,uint64x2_t b) +{ +uint64_t l =vgetq_lane_u64 (a,1 ); +uint64_t m =vgetq_lane_u64 (b,1 ); +uint64x2_t result; + + asm("vmull.p64 %q0, %P1, %P2" :"=w" (result) :"w" (l),"w" (m)); + +return result; +} + +static inline uint64x2_t pmull64_hi_lo (uint64x2_t a,uint64x2_t b) +{ +uint64_t l =vgetq_lane_u64 (a,1 ); +uint64_t m =vgetq_lane_u64 (b,0 ); +uint64x2_t result; + + asm("vmull.p64 %q0, %P1, %P2" :"=w" (result) :"w" (l),"w" (m)); + +return result; +}
lib/crc/arm/crc64.h new file +36 -0
@@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* CRC64 using ARM PMULL instructions +*/ + +#include <asm/simd.h> + +static __ro_after_init DEFINE_STATIC_KEY_FALSE (have_pmull); + +u64 crc64_nvme_neon (u64 crc,const u8 *p,size_t len); + +#define crc64_be_arch crc64_be_generic + +static inline u64 crc64_nvme_arch (u64 crc,const u8 *p,size_t len) +{ +if (len >=128 &&static_branch_likely (&have_pmull) && +likely (may_use_simd ())) { +do { +size_t chunk =min_t (size_t, len & ~15 ,SZ_4K ); + +scoped_ksimd () + crc = crc64_nvme_neon(crc, p, chunk); + + p += chunk; + len -= chunk; + }while (len >=128 ); + } +return crc64_nvme_generic (crc, p, len); +} + +#define crc64_mod_init_arch crc64_mod_init_arch +static void crc64_mod_init_arch (void ) +{ +if (elf_hwcap2 &HWCAP2_PMULL ) +static_branch_enable (&have_pmull); +}
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
Re: [PATCH 5/8] lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64
Hi Ard,
Wow, 20x improvement is nuts.
I like how you handle this change *safely*
Like.
+static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len) +{
+ if (len >= 128 && static_branch_likely(&have_pmull) &&
+ likely(may_use_simd())) {
+ do {
+ size_t chunk = min_t(size_t, len & ~15, SZ_4K);
+
+ scoped_ksimd()
+ crc = crc64_nvme_neon(crc, p, chunk);
+
+ p += chunk;
+ len -= chunk;
+ } while (len >= 128);
+ }
chunking the SIMD work at SZ_4K to avoid hogging the CPU and allowing softirqs/preemption to process is a great detail.
It’s easy to just wing it and throw
the entire buffer at the FPU, but respecting the kernel's latency requirements is better!
Thanks!
PATCH 6/8 crypto: aegis128 - Use neon-intrinsics.h on ARM too
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Use the asm/neon-intrinsics.h header on ARM as well as arm64, so that the calling code does not have to know the difference.
Clean up the Makefile a bit while at it.
--- crypto/Makefile | 10 ++++------ crypto/aegis128-neon-inner.c | 4 +--- 2 files changed, 5 insertions(+), 9 deletions(-)
crypto/Makefile +4 -6
@@ -103,13 +103,14 @@ obj-$(CONFIG_CRYPTO_CHACHA20POLY1305) += chacha20poly1305.o obj-$(CONFIG_CRYPTO_AEGIS128) += aegis128.o aegis128-y := aegis128-core.o +CFLAGS_aegis128-neon-inner.o += $(CC_FLAGS_FPU) +CFLAGS_REMOVE_aegis128-neon-inner.o += $(CC_FLAGS_NO_FPU) ifeq ($(ARCH),arm) -CFLAGS_aegis128-neon-inner.o += -ffreestanding -march=armv8-a -mfloat-abi=softfp -CFLAGS_aegis128-neon-inner.o += -mfpu=crypto-neon-fp-armv8 +CFLAGS_aegis128-neon-inner.o += -march=armv8-a -mfpu=crypto-neon-fp-armv8 aegis128-$(CONFIG_CRYPTO_AEGIS128_SIMD) += aegis128-neon.o aegis128-neon-inner.o endif ifeq ($(ARCH),arm64) -aegis128-cflags-y := -ffreestanding -mcpu=generic+crypto +aegis128-cflags-y := -mcpu=generic+crypto aegis128-cflags-$(CONFIG_CC_IS_GCC) += -ffixed-q16 -ffixed-q17 -ffixed-q18 \ -ffixed-q19 -ffixed-q20 -ffixed-q21 \ -ffixed-q22 -ffixed-q23 -ffixed-q24 \
@@ -117,11 +118,8 @@ aegis128-cflags-$(CONFIG_CC_IS_GCC) += -ffixed-q16 -ffixed-q17 -ffixed-q18 \ -ffixed-q28 -ffixed-q29 -ffixed-q30 \ -ffixed-q31 CFLAGS_aegis128-neon-inner.o += $(aegis128-cflags-y) -CFLAGS_REMOVE_aegis128-neon-inner.o += -mgeneral-regs-only aegis128-$(CONFIG_CRYPTO_AEGIS128_SIMD) += aegis128-neon.o aegis128-neon-inner.o endif -# Enable <arm_neon.h> -CFLAGS_aegis128-neon-inner.o += -isystem $(shell $(CC) -print-file-name=include) obj-$(CONFIG_CRYPTO_PCRYPT) += pcrypt.o obj-$(CONFIG_CRYPTO_CRYPTD) += cryptd.o
crypto/aegis128-neon-inner.c +1 -3
@@ -3,13 +3,11 @@* Copyright (C) 2019 Linaro, Ltd. <[email protected]> */ -#ifdef CONFIG_ARM64 #include <asm/neon-intrinsics.h> +#ifdef CONFIG_ARM64 #define AES_ROUND "aese %0.16b, %1.16b \n\t aesmc %0.16b, %0.16b"#else -#include <arm_neon.h> -#define AES_ROUND "aese.8 %q0, %q1 \n\t aesmc.8 %q0, %q0"#endif
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
Re: [PATCH 6/8] crypto: aegis128 - Use neon-intrinsics.h on ARM too
Hi ard, this is a good cleanup!
Being able to drop <arm_neon.h> and just using <asm/neon-intrinsics.h> across both architectures makes the C code much cleaner.
-# Enable <arm_neon.h>
-CFLAGS_aegis128-neon-inner.o += -isystem $(shell $(CC) -print-file-name=include)
Getting rid of the isystem is good. iirc that was a hack anyway, feel free to correct me on that
PATCH 7/8 lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
arm_neon.h is a compiler header which needs some scaffolding to work correctly in the linux context, and so it is better not to include it directly. Both ARM and arm64 now provide asm/neon-intrinsics.h which takes care of this.
--- lib/raid6/neon.uc | 2 +- lib/raid6/recov_neon_inner.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
lib/raid6/neon.uc +1 -1
@@ -24,7 +24,7 @@ * This file is postprocessed using unroll.awk */ -#include <arm_neon.h> +#include <asm/neon-intrinsics.h> #include "neon.h" typedef uint8x16_t unative_t;
lib/raid6/recov_neon_inner.c +1 -1
@@ -4,7 +4,7 @@* Copyright (C) 2017 Linaro Ltd. <[email protected]> */ -#include <arm_neon.h> +#include <asm/neon-intrinsics.h> #include "neon.h" #ifdef CONFIG_ARM
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
Re: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
Hi ard.
Makes sense here
-#include <arm_neon.h>
+#include <asm/neon-intrinsics.h>
This series is a good (and deserved series)
That's me done! I've reviewed your lib patches for you, have a great day!
Re: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
In reply to Ard BiesheuvelQuoted 6 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
arm_neon.his a compiler header which needs some scaffolding to work correctly in the linux context, and so it is better not to include it directly. Both ARM and arm64 now provideasm/neon-intrinsics.hwhich takes care of this.
This could potentially clash with the raid6 library rework I'm doing for 7.2. Although git has become pretty good about renamed files, so maybe it won't be so bad.
Eric Biggers
maintainerRe: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
Quoted 9 lines · Christoph Hellwig
Quoted 6 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
arm_neon.his a compiler header which needs some scaffolding to work correctly in the linux context, and so it is better not to include it directly. Both ARM and arm64 now provideasm/neon-intrinsics.hwhich takes care of this.
Quoted 4 lines
This could potentially clash with the raid6 library rework I'm doing for 7.2. Although git has become pretty good about renamed files, so maybe it won't be so bad.
I think this patch also breaks the userspace build of lib/raid6/. Which is going away in Christoph's series anyway, but maybe it would make sense to drop this patch (and patch 8 which depends on this, I think) from this series for now? That would make it a bit easier to take the rest through crc-next.
- Eric
Re: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
Quoted 2 lines · Eric Biggers
I think this patch also breaks the userspace build of
lib/raid6/. Which is going away in Christoph's series anyway,
Assuming we're overcoming the objections. Anyway, about to repost this, and maybe Art is another voice for dropping the userspace build support of the RAID code.
Eric Biggers
maintainerRe: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
In reply to Eric BiggersQuoted 15 lines · Eric Biggers
Quoted 9 lines · Christoph Hellwig
Quoted 6 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
arm_neon.his a compiler header which needs some scaffolding to work correctly in the linux context, and so it is better not to include it directly. Both ARM and arm64 now provideasm/neon-intrinsics.hwhich takes care of this.Quoted 4 lines
This could potentially clash with the raid6 library rework I'm doing for 7.2. Although git has become pretty good about renamed files, so maybe it won't be so bad.
Quoted 5 lines
I think this patch also breaks the userspace build of
lib/raid6/. Which is going away in Christoph's series anyway, but maybe it would make sense to drop this patch (and patch 8 which depends on this, I think) from this series for now? That would make it a bit easier to take the rest through crc-next.
Ard, are you okay with me applying just patches 1-6 to crc-next?
- Eric
Ard Biesheuvel
reviewerRe: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
Quoted 16 lines · Eric Biggers
Quoted 15 lines · Eric Biggers
Quoted 13 lines · Christoph Hellwig
Quoted 6 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
arm_neon.his a compiler header which needs some scaffolding to work correctly in the linux context, and so it is better not to include it directly. Both ARM and arm64 now provideasm/neon-intrinsics.hwhich takes care of this.This could potentially clash with the raid6 library rework I'm doing for 7.2. Although git has become pretty good about renamed files, so maybe it won't be so bad.
Quoted 7 lines
Quoted 5 lines
I think this patch also breaks the userspace build of
lib/raid6/. Which is going away in Christoph's series anyway, but maybe it would make sense to drop this patch (and patch 8 which depends on this, I think) from this series for now? That would make it a bit easier to take the rest through crc-next.Ard, are you okay with me applying just patches 1-6 to crc-next?
Yes that's fine - thanks.
PATCH 8/8 ARM: Remove hacked-up asm/types.h header
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
ARM has a special version of asm/types.h which contains overrides for certain #define's related to the C types used to back C99 types such as uint32_t and uintptr_t.
This is only needed when pulling in system headers such as stdint.h during the build, and this only happens when using NEON intrinsics, for which there is now a dedicated header file.
So drop this header entirely, and revert to the asm-generic one.
--- arch/arm/include/uapi/asm/types.h | 41 -------------------- 1 file changed, 41 deletions(-)
arch/arm/include/uapi/asm/types.h deleted +0 -41
@@ -1,41 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ -#ifndef _UAPI_ASM_TYPES_H -#define _UAPI_ASM_TYPES_H - -#include <asm-generic/int-ll64.h> - -/* -* The C99 types uintXX_t that are usually defined in 'stdint.h' are not as -* unambiguous on ARM as you would expect. For the types below, there is a -* difference on ARM between GCC built for bare metal ARM, GCC built for glibc -* and the kernel itself, which results in build errors if you try to build with -* -ffreestanding and include 'stdint.h' (such as when you include 'arm_neon.h' -* in order to use NEON intrinsics) -* -* As the typedefs for these types in 'stdint.h' are based on builtin defines -* supplied by GCC, we can tweak these to align with the kernel's idea of those -* types, so 'linux/types.h' and 'stdint.h' can be safely included from the same -* source file (provided that -ffreestanding is used). -* -* int32_t uint32_t uintptr_t -* bare metal GCC long unsigned long unsigned int -* glibc GCC int unsigned int unsigned int -* kernel int unsigned int unsigned long -*/ - -#ifdef __INT32_TYPE__ -#undef __INT32_TYPE__ -#define __INT32_TYPE__ int -#endif - -#ifdef __UINT32_TYPE__ -#undef __UINT32_TYPE__ -#define __UINT32_TYPE__ unsigned int -#endif - -#ifdef __UINTPTR_TYPE__ -#undef __UINTPTR_TYPE__ -#define __UINTPTR_TYPE__ unsigned long -#endif - -#endif /* _UAPI_ASM_TYPES_H */
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
Eric Biggers
maintainerRe: [PATCH 8/8] ARM: Remove hacked-up asm/types.h header
Quoted 10 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
ARM has a special version of
asm/types.hwhich contains overrides for certain #define's related to the C types used to back C99 types such asuint32_tanduintptr_t.This is only needed when pulling in system headers such as
stdint.hduring the build, and this only happens when using NEON intrinsics, for which there is now a dedicated header file.
Quoted 7 lines
So drop this header entirely, and revert to the asm-generic one.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm/include/uapi/asm/types.h | 41 -------------------- 1 file changed, 41 deletions(-)
This is actually a UAPI header. I guess it got put there accidentally and isn't actually needed there?
- Eric
Arnd Bergmann
Reviewed-byRe: [PATCH 8/8] ARM: Remove hacked-up asm/types.h header
Quoted 7 lines · Eric Biggers
Quoted 6 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
ARM has a special version of
asm/types.hwhich contains overrides for certain #define's related to the C types used to back C99 types such asuint32_tanduintptr_t.
Quoted 7 lines
Quoted 7 lines
This is only needed when pulling in system headers such as
stdint.hduring the build, and this only happens when using NEON intrinsics, for which there is now a dedicated header file.So drop this header entirely, and revert to the asm-generic one.
Signed-off-by Ard Biesheuvel <[email protected]>
Quoted 2 lines
This is actually a UAPI header. I guess it got put there accidentally and isn't actually needed there?
Yes, commit ed79c9d34f4f ("ARM: put types.h in uapi") has some explanations.
I can't think of any case where this would actually be used from userland, and lots of ways it could cause trouble in theory, even if it never has in practice.
Arnd
Eric Biggers
maintainerQuoted 32 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
This is a follow-up to both [0] and [1], both of which included patch #1 of this series, which introduces the
asm/neon-intrinsics.hheader on 32-bit ARM. The remaining changes rely on this.The purpose of this series is to streamline / clean up the use of NEON intrinsics on 32-bit ARM, by sharing more code, clean up Make rules and finally, getting rid of the hacked up
types.hheader, which does some nasty things that are only needed when building NEON intrinsics code.Patches #2 and #3 replace the ARM autovectorized XOR implementation with the NEON intrinsics version used by arm64.
Patches #4 and #5 enable the arm64 NEON intrinsics implementation of crc64 on 32-bit ARM.
Patches #6 and #7 drop the direct includes of <
arm_neon.h> and perform some additional cleanup to reduce the delta between ARM and arm64 code and Make rules.It would probably be easiest to take all these changes through a single tree, and the CRC tree seems like a suitable candidate, if Eric agrees.
Cc Christoph Hellwig <[email protected]>Cc Russell King <[email protected]>Cc Arnd Bergmann <[email protected]>Cc Eric Biggers <[email protected]>[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/
Quoted 9 lines
Ard Biesheuvel (8):
ARM: Add aneon-intrinsics.hheader like on arm64 xor/arm: Replace vectorized implementation with arm64's intrinsics xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM lib/crc: Turn NEON intrinsics crc64 implementation into common code lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64
crypto: aegis128 - Useneon-intrinsics.hon ARM too lib/raid6: Includeasm/neon-intrinsics.hrather thanarm_neon.h
ARM: Remove hacked-upasm/types.hheader
Applied patches 1-6 to
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next
- Eric