Skip to contentSkip to conversation

ARM crc64 and XOR using NEON intrinsics

PATCH In review Ard Biesheuvel 6 participants 26 messages Apr 22 – May 28

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover ARM crc64 and XOR using NEON intrinsics

From: Ard Biesheuvel <[email protected]>

This is a follow-up to both [0] and [1], both of which included patch #1 of this series, which introduces the asm/neon-intrinsics.h header on 32-bit ARM. The remaining changes rely on this.

The purpose of this series is to streamline / clean up the use of NEON intrinsics on 32-bit ARM, by sharing more code, clean up Make rules and finally, getting rid of the hacked up types.h header, which does some nasty things that are only needed when building NEON intrinsics code.

Patches #2 and #3 replace the ARM autovectorized XOR implementation with the NEON intrinsics version used by arm64.

Patches #4 and #5 enable the arm64 NEON intrinsics implementation of crc64 on 32-bit ARM.

Patches #6 and #7 drop the direct includes of <arm_neon.h> and perform some additional cleanup to reduce the delta between ARM and arm64 code and Make rules.

It would probably be easiest to take all these changes through a single tree, and the CRC tree seems like a suitable candidate, if Eric agrees.

Cc Christoph Hellwig <[email protected]>
Cc Russell King <[email protected]>
Cc Arnd Bergmann <[email protected]>
Cc Eric Biggers <[email protected]>

[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/

Ard Biesheuvel (8):
ARM: Add a neon-intrinsics.h header like on arm64 xor/arm: Replace vectorized implementation with arm64's intrinsics xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM lib/crc: Turn NEON intrinsics crc64 implementation into common code lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64
crypto: aegis128 - Use neon-intrinsics.h on ARM too lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
ARM: Remove hacked-up asm/types.h header

 Documentation/arch/arm/kernel_mode_neon.rst        |   4 +-
 arch/arm/include/asm/neon-intrinsics.h             |  60 ++++++++
 arch/arm/include/uapi/asm/types.h                  |  41 ------
 crypto/Makefile                                    |  10 +-
 crypto/aegis128-neon-inner.c                       |   4 +-
 lib/crc/Kconfig                                    |   1 +
 lib/crc/Makefile                                   |   9 +-
 lib/crc/arm/crc64-neon.h                           |  34 +++++
 lib/crc/arm/crc64.h                                |  36 +++++
 lib/crc/arm64/crc64-neon.h                         |  21 +++
 lib/crc/arm64/crc64.h                              |   4 +-
 lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} |  26 +---
 lib/raid/xor/Makefile                              |  13 +-
 lib/raid/xor/arm/xor-neon.c                        |  26 ----
 lib/raid/xor/arm/xor-neon.h                        |   7 +
 lib/raid/xor/arm/xor_arch.h                        |   7 +-
 lib/raid/xor/arm64/xor-eor3.c                      | 146 ++++++++++++++++++++
 lib/raid/xor/xor-8regs.c                           |   2 -
 lib/raid/xor/{arm64 => }/xor-neon.c                | 143 +------------------
 lib/raid6/neon.uc                                  |   2 +-
 lib/raid6/recov_neon_inner.c                       |   2 +-
 21 files changed, 340 insertions(+), 258 deletions(-)
 create mode 100644 arch/arm/include/asm/neon-intrinsics.h
 delete mode 100644 arch/arm/include/uapi/asm/types.h
 create mode 100644 lib/crc/arm/crc64-neon.h
 create mode 100644 lib/crc/arm/crc64.h
 create mode 100644 lib/crc/arm64/crc64-neon.h
 rename lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} (62%)
 delete mode 100644 lib/raid/xor/arm/xor-neon.c
 create mode 100644 lib/raid/xor/arm/xor-neon.h
 create mode 100644 lib/raid/xor/arm64/xor-eor3.c
 rename lib/raid/xor/{arm64 => }/xor-neon.c (56%)

base-commit: 6596a02b207886e9e00bb0161c7fd59fea53c081

Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
#1 lore raw

PATCH 1/8 ARM: Add a neon-intrinsics.h header like on arm64

From: Ard Biesheuvel <[email protected]>

Add a header asm/neon-intrinsics.h similar to the one that arm64 has. This makes it possible for NEON intrinsics code to be shared seamlessly between ARM and arm64.

---
 Documentation/arch/arm/kernel_mode_neon.rst |  4 +-
 arch/arm/include/asm/neon-intrinsics.h      | 60 ++++++++++++++++++++
 2 files changed, 63 insertions(+), 1 deletion(-)
Documentation/arch/arm/kernel_mode_neon.rst +3 -1
index 9bfb71a2a9b9..1efb6d35b7bd 100644
@@ -121,4 +121,6 @@ observe the following in addition to the rules above:
 * Compile the unit containing the NEON intrinsics with '-ffreestanding' so GCC
   uses its builtin version of <stdint.h> (this is a C99 header which the kernel
   does not supply);
-* Include <arm_neon.h> last, or at least after <linux/types.h>
+* Do not include <arm_neon.h> directly: instead, include <asm/neon-intrinsics.h>,
+  which tweaks some macro definitions so that system headers can be included
+  safely.
arch/arm/include/asm/neon-intrinsics.h new file +60 -0

60 lines not on this page · raw message

Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
#2 lore raw

PATCH 2/8 xor/arm: Replace vectorized implementation with arm64's intrinsics

In reply to Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

Drop the XOR implementation generated by the vectorizer: this has always been a bit of a hack, and now that arm64 has an intrinsics version that works on ARM too, let's use that instead.

So copy the part of the arm64 code that can be shared (so not the EOR3 version). The arm64 code will be updated in a subsequent patch to share this implementation.

Performance (QEMU mach-virt VM running on Synquacer [Cortex-A53 @ 1 GHz]

Before:

[    3.519687] xor: measuring software checksum speed
[    3.521725]    neon            :  1660 MB/sec
[    3.524733]    32regs          :  1105 MB/sec
[    3.527751]    8regs           :  1098 MB/sec
[    3.529911]    arm4regs        :  1540 MB/sec

After:

[    3.517654] xor: measuring software checksum speed
[    3.519454]    neon            :  1896 MB/sec
[    3.522499]    32regs          :  1090 MB/sec
[    3.525560]    8regs           :  1083 MB/sec
[    3.527700]    arm4regs        :  1556 MB/sec
---
 lib/raid/xor/Makefile       |   6 +-
 lib/raid/xor/arm/xor-neon.c |  26 ---
 lib/raid/xor/arm/xor-neon.h |   7 +
 lib/raid/xor/arm/xor_arch.h |   7 +-
 lib/raid/xor/xor-8regs.c    |   2 -
 lib/raid/xor/xor-neon.c     | 175 ++++++++++++++++++++
 6 files changed, 186 insertions(+), 37 deletions(-)
lib/raid/xor/Makefile +3 -3
index 4d633dfd5b90..d78400f2427a 100644
@@ -17,7 +17,7 @@ endif
 xor-$(CONFIG_ALPHA)		+= alpha/xor.o
 xor-$(CONFIG_ARM)		+= arm/xor.o
 ifeq ($(CONFIG_ARM),y)
-xor-$(CONFIG_KERNEL_MODE_NEON)	+= arm/xor-neon.o arm/xor-neon-glue.o
+xor-$(CONFIG_KERNEL_MODE_NEON)	+= xor-neon.o arm/xor-neon-glue.o
 endif
 xor-$(CONFIG_ARM64)		+= arm64/xor-neon.o arm64/xor-neon-glue.o
 xor-$(CONFIG_CPU_HAS_LSX)	+= loongarch/xor_simd.o
@@ -31,8 +31,8 @@ xor-$(CONFIG_X86_32)		+= x86/xor-avx.o x86/xor-sse.o x86/xor-mmx.o
 xor-$(CONFIG_X86_64)		+= x86/xor-avx.o x86/xor-sse.o
 obj-y				+= tests/
 
-CFLAGS_arm/xor-neon.o		+= $(CC_FLAGS_FPU)
-CFLAGS_REMOVE_arm/xor-neon.o	+= $(CC_FLAGS_NO_FPU)
+CFLAGS_xor-neon.o		+= $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH)
+CFLAGS_REMOVE_xor-neon.o	+= $(CC_FLAGS_NO_FPU)
 
 CFLAGS_arm64/xor-neon.o		+= $(CC_FLAGS_FPU)
 CFLAGS_REMOVE_arm64/xor-neon.o	+= $(CC_FLAGS_NO_FPU)
lib/raid/xor/arm/xor-neon.c deleted +0 -26
deleted file mode 100644
index 23147e3a7904..000000000000
@@ -1,26 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Copyright (C) 2013 Linaro Ltd <[email protected]>
- */
-
-#include "xor_impl.h"
-#include "xor_arch.h"
-
-#ifndef __ARM_NEON__
-#error You should compile this file with '-march=armv7-a -mfloat-abi=softfp -mfpu=neon'
-#endif
-
-/*
- * Pull in the reference implementations while instructing GCC (through
- * -ftree-vectorize) to attempt to exploit implicit parallelism and emit
- * NEON instructions. Clang does this by default at O2 so no pragma is
- * needed.
- */
-#ifdef CONFIG_CC_IS_GCC
-#pragma GCC optimize "tree-vectorize"
-#endif
-
-#define NO_TEMPLATE
-#include "../xor-8regs.c"
-
-__DO_XOR_BLOCKS(neon_inner, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);
lib/raid/xor/arm/xor-neon.h new file +7 -0
new file mode 100644
index 000000000000..406e0356f05b
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+extern struct xor_block_template xor_block_arm4regs;
+extern struct xor_block_template xor_block_neon;
+
+void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt,
+		unsigned int bytes);
lib/raid/xor/arm/xor_arch.h +1 -6
index 775ff835df65..f1ddb64fe62a 100644
@@ -3,12 +3,7 @@
  *  Copyright (C) 2001 Russell King
  */
 #include <asm/neon.h>
-
-extern struct xor_block_template xor_block_arm4regs;
-extern struct xor_block_template xor_block_neon;
-
-void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt,
-		unsigned int bytes);
+#include "xor-neon.h"
 
 static __always_inline void __init arch_xor_init(void)
 {
lib/raid/xor/xor-8regs.c +0 -2
index 1edaed8acffe..46b3c8bdc27f 100644
@@ -93,11 +93,9 @@ xor_8regs_5(unsigned long bytes, unsigned long * __restrict p1,
 	} while (--lines > 0);
 }
 
-#ifndef NO_TEMPLATE
 DO_XOR_BLOCKS(8regs, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);
 
 struct xor_block_template xor_block_8regs = {
 	.name		= "8regs",
 	.xor_gen	= xor_gen_8regs,
 };
-#endif /* NO_TEMPLATE */
lib/raid/xor/xor-neon.c new file +175 -0

175 lines not on this page · raw message

Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
#5 lore raw

PATCH 3/8 xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM

In reply to Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

Tweak the arm64 code so that the pure NEON intrinsics implementation of XOR is shared between arm64 and ARM. While at it, rename the arm64 specific piece xor-eor3.c to reflect that only the version based on the EOR3 instruction is kept there.

---
 lib/raid/xor/Makefile         |   7 +-
 lib/raid/xor/arm64/xor-eor3.c | 146 +++++++++
 lib/raid/xor/arm64/xor-neon.c | 312 --------------------
 lib/raid/xor/xor-neon.c       |   4 +
 4 files changed, 154 insertions(+), 315 deletions(-)
lib/raid/xor/Makefile +4 -3
index d78400f2427a..e8ecec3c09f9 100644
@@ -19,7 +19,8 @@ xor-$(CONFIG_ARM)		+= arm/xor.o
 ifeq ($(CONFIG_ARM),y)
 xor-$(CONFIG_KERNEL_MODE_NEON)	+= xor-neon.o arm/xor-neon-glue.o
 endif
-xor-$(CONFIG_ARM64)		+= arm64/xor-neon.o arm64/xor-neon-glue.o
+xor-$(CONFIG_ARM64)		+= xor-neon.o arm64/xor-eor3.o \
+				   arm64/xor-neon-glue.o
 xor-$(CONFIG_CPU_HAS_LSX)	+= loongarch/xor_simd.o
 xor-$(CONFIG_CPU_HAS_LSX)	+= loongarch/xor_simd_glue.o
 xor-$(CONFIG_ALTIVEC)		+= powerpc/xor_vmx.o powerpc/xor_vmx_glue.o
@@ -34,8 +35,8 @@ obj-y				+= tests/
 CFLAGS_xor-neon.o		+= $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH)
 CFLAGS_REMOVE_xor-neon.o	+= $(CC_FLAGS_NO_FPU)
 
-CFLAGS_arm64/xor-neon.o		+= $(CC_FLAGS_FPU)
-CFLAGS_REMOVE_arm64/xor-neon.o	+= $(CC_FLAGS_NO_FPU)
+CFLAGS_arm64/xor-eor3.o		+= $(CC_FLAGS_FPU)
+CFLAGS_REMOVE_arm64/xor-eor3.o	+= $(CC_FLAGS_NO_FPU)
 
 CFLAGS_powerpc/xor_vmx.o	+= -mhard-float -maltivec \
 				   $(call cc-option,-mabi=altivec) \
lib/raid/xor/arm64/xor-eor3.c new file +146 -0

146 lines not on this page · raw message

lib/raid/xor/arm64/xor-neon.c deleted +0 -312

312 lines not on this page · raw message

lib/raid/xor/xor-neon.c +4 -0
index a3e2b4af8d36..c7c3cf634e23 100644
@@ -173,3 +173,7 @@ static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
 
 __DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4,
 		__xor_neon_5);
+
+#ifdef CONFIG_ARM64
+extern typeof(__xor_neon_2) __xor_eor3_2 __alias(__xor_neon_2);
+#endif
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
#9 lore raw

PATCH 4/8 lib/crc: Turn NEON intrinsics crc64 implementation into common code

In reply to Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

Move and rename the CRC64 NEON intrinsics implementation source file and rename the function name to reflect that it is NEON code that can be shared. This will be wired up for 32-bit ARM in a subsequent patch.

---
 lib/crc/Makefile                                   |  6 ++---
 lib/crc/arm64/crc64-neon.h                         | 21 ++++++++++++++++
 lib/crc/arm64/crc64.h                              |  4 +--
 lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} | 26 +++-----------------
 4 files changed, 30 insertions(+), 27 deletions(-)
lib/crc/Makefile +3 -3
index ff213590e4e3..193257ae466f 100644
@@ -39,9 +39,9 @@ crc64-y := crc64-main.o
 ifeq ($(CONFIG_CRC64_ARCH),y)
 CFLAGS_crc64-main.o += -I$(src)/$(SRCARCH)
 
-CFLAGS_REMOVE_arm64/crc64-neon-inner.o += $(CC_FLAGS_NO_FPU)
-CFLAGS_arm64/crc64-neon-inner.o += $(CC_FLAGS_FPU) -march=armv8-a+crypto
-crc64-$(CONFIG_ARM64) += arm64/crc64-neon-inner.o
+CFLAGS_REMOVE_crc64-neon.o += $(CC_FLAGS_NO_FPU)
+CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) -march=armv8-a+crypto
+crc64-$(CONFIG_ARM64) += crc64-neon.o
 
 crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o
 crc64-$(CONFIG_X86) += x86/crc64-pclmul.o
lib/crc/arm64/crc64-neon.h new file +21 -0
new file mode 100644
index 000000000000..fcd5b1e6f812
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+static inline uint64x2_t pmull64(uint64x2_t a, uint64x2_t b)
+{
+	return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 0),
+						vgetq_lane_u64(b, 0)));
+}
+
+static inline uint64x2_t pmull64_high(uint64x2_t a, uint64x2_t b)
+{
+	poly64x2_t l = vreinterpretq_p64_u64(a);
+	poly64x2_t m = vreinterpretq_p64_u64(b);
+
+	return vreinterpretq_u64_p128(vmull_high_p64(l, m));
+}
+
+static inline uint64x2_t pmull64_hi_lo(uint64x2_t a, uint64x2_t b)
+{
+	return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 1),
+						vgetq_lane_u64(b, 0)));
+}
lib/crc/arm64/crc64.h +2 -2
index 60151ec3035a..c7a69e1f3d8f 100644
@@ -8,7 +8,7 @@
 #include <linux/minmax.h>
 #include <linux/sizes.h>
 
-u64 crc64_nvme_arm64_c(u64 crc, const u8 *p, size_t len);
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len);
 
 #define crc64_be_arch crc64_be_generic
 
@@ -19,7 +19,7 @@ static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)
 		size_t chunk = len & ~15;
 
 		scoped_ksimd()
-			crc = crc64_nvme_arm64_c(crc, p, chunk);
+			crc = crc64_nvme_neon(crc, p, chunk);
 
 		p += chunk;
 		len &= 15;
lib/crc/crc64-neon.c renamed from lib/crc/arm64/crc64-neon-inner.c +4 -22
similarity index 62%
rename from lib/crc/arm64/crc64-neon-inner.c
rename to lib/crc/crc64-neon.c
index 28527e544ff6..4753fb94a4be 100644
@@ -6,7 +6,9 @@
 #include <linux/types.h>
 #include <asm/neon-intrinsics.h>
 
-u64 crc64_nvme_arm64_c(u64 crc, const u8 *p, size_t len);
+#include "crc64-neon.h"
+
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len);
 
 /* x^191 mod G, x^127 mod G */
 static const u64 fold_consts_val[2] = { 0xeadc41fd2ba3d420ULL,
@@ -15,27 +17,7 @@ static const u64 fold_consts_val[2] = { 0xeadc41fd2ba3d420ULL,
 static const u64 bconsts_val[2] = { 0x27ecfa329aef9f77ULL,
 				    0x34d926535897936aULL };
 
-static inline uint64x2_t pmull64(uint64x2_t a, uint64x2_t b)
-{
-	return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 0),
-						vgetq_lane_u64(b, 0)));
-}
-
-static inline uint64x2_t pmull64_high(uint64x2_t a, uint64x2_t b)
-{
-	poly64x2_t l = vreinterpretq_p64_u64(a);
-	poly64x2_t m = vreinterpretq_p64_u64(b);
-
-	return vreinterpretq_u64_p128(vmull_high_p64(l, m));
-}
-
-static inline uint64x2_t pmull64_hi_lo(uint64x2_t a, uint64x2_t b)
-{
-	return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 1),
-						vgetq_lane_u64(b, 0)));
-}
-
-u64 crc64_nvme_arm64_c(u64 crc, const u8 *p, size_t len)
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len)
 {
 	uint64x2_t fold_consts = vld1q_u64(fold_consts_val);
 	uint64x2_t v0 = { crc, 0 };
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
#11 lore raw

PATCH 5/8 lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64

In reply to Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

Tweak the NEON intrinsics crc64 code written for arm64 so it can be built for 32-bit ARM as well. The only workaround needed is to provide alternatives for vmull_p64() and vmull_high_p64() on Clang, which only defines those when building for the AArch64 or arm64ec ISA. Use the same helpers for GCC too, to avoid doubling the size of the test/validation matrix.

KUnit benchmark results (Cortex-A53 @ 1 Ghz)

Before:

# crc64_nvme_benchmark: len=1: 35 MB/s
# crc64_nvme_benchmark: len=16: 78 MB/s
# crc64_nvme_benchmark: len=64: 87 MB/s
# crc64_nvme_benchmark: len=127: 88 MB/s
# crc64_nvme_benchmark: len=128: 88 MB/s
# crc64_nvme_benchmark: len=200: 89 MB/s
# crc64_nvme_benchmark: len=256: 89 MB/s
# crc64_nvme_benchmark: len=511: 89 MB/s
# crc64_nvme_benchmark: len=512: 89 MB/s
# crc64_nvme_benchmark: len=1024: 90 MB/s
# crc64_nvme_benchmark: len=3173: 90 MB/s
# crc64_nvme_benchmark: len=4096: 90 MB/s
# crc64_nvme_benchmark: len=16384: 90 MB/s

After:

# crc64_nvme_benchmark: len=1: 32 MB/s
# crc64_nvme_benchmark: len=16: 76 MB/s
# crc64_nvme_benchmark: len=64: 71 MB/s
# crc64_nvme_benchmark: len=127: 88 MB/s
# crc64_nvme_benchmark: len=128: 618 MB/s
# crc64_nvme_benchmark: len=200: 542 MB/s
# crc64_nvme_benchmark: len=256: 920 MB/s
# crc64_nvme_benchmark: len=511: 836 MB/s
# crc64_nvme_benchmark: len=512: 1261 MB/s
# crc64_nvme_benchmark: len=1024: 1531 MB/s
# crc64_nvme_benchmark: len=3173: 1731 MB/s
# crc64_nvme_benchmark: len=4096: 1851 MB/s
# crc64_nvme_benchmark: len=16384: 1858 MB/s

Don't bother with big-endian, as it doesn't work correctly on Clang, and is barely used these days.

Note that ARM disables preemption and softirq processing when using kernel mode SIMD, so take care not to hog the CPU for too long.

---
 lib/crc/Kconfig          |  1 +
 lib/crc/Makefile         |  5 ++-
 lib/crc/arm/crc64-neon.h | 34 ++++++++++++++++++
 lib/crc/arm/crc64.h      | 36 ++++++++++++++++++++
 4 files changed, 75 insertions(+), 1 deletion(-)
lib/crc/Kconfig +1 -0
index 31038c8d111a..86a0e4bfec77 100644
@@ -82,6 +82,7 @@ config CRC64
 config CRC64_ARCH
 	bool
 	depends on CRC64 && CRC_OPTIMIZATIONS
+	default y if ARM && KERNEL_MODE_NEON && !CPU_BIG_ENDIAN
 	default y if ARM64
 	default y if RISCV && RISCV_ISA_ZBC && 64BIT
 	default y if X86_64
lib/crc/Makefile +4 -1
index 193257ae466f..386e9c175263 100644
@@ -39,8 +39,11 @@ crc64-y := crc64-main.o
 ifeq ($(CONFIG_CRC64_ARCH),y)
 CFLAGS_crc64-main.o += -I$(src)/$(SRCARCH)
 
+crc64-cflags-$(CONFIG_ARM) += -march=armv8-a -mfpu=crypto-neon-fp-armv8
+crc64-cflags-$(CONFIG_ARM64) += -march=armv8-a+crypto
 CFLAGS_REMOVE_crc64-neon.o += $(CC_FLAGS_NO_FPU)
-CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) -march=armv8-a+crypto
+CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) $(crc64-cflags-y)
+crc64-$(CONFIG_ARM) += crc64-neon.o
 crc64-$(CONFIG_ARM64) += crc64-neon.o
 
 crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o
lib/crc/arm/crc64-neon.h new file +34 -0
new file mode 100644
index 000000000000..645f553220ff
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+static inline uint64x2_t pmull64(uint64x2_t a, uint64x2_t b)
+{
+	uint64_t l = vgetq_lane_u64(a, 0);
+	uint64_t m = vgetq_lane_u64(b, 0);
+	uint64x2_t result;
+
+	asm("vmull.p64	%q0, %P1, %P2" : "=w"(result) : "w"(l), "w"(m));
+
+	return result;
+}
+
+static inline uint64x2_t pmull64_high(uint64x2_t a, uint64x2_t b)
+{
+	uint64_t l = vgetq_lane_u64(a, 1);
+	uint64_t m = vgetq_lane_u64(b, 1);
+	uint64x2_t result;
+
+	asm("vmull.p64	%q0, %P1, %P2" : "=w"(result) : "w"(l), "w"(m));
+
+	return result;
+}
+
+static inline uint64x2_t pmull64_hi_lo(uint64x2_t a, uint64x2_t b)
+{
+	uint64_t l = vgetq_lane_u64(a, 1);
+	uint64_t m = vgetq_lane_u64(b, 0);
+	uint64x2_t result;
+
+	asm("vmull.p64	%q0, %P1, %P2" : "=w"(result) : "w"(l), "w"(m));
+
+	return result;
+}
lib/crc/arm/crc64.h new file +36 -0
new file mode 100644
index 000000000000..de274288af61
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * CRC64 using ARM PMULL instructions
+ */
+
+#include <asm/simd.h>
+
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull);
+
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len);
+
+#define crc64_be_arch crc64_be_generic
+
+static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)
+{
+	if (len >= 128 && static_branch_likely(&have_pmull) &&
+	    likely(may_use_simd())) {
+		do {
+			size_t chunk = min_t(size_t, len & ~15, SZ_4K);
+
+			scoped_ksimd()
+				crc = crc64_nvme_neon(crc, p, chunk);
+
+			p += chunk;
+			len -= chunk;
+		} while (len >= 128);
+	}
+	return crc64_nvme_generic(crc, p, len);
+}
+
+#define crc64_mod_init_arch crc64_mod_init_arch
+static void crc64_mod_init_arch(void)
+{
+	if (elf_hwcap2 & HWCAP2_PMULL)
+		static_branch_enable(&have_pmull);
+}
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
#13 lore raw

PATCH 6/8 crypto: aegis128 - Use neon-intrinsics.h on ARM too

In reply to Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

Use the asm/neon-intrinsics.h header on ARM as well as arm64, so that the calling code does not have to know the difference.

Clean up the Makefile a bit while at it.

---
 crypto/Makefile              | 10 ++++------
 crypto/aegis128-neon-inner.c |  4 +---
 2 files changed, 5 insertions(+), 9 deletions(-)
crypto/Makefile +4 -6
index 162242593c7c..69d1a18e8519 100644
@@ -103,13 +103,14 @@ obj-$(CONFIG_CRYPTO_CHACHA20POLY1305) += chacha20poly1305.o
 obj-$(CONFIG_CRYPTO_AEGIS128) += aegis128.o
 aegis128-y := aegis128-core.o
 
+CFLAGS_aegis128-neon-inner.o += $(CC_FLAGS_FPU)
+CFLAGS_REMOVE_aegis128-neon-inner.o += $(CC_FLAGS_NO_FPU)
 ifeq ($(ARCH),arm)
-CFLAGS_aegis128-neon-inner.o += -ffreestanding -march=armv8-a -mfloat-abi=softfp
-CFLAGS_aegis128-neon-inner.o += -mfpu=crypto-neon-fp-armv8
+CFLAGS_aegis128-neon-inner.o += -march=armv8-a -mfpu=crypto-neon-fp-armv8
 aegis128-$(CONFIG_CRYPTO_AEGIS128_SIMD) += aegis128-neon.o aegis128-neon-inner.o
 endif
 ifeq ($(ARCH),arm64)
-aegis128-cflags-y := -ffreestanding -mcpu=generic+crypto
+aegis128-cflags-y := -mcpu=generic+crypto
 aegis128-cflags-$(CONFIG_CC_IS_GCC) += -ffixed-q16 -ffixed-q17 -ffixed-q18 \
 				       -ffixed-q19 -ffixed-q20 -ffixed-q21 \
 				       -ffixed-q22 -ffixed-q23 -ffixed-q24 \
@@ -117,11 +118,8 @@ aegis128-cflags-$(CONFIG_CC_IS_GCC) += -ffixed-q16 -ffixed-q17 -ffixed-q18 \
 				       -ffixed-q28 -ffixed-q29 -ffixed-q30 \
 				       -ffixed-q31
 CFLAGS_aegis128-neon-inner.o += $(aegis128-cflags-y)
-CFLAGS_REMOVE_aegis128-neon-inner.o += -mgeneral-regs-only
 aegis128-$(CONFIG_CRYPTO_AEGIS128_SIMD) += aegis128-neon.o aegis128-neon-inner.o
 endif
-# Enable <arm_neon.h>
-CFLAGS_aegis128-neon-inner.o += -isystem $(shell $(CC) -print-file-name=include)
 
 obj-$(CONFIG_CRYPTO_PCRYPT) += pcrypt.o
 obj-$(CONFIG_CRYPTO_CRYPTD) += cryptd.o
crypto/aegis128-neon-inner.c +1 -3
index b6a52a386b22..56b534eeb680 100644
@@ -3,13 +3,11 @@
  * Copyright (C) 2019 Linaro, Ltd. <[email protected]>
  */
 
-#ifdef CONFIG_ARM64
 #include <asm/neon-intrinsics.h>
 
+#ifdef CONFIG_ARM64
 #define AES_ROUND	"aese %0.16b, %1.16b \n\t aesmc %0.16b, %0.16b"
 #else
-#include <arm_neon.h>
-
 #define AES_ROUND	"aese.8 %q0, %q1 \n\t aesmc.8 %q0, %q0"
 #endif
 
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
#15 lore raw

PATCH 7/8 lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h

In reply to Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

arm_neon.h is a compiler header which needs some scaffolding to work correctly in the linux context, and so it is better not to include it directly. Both ARM and arm64 now provide asm/neon-intrinsics.h which takes care of this.

---
 lib/raid6/neon.uc            | 2 +-
 lib/raid6/recov_neon_inner.c | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
lib/raid6/neon.uc +1 -1
index 355270af0cd6..3dc20511103a 100644
@@ -24,7 +24,7 @@
  * This file is postprocessed using unroll.awk
  */
 
-#include <arm_neon.h>
+#include <asm/neon-intrinsics.h>
 #include "neon.h"
 
 typedef uint8x16_t unative_t;
lib/raid6/recov_neon_inner.c +1 -1
index f9e7e8f5a151..06b2967fb8b6 100644
@@ -4,7 +4,7 @@
  * Copyright (C) 2017 Linaro Ltd. <[email protected]>
  */
 
-#include <arm_neon.h>
+#include <asm/neon-intrinsics.h>
 #include "neon.h"
 
 #ifdef CONFIG_ARM
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
#22 lore raw

PATCH 8/8 ARM: Remove hacked-up asm/types.h header

In reply to Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

ARM has a special version of asm/types.h which contains overrides for certain #define's related to the C types used to back C99 types such as uint32_t and uintptr_t.

This is only needed when pulling in system headers such as stdint.h during the build, and this only happens when using NEON intrinsics, for which there is now a dedicated header file.

So drop this header entirely, and revert to the asm-generic one.

---
 arch/arm/include/uapi/asm/types.h | 41 --------------------
 1 file changed, 41 deletions(-)
arch/arm/include/uapi/asm/types.h deleted +0 -41
deleted file mode 100644
index 1a667bc26510..000000000000
@@ -1,41 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
-#ifndef _UAPI_ASM_TYPES_H
-#define _UAPI_ASM_TYPES_H
-
-#include <asm-generic/int-ll64.h>
-
-/*
- * The C99 types uintXX_t that are usually defined in 'stdint.h' are not as
- * unambiguous on ARM as you would expect. For the types below, there is a
- * difference on ARM between GCC built for bare metal ARM, GCC built for glibc
- * and the kernel itself, which results in build errors if you try to build with
- * -ffreestanding and include 'stdint.h' (such as when you include 'arm_neon.h'
- * in order to use NEON intrinsics)
- *
- * As the typedefs for these types in 'stdint.h' are based on builtin defines
- * supplied by GCC, we can tweak these to align with the kernel's idea of those
- * types, so 'linux/types.h' and 'stdint.h' can be safely included from the same
- * source file (provided that -ffreestanding is used).
- *
- *                    int32_t         uint32_t               uintptr_t
- * bare metal GCC     long            unsigned long          unsigned int
- * glibc GCC          int             unsigned int           unsigned int
- * kernel             int             unsigned int           unsigned long
- */
-
-#ifdef __INT32_TYPE__
-#undef __INT32_TYPE__
-#define __INT32_TYPE__		int
-#endif
-
-#ifdef __UINT32_TYPE__
-#undef __UINT32_TYPE__
-#define __UINT32_TYPE__	unsigned int
-#endif
-
-#ifdef __UINTPTR_TYPE__
-#undef __UINTPTR_TYPE__
-#define __UINTPTR_TYPE__	unsigned long
-#endif
-
-#endif /* _UAPI_ASM_TYPES_H */
Signature · 1 line
2.54.0.rc1.555.g9c883467ad-goog
Reviewed-by on v1 2/8
#3 lore raw

Re: [PATCH 2/8] xor/arm: Replace vectorized implementation with arm64's intrinsics

In reply to Ard Biesheuvel

Hi ard.

I like this patch.

So, I'd be crazy not to say what I love here.

+ /* p1 ^= p2 */

+		v0 = veorq_u64(vld1q_u64(dp1 +  0), vld1q_u64(dp2 +  0));
+		v1 = veorq_u64(vld1q_u64(dp1 +  2), vld1q_u64(dp2 +  2));
+		v2 = veorq_u64(vld1q_u64(dp1 +  4), vld1q_u64(dp2 +  4));
+		v3 = veorq_u64(vld1q_u64(dp1 +  6), vld1q_u64(dp2 +  6));
+
+		/* p1 ^= p3 */
+		v0 = veorq_u64(v0, vld1q_u64(dp3 +  0));
+		v1 = veorq_u64(v1, vld1q_u64(dp3 +  2));
+		v2 = veorq_u64(v2, vld1q_u64(dp3 +  4));
+		v3 = veorq_u64(v3, vld1q_u64(dp3 +  6));
+
+		/* store */
+		vst1q_u64(dp1 +  0, v0);
+		vst1q_u64(dp1 +  2, v1);
+		vst1q_u64(dp1 +  4, v2);
+		vst1q_u64(dp1 +  6, v3);

+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ } while (--lines > 0);
+}

I really like how clean this is, I'm personally nodding my head here

Taking the "bad" guesswork of the compiler here is also amazing, it also guarantees we won't get stupid regressions in the future.

Also, that performance boost is even better ;)

I'm not the biggest expert of this subdirectory, but I understand it well.

So well,

Thanks! (I will review your lib patches) :)

Reviewed-by on v1 3/8
#6 lore raw

Re: [PATCH 3/8] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM

In reply to Ard Biesheuvel

Hi ard.

lib/raid/xor/xor-neon.c static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1, · line 179

Quoted 3 lines
lib/raid/xor/xor-neon.c +3 -0
+#ifdef CONFIG_ARM64
+extern typeof(__xor_neon_2) >__xor_eor3_2 >__alias(__xor_neon_2);
+#endif

Creative. A reduction of about 150 lines of duplicate code while maintaining
the __alias for the 2 input case is great.

Thanks!

Reviewed-by on v1 4/8
#10 lore raw

Re: [PATCH 4/8] lib/crc: Turn NEON intrinsics crc64 implementation into common code

In reply to Ard Biesheuvel

Hi ard.

lib/crc/arm64/crc64-neon.h new file +4 -0
new file mode 100644
index 000000000000..fcd5b1e6f812
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+static inline uint64x2_t pmull64(uint64x2_t a, uint64x2_t b)
+{

return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 0), vgetq_lane_u64(b, 0)));
+}
+static inline uint64x2_t pmull64_high(uint64x2_t a, uint64x2_t b) +{
poly64x2_t l = vreinterpretq_p64_u64(a);
poly64x2_t m = vreinterpretq_p64_u64(b);
return vreinterpretq_u64_p128(vmull_high_p64(l, m));
+}

Makes sense.

Moving these polynomial multiplication wrappers into their own header is good.

Thanks!

Reviewed-by on v1 5/8
#12 lore raw

Re: [PATCH 5/8] lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64

In reply to Ard Biesheuvel

Hi Ard,

Wow, 20x improvement is nuts.

I like how you handle this change *safely*

Like.

+static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len) +{
+ if (len >= 128 && static_branch_likely(&have_pmull) &&
+ likely(may_use_simd())) {
+ do {
+ size_t chunk = min_t(size_t, len & ~15, SZ_4K);
+
+ scoped_ksimd()
+ crc = crc64_nvme_neon(crc, p, chunk);
+
+ p += chunk;
+ len -= chunk;
+ } while (len >= 128);
+ }

chunking the SIMD work at SZ_4K to avoid hogging the CPU and allowing softirqs/preemption to process is a great detail.

It’s easy to just wing it and throw
the entire buffer at the FPU, but respecting the kernel's latency requirements is better!

Thanks!

Reviewed-by on v1 6/8
#14 lore raw

Re: [PATCH 6/8] crypto: aegis128 - Use neon-intrinsics.h on ARM too

In reply to Ard Biesheuvel

Hi ard, this is a good cleanup!

Being able to drop <arm_neon.h> and just using <asm/neon-intrinsics.h> across both architectures makes the C code much cleaner.

-# Enable <arm_neon.h>
-CFLAGS_aegis128-neon-inner.o += -isystem $(shell $(CC) -print-file-name=include)

Getting rid of the isystem is good. iirc that was a hack anyway, feel free to correct me on that

Reviewed-by on v1 7/8
#16 lore raw

Re: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h

In reply to Ard Biesheuvel

Hi ard.

Makes sense here

-#include <arm_neon.h>
+#include <asm/neon-intrinsics.h>

This series is a good (and deserved series)

That's me done! I've reviewed your lib patches for you, have a great day!

#7 lore raw

Re: [PATCH 3/8] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM

In reply to Ard Biesheuvel

lib/raid/xor/arm64/xor-eor3.c line 10

Quoted 2 lines
lib/raid/xor/arm64/xor-eor3.c new file +2 -0
+extern void __xor_eor3_2(unsigned long bytes, unsigned long * __restrict p1,
+		const unsigned long * __restrict p2);

Does the alias magic prevent this from being in a header? If so a comment would be nice, otherwise moving it to a header would be even better.

#17 lore raw

Re: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h

In reply to Ard Biesheuvel
Quoted 6 lines · Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

arm_neon.h is a compiler header which needs some scaffolding to work correctly in the linux context, and so it is better not to include it directly. Both ARM and arm64 now provide asm/neon-intrinsics.h which takes care of this.

This could potentially clash with the raid6 library rework I'm doing for 7.2. Although git has become pretty good about renamed files, so maybe it won't be so bad.

#8 lore raw

Re: [PATCH 3/8] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM

In reply to Christoph Hellwig
Quoted 4 lines · Christoph Hellwig

lib/raid/xor/arm64/xor-eor3.c line 10

Quoted 2 lines
lib/raid/xor/arm64/xor-eor3.c new file +2 -0
+extern void __xor_eor3_2(unsigned long bytes, unsigned long * __restrict p1,
+		const unsigned long * __restrict p2);

Does the alias magic prevent this from being in a header?

Yes, it emits the ELF symbol for the alias, and this is only permitted in the compilation unit that defines the original.

Quoted 2 lines

If so a comment
would be nice, otherwise moving it to a header would be even better.

Ack.

maintainer
#23 lore raw

Re: [PATCH 8/8] ARM: Remove hacked-up asm/types.h header

In reply to Ard Biesheuvel
Quoted 10 lines · Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

ARM has a special version of asm/types.h which contains overrides for certain #define's related to the C types used to back C99 types such as uint32_t and uintptr_t.

This is only needed when pulling in system headers such as stdint.h during the build, and this only happens when using NEON intrinsics, for which there is now a dedicated header file.

Quoted 7 lines

So drop this header entirely, and revert to the asm-generic one.

---
 arch/arm/include/uapi/asm/types.h | 41 --------------------
 1 file changed, 41 deletions(-)

This is actually a UAPI header. I guess it got put there accidentally and isn't actually needed there?

- Eric

maintainer
#18 lore raw

Re: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h

In reply to Christoph Hellwig
Quoted 9 lines · Christoph Hellwig
Quoted 6 lines · Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

arm_neon.h is a compiler header which needs some scaffolding to work correctly in the linux context, and so it is better not to include it directly. Both ARM and arm64 now provide asm/neon-intrinsics.h which takes care of this.

Quoted 4 lines

This could potentially clash with the raid6 library rework I'm doing for 7.2. Although git has become pretty good about renamed files, so maybe it won't be so bad.

I think this patch also breaks the userspace build of lib/raid6/. Which is going away in Christoph's series anyway, but maybe it would make sense to drop this patch (and patch 8 which depends on this, I think) from this series for now? That would make it a bit easier to take the rest through crc-next.

- Eric

Reviewed-by
#24 lore raw

Re: [PATCH 8/8] ARM: Remove hacked-up asm/types.h header

In reply to Eric Biggers
Quoted 7 lines · Eric Biggers
Quoted 6 lines · Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

ARM has a special version of asm/types.h which contains overrides for certain #define's related to the C types used to back C99 types such as uint32_t and uintptr_t.

Quoted 7 lines
Quoted 7 lines

This is only needed when pulling in system headers such as stdint.h during the build, and this only happens when using NEON intrinsics, for which there is now a dedicated header file.

So drop this header entirely, and revert to the asm-generic one.

Quoted 2 lines

This is actually a UAPI header. I guess it got put there accidentally and isn't actually needed there?

Yes, commit ed79c9d34f4f ("ARM: put types.h in uapi") has some explanations.

I can't think of any case where this would actually be used from userland, and lots of ways it could cause trouble in theory, even if it never has in practice.

Arnd

#19 lore raw

Re: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h

In reply to Eric Biggers
Quoted 2 lines · Eric Biggers

I think this patch also breaks the userspace build of lib/raid6/. Which is going away in Christoph's series anyway,

Assuming we're overcoming the objections. Anyway, about to repost this, and maybe Art is another voice for dropping the userspace build support of the RAID code.

maintainer
#20 lore raw

Re: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h

In reply to Eric Biggers
Quoted 15 lines · Eric Biggers
Quoted 9 lines · Christoph Hellwig
Quoted 6 lines · Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

arm_neon.h is a compiler header which needs some scaffolding to work correctly in the linux context, and so it is better not to include it directly. Both ARM and arm64 now provide asm/neon-intrinsics.h which takes care of this.

Quoted 4 lines

This could potentially clash with the raid6 library rework I'm doing for 7.2. Although git has become pretty good about renamed files, so maybe it won't be so bad.

Quoted 5 lines

I think this patch also breaks the userspace build of lib/raid6/. Which is going away in Christoph's series anyway, but maybe it would make sense to drop this patch (and patch 8 which depends on this, I think) from this series for now? That would make it a bit easier to take the rest through crc-next.

Ard, are you okay with me applying just patches 1-6 to crc-next?

- Eric

#21 lore raw

Re: [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h

Quoted 16 lines · Eric Biggers
Quoted 15 lines · Eric Biggers
Quoted 13 lines · Christoph Hellwig
Quoted 6 lines · Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

arm_neon.h is a compiler header which needs some scaffolding to work correctly in the linux context, and so it is better not to include it directly. Both ARM and arm64 now provide asm/neon-intrinsics.h which takes care of this.

This could potentially clash with the raid6 library rework I'm doing for 7.2. Although git has become pretty good about renamed files, so maybe it won't be so bad.

Quoted 7 lines
Quoted 5 lines

I think this patch also breaks the userspace build of lib/raid6/. Which is going away in Christoph's series anyway, but maybe it would make sense to drop this patch (and patch 8 which depends on this, I think) from this series for now? That would make it a bit easier to take the rest through crc-next.

Ard, are you okay with me applying just patches 1-6 to crc-next?

Yes that's fine - thanks.

maintainer
#25 lore raw
In reply to Ard Biesheuvel
Quoted 32 lines · Ard Biesheuvel

From: Ard Biesheuvel <[email protected]>

This is a follow-up to both [0] and [1], both of which included patch #1 of this series, which introduces the asm/neon-intrinsics.h header on 32-bit ARM. The remaining changes rely on this.

The purpose of this series is to streamline / clean up the use of NEON intrinsics on 32-bit ARM, by sharing more code, clean up Make rules and finally, getting rid of the hacked up types.h header, which does some nasty things that are only needed when building NEON intrinsics code.

Patches #2 and #3 replace the ARM autovectorized XOR implementation with the NEON intrinsics version used by arm64.

Patches #4 and #5 enable the arm64 NEON intrinsics implementation of crc64 on 32-bit ARM.

Patches #6 and #7 drop the direct includes of <arm_neon.h> and perform some additional cleanup to reduce the delta between ARM and arm64 code and Make rules.

It would probably be easiest to take all these changes through a single tree, and the CRC tree seems like a suitable candidate, if Eric agrees.

Cc Christoph Hellwig <[email protected]>
Cc Russell King <[email protected]>
Cc Arnd Bergmann <[email protected]>
Cc Eric Biggers <[email protected]>

[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/

Quoted 9 lines

Ard Biesheuvel (8):
ARM: Add a neon-intrinsics.h header like on arm64 xor/arm: Replace vectorized implementation with arm64's intrinsics xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM lib/crc: Turn NEON intrinsics crc64 implementation into common code lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64
crypto: aegis128 - Use neon-intrinsics.h on ARM too lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
ARM: Remove hacked-up asm/types.h header

Applied patches 1-6 to
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next

- Eric