lib/crc: x86: convert the CRC assembly to C intrinsics
Subsystems (4)
v2 cover lib/crc: x86: convert the CRC assembly to C intrinsics
[CHANGELOG]
v2:
- Use compiler's intrinsics header directly
- Convert all x86 asm files to C
Currently the x86 CRC code is all assembly: the generic [V]PCLMULQDQ template (12 functions: crc16, crc32 and crc64, three vector lengths each) and crc32c-3way.S. arm and arm64 already use C intrinsics.
This series converts all of it to C with the compiler's intrinsics, and removes the assembly. No algorithm, constant or calling convention changes, so the glue is untouched.
The goal is maintainability and portability, not speed: C is easier to read, review and extend than the macro-heavy assembly.
And it is also easier to port to userspace tools like btrfs-progs and xfsprogs which will require optimized CRC for offline operations.
Tests on a Zen5 VM show no obvious change in performance, except for clang with the MSB-first SSE and AVX2 functions, see below.
Patches:
1: asm/x86-intrinsics.h, a wrapper around <immintrin.h>.
2: crc32c_x86_3way().
3-6: crc32_lsb, crc64_msb, crc64_lsb and crc16_msb, each with the SSE4.1,
AVX2 and AVX512BW/VL variants. The template crc-pclmul-impl.h grows with them, and the assembly is removed in the last one.
Design decisions and trade-offs
===============================
- ISA selection: target attribute instead of per-file flags
Each function and helper has __attribute__((target(...))), the code is not built with -msse4.1, -mavx2 or -mavx512*. Per-file flags would need one object per ISA level, or the SSE function could use AVX-512.
Pros: all vector lengths of a CRC are in one file, and the compiler can only use an extension in the functions that run after the CPU feature checks (checked in the objdump).
Cons: no precedent in the kernel for selecting x86 ISA levels this way, and this is completely compiler dependent. See the GCC 8.1/8.2 issue below.
- One template, included once per function
The 12 functions differ only in vector length, CRC width and bit order, so crc-pclmul-impl.h generates one per inclusion, after the includer defined VL, CRC_BITS, CRC_MSB and CRC_FUNC.
I tried two different solutions:
* Nine standalone functions
About 1400 lines against about 490 with the template. And any fututre fixes needs to be duplicated 9 times.
* A DEFINE_CRC_PCLMUL_FUNCS() style macro
It can work, but the body cannot have #if, so the wide-vector stages become constant if()s and GCC warns about the dead code (-Wpsabi).
That macro also makes checkpatch warn about flow control in a macro, and compiler errors, gdb and perf point at the invocation line.
The include-as-template is not a common practice, but would be the least-worst solution I can come up with.
- Generated code and performance
Against the assembly (kbuild objects, GCC 16.2, clang 23): clang has the same total size for the 12 pclmul functions (7453 against 7482 bytes) and the same instruction count in each main loop.
GCC is 18% bigger (SSE +12%, AVX2 +22%, AVX-512 +19%): stack realignment in every function and, for AVX-512, loads not folded into vpternlogq (19 instead of 16 instructions in the LSB-first main loop).
crc32c_x86_3way() grows from 400 bytes to 603 (GCC) and 783 (clang), mostly prologue and unrolling of the tail. The main loop is 15, 17 and 16 instructions (asm, GCC, clang).
Every C function also gets endbr64 and the ftrace call, which the assembly never had. The call is patched to a nop at boot.
Throughput comes from a userspace harness using the same sources, built with the kernel's -mno-sse flags, on a Zen5 VM (Ryzen AI 9 HX 370): one pinned CPU, median of 7 runs. No in-kernel benchmark yet.
Peak is about 7.7, 15.4 and 30.7 GB/s for SSE, AVX2 and AVX-512. GCC is within 2% of the assembly from 1 KiB, up to 4-6% slower at 64-200 bytes and 8% for AVX2 at 512 bytes.
clang is the same, except crc64_msb and crc16_msb: SSE is 5-8% slower from 512 bytes, AVX2 5-7% from 1 KiB, but 14% faster at 512 bytes. AVX-512 is equal.
The clang SSE loop is 175 bytes against 153 for the assembly, as it keeps data in xmm8-xmm11 (REX prefixes). That is likely the cause, changing the XOR order did not help.
crc32c_x86_3way() is within 3% of the assembly: 8.6 to 9.0 GB/s at 200 bytes, 14.8 to 14.9 GB/s at 64 KiB.
Known issue: GCC 8.1 and 8.2
GCC 8.1 and 8.2 cannot build the AVX2 functions. Their <vpclmulqdqintrin.h> declares _mm256_clmulepi64_epi128() with avx512vl in its target, which fold256() lacks, so it cannot be inlined.
GCC 8.3 and later only need avx. I reproduced it on Compiler Explorer, where 8.3 to 9.5 build the code cleanly.
Calling __builtin_ia32_vpclmulqdq_v4di() directly for GCC older than 9 fixes it, but it is not included. 8.1 is still the documented minimum,
so: add that workaround, raise the minimum, or ignore 8.1 and 8.2?
Qu Wenruo (6):
x86: Add a wrapper header for the compiler's SIMD intrinsics lib/crc: x86: Convert crc32c_x86_3way() from assembly to C intrinsics lib/crc: x86: Convert crc32_lsb_*() from assembly to C lib/crc: x86: Convert crc64_msb_*() from assembly to C lib/crc: x86: Convert crc64_lsb_*() from assembly to C lib/crc: x86: Convert crc16_msb_*() from assembly to C
arch/x86/include/asm/x86-intrinsics.h | 49 +++ lib/crc/Makefile | 11 +- lib/crc/x86/crc-pclmul-helpers.h | 90 ++++ lib/crc/x86/crc-pclmul-impl.h | 304 ++++++++++++++ lib/crc/x86/crc-pclmul-template.S | 575 -------------------------- lib/crc/x86/crc-pclmul-template.h | 2 +- lib/crc/x86/crc16-msb-pclmul.S | 6 - lib/crc/x86/crc16-msb-pclmul.c | 23 ++ lib/crc/x86/crc32-pclmul.S | 6 - lib/crc/x86/crc32-pclmul.c | 23 ++ lib/crc/x86/crc32.h | 2 +- lib/crc/x86/crc32c-3way.S | 360 ---------------- lib/crc/x86/crc32c-3way.c | 264 ++++++++++++ lib/crc/x86/crc64-lsb-pclmul.c | 23 ++ lib/crc/x86/crc64-msb-pclmul.c | 23 ++ lib/crc/x86/crc64-pclmul.S | 7 - 16 files changed, 811 insertions(+), 957 deletions(-) create mode 100644 arch/x86/include/asm/x86-intrinsics.h create mode 100644 lib/crc/x86/crc-pclmul-helpers.h create mode 100644 lib/crc/x86/crc-pclmul-impl.h delete mode 100644 lib/crc/x86/crc-pclmul-template.S delete mode 100644 lib/crc/x86/crc16-msb-pclmul.S create mode 100644 lib/crc/x86/crc16-msb-pclmul.c delete mode 100644 lib/crc/x86/crc32-pclmul.S create mode 100644 lib/crc/x86/crc32-pclmul.c delete mode 100644 lib/crc/x86/crc32c-3way.S create mode 100644 lib/crc/x86/crc32c-3way.c create mode 100644 lib/crc/x86/crc64-lsb-pclmul.c create mode 100644 lib/crc/x86/crc64-msb-pclmul.c delete mode 100644 lib/crc/x86/crc64-pclmul.S
Signature · 1 line
2.56.0
RFC PATCH v2 1/6 x86: Add a wrapper header for the compiler's SIMD intrinsics
Allow kernel code to use SSE/AVX/PCLMULQDQ etc. from C through the compiler's intrinsics, as arm and arm64 already do through asm/neon-intrinsics.h.
There are some points to notice:
- Workaround for <mm_malloc.h> header
<immintrin.h> pulls in <xmmintrin.h> then <mm_malloc.h> for _mm_malloc() and mm_free(), which will further pull in <stdlib.h>.
No intrinsic depends on those memory allocation/free helpers. Workaround by pretending they are already included for both GCC and clang.
- Skip intrinsics for genksyms
Intrinsics expands to 30~50K lines, but nothing is exported. No need to expose those for genksyms. This is the same as arm64 NEON wrapper.
--- arch/x86/include/asm/x86-intrinsics.h | 49 +++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 arch/x86/include/asm/x86-intrinsics.h
arch/x86/include/asm/x86-intrinsics.h new file +49 -0
@@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Wrapper around the compiler's x86 intrinsics headers, so that kernel code +* can use e.g. _mm_crc32_u64() and _mm_clmulepi64_si128() from C. +* +* Modeled after arch/arm64/include/asm/neon-intrinsics.h. +* +* Users of this header should: +* +* - Enable the needed ISA extensions per function with +* __attribute__((target("..."))), e.g. target("sse4.2,pclmul"). +* The kernel is built without them (-mno-sse etc.), and this way the +* compiler can only use them inside the functions that are meant to +* run on CPUs that have them. The helpers calling intrinsics need the +* attribute too. +* +* - Build the object with -isystem $(CC -print-file-name=include). +* This is to allow including the compiler's intrinsics headers, which +* are not reachable by -nostdinc. +*/ + +#ifndef _ASM_X86_X86_INTRINSICS_H +#define _ASM_X86_X86_INTRINSICS_H + +#include <asm-generic/int-ll64.h> + +/* +* The compiler's <xmmintrin.h> pulls in <mm_malloc.h>, which needs <stdlib.h> +* for posix_memalign() and free(). Neither is wanted in the kernel, so pretend +* that <mm_malloc.h> was already included. +* +* The first is for GCC's header, the second for clang's. +*/ +#ifndef _MM_MALLOC_H_INCLUDED +#define _MM_MALLOC_H_INCLUDED +#endif +#ifndef __MM_MALLOC_H +#define __MM_MALLOC_H +#endif + +/* +* The intrinsics headers expand to 30~50K lines, but nothing exported. +* Hide them from genksyms, like the NEON wrapper does. +*/ +#ifndef __GENKSYMS__ +#include <immintrin.h> +#endif + +#endif /* _ASM_X86_X86_INTRINSICS_H */
Signature · 1 line
2.56.0
RFC PATCH v2 2/6 lib/crc: x86: Convert crc32c_x86_3way() from assembly to C intrinsics
In reply to Qu WenruoRewrite crc32c-3way.S in C using the new asm/x86-intrinsics.h. The algorithm and the constant table are unchanged: three lanes of up to 128 qwords are CRCed with crc32q, then combined with PCLMULQDQ.
Build changes:
- Enable SSE4.2 and PCLMULQDQ for the function only, with the target attribute, instead of per-object ISA flags.
- Add -isystem for the compiler's include directory, like arm64 does. It is in x86-intrin-cflags, for the later conversions to share.
- Use the path-qualified CFLAGS_x86/crc32c-3way.o, as it is in a subdir.
- Drop asmlinkage from the prototype in crc32.h, as it is C now.
- Use #pragma to keep the 4x unrolled loop of 3 crc32_u64() chains
Behavior changes:
- Always handle the unaligned leading part first Previously it's only for len >= SMALL_SIZE, which means if len < SMALL_SIZE, we can skip the unaligned handling.
This is not a big deal before, as later instructions can handle unaligned address, but still doesn't looks good to me.
Short unaligned input will pay up to 7 crc32_u8(), but should be rare.
The license header and authors are kept from the assembly file, and the SPDX tag now matches its dual license.
--- lib/crc/Makefile | 5 + lib/crc/x86/crc32.h | 2 +- lib/crc/x86/crc32c-3way.S | 360 -------------------------------------- lib/crc/x86/crc32c-3way.c | 264 ++++++++++++++++++++++++++++ 4 files changed, 270 insertions(+), 361 deletions(-) delete mode 100644 lib/crc/x86/crc32c-3way.S create mode 100644 lib/crc/x86/crc32c-3way.c
lib/crc/Makefile +5 -0
@@ -9,6 +9,10 @@ obj-$(CONFIG_CRC16) += crc16.o obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o +ifdef CONFIG_X86 +x86-intrin-cflags := -isystem $(shell $(CC) -print-file-name=include) +endif + obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o crc-t10dif-y := crc-t10dif-main.o ifeq ($(CONFIG_CRC_T10DIF_ARCH),y)
@@ -32,6 +36,7 @@ crc32-$(CONFIG_S390) += s390/crc32le-vx.o s390/crc32be-vx.o crc32-$(CONFIG_SPARC) += sparc/crc32c_asm.o crc32-$(CONFIG_X86) += x86/crc32-pclmul.o crc32-$(CONFIG_X86_64) += x86/crc32c-3way.o +CFLAGS_x86/crc32c-3way.o += $(x86-intrin-cflags) endif obj-$(CONFIG_CRC64) += crc64.o
lib/crc/x86/crc32.h +1 -1
@@ -34,7 +34,7 @@ static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)*/ #define CRC32C_PCLMUL_BREAKEVEN 512 -asmlinkage u32 crc32c_x86_3way(u32 crc, const u8 *buffer, size_t len); +u32 crc32c_x86_3way (u32 crc,const u8 *buffer,size_t len);static inline u32 crc32c_arch (u32 crc,const u8 *p,size_t len) {
lib/crc/x86/crc32c-3way.S deleted +0 -360
360 lines not on this page · raw message
lib/crc/x86/crc32c-3way.c new file +264 -0
264 lines not on this page · raw message
Signature · 1 line
2.56.0
RFC PATCH v2 3/6 lib/crc: x86: Convert crc32_lsb_*() from assembly to C
In reply to Qu WenruoRewrite the three crc32_lsb functions (PCLMULQDQ with SSE4.1, and VPCLMULQDQ with AVX2 or AVX512BW/VL) in C using the compiler's intrinsics, through asm/x86-intrinsics.h.
crc-pclmul-impl.h is the C version of crc-pclmul-template.S: a template, included once per function with the vector length, CRC width and bit order defined by the includer. Only what crc32_lsb needs is in it for now.
crc-pclmul-helpers.h has the small helpers: the loads and the fold step for each vector length.
Each function and helper has a target attribute for the ISA extensions it needs, so all the vector lengths are in one file. The compiler can only use those extensions inside the functions that run after the CPU feature checks.
Build changes:
- crc32-pclmul.c replaces crc32-pclmul.S. It only needs the compiler's include path, in x86-intrin-cflags.
Known issue: GCC 8.1 and 8.2, the oldest compilers the kernel supports, can not build the AVX2 function. Their <vpclmulqdqintrin.h> declares _mm256_clmulepi64_epi128() with avx512vl in its target, which fold256() does not have, so the call can not be inlined:
error: inlining failed in call to always_inline '_mm256_clmulepi64_epi128': target specific option mismatch
GCC 8.3 and later only need avx there. Not sure if we still need to support 8.1 and 8.2.
--- lib/crc/Makefile | 1 + lib/crc/x86/crc-pclmul-helpers.h | 72 +++++++++ lib/crc/x86/crc-pclmul-impl.h | 258 +++++++++++++++++++++++++++++++ lib/crc/x86/crc32-pclmul.S | 6 - lib/crc/x86/crc32-pclmul.c | 22 +++ 5 files changed, 353 insertions(+), 6 deletions(-) create mode 100644 lib/crc/x86/crc-pclmul-helpers.h create mode 100644 lib/crc/x86/crc-pclmul-impl.h delete mode 100644 lib/crc/x86/crc32-pclmul.S create mode 100644 lib/crc/x86/crc32-pclmul.c
lib/crc/Makefile +1 -0
@@ -37,6 +37,7 @@ crc32-$(CONFIG_SPARC) += sparc/crc32c_asm.o crc32-$(CONFIG_X86) += x86/crc32-pclmul.o crc32-$(CONFIG_X86_64) += x86/crc32c-3way.o CFLAGS_x86/crc32c-3way.o += $(x86-intrin-cflags) +CFLAGS_x86/crc32-pclmul.o += $(x86-intrin-cflags) endif obj-$(CONFIG_CRC64) += crc64.o
lib/crc/x86/crc-pclmul-helpers.h new file +72 -0
72 lines not on this page · raw message
lib/crc/x86/crc-pclmul-impl.h new file +258 -0
258 lines not on this page · raw message
lib/crc/x86/crc32-pclmul.S deleted +0 -6
@@ -1,6 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -// Copyright 2025 Google LLC - -#include "crc-pclmul-template.S" - -DEFINE_CRC_PCLMUL_FUNCS(crc32_lsb, /* bits= */ 32, /* lsb= */ 1)
lib/crc/x86/crc32-pclmul.c new file +22 -0
@@ -0,0 +1,22 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* CRC-32 using [V]PCLMULQDQ, via C intrinsics +* +* Copyright 2025 Google LLC +*/ + +#include <linux/types.h> + +#define CRC_BITS 32 + +#define VL 16 +#define CRC_FUNC crc32_lsb_pclmul_sse +#include "crc-pclmul-impl.h" + +#define VL 32 +#define CRC_FUNC crc32_lsb_vpclmul_avx2 +#include "crc-pclmul-impl.h" + +#define VL 64 +#define CRC_FUNC crc32_lsb_vpclmul_avx512 +#include "crc-pclmul-impl.h"
Signature · 1 line
2.56.0
[email protected]
AI reviewRe: [PATCH RFC v2 3/6] lib/crc: x86: Convert crc32_lsb_*() from assembly to C
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] lib/crc: x86: build failure in fold256() on GCC 8.1 and 8.2 --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] lib/crc: x86: build failure in fold256() on GCC 8.1 and 8.2 --
commit 4e73f2c859c18b715ef71e051a7e40e392c143ab
Author: Qu Wenruo <[email protected]>
lib/crc: x86: Convert crc32_lsb_*() from assembly to C
This commit rewrites the x86 PCLMULQDQ and VPCLMULQDQ crc32_lsb functions from assembly to C using compiler intrinsics.
Quoted 6 lines
lib/crc/x86/crc-pclmul-helpers.h new file +0
-0@@ -0,0 +1,72 @@
[ ... ]
Quoted 9 lines
lib/crc/x86/crc-pclmul-helpers.h new file +9
-0+/* The ISA extensions needed for each vector length */ +#define CRC_TARGET_128 "sse4.1,pclmul" +#define CRC_TARGET_256 "sse4.1,pclmul,avx2,vpclmulqdq" +#define CRC_TARGET_512 "sse4.1,pclmul,avx2,vpclmulqdq,avx512f,avx512bw,avx512vl" + +/* The target attributes of the helpers, to keep their signatures short */ +#define __crc128 __attribute__((target(CRC_TARGET_128))) +#define __crc256 __attribute__((target(CRC_TARGET_256))) +#define __crc512 __attribute__((target(CRC_TARGET_512)))
[ ... ]
lib/crc/x86/crc-pclmul-helpers.h line 57
Quoted 7 lines
lib/crc/x86/crc-pclmul-helpers.h new file +7
-0+static __always_inline __crc256 __m256i fold256(__m256i acc, __m256i data, __m256i k) +{ +__m256i lo =_mm256_clmulepi64_epi128 (acc, k,0x00 ); +__m256i hi =_mm256_clmulepi64_epi128 (acc, k,0x11 ); + +return _mm256_xor_si256 (_mm256_xor_si256 (hi, data), lo); +}
[Severity: Medium]
Does fold256() break x86 builds with CONFIG_CRC32_ARCH=y on GCC 8.1 and 8.2?
As noted in the commit message, GCC 8.1 and 8.2 declare _mm256_clmulepi64_epi128() under #pragma GCC target("vpclmulqdq,avx512vl"), which fails to inline into fold256() because CRC_TARGET_256 omits avx512vl. Since scripts/min-tool-version.sh still declares 8.1.0 as the minimum supported GCC version, fold256() needs a workaround for GCC < 8.3 (such as calling __builtin_ia32_vpclmulqdq_v4di directly) unless the minimum GCC version is raised first.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
RFC PATCH v2 4/6 lib/crc: x86: Convert crc64_msb_*() from assembly to C
In reply to Qu WenruoRewrite the three crc64_msb functions in C using the template, which now supports MSB-first CRCs, 64 bits wide.
MSB-first CRCs load the data byte-reflected, with the new load_bswap*() helpers, put the initial CRC at the other end of the vector, and index the shuffle table from the other end.
crc64-pclmul.S now only has the crc64_lsb functions.
Build changes:
- Add crc64-msb-pclmul.o, built with x86-intrin-cflags.
--- lib/crc/Makefile | 3 +- lib/crc/x86/crc-pclmul-helpers.h | 18 +++++++++++ lib/crc/x86/crc-pclmul-impl.h | 52 ++++++++++++++++++++++++-------- lib/crc/x86/crc32-pclmul.c | 1 + lib/crc/x86/crc64-msb-pclmul.c | 23 ++++++++++++++ lib/crc/x86/crc64-pclmul.S | 1 - 6 files changed, 83 insertions(+), 15 deletions(-) create mode 100644 lib/crc/x86/crc64-msb-pclmul.c
lib/crc/Makefile +2 -1
@@ -53,7 +53,8 @@ crc64-$(CONFIG_ARM) += crc64-neon.o crc64-$(CONFIG_ARM64) += crc64-neon.o crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o -crc64-$(CONFIG_X86) += x86/crc64-pclmul.o +crc64-$(CONFIG_X86) += x86/crc64-pclmul.o x86/crc64-msb-pclmul.o +CFLAGS_x86/crc64-msb-pclmul.o += $(x86-intrin-cflags) endif # CONFIG_CRC64_ARCH obj-y += tests/
lib/crc/x86/crc-pclmul-helpers.h +18 -0
@@ -30,6 +30,12 @@ static __always_inline __crc128 __m128i loadu128(const u8 *p)return _mm_loadu_si128 ((const __m128i *)p); } +/* Load 16 bytes, and reflect the bytes within each 128-bit lane. */ +static __always_inline __crc128 __m128i load_bswap128(const u8 *p, __m128i bswap) +{ +return _mm_shuffle_epi8 (loadu128 (p), bswap); +} +/* * Fold acc into data, across the distance that k was generated for: the qwords * of k and acc with the same index are multiplied. The compilers turn the
@@ -48,6 +54,12 @@ static __always_inline __crc256 __m256i loadu256(const u8 *p)return _mm256_loadu_si256 ((const __m256i *)p); } +/* Load 32 bytes, and reflect the bytes within each 128-bit lane. */ +static __always_inline __crc256 __m256i load_bswap256(const u8 *p, __m256i bswap) +{ +return _mm256_shuffle_epi8 (loadu256 (p), bswap); +} + static __always_inline __crc256 __m256i fold256(__m256i acc, __m256i data, __m256i k) {__m256i lo =_mm256_clmulepi64_epi128 (acc, k,0x00 );
@@ -61,6 +73,12 @@ static __always_inline __crc512 __m512i loadu512(const u8 *p)return _mm512_loadu_si512 (p); } +/* Load 64 bytes, and reflect the bytes within each 128-bit lane. */ +static __always_inline __crc512 __m512i load_bswap512(const u8 *p, __m512i bswap) +{ +return _mm512_shuffle_epi8 (loadu512 (p), bswap); +} + static __always_inline __crc512 __m512i fold512(__m512i acc, __m512i data, __m512i k) {__m512i lo =_mm512_clmulepi64_epi128 (acc, k,0x00 );
lib/crc/x86/crc-pclmul-impl.h +39 -13
138 lines not on this page · raw message
lib/crc/x86/crc32-pclmul.c +1 -0
@@ -8,6 +8,7 @@#include <linux/types.h> #define CRC_BITS 32 +#define CRC_MSB 0#define VL 16#define CRC_FUNC crc32_lsb_pclmul_sse
lib/crc/x86/crc64-msb-pclmul.c new file +23 -0
@@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* CRC-64 (MSB-first) using [V]PCLMULQDQ, via C intrinsics +* +* Copyright 2025 Google LLC +*/ + +#include <linux/types.h> + +#define CRC_BITS 64 +#define CRC_MSB 1 + +#define VL 16 +#define CRC_FUNC crc64_msb_pclmul_sse +#include "crc-pclmul-impl.h" + +#define VL 32 +#define CRC_FUNC crc64_msb_vpclmul_avx2 +#include "crc-pclmul-impl.h" + +#define VL 64 +#define CRC_FUNC crc64_msb_vpclmul_avx512 +#include "crc-pclmul-impl.h"
lib/crc/x86/crc64-pclmul.S +0 -1
@@ -3,5 +3,4 @@ #include "crc-pclmul-template.S" -DEFINE_CRC_PCLMUL_FUNCS(crc64_msb, /* bits= */ 64, /* lsb= */ 0) DEFINE_CRC_PCLMUL_FUNCS(crc64_lsb, /* bits= */ 64, /* lsb= */ 1)
Signature · 1 line
2.56.0
RFC PATCH v2 5/6 lib/crc: x86: Convert crc64_lsb_*() from assembly to C
In reply to Qu WenruoRewrite the three crc64_lsb functions in C using the template, which now supports the final reduction of LSB-first 64-bit CRCs.
Remove crc64-pclmul.S, as nothing is left in it.
Build changes:
- Add crc64-lsb-pclmul.o, built with x86-intrin-cflags, and drop crc64-pclmul.o.
--- lib/crc/Makefile | 3 ++- lib/crc/x86/crc-pclmul-impl.h | 14 ++++++++++++++ lib/crc/x86/crc64-lsb-pclmul.c | 23 +++++++++++++++++++++++ lib/crc/x86/crc64-pclmul.S | 6 ------ 4 files changed, 39 insertions(+), 7 deletions(-) create mode 100644 lib/crc/x86/crc64-lsb-pclmul.c delete mode 100644 lib/crc/x86/crc64-pclmul.S
lib/crc/Makefile +2 -1
@@ -53,8 +53,9 @@ crc64-$(CONFIG_ARM) += crc64-neon.o crc64-$(CONFIG_ARM64) += crc64-neon.o crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o -crc64-$(CONFIG_X86) += x86/crc64-pclmul.o x86/crc64-msb-pclmul.o +crc64-$(CONFIG_X86) += x86/crc64-msb-pclmul.o x86/crc64-lsb-pclmul.o CFLAGS_x86/crc64-msb-pclmul.o += $(x86-intrin-cflags) +CFLAGS_x86/crc64-lsb-pclmul.o += $(x86-intrin-cflags) endif # CONFIG_CRC64_ARCH obj-y += tests/
lib/crc/x86/crc-pclmul-impl.h +14 -0
@@ -264,6 +264,20 @@ crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts_ptr) x1 =_mm_clmulepi64_si128 (x1, k128,0x10 ); x0 =_mm_xor_si128 (x0, x1);return _mm_extract_epi32 (x0,2 ); +#else +/* +* For an LSB-first CRC-64 the low qword of x1, floor(t0 / G), must also +* be XORed into the CRC. Move it to the high qword, the one that is +* extracted, so that the two XORs can be done together. +*/ + x1 =_mm_clmulepi64_si128 (x0, k128,0x10 ); + x0 =_mm_xor_si128 (_mm_srli_si128 (x0,8 ), x1); + k128 = consts[4 ]; + x1 =_mm_clmulepi64_si128 (x0, k128,0x00 ); + x0 =_mm_xor_si128 (x0,_mm_unpacklo_epi64 (x1, x1)); + x1 =_mm_clmulepi64_si128 (x1, k128,0x10 ); + x0 =_mm_xor_si128 (x0, x1); +return _mm_extract_epi64 (x0,1 );#endif /* CRC_MSB */ }
lib/crc/x86/crc64-lsb-pclmul.c new file +23 -0
@@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* CRC-64 (LSB-first) using [V]PCLMULQDQ, via C intrinsics +* +* Copyright 2025 Google LLC +*/ + +#include <linux/types.h> + +#define CRC_BITS 64 +#define CRC_MSB 0 + +#define VL 16 +#define CRC_FUNC crc64_lsb_pclmul_sse +#include "crc-pclmul-impl.h" + +#define VL 32 +#define CRC_FUNC crc64_lsb_vpclmul_avx2 +#include "crc-pclmul-impl.h" + +#define VL 64 +#define CRC_FUNC crc64_lsb_vpclmul_avx512 +#include "crc-pclmul-impl.h"
lib/crc/x86/crc64-pclmul.S deleted +0 -6
@@ -1,6 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -// Copyright 2025 Google LLC - -#include "crc-pclmul-template.S" - -DEFINE_CRC_PCLMUL_FUNCS(crc64_lsb, /* bits= */ 64, /* lsb= */ 1)
Signature · 1 line
2.56.0
RFC PATCH v2 6/6 lib/crc: x86: Convert crc16_msb_*() from assembly to C
In reply to Qu WenruoRewrite the three crc16_msb functions in C using the template, which now supports 16-bit CRCs.
Remove crc16-msb-pclmul.S, and crc-pclmul-template.S which is not used by anything any more.
Build changes:
- crc16-msb-pclmul.c replaces crc16-msb-pclmul.S, built with x86-intrin-cflags.
--- lib/crc/Makefile | 1 + lib/crc/x86/crc-pclmul-impl.h | 8 +- lib/crc/x86/crc-pclmul-template.S | 575 ------------------------------ lib/crc/x86/crc-pclmul-template.h | 2 +- lib/crc/x86/crc16-msb-pclmul.S | 6 - lib/crc/x86/crc16-msb-pclmul.c | 23 ++ 6 files changed, 32 insertions(+), 583 deletions(-) delete mode 100644 lib/crc/x86/crc-pclmul-template.S delete mode 100644 lib/crc/x86/crc16-msb-pclmul.S create mode 100644 lib/crc/x86/crc16-msb-pclmul.c
lib/crc/Makefile +1 -0
@@ -22,6 +22,7 @@ crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o crc-t10dif-$(CONFIG_PPC) += powerpc/crct10dif-vpmsum_asm.o crc-t10dif-$(CONFIG_RISCV) += riscv/crc16_msb.o crc-t10dif-$(CONFIG_X86) += x86/crc16-msb-pclmul.o +CFLAGS_x86/crc16-msb-pclmul.o += $(x86-intrin-cflags) endif obj-$(CONFIG_CRC32) += crc32.o
lib/crc/x86/crc-pclmul-impl.h +7 -1
@@ -10,7 +10,8 @@* VL The vector length in bytes: 16 (PCLMULQDQ with SSE4.1), 32 * (VPCLMULQDQ with AVX2) or 64 (VPCLMULQDQ with AVX512BW/VL). The * function gets the matching target attribute. -* CRC_BITS The number of bits in the CRC. Currently only 32 is supported. +* CRC_BITS The number of bits in the CRC. Currently 16, 32 and 64 are +* supported. * CRC_MSB 1 if the CRC processes the most significant bit of each byte * first, i.e. maps bit0 to x^0, ..., bit7 to x^7. 0 if it * processes the least significant bit first, i.e. maps bit0 to x^7,
@@ -37,6 +38,9 @@#if CRC_BITS ==64 #define crc_t u64#define CRC_TO_VEC (crc) _mm_cvtsi64_si128(crc) +#elif CRC_BITS ==16 +#define crc_t u16 +#define CRC_TO_VEC (crc) _mm_cvtsi32_si128(crc)#else #define crc_t u32#define CRC_TO_VEC (crc) _mm_cvtsi32_si128(crc)
@@ -255,6 +259,8 @@ crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts_ptr) x0 =_mm_xor_si128 (x0, x1);#if CRC_BITS ==64 return _mm_cvtsi128_si64 (x0); +#elif CRC_BITS ==16 +return _mm_extract_epi16 (x0,3 );#endif /* CRC_BITS == 64 */ #elif CRC_BITS ==32 x1 =_mm_clmulepi64_si128 (x0, k128,0x10 );
lib/crc/x86/crc-pclmul-template.S deleted +0 -575
575 lines not on this page · raw message
lib/crc/x86/crc-pclmul-template.h +1 -1
@@ -1,7 +1,7 @@/* SPDX-License-Identifier: GPL-2.0-or-later */ /* * Macros for accessing the [V]PCLMULQDQ-based CRC functions that are -* instantiated by crc-pclmul-template.S +* implemented by the crc*-pclmul.c files * * Copyright 2025 Google LLC *
lib/crc/x86/crc16-msb-pclmul.S deleted +0 -6
@@ -1,6 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -// Copyright 2025 Google LLC - -#include "crc-pclmul-template.S" - -DEFINE_CRC_PCLMUL_FUNCS(crc16_msb, /* bits= */ 16, /* lsb= */ 0)
lib/crc/x86/crc16-msb-pclmul.c new file +23 -0
@@ -0,0 +1,23 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* CRC-16 (MSB-first, as used by T10 DIF) using [V]PCLMULQDQ, via C intrinsics +* +* Copyright 2025 Google LLC +*/ + +#include <linux/types.h> + +#define CRC_BITS 16 +#define CRC_MSB 1 + +#define VL 16 +#define CRC_FUNC crc16_msb_pclmul_sse +#include "crc-pclmul-impl.h" + +#define VL 32 +#define CRC_FUNC crc16_msb_vpclmul_avx2 +#include "crc-pclmul-impl.h" + +#define VL 64 +#define CRC_FUNC crc16_msb_vpclmul_avx512 +#include "crc-pclmul-impl.h"
Signature · 1 line
2.56.0
Eric Biggers
maintainer[ +Cc [email protected] ]
Hi Qu,
Quoted 148 lines · Qu Wenruo
[CHANGELOG]
v2:
- Use compiler's intrinsics header directly
- Convert all x86 asm files to CCurrently the x86 CRC code is all assembly: the generic [V]PCLMULQDQ template (12 functions: crc16, crc32 and crc64, three vector lengths each) and
crc32c-3way.S. arm and arm64 already use C intrinsics.This series converts all of it to C with the compiler's intrinsics, and removes the assembly. No algorithm, constant or calling convention changes, so the glue is untouched.
The goal is maintainability and portability, not speed: C is easier to read, review and extend than the macro-heavy assembly.
And it is also easier to port to userspace tools like btrfs-progs and xfsprogs which will require optimized CRC for offline operations.Tests on a Zen5 VM show no obvious change in performance, except for clang with the MSB-first SSE and AVX2 functions, see below.
Patches:
1:
asm/x86-intrinsics.h, a wrapper around <immintrin.h>.
2:crc32c_x86_3way().
3-6:crc32_lsb,crc64_msb,crc64_lsbandcrc16_msb, each with the SSE4.1,
AVX2 and AVX512BW/VL variants. The templatecrc-pclmul-impl.hgrows with them, and the assembly is removed in the last one.Design decisions and trade-offs
===============================(quoted diff) +0
-3- ISA selection: target attribute instead of per-file flags Each function and helper has __attribute__((target(...))), the code is not built with -msse4.1, -mavx2 or -mavx512*. Per-file flags would need one object per ISA level, or the SSE function could use AVX-512. Pros: all vector lengths of a CRC are in one file, and the compiler can only use an extension in the functions that run after the CPU feature checks (checked in the objdump). Cons: no precedent in the kernel for selecting x86 ISA levels this way, and this is completely compiler dependent. See the GCC 8.1/8.2 issue below. - One template, included once per function The 12 functions differ only in vector length, CRC width and bit order, so crc-pclmul-impl.h generates one per inclusion, after the includer defined VL, CRC_BITS, CRC_MSB and CRC_FUNC. I tried two different solutions: * Nine standalone functions About 1400 lines against about 490 with the template. And any fututre fixes needs to be duplicated 9 times. * A DEFINE_CRC_PCLMUL_FUNCS() style macro It can work, but the body cannot have #if, so the wide-vector stages become constant if()s and GCC warns about the dead code (-Wpsabi). That macro also makes checkpatch warn about flow control in a macro, and compiler errors, gdb and perf point at the invocation line. The include-as-template is not a common practice, but would be the least-worst solution I can come up with. - Generated code and performance Against the assembly (kbuild objects, GCC 16.2, clang 23): clang has the same total size for the 12 pclmul functions (7453 against 7482 bytes) and the same instruction count in each main loop. GCC is 18% bigger (SSE +12%, AVX2 +22%, AVX-512 +19%): stack realignment in every function and, for AVX-512, loads not folded into vpternlogq (19 instead of 16 instructions in the LSB-first main loop). crc32c_x86_3way() grows from 400 bytes to 603 (GCC) and 783 (clang), mostly prologue and unrolling of the tail. The main loop is 15, 17 and 16 instructions (asm, GCC, clang). Every C function also gets endbr64 and the ftrace call, which the assembly never had. The call is patched to a nop at boot. Throughput comes from a userspace harness using the same sources, built with the kernel's -mno-sse flags, on a Zen5 VM (Ryzen AI 9 HX 370): one pinned CPU, median of 7 runs. No in-kernel benchmark yet. Peak is about 7.7, 15.4 and 30.7 GB/s for SSE, AVX2 and AVX-512. GCC is within 2% of the assembly from 1 KiB, up to 4-6% slower at 64-200 bytes and 8% for AVX2 at 512 bytes. clang is the same, except crc64_msb and crc16_msb: SSE is 5-8% slower from 512 bytes, AVX2 5-7% from 1 KiB, but 14% faster at 512 bytes. AVX-512 is equal. The clang SSE loop is 175 bytes against 153 for the assembly, as it keeps data in xmm8-xmm11 (REX prefixes). That is likely the cause, changing the XOR order did not help. crc32c_x86_3way() is within 3% of the assembly: 8.6 to 9.0 GB/s at 200 bytes, 14.8 to 14.9 GB/s at 64 KiB.Known issue: GCC 8.1 and 8.2
GCC 8.1 and 8.2 cannot build the AVX2 functions. Their <
vpclmulqdqintrin.h> declares_mm256_clmulepi64_epi128()with avx512vl in its target, whichfold256()lacks, so it cannot be inlined.GCC 8.3 and later only need avx. I reproduced it on Compiler Explorer, where 8.3 to 9.5 build the code cleanly.
Calling
__builtin_ia32_vpclmulqdq_v4di()directly for GCC older than 9 fixes it, but it is not included. 8.1 is still the documented minimum,
so: add that workaround, raise the minimum, or ignore 8.1 and 8.2?Qu Wenruo (6):
x86: Add a wrapper header for the compiler's SIMD intrinsics lib/crc: x86: Convertcrc32c_x86_3way()from assembly to C intrinsics lib/crc: x86: Convert crc32_lsb_*() from assembly to C lib/crc: x86: Convert crc64_msb_*() from assembly to C lib/crc: x86: Convert crc64_lsb_*() from assembly to C lib/crc: x86: Convert crc16_msb_*() from assembly to Carch/x86/include/asm/x86-intrinsics.h | 49 +++ lib/crc/Makefile | 11 +- lib/crc/x86/crc-pclmul-helpers.h | 90 ++++ lib/crc/x86/crc-pclmul-impl.h | 304 ++++++++++++++ lib/crc/x86/crc-pclmul-template.S | 575 -------------------------- lib/crc/x86/crc-pclmul-template.h | 2 +- lib/crc/x86/crc16-msb-pclmul.S | 6 - lib/crc/x86/crc16-msb-pclmul.c | 23 ++ lib/crc/x86/crc32-pclmul.S | 6 - lib/crc/x86/crc32-pclmul.c | 23 ++ lib/crc/x86/crc32.h | 2 +- lib/crc/x86/crc32c-3way.S | 360 ---------------- lib/crc/x86/crc32c-3way.c | 264 ++++++++++++ lib/crc/x86/crc64-lsb-pclmul.c | 23 ++ lib/crc/x86/crc64-msb-pclmul.c | 23 ++ lib/crc/x86/crc64-pclmul.S | 7 - 16 files changed, 811 insertions(+), 957 deletions(-) create mode 100644 arch/x86/include/asm/x86-intrinsics.h create mode 100644 lib/crc/x86/crc-pclmul-helpers.h create mode 100644 lib/crc/x86/crc-pclmul-impl.h delete mode 100644 lib/crc/x86/crc-pclmul-template.S delete mode 100644 lib/crc/x86/crc16-msb-pclmul.S
Quoted 8 lines
create mode 100644 lib/crc/x86/crc16-msb-pclmul.c delete mode 100644 lib/crc/x86/crc32-pclmul.S create mode 100644 lib/crc/x86/crc32-pclmul.c delete mode 100644 lib/crc/x86/crc32c-3way.S create mode 100644 lib/crc/x86/crc32c-3way.c create mode 100644 lib/crc/x86/crc64-lsb-pclmul.c create mode 100644 lib/crc/x86/crc64-msb-pclmul.c delete mode 100644 lib/crc/x86/crc64-pclmul.S
Thanks for exploring this.
I think we should be honest: this would be a negative for maintainability at least in the short term. We'd be replacing existing well-optimized and well-documented code with an experimental use of x86 C intrinsics in the kernel, with many of the original comments stripped out or made inconsistent, and signing up to fight with compiler issues (including inconsistent support between compilers, having to use error-prone hacks to use the x86 intrinsics in the kernel at all, and inefficient code generation which in some cases might not be solvable without contributing fixes to gcc/clang).
I've used the x86 C intrinsics extensively in userspace projects. So sure, some of these compiler issues (minus the ones specific to trying to use the intrinsics in the kernel) are things I'm used to already. It's just been nice to not have to deal with them in the kernel.
Note that gcc can't even generate simple loops using vector intrinsics without emitting unnecessary move instructions. I reported it years ago
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107892) and it's still unfixed. So realistically, we either need to simply tolerate bad codegen like that, or actually step up and actively contribute fixes to the compilers (as merely opening a bug report does not result in a fix).
And as you mentioned in your cover letter, a lot of the code grew in size and became slower, especially for short inputs which are important for some in-kernel use cases. So clearly there is some bad codegen going on compared to what was there before.
I kind of feel this could only be a benefit in the long term if we actually go all in and commit to switching the crypto code over to this style of implementation as well. But that area has a lot more code. And that code also has another thorny issue: when operating on secrets, it has to make sure to only use x86 instructions that are in the DOITM (Data Operand Independent Timing Mode) subset. Yet there's no compiler support for restricting the used instructions to that subset, and no announced plans from anyone (Intel, AMD, or otherwise) to implement it.
I'll also note that the hardest part to understand about this code (IMO) is actually the polynomial math, especially the Barrett reduction. And yet this series deletes or greatly shortens many of the comments explaining it, leaving unexplained operations, factually wrong comments (especially for the Barrett reduction), and dangling references. That's most of the negative diff, too.
So if we're going to proceed, I'd at least like to see a version that compares a bit more favorably to the original. But also I think before diving into this approach I'd like wider agreement in the Linux community (x86 maintainers, crypto, etc.) that the x86 intrinsics are the way to go, and for there to be a path forward for DOITM as well, and hopefully some interest in improving the compilers too.
- Eric
Quoted 5 lines · Eric Biggers
[ +Cc [email protected] ]
Hi Qu,
On Fri, Nov 20, 2026 at 11:48:58AM +1030, Qu Wenruo wrote:
[...]
Quoted 8 lines
Thanks for exploring this.
I think we should be honest: this would be a negative for maintainability at least in the short term. We'd be replacing existing well-optimized and well-documented code with an experimental use of x86 C intrinsics in the kernel, with many of the original comments stripped out or made inconsistent,
That's indeed a huge problem, thanks for pointing this out.
Sorry that I didn't manually compare the comment part before and after the change, that's all my bad.
That's in fact the most important part for readability, and my focus on the generated code blinded me on this important part.
Quoted 3 lines
and signing up to fight with compiler issues
(including inconsistent support between compilers, having to use error-prone hacks to use the x86 intrinsics in the kernel at all,
To be honest, to make kernel use intrinsics with current latest gcc/clang is easier than my initial expectation.
And even for ancient ones, only GCC 8.1/8.2 are really problematic, again better than my expectation, but still requires extra adhoc fixes for certain compilers.
Quoted 9 lines
and
inefficient code generation which in some cases might not be solvable without contributing fixes to gcc/clang).I've used the x86 C intrinsics extensively in userspace projects. So sure, some of these compiler issues (minus the ones specific to trying to use the intrinsics in the kernel) are things I'm used to already. It's just been nice to not have to deal with them in the kernel.
Quoted 6 lines
Note that gcc can't even generate simple loops using vector intrinsics without emitting unnecessary move instructions. I reported it years ago
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107892) and it's still unfixed. So realistically, we either need to simply tolerate bad codegen like that, or actually step up and actively contribute fixes to the compilers (as merely opening a bug report does not result in a fix).
Personally speaking I'd also prefer the compiler to do a lot of things better, include not pulling in the memory allocation headers just for x86 intrinsics.
Quoted 22 lines
And as you mentioned in your cover letter, a lot of the code grew in size and became slower, especially for short inputs which are important for some in-kernel use cases. So clearly there is some bad codegen going on compared to what was there before.
I kind of feel this could only be a benefit in the long term if we actually go all in and commit to switching the crypto code over to this style of implementation as well. But that area has a lot more code. And that code also has another thorny issue: when operating on secrets, it has to make sure to only use x86 instructions that are in the DOITM (Data Operand Independent Timing Mode) subset. Yet there's no compiler support for restricting the used instructions to that subset, and no announced plans from anyone (Intel, AMD, or otherwise) to implement it.
I'll also note that the hardest part to understand about this code (IMO) is actually the polynomial math, especially the Barrett reduction. And yet this series deletes or greatly shortens many of the comments explaining it, leaving unexplained operations, factually wrong comments (especially for the Barrett reduction), and dangling references. That's most of the negative diff, too.
Quoted 6 lines
So if we're going to proceed, I'd at least like to see a version that compares a bit more favorably to the original. But also I think before diving into this approach I'd like wider agreement in the Linux community (x86 maintainers, crypto, etc.) that the x86 intrinsics are the way to go, and for there to be a path forward for DOITM as well, and hopefully some interest in improving the compilers too.
Sure, that's a very reasonable argument, and I'm definitely stepping beyond my ability and initial objective (just to use the same intrinsics code in both kernel and progs).
I can definitely wait for more feedback on this topic before continue any future work.
And to my initial objective, I'd check other user space CRC32C implementation as a reference for now.
Thanks,
Qu
Quoted 2 lines
- Eric
Quoted 6 lines · Eric Biggers
So if we're going to proceed, I'd at least like to see a version that compares a bit more favorably to the original. But also I think before diving into this approach I'd like wider agreement in the Linux community (x86 maintainers, crypto, etc.) that the x86 intrinsics are the way to go, and for there to be a path forward for DOITM as well, and hopefully some interest in improving the compilers too.
My 2cents go towards keeping the asm. There is no point in fighting compilers over this. The very same is true for atomics.
Quoted 9 lines · On Fri, 20 Nov 2026 09:41:58 +0100 Peter Zijlstra <[email protected]> wrote:
Quoted 6 lines · Eric Biggers
So if we're going to proceed, I'd at least like to see a version that compares a bit more favorably to the original. But also I think before diving into this approach I'd like wider agreement in the Linux community (x86 maintainers, crypto, etc.) that the x86 intrinsics are the way to go, and for there to be a path forward for DOITM as well, and hopefully some interest in improving the compilers too.
Quoted 3 lines
My 2cents go towards keeping the asm. There is no point in fighting compilers over this. The very same is true for atomics.
Indeed.
IIRC the code loop had to go asm block because the compiler pretty much refused to generate the loop that matched the source, instead it generated one that added extra clock(s) to the loop making it slower.
I've also use some of the 'intrinsics' to write a short C function. I basically worked out the asm instructions I wanted then tried to find the correct intrinsic.
The whole thing would have been just as readable in asm (possibly with some #defines to give the registers names).
David