RISC-V CRC optimizations

PATCH In review Eric Biggers 5 participants 12 messages Feb 16, 2025 – Mar 10, 2025

Subsystems (3)

RISC-V Crypto API CRC library

lore

Share & tools

lore

Eric Biggers

author maintainer

2025-02-16 23:00 UTC #0 lore raw

cover RISC-V CRC optimizations

This patchset is a replacement for
"[PATCH v4] riscv: Optimize crct10dif with Zbc extension"
(https://lore.kernel.org/r/[email protected]/).
It adopts the approach that I'm taking for x86 where code is shared among CRC variants. It replaces the existing Zbc optimized CRC32 functions, then adds Zbc optimized CRC-T10DIF and CRC64 functions.

This new code should be significantly faster than the current Zbc optimized CRC32 code and the previously proposed CRC-T10DIF code. It uses "folding" instead of just Barrett reduction, and it also implements Barrett reduction more efficiently.

This applies to crc-next at
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next.
It depends on other patches that are queued there for 6.15, so I plan to take it through there if there are no objections.

Tested with crc_kunit in QEMU (set CONFIG_CRC_KUNIT_TEST=y and CONFIG_CRC_BENCHMARK=y), both 32-bit and 64-bit. I don't have real Zbc capable hardware to benchmark this on, but the new code should work very well; similar optimizations work very well on other architectures.

Eric Biggers (4):
riscv/crc: add "template" for Zbc optimized CRC functions riscv/crc32: reimplement the CRC32 functions using new template riscv/crc-t10dif: add Zbc optimized CRC-T10DIF function riscv/crc64: add Zbc optimized CRC64 functions

 arch/riscv/Kconfig                  |   2 +
 arch/riscv/lib/Makefile             |   5 +
 arch/riscv/lib/crc-clmul-consts.h   | 122 +++++++++++
 arch/riscv/lib/crc-clmul-template.h | 265 ++++++++++++++++++++++++
 arch/riscv/lib/crc-clmul.h          |  23 +++
 arch/riscv/lib/crc-t10dif.c         |  24 +++
 arch/riscv/lib/crc16_msb.c          |  18 ++
 arch/riscv/lib/crc32-riscv.c        | 310 ----------------------------
 arch/riscv/lib/crc32.c              |  53 +++++
 arch/riscv/lib/crc32_lsb.c          |  18 ++
 arch/riscv/lib/crc32_msb.c          |  18 ++
 arch/riscv/lib/crc64.c              |  34 +++
 arch/riscv/lib/crc64_lsb.c          |  18 ++
 arch/riscv/lib/crc64_msb.c          |  18 ++
 scripts/gen-crc-consts.py           |  55 ++++-
 15 files changed, 672 insertions(+), 311 deletions(-)
 create mode 100644 arch/riscv/lib/crc-clmul-consts.h
 create mode 100644 arch/riscv/lib/crc-clmul-template.h
 create mode 100644 arch/riscv/lib/crc-clmul.h
 create mode 100644 arch/riscv/lib/crc-t10dif.c
 create mode 100644 arch/riscv/lib/crc16_msb.c
 delete mode 100644 arch/riscv/lib/crc32-riscv.c
 create mode 100644 arch/riscv/lib/crc32.c
 create mode 100644 arch/riscv/lib/crc32_lsb.c
 create mode 100644 arch/riscv/lib/crc32_msb.c
 create mode 100644 arch/riscv/lib/crc64.c
 create mode 100644 arch/riscv/lib/crc64_lsb.c
 create mode 100644 arch/riscv/lib/crc64_msb.c

base-commit: cf1ea3a7c1f63cba7d1dd313ee3accde0c0c8988

Signature · 1 line

2.48.1

Eric Biggers

author maintainer

2025-02-16 23:00 UTC #1 lore raw

PATCH 1/4 riscv/crc: add "template" for Zbc optimized CRC functions

From: Eric Biggers <[email protected]>

Add a "template" crc-clmul-template.h that can generate RISC-V Zbc optimized CRC functions. Each generated CRC function is parameterized by CRC length and bit order, and it accepts a pointer to the constants struct required for the specific CRC polynomial desired. Update gen-crc-consts.py to support generating the needed constants structs.

This makes it possible to easily wire up a Zbc optimized implementation of almost any CRC.

The design generally follows what I did for x86, but it is simplified by using RISC-V's scalar carryless multiplication Zbc, which has no equivalent on x86. RISC-V's clmulr instruction is also helpful. A potential switch to Zvbc (or support for Zvbc alongside Zbc) is left for future work. For long messages Zvbc should be fastest, but it would need to be shown to be worthwhile over just using Zbc which is significantly more convenient to use, especially in the kernel context.

Compared to the existing Zbc-optimized CRC32 code and the earlier proposed Zbc-optimized CRC-T10DIF code
(https://lore.kernel.org/r/[email protected]),
this submission deduplicates the code among CRC variants and is significantly more optimized. It uses "folding" to take better advantage of instruction-level parallelism (to a more limited extent than x86 for now, but it could be extended to more), it reworks the Barrett reduction to eliminate unnecessary instructions, and it documents all the math used and makes all the constants reproducible.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/riscv/lib/crc-clmul-template.h | 265 ++++++++++++++++++++++++++++
 scripts/gen-crc-consts.py           |  55 +++++-
 2 files changed, 319 insertions(+), 1 deletion(-)
 create mode 100644 arch/riscv/lib/crc-clmul-template.h

arch/riscv/lib/crc-clmul-template.h new file +265 -0

new file mode 100644
index 000000000000..77187e7f1762
@@ -0,0 +1,265 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* Copyright 2025 Google LLC */
+
+/*
+ * This file is a "template" that generates a CRC function optimized using the
+ * RISC-V Zbc (scalar carryless multiplication) extension.  The includer of this
+ * file must define the following parameters to specify the type of CRC:
+ *
+ *	crc_t: the data type of the CRC, e.g. u32 for a 32-bit CRC
+ *	LSB_CRC: 0 for a msb (most-significant-bit) first CRC, i.e. natural
+ *		 mapping between bits and polynomial coefficients
+ *	         1 for a lsb (least-significant-bit) first CRC, i.e. reflected
+ *	         mapping between bits and polynomial coefficients
+ */
+
+#include <asm/byteorder.h>
+#include <linux/minmax.h>
+
+#define CRC_BITS	(8 * sizeof(crc_t))	/* a.k.a. 'n' */
+
+static inline unsigned long clmul(unsigned long a, unsigned long b)
+{
+	unsigned long res;
+
+	asm(".option push\n"
+	    ".option arch,+zbc\n"
+	    "clmul %0, %1, %2\n"
+	    ".option pop\n"
+	    : "=r" (res) : "r" (a), "r" (b));
+	return res;
+}
+
+static inline unsigned long clmulh(unsigned long a, unsigned long b)
+{
+	unsigned long res;
+
+	asm(".option push\n"
+	    ".option arch,+zbc\n"
+	    "clmulh %0, %1, %2\n"
+	    ".option pop\n"
+	    : "=r" (res) : "r" (a), "r" (b));
+	return res;
+}
+
+static inline unsigned long clmulr(unsigned long a, unsigned long b)
+{
+	unsigned long res;
+
+	asm(".option push\n"
+	    ".option arch,+zbc\n"
+	    "clmulr %0, %1, %2\n"
+	    ".option pop\n"
+	    : "=r" (res) : "r" (a), "r" (b));
+	return res;
+}
+
+/*
+ * crc_load_long() loads one "unsigned long" of aligned data bytes, producing a
+ * polynomial whose bit order matches the CRC's bit order.
+ */
+#ifdef CONFIG_64BIT
+#  if LSB_CRC
+#    define crc_load_long(x)	le64_to_cpup(x)
+#  else
+#    define crc_load_long(x)	be64_to_cpup(x)
+#  endif
+#else
+#  if LSB_CRC
+#    define crc_load_long(x)	le32_to_cpup(x)
+#  else
+#    define crc_load_long(x)	be32_to_cpup(x)
+#  endif
+#endif
+
+/* XOR @crc into the end of @msgpoly that represents the high-order terms. */
+static inline unsigned long
+crc_clmul_prep(crc_t crc, unsigned long msgpoly)
+{
+#if LSB_CRC
+	return msgpoly ^ crc;
+#else
+	return msgpoly ^ ((unsigned long)crc << (BITS_PER_LONG - CRC_BITS));
+#endif
+}
+
+/*
+ * Multiply the long-sized @msgpoly by x^n (a.k.a. x^CRC_BITS) and reduce it
+ * modulo the generator polynomial G.  This gives the CRC of @msgpoly.
+ */
+static inline crc_t
+crc_clmul_long(unsigned long msgpoly, const struct crc_clmul_consts *consts)
+{
+	unsigned long tmp;
+
+	/*
+	 * First step of Barrett reduction with integrated multiplication by
+	 * x^n: calculate floor((msgpoly * x^n) / G).  This is the value by
+	 * which G needs to be multiplied to cancel out the x^n and higher terms
+	 * of msgpoly * x^n.  Do it using the following formula:
+	 *
+	 * msb-first:
+	 *    floor((msgpoly * floor(x^(BITS_PER_LONG-1+n) / G)) / x^(BITS_PER_LONG-1))
+	 * lsb-first:
+	 *    floor((msgpoly * floor(x^(BITS_PER_LONG-1+n) / G) * x) / x^BITS_PER_LONG)
+	 *
+	 * barrett_reduction_const_1 contains floor(x^(BITS_PER_LONG-1+n) / G),
+	 * which fits a long exactly.  Using any lower power of x there would
+	 * not carry enough precision through the calculation, while using any
+	 * higher power of x would require extra instructions to handle a wider
+	 * multiplication.  In the msb-first case, using this power of x results
+	 * in needing a floored division by x^(BITS_PER_LONG-1), which matches
+	 * what clmulr produces.  In the lsb-first case, a factor of x gets
+	 * implicitly introduced by each carryless multiplication (shown as
+	 * '* x' above), and the floored division instead needs to be by
+	 * x^BITS_PER_LONG which matches what clmul produces.
+	 */
+#if LSB_CRC
+	tmp = clmul(msgpoly, consts->barrett_reduction_const_1);
+#else
+	tmp = clmulr(msgpoly, consts->barrett_reduction_const_1);
+#endif
+
+	/*
+	 * Second step of Barrett reduction:
+	 *
+	 *    crc := (msgpoly * x^n) + (G * floor((msgpoly * x^n) / G))
+	 *
+	 * This reduces (msgpoly * x^n) modulo G by adding the appropriate
+	 * multiple of G to it.  The result uses only the x^0..x^(n-1) terms.
+	 * HOWEVER, since the unreduced value (msgpoly * x^n) is zero in those
+	 * terms in the first place, it is more efficient to do the equivalent:
+	 *
+	 *    crc := ((G - x^n) * floor((msgpoly * x^n) / G)) mod x^n
+	 *
+	 * In the lsb-first case further modify it to the following which avoids
+	 * a shift, as the crc ends up in the physically low n bits from clmulr:
+	 *
+	 *    product := ((G - x^n) * x^(BITS_PER_LONG - n)) * floor((msgpoly * x^n) / G) * x
+	 *    crc := floor(product / x^(BITS_PER_LONG + 1 - n)) mod x^n
+	 *
+	 * barrett_reduction_const_2 contains the constant multiplier (G - x^n)
+	 * or (G - x^n) * x^(BITS_PER_LONG - n) from the formulas above.  The
+	 * cast of the result to crc_t is essential, as it applies the mod x^n!
+	 */
+#if LSB_CRC
+	return clmulr(tmp, consts->barrett_reduction_const_2);
+#else
+	return clmul(tmp, consts->barrett_reduction_const_2);
+#endif
+}
+
+/* Update @crc with the data from @msgpoly. */
+static inline crc_t
+crc_clmul_update_long(crc_t crc, unsigned long msgpoly,
+		      const struct crc_clmul_consts *consts)
+{
+	return crc_clmul_long(crc_clmul_prep(crc, msgpoly), consts);
+}
+
+/* Update @crc with 1 <= @len < sizeof(unsigned long) bytes of data. */
+static inline crc_t
+crc_clmul_update_partial(crc_t crc, const u8 *p, size_t len,
+			 const struct crc_clmul_consts *consts)
+{
+	unsigned long msgpoly;
+	size_t i;
+
+#if LSB_CRC
+	msgpoly = (unsigned long)p[0] << (BITS_PER_LONG - 8);
+	for (i = 1; i < len; i++)
+		msgpoly = (msgpoly >> 8) ^ ((unsigned long)p[i] << (BITS_PER_LONG - 8));
+#else
+	msgpoly = p[0];
+	for (i = 1; i < len; i++)
+		msgpoly = (msgpoly << 8) ^ p[i];
+#endif
+
+	if (len >= sizeof(crc_t)) {
+	#if LSB_CRC
+		msgpoly ^= (unsigned long)crc << (BITS_PER_LONG - 8*len);
+	#else
+		msgpoly ^= (unsigned long)crc << (8*len - CRC_BITS);
+	#endif
+		return crc_clmul_long(msgpoly, consts);
+	}
+#if LSB_CRC
+	msgpoly ^= (unsigned long)crc << (BITS_PER_LONG - 8*len);
+	return crc_clmul_long(msgpoly, consts) ^ (crc >> (8*len));
+#else
+	msgpoly ^= crc >> (CRC_BITS - 8*len);
+	return crc_clmul_long(msgpoly, consts) ^ (crc << (8*len));
+#endif
+}
+
+static inline crc_t
+crc_clmul(crc_t crc, const void *p, size_t len,
+	  const struct crc_clmul_consts *consts)
+{
+	size_t align;
+
+	/* This implementation assumes that the CRC fits in an unsigned long. */
+	BUILD_BUG_ON(sizeof(crc_t) > sizeof(unsigned long));
+
+	/* If the buffer is not long-aligned, align it. */
+	align = (unsigned long)p % sizeof(unsigned long);
+	if (align && len) {
+		align = min(sizeof(unsigned long) - align, len);
+		crc = crc_clmul_update_partial(crc, p, align, consts);
+		p += align;
+		len -= align;
+	}
+
+	if (len >= 4 * sizeof(unsigned long)) {
+		unsigned long m0, m1;
+
+		m0 = crc_clmul_prep(crc, crc_load_long(p));
+		m1 = crc_load_long(p + sizeof(unsigned long));
+		p += 2 * sizeof(unsigned long);
+		len -= 2 * sizeof(unsigned long);
+		/*
+		 * Main loop.  Each iteration starts with a message polynomial
+		 * (x^BITS_PER_LONG)*m0 + m1, then logically extends it by two
+		 * more longs of data to form x^(3*BITS_PER_LONG)*m0 +
+		 * x^(2*BITS_PER_LONG)*m1 + x^BITS_PER_LONG*m2 + m3, then
+		 * "folds" that back into a congruent (modulo G) value that uses
+		 * just m0 and m1 again.  This is done by multiplying m0 by the
+		 * precomputed constant (x^(3*BITS_PER_LONG) mod G) and m1 by
+		 * the precomputed constant (x^(2*BITS_PER_LONG) mod G), then
+		 * adding the results to m2 and m3 as appropriate.  Each such
+		 * multiplication produces a result twice the length of a long,
+		 * which in RISC-V is two instructions clmul and clmulh.
+		 *
+		 * This could be changed to fold across more than 2 longs at a
+		 * time if there is a CPU that can take advantage of it.
+		 */
+		do {
+			unsigned long p0, p1, p2, p3;
+
+			p0 = clmulh(m0, consts->fold_across_2_longs_const_hi);
+			p1 = clmul(m0, consts->fold_across_2_longs_const_hi);
+			p2 = clmulh(m1, consts->fold_across_2_longs_const_lo);
+			p3 = clmul(m1, consts->fold_across_2_longs_const_lo);
+			m0 = (LSB_CRC ? p1 ^ p3 : p0 ^ p2) ^ crc_load_long(p);
+			m1 = (LSB_CRC ? p0 ^ p2 : p1 ^ p3) ^
+			     crc_load_long(p + sizeof(unsigned long));
+
+			p += 2 * sizeof(unsigned long);
+			len -= 2 * sizeof(unsigned long);
+		} while (len >= 2 * sizeof(unsigned long));
+
+		crc = crc_clmul_long(m0, consts);
+		crc = crc_clmul_update_long(crc, m1, consts);
+	}
+
+	while (len >= sizeof(unsigned long)) {
+		crc = crc_clmul_update_long(crc, crc_load_long(p), consts);
+		p += sizeof(unsigned long);
+		len -= sizeof(unsigned long);
+	}
+
+	if (len)
+		crc = crc_clmul_update_partial(crc, p, len, consts);
+
+	return crc;
+}

scripts/gen-crc-consts.py +54 -1

84 lines not on this page · raw message

Signature · 1 line

2.48.1

Eric Biggers

author maintainer

2025-02-16 23:00 UTC #2 lore raw

PATCH 2/4 riscv/crc32: reimplement the CRC32 functions using new template

In reply to Eric Biggers

From: Eric Biggers <[email protected]>

Delete the previous Zbc optimized CRC32 code, and re-implement it using the new template. The new implementation is more optimized and shares more code among CRC variants.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/riscv/lib/Makefile           |   1 +
 arch/riscv/lib/crc-clmul-consts.h |  72 +++++++
 arch/riscv/lib/crc-clmul.h        |  15 ++
 arch/riscv/lib/crc32-riscv.c      | 310 ------------------------------
 arch/riscv/lib/crc32.c            |  53 +++++
 arch/riscv/lib/crc32_lsb.c        |  18 ++
 arch/riscv/lib/crc32_msb.c        |  18 ++
 7 files changed, 177 insertions(+), 310 deletions(-)
 create mode 100644 arch/riscv/lib/crc-clmul-consts.h
 create mode 100644 arch/riscv/lib/crc-clmul.h
 delete mode 100644 arch/riscv/lib/crc32-riscv.c
 create mode 100644 arch/riscv/lib/crc32.c
 create mode 100644 arch/riscv/lib/crc32_lsb.c
 create mode 100644 arch/riscv/lib/crc32_msb.c

arch/riscv/lib/Makefile +1 -0

index 79368a895fee..7b32d3e88337 100644
@@ -14,8 +14,9 @@ lib-$(CONFIG_RISCV_ISA_V)	+= uaccess_vector.o
 endif
 lib-$(CONFIG_MMU)	+= uaccess.o
 lib-$(CONFIG_64BIT)	+= tishift.o
 lib-$(CONFIG_RISCV_ISA_ZICBOZ)	+= clear_page.o
 obj-$(CONFIG_CRC32_ARCH)	+= crc32-riscv.o
+crc32-riscv-y := crc32.o crc32_msb.o crc32_lsb.o
 obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
 lib-$(CONFIG_RISCV_ISA_V)	+= xor.o
 lib-$(CONFIG_RISCV_ISA_V)	+= riscv_v_helpers.o

arch/riscv/lib/crc-clmul-consts.h new file +72 -0

72 lines not on this page · raw message

arch/riscv/lib/crc-clmul.h new file +15 -0

new file mode 100644
index 000000000000..62bd41080785
@@ -0,0 +1,15 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* Copyright 2025 Google LLC */
+
+#ifndef _RISCV_CRC_CLMUL_H
+#define _RISCV_CRC_CLMUL_H
+
+#include <linux/types.h>
+#include "crc-clmul-consts.h"
+
+u32 crc32_msb_clmul(u32 crc, const void *p, size_t len,
+		    const struct crc_clmul_consts *consts);
+u32 crc32_lsb_clmul(u32 crc, const void *p, size_t len,
+		    const struct crc_clmul_consts *consts);
+
+#endif /* _RISCV_CRC_CLMUL_H */

arch/riscv/lib/crc32-riscv.c deleted +0 ~~-310~~

310 lines not on this page · raw message

arch/riscv/lib/crc32.c new file +53 -0

53 lines not on this page · raw message

arch/riscv/lib/crc32_lsb.c new file +18 -0

new file mode 100644
index 000000000000..72fd67e7470c
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * RISC-V optimized least-significant-bit-first CRC32
+ *
+ * Copyright 2025 Google LLC
+ */
+
+#include "crc-clmul.h"
+
+typedef u32 crc_t;
+#define LSB_CRC 1
+#include "crc-clmul-template.h"
+
+u32 crc32_lsb_clmul(u32 crc, const void *p, size_t len,
+		    const struct crc_clmul_consts *consts)
+{
+	return crc_clmul(crc, p, len, consts);
+}

arch/riscv/lib/crc32_msb.c new file +18 -0

new file mode 100644
index 000000000000..fdbeaccc369f
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * RISC-V optimized most-significant-bit-first CRC32
+ *
+ * Copyright 2025 Google LLC
+ */
+
+#include "crc-clmul.h"
+
+typedef u32 crc_t;
+#define LSB_CRC 0
+#include "crc-clmul-template.h"
+
+u32 crc32_msb_clmul(u32 crc, const void *p, size_t len,
+		    const struct crc_clmul_consts *consts)
+{
+	return crc_clmul(crc, p, len, consts);
+}

Signature · 1 line

2.48.1

Eric Biggers

author maintainer

2025-02-16 23:00 UTC #3 lore raw

PATCH 3/4 riscv/crc-t10dif: add Zbc optimized CRC-T10DIF function

In reply to Eric Biggers

From: Eric Biggers <[email protected]>

Wire up crc_t10dif_arch() for RISC-V using crc-clmul-template.h. This greatly improves CRC-T10DIF performance on Zbc-capable CPUs.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/riscv/Kconfig                |  1 +
 arch/riscv/lib/Makefile           |  2 ++
 arch/riscv/lib/crc-clmul-consts.h | 20 +++++++++++++++++++-
 arch/riscv/lib/crc-clmul.h        |  2 ++
 arch/riscv/lib/crc-t10dif.c       | 24 ++++++++++++++++++++++++
 arch/riscv/lib/crc16_msb.c        | 18 ++++++++++++++++++
 6 files changed, 66 insertions(+), 1 deletion(-)
 create mode 100644 arch/riscv/lib/crc-t10dif.c
 create mode 100644 arch/riscv/lib/crc16_msb.c

arch/riscv/Kconfig +1 -0

index 7612c52e9b1e..db1cf9666dfd 100644
@@ -23,10 +23,11 @@ config RISCV
 	select ARCH_ENABLE_MEMORY_HOTREMOVE if MEMORY_HOTPLUG
 	select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
 	select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
 	select ARCH_HAS_BINFMT_FLAT
 	select ARCH_HAS_CRC32 if RISCV_ISA_ZBC
+	select ARCH_HAS_CRC_T10DIF if RISCV_ISA_ZBC
 	select ARCH_HAS_CURRENT_STACK_POINTER
 	select ARCH_HAS_DEBUG_VIRTUAL if MMU
 	select ARCH_HAS_DEBUG_VM_PGTABLE
 	select ARCH_HAS_DEBUG_WX
 	select ARCH_HAS_FAST_MULTIPLIER

arch/riscv/lib/Makefile +2 -0

index 7b32d3e88337..06d9552b9c8b 100644
@@ -15,8 +15,10 @@ endif
 lib-$(CONFIG_MMU)	+= uaccess.o
 lib-$(CONFIG_64BIT)	+= tishift.o
 lib-$(CONFIG_RISCV_ISA_ZICBOZ)	+= clear_page.o
 obj-$(CONFIG_CRC32_ARCH)	+= crc32-riscv.o
 crc32-riscv-y := crc32.o crc32_msb.o crc32_lsb.o
+obj-$(CONFIG_CRC_T10DIF_ARCH)	+= crc-t10dif-riscv.o
+crc-t10dif-riscv-y := crc-t10dif.o crc16_msb.o
 obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
 lib-$(CONFIG_RISCV_ISA_V)	+= xor.o
 lib-$(CONFIG_RISCV_ISA_V)	+= riscv_v_helpers.o

arch/riscv/lib/crc-clmul-consts.h +19 -1

index 6fdf10648a20..b3a02b9096cd 100644
@@ -1,10 +1,10 @@
 /* SPDX-License-Identifier: GPL-2.0-or-later */
 /*
  * CRC constants generated by:
  *
- *	./scripts/gen-crc-consts.py riscv_clmul crc32_msb_0x04c11db7,crc32_lsb_0xedb88320,crc32_lsb_0x82f63b78
+ *	./scripts/gen-crc-consts.py riscv_clmul crc16_msb_0x8bb7,crc32_msb_0x04c11db7,crc32_lsb_0xedb88320,crc32_lsb_0x82f63b78
  *
  * Do not edit manually.
  */
 
 struct crc_clmul_consts {

@@ -12,10 +12,28 @@ struct crc_clmul_consts {
 	unsigned long fold_across_2_longs_const_lo;
 	unsigned long barrett_reduction_const_1;
 	unsigned long barrett_reduction_const_2;
 };
 
+/*
+ * Constants generated for most-significant-bit-first CRC-16 using
+ * G(x) = x^16 + x^15 + x^11 + x^9 + x^8 + x^7 + x^5 + x^4 + x^2 + x^1 + x^0
+ */
+static const struct crc_clmul_consts crc16_msb_0x8bb7_consts __maybe_unused = {
+#ifdef CONFIG_64BIT
+	.fold_across_2_longs_const_hi = 0x0000000000001faa, /* x^192 mod G */
+	.fold_across_2_longs_const_lo = 0x000000000000a010, /* x^128 mod G */
+	.barrett_reduction_const_1 = 0xfb2d2bfc0e99d245, /* floor(x^79 / G) */
+	.barrett_reduction_const_2 = 0x0000000000008bb7, /* G - x^16 */
+#else
+	.fold_across_2_longs_const_hi = 0x00005890, /* x^96 mod G */
+	.fold_across_2_longs_const_lo = 0x0000f249, /* x^64 mod G */
+	.barrett_reduction_const_1 = 0xfb2d2bfc, /* floor(x^47 / G) */
+	.barrett_reduction_const_2 = 0x00008bb7, /* G - x^16 */
+#endif
+};
+
 /*
  * Constants generated for most-significant-bit-first CRC-32 using
  * G(x) = x^32 + x^26 + x^23 + x^22 + x^16 + x^12 + x^11 + x^10 + x^8 + x^7 +
  *        x^5 + x^4 + x^2 + x^1 + x^0
  */

arch/riscv/lib/crc-clmul.h +2 -0

index 62bd41080785..162c1b12b219 100644
@@ -5,10 +5,12 @@
 #define _RISCV_CRC_CLMUL_H
 
 #include <linux/types.h>
 #include "crc-clmul-consts.h"
 
+u16 crc16_msb_clmul(u16 crc, const void *p, size_t len,
+		    const struct crc_clmul_consts *consts);
 u32 crc32_msb_clmul(u32 crc, const void *p, size_t len,
 		    const struct crc_clmul_consts *consts);
 u32 crc32_lsb_clmul(u32 crc, const void *p, size_t len,
 		    const struct crc_clmul_consts *consts);

arch/riscv/lib/crc-t10dif.c new file +24 -0

new file mode 100644
index 000000000000..e6b0051ccd86
@@ -0,0 +1,24 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * RISC-V optimized CRC-T10DIF function
+ *
+ * Copyright 2025 Google LLC
+ */
+
+#include <asm/hwcap.h>
+#include <asm/alternative-macros.h>
+#include <linux/crc-t10dif.h>
+#include <linux/module.h>
+
+#include "crc-clmul.h"
+
+u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len)
+{
+	if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC))
+		return crc16_msb_clmul(crc, p, len, &crc16_msb_0x8bb7_consts);
+	return crc_t10dif_generic(crc, p, len);
+}
+EXPORT_SYMBOL(crc_t10dif_arch);
+
+MODULE_DESCRIPTION("RISC-V optimized CRC-T10DIF function");
+MODULE_LICENSE("GPL");

arch/riscv/lib/crc16_msb.c new file +18 -0

new file mode 100644
index 000000000000..554d295e95f5
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * RISC-V optimized most-significant-bit-first CRC16
+ *
+ * Copyright 2025 Google LLC
+ */
+
+#include "crc-clmul.h"
+
+typedef u16 crc_t;
+#define LSB_CRC 0
+#include "crc-clmul-template.h"
+
+u16 crc16_msb_clmul(u16 crc, const void *p, size_t len,
+		    const struct crc_clmul_consts *consts)
+{
+	return crc_clmul(crc, p, len, consts);
+}

Signature · 1 line

2.48.1

Eric Biggers

author maintainer

2025-02-16 23:00 UTC #4 lore raw

PATCH 4/4 riscv/crc64: add Zbc optimized CRC64 functions

In reply to Eric Biggers

From: Eric Biggers <[email protected]>

Wire up crc64_be_arch() and crc64_nvme_arch() for 64-bit RISC-V using crc-clmul-template.h. This greatly improves the performance of these CRCs on Zbc-capable CPUs in 64-bit kernels.

These optimized CRC64 functions are not yet supported in 32-bit kernels, since crc-clmul-template.h assumes that the CRC fits in an unsigned long. That implementation limitation could be addressed, but it would add a fair bit of complexity, so it has been omitted for now.

Signed-off-by Eric Biggers <[email protected]>

---
 arch/riscv/Kconfig                |  1 +
 arch/riscv/lib/Makefile           |  2 ++
 arch/riscv/lib/crc-clmul-consts.h | 34 ++++++++++++++++++++++++++++++-
 arch/riscv/lib/crc-clmul.h        |  6 ++++++
 arch/riscv/lib/crc64.c            | 34 +++++++++++++++++++++++++++++++
 arch/riscv/lib/crc64_lsb.c        | 18 ++++++++++++++++
 arch/riscv/lib/crc64_msb.c        | 18 ++++++++++++++++
 7 files changed, 112 insertions(+), 1 deletion(-)
 create mode 100644 arch/riscv/lib/crc64.c
 create mode 100644 arch/riscv/lib/crc64_lsb.c
 create mode 100644 arch/riscv/lib/crc64_msb.c

arch/riscv/Kconfig +1 -0

index db1cf9666dfd..e10dda2d0bfe 100644
@@ -23,10 +23,11 @@ config RISCV
 	select ARCH_ENABLE_MEMORY_HOTREMOVE if MEMORY_HOTPLUG
 	select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
 	select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
 	select ARCH_HAS_BINFMT_FLAT
 	select ARCH_HAS_CRC32 if RISCV_ISA_ZBC
+	select ARCH_HAS_CRC64 if 64BIT && RISCV_ISA_ZBC
 	select ARCH_HAS_CRC_T10DIF if RISCV_ISA_ZBC
 	select ARCH_HAS_CURRENT_STACK_POINTER
 	select ARCH_HAS_DEBUG_VIRTUAL if MMU
 	select ARCH_HAS_DEBUG_VM_PGTABLE
 	select ARCH_HAS_DEBUG_WX

arch/riscv/lib/Makefile +2 -0

index 06d9552b9c8b..b1c46153606a 100644
@@ -15,10 +15,12 @@ endif
 lib-$(CONFIG_MMU)	+= uaccess.o
 lib-$(CONFIG_64BIT)	+= tishift.o
 lib-$(CONFIG_RISCV_ISA_ZICBOZ)	+= clear_page.o
 obj-$(CONFIG_CRC32_ARCH)	+= crc32-riscv.o
 crc32-riscv-y := crc32.o crc32_msb.o crc32_lsb.o
+obj-$(CONFIG_CRC64_ARCH) += crc64-riscv.o
+crc64-riscv-y := crc64.o crc64_msb.o crc64_lsb.o
 obj-$(CONFIG_CRC_T10DIF_ARCH)	+= crc-t10dif-riscv.o
 crc-t10dif-riscv-y := crc-t10dif.o crc16_msb.o
 obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
 lib-$(CONFIG_RISCV_ISA_V)	+= xor.o
 lib-$(CONFIG_RISCV_ISA_V)	+= riscv_v_helpers.o

arch/riscv/lib/crc-clmul-consts.h +33 -1

index b3a02b9096cd..8d73449235ef 100644
@@ -1,10 +1,10 @@
 /* SPDX-License-Identifier: GPL-2.0-or-later */
 /*
  * CRC constants generated by:
  *
- *	./scripts/gen-crc-consts.py riscv_clmul crc16_msb_0x8bb7,crc32_msb_0x04c11db7,crc32_lsb_0xedb88320,crc32_lsb_0x82f63b78
+ *	./scripts/gen-crc-consts.py riscv_clmul crc16_msb_0x8bb7,crc32_msb_0x04c11db7,crc32_lsb_0xedb88320,crc32_lsb_0x82f63b78,crc64_msb_0x42f0e1eba9ea3693,crc64_lsb_0x9a6c9329ac4bc9b5
  *
  * Do not edit manually.
  */
 
 struct crc_clmul_consts {

@@ -86,5 +86,37 @@ static const struct crc_clmul_consts crc32_lsb_0x82f63b78_consts __maybe_unused
 	.fold_across_2_longs_const_lo = 0xdd45aab8, /* x^63 mod G */
 	.barrett_reduction_const_1 = 0xdea713f1, /* floor(x^63 / G) */
 	.barrett_reduction_const_2 = 0x82f63b78, /* (G - x^32) * x^0 */
 #endif
 };
+
+/*
+ * Constants generated for most-significant-bit-first CRC-64 using
+ * G(x) = x^64 + x^62 + x^57 + x^55 + x^54 + x^53 + x^52 + x^47 + x^46 + x^45 +
+ *        x^40 + x^39 + x^38 + x^37 + x^35 + x^33 + x^32 + x^31 + x^29 + x^27 +
+ *        x^24 + x^23 + x^22 + x^21 + x^19 + x^17 + x^13 + x^12 + x^10 + x^9 +
+ *        x^7 + x^4 + x^1 + x^0
+ */
+#ifdef CONFIG_64BIT
+static const struct crc_clmul_consts crc64_msb_0x42f0e1eba9ea3693_consts __maybe_unused = {
+	.fold_across_2_longs_const_hi = 0x4eb938a7d257740e, /* x^192 mod G */
+	.fold_across_2_longs_const_lo = 0x05f5c3c7eb52fab6, /* x^128 mod G */
+	.barrett_reduction_const_1 = 0xabc694e836627c39, /* floor(x^127 / G) */
+	.barrett_reduction_const_2 = 0x42f0e1eba9ea3693, /* G - x^64 */
+};
+#endif
+
+/*
+ * Constants generated for least-significant-bit-first CRC-64 using
+ * G(x) = x^64 + x^63 + x^61 + x^59 + x^58 + x^56 + x^55 + x^52 + x^49 + x^48 +
+ *        x^47 + x^46 + x^44 + x^41 + x^37 + x^36 + x^34 + x^32 + x^31 + x^28 +
+ *        x^26 + x^23 + x^22 + x^19 + x^16 + x^13 + x^12 + x^10 + x^9 + x^6 +
+ *        x^4 + x^3 + x^0
+ */
+#ifdef CONFIG_64BIT
+static const struct crc_clmul_consts crc64_lsb_0x9a6c9329ac4bc9b5_consts __maybe_unused = {
+	.fold_across_2_longs_const_hi = 0xeadc41fd2ba3d420, /* x^191 mod G */
+	.fold_across_2_longs_const_lo = 0x21e9761e252621ac, /* x^127 mod G */
+	.barrett_reduction_const_1 = 0x27ecfa329aef9f77, /* floor(x^127 / G) */
+	.barrett_reduction_const_2 = 0x9a6c9329ac4bc9b5, /* (G - x^64) * x^0 */
+};
+#endif

arch/riscv/lib/crc-clmul.h +6 -0

index 162c1b12b219..dd1736245815 100644
@@ -11,7 +11,13 @@ u16 crc16_msb_clmul(u16 crc, const void *p, size_t len,
 		    const struct crc_clmul_consts *consts);
 u32 crc32_msb_clmul(u32 crc, const void *p, size_t len,
 		    const struct crc_clmul_consts *consts);
 u32 crc32_lsb_clmul(u32 crc, const void *p, size_t len,
 		    const struct crc_clmul_consts *consts);
+#ifdef CONFIG_64BIT
+u64 crc64_msb_clmul(u64 crc, const void *p, size_t len,
+		    const struct crc_clmul_consts *consts);
+u64 crc64_lsb_clmul(u64 crc, const void *p, size_t len,
+		    const struct crc_clmul_consts *consts);
+#endif
 
 #endif /* _RISCV_CRC_CLMUL_H */

arch/riscv/lib/crc64.c new file +34 -0

new file mode 100644
index 000000000000..f0015a27836a
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * RISC-V optimized CRC64 functions
+ *
+ * Copyright 2025 Google LLC
+ */
+
+#include <asm/hwcap.h>
+#include <asm/alternative-macros.h>
+#include <linux/crc64.h>
+#include <linux/module.h>
+
+#include "crc-clmul.h"
+
+u64 crc64_be_arch(u64 crc, const u8 *p, size_t len)
+{
+	if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC))
+		return crc64_msb_clmul(crc, p, len,
+				       &crc64_msb_0x42f0e1eba9ea3693_consts);
+	return crc64_be_generic(crc, p, len);
+}
+EXPORT_SYMBOL(crc64_be_arch);
+
+u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)
+{
+	if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC))
+		return crc64_lsb_clmul(crc, p, len,
+				       &crc64_lsb_0x9a6c9329ac4bc9b5_consts);
+	return crc64_nvme_generic(crc, p, len);
+}
+EXPORT_SYMBOL(crc64_nvme_arch);
+
+MODULE_DESCRIPTION("RISC-V optimized CRC64 functions");
+MODULE_LICENSE("GPL");

arch/riscv/lib/crc64_lsb.c new file +18 -0

new file mode 100644
index 000000000000..c5371bb85d90
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * RISC-V optimized least-significant-bit-first CRC64
+ *
+ * Copyright 2025 Google LLC
+ */
+
+#include "crc-clmul.h"
+
+typedef u64 crc_t;
+#define LSB_CRC 1
+#include "crc-clmul-template.h"
+
+u64 crc64_lsb_clmul(u64 crc, const void *p, size_t len,
+		    const struct crc_clmul_consts *consts)
+{
+	return crc_clmul(crc, p, len, consts);
+}

arch/riscv/lib/crc64_msb.c new file +18 -0

new file mode 100644
index 000000000000..1925d1dbe225
@@ -0,0 +1,18 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * RISC-V optimized most-significant-bit-first CRC64
+ *
+ * Copyright 2025 Google LLC
+ */
+
+#include "crc-clmul.h"
+
+typedef u64 crc_t;
+#define LSB_CRC 0
+#include "crc-clmul-template.h"
+
+u64 crc64_msb_clmul(u64 crc, const void *p, size_t len,
+		    const struct crc_clmul_consts *consts)
+{
+	return crc_clmul(crc, p, len, consts);
+}

Signature · 1 line

2.48.1

Eric Biggers

author maintainer

2025-02-24 18:06 UTC #5 lore raw

In reply to Eric Biggers

Quoted 17 lines · Eric Biggers

This patchset is a replacement for
"[PATCH v4] riscv: Optimize crct10dif with Zbc extension"
(https://lore.kernel.org/r/[email protected]/).
It adopts the approach that I'm taking for x86 where code is shared among CRC variants. It replaces the existing Zbc optimized CRC32 functions, then adds Zbc optimized CRC-T10DIF and CRC64 functions.
This new code should be significantly faster than the current Zbc optimized CRC32 code and the previously proposed CRC-T10DIF code. It uses "folding" instead of just Barrett reduction, and it also implements Barrett reduction more efficiently.
This applies to crc-next at
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next.
It depends on other patches that are queued there for 6.15, so I plan to take it through there if there are no objections.

Quoted 4 lines

Tested with crc_kunit in QEMU (set CONFIG_CRC_KUNIT_TEST=y and CONFIG_CRC_BENCHMARK=y), both 32-bit and 64-bit. I don't have real Zbc capable hardware to benchmark this on, but the new code should work very well; similar optimizations work very well on other architectures.

Any feedback on this series from the RISC-V side?

- Eric

Björn Töpel

Tested-by

2025-03-02 18:56 UTC #6 lore raw

Eric!

Quoted 18 lines · Eric Biggers

Quoted 17 lines · Eric Biggers
This patchset is a replacement for
"[PATCH v4] riscv: Optimize crct10dif with Zbc extension"
(https://lore.kernel.org/r/[email protected]/).
It adopts the approach that I'm taking for x86 where code is shared among CRC variants. It replaces the existing Zbc optimized CRC32 functions, then adds Zbc optimized CRC-T10DIF and CRC64 functions.
This new code should be significantly faster than the current Zbc optimized CRC32 code and the previously proposed CRC-T10DIF code. It uses "folding" instead of just Barrett reduction, and it also implements Barrett reduction more efficiently.
This applies to crc-next at
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next.
It depends on other patches that are queued there for 6.15, so I plan to take it through there if there are no objections.

Quoted 6 lines

Quoted 4 lines
Tested with crc_kunit in QEMU (set CONFIG_CRC_KUNIT_TEST=y and CONFIG_CRC_BENCHMARK=y), both 32-bit and 64-bit. I don't have real Zbc capable hardware to benchmark this on, but the new code should work very well; similar optimizations work very well on other architectures.
Any feedback on this series from the RISC-V side?

I have not reviewed your series, but I did a testrun the Milk-V Jupiter which sports a Spacemit K1 that has Zbc.

I based the run on commit 1973160c90d7 ("Merge tag 'gpio-fixes-for-v6.14-rc5' of
git://git.kernel.org/pub/scm/linux/kernel/git/brgl/linux"), plus your crc-next branch (commit a0bd462f3a13 ("x86/crc: add ANNOTATE_NOENDBR to suppress objtool warnings")) merged:

  | --- base1.txt	2025-03-02 18:31:16.169438876 +0000
  | +++ eric.txt	2025-03-02 18:35:58.683017223 +0000
  | @@ -11,7 +11,7 @@
  |      # crc16_benchmark: len=127: 153 MB/s
  |      # crc16_benchmark: len=128: 153 MB/s
  |      # crc16_benchmark: len=200: 153 MB/s
  | -    # crc16_benchmark: len=256: 153 MB/s
  | +    # crc16_benchmark: len=256: 154 MB/s
  |      # crc16_benchmark: len=511: 154 MB/s
  |      # crc16_benchmark: len=512: 154 MB/s
  |      # crc16_benchmark: len=1024: 155 MB/s
  | @@ -20,94 +20,94 @@
  |      # crc16_benchmark: len=16384: 155 MB/s
  |      ok 2 crc16_benchmark
  |      ok 3 crc_t10dif_test
  | -    # crc_t10dif_benchmark: len=1: 48 MB/s
  | -    # crc_t10dif_benchmark: len=16: 125 MB/s
  | -    # crc_t10dif_benchmark: len=64: 136 MB/s
  | -    # crc_t10dif_benchmark: len=127: 138 MB/s
  | -    # crc_t10dif_benchmark: len=128: 138 MB/s
  | -    # crc_t10dif_benchmark: len=200: 138 MB/s
  | -    # crc_t10dif_benchmark: len=256: 138 MB/s
  | -    # crc_t10dif_benchmark: len=511: 139 MB/s
  | -    # crc_t10dif_benchmark: len=512: 139 MB/s
  | -    # crc_t10dif_benchmark: len=1024: 139 MB/s
  | -    # crc_t10dif_benchmark: len=3173: 140 MB/s
  | -    # crc_t10dif_benchmark: len=4096: 140 MB/s
  | -    # crc_t10dif_benchmark: len=16384: 140 MB/s
  | +    # crc_t10dif_benchmark: len=1: 28 MB/s
  | +    # crc_t10dif_benchmark: len=16: 236 MB/s
  | +    # crc_t10dif_benchmark: len=64: 450 MB/s
  | +    # crc_t10dif_benchmark: len=127: 480 MB/s
  | +    # crc_t10dif_benchmark: len=128: 540 MB/s
  | +    # crc_t10dif_benchmark: len=200: 559 MB/s
  | +    # crc_t10dif_benchmark: len=256: 600 MB/s
  | +    # crc_t10dif_benchmark: len=511: 613 MB/s
  | +    # crc_t10dif_benchmark: len=512: 635 MB/s
  | +    # crc_t10dif_benchmark: len=1024: 654 MB/s
  | +    # crc_t10dif_benchmark: len=3173: 665 MB/s
  | +    # crc_t10dif_benchmark: len=4096: 669 MB/s
  | +    # crc_t10dif_benchmark: len=16384: 673 MB/s
  |      ok 4 crc_t10dif_benchmark
  |      ok 5 crc32_le_test
  |      # crc32_le_benchmark: len=1: 31 MB/s
  | -    # crc32_le_benchmark: len=16: 456 MB/s
  | -    # crc32_le_benchmark: len=64: 682 MB/s
  | -    # crc32_le_benchmark: len=127: 620 MB/s
  | -    # crc32_le_benchmark: len=128: 744 MB/s
  | -    # crc32_le_benchmark: len=200: 768 MB/s
  | -    # crc32_le_benchmark: len=256: 777 MB/s
  | -    # crc32_le_benchmark: len=511: 758 MB/s
  | -    # crc32_le_benchmark: len=512: 798 MB/s
  | -    # crc32_le_benchmark: len=1024: 807 MB/s
  | -    # crc32_le_benchmark: len=3173: 807 MB/s
  | -    # crc32_le_benchmark: len=4096: 814 MB/s
  | -    # crc32_le_benchmark: len=16384: 816 MB/s
  | +    # crc32_le_benchmark: len=16: 439 MB/s
  | +    # crc32_le_benchmark: len=64: 1209 MB/s
  | +    # crc32_le_benchmark: len=127: 1067 MB/s
  | +    # crc32_le_benchmark: len=128: 1616 MB/s
  | +    # crc32_le_benchmark: len=200: 1739 MB/s
  | +    # crc32_le_benchmark: len=256: 1951 MB/s
  | +    # crc32_le_benchmark: len=511: 1855 MB/s
  | +    # crc32_le_benchmark: len=512: 2174 MB/s
  | +    # crc32_le_benchmark: len=1024: 2301 MB/s
  | +    # crc32_le_benchmark: len=3173: 2347 MB/s
  | +    # crc32_le_benchmark: len=4096: 2407 MB/s
  | +    # crc32_le_benchmark: len=16384: 2440 MB/s
  |      ok 6 crc32_le_benchmark
  |      ok 7 crc32_be_test
  | -    # crc32_be_benchmark: len=1: 27 MB/s
  | -    # crc32_be_benchmark: len=16: 258 MB/s
  | -    # crc32_be_benchmark: len=64: 388 MB/s
  | -    # crc32_be_benchmark: len=127: 402 MB/s
  | -    # crc32_be_benchmark: len=128: 424 MB/s
  | -    # crc32_be_benchmark: len=200: 438 MB/s
  | -    # crc32_be_benchmark: len=256: 444 MB/s
  | -    # crc32_be_benchmark: len=511: 449 MB/s
  | -    # crc32_be_benchmark: len=512: 455 MB/s
  | -    # crc32_be_benchmark: len=1024: 461 MB/s
  | -    # crc32_be_benchmark: len=3173: 463 MB/s
  | -    # crc32_be_benchmark: len=4096: 465 MB/s
  | -    # crc32_be_benchmark: len=16384: 466 MB/s
  | +    # crc32_be_benchmark: len=1: 25 MB/s
  | +    # crc32_be_benchmark: len=16: 251 MB/s
  | +    # crc32_be_benchmark: len=64: 458 MB/s
  | +    # crc32_be_benchmark: len=127: 496 MB/s
  | +    # crc32_be_benchmark: len=128: 547 MB/s
  | +    # crc32_be_benchmark: len=200: 569 MB/s
  | +    # crc32_be_benchmark: len=256: 605 MB/s
  | +    # crc32_be_benchmark: len=511: 621 MB/s
  | +    # crc32_be_benchmark: len=512: 637 MB/s
  | +    # crc32_be_benchmark: len=1024: 657 MB/s
  | +    # crc32_be_benchmark: len=3173: 668 MB/s
  | +    # crc32_be_benchmark: len=4096: 671 MB/s
  | +    # crc32_be_benchmark: len=16384: 674 MB/s
  |      ok 8 crc32_be_benchmark
  |      ok 9 crc32c_test
  |      # crc32c_benchmark: len=1: 31 MB/s
  | -    # crc32c_benchmark: len=16: 457 MB/s
  | -    # crc32c_benchmark: len=64: 682 MB/s
  | -    # crc32c_benchmark: len=127: 620 MB/s
  | -    # crc32c_benchmark: len=128: 744 MB/s
  | -    # crc32c_benchmark: len=200: 769 MB/s
  | -    # crc32c_benchmark: len=256: 779 MB/s
  | -    # crc32c_benchmark: len=511: 758 MB/s
  | -    # crc32c_benchmark: len=512: 797 MB/s
  | -    # crc32c_benchmark: len=1024: 807 MB/s
  | -    # crc32c_benchmark: len=3173: 806 MB/s
  | -    # crc32c_benchmark: len=4096: 813 MB/s
  | -    # crc32c_benchmark: len=16384: 816 MB/s
  | +    # crc32c_benchmark: len=16: 446 MB/s
  | +    # crc32c_benchmark: len=64: 1188 MB/s
  | +    # crc32c_benchmark: len=127: 1066 MB/s
  | +    # crc32c_benchmark: len=128: 1600 MB/s
  | +    # crc32c_benchmark: len=200: 1727 MB/s
  | +    # crc32c_benchmark: len=256: 1941 MB/s
  | +    # crc32c_benchmark: len=511: 1854 MB/s
  | +    # crc32c_benchmark: len=512: 2164 MB/s
  | +    # crc32c_benchmark: len=1024: 2300 MB/s
  | +    # crc32c_benchmark: len=3173: 2345 MB/s
  | +    # crc32c_benchmark: len=4096: 2402 MB/s
  | +    # crc32c_benchmark: len=16384: 2437 MB/s
  |      ok 10 crc32c_benchmark
  |      ok 11 crc64_be_test
  | -    # crc64_be_benchmark: len=1: 64 MB/s
  | -    # crc64_be_benchmark: len=16: 144 MB/s
  | -    # crc64_be_benchmark: len=64: 154 MB/s
  | -    # crc64_be_benchmark: len=127: 156 MB/s
  | -    # crc64_be_benchmark: len=128: 156 MB/s
  | -    # crc64_be_benchmark: len=200: 156 MB/s
  | -    # crc64_be_benchmark: len=256: 156 MB/s
  | -    # crc64_be_benchmark: len=511: 157 MB/s
  | -    # crc64_be_benchmark: len=512: 157 MB/s
  | -    # crc64_be_benchmark: len=1024: 157 MB/s
  | -    # crc64_be_benchmark: len=3173: 158 MB/s
  | -    # crc64_be_benchmark: len=4096: 158 MB/s
  | -    # crc64_be_benchmark: len=16384: 158 MB/s
  | +    # crc64_be_benchmark: len=1: 29 MB/s
  | +    # crc64_be_benchmark: len=16: 264 MB/s
  | +    # crc64_be_benchmark: len=64: 476 MB/s
  | +    # crc64_be_benchmark: len=127: 499 MB/s
  | +    # crc64_be_benchmark: len=128: 558 MB/s
  | +    # crc64_be_benchmark: len=200: 576 MB/s
  | +    # crc64_be_benchmark: len=256: 611 MB/s
  | +    # crc64_be_benchmark: len=511: 621 MB/s
  | +    # crc64_be_benchmark: len=512: 638 MB/s
  | +    # crc64_be_benchmark: len=1024: 659 MB/s
  | +    # crc64_be_benchmark: len=3173: 667 MB/s
  | +    # crc64_be_benchmark: len=4096: 671 MB/s
  | +    # crc64_be_benchmark: len=16384: 674 MB/s
  |      ok 12 crc64_be_benchmark
  |      ok 13 crc64_nvme_test
  | -    # crc64_nvme_benchmark: len=1: 64 MB/s
  | -    # crc64_nvme_benchmark: len=16: 144 MB/s
  | -    # crc64_nvme_benchmark: len=64: 154 MB/s
  | -    # crc64_nvme_benchmark: len=127: 156 MB/s
  | -    # crc64_nvme_benchmark: len=128: 156 MB/s
  | -    # crc64_nvme_benchmark: len=200: 156 MB/s
  | -    # crc64_nvme_benchmark: len=256: 156 MB/s
  | -    # crc64_nvme_benchmark: len=511: 157 MB/s
  | -    # crc64_nvme_benchmark: len=512: 157 MB/s
  | -    # crc64_nvme_benchmark: len=1024: 157 MB/s
  | -    # crc64_nvme_benchmark: len=3173: 158 MB/s
  | -    # crc64_nvme_benchmark: len=4096: 158 MB/s
  | -    # crc64_nvme_benchmark: len=16384: 158 MB/s
  | +    # crc64_nvme_benchmark: len=1: 36 MB/s
  | +    # crc64_nvme_benchmark: len=16: 479 MB/s
  | +    # crc64_nvme_benchmark: len=64: 1340 MB/s
  | +    # crc64_nvme_benchmark: len=127: 1179 MB/s
  | +    # crc64_nvme_benchmark: len=128: 1766 MB/s
  | +    # crc64_nvme_benchmark: len=200: 1965 MB/s
  | +    # crc64_nvme_benchmark: len=256: 2201 MB/s
  | +    # crc64_nvme_benchmark: len=511: 2087 MB/s
  | +    # crc64_nvme_benchmark: len=512: 2464 MB/s
  | +    # crc64_nvme_benchmark: len=1024: 2331 MB/s
  | +    # crc64_nvme_benchmark: len=3173: 2673 MB/s
  | +    # crc64_nvme_benchmark: len=4096: 2745 MB/s
  | +    # crc64_nvme_benchmark: len=16384: 2782 MB/s
  |      ok 14 crc64_nvme_benchmark
  |  # crc: pass:14 fail:0 skip:0 total:14
  |  # Totals: pass:14 fail:0 skip:0 total:14

That's a significant speed up for this popular SoC, and it would be great to get this series in for the next merge window! Thank you!

Tested-by Björn Töpel <[email protected]>

Björn

Eric Biggers

author maintainer

2025-03-02 22:04 UTC #7 lore raw

Quoted 221 lines · Björn Töpel

Eric!

Quoted 18 lines · Eric Biggers

Quoted 17 lines · Eric Biggers
This patchset is a replacement for
"[PATCH v4] riscv: Optimize crct10dif with Zbc extension"
(https://lore.kernel.org/r/[email protected]/).
It adopts the approach that I'm taking for x86 where code is shared among CRC variants. It replaces the existing Zbc optimized CRC32 functions, then adds Zbc optimized CRC-T10DIF and CRC64 functions.
This new code should be significantly faster than the current Zbc optimized CRC32 code and the previously proposed CRC-T10DIF code. It uses "folding" instead of just Barrett reduction, and it also implements Barrett reduction more efficiently.
This applies to crc-next at
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next.
It depends on other patches that are queued there for 6.15, so I plan to take it through there if there are no objections.

Quoted 6 lines

Quoted 4 lines
Tested with crc_kunit in QEMU (set CONFIG_CRC_KUNIT_TEST=y and CONFIG_CRC_BENCHMARK=y), both 32-bit and 64-bit. I don't have real Zbc capable hardware to benchmark this on, but the new code should work very well; similar optimizations work very well on other architectures.
Any feedback on this series from the RISC-V side?

I have not reviewed your series, but I did a testrun the Milk-V Jupiter which sports a Spacemit K1 that has Zbc.

  | --- base1.txt	2025-03-02 18:31:16.169438876 +0000
  | +++ eric.txt	2025-03-02 18:35:58.683017223 +0000
  | @@ -11,7 +11,7 @@
  |      # crc16_benchmark: len=127: 153 MB/s
  |      # crc16_benchmark: len=128: 153 MB/s
  |      # crc16_benchmark: len=200: 153 MB/s
  | -    # crc16_benchmark: len=256: 153 MB/s
  | +    # crc16_benchmark: len=256: 154 MB/s
  |      # crc16_benchmark: len=511: 154 MB/s
  |      # crc16_benchmark: len=512: 154 MB/s
  |      # crc16_benchmark: len=1024: 155 MB/s
  | @@ -20,94 +20,94 @@
  |      # crc16_benchmark: len=16384: 155 MB/s
  |      ok 2 crc16_benchmark
  |      ok 3 crc_t10dif_test
  | -    # crc_t10dif_benchmark: len=1: 48 MB/s
  | -    # crc_t10dif_benchmark: len=16: 125 MB/s
  | -    # crc_t10dif_benchmark: len=64: 136 MB/s
  | -    # crc_t10dif_benchmark: len=127: 138 MB/s
  | -    # crc_t10dif_benchmark: len=128: 138 MB/s
  | -    # crc_t10dif_benchmark: len=200: 138 MB/s
  | -    # crc_t10dif_benchmark: len=256: 138 MB/s
  | -    # crc_t10dif_benchmark: len=511: 139 MB/s
  | -    # crc_t10dif_benchmark: len=512: 139 MB/s
  | -    # crc_t10dif_benchmark: len=1024: 139 MB/s
  | -    # crc_t10dif_benchmark: len=3173: 140 MB/s
  | -    # crc_t10dif_benchmark: len=4096: 140 MB/s
  | -    # crc_t10dif_benchmark: len=16384: 140 MB/s
  | +    # crc_t10dif_benchmark: len=1: 28 MB/s
  | +    # crc_t10dif_benchmark: len=16: 236 MB/s
  | +    # crc_t10dif_benchmark: len=64: 450 MB/s
  | +    # crc_t10dif_benchmark: len=127: 480 MB/s
  | +    # crc_t10dif_benchmark: len=128: 540 MB/s
  | +    # crc_t10dif_benchmark: len=200: 559 MB/s
  | +    # crc_t10dif_benchmark: len=256: 600 MB/s
  | +    # crc_t10dif_benchmark: len=511: 613 MB/s
  | +    # crc_t10dif_benchmark: len=512: 635 MB/s
  | +    # crc_t10dif_benchmark: len=1024: 654 MB/s
  | +    # crc_t10dif_benchmark: len=3173: 665 MB/s
  | +    # crc_t10dif_benchmark: len=4096: 669 MB/s
  | +    # crc_t10dif_benchmark: len=16384: 673 MB/s
  |      ok 4 crc_t10dif_benchmark
  |      ok 5 crc32_le_test
  |      # crc32_le_benchmark: len=1: 31 MB/s
  | -    # crc32_le_benchmark: len=16: 456 MB/s
  | -    # crc32_le_benchmark: len=64: 682 MB/s
  | -    # crc32_le_benchmark: len=127: 620 MB/s
  | -    # crc32_le_benchmark: len=128: 744 MB/s
  | -    # crc32_le_benchmark: len=200: 768 MB/s
  | -    # crc32_le_benchmark: len=256: 777 MB/s
  | -    # crc32_le_benchmark: len=511: 758 MB/s
  | -    # crc32_le_benchmark: len=512: 798 MB/s
  | -    # crc32_le_benchmark: len=1024: 807 MB/s
  | -    # crc32_le_benchmark: len=3173: 807 MB/s
  | -    # crc32_le_benchmark: len=4096: 814 MB/s
  | -    # crc32_le_benchmark: len=16384: 816 MB/s
  | +    # crc32_le_benchmark: len=16: 439 MB/s
  | +    # crc32_le_benchmark: len=64: 1209 MB/s
  | +    # crc32_le_benchmark: len=127: 1067 MB/s
  | +    # crc32_le_benchmark: len=128: 1616 MB/s
  | +    # crc32_le_benchmark: len=200: 1739 MB/s
  | +    # crc32_le_benchmark: len=256: 1951 MB/s
  | +    # crc32_le_benchmark: len=511: 1855 MB/s
  | +    # crc32_le_benchmark: len=512: 2174 MB/s
  | +    # crc32_le_benchmark: len=1024: 2301 MB/s
  | +    # crc32_le_benchmark: len=3173: 2347 MB/s
  | +    # crc32_le_benchmark: len=4096: 2407 MB/s
  | +    # crc32_le_benchmark: len=16384: 2440 MB/s
  |      ok 6 crc32_le_benchmark
  |      ok 7 crc32_be_test
  | -    # crc32_be_benchmark: len=1: 27 MB/s
  | -    # crc32_be_benchmark: len=16: 258 MB/s
  | -    # crc32_be_benchmark: len=64: 388 MB/s
  | -    # crc32_be_benchmark: len=127: 402 MB/s
  | -    # crc32_be_benchmark: len=128: 424 MB/s
  | -    # crc32_be_benchmark: len=200: 438 MB/s
  | -    # crc32_be_benchmark: len=256: 444 MB/s
  | -    # crc32_be_benchmark: len=511: 449 MB/s
  | -    # crc32_be_benchmark: len=512: 455 MB/s
  | -    # crc32_be_benchmark: len=1024: 461 MB/s
  | -    # crc32_be_benchmark: len=3173: 463 MB/s
  | -    # crc32_be_benchmark: len=4096: 465 MB/s
  | -    # crc32_be_benchmark: len=16384: 466 MB/s
  | +    # crc32_be_benchmark: len=1: 25 MB/s
  | +    # crc32_be_benchmark: len=16: 251 MB/s
  | +    # crc32_be_benchmark: len=64: 458 MB/s
  | +    # crc32_be_benchmark: len=127: 496 MB/s
  | +    # crc32_be_benchmark: len=128: 547 MB/s
  | +    # crc32_be_benchmark: len=200: 569 MB/s
  | +    # crc32_be_benchmark: len=256: 605 MB/s
  | +    # crc32_be_benchmark: len=511: 621 MB/s
  | +    # crc32_be_benchmark: len=512: 637 MB/s
  | +    # crc32_be_benchmark: len=1024: 657 MB/s
  | +    # crc32_be_benchmark: len=3173: 668 MB/s
  | +    # crc32_be_benchmark: len=4096: 671 MB/s
  | +    # crc32_be_benchmark: len=16384: 674 MB/s
  |      ok 8 crc32_be_benchmark
  |      ok 9 crc32c_test
  |      # crc32c_benchmark: len=1: 31 MB/s
  | -    # crc32c_benchmark: len=16: 457 MB/s
  | -    # crc32c_benchmark: len=64: 682 MB/s
  | -    # crc32c_benchmark: len=127: 620 MB/s
  | -    # crc32c_benchmark: len=128: 744 MB/s
  | -    # crc32c_benchmark: len=200: 769 MB/s
  | -    # crc32c_benchmark: len=256: 779 MB/s
  | -    # crc32c_benchmark: len=511: 758 MB/s
  | -    # crc32c_benchmark: len=512: 797 MB/s
  | -    # crc32c_benchmark: len=1024: 807 MB/s
  | -    # crc32c_benchmark: len=3173: 806 MB/s
  | -    # crc32c_benchmark: len=4096: 813 MB/s
  | -    # crc32c_benchmark: len=16384: 816 MB/s
  | +    # crc32c_benchmark: len=16: 446 MB/s
  | +    # crc32c_benchmark: len=64: 1188 MB/s
  | +    # crc32c_benchmark: len=127: 1066 MB/s
  | +    # crc32c_benchmark: len=128: 1600 MB/s
  | +    # crc32c_benchmark: len=200: 1727 MB/s
  | +    # crc32c_benchmark: len=256: 1941 MB/s
  | +    # crc32c_benchmark: len=511: 1854 MB/s
  | +    # crc32c_benchmark: len=512: 2164 MB/s
  | +    # crc32c_benchmark: len=1024: 2300 MB/s
  | +    # crc32c_benchmark: len=3173: 2345 MB/s
  | +    # crc32c_benchmark: len=4096: 2402 MB/s
  | +    # crc32c_benchmark: len=16384: 2437 MB/s
  |      ok 10 crc32c_benchmark
  |      ok 11 crc64_be_test
  | -    # crc64_be_benchmark: len=1: 64 MB/s
  | -    # crc64_be_benchmark: len=16: 144 MB/s
  | -    # crc64_be_benchmark: len=64: 154 MB/s
  | -    # crc64_be_benchmark: len=127: 156 MB/s
  | -    # crc64_be_benchmark: len=128: 156 MB/s
  | -    # crc64_be_benchmark: len=200: 156 MB/s
  | -    # crc64_be_benchmark: len=256: 156 MB/s
  | -    # crc64_be_benchmark: len=511: 157 MB/s
  | -    # crc64_be_benchmark: len=512: 157 MB/s
  | -    # crc64_be_benchmark: len=1024: 157 MB/s
  | -    # crc64_be_benchmark: len=3173: 158 MB/s
  | -    # crc64_be_benchmark: len=4096: 158 MB/s
  | -    # crc64_be_benchmark: len=16384: 158 MB/s
  | +    # crc64_be_benchmark: len=1: 29 MB/s
  | +    # crc64_be_benchmark: len=16: 264 MB/s
  | +    # crc64_be_benchmark: len=64: 476 MB/s
  | +    # crc64_be_benchmark: len=127: 499 MB/s
  | +    # crc64_be_benchmark: len=128: 558 MB/s
  | +    # crc64_be_benchmark: len=200: 576 MB/s
  | +    # crc64_be_benchmark: len=256: 611 MB/s
  | +    # crc64_be_benchmark: len=511: 621 MB/s
  | +    # crc64_be_benchmark: len=512: 638 MB/s
  | +    # crc64_be_benchmark: len=1024: 659 MB/s
  | +    # crc64_be_benchmark: len=3173: 667 MB/s
  | +    # crc64_be_benchmark: len=4096: 671 MB/s
  | +    # crc64_be_benchmark: len=16384: 674 MB/s
  |      ok 12 crc64_be_benchmark
  |      ok 13 crc64_nvme_test
  | -    # crc64_nvme_benchmark: len=1: 64 MB/s
  | -    # crc64_nvme_benchmark: len=16: 144 MB/s
  | -    # crc64_nvme_benchmark: len=64: 154 MB/s
  | -    # crc64_nvme_benchmark: len=127: 156 MB/s
  | -    # crc64_nvme_benchmark: len=128: 156 MB/s
  | -    # crc64_nvme_benchmark: len=200: 156 MB/s
  | -    # crc64_nvme_benchmark: len=256: 156 MB/s
  | -    # crc64_nvme_benchmark: len=511: 157 MB/s
  | -    # crc64_nvme_benchmark: len=512: 157 MB/s
  | -    # crc64_nvme_benchmark: len=1024: 157 MB/s
  | -    # crc64_nvme_benchmark: len=3173: 158 MB/s
  | -    # crc64_nvme_benchmark: len=4096: 158 MB/s
  | -    # crc64_nvme_benchmark: len=16384: 158 MB/s
  | +    # crc64_nvme_benchmark: len=1: 36 MB/s
  | +    # crc64_nvme_benchmark: len=16: 479 MB/s
  | +    # crc64_nvme_benchmark: len=64: 1340 MB/s
  | +    # crc64_nvme_benchmark: len=127: 1179 MB/s
  | +    # crc64_nvme_benchmark: len=128: 1766 MB/s
  | +    # crc64_nvme_benchmark: len=200: 1965 MB/s
  | +    # crc64_nvme_benchmark: len=256: 2201 MB/s
  | +    # crc64_nvme_benchmark: len=511: 2087 MB/s
  | +    # crc64_nvme_benchmark: len=512: 2464 MB/s
  | +    # crc64_nvme_benchmark: len=1024: 2331 MB/s
  | +    # crc64_nvme_benchmark: len=3173: 2673 MB/s
  | +    # crc64_nvme_benchmark: len=4096: 2745 MB/s
  | +    # crc64_nvme_benchmark: len=16384: 2782 MB/s
  |      ok 14 crc64_nvme_benchmark
  |  # crc: pass:14 fail:0 skip:0 total:14
  |  # Totals: pass:14 fail:0 skip:0 total:14

Quoted 4 lines

That's a significant speed up for this popular SoC, and it would be great to get this series in for the next merge window! Thank you!
Tested-by Björn Töpel <[email protected]>

Thanks for testing this patchset! So to summarize, on long messages the results were roughly:

lsb-first CRCs (crc32_le, crc32c, crc64_nvme):
Generic table-based code: 158 MB/s
Old Zbc-optimized code (crc32* only): 816 MB/s
New Zbc-optimized code: 2440 MB/s

mst-first CRCs (crc_t10dif, crc32_be, crc64_be):
Generic table-based code: 158 MB/s
Old Zbc-optimized code (crc32* only): 466 MB/s
New Zbc-optimized code: 674 MB/s

So, quite positive results. Though, the fact the msb-first CRCs are (still) so much slower than lsb-first ones indicates that be64_to_cpu() is super slow on RISC-V. That seems to be caused by the rev8 instruction from Zbb not being used. I wonder if there are any plans to make the endianness swap macros use rev8, or if I'm going to have to roll my own endianness swap in the CRC code. (I assume it would be fine for the CRC code to depend on both Zbb and Zbc.)

Anyway, I've applied this series to the crc tree
(https://web.git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next).

Palmer, I'd appreciate your ack though!

- Eric

Ignacio Encinas Rubio

2025-03-08 12:58 UTC #8 lore raw

Hello!

Quoted 6 lines · Eric Biggers

So, quite positive results. Though, the fact the msb-first CRCs are (still) so much slower than lsb-first ones indicates that be64_to_cpu() is super slow on RISC-V. That seems to be caused by the rev8 instruction from Zbb not being used. I wonder if there are any plans to make the endianness swap macros use rev8, or if I'm going to have to roll my own endianness swap in the CRC code. (I assume it would be fine for the CRC code to depend on both Zbb and Zbc.)

I saw this message the other day and started working on a patch, but I would like to double-check I'm on the right track:

- be64_to_cpu ends up being __swab64 (include/uapi/linux/swab.h)

If Zbb was part of the base ISA, turning CONFIG_ARCH_USE_BUILTIN_BSWAP would take care of the problem, but it is not the case.

Therefore, we have to define __arch_swab<X> like some "arches"(?) do in arch/<ARCH>/include/uapi/asm/swab.h

For those functions to be correct in generic kernels, we would need to use ALTERNATIVE() macros like in arch/riscv/include/asm/bitops.h. Would this be ok? I'm not sure if the ALTERNATIVEs overhead can be a problem here.

Thanks in advance :)

Alexandre Ghiti

reviewer

2025-03-10 12:34 UTC #9 lore raw

Hi Ignacio,

Quoted 17 lines · Ignacio Encinas Rubio

Hello!
Quoted 6 lines · Eric Biggers
So, quite positive results. Though, the fact the msb-first CRCs are (still) so much slower than lsb-first ones indicates that be64_to_cpu() is super slow on RISC-V. That seems to be caused by the rev8 instruction from Zbb not being used. I wonder if there are any plans to make the endianness swap macros use rev8, or if I'm going to have to roll my own endianness swap in the CRC code. (I assume it would be fine for the CRC code to depend on both Zbb and Zbc.)
I saw this message the other day and started working on a patch, but I would like to double-check I'm on the right track:
- be64_to_cpu ends up being __swab64 (include/uapi/linux/swab.h)
If Zbb was part of the base ISA, turning CONFIG_ARCH_USE_BUILTIN_BSWAP would take care of the problem, but it is not the case.

Quoted 7 lines

Therefore, we have to define __arch_swab<X> like some "arches"(?) do in arch/<ARCH>/include/uapi/asm/swab.h
For those functions to be correct in generic kernels, we would need to use ALTERNATIVE() macros like in arch/riscv/include/asm/bitops.h. Would this be ok? I'm not sure if the ALTERNATIVEs overhead can be a problem here.

Yes, using alternatives here is the right way to go. And the only overhead when Zbb is available would be a nop (take a look at lib/csum.c).

Thanks for working on this, looking forward to your patch,

Alex

Quoted 6 lines

Thanks in advance :)
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

Alexandre Ghiti

reviewer Acked-by

2025-03-10 12:44 UTC #10 lore raw

In reply to Eric Biggers

Hi Eric,

Quoted 244 lines · Eric Biggers

Quoted 219 lines · Björn Töpel

Eric!

Quoted 18 lines · Eric Biggers

Quoted 17 lines · Eric Biggers
This patchset is a replacement for
"[PATCH v4] riscv: Optimize crct10dif with Zbc extension"
(https://lore.kernel.org/r/[email protected]/).
It adopts the approach that I'm taking for x86 where code is shared among CRC variants. It replaces the existing Zbc optimized CRC32 functions, then adds Zbc optimized CRC-T10DIF and CRC64 functions.
This new code should be significantly faster than the current Zbc optimized CRC32 code and the previously proposed CRC-T10DIF code. It uses "folding" instead of just Barrett reduction, and it also implements Barrett reduction more efficiently.
This applies to crc-next at
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next.
It depends on other patches that are queued there for 6.15, so I plan to take it through there if there are no objections.

Quoted 5 lines

Quoted 4 lines
Tested with crc_kunit in QEMU (set CONFIG_CRC_KUNIT_TEST=y and CONFIG_CRC_BENCHMARK=y), both 32-bit and 64-bit. I don't have real Zbc capable hardware to benchmark this on, but the new code should work very well; similar optimizations work very well on other architectures.
Any feedback on this series from the RISC-V side?

I have not reviewed your series, but I did a testrun the Milk-V Jupiter which sports a Spacemit K1 that has Zbc.

   | --- base1.txt	2025-03-02 18:31:16.169438876 +0000
   | +++ eric.txt	2025-03-02 18:35:58.683017223 +0000
   | @@ -11,7 +11,7 @@
   |      # crc16_benchmark: len=127: 153 MB/s
   |      # crc16_benchmark: len=128: 153 MB/s
   |      # crc16_benchmark: len=200: 153 MB/s
   | -    # crc16_benchmark: len=256: 153 MB/s
   | +    # crc16_benchmark: len=256: 154 MB/s
   |      # crc16_benchmark: len=511: 154 MB/s
   |      # crc16_benchmark: len=512: 154 MB/s
   |      # crc16_benchmark: len=1024: 155 MB/s
   | @@ -20,94 +20,94 @@
   |      # crc16_benchmark: len=16384: 155 MB/s
   |      ok 2 crc16_benchmark
   |      ok 3 crc_t10dif_test
   | -    # crc_t10dif_benchmark: len=1: 48 MB/s
   | -    # crc_t10dif_benchmark: len=16: 125 MB/s
   | -    # crc_t10dif_benchmark: len=64: 136 MB/s
   | -    # crc_t10dif_benchmark: len=127: 138 MB/s
   | -    # crc_t10dif_benchmark: len=128: 138 MB/s
   | -    # crc_t10dif_benchmark: len=200: 138 MB/s
   | -    # crc_t10dif_benchmark: len=256: 138 MB/s
   | -    # crc_t10dif_benchmark: len=511: 139 MB/s
   | -    # crc_t10dif_benchmark: len=512: 139 MB/s
   | -    # crc_t10dif_benchmark: len=1024: 139 MB/s
   | -    # crc_t10dif_benchmark: len=3173: 140 MB/s
   | -    # crc_t10dif_benchmark: len=4096: 140 MB/s
   | -    # crc_t10dif_benchmark: len=16384: 140 MB/s
   | +    # crc_t10dif_benchmark: len=1: 28 MB/s
   | +    # crc_t10dif_benchmark: len=16: 236 MB/s
   | +    # crc_t10dif_benchmark: len=64: 450 MB/s
   | +    # crc_t10dif_benchmark: len=127: 480 MB/s
   | +    # crc_t10dif_benchmark: len=128: 540 MB/s
   | +    # crc_t10dif_benchmark: len=200: 559 MB/s
   | +    # crc_t10dif_benchmark: len=256: 600 MB/s
   | +    # crc_t10dif_benchmark: len=511: 613 MB/s
   | +    # crc_t10dif_benchmark: len=512: 635 MB/s
   | +    # crc_t10dif_benchmark: len=1024: 654 MB/s
   | +    # crc_t10dif_benchmark: len=3173: 665 MB/s
   | +    # crc_t10dif_benchmark: len=4096: 669 MB/s
   | +    # crc_t10dif_benchmark: len=16384: 673 MB/s
   |      ok 4 crc_t10dif_benchmark
   |      ok 5 crc32_le_test
   |      # crc32_le_benchmark: len=1: 31 MB/s
   | -    # crc32_le_benchmark: len=16: 456 MB/s
   | -    # crc32_le_benchmark: len=64: 682 MB/s
   | -    # crc32_le_benchmark: len=127: 620 MB/s
   | -    # crc32_le_benchmark: len=128: 744 MB/s
   | -    # crc32_le_benchmark: len=200: 768 MB/s
   | -    # crc32_le_benchmark: len=256: 777 MB/s
   | -    # crc32_le_benchmark: len=511: 758 MB/s
   | -    # crc32_le_benchmark: len=512: 798 MB/s
   | -    # crc32_le_benchmark: len=1024: 807 MB/s
   | -    # crc32_le_benchmark: len=3173: 807 MB/s
   | -    # crc32_le_benchmark: len=4096: 814 MB/s
   | -    # crc32_le_benchmark: len=16384: 816 MB/s
   | +    # crc32_le_benchmark: len=16: 439 MB/s
   | +    # crc32_le_benchmark: len=64: 1209 MB/s
   | +    # crc32_le_benchmark: len=127: 1067 MB/s
   | +    # crc32_le_benchmark: len=128: 1616 MB/s
   | +    # crc32_le_benchmark: len=200: 1739 MB/s
   | +    # crc32_le_benchmark: len=256: 1951 MB/s
   | +    # crc32_le_benchmark: len=511: 1855 MB/s
   | +    # crc32_le_benchmark: len=512: 2174 MB/s
   | +    # crc32_le_benchmark: len=1024: 2301 MB/s
   | +    # crc32_le_benchmark: len=3173: 2347 MB/s
   | +    # crc32_le_benchmark: len=4096: 2407 MB/s
   | +    # crc32_le_benchmark: len=16384: 2440 MB/s
   |      ok 6 crc32_le_benchmark
   |      ok 7 crc32_be_test
   | -    # crc32_be_benchmark: len=1: 27 MB/s
   | -    # crc32_be_benchmark: len=16: 258 MB/s
   | -    # crc32_be_benchmark: len=64: 388 MB/s
   | -    # crc32_be_benchmark: len=127: 402 MB/s
   | -    # crc32_be_benchmark: len=128: 424 MB/s
   | -    # crc32_be_benchmark: len=200: 438 MB/s
   | -    # crc32_be_benchmark: len=256: 444 MB/s
   | -    # crc32_be_benchmark: len=511: 449 MB/s
   | -    # crc32_be_benchmark: len=512: 455 MB/s
   | -    # crc32_be_benchmark: len=1024: 461 MB/s
   | -    # crc32_be_benchmark: len=3173: 463 MB/s
   | -    # crc32_be_benchmark: len=4096: 465 MB/s
   | -    # crc32_be_benchmark: len=16384: 466 MB/s
   | +    # crc32_be_benchmark: len=1: 25 MB/s
   | +    # crc32_be_benchmark: len=16: 251 MB/s
   | +    # crc32_be_benchmark: len=64: 458 MB/s
   | +    # crc32_be_benchmark: len=127: 496 MB/s
   | +    # crc32_be_benchmark: len=128: 547 MB/s
   | +    # crc32_be_benchmark: len=200: 569 MB/s
   | +    # crc32_be_benchmark: len=256: 605 MB/s
   | +    # crc32_be_benchmark: len=511: 621 MB/s
   | +    # crc32_be_benchmark: len=512: 637 MB/s
   | +    # crc32_be_benchmark: len=1024: 657 MB/s
   | +    # crc32_be_benchmark: len=3173: 668 MB/s
   | +    # crc32_be_benchmark: len=4096: 671 MB/s
   | +    # crc32_be_benchmark: len=16384: 674 MB/s
   |      ok 8 crc32_be_benchmark
   |      ok 9 crc32c_test
   |      # crc32c_benchmark: len=1: 31 MB/s
   | -    # crc32c_benchmark: len=16: 457 MB/s
   | -    # crc32c_benchmark: len=64: 682 MB/s
   | -    # crc32c_benchmark: len=127: 620 MB/s
   | -    # crc32c_benchmark: len=128: 744 MB/s
   | -    # crc32c_benchmark: len=200: 769 MB/s
   | -    # crc32c_benchmark: len=256: 779 MB/s
   | -    # crc32c_benchmark: len=511: 758 MB/s
   | -    # crc32c_benchmark: len=512: 797 MB/s
   | -    # crc32c_benchmark: len=1024: 807 MB/s
   | -    # crc32c_benchmark: len=3173: 806 MB/s
   | -    # crc32c_benchmark: len=4096: 813 MB/s
   | -    # crc32c_benchmark: len=16384: 816 MB/s
   | +    # crc32c_benchmark: len=16: 446 MB/s
   | +    # crc32c_benchmark: len=64: 1188 MB/s
   | +    # crc32c_benchmark: len=127: 1066 MB/s
   | +    # crc32c_benchmark: len=128: 1600 MB/s
   | +    # crc32c_benchmark: len=200: 1727 MB/s
   | +    # crc32c_benchmark: len=256: 1941 MB/s
   | +    # crc32c_benchmark: len=511: 1854 MB/s
   | +    # crc32c_benchmark: len=512: 2164 MB/s
   | +    # crc32c_benchmark: len=1024: 2300 MB/s
   | +    # crc32c_benchmark: len=3173: 2345 MB/s
   | +    # crc32c_benchmark: len=4096: 2402 MB/s
   | +    # crc32c_benchmark: len=16384: 2437 MB/s
   |      ok 10 crc32c_benchmark
   |      ok 11 crc64_be_test
   | -    # crc64_be_benchmark: len=1: 64 MB/s
   | -    # crc64_be_benchmark: len=16: 144 MB/s
   | -    # crc64_be_benchmark: len=64: 154 MB/s
   | -    # crc64_be_benchmark: len=127: 156 MB/s
   | -    # crc64_be_benchmark: len=128: 156 MB/s
   | -    # crc64_be_benchmark: len=200: 156 MB/s
   | -    # crc64_be_benchmark: len=256: 156 MB/s
   | -    # crc64_be_benchmark: len=511: 157 MB/s
   | -    # crc64_be_benchmark: len=512: 157 MB/s
   | -    # crc64_be_benchmark: len=1024: 157 MB/s
   | -    # crc64_be_benchmark: len=3173: 158 MB/s
   | -    # crc64_be_benchmark: len=4096: 158 MB/s
   | -    # crc64_be_benchmark: len=16384: 158 MB/s
   | +    # crc64_be_benchmark: len=1: 29 MB/s
   | +    # crc64_be_benchmark: len=16: 264 MB/s
   | +    # crc64_be_benchmark: len=64: 476 MB/s
   | +    # crc64_be_benchmark: len=127: 499 MB/s
   | +    # crc64_be_benchmark: len=128: 558 MB/s
   | +    # crc64_be_benchmark: len=200: 576 MB/s
   | +    # crc64_be_benchmark: len=256: 611 MB/s
   | +    # crc64_be_benchmark: len=511: 621 MB/s
   | +    # crc64_be_benchmark: len=512: 638 MB/s
   | +    # crc64_be_benchmark: len=1024: 659 MB/s
   | +    # crc64_be_benchmark: len=3173: 667 MB/s
   | +    # crc64_be_benchmark: len=4096: 671 MB/s
   | +    # crc64_be_benchmark: len=16384: 674 MB/s
   |      ok 12 crc64_be_benchmark
   |      ok 13 crc64_nvme_test
   | -    # crc64_nvme_benchmark: len=1: 64 MB/s
   | -    # crc64_nvme_benchmark: len=16: 144 MB/s
   | -    # crc64_nvme_benchmark: len=64: 154 MB/s
   | -    # crc64_nvme_benchmark: len=127: 156 MB/s
   | -    # crc64_nvme_benchmark: len=128: 156 MB/s
   | -    # crc64_nvme_benchmark: len=200: 156 MB/s
   | -    # crc64_nvme_benchmark: len=256: 156 MB/s
   | -    # crc64_nvme_benchmark: len=511: 157 MB/s
   | -    # crc64_nvme_benchmark: len=512: 157 MB/s
   | -    # crc64_nvme_benchmark: len=1024: 157 MB/s
   | -    # crc64_nvme_benchmark: len=3173: 158 MB/s
   | -    # crc64_nvme_benchmark: len=4096: 158 MB/s
   | -    # crc64_nvme_benchmark: len=16384: 158 MB/s
   | +    # crc64_nvme_benchmark: len=1: 36 MB/s
   | +    # crc64_nvme_benchmark: len=16: 479 MB/s
   | +    # crc64_nvme_benchmark: len=64: 1340 MB/s
   | +    # crc64_nvme_benchmark: len=127: 1179 MB/s
   | +    # crc64_nvme_benchmark: len=128: 1766 MB/s
   | +    # crc64_nvme_benchmark: len=200: 1965 MB/s
   | +    # crc64_nvme_benchmark: len=256: 2201 MB/s
   | +    # crc64_nvme_benchmark: len=511: 2087 MB/s
   | +    # crc64_nvme_benchmark: len=512: 2464 MB/s
   | +    # crc64_nvme_benchmark: len=1024: 2331 MB/s
   | +    # crc64_nvme_benchmark: len=3173: 2673 MB/s
   | +    # crc64_nvme_benchmark: len=4096: 2745 MB/s
   | +    # crc64_nvme_benchmark: len=16384: 2782 MB/s
   |      ok 14 crc64_nvme_benchmark
   |  # crc: pass:14 fail:0 skip:0 total:14
   |  # Totals: pass:14 fail:0 skip:0 total:14

Quoted 4 lines

That's a significant speed up for this popular SoC, and it would be great to get this series in for the next merge window! Thank you!
Tested-by Björn Töpel <[email protected]>

Thanks for testing this patchset! So to summarize, on long messages the results were roughly:

lsb-first CRCs (crc32_le, crc32c, crc64_nvme):
Generic table-based code: 158 MB/s
Old Zbc-optimized code (crc32* only): 816 MB/s
New Zbc-optimized code: 2440 MB/s

mst-first CRCs (crc_t10dif, crc32_be, crc64_be):
Generic table-based code: 158 MB/s
Old Zbc-optimized code (crc32* only): 466 MB/s
New Zbc-optimized code: 674 MB/s

Quoted 4 lines

Anyway, I've applied this series to the crc tree
(https://web.git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next).
Palmer, I'd appreciate your ack though!

Very hard to review but given Bjorn's tests on a Zbc platform, you can add:

Acked-by Alexandre Ghiti <[email protected]>

Thank you very much for working on this and unveiling the issue with the beXX_to_cpu() macros!

Alex

Quoted 7 lines

- Eric
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

Zhihang Shao

2025-03-03 06:53 UTC #11 lore raw

In reply to Björn Töpel

Quoted 219 lines · Björn Töpel

Eric!

Quoted 18 lines · Eric Biggers

Quoted 17 lines · Eric Biggers
This patchset is a replacement for
"[PATCH v4] riscv: Optimize crct10dif with Zbc extension"
(https://lore.kernel.org/r/[email protected]/).
It adopts the approach that I'm taking for x86 where code is shared among CRC variants. It replaces the existing Zbc optimized CRC32 functions, then adds Zbc optimized CRC-T10DIF and CRC64 functions.
This new code should be significantly faster than the current Zbc optimized CRC32 code and the previously proposed CRC-T10DIF code. It uses "folding" instead of just Barrett reduction, and it also implements Barrett reduction more efficiently.
This applies to crc-next at
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next.
It depends on other patches that are queued there for 6.15, so I plan to take it through there if there are no objections.

Quoted 5 lines

Quoted 4 lines
Tested with crc_kunit in QEMU (set CONFIG_CRC_KUNIT_TEST=y and CONFIG_CRC_BENCHMARK=y), both 32-bit and 64-bit. I don't have real Zbc capable hardware to benchmark this on, but the new code should work very well; similar optimizations work very well on other architectures.
Any feedback on this series from the RISC-V side?

I have not reviewed your series, but I did a testrun the Milk-V Jupiter which sports a Spacemit K1 that has Zbc.

   | --- base1.txt	2025-03-02 18:31:16.169438876 +0000
   | +++ eric.txt	2025-03-02 18:35:58.683017223 +0000
   | @@ -11,7 +11,7 @@
   |      # crc16_benchmark: len=127: 153 MB/s
   |      # crc16_benchmark: len=128: 153 MB/s
   |      # crc16_benchmark: len=200: 153 MB/s
   | -    # crc16_benchmark: len=256: 153 MB/s
   | +    # crc16_benchmark: len=256: 154 MB/s
   |      # crc16_benchmark: len=511: 154 MB/s
   |      # crc16_benchmark: len=512: 154 MB/s
   |      # crc16_benchmark: len=1024: 155 MB/s
   | @@ -20,94 +20,94 @@
   |      # crc16_benchmark: len=16384: 155 MB/s
   |      ok 2 crc16_benchmark
   |      ok 3 crc_t10dif_test
   | -    # crc_t10dif_benchmark: len=1: 48 MB/s
   | -    # crc_t10dif_benchmark: len=16: 125 MB/s
   | -    # crc_t10dif_benchmark: len=64: 136 MB/s
   | -    # crc_t10dif_benchmark: len=127: 138 MB/s
   | -    # crc_t10dif_benchmark: len=128: 138 MB/s
   | -    # crc_t10dif_benchmark: len=200: 138 MB/s
   | -    # crc_t10dif_benchmark: len=256: 138 MB/s
   | -    # crc_t10dif_benchmark: len=511: 139 MB/s
   | -    # crc_t10dif_benchmark: len=512: 139 MB/s
   | -    # crc_t10dif_benchmark: len=1024: 139 MB/s
   | -    # crc_t10dif_benchmark: len=3173: 140 MB/s
   | -    # crc_t10dif_benchmark: len=4096: 140 MB/s
   | -    # crc_t10dif_benchmark: len=16384: 140 MB/s
   | +    # crc_t10dif_benchmark: len=1: 28 MB/s
   | +    # crc_t10dif_benchmark: len=16: 236 MB/s
   | +    # crc_t10dif_benchmark: len=64: 450 MB/s
   | +    # crc_t10dif_benchmark: len=127: 480 MB/s
   | +    # crc_t10dif_benchmark: len=128: 540 MB/s
   | +    # crc_t10dif_benchmark: len=200: 559 MB/s
   | +    # crc_t10dif_benchmark: len=256: 600 MB/s
   | +    # crc_t10dif_benchmark: len=511: 613 MB/s
   | +    # crc_t10dif_benchmark: len=512: 635 MB/s
   | +    # crc_t10dif_benchmark: len=1024: 654 MB/s
   | +    # crc_t10dif_benchmark: len=3173: 665 MB/s
   | +    # crc_t10dif_benchmark: len=4096: 669 MB/s
   | +    # crc_t10dif_benchmark: len=16384: 673 MB/s
   |      ok 4 crc_t10dif_benchmark
   |      ok 5 crc32_le_test
   |      # crc32_le_benchmark: len=1: 31 MB/s
   | -    # crc32_le_benchmark: len=16: 456 MB/s
   | -    # crc32_le_benchmark: len=64: 682 MB/s
   | -    # crc32_le_benchmark: len=127: 620 MB/s
   | -    # crc32_le_benchmark: len=128: 744 MB/s
   | -    # crc32_le_benchmark: len=200: 768 MB/s
   | -    # crc32_le_benchmark: len=256: 777 MB/s
   | -    # crc32_le_benchmark: len=511: 758 MB/s
   | -    # crc32_le_benchmark: len=512: 798 MB/s
   | -    # crc32_le_benchmark: len=1024: 807 MB/s
   | -    # crc32_le_benchmark: len=3173: 807 MB/s
   | -    # crc32_le_benchmark: len=4096: 814 MB/s
   | -    # crc32_le_benchmark: len=16384: 816 MB/s
   | +    # crc32_le_benchmark: len=16: 439 MB/s
   | +    # crc32_le_benchmark: len=64: 1209 MB/s
   | +    # crc32_le_benchmark: len=127: 1067 MB/s
   | +    # crc32_le_benchmark: len=128: 1616 MB/s
   | +    # crc32_le_benchmark: len=200: 1739 MB/s
   | +    # crc32_le_benchmark: len=256: 1951 MB/s
   | +    # crc32_le_benchmark: len=511: 1855 MB/s
   | +    # crc32_le_benchmark: len=512: 2174 MB/s
   | +    # crc32_le_benchmark: len=1024: 2301 MB/s
   | +    # crc32_le_benchmark: len=3173: 2347 MB/s
   | +    # crc32_le_benchmark: len=4096: 2407 MB/s
   | +    # crc32_le_benchmark: len=16384: 2440 MB/s
   |      ok 6 crc32_le_benchmark
   |      ok 7 crc32_be_test
   | -    # crc32_be_benchmark: len=1: 27 MB/s
   | -    # crc32_be_benchmark: len=16: 258 MB/s
   | -    # crc32_be_benchmark: len=64: 388 MB/s
   | -    # crc32_be_benchmark: len=127: 402 MB/s
   | -    # crc32_be_benchmark: len=128: 424 MB/s
   | -    # crc32_be_benchmark: len=200: 438 MB/s
   | -    # crc32_be_benchmark: len=256: 444 MB/s
   | -    # crc32_be_benchmark: len=511: 449 MB/s
   | -    # crc32_be_benchmark: len=512: 455 MB/s
   | -    # crc32_be_benchmark: len=1024: 461 MB/s
   | -    # crc32_be_benchmark: len=3173: 463 MB/s
   | -    # crc32_be_benchmark: len=4096: 465 MB/s
   | -    # crc32_be_benchmark: len=16384: 466 MB/s
   | +    # crc32_be_benchmark: len=1: 25 MB/s
   | +    # crc32_be_benchmark: len=16: 251 MB/s
   | +    # crc32_be_benchmark: len=64: 458 MB/s
   | +    # crc32_be_benchmark: len=127: 496 MB/s
   | +    # crc32_be_benchmark: len=128: 547 MB/s
   | +    # crc32_be_benchmark: len=200: 569 MB/s
   | +    # crc32_be_benchmark: len=256: 605 MB/s
   | +    # crc32_be_benchmark: len=511: 621 MB/s
   | +    # crc32_be_benchmark: len=512: 637 MB/s
   | +    # crc32_be_benchmark: len=1024: 657 MB/s
   | +    # crc32_be_benchmark: len=3173: 668 MB/s
   | +    # crc32_be_benchmark: len=4096: 671 MB/s
   | +    # crc32_be_benchmark: len=16384: 674 MB/s
   |      ok 8 crc32_be_benchmark
   |      ok 9 crc32c_test
   |      # crc32c_benchmark: len=1: 31 MB/s
   | -    # crc32c_benchmark: len=16: 457 MB/s
   | -    # crc32c_benchmark: len=64: 682 MB/s
   | -    # crc32c_benchmark: len=127: 620 MB/s
   | -    # crc32c_benchmark: len=128: 744 MB/s
   | -    # crc32c_benchmark: len=200: 769 MB/s
   | -    # crc32c_benchmark: len=256: 779 MB/s
   | -    # crc32c_benchmark: len=511: 758 MB/s
   | -    # crc32c_benchmark: len=512: 797 MB/s
   | -    # crc32c_benchmark: len=1024: 807 MB/s
   | -    # crc32c_benchmark: len=3173: 806 MB/s
   | -    # crc32c_benchmark: len=4096: 813 MB/s
   | -    # crc32c_benchmark: len=16384: 816 MB/s
   | +    # crc32c_benchmark: len=16: 446 MB/s
   | +    # crc32c_benchmark: len=64: 1188 MB/s
   | +    # crc32c_benchmark: len=127: 1066 MB/s
   | +    # crc32c_benchmark: len=128: 1600 MB/s
   | +    # crc32c_benchmark: len=200: 1727 MB/s
   | +    # crc32c_benchmark: len=256: 1941 MB/s
   | +    # crc32c_benchmark: len=511: 1854 MB/s
   | +    # crc32c_benchmark: len=512: 2164 MB/s
   | +    # crc32c_benchmark: len=1024: 2300 MB/s
   | +    # crc32c_benchmark: len=3173: 2345 MB/s
   | +    # crc32c_benchmark: len=4096: 2402 MB/s
   | +    # crc32c_benchmark: len=16384: 2437 MB/s
   |      ok 10 crc32c_benchmark
   |      ok 11 crc64_be_test
   | -    # crc64_be_benchmark: len=1: 64 MB/s
   | -    # crc64_be_benchmark: len=16: 144 MB/s
   | -    # crc64_be_benchmark: len=64: 154 MB/s
   | -    # crc64_be_benchmark: len=127: 156 MB/s
   | -    # crc64_be_benchmark: len=128: 156 MB/s
   | -    # crc64_be_benchmark: len=200: 156 MB/s
   | -    # crc64_be_benchmark: len=256: 156 MB/s
   | -    # crc64_be_benchmark: len=511: 157 MB/s
   | -    # crc64_be_benchmark: len=512: 157 MB/s
   | -    # crc64_be_benchmark: len=1024: 157 MB/s
   | -    # crc64_be_benchmark: len=3173: 158 MB/s
   | -    # crc64_be_benchmark: len=4096: 158 MB/s
   | -    # crc64_be_benchmark: len=16384: 158 MB/s
   | +    # crc64_be_benchmark: len=1: 29 MB/s
   | +    # crc64_be_benchmark: len=16: 264 MB/s
   | +    # crc64_be_benchmark: len=64: 476 MB/s
   | +    # crc64_be_benchmark: len=127: 499 MB/s
   | +    # crc64_be_benchmark: len=128: 558 MB/s
   | +    # crc64_be_benchmark: len=200: 576 MB/s
   | +    # crc64_be_benchmark: len=256: 611 MB/s
   | +    # crc64_be_benchmark: len=511: 621 MB/s
   | +    # crc64_be_benchmark: len=512: 638 MB/s
   | +    # crc64_be_benchmark: len=1024: 659 MB/s
   | +    # crc64_be_benchmark: len=3173: 667 MB/s
   | +    # crc64_be_benchmark: len=4096: 671 MB/s
   | +    # crc64_be_benchmark: len=16384: 674 MB/s
   |      ok 12 crc64_be_benchmark
   |      ok 13 crc64_nvme_test
   | -    # crc64_nvme_benchmark: len=1: 64 MB/s
   | -    # crc64_nvme_benchmark: len=16: 144 MB/s
   | -    # crc64_nvme_benchmark: len=64: 154 MB/s
   | -    # crc64_nvme_benchmark: len=127: 156 MB/s
   | -    # crc64_nvme_benchmark: len=128: 156 MB/s
   | -    # crc64_nvme_benchmark: len=200: 156 MB/s
   | -    # crc64_nvme_benchmark: len=256: 156 MB/s
   | -    # crc64_nvme_benchmark: len=511: 157 MB/s
   | -    # crc64_nvme_benchmark: len=512: 157 MB/s
   | -    # crc64_nvme_benchmark: len=1024: 157 MB/s
   | -    # crc64_nvme_benchmark: len=3173: 158 MB/s
   | -    # crc64_nvme_benchmark: len=4096: 158 MB/s
   | -    # crc64_nvme_benchmark: len=16384: 158 MB/s
   | +    # crc64_nvme_benchmark: len=1: 36 MB/s
   | +    # crc64_nvme_benchmark: len=16: 479 MB/s
   | +    # crc64_nvme_benchmark: len=64: 1340 MB/s
   | +    # crc64_nvme_benchmark: len=127: 1179 MB/s
   | +    # crc64_nvme_benchmark: len=128: 1766 MB/s
   | +    # crc64_nvme_benchmark: len=200: 1965 MB/s
   | +    # crc64_nvme_benchmark: len=256: 2201 MB/s
   | +    # crc64_nvme_benchmark: len=511: 2087 MB/s
   | +    # crc64_nvme_benchmark: len=512: 2464 MB/s
   | +    # crc64_nvme_benchmark: len=1024: 2331 MB/s
   | +    # crc64_nvme_benchmark: len=3173: 2673 MB/s
   | +    # crc64_nvme_benchmark: len=4096: 2745 MB/s
   | +    # crc64_nvme_benchmark: len=16384: 2782 MB/s
   |      ok 14 crc64_nvme_benchmark
   |  # crc: pass:14 fail:0 skip:0 total:14
   |  # Totals: pass:14 fail:0 skip:0 total:14

Quoted 7 lines

That's a significant speed up for this popular SoC, and it would be great to get this series in for the next merge window! Thank you!
Tested-by Björn Töpel <[email protected]>
Björn

I am happy to see that crc-t10dif can achieve remarkable performance improvements under the optimization of the ZBC extension. It has been a great honor for me to contribute to such impactful work.

Zhihang