Skip to contentSkip to conversation

RISC-V: support some cryptography accelerations

PATCH In review Heiko Stuebner 11 participants 50 messages Jul 11, 2023 – Nov 28, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v4 cover RISC-V: support some cryptography accelerations

From: Heiko Stuebner <[email protected]>

This series provides cryptographic implementations using the vector crypto extensions.

v13 of the vector patchset dropped the patches for in-kernel usage of vector instructions, I picked the ones from v12 over into this series for now.

My basic goal was to not re-invent cryptographic code, so the heavy lifting is done by those perl-asm scripts used in openssl and the perl code used here-in stems from code that is targetted at openssl [0] and is unmodified from there to limit needed review effort.

With a matching qemu (there are patches for vector-crypto flying around) the in-kernel crypto-selftests (also the extended ones) are very happy so far.

changes in v4:
- split off from scalar crypto patches but base on top of them
- adapt to pending openssl code [0] using the now frozen vector crypto extensions - with all its changes
[0] https://github.com/openssl/openssl/pull/20149

changes in v3:
- rebase on top of 6.3-rc2
- rebase on top of vector-v14 patchset
- add the missing Co-developed-by mentions to showcase the people that did the actual openSSL crypto code

changes in v2:
- rebased on 6.2 + zbb series, so don't include already applied changes anymore
- refresh code picked from openssl as that side matures
- more algorithms (SHA512, AES, SM3, SM4)

Greentime Hu (2):
riscv: Add support for kernel mode vector
riscv: Add vector extension XOR implementation

Heiko Stuebner (10):
RISC-V: add helper function to read the vector VLEN
RISC-V: add vector crypto extension detection
RISC-V: crypto: update perl include with helpers for vector (crypto)
instructions
RISC-V: crypto: add Zvbb+Zvbc accelerated GCM GHASH implementation
RISC-V: crypto: add Zvkg accelerated GCM GHASH implementation
RISC-V: crypto: add a vector-crypto-accelerated SHA256 implementation
RISC-V: crypto: add a vector-crypto-accelerated SHA512 implementation
RISC-V: crypto: add Zvkned accelerated AES encryption implementation
RISC-V: crypto: add Zvksed accelerated SM4 encryption implementation
RISC-V: crypto: add Zvksh accelerated SM3 hash implementation

 arch/riscv/crypto/Kconfig                     |  68 ++-
 arch/riscv/crypto/Makefile                    |  44 +-
 arch/riscv/crypto/aes-riscv-glue.c            | 168 ++++++
 arch/riscv/crypto/aes-riscv64-zvkned.pl       | 530 ++++++++++++++++++
 arch/riscv/crypto/ghash-riscv64-glue.c        | 245 ++++++++
 arch/riscv/crypto/ghash-riscv64-zvbb-zvbc.pl  | 380 +++++++++++++
 arch/riscv/crypto/ghash-riscv64-zvkg.pl       | 168 ++++++
 arch/riscv/crypto/riscv.pm                    | 433 +++++++++++++-
 arch/riscv/crypto/sha256-riscv64-glue.c       | 115 ++++
 .../crypto/sha256-riscv64-zvbb-zvknha.pl      | 314 +++++++++++
 arch/riscv/crypto/sha512-riscv64-glue.c       | 106 ++++
 .../crypto/sha512-riscv64-zvbb-zvknhb.pl      | 377 +++++++++++++
 arch/riscv/crypto/sm3-riscv64-glue.c          | 112 ++++
 arch/riscv/crypto/sm3-riscv64-zvksh.pl        | 225 ++++++++
 arch/riscv/crypto/sm4-riscv64-glue.c          | 162 ++++++
 arch/riscv/crypto/sm4-riscv64-zvksed.pl       | 300 ++++++++++
 arch/riscv/include/asm/hwcap.h                |   9 +
 arch/riscv/include/asm/vector.h               |  28 +
 arch/riscv/include/asm/xor.h                  |  82 +++
 arch/riscv/kernel/Makefile                    |   1 +
 arch/riscv/kernel/cpu.c                       |   8 +
 arch/riscv/kernel/cpufeature.c                |  50 ++
 arch/riscv/kernel/kernel_mode_vector.c        | 132 +++++
 arch/riscv/lib/Makefile                       |   1 +
 arch/riscv/lib/xor.S                          |  81 +++
 25 files changed, 4136 insertions(+), 3 deletions(-)
 create mode 100644 arch/riscv/crypto/aes-riscv-glue.c
 create mode 100644 arch/riscv/crypto/aes-riscv64-zvkned.pl
 create mode 100644 arch/riscv/crypto/ghash-riscv64-zvbb-zvbc.pl
 create mode 100644 arch/riscv/crypto/ghash-riscv64-zvkg.pl
 create mode 100644 arch/riscv/crypto/sha256-riscv64-glue.c
 create mode 100644 arch/riscv/crypto/sha256-riscv64-zvbb-zvknha.pl
 create mode 100644 arch/riscv/crypto/sha512-riscv64-glue.c
 create mode 100644 arch/riscv/crypto/sha512-riscv64-zvbb-zvknhb.pl
 create mode 100644 arch/riscv/crypto/sm3-riscv64-glue.c
 create mode 100644 arch/riscv/crypto/sm3-riscv64-zvksh.pl
 create mode 100644 arch/riscv/crypto/sm4-riscv64-glue.c
 create mode 100644 arch/riscv/crypto/sm4-riscv64-zvksed.pl
 create mode 100644 arch/riscv/include/asm/xor.h
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
 create mode 100644 arch/riscv/lib/xor.S
Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#1 lore raw

PATCH v4 1/12 riscv: Add support for kernel mode vector

From: Greentime Hu <[email protected]>

Add kernel_rvv_begin() and kernel_rvv_end() function declarations and corresponding definitions in kernel_mode_vector.c

These are needed to wrap uses of vector in kernel mode.

---
 arch/riscv/include/asm/vector.h        |  17 ++++
 arch/riscv/kernel/Makefile             |   1 +
 arch/riscv/kernel/kernel_mode_vector.c | 132 +++++++++++++++++++++++++
 3 files changed, 150 insertions(+)
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
arch/riscv/include/asm/vector.h +17 -0
index 3d78930cab51..ac2c23045eec 100644
@@ -196,6 +196,23 @@ static inline void __switch_to_vector(struct task_struct *prev,
 void riscv_v_vstate_ctrl_init(struct task_struct *tsk);
 bool riscv_v_vstate_ctrl_user_allowed(void);
 
+static inline void riscv_v_flush_cpu_state(void)
+{
+	asm volatile (
+		".option push\n\t"
+		".option arch, +v\n\t"
+		"vsetvli	t0, x0, e8, m8, ta, ma\n\t"
+		"vmv.v.i	v0, 0\n\t"
+		"vmv.v.i	v8, 0\n\t"
+		"vmv.v.i	v16, 0\n\t"
+		"vmv.v.i	v24, 0\n\t"
+		".option pop\n\t"
+		: : : "t0");
+}
+
+void kernel_rvv_begin(void);
+void kernel_rvv_end(void);
+
 #else /* ! CONFIG_RISCV_ISA_V  */
 
 struct pt_regs;
arch/riscv/kernel/Makefile +1 -0
index 506cc4a9a45a..3f4435746af7 100644
@@ -61,6 +61,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/
 obj-$(CONFIG_RISCV_M_MODE)	+= traps_misaligned.o
 obj-$(CONFIG_FPU)		+= fpu.o
 obj-$(CONFIG_RISCV_ISA_V)	+= vector.o
+obj-$(CONFIG_RISCV_ISA_V)	+= kernel_mode_vector.o
 obj-$(CONFIG_SMP)		+= smpboot.o
 obj-$(CONFIG_SMP)		+= smp.o
 obj-$(CONFIG_SMP)		+= cpu_ops.o
arch/riscv/kernel/kernel_mode_vector.c new file +132 -0

132 lines not on this page · raw message

Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#2 lore raw

Re: [PATCH v4 01/12] riscv: Add support for kernel mode vector

	Hi,
Quoted 28 lines · Le tiistaina 11. heinäkuuta 2023, 18.37.32 EEST Heiko Stuebner a écrit :

From: Greentime Hu <[email protected]>

Add kernel_rvv_begin() and kernel_rvv_end() function declarations and corresponding definitions in kernel_mode_vector.c

These are needed to wrap uses of vector in kernel mode.

---
 arch/riscv/include/asm/vector.h        |  17 ++++
 arch/riscv/kernel/Makefile             |   1 +
 arch/riscv/kernel/kernel_mode_vector.c | 132 +++++++++++++++++++++++++
 3 files changed, 150 insertions(+)
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
(quoted diff) +0 -0

b/arch/riscv/include/asm/vector.h index 3d78930cab51..ac2c23045eec 100644

arch/riscv/include/asm/vector.h +2 -0
@@ -196,6 +196,23 @@ static inline void __switch_to_vector(struct
 task_struct *prev, void riscv_v_vstate_ctrl_init(struct task_struct *tsk);
 bool riscv_v_vstate_ctrl_user_allowed(void);
 
+static inline void riscv_v_flush_cpu_state(void)
+{

arch/riscv/include/asm/vector.h static inline void __switch_to_vector(struct · line 210

Quoted 10 lines
arch/riscv/include/asm/vector.h +10 -0
+	asm volatile (
+		".option push\n\t"
+		".option arch, +v\n\t"
+		"vsetvli	t0, x0, e8, m8, ta, ma\n\t"
+		"vmv.v.i	v0, 0\n\t"
+		"vmv.v.i	v8, 0\n\t"
+		"vmv.v.i	v16, 0\n\t"
+		"vmv.v.i	v24, 0\n\t"
+		".option pop\n\t"
+		: : : "t0");

Why bother with zeroing out the vectors before kernel use? That sounds like it will only hide bugs in kernel code - implicitly assuming that everything is initially zero. Ditto initialising the vector configuration; if you really want to have a fixed initial value rather than "leak" whatever user set, better use an invalid configuration (vill=1), IMO.

Quoted 140 lines
arch/riscv/include/asm/vector.h +5 -0
+}
+
+void kernel_rvv_begin(void);
+void kernel_rvv_end(void);
+
 #else /* ! CONFIG_RISCV_ISA_V  */
 
 struct pt_regs;
arch/riscv/kernel/Makefile +1 -0
index 506cc4a9a45a..3f4435746af7 100644
@@ -61,6 +61,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/
 obj-$(CONFIG_RISCV_M_MODE)	+= traps_misaligned.o
 obj-$(CONFIG_FPU)		+= fpu.o
 obj-$(CONFIG_RISCV_ISA_V)	+= vector.o
+obj-$(CONFIG_RISCV_ISA_V)	+= kernel_mode_vector.o
 obj-$(CONFIG_SMP)		+= smpboot.o
 obj-$(CONFIG_SMP)		+= smp.o
 obj-$(CONFIG_SMP)		+= cpu_ops.o
(quoted diff) +0 -0

b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..2d704190c054

arch/riscv/kernel/kernel_mode_vector.c new file +42 -0
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2012 ARM Ltd.
+ * Author: Catalin Marinas <[email protected]>
+ * Copyright (C) 2017 Linaro Ltd. <[email protected]>
+ * Copyright (C) 2021 SiFive
+ */
+#include <linux/compiler.h>
+#include <linux/irqflags.h>
+#include <linux/percpu.h>
+#include <linux/preempt.h>
+#include <linux/types.h>
+
+#include <asm/vector.h>
+#include <asm/switch_to.h>
+
+DECLARE_PER_CPU(bool, vector_context_busy);
+DEFINE_PER_CPU(bool, vector_context_busy);
+
+/*
+ * may_use_vector - whether it is allowable at this time to issue vector
+ *                instructions or access the vector register file
+ *
+ * Callers must not assume that the result remains true beyond the next
+ * preempt_enable() or return from softirq context.
+ */
+static __must_check inline bool may_use_vector(void)
+{
+	/*
+	 * vector_context_busy is only set while preemption is disabled,
+	 * and is clear whenever preemption is enabled. Since
+	 * this_cpu_read() is atomic w.r.t. preemption, vector_context_busy
+	 * cannot change under our feet -- if it's set we cannot be
+	 * migrated, and if it's clear we cannot be migrated to a CPU
+	 * where it is set.
+	 */
+	return !in_irq() && !irqs_disabled() && !in_nmi() &&
+	       !this_cpu_read(vector_context_busy);
+}
+
+/*
+ * Claim ownership of the CPU vector context for use by the calling

context. + *
+ * The caller may freely manipulate the vector context metadata until
+ * put_cpu_vector_context() is called.
+ */

arch/riscv/kernel/kernel_mode_vector.c new file +27 -0
+static void get_cpu_vector_context(void)
+{
+	bool busy;
+
+	preempt_disable();
+	busy = __this_cpu_xchg(vector_context_busy, true);
+
+	WARN_ON(busy);
+}
+
+/*
+ * Release the CPU vector context.
+ *
+ * Must be called from a context in which get_cpu_vector_context() was
+ * previously called, with no call to put_cpu_vector_context() in the
+ * meantime.
+ */
+static void put_cpu_vector_context(void)
+{
+	bool busy = __this_cpu_xchg(vector_context_busy, false);
+
+	WARN_ON(!busy);
+	preempt_enable();
+}
+
+/*
+ * kernel_rvv_begin(): obtain the CPU vector registers for use by the

calling + * context
+ *
+ * Must not be called unless may_use_vector() returns true.
+ * Task context in the vector registers is saved back to memory as necessary. + *
+ * A matching call to kernel_rvv_end() must be made before returning from the + * calling context.
+ *
+ * The caller may freely use the vector registers until kernel_rvv_end() is
+ * called.
+ */

arch/riscv/kernel/kernel_mode_vector.c new file +28 -0
+void kernel_rvv_begin(void)
+{
+	if (WARN_ON(!has_vector()))
+		return;
+
+	WARN_ON(!may_use_vector());
+
+	/* Acquire kernel mode vector */
+	get_cpu_vector_context();
+
+	/* Save vector state, if any */
+	riscv_v_vstate_save(current, task_pt_regs(current));
+
+	/* Enable vector */
+	riscv_v_enable();
+
+	/* Invalidate vector regs */
+	riscv_v_flush_cpu_state();
+}
+EXPORT_SYMBOL_GPL(kernel_rvv_begin);
+
+/*
+ * kernel_rvv_end(): give the CPU vector registers back to the current task
+ *
+ * Must be called from a context in which kernel_rvv_begin() was previously
+ * called, with no call to kernel_rvv_end() in the meantime.
+ *
+ * The caller must not use the vector registers after this function is

called, + * unless kernel_rvv_begin() is called again in the meantime.
+ */

arch/riscv/kernel/kernel_mode_vector.c line 124

Quoted 10 lines
arch/riscv/kernel/kernel_mode_vector.c new file +10 -0
+void kernel_rvv_end(void)
+{
+	if (WARN_ON(!has_vector()))
+		return;
+
+	/* Invalidate vector regs */
+	riscv_v_flush_cpu_state();
+
+	/* Restore vector state, if any */
+	riscv_v_vstate_restore(current, task_pt_regs(current));

I thought that the kernel was already nuking user vectors on every system call, since the RVV spec says so.

Are you trying to use vectors from interrupts? Otherwise, isn't this flush & restore superfluous?

Quoted 8 lines
arch/riscv/kernel/kernel_mode_vector.c new file +8 -0
+
+	/* disable vector */
+	riscv_v_disable();
+
+	/* release kernel mode vector */
+	put_cpu_vector_context();
+}
+EXPORT_SYMBOL_GPL(kernel_rvv_end);
Signature · 2 lines
雷米‧德尼-库尔蒙
http://www.remlab.net/
#3 lore raw

Re: [PATCH v4 01/12] riscv: Add support for kernel mode vector

Hi, Heiko

Quoted 40 lines · Rémi Denis-Courmont
        Hi,
Quoted 36 lines · Le tiistaina 11. heinäkuuta 2023, 18.37.32 EEST Heiko Stuebner a écrit :

From: Greentime Hu <[email protected]>

Add kernel_rvv_begin() and kernel_rvv_end() function declarations and corresponding definitions in kernel_mode_vector.c

These are needed to wrap uses of vector in kernel mode.

---
 arch/riscv/include/asm/vector.h        |  17 ++++
 arch/riscv/kernel/Makefile             |   1 +
 arch/riscv/kernel/kernel_mode_vector.c | 132 +++++++++++++++++++++++++
 3 files changed, 150 insertions(+)
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
(quoted diff) +0 -0

b/arch/riscv/include/asm/vector.h index 3d78930cab51..ac2c23045eec 100644

arch/riscv/include/asm/vector.h +10 -0
@@ -196,6 +196,23 @@ static inline void __switch_to_vector(struct
 task_struct *prev, void riscv_v_vstate_ctrl_init(struct task_struct *tsk);
 bool riscv_v_vstate_ctrl_user_allowed(void);
 
+static inline void riscv_v_flush_cpu_state(void)
+{
+     asm volatile (
+             ".option push\n\t"
+             ".option arch, +v\n\t"
+             "vsetvli        t0, x0, e8, m8, ta, ma\n\t"
+             "vmv.v.i        v0, 0\n\t"
+             "vmv.v.i        v8, 0\n\t"
+             "vmv.v.i        v16, 0\n\t"
+             "vmv.v.i        v24, 0\n\t"
Quoted 8 lines

arch/riscv/include/asm/vector.h static inline void __switch_to_vector(struct task_struct *prev, · line 210

Quoted 2 lines
arch/riscv/include/asm/vector.h +2 -0
+             ".option pop\n\t"
+             : : : "t0");

Why bother with zeroing out the vectors before kernel use? That sounds like it will only hide bugs in kernel code - implicitly assuming that everything is initially zero. Ditto initialising the vector configuration; if you really want to have a fixed initial value rather than "leak" whatever user set, better use an invalid configuration (vill=1), IMO.

Yes, I agree that we don't have to zero out (or invalid) v registers before any kernel uses. And we only have to restore user's v registers once, before really returning back to the user space. Actually I am going to send out the series (for kernel-mode vector) having these improvements in a few days. Does it seem ok to you to drop the first two patches and rebase on top of mine at the next respin for the crypto vector? Or is there any good way that you can think of to let us cooperate on this?

Quoted 175 lines
Quoted 141 lines
arch/riscv/include/asm/vector.h +5 -0
+}
+
+void kernel_rvv_begin(void);
+void kernel_rvv_end(void);
+
 #else /* ! CONFIG_RISCV_ISA_V  */
 
 struct pt_regs;
arch/riscv/kernel/Makefile +1 -0
index 506cc4a9a45a..3f4435746af7 100644
@@ -61,6 +61,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/
 obj-$(CONFIG_RISCV_M_MODE)   += traps_misaligned.o
 obj-$(CONFIG_FPU)            += fpu.o
 obj-$(CONFIG_RISCV_ISA_V)    += vector.o
+obj-$(CONFIG_RISCV_ISA_V)    += kernel_mode_vector.o
 obj-$(CONFIG_SMP)            += smpboot.o
 obj-$(CONFIG_SMP)            += smp.o
 obj-$(CONFIG_SMP)            += cpu_ops.o
(quoted diff) +0 -0

b/arch/riscv/kernel/kernel_mode_vector.c new file mode 100644 index 000000000000..2d704190c054

arch/riscv/kernel/kernel_mode_vector.c new file +42 -0
@@ -0,0 +1,132 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2012 ARM Ltd.
+ * Author: Catalin Marinas <[email protected]>
+ * Copyright (C) 2017 Linaro Ltd. <[email protected]>
+ * Copyright (C) 2021 SiFive
+ */
+#include <linux/compiler.h>
+#include <linux/irqflags.h>
+#include <linux/percpu.h>
+#include <linux/preempt.h>
+#include <linux/types.h>
+
+#include <asm/vector.h>
+#include <asm/switch_to.h>
+
+DECLARE_PER_CPU(bool, vector_context_busy);
+DEFINE_PER_CPU(bool, vector_context_busy);
+
+/*
+ * may_use_vector - whether it is allowable at this time to issue vector
+ *                instructions or access the vector register file
+ *
+ * Callers must not assume that the result remains true beyond the next
+ * preempt_enable() or return from softirq context.
+ */
+static __must_check inline bool may_use_vector(void)
+{
+     /*
+      * vector_context_busy is only set while preemption is disabled,
+      * and is clear whenever preemption is enabled. Since
+      * this_cpu_read() is atomic w.r.t. preemption, vector_context_busy
+      * cannot change under our feet -- if it's set we cannot be
+      * migrated, and if it's clear we cannot be migrated to a CPU
+      * where it is set.
+      */
+     return !in_irq() && !irqs_disabled() && !in_nmi() &&
+            !this_cpu_read(vector_context_busy);
+}
+
+/*
+ * Claim ownership of the CPU vector context for use by the calling

context. + *
+ * The caller may freely manipulate the vector context metadata until
+ * put_cpu_vector_context() is called.
+ */

arch/riscv/kernel/kernel_mode_vector.c new file +27 -0
+static void get_cpu_vector_context(void)
+{
+     bool busy;
+
+     preempt_disable();
+     busy = __this_cpu_xchg(vector_context_busy, true);
+
+     WARN_ON(busy);
+}
+
+/*
+ * Release the CPU vector context.
+ *
+ * Must be called from a context in which get_cpu_vector_context() was
+ * previously called, with no call to put_cpu_vector_context() in the
+ * meantime.
+ */
+static void put_cpu_vector_context(void)
+{
+     bool busy = __this_cpu_xchg(vector_context_busy, false);
+
+     WARN_ON(!busy);
+     preempt_enable();
+}
+
+/*
+ * kernel_rvv_begin(): obtain the CPU vector registers for use by the

calling + * context
+ *
+ * Must not be called unless may_use_vector() returns true.
+ * Task context in the vector registers is saved back to memory as necessary. + *
+ * A matching call to kernel_rvv_end() must be made before returning from the + * calling context.
+ *
+ * The caller may freely use the vector registers until kernel_rvv_end() is
+ * called.
+ */

arch/riscv/kernel/kernel_mode_vector.c new file +28 -0
+void kernel_rvv_begin(void)
+{
+     if (WARN_ON(!has_vector()))
+             return;
+
+     WARN_ON(!may_use_vector());
+
+     /* Acquire kernel mode vector */
+     get_cpu_vector_context();
+
+     /* Save vector state, if any */
+     riscv_v_vstate_save(current, task_pt_regs(current));
+
+     /* Enable vector */
+     riscv_v_enable();
+
+     /* Invalidate vector regs */
+     riscv_v_flush_cpu_state();
+}
+EXPORT_SYMBOL_GPL(kernel_rvv_begin);
+
+/*
+ * kernel_rvv_end(): give the CPU vector registers back to the current task
+ *
+ * Must be called from a context in which kernel_rvv_begin() was previously
+ * called, with no call to kernel_rvv_end() in the meantime.
+ *
+ * The caller must not use the vector registers after this function is

called, + * unless kernel_rvv_begin() is called again in the meantime.
+ */

arch/riscv/kernel/kernel_mode_vector.c line 124

Quoted 10 lines
arch/riscv/kernel/kernel_mode_vector.c new file +10 -0
+void kernel_rvv_end(void)
+{
+     if (WARN_ON(!has_vector()))
+             return;
+
+     /* Invalidate vector regs */
+     riscv_v_flush_cpu_state();
+
+     /* Restore vector state, if any */
+     riscv_v_vstate_restore(current, task_pt_regs(current));

I thought that the kernel was already nuking user vectors on every system call, since the RVV spec says so.

Are you trying to use vectors from interrupts? Otherwise, isn't this flush & restore superfluous?

arch/riscv/kernel/kernel_mode_vector.c line 132

Quoted 8 lines
arch/riscv/kernel/kernel_mode_vector.c new file +8 -0
+
+     /* disable vector */
+     riscv_v_disable();
+
+     /* release kernel mode vector */
+     put_cpu_vector_context();
+}
+EXPORT_SYMBOL_GPL(kernel_rvv_end);

--
雷米‧德尼-库尔蒙
http://www.remlab.net/

Quoted 4 lines

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

Thanks.

Andy

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#4 lore raw

PATCH v4 2/12 riscv: Add vector extension XOR implementation

In reply to Heiko Stuebner

From: Greentime Hu <[email protected]>

This patch adds support for vector optimized XOR and it is tested in qemu.

---
 arch/riscv/include/asm/xor.h | 82 ++++++++++++++++++++++++++++++++++++
 arch/riscv/lib/Makefile      |  1 +
 arch/riscv/lib/xor.S         | 81 +++++++++++++++++++++++++++++++++++
 3 files changed, 164 insertions(+)
 create mode 100644 arch/riscv/include/asm/xor.h
 create mode 100644 arch/riscv/lib/xor.S
arch/riscv/include/asm/xor.h new file +82 -0
new file mode 100644
index 000000000000..74867c7fd955
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2021 SiFive
+ */
+
+#include <linux/hardirq.h>
+#include <asm-generic/xor.h>
+#ifdef CONFIG_VECTOR
+#include <asm/vector.h>
+#include <asm/switch_to.h>
+
+void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2);
+void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2,
+		 const unsigned long *__restrict p3);
+void xor_regs_4_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2,
+		 const unsigned long *__restrict p3,
+		 const unsigned long *__restrict p4);
+void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2,
+		 const unsigned long *__restrict p3,
+		 const unsigned long *__restrict p4,
+		 const unsigned long *__restrict p5);
+
+static void xor_rvv_2(unsigned long bytes, unsigned long *__restrict p1,
+		      const unsigned long *__restrict p2)
+{
+	kernel_rvv_begin();
+	xor_regs_2_(bytes, p1, p2);
+	kernel_rvv_end();
+}
+
+static void xor_rvv_3(unsigned long bytes, unsigned long *__restrict p1,
+		      const unsigned long *__restrict p2,
+		      const unsigned long *__restrict p3)
+{
+	kernel_rvv_begin();
+	xor_regs_3_(bytes, p1, p2, p3);
+	kernel_rvv_end();
+}
+
+static void xor_rvv_4(unsigned long bytes, unsigned long *__restrict p1,
+		      const unsigned long *__restrict p2,
+		      const unsigned long *__restrict p3,
+		      const unsigned long *__restrict p4)
+{
+	kernel_rvv_begin();
+	xor_regs_4_(bytes, p1, p2, p3, p4);
+	kernel_rvv_end();
+}
+
+static void xor_rvv_5(unsigned long bytes, unsigned long *__restrict p1,
+		      const unsigned long *__restrict p2,
+		      const unsigned long *__restrict p3,
+		      const unsigned long *__restrict p4,
+		      const unsigned long *__restrict p5)
+{
+	kernel_rvv_begin();
+	xor_regs_5_(bytes, p1, p2, p3, p4, p5);
+	kernel_rvv_end();
+}
+
+static struct xor_block_template xor_block_rvv = {
+	.name = "rvv",
+	.do_2 = xor_rvv_2,
+	.do_3 = xor_rvv_3,
+	.do_4 = xor_rvv_4,
+	.do_5 = xor_rvv_5
+};
+
+#undef XOR_TRY_TEMPLATES
+#define XOR_TRY_TEMPLATES           \
+	do {        \
+		xor_speed(&xor_block_8regs);    \
+		xor_speed(&xor_block_32regs);    \
+		if (has_vector()) { \
+			xor_speed(&xor_block_rvv);\
+		} \
+	} while (0)
+#endif
arch/riscv/lib/Makefile +1 -0
index 26cb2502ecf8..3164112680f1 100644
@@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT)	+= tishift.o
 lib-$(CONFIG_RISCV_ISA_ZICBOZ)	+= clear_page.o
 
 obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
+lib-$(CONFIG_VECTOR)	+= xor.o
arch/riscv/lib/xor.S new file +81 -0

81 lines not on this page · raw message

Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#5 lore raw

Re: [PATCH v4 02/12] riscv: Add vector extension XOR implementation

Quoted 6 lines · Le tiistaina 11. heinäkuuta 2023, 18.37.33 EEST Heiko Stuebner a écrit :
arch/riscv/lib/xor.S new file +0 -0
new file mode 100644
index 000000000000..3bc059e18171

arch/riscv/lib/xor.S line 10

Quoted 10 lines
arch/riscv/lib/xor.S new file +10 -0
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2021 SiFive
+ */
+#include <linux/linkage.h>
+#include <asm-generic/export.h>
+#include <asm/asm.h>
+
+ENTRY(xor_regs_2_)
+	vsetvli a3, a0, e8, m8, ta, ma

AFAICT, so far, Linux only uses vsetvli to save/restore/flush vectors, and that's, of course, with LMUL=8, so that's not really telling much anything. This function could be the first actual vector optimisation in kernel if/when it gets merged.

Should the same group multiplier be used for "actual" vector loops throughout the kernel? I've seen conflicting advises or opinions here. Should kernel code always use the maximum possible LMUL, depending on register pressure of the loop? Or will that just increase latency with no bandwidth gains compared to, say, LMUL=1 or LMUL=2?

Quoted 71 lines
arch/riscv/lib/xor.S new file +71 -0
+	vle8.v v0, (a1)
+	vle8.v v8, (a2)
+	sub a0, a0, a3
+	vxor.vv v16, v0, v8
+	add a2, a2, a3
+	vse8.v v16, (a1)
+	add a1, a1, a3
+	bnez a0, xor_regs_2_
+	ret
+END(xor_regs_2_)
+EXPORT_SYMBOL(xor_regs_2_)
+
+ENTRY(xor_regs_3_)
+	vsetvli a4, a0, e8, m8, ta, ma
+	vle8.v v0, (a1)
+	vle8.v v8, (a2)
+	sub a0, a0, a4
+	vxor.vv v0, v0, v8
+	vle8.v v16, (a3)
+	add a2, a2, a4
+	vxor.vv v16, v0, v16
+	add a3, a3, a4
+	vse8.v v16, (a1)
+	add a1, a1, a4
+	bnez a0, xor_regs_3_
+	ret
+END(xor_regs_3_)
+EXPORT_SYMBOL(xor_regs_3_)
+
+ENTRY(xor_regs_4_)
+	vsetvli a5, a0, e8, m8, ta, ma
+	vle8.v v0, (a1)
+	vle8.v v8, (a2)
+	sub a0, a0, a5
+	vxor.vv v0, v0, v8
+	vle8.v v16, (a3)
+	add a2, a2, a5
+	vxor.vv v0, v0, v16
+	vle8.v v24, (a4)
+	add a3, a3, a5
+	vxor.vv v16, v0, v24
+	add a4, a4, a5
+	vse8.v v16, (a1)
+	add a1, a1, a5
+	bnez a0, xor_regs_4_
+	ret
+END(xor_regs_4_)
+EXPORT_SYMBOL(xor_regs_4_)
+
+ENTRY(xor_regs_5_)
+	vsetvli a6, a0, e8, m8, ta, ma
+	vle8.v v0, (a1)
+	vle8.v v8, (a2)
+	sub a0, a0, a6
+	vxor.vv v0, v0, v8
+	vle8.v v16, (a3)
+	add a2, a2, a6
+	vxor.vv v0, v0, v16
+	vle8.v v24, (a4)
+	add a3, a3, a6
+	vxor.vv v0, v0, v24
+	vle8.v v8, (a5)
+	add a4, a4, a6
+	vxor.vv v16, v0, v8
+	add a5, a5, a6
+	vse8.v v16, (a1)
+	add a1, a1, a6
+	bnez a0, xor_regs_5_
+	ret
+END(xor_regs_5_)
+EXPORT_SYMBOL(xor_regs_5_)
Signature · 10 lines
レミ・デニ-クールモン
http://www.remlab.net/




_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#6 lore raw

PATCH v4 3/12 RISC-V: add helper function to read the vector VLEN

In reply to Heiko Stuebner

From: Heiko Stuebner <[email protected]>

VLEN describes the length of each vector register and some instructions need specific minimal VLENs to work correctly.

The vector code already includes a variable riscv_vsize that contains the value of "32 vector registers with vlenb length" that gets filled during boot. vlenb is the value contained in the CSR_VLENB register and the value represents "VLEN / 8".

So add riscv_vector_vlen() to return the actual VLEN value for in-kernel users when they need to check the available VLEN.

---
 arch/riscv/include/asm/vector.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
arch/riscv/include/asm/vector.h +11 -0
index ac2c23045eec..88cf76a2316d 100644
@@ -232,4 +232,15 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; }
 
 #endif /* CONFIG_RISCV_ISA_V */
 
+/*
+ * Return the implementation's vlen value.
+ *
+ * riscv_vsize contains the value of "32 vector registers with vlenb length"
+ * so rebuild the vlen value in bits from it.
+ */
+static inline int riscv_vector_vlen(void)
+{
+	return riscv_v_vsize / 32 * 8;
+}
+
 #endif /* ! __ASM_RISCV_VECTOR_H */
Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#7 lore raw

Re: [PATCH v4 03/12] RISC-V: add helper function to read the vector VLEN

Quoted 33 lines · Le tiistaina 11. heinäkuuta 2023, 18.37.34 EEST Heiko Stuebner a écrit :

From: Heiko Stuebner <[email protected]>

VLEN describes the length of each vector register and some instructions need specific minimal VLENs to work correctly.

The vector code already includes a variable riscv_vsize that contains the value of "32 vector registers with vlenb length" that gets filled during boot. vlenb is the value contained in the CSR_VLENB register and the value represents "VLEN / 8".

So add riscv_vector_vlen() to return the actual VLEN value for in-kernel users when they need to check the available VLEN.

---
 arch/riscv/include/asm/vector.h | 11 +++++++++++
 1 file changed, 11 insertions(+)
(quoted diff) +0 -0

b/arch/riscv/include/asm/vector.h index ac2c23045eec..88cf76a2316d 100644

arch/riscv/include/asm/vector.h +4 -0
@@ -232,4 +232,15 @@ static inline bool
 riscv_v_vstate_ctrl_user_allowed(void) { return false; }
 
 #endif /* CONFIG_RISCV_ISA_V */
 
+/*
+ * Return the implementation's vlen value.
+ *
+ * riscv_vsize contains the value of "32 vector registers with vlenb

length" + * so rebuild the vlen value in bits from it.
+ */

arch/riscv/include/asm/vector.h static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; } · line 244

Quoted 4 lines
arch/riscv/include/asm/vector.h +4 -0
+static inline int riscv_vector_vlen(void)
+{
+	return riscv_v_vsize / 32 * 8;
+}

KVM already has a bespoke conversion to bytes (rather than bits). Factor code?

Quoted 2 lines

+
#endif /* ! __ASM_RISCV_VECTOR_H */

Signature · 10 lines
レミ・デニ-クールモン
http://www.remlab.net/




_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#8 lore raw

PATCH v4 4/12 RISC-V: add vector crypto extension detection

In reply to Heiko Stuebner

From: Heiko Stuebner <[email protected]>

Add detection for some extensions of the vector-crypto specification:
- Zvkb: Vector Bit-manipulation used in Cryptography
- Zvkg: Vector GCM/GMAC
- Zvknha and Zvknhb: NIST Algorithm Suite
- Zvkns: AES-128, AES-256 Single Round Suite
- Zvksed: ShangMi Algorithm Suite
- Zvksh: ShangMi Algorithm Suite

As their use is very specific and will likely be limited to special places we expect current code to just pre-encode those instructions, so right now we don't introduce toolchain requirements.

---
 arch/riscv/include/asm/hwcap.h |  9 ++++++
 arch/riscv/kernel/cpu.c        |  8 ++++++
 arch/riscv/kernel/cpufeature.c | 50 ++++++++++++++++++++++++++++++++++
 3 files changed, 67 insertions(+)
arch/riscv/include/asm/hwcap.h +9 -0
index b80ca6e77088..0f5172fa87b0 100644
@@ -64,6 +64,15 @@
 #define RISCV_ISA_EXT_ZKSED		51
 #define RISCV_ISA_EXT_ZKSH		52
 #define RISCV_ISA_EXT_ZKT		53
+#define RISCV_ISA_EXT_ZVBB		54
+#define RISCV_ISA_EXT_ZVBC		55
+#define RISCV_ISA_EXT_ZVKG		56
+#define RISCV_ISA_EXT_ZVKNED		57
+#define RISCV_ISA_EXT_ZVKNHA		58
+#define RISCV_ISA_EXT_ZVKNHB		59
+#define RISCV_ISA_EXT_ZVKSED		60
+#define RISCV_ISA_EXT_ZVKSH		61
+#define RISCV_ISA_EXT_ZVKT		62
 
 #define RISCV_ISA_EXT_MAX		64
 #define RISCV_ISA_EXT_NAME_LEN_MAX	32
arch/riscv/kernel/cpu.c +8 -0
index 10524322a4c0..925241e25db2 100644
@@ -227,6 +227,14 @@ static struct riscv_isa_ext_data isa_ext_arr[] = {
 	__RISCV_ISA_EXT_DATA(zksed, RISCV_ISA_EXT_ZKSED),
 	__RISCV_ISA_EXT_DATA(zksh, RISCV_ISA_EXT_ZKSH),
 	__RISCV_ISA_EXT_DATA(zkt, RISCV_ISA_EXT_ZKT),
+	__RISCV_ISA_EXT_DATA(zvbb, RISCV_ISA_EXT_ZVBB),
+	__RISCV_ISA_EXT_DATA(zvbc, RISCV_ISA_EXT_ZVBC),
+	__RISCV_ISA_EXT_DATA(zvkg, RISCV_ISA_EXT_ZVKG),
+	__RISCV_ISA_EXT_DATA(zvkned, RISCV_ISA_EXT_ZVKNED),
+	__RISCV_ISA_EXT_DATA(zvknha, RISCV_ISA_EXT_ZVKNHA),
+	__RISCV_ISA_EXT_DATA(zvknhb, RISCV_ISA_EXT_ZVKNHB),
+	__RISCV_ISA_EXT_DATA(zvksed, RISCV_ISA_EXT_ZVKSED),
+	__RISCV_ISA_EXT_DATA(zvksh, RISCV_ISA_EXT_ZVKSH),
 	__RISCV_ISA_EXT_DATA(smaia, RISCV_ISA_EXT_SMAIA),
 	__RISCV_ISA_EXT_DATA(ssaia, RISCV_ISA_EXT_SSAIA),
 	__RISCV_ISA_EXT_DATA(sscofpmf, RISCV_ISA_EXT_SSCOFPMF),
arch/riscv/kernel/cpufeature.c +50 -0

56 lines not on this page · raw message

Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#9 lore raw

Re: [PATCH v4 04/12] RISC-V: add vector crypto extension detection

Quoted 10 lines · Heiko Stuebner

From: Heiko Stuebner <[email protected]>

Add detection for some extensions of the vector-crypto specification:
- Zvkb: Vector Bit-manipulation used in Cryptography
- Zvkg: Vector GCM/GMAC
- Zvknha and Zvknhb: NIST Algorithm Suite
- Zvkns: AES-128, AES-256 Single Round Suite
- Zvksed: ShangMi Algorithm Suite
- Zvksh: ShangMi Algorithm Suite

Any plan to allow user-space to detect these extensions via HWPROBE ?

Regards,
Anup

Quoted 113 lines

As their use is very specific and will likely be limited to special places we expect current code to just pre-encode those instructions, so right now we don't introduce toolchain requirements.

---
 arch/riscv/include/asm/hwcap.h |  9 ++++++
 arch/riscv/kernel/cpu.c        |  8 ++++++
 arch/riscv/kernel/cpufeature.c | 50 ++++++++++++++++++++++++++++++++++
 3 files changed, 67 insertions(+)
arch/riscv/include/asm/hwcap.h +9 -0
index b80ca6e77088..0f5172fa87b0 100644
@@ -64,6 +64,15 @@
 #define RISCV_ISA_EXT_ZKSED            51
 #define RISCV_ISA_EXT_ZKSH             52
 #define RISCV_ISA_EXT_ZKT              53
+#define RISCV_ISA_EXT_ZVBB             54
+#define RISCV_ISA_EXT_ZVBC             55
+#define RISCV_ISA_EXT_ZVKG             56
+#define RISCV_ISA_EXT_ZVKNED           57
+#define RISCV_ISA_EXT_ZVKNHA           58
+#define RISCV_ISA_EXT_ZVKNHB           59
+#define RISCV_ISA_EXT_ZVKSED           60
+#define RISCV_ISA_EXT_ZVKSH            61
+#define RISCV_ISA_EXT_ZVKT             62
 
 #define RISCV_ISA_EXT_MAX              64
 #define RISCV_ISA_EXT_NAME_LEN_MAX     32
arch/riscv/kernel/cpu.c +8 -0
index 10524322a4c0..925241e25db2 100644
@@ -227,6 +227,14 @@ static struct riscv_isa_ext_data isa_ext_arr[] = {
        __RISCV_ISA_EXT_DATA(zksed, RISCV_ISA_EXT_ZKSED),
        __RISCV_ISA_EXT_DATA(zksh, RISCV_ISA_EXT_ZKSH),
        __RISCV_ISA_EXT_DATA(zkt, RISCV_ISA_EXT_ZKT),
+       __RISCV_ISA_EXT_DATA(zvbb, RISCV_ISA_EXT_ZVBB),
+       __RISCV_ISA_EXT_DATA(zvbc, RISCV_ISA_EXT_ZVBC),
+       __RISCV_ISA_EXT_DATA(zvkg, RISCV_ISA_EXT_ZVKG),
+       __RISCV_ISA_EXT_DATA(zvkned, RISCV_ISA_EXT_ZVKNED),
+       __RISCV_ISA_EXT_DATA(zvknha, RISCV_ISA_EXT_ZVKNHA),
+       __RISCV_ISA_EXT_DATA(zvknhb, RISCV_ISA_EXT_ZVKNHB),
+       __RISCV_ISA_EXT_DATA(zvksed, RISCV_ISA_EXT_ZVKSED),
+       __RISCV_ISA_EXT_DATA(zvksh, RISCV_ISA_EXT_ZVKSH),
        __RISCV_ISA_EXT_DATA(smaia, RISCV_ISA_EXT_SMAIA),
        __RISCV_ISA_EXT_DATA(ssaia, RISCV_ISA_EXT_SSAIA),
        __RISCV_ISA_EXT_DATA(sscofpmf, RISCV_ISA_EXT_SSCOFPMF),
arch/riscv/kernel/cpufeature.c +50 -1
index 9a872a2007a5..13556fd16bf6 100644
@@ -343,6 +343,56 @@ void __init riscv_fill_hwcap(void)
                                SET_ISA_EXT_MAP("zksh", RISCV_ISA_EXT_ZKSH);
                                SET_ISA_EXT_MAP("zkr", RISCV_ISA_EXT_ZKR);
                                SET_ISA_EXT_MAP("zkt", RISCV_ISA_EXT_ZKT);
+                               SET_ISA_EXT_MAP("zvbb", RISCV_ISA_EXT_ZVBB);
+                               SET_ISA_EXT_MAP("zvbc", RISCV_ISA_EXT_ZVBC);
+                               SET_ISA_EXT_MAP("zvkg", RISCV_ISA_EXT_ZVKG);
+                               SET_ISA_EXT_MAP("zvkned", RISCV_ISA_EXT_ZVKNED);
+                               SET_ISA_EXT_MAP("zvknha", RISCV_ISA_EXT_ZVKNHA);
+                               SET_ISA_EXT_MAP("zvknhb", RISCV_ISA_EXT_ZVKNHB);
+                               SET_ISA_EXT_MAP("zvksed", RISCV_ISA_EXT_ZVKSED);
+                               SET_ISA_EXT_MAP("zvksh", RISCV_ISA_EXT_ZVKSH);
+                               SET_ISA_EXT_MAP("zvkt", RISCV_ISA_EXT_ZVKT);
+
+                               /* NIST Algorithm Suite */
+                               SET_ISA_EXT_MAP("zvkn", RISCV_ISA_EXT_ZVKNED);
+                               SET_ISA_EXT_MAP("zvkn", RISCV_ISA_EXT_ZVKNHB);
+                               SET_ISA_EXT_MAP("zvkn", RISCV_ISA_EXT_ZVBB);
+                               SET_ISA_EXT_MAP("zvkn", RISCV_ISA_EXT_ZVKT);
+
+                               /* NIST Algorithm Suite with carryless multiply */
+                               SET_ISA_EXT_MAP("zvknc", RISCV_ISA_EXT_ZVKNED);
+                               SET_ISA_EXT_MAP("zvknc", RISCV_ISA_EXT_ZVKNHB);
+                               SET_ISA_EXT_MAP("zvknc", RISCV_ISA_EXT_ZVBB);
+                               SET_ISA_EXT_MAP("zvknc", RISCV_ISA_EXT_ZVKT);
+                               SET_ISA_EXT_MAP("zvknc", RISCV_ISA_EXT_ZVBC);
+
+                               /* NIST Algorithm Suite with GCM */
+                               SET_ISA_EXT_MAP("zvkng", RISCV_ISA_EXT_ZVKNED);
+                               SET_ISA_EXT_MAP("zvkng", RISCV_ISA_EXT_ZVKNHB);
+                               SET_ISA_EXT_MAP("zvkng", RISCV_ISA_EXT_ZVBB);
+                               SET_ISA_EXT_MAP("zvkng", RISCV_ISA_EXT_ZVKT);
+                               SET_ISA_EXT_MAP("zvkng", RISCV_ISA_EXT_ZVKG);
+
+                               /*  ShangMi Algorithm Suite */
+                               SET_ISA_EXT_MAP("zvks", RISCV_ISA_EXT_ZVKSED);
+                               SET_ISA_EXT_MAP("zvks", RISCV_ISA_EXT_ZVKSH);
+                               SET_ISA_EXT_MAP("zvks", RISCV_ISA_EXT_ZVBB);
+                               SET_ISA_EXT_MAP("zvks", RISCV_ISA_EXT_ZVKT);
+
+                               /* ShangMi Algorithm Suite with carryless multiply */
+                               SET_ISA_EXT_MAP("zvksc", RISCV_ISA_EXT_ZVKSED);
+                               SET_ISA_EXT_MAP("zvksc", RISCV_ISA_EXT_ZVKSH);
+                               SET_ISA_EXT_MAP("zvksc", RISCV_ISA_EXT_ZVBB);
+                               SET_ISA_EXT_MAP("zvksc", RISCV_ISA_EXT_ZVKT);
+                               SET_ISA_EXT_MAP("zvksc", RISCV_ISA_EXT_ZVBC);
+
+                               /* ShangMi Algorithm Suite with GCM */
+                               SET_ISA_EXT_MAP("zvksg", RISCV_ISA_EXT_ZVKSED);
+                               SET_ISA_EXT_MAP("zvksg", RISCV_ISA_EXT_ZVKSH);
+                               SET_ISA_EXT_MAP("zvksg", RISCV_ISA_EXT_ZVBB);
+                               SET_ISA_EXT_MAP("zvksg", RISCV_ISA_EXT_ZVKT);
+                               SET_ISA_EXT_MAP("zvksg", RISCV_ISA_EXT_ZVKG);
+
                        }
 #undef SET_ISA_EXT_MAP
                }
--
Quoted 2 lines

2.39.2

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#10 lore raw

Re: [PATCH v4 04/12] RISC-V: add vector crypto extension detection

In reply to Heiko Stuebner

Hey Heiko,

arch/riscv/kernel/cpufeature.c void __init riscv_fill_hwcap(void)

Quoted 65 lines · Heiko Stuebner

From: Heiko Stuebner <[email protected]>

Add detection for some extensions of the vector-crypto specification:
- Zvkb: Vector Bit-manipulation used in Cryptography
- Zvkg: Vector GCM/GMAC
- Zvknha and Zvknhb: NIST Algorithm Suite
- Zvkns: AES-128, AES-256 Single Round Suite
- Zvksed: ShangMi Algorithm Suite
- Zvksh: ShangMi Algorithm Suite

As their use is very specific and will likely be limited to special places we expect current code to just pre-encode those instructions, so right now we don't introduce toolchain requirements.

---
 arch/riscv/include/asm/hwcap.h |  9 ++++++
 arch/riscv/kernel/cpu.c        |  8 ++++++
 arch/riscv/kernel/cpufeature.c | 50 ++++++++++++++++++++++++++++++++++
 3 files changed, 67 insertions(+)
arch/riscv/include/asm/hwcap.h +9 -0
index b80ca6e77088..0f5172fa87b0 100644
@@ -64,6 +64,15 @@
 #define RISCV_ISA_EXT_ZKSED		51
 #define RISCV_ISA_EXT_ZKSH		52
 #define RISCV_ISA_EXT_ZKT		53
+#define RISCV_ISA_EXT_ZVBB		54
+#define RISCV_ISA_EXT_ZVBC		55
+#define RISCV_ISA_EXT_ZVKG		56
+#define RISCV_ISA_EXT_ZVKNED		57
+#define RISCV_ISA_EXT_ZVKNHA		58
+#define RISCV_ISA_EXT_ZVKNHB		59
+#define RISCV_ISA_EXT_ZVKSED		60
+#define RISCV_ISA_EXT_ZVKSH		61
+#define RISCV_ISA_EXT_ZVKT		62
 
 #define RISCV_ISA_EXT_MAX		64
 #define RISCV_ISA_EXT_NAME_LEN_MAX	32
arch/riscv/kernel/cpu.c +8 -0
index 10524322a4c0..925241e25db2 100644
@@ -227,6 +227,14 @@ static struct riscv_isa_ext_data isa_ext_arr[] = {
 	__RISCV_ISA_EXT_DATA(zksed, RISCV_ISA_EXT_ZKSED),
 	__RISCV_ISA_EXT_DATA(zksh, RISCV_ISA_EXT_ZKSH),
 	__RISCV_ISA_EXT_DATA(zkt, RISCV_ISA_EXT_ZKT),
+	__RISCV_ISA_EXT_DATA(zvbb, RISCV_ISA_EXT_ZVBB),
+	__RISCV_ISA_EXT_DATA(zvbc, RISCV_ISA_EXT_ZVBC),
+	__RISCV_ISA_EXT_DATA(zvkg, RISCV_ISA_EXT_ZVKG),
+	__RISCV_ISA_EXT_DATA(zvkned, RISCV_ISA_EXT_ZVKNED),
+	__RISCV_ISA_EXT_DATA(zvknha, RISCV_ISA_EXT_ZVKNHA),
+	__RISCV_ISA_EXT_DATA(zvknhb, RISCV_ISA_EXT_ZVKNHB),
+	__RISCV_ISA_EXT_DATA(zvksed, RISCV_ISA_EXT_ZVKSED),
+	__RISCV_ISA_EXT_DATA(zvksh, RISCV_ISA_EXT_ZVKSH),
 	__RISCV_ISA_EXT_DATA(smaia, RISCV_ISA_EXT_SMAIA),
 	__RISCV_ISA_EXT_DATA(ssaia, RISCV_ISA_EXT_SSAIA),
 	__RISCV_ISA_EXT_DATA(sscofpmf, RISCV_ISA_EXT_SSCOFPMF),
arch/riscv/kernel/cpufeature.c +0 -0
index 9a872a2007a5..13556fd16bf6 100644
@@ -343,6 +343,56 @@ void __init riscv_fill_hwcap(void)

All of these need to be documented in dt-bindings.

At least one of these lists will go away iff Palmer merges my rework of this stuff & hopefully we'll get one of the ways to avoid repeating the SET_ISA_EXT_MAP stuff ad nauseam.

Cheers,
Conor.

#11 lore raw

Re: [PATCH v4 04/12] RISC-V: add vector crypto extension detection

In reply to Heiko Stuebner
Quoted 28 lines · Heiko Stuebner

From: Heiko Stuebner <[email protected]>

Add detection for some extensions of the vector-crypto specification:
- Zvkb: Vector Bit-manipulation used in Cryptography
- Zvkg: Vector GCM/GMAC
- Zvknha and Zvknhb: NIST Algorithm Suite
- Zvkns: AES-128, AES-256 Single Round Suite
- Zvksed: ShangMi Algorithm Suite
- Zvksh: ShangMi Algorithm Suite

As their use is very specific and will likely be limited to special places we expect current code to just pre-encode those instructions, so right now we don't introduce toolchain requirements.

---
 arch/riscv/include/asm/hwcap.h |  9 ++++++
 arch/riscv/kernel/cpu.c        |  8 ++++++
 arch/riscv/kernel/cpufeature.c | 50 ++++++++++++++++++++++++++++++++++
 3 files changed, 67 insertions(+)
arch/riscv/include/asm/hwcap.h +0 -0
index b80ca6e77088..0f5172fa87b0 100644
@@ -64,6 +64,15 @@
 #define RISCV_ISA_EXT_ZKSED		51
 #define RISCV_ISA_EXT_ZKSH		52

arch/riscv/include/asm/hwcap.h line 75

Quoted 10 lines
arch/riscv/include/asm/hwcap.h +9 -0
 #define RISCV_ISA_EXT_ZKT		53
+#define RISCV_ISA_EXT_ZVBB		54
+#define RISCV_ISA_EXT_ZVBC		55
+#define RISCV_ISA_EXT_ZVKG		56
+#define RISCV_ISA_EXT_ZVKNED		57
+#define RISCV_ISA_EXT_ZVKNHA		58
+#define RISCV_ISA_EXT_ZVKNHB		59
+#define RISCV_ISA_EXT_ZVKSED		60
+#define RISCV_ISA_EXT_ZVKSH		61
+#define RISCV_ISA_EXT_ZVKT		62

It would be helpful if each RISCV_ISA_EXT_* definition had a comment that spells out what it stands for, similar to what arch/x86/include/asm/cpufeatures.h does. I know they can all be looked up, and they're sort of mnemonic, but it would be helpful.

- Eric

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#12 lore raw

PATCH v4 5/12 RISC-V: crypto: update perl include with helpers for vector (crypto) instructions

In reply to Heiko Stuebner

From: Heiko Stuebner <[email protected]>

The openSSL scripts use a number of helpers for handling vector instructions and instructions from the vector-crypto-extensions.

Therefore port these over from openSSL.

---
 arch/riscv/crypto/riscv.pm | 433 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 432 insertions(+), 1 deletion(-)
arch/riscv/crypto/riscv.pm +431 -0
index a44edc68d1a6..89e4cb7cfd8c 100644
@@ -79,6 +79,29 @@ sub read_reg {
     return $1;
 }
 
+my @vregs = map("v$_",(0..31));
+my %vreglookup;
+@vreglookup{@vregs} = @vregs;
+
+sub read_vreg {
+    my $vreg = lc shift;
+    if (!exists($vreglookup{$vreg})) {
+        my $trace = "";
+        if ($have_stacktrace) {
+            $trace = Devel::StackTrace->new->as_string;
+        }
+        die("Unknown vector register ".$vreg."\n".$trace);
+    }
+    if (!($vreg =~ /^v([0-9]+)$/)) {
+        my $trace = "";
+        if ($have_stacktrace) {
+            $trace = Devel::StackTrace->new->as_string;
+        }
+        die("Could not process vector register ".$vreg."\n".$trace);
+    }
+    return $1;
+}
+
 # Helper functions
 
 sub brev8_rv64i {
@@ -258,4 +281,412 @@ sub rev8 {
     return ".word ".($template | ($rs << 15) | ($rd << 7));
 }
 
+# Vector instructions
+
+sub vadd_vv {
+    # vadd.vv vd, vs2, vs1
+    my $template = 0b0000001_00000_00000_000_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20) | ($vs1 << 15) | ($vd << 7));
+}
+
+sub vid_v {
+    # vid.v vd
+    my $template = 0b0101001_00000_10001_010_00000_1010111;
+    my $vd = read_vreg shift;
+    return ".word ".($template | ($vd << 7));
+}
+
+sub vle32_v {
+    # vle32.v vd, (rs1)
+    my $template = 0b0000001_00000_00000_110_00000_0000111;
+    my $vd = read_vreg shift;
+    my $rs1 = read_reg shift;
+    return ".word ".($template | ($rs1 << 15) | ($vd << 7));
+}
+
+sub vle64_v {
+    # vle64.v vd, (rs1)
+    my $template = 0b0000001_00000_00000_111_00000_0000111;
+    my $vd = read_vreg shift;
+    my $rs1 = read_reg shift;
+    return ".word ".($template | ($rs1 << 15) | ($vd << 7));
+}
+
+sub vlse32_v {
+    # vlse32.v vd, (rs1), rs2
+    my $template = 0b0000101_00000_00000_110_00000_0000111;
+    my $vd = read_vreg shift;
+    my $rs1 = read_reg shift;
+    my $rs2 = read_reg shift;
+    return ".word ".($template | ($rs2 << 20) | ($rs1 << 15) | ($vd << 7));
+}
+
+sub vlse64_v {
+    # vlse64.v vd, (rs1), rs2
+    my $template = 0b0000101_00000_00000_111_00000_0000111;
+    my $vd = read_vreg shift;
+    my $rs1 = read_reg shift;
+    my $rs2 = read_reg shift;
+    return ".word ".($template | ($rs2 << 20) | ($rs1 << 15) | ($vd << 7));
+}
+
+sub vmerge_vim {
+    # vmerge.vim vd, vs2, imm, v0
+    my $template = 0b0101110_00000_00000_011_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $imm = shift;
+    return ".word ".($template | ($vs2 << 20) | ($imm << 15) | ($vd << 7));
+}
+
+sub vmerge_vvm {
+    # vmerge.vvm vd vs2 vs1
+    my $template = 0b0101110_00000_00000_000_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20) | ($vs1 <<   15) | ($vd << 7))
+}
+
+sub vmseq_vi {
+    # vmseq vd vs1, imm
+    my $template = 0b0110001_00000_00000_011_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    my $imm = shift;
+    return ".word ".($template | ($vs1 << 20) | ($imm <<   15) | ($vd << 7))
+}
+
+sub vmv_v_i {
+    # vmv.v.i vd, imm
+    my $template = 0b0101111_00000_00000_011_00000_1010111;
+    my $vd = read_vreg shift;
+    my $imm = shift;
+    return ".word ".($template | ($imm << 15) | ($vd << 7));
+}
+
+sub vmv_v_v {
+    # vmv.v.v vd, vs1
+    my $template = 0b0101111_00000_00000_000_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    return ".word ".($template | ($vs1 << 15) | ($vd << 7));
+}
+
+sub vor_vv_v0t {
+    # vor.vv vd, vs2, vs1, v0.t
+    my $template = 0b0010100_00000_00000_000_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20) | ($vs1 << 15) | ($vd << 7));
+}
+
+sub vse32_v {
+    # vse32.v vd, (rs1)
+    my $template = 0b0000001_00000_00000_110_00000_0100111;
+    my $vd = read_vreg shift;
+    my $rs1 = read_reg shift;
+    return ".word ".($template | ($rs1 << 15) | ($vd << 7));
+}
+
+sub vse64_v {
+    # vse64.v vd, (rs1)
+    my $template = 0b0000001_00000_00000_111_00000_0100111;
+    my $vd = read_vreg shift;
+    my $rs1 = read_reg shift;
+    return ".word ".($template | ($rs1 << 15) | ($vd << 7));
+}
+
+sub vsetivli__x0_2_e64_m1_ta_ma {
+    # vsetivli x0, 2, e64, m1, ta, ma
+    return ".word 0xcd817057";
+}
+
+sub vsetivli__x0_4_e32_m1_ta_ma {
+    # vsetivli x0, 4, e32, m1, ta, ma
+    return ".word 0xcd027057";
+}
+
+sub vsetivli__x0_4_e64_m1_ta_ma {
+    # vsetivli x0,4,e64,m1,ta,ma
+    return ".word 0xcd827057";
+}
+
+sub vsetivli__x0_8_e32_m1_ta_ma {
+    return ".word 0xcd047057";
+}
+
+sub vslidedown_vi {
+    # vslidedown.vi vd, vs2, uimm
+    my $template = 0b0011111_00000_00000_011_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $uimm = shift;
+    return ".word ".($template | ($vs2 << 20) | ($uimm << 15) | ($vd << 7));
+}
+
+sub vslideup_vi_v0t {
+    # vslideup.vi vd, vs2, uimm, v0.t
+    my $template = 0b0011100_00000_00000_011_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $uimm = shift;
+    return ".word ".($template | ($vs2 << 20) | ($uimm << 15) | ($vd << 7));
+}
+
+sub vslideup_vi {
+    # vslideup.vi vd, vs2, uimm
+    my $template = 0b0011101_00000_00000_011_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $uimm = shift;
+    return ".word ".($template | ($vs2 << 20) | ($uimm << 15) | ($vd << 7));
+}
+
+sub vsll_vi {
+    # vsll.vi vd, vs2, uimm, vm
+    my $template = 0b1001011_00000_00000_011_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $uimm = shift;
+    return ".word ".($template | ($vs2 << 20) | ($uimm << 15) | ($vd << 7));
+}
+
+sub vsrl_vx {
+    # vsrl.vx vd, vs2, rs1
+    my $template = 0b1010001_00000_00000_100_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $rs1 = read_reg shift;
+    return ".word ".($template | ($vs2 << 20) | ($rs1 << 15) | ($vd << 7));
+}
+
+sub vsse32_v {
+    # vse32.v vs3, (rs1), rs2
+    my $template = 0b0000101_00000_00000_110_00000_0100111;
+    my $vs3 = read_vreg shift;
+    my $rs1 = read_reg shift;
+    my $rs2 = read_reg shift;
+    return ".word ".($template | ($rs2 << 20) | ($rs1 << 15) | ($vs3 << 7));
+}
+
+sub vsse64_v {
+    # vsse64.v vs3, (rs1), rs2
+    my $template = 0b0000101_00000_00000_111_00000_0100111;
+    my $vs3 = read_vreg shift;
+    my $rs1 = read_reg shift;
+    my $rs2 = read_reg shift;
+    return ".word ".($template | ($rs2 << 20) | ($rs1 << 15) | ($vs3 << 7));
+}
+
+sub vxor_vv_v0t {
+    # vxor.vv vd, vs2, vs1, v0.t
+    my $template = 0b0010110_00000_00000_000_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20) | ($vs1 << 15) | ($vd << 7));
+}
+
+sub vxor_vv {
+    # vxor.vv vd, vs2, vs1
+    my $template = 0b0010111_00000_00000_000_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20) | ($vs1 << 15) | ($vd << 7));
+}
+
+# Vector crypto instructions
+
+## Zvbb instructions
+
+sub vrev8_v {
+    # vrev8.v vd, vs2
+    my $template = 0b0100101_00000_01001_010_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20) | ($vd << 7));
+}
+
+## Zvbc instructions
+
+sub vclmulh_vx {
+    # vclmulh.vx vd, vs2, rs1
+    my $template = 0b0011011_00000_00000_110_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $rs1 = read_reg shift;
+    return ".word ".($template | ($vs2 << 20) | ($rs1 << 15) | ($vd << 7));
+}
+
+sub vclmul_vx_v0t {
+    # vclmul.vx vd, vs2, rs1, v0.t
+    my $template = 0b0011000_00000_00000_110_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $rs1 = read_reg shift;
+    return ".word ".($template | ($vs2 << 20) | ($rs1 << 15) | ($vd << 7));
+}
+
+sub vclmul_vx {
+    # vclmul.vx vd, vs2, rs1
+    my $template = 0b0011001_00000_00000_110_00000_1010111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $rs1 = read_reg shift;
+    return ".word ".($template | ($vs2 << 20) | ($rs1 << 15) | ($vd << 7));
+}
+
+## Zvkg instructions
+
+sub vghsh_vv {
+    # vghsh.vv vd, vs2, vs1
+    my $template = 0b1011001_00000_00000_010_00000_1110111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20) | ($vs1 << 15) | ($vd << 7));
+}
+
+sub vgmul_vv {
+    # vgmul.vv vd, vs2
+    my $template = 0b1010001_00000_10001_010_00000_1110111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20) | ($vd << 7));
+}
+
+## Zvkned instructions
+
+sub vaesdf_vs {
+    # vaesdf.vs vd, vs2
+    my $template = 0b101001_1_00000_00001_010_00000_1110111;
+    my $vd = read_vreg  shift;
+    my $vs2 = read_vreg  shift;
+    return ".word ".($template | ($vs2 << 20) | ($vd << 7));
+}
+
+sub vaesdm_vs {
+    # vaesdm.vs vd, vs2
+    my $template = 0b101001_1_00000_00000_010_00000_1110111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20) | ($vd << 7));
+}
+
+sub vaesef_vs {
+    # vaesef.vs vd, vs2
+    my $template = 0b101001_1_00000_00011_010_00000_1110111;
+    my $vd = read_vreg  shift;
+    my $vs2 = read_vreg  shift;
+    return ".word ".($template | ($vs2 << 20) | ($vd << 7));
+}
+
+sub vaesem_vs {
+    # vaesem.vs vd, vs2
+    my $template = 0b101001_1_00000_00010_010_00000_1110111;
+    my $vd = read_vreg  shift;
+    my $vs2 = read_vreg  shift;
+    return ".word ".($template | ($vs2 << 20) | ($vd << 7));
+}
+
+sub vaeskf1_vi {
+    # vaeskf1.vi vd, vs2, uimmm
+    my $template = 0b100010_1_00000_00000_010_00000_1110111;
+    my $vd = read_vreg  shift;
+    my $vs2 = read_vreg  shift;
+    my $uimm = shift;
+    return ".word ".($template | ($uimm << 15) | ($vs2 << 20) | ($vd << 7));
+}
+
+sub vaeskf2_vi {
+    # vaeskf2.vi vd, vs2, uimm
+    my $template = 0b101010_1_00000_00000_010_00000_1110111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $uimm = shift;
+    return ".word ".($template | ($vs2 << 20) | ($uimm << 15) | ($vd << 7));
+}
+
+sub vaesz_vs {
+    # vaesz.vs vd, vs2
+    my $template = 0b101001_1_00000_00111_010_00000_1110111;
+    my $vd = read_vreg  shift;
+    my $vs2 = read_vreg  shift;
+    return ".word ".($template | ($vs2 << 20) | ($vd << 7));
+}
+
+## Zvknha and Zvknhb instructions
+
+sub vsha2ms_vv {
+    # vsha2ms.vv vd, vs2, vs1
+    my $template = 0b1011011_00000_00000_010_00000_1110111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20)| ($vs1 << 15 )| ($vd << 7));
+}
+
+sub vsha2ch_vv {
+    # vsha2ch.vv vd, vs2, vs1
+    my $template = 0b101110_10000_00000_001_00000_01110111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20)| ($vs1 << 15 )| ($vd << 7));
+}
+
+sub vsha2cl_vv {
+    # vsha2cl.vv vd, vs2, vs1
+    my $template = 0b101111_10000_00000_001_00000_01110111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20)| ($vs1 << 15 )| ($vd << 7));
+}
+
+## Zvksed instructions
+
+sub vsm4k_vi {
+    # vsm4k.vi vd, vs2, uimm
+    my $template = 0b1000011_00000_00000_010_00000_1110111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $uimm = shift;
+    return ".word ".($template | ($vs2 << 20) | ($uimm << 15) | ($vd << 7));
+}
+
+sub vsm4r_vs {
+    # vsm4r.vs vd, vs2
+    my $template = 0b1010011_00000_10000_010_00000_1110111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20) | ($vd << 7));
+}
+
+## zvksh instructions
+
+sub vsm3c_vi {
+    # vsm3c.vi vd, vs2, uimm
+    my $template = 0b1010111_00000_00000_010_00000_1110111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $uimm = shift;
+    return ".word ".($template | ($vs2 << 20) | ($uimm << 15 ) | ($vd << 7));
+}
+
+sub vsm3me_vv {
+    # vsm3me.vv vd, vs2, vs1
+    my $template = 0b1000001_00000_00000_010_00000_1110111;
+    my $vd = read_vreg shift;
+    my $vs2 = read_vreg shift;
+    my $vs1 = read_vreg shift;
+    return ".word ".($template | ($vs2 << 20) | ($vs1 << 15 ) | ($vd << 7));
+}
+
 1;
Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#13 lore raw

Re: [PATCH v4 05/12] RISC-V: crypto: update perl include with helpers for vector (crypto) instructions

Quoted 4 lines · Le tiistaina 11. heinäkuuta 2023, 18.37.36 EEST Heiko Stuebner a écrit :

From: Heiko Stuebner <[email protected]>

The openSSL scripts use a number of helpers for handling vector instructions and instructions from the vector-crypto-extensions.

Uh but the kernel RVV code requires an assembler that supports the .option arch directive and the V extension anyway.

Is there a need to wrap vector load/store and ALU instructions from the Vector spec? This trick should only be necessary for the Zvk*-specific stuff, AFAICT.

(Also FWIW, this can be done directly with .macro inside a header file, without involving Perl.)

Signature · 10 lines
Rémi Denis-Courmont
http://www.remlab.net/




_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#14 lore raw

PATCH v4 6/12 RISC-V: crypto: add Zvbb+Zvbc accelerated GCM GHASH implementation

In reply to Heiko Stuebner

From: Heiko Stuebner <[email protected]>

Add a gcm hash implementation using the Zvbb+Zvbc crypto extensions. It gets possibly registered alongside the Zbc-based variant, with a higher priority so that the crypto subsystem will be able to select the most performant variant, but the algorithm itself will still be part of the crypto selftests that run during registration.

---
 arch/riscv/crypto/Kconfig                    |   3 +-
 arch/riscv/crypto/Makefile                   |   8 +-
 arch/riscv/crypto/ghash-riscv64-glue.c       | 150 ++++++++
 arch/riscv/crypto/ghash-riscv64-zvbb-zvbc.pl | 380 +++++++++++++++++++
 4 files changed, 539 insertions(+), 2 deletions(-)
 create mode 100644 arch/riscv/crypto/ghash-riscv64-zvbb-zvbc.pl
arch/riscv/crypto/Kconfig +2 -1
index cd2237923e68..41b8fdfe1d92 100644
@@ -4,7 +4,7 @@ menu "Accelerated Cryptographic Algorithms for CPU (riscv)"
 
 config CRYPTO_GHASH_RISCV64
 	tristate "Hash functions: GHASH"
-	depends on 64BIT && RISCV_ISA_ZBC
+	depends on 64BIT && (RISCV_ISA_ZBC || RISCV_ISA_V)
 	select CRYPTO_HASH
 	select CRYPTO_LIB_GF128MUL
 	help
@@ -14,5 +14,6 @@ config CRYPTO_GHASH_RISCV64
 	  - Zbc extension
 	  - Zbc + Zbb extensions
 	  - Zbc + Zbkb extensions
+	  - Zvbb vector crypto extension
 
 endmenu
arch/riscv/crypto/Makefile +7 -1
index 0a158919e9da..81190941ba78 100644
@@ -8,6 +8,9 @@ ghash-riscv64-y := ghash-riscv64-glue.o
 ifdef CONFIG_RISCV_ISA_ZBC
 ghash-riscv64-y += ghash-riscv64-zbc.o
 endif
+ifdef CONFIG_RISCV_ISA_V
+ghash-riscv64-y += ghash-riscv64-zvbb-zvbc.o
+endif
 
 quiet_cmd_perlasm = PERLASM $@
       cmd_perlasm = $(PERL) $(<) void $(@)
@@ -15,4 +18,7 @@ quiet_cmd_perlasm = PERLASM $@
 $(obj)/ghash-riscv64-zbc.S: $(src)/ghash-riscv64-zbc.pl
 	$(call cmd,perlasm)
 
-clean-files += ghash-riscv64-zbc.S
+$(obj)/ghash-riscv64-zvbb-zvbc.S: $(src)/ghash-riscv64-zvbb-zvbc.pl
+	$(call cmd,perlasm)
+
+clean-files += ghash-riscv64-zbc.S ghash-riscv64-zvkb.S
arch/riscv/crypto/ghash-riscv64-glue.c +150 -0

174 lines not on this page · raw message

arch/riscv/crypto/ghash-riscv64-zvbb-zvbc.pl new file +380 -0

380 lines not on this page · raw message

Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#15 lore raw

Re: [PATCH v4 06/12] RISC-V: crypto: add Zvbb+Zvbc accelerated GCM GHASH implementation

On Tue, Jul 11, 2023 at 05:37:37PM +0200, Heiko Stuebner wrote:
Hi Heiko,

Quoted 8 lines

From: Heiko Stuebner <[email protected]>

Add a gcm hash implementation using the Zvbb+Zvbc crypto extensions. It gets possibly registered alongside the Zbc-based variant, with a higher priority so that the crypto subsystem will be able to select the most performant variant, but the algorithm itself will still be part of the crypto selftests that run during registration.

All newly added crypto algorithms are passing on my side, except for this one. I was testing on a QEMU and toolchain that support the frozen spec.

It seems like it was failing on a small 16-Byte input. Here are the input, expected digest and the (mismatched-)result.

(gdb) x/2gx vec->key
0xffffffff8163cc38: 0x03db81ed4dbfa6df 0x61f030f895ffcaff
(gdb) x/2gx vec->plaintext
0xffffffff8163cc50: 0xc04a60a5562a2b95 0xb6405ba056662bb3
(gdb) x/2gx vec->digest
0xffffffff8163cc68: 0xb65bc5d20aeb53da 0x60dafec32c80c44f
(gdb) x/2gx result
0xff20000000943bb8: 0x0000000000000000 0xb18de0d5e7abcf10

And here is the bootlog, do you have any idea?

[    5.007043] alg: shash: riscv64_zvbb_zvbc_ghash test failed (wrong result) on test vector 0, cfg="init+update+final aligned buffer"
[    5.008164] alg: self-tests for ghash using riscv64_zvbb_zvbc_ghash failed (rc=-22)
[    5.008450] ------------[ cut here ]------------
[    5.009226] alg: self-tests for ghash using riscv64_zvbb_zvbc_ghash failed (rc=-22)
[    5.010678] WARNING: CPU: 1 PID: 87 at crypto/testmgr.c:5867 alg_test+0x3e2/0x41e
[    5.011792] Modules linked in:
[    5.013314] CPU: 1 PID: 87 Comm: cryptomgr_test Not tainted 6.2.2-02529-g4b0fb43edd0f-dirty #37
[    5.014037] Hardware name: riscv-virtio,qemu (DT)
[    5.014582] epc : alg_test+0x3e2/0x41e
[    5.014938]  ra : alg_test+0x3e2/0x41e
[    5.015256] epc : ffffffff80677744 ra : ffffffff80677744 sp : ff2000000095bd70
[    5.015718]  gp : ffffffff81c896b8 tp : ff6000000464d280 t0 : ffffffff81a2c970
[    5.016171]  t1 : ffffffffffffffff t2 : 2d2d2d2d2d2d2d2d s0 : ff2000000095be80
[    5.016616]  s1 : ffffffffffffffea a0 : 0000000000000047 a1 : ffffffff81a97c70
[    5.017078]  a2 : 0000000000000010 a3 : fffffffffffffffe a4 : 0000000000000000
[    5.017582]  a5 : 0000000000000000 a6 : 0000000000000000 a7 : 0000000000000000
[    5.018079]  s2 : 000000000000000e s3 : ff60000002adf200 s4 : ff60000002adf280
[    5.018576]  s5 : 0000000000000171 s6 : 00000000000000b8 s7 : 0000000000000088
[    5.019083]  s8 : ffffffffffffffff s9 : 00000000000000b8 s10: 0000000000002e00
[    5.019584]  s11: ffffffff8127fd78 t3 : ffffffff81ca0f17 t4 : ffffffff81ca0f17
[    5.020074]  t5 : ffffffff81ca0f18 t6 : ff2000000095bb88
[    5.020455] status: 0000000200000120 badaddr: 0000000000000000 cause: 0000000000000003
[    5.021234] [<ffffffff80677744>] alg_test+0x3e2/0x41e
[    5.021906] [<ffffffff8067490e>] cryptomgr_test+0x28/0x4a
[    5.022306] [<ffffffff80055ba0>] kthread+0xe0/0xf6
[    5.022710] [<ffffffff80003edc>] ret_from_exception+0x0/0x16
[    5.023755] ---[ end trace 0000000000000000 ]---

Thanks,
Andy

#16 lore raw

PATCH v4 7/12 RISC-V: crypto: add Zvkg accelerated GCM GHASH implementation

In reply to Heiko Stuebner

From: Heiko Stuebner <[email protected]>

When the Zvkg vector crypto extension is available another optimized gcm ghash variant is possible, so add it as another implementation.

---
 arch/riscv/crypto/Kconfig               |   1 +
 arch/riscv/crypto/Makefile              |   7 +-
 arch/riscv/crypto/ghash-riscv64-glue.c  |  95 ++++++++++++++
 arch/riscv/crypto/ghash-riscv64-zvkg.pl | 168 ++++++++++++++++++++++++
 4 files changed, 269 insertions(+), 2 deletions(-)
 create mode 100644 arch/riscv/crypto/ghash-riscv64-zvkg.pl
arch/riscv/crypto/Kconfig +1 -0
index 41b8fdfe1d92..a1493b556993 100644
@@ -15,5 +15,6 @@ config CRYPTO_GHASH_RISCV64
 	  - Zbc + Zbb extensions
 	  - Zbc + Zbkb extensions
 	  - Zvbb vector crypto extension
+	  - Zvkg vector crypto extension
 
 endmenu
arch/riscv/crypto/Makefile +5 -2
index 81190941ba78..496e784984cc 100644
@@ -9,7 +9,7 @@ ifdef CONFIG_RISCV_ISA_ZBC
 ghash-riscv64-y += ghash-riscv64-zbc.o
 endif
 ifdef CONFIG_RISCV_ISA_V
-ghash-riscv64-y += ghash-riscv64-zvbb-zvbc.o
+ghash-riscv64-y += ghash-riscv64-zvbb-zvbc.o ghash-riscv64-zvkg.o
 endif
 
 quiet_cmd_perlasm = PERLASM $@
@@ -21,4 +21,7 @@ $(obj)/ghash-riscv64-zbc.S: $(src)/ghash-riscv64-zbc.pl
 $(obj)/ghash-riscv64-zvbb-zvbc.S: $(src)/ghash-riscv64-zvbb-zvbc.pl
 	$(call cmd,perlasm)
 
-clean-files += ghash-riscv64-zbc.S ghash-riscv64-zvkb.S
+$(obj)/ghash-riscv64-zvkg.S: $(src)/ghash-riscv64-zvkg.pl
+	$(call cmd,perlasm)
+
+clean-files += ghash-riscv64-zbc.S ghash-riscv64-zvkb.S ghash-riscv64-zvkg.S
arch/riscv/crypto/ghash-riscv64-glue.c +95 -0

122 lines not on this page · raw message

arch/riscv/crypto/ghash-riscv64-zvkg.pl new file +168 -0

168 lines not on this page · raw message

Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#17 lore raw

PATCH v4 8/12 RISC-V: crypto: add a vector-crypto-accelerated SHA256 implementation

In reply to Heiko Stuebner

From: Heiko Stuebner <[email protected]>

This adds an accelerated SHA256 algorithm using either the Zvknha or Zvknhb vector crypto extensions. The spec says that

Zvknhb supports SHA-256 and SHA-512. Zvknha supports only SHA-256.

so the relevant acclerating instructions are included in both.

---
 arch/riscv/crypto/Kconfig                     |  11 +
 arch/riscv/crypto/Makefile                    |   7 +
 arch/riscv/crypto/sha256-riscv64-glue.c       | 115 +++++++
 .../crypto/sha256-riscv64-zvbb-zvknha.pl      | 314 ++++++++++++++++++
 4 files changed, 447 insertions(+)
 create mode 100644 arch/riscv/crypto/sha256-riscv64-glue.c
 create mode 100644 arch/riscv/crypto/sha256-riscv64-zvbb-zvknha.pl
arch/riscv/crypto/Kconfig +11 -0
index a1493b556993..860919d230aa 100644
@@ -17,4 +17,15 @@ config CRYPTO_GHASH_RISCV64
 	  - Zvbb vector crypto extension
 	  - Zvkg vector crypto extension
 
+config CRYPTO_SHA256_RISCV64
+	tristate "Hash functions: SHA-256"
+	depends on 64BIT && RISCV_ISA_V
+	select CRYPTO_HASH
+	select CRYPTO_LIB_SHA256
+	help
+	  SHA-256 secure hash algorithm (FIPS 180)
+
+	  Architecture: riscv64 using
+	  - Zvknha or Zvknhb vector crypto extensions
+
 endmenu
arch/riscv/crypto/Makefile +7 -0
index 496e784984cc..cae2f255ceae 100644
@@ -12,6 +12,9 @@ ifdef CONFIG_RISCV_ISA_V
 ghash-riscv64-y += ghash-riscv64-zvbb-zvbc.o ghash-riscv64-zvkg.o
 endif
 
+obj-$(CONFIG_CRYPTO_SHA256_RISCV64) += sha256-riscv64.o
+sha256-riscv64-y := sha256-riscv64-glue.o sha256-riscv64-zvbb-zvknha.o
+
 quiet_cmd_perlasm = PERLASM $@
       cmd_perlasm = $(PERL) $(<) void $(@)
 
@@ -24,4 +27,8 @@ $(obj)/ghash-riscv64-zvbb-zvbc.S: $(src)/ghash-riscv64-zvbb-zvbc.pl
 $(obj)/ghash-riscv64-zvkg.S: $(src)/ghash-riscv64-zvkg.pl
 	$(call cmd,perlasm)
 
+$(obj)/sha256-riscv64-zvbb-zvknha.S: $(src)/sha256-riscv64-zvbb-zvknha.pl
+	$(call cmd,perlasm)
+
 clean-files += ghash-riscv64-zbc.S ghash-riscv64-zvkb.S ghash-riscv64-zvkg.S
+clean-files += sha256-riscv64-zvknha.S
arch/riscv/crypto/sha256-riscv64-glue.c new file +115 -0

115 lines not on this page · raw message

arch/riscv/crypto/sha256-riscv64-zvbb-zvknha.pl new file +314 -0

314 lines not on this page · raw message

Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#18 lore raw

Re: [PATCH v4 08/12] RISC-V: crypto: add a vector-crypto-accelerated SHA256 implementation

Quoted 21 lines · Heiko Stuebner
arch/riscv/crypto/sha256-riscv64-glue.c new file +15 -0
new file mode 100644
index 000000000000..1c9c88029f60
@@ -0,0 +1,115 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Linux/riscv64 port of the OpenSSL SHA256 implementation for RISCV64
+ *
+ * Copyright (C) 2022 VRULL GmbH
+ * Author: Heiko Stuebner <[email protected]>
+ */
+
+#include <linux/module.h>
+#include <linux/types.h>
+#include <asm/simd.h>
+#include <asm/vector.h>
+#include <crypto/internal/hash.h>
+#include <crypto/internal/simd.h>
+#include <crypto/sha2.h>

arch/riscv/crypto/sha256-riscv64-glue.c line 25

Quoted 10 lines
arch/riscv/crypto/sha256-riscv64-glue.c new file +10 -0
+#include <crypto/sha256_base.h>
+
+asmlinkage void sha256_block_data_order_zvbb_zvknha(u32 *digest, const void *data,
+					unsigned int num_blks);
+
+static void __sha256_block_data_order(struct sha256_state *sst, u8 const *src,
+				      int blocks)
+{
+	sha256_block_data_order_zvbb_zvknha(sst->state, src, blocks);
+}

Having a double-underscored function wrap around a non-underscored one like this isn't conventional for Linux kernel code. IIRC some of the other crypto code happens to do this, but it really is supposed to be the other way around.

I think you should just declare the assembly function to take a 'struct sha256_state', with a comment mentioning that only the 'u32 state[8]' at the beginning is actually used. That's what arch/x86/crypto/sha256_ssse3_glue.c does, for example. Then, __sha256_block_data_order() would be unneeded.

arch/riscv/crypto/sha256-riscv64-glue.c line 30

Quoted 4 lines
arch/riscv/crypto/sha256-riscv64-glue.c new file +4 -0
+static int riscv64_sha256_update(struct shash_desc *desc, const u8 *data,
+			 unsigned int len)
+{
+	if (crypto_simd_usable()) {

crypto_simd_usable() uses may_use_simd() which isn't wired up for RISC-V, so it gets the default implementation of '!in_interrupt()'. RISC-V does have may_use_vector() which looks like right thing. I think RISC-V needs a header arch/riscv/include/asm/simd.h which defines may_use_simd() as a wrapper around may_use_vector().

Quoted 11 lines
arch/riscv/crypto/sha256-riscv64-glue.c new file +11 -0
+		int ret;
+
+		kernel_rvv_begin();
+		ret = sha256_base_do_update(desc, data, len,
+					    __sha256_block_data_order);
+		kernel_rvv_end();
+		return ret;
+	} else {
+		sha256_update(shash_desc_ctx(desc), data, len);
+		return 0;
+	}

arch/riscv/crypto/sha256-riscv64-glue.c line 51

Quoted 10 lines
arch/riscv/crypto/sha256-riscv64-glue.c new file +10 -0
+}
+
+static int riscv64_sha256_finup(struct shash_desc *desc, const u8 *data,
+			unsigned int len, u8 *out)
+{
+	if (!crypto_simd_usable()) {
+		sha256_update(shash_desc_ctx(desc), data, len);
+		sha256_final(shash_desc_ctx(desc), out);
+		return 0;
+	}

Keep things consistent please. riscv64_sha256_update() could use !crypto_simd_usable() and an early return too.

arch/riscv/crypto/sha256-riscv64-glue.c line 83

Quoted 1 line
arch/riscv/crypto/sha256-riscv64-glue.c new file +1 -0
+static int __init sha256_mod_init(void)

riscv64_sha256_mod_init()

Quoted 6 lines
arch/riscv/crypto/sha256-riscv64-glue.c new file +6 -0
+{
+	/*
+	 * From the spec:
+	 * Zvknhb supports SHA-256 and SHA-512. Zvknha supports only SHA-256.
+	 */
+	if ((riscv_isa_extension_available(NULL, ZVKNHA) ||

arch/riscv/crypto/sha256-riscv64-glue.c line 99

Quoted 10 lines
arch/riscv/crypto/sha256-riscv64-glue.c new file +10 -0
+	     riscv_isa_extension_available(NULL, ZVKNHB)) &&
+	     riscv_isa_extension_available(NULL, ZVBB) &&
+	     riscv_vector_vlen() >= 128)
+
+		return crypto_register_shash(&sha256_alg);
+
+	return 0;
+}
+
+static void __exit sha256_mod_fini(void)

riscv64_sha256_mod_exit()

arch/riscv/crypto/sha256-riscv64-glue.c line 106

Quoted 7 lines
arch/riscv/crypto/sha256-riscv64-glue.c new file +7 -0
+{
+	if ((riscv_isa_extension_available(NULL, ZVKNHA) ||
+	     riscv_isa_extension_available(NULL, ZVKNHB)) &&
+	     riscv_isa_extension_available(NULL, ZVBB) &&
+	     riscv_vector_vlen() >= 128)
+		crypto_unregister_shash(&sha256_alg);
+}

If the needed CPU features aren't present, return -ENODEV from the module_init function instead of 0. Then, the module_exit function can unconditionally unregister the algorithm.

- Eric

#19 lore raw

PATCH v4 9/12 RISC-V: crypto: add a vector-crypto-accelerated SHA512 implementation

In reply to Heiko Stuebner

From: Heiko Stuebner <[email protected]>

This adds an accelerated SHA512 algorithm using either the Zvknhb vector crypto extension.

---
 arch/riscv/crypto/Kconfig                     |  11 +
 arch/riscv/crypto/Makefile                    |   8 +-
 arch/riscv/crypto/sha512-riscv64-glue.c       | 106 +++++
 .../crypto/sha512-riscv64-zvbb-zvknhb.pl      | 377 ++++++++++++++++++
 4 files changed, 501 insertions(+), 1 deletion(-)
 create mode 100644 arch/riscv/crypto/sha512-riscv64-glue.c
 create mode 100644 arch/riscv/crypto/sha512-riscv64-zvbb-zvknhb.pl
arch/riscv/crypto/Kconfig +11 -0
index 860919d230aa..e564f861d95e 100644
@@ -28,4 +28,15 @@ config CRYPTO_SHA256_RISCV64
 	  Architecture: riscv64 using
 	  - Zvknha or Zvknhb vector crypto extensions
 
+config CRYPTO_SHA512_RISCV64
+	tristate "Hash functions: SHA-512"
+	depends on 64BIT && RISCV_ISA_V
+	select CRYPTO_HASH
+	select CRYPTO_SHA512
+	help
+	  SHA-512 secure hash algorithm (FIPS 180)
+
+	  Architecture: riscv64
+	  - Zvknhb vector crypto extension
+
 endmenu
arch/riscv/crypto/Makefile +7 -1
index cae2f255ceae..b12c925172db 100644
@@ -15,6 +15,9 @@ endif
 obj-$(CONFIG_CRYPTO_SHA256_RISCV64) += sha256-riscv64.o
 sha256-riscv64-y := sha256-riscv64-glue.o sha256-riscv64-zvbb-zvknha.o
 
+obj-$(CONFIG_CRYPTO_SHA512_RISCV64) += sha512-riscv64.o
+sha512-riscv64-y := sha512-riscv64-glue.o sha512-riscv64-zvbb-zvknhb.o
+
 quiet_cmd_perlasm = PERLASM $@
       cmd_perlasm = $(PERL) $(<) void $(@)
 
@@ -30,5 +33,8 @@ $(obj)/ghash-riscv64-zvkg.S: $(src)/ghash-riscv64-zvkg.pl
 $(obj)/sha256-riscv64-zvbb-zvknha.S: $(src)/sha256-riscv64-zvbb-zvknha.pl
 	$(call cmd,perlasm)
 
+$(obj)/sha512-riscv64-zvbb-zvknhb.S: $(src)/sha512-riscv64-zvbb-zvknhb.pl
+	$(call cmd,perlasm)
+
 clean-files += ghash-riscv64-zbc.S ghash-riscv64-zvkb.S ghash-riscv64-zvkg.S
-clean-files += sha256-riscv64-zvknha.S
+clean-files += sha256-riscv64-zvknha.S sha512-riscv64-zvknhb.S
arch/riscv/crypto/sha512-riscv64-glue.c new file +106 -0

106 lines not on this page · raw message

arch/riscv/crypto/sha512-riscv64-zvbb-zvknhb.pl new file +377 -0

377 lines not on this page · raw message

Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#20 lore raw

PATCH v4 10/12 RISC-V: crypto: add Zvkned accelerated AES encryption implementation

In reply to Heiko Stuebner

From: Heiko Stuebner <[email protected]>

This adds an AES implementation using the Zvkned vector crypto instructions.

---
 arch/riscv/crypto/Kconfig               |  14 +
 arch/riscv/crypto/Makefile              |   7 +
 arch/riscv/crypto/aes-riscv-glue.c      | 168 ++++++++
 arch/riscv/crypto/aes-riscv64-zvkned.pl | 530 ++++++++++++++++++++++++
 4 files changed, 719 insertions(+)
 create mode 100644 arch/riscv/crypto/aes-riscv-glue.c
 create mode 100644 arch/riscv/crypto/aes-riscv64-zvkned.pl
arch/riscv/crypto/Kconfig +14 -0
index e564f861d95e..8579ce43546d 100644
@@ -2,6 +2,20 @@
 
 menu "Accelerated Cryptographic Algorithms for CPU (riscv)"
 
+config CRYPTO_AES_RISCV
+	tristate "Ciphers: AES (RISCV)"
+	depends on 64BIT && RISCV_ISA_V
+	select CRYPTO_AES
+	help
+	  Block ciphers: AES cipher algorithms (FIPS-197)
+	  Length-preserving ciphers: AES with ECB, CBC, CTR, CTS,
+	    XCTR, and XTS modes
+	  AEAD cipher: AES with CBC, ESSIV, and SHA-256
+	    for fscrypt and dm-crypt
+
+	  Architecture: riscv using one of
+	  - Zvkns
+
 config CRYPTO_GHASH_RISCV64
 	tristate "Hash functions: GHASH"
 	depends on 64BIT && (RISCV_ISA_ZBC || RISCV_ISA_V)
arch/riscv/crypto/Makefile +7 -0
index b12c925172db..38ee741a9777 100644
@@ -3,6 +3,9 @@
 # linux/arch/riscv/crypto/Makefile
 #
 
+obj-$(CONFIG_CRYPTO_AES_RISCV) += aes-riscv.o
+aes-riscv-y := aes-riscv-glue.o aes-riscv64-zvkned.o
+
 obj-$(CONFIG_CRYPTO_GHASH_RISCV64) += ghash-riscv64.o
 ghash-riscv64-y := ghash-riscv64-glue.o
 ifdef CONFIG_RISCV_ISA_ZBC
@@ -21,6 +24,9 @@ sha512-riscv64-y := sha512-riscv64-glue.o sha512-riscv64-zvbb-zvknhb.o
 quiet_cmd_perlasm = PERLASM $@
       cmd_perlasm = $(PERL) $(<) void $(@)
 
+$(obj)/aes-riscv64-zvkned.S: $(src)/aes-riscv64-zvkned.pl
+	$(call cmd,perlasm)
+
 $(obj)/ghash-riscv64-zbc.S: $(src)/ghash-riscv64-zbc.pl
 	$(call cmd,perlasm)
 
@@ -36,5 +42,6 @@ $(obj)/sha256-riscv64-zvbb-zvknha.S: $(src)/sha256-riscv64-zvbb-zvknha.pl
 $(obj)/sha512-riscv64-zvbb-zvknhb.S: $(src)/sha512-riscv64-zvbb-zvknhb.pl
 	$(call cmd,perlasm)
 
+clean-files += aes-riscv64-zvkned.S
 clean-files += ghash-riscv64-zbc.S ghash-riscv64-zvkb.S ghash-riscv64-zvkg.S
 clean-files += sha256-riscv64-zvknha.S sha512-riscv64-zvknhb.S
arch/riscv/crypto/aes-riscv-glue.c new file +168 -0

168 lines not on this page · raw message

arch/riscv/crypto/aes-riscv64-zvkned.pl new file +530 -0

530 lines not on this page · raw message

Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#21 lore raw

Re: [PATCH v4 10/12] RISC-V: crypto: add Zvkned accelerated AES encryption implementation

arch/riscv/crypto/Kconfig line 17

Quoted 13 lines · Heiko Stuebner
arch/riscv/crypto/Kconfig +13 -0
+config CRYPTO_AES_RISCV
+	tristate "Ciphers: AES (RISCV)"
+	depends on 64BIT && RISCV_ISA_V
+	select CRYPTO_AES
+	help
+	  Block ciphers: AES cipher algorithms (FIPS-197)
+	  Length-preserving ciphers: AES with ECB, CBC, CTR, CTS,
+	    XCTR, and XTS modes
+	  AEAD cipher: AES with CBC, ESSIV, and SHA-256
+	    for fscrypt and dm-crypt
+
+	  Architecture: riscv using one of
+	  - Zvkns

I'm looking forward to having direct support for these AES modes, especially the modes needed for storage encryption: XTS, and CBC or CTS! None of these AES modes is actually implemented in this patch yet, though, so they can't be claimed in the kconfig help text yet. This patch is just a starting point, as it just adds support for the bare AES block cipher ("aes" in the crypto API).

(BTW, I'm much more interested in, say, AES-XTS support than SM4 support, which this patchset does include. SM4 is a "national pride cipher" which is somewhat of a niche thing. I suppose there are already people pushing it for RISC-V though, as they are everywhere else, so that's to be expected...)

Quoted 34 lines
arch/riscv/crypto/aes-riscv-glue.c new file +28 -0
new file mode 100644
index 000000000000..85e1187aee22
@@ -0,0 +1,168 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Linux/riscv port of the OpenSSL AES implementation for RISCV
+ *
+ * Copyright (C) 2023 VRULL GmbH
+ * Author: Heiko Stuebner <[email protected]>
+ */
+
+#include <linux/crypto.h>
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/module.h>
+#include <linux/types.h>
+#include <asm/simd.h>
+#include <asm/vector.h>
+#include <crypto/aes.h>
+#include <crypto/internal/cipher.h>
+#include <crypto/internal/simd.h>
+
+struct aes_key {
+	u8 key[AES_MAX_KEYLENGTH];
+	int rounds;
+};
+
+/* variant using the zvkned vector crypto extension */
+void rv64i_zvkned_encrypt(const u8 *in, u8 *out, const struct aes_key *key);
+void rv64i_zvkned_decrypt(const u8 *in, u8 *out, const struct aes_key *key);
+int rv64i_zvkned_set_encrypt_key(const u8 *userKey, const int bits,

arch/riscv/crypto/aes-riscv-glue.c line 38

Quoted 10 lines
arch/riscv/crypto/aes-riscv-glue.c new file +10 -0
+				struct aes_key *key);
+int rv64i_zvkned_set_decrypt_key(const u8 *userKey, const int bits,
+				struct aes_key *key);
+
+struct riscv_aes_ctx {
+	struct crypto_cipher *fallback;
+	struct aes_key enc_key;
+	struct aes_key dec_key;
+	unsigned int keylen;
+};

Can it just use 'struct crypto_aes_ctx'? That's what most of the other AES implementations use.

Quoted 48 lines
arch/riscv/crypto/aes-riscv-glue.c new file +48 -0
+static int riscv64_aes_init_zvkned(struct crypto_tfm *tfm)
+{
+	struct riscv_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+	const char *alg = crypto_tfm_alg_name(tfm);
+	struct crypto_cipher *fallback;
+
+	fallback = crypto_alloc_cipher(alg, 0, CRYPTO_ALG_NEED_FALLBACK);
+	if (IS_ERR(fallback)) {
+		pr_err("Failed to allocate transformation for '%s': %ld\n",
+		       alg, PTR_ERR(fallback));
+		return PTR_ERR(fallback);
+	}
+
+	crypto_cipher_set_flags(fallback,
+				crypto_cipher_get_flags((struct
+							 crypto_cipher *)
+							tfm));
+	ctx->fallback = fallback;
+
+	return 0;
+}
+
+static void riscv_aes_exit(struct crypto_tfm *tfm)
+{
+	struct riscv_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+
+	if (ctx->fallback) {
+		crypto_free_cipher(ctx->fallback);
+		ctx->fallback = NULL;
+	}
+}
+
+static int riscv64_aes_setkey_zvkned(struct crypto_tfm *tfm, const u8 *key,
+			 unsigned int keylen)
+{
+	struct riscv_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+	int ret;
+
+	ctx->keylen = keylen;
+
+	if (keylen == 16 || keylen == 32) {
+		kernel_rvv_begin();
+		ret = rv64i_zvkned_set_encrypt_key(key, keylen * 8, &ctx->enc_key);
+		if (ret != 1) {
+			kernel_rvv_end();
+			return -EINVAL;
+		}
+

arch/riscv/crypto/aes-riscv-glue.c line 97

Quoted 10 lines
arch/riscv/crypto/aes-riscv-glue.c new file +10 -0
+		ret = rv64i_zvkned_set_decrypt_key(key, keylen * 8, &ctx->dec_key);
+		kernel_rvv_end();
+		if (ret != 1)
+			return -EINVAL;
+	}
+
+	ret = crypto_cipher_setkey(ctx->fallback, key, keylen);
+
+	return ret ? -EINVAL : 0;
+}

It's a bit annoying that RISC-V doesn't support AES-192, though also not particularly surprising, seeing as AES-192 is almost never used. (Intel's Key Locker, for example, is another recent CPU feature that doesn't support AES-192.) IMO the issue here is really with the kernel crypto API -- it should treat AES-128, AES-192, and AES-256 as separate algorithms so that implementations aren't forced to support all three key sizes...

Anyway, for now, as you noticed you do need a fallback to handle AES-192 to make the kernel crypto API happy.

But, the fallback doesn't have to be a 'crypto_cipher' as you've implemented. You could just use the AES library. See what arch/arm64/crypto/aes-ce-glue.c does, for example. Have you considered that? It would be simpler than the crypto_cipher based approach.

arch/riscv/crypto/aes-riscv-glue.c line 101

Quoted 4 lines
arch/riscv/crypto/aes-riscv-glue.c new file +4 -0
+
+static void riscv64_aes_encrypt_zvkned(struct crypto_tfm *tfm, u8 *dst, const u8 *src)
+{
+	struct riscv_aes_ctx *ctx = crypto_tfm_ctx(tfm);

Always use 'const' for the tfm_ctx in encrypt and decrypt functions, please, as it must never be modified there.

arch/riscv/crypto/aes-riscv-glue.c line 125

Quoted 1 line
arch/riscv/crypto/aes-riscv-glue.c new file +1 -0
+struct crypto_alg riscv64_aes_zvkned_alg = {

static

arch/riscv/crypto/aes-riscv-glue.c line 130

Quoted 1 line
arch/riscv/crypto/aes-riscv-glue.c new file +1 -0
+	.cra_type = NULL,

Omit that line

arch/riscv/crypto/aes-riscv-glue.c line 132

Quoted 1 line
arch/riscv/crypto/aes-riscv-glue.c new file +1 -0
+	.cra_alignmask = 0,

Omit that line

arch/riscv/crypto/aes-riscv-glue.c line 165

Quoted 1 line
arch/riscv/crypto/aes-riscv-glue.c new file +1 -0
+MODULE_DESCRIPTION("AES (accelerated)");

Maybe "RISC-V accelerated"?

- Eric

#22 lore raw

Re: [PATCH v4 10/12] RISC-V: crypto: add Zvkned accelerated AES encryption implementation

Quoted 2 lines · Eric Biggers

On Tue, Jul 11, 2023 at 05:37:41PM +0200, Heiko Stuebner wrote:

...

Quoted 9 lines
Quoted 9 lines
arch/riscv/crypto/aes-riscv-glue.c new file +9 -0
+static int riscv64_aes_setkey_zvkned(struct crypto_tfm *tfm, const u8 *key,
+                      unsigned int keylen)
+{
+     struct riscv_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+     int ret;
+
+     ctx->keylen = keylen;
+
+     if (keylen == 16 || keylen == 32) {
Quoted 8 lines

arch/riscv/crypto/aes-riscv-glue.c line 88

Quoted 8 lines
arch/riscv/crypto/aes-riscv-glue.c new file +8 -0
+             kernel_rvv_begin();
+             ret = rv64i_zvkned_set_encrypt_key(key, keylen * 8, &ctx->enc_key);
+             if (ret != 1) {
+                     kernel_rvv_end();
+                     return -EINVAL;
+             }
+
+             ret = rv64i_zvkned_set_decrypt_key(key, keylen * 8, &ctx->dec_key);

The asm suggests that the encryption and decryption key schedules are the same, and the decryption algorithm does not implement the Equivalent Inverse Cipher, but simply iterates over they key schedule in reverse order. This makes much more sense for instruction based AES, so it doesn't surprise me but it does mean you can just drop this part, and pass enc_key everywhere.

Quoted 8 lines
Quoted 8 lines
arch/riscv/crypto/aes-riscv-glue.c new file +8 -0
+             kernel_rvv_end();
+             if (ret != 1)
+                     return -EINVAL;
+     }
+
+     ret = crypto_cipher_setkey(ctx->fallback, key, keylen);
+
+     return ret ? -EINVAL : 0;
Quoted 9 lines
Quoted 1 line

+}

It's a bit annoying that RISC-V doesn't support AES-192, though also not particularly surprising, seeing as AES-192 is almost never used. (Intel's Key Locker, for example, is another recent CPU feature that doesn't support AES-192.) IMO the issue here is really with the kernel crypto API -- it should treat AES-128, AES-192, and AES-256 as separate algorithms so that implementations aren't forced to support all three key sizes...

Why is this a fundamental limitation? AES-192 uses the same AES block size and round structure, the only difference is the number of rounds and how the round keys are calculated.

Creating the key schedule should never be performance critical, so if the lack of AES-192 support is due to a limitation in the key schedule generation instructions, I'd suggest to avoid those if possible and just use the generic library code to derive the key schedule. If that works, I'm pretty sure AES-192 support is just a matter of implementing a 12-round variant modeled after the existing 10/14 round ones.

#23 lore raw

Re: [PATCH v4 10/12] RISC-V: crypto: add Zvkned accelerated AES encryption implementation

Quoted 52 lines · Ard Biesheuvel
Quoted 2 lines · Eric Biggers

On Tue, Jul 11, 2023 at 05:37:41PM +0200, Heiko Stuebner wrote:

...

Quoted 9 lines
Quoted 9 lines
arch/riscv/crypto/aes-riscv-glue.c new file +9 -0
+static int riscv64_aes_setkey_zvkned(struct crypto_tfm *tfm, const u8 *key,
+                      unsigned int keylen)
+{
+     struct riscv_aes_ctx *ctx = crypto_tfm_ctx(tfm);
+     int ret;
+
+     ctx->keylen = keylen;
+
+     if (keylen == 16 || keylen == 32) {
Quoted 8 lines

arch/riscv/crypto/aes-riscv-glue.c line 88

Quoted 8 lines
arch/riscv/crypto/aes-riscv-glue.c new file +8 -0
+             kernel_rvv_begin();
+             ret = rv64i_zvkned_set_encrypt_key(key, keylen * 8, &ctx->enc_key);
+             if (ret != 1) {
+                     kernel_rvv_end();
+                     return -EINVAL;
+             }
+
+             ret = rv64i_zvkned_set_decrypt_key(key, keylen * 8, &ctx->dec_key);

The asm suggests that the encryption and decryption key schedules are the same, and the decryption algorithm does not implement the Equivalent Inverse Cipher, but simply iterates over they key schedule in reverse order. This makes much more sense for instruction based AES, so it doesn't surprise me but it does mean you can just drop this part, and pass enc_key everywhere.

Quoted 8 lines
Quoted 8 lines
arch/riscv/crypto/aes-riscv-glue.c new file +8 -0
+             kernel_rvv_end();
+             if (ret != 1)
+                     return -EINVAL;
+     }
+
+     ret = crypto_cipher_setkey(ctx->fallback, key, keylen);
+
+     return ret ? -EINVAL : 0;
Quoted 9 lines
Quoted 1 line

+}

It's a bit annoying that RISC-V doesn't support AES-192, though also not particularly surprising, seeing as AES-192 is almost never used. (Intel's Key Locker, for example, is another recent CPU feature that doesn't support AES-192.) IMO the issue here is really with the kernel crypto API -- it should treat AES-128, AES-192, and AES-256 as separate algorithms so that implementations aren't forced to support all three key sizes...

Why is this a fundamental limitation? AES-192 uses the same AES block size and round structure, the only difference is the number of rounds and how the round keys are calculated.

Quoted 7 lines

Creating the key schedule should never be performance critical, so if the lack of AES-192 support is due to a limitation in the key schedule generation instructions, I'd suggest to avoid those if possible and just use the generic library code to derive the key schedule. If that works, I'm pretty sure AES-192 support is just a matter of implementing a 12-round variant modeled after the existing 10/14 round ones.

This seems to work:
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=riscv-crypto

Feel free to incorporate/squash any of those changes into your series.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#24 lore raw

Re: [PATCH v4 10/12] RISC-V: crypto: add Zvkned accelerated AES encryption implementation

In reply to Eric Biggers
Quoted 11 lines · Eric Biggers

I'm looking forward to having direct support for these AES modes, especially the modes needed for storage encryption: XTS, and CBC or CTS! None of these AES modes is actually implemented in this patch yet, though, so they can't be claimed in the kconfig help text yet. This patch is just a starting point, as it just adds support for the bare AES block cipher ("aes" in the crypto API).

(BTW, I'm much more interested in, say, AES-XTS support than SM4 support, which this patchset does include. SM4 is a "national pride cipher" which is somewhat of a niche thing. I suppose there are already people pushing it for RISC-V though, as they are everywhere else, so that's to be expected...)

We have further optimization for RISC-V platform in OpenSSL PR[1]. It will include AES with CBC, CTR, and XTS mode. Comparing to the generic AES implementation, the specialized AES-XTS one have about 3X performance improvement using OpenSSL benchmark tool. If OpenSSL accepts that PR, we will create the corresponding patch for Linux kernel.

[1]
https://github.com/openssl/openssl/pull/21923

-Jerry

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#25 lore raw

Re: [PATCH v4 10/12] RISC-V: crypto: add Zvkned accelerated AES encryption implementation

Quoted 15 lines · Jerry Shih
Quoted 11 lines · Eric Biggers

I'm looking forward to having direct support for these AES modes, especially the modes needed for storage encryption: XTS, and CBC or CTS! None of these AES modes is actually implemented in this patch yet, though, so they can't be claimed in the kconfig help text yet. This patch is just a starting point, as it just adds support for the bare AES block cipher ("aes" in the crypto API).

(BTW, I'm much more interested in, say, AES-XTS support than SM4 support, which this patchset does include. SM4 is a "national pride cipher" which is somewhat of a niche thing. I suppose there are already people pushing it for RISC-V though, as they are everywhere else, so that's to be expected...)

Quoted 9 lines

We have further optimization for RISC-V platform in OpenSSL PR[1]. It will include AES with CBC, CTR, and XTS mode. Comparing to the generic AES implementation, the specialized AES-XTS one have about 3X performance improvement using OpenSSL benchmark tool. If OpenSSL accepts that PR, we will create the corresponding patch for Linux kernel.

[1]
https://github.com/openssl/openssl/pull/21923

This pull request doesn't appear to contain any XTS code at all, only CBC.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#26 lore raw

Re: [PATCH v4 10/12] RISC-V: crypto: add Zvkned accelerated AES encryption implementation

Quoted 11 lines · Ard Biesheuvel
Quoted 9 lines

We have further optimization for RISC-V platform in OpenSSL PR[1]. It will include AES with CBC, CTR, and XTS mode. Comparing to the generic AES implementation, the specialized AES-XTS one have about 3X performance improvement using OpenSSL benchmark tool. If OpenSSL accepts that PR, we will create the corresponding patch for Linux kernel.

[1]
https://github.com/openssl/openssl/pull/21923

This pull request doesn't appear to contain any XTS code at all, only CBC.

We have some license issues for upstream. We will append the specialized AES modes soon.

-Jerry

#27 lore raw

Re: [PATCH v4 10/12] RISC-V: crypto: add Zvkned accelerated AES encryption implementation

Quoted 4 lines · Jerry Shih
Quoted 1 line

This pull request doesn't appear to contain any XTS code at all, only CBC.

We have some license issues for upstream. We will append the specialized AES modes soon.

We have the XTS and other specialized AES modes in OpenSSL PR[1] now. The specialized implementations all perform better than generic implementation on FPGA.
We will try to make that implementations happen in kernel.

-Jerry

[1]
https://github.com/openssl/openssl/pull/21923

#28 lore raw

PATCH v4 11/12 RISC-V: crypto: add Zvksed accelerated SM4 encryption implementation

In reply to Heiko Stuebner

From: Heiko Stuebner <[email protected]>

Add support for the SM4 symmetric cipher implemented using the special instructions provided by the Zvksed vector crypto instructions.

---
 arch/riscv/crypto/Kconfig               |  17 ++
 arch/riscv/crypto/Makefile              |   7 +
 arch/riscv/crypto/sm4-riscv64-glue.c    | 162 +++++++++++++
 arch/riscv/crypto/sm4-riscv64-zvksed.pl | 300 ++++++++++++++++++++++++
 4 files changed, 486 insertions(+)
 create mode 100644 arch/riscv/crypto/sm4-riscv64-glue.c
 create mode 100644 arch/riscv/crypto/sm4-riscv64-zvksed.pl
arch/riscv/crypto/Kconfig +17 -0
index 8579ce43546d..d1e22482f7c4 100644
@@ -53,4 +53,21 @@ config CRYPTO_SHA512_RISCV64
 	  Architecture: riscv64
 	  - Zvknhb vector crypto extension
 
+config CRYPTO_SM4_RISCV64
+	tristate "Ciphers: SM4 (ShangMi 4)"
+	depends on 64BIT && RISCV_ISA_V
+	select CRYPTO_ALGAPI
+	select CRYPTO_SM4
+	select CRYPTO_SM4_GENERIC
+	help
+	  SM4 cipher algorithms (OSCCA GB/T 32907-2016,
+	  ISO/IEC 18033-3:2010/Amd 1:2021)
+
+	  SM4 (GBT.32907-2016) is a cryptographic standard issued by the
+	  Organization of State Commercial Administration of China (OSCCA)
+	  as an authorized cryptographic algorithms for the use within China.
+
+	  Architecture: riscv64
+	  - Zvksed vector crypto extension
+
 endmenu
arch/riscv/crypto/Makefile +7 -0
index 38ee741a9777..1a9f31b185de 100644
@@ -21,6 +21,9 @@ sha256-riscv64-y := sha256-riscv64-glue.o sha256-riscv64-zvbb-zvknha.o
 obj-$(CONFIG_CRYPTO_SHA512_RISCV64) += sha512-riscv64.o
 sha512-riscv64-y := sha512-riscv64-glue.o sha512-riscv64-zvbb-zvknhb.o
 
+obj-$(CONFIG_CRYPTO_SM4_RISCV64) += sm4-riscv64.o
+sm4-riscv64-y := sm4-riscv64-glue.o sm4-riscv64-zvksed.o
+
 quiet_cmd_perlasm = PERLASM $@
       cmd_perlasm = $(PERL) $(<) void $(@)
 
@@ -42,6 +45,10 @@ $(obj)/sha256-riscv64-zvbb-zvknha.S: $(src)/sha256-riscv64-zvbb-zvknha.pl
 $(obj)/sha512-riscv64-zvbb-zvknhb.S: $(src)/sha512-riscv64-zvbb-zvknhb.pl
 	$(call cmd,perlasm)
 
+$(obj)/sm4-riscv64-zvksed.S: $(src)/sm4-riscv64-zvksed.pl
+	$(call cmd,perlasm)
+
 clean-files += aes-riscv64-zvkned.S
 clean-files += ghash-riscv64-zbc.S ghash-riscv64-zvkb.S ghash-riscv64-zvkg.S
 clean-files += sha256-riscv64-zvknha.S sha512-riscv64-zvknhb.S
+clean-files += sm4-riscv64-zvksed.S
arch/riscv/crypto/sm4-riscv64-glue.c new file +162 -0

162 lines not on this page · raw message

arch/riscv/crypto/sm4-riscv64-zvksed.pl new file +300 -0

300 lines not on this page · raw message

Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#29 lore raw

PATCH v4 12/12 RISC-V: crypto: add Zvksh accelerated SM3 hash implementation

In reply to Heiko Stuebner

From: Heiko Stuebner <[email protected]>

Add support for the SM3 hash function implemented using the special instructions provided by the Zvksh vector crypto instructions.

---
 arch/riscv/crypto/Kconfig              |  11 ++
 arch/riscv/crypto/Makefile             |   8 +-
 arch/riscv/crypto/sm3-riscv64-glue.c   | 112 ++++++++++++
 arch/riscv/crypto/sm3-riscv64-zvksh.pl | 225 +++++++++++++++++++++++++
 4 files changed, 355 insertions(+), 1 deletion(-)
 create mode 100644 arch/riscv/crypto/sm3-riscv64-glue.c
 create mode 100644 arch/riscv/crypto/sm3-riscv64-zvksh.pl
arch/riscv/crypto/Kconfig +11 -0
index d1e22482f7c4..0c4e575ba8d6 100644
@@ -53,6 +53,17 @@ config CRYPTO_SHA512_RISCV64
 	  Architecture: riscv64
 	  - Zvknhb vector crypto extension
 
+config CRYPTO_SM3_RISCV64
+	tristate "Hash functions: SM3 (ShangMi 3)"
+	depends on 64BIT && RISCV_ISA_V
+	select CRYPTO_HASH
+	select CRYPTO_SM3
+	help
+	  SHA-512 secure hash algorithm (FIPS 180)
+
+	  Architecture: riscv64
+	  - Zvknhb vector crypto extension
+
 config CRYPTO_SM4_RISCV64
 	tristate "Ciphers: SM4 (ShangMi 4)"
 	depends on 64BIT && RISCV_ISA_V
arch/riscv/crypto/Makefile +7 -1
index 1a9f31b185de..f7faba6c12c9 100644
@@ -21,6 +21,9 @@ sha256-riscv64-y := sha256-riscv64-glue.o sha256-riscv64-zvbb-zvknha.o
 obj-$(CONFIG_CRYPTO_SHA512_RISCV64) += sha512-riscv64.o
 sha512-riscv64-y := sha512-riscv64-glue.o sha512-riscv64-zvbb-zvknhb.o
 
+obj-$(CONFIG_CRYPTO_SM3_RISCV64) += sm3-riscv64.o
+sm3-riscv64-y := sm3-riscv64-glue.o sm3-riscv64-zvksh.o
+
 obj-$(CONFIG_CRYPTO_SM4_RISCV64) += sm4-riscv64.o
 sm4-riscv64-y := sm4-riscv64-glue.o sm4-riscv64-zvksed.o
 
@@ -45,10 +48,13 @@ $(obj)/sha256-riscv64-zvbb-zvknha.S: $(src)/sha256-riscv64-zvbb-zvknha.pl
 $(obj)/sha512-riscv64-zvbb-zvknhb.S: $(src)/sha512-riscv64-zvbb-zvknhb.pl
 	$(call cmd,perlasm)
 
+$(obj)/sm3-riscv64-zvksh.S: $(src)/sm3-riscv64-zvksh.pl
+	$(call cmd,perlasm)
+
 $(obj)/sm4-riscv64-zvksed.S: $(src)/sm4-riscv64-zvksed.pl
 	$(call cmd,perlasm)
 
 clean-files += aes-riscv64-zvkned.S
 clean-files += ghash-riscv64-zbc.S ghash-riscv64-zvkb.S ghash-riscv64-zvkg.S
 clean-files += sha256-riscv64-zvknha.S sha512-riscv64-zvknhb.S
-clean-files += sm4-riscv64-zvksed.S
+clean-files += sm3-riscv64-zvksh.S sm4-riscv64-zvksed.S
arch/riscv/crypto/sm3-riscv64-glue.c new file +112 -0

112 lines not on this page · raw message

arch/riscv/crypto/sm3-riscv64-zvksh.pl new file +225 -0

225 lines not on this page · raw message

Signature · 7 lines
2.39.2


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#30 lore raw
In reply to Heiko Stuebner
Quoted 9 lines · Heiko Stuebner

From: Heiko Stuebner <[email protected]>

This series provides cryptographic implementations using the vector crypto extensions.

v13 of the vector patchset dropped the patches for in-kernel usage of vector instructions, I picked the ones from v12 over into this series for now.

Quoted 8 lines

My basic goal was to not re-invent cryptographic code, so the heavy lifting is done by those perl-asm scripts used in openssl and the perl code used here-in stems from code that is targetted at openssl [0] and is unmodified from there to limit needed review effort.

With a matching qemu (there are patches for vector-crypto flying around) the in-kernel crypto-selftests (also the extended ones) are very happy so far.

Where does this patchset apply to? I tried torvalds/master, linux-next/master, riscv/for-next, and cryptodev/master. Nothing worked. When sending a patch(set), please always use the '--base' option to 'git format-patch', or explicitly mention where it applies to, or provide a link to a git repo.

- Eric

#31 lore raw
Quoted 15 lines · Eric Biggers
Quoted 14 lines · Heiko Stuebner

From: Heiko Stuebner <[email protected]>

This series provides cryptographic implementations using the vector crypto extensions.

v13 of the vector patchset dropped the patches for in-kernel usage of vector instructions, I picked the ones from v12 over into this series for now.

My basic goal was to not re-invent cryptographic code, so the heavy lifting is done by those perl-asm scripts used in openssl and the perl code used here-in stems from code that is targetted at openssl [0] and is unmodified from there to limit needed review effort.

Quoted 9 lines
Quoted 3 lines

With a matching qemu (there are patches for vector-crypto flying around) the in-kernel crypto-selftests (also the extended ones) are very happy so far.

Where does this patchset apply to? I tried torvalds/master, linux-next/master, riscv/for-next, and cryptodev/master. Nothing worked. When sending a patch(set), please always use the '--base' option to 'git format-patch', or explicitly mention where it applies to, or provide a link to a git repo.

Hi Heiko, any update on this? I would like to review, and maybe test, this patchset but there's no way for me to do so.

- Eric

#32 lore raw

Hi Eric,

Am Freitag, 14. Juli 2023, 08:27:08 CEST schrieb Eric Biggers:

Quoted 20 lines · Eric Biggers
Quoted 19 lines · Heiko Stuebner
Quoted 17 lines

From: Heiko Stuebner <[email protected]>

This series provides cryptographic implementations using the vector crypto extensions.

v13 of the vector patchset dropped the patches for in-kernel usage of vector instructions, I picked the ones from v12 over into this series for now.

My basic goal was to not re-invent cryptographic code, so the heavy lifting is done by those perl-asm scripts used in openssl and the perl code used here-in stems from code that is targetted at openssl [0] and is unmodified from there to limit needed review effort.

With a matching qemu (there are patches for vector-crypto flying around) the in-kernel crypto-selftests (also the extended ones) are very happy so far.

Quoted 8 lines
Quoted 5 lines

Where does this patchset apply to? I tried torvalds/master, linux-next/master, riscv/for-next, and cryptodev/master. Nothing worked. When sending a patch(set), please always use the '--base' option to 'git format-patch', or explicitly mention where it applies to, or provide a link to a git repo.

Hi Heiko, any update on this? I would like to review, and maybe test, this patchset but there's no way for me to do so.

sorry about that. As you said, this should've been mentioned in the cover-letter.

This patchset goes on top of the v6 scalar one [0] which in turn goes on top of the arch-random patchset [1] and that in turn sits on top of 6.5-rc1 for me.

Heiko

[0] https://lore.kernel.org/r/[email protected]
[1] https://lore.kernel.org/r/[email protected]

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#33 lore raw
In reply to Heiko Stuebner

Hi Heiko,

Quoted 87 lines · Heiko Stuebner

From: Heiko Stuebner <[email protected]>

This series provides cryptographic implementations using the vector crypto extensions.

v13 of the vector patchset dropped the patches for in-kernel usage of vector instructions, I picked the ones from v12 over into this series for now.

My basic goal was to not re-invent cryptographic code, so the heavy lifting is done by those perl-asm scripts used in openssl and the perl code used here-in stems from code that is targetted at openssl [0] and is unmodified from there to limit needed review effort.

With a matching qemu (there are patches for vector-crypto flying around) the in-kernel crypto-selftests (also the extended ones) are very happy so far.

changes in v4:
- split off from scalar crypto patches but base on top of them
- adapt to pending openssl code [0] using the now frozen vector crypto extensions - with all its changes
[0] https://github.com/openssl/openssl/pull/20149

changes in v3:
- rebase on top of 6.3-rc2
- rebase on top of vector-v14 patchset
- add the missing Co-developed-by mentions to showcase the people that did the actual openSSL crypto code

changes in v2:
- rebased on 6.2 + zbb series, so don't include already applied changes anymore
- refresh code picked from openssl as that side matures
- more algorithms (SHA512, AES, SM3, SM4)

Greentime Hu (2):
riscv: Add support for kernel mode vector
riscv: Add vector extension XOR implementation

Heiko Stuebner (10):
RISC-V: add helper function to read the vector VLEN
RISC-V: add vector crypto extension detection
RISC-V: crypto: update perl include with helpers for vector (crypto)
instructions
RISC-V: crypto: add Zvbb+Zvbc accelerated GCM GHASH implementation
RISC-V: crypto: add Zvkg accelerated GCM GHASH implementation
RISC-V: crypto: add a vector-crypto-accelerated SHA256 implementation
RISC-V: crypto: add a vector-crypto-accelerated SHA512 implementation
RISC-V: crypto: add Zvkned accelerated AES encryption implementation
RISC-V: crypto: add Zvksed accelerated SM4 encryption implementation
RISC-V: crypto: add Zvksh accelerated SM3 hash implementation

 arch/riscv/crypto/Kconfig                     |  68 ++-
 arch/riscv/crypto/Makefile                    |  44 +-
 arch/riscv/crypto/aes-riscv-glue.c            | 168 ++++++
 arch/riscv/crypto/aes-riscv64-zvkned.pl       | 530 ++++++++++++++++++
 arch/riscv/crypto/ghash-riscv64-glue.c        | 245 ++++++++
 arch/riscv/crypto/ghash-riscv64-zvbb-zvbc.pl  | 380 +++++++++++++
 arch/riscv/crypto/ghash-riscv64-zvkg.pl       | 168 ++++++
 arch/riscv/crypto/riscv.pm                    | 433 +++++++++++++-
 arch/riscv/crypto/sha256-riscv64-glue.c       | 115 ++++
 .../crypto/sha256-riscv64-zvbb-zvknha.pl      | 314 +++++++++++
 arch/riscv/crypto/sha512-riscv64-glue.c       | 106 ++++
 .../crypto/sha512-riscv64-zvbb-zvknhb.pl      | 377 +++++++++++++
 arch/riscv/crypto/sm3-riscv64-glue.c          | 112 ++++
 arch/riscv/crypto/sm3-riscv64-zvksh.pl        | 225 ++++++++
 arch/riscv/crypto/sm4-riscv64-glue.c          | 162 ++++++
 arch/riscv/crypto/sm4-riscv64-zvksed.pl       | 300 ++++++++++
 arch/riscv/include/asm/hwcap.h                |   9 +
 arch/riscv/include/asm/vector.h               |  28 +
 arch/riscv/include/asm/xor.h                  |  82 +++
 arch/riscv/kernel/Makefile                    |   1 +
 arch/riscv/kernel/cpu.c                       |   8 +
 arch/riscv/kernel/cpufeature.c                |  50 ++
 arch/riscv/kernel/kernel_mode_vector.c        | 132 +++++
 arch/riscv/lib/Makefile                       |   1 +
 arch/riscv/lib/xor.S                          |  81 +++
 25 files changed, 4136 insertions(+), 3 deletions(-)
 create mode 100644 arch/riscv/crypto/aes-riscv-glue.c
 create mode 100644 arch/riscv/crypto/aes-riscv64-zvkned.pl
 create mode 100644 arch/riscv/crypto/ghash-riscv64-zvbb-zvbc.pl
 create mode 100644 arch/riscv/crypto/ghash-riscv64-zvkg.pl
 create mode 100644 arch/riscv/crypto/sha256-riscv64-glue.c
 create mode 100644 arch/riscv/crypto/sha256-riscv64-zvbb-zvknha.pl
 create mode 100644 arch/riscv/crypto/sha512-riscv64-glue.c
Quoted 9 lines
 create mode 100644 arch/riscv/crypto/sha512-riscv64-zvbb-zvknhb.pl
 create mode 100644 arch/riscv/crypto/sm3-riscv64-glue.c
 create mode 100644 arch/riscv/crypto/sm3-riscv64-zvksh.pl
 create mode 100644 arch/riscv/crypto/sm4-riscv64-glue.c
 create mode 100644 arch/riscv/crypto/sm4-riscv64-zvksed.pl
 create mode 100644 arch/riscv/include/asm/xor.h
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
 create mode 100644 arch/riscv/lib/xor.S

Thanks for working on this patchset! I'm glad to see that you and others are working on this and the code in OpenSSL. And thanks for running all the kernel crypto self-tests and verifying that they pass.

I'm still a bit worried about there being two competing sets of crypto extensions for RISC-V: scalar and vector.

However the vector crypto extensions are moving forwards (they were recently frozen), from what I've heard are being implemented in CPUs, and based on this patchset implementations of most algorithms are ready already.

So I'm wondering: do you still think that it's valuable to continue with your other patchset that adds GHASH acceleration using the scalar extensions (which this patchset is still based on)?

I'm wondering if we should be 100% focused on the vector extensions for now to avoid fragmentation of effort.

It's just not super clear to me what is driving the scalar crypto support right now. Maybe embedded systems? Maybe it was just a mistep, perhaps due to being started before the CPU even had a vector unit? I don't know. If you do indeed have a strong reason for it, then you can go ahead -- I just wanted to make sure we don't end up doing twice as much work unnecessarily.

- Eric

#34 lore raw
In reply to Heiko Stuebner
Quoted 9 lines · Heiko Stuebner

From: Heiko Stuebner <[email protected]>

This series provides cryptographic implementations using the vector crypto extensions.

v13 of the vector patchset dropped the patches for in-kernel usage of vector instructions, I picked the ones from v12 over into this series for now.

Quoted 8 lines

My basic goal was to not re-invent cryptographic code, so the heavy lifting is done by those perl-asm scripts used in openssl and the perl code used here-in stems from code that is targetted at openssl [0] and is unmodified from there to limit needed review effort.

With a matching qemu (there are patches for vector-crypto flying around) the in-kernel crypto-selftests (also the extended ones) are very happy so far.

Hi Heiko! Are you still working on this patchset? And which of its prerequisites still haven't been merged upstream?

- Eric

#35 lore raw
Quoted 22 lines · Eric Biggers
Quoted 9 lines · Heiko Stuebner

From: Heiko Stuebner <[email protected]>

This series provides cryptographic implementations using the vector crypto extensions.

v13 of the vector patchset dropped the patches for in-kernel usage of vector instructions, I picked the ones from v12 over into this series for now.

Quoted 8 lines

My basic goal was to not re-invent cryptographic code, so the heavy lifting is done by those perl-asm scripts used in openssl and the perl code used here-in stems from code that is targetted at openssl [0] and is unmodified from there to limit needed review effort.

With a matching qemu (there are patches for vector-crypto flying around) the in-kernel crypto-selftests (also the extended ones) are very happy so far.

Hi Heiko! Are you still working on this patchset? And which of its prerequisites still haven't been merged upstream?

Quoted 6 lines

- Eric

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

It is my understanding that Heiko is taking a break from development, I don't think he will be working on this soon.

- Charlie

#36 lore raw
Quoted 9 lines · Charlie Jenkins
Quoted 6 lines · Eric Biggers

On Tue, Jul 11, 2023 at 05:37:31PM +0200, Heiko Stuebner wrote:

Hi Heiko! Are you still working on this patchset? And which of its prerequisites still haven't been merged upstream?

- Eric

It is my understanding that Heiko is taking a break from development, I don't think he will be working on this soon.

We would like to take over these RISC-V vector crypto implementations. Our proposed implementations is under reviewing in OpenSSL pr. And I will check the gluing parts between Linux kernel and OpenSSL.

-Jerry

#37 lore raw
Quoted 8 lines · He-Jie Shih
Quoted 6 lines · Charlie Jenkins
Quoted 5 lines · Eric Biggers

On Tue, Jul 11, 2023 at 05:37:31PM +0200, Heiko Stuebner wrote:

Hi Heiko! Are you still working on this patchset? And which of its prerequisites still haven't been merged upstream?

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

- Eric

It is my understanding that Heiko is taking a break from development, I don't think he will be working on this soon.

We would like to take over these RISC-V vector crypto implementations. Our proposed implementations is under reviewing in OpenSSL pr. And I will check the gluing parts between Linux kernel and OpenSSL.

The OpenSSL PR is at [1].
And we are from SiFive.

-Jerry

[1]
https://github.com/openssl/openssl/pull/21923

#38 lore raw
Quoted 18 lines · Jerry Shih
Quoted 8 lines · He-Jie Shih
Quoted 6 lines · Charlie Jenkins
Quoted 5 lines · Eric Biggers

On Tue, Jul 11, 2023 at 05:37:31PM +0200, Heiko Stuebner wrote:

Hi Heiko! Are you still working on this patchset? And which of its prerequisites still haven't been merged upstream?

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

- Eric

It is my understanding that Heiko is taking a break from development, I don't think he will be working on this soon.

We would like to take over these RISC-V vector crypto implementations. Our proposed implementations is under reviewing in OpenSSL pr. And I will check the gluing parts between Linux kernel and OpenSSL.

Quoted 7 lines

The OpenSSL PR is at [1].
And we are from SiFive.

-Jerry

[1]
https://github.com/openssl/openssl/pull/21923

Hi Jerry, I'm wondering if you have an update on this? Do you need any help?

I'm also wondering about riscv.pm and the choice of generating the crypto instructions from .words instead of using the assembler. It makes it significantly harder to review the code, IMO. Can we depend on assembler support for these instructions, or is that just not ready yet?

- Eric

#39 lore raw
Quoted 11 lines · Eric Biggers
Quoted 8 lines · Jerry Shih

On Sep 15, 2023, at 09:48, He-Jie Shih <[email protected]> wrote:
The OpenSSL PR is at [1].
And we are from SiFive.

-Jerry

[1]
https://github.com/openssl/openssl/pull/21923

Hi Jerry, I'm wondering if you have an update on this? Do you need any help?

We have specialized aes-cbc/ecb/ctr patch locally and pass the testmgr test cases. But the test patterns in testmgr are quite simple, I think it doesn't test the corner case(e.g. aes-xts with tail element).

For aes-xts, I'm trying to update the implementation from OpenSSL. The design philosophy is different between OpenSSL and linux. In linux crypto, the data will be split into scatterlist. I need to preserve the aes-xts's iv for each scatterlist entry call. And I'm thinking about how to handle the tail data in a simple way. By the way, the xts(aes) implementation for arm and x86 are using cra_blocksize= AES_BLOCK_SIZE. I don't know why we need to handle the tail element. I think we will hit EINVAL error in skcipher_walk_next() if the data size it not be a multiple of block size.

Overall, we will have
1) aes cipher
2) aes with cbc/ecb/ctr/xts mode
3) sha256/512 for vlen>=128 platform
4) sm3 for vlen>=128 platform
5) sm4
6) ghash
7) chacha20 stream cipher

The vector crypto pr in OpenSSL is under reviewing, we are still updating the perl file into linux.

The most complicated gcm(aes) mode will be in our next plan.

Quoted 4 lines

I'm also wondering about riscv.pm and the choice of generating the crypto instructions from .words instead of using the assembler. It makes it significantly harder to review the code, IMO. Can we depend on assembler support for these instructions, or is that just not ready yet?

I have asked the same question before[1]. The reason is that Openssl could use very old compiler for compiling. Thus, the assembler might not know the standard rvv 1.0[2] and other vector crypto[3] instructions. That's why we use opcode for all vector instructions. IMO, I would prefer to use opcode for vector crypto only. The gcc-12 and clang-14 are already supporting rvv 1.0. Actually, I just read the perl file instead of the actually generated opcode for OpenSSL pr reviewing. And it's not hard to read the perl code.

Thanks,
- Jerry

[1]
https://github.com/openssl/openssl/pull/20149#discussion_r1244655440
[2]
https://github.com/riscv/riscv-v-spec/blob/master/v-spec.adoc
[3
https://github.com/riscv/riscv-crypto/blob/main/doc/vector/riscv-crypto-spec-vector.adoc]

#40 lore raw
Quoted 9 lines · He-Jie Shih
Quoted 7 lines · Eric Biggers
Quoted 6 lines · Jerry Shih

On Sep 15, 2023, at 09:48, He-Jie Shih <[email protected]> wrote:
The OpenSSL PR is at [1].
And we are from SiFive.

-Jerry

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

[1]
https://github.com/openssl/openssl/pull/21923

Hi Jerry, I'm wondering if you have an update on this? Do you need any help?

We have specialized aes-cbc/ecb/ctr patch locally and pass the testmgr test cases. But the test patterns in testmgr are quite simple, I think it doesn't test the corner case(e.g. aes-xts with tail element).

There should be test cases for that.

Quoted 4 lines

For aes-xts, I'm trying to update the implementation from OpenSSL. The design philosophy is different between OpenSSL and linux. In linux crypto, the data will be split into scatterlist. I need to preserve the aes-xts's iv for each scatterlist entry call.

Yes, this applies to all block ciphers that take an IV.

Quoted 1 line

And I'm thinking about how to handle the tail data in a simple way.

The RISC-V vector ISA is quite advanced, so there may be a better trick using predicates etc but otherwise, I suppose you could reuse the same trick that other asm implementations use, which is to use unaligned loads and stores for the final blocks, and to use a vector permute with a permute table to shift the bytes in the registers. But this is not performance critical, given that existing in-kernel users use sector or page size inputs only.

Quoted 5 lines

By the way, the xts(aes) implementation for arm and x86 are using cra_blocksize= AES_BLOCK_SIZE. I don't know why we need to handle the tail element. I think we will hit EINVAL error in skcipher_walk_next() if the data size it not be a multiple of block size.

No, both XTS and CBC-CTS permit inputs that are not a multiple of the block size, and will use some form of ciphertext stealing for the final tail. There is a generic CTS template that wraps CBC but combining them in the same way (e.g., using vector permute) will speed up things substantially. *However*, I'm not sure how relevant CBC-CTS is in the kernel, given that only fscrypt uses it IIRC but actually prefers something else so for new systems perhaps you shouldn't bother.

Quoted 19 lines

Overall, we will have
1) aes cipher
2) aes with cbc/ecb/ctr/xts mode
3) sha256/512 for vlen>=128 platform
4) sm3 for vlen>=128 platform
5) sm4
6) ghash
7) chacha20 stream cipher

The vector crypto pr in OpenSSL is under reviewing, we are still updating the perl file into linux.

The most complicated gcm(aes) mode will be in our next plan.

Quoted 4 lines

I'm also wondering about riscv.pm and the choice of generating the crypto instructions from .words instead of using the assembler. It makes it significantly harder to review the code, IMO. Can we depend on assembler support for these instructions, or is that just not ready yet?

Quoted 8 lines

I have asked the same question before[1]. The reason is that Openssl could use very old compiler for compiling. Thus, the assembler might not know the standard rvv 1.0[2] and other vector crypto[3] instructions. That's why we use opcode for all vector instructions. IMO, I would prefer to use opcode for vector crypto only. The gcc-12 and clang-14 are already supporting rvv 1.0. Actually, I just read the perl file instead of the actually generated opcode for OpenSSL pr reviewing. And it's not hard to read the perl code.

I understand the desire to reuse code, and OpenSSL already relies on so-called perlasm for this, but I think this is not a great choice, and I actually think this was a mistake for RISC-V. OpenSSL relies on perlasm for things like emittting different function pro-/epilogues depending on the calling convention (SysV versus MS on x86_64, for instance), but RISC-V does not have that much variety, and already supports the insn_r / insn_i pseudo instructions to emit arbitrary opcodes while still supporting named registers as usual. [Maybe my experience does not quite extrapolate to the vector ISA, but I managed to implement scalar AES [0] using the insn_r and insn_i pseudo instructions (which are generally provided by the assembler but Linux has fallback CPP macros for them as well), and this results on much more maintainable code IMO.[

We are using some of the OpenSSL perlasm in the kernel already (and some of it was introduced by me) but I don't think we should blindly reuse all of the RISC-V code if some of it can straight-forwardly be written as normal .S files.

[0] https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=riscv-scalar-aes

#41 lore raw
Quoted 13 lines · Ard Biesheuvel
Quoted 12 lines · He-Jie Shih
Quoted 10 lines · Eric Biggers
Quoted 8 lines · Jerry Shih

On Sep 15, 2023, at 09:48, He-Jie Shih <[email protected]> wrote:
The OpenSSL PR is at [1].
And we are from SiFive.

-Jerry

[1]
https://github.com/openssl/openssl/pull/21923

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Hi Jerry, I'm wondering if you have an update on this? Do you need any help?

We have specialized aes-cbc/ecb/ctr patch locally and pass the testmgr test cases. But the test patterns in testmgr are quite simple, I think it doesn't test the corner case(e.g. aes-xts with tail element).

There should be test cases for that.

Yes, non-block-aligned AES-XTS test vectors should be added to crypto/testmgr.h. Though, that case should be already covered by the randomized tests enabled by CONFIG_CRYPTO_MANAGER_EXTRA_TESTS=y, which I very strongly recommend enabling during development.

Quoted 75 lines
Quoted 5 lines

For aes-xts, I'm trying to update the implementation from OpenSSL. The design philosophy is different between OpenSSL and linux. In linux crypto, the data will be split into scatterlist. I need to preserve the aes-xts's iv for each scatterlist entry call.

Yes, this applies to all block ciphers that take an IV.

Quoted 1 line

And I'm thinking about how to handle the tail data in a simple way.

The RISC-V vector ISA is quite advanced, so there may be a better trick using predicates etc but otherwise, I suppose you could reuse the same trick that other asm implementations use, which is to use unaligned loads and stores for the final blocks, and to use a vector permute with a permute table to shift the bytes in the registers. But this is not performance critical, given that existing in-kernel users use sector or page size inputs only.

Quoted 5 lines

By the way, the xts(aes) implementation for arm and x86 are using cra_blocksize= AES_BLOCK_SIZE. I don't know why we need to handle the tail element. I think we will hit EINVAL error in skcipher_walk_next() if the data size it not be a multiple of block size.

No, both XTS and CBC-CTS permit inputs that are not a multiple of the block size, and will use some form of ciphertext stealing for the final tail. There is a generic CTS template that wraps CBC but combining them in the same way (e.g., using vector permute) will speed up things substantially. *However*, I'm not sure how relevant CBC-CTS is in the kernel, given that only fscrypt uses it IIRC but actually prefers something else so for new systems perhaps you shouldn't bother.

Quoted 19 lines

Overall, we will have
1) aes cipher
2) aes with cbc/ecb/ctr/xts mode
3) sha256/512 for vlen>=128 platform
4) sm3 for vlen>=128 platform
5) sm4
6) ghash
7) chacha20 stream cipher

The vector crypto pr in OpenSSL is under reviewing, we are still updating the perl file into linux.

The most complicated gcm(aes) mode will be in our next plan.

Quoted 4 lines

I'm also wondering about riscv.pm and the choice of generating the crypto instructions from .words instead of using the assembler. It makes it significantly harder to review the code, IMO. Can we depend on assembler support for these instructions, or is that just not ready yet?

Quoted 8 lines

I have asked the same question before[1]. The reason is that Openssl could use very old compiler for compiling. Thus, the assembler might not know the standard rvv 1.0[2] and other vector crypto[3] instructions. That's why we use opcode for all vector instructions. IMO, I would prefer to use opcode for vector crypto only. The gcc-12 and clang-14 are already supporting rvv 1.0. Actually, I just read the perl file instead of the actually generated opcode for OpenSSL pr reviewing. And it's not hard to read the perl code.

I understand the desire to reuse code, and OpenSSL already relies on so-called perlasm for this, but I think this is not a great choice, and I actually think this was a mistake for RISC-V. OpenSSL relies on perlasm for things like emittting different function pro-/epilogues depending on the calling convention (SysV versus MS on x86_64, for instance), but RISC-V does not have that much variety, and already supports the insn_r / insn_i pseudo instructions to emit arbitrary opcodes while still supporting named registers as usual. [Maybe my experience does not quite extrapolate to the vector ISA, but I managed to implement scalar AES [0] using the insn_r and insn_i pseudo instructions (which are generally provided by the assembler but Linux has fallback CPP macros for them as well), and this results on much more maintainable code IMO.[

Quoted 6 lines

We are using some of the OpenSSL perlasm in the kernel already (and some of it was introduced by me) but I don't think we should blindly reuse all of the RISC-V code if some of it can straight-forwardly be written as normal .S files.

[0] https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=riscv-scalar-aes

I'm not a huge fan of perlasm either. Normal .S files can be much easier to understand, and they do still support basic features like macros. (Of course, this only works if the .S file is the real source code. If the real source code is the perlasm, dumping it to a .S file doesn't make it more readable.)

Ultimately, it needs to decided on an algorithm-by-algorithm basis whether it makes sense to use the .pl file directly from OpenSSL or write a normal .S file. Sharing code can save time, but it can also waste time if/when things don't match up and need to be changed for the kernel anyway. If you look at the other architectures, sharing the OpenSSL .pl file is most common for Poly1305 and SHA-2. It's rarer for AES modes.

In any case, regardless of .pl or .S, it would be nice to rely on the assembler for the mapping from readable instruction to 32-bit words. Yes, I understand that the algorithm code reads mostly the some either way, but it introduces nonstandard notation (e.g. due to having to avoid the period character) and a possibility for error. It's not the 1940s anymore; we should be able to have an assembler. Why not make OpenSSL and Linux only enable this code when the assembler supports it? Note that Linux already does this for many of the x86 extensions, so there is precedent for this; see arch/x86/Kconfig.assembler.

- Eric

#43 lore raw
In reply to Eric Biggers
Quoted 6 lines · Eric Biggers
Quoted 5 lines · Jerry Shih

On Sep 15, 2023, at 09:48, He-Jie Shih <[email protected]> wrote:

The OpenSSL PR is at [1].
And we are from SiFive.

Quoted 6 lines
Quoted 4 lines

-Jerry

[1]
https://github.com/openssl/openssl/pull/21923

Hi Jerry, I'm wondering if you have an update on this? Do you need any help?

The RISC-V vector crypto OpenSSL pr[1] is merged.
And we also sent the vector-crypto patch based on Heiko's and OpenSSL works.
Here is the link:
https://lore.kernel.org/all/[email protected]/

[1]
https://github.com/openssl/openssl/pull/21923

Quoted 6 lines

I'm also wondering about riscv.pm and the choice of generating the crypto instructions from .words instead of using the assembler. It makes it significantly harder to review the code, IMO. Can we depend on assembler support for these instructions, or is that just not ready yet?

- Eric

There is no public assembler supports the vector-crypto asm mnemonics. We should still use opcode for vector-crypto instructions. But we might use asm for standard rvv parts.
In order to reuse the codes in OpenSSL as much as possible, we still use the riscv.pm for all standard rvv and vector-crypto instructions. If the asm mnemonic is still a better approach, I will rewrite all standard rvv parts with asm mnemonics in next patch.

-Jerry

#44 lore raw

Hi Jerry,

(Just so you know, you still need to fix your email configuration. Your emails have a bogus Reply-To header, which makes replies not be sent to you by default. I had to manually set the "To:" address when replying.)

Quoted 9 lines · Jerry Shih

The RISC-V vector crypto OpenSSL pr[1] is merged.
And we also sent the vector-crypto patch based on Heiko's and OpenSSL works.
Here is the link:
https://lore.kernel.org/all/[email protected]/

[1]
https://github.com/openssl/openssl/pull/21923

Awesome, thanks!

Quoted 8 lines
Quoted 7 lines

I'm also wondering about riscv.pm and the choice of generating the crypto instructions from .words instead of using the assembler. It makes it significantly harder to review the code, IMO. Can we depend on assembler support for these instructions, or is that just not ready yet?

- Eric

Quoted 7 lines

There is no public assembler supports the vector-crypto asm mnemonics. We should still use opcode for vector-crypto instructions. But we might use asm for standard rvv parts.
In order to reuse the codes in OpenSSL as much as possible, we still use the riscv.pm for all standard rvv and vector-crypto instructions. If the asm mnemonic is still a better approach, I will rewrite all standard rvv parts with asm mnemonics in next patch.

Tip-of-tree gcc + binutils seems to support them. Building some of the sample code from the riscv-crypto repository:

    $ riscv64-linux-gnu-as --version
    GNU assembler (GNU Binutils) 2.41.50.20231021
    $ riscv64-linux-gnu-gcc --version
    riscv64-linux-gnu-gcc (GCC) 14.0.0 20231021 (experimental)
    $ riscv64-linux-gnu-gcc -march=rv64ivzvkned -c riscv-crypto/doc/vector/code-samples/zvkned.s

And tip-of-tree clang supports them experimentally:

    $ clang --version
    clang version 18.0.0 (https://github.com/llvm/llvm-project 30416f39be326b403e19f23da387009736483119)
    $ clang -menable-experimental-extensions -target riscv64-linux-gnu -march=rv64ivzvkned1 -c riscv-crypto/doc/vector/code-samples/zvkned.s

It would be nice to use a real assembler, so that people won't have to worry about potential mistakes or inconsistencies in the perl-based "assembler". Also keep in mind that if we allow people to compile this code without the real assembler support from the beginning, it might end up staying that way for quite a while in order to avoid breaking the build for people.

Ultimately it's up to you though; I think that you and others who have been working on RISC-V crypto can make the best decision about what to do here. I also don't want this patchset to be delayed waiting for other projects, so maybe that indeed means the perl-based "assembler" needs to be used for now.

- Eric

#45 lore raw
Quoted 30 lines · Eric Biggers
Quoted 8 lines

There is no public assembler supports the vector-crypto asm mnemonics. We should still use opcode for vector-crypto instructions. But we might use asm for standard rvv parts.
In order to reuse the codes in OpenSSL as much as possible, we still use the riscv.pm for all standard rvv and vector-crypto instructions. If the asm mnemonic is still a better approach, I will rewrite all standard rvv parts with asm mnemonics in next patch.

Tip-of-tree gcc + binutils seems to support them. Building some of the sample code from the riscv-crypto repository:

    $ riscv64-linux-gnu-as --version
    GNU assembler (GNU Binutils) 2.41.50.20231021
    $ riscv64-linux-gnu-gcc --version
    riscv64-linux-gnu-gcc (GCC) 14.0.0 20231021 (experimental)
    $ riscv64-linux-gnu-gcc -march=rv64ivzvkned -c riscv-crypto/doc/vector/code-samples/zvkned.s

And tip-of-tree clang supports them experimentally:

    $ clang --version
    clang version 18.0.0 (https://github.com/llvm/llvm-project 30416f39be326b403e19f23da387009736483119)
    $ clang -menable-experimental-extensions -target riscv64-linux-gnu -march=rv64ivzvkned1 -c riscv-crypto/doc/vector/code-samples/zvkned.s

It would be nice to use a real assembler, so that people won't have to worry about potential mistakes or inconsistencies in the perl-based "assembler". Also keep in mind that if we allow people to compile this code without the real assembler support from the beginning, it might end up staying that way for quite a while in order to avoid breaking the build for people.

Quoted 5 lines

Ultimately it's up to you though; I think that you and others who have been working on RISC-V crypto can make the best decision about what to do here. I also don't want this patchset to be delayed waiting for other projects, so maybe that indeed means the perl-based "assembler" needs to be used for now.

Just wanted to bump up this discussion again. In binutils, the vector crypto v1.0.0 support was released 4 months ago in 2.41. See the NEWS file at
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob_plain;f=binutils/NEWS;hb=refs/heads/binutils-2_41-branch

* The RISC-V port now supports the following new standard extensions:
- Zicond (conditional zero instructions)
- Zfa (additional floating-point instructions)
- Zvbb, Zvbc, Zvkg, Zvkned, Zvknh[ab], Zvksed, Zvksh, Zvkn, Zvknc, Zvkng, Zvks, Zvksc, Zvkg, Zvkt (vector crypto instructions)

That's every extension listed in the vector crypto v1.0.0 specification
(https://github.com/riscv/riscv-crypto/releases/download/v1.0.0/riscv-crypto-spec-vector.pdf).

LLVM still has the vector crypto extensions marked as "experimental" extensions. However, there is an open pull request to mark them non-experimental:
https://github.com/llvm/llvm-project/pull/69000

Could we just go ahead and remove riscv.pm, develop with binutils for now, and prioritize getting the LLVM support finished?

- Eric

#46 lore raw
Quoted 17 lines · Eric Biggers
Quoted 11 lines · Eric Biggers

It would be nice to use a real assembler, so that people won't have to worry about potential mistakes or inconsistencies in the perl-based "assembler". Also keep in mind that if we allow people to compile this code without the real assembler support from the beginning, it might end up staying that way for quite a while in order to avoid breaking the build for people.

Ultimately it's up to you though; I think that you and others who have been working on RISC-V crypto can make the best decision about what to do here. I also don't want this patchset to be delayed waiting for other projects, so maybe that indeed means the perl-based "assembler" needs to be used for now.

Just wanted to bump up this discussion again. In binutils, the vector crypto v1.0.0 support was released 4 months ago in 2.41. See the NEWS file at
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob_plain;f=binutils/NEWS;hb=refs/heads/binutils-2_41-branch

Quoted 8 lines

* The RISC-V port now supports the following new standard extensions:
- Zicond (conditional zero instructions)
- Zfa (additional floating-point instructions)
- Zvbb, Zvbc, Zvkg, Zvkned, Zvknh[ab], Zvksed, Zvksh, Zvkn, Zvknc, Zvkng, Zvks, Zvksc, Zvkg, Zvkt (vector crypto instructions)

That's every extension listed in the vector crypto v1.0.0 specification
(https://github.com/riscv/riscv-crypto/releases/download/v1.0.0/riscv-crypto-spec-vector.pdf).

It doesn't fit all v1.0 spec.
The Zvkb is missed in binutils. It's the subset of Zvbb. We needs some extra works if user just try to use Zvkb.
https://github.com/riscv/riscv-crypto/blob/main/doc/vector/riscv-crypto-vector-zvkb.adoc
Some crypto algorithms are already checking for Zvkb instead of Zvbb.

Quoted 6 lines

LLVM still has the vector crypto extensions marked as "experimental" extensions. However, there is an open pull request to mark them non-experimental:
https://github.com/llvm/llvm-project/pull/69000

Could we just go ahead and remove riscv.pm, develop with binutils for now, and prioritize getting the LLVM support finished?

Yes, we could.
But we need to handle the extensions checking for toolchains like:
https://github.com/torvalds/linux/commit/b6fcdb191e36f82336f9b5e126d51c02e7323480
I could do that, but I think I need some times to test the builds. And it will introduce more dependency patch which is not related to actual crypto algorithms and the gluing code in kernel. I will send another patch for toolchain part after the v2 patch. And I am working for v2 patch with your new review comments. The v2 will still use perlasm.
And we could move this discussion to this thread.
https://lore.kernel.org/all/[email protected]/

-Jerry

#47 lore raw
Quoted 24 lines · Jerry Shih
Quoted 23 lines · Eric Biggers
Quoted 11 lines · Eric Biggers

It would be nice to use a real assembler, so that people won't have to worry about potential mistakes or inconsistencies in the perl-based "assembler". Also keep in mind that if we allow people to compile this code without the real assembler support from the beginning, it might end up staying that way for quite a while in order to avoid breaking the build for people.

Ultimately it's up to you though; I think that you and others who have been working on RISC-V crypto can make the best decision about what to do here. I also don't want this patchset to be delayed waiting for other projects, so maybe that indeed means the perl-based "assembler" needs to be used for now.

Just wanted to bump up this discussion again. In binutils, the vector crypto v1.0.0 support was released 4 months ago in 2.41. See the NEWS file at
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob_plain;f=binutils/NEWS;hb=refs/heads/binutils-2_41-branch

* The RISC-V port now supports the following new standard extensions:
- Zicond (conditional zero instructions)
- Zfa (additional floating-point instructions)
- Zvbb, Zvbc, Zvkg, Zvkned, Zvknh[ab], Zvksed, Zvksh, Zvkn, Zvknc, Zvkng, Zvks, Zvksc, Zvkg, Zvkt (vector crypto instructions)

Quoted 8 lines
Quoted 2 lines

That's every extension listed in the vector crypto v1.0.0 specification
(https://github.com/riscv/riscv-crypto/releases/download/v1.0.0/riscv-crypto-spec-vector.pdf).

It doesn't fit all v1.0 spec.
The Zvkb is missed in binutils. It's the subset of Zvbb. We needs some extra works if user just try to use Zvkb.
https://github.com/riscv/riscv-crypto/blob/main/doc/vector/riscv-crypto-vector-zvkb.adoc
Some crypto algorithms are already checking for Zvkb instead of Zvbb.

Yeah, that's unfortunate that Zvkb got missed in binutils. However, since all Zvkb instructions are part of Zvbb, which is supported, assembling Zvkb instructions should still work --- right?

Quoted 7 lines
Quoted 6 lines

LLVM still has the vector crypto extensions marked as "experimental" extensions. However, there is an open pull request to mark them non-experimental:
https://github.com/llvm/llvm-project/pull/69000

Could we just go ahead and remove riscv.pm, develop with binutils for now, and prioritize getting the LLVM support finished?

Quoted 8 lines

Yes, we could.
But we need to handle the extensions checking for toolchains like:
https://github.com/torvalds/linux/commit/b6fcdb191e36f82336f9b5e126d51c02e7323480
I could do that, but I think I need some times to test the builds. And it will introduce more dependency patch which is not related to actual crypto algorithms and the gluing code in kernel. I will send another patch for toolchain part after the v2 patch. And I am working for v2 patch with your new review comments. The v2 will still use perlasm.

Note that perlasm (.pl) vs assembly (.S), and ".inst" vs real assembler instructions, are actually separate concerns. We could use real assembler instructions while still using perlasm. Or we could use assembly while still using macros that generate the instructions as .inst.

My preference is indeed both: assembly (.S) with real assembler instructions. That would keep things more straightforward.

We do not necessarily need to do both before merging the code, though. It will be beneficial to get this code merged sooner rather than later, so that other people can work on improving it.

I would prioritize the change to use real assembler instructions. I do think it's worth thinking about getting that change in from the beginning, so that the toolchain prerequisites are properly in place from the beginning and people can properly account for them and prioritize the toolchain work as needed.

- Eric

#48 lore raw
Quoted 35 lines · Eric Biggers
Quoted 24 lines · Jerry Shih
Quoted 23 lines · Eric Biggers
Quoted 11 lines · Eric Biggers

It would be nice to use a real assembler, so that people won't have to worry about potential mistakes or inconsistencies in the perl-based "assembler". Also keep in mind that if we allow people to compile this code without the real assembler support from the beginning, it might end up staying that way for quite a while in order to avoid breaking the build for people.

Ultimately it's up to you though; I think that you and others who have been working on RISC-V crypto can make the best decision about what to do here. I also don't want this patchset to be delayed waiting for other projects, so maybe that indeed means the perl-based "assembler" needs to be used for now.

Just wanted to bump up this discussion again. In binutils, the vector crypto v1.0.0 support was released 4 months ago in 2.41. See the NEWS file at
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob_plain;f=binutils/NEWS;hb=refs/heads/binutils-2_41-branch

* The RISC-V port now supports the following new standard extensions:
- Zicond (conditional zero instructions)
- Zfa (additional floating-point instructions)
- Zvbb, Zvbc, Zvkg, Zvkned, Zvknh[ab], Zvksed, Zvksh, Zvkn, Zvknc, Zvkng, Zvks, Zvksc, Zvkg, Zvkt (vector crypto instructions)

Quoted 8 lines
Quoted 2 lines

That's every extension listed in the vector crypto v1.0.0 specification
(https://github.com/riscv/riscv-crypto/releases/download/v1.0.0/riscv-crypto-spec-vector.pdf).

It doesn't fit all v1.0 spec.
The Zvkb is missed in binutils. It's the subset of Zvbb. We needs some extra works if user just try to use Zvkb.
https://github.com/riscv/riscv-crypto/blob/main/doc/vector/riscv-crypto-vector-zvkb.adoc
Some crypto algorithms are already checking for Zvkb instead of Zvbb.

Quoted 3 lines

Yeah, that's unfortunate that Zvkb got missed in binutils. However, since all Zvkb instructions are part of Zvbb, which is supported, assembling Zvkb instructions should still work --- right?

Not forgotten, but the Zvkb extension did not exist when the patchset was merged.
RISC-V extension support is typically merged when specifications are "frozen". This means a high bar for changes, but they are possible until the spec is ratified.
Often nothing is changed until ratification, but here Zvkb has been (re-)introduced.

I was not aware of this untils I read this email, so I just wrote a patch that fills the gap:
https://sourceware.org/pipermail/binutils/2023-November/130762.html

Thanks for reporting!

BR
Christoph

Quoted 35 lines
Quoted 8 lines
Quoted 7 lines

LLVM still has the vector crypto extensions marked as "experimental" extensions. However, there is an open pull request to mark them non-experimental:
https://github.com/llvm/llvm-project/pull/69000

Could we just go ahead and remove riscv.pm, develop with binutils for now, and prioritize getting the LLVM support finished?

Quoted 8 lines

Yes, we could.
But we need to handle the extensions checking for toolchains like:
https://github.com/torvalds/linux/commit/b6fcdb191e36f82336f9b5e126d51c02e7323480
I could do that, but I think I need some times to test the builds. And it will introduce more dependency patch which is not related to actual crypto algorithms and the gluing code in kernel. I will send another patch for toolchain part after the v2 patch. And I am working for v2 patch with your new review comments. The v2 will still use perlasm.

Note that perlasm (.pl) vs assembly (.S), and ".inst" vs real assembler instructions, are actually separate concerns. We could use real assembler instructions while still using perlasm. Or we could use assembly while still using macros that generate the instructions as .inst.

My preference is indeed both: assembly (.S) with real assembler instructions. That would keep things more straightforward.

We do not necessarily need to do both before merging the code, though. It will be beneficial to get this code merged sooner rather than later, so that other people can work on improving it.

I would prioritize the change to use real assembler instructions. I do think it's worth thinking about getting that change in from the beginning, so that the toolchain prerequisites are properly in place from the beginning and people can properly account for them and prioritize the toolchain work as needed.

- Eric

#49 lore raw
Quoted 48 lines · Christoph Müllner
Quoted 35 lines · Eric Biggers
Quoted 24 lines · Jerry Shih
Quoted 23 lines · Eric Biggers
Quoted 11 lines · Eric Biggers

It would be nice to use a real assembler, so that people won't have to worry about potential mistakes or inconsistencies in the perl-based "assembler". Also keep in mind that if we allow people to compile this code without the real assembler support from the beginning, it might end up staying that way for quite a while in order to avoid breaking the build for people.

Ultimately it's up to you though; I think that you and others who have been working on RISC-V crypto can make the best decision about what to do here. I also don't want this patchset to be delayed waiting for other projects, so maybe that indeed means the perl-based "assembler" needs to be used for now.

Just wanted to bump up this discussion again. In binutils, the vector crypto v1.0.0 support was released 4 months ago in 2.41. See the NEWS file at
https://sourceware.org/git/gitweb.cgi?p=binutils-gdb.git;a=blob_plain;f=binutils/NEWS;hb=refs/heads/binutils-2_41-branch

* The RISC-V port now supports the following new standard extensions:
- Zicond (conditional zero instructions)
- Zfa (additional floating-point instructions)
- Zvbb, Zvbc, Zvkg, Zvkned, Zvknh[ab], Zvksed, Zvksh, Zvkn, Zvknc, Zvkng, Zvks, Zvksc, Zvkg, Zvkt (vector crypto instructions)

Quoted 8 lines
Quoted 2 lines

That's every extension listed in the vector crypto v1.0.0 specification
(https://github.com/riscv/riscv-crypto/releases/download/v1.0.0/riscv-crypto-spec-vector.pdf).

It doesn't fit all v1.0 spec.
The Zvkb is missed in binutils. It's the subset of Zvbb. We needs some extra works if user just try to use Zvkb.
https://github.com/riscv/riscv-crypto/blob/main/doc/vector/riscv-crypto-vector-zvkb.adoc
Some crypto algorithms are already checking for Zvkb instead of Zvbb.

Quoted 3 lines

Yeah, that's unfortunate that Zvkb got missed in binutils. However, since all Zvkb instructions are part of Zvbb, which is supported, assembling Zvkb instructions should still work --- right?

Not forgotten, but the Zvkb extension did not exist when the patchset was merged.
RISC-V extension support is typically merged when specifications are "frozen". This means a high bar for changes, but they are possible until the spec is ratified.
Often nothing is changed until ratification, but here Zvkb has been (re-)introduced.

Quoted 4 lines

I was not aware of this untils I read this email, so I just wrote a patch that fills the gap:
https://sourceware.org/pipermail/binutils/2023-November/130762.html

Thanks Christoph! That binutils patch looks good to me.

- Eric