RISC-V BLAKE2s Vector implementation
Subsystems (4)
cover RISC-V BLAKE2s Vector implementation
From: Björn Töpel <[email protected]>
Hi,
This is Andy's kernel mode vector V2 series [1], with my BLAKE2s AVX-512-to-RISC-V translation patch appended.
I've tagged it as RFC, since Andy's series is still not in-tree yet.
It's a first step towards a Vector aided Wireguard! ;-)
Cheers,
Björn
[1] https://lore.kernel.org/linux-riscv/[email protected]/
Andy Chiu (3):
riscv: sched: defer restoring Vector context for user
riscv: vector: do not pass task_struct into
riscv_v_vstate_{save,restore}()riscv: vector: allow kernel-mode Vector with preemption
Björn Töpel (1):
riscv: Add BLAKE2s V implementation
Greentime Hu (2):
riscv: Add support for kernel mode vector
riscv: Add vector extension XOR implementation
arch/riscv/Kbuild | 2 +- arch/riscv/Kconfig | 10 ++ arch/riscv/crypto/Kconfig | 16 +++ arch/riscv/crypto/Makefile | 6 + arch/riscv/crypto/blake2s-glue.c | 39 ++++++ arch/riscv/crypto/blake2s-v.S | 164 +++++++++++++++++++++++++ arch/riscv/include/asm/entry-common.h | 13 ++ arch/riscv/include/asm/processor.h | 2 + arch/riscv/include/asm/simd.h | 52 ++++++++ arch/riscv/include/asm/thread_info.h | 6 + arch/riscv/include/asm/vector.h | 50 ++++++-- arch/riscv/include/asm/xor.h | 82 +++++++++++++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 2 + arch/riscv/kernel/entry.S | 45 +++++++ arch/riscv/kernel/kernel_mode_vector.c | 146 ++++++++++++++++++++++ arch/riscv/kernel/process.c | 10 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 4 +- arch/riscv/kernel/vector.c | 5 +- arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 ++++++++++++ crypto/Kconfig | 3 + drivers/net/Kconfig | 1 + 24 files changed, 725 insertions(+), 18 deletions(-) create mode 100644 arch/riscv/crypto/Kconfig create mode 100644 arch/riscv/crypto/Makefile create mode 100644 arch/riscv/crypto/blake2s-glue.c create mode 100644 arch/riscv/crypto/blake2s-v.S create mode 100644 arch/riscv/include/asm/simd.h create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c create mode 100644 arch/riscv/lib/xor.S
base-commit: 0bb80ecc33a8fb5a682236443c1e740d5c917d1d
Signature · 1 line
2.39.2
RFC PATCH 1/6 riscv: sched: defer restoring Vector context for user
From: Andy Chiu <[email protected]>
User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register.
Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times for the upcoming kenel-mode Vector patches.
--- arch/riscv/include/asm/entry-common.h | 13 +++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/include/asm/vector.h | 11 ++++++++++- arch/riscv/kernel/process.c | 2 ++ arch/riscv/kernel/signal.c | 2 +- arch/riscv/kernel/vector.c | 2 +- 6 files changed, 29 insertions(+), 3 deletions(-)
arch/riscv/include/asm/entry-common.h +13 -0
@@ -4,6 +4,19 @@#define _ASM_RISCV_ENTRY_COMMON_H#include <asm/stacktrace.h> +#include <asm/thread_info.h> +#include <asm/vector.h> + +static inline void arch_exit_to_user_mode_prepare (struct pt_regs *regs, +unsigned long ti_work) +{ +if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { +clear_thread_flag (TIF_RISCV_V_DEFER_RESTORE ); +riscv_v_vstate_restore (current, regs); + } +} + +#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_preparevoid handle_page_fault (struct pt_regs *regs);void handle_break (struct pt_regs *regs);
arch/riscv/include/asm/thread_info.h +2 -0
@@ -93,12 +93,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define TIF_NOTIFY_SIGNAL 9/* signal notifications exist */ #define TIF_UPROBE 10/* uprobe breakpoint or singlestep */ #define TIF_32BIT 11/* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12/* restore Vector before returing to user */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL)#define _TIF_UPROBE (1 << TIF_UPROBE) +#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE)#define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \
arch/riscv/include/asm/vector.h +10 -1
@@ -184,6 +184,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task, } } +static inline void riscv_v_vstate_set_restore (struct task_struct *task, +struct pt_regs *regs) +{ +if ((regs->status &SR_VS ) !=SR_VS_OFF ) { +set_tsk_thread_flag (task,TIF_RISCV_V_DEFER_RESTORE ); +riscv_v_vstate_on (regs); + } +} +static inline void __switch_to_vector (struct task_struct *prev,struct task_struct *next) {
@@ -191,7 +200,7 @@ static inline void __switch_to_vector(struct task_struct *prev, regs =task_pt_regs (prev);riscv_v_vstate_save (prev, regs); -riscv_v_vstate_restore (next,task_pt_regs (next)); +riscv_v_vstate_set_restore (next,task_pt_regs (next)); }void riscv_v_vstate_ctrl_init (struct task_struct *tsk);
arch/riscv/kernel/process.c +2 -0
@@ -153,6 +153,7 @@ void flush_thread(void)riscv_v_vstate_off (task_pt_regs (current));kfree (current->thread.vstate.datap);memset (¤t->thread.vstate,0 ,sizeof (struct __riscv_v_ext_state )); +clear_tsk_thread_flag (current,TIF_RISCV_V_DEFER_RESTORE );#endif }
@@ -169,6 +170,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src;/* clear entire V context, including datap for a new task */ memset (&dst->thread.vstate,0 ,sizeof (struct __riscv_v_ext_state )); +clear_tsk_thread_flag (dst,TIF_RISCV_V_DEFER_RESTORE );return 0 ; }
arch/riscv/kernel/signal.c +1 -1
@@ -134,7 +134,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec)if (unlikely (err))return err; -riscv_v_vstate_restore (current, regs); +riscv_v_vstate_set_restore (current, regs);return err; }
arch/riscv/kernel/vector.c +1 -1
@@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs)return true; }riscv_v_vstate_on (regs); -riscv_v_vstate_restore (current, regs); +riscv_v_vstate_set_restore (current, regs);return true; }
Signature · 1 line
2.39.2
RFC PATCH 2/6 riscv: Add support for kernel mode vector
In reply to Björn TöpelFrom: Greentime Hu <[email protected]>
Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c
These are needed to wrap uses of vector in kernel mode.
--- arch/riscv/include/asm/simd.h | 50 ++++++++++++ arch/riscv/include/asm/vector.h | 2 + arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 101 +++++++++++++++++++++++++ 4 files changed, 154 insertions(+) create mode 100644 arch/riscv/include/asm/simd.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
arch/riscv/include/asm/simd.h new file +50 -0
@@ -0,0 +1,50 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Copyright (C) 2017 Linaro Ltd. <[email protected]> +* Copyright (C) 2023 SiFive +*/ + +#ifndef __ASM_SIMD_H +#define __ASM_SIMD_H + +#include <linux/compiler.h> +#include <linux/irqflags.h> +#include <linux/percpu.h> +#include <linux/preempt.h> +#include <linux/types.h> + +#ifdef CONFIG_RISCV_ISA_V + +DECLARE_PER_CPU (bool, vector_context_busy); + +/* +* may_use_simd - whether it is allowable at this time to issue vector +* instructions or access the vector register file +* +* Callers must not assume that the result remains true beyond the next +* preempt_enable() or return from softirq context. +*/ +static __must_check inline bool may_use_simd (void ) +{ +/* +* vector_context_busy is only set while preemption is disabled, +* and is clear whenever preemption is enabled. Since +* this_cpu_read() is atomic w.r.t. preemption, vector_context_busy +* cannot change under our feet -- if it's set we cannot be +* migrated, and if it's clear we cannot be migrated to a CPU +* where it is set. +*/ +return !in_irq () && !irqs_disabled () && !in_nmi () && + !this_cpu_read (vector_context_busy); +} + +#else /* ! CONFIG_RISCV_ISA_V */ + +static __must_check inline bool may_use_simd (void ) +{ +return false; +} + +#endif /* ! CONFIG_RISCV_ISA_V */ + +#endif
arch/riscv/include/asm/vector.h +2 -0
@@ -22,6 +22,8 @@extern unsigned long riscv_v_vsize;int riscv_v_setup_vsize (void );bool riscv_v_first_use_handler (struct pt_regs *regs); +void kernel_vector_begin (void ); +void kernel_vector_end (void ); static __always_inline bool has_vector(void) {
arch/riscv/kernel/Makefile +1 -0
@@ -62,6 +62,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_M_MODE) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_RISCV_ISA_V) += vector.o +obj-$(CONFIG_RISCV_ISA_V) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o
arch/riscv/kernel/kernel_mode_vector.c new file +101 -0
101 lines not on this page · raw message
Signature · 7 lines
2.39.2 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 3/6 riscv: Add vector extension XOR implementation
In reply to Björn TöpelFrom: Greentime Hu <[email protected]>
This patch adds support for vector optimized XOR and it is tested in qemu.
--- arch/riscv/include/asm/xor.h | 82 ++++++++++++++++++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++++++++++++ 3 files changed, 164 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.S
arch/riscv/include/asm/xor.h new file +82 -0
@@ -0,0 +1,82 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Copyright (C) 2021 SiFive +*/ + +#include <linux/hardirq.h> +#include <asm-generic/xor.h> +#ifdef CONFIG_RISCV_ISA_V +#include <asm/vector.h> +#include <asm/switch_to.h> + +void xor_regs_2_ (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2); +void xor_regs_3_ (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3); +void xor_regs_4_ (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3, +const unsigned long *__restrict p4); +void xor_regs_5_ (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3, +const unsigned long *__restrict p4, +const unsigned long *__restrict p5); + +static void xor_vector_2 (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2) +{ +kernel_vector_begin (); +xor_regs_2_ (bytes, p1, p2); +kernel_vector_end (); +} + +static void xor_vector_3 (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3) +{ +kernel_vector_begin (); +xor_regs_3_ (bytes, p1, p2, p3); +kernel_vector_end (); +} + +static void xor_vector_4 (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3, +const unsigned long *__restrict p4) +{ +kernel_vector_begin (); +xor_regs_4_ (bytes, p1, p2, p3, p4); +kernel_vector_end (); +} + +static void xor_vector_5 (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3, +const unsigned long *__restrict p4, +const unsigned long *__restrict p5) +{ +kernel_vector_begin (); +xor_regs_5_ (bytes, p1, p2, p3, p4, p5); +kernel_vector_end (); +} + +static struct xor_block_template xor_block_rvv = { + .name ="rvv" , + .do_2 = xor_vector_2, + .do_3 = xor_vector_3, + .do_4 = xor_vector_4, + .do_5 = xor_vector_5 +}; + +#undef XOR_TRY_TEMPLATES +#define XOR_TRY_TEMPLATES \ + do { \ + xor_speed(&xor_block_8regs); \ + xor_speed(&xor_block_32regs); \ + if (has_vector()) { \ + xor_speed(&xor_block_rvv);\ + } \ + } while (0) +#endif
arch/riscv/lib/Makefile +1 -0
@@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +lib-$(CONFIG_RISCV_ISA_V) += xor.o
arch/riscv/lib/xor.S new file +81 -0
81 lines not on this page · raw message
Signature · 7 lines
2.39.2 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 4/6 riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}()
In reply to Björn TöpelFrom: Andy Chiu <[email protected]>
riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused.
--- arch/riscv/include/asm/entry-common.h | 2 +- arch/riscv/include/asm/vector.h | 14 +++++--------- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 2 +- 5 files changed, 9 insertions(+), 13 deletions(-)
arch/riscv/include/asm/entry-common.h +1 -1
@@ -12,7 +12,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, {if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) {clear_thread_flag (TIF_RISCV_V_DEFER_RESTORE ); -riscv_v_vstate_restore (current, regs); +riscv_v_vstate_restore (¤t->thread.vstate, regs); } }
arch/riscv/include/asm/vector.h +5 -9
@@ -164,23 +164,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs)__riscv_v_vstate_dirty (regs); } -static inline void riscv_v_vstate_save (struct task_struct *task, +static inline void riscv_v_vstate_save (struct __riscv_v_ext_state *vstate,struct pt_regs *regs) {if ((regs->status &SR_VS ) ==SR_VS_DIRTY ) { -struct __riscv_v_ext_state *vstate = &task->thread.vstate; -__riscv_v_vstate_save (vstate, vstate->datap);__riscv_v_vstate_clean (regs); } } -static inline void riscv_v_vstate_restore (struct task_struct *task, +static inline void riscv_v_vstate_restore (struct __riscv_v_ext_state *vstate,struct pt_regs *regs) {if ((regs->status &SR_VS ) !=SR_VS_OFF ) { -struct __riscv_v_ext_state *vstate = &task->thread.vstate; -__riscv_v_vstate_restore (vstate, vstate->datap);__riscv_v_vstate_clean (regs); }
@@ -201,7 +197,7 @@ static inline void __switch_to_vector(struct task_struct *prev,struct pt_regs *regs; regs =task_pt_regs (prev); -riscv_v_vstate_save (prev, regs); +riscv_v_vstate_save (&prev->thread.vstate, regs);riscv_v_vstate_set_restore (next,task_pt_regs (next)); }
@@ -219,8 +215,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; }static inline bool riscv_v_vstate_ctrl_user_allowed (void ) {return false; }#define riscv_v_vsize (0)#define riscv_v_vstate_discard (regs) do {} while (0) -#define riscv_v_vstate_save (task, regs) do {} while (0) -#define riscv_v_vstate_restore (task, regs) do {} while (0) +#define riscv_v_vstate_save (vstate, regs) do {} while (0) +#define riscv_v_vstate_restore (vstate, regs) do {} while (0)#define __switch_to_vector (__prev, __next) do {} while (0)#define riscv_v_vstate_off (regs) do {} while (0)#define riscv_v_vstate_on (regs) do {} while (0)
arch/riscv/kernel/kernel_mode_vector.c +1 -1
@@ -68,7 +68,7 @@ void kernel_vector_begin(void)BUG_ON (!may_use_simd ()); -riscv_v_vstate_save (current,task_pt_regs (current)); +riscv_v_vstate_save (¤t->thread.vstate,task_pt_regs (current));get_cpu_vector_context ();
arch/riscv/kernel/ptrace.c +1 -1
@@ -100,7 +100,7 @@ static int riscv_vr_get(struct task_struct *target,* copying them to membuf. */ if (target == current) -riscv_v_vstate_save (current,task_pt_regs (current)); +riscv_v_vstate_save (¤t->thread.vstate,task_pt_regs (current)); ptrace_vstate.vstart = vstate->vstart; ptrace_vstate.vl = vstate->vl;
arch/riscv/kernel/signal.c +1 -1
@@ -86,7 +86,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec)/* datap is designed to be 16 byte aligned for better performance */ WARN_ON (unlikely (!IS_ALIGNED ((unsigned long )datap,16 ))); -riscv_v_vstate_save (current, regs); +riscv_v_vstate_save (¤t->thread.vstate, regs);/* Copy everything of vstate but datap. */ err =__copy_to_user (&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state , datap));
Signature · 7 lines
2.39.2 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 5/6 riscv: vector: allow kernel-mode Vector with preemption
In reply to Björn TöpelFrom: Andy Chiu <[email protected]>
Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide trap_pt_regs to let context save/restore routine reference status.VS at which the trap takes place. The thread flag TIF_RISCV_V_KERNEL_MODE indicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs to kernel_vstate and restore V-regs immediately from kernel_vstate if the bit is set.
Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the RISCV_V_VSTATE_CTRL_PREEMPTIBLE mask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code.
Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context swicth overhead.
--- arch/riscv/Kconfig | 10 +++++ arch/riscv/include/asm/processor.h | 2 + arch/riscv/include/asm/simd.h | 4 +- arch/riscv/include/asm/thread_info.h | 4 ++ arch/riscv/include/asm/vector.h | 27 +++++++++++-- arch/riscv/kernel/asm-offsets.c | 2 + arch/riscv/kernel/entry.S | 45 ++++++++++++++++++++++ arch/riscv/kernel/kernel_mode_vector.c | 53 ++++++++++++++++++++++++-- arch/riscv/kernel/process.c | 8 +++- arch/riscv/kernel/vector.c | 3 +- 10 files changed, 148 insertions(+), 10 deletions(-)
arch/riscv/Kconfig +10 -0
@@ -520,6 +520,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE If you don't know what to do here, say Y. +config RISCV_ISA_V_PREEMPTIVE + bool "Run kernel-mode Vector with kernel preemption" + depends on PREEMPTION + depends on RISCV_ISA_V + default y + help + Ordinarily the kernel disables preemption before running in-kernel + Vector code. This config frees the kernel from disabling preemption + by adding memory on demand for tracking kernel's V-context. + config TOOLCHAIN_HAS_ZBB bool default y
arch/riscv/include/asm/processor.h +2 -0
@@ -82,6 +82,8 @@ struct thread_struct {unsigned long bad_cause;unsigned long vstate_ctrl;struct __riscv_v_ext_state vstate; +struct pt_regs *trap_pt_regs; +struct __riscv_v_ext_state kernel_vstate; };/* Whitelist the fstate from the task_struct for hardened usercopy */
arch/riscv/include/asm/simd.h +3 -1
@@ -12,6 +12,7 @@#include <linux/percpu.h> #include <linux/preempt.h> #include <linux/types.h> +#include <linux/thread_info.h> #ifdef CONFIG_RISCV_ISA_V
@@ -35,7 +36,8 @@ static __must_check inline bool may_use_simd(void)* where it is set. */ return !in_irq () && !irqs_disabled () && !in_nmi () && - !this_cpu_read(vector_context_busy); + !this_cpu_read (vector_context_busy) && + !test_thread_flag(TIF_RISCV_V_KERNEL_MODE); }#else /* ! CONFIG_RISCV_ISA_V */
arch/riscv/include/asm/thread_info.h +4 -0
@@ -94,6 +94,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define TIF_UPROBE 10/* uprobe breakpoint or singlestep */ #define TIF_32BIT 11/* compat-mode 32bit process */ #define TIF_RISCV_V_DEFER_RESTORE 12/* restore Vector before returing to user */ +#define TIF_RISCV_V_KERNEL_MODE 13/* kernel-mode Vector run with preemption-on */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)
@@ -101,9 +102,12 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL)#define _TIF_UPROBE (1 << TIF_UPROBE)#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) +#define _TIF_RISCV_V_KERNEL_MODE (1 << TIF_RISCV_V_KERNEL_MODE)#define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ _TIF_NOTIFY_SIGNAL | _TIF_UPROBE) +#define RISCV_V_VSTATE_CTRL_PREEMPTIBLE 0x20 +#endif /* _ASM_RISCV_THREAD_INFO_H */
arch/riscv/include/asm/vector.h +24 -3
@@ -196,9 +196,24 @@ static inline void __switch_to_vector(struct task_struct *prev, {struct pt_regs *regs; - regs =task_pt_regs (prev); -riscv_v_vstate_save (&prev->thread.vstate, regs); -riscv_v_vstate_set_restore (next,task_pt_regs (next)); +if (IS_ENABLED (CONFIG_RISCV_ISA_V_PREEMPTIVE ) && +test_tsk_thread_flag (prev,TIF_RISCV_V_KERNEL_MODE )) { + regs = prev->thread.trap_pt_regs; +WARN_ON (!regs); +riscv_v_vstate_save (&prev->thread.kernel_vstate, regs); + }else { + regs =task_pt_regs (prev); +riscv_v_vstate_save (&prev->thread.vstate, regs); + } + +if (IS_ENABLED (CONFIG_RISCV_ISA_V_PREEMPTIVE ) && +test_tsk_thread_flag (next,TIF_RISCV_V_KERNEL_MODE )) { + regs = next->thread.trap_pt_regs; +WARN_ON (!regs); +riscv_v_vstate_restore (&next->thread.kernel_vstate, regs); + }else { +riscv_v_vstate_set_restore (next,task_pt_regs (next)); + } }void riscv_v_vstate_ctrl_init (struct task_struct *tsk);
@@ -223,4 +238,10 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; }#endif /* CONFIG_RISCV_ISA_V */ +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE +void kernel_vector_allow_preemption (void ); +#else +#define kernel_vector_allow_preemption () do {} while (0) +#endif +#endif /* ! __ASM_RISCV_VECTOR_H */
arch/riscv/kernel/asm-offsets.c +2 -0
@@ -38,6 +38,8 @@ void asm_offsets(void)OFFSET (TASK_TI_PREEMPT_COUNT , task_struct, thread_info.preempt_count);OFFSET (TASK_TI_KERNEL_SP , task_struct, thread_info.kernel_sp);OFFSET (TASK_TI_USER_SP , task_struct, thread_info.user_sp); +OFFSET (TASK_THREAD_TRAP_REGP , task_struct, thread.trap_pt_regs); +OFFSET (TASK_THREAD_VSTATE_CTRL , task_struct, thread.vstate_ctrl);OFFSET (TASK_THREAD_F0 , task_struct, thread.fstate.f[0 ]);OFFSET (TASK_THREAD_F1 , task_struct, thread.fstate.f[1 ]);
arch/riscv/kernel/entry.S +45 -0
57 lines not on this page · raw message
arch/riscv/kernel/kernel_mode_vector.c +49 -4
78 lines not on this page · raw message
arch/riscv/kernel/process.c +7 -1
@@ -160,8 +160,11 @@ void flush_thread(void)void arch_release_task_struct (struct task_struct *tsk) {/* Free the vector context of datap. */ -if (has_vector ()) +if (has_vector ()) {kfree (tsk->thread.vstate.datap); +if (IS_ENABLED (CONFIG_RISCV_ISA_V_PREEMPTIVE )) +kfree (tsk->thread.kernel_vstate.datap); + } }int arch_dup_task_struct (struct task_struct *dst,struct task_struct *src)
@@ -170,7 +173,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src;/* clear entire V context, including datap for a new task */ memset (&dst->thread.vstate,0 ,sizeof (struct __riscv_v_ext_state )); +memset (&dst->thread.kernel_vstate,0 ,sizeof (struct __riscv_v_ext_state ));clear_tsk_thread_flag (dst,TIF_RISCV_V_DEFER_RESTORE ); +clear_tsk_thread_flag (dst,TIF_RISCV_V_KERNEL_MODE );return 0 ; }
@@ -205,6 +210,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) childregs->a0 =0 ;/* Return value of fork() */ p->thread.s[0 ] =0 ; } +kernel_vector_allow_preemption (); p->thread.ra = (unsigned long )ret_from_fork; p->thread.sp = (unsigned long )childregs;/* kernel sp */ return 0 ;
arch/riscv/kernel/vector.c +2 -1
@@ -122,7 +122,8 @@ static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt, ctrl |=VSTATE_CTRL_MAKE_NEXT (nxt);if (inherit) ctrl |=PR_RISCV_V_VSTATE_CTRL_INHERIT ; - tsk->thread.vstate_ctrl = ctrl; + tsk->thread.vstate_ctrl &= ~PR_RISCV_V_VSTATE_CTRL_MASK ; + tsk->thread.vstate_ctrl |= ctrl; }bool riscv_v_vstate_ctrl_user_allowed (void )
Signature · 7 lines
2.39.2 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
RFC PATCH 6/6 riscv: Add BLAKE2s V implementation
In reply to Björn TöpelFrom: Björn Töpel <[email protected]>
BLAKE2s is used in various places, most notably in Wireguard, and as of v5.17 in drivers/char/random.
Add a BLAKE2s implementation using the V-extension. This is a translation of the x86-64 AVX512 variant (arch/x86/crypto/blake2s-core.S) to the RISC-V V-extension.
The AVX512 variant requires registers >= 256b (ymm*), and hence this implementation requires a VLEN of >=256.
The implementation passes the kernel BLAKE2s selftest, and has been tested on spike and qemu.
Instruction-wise, the V-variant uses 60% less instructions than the generic, C-based, implementation.
--- arch/riscv/Kbuild | 2 +- arch/riscv/crypto/Kconfig | 16 +++ arch/riscv/crypto/Makefile | 6 ++ arch/riscv/crypto/blake2s-glue.c | 39 ++++++++ arch/riscv/crypto/blake2s-v.S | 164 +++++++++++++++++++++++++++++++ crypto/Kconfig | 3 + drivers/net/Kconfig | 1 + 7 files changed, 230 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/crypto/Kconfig create mode 100644 arch/riscv/crypto/Makefile create mode 100644 arch/riscv/crypto/blake2s-glue.c create mode 100644 arch/riscv/crypto/blake2s-v.S
arch/riscv/Kbuild +1 -1
@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -obj-y += kernel/ mm/ net/ +obj-y += kernel/ mm/ net/ crypto/ obj-$(CONFIG_BUILTIN_DTB) += boot/dts/ obj-y += errata/ obj-$(CONFIG_KVM) += kvm/
arch/riscv/crypto/Kconfig new file +16 -0
@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0 + +menu "Accelerated Cryptographic Algorithms for CPU (RISC-V)" + +config CRYPTO_BLAKE2S_RISCV + bool "Hash functions: BLAKE2s (V)" + depends on 64BIT && MMU && RISCV_ISA_V + select CRYPTO_LIB_BLAKE2S_GENERIC + select CRYPTO_ARCH_HAVE_LIB_BLAKE2S + help + BLAKE2s cryptographic hash function (RFC 7693) + + Architecture: riscv64 using: + - V extension (VLEN >= 256) + +endmenu
arch/riscv/crypto/Makefile new file +6 -0
@@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# RISC-V crypto algorithms + +obj-$(CONFIG_CRYPTO_BLAKE2S_RISCV) += libblake2s-riscv.o +libblake2s-riscv-y := blake2s-v.o blake2s-glue.o
arch/riscv/crypto/blake2s-glue.c new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 OR MIT +/* +* Copyright (C) 2022 Rivos, Inc. All Rights Reserved. +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <crypto/internal/blake2s.h> + +#include <linux/types.h> +#include <linux/minmax.h> +#include <linux/sizes.h> + +#include <linux/sched.h> +#include <linux/sched/task_stack.h> +#include <asm/simd.h> +#include <asm/vector.h> + +asmlinkage void blake2s_compress_vector(struct blake2s_state *state, const u8 *block, +const size_t nblocks,const u32 inc); + +void blake2s_compress (struct blake2s_state *state,const u8 *block,size_t nblocks,const u32 inc) +{ +if (!(has_vector () && riscv_v_vsize >=256 *32 /8 ) || !may_use_simd ()) { +blake2s_compress_generic (state, block, nblocks, inc); +return ; + } + +do { +const size_t blocks =min_t (size_t, nblocks,SZ_4K /BLAKE2S_BLOCK_SIZE ); + +kernel_vector_begin (); +blake2s_compress_vector (state, block, blocks, inc); +kernel_vector_end (); + + nblocks -= blocks; + block += blocks *BLAKE2S_BLOCK_SIZE ; + }while (nblocks); +} +EXPORT_SYMBOL (blake2s_compress);
arch/riscv/crypto/blake2s-v.S new file +164 -0
164 lines not on this page · raw message
crypto/Kconfig +3 -0
@@ -1436,6 +1436,9 @@ endif if PPC source "arch/powerpc/crypto/Kconfig" endif +if RISCV +source "arch/riscv/crypto/Kconfig" +endif if S390 source "arch/s390/crypto/Kconfig" endif
drivers/net/Kconfig +1 -0
@@ -94,6 +94,7 @@ config WIREGUARD select CRYPTO_CHACHA_MIPS if CPU_MIPS32_R2 select CRYPTO_POLY1305_MIPS if MIPS select CRYPTO_CHACHA_S390 if S390 + select CRYPTO_BLAKE2S_RISCV if RISCV && 64BIT help WireGuard is a secure, fast, and easy to use replacement for IPSec that uses modern cryptography and clever networking tricks. It's
Signature · 7 lines
2.39.2 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [RFC PATCH 6/6] riscv: Add BLAKE2s V implementation
Quoted 15 lines · Björn Töpel
From: Björn Töpel <[email protected]>
BLAKE2s is used in various places, most notably in Wireguard, and as of v5.17 in
drivers/char/random.Add a BLAKE2s implementation using the V-extension. This is a translation of the x86-64 AVX512 variant (
arch/x86/crypto/blake2s-core.S) to the RISC-V V-extension.The AVX512 variant requires registers >= 256b (ymm*), and hence this implementation requires a VLEN of >=256.
The implementation passes the kernel BLAKE2s selftest, and has been tested on spike and qemu.
Quoted 6 lines
Instruction-wise, the V-variant uses 60% less instructions than the generic, C-based, implementation.
Signed-off-by Björn Töpel <[email protected]>arch/riscv/crypto/blake2s-v.S | 164 +++++++++++++++++++++++++++++++
This file is currently being built for !RISCV_ISA_V, leading to build errors
for obvious reasons. Eg:../arch/riscv/crypto/blake2s-v.S:62:2: error: instruction requires the following: 'V' (Vector Extension for Application Processors), 'Zve32x' or 'Zve64x' (Vector Extensions for Embedded Processors)
Re: [RFC PATCH 6/6] riscv: Add BLAKE2s V implementation
Quoted 21 lines · Conor Dooley
Quoted 19 lines · Björn Töpel
From: Björn Töpel <[email protected]>
BLAKE2s is used in various places, most notably in Wireguard, and as of v5.17 in
drivers/char/random.Add a BLAKE2s implementation using the V-extension. This is a translation of the x86-64 AVX512 variant (
arch/x86/crypto/blake2s-core.S) to the RISC-V V-extension.The AVX512 variant requires registers >= 256b (ymm*), and hence this implementation requires a VLEN of >=256.
The implementation passes the kernel BLAKE2s selftest, and has been tested on spike and qemu.
Instruction-wise, the V-variant uses 60% less instructions than the generic, C-based, implementation.
Signed-off-by Björn Töpel <[email protected]>
Quoted 8 lines
Quoted 1 line
arch/riscv/crypto/blake2s-v.S | 164 +++++++++++++++++++++++++++++++This file is currently being built for !RISCV_ISA_V, leading to build errors for obvious reasons. Eg:
../arch/riscv/crypto/blake2s-v.S:62:2: error: instruction requires the
following: 'V' (Vector Extension for Application Processors), 'Zve32x' or 'Zve64x' (Vector Extensions for Embedded Processors)
Thanks, noted for the next rev!
Re: [RFC PATCH 6/6] riscv: Add BLAKE2s V implementation
In reply to Björn TöpelQuoted 170 lines · Björn Töpel
From: Björn Töpel <[email protected]>
BLAKE2s is used in various places, most notably in Wireguard, and as of v5.17 in
drivers/char/random.Add a BLAKE2s implementation using the V-extension. This is a translation of the x86-64 AVX512 variant (
arch/x86/crypto/blake2s-core.S) to the RISC-V V-extension.The AVX512 variant requires registers >= 256b (ymm*), and hence this implementation requires a VLEN of >=256.
The implementation passes the kernel BLAKE2s selftest, and has been tested on spike and qemu.
Instruction-wise, the V-variant uses 60% less instructions than the generic, C-based, implementation.
Signed-off-by Björn Töpel <[email protected]>--- arch/riscv/Kbuild | 2 +- arch/riscv/crypto/Kconfig | 16 +++ arch/riscv/crypto/Makefile | 6 ++ arch/riscv/crypto/blake2s-glue.c | 39 ++++++++ arch/riscv/crypto/blake2s-v.S | 164 +++++++++++++++++++++++++++++++ crypto/Kconfig | 3 + drivers/net/Kconfig | 1 + 7 files changed, 230 insertions(+), 1 deletion(-) create mode 100644 arch/riscv/crypto/Kconfig create mode 100644 arch/riscv/crypto/Makefile create mode 100644 arch/riscv/crypto/blake2s-glue.c create mode 100644 arch/riscv/crypto/blake2s-v.Sarch/riscv/Kbuild +1
-1@@ -1,6 +1,6 @@ # SPDX-License-Identifier: GPL-2.0-only -obj-y += kernel/ mm/ net/ +obj-y += kernel/ mm/ net/ crypto/ obj-$(CONFIG_BUILTIN_DTB) += boot/dts/ obj-y += errata/ obj-$(CONFIG_KVM) += kvm/arch/riscv/crypto/Kconfig new file +16
-0@@ -0,0 +1,16 @@ +# SPDX-License-Identifier: GPL-2.0 + +menu "Accelerated Cryptographic Algorithms for CPU (RISC-V)" + +config CRYPTO_BLAKE2S_RISCV + bool "Hash functions: BLAKE2s (V)" + depends on 64BIT && MMU && RISCV_ISA_V + select CRYPTO_LIB_BLAKE2S_GENERIC + select CRYPTO_ARCH_HAVE_LIB_BLAKE2S + help + BLAKE2s cryptographic hash function (RFC 7693) + + Architecture: riscv64 using: + - V extension (VLEN >= 256) + +endmenuarch/riscv/crypto/Makefile new file +6
-0@@ -0,0 +1,6 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# RISC-V crypto algorithms + +obj-$(CONFIG_CRYPTO_BLAKE2S_RISCV) += libblake2s-riscv.o +libblake2s-riscv-y := blake2s-v.o blake2s-glue.oarch/riscv/crypto/blake2s-glue.c new file +39
-0@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: GPL-2.0 OR MIT +/* +* Copyright (C) 2022 Rivos, Inc. All Rights Reserved. +* Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. +*/ + +#include <crypto/internal/blake2s.h> + +#include <linux/types.h> +#include <linux/minmax.h> +#include <linux/sizes.h> + +#include <linux/sched.h> +#include <linux/sched/task_stack.h> +#include <asm/simd.h> +#include <asm/vector.h> + +asmlinkage void blake2s_compress_vector(struct blake2s_state *state, const u8 *block, +const size_t nblocks,const u32 inc); + +void blake2s_compress (struct blake2s_state *state,const u8 *block,size_t nblocks,const u32 inc) +{ +if (!(has_vector () && riscv_v_vsize >=256 *32 /8 ) || !may_use_simd ()) { +blake2s_compress_generic (state, block, nblocks, inc); +return ; + } + +do { +const size_t blocks =min_t (size_t, nblocks,SZ_4K /BLAKE2S_BLOCK_SIZE ); + +kernel_vector_begin (); +blake2s_compress_vector (state, block, blocks, inc); +kernel_vector_end (); + + nblocks -= blocks; + block += blocks *BLAKE2S_BLOCK_SIZE ; + }while (nblocks); +} +EXPORT_SYMBOL (blake2s_compress);arch/riscv/crypto/blake2s-v.S new file +40
-0@@ -0,0 +1,164 @@ +/* SPDX-License-Identifier: GPL-2.0 OR MIT */ +/* + * BLAKE2s RISC-V V implementation + * + * Copyright (C) 2022 Rivos, Inc. All Rights Reserved. + * Copyright (C) 2015-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved. + * Copyright (C) 2017-2019 Samuel Neves <[email protected]>. All Rights Reserved. + */ + +#include <linux/linkage.h> +#include <asm/asm.h> + +.section .rodata, "a", @progbits + +.align 8 +IV: +.octa 0xA54FF53A3C6EF372BB67AE856A09E667 +.octa 0x5BE0CD191F83D9AB9B05688C510E527F + +.align 8 +SIGMA2: +.long 0, 2, 4, 6, 1, 3, 5, 7, 14, 8, 10, 12, 15, 9, 11, 13 +.long 8, 2, 13, 15, 10, 9, 12, 3, 6, 4, 0, 14, 5, 11, 1, 7 +.long 11, 13, 8, 6, 5, 10, 14, 3, 2, 4, 12, 15, 1, 0, 7, 9 +.long 11, 10, 7, 0, 8, 15, 1, 13, 3, 6, 2, 12, 4, 14, 9, 5 +.long 4, 10, 9, 14, 15, 0, 11, 8, 1, 7, 3, 13, 2, 5, 6, 12 +.long 2, 11, 4, 15, 14, 3, 10, 8, 13, 6, 5, 7, 0, 12, 1, 9 +.long 4, 8, 15, 9, 14, 11, 13, 5, 3, 2, 1, 12, 6, 10, 7, 0 +.long 6, 13, 0, 14, 12, 2, 1, 11, 15, 4, 5, 8, 7, 9, 3, 10 +.long 15, 5, 4, 13, 10, 7, 3, 11, 12, 2, 0, 6, 9, 8, 1, 14 +.long 8, 7, 14, 11, 13, 15, 0, 12, 10, 4, 5, 6, 3, 2, 1, 9 + +.align 8 +GATHER_DIAG1: +.long 3, 0, 1, 2 +.align 8 +GATHER_DIAG2: +.long 2, 3, 0, 1 +.align 8 +GATHER_DIAG3:
arch/riscv/crypto/blake2s-v.S line 50
Quoted 10 lines
arch/riscv/crypto/blake2s-v.S new file +10
-0+.long 1, 2, 3, 0 + +.section .text + +.macro VROR REG BITS TMPREG + vmv.v.v \TMPREG, \REG + vsrl.vi \REG, \REG, \BITS + vsll.vi \TMPREG, \TMPREG, 32-\BITS + vor.vv \REG, \REG, \TMPREG +.endm
This seems like it would be a good candidate to use the Zvkb extension that has vector rotate [1]. It would be a pain to use because you need two versions of the roundloop, but you could get away with 1 extra branch/nop and save 3 instructions per VROR which looks like it happens 8*10 times, so 239 fewer instructions.
[1] https://github.com/riscv/riscv-crypto/releases
- Charlie
Quoted 148 lines
arch/riscv/crypto/blake2s-v.S new file +114
-0+ +/* + * void blake2s_compress_vector(struct blake2s_state *state, + * const u8 *block, size_t nblocks, const u32 inc) + * + * blake2s_compress_vector is a translation of the AVX512 variant + * (arch/x86/blake2s-core.S) to RISC-V V. The implementation requires + * VLEN>=256b. + * + */ +SYM_FUNC_START(blake2s_compress_vector) + vsetivli t0, 1, e32, m1, ta, ma + vmv.v.x v5, a3 + vsetivli t0, 4, e32, m1, ta, ma + vle32.v v0, (a0) + addi t0, a0, 0x10 + vle32.v v1, (t0) + addi t0, t0, 0x10 + vle32.v v4, (t0) + la t0, IV + vle32.v v14, (t0) + la t0, IV+16 + vle32.v v15, (t0) + la t0, GATHER_DIAG1 + vle32.v v16, (t0) + la t0, GATHER_DIAG2 + vle32.v v18, (t0) + la t0, GATHER_DIAG3 + vle32.v v19, (t0) +.Lblake2s_compress_vector_mainloop: + vmv.v.v v10, v0 + vmv.v.v v11, v1 + vadd.vv v4, v5, v4 + vmv.v.v v2, v14 + vxor.vv v3, v15, v4 + vsetivli t0, 8, e32, m1, ta, ma + vle32.v v6, (a1) + addi a1, a1, 0x20 + vle32.v v7, (a1) + addi a1, a1, 0x20 + la t1, SIGMA2 + li t2, 0xa +.Lblake2s_compress_vector_roundloop: + vsetivli t0, 8, e32, m1, ta, ma + vle32.v v12, (t1) + addi t1, t1, 0x20 + vle32.v v13, (t1) + addi t1, t1, 0x20 + vsetivli t0, 16, e32, m2, ta, ma + vrgather.vv v8, v6, v12 + vmv.v.v v6, v8 + vsetivli t0, 4, e32, m1, ta, ma + vadd.vv v0, v8, v0 + vadd.vv v0, v1, v0 + vxor.vv v3, v0, v3 + VROR v3 16 v17 + vadd.vv v2, v3, v2 + vxor.vv v1, v2, v1 + VROR v1 12 v17 + vsetivli t0, 8, e32, m1, ta, ma + vslidedown.vi v8, v8, 4 + vsetivli t0, 4, e32, m1, ta, ma + vadd.vv v0, v8, v0 + vadd.vv v0, v1, v0 + vxor.vv v3, v0, v3 + VROR v3 8 v17 + vadd.vv v2, v3, v2 + vxor.vv v1, v2, v1 + VROR v1, 7, v17 + vmv.v.v v17, v0 + vrgather.vv v0, v17, v16 + vmv.v.v v17, v3 + vrgather.vv v3, v17, v18 + vmv.v.v v17, v2 + vrgather.vv v2, v17, v19 + vadd.vv v0, v9, v0 + vadd.vv v0, v1, v0 + vxor.vv v3, v0, v3 + VROR v3 16 v17 + vadd.vv v2, v3, v2 + vxor.vv v1, v2, v1 + VROR v1 12 v17 + vsetivli t0, 8, e32, m1, ta, ma + vslidedown.vi v9, v9, 4 + vsetivli t0, 4, e32, m1, ta, ma + vadd.vv v0, v9, v0 + vadd.vv v0, v1, v0 + vxor.vv v3, v0, v3 + VROR v3, 8, v17 + vadd.vv v2, v3, v2 + vxor.vv v1, v2, v1 + VROR v1 7 v17 + vmv.v.v v17, v0 + vrgather.vv v0, v17, v19 + vmv.v.v v17, v3 + vrgather.vv v3, v17, v18 + vmv.v.v v17, v2 + vrgather.vv v2, v17, v16 + addi t2, t2, -1 + bne t2, x0, .Lblake2s_compress_vector_roundloop + vxor.vv v0, v10, v0 + vxor.vv v1, v11, v1 + vxor.vv v0, v2, v0 + vxor.vv v1, v3, v1 + addi a2, a2, -1 + bne a2, x0, .Lblake2s_compress_vector_mainloop + vse32.v v0, (a0) + addi a0, a0, 0x10 + vse32.v v1, (a0) + addi a0, a0, 0x10 + vse32.v v4, (a0) + ret +SYM_FUNC_END(blake2s_compress_vector) +crypto/Kconfig +3
-0@@ -1436,6 +1436,9 @@ endif if PPC source "arch/powerpc/crypto/Kconfig" endif +if RISCV +source "arch/riscv/crypto/Kconfig" +endif if S390 source "arch/s390/crypto/Kconfig" endifdrivers/net/Kconfig +1
-0@@ -94,6 +94,7 @@ config WIREGUARD select CRYPTO_CHACHA_MIPS if CPU_MIPS32_R2 select CRYPTO_POLY1305_MIPS if MIPS select CRYPTO_CHACHA_S390 if S390 + select CRYPTO_BLAKE2S_RISCV if RISCV && 64BIT help WireGuard is a secure, fast, and easy to use replacement for IPSec that uses modern cryptography and clever networking tricks. It'sSignature · 7 lines
2.39.2 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [RFC PATCH 6/6] riscv: Add BLAKE2s V implementation
Quoted 13 lines · Charlie Jenkins
arch/riscv/crypto/blake2s-v.S line 50
Quoted 6 lines
+.macro VROR REG BITS TMPREG
arch/riscv/crypto/blake2s-v.S new file +5
-0+ vmv.v.v \TMPREG, \REG + vsrl.vi \REG, \REG, \BITS + vsll.vi \TMPREG, \TMPREG, 32-\BITS + vor.vv \REG, \REG, \TMPREG +.endmThis seems like it would be a good candidate to use the Zvkb extension that has vector rotate [1]. It would be a pain to use because you need two versions of the roundloop, but you could get away with 1 extra branch/nop and save 3 instructions per VROR which looks like it happens 8*10 times, so 239 fewer instructions.
Quoted 1 line
Indeed! Thanks!
Björn
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 10 lines · Björn Töpel
From: Björn Töpel <[email protected]>
Hi,
This is Andy's kernel mode vector V2 series [1], with my BLAKE2s AVX-512-to-RISC-V translation patch appended.
I've tagged it as RFC, since Andy's series is still not in-tree yet.
It's a first step towards a Vector aided Wireguard! ;-)
This has the same problems as Andy's stuff & doesn't build properly for the automation. What is the plan between yourself and Andy for submitting a version of the in-kernel vector support that passes build testing?
Cheers,
Conor.
Quoted 59 lines
Cheers,
Björn[1] https://lore.kernel.org/linux-riscv/[email protected]/
Andy Chiu (3):
riscv: sched: defer restoring Vector context for user
riscv: vector: do not passtask_structintoriscv_v_vstate_{save,restore}()riscv: vector: allow kernel-mode Vector with preemption
Björn Töpel (1):
riscv: Add BLAKE2s V implementationGreentime Hu (2):
riscv: Add support for kernel mode vector
riscv: Add vector extension XOR implementationarch/riscv/Kbuild | 2 +- arch/riscv/Kconfig | 10 ++ arch/riscv/crypto/Kconfig | 16 +++ arch/riscv/crypto/Makefile | 6 + arch/riscv/crypto/blake2s-glue.c | 39 ++++++ arch/riscv/crypto/blake2s-v.S | 164 +++++++++++++++++++++++++ arch/riscv/include/asm/entry-common.h | 13 ++ arch/riscv/include/asm/processor.h | 2 + arch/riscv/include/asm/simd.h | 52 ++++++++ arch/riscv/include/asm/thread_info.h | 6 + arch/riscv/include/asm/vector.h | 50 ++++++-- arch/riscv/include/asm/xor.h | 82 +++++++++++++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 2 + arch/riscv/kernel/entry.S | 45 +++++++ arch/riscv/kernel/kernel_mode_vector.c | 146 ++++++++++++++++++++++ arch/riscv/kernel/process.c | 10 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 4 +- arch/riscv/kernel/vector.c | 5 +- arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 ++++++++++++ crypto/Kconfig | 3 + drivers/net/Kconfig | 1 + 24 files changed, 725 insertions(+), 18 deletions(-) create mode 100644 arch/riscv/crypto/Kconfig create mode 100644 arch/riscv/crypto/Makefile create mode 100644 arch/riscv/crypto/blake2s-glue.c create mode 100644 arch/riscv/crypto/blake2s-v.S create mode 100644 arch/riscv/include/asm/simd.h create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c create mode 100644 arch/riscv/lib/xor.Sbase-commit:
0bb80ecc33a8fb5a682236443c1e740d5c917d1dSignature · 1 line
2.39.2
Quoted 8 lines · Conor Dooley
Quoted 7 lines · Björn Töpel
From: Björn Töpel <[email protected]>
Hi,
This is Andy's kernel mode vector V2 series [1], with my BLAKE2s AVX-512-to-RISC-V translation patch appended.
Quoted 7 lines
Quoted 3 lines
I've tagged it as RFC, since Andy's series is still not in-tree yet.
It's a first step towards a Vector aided Wireguard! ;-)
This has the same problems as Andy's stuff & doesn't build properly for the automation. What is the plan between yourself and Andy for submitting a version of the in-kernel vector support that passes build testing?
I'll synch up with Andy! I'm not even sure the blake2s patch should part of the "in-kernel vector" series at all.
Thanks,
Björn
Quoted 12 lines · Björn Töpel
Quoted 10 lines · Conor Dooley
Quoted 9 lines · Björn Töpel
From: Björn Töpel <[email protected]>
Hi,
This is Andy's kernel mode vector V2 series [1], with my BLAKE2s AVX-512-to-RISC-V translation patch appended.
I've tagged it as RFC, since Andy's series is still not in-tree yet.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
It's a first step towards a Vector aided Wireguard! ;-)
This has the same problems as Andy's stuff & doesn't build properly for the automation. What is the plan between yourself and Andy for submitting a version of the in-kernel vector support that passes build testing?
I'll synch up with Andy! I'm not even sure the blake2s patch should part of the "in-kernel vector" series at all.
The in-kernel vector stuff should come with a user, otherwise it's dead code :)
Quoted 19 lines · Conor Dooley
Quoted 18 lines · Björn Töpel
Quoted 15 lines · Conor Dooley
Quoted 10 lines · Björn Töpel
From: Björn Töpel <[email protected]>
Hi,
This is Andy's kernel mode vector V2 series [1], with my BLAKE2s AVX-512-to-RISC-V translation patch appended.
I've tagged it as RFC, since Andy's series is still not in-tree yet.
It's a first step towards a Vector aided Wireguard! ;-)
This has the same problems as Andy's stuff & doesn't build properly for the automation. What is the plan between yourself and Andy for submitting a version of the in-kernel vector support that passes build testing?
Quoted 5 lines
Quoted 2 lines
I'll synch up with Andy! I'm not even sure the blake2s patch should part of the "in-kernel vector" series at all.
The in-kernel vector stuff should come with a user, otherwise it's dead code :)
Sure, just so we're on the same page; Patch 3 (Vector XOR) is a user from my perspective, no?
Quoted 21 lines · Björn Töpel
Quoted 19 lines · Conor Dooley
Quoted 18 lines · Björn Töpel
Quoted 15 lines · Conor Dooley
Quoted 10 lines · Björn Töpel
From: Björn Töpel <[email protected]>
Hi,
This is Andy's kernel mode vector V2 series [1], with my BLAKE2s AVX-512-to-RISC-V translation patch appended.
I've tagged it as RFC, since Andy's series is still not in-tree yet.
It's a first step towards a Vector aided Wireguard! ;-)
This has the same problems as Andy's stuff & doesn't build properly for the automation. What is the plan between yourself and Andy for submitting a version of the in-kernel vector support that passes build testing?
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I'll synch up with Andy! I'm not even sure the blake2s patch should part of the "in-kernel vector" series at all.
The in-kernel vector stuff should come with a user, otherwise it's dead code :)
Sure, just so we're on the same page; Patch 3 (Vector XOR) is a user from my perspective, no?
D'oh
Quoted 17 lines · Conor Dooley
Quoted 15 lines · Björn Töpel
Quoted 13 lines · Conor Dooley
Quoted 12 lines · Björn Töpel
Quoted 10 lines · Conor Dooley
Quoted 9 lines · Björn Töpel
From: Björn Töpel <[email protected]>
Hi,
This is Andy's kernel mode vector V2 series [1], with my BLAKE2s AVX-512-to-RISC-V translation patch appended.
I've tagged it as RFC, since Andy's series is still not in-tree yet.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
It's a first step towards a Vector aided Wireguard! ;-)
This has the same problems as Andy's stuff & doesn't build properly for the automation. What is the plan between yourself and Andy for submitting a version of the in-kernel vector support that passes build testing?
I'll synch up with Andy! I'm not even sure the blake2s patch should part of the "in-kernel vector" series at all.
Hi, yes, I have a plan to fix it recently. Please expect a respin of the kernel-mode vector in 1~2 weeks, if this doesn't seem too long to you.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The in-kernel vector stuff should come with a user, otherwise it's dead code :)
Sure, just so we're on the same page; Patch 3 (Vector XOR) is a user from my perspective, no?
D'oh
Thanks,
Andy