riscv: support kernel-mode Vector
Subsystems (2)
cover riscv: support kernel-mode Vector
This series provides support for running Vector code in kernel mode. The implementation is based on the v12 series of the Vector series, but with some additions. First, we introduce a mechanism to defer restoring Vector context for userspace programs (patch 1). This is similar to arm64 and x86's approaches when dealing with extra userspace register context. And it is benefitial to both Vector in user and kernel-mode. Then, patch 2, 3 add the kernel-mode Vector patch from v12 with minor modifications. At the end of the series, patch 4, 5, 6 add supports for making kernel-mode Vector code preemptible. We do this by adding kernel-mode Vector context, and keeping track of the frame where V context is last valid. We believe that enabling preemption of running V is a critical path for getting V more generally available in the kernel-mode. Besides, with status.VS, we can easily tell if saving/restoring V is required. This reduce the level of cost when running SIMD in kernel mode as compared to other arches. Other arches usually do not have a way to tell if extra context is dirty. Thus, if they also want to support running preemptible code with extra registers, then they must save/restore extra context at each context switch even if registers are not dirty.
The series is tested by loading a kernel module on a preemptive kernel. The module launches multiple kworkers which run Vector operations and verifies with scalar code. Also, the module provides userspace intefaces via fops to verify if we can run Vector code on syscall path.
Changes from the vector v12 series (for patch 2, 3):
- return a failure code when kernel_rvv_begin() fails.
- Do not immediately restore user's V context.
Andy Chiu (4):
riscv: sched: defer restoring Vector context for user
riscv: vector: do not pass task_struct into
riscv_v_vstate_{save,restore}() riscv: vector: allow kernel-mode Vector with preemption
riscv: vector: enable preemptive kernel-mode Vector to be built
Greentime Hu (2):
riscv: Add support for kernel mode vector
riscv: Add vector extension XOR implementation
arch/riscv/Kconfig | 10 ++ arch/riscv/include/asm/entry-common.h | 13 ++ arch/riscv/include/asm/processor.h | 2 + arch/riscv/include/asm/thread_info.h | 6 + arch/riscv/include/asm/vector.h | 50 +++++-- arch/riscv/include/asm/xor.h | 82 +++++++++++ arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/asm-offsets.c | 2 + arch/riscv/kernel/entry.S | 41 ++++++ arch/riscv/kernel/kernel_mode_vector.c | 180 +++++++++++++++++++++++++ arch/riscv/kernel/process.c | 10 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 4 +- arch/riscv/kernel/vector.c | 5 +- arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++ 16 files changed, 473 insertions(+), 17 deletions(-) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/kernel/kernel_mode_vector.c create mode 100644 arch/riscv/lib/xor.S
Signature · 7 lines
2.17.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH 1/6 riscv: sched: defer restoring Vector context for user
User's will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs context swicth from A->B->C, and returns to C's userspace, then there is no need for restoring B's V-register.
Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times for the upcoming kenel-mode Vector patches.
--- arch/riscv/include/asm/entry-common.h | 13 +++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/include/asm/vector.h | 11 ++++++++++- arch/riscv/kernel/process.c | 2 ++ arch/riscv/kernel/signal.c | 2 +- arch/riscv/kernel/vector.c | 2 +- 6 files changed, 29 insertions(+), 3 deletions(-)
arch/riscv/include/asm/entry-common.h +13 -0
@@ -4,6 +4,19 @@#define _ASM_RISCV_ENTRY_COMMON_H#include <asm/stacktrace.h> +#include <asm/thread_info.h> +#include <asm/vector.h> + +static inline void arch_exit_to_user_mode_prepare (struct pt_regs *regs, +unsigned long ti_work) +{ +if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) { +clear_thread_flag (TIF_RISCV_V_DEFER_RESTORE ); +riscv_v_vstate_restore (current, regs); + } +} + +#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_preparevoid handle_page_fault (struct pt_regs *regs);void handle_break (struct pt_regs *regs);
arch/riscv/include/asm/thread_info.h +2 -0
@@ -101,12 +101,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define TIF_NOTIFY_SIGNAL 9/* signal notifications exist */ #define TIF_UPROBE 10/* uprobe breakpoint or singlestep */ #define TIF_32BIT 11/* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL)#define _TIF_UPROBE (1 << TIF_UPROBE) +#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE)#define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \
arch/riscv/include/asm/vector.h +10 -1
@@ -183,6 +183,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task, } } +static inline void riscv_v_vstate_set_restore (struct task_struct *task, +struct pt_regs *regs) +{ +if ((regs->status &SR_VS ) !=SR_VS_OFF ) { +set_tsk_thread_flag (task,TIF_RISCV_V_DEFER_RESTORE ); +riscv_v_vstate_on (regs); + } +} +static inline void __switch_to_vector (struct task_struct *prev,struct task_struct *next) {
@@ -190,7 +199,7 @@ static inline void __switch_to_vector(struct task_struct *prev, regs =task_pt_regs (prev);riscv_v_vstate_save (prev, regs); -riscv_v_vstate_restore (next,task_pt_regs (next)); +riscv_v_vstate_set_restore (next,task_pt_regs (next)); }void riscv_v_vstate_ctrl_init (struct task_struct *tsk);
arch/riscv/kernel/process.c +2 -0
@@ -153,6 +153,7 @@ void flush_thread(void)riscv_v_vstate_off (task_pt_regs (current));kfree (current->thread.vstate.datap);memset (¤t->thread.vstate,0 ,sizeof (struct __riscv_v_ext_state )); +clear_tsk_thread_flag (current,TIF_RISCV_V_DEFER_RESTORE );#endif }
@@ -169,6 +170,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src;/* clear entire V context, including datap for a new task */ memset (&dst->thread.vstate,0 ,sizeof (struct __riscv_v_ext_state )); +clear_tsk_thread_flag (dst,TIF_RISCV_V_DEFER_RESTORE );return 0 ; }
arch/riscv/kernel/signal.c +1 -1
@@ -134,7 +134,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec)if (unlikely (err))return err; -riscv_v_vstate_restore (current, regs); +riscv_v_vstate_set_restore (current, regs);return err; }
arch/riscv/kernel/vector.c +1 -1
@@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs)return true; }riscv_v_vstate_on (regs); -riscv_v_vstate_restore (current, regs); +riscv_v_vstate_set_restore (current, regs);return true; }
Signature · 7 lines
2.17.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [v1, 1/6] riscv: sched: defer restoring Vector context for user
Hey Andy,
Small bit of minor nitpickery..
Quoted 1 line · Andy Chiu
User's will use its Vector registers only after the kernel really
Looks like the ' here can be removed.
Quoted 6 lines
returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs context swicth from
A->B->C, and returns
"a context switch"
Quoted 1 line
to C's userspace, then there is no need for restoring B's V-register.
"to restore"
Quoted 13 lines
Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times for the upcoming kenel-mode Vector patches.
Signed-off-by Andy Chiu <[email protected]>---
arch/riscv/include/asm/thread_info.h +0
-0
arch/riscv/include/asm/thread_info.h int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); · line 104
Quoted 4 lines
arch/riscv/include/asm/thread_info.h +1
-0@@ -101,12 +101,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define TIF_NOTIFY_SIGNAL 9/* signal notifications exist */ #define TIF_UPROBE 10/* uprobe breakpoint or singlestep */ #define TIF_32BIT 11/* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12
The rest of these have a comment, should the new addition?
Anyway, no meaningful comments from me here Andy,
Thanks,
Conor.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [v1, 1/6] riscv: sched: defer restoring Vector context for user
Quoted 16 lines · Conor Dooley
Hey Andy,
Small bit of minor nitpickery..Quoted 1 line · Andy Chiu
User's will use its Vector registers only after the kernel really
Looks like the ' here can be removed.
Quoted 6 lines
returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs context swicth from
A->B->C, and returns
Quoted 5 lines
"a context switch"
Quoted 1 line
to C's userspace, then there is no need for restoring B's V-register.
"to restore"
Sorry for the poor english. Let me fix it in the next spin.
Quoted 9 lines
Quoted 8 lines
Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times for the upcoming kenel-mode Vector patches.
Signed-off-by Andy Chiu <[email protected]>---
Quoted 11 lines
arch/riscv/include/asm/thread_info.h int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); · line 104
Quoted 9 lines
arch/riscv/include/asm/thread_info.h +1
-0@@ -101,12 +101,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define TIF_NOTIFY_SIGNAL 9/* signal notifications exist */ #define TIF_UPROBE 10/* uprobe breakpoint or singlestep */ #define TIF_32BIT 11/* compat-mode 32bit process */ +#define TIF_RISCV_V_DEFER_RESTORE 12The rest of these have a comment, should the new addition?
Yes, it should. How about this "defer restoring process's V-context"
Quoted 6 lines
Anyway, no meaningful comments from me here Andy,
Acked-by Conor Dooley <[email protected]>Thanks,
Conor.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH 2/6 riscv: Add support for kernel mode vector
In reply to Andy ChiuFrom: Greentime Hu <[email protected]>
Add kernel_rvv_begin() and kernel_rvv_end() function declarations and corresponding definitions in kernel_mode_vector.c
These are needed to wrap uses of vector in kernel mode.
--- arch/riscv/include/asm/vector.h | 2 + arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 129 +++++++++++++++++++++++++ 3 files changed, 132 insertions(+) create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
arch/riscv/include/asm/vector.h +2 -0
@@ -22,6 +22,8 @@extern unsigned long riscv_v_vsize;int riscv_v_setup_vsize (void );bool riscv_v_first_use_handler (struct pt_regs *regs); +int kernel_rvv_begin (void ); +void kernel_rvv_end (void ); static __always_inline bool has_vector(void) {
arch/riscv/kernel/Makefile +1 -0
@@ -57,6 +57,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_M_MODE) += traps_misaligned.o obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_RISCV_ISA_V) += vector.o +obj-$(CONFIG_RISCV_ISA_V) += kernel_mode_vector.o obj-$(CONFIG_SMP) += smpboot.o obj-$(CONFIG_SMP) += smp.o obj-$(CONFIG_SMP) += cpu_ops.o
arch/riscv/kernel/kernel_mode_vector.c new file +129 -0
129 lines not on this page · raw message
Signature · 7 lines
2.17.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [v1, 2/6] riscv: Add support for kernel mode vector
Quoted 23 lines · Andy Chiu
From: Greentime Hu <[email protected]>
Add
kernel_rvv_begin()andkernel_rvv_end()function declarations and corresponding definitions inkernel_mode_vector.cThese are needed to wrap uses of vector in kernel mode.
Co-developed-by Vincent Chen <[email protected]>Signed-off-by Vincent Chen <[email protected]>Signed-off-by Greentime Hu <[email protected]>Signed-off-by Andy Chiu <[email protected]>--- arch/riscv/include/asm/vector.h | 2 + arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 129 +++++++++++++++++++++++++ 3 files changed, 132 insertions(+) create mode 100644 arch/riscv/kernel/kernel_mode_vector.carch/riscv/include/asm/vector.h +0
-0
arch/riscv/include/asm/vector.h line 26
Quoted 5 lines
arch/riscv/include/asm/vector.h +2
-0@@ -22,6 +22,8 @@extern unsigned long riscv_v_vsize;int riscv_v_setup_vsize (void );bool riscv_v_first_use_handler (struct pt_regs *regs); +int kernel_rvv_begin (void ); +void kernel_rvv_end (void );
So, we ditched all of the "rvv" stuff in the last series, using either "vector" - has_vector() - or "riscv_v". I'd rather not introduce a third naming scheme for vector related things...
Given what you add below is full of other things that use "vector", how does s/rvv/vector/ sound here?
Quoted 10 lines
arch/riscv/kernel/kernel_mode_vector.c new file +4
-0+/* +* kernel_rvv_begin(): obtain the CPU vector registers for use by the calling +* context +*
arch/riscv/kernel/kernel_mode_vector.c line 85
Quoted 10 lines
arch/riscv/kernel/kernel_mode_vector.c new file +10
-0+* Must not be called unless may_use_vector() returns true. +* Task context in the vector registers is saved back to memory as necessary. +* +* A matching call to kernel_rvv_end() must be made before returning from the +* calling context. +* +* The caller may freely use the vector registers until kernel_rvv_end() is +* called. +*/ +int kernel_rvv_begin (void )
How come this returns an int, but you never actually check the result? The other kernel_*_begin()s don't seem to return anything other than void.
arch/riscv/kernel/kernel_mode_vector.c line 91
Quoted 6 lines
arch/riscv/kernel/kernel_mode_vector.c new file +6
-0+{ +if (!has_vector ()) +return -EOPNOTSUPP ; + +if (!may_use_vector ()) +return -EPERM ;
I notice arm64 takes a stronger approach. There, if the has() call fails, it has a WARN_ON(). For the !may_use() case, it BUG()s. Since users are forced to check may_use_vector() before calling kernel_rvv_begin().
Is there a reason that we should not take the same approach?
arch/riscv/kernel/kernel_mode_vector.c line 99
Quoted 8 lines
arch/riscv/kernel/kernel_mode_vector.c new file +8
-0+ +/* Save vector state, if any */ +riscv_v_vstate_save (current,task_pt_regs (current)); + +/* Acquire kernel mode vector */ +get_cpu_vector_context (); + +/* Enable vector */
These three comments are mostly a statement of the obvious, no?
Quoted 8 lines
arch/riscv/kernel/kernel_mode_vector.c new file +8
-0+riscv_v_enable (); + +return 0 ; +} +EXPORT_SYMBOL_GPL (kernel_rvv_begin); + +/* +* kernel_rvv_end(): give the CPU vector registers back to the current task
arch/riscv/kernel/kernel_mode_vector.c line 117
Quoted 10 lines
arch/riscv/kernel/kernel_mode_vector.c new file +10
-0+* +* Must be called from a context in which kernel_rvv_begin() was previously +* called, with no call to kernel_rvv_end() in the meantime. +* +* The caller must not use the vector registers after this function is called, +* unless kernel_rvv_begin() is called again in the meantime. +*/ +void kernel_rvv_end (void ) +{ +if (WARN_ON (!has_vector ()))
But there is a WARN_ON() here...
arch/riscv/kernel/kernel_mode_vector.c line 126
Quoted 9 lines
arch/riscv/kernel/kernel_mode_vector.c new file +9
-0+return ; + +/* Restore vector state, if any */ +riscv_v_vstate_set_restore (current,task_pt_regs (current)); + +/* disable vector */ +riscv_v_disable (); + +/* release kernel mode vector */
Again, comments kinda state the obvious, no?
Otherwise, this stuff looks generally fine to me & similar to what is being done elsewhere.
Thanks,
Conor.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [v1, 2/6] riscv: Add support for kernel mode vector
Quoted 35 lines · Conor Dooley
Quoted 23 lines · Andy Chiu
From: Greentime Hu <[email protected]>
Add
kernel_rvv_begin()andkernel_rvv_end()function declarations and corresponding definitions inkernel_mode_vector.cThese are needed to wrap uses of vector in kernel mode.
Co-developed-by Vincent Chen <[email protected]>Signed-off-by Vincent Chen <[email protected]>Signed-off-by Greentime Hu <[email protected]>Signed-off-by Andy Chiu <[email protected]>--- arch/riscv/include/asm/vector.h | 2 + arch/riscv/kernel/Makefile | 1 + arch/riscv/kernel/kernel_mode_vector.c | 129 +++++++++++++++++++++++++ 3 files changed, 132 insertions(+) create mode 100644 arch/riscv/kernel/kernel_mode_vector.carch/riscv/include/asm/vector.h +0
-0arch/riscv/include/asm/vector.h line 26
Quoted 5 lines
arch/riscv/include/asm/vector.h +2
-0@@ -22,6 +22,8 @@extern unsigned long riscv_v_vsize;int riscv_v_setup_vsize (void );bool riscv_v_first_use_handler (struct pt_regs *regs); +int kernel_rvv_begin (void ); +void kernel_rvv_end (void );So, we ditched all of the "rvv" stuff in the last series, using either "vector" -
has_vector()- or "riscv_v". I'd rather not introduce a third naming scheme for vector related things...
Quoted 2 lines
Given what you add below is full of other things that use "vector", how does s/rvv/vector/ sound here?
Yes, I agree. Let's use 'vector' instead of rvv.
Quoted 38 lines
Quoted 12 lines
arch/riscv/kernel/kernel_mode_vector.c new file +4
-0+/* +* kernel_rvv_begin(): obtain the CPU vector registers for use by the calling +* context +* arch/riscv/kernel/kernel_mode_vector.c line 85
Quoted 10 lines
arch/riscv/kernel/kernel_mode_vector.c new file +10
-0+* Must not be called unless may_use_vector() returns true. +* Task context in the vector registers is saved back to memory as necessary. +* +* A matching call to kernel_rvv_end() must be made before returning from the +* calling context. +* +* The caller may freely use the vector registers until kernel_rvv_end() is +* called. +*/ +int kernel_rvv_begin (void )How come this returns an int, but you never actually check the result? The other kernel_*_begin()s don't seem to return anything other than void.
arch/riscv/kernel/kernel_mode_vector.c line 91
Quoted 6 lines
arch/riscv/kernel/kernel_mode_vector.c new file +6
-0+{ +if (!has_vector ()) +return -EOPNOTSUPP ; + +if (!may_use_vector ()) +return -EPERM ;I notice arm64 takes a stronger approach. There, if the
has()call fails, it has aWARN_ON(). For the !may_use() case, it BUG()s. Since users are forced to checkmay_use_vector()before callingkernel_rvv_begin().
Quoted 1 line
Is there a reason that we should not take the same approach?
Yes, I agree that it is better to return nothing after some thinking. Originally I was hoping to return a failure code if the preemptible kernel-mode Vector failed to allocate memory in riscv_v_start_kernel_context(). However, returning things here just complicates the programming model for kernel-mode Vector. So, let's just make it transparent to users of kernel_vector_begin() and return nothing here.
Quoted 11 lines
arch/riscv/kernel/kernel_mode_vector.c line 99
Quoted 9 lines
arch/riscv/kernel/kernel_mode_vector.c new file +8
-0+ +/* Save vector state, if any */ +riscv_v_vstate_save (current,task_pt_regs (current)); + +/* Acquire kernel mode vector */ +get_cpu_vector_context (); + +/* Enable vector */ These three comments are mostly a statement of the obvious, no?
Agree, I will drop those comments.
Quoted 34 lines
Quoted 9 lines
arch/riscv/kernel/kernel_mode_vector.c new file +8
-0+riscv_v_enable (); + +return 0 ; +} +EXPORT_SYMBOL_GPL (kernel_rvv_begin); + +/* +* kernel_rvv_end(): give the CPU vector registers back to the current task arch/riscv/kernel/kernel_mode_vector.c line 117
Quoted 10 lines
arch/riscv/kernel/kernel_mode_vector.c new file +10
-0+* +* Must be called from a context in which kernel_rvv_begin() was previously +* called, with no call to kernel_rvv_end() in the meantime. +* +* The caller must not use the vector registers after this function is called, +* unless kernel_rvv_begin() is called again in the meantime. +*/ +void kernel_rvv_end (void ) +{ +if (WARN_ON (!has_vector ()))But there is a
WARN_ON()here...arch/riscv/kernel/kernel_mode_vector.c line 126
Quoted 9 lines
arch/riscv/kernel/kernel_mode_vector.c new file +9
-0+return ; + +/* Restore vector state, if any */ +riscv_v_vstate_set_restore (current,task_pt_regs (current)); + +/* disable vector */ +riscv_v_disable (); + +/* release kernel mode vector */ Again, comments kinda state the obvious, no?
Quoted 5 lines
Otherwise, this stuff looks generally fine to me & similar to what is being done elsewhere.
Thanks,
Conor.
Thanks,
Andy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH 3/6 riscv: Add vector extension XOR implementation
In reply to Andy ChiuFrom: Greentime Hu <[email protected]>
This patch adds support for vector optimized XOR and it is tested in qemu.
--- arch/riscv/include/asm/xor.h | 82 ++++++++++++++++++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++++++++++++ 3 files changed, 164 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.S
arch/riscv/include/asm/xor.h new file +82 -0
@@ -0,0 +1,82 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Copyright (C) 2021 SiFive +*/ + +#include <linux/hardirq.h> +#include <asm-generic/xor.h> +#ifdef CONFIG_RISCV_ISA_V +#include <asm/vector.h> +#include <asm/switch_to.h> + +void xor_regs_2_ (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2); +void xor_regs_3_ (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3); +void xor_regs_4_ (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3, +const unsigned long *__restrict p4); +void xor_regs_5_ (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3, +const unsigned long *__restrict p4, +const unsigned long *__restrict p5); + +static void xor_rvv_2 (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2) +{ +kernel_rvv_begin (); +xor_regs_2_ (bytes, p1, p2); +kernel_rvv_end (); +} + +static void xor_rvv_3 (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3) +{ +kernel_rvv_begin (); +xor_regs_3_ (bytes, p1, p2, p3); +kernel_rvv_end (); +} + +static void xor_rvv_4 (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3, +const unsigned long *__restrict p4) +{ +kernel_rvv_begin (); +xor_regs_4_ (bytes, p1, p2, p3, p4); +kernel_rvv_end (); +} + +static void xor_rvv_5 (unsigned long bytes,unsigned long *__restrict p1, +const unsigned long *__restrict p2, +const unsigned long *__restrict p3, +const unsigned long *__restrict p4, +const unsigned long *__restrict p5) +{ +kernel_rvv_begin (); +xor_regs_5_ (bytes, p1, p2, p3, p4, p5); +kernel_rvv_end (); +} + +static struct xor_block_template xor_block_rvv = { + .name ="rvv" , + .do_2 = xor_rvv_2, + .do_3 = xor_rvv_3, + .do_4 = xor_rvv_4, + .do_5 = xor_rvv_5 +}; + +#undef XOR_TRY_TEMPLATES +#define XOR_TRY_TEMPLATES \ + do { \ + xor_speed(&xor_block_8regs); \ + xor_speed(&xor_block_32regs); \ + if (has_vector()) { \ + xor_speed(&xor_block_rvv);\ + } \ + } while (0) +#endif
arch/riscv/lib/Makefile +1 -0
@@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT) += tishift.o lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o +lib-$(CONFIG_RISCV_ISA_V) += xor.o
arch/riscv/lib/xor.S new file +81 -0
81 lines not on this page · raw message
Signature · 7 lines
2.17.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [v1, 3/6] riscv: Add vector extension XOR implementation
Quoted 4 lines · Andy Chiu
From: Greentime Hu <[email protected]>
This patch adds support for vector optimized XOR and it is tested in qemu.
Since this patch was originally written, has it been tested in hardware?
Quoted 35 lines
Co-developed-by Han-Kuan Chen <[email protected]>Signed-off-by Han-Kuan Chen <[email protected]>Signed-off-by Greentime Hu <[email protected]>Signed-off-by Andy Chiu <[email protected]>--- arch/riscv/include/asm/xor.h | 82 ++++++++++++++++++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++++++++++++ 3 files changed, 164 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.Sarch/riscv/include/asm/xor.h new file +14
-0+static void xor_rvv_2(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2) +static void xor_rvv_3(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3) +static void xor_rvv_4(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4) + +static void xor_rvv_5(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4,
arch/riscv/include/asm/xor.h line 71
Quoted 10 lines
arch/riscv/include/asm/xor.h new file +9
-0+ const unsigned long *__restrict p5) + +static struct xor_block_template xor_block_rvv = { + .name ="rvv" , + .do_2 = xor_rvv_2, + .do_3 = xor_rvv_3, + .do_4 = xor_rvv_4, + .do_5 = xor_rvv_5 +};
Same naming scheme comments as the main vector patchset and 2/6 apply here too.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [v1, 3/6] riscv: Add vector extension XOR implementation
Quoted 8 lines · Conor Dooley
Quoted 4 lines · Andy Chiu
From: Greentime Hu <[email protected]>
This patch adds support for vector optimized XOR and it is tested in qemu.
Since this patch was originally written, has it been tested in hardware?
We've run it on internal FPGAs but FPGAs don't count, right? ;)
Quoted 38 lines
Quoted 37 lines
Co-developed-by Han-Kuan Chen <[email protected]>Signed-off-by Han-Kuan Chen <[email protected]>Signed-off-by Greentime Hu <[email protected]>Signed-off-by Andy Chiu <[email protected]>--- arch/riscv/include/asm/xor.h | 82 ++++++++++++++++++++++++++++++++++++ arch/riscv/lib/Makefile | 1 + arch/riscv/lib/xor.S | 81 +++++++++++++++++++++++++++++++++++ 3 files changed, 164 insertions(+) create mode 100644 arch/riscv/include/asm/xor.h create mode 100644 arch/riscv/lib/xor.Sarch/riscv/include/asm/xor.h new file +15
-0+static void xor_rvv_2(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2) +static void xor_rvv_3(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3) +static void xor_rvv_4(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4) + +static void xor_rvv_5(unsigned long bytes, unsigned long *__restrict p1, + const unsigned long *__restrict p2, + const unsigned long *__restrict p3, + const unsigned long *__restrict p4, + const unsigned long *__restrict p5)
Quoted 11 lines
arch/riscv/include/asm/xor.h line 71
Quoted 8 lines
arch/riscv/include/asm/xor.h new file +8
-0+ +static struct xor_block_template xor_block_rvv = { + .name ="rvv" , + .do_2 = xor_rvv_2, + .do_3 = xor_rvv_3, + .do_4 = xor_rvv_4, + .do_5 = xor_rvv_5 +};Same naming scheme comments as the main vector patchset and 2/6 apply here too.
Yep, I'm doing s/xor_rvv/xor_vector
Thanks,
Andy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH 4/6 riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}()
In reply to Andy Chiuriscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused.
--- arch/riscv/include/asm/entry-common.h | 2 +- arch/riscv/include/asm/vector.h | 14 +++++--------- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 2 +- 5 files changed, 9 insertions(+), 13 deletions(-)
arch/riscv/include/asm/entry-common.h +1 -1
@@ -12,7 +12,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, {if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) {clear_thread_flag (TIF_RISCV_V_DEFER_RESTORE ); -riscv_v_vstate_restore (current, regs); +riscv_v_vstate_restore (¤t->thread.vstate, regs); } }
arch/riscv/include/asm/vector.h +5 -9
@@ -163,23 +163,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs)__riscv_v_vstate_dirty (regs); } -static inline void riscv_v_vstate_save (struct task_struct *task, +static inline void riscv_v_vstate_save (struct __riscv_v_ext_state *vstate,struct pt_regs *regs) {if ((regs->status &SR_VS ) ==SR_VS_DIRTY ) { -struct __riscv_v_ext_state *vstate = &task->thread.vstate; -__riscv_v_vstate_save (vstate, vstate->datap);__riscv_v_vstate_clean (regs); } } -static inline void riscv_v_vstate_restore (struct task_struct *task, +static inline void riscv_v_vstate_restore (struct __riscv_v_ext_state *vstate,struct pt_regs *regs) {if ((regs->status &SR_VS ) !=SR_VS_OFF ) { -struct __riscv_v_ext_state *vstate = &task->thread.vstate; -__riscv_v_vstate_restore (vstate, vstate->datap);__riscv_v_vstate_clean (regs); }
@@ -200,7 +196,7 @@ static inline void __switch_to_vector(struct task_struct *prev,struct pt_regs *regs; regs =task_pt_regs (prev); -riscv_v_vstate_save (prev, regs); +riscv_v_vstate_save (prev->thread.vstate, regs);riscv_v_vstate_set_restore (next,task_pt_regs (next)); }
@@ -218,8 +214,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; }static inline bool riscv_v_vstate_ctrl_user_allowed (void ) {return false; }#define riscv_v_vsize (0)#define riscv_v_vstate_discard (regs) do {} while (0) -#define riscv_v_vstate_save (task, regs) do {} while (0) -#define riscv_v_vstate_restore (task, regs) do {} while (0) +#define riscv_v_vstate_save (vstate, regs) do {} while (0) +#define riscv_v_vstate_restore (vstate, regs) do {} while (0)#define __switch_to_vector (__prev, __next) do {} while (0)#define riscv_v_vstate_off (regs) do {} while (0)#define riscv_v_vstate_on (regs) do {} while (0)
arch/riscv/kernel/kernel_mode_vector.c +1 -1
@@ -91,7 +91,7 @@ int kernel_rvv_begin(void)return -EPERM ;/* Save vector state, if any */ -riscv_v_vstate_save (current,task_pt_regs (current)); +riscv_v_vstate_save (¤t->thread.vstate,task_pt_regs (current));/* Acquire kernel mode vector */ get_cpu_vector_context ();
arch/riscv/kernel/ptrace.c +1 -1
@@ -99,7 +99,7 @@ static int riscv_vr_get(struct task_struct *target,* copying them to membuf. */ if (target == current) -riscv_v_vstate_save (current,task_pt_regs (current)); +riscv_v_vstate_save (¤t->thread.vstate,task_pt_regs (current));/* Copy vector header from vstate. */ membuf_write (&to, vstate, offsetof(struct __riscv_v_ext_state , datap));
arch/riscv/kernel/signal.c +1 -1
@@ -86,7 +86,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec)/* datap is designed to be 16 byte aligned for better performance */ WARN_ON (unlikely (!IS_ALIGNED ((unsigned long )datap,16 ))); -riscv_v_vstate_save (current, regs); +riscv_v_vstate_save (¤t->thread.vstate, regs);/* Copy everything of vstate but datap. */ err =__copy_to_user (&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state , datap));
Signature · 7 lines
2.17.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [v1, 4/6] riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}()
Quoted 7 lines · Andy Chiu
riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct
__riscv_v_ext_state, and structpt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused.Signed-off-by Andy Chiu <[email protected]>
Breaks the build chief:../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save' ../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'
rv64 allmodconfig w/ gcc.
Thanks,
Conor.
Quoted 118 lines
--- arch/riscv/include/asm/entry-common.h | 2 +- arch/riscv/include/asm/vector.h | 14 +++++--------- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 2 +- 5 files changed, 9 insertions(+), 13 deletions(-)arch/riscv/include/asm/entry-common.h +1
-1@@ -12,7 +12,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, {if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) {clear_thread_flag (TIF_RISCV_V_DEFER_RESTORE ); -riscv_v_vstate_restore (current, regs); +riscv_v_vstate_restore (¤t->thread.vstate, regs); } }arch/riscv/include/asm/vector.h +5
-9@@ -163,23 +163,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs)__riscv_v_vstate_dirty (regs); } -static inline void riscv_v_vstate_save (struct task_struct *task, +static inline void riscv_v_vstate_save (struct __riscv_v_ext_state *vstate,struct pt_regs *regs) {if ((regs->status &SR_VS ) ==SR_VS_DIRTY ) { -struct __riscv_v_ext_state *vstate = &task->thread.vstate; -__riscv_v_vstate_save (vstate, vstate->datap);__riscv_v_vstate_clean (regs); } } -static inline void riscv_v_vstate_restore (struct task_struct *task, +static inline void riscv_v_vstate_restore (struct __riscv_v_ext_state *vstate,struct pt_regs *regs) {if ((regs->status &SR_VS ) !=SR_VS_OFF ) { -struct __riscv_v_ext_state *vstate = &task->thread.vstate; -__riscv_v_vstate_restore (vstate, vstate->datap);__riscv_v_vstate_clean (regs); }@@ -200,7 +196,7 @@ static inline void __switch_to_vector(struct task_struct *prev,struct pt_regs *regs; regs =task_pt_regs (prev); -riscv_v_vstate_save (prev, regs); +riscv_v_vstate_save (prev->thread.vstate, regs);riscv_v_vstate_set_restore (next,task_pt_regs (next)); }@@ -218,8 +214,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; }static inline bool riscv_v_vstate_ctrl_user_allowed (void ) {return false; }#define riscv_v_vsize (0)#define riscv_v_vstate_discard (regs) do {} while (0) -#define riscv_v_vstate_save (task, regs) do {} while (0) -#define riscv_v_vstate_restore (task, regs) do {} while (0) +#define riscv_v_vstate_save (vstate, regs) do {} while (0) +#define riscv_v_vstate_restore (vstate, regs) do {} while (0)#define __switch_to_vector (__prev, __next) do {} while (0)#define riscv_v_vstate_off (regs) do {} while (0)#define riscv_v_vstate_on (regs) do {} while (0)arch/riscv/kernel/kernel_mode_vector.c +1
-1@@ -91,7 +91,7 @@ int kernel_rvv_begin(void)return -EPERM ;/* Save vector state, if any */ -riscv_v_vstate_save (current,task_pt_regs (current)); +riscv_v_vstate_save (¤t->thread.vstate,task_pt_regs (current));/* Acquire kernel mode vector */ get_cpu_vector_context ();arch/riscv/kernel/ptrace.c +1
-1@@ -99,7 +99,7 @@ static int riscv_vr_get(struct task_struct *target,* copying them to membuf. */ if (target == current) -riscv_v_vstate_save (current,task_pt_regs (current)); +riscv_v_vstate_save (¤t->thread.vstate,task_pt_regs (current));/* Copy vector header from vstate. */ membuf_write (&to, vstate, offsetof(struct __riscv_v_ext_state , datap));arch/riscv/kernel/signal.c +1
-1@@ -86,7 +86,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec)/* datap is designed to be 16 byte aligned for better performance */ WARN_ON (unlikely (!IS_ALIGNED ((unsigned long )datap,16 ))); -riscv_v_vstate_save (current, regs); +riscv_v_vstate_save (¤t->thread.vstate, regs);/* Copy everything of vstate but datap. */ err =__copy_to_user (&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state , datap));Signature · 7 lines
2.17.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [v1, 4/6] riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}()
Quoted 30 lines · Conor Dooley
Quoted 7 lines · Andy Chiu
riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct
__riscv_v_ext_state, and structpt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused.Signed-off-by Andy Chiu <[email protected]>Breaks the build chief:
../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'../arch/riscv/include/asm/vector.h:199:41: error: incompatible type for argument 1 of 'riscv_v_vstate_save'
Quoted 1 line
rv64 allmodconfig w/ gcc.
Thanks for catching this. This bug was buried at the next patch and I was not careful enough to carry the fix back to this patch.
Quoted 118 lines
Thanks,
Conor.Quoted 114 lines
--- arch/riscv/include/asm/entry-common.h | 2 +- arch/riscv/include/asm/vector.h | 14 +++++--------- arch/riscv/kernel/kernel_mode_vector.c | 2 +- arch/riscv/kernel/ptrace.c | 2 +- arch/riscv/kernel/signal.c | 2 +- 5 files changed, 9 insertions(+), 13 deletions(-)arch/riscv/include/asm/entry-common.h +1
-1@@ -12,7 +12,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, {if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) {clear_thread_flag (TIF_RISCV_V_DEFER_RESTORE ); -riscv_v_vstate_restore (current, regs); +riscv_v_vstate_restore (¤t->thread.vstate, regs); } }arch/riscv/include/asm/vector.h +5
-9@@ -163,23 +163,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs)__riscv_v_vstate_dirty (regs); } -static inline void riscv_v_vstate_save (struct task_struct *task, +static inline void riscv_v_vstate_save (struct __riscv_v_ext_state *vstate,struct pt_regs *regs) {if ((regs->status &SR_VS ) ==SR_VS_DIRTY ) { -struct __riscv_v_ext_state *vstate = &task->thread.vstate; -__riscv_v_vstate_save (vstate, vstate->datap);__riscv_v_vstate_clean (regs); } } -static inline void riscv_v_vstate_restore (struct task_struct *task, +static inline void riscv_v_vstate_restore (struct __riscv_v_ext_state *vstate,struct pt_regs *regs) {if ((regs->status &SR_VS ) !=SR_VS_OFF ) { -struct __riscv_v_ext_state *vstate = &task->thread.vstate; -__riscv_v_vstate_restore (vstate, vstate->datap);__riscv_v_vstate_clean (regs); }@@ -200,7 +196,7 @@ static inline void __switch_to_vector(struct task_struct *prev,struct pt_regs *regs; regs =task_pt_regs (prev); -riscv_v_vstate_save (prev, regs); +riscv_v_vstate_save (prev->thread.vstate, regs);riscv_v_vstate_set_restore (next,task_pt_regs (next)); }@@ -218,8 +214,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; }static inline bool riscv_v_vstate_ctrl_user_allowed (void ) {return false; }#define riscv_v_vsize (0)#define riscv_v_vstate_discard (regs) do {} while (0) -#define riscv_v_vstate_save (task, regs) do {} while (0) -#define riscv_v_vstate_restore (task, regs) do {} while (0) +#define riscv_v_vstate_save (vstate, regs) do {} while (0) +#define riscv_v_vstate_restore (vstate, regs) do {} while (0)#define __switch_to_vector (__prev, __next) do {} while (0)#define riscv_v_vstate_off (regs) do {} while (0)#define riscv_v_vstate_on (regs) do {} while (0)arch/riscv/kernel/kernel_mode_vector.c +1
-1@@ -91,7 +91,7 @@ int kernel_rvv_begin(void)return -EPERM ;/* Save vector state, if any */ -riscv_v_vstate_save (current,task_pt_regs (current)); +riscv_v_vstate_save (¤t->thread.vstate,task_pt_regs (current));/* Acquire kernel mode vector */ get_cpu_vector_context ();arch/riscv/kernel/ptrace.c +1
-1@@ -99,7 +99,7 @@ static int riscv_vr_get(struct task_struct *target,* copying them to membuf. */ if (target == current) -riscv_v_vstate_save (current,task_pt_regs (current)); +riscv_v_vstate_save (¤t->thread.vstate,task_pt_regs (current));/* Copy vector header from vstate. */ membuf_write (&to, vstate, offsetof(struct __riscv_v_ext_state , datap));arch/riscv/kernel/signal.c +1
-2@@ -86,7 +86,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec)/* datap is designed to be 16 byte aligned for better performance */ WARN_ON (unlikely (!IS_ALIGNED ((unsigned long )datap,16 ))); -riscv_v_vstate_save (current, regs); +riscv_v_vstate_save (¤t->thread.vstate, regs);/* Copy everything of vstate but datap. */ err =__copy_to_user (&state->v_state, ¤t->thread.vstate, offsetof(struct __riscv_v_ext_state , datap)); --2.17.1
Quoted 4 lines
Quoted 4 lines
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Thanks,
Andy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH 5/6 riscv: vector: allow kernel-mode Vector with preemption
In reply to Andy ChiuAdd kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide trap_pt_regs to let context save/restore routine reference status.VS at which the trap takes place. The thread flag TIF_RISCV_V_KMV indicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs to kernel_vstate and restore V-regs immediately from kernel_vstate if the bit is set.
Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE mask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code.
--- arch/riscv/include/asm/processor.h | 2 + arch/riscv/include/asm/thread_info.h | 4 ++ arch/riscv/include/asm/vector.h | 27 ++++++++++-- arch/riscv/kernel/asm-offsets.c | 2 + arch/riscv/kernel/entry.S | 41 ++++++++++++++++++ arch/riscv/kernel/kernel_mode_vector.c | 57 ++++++++++++++++++++++++-- arch/riscv/kernel/process.c | 8 +++- arch/riscv/kernel/vector.c | 3 +- 8 files changed, 136 insertions(+), 8 deletions(-)
arch/riscv/include/asm/processor.h +2 -0
@@ -42,6 +42,8 @@ struct thread_struct {unsigned long bad_cause;unsigned long vstate_ctrl;struct __riscv_v_ext_state vstate; +struct pt_regs *trap_pt_regs; +struct __riscv_v_ext_state kernel_vstate; };/* Whitelist the fstate from the task_struct for hardened usercopy */
arch/riscv/include/asm/thread_info.h +4 -0
@@ -102,6 +102,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define TIF_UPROBE 10/* uprobe breakpoint or singlestep */ #define TIF_32BIT 11/* compat-mode 32bit process */ #define TIF_RISCV_V_DEFER_RESTORE 12 +#define TIF_RISCV_V_KMV 13#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)
@@ -109,9 +110,12 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL)#define _TIF_UPROBE (1 << TIF_UPROBE)#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) +#define _TIF_RISCV_V_KMV (1 << TIF_RISCV_V_KMV_TASK)#define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ _TIF_NOTIFY_SIGNAL | _TIF_UPROBE) +#define RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE 0x20 +#endif /* _ASM_RISCV_THREAD_INFO_H */
arch/riscv/include/asm/vector.h +24 -3
@@ -25,6 +25,12 @@ bool riscv_v_first_use_handler(struct pt_regs *regs);int kernel_rvv_begin (void );void kernel_rvv_end (void ); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV +void riscv_v_vstate_ctrl_config_kmv (bool preemptive_kmv); +#else +#define riscv_v_vstate_ctrl_config_kmv (preemptive_kmv) do {} while (0) +#endif + static __always_inline bool has_vector(void) {return riscv_has_extension_unlikely (RISCV_ISA_EXT_v);
@@ -195,9 +201,24 @@ static inline void __switch_to_vector(struct task_struct *prev, {struct pt_regs *regs; - regs =task_pt_regs (prev); -riscv_v_vstate_save (prev->thread.vstate, regs); -riscv_v_vstate_set_restore (next,task_pt_regs (next)); +if (IS_ENABLED (CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV ) && +test_tsk_thread_flag (prev,TIF_RISCV_V_KMV )) { + regs = prev->thread.trap_pt_regs; +WARN_ON (!regs); +riscv_v_vstate_save (&prev->thread.kernel_vstate, regs); + }else { + regs =task_pt_regs (prev); +riscv_v_vstate_save (&prev->thread.vstate, regs); + } + +if (IS_ENABLED (CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV ) && +test_tsk_thread_flag (next,TIF_RISCV_V_KMV )) { + regs = next->thread.trap_pt_regs; +WARN_ON (!regs); +riscv_v_vstate_restore (&next->thread.kernel_vstate, regs); + }else { +riscv_v_vstate_set_restore (next,task_pt_regs (next)); + } }void riscv_v_vstate_ctrl_init (struct task_struct *tsk);
arch/riscv/kernel/asm-offsets.c +2 -0
@@ -38,6 +38,8 @@ void asm_offsets(void)OFFSET (TASK_TI_PREEMPT_COUNT , task_struct, thread_info.preempt_count);OFFSET (TASK_TI_KERNEL_SP , task_struct, thread_info.kernel_sp);OFFSET (TASK_TI_USER_SP , task_struct, thread_info.user_sp); +OFFSET (TASK_THREAD_TRAP_REGP , task_struct, thread.trap_pt_regs); +OFFSET (TASK_THREAD_VSTATE_CTRL , task_struct, thread.vstate_ctrl);OFFSET (TASK_THREAD_F0 , task_struct, thread.fstate.f[0 ]);OFFSET (TASK_THREAD_F1 , task_struct, thread.fstate.f[1 ]);
arch/riscv/kernel/entry.S +41 -0
53 lines not on this page · raw message
arch/riscv/kernel/kernel_mode_vector.c +54 -3
86 lines not on this page · raw message
arch/riscv/kernel/process.c +7 -1
@@ -160,8 +160,11 @@ void flush_thread(void)void arch_release_task_struct (struct task_struct *tsk) {/* Free the vector context of datap. */ -if (has_vector ()) +if (has_vector ()) {kfree (tsk->thread.vstate.datap); +if (IS_ENABLED (CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV )) +kfree (tsk->thread.kernel_vstate.datap); + } }int arch_dup_task_struct (struct task_struct *dst,struct task_struct *src)
@@ -170,7 +173,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) *dst = *src;/* clear entire V context, including datap for a new task */ memset (&dst->thread.vstate,0 ,sizeof (struct __riscv_v_ext_state )); +memset (&dst->thread.kernel_vstate,0 ,sizeof (struct __riscv_v_ext_state ));clear_tsk_thread_flag (dst,TIF_RISCV_V_DEFER_RESTORE ); +clear_tsk_thread_flag (dst,TIF_RISCV_V_KMV );return 0 ; }
@@ -205,6 +210,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) childregs->a0 =0 ;/* Return value of fork() */ p->thread.s[0 ] =0 ; } +riscv_v_vstate_ctrl_config_kmv (true); p->thread.ra = (unsigned long )ret_from_fork; p->thread.sp = (unsigned long )childregs;/* kernel sp */ return 0 ;
arch/riscv/kernel/vector.c +2 -1
@@ -122,7 +122,8 @@ static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt, ctrl |=VSTATE_CTRL_MAKE_NEXT (nxt);if (inherit) ctrl |=PR_RISCV_V_VSTATE_CTRL_INHERIT ; - tsk->thread.vstate_ctrl = ctrl; + tsk->thread.vstate_ctrl &= ~PR_RISCV_V_VSTATE_CTRL_MASK ; + tsk->thread.vstate_ctrl |= ctrl; }bool riscv_v_vstate_ctrl_user_allowed (void )
Signature · 7 lines
2.17.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [v1, 5/6] riscv: vector: allow kernel-mode Vector with preemption
Quoted 44 lines · Andy Chiu
Add
kernel_vstateto keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, providetrap_pt_regsto let context save/restore routine reference status.VS at which the trap takes place. The thread flagTIF_RISCV_V_KMVindicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs tokernel_vstateand restore V-regs immediately fromkernel_vstateif the bit is set.Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the
RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLEmask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code.Signed-off-by Andy Chiu <[email protected]>--- arch/riscv/include/asm/processor.h | 2 + arch/riscv/include/asm/thread_info.h | 4 ++ arch/riscv/include/asm/vector.h | 27 ++++++++++-- arch/riscv/kernel/asm-offsets.c | 2 + arch/riscv/kernel/entry.S | 41 ++++++++++++++++++ arch/riscv/kernel/kernel_mode_vector.c | 57 ++++++++++++++++++++++++-- arch/riscv/kernel/process.c | 8 +++- arch/riscv/kernel/vector.c | 3 +- 8 files changed, 136 insertions(+), 8 deletions(-)arch/riscv/include/asm/processor.h +2
-0@@ -42,6 +42,8 @@ struct thread_struct {unsigned long bad_cause;unsigned long vstate_ctrl;struct __riscv_v_ext_state vstate; +struct pt_regs *trap_pt_regs; +struct __riscv_v_ext_state kernel_vstate; };/* Whitelist the fstate from the task_struct for hardened usercopy */ arch/riscv/include/asm/thread_info.h +0
-0
arch/riscv/include/asm/thread_info.h int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); · line 105
Quoted 4 lines
arch/riscv/include/asm/thread_info.h +1
-0@@ -102,6 +102,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define TIF_UPROBE 10/* uprobe breakpoint or singlestep */ #define TIF_32BIT 11/* compat-mode 32bit process */ #define TIF_RISCV_V_DEFER_RESTORE 12 +#define TIF_RISCV_V_KMV 13
Same comment about comments.
Also, the "V" here is a dupe, since you have RISCV_V in the name. Ditto everywhere else, afaict. Perhaps you could do s/KMV/KERNEL_MODE/?
arch/riscv/include/asm/thread_info.h int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); · line 113
Quoted 7 lines
arch/riscv/include/asm/thread_info.h +1
-0#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)@@ -109,9 +110,12 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL)#define _TIF_UPROBE (1 << TIF_UPROBE)#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) +#define _TIF_RISCV_V_KMV (1 << TIF_RISCV_V_KMV_TASK)
Where is KMV_TASK defined?
Quoted 13 lines
arch/riscv/include/asm/thread_info.h +2
-0#define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ _TIF_NOTIFY_SIGNAL | _TIF_UPROBE) +#define RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE 0x20 +#endif /* _ASM_RISCV_THREAD_INFO_H */ arch/riscv/include/asm/vector.h +0
-0
arch/riscv/include/asm/vector.h bool riscv_v_first_use_handler(struct pt_regs *regs); · line 32
Quoted 8 lines
arch/riscv/include/asm/vector.h +5
-0@@ -25,6 +25,12 @@ bool riscv_v_first_use_handler(struct pt_regs *regs);int kernel_rvv_begin (void );void kernel_rvv_end (void ); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV +void riscv_v_vstate_ctrl_config_kmv (bool preemptive_kmv); +#else +#define riscv_v_vstate_ctrl_config_kmv (preemptive_kmv) do {} while (0) +#endif
For clang/llvm allmodconfig:../arch/riscv/kernel/process.c:213:2: error: call to undeclared function 'riscv_v_vstate_ctrl_config_kmv'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
Probably also happens when vector is disabled?
arch/riscv/include/asm/vector.h static inline void __switch_to_vector(struct task_struct *prev, · line 204
Quoted 12 lines
arch/riscv/include/asm/vector.h +2
-3+ static __always_inline bool has_vector(void) {return riscv_has_extension_unlikely (RISCV_ISA_EXT_v);@@ -195,9 +201,24 @@ static inline void __switch_to_vector(struct task_struct *prev, {struct pt_regs *regs; - regs =task_pt_regs (prev); -riscv_v_vstate_save (prev->thread.vstate, regs); -riscv_v_vstate_set_restore (next,task_pt_regs (next)); + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV) &&
w.r.t. this symbol, just drop the KMV?
arch/riscv/include/asm/vector.h static inline void __switch_to_vector(struct task_struct *prev, · line 214
Quoted 10 lines
arch/riscv/include/asm/vector.h +10
-0+ test_tsk_thread_flag(prev, TIF_RISCV_V_KMV)) { + regs = prev->thread.trap_pt_regs; + WARN_ON(!regs); + riscv_v_vstate_save(&prev->thread.kernel_vstate, regs); + } else { + regs =task_pt_regs (prev); +riscv_v_vstate_save (&prev->thread.vstate, regs); + } + + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV) &&
Possibly stupid question, but not explained by the patch, why would we ever want to have RISCV_ISA_V_PREEMPTIVE_KMV disabled?
Quoted 28 lines
arch/riscv/include/asm/vector.h +7
-0+ test_tsk_thread_flag(next, TIF_RISCV_V_KMV)) { + regs = next->thread.trap_pt_regs; + WARN_ON(!regs); + riscv_v_vstate_restore(&next->thread.kernel_vstate, regs); + } else { +riscv_v_vstate_set_restore (next,task_pt_regs (next)); + } }void riscv_v_vstate_ctrl_init (struct task_struct *tsk);arch/riscv/kernel/asm-offsets.c +2
-0@@ -38,6 +38,8 @@ void asm_offsets(void)OFFSET (TASK_TI_PREEMPT_COUNT , task_struct, thread_info.preempt_count);OFFSET (TASK_TI_KERNEL_SP , task_struct, thread_info.kernel_sp);OFFSET (TASK_TI_USER_SP , task_struct, thread_info.user_sp); +OFFSET (TASK_THREAD_TRAP_REGP , task_struct, thread.trap_pt_regs); +OFFSET (TASK_THREAD_VSTATE_CTRL , task_struct, thread.vstate_ctrl);OFFSET (TASK_THREAD_F0 , task_struct, thread.fstate.f[0 ]);OFFSET (TASK_THREAD_F1 , task_struct, thread.fstate.f[1 ]);arch/riscv/kernel/entry.S +0
-0
arch/riscv/kernel/entry.S _save_context: · line 70
Quoted 5 lines
arch/riscv/kernel/entry.S +2
-0@@ -66,6 +66,27 @@ _save_context: REG_S s4, PT_CAUSE(sp) REG_S s5, PT_TP(sp) + /* + * Reocrd the register set at the frame where in-kernel V registers are
nit: s/Reocrd/Record/
Quoted 23 lines
arch/riscv/kernel/kernel_mode_vector.c +3
-1@@ -10,6 +10,7 @@#include <linux/percpu.h> #include <linux/preempt.h> #include <linux/types.h> +#include <linux/slab.h> #include <asm/vector.h> #include <asm/switch_to.h> @@ -35,7 +36,8 @@ static __must_check inline bool may_use_vector(void)* where it is set. */ return !in_irq () && !irqs_disabled () && !in_nmi () && - !this_cpu_read (vector_context_busy); + !this_cpu_read (vector_context_busy) && + !test_thread_flag (TIF_RISCV_V_KMV ); }/*
arch/riscv/kernel/kernel_mode_vector.c static void put_cpu_vector_context(void) · line 75
Quoted 5 lines
arch/riscv/kernel/kernel_mode_vector.c +2
-0@@ -69,6 +71,47 @@ static void put_cpu_vector_context(void)preempt_enable (); } +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV +void riscv_v_vstate_ctrl_config_kmv (bool preemptive_kmv)
I don't understand what this function is trying to do, based on the function name. The lack of a verb in it is somewhat confusing.
arch/riscv/kernel/kernel_mode_vector.c static void put_cpu_vector_context(void) · line 83
Quoted 8 lines
arch/riscv/kernel/kernel_mode_vector.c +8
-0+{ +if (preemptive_kmv) + current->thread.vstate_ctrl |=RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE ; +else + current->thread.vstate_ctrl &= ~RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE ; +} + +static bool riscv_v_kmv_preempitble (void )
Beyond the ible/able stuff, there's a typo in this function name.
arch/riscv/kernel/kernel_mode_vector.c static void put_cpu_vector_context(void) · line 86
Quoted 3 lines
arch/riscv/kernel/kernel_mode_vector.c +3
-0+{ +return !!(current->thread.vstate_ctrl &RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE ); +}
Little comment on the rest, not qualified to do so :)
Thanks,
Conor.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [v1, 5/6] riscv: vector: allow kernel-mode Vector with preemption
Quoted 40 lines · Conor Dooley
Quoted 38 lines · Andy Chiu
Add
kernel_vstateto keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, providetrap_pt_regsto let context save/restore routine reference status.VS at which the trap takes place. The thread flagTIF_RISCV_V_KMVindicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs tokernel_vstateand restore V-regs immediately fromkernel_vstateif the bit is set.Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the
RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLEmask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code.Signed-off-by Andy Chiu <[email protected]>--- arch/riscv/include/asm/processor.h | 2 + arch/riscv/include/asm/thread_info.h | 4 ++ arch/riscv/include/asm/vector.h | 27 ++++++++++-- arch/riscv/kernel/asm-offsets.c | 2 + arch/riscv/kernel/entry.S | 41 ++++++++++++++++++ arch/riscv/kernel/kernel_mode_vector.c | 57 ++++++++++++++++++++++++-- arch/riscv/kernel/process.c | 8 +++- arch/riscv/kernel/vector.c | 3 +- 8 files changed, 136 insertions(+), 8 deletions(-)arch/riscv/include/asm/processor.h +2
-0@@ -42,6 +42,8 @@ struct thread_struct {unsigned long bad_cause;unsigned long vstate_ctrl;struct __riscv_v_ext_state vstate; +struct pt_regs *trap_pt_regs; +struct __riscv_v_ext_state kernel_vstate; };
Quoted 12 lines
arch/riscv/include/asm/thread_info.h int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); · line 105
Quoted 10 lines
arch/riscv/include/asm/processor.h +0
-0/* Whitelist the fstate from the task_struct for hardened usercopy */ arch/riscv/include/asm/thread_info.h +1
-0@@ -102,6 +102,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define TIF_UPROBE 10/* uprobe breakpoint or singlestep */ #define TIF_32BIT 11/* compat-mode 32bit process */ #define TIF_RISCV_V_DEFER_RESTORE 12 +#define TIF_RISCV_V_KMV 13Same comment about comments.
Adding /* kernel-mode Vector run with preemption-on */
Quoted 3 lines
Also, the "V" here is a dupe, since you have
RISCV_Vin the name. Ditto everywhere else, afaict. Perhaps you could do s/KMV/KERNEL_MODE/?
Good idea.
Quoted 10 lines
arch/riscv/include/asm/thread_info.h int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src); · line 113
Quoted 8 lines
arch/riscv/include/asm/thread_info.h +1
-0#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)@@ -109,9 +110,12 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);#define _TIF_NOTIFY_SIGNAL (1 << TIF_NOTIFY_SIGNAL)#define _TIF_UPROBE (1 << TIF_UPROBE)#define _TIF_RISCV_V_DEFER_RESTORE (1 << TIF_RISCV_V_DEFER_RESTORE) +#define _TIF_RISCV_V_KMV (1 << TIF_RISCV_V_KMV_TASK)Where is
KMV_TASKdefined?
My bad, it should be TIF_RISCV_V_KMV. Also, I'm changing it to TIF_RISCV_V_KERNEL_MODE now.
Quoted 26 lines
Quoted 14 lines
arch/riscv/include/asm/thread_info.h +2
-0#define _TIF_WORK_MASK \ (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \ _TIF_NOTIFY_SIGNAL | _TIF_UPROBE) +#define RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE 0x20 +#endif /* _ASM_RISCV_THREAD_INFO_H */ arch/riscv/include/asm/vector.h +0
-0arch/riscv/include/asm/vector.h bool riscv_v_first_use_handler(struct pt_regs *regs); · line 32
Quoted 8 lines
arch/riscv/include/asm/vector.h +5
-0@@ -25,6 +25,12 @@ bool riscv_v_first_use_handler(struct pt_regs *regs);int kernel_rvv_begin (void );void kernel_rvv_end (void ); +#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV +void riscv_v_vstate_ctrl_config_kmv (bool preemptive_kmv); +#else +#define riscv_v_vstate_ctrl_config_kmv (preemptive_kmv) do {} while (0) +#endif For clang/llvm allmodconfig:
../arch/riscv/kernel/process.c:213:2: error: call to undeclared function 'riscv_v_vstate_ctrl_config_kmv'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
Quoted 1 line
Probably also happens when vector is disabled?
Yes, I'm going to move the entire block out of CONFIG_RISCV_ISA_V to resolve that.
Quoted 28 lines
Quoted 7 lines
arch/riscv/include/asm/vector.h +1
-0+ static __always_inline bool has_vector(void) {return riscv_has_extension_unlikely (RISCV_ISA_EXT_v);arch/riscv/include/asm/vector.h static inline void __switch_to_vector(struct task_struct *prev, · line 204
Quoted 7 lines
arch/riscv/include/asm/vector.h +1
-3@@ -195,9 +201,24 @@ static inline void __switch_to_vector(struct task_struct *prev, {struct pt_regs *regs; - regs =task_pt_regs (prev); -riscv_v_vstate_save (prev->thread.vstate, regs); -riscv_v_vstate_set_restore (next,task_pt_regs (next)); + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV) &&w.r.t. this symbol, just drop the KMV?
arch/riscv/include/asm/vector.h static inline void __switch_to_vector(struct task_struct *prev, · line 214
Quoted 10 lines
arch/riscv/include/asm/vector.h +10
-0+ test_tsk_thread_flag(prev, TIF_RISCV_V_KMV)) { + regs = prev->thread.trap_pt_regs; + WARN_ON(!regs); + riscv_v_vstate_save(&prev->thread.kernel_vstate, regs); + } else { + regs =task_pt_regs (prev); +riscv_v_vstate_save (&prev->thread.vstate, regs); + } + + if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV) &&
Quoted 2 lines
Possibly stupid question, but not explained by the patch, why would we ever want to have
RISCV_ISA_V_PREEMPTIVE_KMVdisabled?
Sorry, it's not obvious here. Below is the commit message that I will add for describing usecase of RISCV_ISA_V_PREEMPTIVE_KMV (now RISCV_ISA_V_PREEMPTIVE):
provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking per thread's kernel-mode V context. Or, users might as well want to disable it if all
kernel-mode Vector code is time sensitive and cannot tolerate context switch overhead.
Quoted 32 lines
Quoted 32 lines
arch/riscv/include/asm/vector.h +7
-0+ test_tsk_thread_flag(next, TIF_RISCV_V_KMV)) { + regs = next->thread.trap_pt_regs; + WARN_ON(!regs); + riscv_v_vstate_restore(&next->thread.kernel_vstate, regs); + } else { +riscv_v_vstate_set_restore (next,task_pt_regs (next)); + } }void riscv_v_vstate_ctrl_init (struct task_struct *tsk);arch/riscv/kernel/asm-offsets.c +2
-0@@ -38,6 +38,8 @@ void asm_offsets(void)OFFSET (TASK_TI_PREEMPT_COUNT , task_struct, thread_info.preempt_count);OFFSET (TASK_TI_KERNEL_SP , task_struct, thread_info.kernel_sp);OFFSET (TASK_TI_USER_SP , task_struct, thread_info.user_sp); +OFFSET (TASK_THREAD_TRAP_REGP , task_struct, thread.trap_pt_regs); +OFFSET (TASK_THREAD_VSTATE_CTRL , task_struct, thread.vstate_ctrl);OFFSET (TASK_THREAD_F0 , task_struct, thread.fstate.f[0 ]);OFFSET (TASK_THREAD_F1 , task_struct, thread.fstate.f[1 ]);arch/riscv/kernel/entry.S +0
-0@@ -66,6 +66,27 @@ _save_context: REG_S s4, PT_CAUSE(sp) REG_S s5, PT_TP(sp)
Quoted 4 lines
arch/riscv/kernel/entry.S _save_context: · line 70
Quoted 2 lines
arch/riscv/kernel/entry.S +2
-0+ /* + * Reocrd the register set at the frame where in-kernel V registers arenit: s/Reocrd/Record/
Oops.
Quoted 27 lines
Quoted 27 lines
arch/riscv/kernel/kernel_mode_vector.c +3
-1@@ -10,6 +10,7 @@#include <linux/percpu.h> #include <linux/preempt.h> #include <linux/types.h> +#include <linux/slab.h> #include <asm/vector.h> #include <asm/switch_to.h> @@ -35,7 +36,8 @@ static __must_check inline bool may_use_vector(void)* where it is set. */ return !in_irq () && !irqs_disabled () && !in_nmi () && - !this_cpu_read (vector_context_busy); + !this_cpu_read (vector_context_busy) && + !test_thread_flag (TIF_RISCV_V_KMV ); }/* @@ -69,6 +71,47 @@ static void put_cpu_vector_context(void)preempt_enable (); }
Quoted 5 lines
arch/riscv/kernel/kernel_mode_vector.c static void put_cpu_vector_context(void) · line 75
Quoted 2 lines
arch/riscv/kernel/kernel_mode_vector.c +2
-0+#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV +void riscv_v_vstate_ctrl_config_kmv (bool preemptive_kmv)I don't understand what this function is trying to do, based on the function name. The lack of a verb in it is somewhat confusing.
The purpose of this function is to allow/disallow kernel-mode Vector to be executed with kernel preemption. I am going to change the function name to kernel_vector_allow_preemption() since there is only one user of this function and the only purpose is to initialize it to be "allowed" when the config is y.
Quoted 11 lines
arch/riscv/kernel/kernel_mode_vector.c static void put_cpu_vector_context(void) · line 83
Quoted 9 lines
arch/riscv/kernel/kernel_mode_vector.c +8
-0+{ +if (preemptive_kmv) + current->thread.vstate_ctrl |=RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE ; +else + current->thread.vstate_ctrl &= ~RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE ; +} + +static bool riscv_v_kmv_preempitble (void )Beyond the ible/able stuff, there's a typo in this function name.
I am going to change the function name to kernel_vector_preemptible to match the naming scheme above.
Quoted 9 lines
arch/riscv/kernel/kernel_mode_vector.c static void put_cpu_vector_context(void) · line 86
Quoted 4 lines
arch/riscv/kernel/kernel_mode_vector.c +3
-0+{ +return !!(current->thread.vstate_ctrl &RISCV_V_VSTATE_CTRL_KMV_PREEMPTIBLE ); +}Little comment on the rest, not qualified to do so :)
Thanks,
Conor.
Thanks,
Andy
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH 6/6 riscv: vector: enable preemptive kernel-mode Vector to be built
In reply to Andy ChiuAdd a Kconfig to let user decides whether kernel-mode Vector in a preemptive kernel should also run with preemption. If the config is 'N', then all kernel-mode Vector code are run with preemption disabled.
--- arch/riscv/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/riscv/Kconfig +10 -0
@@ -500,6 +500,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE If you don't know what to do here, say Y. +config RISCV_ISA_V_PREEMPTIVE_KMV + bool "Run kernel-mode Vector with kernel preemption" + depends on PREEMPTION + depends on RISCV_ISA_V + default y + help + Ordinarily the kernel disables preemption before running in-kernel + Vector code. This config frees the kernel from disabling preemption + by adding meory on demand for tracking kernel's V-context. + config TOOLCHAIN_HAS_ZBB bool default y
Signature · 7 lines
2.17.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [v1, 6/6] riscv: vector: enable preemptive kernel-mode Vector to be built
Hey Andy,
Quoted 3 lines · Andy Chiu
Add a
Kconfigto let user decides whether kernel-mode Vector in a preemptive kernel should also run with preemption. If the config is 'N', then all kernel-mode Vector code are run with preemption disabled.
nit: "is run".
Quoted 13 lines
Signed-off-by Andy Chiu <[email protected]>--- arch/riscv/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)arch/riscv/Kconfig +0
-0@@ -500,6 +500,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE If you don't know what to do here, say Y.
arch/riscv/Kconfig config RISCV_ISA_V_DEFAULT_ENABLE · line 511
Quoted 10 lines
arch/riscv/Kconfig +9
-0+config RISCV_ISA_V_PREEMPTIVE_KMV + bool "Run kernel-mode Vector with kernel preemption" + depends on PREEMPTION + depends on RISCV_ISA_V + default y + help + Ordinarily the kernel disables preemption before running in-kernel + Vector code. This config frees the kernel from disabling preemption + by adding meory on demand for tracking kernel's V-context.
s/meory/memory/
This should be part of the previous patch.
Thanks,
Conor.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Am Samstag, 15. Juli 2023, 17:00:26 CEST schrieb Andy Chiu:
Quoted 20 lines
This series provides support for running Vector code in kernel mode. The implementation is based on the v12 series of the Vector series, but with some additions. First, we introduce a mechanism to defer restoring Vector context for userspace programs (patch 1). This is similar to arm64 and x86's approaches when dealing with extra userspace register context. And it is benefitial to both Vector in user and kernel-mode. Then, patch 2, 3 add the kernel-mode Vector patch from v12 with minor modifications. At the end of the series, patch 4, 5, 6 add supports for making kernel-mode Vector code preemptible. We do this by adding kernel-mode Vector context, and keeping track of the frame where V context is last valid. We believe that enabling preemption of running V is a critical path for getting V more generally available in the kernel-mode. Besides, with status.VS, we can easily tell if saving/restoring V is required. This reduce the level of cost when running SIMD in kernel mode as compared to other arches. Other arches usually do not have a way to tell if extra context is dirty. Thus, if they also want to support running preemptible code with extra registers, then they must save/restore extra context at each context switch even if registers are not dirty.
Quoted 8 lines
The series is tested by loading a kernel module on a preemptive kernel. The module launches multiple kworkers which run Vector operations and verifies with scalar code. Also, the module provides userspace intefaces via fops to verify if we can run Vector code on syscall path.
Changes from the vector v12 series (for patch 2, 3):
- return a failure code whenkernel_rvv_begin()fails.
- Do not immediately restore user's V context.
This works nicely with my vector crypto patchset rebased on top of it:
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv