Skip to contentSkip to conversation

riscv: support kernel-mode Vector

PATCH Superseded Andy Chiu 5 participants 17 messages Jul 21, 2023 – Aug 16, 2023

Subsystems (2)

v2 of 11 Superseded Latest is v11 →

lore
Share & tools
lore

Andy Chiu

author
#0 lore raw

v2 cover riscv: support kernel-mode Vector

This series provides support for running Vector code in kernel mode. The implementation is based on the v12 series of the Vector series, but with some additions. First, we introduce a mechanism to defer restoring Vector context for userspace programs (patch 1). This is similar to arm64 and x86's approaches when dealing with extra userspace register context. And it is benefitial to both Vector in user and kernel-mode. Then, patch 2, 3 add the kernel-mode Vector patch from v12 with minor modifications. At the end of the series, patch 4, 5 add supports for making kernel-mode Vector code preemptible. We do this by adding kernel-mode Vector context, and keeping track of the frame where V context is last valid. We believe that enabling preemption of running V is a critical path for getting V more generally available in the kernel-mode. Besides, with status.VS, we can easily tell if saving/restoring V is required. This reduce the level of cost when running SIMD in kernel mode as compared to other arches. Other arches usually do not have a way to tell if extra context is dirty. Thus, if they also want to support running preemptible code with extra registers, then they must save/restore extra context at each context switch even if registers are not dirty.

The series is tested by loading a kernel module on a preemptive kernel. The module launches multiple kworkers which run Vector operations and verifies with scalar code. Also, the module provides userspace intefaces via fops to verify if we can run Vector code on syscall path.

Updated patches: 1, 2, 3, 4, 5
New patches: -
Unchanged patches: -
Deleted patches: 6 (moved to 5)

Changelog v2:
- fix build issues
- Follow arm's way of starting kernel-mode simd code:
- add include/asm/simd.h and rename may_use_vector() ->

     may_use_simd()

- return void in kernel_vector_begin(), and BUG_ON if may_use_simd() fails
- Change naming scheme for functions/macros (Conor):
- remove KMV
- 's/rvv/vector/'
- 's/RISCV_ISA_V_PREEMPTIVE_KMV/RISCV_ISA_V_PREEMPTIVE/'
- 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE/'

Changes from the vector v12 series (for patch 2, 3):
- return a failure code when kernel_vector_begin() fails.
- Do not immediately restore user's V context.

Andy Chiu (3):
riscv: sched: defer restoring Vector context for user
riscv: vector: do not pass task_struct into

    riscv_v_vstate_{save,restore}()

riscv: vector: allow kernel-mode Vector with preemption

Greentime Hu (2):
riscv: Add support for kernel mode vector
riscv: Add vector extension XOR implementation

 arch/riscv/Kconfig                     |  10 ++
 arch/riscv/include/asm/entry-common.h  |  13 +++
 arch/riscv/include/asm/processor.h     |   2 +
 arch/riscv/include/asm/simd.h          |  52 +++++++++
 arch/riscv/include/asm/thread_info.h   |   6 +
 arch/riscv/include/asm/vector.h        |  50 +++++++--
 arch/riscv/include/asm/xor.h           |  82 ++++++++++++++
 arch/riscv/kernel/Makefile             |   1 +
 arch/riscv/kernel/asm-offsets.c        |   2 +
 arch/riscv/kernel/entry.S              |  45 ++++++++
 arch/riscv/kernel/kernel_mode_vector.c | 146 +++++++++++++++++++++++++
 arch/riscv/kernel/process.c            |  10 +-
 arch/riscv/kernel/ptrace.c             |   2 +-
 arch/riscv/kernel/signal.c             |   4 +-
 arch/riscv/kernel/vector.c             |   5 +-
 arch/riscv/lib/Makefile                |   1 +
 arch/riscv/lib/xor.S                   |  81 ++++++++++++++
 17 files changed, 495 insertions(+), 17 deletions(-)
 create mode 100644 arch/riscv/include/asm/simd.h
 create mode 100644 arch/riscv/include/asm/xor.h
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
 create mode 100644 arch/riscv/lib/xor.S
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
author
#1 lore raw

PATCH v2 1/5 riscv: sched: defer restoring Vector context for user

User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register.

Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times for the upcoming kenel-mode Vector patches.

---
Changelog v2:
- rename and add comment for the new thread flag (Conor)

---
 arch/riscv/include/asm/entry-common.h | 13 +++++++++++++
 arch/riscv/include/asm/thread_info.h  |  2 ++
 arch/riscv/include/asm/vector.h       | 11 ++++++++++-
 arch/riscv/kernel/process.c           |  2 ++
 arch/riscv/kernel/signal.c            |  2 +-
 arch/riscv/kernel/vector.c            |  2 +-
 6 files changed, 29 insertions(+), 3 deletions(-)
arch/riscv/include/asm/entry-common.h +13 -0
index 6e4dee49d84b..52926f4d8d7c 100644
@@ -4,6 +4,19 @@
 #define _ASM_RISCV_ENTRY_COMMON_H
 
 #include <asm/stacktrace.h>
+#include <asm/thread_info.h>
+#include <asm/vector.h>
+
+static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs,
+						  unsigned long ti_work)
+{
+	if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) {
+		clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE);
+		riscv_v_vstate_restore(current, regs);
+	}
+}
+
+#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare
 
 void handle_page_fault(struct pt_regs *regs);
 void handle_break(struct pt_regs *regs);
arch/riscv/include/asm/thread_info.h +2 -0
index 1833beb00489..b182f2d03e25 100644
@@ -93,12 +93,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define TIF_NOTIFY_SIGNAL	9	/* signal notifications exist */
 #define TIF_UPROBE		10	/* uprobe breakpoint or singlestep */
 #define TIF_32BIT		11	/* compat-mode 32bit process */
+#define TIF_RISCV_V_DEFER_RESTORE	12 /* restore Vector before returing to user */
 
 #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
 #define _TIF_NOTIFY_SIGNAL	(1 << TIF_NOTIFY_SIGNAL)
 #define _TIF_UPROBE		(1 << TIF_UPROBE)
+#define _TIF_RISCV_V_DEFER_RESTORE	(1 << TIF_RISCV_V_DEFER_RESTORE)
 
 #define _TIF_WORK_MASK \
 	(_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \
arch/riscv/include/asm/vector.h +10 -1
index 3d78930cab51..a4f3705fd144 100644
@@ -183,6 +183,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task,
 	}
 }
 
+static inline void riscv_v_vstate_set_restore(struct task_struct *task,
+					      struct pt_regs *regs)
+{
+	if ((regs->status & SR_VS) != SR_VS_OFF) {
+		set_tsk_thread_flag(task, TIF_RISCV_V_DEFER_RESTORE);
+		riscv_v_vstate_on(regs);
+	}
+}
+
 static inline void __switch_to_vector(struct task_struct *prev,
 				      struct task_struct *next)
 {
@@ -190,7 +199,7 @@ static inline void __switch_to_vector(struct task_struct *prev,
 
 	regs = task_pt_regs(prev);
 	riscv_v_vstate_save(prev, regs);
-	riscv_v_vstate_restore(next, task_pt_regs(next));
+	riscv_v_vstate_set_restore(next, task_pt_regs(next));
 }
 
 void riscv_v_vstate_ctrl_init(struct task_struct *tsk);
arch/riscv/kernel/process.c +2 -0
index e32d737e039f..ec89e7edb6fd 100644
@@ -153,6 +153,7 @@ void flush_thread(void)
 	riscv_v_vstate_off(task_pt_regs(current));
 	kfree(current->thread.vstate.datap);
 	memset(&current->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
+	clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE);
 #endif
 }
 
@@ -169,6 +170,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
 	*dst = *src;
 	/* clear entire V context, including datap for a new task */
 	memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
+	clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE);
 
 	return 0;
 }
arch/riscv/kernel/signal.c +1 -1
index 180d951d3624..0fca2c128b5f 100644
@@ -134,7 +134,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec)
 	if (unlikely(err))
 		return err;
 
-	riscv_v_vstate_restore(current, regs);
+	riscv_v_vstate_set_restore(current, regs);
 
 	return err;
 }
arch/riscv/kernel/vector.c +1 -1
index 8d92fb6c522c..9d583b760db4 100644
@@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs)
 		return true;
 	}
 	riscv_v_vstate_on(regs);
-	riscv_v_vstate_restore(current, regs);
+	riscv_v_vstate_set_restore(current, regs);
 	return true;
 }
 
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Reviewed-by on v2 1/5
#2 lore raw

Re: [v2, 1/5] riscv: sched: defer restoring Vector context for user

Quoted 10 lines · Andy Chiu

User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register.

Quoted 6 lines

Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times for the upcoming kenel-mode Vector patches.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#3 lore raw

PATCH v2 2/5 riscv: Add support for kernel mode vector

In reply to Andy Chiu

From: Greentime Hu <[email protected]>

Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c

These are needed to wrap uses of vector in kernel mode.

---
Changelog v2:
- 's/kernel_rvv/kernel_vector' and return void in kernel_vector_begin (Conor)
- export may_use_simd to include/asm/simd.h

---
 arch/riscv/include/asm/simd.h          |  50 ++++++++++++
 arch/riscv/include/asm/vector.h        |   2 +
 arch/riscv/kernel/Makefile             |   1 +
 arch/riscv/kernel/kernel_mode_vector.c | 101 +++++++++++++++++++++++++
 4 files changed, 154 insertions(+)
 create mode 100644 arch/riscv/include/asm/simd.h
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
arch/riscv/include/asm/simd.h new file +50 -0
new file mode 100644
index 000000000000..ef70af78005d
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2017 Linaro Ltd. <[email protected]>
+ * Copyright (C) 2023 SiFive
+ */
+
+#ifndef __ASM_SIMD_H
+#define __ASM_SIMD_H
+
+#include <linux/compiler.h>
+#include <linux/irqflags.h>
+#include <linux/percpu.h>
+#include <linux/preempt.h>
+#include <linux/types.h>
+
+#ifdef CONFIG_RISCV_ISA_V
+
+DECLARE_PER_CPU(bool, vector_context_busy);
+
+/*
+ * may_use_simd - whether it is allowable at this time to issue vector
+ *                instructions or access the vector register file
+ *
+ * Callers must not assume that the result remains true beyond the next
+ * preempt_enable() or return from softirq context.
+ */
+static __must_check inline bool may_use_simd(void)
+{
+	/*
+	 * vector_context_busy is only set while preemption is disabled,
+	 * and is clear whenever preemption is enabled. Since
+	 * this_cpu_read() is atomic w.r.t. preemption, vector_context_busy
+	 * cannot change under our feet -- if it's set we cannot be
+	 * migrated, and if it's clear we cannot be migrated to a CPU
+	 * where it is set.
+	 */
+	return !in_irq() && !irqs_disabled() && !in_nmi() &&
+	       !this_cpu_read(vector_context_busy);
+}
+
+#else /* ! CONFIG_RISCV_ISA_V */
+
+static __must_check inline bool may_use_simd(void)
+{
+	return false;
+}
+
+#endif /* ! CONFIG_RISCV_ISA_V */
+
+#endif
arch/riscv/include/asm/vector.h +2 -0
index a4f3705fd144..b46b8f3261fa 100644
@@ -22,6 +22,8 @@
 extern unsigned long riscv_v_vsize;
 int riscv_v_setup_vsize(void);
 bool riscv_v_first_use_handler(struct pt_regs *regs);
+void kernel_vector_begin(void);
+void kernel_vector_end(void);
 
 static __always_inline bool has_vector(void)
 {
arch/riscv/kernel/Makefile +1 -0
index 506cc4a9a45a..3f4435746af7 100644
@@ -61,6 +61,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/
 obj-$(CONFIG_RISCV_M_MODE)	+= traps_misaligned.o
 obj-$(CONFIG_FPU)		+= fpu.o
 obj-$(CONFIG_RISCV_ISA_V)	+= vector.o
+obj-$(CONFIG_RISCV_ISA_V)	+= kernel_mode_vector.o
 obj-$(CONFIG_SMP)		+= smpboot.o
 obj-$(CONFIG_SMP)		+= smp.o
 obj-$(CONFIG_SMP)		+= cpu_ops.o
arch/riscv/kernel/kernel_mode_vector.c new file +101 -0

101 lines not on this page · raw message

Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#4 lore raw

Re: [v2, 2/5] riscv: Add support for kernel mode vector

Hey Andy,

Quoted 36 lines · Andy Chiu

From: Greentime Hu <[email protected]>

Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c

These are needed to wrap uses of vector in kernel mode.

---
Changelog v2:
- 's/kernel_rvv/kernel_vector' and return void in kernel_vector_begin (Conor)
- export may_use_simd to include/asm/simd.h
---

arch/riscv/kernel/kernel_mode_vector.c new file +18 -0
+/*
+ * kernel_vector_begin(): obtain the CPU vector registers for use by the calling
+ * context
+ *
+ * Must not be called unless may_use_simd() returns true.
+ * Task context in the vector registers is saved back to memory as necessary.
+ *
+ * A matching call to kernel_vector_end() must be made before returning from the
+ * calling context.
+ *
+ * The caller may freely use the vector registers until kernel_vector_end() is
+ * called.
+ */
+void kernel_vector_begin(void)
+{
+	if (WARN_ON(!has_vector()))
+		return;
+

arch/riscv/kernel/kernel_mode_vector.c line 77

Quoted 10 lines
arch/riscv/kernel/kernel_mode_vector.c new file +9 -0
+	BUG_ON(!may_use_simd());
+
+	riscv_v_vstate_save(current, task_pt_regs(current));
+
+	get_cpu_vector_context();
+
+	riscv_v_enable();
+
 
+	return 0;

This breaks the build as you have made the function void.

Otherwise, this looks fine to me.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#5 lore raw

Re: [v2, 2/5] riscv: Add support for kernel mode vector

Quoted 49 lines · Conor Dooley

Hey Andy,

Quoted 44 lines · Andy Chiu

From: Greentime Hu <[email protected]>

Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c

These are needed to wrap uses of vector in kernel mode.

---
Changelog v2:
- 's/kernel_rvv/kernel_vector' and return void in kernel_vector_begin (Conor)
- export may_use_simd to include/asm/simd.h
---

arch/riscv/kernel/kernel_mode_vector.c new file +26 -0
+/*
+ * kernel_vector_begin(): obtain the CPU vector registers for use by the calling
+ * context
+ *
+ * Must not be called unless may_use_simd() returns true.
+ * Task context in the vector registers is saved back to memory as necessary.
+ *
+ * A matching call to kernel_vector_end() must be made before returning from the
+ * calling context.
+ *
+ * The caller may freely use the vector registers until kernel_vector_end() is
+ * called.
+ */
+void kernel_vector_begin(void)
+{
+     if (WARN_ON(!has_vector()))
+             return;
+
+     BUG_ON(!may_use_simd());
+
+     riscv_v_vstate_save(current, task_pt_regs(current));
+
+     get_cpu_vector_context();
+
+     riscv_v_enable();
+
Quoted 3 lines
Quoted 1 line

+ return 0;

This breaks the build as you have made the function void.

Sorry, my bad again..
I will send a v3 to address this when we close the discussion on patch 5/5.

Quoted 2 lines

Otherwise, this looks fine to me.

Thanks,
Andy

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#6 lore raw

Re: [v2, 2/5] riscv: Add support for kernel mode vector

In reply to Andy Chiu
Quoted 58 lines · Andy Chiu

From: Greentime Hu <[email protected]>

Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c

These are needed to wrap uses of vector in kernel mode.

---
Changelog v2:
- 's/kernel_rvv/kernel_vector' and return void in kernel_vector_begin (Conor)
- export may_use_simd to include/asm/simd.h

---
 arch/riscv/include/asm/simd.h          |  50 ++++++++++++
 arch/riscv/include/asm/vector.h        |   2 +
 arch/riscv/kernel/Makefile             |   1 +
 arch/riscv/kernel/kernel_mode_vector.c | 101 +++++++++++++++++++++++++
 4 files changed, 154 insertions(+)
 create mode 100644 arch/riscv/include/asm/simd.h
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
arch/riscv/include/asm/simd.h new file +27 -0
new file mode 100644
index 000000000000..ef70af78005d
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2017 Linaro Ltd. <[email protected]>
+ * Copyright (C) 2023 SiFive
+ */
+
+#ifndef __ASM_SIMD_H
+#define __ASM_SIMD_H
+
+#include <linux/compiler.h>
+#include <linux/irqflags.h>
+#include <linux/percpu.h>
+#include <linux/preempt.h>
+#include <linux/types.h>
+
+#ifdef CONFIG_RISCV_ISA_V
+
+DECLARE_PER_CPU(bool, vector_context_busy);
+
+/*
+ * may_use_simd - whether it is allowable at this time to issue vector
+ *                instructions or access the vector register file
+ *
+ * Callers must not assume that the result remains true beyond the next
+ * preempt_enable() or return from softirq context.
+ */
+static __must_check inline bool may_use_simd(void)

arch/riscv/include/asm/simd.h line 37

Quoted 10 lines
arch/riscv/include/asm/simd.h new file +10 -0
+{
+	/*
+	 * vector_context_busy is only set while preemption is disabled,
+	 * and is clear whenever preemption is enabled. Since
+	 * this_cpu_read() is atomic w.r.t. preemption, vector_context_busy
+	 * cannot change under our feet -- if it's set we cannot be
+	 * migrated, and if it's clear we cannot be migrated to a CPU
+	 * where it is set.
+	 */
+	return !in_irq() && !irqs_disabled() && !in_nmi() &&

in_irq() is deprecated, use in_hardirq() instead.

Could you elaborate why !irqs_disabled() is required?

Nit: Use the full 100 chars for the line.

Björn

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#7 lore raw

Re: [v2, 2/5] riscv: Add support for kernel mode vector

In reply to Andy Chiu
Quoted 175 lines · Andy Chiu

From: Greentime Hu <[email protected]>

Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c

These are needed to wrap uses of vector in kernel mode.

---
Changelog v2:
- 's/kernel_rvv/kernel_vector' and return void in kernel_vector_begin (Conor)
- export may_use_simd to include/asm/simd.h

---
 arch/riscv/include/asm/simd.h          |  50 ++++++++++++
 arch/riscv/include/asm/vector.h        |   2 +
 arch/riscv/kernel/Makefile             |   1 +
 arch/riscv/kernel/kernel_mode_vector.c | 101 +++++++++++++++++++++++++
 4 files changed, 154 insertions(+)
 create mode 100644 arch/riscv/include/asm/simd.h
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
arch/riscv/include/asm/simd.h new file +50 -0
new file mode 100644
index 000000000000..ef70af78005d
@@ -0,0 +1,50 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2017 Linaro Ltd. <[email protected]>
+ * Copyright (C) 2023 SiFive
+ */
+
+#ifndef __ASM_SIMD_H
+#define __ASM_SIMD_H
+
+#include <linux/compiler.h>
+#include <linux/irqflags.h>
+#include <linux/percpu.h>
+#include <linux/preempt.h>
+#include <linux/types.h>
+
+#ifdef CONFIG_RISCV_ISA_V
+
+DECLARE_PER_CPU(bool, vector_context_busy);
+
+/*
+ * may_use_simd - whether it is allowable at this time to issue vector
+ *                instructions or access the vector register file
+ *
+ * Callers must not assume that the result remains true beyond the next
+ * preempt_enable() or return from softirq context.
+ */
+static __must_check inline bool may_use_simd(void)
+{
+	/*
+	 * vector_context_busy is only set while preemption is disabled,
+	 * and is clear whenever preemption is enabled. Since
+	 * this_cpu_read() is atomic w.r.t. preemption, vector_context_busy
+	 * cannot change under our feet -- if it's set we cannot be
+	 * migrated, and if it's clear we cannot be migrated to a CPU
+	 * where it is set.
+	 */
+	return !in_irq() && !irqs_disabled() && !in_nmi() &&
+	       !this_cpu_read(vector_context_busy);
+}
+
+#else /* ! CONFIG_RISCV_ISA_V */
+
+static __must_check inline bool may_use_simd(void)
+{
+	return false;
+}
+
+#endif /* ! CONFIG_RISCV_ISA_V */
+
+#endif
arch/riscv/include/asm/vector.h +2 -0
index a4f3705fd144..b46b8f3261fa 100644
@@ -22,6 +22,8 @@
 extern unsigned long riscv_v_vsize;
 int riscv_v_setup_vsize(void);
 bool riscv_v_first_use_handler(struct pt_regs *regs);
+void kernel_vector_begin(void);
+void kernel_vector_end(void);
 
 static __always_inline bool has_vector(void)
 {
arch/riscv/kernel/Makefile +1 -0
index 506cc4a9a45a..3f4435746af7 100644
@@ -61,6 +61,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/
 obj-$(CONFIG_RISCV_M_MODE)	+= traps_misaligned.o
 obj-$(CONFIG_FPU)		+= fpu.o
 obj-$(CONFIG_RISCV_ISA_V)	+= vector.o
+obj-$(CONFIG_RISCV_ISA_V)	+= kernel_mode_vector.o
 obj-$(CONFIG_SMP)		+= smpboot.o
 obj-$(CONFIG_SMP)		+= smp.o
 obj-$(CONFIG_SMP)		+= cpu_ops.o
arch/riscv/kernel/kernel_mode_vector.c new file +63 -0
new file mode 100644
index 000000000000..1c3b32d2b340
@@ -0,0 +1,101 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2012 ARM Ltd.
+ * Author: Catalin Marinas <[email protected]>
+ * Copyright (C) 2017 Linaro Ltd. <[email protected]>
+ * Copyright (C) 2021 SiFive
+ */
+#include <linux/compiler.h>
+#include <linux/irqflags.h>
+#include <linux/percpu.h>
+#include <linux/preempt.h>
+#include <linux/types.h>
+
+#include <asm/vector.h>
+#include <asm/switch_to.h>
+#include <asm/simd.h>
+
+DEFINE_PER_CPU(bool, vector_context_busy);
+
+/*
+ * Claim ownership of the CPU vector context for use by the calling context.
+ *
+ * The caller may freely manipulate the vector context metadata until
+ * put_cpu_vector_context() is called.
+ */
+static void get_cpu_vector_context(void)
+{
+	bool busy;
+
+	preempt_disable();
+	busy = __this_cpu_xchg(vector_context_busy, true);
+
+	WARN_ON(busy);
+}
+
+/*
+ * Release the CPU vector context.
+ *
+ * Must be called from a context in which get_cpu_vector_context() was
+ * previously called, with no call to put_cpu_vector_context() in the
+ * meantime.
+ */
+static void put_cpu_vector_context(void)
+{
+	bool busy = __this_cpu_xchg(vector_context_busy, false);
+
+	WARN_ON(!busy);
+	preempt_enable();
+}
+
+/*
+ * kernel_vector_begin(): obtain the CPU vector registers for use by the calling
+ * context
+ *
+ * Must not be called unless may_use_simd() returns true.
+ * Task context in the vector registers is saved back to memory as necessary.
+ *
+ * A matching call to kernel_vector_end() must be made before returning from the
+ * calling context.
+ *
+ * The caller may freely use the vector registers until kernel_vector_end() is
+ * called.
+ */

arch/riscv/kernel/kernel_mode_vector.c line 73

Quoted 10 lines
arch/riscv/kernel/kernel_mode_vector.c new file +10 -0
+void kernel_vector_begin(void)
+{
+	if (WARN_ON(!has_vector()))
+		return;
+
+	BUG_ON(!may_use_simd());
+
+	riscv_v_vstate_save(current, task_pt_regs(current));
+
+	get_cpu_vector_context();

Could we do riscv_v_vstate_save() during preempt_enable()?

Should it be:
get_cpu_vector_context();

riscv_v_vstate_save(current, task_pt_regs(current));

Quoted 16 lines
arch/riscv/kernel/kernel_mode_vector.c new file +16 -0
+
+	riscv_v_enable();
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(kernel_vector_begin);
+
+/*
+ * kernel_vector_end(): give the CPU vector registers back to the current task
+ *
+ * Must be called from a context in which kernel_vector_begin() was previously
+ * called, with no call to kernel_vector_end() in the meantime.
+ *
+ * The caller must not use the vector registers after this function is called,
+ * unless kernel_vector_begin() is called again in the meantime.
+ */

arch/riscv/kernel/kernel_mode_vector.c line 99

Quoted 10 lines
arch/riscv/kernel/kernel_mode_vector.c new file +10 -0
+void kernel_vector_end(void)
+{
+	if (WARN_ON(!has_vector()))
+		return;
+
+	riscv_v_vstate_set_restore(current, task_pt_regs(current));
+
+	riscv_v_disable();
+
+	put_cpu_vector_context();

Seems you know the issue, but why above stuff missed?

Quoted 11 lines
arch/riscv/kernel/kernel_mode_vector.c new file +2 -0
+}
+EXPORT_SYMBOL_GPL(kernel_vector_end);
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#8 lore raw

PATCH v2 3/5 riscv: Add vector extension XOR implementation

In reply to Andy Chiu

From: Greentime Hu <[email protected]>

This patch adds support for vector optimized XOR and it is tested in qemu.

---
Changelog v2:
- 's/rvv/vector/' (Conor)

---
 arch/riscv/include/asm/xor.h | 82 ++++++++++++++++++++++++++++++++++++
 arch/riscv/lib/Makefile      |  1 +
 arch/riscv/lib/xor.S         | 81 +++++++++++++++++++++++++++++++++++
 3 files changed, 164 insertions(+)
 create mode 100644 arch/riscv/include/asm/xor.h
 create mode 100644 arch/riscv/lib/xor.S
arch/riscv/include/asm/xor.h new file +82 -0
new file mode 100644
index 000000000000..903c3275f8d0
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2021 SiFive
+ */
+
+#include <linux/hardirq.h>
+#include <asm-generic/xor.h>
+#ifdef CONFIG_RISCV_ISA_V
+#include <asm/vector.h>
+#include <asm/switch_to.h>
+
+void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2);
+void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2,
+		 const unsigned long *__restrict p3);
+void xor_regs_4_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2,
+		 const unsigned long *__restrict p3,
+		 const unsigned long *__restrict p4);
+void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2,
+		 const unsigned long *__restrict p3,
+		 const unsigned long *__restrict p4,
+		 const unsigned long *__restrict p5);
+
+static void xor_vector_2(unsigned long bytes, unsigned long *__restrict p1,
+			 const unsigned long *__restrict p2)
+{
+	kernel_vector_begin();
+	xor_regs_2_(bytes, p1, p2);
+	kernel_vector_end();
+}
+
+static void xor_vector_3(unsigned long bytes, unsigned long *__restrict p1,
+			 const unsigned long *__restrict p2,
+			 const unsigned long *__restrict p3)
+{
+	kernel_vector_begin();
+	xor_regs_3_(bytes, p1, p2, p3);
+	kernel_vector_end();
+}
+
+static void xor_vector_4(unsigned long bytes, unsigned long *__restrict p1,
+			 const unsigned long *__restrict p2,
+			 const unsigned long *__restrict p3,
+			 const unsigned long *__restrict p4)
+{
+	kernel_vector_begin();
+	xor_regs_4_(bytes, p1, p2, p3, p4);
+	kernel_vector_end();
+}
+
+static void xor_vector_5(unsigned long bytes, unsigned long *__restrict p1,
+			 const unsigned long *__restrict p2,
+			 const unsigned long *__restrict p3,
+			 const unsigned long *__restrict p4,
+			 const unsigned long *__restrict p5)
+{
+	kernel_vector_begin();
+	xor_regs_5_(bytes, p1, p2, p3, p4, p5);
+	kernel_vector_end();
+}
+
+static struct xor_block_template xor_block_rvv = {
+	.name = "rvv",
+	.do_2 = xor_vector_2,
+	.do_3 = xor_vector_3,
+	.do_4 = xor_vector_4,
+	.do_5 = xor_vector_5
+};
+
+#undef XOR_TRY_TEMPLATES
+#define XOR_TRY_TEMPLATES           \
+	do {        \
+		xor_speed(&xor_block_8regs);    \
+		xor_speed(&xor_block_32regs);    \
+		if (has_vector()) { \
+			xor_speed(&xor_block_rvv);\
+		} \
+	} while (0)
+#endif
arch/riscv/lib/Makefile +1 -0
index 26cb2502ecf8..494f9cd1a00c 100644
@@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT)	+= tishift.o
 lib-$(CONFIG_RISCV_ISA_ZICBOZ)	+= clear_page.o
 
 obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
+lib-$(CONFIG_RISCV_ISA_V)	+= xor.o
arch/riscv/lib/xor.S new file +81 -0

81 lines not on this page · raw message

Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Acked-by on v2 3/5
#9 lore raw

Re: [v2, 3/5] riscv: Add vector extension XOR implementation

Quoted 9 lines · Andy Chiu

From: Greentime Hu <[email protected]>

This patch adds support for vector optimized XOR and it is tested in qemu.

Other than the inherited build failure from 2/5, this seems okay to me. I have no opinion on the asm bits, so

Thanks,
Conor.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#10 lore raw

PATCH v2 4/5 riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}()

In reply to Andy Chiu

riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused.

---
Changelog v2:
- fix build fail that get caught on this patch (Conor)

---
 arch/riscv/include/asm/entry-common.h  |  2 +-
 arch/riscv/include/asm/vector.h        | 14 +++++---------
 arch/riscv/kernel/kernel_mode_vector.c |  2 +-
 arch/riscv/kernel/ptrace.c             |  2 +-
 arch/riscv/kernel/signal.c             |  2 +-
 5 files changed, 9 insertions(+), 13 deletions(-)
arch/riscv/include/asm/entry-common.h +1 -1
index 52926f4d8d7c..aa1b9e50d6c8 100644
@@ -12,7 +12,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs,
 {
 	if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) {
 		clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE);
-		riscv_v_vstate_restore(current, regs);
+		riscv_v_vstate_restore(&current->thread.vstate, regs);
 	}
 }
 
arch/riscv/include/asm/vector.h +5 -9
index b46b8f3261fa..3b783b317112 100644
@@ -163,23 +163,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs)
 	__riscv_v_vstate_dirty(regs);
 }
 
-static inline void riscv_v_vstate_save(struct task_struct *task,
+static inline void riscv_v_vstate_save(struct __riscv_v_ext_state *vstate,
 				       struct pt_regs *regs)
 {
 	if ((regs->status & SR_VS) == SR_VS_DIRTY) {
-		struct __riscv_v_ext_state *vstate = &task->thread.vstate;
-
 		__riscv_v_vstate_save(vstate, vstate->datap);
 		__riscv_v_vstate_clean(regs);
 	}
 }
 
-static inline void riscv_v_vstate_restore(struct task_struct *task,
+static inline void riscv_v_vstate_restore(struct __riscv_v_ext_state *vstate,
 					  struct pt_regs *regs)
 {
 	if ((regs->status & SR_VS) != SR_VS_OFF) {
-		struct __riscv_v_ext_state *vstate = &task->thread.vstate;
-
 		__riscv_v_vstate_restore(vstate, vstate->datap);
 		__riscv_v_vstate_clean(regs);
 	}
@@ -200,7 +196,7 @@ static inline void __switch_to_vector(struct task_struct *prev,
 	struct pt_regs *regs;
 
 	regs = task_pt_regs(prev);
-	riscv_v_vstate_save(prev, regs);
+	riscv_v_vstate_save(&prev->thread.vstate, regs);
 	riscv_v_vstate_set_restore(next, task_pt_regs(next));
 }
 
@@ -218,8 +214,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; }
 static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; }
 #define riscv_v_vsize (0)
 #define riscv_v_vstate_discard(regs)		do {} while (0)
-#define riscv_v_vstate_save(task, regs)		do {} while (0)
-#define riscv_v_vstate_restore(task, regs)	do {} while (0)
+#define riscv_v_vstate_save(vstate, regs)	do {} while (0)
+#define riscv_v_vstate_restore(vstate, regs)	do {} while (0)
 #define __switch_to_vector(__prev, __next)	do {} while (0)
 #define riscv_v_vstate_off(regs)		do {} while (0)
 #define riscv_v_vstate_on(regs)			do {} while (0)
arch/riscv/kernel/kernel_mode_vector.c +1 -1
index 1c3b32d2b340..d9e097e68937 100644
@@ -68,7 +68,7 @@ void kernel_vector_begin(void)
 
 	BUG_ON(!may_use_simd());
 
-	riscv_v_vstate_save(current, task_pt_regs(current));
+	riscv_v_vstate_save(&current->thread.vstate, task_pt_regs(current));
 
 	get_cpu_vector_context();
 
arch/riscv/kernel/ptrace.c +1 -1
index 1d572cf3140f..85e7167245cc 100644
@@ -99,7 +99,7 @@ static int riscv_vr_get(struct task_struct *target,
 	 * copying them to membuf.
 	 */
 	if (target == current)
-		riscv_v_vstate_save(current, task_pt_regs(current));
+		riscv_v_vstate_save(&current->thread.vstate, task_pt_regs(current));
 
 	/* Copy vector header from vstate. */
 	membuf_write(&to, vstate, offsetof(struct __riscv_v_ext_state, datap));
arch/riscv/kernel/signal.c +1 -1
index 0fca2c128b5f..75fd8cc05e10 100644
@@ -86,7 +86,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec)
 	/* datap is designed to be 16 byte aligned for better performance */
 	WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16)));
 
-	riscv_v_vstate_save(current, regs);
+	riscv_v_vstate_save(&current->thread.vstate, regs);
 	/* Copy everything of vstate but datap. */
 	err = __copy_to_user(&state->v_state, &current->thread.vstate,
 			     offsetof(struct __riscv_v_ext_state, datap));
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
author
#11 lore raw

PATCH v2 5/5 riscv: vector: allow kernel-mode Vector with preemption

In reply to Andy Chiu

Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide trap_pt_regs to let context save/restore routine reference status.VS at which the trap takes place. The thread flag TIF_RISCV_V_KERNEL_MODE indicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs to kernel_vstate and restore V-regs immediately from kernel_vstate if the bit is set.

Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the RISCV_V_VSTATE_CTRL_PREEMPTIBLE mask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code.

Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context swicth overhead.

---
Changelog v2:
- fix build fail when compiling without RISCV_ISA_V (Conor)
- 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE' and add comment (Conor)
- merge Kconfig patch into this oine (Conor).
- 's/CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV/CONFIG_RISCV_ISA_V_PREEMPTIVE/' (Conor)
- fix some typos (Conor)
- enclose assembly with RISCV_ISA_V_PREEMPTIVE.
- change riscv_v_vstate_ctrl_config_kmv() to kernel_vector_allow_preemption() for better understanding. (Conor)
- 's/riscv_v_kmv_preempitble/kernel_vector_preemptible/'

---
 arch/riscv/Kconfig                     | 10 +++++
 arch/riscv/include/asm/processor.h     |  2 +
 arch/riscv/include/asm/simd.h          |  4 +-
 arch/riscv/include/asm/thread_info.h   |  4 ++
 arch/riscv/include/asm/vector.h        | 27 +++++++++++--
 arch/riscv/kernel/asm-offsets.c        |  2 +
 arch/riscv/kernel/entry.S              | 45 ++++++++++++++++++++++
 arch/riscv/kernel/kernel_mode_vector.c | 53 ++++++++++++++++++++++++--
 arch/riscv/kernel/process.c            |  8 +++-
 arch/riscv/kernel/vector.c             |  3 +-
 10 files changed, 148 insertions(+), 10 deletions(-)
arch/riscv/Kconfig +10 -0
index 4c07b9189c86..0622951b15dd 100644
@@ -507,6 +507,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
 
 	  If you don't know what to do here, say Y.
 
+config RISCV_ISA_V_PREEMPTIVE
+	bool "Run kernel-mode Vector with kernel preemption"
+	depends on PREEMPTION
+	depends on RISCV_ISA_V
+	default y
+	help
+	  Ordinarily the kernel disables preemption before running in-kernel
+	  Vector code. This config frees the kernel from disabling preemption
+	  by adding memory on demand for tracking kernel's V-context.
+
 config TOOLCHAIN_HAS_ZBB
 	bool
 	default y
arch/riscv/include/asm/processor.h +2 -0
index c950a8d9edef..497c0dd30b2a 100644
@@ -42,6 +42,8 @@ struct thread_struct {
 	unsigned long bad_cause;
 	unsigned long vstate_ctrl;
 	struct __riscv_v_ext_state vstate;
+	struct pt_regs *trap_pt_regs;
+	struct __riscv_v_ext_state kernel_vstate;
 };
 
 /* Whitelist the fstate from the task_struct for hardened usercopy */
arch/riscv/include/asm/simd.h +3 -1
index ef70af78005d..a54a0ce58f4d 100644
@@ -12,6 +12,7 @@
 #include <linux/percpu.h>
 #include <linux/preempt.h>
 #include <linux/types.h>
+#include <linux/thread_info.h>
 
 #ifdef CONFIG_RISCV_ISA_V
 
@@ -35,7 +36,8 @@ static __must_check inline bool may_use_simd(void)
 	 * where it is set.
 	 */
 	return !in_irq() && !irqs_disabled() && !in_nmi() &&
-	       !this_cpu_read(vector_context_busy);
+	       !this_cpu_read(vector_context_busy) &&
+	       !test_thread_flag(TIF_RISCV_V_KERNEL_MODE);
 }
 
 #else /* ! CONFIG_RISCV_ISA_V */
arch/riscv/include/asm/thread_info.h +4 -0
index b182f2d03e25..8797d520e8ef 100644
@@ -94,6 +94,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define TIF_UPROBE		10	/* uprobe breakpoint or singlestep */
 #define TIF_32BIT		11	/* compat-mode 32bit process */
 #define TIF_RISCV_V_DEFER_RESTORE	12 /* restore Vector before returing to user */
+#define TIF_RISCV_V_KERNEL_MODE			13 /* kernel-mode Vector run with preemption-on */
 
 #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
@@ -101,9 +102,12 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define _TIF_NOTIFY_SIGNAL	(1 << TIF_NOTIFY_SIGNAL)
 #define _TIF_UPROBE		(1 << TIF_UPROBE)
 #define _TIF_RISCV_V_DEFER_RESTORE	(1 << TIF_RISCV_V_DEFER_RESTORE)
+#define _TIF_RISCV_V_KERNEL_MODE	(1 << TIF_RISCV_V_KERNEL_MODE)
 
 #define _TIF_WORK_MASK \
 	(_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \
 	 _TIF_NOTIFY_SIGNAL | _TIF_UPROBE)
 
+#define RISCV_V_VSTATE_CTRL_PREEMPTIBLE	0x20
+
 #endif /* _ASM_RISCV_THREAD_INFO_H */
arch/riscv/include/asm/vector.h +24 -3
index 3b783b317112..c2776851d50d 100644
@@ -195,9 +195,24 @@ static inline void __switch_to_vector(struct task_struct *prev,
 {
 	struct pt_regs *regs;
 
-	regs = task_pt_regs(prev);
-	riscv_v_vstate_save(&prev->thread.vstate, regs);
-	riscv_v_vstate_set_restore(next, task_pt_regs(next));
+	if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) &&
+	    test_tsk_thread_flag(prev, TIF_RISCV_V_KERNEL_MODE)) {
+		regs = prev->thread.trap_pt_regs;
+		WARN_ON(!regs);
+		riscv_v_vstate_save(&prev->thread.kernel_vstate, regs);
+	} else {
+		regs = task_pt_regs(prev);
+		riscv_v_vstate_save(&prev->thread.vstate, regs);
+	}
+
+	if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) &&
+	    test_tsk_thread_flag(next, TIF_RISCV_V_KERNEL_MODE)) {
+		regs = next->thread.trap_pt_regs;
+		WARN_ON(!regs);
+		riscv_v_vstate_restore(&next->thread.kernel_vstate, regs);
+	} else {
+		riscv_v_vstate_set_restore(next, task_pt_regs(next));
+	}
 }
 
 void riscv_v_vstate_ctrl_init(struct task_struct *tsk);
@@ -222,4 +237,10 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; }
 
 #endif /* CONFIG_RISCV_ISA_V */
 
+#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
+void kernel_vector_allow_preemption(void);
+#else
+#define kernel_vector_allow_preemption()	do {} while (0)
+#endif
+
 #endif /* ! __ASM_RISCV_VECTOR_H */
arch/riscv/kernel/asm-offsets.c +2 -0
index d6a75aac1d27..4b062f7741b2 100644
@@ -38,6 +38,8 @@ void asm_offsets(void)
 	OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count);
 	OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp);
 	OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp);
+	OFFSET(TASK_THREAD_TRAP_REGP, task_struct, thread.trap_pt_regs);
+	OFFSET(TASK_THREAD_VSTATE_CTRL, task_struct, thread.vstate_ctrl);
 
 	OFFSET(TASK_THREAD_F0,  task_struct, thread.fstate.f[0]);
 	OFFSET(TASK_THREAD_F1,  task_struct, thread.fstate.f[1]);
arch/riscv/kernel/entry.S +45 -0

57 lines not on this page · raw message

arch/riscv/kernel/kernel_mode_vector.c +49 -4

78 lines not on this page · raw message

arch/riscv/kernel/process.c +7 -1
index ec89e7edb6fd..18cb37c305ab 100644
@@ -160,8 +160,11 @@ void flush_thread(void)
 void arch_release_task_struct(struct task_struct *tsk)
 {
 	/* Free the vector context of datap. */
-	if (has_vector())
+	if (has_vector()) {
 		kfree(tsk->thread.vstate.datap);
+		if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE))
+			kfree(tsk->thread.kernel_vstate.datap);
+	}
 }
 
 int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
@@ -170,7 +173,9 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
 	*dst = *src;
 	/* clear entire V context, including datap for a new task */
 	memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
+	memset(&dst->thread.kernel_vstate, 0, sizeof(struct __riscv_v_ext_state));
 	clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE);
+	clear_tsk_thread_flag(dst, TIF_RISCV_V_KERNEL_MODE);
 
 	return 0;
 }
@@ -205,6 +210,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 		childregs->a0 = 0; /* Return value of fork() */
 		p->thread.s[0] = 0;
 	}
+	kernel_vector_allow_preemption();
 	p->thread.ra = (unsigned long)ret_from_fork;
 	p->thread.sp = (unsigned long)childregs; /* kernel sp */
 	return 0;
arch/riscv/kernel/vector.c +2 -1
index 9d583b760db4..42f227077ee5 100644
@@ -122,7 +122,8 @@ static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt,
 	ctrl |= VSTATE_CTRL_MAKE_NEXT(nxt);
 	if (inherit)
 		ctrl |= PR_RISCV_V_VSTATE_CTRL_INHERIT;
-	tsk->thread.vstate_ctrl = ctrl;
+	tsk->thread.vstate_ctrl &= ~PR_RISCV_V_VSTATE_CTRL_MASK;
+	tsk->thread.vstate_ctrl |= ctrl;
 }
 
 bool riscv_v_vstate_ctrl_user_allowed(void)
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#12 lore raw

Re: [v2, 5/5] riscv: vector: allow kernel-mode Vector with preemption

Hey Andy,

Quoted 134 lines · Andy Chiu

Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide trap_pt_regs to let context save/restore routine reference status.VS at which the trap takes place. The thread flag TIF_RISCV_V_KERNEL_MODE indicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs to kernel_vstate and restore V-regs immediately from kernel_vstate if the bit is set.

Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the RISCV_V_VSTATE_CTRL_PREEMPTIBLE mask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code.

Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context swicth overhead.

---
Changelog v2:
- fix build fail when compiling without RISCV_ISA_V (Conor)
- 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE' and add comment (Conor)
- merge Kconfig patch into this oine (Conor).
- 's/CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV/CONFIG_RISCV_ISA_V_PREEMPTIVE/' (Conor)
- fix some typos (Conor)
- enclose assembly with RISCV_ISA_V_PREEMPTIVE.
- change riscv_v_vstate_ctrl_config_kmv() to kernel_vector_allow_preemption() for better understanding. (Conor)
- 's/riscv_v_kmv_preempitble/kernel_vector_preemptible/'

---
 arch/riscv/Kconfig                     | 10 +++++
 arch/riscv/include/asm/processor.h     |  2 +
 arch/riscv/include/asm/simd.h          |  4 +-
 arch/riscv/include/asm/thread_info.h   |  4 ++
 arch/riscv/include/asm/vector.h        | 27 +++++++++++--
 arch/riscv/kernel/asm-offsets.c        |  2 +
 arch/riscv/kernel/entry.S              | 45 ++++++++++++++++++++++
 arch/riscv/kernel/kernel_mode_vector.c | 53 ++++++++++++++++++++++++--
 arch/riscv/kernel/process.c            |  8 +++-
 arch/riscv/kernel/vector.c             |  3 +-
 10 files changed, 148 insertions(+), 10 deletions(-)
arch/riscv/Kconfig +10 -0
index 4c07b9189c86..0622951b15dd 100644
@@ -507,6 +507,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
 
 	  If you don't know what to do here, say Y.
 
+config RISCV_ISA_V_PREEMPTIVE
+	bool "Run kernel-mode Vector with kernel preemption"
+	depends on PREEMPTION
+	depends on RISCV_ISA_V
+	default y
+	help
+	  Ordinarily the kernel disables preemption before running in-kernel
+	  Vector code. This config frees the kernel from disabling preemption
+	  by adding memory on demand for tracking kernel's V-context.
+
 config TOOLCHAIN_HAS_ZBB
 	bool
 	default y
arch/riscv/include/asm/processor.h +2 -0
index c950a8d9edef..497c0dd30b2a 100644
@@ -42,6 +42,8 @@ struct thread_struct {
 	unsigned long bad_cause;
 	unsigned long vstate_ctrl;
 	struct __riscv_v_ext_state vstate;
+	struct pt_regs *trap_pt_regs;
+	struct __riscv_v_ext_state kernel_vstate;
 };
 
 /* Whitelist the fstate from the task_struct for hardened usercopy */
arch/riscv/include/asm/simd.h +3 -1
index ef70af78005d..a54a0ce58f4d 100644
@@ -12,6 +12,7 @@
 #include <linux/percpu.h>
 #include <linux/preempt.h>
 #include <linux/types.h>
+#include <linux/thread_info.h>
 
 #ifdef CONFIG_RISCV_ISA_V
 
@@ -35,7 +36,8 @@ static __must_check inline bool may_use_simd(void)
 	 * where it is set.
 	 */
 	return !in_irq() && !irqs_disabled() && !in_nmi() &&
-	       !this_cpu_read(vector_context_busy);
+	       !this_cpu_read(vector_context_busy) &&
+	       !test_thread_flag(TIF_RISCV_V_KERNEL_MODE);
 }
 
 #else /* ! CONFIG_RISCV_ISA_V */
arch/riscv/include/asm/thread_info.h +4 -0
index b182f2d03e25..8797d520e8ef 100644
@@ -94,6 +94,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define TIF_UPROBE		10	/* uprobe breakpoint or singlestep */
 #define TIF_32BIT		11	/* compat-mode 32bit process */
 #define TIF_RISCV_V_DEFER_RESTORE	12 /* restore Vector before returing to user */
+#define TIF_RISCV_V_KERNEL_MODE			13 /* kernel-mode Vector run with preemption-on */
 
 #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
@@ -101,9 +102,12 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define _TIF_NOTIFY_SIGNAL	(1 << TIF_NOTIFY_SIGNAL)
 #define _TIF_UPROBE		(1 << TIF_UPROBE)
 #define _TIF_RISCV_V_DEFER_RESTORE	(1 << TIF_RISCV_V_DEFER_RESTORE)
+#define _TIF_RISCV_V_KERNEL_MODE	(1 << TIF_RISCV_V_KERNEL_MODE)
 
 #define _TIF_WORK_MASK \
 	(_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \
 	 _TIF_NOTIFY_SIGNAL | _TIF_UPROBE)
 
+#define RISCV_V_VSTATE_CTRL_PREEMPTIBLE	0x20
+
 #endif /* _ASM_RISCV_THREAD_INFO_H */
arch/riscv/include/asm/vector.h +0 -0
index 3b783b317112..c2776851d50d 100644

arch/riscv/include/asm/vector.h static inline void __switch_to_vector(struct task_struct *prev, · line 201

Quoted 10 lines
arch/riscv/include/asm/vector.h +4 -3
@@ -195,9 +195,24 @@ static inline void __switch_to_vector(struct task_struct *prev,
 {
 	struct pt_regs *regs;
 
-	regs = task_pt_regs(prev);
-	riscv_v_vstate_save(&prev->thread.vstate, regs);
-	riscv_v_vstate_set_restore(next, task_pt_regs(next));
+	if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) &&
+	    test_tsk_thread_flag(prev, TIF_RISCV_V_KERNEL_MODE)) {
+		regs = prev->thread.trap_pt_regs;
+		WARN_ON(!regs);

In what cases could these WARN_ON()s be triggered?

Quoted 21 lines
arch/riscv/include/asm/vector.h +14 -0
+		riscv_v_vstate_save(&prev->thread.kernel_vstate, regs);
+	} else {
+		regs = task_pt_regs(prev);
+		riscv_v_vstate_save(&prev->thread.vstate, regs);
+	}
+
+	if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) &&
+	    test_tsk_thread_flag(next, TIF_RISCV_V_KERNEL_MODE)) {
+		regs = next->thread.trap_pt_regs;
+		WARN_ON(!regs);
+		riscv_v_vstate_restore(&next->thread.kernel_vstate, regs);
+	} else {
+		riscv_v_vstate_set_restore(next, task_pt_regs(next));
+	}
 }
 
 

/*
* kernel_vector_begin(): obtain the CPU vector registers for use by the calling
* context

arch/riscv/include/asm/vector.h void kernel_vector_begin(void) · line 116

Quoted 9 lines
arch/riscv/include/asm/vector.h +5 -1
@@ -70,11 +109,14 @@ void kernel_vector_begin(void)
 
 	riscv_v_vstate_save(&current->thread.vstate, task_pt_regs(current));
 
-	get_cpu_vector_context();
+	if (!preemptible() || !kernel_vector_preemptible()) {
+		get_cpu_vector_context();
+	} else {
+		if (riscv_v_start_kernel_context())
+			get_cpu_vector_context();

What happens here if riscv_v_start_kernel_context() fails w/ -ENOMEM?

Quoted 9 lines
arch/riscv/include/asm/vector.h +1 -2
+	}
 
 	riscv_v_enable();
-
-	return 0;
 }
 EXPORT_SYMBOL_GPL(kernel_vector_begin);
 

arch/riscv/include/asm/vector.h void kernel_vector_end(void) · line 145

Quoted 9 lines
arch/riscv/include/asm/vector.h +4 -1
@@ -96,6 +138,9 @@  void kernel_vector_end(void)
 
 	riscv_v_disable();
 
-	put_cpu_vector_context();
+	if (!test_thread_flag(TIF_RISCV_V_KERNEL_MODE))
+		put_cpu_vector_context();
+	else
+		riscv_v_stop_kernel_context();
 }

Probably just missing something here, but how come we don't need to call put_cpu_vector_context() here. I'm just a little confused, since, in kernel_vector_begin, get_cpu_vector_context() is called.

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#13 lore raw

Re: [v2, 5/5] riscv: vector: allow kernel-mode Vector with preemption

Quoted 141 lines · Conor Dooley

Hey Andy,

Quoted 137 lines · Andy Chiu

Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide trap_pt_regs to let context save/restore routine reference status.VS at which the trap takes place. The thread flag TIF_RISCV_V_KERNEL_MODE indicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs to kernel_vstate and restore V-regs immediately from kernel_vstate if the bit is set.

Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the RISCV_V_VSTATE_CTRL_PREEMPTIBLE mask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code.

Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context swicth overhead.

---
Changelog v2:
- fix build fail when compiling without RISCV_ISA_V (Conor)
- 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE' and add comment (Conor)
- merge Kconfig patch into this oine (Conor).
- 's/CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV/CONFIG_RISCV_ISA_V_PREEMPTIVE/' (Conor)
- fix some typos (Conor)
- enclose assembly with RISCV_ISA_V_PREEMPTIVE.
- change riscv_v_vstate_ctrl_config_kmv() to kernel_vector_allow_preemption() for better understanding. (Conor)
- 's/riscv_v_kmv_preempitble/kernel_vector_preemptible/'

---
 arch/riscv/Kconfig                     | 10 +++++
 arch/riscv/include/asm/processor.h     |  2 +
 arch/riscv/include/asm/simd.h          |  4 +-
 arch/riscv/include/asm/thread_info.h   |  4 ++
 arch/riscv/include/asm/vector.h        | 27 +++++++++++--
 arch/riscv/kernel/asm-offsets.c        |  2 +
 arch/riscv/kernel/entry.S              | 45 ++++++++++++++++++++++
 arch/riscv/kernel/kernel_mode_vector.c | 53 ++++++++++++++++++++++++--
 arch/riscv/kernel/process.c            |  8 +++-
 arch/riscv/kernel/vector.c             |  3 +-
 10 files changed, 148 insertions(+), 10 deletions(-)
arch/riscv/Kconfig +10 -0
index 4c07b9189c86..0622951b15dd 100644
@@ -507,6 +507,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
 
        If you don't know what to do here, say Y.
 
+config RISCV_ISA_V_PREEMPTIVE
+     bool "Run kernel-mode Vector with kernel preemption"
+     depends on PREEMPTION
+     depends on RISCV_ISA_V
+     default y
+     help
+       Ordinarily the kernel disables preemption before running in-kernel
+       Vector code. This config frees the kernel from disabling preemption
+       by adding memory on demand for tracking kernel's V-context.
+
 config TOOLCHAIN_HAS_ZBB
      bool
      default y
arch/riscv/include/asm/processor.h +2 -0
index c950a8d9edef..497c0dd30b2a 100644
@@ -42,6 +42,8 @@ struct thread_struct {
      unsigned long bad_cause;
      unsigned long vstate_ctrl;
      struct __riscv_v_ext_state vstate;
+     struct pt_regs *trap_pt_regs;
+     struct __riscv_v_ext_state kernel_vstate;
 };
 
 /* Whitelist the fstate from the task_struct for hardened usercopy */
arch/riscv/include/asm/simd.h +3 -1
index ef70af78005d..a54a0ce58f4d 100644
@@ -12,6 +12,7 @@
 #include <linux/percpu.h>
 #include <linux/preempt.h>
 #include <linux/types.h>
+#include <linux/thread_info.h>
 
 #ifdef CONFIG_RISCV_ISA_V
 
@@ -35,7 +36,8 @@ static __must_check inline bool may_use_simd(void)
       * where it is set.
       */
      return !in_irq() && !irqs_disabled() && !in_nmi() &&
-            !this_cpu_read(vector_context_busy);
+            !this_cpu_read(vector_context_busy) &&
+            !test_thread_flag(TIF_RISCV_V_KERNEL_MODE);
 }
 
 #else /* ! CONFIG_RISCV_ISA_V */
arch/riscv/include/asm/thread_info.h +4 -0
index b182f2d03e25..8797d520e8ef 100644
@@ -94,6 +94,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define TIF_UPROBE           10      /* uprobe breakpoint or singlestep */
 #define TIF_32BIT            11      /* compat-mode 32bit process */
 #define TIF_RISCV_V_DEFER_RESTORE    12 /* restore Vector before returing to user */
+#define TIF_RISCV_V_KERNEL_MODE                      13 /* kernel-mode Vector run with preemption-on */
 
 #define _TIF_NOTIFY_RESUME   (1 << TIF_NOTIFY_RESUME)
 #define _TIF_SIGPENDING              (1 << TIF_SIGPENDING)
@@ -101,9 +102,12 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define _TIF_NOTIFY_SIGNAL   (1 << TIF_NOTIFY_SIGNAL)
 #define _TIF_UPROBE          (1 << TIF_UPROBE)
 #define _TIF_RISCV_V_DEFER_RESTORE   (1 << TIF_RISCV_V_DEFER_RESTORE)
+#define _TIF_RISCV_V_KERNEL_MODE     (1 << TIF_RISCV_V_KERNEL_MODE)
 
 #define _TIF_WORK_MASK \
      (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \
       _TIF_NOTIFY_SIGNAL | _TIF_UPROBE)
 
+#define RISCV_V_VSTATE_CTRL_PREEMPTIBLE      0x20
+
 #endif /* _ASM_RISCV_THREAD_INFO_H */
arch/riscv/include/asm/vector.h +0 -0
index 3b783b317112..c2776851d50d 100644
@@ -195,9 +195,24 @@ static inline void __switch_to_vector(struct task_struct *prev,
 {
      struct pt_regs *regs;
 
Quoted 9 lines

arch/riscv/include/asm/vector.h static inline void __switch_to_vector(struct task_struct *prev, · line 201

Quoted 7 lines
arch/riscv/include/asm/vector.h +4 -3
-     regs = task_pt_regs(prev);
-     riscv_v_vstate_save(&prev->thread.vstate, regs);
-     riscv_v_vstate_set_restore(next, task_pt_regs(next));
+     if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) &&
+         test_tsk_thread_flag(prev, TIF_RISCV_V_KERNEL_MODE)) {
+             regs = prev->thread.trap_pt_regs;
+             WARN_ON(!regs);

In what cases could these WARN_ON()s be triggered?

It probably happens when a kernel thread calls schedule() in the middle of preemptible kernel mode Vector code. Because the kernel sets trap_pt_regs only at trap entries. For example

// assume preemption = "ON" and memory allocation
// for kernel_vstate.datap success
kernel_vector_begin();
// some vector code
...
schedule();
...
kernel_vector_end();

It is possible to support making scheduler calls in preemptible kernel mode Vector though. We just need to save nothing (all V regs are caller-save) and set an appropriate status.VS for the "next" process.

Quoted 25 lines
Quoted 25 lines
arch/riscv/include/asm/vector.h +14 -0
+             riscv_v_vstate_save(&prev->thread.kernel_vstate, regs);
+     } else {
+             regs = task_pt_regs(prev);
+             riscv_v_vstate_save(&prev->thread.vstate, regs);
+     }
+
+     if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) &&
+         test_tsk_thread_flag(next, TIF_RISCV_V_KERNEL_MODE)) {
+             regs = next->thread.trap_pt_regs;
+             WARN_ON(!regs);
+             riscv_v_vstate_restore(&next->thread.kernel_vstate, regs);
+     } else {
+             riscv_v_vstate_set_restore(next, task_pt_regs(next));
+     }
 }
 
 

/*
* kernel_vector_begin(): obtain the CPU vector registers for use by the calling
* context

arch/riscv/include/asm/vector.h +0 -0
@@ -70,11 +109,14 @@ void kernel_vector_begin(void)
 
      riscv_v_vstate_save(&current->thread.vstate, task_pt_regs(current));
 
Quoted 8 lines

arch/riscv/include/asm/vector.h void kernel_vector_begin(void) · line 116

Quoted 6 lines
arch/riscv/include/asm/vector.h +5 -1
-     get_cpu_vector_context();
+     if (!preemptible() || !kernel_vector_preemptible()) {
+             get_cpu_vector_context();
+     } else {
+             if (riscv_v_start_kernel_context())
+                     get_cpu_vector_context();

What happens here if riscv_v_start_kernel_context() fails w/ -ENOMEM?

Here we would fallback to starting kernel-mode Vector with preemption disabled, by calling get_cpu_vector_context(). This makes calling kernel_vector_begin() end up with 2 possible consequences, if the caller runs in a preemptible context. One, which is the success path of riscv_v_start_kernel_context(), will not alter the preemption status but may increase memory usage if the context does not exist yet.

However, if, on the other path, riscv_v_start_kernel_context() fails with -ENOMEM, then the kernel-mode Vector code will be executed with preemption "off".

Another way of solving this ambiguity is to add another function to enable kernel mode Vector with preemption, and let the user check if the allocation fails. So users who really want to run their Vector code with preemption shall make this call. Otherwise, kernel mode Vector runs with preemption off. However, I don't really want to add it because I'd like to make the "upgrade" transparent to the caller.

Quoted 13 lines
Quoted 13 lines
arch/riscv/include/asm/vector.h +1 -2
+     }
 
      riscv_v_enable();
-
-     return 0;
 }
 EXPORT_SYMBOL_GPL(kernel_vector_begin);
 
@@ -96,6 +138,9 @@  void kernel_vector_end(void)
 
      riscv_v_disable();
 
Quoted 10 lines

arch/riscv/include/asm/vector.h void kernel_vector_end(void) · line 145

Quoted 6 lines
arch/riscv/include/asm/vector.h +4 -1
-     put_cpu_vector_context();
+     if (!test_thread_flag(TIF_RISCV_V_KERNEL_MODE))
+             put_cpu_vector_context();
+     else
+             riscv_v_stop_kernel_context();
 }

Probably just missing something here, but how come we don't need to call put_cpu_vector_context() here. I'm just a little confused, since, in kernel_vector_begin, get_cpu_vector_context() is called.

If "TIF_RISCV_V_KERNEL_MODE" is set, then we are running kernel-mode Vector with preemption "ON". In such cases we don't need to call put_cpu_vector_context(), which is the epilogue of kernel-mode Vector with preemption "OFF". Instead, we should call riscv_v_stop_kernel_context() to end the session.

Thanks,
Andy

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#14 lore raw

Re: [v2, 5/5] riscv: vector: allow kernel-mode Vector with preemption

Quoted 162 lines · Andy Chiu
Quoted 138 lines · Conor Dooley
Quoted 137 lines · Andy Chiu

Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide trap_pt_regs to let context save/restore routine reference status.VS at which the trap takes place. The thread flag TIF_RISCV_V_KERNEL_MODE indicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs to kernel_vstate and restore V-regs immediately from kernel_vstate if the bit is set.

Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the RISCV_V_VSTATE_CTRL_PREEMPTIBLE mask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code.

Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context swicth overhead.

---
Changelog v2:
- fix build fail when compiling without RISCV_ISA_V (Conor)
- 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE' and add comment (Conor)
- merge Kconfig patch into this oine (Conor).
- 's/CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV/CONFIG_RISCV_ISA_V_PREEMPTIVE/' (Conor)
- fix some typos (Conor)
- enclose assembly with RISCV_ISA_V_PREEMPTIVE.
- change riscv_v_vstate_ctrl_config_kmv() to kernel_vector_allow_preemption() for better understanding. (Conor)
- 's/riscv_v_kmv_preempitble/kernel_vector_preemptible/'

---
 arch/riscv/Kconfig                     | 10 +++++
 arch/riscv/include/asm/processor.h     |  2 +
 arch/riscv/include/asm/simd.h          |  4 +-
 arch/riscv/include/asm/thread_info.h   |  4 ++
 arch/riscv/include/asm/vector.h        | 27 +++++++++++--
 arch/riscv/kernel/asm-offsets.c        |  2 +
 arch/riscv/kernel/entry.S              | 45 ++++++++++++++++++++++
 arch/riscv/kernel/kernel_mode_vector.c | 53 ++++++++++++++++++++++++--
 arch/riscv/kernel/process.c            |  8 +++-
 arch/riscv/kernel/vector.c             |  3 +-
 10 files changed, 148 insertions(+), 10 deletions(-)
arch/riscv/Kconfig +10 -0
index 4c07b9189c86..0622951b15dd 100644
@@ -507,6 +507,16 @@ config RISCV_ISA_V_DEFAULT_ENABLE
 
        If you don't know what to do here, say Y.
 
+config RISCV_ISA_V_PREEMPTIVE
+     bool "Run kernel-mode Vector with kernel preemption"
+     depends on PREEMPTION
+     depends on RISCV_ISA_V
+     default y
+     help
+       Ordinarily the kernel disables preemption before running in-kernel
+       Vector code. This config frees the kernel from disabling preemption
+       by adding memory on demand for tracking kernel's V-context.
+
 config TOOLCHAIN_HAS_ZBB
      bool
      default y
arch/riscv/include/asm/processor.h +2 -0
index c950a8d9edef..497c0dd30b2a 100644
@@ -42,6 +42,8 @@ struct thread_struct {
      unsigned long bad_cause;
      unsigned long vstate_ctrl;
      struct __riscv_v_ext_state vstate;
+     struct pt_regs *trap_pt_regs;
+     struct __riscv_v_ext_state kernel_vstate;
 };
 
 /* Whitelist the fstate from the task_struct for hardened usercopy */
arch/riscv/include/asm/simd.h +3 -1
index ef70af78005d..a54a0ce58f4d 100644
@@ -12,6 +12,7 @@
 #include <linux/percpu.h>
 #include <linux/preempt.h>
 #include <linux/types.h>
+#include <linux/thread_info.h>
 
 #ifdef CONFIG_RISCV_ISA_V
 
@@ -35,7 +36,8 @@ static __must_check inline bool may_use_simd(void)
       * where it is set.
       */
      return !in_irq() && !irqs_disabled() && !in_nmi() &&
-            !this_cpu_read(vector_context_busy);
+            !this_cpu_read(vector_context_busy) &&
+            !test_thread_flag(TIF_RISCV_V_KERNEL_MODE);
 }
 
 #else /* ! CONFIG_RISCV_ISA_V */
arch/riscv/include/asm/thread_info.h +4 -0
index b182f2d03e25..8797d520e8ef 100644
@@ -94,6 +94,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define TIF_UPROBE           10      /* uprobe breakpoint or singlestep */
 #define TIF_32BIT            11      /* compat-mode 32bit process */
 #define TIF_RISCV_V_DEFER_RESTORE    12 /* restore Vector before returing to user */
+#define TIF_RISCV_V_KERNEL_MODE                      13 /* kernel-mode Vector run with preemption-on */
 
 #define _TIF_NOTIFY_RESUME   (1 << TIF_NOTIFY_RESUME)
 #define _TIF_SIGPENDING              (1 << TIF_SIGPENDING)
@@ -101,9 +102,12 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define _TIF_NOTIFY_SIGNAL   (1 << TIF_NOTIFY_SIGNAL)
 #define _TIF_UPROBE          (1 << TIF_UPROBE)
 #define _TIF_RISCV_V_DEFER_RESTORE   (1 << TIF_RISCV_V_DEFER_RESTORE)
+#define _TIF_RISCV_V_KERNEL_MODE     (1 << TIF_RISCV_V_KERNEL_MODE)
 
 #define _TIF_WORK_MASK \
      (_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \
       _TIF_NOTIFY_SIGNAL | _TIF_UPROBE)
 
+#define RISCV_V_VSTATE_CTRL_PREEMPTIBLE      0x20
+
 #endif /* _ASM_RISCV_THREAD_INFO_H */
arch/riscv/include/asm/vector.h +0 -0
index 3b783b317112..c2776851d50d 100644
@@ -195,9 +195,24 @@ static inline void __switch_to_vector(struct task_struct *prev,
 {
      struct pt_regs *regs;
 
Quoted 9 lines

arch/riscv/include/asm/vector.h static inline void __switch_to_vector(struct task_struct *prev, · line 201

Quoted 7 lines
arch/riscv/include/asm/vector.h +4 -3
-     regs = task_pt_regs(prev);
-     riscv_v_vstate_save(&prev->thread.vstate, regs);
-     riscv_v_vstate_set_restore(next, task_pt_regs(next));
+     if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) &&
+         test_tsk_thread_flag(prev, TIF_RISCV_V_KERNEL_MODE)) {
+             regs = prev->thread.trap_pt_regs;
+             WARN_ON(!regs);

In what cases could these WARN_ON()s be triggered?

It probably happens when a kernel thread calls schedule() in the middle of preemptible kernel mode Vector code. Because the kernel sets trap_pt_regs only at trap entries. For example

// assume preemption = "ON" and memory allocation
// for kernel_vstate.datap success
kernel_vector_begin();
// some vector code
...
schedule();
...
kernel_vector_end();

Quoted 3 lines

It is possible to support making scheduler calls in preemptible kernel mode Vector though. We just need to save nothing (all V regs are caller-save) and set an appropriate status.VS for the "next" process.

I'm struggling to theorycraft where this can go wrong, because my knowledge in this area is limited. If the only way it can go wrong is by calling schedule() in a "protected" section like this, that seems "okay". Are there not non-trap induced context switches that we need to worry about?

Quoted 76 lines
Quoted 24 lines
Quoted 24 lines
arch/riscv/include/asm/vector.h +14 -0
+             riscv_v_vstate_save(&prev->thread.kernel_vstate, regs);
+     } else {
+             regs = task_pt_regs(prev);
+             riscv_v_vstate_save(&prev->thread.vstate, regs);
+     }
+
+     if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) &&
+         test_tsk_thread_flag(next, TIF_RISCV_V_KERNEL_MODE)) {
+             regs = next->thread.trap_pt_regs;
+             WARN_ON(!regs);
+             riscv_v_vstate_restore(&next->thread.kernel_vstate, regs);
+     } else {
+             riscv_v_vstate_set_restore(next, task_pt_regs(next));
+     }
 }
 
 

/*
* kernel_vector_begin(): obtain the CPU vector registers for use by the calling
* context

arch/riscv/include/asm/vector.h +0 -0
@@ -70,11 +109,14 @@ void kernel_vector_begin(void)
 
      riscv_v_vstate_save(&current->thread.vstate, task_pt_regs(current));
 
Quoted 8 lines

arch/riscv/include/asm/vector.h void kernel_vector_begin(void) · line 116

Quoted 6 lines
arch/riscv/include/asm/vector.h +5 -1
-     get_cpu_vector_context();
+     if (!preemptible() || !kernel_vector_preemptible()) {
+             get_cpu_vector_context();
+     } else {
+             if (riscv_v_start_kernel_context())
+                     get_cpu_vector_context();

What happens here if riscv_v_start_kernel_context() fails w/ -ENOMEM?

Here we would fallback to starting kernel-mode Vector with preemption disabled, by calling get_cpu_vector_context(). This makes calling kernel_vector_begin() end up with 2 possible consequences, if the caller runs in a preemptible context. One, which is the success path of riscv_v_start_kernel_context(), will not alter the preemption status but may increase memory usage if the context does not exist yet.

However, if, on the other path, riscv_v_start_kernel_context() fails with -ENOMEM, then the kernel-mode Vector code will be executed with preemption "off".

Another way of solving this ambiguity is to add another function to enable kernel mode Vector with preemption, and let the user check if the allocation fails. So users who really want to run their Vector code with preemption shall make this call. Otherwise, kernel mode Vector runs with preemption off. However, I don't really want to add it because I'd like to make the "upgrade" transparent to the caller.

Quoted 13 lines
Quoted 13 lines
arch/riscv/include/asm/vector.h +1 -2
+     }
 
      riscv_v_enable();
-
-     return 0;
 }
 EXPORT_SYMBOL_GPL(kernel_vector_begin);
 
@@ -96,6 +138,9 @@  void kernel_vector_end(void)
 
      riscv_v_disable();
 
Quoted 10 lines

arch/riscv/include/asm/vector.h void kernel_vector_end(void) · line 145

Quoted 6 lines
arch/riscv/include/asm/vector.h +4 -1
-     put_cpu_vector_context();
+     if (!test_thread_flag(TIF_RISCV_V_KERNEL_MODE))
+             put_cpu_vector_context();
+     else
+             riscv_v_stop_kernel_context();
 }

Probably just missing something here, but how come we don't need to call put_cpu_vector_context() here. I'm just a little confused, since, in kernel_vector_begin, get_cpu_vector_context() is called.

Quoted 5 lines

If "TIF_RISCV_V_KERNEL_MODE" is set, then we are running kernel-mode Vector with preemption "ON". In such cases we don't need to call put_cpu_vector_context(), which is the epilogue of kernel-mode Vector with preemption "OFF". Instead, we should call riscv_v_stop_kernel_context() to end the session.

I think, for these last two comments, I screwed up. I misread if (riscv_v_start_kernel_context())
as
if (!riscv_v_start_kernel_context())
which is the source of my confusion about this being imbalanced.

Thanks for your explanations however!

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#15 lore raw

Re: [v2, 5/5] riscv: vector: allow kernel-mode Vector with preemption

In reply to Andy Chiu
Quoted 14 lines · Andy Chiu

Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide trap_pt_regs to let context save/restore routine reference status.VS at which the trap takes place. The thread flag TIF_RISCV_V_KERNEL_MODE indicates whether a task is running in kernel-mode Vector with preemption 'ON'. So context switch routines know and would save V-regs to kernel_vstate and restore V-regs immediately from kernel_vstate if the bit is set.

Apart from a task's preemption status, the capability of running preemptive kernel-mode Vector is jointly controlled by the RISCV_V_VSTATE_CTRL_PREEMPTIBLE mask in the task's thread.vstate_ctrl. This bit is masked whenever a trap takes place in kernel mode while executing preemptive Vector code.

Quoted 7 lines

Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context swicth overhead.

Nice idea! Did you perform any benchmarking? Would be really interesting to get some numbers.

Nit: "switch"

I like that the most "controversial" patch is last, so it can easily be dropped if the discussions doesn't settle! It would be nice with kernel vector support in 6.6!

Quoted 1 line
arch/riscv/kernel/kernel_mode_vector.c +0 -0

[...]

arch/riscv/kernel/kernel_mode_vector.c void kernel_vector_begin(void) · line 117

Quoted 11 lines
arch/riscv/kernel/kernel_mode_vector.c +6 -1
@@ -70,11 +109,14 @@ void kernel_vector_begin(void)
 
 	riscv_v_vstate_save(&current->thread.vstate, task_pt_regs(current));
 
-	get_cpu_vector_context();
+	if (!preemptible() || !kernel_vector_preemptible()) {
+		get_cpu_vector_context();
+	} else {
+		if (riscv_v_start_kernel_context())
+			get_cpu_vector_context();
+	}

Wdyt about replacing this with:

        if (!preemptible() || !kernel_vector_preemptible() || riscv_v_start_kernel_context())
                get_cpu_vector_context();

Björn

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#16 lore raw
In reply to Andy Chiu
Quoted 20 lines · Andy Chiu

This series provides support for running Vector code in kernel mode. The implementation is based on the v12 series of the Vector series, but with some additions. First, we introduce a mechanism to defer restoring Vector context for userspace programs (patch 1). This is similar to arm64 and x86's approaches when dealing with extra userspace register context. And it is benefitial to both Vector in user and kernel-mode. Then, patch 2, 3 add the kernel-mode Vector patch from v12 with minor modifications. At the end of the series, patch 4, 5 add supports for making kernel-mode Vector code preemptible. We do this by adding kernel-mode Vector context, and keeping track of the frame where V context is last valid. We believe that enabling preemption of running V is a critical path for getting V more generally available in the kernel-mode. Besides, with status.VS, we can easily tell if saving/restoring V is required. This reduce the level of cost when running SIMD in kernel mode as compared to other arches. Other arches usually do not have a way to tell if extra context is dirty. Thus, if they also want to support running preemptible code with extra registers, then they must save/restore extra context at each context switch even if registers are not dirty.

Quoted 4 lines

The series is tested by loading a kernel module on a preemptive kernel. The module launches multiple kworkers which run Vector operations and verifies with scalar code. Also, the module provides userspace intefaces via fops to verify if we can run Vector code on syscall path.

Would it be contributed to kernel tools/testing/selftests/riscv/?

Quoted 47 lines

Updated patches: 1, 2, 3, 4, 5
New patches: -
Unchanged patches: -
Deleted patches: 6 (moved to 5)

Changelog v2:
- fix build issues
- Follow arm's way of starting kernel-mode simd code:
- add include/asm/simd.h and rename may_use_vector() ->

     may_use_simd()

- return void in kernel_vector_begin(), and BUG_ON if may_use_simd() fails
- Change naming scheme for functions/macros (Conor):
- remove KMV
- 's/rvv/vector/'
- 's/RISCV_ISA_V_PREEMPTIVE_KMV/RISCV_ISA_V_PREEMPTIVE/'
- 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE/'

Changes from the vector v12 series (for patch 2, 3):
- return a failure code when kernel_vector_begin() fails.
- Do not immediately restore user's V context.

Andy Chiu (3):
riscv: sched: defer restoring Vector context for user
riscv: vector: do not pass task_struct into

    riscv_v_vstate_{save,restore}()

riscv: vector: allow kernel-mode Vector with preemption

Greentime Hu (2):
riscv: Add support for kernel mode vector
riscv: Add vector extension XOR implementation

 arch/riscv/Kconfig                     |  10 ++
 arch/riscv/include/asm/entry-common.h  |  13 +++
 arch/riscv/include/asm/processor.h     |   2 +
 arch/riscv/include/asm/simd.h          |  52 +++++++++
 arch/riscv/include/asm/thread_info.h   |   6 +
 arch/riscv/include/asm/vector.h        |  50 +++++++--
 arch/riscv/include/asm/xor.h           |  82 ++++++++++++++
 arch/riscv/kernel/Makefile             |   1 +
 arch/riscv/kernel/asm-offsets.c        |   2 +
 arch/riscv/kernel/entry.S              |  45 ++++++++
 arch/riscv/kernel/kernel_mode_vector.c | 146 +++++++++++++++++++++++++
 arch/riscv/kernel/process.c            |  10 +-
 arch/riscv/kernel/ptrace.c             |   2 +-
 arch/riscv/kernel/signal.c             |   4 +-
Quoted 18 lines
 arch/riscv/kernel/vector.c             |   5 +-
 arch/riscv/lib/Makefile                |   1 +
 arch/riscv/lib/xor.S                   |  81 ++++++++++++++
 17 files changed, 495 insertions(+), 17 deletions(-)
 create mode 100644 arch/riscv/include/asm/simd.h
 create mode 100644 arch/riscv/include/asm/xor.h
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
 create mode 100644 arch/riscv/lib/xor.S
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv