Skip to contentSkip to conversation

riscv: support kernel-mode Vector

PATCH Superseded Andy Chiu 2 participants 16 messages Dec 21, 2023 – Dec 26, 2023

Subsystems (2)

v7 of 11 Superseded Latest is v11 →

lore
Share & tools
lore

Andy Chiu

author
#0 lore raw

v7 cover riscv: support kernel-mode Vector

This series provides support for running Vector code in kernel mode. Along with the suport, we add some Vector optimized routines. And provide a simple threshold to decide when to run the vectorized functions.

I am adding back preemptible Vector as they are ready for review now. Having them together makes it easier to be reviewed and tested. Meanwhile, as they are appended at the end, this series can be merged incrementally if we feel any part of {1~4, 5~6, 7~10} is mature enough.

This patch is tested on a QEMU with V and verified that booting, normal userspace operations all work as usual with thresholds set to 0. Also, we test by launching multiple kernel threads which continuously executes and verifies Vector operations in the background. The module that tests these operation is expected to be upstream later.

v6 of this series can be found at [1]

Patch summary:
- Updated patches: 1, 3, 6
- New patch: none
- Unchanged patch: 2, 4, 5, 7, 8, 9, 10
- Deleted patch: none

Changelog v7:
- Fix build fail for allmodconfig and test building the series with allmodconfig/allyesconfig

Changelog v6:
- Provide a more robust check on the use of non-preemptible Vector.
- Add Kconfigs to set threshold value at compile time. (Charlie)
- Add a patch to utilize kmem_cache_* for V context allocations.
- Re-write and add preemptible Vector.

Changelog v5:
- Rebase on top of riscv for-next (6.7-rc1)
Changelog v4:
- Use kernel_v_flags and helpers to track vector context.
- Prevent softirq from nesting V context for non-preempt V
- Add user copy and mem* routines

Changelog v3:
- Rebase on top of riscv for-next (6.6-rc1)
- Fix a build issue (Conor)
- Guard vstate_save, vstate_restore with {get,put}_cpu_vector_context.
- Save V context after disabling preemption. (Guo)
- Remove irqs_disabled() check from may_use_simd(). (Björn)
- Comment about nesting V context.

Changelog v2:
- fix build issues
- Follow arm's way of starting kernel-mode simd code:
- add include/asm/simd.h and rename may_use_vector() ->

     may_use_simd()

- return void in kernel_vector_begin(), and BUG_ON if may_use_simd() fails
- Change naming scheme for functions/macros (Conor):
- remove KMV
- 's/rvv/vector/'
- 's/RISCV_ISA_V_PREEMPTIVE_KMV/RISCV_ISA_V_PREEMPTIVE/'
- 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE/'

Andy Chiu (8):
riscv: vector: make Vector always available for softirq context
riscv: sched: defer restoring Vector context for user
riscv: lib: vectorize copy_to_user/copy_from_user
riscv: lib: add vectorized mem* routines
riscv: vector: do not pass task_struct into

    riscv_v_vstate_{save,restore}()

riscv: vector: use a mask to write vstate_ctrl
riscv: vector: use kmem_cache to manage vector context
riscv: vector: allow kernel-mode Vector with preemption

Greentime Hu (2):
riscv: Add support for kernel mode vector
riscv: Add vector extension XOR implementation

 arch/riscv/Kconfig                      |  46 +++++
 arch/riscv/include/asm/asm-prototypes.h |  14 ++
 arch/riscv/include/asm/entry-common.h   |  17 ++
 arch/riscv/include/asm/processor.h      |  39 ++++-
 arch/riscv/include/asm/simd.h           |  64 +++++++
 arch/riscv/include/asm/thread_info.h    |   2 +
 arch/riscv/include/asm/vector.h         | 103 ++++++++++--
 arch/riscv/include/asm/xor.h            |  68 ++++++++
 arch/riscv/kernel/Makefile              |   1 +
 arch/riscv/kernel/entry.S               |   8 +
 arch/riscv/kernel/kernel_mode_vector.c  | 212 ++++++++++++++++++++++++
 arch/riscv/kernel/process.c             |  14 +-
 arch/riscv/kernel/ptrace.c              |   7 +-
 arch/riscv/kernel/signal.c              |   7 +-
 arch/riscv/kernel/vector.c              |  50 +++++-
 arch/riscv/lib/Makefile                 |   6 +
 arch/riscv/lib/memcpy_vector.S          |  29 ++++
 arch/riscv/lib/memmove_vector.S         |  49 ++++++
 arch/riscv/lib/memset_vector.S          |  33 ++++
 arch/riscv/lib/riscv_v_helpers.c        |  64 +++++++
 arch/riscv/lib/uaccess.S                |  10 ++
 arch/riscv/lib/uaccess_vector.S         |  50 ++++++
 arch/riscv/lib/xor.S                    |  81 +++++++++
 23 files changed, 948 insertions(+), 26 deletions(-)
 create mode 100644 arch/riscv/include/asm/simd.h
 create mode 100644 arch/riscv/include/asm/xor.h
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
 create mode 100644 arch/riscv/lib/memcpy_vector.S
 create mode 100644 arch/riscv/lib/memmove_vector.S
 create mode 100644 arch/riscv/lib/memset_vector.S
 create mode 100644 arch/riscv/lib/riscv_v_helpers.c
 create mode 100644 arch/riscv/lib/uaccess_vector.S
 create mode 100644 arch/riscv/lib/xor.S
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
author
#1 lore raw

PATCH v7 1/10 riscv: Add support for kernel mode vector

From: Greentime Hu <[email protected]>

Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c

These are needed to wrap uses of vector in kernel mode.

---
Changelog v7:
- fix build fail for allmodconfig
Changelog v6:
- Use 8 bits to track non-preemptible vector context to provide better WARN coverage.
Changelog v4:
- Use kernel_v_flags and helpers to track vector context.
Changelog v3:
- Reorder patch 1 to patch 3 to make use of {get,put}_cpu_vector_context later.
- Export {get,put}_cpu_vector_context.
- Save V context after disabling preemption. (Guo)
- Fix a build fail. (Conor)
- Remove irqs_disabled() check as it is not needed, fix styling. (Björn)
Changelog v2:
- 's/kernel_rvv/kernel_vector' and return void in kernel_vector_begin (Conor)
- export may_use_simd to include/asm/simd.h

---
 arch/riscv/include/asm/processor.h     | 17 ++++-
 arch/riscv/include/asm/simd.h          | 44 ++++++++++++
 arch/riscv/include/asm/vector.h        | 21 ++++++
 arch/riscv/kernel/Makefile             |  1 +
 arch/riscv/kernel/kernel_mode_vector.c | 95 ++++++++++++++++++++++++++
 arch/riscv/kernel/process.c            |  2 +-
 6 files changed, 178 insertions(+), 2 deletions(-)
 create mode 100644 arch/riscv/include/asm/simd.h
 create mode 100644 arch/riscv/kernel/kernel_mode_vector.c
arch/riscv/include/asm/processor.h +16 -1
index f19f861cda54..15781e2232e0 100644
@@ -73,6 +73,20 @@
 struct task_struct;
 struct pt_regs;
 
+/*
+ * We use a flag to track in-kernel Vector context. Currently the flag has the
+ * following meaning:
+ *
+ *  - bit 0-7 indicates whether the in-kernel Vector context is active. The
+ *    activation of this state disables the preemption. On a non-RT kernel, it
+ *    also disable bh. Currently only 0 and 1 are valid value for this field.
+ *    Other values are reserved for future uses.
+ */
+
+#define RISCV_KERNEL_MODE_V_MASK	0xff
+
+#define RISCV_KERNEL_MODE_V	0x1
+
 /* CPU-specific state of a task */
 struct thread_struct {
 	/* Callee-saved registers */
@@ -81,7 +95,8 @@ struct thread_struct {
 	unsigned long s[12];	/* s[0]: frame pointer */
 	struct __riscv_d_ext_state fstate;
 	unsigned long bad_cause;
-	unsigned long vstate_ctrl;
+	u32 riscv_v_flags;
+	u32 vstate_ctrl;
 	struct __riscv_v_ext_state vstate;
 	unsigned long align_ctl;
 };
arch/riscv/include/asm/simd.h new file +44 -0

44 lines not on this page · raw message

arch/riscv/include/asm/vector.h +21 -0
index 87aaef656257..6254830c0668 100644
@@ -22,6 +22,27 @@
 extern unsigned long riscv_v_vsize;
 int riscv_v_setup_vsize(void);
 bool riscv_v_first_use_handler(struct pt_regs *regs);
+void kernel_vector_begin(void);
+void kernel_vector_end(void);
+void get_cpu_vector_context(void);
+void put_cpu_vector_context(void);
+
+static inline void riscv_v_ctx_cnt_add(u32 offset)
+{
+	current->thread.riscv_v_flags += offset;
+	barrier();
+}
+
+static inline void riscv_v_ctx_cnt_sub(u32 offset)
+{
+	barrier();
+	current->thread.riscv_v_flags -= offset;
+}
+
+static inline u32 riscv_v_ctx_cnt(void)
+{
+	return READ_ONCE(current->thread.riscv_v_flags);
+}
 
 static __always_inline bool has_vector(void)
 {
arch/riscv/kernel/Makefile +1 -0
index fee22a3d1b53..8c58595696b3 100644
@@ -63,6 +63,7 @@ obj-$(CONFIG_MMU) += vdso.o vdso/
 obj-$(CONFIG_RISCV_MISALIGNED)	+= traps_misaligned.o
 obj-$(CONFIG_FPU)		+= fpu.o
 obj-$(CONFIG_RISCV_ISA_V)	+= vector.o
+obj-$(CONFIG_RISCV_ISA_V)	+= kernel_mode_vector.o
 obj-$(CONFIG_SMP)		+= smpboot.o
 obj-$(CONFIG_SMP)		+= smp.o
 obj-$(CONFIG_SMP)		+= cpu_ops.o
arch/riscv/kernel/kernel_mode_vector.c new file +95 -0

95 lines not on this page · raw message

arch/riscv/kernel/process.c +1 -1
index 4f21d970a129..5c4dcf518684 100644
@@ -187,7 +187,6 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
 	*dst = *src;
 	/* clear entire V context, including datap for a new task */
 	memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
-
 	return 0;
 }
 
@@ -221,6 +220,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 		childregs->a0 = 0; /* Return value of fork() */
 		p->thread.s[0] = 0;
 	}
+	p->thread.riscv_v_flags = 0;
 	p->thread.ra = (unsigned long)ret_from_fork;
 	p->thread.sp = (unsigned long)childregs; /* kernel sp */
 	return 0;
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#2 lore raw

Re: [v7, 01/10] riscv: Add support for kernel mode vector

arch/riscv/include/asm/processor.h line 88

Quoted 13 lines · Andy Chiu
arch/riscv/include/asm/processor.h +13 -0
+/*
+ * We use a flag to track in-kernel Vector context. Currently the flag has the
+ * following meaning:
+ *
+ *  - bit 0-7 indicates whether the in-kernel Vector context is active. The
+ *    activation of this state disables the preemption. On a non-RT kernel, it
+ *    also disable bh. Currently only 0 and 1 are valid value for this field.
+ *    Other values are reserved for future uses.
+ */
+
+#define RISCV_KERNEL_MODE_V_MASK	0xff
+
+#define RISCV_KERNEL_MODE_V	0x1

Is there a reason this isn't just a single bit flag?

arch/riscv/include/asm/vector.h line 40

Quoted 11 lines
arch/riscv/include/asm/vector.h +11 -0
+static inline void riscv_v_ctx_cnt_add(u32 offset)
+{
+	current->thread.riscv_v_flags += offset;
+	barrier();
+}
+
+static inline void riscv_v_ctx_cnt_sub(u32 offset)
+{
+	barrier();
+	current->thread.riscv_v_flags -= offset;
+}

What is the purpose of the barriers above?

arch/riscv/include/asm/vector.h line 45

Quoted 4 lines
arch/riscv/include/asm/vector.h +4 -0
+static inline u32 riscv_v_ctx_cnt(void)
+{
+	return READ_ONCE(current->thread.riscv_v_flags);
+}

What is the purpose of using READ_ONCE() here?

arch/riscv/kernel/process.c int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) · line 191

Quoted 11 lines
arch/riscv/kernel/process.c +0 -1
index 4f21d970a129..5c4dcf518684 100644
@@ -187,7 +187,6 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
 	*dst = *src;
 	/* clear entire V context, including datap for a new task */
 	memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
-
 	return 0;
 }

Unnecessary whitespace change.

Otherwise this patch looks good, thanks!

- Eric

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#3 lore raw

Re: [v7, 01/10] riscv: Add support for kernel mode vector

Quoted 17 lines · Eric Biggers

arch/riscv/include/asm/processor.h line 88

Quoted 13 lines · Andy Chiu
arch/riscv/include/asm/processor.h +13 -0
+/*
+ * We use a flag to track in-kernel Vector context. Currently the flag has the
+ * following meaning:
+ *
+ *  - bit 0-7 indicates whether the in-kernel Vector context is active. The
+ *    activation of this state disables the preemption. On a non-RT kernel, it
+ *    also disable bh. Currently only 0 and 1 are valid value for this field.
+ *    Other values are reserved for future uses.
+ */
+
+#define RISCV_KERNEL_MODE_V_MASK     0xff
+
+#define RISCV_KERNEL_MODE_V  0x1

Is there a reason this isn't just a single bit flag?

I have not yet drawn a conclusion on this. But I am thinking if it would be useful to allow calling kernel_vector_begin multiple times on a call chain. Then these extra bits would be useful if we were to allow and implement it. For example, the use case would be like

kernel_vector_begin();
memset(); //chained to another kernel_vector_begin and vectorized memset do_things_with_vector();
kernel_vector_end();

Quoted 15 lines

arch/riscv/include/asm/vector.h line 40

Quoted 12 lines
arch/riscv/include/asm/vector.h +11 -0
+static inline void riscv_v_ctx_cnt_add(u32 offset)
+{
+     current->thread.riscv_v_flags += offset;
+     barrier();
+}
+
+static inline void riscv_v_ctx_cnt_sub(u32 offset)
+{
+     barrier();
+     current->thread.riscv_v_flags -= offset;
+}

What is the purpose of the barriers above?

Quoted 6 lines

arch/riscv/include/asm/vector.h line 45

Quoted 4 lines
arch/riscv/include/asm/vector.h +4 -0
+static inline u32 riscv_v_ctx_cnt(void)
+{
+     return READ_ONCE(current->thread.riscv_v_flags);
+}

What is the purpose of using READ_ONCE() here?

These codes provide compiler barriers, e.g. to prevent riscv_v_flag counting slips into Vector operations. Currently. it should be fine for non-preemptible Vector as riscv_v_ctx_cnt_add()/sub() are guarded with preempt_disable()/local_bh_disable(). However, if preempt_v or the above use-case is concerned, then these compiler barriers are needed and should not be mixed with Vector operations afterward. Or, it would confuse the context tracking when traps take place.

riscv_v_ctx_cnt_add()
do things with Vector, maybe inlined (vstate_save) riscv_v_ctx_cnt_sub()

Quoted 14 lines

arch/riscv/kernel/process.c int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) · line 191

Quoted 12 lines
arch/riscv/kernel/process.c +0 -1
index 4f21d970a129..5c4dcf518684 100644
@@ -187,7 +187,6 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
      *dst = *src;
      /* clear entire V context, including datap for a new task */
      memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
-
      return 0;
 }

Unnecessary whitespace change.

This will be fixed in v8, thanks!

Quoted 4 lines

Otherwise this patch looks good, thanks!

- Eric

Thanks,
Andy

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

#4 lore raw

Re: [v7, 01/10] riscv: Add support for kernel mode vector

Quoted 24 lines · Andy Chiu
Quoted 17 lines · Eric Biggers

arch/riscv/include/asm/processor.h line 88

Quoted 13 lines · Andy Chiu
arch/riscv/include/asm/processor.h +13 -0
+/*
+ * We use a flag to track in-kernel Vector context. Currently the flag has the
+ * following meaning:
+ *
+ *  - bit 0-7 indicates whether the in-kernel Vector context is active. The
+ *    activation of this state disables the preemption. On a non-RT kernel, it
+ *    also disable bh. Currently only 0 and 1 are valid value for this field.
+ *    Other values are reserved for future uses.
+ */
+
+#define RISCV_KERNEL_MODE_V_MASK     0xff
+
+#define RISCV_KERNEL_MODE_V  0x1

Is there a reason this isn't just a single bit flag?

I have not yet drawn a conclusion on this. But I am thinking if it would be useful to allow calling kernel_vector_begin multiple times on a call chain. Then these extra bits would be useful if we were to allow and implement it. For example, the use case would be like

Quoted 4 lines

kernel_vector_begin();
memset(); //chained to another kernel_vector_begin and vectorized memset do_things_with_vector();
kernel_vector_end();

Maybe it should just be a single bit for now, and it can be changed later if the more complex version actually turns out to be needed?

FWIW, the existing architectures don't allow reentrant enabling of SIMD.

Quoted 29 lines
Quoted 14 lines

arch/riscv/include/asm/vector.h line 40

Quoted 11 lines
arch/riscv/include/asm/vector.h +11 -0
+static inline void riscv_v_ctx_cnt_add(u32 offset)
+{
+     current->thread.riscv_v_flags += offset;
+     barrier();
+}
+
+static inline void riscv_v_ctx_cnt_sub(u32 offset)
+{
+     barrier();
+     current->thread.riscv_v_flags -= offset;
+}

What is the purpose of the barriers above?

Quoted 6 lines

arch/riscv/include/asm/vector.h line 45

Quoted 4 lines
arch/riscv/include/asm/vector.h +4 -0
+static inline u32 riscv_v_ctx_cnt(void)
+{
+     return READ_ONCE(current->thread.riscv_v_flags);
+}

What is the purpose of using READ_ONCE() here?

These codes provide compiler barriers, e.g. to prevent riscv_v_flag counting slips into Vector operations. Currently. it should be fine for non-preemptible Vector as riscv_v_ctx_cnt_add()/sub() are guarded with preempt_disable()/local_bh_disable(). However, if preempt_v or the above use-case is concerned, then these compiler barriers are needed and should not be mixed with Vector operations afterward. Or, it would confuse the context tracking when traps take place.

Quoted 3 lines

riscv_v_ctx_cnt_add()
do things with Vector, maybe inlined (vstate_save) riscv_v_ctx_cnt_sub()

Can you leave comments in the code explaining this?

Also, if these barriers aren't needed until preemptible vector support, they probably should be part of that patch, not this one.

- Eric

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#5 lore raw

Re: [v7, 01/10] riscv: Add support for kernel mode vector

Quoted 26 lines · Eric Biggers
Quoted 24 lines · Andy Chiu
Quoted 17 lines · Eric Biggers

arch/riscv/include/asm/processor.h line 88

Quoted 13 lines · Andy Chiu
arch/riscv/include/asm/processor.h +13 -0
+/*
+ * We use a flag to track in-kernel Vector context. Currently the flag has the
+ * following meaning:
+ *
+ *  - bit 0-7 indicates whether the in-kernel Vector context is active. The
+ *    activation of this state disables the preemption. On a non-RT kernel, it
+ *    also disable bh. Currently only 0 and 1 are valid value for this field.
+ *    Other values are reserved for future uses.
+ */
+
+#define RISCV_KERNEL_MODE_V_MASK     0xff
+
+#define RISCV_KERNEL_MODE_V  0x1

Is there a reason this isn't just a single bit flag?

I have not yet drawn a conclusion on this. But I am thinking if it would be useful to allow calling kernel_vector_begin multiple times on a call chain. Then these extra bits would be useful if we were to allow and implement it. For example, the use case would be like

Quoted 7 lines
Quoted 4 lines

kernel_vector_begin();
memset(); //chained to another kernel_vector_begin and vectorized memset do_things_with_vector();
kernel_vector_end();

Maybe it should just be a single bit for now, and it can be changed later if the more complex version actually turns out to be needed?

Having more bits allows us to check against overflow [1]. For example, we can detect if people accidentally call riscv_v_ctx_cnt_add() even number of times with a few more bits. Another way to do this is to use test_and_* operation on riscv_v_flags, but then we will have to maintain two kinds of accessors to the flag. E.g. the preempt_v one will still have to do counter add/sub operations on the flag.

Quoted 32 lines

FWIW, the existing architectures don't allow reentrant enabling of SIMD.

Quoted 29 lines
Quoted 14 lines

arch/riscv/include/asm/vector.h line 40

Quoted 11 lines
arch/riscv/include/asm/vector.h +11 -0
+static inline void riscv_v_ctx_cnt_add(u32 offset)
+{
+     current->thread.riscv_v_flags += offset;
+     barrier();
+}
+
+static inline void riscv_v_ctx_cnt_sub(u32 offset)
+{
+     barrier();
+     current->thread.riscv_v_flags -= offset;
+}

What is the purpose of the barriers above?

Quoted 6 lines

arch/riscv/include/asm/vector.h line 45

Quoted 4 lines
arch/riscv/include/asm/vector.h +4 -0
+static inline u32 riscv_v_ctx_cnt(void)
+{
+     return READ_ONCE(current->thread.riscv_v_flags);
+}

What is the purpose of using READ_ONCE() here?

These codes provide compiler barriers, e.g. to prevent riscv_v_flag counting slips into Vector operations. Currently. it should be fine for non-preemptible Vector as riscv_v_ctx_cnt_add()/sub() are guarded with preempt_disable()/local_bh_disable(). However, if preempt_v or the above use-case is concerned, then these compiler barriers are needed and should not be mixed with Vector operations afterward. Or, it would confuse the context tracking when traps take place.

Quoted 8 lines
Quoted 3 lines

riscv_v_ctx_cnt_add()
do things with Vector, maybe inlined (vstate_save) riscv_v_ctx_cnt_sub()

Can you leave comments in the code explaining this?

Also, if these barriers aren't needed until preemptible vector support, they probably should be part of that patch, not this one.

Yes, I think these barriers are not needed until then. I will move them to patch 10

Quoted 2 lines

- Eric

- [1]: https://lore.kernel.org/all/ZXvxIuZwCQ8zeXhr@ghost/

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#6 lore raw

PATCH v7 2/10 riscv: vector: make Vector always available for softirq context

In reply to Andy Chiu

By disabling bottom halves in active kerne-mode Vector, softirq will not be able to nest on top of any kernel-mode Vector.

After this patch, Vector context cannot start with irqs disabled. Otherwise local_bh_enable() may run in a wrong context.

Disabling bh is not enough for RT-kernel to prevent preeemption. So we must disable preemption, which also implies disabling bh on RT.

Related-to: commit 696207d4258b ("arm64/sve: Make kernel FPU protection RT friendly")
Related-to: commit 66c3ec5a7120 ("arm64: neon: Forbid when irqs are disabled")

---
Changelog v4:
- new patch since v4

---
 arch/riscv/include/asm/simd.h          |  6 +++++-
 arch/riscv/kernel/kernel_mode_vector.c | 10 ++++++++--
 2 files changed, 13 insertions(+), 3 deletions(-)
arch/riscv/include/asm/simd.h +5 -1
index 3b603e47c5d8..7df5a976a80a 100644
@@ -28,8 +28,12 @@ static __must_check inline bool may_use_simd(void)
 	/*
 	 * RISCV_KERNEL_MODE_V is only set while preemption is disabled,
 	 * and is clear whenever preemption is enabled.
+	 *
+	 * Kernel-mode Vector temperarily disables bh. So we must not return
+	 * true on irq_disabled(). Otherwise we would fail the lockdep check
+	 * calling local_bh_enable()
 	 */
-	return !in_hardirq() && !in_nmi() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK);
+	return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK);
 }
 
 #else /* ! CONFIG_RISCV_ISA_V */
arch/riscv/kernel/kernel_mode_vector.c +8 -2
index 105147c7d2da..db0cf06f2abf 100644
@@ -23,7 +23,10 @@
  */
 void get_cpu_vector_context(void)
 {
-	preempt_disable();
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+		local_bh_disable();
+	else
+		preempt_disable();
 
 	WARN_ON((riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK) != 0);
 	riscv_v_ctx_cnt_add(RISCV_KERNEL_MODE_V);
@@ -41,7 +44,10 @@ void put_cpu_vector_context(void)
 	WARN_ON((riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK) != RISCV_KERNEL_MODE_V);
 	riscv_v_ctx_cnt_sub(RISCV_KERNEL_MODE_V);
 
-	preempt_enable();
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+		local_bh_enable();
+	else
+		preempt_enable();
 }
 
 /*
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#7 lore raw

Re: [v7, 02/10] riscv: vector: make Vector always available for softirq context

Quoted 2 lines · Andy Chiu

By disabling bottom halves in active kerne-mode Vector, softirq will not be able to nest on top of any kernel-mode Vector.

kerne => kernel

Quoted 6 lines

After this patch, Vector context cannot start with irqs disabled. Otherwise local_bh_enable() may run in a wrong context.

Disabling bh is not enough for RT-kernel to prevent preeemption. So we must disable preemption, which also implies disabling bh on RT.

The body of the commit message should explicitly mention the goal of the patch (support kernel vector in softirq context) and why this is a desirable goal (e.g., make it so that some of the crypto algorithms won't need fallbacks).

arch/riscv/include/asm/simd.h static __must_check inline bool may_use_simd(void) · line 34

Quoted 12 lines
arch/riscv/include/asm/simd.h +4 -0
index 3b603e47c5d8..7df5a976a80a 100644
@@ -28,8 +28,12 @@ static __must_check inline bool may_use_simd(void)
 	/*
 	 * RISCV_KERNEL_MODE_V is only set while preemption is disabled,
 	 * and is clear whenever preemption is enabled.
+	 *
+	 * Kernel-mode Vector temperarily disables bh. So we must not return
+	 * true on irq_disabled(). Otherwise we would fail the lockdep check
+	 * calling local_bh_enable()

temperarily => temporarily

arch/riscv/kernel/kernel_mode_vector.c line 29

Quoted 13 lines
arch/riscv/kernel/kernel_mode_vector.c +4 -1
index 105147c7d2da..db0cf06f2abf 100644
@@ -23,7 +23,10 @@
  */
 void get_cpu_vector_context(void)
 {
-	preempt_disable();
+	if (!IS_ENABLED(CONFIG_PREEMPT_RT))
+		local_bh_disable();
+	else
+		preempt_disable();

Maybe leave a comment here explaining why softirqs are being disabled?

Otherwise this patch looks good, thanks!

- Eric

_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv

author
#8 lore raw

PATCH v7 3/10 riscv: Add vector extension XOR implementation

In reply to Andy Chiu

From: Greentime Hu <[email protected]>

This patch adds support for vector optimized XOR and it is tested in qemu.

---
Changelog v7:
- fix build warning message and use proper entry/exit macro for assembly. Drop Conor's A-b
Changelog v2:
- 's/rvv/vector/' (Conor)

---
 arch/riscv/include/asm/asm-prototypes.h | 14 +++++
 arch/riscv/include/asm/xor.h            | 68 +++++++++++++++++++++
 arch/riscv/lib/Makefile                 |  1 +
 arch/riscv/lib/xor.S                    | 81 +++++++++++++++++++++++++
 4 files changed, 164 insertions(+)
 create mode 100644 arch/riscv/include/asm/xor.h
 create mode 100644 arch/riscv/lib/xor.S
arch/riscv/include/asm/asm-prototypes.h +14 -0
index 36b955c762ba..b34b68a99855 100644
@@ -9,6 +9,20 @@ long long __lshrti3(long long a, int b);
 long long __ashrti3(long long a, int b);
 long long __ashlti3(long long a, int b);
 
+void xor_regs_2_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2);
+void xor_regs_3_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2,
+		 const unsigned long *__restrict p3);
+void xor_regs_4_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2,
+		 const unsigned long *__restrict p3,
+		 const unsigned long *__restrict p4);
+void xor_regs_5_(unsigned long bytes, unsigned long *__restrict p1,
+		 const unsigned long *__restrict p2,
+		 const unsigned long *__restrict p3,
+		 const unsigned long *__restrict p4,
+		 const unsigned long *__restrict p5);
 
 #define DECLARE_DO_ERROR_INFO(name)	asmlinkage void name(struct pt_regs *regs)
 
arch/riscv/include/asm/xor.h new file +68 -0

68 lines not on this page · raw message

arch/riscv/lib/Makefile +1 -0
index 26cb2502ecf8..494f9cd1a00c 100644
@@ -11,3 +11,4 @@ lib-$(CONFIG_64BIT)	+= tishift.o
 lib-$(CONFIG_RISCV_ISA_ZICBOZ)	+= clear_page.o
 
 obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
+lib-$(CONFIG_RISCV_ISA_V)	+= xor.o
arch/riscv/lib/xor.S new file +81 -0

81 lines not on this page · raw message

Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
author
#9 lore raw

PATCH v7 4/10 riscv: sched: defer restoring Vector context for user

In reply to Andy Chiu

User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register.

Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times.

The cost of this is that we must disable preemption and mark vector as busy during vstate_{save,restore}. Because then the V context will not get restored back immediately when a trap-causing context switch happens in the middle of vstate_{save,restore}.

---
Changelog v4:
- fix typos and re-add Conor's A-b.
Changelog v3:
- Guard {get,put}_cpu_vector_context between vstate_* operation and explain it in the commit msg.
- Drop R-b from Björn and A-b from Conor.
Changelog v2:
- rename and add comment for the new thread flag (Conor)

---
 arch/riscv/include/asm/entry-common.h  | 17 +++++++++++++++++
 arch/riscv/include/asm/thread_info.h   |  2 ++
 arch/riscv/include/asm/vector.h        | 11 ++++++++++-
 arch/riscv/kernel/kernel_mode_vector.c |  2 +-
 arch/riscv/kernel/process.c            |  2 ++
 arch/riscv/kernel/ptrace.c             |  5 ++++-
 arch/riscv/kernel/signal.c             |  5 ++++-
 arch/riscv/kernel/vector.c             |  2 +-
 8 files changed, 41 insertions(+), 5 deletions(-)
arch/riscv/include/asm/entry-common.h +17 -0
index 7ab5e34318c8..6361a8488642 100644
@@ -4,6 +4,23 @@
 #define _ASM_RISCV_ENTRY_COMMON_H
 
 #include <asm/stacktrace.h>
+#include <asm/thread_info.h>
+#include <asm/vector.h>
+
+static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs,
+						  unsigned long ti_work)
+{
+	if (ti_work & _TIF_RISCV_V_DEFER_RESTORE) {
+		clear_thread_flag(TIF_RISCV_V_DEFER_RESTORE);
+		/*
+		 * We are already called with irq disabled, so go without
+		 * keeping track of vector_context_busy.
+		 */
+		riscv_v_vstate_restore(current, regs);
+	}
+}
+
+#define arch_exit_to_user_mode_prepare arch_exit_to_user_mode_prepare
 
 void handle_page_fault(struct pt_regs *regs);
 void handle_break(struct pt_regs *regs);
arch/riscv/include/asm/thread_info.h +2 -0
index 574779900bfb..1047a97ddbc8 100644
@@ -103,12 +103,14 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
 #define TIF_NOTIFY_SIGNAL	9	/* signal notifications exist */
 #define TIF_UPROBE		10	/* uprobe breakpoint or singlestep */
 #define TIF_32BIT		11	/* compat-mode 32bit process */
+#define TIF_RISCV_V_DEFER_RESTORE	12 /* restore Vector before returing to user */
 
 #define _TIF_NOTIFY_RESUME	(1 << TIF_NOTIFY_RESUME)
 #define _TIF_SIGPENDING		(1 << TIF_SIGPENDING)
 #define _TIF_NEED_RESCHED	(1 << TIF_NEED_RESCHED)
 #define _TIF_NOTIFY_SIGNAL	(1 << TIF_NOTIFY_SIGNAL)
 #define _TIF_UPROBE		(1 << TIF_UPROBE)
+#define _TIF_RISCV_V_DEFER_RESTORE	(1 << TIF_RISCV_V_DEFER_RESTORE)
 
 #define _TIF_WORK_MASK \
 	(_TIF_NOTIFY_RESUME | _TIF_SIGPENDING | _TIF_NEED_RESCHED | \
arch/riscv/include/asm/vector.h +10 -1
index 6254830c0668..e706613aae2c 100644
@@ -205,6 +205,15 @@ static inline void riscv_v_vstate_restore(struct task_struct *task,
 	}
 }
 
+static inline void riscv_v_vstate_set_restore(struct task_struct *task,
+					      struct pt_regs *regs)
+{
+	if ((regs->status & SR_VS) != SR_VS_OFF) {
+		set_tsk_thread_flag(task, TIF_RISCV_V_DEFER_RESTORE);
+		riscv_v_vstate_on(regs);
+	}
+}
+
 static inline void __switch_to_vector(struct task_struct *prev,
 				      struct task_struct *next)
 {
@@ -212,7 +221,7 @@ static inline void __switch_to_vector(struct task_struct *prev,
 
 	regs = task_pt_regs(prev);
 	riscv_v_vstate_save(prev, regs);
-	riscv_v_vstate_restore(next, task_pt_regs(next));
+	riscv_v_vstate_set_restore(next, task_pt_regs(next));
 }
 
 void riscv_v_vstate_ctrl_init(struct task_struct *tsk);
arch/riscv/kernel/kernel_mode_vector.c +1 -1
index db0cf06f2abf..3f1d67109e5a 100644
@@ -92,7 +92,7 @@ void kernel_vector_end(void)
 	if (WARN_ON(!has_vector()))
 		return;
 
-	riscv_v_vstate_restore(current, task_pt_regs(current));
+	riscv_v_vstate_set_restore(current, task_pt_regs(current));
 
 	riscv_v_disable();
 
arch/riscv/kernel/process.c +2 -0
index 5c4dcf518684..58127b1c6c71 100644
@@ -171,6 +171,7 @@ void flush_thread(void)
 	riscv_v_vstate_off(task_pt_regs(current));
 	kfree(current->thread.vstate.datap);
 	memset(&current->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
+	clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE);
 #endif
 }
 
@@ -187,6 +188,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
 	*dst = *src;
 	/* clear entire V context, including datap for a new task */
 	memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
+	clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE);
 	return 0;
 }
 
arch/riscv/kernel/ptrace.c +4 -1
index 2afe460de16a..7b93bcbdf9fa 100644
@@ -99,8 +99,11 @@ static int riscv_vr_get(struct task_struct *target,
 	 * Ensure the vector registers have been saved to the memory before
 	 * copying them to membuf.
 	 */
-	if (target == current)
+	if (target == current) {
+		get_cpu_vector_context();
 		riscv_v_vstate_save(current, task_pt_regs(current));
+		put_cpu_vector_context();
+	}
 
 	ptrace_vstate.vstart = vstate->vstart;
 	ptrace_vstate.vl = vstate->vl;
arch/riscv/kernel/signal.c +4 -1
index 88b6220b2608..aca4a12c8416 100644
@@ -86,7 +86,10 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec)
 	/* datap is designed to be 16 byte aligned for better performance */
 	WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16)));
 
+	get_cpu_vector_context();
 	riscv_v_vstate_save(current, regs);
+	put_cpu_vector_context();
+
 	/* Copy everything of vstate but datap. */
 	err = __copy_to_user(&state->v_state, &current->thread.vstate,
 			     offsetof(struct __riscv_v_ext_state, datap));
@@ -134,7 +137,7 @@ static long __restore_v_state(struct pt_regs *regs, void __user *sc_vec)
 	if (unlikely(err))
 		return err;
 
-	riscv_v_vstate_restore(current, regs);
+	riscv_v_vstate_set_restore(current, regs);
 
 	return err;
 }
arch/riscv/kernel/vector.c +1 -1
index 578b6292487e..66e8c6ab09d2 100644
@@ -167,7 +167,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs)
 		return true;
 	}
 	riscv_v_vstate_on(regs);
-	riscv_v_vstate_restore(current, regs);
+	riscv_v_vstate_set_restore(current, regs);
 	return true;
 }
 
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
author
#10 lore raw

PATCH v7 5/10 riscv: lib: vectorize copy_to_user/copy_from_user

In reply to Andy Chiu

This patch utilizes Vector to perform copy_to_user/copy_from_user. If Vector is available and the size of copy is large enough for Vector to perform better than scalar, then direct the kernel to do Vector copies for userspace. Though the best programming practice for users is to reduce the copy, this provides a faster variant when copies are inevitable.

The optimal size for using Vector, copy_to_user_thres, is only a heuristic for now. We can add DT parsing if people feel the need of customizing it.

The exception fixup code of the __asm_vector_usercopy must fallback to the scalar one because accessing user pages might fault, and must be sleepable. Current kernel-mode Vector does not allow tasks to be preemptible, so we must disactivate Vector and perform a scalar fallback in such case.

The original implementation of Vector operations comes from
https://github.com/sifive/sifive-libc, which we agree to contribute to Linux kernel.

---
Changelog v6:
- Add a kconfig entry to configure threshold values (Charlie)
- Refine assembly code (Charlie)
Changelog v4:
- new patch since v4

---
 arch/riscv/Kconfig               |  8 +++++
 arch/riscv/lib/Makefile          |  2 ++
 arch/riscv/lib/riscv_v_helpers.c | 38 ++++++++++++++++++++++++
 arch/riscv/lib/uaccess.S         | 10 +++++++
 arch/riscv/lib/uaccess_vector.S  | 50 ++++++++++++++++++++++++++++++++
 5 files changed, 108 insertions(+)
 create mode 100644 arch/riscv/lib/riscv_v_helpers.c
 create mode 100644 arch/riscv/lib/uaccess_vector.S
arch/riscv/Kconfig +8 -0
index 95a2a06acc6a..3c5ba05e8a2d 100644
@@ -525,6 +525,14 @@ config RISCV_ISA_V_DEFAULT_ENABLE
 
 	  If you don't know what to do here, say Y.
 
+config RISCV_ISA_V_UCOPY_THRESHOLD
+	int "Threshold size for vectorized user copies"
+	depends on RISCV_ISA_V
+	default 768
+	help
+	  Prefer using vectorized copy_to_user()/copy_from_user() when the
+	  workload size exceeds this value.
+
 config TOOLCHAIN_HAS_ZBB
 	bool
 	default y
arch/riscv/lib/Makefile +2 -0
index 494f9cd1a00c..1fe8d797e0f2 100644
@@ -12,3 +12,5 @@ lib-$(CONFIG_RISCV_ISA_ZICBOZ)	+= clear_page.o
 
 obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
 lib-$(CONFIG_RISCV_ISA_V)	+= xor.o
+lib-$(CONFIG_RISCV_ISA_V)	+= riscv_v_helpers.o
+lib-$(CONFIG_RISCV_ISA_V)	+= uaccess_vector.o
arch/riscv/lib/riscv_v_helpers.c new file +38 -0
new file mode 100644
index 000000000000..139e5de1b793
@@ -0,0 +1,38 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2023 SiFive
+ * Author: Andy Chiu <[email protected]>
+ */
+#include <linux/linkage.h>
+#include <asm/asm.h>
+
+#include <asm/vector.h>
+#include <asm/simd.h>
+
+size_t riscv_v_usercopy_threshold = CONFIG_RISCV_ISA_V_UCOPY_THRESHOLD;
+int __asm_vector_usercopy(void *dst, void *src, size_t n);
+int fallback_scalar_usercopy(void *dst, void *src, size_t n);
+asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n)
+{
+	size_t remain, copied;
+
+	/* skip has_vector() check because it has been done by the asm  */
+	if (!may_use_simd())
+		goto fallback;
+
+	kernel_vector_begin();
+	remain = __asm_vector_usercopy(dst, src, n);
+	kernel_vector_end();
+
+	if (remain) {
+		copied = n - remain;
+		dst += copied;
+		src += copied;
+		goto fallback;
+	}
+
+	return remain;
+
+fallback:
+	return fallback_scalar_usercopy(dst, src, n);
+}
arch/riscv/lib/uaccess.S +10 -0
index 3ab438f30d13..a1e4a3c42925 100644
@@ -3,6 +3,8 @@
 #include <asm/asm.h>
 #include <asm/asm-extable.h>
 #include <asm/csr.h>
+#include <asm/hwcap.h>
+#include <asm/alternative-macros.h>
 
 	.macro fixup op reg addr lbl
 100:
@@ -11,6 +13,13 @@
 	.endm
 
 SYM_FUNC_START(__asm_copy_to_user)
+#ifdef CONFIG_RISCV_ISA_V
+	ALTERNATIVE("j fallback_scalar_usercopy", "nop", 0, RISCV_ISA_EXT_v, CONFIG_RISCV_ISA_V)
+	REG_L	t0, riscv_v_usercopy_threshold
+	bltu	a2, t0, fallback_scalar_usercopy
+	tail enter_vector_usercopy
+#endif
+SYM_FUNC_START(fallback_scalar_usercopy)
 
 	/* Enable access to user memory */
 	li t6, SR_SUM
@@ -181,6 +190,7 @@ SYM_FUNC_START(__asm_copy_to_user)
 	sub a0, t5, a0
 	ret
 SYM_FUNC_END(__asm_copy_to_user)
+SYM_FUNC_END(fallback_scalar_usercopy)
 EXPORT_SYMBOL(__asm_copy_to_user)
 SYM_FUNC_ALIAS(__asm_copy_from_user, __asm_copy_to_user)
 EXPORT_SYMBOL(__asm_copy_from_user)
arch/riscv/lib/uaccess_vector.S new file +50 -0

50 lines not on this page · raw message

Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
author
#11 lore raw

PATCH v7 6/10 riscv: lib: add vectorized mem* routines

In reply to Andy Chiu

Provide vectorized memcpy/memset/memmove to accelerate common memory operations. Also, group them into V_OPT_TEMPLATE3 macro because their setup/tear-down and fallback logics are the same.

The optimal size for the kernel to preference Vector over scalar, riscv_v_mem*_threshold, is only a heuristic for now. We can add DT parsing if people feel the need of customizing it.

The original implementation of Vector operations comes from
https://github.com/sifive/sifive-libc, which we agree to contribute to Linux kernel.

---
Changelog v7:
- add __NO_FORTIFY to prevent conflicting function declaration with macro for mem* functions.
Changelog v6:
- provide kconfig to set threshold for vectorized functions (Charlie)
- rename *thres to *threshold (Charlie)
Changelog v4:
- new patch since v4

---
 arch/riscv/Kconfig               | 24 ++++++++++++++++
 arch/riscv/lib/Makefile          |  3 ++
 arch/riscv/lib/memcpy_vector.S   | 29 +++++++++++++++++++
 arch/riscv/lib/memmove_vector.S  | 49 ++++++++++++++++++++++++++++++++
 arch/riscv/lib/memset_vector.S   | 33 +++++++++++++++++++++
 arch/riscv/lib/riscv_v_helpers.c | 26 +++++++++++++++++
 6 files changed, 164 insertions(+)
 create mode 100644 arch/riscv/lib/memcpy_vector.S
 create mode 100644 arch/riscv/lib/memmove_vector.S
 create mode 100644 arch/riscv/lib/memset_vector.S
arch/riscv/Kconfig +24 -0
index 3c5ba05e8a2d..cba53dcc2ae0 100644
@@ -533,6 +533,30 @@ config RISCV_ISA_V_UCOPY_THRESHOLD
 	  Prefer using vectorized copy_to_user()/copy_from_user() when the
 	  workload size exceeds this value.
 
+config RISCV_ISA_V_MEMSET_THRESHOLD
+	int "Threshold size for vectorized memset()"
+	depends on RISCV_ISA_V
+	default 1280
+	help
+	  Prefer using vectorized memset() when the workload size exceeds this
+	  value.
+
+config RISCV_ISA_V_MEMCPY_THRESHOLD
+	int "Threshold size for vectorized memcpy()"
+	depends on RISCV_ISA_V
+	default 768
+	help
+	  Prefer using vectorized memcpy() when the workload size exceeds this
+	  value.
+
+config RISCV_ISA_V_MEMMOVE_THRESHOLD
+	int "Threshold size for vectorized memmove()"
+	depends on RISCV_ISA_V
+	default 512
+	help
+	  Prefer using vectorized memmove() when the workload size exceeds this
+	  value.
+
 config TOOLCHAIN_HAS_ZBB
 	bool
 	default y
arch/riscv/lib/Makefile +3 -0
index 1fe8d797e0f2..3111863afd2e 100644
@@ -14,3 +14,6 @@ obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
 lib-$(CONFIG_RISCV_ISA_V)	+= xor.o
 lib-$(CONFIG_RISCV_ISA_V)	+= riscv_v_helpers.o
 lib-$(CONFIG_RISCV_ISA_V)	+= uaccess_vector.o
+lib-$(CONFIG_RISCV_ISA_V)	+= memset_vector.o
+lib-$(CONFIG_RISCV_ISA_V)	+= memcpy_vector.o
+lib-$(CONFIG_RISCV_ISA_V)	+= memmove_vector.o
arch/riscv/lib/memcpy_vector.S new file +29 -0
new file mode 100644
index 000000000000..4176b6e0a53c
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#include <linux/linkage.h>
+#include <asm/asm.h>
+
+#define pDst a0
+#define pSrc a1
+#define iNum a2
+
+#define iVL a3
+#define pDstPtr a4
+
+#define ELEM_LMUL_SETTING m8
+#define vData v0
+
+
+/* void *memcpy(void *, const void *, size_t) */
+SYM_FUNC_START(__asm_memcpy_vector)
+	mv pDstPtr, pDst
+loop:
+	vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
+	vle8.v vData, (pSrc)
+	sub iNum, iNum, iVL
+	add pSrc, pSrc, iVL
+	vse8.v vData, (pDstPtr)
+	add pDstPtr, pDstPtr, iVL
+	bnez iNum, loop
+	ret
+SYM_FUNC_END(__asm_memcpy_vector)
arch/riscv/lib/memmove_vector.S new file +49 -0

49 lines not on this page · raw message

arch/riscv/lib/memset_vector.S new file +33 -0
new file mode 100644
index 000000000000..4611feed72ac
@@ -0,0 +1,33 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#include <linux/linkage.h>
+#include <asm/asm.h>
+
+#define pDst a0
+#define iValue a1
+#define iNum a2
+
+#define iVL a3
+#define iTemp a4
+#define pDstPtr a5
+
+#define ELEM_LMUL_SETTING m8
+#define vData v0
+
+/* void *memset(void *, int, size_t) */
+SYM_FUNC_START(__asm_memset_vector)
+
+    mv pDstPtr, pDst
+
+    vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
+    vmv.v.x vData, iValue
+
+loop:
+    vse8.v vData, (pDstPtr)
+    sub iNum, iNum, iVL
+    add pDstPtr, pDstPtr, iVL
+    vsetvli iVL, iNum, e8, ELEM_LMUL_SETTING, ta, ma
+    bnez iNum, loop
+
+    ret
+
+SYM_FUNC_END(__asm_memset_vector)
arch/riscv/lib/riscv_v_helpers.c +26 -0
index 139e5de1b793..28467d737faf 100644
@@ -3,9 +3,13 @@
  * Copyright (C) 2023 SiFive
  * Author: Andy Chiu <[email protected]>
  */
+#ifndef __NO_FORTIFY
+# define __NO_FORTIFY
+#endif
 #include <linux/linkage.h>
 #include <asm/asm.h>
 
+#include <asm/string.h>
 #include <asm/vector.h>
 #include <asm/simd.h>
 
@@ -36,3 +40,25 @@ asmlinkage int enter_vector_usercopy(void *dst, void *src, size_t n)
 fallback:
 	return fallback_scalar_usercopy(dst, src, n);
 }
+
+#define V_OPT_TEMPLATE3(prefix, type_r, type_0, type_1)				\
+extern type_r __asm_##prefix##_vector(type_0, type_1, size_t n);		\
+type_r prefix(type_0 a0, type_1 a1, size_t n)					\
+{										\
+	type_r ret;								\
+	if (has_vector() && may_use_simd() &&					\
+	    n > riscv_v_##prefix##_threshold) {					\
+		kernel_vector_begin();						\
+		ret = __asm_##prefix##_vector(a0, a1, n);			\
+		kernel_vector_end();						\
+		return ret;							\
+	}									\
+	return __##prefix(a0, a1, n);						\
+}
+
+static size_t riscv_v_memset_threshold = CONFIG_RISCV_ISA_V_MEMSET_THRESHOLD;
+V_OPT_TEMPLATE3(memset, void *, void*, int)
+static size_t riscv_v_memcpy_threshold = CONFIG_RISCV_ISA_V_MEMCPY_THRESHOLD;
+V_OPT_TEMPLATE3(memcpy, void *, void*, const void *)
+static size_t riscv_v_memmove_threshold = CONFIG_RISCV_ISA_V_MEMMOVE_THRESHOLD;
+V_OPT_TEMPLATE3(memmove, void *, void*, const void *)
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
author
#12 lore raw

PATCH v7 7/10 riscv: vector: do not pass task_struct into riscv_v_vstate_{save,restore}()

In reply to Andy Chiu

riscv_v_vstate_{save,restore}() can operate only on the knowlege of struct __riscv_v_ext_state, and struct pt_regs. Let the caller decides which should be passed into the function. Meanwhile, the kernel-mode Vector is going to introduce another vstate, so this also makes functions potentially able to be reused.

---
Changelog v6:
- re-added for v6
Changelog v3:
- save V context after get_cpu_vector_context
Changelog v2:
- fix build fail that get caught on this patch (Conor)

---
 arch/riscv/include/asm/entry-common.h  |  2 +-
 arch/riscv/include/asm/vector.h        | 14 +++++---------
 arch/riscv/kernel/kernel_mode_vector.c |  2 +-
 arch/riscv/kernel/ptrace.c             |  2 +-
 arch/riscv/kernel/signal.c             |  2 +-
 5 files changed, 9 insertions(+), 13 deletions(-)
arch/riscv/include/asm/entry-common.h +1 -1
index 6361a8488642..08fe8cdbf33e 100644
@@ -16,7 +16,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs,
 		 * We are already called with irq disabled, so go without
 		 * keeping track of vector_context_busy.
 		 */
-		riscv_v_vstate_restore(current, regs);
+		riscv_v_vstate_restore(&current->thread.vstate, regs);
 	}
 }
 
arch/riscv/include/asm/vector.h +5 -9
index e706613aae2c..c5a83c277583 100644
@@ -183,23 +183,19 @@ static inline void riscv_v_vstate_discard(struct pt_regs *regs)
 	__riscv_v_vstate_dirty(regs);
 }
 
-static inline void riscv_v_vstate_save(struct task_struct *task,
+static inline void riscv_v_vstate_save(struct __riscv_v_ext_state *vstate,
 				       struct pt_regs *regs)
 {
 	if ((regs->status & SR_VS) == SR_VS_DIRTY) {
-		struct __riscv_v_ext_state *vstate = &task->thread.vstate;
-
 		__riscv_v_vstate_save(vstate, vstate->datap);
 		__riscv_v_vstate_clean(regs);
 	}
 }
 
-static inline void riscv_v_vstate_restore(struct task_struct *task,
+static inline void riscv_v_vstate_restore(struct __riscv_v_ext_state *vstate,
 					  struct pt_regs *regs)
 {
 	if ((regs->status & SR_VS) != SR_VS_OFF) {
-		struct __riscv_v_ext_state *vstate = &task->thread.vstate;
-
 		__riscv_v_vstate_restore(vstate, vstate->datap);
 		__riscv_v_vstate_clean(regs);
 	}
@@ -220,7 +216,7 @@ static inline void __switch_to_vector(struct task_struct *prev,
 	struct pt_regs *regs;
 
 	regs = task_pt_regs(prev);
-	riscv_v_vstate_save(prev, regs);
+	riscv_v_vstate_save(&prev->thread.vstate, regs);
 	riscv_v_vstate_set_restore(next, task_pt_regs(next));
 }
 
@@ -238,8 +234,8 @@ static inline bool riscv_v_vstate_query(struct pt_regs *regs) { return false; }
 static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; }
 #define riscv_v_vsize (0)
 #define riscv_v_vstate_discard(regs)		do {} while (0)
-#define riscv_v_vstate_save(task, regs)		do {} while (0)
-#define riscv_v_vstate_restore(task, regs)	do {} while (0)
+#define riscv_v_vstate_save(vstate, regs)	do {} while (0)
+#define riscv_v_vstate_restore(vstate, regs)	do {} while (0)
 #define __switch_to_vector(__prev, __next)	do {} while (0)
 #define riscv_v_vstate_off(regs)		do {} while (0)
 #define riscv_v_vstate_on(regs)			do {} while (0)
arch/riscv/kernel/kernel_mode_vector.c +1 -1
index 3f1d67109e5a..238154cb4fce 100644
@@ -72,7 +72,7 @@ void kernel_vector_begin(void)
 
 	get_cpu_vector_context();
 
-	riscv_v_vstate_save(current, task_pt_regs(current));
+	riscv_v_vstate_save(&current->thread.vstate, task_pt_regs(current));
 
 	riscv_v_enable();
 }
arch/riscv/kernel/ptrace.c +1 -1
index 7b93bcbdf9fa..e8515aa9d80b 100644
@@ -101,7 +101,7 @@ static int riscv_vr_get(struct task_struct *target,
 	 */
 	if (target == current) {
 		get_cpu_vector_context();
-		riscv_v_vstate_save(current, task_pt_regs(current));
+		riscv_v_vstate_save(&current->thread.vstate, task_pt_regs(current));
 		put_cpu_vector_context();
 	}
 
arch/riscv/kernel/signal.c +1 -1
index aca4a12c8416..5d69f4db9e8f 100644
@@ -87,7 +87,7 @@ static long save_v_state(struct pt_regs *regs, void __user **sc_vec)
 	WARN_ON(unlikely(!IS_ALIGNED((unsigned long)datap, 16)));
 
 	get_cpu_vector_context();
-	riscv_v_vstate_save(current, regs);
+	riscv_v_vstate_save(&current->thread.vstate, regs);
 	put_cpu_vector_context();
 
 	/* Copy everything of vstate but datap. */
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
author
#13 lore raw

PATCH v7 8/10 riscv: vector: use a mask to write vstate_ctrl

In reply to Andy Chiu

riscv_v_ctrl_set() should only touch bits within PR_RISCV_V_VSTATE_CTRL_MASK. So, use the mask when we really set task's vstate_ctrl.

---
Changelog v6:
- splitted out from v3

---
 arch/riscv/kernel/vector.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/riscv/kernel/vector.c +2 -1
index 66e8c6ab09d2..c1f28bc89ec6 100644
@@ -122,7 +122,8 @@ static inline void riscv_v_ctrl_set(struct task_struct *tsk, int cur, int nxt,
 	ctrl |= VSTATE_CTRL_MAKE_NEXT(nxt);
 	if (inherit)
 		ctrl |= PR_RISCV_V_VSTATE_CTRL_INHERIT;
-	tsk->thread.vstate_ctrl = ctrl;
+	tsk->thread.vstate_ctrl &= ~PR_RISCV_V_VSTATE_CTRL_MASK;
+	tsk->thread.vstate_ctrl |= ctrl;
 }
 
 bool riscv_v_vstate_ctrl_user_allowed(void)
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
author
#14 lore raw

PATCH v7 9/10 riscv: vector: use kmem_cache to manage vector context

In reply to Andy Chiu

The allocation size of thread.vstate.datap is always riscv_v_vsize. So it is possbile to use kmem_cache_* to manage the allocation. This gives users more information regarding allocation of vector context via /proc/slabinfo. And it potentially reduces the latency of the first-use trap because of the allocation caches.

---
Changelog v6:
- new patch since v6

---
 arch/riscv/include/asm/vector.h |  4 ++++
 arch/riscv/kernel/process.c     |  7 ++++++-
 arch/riscv/kernel/vector.c      | 16 +++++++++++++++-
 3 files changed, 25 insertions(+), 2 deletions(-)
arch/riscv/include/asm/vector.h +4 -0
index c5a83c277583..0e6741dd9ef3 100644
@@ -26,6 +26,8 @@ void kernel_vector_begin(void);
 void kernel_vector_end(void);
 void get_cpu_vector_context(void);
 void put_cpu_vector_context(void);
+void riscv_v_thread_free(struct task_struct *tsk);
+void __init riscv_v_setup_ctx_cache(void);
 
 static inline void riscv_v_ctx_cnt_add(u32 offset)
 {
@@ -239,6 +241,8 @@ static inline bool riscv_v_vstate_ctrl_user_allowed(void) { return false; }
 #define __switch_to_vector(__prev, __next)	do {} while (0)
 #define riscv_v_vstate_off(regs)		do {} while (0)
 #define riscv_v_vstate_on(regs)			do {} while (0)
+#define riscv_v_thread_free(tsk)		do {} while (0)
+#define  riscv_v_setup_ctx_cache()		do {} while (0)
 
 #endif /* CONFIG_RISCV_ISA_V */
 
arch/riscv/kernel/process.c +6 -1
index 58127b1c6c71..38bdbcf9b81d 100644
@@ -179,7 +179,7 @@ void arch_release_task_struct(struct task_struct *tsk)
 {
 	/* Free the vector context of datap. */
 	if (has_vector())
-		kfree(tsk->thread.vstate.datap);
+		riscv_v_thread_free(tsk);
 }
 
 int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
@@ -227,3 +227,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 	p->thread.sp = (unsigned long)childregs; /* kernel sp */
 	return 0;
 }
+
+void __init arch_task_cache_init(void)
+{
+	riscv_v_setup_ctx_cache();
+}
arch/riscv/kernel/vector.c +15 -1
index c1f28bc89ec6..1fe140e34557 100644
@@ -21,6 +21,7 @@
 #include <asm/bug.h>
 
 static bool riscv_v_implicit_uacc = IS_ENABLED(CONFIG_RISCV_ISA_V_DEFAULT_ENABLE);
+static struct kmem_cache *riscv_v_user_cachep;
 
 unsigned long riscv_v_vsize __read_mostly;
 EXPORT_SYMBOL_GPL(riscv_v_vsize);
@@ -47,6 +48,13 @@ int riscv_v_setup_vsize(void)
 	return 0;
 }
 
+void __init riscv_v_setup_ctx_cache(void)
+{
+	riscv_v_user_cachep = kmem_cache_create_usercopy("riscv_vector_ctx",
+							 riscv_v_vsize, 16, SLAB_PANIC,
+							 0, riscv_v_vsize, NULL);
+}
+
 static bool insn_is_vector(u32 insn_buf)
 {
 	u32 opcode = insn_buf & __INSN_OPCODE_MASK;
@@ -84,7 +92,7 @@ static int riscv_v_thread_zalloc(void)
 {
 	void *datap;
 
-	datap = kzalloc(riscv_v_vsize, GFP_KERNEL);
+	datap = kmem_cache_zalloc(riscv_v_user_cachep, GFP_KERNEL);
 	if (!datap)
 		return -ENOMEM;
 
@@ -94,6 +102,12 @@ static int riscv_v_thread_zalloc(void)
 	return 0;
 }
 
+void riscv_v_thread_free(struct task_struct *tsk)
+{
+	if (tsk->thread.vstate.datap)
+		kmem_cache_free(riscv_v_user_cachep, tsk->thread.vstate.datap);
+}
+
 #define VSTATE_CTRL_GET_CUR(x) ((x) & PR_RISCV_V_VSTATE_CTRL_CUR_MASK)
 #define VSTATE_CTRL_GET_NEXT(x) (((x) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK) >> 2)
 #define VSTATE_CTRL_MAKE_NEXT(x) (((x) << 2) & PR_RISCV_V_VSTATE_CTRL_NEXT_MASK)
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
author
#15 lore raw

PATCH v7 10/10 riscv: vector: allow kernel-mode Vector with preemption

In reply to Andy Chiu

Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide riscv_v_flags to let context save/restore routine track context status. Context tracking happens whenever the core starts its in-kernel Vector executions. An active (dirty) kernel task's V contexts will be saved to memory whenever a trap-introduced context switch happens. Or, when a softirq, which happens to nest on top of it, uses Vector. Context retoring happens when the execution transfer back to the original Kernel context where it first enable preempt_v.

Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking of per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context switch overhead.

---
Changelog v6:
- re-write patch to handle context nesting for softirqs
- drop thread flag and track context instead in riscv_v_flags
- refine some asm code and constraint it into C functions
- preallocate v context for preempt_v
- Return non-zero in riscv_v_start_kernel_context with non-preemptible kernel-mode Vector
Changelog v4:
- dropped from v4
Changelog v3:
- Guard vstate_save with {get,set}_cpu_vector_context
- Add comments on preventions of nesting V contexts
- remove warnings in context switch when trap's reg is not pressent (Conor)
- refactor code (Björn)
Changelog v2:
- fix build fail when compiling without RISCV_ISA_V (Conor)
- 's/TIF_RISCV_V_KMV/TIF_RISCV_V_KERNEL_MODE' and add comment (Conor)
- merge Kconfig patch into this oine (Conor).
- 's/CONFIG_RISCV_ISA_V_PREEMPTIVE_KMV/CONFIG_RISCV_ISA_V_PREEMPTIVE/' (Conor)
- fix some typos (Conor)
- enclose assembly with RISCV_ISA_V_PREEMPTIVE.
- change riscv_v_vstate_ctrl_config_kmv() to kernel_vector_allow_preemption() for better understanding. (Conor)
- 's/riscv_v_kmv_preempitble/kernel_vector_preemptible/'

---
 arch/riscv/Kconfig                     |  14 +++
 arch/riscv/include/asm/processor.h     |  26 +++++-
 arch/riscv/include/asm/simd.h          |  26 +++++-
 arch/riscv/include/asm/vector.h        |  57 +++++++++++-
 arch/riscv/kernel/entry.S              |   8 ++
 arch/riscv/kernel/kernel_mode_vector.c | 121 ++++++++++++++++++++++++-
 arch/riscv/kernel/process.c            |   3 +
 arch/riscv/kernel/vector.c             |  31 +++++--
 8 files changed, 265 insertions(+), 21 deletions(-)
arch/riscv/Kconfig +14 -0
index cba53dcc2ae0..70603c486593 100644
@@ -557,6 +557,20 @@ config RISCV_ISA_V_MEMMOVE_THRESHOLD
 	  Prefer using vectorized memmove() when the workload size exceeds this
 	  value.
 
+config RISCV_ISA_V_PREEMPTIVE
+	bool "Run kernel-mode Vector with kernel preemption"
+	depends on PREEMPTION
+	depends on RISCV_ISA_V
+	default y
+	help
+	  Usually, in-kernel SIMD routines are run with preemption disabled.
+	  Functions which envoke long running SIMD thus must yield core's
+	  vector unit to prevent blocking other tasks for too long.
+
+	  This config allows kernel to run SIMD without explicitly disable
+	  preemption. Enabling this config will result in higher memory
+	  consumption due to the allocation of per-task's kernel Vector context.
+
 config TOOLCHAIN_HAS_ZBB
 	bool
 	default y
arch/riscv/include/asm/processor.h +24 -2
index 15781e2232e0..4de9124bcf4f 100644
@@ -81,11 +81,32 @@ struct pt_regs;
  *    activation of this state disables the preemption. On a non-RT kernel, it
  *    also disable bh. Currently only 0 and 1 are valid value for this field.
  *    Other values are reserved for future uses.
+ *  - bits 8-15 are used for tracking preemptible kernel-mode Vector, when
+ *    RISCV_ISA_V_PREEMPTIVE is set. Calling kernel_vector_begin() does not
+ *    disable the preemption if the thread's kernel_vstate.datap is allocated.
+ *    Instead, the kernel adds 1 into this field. Then the trap entry/exit code
+ *    knows if we are entering/exiting the context that owns preempt_v.
+ *     - 0: the task is not using preempt_v
+ *     - 1: the task is actively using, and owns preempt_v
+ *     - >1: the task was using preempt_v, but then took a trap within. Thus,
+ *       the task does not own preempt_v. Any use of Vector will have to save
+ *       preempt_v, if dirty, and fallback to non-preemptible kernel-mode
+ *       Vector.
+ *   - bit 30: The in-kernel preempt_v context is saved, and requries to be
+ *     restored when returning to the context that owns the preempt_v.
+ *   - bit 31: The in-kernel preempt_v context is dirty, as signaled by the
+ *     trap entry code. Any context switches out-of current task need to save
+ *     it to the task's in-kernel V context. Also, any traps nesting on-top-of
+ *     preempt_v requesting to use V needs a save.
  */
 
-#define RISCV_KERNEL_MODE_V_MASK	0xff
+#define RISCV_KERNEL_MODE_V_MASK	0x000000ff
+#define RISCV_PREEMPT_V_MASK		0x0000ff00
 
-#define RISCV_KERNEL_MODE_V	0x1
+#define RISCV_KERNEL_MODE_V		0x00000001
+#define RISCV_PREEMPT_V			0x00000100
+#define RISCV_PREEMPT_V_DIRTY		0x80000000
+#define RISCV_PREEMPT_V_NEED_RESTORE	0x40000000
 
 /* CPU-specific state of a task */
 struct thread_struct {
@@ -99,6 +120,7 @@ struct thread_struct {
 	u32 vstate_ctrl;
 	struct __riscv_v_ext_state vstate;
 	unsigned long align_ctl;
+	struct __riscv_v_ext_state kernel_vstate;
 };
 
 /* Whitelist the fstate from the task_struct for hardened usercopy */
arch/riscv/include/asm/simd.h +21 -5
index 7df5a976a80a..3fe6f201ee6e 100644
@@ -12,6 +12,7 @@
 #include <linux/percpu.h>
 #include <linux/preempt.h>
 #include <linux/types.h>
+#include <linux/thread_info.h>
 
 #include <asm/vector.h>
 
@@ -28,12 +29,27 @@ static __must_check inline bool may_use_simd(void)
 	/*
 	 * RISCV_KERNEL_MODE_V is only set while preemption is disabled,
 	 * and is clear whenever preemption is enabled.
-	 *
-	 * Kernel-mode Vector temperarily disables bh. So we must not return
-	 * true on irq_disabled(). Otherwise we would fail the lockdep check
-	 * calling local_bh_enable()
 	 */
-	return !in_hardirq() && !in_nmi() && !irqs_disabled() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK);
+	if (in_hardirq() || in_nmi())
+		return false;
+
+	/*
+	 * Nesting is acheived in preempt_v by spreading the control for
+	 * preemptible and non-preemptible kernel-mode Vector into two fields.
+	 * Always try to match with prempt_v if kernel V-context exists. Then,
+	 * fallback to check non preempt_v if nesting happens, or if the config
+	 * is not set.
+	 */
+	if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) && current->thread.kernel_vstate.datap) {
+		if (!riscv_preempt_v_started(current))
+			return true;
+	}
+	/*
+	 * Non-preemptible kernel-mode Vector temperarily disables bh. So we
+	 * must not return true on irq_disabled(). Otherwise we would fail the
+	 * lockdep check calling local_bh_enable()
+	 */
+	return !irqs_disabled() && !(riscv_v_ctx_cnt() & RISCV_KERNEL_MODE_V_MASK);
 }
 
 #else /* ! CONFIG_RISCV_ISA_V */
arch/riscv/include/asm/vector.h +54 -3

80 lines not on this page · raw message

arch/riscv/kernel/entry.S +8 -0
index 54ca4564a926..9d1a305d5508 100644
@@ -83,6 +83,10 @@ SYM_CODE_START(handle_exception)
 	/* Load the kernel shadow call stack pointer if coming from userspace */
 	scs_load_current_if_task_changed s5
 
+#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
+	move a0, sp
+	call riscv_v_context_nesting_start
+#endif
 	move a0, sp /* pt_regs */
 	la ra, ret_from_exception
 
@@ -138,6 +142,10 @@ SYM_CODE_START_NOALIGN(ret_from_exception)
 	 */
 	csrw CSR_SCRATCH, tp
 1:
+#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
+	move a0, sp
+	call riscv_v_context_nesting_end
+#endif
 	REG_L a0, PT_STATUS(sp)
 	/*
 	 * The current load reservation is effectively part of the processor's
arch/riscv/kernel/kernel_mode_vector.c +116 -5

146 lines not on this page · raw message

arch/riscv/kernel/process.c +3 -0
index 38bdbcf9b81d..1afdec4aeda6 100644
@@ -188,6 +188,7 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
 	*dst = *src;
 	/* clear entire V context, including datap for a new task */
 	memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
+	memset(&dst->thread.kernel_vstate, 0, sizeof(struct __riscv_v_ext_state));
 	clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE);
 	return 0;
 }
@@ -223,6 +224,8 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
 		p->thread.s[0] = 0;
 	}
 	p->thread.riscv_v_flags = 0;
+	if (has_vector())
+		riscv_v_thread_alloc(p);
 	p->thread.ra = (unsigned long)ret_from_fork;
 	p->thread.sp = (unsigned long)childregs; /* kernel sp */
 	return 0;
arch/riscv/kernel/vector.c +25 -6
index 1fe140e34557..f9769703fd39 100644
@@ -22,6 +22,9 @@
 
 static bool riscv_v_implicit_uacc = IS_ENABLED(CONFIG_RISCV_ISA_V_DEFAULT_ENABLE);
 static struct kmem_cache *riscv_v_user_cachep;
+#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
+static struct kmem_cache *riscv_v_kernel_cachep;
+#endif
 
 unsigned long riscv_v_vsize __read_mostly;
 EXPORT_SYMBOL_GPL(riscv_v_vsize);
@@ -53,6 +56,11 @@ void __init riscv_v_setup_ctx_cache(void)
 	riscv_v_user_cachep = kmem_cache_create_usercopy("riscv_vector_ctx",
 							 riscv_v_vsize, 16, SLAB_PANIC,
 							 0, riscv_v_vsize, NULL);
+#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
+	riscv_v_kernel_cachep = kmem_cache_create("riscv_vector_kctx",
+						  riscv_v_vsize, 16,
+						  SLAB_PANIC, NULL);
+#endif
 }
 
 static bool insn_is_vector(u32 insn_buf)
@@ -88,24 +96,35 @@ static bool insn_is_vector(u32 insn_buf)
 	return false;
 }
 
-static int riscv_v_thread_zalloc(void)
+static int riscv_v_thread_zalloc(struct kmem_cache *cache,
+				 struct __riscv_v_ext_state *ctx)
 {
 	void *datap;
 
-	datap = kmem_cache_zalloc(riscv_v_user_cachep, GFP_KERNEL);
+	datap = kmem_cache_zalloc(cache, GFP_KERNEL);
 	if (!datap)
 		return -ENOMEM;
 
-	current->thread.vstate.datap = datap;
-	memset(&current->thread.vstate, 0, offsetof(struct __riscv_v_ext_state,
-						    datap));
+	ctx->datap = datap;
+	memset(ctx, 0, offsetof(struct __riscv_v_ext_state, datap));
 	return 0;
 }
 
+void riscv_v_thread_alloc(struct task_struct *tsk)
+{
+#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
+	riscv_v_thread_zalloc(riscv_v_kernel_cachep, &tsk->thread.kernel_vstate);
+#endif
+}
+
 void riscv_v_thread_free(struct task_struct *tsk)
 {
 	if (tsk->thread.vstate.datap)
 		kmem_cache_free(riscv_v_user_cachep, tsk->thread.vstate.datap);
+#ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE
+	if (tsk->thread.kernel_vstate.datap)
+		kmem_cache_free(riscv_v_kernel_cachep, tsk->thread.kernel_vstate.datap);
+#endif
 }
 
 #define VSTATE_CTRL_GET_CUR(x) ((x) & PR_RISCV_V_VSTATE_CTRL_CUR_MASK)
@@ -177,7 +196,7 @@ bool riscv_v_first_use_handler(struct pt_regs *regs)
 	 * context where VS has been off. So, try to allocate the user's V
 	 * context and resume execution.
 	 */
-	if (riscv_v_thread_zalloc()) {
+	if (riscv_v_thread_zalloc(riscv_v_user_cachep, &current->thread.vstate)) {
 		force_sig(SIGBUS);
 		return true;
 	}
Signature · 7 lines
2.17.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv