Skip to contentSkip to conversation

riscv: Add independent irq/softirq stacks

PATCH Superseded [email protected] 2 participants 6 messages May 29, 2023 – Jun 13, 2023

Subsystems (2)

v12 of 13 Superseded Latest is v13 →

lore
Share & tools
lore
#0 lore raw

v12 cover riscv: Add independent irq/softirq stacks

From: Guo Ren <[email protected]>

This patch series adds independent irq/softirq stacks to decrease the press of the thread stack. Also, add a thread STACK_SIZE config for users to adjust the proper size during compile time.

This patch series belonged to the generic entry, which has been merged to for-next now.

v12:
- Rebase on palmer/for-next (20230529)
- Move DECLARE_PER_CPU(ulong *, irq_stack_ptr) into irq_stack.h (Thx Conor)
- Optimize commit msg

v11:
https://lore.kernel.org/linux-riscv/[email protected]/
- Rebase on palmer/for-next (20230324)
- Separate from generic entry patch series.

v10:
https://lore.kernel.org/linux-riscv/[email protected]/
- Rebase on palmer/for-next branch (20221208)
- Remove unrelated patches from the series (Suggested-by: Bjorn)
- Fixup Typos.

v9:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup NR_syscalls check (by Ben Hutchings)
- Add Tested-by: Jisheng Zhang

v8:
https://lore.kernel.org/linux-riscv/[email protected]/
- Rebase on palmer/for-next branch (20221102)
- Add save/restore_from_x5_to_x31 .macro (JishengZhang)
- Consolidate ret_from_kernel_thread into ret_from_fork (JishengZhang)
- Optimize __noinstr_section comment (JiangshanLai)

v7:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup regs_irqs_disabled with SR_PIE
- Optimize stackleak_erase -> stackleak_erase_on_task_stack (Thx Mark Rutland)
- Add BUG_ON(!irqs_disabled()) in trap handlers
- Using regs_irqs_disabled in __do_page_fault
- Remove unnecessary irq disable in ret_from_exception and add comment

v6:
https://lore.kernel.org/linux-riscv/[email protected]/
- Use THEAD_SIZE_ORDER for thread size adjustment in kconfig (Thx Arnd)
- Move call_on_stack to inline style (Thx Peter Zijlstra)
- Fixup fp chain broken (Thx Chen Zhongjin)
- Remove common entry modification, and fixup page_fault entry (Thx Peter Zijlstra)
- Treat some traps as nmi entry (Thx Peter Zijlstra)

v5:
https://lore.kernel.org/linux-riscv/[email protected]/
- Add riscv own stackleak patch instead of generic entry modification (by Mark Rutland)
- Add EXPERT dependency for THREAD_SIZE (by Arnd)
- Add EXPERT dependency for IRQ_STACK (by Sebastian, David Laight)
- Corrected __trap_section (by Peter Zijlstra)
- Add Tested-by (Yipeng Zou)
- Use CONFIG_SOFTIRQ_ON_OWN_STACK replace "#ifndef CONFIG_PREEMPT_RT"
- Fixup systrace_enter compile error
- Fixup exit_to_user_mode_prepare preempt_disable warning

V4:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup entry.S with "la" bug (by Conor.Dooley)
- Fixup missing noinstr bug (by Peter Zijlstra)

V3:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup CONFIG_COMPAT=n compile error
- Add THREAD_SIZE_ORDER config
- Optimize elf_kexec.c warning fixup
- Add static to irq_stack_ptr definition

V2:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup compile error by include "riscv: ptrace: Remove duplicate operation"
- Fixup compile warning

- Add test repo link in cover letter

V1:
https://lore.kernel.org/linux-riscv/[email protected]/

Guo Ren (3):
riscv: stack: Support HAVE_IRQ_EXIT_ON_IRQ_STACK
riscv: stack: Support HAVE_SOFTIRQ_ON_OWN_STACK
riscv: stack: Add config of thread stack size

 arch/riscv/Kconfig                   | 19 ++++++++
 arch/riscv/include/asm/irq_stack.h   | 32 +++++++++++++
 arch/riscv/include/asm/thread_info.h | 14 ++----
 arch/riscv/kernel/irq.c              | 68 ++++++++++++++++++++++++++++
 arch/riscv/kernel/traps.c            | 35 +++++++++++++-
 5 files changed, 155 insertions(+), 13 deletions(-)
 create mode 100644 arch/riscv/include/asm/irq_stack.h
Signature · 1 line
2.36.1
#1 lore raw

PATCH v12 1/3 riscv: stack: Support HAVE_IRQ_EXIT_ON_IRQ_STACK

From: Guo Ren <[email protected]>

Add independent irq stacks for percpu to prevent kernel stack overflows. It is also compatible with VMAP_STACK by arch_alloc_vmap_stack.

---
 arch/riscv/Kconfig                   |  7 ++++++
 arch/riscv/include/asm/irq_stack.h   | 32 +++++++++++++++++++++++++
 arch/riscv/include/asm/thread_info.h |  2 ++
 arch/riscv/kernel/irq.c              | 33 ++++++++++++++++++++++++++
 arch/riscv/kernel/traps.c            | 35 ++++++++++++++++++++++++++--
 5 files changed, 107 insertions(+), 2 deletions(-)
 create mode 100644 arch/riscv/include/asm/irq_stack.h
arch/riscv/Kconfig +7 -0
index fa256f2e23c1..44b4c9690f94 100644
@@ -588,6 +588,13 @@ config FPU
 
 	  If you don't know what to do here, say Y.
 
+config IRQ_STACKS
+	bool "Independent irq stacks" if EXPERT
+	default y
+	select HAVE_IRQ_EXIT_ON_IRQ_STACK
+	help
+	  Add independent irq stacks for percpu to prevent kernel stack overflows.
+
 endmenu # "Platform type"
 
 menu "Kernel features"
arch/riscv/include/asm/irq_stack.h new file +32 -0
new file mode 100644
index 000000000000..b0dcee9a3fa2
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _ASM_RISCV_IRQ_STACK_H
+#define _ASM_RISCV_IRQ_STACK_H
+
+#include <linux/bug.h>
+#include <linux/gfp.h>
+#include <linux/kconfig.h>
+#include <linux/vmalloc.h>
+#include <linux/pgtable.h>
+#include <asm/thread_info.h>
+
+DECLARE_PER_CPU(ulong *, irq_stack_ptr);
+
+#ifdef CONFIG_VMAP_STACK
+/*
+ * To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd
+ * stacks need to have the same alignment.
+ */
+static inline unsigned long *arch_alloc_vmap_stack(size_t stack_size, int node)
+{
+	void *p;
+
+	BUILD_BUG_ON(!IS_ENABLED(CONFIG_VMAP_STACK));
+
+	p = __vmalloc_node(stack_size, THREAD_ALIGN, THREADINFO_GFP, node,
+			__builtin_return_address(0));
+	return kasan_reset_tag(p);
+}
+#endif /* CONFIG_VMAP_STACK */
+
+#endif /* _ASM_RISCV_IRQ_STACK_H */
arch/riscv/include/asm/thread_info.h +2 -0
index 97e6f65ec176..2f32875276b0 100644
@@ -40,6 +40,8 @@
 #define OVERFLOW_STACK_SIZE     SZ_4K
 #define SHADOW_OVERFLOW_STACK_SIZE (1024)
 
+#define IRQ_STACK_SIZE		THREAD_SIZE
+
 #ifndef __ASSEMBLY__
 
 extern long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE / sizeof(long)];
arch/riscv/kernel/irq.c +33 -0
index eb9a68a539e6..a1dcf8e43b3c 100644
@@ -28,6 +28,38 @@ struct fwnode_handle *riscv_get_intc_hwnode(void)
 }
 EXPORT_SYMBOL_GPL(riscv_get_intc_hwnode);
 
+#ifdef CONFIG_IRQ_STACKS
+#include <asm/irq_stack.h>
+
+DEFINE_PER_CPU(ulong *, irq_stack_ptr);
+
+#ifdef CONFIG_VMAP_STACK
+static void init_irq_stacks(void)
+{
+	int cpu;
+	ulong *p;
+
+	for_each_possible_cpu(cpu) {
+		p = arch_alloc_vmap_stack(IRQ_STACK_SIZE, cpu_to_node(cpu));
+		per_cpu(irq_stack_ptr, cpu) = p;
+	}
+}
+#else
+/* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */
+DEFINE_PER_CPU_ALIGNED(ulong [IRQ_STACK_SIZE/sizeof(ulong)], irq_stack);
+
+static void init_irq_stacks(void)
+{
+	int cpu;
+
+	for_each_possible_cpu(cpu)
+		per_cpu(irq_stack_ptr, cpu) = per_cpu(irq_stack, cpu);
+}
+#endif /* CONFIG_VMAP_STACK */
+#else
+static void init_irq_stacks(void) {}
+#endif /* CONFIG_IRQ_STACKS */
+
 int arch_show_interrupts(struct seq_file *p, int prec)
 {
 	show_ipi_stats(p, prec);
@@ -36,6 +68,7 @@ int arch_show_interrupts(struct seq_file *p, int prec)
 
 void __init init_IRQ(void)
 {
+	init_irq_stacks();
 	irqchip_init();
 	if (!handle_arch_irq)
 		panic("No interrupt controller found.");
arch/riscv/kernel/traps.c +33 -2
index 24d309c6ab8d..9e0426cfcc60 100644
@@ -27,6 +27,7 @@
 #include <asm/syscall.h>
 #include <asm/thread_info.h>
 #include <asm/vector.h>
+#include <asm/irq_stack.h>
 
 int show_unhandled_signals = 1;
 
@@ -327,16 +328,46 @@ asmlinkage __visible noinstr void do_page_fault(struct pt_regs *regs)
 }
 #endif
 
-asmlinkage __visible noinstr void do_irq(struct pt_regs *regs)
+static void noinstr handle_riscv_irq(struct pt_regs *regs)
 {
 	struct pt_regs *old_regs;
-	irqentry_state_t state = irqentry_enter(regs);
 
 	irq_enter_rcu();
 	old_regs = set_irq_regs(regs);
 	handle_arch_irq(regs);
 	set_irq_regs(old_regs);
 	irq_exit_rcu();
+}
+
+asmlinkage void noinstr do_irq(struct pt_regs *regs)
+{
+	irqentry_state_t state = irqentry_enter(regs);
+#ifdef CONFIG_IRQ_STACKS
+	if (on_thread_stack()) {
+		ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id())
+					+ IRQ_STACK_SIZE/sizeof(ulong);
+		__asm__ __volatile(
+		"addi	sp, sp, -"RISCV_SZPTR  "\n"
+		REG_S"  ra, (sp)		\n"
+		"addi	sp, sp, -"RISCV_SZPTR  "\n"
+		REG_S"  s0, (sp)		\n"
+		"addi	s0, sp, 2*"RISCV_SZPTR "\n"
+		"move	sp, %[sp]		\n"
+		"move	a0, %[regs]		\n"
+		"call	handle_riscv_irq	\n"
+		"addi	sp, s0, -2*"RISCV_SZPTR"\n"
+		REG_L"  s0, (sp)		\n"
+		"addi	sp, sp, "RISCV_SZPTR   "\n"
+		REG_L"  ra, (sp)		\n"
+		"addi	sp, sp, "RISCV_SZPTR   "\n"
+		:
+		: [sp] "r" (sp), [regs] "r" (regs)
+		: "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7",
+		  "t0", "t1", "t2", "t3", "t4", "t5", "t6",
+		  "memory");
+	} else
+#endif
+		handle_riscv_irq(regs);
 
 	irqentry_exit(regs, state);
 }
Signature · 7 lines
2.36.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#2 lore raw

Re: [PATCH -next V12 1/3] riscv: stack: Support HAVE_IRQ_EXIT_ON_IRQ_STACK

Quoted 56 lines · On 29/05/2023 10:45, [email protected] wrote:

From: Guo Ren <[email protected]>

Add independent irq stacks for percpu to prevent kernel stack overflows. It is also compatible with VMAP_STACK by arch_alloc_vmap_stack.

---
  arch/riscv/Kconfig                   |  7 ++++++
  arch/riscv/include/asm/irq_stack.h   | 32 +++++++++++++++++++++++++
  arch/riscv/include/asm/thread_info.h |  2 ++
  arch/riscv/kernel/irq.c              | 33 ++++++++++++++++++++++++++
  arch/riscv/kernel/traps.c            | 35 ++++++++++++++++++++++++++--
  5 files changed, 107 insertions(+), 2 deletions(-)

create mode 100644 arch/riscv/include/asm/irq_stack.h

arch/riscv/Kconfig +7 -0
index fa256f2e23c1..44b4c9690f94 100644
@@ -588,6 +588,13 @@ config FPU
 
  	  If you don't know what to do here, say Y.
 
+config IRQ_STACKS
+	bool "Independent irq stacks" if EXPERT
+	default y
+	select HAVE_IRQ_EXIT_ON_IRQ_STACK
+	help
+	  Add independent irq stacks for percpu to prevent kernel stack overflows.
+
  endmenu # "Platform type"
 
  menu "Kernel features"
arch/riscv/include/asm/irq_stack.h new file +14 -0
new file mode 100644
index 000000000000..b0dcee9a3fa2
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _ASM_RISCV_IRQ_STACK_H
+#define _ASM_RISCV_IRQ_STACK_H
+
+#include <linux/bug.h>
+#include <linux/gfp.h>
+#include <linux/kconfig.h>
+#include <linux/vmalloc.h>
+#include <linux/pgtable.h>
+#include <asm/thread_info.h>
+
+DECLARE_PER_CPU(ulong *, irq_stack_ptr);
+

arch/riscv/include/asm/irq_stack.h line 24

Quoted 10 lines
arch/riscv/include/asm/irq_stack.h new file +10 -0
+#ifdef CONFIG_VMAP_STACK
+/*
+ * To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd
+ * stacks need to have the same alignment.
+ */
+static inline unsigned long *arch_alloc_vmap_stack(size_t stack_size, int node)
+{
+	void *p;
+
+	BUILD_BUG_ON(!IS_ENABLED(CONFIG_VMAP_STACK));

Hi Guo,

Since this function is already guarded with #ifdef CONFIG_VMAP_STACK, I guess this BUILD_BUG_ON() is unnecessary).

Clément

author
#3 lore raw

Re: [PATCH -next V12 1/3] riscv: stack: Support HAVE_IRQ_EXIT_ON_IRQ_STACK

Quoted 73 lines · Clément Léger
Quoted 56 lines · On 29/05/2023 10:45, [email protected] wrote:

From: Guo Ren <[email protected]>

Add independent irq stacks for percpu to prevent kernel stack overflows. It is also compatible with VMAP_STACK by arch_alloc_vmap_stack.

---
  arch/riscv/Kconfig                   |  7 ++++++
  arch/riscv/include/asm/irq_stack.h   | 32 +++++++++++++++++++++++++
  arch/riscv/include/asm/thread_info.h |  2 ++
  arch/riscv/kernel/irq.c              | 33 ++++++++++++++++++++++++++
  arch/riscv/kernel/traps.c            | 35 ++++++++++++++++++++++++++--
  5 files changed, 107 insertions(+), 2 deletions(-)

create mode 100644 arch/riscv/include/asm/irq_stack.h

arch/riscv/Kconfig +7 -0
index fa256f2e23c1..44b4c9690f94 100644
@@ -588,6 +588,13 @@ config FPU
 
        If you don't know what to do here, say Y.
 
+config IRQ_STACKS
+     bool "Independent irq stacks" if EXPERT
+     default y
+     select HAVE_IRQ_EXIT_ON_IRQ_STACK
+     help
+       Add independent irq stacks for percpu to prevent kernel stack overflows.
+
  endmenu # "Platform type"
 
  menu "Kernel features"
arch/riscv/include/asm/irq_stack.h new file +14 -0
new file mode 100644
index 000000000000..b0dcee9a3fa2
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#ifndef _ASM_RISCV_IRQ_STACK_H
+#define _ASM_RISCV_IRQ_STACK_H
+
+#include <linux/bug.h>
+#include <linux/gfp.h>
+#include <linux/kconfig.h>
+#include <linux/vmalloc.h>
+#include <linux/pgtable.h>
+#include <asm/thread_info.h>
+
+DECLARE_PER_CPU(ulong *, irq_stack_ptr);
+

arch/riscv/include/asm/irq_stack.h line 24

Quoted 10 lines
arch/riscv/include/asm/irq_stack.h new file +10 -0
+#ifdef CONFIG_VMAP_STACK
+/*
+ * To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd
+ * stacks need to have the same alignment.
+ */
+static inline unsigned long *arch_alloc_vmap_stack(size_t stack_size, int node)
+{
+     void *p;
+
+     BUILD_BUG_ON(!IS_ENABLED(CONFIG_VMAP_STACK));

Hi Guo,

Quoted 2 lines

Since this function is already guarded with #ifdef CONFIG_VMAP_STACK, I guess this BUILD_BUG_ON() is unnecessary).

Yes, my carelessness. I would remove it in the next version.

Quoted 2 lines

Clément

--
Best Regards
Guo Ren

#4 lore raw

PATCH v12 2/3 riscv: stack: Support HAVE_SOFTIRQ_ON_OWN_STACK

In reply to [email protected]

From: Guo Ren <[email protected]>

Add the HAVE_SOFTIRQ_ON_OWN_STACK feature for the IRQ_STACKS config, and the irq and softirq use the same irq_stack of percpu.

---
 arch/riscv/Kconfig      |  6 ++++--
 arch/riscv/kernel/irq.c | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 39 insertions(+), 2 deletions(-)
arch/riscv/Kconfig +4 -2
index 44b4c9690f94..9567bf5fd5ed 100644
@@ -589,11 +589,13 @@ config FPU
 	  If you don't know what to do here, say Y.
 
 config IRQ_STACKS
-	bool "Independent irq stacks" if EXPERT
+	bool "Independent irq & softirq stacks" if EXPERT
 	default y
 	select HAVE_IRQ_EXIT_ON_IRQ_STACK
+	select HAVE_SOFTIRQ_ON_OWN_STACK
 	help
-	  Add independent irq stacks for percpu to prevent kernel stack overflows.
+	  Add independent irq & softirq stacks for percpu to prevent kernel stack
+	  overflows. We may save some memory footprint by disabling IRQ_STACKS.
 
 endmenu # "Platform type"
 
arch/riscv/kernel/irq.c +35 -0
index a1dcf8e43b3c..d0577cc6a081 100644
@@ -11,6 +11,9 @@
 #include <linux/module.h>
 #include <linux/seq_file.h>
 #include <asm/sbi.h>
+#include <asm/smp.h>
+#include <asm/softirq_stack.h>
+#include <asm/stacktrace.h>
 
 static struct fwnode_handle *(*__get_intc_node)(void);
 
@@ -56,6 +59,38 @@ static void init_irq_stacks(void)
 		per_cpu(irq_stack_ptr, cpu) = per_cpu(irq_stack, cpu);
 }
 #endif /* CONFIG_VMAP_STACK */
+
+#ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK
+void do_softirq_own_stack(void)
+{
+#ifdef CONFIG_IRQ_STACKS
+	if (on_thread_stack()) {
+		ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id())
+					+ IRQ_STACK_SIZE/sizeof(ulong);
+		__asm__ __volatile(
+		"addi	sp, sp, -"RISCV_SZPTR  "\n"
+		REG_S"  ra, (sp)		\n"
+		"addi	sp, sp, -"RISCV_SZPTR  "\n"
+		REG_S"  s0, (sp)		\n"
+		"addi	s0, sp, 2*"RISCV_SZPTR "\n"
+		"move	sp, %[sp]		\n"
+		"call	__do_softirq		\n"
+		"addi	sp, s0, -2*"RISCV_SZPTR"\n"
+		REG_L"  s0, (sp)		\n"
+		"addi	sp, sp, "RISCV_SZPTR   "\n"
+		REG_L"  ra, (sp)		\n"
+		"addi	sp, sp, "RISCV_SZPTR   "\n"
+		:
+		: [sp] "r" (sp)
+		: "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7",
+		  "t0", "t1", "t2", "t3", "t4", "t5", "t6",
+		  "memory");
+	} else
+#endif
+		__do_softirq();
+}
+#endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */
+
 #else
 static void init_irq_stacks(void) {}
 #endif /* CONFIG_IRQ_STACKS */
Signature · 7 lines
2.36.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
#5 lore raw

PATCH v12 3/3 riscv: stack: Add config of thread stack size

In reply to [email protected]

From: Guo Ren <[email protected]>

The commit 0cac21b02ba5 ("riscv: use 16KB kernel stack on 64-bit") increases the thread size mandatory, but some scenarios, such as D1 with a small memory footprint, would suffer from that. After independent irq stack support, let's give users a choice to determine their custom stack size.

---
 arch/riscv/Kconfig                   | 10 ++++++++++
 arch/riscv/include/asm/thread_info.h | 12 +-----------
 2 files changed, 11 insertions(+), 11 deletions(-)
arch/riscv/Kconfig +10 -0
index 9567bf5fd5ed..e82be82882fd 100644
@@ -597,6 +597,16 @@ config IRQ_STACKS
 	  Add independent irq & softirq stacks for percpu to prevent kernel stack
 	  overflows. We may save some memory footprint by disabling IRQ_STACKS.
 
+config THREAD_SIZE_ORDER
+	int "Kernel stack size (in power-of-two numbers of page size)" if VMAP_STACK && EXPERT
+	range 0 4
+	default 1 if 32BIT && !KASAN
+	default 3 if 64BIT && KASAN
+	default 2
+	help
+	  Specify the Pages of thread stack size (from 4KB to 64KB), which also
+	  affects irq stack size, which is equal to thread stack size.
+
 endmenu # "Platform type"
 
 menu "Kernel features"
arch/riscv/include/asm/thread_info.h +1 -11
index 2f32875276b0..1833beb00489 100644
@@ -11,18 +11,8 @@
 #include <asm/page.h>
 #include <linux/const.h>
 
-#ifdef CONFIG_KASAN
-#define KASAN_STACK_ORDER 1
-#else
-#define KASAN_STACK_ORDER 0
-#endif
-
 /* thread information allocation */
-#ifdef CONFIG_64BIT
-#define THREAD_SIZE_ORDER	(2 + KASAN_STACK_ORDER)
-#else
-#define THREAD_SIZE_ORDER	(1 + KASAN_STACK_ORDER)
-#endif
+#define THREAD_SIZE_ORDER	CONFIG_THREAD_SIZE_ORDER
 #define THREAD_SIZE		(PAGE_SIZE << THREAD_SIZE_ORDER)
 
 /*
Signature · 7 lines
2.36.1


_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv