riscv: Add independent irq/softirq stacks
Subsystems (2)
v12 cover riscv: Add independent irq/softirq stacks
From: Guo Ren <[email protected]>
This patch series adds independent irq/softirq stacks to decrease the press of the thread stack. Also, add a thread STACK_SIZE config for users to adjust the proper size during compile time.
This patch series belonged to the generic entry, which has been merged to for-next now.
v12:
- Rebase on palmer/for-next (20230529)
- Move DECLARE_PER_CPU(ulong *, irq_stack_ptr) into irq_stack.h (Thx Conor)
- Optimize commit msg
v11:
https://lore.kernel.org/linux-riscv/[email protected]/
- Rebase on palmer/for-next (20230324)
- Separate from generic entry patch series.
v10:
https://lore.kernel.org/linux-riscv/[email protected]/
- Rebase on palmer/for-next branch (20221208)
- Remove unrelated patches from the series (Suggested-by: Bjorn)
- Fixup Typos.
v9:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup NR_syscalls check (by Ben Hutchings)
- Add Tested-by: Jisheng Zhang
v8:
https://lore.kernel.org/linux-riscv/[email protected]/
- Rebase on palmer/for-next branch (20221102)
- Add save/restore_from_x5_to_x31 .macro (JishengZhang)
- Consolidate ret_from_kernel_thread into ret_from_fork (JishengZhang)
- Optimize __noinstr_section comment (JiangshanLai)
v7:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup regs_irqs_disabled with SR_PIE
- Optimize stackleak_erase -> stackleak_erase_on_task_stack (Thx Mark Rutland)
- Add BUG_ON(!irqs_disabled()) in trap handlers
- Using regs_irqs_disabled in __do_page_fault
- Remove unnecessary irq disable in ret_from_exception and add comment
v6:
https://lore.kernel.org/linux-riscv/[email protected]/
- Use THEAD_SIZE_ORDER for thread size adjustment in kconfig (Thx Arnd)
- Move call_on_stack to inline style (Thx Peter Zijlstra)
- Fixup fp chain broken (Thx Chen Zhongjin)
- Remove common entry modification, and fixup page_fault entry (Thx Peter Zijlstra)
- Treat some traps as nmi entry (Thx Peter Zijlstra)
v5:
https://lore.kernel.org/linux-riscv/[email protected]/
- Add riscv own stackleak patch instead of generic entry modification (by Mark Rutland)
- Add EXPERT dependency for THREAD_SIZE (by Arnd)
- Add EXPERT dependency for IRQ_STACK (by Sebastian, David Laight)
- Corrected __trap_section (by Peter Zijlstra)
- Add Tested-by (Yipeng Zou)
- Use CONFIG_SOFTIRQ_ON_OWN_STACK replace "#ifndef CONFIG_PREEMPT_RT"
- Fixup systrace_enter compile error
- Fixup exit_to_user_mode_prepare preempt_disable warning
V4:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup entry.S with "la" bug (by Conor.Dooley)
- Fixup missing noinstr bug (by Peter Zijlstra)
V3:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup CONFIG_COMPAT=n compile error
- Add THREAD_SIZE_ORDER config
- Optimize elf_kexec.c warning fixup
- Add static to irq_stack_ptr definition
V2:
https://lore.kernel.org/linux-riscv/[email protected]/
- Fixup compile error by include "riscv: ptrace: Remove duplicate operation"
- Fixup compile warning
- Add test repo link in cover letter
V1:
https://lore.kernel.org/linux-riscv/[email protected]/
Guo Ren (3):
riscv: stack: Support HAVE_IRQ_EXIT_ON_IRQ_STACK
riscv: stack: Support HAVE_SOFTIRQ_ON_OWN_STACK
riscv: stack: Add config of thread stack size
arch/riscv/Kconfig | 19 ++++++++ arch/riscv/include/asm/irq_stack.h | 32 +++++++++++++ arch/riscv/include/asm/thread_info.h | 14 ++---- arch/riscv/kernel/irq.c | 68 ++++++++++++++++++++++++++++ arch/riscv/kernel/traps.c | 35 +++++++++++++- 5 files changed, 155 insertions(+), 13 deletions(-) create mode 100644 arch/riscv/include/asm/irq_stack.h
Signature · 1 line
2.36.1
PATCH v12 1/3 riscv: stack: Support HAVE_IRQ_EXIT_ON_IRQ_STACK
From: Guo Ren <[email protected]>
Add independent irq stacks for percpu to prevent kernel stack overflows. It is also compatible with VMAP_STACK by arch_alloc_vmap_stack.
--- arch/riscv/Kconfig | 7 ++++++ arch/riscv/include/asm/irq_stack.h | 32 +++++++++++++++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/kernel/irq.c | 33 ++++++++++++++++++++++++++ arch/riscv/kernel/traps.c | 35 ++++++++++++++++++++++++++-- 5 files changed, 107 insertions(+), 2 deletions(-) create mode 100644 arch/riscv/include/asm/irq_stack.h
arch/riscv/Kconfig +7 -0
@@ -588,6 +588,13 @@ config FPU If you don't know what to do here, say Y. +config IRQ_STACKS + bool "Independent irq stacks" if EXPERT + default y + select HAVE_IRQ_EXIT_ON_IRQ_STACK + help + Add independent irq stacks for percpu to prevent kernel stack overflows. + endmenu # "Platform type" menu "Kernel features"
arch/riscv/include/asm/irq_stack.h new file +32 -0
@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_RISCV_IRQ_STACK_H +#define _ASM_RISCV_IRQ_STACK_H + +#include <linux/bug.h> +#include <linux/gfp.h> +#include <linux/kconfig.h> +#include <linux/vmalloc.h> +#include <linux/pgtable.h> +#include <asm/thread_info.h> + +DECLARE_PER_CPU(ulong *, irq_stack_ptr); + +#ifdef CONFIG_VMAP_STACK +/* +* To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd +* stacks need to have the same alignment. +*/ +static inline unsigned long *arch_alloc_vmap_stack (size_t stack_size,int node) +{ +void *p; + +BUILD_BUG_ON (!IS_ENABLED (CONFIG_VMAP_STACK )); + + p =__vmalloc_node (stack_size,THREAD_ALIGN ,THREADINFO_GFP , node, +__builtin_return_address (0 )); +return kasan_reset_tag (p); +} +#endif /* CONFIG_VMAP_STACK */ + +#endif /* _ASM_RISCV_IRQ_STACK_H */
arch/riscv/include/asm/thread_info.h +2 -0
@@ -40,6 +40,8 @@#define OVERFLOW_STACK_SIZE SZ_4K#define SHADOW_OVERFLOW_STACK_SIZE (1024) +#define IRQ_STACK_SIZE THREAD_SIZE +#ifndef __ASSEMBLY__extern long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE /sizeof (long )];
arch/riscv/kernel/irq.c +33 -0
@@ -28,6 +28,38 @@ struct fwnode_handle *riscv_get_intc_hwnode(void) }EXPORT_SYMBOL_GPL (riscv_get_intc_hwnode); +#ifdef CONFIG_IRQ_STACKS +#include <asm/irq_stack.h> + +DEFINE_PER_CPU(ulong *, irq_stack_ptr); + +#ifdef CONFIG_VMAP_STACK +static void init_irq_stacks (void ) +{ +int cpu; +ulong *p; + +for_each_possible_cpu (cpu) { + p =arch_alloc_vmap_stack (IRQ_STACK_SIZE ,cpu_to_node (cpu)); +per_cpu (irq_stack_ptr, cpu) = p; + } +} +#else +/* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */ +DEFINE_PER_CPU_ALIGNED (ulong [IRQ_STACK_SIZE /sizeof (ulong)], irq_stack); + +static void init_irq_stacks (void ) +{ +int cpu; + +for_each_possible_cpu (cpu) +per_cpu (irq_stack_ptr, cpu) =per_cpu (irq_stack, cpu); +} +#endif /* CONFIG_VMAP_STACK */ +#else +static void init_irq_stacks (void ) {} +#endif /* CONFIG_IRQ_STACKS */ +int arch_show_interrupts (struct seq_file *p,int prec) {show_ipi_stats (p, prec);
@@ -36,6 +68,7 @@ int arch_show_interrupts(struct seq_file *p, int prec)void __init init_IRQ (void ) { +init_irq_stacks ();irqchip_init ();if (!handle_arch_irq)panic ("No interrupt controller found." );
arch/riscv/kernel/traps.c +33 -2
@@ -27,6 +27,7 @@#include <asm/syscall.h> #include <asm/thread_info.h> #include <asm/vector.h> +#include <asm/irq_stack.h> int show_unhandled_signals =1 ;
@@ -327,16 +328,46 @@ asmlinkage __visible noinstr void do_page_fault(struct pt_regs *regs) }#endif -asmlinkage __visible noinstr void do_irq(struct pt_regs *regs) +static void noinstr handle_riscv_irq(struct pt_regs *regs) {struct pt_regs *old_regs; -irqentry_state_t state =irqentry_enter (regs);irq_enter_rcu (); old_regs =set_irq_regs (regs);handle_arch_irq (regs);set_irq_regs (old_regs);irq_exit_rcu (); +} + +asmlinkage void noinstr do_irq(struct pt_regs *regs) +{ + irqentry_state_t state = irqentry_enter(regs); +#ifdef CONFIG_IRQ_STACKS + if (on_thread_stack()) { + ulong *sp = per_cpu(irq_stack_ptr, smp_processor_id()) + + IRQ_STACK_SIZE/sizeof(ulong); + __asm__ __volatile( + "addi sp, sp, -"RISCV_SZPTR "\n" + REG_S" ra, (sp) \n" + "addi sp, sp, -"RISCV_SZPTR "\n" + REG_S" s0, (sp) \n" + "addi s0, sp, 2*"RISCV_SZPTR "\n" + "move sp, %[sp] \n" + "move a0, %[regs] \n" + "call handle_riscv_irq \n" + "addi sp, s0, -2*"RISCV_SZPTR"\n" + REG_L" s0, (sp) \n" + "addi sp, sp, "RISCV_SZPTR "\n" + REG_L" ra, (sp) \n" + "addi sp, sp, "RISCV_SZPTR "\n" + : + : [sp] "r" (sp), [regs] "r" (regs) + : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", + "t0", "t1", "t2", "t3", "t4", "t5", "t6", + "memory"); + } else +#endif + handle_riscv_irq(regs); irqentry_exit(regs, state); }
Signature · 7 lines
2.36.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
Re: [PATCH -next V12 1/3] riscv: stack: Support HAVE_IRQ_EXIT_ON_IRQ_STACK
Quoted 56 lines · On 29/05/2023 10:45, [email protected] wrote:
From: Guo Ren <[email protected]>
Add independent irq stacks for percpu to prevent kernel stack overflows. It is also compatible with
VMAP_STACKbyarch_alloc_vmap_stack.Tested-by Jisheng Zhang <[email protected]>Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>Cc Conor Dooley <[email protected]>--- arch/riscv/Kconfig | 7 ++++++ arch/riscv/include/asm/irq_stack.h | 32 +++++++++++++++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/kernel/irq.c | 33 ++++++++++++++++++++++++++ arch/riscv/kernel/traps.c | 35 ++++++++++++++++++++++++++-- 5 files changed, 107 insertions(+), 2 deletions(-)create mode 100644
arch/riscv/include/asm/irq_stack.harch/riscv/Kconfig +7
-0@@ -588,6 +588,13 @@ config FPU If you don't know what to do here, say Y. +config IRQ_STACKS + bool "Independent irq stacks" if EXPERT + default y + select HAVE_IRQ_EXIT_ON_IRQ_STACK + help + Add independent irq stacks for percpu to prevent kernel stack overflows. + endmenu # "Platform type" menu "Kernel features"arch/riscv/include/asm/irq_stack.h new file +14
-0@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_RISCV_IRQ_STACK_H +#define _ASM_RISCV_IRQ_STACK_H + +#include <linux/bug.h> +#include <linux/gfp.h> +#include <linux/kconfig.h> +#include <linux/vmalloc.h> +#include <linux/pgtable.h> +#include <asm/thread_info.h> + +DECLARE_PER_CPU(ulong *, irq_stack_ptr); +
arch/riscv/include/asm/irq_stack.h line 24
Quoted 10 lines
arch/riscv/include/asm/irq_stack.h new file +10
-0+#ifdef CONFIG_VMAP_STACK +/* +* To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd +* stacks need to have the same alignment. +*/ +static inline unsigned long *arch_alloc_vmap_stack (size_t stack_size,int node) +{ +void *p; + +BUILD_BUG_ON (!IS_ENABLED (CONFIG_VMAP_STACK ));
Hi Guo,
Since this function is already guarded with #ifdef CONFIG_VMAP_STACK, I guess this BUILD_BUG_ON() is unnecessary).
Clément
Re: [PATCH -next V12 1/3] riscv: stack: Support HAVE_IRQ_EXIT_ON_IRQ_STACK
Quoted 73 lines · Clément Léger
Quoted 56 lines · On 29/05/2023 10:45, [email protected] wrote:
From: Guo Ren <[email protected]>
Add independent irq stacks for percpu to prevent kernel stack overflows. It is also compatible with
VMAP_STACKbyarch_alloc_vmap_stack.Tested-by Jisheng Zhang <[email protected]>Signed-off-by Guo Ren <[email protected]>Signed-off-by Guo Ren <[email protected]>Cc Conor Dooley <[email protected]>--- arch/riscv/Kconfig | 7 ++++++ arch/riscv/include/asm/irq_stack.h | 32 +++++++++++++++++++++++++ arch/riscv/include/asm/thread_info.h | 2 ++ arch/riscv/kernel/irq.c | 33 ++++++++++++++++++++++++++ arch/riscv/kernel/traps.c | 35 ++++++++++++++++++++++++++-- 5 files changed, 107 insertions(+), 2 deletions(-)create mode 100644
arch/riscv/include/asm/irq_stack.harch/riscv/Kconfig +7
-0@@ -588,6 +588,13 @@ config FPU If you don't know what to do here, say Y. +config IRQ_STACKS + bool "Independent irq stacks" if EXPERT + default y + select HAVE_IRQ_EXIT_ON_IRQ_STACK + help + Add independent irq stacks for percpu to prevent kernel stack overflows. + endmenu # "Platform type" menu "Kernel features"arch/riscv/include/asm/irq_stack.h new file +14
-0@@ -0,0 +1,32 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_RISCV_IRQ_STACK_H +#define _ASM_RISCV_IRQ_STACK_H + +#include <linux/bug.h> +#include <linux/gfp.h> +#include <linux/kconfig.h> +#include <linux/vmalloc.h> +#include <linux/pgtable.h> +#include <asm/thread_info.h> + +DECLARE_PER_CPU(ulong *, irq_stack_ptr); +arch/riscv/include/asm/irq_stack.h line 24
Quoted 10 lines
arch/riscv/include/asm/irq_stack.h new file +10
-0+#ifdef CONFIG_VMAP_STACK +/* +* To ensure that VMAP'd stack overflow detection works correctly, all VMAP'd +* stacks need to have the same alignment. +*/ +static inline unsigned long *arch_alloc_vmap_stack (size_t stack_size,int node) +{ +void *p; + +BUILD_BUG_ON (!IS_ENABLED (CONFIG_VMAP_STACK ));Hi Guo,
Quoted 2 lines
Since this function is already guarded with #ifdef
CONFIG_VMAP_STACK, I guess thisBUILD_BUG_ON()is unnecessary).
Yes, my carelessness. I would remove it in the next version.
Quoted 2 lines
Clément
--
Best Regards
Guo Ren
PATCH v12 2/3 riscv: stack: Support HAVE_SOFTIRQ_ON_OWN_STACK
In reply to [email protected]From: Guo Ren <[email protected]>
Add the HAVE_SOFTIRQ_ON_OWN_STACK feature for the IRQ_STACKS config, and the irq and softirq use the same irq_stack of percpu.
--- arch/riscv/Kconfig | 6 ++++-- arch/riscv/kernel/irq.c | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 2 deletions(-)
arch/riscv/Kconfig +4 -2
@@ -589,11 +589,13 @@ config FPU If you don't know what to do here, say Y. config IRQ_STACKS - bool "Independent irq stacks" if EXPERT + bool "Independent irq & softirq stacks" if EXPERT default y select HAVE_IRQ_EXIT_ON_IRQ_STACK + select HAVE_SOFTIRQ_ON_OWN_STACK help - Add independent irq stacks for percpu to prevent kernel stack overflows. + Add independent irq & softirq stacks for percpu to prevent kernel stack + overflows. We may save some memory footprint by disabling IRQ_STACKS. endmenu # "Platform type"
arch/riscv/kernel/irq.c +35 -0
@@ -11,6 +11,9 @@#include <linux/module.h> #include <linux/seq_file.h> #include <asm/sbi.h> +#include <asm/smp.h> +#include <asm/softirq_stack.h> +#include <asm/stacktrace.h> static struct fwnode_handle *(*__get_intc_node)(void );
@@ -56,6 +59,38 @@ static void init_irq_stacks(void)per_cpu (irq_stack_ptr, cpu) =per_cpu (irq_stack, cpu); }#endif /* CONFIG_VMAP_STACK */ + +#ifdef CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK +void do_softirq_own_stack (void ) +{ +#ifdef CONFIG_IRQ_STACKS +if (on_thread_stack ()) { +ulong *sp =per_cpu (irq_stack_ptr,smp_processor_id ()) + +IRQ_STACK_SIZE /sizeof (ulong); + __asm__ __volatile( +"addi sp, sp, -" RISCV_SZPTR "\n" +REG_S " ra, (sp) \n" +"addi sp, sp, -" RISCV_SZPTR "\n" +REG_S " s0, (sp) \n" +"addi s0, sp, 2*" RISCV_SZPTR "\n" +"move sp, %[sp] \n" +"call __do_softirq \n" +"addi sp, s0, -2*" RISCV_SZPTR "\n" +REG_L " s0, (sp) \n" +"addi sp, sp, " RISCV_SZPTR "\n" +REG_L " ra, (sp) \n" +"addi sp, sp, " RISCV_SZPTR "\n" + : + : [sp] "r" (sp) + : "a0", "a1", "a2", "a3", "a4", "a5", "a6", "a7", +"t0" ,"t1" ,"t2" ,"t3" ,"t4" ,"t5" ,"t6" , +"memory" ); + } else +#endif +__do_softirq (); +} +#endif /* CONFIG_HAVE_SOFTIRQ_ON_OWN_STACK */ +#else static void init_irq_stacks (void ) {}#endif /* CONFIG_IRQ_STACKS */
Signature · 7 lines
2.36.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv
PATCH v12 3/3 riscv: stack: Add config of thread stack size
In reply to [email protected]From: Guo Ren <[email protected]>
The commit 0cac21b02ba5 ("riscv: use 16KB kernel stack on 64-bit") increases the thread size mandatory, but some scenarios, such as D1 with a small memory footprint, would suffer from that. After independent irq stack support, let's give users a choice to determine their custom stack size.
--- arch/riscv/Kconfig | 10 ++++++++++ arch/riscv/include/asm/thread_info.h | 12 +----------- 2 files changed, 11 insertions(+), 11 deletions(-)
arch/riscv/Kconfig +10 -0
@@ -597,6 +597,16 @@ config IRQ_STACKS Add independent irq & softirq stacks for percpu to prevent kernel stack overflows. We may save some memory footprint by disabling IRQ_STACKS. +config THREAD_SIZE_ORDER + int "Kernel stack size (in power-of-two numbers of page size)" if VMAP_STACK && EXPERT + range 0 4 + default 1 if 32BIT && !KASAN + default 3 if 64BIT && KASAN + default 2 + help + Specify the Pages of thread stack size (from 4KB to 64KB), which also + affects irq stack size, which is equal to thread stack size. + endmenu # "Platform type" menu "Kernel features"
arch/riscv/include/asm/thread_info.h +1 -11
@@ -11,18 +11,8 @@#include <asm/page.h> #include <linux/const.h> -#ifdef CONFIG_KASAN -#define KASAN_STACK_ORDER 1 -#else -#define KASAN_STACK_ORDER 0 -#endif -/* thread information allocation */ -#ifdef CONFIG_64BIT -#define THREAD_SIZE_ORDER (2 + KASAN_STACK_ORDER) -#else -#define THREAD_SIZE_ORDER (1 + KASAN_STACK_ORDER) -#endif +#define THREAD_SIZE_ORDER CONFIG_THREAD_SIZE_ORDER#define THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER)/*
Signature · 7 lines
2.36.1 _______________________________________________ linux-riscv mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-riscv