riscv: add irq stack support
PATCH v2 1/1 riscv: add irq stack support
Currently, IRQs are still handled on the kernel stack of the current task on riscv platforms. If the task has a deep call stack at the time of interrupt, and handling the interrupt also requires a deep stack, it's possible to see stack overflow.
Before this patch, the stack_max_size of a v5.17-rc1 kernel running on a lichee RV board gave:
~ # cat /sys/kernel/debug/tracing/stack_max_size
3736
After this patch,
~ # cat /sys/kernel/debug/tracing/stack_max_size
3176
We reduce the max kernel stack usage by 560 bytes!
From another side, after this patch, it's possible to reduce the THREAD_SIZE to 8KB for RV64 platforms. This is especially useful for those systems with small memory size, e.g the Allwinner D1S platform which is RV64 but only has 64MB DDR.
---
since v1:
- add __ro_after_init to the irq_stack[] array.
arch/riscv/include/asm/thread_info.h | 1 + arch/riscv/kernel/asm-offsets.c | 2 ++ arch/riscv/kernel/entry.S | 33 +++++++++++++++++++++++++--- arch/riscv/kernel/irq.c | 16 ++++++++++++++ 4 files changed, 49 insertions(+), 3 deletions(-)
arch/riscv/include/asm/thread_info.h +1 -0
@@ -19,6 +19,7 @@#endif #define THREAD_SIZE (PAGE_SIZE << THREAD_SIZE_ORDER) +#define IRQ_STACK_SIZE THREAD_SIZE/* * By aligning VMAP'd stacks to 2 * THREAD_SIZE, we can detect overflow by * checking sp & (1 << THREAD_SHIFT), which we can do cheaply in the entry
arch/riscv/kernel/asm-offsets.c +2 -0
@@ -36,6 +36,8 @@ void asm_offsets(void)OFFSET (TASK_TI_PREEMPT_COUNT , task_struct, thread_info.preempt_count);OFFSET (TASK_TI_KERNEL_SP , task_struct, thread_info.kernel_sp);OFFSET (TASK_TI_USER_SP , task_struct, thread_info.user_sp); +OFFSET (TASK_TI_CPU , task_struct, thread_info.cpu); +OFFSET (TASK_STACK , task_struct, stack);OFFSET (TASK_THREAD_F0 , task_struct, thread.fstate.f[0 ]);OFFSET (TASK_THREAD_F1 , task_struct, thread.fstate.f[1 ]);
arch/riscv/kernel/entry.S +30 -3
@@ -126,12 +126,39 @@ skip_context_tracking: */ bge s4, zero, 1f - la ra, ret_from_exception + /* preserve the sp */ + move s0, sp - /* Handle interrupts */ move a0, sp /* pt_regs */ + + /* + * Compare sp with the base of the task stack. + * If the top ~(THREAD_SIZE - 1) bits match, we are on a task stack, + * and should switch to the irq stack. + */ + REG_L t0, TASK_STACK(tp) + xor t0, t0, s0 + li t1, ~(THREAD_SIZE - 1) + and t0, t0, t1 + bnez t0, 2f + + la t1, irq_stack + REG_L t2, TASK_TI_CPU(tp) + slli t2, t2, RISCV_LGPTR + add t1, t1, t2 + REG_L t2, 0(t1) + li t1, IRQ_STACK_SIZE + /* switch to the irq stack */ + add sp, t2, t1 + +2: + /* Handle interrupts */ la a1, generic_handle_arch_irq - jr a1 + jalr a1 + + /* Restore sp */ + move sp, s0 + j ret_from_exception 1: /* * Exceptions run with interrupts enabled or disabled depending on the
arch/riscv/kernel/irq.c +16 -0
@@ -10,6 +10,8 @@#include <linux/seq_file.h> #include <asm/smp.h> +void *irq_stack[NR_CPUS ]__ro_after_init ; +int arch_show_interrupts (struct seq_file *p,int prec) {show_ipi_stats (p, prec);
@@ -18,7 +20,21 @@ int arch_show_interrupts(struct seq_file *p, int prec)void __init init_IRQ (void ) { +int cpu; +irqchip_init ();if (!handle_arch_irq)panic ("No interrupt controller found." ); + +for_each_possible_cpu (cpu) { +#ifdef CONFIG_VMAP_STACK +void *s =__vmalloc_node (IRQ_STACK_SIZE ,THREAD_ALIGN , +THREADINFO_GFP ,cpu_to_node (cpu), +__builtin_return_address (0 )); +#else +void *s = (void *)__get_free_pages (GFP_KERNEL ,get_order (IRQ_STACK_SIZE )); +#endif + + irq_stack[cpu] = s; + } }
Signature · 1 line
2.34.1
From: Jisheng Zhang
Quoted 6 lines
Sent: 07 March 2022 14:08
Currently, IRQs are still handled on the kernel stack of the current task on riscv platforms. If the task has a deep call stack at the time of interrupt, and handling the interrupt also requires a deep stack, it's possible to see stack overflow.
...
I'd have thought that a single page is (probably) enough for the IRQ stack.
Certainly its sizing isn't really related to the normal thread stack size.
Quoted 4 lines
From another side, after this patch, it's possible to reduce the
THREAD_SIZEto 8KB for RV64 platforms. This is especially useful for those systems with small memory size, e.g the Allwinner D1S platform which is RV64 but only has 64MB DDR.
Are you sure?
Is the stack use likely to be very much less than that of x86-64? The real problem isn't the stack use of the test you are doing, but the horrid worst case stack of some path that has multiple 1k+ buffers on stack.
Apart from compiler fubar (which usually hit KASAN) that stack is actually likely to be architecture independent. (The difference between 32bit and 64bit is also likely to be relatively small - unless there are on-stack arrays of 'long'.)
For VMAP stacks is there a 'guard' KVA page allocated below all of the stacks?
64bit systems should have lots of KVA so this shouldn't be a problem.
Then stack overruns will fault and panic rather than trashing another data area - which is really hard to debug.
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Quoted 15 lines · Jisheng Zhang
Currently, IRQs are still handled on the kernel stack of the current task on riscv platforms. If the task has a deep call stack at the time of interrupt, and handling the interrupt also requires a deep stack, it's possible to see stack overflow.
Before this patch, the
stack_max_sizeof a v5.17-rc1 kernel running on a lichee RV board gave:
~ # cat/sys/kernel/debug/tracing/stack_max_size
3736After this patch,
~ # cat/sys/kernel/debug/tracing/stack_max_size
3176
Quoted 8 lines
We reduce the max kernel stack usage by 560 bytes!
From another side, after this patch, it's possible to reduce the
THREAD_SIZEto 8KB for RV64 platforms. This is especially useful for those systems with small memory size, e.g the Allwinner D1S platform which is RV64 but only has 64MB DDR.Signed-off-by Jisheng Zhang <[email protected]>
Very nice!
Quoted 26 lines
arch/riscv/kernel/entry.S +14
-2@@ -126,12 +126,39 @@ skip_context_tracking: */ bge s4, zero, 1f - la ra, ret_from_exception + /* preserve the sp */ + move s0, sp - /* Handle interrupts */ move a0, sp /* pt_regs */ + + /* + * Compare sp with the base of the task stack. + * If the top ~(THREAD_SIZE - 1) bits match, we are on a task stack, + * and should switch to the irq stack. + */ + REG_L t0, TASK_STACK(tp) + xor t0, t0, s0 + li t1, ~(THREAD_SIZE - 1) + and t0, t0, t1 + bnez t0, 2f +
arch/riscv/kernel/entry.S skip_context_tracking: · line 154
Quoted 10 lines
arch/riscv/kernel/entry.S +10
-0+ la t1, irq_stack + REG_L t2, TASK_TI_CPU(tp) + slli t2, t2, RISCV_LGPTR + add t1, t1, t2 + REG_L t2, 0(t1) + li t1, IRQ_STACK_SIZE + /* switch to the irq stack */ + add sp, t2, t1 + +2:
What is the benefit of doing this in assembler? Is it measurably faster?
I see that arm64 does the same thing in C code, and it would be best to have a common implementation for doing this, in terms of maintainability.
arch/riscv/kernel/entry.S skip_context_tracking: · line 163
Quoted 9 lines
arch/riscv/kernel/entry.S +9
-0+ + for_each_possible_cpu(cpu) { +#ifdef CONFIG_VMAP_STACK + void *s = __vmalloc_node(IRQ_STACK_SIZE, THREAD_ALIGN, + THREADINFO_GFP, cpu_to_node(cpu), + __builtin_return_address(0)); +#else + void *s = (void *)__get_free_pages(GFP_KERNEL, get_order(IRQ_STACK_SIZE)); +#endif
On a related topic: is there a reason to still keep the non-VMAP_STACK code path around? I see that it currently is optional for 64-bit with MMU, but not available otherwise. The benefits should still outweigh the downside (virtual address space usage mainly) on 32-bit, especially when this allows a common implementation. Not sure about NOMMU, but I would guess that it's not a big issue to use the same code there as well, since nommu vmalloc just turns into a kmalloc as well.
Arnd
Quoted 66 lines · Arnd Bergmann
Quoted 15 lines · Jisheng Zhang
Currently, IRQs are still handled on the kernel stack of the current task on riscv platforms. If the task has a deep call stack at the time of interrupt, and handling the interrupt also requires a deep stack, it's possible to see stack overflow.
Before this patch, the
stack_max_sizeof a v5.17-rc1 kernel running on a lichee RV board gave:
~ # cat/sys/kernel/debug/tracing/stack_max_size
3736After this patch,
~ # cat/sys/kernel/debug/tracing/stack_max_size
3176Quoted 8 lines
We reduce the max kernel stack usage by 560 bytes!
From another side, after this patch, it's possible to reduce the
THREAD_SIZEto 8KB for RV64 platforms. This is especially useful for those systems with small memory size, e.g the Allwinner D1S platform which is RV64 but only has 64MB DDR.Signed-off-by Jisheng Zhang <[email protected]>Very nice!
Quoted 26 lines
arch/riscv/kernel/entry.S +14
-2@@ -126,12 +126,39 @@ skip_context_tracking: */ bge s4, zero, 1f - la ra, ret_from_exception + /* preserve the sp */ + move s0, sp - /* Handle interrupts */ move a0, sp /* pt_regs */ + + /* + * Compare sp with the base of the task stack. + * If the top ~(THREAD_SIZE - 1) bits match, we are on a task stack, + * and should switch to the irq stack. + */ + REG_L t0, TASK_STACK(tp) + xor t0, t0, s0 + li t1, ~(THREAD_SIZE - 1) + and t0, t0, t1 + bnez t0, 2f +arch/riscv/kernel/entry.S skip_context_tracking: · line 154
Quoted 10 lines
arch/riscv/kernel/entry.S +10
-0+ la t1, irq_stack + REG_L t2, TASK_TI_CPU(tp) + slli t2, t2, RISCV_LGPTR + add t1, t1, t2 + REG_L t2, 0(t1) + li t1, IRQ_STACK_SIZE + /* switch to the irq stack */ + add sp, t2, t1 + +2:What is the benefit of doing this in assembler? Is it measurably faster?
Quoted 3 lines
I see that arm64 does the same thing in C code, and it would be best to have a common implementation for doing this, in terms of maintainability.
Hi Arnd,
Sorry for delay. The assembler code is mainly to cal the stack ptr then change the SP to use the stack, which equals to arm64 call_on_irq_stack() which is implemented in assembler too.
Quoted 11 lines
arch/riscv/kernel/entry.S skip_context_tracking: · line 163
Quoted 9 lines
arch/riscv/kernel/entry.S +9
-0+ + for_each_possible_cpu(cpu) { +#ifdef CONFIG_VMAP_STACK + void *s = __vmalloc_node(IRQ_STACK_SIZE, THREAD_ALIGN, + THREADINFO_GFP, cpu_to_node(cpu), + __builtin_return_address(0)); +#else + void *s = (void *)__get_free_pages(GFP_KERNEL, get_order(IRQ_STACK_SIZE)); +#endifOn a related topic: is there a reason to still keep the non-VMAP_STACK
irq stack is 16KB on RV64 now, vmalloc doesn't gurantee physical continuous pages, I want to keep the stack physical continuous characteristic for !VMAP_STACK case.
Thanks
Quoted 7 lines
code path around? I see that it currently is optional for 64-bit with MMU, but not available otherwise. The benefits should still outweigh the downside (virtual address space usage mainly) on 32-bit, especially when this allows a common implementation. Not sure about NOMMU, but I would guess that it's not a big issue to use the same code there as well, since nommu vmalloc just turns into a kmalloc as well.
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 18 lines · David Laight
From: Jisheng Zhang
Quoted 6 lines
Sent: 07 March 2022 14:08
Currently, IRQs are still handled on the kernel stack of the current task on riscv platforms. If the task has a deep call stack at the time of interrupt, and handling the interrupt also requires a deep stack, it's possible to see stack overflow....
I'd have thought that a single page is (probably) enough for the IRQ stack.
Certainly its sizing isn't really related to the normal thread stack size.Quoted 4 lines
From another side, after this patch, it's possible to reduce the
THREAD_SIZEto 8KB for RV64 platforms. This is especially useful for those systems with small memory size, e.g the Allwinner D1S platform which is RV64 but only has 64MB DDR.
Quoted 5 lines
Are you sure?
Is the stack use likely to be very much less than that of x86-64? The real problem isn't the stack use of the test you are doing, but the horrid worst case stack of some path that has multiple 1k+ buffers on stack.
Hi David,
Sorry for delay. I think you are right, at least I should not put the confusing "it's possible to reduce the THREAD_SIZE to 8KB for RV64 platforms..." in the commit msg. For one thing, the 8KB IRQ stack isn't available in the mainline w/o a small patch; For another, I only do tests on Allwinner D1 platform. So I remove the section in V3's commit msg.
Thanks
Quoted 13 lines
Apart from compiler fubar (which usually hit KASAN) that stack is actually likely to be architecture independent. (The difference between 32bit and 64bit is also likely to be relatively small - unless there are on-stack arrays of 'long'.)
For VMAP stacks is there a 'guard' KVA page allocated below all of the stacks?
64bit systems should have lots of KVA so this shouldn't be a problem.
Then stack overruns will fault and panic rather than trashing another data area - which is really hard to debug.
Quoted 6 lines
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
_______________________________________________
linux-riscv mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-riscv
Quoted 6 lines · Jisheng Zhang
Quoted 5 lines · Arnd Bergmann
Quoted 1 line · Jisheng Zhang
+2:
What is the benefit of doing this in assembler? Is it measurably faster?
Quoted 7 lines
Quoted 3 lines
I see that arm64 does the same thing in C code, and it would be best to have a common implementation for doing this, in terms of maintainability.
Sorry for delay. The assembler code is mainly to cal the stack ptr then change the SP to use the stack, which equals to arm64
call_on_irq_stack()which is implemented in assembler too.
I understand that you need to be in asm code to switch the stack, it just felt that the arm64 method is a bit easier to debug here.
I suppose being able to keep using generic_handle_arch_irq() is also beneficial, so it doesn't make much difference either way.
Quoted 11 lines
Quoted 10 lines
arch/riscv/kernel/irq.c int arch_show_interrupts(struct seq_file *p, int prec) · line 36
Quoted 8 lines
arch/riscv/kernel/irq.c +8
-0+for_each_possible_cpu (cpu) { +#ifdef CONFIG_VMAP_STACK +void *s =__vmalloc_node (IRQ_STACK_SIZE ,THREAD_ALIGN , +THREADINFO_GFP ,cpu_to_node (cpu), +__builtin_return_address (0 )); +#else +void *s = (void *)__get_free_pages (GFP_KERNEL ,get_order (IRQ_STACK_SIZE )); +#endif On a related topic: is there a reason to still keep the non-VMAP_STACK
Quoted 3 lines
irq stack is 16KB on RV64 now, vmalloc doesn't gurantee physical continuous pages, I want to keep the stack physical continuous characteristic for !VMAP_STACK case.
I don't understand. What is the benefit of having a physically continuous stack? If this is required for something, you could still get that with a VMAP stack by using alloc_pages() to allocate the stack and them using vmap() to put it into the vmalloc range with appropriate guard pages.
I think we really want to avoid the case of missing guard pages around the stack, and eliminate the part where the stack is in the linear map.
Arnd
Quoted 15 lines · Arnd Bergmann
Quoted 11 lines
Quoted 10 lines
arch/riscv/kernel/irq.c int arch_show_interrupts(struct seq_file *p, int prec) · line 36
Quoted 8 lines
arch/riscv/kernel/irq.c +8
-0+for_each_possible_cpu (cpu) { +#ifdef CONFIG_VMAP_STACK +void *s =__vmalloc_node (IRQ_STACK_SIZE ,THREAD_ALIGN , +THREADINFO_GFP ,cpu_to_node (cpu), +__builtin_return_address (0 )); +#else +void *s = (void *)__get_free_pages (GFP_KERNEL ,get_order (IRQ_STACK_SIZE )); +#endif On a related topic: is there a reason to still keep the non-VMAP_STACK
Quoted 3 lines
irq stack is 16KB on RV64 now, vmalloc doesn't gurantee physical continuous pages, I want to keep the stack physical continuous characteristic for !VMAP_STACK case.
Quoted 7 lines
I don't understand. What is the benefit of having a physically continuous stack? If this is required for something, you could still get that with a VMAP stack by using
alloc_pages()to allocate the stack and them usingvmap()to put it into the vmalloc range with appropriate guard pages.I think we really want to avoid the case of missing guard pages around the stack, and eliminate the part where the stack is in the linear map.
I was actually confused here and mixed up a few things: I thought this was about whether to use vmap stacks unconditionally, and this is in fact not even an architecture specific decision, it's a global option as you are probably aware.
Since one can still turn off VMAP_STACK for normal thread stacks, it doesn't make much of a difference whether one can do the same for IRQ stacks. Please just ignore what I said above. I see you already sent a modified v3, and I think either way is fine, feel free to revert back to this method if it makes life easier.
Arnd