ARM: add support for IRQ stacks
cover ARM: add support for IRQ stacks
Compared to user space, the kernel's task stacks are tiny and inflexible, as we don't grow them dynamically using demand paging. This is the reason we tend to obsess about functions with disproportionately large stack frames, given that it is hard to predict statically how calls to those functions may combine at runtime, and exhaust the stack and crash the kernel.
This becomes even less predictable when taking interrupt handling into account, as their handlers are normally executed on the stack of the task that was interrupted, regardless of how deep its call stack was at the time of the interruption. To decouple these, and reduce the risk of hitting a pathological worst case where IRQ handling and the task below it happen to exhaust the available stack space when combined, we can switch to a different stack when taking interrupts, similar to how this is done already on some other architectures. This series implements this approach for ARM.
Note that a good chunk of the changes below are related to supporting non-contiguous call stacks, which is also relevant in the context of vmap'ed stacks, which use a separate overflow stack to handle stack overflows. The changes preserve all functionality related to walking the call stack and dumping exception stacks and register contents, with one small exception: when using CONFIG_UNWINDER_FRAME_POINTER=y with Clang, (which does not record the SP value at function entry), we cannot dump the exception stack passed by the IRQ entry code if a stack switch occurred. All other scenarios have been tested and should work as before.
Patch #1 fixes a bug that broke the exception stack dumping code.
Patch #2 removes some code that I spotted that is no longer used.
Patch #3 introduces a pair of macros that will be used later in the series to emit the optimal indirect call sequence for older and newer cores.
Patch #4 tweaks the IRQ asm entry point to generate better code for v7 CPUs.
Patch #5 updates the unwind info based unwinder so it can deal with call stacks that are non-contiguous.
Patch #6 exports dump_mem() to other compilation units so the ARM unwinder can cal it directly. This is needed by the next patch.
Patch #7 refactors the ARM unwinder to dump the exception stack from a context where it can figure out if it lives on the current stack or on the task stack.
Patch #8 implements the actual IRQ stacks support, by allocating one for each CPU, and adding the code to switch to it in the IRQ entry path. It also contains some related changes to allow the frame pointer based unwinder to deal with the new situation.
Patch #9 modifies call_with_stack() so both the frame pointer unwinder as well as the ARM unwinder know how to deal with it.
Patch #10 adds the IRQ stack switching for softIRQ handling initiated from task context.
The patches are based on my arm32-ti-in-task-v5 branch [0], which is a prerequisite for these changes, given that we can no longer rely on thread_info being accessible by masking the stack pointer when we are jumping between stacks.
[0] https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=arm32-ti-in-task-v5
Ard Biesheuvel (10):
ARM: fix incorrect use of get_kernel_nofault()
ARM: remove some dead code
ARM: assembler: introduce bl_r and bl_m macros
ARM: optimize indirect call to handle_arch_irq for v7 cores
ARM: unwind: support unwinding across multiple stacks
ARM: export dump_mem() to other objects
ARM: unwind: dump exception stack from calling frame
ARM: implement IRQ stacks
ARM: call_with_stack: add unwind support
ARM: run softirqs on the per-CPU IRQ stack
arch/arm/Kconfig | 6 +++ arch/arm/include/asm/assembler.h | 37 ++++++++++++++++ arch/arm/include/asm/entry-macro-multi.S | 24 ---------- arch/arm/include/asm/smp.h | 5 --- arch/arm/include/asm/stacktrace.h | 13 ++++++ arch/arm/kernel/entry-armv.S | 46 +++++++++++++++++--- arch/arm/kernel/irq.c | 31 +++++++++++++ arch/arm/kernel/smp.c | 5 --- arch/arm/kernel/traps.c | 27 +++++++++--- arch/arm/kernel/unwind.c | 33 +++++++++----- arch/arm/lib/backtrace-clang.S | 8 ++++ arch/arm/lib/backtrace.S | 8 ++++ arch/arm/lib/call_with_stack.S | 44 ++++++++++++++++--- 13 files changed, 223 insertions(+), 64 deletions(-)
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 1/10 ARM: fix incorrect use of get_kernel_nofault()
Commit 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead of set_fs()") replaced an occurrence of __get_user() with get_kernel_nofault(), but inverted the sense of the conditional in the process, resulting in no values to be printed at all.
I.e., every exception stack now looks like this:
Exception stack(0xc18d1fb0 to 0xc18d1ff8)
1fa0: ???????? ???????? ???????? ????????
1fc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? 1fe0: ???????? ???????? ???????? ???????? ???????? ????????
which is rather unhelpful.
--- arch/arm/kernel/traps.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm/kernel/traps.c +1 -1
@@ -136,7 +136,7 @@ static void dump_mem(const char *lvl, const char *str, unsigned long bottom,for (p = first, i =0 ; i <8 && p < top; i++, p +=4 ) {if (p >= bottom && p < top) {unsigned long val; -if (get_kernel_nofault (val, (unsigned long *)p)) +if (!get_kernel_nofault (val, (unsigned long *)p))sprintf (str + i *9 ," %08lx" , val);else sprintf (str + i *9 ," ????????" );
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH 01/10] ARM: fix incorrect use of get_kernel_nofault()
Quoted 9 lines · Ard Biesheuvel
Commit
344179fc7ef4("ARM: 9106/1: traps: useget_kernel_nofaultinstead ofset_fs()") replaced an occurrence of__get_user()withget_kernel_nofault(), but inverted the sense of the conditional in the process, resulting in no values to be printed at all.I.e., every exception stack now looks like this:
Exception stack(0xc18d1fb0 to 0xc18d1ff8)
Quoted 8 lines
1fa0: ???????? ???????? ???????? ????????
1fc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? 1fe0: ???????? ???????? ???????? ???????? ???????? ????????which is rather unhelpful.
Fixes 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead of set_fs()")Signed-off-by Ard Biesheuvel <[email protected]>
Thanks for catching this!
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Russell King (Oracle)
maintainerRe: [PATCH 01/10] ARM: fix incorrect use of get_kernel_nofault()
In reply to Ard BiesheuvelQuoted 8 lines · Ard Biesheuvel
Commit
344179fc7ef4("ARM: 9106/1: traps: useget_kernel_nofaultinstead ofset_fs()") replaced an occurrence of__get_user()withget_kernel_nofault(), but inverted the sense of the conditional in the process, resulting in no values to be printed at all.I.e., every exception stack now looks like this:
Exception stack(0xc18d1fb0 to 0xc18d1ff8)
Quoted 8 lines
1fa0: ???????? ???????? ???????? ????????
1fc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? 1fe0: ???????? ???????? ???????? ???????? ???????? ????????which is rather unhelpful.
Fixes 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead of set_fs()")Signed-off-by Ard Biesheuvel <[email protected]>
This is definitely a bug fix to one of the previous series, and should be independent of the rest of the series.
Please put it in the patch system, thanks.
Signature · 7 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH 01/10] ARM: fix incorrect use of get_kernel_nofault()
Quoted 16 lines · Russell King (Oracle)
Quoted 14 lines · Ard Biesheuvel
Commit
344179fc7ef4("ARM: 9106/1: traps: useget_kernel_nofaultinstead ofset_fs()") replaced an occurrence of__get_user()withget_kernel_nofault(), but inverted the sense of the conditional in the process, resulting in no values to be printed at all.I.e., every exception stack now looks like this:
Exception stack(0xc18d1fb0 to 0xc18d1ff8)1fa0: ???????? ???????? ???????? ????????
1fc0: ???????? ???????? ???????? ???????? ???????? ???????? ???????? ???????? 1fe0: ???????? ???????? ???????? ???????? ???????? ????????which is rather unhelpful.
Quoted 8 lines
Quoted 2 lines
Fixes 344179fc7ef4 ("ARM: 9106/1: traps: use get_kernel_nofault instead of set_fs()")Signed-off-by Ard Biesheuvel <[email protected]>This is definitely a bug fix to one of the previous series, and should be independent of the rest of the series.
Please put it in the patch system, thanks.
Queued up as 9125/1
Signature · 6 lines
Ard. _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 2/10 ARM: remove some dead code
In reply to Ard BiesheuvelThis code appears to be no longer used so let's get rid of it.
--- arch/arm/include/asm/entry-macro-multi.S | 24 -------------------- arch/arm/include/asm/smp.h | 5 ---- arch/arm/kernel/smp.c | 5 ---- 3 files changed, 34 deletions(-)
arch/arm/include/asm/entry-macro-multi.S +0 -24
@@ -13,28 +13,4 @@ @ badrne lr, 1b bne asm_do_IRQ - -#ifdef CONFIG_SMP - /* - * XXX - * - * this macro assumes that irqstat (r2) and base (r6) are - * preserved from get_irqnr_and_base above - */ - ALT_SMP(test_for_ipi r0, r2, r6, lr) - ALT_UP_B(9997f) - movne r1, sp - badrne lr, 1b - bne do_IPI -#endif -9997: - .endm - - .macro arch_irq_handler, symbol_name - .align 5 - .global \symbol_name -\symbol_name: - mov r8, lr - arch_irq_handler_default - ret r8 .endm
arch/arm/include/asm/smp.h +0 -5
@@ -24,11 +24,6 @@ struct seq_file;*/ extern void show_ipi_list (struct seq_file *,int ); -/* -* Called from assembly code, this handles an IPI. -*/ -asmlinkage void do_IPI(int ipinr, struct pt_regs *regs); -/* * Called from C code, this handles an IPI. */
arch/arm/kernel/smp.c +0 -5
@@ -631,11 +631,6 @@ static void ipi_complete(unsigned int cpu)/* * Main handler for inter-processor interrupts */ -asmlinkage void __exception_irq_entry do_IPI(int ipinr, struct pt_regs *regs) -{ -handle_IPI (ipinr, regs); -} -static void do_handle_IPI (int ipinr) {unsigned int cpu =smp_processor_id ();
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 3/10 ARM: assembler: introduce bl_r and bl_m macros
In reply to Ard BiesheuvelAdd some macros that abstract the difference between the ways indirect calls are performed on older and newer ARM architecture revisions.
The main difference is to prefer blx instructions over explicit LR assignments when possible, as these tend to confuse the prediction logic in out-of-order cores when speculating across a function return.
--- arch/arm/include/asm/assembler.h | 37 ++++++++++++++++++++ 1 file changed, 37 insertions(+)
arch/arm/include/asm/assembler.h +37 -0
@@ -624,4 +624,41 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) .endif .endm + /* + * bl_r - branch and link to register + * + * @dst: target to branch to + * @c: conditional opcode suffix + */ + .macro bl_r, dst:req, c + .if __LINUX_ARM_ARCH__ < 6 + mov\c lr, pc + mov\c pc, \dst + .else + blx\c \dst + .endif + .endm + +/* +* bl_m - branch and link to address held in memory +* +* @dst: memory reference to load the branch target from +* +* This uses the same syntax as ldr instructions, including things like +* register writeback (where it makes sense). For example, +* +* bl_m [ip, #<offset>] +* +* Note that @dst should not reference lr as it may be clobbered early. +*/ + .macro bl_m, dst:vararg + .if __LINUX_ARM_ARCH__ < 6 + mov lr, pc + ldr pc, \dst + .else + ldr lr, \dst + blx lr + .endif + .endm + #endif /* __ASM_ASSEMBLER_H__ */
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 4/10 ARM: optimize indirect call to handle_arch_irq for v7 cores
In reply to Ard BiesheuvelThe current irq_handler macro uses a literal load followed by an explicit assignment of lr before an indirect call, both of which are sub-optimal on recent ARM cores. Replace it by a mov_l call, which will evaluate to a movw/movt pair on v7 cores, avoiding one of the two loads, followed by a call to a newly introduced 'bl_m' macro, which will evaluate to a blx instruction on cores that support it. (Setting lr explicitly rather than via a bl or blx instruction confuses the return address prediction that modern out-of-order cores use to speculatively perform the function return.)
--- arch/arm/kernel/entry-armv.S | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
arch/arm/kernel/entry-armv.S +2 -4
@@ -38,14 +38,12 @@ */ .macro irq_handler #ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER - ldr r1, =handle_arch_irq + mov_l r1, handle_arch_irq mov r0, sp - badr lr, 9997f - ldr pc, [r1] + bl_m [r1] #else arch_irq_handler_default #endif -9997: .endm .macro pabt_helper
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 5/10 ARM: unwind: support unwinding across multiple stacks
In reply to Ard BiesheuvelImplement support in the unwinder for dealing with multiple stacks. This will be needed once we add support for IRQ stacks, or for the overflow stack used by the vmap'ed stacks code.
This involves tracking the unwind opcodes that either update the virtual stack pointer from another virtual register, or perform an explicit subtract on the virtual stack pointer, and updating the low and high bounds that we use to sanitize the stack pointer accordingly.
--- arch/arm/kernel/unwind.c | 25 +++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-)
arch/arm/kernel/unwind.c +16 -9
@@ -52,6 +52,7 @@ EXPORT_SYMBOL(__aeabi_unwind_cpp_pr2);struct unwind_ctrl_block {unsigned long vrs[16 ];/* virtual register set */ const unsigned long *insn;/* pointer to the current instructions word */ +unsigned long sp_low;/* lowest value of sp allowed */ unsigned long sp_high;/* highest value of sp allowed */ /* * 1 : check for stack overflow for each register pop.
@@ -256,8 +257,12 @@ static int unwind_exec_pop_subset_r4_to_r13(struct unwind_ctrl_block *ctrl, mask >>=1 ; reg++; } -if (!load_sp) +if (!load_sp) { ctrl->vrs[SP ] = (unsigned long )vsp; + }else { + ctrl->sp_low = ctrl->vrs[SP ]; + ctrl->sp_high =ALIGN (ctrl->sp_low,THREAD_SIZE ); + }return URC_OK ; }
@@ -313,9 +318,10 @@ static int unwind_exec_insn(struct unwind_ctrl_block *ctrl)if ((insn &0xc0 ) ==0x00 ) ctrl->vrs[SP ] += ((insn &0x3f ) <<2 ) +4 ; -else if ((insn &0xc0 ) ==0x40 ) +else if ((insn &0xc0 ) ==0x40 ) { ctrl->vrs[SP ] -= ((insn &0x3f ) <<2 ) +4 ; -else if ((insn &0xf0 ) ==0x80 ) { + ctrl->sp_low = ctrl->vrs[SP ]; + }else if ((insn &0xf0 ) ==0x80 ) {unsigned long mask; insn = (insn <<8 ) |unwind_get_byte (ctrl);
@@ -330,9 +336,11 @@ static int unwind_exec_insn(struct unwind_ctrl_block *ctrl)if (ret) goto error; }else if ((insn &0xf0 ) ==0x90 && - (insn &0x0d ) !=0x0d ) + (insn &0x0d ) !=0x0d ) { ctrl->vrs[SP ] = ctrl->vrs[insn &0x0f ]; -else if ((insn &0xf0 ) ==0xa0 ) { + ctrl->sp_low = ctrl->vrs[SP ]; + ctrl->sp_high =ALIGN (ctrl->sp_low,THREAD_SIZE ); + }else if ((insn &0xf0 ) ==0xa0 ) { ret =unwind_exec_pop_r4_to_rN (ctrl, insn);if (ret) goto error;
@@ -375,13 +383,12 @@ static int unwind_exec_insn(struct unwind_ctrl_block *ctrl)*/ int unwind_frame (struct stackframe *frame) { -unsigned long low;const struct unwind_idx *idx;struct unwind_ctrl_block ctrl;/* store the highest address on the stack to avoid crossing it*/ - low = frame->sp; - ctrl.sp_high =ALIGN (low,THREAD_SIZE ); + ctrl.sp_low = frame->sp; + ctrl.sp_high =ALIGN (ctrl.sp_low,THREAD_SIZE );pr_debug ("%s(pc = %08lx lr = %08lx sp = %08lx)\n" , __func__, frame->pc, frame->lr, frame->sp);
@@ -437,7 +444,7 @@ int unwind_frame(struct stackframe *frame) urc =unwind_exec_insn (&ctrl);if (urc <0 )return urc; -if (ctrl.vrs[SP ] < low || ctrl.vrs[SP ] >= ctrl.sp_high) +if (ctrl.vrs[SP ] < ctrl.sp_low || ctrl.vrs[SP ] > ctrl.sp_high)return -URC_FAILURE ; }
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 6/10 ARM: export dump_mem() to other objects
In reply to Ard BiesheuvelThe unwind info based stack unwinder will make its own call to dump_mem() to dump the exception stack, so give it external linkage.
--- arch/arm/include/asm/stacktrace.h | 2 ++ arch/arm/kernel/traps.c | 7 +++---- 2 files changed, 5 insertions(+), 4 deletions(-)
arch/arm/include/asm/stacktrace.h +2 -0
@@ -27,5 +27,7 @@ void arm_get_current_stackframe(struct pt_regs *regs, struct stackframe *frame)extern int unwind_frame (struct stackframe *frame);extern void walk_stackframe (struct stackframe *frame,int (*fn)(struct stackframe *,void *),void *data); +extern void dump_mem (const char *lvl,const char *str,unsigned long bottom, +unsigned long top);#endif /* __ASM_STACKTRACE_H */
arch/arm/kernel/traps.c +3 -4
@@ -35,6 +35,7 @@#include <asm/ptrace.h> #include <asm/unwind.h> #include <asm/tls.h> +#include <asm/stacktrace.h> #include <asm/system_misc.h> #include <asm/opcodes.h>
@@ -60,8 +61,6 @@ static int __init user_debug_setup(char *str)__setup ("user_debug=" , user_debug_setup);#endif -static void dump_mem (const char *,const char *,unsigned long ,unsigned long ); -void dump_backtrace_entry (unsigned long where,unsigned long from,unsigned long frame,const char *loglvl) {
@@ -118,8 +117,8 @@ static int verify_stack(unsigned long sp)/* * Dump out the contents of some memory nicely... */ -static void dump_mem (const char *lvl,const char *str,unsigned long bottom, -unsigned long top) +void dump_mem (const char *lvl,const char *str,unsigned long bottom, +unsigned long top) {unsigned long first;int i;
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 7/10 ARM: unwind: dump exception stack from calling frame
In reply to Ard BiesheuvelThe existing code that dumps the contents of the pt_regs structure passed to __entry routines does so while unwinding the callee frame, and dereferences the stack pointer as a struct pt_regs*. This will no longer work when we enable support for IRQ or overflow stacks, because the struct pt_regs may live on the task stack, while we are executing from another stack.
The unwinder has access to this information, but only while unwinding the calling frame. So let's combine the exception stack dumping code with the handling of the calling frame as well. By printing it before dumping the caller/callee addresses, the output order is preserved.
--- arch/arm/include/asm/stacktrace.h | 11 +++++++++++ arch/arm/kernel/traps.c | 3 ++- arch/arm/kernel/unwind.c | 8 +++++++- 3 files changed, 20 insertions(+), 2 deletions(-)
arch/arm/include/asm/stacktrace.h +11 -0
@@ -13,6 +13,17 @@ struct stackframe {unsigned long sp;unsigned long lr;unsigned long pc; + +#ifdef CONFIG_ARM_UNWIND +/* +* This field is used to track the stack pointer value when calling +* __entry routines. This is needed when IRQ stacks and overflow stacks +* are used, because in that case, the struct pt_regs passed to these +* __entry routines may be at the top of the task stack, while we are +* executing from another stack. +*/ +unsigned long sp_low; +#endif }; static __always_inline
arch/arm/kernel/traps.c +2 -1
@@ -74,7 +74,8 @@ void dump_backtrace_entry(unsigned long where, unsigned long from, loglvl, where, from);#endif -if (in_entry_text (from) && end <=ALIGN (frame,THREAD_SIZE )) +if (!IS_ENABLED (CONFIG_UNWINDER_ARM ) && +in_entry_text (from) && end <=ALIGN (frame,THREAD_SIZE ))dump_mem (loglvl,"Exception stack" , frame +4 , end); }
arch/arm/kernel/unwind.c +7 -1
@@ -29,6 +29,7 @@#include <linux/spinlock.h> #include <linux/list.h> +#include <asm/sections.h> #include <asm/stacktrace.h> #include <asm/traps.h> #include <asm/unwind.h>
@@ -459,6 +460,7 @@ int unwind_frame(struct stackframe *frame) frame->sp = ctrl.vrs[SP ]; frame->lr = ctrl.vrs[LR ]; frame->pc = ctrl.vrs[PC ]; + frame->sp_low = ctrl.sp_low;return URC_OK ; }
@@ -502,7 +504,11 @@ void unwind_backtrace(struct pt_regs *regs, struct task_struct *tsk, urc =unwind_frame (&frame);if (urc <0 )break ; -dump_backtrace_entry (where, frame.pc, frame.sp -4 , loglvl); +if (in_entry_text (where)) +dump_mem (loglvl,"Exception stack" , frame.sp_low, + frame.sp_low +sizeof (struct pt_regs )); + +dump_backtrace_entry (where, frame.pc,0 , loglvl); } }
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 8/10 ARM: implement IRQ stacks
In reply to Ard BiesheuvelNow that we no longer rely on the stack pointer to access the current task struct or thread info, we can implement support for IRQ stacks cleanly as well.
Define a per-CPU IRQ stack and switch to this stack when taking an IRQ, provided that we were not already using that stack in the interrupted context. This is never the case for IRQs taken from user space, but ones taken while running in the kernel could fire while one taken from user space has not completed yet.
--- arch/arm/Kconfig | 4 ++ arch/arm/kernel/entry-armv.S | 40 ++++++++++++++++++-- arch/arm/kernel/irq.c | 23 +++++++++++ arch/arm/kernel/traps.c | 15 +++++++- arch/arm/lib/backtrace-clang.S | 8 ++++ arch/arm/lib/backtrace.S | 8 ++++ 6 files changed, 94 insertions(+), 4 deletions(-)
arch/arm/Kconfig +4 -0
@@ -1163,6 +1163,10 @@ config CURRENT_POINTER_IN_TPIDRURO def_bool y depends on SMP && CPU_32v6K && !CPU_V6 +config IRQSTACKS + def_bool y + depends on GENERIC_IRQ_MULTI_HANDLER && THREAD_INFO_IN_TASK + config ARM_CPU_TOPOLOGY bool "Support cpu topology definition" depends on SMP && CPU_V7
arch/arm/kernel/entry-armv.S +37 -3
@@ -36,11 +36,45 @@ /* * Interrupt handling. */ - .macro irq_handler + .macro irq_handler, from_user:req #ifdef CONFIG_GENERIC_IRQ_MULTI_HANDLER mov_l r1, handle_arch_irq mov r0, sp +#ifdef CONFIG_IRQSTACKS + mov_l r2, irq_stack_ptr @ Take base address + mrc p15, 0, r3, c13, c0, 4 @ Get CPU offset + mov r7, sp @ Preserve original SP + ldr sp, [r2, r3] @ Load SP from per-CPU var + .if \from_user == 0 + @ + @ If we took the interrupt while running in the kernel, we may already + @ be using the IRQ stack, so revert to the original value in that case. + @ + subs r3, sp, r7 @ SP above bottom of IRQ stack? + rsbscs r3, r3, #THREAD_SIZE @ ... and below the top? + movcs sp, r7 @ If so, revert to incoming SP + + @ + @ Inform the unwinder where the next frame lives + @ +#if defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_GCC) + movcc ip, r7 @ Store the old SP in the frame record. + movcc lr, pc @ Make LR point into .entry.text so + @ that we will get a dump of the + @ exception stack for this frame. + stmdbcc sp!, {fp, ip, lr, pc} + addcc fp, sp, #12 +#elif defined(CONFIG_UNWINDER_ARM) +ARM( mov fp, r7 ) +ARM( .setfp fp, sp ) +THUMB( .setfp r7, sp ) +#endif + .endif +#endif bl_m [r1] +#ifdef CONFIG_IRQSTACKS + mov sp, r7 @ Restore original SP +#endif #else arch_irq_handler_default #endif
@@ -200,7 +234,7 @@ ENDPROC(__dabt_svc) .align 5 __irq_svc: svc_entry - irq_handler + irq_handler from_user=0 #ifdef CONFIG_PREEMPTION ldr r8, [tsk, #TI_PREEMPT] @ get preempt count
@@ -427,7 +461,7 @@ ENDPROC(__dabt_usr) __irq_usr: usr_entry kuser_cmpxchg_check - irq_handler + irq_handler from_user=1 get_thread_info tsk mov why, #0 b ret_to_user_from_irq
arch/arm/kernel/irq.c +23 -0
@@ -43,6 +43,25 @@unsigned long irq_err_count; +#ifdef CONFIG_IRQSTACKS + +asmlinkage DEFINE_PER_CPU_READ_MOSTLY (u8 *,irq_stack_ptr ); + +static void __init init_irq_stacks (void ) +{ +u8 *stack; +int cpu; + +for_each_possible_cpu (cpu) { + stack = (u8 *)__get_free_pages (GFP_KERNEL ,THREAD_SIZE_ORDER ); +if (WARN_ON (!stack)) +break ; +per_cpu (irq_stack_ptr, cpu) = &stack[THREAD_SIZE ]; + } +} + +#endif +int arch_show_interrupts (struct seq_file *p,int prec) {#ifdef CONFIG_FIQ
@@ -99,6 +118,10 @@ void __init init_IRQ(void) {int ret; +#ifdef CONFIG_IRQSTACKS +init_irq_stacks (); +#endif +if (IS_ENABLED (CONFIG_OF ) && !machine_desc->init_irq)irqchip_init ();else
arch/arm/kernel/traps.c +14 -1
@@ -66,6 +66,19 @@ void dump_backtrace_entry(unsigned long where, unsigned long from, {unsigned long end = frame +4 +sizeof (struct pt_regs ); +if (IS_ENABLED (CONFIG_UNWINDER_FRAME_POINTER ) && +IS_ENABLED (CONFIG_CC_IS_GCC ) && + end >ALIGN (frame,THREAD_SIZE )) { +/* +* If we are walking past the end of the stack, it may be due +* to the fact that we are on an IRQ or overflow stack. In this +* case, we can load the address of the other stack from the +* frame record. +*/ + frame = ((unsigned long *)frame)[-2 ] -4 ; + end = frame +4 +sizeof (struct pt_regs ); + } +#ifdef CONFIG_KALLSYMS printk ("%s[<%08lx>] (%ps) from [<%08lx>] (%pS)\n" , loglvl, where, (void *)where, from, (void *)from);
@@ -278,7 +291,7 @@ static int __die(const char *str, int err, struct pt_regs *regs)if (!user_mode (regs) ||in_interrupt ()) {dump_mem (KERN_EMERG ,"Stack: " , regs->ARM_sp, -THREAD_SIZE + (unsigned long )task_stack_page (tsk)); +ALIGN (regs->ARM_sp,THREAD_SIZE ));dump_backtrace (regs, tsk,KERN_EMERG );dump_instr (KERN_EMERG , regs); }
arch/arm/lib/backtrace-clang.S +8 -0
@@ -197,6 +197,14 @@ finished_setup: cmp sv_fp, frame @ next frame must be mov frame, sv_fp @ above the current frame +#ifdef CONFIG_IRQSTACKS + @ + @ Kernel stacks may be discontiguous in memory. If the next + @ frame is below the previous frame, accept it as long as it + @ lives in kernel memory. + @ + cmpls sv_fp, #PAGE_OFFSET +#endif bhi for_each_frame 1006: adr r0, .Lbad
arch/arm/lib/backtrace.S +8 -0
@@ -98,6 +98,14 @@ for_each_frame: tst frame, mask @ Check for address exceptions cmp sv_fp, frame @ next frame must be mov frame, sv_fp @ above the current frame +#ifdef CONFIG_IRQSTACKS + @ + @ Kernel stacks may be discontiguous in memory. If the next + @ frame is below the previous frame, accept it as long as it + @ lives in kernel memory. + @ + cmpls sv_fp, #PAGE_OFFSET +#endif bhi for_each_frame 1006: adr r0, .Lbad
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 9/10 ARM: call_with_stack: add unwind support
In reply to Ard BiesheuvelRestructure the code and add the unwind annotations so that both the frame pointer unwinder as well as the ELF unwind info based unwinder will be able to follow the call stack through call_with_stack().
Note that the former cannot support GCC and Clang at the same time, as they use a different idiom for the prologue/epilogue. So the code uses the GCC idiom, adding full frame pointer based unwind support for GCC while preserving the existing behavior of the Clang version, which simply omits call_with_stack() from its call stack.
--- arch/arm/lib/call_with_stack.S | 44 +++++++++++++++++--- 1 file changed, 38 insertions(+), 6 deletions(-)
arch/arm/lib/call_with_stack.S +38 -6
@@ -8,25 +8,57 @@ #include <linux/linkage.h> #include <asm/assembler.h> +#include <asm/unwind.h> /* * void call_with_stack(void (*fn)(void *), void *arg, void *sp) * * Change the stack to that pointed at by sp, then invoke fn(arg) with * the new stack. + * + * The sequence below follows the APCS frame convention for frame pointer + * unwinding, and implements the unwinder annotations needed by the EABI + * unwinder. + */ + +#if defined(CONFIG_THUMB2_KERNEL) || \ + (defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_CLANG)) +/* + * Thumb-2 builds must use R7 as the frame pointer due to the way our unwind + * info based unwinder is constructed. + * + * The code below uses the GCC idiom for managing the frame pointer in the + * function prologue and epilogue, which Clang does not support. So the best we + * can do here is not touch the frame pointer at all: this will simply omit + * this frame when unwinding the call stack. So use R7 in this case as well, + * and leave R11 unmodified. */ + fpreg .req r7 +#else + fpreg .req fp +#endif + ENTRY(call_with_stack) - str sp, [r2, #-4]! - str lr, [r2, #-4]! +UNWIND( .fnstart ) +UNWIND( .movsp ip ) + mov ip, sp + +UNWIND( .pad #4 ) +UNWIND( .save {fpreg, ip, lr} ) +THUMB( sub sp, #4 ) + push {fpreg, ip, lr ARM(, pc)} + +UNWIND( .setfp fpreg, ip, #-4 ) + sub fpreg, ip, #4 mov sp, r2 mov r2, r0 mov r0, r1 - badr lr, 1f - ret r2 + bl_r r2 -1: ldr lr, [sp] - ldr sp, [sp, #4] + ldmdb fpreg, {fpreg, ip, lr} + mov sp, ip ret lr +UNWIND( .fnend ) ENDPROC(call_with_stack)
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 10/10 ARM: run softirqs on the per-CPU IRQ stack
In reply to Ard BiesheuvelNow that we have enabled IRQ stacks, any softIRQs that are handled over the back of a hard IRQ will run from the IRQ stack as well. However, any synchronous softirq processing that happens when re-enabling softIRQs from task context will still execute on that task's stack.
So let's wire up the existing infrastructure to run these softIRQs from the IRQ stack as well.
--- arch/arm/Kconfig | 2 ++ arch/arm/kernel/irq.c | 8 ++++++++ 2 files changed, 10 insertions(+)
arch/arm/Kconfig +2 -0
@@ -1166,6 +1166,8 @@ config CURRENT_POINTER_IN_TPIDRURO config IRQSTACKS def_bool y depends on GENERIC_IRQ_MULTI_HANDLER && THREAD_INFO_IN_TASK + select HAVE_IRQ_EXIT_ON_IRQ_STACK + select HAVE_SOFTIRQ_ON_OWN_STACK config ARM_CPU_TOPOLOGY bool "Support cpu topology definition"
arch/arm/kernel/irq.c +8 -0
@@ -41,6 +41,8 @@#include <asm/mach/irq.h> #include <asm/mach/time.h> +#include "reboot.h" +unsigned long irq_err_count;#ifdef CONFIG_IRQSTACKS
@@ -60,6 +62,12 @@ static void __init init_irq_stacks(void) } } +void do_softirq_own_stack (void ) +{ +call_with_stack ((void *)__do_softirq,NULL , +__this_cpu_read (irq_stack_ptr)); +} +#endif int arch_show_interrupts (struct seq_file *p,int prec)
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel