ARM: add support for IRQ stacks
v4 cover ARM: add support for IRQ stacks
Compared to user space, the kernel's task stacks are tiny and inflexible, as we don't grow them dynamically using demand paging. This is the reason we tend to obsess about functions with disproportionately large stack frames, given that it is hard to predict statically how calls to those functions may combine at runtime, and exhaust the stack and crash the kernel.
This becomes even less predictable when taking interrupt handling into account, as their handlers are normally executed on the stack of the task that was interrupted, regardless of how deep its call stack was at the time of the interruption. To decouple these, and reduce the risk of hitting a pathological worst case where IRQ handling and the task below it happen to exhaust the available stack space when combined, we can switch to a different stack when taking interrupts, similar to how this is done already on some other architectures. This series implements this approach for ARM.
Note that a good chunk of the changes below are related to supporting non-contiguous call stacks, which is also relevant in the context of vmap'ed stacks, which use a separate overflow stack to handle stack overflows. The changes preserve all functionality related to walking the call stack and dumping exception stacks and register contents.
Changes since v3:
- rebase onto v5.16-rc1
- dropped patch abstracting the call to (*handle_arch_irq)() from entry.S, which is no longer needed due to cross-arch rework that landed during the merge window
- add some acks from Nick
Changes since v2:
- improve Clang support, by emitting code that is compatible with its frame pointer unwinder if that is the unwinder being used;
- add acks from Arnd and Linus (thanks!)
Changes since v1:
- drop the first bugfix patch, which has been queued as a fix in the meantime;
- preserve/restore FP in the irq_handler entry code;
- add missing include to arch/arm/kernel/irq.c to silence warnings about missing prototypes.
Patch #1 removes some code that I spotted that is no longer used.
Patch #2 introduces a macro that will be used later in the series to emit the optimal indirect call sequence for older and newer cores.
Patch #3 updates the unwind info based unwinder so it can deal with call stacks that are non-contiguous.
Patch #4 exports dump_mem() to other compilation units so the ARM unwinder can call it directly. This is needed by the next patch.
Patch #5 refactors the ARM unwinder to dump the exception stack from a context where it is able to figure out if it lives on the current stack or on the task stack.
Patch #6 fixes an issue in the Clang frame pointer unwinder, which may get into an endless recursive fault if any of the stack frames have a bogus value for the link register.
Patch #7 implements the actual IRQ stacks support, by allocating one for each CPU, and adding the code to switch to it in the IRQ entry path. It also contains some related changes to allow the frame pointer based unwinder to deal with the new situation.
Patch #8 modifies call_with_stack() so both the frame pointer unwinder as well as the ARM unwinder know how to deal with it.
Patch #9 adds the IRQ stack switching for softIRQ handling initiated from task context.
Ard Biesheuvel (9):
ARM: remove some dead code
ARM: assembler: introduce bl_r macro
ARM: unwind: support unwinding across multiple stacks
ARM: export dump_mem() to other objects
ARM: unwind: dump exception stack from calling frame
ARM: backtrace-clang: avoid crash on bogus frame pointer
ARM: implement IRQ stacks
ARM: call_with_stack: add unwind support
ARM: run softirqs on the per-CPU IRQ stack
arch/arm/Kconfig | 6 +++ arch/arm/include/asm/assembler.h | 19 +++++++ arch/arm/include/asm/entry-macro-multi.S | 24 --------- arch/arm/include/asm/smp.h | 5 -- arch/arm/include/asm/stacktrace.h | 12 +++++ arch/arm/kernel/entry-armv.S | 54 ++++++++++++++++++-- arch/arm/kernel/irq.c | 37 ++++++++++++++ arch/arm/kernel/smp.c | 5 -- arch/arm/kernel/traps.c | 25 ++++++--- arch/arm/kernel/unwind.c | 33 ++++++++---- arch/arm/lib/backtrace-clang.S | 14 +++-- arch/arm/lib/backtrace.S | 8 +++ arch/arm/lib/call_with_stack.S | 33 +++++++++--- 13 files changed, 211 insertions(+), 64 deletions(-)
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 1/9 ARM: remove some dead code
This code appears to be no longer used so let's get rid of it.
--- arch/arm/include/asm/entry-macro-multi.S | 24 -------------------- arch/arm/include/asm/smp.h | 5 ---- arch/arm/kernel/smp.c | 5 ---- 3 files changed, 34 deletions(-)
arch/arm/include/asm/entry-macro-multi.S +0 -24
@@ -13,28 +13,4 @@ @ badrne lr, 1b bne asm_do_IRQ - -#ifdef CONFIG_SMP - /* - * XXX - * - * this macro assumes that irqstat (r2) and base (r6) are - * preserved from get_irqnr_and_base above - */ - ALT_SMP(test_for_ipi r0, r2, r6, lr) - ALT_UP_B(9997f) - movne r1, sp - badrne lr, 1b - bne do_IPI -#endif -9997: - .endm - - .macro arch_irq_handler, symbol_name - .align 5 - .global \symbol_name -\symbol_name: - mov r8, lr - arch_irq_handler_default - ret r8 .endm
arch/arm/include/asm/smp.h +0 -5
@@ -24,11 +24,6 @@ struct seq_file;*/ extern void show_ipi_list (struct seq_file *,int ); -/* -* Called from assembly code, this handles an IPI. -*/ -asmlinkage void do_IPI(int ipinr, struct pt_regs *regs); -/* * Called from C code, this handles an IPI. */
arch/arm/kernel/smp.c +0 -5
@@ -631,11 +631,6 @@ static void ipi_complete(unsigned int cpu)/* * Main handler for inter-processor interrupts */ -asmlinkage void __exception_irq_entry do_IPI(int ipinr, struct pt_regs *regs) -{ -handle_IPI (ipinr, regs); -} -static void do_handle_IPI (int ipinr) {unsigned int cpu =smp_processor_id ();
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 2/9 ARM: assembler: introduce bl_r macro
In reply to Ard BiesheuvelAdd a bl_r macro that abstract the difference between the ways indirect calls are performed on older and newer ARM architecture revisions.
The main difference is to prefer blx instructions over explicit LR assignments when possible, as these tend to confuse the prediction logic in out-of-order cores when speculating across a function return.
--- arch/arm/include/asm/assembler.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/arm/include/asm/assembler.h +15 -0
@@ -624,4 +624,19 @@ THUMB( orr \reg , \reg , #PSR_T_BIT ) .endif .endm + /* + * bl_r - branch and link to register + * + * @dst: target to branch to + * @c: conditional opcode suffix + */ + .macro bl_r, dst:req, c + .if __LINUX_ARM_ARCH__ < 6 + mov\c lr, pc + mov\c pc, \dst + .else + blx\c \dst + .endif + .endm +#endif /* __ASM_ASSEMBLER_H__ */
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 3/9 ARM: unwind: support unwinding across multiple stacks
In reply to Ard BiesheuvelImplement support in the unwinder for dealing with multiple stacks. This will be needed once we add support for IRQ stacks, or for the overflow stack used by the vmap'ed stacks code.
This involves tracking the unwind opcodes that either update the virtual stack pointer from another virtual register, or perform an explicit subtract on the virtual stack pointer, and updating the low and high bounds that we use to sanitize the stack pointer accordingly.
--- arch/arm/kernel/unwind.c | 25 +++++++++++++------- 1 file changed, 16 insertions(+), 9 deletions(-)
arch/arm/kernel/unwind.c +16 -9
@@ -52,6 +52,7 @@ EXPORT_SYMBOL(__aeabi_unwind_cpp_pr2);struct unwind_ctrl_block {unsigned long vrs[16 ];/* virtual register set */ const unsigned long *insn;/* pointer to the current instructions word */ +unsigned long sp_low;/* lowest value of sp allowed */ unsigned long sp_high;/* highest value of sp allowed */ /* * 1 : check for stack overflow for each register pop.
@@ -256,8 +257,12 @@ static int unwind_exec_pop_subset_r4_to_r13(struct unwind_ctrl_block *ctrl, mask >>=1 ; reg++; } -if (!load_sp) +if (!load_sp) { ctrl->vrs[SP ] = (unsigned long )vsp; + }else { + ctrl->sp_low = ctrl->vrs[SP ]; + ctrl->sp_high =ALIGN (ctrl->sp_low,THREAD_SIZE ); + }return URC_OK ; }
@@ -313,9 +318,10 @@ static int unwind_exec_insn(struct unwind_ctrl_block *ctrl)if ((insn &0xc0 ) ==0x00 ) ctrl->vrs[SP ] += ((insn &0x3f ) <<2 ) +4 ; -else if ((insn &0xc0 ) ==0x40 ) +else if ((insn &0xc0 ) ==0x40 ) { ctrl->vrs[SP ] -= ((insn &0x3f ) <<2 ) +4 ; -else if ((insn &0xf0 ) ==0x80 ) { + ctrl->sp_low = ctrl->vrs[SP ]; + }else if ((insn &0xf0 ) ==0x80 ) {unsigned long mask; insn = (insn <<8 ) |unwind_get_byte (ctrl);
@@ -330,9 +336,11 @@ static int unwind_exec_insn(struct unwind_ctrl_block *ctrl)if (ret) goto error; }else if ((insn &0xf0 ) ==0x90 && - (insn &0x0d ) !=0x0d ) + (insn &0x0d ) !=0x0d ) { ctrl->vrs[SP ] = ctrl->vrs[insn &0x0f ]; -else if ((insn &0xf0 ) ==0xa0 ) { + ctrl->sp_low = ctrl->vrs[SP ]; + ctrl->sp_high =ALIGN (ctrl->sp_low,THREAD_SIZE ); + }else if ((insn &0xf0 ) ==0xa0 ) { ret =unwind_exec_pop_r4_to_rN (ctrl, insn);if (ret) goto error;
@@ -375,13 +383,12 @@ static int unwind_exec_insn(struct unwind_ctrl_block *ctrl)*/ int unwind_frame (struct stackframe *frame) { -unsigned long low;const struct unwind_idx *idx;struct unwind_ctrl_block ctrl;/* store the highest address on the stack to avoid crossing it*/ - low = frame->sp; - ctrl.sp_high =ALIGN (low,THREAD_SIZE ); + ctrl.sp_low = frame->sp; + ctrl.sp_high =ALIGN (ctrl.sp_low,THREAD_SIZE );pr_debug ("%s(pc = %08lx lr = %08lx sp = %08lx)\n" , __func__, frame->pc, frame->lr, frame->sp);
@@ -437,7 +444,7 @@ int unwind_frame(struct stackframe *frame) urc =unwind_exec_insn (&ctrl);if (urc <0 )return urc; -if (ctrl.vrs[SP ] < low || ctrl.vrs[SP ] >= ctrl.sp_high) +if (ctrl.vrs[SP ] < ctrl.sp_low || ctrl.vrs[SP ] > ctrl.sp_high)return -URC_FAILURE ; }
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 4/9 ARM: export dump_mem() to other objects
In reply to Ard BiesheuvelThe unwind info based stack unwinder will make its own call to dump_mem() to dump the exception stack, so give it external linkage.
--- arch/arm/include/asm/stacktrace.h | 2 ++ arch/arm/kernel/traps.c | 7 +++---- 2 files changed, 5 insertions(+), 4 deletions(-)
arch/arm/include/asm/stacktrace.h +2 -0
@@ -36,5 +36,7 @@ void arm_get_current_stackframe(struct pt_regs *regs, struct stackframe *frame)extern int unwind_frame (struct stackframe *frame);extern void walk_stackframe (struct stackframe *frame,int (*fn)(struct stackframe *,void *),void *data); +extern void dump_mem (const char *lvl,const char *str,unsigned long bottom, +unsigned long top);#endif /* __ASM_STACKTRACE_H */
arch/arm/kernel/traps.c +3 -4
@@ -35,6 +35,7 @@#include <asm/ptrace.h> #include <asm/unwind.h> #include <asm/tls.h> +#include <asm/stacktrace.h> #include <asm/system_misc.h> #include <asm/opcodes.h>
@@ -60,8 +61,6 @@ static int __init user_debug_setup(char *str)__setup ("user_debug=" , user_debug_setup);#endif -static void dump_mem (const char *,const char *,unsigned long ,unsigned long ); -void dump_backtrace_entry (unsigned long where,unsigned long from,unsigned long frame,const char *loglvl) {
@@ -118,8 +117,8 @@ static int verify_stack(unsigned long sp)/* * Dump out the contents of some memory nicely... */ -static void dump_mem (const char *lvl,const char *str,unsigned long bottom, -unsigned long top) +void dump_mem (const char *lvl,const char *str,unsigned long bottom, +unsigned long top) {unsigned long first;int i;
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 5/9 ARM: unwind: dump exception stack from calling frame
In reply to Ard BiesheuvelThe existing code that dumps the contents of the pt_regs structure passed to __entry routines does so while unwinding the callee frame, and dereferences the stack pointer as a struct pt_regs*. This will no longer work when we enable support for IRQ or overflow stacks, because the struct pt_regs may live on the task stack, while we are executing from another stack.
The unwinder has access to this information, but only while unwinding the calling frame. So let's combine the exception stack dumping code with the handling of the calling frame as well. By printing it before dumping the caller/callee addresses, the output order is preserved.
--- arch/arm/include/asm/stacktrace.h | 10 ++++++++++ arch/arm/kernel/traps.c | 3 ++- arch/arm/kernel/unwind.c | 8 +++++++- 3 files changed, 19 insertions(+), 2 deletions(-)
arch/arm/include/asm/stacktrace.h +10 -0
@@ -18,6 +18,16 @@ struct stackframe {struct llist_node *kr_cur;struct task_struct *tsk;#endif +#ifdef CONFIG_ARM_UNWIND +/* +* This field is used to track the stack pointer value when calling +* __entry routines. This is needed when IRQ stacks and overflow stacks +* are used, because in that case, the struct pt_regs passed to these +* __entry routines may be at the top of the task stack, while we are +* executing from another stack. +*/ +unsigned long sp_low; +#endif }; static __always_inline
arch/arm/kernel/traps.c +2 -1
@@ -74,7 +74,8 @@ void dump_backtrace_entry(unsigned long where, unsigned long from, loglvl, where, from);#endif -if (in_entry_text (from) && end <=ALIGN (frame,THREAD_SIZE )) +if (!IS_ENABLED (CONFIG_UNWINDER_ARM ) && +in_entry_text (from) && end <=ALIGN (frame,THREAD_SIZE ))dump_mem (loglvl,"Exception stack" , frame +4 , end); }
arch/arm/kernel/unwind.c +7 -1
@@ -29,6 +29,7 @@#include <linux/spinlock.h> #include <linux/list.h> +#include <asm/sections.h> #include <asm/stacktrace.h> #include <asm/traps.h> #include <asm/unwind.h>
@@ -459,6 +460,7 @@ int unwind_frame(struct stackframe *frame) frame->sp = ctrl.vrs[SP ]; frame->lr = ctrl.vrs[LR ]; frame->pc = ctrl.vrs[PC ]; + frame->sp_low = ctrl.sp_low;return URC_OK ; }
@@ -502,7 +504,11 @@ void unwind_backtrace(struct pt_regs *regs, struct task_struct *tsk, urc =unwind_frame (&frame);if (urc <0 )break ; -dump_backtrace_entry (where, frame.pc, frame.sp -4 , loglvl); +if (in_entry_text (where)) +dump_mem (loglvl,"Exception stack" , frame.sp_low, + frame.sp_low +sizeof (struct pt_regs )); + +dump_backtrace_entry (where, frame.pc,0 , loglvl); } }
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 6/9 ARM: backtrace-clang: avoid crash on bogus frame pointer
In reply to Ard BiesheuvelThe Clang backtrace code dereferences the link register value pulled from the stack to decide whether the caller was a branch-and-link instruction, in order to subsequently decode the offset to find the start of the calling function. Unlike other loads in this routine, this one is not protected by a fixup, and may therefore cause a crash if the address in question is bogus.
So let's fix this, by treating the fault as a failure to decode the 'bl' instruction. To avoid a label renum, reuse a fixup label that guards an instruction that cannot fault to begin with.
--- arch/arm/lib/backtrace-clang.S | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/arm/lib/backtrace-clang.S +3 -3
@@ -144,7 +144,7 @@ for_each_frame: tst frame, mask @ Check for address exceptions */ 1003: ldr sv_lr, [sv_fp, #4] @ get saved lr from next frame - ldr r0, [sv_lr, #-4] @ get call instruction +1004: ldr r0, [sv_lr, #-4] @ get call instruction ldr r3, .Lopcode+4 and r2, r3, r0 @ is this a bl call teq r2, r3
@@ -164,7 +164,7 @@ finished_setup: /* * Print the function (sv_pc) and where it was called from (sv_lr). */ -1004: mov r0, sv_pc + mov r0, sv_pc mov r1, sv_lr mov r2, frame
@@ -210,7 +210,7 @@ ENDPROC(c_backtrace) .long 1001b, 1006b .long 1002b, 1006b .long 1003b, 1006b - .long 1004b, 1006b + .long 1004b, finished_setup .long 1005b, 1006b .popsection
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 7/9 ARM: implement IRQ stacks
In reply to Ard BiesheuvelNow that we no longer rely on the stack pointer to access the current task struct or thread info, we can implement support for IRQ stacks cleanly as well.
Define a per-CPU IRQ stack and switch to this stack when taking an IRQ, provided that we were not already using that stack in the interrupted context. This is never the case for IRQs taken from user space, but ones taken while running in the kernel could fire while one taken from user space has not completed yet.
--- arch/arm/Kconfig | 4 ++ arch/arm/include/asm/assembler.h | 4 ++ arch/arm/kernel/entry-armv.S | 54 ++++++++++++++++++-- arch/arm/kernel/irq.c | 23 +++++++++ arch/arm/kernel/traps.c | 15 +++++- arch/arm/lib/backtrace-clang.S | 8 +++ arch/arm/lib/backtrace.S | 8 +++ 7 files changed, 112 insertions(+), 4 deletions(-)
arch/arm/Kconfig +4 -0
@@ -1166,6 +1166,10 @@ config CURRENT_POINTER_IN_TPIDRURO def_bool y depends on SMP && CPU_32v6K && !CPU_V6 +config IRQSTACKS + def_bool y + depends on GENERIC_IRQ_MULTI_HANDLER && THREAD_INFO_IN_TASK + config ARM_CPU_TOPOLOGY bool "Support cpu topology definition" depends on SMP && CPU_V7
arch/arm/include/asm/assembler.h +4 -0
@@ -86,6 +86,10 @@#define IMM12_MASK 0xfff +/* the frame pointer used for stack unwinding */ +ARM( fpreg .req r11 ) +THUMB( fpreg .req r7 ) +/* * Enable and disable interrupts */
arch/arm/kernel/entry-armv.S +51 -3
75 lines not on this page · raw message
arch/arm/kernel/irq.c +23 -0
@@ -43,6 +43,25 @@unsigned long irq_err_count; +#ifdef CONFIG_IRQSTACKS + +asmlinkage DEFINE_PER_CPU_READ_MOSTLY (u8 *,irq_stack_ptr ); + +static void __init init_irq_stacks (void ) +{ +u8 *stack; +int cpu; + +for_each_possible_cpu (cpu) { + stack = (u8 *)__get_free_pages (GFP_KERNEL ,THREAD_SIZE_ORDER ); +if (WARN_ON (!stack)) +break ; +per_cpu (irq_stack_ptr, cpu) = &stack[THREAD_SIZE ]; + } +} + +#endif +int arch_show_interrupts (struct seq_file *p,int prec) {#ifdef CONFIG_FIQ
@@ -101,6 +120,10 @@ void __init init_IRQ(void) {int ret; +#ifdef CONFIG_IRQSTACKS +init_irq_stacks (); +#endif +if (IS_ENABLED (CONFIG_OF ) && !machine_desc->init_irq)irqchip_init ();else
arch/arm/kernel/traps.c +14 -1
@@ -66,6 +66,19 @@ void dump_backtrace_entry(unsigned long where, unsigned long from, {unsigned long end = frame +4 +sizeof (struct pt_regs ); +if (IS_ENABLED (CONFIG_UNWINDER_FRAME_POINTER ) && +IS_ENABLED (CONFIG_CC_IS_GCC ) && + end >ALIGN (frame,THREAD_SIZE )) { +/* +* If we are walking past the end of the stack, it may be due +* to the fact that we are on an IRQ or overflow stack. In this +* case, we can load the address of the other stack from the +* frame record. +*/ + frame = ((unsigned long *)frame)[-2 ] -4 ; + end = frame +4 +sizeof (struct pt_regs ); + } +#ifdef CONFIG_KALLSYMS printk ("%s[<%08lx>] (%ps) from [<%08lx>] (%pS)\n" , loglvl, where, (void *)where, from, (void *)from);
@@ -278,7 +291,7 @@ static int __die(const char *str, int err, struct pt_regs *regs)if (!user_mode (regs) ||in_interrupt ()) {dump_mem (KERN_EMERG ,"Stack: " , regs->ARM_sp, -THREAD_SIZE + (unsigned long )task_stack_page (tsk)); +ALIGN (regs->ARM_sp,THREAD_SIZE ));dump_backtrace (regs, tsk,KERN_EMERG );dump_instr (KERN_EMERG , regs); }
arch/arm/lib/backtrace-clang.S +8 -0
@@ -197,6 +197,14 @@ finished_setup: cmp sv_fp, frame @ next frame must be mov frame, sv_fp @ above the current frame +#ifdef CONFIG_IRQSTACKS + @ + @ Kernel stacks may be discontiguous in memory. If the next + @ frame is below the previous frame, accept it as long as it + @ lives in kernel memory. + @ + cmpls sv_fp, #PAGE_OFFSET +#endif bhi for_each_frame 1006: adr r0, .Lbad
arch/arm/lib/backtrace.S +8 -0
@@ -98,6 +98,14 @@ for_each_frame: tst frame, mask @ Check for address exceptions cmp sv_fp, frame @ next frame must be mov frame, sv_fp @ above the current frame +#ifdef CONFIG_IRQSTACKS + @ + @ Kernel stacks may be discontiguous in memory. If the next + @ frame is below the previous frame, accept it as long as it + @ lives in kernel memory. + @ + cmpls sv_fp, #PAGE_OFFSET +#endif bhi for_each_frame 1006: adr r0, .Lbad
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 8/9 ARM: call_with_stack: add unwind support
In reply to Ard BiesheuvelRestructure the code and add the unwind annotations so that both the frame pointer unwinder as well as the EHABI unwind info based unwinder will be able to follow the call stack through call_with_stack().
Since GCC and Clang use different formats for the stack frame, two methods are implemented: a GCC version that pushes fp, sp, lr and pc for compatibility with the frame pointer unwinder, and a second version that works with Clang, as well as with the EHABI unwinder both in ARM and Thumb2 modes.
--- arch/arm/lib/call_with_stack.S | 33 +++++++++++++++----- 1 file changed, 25 insertions(+), 8 deletions(-)
arch/arm/lib/call_with_stack.S +25 -8
@@ -8,25 +8,42 @@ #include <linux/linkage.h> #include <asm/assembler.h> +#include <asm/unwind.h> /* * void call_with_stack(void (*fn)(void *), void *arg, void *sp) * * Change the stack to that pointed at by sp, then invoke fn(arg) with * the new stack. + * + * The sequence below follows the APCS frame convention for frame pointer + * unwinding, and implements the unwinder annotations needed by the EABI + * unwinder. */ -ENTRY(call_with_stack) - str sp, [r2, #-4]! - str lr, [r2, #-4]! +ENTRY(call_with_stack) +#if defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_GCC) + mov ip, sp + push {fp, ip, lr, pc} + sub fp, ip, #4 +#else +UNWIND( .fnstart ) +UNWIND( .save {fpreg, lr} ) + push {fpreg, lr} +UNWIND( .setfp fpreg, sp ) + mov fpreg, sp +#endif mov sp, r2 mov r2, r0 mov r0, r1 - badr lr, 1f - ret r2 + bl_r r2 -1: ldr lr, [sp] - ldr sp, [sp, #4] - ret lr +#if defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_GCC) + ldmdb fp, {fp, sp, pc} +#else + mov sp, fpreg + pop {fpreg, pc} +UNWIND( .fnend ) +#endif ENDPROC(call_with_stack)
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 9/9 ARM: run softirqs on the per-CPU IRQ stack
In reply to Ard BiesheuvelNow that we have enabled IRQ stacks, any softIRQs that are handled over the back of a hard IRQ will run from the IRQ stack as well. However, any synchronous softirq processing that happens when re-enabling softIRQs from task context will still execute on that task's stack.
Since any call to local_bh_enable() at any level in the task's call stack may trigger a softIRQ processing run, which could potentially cause a task stack overflow if the combined stack footprints exceed the stack's size, let's run these synchronous invocations of do_softirq() on the IRQ stack as well.
--- arch/arm/Kconfig | 2 ++ arch/arm/kernel/irq.c | 14 ++++++++++++++ 2 files changed, 16 insertions(+)
arch/arm/Kconfig +2 -0
@@ -1169,6 +1169,8 @@ config CURRENT_POINTER_IN_TPIDRURO config IRQSTACKS def_bool y depends on GENERIC_IRQ_MULTI_HANDLER && THREAD_INFO_IN_TASK + select HAVE_IRQ_EXIT_ON_IRQ_STACK + select HAVE_SOFTIRQ_ON_OWN_STACK config ARM_CPU_TOPOLOGY bool "Support cpu topology definition"
arch/arm/kernel/irq.c +14 -0
@@ -36,11 +36,14 @@#include <asm/hardware/cache-l2x0.h> #include <asm/hardware/cache-uniphier.h> #include <asm/outercache.h> +#include <asm/softirq_stack.h> #include <asm/exception.h> #include <asm/mach/arch.h> #include <asm/mach/irq.h> #include <asm/mach/time.h> +#include "reboot.h" +unsigned long irq_err_count;#ifdef CONFIG_IRQSTACKS
@@ -60,6 +63,17 @@ static void __init init_irq_stacks(void) } } +static void ____do_softirq (void *arg) +{ +__do_softirq (); +} + +void do_softirq_own_stack (void ) +{ +call_with_stack (____do_softirq,NULL , +__this_cpu_read (irq_stack_ptr)); +} +#endif int arch_show_interrupts (struct seq_file *p,int prec)
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel