ARM: add support for IRQ stacks

This code appears to be no longer used so let's get rid of it.

Signed-off-by Ard Biesheuvel <[email protected]>

Add some macros that abstract the difference between the ways indirect calls are performed on older and newer ARM architecture revisions.

The main difference is to prefer blx instructions over explicit LR assignments when possible, as these tend to confuse the prediction logic in out-of-order cores when speculating across a function return.

Signed-off-by Ard Biesheuvel <[email protected]>

Quoted 9 lines · Ard Biesheuvel

Add some macros that abstract the difference between the ways indirect calls are performed on older and newer ARM architecture revisions.

The main difference is to prefer blx instructions over explicit LR assignments when possible, as these tend to confuse the prediction logic in out-of-order cores when speculating across a function return.

Signed-off-by Ard Biesheuvel <[email protected]>

Reviewed-by Arnd Bergmann <[email protected]>

arch/arm/include/asm/assembler.h THUMB( orr \reg , \reg , #PSR_T_BIT ) · line 640

Quoted 8 lines

arch/arm/include/asm/assembler.h +8 -0

+       .macro          bl_r, dst:req, c
+       .if             __LINUX_ARM_ARCH__ < 6
+       mov\c           lr, pc
+       mov\c           pc, \dst
+       .else
+       blx\c           \dst
+       .endif
+       .endm

I suppose we could use blx on ARMv5 as well, but it wouldn't make a difference there, right?

The current irq_handler macro uses a literal load followed by an explicit assignment of lr before an indirect call, both of which are sub-optimal on recent ARM cores. Replace it by a mov_l call, which will evaluate to a movw/movt pair on v7 cores, avoiding one of the two loads, followed by a call to a newly introduced 'bl_m' macro, which will evaluate to a blx instruction on cores that support it. (Setting lr explicitly rather than via a bl or blx instruction confuses the return address prediction that modern out-of-order cores use to speculatively perform the function return.)

Signed-off-by Ard Biesheuvel <[email protected]>

Implement support in the unwinder for dealing with multiple stacks. This will be needed once we add support for IRQ stacks, or for the overflow stack used by the vmap'ed stacks code.

This involves tracking the unwind opcodes that either update the virtual stack pointer from another virtual register, or perform an explicit subtract on the virtual stack pointer, and updating the low and high bounds that we use to sanitize the stack pointer accordingly.

Signed-off-by Ard Biesheuvel <[email protected]>

The unwind info based stack unwinder will make its own call to dump_mem() to dump the exception stack, so give it external linkage.

Signed-off-by Ard Biesheuvel <[email protected]>

The existing code that dumps the contents of the pt_regs structure passed to __entry routines does so while unwinding the callee frame, and dereferences the stack pointer as a struct pt_regs*. This will no longer work when we enable support for IRQ or overflow stacks, because the struct pt_regs may live on the task stack, while we are executing from another stack.

The unwinder has access to this information, but only while unwinding the calling frame. So let's combine the exception stack dumping code with the handling of the calling frame as well. By printing it before dumping the caller/callee addresses, the output order is preserved.

Signed-off-by Ard Biesheuvel <[email protected]>

Now that we no longer rely on the stack pointer to access the current task struct or thread info, we can implement support for IRQ stacks cleanly as well.

Define a per-CPU IRQ stack and switch to this stack when taking an IRQ, provided that we were not already using that stack in the interrupted context. This is never the case for IRQs taken from user space, but ones taken while running in the kernel could fire while one taken from user space has not completed yet.

Signed-off-by Ard Biesheuvel <[email protected]>

Restructure the code and add the unwind annotations so that both the frame pointer unwinder as well as the ELF unwind info based unwinder will be able to follow the call stack through call_with_stack().

Note that the former cannot support GCC and Clang at the same time, as they use a different idiom for the prologue/epilogue. So the code uses the GCC idiom, adding full frame pointer based unwind support for GCC while preserving the existing behavior of the Clang version, which simply omits call_with_stack() from its call stack.

Signed-off-by Ard Biesheuvel <[email protected]>

---
 arch/arm/lib/call_with_stack.S | 44 +++++++++++++++++---
 1 file changed, 38 insertions(+), 6 deletions(-)

arch/arm/lib/call_with_stack.S +38 -7

index 28b0341ae786..133dffa2404a 100644
@@ -8,25 +8,57 @@
 
 #include <linux/linkage.h>
 #include <asm/assembler.h>
+#include <asm/unwind.h>
 
 /*
  * void call_with_stack(void (*fn)(void *), void *arg, void *sp)
  *
  * Change the stack to that pointed at by sp, then invoke fn(arg) with
  * the new stack.
+ *
+ * The sequence below follows the APCS frame convention for frame pointer
+ * unwinding, and implements the unwinder annotations needed by the EABI
+ * unwinder.
+ */
+
+#if defined(CONFIG_THUMB2_KERNEL) || \
+    (defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_CLANG))
+/*
+ * Thumb-2 builds must use R7 as the frame pointer due to the way our unwind
+ * info based unwinder is constructed.
+ *
+ * The code below uses the GCC idiom for managing the frame pointer in the
+ * function prologue and epilogue, which Clang does not support. So the best we
+ * can do here is not touch the frame pointer at all: this will simply omit
+ * this frame when unwinding the call stack. So use R7 in this case as well,
+ * and leave R11 unmodified.
  */
+       fpreg   .req    r7
+#else
+       fpreg   .req    fp
+#endif
+
 ENTRY(call_with_stack)
-       str     sp, [r2, #-4]!
-       str     lr, [r2, #-4]!
+UNWIND(        .fnstart                )
+UNWIND(        .movsp  ip              )
+       mov     ip, sp
+
+UNWIND(        .pad    #4              )
+UNWIND(        .save   {fpreg, ip, lr} )
+THUMB( sub     sp, #4          )
+       push    {fpreg, ip, lr ARM(, pc)}
+
+UNWIND(        .setfp  fpreg, ip, #-4  )
+       sub     fpreg, ip, #4
 
        mov     sp, r2
        mov     r2, r0
        mov     r0, r1
 
-       badr    lr, 1f
-       ret     r2
+       bl_r    r2
 
-1:     ldr     lr, [sp]
-       ldr     sp, [sp, #4]
+       ldmdb   fpreg, {fpreg, ip, lr}
+       mov     sp, ip
        ret     lr
+UNWIND(        .fnend                  )
 ENDPROC(call_with_stack)
--

2.30.2

Quoted 5 lines · Ard Biesheuvel

Restructure the code and add the unwind annotations so that both the frame pointer unwinder as well as the ELF unwind info based unwinder will be able to follow the call stack through call_with_stack().

Quoted 7 lines

Note that the former cannot support GCC and Clang at the same time, as they use a different idiom for the prologue/epilogue. So the code uses the GCC idiom, adding full frame pointer based unwind support for GCC while preserving the existing behavior of the Clang version, which simply omits call_with_stack() from its call stack.

Signed-off-by Ard Biesheuvel <[email protected]>

I would like Nick to take a look at this for the clang support, he spent some time on getting the frame pointer unwinder working with clang, so he may have additional comments about this.

Quoted 14 lines

---
 arch/arm/lib/call_with_stack.S | 44 +++++++++++++++++---
 1 file changed, 38 insertions(+), 6 deletions(-)

arch/arm/lib/call_with_stack.S +1 -0

index 28b0341ae786..133dffa2404a 100644
@@ -8,25 +8,57 @@
 
 #include <linux/linkage.h>
 #include <asm/assembler.h>
+#include <asm/unwind.h>
 

arch/arm/lib/call_with_stack.S line 25

Quoted 13 lines

arch/arm/lib/call_with_stack.S +8 -0

 /*
  * void call_with_stack(void (*fn)(void *), void *arg, void *sp)
  *
  * Change the stack to that pointed at by sp, then invoke fn(arg) with
  * the new stack.
+ *
+ * The sequence below follows the APCS frame convention for frame pointer
+ * unwinding, and implements the unwinder annotations needed by the EABI
+ * unwinder.
+ */
+
+#if defined(CONFIG_THUMB2_KERNEL) || \
+    (defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_CLANG))

arch/arm/lib/call_with_stack.S line 31

Quoted 6 lines

arch/arm/lib/call_with_stack.S +6 -0

+/*
+ * Thumb-2 builds must use R7 as the frame pointer due to the way our unwind
+ * info based unwinder is constructed.
+ *
+ * The code below uses the GCC idiom for managing the frame pointer in the
+ * function prologue and epilogue, which Clang does not support. So the best we

arch/arm/lib/call_with_stack.S line 36

Quoted 5 lines

arch/arm/lib/call_with_stack.S +4 -0

+ * can do here is not touch the frame pointer at all: this will simply omit
+ * this frame when unwinding the call stack. So use R7 in this case as well,
+ * and leave R11 unmodified.
  */
+       fpreg   .req    r7

Quoted 37 lines

arch/arm/lib/call_with_stack.S +19 -7

+#else
+       fpreg   .req    fp
+#endif
+
 ENTRY(call_with_stack)
-       str     sp, [r2, #-4]!
-       str     lr, [r2, #-4]!
+UNWIND(        .fnstart                )
+UNWIND(        .movsp  ip              )
+       mov     ip, sp
+
+UNWIND(        .pad    #4              )
+UNWIND(        .save   {fpreg, ip, lr} )
+THUMB( sub     sp, #4          )
+       push    {fpreg, ip, lr ARM(, pc)}
+
+UNWIND(        .setfp  fpreg, ip, #-4  )
+       sub     fpreg, ip, #4
 
        mov     sp, r2
        mov     r2, r0
        mov     r0, r1
 
-       badr    lr, 1f
-       ret     r2
+       bl_r    r2
 
-1:     ldr     lr, [sp]
-       ldr     sp, [sp, #4]
+       ldmdb   fpreg, {fpreg, ip, lr}
+       mov     sp, ip
        ret     lr
+UNWIND(        .fnend                  )
 ENDPROC(call_with_stack)
--

2.30.2

Quoted 43 lines · Arnd Bergmann

Quoted 5 lines · Ard Biesheuvel

Restructure the code and add the unwind annotations so that both the frame pointer unwinder as well as the ELF unwind info based unwinder will be able to follow the call stack through call_with_stack().

Quoted 7 lines

Note that the former cannot support GCC and Clang at the same time, as they use a different idiom for the prologue/epilogue. So the code uses the GCC idiom, adding full frame pointer based unwind support for GCC while preserving the existing behavior of the Clang version, which simply omits call_with_stack() from its call stack.

Signed-off-by Ard Biesheuvel <[email protected]>

I would like Nick to take a look at this for the clang support, he spent some time on getting the frame pointer unwinder working with clang, so he may have additional comments about this.

Quoted 24 lines

---
 arch/arm/lib/call_with_stack.S | 44 +++++++++++++++++---
 1 file changed, 38 insertions(+), 6 deletions(-)

arch/arm/lib/call_with_stack.S +6 -0

index 28b0341ae786..133dffa2404a 100644
@@ -8,25 +8,57 @@
 
 #include <linux/linkage.h>
 #include <asm/assembler.h>
+#include <asm/unwind.h>
 
 /*
  * void call_with_stack(void (*fn)(void *), void *arg, void *sp)
  *
  * Change the stack to that pointed at by sp, then invoke fn(arg) with
  * the new stack.
+ *
+ * The sequence below follows the APCS frame convention for frame pointer
+ * unwinding, and implements the unwinder annotations needed by the EABI
+ * unwinder.
+ */

Quoted 3 lines

arch/arm/lib/call_with_stack.S line 25

Quoted 3 lines

arch/arm/lib/call_with_stack.S +3 -0

+
+#if defined(CONFIG_THUMB2_KERNEL) || \
+    (defined(CONFIG_UNWINDER_FRAME_POINTER) && defined(CONFIG_CC_IS_CLANG))

Doesn't clang use r11 (fp) as the frame pointer in ARM mode?
https://godbolt.org/z/1x4x99M1x
Or is this what you meant by "So the best we can do here is not touch the frame pointer at all"?

Quoted 6 lines

arch/arm/lib/call_with_stack.S line 31

Quoted 6 lines

arch/arm/lib/call_with_stack.S +6 -0

+/*
+ * Thumb-2 builds must use R7 as the frame pointer due to the way our unwind
+ * info based unwinder is constructed.
+ *
+ * The code below uses the GCC idiom for managing the frame pointer in the
+ * function prologue and epilogue, which Clang does not support. So the best we

IIRC, it's only slightly different; it's just that FP points to the previous FP in clang, rather than LR; at a fixed offset. At least when looking through Doug's notes and diagrams:

Though looking at the diagram, it looks like neither toolchain implements APCS...did I understand that correctly?

There's also some documentation in
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm/kernel/stacktrace.c#n11
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm/lib/backtrace-clang.S#n31

I guess I'm more so curious about this code when built with clang, both before and after this patch. Was it broken for either unwinder on ARM or THUMB2+UNWINDER_ARM? Does it regress with this patch? What's the best way to test/verify this?

Quoted 5 lines

arch/arm/lib/call_with_stack.S line 36

Quoted 5 lines

arch/arm/lib/call_with_stack.S +4 -0

+ * can do here is not touch the frame pointer at all: this will simply omit
+ * this frame when unwinding the call stack. So use R7 in this case as well,
+ * and leave R11 unmodified.
  */
+       fpreg   .req    r7

TIL about .req: https://sourceware.org/binutils/docs/as/ARM-Directives.html
This patch demonstrates the usage of quite a few of these!

Quoted 27 lines

Quoted 27 lines

arch/arm/lib/call_with_stack.S +16 -4

+#else
+       fpreg   .req    fp
+#endif
+
 ENTRY(call_with_stack)
-       str     sp, [r2, #-4]!
-       str     lr, [r2, #-4]!
+UNWIND(        .fnstart                )
+UNWIND(        .movsp  ip              )
+       mov     ip, sp
+
+UNWIND(        .pad    #4              )
+UNWIND(        .save   {fpreg, ip, lr} )
+THUMB( sub     sp, #4          )
+       push    {fpreg, ip, lr ARM(, pc)}
+
+UNWIND(        .setfp  fpreg, ip, #-4  )
+       sub     fpreg, ip, #4
 
        mov     sp, r2
        mov     r2, r0
        mov     r0, r1
 
-       badr    lr, 1f
-       ret     r2
+       bl_r    r2
 

Quoted 10 lines

Quoted 10 lines

arch/arm/lib/call_with_stack.S +3 -3

-1:     ldr     lr, [sp]
-       ldr     sp, [sp, #4]
+       ldmdb   fpreg, {fpreg, ip, lr}
+       mov     sp, ip
        ret     lr
+UNWIND(        .fnend                  )
 ENDPROC(call_with_stack)
--

2.30.2

--
Thanks,
~Nick Desaulniers

Now that we have enabled IRQ stacks, any softIRQs that are handled over the back of a hard IRQ will run from the IRQ stack as well. However, any synchronous softirq processing that happens when re-enabling softIRQs from task context will still execute on that task's stack.

So let's wire up the existing infrastructure to run these softIRQs from the IRQ stack as well.

Signed-off-by Ard Biesheuvel <[email protected]>

Quoted 9 lines · Ard Biesheuvel

Now that we have enabled IRQ stacks, any softIRQs that are handled over the back of a hard IRQ will run from the IRQ stack as well. However, any synchronous softirq processing that happens when re-enabling softIRQs from task context will still execute on that task's stack.

So let's wire up the existing infrastructure to run these softIRQs from the IRQ stack as well.

Quoted 1 line

Signed-off-by Ard Biesheuvel <[email protected]>

The change looks reasonable, but maybe try to explain why it is a good idea to do this.

Reviewed-by Arnd Bergmann <[email protected]>

Ard Biesheuvel (9):
ARM: remove some dead code
ARM: assembler: introduce bl_r and bl_m macros
ARM: optimize indirect call to handle_arch_irq for v7 cores
ARM: unwind: support unwinding across multiple stacks
ARM: export dump_mem() to other objects
ARM: unwind: dump exception stack from calling frame
ARM: implement IRQ stacks
ARM: call_with_stack: add unwind support
ARM: run softirqs on the per-CPU IRQ stack

Compared to user space, the kernel's task stacks are tiny and inflexible, as we don't grow them dynamically using demand paging. This is the reason we tend to obsess about functions with disproportionately large stack frames, given that it is hard to predict statically how calls to those functions may combine at runtime, and exhaust the stack and crash the kernel.