arm64: implement support for static call trampolines

This turned nasty really quickly when I realized that any sleeping task could have been interrupted right in the middle of the ADRP/ADD pair that we emit for static call targets that are out of immediate branching range.

+/*
+ * The static call trampoline consists of one of the following sequences:
+ *
+ * (A) (B) (C) (D) (E)

+ * 00: BTI  C        BTI  C        BTI  C        BTI  C        BTI  C
+ * 04: B    fn       NOP           NOP           NOP           NOP
+ * 08: RET           RET           ADRP X16, fn  ADRP X16, fn  ADRP X16, fn
+ * 0c: NOP           NOP           ADD  X16, fn  ADD  X16, fn  ADD  X16, fn
+ * 10:                             BR   X16      RET           NOP
+ * 14:                                                         ADRP X16, &fn
+ * 18:                                                         LDR  X16, [X16, &fn]
+ * 1c:                                                         BR   X16

+ *
+ * The architecture permits us to patch B instructions into NOPs or vice versa
+ * directly, but patching any other instruction sequence requires careful
+ * synchronization. Since branch targets may be out of range for ordinary
+ * immediate branch instructions, we may have to fall back to ADRP/ADD/BR
+ * sequences in some cases, which complicates things considerably; since any
+ * sleeping tasks may have been preempted right in the middle of any of these
+ * sequences, we have to carefully transform one into the other, and ensure
+ * that it is safe to resume execution at any point in the sequence for tasks
+ * that have already executed part of it.
+ *
+ * So the rules are:
+ * - we start out with (A) or (B)
+ * - a branch within immediate range can always be patched in at offset 0x4;
+ * - sequence (A) can be turned into (B) for NULL branch targets;
+ * - a branch outside of immediate range can be patched using (C), but only if
+ * . the sequence being updated is (A) or (B), or
+ * . the branch target address modulo 4k results in the same ADD opcode
+ * (which could occur when patching the same far target a second time)
+ * - once we have patched in (C) we cannot go back to (A) or (B), so patching
+ * in a NULL target now requires sequence (D);
+ * - if we cannot patch a far target using (C), we fall back to sequence (E),
+ * which loads the function pointer from memory.
+ *
+ * If we abide by these rules, then the following must hold for tasks that were
+ * interrupted halfway through execution of the trampoline:
+ * - when resuming at offset 0x8, we can only encounter a RET if (B) or (D)
+ * was patched in at any point, and therefore a NULL target is valid;
+ * - when resuming at offset 0xc, we are executing the ADD opcode that is only

+ * reachable via the preceding ADRP, and which is patched in only a single
+ * time, and is therefore guaranteed to be consistent with the ADRP target;
+ * - when resuming at offset 0x10, X16 must refer to a valid target, since it
+ * is only reachable via a ADRP/ADD pair that is guaranteed to be consistent.
+ *
+ * Note that sequence (E) is only used when switching between multiple far
+ * targets, and that it is not a terminal degraded state.
+ */

Quoted 41 lines · Ard Biesheuvel

+/*
+ * The static call trampoline consists of one of the following sequences:
+ *
+ * (A) (B) (C) (D) (E)

+ * 00: BTI  C        BTI  C        BTI  C        BTI  C        BTI  C
+ * 04: B    fn       NOP           NOP           NOP           NOP
+ * 08: RET           RET           ADRP X16, fn  ADRP X16, fn  ADRP X16, fn
+ * 0c: NOP           NOP           ADD  X16, fn  ADD  X16, fn  ADD  X16, fn
+ * 10:                             BR   X16      RET           NOP
+ * 14:                                                         ADRP X16, &fn
+ * 18:                                                         LDR  X16, [X16, &fn]
+ * 1c:                                                         BR   X16

+ *
+ * The architecture permits us to patch B instructions into NOPs or vice versa
+ * directly, but patching any other instruction sequence requires careful
+ * synchronization. Since branch targets may be out of range for ordinary
+ * immediate branch instructions, we may have to fall back to ADRP/ADD/BR
+ * sequences in some cases, which complicates things considerably; since any
+ * sleeping tasks may have been preempted right in the middle of any of these
+ * sequences, we have to carefully transform one into the other, and ensure
+ * that it is safe to resume execution at any point in the sequence for tasks
+ * that have already executed part of it.
+ *
+ * So the rules are:
+ * - we start out with (A) or (B)
+ * - a branch within immediate range can always be patched in at offset 0x4;
+ * - sequence (A) can be turned into (B) for NULL branch targets;
+ * - a branch outside of immediate range can be patched using (C), but only if
+ * . the sequence being updated is (A) or (B), or
+ * . the branch target address modulo 4k results in the same ADD opcode
+ * (which could occur when patching the same far target a second time)
+ * - once we have patched in (C) we cannot go back to (A) or (B), so patching
+ * in a NULL target now requires sequence (D);
+ * - if we cannot patch a far target using (C), we fall back to sequence (E),
+ * which loads the function pointer from memory.
+ *
+ * If we abide by these rules, then the following must hold for tasks that were
+ * interrupted halfway through execution of the trampoline:
+ * - when resuming at offset 0x8, we can only encounter a RET if (B) or (D)
+ * was patched in at any point, and therefore a NULL target is valid;
+ * - when resuming at offset 0xc, we are executing the ADD opcode that is only

Quoted 8 lines

+ * reachable via the preceding ADRP, and which is patched in only a single
+ * time, and is therefore guaranteed to be consistent with the ADRP target;
+ * - when resuming at offset 0x10, X16 must refer to a valid target, since it
+ * is only reachable via a ADRP/ADD pair that is guaranteed to be consistent.
+ *
+ * Note that sequence (E) is only used when switching between multiple far
+ * targets, and that it is not a terminal degraded state.
+ */

Would it make things easier if your trampoline consisted of two complete slots, between which you can flip?

Something like:

        0x00    B 0x24 / NOP
        0x04    < slot 1 >
                ....
        0x20
        0x24    < slot 2 >
                ....
        0x40

Then each (20 byte) slot can contain any of the variants above and you can write the unused slot without stop-machine. Then, when the unused slot is populated, flip the initial instruction (like a static-branch), issue synchronize_rcu_tasks() and flip to using the other slot for next time.

Alternatively, you can patch the call-sites to point to the alternative trampoline slot, but that might be pushing things a bit.

Quoted 13 lines · Peter Zijlstra

Would it make things easier if your trampoline consisted of two complete slots, between which you can flip?

Something like:

        0x00    B 0x24 / NOP
        0x04    < slot 1 >
                ....
        0x20
        0x24    < slot 2 >
                ....
        0x40

Quoted 6 lines

Then each (20 byte) slot can contain any of the variants above and you can write the unused slot without stop-machine. Then, when the unused slot is populated, flip the initial instruction (like a static-branch), issue synchronize_rcu_tasks() and flip to using the other slot for next time.

Once we've populated a slot and activated it, we have to assume that it is live and we can no longer modify it freely.

Quoted 22 lines · Ard Biesheuvel

Quoted 19 lines · Peter Zijlstra

Would it make things easier if your trampoline consisted of two complete slots, between which you can flip?

Something like:

        0x00    B 0x24 / NOP
        0x04    < slot 1 >
                ....
        0x20
        0x24    < slot 2 >
                ....
        0x40

Then each (20 byte) slot can contain any of the variants above and you can write the unused slot without stop-machine. Then, when the unused slot is populated, flip the initial instruction (like a static-branch), issue synchronize_rcu_tasks() and flip to using the other slot for next time.

Quoted 2 lines

Once we've populated a slot and activated it, we have to assume that it is live and we can no longer modify it freely.

Urhm how so? Once you pass through synchronize_rcu_tasks() nobody should still be using the old one.

Quoted 24 lines · Peter Zijlstra

Quoted 22 lines · Ard Biesheuvel

Quoted 19 lines · Peter Zijlstra

Would it make things easier if your trampoline consisted of two complete slots, between which you can flip?

Something like:

        0x00    B 0x24 / NOP
        0x04    < slot 1 >
                ....
        0x20
        0x24    < slot 2 >
                ....
        0x40

Then each (20 byte) slot can contain any of the variants above and you can write the unused slot without stop-machine. Then, when the unused slot is populated, flip the initial instruction (like a static-branch), issue synchronize_rcu_tasks() and flip to using the other slot for next time.

Quoted 5 lines

Quoted 2 lines

Once we've populated a slot and activated it, we have to assume that it is live and we can no longer modify it freely.

Urhm how so? Once you pass through synchronize_rcu_tasks() nobody should still be using the old one.

But doesn't that require some RCU foo in the static call wrapper itself?

Quoted 31 lines · Ard Biesheuvel

Quoted 29 lines · Peter Zijlstra

Quoted 22 lines · Ard Biesheuvel

Quoted 13 lines · Peter Zijlstra

Would it make things easier if your trampoline consisted of two complete slots, between which you can flip?

Something like:

        0x00    B 0x24 / NOP
        0x04    < slot 1 >
                ....
        0x20
        0x24    < slot 2 >
                ....
        0x40

Quoted 6 lines

Then each (20 byte) slot can contain any of the variants above and you can write the unused slot without stop-machine. Then, when the unused slot is populated, flip the initial instruction (like a static-branch), issue synchronize_rcu_tasks() and flip to using the other slot for next time.

Quoted 2 lines

Once we've populated a slot and activated it, we have to assume that it is live and we can no longer modify it freely.

Urhm how so? Once you pass through synchronize_rcu_tasks() nobody should still be using the old one.

Quoted 1 line

But doesn't that require some RCU foo in the static call wrapper itself?

Nope, synchronize_rcu_tasks() ensures that every task will pass a safe point (aka. voluntary schedule()) before returning. This then guarantees that, no matter where they were preempted, they'll have left (and will observe the new state).

arch/arm64/include/asm/static_call.h new file +14 -0

new file mode 100644
index 000000000000..7ddf939d57f5
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_STATIC_CALL_H
+#define _ASM_STATIC_CALL_H
+
+/*
+ * We have to account for the possibility that the static call site may
+ * be updated to refer to a target that is out of range for an ordinary
+ * 'B' branch instruction, and so we need to pre-allocate some space for
+ * a ADRP/ADD/BR sequence.
+ */
+#define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, insn)			    \
+	asm(".pushsection	.static_call.text, \"ax\"		\n" \
+	    ".align		5					\n" \
+	    ".globl		" STATIC_CALL_TRAMP_STR(name) "		\n" \

Hi Ard,

Quoted 20 lines · Ard Biesheuvel

arch/arm64/include/asm/static_call.h new file +14 -0

new file mode 100644
index 000000000000..7ddf939d57f5
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_STATIC_CALL_H
+#define _ASM_STATIC_CALL_H
+
+/*
+ * We have to account for the possibility that the static call site may
+ * be updated to refer to a target that is out of range for an ordinary
+ * 'B' branch instruction, and so we need to pre-allocate some space for
+ * a ADRP/ADD/BR sequence.
+ */
+#define __ARCH_DEFINE_STATIC_CALL_TRAMP(name, insn)                      \
+     asm(".pushsection       .static_call.text, \"ax\"               \n" \
+         ".align             5                                       \n" \
+         ".globl             " STATIC_CALL_TRAMP_STR(name) "         \n" \

arch/arm64/include/asm/static_call.h line 24

Quoted 10 lines

arch/arm64/include/asm/static_call.h new file +10 -0

+         STATIC_CALL_TRAMP_STR(name) ":                              \n" \
+         "hint       34      /* BTI C */                             \n" \
+         insn "                                                      \n" \
+         "ret                                                        \n" \
+         "nop                                                        \n" \
+         "nop                                                        \n" \
+         "adrp       x16, " STATIC_CALL_KEY(name) "                  \n" \
+         "ldr        x16, [x16, :lo12:" STATIC_CALL_KEY(name) "]     \n" \
+         "br         x16                                             \n" \
+         ".popsection                                                \n")

Still trying to understand all this in details, so bear with me, but is there any way this could be turned into the 'inline' static call variant?

That is, could we have a piece of inline assembly at all static_call locations that would do a branch-link, with basically x0-x18 and x29,x30 in the clobber list (+ some have some magic to place the parameters in the right register, like we do for SMCCC calls for instance). That'd save a branch to the trampoline.

Ideally we'd need a way to tell the compile 'this inline assembly code does a function call', but I'm not aware of any easy way to do that (other that marking register clobbered + probably other things that I'm missing).

In any case, I was looking forward to an arm64 static call port so thanks for putting that together.

However, as I discussed with Will offline yesterday as well, the question that got snowed under is whether we need any of this on arm64 in the first place. It seems highly unlikely that inline static calls are worth it, and even out-of-line static calls are probably not worth the hassle as we don't have the retpoline problem.

So this code should be considered an invitation for discussion, and perhaps someone can invent a use case where benchmarks can show a worthwhile improvement. But let's not get ahead of ourselves.

IIRC Steve had benchmarks for the ftrace conversion, which is now upstream, so that should be simple enough to run.

Steve, remember how to get numbers out of that?

We'll need tooling along the lines of the GCC plugin I wrote [0] to implement inline static calls - doing function calls from inline assembly is too messy, too fragile and too error prone to rely on.

However, as I discussed with Will offline yesterday as well, the question that got snowed under is whether we need any of this on arm64 in the first place. It seems highly unlikely that inline static calls are worth it, and even out-of-line static calls are probably not worth the hassle as we don't have the retpoline problem.

So this code should be considered an invitation for discussion, and perhaps someone can invent a use case where benchmarks can show a worthwhile improvement. But let's not get ahead of ourselves.

Quoted 3 lines · Ard Biesheuvel

We'll need tooling along the lines of the GCC plugin I wrote [0] to implement inline static calls - doing function calls from inline assembly is too messy, too fragile and too error prone to rely on.

Right, and that is the gut feeling I had too, but I can't quite put my finger on what exactly can go wrong. Any pointers?

Quoted 9 lines

However, as I discussed with Will offline yesterday as well, the question that got snowed under is whether we need any of this on arm64 in the first place. It seems highly unlikely that inline static calls are worth it, and even out-of-line static calls are probably not worth the hassle as we don't have the retpoline problem.

So this code should be considered an invitation for discussion, and perhaps someone can invent a use case where benchmarks can show a worthwhile improvement. But let's not get ahead of ourselves.

The reason I'm interested in this is because Android makes heavy use of trace points/hooks, so any potential improvement in this area would be welcome. Now I agree we need numbers to show the benefit is real before this can be considered for inclusion in the kernel. I'll try and see if we can get something.

Thanks,
Quentin

The reason I'm interested in this is because Android makes heavy use of trace points/hooks, so any potential improvement in this area would be welcome. Now I agree we need numbers to show the benefit is real before this can be considered for inclusion in the kernel. I'll try and see if we can get something.

Quoted 5 lines · Quentin Perret

The reason I'm interested in this is because Android makes heavy use of trace points/hooks, so any potential improvement in this area would be welcome. Now I agree we need numbers to show the benefit is real before this can be considered for inclusion in the kernel. I'll try and see if we can get something.

Following up on this as we've just figured out what was causing performance issues in our use-case. Basically, we have a setup where some modules attach to trace hooks for a few things (e.g. the pelt scheduler hooks + other Android-specific hooks), and that appeared to cause up ~6% perf regression on the Androbench benchmark.

The bulk of the regression came from a feature that is currently Android-specific but should hopefully make it upstream (soon?): Control Flow Integrity (CFI) -- see [1] for more details. In essence CFI is a software-based cousin of BTI, which is basically about ensuring the target of an indirect function call has a compatible prototype. This can be relatively easily checked for potential targets that are known at compile-time, but is a little harder when the targets are dynamically loaded, hence causing extra overhead when the target is in a module.

Anyway, I don't think any of the above is particularly relevant to upstream just yet, but I figured this would interesting to share. The short-term fix for Android was to locally disable CFI checks around the trace hooks that cause the perf regression, but I think static-calls would be a preferable alternative to that (I'll try to confirm that experimentally). And when/if CFI makes it upstream, then that may become relevant to upstream as well, though the integration of CFI and static-calls is not very clear yet.

OK, so that would suggest that having at least the out-of-line trampoline would help with CFI, but only because the indirect call is decorated with CFI checks, not because the indirect call itself is any slower.

So that suggests that something like

  bti    c
  ldr    x16, 0f
  br     x16
0:.quad  <target>

may well be sufficient in the arm64 case - it is hidden from the assembler, so we don't get the CFI overhead, and since it is emitted as .text (and therefore requires code patching to be updated), it does not need the same level of protection that CFI offers elsewhere when it comes to indirect calls.

Implement arm64 support for the 'unoptimized' static call variety, which routes all calls through a single trampoline that is patched to perform a tail call to the selected function.

Since static call targets may be located in modules loaded out of direct branching range, we need to be able to fall back to issuing a ADRP/ADD pair to load the branch target into R16 and use a BR instruction. As this involves patching more than a single B or NOP instruction (for which the architecture makes special provisions in terms of the synchronization needed), we may need to run the full blown instruction patching logic that uses stop_machine(). It also means that once we've patched in a ADRP/ADD pair once, we are quite restricted in the patching we can code subsequently, and we may end up using an indirect call after all (note that

v2:
This turned nasty really quickly when I realized that any sleeping task could have been interrupted right in the middle of the ADRP/ADD pair that we emit for static call targets that are out of immediate branching range.

+/*
+ * The static call trampoline consists of one of the following sequences:
+ *
+ * (A) (B) (C) (D) (E)

+ * 00: BTI  C        BTI  C        BTI  C        BTI  C        BTI  C
+ * 04: B    fn       NOP           NOP           NOP           NOP
+ * 08: RET           RET           ADRP X16, fn  ADRP X16, fn  ADRP X16, fn
+ * 0c: NOP           NOP           ADD  X16, fn  ADD  X16, fn  ADD  X16, fn
+ * 10:                             BR   X16      RET           NOP
+ * 14:                                                         ADRP X16, &fn
+ * 18:                                                         LDR  X16, [X16, &fn]
+ * 1c:                                                         BR   X16

+ *
+ * The architecture permits us to patch B instructions into NOPs or vice versa
+ * directly, but patching any other instruction sequence requires careful
+ * synchronization. Since branch targets may be out of range for ordinary
+ * immediate branch instructions, we may have to fall back to ADRP/ADD/BR
+ * sequences in some cases, which complicates things considerably; since any
+ * sleeping tasks may have been preempted right in the middle of any of these
+ * sequences, we have to carefully transform one into the other, and ensure
+ * that it is safe to resume execution at any point in the sequence for tasks
+ * that have already executed part of it.
+ *
+ * So the rules are:
+ * - we start out with (A) or (B)
+ * - a branch within immediate range can always be patched in at offset 0x4;
+ * - sequence (A) can be turned into (B) for NULL branch targets;

+ * - a branch outside of immediate range can be patched using (C), but only if
+ * . the sequence being updated is (A) or (B), or
+ * . the branch target address modulo 4k results in the same ADD opcode
+ * (which could occur when patching the same far target a second time)
+ * - once we have patched in (C) we cannot go back to (A) or (B), so patching
+ * in a NULL target now requires sequence (D);
+ * - if we cannot patch a far target using (C), we fall back to sequence (E),
+ * which loads the function pointer from memory.

+ * Note that sequence (E) is only used when switching between multiple far
+ * targets, and that it is not a terminal degraded state.

Hi Ard,

Quoted 3 lines · Ard Biesheuvel

Implement arm64 support for the 'unoptimized' static call variety, which routes all calls through a single trampoline that is patched to perform a tail call to the selected function.

Given the complexity and subtlety here, do we actually need this?

Quoted 7 lines · Mark Rutland

Hi Ard,

Quoted 3 lines · Ard Biesheuvel

Implement arm64 support for the 'unoptimized' static call variety, which routes all calls through a single trampoline that is patched to perform a tail call to the selected function.

Quoted 1 line

Given the complexity and subtlety here, do we actually need this?

Only if you can get a performance win. The obvious benefit is loosing the load that's inherent in indirect function calls. The down-side of the indirect static-call implementation is that it will incur an extra I$ miss.

So it might be a wash, loose a data load miss, gain an I$ miss.

The direct method (patching the call-site, where possible) would alleviate that (mostly) and be more of a win.

Hi Ard,

Quoted 3 lines · Ard Biesheuvel

Implement arm64 support for the 'unoptimized' static call variety, which routes all calls through a single trampoline that is patched to perform a tail call to the selected function.

Given the complexity and subtlety here, do we actually need this?

If there's some core feature that depends on this (or wants to), it'd be nice to call that out in the commit message.

Quoted 10 lines

Since static call targets may be located in modules loaded out of direct branching range, we need to be able to fall back to issuing a ADRP/ADD pair to load the branch target into R16 and use a BR instruction. As this involves patching more than a single B or NOP instruction (for which the architecture makes special provisions in terms of the synchronization needed), we may need to run the full blown instruction patching logic that uses stop_machine(). It also means that once we've patched in a ADRP/ADD pair once, we are quite restricted in the patching we can code subsequently, and we may end up using an indirect call after all (note that

Noted. I guess we

[...]

Quoted 10 lines

v2:
This turned nasty really quickly when I realized that any sleeping task could have been interrupted right in the middle of the ADRP/ADD pair that we emit for static call targets that are out of immediate branching range.

+/*
+ * The static call trampoline consists of one of the following sequences:
+ *
+ * (A) (B) (C) (D) (E)

Quoted 8 lines

+ * 00: BTI  C        BTI  C        BTI  C        BTI  C        BTI  C
+ * 04: B    fn       NOP           NOP           NOP           NOP
+ * 08: RET           RET           ADRP X16, fn  ADRP X16, fn  ADRP X16, fn
+ * 0c: NOP           NOP           ADD  X16, fn  ADD  X16, fn  ADD  X16, fn
+ * 10:                             BR   X16      RET           NOP
+ * 14:                                                         ADRP X16, &fn
+ * 18:                                                         LDR  X16, [X16, &fn]
+ * 1c:                                                         BR   X16

Are these all padded with trailing NOPs or UDFs? I assume the same space is statically reserved.

Quoted 15 lines

+ *
+ * The architecture permits us to patch B instructions into NOPs or vice versa
+ * directly, but patching any other instruction sequence requires careful
+ * synchronization. Since branch targets may be out of range for ordinary
+ * immediate branch instructions, we may have to fall back to ADRP/ADD/BR
+ * sequences in some cases, which complicates things considerably; since any
+ * sleeping tasks may have been preempted right in the middle of any of these
+ * sequences, we have to carefully transform one into the other, and ensure
+ * that it is safe to resume execution at any point in the sequence for tasks
+ * that have already executed part of it.
+ *
+ * So the rules are:
+ * - we start out with (A) or (B)
+ * - a branch within immediate range can always be patched in at offset 0x4;
+ * - sequence (A) can be turned into (B) for NULL branch targets;

Quoted 8 lines

+ * - a branch outside of immediate range can be patched using (C), but only if
+ * . the sequence being updated is (A) or (B), or
+ * . the branch target address modulo 4k results in the same ADD opcode
+ * (which could occur when patching the same far target a second time)
+ * - once we have patched in (C) we cannot go back to (A) or (B), so patching
+ * in a NULL target now requires sequence (D);
+ * - if we cannot patch a far target using (C), we fall back to sequence (E),
+ * which loads the function pointer from memory.

Cases C-E all use an indirect branch, which goes against one of the arguments for having static calls (the assumption that CPUs won't mis-predict direct branches). Similarly case E is a literal pool with more steps.

That means that for us, static calls would only be an opportunistic optimization rather than a hardening feature. Do they actually save us much, or could we get by with an inline literal pool in the trampoline?

It'd be much easier to reason consistently if the trampoline were always:

|       BTI C
|       LDR X16, _literal // pc-relative load
|       BR X16
| _literal:
|       < patch a 64-bit value here atomically >

... and I would strongly prefer that to having multiple sequences that could all be live -- I'm really not keen on the complexity and subtlety that entails.

[...]

Quoted 2 lines

+ * Note that sequence (E) is only used when switching between multiple far
+ * targets, and that it is not a terminal degraded state.

I think what you're saying here is that you can go from (E) to (C) if switching to a new far branch that's in ADRP+ADD range, but it can be misread to mean (E) is a transient step while patching a branch rather than some branches only being possible to encode with (E).

Quoted 6 lines · Mark Rutland

Quoted 3 lines · Ard Biesheuvel

Implement arm64 support for the 'unoptimized' static call variety, which routes all calls through a single trampoline that is patched to perform a tail call to the selected function.

Given the complexity and subtlety here, do we actually need this?

The more time I spend on this, the more I lean towards 'no' :-)

Quoted 11 lines

Quoted 10 lines

Since static call targets may be located in modules loaded out of direct branching range, we need to be able to fall back to issuing a ADRP/ADD pair to load the branch target into R16 and use a BR instruction. As this involves patching more than a single B or NOP instruction (for which the architecture makes special provisions in terms of the synchronization needed), we may need to run the full blown instruction patching logic that uses stop_machine(). It also means that once we've patched in a ADRP/ADD pair once, we are quite restricted in the patching we can code subsequently, and we may end up using an indirect call after all (note that

Quoted 4 lines

Noted. I guess we

[...]

?

Quoted 29 lines

Quoted 15 lines

+ *
+ * The architecture permits us to patch B instructions into NOPs or vice versa
+ * directly, but patching any other instruction sequence requires careful
+ * synchronization. Since branch targets may be out of range for ordinary
+ * immediate branch instructions, we may have to fall back to ADRP/ADD/BR
+ * sequences in some cases, which complicates things considerably; since any
+ * sleeping tasks may have been preempted right in the middle of any of these
+ * sequences, we have to carefully transform one into the other, and ensure
+ * that it is safe to resume execution at any point in the sequence for tasks
+ * that have already executed part of it.
+ *
+ * So the rules are:
+ * - we start out with (A) or (B)
+ * - a branch within immediate range can always be patched in at offset 0x4;
+ * - sequence (A) can be turned into (B) for NULL branch targets;

Quoted 8 lines

+ * - a branch outside of immediate range can be patched using (C), but only if
+ * . the sequence being updated is (A) or (B), or
+ * . the branch target address modulo 4k results in the same ADD opcode
+ * (which could occur when patching the same far target a second time)
+ * - once we have patched in (C) we cannot go back to (A) or (B), so patching
+ * in a NULL target now requires sequence (D);
+ * - if we cannot patch a far target using (C), we fall back to sequence (E),
+ * which loads the function pointer from memory.

Cases C-E all use an indirect branch, which goes against one of the arguments for having static calls (the assumption that CPUs won't mis-predict direct branches). Similarly case E is a literal pool with more steps.

Quoted 3 lines

That means that for us, static calls would only be an opportunistic optimization rather than a hardening feature. Do they actually save us much, or could we get by with an inline literal pool in the trampoline?

Another assumption this is based on is that a literal load is more costly than a ADRP/ADD.

Quoted 9 lines

It'd be much easier to reason consistently if the trampoline were always:

|       BTI C
|       LDR X16, _literal // pc-relative load
|       BR X16
| _literal:
|       < patch a 64-bit value here atomically >

Quoted 3 lines

... and I would strongly prefer that to having multiple sequences that could all be live -- I'm really not keen on the complexity and subtlety that entails.

I don't see this having any benefit over a ADRP/LDR pair that accesses the static call key struct directly.