Skip to contentSkip to conversation

Out-of-line static calls for powerpc64 ELF V2

PATCH In review Benjamin Gray 2 participants 11 messages Oct 10, 2022 – Oct 25, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v4 cover Out-of-line static calls for powerpc64 ELF V2

Implementation of out-of-line static calls for PowerPC 64-bit ELF V2 ABI. Static calls patch an indirect branch into a direct branch at runtime. Out-of-line specifically has a caller directly call a trampoline, and the trampoline gets patched to directly call the target.

Previous versions here:
V3: https://lore.kernel.org/all/[email protected]/
V2: https://lore.kernel.org/all/[email protected]/
V1: https://lore.kernel.org/all/[email protected]/
RFC: https://lore.kernel.org/linuxppc-dev/[email protected]/

Still rebased on top of
https://lore.kernel.org/all/0df84a2eea551c1d000c34c36d0c1d23cbe26a97.1664289176.git.christophe.leroy@csgroup.eu/

Changed for V4:

[1/6]
* IS_ENABLED(CONFIG_PPC64) added to patch size condition for better compiler optimisation

[2/6]
* Reworded message
* Refactored need r2 save expression. Parens are intentional, C operator precedence can be difficult to remember, especially when three different kinds are used in a single expression.

Benjamin Gray (6):
powerpc/code-patching: Implement generic text patching function powerpc/module: Handle caller-saved TOC in module linker powerpc/module: Optimise nearby branches in ELF V2 ABI stub
static_call: Move static call selftest to static_call_selftest.c powerpc/64: Add support for out-of-line static calls
powerpc: Add tests for out-of-line static calls

 arch/powerpc/Kconfig                     |  26 ++-
 arch/powerpc/include/asm/code-patching.h |  30 +++
 arch/powerpc/include/asm/static_call.h   |  80 ++++++-
 arch/powerpc/kernel/Makefile             |   4 +-
 arch/powerpc/kernel/module_64.c          |  26 ++-
 arch/powerpc/kernel/static_call.c        | 183 +++++++++++++++-
 arch/powerpc/kernel/static_call_test.c   | 263 +++++++++++++++++++++++
 arch/powerpc/kernel/static_call_test.h   |  56 +++++
 arch/powerpc/lib/code-patching.c         |  73 +++++--
 kernel/Makefile                          |   1 +
 kernel/static_call_inline.c              |  43 ----
 kernel/static_call_selftest.c            |  41 ++++
 12 files changed, 753 insertions(+), 73 deletions(-)
 create mode 100644 arch/powerpc/kernel/static_call_test.c
 create mode 100644 arch/powerpc/kernel/static_call_test.h
 create mode 100644 kernel/static_call_selftest.c

base-commit: 86de6882270620777fe9cc16ece97e9ee407122c --
2.37.3

#1 lore raw

PATCH v4 1/6 powerpc/code-patching: Implement generic text patching function

Adds a generic memory patching mechanism for patches of size int or long bytes.

The patch_instruction function is reimplemented in terms of this more generic function. This generic implementation allows patching of arbitrary long data, such as pointers on 64-bit. As a performance optimisation the patch will unconditionally flush the icache, as patch_instruction is used significantly more often and in more performance sensitive contexts (e.g., ftrace activation).

On 32-bit patch_int is marked noinline to prevent a mis-optimisation. Without noinline, inside patch_branch the compiler may inline all the way to do_patch_memory, preventing the compiler from inlining do_patch_memory into patch_int. This would needlessly force patch_int to be a branch to do_patch_memory.

The 'IS_ENABLED(CONFIG_PPC64) && ' part of the patch size condition helps GCC inline __patch_memory properly. Only 64-bit requires doubleword writes anyway, as ints and longs are the same size on 32-bit.

---
 arch/powerpc/include/asm/code-patching.h | 29 ++++++++++
 arch/powerpc/lib/code-patching.c         | 73 ++++++++++++++++++------
 2 files changed, 85 insertions(+), 17 deletions(-)
arch/powerpc/include/asm/code-patching.h +29 -0
index 3f881548fb61..170bfa848c7c 100644
@@ -72,7 +72,36 @@ static inline int create_branch(ppc_inst_t *instr, const u32 *addr,
 int create_cond_branch(ppc_inst_t *instr, const u32 *addr,
 		       unsigned long target, int flags);
 int patch_branch(u32 *addr, unsigned long target, int flags);
+
+/* patch_uint and patch_ulong must only be called on addresses where the patch
+ * does not cross a cacheline, otherwise it may not be flushed properly and
+ * mixes of new and stale data may be observed.
+ *
+ * patch_instruction and other instruction patchers automatically satisfy this
+ * requirement due to instruction alignment requirements.
+ */
+
+int patch_uint(void *addr, unsigned int val);
+
+#ifdef CONFIG_PPC64
+
+int patch_ulong(void *addr, unsigned long val);
 int patch_instruction(u32 *addr, ppc_inst_t instr);
+
+#else
+
+static inline int patch_ulong(void *addr, unsigned long val)
+{
+	return patch_uint(addr, val);
+}
+
+static inline int patch_instruction(u32 *addr, ppc_inst_t instr)
+{
+	return patch_uint(addr, ppc_inst_val(instr));
+}
+
+#endif
+
 int raw_patch_instruction(u32 *addr, ppc_inst_t instr);
 
 static inline unsigned long patch_site_addr(s32 *site)
arch/powerpc/lib/code-patching.c +56 -17

119 lines not on this page · raw message

Signature · 1 line
2.37.3
#2 lore raw

Re: [PATCH v4 1/6] powerpc/code-patching: Implement generic text patching function

Quoted 17 lines · Benjamin Gray

Adds a generic memory patching mechanism for patches of size int or long
bytes.

The patch_instruction function is reimplemented in terms of this more generic function. This generic implementation allows patching of arbitrary long data, such as pointers on 64-bit. As a performance optimisation the patch will unconditionally flush the icache, as patch_instruction is used significantly more often and in more performance sensitive contexts (e.g., ftrace activation).

On 32-bit patch_int is marked noinline to prevent a mis-optimisation. Without noinline, inside patch_branch the compiler may inline all the way to do_patch_memory, preventing the compiler from inlining do_patch_memory into patch_int. This would needlessly force patch_int to be a branch to do_patch_memory.

Quoted 6 lines

The 'IS_ENABLED(CONFIG_PPC64) && ' part of the patch size condition helps GCC inline __patch_memory properly. Only 64-bit requires doubleword writes anyway, as ints and longs are the same size on 32- bit.

It looks like this needs rebasing.

Otherwise I gave this a read over and didn't spot anything, but I'll hold off on a reviewed-by tag until it's rebased.

Quoted 198 lines

---
 arch/powerpc/include/asm/code-patching.h | 29 ++++++++++ arch/powerpc/lib/code-patching.c         | 73 ++++++++++++++++++---- --

 2 files changed, 85 insertions(+), 17 deletions(-)
arch/powerpc/include/asm/code-patching.h +2 -0
index 3f881548fb61..170bfa848c7c 100644
@@ -72,7 +72,36 @@ static inline int create_branch(ppc_inst_t *instr,
 const u32 *addr,
  int create_cond_branch(ppc_inst_t *instr, const u32 *addr,
                        unsigned long target, int flags);
  int patch_branch(u32 *addr, unsigned long target, int flags);
+
+/* patch_uint and patch_ulong must only be called on addresses where

the patch
+ * does not cross a cacheline, otherwise it may not be flushed properly and
+ * mixes of new and stale data may be observed.
+ *
+ * patch_instruction and other instruction patchers automatically satisfy this
+ * requirement due to instruction alignment requirements.
+ */

arch/powerpc/include/asm/code-patching.h +6 -0
+
+int patch_uint(void *addr, unsigned int val);
+
+#ifdef CONFIG_PPC64
+
+int patch_ulong(void *addr, unsigned long val);

 int patch_instruction(u32 *addr, ppc_inst_t instr);

arch/powerpc/include/asm/code-patching.h +15 -0
+
+#else
+
+static inline int patch_ulong(void *addr, unsigned long val)
+{
+       return patch_uint(addr, val);
+}
+
+static inline int patch_instruction(u32 *addr, ppc_inst_t instr)
+{
+       return patch_uint(addr, ppc_inst_val(instr));
+}
+
+#endif
+

 int raw_patch_instruction(u32 *addr, ppc_inst_t instr);
 static inline unsigned long patch_site_addr(s32 *site)

arch/powerpc/lib/code-patching.c +0 -0
index 125c55e3e148..e41c0ccec79f 100644
@@ -15,20 +15,24 @@
  #include <asm/code-patching.h>
  #include <asm/inst.h>


-static int __patch_instruction(u32 *exec_addr, ppc_inst_t instr, u32 *patch_addr)
+static int __patch_memory(void *patch_addr, unsigned long val, void *exec_addr,
+                         bool is_dword)
 {
-       if (!ppc_inst_prefixed(instr)) {
-               u32 val = ppc_inst_val(instr);
-
-               __put_kernel_nofault(patch_addr, &val, u32, failed); -       } else {
-               u64 val = ppc_inst_as_ulong(instr);
+       /* Prefixed instruction may cross cacheline if cacheline smaller than 64 bytes */
+       BUILD_BUG_ON(IS_ENABLED(CONFIG_PPC64) && L1_CACHE_BYTES < 64);

+       if (IS_ENABLED(CONFIG_PPC64) && unlikely(is_dword)) __put_kernel_nofault(patch_addr, &val, u64, failed); -       }
+       else
+               __put_kernel_nofault(patch_addr, &val, u32, failed);
-       asm ("dcbst 0, %0; sync; icbi 0,%1; sync; isync" :: "r" (patch_addr),
-                                                           "r" (exec_addr));
+       /* Assume data is inside a single cacheline */
+       dcbst(patch_addr);
+       mb(); /* sync */
+       /* Flush on the EA that may be executed in case of a non- coherent icache */
+       icbi(exec_addr);
+       mb(); /* sync */
+       isync();

        return 0;

arch/powerpc/lib/code-patching.c +0 -0
@@ -38,7 +42,10 @@ static int __patch_instruction(u32 *exec_addr,

ppc_inst_t instr, u32 *patch_addr

 int raw_patch_instruction(u32 *addr, ppc_inst_t instr)
 {
-       return __patch_instruction(addr, instr, addr);
+       if (ppc_inst_prefixed(instr))
+               return __patch_memory(addr, ppc_inst_as_ulong(instr), addr, true);
+       else
+               return __patch_memory(addr, ppc_inst_val(instr), addr, false);
 }

 static DEFINE_PER_CPU(struct vm_struct *, text_poke_area);

arch/powerpc/lib/code-patching.c +0 -0
@@ -149,7 +156,7 @@ static void unmap_patch_area(unsigned long addr)
         flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
  }

arch/powerpc/lib/code-patching.c +1 -1
-static int __do_patch_instruction(u32 *addr, ppc_inst_t instr)
+static int __do_patch_memory(void *addr, unsigned long val, bool

is_dword)
 {
        int err;
        u32 *patch_addr;

arch/powerpc/lib/code-patching.c +0 -0
@@ -166,7 +173,7 @@ static int __do_patch_instruction(u32 *addr,
 ppc_inst_t instr)
         if (radix_enabled())
                 asm volatile("ptesync": : :"memory");


-       err = __patch_instruction(addr, instr, patch_addr); +       err = __patch_memory(patch_addr, val, addr, is_dword);
        pte_clear(&init_mm, text_poke_addr, pte); flush_tlb_kernel_range(text_poke_addr, text_poke_addr + PAGE_SIZE);

arch/powerpc/lib/code-patching.c +0 -0
@@ -174,7 +181,7 @@ static int __do_patch_instruction(u32 *addr,
 ppc_inst_t instr)
         return err;
  }

arch/powerpc/lib/code-patching.c +1 -1
-int patch_instruction(u32 *addr, ppc_inst_t instr)
+static int do_patch_memory(void *addr, unsigned long val, bool

is_dword)
 {
        int err;
        unsigned long flags;

arch/powerpc/lib/code-patching.c +0 -0
@@ -186,15 +193,47 @@ int patch_instruction(u32 *addr, ppc_inst_t
 instr)
          */

        if (!IS_ENABLED(CONFIG_STRICT_KERNEL_RWX) || !static_branch_likely(&poking_init_done))
-               return raw_patch_instruction(addr, instr); +               return __patch_memory(addr, val, addr, is_dword);
        local_irq_save(flags);
-       err = __do_patch_instruction(addr, instr);
+       err = __do_patch_memory(addr, val, is_dword); local_irq_restore(flags);

        return err;
 }

arch/powerpc/lib/code-patching.c +33 -1
-NOKPROBE_SYMBOL(patch_instruction);
+
+#ifdef CONFIG_PPC64
+
+int patch_uint(void *addr, unsigned int val)
+{
+       return do_patch_memory(addr, val, false);
+}
+NOKPROBE_SYMBOL(patch_uint)
+
+int patch_ulong(void *addr, unsigned long val)
+{
+       return do_patch_memory(addr, val, true);
+}
+NOKPROBE_SYMBOL(patch_ulong)
+
+int patch_instruction(u32 *addr, ppc_inst_t instr)
+{
+       if (ppc_inst_prefixed(instr))
+               return patch_ulong(addr, ppc_inst_as_ulong(instr));
+       else
+               return patch_uint(addr, ppc_inst_val(instr));
+}
+NOKPROBE_SYMBOL(patch_instruction)
+
+#else
+
+noinline int patch_uint(void *addr, unsigned int val)
+{
+       return do_patch_memory(addr, val, false);
+}
+NOKPROBE_SYMBOL(patch_uint)
+
+#endif


 int patch_branch(u32 *addr, unsigned long target, int flags) {

Signature · 2 lines
Andrew Donnellan    OzLabs, ADL Canberra
[email protected]   IBM Australia Limited
Reviewed-by
#3 lore raw

Re: [PATCH v4 1/6] powerpc/code-patching: Implement generic text patching function

Quoted 28 lines · Andrew Donnellan
Quoted 27 lines · Benjamin Gray

Adds a generic memory patching mechanism for patches of size int or long
bytes.

The patch_instruction function is reimplemented in terms of this more generic function. This generic implementation allows patching of
arbitrary long data, such as pointers on 64-bit. As a performance optimisation the patch will unconditionally flush the icache, as patch_instruction is used significantly more often and in more performance sensitive contexts (e.g., ftrace activation).

On 32-bit patch_int is marked noinline to prevent a mis- optimisation.
Without noinline, inside patch_branch the compiler may inline all the
way to do_patch_memory, preventing the compiler from inlining do_patch_memory into patch_int. This would needlessly force patch_int
to be a branch to do_patch_memory.

The 'IS_ENABLED(CONFIG_PPC64) && ' part of the patch size condition helps GCC inline __patch_memory properly. Only 64-bit requires doubleword writes anyway, as ints and longs are the same size on 32-
bit.

Quoted 3 lines
Quoted 1 line

It looks like this needs rebasing.

Alternatively, I just need to learn to read cover letters properly and notice that this is based off another series...

Quoted 4 lines

Otherwise I gave this a read over and didn't spot anything, but I'll hold off on a reviewed-by tag until it's rebased.

Quoted 209 lines
Quoted 208 lines

---
 arch/powerpc/include/asm/code-patching.h | 29 ++++++++++ arch/powerpc/lib/code-patching.c         | 73 ++++++++++++++++++-- --
--

 2 files changed, 85 insertions(+), 17 deletions(-)
arch/powerpc/include/asm/code-patching.h +2 -0
index 3f881548fb61..170bfa848c7c 100644
@@ -72,7 +72,36 @@ static inline int create_branch(ppc_inst_t
 *instr,
 const u32 *addr,
  int create_cond_branch(ppc_inst_t *instr, const u32 *addr,
                        unsigned long target, int flags);
  int patch_branch(u32 *addr, unsigned long target, int flags);
+
+/* patch_uint and patch_ulong must only be called on addresses

where
the patch
+ * does not cross a cacheline, otherwise it may not be flushed properly and
+ * mixes of new and stale data may be observed.
+ *
+ * patch_instruction and other instruction patchers automatically satisfy this
+ * requirement due to instruction alignment requirements.
+ */

arch/powerpc/include/asm/code-patching.h +6 -0
+
+int patch_uint(void *addr, unsigned int val);
+
+#ifdef CONFIG_PPC64
+
+int patch_ulong(void *addr, unsigned long val);

 int patch_instruction(u32 *addr, ppc_inst_t instr);

arch/powerpc/include/asm/code-patching.h +15 -0
+
+#else
+
+static inline int patch_ulong(void *addr, unsigned long val)
+{
+       return patch_uint(addr, val);
+}
+
+static inline int patch_instruction(u32 *addr, ppc_inst_t instr)
+{
+       return patch_uint(addr, ppc_inst_val(instr));
+}
+
+#endif
+

 int raw_patch_instruction(u32 *addr, ppc_inst_t instr);
 static inline unsigned long patch_site_addr(s32 *site)

arch/powerpc/lib/code-patching.c +0 -0
index 125c55e3e148..e41c0ccec79f 100644
@@ -15,20 +15,24 @@
  #include <asm/code-patching.h>
  #include <asm/inst.h>


-static int __patch_instruction(u32 *exec_addr, ppc_inst_t instr, u32
*patch_addr)
+static int __patch_memory(void *patch_addr, unsigned long val, void
*exec_addr,
+                         bool is_dword)
 {
-       if (!ppc_inst_prefixed(instr)) {
-               u32 val = ppc_inst_val(instr);
-
-               __put_kernel_nofault(patch_addr, &val, u32, failed);
-       } else {
-               u64 val = ppc_inst_as_ulong(instr);
+       /* Prefixed instruction may cross cacheline if cacheline smaller than 64 bytes */
+       BUILD_BUG_ON(IS_ENABLED(CONFIG_PPC64) && L1_CACHE_BYTES < 64);

+       if (IS_ENABLED(CONFIG_PPC64) && unlikely(is_dword)) __put_kernel_nofault(patch_addr, &val, u64, failed);
-       }
+       else
+               __put_kernel_nofault(patch_addr, &val, u32, failed);

-       asm ("dcbst 0, %0; sync; icbi 0,%1; sync; isync" :: "r" (patch_addr),
-                                                           "r" (exec_addr));
+       /* Assume data is inside a single cacheline */ +       dcbst(patch_addr);
+       mb(); /* sync */
+       /* Flush on the EA that may be executed in case of a non- coherent icache */
+       icbi(exec_addr);
+       mb(); /* sync */
+       isync();

        return 0;

arch/powerpc/lib/code-patching.c +0 -0
@@ -38,7 +42,10 @@ static int __patch_instruction(u32 *exec_addr,

ppc_inst_t instr, u32 *patch_addr

 int raw_patch_instruction(u32 *addr, ppc_inst_t instr) {
-       return __patch_instruction(addr, instr, addr); +       if (ppc_inst_prefixed(instr))
+               return __patch_memory(addr, ppc_inst_as_ulong(instr),
addr, true);
+       else
+               return __patch_memory(addr, ppc_inst_val(instr), addr, false);
 }

 static DEFINE_PER_CPU(struct vm_struct *, text_poke_area);

arch/powerpc/lib/code-patching.c +0 -0
@@ -149,7 +156,7 @@ static void unmap_patch_area(unsigned long
 addr)
         flush_tlb_kernel_range(addr, addr + PAGE_SIZE);
  }

arch/powerpc/lib/code-patching.c +1 -1
-static int __do_patch_instruction(u32 *addr, ppc_inst_t instr)
+static int __do_patch_memory(void *addr, unsigned long val, bool

is_dword)
 {
        int err;
        u32 *patch_addr;

arch/powerpc/lib/code-patching.c +0 -0
@@ -166,7 +173,7 @@ static int __do_patch_instruction(u32 *addr,
 ppc_inst_t instr)
         if (radix_enabled())
                 asm volatile("ptesync": : :"memory");


-       err = __patch_instruction(addr, instr, patch_addr); +       err = __patch_memory(patch_addr, val, addr, is_dword);
        pte_clear(&init_mm, text_poke_addr, pte); flush_tlb_kernel_range(text_poke_addr, text_poke_addr + PAGE_SIZE);

arch/powerpc/lib/code-patching.c +0 -0
@@ -174,7 +181,7 @@ static int __do_patch_instruction(u32 *addr,
 ppc_inst_t instr)
         return err;
  }

arch/powerpc/lib/code-patching.c +1 -1
-int patch_instruction(u32 *addr, ppc_inst_t instr)
+static int do_patch_memory(void *addr, unsigned long val, bool

is_dword)
 {
        int err;
        unsigned long flags;

arch/powerpc/lib/code-patching.c +0 -0
@@ -186,15 +193,47 @@ int patch_instruction(u32 *addr, ppc_inst_t
 instr)
          */

        if (!IS_ENABLED(CONFIG_STRICT_KERNEL_RWX) || !static_branch_likely(&poking_init_done))
-               return raw_patch_instruction(addr, instr); +               return __patch_memory(addr, val, addr, is_dword);
        local_irq_save(flags);
-       err = __do_patch_instruction(addr, instr);
+       err = __do_patch_memory(addr, val, is_dword); local_irq_restore(flags);

        return err;
 }

arch/powerpc/lib/code-patching.c +33 -1
-NOKPROBE_SYMBOL(patch_instruction);
+
+#ifdef CONFIG_PPC64
+
+int patch_uint(void *addr, unsigned int val)
+{
+       return do_patch_memory(addr, val, false);
+}
+NOKPROBE_SYMBOL(patch_uint)
+
+int patch_ulong(void *addr, unsigned long val)
+{
+       return do_patch_memory(addr, val, true);
+}
+NOKPROBE_SYMBOL(patch_ulong)
+
+int patch_instruction(u32 *addr, ppc_inst_t instr)
+{
+       if (ppc_inst_prefixed(instr))
+               return patch_ulong(addr, ppc_inst_as_ulong(instr));
+       else
+               return patch_uint(addr, ppc_inst_val(instr));
+}
+NOKPROBE_SYMBOL(patch_instruction)
+
+#else
+
+noinline int patch_uint(void *addr, unsigned int val)
+{
+       return do_patch_memory(addr, val, false);
+}
+NOKPROBE_SYMBOL(patch_uint)
+
+#endif


 int patch_branch(u32 *addr, unsigned long target, int flags) {

Signature · 2 lines
Andrew Donnellan    OzLabs, ADL Canberra
[email protected]   IBM Australia Limited
#4 lore raw

PATCH v4 2/6 powerpc/module: Handle caller-saved TOC in module linker

In reply to Benjamin Gray

A function symbol may set a value in the st_other field to indicate the TOC should be treated as caller-saved. The linker should ensure the current TOC is saved before calling it and restore the TOC afterwards, much like external calls.

This is necessary for supporting V2 ABI static calls that do not preserve the TOC.

---
 arch/powerpc/kernel/module_64.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
arch/powerpc/kernel/module_64.c +13 -1
index 7e45dc98df8a..83a6f6e22e3b 100644
@@ -55,6 +55,12 @@ static unsigned int local_entry_offset(const Elf64_Sym *sym)
 	 * of function and try to derive r2 from it). */
 	return PPC64_LOCAL_ENTRY_OFFSET(sym->st_other);
 }
+
+static bool need_r2save_stub(unsigned char st_other)
+{
+	return (st_other & STO_PPC64_LOCAL_MASK) == (1 << STO_PPC64_LOCAL_BIT);
+}
+
 #else
 
 static func_desc_t func_desc(unsigned long addr)
@@ -66,6 +72,11 @@ static unsigned int local_entry_offset(const Elf64_Sym *sym)
 	return 0;
 }
 
+static bool need_r2save_stub(unsigned char st_other)
+{
+	return false;
+}
+
 void *dereference_module_function_descriptor(struct module *mod, void *ptr)
 {
 	if (ptr < (void *)mod->arch.start_opd ||
@@ -632,7 +643,8 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,
 		case R_PPC_REL24:
 			/* FIXME: Handle weak symbols here --RR */
 			if (sym->st_shndx == SHN_UNDEF ||
-			    sym->st_shndx == SHN_LIVEPATCH) {
+			    sym->st_shndx == SHN_LIVEPATCH ||
+			    need_r2save_stub(sym->st_other)) {
 				/* External: go via stub */
 				value = stub_for_addr(sechdrs, value, me,
 						strtab + sym->st_name);
Signature · 1 line
2.37.3
Reviewed-by on v4 2/6
#5 lore raw

Re: [PATCH v4 2/6] powerpc/module: Handle caller-saved TOC in module linker

Quoted 6 lines · Benjamin Gray
Quoted 6 lines

A function symbol may set a value in the st_other field to indicate the TOC should be treated as caller-saved. The linker should ensure> the
current TOC is saved before calling it and restore the TOC> afterwards,
much like external calls.

As I suggested on the last revision, worth mentioning here that it's the '.localentry <NAME>, 1' directive we're talking about here.

Quoted 43 lines
Quoted 43 lines

This is necessary for supporting V2 ABI static calls that do not preserve the TOC.

---
 arch/powerpc/kernel/module_64.c | 14 +++++++++++++-

 1 file changed, 13 insertions(+), 1 deletion(-)
arch/powerpc/kernel/module_64.c +6 -0
index 7e45dc98df8a..83a6f6e22e3b 100644
@@ -55,6 +55,12 @@ static unsigned int local_entry_offset(const>
 Elf64_Sym *sym)
          * of function and try to derive r2 from it). */
         return PPC64_LOCAL_ENTRY_OFFSET(sym->st_other);
  }
+
+static bool need_r2save_stub(unsigned char st_other)
+{
+       return (st_other & STO_PPC64_LOCAL_MASK) == (1 <<>
 STO_PPC64_LOCAL_BIT);
+}
+
  #else


 static func_desc_t func_desc(unsigned long addr)

arch/powerpc/kernel/module_64.c +0 -0
@@ -66,6 +72,11 @@ static unsigned int local_entry_offset(const>
 Elf64_Sym *sym)
         return 0;
  }

arch/powerpc/kernel/module_64.c +5 -0
+static bool need_r2save_stub(unsigned char st_other)
+{
+       return false;
+}
+

 void *dereference_module_function_descriptor(struct module *mod,> void *ptr)
 {
        if (ptr < (void *)mod->arch.start_opd ||

Quoted 8 lines
Quoted 8 lines
arch/powerpc/kernel/module_64.c +0 -0
@@ -632,7 +643,8 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,

                case R_PPC_REL24:
                        /* FIXME: Handle weak symbols here --RR */ if (sym->st_shndx == SHN_UNDEF || -                           sym->st_shndx == SHN_LIVEPATCH) { +                           sym->st_shndx == SHN_LIVEPATCH || +                           need_r2save_stub(sym->st_other)) { /* External: go via stub */

Perhaps this comment should be updated to mention that there are non- external but external-like calls?

Otherwise

Quoted 4 lines
Quoted 4 lines

                                value = stub_for_addr(sechdrs, value,> me,
                                                strtab +> sym-

Quoted 1 line

st_name);

Signature · 2 lines
Andrew Donnellan    OzLabs, ADL Canberra
[email protected]   IBM Australia Limited
#6 lore raw

Re: [PATCH v4 2/6] powerpc/module: Handle caller-saved TOC in module linker

Quoted 11 lines · Andrew Donnellan
Quoted 7 lines · Benjamin Gray
Quoted 7 lines

A function symbol may set a value in the st_other field to indicate
the TOC should be treated as caller-saved. The linker should ensure> the
current TOC is saved before calling it and restore the TOC> afterwards,
much like external calls.

As I suggested on the last revision, worth mentioning here that it's the '.localentry <NAME>, 1' directive we're talking about here.

Ah right, whoops. Added "For example, GCC and Clang support a '.localentry <NAME>, 1' directive to set this explicitly in assembly."

The exact method for doing this seems to be nonstandard (GCC supports arbitrary value in 1--7, Clang special cases only 1), so originally I was avoiding specifying how it is set.

Quoted 13 lines
Quoted 9 lines
Quoted 9 lines
(quoted diff) +0 -0
@@ -632,7 +643,8 @@ int apply_relocate_add(Elf64_Shdr *sechdrs,

                case R_PPC_REL24:
                        /* FIXME: Handle weak symbols here --RR */
                        if (sym->st_shndx == SHN_UNDEF || -                           sym->st_shndx == SHN_LIVEPATCH) { +                           sym->st_shndx == SHN_LIVEPATCH || +                           need_r2save_stub(sym->st_other)) { /* External: go via stub */

Perhaps this comment should be updated to mention that there are non- external but external-like calls?

Quoted 3 lines

Otherwise

Updated to "/* May use different / not preserve TOC: go via stub */", will add your reviewed-by in the next version. For now I'll wait for any other feedback before sending it.

Quoted 1 line
Quoted 1 line
#7 lore raw

PATCH v4 3/6 powerpc/module: Optimise nearby branches in ELF V2 ABI stub

In reply to Benjamin Gray

Inserts a direct branch to the stub target when possible, replacing the mtctr/btctr sequence.

The load into r12 could potentially be skipped too, but that change would need to refactor the arguments to indicate that the address does not have a separate local entry point.

This helps the static call implementation, where modules calling their own trampolines are called through this stub and the trampoline is easily within range of a direct branch.

---
 arch/powerpc/kernel/module_64.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
arch/powerpc/kernel/module_64.c +12 -0
index 83a6f6e22e3b..3a90043b006c 100644
@@ -141,6 +141,12 @@ static u32 ppc64_stub_insns[] = {
 	PPC_RAW_BCTR(),
 };
 
+#ifdef CONFIG_PPC64_ELF_ABI_V1
+#define PPC64_STUB_MTCTR_OFFSET 5
+#else
+#define PPC64_STUB_MTCTR_OFFSET 4
+#endif
+
 /* Count how many different 24-bit relocations (different symbol,
    different addend) */
 static unsigned int count_relocs(const Elf64_Rela *rela, unsigned int num)
@@ -426,6 +432,7 @@ static inline int create_stub(const Elf64_Shdr *sechdrs,
 			      struct module *me,
 			      const char *name)
 {
+	int err;
 	long reladdr;
 	func_desc_t desc;
 	int i;
@@ -439,6 +446,11 @@ static inline int create_stub(const Elf64_Shdr *sechdrs,
 			return 0;
 	}
 
+	/* Replace indirect branch sequence with direct branch where possible */
+	err = patch_branch(&entry->jump[PPC64_STUB_MTCTR_OFFSET], addr, 0);
+	if (err && err != -ERANGE)
+		return 0;
+
 	/* Stub uses address relative to r2. */
 	reladdr = (unsigned long)entry - my_r2(sechdrs, me);
 	if (reladdr > 0x7FFFFFFF || reladdr < -(0x80000000L)) {
Signature · 1 line
2.37.3
#8 lore raw

PATCH v4 4/6 static_call: Move static call selftest to static_call_selftest.c

In reply to Benjamin Gray

These tests are out-of-line only, so moving them to the their own file allows them to be run when an arch does not implement inline static calls.

---
 kernel/Makefile               |  1 +
 kernel/static_call_inline.c   | 43 -----------------------------------
 kernel/static_call_selftest.c | 41 +++++++++++++++++++++++++++++++++
 3 files changed, 42 insertions(+), 43 deletions(-)
 create mode 100644 kernel/static_call_selftest.c
kernel/Makefile +1 -0
index 318789c728d3..8ce8beaa3cc0 100644
@@ -113,6 +113,7 @@ obj-$(CONFIG_KCSAN) += kcsan/
 obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o
 obj-$(CONFIG_HAVE_STATIC_CALL) += static_call.o
 obj-$(CONFIG_HAVE_STATIC_CALL_INLINE) += static_call_inline.o
+obj-$(CONFIG_STATIC_CALL_SELFTEST) += static_call_selftest.o
 obj-$(CONFIG_CFI_CLANG) += cfi.o
 
 obj-$(CONFIG_PERF_EVENTS) += events/
kernel/static_call_inline.c +0 -43

46 lines not on this page · raw message

kernel/static_call_selftest.c new file +41 -0

41 lines not on this page · raw message

Signature · 1 line
2.37.3
#9 lore raw

PATCH v4 5/6 powerpc/64: Add support for out-of-line static calls

In reply to Benjamin Gray

Implement static call support for 64 bit V2 ABI. This requires making sure the TOC is kept correct across kernel-module boundaries. As a secondary concern, it tries to use the local entry point of a target wherever possible. It does so by checking if both tramp & target are kernel code, and falls back to detecting the common global entry point patterns if modules are involved. Detecting the global entry point is also required for setting the local entry point as the trampoline
target: if we cannot detect the local entry point, then we need to convservatively initialise r12 and use the global entry point.

The trampolines are marked with .localentry NAME, 1 to make the linker save and restore the TOC on each call to the trampoline. This allows the trampoline to safely target functions with different TOC values.

However this directive also implies the TOC is not initialised on entry to the trampoline. The kernel TOC is easily found in the PACA, but not an arbitrary module TOC. Therefore the trampoline implementation depends on whether it's in the kernel or not. If in the kernel, we initialise the TOC using the PACA. If in a module, we have to initialise the TOC with zero context, so it's quite expensive.

---
 arch/powerpc/Kconfig                     |  14 ++-
 arch/powerpc/include/asm/code-patching.h |   1 +
 arch/powerpc/include/asm/static_call.h   |  80 +++++++++++++-
 arch/powerpc/kernel/Makefile             |   3 +-
 arch/powerpc/kernel/static_call.c        | 130 +++++++++++++++++++++--
 5 files changed, 216 insertions(+), 12 deletions(-)
arch/powerpc/Kconfig +13 -1
index 4c466acdc70d..962e36ec34ec 100644
@@ -102,6 +102,18 @@ config GENERIC_HWEIGHT
 	bool
 	default y
 
+config TOOLCHAIN_SUPPORTS_LOCALENTRY1
+	bool
+	depends on PPC64_ELF_ABI_V2
+	default y if LD_VERSION >= 23200 || LLD_VERSION >= 110000
+	help
+	  A section of the ELF symbol st_other field can be given the value 1
+	  using the directive '.localentry NAME, 1' to mean the local and global
+	  entry points are the same, and r2 should be treated as caller-saved.
+
+	  Older versions of Clang and binutils do not recognise this form of the
+	  directive and will error if it is used.
+
 config PPC
 	bool
 	default y
@@ -248,7 +260,7 @@ config PPC
 	select HAVE_SOFTIRQ_ON_OWN_STACK
 	select HAVE_STACKPROTECTOR		if PPC32 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r2)
 	select HAVE_STACKPROTECTOR		if PPC64 && $(cc-option,-mstack-protector-guard=tls -mstack-protector-guard-reg=r13)
-	select HAVE_STATIC_CALL			if PPC32
+	select HAVE_STATIC_CALL			if PPC32 || (PPC64_ELF_ABI_V2 && TOOLCHAIN_SUPPORTS_LOCALENTRY1)
 	select HAVE_SYSCALL_TRACEPOINTS
 	select HAVE_VIRT_CPU_ACCOUNTING
 	select HUGETLB_PAGE_SIZE_VARIABLE	if PPC_BOOK3S_64 && HUGETLB_PAGE
arch/powerpc/include/asm/code-patching.h +1 -0
index 170bfa848c7c..cb4629e55e57 100644
@@ -152,6 +152,7 @@ int translate_branch(ppc_inst_t *instr, const u32 *dest, const u32 *src);
 bool is_conditional_branch(ppc_inst_t instr);
 
 #define OP_RT_RA_MASK	0xffff0000UL
+#define OP_SI_MASK	0x0000ffffUL
 #define LIS_R2		(PPC_RAW_LIS(_R2, 0))
 #define ADDIS_R2_R12	(PPC_RAW_ADDIS(_R2, _R12, 0))
 #define ADDI_R2_R2	(PPC_RAW_ADDI(_R2, _R2, 0))
arch/powerpc/include/asm/static_call.h +76 -4

99 lines not on this page · raw message

arch/powerpc/kernel/Makefile +2 -1
index 06d2d1f78f71..a30d0d0f5499 100644
@@ -128,8 +128,9 @@ extra-y				+= vmlinux.lds
 
 obj-$(CONFIG_RELOCATABLE)	+= reloc_$(BITS).o
 
-obj-$(CONFIG_PPC32)		+= entry_32.o setup_32.o early_32.o static_call.o
+obj-$(CONFIG_PPC32)		+= entry_32.o setup_32.o early_32.o
 obj-$(CONFIG_PPC64)		+= dma-iommu.o iommu.o
+obj-$(CONFIG_HAVE_STATIC_CALL)	+= static_call.o
 obj-$(CONFIG_KGDB)		+= kgdb.o
 obj-$(CONFIG_BOOTX_TEXT)	+= btext.o
 obj-$(CONFIG_SMP)		+= smp.o
arch/powerpc/kernel/static_call.c +124 -6

157 lines not on this page · raw message

Signature · 1 line
2.37.3
#10 lore raw

PATCH v4 6/6 powerpc: Add tests for out-of-line static calls

In reply to Benjamin Gray

KUnit tests for the various combinations of caller/trampoline/target and kernel/module. They must be run from a module loaded at runtime to guarantee they have a different TOC to the kernel (64-bit ELF V2) and increase the chance of testing the non-direct branch path of the trampoline.

For 64-bit ELF V2 ABI the tests try to mitigate the chance of panicking by restoring the TOC after every static call. Not all possible errors can be caught by this (we can't stop a trampoline from using a bad TOC itself), but it makes certain errors easier to debug.

---
 arch/powerpc/Kconfig                   |  12 ++
 arch/powerpc/kernel/Makefile           |   1 +
 arch/powerpc/kernel/static_call.c      |  53 +++++
 arch/powerpc/kernel/static_call_test.c | 263 +++++++++++++++++++++++++
 arch/powerpc/kernel/static_call_test.h |  56 ++++++
 5 files changed, 385 insertions(+)
 create mode 100644 arch/powerpc/kernel/static_call_test.c
 create mode 100644 arch/powerpc/kernel/static_call_test.h
arch/powerpc/Kconfig +12 -0
index 962e36ec34ec..5b9d5fa96a9e 100644
@@ -1035,6 +1035,18 @@ config PPC_RTAS_FILTER
 	  Say Y unless you know what you are doing and the filter is causing
 	  problems for you.
 
+config PPC_STATIC_CALL_KUNIT_TEST
+	tristate "KUnit tests for static calls"
+	default KUNIT_ALL_TESTS
+	depends on HAVE_STATIC_CALL && KUNIT && m
+	help
+	  Tests for static calls across all combinations of caller/trampoline/target
+	  being kernel/module. On ELF ABI V2 the tests check the TOC is kept consistent.
+
+	  Must be built as a module and loaded at runtime to ensure the module has
+	  a different TOC to the kernel and make it likely that non-direct branch
+	  path of the trampoline is tested.
+
 endmenu
 
 config ISA_DMA_API
arch/powerpc/kernel/Makefile +1 -0
index a30d0d0f5499..22c07e3d34df 100644
@@ -131,6 +131,7 @@ obj-$(CONFIG_RELOCATABLE)	+= reloc_$(BITS).o
 obj-$(CONFIG_PPC32)		+= entry_32.o setup_32.o early_32.o
 obj-$(CONFIG_PPC64)		+= dma-iommu.o iommu.o
 obj-$(CONFIG_HAVE_STATIC_CALL)	+= static_call.o
+obj-$(CONFIG_PPC_STATIC_CALL_KUNIT_TEST)	+= static_call_test.o
 obj-$(CONFIG_KGDB)		+= kgdb.o
 obj-$(CONFIG_BOOTX_TEXT)	+= btext.o
 obj-$(CONFIG_SMP)		+= smp.o
arch/powerpc/kernel/static_call.c +53 -0

56 lines not on this page · raw message

arch/powerpc/kernel/static_call_test.c new file +263 -0

263 lines not on this page · raw message

arch/powerpc/kernel/static_call_test.h new file +56 -0

56 lines not on this page · raw message

Signature · 1 line
2.37.3