Static calls
Subsystems (6)
cover Static calls
These patches are related to two similar patch sets from Ard and Steve:
- https://lkml.kernel.org/r/[email protected]
- https://lkml.kernel.org/r/[email protected]
The code is also heavily inspired by the jump label code, as some of the concepts are very similar.
There are three separate implementations, depending on what the arch supports:
1) CONFIG_HAVE_STATIC_CALL_OPTIMIZED: patched call sites - requires objtool and a small amount of arch code
2) CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED: patched trampolines - requires a small amount of arch code
3) If no arch support, fall back to regular function pointers
TODO:
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?
- Find some actual users of the interfaces (tracepoints? crypto?)
Details (cribbed from comments in include/linux/static_call.h):
------------------------------------------------------------------------
Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.
API overview:
DECLARE_STATIC_CALL(key, func);
DEFINE_STATIC_CALL(key, func);
static_call(key, args...);
static_call_update(key, func);
Usage example:
# Start with the following functions (with identical prototypes): int func_a(int arg1, int arg2); int func_b(int arg1, int arg2); # Define a 'my_key' reference, associated with func_a() by default DEFINE_STATIC_CALL(my_key, func_a); # Call func_a() static_call(my_key, arg1, arg2); # Update 'my_key' to point to func_b() static_call_update(my_key, func_b); # Call func_b() static_call(my_key, arg1, arg2);
Implementation details:
There are three different implementations:
1) Optimized static calls (patched call sites)
This requires objtool, which detects all the static_call() sites and annotates them in the '.static_call_sites' section. By default, the call sites will call into a temporary per-key trampoline which has an indirect branch to the current destination function associated with the key. During system boot (or module init), all call sites are patched to call their destination functions directly. Updates to a key will patch all call sites associated with that key.
2) Unoptimized static calls (patched trampolines)
Each static_call() site calls into a permanent trampoline associated with the key. The trampoline has a direct branch to the default function. Updates to a key will modify the direct branch in the key's trampoline.
3) Generic implementation
This is the default implementation if the architecture hasn't implemented CONFIG_HAVE_STATIC_CALL_[UN]OPTIMIZED. In this case, a basic function pointer is used.
Josh Poimboeuf (3):
static_call: Add static call infrastructure x86/static_call: Add x86 unoptimized static call implementation x86/static_call: Add optimized static call implementation for 64-bit
arch/Kconfig | 6 + arch/x86/Kconfig | 4 +- arch/x86/include/asm/static_call.h | 42 +++ arch/x86/kernel/Makefile | 1 + arch/x86/kernel/static_call.c | 84 +++++ include/asm-generic/vmlinux.lds.h | 11 + include/linux/module.h | 10 + include/linux/static_call.h | 186 +++++++++++ include/linux/static_call_types.h | 19 ++ kernel/Makefile | 1 + kernel/module.c | 5 + kernel/static_call.c | 297 ++++++++++++++++++ tools/objtool/Makefile | 3 +- tools/objtool/check.c | 126 +++++++- tools/objtool/check.h | 2 + tools/objtool/elf.h | 1 + .../objtool/include/linux/static_call_types.h | 19 ++ tools/objtool/sync-check.sh | 1 + 18 files changed, 815 insertions(+), 3 deletions(-) create mode 100644 arch/x86/include/asm/static_call.h create mode 100644 arch/x86/kernel/static_call.c create mode 100644 include/linux/static_call.h create mode 100644 include/linux/static_call_types.h create mode 100644 kernel/static_call.c create mode 100644 tools/objtool/include/linux/static_call_types.h
Signature · 1 line
2.17.2
RFC PATCH 1/3 static_call: Add static call infrastructure
Add a static call infrastructure. Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.
This code is heavily inspired by the jump label code (aka "static jumps"), as some of the concepts are very similar.
There are three implementations, depending on arch support:
1) optimized: patched call sites (CONFIG_HAVE_STATIC_CALL_OPTIMIZED)
2) unoptimized: patched trampolines (CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)
3) basic function pointers
For more details, see the comments in include/linux/static_call.h.
--- arch/Kconfig | 6 + include/asm-generic/vmlinux.lds.h | 11 ++ include/linux/module.h | 10 + include/linux/static_call.h | 186 +++++++++++++++++++ include/linux/static_call_types.h | 19 ++ kernel/Makefile | 1 + kernel/module.c | 5 + kernel/static_call.c | 297 ++++++++++++++++++++++++++++++ 8 files changed, 535 insertions(+) create mode 100644 include/linux/static_call.h create mode 100644 include/linux/static_call_types.h create mode 100644 kernel/static_call.c
arch/Kconfig +6 -0
@@ -879,6 +879,12 @@ config HAVE_ARCH_PREL32_RELOCATIONS architectures, and don't require runtime relocation on relocatable kernels. +config HAVE_STATIC_CALL_OPTIMIZED + bool + +config HAVE_STATIC_CALL_UNOPTIMIZED + bool + source "kernel/gcov/Kconfig" source "scripts/gcc-plugins/Kconfig"
include/asm-generic/vmlinux.lds.h +11 -0
@@ -320,6 +320,7 @@ __start_ro_after_init = .; \ *(.data..ro_after_init) \ JUMP_TABLE_DATA \ + STATIC_CALL_SITES \ __end_ro_after_init = .;#endif
@@ -725,6 +726,16 @@#define BUG_TABLE #endif +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED +#define STATIC_CALL_SITES \ + . = ALIGN(8); \ + __start_static_call_sites = .; \ + KEEP(*(.static_call_sites)) \ + __stop_static_call_sites = .; +#else +#define STATIC_CALL_SITES +#endif +#ifdef CONFIG_UNWINDER_ORC #define ORC_UNWIND_TABLE \ . = ALIGN(4); \
include/linux/module.h +10 -0
@@ -21,6 +21,7 @@#include <linux/rbtree_latch.h> #include <linux/error-injection.h> #include <linux/tracepoint-defs.h> +#include <linux/static_call_types.h> #include <linux/percpu.h> #include <asm/module.h>
@@ -450,6 +451,10 @@ struct module {unsigned int num_ftrace_callsites;unsigned long *ftrace_callsites;#endif +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED +int num_static_call_sites; +struct static_call_site *static_call_sites; +#endif #ifdef CONFIG_LIVEPATCH bool klp;/* Is this a livepatch module? */
@@ -682,6 +687,11 @@ static inline bool is_module_text_address(unsigned long addr)return false; } +static inline bool within_module_init (unsigned long addr,const struct module *mod) +{ +return false; +} +/* Get/put a kernel symbol (calls should be symmetric) */ #define symbol_get (x) ({ extern typeof(x) x __attribute__((weak)); &(x); })#define symbol_put (x) do { } while (0)
include/linux/static_call.h new file +186 -0
186 lines not on this page · raw message
include/linux/static_call_types.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _STATIC_CALL_TYPES_H +#define _STATIC_CALL_TYPES_H + +#include <linux/stringify.h> + +#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_ +#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX) + +#define STATIC_CALL_TRAMP (key) STATIC_CALL_TRAMP_PREFIX##key +#define STATIC_CALL_TRAMP_STR (key) __stringify(STATIC_CALL_TRAMP(key)) + +/* The static call site table is created by objtool. */ +struct static_call_site { +s32 addr; +s32 key; +}; + +#endif /* _STATIC_CALL_TYPES_H */
kernel/Makefile +1 -0
@@ -103,6 +103,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_HAVE_STATIC_CALL_OPTIMIZED) += static_call.o obj-$(CONFIG_PERF_EVENTS) += events/
kernel/module.c +5 -0
@@ -3121,6 +3121,11 @@ static int find_module_sections(struct module *mod, struct load_info *info) mod->ei_funcs =section_objs (info,"_error_injection_whitelist" ,sizeof (*mod->ei_funcs), &mod->num_ei_funcs); +#endif +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED + mod->static_call_sites =section_objs (info,".static_call_sites" , +sizeof (*mod->static_call_sites), + &mod->num_static_call_sites);#endif mod->extable =section_objs (info,"__ex_table" ,sizeof (*mod->extable), &mod->num_exentries);
kernel/static_call.c new file +297 -0
297 lines not on this page · raw message
Signature · 1 line
2.17.2
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Hi Josh,
Thanks a lot for looking into this.
Quoted 6 lines · Josh Poimboeuf
Add a static call infrastructure. Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.
Quoted 7 lines
This code is heavily inspired by the jump label code (aka "static jumps"), as some of the concepts are very similar.
There are three implementations, depending on arch support:
1) optimized: patched call sites (
CONFIG_HAVE_STATIC_CALL_OPTIMIZED)
2) unoptimized: patched trampolines (CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)
Could we move to an idiom like inline/out of line? The unoptimized variant may be perfectly adequate for many arches, and 'unoptimized' sounds like there is something wrong with it.
Quoted 136 lines
3) basic function pointers
For more details, see the comments in
include/linux/static_call.h.Signed-off-by Josh Poimboeuf <[email protected]>--- arch/Kconfig | 6 + include/asm-generic/vmlinux.lds.h | 11 ++ include/linux/module.h | 10 + include/linux/static_call.h | 186 +++++++++++++++++++ include/linux/static_call_types.h | 19 ++ kernel/Makefile | 1 + kernel/module.c | 5 + kernel/static_call.c | 297 ++++++++++++++++++++++++++++++ 8 files changed, 535 insertions(+) create mode 100644 include/linux/static_call.h create mode 100644 include/linux/static_call_types.h create mode 100644 kernel/static_call.carch/Kconfig +6
-0@@ -879,6 +879,12 @@ config HAVE_ARCH_PREL32_RELOCATIONS architectures, and don't require runtime relocation on relocatable kernels. +config HAVE_STATIC_CALL_OPTIMIZED + bool + +config HAVE_STATIC_CALL_UNOPTIMIZED + bool + source "kernel/gcov/Kconfig" source "scripts/gcc-plugins/Kconfig"include/asm-generic/vmlinux.lds.h +11
-0@@ -320,6 +320,7 @@ __start_ro_after_init = .; \ *(.data..ro_after_init) \ JUMP_TABLE_DATA \ + STATIC_CALL_SITES \ __end_ro_after_init = .;#endif @@ -725,6 +726,16 @@#define BUG_TABLE #endif +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED +#define STATIC_CALL_SITES \ + . = ALIGN(8); \ + __start_static_call_sites = .; \ + KEEP(*(.static_call_sites)) \ + __stop_static_call_sites = .; +#else +#define STATIC_CALL_SITES +#endif +#ifdef CONFIG_UNWINDER_ORC #define ORC_UNWIND_TABLE \ . = ALIGN(4); \include/linux/module.h +10
-0@@ -21,6 +21,7 @@#include <linux/rbtree_latch.h> #include <linux/error-injection.h> #include <linux/tracepoint-defs.h> +#include <linux/static_call_types.h> #include <linux/percpu.h> #include <asm/module.h> @@ -450,6 +451,10 @@ struct module {unsigned int num_ftrace_callsites;unsigned long *ftrace_callsites;#endif +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED +int num_static_call_sites; +struct static_call_site *static_call_sites; +#endif #ifdef CONFIG_LIVEPATCH bool klp;/* Is this a livepatch module? */ @@ -682,6 +687,11 @@ static inline bool is_module_text_address(unsigned long addr)return false; } +static inline bool within_module_init (unsigned long addr,const struct module *mod) +{ +return false; +} +/* Get/put a kernel symbol (calls should be symmetric) */ #define symbol_get (x) ({ extern typeof(x) x __attribute__((weak)); &(x); })#define symbol_put (x) do { } while (0)include/linux/static_call.h new file +30
-0@@ -0,0 +1,186 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_STATIC_CALL_H +#define _LINUX_STATIC_CALL_H + +/* +* Static call support +* +* Static calls use code patching to hard-code function pointers into direct +* branch instructions. They give the flexibility of function pointers, but +* with improved performance. This is especially important for cases where +* retpolines would otherwise be used, as retpolines can significantly impact +* performance. +* +* +* API overview: +* +* DECLARE_STATIC_CALL(key, func); +* DEFINE_STATIC_CALL(key, func); +* static_call(key, args...); +* static_call_update(key, func); +* +* +* Usage example: +* +* # Start with the following functions (with identical prototypes): +* int func_a(int arg1, int arg2); +* int func_b(int arg1, int arg2); +* +* # Define a 'my_key' reference, associated with func_a() by default +* DEFINE_STATIC_CALL(my_key, func_a);
include/linux/static_call.h line 40
Quoted 10 lines
include/linux/static_call.h new file +10
-0+* +* # Call func_a() +* static_call(my_key, arg1, arg2); +* +* # Update 'my_key' to point to func_b() +* static_call_update(my_key, func_b); +* +* # Call func_b() +* static_call(my_key, arg1, arg2); +*
Any way we can revert to the default implementation? That would be useful for, e.g., unloading modules that provide an alternative implementation.
Quoted 154 lines
include/linux/static_call.h new file +146
-0+* +* Implementation details: +* +* There are three different implementations: +* +* 1) Optimized static calls (patched call sites) +* +* This requires objtool, which detects all the static_call() sites and +* annotates them in the '.static_call_sites' section. By default, the call +* sites will call into a temporary per-key trampoline which has an indirect +* branch to the current destination function associated with the key. +* During system boot (or module init), all call sites are patched to call +* their destination functions directly. Updates to a key will patch all +* call sites associated with that key. +* +* 2) Unoptimized static calls (patched trampolines) +* +* Each static_call() site calls into a permanent trampoline associated with +* the key. The trampoline has a direct branch to the default function. +* Updates to a key will modify the direct branch in the key's trampoline. +* +* 3) Generic implementation +* +* This is the default implementation if the architecture hasn't implemented +* CONFIG_HAVE_STATIC_CALL_[UN]OPTIMIZED. In this case, a basic +* function pointer is used. +*/ + +#include <linux/types.h> +#include <linux/cpu.h> +#include <linux/static_call_types.h> + +#if defined(CONFIG_HAVE_STATIC_CALL_OPTIMIZED ) || \ + defined(CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED ) + +#include <asm/static_call.h> + +extern void arch_static_call_transform (unsigned long insn,void *dest); + +#endif /* CONFIG_HAVE_STATIC_CALL_[UN]OPTIMIZED */ + +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED +/* Optimized implementation */ + +struct static_call_mod { +struct list_head list; +struct module *mod;/* NULL means vmlinux */ +struct static_call_site *sites; +}; + +struct static_call_key { +/* +* This field should always be first because the trampolines expect it. +* This points to the current call destination. +*/ +void *func; + +/* +* List of modules (including vmlinux) and the call sites which need to +* be patched for this key. +*/ +struct list_head site_mods; +}; + +extern void __static_call_update (struct static_call_key *key,void *func); +extern void arch_static_call_poison_tramp (unsigned long insn); + +#define DECLARE_STATIC_CALL (key, func) \ + extern struct static_call_key key; \ + extern typeof(func) STATIC_CALL_TRAMP(key); \ +/* Preserve the ELF symbol so objtool can access it: */ \ + __ADDRESSABLE(key) + +#define DEFINE_STATIC_CALL (key, _func) \ + DECLARE_STATIC_CALL(key, _func); \ + struct static_call_key key = { \ + .func = _func, \ + .site_mods = LIST_HEAD_INIT(key.site_mods), \ + }; \ + ARCH_STATIC_CALL_TEMPORARY_TRAMP(key) + +#define static_call(key, args...) STATIC_CALL_TRAMP(key)(args) + +#define static_call_update (key, func) \ +({ \ + BUILD_BUG_ON(!__same_type(typeof(func), typeof(STATIC_CALL_TRAMP(key)))); \ + __static_call_update(&key, func); \ +}) + +#define EXPORT_STATIC_CALL (key) \ + EXPORT_SYMBOL(key); \ + EXPORT_SYMBOL(STATIC_CALL_TRAMP(key)) + +#define EXPORT_STATIC_CALL_GPL (key) \ + EXPORT_SYMBOL_GPL(key); \ + EXPORT_SYMBOL_GPL(STATIC_CALL_TRAMP(key)) + + +#elif defined(CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED ) +/* Unoptimized implementation */ + +#define DECLARE_STATIC_CALL (key, func) \ + extern typeof(func) STATIC_CALL_TRAMP(key) + +#define DEFINE_STATIC_CALL (key, func) \ + DECLARE_STATIC_CALL(key, func); \ + ARCH_STATIC_CALL_TRAMP(key, func) + +#define static_call(key, args...) STATIC_CALL_TRAMP(key)(args) + +#define static_call_update (key, func) \ +({ \ + BUILD_BUG_ON(!__same_type(func, STATIC_CALL_TRAMP(key))); \ + cpus_read_lock(); \ + arch_static_call_transform((unsigned long)STATIC_CALL_TRAMP(key),\ + func); \ + cpus_read_unlock(); \ +}) + +#define EXPORT_STATIC_CALL (key) \ + EXPORT_SYMBOL(STATIC_CALL_TRAMP(key)) + +#define EXPORT_STATIC_CALL_GPL (key) \ + EXPORT_SYMBOL_GPL(STATIC_CALL_TRAMP(key)) + + +#else /* Generic implementation */ + +#define DECLARE_STATIC_CALL (key, func) \ + extern typeof(func) *key + +#define DEFINE_STATIC_CALL (key, func) \ + typeof(func) *key = func + +#define static_call(key, args...) \ + key(args) + +#define static_call_update (key, func) \ + WRITE_ONCE(key, func) + +#define EXPORT_STATIC_CALL (key) EXPORT_SYMBOL(key) +#define EXPORT_STATIC_CALL_GPL (key) EXPORT_SYMBOL_GPL(key) + +#endif /* CONFIG_HAVE_STATIC_CALL_OPTIMIZED */ + +#endif /* _LINUX_STATIC_CALL_H */ include/linux/static_call_types.h new file +2
-0@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _STATIC_CALL_TYPES_H
include/linux/static_call_types.h line 12
Quoted 10 lines
include/linux/static_call_types.h new file +10
-0+#define _STATIC_CALL_TYPES_H + +#include <linux/stringify.h> + +#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_ +#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX) + +#define STATIC_CALL_TRAMP (key) STATIC_CALL_TRAMP_PREFIX##key +#define STATIC_CALL_TRAMP_STR (key) __stringify(STATIC_CALL_TRAMP(key)) +
I needed to apply
(quoted diff) +0 -0
b/include/linux/static_call_types.h
index 7dd4b3d7ec2b..6859b208de6e 100644
include/linux/static_call_types.h +1 -1
@@ -7,7 +7,7 @@#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX) -#define STATIC_CALL_TRAMP (key) STATIC_CALL_TRAMP_PREFIX##key +#define STATIC_CALL_TRAMP (key) __PASTE(STATIC_CALL_TRAMP_PREFIX, key)#define STATIC_CALL_TRAMP_STR (key) __stringify(STATIC_CALL_TRAMP(key))/* The static call site table is created by objtool. */
or I end up with
0000000000000000 <STATIC_CALL_TRAMP_PREFIXfoo>:
0: 14000000 b a8 <foo_a>
4: d503201f nop
Quoted 341 lines
include/linux/static_call_types.h new file +7
-0+/* The static call site table is created by objtool. */ +struct static_call_site { +s32 addr; +s32 key; +}; + +#endif /* _STATIC_CALL_TYPES_H */ kernel/Makefile +1
-0@@ -103,6 +103,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_HAVE_STATIC_CALL_OPTIMIZED) += static_call.o obj-$(CONFIG_PERF_EVENTS) += events/kernel/module.c +5
-0@@ -3121,6 +3121,11 @@ static int find_module_sections(struct module *mod, struct load_info *info) mod->ei_funcs =section_objs (info,"_error_injection_whitelist" ,sizeof (*mod->ei_funcs), &mod->num_ei_funcs); +#endif +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED + mod->static_call_sites =section_objs (info,".static_call_sites" , +sizeof (*mod->static_call_sites), + &mod->num_static_call_sites);#endif mod->extable =section_objs (info,"__ex_table" ,sizeof (*mod->extable), &mod->num_exentries);kernel/static_call.c new file +297
-1@@ -0,0 +1,297 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include <linux/init.h> +#include <linux/static_call.h> +#include <linux/bug.h> +#include <linux/smp.h> +#include <linux/sort.h> +#include <linux/slab.h> +#include <linux/module.h> +#include <linux/cpu.h> +#include <linux/processor.h> +#include <asm/sections.h> + +extern struct static_call_site __start_static_call_sites[], + __stop_static_call_sites[]; + +static bool static_call_initialized; + +/* mutex to protect key modules/sites */ +static DEFINE_MUTEX (static_call_mutex); + +static void static_call_lock (void ) +{ +mutex_lock (&static_call_mutex); +} + +static void static_call_unlock (void ) +{ +mutex_unlock (&static_call_mutex); +} + +static inline unsigned long static_call_addr (struct static_call_site *site) +{ +return (long )site->addr + (long )&site->addr; +} + +static inline struct static_call_key *static_call_key (const struct static_call_site *site) +{ +return (struct static_call_key *)((long )site->key + (long )&site->key); +} + +static int static_call_site_cmp (const void *_a,const void *_b) +{ +const struct static_call_site *a = _a; +const struct static_call_site *b = _b; +const struct static_call_key *key_a =static_call_key (a); +const struct static_call_key *key_b =static_call_key (b); + +if (key_a < key_b) +return -1 ; + +if (key_a > key_b) +return 1 ; + +return 0 ; +} + +static void static_call_site_swap (void *_a,void *_b,int size) +{ +long delta = (unsigned long )_a - (unsigned long )_b; +struct static_call_site *a = _a; +struct static_call_site *b = _b; +struct static_call_site tmp = *a; + + a->addr = b->addr - delta; + a->key = b->key - delta; + + b->addr = tmp.addr + delta; + b->key = tmp.key + delta; +} + +static inline void static_call_sort_entries (struct static_call_site *start, +struct static_call_site *stop) +{ +sort (start, stop - start,sizeof (struct static_call_site ), + static_call_site_cmp, static_call_site_swap); +} + +void __static_call_update (struct static_call_key *key,void *func) +{ +struct static_call_mod *mod; +struct static_call_site *site, *stop; + +cpus_read_lock (); +static_call_lock (); + +if (key->func == func) + goto done; + + key->func = func; + +/* +* If called before init, leave the call sites unpatched for now. +* In the meantime they'll continue to call the temporary trampoline. +*/ +if (!static_call_initialized) + goto done; + +list_for_each_entry (mod, &key->site_mods, list) { +if (!mod->sites) { +/* +* This can happen if the static call key is defined in +* a module which doesn't use it. +*/ +continue ; + } + + stop = __stop_static_call_sites; + +#ifdef CONFIG_MODULES +if (mod->mod) { + stop = mod->mod->static_call_sites + + mod->mod->num_static_call_sites; + } +#endif + +for (site = mod->sites; + site < stop &&static_call_key (site) == key; site++) { +unsigned long addr =static_call_addr (site); + +if (!mod->mod &&init_section_contains ((void *)addr,1 )) +continue ; +if (mod->mod &&within_module_init (addr, mod->mod)) +continue ; + +arch_static_call_transform (addr, func); + } + } + +done: +static_call_unlock (); +cpus_read_unlock (); +} +EXPORT_SYMBOL_GPL (__static_call_update); + +#ifdef CONFIG_MODULES + +static int static_call_add_module (struct module *mod) +{ +struct static_call_site *start = mod->static_call_sites; +struct static_call_site *stop = mod->static_call_sites + + mod->num_static_call_sites; +struct static_call_site *site; +struct static_call_key *key, *prev_key =NULL ; +struct static_call_mod *static_call_mod; + +if (start == stop) +return 0 ; + +module_disable_ro (mod); +static_call_sort_entries (start, stop); +module_enable_ro (mod, false); + +for (site = start; site < stop; site++) { +unsigned long addr =static_call_addr (site); + +if (within_module_init (addr, mod)) +continue ; + + key =static_call_key (site); +if (key != prev_key) { + prev_key = key; + + static_call_mod =kzalloc (sizeof (*static_call_mod),GFP_KERNEL ); +if (!static_call_mod) +return -ENOMEM ; + + static_call_mod->mod = mod; + static_call_mod->sites = site; +list_add_tail (&static_call_mod->list, &key->site_mods); + +if (is_module_address ((unsigned long )key)) { +/* +* The trampoline should no longer be used. +* Poison it it with a BUG() to catch any stray +* callers. +*/ +arch_static_call_poison_tramp (addr); + } + } + +arch_static_call_transform (addr, key->func); + } + +return 0 ; +} + +static void static_call_del_module (struct module *mod) +{ +struct static_call_site *start = mod->static_call_sites; +struct static_call_site *stop = mod->static_call_sites + + mod->num_static_call_sites; +struct static_call_site *site; +struct static_call_key *key, *prev_key =NULL ; +struct static_call_mod *static_call_mod; + +for (site = start; site < stop; site++) { + key =static_call_key (site); +if (key == prev_key) +continue ; + prev_key = key; + +list_for_each_entry (static_call_mod, &key->site_mods, list) { +if (static_call_mod->mod == mod) { +list_del (&static_call_mod->list); +kfree (static_call_mod); +break ; + } + } + } +} + +static int static_call_module_notify (struct notifier_block *nb, +unsigned long val,void *data) +{ +struct module *mod = data; +int ret =0 ; + +cpus_read_lock (); +static_call_lock (); + +switch (val) { +case MODULE_STATE_COMING : + ret =static_call_add_module (mod); +if (ret) { +WARN (1 ,"Failed to allocate memory for static calls" ); +static_call_del_module (mod); + } +break ; +case MODULE_STATE_GOING : +static_call_del_module (mod); +break ; + } + +static_call_unlock (); +cpus_read_unlock (); + +return notifier_from_errno (ret); +} + +static struct notifier_block static_call_module_nb = { + .notifier_call = static_call_module_notify, +}; + +#endif /* CONFIG_MODULES */ + +static void __init static_call_init (void ) +{ +struct static_call_site *start = __start_static_call_sites; +struct static_call_site *stop = __stop_static_call_sites; +struct static_call_site *site; + +if (start == stop) { +pr_warn ("WARNING: empty static call table\n" ); +return ; + } + +cpus_read_lock (); +static_call_lock (); + +static_call_sort_entries (start, stop); + +for (site = start; site < stop; site++) { +struct static_call_key *key =static_call_key (site); +unsigned long addr =static_call_addr (site); + +if (list_empty (&key->site_mods)) { +struct static_call_mod *mod; + + mod =kzalloc (sizeof (*mod),GFP_KERNEL ); +if (!mod) { +WARN (1 ,"Failed to allocate memory for static calls" ); +return ; + } + + mod->sites = site; +list_add_tail (&mod->list, &key->site_mods); + +/* +* The trampoline should no longer be used. Poison it +* it with a BUG() to catch any stray callers. +*/ +arch_static_call_poison_tramp (addr); + } + +arch_static_call_transform (addr, key->func); + } + + static_call_initialized = true; + +static_call_unlock (); +cpus_read_unlock (); + +#ifdef CONFIG_MODULES +register_module_notifier (&static_call_module_nb); +#endif +} +early_initcall (static_call_init); --2.17.2
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 14 lines · Ard Biesheuvel
Hi Josh,
Thanks a lot for looking into this.
Quoted 9 lines · Josh Poimboeuf
Add a static call infrastructure. Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.
This code is heavily inspired by the jump label code (aka "static jumps"), as some of the concepts are very similar.
Quoted 8 lines
Quoted 4 lines
There are three implementations, depending on arch support:
1) optimized: patched call sites (
CONFIG_HAVE_STATIC_CALL_OPTIMIZED)
2) unoptimized: patched trampolines (CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)Could we move to an idiom like inline/out of line? The unoptimized variant may be perfectly adequate for many arches, and 'unoptimized' sounds like there is something wrong with it.
Yeah, inline/out-of-line would be better. It's both more descriptive and less derogatory :-)
Quoted 19 lines
Quoted 5 lines
+ * Usage example:
+ *+ * # Start with the following functions (with identical prototypes): + * int func_a(int arg1, int arg2); + * int func_b(int arg1, int arg2);Quoted 13 lines
+ *
+ * # Define a 'my_key' reference, associated withfunc_a()by default
+ *DEFINE_STATIC_CALL(my_key, func_a);
+ *
+ * # Callfunc_a()
+ *static_call(my_key, arg1, arg2);
+ *
+ * # Update 'my_key' to point tofunc_b()
+ *static_call_update(my_key, func_b);
+ *
+ * # Callfunc_b()
+ *static_call(my_key, arg1, arg2);
+ *
Quoted 3 lines
Any way we can revert to the default implementation? That would be useful for, e.g., unloading modules that provide an alternative implementation.
I saw your original implementation had a "reset to default", but from what I can tell, most users wouldn't need that.
Assuming the caller knows what the original dest function was, can't they just call static_call_update(my_key, orig_func) directly?
Quoted 36 lines
Quoted 8 lines
include/linux/static_call_types.h new file +2
-0@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _STATIC_CALL_TYPES_Hinclude/linux/static_call_types.h line 12
Quoted 10 lines
include/linux/static_call_types.h new file +10
-0+#define _STATIC_CALL_TYPES_H + +#include <linux/stringify.h> + +#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_ +#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX) + +#define STATIC_CALL_TRAMP (key) STATIC_CALL_TRAMP_PREFIX##key +#define STATIC_CALL_TRAMP_STR (key) __stringify(STATIC_CALL_TRAMP(key)) +I needed to apply
include/linux/static_call_types.h +1
-1@@ -7,7 +7,7 @@#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX) -#define STATIC_CALL_TRAMP (key) STATIC_CALL_TRAMP_PREFIX##key +#define STATIC_CALL_TRAMP (key) __PASTE(STATIC_CALL_TRAMP_PREFIX, key)#define STATIC_CALL_TRAMP_STR (key) __stringify(STATIC_CALL_TRAMP(key))/* The static call site table is created by objtool. */
Quoted 5 lines
or I end up with
0000000000000000 <
STATIC_CALL_TRAMP_PREFIXfoo>:0: 14000000 b a8 <foo_a> 4: d503201f nop
Ha, oops. That was part of a last minute cleanup to share the macros with objtool.
Signature · 1 line
Josh
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
In reply to Josh PoimboeufQuoted 37 lines · Josh Poimboeuf
Add a static call infrastructure. Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.
This code is heavily inspired by the jump label code (aka "static jumps"), as some of the concepts are very similar.
There are three implementations, depending on arch support:
1) optimized: patched call sites (
CONFIG_HAVE_STATIC_CALL_OPTIMIZED)
2) unoptimized: patched trampolines (CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)
3) basic function pointersFor more details, see the comments in
include/linux/static_call.h.Signed-off-by Josh Poimboeuf <[email protected]>--- arch/Kconfig | 6 + include/asm-generic/vmlinux.lds.h | 11 ++ include/linux/module.h | 10 + include/linux/static_call.h | 186 +++++++++++++++++++ include/linux/static_call_types.h | 19 ++ kernel/Makefile | 1 + kernel/module.c | 5 + kernel/static_call.c | 297 ++++++++++++++++++++++++++++++ 8 files changed, 535 insertions(+) create mode 100644 include/linux/static_call.h create mode 100644 include/linux/static_call_types.h create mode 100644 kernel/static_call.ckernel/static_call.c new file +0
-0
...
Quoted 27 lines
kernel/static_call.c new file +27
-0+static void __init static_call_init (void ) +{ +struct static_call_site *start = __start_static_call_sites; +struct static_call_site *stop = __stop_static_call_sites; +struct static_call_site *site; + +if (start == stop) { +pr_warn ("WARNING: empty static call table\n" ); +return ; + } + +cpus_read_lock (); +static_call_lock (); + +static_call_sort_entries (start, stop); + +for (site = start; site < stop; site++) { +struct static_call_key *key =static_call_key (site); +unsigned long addr =static_call_addr (site); + +if (list_empty (&key->site_mods)) { +struct static_call_mod *mod; + + mod =kzalloc (sizeof (*mod),GFP_KERNEL ); +if (!mod) { +WARN (1 ,"Failed to allocate memory for static calls" ); +return ;
kernel/static_call.c
Quoted 10 lines
kernel/static_call.c new file +10
-0+ } + + mod->sites = site; +list_add_tail (&mod->list, &key->site_mods); + +/* +* The trampoline should no longer be used. Poison it +* it with a BUG() to catch any stray callers. +*/ +arch_static_call_poison_tramp (addr);
This patches the wrong thing: the trampoline is at key->func not addr.
However, patching it here means we poison it before all users are patched. I added this on top
kernel/static_call.c +3 -1
@@ -248,6 +248,7 @@ static void __init static_call_init(void)struct static_call_site *start = __start_static_call_sites;struct static_call_site *stop = __stop_static_call_sites;struct static_call_site *site; +struct static_call_key *prev_key =NULL ;if (start == stop) {pr_warn ("WARNING: empty static call table\n" );
@@ -279,7 +280,9 @@ static void __init static_call_init(void)* The trampoline should no longer be used. Poison it * it with a BUG() to catch any stray callers. */ -arch_static_call_poison_tramp (addr); +if (prev_key) +
arch_static_call_poison_tramp((unsigned long)prev_key->func);
+ prev_key = key;
}
arch_static_call_transform(addr, key->func);Quoted 18 lines
kernel/static_call.c new file +15
-1+ } + +arch_static_call_transform (addr, key->func); + } + + static_call_initialized = true; + +static_call_unlock (); +cpus_read_unlock (); + +#ifdef CONFIG_MODULES +register_module_notifier (&static_call_module_nb); +#endif +} +early_initcall (static_call_init); --2.17.2
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15
-0+ for (site = start; site < stop; site++) { + struct static_call_key *key = static_call_key(site); + unsigned long addr = static_call_addr(site); + + if (list_empty(&key->site_mods)) { + struct static_call_mod *mod; + + mod = kzalloc(sizeof(*mod), GFP_KERNEL); + if (!mod) { + WARN(1, "Failed to allocate memory for static calls"); + return; + } + + mod->sites = site; + list_add_tail(&mod->list, &key->site_mods);
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + /* + * The trampoline should no longer be used. Poison it + * it with a BUG() to catch any stray callers. + */ + arch_static_call_poison_tramp(addr);This patches the wrong thing: the trampoline is at
key->funcnot addr.
If you look at the x86 implementation, it actually does poison the trampoline.
The address of the trampoline isn't actually known here. key->func isn't the trampoline address; it's the destination func address.
So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).
The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into arch_static_call_poison_tramp().
Quoted 26 lines
However, patching it here means we poison it before all users are patched. I added this on top
kernel/static_call.c +4
-1@@ -248,6 +248,7 @@ static void __init static_call_init(void)struct static_call_site *start = __start_static_call_sites;struct static_call_site *stop = __stop_static_call_sites;struct static_call_site *site; +struct static_call_key *prev_key =NULL ;if (start == stop) {pr_warn ("WARNING: empty static call table\n" );@@ -279,7 +280,9 @@ static void __init static_call_init(void)* The trampoline should no longer be used. Poison it * it with a BUG() to catch any stray callers. */ -arch_static_call_poison_tramp (addr); +if (prev_key) +arch_static_call_poison_tramp ((unsigned long )prev_key->func); + prev_key = key; }
Quoted 1 line
arch_static_call_transform(addr, key->func);
While it does indeed poison the trampoline before all users are patched, I had been thinking that it didn't really matter because this is before the other CPUs have been booted.
But I believe interrupts are enabled at this point during the boot, so it would indeed be wise to poison it afterwards, in case an irq handler makes a static call.
Signature · 1 line
Josh
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 31 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15
-0+ for (site = start; site < stop; site++) { + struct static_call_key *key = static_call_key(site); + unsigned long addr = static_call_addr(site); + + if (list_empty(&key->site_mods)) { + struct static_call_mod *mod; + + mod = kzalloc(sizeof(*mod), GFP_KERNEL); + if (!mod) { + WARN(1, "Failed to allocate memory for static calls"); + return; + } + + mod->sites = site; + list_add_tail(&mod->list, &key->site_mods);Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + /* + * The trampoline should no longer be used. Poison it + * it with a BUG() to catch any stray callers. + */ + arch_static_call_poison_tramp(addr);This patches the wrong thing: the trampoline is at
key->funcnot addr.If you look at the x86 implementation, it actually does poison the trampoline.
The address of the trampoline isn't actually known here.
key->funcisn't the trampoline address; it's the destination func address.
Quoted 7 lines
So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).
The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into
arch_static_call_poison_tramp().
Ah right, so I am basically missing a dereference in my arch_static_call_poison_tramp() code if this breaks.
Quoted 32 lines
Quoted 26 lines
However, patching it here means we poison it before all users are patched. I added this on top
kernel/static_call.c +4
-1@@ -248,6 +248,7 @@ static void __init static_call_init(void)struct static_call_site *start = __start_static_call_sites;struct static_call_site *stop = __stop_static_call_sites;struct static_call_site *site; +struct static_call_key *prev_key =NULL ;if (start == stop) {pr_warn ("WARNING: empty static call table\n" );@@ -279,7 +280,9 @@ static void __init static_call_init(void)* The trampoline should no longer be used. Poison it * it with a BUG() to catch any stray callers. */ -arch_static_call_poison_tramp (addr); +if (prev_key) +arch_static_call_poison_tramp ((unsigned long )prev_key->func); + prev_key = key; }Quoted 1 line
arch_static_call_transform(addr, key->func);While it does indeed poison the trampoline before all users are patched, I had been thinking that it didn't really matter because this is before the other CPUs have been booted.
Quoted 4 lines
But I believe interrupts are enabled at this point during the boot, so it would indeed be wise to poison it afterwards, in case an irq handler makes a static call.
And kmalloc(GFP_KERNEL) itself could cascade into lots of other code as well.
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 35 lines · Ard Biesheuvel
Quoted 34 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15
-0+ for (site = start; site < stop; site++) { + struct static_call_key *key = static_call_key(site); + unsigned long addr = static_call_addr(site); + + if (list_empty(&key->site_mods)) { + struct static_call_mod *mod; + + mod = kzalloc(sizeof(*mod), GFP_KERNEL); + if (!mod) { + WARN(1, "Failed to allocate memory for static calls"); + return; + } + + mod->sites = site; + list_add_tail(&mod->list, &key->site_mods);Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + /* + * The trampoline should no longer be used. Poison it + * it with a BUG() to catch any stray callers. + */ + arch_static_call_poison_tramp(addr);This patches the wrong thing: the trampoline is at
key->funcnot addr.If you look at the x86 implementation, it actually does poison the trampoline.
The address of the trampoline isn't actually known here.
key->funcisn't the trampoline address; it's the destination func address.So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).
Quoted 8 lines
Quoted 4 lines
The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into
arch_static_call_poison_tramp().Ah right, so I am basically missing a dereference in my
arch_static_call_poison_tramp()code if this breaks.
Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 41 lines · Ard Biesheuvel
Quoted 40 lines · Ard Biesheuvel
Quoted 38 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15
-0+ for (site = start; site < stop; site++) { + struct static_call_key *key = static_call_key(site); + unsigned long addr = static_call_addr(site); + + if (list_empty(&key->site_mods)) { + struct static_call_mod *mod; + + mod = kzalloc(sizeof(*mod), GFP_KERNEL); + if (!mod) { + WARN(1, "Failed to allocate memory for static calls"); + return; + } + + mod->sites = site; + list_add_tail(&mod->list, &key->site_mods);Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + /* + * The trampoline should no longer be used. Poison it + * it with a BUG() to catch any stray callers. + */ + arch_static_call_poison_tramp(addr);This patches the wrong thing: the trampoline is at
key->funcnot addr.If you look at the x86 implementation, it actually does poison the trampoline.
The address of the trampoline isn't actually known here.
key->funcisn't the trampoline address; it's the destination func address.So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).
The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into
arch_static_call_poison_tramp().
Quoted 7 lines
Quoted 3 lines
Ah right, so I am basically missing a dereference in my
arch_static_call_poison_tramp()code if this breaks.Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.
Ok, but doesn't that defeat the purpose of the inline approach?
Signature · 1 line
Josh
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 46 lines · Josh Poimboeuf
Quoted 45 lines · Ard Biesheuvel
Quoted 43 lines · Ard Biesheuvel
Quoted 31 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15
-0+ for (site = start; site < stop; site++) { + struct static_call_key *key = static_call_key(site); + unsigned long addr = static_call_addr(site); + + if (list_empty(&key->site_mods)) { + struct static_call_mod *mod; + + mod = kzalloc(sizeof(*mod), GFP_KERNEL); + if (!mod) { + WARN(1, "Failed to allocate memory for static calls"); + return; + } + + mod->sites = site; + list_add_tail(&mod->list, &key->site_mods);Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + /* + * The trampoline should no longer be used. Poison it + * it with a BUG() to catch any stray callers. + */ + arch_static_call_poison_tramp(addr);This patches the wrong thing: the trampoline is at
key->funcnot addr.If you look at the x86 implementation, it actually does poison the trampoline.
The address of the trampoline isn't actually known here.
key->funcisn't the trampoline address; it's the destination func address.Quoted 7 lines
So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).
The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into
arch_static_call_poison_tramp().Ah right, so I am basically missing a dereference in my
arch_static_call_poison_tramp()code if this breaks.
Quoted 6 lines
Quoted 3 lines
Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.
Ok, but doesn't that defeat the purpose of the inline approach?
It does. But this only occurs when a module is loaded far away, and this will only happen if you have 2 GB range KASLR enabled, or your 128 MB module region gets exhausted for some reason, so the majority of calls should use a single relative branch.
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 51 lines · Ard Biesheuvel
Quoted 50 lines · Josh Poimboeuf
Quoted 48 lines · Ard Biesheuvel
Quoted 35 lines · Ard Biesheuvel
Quoted 34 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15
-0+ for (site = start; site < stop; site++) { + struct static_call_key *key = static_call_key(site); + unsigned long addr = static_call_addr(site); + + if (list_empty(&key->site_mods)) { + struct static_call_mod *mod; + + mod = kzalloc(sizeof(*mod), GFP_KERNEL); + if (!mod) { + WARN(1, "Failed to allocate memory for static calls"); + return; + } + + mod->sites = site; + list_add_tail(&mod->list, &key->site_mods);Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + /* + * The trampoline should no longer be used. Poison it + * it with a BUG() to catch any stray callers. + */ + arch_static_call_poison_tramp(addr);This patches the wrong thing: the trampoline is at
key->funcnot addr.If you look at the x86 implementation, it actually does poison the trampoline.
The address of the trampoline isn't actually known here.
key->funcisn't the trampoline address; it's the destination func address.So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).
Quoted 8 lines
Quoted 4 lines
The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into
arch_static_call_poison_tramp().Ah right, so I am basically missing a dereference in my
arch_static_call_poison_tramp()code if this breaks.Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.
Quoted 7 lines
Quoted 2 lines
Ok, but doesn't that defeat the purpose of the inline approach?
It does. But this only occurs when a module is loaded far away, and this will only happen if you have 2 GB range KASLR enabled, or your 128 MB module region gets exhausted for some reason, so the majority of calls should use a single relative branch.
Makes sense. Do you also account for the possibility that the original call emitted by GCC was far away and thus used the PLT?
Signature · 1 line
Josh
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 55 lines · Josh Poimboeuf
Quoted 54 lines · Ard Biesheuvel
Quoted 52 lines · Josh Poimboeuf
Quoted 41 lines · Ard Biesheuvel
Quoted 40 lines · Ard Biesheuvel
Quoted 38 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15
-0+ for (site = start; site < stop; site++) { + struct static_call_key *key = static_call_key(site); + unsigned long addr = static_call_addr(site); + + if (list_empty(&key->site_mods)) { + struct static_call_mod *mod; + + mod = kzalloc(sizeof(*mod), GFP_KERNEL); + if (!mod) { + WARN(1, "Failed to allocate memory for static calls"); + return; + } + + mod->sites = site; + list_add_tail(&mod->list, &key->site_mods);Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + /* + * The trampoline should no longer be used. Poison it + * it with a BUG() to catch any stray callers. + */ + arch_static_call_poison_tramp(addr);This patches the wrong thing: the trampoline is at
key->funcnot addr.If you look at the x86 implementation, it actually does poison the trampoline.
The address of the trampoline isn't actually known here.
key->funcisn't the trampoline address; it's the destination func address.So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).
The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into
arch_static_call_poison_tramp().Quoted 7 lines
Quoted 3 lines
Ah right, so I am basically missing a dereference in my
arch_static_call_poison_tramp()code if this breaks.Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.
Ok, but doesn't that defeat the purpose of the inline approach?
Quoted 8 lines
Quoted 4 lines
It does. But this only occurs when a module is loaded far away, and this will only happen if you have 2 GB range KASLR enabled, or your 128 MB module region gets exhausted for some reason, so the majority of calls should use a single relative branch.
Makes sense. Do you also account for the possibility that the original call emitted by GCC was far away and thus used the PLT?
That doesn't really happen. vmlinux is never over 128 MB, and the modules are only partially linked, so the linker never gets to the stage where it needs to emit veneers.
It's a bit fiddly since inline and out-of-line both use arch_static_call_transform(), but what I need to do is basically:
- for out-of-line, the trampoline needs to be patched into a movn/movk/movk/br sequence if the target is too far
- for inline, the trampoline itself needs to be patched from adrp/ldr/br (which does a load and a indirect branch) to movn/movk/movk/br (which uses immediates), and the call sites need to be patched into calls to the veneer if the target is out of range.
So arch_static_call_transform() needs to know where the trampoline is for this use case, so could we perhaps add a 'void *orig' in the key struct that keeps track of the original value of 'addr' ?
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 79 lines · Ard Biesheuvel
Quoted 55 lines · Josh Poimboeuf
Quoted 54 lines · Ard Biesheuvel
Quoted 52 lines · Josh Poimboeuf
Quoted 41 lines · Ard Biesheuvel
Quoted 40 lines · Ard Biesheuvel
Quoted 38 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15
-0+ for (site = start; site < stop; site++) { + struct static_call_key *key = static_call_key(site); + unsigned long addr = static_call_addr(site); + + if (list_empty(&key->site_mods)) { + struct static_call_mod *mod; + + mod = kzalloc(sizeof(*mod), GFP_KERNEL); + if (!mod) { + WARN(1, "Failed to allocate memory for static calls"); + return; + } + + mod->sites = site; + list_add_tail(&mod->list, &key->site_mods);Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + /* + * The trampoline should no longer be used. Poison it + * it with a BUG() to catch any stray callers. + */ + arch_static_call_poison_tramp(addr);This patches the wrong thing: the trampoline is at
key->funcnot addr.If you look at the x86 implementation, it actually does poison the trampoline.
The address of the trampoline isn't actually known here.
key->funcisn't the trampoline address; it's the destination func address.So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).
The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into
arch_static_call_poison_tramp().Quoted 7 lines
Quoted 3 lines
Ah right, so I am basically missing a dereference in my
arch_static_call_poison_tramp()code if this breaks.Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.
Ok, but doesn't that defeat the purpose of the inline approach?
Quoted 8 lines
Quoted 4 lines
It does. But this only occurs when a module is loaded far away, and this will only happen if you have 2 GB range KASLR enabled, or your 128 MB module region gets exhausted for some reason, so the majority of calls should use a single relative branch.
Makes sense. Do you also account for the possibility that the original call emitted by GCC was far away and thus used the PLT?
That doesn't really happen. vmlinux is never over 128 MB, and the modules are only partially linked, so the linker never gets to the stage where it needs to emit veneers.
It's a bit fiddly since inline and out-of-line both use
arch_static_call_transform(), but what I need to do is basically:- for out-of-line, the trampoline needs to be patched into a movn/movk/movk/br sequence if the target is too far
- for inline, the trampoline itself needs to be patched from adrp/ldr/br (which does a load and a indirect branch) to movn/movk/movk/br (which uses immediates), and the call sites need to be patched into calls to the veneer if the target is out of range.
Quoted 3 lines
So
arch_static_call_transform()needs to know where the trampoline is for this use case, so could we perhaps add a 'void *orig' in the key struct that keeps track of the original value of 'addr' ?
... and pass it to arch_static_call_transform
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 10 lines · Ard Biesheuvel
Quoted 10 lines
It's a bit fiddly since inline and out-of-line both use
arch_static_call_transform(), but what I need to do is basically:- for out-of-line, the trampoline needs to be patched into a movn/movk/movk/br sequence if the target is too far
- for inline, the trampoline itself needs to be patched from adrp/ldr/br (which does a load and a indirect branch) to movn/movk/movk/br (which uses immediates), and the call sites need to be patched into calls to the veneer if the target is out of range.
Quoted 5 lines
Quoted 3 lines
So
arch_static_call_transform()needs to know where the trampoline is for this use case, so could we perhaps add a 'void *orig' in the key struct that keeps track of the original value of 'addr' ?... and pass it to
arch_static_call_transform
Ok. So in both cases, you need both the call site address and the tramp address, right? We could add a 'tramp' pointer to the key struct -- I assume that's what you mean?
And also change
void arch_static_call_transform(unsigned long insn, void *dest)
to
void arch_static_call_transform(void *call_site, void *tramp, void *dest)
Signature · 1 line
Josh
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 15 lines · Josh Poimboeuf
Quoted 14 lines · Ard Biesheuvel
Quoted 13 lines
It's a bit fiddly since inline and out-of-line both use
arch_static_call_transform(), but what I need to do is basically:- for out-of-line, the trampoline needs to be patched into a movn/movk/movk/br sequence if the target is too far
- for inline, the trampoline itself needs to be patched from adrp/ldr/br (which does a load and a indirect branch) to movn/movk/movk/br (which uses immediates), and the call sites need to be patched into calls to the veneer if the target is out of range.So
arch_static_call_transform()needs to know where the trampoline is for this use case, so could we perhaps add a 'void *orig' in the key struct that keeps track of the original value of 'addr' ?
Quoted 6 lines
Quoted 1 line
... and pass it to
arch_static_call_transformOk. So in both cases, you need both the call site address and the tramp address, right? We could add a 'tramp' pointer to the key struct -- I assume that's what you mean?
Yes.
Quoted 8 lines
And also change
void
arch_static_call_transform(unsigned long insn, void *dest)to
void
arch_static_call_transform(void *call_site, void *tramp, void *dest)
Indeed.
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
In reply to Josh PoimboeufQuoted 46 lines · Josh Poimboeuf
Quoted 45 lines · Ard Biesheuvel
Quoted 43 lines · Ard Biesheuvel
Quoted 31 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15
-0+ for (site = start; site < stop; site++) { + struct static_call_key *key = static_call_key(site); + unsigned long addr = static_call_addr(site); + + if (list_empty(&key->site_mods)) { + struct static_call_mod *mod; + + mod = kzalloc(sizeof(*mod), GFP_KERNEL); + if (!mod) { + WARN(1, "Failed to allocate memory for static calls"); + return; + } + + mod->sites = site; + list_add_tail(&mod->list, &key->site_mods);Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ + /* + * The trampoline should no longer be used. Poison it + * it with a BUG() to catch any stray callers. + */ + arch_static_call_poison_tramp(addr);This patches the wrong thing: the trampoline is at
key->funcnot addr.If you look at the x86 implementation, it actually does poison the trampoline.
The address of the trampoline isn't actually known here.
key->funcisn't the trampoline address; it's the destination func address.Quoted 7 lines
So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).
The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into
arch_static_call_poison_tramp().Ah right, so I am basically missing a dereference in my
arch_static_call_poison_tramp()code if this breaks.
Quoted 5 lines
Quoted 3 lines
Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.
Ok, but doesn't that defeat the purpose of the inline approach?
Or are you only going to use the trampoline for out-of-range calls, otherwise just do direct calls?
Signature · 1 line
Josh
Steven Rostedt
reviewerRe: [RFC PATCH 1/3] static_call: Add static call infrastructure
In reply to Josh PoimboeufQuoted 6 lines · On Thu, 8 Nov 2018 15:15:51 -0600 Josh Poimboeuf <[email protected]> wrote:
Add a static call infrastructure. Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.
Quoted 6 lines
This code is heavily inspired by the jump label code (aka "static jumps"), as some of the concepts are very similar.
There are three implementations, depending on arch support:
1) optimized: patched call sites (
CONFIG_HAVE_STATIC_CALL_OPTIMIZED)
Thanks for doing this.
Quoted 1 line
2) unoptimized: patched trampolines (
CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)
#2 looks like my approach.
Quoted 24 lines
3) basic function pointers
For more details, see the comments in
include/linux/static_call.h.Signed-off-by Josh Poimboeuf <[email protected]>--- arch/Kconfig | 6 + include/asm-generic/vmlinux.lds.h | 11 ++ include/linux/module.h | 10 + include/linux/static_call.h | 186 +++++++++++++++++++ include/linux/static_call_types.h | 19 ++ kernel/Makefile | 1 + kernel/module.c | 5 + kernel/static_call.c | 297 ++++++++++++++++++++++++++++++ 8 files changed, 535 insertions(+) create mode 100644 include/linux/static_call.h create mode 100644 include/linux/static_call_types.h create mode 100644 kernel/static_call.carch/Kconfig +0
-0
arch/Kconfig config HAVE_ARCH_PREL32_RELOCATIONS · line 887
Quoted 9 lines
arch/Kconfig +6
-0@@ -879,6 +879,12 @@ config HAVE_ARCH_PREL32_RELOCATIONS architectures, and don't require runtime relocation on relocatable kernels. +config HAVE_STATIC_CALL_OPTIMIZED + bool + +config HAVE_STATIC_CALL_UNOPTIMIZED + bool +
Let's also have
config HAVE_STATIC_CALL
def_bool Y depends on HAVE_STATIC_CALL_OPTIMIZED || HAVE_STATIC_CALL_UNOPTIMIZED
Quoted 137 lines
arch/Kconfig +0
-0source "kernel/gcov/Kconfig" source "scripts/gcc-plugins/Kconfig"include/asm-generic/vmlinux.lds.h +11
-0@@ -320,6 +320,7 @@ __start_ro_after_init = .; \ *(.data..ro_after_init) \ JUMP_TABLE_DATA \ + STATIC_CALL_SITES \ __end_ro_after_init = .;#endif @@ -725,6 +726,16 @@#define BUG_TABLE #endif +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED +#define STATIC_CALL_SITES \ + . = ALIGN(8); \ + __start_static_call_sites = .; \ + KEEP(*(.static_call_sites)) \ + __stop_static_call_sites = .; +#else +#define STATIC_CALL_SITES +#endif +#ifdef CONFIG_UNWINDER_ORC #define ORC_UNWIND_TABLE \ . = ALIGN(4); \include/linux/module.h +10
-0@@ -21,6 +21,7 @@#include <linux/rbtree_latch.h> #include <linux/error-injection.h> #include <linux/tracepoint-defs.h> +#include <linux/static_call_types.h> #include <linux/percpu.h> #include <asm/module.h> @@ -450,6 +451,10 @@ struct module {unsigned int num_ftrace_callsites;unsigned long *ftrace_callsites;#endif +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED +int num_static_call_sites; +struct static_call_site *static_call_sites; +#endif #ifdef CONFIG_LIVEPATCH bool klp;/* Is this a livepatch module? */ @@ -682,6 +687,11 @@ static inline bool is_module_text_address(unsigned long addr)return false; } +static inline bool within_module_init (unsigned long addr,const struct module *mod) +{ +return false; +} +/* Get/put a kernel symbol (calls should be symmetric) */ #define symbol_get (x) ({ extern typeof(x) x __attribute__((weak)); &(x); })#define symbol_put (x) do { } while (0)include/linux/static_call.h new file +64
-0@@ -0,0 +1,186 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_STATIC_CALL_H +#define _LINUX_STATIC_CALL_H + +/* +* Static call support +* +* Static calls use code patching to hard-code function pointers into direct +* branch instructions. They give the flexibility of function pointers, but +* with improved performance. This is especially important for cases where +* retpolines would otherwise be used, as retpolines can significantly impact +* performance. +* +* +* API overview: +* +* DECLARE_STATIC_CALL(key, func); +* DEFINE_STATIC_CALL(key, func); +* static_call(key, args...); +* static_call_update(key, func); +* +* +* Usage example: +* +* # Start with the following functions (with identical prototypes): +* int func_a(int arg1, int arg2); +* int func_b(int arg1, int arg2); +* +* # Define a 'my_key' reference, associated with func_a() by default +* DEFINE_STATIC_CALL(my_key, func_a); +* +* # Call func_a() +* static_call(my_key, arg1, arg2); +* +* # Update 'my_key' to point to func_b() +* static_call_update(my_key, func_b); +* +* # Call func_b() +* static_call(my_key, arg1, arg2); +* +* +* Implementation details: +* +* There are three different implementations: +* +* 1) Optimized static calls (patched call sites) +* +* This requires objtool, which detects all the static_call() sites and +* annotates them in the '.static_call_sites' section. By default, the call +* sites will call into a temporary per-key trampoline which has an indirect +* branch to the current destination function associated with the key. +* During system boot (or module init), all call sites are patched to call +* their destination functions directly. Updates to a key will patch all +* call sites associated with that key. +* +* 2) Unoptimized static calls (patched trampolines) +* +* Each static_call() site calls into a permanent trampoline associated with +* the key. The trampoline has a direct branch to the default function. +* Updates to a key will modify the direct branch in the key's trampoline. +* +* 3) Generic implementation +* +* This is the default implementation if the architecture hasn't implemented
include/linux/static_call.h line 74
Quoted 10 lines
include/linux/static_call.h new file +10
-0+* CONFIG_HAVE_STATIC_CALL_[UN]OPTIMIZED. In this case, a basic +* function pointer is used. +*/ + +#include <linux/types.h> +#include <linux/cpu.h> +#include <linux/static_call_types.h> + +#if defined(CONFIG_HAVE_STATIC_CALL_OPTIMIZED ) || \ + defined(CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED )
Now we can have:
#ifdef CONFIG_HAVE_STATIC_CALL
Quoted 28 lines
include/linux/static_call.h new file +28
-0+ +#include <asm/static_call.h> + +extern void arch_static_call_transform (unsigned long insn,void *dest); + +#endif /* CONFIG_HAVE_STATIC_CALL_[UN]OPTIMIZED */ + +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED +/* Optimized implementation */ + +struct static_call_mod { + struct list_head list; + struct module *mod; /* NULL means vmlinux */ + struct static_call_site *sites; +}; + +struct static_call_key { + /* + * This field should always be first because the trampolines expect it. + * This points to the current call destination. + */ + void *func; + + /* + * List of modules (including vmlinux) and the call sites which need to + * be patched for this key. + */ + struct list_head site_mods;
include/linux/static_call.h line 112
Quoted 10 lines
include/linux/static_call.h new file +10
-0+}; + +extern void __static_call_update (struct static_call_key *key,void *func); +extern void arch_static_call_poison_tramp (unsigned long insn); + +#define DECLARE_STATIC_CALL (key, func) \ + extern struct static_call_key key; \ + extern typeof(func) STATIC_CALL_TRAMP(key); \ +/* Preserve the ELF symbol so objtool can access it: */ \ +__ADDRESSABLE (key)
Does the __ADDRESSABLE(key) need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.
Quoted 221 lines
include/linux/static_call.h new file +74
-0+ +#define DEFINE_STATIC_CALL (key, _func) \ + DECLARE_STATIC_CALL(key, _func); \ + struct static_call_key key = { \ + .func = _func, \ + .site_mods = LIST_HEAD_INIT(key.site_mods), \ + }; \ + ARCH_STATIC_CALL_TEMPORARY_TRAMP(key) + +#define static_call(key, args...) STATIC_CALL_TRAMP(key)(args) + +#define static_call_update (key, func) \ +({ \ + BUILD_BUG_ON(!__same_type(typeof(func), typeof(STATIC_CALL_TRAMP(key)))); \ + __static_call_update(&key, func); \ +}) + +#define EXPORT_STATIC_CALL (key) \ + EXPORT_SYMBOL(key); \ + EXPORT_SYMBOL(STATIC_CALL_TRAMP(key)) + +#define EXPORT_STATIC_CALL_GPL (key) \ + EXPORT_SYMBOL_GPL(key); \ + EXPORT_SYMBOL_GPL(STATIC_CALL_TRAMP(key)) + + +#elif defined(CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED ) +/* Unoptimized implementation */ + +#define DECLARE_STATIC_CALL (key, func) \ + extern typeof(func) STATIC_CALL_TRAMP(key) + +#define DEFINE_STATIC_CALL (key, func) \ + DECLARE_STATIC_CALL(key, func); \ + ARCH_STATIC_CALL_TRAMP(key, func) + +#define static_call(key, args...) STATIC_CALL_TRAMP(key)(args) + +#define static_call_update (key, func) \ +({ \ + BUILD_BUG_ON(!__same_type(func, STATIC_CALL_TRAMP(key))); \ + cpus_read_lock(); \ + arch_static_call_transform((unsigned long)STATIC_CALL_TRAMP(key),\ + func); \ + cpus_read_unlock(); \ +}) + +#define EXPORT_STATIC_CALL (key) \ + EXPORT_SYMBOL(STATIC_CALL_TRAMP(key)) + +#define EXPORT_STATIC_CALL_GPL (key) \ + EXPORT_SYMBOL_GPL(STATIC_CALL_TRAMP(key)) + + +#else /* Generic implementation */ + +#define DECLARE_STATIC_CALL (key, func) \ + extern typeof(func) *key + +#define DEFINE_STATIC_CALL (key, func) \ + typeof(func) *key = func + +#define static_call(key, args...) \ + key(args) + +#define static_call_update (key, func) \ + WRITE_ONCE(key, func) + +#define EXPORT_STATIC_CALL (key) EXPORT_SYMBOL(key) +#define EXPORT_STATIC_CALL_GPL (key) EXPORT_SYMBOL_GPL(key) + +#endif /* CONFIG_HAVE_STATIC_CALL_OPTIMIZED */ + +#endif /* _LINUX_STATIC_CALL_H */ include/linux/static_call_types.h new file +19
-0@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _STATIC_CALL_TYPES_H +#define _STATIC_CALL_TYPES_H + +#include <linux/stringify.h> + +#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_ +#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX) + +#define STATIC_CALL_TRAMP (key) STATIC_CALL_TRAMP_PREFIX##key +#define STATIC_CALL_TRAMP_STR (key) __stringify(STATIC_CALL_TRAMP(key)) + +/* The static call site table is created by objtool. */ +struct static_call_site { +s32 addr; +s32 key; +}; + +#endif /* _STATIC_CALL_TYPES_H */ kernel/Makefile +1
-0@@ -103,6 +103,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/ obj-$(CONFIG_IRQ_WORK) += irq_work.o obj-$(CONFIG_CPU_PM) += cpu_pm.o obj-$(CONFIG_BPF) += bpf/ +obj-$(CONFIG_HAVE_STATIC_CALL_OPTIMIZED) += static_call.o obj-$(CONFIG_PERF_EVENTS) += events/kernel/module.c +5
-0@@ -3121,6 +3121,11 @@ static int find_module_sections(struct module *mod, struct load_info *info) mod->ei_funcs =section_objs (info,"_error_injection_whitelist" ,sizeof (*mod->ei_funcs), &mod->num_ei_funcs); +#endif +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED + mod->static_call_sites =section_objs (info,".static_call_sites" , +sizeof (*mod->static_call_sites), + &mod->num_static_call_sites);#endif mod->extable =section_objs (info,"__ex_table" ,sizeof (*mod->extable), &mod->num_exentries);kernel/static_call.c new file +88
-0@@ -0,0 +1,297 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#include <linux/init.h> +#include <linux/static_call.h> +#include <linux/bug.h> +#include <linux/smp.h> +#include <linux/sort.h> +#include <linux/slab.h> +#include <linux/module.h> +#include <linux/cpu.h> +#include <linux/processor.h> +#include <asm/sections.h> + +extern struct static_call_site __start_static_call_sites[], + __stop_static_call_sites[]; + +static bool static_call_initialized; + +/* mutex to protect key modules/sites */ +static DEFINE_MUTEX (static_call_mutex); + +static void static_call_lock (void ) +{ +mutex_lock (&static_call_mutex); +} + +static void static_call_unlock (void ) +{ +mutex_unlock (&static_call_mutex); +} + +static inline unsigned long static_call_addr (struct static_call_site *site) +{ +return (long )site->addr + (long )&site->addr; +} + +static inline struct static_call_key *static_call_key (const struct static_call_site *site) +{ +return (struct static_call_key *)((long )site->key + (long )&site->key); +} + +static int static_call_site_cmp (const void *_a,const void *_b) +{ +const struct static_call_site *a = _a; +const struct static_call_site *b = _b; +const struct static_call_key *key_a =static_call_key (a); +const struct static_call_key *key_b =static_call_key (b); + +if (key_a < key_b) +return -1 ; + +if (key_a > key_b) +return 1 ; + +return 0 ; +} + +static void static_call_site_swap (void *_a,void *_b,int size) +{ +long delta = (unsigned long )_a - (unsigned long )_b; +struct static_call_site *a = _a; +struct static_call_site *b = _b; +struct static_call_site tmp = *a; + + a->addr = b->addr - delta; + a->key = b->key - delta; + + b->addr = tmp.addr + delta; + b->key = tmp.key + delta; +} + +static inline void static_call_sort_entries (struct static_call_site *start, +struct static_call_site *stop) +{ +sort (start, stop - start,sizeof (struct static_call_site ), + static_call_site_cmp, static_call_site_swap); +} + +void __static_call_update (struct static_call_key *key,void *func) +{ +struct static_call_mod *mod; +struct static_call_site *site, *stop; + +cpus_read_lock (); +static_call_lock (); + +if (key->func == func) + goto done; +
kernel/static_call.c line 98
Quoted 10 lines
kernel/static_call.c new file +10
-0+ key->func = func; + +/* +* If called before init, leave the call sites unpatched for now. +* In the meantime they'll continue to call the temporary trampoline. +*/ +if (!static_call_initialized) + goto done; + +list_for_each_entry (mod, &key->site_mods, list) {
Since I'm expecting a lot of sites, I'm wondering if we should just do this as an array, like I do with the ftrace call sites.
But this can be an enhancement for later. Let's focus on getting this working first.
kernel/static_call.c line 110
Quoted 12 lines
kernel/static_call.c new file +12
-0+if (!mod->sites) { +/* +* This can happen if the static call key is defined in +* a module which doesn't use it. +*/ +continue ; + } + + stop = __stop_static_call_sites; + +#ifdef CONFIG_MODULES +if (mod->mod) {
BTW, "mod" is an unfortunate name.
Quoted 4 lines
kernel/static_call.c new file +4
-0+ stop = mod->mod->static_call_sites + + mod->mod->num_static_call_sites; + } +#endif
kernel/static_call.c line 124
Quoted 10 lines
kernel/static_call.c new file +10
-0+ +for (site = mod->sites; + site < stop &&static_call_key (site) == key; site++) { +unsigned long addr =static_call_addr (site); + +if (!mod->mod &&init_section_contains ((void *)addr,1 )) +continue ; +if (mod->mod &&within_module_init (addr, mod->mod)) +continue ; +
So what's the reason for skipping init calls?
-- Steve
Quoted 173 lines
kernel/static_call.c new file +173
-0+arch_static_call_transform (addr, func); + } + } + +done: +static_call_unlock (); +cpus_read_unlock (); +} +EXPORT_SYMBOL_GPL (__static_call_update); + +#ifdef CONFIG_MODULES + +static int static_call_add_module (struct module *mod) +{ +struct static_call_site *start = mod->static_call_sites; +struct static_call_site *stop = mod->static_call_sites + + mod->num_static_call_sites; +struct static_call_site *site; +struct static_call_key *key, *prev_key =NULL ; +struct static_call_mod *static_call_mod; + +if (start == stop) +return 0 ; + +module_disable_ro (mod); +static_call_sort_entries (start, stop); +module_enable_ro (mod, false); + +for (site = start; site < stop; site++) { +unsigned long addr =static_call_addr (site); + +if (within_module_init (addr, mod)) +continue ; + + key =static_call_key (site); +if (key != prev_key) { + prev_key = key; + + static_call_mod =kzalloc (sizeof (*static_call_mod),GFP_KERNEL ); +if (!static_call_mod) +return -ENOMEM ; + + static_call_mod->mod = mod; + static_call_mod->sites = site; +list_add_tail (&static_call_mod->list, &key->site_mods); + +if (is_module_address ((unsigned long )key)) { +/* +* The trampoline should no longer be used. +* Poison it it with a BUG() to catch any stray +* callers. +*/ +arch_static_call_poison_tramp (addr); + } + } + +arch_static_call_transform (addr, key->func); + } + +return 0 ; +} + +static void static_call_del_module (struct module *mod) +{ +struct static_call_site *start = mod->static_call_sites; +struct static_call_site *stop = mod->static_call_sites + + mod->num_static_call_sites; +struct static_call_site *site; +struct static_call_key *key, *prev_key =NULL ; +struct static_call_mod *static_call_mod; + +for (site = start; site < stop; site++) { + key =static_call_key (site); +if (key == prev_key) +continue ; + prev_key = key; + +list_for_each_entry (static_call_mod, &key->site_mods, list) { +if (static_call_mod->mod == mod) { +list_del (&static_call_mod->list); +kfree (static_call_mod); +break ; + } + } + } +} + +static int static_call_module_notify (struct notifier_block *nb, +unsigned long val,void *data) +{ +struct module *mod = data; +int ret =0 ; + +cpus_read_lock (); +static_call_lock (); + +switch (val) { +case MODULE_STATE_COMING : + ret =static_call_add_module (mod); +if (ret) { +WARN (1 ,"Failed to allocate memory for static calls" ); +static_call_del_module (mod); + } +break ; +case MODULE_STATE_GOING : +static_call_del_module (mod); +break ; + } + +static_call_unlock (); +cpus_read_unlock (); + +return notifier_from_errno (ret); +} + +static struct notifier_block static_call_module_nb = { + .notifier_call = static_call_module_notify, +}; + +#endif /* CONFIG_MODULES */ + +static void __init static_call_init (void ) +{ +struct static_call_site *start = __start_static_call_sites; +struct static_call_site *stop = __stop_static_call_sites; +struct static_call_site *site; + +if (start == stop) { +pr_warn ("WARNING: empty static call table\n" ); +return ; + } + +cpus_read_lock (); +static_call_lock (); + +static_call_sort_entries (start, stop); + +for (site = start; site < stop; site++) { +struct static_call_key *key =static_call_key (site); +unsigned long addr =static_call_addr (site); + +if (list_empty (&key->site_mods)) { +struct static_call_mod *mod; + + mod =kzalloc (sizeof (*mod),GFP_KERNEL ); +if (!mod) { +WARN (1 ,"Failed to allocate memory for static calls" ); +return ; + } + + mod->sites = site; +list_add_tail (&mod->list, &key->site_mods); + +/* +* The trampoline should no longer be used. Poison it +* it with a BUG() to catch any stray callers. +*/ +arch_static_call_poison_tramp (addr); + } + +arch_static_call_transform (addr, key->func); + } + + static_call_initialized = true; + +static_call_unlock (); +cpus_read_unlock (); + +#ifdef CONFIG_MODULES +register_module_notifier (&static_call_module_nb); +#endif +} +early_initcall (static_call_init);
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 13 lines · Steven Rostedt
arch/Kconfig config HAVE_ARCH_PREL32_RELOCATIONS · line 887
Quoted 6 lines
arch/Kconfig +6
-0+config HAVE_STATIC_CALL_OPTIMIZED + bool + +config HAVE_STATIC_CALL_UNOPTIMIZED + bool +Let's also have
config
HAVE_STATIC_CALLdef_bool Y depends on HAVE_STATIC_CALL_OPTIMIZED || HAVE_STATIC_CALL_UNOPTIMIZED
Quoted 6 lines
include/linux/static_call.h line 74
Quoted 2 lines
include/linux/static_call.h new file +2
-0+#if defined(CONFIG_HAVE_STATIC_CALL_OPTIMIZED ) || \ + defined(CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED )Now we can have:
#ifdef
CONFIG_HAVE_STATIC_CALL
Yeah, that's better.
Quoted 9 lines
include/linux/static_call.h line 112
Quoted 5 lines
include/linux/static_call.h new file +5
-0+#define DECLARE_STATIC_CALL (key, func) \ + extern struct static_call_key key; \ + extern typeof(func) STATIC_CALL_TRAMP(key); \ +/* Preserve the ELF symbol so objtool can access it: */ \ +__ADDRESSABLE (key)Does the
__ADDRESSABLE(key)need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.
For each call site, objtool creates a struct in .static_call_sites:
struct static_call_site {
s32 addr;
s32 key;
};In order to do that, it needs to create a relocation which references the key symbol. If the key is defined in another .o file, then the current .o will not have an ELF symbol associated with the key. The __ADDRESSABLE(key) thing tells GCC to leave the key symbol in the .o file, even though it's not referenced anywhere. That makes objtool's job easier, so it doesn't have to edit the symbol table.
I could add a comment saying as much, though it's hard to explain it in fewer words than I just did :-)
Quoted 12 lines
kernel/static_call.c line 98
Quoted 8 lines
kernel/static_call.c new file +8
-0+/* +* If called before init, leave the call sites unpatched for now. +* In the meantime they'll continue to call the temporary trampoline. +*/ +if (!static_call_initialized) + goto done; + +list_for_each_entry (mod, &key->site_mods, list) {Since I'm expecting a lot of sites, I'm wondering if we should just do this as an array, like I do with the ftrace call sites.
Quoted 2 lines
But this can be an enhancement for later. Let's focus on getting this working first.
But it's not a static list. It can grow/shrink as modules are loaded/unload.
Quoted 15 lines
kernel/static_call.c line 110
Quoted 13 lines
kernel/static_call.c new file +12
-0+if (!mod->sites) { +/* +* This can happen if the static call key is defined in +* a module which doesn't use it. +*/ +continue ; + } + + stop = __stop_static_call_sites; + +#ifdef CONFIG_MODULES +if (mod->mod) {BTW, "mod" is an unfortunate name.
True. I'll change it to 'site_mod'.
Quoted 10 lines
Quoted 10 lines
kernel/static_call.c new file +9
-0+ stop = mod->mod->static_call_sites + + mod->mod->num_static_call_sites; + } +#endif + +for (site = mod->sites; + site < stop &&static_call_key (site) == key; site++) { +unsigned long addr =static_call_addr (site); +
Quoted 8 lines
kernel/static_call.c line 124
Quoted 5 lines
kernel/static_call.c new file +5
-0+if (!mod->mod &&init_section_contains ((void *)addr,1 )) +continue ; +if (mod->mod &&within_module_init (addr, mod->mod)) +continue ; +So what's the reason for skipping init calls?
This is the runtime changing code (static_call_update). Presumably the init sections no longer exist and we shouldn't write to any (former) call sites there.
That's probably a dangerous assumption though... If static_call_update() were called early, some init code might not get patched and then call into the wrong function.
I'm thinking we should just disallow static call sites in init sections. I can't think of a good reason why they would be needed in init code. We can WARN when detecting them during boot / module init.
Signature · 1 line
Josh
Steven Rostedt
reviewerRe: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 24 lines · On Fri, 9 Nov 2018 13:35:05 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 9 lines
include/linux/static_call.h line 112
Quoted 5 lines
include/linux/static_call.h new file +5
-0+#define DECLARE_STATIC_CALL (key, func) \ + extern struct static_call_key key; \ + extern typeof(func) STATIC_CALL_TRAMP(key); \ +/* Preserve the ELF symbol so objtool can access it: */ \ +__ADDRESSABLE (key)Does the
__ADDRESSABLE(key)need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.For each call site, objtool creates a struct in .static_call_sites:
struct static_call_site { s32 addr; s32 key; };In order to do that, it needs to create a relocation which references the key symbol. If the key is defined in another .o file, then the current .o will not have an ELF symbol associated with the key. The
__ADDRESSABLE(key)thing tells GCC to leave the key symbol in the .o file, even though it's not referenced anywhere. That makes objtool's job easier, so it doesn't have to edit the symbol table.
Quoted 2 lines
I could add a comment saying as much, though it's hard to explain it in fewer words than I just did :-)
Does this have to do with adding the references by relative address?
In record_mcount, I just picked an existing symbol and referenced that.. But perhaps this is a cleaner way.
Adding a more in depth comment wont hurt.
Quoted 13 lines
Quoted 13 lines
kernel/static_call.c line 98
Quoted 9 lines
kernel/static_call.c new file +8
-0+/* +* If called before init, leave the call sites unpatched for now. +* In the meantime they'll continue to call the temporary trampoline. +*/ +if (!static_call_initialized) + goto done; + +list_for_each_entry (mod, &key->site_mods, list) {Since I'm expecting a lot of sites, I'm wondering if we should just do this as an array, like I do with the ftrace call sites.
Quoted 5 lines
Quoted 2 lines
But this can be an enhancement for later. Let's focus on getting this working first.
But it's not a static list. It can grow/shrink as modules are loaded/unload.
Neither is ftrace :-) What I did was make one array for the core kernel code, and an array for each module, and link list those (single link, although double link may not be hard either). That will save a lot of memory than having each instance have a link pointer, as it only grows or shrinks in chunks.
Quoted 42 lines
Quoted 16 lines
Quoted 4 lines
kernel/static_call.c new file +2
-0+if (!mod->sites) { +/* kernel/static_call.c line 110
Quoted 10 lines
kernel/static_call.c new file +10
-0+* This can happen if the static call key is defined in +* a module which doesn't use it. +*/ +continue ; + } + + stop = __stop_static_call_sites; + +#ifdef CONFIG_MODULES +if (mod->mod) {BTW, "mod" is an unfortunate name.
True. I'll change it to '
site_mod'.Quoted 10 lines
Quoted 10 lines
kernel/static_call.c new file +9
-0+ stop = mod->mod->static_call_sites + + mod->mod->num_static_call_sites; + } +#endif + +for (site = mod->sites; + site < stop &&static_call_key (site) == key; site++) { +unsigned long addr =static_call_addr (site); +Quoted 8 lines
kernel/static_call.c line 124
Quoted 5 lines
kernel/static_call.c new file +5
-0+if (!mod->mod &&init_section_contains ((void *)addr,1 )) +continue ; +if (mod->mod &&within_module_init (addr, mod->mod)) +continue ; +So what's the reason for skipping init calls?
This is the runtime changing code (
static_call_update). Presumably the init sections no longer exist and we shouldn't write to any (former) call sites there.
Quoted 8 lines
That's probably a dangerous assumption though... If
static_call_update()were called early, some init code might not get patched and then call into the wrong function.I'm thinking we should just disallow static call sites in init sections. I can't think of a good reason why they would be needed in init code. We can WARN when detecting them during boot / module init.
What I would do is to allow init (like ftrace now does). I have ftrace_free_init_mem() that removes all the mcount references for init calls from its list. You could add a static_call_free_init() to kernel_init() in init/main.c too.
-- Steve
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 28 lines · Steven Rostedt
Quoted 24 lines · On Fri, 9 Nov 2018 13:35:05 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 9 lines
include/linux/static_call.h line 112
Quoted 5 lines
include/linux/static_call.h new file +5
-0+#define DECLARE_STATIC_CALL (key, func) \ + extern struct static_call_key key; \ + extern typeof(func) STATIC_CALL_TRAMP(key); \ +/* Preserve the ELF symbol so objtool can access it: */ \ +__ADDRESSABLE (key)Does the
__ADDRESSABLE(key)need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.For each call site, objtool creates a struct in .static_call_sites:
struct static_call_site { s32 addr; s32 key; };In order to do that, it needs to create a relocation which references the key symbol. If the key is defined in another .o file, then the current .o will not have an ELF symbol associated with the key. The
__ADDRESSABLE(key)thing tells GCC to leave the key symbol in the .o file, even though it's not referenced anywhere. That makes objtool's job easier, so it doesn't have to edit the symbol table.
Quoted 7 lines
Quoted 2 lines
I could add a comment saying as much, though it's hard to explain it in fewer words than I just did :-)
Does this have to do with adding the references by relative address?
In
record_mcount, I just picked an existing symbol and referenced that.. But perhaps this is a cleaner way.
I think recordmcount is different. It creates references (in __mcount_loc) to functions which are already in the object file, so they already have symbols associated with them.
But in this case, when objtool is creating references, the symbol it needs to reference is outside the .o file, so there's no symbol to associate it with.
Quoted 18 lines
Adding a more in depth comment wont hurt.
Quoted 16 lines
Quoted 15 lines
kernel/static_call.c line 98
Quoted 9 lines
kernel/static_call.c new file +8
-0+/* +* If called before init, leave the call sites unpatched for now. +* In the meantime they'll continue to call the temporary trampoline. +*/ +if (!static_call_initialized) + goto done; + +list_for_each_entry (mod, &key->site_mods, list) {Since I'm expecting a lot of sites, I'm wondering if we should just do this as an array, like I do with the ftrace call sites.
But this can be an enhancement for later. Let's focus on getting this working first.
Quoted 8 lines
Quoted 2 lines
But it's not a static list. It can grow/shrink as modules are loaded/unload.
Neither is ftrace :-) What I did was make one array for the core kernel code, and an array for each module, and link list those (single link, although double link may not be hard either). That will save a lot of memory than having each instance have a link pointer, as it only grows or shrinks in chunks.
That sounds exactly like what I did :-)
The site_mods list is a list of static_call_mods. Each static_call_mod has a pointer to an array (which is that module/vmlinux's .static_call_site section).
Quoted 15 lines
Quoted 6 lines
Quoted 1 line
So what's the reason for skipping init calls?
This is the runtime changing code (
static_call_update). Presumably the init sections no longer exist and we shouldn't write to any (former) call sites there.Quoted 8 lines
That's probably a dangerous assumption though... If
static_call_update()were called early, some init code might not get patched and then call into the wrong function.I'm thinking we should just disallow static call sites in init sections. I can't think of a good reason why they would be needed in init code. We can WARN when detecting them during boot / module init.
Quoted 4 lines
What I would do is to allow init (like ftrace now does). I have
ftrace_free_init_mem()that removes all the mcount references for init calls from its list. You could add astatic_call_free_init()tokernel_init()ininit/main.ctoo.
That makes sense for ftrace, but I don't see much point in allowing it for static calls. Maybe we could just add support for it later if it turns out to be useful.
Signature · 1 line
Josh
Steven Rostedt
reviewerRe: [RFC PATCH 1/3] static_call: Add static call infrastructure
On Fri, 9 Nov 2018 14:34:59 -0600
Josh Poimboeuf <[email protected]> wrote:
I'm slowly massaging this to work with tracepoints.
But I hit a snag on this patch.
Quoted 37 lines · Steven Rostedt
Quoted 28 lines
Quoted 24 lines · On Fri, 9 Nov 2018 13:35:05 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 9 lines
include/linux/static_call.h line 112
Quoted 5 lines
include/linux/static_call.h new file +5
-0+#define DECLARE_STATIC_CALL (key, func) \ + extern struct static_call_key key; \ + extern typeof(func) STATIC_CALL_TRAMP(key); \ +/* Preserve the ELF symbol so objtool can access it: */ \ +__ADDRESSABLE (key)Does the
__ADDRESSABLE(key)need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.For each call site, objtool creates a struct in .static_call_sites:
struct static_call_site { s32 addr; s32 key; };In order to do that, it needs to create a relocation which references the key symbol. If the key is defined in another .o file, then the current .o will not have an ELF symbol associated with the key. The
__ADDRESSABLE(key)thing tells GCC to leave the key symbol in the .o file, even though it's not referenced anywhere. That makes objtool's job easier, so it doesn't have to edit the symbol table.Quoted 7 lines
Quoted 2 lines
I could add a comment saying as much, though it's hard to explain it in fewer words than I just did :-)
Does this have to do with adding the references by relative address?
In
record_mcount, I just picked an existing symbol and referenced that.. But perhaps this is a cleaner way.
Quoted 8 lines
I think recordmcount is different. It creates references (in
__mcount_loc) to functions which are already in the object file, so they already have symbols associated with them.But in this case, when objtool is creating references, the symbol it needs to reference is outside the .o file, so there's no symbol to associate it with.
The __ADDRESSABLE() appears to fail if you have a header with a DECLARE_STATIC_CALL() that is included where the DEFINE_STATIC_CALL() is, because I'm getting this:
In file included from <command-line>:
/work/git/linux-trace.git/include/linux/compiler.h:285:11: error: redefinition of ‘__addressable___tp_func_sys_enter40’
__PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
^~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/compiler_types.h:53:23: note: in definition of macro ‘___PASTE’
#define ___PASTE(a,b) a##b
^/work/git/linux-trace.git/include/linux/compiler.h:285:3: note: in expansion of macro ‘__PASTE’
__PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
^~~~~~~
/work/git/linux-trace.git/include/linux/static_call.h:112:2: note: in expansion of macro ‘__ADDRESSABLE’
__ADDRESSABLE(key)
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/static_call.h:115:2: note: in expansion of macro ‘DECLARE_STATIC_CALL’
DECLARE_STATIC_CALL(key, _func); \
^~~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:310:2: note: in expansion of macro ‘DEFINE_STATIC_CALL’
DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);
^~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/define_trace.h:42:2: note: in expansion of macro ‘DEFINE_TRACE_FN’
DEFINE_TRACE_FN(name, reg, unreg, PARAMS(proto), PARAMS(args))
^~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/events/syscalls.h:18:1: note: in expansion of macro ‘TRACE_EVENT_FN’
TRACE_EVENT_FN(sys_enter,
^~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/compiler.h:285:11: note: previous definition of ‘__addressable___tp_func_sys_enter40’ was here
__PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
^~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/compiler_types.h:53:23: note: in definition of macro ‘___PASTE’
#define ___PASTE(a,b) a##b
^/work/git/linux-trace.git/include/linux/compiler.h:285:3: note: in expansion of macro ‘__PASTE’
__PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
^~~~~~~
/work/git/linux-trace.git/include/linux/static_call.h:112:2: note: in expansion of macro ‘__ADDRESSABLE’
__ADDRESSABLE(key)
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:234:2: note: in expansion of macro ‘DECLARE_STATIC_CALL’
DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \
^~~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:421:2: note: in expansion of macro ‘__DECLARE_TRACE’
__DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \
^~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:560:2: note: in expansion of macro ‘DECLARE_TRACE’
DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/events/syscalls.h:18:1: note: in expansion of macro ‘TRACE_EVENT_FN’
TRACE_EVENT_FN(sys_enter,
The complaint is on:
DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);
And the previous definition is on:
DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \
-- Steve
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 106 lines · Steven Rostedt
On Fri, 9 Nov 2018 14:34:59 -0600
Josh Poimboeuf <[email protected]> wrote:I'm slowly massaging this to work with tracepoints.
But I hit a snag on this patch.
Quoted 37 lines · Steven Rostedt
Quoted 28 lines
Quoted 24 lines · On Fri, 9 Nov 2018 13:35:05 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 9 lines
include/linux/static_call.h line 112
Quoted 5 lines
include/linux/static_call.h new file +5
-0+#define DECLARE_STATIC_CALL (key, func) \ + extern struct static_call_key key; \ + extern typeof(func) STATIC_CALL_TRAMP(key); \ +/* Preserve the ELF symbol so objtool can access it: */ \ +__ADDRESSABLE (key)Does the
__ADDRESSABLE(key)need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.For each call site, objtool creates a struct in .static_call_sites:
struct static_call_site { s32 addr; s32 key; };In order to do that, it needs to create a relocation which references the key symbol. If the key is defined in another .o file, then the current .o will not have an ELF symbol associated with the key. The
__ADDRESSABLE(key)thing tells GCC to leave the key symbol in the .o file, even though it's not referenced anywhere. That makes objtool's job easier, so it doesn't have to edit the symbol table.Quoted 7 lines
Quoted 2 lines
I could add a comment saying as much, though it's hard to explain it in fewer words than I just did :-)
Does this have to do with adding the references by relative address?
In
record_mcount, I just picked an existing symbol and referenced that.. But perhaps this is a cleaner way.Quoted 8 lines
I think recordmcount is different. It creates references (in
__mcount_loc) to functions which are already in the object file, so they already have symbols associated with them.But in this case, when objtool is creating references, the symbol it needs to reference is outside the .o file, so there's no symbol to associate it with.
The
__ADDRESSABLE()appears to fail if you have a header with aDECLARE_STATIC_CALL()that is included where theDEFINE_STATIC_CALL()is, because I'm getting this:In file included from <command-line>:
/work/git/linux-trace.git/include/linux/compiler.h:285:11: error: redefinition of ‘__addressable___tp_func_sys_enter40’__PASTE(__addressable_##sym, __LINE__) = (void *)&sym; ^~~~~~~~~~~~~~ /work/git/linux-trace.git/include/linux/compiler_types.h:53:23: note: in definition of macro ‘___PASTE’ #define ___PASTE(a,b) a##b ^ /work/git/linux-trace.git/include/linux/compiler.h:285:3: note: in expansion of macro ‘__PASTE’ __PASTE(__addressable_##sym, __LINE__) = (void *)&sym; ^~~~~~~/work/git/linux-trace.git/include/linux/static_call.h:112:2: note: in expansion of macro ‘__ADDRESSABLE’
__ADDRESSABLE(key)
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/static_call.h:115:2: note: in expansion of macro ‘DECLARE_STATIC_CALL’
DECLARE_STATIC_CALL(key, _func); \
^~~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:310:2: note: in expansion of macro ‘DEFINE_STATIC_CALL’
DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);
^~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/define_trace.h:42:2: note: in expansion of macro ‘DEFINE_TRACE_FN’
DEFINE_TRACE_FN(name, reg, unreg, PARAMS(proto), PARAMS(args))
^~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/events/syscalls.h:18:1: note: in expansion of macro ‘TRACE_EVENT_FN’
TRACE_EVENT_FN(sys_enter,
^~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/compiler.h:285:11: note: previous definition of ‘__addressable___tp_func_sys_enter40’ was here__PASTE(__addressable_##sym, __LINE__) = (void *)&sym; ^~~~~~~~~~~~~~ /work/git/linux-trace.git/include/linux/compiler_types.h:53:23: note: in definition of macro ‘___PASTE’ #define ___PASTE(a,b) a##b ^ /work/git/linux-trace.git/include/linux/compiler.h:285:3: note: in expansion of macro ‘__PASTE’ __PASTE(__addressable_##sym, __LINE__) = (void *)&sym; ^~~~~~~/work/git/linux-trace.git/include/linux/static_call.h:112:2: note: in expansion of macro ‘__ADDRESSABLE’
__ADDRESSABLE(key)
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:234:2: note: in expansion of macro ‘DECLARE_STATIC_CALL’
DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \
^~~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:421:2: note: in expansion of macro ‘__DECLARE_TRACE’
__DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \
^~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:560:2: note: in expansion of macro ‘DECLARE_TRACE’
DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/events/syscalls.h:18:1: note: in expansion of macro ‘TRACE_EVENT_FN’
TRACE_EVENT_FN(sys_enter,
Quoted 8 lines
The complaint is on:
DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);And the previous definition is on:
DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \
Does the DECLARE really need the __ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.
Steven Rostedt
reviewerRe: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 6 lines · On Sat, 10 Nov 2018 12:58:08 +0100 Ard Biesheuvel <[email protected]>…
Quoted 6 lines
The complaint is on:
DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);And the previous definition is on:
Quoted 7 lines
Quoted 2 lines
DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \Does the DECLARE really need the
__ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.
I applied the following change and it compiled fine:
include/linux/static_call.h +3 -3
@@ -108,8 +108,6 @@ extern void arch_static_call_poison_tramp(unsigned long insn);#define DECLARE_STATIC_CALL (key, func) \ extern struct static_call_key key; \ extern typeof(func) STATIC_CALL_TRAMP(key); \ -/* Preserve the ELF symbol so objtool can access it: */ \ - __ADDRESSABLE(key)#define DEFINE_STATIC_CALL (key, _func) \ DECLARE_STATIC_CALL(key, _func); \
@@ -117,7 +115,9 @@ extern void arch_static_call_poison_tramp(unsigned long insn); .func = _func, \ .site_mods = LIST_HEAD_INIT(key.site_mods), \ }; \ - ARCH_STATIC_CALL_TEMPORARY_TRAMP(key) +ARCH_STATIC_CALL_TEMPORARY_TRAMP (key); \ +/* Preserve the ELF symbol so objtool can access it: */ \ + __ADDRESSABLE(key) #define static_call(key, args...) STATIC_CALL_TRAMP(key)(args)
Thanks!
-- Steve
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Sat, 10 Nov 2018 12:58:08 +0100 Ard Biesheuvel <[email protected]>…
Quoted 6 lines
The complaint is on:
DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);And the previous definition is on:
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \Does the DECLARE really need the
__ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.
Yes, DECLARE needs the __ADDRESSABLE. In the case where DECLARE is used, but DEFINE is not, GCC strips the symbol.
Quoted 16 lines
I applied the following change and it compiled fine:
include/linux/static_call.h +0
-2@@ -108,8 +108,6 @@ extern void arch_static_call_poison_tramp(unsigned long insn);#define DECLARE_STATIC_CALL (key, func) \ extern struct static_call_key key; \ extern typeof(func) STATIC_CALL_TRAMP(key); \ -/* Preserve the ELF symbol so objtool can access it: */ \ - __ADDRESSABLE(key)#define DEFINE_STATIC_CALL (key, _func) \ DECLARE_STATIC_CALL(key, _func); \
include/linux/static_call.h extern void arch_static_call_poison_tramp(unsigned long insn); · line 122
Quoted 9 lines
include/linux/static_call.h +3
-1@@ -117,7 +115,9 @@ extern void arch_static_call_poison_tramp(unsigned long insn); .func = _func, \ .site_mods = LIST_HEAD_INIT(key.site_mods), \ }; \ - ARCH_STATIC_CALL_TEMPORARY_TRAMP(key) +ARCH_STATIC_CALL_TEMPORARY_TRAMP (key); \ +/* Preserve the ELF symbol so objtool can access it: */ \ + __ADDRESSABLE(key) #define static_call(key, args...) STATIC_CALL_TRAMP(key)(args)
Adding __ADDRESSABLE to the DEFINE macro doesn't do any good. By definition, the key is defined in the .o file, so the symbol already exists.
The issue you're seeing is really an issue with the __ADDRESSABLE macro not creating unique symbol names. This should fix it:
include/linux/compiler.h +1 -1
@@ -282,7 +282,7 @@ unsigned long read_word_at_a_time(const void *addr)*/ #define __ADDRESSABLE (sym) \ static void * __section(".discard.addressable") __used \ - __PASTE(__addressable_##sym, __LINE__) = (void *)&sym; + __UNIQUE_ID(__addressable_##sym) = (void *)&sym;/** * offset_to_ptr - convert a relative memory offset to an absolute pointer
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 14 lines · Josh Poimboeuf
Quoted 13 lines · Steven Rostedt
Quoted 9 lines · On Sat, 10 Nov 2018 12:58:08 +0100 Ard Biesheuvel <[email protected]>…
Quoted 8 lines
The complaint is on:
DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);And the previous definition is on:
DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Does the DECLARE really need the
__ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.Yes, DECLARE needs the
__ADDRESSABLE. In the case where DECLARE is used, but DEFINE is not, GCC strips the symbol.
I assume DECLARE() is intended for use in header files, and DEFINE() for source files, no?
Doesn't that mean that whatever symbol __ADDRESSABLE() refers to should have external linkage, in which case it it addressable anyway? Or are we talking about some LTO / --gc-sections use case here?
Quoted 38 lines
Quoted 16 lines
I applied the following change and it compiled fine:
include/linux/static_call.h +0
-2@@ -108,8 +108,6 @@ extern void arch_static_call_poison_tramp(unsigned long insn);#define DECLARE_STATIC_CALL (key, func) \ extern struct static_call_key key; \ extern typeof(func) STATIC_CALL_TRAMP(key); \ -/* Preserve the ELF symbol so objtool can access it: */ \ - __ADDRESSABLE(key)#define DEFINE_STATIC_CALL (key, _func) \ DECLARE_STATIC_CALL(key, _func); \include/linux/static_call.h extern void arch_static_call_poison_tramp(unsigned long insn); · line 122
Quoted 9 lines
include/linux/static_call.h +3
-1@@ -117,7 +115,9 @@ extern void arch_static_call_poison_tramp(unsigned long insn); .func = _func, \ .site_mods = LIST_HEAD_INIT(key.site_mods), \ }; \ - ARCH_STATIC_CALL_TEMPORARY_TRAMP(key) +ARCH_STATIC_CALL_TEMPORARY_TRAMP (key); \ +/* Preserve the ELF symbol so objtool can access it: */ \ + __ADDRESSABLE(key) #define static_call(key, args...) STATIC_CALL_TRAMP(key)(args)Adding
__ADDRESSABLEto the DEFINE macro doesn't do any good. By definition, the key is defined in the .o file, so the symbol already exists.The issue you're seeing is really an issue with the
__ADDRESSABLEmacro not creating unique symbol names. This should fix it:include/linux/compiler.h +0
-0
include/linux/compiler.h unsigned long read_word_at_a_time(const void *addr) · line 288
Quoted 8 lines
include/linux/compiler.h +1
-1@@ -282,7 +282,7 @@ unsigned long read_word_at_a_time(const void *addr)*/ #define __ADDRESSABLE (sym) \ static void * __section(".discard.addressable") __used \ - __PASTE(__addressable_##sym, __LINE__) = (void *)&sym; + __UNIQUE_ID(__addressable_##sym) = (void *)&sym;/** * offset_to_ptr - convert a relative memory offset to an absolute pointer
Not sure if it matters, but we'll end up with a lot more stuff in .discard.addressable this way.
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 20 lines · Ard Biesheuvel
Quoted 19 lines · Josh Poimboeuf
Quoted 17 lines · Steven Rostedt
Quoted 13 lines · On Sat, 10 Nov 2018 12:58:08 +0100 Ard Biesheuvel <[email protected]>…
Quoted 8 lines
The complaint is on:
DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);And the previous definition is on:
DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \Does the DECLARE really need the
__ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.
Quoted 6 lines
Quoted 3 lines
Yes, DECLARE needs the
__ADDRESSABLE. In the case where DECLARE is used, but DEFINE is not, GCC strips the symbol.I assume
DECLARE()is intended for use in header files, andDEFINE()for source files, no?
Right.
Quoted 3 lines
Doesn't that mean that whatever symbol
__ADDRESSABLE()refers to should have external linkage, in which case it it addressable anyway? Or are we talking about some LTO / --gc-sections use case here?
If the key is declared, but not used, GCC doesn't put the key's ELF symbol in the binary's symbol table. That makes objtool's life harder, because if the file has a call site, then objtool has to add the key symbol to the symbol table, so that it can create a relocation (in the call site table) which references the symbol.
Signature · 1 line
Josh
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
Quoted 34 lines · Josh Poimboeuf
Quoted 20 lines · Ard Biesheuvel
Quoted 19 lines · Josh Poimboeuf
Quoted 17 lines · Steven Rostedt
Quoted 13 lines · On Sat, 10 Nov 2018 12:58:08 +0100 Ard Biesheuvel <[email protected]>…
Quoted 8 lines
The complaint is on:
DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);And the previous definition is on:
DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \Does the DECLARE really need the
__ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.Quoted 6 lines
Quoted 3 lines
Yes, DECLARE needs the
__ADDRESSABLE. In the case where DECLARE is used, but DEFINE is not, GCC strips the symbol.I assume
DECLARE()is intended for use in header files, andDEFINE()for source files, no?Right.
Quoted 3 lines
Doesn't that mean that whatever symbol
__ADDRESSABLE()refers to should have external linkage, in which case it it addressable anyway? Or are we talking about some LTO / --gc-sections use case here?
Quoted 6 lines
If the key is declared, but not used, GCC doesn't put the key's ELF symbol in the binary's symbol table. That makes objtool's life harder, because if the file has a call site, then objtool has to add the key symbol to the symbol table, so that it can create a relocation (in the call site table) which references the symbol.
Ah, right. So this is specific to objtool rather than a generic issue.
Should we perhaps wrap __ADDRESSABLE() into something else that resolves to a NOP unless on X86, and give it a clear name?
Re: [RFC PATCH 1/3] static_call: Add static call infrastructure
In reply to Josh PoimboeufQuoted 4 lines · Josh Poimboeuf
Quoted 3 lines · Steven Rostedt
On Fri, 9 Nov 2018 13:35:05 -0600
Josh Poimboeuf <[email protected]> wrote:
..
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 1 line
So what's the reason for skipping init calls?
This is the runtime changing code (
static_call_update). Presumably the init sections no longer exist and we shouldn't write to any (former) call sites there.That's probably a dangerous assumption though... If
static_call_update()were called early, some init code might not get patched and then call into the wrong function.I'm thinking we should just disallow static call sites in init sections. I can't think of a good reason why they would be needed in init code. We can WARN when detecting them during boot / module init.
Quoted 9 lines
Quoted 4 lines
What I would do is to allow init (like ftrace now does). I have
ftrace_free_init_mem()that removes all the mcount references for init calls from its list. You could add astatic_call_free_init()tokernel_init()ininit/main.ctoo.That makes sense for ftrace, but I don't see much point in allowing it for static calls. Maybe we could just add support for it later if it turns out to be useful.
I don't see how you can prevent that. Some arch may use a static call in its version of some library code which could be used anywhere, including in .init code.
RFC PATCH 2/3 x86/static_call: Add x86 unoptimized static call implementation
In reply to Josh PoimboeufAdd the x86 unoptimized static call implementation. For each key, it creates a permanent trampoline which is the destination for all static calls for the given key. The trampoline has a direct jump which gets patched by static_call_update() when the destination changes.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/static_call.h | 18 +++++++++++ arch/x86/kernel/Makefile | 1 + arch/x86/kernel/static_call.c | 51 ++++++++++++++++++++++++++++++ 4 files changed, 71 insertions(+) create mode 100644 arch/x86/include/asm/static_call.h create mode 100644 arch/x86/kernel/static_call.c
arch/x86/Kconfig +1 -0
@@ -189,6 +189,7 @@ config X86 select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR select HAVE_STACK_VALIDATION if X86_64 + select HAVE_STATIC_CALL_UNOPTIMIZED select HAVE_RSEQ select HAVE_SYSCALL_TRACEPOINTS select HAVE_UNSTABLE_SCHED_CLOCK
arch/x86/include/asm/static_call.h new file +18 -0
@@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_STATIC_CALL_H +#define _ASM_STATIC_CALL_H + +/* +* This is a permanent trampoline which is the destination for all static calls +* for the given key. The direct jump gets patched by static_call_update(). +*/ +#define ARCH_STATIC_CALL_TRAMP (key, func) \ + asm(".pushsection .text, \"ax\" \n" \ + ".align 4 \n" \ + ".globl " STATIC_CALL_TRAMP_STR(key) " \n" \ + ".type " STATIC_CALL_TRAMP_STR(key) ", @function \n" \ + STATIC_CALL_TRAMP_STR(key) ": \n" \ + "jmp " #func " \n" \ + ".popsection \n") + +#endif /* _ASM_STATIC_CALL_H */
arch/x86/kernel/Makefile +1 -0
@@ -62,6 +62,7 @@ obj-y += tsc.o tsc_msr.o io_delay.o rtc.o obj-y += pci-iommu_table.o obj-y += resource.o obj-y += irqflags.o +obj-y += static_call.o obj-y += process.o obj-y += fpu/
arch/x86/kernel/static_call.c new file +51 -0
51 lines not on this page · raw message
Signature · 1 line
2.17.2
RFC PATCH 3/3 x86/static_call: Add optimized static call implementation for 64-bit
In reply to Josh PoimboeufAdd the optimized static call implementation for x86-64. For each key, a temporary trampoline is created, named __static_call_tramp_<key>. The trampoline has an indirect jump to the destination function.
Objtool uses the trampoline naming convention to detect all the call sites. It then annotates those call sites in the .static_call_sites section.
During boot (and module init), the call sites are patched to call directly into the destination function. The temporary trampoline is then no longer used.
--- arch/x86/Kconfig | 5 +- arch/x86/include/asm/static_call.h | 24 ++++ arch/x86/kernel/static_call.c | 35 ++++- tools/objtool/Makefile | 3 +- tools/objtool/check.c | 126 +++++++++++++++++- tools/objtool/check.h | 2 + tools/objtool/elf.h | 1 + .../objtool/include/linux/static_call_types.h | 19 +++ tools/objtool/sync-check.sh | 1 + 9 files changed, 211 insertions(+), 5 deletions(-) create mode 100644 tools/objtool/include/linux/static_call_types.h
arch/x86/Kconfig +3 -2
@@ -189,7 +189,8 @@ config X86 select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_STACKPROTECTOR if CC_HAS_SANE_STACKPROTECTOR select HAVE_STACK_VALIDATION if X86_64 - select HAVE_STATIC_CALL_UNOPTIMIZED + select HAVE_STATIC_CALL_OPTIMIZED if HAVE_STACK_VALIDATION + select HAVE_STATIC_CALL_UNOPTIMIZED if !HAVE_STACK_VALIDATION select HAVE_RSEQ select HAVE_SYSCALL_TRACEPOINTS select HAVE_UNSTABLE_SCHED_CLOCK
@@ -203,6 +204,7 @@ config X86 select RTC_MC146818_LIB select SPARSE_IRQ select SRCU + select STACK_VALIDATION if HAVE_STACK_VALIDATION && (HAVE_STATIC_CALL_OPTIMIZED || RETPOLINE) select SYSCTL_EXCEPTION_TRACE select THREAD_INFO_IN_TASK select USER_STACKTRACE_SUPPORT
@@ -438,7 +440,6 @@ config GOLDFISH config RETPOLINE bool "Avoid speculative indirect branches in kernel" default y - select STACK_VALIDATION if HAVE_STACK_VALIDATION help Compile kernel with the retpoline compiler options to guard against kernel-to-user data leaks by avoiding speculative indirect
arch/x86/include/asm/static_call.h +24 -0
@@ -2,6 +2,27 @@#ifndef _ASM_STATIC_CALL_H#define _ASM_STATIC_CALL_H +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED +/* +* This is a temporary trampoline which is only used during boot (before the +* call sites have been patched). It uses the current value of the key->func +* pointer to do an indirect jump to the function. +* +* The name of this function has a magical aspect. Objtool uses it to find +* static call sites so that it can create the .static_call_sites section. +*/ +#define ARCH_STATIC_CALL_TEMPORARY_TRAMP (key) \ + asm(".pushsection .text, \"ax\" \n" \ + ".align 4 \n" \ + ".globl " STATIC_CALL_TRAMP_STR(key) " \n" \ + ".type " STATIC_CALL_TRAMP_STR(key) ", @function \n" \ + STATIC_CALL_TRAMP_STR(key) ": \n" \ + ANNOTATE_RETPOLINE_SAFE " \n" \ + "jmpq *" __stringify(key) "(%rip) \n" \ + ".popsection \n") + +#else /* !CONFIG_HAVE_STATIC_CALL_OPTIMIZED */ +/* * This is a permanent trampoline which is the destination for all static calls * for the given key. The direct jump gets patched by static_call_update().
@@ -15,4 +36,7 @@ "jmp " #func " \n" \ ".popsection \n") + +#endif /* !CONFIG_HAVE_STATIC_CALL_OPTIMIZED */ +#endif /* _ASM_STATIC_CALL_H */
arch/x86/kernel/static_call.c +34 -1
@@ -10,6 +10,22 @@void static_call_bp_handler (void );void *bp_handler_dest; +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED + +void *bp_handler_continue; + +asm(".pushsection .text, \"ax\" \n" +".globl static_call_bp_handler \n" +".type static_call_bp_handler, @function \n" +"static_call_bp_handler: \n" +"ANNOTATE_RETPOLINE_SAFE \n" +"call *bp_handler_dest \n" +"ANNOTATE_RETPOLINE_SAFE \n" +"jmp *bp_handler_continue \n" +".popsection \n" ); + +#else /* !CONFIG_HAVE_STATIC_CALL_OPTIMIZED */ + asm(".pushsection .text, \"ax\" \n" ".globl static_call_bp_handler \n" ".type static_call_bp_handler, @function \n"
@@ -18,6 +34,8 @@ asm(".pushsection .text, \"ax\" \n""jmp *bp_handler_dest \n" ".popsection \n"); +#endif /* !CONFIG_HAVE_STATIC_CALL_OPTIMIZED */ +void arch_static_call_transform (unsigned long insn,void *dest) {s32 dest_relative;
@@ -38,8 +56,11 @@ void arch_static_call_transform(unsigned long insn, void *dest) opcodes[0 ] = insn_opcode;memcpy (&opcodes[1 ], &dest_relative,CALL_INSN_SIZE -1 ); -/* Set up the variable for the breakpoint handler: */ +/* Set up the variables for the breakpoint handler: */ bp_handler_dest = dest; +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED + bp_handler_continue = (void *)(insn +CALL_INSN_SIZE ); +#endif /* Patch the call site: */ text_poke_bp((void *)insn, opcodes, CALL_INSN_SIZE,
@@ -49,3 +70,15 @@ void arch_static_call_transform(unsigned long insn, void *dest)mutex_unlock (&text_mutex); }EXPORT_SYMBOL_GPL (arch_static_call_transform); + +#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED +void arch_static_call_poison_tramp (unsigned long insn) +{ +unsigned long tramp = insn +CALL_INSN_SIZE + *(s32 *)(insn +1 ); +unsigned short opcode =INSN_UD2 ; + +mutex_lock (&text_mutex); +text_poke ((void *)tramp, &opcode,2 ); +mutex_unlock (&text_mutex); +} +#endif
tools/objtool/Makefile +2 -1
@@ -29,7 +29,8 @@ all: $(OBJTOOL) INCLUDES := -I$(srctree)/tools/include \ -I$(srctree)/tools/arch/$(HOSTARCH)/include/uapi \ - -I$(srctree)/tools/objtool/arch/$(ARCH)/include + -I$(srctree)/tools/objtool/arch/$(ARCH)/include \ + -I$(srctree)/tools/objtool/include WARNINGS := $(EXTRA_WARNINGS) -Wno-switch-default -Wno-switch-enum -Wno-packed CFLAGS += -Werror $(WARNINGS) $(KBUILD_HOSTCFLAGS) -g $(INCLUDES) LDFLAGS += -lelf $(LIBSUBCMD) $(KBUILD_HOSTLDFLAGS)
tools/objtool/check.c +125 -1
192 lines not on this page · raw message
tools/objtool/check.h +2 -0
@@ -39,6 +39,7 @@ struct insn_state {struct instruction {struct list_head list;struct hlist_node hash; +struct list_head static_call_node;struct section *sec;unsigned long offset;unsigned int len;
@@ -60,6 +61,7 @@ struct objtool_file {struct elf *elf;struct list_head insn_list;DECLARE_HASHTABLE (insn_hash,16 ); +struct list_head static_call_list;struct section *whitelist;bool ignore_unreachables, c_file, hints, rodata; };
tools/objtool/elf.h +1 -0
@@ -62,6 +62,7 @@ struct symbol {unsigned long offset;unsigned int len;struct symbol *pfunc, *cfunc; +bool static_call_tramp; };struct rela {
tools/objtool/include/linux/static_call_types.h new file +19 -0
@@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _STATIC_CALL_TYPES_H +#define _STATIC_CALL_TYPES_H + +#include <linux/stringify.h> + +#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_ +#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX) + +#define STATIC_CALL_TRAMP (key) STATIC_CALL_TRAMP_PREFIX##key +#define STATIC_CALL_TRAMP_STR (key) __stringify(STATIC_CALL_TRAMP(key)) + +/* The static call site table is created by objtool. */ +struct static_call_site { +s32 addr; +s32 key; +}; + +#endif /* _STATIC_CALL_TYPES_H */
tools/objtool/sync-check.sh +1 -0
@@ -10,6 +10,7 @@ arch/x86/include/asm/insn.h arch/x86/include/asm/inat.h arch/x86/include/asm/inat_types.h arch/x86/include/asm/orc_types.h +include/linux/static_call_types.h ' check()
Signature · 1 line
2.17.2
Quoted 2 lines · Josh Poimboeuf
- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?
I forgot Andy mentioned that we might be able to use this to clean up paravirt patching, in which case it would have a lot of value, retpolines or not...
Signature · 1 line
Josh
Quoted 17 lines · Josh Poimboeuf
These patches are related to two similar patch sets from Ard and Steve:
- https://lkml.kernel.org/r/[email protected]
- https://lkml.kernel.org/r/[email protected]The code is also heavily inspired by the jump label code, as some of the concepts are very similar.
There are three separate implementations, depending on what the arch supports:
1)
CONFIG_HAVE_STATIC_CALL_OPTIMIZED: patched call sites - requires objtool and a small amount of arch code2)
CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED: patched trampolines - requires a small amount of arch code
Quoted 8 lines
3) If no arch support, fall back to regular function pointers
TODO:
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
Quoted 2 lines
- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?
Paravirt patching, as you mention in your later reply?
Quoted 1 line
- Find some actual users of the interfaces (tracepoints? crypto?)
I'd be very happy with a demonstrated paravirt optimization already - i.e. seeing the before/after effect on the vmlinux with an x86 distro config.
All major Linux distributions enable CONFIG_PARAVIRT=y and CONFIG_PARAVIRT_XXL=y on x86 at the moment, so optimizing it away as much as possible in the 99.999% cases where it's not used is a primary concern.
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
Thanks,
Ingo
Quoted 4 lines · Ingo Molnar
Quoted 2 lines
- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?
Paravirt patching, as you mention in your later reply?
BTW., to look for candidates of this API, I'd suggest looking at the function call frequency of my (almost-)distro kernel vmlinux:
$ objdump -d vmlinux | grep -w callq | cut -f3- | sort | uniq -c | sort -n | tail -100
which gives:
502 callq ffffffff8157d050 <nla_put>
522 callq ffffffff81aaf420 <down_write>
536 callq ffffffff81547e60 <_copy_to_user>
615 callq ffffffff81a97700 <snprintf>
624 callq *0xffffffff82648428
624 callq ffffffff810cc810 <__might_sleep>
625 callq ffffffff81a93b90 <strcmp>
649 callq ffffffff81547dd0 <_copy_from_user>
651 callq ffffffff811ba930 <trace_seq_printf>
654 callq ffffffff8170b6f0 <_dev_warn>
691 callq ffffffff81a93790 <strlen>
693 callq ffffffff81a88dc0 <cpumask_next>
709 callq *0xffffffff82648438
723 callq ffffffff811bdbd0 <trace_hardirqs_on>
735 callq ffffffff810feac0 <up_write>
750 callq ffffffff8163e9f0 <acpi_ut_status_exit>
768 callq *0xffffffff82648430
814 callq ffffffff81ab2710 <_raw_spin_lock_irq>
841 callq ffffffff81a9e680 <__memcpy>
863 callq ffffffff812ae3d0 <__kmalloc>
899 callq ffffffff8126ac80 <__might_fault>
912 callq ffffffff81ab2970 <_raw_spin_unlock_irq>
939 callq ffffffff81aaaf10 <_cond_resched>
966 callq ffffffff811bda00 <trace_hardirqs_off>
1069 callq ffffffff81126f50 <rcu_read_lock_sched_held>
1078 callq ffffffff81097760 <__warn_printk>
1081 callq ffffffff8157b140 <__dynamic_dev_dbg>
1351 callq ffffffff8170b630 <_dev_err>
1365 callq ffffffff811050c0 <lock_is_held_type>
1373 callq ffffffff81a977f0 <sprintf>
1390 callq ffffffff8157b090 <__dynamic_pr_debug>
1453 callq ffffffff8155c650 <__list_add_valid>
1501 callq ffffffff812ad6f0 <kmem_cache_alloc_trace>
1509 callq ffffffff8155c6c0 <__list_del_entry_valid>
1513 callq ffffffff81310ce0 <seq_printf>
1571 callq ffffffff81ab2780 <_raw_spin_lock_irqsave>
1624 callq ffffffff81ab29b0 <_raw_spin_unlock_irqrestore>
1661 callq ffffffff81126fd0 <rcu_read_lock_held>
1986 callq ffffffff81104940 <lock_acquire>
2050 callq ffffffff811c5110 <trace_define_field>
2133 callq ffffffff81102c70 <lock_release>
2507 callq ffffffff81ab2560 <_raw_spin_lock>
2676 callq ffffffff81aadc40 <mutex_lock_nested>
3056 callq ffffffff81ab2900 <_raw_spin_unlock>
3294 callq ffffffff81aac610 <mutex_unlock>
3628 callq ffffffff81129100 <rcu_is_watching>
4462 callq ffffffff812ac2c0 <kfree>
6454 callq ffffffff8111a51e <printk>
6676 callq ffffffff81101420 <lockdep_rcu_suspicious>
7328 callq ffffffff81e014b0 <__x86_indirect_thunk_rax>
7598 callq ffffffff81126f30 <debug_lockdep_rcu_enabled>
9065 callq ffffffff810979f0 <__stack_chk_fail>The most prominent callers which are already function call pointers today are:
$ objdump -d vmlinux | grep -w callq | grep \* | cut -f3- | sort | uniq -c | sort -n | tail -10
109 callq *0xffffffff82648530
134 callq *0xffffffff82648568
154 callq *0xffffffff826483d0
260 callq *0xffffffff826483d8
297 callq *0xffffffff826483e0
345 callq *0xffffffff82648440
345 callq *0xffffffff82648558
624 callq *0xffffffff82648428
709 callq *0xffffffff82648438
768 callq *0xffffffff82648430That's all pv_ops->*() method calls:
ffffffff82648300 D pv_ops
ffffffff826485d0 D pv_info
Optimizing those thousands of function pointer calls would already be a nice improvement.
But retpolines:
7328 callq ffffffff81e014b0 <__x86_indirect_thunk_rax>
ffffffff81e014b0 <__x86_indirect_thunk_rax>:
ffffffff81e014b0: ff e0 jmpq *%rax
... are even more prominent, and turned on in every distro as well, obviously.
Thanks,
Ingo
Quoted 25 lines · Ingo Molnar
Quoted 22 lines · Josh Poimboeuf
These patches are related to two similar patch sets from Ard and Steve:
- https://lkml.kernel.org/r/[email protected]
- https://lkml.kernel.org/r/[email protected]The code is also heavily inspired by the jump label code, as some of the concepts are very similar.
There are three separate implementations, depending on what the arch supports:
1)
CONFIG_HAVE_STATIC_CALL_OPTIMIZED: patched call sites - requires objtool and a small amount of arch code2)
CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED: patched trampolines - requires a small amount of arch code3) If no arch support, fall back to regular function pointers
TODO:
Quoted 8 lines
Quoted 3 lines
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
I implemented the GCC plugin approach here for arm64
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=static-calls
That implements both the unoptimized and the optimized versions.
I do take your point about GCC and other compilers, but on arm64 we don't have a lot of choice.
As far as I can tell, the GCC plugin is generic (i.e., it does not rely on any ARM specific passes, but obviously, this requires a *lot* of testing and validation to be taken seriously.
Quoted 25 lines
Quoted 2 lines
- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?
Paravirt patching, as you mention in your later reply?
Quoted 1 line
- Find some actual users of the interfaces (tracepoints? crypto?)
I'd be very happy with a demonstrated paravirt optimization already - i.e. seeing the before/after effect on the vmlinux with an x86 distro config.
All major Linux distributions enable
CONFIG_PARAVIRT=yandCONFIG_PARAVIRT_XXL=yon x86 at the moment, so optimizing it away as much as possible in the 99.999% cases where it's not used is a primary concern.All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
Thanks,
Ingo
Quoted 35 lines · Ard Biesheuvel
Quoted 25 lines · Ingo Molnar
Quoted 22 lines · Josh Poimboeuf
These patches are related to two similar patch sets from Ard and Steve:
- https://lkml.kernel.org/r/[email protected]
- https://lkml.kernel.org/r/[email protected]The code is also heavily inspired by the jump label code, as some of the concepts are very similar.
There are three separate implementations, depending on what the arch supports:
1)
CONFIG_HAVE_STATIC_CALL_OPTIMIZED: patched call sites - requires objtool and a small amount of arch code2)
CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED: patched trampolines - requires a small amount of arch code3) If no arch support, fall back to regular function pointers
TODO:
Quoted 8 lines
Quoted 3 lines
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
Quoted 5 lines
I implemented the GCC plugin approach here for arm64
https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=static-calls
That implements both the unoptimized and the optimized versions.
Nice! That was fast :-)
Quoted 6 lines
I do take your point about GCC and other compilers, but on arm64 we don't have a lot of choice.
As far as I can tell, the GCC plugin is generic (i.e., it does not rely on any ARM specific passes, but obviously, this requires a *lot* of testing and validation to be taken seriously.
Yeah. I haven't had a chance to try your plugin on x86 yet, but in theory it should be arch-independent.
Signature · 1 line
Josh
Peter Zijlstra
maintainerQuoted 11 lines · Ard Biesheuvel
Quoted 8 lines · Ingo Molnar
Quoted 3 lines
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
I implemented the GCC plugin approach here for arm64
I'm confused; I though we only needed objtool for variable instruction length architectures, because we can't reliably decode our instruction stream. Otherwise we can fairly trivially use the DWARF relocation data, no?
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Ard Biesheuvel
Quoted 8 lines · Ingo Molnar
Quoted 3 lines
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
Quoted 6 lines
Quoted 1 line
I implemented the GCC plugin approach here for arm64
I'm confused; I though we only needed objtool for variable instruction length architectures, because we can't reliably decode our instruction stream. Otherwise we can fairly trivially use the DWARF relocation data, no?
How would that work? We could build vmlinux with --emit-relocs, filter out the static jump/call relocations and resolve the symbol names to filter the ones associated with calls to trampolines. But then, we have to build the static_call_sites section and reinject it back into the image in some way, which is essentially objtool, no?
Peter Zijlstra
maintainerQuoted 19 lines · Ard Biesheuvel
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Ard Biesheuvel
Quoted 8 lines · Ingo Molnar
Quoted 3 lines
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
Quoted 6 lines
Quoted 1 line
I implemented the GCC plugin approach here for arm64
I'm confused; I though we only needed objtool for variable instruction length architectures, because we can't reliably decode our instruction stream. Otherwise we can fairly trivially use the DWARF relocation data, no?
Quoted 5 lines
How would that work? We could build vmlinux with --emit-relocs, filter out the static jump/call relocations and resolve the symbol names to filter the ones associated with calls to trampolines. But then, we have to build the
static_call_sitessection and reinject it back into the image in some way, which is essentially objtool, no?
It's a _much_ simpler tool than objtool, but yes, we need a tool that reads the relocation stuff and (re)injects it in a new section -- we don't need it on a vmlinux level, it can be done per TU.
Anyway, a GCC plugin (I still have to have a peek at your thing) sounds like it should work just fine too.
Quoted 7 lines · Ingo Molnar
Quoted 3 lines
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.
The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).
Quoted 7 lines
Quoted 2 lines
- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?
Paravirt patching, as you mention in your later reply?
Quoted 1 line
- Find some actual users of the interfaces (tracepoints? crypto?)
Quoted 8 lines
I'd be very happy with a demonstrated paravirt optimization already - i.e. seeing the before/after effect on the vmlinux with an x86 distro config.
All major Linux distributions enable
CONFIG_PARAVIRT=yandCONFIG_PARAVIRT_XXL=yon x86 at the moment, so optimizing it away as much as possible in the 99.999% cases where it's not used is a primary concern.
For paravirt, I was thinking of it as more of a cleanup than an optimization. The paravirt patching code already replaces indirect branches with direct ones -- see paravirt_patch_default().
Though it *would* reduce the instruction footprint a bit, as the 7-byte indirect calls (later patched to 5-byte direct + 2-byte nop) would instead be 5-byte direct calls to begin with.
Quoted 5 lines
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
With retpolines, and with tracepoints enabled, it's definitely a major win. Steve measured an 8.9% general slowdown on hackbench caused by retpolines.
But with tracepoints disabled, I believe static jumps are used, which already minimizes the impact on hot paths.
Signature · 1 line
Josh
Quoted 9 lines · Josh Poimboeuf
Quoted 7 lines · Ingo Molnar
Quoted 3 lines
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
Quoted 6 lines
The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.
The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).
Which architectures would benefit from ORC support the most?
I really think that hard reliance on GCC plugins is foolish - but maybe Clang's plugin infrastructure is a guarantee that it remains a sane and usable interface.
Quoted 9 lines
Quoted 8 lines
I'd be very happy with a demonstrated paravirt optimization already - i.e. seeing the before/after effect on the vmlinux with an x86 distro config.
All major Linux distributions enable
CONFIG_PARAVIRT=yandCONFIG_PARAVIRT_XXL=yon x86 at the moment, so optimizing it away as much as possible in the 99.999% cases where it's not used is a primary concern.
Quoted 7 lines
For paravirt, I was thinking of it as more of a cleanup than an optimization. The paravirt patching code already replaces indirect branches with direct ones -- see
paravirt_patch_default().Though it *would* reduce the instruction footprint a bit, as the 7-byte indirect calls (later patched to 5-byte direct + 2-byte nop) would instead be 5-byte direct calls to begin with.
Yes.
Quoted 9 lines
Quoted 5 lines
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
With retpolines, and with tracepoints enabled, it's definitely a major win. Steve measured an 8.9% general slowdown on hackbench caused by retpolines.
How much of that slowdown is reversed?
Quoted 2 lines
But with tracepoints disabled, I believe static jumps are used, which already minimizes the impact on hot paths.
Yeah.
Thanks,
Ing
Quoted 12 lines · Ingo Molnar
Quoted 9 lines · Josh Poimboeuf
Quoted 7 lines · Ingo Molnar
Quoted 3 lines
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
Quoted 8 lines
Quoted 6 lines
The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.
The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).
Which architectures would benefit from ORC support the most?
According to my (limited and potentially flawed) knowledge, I think arm64 would benefit the most performance-wise, whereas powerpc and s390 gains would be quite a bit less.
We may have to port objtool to arm64 anyway, for live patching. But that will be a lot more work than it took for Ard to write a GCC plugin.
Quoted 1 line
I really think that hard reliance on GCC plugins is foolish
Funny, I feel the same way about hard reliance on objtool :-)
Quoted 2 lines
- but maybe Clang's plugin infrastructure is a guarantee that it remains a sane and usable interface.
Hopefully so. If it breaks, we could always write another tool, as the work is straightforward. Or we could make it an objtool subcommand which works on all arches.
Quoted 11 lines
Quoted 9 lines
Quoted 5 lines
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
With retpolines, and with tracepoints enabled, it's definitely a major win. Steve measured an 8.9% general slowdown on hackbench caused by retpolines.
How much of that slowdown is reversed?
In theory, it should reverse all of the slowdown, and actually may even speed it up a little. Steve is working on measuring that now.
Signature · 1 line
Josh
Quoted 10 lines · Josh Poimboeuf
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Josh Poimboeuf
Quoted 4 lines · Ingo Molnar
Quoted 3 lines
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.
Having looked into this, I don't think they will share any code at all, to be honest. Perhaps some macros and string templates, that's all.
Quoted 10 lines
Quoted 5 lines
Quoted 3 lines
The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).
Which architectures would benefit from ORC support the most?
According to my (limited and potentially flawed) knowledge, I think arm64 would benefit the most performance-wise, whereas powerpc and s390 gains would be quite a bit less.
What would arm64 gain from ORC and/or objtool?
Quoted 1 line
We may have to port objtool to arm64 anyway, for live patching.
Is this about the reliable stack traces, i.e., the ability to detect non-leaf functions that don't create stack frames? I think we should be able to manage this without objtool on arm64 tbh.
Quoted 7 lines
But
that will be a lot more work than it took for Ard to write a GCC plugin.Quoted 1 line
I really think that hard reliance on GCC plugins is foolish
Funny, I feel the same way about hard reliance on objtool :-)
I tend to agree here. I think objtool is a necessary evil (as are compiler plugins, for that matter) which I hope does not spread to other architectures.
But the main difference is that the GCC plugin is only ~50 lines (for this particular use case, and minus another 50 lines of boilerplate), whereas objtool (AIUI) duplicates lots and lots of functionality of the compiler, assembler and/or linker, to mangle relocations, create new sections etc etc. Porting this to other architectures is going to be a major maintenance effort, especially when I think of, e.g., 32-bit ARM with its Thumb2 quirks and other idiosyncrasies that are currently hidden in the toolchain. Other architectures should be first class citizens if objtool gains support for them, which means that the x86 people that own it currently are on the hook for testing their changes against architectures they are not familiar with.
This obviously applies equally to compiler plugins, but those have a lot more focus.
Quoted 24 lines
Quoted 2 lines
- but maybe Clang's plugin infrastructure is a guarantee that it remains a sane and usable interface.
Hopefully so. If it breaks, we could always write another tool, as the work is straightforward. Or we could make it an objtool subcommand which works on all arches.
Quoted 11 lines
Quoted 9 lines
Quoted 5 lines
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
With retpolines, and with tracepoints enabled, it's definitely a major win. Steve measured an 8.9% general slowdown on hackbench caused by retpolines.
How much of that slowdown is reversed?
In theory, it should reverse all of the slowdown, and actually may even speed it up a little. Steve is working on measuring that now.
--
Josh
Quoted 15 lines · Ard Biesheuvel
Quoted 14 lines · Josh Poimboeuf
Quoted 12 lines · Ingo Molnar
Quoted 9 lines · Josh Poimboeuf
Quoted 7 lines · Ingo Molnar
Quoted 3 lines
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.
Having looked into this, I don't think they will share any code at all, to be honest. Perhaps some macros and string templates, that's all.
Oh well. That should still be easier to maintain than objtool across all arches at this point.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Which architectures would benefit from ORC support the most?
According to my (limited and potentially flawed) knowledge, I think arm64 would benefit the most performance-wise, whereas powerpc and s390 gains would be quite a bit less.
What would arm64 gain from ORC and/or objtool?
Other than live patching, the biggest benefit would be an across-the-board performance improvement from disabling frame pointers. It would be interesting to see some arm64 performance numbers there, for a kernel compiled with -fomit-frame-pointer.
For more details (and benefits of) ORC seeDocumentation/x86/orc-unwinder.txt.
Objtool has also come in handy for other cases, like ensuring retpolines are used everywhere.
Over time, I would like to move some objtool functionality to compiler plugins, such that it would be easier to port it to other arches.
Quoted 5 lines
Quoted 1 line
We may have to port objtool to arm64 anyway, for live patching.
Is this about the reliable stack traces, i.e., the ability to detect non-leaf functions that don't create stack frames? I think we should be able to manage this without objtool on arm64 tbh.
Hm? How else would you ensure all functions honor CONFIG_FRAME_POINTER, and continue to do so indefinitely?
Quoted 11 lines
Quoted 7 lines
But
that will be a lot more work than it took for Ard to write a GCC plugin.Quoted 1 line
I really think that hard reliance on GCC plugins is foolish
Funny, I feel the same way about hard reliance on objtool :-)
I tend to agree here. I think objtool is a necessary evil (as are compiler plugins, for that matter) which I hope does not spread to other architectures.
I agree that it's a necessary evil, but it may be necessary on arm64 for live patching.
Quoted 11 lines
But the main difference is that the GCC plugin is only ~50 lines (for this particular use case, and minus another 50 lines of boilerplate), whereas objtool (AIUI) duplicates lots and lots of functionality of the compiler, assembler and/or linker, to mangle relocations, create new sections etc etc. Porting this to other architectures is going to be a major maintenance effort, especially when I think of, e.g., 32-bit ARM with its Thumb2 quirks and other idiosyncrasies that are currently hidden in the toolchain. Other architectures should be first class citizens if objtool gains support for them, which means that the x86 people that own it currently are on the hook for testing their changes against architectures they are not familiar with.
Sounds like we could use you as a co-maintainer then :-)
BTW, AFAIK, there are no plans to support live patching for 32-bit ARM.
Signature · 1 line
Josh
Steven Rostedt
reviewerQuoted 4 lines · On Sun, 11 Nov 2018 23:30:55 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 1 line
How much of that slowdown is reversed?
In theory, it should reverse all of the slowdown, and actually may even speed it up a little. Steve is working on measuring that now.
When I'm able to get it to work! Hopefully that last patch snippet you posted will help. If not, I'm assuming you'll be in Vancouver this week, and we could sit down and work it out.
That said, I don't expect a 100% improvement. Because the retpoline causes slow down in other areas than just tracing, which is not being fixed by this. I'm expecting a substantial improvement (which I see good improvement with the unoptimized static calls), and hoping for much more with the optimized one (when I get it working). But not 100%, as stated above.
-- Steve
Quoted 11 lines · Steven Rostedt
Quoted 4 lines · On Sun, 11 Nov 2018 23:30:55 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 1 line
How much of that slowdown is reversed?
In theory, it should reverse all of the slowdown, and actually may even speed it up a little. Steve is working on measuring that now.
When I'm able to get it to work! Hopefully that last patch snippet you posted will help. If not, I'm assuming you'll be in Vancouver this week, and we could sit down and work it out.
Sure, I'm already in Vancouver. Just grab me if you see me, or ping me on IRC/email. Or feel free to send me your patches if it's still giving you trouble.
Quoted 6 lines
That said, I don't expect a 100% improvement. Because the retpoline causes slow down in other areas than just tracing, which is not being fixed by this. I'm expecting a substantial improvement (which I see good improvement with the unoptimized static calls), and hoping for much more with the optimized one (when I get it working). But not 100%, as stated above.
Ah, ok. Makes sense.
Signature · 1 line
Josh
Quoted 39 lines · Ingo Molnar
Quoted 9 lines · Josh Poimboeuf
Quoted 7 lines · Ingo Molnar
Quoted 3 lines
- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.
I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.
Quoted 6 lines
The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.
The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).
Which architectures would benefit from ORC support the most?
I really think that hard reliance on GCC plugins is foolish - but maybe Clang's plugin infrastructure is a guarantee that it remains a sane and usable interface.
Quoted 13 lines
Quoted 8 lines
I'd be very happy with a demonstrated paravirt optimization already - i.e. seeing the before/after effect on the vmlinux with an x86 distro config.
All major Linux distributions enable
CONFIG_PARAVIRT=yandCONFIG_PARAVIRT_XXL=yon x86 at the moment, so optimizing it away as much as possible in the 99.999% cases where it's not used is a primary concern.For paravirt, I was thinking of it as more of a cleanup than an optimization. The paravirt patching code already replaces indirect branches with direct ones -- see
paravirt_patch_default().
Quoted 5 lines
Quoted 3 lines
Though it *would* reduce the instruction footprint a bit, as the 7-byte indirect calls (later patched to 5-byte direct + 2-byte nop) would instead be 5-byte direct calls to begin with.
Yes.
It would be a huge cleanup IMO -- the existing PVOP call stuff is really quite ugly IMO. Also, the existing stuff tries to emulate the semantics of passing parameters of unknown types using asm constraints, and I just don't believe that GCC does what we want it to do. In general, passing the *value* of a pointer to asm doesn't seem to convince gcc that the pointed-to value is used by the asm, and this makes me nervous. See commit 715bd9d12f84d8f5cc8ad21d888f9bc304a8eb0b as an example of this. In a similar vein, the existing PVOP calls have a "memory" clobber, and that's not free.
Quoted 7 lines · Ingo Molnar
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
For tracing, we'd want static_call_set_to_nop() or something like that, right?
--Andy
Quoted 10 lines · Andy Lutomirski
Quoted 7 lines · Ingo Molnar
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
For tracing, we'd want
static_call_set_to_nop()or something like that, right?
Are we talking about tracepoints? Or ftrace?
Signature · 1 line
Josh
Quoted 10 lines · Josh Poimboeuf
Quoted 9 lines · Andy Lutomirski
Quoted 7 lines · Ingo Molnar
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
Quoted 3 lines
Quoted 1 line
For tracing, we'd want
static_call_set_to_nop()or something like that, right?Are we talking about tracepoints? Or ftrace?
Since ftrace changes calls to nops, and vice versa, I assume you meant ftrace. I don't think ftrace is a good candidate for this, as it's inherently more flexible than this API would reasonably allow.
Signature · 1 line
Josh
Steven Rostedt
reviewerQuoted 11 lines · On Fri, 9 Nov 2018 10:41:37 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 10 lines · Josh Poimboeuf
Quoted 9 lines · Andy Lutomirski
Quoted 7 lines · Ingo Molnar
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
For tracing, we'd want
static_call_set_to_nop()or something like that, right?Are we talking about tracepoints? Or ftrace?
Since ftrace changes calls to nops, and vice versa, I assume you meant ftrace. I don't think ftrace is a good candidate for this, as it's inherently more flexible than this API would reasonably allow.
Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.
-- Steve
Quoted 27 lines · Steven Rostedt
Quoted 11 lines · On Fri, 9 Nov 2018 10:41:37 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 11 lines · Josh Poimboeuf
Quoted 10 lines · Andy Lutomirski
Quoted 8 lines · Ingo Molnar
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
For tracing, we'd want
static_call_set_to_nop()or something like that, right?Are we talking about tracepoints? Or ftrace?
Since ftrace changes calls to nops, and vice versa, I assume you meant ftrace. I don't think ftrace is a good candidate for this, as it's inherently more flexible than this API would reasonably allow.
Quoted 6 lines
Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.
Indeed.
Although I had assumed that tracepoints already had appropriate jump label magic.
Steven Rostedt
reviewerQuoted 10 lines · On Fri, 9 Nov 2018 11:05:51 -0800 Andy Lutomirski <[email protected]> wrote:
Quoted 6 lines
Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.
Indeed.
Although I had assumed that tracepoints already had appropriate jump label magic.
It does. But that's not the problem I was trying to solve. It's that tracing took a 8% noise dive with retpolines when enabled (hackbench slowed down by 8% with all the trace events enabled compared to all trace events enabled without retpoline). That is, normal users (those not tracinng) are not affected by trace events slowing down by retpoline. Those that care about performance when they are tracing, are affected by retpoline, quite drastically.
I'm doing another test run and measurements, to see how the unoptimized trampolines help, followed by the trampoline case.
-- Steve
Quoted 22 lines · Steven Rostedt
Quoted 10 lines · On Fri, 9 Nov 2018 11:05:51 -0800 Andy Lutomirski <[email protected]> wrote:
Quoted 6 lines
Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.
Indeed.
Although I had assumed that tracepoints already had appropriate jump label magic.
It does. But that's not the problem I was trying to solve. It's that tracing took a 8% noise dive with retpolines when enabled (hackbench slowed down by 8% with all the trace events enabled compared to all trace events enabled without retpoline). That is, normal users (those not tracinng) are not affected by trace events slowing down by retpoline. Those that care about performance when they are tracing, are affected by retpoline, quite drastically.
Quoted 2 lines
I'm doing another test run and measurements, to see how the unoptimized trampolines help, followed by the trampoline case.
Are you sure you're using unoptimized? Optimized is the default on x86-64 (with my third patch).
Signature · 1 line
Josh
Steven Rostedt
reviewerQuoted 23 lines · On Fri, 9 Nov 2018 13:44:09 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 22 lines · Steven Rostedt
Quoted 10 lines · On Fri, 9 Nov 2018 11:05:51 -0800 Andy Lutomirski <[email protected]> wrote:
Quoted 6 lines
Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.
Indeed.
Although I had assumed that tracepoints already had appropriate jump label magic.
It does. But that's not the problem I was trying to solve. It's that tracing took a 8% noise dive with retpolines when enabled (hackbench slowed down by 8% with all the trace events enabled compared to all trace events enabled without retpoline). That is, normal users (those not tracinng) are not affected by trace events slowing down by retpoline. Those that care about performance when they are tracing, are affected by retpoline, quite drastically.
Quoted 6 lines
Quoted 2 lines
I'm doing another test run and measurements, to see how the unoptimized trampolines help, followed by the trampoline case.
Are you sure you're using unoptimized? Optimized is the default on x86-64 (with my third patch).
Yes, because I haven't applied that third patch yet ;-)
Then I'll apply it and see how much that improves things.
-- Steve
Quoted 29 lines · Steven Rostedt
Quoted 26 lines · On Fri, 9 Nov 2018 13:44:09 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 24 lines · Steven Rostedt
Quoted 10 lines · On Fri, 9 Nov 2018 11:05:51 -0800 Andy Lutomirski <[email protected]> wrote:
Quoted 6 lines
Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.
Indeed.
Although I had assumed that tracepoints already had appropriate jump label magic.
It does. But that's not the problem I was trying to solve. It's that tracing took a 8% noise dive with retpolines when enabled (hackbench slowed down by 8% with all the trace events enabled compared to all trace events enabled without retpoline). That is, normal users (those not tracinng) are not affected by trace events slowing down by retpoline. Those that care about performance when they are tracing, are affected by retpoline, quite drastically.
I'm doing another test run and measurements, to see how the unoptimized trampolines help, followed by the trampoline case.
Quoted 7 lines
Quoted 3 lines
Are you sure you're using unoptimized? Optimized is the default on x86-64 (with my third patch).
Yes, because I haven't applied that third patch yet ;-)
Then I'll apply it and see how much that improves things.
Ah, good. That will be interesting to see the difference between optimized/unoptimized.
Signature · 1 line
Josh
Quoted 34 lines · On Fri, 9 Nov 2018 11:05:51 -0800 Andy Lutomirski <[email protected]> wrote:
Quoted 27 lines · Steven Rostedt
Quoted 11 lines · On Fri, 9 Nov 2018 10:41:37 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 11 lines · Josh Poimboeuf
Quoted 10 lines · Andy Lutomirski
Quoted 8 lines · Ingo Molnar
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
For tracing, we'd want
static_call_set_to_nop()or something like that, right?Are we talking about tracepoints? Or ftrace?
Since ftrace changes calls to nops, and vice versa, I assume you meant ftrace. I don't think ftrace is a good candidate for this, as it's inherently more flexible than this API would reasonably allow.
Quoted 6 lines
Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.
Quoted 3 lines
Indeed.
Although I had assumed that tracepoints already had appropriate jump label magic.
As far as I know, the jump label magic is for reducing the overhead when the tracepoint is OFF (because it can skip function parameter preparation), and this static call will be good when the tracepoint is ON (enabled) because of this can avoid retpoline performance degradation.
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
Quoted 11 lines · Andy Lutomirski
Quoted 7 lines · Ingo Molnar
All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.
For tracing, we'd want
static_call_set_to_nop()or something like that, right?
Hm. IIUC, when gcc sees static_call(key)(...), it has to generate code to put the right values in %rdi, %rsi etc.. Even if the function is void (*)(void), gcc would still need to shuffle things around (either spill and reload, or move %rdi to some callee saved register). So if the static_call is noop'ed out most of the time, that seems like a net loss? With an unlikely static_key, gcc can do all the parameter setup and reloading in an out-of-line chunk of code.
static calls seems like a quite useful concept, but only/mostly if _some_ function needs to be called at that spot.
Aside: there should be some compile-time check that static_call_set_to_nop can only be used if the return type is void.
Rasmus