Skip to contentSkip to conversation

Static calls

RFC PATCH Superseded Josh Poimboeuf 9 participants 57 messages Nov 8, 2018 – Nov 12, 2018

Subsystems (6)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover Static calls

These patches are related to two similar patch sets from Ard and Steve:

- https://lkml.kernel.org/r/[email protected]
- https://lkml.kernel.org/r/[email protected]

The code is also heavily inspired by the jump label code, as some of the concepts are very similar.

There are three separate implementations, depending on what the arch supports:

1) CONFIG_HAVE_STATIC_CALL_OPTIMIZED: patched call sites - requires objtool and a small amount of arch code

2) CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED: patched trampolines - requires a small amount of arch code

3) If no arch support, fall back to regular function pointers

TODO:

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?

- Find some actual users of the interfaces (tracepoints? crypto?)

Details (cribbed from comments in include/linux/static_call.h):
------------------------------------------------------------------------

Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.

API overview:

DECLARE_STATIC_CALL(key, func);
DEFINE_STATIC_CALL(key, func);
static_call(key, args...);
static_call_update(key, func);

Usage example:

  # Start with the following functions (with identical prototypes):
  int func_a(int arg1, int arg2);
  int func_b(int arg1, int arg2);

  # Define a 'my_key' reference, associated with func_a() by default
  DEFINE_STATIC_CALL(my_key, func_a);

  # Call func_a()
  static_call(my_key, arg1, arg2);

  # Update 'my_key' to point to func_b()
  static_call_update(my_key, func_b);

  # Call func_b()
  static_call(my_key, arg1, arg2);

Implementation details:

There are three different implementations:

1) Optimized static calls (patched call sites)

This requires objtool, which detects all the static_call() sites and annotates them in the '.static_call_sites' section. By default, the call sites will call into a temporary per-key trampoline which has an indirect branch to the current destination function associated with the key. During system boot (or module init), all call sites are patched to call their destination functions directly. Updates to a key will patch all call sites associated with that key.

2) Unoptimized static calls (patched trampolines)

Each static_call() site calls into a permanent trampoline associated with the key. The trampoline has a direct branch to the default function. Updates to a key will modify the direct branch in the key's trampoline.

3) Generic implementation

This is the default implementation if the architecture hasn't implemented CONFIG_HAVE_STATIC_CALL_[UN]OPTIMIZED. In this case, a basic function pointer is used.

Josh Poimboeuf (3):
static_call: Add static call infrastructure x86/static_call: Add x86 unoptimized static call implementation x86/static_call: Add optimized static call implementation for 64-bit

 arch/Kconfig                                  |   6 +
 arch/x86/Kconfig                              |   4 +-
 arch/x86/include/asm/static_call.h            |  42 +++
 arch/x86/kernel/Makefile                      |   1 +
 arch/x86/kernel/static_call.c                 |  84 +++++
 include/asm-generic/vmlinux.lds.h             |  11 +
 include/linux/module.h                        |  10 +
 include/linux/static_call.h                   | 186 +++++++++++
 include/linux/static_call_types.h             |  19 ++
 kernel/Makefile                               |   1 +
 kernel/module.c                               |   5 +
 kernel/static_call.c                          | 297 ++++++++++++++++++
 tools/objtool/Makefile                        |   3 +-
 tools/objtool/check.c                         | 126 +++++++-
 tools/objtool/check.h                         |   2 +
 tools/objtool/elf.h                           |   1 +
 .../objtool/include/linux/static_call_types.h |  19 ++
 tools/objtool/sync-check.sh                   |   1 +
 18 files changed, 815 insertions(+), 3 deletions(-)
 create mode 100644 arch/x86/include/asm/static_call.h
 create mode 100644 arch/x86/kernel/static_call.c
 create mode 100644 include/linux/static_call.h
 create mode 100644 include/linux/static_call_types.h
 create mode 100644 kernel/static_call.c
 create mode 100644 tools/objtool/include/linux/static_call_types.h
Signature · 1 line
2.17.2
#28 lore raw

RFC PATCH 2/3 x86/static_call: Add x86 unoptimized static call implementation

Add the x86 unoptimized static call implementation. For each key, it creates a permanent trampoline which is the destination for all static calls for the given key. The trampoline has a direct jump which gets patched by static_call_update() when the destination changes.

---
 arch/x86/Kconfig                   |  1 +
 arch/x86/include/asm/static_call.h | 18 +++++++++++
 arch/x86/kernel/Makefile           |  1 +
 arch/x86/kernel/static_call.c      | 51 ++++++++++++++++++++++++++++++
 4 files changed, 71 insertions(+)
 create mode 100644 arch/x86/include/asm/static_call.h
 create mode 100644 arch/x86/kernel/static_call.c
arch/x86/Kconfig +1 -0
index b5286ad2a982..9a83c3edd839 100644
@@ -189,6 +189,7 @@ config X86
 	select HAVE_FUNCTION_ARG_ACCESS_API
 	select HAVE_STACKPROTECTOR		if CC_HAS_SANE_STACKPROTECTOR
 	select HAVE_STACK_VALIDATION		if X86_64
+	select HAVE_STATIC_CALL_UNOPTIMIZED
 	select HAVE_RSEQ
 	select HAVE_SYSCALL_TRACEPOINTS
 	select HAVE_UNSTABLE_SCHED_CLOCK
arch/x86/include/asm/static_call.h new file +18 -0
new file mode 100644
index 000000000000..de6b032cf809
@@ -0,0 +1,18 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_STATIC_CALL_H
+#define _ASM_STATIC_CALL_H
+
+/*
+ * This is a permanent trampoline which is the destination for all static calls
+ * for the given key.  The direct jump gets patched by static_call_update().
+ */
+#define ARCH_STATIC_CALL_TRAMP(key, func)				\
+	asm(".pushsection .text, \"ax\"				\n"	\
+	    ".align 4						\n"	\
+	    ".globl " STATIC_CALL_TRAMP_STR(key) "		\n"	\
+	    ".type " STATIC_CALL_TRAMP_STR(key) ", @function	\n"	\
+	    STATIC_CALL_TRAMP_STR(key) ":			\n"	\
+	    "jmp " #func "					\n"	\
+	    ".popsection					\n")
+
+#endif /* _ASM_STATIC_CALL_H */
arch/x86/kernel/Makefile +1 -0
index 8824d01c0c35..82acc8a28429 100644
@@ -62,6 +62,7 @@ obj-y			+= tsc.o tsc_msr.o io_delay.o rtc.o
 obj-y			+= pci-iommu_table.o
 obj-y			+= resource.o
 obj-y			+= irqflags.o
+obj-y			+= static_call.o
 
 obj-y				+= process.o
 obj-y				+= fpu/
arch/x86/kernel/static_call.c new file +51 -0

51 lines not on this page · raw message

Signature · 1 line
2.17.2
#29 lore raw

RFC PATCH 3/3 x86/static_call: Add optimized static call implementation for 64-bit

In reply to Josh Poimboeuf

Add the optimized static call implementation for x86-64. For each key, a temporary trampoline is created, named __static_call_tramp_<key>. The trampoline has an indirect jump to the destination function.

Objtool uses the trampoline naming convention to detect all the call sites. It then annotates those call sites in the .static_call_sites section.

During boot (and module init), the call sites are patched to call directly into the destination function. The temporary trampoline is then no longer used.

---
 arch/x86/Kconfig                              |   5 +-
 arch/x86/include/asm/static_call.h            |  24 ++++
 arch/x86/kernel/static_call.c                 |  35 ++++-
 tools/objtool/Makefile                        |   3 +-
 tools/objtool/check.c                         | 126 +++++++++++++++++-
 tools/objtool/check.h                         |   2 +
 tools/objtool/elf.h                           |   1 +
 .../objtool/include/linux/static_call_types.h |  19 +++
 tools/objtool/sync-check.sh                   |   1 +
 9 files changed, 211 insertions(+), 5 deletions(-)
 create mode 100644 tools/objtool/include/linux/static_call_types.h
arch/x86/Kconfig +3 -2
index 9a83c3edd839..1d9b0e06b3b0 100644
@@ -189,7 +189,8 @@ config X86
 	select HAVE_FUNCTION_ARG_ACCESS_API
 	select HAVE_STACKPROTECTOR		if CC_HAS_SANE_STACKPROTECTOR
 	select HAVE_STACK_VALIDATION		if X86_64
-	select HAVE_STATIC_CALL_UNOPTIMIZED
+	select HAVE_STATIC_CALL_OPTIMIZED	if HAVE_STACK_VALIDATION
+	select HAVE_STATIC_CALL_UNOPTIMIZED	if !HAVE_STACK_VALIDATION
 	select HAVE_RSEQ
 	select HAVE_SYSCALL_TRACEPOINTS
 	select HAVE_UNSTABLE_SCHED_CLOCK
@@ -203,6 +204,7 @@ config X86
 	select RTC_MC146818_LIB
 	select SPARSE_IRQ
 	select SRCU
+	select STACK_VALIDATION			if HAVE_STACK_VALIDATION && (HAVE_STATIC_CALL_OPTIMIZED || RETPOLINE)
 	select SYSCTL_EXCEPTION_TRACE
 	select THREAD_INFO_IN_TASK
 	select USER_STACKTRACE_SUPPORT
@@ -438,7 +440,6 @@ config GOLDFISH
 config RETPOLINE
 	bool "Avoid speculative indirect branches in kernel"
 	default y
-	select STACK_VALIDATION if HAVE_STACK_VALIDATION
 	help
 	  Compile kernel with the retpoline compiler options to guard against
 	  kernel-to-user data leaks by avoiding speculative indirect
arch/x86/include/asm/static_call.h +24 -0
index de6b032cf809..d3b40c34a00f 100644
@@ -2,6 +2,27 @@
 #ifndef _ASM_STATIC_CALL_H
 #define _ASM_STATIC_CALL_H
 
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+/*
+ * This is a temporary trampoline which is only used during boot (before the
+ * call sites have been patched).  It uses the current value of the key->func
+ * pointer to do an indirect jump to the function.
+ *
+ * The name of this function has a magical aspect.  Objtool uses it to find
+ * static call sites so that it can create the .static_call_sites section.
+ */
+#define ARCH_STATIC_CALL_TEMPORARY_TRAMP(key)				\
+	asm(".pushsection .text, \"ax\"				\n"	\
+	    ".align 4						\n"	\
+	    ".globl " STATIC_CALL_TRAMP_STR(key) "		\n"	\
+	    ".type " STATIC_CALL_TRAMP_STR(key) ", @function	\n"	\
+	    STATIC_CALL_TRAMP_STR(key) ":			\n"	\
+	    ANNOTATE_RETPOLINE_SAFE "				\n"	\
+	    "jmpq *" __stringify(key) "(%rip)			\n"	\
+	    ".popsection					\n")
+
+#else /* !CONFIG_HAVE_STATIC_CALL_OPTIMIZED */
+
 /*
  * This is a permanent trampoline which is the destination for all static calls
  * for the given key.  The direct jump gets patched by static_call_update().
@@ -15,4 +36,7 @@
 	    "jmp " #func "					\n"	\
 	    ".popsection					\n")
 
+
+#endif /* !CONFIG_HAVE_STATIC_CALL_OPTIMIZED */
+
 #endif /* _ASM_STATIC_CALL_H */
arch/x86/kernel/static_call.c +34 -1
index 47ddc655ccda..f3176a7ea6d9 100644
@@ -10,6 +10,22 @@
 void static_call_bp_handler(void);
 void *bp_handler_dest;
 
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+
+void *bp_handler_continue;
+
+asm(".pushsection .text, \"ax\"						\n"
+    ".globl static_call_bp_handler					\n"
+    ".type static_call_bp_handler, @function				\n"
+    "static_call_bp_handler:						\n"
+    "ANNOTATE_RETPOLINE_SAFE						\n"
+    "call *bp_handler_dest						\n"
+    "ANNOTATE_RETPOLINE_SAFE						\n"
+    "jmp *bp_handler_continue						\n"
+    ".popsection							\n");
+
+#else /* !CONFIG_HAVE_STATIC_CALL_OPTIMIZED */
+
 asm(".pushsection .text, \"ax\"						\n"
     ".globl static_call_bp_handler					\n"
     ".type static_call_bp_handler, @function				\n"
@@ -18,6 +34,8 @@ asm(".pushsection .text, \"ax\"						\n"
     "jmp *bp_handler_dest						\n"
     ".popsection							\n");
 
+#endif /* !CONFIG_HAVE_STATIC_CALL_OPTIMIZED */
+
 void arch_static_call_transform(unsigned long insn, void *dest)
 {
 	s32 dest_relative;
@@ -38,8 +56,11 @@ void arch_static_call_transform(unsigned long insn, void *dest)
 	opcodes[0] = insn_opcode;
 	memcpy(&opcodes[1], &dest_relative, CALL_INSN_SIZE - 1);
 
-	/* Set up the variable for the breakpoint handler: */
+	/* Set up the variables for the breakpoint handler: */
 	bp_handler_dest = dest;
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+	bp_handler_continue = (void *)(insn + CALL_INSN_SIZE);
+#endif
 
 	/* Patch the call site: */
 	text_poke_bp((void *)insn, opcodes, CALL_INSN_SIZE,
@@ -49,3 +70,15 @@ void arch_static_call_transform(unsigned long insn, void *dest)
 	mutex_unlock(&text_mutex);
 }
 EXPORT_SYMBOL_GPL(arch_static_call_transform);
+
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+void arch_static_call_poison_tramp(unsigned long insn)
+{
+	unsigned long tramp = insn + CALL_INSN_SIZE + *(s32 *)(insn + 1);
+	unsigned short opcode = INSN_UD2;
+
+	mutex_lock(&text_mutex);
+	text_poke((void *)tramp, &opcode, 2);
+	mutex_unlock(&text_mutex);
+}
+#endif
tools/objtool/Makefile +2 -1
index c9d038f91af6..fb1afa34f10d 100644
@@ -29,7 +29,8 @@ all: $(OBJTOOL)
 
 INCLUDES := -I$(srctree)/tools/include \
 	    -I$(srctree)/tools/arch/$(HOSTARCH)/include/uapi \
-	    -I$(srctree)/tools/objtool/arch/$(ARCH)/include
+	    -I$(srctree)/tools/objtool/arch/$(ARCH)/include \
+	    -I$(srctree)/tools/objtool/include
 WARNINGS := $(EXTRA_WARNINGS) -Wno-switch-default -Wno-switch-enum -Wno-packed
 CFLAGS   += -Werror $(WARNINGS) $(KBUILD_HOSTCFLAGS) -g $(INCLUDES)
 LDFLAGS  += -lelf $(LIBSUBCMD) $(KBUILD_HOSTLDFLAGS)
tools/objtool/check.c +125 -1

192 lines not on this page · raw message

tools/objtool/check.h +2 -0
index e6e8a655b556..56b8b7fb1bd1 100644
@@ -39,6 +39,7 @@ struct insn_state {
 struct instruction {
 	struct list_head list;
 	struct hlist_node hash;
+	struct list_head static_call_node;
 	struct section *sec;
 	unsigned long offset;
 	unsigned int len;
@@ -60,6 +61,7 @@ struct objtool_file {
 	struct elf *elf;
 	struct list_head insn_list;
 	DECLARE_HASHTABLE(insn_hash, 16);
+	struct list_head static_call_list;
 	struct section *whitelist;
 	bool ignore_unreachables, c_file, hints, rodata;
 };
tools/objtool/elf.h +1 -0
index bc97ed86b9cd..3cf44d7cc3ac 100644
@@ -62,6 +62,7 @@ struct symbol {
 	unsigned long offset;
 	unsigned int len;
 	struct symbol *pfunc, *cfunc;
+	bool static_call_tramp;
 };
 
 struct rela {
tools/objtool/include/linux/static_call_types.h new file +19 -0
new file mode 100644
index 000000000000..7dd4b3d7ec2b
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _STATIC_CALL_TYPES_H
+#define _STATIC_CALL_TYPES_H
+
+#include <linux/stringify.h>
+
+#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_
+#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX)
+
+#define STATIC_CALL_TRAMP(key) STATIC_CALL_TRAMP_PREFIX##key
+#define STATIC_CALL_TRAMP_STR(key) __stringify(STATIC_CALL_TRAMP(key))
+
+/* The static call site table is created by objtool. */
+struct static_call_site {
+	s32 addr;
+	s32 key;
+};
+
+#endif /* _STATIC_CALL_TYPES_H */
tools/objtool/sync-check.sh +1 -0
index 1470e74e9d66..e1a204bf3556 100755
@@ -10,6 +10,7 @@ arch/x86/include/asm/insn.h
 arch/x86/include/asm/inat.h
 arch/x86/include/asm/inat_types.h
 arch/x86/include/asm/orc_types.h
+include/linux/static_call_types.h
 '
 
 check()
Signature · 1 line
2.17.2
#1 lore raw

RFC PATCH 1/3 static_call: Add static call infrastructure

In reply to Josh Poimboeuf

Add a static call infrastructure. Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.

This code is heavily inspired by the jump label code (aka "static jumps"), as some of the concepts are very similar.

There are three implementations, depending on arch support:

1) optimized: patched call sites (CONFIG_HAVE_STATIC_CALL_OPTIMIZED)
2) unoptimized: patched trampolines (CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)
3) basic function pointers

For more details, see the comments in include/linux/static_call.h.

---
 arch/Kconfig                      |   6 +
 include/asm-generic/vmlinux.lds.h |  11 ++
 include/linux/module.h            |  10 +
 include/linux/static_call.h       | 186 +++++++++++++++++++
 include/linux/static_call_types.h |  19 ++
 kernel/Makefile                   |   1 +
 kernel/module.c                   |   5 +
 kernel/static_call.c              | 297 ++++++++++++++++++++++++++++++
 8 files changed, 535 insertions(+)
 create mode 100644 include/linux/static_call.h
 create mode 100644 include/linux/static_call_types.h
 create mode 100644 kernel/static_call.c
arch/Kconfig +6 -0
index e1e540ffa979..9657ad5a80d1 100644
@@ -879,6 +879,12 @@ config HAVE_ARCH_PREL32_RELOCATIONS
 	  architectures, and don't require runtime relocation on relocatable
 	  kernels.
 
+config HAVE_STATIC_CALL_OPTIMIZED
+	bool
+
+config HAVE_STATIC_CALL_UNOPTIMIZED
+	bool
+
 source "kernel/gcov/Kconfig"
 
 source "scripts/gcc-plugins/Kconfig"
include/asm-generic/vmlinux.lds.h +11 -0
index 3d7a6a9c2370..45d8ced28bd0 100644
@@ -320,6 +320,7 @@
 	__start_ro_after_init = .;					\
 	*(.data..ro_after_init)						\
 	JUMP_TABLE_DATA							\
+	STATIC_CALL_SITES						\
 	__end_ro_after_init = .;
 #endif
 
@@ -725,6 +726,16 @@
 #define BUG_TABLE
 #endif
 
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+#define STATIC_CALL_SITES						\
+	. = ALIGN(8);							\
+	__start_static_call_sites = .;					\
+	KEEP(*(.static_call_sites))					\
+	__stop_static_call_sites = .;
+#else
+#define STATIC_CALL_SITES
+#endif
+
 #ifdef CONFIG_UNWINDER_ORC
 #define ORC_UNWIND_TABLE						\
 	. = ALIGN(4);							\
include/linux/module.h +10 -0
index fce6b4335e36..91baf181247a 100644
@@ -21,6 +21,7 @@
 #include <linux/rbtree_latch.h>
 #include <linux/error-injection.h>
 #include <linux/tracepoint-defs.h>
+#include <linux/static_call_types.h>
 
 #include <linux/percpu.h>
 #include <asm/module.h>
@@ -450,6 +451,10 @@ struct module {
 	unsigned int num_ftrace_callsites;
 	unsigned long *ftrace_callsites;
 #endif
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+	int num_static_call_sites;
+	struct static_call_site *static_call_sites;
+#endif
 
 #ifdef CONFIG_LIVEPATCH
 	bool klp; /* Is this a livepatch module? */
@@ -682,6 +687,11 @@ static inline bool is_module_text_address(unsigned long addr)
 	return false;
 }
 
+static inline bool within_module_init(unsigned long addr, const struct module *mod)
+{
+	return false;
+}
+
 /* Get/put a kernel symbol (calls should be symmetric) */
 #define symbol_get(x) ({ extern typeof(x) x __attribute__((weak)); &(x); })
 #define symbol_put(x) do { } while (0)
include/linux/static_call.h new file +186 -0

186 lines not on this page · raw message

include/linux/static_call_types.h new file +19 -0
new file mode 100644
index 000000000000..7dd4b3d7ec2b
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _STATIC_CALL_TYPES_H
+#define _STATIC_CALL_TYPES_H
+
+#include <linux/stringify.h>
+
+#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_
+#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX)
+
+#define STATIC_CALL_TRAMP(key) STATIC_CALL_TRAMP_PREFIX##key
+#define STATIC_CALL_TRAMP_STR(key) __stringify(STATIC_CALL_TRAMP(key))
+
+/* The static call site table is created by objtool. */
+struct static_call_site {
+	s32 addr;
+	s32 key;
+};
+
+#endif /* _STATIC_CALL_TYPES_H */
kernel/Makefile +1 -0
index 7343b3a9bff0..888bb476ddaf 100644
@@ -103,6 +103,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/
 obj-$(CONFIG_IRQ_WORK) += irq_work.o
 obj-$(CONFIG_CPU_PM) += cpu_pm.o
 obj-$(CONFIG_BPF) += bpf/
+obj-$(CONFIG_HAVE_STATIC_CALL_OPTIMIZED) += static_call.o
 
 obj-$(CONFIG_PERF_EVENTS) += events/
 
kernel/module.c +5 -0
index 49a405891587..11124d991fcd 100644
@@ -3121,6 +3121,11 @@ static int find_module_sections(struct module *mod, struct load_info *info)
 	mod->ei_funcs = section_objs(info, "_error_injection_whitelist",
 					    sizeof(*mod->ei_funcs),
 					    &mod->num_ei_funcs);
+#endif
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+	mod->static_call_sites = section_objs(info, ".static_call_sites",
+					      sizeof(*mod->static_call_sites),
+					      &mod->num_static_call_sites);
 #endif
 	mod->extable = section_objs(info, "__ex_table",
 				    sizeof(*mod->extable), &mod->num_exentries);
kernel/static_call.c new file +297 -0

297 lines not on this page · raw message

Signature · 1 line
2.17.2
#30 lore raw
In reply to Josh Poimboeuf
Quoted 2 lines · Josh Poimboeuf

- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?

I forgot Andy mentioned that we might be able to use this to clean up paravirt patching, in which case it would have a lot of value, retpolines or not...

Signature · 1 line
Josh
#31 lore raw
In reply to Josh Poimboeuf
Quoted 17 lines · Josh Poimboeuf

These patches are related to two similar patch sets from Ard and Steve:

- https://lkml.kernel.org/r/[email protected]
- https://lkml.kernel.org/r/[email protected]

The code is also heavily inspired by the jump label code, as some of the concepts are very similar.

There are three separate implementations, depending on what the arch supports:

1) CONFIG_HAVE_STATIC_CALL_OPTIMIZED: patched call sites - requires objtool and a small amount of arch code

2) CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED: patched trampolines - requires a small amount of arch code

Quoted 8 lines

3) If no arch support, fall back to regular function pointers

TODO:

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

Quoted 2 lines

- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?

Paravirt patching, as you mention in your later reply?

Quoted 1 line

- Find some actual users of the interfaces (tracepoints? crypto?)

I'd be very happy with a demonstrated paravirt optimization already - i.e. seeing the before/after effect on the vmlinux with an x86 distro config.

All major Linux distributions enable CONFIG_PARAVIRT=y and CONFIG_PARAVIRT_XXL=y on x86 at the moment, so optimizing it away as much as possible in the 99.999% cases where it's not used is a primary concern.

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

Thanks,

Ingo

#32 lore raw
Quoted 4 lines · Ingo Molnar
Quoted 2 lines

- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?

Paravirt patching, as you mention in your later reply?

BTW., to look for candidates of this API, I'd suggest looking at the function call frequency of my (almost-)distro kernel vmlinux:

  $ objdump -d vmlinux | grep -w callq | cut -f3- | sort | uniq -c | sort -n | tail -100

which gives:

    502 callq  ffffffff8157d050 <nla_put>
    522 callq  ffffffff81aaf420 <down_write>
    536 callq  ffffffff81547e60 <_copy_to_user>
    615 callq  ffffffff81a97700 <snprintf>
    624 callq  *0xffffffff82648428
    624 callq  ffffffff810cc810 <__might_sleep>
    625 callq  ffffffff81a93b90 <strcmp>
    649 callq  ffffffff81547dd0 <_copy_from_user>
    651 callq  ffffffff811ba930 <trace_seq_printf>
    654 callq  ffffffff8170b6f0 <_dev_warn>
    691 callq  ffffffff81a93790 <strlen>
    693 callq  ffffffff81a88dc0 <cpumask_next>
    709 callq  *0xffffffff82648438
    723 callq  ffffffff811bdbd0 <trace_hardirqs_on>
    735 callq  ffffffff810feac0 <up_write>
    750 callq  ffffffff8163e9f0 <acpi_ut_status_exit>
    768 callq  *0xffffffff82648430
    814 callq  ffffffff81ab2710 <_raw_spin_lock_irq>
    841 callq  ffffffff81a9e680 <__memcpy>
    863 callq  ffffffff812ae3d0 <__kmalloc>
    899 callq  ffffffff8126ac80 <__might_fault>
    912 callq  ffffffff81ab2970 <_raw_spin_unlock_irq>
    939 callq  ffffffff81aaaf10 <_cond_resched>
    966 callq  ffffffff811bda00 <trace_hardirqs_off>
   1069 callq  ffffffff81126f50 <rcu_read_lock_sched_held>
   1078 callq  ffffffff81097760 <__warn_printk>
   1081 callq  ffffffff8157b140 <__dynamic_dev_dbg>
   1351 callq  ffffffff8170b630 <_dev_err>
   1365 callq  ffffffff811050c0 <lock_is_held_type>
   1373 callq  ffffffff81a977f0 <sprintf>
   1390 callq  ffffffff8157b090 <__dynamic_pr_debug>
   1453 callq  ffffffff8155c650 <__list_add_valid>
   1501 callq  ffffffff812ad6f0 <kmem_cache_alloc_trace>
   1509 callq  ffffffff8155c6c0 <__list_del_entry_valid>
   1513 callq  ffffffff81310ce0 <seq_printf>
   1571 callq  ffffffff81ab2780 <_raw_spin_lock_irqsave>
   1624 callq  ffffffff81ab29b0 <_raw_spin_unlock_irqrestore>
   1661 callq  ffffffff81126fd0 <rcu_read_lock_held>
   1986 callq  ffffffff81104940 <lock_acquire>
   2050 callq  ffffffff811c5110 <trace_define_field>
   2133 callq  ffffffff81102c70 <lock_release>
   2507 callq  ffffffff81ab2560 <_raw_spin_lock>
   2676 callq  ffffffff81aadc40 <mutex_lock_nested>
   3056 callq  ffffffff81ab2900 <_raw_spin_unlock>
   3294 callq  ffffffff81aac610 <mutex_unlock>
   3628 callq  ffffffff81129100 <rcu_is_watching>
   4462 callq  ffffffff812ac2c0 <kfree>
   6454 callq  ffffffff8111a51e <printk>
   6676 callq  ffffffff81101420 <lockdep_rcu_suspicious>
   7328 callq  ffffffff81e014b0 <__x86_indirect_thunk_rax>
   7598 callq  ffffffff81126f30 <debug_lockdep_rcu_enabled>
   9065 callq  ffffffff810979f0 <__stack_chk_fail>

The most prominent callers which are already function call pointers today are:

  $ objdump -d vmlinux | grep -w callq | grep \* | cut -f3- | sort | uniq -c | sort -n | tail -10
    109 callq  *0xffffffff82648530
    134 callq  *0xffffffff82648568
    154 callq  *0xffffffff826483d0
    260 callq  *0xffffffff826483d8
    297 callq  *0xffffffff826483e0
    345 callq  *0xffffffff82648440
    345 callq  *0xffffffff82648558
    624 callq  *0xffffffff82648428
    709 callq  *0xffffffff82648438
    768 callq  *0xffffffff82648430

That's all pv_ops->*() method calls:

ffffffff82648300 D pv_ops
ffffffff826485d0 D pv_info

Optimizing those thousands of function pointer calls would already be a nice improvement.

But retpolines:

7328 callq ffffffff81e014b0 <__x86_indirect_thunk_rax>

ffffffff81e014b0 <__x86_indirect_thunk_rax>:
ffffffff81e014b0: ff e0 jmpq *%rax

... are even more prominent, and turned on in every distro as well, obviously.

Thanks,

Ingo

#2 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Josh Poimboeuf

Hi Josh,

Thanks a lot for looking into this.

Quoted 6 lines · Josh Poimboeuf

Add a static call infrastructure. Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.

Quoted 7 lines

This code is heavily inspired by the jump label code (aka "static jumps"), as some of the concepts are very similar.

There are three implementations, depending on arch support:

1) optimized: patched call sites (CONFIG_HAVE_STATIC_CALL_OPTIMIZED)
2) unoptimized: patched trampolines (CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)

Could we move to an idiom like inline/out of line? The unoptimized variant may be perfectly adequate for many arches, and 'unoptimized' sounds like there is something wrong with it.

Quoted 136 lines

3) basic function pointers

For more details, see the comments in include/linux/static_call.h.

---
 arch/Kconfig                      |   6 +
 include/asm-generic/vmlinux.lds.h |  11 ++
 include/linux/module.h            |  10 +
 include/linux/static_call.h       | 186 +++++++++++++++++++
 include/linux/static_call_types.h |  19 ++
 kernel/Makefile                   |   1 +
 kernel/module.c                   |   5 +
 kernel/static_call.c              | 297 ++++++++++++++++++++++++++++++
 8 files changed, 535 insertions(+)
 create mode 100644 include/linux/static_call.h
 create mode 100644 include/linux/static_call_types.h
 create mode 100644 kernel/static_call.c
arch/Kconfig +6 -0
index e1e540ffa979..9657ad5a80d1 100644
@@ -879,6 +879,12 @@ config HAVE_ARCH_PREL32_RELOCATIONS
          architectures, and don't require runtime relocation on relocatable
          kernels.
 
+config HAVE_STATIC_CALL_OPTIMIZED
+       bool
+
+config HAVE_STATIC_CALL_UNOPTIMIZED
+       bool
+
 source "kernel/gcov/Kconfig"
 
 source "scripts/gcc-plugins/Kconfig"
include/asm-generic/vmlinux.lds.h +11 -0
index 3d7a6a9c2370..45d8ced28bd0 100644
@@ -320,6 +320,7 @@
        __start_ro_after_init = .;                                      \
        *(.data..ro_after_init)                                         \
        JUMP_TABLE_DATA                                                 \
+       STATIC_CALL_SITES                                               \
        __end_ro_after_init = .;
 #endif
 
@@ -725,6 +726,16 @@
 #define BUG_TABLE
 #endif
 
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+#define STATIC_CALL_SITES                                              \
+       . = ALIGN(8);                                                   \
+       __start_static_call_sites = .;                                  \
+       KEEP(*(.static_call_sites))                                     \
+       __stop_static_call_sites = .;
+#else
+#define STATIC_CALL_SITES
+#endif
+
 #ifdef CONFIG_UNWINDER_ORC
 #define ORC_UNWIND_TABLE                                               \
        . = ALIGN(4);                                                   \
include/linux/module.h +10 -0
index fce6b4335e36..91baf181247a 100644
@@ -21,6 +21,7 @@
 #include <linux/rbtree_latch.h>
 #include <linux/error-injection.h>
 #include <linux/tracepoint-defs.h>
+#include <linux/static_call_types.h>
 
 #include <linux/percpu.h>
 #include <asm/module.h>
@@ -450,6 +451,10 @@ struct module {
        unsigned int num_ftrace_callsites;
        unsigned long *ftrace_callsites;
 #endif
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+       int num_static_call_sites;
+       struct static_call_site *static_call_sites;
+#endif
 
 #ifdef CONFIG_LIVEPATCH
        bool klp; /* Is this a livepatch module? */
@@ -682,6 +687,11 @@ static inline bool is_module_text_address(unsigned long addr)
        return false;
 }
 
+static inline bool within_module_init(unsigned long addr, const struct module *mod)
+{
+       return false;
+}
+
 /* Get/put a kernel symbol (calls should be symmetric) */
 #define symbol_get(x) ({ extern typeof(x) x __attribute__((weak)); &(x); })
 #define symbol_put(x) do { } while (0)
include/linux/static_call.h new file +30 -0
new file mode 100644
index 000000000000..b22987057cf1
@@ -0,0 +1,186 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_STATIC_CALL_H
+#define _LINUX_STATIC_CALL_H
+
+/*
+ * Static call support
+ *
+ * Static calls use code patching to hard-code function pointers into direct
+ * branch instructions.  They give the flexibility of function pointers, but
+ * with improved performance.  This is especially important for cases where
+ * retpolines would otherwise be used, as retpolines can significantly impact
+ * performance.
+ *
+ *
+ * API overview:
+ *
+ *   DECLARE_STATIC_CALL(key, func);
+ *   DEFINE_STATIC_CALL(key, func);
+ *   static_call(key, args...);
+ *   static_call_update(key, func);
+ *
+ *
+ * Usage example:
+ *
+ *   # Start with the following functions (with identical prototypes):
+ *   int func_a(int arg1, int arg2);
+ *   int func_b(int arg1, int arg2);
+ *
+ *   # Define a 'my_key' reference, associated with func_a() by default
+ *   DEFINE_STATIC_CALL(my_key, func_a);

include/linux/static_call.h line 40

Quoted 10 lines
include/linux/static_call.h new file +10 -0
+ *
+ *   # Call func_a()
+ *   static_call(my_key, arg1, arg2);
+ *
+ *   # Update 'my_key' to point to func_b()
+ *   static_call_update(my_key, func_b);
+ *
+ *   # Call func_b()
+ *   static_call(my_key, arg1, arg2);
+ *

Any way we can revert to the default implementation? That would be useful for, e.g., unloading modules that provide an alternative implementation.

Quoted 154 lines
include/linux/static_call.h new file +146 -0
+ *
+ * Implementation details:
+ *
+ * There are three different implementations:
+ *
+ * 1) Optimized static calls (patched call sites)
+ *
+ *    This requires objtool, which detects all the static_call() sites and
+ *    annotates them in the '.static_call_sites' section.  By default, the call
+ *    sites will call into a temporary per-key trampoline which has an indirect
+ *    branch to the current destination function associated with the key.
+ *    During system boot (or module init), all call sites are patched to call
+ *    their destination functions directly.  Updates to a key will patch all
+ *    call sites associated with that key.
+ *
+ * 2) Unoptimized static calls (patched trampolines)
+ *
+ *    Each static_call() site calls into a permanent trampoline associated with
+ *    the key.  The trampoline has a direct branch to the default function.
+ *    Updates to a key will modify the direct branch in the key's trampoline.
+ *
+ * 3) Generic implementation
+ *
+ *    This is the default implementation if the architecture hasn't implemented
+ *    CONFIG_HAVE_STATIC_CALL_[UN]OPTIMIZED.  In this case, a basic
+ *    function pointer is used.
+ */
+
+#include <linux/types.h>
+#include <linux/cpu.h>
+#include <linux/static_call_types.h>
+
+#if defined(CONFIG_HAVE_STATIC_CALL_OPTIMIZED) || \
+    defined(CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)
+
+#include <asm/static_call.h>
+
+extern void arch_static_call_transform(unsigned long insn, void *dest);
+
+#endif /* CONFIG_HAVE_STATIC_CALL_[UN]OPTIMIZED */
+
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+/* Optimized implementation */
+
+struct static_call_mod {
+       struct list_head list;
+       struct module *mod; /* NULL means vmlinux */
+       struct static_call_site *sites;
+};
+
+struct static_call_key {
+       /*
+        * This field should always be first because the trampolines expect it.
+        * This points to the current call destination.
+        */
+       void *func;
+
+       /*
+        * List of modules (including vmlinux) and the call sites which need to
+        * be patched for this key.
+        */
+       struct list_head site_mods;
+};
+
+extern void __static_call_update(struct static_call_key *key, void *func);
+extern void arch_static_call_poison_tramp(unsigned long insn);
+
+#define DECLARE_STATIC_CALL(key, func)                                 \
+       extern struct static_call_key key;                              \
+       extern typeof(func) STATIC_CALL_TRAMP(key);                     \
+       /* Preserve the ELF symbol so objtool can access it: */         \
+       __ADDRESSABLE(key)
+
+#define DEFINE_STATIC_CALL(key, _func)                                 \
+       DECLARE_STATIC_CALL(key, _func);                                \
+       struct static_call_key key = {                                  \
+               .func = _func,                                          \
+               .site_mods = LIST_HEAD_INIT(key.site_mods),             \
+       };                                                              \
+       ARCH_STATIC_CALL_TEMPORARY_TRAMP(key)
+
+#define static_call(key, args...) STATIC_CALL_TRAMP(key)(args)
+
+#define static_call_update(key, func)                                  \
+({                                                                     \
+       BUILD_BUG_ON(!__same_type(typeof(func), typeof(STATIC_CALL_TRAMP(key)))); \
+       __static_call_update(&key, func);                               \
+})
+
+#define EXPORT_STATIC_CALL(key)                                                \
+       EXPORT_SYMBOL(key);                                             \
+       EXPORT_SYMBOL(STATIC_CALL_TRAMP(key))
+
+#define EXPORT_STATIC_CALL_GPL(key)                                    \
+       EXPORT_SYMBOL_GPL(key);                                         \
+       EXPORT_SYMBOL_GPL(STATIC_CALL_TRAMP(key))
+
+
+#elif defined(CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)
+/* Unoptimized implementation */
+
+#define DECLARE_STATIC_CALL(key, func)                                 \
+       extern typeof(func) STATIC_CALL_TRAMP(key)
+
+#define DEFINE_STATIC_CALL(key, func)                                  \
+       DECLARE_STATIC_CALL(key, func);                                 \
+       ARCH_STATIC_CALL_TRAMP(key, func)
+
+#define static_call(key, args...) STATIC_CALL_TRAMP(key)(args)
+
+#define static_call_update(key, func)                                  \
+({                                                                     \
+       BUILD_BUG_ON(!__same_type(func, STATIC_CALL_TRAMP(key)));       \
+       cpus_read_lock();                                               \
+       arch_static_call_transform((unsigned long)STATIC_CALL_TRAMP(key),\
+                                  func);                               \
+       cpus_read_unlock();                                             \
+})
+
+#define EXPORT_STATIC_CALL(key)                                                \
+       EXPORT_SYMBOL(STATIC_CALL_TRAMP(key))
+
+#define EXPORT_STATIC_CALL_GPL(key)                                    \
+       EXPORT_SYMBOL_GPL(STATIC_CALL_TRAMP(key))
+
+
+#else /* Generic implementation */
+
+#define DECLARE_STATIC_CALL(key, func)                                 \
+       extern typeof(func) *key
+
+#define DEFINE_STATIC_CALL(key, func)                                  \
+       typeof(func) *key = func
+
+#define static_call(key, args...)                                      \
+       key(args)
+
+#define static_call_update(key, func)                                  \
+       WRITE_ONCE(key, func)
+
+#define EXPORT_STATIC_CALL(key) EXPORT_SYMBOL(key)
+#define EXPORT_STATIC_CALL_GPL(key) EXPORT_SYMBOL_GPL(key)
+
+#endif /* CONFIG_HAVE_STATIC_CALL_OPTIMIZED */
+
+#endif /* _LINUX_STATIC_CALL_H */
include/linux/static_call_types.h new file +2 -0
new file mode 100644
index 000000000000..7dd4b3d7ec2b
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _STATIC_CALL_TYPES_H

include/linux/static_call_types.h line 12

Quoted 10 lines
include/linux/static_call_types.h new file +10 -0
+#define _STATIC_CALL_TYPES_H
+
+#include <linux/stringify.h>
+
+#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_
+#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX)
+
+#define STATIC_CALL_TRAMP(key) STATIC_CALL_TRAMP_PREFIX##key
+#define STATIC_CALL_TRAMP_STR(key) __stringify(STATIC_CALL_TRAMP(key))
+

I needed to apply

(quoted diff) +0 -0

b/include/linux/static_call_types.h
index 7dd4b3d7ec2b..6859b208de6e 100644

include/linux/static_call_types.h +1 -1
@@ -7,7 +7,7 @@
 #define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_
 #define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX)
 
-#define STATIC_CALL_TRAMP(key) STATIC_CALL_TRAMP_PREFIX##key
+#define STATIC_CALL_TRAMP(key) __PASTE(STATIC_CALL_TRAMP_PREFIX, key)
 #define STATIC_CALL_TRAMP_STR(key) __stringify(STATIC_CALL_TRAMP(key))
 
 /* The static call site table is created by objtool. */

or I end up with

0000000000000000 <STATIC_CALL_TRAMP_PREFIXfoo>:
0: 14000000 b a8 <foo_a>
4: d503201f nop

Quoted 341 lines
include/linux/static_call_types.h new file +7 -0
+/* The static call site table is created by objtool. */
+struct static_call_site {
+       s32 addr;
+       s32 key;
+};
+
+#endif /* _STATIC_CALL_TYPES_H */
kernel/Makefile +1 -0
index 7343b3a9bff0..888bb476ddaf 100644
@@ -103,6 +103,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/
 obj-$(CONFIG_IRQ_WORK) += irq_work.o
 obj-$(CONFIG_CPU_PM) += cpu_pm.o
 obj-$(CONFIG_BPF) += bpf/
+obj-$(CONFIG_HAVE_STATIC_CALL_OPTIMIZED) += static_call.o
 
 obj-$(CONFIG_PERF_EVENTS) += events/
 
kernel/module.c +5 -0
index 49a405891587..11124d991fcd 100644
@@ -3121,6 +3121,11 @@ static int find_module_sections(struct module *mod, struct load_info *info)
        mod->ei_funcs = section_objs(info, "_error_injection_whitelist",
                                            sizeof(*mod->ei_funcs),
                                            &mod->num_ei_funcs);
+#endif
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+       mod->static_call_sites = section_objs(info, ".static_call_sites",
+                                             sizeof(*mod->static_call_sites),
+                                             &mod->num_static_call_sites);
 #endif
        mod->extable = section_objs(info, "__ex_table",
                                    sizeof(*mod->extable), &mod->num_exentries);
kernel/static_call.c new file +297 -1
new file mode 100644
index 000000000000..599ebc6fc4f1
@@ -0,0 +1,297 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <linux/init.h>
+#include <linux/static_call.h>
+#include <linux/bug.h>
+#include <linux/smp.h>
+#include <linux/sort.h>
+#include <linux/slab.h>
+#include <linux/module.h>
+#include <linux/cpu.h>
+#include <linux/processor.h>
+#include <asm/sections.h>
+
+extern struct static_call_site __start_static_call_sites[],
+                              __stop_static_call_sites[];
+
+static bool static_call_initialized;
+
+/* mutex to protect key modules/sites */
+static DEFINE_MUTEX(static_call_mutex);
+
+static void static_call_lock(void)
+{
+       mutex_lock(&static_call_mutex);
+}
+
+static void static_call_unlock(void)
+{
+       mutex_unlock(&static_call_mutex);
+}
+
+static inline unsigned long static_call_addr(struct static_call_site *site)
+{
+       return (long)site->addr + (long)&site->addr;
+}
+
+static inline struct static_call_key *static_call_key(const struct static_call_site *site)
+{
+       return (struct static_call_key *)((long)site->key + (long)&site->key);
+}
+
+static int static_call_site_cmp(const void *_a, const void *_b)
+{
+       const struct static_call_site *a = _a;
+       const struct static_call_site *b = _b;
+       const struct static_call_key *key_a = static_call_key(a);
+       const struct static_call_key *key_b = static_call_key(b);
+
+       if (key_a < key_b)
+               return -1;
+
+       if (key_a > key_b)
+               return 1;
+
+       return 0;
+}
+
+static void static_call_site_swap(void *_a, void *_b, int size)
+{
+       long delta = (unsigned long)_a - (unsigned long)_b;
+       struct static_call_site *a = _a;
+       struct static_call_site *b = _b;
+       struct static_call_site tmp = *a;
+
+       a->addr = b->addr  - delta;
+       a->key  = b->key   - delta;
+
+       b->addr = tmp.addr + delta;
+       b->key  = tmp.key  + delta;
+}
+
+static inline void static_call_sort_entries(struct static_call_site *start,
+                                           struct static_call_site *stop)
+{
+       sort(start, stop - start, sizeof(struct static_call_site),
+            static_call_site_cmp, static_call_site_swap);
+}
+
+void __static_call_update(struct static_call_key *key, void *func)
+{
+       struct static_call_mod *mod;
+       struct static_call_site *site, *stop;
+
+       cpus_read_lock();
+       static_call_lock();
+
+       if (key->func == func)
+               goto done;
+
+       key->func = func;
+
+       /*
+        * If called before init, leave the call sites unpatched for now.
+        * In the meantime they'll continue to call the temporary trampoline.
+        */
+       if (!static_call_initialized)
+               goto done;
+
+       list_for_each_entry(mod, &key->site_mods, list) {
+               if (!mod->sites) {
+                       /*
+                        * This can happen if the static call key is defined in
+                        * a module which doesn't use it.
+                        */
+                       continue;
+               }
+
+               stop = __stop_static_call_sites;
+
+#ifdef CONFIG_MODULES
+               if (mod->mod) {
+                       stop = mod->mod->static_call_sites +
+                              mod->mod->num_static_call_sites;
+               }
+#endif
+
+               for (site = mod->sites;
+                    site < stop && static_call_key(site) == key; site++) {
+                       unsigned long addr = static_call_addr(site);
+
+                       if (!mod->mod && init_section_contains((void *)addr, 1))
+                               continue;
+                       if (mod->mod && within_module_init(addr, mod->mod))
+                               continue;
+
+                       arch_static_call_transform(addr, func);
+               }
+       }
+
+done:
+       static_call_unlock();
+       cpus_read_unlock();
+}
+EXPORT_SYMBOL_GPL(__static_call_update);
+
+#ifdef CONFIG_MODULES
+
+static int static_call_add_module(struct module *mod)
+{
+       struct static_call_site *start = mod->static_call_sites;
+       struct static_call_site *stop = mod->static_call_sites +
+                                       mod->num_static_call_sites;
+       struct static_call_site *site;
+       struct static_call_key *key, *prev_key = NULL;
+       struct static_call_mod *static_call_mod;
+
+       if (start == stop)
+               return 0;
+
+       module_disable_ro(mod);
+       static_call_sort_entries(start, stop);
+       module_enable_ro(mod, false);
+
+       for (site = start; site < stop; site++) {
+               unsigned long addr = static_call_addr(site);
+
+               if (within_module_init(addr, mod))
+                       continue;
+
+               key = static_call_key(site);
+               if (key != prev_key) {
+                       prev_key = key;
+
+                       static_call_mod = kzalloc(sizeof(*static_call_mod), GFP_KERNEL);
+                       if (!static_call_mod)
+                               return -ENOMEM;
+
+                       static_call_mod->mod = mod;
+                       static_call_mod->sites = site;
+                       list_add_tail(&static_call_mod->list, &key->site_mods);
+
+                       if (is_module_address((unsigned long)key)) {
+                               /*
+                                * The trampoline should no longer be used.
+                                * Poison it it with a BUG() to catch any stray
+                                * callers.
+                                */
+                               arch_static_call_poison_tramp(addr);
+                       }
+               }
+
+               arch_static_call_transform(addr, key->func);
+       }
+
+       return 0;
+}
+
+static void static_call_del_module(struct module *mod)
+{
+       struct static_call_site *start = mod->static_call_sites;
+       struct static_call_site *stop = mod->static_call_sites +
+                                       mod->num_static_call_sites;
+       struct static_call_site *site;
+       struct static_call_key *key, *prev_key = NULL;
+       struct static_call_mod *static_call_mod;
+
+       for (site = start; site < stop; site++) {
+               key = static_call_key(site);
+               if (key == prev_key)
+                       continue;
+               prev_key = key;
+
+               list_for_each_entry(static_call_mod, &key->site_mods, list) {
+                       if (static_call_mod->mod == mod) {
+                               list_del(&static_call_mod->list);
+                               kfree(static_call_mod);
+                               break;
+                       }
+               }
+       }
+}
+
+static int static_call_module_notify(struct notifier_block *nb,
+                                    unsigned long val, void *data)
+{
+       struct module *mod = data;
+       int ret = 0;
+
+       cpus_read_lock();
+       static_call_lock();
+
+       switch (val) {
+       case MODULE_STATE_COMING:
+               ret = static_call_add_module(mod);
+               if (ret) {
+                       WARN(1, "Failed to allocate memory for static calls");
+                       static_call_del_module(mod);
+               }
+               break;
+       case MODULE_STATE_GOING:
+               static_call_del_module(mod);
+               break;
+       }
+
+       static_call_unlock();
+       cpus_read_unlock();
+
+       return notifier_from_errno(ret);
+}
+
+static struct notifier_block static_call_module_nb = {
+       .notifier_call = static_call_module_notify,
+};
+
+#endif /* CONFIG_MODULES */
+
+static void __init static_call_init(void)
+{
+       struct static_call_site *start = __start_static_call_sites;
+       struct static_call_site *stop  = __stop_static_call_sites;
+       struct static_call_site *site;
+
+       if (start == stop) {
+               pr_warn("WARNING: empty static call table\n");
+               return;
+       }
+
+       cpus_read_lock();
+       static_call_lock();
+
+       static_call_sort_entries(start, stop);
+
+       for (site = start; site < stop; site++) {
+               struct static_call_key *key = static_call_key(site);
+               unsigned long addr = static_call_addr(site);
+
+               if (list_empty(&key->site_mods)) {
+                       struct static_call_mod *mod;
+
+                       mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+                       if (!mod) {
+                               WARN(1, "Failed to allocate memory for static calls");
+                               return;
+                       }
+
+                       mod->sites = site;
+                       list_add_tail(&mod->list, &key->site_mods);
+
+                       /*
+                        * The trampoline should no longer be used.  Poison it
+                        * it with a BUG() to catch any stray callers.
+                        */
+                       arch_static_call_poison_tramp(addr);
+               }
+
+               arch_static_call_transform(addr, key->func);
+       }
+
+       static_call_initialized = true;
+
+       static_call_unlock();
+       cpus_read_unlock();
+
+#ifdef CONFIG_MODULES
+       register_module_notifier(&static_call_module_nb);
+#endif
+}
+early_initcall(static_call_init);
--

2.17.2

#4 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Josh Poimboeuf
Quoted 37 lines · Josh Poimboeuf

Add a static call infrastructure. Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.

This code is heavily inspired by the jump label code (aka "static jumps"), as some of the concepts are very similar.

There are three implementations, depending on arch support:

1) optimized: patched call sites (CONFIG_HAVE_STATIC_CALL_OPTIMIZED)
2) unoptimized: patched trampolines (CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)
3) basic function pointers

For more details, see the comments in include/linux/static_call.h.

---
 arch/Kconfig                      |   6 +
 include/asm-generic/vmlinux.lds.h |  11 ++
 include/linux/module.h            |  10 +
 include/linux/static_call.h       | 186 +++++++++++++++++++
 include/linux/static_call_types.h |  19 ++
 kernel/Makefile                   |   1 +
 kernel/module.c                   |   5 +
 kernel/static_call.c              | 297 ++++++++++++++++++++++++++++++
 8 files changed, 535 insertions(+)
 create mode 100644 include/linux/static_call.h
 create mode 100644 include/linux/static_call_types.h
 create mode 100644 kernel/static_call.c
kernel/static_call.c new file +0 -0
new file mode 100644
index 000000000000..599ebc6fc4f1

...

Quoted 27 lines
kernel/static_call.c new file +27 -0
+static void __init static_call_init(void)
+{
+       struct static_call_site *start = __start_static_call_sites;
+       struct static_call_site *stop  = __stop_static_call_sites;
+       struct static_call_site *site;
+
+       if (start == stop) {
+               pr_warn("WARNING: empty static call table\n");
+               return;
+       }
+
+       cpus_read_lock();
+       static_call_lock();
+
+       static_call_sort_entries(start, stop);
+
+       for (site = start; site < stop; site++) {
+               struct static_call_key *key = static_call_key(site);
+               unsigned long addr = static_call_addr(site);
+
+               if (list_empty(&key->site_mods)) {
+                       struct static_call_mod *mod;
+
+                       mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+                       if (!mod) {
+                               WARN(1, "Failed to allocate memory for static calls");
+                               return;

kernel/static_call.c

Quoted 10 lines
kernel/static_call.c new file +10 -0
+                       }
+
+                       mod->sites = site;
+                       list_add_tail(&mod->list, &key->site_mods);
+
+                       /*
+                        * The trampoline should no longer be used.  Poison it
+                        * it with a BUG() to catch any stray callers.
+                        */
+                       arch_static_call_poison_tramp(addr);

This patches the wrong thing: the trampoline is at key->func not addr.

However, patching it here means we poison it before all users are patched. I added this on top

kernel/static_call.c +3 -1
index 599ebc6fc4f1..d9562329bec6 100644
@@ -248,6 +248,7 @@ static void __init static_call_init(void)
        struct static_call_site *start = __start_static_call_sites;
        struct static_call_site *stop  = __stop_static_call_sites;
        struct static_call_site *site;
+       struct static_call_key *prev_key = NULL;
 
        if (start == stop) {
                pr_warn("WARNING: empty static call table\n");
@@ -279,7 +280,9 @@ static void __init static_call_init(void)
                         * The trampoline should no longer be used.  Poison it
                         * it with a BUG() to catch any stray callers.
                         */
-                       arch_static_call_poison_tramp(addr);
+                       if (prev_key)
+

arch_static_call_poison_tramp((unsigned long)prev_key->func);

+                       prev_key = key;
                }

                arch_static_call_transform(addr, key->func);
Quoted 18 lines
kernel/static_call.c new file +15 -1
+               }
+
+               arch_static_call_transform(addr, key->func);
+       }
+
+       static_call_initialized = true;
+
+       static_call_unlock();
+       cpus_read_unlock();
+
+#ifdef CONFIG_MODULES
+       register_module_notifier(&static_call_module_nb);
+#endif
+}
+early_initcall(static_call_init);
--

2.17.2

#33 lore raw
In reply to Ingo Molnar
Quoted 25 lines · Ingo Molnar
Quoted 22 lines · Josh Poimboeuf

These patches are related to two similar patch sets from Ard and Steve:

- https://lkml.kernel.org/r/[email protected]
- https://lkml.kernel.org/r/[email protected]

The code is also heavily inspired by the jump label code, as some of the concepts are very similar.

There are three separate implementations, depending on what the arch supports:

1) CONFIG_HAVE_STATIC_CALL_OPTIMIZED: patched call sites - requires objtool and a small amount of arch code

2) CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED: patched trampolines - requires a small amount of arch code

3) If no arch support, fall back to regular function pointers

TODO:

Quoted 8 lines
Quoted 3 lines

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

I implemented the GCC plugin approach here for arm64

https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=static-calls

That implements both the unoptimized and the optimized versions.

I do take your point about GCC and other compilers, but on arm64 we don't have a lot of choice.

As far as I can tell, the GCC plugin is generic (i.e., it does not rely on any ARM specific passes, but obviously, this requires a *lot* of testing and validation to be taken seriously.

Quoted 25 lines
Quoted 2 lines

- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?

Paravirt patching, as you mention in your later reply?

Quoted 1 line

- Find some actual users of the interfaces (tracepoints? crypto?)

I'd be very happy with a demonstrated paravirt optimization already - i.e. seeing the before/after effect on the vmlinux with an x86 distro config.

All major Linux distributions enable CONFIG_PARAVIRT=y and CONFIG_PARAVIRT_XXL=y on x86 at the moment, so optimizing it away as much as possible in the 99.999% cases where it's not used is a primary concern.

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

Thanks,

Ingo

#38 lore raw
In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar
Quoted 3 lines

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.

The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).

Quoted 7 lines
Quoted 2 lines

- Does this feature have much value without retpolines? If not, should we make it depend on retpolines somehow?

Paravirt patching, as you mention in your later reply?

Quoted 1 line

- Find some actual users of the interfaces (tracepoints? crypto?)

Quoted 8 lines

I'd be very happy with a demonstrated paravirt optimization already - i.e. seeing the before/after effect on the vmlinux with an x86 distro config.

All major Linux distributions enable CONFIG_PARAVIRT=y and CONFIG_PARAVIRT_XXL=y on x86 at the moment, so optimizing it away as much as possible in the 99.999% cases where it's not used is a primary concern.

For paravirt, I was thinking of it as more of a cleanup than an optimization. The paravirt patching code already replaces indirect branches with direct ones -- see paravirt_patch_default().

Though it *would* reduce the instruction footprint a bit, as the 7-byte indirect calls (later patched to 5-byte direct + 2-byte nop) would instead be 5-byte direct calls to begin with.

Quoted 5 lines

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

With retpolines, and with tracepoints enabled, it's definitely a major win. Steve measured an 8.9% general slowdown on hackbench caused by retpolines.

But with tracepoints disabled, I believe static jumps are used, which already minimizes the impact on hot paths.

Signature · 1 line
Josh
#3 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Ard Biesheuvel
Quoted 14 lines · Ard Biesheuvel

Hi Josh,

Thanks a lot for looking into this.

Quoted 9 lines · Josh Poimboeuf

Add a static call infrastructure. Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.

This code is heavily inspired by the jump label code (aka "static jumps"), as some of the concepts are very similar.

Quoted 8 lines
Quoted 4 lines

There are three implementations, depending on arch support:

1) optimized: patched call sites (CONFIG_HAVE_STATIC_CALL_OPTIMIZED)
2) unoptimized: patched trampolines (CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)

Could we move to an idiom like inline/out of line? The unoptimized variant may be perfectly adequate for many arches, and 'unoptimized' sounds like there is something wrong with it.

Yeah, inline/out-of-line would be better. It's both more descriptive and less derogatory :-)

Quoted 19 lines
Quoted 5 lines

+ * Usage example:
+ *

+ *   # Start with the following functions (with identical prototypes):
+ *   int func_a(int arg1, int arg2);
+ *   int func_b(int arg1, int arg2);
Quoted 13 lines

+ *
+ * # Define a 'my_key' reference, associated with func_a() by default
+ * DEFINE_STATIC_CALL(my_key, func_a);
+ *
+ * # Call func_a()
+ * static_call(my_key, arg1, arg2);
+ *
+ * # Update 'my_key' to point to func_b()
+ * static_call_update(my_key, func_b);
+ *
+ * # Call func_b()
+ * static_call(my_key, arg1, arg2);
+ *

Quoted 3 lines

Any way we can revert to the default implementation? That would be useful for, e.g., unloading modules that provide an alternative implementation.

I saw your original implementation had a "reset to default", but from what I can tell, most users wouldn't need that.

Assuming the caller knows what the original dest function was, can't they just call static_call_update(my_key, orig_func) directly?

Quoted 36 lines
Quoted 8 lines
include/linux/static_call_types.h new file +2 -0
new file mode 100644
index 000000000000..7dd4b3d7ec2b
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _STATIC_CALL_TYPES_H

include/linux/static_call_types.h line 12

Quoted 10 lines
include/linux/static_call_types.h new file +10 -0
+#define _STATIC_CALL_TYPES_H
+
+#include <linux/stringify.h>
+
+#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_
+#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX)
+
+#define STATIC_CALL_TRAMP(key) STATIC_CALL_TRAMP_PREFIX##key
+#define STATIC_CALL_TRAMP_STR(key) __stringify(STATIC_CALL_TRAMP(key))
+

I needed to apply

include/linux/static_call_types.h +1 -1
index 7dd4b3d7ec2b..6859b208de6e 100644
@@ -7,7 +7,7 @@
 #define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_
 #define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX)
 
-#define STATIC_CALL_TRAMP(key) STATIC_CALL_TRAMP_PREFIX##key
+#define STATIC_CALL_TRAMP(key) __PASTE(STATIC_CALL_TRAMP_PREFIX, key)
 #define STATIC_CALL_TRAMP_STR(key) __stringify(STATIC_CALL_TRAMP(key))
 
 /* The static call site table is created by objtool. */
Quoted 5 lines

or I end up with

0000000000000000 <STATIC_CALL_TRAMP_PREFIXfoo>:

   0: 14000000 b a8 <foo_a>
   4: d503201f nop

Ha, oops. That was part of a last minute cleanup to share the macros with objtool.

Signature · 1 line
Josh
#5 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Ard Biesheuvel
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15 -0
+       for (site = start; site < stop; site++) {
+               struct static_call_key *key = static_call_key(site);
+               unsigned long addr = static_call_addr(site);
+
+               if (list_empty(&key->site_mods)) {
+                       struct static_call_mod *mod;
+
+                       mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+                       if (!mod) {
+                               WARN(1, "Failed to allocate memory for static calls");
+                               return;
+                       }
+
+                       mod->sites = site;
+                       list_add_tail(&mod->list, &key->site_mods);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+                       /*
+                        * The trampoline should no longer be used.  Poison it
+                        * it with a BUG() to catch any stray callers.
+                        */
+                       arch_static_call_poison_tramp(addr);

This patches the wrong thing: the trampoline is at key->func not addr.

If you look at the x86 implementation, it actually does poison the trampoline.

The address of the trampoline isn't actually known here. key->func isn't the trampoline address; it's the destination func address.

So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).

The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into arch_static_call_poison_tramp().

Quoted 26 lines

However, patching it here means we poison it before all users are patched. I added this on top

kernel/static_call.c +4 -1
index 599ebc6fc4f1..d9562329bec6 100644
@@ -248,6 +248,7 @@ static void __init static_call_init(void)
        struct static_call_site *start = __start_static_call_sites;
        struct static_call_site *stop  = __stop_static_call_sites;
        struct static_call_site *site;
+       struct static_call_key *prev_key = NULL;
 
        if (start == stop) {
                pr_warn("WARNING: empty static call table\n");
@@ -279,7 +280,9 @@ static void __init static_call_init(void)
                         * The trampoline should no longer be used.  Poison it
                         * it with a BUG() to catch any stray callers.
                         */
-                       arch_static_call_poison_tramp(addr);
+                       if (prev_key)
+
 arch_static_call_poison_tramp((unsigned long)prev_key->func);
+                       prev_key = key;
                }
 
Quoted 1 line
                arch_static_call_transform(addr, key->func);

While it does indeed poison the trampoline before all users are patched, I had been thinking that it didn't really matter because this is before the other CPUs have been booted.

But I believe interrupts are enabled at this point during the boot, so it would indeed be wise to poison it afterwards, in case an irq handler makes a static call.

Signature · 1 line
Josh
#6 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

Quoted 31 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15 -0
+       for (site = start; site < stop; site++) {
+               struct static_call_key *key = static_call_key(site);
+               unsigned long addr = static_call_addr(site);
+
+               if (list_empty(&key->site_mods)) {
+                       struct static_call_mod *mod;
+
+                       mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+                       if (!mod) {
+                               WARN(1, "Failed to allocate memory for static calls");
+                               return;
+                       }
+
+                       mod->sites = site;
+                       list_add_tail(&mod->list, &key->site_mods);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+                       /*
+                        * The trampoline should no longer be used.  Poison it
+                        * it with a BUG() to catch any stray callers.
+                        */
+                       arch_static_call_poison_tramp(addr);

This patches the wrong thing: the trampoline is at key->func not addr.

If you look at the x86 implementation, it actually does poison the trampoline.

The address of the trampoline isn't actually known here. key->func isn't the trampoline address; it's the destination func address.

Quoted 7 lines

So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).

The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into arch_static_call_poison_tramp().

Ah right, so I am basically missing a dereference in my arch_static_call_poison_tramp() code if this breaks.

Quoted 32 lines
Quoted 26 lines

However, patching it here means we poison it before all users are patched. I added this on top

kernel/static_call.c +4 -1
index 599ebc6fc4f1..d9562329bec6 100644
@@ -248,6 +248,7 @@ static void __init static_call_init(void)
        struct static_call_site *start = __start_static_call_sites;
        struct static_call_site *stop  = __stop_static_call_sites;
        struct static_call_site *site;
+       struct static_call_key *prev_key = NULL;
 
        if (start == stop) {
                pr_warn("WARNING: empty static call table\n");
@@ -279,7 +280,9 @@ static void __init static_call_init(void)
                         * The trampoline should no longer be used.  Poison it
                         * it with a BUG() to catch any stray callers.
                         */
-                       arch_static_call_poison_tramp(addr);
+                       if (prev_key)
+
 arch_static_call_poison_tramp((unsigned long)prev_key->func);
+                       prev_key = key;
                }
 
Quoted 1 line
                arch_static_call_transform(addr, key->func);

While it does indeed poison the trampoline before all users are patched, I had been thinking that it didn't really matter because this is before the other CPUs have been booted.

Quoted 4 lines

But I believe interrupts are enabled at this point during the boot, so it would indeed be wise to poison it afterwards, in case an irq handler makes a static call.

And kmalloc(GFP_KERNEL) itself could cascade into lots of other code as well.

#46 lore raw
In reply to Ingo Molnar
Quoted 7 lines · Ingo Molnar

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

For tracing, we'd want static_call_set_to_nop() or something like that, right?

--Andy

#34 lore raw
In reply to Ard Biesheuvel
Quoted 35 lines · Ard Biesheuvel
Quoted 25 lines · Ingo Molnar
Quoted 22 lines · Josh Poimboeuf

These patches are related to two similar patch sets from Ard and Steve:

- https://lkml.kernel.org/r/[email protected]
- https://lkml.kernel.org/r/[email protected]

The code is also heavily inspired by the jump label code, as some of the concepts are very similar.

There are three separate implementations, depending on what the arch supports:

1) CONFIG_HAVE_STATIC_CALL_OPTIMIZED: patched call sites - requires objtool and a small amount of arch code

2) CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED: patched trampolines - requires a small amount of arch code

3) If no arch support, fall back to regular function pointers

TODO:

Quoted 8 lines
Quoted 3 lines

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

Quoted 5 lines

I implemented the GCC plugin approach here for arm64

https://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux.git/log/?h=static-calls

That implements both the unoptimized and the optimized versions.

Nice! That was fast :-)

Quoted 6 lines

I do take your point about GCC and other compilers, but on arm64 we don't have a lot of choice.

As far as I can tell, the GCC plugin is generic (i.e., it does not rely on any ARM specific passes, but obviously, this requires a *lot* of testing and validation to be taken seriously.

Yeah. I haven't had a chance to try your plugin on x86 yet, but in theory it should be arch-independent.

Signature · 1 line
Josh
#47 lore raw
In reply to Andy Lutomirski
Quoted 10 lines · Andy Lutomirski
Quoted 7 lines · Ingo Molnar

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

For tracing, we'd want static_call_set_to_nop() or something like that, right?

Are we talking about tracepoints? Or ftrace?

Signature · 1 line
Josh
#48 lore raw
Quoted 10 lines · Josh Poimboeuf
Quoted 9 lines · Andy Lutomirski
Quoted 7 lines · Ingo Molnar

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

Quoted 3 lines
Quoted 1 line

For tracing, we'd want static_call_set_to_nop() or something like that, right?

Are we talking about tracepoints? Or ftrace?

Since ftrace changes calls to nops, and vice versa, I assume you meant ftrace. I don't think ftrace is a good candidate for this, as it's inherently more flexible than this API would reasonably allow.

Signature · 1 line
Josh
#7 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Ard Biesheuvel
Quoted 35 lines · Ard Biesheuvel
Quoted 34 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15 -0
+       for (site = start; site < stop; site++) {
+               struct static_call_key *key = static_call_key(site);
+               unsigned long addr = static_call_addr(site);
+
+               if (list_empty(&key->site_mods)) {
+                       struct static_call_mod *mod;
+
+                       mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+                       if (!mod) {
+                               WARN(1, "Failed to allocate memory for static calls");
+                               return;
+                       }
+
+                       mod->sites = site;
+                       list_add_tail(&mod->list, &key->site_mods);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+                       /*
+                        * The trampoline should no longer be used.  Poison it
+                        * it with a BUG() to catch any stray callers.
+                        */
+                       arch_static_call_poison_tramp(addr);

This patches the wrong thing: the trampoline is at key->func not addr.

If you look at the x86 implementation, it actually does poison the trampoline.

The address of the trampoline isn't actually known here. key->func isn't the trampoline address; it's the destination func address.

So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).

Quoted 8 lines
Quoted 4 lines

The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into arch_static_call_poison_tramp().

Ah right, so I am basically missing a dereference in my arch_static_call_poison_tramp() code if this breaks.

Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.

#8 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

Quoted 41 lines · Ard Biesheuvel
Quoted 40 lines · Ard Biesheuvel
Quoted 38 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15 -0
+       for (site = start; site < stop; site++) {
+               struct static_call_key *key = static_call_key(site);
+               unsigned long addr = static_call_addr(site);
+
+               if (list_empty(&key->site_mods)) {
+                       struct static_call_mod *mod;
+
+                       mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+                       if (!mod) {
+                               WARN(1, "Failed to allocate memory for static calls");
+                               return;
+                       }
+
+                       mod->sites = site;
+                       list_add_tail(&mod->list, &key->site_mods);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+                       /*
+                        * The trampoline should no longer be used.  Poison it
+                        * it with a BUG() to catch any stray callers.
+                        */
+                       arch_static_call_poison_tramp(addr);

This patches the wrong thing: the trampoline is at key->func not addr.

If you look at the x86 implementation, it actually does poison the trampoline.

The address of the trampoline isn't actually known here. key->func isn't the trampoline address; it's the destination func address.

So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).

The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into arch_static_call_poison_tramp().

Quoted 7 lines
Quoted 3 lines

Ah right, so I am basically missing a dereference in my arch_static_call_poison_tramp() code if this breaks.

Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.

Ok, but doesn't that defeat the purpose of the inline approach?

Signature · 1 line
Josh
#9 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

Quoted 46 lines · Josh Poimboeuf
Quoted 45 lines · Ard Biesheuvel
Quoted 43 lines · Ard Biesheuvel
Quoted 31 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15 -0
+       for (site = start; site < stop; site++) {
+               struct static_call_key *key = static_call_key(site);
+               unsigned long addr = static_call_addr(site);
+
+               if (list_empty(&key->site_mods)) {
+                       struct static_call_mod *mod;
+
+                       mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+                       if (!mod) {
+                               WARN(1, "Failed to allocate memory for static calls");
+                               return;
+                       }
+
+                       mod->sites = site;
+                       list_add_tail(&mod->list, &key->site_mods);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+                       /*
+                        * The trampoline should no longer be used.  Poison it
+                        * it with a BUG() to catch any stray callers.
+                        */
+                       arch_static_call_poison_tramp(addr);

This patches the wrong thing: the trampoline is at key->func not addr.

If you look at the x86 implementation, it actually does poison the trampoline.

The address of the trampoline isn't actually known here. key->func isn't the trampoline address; it's the destination func address.

Quoted 7 lines

So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).

The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into arch_static_call_poison_tramp().

Ah right, so I am basically missing a dereference in my arch_static_call_poison_tramp() code if this breaks.

Quoted 6 lines
Quoted 3 lines

Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.

Ok, but doesn't that defeat the purpose of the inline approach?

It does. But this only occurs when a module is loaded far away, and this will only happen if you have 2 GB range KASLR enabled, or your 128 MB module region gets exhausted for some reason, so the majority of calls should use a single relative branch.

#15 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Josh Poimboeuf
Quoted 46 lines · Josh Poimboeuf
Quoted 45 lines · Ard Biesheuvel
Quoted 43 lines · Ard Biesheuvel
Quoted 31 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15 -0
+       for (site = start; site < stop; site++) {
+               struct static_call_key *key = static_call_key(site);
+               unsigned long addr = static_call_addr(site);
+
+               if (list_empty(&key->site_mods)) {
+                       struct static_call_mod *mod;
+
+                       mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+                       if (!mod) {
+                               WARN(1, "Failed to allocate memory for static calls");
+                               return;
+                       }
+
+                       mod->sites = site;
+                       list_add_tail(&mod->list, &key->site_mods);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+                       /*
+                        * The trampoline should no longer be used.  Poison it
+                        * it with a BUG() to catch any stray callers.
+                        */
+                       arch_static_call_poison_tramp(addr);

This patches the wrong thing: the trampoline is at key->func not addr.

If you look at the x86 implementation, it actually does poison the trampoline.

The address of the trampoline isn't actually known here. key->func isn't the trampoline address; it's the destination func address.

Quoted 7 lines

So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).

The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into arch_static_call_poison_tramp().

Ah right, so I am basically missing a dereference in my arch_static_call_poison_tramp() code if this breaks.

Quoted 5 lines
Quoted 3 lines

Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.

Ok, but doesn't that defeat the purpose of the inline approach?

Or are you only going to use the trampoline for out-of-range calls, otherwise just do direct calls?

Signature · 1 line
Josh
#10 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Ard Biesheuvel
Quoted 51 lines · Ard Biesheuvel
Quoted 50 lines · Josh Poimboeuf
Quoted 48 lines · Ard Biesheuvel
Quoted 35 lines · Ard Biesheuvel
Quoted 34 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15 -0
+       for (site = start; site < stop; site++) {
+               struct static_call_key *key = static_call_key(site);
+               unsigned long addr = static_call_addr(site);
+
+               if (list_empty(&key->site_mods)) {
+                       struct static_call_mod *mod;
+
+                       mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+                       if (!mod) {
+                               WARN(1, "Failed to allocate memory for static calls");
+                               return;
+                       }
+
+                       mod->sites = site;
+                       list_add_tail(&mod->list, &key->site_mods);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+                       /*
+                        * The trampoline should no longer be used.  Poison it
+                        * it with a BUG() to catch any stray callers.
+                        */
+                       arch_static_call_poison_tramp(addr);

This patches the wrong thing: the trampoline is at key->func not addr.

If you look at the x86 implementation, it actually does poison the trampoline.

The address of the trampoline isn't actually known here. key->func isn't the trampoline address; it's the destination func address.

So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).

Quoted 8 lines
Quoted 4 lines

The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into arch_static_call_poison_tramp().

Ah right, so I am basically missing a dereference in my arch_static_call_poison_tramp() code if this breaks.

Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.

Quoted 7 lines
Quoted 2 lines

Ok, but doesn't that defeat the purpose of the inline approach?

It does. But this only occurs when a module is loaded far away, and this will only happen if you have 2 GB range KASLR enabled, or your 128 MB module region gets exhausted for some reason, so the majority of calls should use a single relative branch.

Makes sense. Do you also account for the possibility that the original call emitted by GCC was far away and thus used the PLT?

Signature · 1 line
Josh
#11 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

Quoted 55 lines · Josh Poimboeuf
Quoted 54 lines · Ard Biesheuvel
Quoted 52 lines · Josh Poimboeuf
Quoted 41 lines · Ard Biesheuvel
Quoted 40 lines · Ard Biesheuvel
Quoted 38 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15 -0
+       for (site = start; site < stop; site++) {
+               struct static_call_key *key = static_call_key(site);
+               unsigned long addr = static_call_addr(site);
+
+               if (list_empty(&key->site_mods)) {
+                       struct static_call_mod *mod;
+
+                       mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+                       if (!mod) {
+                               WARN(1, "Failed to allocate memory for static calls");
+                               return;
+                       }
+
+                       mod->sites = site;
+                       list_add_tail(&mod->list, &key->site_mods);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+                       /*
+                        * The trampoline should no longer be used.  Poison it
+                        * it with a BUG() to catch any stray callers.
+                        */
+                       arch_static_call_poison_tramp(addr);

This patches the wrong thing: the trampoline is at key->func not addr.

If you look at the x86 implementation, it actually does poison the trampoline.

The address of the trampoline isn't actually known here. key->func isn't the trampoline address; it's the destination func address.

So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).

The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into arch_static_call_poison_tramp().

Quoted 7 lines
Quoted 3 lines

Ah right, so I am basically missing a dereference in my arch_static_call_poison_tramp() code if this breaks.

Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.

Ok, but doesn't that defeat the purpose of the inline approach?

Quoted 8 lines
Quoted 4 lines

It does. But this only occurs when a module is loaded far away, and this will only happen if you have 2 GB range KASLR enabled, or your 128 MB module region gets exhausted for some reason, so the majority of calls should use a single relative branch.

Makes sense. Do you also account for the possibility that the original call emitted by GCC was far away and thus used the PLT?

That doesn't really happen. vmlinux is never over 128 MB, and the modules are only partially linked, so the linker never gets to the stage where it needs to emit veneers.

It's a bit fiddly since inline and out-of-line both use arch_static_call_transform(), but what I need to do is basically:

- for out-of-line, the trampoline needs to be patched into a movn/movk/movk/br sequence if the target is too far
- for inline, the trampoline itself needs to be patched from adrp/ldr/br (which does a load and a indirect branch) to movn/movk/movk/br (which uses immediates), and the call sites need to be patched into calls to the veneer if the target is out of range.

So arch_static_call_transform() needs to know where the trampoline is for this use case, so could we perhaps add a 'void *orig' in the key struct that keeps track of the original value of 'addr' ?

#12 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

Quoted 79 lines · Ard Biesheuvel
Quoted 55 lines · Josh Poimboeuf
Quoted 54 lines · Ard Biesheuvel
Quoted 52 lines · Josh Poimboeuf
Quoted 41 lines · Ard Biesheuvel
Quoted 40 lines · Ard Biesheuvel
Quoted 38 lines · Josh Poimboeuf
Quoted 15 lines · Ard Biesheuvel
Quoted 15 lines
(quoted diff) +15 -0
+       for (site = start; site < stop; site++) {
+               struct static_call_key *key = static_call_key(site);
+               unsigned long addr = static_call_addr(site);
+
+               if (list_empty(&key->site_mods)) {
+                       struct static_call_mod *mod;
+
+                       mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+                       if (!mod) {
+                               WARN(1, "Failed to allocate memory for static calls");
+                               return;
+                       }
+
+                       mod->sites = site;
+                       list_add_tail(&mod->list, &key->site_mods);
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+
+                       /*
+                        * The trampoline should no longer be used.  Poison it
+                        * it with a BUG() to catch any stray callers.
+                        */
+                       arch_static_call_poison_tramp(addr);

This patches the wrong thing: the trampoline is at key->func not addr.

If you look at the x86 implementation, it actually does poison the trampoline.

The address of the trampoline isn't actually known here. key->func isn't the trampoline address; it's the destination func address.

So instead I passed the address of the call instruction. The arch code then reads the instruction to find the callee (the trampoline).

The code is a bit confusing. To make it more obvious, maybe we should add another arch function to read the call destination. Then this code can pass that into arch_static_call_poison_tramp().

Quoted 7 lines
Quoted 3 lines

Ah right, so I am basically missing a dereference in my arch_static_call_poison_tramp() code if this breaks.

Could we call it 'defuse' rather than 'poision'? On arm64, we will need to keep it around to bounce function calls that are out of range, and replace it with a PLT sequence.

Ok, but doesn't that defeat the purpose of the inline approach?

Quoted 8 lines
Quoted 4 lines

It does. But this only occurs when a module is loaded far away, and this will only happen if you have 2 GB range KASLR enabled, or your 128 MB module region gets exhausted for some reason, so the majority of calls should use a single relative branch.

Makes sense. Do you also account for the possibility that the original call emitted by GCC was far away and thus used the PLT?

That doesn't really happen. vmlinux is never over 128 MB, and the modules are only partially linked, so the linker never gets to the stage where it needs to emit veneers.

It's a bit fiddly since inline and out-of-line both use arch_static_call_transform(), but what I need to do is basically:

- for out-of-line, the trampoline needs to be patched into a movn/movk/movk/br sequence if the target is too far
- for inline, the trampoline itself needs to be patched from adrp/ldr/br (which does a load and a indirect branch) to movn/movk/movk/br (which uses immediates), and the call sites need to be patched into calls to the veneer if the target is out of range.

Quoted 3 lines

So arch_static_call_transform() needs to know where the trampoline is for this use case, so could we perhaps add a 'void *orig' in the key struct that keeps track of the original value of 'addr' ?

... and pass it to arch_static_call_transform

#16 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Josh Poimboeuf
Quoted 6 lines · On Thu, 8 Nov 2018 15:15:51 -0600 Josh Poimboeuf <[email protected]> wrote:

Add a static call infrastructure. Static calls use code patching to hard-code function pointers into direct branch instructions. They give the flexibility of function pointers, but with improved performance. This is especially important for cases where retpolines would otherwise be used, as retpolines can significantly impact performance.

Quoted 6 lines

This code is heavily inspired by the jump label code (aka "static jumps"), as some of the concepts are very similar.

There are three implementations, depending on arch support:

1) optimized: patched call sites (CONFIG_HAVE_STATIC_CALL_OPTIMIZED)

Thanks for doing this.

Quoted 1 line

2) unoptimized: patched trampolines (CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)

#2 looks like my approach.

Quoted 24 lines

3) basic function pointers

For more details, see the comments in include/linux/static_call.h.

---
 arch/Kconfig                      |   6 +
 include/asm-generic/vmlinux.lds.h |  11 ++
 include/linux/module.h            |  10 +
 include/linux/static_call.h       | 186 +++++++++++++++++++
 include/linux/static_call_types.h |  19 ++
 kernel/Makefile                   |   1 +
 kernel/module.c                   |   5 +
 kernel/static_call.c              | 297 ++++++++++++++++++++++++++++++
 8 files changed, 535 insertions(+)
 create mode 100644 include/linux/static_call.h
 create mode 100644 include/linux/static_call_types.h
 create mode 100644 kernel/static_call.c
arch/Kconfig +0 -0
index e1e540ffa979..9657ad5a80d1 100644

arch/Kconfig config HAVE_ARCH_PREL32_RELOCATIONS · line 887

Quoted 9 lines
arch/Kconfig +6 -0
@@ -879,6 +879,12 @@ config HAVE_ARCH_PREL32_RELOCATIONS
 	  architectures, and don't require runtime relocation on relocatable
 	  kernels.
 
+config HAVE_STATIC_CALL_OPTIMIZED
+	bool
+
+config HAVE_STATIC_CALL_UNOPTIMIZED
+	bool
+

Let's also have

config HAVE_STATIC_CALL

	def_bool Y
	depends on HAVE_STATIC_CALL_OPTIMIZED || HAVE_STATIC_CALL_UNOPTIMIZED
Quoted 137 lines
arch/Kconfig +0 -0
 source "kernel/gcov/Kconfig"
 
 source "scripts/gcc-plugins/Kconfig"
include/asm-generic/vmlinux.lds.h +11 -0
index 3d7a6a9c2370..45d8ced28bd0 100644
@@ -320,6 +320,7 @@
 	__start_ro_after_init = .;					\
 	*(.data..ro_after_init)						\
 	JUMP_TABLE_DATA							\
+	STATIC_CALL_SITES						\
 	__end_ro_after_init = .;
 #endif
 
@@ -725,6 +726,16 @@
 #define BUG_TABLE
 #endif
 
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+#define STATIC_CALL_SITES						\
+	. = ALIGN(8);							\
+	__start_static_call_sites = .;					\
+	KEEP(*(.static_call_sites))					\
+	__stop_static_call_sites = .;
+#else
+#define STATIC_CALL_SITES
+#endif
+
 #ifdef CONFIG_UNWINDER_ORC
 #define ORC_UNWIND_TABLE						\
 	. = ALIGN(4);							\
include/linux/module.h +10 -0
index fce6b4335e36..91baf181247a 100644
@@ -21,6 +21,7 @@
 #include <linux/rbtree_latch.h>
 #include <linux/error-injection.h>
 #include <linux/tracepoint-defs.h>
+#include <linux/static_call_types.h>
 
 #include <linux/percpu.h>
 #include <asm/module.h>
@@ -450,6 +451,10 @@ struct module {
 	unsigned int num_ftrace_callsites;
 	unsigned long *ftrace_callsites;
 #endif
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+	int num_static_call_sites;
+	struct static_call_site *static_call_sites;
+#endif
 
 #ifdef CONFIG_LIVEPATCH
 	bool klp; /* Is this a livepatch module? */
@@ -682,6 +687,11 @@ static inline bool is_module_text_address(unsigned long addr)
 	return false;
 }
 
+static inline bool within_module_init(unsigned long addr, const struct module *mod)
+{
+	return false;
+}
+
 /* Get/put a kernel symbol (calls should be symmetric) */
 #define symbol_get(x) ({ extern typeof(x) x __attribute__((weak)); &(x); })
 #define symbol_put(x) do { } while (0)
include/linux/static_call.h new file +64 -0
new file mode 100644
index 000000000000..b22987057cf1
@@ -0,0 +1,186 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_STATIC_CALL_H
+#define _LINUX_STATIC_CALL_H
+
+/*
+ * Static call support
+ *
+ * Static calls use code patching to hard-code function pointers into direct
+ * branch instructions.  They give the flexibility of function pointers, but
+ * with improved performance.  This is especially important for cases where
+ * retpolines would otherwise be used, as retpolines can significantly impact
+ * performance.
+ *
+ *
+ * API overview:
+ *
+ *   DECLARE_STATIC_CALL(key, func);
+ *   DEFINE_STATIC_CALL(key, func);
+ *   static_call(key, args...);
+ *   static_call_update(key, func);
+ *
+ *
+ * Usage example:
+ *
+ *   # Start with the following functions (with identical prototypes):
+ *   int func_a(int arg1, int arg2);
+ *   int func_b(int arg1, int arg2);
+ *
+ *   # Define a 'my_key' reference, associated with func_a() by default
+ *   DEFINE_STATIC_CALL(my_key, func_a);
+ *
+ *   # Call func_a()
+ *   static_call(my_key, arg1, arg2);
+ *
+ *   # Update 'my_key' to point to func_b()
+ *   static_call_update(my_key, func_b);
+ *
+ *   # Call func_b()
+ *   static_call(my_key, arg1, arg2);
+ *
+ *
+ * Implementation details:
+ *
+ * There are three different implementations:
+ *
+ * 1) Optimized static calls (patched call sites)
+ *
+ *    This requires objtool, which detects all the static_call() sites and
+ *    annotates them in the '.static_call_sites' section.  By default, the call
+ *    sites will call into a temporary per-key trampoline which has an indirect
+ *    branch to the current destination function associated with the key.
+ *    During system boot (or module init), all call sites are patched to call
+ *    their destination functions directly.  Updates to a key will patch all
+ *    call sites associated with that key.
+ *
+ * 2) Unoptimized static calls (patched trampolines)
+ *
+ *    Each static_call() site calls into a permanent trampoline associated with
+ *    the key.  The trampoline has a direct branch to the default function.
+ *    Updates to a key will modify the direct branch in the key's trampoline.
+ *
+ * 3) Generic implementation
+ *
+ *    This is the default implementation if the architecture hasn't implemented

include/linux/static_call.h line 74

Quoted 10 lines
include/linux/static_call.h new file +10 -0
+ *    CONFIG_HAVE_STATIC_CALL_[UN]OPTIMIZED.  In this case, a basic
+ *    function pointer is used.
+ */
+
+#include <linux/types.h>
+#include <linux/cpu.h>
+#include <linux/static_call_types.h>
+
+#if defined(CONFIG_HAVE_STATIC_CALL_OPTIMIZED) || \
+    defined(CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)

Now we can have:

#ifdef CONFIG_HAVE_STATIC_CALL

Quoted 28 lines
include/linux/static_call.h new file +28 -0
+
+#include <asm/static_call.h>
+
+extern void arch_static_call_transform(unsigned long insn, void *dest);
+
+#endif /* CONFIG_HAVE_STATIC_CALL_[UN]OPTIMIZED */
+
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+/* Optimized implementation */
+
+struct static_call_mod {
+	struct list_head list;
+	struct module *mod; /* NULL means vmlinux */
+	struct static_call_site *sites;
+};
+
+struct static_call_key {
+	/*
+	 * This field should always be first because the trampolines expect it.
+	 * This points to the current call destination.
+	 */
+	void *func;
+
+	/*
+	 * List of modules (including vmlinux) and the call sites which need to
+	 * be patched for this key.
+	 */
+	struct list_head site_mods;

include/linux/static_call.h line 112

Quoted 10 lines
include/linux/static_call.h new file +10 -0
+};
+
+extern void __static_call_update(struct static_call_key *key, void *func);
+extern void arch_static_call_poison_tramp(unsigned long insn);
+
+#define DECLARE_STATIC_CALL(key, func)					\
+	extern struct static_call_key key;				\
+	extern typeof(func) STATIC_CALL_TRAMP(key);			\
+	/* Preserve the ELF symbol so objtool can access it: */		\
+	__ADDRESSABLE(key)

Does the __ADDRESSABLE(key) need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.

Quoted 221 lines
include/linux/static_call.h new file +74 -0
+
+#define DEFINE_STATIC_CALL(key, _func)					\
+	DECLARE_STATIC_CALL(key, _func);				\
+	struct static_call_key key = {					\
+		.func = _func,						\
+		.site_mods = LIST_HEAD_INIT(key.site_mods),		\
+	};								\
+	ARCH_STATIC_CALL_TEMPORARY_TRAMP(key)
+
+#define static_call(key, args...) STATIC_CALL_TRAMP(key)(args)
+
+#define static_call_update(key, func)					\
+({									\
+	BUILD_BUG_ON(!__same_type(typeof(func), typeof(STATIC_CALL_TRAMP(key)))); \
+	__static_call_update(&key, func);				\
+})
+
+#define EXPORT_STATIC_CALL(key)						\
+	EXPORT_SYMBOL(key);						\
+	EXPORT_SYMBOL(STATIC_CALL_TRAMP(key))
+
+#define EXPORT_STATIC_CALL_GPL(key)					\
+	EXPORT_SYMBOL_GPL(key);						\
+	EXPORT_SYMBOL_GPL(STATIC_CALL_TRAMP(key))
+
+
+#elif defined(CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)
+/* Unoptimized implementation */
+
+#define DECLARE_STATIC_CALL(key, func)					\
+	extern typeof(func) STATIC_CALL_TRAMP(key)
+
+#define DEFINE_STATIC_CALL(key, func)					\
+	DECLARE_STATIC_CALL(key, func);					\
+	ARCH_STATIC_CALL_TRAMP(key, func)
+
+#define static_call(key, args...) STATIC_CALL_TRAMP(key)(args)
+
+#define static_call_update(key, func)					\
+({									\
+	BUILD_BUG_ON(!__same_type(func, STATIC_CALL_TRAMP(key)));	\
+	cpus_read_lock();						\
+	arch_static_call_transform((unsigned long)STATIC_CALL_TRAMP(key),\
+				   func);				\
+	cpus_read_unlock();						\
+})
+
+#define EXPORT_STATIC_CALL(key)						\
+	EXPORT_SYMBOL(STATIC_CALL_TRAMP(key))
+
+#define EXPORT_STATIC_CALL_GPL(key)					\
+	EXPORT_SYMBOL_GPL(STATIC_CALL_TRAMP(key))
+
+
+#else /* Generic implementation */
+
+#define DECLARE_STATIC_CALL(key, func)					\
+	extern typeof(func) *key
+
+#define DEFINE_STATIC_CALL(key, func)					\
+	typeof(func) *key = func
+
+#define static_call(key, args...)					\
+	key(args)
+
+#define static_call_update(key, func)					\
+	WRITE_ONCE(key, func)
+
+#define EXPORT_STATIC_CALL(key) EXPORT_SYMBOL(key)
+#define EXPORT_STATIC_CALL_GPL(key) EXPORT_SYMBOL_GPL(key)
+
+#endif /* CONFIG_HAVE_STATIC_CALL_OPTIMIZED */
+
+#endif /* _LINUX_STATIC_CALL_H */
include/linux/static_call_types.h new file +19 -0
new file mode 100644
index 000000000000..7dd4b3d7ec2b
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _STATIC_CALL_TYPES_H
+#define _STATIC_CALL_TYPES_H
+
+#include <linux/stringify.h>
+
+#define STATIC_CALL_TRAMP_PREFIX ____static_call_tramp_
+#define STATIC_CALL_TRAMP_PREFIX_STR __stringify(STATIC_CALL_TRAMP_PREFIX)
+
+#define STATIC_CALL_TRAMP(key) STATIC_CALL_TRAMP_PREFIX##key
+#define STATIC_CALL_TRAMP_STR(key) __stringify(STATIC_CALL_TRAMP(key))
+
+/* The static call site table is created by objtool. */
+struct static_call_site {
+	s32 addr;
+	s32 key;
+};
+
+#endif /* _STATIC_CALL_TYPES_H */
kernel/Makefile +1 -0
index 7343b3a9bff0..888bb476ddaf 100644
@@ -103,6 +103,7 @@ obj-$(CONFIG_TRACEPOINTS) += trace/
 obj-$(CONFIG_IRQ_WORK) += irq_work.o
 obj-$(CONFIG_CPU_PM) += cpu_pm.o
 obj-$(CONFIG_BPF) += bpf/
+obj-$(CONFIG_HAVE_STATIC_CALL_OPTIMIZED) += static_call.o
 
 obj-$(CONFIG_PERF_EVENTS) += events/
 
kernel/module.c +5 -0
index 49a405891587..11124d991fcd 100644
@@ -3121,6 +3121,11 @@ static int find_module_sections(struct module *mod, struct load_info *info)
 	mod->ei_funcs = section_objs(info, "_error_injection_whitelist",
 					    sizeof(*mod->ei_funcs),
 					    &mod->num_ei_funcs);
+#endif
+#ifdef CONFIG_HAVE_STATIC_CALL_OPTIMIZED
+	mod->static_call_sites = section_objs(info, ".static_call_sites",
+					      sizeof(*mod->static_call_sites),
+					      &mod->num_static_call_sites);
 #endif
 	mod->extable = section_objs(info, "__ex_table",
 				    sizeof(*mod->extable), &mod->num_exentries);
kernel/static_call.c new file +88 -0
new file mode 100644
index 000000000000..599ebc6fc4f1
@@ -0,0 +1,297 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <linux/init.h>
+#include <linux/static_call.h>
+#include <linux/bug.h>
+#include <linux/smp.h>
+#include <linux/sort.h>
+#include <linux/slab.h>
+#include <linux/module.h>
+#include <linux/cpu.h>
+#include <linux/processor.h>
+#include <asm/sections.h>
+
+extern struct static_call_site __start_static_call_sites[],
+			       __stop_static_call_sites[];
+
+static bool static_call_initialized;
+
+/* mutex to protect key modules/sites */
+static DEFINE_MUTEX(static_call_mutex);
+
+static void static_call_lock(void)
+{
+	mutex_lock(&static_call_mutex);
+}
+
+static void static_call_unlock(void)
+{
+	mutex_unlock(&static_call_mutex);
+}
+
+static inline unsigned long static_call_addr(struct static_call_site *site)
+{
+	return (long)site->addr + (long)&site->addr;
+}
+
+static inline struct static_call_key *static_call_key(const struct static_call_site *site)
+{
+	return (struct static_call_key *)((long)site->key + (long)&site->key);
+}
+
+static int static_call_site_cmp(const void *_a, const void *_b)
+{
+	const struct static_call_site *a = _a;
+	const struct static_call_site *b = _b;
+	const struct static_call_key *key_a = static_call_key(a);
+	const struct static_call_key *key_b = static_call_key(b);
+
+	if (key_a < key_b)
+		return -1;
+
+	if (key_a > key_b)
+		return 1;
+
+	return 0;
+}
+
+static void static_call_site_swap(void *_a, void *_b, int size)
+{
+	long delta = (unsigned long)_a - (unsigned long)_b;
+	struct static_call_site *a = _a;
+	struct static_call_site *b = _b;
+	struct static_call_site tmp = *a;
+
+	a->addr = b->addr  - delta;
+	a->key  = b->key   - delta;
+
+	b->addr = tmp.addr + delta;
+	b->key  = tmp.key  + delta;
+}
+
+static inline void static_call_sort_entries(struct static_call_site *start,
+					    struct static_call_site *stop)
+{
+	sort(start, stop - start, sizeof(struct static_call_site),
+	     static_call_site_cmp, static_call_site_swap);
+}
+
+void __static_call_update(struct static_call_key *key, void *func)
+{
+	struct static_call_mod *mod;
+	struct static_call_site *site, *stop;
+
+	cpus_read_lock();
+	static_call_lock();
+
+	if (key->func == func)
+		goto done;
+

kernel/static_call.c line 98

Quoted 10 lines
kernel/static_call.c new file +10 -0
+	key->func = func;
+
+	/*
+	 * If called before init, leave the call sites unpatched for now.
+	 * In the meantime they'll continue to call the temporary trampoline.
+	 */
+	if (!static_call_initialized)
+		goto done;
+
+	list_for_each_entry(mod, &key->site_mods, list) {

Since I'm expecting a lot of sites, I'm wondering if we should just do this as an array, like I do with the ftrace call sites.

But this can be an enhancement for later. Let's focus on getting this working first.

kernel/static_call.c line 110

Quoted 12 lines
kernel/static_call.c new file +12 -0
+		if (!mod->sites) {
+			/*
+			 * This can happen if the static call key is defined in
+			 * a module which doesn't use it.
+			 */
+			continue;
+		}
+
+		stop = __stop_static_call_sites;
+
+#ifdef CONFIG_MODULES
+		if (mod->mod) {

BTW, "mod" is an unfortunate name.

Quoted 4 lines
kernel/static_call.c new file +4 -0
+			stop = mod->mod->static_call_sites +
+			       mod->mod->num_static_call_sites;
+		}
+#endif

kernel/static_call.c line 124

Quoted 10 lines
kernel/static_call.c new file +10 -0
+
+		for (site = mod->sites;
+		     site < stop && static_call_key(site) == key; site++) {
+			unsigned long addr = static_call_addr(site);
+
+			if (!mod->mod && init_section_contains((void *)addr, 1))
+				continue;
+			if (mod->mod && within_module_init(addr, mod->mod))
+				continue;
+

So what's the reason for skipping init calls?

-- Steve

Quoted 173 lines
kernel/static_call.c new file +173 -0
+			arch_static_call_transform(addr, func);
+		}
+	}
+
+done:
+	static_call_unlock();
+	cpus_read_unlock();
+}
+EXPORT_SYMBOL_GPL(__static_call_update);
+
+#ifdef CONFIG_MODULES
+
+static int static_call_add_module(struct module *mod)
+{
+	struct static_call_site *start = mod->static_call_sites;
+	struct static_call_site *stop = mod->static_call_sites +
+					mod->num_static_call_sites;
+	struct static_call_site *site;
+	struct static_call_key *key, *prev_key = NULL;
+	struct static_call_mod *static_call_mod;
+
+	if (start == stop)
+		return 0;
+
+	module_disable_ro(mod);
+	static_call_sort_entries(start, stop);
+	module_enable_ro(mod, false);
+
+	for (site = start; site < stop; site++) {
+		unsigned long addr = static_call_addr(site);
+
+		if (within_module_init(addr, mod))
+			continue;
+
+		key = static_call_key(site);
+		if (key != prev_key) {
+			prev_key = key;
+
+			static_call_mod = kzalloc(sizeof(*static_call_mod), GFP_KERNEL);
+			if (!static_call_mod)
+				return -ENOMEM;
+
+			static_call_mod->mod = mod;
+			static_call_mod->sites = site;
+			list_add_tail(&static_call_mod->list, &key->site_mods);
+
+			if (is_module_address((unsigned long)key)) {
+				/*
+				 * The trampoline should no longer be used.
+				 * Poison it it with a BUG() to catch any stray
+				 * callers.
+				 */
+				arch_static_call_poison_tramp(addr);
+			}
+		}
+
+		arch_static_call_transform(addr, key->func);
+	}
+
+	return 0;
+}
+
+static void static_call_del_module(struct module *mod)
+{
+	struct static_call_site *start = mod->static_call_sites;
+	struct static_call_site *stop = mod->static_call_sites +
+					mod->num_static_call_sites;
+	struct static_call_site *site;
+	struct static_call_key *key, *prev_key = NULL;
+	struct static_call_mod *static_call_mod;
+
+	for (site = start; site < stop; site++) {
+		key = static_call_key(site);
+		if (key == prev_key)
+			continue;
+		prev_key = key;
+
+		list_for_each_entry(static_call_mod, &key->site_mods, list) {
+			if (static_call_mod->mod == mod) {
+				list_del(&static_call_mod->list);
+				kfree(static_call_mod);
+				break;
+			}
+		}
+	}
+}
+
+static int static_call_module_notify(struct notifier_block *nb,
+				     unsigned long val, void *data)
+{
+	struct module *mod = data;
+	int ret = 0;
+
+	cpus_read_lock();
+	static_call_lock();
+
+	switch (val) {
+	case MODULE_STATE_COMING:
+		ret = static_call_add_module(mod);
+		if (ret) {
+			WARN(1, "Failed to allocate memory for static calls");
+			static_call_del_module(mod);
+		}
+		break;
+	case MODULE_STATE_GOING:
+		static_call_del_module(mod);
+		break;
+	}
+
+	static_call_unlock();
+	cpus_read_unlock();
+
+	return notifier_from_errno(ret);
+}
+
+static struct notifier_block static_call_module_nb = {
+	.notifier_call = static_call_module_notify,
+};
+
+#endif /* CONFIG_MODULES */
+
+static void __init static_call_init(void)
+{
+	struct static_call_site *start = __start_static_call_sites;
+	struct static_call_site *stop  = __stop_static_call_sites;
+	struct static_call_site *site;
+
+	if (start == stop) {
+		pr_warn("WARNING: empty static call table\n");
+		return;
+	}
+
+	cpus_read_lock();
+	static_call_lock();
+
+	static_call_sort_entries(start, stop);
+
+	for (site = start; site < stop; site++) {
+		struct static_call_key *key = static_call_key(site);
+		unsigned long addr = static_call_addr(site);
+
+		if (list_empty(&key->site_mods)) {
+			struct static_call_mod *mod;
+
+			mod = kzalloc(sizeof(*mod), GFP_KERNEL);
+			if (!mod) {
+				WARN(1, "Failed to allocate memory for static calls");
+				return;
+			}
+
+			mod->sites = site;
+			list_add_tail(&mod->list, &key->site_mods);
+
+			/*
+			 * The trampoline should no longer be used.  Poison it
+			 * it with a BUG() to catch any stray callers.
+			 */
+			arch_static_call_poison_tramp(addr);
+		}
+
+		arch_static_call_transform(addr, key->func);
+	}
+
+	static_call_initialized = true;
+
+	static_call_unlock();
+	cpus_read_unlock();
+
+#ifdef CONFIG_MODULES
+	register_module_notifier(&static_call_module_nb);
+#endif
+}
+early_initcall(static_call_init);
#49 lore raw
In reply to Josh Poimboeuf
Quoted 11 lines · On Fri, 9 Nov 2018 10:41:37 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 10 lines · Josh Poimboeuf
Quoted 9 lines · Andy Lutomirski
Quoted 7 lines · Ingo Molnar

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

For tracing, we'd want static_call_set_to_nop() or something like that, right?

Are we talking about tracepoints? Or ftrace?

Since ftrace changes calls to nops, and vice versa, I assume you meant ftrace. I don't think ftrace is a good candidate for this, as it's inherently more flexible than this API would reasonably allow.

Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.

-- Steve

#13 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Ard Biesheuvel
Quoted 10 lines · Ard Biesheuvel
Quoted 10 lines

It's a bit fiddly since inline and out-of-line both use arch_static_call_transform(), but what I need to do is basically:

- for out-of-line, the trampoline needs to be patched into a movn/movk/movk/br sequence if the target is too far
- for inline, the trampoline itself needs to be patched from adrp/ldr/br (which does a load and a indirect branch) to movn/movk/movk/br (which uses immediates), and the call sites need to be patched into calls to the veneer if the target is out of range.

Quoted 5 lines
Quoted 3 lines

So arch_static_call_transform() needs to know where the trampoline is for this use case, so could we perhaps add a 'void *orig' in the key struct that keeps track of the original value of 'addr' ?

... and pass it to arch_static_call_transform

Ok. So in both cases, you need both the call site address and the tramp address, right? We could add a 'tramp' pointer to the key struct -- I assume that's what you mean?

And also change

void arch_static_call_transform(unsigned long insn, void *dest)

to

void arch_static_call_transform(void *call_site, void *tramp, void *dest)

Signature · 1 line
Josh
#50 lore raw
In reply to Steven Rostedt
Quoted 27 lines · Steven Rostedt
Quoted 11 lines · On Fri, 9 Nov 2018 10:41:37 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 11 lines · Josh Poimboeuf
Quoted 10 lines · Andy Lutomirski
Quoted 8 lines · Ingo Molnar

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

For tracing, we'd want static_call_set_to_nop() or something like that, right?

Are we talking about tracepoints? Or ftrace?

Since ftrace changes calls to nops, and vice versa, I assume you meant ftrace. I don't think ftrace is a good candidate for this, as it's inherently more flexible than this API would reasonably allow.

Quoted 6 lines

Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.

Indeed.

Although I had assumed that tracepoints already had appropriate jump label magic.

#14 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Josh Poimboeuf
Quoted 15 lines · Josh Poimboeuf
Quoted 14 lines · Ard Biesheuvel
Quoted 13 lines

It's a bit fiddly since inline and out-of-line both use arch_static_call_transform(), but what I need to do is basically:

- for out-of-line, the trampoline needs to be patched into a movn/movk/movk/br sequence if the target is too far
- for inline, the trampoline itself needs to be patched from adrp/ldr/br (which does a load and a indirect branch) to movn/movk/movk/br (which uses immediates), and the call sites need to be patched into calls to the veneer if the target is out of range.

So arch_static_call_transform() needs to know where the trampoline is for this use case, so could we perhaps add a 'void *orig' in the key struct that keeps track of the original value of 'addr' ?

Quoted 6 lines
Quoted 1 line

... and pass it to arch_static_call_transform

Ok. So in both cases, you need both the call site address and the tramp address, right? We could add a 'tramp' pointer to the key struct -- I assume that's what you mean?

Yes.

Quoted 8 lines

And also change

void arch_static_call_transform(unsigned long insn, void *dest)

to

void arch_static_call_transform(void *call_site, void *tramp, void *dest)

Indeed.

#17 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Steven Rostedt
Quoted 13 lines · Steven Rostedt

arch/Kconfig config HAVE_ARCH_PREL32_RELOCATIONS · line 887

Quoted 6 lines
arch/Kconfig +6 -0
+config HAVE_STATIC_CALL_OPTIMIZED
+	bool
+
+config HAVE_STATIC_CALL_UNOPTIMIZED
+	bool
+

Let's also have

config HAVE_STATIC_CALL

	def_bool Y
	depends on HAVE_STATIC_CALL_OPTIMIZED || HAVE_STATIC_CALL_UNOPTIMIZED
Quoted 6 lines

include/linux/static_call.h line 74

Quoted 2 lines
include/linux/static_call.h new file +2 -0
+#if defined(CONFIG_HAVE_STATIC_CALL_OPTIMIZED) || \
+    defined(CONFIG_HAVE_STATIC_CALL_UNOPTIMIZED)

Now we can have:

#ifdef CONFIG_HAVE_STATIC_CALL

Yeah, that's better.

Quoted 9 lines

include/linux/static_call.h line 112

Quoted 5 lines
include/linux/static_call.h new file +5 -0
+#define DECLARE_STATIC_CALL(key, func)					\
+	extern struct static_call_key key;				\
+	extern typeof(func) STATIC_CALL_TRAMP(key);			\
+	/* Preserve the ELF symbol so objtool can access it: */		\
+	__ADDRESSABLE(key)

Does the __ADDRESSABLE(key) need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.

For each call site, objtool creates a struct in .static_call_sites:

	struct static_call_site {
		s32 addr;
		s32 key;
	};

In order to do that, it needs to create a relocation which references the key symbol. If the key is defined in another .o file, then the current .o will not have an ELF symbol associated with the key. The __ADDRESSABLE(key) thing tells GCC to leave the key symbol in the .o file, even though it's not referenced anywhere. That makes objtool's job easier, so it doesn't have to edit the symbol table.

I could add a comment saying as much, though it's hard to explain it in fewer words than I just did :-)

Quoted 12 lines

kernel/static_call.c line 98

Quoted 8 lines
kernel/static_call.c new file +8 -0
+	/*
+	 * If called before init, leave the call sites unpatched for now.
+	 * In the meantime they'll continue to call the temporary trampoline.
+	 */
+	if (!static_call_initialized)
+		goto done;
+
+	list_for_each_entry(mod, &key->site_mods, list) {

Since I'm expecting a lot of sites, I'm wondering if we should just do this as an array, like I do with the ftrace call sites.

Quoted 2 lines

But this can be an enhancement for later. Let's focus on getting this working first.

But it's not a static list. It can grow/shrink as modules are loaded/unload.

Quoted 15 lines

kernel/static_call.c line 110

Quoted 13 lines
kernel/static_call.c new file +12 -0
+		if (!mod->sites) {
+			/*
+			 * This can happen if the static call key is defined in
+			 * a module which doesn't use it.
+			 */
+			continue;
+		}
+
+		stop = __stop_static_call_sites;
+
+#ifdef CONFIG_MODULES
+		if (mod->mod) {

BTW, "mod" is an unfortunate name.

True. I'll change it to 'site_mod'.

Quoted 10 lines
Quoted 10 lines
kernel/static_call.c new file +9 -0
+			stop = mod->mod->static_call_sites +
+			       mod->mod->num_static_call_sites;
+		}
+#endif
+
+		for (site = mod->sites;
+		     site < stop && static_call_key(site) == key; site++) {
+			unsigned long addr = static_call_addr(site);
+
Quoted 8 lines

kernel/static_call.c line 124

Quoted 5 lines
kernel/static_call.c new file +5 -0
+			if (!mod->mod && init_section_contains((void *)addr, 1))
+				continue;
+			if (mod->mod && within_module_init(addr, mod->mod))
+				continue;
+

So what's the reason for skipping init calls?

This is the runtime changing code (static_call_update). Presumably the init sections no longer exist and we shouldn't write to any (former) call sites there.

That's probably a dangerous assumption though... If static_call_update() were called early, some init code might not get patched and then call into the wrong function.

I'm thinking we should just disallow static call sites in init sections. I can't think of a good reason why they would be needed in init code. We can WARN when detecting them during boot / module init.

Signature · 1 line
Josh
#51 lore raw
In reply to Andy Lutomirski
Quoted 10 lines · On Fri, 9 Nov 2018 11:05:51 -0800 Andy Lutomirski <[email protected]> wrote:
Quoted 6 lines

Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.

Indeed.

Although I had assumed that tracepoints already had appropriate jump label magic.

It does. But that's not the problem I was trying to solve. It's that tracing took a 8% noise dive with retpolines when enabled (hackbench slowed down by 8% with all the trace events enabled compared to all trace events enabled without retpoline). That is, normal users (those not tracinng) are not affected by trace events slowing down by retpoline. Those that care about performance when they are tracing, are affected by retpoline, quite drastically.

I'm doing another test run and measurements, to see how the unoptimized trampolines help, followed by the trampoline case.

-- Steve

#52 lore raw
Quoted 22 lines · Steven Rostedt
Quoted 10 lines · On Fri, 9 Nov 2018 11:05:51 -0800 Andy Lutomirski <[email protected]> wrote:
Quoted 6 lines

Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.

Indeed.

Although I had assumed that tracepoints already had appropriate jump label magic.

It does. But that's not the problem I was trying to solve. It's that tracing took a 8% noise dive with retpolines when enabled (hackbench slowed down by 8% with all the trace events enabled compared to all trace events enabled without retpoline). That is, normal users (those not tracinng) are not affected by trace events slowing down by retpoline. Those that care about performance when they are tracing, are affected by retpoline, quite drastically.

Quoted 2 lines

I'm doing another test run and measurements, to see how the unoptimized trampolines help, followed by the trampoline case.

Are you sure you're using unoptimized? Optimized is the default on x86-64 (with my third patch).

Signature · 1 line
Josh
#18 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Josh Poimboeuf
Quoted 24 lines · On Fri, 9 Nov 2018 13:35:05 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 9 lines

include/linux/static_call.h line 112

Quoted 5 lines
include/linux/static_call.h new file +5 -0
+#define DECLARE_STATIC_CALL(key, func)					\
+	extern struct static_call_key key;				\
+	extern typeof(func) STATIC_CALL_TRAMP(key);			\
+	/* Preserve the ELF symbol so objtool can access it: */		\
+	__ADDRESSABLE(key)  

Does the __ADDRESSABLE(key) need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.

For each call site, objtool creates a struct in .static_call_sites:

	struct static_call_site {
		s32 addr;
		s32 key;
	};

In order to do that, it needs to create a relocation which references the key symbol. If the key is defined in another .o file, then the current .o will not have an ELF symbol associated with the key. The __ADDRESSABLE(key) thing tells GCC to leave the key symbol in the .o file, even though it's not referenced anywhere. That makes objtool's job easier, so it doesn't have to edit the symbol table.

Quoted 2 lines

I could add a comment saying as much, though it's hard to explain it in fewer words than I just did :-)

Does this have to do with adding the references by relative address?

In record_mcount, I just picked an existing symbol and referenced that.. But perhaps this is a cleaner way.

Adding a more in depth comment wont hurt.

Quoted 13 lines
Quoted 13 lines

kernel/static_call.c line 98

Quoted 9 lines
kernel/static_call.c new file +8 -0
+	/*
+	 * If called before init, leave the call sites unpatched for now.
+	 * In the meantime they'll continue to call the temporary trampoline.
+	 */
+	if (!static_call_initialized)
+		goto done;
+
+	list_for_each_entry(mod, &key->site_mods, list) {  

Since I'm expecting a lot of sites, I'm wondering if we should just do this as an array, like I do with the ftrace call sites.

Quoted 5 lines
Quoted 2 lines

But this can be an enhancement for later. Let's focus on getting this working first.

But it's not a static list. It can grow/shrink as modules are loaded/unload.

Neither is ftrace :-) What I did was make one array for the core kernel code, and an array for each module, and link list those (single link, although double link may not be hard either). That will save a lot of memory than having each instance have a link pointer, as it only grows or shrinks in chunks.

Quoted 42 lines
Quoted 16 lines
Quoted 4 lines
kernel/static_call.c new file +2 -0
+		if (!mod->sites) {
+			/*

kernel/static_call.c line 110

Quoted 10 lines
kernel/static_call.c new file +10 -0
+			 * This can happen if the static call key is defined in
+			 * a module which doesn't use it.
+			 */
+			continue;
+		}
+
+		stop = __stop_static_call_sites;
+
+#ifdef CONFIG_MODULES
+		if (mod->mod) {  

BTW, "mod" is an unfortunate name.

True. I'll change it to 'site_mod'.

Quoted 10 lines
Quoted 10 lines
kernel/static_call.c new file +9 -0
+			stop = mod->mod->static_call_sites +
+			       mod->mod->num_static_call_sites;
+		}
+#endif
+
+		for (site = mod->sites;
+		     site < stop && static_call_key(site) == key; site++) {
+			unsigned long addr = static_call_addr(site);
+
Quoted 8 lines

kernel/static_call.c line 124

Quoted 5 lines
kernel/static_call.c new file +5 -0
+			if (!mod->mod && init_section_contains((void *)addr, 1))
+				continue;
+			if (mod->mod && within_module_init(addr, mod->mod))
+				continue;
+  

So what's the reason for skipping init calls?

This is the runtime changing code (static_call_update). Presumably the init sections no longer exist and we shouldn't write to any (former) call sites there.

Quoted 8 lines

That's probably a dangerous assumption though... If static_call_update() were called early, some init code might not get patched and then call into the wrong function.

I'm thinking we should just disallow static call sites in init sections. I can't think of a good reason why they would be needed in init code. We can WARN when detecting them during boot / module init.

What I would do is to allow init (like ftrace now does). I have ftrace_free_init_mem() that removes all the mcount references for init calls from its list. You could add a static_call_free_init() to kernel_init() in init/main.c too.

-- Steve

#53 lore raw
In reply to Josh Poimboeuf
Quoted 23 lines · On Fri, 9 Nov 2018 13:44:09 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 22 lines · Steven Rostedt
Quoted 10 lines · On Fri, 9 Nov 2018 11:05:51 -0800 Andy Lutomirski <[email protected]> wrote:
Quoted 6 lines

Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.

Indeed.

Although I had assumed that tracepoints already had appropriate jump label magic.

It does. But that's not the problem I was trying to solve. It's that tracing took a 8% noise dive with retpolines when enabled (hackbench slowed down by 8% with all the trace events enabled compared to all trace events enabled without retpoline). That is, normal users (those not tracinng) are not affected by trace events slowing down by retpoline. Those that care about performance when they are tracing, are affected by retpoline, quite drastically.

Quoted 6 lines
Quoted 2 lines

I'm doing another test run and measurements, to see how the unoptimized trampolines help, followed by the trampoline case.

Are you sure you're using unoptimized? Optimized is the default on x86-64 (with my third patch).

Yes, because I haven't applied that third patch yet ;-)

Then I'll apply it and see how much that improves things.

-- Steve

#19 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Steven Rostedt
Quoted 28 lines · Steven Rostedt
Quoted 24 lines · On Fri, 9 Nov 2018 13:35:05 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 9 lines

include/linux/static_call.h line 112

Quoted 5 lines
include/linux/static_call.h new file +5 -0
+#define DECLARE_STATIC_CALL(key, func)					\
+	extern struct static_call_key key;				\
+	extern typeof(func) STATIC_CALL_TRAMP(key);			\
+	/* Preserve the ELF symbol so objtool can access it: */		\
+	__ADDRESSABLE(key)  

Does the __ADDRESSABLE(key) need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.

For each call site, objtool creates a struct in .static_call_sites:

	struct static_call_site {
		s32 addr;
		s32 key;
	};

In order to do that, it needs to create a relocation which references the key symbol. If the key is defined in another .o file, then the current .o will not have an ELF symbol associated with the key. The __ADDRESSABLE(key) thing tells GCC to leave the key symbol in the .o file, even though it's not referenced anywhere. That makes objtool's job easier, so it doesn't have to edit the symbol table.

Quoted 7 lines
Quoted 2 lines

I could add a comment saying as much, though it's hard to explain it in fewer words than I just did :-)

Does this have to do with adding the references by relative address?

In record_mcount, I just picked an existing symbol and referenced that.. But perhaps this is a cleaner way.

I think recordmcount is different. It creates references (in __mcount_loc) to functions which are already in the object file, so they already have symbols associated with them.

But in this case, when objtool is creating references, the symbol it needs to reference is outside the .o file, so there's no symbol to associate it with.

Quoted 18 lines

Adding a more in depth comment wont hurt.

Quoted 16 lines
Quoted 15 lines

kernel/static_call.c line 98

Quoted 9 lines
kernel/static_call.c new file +8 -0
+	/*
+	 * If called before init, leave the call sites unpatched for now.
+	 * In the meantime they'll continue to call the temporary trampoline.
+	 */
+	if (!static_call_initialized)
+		goto done;
+
+	list_for_each_entry(mod, &key->site_mods, list) {  

Since I'm expecting a lot of sites, I'm wondering if we should just do this as an array, like I do with the ftrace call sites.

But this can be an enhancement for later. Let's focus on getting this working first.

Quoted 8 lines
Quoted 2 lines

But it's not a static list. It can grow/shrink as modules are loaded/unload.

Neither is ftrace :-) What I did was make one array for the core kernel code, and an array for each module, and link list those (single link, although double link may not be hard either). That will save a lot of memory than having each instance have a link pointer, as it only grows or shrinks in chunks.

That sounds exactly like what I did :-)

The site_mods list is a list of static_call_mods. Each static_call_mod has a pointer to an array (which is that module/vmlinux's .static_call_site section).

Quoted 15 lines
Quoted 6 lines
Quoted 1 line

So what's the reason for skipping init calls?

This is the runtime changing code (static_call_update). Presumably the init sections no longer exist and we shouldn't write to any (former) call sites there.

Quoted 8 lines

That's probably a dangerous assumption though... If static_call_update() were called early, some init code might not get patched and then call into the wrong function.

I'm thinking we should just disallow static call sites in init sections. I can't think of a good reason why they would be needed in init code. We can WARN when detecting them during boot / module init.

Quoted 4 lines

What I would do is to allow init (like ftrace now does). I have ftrace_free_init_mem() that removes all the mcount references for init calls from its list. You could add a static_call_free_init() to kernel_init() in init/main.c too.

That makes sense for ftrace, but I don't see much point in allowing it for static calls. Maybe we could just add support for it later if it turns out to be useful.

Signature · 1 line
Josh
#54 lore raw
In reply to Steven Rostedt
Quoted 29 lines · Steven Rostedt
Quoted 26 lines · On Fri, 9 Nov 2018 13:44:09 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 24 lines · Steven Rostedt
Quoted 10 lines · On Fri, 9 Nov 2018 11:05:51 -0800 Andy Lutomirski <[email protected]> wrote:
Quoted 6 lines

Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.

Indeed.

Although I had assumed that tracepoints already had appropriate jump label magic.

It does. But that's not the problem I was trying to solve. It's that tracing took a 8% noise dive with retpolines when enabled (hackbench slowed down by 8% with all the trace events enabled compared to all trace events enabled without retpoline). That is, normal users (those not tracinng) are not affected by trace events slowing down by retpoline. Those that care about performance when they are tracing, are affected by retpoline, quite drastically.

I'm doing another test run and measurements, to see how the unoptimized trampolines help, followed by the trampoline case.

Quoted 7 lines
Quoted 3 lines

Are you sure you're using unoptimized? Optimized is the default on x86-64 (with my third patch).

Yes, because I haven't applied that third patch yet ;-)

Then I'll apply it and see how much that improves things.

Ah, good. That will be interesting to see the difference between optimized/unoptimized.

Signature · 1 line
Josh
#56 lore raw
In reply to Andy Lutomirski
Quoted 11 lines · Andy Lutomirski
Quoted 7 lines · Ingo Molnar

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

For tracing, we'd want static_call_set_to_nop() or something like that, right?

Hm. IIUC, when gcc sees static_call(key)(...), it has to generate code to put the right values in %rdi, %rsi etc.. Even if the function is void (*)(void), gcc would still need to shuffle things around (either spill and reload, or move %rdi to some callee saved register). So if the static_call is noop'ed out most of the time, that seems like a net loss? With an unlikely static_key, gcc can do all the parameter setup and reloading in an out-of-line chunk of code.

static calls seems like a quite useful concept, but only/mostly if _some_ function needs to be called at that spot.

Aside: there should be some compile-time check that static_call_set_to_nop can only be used if the return type is void.

Rasmus

#20 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Josh Poimboeuf

On Fri, 9 Nov 2018 14:34:59 -0600
Josh Poimboeuf <[email protected]> wrote:

I'm slowly massaging this to work with tracepoints.

But I hit a snag on this patch.

Quoted 37 lines · Steven Rostedt
Quoted 28 lines
Quoted 24 lines · On Fri, 9 Nov 2018 13:35:05 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 9 lines

include/linux/static_call.h line 112

Quoted 5 lines
include/linux/static_call.h new file +5 -0
+#define DECLARE_STATIC_CALL(key, func)					\
+	extern struct static_call_key key;				\
+	extern typeof(func) STATIC_CALL_TRAMP(key);			\
+	/* Preserve the ELF symbol so objtool can access it: */		\
+	__ADDRESSABLE(key)    

Does the __ADDRESSABLE(key) need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.

For each call site, objtool creates a struct in .static_call_sites:

	struct static_call_site {
		s32 addr;
		s32 key;
	};

In order to do that, it needs to create a relocation which references the key symbol. If the key is defined in another .o file, then the current .o will not have an ELF symbol associated with the key. The __ADDRESSABLE(key) thing tells GCC to leave the key symbol in the .o file, even though it's not referenced anywhere. That makes objtool's job easier, so it doesn't have to edit the symbol table.

Quoted 7 lines
Quoted 2 lines

I could add a comment saying as much, though it's hard to explain it in fewer words than I just did :-)

Does this have to do with adding the references by relative address?

In record_mcount, I just picked an existing symbol and referenced that.. But perhaps this is a cleaner way.

Quoted 8 lines

I think recordmcount is different. It creates references (in __mcount_loc) to functions which are already in the object file, so they already have symbols associated with them.

But in this case, when objtool is creating references, the symbol it needs to reference is outside the .o file, so there's no symbol to associate it with.

The __ADDRESSABLE() appears to fail if you have a header with a DECLARE_STATIC_CALL() that is included where the DEFINE_STATIC_CALL() is, because I'm getting this:

In file included from <command-line>:
/work/git/linux-trace.git/include/linux/compiler.h:285:11: error: redefinition of ‘__addressable___tp_func_sys_enter40’

   __PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
           ^~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/compiler_types.h:53:23: note: in definition of macro ‘___PASTE’
 #define ___PASTE(a,b) a##b
                       ^

/work/git/linux-trace.git/include/linux/compiler.h:285:3: note: in expansion of macro ‘__PASTE’
__PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
^~~~~~~
/work/git/linux-trace.git/include/linux/static_call.h:112:2: note: in expansion of macro ‘__ADDRESSABLE’
__ADDRESSABLE(key)
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/static_call.h:115:2: note: in expansion of macro ‘DECLARE_STATIC_CALL’
DECLARE_STATIC_CALL(key, _func); \
^~~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:310:2: note: in expansion of macro ‘DEFINE_STATIC_CALL’
DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);
^~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/define_trace.h:42:2: note: in expansion of macro ‘DEFINE_TRACE_FN’
DEFINE_TRACE_FN(name, reg, unreg, PARAMS(proto), PARAMS(args))
^~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/events/syscalls.h:18:1: note: in expansion of macro ‘TRACE_EVENT_FN’
TRACE_EVENT_FN(sys_enter,
^~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/compiler.h:285:11: note: previous definition of ‘__addressable___tp_func_sys_enter40’ was here

   __PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
           ^~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/compiler_types.h:53:23: note: in definition of macro ‘___PASTE’
 #define ___PASTE(a,b) a##b
                       ^

/work/git/linux-trace.git/include/linux/compiler.h:285:3: note: in expansion of macro ‘__PASTE’
__PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
^~~~~~~
/work/git/linux-trace.git/include/linux/static_call.h:112:2: note: in expansion of macro ‘__ADDRESSABLE’
__ADDRESSABLE(key)
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:234:2: note: in expansion of macro ‘DECLARE_STATIC_CALL’
DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \
^~~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:421:2: note: in expansion of macro ‘__DECLARE_TRACE’
__DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \
^~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:560:2: note: in expansion of macro ‘DECLARE_TRACE’
DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/events/syscalls.h:18:1: note: in expansion of macro ‘TRACE_EVENT_FN’
TRACE_EVENT_FN(sys_enter,

The complaint is on:

	DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);

And the previous definition is on:

	DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \

-- Steve

#27 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Josh Poimboeuf
Quoted 4 lines · Josh Poimboeuf
Quoted 3 lines · Steven Rostedt

On Fri, 9 Nov 2018 13:35:05 -0600
Josh Poimboeuf <[email protected]> wrote:

..

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 1 line

So what's the reason for skipping init calls?

This is the runtime changing code (static_call_update). Presumably the init sections no longer exist and we shouldn't write to any (former) call sites there.

That's probably a dangerous assumption though... If static_call_update() were called early, some init code might not get patched and then call into the wrong function.

I'm thinking we should just disallow static call sites in init sections. I can't think of a good reason why they would be needed in init code. We can WARN when detecting them during boot / module init.

Quoted 9 lines
Quoted 4 lines

What I would do is to allow init (like ftrace now does). I have ftrace_free_init_mem() that removes all the mcount references for init calls from its list. You could add a static_call_free_init() to kernel_init() in init/main.c too.

That makes sense for ftrace, but I don't see much point in allowing it for static calls. Maybe we could just add support for it later if it turns out to be useful.

I don't see how you can prevent that. Some arch may use a static call in its version of some library code which could be used anywhere, including in .init code.

#21 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Steven Rostedt
Quoted 106 lines · Steven Rostedt

On Fri, 9 Nov 2018 14:34:59 -0600
Josh Poimboeuf <[email protected]> wrote:

I'm slowly massaging this to work with tracepoints.

But I hit a snag on this patch.

Quoted 37 lines · Steven Rostedt
Quoted 28 lines
Quoted 24 lines · On Fri, 9 Nov 2018 13:35:05 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 9 lines

include/linux/static_call.h line 112

Quoted 5 lines
include/linux/static_call.h new file +5 -0
+#define DECLARE_STATIC_CALL(key, func)                                       \
+     extern struct static_call_key key;                              \
+     extern typeof(func) STATIC_CALL_TRAMP(key);                     \
+     /* Preserve the ELF symbol so objtool can access it: */         \
+     __ADDRESSABLE(key)

Does the __ADDRESSABLE(key) need to be in the DECLARE part? If so, there needs to be more explanation than just the comment above it.

For each call site, objtool creates a struct in .static_call_sites:

  struct static_call_site {
          s32 addr;
          s32 key;
  };

In order to do that, it needs to create a relocation which references the key symbol. If the key is defined in another .o file, then the current .o will not have an ELF symbol associated with the key. The __ADDRESSABLE(key) thing tells GCC to leave the key symbol in the .o file, even though it's not referenced anywhere. That makes objtool's job easier, so it doesn't have to edit the symbol table.

Quoted 7 lines
Quoted 2 lines

I could add a comment saying as much, though it's hard to explain it in fewer words than I just did :-)

Does this have to do with adding the references by relative address?

In record_mcount, I just picked an existing symbol and referenced that.. But perhaps this is a cleaner way.

Quoted 8 lines

I think recordmcount is different. It creates references (in __mcount_loc) to functions which are already in the object file, so they already have symbols associated with them.

But in this case, when objtool is creating references, the symbol it needs to reference is outside the .o file, so there's no symbol to associate it with.

The __ADDRESSABLE() appears to fail if you have a header with a DECLARE_STATIC_CALL() that is included where the DEFINE_STATIC_CALL() is, because I'm getting this:

In file included from <command-line>:
/work/git/linux-trace.git/include/linux/compiler.h:285:11: error: redefinition of ‘__addressable___tp_func_sys_enter40’

   __PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
           ^~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/compiler_types.h:53:23: note: in definition of macro ‘___PASTE’
 #define ___PASTE(a,b) a##b
                       ^
/work/git/linux-trace.git/include/linux/compiler.h:285:3: note: in expansion of macro ‘__PASTE’
   __PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
   ^~~~~~~

/work/git/linux-trace.git/include/linux/static_call.h:112:2: note: in expansion of macro ‘__ADDRESSABLE’
__ADDRESSABLE(key)
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/static_call.h:115:2: note: in expansion of macro ‘DECLARE_STATIC_CALL’
DECLARE_STATIC_CALL(key, _func); \
^~~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:310:2: note: in expansion of macro ‘DEFINE_STATIC_CALL’
DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);
^~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/define_trace.h:42:2: note: in expansion of macro ‘DEFINE_TRACE_FN’
DEFINE_TRACE_FN(name, reg, unreg, PARAMS(proto), PARAMS(args))
^~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/events/syscalls.h:18:1: note: in expansion of macro ‘TRACE_EVENT_FN’
TRACE_EVENT_FN(sys_enter,
^~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/compiler.h:285:11: note: previous definition of ‘__addressable___tp_func_sys_enter40’ was here

   __PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
           ^~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/compiler_types.h:53:23: note: in definition of macro ‘___PASTE’
 #define ___PASTE(a,b) a##b
                       ^
/work/git/linux-trace.git/include/linux/compiler.h:285:3: note: in expansion of macro ‘__PASTE’
   __PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
   ^~~~~~~

/work/git/linux-trace.git/include/linux/static_call.h:112:2: note: in expansion of macro ‘__ADDRESSABLE’
__ADDRESSABLE(key)
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:234:2: note: in expansion of macro ‘DECLARE_STATIC_CALL’
DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \
^~~~~~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:421:2: note: in expansion of macro ‘__DECLARE_TRACE’
__DECLARE_TRACE(name, PARAMS(proto), PARAMS(args), \
^~~~~~~~~~~~~~~
/work/git/linux-trace.git/include/linux/tracepoint.h:560:2: note: in expansion of macro ‘DECLARE_TRACE’
DECLARE_TRACE(name, PARAMS(proto), PARAMS(args))
^~~~~~~~~~~~~
/work/git/linux-trace.git/include/trace/events/syscalls.h:18:1: note: in expansion of macro ‘TRACE_EVENT_FN’
TRACE_EVENT_FN(sys_enter,

Quoted 8 lines

The complaint is on:

        DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);

And the previous definition is on:

        DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \

Does the DECLARE really need the __ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.

#22 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

Quoted 6 lines · On Sat, 10 Nov 2018 12:58:08 +0100 Ard Biesheuvel <[email protected]>…
Quoted 6 lines

The complaint is on:

        DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);

And the previous definition is on:

Quoted 7 lines
Quoted 2 lines
        DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \

Does the DECLARE really need the __ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.

I applied the following change and it compiled fine:

include/linux/static_call.h +3 -3
index 90b580b95303..5f8a0f0e77be 100644
@@ -108,8 +108,6 @@ extern void arch_static_call_poison_tramp(unsigned long insn);
 #define DECLARE_STATIC_CALL(key, func)					\
 	extern struct static_call_key key;				\
 	extern typeof(func) STATIC_CALL_TRAMP(key);			\
-	/* Preserve the ELF symbol so objtool can access it: */		\
-	__ADDRESSABLE(key)
 
 #define DEFINE_STATIC_CALL(key, _func)					\
 	DECLARE_STATIC_CALL(key, _func);				\
@@ -117,7 +115,9 @@ extern void arch_static_call_poison_tramp(unsigned long insn);
 		.func = _func,						\
 		.site_mods = LIST_HEAD_INIT(key.site_mods),		\
 	};								\
-	ARCH_STATIC_CALL_TEMPORARY_TRAMP(key)
+	ARCH_STATIC_CALL_TEMPORARY_TRAMP(key);				\
+	/* Preserve the ELF symbol so objtool can access it: */		\
+	__ADDRESSABLE(key)
 
 #define static_call(key, args...) STATIC_CALL_TRAMP(key)(args)
 

Thanks!

-- Steve

#55 lore raw
In reply to Andy Lutomirski
Quoted 34 lines · On Fri, 9 Nov 2018 11:05:51 -0800 Andy Lutomirski <[email protected]> wrote:
Quoted 27 lines · Steven Rostedt
Quoted 11 lines · On Fri, 9 Nov 2018 10:41:37 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 11 lines · Josh Poimboeuf
Quoted 10 lines · Andy Lutomirski
Quoted 8 lines · Ingo Molnar

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

For tracing, we'd want static_call_set_to_nop() or something like that, right?

Are we talking about tracepoints? Or ftrace?

Since ftrace changes calls to nops, and vice versa, I assume you meant ftrace. I don't think ftrace is a good candidate for this, as it's inherently more flexible than this API would reasonably allow.

Quoted 6 lines

Not sure what Andy was talking about, but I'm currently implementing tracepoints to use this, as tracepoints use indirect calls, and are a prime candidate for static calls, as I showed in my original RFC of this feature.

Quoted 3 lines

Indeed.

Although I had assumed that tracepoints already had appropriate jump label magic.

As far as I know, the jump label magic is for reducing the overhead when the tracepoint is OFF (because it can skip function parameter preparation), and this static call will be good when the tracepoint is ON (enabled) because of this can avoid retpoline performance degradation.

Thank you,

Signature · 1 line
Masami Hiramatsu <[email protected]>
maintainer
#35 lore raw
In reply to Ard Biesheuvel
Quoted 11 lines · Ard Biesheuvel
Quoted 8 lines · Ingo Molnar
Quoted 3 lines

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

I implemented the GCC plugin approach here for arm64

I'm confused; I though we only needed objtool for variable instruction length architectures, because we can't reliably decode our instruction stream. Otherwise we can fairly trivially use the DWARF relocation data, no?

#36 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Ard Biesheuvel
Quoted 8 lines · Ingo Molnar
Quoted 3 lines

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

Quoted 6 lines
Quoted 1 line

I implemented the GCC plugin approach here for arm64

I'm confused; I though we only needed objtool for variable instruction length architectures, because we can't reliably decode our instruction stream. Otherwise we can fairly trivially use the DWARF relocation data, no?

How would that work? We could build vmlinux with --emit-relocs, filter out the static jump/call relocations and resolve the symbol names to filter the ones associated with calls to trampolines. But then, we have to build the static_call_sites section and reinject it back into the image in some way, which is essentially objtool, no?

maintainer
#37 lore raw
Quoted 19 lines · Ard Biesheuvel
Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Ard Biesheuvel
Quoted 8 lines · Ingo Molnar
Quoted 3 lines

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

Quoted 6 lines
Quoted 1 line

I implemented the GCC plugin approach here for arm64

I'm confused; I though we only needed objtool for variable instruction length architectures, because we can't reliably decode our instruction stream. Otherwise we can fairly trivially use the DWARF relocation data, no?

Quoted 5 lines

How would that work? We could build vmlinux with --emit-relocs, filter out the static jump/call relocations and resolve the symbol names to filter the ones associated with calls to trampolines. But then, we have to build the static_call_sites section and reinject it back into the image in some way, which is essentially objtool, no?

It's a _much_ simpler tool than objtool, but yes, we need a tool that reads the relocation stuff and (re)injects it in a new section -- we don't need it on a vmlinux level, it can be done per TU.

Anyway, a GCC plugin (I still have to have a peek at your thing) sounds like it should work just fine too.

#23 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

In reply to Steven Rostedt
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Sat, 10 Nov 2018 12:58:08 +0100 Ard Biesheuvel <[email protected]>…
Quoted 6 lines

The complaint is on:

        DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);

And the previous definition is on:

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
        DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \

Does the DECLARE really need the __ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.

Yes, DECLARE needs the __ADDRESSABLE. In the case where DECLARE is used, but DEFINE is not, GCC strips the symbol.

Quoted 16 lines

I applied the following change and it compiled fine:

include/linux/static_call.h +0 -2
index 90b580b95303..5f8a0f0e77be 100644
@@ -108,8 +108,6 @@ extern void arch_static_call_poison_tramp(unsigned long insn);
 #define DECLARE_STATIC_CALL(key, func)					\
 	extern struct static_call_key key;				\
 	extern typeof(func) STATIC_CALL_TRAMP(key);			\
-	/* Preserve the ELF symbol so objtool can access it: */		\
-	__ADDRESSABLE(key)
 
 #define DEFINE_STATIC_CALL(key, _func)					\
 	DECLARE_STATIC_CALL(key, _func);				\

include/linux/static_call.h extern void arch_static_call_poison_tramp(unsigned long insn); · line 122

Quoted 9 lines
include/linux/static_call.h +3 -1
@@ -117,7 +115,9 @@ extern void arch_static_call_poison_tramp(unsigned long insn);
 		.func = _func,						\
 		.site_mods = LIST_HEAD_INIT(key.site_mods),		\
 	};								\
-	ARCH_STATIC_CALL_TEMPORARY_TRAMP(key)
+	ARCH_STATIC_CALL_TEMPORARY_TRAMP(key);				\
+	/* Preserve the ELF symbol so objtool can access it: */		\
+	__ADDRESSABLE(key)
 
 #define static_call(key, args...) STATIC_CALL_TRAMP(key)(args)

Adding __ADDRESSABLE to the DEFINE macro doesn't do any good. By definition, the key is defined in the .o file, so the symbol already exists.

The issue you're seeing is really an issue with the __ADDRESSABLE macro not creating unique symbol names. This should fix it:

include/linux/compiler.h +1 -1
index 06396c1cf127..4bb73fd918b5 100644
@@ -282,7 +282,7 @@ unsigned long read_word_at_a_time(const void *addr)
  */
 #define __ADDRESSABLE(sym) \
 	static void * __section(".discard.addressable") __used \
-		__PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
+		__UNIQUE_ID(__addressable_##sym) = (void *)&sym;
 
 /**
  * offset_to_ptr - convert a relative memory offset to an absolute pointer
#24 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

Quoted 14 lines · Josh Poimboeuf
Quoted 13 lines · Steven Rostedt
Quoted 9 lines · On Sat, 10 Nov 2018 12:58:08 +0100 Ard Biesheuvel <[email protected]>…
Quoted 8 lines

The complaint is on:

        DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);

And the previous definition is on:

        DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Does the DECLARE really need the __ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.

Yes, DECLARE needs the __ADDRESSABLE. In the case where DECLARE is used, but DEFINE is not, GCC strips the symbol.

I assume DECLARE() is intended for use in header files, and DEFINE() for source files, no?

Doesn't that mean that whatever symbol __ADDRESSABLE() refers to should have external linkage, in which case it it addressable anyway? Or are we talking about some LTO / --gc-sections use case here?

Quoted 38 lines
Quoted 16 lines

I applied the following change and it compiled fine:

include/linux/static_call.h +0 -2
index 90b580b95303..5f8a0f0e77be 100644
@@ -108,8 +108,6 @@ extern void arch_static_call_poison_tramp(unsigned long insn);
 #define DECLARE_STATIC_CALL(key, func)                                       \
      extern struct static_call_key key;                              \
      extern typeof(func) STATIC_CALL_TRAMP(key);                     \
-     /* Preserve the ELF symbol so objtool can access it: */         \
-     __ADDRESSABLE(key)
 
 #define DEFINE_STATIC_CALL(key, _func)                                       \
      DECLARE_STATIC_CALL(key, _func);                                \

include/linux/static_call.h extern void arch_static_call_poison_tramp(unsigned long insn); · line 122

Quoted 9 lines
include/linux/static_call.h +3 -1
@@ -117,7 +115,9 @@ extern void arch_static_call_poison_tramp(unsigned long insn);
              .func = _func,                                          \
              .site_mods = LIST_HEAD_INIT(key.site_mods),             \
      };                                                              \
-     ARCH_STATIC_CALL_TEMPORARY_TRAMP(key)
+     ARCH_STATIC_CALL_TEMPORARY_TRAMP(key);                          \
+     /* Preserve the ELF symbol so objtool can access it: */         \
+     __ADDRESSABLE(key)
 
 #define static_call(key, args...) STATIC_CALL_TRAMP(key)(args)

Adding __ADDRESSABLE to the DEFINE macro doesn't do any good. By definition, the key is defined in the .o file, so the symbol already exists.

The issue you're seeing is really an issue with the __ADDRESSABLE macro not creating unique symbol names. This should fix it:

include/linux/compiler.h +0 -0
index 06396c1cf127..4bb73fd918b5 100644

include/linux/compiler.h unsigned long read_word_at_a_time(const void *addr) · line 288

Quoted 8 lines
include/linux/compiler.h +1 -1
@@ -282,7 +282,7 @@ unsigned long read_word_at_a_time(const void *addr)
  */
 #define __ADDRESSABLE(sym) \
        static void * __section(".discard.addressable") __used \
-               __PASTE(__addressable_##sym, __LINE__) = (void *)&sym;
+               __UNIQUE_ID(__addressable_##sym) = (void *)&sym;
 
 /**
  * offset_to_ptr - convert a relative memory offset to an absolute pointer

Not sure if it matters, but we'll end up with a lot more stuff in .discard.addressable this way.

#25 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

Quoted 20 lines · Ard Biesheuvel
Quoted 19 lines · Josh Poimboeuf
Quoted 17 lines · Steven Rostedt
Quoted 13 lines · On Sat, 10 Nov 2018 12:58:08 +0100 Ard Biesheuvel <[email protected]>…
Quoted 8 lines

The complaint is on:

        DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);

And the previous definition is on:

        DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \

Does the DECLARE really need the __ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.

Quoted 6 lines
Quoted 3 lines

Yes, DECLARE needs the __ADDRESSABLE. In the case where DECLARE is used, but DEFINE is not, GCC strips the symbol.

I assume DECLARE() is intended for use in header files, and DEFINE() for source files, no?

Right.

Quoted 3 lines

Doesn't that mean that whatever symbol __ADDRESSABLE() refers to should have external linkage, in which case it it addressable anyway? Or are we talking about some LTO / --gc-sections use case here?

If the key is declared, but not used, GCC doesn't put the key's ELF symbol in the binary's symbol table. That makes objtool's life harder, because if the file has a call site, then objtool has to add the key symbol to the symbol table, so that it can create a relocation (in the call site table) which references the symbol.

Signature · 1 line
Josh
#26 lore raw

Re: [RFC PATCH 1/3] static_call: Add static call infrastructure

Quoted 34 lines · Josh Poimboeuf
Quoted 20 lines · Ard Biesheuvel
Quoted 19 lines · Josh Poimboeuf
Quoted 17 lines · Steven Rostedt
Quoted 13 lines · On Sat, 10 Nov 2018 12:58:08 +0100 Ard Biesheuvel <[email protected]>…
Quoted 8 lines

The complaint is on:

        DEFINE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name);

And the previous definition is on:

        DECLARE_STATIC_CALL(__tp_func_##name, __tracepoint_iter_##name); \

Does the DECLARE really need the __ADDRESSABLE? Its purpose is to ensure that symbols with static linkage are not optimized away, but since the reference is from a header file, the symbol should have external linkage anyway.

Quoted 6 lines
Quoted 3 lines

Yes, DECLARE needs the __ADDRESSABLE. In the case where DECLARE is used, but DEFINE is not, GCC strips the symbol.

I assume DECLARE() is intended for use in header files, and DEFINE() for source files, no?

Right.

Quoted 3 lines

Doesn't that mean that whatever symbol __ADDRESSABLE() refers to should have external linkage, in which case it it addressable anyway? Or are we talking about some LTO / --gc-sections use case here?

Quoted 6 lines

If the key is declared, but not used, GCC doesn't put the key's ELF symbol in the binary's symbol table. That makes objtool's life harder, because if the file has a call site, then objtool has to add the key symbol to the symbol table, so that it can create a relocation (in the call site table) which references the symbol.

Ah, right. So this is specific to objtool rather than a generic issue.

Should we perhaps wrap __ADDRESSABLE() into something else that resolves to a NOP unless on X86, and give it a clear name?

#39 lore raw
In reply to Josh Poimboeuf
Quoted 9 lines · Josh Poimboeuf
Quoted 7 lines · Ingo Molnar
Quoted 3 lines

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

Quoted 6 lines

The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.

The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).

Which architectures would benefit from ORC support the most?

I really think that hard reliance on GCC plugins is foolish - but maybe Clang's plugin infrastructure is a guarantee that it remains a sane and usable interface.

Quoted 9 lines
Quoted 8 lines

I'd be very happy with a demonstrated paravirt optimization already - i.e. seeing the before/after effect on the vmlinux with an x86 distro config.

All major Linux distributions enable CONFIG_PARAVIRT=y and CONFIG_PARAVIRT_XXL=y on x86 at the moment, so optimizing it away as much as possible in the 99.999% cases where it's not used is a primary concern.

Quoted 7 lines

For paravirt, I was thinking of it as more of a cleanup than an optimization. The paravirt patching code already replaces indirect branches with direct ones -- see paravirt_patch_default().

Though it *would* reduce the instruction footprint a bit, as the 7-byte indirect calls (later patched to 5-byte direct + 2-byte nop) would instead be 5-byte direct calls to begin with.

Yes.

Quoted 9 lines
Quoted 5 lines

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

With retpolines, and with tracepoints enabled, it's definitely a major win. Steve measured an 8.9% general slowdown on hackbench caused by retpolines.

How much of that slowdown is reversed?

Quoted 2 lines

But with tracepoints disabled, I believe static jumps are used, which already minimizes the impact on hot paths.

Yeah.

Thanks,

Ing

#40 lore raw
Quoted 12 lines · Ingo Molnar
Quoted 9 lines · Josh Poimboeuf
Quoted 7 lines · Ingo Molnar
Quoted 3 lines

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

Quoted 8 lines
Quoted 6 lines

The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.

The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).

Which architectures would benefit from ORC support the most?

According to my (limited and potentially flawed) knowledge, I think arm64 would benefit the most performance-wise, whereas powerpc and s390 gains would be quite a bit less.

We may have to port objtool to arm64 anyway, for live patching. But that will be a lot more work than it took for Ard to write a GCC plugin.

Quoted 1 line

I really think that hard reliance on GCC plugins is foolish

Funny, I feel the same way about hard reliance on objtool :-)

Quoted 2 lines

- but maybe Clang's plugin infrastructure is a guarantee that it remains a sane and usable interface.

Hopefully so. If it breaks, we could always write another tool, as the work is straightforward. Or we could make it an objtool subcommand which works on all arches.

Quoted 11 lines
Quoted 9 lines
Quoted 5 lines

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

With retpolines, and with tracepoints enabled, it's definitely a major win. Steve measured an 8.9% general slowdown on hackbench caused by retpolines.

How much of that slowdown is reversed?

In theory, it should reverse all of the slowdown, and actually may even speed it up a little. Steve is working on measuring that now.

Signature · 1 line
Josh
#45 lore raw
In reply to Ingo Molnar
Quoted 39 lines · Ingo Molnar
Quoted 9 lines · Josh Poimboeuf
Quoted 7 lines · Ingo Molnar
Quoted 3 lines

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

Quoted 6 lines

The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.

The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).

Which architectures would benefit from ORC support the most?

I really think that hard reliance on GCC plugins is foolish - but maybe Clang's plugin infrastructure is a guarantee that it remains a sane and usable interface.

Quoted 13 lines
Quoted 8 lines

I'd be very happy with a demonstrated paravirt optimization already - i.e. seeing the before/after effect on the vmlinux with an x86 distro config.

All major Linux distributions enable CONFIG_PARAVIRT=y and CONFIG_PARAVIRT_XXL=y on x86 at the moment, so optimizing it away as much as possible in the 99.999% cases where it's not used is a primary concern.

For paravirt, I was thinking of it as more of a cleanup than an optimization. The paravirt patching code already replaces indirect branches with direct ones -- see paravirt_patch_default().

Quoted 5 lines
Quoted 3 lines

Though it *would* reduce the instruction footprint a bit, as the 7-byte indirect calls (later patched to 5-byte direct + 2-byte nop) would instead be 5-byte direct calls to begin with.

Yes.

It would be a huge cleanup IMO -- the existing PVOP call stuff is really quite ugly IMO. Also, the existing stuff tries to emulate the semantics of passing parameters of unknown types using asm constraints, and I just don't believe that GCC does what we want it to do. In general, passing the *value* of a pointer to asm doesn't seem to convince gcc that the pointed-to value is used by the asm, and this makes me nervous. See commit 715bd9d12f84d8f5cc8ad21d888f9bc304a8eb0b as an example of this. In a similar vein, the existing PVOP calls have a "memory" clobber, and that's not free.

#41 lore raw
In reply to Josh Poimboeuf
Quoted 10 lines · Josh Poimboeuf
Quoted 8 lines · Ingo Molnar
Quoted 5 lines · Josh Poimboeuf
Quoted 4 lines · Ingo Molnar
Quoted 3 lines

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.

Having looked into this, I don't think they will share any code at all, to be honest. Perhaps some macros and string templates, that's all.

Quoted 10 lines
Quoted 5 lines
Quoted 3 lines

The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).

Which architectures would benefit from ORC support the most?

According to my (limited and potentially flawed) knowledge, I think arm64 would benefit the most performance-wise, whereas powerpc and s390 gains would be quite a bit less.

What would arm64 gain from ORC and/or objtool?

Quoted 1 line

We may have to port objtool to arm64 anyway, for live patching.

Is this about the reliable stack traces, i.e., the ability to detect non-leaf functions that don't create stack frames? I think we should be able to manage this without objtool on arm64 tbh.

Quoted 7 lines

But
that will be a lot more work than it took for Ard to write a GCC plugin.

Quoted 1 line

I really think that hard reliance on GCC plugins is foolish

Funny, I feel the same way about hard reliance on objtool :-)

I tend to agree here. I think objtool is a necessary evil (as are compiler plugins, for that matter) which I hope does not spread to other architectures.

But the main difference is that the GCC plugin is only ~50 lines (for this particular use case, and minus another 50 lines of boilerplate), whereas objtool (AIUI) duplicates lots and lots of functionality of the compiler, assembler and/or linker, to mangle relocations, create new sections etc etc. Porting this to other architectures is going to be a major maintenance effort, especially when I think of, e.g., 32-bit ARM with its Thumb2 quirks and other idiosyncrasies that are currently hidden in the toolchain. Other architectures should be first class citizens if objtool gains support for them, which means that the x86 people that own it currently are on the hook for testing their changes against architectures they are not familiar with.

This obviously applies equally to compiler plugins, but those have a lot more focus.

Quoted 24 lines
Quoted 2 lines

- but maybe Clang's plugin infrastructure is a guarantee that it remains a sane and usable interface.

Hopefully so. If it breaks, we could always write another tool, as the work is straightforward. Or we could make it an objtool subcommand which works on all arches.

Quoted 11 lines
Quoted 9 lines
Quoted 5 lines

All other usecases are bonus, but it would certainly be interesting to investigate the impact of using these APIs for tracing: that too is a feature enabled everywhere but utilized only by a small fraction of Linux users - so literally every single cycle or instruction saved or hot-path shortened is a major win.

With retpolines, and with tracepoints enabled, it's definitely a major win. Steve measured an 8.9% general slowdown on hackbench caused by retpolines.

How much of that slowdown is reversed?

In theory, it should reverse all of the slowdown, and actually may even speed it up a little. Steve is working on measuring that now.

--
Josh

#43 lore raw
In reply to Josh Poimboeuf
Quoted 4 lines · On Sun, 11 Nov 2018 23:30:55 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 1 line

How much of that slowdown is reversed?

In theory, it should reverse all of the slowdown, and actually may even speed it up a little. Steve is working on measuring that now.

When I'm able to get it to work! Hopefully that last patch snippet you posted will help. If not, I'm assuming you'll be in Vancouver this week, and we could sit down and work it out.

That said, I don't expect a 100% improvement. Because the retpoline causes slow down in other areas than just tracing, which is not being fixed by this. I'm expecting a substantial improvement (which I see good improvement with the unoptimized static calls), and hoping for much more with the optimized one (when I get it working). But not 100%, as stated above.

-- Steve

#42 lore raw
In reply to Ard Biesheuvel
Quoted 15 lines · Ard Biesheuvel
Quoted 14 lines · Josh Poimboeuf
Quoted 12 lines · Ingo Molnar
Quoted 9 lines · Josh Poimboeuf
Quoted 7 lines · Ingo Molnar
Quoted 3 lines

- I'm not sure about the objtool approach. Objtool is (currently) x86-64 only, which means we have to use the "unoptimized" version everywhere else. I may experiment with a GCC plugin instead.

I'd prefer the objtool approach. It's a pretty reliable first-principles approach while GCC plugin would have to be replicated for Clang and any other compilers, etc.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

The benefit of a plugin is that we'd only need two of them: GCC and Clang. And presumably, they'd share a lot of code.

Having looked into this, I don't think they will share any code at all, to be honest. Perhaps some macros and string templates, that's all.

Oh well. That should still be easier to maintain than objtool across all arches at this point.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

The prospect of porting objtool to all architectures is going to be much more of a daunting task (though we are at least already considering it for some arches).

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Which architectures would benefit from ORC support the most?

According to my (limited and potentially flawed) knowledge, I think arm64 would benefit the most performance-wise, whereas powerpc and s390 gains would be quite a bit less.

What would arm64 gain from ORC and/or objtool?

Other than live patching, the biggest benefit would be an across-the-board performance improvement from disabling frame pointers. It would be interesting to see some arm64 performance numbers there, for a kernel compiled with -fomit-frame-pointer.

For more details (and benefits of) ORC see
Documentation/x86/orc-unwinder.txt.

Objtool has also come in handy for other cases, like ensuring retpolines are used everywhere.

Over time, I would like to move some objtool functionality to compiler plugins, such that it would be easier to port it to other arches.

Quoted 5 lines
Quoted 1 line

We may have to port objtool to arm64 anyway, for live patching.

Is this about the reliable stack traces, i.e., the ability to detect non-leaf functions that don't create stack frames? I think we should be able to manage this without objtool on arm64 tbh.

Hm? How else would you ensure all functions honor CONFIG_FRAME_POINTER, and continue to do so indefinitely?

Quoted 11 lines
Quoted 7 lines

But
that will be a lot more work than it took for Ard to write a GCC plugin.

Quoted 1 line

I really think that hard reliance on GCC plugins is foolish

Funny, I feel the same way about hard reliance on objtool :-)

I tend to agree here. I think objtool is a necessary evil (as are compiler plugins, for that matter) which I hope does not spread to other architectures.

I agree that it's a necessary evil, but it may be necessary on arm64 for live patching.

Quoted 11 lines

But the main difference is that the GCC plugin is only ~50 lines (for this particular use case, and minus another 50 lines of boilerplate), whereas objtool (AIUI) duplicates lots and lots of functionality of the compiler, assembler and/or linker, to mangle relocations, create new sections etc etc. Porting this to other architectures is going to be a major maintenance effort, especially when I think of, e.g., 32-bit ARM with its Thumb2 quirks and other idiosyncrasies that are currently hidden in the toolchain. Other architectures should be first class citizens if objtool gains support for them, which means that the x86 people that own it currently are on the hook for testing their changes against architectures they are not familiar with.

Sounds like we could use you as a co-maintainer then :-)

BTW, AFAIK, there are no plans to support live patching for 32-bit ARM.

Signature · 1 line
Josh
#44 lore raw
In reply to Steven Rostedt
Quoted 11 lines · Steven Rostedt
Quoted 4 lines · On Sun, 11 Nov 2018 23:30:55 -0600 Josh Poimboeuf <[email protected]> wrote:
Quoted 1 line

How much of that slowdown is reversed?

In theory, it should reverse all of the slowdown, and actually may even speed it up a little. Steve is working on measuring that now.

When I'm able to get it to work! Hopefully that last patch snippet you posted will help. If not, I'm assuming you'll be in Vancouver this week, and we could sit down and work it out.

Sure, I'm already in Vancouver. Just grab me if you see me, or ping me on IRC/email. Or feel free to send me your patches if it's still giving you trouble.

Quoted 6 lines

That said, I don't expect a 100% improvement. Because the retpoline causes slow down in other areas than just tracing, which is not being fixed by this. I'm expecting a substantial improvement (which I see good improvement with the unoptimized static calls), and hoping for much more with the optimized one (when I get it working). But not 100%, as stated above.

Ah, ok. Makes sense.

Signature · 1 line
Josh