Skip to contentSkip to conversation

jump label v11

PATCH In review Jason Baron 11 participants 64 messages Sep 17, 2010 – Oct 18, 2010

Subsystems (12)
lore
Share & tools
lore
#0 lore raw

cover jump label v11

Hi,

updates in -v11:

-consolidated core jump label changes into a single patch -updated sparc64 patch from David Miller -various cleanups

Thanks to Steve Rostedt for helping me clean up the ftrace changes found in patch 1/10.

thanks,

-Jason

David S. Miller (1):
jump label: add sparc64 support

Jason Baron (9):
jump label: x86 - make dynamic no-op selection available outside of ftrace jump label: make text_poke_early() globally visisble
jump label: base patch
jump label: initialize workqueue tracepoints *before* they are registered jump label: jump_label_text_reserved() to reserve our jump points jump label: tracepoint support
jump label: convert dynamic debug to use jump labels
jump label: x86 support
jump label: add docs

 Documentation/jump-label.txt        |  148 ++++++++++++
 Makefile                            |    5 +
 arch/Kconfig                        |    3 +
 arch/sparc/Kconfig                  |    1 +
 arch/sparc/include/asm/jump_label.h |   32 +++
 arch/sparc/kernel/Makefile          |    2 +
 arch/sparc/kernel/jump_label.c      |   46 ++++
 arch/sparc/kernel/module.c          |    6 +
 arch/x86/Kconfig                    |    1 +
 arch/x86/include/asm/alternative.h  |   11 +
 arch/x86/include/asm/jump_label.h   |   47 ++++
 arch/x86/kernel/Makefile            |    2 +-
 arch/x86/kernel/alternative.c       |   68 ++++++-
 arch/x86/kernel/ftrace.c            |   63 +-----
 arch/x86/kernel/jump_label.c        |   49 ++++
 arch/x86/kernel/kprobes.c           |    3 +-
 arch/x86/kernel/module.c            |    3 +
 arch/x86/kernel/setup.c             |    6 +
 include/asm-generic/vmlinux.lds.h   |   10 +
 include/linux/dynamic_debug.h       |   39 ++--
 include/linux/jump_label.h          |   64 ++++++
 include/linux/module.h              |    5 +-
 include/linux/tracepoint.h          |    5 +-
 kernel/Makefile                     |    2 +-
 kernel/jump_label.c                 |  421 +++++++++++++++++++++++++++++++++++
 kernel/kprobes.c                    |    4 +-
 kernel/module.c                     |    6 +
 kernel/trace/trace_workqueue.c      |   10 +-
 kernel/tracepoint.c                 |   14 +-
 lib/dynamic_debug.c                 |   42 +----
 scripts/Makefile.lib                |   11 +-
 scripts/basic/Makefile              |    2 +-
 scripts/basic/hash.c                |   64 ------
 scripts/gcc-goto.sh                 |    5 +
 34 files changed, 991 insertions(+), 209 deletions(-)
 create mode 100644 Documentation/jump-label.txt
 create mode 100644 arch/sparc/include/asm/jump_label.h
 create mode 100644 arch/sparc/kernel/jump_label.c
 create mode 100644 arch/x86/include/asm/jump_label.h
 create mode 100644 arch/x86/kernel/jump_label.c
 create mode 100644 include/linux/jump_label.h
 create mode 100644 kernel/jump_label.c
 delete mode 100644 scripts/basic/hash.c
 create mode 100644 scripts/gcc-goto.sh
#1 lore raw

PATCH 1/10 jump label v11: make dynamic no-op selection available outside of ftrace

Move Steve's code for finding the best 5-byte no-op from ftrace.c to alternative.c. The idea is that other consumers (in this case jump label) want to make use of that code.

---
 arch/x86/include/asm/alternative.h |    8 ++++
 arch/x86/kernel/alternative.c      |   64 ++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/ftrace.c           |   63 +----------------------------------
 arch/x86/kernel/setup.c            |    6 +++
 4 files changed, 79 insertions(+), 62 deletions(-)
arch/x86/include/asm/alternative.h +8 -0
index bc6abb7..27a35b6 100644
@@ -180,4 +180,12 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 extern void *text_poke(void *addr, const void *opcode, size_t len);
 extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
 
+#if defined(CONFIG_DYNAMIC_FTRACE)
+#define IDEAL_NOP_SIZE_5 5
+extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
+extern void arch_init_ideal_nop5(void);
+#else
+static inline void arch_init_ideal_nop5(void) {}
+#endif
+
 #endif /* _ASM_X86_ALTERNATIVE_H */
arch/x86/kernel/alternative.c +64 -0

67 lines not on this page · raw message

arch/x86/kernel/ftrace.c +1 -62

77 lines not on this page · raw message

arch/x86/kernel/setup.c +6 -0
index 1427c36..fb37842 100644
@@ -112,6 +112,7 @@
 #include <asm/numa_64.h>
 #endif
 #include <asm/mce.h>
+#include <asm/alternative.h>
 
 /*
  * end_pfn only includes RAM, while max_pfn_mapped includes all e820 entries.
@@ -689,6 +690,7 @@ void __init setup_arch(char **cmdline_p)
 {
 	int acpi = 0;
 	int k8 = 0;
+	unsigned long flags;
 
 #ifdef CONFIG_X86_32
 	memcpy(&boot_cpu_data, &new_cpu_data, sizeof(new_cpu_data));
@@ -1036,6 +1038,10 @@ void __init setup_arch(char **cmdline_p)
 	x86_init.oem.banner();
 
 	mcheck_init();
+
+	local_irq_save(flags);
+	arch_init_ideal_nop5();
+	local_irq_restore(flags);
 }
 
 #ifdef CONFIG_X86_32
Signature · 1 line
1.7.1
maintainer Acked-by on v1 1/10
#2 lore raw

Re: [PATCH 01/10] jump label v11: make dynamic no-op selection available outside of ftrace

Quoted 5 lines · Jason Baron

Move Steve's code for finding the best 5-byte no-op from ftrace.c to alternative.c. The idea is that other consumers (in this case jump label) want to make use of that code.

-- Steve

Quoted 210 lines
---
 arch/x86/include/asm/alternative.h |    8 ++++
 arch/x86/kernel/alternative.c      |   64 ++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/ftrace.c           |   63 +----------------------------------
 arch/x86/kernel/setup.c            |    6 +++
 4 files changed, 79 insertions(+), 62 deletions(-)
arch/x86/include/asm/alternative.h +8 -0
index bc6abb7..27a35b6 100644
@@ -180,4 +180,12 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 extern void *text_poke(void *addr, const void *opcode, size_t len);
 extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
 
+#if defined(CONFIG_DYNAMIC_FTRACE)
+#define IDEAL_NOP_SIZE_5 5
+extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
+extern void arch_init_ideal_nop5(void);
+#else
+static inline void arch_init_ideal_nop5(void) {}
+#endif
+
 #endif /* _ASM_X86_ALTERNATIVE_H */
arch/x86/kernel/alternative.c +64 -0
index f65ab8b..1849d80 100644
@@ -641,3 +641,67 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)
 	return addr;
 }
 
+#if defined(CONFIG_DYNAMIC_FTRACE)
+
+unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
+
+void __init arch_init_ideal_nop5(void)
+{
+	extern const unsigned char ftrace_test_p6nop[];
+	extern const unsigned char ftrace_test_nop5[];
+	extern const unsigned char ftrace_test_jmp[];
+	int faulted = 0;
+
+	/*
+	 * There is no good nop for all x86 archs.
+	 * We will default to using the P6_NOP5, but first we
+	 * will test to make sure that the nop will actually
+	 * work on this CPU. If it faults, we will then
+	 * go to a lesser efficient 5 byte nop. If that fails
+	 * we then just use a jmp as our nop. This isn't the most
+	 * efficient nop, but we can not use a multi part nop
+	 * since we would then risk being preempted in the middle
+	 * of that nop, and if we enabled tracing then, it might
+	 * cause a system crash.
+	 *
+	 * TODO: check the cpuid to determine the best nop.
+	 */
+	asm volatile (
+		"ftrace_test_jmp:"
+		"jmp ftrace_test_p6nop\n"
+		"nop\n"
+		"nop\n"
+		"nop\n"  /* 2 byte jmp + 3 bytes */
+		"ftrace_test_p6nop:"
+		P6_NOP5
+		"jmp 1f\n"
+		"ftrace_test_nop5:"
+		".byte 0x66,0x66,0x66,0x66,0x90\n"
+		"1:"
+		".section .fixup, \"ax\"\n"
+		"2:	movl $1, %0\n"
+		"	jmp ftrace_test_nop5\n"
+		"3:	movl $2, %0\n"
+		"	jmp 1b\n"
+		".previous\n"
+		_ASM_EXTABLE(ftrace_test_p6nop, 2b)
+		_ASM_EXTABLE(ftrace_test_nop5, 3b)
+		: "=r"(faulted) : "0" (faulted));
+
+	switch (faulted) {
+	case 0:
+		pr_info("converting mcount calls to 0f 1f 44 00 00\n");
+		memcpy(ideal_nop5, ftrace_test_p6nop, IDEAL_NOP_SIZE_5);
+		break;
+	case 1:
+		pr_info("converting mcount calls to 66 66 66 66 90\n");
+		memcpy(ideal_nop5, ftrace_test_nop5, IDEAL_NOP_SIZE_5);
+		break;
+	case 2:
+		pr_info("converting mcount calls to jmp . + 5\n");
+		memcpy(ideal_nop5, ftrace_test_jmp, IDEAL_NOP_SIZE_5);
+		break;
+	}
+
+}
+#endif
arch/x86/kernel/ftrace.c +1 -62
index cd37469..3afb33f 100644
@@ -257,14 +257,9 @@ do_ftrace_mod_code(unsigned long ip, void *new_code)
 	return mod_code_status;
 }
 
-
-
-
-static unsigned char ftrace_nop[MCOUNT_INSN_SIZE];
-
 static unsigned char *ftrace_nop_replace(void)
 {
-	return ftrace_nop;
+	return ideal_nop5;
 }
 
 static int
@@ -338,62 +333,6 @@ int ftrace_update_ftrace_func(ftrace_func_t func)
 
 int __init ftrace_dyn_arch_init(void *data)
 {
-	extern const unsigned char ftrace_test_p6nop[];
-	extern const unsigned char ftrace_test_nop5[];
-	extern const unsigned char ftrace_test_jmp[];
-	int faulted = 0;
-
-	/*
-	 * There is no good nop for all x86 archs.
-	 * We will default to using the P6_NOP5, but first we
-	 * will test to make sure that the nop will actually
-	 * work on this CPU. If it faults, we will then
-	 * go to a lesser efficient 5 byte nop. If that fails
-	 * we then just use a jmp as our nop. This isn't the most
-	 * efficient nop, but we can not use a multi part nop
-	 * since we would then risk being preempted in the middle
-	 * of that nop, and if we enabled tracing then, it might
-	 * cause a system crash.
-	 *
-	 * TODO: check the cpuid to determine the best nop.
-	 */
-	asm volatile (
-		"ftrace_test_jmp:"
-		"jmp ftrace_test_p6nop\n"
-		"nop\n"
-		"nop\n"
-		"nop\n"  /* 2 byte jmp + 3 bytes */
-		"ftrace_test_p6nop:"
-		P6_NOP5
-		"jmp 1f\n"
-		"ftrace_test_nop5:"
-		".byte 0x66,0x66,0x66,0x66,0x90\n"
-		"1:"
-		".section .fixup, \"ax\"\n"
-		"2:	movl $1, %0\n"
-		"	jmp ftrace_test_nop5\n"
-		"3:	movl $2, %0\n"
-		"	jmp 1b\n"
-		".previous\n"
-		_ASM_EXTABLE(ftrace_test_p6nop, 2b)
-		_ASM_EXTABLE(ftrace_test_nop5, 3b)
-		: "=r"(faulted) : "0" (faulted));
-
-	switch (faulted) {
-	case 0:
-		pr_info("converting mcount calls to 0f 1f 44 00 00\n");
-		memcpy(ftrace_nop, ftrace_test_p6nop, MCOUNT_INSN_SIZE);
-		break;
-	case 1:
-		pr_info("converting mcount calls to 66 66 66 66 90\n");
-		memcpy(ftrace_nop, ftrace_test_nop5, MCOUNT_INSN_SIZE);
-		break;
-	case 2:
-		pr_info("converting mcount calls to jmp . + 5\n");
-		memcpy(ftrace_nop, ftrace_test_jmp, MCOUNT_INSN_SIZE);
-		break;
-	}
-
 	/* The return code is retured via data */
 	*(unsigned long *)data = 0;
 
arch/x86/kernel/setup.c +6 -0
index 1427c36..fb37842 100644
@@ -112,6 +112,7 @@
 #include <asm/numa_64.h>
 #endif
 #include <asm/mce.h>
+#include <asm/alternative.h>
 
 /*
  * end_pfn only includes RAM, while max_pfn_mapped includes all e820 entries.
@@ -689,6 +690,7 @@ void __init setup_arch(char **cmdline_p)
 {
 	int acpi = 0;
 	int k8 = 0;
+	unsigned long flags;
 
 #ifdef CONFIG_X86_32
 	memcpy(&boot_cpu_data, &new_cpu_data, sizeof(new_cpu_data));
@@ -1036,6 +1038,10 @@ void __init setup_arch(char **cmdline_p)
 	x86_init.oem.banner();
 
 	mcheck_init();
+
+	local_irq_save(flags);
+	arch_init_ideal_nop5();
+	local_irq_restore(flags);
 }
 
 #ifdef CONFIG_X86_32
#3 lore raw

[tip:perf/core] jump label: Make dynamic no-op selection available outside of ftrace

In reply to Jason Baron
Commit-ID:  f49aa448561fe9215f43405cac6f31eb86317792
Gitweb:     http://git.kernel.org/tip/f49aa448561fe9215f43405cac6f31eb86317792
Author:     Jason Baron <[email protected]>
AuthorDate: Fri, 17 Sep 2010 11:08:51 -0400
Committer:  Steven Rostedt <[email protected]>
CommitDate: Mon, 20 Sep 2010 18:19:39 -0400

jump label: Make dynamic no-op selection available outside of ftrace

Move Steve's code for finding the best 5-byte no-op from ftrace.c to alternative.c. The idea is that other consumers (in this case jump label) want to make use of that code.

LKML-Reference: <96259ae74172dcac99c0020c249743c523a92e18.1284733808.git.jbaron@redhat.com>

---
 arch/x86/include/asm/alternative.h |    8 ++++
 arch/x86/kernel/alternative.c      |   64 ++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/ftrace.c           |   63 +----------------------------------
 arch/x86/kernel/setup.c            |    6 +++
 4 files changed, 79 insertions(+), 62 deletions(-)
arch/x86/include/asm/alternative.h +8 -0
index bc6abb7..27a35b6 100644
@@ -180,4 +180,12 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 extern void *text_poke(void *addr, const void *opcode, size_t len);
 extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
 
+#if defined(CONFIG_DYNAMIC_FTRACE)
+#define IDEAL_NOP_SIZE_5 5
+extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
+extern void arch_init_ideal_nop5(void);
+#else
+static inline void arch_init_ideal_nop5(void) {}
+#endif
+
 #endif /* _ASM_X86_ALTERNATIVE_H */
arch/x86/kernel/alternative.c +64 -0

67 lines not on this page · raw message

arch/x86/kernel/ftrace.c +1 -62

77 lines not on this page · raw message

arch/x86/kernel/setup.c +6 -0
index c3a4fbb..00e1678 100644
@@ -112,6 +112,7 @@
 #include <asm/numa_64.h>
 #endif
 #include <asm/mce.h>
+#include <asm/alternative.h>
 
 /*
  * end_pfn only includes RAM, while max_pfn_mapped includes all e820 entries.
@@ -726,6 +727,7 @@ void __init setup_arch(char **cmdline_p)
 {
 	int acpi = 0;
 	int k8 = 0;
+	unsigned long flags;
 
 #ifdef CONFIG_X86_32
 	memcpy(&boot_cpu_data, &new_cpu_data, sizeof(new_cpu_data));
@@ -1071,6 +1073,10 @@ void __init setup_arch(char **cmdline_p)
 	x86_init.oem.banner();
 
 	mcheck_init();
+
+	local_irq_save(flags);
+	arch_init_ideal_nop5();
+	local_irq_restore(flags);
 }
 
 #ifdef CONFIG_X86_32
#4 lore raw

PATCH 2/10 jump label v11: make text_poke_early() globally visisble

In reply to Jason Baron

Make text_poke_early available outside of alternative.c. The jump label patchset wants to make use of it in order to set up the optimal no-op sequences at run-time.

---
 arch/x86/include/asm/alternative.h |    2 ++
 arch/x86/kernel/alternative.c      |    4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/include/asm/alternative.h +2 -0
index 27a35b6..634bf78 100644
@@ -160,6 +160,8 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 #define __parainstructions_end	NULL
 #endif
 
+extern void *text_poke_early(void *addr, const void *opcode, size_t len);
+
 /*
  * Clear and restore the kernel write-protection flag on the local CPU.
  * Allows the kernel to edit read-only pages.
arch/x86/kernel/alternative.c +2 -2
index 1849d80..083bd01 100644
@@ -195,7 +195,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len)
 
 extern struct alt_instr __alt_instructions[], __alt_instructions_end[];
 extern s32 __smp_locks[], __smp_locks_end[];
-static void *text_poke_early(void *addr, const void *opcode, size_t len);
+void *text_poke_early(void *addr, const void *opcode, size_t len);
 
 /* Replace instructions with better alternatives for this CPU type.
    This runs before SMP is initialized to avoid SMP problems with
@@ -522,7 +522,7 @@ void __init alternative_instructions(void)
  * instructions. And on the local CPU you need to be protected again NMI or MCE
  * handlers seeing an inconsistent instruction while you patch.
  */
-static void *__init_or_module text_poke_early(void *addr, const void *opcode,
+void *__init_or_module text_poke_early(void *addr, const void *opcode,
 					      size_t len)
 {
 	unsigned long flags;
Signature · 1 line
1.7.1
#5 lore raw

[tip:perf/core] jump label: Make text_poke_early() globally visible

Commit-ID:  fa6f2cc77081792e4edca9168420a3422299ef15
Gitweb:     http://git.kernel.org/tip/fa6f2cc77081792e4edca9168420a3422299ef15
Author:     Jason Baron <[email protected]>
AuthorDate: Fri, 17 Sep 2010 11:08:56 -0400
Committer:  Steven Rostedt <[email protected]>
CommitDate: Mon, 20 Sep 2010 18:19:51 -0400

jump label: Make text_poke_early() globally visible

Make text_poke_early available outside of alternative.c. The jump label patchset wants to make use of it in order to set up the optimal no-op sequences at run-time.

LKML-Reference: <04cfddf2ba77bcabfc3e524f1849d871d6a1cf9d.1284733808.git.jbaron@redhat.com>

---
 arch/x86/include/asm/alternative.h |    2 ++
 arch/x86/kernel/alternative.c      |    4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/include/asm/alternative.h +2 -0
index 27a35b6..634bf78 100644
@@ -160,6 +160,8 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 #define __parainstructions_end	NULL
 #endif
 
+extern void *text_poke_early(void *addr, const void *opcode, size_t len);
+
 /*
  * Clear and restore the kernel write-protection flag on the local CPU.
  * Allows the kernel to edit read-only pages.
arch/x86/kernel/alternative.c +2 -2
index 1849d80..083bd01 100644
@@ -195,7 +195,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len)
 
 extern struct alt_instr __alt_instructions[], __alt_instructions_end[];
 extern s32 __smp_locks[], __smp_locks_end[];
-static void *text_poke_early(void *addr, const void *opcode, size_t len);
+void *text_poke_early(void *addr, const void *opcode, size_t len);
 
 /* Replace instructions with better alternatives for this CPU type.
    This runs before SMP is initialized to avoid SMP problems with
@@ -522,7 +522,7 @@ void __init alternative_instructions(void)
  * instructions. And on the local CPU you need to be protected again NMI or MCE
  * handlers seeing an inconsistent instruction while you patch.
  */
-static void *__init_or_module text_poke_early(void *addr, const void *opcode,
+void *__init_or_module text_poke_early(void *addr, const void *opcode,
 					      size_t len)
 {
 	unsigned long flags;
#6 lore raw

PATCH 3/10 jump label v11: base patch

In reply to Jason Baron

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

---
 Makefile                           |    5 +
 arch/Kconfig                       |    3 +
 arch/x86/include/asm/alternative.h |    3 +-
 arch/x86/kernel/alternative.c      |    2 +-
 include/asm-generic/vmlinux.lds.h  |   10 +
 include/linux/jump_label.h         |   58 ++++++
 include/linux/module.h             |    5 +-
 kernel/Makefile                    |    2 +-
 kernel/jump_label.c                |  338 ++++++++++++++++++++++++++++++++++++
 kernel/kprobes.c                   |    1 +
 kernel/module.c                    |    6 +
 scripts/gcc-goto.sh                |    5 +
 12 files changed, 434 insertions(+), 4 deletions(-)
 create mode 100644 include/linux/jump_label.h
 create mode 100644 kernel/jump_label.c
 create mode 100644 scripts/gcc-goto.sh
Makefile +5 -0
index 92ab33f..a906378 100644
@@ -591,6 +591,11 @@ KBUILD_CFLAGS	+= $(call cc-option,-fno-strict-overflow)
 # conserve stack if available
 KBUILD_CFLAGS   += $(call cc-option,-fconserve-stack)
 
+# check for 'asm goto'
+ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC)), y)
+	KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
+endif
+
 # Add user supplied CPPFLAGS, AFLAGS and CFLAGS as the last assignments
 # But warn user when we do so
 warn-assign = \
arch/Kconfig +3 -0
index 4877a8c..1462d84 100644
@@ -158,4 +158,7 @@ config HAVE_PERF_EVENTS_NMI
 	  subsystem.  Also has support for calculating CPU cycle events
 	  to determine how many clock cycles in a given period.
 
+config HAVE_ARCH_JUMP_LABEL
+	bool
+
 source "kernel/gcov/Kconfig"
arch/x86/include/asm/alternative.h +2 -1
index 634bf78..76561d2 100644
@@ -4,6 +4,7 @@
 #include <linux/types.h>
 #include <linux/stddef.h>
 #include <linux/stringify.h>
+#include <linux/jump_label.h>
 #include <asm/asm.h>
 
 /*
@@ -182,7 +183,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);
 extern void *text_poke(void *addr, const void *opcode, size_t len);
 extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
 
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
 #define IDEAL_NOP_SIZE_5 5
 extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
 extern void arch_init_ideal_nop5(void);
arch/x86/kernel/alternative.c +1 -1
index 083bd01..cb0e6d3 100644
@@ -641,7 +641,7 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)
 	return addr;
 }
 
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
 
 unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
 
include/asm-generic/vmlinux.lds.h +10 -0
index 8a92a17..ef2af99 100644
@@ -220,6 +220,8 @@
 									\
 	BUG_TABLE							\
 									\
+	JUMP_TABLE							\
+									\
 	/* PCI quirks */						\
 	.pci_fixup        : AT(ADDR(.pci_fixup) - LOAD_OFFSET) {	\
 		VMLINUX_SYMBOL(__start_pci_fixups_early) = .;		\
@@ -563,6 +565,14 @@
 #define BUG_TABLE
 #endif
 
+#define JUMP_TABLE							\
+	. = ALIGN(8);							\
+	__jump_table : AT(ADDR(__jump_table) - LOAD_OFFSET) {		\
+		VMLINUX_SYMBOL(__start___jump_table) = .;		\
+		*(__jump_table)						\
+		VMLINUX_SYMBOL(__stop___jump_table) = .;		\
+	}
+
 #ifdef CONFIG_PM_TRACE
 #define TRACEDATA							\
 	. = ALIGN(4);							\
include/linux/jump_label.h new file +58 -0

58 lines not on this page · raw message

include/linux/module.h +4 -1
index 8a6b9fd..403ac26 100644
@@ -350,7 +350,10 @@ struct module
 	struct tracepoint *tracepoints;
 	unsigned int num_tracepoints;
 #endif
-
+#ifdef HAVE_JUMP_LABEL
+	struct jump_entry *jump_entries;
+	unsigned int num_jump_entries;
+#endif
 #ifdef CONFIG_TRACING
 	const char **trace_bprintk_fmt_start;
 	unsigned int num_trace_bprintk_fmt;
kernel/Makefile +1 -1
index 92b7420..8a0feb7 100644
@@ -10,7 +10,7 @@ obj-y     = sched.o fork.o exec_domain.o panic.o printk.o \
 	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \
 	    hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
 	    notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \
-	    async.o range.o
+	    async.o range.o jump_label.o
 obj-y += groups.o
 
 ifdef CONFIG_FUNCTION_TRACER
kernel/jump_label.c new file +338 -0

338 lines not on this page · raw message

kernel/kprobes.c +1 -0
index 282035f..798adfa 100644
@@ -47,6 +47,7 @@
 #include <linux/memory.h>
 #include <linux/ftrace.h>
 #include <linux/cpu.h>
+#include <linux/jump_label.h>
 
 #include <asm-generic/sections.h>
 #include <asm/cacheflush.h>
kernel/module.c +6 -0
index d0b5f8d..eba1341 100644
@@ -55,6 +55,7 @@
 #include <linux/async.h>
 #include <linux/percpu.h>
 #include <linux/kmemleak.h>
+#include <linux/jump_label.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/module.h>
@@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)
 					sizeof(*mod->tracepoints),
 					&mod->num_tracepoints);
 #endif
+#ifdef HAVE_JUMP_LABEL
+	mod->jump_entries = section_objs(info, "__jump_table",
+					sizeof(*mod->jump_entries),
+					&mod->num_jump_entries);
+#endif
 #ifdef CONFIG_EVENT_TRACING
 	mod->trace_events = section_objs(info, "_ftrace_events",
 					 sizeof(*mod->trace_events),
scripts/gcc-goto.sh new file +5 -0
new file mode 100644
index 0000000..8e82424
@@ -0,0 +1,5 @@
+#!/bin/sh
+# Test for gcc 'asm goto' suport
+# Copyright (C) 2010, Jason Baron <[email protected]>
+
+echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
Signature · 1 line
1.7.1
maintainer Acked-by on v1 3/10
#7 lore raw

Re: [PATCH 03/10] jump label v11: base patch

From: Jason Baron <[email protected]>
Date: Fri, 17 Sep 2010 11:09:00 -0400

Quoted 8 lines

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

maintainer
#8 lore raw

Re: [PATCH 03/10] jump label v11: base patch

In reply to Jason Baron
Quoted 120 lines · Jason Baron

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

---

kernel/jump_label.c new file +110 -0
+static struct jump_label_entry *add_jump_label_entry(jump_label_t key, int nr_entries, struct jump_entry *table)
+{
+	struct hlist_head *head;
+	struct jump_label_entry *e;
+	u32 hash;
+
+	e = get_jump_label_entry(key);
+	if (e)
+		return ERR_PTR(-EEXIST);
+
+	e = kmalloc(sizeof(struct jump_label_entry), GFP_KERNEL);
+	if (!e)
+		return ERR_PTR(-ENOMEM);
+
+	hash = jhash((void *)&key, sizeof(jump_label_t), 0);
+	head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
+	e->key = key;
+	e->table = table;
+	e->nr_entries = nr_entries;
+	INIT_HLIST_HEAD(&(e->modules));
+	hlist_add_head(&e->hlist, head);
+	return e;
+}
+
+static int build_jump_label_hashtable(struct jump_entry *start, struct jump_entry *stop)
+{
+	struct jump_entry *iter, *iter_begin;
+	struct jump_label_entry *entry;
+	int count;
+
+	sort_jump_label_entries(start, stop);
+	iter = start;
+	while (iter < stop) {
+		entry = get_jump_label_entry(iter->key);
+		if (!entry) {
+			iter_begin = iter;
+			count = 0;
+			while ((iter < stop) &&
+				(iter->key == iter_begin->key)) {
+				iter++;
+				count++;
+			}
+			entry = add_jump_label_entry(iter_begin->key,
+							count, iter_begin);
+			if (IS_ERR(entry))
+				return PTR_ERR(entry);
+		 } else {
+			WARN_ONCE(1, KERN_ERR "build_jump_hashtable: unexpected entry!\n");
+			return -1;
+		}
+	}
+	return 0;
+}
+
+/***
+ * jump_label_update - update jump label text
+ * @key -  key value associated with a a jump label
+ * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
+ *
+ * Will enable/disable the jump for jump label @key, depending on the
+ * value of @type.
+ *
+ */
+
+void jump_label_update(unsigned long key, enum jump_label_type type)
+{
+	struct jump_entry *iter;
+	struct jump_label_entry *entry;
+	struct hlist_node *module_node;
+	struct jump_label_module_entry *e_module;
+	int count;
+
+	mutex_lock(&jump_label_mutex);
+	entry = get_jump_label_entry((jump_label_t)key);
+	if (entry) {
+		count = entry->nr_entries;
+		iter = entry->table;
+		while (count--) {
+			if (kernel_text_address(iter->code))
+				arch_jump_label_transform(iter, type);
+			iter++;
+		}
+		/* eanble/disable jump labels in modules */
+		hlist_for_each_entry(e_module, module_node, &(entry->modules),
+							hlist) {
+			count = e_module->nr_entries;
+			iter = e_module->table;
+			while (count--) {
+				if (kernel_text_address(iter->code))
+					arch_jump_label_transform(iter, type);
+				iter++;
+			}
+		}
+	}
+	mutex_unlock(&jump_label_mutex);
+}
+
+static __init int init_jump_label(void)
+{
+	int ret;
+	struct jump_entry *iter_start = __start___jump_table;
+	struct jump_entry *iter_stop = __stop___jump_table;
+	struct jump_entry *iter;
+
+	mutex_lock(&jump_label_mutex);
+	ret = build_jump_label_hashtable(__start___jump_table,
+					 __stop___jump_table);
+	iter = iter_start;
+	while (iter < iter_stop) {
+		arch_jump_label_text_poke_early(iter->code);

kernel/jump_label.c line 199

Quoted 10 lines
kernel/jump_label.c new file +10 -0
+		iter++;
+	}
+	mutex_unlock(&jump_label_mutex);
+	return ret;
+}
+early_initcall(init_jump_label);
+
+#ifdef CONFIG_MODULES
+
+static struct jump_label_module_entry *add_jump_label_module_entry(struct jump_label_entry *entry, struct jump_entry *iter_begin, int count, struct module *mod)

Eek!!

I'm not that strict on the 80 character limit, but the above can wrap it twice!!!

I'll fix these extended lines myself.

-- Steve

Quoted 186 lines
kernel/jump_label.c new file +139 -0
+{
+	struct jump_label_module_entry *e;
+
+	e = kmalloc(sizeof(struct jump_label_module_entry), GFP_KERNEL);
+	if (!e)
+		return ERR_PTR(-ENOMEM);
+	e->mod = mod;
+	e->nr_entries = count;
+	e->table = iter_begin;
+	hlist_add_head(&e->hlist, &entry->modules);
+	return e;
+}
+
+static int add_jump_label_module(struct module *mod)
+{
+	struct jump_entry *iter, *iter_begin;
+	struct jump_label_entry *entry;
+	struct jump_label_module_entry *module_entry;
+	int count;
+
+	/* if the module doesn't have jump label entries, just return */
+	if (!mod->num_jump_entries)
+		return 0;
+
+	sort_jump_label_entries(mod->jump_entries,
+				mod->jump_entries + mod->num_jump_entries);
+	iter = mod->jump_entries;
+	while (iter < mod->jump_entries + mod->num_jump_entries) {
+		entry = get_jump_label_entry(iter->key);
+		iter_begin = iter;
+		count = 0;
+		while ((iter < mod->jump_entries + mod->num_jump_entries) &&
+			(iter->key == iter_begin->key)) {
+				iter++;
+				count++;
+		}
+		if (!entry) {
+			entry = add_jump_label_entry(iter_begin->key, 0, NULL);
+			if (IS_ERR(entry))
+				return PTR_ERR(entry);
+		}
+		module_entry = add_jump_label_module_entry(entry, iter_begin,
+							   count, mod);
+		if (IS_ERR(module_entry))
+			return PTR_ERR(module_entry);
+	}
+	return 0;
+}
+
+static void remove_jump_label_module(struct module *mod)
+{
+	struct hlist_head *head;
+	struct hlist_node *node, *node_next, *module_node, *module_node_next;
+	struct jump_label_entry *e;
+	struct jump_label_module_entry *e_module;
+	int i;
+
+	/* if the module doesn't have jump label entries, just return */
+	if (!mod->num_jump_entries)
+		return;
+
+	for (i = 0; i < JUMP_LABEL_TABLE_SIZE; i++) {
+		head = &jump_label_table[i];
+		hlist_for_each_entry_safe(e, node, node_next, head, hlist) {
+			hlist_for_each_entry_safe(e_module, module_node,
+						  module_node_next,
+						  &(e->modules), hlist) {
+				if (e_module->mod == mod) {
+					hlist_del(&e_module->hlist);
+					kfree(e_module);
+				}
+			}
+			if (hlist_empty(&e->modules) && (e->nr_entries == 0)) {
+				hlist_del(&e->hlist);
+				kfree(e);
+			}
+		}
+	}
+}
+
+static int jump_label_module_notify(struct notifier_block *self, unsigned long val, void *data)
+{
+	struct module *mod = data;
+	int ret = 0;
+
+	switch (val) {
+	case MODULE_STATE_COMING:
+		mutex_lock(&jump_label_mutex);
+		ret = add_jump_label_module(mod);
+		if (ret)
+			remove_jump_label_module(mod);
+		mutex_unlock(&jump_label_mutex);
+		break;
+	case MODULE_STATE_GOING:
+		mutex_lock(&jump_label_mutex);
+		remove_jump_label_module(mod);
+		mutex_unlock(&jump_label_mutex);
+		break;
+	}
+	return ret;
+}
+
+/***
+ * apply_jump_label_nops - patch module jump labels with arch_get_jump_label_nop()
+ * @mod: module to patch
+ *
+ * Allow for run-time selection of the optimal nops. Before the module
+ * loads patch these with arch_get_jump_label_nop(), which is specified by
+ * the arch specific jump label code.
+ */
+void jump_label_apply_nops(struct module *mod)
+{
+	struct jump_entry *iter;
+
+	/* if the module doesn't have jump label entries, just return */
+	if (!mod->num_jump_entries)
+		return;
+
+	iter = mod->jump_entries;
+	while (iter < mod->jump_entries + mod->num_jump_entries) {
+		arch_jump_label_text_poke_early(iter->code);
+		iter++;
+	}
+}
+
+struct notifier_block jump_label_module_nb = {
+	.notifier_call = jump_label_module_notify,
+	.priority = 0,
+};
+
+static __init int init_jump_label_module(void)
+{
+	return register_module_notifier(&jump_label_module_nb);
+}
+early_initcall(init_jump_label_module);
+
+#endif /* CONFIG_MODULES */
+
+#endif
kernel/kprobes.c +1 -0
index 282035f..798adfa 100644
@@ -47,6 +47,7 @@
 #include <linux/memory.h>
 #include <linux/ftrace.h>
 #include <linux/cpu.h>
+#include <linux/jump_label.h>
 
 #include <asm-generic/sections.h>
 #include <asm/cacheflush.h>
kernel/module.c +6 -0
index d0b5f8d..eba1341 100644
@@ -55,6 +55,7 @@
 #include <linux/async.h>
 #include <linux/percpu.h>
 #include <linux/kmemleak.h>
+#include <linux/jump_label.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/module.h>
@@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)
 					sizeof(*mod->tracepoints),
 					&mod->num_tracepoints);
 #endif
+#ifdef HAVE_JUMP_LABEL
+	mod->jump_entries = section_objs(info, "__jump_table",
+					sizeof(*mod->jump_entries),
+					&mod->num_jump_entries);
+#endif
 #ifdef CONFIG_EVENT_TRACING
 	mod->trace_events = section_objs(info, "_ftrace_events",
 					 sizeof(*mod->trace_events),
scripts/gcc-goto.sh new file +5 -0
new file mode 100644
index 0000000..8e82424
@@ -0,0 +1,5 @@
+#!/bin/sh
+# Test for gcc 'asm goto' suport
+# Copyright (C) 2010, Jason Baron <[email protected]>
+
+echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
#9 lore raw

Re: [PATCH 03/10] jump label v11: base patch

In reply to Jason Baron

include/linux/jump_label.h line 25

Quoted 5 lines · Jason Baron
include/linux/jump_label.h new file +5 -0
+extern void arch_jump_label_transform(struct jump_entry *entry,
+				 enum jump_label_type type);
+extern void jump_label_update(unsigned long key, enum jump_label_type type);
+extern void jump_label_apply_nops(struct module *mod);
+extern void arch_jump_label_text_poke_early(jump_label_t addr);

These function names are too long.

Also it would be better if the types for the pointers are kept instead of casting to unsigned long. All the variables are ints right?

kernel/jump_label.c line 21

Quoted 3 lines
kernel/jump_label.c new file +3 -0
+#define JUMP_LABEL_HASH_BITS 6
+#define JUMP_LABEL_TABLE_SIZE (1 << JUMP_LABEL_HASH_BITS)
+static struct hlist_head jump_label_table[JUMP_LABEL_TABLE_SIZE];

It's not clear to me why this hash table is needed. There should not be that many trace points, is it that big a problem to simply walk all the sections when something is changed?

Or maybe the sections could be just sorted and a binary search used like with exception tables.

I suspect that would simplify a lot of code.

Overall I like the idea, but the current code is too complicated for the benefit I think.

Can it be put on a diet?

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#10 lore raw

Re: [PATCH 03/10] jump label v11: base patch

Quoted 9 lines · Andi Kleen

include/linux/jump_label.h line 25

Quoted 5 lines · Jason Baron
include/linux/jump_label.h new file +5 -0
+extern void arch_jump_label_transform(struct jump_entry *entry,
+				 enum jump_label_type type);
+extern void jump_label_update(unsigned long key, enum jump_label_type type);
+extern void jump_label_apply_nops(struct module *mod);
+extern void arch_jump_label_text_poke_early(jump_label_t addr);

These function names are too long.

Quoted 4 lines

Also it would be better if the types for the pointers are kept instead of casting to unsigned long. All the variables are ints right?

So far, yes. But I didn't want to force this, in case the users of the API wanted to use other types. But I'm ok with 'int *' here.

Quoted 11 lines

kernel/jump_label.c line 21

Quoted 3 lines
kernel/jump_label.c new file +3 -0
+#define JUMP_LABEL_HASH_BITS 6
+#define JUMP_LABEL_TABLE_SIZE (1 << JUMP_LABEL_HASH_BITS)
+static struct hlist_head jump_label_table[JUMP_LABEL_TABLE_SIZE];

It's not clear to me why this hash table is needed. There should not be that many trace points, is it that big a problem to simply walk all the sections when something is changed?

Or maybe the sections could be just sorted and a binary search used like with exception tables.

Quoted 8 lines

I suspect that would simplify a lot of code.

Overall I like the idea, but the current code is too complicated for the benefit I think.

Can it be put on a diet?

-Andi

So there are ~150 tracepoints, but this code is also being proposed for use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.

Also, I think the hash table deals nicely with modules. I create a linked list of only those module sections that are relevant to each hash bucket. If you search through all the section on each enable/disable, its going to be proportional to the number of modules as well.

thanks,

-Jason

#11 lore raw

Re: [PATCH 03/10] jump label v11: base patch

Quoted 5 lines

So there are ~150 tracepoints, but this code is also being proposed for use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.

Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)

Quoted 1 line

Also, I think the hash table deals nicely with modules.

Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.

-Andi

#12 lore raw

Re: [PATCH 03/10] jump label v11: base patch

Quoted 9 lines · Andi Kleen
Quoted 3 lines

So there are ~150 tracepoints, but this code is also being proposed for use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.

Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)

yes, but then for each module, I need to again search through the table to find the correct entires. Although there wouldn't be a 150 tracepoints to search for in each module there are a number of tracepoints that are used by most modules, such as kmalloc, kfree...plus any other users of this infrastructure.

Quoted 7 lines
Quoted 1 line

Also, I think the hash table deals nicely with modules.

Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.

-Andi

its right to realize that the enable/disable paths are not at the heart of this optimization. in fact, the initial few postings of this patchset, didn't include the hash table, but the added support for modules lead me to the current design.

thanks,

-Jason

#13 lore raw

Re: [PATCH 03/10] jump label v11: base patch

In reply to Andi Kleen
Quoted 6 lines · Andi Kleen
Quoted 3 lines

[someone, identity censored by Andi] wrote:

Also, I think the hash table deals nicely with modules.

Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.

Uhm, no - you are not making it simpler, you are making it stupider.

Using a list walk for even 100 items is already borderline horrible and tasteless. We dont want crap code.

Thanks,

Ingo

maintainer
#14 lore raw

Re: [PATCH 03/10] jump label v11: base patch

In reply to Andi Kleen
Quoted 9 lines · Andi Kleen
Quoted 4 lines

So there are ~150 tracepoints, but this code is also being proposed for use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.

Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)

xfs has > 100 tracepoints

Quoted 5 lines
Quoted 2 lines

Also, I think the hash table deals nicely with modules.

Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.

I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.

Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.

-- Steve

#15 lore raw

Re: [PATCH 03/10] jump label v11: base patch

Quoted 7 lines · Andi Kleen
Quoted 6 lines
Quoted 2 lines

So there are ~150 tracepoints, but this code is also being proposed

for

Quoted 2 lines

use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.

Quoted 6 lines
Quoted 4 lines

Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)

xfs has > 100 tracepoints

Doesn

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

Also, I think the hash table deals nicely with modules.

Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.

I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.

The only time you add one is when you load a module, right? When you do that you only sort the section of the new module.

Quoted 2 lines

Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.

Well problem is that the code is very complicated today. I suspect this could be done much simpler if it wasn't so overengin

-Andi

maintainer
#16 lore raw

Re: [PATCH 03/10] jump label v11: base patch

Quoted 7 lines · Andi Kleen
Quoted 7 lines · Andi Kleen
Quoted 6 lines
Quoted 2 lines

So there are ~150 tracepoints, but this code is also being proposed

for

Quoted 2 lines

use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)

xfs has > 100 tracepoints

Doesn

I suppose you were missing a 't'.

Anyway:

 $ find fs/xfs/ -name "*.c" ! -type d | xargs grep "[ ^I]trace_" | wc -l
 313

The jump label occurs at the calling sight, not for defined tracepoints (which can be used in multiple places).

Also take a look at fs/xfs/linux-2.6/xfs_trace.h, you will be surprised.

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

Also, I think the hash table deals nicely with modules.

Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.

Quoted 6 lines
Quoted 3 lines

I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.

The only time you add one is when you load a module, right? When you do that you only sort the section of the new module.

And on removing a module.

Quoted 7 lines
Quoted 3 lines

Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.

Well problem is that the code is very complicated today. I suspect this could be done much simpler if it wasn't so overengin

Perhaps it can be cleaned up. But I have no issues with it now, and using a hash (basic data structures 101) is not where the complexity comes in.

-- Steve

#17 lore raw

Re: [PATCH 03/10] jump label v11: base patch

Quoted 21 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 7 lines · Andi Kleen
Quoted 7 lines · Andi Kleen
Quoted 6 lines
Quoted 2 lines

So there are ~150 tracepoints, but this code is also being proposed

for

Quoted 2 lines

use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines

Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)

xfs has > 100 tracepoints

Doesn

I suppose you were missing a 't'.

Anyway:

Quoted 7 lines
 $ find fs/xfs/ -name "*.c" ! -type d | xargs grep "[ ^I]trace_" | wc -l
 313

The jump label occurs at the calling sight, not for defined tracepoints (which can be used in multiple places).

Also take a look at fs/xfs/linux-2.6/xfs_trace.h, you will be surprised.

Yep, I'd be surprised to see how many tracepoints we can end up having with stuff like kmem tracing (trace_kmalloc). Each instance of the inline function will generate an entry. (!)

Quoted 22 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

Also, I think the hash table deals nicely with modules.

Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.

Quoted 6 lines
Quoted 3 lines

I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.

The only time you add one is when you load a module, right? When you do that you only sort the section of the new module.

And on removing a module.

Quoted 4 lines
Quoted 3 lines

Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.

Quoted 7 lines
Quoted 3 lines

Well problem is that the code is very complicated today. I suspect this could be done much simpler if it wasn't so overengin

Perhaps it can be cleaned up. But I have no issues with it now, and using a hash (basic data structures 101) is not where the complexity comes in.

I agree with Steven, Peter and Jason: due to the large amount of tracepoints we can end up patching, we should keep the hash tables. This code is very similar to what I have in the tracepoints already and in the immediate values. So this code is solid and has been tested over a large user base for quite some time already.

One change I would recommend is to use a separate memory pool to allocate the struct jump_label_entry, to favor better locality. I did not do it in tracepoints and markers because each entry have a variable length, but given that struct jump_label_entry seems to be fixed-size, then we should definitely go for a kmem_cache_alloc().

Thanks,

Mathieu

Quoted 4 lines

-- Steve

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#18 lore raw

Re: [PATCH 03/10] jump label v11: base patch

Quoted 2 lines

I agree with Steven, Peter and Jason: due to the large amount of tracepoints we can end up patching, we should keep the hash tables. This

I suspect when it's cache cold the hash tables will be actually slower. As a general rule memory bloat = slow.

Quoted 3 lines

code is very similar to what I have in the tracepoints already and in the immediate values. So this code is solid and has been tested over a large user base for quite some time already.

FWIW "We always did it this way" is not a good argument in engineering discussions.

Quoted 5 lines

One change I would recommend is to use a separate memory pool to allocate the struct jump_label_entry, to favor better locality. I did not do it in tracepoints and markers because each entry have a variable length, but given that struct jump_label_entry seems to be fixed-size, then we should definitely go for a kmem_cache_alloc().

Yes even more complexity, great idea.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#19 lore raw

Re: [PATCH 03/10] jump label v11: base patch

In reply to Steven Rostedt
Quoted 4 lines
Quoted 2 lines

Doesn

I suppose you were missing a 't'.

Sorry sentence was eaten (I had to type it twice because the webmail thingy eat it first try and probably didn't finish
it second time). I did not doubt your trace point number :)

What I meant is that the number of trace points does not really matter much for binary search, unless you're going to gigantic numbers. 100 is still searched in only a few tries.

What matters is just how many tables you need to walk and that only scales with the number of modules, not the number of trace points.

for ( each module )
binary search in section

Advantage: no additional memory, no setup code other than a simple sort, less code overall.

Also most modules are small so a binary search try is likely very fast.

The methology is also long proven for *_user() exception handling and always worked well here.

Quoted 4 lines
Quoted 2 lines

The only time you add one is when you load a module, right? When you do that you only sort the section of the new module.

And on removing a module.

On removing you simply free the module table, no real work to do.

Quoted 3 lines

Perhaps it can be cleaned up. But I have no issues with it now, and using a hash (basic data structures 101) is not where the complexity comes in.

Hash tables are not complex in themselves, but all the code to maintain them still adds up. I think considerable parts of the new code were simply that. IMHO that can be done simpler at no real loss of functionality or performance.

-Andi

#20 lore raw

Re: [PATCH 03/10] jump label v11: base patch

In reply to Steven Rostedt
Quoted 6 lines · Steven Rostedt
Quoted 5 lines
Quoted 2 lines

Also, I think the hash table deals nicely with modules.

Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.

Quoted 7 lines

I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.

Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.

If all you need is exact matches -- as in this case -- hashes are generally faster than binary anyway.

-hpa

#21 lore raw

Re: [PATCH 03/10] jump label v11: base patch

Quoted 4 lines

If all you need is exact matches -- as in this case -- hashes are generally faster than binary anyway.

Actually only if you have the whole thing in cache. The binary search has a much nicer access pattern if it's cache cold.

And the footprint of directly accessing the section than having separate large list and tables is also significantly more compared to a compact table.

But I don't think the access speed really matters here. Toggling trace points should be rare and is not a critical path operation. What matters is runtime overhead and simplicity.

Right now on the simplicity front there is a problem.

-Andi

#22 lore raw

Re: [PATCH 03/10] jump label v11: base patch

In reply to Steven Rostedt
Quoted 7 lines
Quoted 5 lines

Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)

xfs has > 100 tracepoints

No problem for binary search.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

Also, I think the hash table deals nicely with modules.

Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.

I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.

You only add trace points with new modules right?
In this case you only sort the section of the new module, nothing else. And only once when you load it.

Quoted 2 lines

Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.

The problem I see is that there's a lot of hashing related code and a lot of memory overhead. I suspect with inplace access everything would be much simpler and less overhead.

For me the current implementation simply seems overengineered.

-Andi

#23 lore raw

Re: [PATCH 03/10] jump label v11: base patch

In reply to Jason Baron
Quoted 215 lines · Jason Baron

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

---
 Makefile                           |    5 +
 arch/Kconfig                       |    3 +
 arch/x86/include/asm/alternative.h |    3 +-
 arch/x86/kernel/alternative.c      |    2 +-
 include/asm-generic/vmlinux.lds.h  |   10 +
 include/linux/jump_label.h         |   58 ++++++
 include/linux/module.h             |    5 +-
 kernel/Makefile                    |    2 +-
 kernel/jump_label.c                |  338 ++++++++++++++++++++++++++++++++++++
 kernel/kprobes.c                   |    1 +
 kernel/module.c                    |    6 +
 scripts/gcc-goto.sh                |    5 +
 12 files changed, 434 insertions(+), 4 deletions(-)
 create mode 100644 include/linux/jump_label.h
 create mode 100644 kernel/jump_label.c
 create mode 100644 scripts/gcc-goto.sh
Makefile +5 -0
index 92ab33f..a906378 100644
@@ -591,6 +591,11 @@ KBUILD_CFLAGS	+= $(call cc-option,-fno-strict-overflow)
 # conserve stack if available
 KBUILD_CFLAGS   += $(call cc-option,-fconserve-stack)
 
+# check for 'asm goto'
+ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC)), y)
+	KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
+endif
+
 # Add user supplied CPPFLAGS, AFLAGS and CFLAGS as the last assignments
 # But warn user when we do so
 warn-assign = \
arch/Kconfig +3 -0
index 4877a8c..1462d84 100644
@@ -158,4 +158,7 @@ config HAVE_PERF_EVENTS_NMI
 	  subsystem.  Also has support for calculating CPU cycle events
 	  to determine how many clock cycles in a given period.
 
+config HAVE_ARCH_JUMP_LABEL
+	bool
+
 source "kernel/gcov/Kconfig"
arch/x86/include/asm/alternative.h +2 -1
index 634bf78..76561d2 100644
@@ -4,6 +4,7 @@
 #include <linux/types.h>
 #include <linux/stddef.h>
 #include <linux/stringify.h>
+#include <linux/jump_label.h>
 #include <asm/asm.h>
 
 /*
@@ -182,7 +183,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);
 extern void *text_poke(void *addr, const void *opcode, size_t len);
 extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
 
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
 #define IDEAL_NOP_SIZE_5 5
 extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
 extern void arch_init_ideal_nop5(void);
arch/x86/kernel/alternative.c +1 -1
index 083bd01..cb0e6d3 100644
@@ -641,7 +641,7 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)
 	return addr;
 }
 
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
 
 unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
 
include/asm-generic/vmlinux.lds.h +10 -0
index 8a92a17..ef2af99 100644
@@ -220,6 +220,8 @@
 									\
 	BUG_TABLE							\
 									\
+	JUMP_TABLE							\
+									\
 	/* PCI quirks */						\
 	.pci_fixup        : AT(ADDR(.pci_fixup) - LOAD_OFFSET) {	\
 		VMLINUX_SYMBOL(__start_pci_fixups_early) = .;		\
@@ -563,6 +565,14 @@
 #define BUG_TABLE
 #endif
 
+#define JUMP_TABLE							\
+	. = ALIGN(8);							\
+	__jump_table : AT(ADDR(__jump_table) - LOAD_OFFSET) {		\
+		VMLINUX_SYMBOL(__start___jump_table) = .;		\
+		*(__jump_table)						\
+		VMLINUX_SYMBOL(__stop___jump_table) = .;		\
+	}
+
 #ifdef CONFIG_PM_TRACE
 #define TRACEDATA							\
 	. = ALIGN(4);							\
include/linux/jump_label.h new file +58 -0
new file mode 100644
index 0000000..de58656
@@ -0,0 +1,58 @@
+#ifndef _LINUX_JUMP_LABEL_H
+#define _LINUX_JUMP_LABEL_H
+
+#if defined(CC_HAVE_ASM_GOTO) && defined(CONFIG_HAVE_ARCH_JUMP_LABEL)
+# include <asm/jump_label.h>
+# define HAVE_JUMP_LABEL
+#endif
+
+enum jump_label_type {
+	JUMP_LABEL_ENABLE,
+	JUMP_LABEL_DISABLE
+};
+
+struct module;
+
+#ifdef HAVE_JUMP_LABEL
+
+extern struct jump_entry __start___jump_table[];
+extern struct jump_entry __stop___jump_table[];
+
+extern void arch_jump_label_transform(struct jump_entry *entry,
+				 enum jump_label_type type);
+extern void jump_label_update(unsigned long key, enum jump_label_type type);
+extern void jump_label_apply_nops(struct module *mod);
+extern void arch_jump_label_text_poke_early(jump_label_t addr);
+
+#define enable_jump_label(key) \
+	jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE);
+
+#define disable_jump_label(key) \
+	jump_label_update((unsigned long)key, JUMP_LABEL_DISABLE);
+
+#else
+
+#define JUMP_LABEL(key, label)			\
+do {						\
+	if (unlikely(*key))			\
+		goto label;			\
+} while (0)
+
+#define enable_jump_label(cond_var)	\
+do {					\
+       *(cond_var) = 1;			\
+} while (0)
+
+#define disable_jump_label(cond_var)	\
+do {					\
+       *(cond_var) = 0;			\
+} while (0)
+
+static inline int jump_label_apply_nops(struct module *mod)
+{
+	return 0;
+}
+
+#endif
+
+#endif
include/linux/module.h +4 -1
index 8a6b9fd..403ac26 100644
@@ -350,7 +350,10 @@ struct module
 	struct tracepoint *tracepoints;
 	unsigned int num_tracepoints;
 #endif
-
+#ifdef HAVE_JUMP_LABEL
+	struct jump_entry *jump_entries;
+	unsigned int num_jump_entries;
+#endif
 #ifdef CONFIG_TRACING
 	const char **trace_bprintk_fmt_start;
 	unsigned int num_trace_bprintk_fmt;
kernel/Makefile +1 -1
index 92b7420..8a0feb7 100644
@@ -10,7 +10,7 @@ obj-y     = sched.o fork.o exec_domain.o panic.o printk.o \
 	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \
 	    hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
 	    notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \
-	    async.o range.o
+	    async.o range.o jump_label.o
 obj-y += groups.o
 
 ifdef CONFIG_FUNCTION_TRACER
kernel/jump_label.c new file +0 -0
new file mode 100644
index 0000000..07656ec

kernel/jump_label.c line 4

Quoted 4 lines
kernel/jump_label.c new file +4 -0
@@ -0,0 +1,338 @@
+/*
+ * jump label support
+ *
+ * Copyright (C) 2009 Jason Baron <[email protected]>

Wrong year.

Quoted 114 lines
kernel/jump_label.c new file +114 -0
+ *
+ */
+#include <linux/jump_label.h>
+#include <linux/memory.h>
+#include <linux/uaccess.h>
+#include <linux/module.h>
+#include <linux/list.h>
+#include <linux/jhash.h>
+#include <linux/slab.h>
+#include <linux/sort.h>
+#include <linux/err.h>
+
+#ifdef HAVE_JUMP_LABEL
+
+#define JUMP_LABEL_HASH_BITS 6
+#define JUMP_LABEL_TABLE_SIZE (1 << JUMP_LABEL_HASH_BITS)
+static struct hlist_head jump_label_table[JUMP_LABEL_TABLE_SIZE];
+
+/* mutex to protect coming/going of the the jump_label table */
+static DEFINE_MUTEX(jump_label_mutex);
+
+struct jump_label_entry {
+	struct hlist_node hlist;
+	struct jump_entry *table;
+	int nr_entries;
+	/* hang modules off here */
+	struct hlist_head modules;
+	unsigned long key;
+};
+
+struct jump_label_module_entry {
+	struct hlist_node hlist;
+	struct jump_entry *table;
+	int nr_entries;
+	struct module *mod;
+};
+
+static int jump_label_cmp(const void *a, const void *b)
+{
+	const struct jump_entry *jea = a;
+	const struct jump_entry *jeb = b;
+
+	if (jea->key < jeb->key)
+		return -1;
+
+	if (jea->key > jeb->key)
+		return 1;
+
+	return 0;
+}
+
+static void sort_jump_label_entries(struct jump_entry *start, struct jump_entry *stop)
+{
+	unsigned long size;
+
+	size = (((unsigned long)stop - (unsigned long)start)
+					/ sizeof(struct jump_entry));
+	sort(start, size, sizeof(struct jump_entry), jump_label_cmp, NULL);
+}
+
+static struct jump_label_entry *get_jump_label_entry(jump_label_t key)
+{
+	struct hlist_head *head;
+	struct hlist_node *node;
+	struct jump_label_entry *e;
+	u32 hash = jhash((void *)&key, sizeof(jump_label_t), 0);
+
+	head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
+	hlist_for_each_entry(e, node, head, hlist) {
+		if (key == e->key)
+			return e;
+	}
+	return NULL;
+}
+
+static struct jump_label_entry *add_jump_label_entry(jump_label_t key, int nr_entries, struct jump_entry *table)
+{
+	struct hlist_head *head;
+	struct jump_label_entry *e;
+	u32 hash;
+
+	e = get_jump_label_entry(key);
+	if (e)
+		return ERR_PTR(-EEXIST);
+
+	e = kmalloc(sizeof(struct jump_label_entry), GFP_KERNEL);
+	if (!e)
+		return ERR_PTR(-ENOMEM);
+
+	hash = jhash((void *)&key, sizeof(jump_label_t), 0);
+	head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
+	e->key = key;
+	e->table = table;
+	e->nr_entries = nr_entries;
+	INIT_HLIST_HEAD(&(e->modules));
+	hlist_add_head(&e->hlist, head);
+	return e;
+}
+
+static int build_jump_label_hashtable(struct jump_entry *start, struct jump_entry *stop)
+{
+	struct jump_entry *iter, *iter_begin;
+	struct jump_label_entry *entry;
+	int count;
+
+	sort_jump_label_entries(start, stop);
+	iter = start;
+	while (iter < stop) {
+		entry = get_jump_label_entry(iter->key);
+		if (!entry) {
+			iter_begin = iter;
+			count = 0;
+			while ((iter < stop) &&
+				(iter->key == iter_begin->key)) {

kernel/jump_label.c line 128

Quoted 10 lines
kernel/jump_label.c new file +10 -0
+				iter++;
+				count++;
+			}
+			entry = add_jump_label_entry(iter_begin->key,
+							count, iter_begin);
+			if (IS_ERR(entry))
+				return PTR_ERR(entry);
+		 } else {
+			WARN_ONCE(1, KERN_ERR "build_jump_hashtable: unexpected entry!\n");
+			return -1;

Would it make sense to use a different value? Say -EINVAL?

Quoted 24 lines
kernel/jump_label.c new file +24 -0
+		}
+	}
+	return 0;
+}
+
+/***
+ * jump_label_update - update jump label text
+ * @key -  key value associated with a a jump label
+ * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
+ *
+ * Will enable/disable the jump for jump label @key, depending on the
+ * value of @type.
+ *
+ */
+
+void jump_label_update(unsigned long key, enum jump_label_type type)
+{
+	struct jump_entry *iter;
+	struct jump_label_entry *entry;
+	struct hlist_node *module_node;
+	struct jump_label_module_entry *e_module;
+	int count;
+
+	mutex_lock(&jump_label_mutex);

kernel/jump_label.c line 162

Quoted 10 lines
kernel/jump_label.c new file +10 -0
+	entry = get_jump_label_entry((jump_label_t)key);
+	if (entry) {
+		count = entry->nr_entries;
+		iter = entry->table;
+		while (count--) {
+			if (kernel_text_address(iter->code))
+				arch_jump_label_transform(iter, type);
+			iter++;
+		}
+		/* eanble/disable jump labels in modules */

"enable"

Quoted 23 lines
kernel/jump_label.c new file +23 -0
+		hlist_for_each_entry(e_module, module_node, &(entry->modules),
+							hlist) {
+			count = e_module->nr_entries;
+			iter = e_module->table;
+			while (count--) {
+				if (kernel_text_address(iter->code))
+					arch_jump_label_transform(iter, type);
+				iter++;
+			}
+		}
+	}
+	mutex_unlock(&jump_label_mutex);
+}
+
+static __init int init_jump_label(void)
+{
+	int ret;
+	struct jump_entry *iter_start = __start___jump_table;
+	struct jump_entry *iter_stop = __stop___jump_table;
+	struct jump_entry *iter;
+
+	mutex_lock(&jump_label_mutex);
+	ret = build_jump_label_hashtable(__start___jump_table,

kernel/jump_label.c line 195

Quoted 10 lines
kernel/jump_label.c new file +10 -0
+					 __stop___jump_table);
+	iter = iter_start;
+	while (iter < iter_stop) {
+		arch_jump_label_text_poke_early(iter->code);
+		iter++;
+	}
+	mutex_unlock(&jump_label_mutex);
+	return ret;
+}
+early_initcall(init_jump_label);

Is there any danger of not patching the instructions with NOPs before early_initcall is called? I guess for your uses (tracing) there won't. And even if there are not patched it will just call the function.

Quoted 198 lines
kernel/jump_label.c new file +143 -0
+
+#ifdef CONFIG_MODULES
+
+static struct jump_label_module_entry *add_jump_label_module_entry(struct jump_label_entry *entry, struct jump_entry *iter_begin, int count, struct module *mod)
+{
+	struct jump_label_module_entry *e;
+
+	e = kmalloc(sizeof(struct jump_label_module_entry), GFP_KERNEL);
+	if (!e)
+		return ERR_PTR(-ENOMEM);
+	e->mod = mod;
+	e->nr_entries = count;
+	e->table = iter_begin;
+	hlist_add_head(&e->hlist, &entry->modules);
+	return e;
+}
+
+static int add_jump_label_module(struct module *mod)
+{
+	struct jump_entry *iter, *iter_begin;
+	struct jump_label_entry *entry;
+	struct jump_label_module_entry *module_entry;
+	int count;
+
+	/* if the module doesn't have jump label entries, just return */
+	if (!mod->num_jump_entries)
+		return 0;
+
+	sort_jump_label_entries(mod->jump_entries,
+				mod->jump_entries + mod->num_jump_entries);
+	iter = mod->jump_entries;
+	while (iter < mod->jump_entries + mod->num_jump_entries) {
+		entry = get_jump_label_entry(iter->key);
+		iter_begin = iter;
+		count = 0;
+		while ((iter < mod->jump_entries + mod->num_jump_entries) &&
+			(iter->key == iter_begin->key)) {
+				iter++;
+				count++;
+		}
+		if (!entry) {
+			entry = add_jump_label_entry(iter_begin->key, 0, NULL);
+			if (IS_ERR(entry))
+				return PTR_ERR(entry);
+		}
+		module_entry = add_jump_label_module_entry(entry, iter_begin,
+							   count, mod);
+		if (IS_ERR(module_entry))
+			return PTR_ERR(module_entry);
+	}
+	return 0;
+}
+
+static void remove_jump_label_module(struct module *mod)
+{
+	struct hlist_head *head;
+	struct hlist_node *node, *node_next, *module_node, *module_node_next;
+	struct jump_label_entry *e;
+	struct jump_label_module_entry *e_module;
+	int i;
+
+	/* if the module doesn't have jump label entries, just return */
+	if (!mod->num_jump_entries)
+		return;
+
+	for (i = 0; i < JUMP_LABEL_TABLE_SIZE; i++) {
+		head = &jump_label_table[i];
+		hlist_for_each_entry_safe(e, node, node_next, head, hlist) {
+			hlist_for_each_entry_safe(e_module, module_node,
+						  module_node_next,
+						  &(e->modules), hlist) {
+				if (e_module->mod == mod) {
+					hlist_del(&e_module->hlist);
+					kfree(e_module);
+				}
+			}
+			if (hlist_empty(&e->modules) && (e->nr_entries == 0)) {
+				hlist_del(&e->hlist);
+				kfree(e);
+			}
+		}
+	}
+}
+
+static int jump_label_module_notify(struct notifier_block *self, unsigned long val, void *data)
+{
+	struct module *mod = data;
+	int ret = 0;
+
+	switch (val) {
+	case MODULE_STATE_COMING:
+		mutex_lock(&jump_label_mutex);
+		ret = add_jump_label_module(mod);
+		if (ret)
+			remove_jump_label_module(mod);
+		mutex_unlock(&jump_label_mutex);
+		break;
+	case MODULE_STATE_GOING:
+		mutex_lock(&jump_label_mutex);
+		remove_jump_label_module(mod);
+		mutex_unlock(&jump_label_mutex);
+		break;
+	}
+	return ret;
+}
+
+/***
+ * apply_jump_label_nops - patch module jump labels with arch_get_jump_label_nop()
+ * @mod: module to patch
+ *
+ * Allow for run-time selection of the optimal nops. Before the module
+ * loads patch these with arch_get_jump_label_nop(), which is specified by
+ * the arch specific jump label code.
+ */
+void jump_label_apply_nops(struct module *mod)
+{
+	struct jump_entry *iter;
+
+	/* if the module doesn't have jump label entries, just return */
+	if (!mod->num_jump_entries)
+		return;
+
+	iter = mod->jump_entries;
+	while (iter < mod->jump_entries + mod->num_jump_entries) {
+		arch_jump_label_text_poke_early(iter->code);
+		iter++;
+	}
+}
+
+struct notifier_block jump_label_module_nb = {
+	.notifier_call = jump_label_module_notify,
+	.priority = 0,
+};
+
+static __init int init_jump_label_module(void)
+{
+	return register_module_notifier(&jump_label_module_nb);
+}
+early_initcall(init_jump_label_module);
+
+#endif /* CONFIG_MODULES */
+
+#endif
kernel/kprobes.c +1 -0
index 282035f..798adfa 100644
@@ -47,6 +47,7 @@
 #include <linux/memory.h>
 #include <linux/ftrace.h>
 #include <linux/cpu.h>
+#include <linux/jump_label.h>
 
 #include <asm-generic/sections.h>
 #include <asm/cacheflush.h>
kernel/module.c +6 -0
index d0b5f8d..eba1341 100644
@@ -55,6 +55,7 @@
 #include <linux/async.h>
 #include <linux/percpu.h>
 #include <linux/kmemleak.h>
+#include <linux/jump_label.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/module.h>
@@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)
 					sizeof(*mod->tracepoints),
 					&mod->num_tracepoints);
 #endif
+#ifdef HAVE_JUMP_LABEL
+	mod->jump_entries = section_objs(info, "__jump_table",
+					sizeof(*mod->jump_entries),
+					&mod->num_jump_entries);
+#endif
 #ifdef CONFIG_EVENT_TRACING
 	mod->trace_events = section_objs(info, "_ftrace_events",
 					 sizeof(*mod->trace_events),
scripts/gcc-goto.sh new file +5 -0
new file mode 100644
index 0000000..8e82424
@@ -0,0 +1,5 @@
+#!/bin/sh
+# Test for gcc 'asm goto' suport
+# Copyright (C) 2010, Jason Baron <[email protected]>
+
+echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
Signature · 7 lines
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
#24 lore raw

Re: [PATCH 03/10] jump label v11: base patch

Quoted 5 lines
Quoted 1 line

+early_initcall(init_jump_label);

Is there any danger of not patching the instructions with NOPs before early_initcall is called? I guess for your uses (tracing) there won't. And even if there are not patched it will just call the function.

Ah, nevermind - if you don't patch the addresses which are in _jump_label section, the untouched code will just do far jump (jmpq) to right after the macro, which is ok.

#25 lore raw

Re: [PATCH 03/10] jump label v11: base patch

Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 5 lines
Quoted 1 line

+early_initcall(init_jump_label);

Is there any danger of not patching the instructions with NOPs before early_initcall is called? I guess for your uses (tracing) there won't. And even if there are not patched it will just call the function.

Ah, nevermind - if you don't patch the addresses which are in _jump_label section, the untouched code will just do far jump (jmpq) to right after the macro, which is ok.

Technical point of order: jmpq is *NOT* a far jump. A far jump (called ljmp in gas) is a completely different x86 instruction with *very* different semantics.

-hpa

#26 lore raw

[tip:perf/core] jump label: Base patch for jump label

In reply to Jason Baron
Commit-ID:  bf5438fca2950b03c21ad868090cc1a8fcd49536
Gitweb:     http://git.kernel.org/tip/bf5438fca2950b03c21ad868090cc1a8fcd49536
Author:     Jason Baron <[email protected]>
AuthorDate: Fri, 17 Sep 2010 11:09:00 -0400
Committer:  Steven Rostedt <[email protected]>
CommitDate: Wed, 22 Sep 2010 16:29:41 -0400

jump label: Base patch for jump label

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

LKML-Reference: <ee8b3595967989fdaf84e698dc7447d315ce972a.1284733808.git.jbaron@redhat.com>

[ cleaned up some formating ]

---
 Makefile                           |    5 +
 arch/Kconfig                       |    3 +
 arch/x86/include/asm/alternative.h |    3 +-
 arch/x86/kernel/alternative.c      |    2 +-
 include/asm-generic/vmlinux.lds.h  |   10 +
 include/linux/jump_label.h         |   58 ++++++
 include/linux/module.h             |    5 +-
 kernel/Makefile                    |    2 +-
 kernel/jump_label.c                |  346 ++++++++++++++++++++++++++++++++++++
 kernel/kprobes.c                   |    1 +
 kernel/module.c                    |    6 +
 scripts/gcc-goto.sh                |    5 +
 12 files changed, 442 insertions(+), 4 deletions(-)
Makefile +5 -0
index 92ab33f..a906378 100644
@@ -591,6 +591,11 @@ KBUILD_CFLAGS	+= $(call cc-option,-fno-strict-overflow)
 # conserve stack if available
 KBUILD_CFLAGS   += $(call cc-option,-fconserve-stack)
 
+# check for 'asm goto'
+ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC)), y)
+	KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
+endif
+
 # Add user supplied CPPFLAGS, AFLAGS and CFLAGS as the last assignments
 # But warn user when we do so
 warn-assign = \
arch/Kconfig +3 -0
index 4877a8c..1462d84 100644
@@ -158,4 +158,7 @@ config HAVE_PERF_EVENTS_NMI
 	  subsystem.  Also has support for calculating CPU cycle events
 	  to determine how many clock cycles in a given period.
 
+config HAVE_ARCH_JUMP_LABEL
+	bool
+
 source "kernel/gcov/Kconfig"
arch/x86/include/asm/alternative.h +2 -1
index 634bf78..76561d2 100644
@@ -4,6 +4,7 @@
 #include <linux/types.h>
 #include <linux/stddef.h>
 #include <linux/stringify.h>
+#include <linux/jump_label.h>
 #include <asm/asm.h>
 
 /*
@@ -182,7 +183,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);
 extern void *text_poke(void *addr, const void *opcode, size_t len);
 extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
 
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
 #define IDEAL_NOP_SIZE_5 5
 extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
 extern void arch_init_ideal_nop5(void);
arch/x86/kernel/alternative.c +1 -1
index 083bd01..cb0e6d3 100644
@@ -641,7 +641,7 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)
 	return addr;
 }
 
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
 
 unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
 
include/asm-generic/vmlinux.lds.h +10 -0
index 8a92a17..ef2af99 100644
@@ -220,6 +220,8 @@
 									\
 	BUG_TABLE							\
 									\
+	JUMP_TABLE							\
+									\
 	/* PCI quirks */						\
 	.pci_fixup        : AT(ADDR(.pci_fixup) - LOAD_OFFSET) {	\
 		VMLINUX_SYMBOL(__start_pci_fixups_early) = .;		\
@@ -563,6 +565,14 @@
 #define BUG_TABLE
 #endif
 
+#define JUMP_TABLE							\
+	. = ALIGN(8);							\
+	__jump_table : AT(ADDR(__jump_table) - LOAD_OFFSET) {		\
+		VMLINUX_SYMBOL(__start___jump_table) = .;		\
+		*(__jump_table)						\
+		VMLINUX_SYMBOL(__stop___jump_table) = .;		\
+	}
+
 #ifdef CONFIG_PM_TRACE
 #define TRACEDATA							\
 	. = ALIGN(4);							\
include/linux/jump_label.h new file +58 -0

58 lines not on this page · raw message

include/linux/module.h +4 -1
index 8a6b9fd..403ac26 100644
@@ -350,7 +350,10 @@ struct module
 	struct tracepoint *tracepoints;
 	unsigned int num_tracepoints;
 #endif
-
+#ifdef HAVE_JUMP_LABEL
+	struct jump_entry *jump_entries;
+	unsigned int num_jump_entries;
+#endif
 #ifdef CONFIG_TRACING
 	const char **trace_bprintk_fmt_start;
 	unsigned int num_trace_bprintk_fmt;
kernel/Makefile +1 -1
index 0b72d1a..d52b473 100644
@@ -10,7 +10,7 @@ obj-y     = sched.o fork.o exec_domain.o panic.o printk.o \
 	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \
 	    hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
 	    notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \
-	    async.o range.o
+	    async.o range.o jump_label.o
 obj-$(CONFIG_HAVE_EARLY_RES) += early_res.o
 obj-y += groups.o
 
kernel/jump_label.c new file +346 -0

346 lines not on this page · raw message

kernel/kprobes.c +1 -0
index 6dd5359..18904e4 100644
@@ -47,6 +47,7 @@
 #include <linux/memory.h>
 #include <linux/ftrace.h>
 #include <linux/cpu.h>
+#include <linux/jump_label.h>
 
 #include <asm-generic/sections.h>
 #include <asm/cacheflush.h>
kernel/module.c +6 -0
index d0b5f8d..eba1341 100644
@@ -55,6 +55,7 @@
 #include <linux/async.h>
 #include <linux/percpu.h>
 #include <linux/kmemleak.h>
+#include <linux/jump_label.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/module.h>
@@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)
 					sizeof(*mod->tracepoints),
 					&mod->num_tracepoints);
 #endif
+#ifdef HAVE_JUMP_LABEL
+	mod->jump_entries = section_objs(info, "__jump_table",
+					sizeof(*mod->jump_entries),
+					&mod->num_jump_entries);
+#endif
 #ifdef CONFIG_EVENT_TRACING
 	mod->trace_events = section_objs(info, "_ftrace_events",
 					 sizeof(*mod->trace_events),
scripts/gcc-goto.sh new file +5 -0
new file mode 100644
index 0000000..8e82424
@@ -0,0 +1,5 @@
+#!/bin/sh
+# Test for gcc 'asm goto' suport
+# Copyright (C) 2010, Jason Baron <[email protected]>
+
+echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
#27 lore raw

PATCH 4/10 jump label v11: initialize workqueue tracepoints *before* they are registered

In reply to Jason Baron

Initialize the workqueue data structures *before* they are registered so that they are ready for callbacks.

---
 kernel/trace/trace_workqueue.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)
kernel/trace/trace_workqueue.c +5 -5
index a7cc379..209b379 100644
@@ -263,6 +263,11 @@ int __init trace_workqueue_early_init(void)
 {
 	int ret, cpu;
 
+	for_each_possible_cpu(cpu) {
+		spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
+		INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
+	}
+
 	ret = register_trace_workqueue_insertion(probe_workqueue_insertion, NULL);
 	if (ret)
 		goto out;
@@ -279,11 +284,6 @@ int __init trace_workqueue_early_init(void)
 	if (ret)
 		goto no_creation;
 
-	for_each_possible_cpu(cpu) {
-		spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
-		INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
-	}
-
 	return 0;
 
 no_creation:
Signature · 1 line
1.7.1
#28 lore raw

[tip:perf/core] jump label: Initialize workqueue tracepoints *before* they are registered

Commit-ID:  e0cf0cd49632552f063fb3ae58691946da45fb2e
Gitweb:     http://git.kernel.org/tip/e0cf0cd49632552f063fb3ae58691946da45fb2e
Author:     Jason Baron <[email protected]>
AuthorDate: Fri, 17 Sep 2010 11:09:04 -0400
Committer:  Steven Rostedt <[email protected]>
CommitDate: Wed, 22 Sep 2010 16:30:03 -0400

jump label: Initialize workqueue tracepoints *before* they are registered

Initialize the workqueue data structures *before* they are registered so that they are ready for callbacks.

LKML-Reference: <e3a3383fc370ac7086625bebe89d9480d7caf372.1284733808.git.jbaron@redhat.com>

---
 kernel/trace/trace_workqueue.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)
kernel/trace/trace_workqueue.c +5 -5
index a7cc379..209b379 100644
@@ -263,6 +263,11 @@ int __init trace_workqueue_early_init(void)
 {
 	int ret, cpu;
 
+	for_each_possible_cpu(cpu) {
+		spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
+		INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
+	}
+
 	ret = register_trace_workqueue_insertion(probe_workqueue_insertion, NULL);
 	if (ret)
 		goto out;
@@ -279,11 +284,6 @@ int __init trace_workqueue_early_init(void)
 	if (ret)
 		goto no_creation;
 
-	for_each_possible_cpu(cpu) {
-		spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
-		INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
-	}
-
 	return 0;
 
 no_creation:
#29 lore raw

PATCH 5/10 jump label v11: jump_label_text_reserved() to reserve our jump points

In reply to Jason Baron

Add a jump_label_text_reserved(void *start, void *end), so that other pieces of code that want to modify kernel text, can first verify that jump label has not reserved the instruction.

---
 arch/x86/kernel/kprobes.c  |    3 +-
 include/linux/jump_label.h |    8 ++++-
 kernel/jump_label.c        |   83 ++++++++++++++++++++++++++++++++++++++++++++
 kernel/kprobes.c           |    3 +-
 4 files changed, 94 insertions(+), 3 deletions(-)
arch/x86/kernel/kprobes.c +2 -1
index 770ebfb..788fade 100644
@@ -1221,7 +1221,8 @@ static int __kprobes copy_optimized_instructions(u8 *dest, u8 *src)
 	}
 	/* Check whether the address range is reserved */
 	if (ftrace_text_reserved(src, src + len - 1) ||
-	    alternatives_text_reserved(src, src + len - 1))
+	    alternatives_text_reserved(src, src + len - 1) ||
+	    jump_label_text_reserved(src, src + len - 1))
 		return -EBUSY;
 
 	return len;
include/linux/jump_label.h +7 -1
index de58656..b72cd9f 100644
@@ -20,9 +20,10 @@ extern struct jump_entry __stop___jump_table[];
 
 extern void arch_jump_label_transform(struct jump_entry *entry,
 				 enum jump_label_type type);
+extern void arch_jump_label_text_poke_early(jump_label_t addr);
 extern void jump_label_update(unsigned long key, enum jump_label_type type);
 extern void jump_label_apply_nops(struct module *mod);
-extern void arch_jump_label_text_poke_early(jump_label_t addr);
+extern int jump_label_text_reserved(void *start, void *end);
 
 #define enable_jump_label(key) \
 	jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE);
@@ -53,6 +54,11 @@ static inline int jump_label_apply_nops(struct module *mod)
 	return 0;
 }
 
+static inline int jump_label_text_reserved(void *start, void *end)
+{
+	return 0;
+}
+
 #endif
 
 #endif
kernel/jump_label.c +83 -0

89 lines not on this page · raw message

kernel/kprobes.c +2 -1
index 798adfa..a53bff3 100644
@@ -1142,7 +1142,8 @@ int __kprobes register_kprobe(struct kprobe *p)
 	preempt_disable();
 	if (!kernel_text_address((unsigned long) p->addr) ||
 	    in_kprobes_functions((unsigned long) p->addr) ||
-	    ftrace_text_reserved(p->addr, p->addr)) {
+	    ftrace_text_reserved(p->addr, p->addr) ||
+	    jump_label_text_reserved(p->addr, p->addr)) {
 		preempt_enable();
 		return -EINVAL;
 	}
Signature · 1 line
1.7.1
#30 lore raw

[tip:perf/core] jump label: Add jump_label_text_reserved() to reserve jump points

Commit-ID:  4c3ef6d79328c0e23ade60cbfc8d496123a6855c
Gitweb:     http://git.kernel.org/tip/4c3ef6d79328c0e23ade60cbfc8d496123a6855c
Author:     Jason Baron <[email protected]>
AuthorDate: Fri, 17 Sep 2010 11:09:08 -0400
Committer:  Steven Rostedt <[email protected]>
CommitDate: Wed, 22 Sep 2010 16:30:46 -0400

jump label: Add jump_label_text_reserved() to reserve jump points

Add a jump_label_text_reserved(void *start, void *end), so that other pieces of code that want to modify kernel text, can first verify that jump label has not reserved the instruction.

LKML-Reference: <06236663a3a7b1c1f13576bb9eccb6d9c17b7bfe.1284733808.git.jbaron@redhat.com>

---
 arch/x86/kernel/kprobes.c  |    3 +-
 include/linux/jump_label.h |    8 ++++-
 kernel/jump_label.c        |   83 ++++++++++++++++++++++++++++++++++++++++++++
 kernel/kprobes.c           |    3 +-
 4 files changed, 94 insertions(+), 3 deletions(-)
arch/x86/kernel/kprobes.c +2 -1
index e05952a..1cbd54c 100644
@@ -1218,7 +1218,8 @@ static int __kprobes copy_optimized_instructions(u8 *dest, u8 *src)
 	}
 	/* Check whether the address range is reserved */
 	if (ftrace_text_reserved(src, src + len - 1) ||
-	    alternatives_text_reserved(src, src + len - 1))
+	    alternatives_text_reserved(src, src + len - 1) ||
+	    jump_label_text_reserved(src, src + len - 1))
 		return -EBUSY;
 
 	return len;
include/linux/jump_label.h +7 -1
index de58656..b72cd9f 100644
@@ -20,9 +20,10 @@ extern struct jump_entry __stop___jump_table[];
 
 extern void arch_jump_label_transform(struct jump_entry *entry,
 				 enum jump_label_type type);
+extern void arch_jump_label_text_poke_early(jump_label_t addr);
 extern void jump_label_update(unsigned long key, enum jump_label_type type);
 extern void jump_label_apply_nops(struct module *mod);
-extern void arch_jump_label_text_poke_early(jump_label_t addr);
+extern int jump_label_text_reserved(void *start, void *end);
 
 #define enable_jump_label(key) \
 	jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE);
@@ -53,6 +54,11 @@ static inline int jump_label_apply_nops(struct module *mod)
 	return 0;
 }
 
+static inline int jump_label_text_reserved(void *start, void *end)
+{
+	return 0;
+}
+
 #endif
 
 #endif
kernel/jump_label.c +83 -0

89 lines not on this page · raw message

kernel/kprobes.c +2 -1
index 18904e4..ec4210c 100644
@@ -1147,7 +1147,8 @@ int __kprobes register_kprobe(struct kprobe *p)
 	preempt_disable();
 	if (!kernel_text_address((unsigned long) p->addr) ||
 	    in_kprobes_functions((unsigned long) p->addr) ||
-	    ftrace_text_reserved(p->addr, p->addr)) {
+	    ftrace_text_reserved(p->addr, p->addr) ||
+	    jump_label_text_reserved(p->addr, p->addr)) {
 		preempt_enable();
 		return -EINVAL;
 	}
#31 lore raw

PATCH 6/10 jump label v11: tracepoint support

In reply to Jason Baron

Make use of the jump label infrastructure for tracepoints.

---
 include/linux/tracepoint.h |    5 ++++-
 kernel/tracepoint.c        |   14 ++++++++++++--
 2 files changed, 16 insertions(+), 3 deletions(-)
include/linux/tracepoint.h +4 -1
index 103d1b6..a4a90b6 100644
@@ -17,6 +17,7 @@
 #include <linux/errno.h>
 #include <linux/types.h>
 #include <linux/rcupdate.h>
+#include <linux/jump_label.h>
 
 struct module;
 struct tracepoint;
@@ -145,7 +146,9 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin,
 	extern struct tracepoint __tracepoint_##name;			\
 	static inline void trace_##name(proto)				\
 	{								\
-		if (unlikely(__tracepoint_##name.state))		\
+		JUMP_LABEL(&__tracepoint_##name.state, do_trace);	\
+		return;							\
+do_trace:								\
 			__DO_TRACE(&__tracepoint_##name,		\
 				TP_PROTO(data_proto),			\
 				TP_ARGS(data_args));			\
kernel/tracepoint.c +12 -2
index c77f3ec..d6073a5 100644
@@ -25,6 +25,7 @@
 #include <linux/err.h>
 #include <linux/slab.h>
 #include <linux/sched.h>
+#include <linux/jump_label.h>
 
 extern struct tracepoint __start___tracepoints[];
 extern struct tracepoint __stop___tracepoints[];
@@ -263,7 +264,13 @@ static void set_tracepoint(struct tracepoint_entry **entry,
 	 * is used.
 	 */
 	rcu_assign_pointer(elem->funcs, (*entry)->funcs);
-	elem->state = active;
+	if (!elem->state && active) {
+		enable_jump_label(&elem->state);
+		elem->state = active;
+	} else if (elem->state && !active) {
+		disable_jump_label(&elem->state);
+		elem->state = active;
+	}
 }
 
 /*
@@ -277,7 +284,10 @@ static void disable_tracepoint(struct tracepoint *elem)
 	if (elem->unregfunc && elem->state)
 		elem->unregfunc();
 
-	elem->state = 0;
+	if (elem->state) {
+		disable_jump_label(&elem->state);
+		elem->state = 0;
+	}
 	rcu_assign_pointer(elem->funcs, NULL);
 }
 
Signature · 1 line
1.7.1
#32 lore raw

[tip:perf/core] jump label: Tracepoint support for jump labels

Commit-ID:  8f7b50c514206211cc282a4247f7b12f18dee674
Gitweb:     http://git.kernel.org/tip/8f7b50c514206211cc282a4247f7b12f18dee674
Author:     Jason Baron <[email protected]>
AuthorDate: Fri, 17 Sep 2010 11:09:13 -0400
Committer:  Steven Rostedt <[email protected]>
CommitDate: Wed, 22 Sep 2010 16:31:01 -0400

jump label: Tracepoint support for jump labels

Make use of the jump label infrastructure for tracepoints.

LKML-Reference: <a9ba2056e2c9cf332c3c300b577463ce66ff23a8.1284733808.git.jbaron@redhat.com>

---
 include/linux/tracepoint.h |    5 ++++-
 kernel/tracepoint.c        |   14 ++++++++++++--
 2 files changed, 16 insertions(+), 3 deletions(-)
include/linux/tracepoint.h +4 -1
index 103d1b6..a4a90b6 100644
@@ -17,6 +17,7 @@
 #include <linux/errno.h>
 #include <linux/types.h>
 #include <linux/rcupdate.h>
+#include <linux/jump_label.h>
 
 struct module;
 struct tracepoint;
@@ -145,7 +146,9 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin,
 	extern struct tracepoint __tracepoint_##name;			\
 	static inline void trace_##name(proto)				\
 	{								\
-		if (unlikely(__tracepoint_##name.state))		\
+		JUMP_LABEL(&__tracepoint_##name.state, do_trace);	\
+		return;							\
+do_trace:								\
 			__DO_TRACE(&__tracepoint_##name,		\
 				TP_PROTO(data_proto),			\
 				TP_ARGS(data_args));			\
kernel/tracepoint.c +12 -2
index c77f3ec..d6073a5 100644
@@ -25,6 +25,7 @@
 #include <linux/err.h>
 #include <linux/slab.h>
 #include <linux/sched.h>
+#include <linux/jump_label.h>
 
 extern struct tracepoint __start___tracepoints[];
 extern struct tracepoint __stop___tracepoints[];
@@ -263,7 +264,13 @@ static void set_tracepoint(struct tracepoint_entry **entry,
 	 * is used.
 	 */
 	rcu_assign_pointer(elem->funcs, (*entry)->funcs);
-	elem->state = active;
+	if (!elem->state && active) {
+		enable_jump_label(&elem->state);
+		elem->state = active;
+	} else if (elem->state && !active) {
+		disable_jump_label(&elem->state);
+		elem->state = active;
+	}
 }
 
 /*
@@ -277,7 +284,10 @@ static void disable_tracepoint(struct tracepoint *elem)
 	if (elem->unregfunc && elem->state)
 		elem->unregfunc();
 
-	elem->state = 0;
+	if (elem->state) {
+		disable_jump_label(&elem->state);
+		elem->state = 0;
+	}
 	rcu_assign_pointer(elem->funcs, NULL);
 }
 
#33 lore raw

PATCH 7/10 jump label v11: convert dynamic debug to use jump labels

In reply to Jason Baron

Convert the 'dynamic debug' infrastructure to use jump labels.

---
 include/linux/dynamic_debug.h |   39 +++++++++++++-----------
 lib/dynamic_debug.c           |   42 ++-------------------------
 scripts/Makefile.lib          |   11 +------
 scripts/basic/Makefile        |    2 +-
 scripts/basic/hash.c          |   64 -----------------------------------------
 5 files changed, 26 insertions(+), 132 deletions(-)
 delete mode 100644 scripts/basic/hash.c
include/linux/dynamic_debug.h +21 -18
index 52c0da4..bef3cda 100644
@@ -1,6 +1,8 @@
 #ifndef _DYNAMIC_DEBUG_H
 #define _DYNAMIC_DEBUG_H
 
+#include <linux/jump_label.h>
+
 /* dynamic_printk_enabled, and dynamic_printk_enabled2 are bitmasks in which
  * bit n is set to 1 if any modname hashes into the bucket n, 0 otherwise. They
  * use independent hash functions, to reduce the chance of false positives.
@@ -22,8 +24,6 @@ struct _ddebug {
 	const char *function;
 	const char *filename;
 	const char *format;
-	char primary_hash;
-	char secondary_hash;
 	unsigned int lineno:24;
 	/*
  	 * The flags field controls the behaviour at the callsite.
@@ -33,6 +33,7 @@ struct _ddebug {
 #define _DPRINTK_FLAGS_PRINT   (1<<0)  /* printk() a message using the format */
 #define _DPRINTK_FLAGS_DEFAULT 0
 	unsigned int flags:8;
+	char enabled;
 } __attribute__((aligned(8)));
 
 
@@ -42,33 +43,35 @@ int ddebug_add_module(struct _ddebug *tab, unsigned int n,
 #if defined(CONFIG_DYNAMIC_DEBUG)
 extern int ddebug_remove_module(const char *mod_name);
 
-#define __dynamic_dbg_enabled(dd)  ({	     \
-	int __ret = 0;							     \
-	if (unlikely((dynamic_debug_enabled & (1LL << DEBUG_HASH)) &&	     \
-			(dynamic_debug_enabled2 & (1LL << DEBUG_HASH2))))   \
-				if (unlikely(dd.flags))			     \
-					__ret = 1;			     \
-	__ret; })
-
 #define dynamic_pr_debug(fmt, ...) do {					\
+	__label__ do_printk;						\
+	__label__ out;							\
 	static struct _ddebug descriptor				\
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
-	{ KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH,	\
-		DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT };	\
-	if (__dynamic_dbg_enabled(descriptor))				\
-		printk(KERN_DEBUG pr_fmt(fmt),	##__VA_ARGS__);		\
+	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
+		_DPRINTK_FLAGS_DEFAULT };				\
+	JUMP_LABEL(&descriptor.enabled, do_printk);			\
+	goto out;							\
+do_printk:								\
+	printk(KERN_DEBUG pr_fmt(fmt),	##__VA_ARGS__);			\
+out:	;								\
 	} while (0)
 
 
 #define dynamic_dev_dbg(dev, fmt, ...) do {				\
+	__label__ do_printk;						\
+	__label__ out;							\
 	static struct _ddebug descriptor				\
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
-	{ KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH,	\
-		DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT };	\
-	if (__dynamic_dbg_enabled(descriptor))				\
-		dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__);	\
+	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
+		_DPRINTK_FLAGS_DEFAULT };				\
+	JUMP_LABEL(&descriptor.enabled, do_printk);			\
+	goto out;							\
+do_printk:								\
+	dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__);		\
+out:	;								\
 	} while (0)
 
 #else
lib/dynamic_debug.c +3 -39

65 lines not on this page · raw message

scripts/Makefile.lib +1 -10
index 54fd1b7..7bfcf1a 100644
@@ -101,14 +101,6 @@ basename_flags = -D"KBUILD_BASENAME=KBUILD_STR($(call name-fix,$(basetarget)))"
 modname_flags  = $(if $(filter 1,$(words $(modname))),\
                  -D"KBUILD_MODNAME=KBUILD_STR($(call name-fix,$(modname)))")
 
-#hash values
-ifdef CONFIG_DYNAMIC_DEBUG
-debug_flags = -D"DEBUG_HASH=$(shell ./scripts/basic/hash djb2 $(@D)$(modname))"\
-              -D"DEBUG_HASH2=$(shell ./scripts/basic/hash r5 $(@D)$(modname))"
-else
-debug_flags =
-endif
-
 orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
                  $(ccflags-y) $(CFLAGS_$(basetarget).o)
 _c_flags       = $(filter-out $(CFLAGS_REMOVE_$(basetarget).o), $(orig_c_flags))
@@ -152,8 +144,7 @@ endif
 
 c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__c_flags) $(modkern_cflags)                           \
-		 -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags) \
-		  $(debug_flags)
+		 -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags)
 
 a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__a_flags) $(modkern_aflags)
scripts/basic/Makefile +1 -1
index 0955995..4c324a1 100644
@@ -9,7 +9,7 @@
 # fixdep: 	 Used to generate dependency information during build process
 # docproc:	 Used in Documentation/DocBook
 
-hostprogs-y	:= fixdep docproc hash
+hostprogs-y	:= fixdep docproc
 always		:= $(hostprogs-y)
 
 # fixdep is needed to compile other host programs
scripts/basic/hash.c deleted +0 -64

64 lines not on this page · raw message

Signature · 1 line
1.7.1
#34 lore raw

[tip:perf/core] jump label: Convert dynamic debug to use jump labels

Commit-ID:  52159d98be6f26c48f5e02c7ab3c9848a85979b5
Gitweb:     http://git.kernel.org/tip/52159d98be6f26c48f5e02c7ab3c9848a85979b5
Author:     Jason Baron <[email protected]>
AuthorDate: Fri, 17 Sep 2010 11:09:17 -0400
Committer:  Steven Rostedt <[email protected]>
CommitDate: Wed, 22 Sep 2010 16:31:19 -0400

jump label: Convert dynamic debug to use jump labels

Convert the 'dynamic debug' infrastructure to use jump labels.

LKML-Reference: <b77627358cea3e27d7be4386f45f66219afb8452.1284733808.git.jbaron@redhat.com>

---
 include/linux/dynamic_debug.h |   39 +++++++++++++-----------
 lib/dynamic_debug.c           |   42 ++-------------------------
 scripts/Makefile.lib          |   11 +------
 scripts/basic/Makefile        |    2 +-
 scripts/basic/hash.c          |   64 -----------------------------------------
 5 files changed, 26 insertions(+), 132 deletions(-)
include/linux/dynamic_debug.h +21 -18
index 52c0da4..bef3cda 100644
@@ -1,6 +1,8 @@
 #ifndef _DYNAMIC_DEBUG_H
 #define _DYNAMIC_DEBUG_H
 
+#include <linux/jump_label.h>
+
 /* dynamic_printk_enabled, and dynamic_printk_enabled2 are bitmasks in which
  * bit n is set to 1 if any modname hashes into the bucket n, 0 otherwise. They
  * use independent hash functions, to reduce the chance of false positives.
@@ -22,8 +24,6 @@ struct _ddebug {
 	const char *function;
 	const char *filename;
 	const char *format;
-	char primary_hash;
-	char secondary_hash;
 	unsigned int lineno:24;
 	/*
  	 * The flags field controls the behaviour at the callsite.
@@ -33,6 +33,7 @@ struct _ddebug {
 #define _DPRINTK_FLAGS_PRINT   (1<<0)  /* printk() a message using the format */
 #define _DPRINTK_FLAGS_DEFAULT 0
 	unsigned int flags:8;
+	char enabled;
 } __attribute__((aligned(8)));
 
 
@@ -42,33 +43,35 @@ int ddebug_add_module(struct _ddebug *tab, unsigned int n,
 #if defined(CONFIG_DYNAMIC_DEBUG)
 extern int ddebug_remove_module(const char *mod_name);
 
-#define __dynamic_dbg_enabled(dd)  ({	     \
-	int __ret = 0;							     \
-	if (unlikely((dynamic_debug_enabled & (1LL << DEBUG_HASH)) &&	     \
-			(dynamic_debug_enabled2 & (1LL << DEBUG_HASH2))))   \
-				if (unlikely(dd.flags))			     \
-					__ret = 1;			     \
-	__ret; })
-
 #define dynamic_pr_debug(fmt, ...) do {					\
+	__label__ do_printk;						\
+	__label__ out;							\
 	static struct _ddebug descriptor				\
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
-	{ KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH,	\
-		DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT };	\
-	if (__dynamic_dbg_enabled(descriptor))				\
-		printk(KERN_DEBUG pr_fmt(fmt),	##__VA_ARGS__);		\
+	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
+		_DPRINTK_FLAGS_DEFAULT };				\
+	JUMP_LABEL(&descriptor.enabled, do_printk);			\
+	goto out;							\
+do_printk:								\
+	printk(KERN_DEBUG pr_fmt(fmt),	##__VA_ARGS__);			\
+out:	;								\
 	} while (0)
 
 
 #define dynamic_dev_dbg(dev, fmt, ...) do {				\
+	__label__ do_printk;						\
+	__label__ out;							\
 	static struct _ddebug descriptor				\
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
-	{ KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH,	\
-		DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT };	\
-	if (__dynamic_dbg_enabled(descriptor))				\
-		dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__);	\
+	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
+		_DPRINTK_FLAGS_DEFAULT };				\
+	JUMP_LABEL(&descriptor.enabled, do_printk);			\
+	goto out;							\
+do_printk:								\
+	dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__);		\
+out:	;								\
 	} while (0)
 
 #else
lib/dynamic_debug.c +3 -39

65 lines not on this page · raw message

scripts/Makefile.lib +1 -10
index 54fd1b7..7bfcf1a 100644
@@ -101,14 +101,6 @@ basename_flags = -D"KBUILD_BASENAME=KBUILD_STR($(call name-fix,$(basetarget)))"
 modname_flags  = $(if $(filter 1,$(words $(modname))),\
                  -D"KBUILD_MODNAME=KBUILD_STR($(call name-fix,$(modname)))")
 
-#hash values
-ifdef CONFIG_DYNAMIC_DEBUG
-debug_flags = -D"DEBUG_HASH=$(shell ./scripts/basic/hash djb2 $(@D)$(modname))"\
-              -D"DEBUG_HASH2=$(shell ./scripts/basic/hash r5 $(@D)$(modname))"
-else
-debug_flags =
-endif
-
 orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
                  $(ccflags-y) $(CFLAGS_$(basetarget).o)
 _c_flags       = $(filter-out $(CFLAGS_REMOVE_$(basetarget).o), $(orig_c_flags))
@@ -152,8 +144,7 @@ endif
 
 c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__c_flags) $(modkern_cflags)                           \
-		 -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags) \
-		  $(debug_flags)
+		 -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags)
 
 a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__a_flags) $(modkern_aflags)
scripts/basic/Makefile +1 -1
index 0955995..4c324a1 100644
@@ -9,7 +9,7 @@
 # fixdep: 	 Used to generate dependency information during build process
 # docproc:	 Used in Documentation/DocBook
 
-hostprogs-y	:= fixdep docproc hash
+hostprogs-y	:= fixdep docproc
 always		:= $(hostprogs-y)
 
 # fixdep is needed to compile other host programs
scripts/basic/hash.c deleted +0 -64

64 lines not on this page · raw message

#35 lore raw

PATCH 8/10 jump label v11: x86 support

In reply to Jason Baron

add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.

---
 arch/x86/Kconfig                  |    1 +
 arch/x86/include/asm/jump_label.h |   47 +++++++++++++++++++++++++++++++++++
 arch/x86/kernel/Makefile          |    2 +-
 arch/x86/kernel/jump_label.c      |   49 +++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/module.c          |    3 ++
 5 files changed, 101 insertions(+), 1 deletions(-)
 create mode 100644 arch/x86/include/asm/jump_label.h
 create mode 100644 arch/x86/kernel/jump_label.c
arch/x86/Kconfig +1 -0
index 0c14369..a986079 100644
@@ -60,6 +60,7 @@ config X86
 	select ANON_INODES
 	select HAVE_ARCH_KMEMCHECK
 	select HAVE_USER_RETURN_NOTIFIER
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config INSTRUCTION_DECODER
 	def_bool (KPROBES || PERF_EVENTS)
arch/x86/include/asm/jump_label.h new file +47 -0

47 lines not on this page · raw message

arch/x86/kernel/Makefile +1 -1
index 11a9925..dfa1443 100644
@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o		:= n
 obj-y			:= process_$(BITS).o signal.o entry_$(BITS).o
 obj-y			+= traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o
 obj-y			+= time.o ioport.o ldt.o dumpstack.o
-obj-y			+= setup.o x86_init.o i8259.o irqinit.o
+obj-y			+= setup.o x86_init.o i8259.o irqinit.o jump_label.o
 obj-$(CONFIG_X86_VISWS)	+= visws_quirks.o
 obj-$(CONFIG_X86_32)	+= probe_roms_32.o
 obj-$(CONFIG_X86_32)	+= sys_i386_32.o i386_ksyms_32.o
arch/x86/kernel/jump_label.c new file +49 -0

49 lines not on this page · raw message

arch/x86/kernel/module.c +3 -0
index e0bc186..5399f58 100644
@@ -239,6 +239,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 		apply_paravirt(pseg, pseg + para->sh_size);
 	}
 
+	/* make jump label nops */
+	jump_label_apply_nops(me);
+
 	return module_bug_finalize(hdr, sechdrs, me);
 }
 
Signature · 1 line
1.7.1
maintainer
#36 lore raw

Re: [PATCH 08/10] jump label v11: x86 support

I'm starting to play with this. I just updated my distcc to gcc 4.5.1 so that I can.

Quoted 68 lines · Jason Baron

add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.

---
 arch/x86/Kconfig                  |    1 +
 arch/x86/include/asm/jump_label.h |   47 +++++++++++++++++++++++++++++++++++
 arch/x86/kernel/Makefile          |    2 +-
 arch/x86/kernel/jump_label.c      |   49 +++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/module.c          |    3 ++
 5 files changed, 101 insertions(+), 1 deletions(-)
 create mode 100644 arch/x86/include/asm/jump_label.h
 create mode 100644 arch/x86/kernel/jump_label.c
arch/x86/Kconfig +1 -0
index 0c14369..a986079 100644
@@ -60,6 +60,7 @@ config X86
 	select ANON_INODES
 	select HAVE_ARCH_KMEMCHECK
 	select HAVE_USER_RETURN_NOTIFIER
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config INSTRUCTION_DECODER
 	def_bool (KPROBES || PERF_EVENTS)
arch/x86/include/asm/jump_label.h new file +35 -0
new file mode 100644
index 0000000..b4a2cb4
@@ -0,0 +1,47 @@
+#ifndef _ASM_X86_JUMP_LABEL_H
+#define _ASM_X86_JUMP_LABEL_H
+
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <asm/nops.h>
+
+#define JUMP_LABEL_NOP_SIZE 5
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label)					\
+	do {							\
+		asm goto("1:"					\
+			JUMP_LABEL_INITIAL_NOP			\
+			".pushsection __jump_table,  \"a\" \n\t"\
+			_ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+			".popsection \n\t"			\
+			: :  "i" (key) :  : label);		\
+	} while (0)
+
+#endif /* __KERNEL__ */
+
+#ifdef CONFIG_X86_64
+
+typedef u64 jump_label_t;
+
+struct jump_entry {
+	jump_label_t code;
+	jump_label_t target;
+	jump_label_t key;
+};
+
+#else

arch/x86/include/asm/jump_label.h line 45

Quoted 10 lines
arch/x86/include/asm/jump_label.h new file +10 -0
+
+typedef u32 jump_label_t;
+
+struct jump_entry {
+	jump_label_t code;
+	jump_label_t target;
+	jump_label_t key;
+};
+
+#endif

Why duplicate the structure? You typedef'd jump_label_t and used that in the structure twice!

-- Steve

Quoted 84 lines
arch/x86/include/asm/jump_label.h new file +2 -0
+
+#endif
arch/x86/kernel/Makefile +1 -1
index 11a9925..dfa1443 100644
@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o		:= n
 obj-y			:= process_$(BITS).o signal.o entry_$(BITS).o
 obj-y			+= traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o
 obj-y			+= time.o ioport.o ldt.o dumpstack.o
-obj-y			+= setup.o x86_init.o i8259.o irqinit.o
+obj-y			+= setup.o x86_init.o i8259.o irqinit.o jump_label.o
 obj-$(CONFIG_X86_VISWS)	+= visws_quirks.o
 obj-$(CONFIG_X86_32)	+= probe_roms_32.o
 obj-$(CONFIG_X86_32)	+= sys_i386_32.o i386_ksyms_32.o
arch/x86/kernel/jump_label.c new file +49 -0
new file mode 100644
index 0000000..b3c0f37
@@ -0,0 +1,49 @@
+/*
+ * jump label x86 support
+ *
+ * Copyright (C) 2009 Jason Baron <[email protected]>
+ *
+ */
+#include <linux/jump_label.h>
+#include <linux/memory.h>
+#include <linux/uaccess.h>
+#include <linux/module.h>
+#include <linux/list.h>
+#include <linux/jhash.h>
+#include <linux/cpu.h>
+#include <asm/kprobes.h>
+#include <asm/alternative.h>
+
+#ifdef HAVE_JUMP_LABEL
+
+union jump_code_union {
+	char code[JUMP_LABEL_NOP_SIZE];
+	struct {
+		char jump;
+		int offset;
+	} __attribute__((packed));
+};
+
+void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+{
+	union jump_code_union code;
+
+	if (type == JUMP_LABEL_ENABLE) {
+		code.jump = 0xe9;
+		code.offset = entry->target -
+				(entry->code + JUMP_LABEL_NOP_SIZE);
+	} else
+		memcpy(&code, ideal_nop5, JUMP_LABEL_NOP_SIZE);
+	get_online_cpus();
+	mutex_lock(&text_mutex);
+	text_poke_smp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE);
+	mutex_unlock(&text_mutex);
+	put_online_cpus();
+}
+
+void arch_jump_label_text_poke_early(jump_label_t addr)
+{
+	text_poke_early((void *)addr, ideal_nop5, JUMP_LABEL_NOP_SIZE);
+}
+
+#endif
arch/x86/kernel/module.c +3 -0
index e0bc186..5399f58 100644
@@ -239,6 +239,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 		apply_paravirt(pseg, pseg + para->sh_size);
 	}
 
+	/* make jump label nops */
+	jump_label_apply_nops(me);
+
 	return module_bug_finalize(hdr, sechdrs, me);
 }
 
#37 lore raw

Re: [PATCH 08/10] jump label v11: x86 support

In reply to Jason Baron

arch/x86/Kconfig config X86 · line 63

Quoted 5 lines · Jason Baron

add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.

arch/x86/Kconfig +1 -0
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE

Why?

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#38 lore raw

Re: [PATCH 08/10] jump label v11: x86 support

Quoted 11 lines · H. Peter Anvin

arch/x86/Kconfig config X86 · line 63

Quoted 5 lines · Jason Baron

add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.

arch/x86/Kconfig +1 -0
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE

Why?

-hpa

Hi,

Quoted 1 line

From the documentation patch:

"
The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is set, gcc does not always out of line the not taken label path in the same way that the "if unlikely()" paths are made out of line. Thus, with CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be solved in subsequent gcc versions, that allow us to move labels out of line, while still optimizing for size.
"

This was also discuessed a bit on lkml previously:

http://marc.info/?l=linux-kernel&m=127618233108964&w=2

thanks,

-Jason

#39 lore raw

Re: [PATCH 08/10] jump label v11: x86 support

Quoted 17 lines · Jason Baron
Quoted 11 lines · H. Peter Anvin

arch/x86/Kconfig config X86 · line 63

Quoted 5 lines · Jason Baron

add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.

arch/x86/Kconfig +1 -0
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE

Why?

-hpa

Hi,

Quoted 1 line

From the documentation patch:

Quoted 7 lines

" The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is set, gcc does not always out of line the not taken label path in the same way that the "if unlikely()" paths are made out of line. Thus, with CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be solved in subsequent gcc versions, that allow us to move labels out of line, while still optimizing for size. "

OTOH making a difficult optimization (HAVE_ARCH_JUMP_LABEL) dependent on compiler flags is really asking for trouble.

So how about enabling it unconditionally, and just chalk up the cost under CC_OPTIMIZE_FOR_SIZE as one of the costs it already has? This also has the advantage that future compilers can improve things without having to wait for yet another kernel patch that re-enables HAVE_ARCH_JUMP_LABEL.

Thanks,

Ingo

maintainer
#40 lore raw

Re: [PATCH 08/10] jump label v11: x86 support

Quoted 10 lines · Ingo Molnar
Quoted 9 lines
Quoted 1 line

From the documentation patch:

" The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is set, gcc does not always out of line the not taken label path in the same way that the "if unlikely()" paths are made out of line. Thus, with CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be solved in subsequent gcc versions, that allow us to move labels out of line, while still optimizing for size. "

Quoted 8 lines

OTOH making a difficult optimization (HAVE_ARCH_JUMP_LABEL) dependent on compiler flags is really asking for trouble.

So how about enabling it unconditionally, and just chalk up the cost under CC_OPTIMIZE_FOR_SIZE as one of the costs it already has? This also has the advantage that future compilers can improve things without having to wait for yet another kernel patch that re-enables HAVE_ARCH_JUMP_LABEL.

Agreed,

CC_OPTIMIZE_FOR_SIZE does not mean OPTIMIZE_FOR_PERFORMANCE. Although people have argued that with smaller size you gain better cache performance. I've noticed that the general case is that optimizing for size has decreased performance (although I have not done any official benchmarks, just my own personal observations).

I thought you may have had that there because OPTIMIZE_FOR_SIZE actually broke the code (as some gcc compilers do for function graph tracer). If its just a "we don't perform better with this set". Then get rid of it.

Thanks,

-- Steve

#41 lore raw

Re: [PATCH 08/10] jump label v11: x86 support

Quoted 29 lines · Steven Rostedt
Quoted 10 lines · Ingo Molnar
Quoted 9 lines
Quoted 1 line

From the documentation patch:

" The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is set, gcc does not always out of line the not taken label path in the same way that the "if unlikely()" paths are made out of line. Thus, with CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be solved in subsequent gcc versions, that allow us to move labels out of line, while still optimizing for size. "

Quoted 8 lines

OTOH making a difficult optimization (HAVE_ARCH_JUMP_LABEL) dependent on compiler flags is really asking for trouble.

So how about enabling it unconditionally, and just chalk up the cost under CC_OPTIMIZE_FOR_SIZE as one of the costs it already has? This also has the advantage that future compilers can improve things without having to wait for yet another kernel patch that re-enables HAVE_ARCH_JUMP_LABEL.

Agreed,

CC_OPTIMIZE_FOR_SIZE does not mean OPTIMIZE_FOR_PERFORMANCE. Although people have argued that with smaller size you gain better cache performance. I've noticed that the general case is that optimizing for size has decreased performance (although I have not done any official benchmarks, just my own personal observations).

Quoted 4 lines

I thought you may have had that there because OPTIMIZE_FOR_SIZE actually broke the code (as some gcc compilers do for function graph tracer). If its just a "we don't perform better with this set". Then get rid of it.

No, OPTIMIZE_FOR_SIZE doesn't break anything, its just that the 'dormant' or the cold code path, isn't moved out of line.

I believe the current approach will eventually lead us to an optimal solution, (single no-op in the fast path, with ability to jump patch to out-of-line cold code), even if its not quite there yet under this compiler flag. And the inclusion of this code will provide more of an impetus for gcc to make further optimizations.

thanks,

-Jason

#42 lore raw

Re: [PATCH 08/10] jump label v11: x86 support

In reply to Jason Baron
Quoted 65 lines · Jason Baron

add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.

---
 arch/x86/Kconfig                  |    1 +
 arch/x86/include/asm/jump_label.h |   47 +++++++++++++++++++++++++++++++++++
 arch/x86/kernel/Makefile          |    2 +-
 arch/x86/kernel/jump_label.c      |   49 +++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/module.c          |    3 ++
 5 files changed, 101 insertions(+), 1 deletions(-)
 create mode 100644 arch/x86/include/asm/jump_label.h
 create mode 100644 arch/x86/kernel/jump_label.c
arch/x86/Kconfig +1 -0
index 0c14369..a986079 100644
@@ -60,6 +60,7 @@ config X86
 	select ANON_INODES
 	select HAVE_ARCH_KMEMCHECK
 	select HAVE_USER_RETURN_NOTIFIER
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config INSTRUCTION_DECODER
 	def_bool (KPROBES || PERF_EVENTS)
arch/x86/include/asm/jump_label.h new file +32 -0
new file mode 100644
index 0000000..b4a2cb4
@@ -0,0 +1,47 @@
+#ifndef _ASM_X86_JUMP_LABEL_H
+#define _ASM_X86_JUMP_LABEL_H
+
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <asm/nops.h>
+
+#define JUMP_LABEL_NOP_SIZE 5
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label)					\
+	do {							\
+		asm goto("1:"					\
+			JUMP_LABEL_INITIAL_NOP			\
+			".pushsection __jump_table,  \"a\" \n\t"\
+			_ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+			".popsection \n\t"			\
+			: :  "i" (key) :  : label);		\
+	} while (0)
+
+#endif /* __KERNEL__ */
+
+#ifdef CONFIG_X86_64
+
+typedef u64 jump_label_t;
+
+struct jump_entry {
+	jump_label_t code;
+	jump_label_t target;
+	jump_label_t key;

arch/x86/include/asm/jump_label.h line 42

Quoted 10 lines
arch/x86/include/asm/jump_label.h new file +10 -0
+};
+
+#else
+
+typedef u32 jump_label_t;
+
+struct jump_entry {
+	jump_label_t code;
+	jump_label_t target;
+	jump_label_t key;

The format is the same, why not just put the typedef within the #ifdef?

Quoted 24 lines
arch/x86/include/asm/jump_label.h new file +5 -0
+};
+
+#endif
+
+#endif
arch/x86/kernel/Makefile +1 -1
index 11a9925..dfa1443 100644
@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o		:= n
 obj-y			:= process_$(BITS).o signal.o entry_$(BITS).o
 obj-y			+= traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o
 obj-y			+= time.o ioport.o ldt.o dumpstack.o
-obj-y			+= setup.o x86_init.o i8259.o irqinit.o
+obj-y			+= setup.o x86_init.o i8259.o irqinit.o jump_label.o
 obj-$(CONFIG_X86_VISWS)	+= visws_quirks.o
 obj-$(CONFIG_X86_32)	+= probe_roms_32.o
 obj-$(CONFIG_X86_32)	+= sys_i386_32.o i386_ksyms_32.o
arch/x86/kernel/jump_label.c new file +0 -0
new file mode 100644
index 0000000..b3c0f37

arch/x86/kernel/jump_label.c line 4

Quoted 4 lines
arch/x86/kernel/jump_label.c new file +4 -0
@@ -0,0 +1,49 @@
+/*
+ * jump label x86 support
+ *
+ * Copyright (C) 2009 Jason Baron <[email protected]>

Year.

#43 lore raw

[tip:perf/core] jump label: x86 support

In reply to Jason Baron
Commit-ID:  d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
Gitweb:     http://git.kernel.org/tip/d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
Author:     Jason Baron <[email protected]>
AuthorDate: Fri, 17 Sep 2010 11:09:22 -0400
Committer:  Steven Rostedt <[email protected]>
CommitDate: Wed, 22 Sep 2010 16:33:03 -0400

jump label: x86 support

add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. Hopefully, it wouldn't be too painful for other archs.

LKML-Reference: <f838f49f40fbea0254036194be66dc48b598dcea.1284733808.git.jbaron@redhat.com>

[ cleaned up some formatting ]

---
 arch/x86/Kconfig                  |    1 +
 arch/x86/include/asm/jump_label.h |   47 ++++++++++++++++++++++++++++++++++
 arch/x86/kernel/Makefile          |    2 +-
 arch/x86/kernel/jump_label.c      |   50 +++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/module.c          |    3 ++
 5 files changed, 102 insertions(+), 1 deletions(-)
arch/x86/Kconfig +1 -0
index cea0cd9..afcd663 100644
@@ -59,6 +59,7 @@ config X86
 	select ANON_INODES
 	select HAVE_ARCH_KMEMCHECK
 	select HAVE_USER_RETURN_NOTIFIER
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config INSTRUCTION_DECODER
 	def_bool (KPROBES || PERF_EVENTS)
arch/x86/include/asm/jump_label.h new file +47 -0

47 lines not on this page · raw message

arch/x86/kernel/Makefile +1 -1
index 0925676..24fa171 100644
@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o		:= n
 obj-y			:= process_$(BITS).o signal.o entry_$(BITS).o
 obj-y			+= traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o
 obj-y			+= time.o ioport.o ldt.o dumpstack.o
-obj-y			+= setup.o x86_init.o i8259.o irqinit.o
+obj-y			+= setup.o x86_init.o i8259.o irqinit.o jump_label.o
 obj-$(CONFIG_X86_VISWS)	+= visws_quirks.o
 obj-$(CONFIG_X86_32)	+= probe_roms_32.o
 obj-$(CONFIG_X86_32)	+= sys_i386_32.o i386_ksyms_32.o
arch/x86/kernel/jump_label.c new file +50 -0

50 lines not on this page · raw message

arch/x86/kernel/module.c +3 -0
index e0bc186..5399f58 100644
@@ -239,6 +239,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 		apply_paravirt(pseg, pseg + para->sh_size);
 	}
 
+	/* make jump label nops */
+	jump_label_apply_nops(me);
+
 	return module_bug_finalize(hdr, sechdrs, me);
 }
 
#44 lore raw

Re: [tip:perf/core] jump label: x86 support

Quoted 32 lines · tip-bot for Jason Baron
Commit-ID:  d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
Gitweb:     http://git.kernel.org/tip/d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
Author:     Jason Baron <[email protected]>
AuthorDate: Fri, 17 Sep 2010 11:09:22 -0400
Committer:  Steven Rostedt <[email protected]>
CommitDate: Wed, 22 Sep 2010 16:33:03 -0400

jump label: x86 support

add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. Hopefully, it wouldn't be too painful for other archs.

LKML-Reference: <f838f49f40fbea0254036194be66dc48b598dcea.1284733808.git.jbaron@redhat.com>

[ cleaned up some formatting ]

---
 arch/x86/Kconfig                  |    1 +
 arch/x86/include/asm/jump_label.h |   47 ++++++++++++++++++++++++++++++++++
 arch/x86/kernel/Makefile          |    2 +-
 arch/x86/kernel/jump_label.c      |   50 +++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/module.c          |    3 ++
 5 files changed, 102 insertions(+), 1 deletions(-)
arch/x86/Kconfig +0 -0
index cea0cd9..afcd663 100644

arch/x86/Kconfig config X86 · line 65

Quoted 7 lines
arch/x86/Kconfig +1 -0
@@ -59,6 +59,7 @@ config X86
 	select ANON_INODES
 	select HAVE_ARCH_KMEMCHECK
 	select HAVE_USER_RETURN_NOTIFIER
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config INSTRUCTION_DECODER
 	def_bool (KPROBES || PERF_EVENTS)

What the heck? I had NAKed at least this chunk already...

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#45 lore raw

Re: [tip:perf/core] jump label: x86 support

Quoted 38 lines · H. Peter Anvin
Quoted 37 lines · tip-bot for Jason Baron
Commit-ID:  d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
Gitweb:     http://git.kernel.org/tip/d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e
Author:     Jason Baron <[email protected]>
AuthorDate: Fri, 17 Sep 2010 11:09:22 -0400
Committer:  Steven Rostedt <[email protected]>
CommitDate: Wed, 22 Sep 2010 16:33:03 -0400

jump label: x86 support

add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. Hopefully, it wouldn't be too painful for other archs.

LKML-Reference: <f838f49f40fbea0254036194be66dc48b598dcea.1284733808.git.jbaron@redhat.com>

[ cleaned up some formatting ]

---
 arch/x86/Kconfig                  |    1 +
 arch/x86/include/asm/jump_label.h |   47 ++++++++++++++++++++++++++++++++++
 arch/x86/kernel/Makefile          |    2 +-
 arch/x86/kernel/jump_label.c      |   50 +++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/module.c          |    3 ++
 5 files changed, 102 insertions(+), 1 deletions(-)
arch/x86/Kconfig +1 -0
index cea0cd9..afcd663 100644
@@ -59,6 +59,7 @@ config X86
 	select ANON_INODES
 	select HAVE_ARCH_KMEMCHECK
 	select HAVE_USER_RETURN_NOTIFIER
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
Quoted 7 lines

arch/x86/Kconfig config X86 · line 65

Quoted 2 lines
arch/x86/Kconfig +0 -0
 config INSTRUCTION_DECODER
 	def_bool (KPROBES || PERF_EVENTS)

What the heck? I had NAKed at least this chunk already...

-hpa

The !CC_OPTIMIZE_FOR_SIZE condition is subsequently removed by a follow-on patch as part of the commit series.

thanks,

-Jason

#46 lore raw

Re: [tip:perf/core] jump label: x86 support

Quoted 4 lines · Jason Baron

The !CC_OPTIMIZE_FOR_SIZE condition is subsequently removed by a follow-on patch as part of the commit series.

Ah, okay. No problem then.

-hpa

maintainer
#47 lore raw

Re: [tip:perf/core] jump label: x86 support

Quoted 7 lines · H. Peter Anvin
Quoted 4 lines · Jason Baron

The !CC_OPTIMIZE_FOR_SIZE condition is subsequently removed by a follow-on patch as part of the commit series.

Ah, okay. No problem then.

Sorry, I should have Cc'd you on that patch.

-- Steve

maintainer
#48 lore raw

Re: [tip:perf/core] jump label: x86 support

In reply to tip-bot for Jason Baron
Quoted 7 lines · tip-bot for Jason Baron
arch/x86/kernel/jump_label.c new file +7 -0
+void arch_jump_label_transform(struct jump_entry *entry,
+                              enum jump_label_type type)
+{
+       union jump_code_union code;
+
+       if (type == JUMP_LABEL_ENABLE) {
+               code.jump = 0xe9;

arch/x86/kernel/jump_label.c line 43

Quoted 10 lines
arch/x86/kernel/jump_label.c new file +10 -0
+               code.offset = entry->target -
+                               (entry->code + JUMP_LABEL_NOP_SIZE);
+       } else
+               memcpy(&code, ideal_nop5, JUMP_LABEL_NOP_SIZE);
+       get_online_cpus();
+       mutex_lock(&text_mutex);
+       text_poke_smp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE);
+       mutex_unlock(&text_mutex);
+       put_online_cpus();
+} 

hpa, mathieu, what's the status of stop_machine less text poking? Because the above basically means we have to disable architecture jump_label support for -rt.

#49 lore raw

Re: [tip:perf/core] jump label: x86 support

Quoted 14 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 13 lines · tip-bot for Jason Baron
arch/x86/kernel/jump_label.c new file +13 -0
+void arch_jump_label_transform(struct jump_entry *entry,
+                              enum jump_label_type type)
+{
+       union jump_code_union code;
+
+       if (type == JUMP_LABEL_ENABLE) {
+               code.jump = 0xe9;
+               code.offset = entry->target -
+                               (entry->code + JUMP_LABEL_NOP_SIZE);
+       } else
+               memcpy(&code, ideal_nop5, JUMP_LABEL_NOP_SIZE);
+       get_online_cpus();
+       mutex_lock(&text_mutex);
Quoted 8 lines

arch/x86/kernel/jump_label.c line 43

Quoted 4 lines
arch/x86/kernel/jump_label.c new file +4 -0
+       text_poke_smp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE);
+       mutex_unlock(&text_mutex);
+       put_online_cpus();
+} 

hpa, mathieu, what's the status of stop_machine less text poking? Because the above basically means we have to disable architecture jump_label support for -rt.

hpa got an unofficial answer from Intel OTC, that states the stop_machine-less text poking procedure is valid.

http://lkml.org/lkml/2010/1/12/300

We are still waiting for the "official" blessing though.

Thanks,

Mathieu

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#50 lore raw

PATCH 9/10 jump label 11: add sparc64 support

In reply to Jason Baron

add sparc64 support

---
 arch/sparc/Kconfig                  |    1 +
 arch/sparc/include/asm/jump_label.h |   32 ++++++++++++++++++++++++
 arch/sparc/kernel/Makefile          |    2 +
 arch/sparc/kernel/jump_label.c      |   46 +++++++++++++++++++++++++++++++++++
 arch/sparc/kernel/module.c          |    6 ++++
 5 files changed, 87 insertions(+), 0 deletions(-)
 create mode 100644 arch/sparc/include/asm/jump_label.h
 create mode 100644 arch/sparc/kernel/jump_label.c
arch/sparc/Kconfig +1 -0
index 491e9d6..a81b04e 100644
@@ -30,6 +30,7 @@ config SPARC
 	select PERF_USE_VMALLOC
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_API_DEBUG
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config SPARC32
 	def_bool !64BIT
arch/sparc/include/asm/jump_label.h new file +32 -0
new file mode 100644
index 0000000..62e66d7
@@ -0,0 +1,32 @@
+#ifndef _ASM_SPARC_JUMP_LABEL_H
+#define _ASM_SPARC_JUMP_LABEL_H
+
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <asm/system.h>
+
+#define JUMP_LABEL_NOP_SIZE 4
+
+#define JUMP_LABEL(key, label)					\
+	do {							\
+		asm goto("1:\n\t"				\
+			 "nop\n\t"				\
+			 "nop\n\t"				\
+			 ".pushsection __jump_table,  \"a\"\n\t"\
+			 ".word 1b, %l[" #label "], %c0\n\t"	\
+			 ".popsection \n\t"			\
+			 : :  "i" (key) :  : label);\
+	} while (0)
+
+#endif /* __KERNEL__ */
+
+typedef u32 jump_label_t;
+
+struct jump_entry {
+	jump_label_t code;
+	jump_label_t target;
+	jump_label_t key;
+};
+
+#endif
arch/sparc/kernel/Makefile +2 -0
index 0c2dc1f..599398f 100644
@@ -119,3 +119,5 @@ obj-$(CONFIG_COMPAT)    += $(audit--y)
 
 pc--$(CONFIG_PERF_EVENTS) := perf_event.o
 obj-$(CONFIG_SPARC64)	+= $(pc--y)
+
+obj-$(CONFIG_SPARC64)	+= jump_label.o
arch/sparc/kernel/jump_label.c new file +46 -0

46 lines not on this page · raw message

arch/sparc/kernel/module.c +6 -0
index f848aad..ee3c7dd 100644
@@ -18,6 +18,9 @@
 #include <asm/spitfire.h>
 
 #ifdef CONFIG_SPARC64
+
+#include <linux/jump_label.h>
+
 static void *module_map(unsigned long size)
 {
 	struct vm_struct *area;
@@ -227,6 +230,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 		    const Elf_Shdr *sechdrs,
 		    struct module *me)
 {
+	/* make jump label nops */
+	jump_label_apply_nops(me);
+
 	/* Cheetah's I-cache is fully coherent.  */
 	if (tlb_type == spitfire) {
 		unsigned long va;
Signature · 1 line
1.7.1
maintainer
#51 lore raw

Re: [PATCH 09/10] jump label 11: add sparc64 support

Jason, did you author this patch or did David?

-- Steve

Quoted 150 lines · Jason Baron

add sparc64 support

---
 arch/sparc/Kconfig                  |    1 +
 arch/sparc/include/asm/jump_label.h |   32 ++++++++++++++++++++++++
 arch/sparc/kernel/Makefile          |    2 +
 arch/sparc/kernel/jump_label.c      |   46 +++++++++++++++++++++++++++++++++++
 arch/sparc/kernel/module.c          |    6 ++++
 5 files changed, 87 insertions(+), 0 deletions(-)
 create mode 100644 arch/sparc/include/asm/jump_label.h
 create mode 100644 arch/sparc/kernel/jump_label.c
arch/sparc/Kconfig +1 -0
index 491e9d6..a81b04e 100644
@@ -30,6 +30,7 @@ config SPARC
 	select PERF_USE_VMALLOC
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_API_DEBUG
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config SPARC32
 	def_bool !64BIT
arch/sparc/include/asm/jump_label.h new file +32 -0
new file mode 100644
index 0000000..62e66d7
@@ -0,0 +1,32 @@
+#ifndef _ASM_SPARC_JUMP_LABEL_H
+#define _ASM_SPARC_JUMP_LABEL_H
+
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <asm/system.h>
+
+#define JUMP_LABEL_NOP_SIZE 4
+
+#define JUMP_LABEL(key, label)					\
+	do {							\
+		asm goto("1:\n\t"				\
+			 "nop\n\t"				\
+			 "nop\n\t"				\
+			 ".pushsection __jump_table,  \"a\"\n\t"\
+			 ".word 1b, %l[" #label "], %c0\n\t"	\
+			 ".popsection \n\t"			\
+			 : :  "i" (key) :  : label);\
+	} while (0)
+
+#endif /* __KERNEL__ */
+
+typedef u32 jump_label_t;
+
+struct jump_entry {
+	jump_label_t code;
+	jump_label_t target;
+	jump_label_t key;
+};
+
+#endif
arch/sparc/kernel/Makefile +2 -0
index 0c2dc1f..599398f 100644
@@ -119,3 +119,5 @@ obj-$(CONFIG_COMPAT)    += $(audit--y)
 
 pc--$(CONFIG_PERF_EVENTS) := perf_event.o
 obj-$(CONFIG_SPARC64)	+= $(pc--y)
+
+obj-$(CONFIG_SPARC64)	+= jump_label.o
arch/sparc/kernel/jump_label.c new file +46 -0
new file mode 100644
index 0000000..00b3fcb
@@ -0,0 +1,46 @@
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/mutex.h>
+#include <linux/cpu.h>
+
+#include <linux/jump_label.h>
+#include <linux/memory.h>
+
+#ifdef HAVE_JUMP_LABEL
+
+void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+{
+	u32 val;
+	u32 *insn = (u32 *) (unsigned long) entry->code;
+
+	if (type == JUMP_LABEL_ENABLE) {
+		s32 off = (s32)entry->target - (s32)entry->code;
+
+#ifdef CONFIG_SPARC64
+		/* ba,pt %xcc, . + (off << 2) */
+		val = 0x10680000 | ((u32) off >> 2);
+#else
+		/* ba . + (off << 2) */
+		val = 0x10800000 | ((u32) off >> 2);
+#endif
+	} else {
+		val = 0x01000000;
+	}
+
+	get_online_cpus();
+	mutex_lock(&text_mutex);
+	*insn = val;
+	flushi(insn);
+	mutex_unlock(&text_mutex);
+	put_online_cpus();
+}
+
+void arch_jump_label_text_poke_early(jump_label_t addr)
+{
+	u32 *insn_p = (u32 *) (unsigned long) addr;
+
+	*insn_p = 0x01000000;
+	flushi(insn_p);
+}
+
+#endif
arch/sparc/kernel/module.c +6 -0
index f848aad..ee3c7dd 100644
@@ -18,6 +18,9 @@
 #include <asm/spitfire.h>
 
 #ifdef CONFIG_SPARC64
+
+#include <linux/jump_label.h>
+
 static void *module_map(unsigned long size)
 {
 	struct vm_struct *area;
@@ -227,6 +230,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 		    const Elf_Shdr *sechdrs,
 		    struct module *me)
 {
+	/* make jump label nops */
+	jump_label_apply_nops(me);
+
 	/* Cheetah's I-cache is fully coherent.  */
 	if (tlb_type == spitfire) {
 		unsigned long va;
maintainer
#52 lore raw

Re: [PATCH 09/10] jump label 11: add sparc64 support

From: Steven Rostedt <[email protected]>
Date: Mon, 20 Sep 2010 18:25:46 -0400

Quoted 2 lines

Jason, did you author this patch or did David?

I did.

Jason when you post patches other people have authored, you should add a:

From: Mr. Author <[email protected]>

to the top of your commit message.

This allows automated tools that pull patches into GIT to set the author correctly when it is not you.

Thanks.

maintainer
#53 lore raw

Re: [PATCH 09/10] jump label 11: add sparc64 support

Quoted 11 lines · David Miller

From: Steven Rostedt <[email protected]>
Date: Mon, 20 Sep 2010 18:25:46 -0400

Quoted 2 lines

Jason, did you author this patch or did David?

I did.

Jason when you post patches other people have authored, you should add a:

Quoted 8 lines

From: Mr. Author <[email protected]>

to the top of your commit message.

This allows automated tools that pull patches into GIT to set the author correctly when it is not you.

Thanks.

Thanks, I figured you did when I saw your SoB. I'll add your From line manually, when I pull it.

-- Steve

maintainer
#54 lore raw

Re: [PATCH 09/10] jump label 11: add sparc64 support

In reply to Jason Baron

arch/sparc/Kconfig config SPARC · line 34

Quoted 10 lines · Jason Baron
arch/sparc/Kconfig +1 -0
index 491e9d6..a81b04e 100644
@@ -30,6 +30,7 @@ config SPARC
 	select PERF_USE_VMALLOC
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_API_DEBUG
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 

David,

If you look at the thread for patch #8, do you still want that
" if !CC_OPTIMIZE_FOR_SIZE"?

-- Steve

maintainer
#55 lore raw

Re: [PATCH 09/10] jump label 11: add sparc64 support

From: Steven Rostedt <[email protected]>
Date: Tue, 21 Sep 2010 11:37:33 -0400

Quoted 15 lines · Jason Baron

arch/sparc/Kconfig config SPARC · line 34

Quoted 11 lines
arch/sparc/Kconfig +1 -0
index 491e9d6..a81b04e 100644
@@ -30,6 +30,7 @@ config SPARC
 	select PERF_USE_VMALLOC
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_API_DEBUG
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 

David,

Quoted 2 lines

If you look at the thread for patch #8, do you still want that
" if !CC_OPTIMIZE_FOR_SIZE"?

No :-)

maintainer
#56 lore raw

Re: [PATCH 09/10] jump label 11: add sparc64 support

Quoted 18 lines · David Miller

From: Steven Rostedt <[email protected]>
Date: Tue, 21 Sep 2010 11:37:33 -0400

Quoted 15 lines · Jason Baron

arch/sparc/Kconfig config SPARC · line 34

Quoted 11 lines
arch/sparc/Kconfig +1 -0
index 491e9d6..a81b04e 100644
@@ -30,6 +30,7 @@ config SPARC
 	select PERF_USE_VMALLOC
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_API_DEBUG
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 

David,

Quoted 4 lines
Quoted 2 lines

If you look at the thread for patch #8, do you still want that
" if !CC_OPTIMIZE_FOR_SIZE"?

No :-)

I'll be sending a pull request to Ingo soon. I'll add this change with your Acked-by.

Thanks,

-- Steve

#57 lore raw

[tip:perf/core] jump label: Add sparc64 support

In reply to Jason Baron
Commit-ID:  dff9d3c215251022dd8bb3823c9f75edb4b63fe9
Gitweb:     http://git.kernel.org/tip/dff9d3c215251022dd8bb3823c9f75edb4b63fe9
Author:     David S. Miller <[email protected]>
AuthorDate: Fri, 17 Sep 2010 11:09:25 -0400
Committer:  Steven Rostedt <[email protected]>
CommitDate: Wed, 22 Sep 2010 16:35:09 -0400

jump label: Add sparc64 support

Add jump label support for sparc64.

LKML-Reference: <3b5b071fcdb2afb7f67cacecfa78b14c740278a7.1284733808.git.jbaron@redhat.com>

[ cleaned up some formatting ]

---
 arch/sparc/Kconfig                  |    1 +
 arch/sparc/include/asm/jump_label.h |   32 +++++++++++++++++++++++
 arch/sparc/kernel/Makefile          |    2 +
 arch/sparc/kernel/jump_label.c      |   47 +++++++++++++++++++++++++++++++++++
 arch/sparc/kernel/module.c          |    6 ++++
 5 files changed, 88 insertions(+), 0 deletions(-)
arch/sparc/Kconfig +1 -0
index 491e9d6..a81b04e 100644
@@ -30,6 +30,7 @@ config SPARC
 	select PERF_USE_VMALLOC
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_API_DEBUG
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config SPARC32
 	def_bool !64BIT
arch/sparc/include/asm/jump_label.h new file +32 -0
new file mode 100644
index 0000000..62e66d7
@@ -0,0 +1,32 @@
+#ifndef _ASM_SPARC_JUMP_LABEL_H
+#define _ASM_SPARC_JUMP_LABEL_H
+
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <asm/system.h>
+
+#define JUMP_LABEL_NOP_SIZE 4
+
+#define JUMP_LABEL(key, label)					\
+	do {							\
+		asm goto("1:\n\t"				\
+			 "nop\n\t"				\
+			 "nop\n\t"				\
+			 ".pushsection __jump_table,  \"a\"\n\t"\
+			 ".word 1b, %l[" #label "], %c0\n\t"	\
+			 ".popsection \n\t"			\
+			 : :  "i" (key) :  : label);\
+	} while (0)
+
+#endif /* __KERNEL__ */
+
+typedef u32 jump_label_t;
+
+struct jump_entry {
+	jump_label_t code;
+	jump_label_t target;
+	jump_label_t key;
+};
+
+#endif
arch/sparc/kernel/Makefile +2 -0
index 0c2dc1f..599398f 100644
@@ -119,3 +119,5 @@ obj-$(CONFIG_COMPAT)    += $(audit--y)
 
 pc--$(CONFIG_PERF_EVENTS) := perf_event.o
 obj-$(CONFIG_SPARC64)	+= $(pc--y)
+
+obj-$(CONFIG_SPARC64)	+= jump_label.o
arch/sparc/kernel/jump_label.c new file +47 -0

47 lines not on this page · raw message

arch/sparc/kernel/module.c +6 -0
index f848aad..ee3c7dd 100644
@@ -18,6 +18,9 @@
 #include <asm/spitfire.h>
 
 #ifdef CONFIG_SPARC64
+
+#include <linux/jump_label.h>
+
 static void *module_map(unsigned long size)
 {
 	struct vm_struct *area;
@@ -227,6 +230,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 		    const Elf_Shdr *sechdrs,
 		    struct module *me)
 {
+	/* make jump label nops */
+	jump_label_apply_nops(me);
+
 	/* Cheetah's I-cache is fully coherent.  */
 	if (tlb_type == spitfire) {
 		unsigned long va;
#58 lore raw

PATCH 10/10 jump label v11: add docs

In reply to Jason Baron

Add jump label docs as: Documentation/jump-label.txt

---
 Documentation/jump-label.txt |  148 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 148 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/jump-label.txt
Documentation/jump-label.txt new file +148 -0
new file mode 100644
index 0000000..2e5cff6
@@ -0,0 +1,148 @@
+			Jump Label
+			----------
+
+By: Jason Baron <[email protected]>
+
+
+1) motivation
+
+
+Currently, tracepoints are implemented using a conditional. The conditional
+check requires checking a global variable for each tracepoint. Although,
+the overhead of this check is small, it increases under memory pressure. As we
+increase the number of tracepoints in the kernel this may become more of an
+issue. In addition, tracepoints are often dormant (disabled), and provide no
+direct kernel functionality. Thus, it is highly desirable to reduce their
+impact as much as possible. Although tracepoints are the original motivation
+for this work, other kernel code paths should be able to make use of the jump
+label optimization.
+
+
+2) jump label description/usage
+
+
+gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label.
+http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html
+
+Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as
+follows (x86):
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label)					\
+	do {                                                    \
+		asm goto("1:"                                   \
+			JUMP_LABEL_INITIAL_NOP			\
+                        ".pushsection __jump_table,  \"a\" \n\t"\
+                        _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+                        ".popsection \n\t"                      \
+                        : :  "i" (key) :  : label);		\
+        } while (0)
+
+
+For architectures that have not yet introduced jump label support its simply:
+
+#define JUMP_LABEL(key, label)			\
+	if (unlikely(*key))			\
+		goto label;
+
+which then can be used as:
+
+	....
+        JUMP_LABEL(trace_name, trace_label, jump_enabled);
+        printk("not doing tracing\n");
+	return;
+trace_label:
+        printk("doing tracing: %d\n", file);
+	....
+
+The 'key' argument is thus a pointer to a conditional argument that can be used
+if the optimization is not enabled. Otherwise, this address serves as a unique
+key to identify the particular instance of the jump label.
+
+Thus, when tracing is disabled, we simply have a no-op followed by a jump around
+the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a
+'jump_table' which has the following format:
+
+[instruction address] [jump target] [tracepoint key]
+
+Thus, to enable a tracepoint, we simply patch the 'instruction address' with
+a jump to the 'jump target'.
+
+The call to enable a jump label is: enable_jump_label(key); to disable:
+disable_jump_label(key);
+
+
+3) architecture interface
+
+
+There are a few functions and macros which arches must implement in order to
+take advantage of this optimization. As previously mentioned, if there is no
+architecture support we simply fall back to a traditional, load, test, and
+jump sequence.
+
+* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support
+
+* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h
+
+* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h
+
+* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+	see: arch/x86/kernel/jump_label.c
+
+* void arch_jump_label_text_poke_early(jump_label_t addr)
+	see: arch/x86/kernel/jump_label.c
+
+* finally add a definition for "struct jump_entry".
+	see: arch/x86/include/asm/jump_label.h
+
+
+4) Jump label analysis (x86)
+
+
+I've tested the performance of using 'get_cycles()' calls around the
+tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages):
+
+		idle		after tbench run
+		----		----------------
+old code	 32		  88
+new code	  2		   4
+
+
+The performance improvement can be reproduced reliably on both Intel and AMD
+hardware.
+
+In terms of code analysis the current code for the disabled case is a 'cmpl'
+followed by a 'je' around the tracepoint code. so:
+
+cmpl - 83 3d 0e 77 87 00 00 - 7 bytes
+je   - 74 3e                - 2 bytes
+
+total of 9 instruction bytes.
+
+The new code is a 'nopl' followed by a 'jmp'. Thus:
+
+nopl - 0f 1f 44 00 00 - 5 bytes
+jmp  - eb 3e          - 2 bytes
+
+total of 7 instruction bytes.
+
+So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.
+
+The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is
+set, gcc does not always out of line the not taken label path in the same way
+that the "if unlikely()" paths are made out of line. Thus, with
+CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be
+solved in subsequent gcc versions, that allow us to move labels out of line,
+while still optimizing for size.
+
+
+5) Acknowledgments
+
+
+Thanks to Roland McGrath and Richard Henderson for helping come up with the
+initial 'asm goto' and jump label design.
+
+Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this
+issue, and outlining the requirements of a solution. Mathieu also implemened a
+solution in the form of the "Immediate Values" work.
Signature · 1 line
1.7.1
#59 lore raw

Re: [PATCH 10/10] jump label v11: add docs

Quoted 14 lines · * Jason Baron ([email protected]) wrote:

Add jump label docs as: Documentation/jump-label.txt

---
 Documentation/jump-label.txt |  148 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 148 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/jump-label.txt
Documentation/jump-label.txt new file +0 -0
new file mode 100644
index 0000000..2e5cff6

Documentation/jump-label.txt line 7

Quoted 7 lines
Documentation/jump-label.txt new file +7 -0
@@ -0,0 +1,148 @@
+			Jump Label
+			----------
+
+By: Jason Baron <[email protected]>
+
+
+1) motivation

You might want to start your titles with capital letters (small nit).

Documentation/jump-label.txt line 12

Quoted 5 lines
Documentation/jump-label.txt new file +5 -0
+
+
+Currently, tracepoints are implemented using a conditional. The conditional
+check requires checking a global variable for each tracepoint. Although,
+the overhead of this check is small, it increases under memory pressure. As we

I wonder if it would be worthwhile to spell out the difference between "memory pressure" in terms of memory bandwidth (which is what we care about here) and in terms of "memory space available". I just don't want people to get mixed up between the two when they read this text. What we care a lot about here is just memory bandwidth, which is impacted by cache pressure.

Documentation/jump-label.txt line 21

Quoted 9 lines
Documentation/jump-label.txt new file +9 -0
+increase the number of tracepoints in the kernel this may become more of an
+issue. In addition, tracepoints are often dormant (disabled), and provide no
+direct kernel functionality. Thus, it is highly desirable to reduce their
+impact as much as possible. Although tracepoints are the original motivation
+for this work, other kernel code paths should be able to make use of the jump
+label optimization.
+
+
+2) jump label description/usage

Start with capital letter here too.

Quoted 45 lines
Documentation/jump-label.txt new file +45 -0
+
+
+gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label.
+http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html
+
+Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as
+follows (x86):
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label)					\
+	do {                                                    \
+		asm goto("1:"                                   \
+			JUMP_LABEL_INITIAL_NOP			\
+                        ".pushsection __jump_table,  \"a\" \n\t"\
+                        _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+                        ".popsection \n\t"                      \
+                        : :  "i" (key) :  : label);		\
+        } while (0)
+
+
+For architectures that have not yet introduced jump label support its simply:
+
+#define JUMP_LABEL(key, label)			\
+	if (unlikely(*key))			\
+		goto label;
+
+which then can be used as:
+
+	....
+        JUMP_LABEL(trace_name, trace_label, jump_enabled);
+        printk("not doing tracing\n");
+	return;
+trace_label:
+        printk("doing tracing: %d\n", file);
+	....
+
+The 'key' argument is thus a pointer to a conditional argument that can be used
+if the optimization is not enabled. Otherwise, this address serves as a unique
+key to identify the particular instance of the jump label.
+
+Thus, when tracing is disabled, we simply have a no-op followed by a jump around
+the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a
+'jump_table' which has the following format:
+

Documentation/jump-label.txt line 76

Quoted 10 lines
Documentation/jump-label.txt new file +10 -0
+[instruction address] [jump target] [tracepoint key]
+
+Thus, to enable a tracepoint, we simply patch the 'instruction address' with
+a jump to the 'jump target'.
+
+The call to enable a jump label is: enable_jump_label(key); to disable:
+disable_jump_label(key);
+
+
+3) architecture interface

cap.

Documentation/jump-label.txt line 79

Quoted 3 lines
Documentation/jump-label.txt new file +3 -0
+
+
+There are a few functions and macros which arches must implement in order to

arches -> architectures

Quoted 51 lines
Documentation/jump-label.txt new file +51 -0
+take advantage of this optimization. As previously mentioned, if there is no
+architecture support we simply fall back to a traditional, load, test, and
+jump sequence.
+
+* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support
+
+* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h
+
+* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h
+
+* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+	see: arch/x86/kernel/jump_label.c
+
+* void arch_jump_label_text_poke_early(jump_label_t addr)
+	see: arch/x86/kernel/jump_label.c
+
+* finally add a definition for "struct jump_entry".
+	see: arch/x86/include/asm/jump_label.h
+
+
+4) Jump label analysis (x86)
+
+
+I've tested the performance of using 'get_cycles()' calls around the
+tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages):
+
+		idle		after tbench run
+		----		----------------
+old code	 32		  88
+new code	  2		   4
+
+
+The performance improvement can be reproduced reliably on both Intel and AMD
+hardware.
+
+In terms of code analysis the current code for the disabled case is a 'cmpl'
+followed by a 'je' around the tracepoint code. so:
+
+cmpl - 83 3d 0e 77 87 00 00 - 7 bytes
+je   - 74 3e                - 2 bytes
+
+total of 9 instruction bytes.
+
+The new code is a 'nopl' followed by a 'jmp'. Thus:
+
+nopl - 0f 1f 44 00 00 - 5 bytes
+jmp  - eb 3e          - 2 bytes
+
+total of 7 instruction bytes.
+
+So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.

Documentation/jump-label.txt line 140

Quoted 10 lines
Documentation/jump-label.txt new file +10 -0
+
+The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is
+set, gcc does not always out of line the not taken label path in the same way
+that the "if unlikely()" paths are made out of line. Thus, with
+CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be
+solved in subsequent gcc versions, that allow us to move labels out of line,
+while still optimizing for size.
+
+
+5) Acknowledgments

Acknowledgements (seems to be the usual way to spell it in Documentation/).

Documentation/jump-label.txt line 147

Quoted 7 lines
Documentation/jump-label.txt new file +7 -0
+
+
+Thanks to Roland McGrath and Richard Henderson for helping come up with the
+initial 'asm goto' and jump label design.
+
+Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this
+issue, and outlining the requirements of a solution. Mathieu also implemened a

implemened -> implemented

Thanks!

Mathieu

Quoted 4 lines
Documentation/jump-label.txt new file +1 -0
+solution in the form of the "Immediate Values" work.
Signature · 1 line
1.7.1
Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
maintainer
#60 lore raw

Re: [PATCH 10/10] jump label v11: add docs

Quoted 4 lines · Mathieu Desnoyers
Quoted 3 lines · * Jason Baron ([email protected]) wrote:

Add jump label docs as: Documentation/jump-label.txt

Jason,

Could you post this patch again with any updates you want to take from Mathieu. No need to post the entire series, I'm pulling in the rest. Just this patch.

Thanks,

-- Steve

Quoted 194 lines
Quoted 11 lines
---
 Documentation/jump-label.txt |  148 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 148 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/jump-label.txt
Documentation/jump-label.txt new file +0 -0
new file mode 100644
index 0000000..2e5cff6

Documentation/jump-label.txt line 7

Quoted 7 lines
Documentation/jump-label.txt new file +7 -0
@@ -0,0 +1,148 @@
+			Jump Label
+			----------
+
+By: Jason Baron <[email protected]>
+
+
+1) motivation

You might want to start your titles with capital letters (small nit).

Documentation/jump-label.txt line 12

Quoted 5 lines
Documentation/jump-label.txt new file +5 -0
+
+
+Currently, tracepoints are implemented using a conditional. The conditional
+check requires checking a global variable for each tracepoint. Although,
+the overhead of this check is small, it increases under memory pressure. As we

I wonder if it would be worthwhile to spell out the difference between "memory pressure" in terms of memory bandwidth (which is what we care about here) and in terms of "memory space available". I just don't want people to get mixed up between the two when they read this text. What we care a lot about here is just memory bandwidth, which is impacted by cache pressure.

Documentation/jump-label.txt line 21

Quoted 9 lines
Documentation/jump-label.txt new file +9 -0
+increase the number of tracepoints in the kernel this may become more of an
+issue. In addition, tracepoints are often dormant (disabled), and provide no
+direct kernel functionality. Thus, it is highly desirable to reduce their
+impact as much as possible. Although tracepoints are the original motivation
+for this work, other kernel code paths should be able to make use of the jump
+label optimization.
+
+
+2) jump label description/usage

Start with capital letter here too.

Quoted 45 lines
Documentation/jump-label.txt new file +45 -0
+
+
+gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label.
+http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html
+
+Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as
+follows (x86):
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label)					\
+	do {                                                    \
+		asm goto("1:"                                   \
+			JUMP_LABEL_INITIAL_NOP			\
+                        ".pushsection __jump_table,  \"a\" \n\t"\
+                        _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+                        ".popsection \n\t"                      \
+                        : :  "i" (key) :  : label);		\
+        } while (0)
+
+
+For architectures that have not yet introduced jump label support its simply:
+
+#define JUMP_LABEL(key, label)			\
+	if (unlikely(*key))			\
+		goto label;
+
+which then can be used as:
+
+	....
+        JUMP_LABEL(trace_name, trace_label, jump_enabled);
+        printk("not doing tracing\n");
+	return;
+trace_label:
+        printk("doing tracing: %d\n", file);
+	....
+
+The 'key' argument is thus a pointer to a conditional argument that can be used
+if the optimization is not enabled. Otherwise, this address serves as a unique
+key to identify the particular instance of the jump label.
+
+Thus, when tracing is disabled, we simply have a no-op followed by a jump around
+the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a
+'jump_table' which has the following format:
+

Documentation/jump-label.txt line 76

Quoted 10 lines
Documentation/jump-label.txt new file +10 -0
+[instruction address] [jump target] [tracepoint key]
+
+Thus, to enable a tracepoint, we simply patch the 'instruction address' with
+a jump to the 'jump target'.
+
+The call to enable a jump label is: enable_jump_label(key); to disable:
+disable_jump_label(key);
+
+
+3) architecture interface

cap.

Documentation/jump-label.txt line 79

Quoted 3 lines
Documentation/jump-label.txt new file +3 -0
+
+
+There are a few functions and macros which arches must implement in order to

arches -> architectures

Quoted 51 lines
Documentation/jump-label.txt new file +51 -0
+take advantage of this optimization. As previously mentioned, if there is no
+architecture support we simply fall back to a traditional, load, test, and
+jump sequence.
+
+* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support
+
+* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h
+
+* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h
+
+* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+	see: arch/x86/kernel/jump_label.c
+
+* void arch_jump_label_text_poke_early(jump_label_t addr)
+	see: arch/x86/kernel/jump_label.c
+
+* finally add a definition for "struct jump_entry".
+	see: arch/x86/include/asm/jump_label.h
+
+
+4) Jump label analysis (x86)
+
+
+I've tested the performance of using 'get_cycles()' calls around the
+tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages):
+
+		idle		after tbench run
+		----		----------------
+old code	 32		  88
+new code	  2		   4
+
+
+The performance improvement can be reproduced reliably on both Intel and AMD
+hardware.
+
+In terms of code analysis the current code for the disabled case is a 'cmpl'
+followed by a 'je' around the tracepoint code. so:
+
+cmpl - 83 3d 0e 77 87 00 00 - 7 bytes
+je   - 74 3e                - 2 bytes
+
+total of 9 instruction bytes.
+
+The new code is a 'nopl' followed by a 'jmp'. Thus:
+
+nopl - 0f 1f 44 00 00 - 5 bytes
+jmp  - eb 3e          - 2 bytes
+
+total of 7 instruction bytes.
+
+So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.

Documentation/jump-label.txt line 140

Quoted 10 lines
Documentation/jump-label.txt new file +10 -0
+
+The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is
+set, gcc does not always out of line the not taken label path in the same way
+that the "if unlikely()" paths are made out of line. Thus, with
+CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be
+solved in subsequent gcc versions, that allow us to move labels out of line,
+while still optimizing for size.
+
+
+5) Acknowledgments

Acknowledgements (seems to be the usual way to spell it in Documentation/).

Documentation/jump-label.txt line 147

Quoted 7 lines
Documentation/jump-label.txt new file +7 -0
+
+
+Thanks to Roland McGrath and Richard Henderson for helping come up with the
+initial 'asm goto' and jump label design.
+
+Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this
+issue, and outlining the requirements of a solution. Mathieu also implemened a

implemened -> implemented

Thanks!

Mathieu

Quoted 4 lines
Documentation/jump-label.txt new file +1 -0
+solution in the form of the "Immediate Values" work.
Signature · 1 line
1.7.1
#61 lore raw

Re: [PATCH 10/10] jump label v11: add docs

Quoted 8 lines · Steven Rostedt
Quoted 4 lines · Mathieu Desnoyers
Quoted 3 lines · * Jason Baron ([email protected]) wrote:

Add jump label docs as: Documentation/jump-label.txt

Jason,

Quoted 8 lines

Could you post this patch again with any updates you want to take from Mathieu. No need to post the entire series, I'm pulling in the rest. Just this patch.

Thanks,

-- Steve

Hi Steve,

here's a re-post of the documentation patch. I've incorporated Mathieu's as well as Matt's feedback. Note that I've dropped the '!CC_OPTIMIZE_FOR_SIZE' optimize for size paragraph.

thanks,

-Jason

Add jump label docs as: Documentation/jump-label.txt

---
 Documentation/jump-label.txt |  142 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 142 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/jump-label.txt
Documentation/jump-label.txt new file +142 -0
new file mode 100644
index 0000000..d26df30
@@ -0,0 +1,142 @@
+			Jump Label
+			----------
+
+By: Jason Baron <[email protected]>
+
+
+1) Motivation
+
+
+Currently, tracepoints are implemented using a conditional. The conditional
+check requires checking a global variable for each tracepoint. Although
+the overhead of this check is small, it increases when the memory cache comes
+under pressure (memory cache lines for these global variables may be shared
+with other memory accesses). As we increase the number of tracepoints in the
+kernel this overhead may become more of an issue. In addition, tracepoints are
+often dormant (disabled) and provide no direct kernel functionality. Thus, it
+is highly desirable to reduce their impact as much as possible. Although
+tracepoints are the original motivation for this work, other kernel code paths
+should be able to make use of the jump label optimization.
+
+
+2) Jump label description/usage
+
+
+gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label.
+http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html
+
+Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as
+follows (x86):
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label)					\
+	do {                                                    \
+		asm goto("1:"                                   \
+			JUMP_LABEL_INITIAL_NOP			\
+                        ".pushsection __jump_table,  \"a\" \n\t"\
+                        _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+                        ".popsection \n\t"                      \
+                        : :  "i" (key) :  : label);		\
+        } while (0)
+
+
+For architectures that have not yet introduced jump label support it's simply:
+
+#define JUMP_LABEL(key, label)			\
+	if (unlikely(*key))			\
+		goto label;
+
+which then can be used as:
+
+	....
+        JUMP_LABEL(key, trace_label);
+        printk("not doing tracing\n");
+	return;
+trace_label:
+        printk("doing tracing: %d\n", file);
+	....
+
+The 'key' argument is thus a pointer to a conditional argument that can be used
+if the optimization is not enabled. Otherwise, this address serves as a unique
+key to identify the particular instance of the jump label.
+
+Thus, when tracing is disabled, we simply have a no-op followed by a jump around
+the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro produces a
+'jump_table', which has the following format:
+
+[instruction address] [jump target] [key]
+
+Thus, to enable a tracepoint, we simply patch the 'instruction address' with
+a jump to the 'jump target.'
+
+The calls to enable and disable a jump label are: enable_jump_label(key) and
+disable_jump_label(key).
+
+
+3) Architecture interface
+
+
+There are a few functions and macros that architectures must implement in order
+to take advantage of this optimization. As previously mentioned, if there is no
+architecture support, we simply fall back to a traditional, load, test, and
+jump sequence.
+
+* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support
+
+* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h
+
+* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h
+
+* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+	see: arch/x86/kernel/jump_label.c
+
+* void arch_jump_label_text_poke_early(jump_label_t addr)
+	see: arch/x86/kernel/jump_label.c
+
+* finally add a definition for "struct jump_entry".
+	see: arch/x86/include/asm/jump_label.h
+
+
+4) Jump label analysis (x86)
+
+
+I've tested the performance of using 'get_cycles()' calls around the
+tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages):
+
+		idle		after tbench run
+		----		----------------
+old code	 32		  88
+new code	  2		   4
+
+
+The performance improvement can be reproduced reliably on both Intel and AMD
+hardware.
+
+In terms of code analysis, the current code for the disabled case is a 'cmpl'
+followed by a 'je' around the tracepoint code. So:
+
+cmpl - 83 3d 0e 77 87 00 00 - 7 bytes
+je   - 74 3e                - 2 bytes
+
+total of 9 instruction bytes.
+
+The new code is a 'nopl' followed by a 'jmp'. Thus:
+
+nopl - 0f 1f 44 00 00 - 5 bytes
+jmp  - eb 3e          - 2 bytes
+
+total of 7 instruction bytes.
+
+So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.
+
+
+5) Acknowledgements
+
+
+Thanks to Roland McGrath and Richard Henderson for helping come up with the
+initial 'asm goto' and jump label design.
+
+Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this
+issue, and outlining the requirements of a solution. Mathieu also implemented a
+solution in the form of the "Immediate Values" work.
Signature · 1 line
1.7.1
#62 lore raw

Re: [PATCH 10/10] jump label v11: add docs

In reply to Jason Baron

Documentation/jump-label.txt line 11

Quoted 4 lines · Jason Baron

Add jump label docs as: Documentation/jump-label.txt

Documentation/jump-label.txt new file +2 -0
+Currently, tracepoints are implemented using a conditional. The conditional
+check requires checking a global variable for each tracepoint. Although,

No comma after although here.

Documentation/jump-label.txt line 13

Quoted 2 lines
Documentation/jump-label.txt new file +2 -0
+the overhead of this check is small, it increases under memory pressure. As we
+increase the number of tracepoints in the kernel this may become more of an

Comma after kernel is needed to separate the preposition, and I think the use of "this" in the sentence is unclear.

Quoted 1 line

+issue. In addition, tracepoints are often dormant (disabled), and provide no

Also, no comma before "and" because the second part is a subordinate clause.

Documentation/jump-label.txt line 43

Quoted 7 lines
Documentation/jump-label.txt new file +7 -0
+direct kernel functionality. Thus, it is highly desirable to reduce their
+impact as much as possible. Although tracepoints are the original motivation
+for this work, other kernel code paths should be able to make use of the jump
+label optimization.
+
+
+For architectures that have not yet introduced jump label support its simply:

..."support, it's" ...

Documentation/jump-label.txt line 65

Quoted 4 lines
Documentation/jump-label.txt new file +4 -0
+
+Thus, when tracing is disabled, we simply have a no-op followed by a jump around
+the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a
+'jump_table' which has the following format:

No comma after macro, but a comma is needed after jump_table because what follows is a nonrestrictive clause.

Documentation/jump-label.txt line 70

Quoted 4 lines

+[instruction address] [jump target] [tracepoint key]

Documentation/jump-label.txt new file +3 -0
+
+Thus, to enable a tracepoint, we simply patch the 'instruction address' with
+a jump to the 'jump target'.

Punctuation is suppose to go inside quotes (I know it's ugly and illogical).

Documentation/jump-label.txt line 73

Quoted 3 lines
Documentation/jump-label.txt new file +3 -0
+
+The call to enable a jump label is: enable_jump_label(key); to disable:
+disable_jump_label(key);

Hmm, maybe a better structure would be:
"The calls to enable and disable a jump label are: enable_jump_label(key) and disable_jump_label(key)."

Quoted 1 line

+There are a few functions and macros which arches must implement in order to

"That" should be used here because it is restrictive.

Documentation/jump-label.txt line 81

Quoted 2 lines
Documentation/jump-label.txt new file +2 -0
+take advantage of this optimization. As previously mentioned, if there is no
+architecture support we simply fall back to a traditional, load, test, and

Comma after support.

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+In terms of code analysis the current code for the disabled case is a 'cmpl'

Comma after analysis; it is a preposition.

Quoted 1 line

+followed by a 'je' around the tracepoint code. so:

Capitalize S in "so"

Documentation/jump-label.txt line 136

Quoted 7 lines
Documentation/jump-label.txt new file +7 -0
+
+
+The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is
+set, gcc does not always out of line the not taken label path in the same way
+that the "if unlikely()" paths are made out of line. Thus, with
+CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be
+solved in subsequent gcc versions, that allow us to move labels out of line,

"which" should be used here instead of that.

Otherwise, from a writing standpoint, it looks good.

-mfm

#63 lore raw

Re: [PATCH 10/10] jump label v11: add docs

In reply to Jason Baron
Quoted 56 lines · Jason Baron

Add jump label docs as: Documentation/jump-label.txt

---
 Documentation/jump-label.txt |  148 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 148 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/jump-label.txt
Documentation/jump-label.txt new file +42 -0
new file mode 100644
index 0000000..2e5cff6
@@ -0,0 +1,148 @@
+			Jump Label
+			----------
+
+By: Jason Baron <[email protected]>
+
+
+1) motivation
+
+
+Currently, tracepoints are implemented using a conditional. The conditional
+check requires checking a global variable for each tracepoint. Although,
+the overhead of this check is small, it increases under memory pressure. As we
+increase the number of tracepoints in the kernel this may become more of an
+issue. In addition, tracepoints are often dormant (disabled), and provide no
+direct kernel functionality. Thus, it is highly desirable to reduce their
+impact as much as possible. Although tracepoints are the original motivation
+for this work, other kernel code paths should be able to make use of the jump
+label optimization.
+
+
+2) jump label description/usage
+
+
+gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label.
+http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html
+
+Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as
+follows (x86):
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label)					\
+	do {                                                    \
+		asm goto("1:"                                   \
+			JUMP_LABEL_INITIAL_NOP			\
+                        ".pushsection __jump_table,  \"a\" \n\t"\
+                        _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+                        ".popsection \n\t"                      \
+                        : :  "i" (key) :  : label);		\
+        } while (0)
+
+

Documentation/jump-label.txt line 52

Quoted 10 lines
Documentation/jump-label.txt new file +10 -0
+For architectures that have not yet introduced jump label support its simply:
+
+#define JUMP_LABEL(key, label)			\
+	if (unlikely(*key))			\
+		goto label;
+
+which then can be used as:
+
+	....
+        JUMP_LABEL(trace_name, trace_label, jump_enabled);

No. That is three arguments. The macro only has two.

Quoted 64 lines
Documentation/jump-label.txt new file +64 -0
+        printk("not doing tracing\n");
+	return;
+trace_label:
+        printk("doing tracing: %d\n", file);
+	....
+
+The 'key' argument is thus a pointer to a conditional argument that can be used
+if the optimization is not enabled. Otherwise, this address serves as a unique
+key to identify the particular instance of the jump label.
+
+Thus, when tracing is disabled, we simply have a no-op followed by a jump around
+the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a
+'jump_table' which has the following format:
+
+[instruction address] [jump target] [tracepoint key]
+
+Thus, to enable a tracepoint, we simply patch the 'instruction address' with
+a jump to the 'jump target'.
+
+The call to enable a jump label is: enable_jump_label(key); to disable:
+disable_jump_label(key);
+
+
+3) architecture interface
+
+
+There are a few functions and macros which arches must implement in order to
+take advantage of this optimization. As previously mentioned, if there is no
+architecture support we simply fall back to a traditional, load, test, and
+jump sequence.
+
+* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support
+
+* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h
+
+* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h
+
+* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+	see: arch/x86/kernel/jump_label.c
+
+* void arch_jump_label_text_poke_early(jump_label_t addr)
+	see: arch/x86/kernel/jump_label.c
+
+* finally add a definition for "struct jump_entry".
+	see: arch/x86/include/asm/jump_label.h
+
+
+4) Jump label analysis (x86)
+
+
+I've tested the performance of using 'get_cycles()' calls around the
+tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages):
+
+		idle		after tbench run
+		----		----------------
+old code	 32		  88
+new code	  2		   4
+
+
+The performance improvement can be reproduced reliably on both Intel and AMD
+hardware.
+
+In terms of code analysis the current code for the disabled case is a 'cmpl'
+followed by a 'je' around the tracepoint code. so:

Documentation/jump-label.txt line 126

Quoted 10 lines
Documentation/jump-label.txt new file +10 -0
+
+cmpl - 83 3d 0e 77 87 00 00 - 7 bytes
+je   - 74 3e                - 2 bytes
+
+total of 9 instruction bytes.
+
+The new code is a 'nopl' followed by a 'jmp'. Thus:
+
+nopl - 0f 1f 44 00 00 - 5 bytes
+jmp  - eb 3e          - 2 bytes

Uh, not e9?

Quoted 30 lines
Documentation/jump-label.txt new file +22 -0
+
+total of 7 instruction bytes.
+
+So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.
+
+The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is
+set, gcc does not always out of line the not taken label path in the same way
+that the "if unlikely()" paths are made out of line. Thus, with
+CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be
+solved in subsequent gcc versions, that allow us to move labels out of line,
+while still optimizing for size.
+
+
+5) Acknowledgments
+
+
+Thanks to Roland McGrath and Richard Henderson for helping come up with the
+initial 'asm goto' and jump label design.
+
+Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this
+issue, and outlining the requirements of a solution. Mathieu also implemened a
+solution in the form of the "Immediate Values" work.
Signature · 7 lines
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/