jump label v11
Subsystems (12)
cover jump label v11
Hi,
updates in -v11:
-consolidated core jump label changes into a single patch -updated sparc64 patch from David Miller -various cleanups
Thanks to Steve Rostedt for helping me clean up the ftrace changes found in patch 1/10.
thanks,
-Jason
David S. Miller (1):
jump label: add sparc64 support
Jason Baron (9):
jump label: x86 - make dynamic no-op selection available outside of ftrace jump label: make text_poke_early() globally visisble
jump label: base patch
jump label: initialize workqueue tracepoints *before* they are registered jump label: jump_label_text_reserved() to reserve our jump points jump label: tracepoint support
jump label: convert dynamic debug to use jump labels
jump label: x86 support
jump label: add docs
Documentation/jump-label.txt | 148 ++++++++++++ Makefile | 5 + arch/Kconfig | 3 + arch/sparc/Kconfig | 1 + arch/sparc/include/asm/jump_label.h | 32 +++ arch/sparc/kernel/Makefile | 2 + arch/sparc/kernel/jump_label.c | 46 ++++ arch/sparc/kernel/module.c | 6 + arch/x86/Kconfig | 1 + arch/x86/include/asm/alternative.h | 11 + arch/x86/include/asm/jump_label.h | 47 ++++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/alternative.c | 68 ++++++- arch/x86/kernel/ftrace.c | 63 +----- arch/x86/kernel/jump_label.c | 49 ++++ arch/x86/kernel/kprobes.c | 3 +- arch/x86/kernel/module.c | 3 + arch/x86/kernel/setup.c | 6 + include/asm-generic/vmlinux.lds.h | 10 + include/linux/dynamic_debug.h | 39 ++-- include/linux/jump_label.h | 64 ++++++ include/linux/module.h | 5 +- include/linux/tracepoint.h | 5 +- kernel/Makefile | 2 +- kernel/jump_label.c | 421 +++++++++++++++++++++++++++++++++++ kernel/kprobes.c | 4 +- kernel/module.c | 6 + kernel/trace/trace_workqueue.c | 10 +- kernel/tracepoint.c | 14 +- lib/dynamic_debug.c | 42 +---- scripts/Makefile.lib | 11 +- scripts/basic/Makefile | 2 +- scripts/basic/hash.c | 64 ------ scripts/gcc-goto.sh | 5 + 34 files changed, 991 insertions(+), 209 deletions(-) create mode 100644 Documentation/jump-label.txt create mode 100644 arch/sparc/include/asm/jump_label.h create mode 100644 arch/sparc/kernel/jump_label.c create mode 100644 arch/x86/include/asm/jump_label.h create mode 100644 arch/x86/kernel/jump_label.c create mode 100644 include/linux/jump_label.h create mode 100644 kernel/jump_label.c delete mode 100644 scripts/basic/hash.c create mode 100644 scripts/gcc-goto.sh
PATCH 1/10 jump label v11: make dynamic no-op selection available outside of ftrace
Move Steve's code for finding the best 5-byte no-op from ftrace.c to alternative.c. The idea is that other consumers (in this case jump label) want to make use of that code.
--- arch/x86/include/asm/alternative.h | 8 ++++ arch/x86/kernel/alternative.c | 64 ++++++++++++++++++++++++++++++++++++ arch/x86/kernel/ftrace.c | 63 +---------------------------------- arch/x86/kernel/setup.c | 6 +++ 4 files changed, 79 insertions(+), 62 deletions(-)
arch/x86/include/asm/alternative.h +8 -0
@@ -180,4 +180,12 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,extern void *text_poke (void *addr,const void *opcode,size_t len);extern void *text_poke_smp (void *addr,const void *opcode,size_t len); +#if defined(CONFIG_DYNAMIC_FTRACE ) +#define IDEAL_NOP_SIZE_5 5 +extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5 ]; +extern void arch_init_ideal_nop5 (void ); +#else +static inline void arch_init_ideal_nop5 (void ) {} +#endif +#endif /* _ASM_X86_ALTERNATIVE_H */
arch/x86/kernel/alternative.c +64 -0
67 lines not on this page · raw message
arch/x86/kernel/ftrace.c +1 -62
77 lines not on this page · raw message
arch/x86/kernel/setup.c +6 -0
@@ -112,6 +112,7 @@#include <asm/numa_64.h> #endif #include <asm/mce.h> +#include <asm/alternative.h> /* * end_pfn only includes RAM, while max_pfn_mapped includes all e820 entries.
@@ -689,6 +690,7 @@ void __init setup_arch(char **cmdline_p) {int acpi =0 ;int k8 =0 ; +unsigned long flags;#ifdef CONFIG_X86_32 memcpy (&boot_cpu_data, &new_cpu_data,sizeof (new_cpu_data));
@@ -1036,6 +1038,10 @@ void __init setup_arch(char **cmdline_p) x86_init.oem.banner ();mcheck_init (); + +local_irq_save (flags); +arch_init_ideal_nop5 (); +local_irq_restore (flags); }#ifdef CONFIG_X86_32
Signature · 1 line
1.7.1
Re: [PATCH 01/10] jump label v11: make dynamic no-op selection available outside of ftrace
Quoted 5 lines · Jason Baron
Move Steve's code for finding the best 5-byte no-op from
ftrace.ctoalternative.c. The idea is that other consumers (in this case jump label) want to make use of that code.Signed-off-by Jason Baron <[email protected]>
-- Steve
Quoted 210 lines
--- arch/x86/include/asm/alternative.h | 8 ++++ arch/x86/kernel/alternative.c | 64 ++++++++++++++++++++++++++++++++++++ arch/x86/kernel/ftrace.c | 63 +---------------------------------- arch/x86/kernel/setup.c | 6 +++ 4 files changed, 79 insertions(+), 62 deletions(-)arch/x86/include/asm/alternative.h +8
-0@@ -180,4 +180,12 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,extern void *text_poke (void *addr,const void *opcode,size_t len);extern void *text_poke_smp (void *addr,const void *opcode,size_t len); +#if defined(CONFIG_DYNAMIC_FTRACE ) +#define IDEAL_NOP_SIZE_5 5 +extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5 ]; +extern void arch_init_ideal_nop5 (void ); +#else +static inline void arch_init_ideal_nop5 (void ) {} +#endif +#endif /* _ASM_X86_ALTERNATIVE_H */ arch/x86/kernel/alternative.c +64
-0@@ -641,3 +641,67 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)return addr; } +#if defined(CONFIG_DYNAMIC_FTRACE ) + +unsigned char ideal_nop5[IDEAL_NOP_SIZE_5 ]; + +void __init arch_init_ideal_nop5 (void ) +{ +extern const unsigned char ftrace_test_p6nop[]; +extern const unsigned char ftrace_test_nop5[]; +extern const unsigned char ftrace_test_jmp[]; +int faulted =0 ; + +/* +* There is no good nop for all x86 archs. +* We will default to using the P6_NOP5, but first we +* will test to make sure that the nop will actually +* work on this CPU. If it faults, we will then +* go to a lesser efficient 5 byte nop. If that fails +* we then just use a jmp as our nop. This isn't the most +* efficient nop, but we can not use a multi part nop +* since we would then risk being preempted in the middle +* of that nop, and if we enabled tracing then, it might +* cause a system crash. +* +* TODO: check the cpuid to determine the best nop. +*/ + asmvolatile ( +"ftrace_test_jmp:" +"jmp ftrace_test_p6nop\n" +"nop\n" +"nop\n" +"nop\n" /* 2 byte jmp + 3 bytes */ +"ftrace_test_p6nop:" +P6_NOP5 +"jmp 1f\n" +"ftrace_test_nop5:" +".byte 0x66,0x66,0x66,0x66,0x90\n" +"1:" +".section .fixup, \"ax\"\n" +"2: movl $1, %0\n" +" jmp ftrace_test_nop5\n" +"3: movl $2, %0\n" +" jmp 1b\n" +".previous\n" + _ASM_EXTABLE(ftrace_test_p6nop, 2b) + _ASM_EXTABLE(ftrace_test_nop5, 3b) + : "=r"(faulted) : "0" (faulted)); + + switch (faulted) { +case 0 : +pr_info ("converting mcount calls to 0f 1f 44 00 00\n" ); +memcpy (ideal_nop5, ftrace_test_p6nop,IDEAL_NOP_SIZE_5 ); +break ; +case 1 : +pr_info ("converting mcount calls to 66 66 66 66 90\n" ); +memcpy (ideal_nop5, ftrace_test_nop5,IDEAL_NOP_SIZE_5 ); +break ; +case 2 : +pr_info ("converting mcount calls to jmp . + 5\n" ); +memcpy (ideal_nop5, ftrace_test_jmp,IDEAL_NOP_SIZE_5 ); +break ; + } + +} +#endif arch/x86/kernel/ftrace.c +1
-62@@ -257,14 +257,9 @@ do_ftrace_mod_code(unsigned long ip, void *new_code)return mod_code_status; } - - - -static unsigned char ftrace_nop[MCOUNT_INSN_SIZE ]; -static unsigned char *ftrace_nop_replace (void ) { -return ftrace_nop; +return ideal_nop5; } static int@@ -338,62 +333,6 @@ int ftrace_update_ftrace_func(ftrace_func_t func)int __init ftrace_dyn_arch_init (void *data) { -extern const unsigned char ftrace_test_p6nop[]; -extern const unsigned char ftrace_test_nop5[]; -extern const unsigned char ftrace_test_jmp[]; -int faulted =0 ; - -/* -* There is no good nop for all x86 archs. -* We will default to using the P6_NOP5, but first we -* will test to make sure that the nop will actually -* work on this CPU. If it faults, we will then -* go to a lesser efficient 5 byte nop. If that fails -* we then just use a jmp as our nop. This isn't the most -* efficient nop, but we can not use a multi part nop -* since we would then risk being preempted in the middle -* of that nop, and if we enabled tracing then, it might -* cause a system crash. -* -* TODO: check the cpuid to determine the best nop. -*/ - asmvolatile ( -"ftrace_test_jmp:" -"jmp ftrace_test_p6nop\n" -"nop\n" -"nop\n" -"nop\n" /* 2 byte jmp + 3 bytes */ -"ftrace_test_p6nop:" -P6_NOP5 -"jmp 1f\n" -"ftrace_test_nop5:" -".byte 0x66,0x66,0x66,0x66,0x90\n" -"1:" -".section .fixup, \"ax\"\n" -"2: movl $1, %0\n" -" jmp ftrace_test_nop5\n" -"3: movl $2, %0\n" -" jmp 1b\n" -".previous\n" - _ASM_EXTABLE(ftrace_test_p6nop, 2b) - _ASM_EXTABLE(ftrace_test_nop5, 3b) - : "=r"(faulted) : "0" (faulted)); - - switch (faulted) { -case 0 : -pr_info ("converting mcount calls to 0f 1f 44 00 00\n" ); -memcpy (ftrace_nop, ftrace_test_p6nop,MCOUNT_INSN_SIZE ); -break ; -case 1 : -pr_info ("converting mcount calls to 66 66 66 66 90\n" ); -memcpy (ftrace_nop, ftrace_test_nop5,MCOUNT_INSN_SIZE ); -break ; -case 2 : -pr_info ("converting mcount calls to jmp . + 5\n" ); -memcpy (ftrace_nop, ftrace_test_jmp,MCOUNT_INSN_SIZE ); -break ; - } -/* The return code is retured via data */ *(unsigned long *)data =0 ;arch/x86/kernel/setup.c +6
-0@@ -112,6 +112,7 @@#include <asm/numa_64.h> #endif #include <asm/mce.h> +#include <asm/alternative.h> /* * end_pfn only includes RAM, while max_pfn_mapped includes all e820 entries. @@ -689,6 +690,7 @@ void __init setup_arch(char **cmdline_p) {int acpi =0 ;int k8 =0 ; +unsigned long flags;#ifdef CONFIG_X86_32 memcpy (&boot_cpu_data, &new_cpu_data,sizeof (new_cpu_data));@@ -1036,6 +1038,10 @@ void __init setup_arch(char **cmdline_p) x86_init.oem.banner ();mcheck_init (); + +local_irq_save (flags); +arch_init_ideal_nop5 (); +local_irq_restore (flags); }#ifdef CONFIG_X86_32
[tip:perf/core] jump label: Make dynamic no-op selection available outside of ftrace
In reply to Jason BaronCommit-ID: f49aa448561fe9215f43405cac6f31eb86317792 Gitweb: http://git.kernel.org/tip/f49aa448561fe9215f43405cac6f31eb86317792 Author: Jason Baron <[email protected]> AuthorDate: Fri, 17 Sep 2010 11:08:51 -0400 Committer: Steven Rostedt <[email protected]> CommitDate: Mon, 20 Sep 2010 18:19:39 -0400
jump label: Make dynamic no-op selection available outside of ftrace
Move Steve's code for finding the best 5-byte no-op from ftrace.c to alternative.c. The idea is that other consumers (in this case jump label) want to make use of that code.
LKML-Reference: <96259ae74172dcac99c0020c249743c523a92e18.1284733808.git.jbaron@redhat.com>
--- arch/x86/include/asm/alternative.h | 8 ++++ arch/x86/kernel/alternative.c | 64 ++++++++++++++++++++++++++++++++++++ arch/x86/kernel/ftrace.c | 63 +---------------------------------- arch/x86/kernel/setup.c | 6 +++ 4 files changed, 79 insertions(+), 62 deletions(-)
arch/x86/include/asm/alternative.h +8 -0
@@ -180,4 +180,12 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,extern void *text_poke (void *addr,const void *opcode,size_t len);extern void *text_poke_smp (void *addr,const void *opcode,size_t len); +#if defined(CONFIG_DYNAMIC_FTRACE ) +#define IDEAL_NOP_SIZE_5 5 +extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5 ]; +extern void arch_init_ideal_nop5 (void ); +#else +static inline void arch_init_ideal_nop5 (void ) {} +#endif +#endif /* _ASM_X86_ALTERNATIVE_H */
arch/x86/kernel/alternative.c +64 -0
67 lines not on this page · raw message
arch/x86/kernel/ftrace.c +1 -62
77 lines not on this page · raw message
arch/x86/kernel/setup.c +6 -0
@@ -112,6 +112,7 @@#include <asm/numa_64.h> #endif #include <asm/mce.h> +#include <asm/alternative.h> /* * end_pfn only includes RAM, while max_pfn_mapped includes all e820 entries.
@@ -726,6 +727,7 @@ void __init setup_arch(char **cmdline_p) {int acpi =0 ;int k8 =0 ; +unsigned long flags;#ifdef CONFIG_X86_32 memcpy (&boot_cpu_data, &new_cpu_data,sizeof (new_cpu_data));
@@ -1071,6 +1073,10 @@ void __init setup_arch(char **cmdline_p) x86_init.oem.banner ();mcheck_init (); + +local_irq_save (flags); +arch_init_ideal_nop5 (); +local_irq_restore (flags); }#ifdef CONFIG_X86_32
PATCH 2/10 jump label v11: make text_poke_early() globally visisble
In reply to Jason BaronMake text_poke_early available outside of alternative.c. The jump label patchset wants to make use of it in order to set up the optimal no-op sequences at run-time.
--- arch/x86/include/asm/alternative.h | 2 ++ arch/x86/kernel/alternative.c | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/include/asm/alternative.h +2 -0
@@ -160,6 +160,8 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,#define __parainstructions_end NULL#endif +extern void *text_poke_early (void *addr,const void *opcode,size_t len); +/* * Clear and restore the kernel write-protection flag on the local CPU. * Allows the kernel to edit read-only pages.
arch/x86/kernel/alternative.c +2 -2
@@ -195,7 +195,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len)extern struct alt_instr __alt_instructions[], __alt_instructions_end[];extern s32 __smp_locks[], __smp_locks_end[]; -static void *text_poke_early (void *addr,const void *opcode,size_t len); +void *text_poke_early (void *addr,const void *opcode,size_t len);/* Replace instructions with better alternatives for this CPU type. This runs before SMP is initialized to avoid SMP problems with
@@ -522,7 +522,7 @@ void __init alternative_instructions(void)* instructions. And on the local CPU you need to be protected again NMI or MCE * handlers seeing an inconsistent instruction while you patch. */ -static void *__init_or_module text_poke_early(void *addr, const void *opcode, +void *__init_or_module text_poke_early(void *addr, const void *opcode,size_t len) {unsigned long flags;
Signature · 1 line
1.7.1
[tip:perf/core] jump label: Make text_poke_early() globally visible
Commit-ID: fa6f2cc77081792e4edca9168420a3422299ef15 Gitweb: http://git.kernel.org/tip/fa6f2cc77081792e4edca9168420a3422299ef15 Author: Jason Baron <[email protected]> AuthorDate: Fri, 17 Sep 2010 11:08:56 -0400 Committer: Steven Rostedt <[email protected]> CommitDate: Mon, 20 Sep 2010 18:19:51 -0400
jump label: Make text_poke_early() globally visible
Make text_poke_early available outside of alternative.c. The jump label patchset wants to make use of it in order to set up the optimal no-op sequences at run-time.
LKML-Reference: <04cfddf2ba77bcabfc3e524f1849d871d6a1cf9d.1284733808.git.jbaron@redhat.com>
--- arch/x86/include/asm/alternative.h | 2 ++ arch/x86/kernel/alternative.c | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/include/asm/alternative.h +2 -0
@@ -160,6 +160,8 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,#define __parainstructions_end NULL#endif +extern void *text_poke_early (void *addr,const void *opcode,size_t len); +/* * Clear and restore the kernel write-protection flag on the local CPU. * Allows the kernel to edit read-only pages.
arch/x86/kernel/alternative.c +2 -2
@@ -195,7 +195,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len)extern struct alt_instr __alt_instructions[], __alt_instructions_end[];extern s32 __smp_locks[], __smp_locks_end[]; -static void *text_poke_early (void *addr,const void *opcode,size_t len); +void *text_poke_early (void *addr,const void *opcode,size_t len);/* Replace instructions with better alternatives for this CPU type. This runs before SMP is initialized to avoid SMP problems with
@@ -522,7 +522,7 @@ void __init alternative_instructions(void)* instructions. And on the local CPU you need to be protected again NMI or MCE * handlers seeing an inconsistent instruction while you patch. */ -static void *__init_or_module text_poke_early(void *addr, const void *opcode, +void *__init_or_module text_poke_early(void *addr, const void *opcode,size_t len) {unsigned long flags;
PATCH 3/10 jump label v11: base patch
In reply to Jason Baronbase patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.
--- Makefile | 5 + arch/Kconfig | 3 + arch/x86/include/asm/alternative.h | 3 +- arch/x86/kernel/alternative.c | 2 +- include/asm-generic/vmlinux.lds.h | 10 + include/linux/jump_label.h | 58 ++++++ include/linux/module.h | 5 +- kernel/Makefile | 2 +- kernel/jump_label.c | 338 ++++++++++++++++++++++++++++++++++++ kernel/kprobes.c | 1 + kernel/module.c | 6 + scripts/gcc-goto.sh | 5 + 12 files changed, 434 insertions(+), 4 deletions(-) create mode 100644 include/linux/jump_label.h create mode 100644 kernel/jump_label.c create mode 100644 scripts/gcc-goto.sh
Makefile +5 -0
@@ -591,6 +591,11 @@ KBUILD_CFLAGS += $(call cc-option,-fno-strict-overflow) # conserve stack if available KBUILD_CFLAGS += $(call cc-option,-fconserve-stack) +# check for 'asm goto' +ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC)), y) + KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO +endif + # Add user supplied CPPFLAGS, AFLAGS and CFLAGS as the last assignments # But warn user when we do so warn-assign = \
arch/Kconfig +3 -0
@@ -158,4 +158,7 @@ config HAVE_PERF_EVENTS_NMI subsystem. Also has support for calculating CPU cycle events to determine how many clock cycles in a given period. +config HAVE_ARCH_JUMP_LABEL + bool + source "kernel/gcov/Kconfig"
arch/x86/include/asm/alternative.h +2 -1
@@ -4,6 +4,7 @@#include <linux/types.h> #include <linux/stddef.h> #include <linux/stringify.h> +#include <linux/jump_label.h> #include <asm/asm.h> /*
@@ -182,7 +183,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);extern void *text_poke (void *addr,const void *opcode,size_t len);extern void *text_poke_smp (void *addr,const void *opcode,size_t len); -#if defined(CONFIG_DYNAMIC_FTRACE ) +#if defined(CONFIG_DYNAMIC_FTRACE ) || defined(HAVE_JUMP_LABEL )#define IDEAL_NOP_SIZE_5 5extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5 ];extern void arch_init_ideal_nop5 (void );
arch/x86/kernel/alternative.c +1 -1
@@ -641,7 +641,7 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)return addr; } -#if defined(CONFIG_DYNAMIC_FTRACE ) +#if defined(CONFIG_DYNAMIC_FTRACE ) || defined(HAVE_JUMP_LABEL )unsigned char ideal_nop5[IDEAL_NOP_SIZE_5 ];
include/asm-generic/vmlinux.lds.h +10 -0
@@ -220,6 +220,8 @@ \BUG_TABLE \ \ + JUMP_TABLE \ + \ /* PCI quirks */ \ .pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \ VMLINUX_SYMBOL(__start_pci_fixups_early) = .; \
@@ -563,6 +565,14 @@#define BUG_TABLE #endif +#define JUMP_TABLE \ + . = ALIGN(8); \ + __jump_table : AT(ADDR(__jump_table) - LOAD_OFFSET) { \ + VMLINUX_SYMBOL(__start___jump_table) = .; \ + *(__jump_table) \ + VMLINUX_SYMBOL(__stop___jump_table) = .; \ + } +#ifdef CONFIG_PM_TRACE #define TRACEDATA \ . = ALIGN(4); \
include/linux/jump_label.h new file +58 -0
58 lines not on this page · raw message
include/linux/module.h +4 -1
@@ -350,7 +350,10 @@ struct modulestruct tracepoint *tracepoints;unsigned int num_tracepoints;#endif - +#ifdef HAVE_JUMP_LABEL +struct jump_entry *jump_entries; +unsigned int num_jump_entries; +#endif #ifdef CONFIG_TRACING const char **trace_bprintk_fmt_start;unsigned int num_trace_bprintk_fmt;
kernel/Makefile +1 -1
@@ -10,7 +10,7 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o \ kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \ hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \ notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \ - async.o range.o + async.o range.o jump_label.o obj-y += groups.o ifdef CONFIG_FUNCTION_TRACER
kernel/jump_label.c new file +338 -0
338 lines not on this page · raw message
kernel/kprobes.c +1 -0
@@ -47,6 +47,7 @@#include <linux/memory.h> #include <linux/ftrace.h> #include <linux/cpu.h> +#include <linux/jump_label.h> #include <asm-generic/sections.h> #include <asm/cacheflush.h>
kernel/module.c +6 -0
@@ -55,6 +55,7 @@#include <linux/async.h> #include <linux/percpu.h> #include <linux/kmemleak.h> +#include <linux/jump_label.h> #define CREATE_TRACE_POINTS #include <trace/events/module.h>
@@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)sizeof (*mod->tracepoints), &mod->num_tracepoints);#endif +#ifdef HAVE_JUMP_LABEL + mod->jump_entries =section_objs (info,"__jump_table" , +sizeof (*mod->jump_entries), + &mod->num_jump_entries); +#endif #ifdef CONFIG_EVENT_TRACING mod->trace_events = section_objs(info, "_ftrace_events", sizeof(*mod->trace_events),
scripts/gcc-goto.sh new file +5 -0
@@ -0,0 +1,5 @@ +#!/bin/sh +# Test for gcc 'asm goto' suport +# Copyright (C) 2010, Jason Baron <[email protected]> + +echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
Signature · 1 line
1.7.1
Re: [PATCH 03/10] jump label v11: base patch
From: Jason Baron <[email protected]>
Date: Fri, 17 Sep 2010 11:09:00 -0400
Quoted 8 lines
base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.
Signed-off-by Jason Baron <[email protected]>
Steven Rostedt
maintainerRe: [PATCH 03/10] jump label v11: base patch
In reply to Jason BaronQuoted 120 lines · Jason Baron
base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.
Signed-off-by Jason Baron <[email protected]>---
kernel/jump_label.c new file +110
-0+static struct jump_label_entry *add_jump_label_entry (jump_label_t key,int nr_entries,struct jump_entry *table) +{ +struct hlist_head *head; +struct jump_label_entry *e; +u32 hash; + + e =get_jump_label_entry (key); +if (e) +return ERR_PTR (-EEXIST ); + + e =kmalloc (sizeof (struct jump_label_entry ),GFP_KERNEL ); +if (!e) +return ERR_PTR (-ENOMEM ); + + hash =jhash ((void *)&key,sizeof (jump_label_t),0 ); + head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE -1 )]; + e->key = key; + e->table = table; + e->nr_entries = nr_entries; +INIT_HLIST_HEAD (&(e->modules)); +hlist_add_head (&e->hlist, head); +return e; +} + +static int build_jump_label_hashtable (struct jump_entry *start,struct jump_entry *stop) +{ +struct jump_entry *iter, *iter_begin; +struct jump_label_entry *entry; +int count; + +sort_jump_label_entries (start, stop); + iter = start; +while (iter < stop) { + entry =get_jump_label_entry (iter->key); +if (!entry) { + iter_begin = iter; + count =0 ; +while ((iter < stop) && + (iter->key == iter_begin->key)) { + iter++; + count++; + } + entry =add_jump_label_entry (iter_begin->key, + count, iter_begin); +if (IS_ERR (entry)) +return PTR_ERR (entry); + }else { +WARN_ONCE (1 ,KERN_ERR "build_jump_hashtable: unexpected entry!\n" ); +return -1 ; + } + } +return 0 ; +} + +/*** +* jump_label_update - update jump label text +* @key - key value associated with a a jump label +* @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE +* +* Will enable/disable the jump for jump label @key, depending on the +* value of @type. +* +*/ + +void jump_label_update (unsigned long key,enum jump_label_type type) +{ +struct jump_entry *iter; +struct jump_label_entry *entry; +struct hlist_node *module_node; +struct jump_label_module_entry *e_module; +int count; + +mutex_lock (&jump_label_mutex); + entry =get_jump_label_entry ((jump_label_t )key); +if (entry) { + count = entry->nr_entries; + iter = entry->table; +while (count--) { +if (kernel_text_address (iter->code)) +arch_jump_label_transform (iter, type); + iter++; + } +/* eanble/disable jump labels in modules */ +hlist_for_each_entry (e_module, module_node, &(entry->modules), + hlist) { + count = e_module->nr_entries; + iter = e_module->table; +while (count--) { +if (kernel_text_address (iter->code)) +arch_jump_label_transform (iter, type); + iter++; + } + } + } +mutex_unlock (&jump_label_mutex); +} + +static __init int init_jump_label (void ) +{ +int ret; +struct jump_entry *iter_start = __start___jump_table; +struct jump_entry *iter_stop = __stop___jump_table; +struct jump_entry *iter; + +mutex_lock (&jump_label_mutex); + ret =build_jump_label_hashtable (__start___jump_table, + __stop___jump_table); + iter = iter_start; +while (iter < iter_stop) { +arch_jump_label_text_poke_early (iter->code);
kernel/jump_label.c line 199
Quoted 10 lines
kernel/jump_label.c new file +10
-0+ iter++; + } +mutex_unlock (&jump_label_mutex); +return ret; +} +early_initcall (init_jump_label); + +#ifdef CONFIG_MODULES + +static struct jump_label_module_entry *add_jump_label_module_entry (struct jump_label_entry *entry,struct jump_entry *iter_begin,int count,struct module *mod)
Eek!!
I'm not that strict on the 80 character limit, but the above can wrap it twice!!!
I'll fix these extended lines myself.
-- Steve
Quoted 186 lines
kernel/jump_label.c new file +139
-0+{ +struct jump_label_module_entry *e; + + e =kmalloc (sizeof (struct jump_label_module_entry ),GFP_KERNEL ); +if (!e) +return ERR_PTR (-ENOMEM ); + e->mod = mod; + e->nr_entries = count; + e->table = iter_begin; +hlist_add_head (&e->hlist, &entry->modules); +return e; +} + +static int add_jump_label_module (struct module *mod) +{ +struct jump_entry *iter, *iter_begin; +struct jump_label_entry *entry; +struct jump_label_module_entry *module_entry; +int count; + +/* if the module doesn't have jump label entries, just return */ +if (!mod->num_jump_entries) +return 0 ; + +sort_jump_label_entries (mod->jump_entries, + mod->jump_entries + mod->num_jump_entries); + iter = mod->jump_entries; +while (iter < mod->jump_entries + mod->num_jump_entries) { + entry =get_jump_label_entry (iter->key); + iter_begin = iter; + count =0 ; +while ((iter < mod->jump_entries + mod->num_jump_entries) && + (iter->key == iter_begin->key)) { + iter++; + count++; + } +if (!entry) { + entry =add_jump_label_entry (iter_begin->key,0 ,NULL ); +if (IS_ERR (entry)) +return PTR_ERR (entry); + } + module_entry =add_jump_label_module_entry (entry, iter_begin, + count, mod); +if (IS_ERR (module_entry)) +return PTR_ERR (module_entry); + } +return 0 ; +} + +static void remove_jump_label_module (struct module *mod) +{ +struct hlist_head *head; +struct hlist_node *node, *node_next, *module_node, *module_node_next; +struct jump_label_entry *e; +struct jump_label_module_entry *e_module; +int i; + +/* if the module doesn't have jump label entries, just return */ +if (!mod->num_jump_entries) +return ; + +for (i =0 ; i <JUMP_LABEL_TABLE_SIZE ; i++) { + head = &jump_label_table[i]; +hlist_for_each_entry_safe (e, node, node_next, head, hlist) { +hlist_for_each_entry_safe (e_module, module_node, + module_node_next, + &(e->modules), hlist) { +if (e_module->mod == mod) { +hlist_del (&e_module->hlist); +kfree (e_module); + } + } +if (hlist_empty (&e->modules) && (e->nr_entries ==0 )) { +hlist_del (&e->hlist); +kfree (e); + } + } + } +} + +static int jump_label_module_notify (struct notifier_block *self,unsigned long val,void *data) +{ +struct module *mod = data; +int ret =0 ; + +switch (val) { +case MODULE_STATE_COMING : +mutex_lock (&jump_label_mutex); + ret =add_jump_label_module (mod); +if (ret) +remove_jump_label_module (mod); +mutex_unlock (&jump_label_mutex); +break ; +case MODULE_STATE_GOING : +mutex_lock (&jump_label_mutex); +remove_jump_label_module (mod); +mutex_unlock (&jump_label_mutex); +break ; + } +return ret; +} + +/*** +* apply_jump_label_nops - patch module jump labels with arch_get_jump_label_nop() +* @mod: module to patch +* +* Allow for run-time selection of the optimal nops. Before the module +* loads patch these with arch_get_jump_label_nop(), which is specified by +* the arch specific jump label code. +*/ +void jump_label_apply_nops (struct module *mod) +{ +struct jump_entry *iter; + +/* if the module doesn't have jump label entries, just return */ +if (!mod->num_jump_entries) +return ; + + iter = mod->jump_entries; +while (iter < mod->jump_entries + mod->num_jump_entries) { +arch_jump_label_text_poke_early (iter->code); + iter++; + } +} + +struct notifier_block jump_label_module_nb = { + .notifier_call = jump_label_module_notify, + .priority =0 , +}; + +static __init int init_jump_label_module (void ) +{ +return register_module_notifier (&jump_label_module_nb); +} +early_initcall (init_jump_label_module); + +#endif /* CONFIG_MODULES */ + +#endif kernel/kprobes.c +1
-0@@ -47,6 +47,7 @@#include <linux/memory.h> #include <linux/ftrace.h> #include <linux/cpu.h> +#include <linux/jump_label.h> #include <asm-generic/sections.h> #include <asm/cacheflush.h> kernel/module.c +6
-0@@ -55,6 +55,7 @@#include <linux/async.h> #include <linux/percpu.h> #include <linux/kmemleak.h> +#include <linux/jump_label.h> #define CREATE_TRACE_POINTS #include <trace/events/module.h> @@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)sizeof (*mod->tracepoints), &mod->num_tracepoints);#endif +#ifdef HAVE_JUMP_LABEL + mod->jump_entries =section_objs (info,"__jump_table" , +sizeof (*mod->jump_entries), + &mod->num_jump_entries); +#endif #ifdef CONFIG_EVENT_TRACING mod->trace_events = section_objs(info, "_ftrace_events", sizeof(*mod->trace_events),scripts/gcc-goto.sh new file +5
-0@@ -0,0 +1,5 @@ +#!/bin/sh +# Test for gcc 'asm goto' suport +# Copyright (C) 2010, Jason Baron <[email protected]> + +echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
Re: [PATCH 03/10] jump label v11: base patch
In reply to Jason Baroninclude/linux/jump_label.h line 25
Quoted 5 lines · Jason Baron
include/linux/jump_label.h new file +5
-0+extern void arch_jump_label_transform (struct jump_entry *entry, +enum jump_label_type type); +extern void jump_label_update (unsigned long key,enum jump_label_type type); +extern void jump_label_apply_nops (struct module *mod); +extern void arch_jump_label_text_poke_early (jump_label_t addr);
These function names are too long.
Also it would be better if the types for the pointers are kept instead of casting to unsigned long. All the variables are ints right?
kernel/jump_label.c line 21
Quoted 3 lines
kernel/jump_label.c new file +3
-0+#define JUMP_LABEL_HASH_BITS 6 +#define JUMP_LABEL_TABLE_SIZE (1 << JUMP_LABEL_HASH_BITS) +static struct hlist_head jump_label_table[JUMP_LABEL_TABLE_SIZE ];
It's not clear to me why this hash table is needed. There should not be that many trace points, is it that big a problem to simply walk all the sections when something is changed?
Or maybe the sections could be just sorted and a binary search used like with exception tables.
I suspect that would simplify a lot of code.
Overall I like the idea, but the current code is too complicated for the benefit I think.
Can it be put on a diet?
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [PATCH 03/10] jump label v11: base patch
Quoted 9 lines · Andi Kleen
include/linux/jump_label.h line 25
Quoted 5 lines · Jason Baron
include/linux/jump_label.h new file +5
-0+extern void arch_jump_label_transform (struct jump_entry *entry, +enum jump_label_type type); +extern void jump_label_update (unsigned long key,enum jump_label_type type); +extern void jump_label_apply_nops (struct module *mod); +extern void arch_jump_label_text_poke_early (jump_label_t addr);These function names are too long.
Quoted 4 lines
Also it would be better if the types for the pointers are kept instead of casting to unsigned long. All the variables are ints right?
So far, yes. But I didn't want to force this, in case the users of the API wanted to use other types. But I'm ok with 'int *' here.
Quoted 11 lines
kernel/jump_label.c line 21
Quoted 3 lines
kernel/jump_label.c new file +3
-0+#define JUMP_LABEL_HASH_BITS 6 +#define JUMP_LABEL_TABLE_SIZE (1 << JUMP_LABEL_HASH_BITS) +static struct hlist_head jump_label_table[JUMP_LABEL_TABLE_SIZE ];It's not clear to me why this hash table is needed. There should not be that many trace points, is it that big a problem to simply walk all the sections when something is changed?
Or maybe the sections could be just sorted and a binary search used like with exception tables.
Quoted 8 lines
I suspect that would simplify a lot of code.
Overall I like the idea, but the current code is too complicated for the benefit I think.
Can it be put on a diet?
-Andi
So there are ~150 tracepoints, but this code is also being proposed for use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.
Also, I think the hash table deals nicely with modules. I create a linked list of only those module sections that are relevant to each hash bucket. If you search through all the section on each enable/disable, its going to be proportional to the number of modules as well.
thanks,
-Jason
Re: [PATCH 03/10] jump label v11: base patch
Quoted 5 lines
So there are ~150 tracepoints, but this code is also being proposed for use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.
Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)
Quoted 1 line
Also, I think the hash table deals nicely with modules.
Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.
-Andi
Re: [PATCH 03/10] jump label v11: base patch
Quoted 9 lines · Andi Kleen
Quoted 3 lines
So there are ~150 tracepoints, but this code is also being proposed for use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.
Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)
yes, but then for each module, I need to again search through the table to find the correct entires. Although there wouldn't be a 150 tracepoints to search for in each module there are a number of tracepoints that are used by most modules, such as kmalloc, kfree...plus any other users of this infrastructure.
Quoted 7 lines
Quoted 1 line
Also, I think the hash table deals nicely with modules.
Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.
-Andi
its right to realize that the enable/disable paths are not at the heart of this optimization. in fact, the initial few postings of this patchset, didn't include the hash table, but the added support for modules lead me to the current design.
thanks,
-Jason
Re: [PATCH 03/10] jump label v11: base patch
In reply to Andi KleenQuoted 6 lines · Andi Kleen
Quoted 3 lines
[someone, identity censored by Andi] wrote:
Also, I think the hash table deals nicely with modules.
Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.
Uhm, no - you are not making it simpler, you are making it stupider.
Using a list walk for even 100 items is already borderline horrible and tasteless. We dont want crap code.
Thanks,
Ingo
Steven Rostedt
maintainerRe: [PATCH 03/10] jump label v11: base patch
In reply to Andi KleenQuoted 9 lines · Andi Kleen
Quoted 4 lines
So there are ~150 tracepoints, but this code is also being proposed for use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.
Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)
xfs has > 100 tracepoints
Quoted 5 lines
Quoted 2 lines
Also, I think the hash table deals nicely with modules.
Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.
I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.
Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.
-- Steve
Re: [PATCH 03/10] jump label v11: base patch
Quoted 7 lines · Andi Kleen
Quoted 6 lines
Quoted 2 lines
So there are ~150 tracepoints, but this code is also being proposed
for
Quoted 2 lines
use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.
Quoted 6 lines
Quoted 4 lines
Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)
xfs has > 100 tracepoints
Doesn
Quoted 10 lines
Quoted 6 lines
Quoted 3 lines
Also, I think the hash table deals nicely with modules.
Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.
I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.
The only time you add one is when you load a module, right? When you do that you only sort the section of the new module.
Quoted 2 lines
Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.
Well problem is that the code is very complicated today. I suspect this could be done much simpler if it wasn't so overengin
-Andi
Steven Rostedt
maintainerRe: [PATCH 03/10] jump label v11: base patch
Quoted 7 lines · Andi Kleen
Quoted 7 lines · Andi Kleen
Quoted 6 lines
Quoted 2 lines
So there are ~150 tracepoints, but this code is also being proposed
for
Quoted 2 lines
use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)
xfs has > 100 tracepoints
Doesn
I suppose you were missing a 't'.
Anyway:
$ find fs/xfs/ -name "*.c" ! -type d | xargs grep "[ ^I]trace_" | wc -l 313
The jump label occurs at the calling sight, not for defined tracepoints (which can be used in multiple places).
Also take a look at fs/xfs/linux-2.6/xfs_trace.h, you will be surprised.
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
Also, I think the hash table deals nicely with modules.
Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.
Quoted 6 lines
Quoted 3 lines
I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.
The only time you add one is when you load a module, right? When you do that you only sort the section of the new module.
And on removing a module.
Quoted 7 lines
Quoted 3 lines
Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.
Well problem is that the code is very complicated today. I suspect this could be done much simpler if it wasn't so overengin
Perhaps it can be cleaned up. But I have no issues with it now, and using a hash (basic data structures 101) is not where the complexity comes in.
-- Steve
Re: [PATCH 03/10] jump label v11: base patch
Quoted 21 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 7 lines · Andi Kleen
Quoted 7 lines · Andi Kleen
Quoted 6 lines
Quoted 2 lines
So there are ~150 tracepoints, but this code is also being proposed
for
Quoted 2 lines
use with 'dynamic debug' of which there are > 1000, and I'm hoping for more users moving forward.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)
xfs has > 100 tracepoints
Doesn
I suppose you were missing a 't'.
Anyway:
Quoted 7 lines
$ find fs/xfs/ -name "*.c" ! -type d | xargs grep "[ ^I]trace_" | wc -l 313The jump label occurs at the calling sight, not for defined tracepoints (which can be used in multiple places).
Also take a look at
fs/xfs/linux-2.6/xfs_trace.h, you will be surprised.
Yep, I'd be surprised to see how many tracepoints we can end up having with stuff like kmem tracing (trace_kmalloc). Each instance of the inline function will generate an entry. (!)
Quoted 22 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Also, I think the hash table deals nicely with modules.
Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.
Quoted 6 lines
Quoted 3 lines
I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.
The only time you add one is when you load a module, right? When you do that you only sort the section of the new module.
And on removing a module.
Quoted 4 lines
Quoted 3 lines
Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.
Quoted 7 lines
Quoted 3 lines
Well problem is that the code is very complicated today. I suspect this could be done much simpler if it wasn't so overengin
Perhaps it can be cleaned up. But I have no issues with it now, and using a hash (basic data structures 101) is not where the complexity comes in.
I agree with Steven, Peter and Jason: due to the large amount of tracepoints we can end up patching, we should keep the hash tables. This code is very similar to what I have in the tracepoints already and in the immediate values. So this code is solid and has been tested over a large user base for quite some time already.
One change I would recommend is to use a separate memory pool to allocate the struct jump_label_entry, to favor better locality. I did not do it in tracepoints and markers because each entry have a variable length, but given that struct jump_label_entry seems to be fixed-size, then we should definitely go for a kmem_cache_alloc().
Thanks,
Mathieu
Quoted 4 lines
-- Steve
Signature · 4 lines
Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com
Re: [PATCH 03/10] jump label v11: base patch
Quoted 2 lines
I agree with Steven, Peter and Jason: due to the large amount of tracepoints we can end up patching, we should keep the hash tables. This
I suspect when it's cache cold the hash tables will be actually slower. As a general rule memory bloat = slow.
Quoted 3 lines
code is very similar to what I have in the tracepoints already and in the immediate values. So this code is solid and has been tested over a large user base for quite some time already.
FWIW "We always did it this way" is not a good argument in engineering discussions.
Quoted 5 lines
One change I would recommend is to use a separate memory pool to allocate the struct
jump_label_entry, to favor better locality. I did not do it in tracepoints and markers because each entry have a variable length, but given that structjump_label_entryseems to be fixed-size, then we should definitely go for akmem_cache_alloc().
Yes even more complexity, great idea.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Re: [PATCH 03/10] jump label v11: base patch
In reply to Steven RostedtQuoted 4 lines
Quoted 2 lines
Doesn
I suppose you were missing a 't'.
Sorry sentence was eaten (I had to type it twice because the webmail thingy eat it first try and probably didn't finish
it second time). I did not doubt your trace point number :)
What I meant is that the number of trace points does not really matter much for binary search, unless you're going to gigantic numbers. 100 is still searched in only a few tries.
What matters is just how many tables you need to walk and that only scales with the number of modules, not the number of trace points.
for ( each module )
binary search in section
Advantage: no additional memory, no setup code other than a simple sort, less code overall.
Also most modules are small so a binary search try is likely very fast.
The methology is also long proven for *_user() exception handling and always worked well here.
Quoted 4 lines
Quoted 2 lines
The only time you add one is when you load a module, right? When you do that you only sort the section of the new module.
And on removing a module.
On removing you simply free the module table, no real work to do.
Quoted 3 lines
Perhaps it can be cleaned up. But I have no issues with it now, and using a hash (basic data structures 101) is not where the complexity comes in.
Hash tables are not complex in themselves, but all the code to maintain them still adds up. I think considerable parts of the new code were simply that. IMHO that can be done simpler at no real loss of functionality or performance.
-Andi
H. Peter Anvin
reviewerRe: [PATCH 03/10] jump label v11: base patch
In reply to Steven RostedtQuoted 6 lines · Steven Rostedt
Quoted 5 lines
Quoted 2 lines
Also, I think the hash table deals nicely with modules.
Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.
Quoted 7 lines
I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.
Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.
If all you need is exact matches -- as in this case -- hashes are generally faster than binary anyway.
-hpa
Re: [PATCH 03/10] jump label v11: base patch
Quoted 4 lines
If all you need is exact matches -- as in this case -- hashes are generally faster than binary anyway.
Actually only if you have the whole thing in cache. The binary search has a much nicer access pattern if it's cache cold.
And the footprint of directly accessing the section than having separate large list and tables is also significantly more compared to a compact table.
But I don't think the access speed really matters here. Toggling trace points should be rare and is not a critical path operation. What matters is runtime overhead and simplicity.
Right now on the simplicity front there is a problem.
-Andi
Re: [PATCH 03/10] jump label v11: base patch
In reply to Steven RostedtQuoted 7 lines
Quoted 5 lines
Even 1000 is fine to walk, but if it was sorted a binary search would be much faster anyways. That is then you would still need to search for each module, but that is a relatively small number (< 100)
xfs has > 100 tracepoints
No problem for binary search.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Also, I think the hash table deals nicely with modules.
Maybe but it's also a lot of code. And it seems to me that it is optimizing the wrong thing. Simpler is nicer.
I guess simplicity is in the eye of the beholder. I find hashes easier to deal with than binary searching sorted lists. Every time you add a tracepoint, you need to resort the list.
You only add trace points with new modules right?
In this case you only sort the section of the new module, nothing else. And only once when you load it.
Quoted 2 lines
Hashes are much easier to deal with and scale nicely. I don't think there's enough rational to switch this to a binary list.
The problem I see is that there's a lot of hashing related code and a lot of memory overhead. I suspect with inplace access everything would be much simpler and less overhead.
For me the current implementation simply seems overengineered.
-Andi
Re: [PATCH 03/10] jump label v11: base patch
In reply to Jason BaronQuoted 215 lines · Jason Baron
base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.
Signed-off-by Jason Baron <[email protected]>--- Makefile | 5 + arch/Kconfig | 3 + arch/x86/include/asm/alternative.h | 3 +- arch/x86/kernel/alternative.c | 2 +- include/asm-generic/vmlinux.lds.h | 10 + include/linux/jump_label.h | 58 ++++++ include/linux/module.h | 5 +- kernel/Makefile | 2 +- kernel/jump_label.c | 338 ++++++++++++++++++++++++++++++++++++ kernel/kprobes.c | 1 + kernel/module.c | 6 + scripts/gcc-goto.sh | 5 + 12 files changed, 434 insertions(+), 4 deletions(-) create mode 100644 include/linux/jump_label.h create mode 100644 kernel/jump_label.c create mode 100644 scripts/gcc-goto.shMakefile +5
-0@@ -591,6 +591,11 @@ KBUILD_CFLAGS += $(call cc-option,-fno-strict-overflow) # conserve stack if available KBUILD_CFLAGS += $(call cc-option,-fconserve-stack) +# check for 'asm goto' +ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC)), y) + KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO +endif + # Add user supplied CPPFLAGS, AFLAGS and CFLAGS as the last assignments # But warn user when we do so warn-assign = \arch/Kconfig +3
-0@@ -158,4 +158,7 @@ config HAVE_PERF_EVENTS_NMI subsystem. Also has support for calculating CPU cycle events to determine how many clock cycles in a given period. +config HAVE_ARCH_JUMP_LABEL + bool + source "kernel/gcov/Kconfig"arch/x86/include/asm/alternative.h +2
-1@@ -4,6 +4,7 @@#include <linux/types.h> #include <linux/stddef.h> #include <linux/stringify.h> +#include <linux/jump_label.h> #include <asm/asm.h> /* @@ -182,7 +183,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);extern void *text_poke (void *addr,const void *opcode,size_t len);extern void *text_poke_smp (void *addr,const void *opcode,size_t len); -#if defined(CONFIG_DYNAMIC_FTRACE ) +#if defined(CONFIG_DYNAMIC_FTRACE ) || defined(HAVE_JUMP_LABEL )#define IDEAL_NOP_SIZE_5 5extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5 ];extern void arch_init_ideal_nop5 (void );arch/x86/kernel/alternative.c +1
-1@@ -641,7 +641,7 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)return addr; } -#if defined(CONFIG_DYNAMIC_FTRACE ) +#if defined(CONFIG_DYNAMIC_FTRACE ) || defined(HAVE_JUMP_LABEL )unsigned char ideal_nop5[IDEAL_NOP_SIZE_5 ];include/asm-generic/vmlinux.lds.h +10
-0@@ -220,6 +220,8 @@ \BUG_TABLE \ \ + JUMP_TABLE \ + \ /* PCI quirks */ \ .pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \ VMLINUX_SYMBOL(__start_pci_fixups_early) = .; \@@ -563,6 +565,14 @@#define BUG_TABLE #endif +#define JUMP_TABLE \ + . = ALIGN(8); \ + __jump_table : AT(ADDR(__jump_table) - LOAD_OFFSET) { \ + VMLINUX_SYMBOL(__start___jump_table) = .; \ + *(__jump_table) \ + VMLINUX_SYMBOL(__stop___jump_table) = .; \ + } +#ifdef CONFIG_PM_TRACE #define TRACEDATA \ . = ALIGN(4); \include/linux/jump_label.h new file +58
-0@@ -0,0 +1,58 @@ +#ifndef _LINUX_JUMP_LABEL_H +#define _LINUX_JUMP_LABEL_H + +#if defined(CC_HAVE_ASM_GOTO ) && defined(CONFIG_HAVE_ARCH_JUMP_LABEL ) +# include <asm/jump_label.h> +# define HAVE_JUMP_LABEL +#endif + +enum jump_label_type { +JUMP_LABEL_ENABLE , +JUMP_LABEL_DISABLE +}; + +struct module ; + +#ifdef HAVE_JUMP_LABEL + +extern struct jump_entry __start___jump_table[]; +extern struct jump_entry __stop___jump_table[]; + +extern void arch_jump_label_transform (struct jump_entry *entry, +enum jump_label_type type); +extern void jump_label_update (unsigned long key,enum jump_label_type type); +extern void jump_label_apply_nops (struct module *mod); +extern void arch_jump_label_text_poke_early (jump_label_t addr); + +#define enable_jump_label (key) \ + jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE); + +#define disable_jump_label (key) \ + jump_label_update((unsigned long)key, JUMP_LABEL_DISABLE); + +#else + +#define JUMP_LABEL (key, label) \ +do { \ + if (unlikely(*key)) \ + goto label; \ +} while (0) + +#define enable_jump_label (cond_var) \ +do { \ + *(cond_var) = 1; \ +} while (0) + +#define disable_jump_label (cond_var) \ +do { \ + *(cond_var) = 0; \ +} while (0) + +static inline int jump_label_apply_nops (struct module *mod) +{ +return 0 ; +} + +#endif + +#endif include/linux/module.h +4
-1@@ -350,7 +350,10 @@ struct modulestruct tracepoint *tracepoints;unsigned int num_tracepoints;#endif - +#ifdef HAVE_JUMP_LABEL +struct jump_entry *jump_entries; +unsigned int num_jump_entries; +#endif #ifdef CONFIG_TRACING const char **trace_bprintk_fmt_start;unsigned int num_trace_bprintk_fmt;kernel/Makefile +1
-1@@ -10,7 +10,7 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o \ kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \ hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \ notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \ - async.o range.o + async.o range.o jump_label.o obj-y += groups.o ifdef CONFIG_FUNCTION_TRACERkernel/jump_label.c new file +0
-0
kernel/jump_label.c line 4
Quoted 4 lines
kernel/jump_label.c new file +4
-0@@ -0,0 +1,338 @@ +/* +* jump label support +* +* Copyright (C) 2009 Jason Baron <[email protected]>
Wrong year.
Quoted 114 lines
kernel/jump_label.c new file +114
-0+* +*/ +#include <linux/jump_label.h> +#include <linux/memory.h> +#include <linux/uaccess.h> +#include <linux/module.h> +#include <linux/list.h> +#include <linux/jhash.h> +#include <linux/slab.h> +#include <linux/sort.h> +#include <linux/err.h> + +#ifdef HAVE_JUMP_LABEL + +#define JUMP_LABEL_HASH_BITS 6 +#define JUMP_LABEL_TABLE_SIZE (1 << JUMP_LABEL_HASH_BITS) +static struct hlist_head jump_label_table[JUMP_LABEL_TABLE_SIZE ]; + +/* mutex to protect coming/going of the the jump_label table */ +static DEFINE_MUTEX (jump_label_mutex); + +struct jump_label_entry { +struct hlist_node hlist; +struct jump_entry *table; +int nr_entries; +/* hang modules off here */ +struct hlist_head modules; +unsigned long key; +}; + +struct jump_label_module_entry { +struct hlist_node hlist; +struct jump_entry *table; +int nr_entries; +struct module *mod; +}; + +static int jump_label_cmp (const void *a,const void *b) +{ +const struct jump_entry *jea = a; +const struct jump_entry *jeb = b; + +if (jea->key < jeb->key) +return -1 ; + +if (jea->key > jeb->key) +return 1 ; + +return 0 ; +} + +static void sort_jump_label_entries (struct jump_entry *start,struct jump_entry *stop) +{ +unsigned long size; + + size = (((unsigned long )stop - (unsigned long )start) + /sizeof (struct jump_entry )); +sort (start, size,sizeof (struct jump_entry ), jump_label_cmp,NULL ); +} + +static struct jump_label_entry *get_jump_label_entry (jump_label_t key) +{ +struct hlist_head *head; +struct hlist_node *node; +struct jump_label_entry *e; +u32 hash =jhash ((void *)&key,sizeof (jump_label_t),0 ); + + head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE -1 )]; +hlist_for_each_entry (e, node, head, hlist) { +if (key == e->key) +return e; + } +return NULL ; +} + +static struct jump_label_entry *add_jump_label_entry (jump_label_t key,int nr_entries,struct jump_entry *table) +{ +struct hlist_head *head; +struct jump_label_entry *e; +u32 hash; + + e =get_jump_label_entry (key); +if (e) +return ERR_PTR (-EEXIST ); + + e =kmalloc (sizeof (struct jump_label_entry ),GFP_KERNEL ); +if (!e) +return ERR_PTR (-ENOMEM ); + + hash =jhash ((void *)&key,sizeof (jump_label_t),0 ); + head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE -1 )]; + e->key = key; + e->table = table; + e->nr_entries = nr_entries; +INIT_HLIST_HEAD (&(e->modules)); +hlist_add_head (&e->hlist, head); +return e; +} + +static int build_jump_label_hashtable (struct jump_entry *start,struct jump_entry *stop) +{ +struct jump_entry *iter, *iter_begin; +struct jump_label_entry *entry; +int count; + +sort_jump_label_entries (start, stop); + iter = start; +while (iter < stop) { + entry =get_jump_label_entry (iter->key); +if (!entry) { + iter_begin = iter; + count =0 ; +while ((iter < stop) && + (iter->key == iter_begin->key)) {
kernel/jump_label.c line 128
Quoted 10 lines
kernel/jump_label.c new file +10
-0+ iter++; + count++; + } + entry =add_jump_label_entry (iter_begin->key, + count, iter_begin); +if (IS_ERR (entry)) +return PTR_ERR (entry); + }else { +WARN_ONCE (1 ,KERN_ERR "build_jump_hashtable: unexpected entry!\n" ); +return -1 ;
Would it make sense to use a different value? Say -EINVAL?
Quoted 24 lines
kernel/jump_label.c new file +24
-0+ } + } +return 0 ; +} + +/*** +* jump_label_update - update jump label text +* @key - key value associated with a a jump label +* @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE +* +* Will enable/disable the jump for jump label @key, depending on the +* value of @type. +* +*/ + +void jump_label_update (unsigned long key,enum jump_label_type type) +{ +struct jump_entry *iter; +struct jump_label_entry *entry; +struct hlist_node *module_node; +struct jump_label_module_entry *e_module; +int count; + +mutex_lock (&jump_label_mutex);
kernel/jump_label.c line 162
Quoted 10 lines
kernel/jump_label.c new file +10
-0+ entry =get_jump_label_entry ((jump_label_t )key); +if (entry) { + count = entry->nr_entries; + iter = entry->table; +while (count--) { +if (kernel_text_address (iter->code)) +arch_jump_label_transform (iter, type); + iter++; + } +/* eanble/disable jump labels in modules */
"enable"
Quoted 23 lines
kernel/jump_label.c new file +23
-0+hlist_for_each_entry (e_module, module_node, &(entry->modules), + hlist) { + count = e_module->nr_entries; + iter = e_module->table; +while (count--) { +if (kernel_text_address (iter->code)) +arch_jump_label_transform (iter, type); + iter++; + } + } + } +mutex_unlock (&jump_label_mutex); +} + +static __init int init_jump_label (void ) +{ +int ret; +struct jump_entry *iter_start = __start___jump_table; +struct jump_entry *iter_stop = __stop___jump_table; +struct jump_entry *iter; + +mutex_lock (&jump_label_mutex); + ret = build_jump_label_hashtable(__start___jump_table,
kernel/jump_label.c line 195
Quoted 10 lines
kernel/jump_label.c new file +10
-0+ __stop___jump_table); + iter = iter_start; +while (iter < iter_stop) { +arch_jump_label_text_poke_early (iter->code); + iter++; + } +mutex_unlock (&jump_label_mutex); +return ret; +} +early_initcall (init_jump_label);
Is there any danger of not patching the instructions with NOPs before early_initcall is called? I guess for your uses (tracing) there won't. And even if there are not patched it will just call the function.
Quoted 198 lines
kernel/jump_label.c new file +143
-0+ +#ifdef CONFIG_MODULES + +static struct jump_label_module_entry *add_jump_label_module_entry (struct jump_label_entry *entry,struct jump_entry *iter_begin,int count,struct module *mod) +{ +struct jump_label_module_entry *e; + + e =kmalloc (sizeof (struct jump_label_module_entry ),GFP_KERNEL ); +if (!e) +return ERR_PTR (-ENOMEM ); + e->mod = mod; + e->nr_entries = count; + e->table = iter_begin; +hlist_add_head (&e->hlist, &entry->modules); +return e; +} + +static int add_jump_label_module (struct module *mod) +{ +struct jump_entry *iter, *iter_begin; +struct jump_label_entry *entry; +struct jump_label_module_entry *module_entry; +int count; + +/* if the module doesn't have jump label entries, just return */ +if (!mod->num_jump_entries) +return 0 ; + +sort_jump_label_entries (mod->jump_entries, + mod->jump_entries + mod->num_jump_entries); + iter = mod->jump_entries; +while (iter < mod->jump_entries + mod->num_jump_entries) { + entry =get_jump_label_entry (iter->key); + iter_begin = iter; + count =0 ; +while ((iter < mod->jump_entries + mod->num_jump_entries) && + (iter->key == iter_begin->key)) { + iter++; + count++; + } +if (!entry) { + entry =add_jump_label_entry (iter_begin->key,0 ,NULL ); +if (IS_ERR (entry)) +return PTR_ERR (entry); + } + module_entry =add_jump_label_module_entry (entry, iter_begin, + count, mod); +if (IS_ERR (module_entry)) +return PTR_ERR (module_entry); + } +return 0 ; +} + +static void remove_jump_label_module (struct module *mod) +{ +struct hlist_head *head; +struct hlist_node *node, *node_next, *module_node, *module_node_next; +struct jump_label_entry *e; +struct jump_label_module_entry *e_module; +int i; + +/* if the module doesn't have jump label entries, just return */ +if (!mod->num_jump_entries) +return ; + +for (i =0 ; i <JUMP_LABEL_TABLE_SIZE ; i++) { + head = &jump_label_table[i]; +hlist_for_each_entry_safe (e, node, node_next, head, hlist) { +hlist_for_each_entry_safe (e_module, module_node, + module_node_next, + &(e->modules), hlist) { +if (e_module->mod == mod) { +hlist_del (&e_module->hlist); +kfree (e_module); + } + } +if (hlist_empty (&e->modules) && (e->nr_entries ==0 )) { +hlist_del (&e->hlist); +kfree (e); + } + } + } +} + +static int jump_label_module_notify (struct notifier_block *self,unsigned long val,void *data) +{ +struct module *mod = data; +int ret =0 ; + +switch (val) { +case MODULE_STATE_COMING : +mutex_lock (&jump_label_mutex); + ret =add_jump_label_module (mod); +if (ret) +remove_jump_label_module (mod); +mutex_unlock (&jump_label_mutex); +break ; +case MODULE_STATE_GOING : +mutex_lock (&jump_label_mutex); +remove_jump_label_module (mod); +mutex_unlock (&jump_label_mutex); +break ; + } +return ret; +} + +/*** +* apply_jump_label_nops - patch module jump labels with arch_get_jump_label_nop() +* @mod: module to patch +* +* Allow for run-time selection of the optimal nops. Before the module +* loads patch these with arch_get_jump_label_nop(), which is specified by +* the arch specific jump label code. +*/ +void jump_label_apply_nops (struct module *mod) +{ +struct jump_entry *iter; + +/* if the module doesn't have jump label entries, just return */ +if (!mod->num_jump_entries) +return ; + + iter = mod->jump_entries; +while (iter < mod->jump_entries + mod->num_jump_entries) { +arch_jump_label_text_poke_early (iter->code); + iter++; + } +} + +struct notifier_block jump_label_module_nb = { + .notifier_call = jump_label_module_notify, + .priority =0 , +}; + +static __init int init_jump_label_module (void ) +{ +return register_module_notifier (&jump_label_module_nb); +} +early_initcall (init_jump_label_module); + +#endif /* CONFIG_MODULES */ + +#endif kernel/kprobes.c +1
-0@@ -47,6 +47,7 @@#include <linux/memory.h> #include <linux/ftrace.h> #include <linux/cpu.h> +#include <linux/jump_label.h> #include <asm-generic/sections.h> #include <asm/cacheflush.h> kernel/module.c +6
-0@@ -55,6 +55,7 @@#include <linux/async.h> #include <linux/percpu.h> #include <linux/kmemleak.h> +#include <linux/jump_label.h> #define CREATE_TRACE_POINTS #include <trace/events/module.h> @@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)sizeof (*mod->tracepoints), &mod->num_tracepoints);#endif +#ifdef HAVE_JUMP_LABEL + mod->jump_entries =section_objs (info,"__jump_table" , +sizeof (*mod->jump_entries), + &mod->num_jump_entries); +#endif #ifdef CONFIG_EVENT_TRACING mod->trace_events = section_objs(info, "_ftrace_events", sizeof(*mod->trace_events),scripts/gcc-goto.sh new file +5
-0@@ -0,0 +1,5 @@ +#!/bin/sh +# Test for gcc 'asm goto' suport +# Copyright (C) 2010, Jason Baron <[email protected]> + +echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"Signature · 7 lines
1.7.1 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH 03/10] jump label v11: base patch
Quoted 5 lines
Quoted 1 line
+early_initcall(init_jump_label);
Is there any danger of not patching the instructions with NOPs before
early_initcallis called? I guess for your uses (tracing) there won't. And even if there are not patched it will just call the function.
Ah, nevermind - if you don't patch the addresses which are in _jump_label section, the untouched code will just do far jump (jmpq) to right after the macro, which is ok.
H. Peter Anvin
reviewerRe: [PATCH 03/10] jump label v11: base patch
Quoted 9 lines · Konrad Rzeszutek Wilk
Quoted 5 lines
Quoted 1 line
+early_initcall(init_jump_label);
Is there any danger of not patching the instructions with NOPs before
early_initcallis called? I guess for your uses (tracing) there won't. And even if there are not patched it will just call the function.Ah, nevermind - if you don't patch the addresses which are in
_jump_labelsection, the untouched code will just do far jump (jmpq) to right after the macro, which is ok.
Technical point of order: jmpq is *NOT* a far jump. A far jump (called ljmp in gas) is a completely different x86 instruction with *very* different semantics.
-hpa
[tip:perf/core] jump label: Base patch for jump label
In reply to Jason BaronCommit-ID: bf5438fca2950b03c21ad868090cc1a8fcd49536 Gitweb: http://git.kernel.org/tip/bf5438fca2950b03c21ad868090cc1a8fcd49536 Author: Jason Baron <[email protected]> AuthorDate: Fri, 17 Sep 2010 11:09:00 -0400 Committer: Steven Rostedt <[email protected]> CommitDate: Wed, 22 Sep 2010 16:29:41 -0400
jump label: Base patch for jump label
base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.
LKML-Reference: <ee8b3595967989fdaf84e698dc7447d315ce972a.1284733808.git.jbaron@redhat.com>
[ cleaned up some formating ]
--- Makefile | 5 + arch/Kconfig | 3 + arch/x86/include/asm/alternative.h | 3 +- arch/x86/kernel/alternative.c | 2 +- include/asm-generic/vmlinux.lds.h | 10 + include/linux/jump_label.h | 58 ++++++ include/linux/module.h | 5 +- kernel/Makefile | 2 +- kernel/jump_label.c | 346 ++++++++++++++++++++++++++++++++++++ kernel/kprobes.c | 1 + kernel/module.c | 6 + scripts/gcc-goto.sh | 5 + 12 files changed, 442 insertions(+), 4 deletions(-)
Makefile +5 -0
@@ -591,6 +591,11 @@ KBUILD_CFLAGS += $(call cc-option,-fno-strict-overflow) # conserve stack if available KBUILD_CFLAGS += $(call cc-option,-fconserve-stack) +# check for 'asm goto' +ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC)), y) + KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO +endif + # Add user supplied CPPFLAGS, AFLAGS and CFLAGS as the last assignments # But warn user when we do so warn-assign = \
arch/Kconfig +3 -0
@@ -158,4 +158,7 @@ config HAVE_PERF_EVENTS_NMI subsystem. Also has support for calculating CPU cycle events to determine how many clock cycles in a given period. +config HAVE_ARCH_JUMP_LABEL + bool + source "kernel/gcov/Kconfig"
arch/x86/include/asm/alternative.h +2 -1
@@ -4,6 +4,7 @@#include <linux/types.h> #include <linux/stddef.h> #include <linux/stringify.h> +#include <linux/jump_label.h> #include <asm/asm.h> /*
@@ -182,7 +183,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);extern void *text_poke (void *addr,const void *opcode,size_t len);extern void *text_poke_smp (void *addr,const void *opcode,size_t len); -#if defined(CONFIG_DYNAMIC_FTRACE ) +#if defined(CONFIG_DYNAMIC_FTRACE ) || defined(HAVE_JUMP_LABEL )#define IDEAL_NOP_SIZE_5 5extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5 ];extern void arch_init_ideal_nop5 (void );
arch/x86/kernel/alternative.c +1 -1
@@ -641,7 +641,7 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)return addr; } -#if defined(CONFIG_DYNAMIC_FTRACE ) +#if defined(CONFIG_DYNAMIC_FTRACE ) || defined(HAVE_JUMP_LABEL )unsigned char ideal_nop5[IDEAL_NOP_SIZE_5 ];
include/asm-generic/vmlinux.lds.h +10 -0
@@ -220,6 +220,8 @@ \BUG_TABLE \ \ + JUMP_TABLE \ + \ /* PCI quirks */ \ .pci_fixup : AT(ADDR(.pci_fixup) - LOAD_OFFSET) { \ VMLINUX_SYMBOL(__start_pci_fixups_early) = .; \
@@ -563,6 +565,14 @@#define BUG_TABLE #endif +#define JUMP_TABLE \ + . = ALIGN(8); \ + __jump_table : AT(ADDR(__jump_table) - LOAD_OFFSET) { \ + VMLINUX_SYMBOL(__start___jump_table) = .; \ + *(__jump_table) \ + VMLINUX_SYMBOL(__stop___jump_table) = .; \ + } +#ifdef CONFIG_PM_TRACE #define TRACEDATA \ . = ALIGN(4); \
include/linux/jump_label.h new file +58 -0
58 lines not on this page · raw message
include/linux/module.h +4 -1
@@ -350,7 +350,10 @@ struct modulestruct tracepoint *tracepoints;unsigned int num_tracepoints;#endif - +#ifdef HAVE_JUMP_LABEL +struct jump_entry *jump_entries; +unsigned int num_jump_entries; +#endif #ifdef CONFIG_TRACING const char **trace_bprintk_fmt_start;unsigned int num_trace_bprintk_fmt;
kernel/Makefile +1 -1
@@ -10,7 +10,7 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o \ kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \ hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \ notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \ - async.o range.o + async.o range.o jump_label.o obj-$(CONFIG_HAVE_EARLY_RES) += early_res.o obj-y += groups.o
kernel/jump_label.c new file +346 -0
346 lines not on this page · raw message
kernel/kprobes.c +1 -0
@@ -47,6 +47,7 @@#include <linux/memory.h> #include <linux/ftrace.h> #include <linux/cpu.h> +#include <linux/jump_label.h> #include <asm-generic/sections.h> #include <asm/cacheflush.h>
kernel/module.c +6 -0
@@ -55,6 +55,7 @@#include <linux/async.h> #include <linux/percpu.h> #include <linux/kmemleak.h> +#include <linux/jump_label.h> #define CREATE_TRACE_POINTS #include <trace/events/module.h>
@@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)sizeof (*mod->tracepoints), &mod->num_tracepoints);#endif +#ifdef HAVE_JUMP_LABEL + mod->jump_entries =section_objs (info,"__jump_table" , +sizeof (*mod->jump_entries), + &mod->num_jump_entries); +#endif #ifdef CONFIG_EVENT_TRACING mod->trace_events = section_objs(info, "_ftrace_events", sizeof(*mod->trace_events),
scripts/gcc-goto.sh new file +5 -0
@@ -0,0 +1,5 @@ +#!/bin/sh +# Test for gcc 'asm goto' suport +# Copyright (C) 2010, Jason Baron <[email protected]> + +echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
PATCH 4/10 jump label v11: initialize workqueue tracepoints *before* they are registered
In reply to Jason BaronInitialize the workqueue data structures *before* they are registered so that they are ready for callbacks.
--- kernel/trace/trace_workqueue.c | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-)
kernel/trace/trace_workqueue.c +5 -5
@@ -263,6 +263,11 @@ int __init trace_workqueue_early_init(void) {int ret, cpu; +for_each_possible_cpu (cpu) { +spin_lock_init (&workqueue_cpu_stat (cpu)->lock); +INIT_LIST_HEAD (&workqueue_cpu_stat (cpu)->list); + } + ret =register_trace_workqueue_insertion (probe_workqueue_insertion,NULL );if (ret) goto out;
@@ -279,11 +284,6 @@ int __init trace_workqueue_early_init(void)if (ret) goto no_creation; -for_each_possible_cpu (cpu) { -spin_lock_init (&workqueue_cpu_stat (cpu)->lock); -INIT_LIST_HEAD (&workqueue_cpu_stat (cpu)->list); - } -return 0 ; no_creation:
Signature · 1 line
1.7.1
[tip:perf/core] jump label: Initialize workqueue tracepoints *before* they are registered
Commit-ID: e0cf0cd49632552f063fb3ae58691946da45fb2e Gitweb: http://git.kernel.org/tip/e0cf0cd49632552f063fb3ae58691946da45fb2e Author: Jason Baron <[email protected]> AuthorDate: Fri, 17 Sep 2010 11:09:04 -0400 Committer: Steven Rostedt <[email protected]> CommitDate: Wed, 22 Sep 2010 16:30:03 -0400
jump label: Initialize workqueue tracepoints *before* they are registered
Initialize the workqueue data structures *before* they are registered so that they are ready for callbacks.
LKML-Reference: <e3a3383fc370ac7086625bebe89d9480d7caf372.1284733808.git.jbaron@redhat.com>
--- kernel/trace/trace_workqueue.c | 10 +++++----- 1 files changed, 5 insertions(+), 5 deletions(-)
kernel/trace/trace_workqueue.c +5 -5
@@ -263,6 +263,11 @@ int __init trace_workqueue_early_init(void) {int ret, cpu; +for_each_possible_cpu (cpu) { +spin_lock_init (&workqueue_cpu_stat (cpu)->lock); +INIT_LIST_HEAD (&workqueue_cpu_stat (cpu)->list); + } + ret =register_trace_workqueue_insertion (probe_workqueue_insertion,NULL );if (ret) goto out;
@@ -279,11 +284,6 @@ int __init trace_workqueue_early_init(void)if (ret) goto no_creation; -for_each_possible_cpu (cpu) { -spin_lock_init (&workqueue_cpu_stat (cpu)->lock); -INIT_LIST_HEAD (&workqueue_cpu_stat (cpu)->list); - } -return 0 ; no_creation:
PATCH 5/10 jump label v11: jump_label_text_reserved() to reserve our jump points
In reply to Jason BaronAdd a jump_label_text_reserved(void *start, void *end), so that other pieces of code that want to modify kernel text, can first verify that jump label has not reserved the instruction.
--- arch/x86/kernel/kprobes.c | 3 +- include/linux/jump_label.h | 8 ++++- kernel/jump_label.c | 83 ++++++++++++++++++++++++++++++++++++++++++++ kernel/kprobes.c | 3 +- 4 files changed, 94 insertions(+), 3 deletions(-)
arch/x86/kernel/kprobes.c +2 -1
@@ -1221,7 +1221,8 @@ static int __kprobes copy_optimized_instructions(u8 *dest, u8 *src) }/* Check whether the address range is reserved */ if (ftrace_text_reserved (src, src + len -1 ) || -alternatives_text_reserved (src, src + len -1 )) +alternatives_text_reserved (src, src + len -1 ) || +jump_label_text_reserved (src, src + len -1 ))return -EBUSY ;return len;
include/linux/jump_label.h +7 -1
@@ -20,9 +20,10 @@ extern struct jump_entry __stop___jump_table[];extern void arch_jump_label_transform (struct jump_entry *entry,enum jump_label_type type); +extern void arch_jump_label_text_poke_early (jump_label_t addr);extern void jump_label_update (unsigned long key,enum jump_label_type type);extern void jump_label_apply_nops (struct module *mod); -extern void arch_jump_label_text_poke_early (jump_label_t addr); +extern int jump_label_text_reserved (void *start,void *end);#define enable_jump_label (key) \ jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE);
@@ -53,6 +54,11 @@ static inline int jump_label_apply_nops(struct module *mod)return 0 ; } +static inline int jump_label_text_reserved (void *start,void *end) +{ +return 0 ; +} +#endif #endif
kernel/jump_label.c +83 -0
89 lines not on this page · raw message
kernel/kprobes.c +2 -1
@@ -1142,7 +1142,8 @@ int __kprobes register_kprobe(struct kprobe *p)preempt_disable ();if (!kernel_text_address ((unsigned long ) p->addr) ||in_kprobes_functions ((unsigned long ) p->addr) || -ftrace_text_reserved (p->addr, p->addr)) { +ftrace_text_reserved (p->addr, p->addr) || +jump_label_text_reserved (p->addr, p->addr)) {preempt_enable ();return -EINVAL ; }
Signature · 1 line
1.7.1
[tip:perf/core] jump label: Add jump_label_text_reserved() to reserve jump points
Commit-ID: 4c3ef6d79328c0e23ade60cbfc8d496123a6855c Gitweb: http://git.kernel.org/tip/4c3ef6d79328c0e23ade60cbfc8d496123a6855c Author: Jason Baron <[email protected]> AuthorDate: Fri, 17 Sep 2010 11:09:08 -0400 Committer: Steven Rostedt <[email protected]> CommitDate: Wed, 22 Sep 2010 16:30:46 -0400
jump label: Add jump_label_text_reserved() to reserve jump points
Add a jump_label_text_reserved(void *start, void *end), so that other pieces of code that want to modify kernel text, can first verify that jump label has not reserved the instruction.
LKML-Reference: <06236663a3a7b1c1f13576bb9eccb6d9c17b7bfe.1284733808.git.jbaron@redhat.com>
--- arch/x86/kernel/kprobes.c | 3 +- include/linux/jump_label.h | 8 ++++- kernel/jump_label.c | 83 ++++++++++++++++++++++++++++++++++++++++++++ kernel/kprobes.c | 3 +- 4 files changed, 94 insertions(+), 3 deletions(-)
arch/x86/kernel/kprobes.c +2 -1
@@ -1218,7 +1218,8 @@ static int __kprobes copy_optimized_instructions(u8 *dest, u8 *src) }/* Check whether the address range is reserved */ if (ftrace_text_reserved (src, src + len -1 ) || -alternatives_text_reserved (src, src + len -1 )) +alternatives_text_reserved (src, src + len -1 ) || +jump_label_text_reserved (src, src + len -1 ))return -EBUSY ;return len;
include/linux/jump_label.h +7 -1
@@ -20,9 +20,10 @@ extern struct jump_entry __stop___jump_table[];extern void arch_jump_label_transform (struct jump_entry *entry,enum jump_label_type type); +extern void arch_jump_label_text_poke_early (jump_label_t addr);extern void jump_label_update (unsigned long key,enum jump_label_type type);extern void jump_label_apply_nops (struct module *mod); -extern void arch_jump_label_text_poke_early (jump_label_t addr); +extern int jump_label_text_reserved (void *start,void *end);#define enable_jump_label (key) \ jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE);
@@ -53,6 +54,11 @@ static inline int jump_label_apply_nops(struct module *mod)return 0 ; } +static inline int jump_label_text_reserved (void *start,void *end) +{ +return 0 ; +} +#endif #endif
kernel/jump_label.c +83 -0
89 lines not on this page · raw message
kernel/kprobes.c +2 -1
@@ -1147,7 +1147,8 @@ int __kprobes register_kprobe(struct kprobe *p)preempt_disable ();if (!kernel_text_address ((unsigned long ) p->addr) ||in_kprobes_functions ((unsigned long ) p->addr) || -ftrace_text_reserved (p->addr, p->addr)) { +ftrace_text_reserved (p->addr, p->addr) || +jump_label_text_reserved (p->addr, p->addr)) {preempt_enable ();return -EINVAL ; }
PATCH 6/10 jump label v11: tracepoint support
In reply to Jason BaronMake use of the jump label infrastructure for tracepoints.
--- include/linux/tracepoint.h | 5 ++++- kernel/tracepoint.c | 14 ++++++++++++-- 2 files changed, 16 insertions(+), 3 deletions(-)
include/linux/tracepoint.h +4 -1
@@ -17,6 +17,7 @@#include <linux/errno.h> #include <linux/types.h> #include <linux/rcupdate.h> +#include <linux/jump_label.h> struct module ;struct tracepoint ;
@@ -145,7 +146,9 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin, extern struct tracepoint __tracepoint_##name; \ static inline void trace_##name(proto) \ { \ - if (unlikely(__tracepoint_##name.state)) \ + JUMP_LABEL(&__tracepoint_##name.state, do_trace); \ + return; \ +do_trace: \ __DO_TRACE(&__tracepoint_##name, \ TP_PROTO(data_proto), \ TP_ARGS(data_args)); \
kernel/tracepoint.c +12 -2
@@ -25,6 +25,7 @@#include <linux/err.h> #include <linux/slab.h> #include <linux/sched.h> +#include <linux/jump_label.h> extern struct tracepoint __start___tracepoints[];extern struct tracepoint __stop___tracepoints[];
@@ -263,7 +264,13 @@ static void set_tracepoint(struct tracepoint_entry **entry,* is used. */ rcu_assign_pointer (elem->funcs, (*entry)->funcs); - elem->state = active; +if (!elem->state && active) { +enable_jump_label (&elem->state); + elem->state = active; + }else if (elem->state && !active) { +disable_jump_label (&elem->state); + elem->state = active; + } }/*
@@ -277,7 +284,10 @@ static void disable_tracepoint(struct tracepoint *elem)if (elem->unregfunc && elem->state) elem->unregfunc (); - elem->state =0 ; +if (elem->state) { +disable_jump_label (&elem->state); + elem->state =0 ; + }rcu_assign_pointer (elem->funcs,NULL ); }
Signature · 1 line
1.7.1
[tip:perf/core] jump label: Tracepoint support for jump labels
Commit-ID: 8f7b50c514206211cc282a4247f7b12f18dee674 Gitweb: http://git.kernel.org/tip/8f7b50c514206211cc282a4247f7b12f18dee674 Author: Jason Baron <[email protected]> AuthorDate: Fri, 17 Sep 2010 11:09:13 -0400 Committer: Steven Rostedt <[email protected]> CommitDate: Wed, 22 Sep 2010 16:31:01 -0400
jump label: Tracepoint support for jump labels
Make use of the jump label infrastructure for tracepoints.
LKML-Reference: <a9ba2056e2c9cf332c3c300b577463ce66ff23a8.1284733808.git.jbaron@redhat.com>
--- include/linux/tracepoint.h | 5 ++++- kernel/tracepoint.c | 14 ++++++++++++-- 2 files changed, 16 insertions(+), 3 deletions(-)
include/linux/tracepoint.h +4 -1
@@ -17,6 +17,7 @@#include <linux/errno.h> #include <linux/types.h> #include <linux/rcupdate.h> +#include <linux/jump_label.h> struct module ;struct tracepoint ;
@@ -145,7 +146,9 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin, extern struct tracepoint __tracepoint_##name; \ static inline void trace_##name(proto) \ { \ - if (unlikely(__tracepoint_##name.state)) \ + JUMP_LABEL(&__tracepoint_##name.state, do_trace); \ + return; \ +do_trace: \ __DO_TRACE(&__tracepoint_##name, \ TP_PROTO(data_proto), \ TP_ARGS(data_args)); \
kernel/tracepoint.c +12 -2
@@ -25,6 +25,7 @@#include <linux/err.h> #include <linux/slab.h> #include <linux/sched.h> +#include <linux/jump_label.h> extern struct tracepoint __start___tracepoints[];extern struct tracepoint __stop___tracepoints[];
@@ -263,7 +264,13 @@ static void set_tracepoint(struct tracepoint_entry **entry,* is used. */ rcu_assign_pointer (elem->funcs, (*entry)->funcs); - elem->state = active; +if (!elem->state && active) { +enable_jump_label (&elem->state); + elem->state = active; + }else if (elem->state && !active) { +disable_jump_label (&elem->state); + elem->state = active; + } }/*
@@ -277,7 +284,10 @@ static void disable_tracepoint(struct tracepoint *elem)if (elem->unregfunc && elem->state) elem->unregfunc (); - elem->state =0 ; +if (elem->state) { +disable_jump_label (&elem->state); + elem->state =0 ; + }rcu_assign_pointer (elem->funcs,NULL ); }
PATCH 7/10 jump label v11: convert dynamic debug to use jump labels
In reply to Jason BaronConvert the 'dynamic debug' infrastructure to use jump labels.
--- include/linux/dynamic_debug.h | 39 +++++++++++++----------- lib/dynamic_debug.c | 42 ++------------------------- scripts/Makefile.lib | 11 +------ scripts/basic/Makefile | 2 +- scripts/basic/hash.c | 64 ----------------------------------------- 5 files changed, 26 insertions(+), 132 deletions(-) delete mode 100644 scripts/basic/hash.c
include/linux/dynamic_debug.h +21 -18
@@ -1,6 +1,8 @@#ifndef _DYNAMIC_DEBUG_H#define _DYNAMIC_DEBUG_H +#include <linux/jump_label.h> +/* dynamic_printk_enabled, and dynamic_printk_enabled2 are bitmasks in which * bit n is set to 1 if any modname hashes into the bucket n, 0 otherwise. They * use independent hash functions, to reduce the chance of false positives.
@@ -22,8 +24,6 @@ struct _ddebug {const char *function;const char *filename;const char *format; -char primary_hash; -char secondary_hash; unsigned int lineno:24;/* * The flags field controls the behaviour at the callsite.
@@ -33,6 +33,7 @@ struct _ddebug {#define _DPRINTK_FLAGS_PRINT (1<<0)/* printk() a message using the format */ #define _DPRINTK_FLAGS_DEFAULT 0 unsigned int flags:8; +char enabled; } __attribute__((aligned(8)));
@@ -42,33 +43,35 @@ int ddebug_add_module(struct _ddebug *tab, unsigned int n,#if defined(CONFIG_DYNAMIC_DEBUG )extern int ddebug_remove_module (const char *mod_name); -#define __dynamic_dbg_enabled (dd) ({ \ - int __ret = 0; \ - if (unlikely((dynamic_debug_enabled & (1LL << DEBUG_HASH)) && \ - (dynamic_debug_enabled2 & (1LL << DEBUG_HASH2)))) \ - if (unlikely(dd.flags)) \ - __ret = 1; \ - __ret; }) -#define dynamic_pr_debug (fmt, ...) do { \ + __label__ do_printk; \ + __label__ out; \ static struct _ddebug descriptor \ __used \ __attribute__((section("__verbose"), aligned(8))) = \ - { KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH, \ - DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT }; \ - if (__dynamic_dbg_enabled(descriptor)) \ - printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__); \ + { KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__, \ + _DPRINTK_FLAGS_DEFAULT }; \ + JUMP_LABEL(&descriptor.enabled, do_printk); \ + goto out; \ +do_printk: \ + printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__); \ +out: ; \ } while (0)#define dynamic_dev_dbg (dev, fmt, ...) do { \ + __label__ do_printk; \ + __label__ out; \ static struct _ddebug descriptor \ __used \ __attribute__((section("__verbose"), aligned(8))) = \ - { KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH, \ - DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT }; \ - if (__dynamic_dbg_enabled(descriptor)) \ - dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__); \ + { KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__, \ + _DPRINTK_FLAGS_DEFAULT }; \ + JUMP_LABEL(&descriptor.enabled, do_printk); \ + goto out; \ +do_printk: \ + dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__); \ +out: ; \ } while (0)#else
lib/dynamic_debug.c +3 -39
65 lines not on this page · raw message
scripts/Makefile.lib +1 -10
@@ -101,14 +101,6 @@ basename_flags = -D"KBUILD_BASENAME=KBUILD_STR($(call name-fix,$(basetarget)))" modname_flags = $(if $(filter 1,$(words $(modname))),\ -D"KBUILD_MODNAME=KBUILD_STR($(call name-fix,$(modname)))") -#hash values -ifdef CONFIG_DYNAMIC_DEBUG -debug_flags = -D"DEBUG_HASH=$(shell ./scripts/basic/hash djb2 $(@D)$(modname))"\ - -D"DEBUG_HASH2=$(shell ./scripts/basic/hash r5 $(@D)$(modname))" -else -debug_flags = -endif - orig_c_flags = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \ $(ccflags-y) $(CFLAGS_$(basetarget).o) _c_flags = $(filter-out $(CFLAGS_REMOVE_$(basetarget).o), $(orig_c_flags))
@@ -152,8 +144,7 @@ endif c_flags = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) \ $(__c_flags) $(modkern_cflags) \ - -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags) \ - $(debug_flags) + -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags) a_flags = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) \ $(__a_flags) $(modkern_aflags)
scripts/basic/Makefile +1 -1
@@ -9,7 +9,7 @@ # fixdep: Used to generate dependency information during build process # docproc: Used in Documentation/DocBook -hostprogs-y := fixdep docproc hash +hostprogs-y := fixdep docproc always := $(hostprogs-y) # fixdep is needed to compile other host programs
scripts/basic/hash.c deleted +0 -64
64 lines not on this page · raw message
Signature · 1 line
1.7.1
[tip:perf/core] jump label: Convert dynamic debug to use jump labels
Commit-ID: 52159d98be6f26c48f5e02c7ab3c9848a85979b5 Gitweb: http://git.kernel.org/tip/52159d98be6f26c48f5e02c7ab3c9848a85979b5 Author: Jason Baron <[email protected]> AuthorDate: Fri, 17 Sep 2010 11:09:17 -0400 Committer: Steven Rostedt <[email protected]> CommitDate: Wed, 22 Sep 2010 16:31:19 -0400
jump label: Convert dynamic debug to use jump labels
Convert the 'dynamic debug' infrastructure to use jump labels.
LKML-Reference: <b77627358cea3e27d7be4386f45f66219afb8452.1284733808.git.jbaron@redhat.com>
--- include/linux/dynamic_debug.h | 39 +++++++++++++----------- lib/dynamic_debug.c | 42 ++------------------------- scripts/Makefile.lib | 11 +------ scripts/basic/Makefile | 2 +- scripts/basic/hash.c | 64 ----------------------------------------- 5 files changed, 26 insertions(+), 132 deletions(-)
include/linux/dynamic_debug.h +21 -18
@@ -1,6 +1,8 @@#ifndef _DYNAMIC_DEBUG_H#define _DYNAMIC_DEBUG_H +#include <linux/jump_label.h> +/* dynamic_printk_enabled, and dynamic_printk_enabled2 are bitmasks in which * bit n is set to 1 if any modname hashes into the bucket n, 0 otherwise. They * use independent hash functions, to reduce the chance of false positives.
@@ -22,8 +24,6 @@ struct _ddebug {const char *function;const char *filename;const char *format; -char primary_hash; -char secondary_hash; unsigned int lineno:24;/* * The flags field controls the behaviour at the callsite.
@@ -33,6 +33,7 @@ struct _ddebug {#define _DPRINTK_FLAGS_PRINT (1<<0)/* printk() a message using the format */ #define _DPRINTK_FLAGS_DEFAULT 0 unsigned int flags:8; +char enabled; } __attribute__((aligned(8)));
@@ -42,33 +43,35 @@ int ddebug_add_module(struct _ddebug *tab, unsigned int n,#if defined(CONFIG_DYNAMIC_DEBUG )extern int ddebug_remove_module (const char *mod_name); -#define __dynamic_dbg_enabled (dd) ({ \ - int __ret = 0; \ - if (unlikely((dynamic_debug_enabled & (1LL << DEBUG_HASH)) && \ - (dynamic_debug_enabled2 & (1LL << DEBUG_HASH2)))) \ - if (unlikely(dd.flags)) \ - __ret = 1; \ - __ret; }) -#define dynamic_pr_debug (fmt, ...) do { \ + __label__ do_printk; \ + __label__ out; \ static struct _ddebug descriptor \ __used \ __attribute__((section("__verbose"), aligned(8))) = \ - { KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH, \ - DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT }; \ - if (__dynamic_dbg_enabled(descriptor)) \ - printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__); \ + { KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__, \ + _DPRINTK_FLAGS_DEFAULT }; \ + JUMP_LABEL(&descriptor.enabled, do_printk); \ + goto out; \ +do_printk: \ + printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__); \ +out: ; \ } while (0)#define dynamic_dev_dbg (dev, fmt, ...) do { \ + __label__ do_printk; \ + __label__ out; \ static struct _ddebug descriptor \ __used \ __attribute__((section("__verbose"), aligned(8))) = \ - { KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH, \ - DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT }; \ - if (__dynamic_dbg_enabled(descriptor)) \ - dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__); \ + { KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__, \ + _DPRINTK_FLAGS_DEFAULT }; \ + JUMP_LABEL(&descriptor.enabled, do_printk); \ + goto out; \ +do_printk: \ + dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__); \ +out: ; \ } while (0)#else
lib/dynamic_debug.c +3 -39
65 lines not on this page · raw message
scripts/Makefile.lib +1 -10
@@ -101,14 +101,6 @@ basename_flags = -D"KBUILD_BASENAME=KBUILD_STR($(call name-fix,$(basetarget)))" modname_flags = $(if $(filter 1,$(words $(modname))),\ -D"KBUILD_MODNAME=KBUILD_STR($(call name-fix,$(modname)))") -#hash values -ifdef CONFIG_DYNAMIC_DEBUG -debug_flags = -D"DEBUG_HASH=$(shell ./scripts/basic/hash djb2 $(@D)$(modname))"\ - -D"DEBUG_HASH2=$(shell ./scripts/basic/hash r5 $(@D)$(modname))" -else -debug_flags = -endif - orig_c_flags = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \ $(ccflags-y) $(CFLAGS_$(basetarget).o) _c_flags = $(filter-out $(CFLAGS_REMOVE_$(basetarget).o), $(orig_c_flags))
@@ -152,8 +144,7 @@ endif c_flags = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) \ $(__c_flags) $(modkern_cflags) \ - -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags) \ - $(debug_flags) + -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags) a_flags = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) \ $(__a_flags) $(modkern_aflags)
scripts/basic/Makefile +1 -1
@@ -9,7 +9,7 @@ # fixdep: Used to generate dependency information during build process # docproc: Used in Documentation/DocBook -hostprogs-y := fixdep docproc hash +hostprogs-y := fixdep docproc always := $(hostprogs-y) # fixdep is needed to compile other host programs
scripts/basic/hash.c deleted +0 -64
64 lines not on this page · raw message
PATCH 8/10 jump label v11: x86 support
In reply to Jason Baronadd x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/jump_label.h | 47 +++++++++++++++++++++++++++++++++++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/jump_label.c | 49 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/module.c | 3 ++ 5 files changed, 101 insertions(+), 1 deletions(-) create mode 100644 arch/x86/include/asm/jump_label.h create mode 100644 arch/x86/kernel/jump_label.c
arch/x86/Kconfig +1 -0
@@ -60,6 +60,7 @@ config X86 select ANON_INODES select HAVE_ARCH_KMEMCHECK select HAVE_USER_RETURN_NOTIFIER + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE config INSTRUCTION_DECODER def_bool (KPROBES || PERF_EVENTS)
arch/x86/include/asm/jump_label.h new file +47 -0
47 lines not on this page · raw message
arch/x86/kernel/Makefile +1 -1
@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o := n obj-y := process_$(BITS).o signal.o entry_$(BITS).o obj-y += traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o obj-y += time.o ioport.o ldt.o dumpstack.o -obj-y += setup.o x86_init.o i8259.o irqinit.o +obj-y += setup.o x86_init.o i8259.o irqinit.o jump_label.o obj-$(CONFIG_X86_VISWS) += visws_quirks.o obj-$(CONFIG_X86_32) += probe_roms_32.o obj-$(CONFIG_X86_32) += sys_i386_32.o i386_ksyms_32.o
arch/x86/kernel/jump_label.c new file +49 -0
49 lines not on this page · raw message
arch/x86/kernel/module.c +3 -0
@@ -239,6 +239,9 @@ int module_finalize(const Elf_Ehdr *hdr,apply_paravirt (pseg, pseg + para->sh_size); } +/* make jump label nops */ +jump_label_apply_nops (me); +return module_bug_finalize (hdr, sechdrs, me); }
Signature · 1 line
1.7.1
Steven Rostedt
maintainerRe: [PATCH 08/10] jump label v11: x86 support
I'm starting to play with this. I just updated my distcc to gcc 4.5.1 so that I can.
Quoted 68 lines · Jason Baron
add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.
Signed-off-by Jason Baron <[email protected]>--- arch/x86/Kconfig | 1 + arch/x86/include/asm/jump_label.h | 47 +++++++++++++++++++++++++++++++++++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/jump_label.c | 49 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/module.c | 3 ++ 5 files changed, 101 insertions(+), 1 deletions(-) create mode 100644 arch/x86/include/asm/jump_label.h create mode 100644 arch/x86/kernel/jump_label.carch/x86/Kconfig +1
-0@@ -60,6 +60,7 @@ config X86 select ANON_INODES select HAVE_ARCH_KMEMCHECK select HAVE_USER_RETURN_NOTIFIER + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE config INSTRUCTION_DECODER def_bool (KPROBES || PERF_EVENTS)arch/x86/include/asm/jump_label.h new file +35
-0@@ -0,0 +1,47 @@ +#ifndef _ASM_X86_JUMP_LABEL_H +#define _ASM_X86_JUMP_LABEL_H + +#ifdef __KERNEL__ + +#include <linux/types.h> +#include <asm/nops.h> + +#define JUMP_LABEL_NOP_SIZE 5 + +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t" + +# define JUMP_LABEL (key, label) \ + do { \ + asm goto("1:" \ + JUMP_LABEL_INITIAL_NOP \ + ".pushsection __jump_table, \"a\" \n\t"\ + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \ + ".popsection \n\t" \ + : : "i" (key) : : label); \ + } while (0) + +#endif /* __KERNEL__ */ + +#ifdef CONFIG_X86_64 + +typedef u64 jump_label_t ; + +struct jump_entry { +jump_label_t code; +jump_label_t target; +jump_label_t key; +}; + +#else
arch/x86/include/asm/jump_label.h line 45
Quoted 10 lines
arch/x86/include/asm/jump_label.h new file +10
-0+ +typedef u32 jump_label_t ; + +struct jump_entry { +jump_label_t code; +jump_label_t target; +jump_label_t key; +}; + +#endif
Why duplicate the structure? You typedef'd jump_label_t and used that in the structure twice!
-- Steve
Quoted 84 lines
arch/x86/include/asm/jump_label.h new file +2
-0+ +#endif arch/x86/kernel/Makefile +1
-1@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o := n obj-y := process_$(BITS).o signal.o entry_$(BITS).o obj-y += traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o obj-y += time.o ioport.o ldt.o dumpstack.o -obj-y += setup.o x86_init.o i8259.o irqinit.o +obj-y += setup.o x86_init.o i8259.o irqinit.o jump_label.o obj-$(CONFIG_X86_VISWS) += visws_quirks.o obj-$(CONFIG_X86_32) += probe_roms_32.o obj-$(CONFIG_X86_32) += sys_i386_32.o i386_ksyms_32.oarch/x86/kernel/jump_label.c new file +49
-0@@ -0,0 +1,49 @@ +/* +* jump label x86 support +* +* Copyright (C) 2009 Jason Baron <[email protected]> +* +*/ +#include <linux/jump_label.h> +#include <linux/memory.h> +#include <linux/uaccess.h> +#include <linux/module.h> +#include <linux/list.h> +#include <linux/jhash.h> +#include <linux/cpu.h> +#include <asm/kprobes.h> +#include <asm/alternative.h> + +#ifdef HAVE_JUMP_LABEL + +union jump_code_union { +char code[JUMP_LABEL_NOP_SIZE ]; +struct { +char jump; +int offset; + } __attribute__((packed)); +}; + +void arch_jump_label_transform (struct jump_entry *entry,enum jump_label_type type) +{ +union jump_code_union code; + +if (type ==JUMP_LABEL_ENABLE ) { + code.jump =0xe9 ; + code.offset = entry->target - + (entry->code +JUMP_LABEL_NOP_SIZE ); + }else +memcpy (&code, ideal_nop5,JUMP_LABEL_NOP_SIZE ); +get_online_cpus (); +mutex_lock (&text_mutex); +text_poke_smp ((void *)entry->code, &code,JUMP_LABEL_NOP_SIZE ); +mutex_unlock (&text_mutex); +put_online_cpus (); +} + +void arch_jump_label_text_poke_early (jump_label_t addr) +{ +text_poke_early ((void *)addr, ideal_nop5,JUMP_LABEL_NOP_SIZE ); +} + +#endif arch/x86/kernel/module.c +3
-0@@ -239,6 +239,9 @@ int module_finalize(const Elf_Ehdr *hdr,apply_paravirt (pseg, pseg + para->sh_size); } +/* make jump label nops */ +jump_label_apply_nops (me); +return module_bug_finalize (hdr, sechdrs, me); }
H. Peter Anvin
reviewerRe: [PATCH 08/10] jump label v11: x86 support
In reply to Jason Baronarch/x86/Kconfig config X86 · line 63
Quoted 5 lines · Jason Baron
add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.
arch/x86/Kconfig +1
-0+ select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
Why?
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Re: [PATCH 08/10] jump label v11: x86 support
Quoted 11 lines · H. Peter Anvin
arch/x86/Kconfig config X86 · line 63
Quoted 5 lines · Jason Baron
add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.
arch/x86/Kconfig +1
-0+ select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZEWhy?
-hpa
Hi,
Quoted 1 line
From the documentation patch:
"
The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is set, gcc does not always out of line the not taken label path in the same way that the "if unlikely()" paths are made out of line. Thus, with CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be solved in subsequent gcc versions, that allow us to move labels out of line, while still optimizing for size.
"
This was also discuessed a bit on lkml previously:
http://marc.info/?l=linux-kernel&m=127618233108964&w=2
thanks,
-Jason
Re: [PATCH 08/10] jump label v11: x86 support
Quoted 17 lines · Jason Baron
Quoted 11 lines · H. Peter Anvin
arch/x86/Kconfig config X86 · line 63
Quoted 5 lines · Jason Baron
add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.
arch/x86/Kconfig +1
-0+ select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZEWhy?
-hpa
Hi,
Quoted 1 line
From the documentation patch:
Quoted 7 lines
" The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When
CC_OPTIMIZE_FOR_SIZEis set, gcc does not always out of line the not taken label path in the same way that the "ifunlikely()" paths are made out of line. Thus, withCC_OPTIMIZE_FOR_SIZEset, this optimization is not always optimal. This may be solved in subsequent gcc versions, that allow us to move labels out of line, while still optimizing for size. "
OTOH making a difficult optimization (HAVE_ARCH_JUMP_LABEL) dependent on compiler flags is really asking for trouble.
So how about enabling it unconditionally, and just chalk up the cost under CC_OPTIMIZE_FOR_SIZE as one of the costs it already has? This also has the advantage that future compilers can improve things without having to wait for yet another kernel patch that re-enables HAVE_ARCH_JUMP_LABEL.
Thanks,
Ingo
Steven Rostedt
maintainerRe: [PATCH 08/10] jump label v11: x86 support
Quoted 10 lines · Ingo Molnar
Quoted 9 lines
Quoted 1 line
From the documentation patch:
" The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When
CC_OPTIMIZE_FOR_SIZEis set, gcc does not always out of line the not taken label path in the same way that the "ifunlikely()" paths are made out of line. Thus, withCC_OPTIMIZE_FOR_SIZEset, this optimization is not always optimal. This may be solved in subsequent gcc versions, that allow us to move labels out of line, while still optimizing for size. "
Quoted 8 lines
OTOH making a difficult optimization (
HAVE_ARCH_JUMP_LABEL) dependent on compiler flags is really asking for trouble.So how about enabling it unconditionally, and just chalk up the cost under
CC_OPTIMIZE_FOR_SIZEas one of the costs it already has? This also has the advantage that future compilers can improve things without having to wait for yet another kernel patch that re-enablesHAVE_ARCH_JUMP_LABEL.
Agreed,
CC_OPTIMIZE_FOR_SIZE does not mean OPTIMIZE_FOR_PERFORMANCE. Although people have argued that with smaller size you gain better cache performance. I've noticed that the general case is that optimizing for size has decreased performance (although I have not done any official benchmarks, just my own personal observations).
I thought you may have had that there because OPTIMIZE_FOR_SIZE actually broke the code (as some gcc compilers do for function graph tracer). If its just a "we don't perform better with this set". Then get rid of it.
Thanks,
-- Steve
Re: [PATCH 08/10] jump label v11: x86 support
Quoted 29 lines · Steven Rostedt
Quoted 10 lines · Ingo Molnar
Quoted 9 lines
Quoted 1 line
From the documentation patch:
" The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When
CC_OPTIMIZE_FOR_SIZEis set, gcc does not always out of line the not taken label path in the same way that the "ifunlikely()" paths are made out of line. Thus, withCC_OPTIMIZE_FOR_SIZEset, this optimization is not always optimal. This may be solved in subsequent gcc versions, that allow us to move labels out of line, while still optimizing for size. "Quoted 8 lines
OTOH making a difficult optimization (
HAVE_ARCH_JUMP_LABEL) dependent on compiler flags is really asking for trouble.So how about enabling it unconditionally, and just chalk up the cost under
CC_OPTIMIZE_FOR_SIZEas one of the costs it already has? This also has the advantage that future compilers can improve things without having to wait for yet another kernel patch that re-enablesHAVE_ARCH_JUMP_LABEL.Agreed,
CC_OPTIMIZE_FOR_SIZEdoes not meanOPTIMIZE_FOR_PERFORMANCE. Although people have argued that with smaller size you gain better cache performance. I've noticed that the general case is that optimizing for size has decreased performance (although I have not done any official benchmarks, just my own personal observations).
Quoted 4 lines
I thought you may have had that there because
OPTIMIZE_FOR_SIZEactually broke the code (as some gcc compilers do for function graph tracer). If its just a "we don't perform better with this set". Then get rid of it.
No, OPTIMIZE_FOR_SIZE doesn't break anything, its just that the 'dormant' or the cold code path, isn't moved out of line.
I believe the current approach will eventually lead us to an optimal solution, (single no-op in the fast path, with ability to jump patch to out-of-line cold code), even if its not quite there yet under this compiler flag. And the inclusion of this code will provide more of an impetus for gcc to make further optimizations.
thanks,
-Jason
Re: [PATCH 08/10] jump label v11: x86 support
In reply to Jason BaronQuoted 65 lines · Jason Baron
add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.
Signed-off-by Jason Baron <[email protected]>--- arch/x86/Kconfig | 1 + arch/x86/include/asm/jump_label.h | 47 +++++++++++++++++++++++++++++++++++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/jump_label.c | 49 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/module.c | 3 ++ 5 files changed, 101 insertions(+), 1 deletions(-) create mode 100644 arch/x86/include/asm/jump_label.h create mode 100644 arch/x86/kernel/jump_label.carch/x86/Kconfig +1
-0@@ -60,6 +60,7 @@ config X86 select ANON_INODES select HAVE_ARCH_KMEMCHECK select HAVE_USER_RETURN_NOTIFIER + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE config INSTRUCTION_DECODER def_bool (KPROBES || PERF_EVENTS)arch/x86/include/asm/jump_label.h new file +32
-0@@ -0,0 +1,47 @@ +#ifndef _ASM_X86_JUMP_LABEL_H +#define _ASM_X86_JUMP_LABEL_H + +#ifdef __KERNEL__ + +#include <linux/types.h> +#include <asm/nops.h> + +#define JUMP_LABEL_NOP_SIZE 5 + +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t" + +# define JUMP_LABEL(key, label) \ + do { \ + asm goto("1:" \ + JUMP_LABEL_INITIAL_NOP \ + ".pushsection __jump_table, \"a\" \n\t"\ + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \ + ".popsection \n\t" \ + : : "i" (key) : : label); \ + } while (0) + +#endif /* __KERNEL__ */ + +#ifdef CONFIG_X86_64 + +typedef u64 jump_label_t; + +struct jump_entry { + jump_label_t code; + jump_label_t target; + jump_label_t key;
arch/x86/include/asm/jump_label.h line 42
Quoted 10 lines
arch/x86/include/asm/jump_label.h new file +10
-0+}; + +#else + +typedef u32 jump_label_t; + +struct jump_entry { + jump_label_t code; + jump_label_t target; + jump_label_t key;
The format is the same, why not just put the typedef within the #ifdef?
Quoted 24 lines
arch/x86/include/asm/jump_label.h new file +5
-0+}; + +#endif + +#endif arch/x86/kernel/Makefile +1
-1@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o := n obj-y := process_$(BITS).o signal.o entry_$(BITS).o obj-y += traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o obj-y += time.o ioport.o ldt.o dumpstack.o -obj-y += setup.o x86_init.o i8259.o irqinit.o +obj-y += setup.o x86_init.o i8259.o irqinit.o jump_label.o obj-$(CONFIG_X86_VISWS) += visws_quirks.o obj-$(CONFIG_X86_32) += probe_roms_32.o obj-$(CONFIG_X86_32) += sys_i386_32.o i386_ksyms_32.oarch/x86/kernel/jump_label.c new file +0
-0
arch/x86/kernel/jump_label.c line 4
Quoted 4 lines
arch/x86/kernel/jump_label.c new file +4
-0@@ -0,0 +1,49 @@ +/* +* jump label x86 support +* +* Copyright (C) 2009 Jason Baron <[email protected]>
Year.
[tip:perf/core] jump label: x86 support
In reply to Jason BaronCommit-ID: d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e Gitweb: http://git.kernel.org/tip/d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e Author: Jason Baron <[email protected]> AuthorDate: Fri, 17 Sep 2010 11:09:22 -0400 Committer: Steven Rostedt <[email protected]> CommitDate: Wed, 22 Sep 2010 16:33:03 -0400
jump label: x86 support
add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. Hopefully, it wouldn't be too painful for other archs.
LKML-Reference: <f838f49f40fbea0254036194be66dc48b598dcea.1284733808.git.jbaron@redhat.com>
[ cleaned up some formatting ]
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/jump_label.h | 47 ++++++++++++++++++++++++++++++++++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/jump_label.c | 50 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/module.c | 3 ++ 5 files changed, 102 insertions(+), 1 deletions(-)
arch/x86/Kconfig +1 -0
@@ -59,6 +59,7 @@ config X86 select ANON_INODES select HAVE_ARCH_KMEMCHECK select HAVE_USER_RETURN_NOTIFIER + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE config INSTRUCTION_DECODER def_bool (KPROBES || PERF_EVENTS)
arch/x86/include/asm/jump_label.h new file +47 -0
47 lines not on this page · raw message
arch/x86/kernel/Makefile +1 -1
@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o := n obj-y := process_$(BITS).o signal.o entry_$(BITS).o obj-y += traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o obj-y += time.o ioport.o ldt.o dumpstack.o -obj-y += setup.o x86_init.o i8259.o irqinit.o +obj-y += setup.o x86_init.o i8259.o irqinit.o jump_label.o obj-$(CONFIG_X86_VISWS) += visws_quirks.o obj-$(CONFIG_X86_32) += probe_roms_32.o obj-$(CONFIG_X86_32) += sys_i386_32.o i386_ksyms_32.o
arch/x86/kernel/jump_label.c new file +50 -0
50 lines not on this page · raw message
arch/x86/kernel/module.c +3 -0
@@ -239,6 +239,9 @@ int module_finalize(const Elf_Ehdr *hdr,apply_paravirt (pseg, pseg + para->sh_size); } +/* make jump label nops */ +jump_label_apply_nops (me); +return module_bug_finalize (hdr, sechdrs, me); }
H. Peter Anvin
reviewerRe: [tip:perf/core] jump label: x86 support
Quoted 32 lines · tip-bot for Jason Baron
Commit-ID: d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e Gitweb: http://git.kernel.org/tip/d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e Author: Jason Baron <[email protected]> AuthorDate: Fri, 17 Sep 2010 11:09:22 -0400 Committer: Steven Rostedt <[email protected]> CommitDate: Wed, 22 Sep 2010 16:33:03 -0400jump label: x86 support
add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. Hopefully, it wouldn't be too painful for other archs.
Signed-off-by Jason Baron <[email protected]>LKML-Reference: <f838f49f40fbea0254036194be66dc48b598dcea.1284733808.git.jbaron@redhat.com>
[ cleaned up some formatting ]
Signed-off-by Steven Rostedt <[email protected]>--- arch/x86/Kconfig | 1 + arch/x86/include/asm/jump_label.h | 47 ++++++++++++++++++++++++++++++++++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/jump_label.c | 50 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/module.c | 3 ++ 5 files changed, 102 insertions(+), 1 deletions(-)arch/x86/Kconfig +0
-0
arch/x86/Kconfig config X86 · line 65
Quoted 7 lines
arch/x86/Kconfig +1
-0@@ -59,6 +59,7 @@ config X86 select ANON_INODES select HAVE_ARCH_KMEMCHECK select HAVE_USER_RETURN_NOTIFIER + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE config INSTRUCTION_DECODER def_bool (KPROBES || PERF_EVENTS)
What the heck? I had NAKed at least this chunk already...
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Re: [tip:perf/core] jump label: x86 support
Quoted 38 lines · H. Peter Anvin
Quoted 37 lines · tip-bot for Jason Baron
Commit-ID: d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e Gitweb: http://git.kernel.org/tip/d9f5ab7b1c0a520867af389bab5d5fcdbd0e407e Author: Jason Baron <[email protected]> AuthorDate: Fri, 17 Sep 2010 11:09:22 -0400 Committer: Steven Rostedt <[email protected]> CommitDate: Wed, 22 Sep 2010 16:33:03 -0400jump label: x86 support
add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. Hopefully, it wouldn't be too painful for other archs.
Signed-off-by Jason Baron <[email protected]>LKML-Reference: <f838f49f40fbea0254036194be66dc48b598dcea.1284733808.git.jbaron@redhat.com>
[ cleaned up some formatting ]
Signed-off-by Steven Rostedt <[email protected]>--- arch/x86/Kconfig | 1 + arch/x86/include/asm/jump_label.h | 47 ++++++++++++++++++++++++++++++++++ arch/x86/kernel/Makefile | 2 +- arch/x86/kernel/jump_label.c | 50 +++++++++++++++++++++++++++++++++++++ arch/x86/kernel/module.c | 3 ++ 5 files changed, 102 insertions(+), 1 deletions(-)arch/x86/Kconfig +1
-0@@ -59,6 +59,7 @@ config X86 select ANON_INODES select HAVE_ARCH_KMEMCHECK select HAVE_USER_RETURN_NOTIFIER + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
Quoted 7 lines
arch/x86/Kconfig config X86 · line 65
Quoted 2 lines
arch/x86/Kconfig +0
-0config INSTRUCTION_DECODER def_bool (KPROBES || PERF_EVENTS)What the heck? I had NAKed at least this chunk already...
-hpa
The !CC_OPTIMIZE_FOR_SIZE condition is subsequently removed by a follow-on patch as part of the commit series.
thanks,
-Jason
H. Peter Anvin
reviewerRe: [tip:perf/core] jump label: x86 support
Quoted 4 lines · Jason Baron
The !CC_OPTIMIZE_FOR_SIZE condition is subsequently removed by a follow-on patch as part of the commit series.
Ah, okay. No problem then.
-hpa
Steven Rostedt
maintainerRe: [tip:perf/core] jump label: x86 support
Quoted 7 lines · H. Peter Anvin
Quoted 4 lines · Jason Baron
The !CC_OPTIMIZE_FOR_SIZE condition is subsequently removed by a follow-on patch as part of the commit series.
Ah, okay. No problem then.
Sorry, I should have Cc'd you on that patch.
-- Steve
Peter Zijlstra
maintainerRe: [tip:perf/core] jump label: x86 support
In reply to tip-bot for Jason BaronQuoted 7 lines · tip-bot for Jason Baron
arch/x86/kernel/jump_label.c new file +7
-0+void arch_jump_label_transform (struct jump_entry *entry, +enum jump_label_type type) +{ +union jump_code_union code; + +if (type ==JUMP_LABEL_ENABLE ) { + code.jump =0xe9 ;
arch/x86/kernel/jump_label.c line 43
Quoted 10 lines
arch/x86/kernel/jump_label.c new file +10
-0+ code.offset = entry->target - + (entry->code +JUMP_LABEL_NOP_SIZE ); + }else +memcpy (&code, ideal_nop5,JUMP_LABEL_NOP_SIZE ); +get_online_cpus (); +mutex_lock (&text_mutex); +text_poke_smp ((void *)entry->code, &code,JUMP_LABEL_NOP_SIZE ); +mutex_unlock (&text_mutex); +put_online_cpus (); +}
hpa, mathieu, what's the status of stop_machine less text poking? Because the above basically means we have to disable architecture jump_label support for -rt.
Mathieu Desnoyers
reviewerRe: [tip:perf/core] jump label: x86 support
Quoted 14 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 13 lines · tip-bot for Jason Baron
arch/x86/kernel/jump_label.c new file +13
-0+void arch_jump_label_transform (struct jump_entry *entry, +enum jump_label_type type) +{ +union jump_code_union code; + +if (type ==JUMP_LABEL_ENABLE ) { + code.jump =0xe9 ; + code.offset = entry->target - + (entry->code +JUMP_LABEL_NOP_SIZE ); + }else +memcpy (&code, ideal_nop5,JUMP_LABEL_NOP_SIZE ); +get_online_cpus (); +mutex_lock (&text_mutex);
Quoted 8 lines
arch/x86/kernel/jump_label.c line 43
Quoted 4 lines
arch/x86/kernel/jump_label.c new file +4
-0+text_poke_smp ((void *)entry->code, &code,JUMP_LABEL_NOP_SIZE ); +mutex_unlock (&text_mutex); +put_online_cpus (); +}hpa, mathieu, what's the status of
stop_machineless text poking? Because the above basically means we have to disable architecturejump_labelsupport for -rt.
hpa got an unofficial answer from Intel OTC, that states the stop_machine-less text poking procedure is valid.
http://lkml.org/lkml/2010/1/12/300
We are still waiting for the "official" blessing though.
Thanks,
Mathieu
Signature · 4 lines
Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com
PATCH 9/10 jump label 11: add sparc64 support
In reply to Jason Baronadd sparc64 support
--- arch/sparc/Kconfig | 1 + arch/sparc/include/asm/jump_label.h | 32 ++++++++++++++++++++++++ arch/sparc/kernel/Makefile | 2 + arch/sparc/kernel/jump_label.c | 46 +++++++++++++++++++++++++++++++++++ arch/sparc/kernel/module.c | 6 ++++ 5 files changed, 87 insertions(+), 0 deletions(-) create mode 100644 arch/sparc/include/asm/jump_label.h create mode 100644 arch/sparc/kernel/jump_label.c
arch/sparc/Kconfig +1 -0
@@ -30,6 +30,7 @@ config SPARC select PERF_USE_VMALLOC select HAVE_DMA_ATTRS select HAVE_DMA_API_DEBUG + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE config SPARC32 def_bool !64BIT
arch/sparc/include/asm/jump_label.h new file +32 -0
@@ -0,0 +1,32 @@ +#ifndef _ASM_SPARC_JUMP_LABEL_H +#define _ASM_SPARC_JUMP_LABEL_H + +#ifdef __KERNEL__ + +#include <linux/types.h> +#include <asm/system.h> + +#define JUMP_LABEL_NOP_SIZE 4 + +#define JUMP_LABEL (key, label) \ + do { \ + asm goto("1:\n\t" \ + "nop\n\t" \ + "nop\n\t" \ + ".pushsection __jump_table, \"a\"\n\t"\ + ".word 1b, %l[" #label "], %c0\n\t" \ + ".popsection \n\t" \ + : : "i" (key) : : label);\ + } while (0) + +#endif /* __KERNEL__ */ + +typedef u32 jump_label_t ; + +struct jump_entry { +jump_label_t code; +jump_label_t target; +jump_label_t key; +}; + +#endif
arch/sparc/kernel/Makefile +2 -0
@@ -119,3 +119,5 @@ obj-$(CONFIG_COMPAT) += $(audit--y) pc--$(CONFIG_PERF_EVENTS) := perf_event.o obj-$(CONFIG_SPARC64) += $(pc--y) + +obj-$(CONFIG_SPARC64) += jump_label.o
arch/sparc/kernel/jump_label.c new file +46 -0
46 lines not on this page · raw message
arch/sparc/kernel/module.c +6 -0
@@ -18,6 +18,9 @@#include <asm/spitfire.h> #ifdef CONFIG_SPARC64 + +#include <linux/jump_label.h> +static void *module_map (unsigned long size) {struct vm_struct *area;
@@ -227,6 +230,9 @@ int module_finalize(const Elf_Ehdr *hdr,const Elf_Shdr *sechdrs, struct module *me) { +/* make jump label nops */ +jump_label_apply_nops (me ); +/* Cheetah's I-cache is fully coherent. */ if (tlb_type == spitfire) {unsigned long va;
Signature · 1 line
1.7.1
Steven Rostedt
maintainerRe: [PATCH 09/10] jump label 11: add sparc64 support
Jason, did you author this patch or did David?
-- Steve
Quoted 150 lines · Jason Baron
add sparc64 support
Signed-off-by David S. Miller <[email protected]>Signed-off-by Jason Baron <[email protected]>--- arch/sparc/Kconfig | 1 + arch/sparc/include/asm/jump_label.h | 32 ++++++++++++++++++++++++ arch/sparc/kernel/Makefile | 2 + arch/sparc/kernel/jump_label.c | 46 +++++++++++++++++++++++++++++++++++ arch/sparc/kernel/module.c | 6 ++++ 5 files changed, 87 insertions(+), 0 deletions(-) create mode 100644 arch/sparc/include/asm/jump_label.h create mode 100644 arch/sparc/kernel/jump_label.carch/sparc/Kconfig +1
-0@@ -30,6 +30,7 @@ config SPARC select PERF_USE_VMALLOC select HAVE_DMA_ATTRS select HAVE_DMA_API_DEBUG + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE config SPARC32 def_bool !64BITarch/sparc/include/asm/jump_label.h new file +32
-0@@ -0,0 +1,32 @@ +#ifndef _ASM_SPARC_JUMP_LABEL_H +#define _ASM_SPARC_JUMP_LABEL_H + +#ifdef __KERNEL__ + +#include <linux/types.h> +#include <asm/system.h> + +#define JUMP_LABEL_NOP_SIZE 4 + +#define JUMP_LABEL (key, label) \ + do { \ + asm goto("1:\n\t" \ + "nop\n\t" \ + "nop\n\t" \ + ".pushsection __jump_table, \"a\"\n\t"\ + ".word 1b, %l[" #label "], %c0\n\t" \ + ".popsection \n\t" \ + : : "i" (key) : : label);\ + } while (0) + +#endif /* __KERNEL__ */ + +typedef u32 jump_label_t ; + +struct jump_entry { +jump_label_t code; +jump_label_t target; +jump_label_t key; +}; + +#endif arch/sparc/kernel/Makefile +2
-0@@ -119,3 +119,5 @@ obj-$(CONFIG_COMPAT) += $(audit--y) pc--$(CONFIG_PERF_EVENTS) := perf_event.o obj-$(CONFIG_SPARC64) += $(pc--y) + +obj-$(CONFIG_SPARC64) += jump_label.oarch/sparc/kernel/jump_label.c new file +46
-0@@ -0,0 +1,46 @@ +#include <linux/kernel.h> +#include <linux/types.h> +#include <linux/mutex.h> +#include <linux/cpu.h> + +#include <linux/jump_label.h> +#include <linux/memory.h> + +#ifdef HAVE_JUMP_LABEL + +void arch_jump_label_transform (struct jump_entry *entry,enum jump_label_type type) +{ +u32 val; +u32 *insn = (u32 *) (unsigned long ) entry->code; + +if (type ==JUMP_LABEL_ENABLE ) { +s32 off = (s32 )entry->target - (s32 )entry->code; + +#ifdef CONFIG_SPARC64 +/* ba,pt %xcc, . + (off << 2) */ + val =0x10680000 | ((u32 ) off >>2 ); +#else +/* ba . + (off << 2) */ + val =0x10800000 | ((u32 ) off >>2 ); +#endif + }else { + val =0x01000000 ; + } + +get_online_cpus (); +mutex_lock (&text_mutex); + *insn = val; +flushi (insn); +mutex_unlock (&text_mutex); +put_online_cpus (); +} + +void arch_jump_label_text_poke_early (jump_label_t addr) +{ +u32 *insn_p = (u32 *) (unsigned long ) addr; + + *insn_p =0x01000000 ; +flushi (insn_p); +} + +#endif arch/sparc/kernel/module.c +6
-0@@ -18,6 +18,9 @@#include <asm/spitfire.h> #ifdef CONFIG_SPARC64 + +#include <linux/jump_label.h> +static void *module_map (unsigned long size) {struct vm_struct *area;@@ -227,6 +230,9 @@ int module_finalize(const Elf_Ehdr *hdr,const Elf_Shdr *sechdrs, struct module *me) { +/* make jump label nops */ +jump_label_apply_nops (me ); +/* Cheetah's I-cache is fully coherent. */ if (tlb_type == spitfire) {unsigned long va;
David Miller
maintainerRe: [PATCH 09/10] jump label 11: add sparc64 support
From: Steven Rostedt <[email protected]>
Date: Mon, 20 Sep 2010 18:25:46 -0400
Quoted 2 lines
Jason, did you author this patch or did David?
I did.
Jason when you post patches other people have authored, you should add a:
From: Mr. Author <[email protected]>
to the top of your commit message.
This allows automated tools that pull patches into GIT to set the author correctly when it is not you.
Thanks.
Steven Rostedt
maintainerRe: [PATCH 09/10] jump label 11: add sparc64 support
Quoted 11 lines · David Miller
From: Steven Rostedt <[email protected]>
Date: Mon, 20 Sep 2010 18:25:46 -0400Quoted 2 lines
Jason, did you author this patch or did David?
I did.
Jason when you post patches other people have authored, you should add a:
Quoted 8 lines
From: Mr. Author <[email protected]>
to the top of your commit message.
This allows automated tools that pull patches into GIT to set the author correctly when it is not you.
Thanks.
Thanks, I figured you did when I saw your SoB. I'll add your From line manually, when I pull it.
-- Steve
Steven Rostedt
maintainerRe: [PATCH 09/10] jump label 11: add sparc64 support
In reply to Jason Baronarch/sparc/Kconfig config SPARC · line 34
Quoted 10 lines · Jason Baron
arch/sparc/Kconfig +1
-0@@ -30,6 +30,7 @@ config SPARC select PERF_USE_VMALLOC select HAVE_DMA_ATTRS select HAVE_DMA_API_DEBUG + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
David,
If you look at the thread for patch #8, do you still want that
" if !CC_OPTIMIZE_FOR_SIZE"?
-- Steve
David Miller
maintainerRe: [PATCH 09/10] jump label 11: add sparc64 support
From: Steven Rostedt <[email protected]>
Date: Tue, 21 Sep 2010 11:37:33 -0400
Quoted 15 lines · Jason Baron
arch/sparc/Kconfig config SPARC · line 34
Quoted 11 lines
arch/sparc/Kconfig +1
-0@@ -30,6 +30,7 @@ config SPARC select PERF_USE_VMALLOC select HAVE_DMA_ATTRS select HAVE_DMA_API_DEBUG + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZEDavid,
Quoted 2 lines
If you look at the thread for patch #8, do you still want that
" if !CC_OPTIMIZE_FOR_SIZE"?
No :-)
Steven Rostedt
maintainerRe: [PATCH 09/10] jump label 11: add sparc64 support
Quoted 18 lines · David Miller
From: Steven Rostedt <[email protected]>
Date: Tue, 21 Sep 2010 11:37:33 -0400Quoted 15 lines · Jason Baron
arch/sparc/Kconfig config SPARC · line 34
Quoted 11 lines
arch/sparc/Kconfig +1
-0@@ -30,6 +30,7 @@ config SPARC select PERF_USE_VMALLOC select HAVE_DMA_ATTRS select HAVE_DMA_API_DEBUG + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZEDavid,
Quoted 4 lines
Quoted 2 lines
If you look at the thread for patch #8, do you still want that
" if !CC_OPTIMIZE_FOR_SIZE"?No :-)
I'll be sending a pull request to Ingo soon. I'll add this change with your Acked-by.
Thanks,
-- Steve
tip-bot for David S. Miller
maintainer[tip:perf/core] jump label: Add sparc64 support
In reply to Jason BaronCommit-ID: dff9d3c215251022dd8bb3823c9f75edb4b63fe9 Gitweb: http://git.kernel.org/tip/dff9d3c215251022dd8bb3823c9f75edb4b63fe9 Author: David S. Miller <[email protected]> AuthorDate: Fri, 17 Sep 2010 11:09:25 -0400 Committer: Steven Rostedt <[email protected]> CommitDate: Wed, 22 Sep 2010 16:35:09 -0400
jump label: Add sparc64 support
Add jump label support for sparc64.
LKML-Reference: <3b5b071fcdb2afb7f67cacecfa78b14c740278a7.1284733808.git.jbaron@redhat.com>
[ cleaned up some formatting ]
--- arch/sparc/Kconfig | 1 + arch/sparc/include/asm/jump_label.h | 32 +++++++++++++++++++++++ arch/sparc/kernel/Makefile | 2 + arch/sparc/kernel/jump_label.c | 47 +++++++++++++++++++++++++++++++++++ arch/sparc/kernel/module.c | 6 ++++ 5 files changed, 88 insertions(+), 0 deletions(-)
arch/sparc/Kconfig +1 -0
@@ -30,6 +30,7 @@ config SPARC select PERF_USE_VMALLOC select HAVE_DMA_ATTRS select HAVE_DMA_API_DEBUG + select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE config SPARC32 def_bool !64BIT
arch/sparc/include/asm/jump_label.h new file +32 -0
@@ -0,0 +1,32 @@ +#ifndef _ASM_SPARC_JUMP_LABEL_H +#define _ASM_SPARC_JUMP_LABEL_H + +#ifdef __KERNEL__ + +#include <linux/types.h> +#include <asm/system.h> + +#define JUMP_LABEL_NOP_SIZE 4 + +#define JUMP_LABEL (key, label) \ + do { \ + asm goto("1:\n\t" \ + "nop\n\t" \ + "nop\n\t" \ + ".pushsection __jump_table, \"a\"\n\t"\ + ".word 1b, %l[" #label "], %c0\n\t" \ + ".popsection \n\t" \ + : : "i" (key) : : label);\ + } while (0) + +#endif /* __KERNEL__ */ + +typedef u32 jump_label_t ; + +struct jump_entry { +jump_label_t code; +jump_label_t target; +jump_label_t key; +}; + +#endif
arch/sparc/kernel/Makefile +2 -0
@@ -119,3 +119,5 @@ obj-$(CONFIG_COMPAT) += $(audit--y) pc--$(CONFIG_PERF_EVENTS) := perf_event.o obj-$(CONFIG_SPARC64) += $(pc--y) + +obj-$(CONFIG_SPARC64) += jump_label.o
arch/sparc/kernel/jump_label.c new file +47 -0
47 lines not on this page · raw message
arch/sparc/kernel/module.c +6 -0
@@ -18,6 +18,9 @@#include <asm/spitfire.h> #ifdef CONFIG_SPARC64 + +#include <linux/jump_label.h> +static void *module_map (unsigned long size) {struct vm_struct *area;
@@ -227,6 +230,9 @@ int module_finalize(const Elf_Ehdr *hdr,const Elf_Shdr *sechdrs, struct module *me) { +/* make jump label nops */ +jump_label_apply_nops (me ); +/* Cheetah's I-cache is fully coherent. */ if (tlb_type == spitfire) {unsigned long va;
PATCH 10/10 jump label v11: add docs
In reply to Jason BaronAdd jump label docs as: Documentation/jump-label.txt
--- Documentation/jump-label.txt | 148 ++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 148 insertions(+), 0 deletions(-) create mode 100644 Documentation/jump-label.txt
Documentation/jump-label.txt new file +148 -0
@@ -0,0 +1,148 @@ + Jump Label + ---------- + +By: Jason Baron <[email protected]> + + +1) motivation + + +Currently, tracepoints are implemented using a conditional. The conditional +check requires checking a global variable for each tracepoint. Although, +the overhead of this check is small, it increases under memory pressure. As we +increase the number of tracepoints in the kernel this may become more of an +issue. In addition, tracepoints are often dormant (disabled), and provide no +direct kernel functionality. Thus, it is highly desirable to reduce their +impact as much as possible. Although tracepoints are the original motivation +for this work, other kernel code paths should be able to make use of the jump +label optimization. + + +2) jump label description/usage + + +gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label. +http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html + +Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as +follows (x86): + +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t" + +# define JUMP_LABEL(key, label) \ + do { \ + asm goto("1:" \ + JUMP_LABEL_INITIAL_NOP \ + ".pushsection __jump_table, \"a\" \n\t"\ + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \ + ".popsection \n\t" \ + : : "i" (key) : : label); \ + } while (0) + + +For architectures that have not yet introduced jump label support its simply: + +#define JUMP_LABEL(key, label) \ + if (unlikely(*key)) \ + goto label; + +which then can be used as: + + .... + JUMP_LABEL(trace_name, trace_label, jump_enabled); + printk("not doing tracing\n"); + return; +trace_label: + printk("doing tracing: %d\n", file); + .... + +The 'key' argument is thus a pointer to a conditional argument that can be used +if the optimization is not enabled. Otherwise, this address serves as a unique +key to identify the particular instance of the jump label. + +Thus, when tracing is disabled, we simply have a no-op followed by a jump around +the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a +'jump_table' which has the following format: + +[instruction address] [jump target] [tracepoint key] + +Thus, to enable a tracepoint, we simply patch the 'instruction address' with +a jump to the 'jump target'. + +The call to enable a jump label is: enable_jump_label(key); to disable: +disable_jump_label(key); + + +3) architecture interface + + +There are a few functions and macros which arches must implement in order to +take advantage of this optimization. As previously mentioned, if there is no +architecture support we simply fall back to a traditional, load, test, and +jump sequence. + +* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support + +* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h + +* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h + +* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) + see: arch/x86/kernel/jump_label.c + +* void arch_jump_label_text_poke_early(jump_label_t addr) + see: arch/x86/kernel/jump_label.c + +* finally add a definition for "struct jump_entry". + see: arch/x86/include/asm/jump_label.h + + +4) Jump label analysis (x86) + + +I've tested the performance of using 'get_cycles()' calls around the +tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages): + + idle after tbench run + ---- ---------------- +old code 32 88 +new code 2 4 + + +The performance improvement can be reproduced reliably on both Intel and AMD +hardware. + +In terms of code analysis the current code for the disabled case is a 'cmpl' +followed by a 'je' around the tracepoint code. so: + +cmpl - 83 3d 0e 77 87 00 00 - 7 bytes +je - 74 3e - 2 bytes + +total of 9 instruction bytes. + +The new code is a 'nopl' followed by a 'jmp'. Thus: + +nopl - 0f 1f 44 00 00 - 5 bytes +jmp - eb 3e - 2 bytes + +total of 7 instruction bytes. + +So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint. + +The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is +set, gcc does not always out of line the not taken label path in the same way +that the "if unlikely()" paths are made out of line. Thus, with +CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be +solved in subsequent gcc versions, that allow us to move labels out of line, +while still optimizing for size. + + +5) Acknowledgments + + +Thanks to Roland McGrath and Richard Henderson for helping come up with the +initial 'asm goto' and jump label design. + +Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this +issue, and outlining the requirements of a solution. Mathieu also implemened a +solution in the form of the "Immediate Values" work.
Signature · 1 line
1.7.1
Re: [PATCH 10/10] jump label v11: add docs
Quoted 14 lines · * Jason Baron ([email protected]) wrote:
Add jump label docs as:
Documentation/jump-label.txtSigned-off-by Jason Baron <[email protected]>--- Documentation/jump-label.txt | 148 ++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 148 insertions(+), 0 deletions(-) create mode 100644 Documentation/jump-label.txtDocumentation/jump-label.txt new file +0
-0
Documentation/jump-label.txt line 7
Quoted 7 lines
Documentation/jump-label.txt new file +7
-0@@ -0,0 +1,148 @@ + Jump Label + ---------- + +By: Jason Baron <[email protected]> + + +1) motivation
You might want to start your titles with capital letters (small nit).
Documentation/jump-label.txt line 12
Quoted 5 lines
Documentation/jump-label.txt new file +5
-0+ + +Currently, tracepoints are implemented using a conditional. The conditional +check requires checking a global variable for each tracepoint. Although, +the overhead of this check is small, it increases under memory pressure. As we
I wonder if it would be worthwhile to spell out the difference between "memory pressure" in terms of memory bandwidth (which is what we care about here) and in terms of "memory space available". I just don't want people to get mixed up between the two when they read this text. What we care a lot about here is just memory bandwidth, which is impacted by cache pressure.
Documentation/jump-label.txt line 21
Quoted 9 lines
Documentation/jump-label.txt new file +9
-0+increase the number of tracepoints in the kernel this may become more of an +issue. In addition, tracepoints are often dormant (disabled), and provide no +direct kernel functionality. Thus, it is highly desirable to reduce their +impact as much as possible. Although tracepoints are the original motivation +for this work, other kernel code paths should be able to make use of the jump +label optimization. + + +2) jump label description/usage
Start with capital letter here too.
Quoted 45 lines
Documentation/jump-label.txt new file +45
-0+ + +gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label. +http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html + +Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as +follows (x86): + +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t" + +# define JUMP_LABEL(key, label) \ + do { \ + asm goto("1:" \ + JUMP_LABEL_INITIAL_NOP \ + ".pushsection __jump_table, \"a\" \n\t"\ + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \ + ".popsection \n\t" \ + : : "i" (key) : : label); \ + } while (0) + + +For architectures that have not yet introduced jump label support its simply: + +#define JUMP_LABEL(key, label) \ + if (unlikely(*key)) \ + goto label; + +which then can be used as: + + .... + JUMP_LABEL(trace_name, trace_label, jump_enabled); + printk("not doing tracing\n"); + return; +trace_label: + printk("doing tracing: %d\n", file); + .... + +The 'key' argument is thus a pointer to a conditional argument that can be used +if the optimization is not enabled. Otherwise, this address serves as a unique +key to identify the particular instance of the jump label. + +Thus, when tracing is disabled, we simply have a no-op followed by a jump around +the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a +'jump_table' which has the following format: +
Documentation/jump-label.txt line 76
Quoted 10 lines
Documentation/jump-label.txt new file +10
-0+[instruction address] [jump target] [tracepoint key] + +Thus, to enable a tracepoint, we simply patch the 'instruction address' with +a jump to the 'jump target'. + +The call to enable a jump label is: enable_jump_label(key); to disable: +disable_jump_label(key); + + +3) architecture interface
cap.
Documentation/jump-label.txt line 79
Quoted 3 lines
Documentation/jump-label.txt new file +3
-0+ + +There are a few functions and macros which arches must implement in order to
arches -> architectures
Quoted 51 lines
Documentation/jump-label.txt new file +51
-0+take advantage of this optimization. As previously mentioned, if there is no +architecture support we simply fall back to a traditional, load, test, and +jump sequence. + +* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support + +* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h + +* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h + +* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) + see: arch/x86/kernel/jump_label.c + +* void arch_jump_label_text_poke_early(jump_label_t addr) + see: arch/x86/kernel/jump_label.c + +* finally add a definition for "struct jump_entry". + see: arch/x86/include/asm/jump_label.h + + +4) Jump label analysis (x86) + + +I've tested the performance of using 'get_cycles()' calls around the +tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages): + + idle after tbench run + ---- ---------------- +old code 32 88 +new code 2 4 + + +The performance improvement can be reproduced reliably on both Intel and AMD +hardware. + +In terms of code analysis the current code for the disabled case is a 'cmpl' +followed by a 'je' around the tracepoint code. so: + +cmpl - 83 3d 0e 77 87 00 00 - 7 bytes +je - 74 3e - 2 bytes + +total of 9 instruction bytes. + +The new code is a 'nopl' followed by a 'jmp'. Thus: + +nopl - 0f 1f 44 00 00 - 5 bytes +jmp - eb 3e - 2 bytes + +total of 7 instruction bytes. + +So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.
Documentation/jump-label.txt line 140
Quoted 10 lines
Documentation/jump-label.txt new file +10
-0+ +The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is +set, gcc does not always out of line the not taken label path in the same way +that the "if unlikely()" paths are made out of line. Thus, with +CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be +solved in subsequent gcc versions, that allow us to move labels out of line, +while still optimizing for size. + + +5) Acknowledgments
Acknowledgements (seems to be the usual way to spell it in Documentation/).
Documentation/jump-label.txt line 147
Quoted 7 lines
Documentation/jump-label.txt new file +7
-0+ + +Thanks to Roland McGrath and Richard Henderson for helping come up with the +initial 'asm goto' and jump label design. + +Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this +issue, and outlining the requirements of a solution. Mathieu also implemened a
implemened -> implemented
Thanks!
Mathieu
Quoted 4 lines
Documentation/jump-label.txt new file +1
-0+solution in the form of the "Immediate Values" work.Signature · 1 line
1.7.1
Signature · 4 lines
Mathieu Desnoyers Operating System Efficiency R&D Consultant EfficiOS Inc. http://www.efficios.com
Steven Rostedt
maintainerRe: [PATCH 10/10] jump label v11: add docs
Quoted 4 lines · Mathieu Desnoyers
Quoted 3 lines · * Jason Baron ([email protected]) wrote:
Add jump label docs as:
Documentation/jump-label.txtSigned-off-by Jason Baron <[email protected]>
Jason,
Could you post this patch again with any updates you want to take from Mathieu. No need to post the entire series, I'm pulling in the rest. Just this patch.
Thanks,
-- Steve
Quoted 194 lines
Quoted 11 lines
--- Documentation/jump-label.txt | 148 ++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 148 insertions(+), 0 deletions(-) create mode 100644 Documentation/jump-label.txtDocumentation/jump-label.txt new file +0
-0Documentation/jump-label.txt line 7
Quoted 7 lines
Documentation/jump-label.txt new file +7
-0@@ -0,0 +1,148 @@ + Jump Label + ---------- + +By: Jason Baron <[email protected]> + + +1) motivationYou might want to start your titles with capital letters (small nit).
Documentation/jump-label.txt line 12
Quoted 5 lines
Documentation/jump-label.txt new file +5
-0+ + +Currently, tracepoints are implemented using a conditional. The conditional +check requires checking a global variable for each tracepoint. Although, +the overhead of this check is small, it increases under memory pressure. As weI wonder if it would be worthwhile to spell out the difference between "memory pressure" in terms of memory bandwidth (which is what we care about here) and in terms of "memory space available". I just don't want people to get mixed up between the two when they read this text. What we care a lot about here is just memory bandwidth, which is impacted by cache pressure.
Documentation/jump-label.txt line 21
Quoted 9 lines
Documentation/jump-label.txt new file +9
-0+increase the number of tracepoints in the kernel this may become more of an +issue. In addition, tracepoints are often dormant (disabled), and provide no +direct kernel functionality. Thus, it is highly desirable to reduce their +impact as much as possible. Although tracepoints are the original motivation +for this work, other kernel code paths should be able to make use of the jump +label optimization. + + +2) jump label description/usageStart with capital letter here too.
Quoted 45 lines
Documentation/jump-label.txt new file +45
-0+ + +gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label. +http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html + +Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as +follows (x86): + +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t" + +# define JUMP_LABEL(key, label) \ + do { \ + asm goto("1:" \ + JUMP_LABEL_INITIAL_NOP \ + ".pushsection __jump_table, \"a\" \n\t"\ + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \ + ".popsection \n\t" \ + : : "i" (key) : : label); \ + } while (0) + + +For architectures that have not yet introduced jump label support its simply: + +#define JUMP_LABEL(key, label) \ + if (unlikely(*key)) \ + goto label; + +which then can be used as: + + .... + JUMP_LABEL(trace_name, trace_label, jump_enabled); + printk("not doing tracing\n"); + return; +trace_label: + printk("doing tracing: %d\n", file); + .... + +The 'key' argument is thus a pointer to a conditional argument that can be used +if the optimization is not enabled. Otherwise, this address serves as a unique +key to identify the particular instance of the jump label. + +Thus, when tracing is disabled, we simply have a no-op followed by a jump around +the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a +'jump_table' which has the following format: +Documentation/jump-label.txt line 76
Quoted 10 lines
Documentation/jump-label.txt new file +10
-0+[instruction address] [jump target] [tracepoint key] + +Thus, to enable a tracepoint, we simply patch the 'instruction address' with +a jump to the 'jump target'. + +The call to enable a jump label is: enable_jump_label(key); to disable: +disable_jump_label(key); + + +3) architecture interfacecap.
Documentation/jump-label.txt line 79
Quoted 3 lines
Documentation/jump-label.txt new file +3
-0+ + +There are a few functions and macros which arches must implement in order toarches -> architectures
Quoted 51 lines
Documentation/jump-label.txt new file +51
-0+take advantage of this optimization. As previously mentioned, if there is no +architecture support we simply fall back to a traditional, load, test, and +jump sequence. + +* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support + +* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h + +* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h + +* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) + see: arch/x86/kernel/jump_label.c + +* void arch_jump_label_text_poke_early(jump_label_t addr) + see: arch/x86/kernel/jump_label.c + +* finally add a definition for "struct jump_entry". + see: arch/x86/include/asm/jump_label.h + + +4) Jump label analysis (x86) + + +I've tested the performance of using 'get_cycles()' calls around the +tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages): + + idle after tbench run + ---- ---------------- +old code 32 88 +new code 2 4 + + +The performance improvement can be reproduced reliably on both Intel and AMD +hardware. + +In terms of code analysis the current code for the disabled case is a 'cmpl' +followed by a 'je' around the tracepoint code. so: + +cmpl - 83 3d 0e 77 87 00 00 - 7 bytes +je - 74 3e - 2 bytes + +total of 9 instruction bytes. + +The new code is a 'nopl' followed by a 'jmp'. Thus: + +nopl - 0f 1f 44 00 00 - 5 bytes +jmp - eb 3e - 2 bytes + +total of 7 instruction bytes. + +So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.Documentation/jump-label.txt line 140
Quoted 10 lines
Documentation/jump-label.txt new file +10
-0+ +The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is +set, gcc does not always out of line the not taken label path in the same way +that the "if unlikely()" paths are made out of line. Thus, with +CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be +solved in subsequent gcc versions, that allow us to move labels out of line, +while still optimizing for size. + + +5) AcknowledgmentsAcknowledgements (seems to be the usual way to spell it in
Documentation/).Documentation/jump-label.txt line 147
Quoted 7 lines
Documentation/jump-label.txt new file +7
-0+ + +Thanks to Roland McGrath and Richard Henderson for helping come up with the +initial 'asm goto' and jump label design. + +Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this +issue, and outlining the requirements of a solution. Mathieu also implemened aimplemened -> implemented
Thanks!
Mathieu
Quoted 4 lines
Documentation/jump-label.txt new file +1
-0+solution in the form of the "Immediate Values" work.Signature · 1 line
1.7.1
Re: [PATCH 10/10] jump label v11: add docs
Quoted 8 lines · Steven Rostedt
Quoted 4 lines · Mathieu Desnoyers
Quoted 3 lines · * Jason Baron ([email protected]) wrote:
Add jump label docs as:
Documentation/jump-label.txtSigned-off-by Jason Baron <[email protected]>Jason,
Quoted 8 lines
Could you post this patch again with any updates you want to take from Mathieu. No need to post the entire series, I'm pulling in the rest. Just this patch.
Thanks,
-- Steve
Hi Steve,
here's a re-post of the documentation patch. I've incorporated Mathieu's as well as Matt's feedback. Note that I've dropped the '!CC_OPTIMIZE_FOR_SIZE' optimize for size paragraph.
thanks,
-Jason
Add jump label docs as: Documentation/jump-label.txt
--- Documentation/jump-label.txt | 142 ++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 142 insertions(+), 0 deletions(-) create mode 100644 Documentation/jump-label.txt
Documentation/jump-label.txt new file +142 -0
@@ -0,0 +1,142 @@ + Jump Label + ---------- + +By: Jason Baron <[email protected]> + + +1) Motivation + + +Currently, tracepoints are implemented using a conditional. The conditional +check requires checking a global variable for each tracepoint. Although +the overhead of this check is small, it increases when the memory cache comes +under pressure (memory cache lines for these global variables may be shared +with other memory accesses). As we increase the number of tracepoints in the +kernel this overhead may become more of an issue. In addition, tracepoints are +often dormant (disabled) and provide no direct kernel functionality. Thus, it +is highly desirable to reduce their impact as much as possible. Although +tracepoints are the original motivation for this work, other kernel code paths +should be able to make use of the jump label optimization. + + +2) Jump label description/usage + + +gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label. +http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html + +Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as +follows (x86): + +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t" + +# define JUMP_LABEL(key, label) \ + do { \ + asm goto("1:" \ + JUMP_LABEL_INITIAL_NOP \ + ".pushsection __jump_table, \"a\" \n\t"\ + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \ + ".popsection \n\t" \ + : : "i" (key) : : label); \ + } while (0) + + +For architectures that have not yet introduced jump label support it's simply: + +#define JUMP_LABEL(key, label) \ + if (unlikely(*key)) \ + goto label; + +which then can be used as: + + .... + JUMP_LABEL(key, trace_label); + printk("not doing tracing\n"); + return; +trace_label: + printk("doing tracing: %d\n", file); + .... + +The 'key' argument is thus a pointer to a conditional argument that can be used +if the optimization is not enabled. Otherwise, this address serves as a unique +key to identify the particular instance of the jump label. + +Thus, when tracing is disabled, we simply have a no-op followed by a jump around +the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro produces a +'jump_table', which has the following format: + +[instruction address] [jump target] [key] + +Thus, to enable a tracepoint, we simply patch the 'instruction address' with +a jump to the 'jump target.' + +The calls to enable and disable a jump label are: enable_jump_label(key) and +disable_jump_label(key). + + +3) Architecture interface + + +There are a few functions and macros that architectures must implement in order +to take advantage of this optimization. As previously mentioned, if there is no +architecture support, we simply fall back to a traditional, load, test, and +jump sequence. + +* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support + +* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h + +* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h + +* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) + see: arch/x86/kernel/jump_label.c + +* void arch_jump_label_text_poke_early(jump_label_t addr) + see: arch/x86/kernel/jump_label.c + +* finally add a definition for "struct jump_entry". + see: arch/x86/include/asm/jump_label.h + + +4) Jump label analysis (x86) + + +I've tested the performance of using 'get_cycles()' calls around the +tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages): + + idle after tbench run + ---- ---------------- +old code 32 88 +new code 2 4 + + +The performance improvement can be reproduced reliably on both Intel and AMD +hardware. + +In terms of code analysis, the current code for the disabled case is a 'cmpl' +followed by a 'je' around the tracepoint code. So: + +cmpl - 83 3d 0e 77 87 00 00 - 7 bytes +je - 74 3e - 2 bytes + +total of 9 instruction bytes. + +The new code is a 'nopl' followed by a 'jmp'. Thus: + +nopl - 0f 1f 44 00 00 - 5 bytes +jmp - eb 3e - 2 bytes + +total of 7 instruction bytes. + +So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint. + + +5) Acknowledgements + + +Thanks to Roland McGrath and Richard Henderson for helping come up with the +initial 'asm goto' and jump label design. + +Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this +issue, and outlining the requirements of a solution. Mathieu also implemented a +solution in the form of the "Immediate Values" work.
Signature · 1 line
1.7.1
Re: [PATCH 10/10] jump label v11: add docs
In reply to Jason BaronDocumentation/jump-label.txt line 11
Quoted 4 lines · Jason Baron
Add jump label docs as:
Documentation/jump-label.txtDocumentation/jump-label.txt new file +2
-0+Currently, tracepoints are implemented using a conditional. The conditional +check requires checking a global variable for each tracepoint. Although,
No comma after although here.
Documentation/jump-label.txt line 13
Quoted 2 lines
Documentation/jump-label.txt new file +2
-0+the overhead of this check is small, it increases under memory pressure. As we +increase the number of tracepoints in the kernel this may become more of an
Comma after kernel is needed to separate the preposition, and I think the use of "this" in the sentence is unclear.
Quoted 1 line
+issue. In addition, tracepoints are often dormant (disabled), and provide no
Also, no comma before "and" because the second part is a subordinate clause.
Documentation/jump-label.txt line 43
Quoted 7 lines
Documentation/jump-label.txt new file +7
-0+direct kernel functionality. Thus, it is highly desirable to reduce their +impact as much as possible. Although tracepoints are the original motivation +for this work, other kernel code paths should be able to make use of the jump +label optimization. + + +For architectures that have not yet introduced jump label support its simply:
..."support, it's" ...
Documentation/jump-label.txt line 65
Quoted 4 lines
Documentation/jump-label.txt new file +4
-0+ +Thus, when tracing is disabled, we simply have a no-op followed by a jump around +the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a +'jump_table' which has the following format:
No comma after macro, but a comma is needed after jump_table because what follows is a nonrestrictive clause.
Documentation/jump-label.txt line 70
Quoted 4 lines
+[instruction address] [jump target] [tracepoint key]
Documentation/jump-label.txt new file +3
-0+ +Thus, to enable a tracepoint, we simply patch the 'instruction address' with +a jump to the 'jump target'.
Punctuation is suppose to go inside quotes (I know it's ugly and illogical).
Documentation/jump-label.txt line 73
Quoted 3 lines
Documentation/jump-label.txt new file +3
-0+ +The call to enable a jump label is: enable_jump_label(key); to disable: +disable_jump_label(key);
Hmm, maybe a better structure would be:
"The calls to enable and disable a jump label are: enable_jump_label(key) and disable_jump_label(key)."
Quoted 1 line
+There are a few functions and macros which arches must implement in order to
"That" should be used here because it is restrictive.
Documentation/jump-label.txt line 81
Quoted 2 lines
Documentation/jump-label.txt new file +2
-0+take advantage of this optimization. As previously mentioned, if there is no +architecture support we simply fall back to a traditional, load, test, and
Comma after support.
Review
Quoted 2 lines
(quoted diff) +2
-0+ +In terms of code analysis the current code for the disabled case is a 'cmpl'
Comma after analysis; it is a preposition.
Quoted 1 line
+followed by a 'je' around the tracepoint code. so:
Capitalize S in "so"
Documentation/jump-label.txt line 136
Quoted 7 lines
Documentation/jump-label.txt new file +7
-0+ + +The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is +set, gcc does not always out of line the not taken label path in the same way +that the "if unlikely()" paths are made out of line. Thus, with +CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be +solved in subsequent gcc versions, that allow us to move labels out of line,
"which" should be used here instead of that.
Otherwise, from a writing standpoint, it looks good.
-mfm
Re: [PATCH 10/10] jump label v11: add docs
In reply to Jason BaronQuoted 56 lines · Jason Baron
Add jump label docs as:
Documentation/jump-label.txtSigned-off-by Jason Baron <[email protected]>--- Documentation/jump-label.txt | 148 ++++++++++++++++++++++++++++++++++++++++++ 1 files changed, 148 insertions(+), 0 deletions(-) create mode 100644 Documentation/jump-label.txtDocumentation/jump-label.txt new file +42
-0@@ -0,0 +1,148 @@ + Jump Label + ---------- + +By: Jason Baron <[email protected]> + + +1) motivation + + +Currently, tracepoints are implemented using a conditional. The conditional +check requires checking a global variable for each tracepoint. Although, +the overhead of this check is small, it increases under memory pressure. As we +increase the number of tracepoints in the kernel this may become more of an +issue. In addition, tracepoints are often dormant (disabled), and provide no +direct kernel functionality. Thus, it is highly desirable to reduce their +impact as much as possible. Although tracepoints are the original motivation +for this work, other kernel code paths should be able to make use of the jump +label optimization. + + +2) jump label description/usage + + +gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label. +http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html + +Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as +follows (x86): + +# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t" + +# define JUMP_LABEL(key, label) \ + do { \ + asm goto("1:" \ + JUMP_LABEL_INITIAL_NOP \ + ".pushsection __jump_table, \"a\" \n\t"\ + _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \ + ".popsection \n\t" \ + : : "i" (key) : : label); \ + } while (0) + +
Documentation/jump-label.txt line 52
Quoted 10 lines
Documentation/jump-label.txt new file +10
-0+For architectures that have not yet introduced jump label support its simply: + +#define JUMP_LABEL(key, label) \ + if (unlikely(*key)) \ + goto label; + +which then can be used as: + + .... + JUMP_LABEL(trace_name, trace_label, jump_enabled);
No. That is three arguments. The macro only has two.
Quoted 64 lines
Documentation/jump-label.txt new file +64
-0+ printk("not doing tracing\n"); + return; +trace_label: + printk("doing tracing: %d\n", file); + .... + +The 'key' argument is thus a pointer to a conditional argument that can be used +if the optimization is not enabled. Otherwise, this address serves as a unique +key to identify the particular instance of the jump label. + +Thus, when tracing is disabled, we simply have a no-op followed by a jump around +the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a +'jump_table' which has the following format: + +[instruction address] [jump target] [tracepoint key] + +Thus, to enable a tracepoint, we simply patch the 'instruction address' with +a jump to the 'jump target'. + +The call to enable a jump label is: enable_jump_label(key); to disable: +disable_jump_label(key); + + +3) architecture interface + + +There are a few functions and macros which arches must implement in order to +take advantage of this optimization. As previously mentioned, if there is no +architecture support we simply fall back to a traditional, load, test, and +jump sequence. + +* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support + +* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h + +* #define "JUMP_LABEL(key, label)", arch/x86/include/asm/jump_label.h + +* void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) + see: arch/x86/kernel/jump_label.c + +* void arch_jump_label_text_poke_early(jump_label_t addr) + see: arch/x86/kernel/jump_label.c + +* finally add a definition for "struct jump_entry". + see: arch/x86/include/asm/jump_label.h + + +4) Jump label analysis (x86) + + +I've tested the performance of using 'get_cycles()' calls around the +tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages): + + idle after tbench run + ---- ---------------- +old code 32 88 +new code 2 4 + + +The performance improvement can be reproduced reliably on both Intel and AMD +hardware. + +In terms of code analysis the current code for the disabled case is a 'cmpl' +followed by a 'je' around the tracepoint code. so:
Documentation/jump-label.txt line 126
Quoted 10 lines
Documentation/jump-label.txt new file +10
-0+ +cmpl - 83 3d 0e 77 87 00 00 - 7 bytes +je - 74 3e - 2 bytes + +total of 9 instruction bytes. + +The new code is a 'nopl' followed by a 'jmp'. Thus: + +nopl - 0f 1f 44 00 00 - 5 bytes +jmp - eb 3e - 2 bytes
Uh, not e9?
Quoted 30 lines
Documentation/jump-label.txt new file +22
-0+ +total of 7 instruction bytes. + +So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint. + +The optimization depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is +set, gcc does not always out of line the not taken label path in the same way +that the "if unlikely()" paths are made out of line. Thus, with +CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be +solved in subsequent gcc versions, that allow us to move labels out of line, +while still optimizing for size. + + +5) Acknowledgments + + +Thanks to Roland McGrath and Richard Henderson for helping come up with the +initial 'asm goto' and jump label design. + +Thanks to Mathieu Desnoyers and H. Peter Anvin for calling attention to this +issue, and outlining the requirements of a solution. Mathieu also implemened a +solution in the form of the "Immediate Values" work.Signature · 7 lines
1.7.1 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/