Skip to contentSkip to conversation

jump label v8

PATCH In review Jason Baron 5 participants 22 messages May 19, 2010 – May 25, 2010 4 messages not in the archive

Subsystems (13)
Documentation x86 Kbuild Library code +9 more subsystems
lore
Share & tools
lore

Jason Baron

author
#0 lore raw

cover jump label v8

Hi,

updates for -v8:

*converted jump labels to use a numeric key instead of a string to identify sets of related jump labels. this saves space, since we no longer store all the strings. It also simplifies the jump label call as:

-# define JUMP_LABEL(tag, label, cond)
+# define JUMP_LABEL(key, label)

since we can use &cond for as the key value

*added 'dynamic debug' code as another consumer of the jump label code (in addition to tracepoints). It would be interesting to identify further codepaths that would benefit from this type of optimization

*added a depends on !CC_OPTIMIZE_FOR_SIZE. When CC_OPTIMIZE_FOR_SIZE is set, gcc does not always out of line the not taken label path in the same way that the "if unlikely()" paths are made out of line. Thus, with CC_OPTIMIZE_FOR_SIZE set, this optimization is not always optimal. This may be solved in subsequent gcc versions, that allow us to move labels out of line, while still optimizing for size. In the case of !CC_OPTIMIZE_FOR_SIZE this optimization is seen on high level benchmarks such as tbench where I can get ~1-2% higher throughput. In addition we are within .5% of the performance of no tracepoints compiled in at all.

thanks,

-Jason

David S. Miller (1):
sparc64: Add jump_label support

Jason Baron (11):
jump label: base patch
jump label: x86 support
jump label: tracepoint support
jump label: add module support
jump label: move ftrace_dyn_arch_init to common code jump label: sort jump table at build-time
jump label: initialize workqueue tracepoints *before* they are
registered
jump label: jump_label_text_reserved() to reserve our jump points jump label: add docs
jump label: convert jump label to use a key
jump label: convert dynamic debug to use jump labels.

Mathieu Desnoyers (1):
jump label: notifier atomic call chain notrace

 Documentation/jump-label.txt        |  143 ++++++++++++
 Makefile                            |    6 +-
 arch/Kconfig                        |    3 +
 arch/sparc/Kconfig                  |    1 +
 arch/sparc/include/asm/jump_entry.h |   18 ++
 arch/sparc/include/asm/jump_label.h |   20 ++
 arch/sparc/kernel/Makefile          |    2 +
 arch/sparc/kernel/jump_label.c      |   38 +++
 arch/sparc/kernel/module.c          |    6 +
 arch/x86/Kconfig                    |    1 +
 arch/x86/include/asm/alternative.h  |   14 ++
 arch/x86/include/asm/jump_entry.h   |   32 +++
 arch/x86/include/asm/jump_label.h   |   22 ++
 arch/x86/kernel/Makefile            |    2 +-
 arch/x86/kernel/alternative.c       |   72 ++++++-
 arch/x86/kernel/ftrace.c            |   70 +------
 arch/x86/kernel/jump_label.c        |   47 ++++
 arch/x86/kernel/kprobes.c           |    3 +-
 arch/x86/kernel/module.c            |    3 +
 arch/x86/kernel/ptrace.c            |    1 +
 arch/x86/kernel/setup.c             |    3 +
 include/asm-generic/vmlinux.lds.h   |   22 ++-
 include/linux/dynamic_debug.h       |   42 ++--
 include/linux/jump_label.h          |   61 +++++
 include/linux/module.h              |    5 +-
 include/linux/tracepoint.h          |   33 ++--
 kernel/Makefile                     |    2 +-
 kernel/jump_label.c                 |  426 +++++++++++++++++++++++++++++++++++
 kernel/kprobes.c                    |    3 +-
 kernel/module.c                     |    7 +
 kernel/notifier.c                   |    6 +-
 kernel/trace/ftrace.c               |   13 +-
 kernel/trace/trace_workqueue.c      |   10 +-
 kernel/tracepoint.c                 |    8 +
 lib/dynamic_debug.c                 |   42 +----
 scripts/Makefile.lib                |   11 +-
 scripts/basic/Makefile              |    2 +-
 scripts/basic/hash.c                |   64 ------
 scripts/mod/Makefile                |    1 +
 scripts/mod/modpost.c               |   61 +++++-
 scripts/mod/modpost.h               |    1 +
 41 files changed, 1077 insertions(+), 250 deletions(-)
 create mode 100644 Documentation/jump-label.txt
 create mode 100644 arch/sparc/include/asm/jump_entry.h
 create mode 100644 arch/sparc/include/asm/jump_label.h
 create mode 100644 arch/sparc/kernel/jump_label.c
 create mode 100644 arch/x86/include/asm/jump_entry.h
 create mode 100644 arch/x86/include/asm/jump_label.h
 create mode 100644 arch/x86/kernel/jump_label.c
 create mode 100644 include/linux/jump_label.h
 create mode 100644 kernel/jump_label.c
 delete mode 100644 scripts/basic/hash.c
author
#1 lore raw

PATCH 1/13 jump label v8: notifier atomic call chain notrace

From: Mathieu Desnoyers <[email protected]>

In LTTng, being able to use the atomic notifier from cpu idle entry to ensure the tracer flush the last events in the current subbuffer requires the rcu read-side to be marked "notrace", otherwise it can end up calling back into lockdep and the tracer.

Also apply to the the die notifier.

Signed-off-by Mathieu Desnoyers <[email protected]>
Signed-off-by Jason Baron <[email protected]>
Reviewed-by Masami Hiramatsu <[email protected]>
---
 kernel/notifier.c |    6 +++---
 1 files changed, 3 insertions(+), 3 deletions(-)
kernel/notifier.c +3 -3
index 2488ba7..88453a7 100644
@@ -148,7 +148,7 @@ int atomic_notifier_chain_unregister(struct atomic_notifier_head *nh,
 	spin_lock_irqsave(&nh->lock, flags);
 	ret = notifier_chain_unregister(&nh->head, n);
 	spin_unlock_irqrestore(&nh->lock, flags);
-	synchronize_rcu();
+	synchronize_sched();
 	return ret;
 }
 EXPORT_SYMBOL_GPL(atomic_notifier_chain_unregister);

@@ -178,9 +178,9 @@ int __kprobes __atomic_notifier_call_chain(struct atomic_notifier_head *nh,
 {
 	int ret;
 
-	rcu_read_lock();
+	rcu_read_lock_sched_notrace();
 	ret = notifier_call_chain(&nh->head, val, v, nr_to_call, nr_calls);
-	rcu_read_unlock();
+	rcu_read_unlock_sched_notrace();
 	return ret;
 }
 EXPORT_SYMBOL_GPL(__atomic_notifier_call_chain);
Signature · 1 line
1.7.0.1
author
#2 lore raw

PATCH 2/13 jump label v8: base patch

In reply to Jason Baron

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

Signed-off-by Jason Baron <[email protected]>
---
 arch/Kconfig                      |    3 +
 include/asm-generic/vmlinux.lds.h |   10 ++-
 include/linux/jump_label.h        |   59 ++++++++++++
 kernel/Makefile                   |    2 +-
 kernel/jump_label.c               |  178 +++++++++++++++++++++++++++++++++++++
 5 files changed, 250 insertions(+), 2 deletions(-)
 create mode 100644 include/linux/jump_label.h
 create mode 100644 kernel/jump_label.c
arch/Kconfig +3 -0
index acda512..722e0b7 100644
@@ -151,4 +151,7 @@ config HAVE_MIXED_BREAKPOINTS_REGS
 config HAVE_USER_RETURN_NOTIFIER
 	bool
 
+config HAVE_ARCH_JUMP_LABEL
+	bool
+
 source "kernel/gcov/Kconfig"
include/asm-generic/vmlinux.lds.h +9 -1
index 67e6520..83a469d 100644
@@ -167,7 +167,8 @@
 	BRANCH_PROFILE()						\
 	TRACE_PRINTKS()							\
 	FTRACE_EVENTS()							\
-	TRACE_SYSCALLS()
+	TRACE_SYSCALLS()						\
+	JUMP_TABLE()							\
 
 /*
  * Data section helpers

@@ -206,6 +207,7 @@
 		*(__vermagic)		/* Kernel version magic */	\
 		*(__markers_strings)	/* Markers: strings */		\
 		*(__tracepoints_strings)/* Tracepoints: strings */	\
+		*(__jump_strings)/* Jump: strings */	\
 	}								\
 									\
 	.rodata1          : AT(ADDR(.rodata1) - LOAD_OFFSET) {		\

@@ -557,6 +559,12 @@
 #define BUG_TABLE
 #endif
 
+#define JUMP_TABLE()							\
+	. = ALIGN(64);							\
+		VMLINUX_SYMBOL(__start___jump_table) = .;		\
+		*(__jump_table)						\
+		VMLINUX_SYMBOL(__stop___jump_table) = .;		\
+
 #ifdef CONFIG_PM_TRACE
 #define TRACEDATA							\
 	. = ALIGN(4);							\
include/linux/jump_label.h new file +59 -0

59 lines not on this page · raw message

kernel/Makefile +1 -1
index d9fa7d8..e78aec0 100644
@@ -10,7 +10,7 @@ obj-y     = sched.o fork.o exec_domain.o panic.o printk.o \
 	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \
 	    hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
 	    notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \
-	    async.o range.o
+	    async.o range.o jump_label.o
 obj-$(CONFIG_HAVE_EARLY_RES) += early_res.o
 obj-y += groups.o
kernel/jump_label.c new file +178 -0

178 lines not on this page · raw message

Signature · 1 line
1.7.0.1
author
#3 lore raw

PATCH 3/13 jump label v8: x86 support

In reply to Jason Baron

add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. hopefully, it wouldn't be too painful for other arches.

Signed-off-by Jason Baron <[email protected]>
---
 arch/x86/Kconfig                  |    1 +
 arch/x86/include/asm/jump_entry.h |   28 +++++++++++++++++
 arch/x86/include/asm/jump_label.h |   27 ++++++++++++++++
 arch/x86/kernel/Makefile          |    2 +-
 arch/x86/kernel/jump_label.c      |   60 +++++++++++++++++++++++++++++++++++++
 5 files changed, 117 insertions(+), 1 deletions(-)
 create mode 100644 arch/x86/include/asm/jump_entry.h
 create mode 100644 arch/x86/include/asm/jump_label.h
 create mode 100644 arch/x86/kernel/jump_label.c
arch/x86/Kconfig +1 -0
index d80d2ab..95f84ed 100644
@@ -59,6 +59,7 @@ config X86
 	select ANON_INODES
 	select HAVE_ARCH_KMEMCHECK
 	select HAVE_USER_RETURN_NOTIFIER
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config INSTRUCTION_DECODER
 	def_bool (KPROBES || PERF_EVENTS)
arch/x86/include/asm/jump_entry.h new file +28 -0
new file mode 100644
index 0000000..9b32696
@@ -0,0 +1,28 @@
+#ifndef _LINUX_JUMP_ENTRY_H
+#define _LINUX_JUMP_ENTRY_H
+
+/* Defined in a separate file, so that it can be used by user program modpost.c
+ * This file is included by include/asm/jump_label.h, in turn included by
+ * include/linux/jump_label.h. Kernel sources should only include
+ * include/linux/jump_label.h.
+ **/
+
+#ifdef CONFIG_X86_64
+
+struct jump_entry {
+	__u64 code;
+	__u64 target;
+	__u64 name;
+};
+
+#else
+
+struct jump_entry {
+	__u32 code;
+	__u32 target;
+	__u32 name;
+};
+
+#endif
+
+#endif
arch/x86/include/asm/jump_label.h new file +27 -0
new file mode 100644
index 0000000..70da737
@@ -0,0 +1,27 @@
+#ifndef _ASM_X86_JUMP_LABEL_H
+#define _ASM_X86_JUMP_LABEL_H
+
+#include <linux/types.h>
+#include <asm/nops.h>
+#include <asm/jump_entry.h>
+
+#define JUMP_LABEL_NOP_SIZE 5
+
+#ifdef CONFIG_X86_64
+# define JUMP_LABEL_NOP P6_NOP5
+#else
+# define JUMP_LABEL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+#endif
+
+#define JUMP_LABEL(tag, label, cond)				\
+	do {							\
+		extern const char __jlstrtab_##tag[];		\
+		asm goto("1:"					\
+			JUMP_LABEL_NOP				\
+			".pushsection __jump_table,  \"a\" \n\t"\
+			_ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+			".popsection \n\t"			\
+			: :  "i" (__jlstrtab_##tag) :  : label);\
+	} while (0)
+
+#endif
arch/x86/kernel/Makefile +1 -1
index e77b220..44f6bea 100644
@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o		:= n
 obj-y			:= process_$(BITS).o signal.o entry_$(BITS).o
 obj-y			+= traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o
 obj-y			+= time.o ioport.o ldt.o dumpstack.o
-obj-y			+= setup.o x86_init.o i8259.o irqinit.o
+obj-y			+= setup.o x86_init.o i8259.o irqinit.o jump_label.o
 obj-$(CONFIG_X86_VISWS)	+= visws_quirks.o
 obj-$(CONFIG_X86_32)	+= probe_roms_32.o
 obj-$(CONFIG_X86_32)	+= sys_i386_32.o i386_ksyms_32.o
arch/x86/kernel/jump_label.c new file +60 -0

60 lines not on this page · raw message

Signature · 1 line
1.7.0.1
author
#4 lore raw

PATCH 4/13 jump label v8: tracepoint support

In reply to Jason Baron

Make use of the jump label infrastructure for tracepoints.

Signed-off-by Jason Baron <[email protected]>
---
 include/linux/tracepoint.h |   34 +++++++++++++++++++---------------
 kernel/tracepoint.c        |    8 ++++++++
 2 files changed, 27 insertions(+), 15 deletions(-)
include/linux/tracepoint.h +19 -15
index 1d85f9a..ea78e92 100644
@@ -16,6 +16,7 @@
 
 #include <linux/types.h>
 #include <linux/rcupdate.h>
+#include <linux/jump_label.h>
 
 struct module;
 struct tracepoint;

@@ -122,20 +123,22 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin,
  * not add unwanted padding between the beginning of the section and the
  * structure. Force alignment to the same alignment as the section start.
  */
-#define DECLARE_TRACE(name, proto, args)				\
-	extern struct tracepoint __tracepoint_##name;			\
-	static inline void trace_##name(proto)				\
-	{								\
-		if (unlikely(__tracepoint_##name.state))		\
-			__DO_TRACE(&__tracepoint_##name,		\
-				TP_PROTO(proto), TP_ARGS(args));	\
-	}								\
-	static inline int register_trace_##name(void (*probe)(proto))	\
-	{								\
-		return tracepoint_probe_register(#name, (void *)probe);	\
-	}								\
-	static inline int unregister_trace_##name(void (*probe)(proto))	\
-	{								\
+#define DECLARE_TRACE(name, proto, args)				 \
+	extern struct tracepoint __tracepoint_##name;			 \
+	static inline void trace_##name(proto)				 \
+	{								 \
+		JUMP_LABEL(&__tracepoint_##name, do_trace, __tracepoint_##name.state);\
+		return;							 \
+do_trace:								 \
+		__DO_TRACE(&__tracepoint_##name,			 \
+			   TP_PROTO(proto), TP_ARGS(args));		 \
+	}								 \
+	static inline int register_trace_##name(void (*probe)(proto))	 \
+	{								 \
+		return tracepoint_probe_register(#name, (void *)probe);	 \
+	}								 \
+	static inline int unregister_trace_##name(void (*probe)(proto))	 \
+	{								 \
 		return tracepoint_probe_unregister(#name, (void *)probe);\
 	}
 

@@ -145,7 +148,8 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin,
 	__attribute__((section("__tracepoints_strings"))) = #name;	\
 	struct tracepoint __tracepoint_##name				\
 	__attribute__((section("__tracepoints"), aligned(32))) =	\
-		{ __tpstrtab_##name, 0, reg, unreg, NULL }
+		{ __tpstrtab_##name, 0, reg, unreg, NULL };		\
+	DEFINE_JUMP_LABEL(name);
 
 #define DEFINE_TRACE(name)						\
 	DEFINE_TRACE_FN(name, NULL, NULL);
kernel/tracepoint.c +8 -0
index cc89be5..8acced8 100644
@@ -25,6 +25,7 @@
 #include <linux/err.h>
 #include <linux/slab.h>
 #include <linux/sched.h>
+#include <linux/jump_label.h>
 
 extern struct tracepoint __start___tracepoints[];
 extern struct tracepoint __stop___tracepoints[];

@@ -256,6 +257,10 @@ static void set_tracepoint(struct tracepoint_entry **entry,
 	 * is used.
 	 */
 	rcu_assign_pointer(elem->funcs, (*entry)->funcs);
+	if (!elem->state && active)
+		enable_jump_label(elem->name);
+	if (elem->state && !active)
+		disable_jump_label(elem->name);
 	elem->state = active;
 }
 

@@ -270,6 +275,9 @@ static void disable_tracepoint(struct tracepoint *elem)
 	if (elem->unregfunc && elem->state)
 		elem->unregfunc();
 
+	if (elem->state)
+		disable_jump_label(elem->name);
+
 	elem->state = 0;
 	rcu_assign_pointer(elem->funcs, NULL);
 }
Signature · 1 line
1.7.0.1
author
#5 lore raw

PATCH 5/13 jump label v8: add module support

In reply to Jason Baron

Add support for 'jump label' for modules.

Signed-off-by Jason Baron <[email protected]>
---
 arch/x86/kernel/ptrace.c   |    1 +
 include/linux/jump_label.h |    3 +-
 include/linux/module.h     |    5 +-
 kernel/jump_label.c        |  136 ++++++++++++++++++++++++++++++++++++++++++++
 kernel/module.c            |    7 ++
 5 files changed, 150 insertions(+), 2 deletions(-)
arch/x86/kernel/ptrace.c +1 -0
index 70c4872..8b83cf7 100644
@@ -21,6 +21,7 @@
 #include <linux/signal.h>
 #include <linux/perf_event.h>
 #include <linux/hw_breakpoint.h>
+#include <linux/module.h>
 
 #include <asm/uaccess.h>
 #include <asm/pgtable.h>
include/linux/jump_label.h +2 -1
index 914ff7c..ab879ef 100644
@@ -20,7 +20,8 @@ extern struct jump_entry __stop___jump_table[];
 
 #define DEFINE_JUMP_LABEL(name)						\
 	const char __jlstrtab_##name[]					\
-	__used __attribute__((section("__jump_strings")))  = #name;
+	__used __attribute__((section("__jump_strings")))  = #name;	\
+	EXPORT_SYMBOL_GPL(__jlstrtab_##name);
 
 extern void arch_jump_label_transform(struct jump_entry *entry,
 				 enum jump_label_type type);
include/linux/module.h +4 -1
index 6914fca..4575f89 100644
@@ -343,7 +343,10 @@ struct module
 	struct tracepoint *tracepoints;
 	unsigned int num_tracepoints;
 #endif
-
+#ifdef HAVE_JUMP_LABEL
+	struct jump_entry *jump_entries;
+	unsigned int num_jump_entries;
+#endif
 #ifdef CONFIG_TRACING
 	const char **trace_bprintk_fmt_start;
 	unsigned int num_trace_bprintk_fmt;
kernel/jump_label.c +136 -0

152 lines not on this page · raw message

kernel/module.c +7 -0
index e256458..441a698 100644
@@ -55,6 +55,7 @@
 #include <linux/async.h>
 #include <linux/percpu.h>
 #include <linux/kmemleak.h>
+#include <linux/jump_label.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/module.h>

@@ -2301,6 +2302,12 @@ static noinline struct module *load_module(void __user *umod,
 					sizeof(*mod->tracepoints),
 					&mod->num_tracepoints);
 #endif
+#ifdef HAVE_JUMP_LABEL
+	mod->jump_entries = section_objs(hdr, sechdrs, secstrings,
+					"__jump_table",
+					sizeof(*mod->jump_entries),
+					&mod->num_jump_entries);
+#endif
 #ifdef CONFIG_EVENT_TRACING
 	mod->trace_events = section_objs(hdr, sechdrs, secstrings,
 					 "_ftrace_events",
Signature · 1 line
1.7.0.1
author
#6 lore raw

PATCH 6/13 jump label v8: move ftrace_dyn_arch_init to common code

In reply to Jason Baron

Move Steve's code for finding the best 5-byte no-op from ftrace.c to alternative.c. The idea is that other consumers (in this case jump label) want to make use of that code. I've created a global: 'char ideal_nop[5]', that is setup during setup_arch that can be used.

Signed-off-by Jason Baron <[email protected]>
---
 arch/x86/include/asm/alternative.h |   14 +++++++
 arch/x86/include/asm/jump_label.h  |   10 +---
 arch/x86/kernel/alternative.c      |   72 +++++++++++++++++++++++++++++++++++-
 arch/x86/kernel/ftrace.c           |   70 +----------------------------------
 arch/x86/kernel/jump_label.c       |   19 +--------
 arch/x86/kernel/module.c           |    3 +
 arch/x86/kernel/setup.c            |    3 +
 include/linux/jump_label.h         |    9 ++++
 kernel/jump_label.c                |   33 ++++++++++++++++
 kernel/trace/ftrace.c              |   13 +------
 10 files changed, 140 insertions(+), 106 deletions(-)
arch/x86/include/asm/alternative.h +14 -0
index 03b6bb5..54c4952 100644
@@ -4,6 +4,7 @@
 #include <linux/types.h>
 #include <linux/stddef.h>
 #include <linux/stringify.h>
+#include <linux/jump_label.h>
 #include <asm/asm.h>
 
 /*

@@ -159,6 +160,8 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 #define __parainstructions_end	NULL
 #endif
 
+extern void *text_poke_early(void *addr, const void *opcode, size_t len);
+
 /*
  * Clear and restore the kernel write-protection flag on the local CPU.
  * Allows the kernel to edit read-only pages.

@@ -179,4 +182,15 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 extern void *text_poke(void *addr, const void *opcode, size_t len);
 extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
 
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
+#define IDEAL_NOP_SIZE_5 5
+extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
+extern int arch_init_ideal_nop5(void);
+#else
+static inline int arch_init_ideal_nop5(void)
+{
+	return 0;
+}
+#endif
+
 #endif /* _ASM_X86_ALTERNATIVE_H */
arch/x86/include/asm/jump_label.h +3 -7
index 70da737..a65a71d 100644
@@ -7,17 +7,13 @@
 
 #define JUMP_LABEL_NOP_SIZE 5
 
-#ifdef CONFIG_X86_64
-# define JUMP_LABEL_NOP P6_NOP5
-#else
-# define JUMP_LABEL_NOP ".byte 0xe9 \n\t .long 0\n\t"
-#endif
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
 
-#define JUMP_LABEL(tag, label, cond)				\
+# define JUMP_LABEL(tag, label, cond)				\
 	do {							\
 		extern const char __jlstrtab_##tag[];		\
 		asm goto("1:"					\
-			JUMP_LABEL_NOP				\
+			JUMP_LABEL_INITIAL_NOP			\
 			".pushsection __jump_table,  \"a\" \n\t"\
 			_ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
 			".popsection \n\t"			\
arch/x86/kernel/alternative.c +70 -2

87 lines not on this page · raw message

arch/x86/kernel/ftrace.c +1 -69

84 lines not on this page · raw message

arch/x86/kernel/jump_label.c +3 -16
index c347431..2db569c 100644
@@ -30,19 +30,8 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type ty
 	if (type == JUMP_LABEL_ENABLE) {
 		code.jump = 0xe9;
 		code.offset = entry->target - (entry->code + JUMP_LABEL_NOP_SIZE);
-	} else {
-#ifdef CONFIG_X86_64
-		/* opcode for P6_NOP5 */
-		code.code[0] = 0x0f;
-		code.code[1] = 0x1f;
-		code.code[2] = 0x44;
-		code.code[3] = 0x00;
-		code.code[4] = 0x00;
-#else
-		code.jump = 0xe9;
-		code.offset = 0;
-#endif
-	}
+	} else
+		memcpy(&code, ideal_nop5, JUMP_LABEL_NOP_SIZE);
 	get_online_cpus();
 	mutex_lock(&text_mutex);
 	text_poke_smp((void *)entry->code, &code, JUMP_LABEL_NOP_SIZE);

@@ -50,11 +39,9 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type ty
 	put_online_cpus();
 }
 
-static const u8 jump_label_nop[JUMP_LABEL_NOP_SIZE] = { 0x0f, 0x1f, 0x44, 0x00, 0x00 };
-
 const u8 *arch_get_jump_label_nop()
 {
-	return jump_label_nop;
+	return ideal_nop5;
 }
 
 #endif
arch/x86/kernel/module.c +3 -0
index e0bc186..3d964dc 100644
@@ -239,6 +239,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 		apply_paravirt(pseg, pseg + para->sh_size);
 	}
 
+	/* make jump label nops */
+	apply_jump_label_nops(me);
+
 	return module_bug_finalize(hdr, sechdrs, me);
 }
arch/x86/kernel/setup.c +3 -0
index c4851ef..de0a3e8 100644
@@ -49,6 +49,7 @@
 #include <asm/pci-direct.h>
 #include <linux/init_ohci1394_dma.h>
 #include <linux/kvm_para.h>
+#include <linux/jump_label.h>
 
 #include <linux/errno.h>
 #include <linux/kernel.h>

@@ -1051,6 +1052,8 @@ void __init setup_arch(char **cmdline_p)
 	x86_init.oem.banner();
 
 	mcheck_init();
+
+	arch_init_ideal_nop5();
 }
 
 #ifdef CONFIG_X86_32
include/linux/jump_label.h +9 -0
index ab879ef..7de278d 100644
@@ -13,6 +13,8 @@ enum jump_label_type {
 	JUMP_LABEL_DISABLE
 };
 
+struct module;
+
 #ifdef HAVE_JUMP_LABEL
 
 extern struct jump_entry __start___jump_table[];

@@ -29,6 +31,8 @@ extern const u8 *arch_get_jump_label_nop(void);
 
 extern void jump_label_update(const char *name, enum jump_label_type type);
 
+extern void apply_jump_label_nops(struct module *mod);
+
 #define enable_jump_label(name) \
 	jump_label_update(name, JUMP_LABEL_ENABLE);
 

@@ -55,6 +59,11 @@ static inline int disable_jump_label(const char *name)
 	return 0;
 }
 
+static inline int apply_jump_label_nops(struct module *mod)
+{
+	return 0;
+}
+
 #endif
 
 #endif
kernel/jump_label.c +33 -0
index 59316f0..3ebe2bf 100644
@@ -184,10 +184,19 @@ void jump_label_update(const char *name, enum jump_label_type type)
 static int init_jump_label(void)
 {
 	int ret;
+	struct jump_entry *iter_start = __start___jump_table;
+	struct jump_entry *iter_stop = __stop___jump_table;
+	struct jump_entry *iter;
 
 	mutex_lock(&jump_label_mutex);
 	ret = build_jump_label_hashtable(__start___jump_table,
 					 __stop___jump_table);
+	iter = iter_start;
+	while (iter < iter_stop) {
+		text_poke_early((void *)iter->code, arch_get_jump_label_nop(),
+				JUMP_LABEL_NOP_SIZE);
+		iter++;
+	}
 	mutex_unlock(&jump_label_mutex);
 	return ret;
 }

@@ -298,6 +307,30 @@ static int jump_label_module_notify(struct notifier_block *self, unsigned long v
 	return ret;
 }
 
+/***
+ * apply_jump_label_nops - patch module jump labels with arch_get_jump_label_nop()
+ * @mod: module to patch
+ *
+ * Allow for run-time selection of the optimal nops. Before the module
+ * loads patch these with arch_get_jump_label_nop(), which is specified by
+ * the arch specific jump label code.
+ */
+void apply_jump_label_nops(struct module *mod)
+{
+	struct jump_entry *iter;
+
+	/* if the module doesn't have jump label entries, just return */
+	if (!mod->num_jump_entries)
+		return;
+
+	iter = mod->jump_entries;
+	while (iter < mod->jump_entries + mod->num_jump_entries) {
+		text_poke_early((void *)iter->code, arch_get_jump_label_nop(),
+				JUMP_LABEL_NOP_SIZE);
+		iter++;
+	}
+}
+
 struct notifier_block jump_label_module_nb = {
 	.notifier_call = jump_label_module_notify,
 	.priority = 0,
kernel/trace/ftrace.c +1 -12
index 32837e1..4e65bdf 100644
@@ -2750,20 +2750,9 @@ extern unsigned long __stop_mcount_loc[];
 
 void __init ftrace_init(void)
 {
-	unsigned long count, addr, flags;
+	unsigned long count;
 	int ret;
 
-	/* Keep the ftrace pointer to the stub */
-	addr = (unsigned long)ftrace_stub;
-
-	local_irq_save(flags);
-	ftrace_dyn_arch_init(&addr);
-	local_irq_restore(flags);
-
-	/* ftrace_dyn_arch_init places the return code in addr */
-	if (addr)
-		goto failed;
-
 	count = __stop_mcount_loc - __start_mcount_loc;
 
 	ret = ftrace_dyn_table_alloc(count);
Signature · 1 line
1.7.0.1
author
#7 lore raw

PATCH 7/13 jump label v8: sort jump table at build-time

In reply to Jason Baron

The jump label table is more optimal accessed if the entries are continguous. Sorting the table accomplishes this. Do the sort at build-time. Adds a '-j' option to 'modpost' which replaces the vmlinux, with a sorted jump label section vmlinux. I've tested this on x86 with relocatable and it works fine there as well. Note that I have not sorted the jump label table in modules. This is b/c the jump label names can be exported by the core kernel, and thus I don't have them available at buildtime. This could be solved by either finding the correct ones in the vmlinux, or by embedding the name of the jump label in the module tables (and not just a pointer), but the module tables tend to be smaller, and thus their is less value to this kind of change anyway. The kernel continues to do the sort, just in case, but at least for the vmlinux, this is just a verfication that the jump label table has already been sorted.

Signed-off-by Jason Baron <[email protected]>
---
 Makefile                          |    6 +++-
 include/asm-generic/vmlinux.lds.h |   18 ++++++++--
 scripts/mod/Makefile              |    1 +
 scripts/mod/modpost.c             |   71 +++++++++++++++++++++++++++++++++++--
 scripts/mod/modpost.h             |    2 +
 5 files changed, 91 insertions(+), 7 deletions(-)
Makefile +5 -1
index 205f280..85edb67 100644
@@ -152,7 +152,7 @@ obj		:= $(objtree)
 
 VPATH		:= $(srctree)$(if $(KBUILD_EXTMOD),:$(KBUILD_EXTMOD))
 
-export srctree objtree VPATH
+export srctree objtree VPATH hdr-arch
 
 
 # SUBARCH tells the usermode build what the underlying arch is.  That is set

@@ -845,6 +845,9 @@ define rule_vmlinux-modpost
 	$(Q)echo 'cmd_$@ := $(cmd_vmlinux-modpost)' > $(dot-target).cmd
 endef
 
+quiet_cmd_sort-jump-label = SORT    $@
+      cmd_sort-jump-label = $(srctree)/scripts/mod/modpost -j -s $@
+
 # vmlinux image - including updated kernel symbols
 vmlinux: $(vmlinux-lds) $(vmlinux-init) $(vmlinux-main) vmlinux.o $(kallsyms.o) FORCE
 ifdef CONFIG_HEADERS_CHECK

@@ -858,6 +861,7 @@ ifdef CONFIG_BUILD_DOCSRC
 endif
 	$(call vmlinux-modpost)
 	$(call if_changed_rule,vmlinux__)
+	$(call cmd,sort-jump-label)
 	$(Q)rm -f .old_version
 
 # build vmlinux.o first to catch section mismatch errors early
include/asm-generic/vmlinux.lds.h +15 -3
index 83a469d..f9d8188 100644
@@ -168,7 +168,6 @@
 	TRACE_PRINTKS()							\
 	FTRACE_EVENTS()							\
 	TRACE_SYSCALLS()						\
-	JUMP_TABLE()							\
 
 /*
  * Data section helpers

@@ -207,7 +206,6 @@
 		*(__vermagic)		/* Kernel version magic */	\
 		*(__markers_strings)	/* Markers: strings */		\
 		*(__tracepoints_strings)/* Tracepoints: strings */	\
-		*(__jump_strings)/* Jump: strings */	\
 	}								\
 									\
 	.rodata1          : AT(ADDR(.rodata1) - LOAD_OFFSET) {		\

@@ -216,6 +214,10 @@
 									\
 	BUG_TABLE							\
 									\
+	JUMP_TABLE							\
+									\
+	JUMP_STRINGS							\
+									\
 	/* PCI quirks */						\
 	.pci_fixup        : AT(ADDR(.pci_fixup) - LOAD_OFFSET) {	\
 		VMLINUX_SYMBOL(__start_pci_fixups_early) = .;		\

@@ -559,11 +561,21 @@
 #define BUG_TABLE
 #endif
 
-#define JUMP_TABLE()							\
+#define JUMP_TABLE							\
 	. = ALIGN(64);							\
+	__jump_table : AT(ADDR(__jump_table) - LOAD_OFFSET) {		\
 		VMLINUX_SYMBOL(__start___jump_table) = .;		\
 		*(__jump_table)						\
 		VMLINUX_SYMBOL(__stop___jump_table) = .;		\
+	}
+
+#define JUMP_STRINGS							\
+	. = ALIGN(64);							\
+	__jump_strings : AT(ADDR(__jump_strings) - LOAD_OFFSET) {	\
+		VMLINUX_SYMBOL(__start___jump_strings) = .;		\
+		*(__jump_strings)					\
+		VMLINUX_SYMBOL(__stop___jump_strings) = .;		\
+	}
 
 #ifdef CONFIG_PM_TRACE
 #define TRACEDATA							\
scripts/mod/Makefile +1 -0
index ff954f8..a6dbbe6 100644
@@ -1,4 +1,5 @@
 hostprogs-y	:= modpost mk_elfconfig
+HOST_EXTRACFLAGS += -iquote "$(srctree)/arch/$(hdr-arch)/include/asm"
 always		:= $(hostprogs-y) empty.o
 
 modpost-objs	:= modpost.o file2alias.o sumversion.o
scripts/mod/modpost.c +68 -3

123 lines not on this page · raw message

scripts/mod/modpost.h +2 -0
index be987a4..0875d4b 100644
@@ -126,6 +126,8 @@ struct elf_info {
 	Elf_Section  export_gpl_sec;
 	Elf_Section  export_unused_gpl_sec;
 	Elf_Section  export_gpl_future_sec;
+	Elf_Section  jump_sec;
+	Elf_Section  jump_strings_sec;
 	const char   *strtab;
 	char	     *modinfo;
 	unsigned int modinfo_len;
Signature · 1 line
1.7.0.1
maintainer
#8 lore raw

Re: [PATCH 07/13] jump label v8: sort jump table at build-time

From: Jason Baron <[email protected]>
Date: Wed, 19 May 2010 12:22:17 -0400

scripts/mod/modpost.c line 22

Quoted 11 lines
scripts/mod/modpost.c +2 -0
index 2092361..fe6f8be 100644
@@ -17,6 +17,8 @@
 #include "modpost.h"
 #include "../../include/generated/autoconf.h"
 #include "../../include/linux/license.h"
+#include <linux/types.h>
+#include "jump_entry.h"

This breaks the build on non-jump_label-supporting architectures because only they will have the jump_entry.h header file.

I'm really getting tired trying to test your changes and every single time as I go through doing very basic build testing I always hit one patch that assumes jump label support exists, or assumes some x86'ism.

author
#9 lore raw

Re: [PATCH 07/13] jump label v8: sort jump table at build-time

Quoted 15 lines · David Miller

scripts/mod/modpost.c line 22

Quoted 11 lines
scripts/mod/modpost.c +2 -0
index 2092361..fe6f8be 100644
@@ -17,6 +17,8 @@
 #include "modpost.h"
 #include "../../include/generated/autoconf.h"
 #include "../../include/linux/license.h"
+#include <linux/types.h>
+#include "jump_entry.h"

This breaks the build on non-jump_label-supporting architectures because only they will have the jump_entry.h header file.

Quoted 3 lines

I'm really getting tired trying to test your changes and every single time as I go through doing very basic build testing I always hit one patch that assumes jump label support exists, or assumes some x86'ism.

sorry about that. I mis-tested it. anyways here's the interdiff to this patch to fix. btw, how do the sparc bits look?

thanks,

-Jason

scripts/mod/modpost.c +9 -1
index fe6f8be..cc87012 100644
@@ -18,7 +18,9 @@
 #include "../../include/generated/autoconf.h"
 #include "../../include/linux/license.h"
 #include <linux/types.h>
-#include "jump_entry.h"
+#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL
+ #include "jump_entry.h"
+#endif
 
 /* Some toolchains use a `_' prefix for all user symbols. */
 #ifdef CONFIG_SYMBOL_PREFIX

@@ -377,6 +379,8 @@ void release_file(void *file, unsigned long size)
 	munmap(file, size);
 }
 
+#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL
+
 static void swap_jump_label_entries(struct jump_entry *previous, struct jump_entry *next)
 {
 	struct jump_entry tmp;

@@ -421,6 +425,8 @@ static void sort_jump_label_table(struct elf_info *info, Elf_Ehdr *hdr)
 	} while (swapped == 1);
 }
 
+#endif
+
 static int parse_elf(struct elf_info *info, const char *filename)
 {
 	unsigned int i;

@@ -539,8 +545,10 @@ static int parse_elf(struct elf_info *info, const char *filename)
 		sym->st_size  = TO_NATIVE(sym->st_size);
 	}
 
+#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL
 	if (enable_jump_label)
 		sort_jump_label_table(info, hdr);
+#endif
 
 	return 1;
 }
#10 lore raw

Re: [PATCH 07/13] jump label v8: sort jump table at build-time

Quoted 33 lines · * Jason Baron ([email protected]) wrote:
Quoted 15 lines · David Miller

scripts/mod/modpost.c line 22

Quoted 11 lines
scripts/mod/modpost.c +2 -0
index 2092361..fe6f8be 100644
@@ -17,6 +17,8 @@
 #include "modpost.h"
 #include "../../include/generated/autoconf.h"
 #include "../../include/linux/license.h"
+#include <linux/types.h>
+#include "jump_entry.h"

This breaks the build on non-jump_label-supporting architectures because only they will have the jump_entry.h header file.

Quoted 3 lines

I'm really getting tired trying to test your changes and every single time as I go through doing very basic build testing I always hit one patch that assumes jump label support exists, or assumes some x86'ism.

sorry about that. I mis-tested it. anyways here's the interdiff to this patch to fix. btw, how do the sparc bits look?

thanks,

(quoted diff) +0 -1
-Jason
scripts/mod/modpost.c +0 -0
index fe6f8be..cc87012 100644

scripts/mod/modpost.c line 21

Quoted 4 lines
scripts/mod/modpost.c +0 -1
@@ -18,7 +18,9 @@
 #include "../../include/generated/autoconf.h"
 #include "../../include/linux/license.h"
 #include <linux/types.h>
-#include "jump_entry.h"

includes with "" is a bit unexpected in kernel code. Any reason for this ? Is jump_entry.h shipped in scripts/mod/ ?

scripts/mod/modpost.c line 21

Quoted 1 line
scripts/mod/modpost.c +1 -0
+#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL

ifdefs everywhere cripples the code and make it unreadable. You might want to try to find another way to deal with the problem. E.g.

create linux/jump_entry.h
which contains

#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL
#include <asm/jump_entry.h>
#else

define some empty static inlines, and use them in the .c files.

e.g.

static inline void sort_jump_label_table(struct elf_info *info, Elf_Ehdr *hdr) {
}

#endif

Quoted 24 lines
scripts/mod/modpost.c +6 -0
+ #include "jump_entry.h"
+#endif
 
 /* Some toolchains use a `_' prefix for all user symbols. */
 #ifdef CONFIG_SYMBOL_PREFIX

@@ -377,6 +379,8 @@ void release_file(void *file, unsigned long size)
 	munmap(file, size);
 }
 
+#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL
+
 static void swap_jump_label_entries(struct jump_entry *previous, struct jump_entry *next)
 {
 	struct jump_entry tmp;

@@ -421,6 +425,8 @@ static void sort_jump_label_table(struct elf_info *info, Elf_Ehdr *hdr)
 	} while (swapped == 1);
 }
 
+#endif
+
 static int parse_elf(struct elf_info *info, const char *filename)
 {
 	unsigned int i;

scripts/mod/modpost.c static int parse_elf(struct elf_info *info, const char *filename) · line 551

Quoted 7 lines
scripts/mod/modpost.c +2 -0
@@ -539,8 +545,10 @@ static int parse_elf(struct elf_info *info, const char *filename)
 		sym->st_size  = TO_NATIVE(sym->st_size);
 	}
 
+#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL
 	if (enable_jump_label)
 		sort_jump_label_table(info, hdr);
+#endif

And remove this ifdef.

Thanks,

Mathieu

Quoted 3 lines
scripts/mod/modpost.c +0 -0
 	return 1;
 }
Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
author
#11 lore raw

Re: [PATCH 07/13] jump label v8: sort jump table at build-time

Quoted 29 lines · Mathieu Desnoyers
Quoted 28 lines · * Jason Baron ([email protected]) wrote:
Quoted 15 lines · David Miller

scripts/mod/modpost.c line 22

Quoted 11 lines
scripts/mod/modpost.c +2 -0
index 2092361..fe6f8be 100644
@@ -17,6 +17,8 @@
 #include "modpost.h"
 #include "../../include/generated/autoconf.h"
 #include "../../include/linux/license.h"
+#include <linux/types.h>
+#include "jump_entry.h"

This breaks the build on non-jump_label-supporting architectures because only they will have the jump_entry.h header file.

Quoted 3 lines

I'm really getting tired trying to test your changes and every single time as I go through doing very basic build testing I always hit one patch that assumes jump label support exists, or assumes some x86'ism.

sorry about that. I mis-tested it. anyways here's the interdiff to this patch to fix. btw, how do the sparc bits look?

thanks,

-Jason

Quoted 13 lines

scripts/mod/modpost.c line 21

Quoted 9 lines
scripts/mod/modpost.c +0 -1
index fe6f8be..cc87012 100644
@@ -18,7 +18,9 @@
 #include "../../include/generated/autoconf.h"
 #include "../../include/linux/license.h"
 #include <linux/types.h>
-#include "jump_entry.h"

includes with "" is a bit unexpected in kernel code. Any reason for this ? Is jump_entry.h shipped in scripts/mod/ ?

The jump_entry.h is per arch found in:
arch/<arch>/include/asm/jump_entry.

Its being found in modpost.c, via:

HOST_EXTRACFLAGS += -iquote "$(srctree)/arch/$(hdr-arch)/include/asm"

in scripts/mod/Makefile

I used the quotes so that its less likely to conflict with any other include.

Quoted 14 lines

scripts/mod/modpost.c line 21

Quoted 1 line
scripts/mod/modpost.c +1 -0
+#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL

ifdefs everywhere cripples the code and make it unreadable. You might want to try to find another way to deal with the problem. E.g.

create linux/jump_entry.h
which contains

#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL
#include <asm/jump_entry.h>
#else

define some empty static inlines, and use them in the .c files.

Quoted 8 lines

e.g.

static inline void sort_jump_label_table(struct elf_info *info, Elf_Ehdr *hdr) {
}

#endif

right. i have that defined already as include/linux/jump_label.h. Which is the include file that kernel code uses. However, I can't use it in the modpost.c code, since there are a number of kernel specific references. (I probably could simplify things a bit using #ifdef KERNEL).

Quoted 34 lines
Quoted 24 lines
scripts/mod/modpost.c +6 -0
+ #include "jump_entry.h"
+#endif
 
 /* Some toolchains use a `_' prefix for all user symbols. */
 #ifdef CONFIG_SYMBOL_PREFIX

@@ -377,6 +379,8 @@ void release_file(void *file, unsigned long size)
 	munmap(file, size);
 }
 
+#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL
+
 static void swap_jump_label_entries(struct jump_entry *previous, struct jump_entry *next)
 {
 	struct jump_entry tmp;

@@ -421,6 +425,8 @@ static void sort_jump_label_table(struct elf_info *info, Elf_Ehdr *hdr)
 	} while (swapped == 1);
 }
 
+#endif
+
 static int parse_elf(struct elf_info *info, const char *filename)
 {
 	unsigned int i;

scripts/mod/modpost.c static int parse_elf(struct elf_info *info, const char *filename) · line 551

Quoted 7 lines
scripts/mod/modpost.c +2 -0
@@ -539,8 +545,10 @@ static int parse_elf(struct elf_info *info, const char *filename)
 		sym->st_size  = TO_NATIVE(sym->st_size);
 	}
 
+#ifdef CONFIG_HAVE_ARCH_JUMP_LABEL
 	if (enable_jump_label)
 		sort_jump_label_table(info, hdr);
+#endif

And remove this ifdef.

Quoted 4 lines

Thanks,

Mathieu

agreed. I'll clean this up in the next iteration.

thanks,

-jason

maintainer
#12 lore raw

Re: [PATCH 07/13] jump label v8: sort jump table at build-time

In reply to Jason Baron

From: Jason Baron <[email protected]>
Date: Thu, 20 May 2010 16:17:29 -0400

Quoted 1 line

btw, how do the sparc bits look?

I have to send you a fix because I use the "call" opcode instead of the one for "ba".

author
#13 lore raw

PATCH 8/13 jump label v8: initialize workqueue tracepoints *before* they are registered

In reply to Jason Baron

Initialize the workqueue data structures *before* they are registered so that they are ready for callbacks.

Signed-off-by Jason Baron <[email protected]>
---
 kernel/trace/trace_workqueue.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)
kernel/trace/trace_workqueue.c +5 -5
index cc2d2fa..7705ad4 100644
@@ -259,6 +259,11 @@ int __init trace_workqueue_early_init(void)
 {
 	int ret, cpu;
 
+	for_each_possible_cpu(cpu) {
+		spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
+		INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
+	}
+
 	ret = register_trace_workqueue_insertion(probe_workqueue_insertion);
 	if (ret)
 		goto out;

@@ -275,11 +280,6 @@ int __init trace_workqueue_early_init(void)
 	if (ret)
 		goto no_creation;
 
-	for_each_possible_cpu(cpu) {
-		spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
-		INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
-	}
-
 	return 0;
 
 no_creation:
Signature · 1 line
1.7.0.1
author
#14 lore raw

PATCH 9/13 jump label v8: jump_label_text_reserved() to reserve our jump points

In reply to Jason Baron

Add a jump_label_text_reserved(void *start, void *end), so that other pieces of code that want to modify kernel text, can first verify that jump label has not reserved the instruction.

Signed-off-by Jason Baron <[email protected]>
Acked-by Masami Hiramatsu <[email protected]>
---
 arch/x86/kernel/kprobes.c  |    3 +-
 include/linux/jump_label.h |    6 +++
 kernel/jump_label.c        |   83 ++++++++++++++++++++++++++++++++++++++++++++
 kernel/kprobes.c           |    3 +-
 4 files changed, 93 insertions(+), 2 deletions(-)
arch/x86/kernel/kprobes.c +2 -1
index 345a4b1..87ca919 100644
@@ -1201,7 +1201,8 @@ static int __kprobes copy_optimized_instructions(u8 *dest, u8 *src)
 	}
 	/* Check whether the address range is reserved */
 	if (ftrace_text_reserved(src, src + len - 1) ||
-	    alternatives_text_reserved(src, src + len - 1))
+	    alternatives_text_reserved(src, src + len - 1) ||
+	    jump_label_text_reserved(src, src + len - 1))
 		return -EBUSY;
 
 	return len;
include/linux/jump_label.h +6 -0
index 7de278d..2464348 100644
@@ -30,6 +30,7 @@ extern void arch_jump_label_transform(struct jump_entry *entry,
 extern const u8 *arch_get_jump_label_nop(void);
 
 extern void jump_label_update(const char *name, enum jump_label_type type);
+extern int jump_label_text_reserved(void *start, void *end);
 
 extern void apply_jump_label_nops(struct module *mod);
 

@@ -64,6 +65,11 @@ static inline int apply_jump_label_nops(struct module *mod)
 	return 0;
 }
 
+static inline int jump_label_text_reserved(void *start, void *end)
+{
+	return 0;
+}
+
 #endif
 
 #endif
kernel/jump_label.c +83 -0

89 lines not on this page · raw message

kernel/kprobes.c +2 -1
index 282035f..2f9bea6 100644
@@ -1141,7 +1141,8 @@ int __kprobes register_kprobe(struct kprobe *p)
 	preempt_disable();
 	if (!kernel_text_address((unsigned long) p->addr) ||
 	    in_kprobes_functions((unsigned long) p->addr) ||
-	    ftrace_text_reserved(p->addr, p->addr)) {
+	    ftrace_text_reserved(p->addr, p->addr) ||
+	    jump_label_text_reserved(p->addr, p->addr)) {
 		preempt_enable();
 		return -EINVAL;
 	}
Signature · 1 line
1.7.0.1
author
#15 lore raw

PATCH 10/13 jump label v8: add docs

In reply to Jason Baron

Add jump label docs as: Documentation/jump-label.txt

Signed-off-by Jason Baron <[email protected]>
---
 Documentation/jump-label.txt |  143 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 143 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/jump-label.txt
Documentation/jump-label.txt new file +143 -0
new file mode 100644
index 0000000..7fc18a0
@@ -0,0 +1,143 @@
+			Jump Label
+			----------
+
+By: Jason Baron <[email protected]>
+
+
+1) motivation
+
+
+Currently, tracepoints are implemented using a conditional. The conditional
+check requires checking a global variable for each tracepoint. Although,
+the overhead of this check is small, it increases under memory pressure. As we
+increase the number of tracepoints in the kernel this may become more of an
+issue. In addition, tracepoints are often dormant (disabled), and provide no
+direct kernel functionality. Thus, it is highly desirable to reduce their
+impact as much as possible. Although tracepoints are the original motivation
+for this work, other kernel code paths should be able to make use of the jump
+label optimization.
+
+
+2) jump label description/usage
+
+
+gcc (v4.5) adds a new 'asm goto' statement that allows branching to a label.
+http://gcc.gnu.org/ml/gcc-patches/2009-07/msg01556.html
+
+Thus, this patch set introduces an architecture specific 'JUMP_LABEL()' macro as
+follows (x86):
+
+# define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
+
+# define JUMP_LABEL(key, label)					\
+	do {                                                    \
+		asm goto("1:"                                   \
+			JUMP_LABEL_INITIAL_NOP			\
+                        ".pushsection __jump_table,  \"a\" \n\t"\
+                        _ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
+                        ".popsection \n\t"                      \
+                        : :  "i" (key) :  : label);		\
+        } while (0)
+
+
+For architectures that have not yet introduced jump label support its simply:
+
+#define JUMP_LABEL(key, label)			\
+	if (unlikely(*key))			\
+		goto label;
+
+which then can be used as:
+
+	....
+        JUMP_LABEL(trace_name, trace_label, jump_enabled);
+        printk("not doing tracing\n");
+	return;
+trace_label:
+        printk("doing tracing: %d\n", file);
+	....
+
+The 'key' argument is thus a pointer to a conditional argument that can be used
+if the optimization is not enabled. Otherwise, this address serves as a unique
+key to identify the particular instance of the jump label.
+
+Thus, when tracing is disabled, we simply have a no-op followed by a jump around
+the dormant (disabled) tracing code. The 'JUMP_LABEL()' macro, produces a
+'jump_table' which has the following format:
+
+[instruction address] [jump target] [tracepoint name]
+
+Thus, to enable a tracepoint, we simply patch the 'instruction address' with
+a jump to the 'jump target'.
+
+The call to enable a jump label is: enable_jump_label(trace_name); to disable:
+disable_jump_label(trace_name);
+
+
+3) Jump label analysis (x86)
+
+
+I've tested the performance of using 'get_cycles()' calls around the
+tracepoint call sites. For an Intel Core 2 Quad cpu (in cycles, averages):
+
+		idle		after tbench run
+		----		----------------
+old code	 32		  88
+new code	  2		   4
+
+
+The performance improvement can be reproduced reliably on both Intel and AMD
+hardware.
+
+In terms of code analysis the current code for the disabled case is a 'cmpl'
+followed by a 'je' around the tracepoint code. so:
+
+cmpl - 83 3d 0e 77 87 00 00 - 7 bytes
+je   - 74 3e                - 2 bytes
+
+total of 9 instruction bytes.
+
+The new code is a 'nopl' followed by a 'jmp'. Thus:
+
+nopl - 0f 1f 44 00 00 - 5 bytes
+jmp  - eb 3e          - 2 bytes
+
+total of 7 instruction bytes.
+
+So, the new code also accounts for 2 less bytes in the instruction cache per tracepoint.
+
+
+4) architecture interface
+
+
+There are a few functions and macros which arches must implement in order to
+take advantage of this optimization. As previously mentioned, if there is no
+architecture support we simply fall back to a traditional, load, test, and
+jump sequence.
+
+* add "HAVE_ARCH_JUMP_LABEL" to arch/<arch>/Kconfig to indicate support
+
+* #define JUMP_LABEL_NOP_SIZE, arch/x86/include/asm/jump_label.h
+
+* #define "JUMP_LABEL(tag, label, cond)", arch/x86/include/asm/jump_label.h
+
+* add: void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+				and
+       const u8 *arch_get_jump_label_nop(void)
+
+	see: arch/x86/kernel/jump_label.c
+
+* finally add a definition for "struct jump_entry". This must be done in a
+  separate .h file, b/c the modpost.c code uses this definition to sort the
+  the jump label tabel in the vmlinux, so that it does not have to be sorted at
+  runtime. see: arch/x86/include/asm/jump_entry.h
+
+
+5) Acknowledgments
+
+
+Thanks to Roland McGrath and Richard Henderson for coming up with the initial
+'asm goto' and jump label design.
+
+Thanks to Mathieu Desnoyers for calling attention to this issue, outlining the
+requirements of a solution, and implementing a solution in the form of the
+"Immediate Values" work.
Signature · 1 line
1.7.0.1
author
#16 lore raw

PATCH 11/13 jump label v8: convert jump label to use a key

In reply to Jason Baron

Convert the jump label code to use a key to identify each related set of jump labels instead of a string. This saves space since we no longer need to store strings and their pointers, just the keys. In addition we can simplify the JUMP_LABEL macro to 2 arguments from 3. We use the address of the conditional variable as the key value. If jump labels are not enabled then we can use the conditional variable as usual.

Signed-off-by Jason Baron <[email protected]>
---
 arch/x86/include/asm/jump_entry.h |   16 +++++++++-----
 arch/x86/include/asm/jump_label.h |    5 +--
 include/linux/jump_label.h        |   32 ++++++++----------------------
 include/linux/tracepoint.h        |    5 +--
 kernel/jump_label.c               |   38 ++++++++++++++++--------------------
 kernel/tracepoint.c               |    6 ++--
 scripts/mod/modpost.c             |   14 +-----------
 scripts/mod/modpost.h             |    1 -
 8 files changed, 45 insertions(+), 72 deletions(-)
arch/x86/include/asm/jump_entry.h +10 -6
index 9b32696..4ed5428 100644
@@ -9,18 +9,22 @@
 
 #ifdef CONFIG_X86_64
 
+typedef __u64 jump_label_t;
+
 struct jump_entry {
-	__u64 code;
-	__u64 target;
-	__u64 name;
+	jump_label_t code;
+	jump_label_t target;
+	jump_label_t key;
 };
 
 #else
 
+typedef __u32 jump_label_t;
+
 struct jump_entry {
-	__u32 code;
-	__u32 target;
-	__u32 name;
+	jump_label_t code;
+	jump_label_t target;
+	jump_label_t key;
 };
 
 #endif
arch/x86/include/asm/jump_label.h +2 -3
index a65a71d..07ef68b 100644
@@ -9,15 +9,14 @@
 
 # define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
 
-# define JUMP_LABEL(tag, label, cond)				\
+# define JUMP_LABEL(key, label)					\
 	do {							\
-		extern const char __jlstrtab_##tag[];		\
 		asm goto("1:"					\
 			JUMP_LABEL_INITIAL_NOP			\
 			".pushsection __jump_table,  \"a\" \n\t"\
 			_ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
 			".popsection \n\t"			\
-			: :  "i" (__jlstrtab_##tag) :  : label);\
+			: : "i" (key) :  : label);\
 	} while (0)
 
 #endif
include/linux/jump_label.h +9 -23
index 2464348..1eaf884 100644
@@ -20,45 +20,31 @@ struct module;
 extern struct jump_entry __start___jump_table[];
 extern struct jump_entry __stop___jump_table[];
 
-#define DEFINE_JUMP_LABEL(name)						\
-	const char __jlstrtab_##name[]					\
-	__used __attribute__((section("__jump_strings")))  = #name;	\
-	EXPORT_SYMBOL_GPL(__jlstrtab_##name);
-
 extern void arch_jump_label_transform(struct jump_entry *entry,
 				 enum jump_label_type type);
 extern const u8 *arch_get_jump_label_nop(void);
 
-extern void jump_label_update(const char *name, enum jump_label_type type);
+extern void jump_label_update(jump_label_t key, enum jump_label_type type);
 extern int jump_label_text_reserved(void *start, void *end);
 
 extern void apply_jump_label_nops(struct module *mod);
 
-#define enable_jump_label(name) \
-	jump_label_update(name, JUMP_LABEL_ENABLE);
+#define enable_jump_label(key) \
+	jump_label_update(key, JUMP_LABEL_ENABLE);
 
-#define disable_jump_label(name) \
-	jump_label_update(name, JUMP_LABEL_DISABLE);
+#define disable_jump_label(key) \
+	jump_label_update(key, JUMP_LABEL_DISABLE);
 
 #else
 
-#define DEFINE_JUMP_LABEL(name)
-
-#define JUMP_LABEL(tag, label, cond)		\
+#define JUMP_LABEL(key, label)			\
 do {						\
-	if (unlikely(cond))			\
+	if (unlikely(*key))			\
 		goto label;			\
 } while (0)
 
-static inline int enable_jump_label(const char *name)
-{
-	return 0;
-}
-
-static inline int disable_jump_label(const char *name)
-{
-	return 0;
-}
+#define enable_jump_label(key)
+#define disable_jump_label(key)
 
 static inline int apply_jump_label_nops(struct module *mod)
 {
include/linux/tracepoint.h +2 -3
index ea78e92..81a7ae6 100644
@@ -127,7 +127,7 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin,
 	extern struct tracepoint __tracepoint_##name;			 \
 	static inline void trace_##name(proto)				 \
 	{								 \
-		JUMP_LABEL(&__tracepoint_##name, do_trace, __tracepoint_##name.state);\
+		JUMP_LABEL(&__tracepoint_##name.state, do_trace);	 \
 		return;							 \
 do_trace:								 \
 		__DO_TRACE(&__tracepoint_##name,			 \

@@ -148,8 +148,7 @@ do_trace:								 \
 	__attribute__((section("__tracepoints_strings"))) = #name;	\
 	struct tracepoint __tracepoint_##name				\
 	__attribute__((section("__tracepoints"), aligned(32))) =	\
-		{ __tpstrtab_##name, 0, reg, unreg, NULL };		\
-	DEFINE_JUMP_LABEL(name);
+		{ __tpstrtab_##name, 0, reg, unreg, NULL };
 
 #define DEFINE_TRACE(name)						\
 	DEFINE_TRACE_FN(name, NULL, NULL);
kernel/jump_label.c +17 -21
index ac39662..859a87c 100644
@@ -27,7 +27,7 @@ struct jump_label_entry {
 	int nr_entries;
 	/* hang modules off here */
 	struct hlist_head modules;
-	const char *name;
+	unsigned long key;
 };
 
 struct jump_label_module_entry {

@@ -58,8 +58,7 @@ static void sort_jump_label_entries(struct jump_entry *start, struct jump_entry
 		iter_next = start;
 		iter_next++;
 		for (; iter_next < stop; iter++, iter_next++) {
-			if (strcmp((char *)iter->name,
-						(char *)iter_next->name) > 0) {
+			if (iter->key > iter_next->key) {
 				swap_jump_label_entries(iter, iter_next);
 				swapped = 1;
 			}

@@ -67,29 +66,28 @@ static void sort_jump_label_entries(struct jump_entry *start, struct jump_entry
 	} while (swapped == 1);
 }
 
-static struct jump_label_entry *get_jump_label_entry(const char *name)
+static struct jump_label_entry *get_jump_label_entry(jump_label_t key)
 {
 	struct hlist_head *head;
 	struct hlist_node *node;
 	struct jump_label_entry *e;
-	u32 hash = jhash(name, strlen(name), 0);
+	u32 hash = jhash((void *)&key, sizeof(jump_label_t), 0);
 
 	head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
 	hlist_for_each_entry(e, node, head, hlist) {
-		if (!strcmp((char *)name, (char *)e->name))
+		if (key == e->key)
 			return e;
 	}
 	return NULL;
 }
 
-static struct jump_label_entry *add_jump_label_entry(const char *name, int nr_entries, struct jump_entry *table)
+static struct jump_label_entry *add_jump_label_entry(jump_label_t key, int nr_entries, struct jump_entry *table)
 {
 	struct hlist_head *head;
 	struct jump_label_entry *e;
-	size_t name_len;
 	u32 hash;
 
-	e = get_jump_label_entry(name);
+	e = get_jump_label_entry(key);
 	if (e)
 		return ERR_PTR(-EEXIST);
 

@@ -97,10 +95,9 @@ static struct jump_label_entry *add_jump_label_entry(const char *name, int nr_en
 	if (!e)
 		return ERR_PTR(-ENOMEM);
 
-	name_len = strlen(name) + 1;
-	hash = jhash(name, name_len-1, 0);
+	hash = jhash((void *)&key, sizeof(jump_label_t), 0);
 	head = &jump_label_table[hash & (JUMP_LABEL_TABLE_SIZE - 1)];
-	e->name = name;
+	e->key = key;
 	e->table = table;
 	e->nr_entries = nr_entries;
 	INIT_HLIST_HEAD(&(e->modules));

@@ -117,17 +114,16 @@ static int build_jump_label_hashtable(struct jump_entry *start, struct jump_entr
 	sort_jump_label_entries(start, stop);
 	iter = start;
 	while (iter < stop) {
-		entry = get_jump_label_entry((char *)iter->name);
+		entry = get_jump_label_entry(iter->key);
 		if (!entry) {
 			iter_begin = iter;
 			count = 0;
 			while ((iter < stop) &&
-				(strcmp((char *)iter->name,
-					(char *)iter_begin->name) == 0)) {
+				(iter->key == iter_begin->key)) {
 				iter++;
 				count++;
 			}
-			entry = add_jump_label_entry((char *)iter_begin->name,
+			entry = add_jump_label_entry(iter_begin->key,
 							count, iter_begin);
 			if (IS_ERR(entry))
 				return PTR_ERR(entry);

@@ -148,7 +144,7 @@ static int build_jump_label_hashtable(struct jump_entry *start, struct jump_entr
  *
  */
 
-void jump_label_update(const char *name, enum jump_label_type type)
+void jump_label_update(jump_label_t key, enum jump_label_type type)
 {
 	struct jump_entry *iter;
 	struct jump_label_entry *entry;

@@ -157,7 +153,7 @@ void jump_label_update(const char *name, enum jump_label_type type)
 	int count;
 
 	mutex_lock(&jump_label_mutex);
-	entry = get_jump_label_entry(name);
+	entry = get_jump_label_entry(key);
 	if (entry) {
 		count = entry->nr_entries;
 		iter = entry->table;

@@ -316,16 +312,16 @@ static int add_jump_label_module(struct module *mod)
 				mod->jump_entries + mod->num_jump_entries);
 	iter = mod->jump_entries;
 	while (iter < mod->jump_entries + mod->num_jump_entries) {
-		entry = get_jump_label_entry((char *)iter->name);
+		entry = get_jump_label_entry(iter->key);
 		iter_begin = iter;
 		count = 0;
 		while ((iter < mod->jump_entries + mod->num_jump_entries) &&
-			(strcmp((char *)iter->name, (char *)iter_begin->name) == 0)) {
+			(iter->key == iter_begin->key)) {
 				iter++;
 				count++;
 		}
 		if (!entry) {
-			entry = add_jump_label_entry((char *)iter_begin->name, 0, NULL);
+			entry = add_jump_label_entry(iter_begin->key, 0, NULL);
 			if (IS_ERR(entry))
 				return PTR_ERR(entry);
 		}
kernel/tracepoint.c +3 -3
index 8acced8..d07b42c 100644
@@ -258,9 +258,9 @@ static void set_tracepoint(struct tracepoint_entry **entry,
 	 */
 	rcu_assign_pointer(elem->funcs, (*entry)->funcs);
 	if (!elem->state && active)
-		enable_jump_label(elem->name);
+		enable_jump_label((jump_label_t)&elem->state);
 	if (elem->state && !active)
-		disable_jump_label(elem->name);
+		disable_jump_label((jump_label_t)&elem->state);
 	elem->state = active;
 }
 

@@ -276,7 +276,7 @@ static void disable_tracepoint(struct tracepoint *elem)
 		elem->unregfunc();
 
 	if (elem->state)
-		disable_jump_label(elem->name);
+		disable_jump_label((jump_label_t)&elem->state);
 
 	elem->state = 0;
 	rcu_assign_pointer(elem->funcs, NULL);
scripts/mod/modpost.c +2 -12
index fe6f8be..7fd693e 100644
@@ -390,19 +390,14 @@ static void sort_jump_label_table(struct elf_info *info, Elf_Ehdr *hdr)
 {
 	int swapped = 0;
 	struct jump_entry *iter, *iter_next;
-	char *name, *next_name;
 	Elf_Shdr *sechdrs = info->sechdrs;
 	unsigned long jump_table, jump_table_end;
-	unsigned long jump_strings, jump_strings_addr;
 
-	if ((info->jump_sec == 0) && (info->jump_strings_sec == 0))
+	if (info->jump_sec == 0)
 		return;
 
 	jump_table = (unsigned long)hdr + sechdrs[info->jump_sec].sh_offset;
 	jump_table_end = jump_table + sechdrs[info->jump_sec].sh_size;
-	jump_strings = (unsigned long)hdr +
-				sechdrs[info->jump_strings_sec].sh_offset;
-	jump_strings_addr = sechdrs[info->jump_strings_sec].sh_addr;
 
 	do {
 		swapped = 0;

@@ -410,10 +405,7 @@ static void sort_jump_label_table(struct elf_info *info, Elf_Ehdr *hdr)
 		iter_next++;
 		for (; iter_next < (struct jump_entry *)jump_table_end;
 							iter++, iter_next++) {
-			name = (char *)(jump_strings + (iter->name - jump_strings_addr));
-			next_name = (char *)(jump_strings +
-					(iter_next->name - jump_strings_addr));
-			if (strcmp(name, next_name) > 0) {
+			if (iter->key > iter_next->key) {
 				swap_jump_label_entries(iter, iter_next);
 				swapped = 1;
 			}

@@ -516,8 +508,6 @@ static int parse_elf(struct elf_info *info, const char *filename)
 			info->export_gpl_future_sec = i;
 		else if (strcmp(secname, "__jump_table") == 0)
 			info->jump_sec = i;
-		else if (strcmp(secname, "__jump_strings") == 0)
-			info->jump_strings_sec = i;
 
 		if (sechdrs[i].sh_type != SHT_SYMTAB)
 			continue;
scripts/mod/modpost.h +0 -1
index 0875d4b..362b966 100644
@@ -127,7 +127,6 @@ struct elf_info {
 	Elf_Section  export_unused_gpl_sec;
 	Elf_Section  export_gpl_future_sec;
 	Elf_Section  jump_sec;
-	Elf_Section  jump_strings_sec;
 	const char   *strtab;
 	char	     *modinfo;
 	unsigned int modinfo_len;
Signature · 1 line
1.7.0.1
author
#17 lore raw

PATCH 12/13 jump label v8: convert dynamic debug to use jump labels.

In reply to Jason Baron

Convert the 'dynamic debug' infrastructure to use jump labels.

Signed-off-by Jason Baron <[email protected]>
---
 include/linux/dynamic_debug.h |   42 ++++++++++++++-------------
 lib/dynamic_debug.c           |   42 ++-------------------------
 scripts/Makefile.lib          |   11 +------
 scripts/basic/Makefile        |    2 +-
 scripts/basic/hash.c          |   64 -----------------------------------------
 5 files changed, 27 insertions(+), 134 deletions(-)
 delete mode 100644 scripts/basic/hash.c
include/linux/dynamic_debug.h +22 -20
index f8c2e17..bb5dfaf 100644
@@ -1,6 +1,8 @@
 #ifndef _DYNAMIC_DEBUG_H
 #define _DYNAMIC_DEBUG_H
 
+#include <linux/jump_label.h>
+
 /* dynamic_printk_enabled, and dynamic_printk_enabled2 are bitmasks in which
  * bit n is set to 1 if any modname hashes into the bucket n, 0 otherwise. They
  * use independent hash functions, to reduce the chance of false positives.

@@ -22,9 +24,7 @@ struct _ddebug {
 	const char *function;
 	const char *filename;
 	const char *format;
-	char primary_hash;
-	char secondary_hash;
-	unsigned int lineno:24;
+	unsigned int lineno;
 	/*
  	 * The flags field controls the behaviour at the callsite.
  	 * The bits here are changed dynamically when the user

@@ -32,7 +32,7 @@ struct _ddebug {
 	 */
 #define _DPRINTK_FLAGS_PRINT   (1<<0)  /* printk() a message using the format */
 #define _DPRINTK_FLAGS_DEFAULT 0
-	unsigned int flags:8;
+	char flags;
 } __attribute__((aligned(8)));
 
 

@@ -42,33 +42,35 @@ int ddebug_add_module(struct _ddebug *tab, unsigned int n,
 #if defined(CONFIG_DYNAMIC_DEBUG)
 extern int ddebug_remove_module(char *mod_name);
 
-#define __dynamic_dbg_enabled(dd)  ({	     \
-	int __ret = 0;							     \
-	if (unlikely((dynamic_debug_enabled & (1LL << DEBUG_HASH)) &&	     \
-			(dynamic_debug_enabled2 & (1LL << DEBUG_HASH2))))   \
-				if (unlikely(dd.flags))			     \
-					__ret = 1;			     \
-	__ret; })
-
 #define dynamic_pr_debug(fmt, ...) do {					\
+	__label__ do_printk;						\
+	__label__ out;							\
 	static struct _ddebug descriptor				\
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
-	{ KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH,	\
-		DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT };	\
-	if (__dynamic_dbg_enabled(descriptor))				\
-		printk(KERN_DEBUG pr_fmt(fmt),	##__VA_ARGS__);		\
+	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
+		_DPRINTK_FLAGS_DEFAULT };				\
+	JUMP_LABEL(&descriptor.flags, do_printk);			\
+	goto out;							\
+do_printk:								\
+	printk(KERN_DEBUG pr_fmt(fmt),	##__VA_ARGS__);			\
+out:	;								\
 	} while (0)
 
 
 #define dynamic_dev_dbg(dev, fmt, ...) do {				\
+	__label__ do_printk;						\
+	__label__ out;							\
 	static struct _ddebug descriptor				\
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
-	{ KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH,	\
-		DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT };	\
-	if (__dynamic_dbg_enabled(descriptor))				\
-		dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__);	\
+	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
+		_DPRINTK_FLAGS_DEFAULT };				\
+	JUMP_LABEL(&descriptor.flags, do_printk);			\
+	goto out;							\
+do_printk:								\
+	dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__);		\
+out:	;								\
 	} while (0)
 
 #else
lib/dynamic_debug.c +3 -39

65 lines not on this page · raw message

scripts/Makefile.lib +1 -10
index cbcd654..6a162fc 100644
@@ -101,14 +101,6 @@ basename_flags = -D"KBUILD_BASENAME=KBUILD_STR($(call name-fix,$(basetarget)))"
 modname_flags  = $(if $(filter 1,$(words $(modname))),\
                  -D"KBUILD_MODNAME=KBUILD_STR($(call name-fix,$(modname)))")
 
-#hash values
-ifdef CONFIG_DYNAMIC_DEBUG
-debug_flags = -D"DEBUG_HASH=$(shell ./scripts/basic/hash djb2 $(@D)$(modname))"\
-              -D"DEBUG_HASH2=$(shell ./scripts/basic/hash r5 $(@D)$(modname))"
-else
-debug_flags =
-endif
-
 orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
                  $(ccflags-y) $(CFLAGS_$(basetarget).o)
 _c_flags       = $(filter-out $(CFLAGS_REMOVE_$(basetarget).o), $(orig_c_flags))

@@ -152,8 +144,7 @@ endif
 
 c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__c_flags) $(modkern_cflags)                           \
-		 -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags) \
-		  $(debug_flags)
+		 -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags)
 
 a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__a_flags) $(modkern_aflags)
scripts/basic/Makefile +1 -1
index 0955995..4c324a1 100644
@@ -9,7 +9,7 @@
 # fixdep: 	 Used to generate dependency information during build process
 # docproc:	 Used in Documentation/DocBook
 
-hostprogs-y	:= fixdep docproc hash
+hostprogs-y	:= fixdep docproc
 always		:= $(hostprogs-y)
 
 # fixdep is needed to compile other host programs
scripts/basic/hash.c deleted +0 -64

64 lines not on this page · raw message

Signature · 1 line
1.7.0.1
author
#18 lore raw

PATCH 13/13 jump label v8: sparc64 add jump_label support

In reply to Jason Baron

Add jump label support for sparc64.

Signed-off-by David S. Miller <[email protected]>
Signed-off-by Jason Baron <[email protected]>
---
 arch/sparc/Kconfig                  |    1 +
 arch/sparc/include/asm/jump_entry.h |   18 ++++++++++++++++
 arch/sparc/include/asm/jump_label.h |   20 ++++++++++++++++++
 arch/sparc/kernel/Makefile          |    2 +
 arch/sparc/kernel/jump_label.c      |   38 +++++++++++++++++++++++++++++++++++
 arch/sparc/kernel/module.c          |    6 +++++
 6 files changed, 85 insertions(+), 0 deletions(-)
 create mode 100644 arch/sparc/include/asm/jump_entry.h
 create mode 100644 arch/sparc/include/asm/jump_label.h
 create mode 100644 arch/sparc/kernel/jump_label.c
arch/sparc/Kconfig +1 -0
index d6781ce..0c581d4 100644
@@ -29,6 +29,7 @@ config SPARC
 	select PERF_USE_VMALLOC
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_API_DEBUG
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config SPARC32
 	def_bool !64BIT
arch/sparc/include/asm/jump_entry.h new file +18 -0
new file mode 100644
index 0000000..687452b
@@ -0,0 +1,18 @@
+#ifndef _LINUX_JUMP_ENTRY_H
+#define _LINUX_JUMP_ENTRY_H
+
+/* Defined in a separate file, so that it can be used by user program modpost.c
+ * This file is included by include/asm/jump_label.h, in turn included by
+ * include/linux/jump_label.h. Kernel sources should only include
+ * include/linux/jump_label.h.
+ **/
+
+typedef __u32 jump_label_t;
+
+struct jump_entry {
+	jump_label_t code;
+	jump_label_t target;
+	jump_label_t name;
+};
+
+#endif
arch/sparc/include/asm/jump_label.h new file +20 -0
new file mode 100644
index 0000000..70c7668
@@ -0,0 +1,20 @@
+#ifndef _ASM_SPARC_JUMP_LABEL_H
+#define _ASM_SPARC_JUMP_LABEL_H
+
+#include <linux/types.h>
+#include <asm/jump_entry.h>
+
+#define JUMP_LABEL_NOP_SIZE 4
+
+#define JUMP_LABEL(tag, label)					\
+	do {							\
+		asm goto("1:\n\t"				\
+			 "nop\n\t"				\
+			 "nop\n\t"				\
+			 ".pushsection __jump_table,  \"a\"\n\t"\
+			 ".xword 1b, %l[" #label "], %c0\n\t"	\
+			 ".popsection \n\t"			\
+			 : :  "i" (tag) :  : label);\
+	} while (0)
+
+#endif
arch/sparc/kernel/Makefile +2 -0
index 0c2dc1f..599398f 100644
@@ -119,3 +119,5 @@ obj-$(CONFIG_COMPAT)    += $(audit--y)
 
 pc--$(CONFIG_PERF_EVENTS) := perf_event.o
 obj-$(CONFIG_SPARC64)	+= $(pc--y)
+
+obj-$(CONFIG_SPARC64)	+= jump_label.o
arch/sparc/kernel/jump_label.c new file +38 -0
new file mode 100644
index 0000000..c1aa2d8
@@ -0,0 +1,38 @@
+#include <linux/kernel.h>
+#include <linux/types.h>
+#include <linux/mutex.h>
+#include <linux/cpu.h>
+
+#include <linux/jump_label.h>
+#include <linux/memory.h>
+
+#ifdef HAVE_JUMP_LABEL
+
+void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type)
+{
+	u32 val, *insn = (u32 *) entry->code;
+
+	val = *insn;
+	if (type == JUMP_LABEL_ENABLE) {
+		s32 off = (s32)entry->target - (s32)entry->code;
+		val = 0x40000000 | ((u32) off >> 2);
+	} else {
+		val = 0x01000000;
+	}
+
+	get_online_cpus();
+	mutex_lock(&text_mutex);
+	*insn = val;
+	flushi(insn);
+	mutex_unlock(&text_mutex);
+	put_online_cpus();
+}
+
+static const u8 jump_label_nop[JUMP_LABEL_NOP_SIZE] = { 0x01, 0x00, 0x00, 0x00 };
+
+const u8 *arch_get_jump_label_nop()
+{
+	return jump_label_nop;
+}
+
+#endif
arch/sparc/kernel/module.c +6 -0
index f848aad..37cf439 100644
@@ -18,6 +18,9 @@
 #include <asm/spitfire.h>
 
 #ifdef CONFIG_SPARC64
+
+#include <linux/jump_label.h>
+
 static void *module_map(unsigned long size)
 {
 	struct vm_struct *area;

@@ -227,6 +230,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 		    const Elf_Shdr *sechdrs,
 		    struct module *me)
 {
+	/* make jump label nops */
+	apply_jump_label_nops(me);
+
 	/* Cheetah's I-cache is fully coherent.  */
 	if (tlb_type == spitfire) {
 		unsigned long va;
Signature · 1 line
1.7.0.1

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

maintainer Acked-by
#23 lore raw

[ Added Michal to Cc ]

Quoted 61 lines · Jason Baron
Quoted 38 lines · Ingo Molnar
Quoted 28 lines · Jason Baron
Quoted 20 lines · Ingo Molnar

hi Jason,

Quoted 3 lines · Jason Baron

Hi,

updates for -v8:

I wanted to try your feature, but even after updating to latest rawhide i have:

lyra:~> rpm -q gcc
gcc-4.4.4-3.fc14.x86_64

and IIRC the asm goto feature is new in GCC 4.5, right?

Do you know when there's going to be a GCC 4.5 package for Fedora? I wanted to see how this feature works in practice, with a GCC packaged up for a distro.

Quoted 6 lines

Is there perhaps some repo somewhere with more experimental GCC builds for rawhide or so?

Thanks,

Ingo

Quoted 7 lines

Hi Ingo,

right. So the gcc feature is officially in gcc 4.5 which i believe has been released. however, it has been bacported to the latest fedora gcc even though its 4.4 based. Thus, if you apply something like like the following you can test it out on fedora.

Quoted 8 lines

Hm, could you please add a build-time check to check for the GCC feature instead? That way we'll be able to support any backports straight out of box.

We do this for various GCC features: stackprotector, cfi, etc., so it's a standard technique.

Ingo

ok, here's what we've come up with based on Ingo's suggestion. Any chance this feature can be considered for 2.6.35? I've a got a few other small fixes too...

thanks,

(quoted diff) +0 -1
-Jason
Makefile +0 -0
index 85edb67..b51ddfb 100644

Makefile KBUILD_CFLAGS += $(call cc-option,-fno-dwarf2-cfi-asm) · line 588

Quoted 7 lines
Makefile +4 -0
@@ -582,6 +582,11 @@ KBUILD_CFLAGS	+= $(call cc-option,-fno-dwarf2-cfi-asm)
 # conserve stack if available
 KBUILD_CFLAGS   += $(call cc-option,-fconserve-stack)
 
+# check for 'asm goto'
+ifeq ($(shell sh -c "(echo 'int main(void) { entry: asm goto (\"\"::::entry); return 0; }') | $(CC) -x c - >/dev/null 2>&1 && echo y"), y)
+	KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
+endif

Yeah, also this would help prevent bugs with other archs where the GCC version may not have supported it either.

Acked-by Steven Rostedt <[email protected]>

-- Steve

Quoted 23 lines
Makefile +1 -0
+
 # Add user supplied CPPFLAGS, AFLAGS and CFLAGS as the last assignments
 # But warn user when we do so
 warn-assign = \
include/linux/jump_label.h +3 -5
index 1eaf884..ebe19ae 100644
@@ -1,11 +1,9 @@
 #ifndef _LINUX_JUMP_LABEL_H
 #define _LINUX_JUMP_LABEL_H
 
-#if (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 5))
-# ifdef CONFIG_HAVE_ARCH_JUMP_LABEL
-#  include <asm/jump_label.h>
-#  define HAVE_JUMP_LABEL 1
-# endif
+#if defined(CC_HAVE_ASM_GOTO) && defined(CONFIG_HAVE_ARCH_JUMP_LABEL)
+# include <asm/jump_label.h>
+# define HAVE_JUMP_LABEL
 #endif
 
 enum jump_label_type {
#24 lore raw
Quoted 27 lines · Steven Rostedt

[ Added Michal to Cc ]

Quoted 14 lines

ok, here's what we've come up with based on Ingo's suggestion. Any chance this feature can be considered for 2.6.35? I've a got a few other small fixes too...

thanks,

(quoted diff) +0 -1
-Jason
Makefile +0 -0
index 85edb67..b51ddfb 100644

Makefile KBUILD_CFLAGS += $(call cc-option,-fno-dwarf2-cfi-asm) · line 588

Quoted 7 lines
Makefile +4 -0
@@ -582,6 +582,11 @@ KBUILD_CFLAGS	+= $(call cc-option,-fno-dwarf2-cfi-asm)
 # conserve stack if available
 KBUILD_CFLAGS   += $(call cc-option,-fconserve-stack)
 
+# check for 'asm goto'
+ifeq ($(shell sh -c "(echo 'int main(void) { entry: asm goto (\"\"::::entry); return 0; }') | $(CC) -x c - >/dev/null 2>&1 && echo y"), y)
+	KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
+endif

Yeah, also this would help prevent bugs with other archs where the GCC version may not have supported it either.

Quoted 1 line
Acked-by Steven Rostedt <[email protected]>

Could we move this to a small well-commented helper script like we do with gcc-version.sh?

Then we could simply do something like this in the top-level Makefile:

# Check if gcc support' asm goto'

ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC))),y)

    KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO

endif

The top-level Makefile needs to be simpler - not more complicated.

PS. I think Michal is away until the end of this month.

Sam

maintainer
#25 lore raw
Quoted 6 lines · Sam Ravnborg

On Tue, May 25, 2010 at 03:39:29PM -0400, Steven Rostedt wrote:

Could we move this to a small well-commented helper script like we do with gcc-version.sh?

Then we could simply do something like this in the top-level Makefile:

I'm fine with this idea. We just want to get rid of the binding a feature to a version number.

Quoted 10 lines

# Check if gcc support' asm goto'

ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC))),y)

    KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO

endif

The top-level Makefile needs to be simpler - not more complicated.

PS. I think Michal is away until the end of this month.

Yeah, I got his away-from-office message.

-- Steve