Skip to contentSkip to conversation

jump label: add jump label code

GIT PULL In review Steven Rostedt 5 participants 29 messages Sep 23, 2010 – Sep 24, 2010

Subsystems (11)
x86 Kbuild Library code Tracing +7 more subsystems
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

Ingo,

I finally got around to doing some basic tests. The thing that took the longest, was setting up all my distcc computers with gcc 2.5.1. I also hit a little glitch in one of my tests that would trigger RCU stalls, but that's another story.

Anyway, I took Jason's patches and applied them, with a few formatting clean ups (basically, shortened function prototypes).

I added two patches at the end of the series to remove the duplicate structure and also to remove the dependency on !CC_OPTIMIZE_FOR_SIZE.

Andi,

I saw your patches but have not had time to take a deeper look at them. If you want, you can rebase them against this tree.

Thanks,

Please pull the latest tip/perf/core tree, which can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/perf/core

David S. Miller (1):
jump label: Add sparc64 support

Jason Baron (8):
jump label: Make dynamic no-op selection available outside of ftrace jump label: Make text_poke_early() globally visible
jump label: Base patch for jump label
jump label: Initialize workqueue tracepoints *before* they are registered jump label: Add jump_label_text_reserved() to reserve jump points jump label: Tracepoint support for jump labels
jump label: Convert dynamic debug to use jump labels
jump label: x86 support

Steven Rostedt (2):
jump label: Remove duplicate structure for x86
jump label/x86/sparc64: Remove !CC_OPTIMIZE_FOR_SIZE config conditions

----

 Makefile                            |    5 +
 arch/Kconfig                        |    3 +
 arch/sparc/Kconfig                  |    1 +
 arch/sparc/include/asm/jump_label.h |   32 +++
 arch/sparc/kernel/Makefile          |    2 +
 arch/sparc/kernel/jump_label.c      |   47 ++++
 arch/sparc/kernel/module.c          |    6 +
 arch/x86/Kconfig                    |    1 +
 arch/x86/include/asm/alternative.h  |   11 +
 arch/x86/include/asm/jump_label.h   |   37 +++
 arch/x86/kernel/Makefile            |    2 +-
 arch/x86/kernel/alternative.c       |   68 ++++++-
 arch/x86/kernel/ftrace.c            |   63 +-----
 arch/x86/kernel/jump_label.c        |   50 ++++
 arch/x86/kernel/kprobes.c           |    3 +-
 arch/x86/kernel/module.c            |    3 +
 arch/x86/kernel/setup.c             |    6 +
 include/asm-generic/vmlinux.lds.h   |   10 +
 include/linux/dynamic_debug.h       |   39 ++--
 include/linux/jump_label.h          |   64 ++++++
 include/linux/module.h              |    5 +-
 include/linux/tracepoint.h          |    5 +-
 kernel/Makefile                     |    2 +-
 kernel/jump_label.c                 |  429 +++++++++++++++++++++++++++++++++++
 kernel/kprobes.c                    |    4 +-
 kernel/module.c                     |    6 +
 kernel/trace/trace_workqueue.c      |   10 +-
 kernel/tracepoint.c                 |   14 +-
 lib/dynamic_debug.c                 |   42 +----
 scripts/Makefile.lib                |   11 +-
 scripts/basic/Makefile              |    2 +-
 scripts/basic/hash.c                |   64 ------
 scripts/gcc-goto.sh                 |    5 +
 33 files changed, 843 insertions(+), 209 deletions(-)
author maintainer
#1 lore raw

PATCH 1/11 jump label: Make dynamic no-op selection available outside of ftrace

From: Jason Baron <[email protected]>

Move Steve's code for finding the best 5-byte no-op from ftrace.c to alternative.c. The idea is that other consumers (in this case jump label) want to make use of that code.

LKML-Reference: <96259ae74172dcac99c0020c249743c523a92e18.1284733808.git.jbaron@redhat.com>

---
 arch/x86/include/asm/alternative.h |    8 ++++
 arch/x86/kernel/alternative.c      |   64 ++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/ftrace.c           |   63 +----------------------------------
 arch/x86/kernel/setup.c            |    6 +++
 4 files changed, 79 insertions(+), 62 deletions(-)
arch/x86/include/asm/alternative.h +8 -0
index bc6abb7..27a35b6 100644
@@ -180,4 +180,12 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 extern void *text_poke(void *addr, const void *opcode, size_t len);
 extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
 
+#if defined(CONFIG_DYNAMIC_FTRACE)
+#define IDEAL_NOP_SIZE_5 5
+extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
+extern void arch_init_ideal_nop5(void);
+#else
+static inline void arch_init_ideal_nop5(void) {}
+#endif
+
 #endif /* _ASM_X86_ALTERNATIVE_H */
arch/x86/kernel/alternative.c +64 -0

67 lines not on this page · raw message

arch/x86/kernel/ftrace.c +1 -62

77 lines not on this page · raw message

arch/x86/kernel/setup.c +6 -0
index c3a4fbb..00e1678 100644
@@ -112,6 +112,7 @@
 #include <asm/numa_64.h>
 #endif
 #include <asm/mce.h>
+#include <asm/alternative.h>
 
 /*
  * end_pfn only includes RAM, while max_pfn_mapped includes all e820 entries.
@@ -726,6 +727,7 @@ void __init setup_arch(char **cmdline_p)
 {
 	int acpi = 0;
 	int k8 = 0;
+	unsigned long flags;
 
 #ifdef CONFIG_X86_32
 	memcpy(&boot_cpu_data, &new_cpu_data, sizeof(new_cpu_data));
@@ -1071,6 +1073,10 @@ void __init setup_arch(char **cmdline_p)
 	x86_init.oem.banner();
 
 	mcheck_init();
+
+	local_irq_save(flags);
+	arch_init_ideal_nop5();
+	local_irq_restore(flags);
 }
 
 #ifdef CONFIG_X86_32
Signature · 1 line
1.7.1
author maintainer
#2 lore raw

PATCH 2/11 jump label: Make text_poke_early() globally visible

In reply to Steven Rostedt

From: Jason Baron <[email protected]>

Make text_poke_early available outside of alternative.c. The jump label patchset wants to make use of it in order to set up the optimal no-op sequences at run-time.

LKML-Reference: <04cfddf2ba77bcabfc3e524f1849d871d6a1cf9d.1284733808.git.jbaron@redhat.com>

---
 arch/x86/include/asm/alternative.h |    2 ++
 arch/x86/kernel/alternative.c      |    4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)
arch/x86/include/asm/alternative.h +2 -0
index 27a35b6..634bf78 100644
@@ -160,6 +160,8 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,
 #define __parainstructions_end	NULL
 #endif
 
+extern void *text_poke_early(void *addr, const void *opcode, size_t len);
+
 /*
  * Clear and restore the kernel write-protection flag on the local CPU.
  * Allows the kernel to edit read-only pages.
arch/x86/kernel/alternative.c +2 -2
index 1849d80..083bd01 100644
@@ -195,7 +195,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len)
 
 extern struct alt_instr __alt_instructions[], __alt_instructions_end[];
 extern s32 __smp_locks[], __smp_locks_end[];
-static void *text_poke_early(void *addr, const void *opcode, size_t len);
+void *text_poke_early(void *addr, const void *opcode, size_t len);
 
 /* Replace instructions with better alternatives for this CPU type.
    This runs before SMP is initialized to avoid SMP problems with
@@ -522,7 +522,7 @@ void __init alternative_instructions(void)
  * instructions. And on the local CPU you need to be protected again NMI or MCE
  * handlers seeing an inconsistent instruction while you patch.
  */
-static void *__init_or_module text_poke_early(void *addr, const void *opcode,
+void *__init_or_module text_poke_early(void *addr, const void *opcode,
 					      size_t len)
 {
 	unsigned long flags;
Signature · 1 line
1.7.1
author maintainer
#3 lore raw

PATCH 3/11 jump label: Base patch for jump label

In reply to Steven Rostedt

From: Jason Baron <[email protected]>

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

LKML-Reference: <ee8b3595967989fdaf84e698dc7447d315ce972a.1284733808.git.jbaron@redhat.com>

[ cleaned up some formating ]

---
 Makefile                           |    5 +
 arch/Kconfig                       |    3 +
 arch/x86/include/asm/alternative.h |    3 +-
 arch/x86/kernel/alternative.c      |    2 +-
 include/asm-generic/vmlinux.lds.h  |   10 +
 include/linux/jump_label.h         |   58 ++++++
 include/linux/module.h             |    5 +-
 kernel/Makefile                    |    2 +-
 kernel/jump_label.c                |  346 ++++++++++++++++++++++++++++++++++++
 kernel/kprobes.c                   |    1 +
 kernel/module.c                    |    6 +
 scripts/gcc-goto.sh                |    5 +
 12 files changed, 442 insertions(+), 4 deletions(-)
 create mode 100644 include/linux/jump_label.h
 create mode 100644 kernel/jump_label.c
 create mode 100644 scripts/gcc-goto.sh
Makefile +5 -0
index 92ab33f..a906378 100644
@@ -591,6 +591,11 @@ KBUILD_CFLAGS	+= $(call cc-option,-fno-strict-overflow)
 # conserve stack if available
 KBUILD_CFLAGS   += $(call cc-option,-fconserve-stack)
 
+# check for 'asm goto'
+ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC)), y)
+	KBUILD_CFLAGS += -DCC_HAVE_ASM_GOTO
+endif
+
 # Add user supplied CPPFLAGS, AFLAGS and CFLAGS as the last assignments
 # But warn user when we do so
 warn-assign = \
arch/Kconfig +3 -0
index 4877a8c..1462d84 100644
@@ -158,4 +158,7 @@ config HAVE_PERF_EVENTS_NMI
 	  subsystem.  Also has support for calculating CPU cycle events
 	  to determine how many clock cycles in a given period.
 
+config HAVE_ARCH_JUMP_LABEL
+	bool
+
 source "kernel/gcov/Kconfig"
arch/x86/include/asm/alternative.h +2 -1
index 634bf78..76561d2 100644
@@ -4,6 +4,7 @@
 #include <linux/types.h>
 #include <linux/stddef.h>
 #include <linux/stringify.h>
+#include <linux/jump_label.h>
 #include <asm/asm.h>
 
 /*
@@ -182,7 +183,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);
 extern void *text_poke(void *addr, const void *opcode, size_t len);
 extern void *text_poke_smp(void *addr, const void *opcode, size_t len);
 
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
 #define IDEAL_NOP_SIZE_5 5
 extern unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
 extern void arch_init_ideal_nop5(void);
arch/x86/kernel/alternative.c +1 -1
index 083bd01..cb0e6d3 100644
@@ -641,7 +641,7 @@ void *__kprobes text_poke_smp(void *addr, const void *opcode, size_t len)
 	return addr;
 }
 
-#if defined(CONFIG_DYNAMIC_FTRACE)
+#if defined(CONFIG_DYNAMIC_FTRACE) || defined(HAVE_JUMP_LABEL)
 
 unsigned char ideal_nop5[IDEAL_NOP_SIZE_5];
 
include/asm-generic/vmlinux.lds.h +10 -0
index 8a92a17..ef2af99 100644
@@ -220,6 +220,8 @@
 									\
 	BUG_TABLE							\
 									\
+	JUMP_TABLE							\
+									\
 	/* PCI quirks */						\
 	.pci_fixup        : AT(ADDR(.pci_fixup) - LOAD_OFFSET) {	\
 		VMLINUX_SYMBOL(__start_pci_fixups_early) = .;		\
@@ -563,6 +565,14 @@
 #define BUG_TABLE
 #endif
 
+#define JUMP_TABLE							\
+	. = ALIGN(8);							\
+	__jump_table : AT(ADDR(__jump_table) - LOAD_OFFSET) {		\
+		VMLINUX_SYMBOL(__start___jump_table) = .;		\
+		*(__jump_table)						\
+		VMLINUX_SYMBOL(__stop___jump_table) = .;		\
+	}
+
 #ifdef CONFIG_PM_TRACE
 #define TRACEDATA							\
 	. = ALIGN(4);							\
include/linux/jump_label.h new file +58 -0

58 lines not on this page · raw message

include/linux/module.h +4 -1
index 8a6b9fd..403ac26 100644
@@ -350,7 +350,10 @@ struct module
 	struct tracepoint *tracepoints;
 	unsigned int num_tracepoints;
 #endif
-
+#ifdef HAVE_JUMP_LABEL
+	struct jump_entry *jump_entries;
+	unsigned int num_jump_entries;
+#endif
 #ifdef CONFIG_TRACING
 	const char **trace_bprintk_fmt_start;
 	unsigned int num_trace_bprintk_fmt;
kernel/Makefile +1 -1
index 0b72d1a..d52b473 100644
@@ -10,7 +10,7 @@ obj-y     = sched.o fork.o exec_domain.o panic.o printk.o \
 	    kthread.o wait.o kfifo.o sys_ni.o posix-cpu-timers.o mutex.o \
 	    hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
 	    notifier.o ksysfs.o pm_qos_params.o sched_clock.o cred.o \
-	    async.o range.o
+	    async.o range.o jump_label.o
 obj-$(CONFIG_HAVE_EARLY_RES) += early_res.o
 obj-y += groups.o
 
kernel/jump_label.c new file +346 -0

346 lines not on this page · raw message

kernel/kprobes.c +1 -0
index 6dd5359..18904e4 100644
@@ -47,6 +47,7 @@
 #include <linux/memory.h>
 #include <linux/ftrace.h>
 #include <linux/cpu.h>
+#include <linux/jump_label.h>
 
 #include <asm-generic/sections.h>
 #include <asm/cacheflush.h>
kernel/module.c +6 -0
index d0b5f8d..eba1341 100644
@@ -55,6 +55,7 @@
 #include <linux/async.h>
 #include <linux/percpu.h>
 #include <linux/kmemleak.h>
+#include <linux/jump_label.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/module.h>
@@ -2308,6 +2309,11 @@ static void find_module_sections(struct module *mod, struct load_info *info)
 					sizeof(*mod->tracepoints),
 					&mod->num_tracepoints);
 #endif
+#ifdef HAVE_JUMP_LABEL
+	mod->jump_entries = section_objs(info, "__jump_table",
+					sizeof(*mod->jump_entries),
+					&mod->num_jump_entries);
+#endif
 #ifdef CONFIG_EVENT_TRACING
 	mod->trace_events = section_objs(info, "_ftrace_events",
 					 sizeof(*mod->trace_events),
scripts/gcc-goto.sh new file +5 -0
new file mode 100644
index 0000000..8e82424
@@ -0,0 +1,5 @@
+#!/bin/sh
+# Test for gcc 'asm goto' suport
+# Copyright (C) 2010, Jason Baron <[email protected]>
+
+echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
Signature · 1 line
1.7.1
#4 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

Quoted 9 lines · * Steven Rostedt ([email protected]) wrote:

From: Jason Baron <[email protected]>

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

[...]

Quoted 15 lines
kernel/jump_label.c new file +15 -0
+/***
+ * jump_label_update - update jump label text
+ * @key -  key value associated with a a jump label
+ * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
+ *
+ * Will enable/disable the jump for jump label @key, depending on the
+ * value of @type.
+ *
+ */
+
+void jump_label_update(unsigned long key, enum jump_label_type type)
+{
+	struct jump_entry *iter;
+	struct jump_label_entry *entry;
+	struct hlist_node *module_node;

kernel/jump_label.c line 161

Quoted 10 lines
kernel/jump_label.c new file +10 -0
+	struct jump_label_module_entry *e_module;
+	int count;
+
+	mutex_lock(&jump_label_mutex);
+	entry = get_jump_label_entry((jump_label_t)key);
+	if (entry) {
+		count = entry->nr_entries;
+		iter = entry->table;
+		while (count--) {
+			if (kernel_text_address(iter->code))

As I pointed out in another thread, I'm concerned about the use of kernel_text_address without module mutex here. kernel_text_address calls is_module_text_address(), which calls __module_text_address() with preemption off.

__module_text_address() looks like:

struct module *__module_address(unsigned long addr)

{
        struct module *mod;

        if (addr < module_addr_min || addr > module_addr_max)
                return NULL;

        list_for_each_entry_rcu(mod, &modules, list)
                if (within_module_core(addr, mod)
                    || within_module_init(addr, mod))
                        return mod;
        return NULL;
}

struct module *__module_text_address(unsigned long addr)

{
        struct module *mod = __module_address(addr);
        if (mod) {
                /* Make sure it's within the text section. */
                if (!within(addr, mod->module_init, mod->init_text_size)
                    && !within(addr, mod->module_core, mod->core_text_size))
                        mod = NULL;
        }
        return mod;
}

So the test for the address being in the module core is already problematic, since we hold preempt off only within is_module_text_address(). The is_module_text_address() caller is then free to write to this address even after the module has been unloaded and the module unload grace period ended.

Even worse, such grace period is not waited for at module load time within:

init_module()

       module_free(mod, mod->module_init);
       mod->module_init = NULL;
       mod->init_size = 0;
       mod->init_text_size = 0;

(done with module_mutex held, while the module is already in the
module list)

We'd probably have to hold the module mutex around the is_module_text_address() call and address use (which can be a pain), or to correctly address this part of init_module() with RCU and require that preempt off is held across both __module_text_address() call site and the actual use of that pointer (which does not fit with jump label, which need to sleep, so we'd have to move module.c to a preemptable rcu_read_lock/synchronize_rcu() C.S.).

Thoughts ?

Mathieu

Quoted 16 lines
kernel/jump_label.c new file +16 -0
+				arch_jump_label_transform(iter, type);
+			iter++;
+		}
+		/* eanble/disable jump labels in modules */
+		hlist_for_each_entry(e_module, module_node, &(entry->modules),
+							hlist) {
+			count = e_module->nr_entries;
+			iter = e_module->table;
+			while (count--) {
+				if (kernel_text_address(iter->code))
+					arch_jump_label_transform(iter, type);
+				iter++;
+			}
+		}
+	}
+	mutex_unlock(&jump_label_mutex);
Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#5 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

Quoted 95 lines · Mathieu Desnoyers
Quoted 9 lines · * Steven Rostedt ([email protected]) wrote:

From: Jason Baron <[email protected]>

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

[...]

Quoted 15 lines
kernel/jump_label.c new file +15 -0
+/***
+ * jump_label_update - update jump label text
+ * @key -  key value associated with a a jump label
+ * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
+ *
+ * Will enable/disable the jump for jump label @key, depending on the
+ * value of @type.
+ *
+ */
+
+void jump_label_update(unsigned long key, enum jump_label_type type)
+{
+	struct jump_entry *iter;
+	struct jump_label_entry *entry;
+	struct hlist_node *module_node;

kernel/jump_label.c line 161

Quoted 10 lines
kernel/jump_label.c new file +10 -0
+	struct jump_label_module_entry *e_module;
+	int count;
+
+	mutex_lock(&jump_label_mutex);
+	entry = get_jump_label_entry((jump_label_t)key);
+	if (entry) {
+		count = entry->nr_entries;
+		iter = entry->table;
+		while (count--) {
+			if (kernel_text_address(iter->code))

As I pointed out in another thread, I'm concerned about the use of kernel_text_address without module mutex here. kernel_text_address calls is_module_text_address(), which calls __module_text_address() with preemption off.

__module_text_address() looks like:

struct module *__module_address(unsigned long addr)

{
        struct module *mod;

        if (addr < module_addr_min || addr > module_addr_max)
                return NULL;

        list_for_each_entry_rcu(mod, &modules, list)
                if (within_module_core(addr, mod)
                    || within_module_init(addr, mod))
                        return mod;
        return NULL;
}

struct module *__module_text_address(unsigned long addr)

{
        struct module *mod = __module_address(addr);
        if (mod) {
                /* Make sure it's within the text section. */
                if (!within(addr, mod->module_init, mod->init_text_size)
                    && !within(addr, mod->module_core, mod->core_text_size))
                        mod = NULL;
        }
        return mod;
}

So the test for the address being in the module core is already problematic, since we hold preempt off only within is_module_text_address(). The is_module_text_address() caller is then free to write to this address even after the module has been unloaded and the module unload grace period ended.

Even worse, such grace period is not waited for at module load time within:

init_module()

       module_free(mod, mod->module_init);
       mod->module_init = NULL;
       mod->init_size = 0;
       mod->init_text_size = 0;

(done with module_mutex held, while the module is already in the
module list)

We'd probably have to hold the module mutex around the is_module_text_address() call and address use (which can be a pain), or to correctly address this part of init_module() with RCU and require that preempt off is held across both __module_text_address() call site and the actual use of that pointer (which does not fit with jump label, which need to sleep, so we'd have to move module.c to a preemptable rcu_read_lock/synchronize_rcu() C.S.).

Quoted 2 lines

Thoughts ?

I was thinking about the rcu_read_lock/synchronize_rcu() for this race. We can hold the rcu_read_lock() across the is_module_text_address() check in the jump label code, and then we can do in module.c:

mod->module_init = NULL;
synchronize_rcu();
module_free(mod, mod->module_init);
.
.
.

or we could push the rcu_read_lock() further down into is_module_address()?

thanks,

-Jason

#6 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

Quoted 99 lines · * Jason Baron ([email protected]) wrote:
Quoted 95 lines · Mathieu Desnoyers
Quoted 9 lines · * Steven Rostedt ([email protected]) wrote:

From: Jason Baron <[email protected]>

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

[...]

Quoted 15 lines
kernel/jump_label.c new file +15 -0
+/***
+ * jump_label_update - update jump label text
+ * @key -  key value associated with a a jump label
+ * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
+ *
+ * Will enable/disable the jump for jump label @key, depending on the
+ * value of @type.
+ *
+ */
+
+void jump_label_update(unsigned long key, enum jump_label_type type)
+{
+	struct jump_entry *iter;
+	struct jump_label_entry *entry;
+	struct hlist_node *module_node;

kernel/jump_label.c line 161

Quoted 10 lines
kernel/jump_label.c new file +10 -0
+	struct jump_label_module_entry *e_module;
+	int count;
+
+	mutex_lock(&jump_label_mutex);
+	entry = get_jump_label_entry((jump_label_t)key);
+	if (entry) {
+		count = entry->nr_entries;
+		iter = entry->table;
+		while (count--) {
+			if (kernel_text_address(iter->code))

As I pointed out in another thread, I'm concerned about the use of kernel_text_address without module mutex here. kernel_text_address calls is_module_text_address(), which calls __module_text_address() with preemption off.

__module_text_address() looks like:

struct module *__module_address(unsigned long addr)

{
        struct module *mod;

        if (addr < module_addr_min || addr > module_addr_max)
                return NULL;

        list_for_each_entry_rcu(mod, &modules, list)
                if (within_module_core(addr, mod)
                    || within_module_init(addr, mod))
                        return mod;
        return NULL;
}

struct module *__module_text_address(unsigned long addr)

{
        struct module *mod = __module_address(addr);
        if (mod) {
                /* Make sure it's within the text section. */
                if (!within(addr, mod->module_init, mod->init_text_size)
                    && !within(addr, mod->module_core, mod->core_text_size))
                        mod = NULL;
        }
        return mod;
}

So the test for the address being in the module core is already problematic, since we hold preempt off only within is_module_text_address(). The is_module_text_address() caller is then free to write to this address even after the module has been unloaded and the module unload grace period ended.

Even worse, such grace period is not waited for at module load time within:

init_module()

       module_free(mod, mod->module_init);
       mod->module_init = NULL;
       mod->init_size = 0;
       mod->init_text_size = 0;

(done with module_mutex held, while the module is already in the
module list)

We'd probably have to hold the module mutex around the is_module_text_address() call and address use (which can be a pain), or to correctly address this part of init_module() with RCU and require that preempt off is held across both __module_text_address() call site and the actual use of that pointer (which does not fit with jump label, which need to sleep, so we'd have to move module.c to a preemptable rcu_read_lock/synchronize_rcu() C.S.).

Quoted 2 lines

Thoughts ?

Quoted 7 lines

I was thinking about the rcu_read_lock/synchronize_rcu() for this race. We can hold the rcu_read_lock() across the is_module_text_address() check in the jump label code, and then we can do in module.c:

mod->module_init = NULL;
synchronize_rcu();
module_free(mod, mod->module_init);

Beware that you need to copy the module_init address. Also make sure you audit the "module_free" (per-architecture) to make sure they don't use "mod" in ways you did not foresee.

Quoted 6 lines

.
.
.

or we could push the rcu_read_lock() further down into is_module_address()?

We need to pull rcu_read_lock further _up_. It needs to be held across both is_module_address() and the actual use of the address, otherwise the memory mapping can be removed underneath us.

You can see the rcu read lock as keeping the memory mapping alive for as long as the rcu read lock is held.

We'd also need to add a synchronize_rcu() in module removal.

Thanks,

Mathieu

Quoted 6 lines

thanks,

-Jason

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#7 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

Quoted 119 lines · Mathieu Desnoyers
Quoted 99 lines · * Jason Baron ([email protected]) wrote:
Quoted 95 lines · Mathieu Desnoyers
Quoted 9 lines · * Steven Rostedt ([email protected]) wrote:

From: Jason Baron <[email protected]>

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

[...]

Quoted 15 lines
(quoted diff) +15 -0
+/***
+ * jump_label_update - update jump label text
+ * @key -  key value associated with a a jump label
+ * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
+ *
+ * Will enable/disable the jump for jump label @key, depending on the
+ * value of @type.
+ *
+ */
+
+void jump_label_update(unsigned long key, enum jump_label_type type)
+{
+	struct jump_entry *iter;
+	struct jump_label_entry *entry;
+	struct hlist_node *module_node;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	struct jump_label_module_entry *e_module;
+	int count;
+
+	mutex_lock(&jump_label_mutex);
+	entry = get_jump_label_entry((jump_label_t)key);
+	if (entry) {
+		count = entry->nr_entries;
+		iter = entry->table;
+		while (count--) {
+			if (kernel_text_address(iter->code))

As I pointed out in another thread, I'm concerned about the use of kernel_text_address without module mutex here. kernel_text_address calls is_module_text_address(), which calls __module_text_address() with preemption off.

__module_text_address() looks like:

struct module *__module_address(unsigned long addr)

{
        struct module *mod;

        if (addr < module_addr_min || addr > module_addr_max)
                return NULL;

        list_for_each_entry_rcu(mod, &modules, list)
                if (within_module_core(addr, mod)
                    || within_module_init(addr, mod))
                        return mod;
        return NULL;
}

struct module *__module_text_address(unsigned long addr)

{
        struct module *mod = __module_address(addr);
        if (mod) {
                /* Make sure it's within the text section. */
                if (!within(addr, mod->module_init, mod->init_text_size)
                    && !within(addr, mod->module_core, mod->core_text_size))
                        mod = NULL;
        }
        return mod;
}

So the test for the address being in the module core is already problematic, since we hold preempt off only within is_module_text_address(). The is_module_text_address() caller is then free to write to this address even after the module has been unloaded and the module unload grace period ended.

Even worse, such grace period is not waited for at module load time within:

init_module()

       module_free(mod, mod->module_init);
       mod->module_init = NULL;
       mod->init_size = 0;
       mod->init_text_size = 0;

(done with module_mutex held, while the module is already in the
module list)

We'd probably have to hold the module mutex around the is_module_text_address() call and address use (which can be a pain), or to correctly address this part of init_module() with RCU and require that preempt off is held across both __module_text_address() call site and the actual use of that pointer (which does not fit with jump label, which need to sleep, so we'd have to move module.c to a preemptable rcu_read_lock/synchronize_rcu() C.S.).

Quoted 2 lines

Thoughts ?

Quoted 7 lines

I was thinking about the rcu_read_lock/synchronize_rcu() for this race. We can hold the rcu_read_lock() across the is_module_text_address() check in the jump label code, and then we can do in module.c:

mod->module_init = NULL;
synchronize_rcu();
module_free(mod, mod->module_init);

Beware that you need to copy the module_init address. Also make sure you audit the "module_free" (per-architecture) to make sure they don't use "mod" in ways you did not foresee.

Quoted 6 lines

.
.
.

or we could push the rcu_read_lock() further down into is_module_address()?

Quoted 9 lines

We need to pull rcu_read_lock further _up_. It needs to be held across both is_module_address() and the actual use of the address, otherwise the memory mapping can be removed underneath us.

You can see the rcu read lock as keeping the memory mapping alive for as long as the rcu read lock is held.

We'd also need to add a synchronize_rcu() in module removal.

I agree that we this synchronization for the module __init section.

However, I believe we are ok for module removal case. free_module() is called *after* blocking_notifier_call_chain() call. The blocking_notifier_call_chain() is going to call back into the jump label code, grab the jump_label_mutex and remove the reference to the module that is about to freed. Thus, the jump label code can no longer reference it.

So I think the following patch is all that is required here (lightly tested).

Steve, I'll re-post as a separate patch, if we agree on this fix.

thanks,

-Jason

jump label: fix __init module section race

Jump label uses is_module_text_address() to ensure that the module __init sections are valid before updating them. However, between the check for a valid module __init section and the subsequent jump label update, the module's __init section could be free out from under us.

We fix this potential race putting the address check *and* the jump label update under a rcu_read_lock(), and making sure a grace period has completed before we free the __init section.

Thanks to Mathieu Desnoyers for pointing out this race condition.

---
 kernel/jump_label.c |    2 ++
 kernel/module.c     |    5 ++++-
 2 files changed, 6 insertions(+), 1 deletions(-)
kernel/jump_label.c +2 -0
index f82878b..7830bfb 100644
@@ -160,6 +160,7 @@ void jump_label_update(unsigned long key, enum jump_label_type type)
 			iter++;
 		}
 		/* eanble/disable jump labels in modules */
+		rcu_read_lock();
 		hlist_for_each_entry(e_module, module_node, &(entry->modules),
 							hlist) {
 			count = e_module->nr_entries;
@@ -170,6 +171,7 @@ void jump_label_update(unsigned long key, enum jump_label_type type)
 				iter++;
 			}
 		}
+		rcu_read_unlock();
 	}
 	mutex_unlock(&jump_label_mutex);
 }
kernel/module.c +4 -1
index eba1341..09f7e9e 100644
@@ -2692,6 +2692,7 @@ SYSCALL_DEFINE3(init_module, void __user *, umod,
 		unsigned long, len, const char __user *, uargs)
 {
 	struct module *mod;
+	void *init_code;
 	int ret = 0;
 
 	/* Must have permission */
@@ -2749,8 +2750,10 @@ SYSCALL_DEFINE3(init_module, void __user *, umod,
 	mod->symtab = mod->core_symtab;
 	mod->strtab = mod->core_strtab;
 #endif
-	module_free(mod, mod->module_init);
+	init_code = mod->module_init;
 	mod->module_init = NULL;
+	synchronize_rcu();
+	module_free(mod, init_code);
 	mod->init_size = 0;
 	mod->init_text_size = 0;
 	mutex_unlock(&module_mutex);
Signature · 1 line
1.7.1
#8 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

Quoted 130 lines · * Jason Baron ([email protected]) wrote:
Quoted 119 lines · Mathieu Desnoyers
Quoted 99 lines · * Jason Baron ([email protected]) wrote:
Quoted 95 lines · Mathieu Desnoyers
Quoted 9 lines · * Steven Rostedt ([email protected]) wrote:

From: Jason Baron <[email protected]>

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

[...]

Quoted 15 lines
(quoted diff) +15 -0
+/***
+ * jump_label_update - update jump label text
+ * @key -  key value associated with a a jump label
+ * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
+ *
+ * Will enable/disable the jump for jump label @key, depending on the
+ * value of @type.
+ *
+ */
+
+void jump_label_update(unsigned long key, enum jump_label_type type)
+{
+	struct jump_entry *iter;
+	struct jump_label_entry *entry;
+	struct hlist_node *module_node;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	struct jump_label_module_entry *e_module;
+	int count;
+
+	mutex_lock(&jump_label_mutex);
+	entry = get_jump_label_entry((jump_label_t)key);
+	if (entry) {
+		count = entry->nr_entries;
+		iter = entry->table;
+		while (count--) {
+			if (kernel_text_address(iter->code))

As I pointed out in another thread, I'm concerned about the use of kernel_text_address without module mutex here. kernel_text_address calls is_module_text_address(), which calls __module_text_address() with preemption off.

__module_text_address() looks like:

struct module *__module_address(unsigned long addr)

{
        struct module *mod;

        if (addr < module_addr_min || addr > module_addr_max)
                return NULL;

        list_for_each_entry_rcu(mod, &modules, list)
                if (within_module_core(addr, mod)
                    || within_module_init(addr, mod))
                        return mod;
        return NULL;
}

struct module *__module_text_address(unsigned long addr)

{
        struct module *mod = __module_address(addr);
        if (mod) {
                /* Make sure it's within the text section. */
                if (!within(addr, mod->module_init, mod->init_text_size)
                    && !within(addr, mod->module_core, mod->core_text_size))
                        mod = NULL;
        }
        return mod;
}

So the test for the address being in the module core is already problematic, since we hold preempt off only within is_module_text_address(). The is_module_text_address() caller is then free to write to this address even after the module has been unloaded and the module unload grace period ended.

Even worse, such grace period is not waited for at module load time within:

init_module()

       module_free(mod, mod->module_init);
       mod->module_init = NULL;
       mod->init_size = 0;
       mod->init_text_size = 0;

(done with module_mutex held, while the module is already in the
module list)

We'd probably have to hold the module mutex around the is_module_text_address() call and address use (which can be a pain), or to correctly address this part of init_module() with RCU and require that preempt off is held across both __module_text_address() call site and the actual use of that pointer (which does not fit with jump label, which need to sleep, so we'd have to move module.c to a preemptable rcu_read_lock/synchronize_rcu() C.S.).

Quoted 2 lines

Thoughts ?

Quoted 7 lines

I was thinking about the rcu_read_lock/synchronize_rcu() for this race. We can hold the rcu_read_lock() across the is_module_text_address() check in the jump label code, and then we can do in module.c:

mod->module_init = NULL;
synchronize_rcu();
module_free(mod, mod->module_init);

Beware that you need to copy the module_init address. Also make sure you audit the "module_free" (per-architecture) to make sure they don't use "mod" in ways you did not foresee.

Quoted 6 lines

.
.
.

or we could push the rcu_read_lock() further down into is_module_address()?

Quoted 9 lines

We need to pull rcu_read_lock further _up_. It needs to be held across both is_module_address() and the actual use of the address, otherwise the memory mapping can be removed underneath us.

You can see the rcu read lock as keeping the memory mapping alive for as long as the rcu read lock is held.

We'd also need to add a synchronize_rcu() in module removal.

Quoted 7 lines

I agree that we this synchronization for the module __init section.

However, I believe we are ok for module removal case. free_module() is called *after* blocking_notifier_call_chain() call. The blocking_notifier_call_chain() is going to call back into the jump label code, grab the jump_label_mutex and remove the reference to the module that is about to freed. Thus, the jump label code can no longer reference it.

Ah, yes, this makes sense.

Quoted 3 lines

So I think the following patch is all that is required here (lightly tested).

Some comments below,

Quoted 18 lines

Steve, I'll re-post as a separate patch, if we agree on this fix.

thanks,

-Jason

jump label: fix __init module section race

Jump label uses is_module_text_address() to ensure that the module __init sections are valid before updating them. However, between the check for a valid module __init section and the subsequent jump label update, the module's __init section could be free out from under us.

Quoted 7 lines

We fix this potential race putting the address check *and* the jump label update under a rcu_read_lock(), and making sure a grace period has completed before we free the __init section.

Thanks to Mathieu Desnoyers for pointing out this race condition.

You can put my

kernel/jump_label.c void jump_label_update(unsigned long key, enum jump_label_type type) · line 162

Quoted 13 lines
---
 kernel/jump_label.c |    2 ++
 kernel/module.c     |    5 ++++-
 2 files changed, 6 insertions(+), 1 deletions(-)
kernel/jump_label.c +0 -0
index f82878b..7830bfb 100644
@@ -160,6 +160,7 @@ void jump_label_update(unsigned long key, enum jump_label_type type)
 			iter++;
 		}
 		/* eanble/disable jump labels in modules */

Separate patch for typo: eanble -> enable would be appropriate.

kernel/jump_label.c void jump_label_update(unsigned long key, enum jump_label_type type) · line 174

Quoted 9 lines
kernel/jump_label.c +2 -0
+		rcu_read_lock();
 		hlist_for_each_entry(e_module, module_node, &(entry->modules),
 							hlist) {
 			count = e_module->nr_entries;
@@ -170,6 +171,7 @@ void jump_label_update(unsigned long key, enum jump_label_type type)
 				iter++;
 			}
 		}
+		rcu_read_unlock();

Please note that the impact of this read lock is that synchronize_rcu() cannot be called from text_poke anymore (because text_poke is called within a RCU read-side C.S.). So hopefully Masami did not plan to call synchronize_rcu() from the upcoming breakpoint-based text_poke.

Quoted 16 lines
kernel/jump_label.c +0 -0
 	}
 	mutex_unlock(&jump_label_mutex);
 }
kernel/module.c +1 -0
index eba1341..09f7e9e 100644
@@ -2692,6 +2692,7 @@ SYSCALL_DEFINE3(init_module, void __user *, umod,
 		unsigned long, len, const char __user *, uargs)
 {
 	struct module *mod;
+	void *init_code;
 	int ret = 0;
 
 	/* Must have permission */

kernel/module.c SYSCALL_DEFINE3(init_module, void __user *, umod, · line 2756

Quoted 8 lines
kernel/module.c +3 -1
@@ -2749,8 +2750,10 @@ SYSCALL_DEFINE3(init_module, void __user *, umod,
 	mod->symtab = mod->core_symtab;
 	mod->strtab = mod->core_strtab;
 #endif
-	module_free(mod, mod->module_init);
+	init_code = mod->module_init;
 	mod->module_init = NULL;
+	synchronize_rcu();
+	module_free(mod, init_code);

I'm a bit concerned about the fact that "mod" is passed as parameter to module_free(). It does not seem to be used on x86, but have you reviewed all other architectures ? I would feel more comfortable if we removed this parameter from module_free() if it is indeed unused everywhere.

Thanks,

Mathieu

Quoted 8 lines
kernel/module.c +0 -0
 	mod->init_size = 0;
 	mod->init_text_size = 0;
 	mutex_unlock(&module_mutex);
Signature · 1 line
1.7.1
Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#9 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

Quoted 192 lines · Mathieu Desnoyers
Quoted 130 lines · * Jason Baron ([email protected]) wrote:
Quoted 119 lines · Mathieu Desnoyers
Quoted 99 lines · * Jason Baron ([email protected]) wrote:
Quoted 95 lines · Mathieu Desnoyers
Quoted 9 lines · * Steven Rostedt ([email protected]) wrote:

From: Jason Baron <[email protected]>

base patch to implement 'jump labeling'. Based on a new 'asm goto' inline assembly gcc mechanism, we can now branch to labels from an 'asm goto' statment. This allows us to create a 'no-op' fastpath, which can subsequently be patched with a jump to the slowpath code. This is useful for code which might be rarely used, but which we'd like to be able to call, if needed. Tracepoints are the current usecase that these are being implemented for.

[...]

Quoted 15 lines
(quoted diff) +15 -0
+/***
+ * jump_label_update - update jump label text
+ * @key -  key value associated with a a jump label
+ * @type - enum set to JUMP_LABEL_ENABLE or JUMP_LABEL_DISABLE
+ *
+ * Will enable/disable the jump for jump label @key, depending on the
+ * value of @type.
+ *
+ */
+
+void jump_label_update(unsigned long key, enum jump_label_type type)
+{
+	struct jump_entry *iter;
+	struct jump_label_entry *entry;
+	struct hlist_node *module_node;

Review

Quoted 10 lines
(quoted diff) +10 -0
+	struct jump_label_module_entry *e_module;
+	int count;
+
+	mutex_lock(&jump_label_mutex);
+	entry = get_jump_label_entry((jump_label_t)key);
+	if (entry) {
+		count = entry->nr_entries;
+		iter = entry->table;
+		while (count--) {
+			if (kernel_text_address(iter->code))

As I pointed out in another thread, I'm concerned about the use of kernel_text_address without module mutex here. kernel_text_address calls is_module_text_address(), which calls __module_text_address() with preemption off.

__module_text_address() looks like:

struct module *__module_address(unsigned long addr)

{
        struct module *mod;

        if (addr < module_addr_min || addr > module_addr_max)
                return NULL;

        list_for_each_entry_rcu(mod, &modules, list)
                if (within_module_core(addr, mod)
                    || within_module_init(addr, mod))
                        return mod;
        return NULL;
}

struct module *__module_text_address(unsigned long addr)

{
        struct module *mod = __module_address(addr);
        if (mod) {
                /* Make sure it's within the text section. */
                if (!within(addr, mod->module_init, mod->init_text_size)
                    && !within(addr, mod->module_core, mod->core_text_size))
                        mod = NULL;
        }
        return mod;
}

So the test for the address being in the module core is already problematic, since we hold preempt off only within is_module_text_address(). The is_module_text_address() caller is then free to write to this address even after the module has been unloaded and the module unload grace period ended.

Even worse, such grace period is not waited for at module load time within:

init_module()

       module_free(mod, mod->module_init);
       mod->module_init = NULL;
       mod->init_size = 0;
       mod->init_text_size = 0;

(done with module_mutex held, while the module is already in the
module list)

We'd probably have to hold the module mutex around the is_module_text_address() call and address use (which can be a pain), or to correctly address this part of init_module() with RCU and require that preempt off is held across both __module_text_address() call site and the actual use of that pointer (which does not fit with jump label, which need to sleep, so we'd have to move module.c to a preemptable rcu_read_lock/synchronize_rcu() C.S.).

Quoted 2 lines

Thoughts ?

Quoted 7 lines

I was thinking about the rcu_read_lock/synchronize_rcu() for this race. We can hold the rcu_read_lock() across the is_module_text_address() check in the jump label code, and then we can do in module.c:

mod->module_init = NULL;
synchronize_rcu();
module_free(mod, mod->module_init);

Beware that you need to copy the module_init address. Also make sure you audit the "module_free" (per-architecture) to make sure they don't use "mod" in ways you did not foresee.

Quoted 6 lines

.
.
.

or we could push the rcu_read_lock() further down into is_module_address()?

Quoted 9 lines

We need to pull rcu_read_lock further _up_. It needs to be held across both is_module_address() and the actual use of the address, otherwise the memory mapping can be removed underneath us.

You can see the rcu read lock as keeping the memory mapping alive for as long as the rcu read lock is held.

We'd also need to add a synchronize_rcu() in module removal.

Quoted 7 lines

I agree that we this synchronization for the module __init section.

However, I believe we are ok for module removal case. free_module() is called *after* blocking_notifier_call_chain() call. The blocking_notifier_call_chain() is going to call back into the jump label code, grab the jump_label_mutex and remove the reference to the module that is about to freed. Thus, the jump label code can no longer reference it.

Ah, yes, this makes sense.

Quoted 3 lines

So I think the following patch is all that is required here (lightly tested).

Some comments below,

Quoted 18 lines

Steve, I'll re-post as a separate patch, if we agree on this fix.

thanks,

-Jason

jump label: fix __init module section race

Jump label uses is_module_text_address() to ensure that the module __init sections are valid before updating them. However, between the check for a valid module __init section and the subsequent jump label update, the module's __init section could be free out from under us.

Quoted 7 lines

We fix this potential race putting the address check *and* the jump label update under a rcu_read_lock(), and making sure a grace period has completed before we free the __init section.

Thanks to Mathieu Desnoyers for pointing out this race condition.

You can put my

kernel/jump_label.c void jump_label_update(unsigned long key, enum jump_label_type type) · line 162

Quoted 13 lines
---
 kernel/jump_label.c |    2 ++
 kernel/module.c     |    5 ++++-
 2 files changed, 6 insertions(+), 1 deletions(-)
kernel/jump_label.c +0 -0
index f82878b..7830bfb 100644
@@ -160,6 +160,7 @@ void jump_label_update(unsigned long key, enum jump_label_type type)
 			iter++;
 		}
 		/* eanble/disable jump labels in modules */

Separate patch for typo: eanble -> enable would be appropriate.

Quoted 15 lines

kernel/jump_label.c void jump_label_update(unsigned long key, enum jump_label_type type) · line 174

Quoted 9 lines
kernel/jump_label.c +2 -0
+		rcu_read_lock();
 		hlist_for_each_entry(e_module, module_node, &(entry->modules),
 							hlist) {
 			count = e_module->nr_entries;
@@ -170,6 +171,7 @@ void jump_label_update(unsigned long key, enum jump_label_type type)
 				iter++;
 			}
 		}
+		rcu_read_unlock();

Please note that the impact of this read lock is that synchronize_rcu() cannot be called from text_poke anymore (because text_poke is called within a RCU read-side C.S.). So hopefully Masami did not plan to call synchronize_rcu() from the upcoming breakpoint-based text_poke.

ok, we can re-visit this if it becomes an issue at some future point...thanks for mentioning it.

Quoted 14 lines
Quoted 14 lines
kernel/jump_label.c +0 -0
 	}
 	mutex_unlock(&jump_label_mutex);
 }
kernel/module.c +1 -0
index eba1341..09f7e9e 100644
@@ -2692,6 +2692,7 @@ SYSCALL_DEFINE3(init_module, void __user *, umod,
 		unsigned long, len, const char __user *, uargs)
 {
 	struct module *mod;
+	void *init_code;
 	int ret = 0;
 
Quoted 16 lines

kernel/module.c SYSCALL_DEFINE3(init_module, void __user *, umod, · line 2756

Quoted 10 lines
kernel/module.c +3 -1
 	/* Must have permission */
@@ -2749,8 +2750,10 @@ SYSCALL_DEFINE3(init_module, void __user *, umod,
 	mod->symtab = mod->core_symtab;
 	mod->strtab = mod->core_strtab;
 #endif
-	module_free(mod, mod->module_init);
+	init_code = mod->module_init;
 	mod->module_init = NULL;
+	synchronize_rcu();
+	module_free(mod, init_code);

I'm a bit concerned about the fact that "mod" is passed as parameter to module_free(). It does not seem to be used on x86, but have you reviewed all other architectures ? I would feel more comfortable if we removed this parameter from module_free() if it is indeed unused everywhere.

good point.

So you're concern here is that mod->module_init is used in the 'module_free'? indeed i missed it on ia64:

void
module_free (struct module *mod, void *module_region)

{
        if (mod && mod->arch.init_unw_table &&
            module_region == mod->module_init) {
                unw_remove_unwind_table(mod->arch.init_unw_table);
                mod->arch.init_unw_table = NULL;
        }
        vfree(module_region);
}

So, I don't see why we couldn't s/mod->module_init/module_region, but then I wonder why it wasn't written that way in the first place?

thanks,

-Jason

#10 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

In reply to Mathieu Desnoyers
Quoted 1 line · * Mathieu Desnoyers ([email protected]) wrote:

* Jason Baron ([email protected]) wrote:

[...]

Quoted 8 lines
Quoted 8 lines

jump label: fix __init module section race

Jump label uses is_module_text_address() to ensure that the module __init sections are valid before updating them. However, between the check for a valid module __init section and the subsequent jump label update, the module's __init section could be free out from under us.

Quoted 5 lines
Quoted 5 lines

We fix this potential race putting the address check *and* the jump label update under a rcu_read_lock(), and making sure a grace period has completed before we free the __init section.

Thanks to Mathieu Desnoyers for pointing out this race condition.

Actually, there might be another way to deal with this that would not require holding rcu read lock at all in this path, nor adding synchronize_rcu in the module code.

This involves adding a notifier right before the init code is freed, so the static jump code can disable all static jumps pointing to init code area for the given module.

On the plus side, this means text_poke would be free to still use synchronize_rcu() if it ever needs to. Also, calling a notifier chain before freeing the module init region will hurt module load time much less than waiting for a grace period.

Thoughts ?

Mathieu

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#11 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

In reply to Jason Baron

kernel/module.c SYSCALL_DEFINE3(init_module, void __user *, umod, · line 2759

Quoted 12 lines · Jason Baron
kernel/module.c +3 -1
@@ -2749,8 +2750,10 @@ SYSCALL_DEFINE3(init_module, void __user *, umod,
 	mod->symtab = mod->core_symtab;
 	mod->strtab = mod->core_strtab;
 #endif
-	module_free(mod, mod->module_init);
+	init_code = mod->module_init;
 	mod->module_init = NULL;
+	synchronize_rcu();
+	module_free(mod, init_code);
 	mod->init_size = 0;
 	mod->init_text_size = 0;
 	mutex_unlock(&module_mutex);

When a patch requires more lkml mails than it has lines, it needs a comment.

But step back for a moment: what prompts the jump label update? Why isn't that simply done under the module lock, obviating any complexity?

If you're frobbing kernel text all over the place, you probable want the module lock. You wouldn't be the first: perhaps we should rename that to kernel_text_lock...

Apologies if that's a dumb question,
Rusty.

author maintainer
#12 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

In reply to Jason Baron
Quoted 8 lines · Jason Baron

I agree that we this synchronization for the module __init section.

However, I believe we are ok for module removal case. free_module() is called *after* blocking_notifier_call_chain() call. The blocking_notifier_call_chain() is going to call back into the jump label code, grab the jump_label_mutex and remove the reference to the module that is about to freed. Thus, the jump label code can no longer reference it.

Quoted 4 lines

So I think the following patch is all that is required here (lightly tested).

Steve, I'll re-post as a separate patch, if we agree on this fix.

Ug, I'm struggling to get ready for my Tokyo trip. I'll try to look at it on the flight. But I still need to write my presentation :-)

I may have a response back till Monday or Tuesday.

-- Steve

#13 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

Quoted 13 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 11 lines · Jason Baron

I agree that we this synchronization for the module __init section.

However, I believe we are ok for module removal case. free_module() is called *after* blocking_notifier_call_chain() call. The blocking_notifier_call_chain() is going to call back into the jump label code, grab the jump_label_mutex and remove the reference to the module that is about to freed. Thus, the jump label code can no longer reference it.

So I think the following patch is all that is required here (lightly tested).

Quoted 6 lines
Quoted 1 line

Steve, I'll re-post as a separate patch, if we agree on this fix.

Ug, I'm struggling to get ready for my Tokyo trip. I'll try to look at it on the flight. But I still need to write my presentation :-)

I may have a response back till Monday or Tuesday.

As a note for when you review this patch, I have a strong preference for adding a "pre-init-free notifier callback" rather than taking the rcu read lock + adding a synchronize_rcu() in module.c. This will make our life much easier when we end up doing modifications down in text_poke.

Thanks,

Mathieu

Quoted 4 lines

-- Steve

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#14 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

Quoted 21 lines · Mathieu Desnoyers
Quoted 13 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 11 lines · Jason Baron

I agree that we this synchronization for the module __init section.

However, I believe we are ok for module removal case. free_module() is called *after* blocking_notifier_call_chain() call. The blocking_notifier_call_chain() is going to call back into the jump label code, grab the jump_label_mutex and remove the reference to the module that is about to freed. Thus, the jump label code can no longer reference it.

So I think the following patch is all that is required here (lightly tested).

Quoted 6 lines
Quoted 1 line

Steve, I'll re-post as a separate patch, if we agree on this fix.

Ug, I'm struggling to get ready for my Tokyo trip. I'll try to look at it on the flight. But I still need to write my presentation :-)

I may have a response back till Monday or Tuesday.

Quoted 9 lines

As a note for when you review this patch, I have a strong preference for adding a "pre-init-free notifier callback" rather than taking the rcu read lock + adding a synchronize_rcu() in module.c. This will make our life much easier when we end up doing modifications down in text_poke.

Thanks,

Mathieu

Yes, I think we can make this work with a "pre-init-free notifier callback". In fact, we can use MODULE_STATE_LIVE state that is already in module.c. I'm in the process of testing a patch using this alternative method.

thanks,

-Jason

author maintainer
#15 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

In reply to Jason Baron
Quoted 9 lines · Jason Baron

On Thu, Sep 23, 2010 at 10:37:58AM -0400, Mathieu Desnoyers wrote:

I was thinking about the rcu_read_lock/synchronize_rcu() for this race. We can hold the rcu_read_lock() across the is_module_text_address() check in the jump label code, and then we can do in module.c:

mod->module_init = NULL;
synchronize_rcu();
module_free(mod, mod->module_init);

Um, isn't that last call the same as:

module_free(mod, NULL); ?

I'll spend some time looking at this too.

-- Steve

Quoted 11 lines

.
.
.

or we could push the rcu_read_lock() further down into is_module_address()?

thanks,

-Jason

#16 lore raw

Re: [PATCH 03/11] jump label: Base patch for jump label

Quoted 7 lines · * Steven Rostedt ([email protected]) wrote:
Quoted 6 lines · Jason Baron

On Thu, Sep 23, 2010 at 10:37:58AM -0400, Mathieu Desnoyers wrote:

I was thinking about the rcu_read_lock/synchronize_rcu() for this race. We can hold the rcu_read_lock() across the is_module_text_address() check in the jump label code, and then we can do in module.c:

Quoted 7 lines
Quoted 3 lines

mod->module_init = NULL;
synchronize_rcu();
module_free(mod, mod->module_init);

Um, isn't that last call the same as:

module_free(mod, NULL); ?

Yes, this is incorrect (as I pointed out in my reply). We should do, instead:

module_init = mod->module_init;
mod->module_init = NULL;
synchronize_rcu();
module_free(mod, module_init);

Thanks,

Mathieu

Quoted 18 lines

I'll spend some time looking at this too.

-- Steve

Quoted 11 lines

.
.
.

or we could push the rcu_read_lock() further down into is_module_address()?

thanks,

-Jason

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
author maintainer
#17 lore raw

PATCH 4/11 jump label: Initialize workqueue tracepoints *before* they are registered

In reply to Steven Rostedt

From: Jason Baron <[email protected]>

Initialize the workqueue data structures *before* they are registered so that they are ready for callbacks.

LKML-Reference: <e3a3383fc370ac7086625bebe89d9480d7caf372.1284733808.git.jbaron@redhat.com>

---
 kernel/trace/trace_workqueue.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)
kernel/trace/trace_workqueue.c +5 -5
index a7cc379..209b379 100644
@@ -263,6 +263,11 @@ int __init trace_workqueue_early_init(void)
 {
 	int ret, cpu;
 
+	for_each_possible_cpu(cpu) {
+		spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
+		INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
+	}
+
 	ret = register_trace_workqueue_insertion(probe_workqueue_insertion, NULL);
 	if (ret)
 		goto out;
@@ -279,11 +284,6 @@ int __init trace_workqueue_early_init(void)
 	if (ret)
 		goto no_creation;
 
-	for_each_possible_cpu(cpu) {
-		spin_lock_init(&workqueue_cpu_stat(cpu)->lock);
-		INIT_LIST_HEAD(&workqueue_cpu_stat(cpu)->list);
-	}
-
 	return 0;
 
 no_creation:
Signature · 1 line
1.7.1
author maintainer
#18 lore raw

PATCH 5/11 jump label: Add jump_label_text_reserved() to reserve jump points

In reply to Steven Rostedt

From: Jason Baron <[email protected]>

Add a jump_label_text_reserved(void *start, void *end), so that other pieces of code that want to modify kernel text, can first verify that jump label has not reserved the instruction.

LKML-Reference: <06236663a3a7b1c1f13576bb9eccb6d9c17b7bfe.1284733808.git.jbaron@redhat.com>

---
 arch/x86/kernel/kprobes.c  |    3 +-
 include/linux/jump_label.h |    8 ++++-
 kernel/jump_label.c        |   83 ++++++++++++++++++++++++++++++++++++++++++++
 kernel/kprobes.c           |    3 +-
 4 files changed, 94 insertions(+), 3 deletions(-)
arch/x86/kernel/kprobes.c +2 -1
index e05952a..1cbd54c 100644
@@ -1218,7 +1218,8 @@ static int __kprobes copy_optimized_instructions(u8 *dest, u8 *src)
 	}
 	/* Check whether the address range is reserved */
 	if (ftrace_text_reserved(src, src + len - 1) ||
-	    alternatives_text_reserved(src, src + len - 1))
+	    alternatives_text_reserved(src, src + len - 1) ||
+	    jump_label_text_reserved(src, src + len - 1))
 		return -EBUSY;
 
 	return len;
include/linux/jump_label.h +7 -1
index de58656..b72cd9f 100644
@@ -20,9 +20,10 @@ extern struct jump_entry __stop___jump_table[];
 
 extern void arch_jump_label_transform(struct jump_entry *entry,
 				 enum jump_label_type type);
+extern void arch_jump_label_text_poke_early(jump_label_t addr);
 extern void jump_label_update(unsigned long key, enum jump_label_type type);
 extern void jump_label_apply_nops(struct module *mod);
-extern void arch_jump_label_text_poke_early(jump_label_t addr);
+extern int jump_label_text_reserved(void *start, void *end);
 
 #define enable_jump_label(key) \
 	jump_label_update((unsigned long)key, JUMP_LABEL_ENABLE);
@@ -53,6 +54,11 @@ static inline int jump_label_apply_nops(struct module *mod)
 	return 0;
 }
 
+static inline int jump_label_text_reserved(void *start, void *end)
+{
+	return 0;
+}
+
 #endif
 
 #endif
kernel/jump_label.c +83 -0

89 lines not on this page · raw message

kernel/kprobes.c +2 -1
index 18904e4..ec4210c 100644
@@ -1147,7 +1147,8 @@ int __kprobes register_kprobe(struct kprobe *p)
 	preempt_disable();
 	if (!kernel_text_address((unsigned long) p->addr) ||
 	    in_kprobes_functions((unsigned long) p->addr) ||
-	    ftrace_text_reserved(p->addr, p->addr)) {
+	    ftrace_text_reserved(p->addr, p->addr) ||
+	    jump_label_text_reserved(p->addr, p->addr)) {
 		preempt_enable();
 		return -EINVAL;
 	}
Signature · 1 line
1.7.1
author maintainer
#19 lore raw

PATCH 6/11 jump label: Tracepoint support for jump labels

In reply to Steven Rostedt

From: Jason Baron <[email protected]>

Make use of the jump label infrastructure for tracepoints.

LKML-Reference: <a9ba2056e2c9cf332c3c300b577463ce66ff23a8.1284733808.git.jbaron@redhat.com>

---
 include/linux/tracepoint.h |    5 ++++-
 kernel/tracepoint.c        |   14 ++++++++++++--
 2 files changed, 16 insertions(+), 3 deletions(-)
include/linux/tracepoint.h +4 -1
index 103d1b6..a4a90b6 100644
@@ -17,6 +17,7 @@
 #include <linux/errno.h>
 #include <linux/types.h>
 #include <linux/rcupdate.h>
+#include <linux/jump_label.h>
 
 struct module;
 struct tracepoint;
@@ -145,7 +146,9 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin,
 	extern struct tracepoint __tracepoint_##name;			\
 	static inline void trace_##name(proto)				\
 	{								\
-		if (unlikely(__tracepoint_##name.state))		\
+		JUMP_LABEL(&__tracepoint_##name.state, do_trace);	\
+		return;							\
+do_trace:								\
 			__DO_TRACE(&__tracepoint_##name,		\
 				TP_PROTO(data_proto),			\
 				TP_ARGS(data_args));			\
kernel/tracepoint.c +12 -2
index c77f3ec..d6073a5 100644
@@ -25,6 +25,7 @@
 #include <linux/err.h>
 #include <linux/slab.h>
 #include <linux/sched.h>
+#include <linux/jump_label.h>
 
 extern struct tracepoint __start___tracepoints[];
 extern struct tracepoint __stop___tracepoints[];
@@ -263,7 +264,13 @@ static void set_tracepoint(struct tracepoint_entry **entry,
 	 * is used.
 	 */
 	rcu_assign_pointer(elem->funcs, (*entry)->funcs);
-	elem->state = active;
+	if (!elem->state && active) {
+		enable_jump_label(&elem->state);
+		elem->state = active;
+	} else if (elem->state && !active) {
+		disable_jump_label(&elem->state);
+		elem->state = active;
+	}
 }
 
 /*
@@ -277,7 +284,10 @@ static void disable_tracepoint(struct tracepoint *elem)
 	if (elem->unregfunc && elem->state)
 		elem->unregfunc();
 
-	elem->state = 0;
+	if (elem->state) {
+		disable_jump_label(&elem->state);
+		elem->state = 0;
+	}
 	rcu_assign_pointer(elem->funcs, NULL);
 }
 
Signature · 1 line
1.7.1
author maintainer
#20 lore raw

PATCH 7/11 jump label: Convert dynamic debug to use jump labels

In reply to Steven Rostedt

From: Jason Baron <[email protected]>

Convert the 'dynamic debug' infrastructure to use jump labels.

LKML-Reference: <b77627358cea3e27d7be4386f45f66219afb8452.1284733808.git.jbaron@redhat.com>

---
 include/linux/dynamic_debug.h |   39 +++++++++++++-----------
 lib/dynamic_debug.c           |   42 ++-------------------------
 scripts/Makefile.lib          |   11 +------
 scripts/basic/Makefile        |    2 +-
 scripts/basic/hash.c          |   64 -----------------------------------------
 5 files changed, 26 insertions(+), 132 deletions(-)
 delete mode 100644 scripts/basic/hash.c
include/linux/dynamic_debug.h +21 -18
index 52c0da4..bef3cda 100644
@@ -1,6 +1,8 @@
 #ifndef _DYNAMIC_DEBUG_H
 #define _DYNAMIC_DEBUG_H
 
+#include <linux/jump_label.h>
+
 /* dynamic_printk_enabled, and dynamic_printk_enabled2 are bitmasks in which
  * bit n is set to 1 if any modname hashes into the bucket n, 0 otherwise. They
  * use independent hash functions, to reduce the chance of false positives.
@@ -22,8 +24,6 @@ struct _ddebug {
 	const char *function;
 	const char *filename;
 	const char *format;
-	char primary_hash;
-	char secondary_hash;
 	unsigned int lineno:24;
 	/*
  	 * The flags field controls the behaviour at the callsite.
@@ -33,6 +33,7 @@ struct _ddebug {
 #define _DPRINTK_FLAGS_PRINT   (1<<0)  /* printk() a message using the format */
 #define _DPRINTK_FLAGS_DEFAULT 0
 	unsigned int flags:8;
+	char enabled;
 } __attribute__((aligned(8)));
 
 
@@ -42,33 +43,35 @@ int ddebug_add_module(struct _ddebug *tab, unsigned int n,
 #if defined(CONFIG_DYNAMIC_DEBUG)
 extern int ddebug_remove_module(const char *mod_name);
 
-#define __dynamic_dbg_enabled(dd)  ({	     \
-	int __ret = 0;							     \
-	if (unlikely((dynamic_debug_enabled & (1LL << DEBUG_HASH)) &&	     \
-			(dynamic_debug_enabled2 & (1LL << DEBUG_HASH2))))   \
-				if (unlikely(dd.flags))			     \
-					__ret = 1;			     \
-	__ret; })
-
 #define dynamic_pr_debug(fmt, ...) do {					\
+	__label__ do_printk;						\
+	__label__ out;							\
 	static struct _ddebug descriptor				\
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
-	{ KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH,	\
-		DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT };	\
-	if (__dynamic_dbg_enabled(descriptor))				\
-		printk(KERN_DEBUG pr_fmt(fmt),	##__VA_ARGS__);		\
+	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
+		_DPRINTK_FLAGS_DEFAULT };				\
+	JUMP_LABEL(&descriptor.enabled, do_printk);			\
+	goto out;							\
+do_printk:								\
+	printk(KERN_DEBUG pr_fmt(fmt),	##__VA_ARGS__);			\
+out:	;								\
 	} while (0)
 
 
 #define dynamic_dev_dbg(dev, fmt, ...) do {				\
+	__label__ do_printk;						\
+	__label__ out;							\
 	static struct _ddebug descriptor				\
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
-	{ KBUILD_MODNAME, __func__, __FILE__, fmt, DEBUG_HASH,	\
-		DEBUG_HASH2, __LINE__, _DPRINTK_FLAGS_DEFAULT };	\
-	if (__dynamic_dbg_enabled(descriptor))				\
-		dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__);	\
+	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
+		_DPRINTK_FLAGS_DEFAULT };				\
+	JUMP_LABEL(&descriptor.enabled, do_printk);			\
+	goto out;							\
+do_printk:								\
+	dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__);		\
+out:	;								\
 	} while (0)
 
 #else
lib/dynamic_debug.c +3 -39

65 lines not on this page · raw message

scripts/Makefile.lib +1 -10
index 54fd1b7..7bfcf1a 100644
@@ -101,14 +101,6 @@ basename_flags = -D"KBUILD_BASENAME=KBUILD_STR($(call name-fix,$(basetarget)))"
 modname_flags  = $(if $(filter 1,$(words $(modname))),\
                  -D"KBUILD_MODNAME=KBUILD_STR($(call name-fix,$(modname)))")
 
-#hash values
-ifdef CONFIG_DYNAMIC_DEBUG
-debug_flags = -D"DEBUG_HASH=$(shell ./scripts/basic/hash djb2 $(@D)$(modname))"\
-              -D"DEBUG_HASH2=$(shell ./scripts/basic/hash r5 $(@D)$(modname))"
-else
-debug_flags =
-endif
-
 orig_c_flags   = $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) $(KBUILD_SUBDIR_CCFLAGS) \
                  $(ccflags-y) $(CFLAGS_$(basetarget).o)
 _c_flags       = $(filter-out $(CFLAGS_REMOVE_$(basetarget).o), $(orig_c_flags))
@@ -152,8 +144,7 @@ endif
 
 c_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__c_flags) $(modkern_cflags)                           \
-		 -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags) \
-		  $(debug_flags)
+		 -D"KBUILD_STR(s)=\#s" $(basename_flags) $(modname_flags)
 
 a_flags        = -Wp,-MD,$(depfile) $(NOSTDINC_FLAGS) $(LINUXINCLUDE)     \
 		 $(__a_flags) $(modkern_aflags)
scripts/basic/Makefile +1 -1
index 0955995..4c324a1 100644
@@ -9,7 +9,7 @@
 # fixdep: 	 Used to generate dependency information during build process
 # docproc:	 Used in Documentation/DocBook
 
-hostprogs-y	:= fixdep docproc hash
+hostprogs-y	:= fixdep docproc
 always		:= $(hostprogs-y)
 
 # fixdep is needed to compile other host programs
scripts/basic/hash.c deleted +0 -64

64 lines not on this page · raw message

Signature · 1 line
1.7.1
author maintainer
#21 lore raw

PATCH 8/11 jump label: x86 support

In reply to Steven Rostedt

From: Jason Baron <[email protected]>

add x86 support for jump label. I'm keeping this patch separate so its clear to arch maintainers what was required for x86 support this new feature. Hopefully, it wouldn't be too painful for other archs.

LKML-Reference: <f838f49f40fbea0254036194be66dc48b598dcea.1284733808.git.jbaron@redhat.com>

[ cleaned up some formatting ]

---
 arch/x86/Kconfig                  |    1 +
 arch/x86/include/asm/jump_label.h |   47 ++++++++++++++++++++++++++++++++++
 arch/x86/kernel/Makefile          |    2 +-
 arch/x86/kernel/jump_label.c      |   50 +++++++++++++++++++++++++++++++++++++
 arch/x86/kernel/module.c          |    3 ++
 5 files changed, 102 insertions(+), 1 deletions(-)
 create mode 100644 arch/x86/include/asm/jump_label.h
 create mode 100644 arch/x86/kernel/jump_label.c
arch/x86/Kconfig +1 -0
index cea0cd9..afcd663 100644
@@ -59,6 +59,7 @@ config X86
 	select ANON_INODES
 	select HAVE_ARCH_KMEMCHECK
 	select HAVE_USER_RETURN_NOTIFIER
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config INSTRUCTION_DECODER
 	def_bool (KPROBES || PERF_EVENTS)
arch/x86/include/asm/jump_label.h new file +47 -0

47 lines not on this page · raw message

arch/x86/kernel/Makefile +1 -1
index 0925676..24fa171 100644
@@ -32,7 +32,7 @@ GCOV_PROFILE_paravirt.o		:= n
 obj-y			:= process_$(BITS).o signal.o entry_$(BITS).o
 obj-y			+= traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o
 obj-y			+= time.o ioport.o ldt.o dumpstack.o
-obj-y			+= setup.o x86_init.o i8259.o irqinit.o
+obj-y			+= setup.o x86_init.o i8259.o irqinit.o jump_label.o
 obj-$(CONFIG_X86_VISWS)	+= visws_quirks.o
 obj-$(CONFIG_X86_32)	+= probe_roms_32.o
 obj-$(CONFIG_X86_32)	+= sys_i386_32.o i386_ksyms_32.o
arch/x86/kernel/jump_label.c new file +50 -0

50 lines not on this page · raw message

arch/x86/kernel/module.c +3 -0
index e0bc186..5399f58 100644
@@ -239,6 +239,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 		apply_paravirt(pseg, pseg + para->sh_size);
 	}
 
+	/* make jump label nops */
+	jump_label_apply_nops(me);
+
 	return module_bug_finalize(hdr, sechdrs, me);
 }
 
Signature · 1 line
1.7.1
author maintainer
#22 lore raw

PATCH 9/11 jump label: Add sparc64 support

In reply to Steven Rostedt

From: David S. Miller <[email protected]>

Add jump label support for sparc64.

LKML-Reference: <3b5b071fcdb2afb7f67cacecfa78b14c740278a7.1284733808.git.jbaron@redhat.com>

[ cleaned up some formatting ]

---
 arch/sparc/Kconfig                  |    1 +
 arch/sparc/include/asm/jump_label.h |   32 +++++++++++++++++++++++
 arch/sparc/kernel/Makefile          |    2 +
 arch/sparc/kernel/jump_label.c      |   47 +++++++++++++++++++++++++++++++++++
 arch/sparc/kernel/module.c          |    6 ++++
 5 files changed, 88 insertions(+), 0 deletions(-)
 create mode 100644 arch/sparc/include/asm/jump_label.h
 create mode 100644 arch/sparc/kernel/jump_label.c
arch/sparc/Kconfig +1 -0
index 491e9d6..a81b04e 100644
@@ -30,6 +30,7 @@ config SPARC
 	select PERF_USE_VMALLOC
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_API_DEBUG
+	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
 
 config SPARC32
 	def_bool !64BIT
arch/sparc/include/asm/jump_label.h new file +32 -0
new file mode 100644
index 0000000..62e66d7
@@ -0,0 +1,32 @@
+#ifndef _ASM_SPARC_JUMP_LABEL_H
+#define _ASM_SPARC_JUMP_LABEL_H
+
+#ifdef __KERNEL__
+
+#include <linux/types.h>
+#include <asm/system.h>
+
+#define JUMP_LABEL_NOP_SIZE 4
+
+#define JUMP_LABEL(key, label)					\
+	do {							\
+		asm goto("1:\n\t"				\
+			 "nop\n\t"				\
+			 "nop\n\t"				\
+			 ".pushsection __jump_table,  \"a\"\n\t"\
+			 ".word 1b, %l[" #label "], %c0\n\t"	\
+			 ".popsection \n\t"			\
+			 : :  "i" (key) :  : label);\
+	} while (0)
+
+#endif /* __KERNEL__ */
+
+typedef u32 jump_label_t;
+
+struct jump_entry {
+	jump_label_t code;
+	jump_label_t target;
+	jump_label_t key;
+};
+
+#endif
arch/sparc/kernel/Makefile +2 -0
index 0c2dc1f..599398f 100644
@@ -119,3 +119,5 @@ obj-$(CONFIG_COMPAT)    += $(audit--y)
 
 pc--$(CONFIG_PERF_EVENTS) := perf_event.o
 obj-$(CONFIG_SPARC64)	+= $(pc--y)
+
+obj-$(CONFIG_SPARC64)	+= jump_label.o
arch/sparc/kernel/jump_label.c new file +47 -0

47 lines not on this page · raw message

arch/sparc/kernel/module.c +6 -0
index f848aad..ee3c7dd 100644
@@ -18,6 +18,9 @@
 #include <asm/spitfire.h>
 
 #ifdef CONFIG_SPARC64
+
+#include <linux/jump_label.h>
+
 static void *module_map(unsigned long size)
 {
 	struct vm_struct *area;
@@ -227,6 +230,9 @@ int module_finalize(const Elf_Ehdr *hdr,
 		    const Elf_Shdr *sechdrs,
 		    struct module *me)
 {
+	/* make jump label nops */
+	jump_label_apply_nops(me);
+
 	/* Cheetah's I-cache is fully coherent.  */
 	if (tlb_type == spitfire) {
 		unsigned long va;
Signature · 1 line
1.7.1
author maintainer
#23 lore raw

PATCH 10/11 jump label: Remove duplicate structure for x86

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

The structure in the x86 jump label code uses the typedef jump_label_t, which is defined by the #ifdef arch type. The structure does not need to be duplicated there.

---
 arch/x86/include/asm/jump_label.h |   12 +-----------
 1 files changed, 1 insertions(+), 11 deletions(-)
arch/x86/include/asm/jump_label.h +1 -11
index b4a2cb4..f52d42e 100644
@@ -23,18 +23,10 @@
 #endif /* __KERNEL__ */
 
 #ifdef CONFIG_X86_64
-
 typedef u64 jump_label_t;
-
-struct jump_entry {
-	jump_label_t code;
-	jump_label_t target;
-	jump_label_t key;
-};
-
 #else
-
 typedef u32 jump_label_t;
+#endif
 
 struct jump_entry {
 	jump_label_t code;
@@ -43,5 +35,3 @@ struct jump_entry {
 };
 
 #endif
-
-#endif
Signature · 1 line
1.7.1
author maintainer
#24 lore raw

PATCH 11/11 jump label/x86/sparc64: Remove !CC_OPTIMIZE_FOR_SIZE config conditions

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

The !CC_OPTIMIZE_FOR_SIZE was added to enable the jump label functionality because Jason noticed that the gcc option would not optimize the labels and may even hurt performance.

But this is a gcc problem not a kernel one. Removing this condition should add motivation to the gcc developers to actually fix it.

---
 arch/sparc/Kconfig |    2 +-
 arch/x86/Kconfig   |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/sparc/Kconfig +1 -1
index a81b04e..9212cd4 100644
@@ -30,7 +30,7 @@ config SPARC
 	select PERF_USE_VMALLOC
 	select HAVE_DMA_ATTRS
 	select HAVE_DMA_API_DEBUG
-	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
+	select HAVE_ARCH_JUMP_LABEL
 
 config SPARC32
 	def_bool !64BIT
arch/x86/Kconfig +1 -1
index afcd663..b431a08 100644
@@ -59,7 +59,7 @@ config X86
 	select ANON_INODES
 	select HAVE_ARCH_KMEMCHECK
 	select HAVE_USER_RETURN_NOTIFIER
-	select HAVE_ARCH_JUMP_LABEL if !CC_OPTIMIZE_FOR_SIZE
+	select HAVE_ARCH_JUMP_LABEL
 
 config INSTRUCTION_DECODER
 	def_bool (KPROBES || PERF_EVENTS)
Signature · 1 line
1.7.1
author maintainer
#25 lore raw
In reply to Steven Rostedt

Hmm, interesting that the Subject is missing the [PATCH 00/11].

Ah! doing a test, I see that "quilt mail" will not add that, if you delete the space after "Subject:". Yes, I develop in git, but send out patches in quilt. That's just the way I am.

-- Steve

#26 lore raw
Quoted 5 lines · Steven Rostedt

Hmm, interesting that the Subject is missing the [PATCH 00/11].

Ah! doing a test, I see that "quilt mail" will not add that, if you delete the space after "Subject:". Yes, I develop in git, but send out patches in quilt. That's just the way I am.

The lack of 00/11 doesnt even look bad - the threading of the emails you sent was correct and i didnt even notice that the 00/11 is missing. The numbering of the rest is what matters, and proper threading.

( The more annoying variants are the recursive-nested Git sends - and
the totally unthreaded random-lkml-order-of-the-day patch dumps. )

Thanks,

Ingo

author maintainer
#27 lore raw
In reply to Steven Rostedt
Quoted 9 lines · Steven Rostedt

Jason Baron (8):
jump label: Make dynamic no-op selection available outside of ftrace jump label: Make text_poke_early() globally visible
jump label: Base patch for jump label
jump label: Initialize workqueue tracepoints *before* they are registered jump label: Add jump_label_text_reserved() to reserve jump points jump label: Tracepoint support for jump labels
jump label: Convert dynamic debug to use jump labels
jump label: x86 support

Jason,

If you noticed, I left out your last patch: the documentation one. I did this because there was enough comments to require a resubmit. Could you just send that one patch, with the corrections, by itself?

Thanks,

-- Steve

#28 lore raw

[tip:perf/core] jump label: Fix GCC feature check when distcc is used

In reply to Steven Rostedt
Commit-ID:  d6dad199a10423ce37b8bfec1f055c380dc4a3d5
Gitweb:     http://git.kernel.org/tip/d6dad199a10423ce37b8bfec1f055c380dc4a3d5
Author:     Ingo Molnar <[email protected]>
AuthorDate: Fri, 24 Sep 2010 09:12:25 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 24 Sep 2010 09:12:25 +0200

jump label: Fix GCC feature check when distcc is used

The following build bug occurs on distcc builds:

CC arch/x86/kernel/asm-offsets.s
In file included from include/linux/module.h:24,

                  from include/linux/crypto.h:22,
                  from arch/x86/kernel/asm-offsets_64.c:9,
                  from arch/x86/kernel/asm-offsets.c:5:

include/trace/events/module.h: In function 'trace_module_load':
include/trace/events/module.h:18: error: expected '(' before 'goto' include/trace/events/module.h:18: error: expected identifier or '*' before '(' token

It triggers because distcc is invoked by turning $CC into "distcc gcc", but gcc-goto.sh check script was using $1 not $@ to expand parameters.

Cc: Jason Baron <[email protected]>
Cc: Steven Rostedt <[email protected]>
LKML-Reference: <[email protected]>

---
 scripts/gcc-goto.sh |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
scripts/gcc-goto.sh +1 -1
index 8e82424..520d16b 100644
@@ -2,4 +2,4 @@
 # Test for gcc 'asm goto' suport
 # Copyright (C) 2010, Jason Baron <[email protected]>
 
-echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $1 -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"
+echo "int main(void) { entry: asm goto (\"\"::::entry); return 0; }" | $@ -x c - -c -o /dev/null >/dev/null 2>&1 && echo "y"