Skip to contentSkip to conversation

jump label: simplify API

PATCH In review Jason Baron 5 participants 21 messages Dec 16, 2010 – Dec 17, 2010

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover jump label: simplify API

Hi,

The first patch uses the storage space of the jump label key address as a pointer into the update table. In this way, we can find all the addresses that need to be updated without hashing.

The second patch introduces:

static __always_inline bool unlikely_switch(struct jump_label_key *key);

instead of the old JUMP_LABEL(key, label) macro.

In this way, jump labels become really easy to use:

Define:

	struct jump_label_key jump_key;

Can be used as:

if (unlikely_switch(&jump_key))
do unlikely code

enable/disale via:

	jump_label_enable(&jump_key);
	jump_label_disable(&jump_key);

that's it!

Thanks to H. Peter Anvin for suggesting the simpler 'unlikely_switch()' function.

thanks,

-Jason

Jason Baron (2):
jump label: make enable/disable o(1)
jump label: introduce unlikely_switch()

 arch/x86/include/asm/jump_label.h |   22 +++--
 arch/x86/kernel/jump_label.c      |    2 +-
 include/linux/dynamic_debug.h     |   24 ++----
 include/linux/jump_label.h        |   72 ++++++++++-------
 include/linux/jump_label_ref.h    |   41 ++++++----
 include/linux/perf_event.h        |   25 +++---
 include/linux/tracepoint.h        |    8 +-
 kernel/jump_label.c               |  159 ++++++++++++++++++++++++++++++-------
 kernel/perf_event.c               |    4 +-
 kernel/tracepoint.c               |   22 ++---
 10 files changed, 243 insertions(+), 136 deletions(-)
#1 lore raw

PATCH 1/2 jump label: make enable/disable o(1)

Previously, I allowed any variable type to be used as the 'key' for the jump label. However, by enforcing a type, we can make use of the contents of the 'key'. This patch thus introduces:

struct jump_label_key {
       void *ptr;
};

The 'ptr' is used a pointer into the jump label table of the corresponding addresses that need to be updated. Thus, when jump labels are enabled/disabled we have a constant time algorithm. There is no longer any hashing.

When jump lables are disabled we simply have:

struct jump_label_key {
        int state;
};

I've also defined an analogous structure for ref counted jump labels as per a request from Peter.

struct jump_label_keyref {
       void *ptr;
};

And for the jump labels disabled case:

struct jump_label_keyref {
        atomic_t refcount;
};

The reason I've introduced an additional structure for the reference counted jump labels is twofold:

1) For the jump labels disabled case, reference counted jump labels use an atomic_read(). I didn't want to impact the jump labels disabled case for tracepoints which simply accesses an 'int'.

2) By introducing a second type, we have two parallel APIs:

extern void jump_label_enable(struct jump_label_key *key); extern void jump_label_disable(struct jump_label_key *key);

static inline void jump_label_inc(struct jump_label_keyref *key) static inline void jump_label_dec(struct jump_label_keyref *key)

In this way, we can't mix up the reference counted API, with the straight enable/disable API since they accept different types.

I tested enable/disable times on x86 on a quad core via:

time echo 1 > /sys/kernel/debug/tracing/events/enable

With this patch, runs average .03s. Prior to the jump label infrastructure this command averaged around .01s.

We can speed this path up further via batching the enable/disables.

thanks,

-Jason

---
 include/linux/dynamic_debug.h  |    6 +-
 include/linux/jump_label.h     |   50 +++++++++----
 include/linux/jump_label_ref.h |   26 ++++---
 include/linux/perf_event.h     |    4 +-
 include/linux/tracepoint.h     |    6 +-
 kernel/jump_label.c            |  157 +++++++++++++++++++++++++++++++++-------
 kernel/perf_event.c            |    4 +-
 kernel/tracepoint.c            |   22 ++----
 8 files changed, 196 insertions(+), 79 deletions(-)
include/linux/dynamic_debug.h +3 -3
index a90b389..ddf7bae 100644
@@ -33,7 +33,7 @@ struct _ddebug {
 #define _DPRINTK_FLAGS_PRINT   (1<<0)  /* printk() a message using the format */
 #define _DPRINTK_FLAGS_DEFAULT 0
 	unsigned int flags:8;
-	char enabled;
+	struct jump_label_key enabled;
 } __attribute__((aligned(8)));
 
 
@@ -50,7 +50,7 @@ extern int ddebug_remove_module(const char *mod_name);
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
 	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
-		_DPRINTK_FLAGS_DEFAULT };				\
+		_DPRINTK_FLAGS_DEFAULT, JUMP_LABEL_INIT };		\
 	JUMP_LABEL(&descriptor.enabled, do_printk);			\
 	goto out;							\
 do_printk:								\
@@ -66,7 +66,7 @@ out:	;								\
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
 	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
-		_DPRINTK_FLAGS_DEFAULT };				\
+		_DPRINTK_FLAGS_DEFAULT, JUMP_LABEL_INIT };		\
 	JUMP_LABEL(&descriptor.enabled, do_printk);			\
 	goto out;							\
 do_printk:								\
include/linux/jump_label.h +34 -16

79 lines not on this page · raw message

include/linux/jump_label_ref.h +14 -12
index e5d012a..2dc9ddc 100644
@@ -6,36 +6,38 @@
 
 #ifdef HAVE_JUMP_LABEL
 
-static inline void jump_label_inc(atomic_t *key)
+static inline void jump_label_inc(struct jump_label_keyref *key)
 {
-	if (atomic_add_return(1, key) == 1)
-		jump_label_enable(key);
+	__jump_label_inc((struct jump_label_key *)key);
 }
 
-static inline void jump_label_dec(atomic_t *key)
+static inline void jump_label_dec(struct jump_label_keyref *key)
 {
-	if (atomic_dec_and_test(key))
-		jump_label_disable(key);
+	__jump_label_dec((struct jump_label_key *)key);
 }
 
 #else /* !HAVE_JUMP_LABEL */
 
-static inline void jump_label_inc(atomic_t *key)
+struct jump_label_keyref {
+	atomic_t refcount;
+};
+
+static inline void jump_label_inc(struct jump_label_keyref *key)
 {
-	atomic_inc(key);
+	atomic_inc(&key->refcount);
 }
 
-static inline void jump_label_dec(atomic_t *key)
+static inline void jump_label_dec(struct jump_label_keyref *key)
 {
-	atomic_dec(key);
+	atomic_dec(&key->refcount);
 }
 
 #undef JUMP_LABEL
 #define JUMP_LABEL(key, label)						\
 do {									\
 	if (unlikely(__builtin_choose_expr(				\
-	      __builtin_types_compatible_p(typeof(key), atomic_t *),	\
-	      atomic_read((atomic_t *)(key)), *(key))))			\
+	      __builtin_types_compatible_p(typeof(key), struct jump_label_keyref *),\
+	      atomic_read(&(((struct jump_label_keyref *)key)->refcount)), (((struct jump_label_key *)key)->state))))			\
 		goto label;						\
 } while (0)
 
include/linux/perf_event.h +2 -2
index dda5b0a..77c4645 100644
@@ -1000,7 +1000,7 @@ static inline int is_software_event(struct perf_event *event)
 	return event->pmu->task_ctx_nr == perf_sw_context;
 }
 
-extern atomic_t perf_swevent_enabled[PERF_COUNT_SW_MAX];
+extern struct jump_label_keyref perf_swevent_enabled[PERF_COUNT_SW_MAX];
 
 extern void __perf_sw_event(u32, u64, int, struct pt_regs *, u64);
 
@@ -1040,7 +1040,7 @@ have_event:
 	__perf_sw_event(event_id, nr, nmi, regs, addr);
 }
 
-extern atomic_t perf_task_events;
+extern struct jump_label_keyref perf_task_events;
 
 static inline void perf_event_task_sched_in(struct task_struct *task)
 {
include/linux/tracepoint.h +3 -3
index 5a6074f..e6f9793 100644
@@ -29,7 +29,7 @@ struct tracepoint_func {
 
 struct tracepoint {
 	const char *name;		/* Tracepoint name */
-	int state;			/* State. */
+	struct jump_label_key key;
 	void (*regfunc)(void);
 	void (*unregfunc)(void);
 	struct tracepoint_func *funcs;
@@ -146,7 +146,7 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin,
 	extern struct tracepoint __tracepoint_##name;			\
 	static inline void trace_##name(proto)				\
 	{								\
-		JUMP_LABEL(&__tracepoint_##name.state, do_trace);	\
+		JUMP_LABEL(&__tracepoint_##name.key, do_trace);		\
 		return;							\
 do_trace:								\
 			__DO_TRACE(&__tracepoint_##name,		\
@@ -175,7 +175,7 @@ do_trace:								\
 	__attribute__((section("__tracepoints_strings"))) = #name;	\
 	struct tracepoint __tracepoint_##name				\
 	__attribute__((section("__tracepoints"), aligned(32))) =	\
-		{ __tpstrtab_##name, 0, reg, unreg, NULL }
+		{ __tpstrtab_##name, JUMP_LABEL_INIT, reg, unreg, NULL }
 
 #define DEFINE_TRACE(name)						\
 	DEFINE_TRACE_FN(name, NULL, NULL);
kernel/jump_label.c +130 -27

227 lines not on this page · raw message

kernel/perf_event.c +2 -2
index 11847bf..d8b6188 100644
@@ -38,7 +38,7 @@
 
 #include <asm/irq_regs.h>
 
-atomic_t perf_task_events __read_mostly;
+struct jump_label_keyref perf_task_events __read_mostly;
 static atomic_t nr_mmap_events __read_mostly;
 static atomic_t nr_comm_events __read_mostly;
 static atomic_t nr_task_events __read_mostly;
@@ -4821,7 +4821,7 @@ fail:
 	return err;
 }
 
-atomic_t perf_swevent_enabled[PERF_COUNT_SW_MAX];
+struct jump_label_keyref perf_swevent_enabled[PERF_COUNT_SW_MAX];
 
 static void sw_perf_event_destroy(struct perf_event *event)
 {
kernel/tracepoint.c +8 -14
index e95ee7f..d54b434 100644
@@ -251,9 +251,9 @@ static void set_tracepoint(struct tracepoint_entry **entry,
 {
 	WARN_ON(strcmp((*entry)->name, elem->name) != 0);
 
-	if (elem->regfunc && !elem->state && active)
+	if (elem->regfunc && !jump_label_enabled(&elem->key) && active)
 		elem->regfunc();
-	else if (elem->unregfunc && elem->state && !active)
+	else if (elem->unregfunc && jump_label_enabled(&elem->key) && !active)
 		elem->unregfunc();
 
 	/*
@@ -264,13 +264,10 @@ static void set_tracepoint(struct tracepoint_entry **entry,
 	 * is used.
 	 */
 	rcu_assign_pointer(elem->funcs, (*entry)->funcs);
-	if (!elem->state && active) {
-		jump_label_enable(&elem->state);
-		elem->state = active;
-	} else if (elem->state && !active) {
-		jump_label_disable(&elem->state);
-		elem->state = active;
-	}
+	if (active)
+		jump_label_enable(&elem->key);
+	else if (!active)
+		jump_label_disable(&elem->key);
 }
 
 /*
@@ -281,13 +278,10 @@ static void set_tracepoint(struct tracepoint_entry **entry,
  */
 static void disable_tracepoint(struct tracepoint *elem)
 {
-	if (elem->unregfunc && elem->state)
+	if (elem->unregfunc && jump_label_enabled(&elem->key))
 		elem->unregfunc();
 
-	if (elem->state) {
-		jump_label_disable(&elem->state);
-		elem->state = 0;
-	}
+	jump_label_disable(&elem->key);
 	rcu_assign_pointer(elem->funcs, NULL);
 }
 
Signature · 1 line
1.7.1
maintainer
#2 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 42 lines · Jason Baron

Previously, I allowed any variable type to be used as the 'key' for the jump label. However, by enforcing a type, we can make use of the contents of the 'key'. This patch thus introduces:

struct jump_label_key {
       void *ptr;
};

The 'ptr' is used a pointer into the jump label table of the corresponding addresses that need to be updated. Thus, when jump labels are enabled/disabled we have a constant time algorithm. There is no longer any hashing.

When jump lables are disabled we simply have:

struct jump_label_key {
        int state;
};

I've also defined an analogous structure for ref counted jump labels as per a request from Peter.

struct jump_label_keyref {
       void *ptr;
};

And for the jump labels disabled case:

struct jump_label_keyref {
        atomic_t refcount;
};

The reason I've introduced an additional structure for the reference counted jump labels is twofold:

1) For the jump labels disabled case, reference counted jump labels use an atomic_read(). I didn't want to impact the jump labels disabled case for tracepoints which simply accesses an 'int'.

2) By introducing a second type, we have two parallel APIs:

Quoted 8 lines

extern void jump_label_enable(struct jump_label_key *key); extern void jump_label_disable(struct jump_label_key *key);

static inline void jump_label_inc(struct jump_label_keyref *key) static inline void jump_label_dec(struct jump_label_keyref *key)

In this way, we can't mix up the reference counted API, with the straight enable/disable API since they accept different types.

But why do we want to have two APIs?

#3 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 46 lines · Peter Zijlstra
Quoted 45 lines · Jason Baron

Previously, I allowed any variable type to be used as the 'key' for the jump label. However, by enforcing a type, we can make use of the contents of the 'key'. This patch thus introduces:

struct jump_label_key {
       void *ptr;
};

The 'ptr' is used a pointer into the jump label table of the corresponding addresses that need to be updated. Thus, when jump labels are enabled/disabled we have a constant time algorithm. There is no longer any hashing.

When jump lables are disabled we simply have:

struct jump_label_key {
        int state;
};

I've also defined an analogous structure for ref counted jump labels as per a request from Peter.

struct jump_label_keyref {
       void *ptr;
};

And for the jump labels disabled case:

struct jump_label_keyref {
        atomic_t refcount;
};

The reason I've introduced an additional structure for the reference counted jump labels is twofold:

1) For the jump labels disabled case, reference counted jump labels use an atomic_read(). I didn't want to impact the jump labels disabled case for tracepoints which simply accesses an 'int'.

2) By introducing a second type, we have two parallel APIs:

extern void jump_label_enable(struct jump_label_key *key); extern void jump_label_disable(struct jump_label_key *key);

Quoted 7 lines
Quoted 5 lines

static inline void jump_label_inc(struct jump_label_keyref *key) static inline void jump_label_dec(struct jump_label_keyref *key)

In this way, we can't mix up the reference counted API, with the straight enable/disable API since they accept different types.

But why do we want to have two APIs?

its not 2 APIS doing the same thing...one does refcounting, the other is a straight enable/disable.

For the jump label disabled case, perf is using atomic_inc/dec and atomic_read to check if enabled. While other consumers (tracepoints) are just using an 'int'. I didn't want hurt the jump label disabled case for tracepoints. If we can agree to use atomic ops for tracepoints, or drop atomics from perf, that would simplify things.

For the jump albel enabled case, there is no issue.

thanks,

-Jason

maintainer
#4 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 6 lines · Jason Baron

For the jump label disabled case, perf is using atomic_inc/dec and atomic_read to check if enabled. While other consumers (tracepoints) are just using an 'int'. I didn't want hurt the jump label disabled case for tracepoints. If we can agree to use atomic ops for tracepoints, or drop atomics from perf, that would simplify things.

I had a quick look at the tracepoint stuff but got lost, but surely it has a reference count somewhere as well, it needs to know when the last probe goes away.. or does it check if the list is empty?

Anyway, tracepoint enable/disable isn't a real fast-path, surely it could suffer an atomic op?

#5 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 8 lines · Peter Zijlstra
Quoted 6 lines · Jason Baron

For the jump label disabled case, perf is using atomic_inc/dec and atomic_read to check if enabled. While other consumers (tracepoints) are just using an 'int'. I didn't want hurt the jump label disabled case for tracepoints. If we can agree to use atomic ops for tracepoints, or drop atomics from perf, that would simplify things.

Quoted 6 lines

I had a quick look at the tracepoint stuff but got lost, but surely it has a reference count somewhere as well, it needs to know when the last probe goes away.. or does it check if the list is empty?

Anyway, tracepoint enable/disable isn't a real fast-path, surely it could suffer an atomic op?

It is the atomic_read() at the tracepoint site that I am concerned about.

thanks,

-Jason

maintainer
#6 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 13 lines · Jason Baron
Quoted 12 lines · Peter Zijlstra
Quoted 6 lines · Jason Baron

For the jump label disabled case, perf is using atomic_inc/dec and atomic_read to check if enabled. While other consumers (tracepoints) are just using an 'int'. I didn't want hurt the jump label disabled case for tracepoints. If we can agree to use atomic ops for tracepoints, or drop atomics from perf, that would simplify things.

I had a quick look at the tracepoint stuff but got lost, but surely it has a reference count somewhere as well, it needs to know when the last probe goes away.. or does it check if the list is empty?

Quoted 5 lines
Quoted 2 lines

Anyway, tracepoint enable/disable isn't a real fast-path, surely it could suffer an atomic op?

It is the atomic_read() at the tracepoint site that I am concerned about.

Look at the implementation :-), its just wrapper foo, its a regular read for everything except some really weird archs (you really shouldn't care about).

static inline int atomic_read(const atomic_t *v)

{
        return (*(volatile int *)&(v)->counter);
}

The volatile simply forces a load to be emitted.

#7 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Jason Baron
Quoted 12 lines · Peter Zijlstra
Quoted 6 lines · Jason Baron

For the jump label disabled case, perf is using atomic_inc/dec and atomic_read to check if enabled. While other consumers (tracepoints) are just using an 'int'. I didn't want hurt the jump label disabled case for tracepoints. If we can agree to use atomic ops for tracepoints, or drop atomics from perf, that would simplify things.

I had a quick look at the tracepoint stuff but got lost, but surely it has a reference count somewhere as well, it needs to know when the last probe goes away.. or does it check if the list is empty?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Anyway, tracepoint enable/disable isn't a real fast-path, surely it could suffer an atomic op?

It is the atomic_read() at the tracepoint site that I am concerned about.

Look at the implementation :-), its just wrapper foo, its a regular read

i did.

Quoted 2 lines

for everything except some really weird archs (you really shouldn't care about).

right, I wasn't sure how much those mattered.

Quoted 7 lines

static inline int atomic_read(const atomic_t *v)

{
        return (*(volatile int *)&(v)->counter);
}

The volatile simply forces a load to be emitted.

Mathieu, what do you think? Are you ok with an atomic_read() for checking if a tracepoint is enabled, when jump labels are disabled?

thanks,

-Jason

maintainer
#8 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 10 lines · Jason Baron
Quoted 7 lines

static inline int atomic_read(const atomic_t *v)

{
        return (*(volatile int *)&(v)->counter);
}

The volatile simply forces a load to be emitted.

Mathieu, what do you think? Are you ok with an atomic_read() for checking if a tracepoint is enabled, when jump labels are disabled?

Note, I'm fine with this method too. An atomic_read() is extremely fast. The worse that it does is to prevent gcc from optimizing a little, which we already cause it to do due to the asm goto that we use.

-- Steve

#9 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

In reply to Jason Baron
Quoted 36 lines · * Jason Baron ([email protected]) wrote:
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Jason Baron
Quoted 12 lines · Peter Zijlstra
Quoted 6 lines · Jason Baron

For the jump label disabled case, perf is using atomic_inc/dec and atomic_read to check if enabled. While other consumers (tracepoints) are just using an 'int'. I didn't want hurt the jump label disabled case for tracepoints. If we can agree to use atomic ops for tracepoints, or drop atomics from perf, that would simplify things.

I had a quick look at the tracepoint stuff but got lost, but surely it has a reference count somewhere as well, it needs to know when the last probe goes away.. or does it check if the list is empty?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Anyway, tracepoint enable/disable isn't a real fast-path, surely it could suffer an atomic op?

It is the atomic_read() at the tracepoint site that I am concerned about.

Look at the implementation :-), its just wrapper foo, its a regular read

i did.

Quoted 2 lines

for everything except some really weird archs (you really shouldn't care about).

right, I wasn't sure how much those mattered.

Quoted 6 lines

static inline int atomic_read(const atomic_t *v)

{
        return (*(volatile int *)&(v)->counter);
}
Quoted 4 lines
Quoted 1 line

The volatile simply forces a load to be emitted.

Mathieu, what do you think? Are you ok with an atomic_read() for checking if a tracepoint is enabled, when jump labels are disabled?

volatiles are also ordered one with respect to another by the compiler, so I'd like to avoid doing the volatile read on the fast path unless utterly necessary for architectures not supporting static jump patching yet.

Tracepoints keep their own reference counts for enable/disable, so a simple "enable/disable" is fine as far as tracepoints are concerned. Why does perf need that refcounting done by the static jumps ?

I'd be tempted to use a "char" instead of a int/atomic_t for the key, and keep the reference counter out of the picture (and leave that to the caller). A single byte can be updated and read atomically. This should save memory for the jump label tables.

Thoughts ?

Thanks,

Mathieu

Quoted 4 lines

thanks,

-Jason

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
maintainer
#10 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 3 lines · Mathieu Desnoyers

Tracepoints keep their own reference counts for enable/disable, so a simple "enable/disable" is fine as far as tracepoints are concerned. Why does perf need that refcounting done by the static jumps ?

Because the refcount is all we have... Why not replace that tracepoint refcount with the jumplabel thing?

#11 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 7 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 3 lines · Mathieu Desnoyers

Tracepoints keep their own reference counts for enable/disable, so a simple "enable/disable" is fine as far as tracepoints are concerned. Why does perf need that refcounting done by the static jumps ?

Because the refcount is all we have... Why not replace that tracepoint refcount with the jumplabel thing?

The reason why tracepoints need to keep their own refcount is because they support dynamically loadable modules, and hence the refcount must be kept outside of the modules, in a table internal to tracepoints, so we can attach a probe to a yet unloaded module. Therefore, relying on this lower level jump label to keep the refcount is not appropriate for tracepoints, because the refcount only exists when the module is live.

I know that your point of view is "let users of modules suffer", but this represents a very large portion of Linux users I am not willing to let suffer knowingly.

Thanks,

Mathieu

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
maintainer
#12 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 9 lines · Mathieu Desnoyers
Quoted 7 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 3 lines · Mathieu Desnoyers

Tracepoints keep their own reference counts for enable/disable, so a simple "enable/disable" is fine as far as tracepoints are concerned. Why does perf need that refcounting done by the static jumps ?

Because the refcount is all we have... Why not replace that tracepoint refcount with the jumplabel thing?

Quoted 6 lines

The reason why tracepoints need to keep their own refcount is because they support dynamically loadable modules, and hence the refcount must be kept outside of the modules, in a table internal to tracepoints, so we can attach a probe to a yet unloaded module. Therefore, relying on this lower level jump label to keep the refcount is not appropriate for tracepoints, because the refcount only exists when the module is live.

That's not a logical conclusion, you can keep these jump_label keys outside of the module just fine.

Quoted 3 lines

I know that your point of view is "let users of modules suffer", but this represents a very large portion of Linux users I am not willing to let suffer knowingly.

Feh, I'd argue to remove this special tracepoint crap, the only in-kernel user (ftrace) doesn't even make use of it. This weird ass tracepoint semantic being different from the ftrace trace_event semantics has caused trouble before.

#13 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 20 lines · Peter Zijlstra
Quoted 9 lines · Mathieu Desnoyers
Quoted 7 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 3 lines · Mathieu Desnoyers

Tracepoints keep their own reference counts for enable/disable, so a simple "enable/disable" is fine as far as tracepoints are concerned. Why does perf need that refcounting done by the static jumps ?

Because the refcount is all we have... Why not replace that tracepoint refcount with the jumplabel thing?

Quoted 6 lines

The reason why tracepoints need to keep their own refcount is because they support dynamically loadable modules, and hence the refcount must be kept outside of the modules, in a table internal to tracepoints, so we can attach a probe to a yet unloaded module. Therefore, relying on this lower level jump label to keep the refcount is not appropriate for tracepoints, because the refcount only exists when the module is live.

That's not a logical conclusion, you can keep these jump_label keys outside of the module just fine.

Quoted 10 lines
Quoted 3 lines

I know that your point of view is "let users of modules suffer", but this represents a very large portion of Linux users I am not willing to let suffer knowingly.

Feh, I'd argue to remove this special tracepoint crap, the only in-kernel user (ftrace) doesn't even make use of it. This weird ass tracepoint semantic being different from the ftrace trace_event semantics has caused trouble before.

Hi,

since atomic_t is just an 'int' from include/linux/types.h, so for all arches. We can cast any refernces to an atomic_t in include/linux/jump_label_ref.h

So for when jump labels are disabled case we could have one struct:

struct jump_label_key {
	int state;
}

and then we could then have (rough c code):

jump_label_enable(struct jump_label_key *key)

{
	key->state = 1;
}

jump_label_disable(struct jump_label_key *key)

{
	key->state = 0;
}

jump_label_inc(struct jump_label_key *key)

{
	atomic_inc((atomic_t *)key)
}

jump_label_dec(struct jump_label_key *key)

{
	atomic_dec((atomic_t *)key)
}

bool unlikely_switch(struct jump_label_key *key)

{
	if (key->state)
		return true;
	return false;
}

bool unlikely_switch_atomic(struct jump_label_key *key)

{
	if (atomic_read((atomic_t *)key)
		return true;
	return false;
}

can we agree on something like this?

thanks,

-Jason

#14 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 32 lines · Jason Baron
Quoted 20 lines · Peter Zijlstra
Quoted 9 lines · Mathieu Desnoyers
Quoted 7 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 3 lines · Mathieu Desnoyers

Tracepoints keep their own reference counts for enable/disable, so a simple "enable/disable" is fine as far as tracepoints are concerned. Why does perf need that refcounting done by the static jumps ?

Because the refcount is all we have... Why not replace that tracepoint refcount with the jumplabel thing?

Quoted 6 lines

The reason why tracepoints need to keep their own refcount is because they support dynamically loadable modules, and hence the refcount must be kept outside of the modules, in a table internal to tracepoints, so we can attach a probe to a yet unloaded module. Therefore, relying on this lower level jump label to keep the refcount is not appropriate for tracepoints, because the refcount only exists when the module is live.

That's not a logical conclusion, you can keep these jump_label keys outside of the module just fine.

Quoted 10 lines
Quoted 3 lines

I know that your point of view is "let users of modules suffer", but this represents a very large portion of Linux users I am not willing to let suffer knowingly.

Feh, I'd argue to remove this special tracepoint crap, the only in-kernel user (ftrace) doesn't even make use of it. This weird ass tracepoint semantic being different from the ftrace trace_event semantics has caused trouble before.

Quoted 6 lines

Hi,

since atomic_t is just an 'int' from include/linux/types.h, so for all arches. We can cast any refernces to an atomic_t in include/linux/jump_label_ref.h

Not acceptable I would think.

How about:

union fubar {
int key_as_non_atomic;
atomic_t key_as_atomic;
};

Now explain the exact semantics of this thing including how you guarantee no conflicting accesses *ever* occur.

Quoted 36 lines

So for when jump labels are disabled case we could have one struct:

struct jump_label_key {
	int state;
}

and then we could then have (rough c code):

jump_label_enable(struct jump_label_key *key)

{
	key->state = 1;
}

jump_label_disable(struct jump_label_key *key)

{
	key->state = 0;
}

jump_label_inc(struct jump_label_key *key)

{
	atomic_inc((atomic_t *)key)
}

jump_label_dec(struct jump_label_key *key)

{
	atomic_dec((atomic_t *)key)
}

bool unlikely_switch(struct jump_label_key *key)

{
	if (key->state)
		return true;
	return false;
}
Quoted 8 lines

bool unlikely_switch_atomic(struct jump_label_key *key)

{
	if (atomic_read((atomic_t *)key)
		return true;
	return false;
}

can we agree on something like this?

I get a sick feeling whenever casting is used to give types with well defined semantics (atomic_t) poorly defined semantics (your usage).

David Daney

maintainer
#15 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 10 lines · David Daney

On 12/17/2010 12:07 PM, Jason Baron wrote:

Not acceptable I would think.

How about:

union fubar {
    int key_as_non_atomic;
    atomic_t key_as_atomic;
};

I don't even like this union.

Quoted 3 lines

Now explain the exact semantics of this thing including how you guarantee no conflicting accesses *ever* occur.

I don't like the mixed semantics at all.

Quoted 6 lines
Quoted 6 lines

So for when jump labels are disabled case we could have one struct:

struct jump_label_key {

	atomic_t state;
Quoted 6 lines
Quoted 6 lines

}

and then we could then have (rough c code):

jump_label_enable(struct jump_label_key *key)
{

	if (atomic_read(&key->state))
		return;
	atomic_inc(&key->state);
Quoted 4 lines
Quoted 4 lines

}

jump_label_disable(struct jump_label_key *key)
{

	if (!atomic_read(&key->state))
		return;
	atomic_dec(&key->state);
	WARN_ON(atomic_read(&key->state);
Quoted 4 lines
Quoted 4 lines

}

jump_label_inc(struct jump_label_key *key)
{

atomic_inc(&key->state)

Quoted 4 lines
Quoted 4 lines

}

jump_label_dec(struct jump_label_key *key)
{

atomic_dec((&key->state)

Quoted 4 lines
Quoted 4 lines

}

bool unlikely_switch(struct jump_label_key *key)
{

if (atomic_read(&key->state))

Quoted 4 lines
Quoted 4 lines
		return true;
	return false;
}

There, now you are guaranteed that you have proper semantics.

Quoted 5 lines
Quoted 2 lines

can we agree on something like this?

I get a sick feeling whenever casting is used to give types with well defined semantics (atomic_t) poorly defined semantics (your usage).

Exactly, I like to avoid (void*) anything or even worse, casting one type to another for some strange semantics. This is guaranteed nightmare of maintenance.

-- Steve

#16 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

Quoted 61 lines · Steven Rostedt
Quoted 10 lines · Jason Baron

Not acceptable I would think.

How about:

union fubar {
    int key_as_non_atomic;
    atomic_t key_as_atomic;
};

I don't even like this union.

Quoted 3 lines

Now explain the exact semantics of this thing including how you guarantee no conflicting accesses *ever* occur.

I don't like the mixed semantics at all.

Quoted 6 lines
Quoted 6 lines

So for when jump labels are disabled case we could have one struct:

struct jump_label_key {

	atomic_t state;
Quoted 6 lines
Quoted 6 lines

}

and then we could then have (rough c code):

jump_label_enable(struct jump_label_key *key)
{

	if (atomic_read(&key->state))
		return;
	atomic_inc(&key->state);
Quoted 4 lines
Quoted 4 lines

}

jump_label_disable(struct jump_label_key *key)
{

	if (!atomic_read(&key->state))
		return;
	atomic_dec(&key->state);
	WARN_ON(atomic_read(&key->state);
Quoted 4 lines
Quoted 4 lines

}

jump_label_inc(struct jump_label_key *key)
{

atomic_inc(&key->state)

Quoted 4 lines
Quoted 4 lines

}

jump_label_dec(struct jump_label_key *key)
{

Quoted 14 lines

atomic_dec((&key->state)

Quoted 4 lines
Quoted 4 lines

}

bool unlikely_switch(struct jump_label_key *key)
{

if (atomic_read(&key->state))

Quoted 4 lines
Quoted 4 lines
		return true;
	return false;
}

hmmm...we were trying to avoid having the atomic_read() for tracepoints b/c of potential extra cost that Mathieu was concerned about.

Quoted 3 lines

There, now you are guaranteed that you have proper semantics.

Quoted 1 line
Quoted 1 line

The other issue here was that jump_label.h gets included by asm/atomic.h, so there a dependency issue to be addressed here as well....

thanks,

-Jason

#17 lore raw

Re: [PATCH/RFC 1/2] jump label: make enable/disable o(1)

In reply to Jason Baron
Quoted 36 lines · * Jason Baron ([email protected]) wrote:
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Jason Baron
Quoted 12 lines · Peter Zijlstra
Quoted 6 lines · Jason Baron

For the jump label disabled case, perf is using atomic_inc/dec and atomic_read to check if enabled. While other consumers (tracepoints) are just using an 'int'. I didn't want hurt the jump label disabled case for tracepoints. If we can agree to use atomic ops for tracepoints, or drop atomics from perf, that would simplify things.

I had a quick look at the tracepoint stuff but got lost, but surely it has a reference count somewhere as well, it needs to know when the last probe goes away.. or does it check if the list is empty?

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Anyway, tracepoint enable/disable isn't a real fast-path, surely it could suffer an atomic op?

It is the atomic_read() at the tracepoint site that I am concerned about.

Look at the implementation :-), its just wrapper foo, its a regular read

i did.

Quoted 2 lines

for everything except some really weird archs (you really shouldn't care about).

right, I wasn't sure how much those mattered.

Quoted 6 lines

static inline int atomic_read(const atomic_t *v)

{
        return (*(volatile int *)&(v)->counter);
}
Quoted 4 lines
Quoted 1 line

The volatile simply forces a load to be emitted.

Mathieu, what do you think? Are you ok with an atomic_read() for checking if a tracepoint is enabled, when jump labels are disabled?

[Steven:]

Note, I'm fine with this method too. An atomic_read() is extremely fast. The worse that it does is to prevent gcc from optimizing a little, which we already cause it to do due to the asm goto that we use.

[my reply to Steven]

How does the asm goto we use prohibit the compiler from moving code in any way more than an standard branch ? It's not an "asm volatile goto", just an asm goto.

Thanks,

Mathieu

Quoted 4 lines

thanks,

-Jason

Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
#18 lore raw

PATCH 2/2 jump label: introduce unlikely_switch()

In reply to Jason Baron

Introduce:

static __always_inline bool unlikely_switch(struct jump_label_key *key);

to replace the old JUMP_LABEL(key, label) macro.

The new unlikely_switch(), simplifies the usage of jump labels. Since unlikely_switch() returns a boolean, it can be used as part of an if() construct. It also, allows us to drop the 'label' argument from the prototype. Its probably best understood with an example, here is the part of the patch that converts the tracepoints to use unlikely_switch():

include/linux/tracepoint.h +1 -3
@@ -146,9 +146,7 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin,
 	extern struct tracepoint __tracepoint_##name;			\
 	static inline void trace_##name(proto)				\
 	{								\
-		JUMP_LABEL(&__tracepoint_##name.key, do_trace);		\
-		return;							\
-do_trace:								\
+		if (unlikely_switch(&__tracepoint_##name.key))		\
 			__DO_TRACE(&__tracepoint_##name,		\
 				TP_PROTO(data_proto),			\
 				TP_ARGS(data_args));			\

The name 'unlikely_switch' is meant to invoke the notion of 'unlikely', in that we expect this code path to be disabled most of the time. The 'switch' notion is that we really have a switch here that can be flipped, by patching either a 'nop' or a 'jmp' at the switchpoint.

I analyzed the code produced by unlikely_switch(), and it seems to be at least as good as the code generated by the JUMP_LABEL(). As a reminder, we get a single nop in the fastpath for -02. But will often times get a 'double jmp' in the -Os case. That is, 'jmp 0', followed by a jmp around the disabled code. We believe that future gcc tweaks to allow block re-ordering in the -Os, will solve the -Os case in the future.

I also saw a 1-2% tbench throughput improvement when compiling with jump labels in the -02 case.

Thanks to H. Peter Anvin for suggesting this improved syntax as well as the name 'switchpoint'.

---
 arch/x86/include/asm/jump_label.h |   22 +++++++++++++---------
 arch/x86/kernel/jump_label.c      |    2 +-
 include/linux/dynamic_debug.h     |   18 ++++--------------
 include/linux/jump_label.h        |   26 +++++++++++---------------
 include/linux/jump_label_ref.h    |   19 +++++++++++--------
 include/linux/perf_event.h        |   21 ++++++++++-----------
 include/linux/tracepoint.h        |    4 +---
 kernel/jump_label.c               |    2 +-
 8 files changed, 52 insertions(+), 62 deletions(-)
arch/x86/include/asm/jump_label.h +13 -9
index f52d42e..172af9b 100644
@@ -5,20 +5,24 @@
 
 #include <linux/types.h>
 #include <asm/nops.h>
+#include <asm/asm.h>
 
 #define JUMP_LABEL_NOP_SIZE 5
 
 # define JUMP_LABEL_INITIAL_NOP ".byte 0xe9 \n\t .long 0\n\t"
 
-# define JUMP_LABEL(key, label)					\
-	do {							\
-		asm goto("1:"					\
-			JUMP_LABEL_INITIAL_NOP			\
-			".pushsection __jump_table,  \"a\" \n\t"\
-			_ASM_PTR "1b, %l[" #label "], %c0 \n\t" \
-			".popsection \n\t"			\
-			: :  "i" (key) :  : label);		\
-	} while (0)
+static __always_inline bool __unlikely_switch(struct jump_label_key *key)
+{
+	asm goto("1:"
+		JUMP_LABEL_INITIAL_NOP
+		".pushsection __jump_table,  \"a\" \n\t"
+		_ASM_PTR "1b, %l[l_yes], %c0 \n\t"
+		".popsection \n\t"
+		: :  "i" (key) : : l_yes );
+	return false;
+l_yes:
+	return true;
+}
 
 #endif /* __KERNEL__ */
 
arch/x86/kernel/jump_label.c +1 -1
index 961b6b3..dfa4c3c 100644
@@ -4,13 +4,13 @@
  * Copyright (C) 2009 Jason Baron <[email protected]>
  *
  */
-#include <linux/jump_label.h>
 #include <linux/memory.h>
 #include <linux/uaccess.h>
 #include <linux/module.h>
 #include <linux/list.h>
 #include <linux/jhash.h>
 #include <linux/cpu.h>
+#include <linux/jump_label.h>
 #include <asm/kprobes.h>
 #include <asm/alternative.h>
 
include/linux/dynamic_debug.h +4 -14
index ddf7bae..71d18a8 100644
@@ -44,34 +44,24 @@ int ddebug_add_module(struct _ddebug *tab, unsigned int n,
 extern int ddebug_remove_module(const char *mod_name);
 
 #define dynamic_pr_debug(fmt, ...) do {					\
-	__label__ do_printk;						\
-	__label__ out;							\
 	static struct _ddebug descriptor				\
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
 	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
 		_DPRINTK_FLAGS_DEFAULT, JUMP_LABEL_INIT };		\
-	JUMP_LABEL(&descriptor.enabled, do_printk);			\
-	goto out;							\
-do_printk:								\
-	printk(KERN_DEBUG pr_fmt(fmt),	##__VA_ARGS__);			\
-out:	;								\
+	if (unlikely_switch(&descriptor.enabled))			\
+		printk(KERN_DEBUG pr_fmt(fmt),	##__VA_ARGS__);		\
 	} while (0)
 
 
 #define dynamic_dev_dbg(dev, fmt, ...) do {				\
-	__label__ do_printk;						\
-	__label__ out;							\
 	static struct _ddebug descriptor				\
 	__used								\
 	__attribute__((section("__verbose"), aligned(8))) =		\
 	{ KBUILD_MODNAME, __func__, __FILE__, fmt, __LINE__,		\
 		_DPRINTK_FLAGS_DEFAULT, JUMP_LABEL_INIT };		\
-	JUMP_LABEL(&descriptor.enabled, do_printk);			\
-	goto out;							\
-do_printk:								\
-	dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__);		\
-out:	;								\
+	if (unlikely_switch(&descriptor.enabled))			\
+		dev_printk(KERN_DEBUG, dev, fmt, ##__VA_ARGS__);	\
 	} while (0)
 
 #else
include/linux/jump_label.h +11 -15
index 3e56668..5da64b7 100644
@@ -26,6 +26,11 @@ struct module;
 
 #ifdef HAVE_JUMP_LABEL
 
+static __always_inline bool unlikely_switch(struct jump_label_key *key)
+{
+	return __unlikely_switch(key);
+}
+
 extern struct jump_entry __start___jump_table[];
 extern struct jump_entry __stop___jump_table[];
 
@@ -48,11 +53,12 @@ struct jump_label_key {
 	int state;
 };
 
-#define JUMP_LABEL(key, label)			\
-do {						\
-	if (unlikely(((struct jump_label_key *)key)->state))		\
-		goto label;			\
-} while (0)
+static __always_inline bool unlikely_switch(struct jump_label_key *key)
+{
+	if (unlikely(key->state))
+		return true;
+	return false;
+}
 
 static inline int jump_label_enabled(struct jump_label_key *key)
 {
@@ -84,14 +90,4 @@ static inline void jump_label_unlock(void) {}
 
 #endif
 
-#define COND_STMT(key, stmt)					\
-do {								\
-	__label__ jl_enabled;					\
-	JUMP_LABEL(key, jl_enabled);				\
-	if (0) {						\
-jl_enabled:							\
-		stmt;						\
-	}							\
-} while (0)
-
 #endif
include/linux/jump_label_ref.h +11 -8
index 2dc9ddc..c1ae838 100644
@@ -16,6 +16,11 @@ static inline void jump_label_dec(struct jump_label_keyref *key)
 	__jump_label_dec((struct jump_label_key *)key);
 }
 
+static __always_inline bool unlikely_switch_refcount(struct jump_label_keyref *key)
+{
+	return __unlikely_switch((struct jump_label_key *)key);
+}
+
 #else /* !HAVE_JUMP_LABEL */
 
 struct jump_label_keyref {
@@ -32,14 +37,12 @@ static inline void jump_label_dec(struct jump_label_keyref *key)
 	atomic_dec(&key->refcount);
 }
 
-#undef JUMP_LABEL
-#define JUMP_LABEL(key, label)						\
-do {									\
-	if (unlikely(__builtin_choose_expr(				\
-	      __builtin_types_compatible_p(typeof(key), struct jump_label_keyref *),\
-	      atomic_read(&(((struct jump_label_keyref *)key)->refcount)), (((struct jump_label_key *)key)->state))))			\
-		goto label;						\
-} while (0)
+static __always_inline bool unlikely_switch_refcount(struct jump_label_keyref *key)
+{
+	if (unlikely(atomic_read(&key->refcount)))
+		return true;
+	return false;
+}
 
 #endif /* HAVE_JUMP_LABEL */
 
include/linux/perf_event.h +10 -11
index 77c4645..142e9b2 100644
@@ -1029,30 +1029,29 @@ perf_sw_event(u32 event_id, u64 nr, int nmi, struct pt_regs *regs, u64 addr)
 {
 	struct pt_regs hot_regs;
 
-	JUMP_LABEL(&perf_swevent_enabled[event_id], have_event);
-	return;
-
-have_event:
-	if (!regs) {
-		perf_fetch_caller_regs(&hot_regs);
-		regs = &hot_regs;
+	if (unlikely_switch_refcount(&perf_swevent_enabled[event_id])) {
+		if (!regs) {
+			perf_fetch_caller_regs(&hot_regs);
+			regs = &hot_regs;
+		}
+		__perf_sw_event(event_id, nr, nmi, regs, addr);
 	}
-	__perf_sw_event(event_id, nr, nmi, regs, addr);
 }
 
 extern struct jump_label_keyref perf_task_events;
 
 static inline void perf_event_task_sched_in(struct task_struct *task)
 {
-	COND_STMT(&perf_task_events, __perf_event_task_sched_in(task));
+	if (unlikely_switch_refcount(&perf_task_events))
+		__perf_event_task_sched_in(task);
 }
 
 static inline
 void perf_event_task_sched_out(struct task_struct *task, struct task_struct *next)
 {
 	perf_sw_event(PERF_COUNT_SW_CONTEXT_SWITCHES, 1, 1, NULL, 0);
-
-	COND_STMT(&perf_task_events, __perf_event_task_sched_out(task, next));
+	if (unlikely_switch_refcount(&perf_task_events))
+		__perf_event_task_sched_out(task, next);
 }
 
 extern void perf_event_mmap(struct vm_area_struct *vma);
include/linux/tracepoint.h +1 -3
index e6f9793..f1bcc5e 100644
@@ -146,9 +146,7 @@ static inline void tracepoint_update_probe_range(struct tracepoint *begin,
 	extern struct tracepoint __tracepoint_##name;			\
 	static inline void trace_##name(proto)				\
 	{								\
-		JUMP_LABEL(&__tracepoint_##name.key, do_trace);		\
-		return;							\
-do_trace:								\
+		if (unlikely_switch(&__tracepoint_##name.key))		\
 			__DO_TRACE(&__tracepoint_##name,		\
 				TP_PROTO(data_proto),			\
 				TP_ARGS(data_args));			\
kernel/jump_label.c +1 -1
index f8869d6..0c9f4d5 100644
@@ -4,7 +4,6 @@
  * Copyright (C) 2009 Jason Baron <[email protected]>
  *
  */
-#include <linux/jump_label.h>
 #include <linux/memory.h>
 #include <linux/uaccess.h>
 #include <linux/module.h>
@@ -13,6 +12,7 @@
 #include <linux/slab.h>
 #include <linux/sort.h>
 #include <linux/err.h>
+#include <linux/jump_label.h>
 
 #ifdef HAVE_JUMP_LABEL
 
Signature · 1 line
1.7.1
#19 lore raw
In reply to Jason Baron
Quoted 14 lines · * Jason Baron ([email protected]) wrote:

Hi,

The first patch uses the storage space of the jump label key address as a pointer into the update table. In this way, we can find all the addresses that need to be updated without hashing.

The second patch introduces:

static __always_inline bool unlikely_switch(struct jump_label_key *key);

instead of the old JUMP_LABEL(key, label) macro.

In this way, jump labels become really easy to use:

Quoted 8 lines

Define:

	struct jump_label_key jump_key;

Can be used as:

if (unlikely_switch(&jump_key))
do unlikely code

Ah, yes, that's an improvement!

I'm just wondering about the terminology here. Isn't that more a "branch" than a "switch" ?

I'm concerned about the fact that if we ever want to use the asm goto ability to jump to multiple targets (which is closer to a statically computed switch than a branch), we might want to reserve "switch" name for that rather than the branch.

I wonder if the "if (unlikely_switch(&jump_key))" you propose above is the right thing to do. Why does the unlikely_ have to be included in the name ? Maybe there is a good reason for it, but it would be nice to have it spelled out. We might consider:

if (unlikely(static_branch(&jump_key)))
...

instead.

For the switch statement, from the top of my head the idea would be to get something close to the following:

static __always_inline
int static_switch_{3,4,5,6...}(struct jump_label_key *key);

e.g.:

static __always_inline
int static_switch_3(struct jump_label_key *key)

{
	asm goto("1:"
		JUMP_LABEL_INITIAL_NOP
		".pushsection __switch_table_3,  \"a\" \n\t"
		_ASM_PTR "%c0, 1b, %l[l_1], %l[l_2] \n\t"
		".popsection \n\t"
		: :  "i" (key) : : l_1, l_2 );
	return 0;
l_1:
	return 1;
l_2:
	return 2;
}
switch(static_switch_3(&switch_key)) {
	case 0: .....
		break;
	case 1: .....
		break;
	case 2: .....
		break;
}

(I have not tried to give that to gcc 4.5.x to see how the resulting assembly looks like. It would be interesting to see if it handles this case well)

Thoughts ?

Thanks,

Mathieu

Quoted 32 lines

enable/disale via:

	jump_label_enable(&jump_key);
	jump_label_disable(&jump_key);

that's it!

Thanks to H. Peter Anvin for suggesting the simpler 'unlikely_switch()' function.

thanks,

-Jason

Jason Baron (2):
jump label: make enable/disable o(1)
jump label: introduce unlikely_switch()

 arch/x86/include/asm/jump_label.h |   22 +++--
 arch/x86/kernel/jump_label.c      |    2 +-
 include/linux/dynamic_debug.h     |   24 ++----
 include/linux/jump_label.h        |   72 ++++++++++-------
 include/linux/jump_label_ref.h    |   41 ++++++----
 include/linux/perf_event.h        |   25 +++---
 include/linux/tracepoint.h        |    8 +-
 kernel/jump_label.c               |  159 ++++++++++++++++++++++++++++++-------
 kernel/perf_event.c               |    4 +-
 kernel/tracepoint.c               |   22 ++---
 10 files changed, 243 insertions(+), 136 deletions(-)
Signature · 4 lines
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
maintainer
#20 lore raw
Quoted 26 lines · Mathieu Desnoyers
Quoted 14 lines · * Jason Baron ([email protected]) wrote:

Hi,

The first patch uses the storage space of the jump label key address as a pointer into the update table. In this way, we can find all the addresses that need to be updated without hashing.

The second patch introduces:

static __always_inline bool unlikely_switch(struct jump_label_key *key);

instead of the old JUMP_LABEL(key, label) macro.

In this way, jump labels become really easy to use:

Quoted 8 lines

Define:

	struct jump_label_key jump_key;

Can be used as:

if (unlikely_switch(&jump_key))
do unlikely code

Ah, yes, that's an improvement!

Quoted 7 lines

I'm just wondering about the terminology here. Isn't that more a "branch" than a "switch" ?

I'm concerned about the fact that if we ever want to use the asm goto ability to jump to multiple targets (which is closer to a statically computed switch than a branch), we might want to reserve "switch" name for that rather than the branch.

Good point.

Quoted 10 lines

I wonder if the "if (unlikely_switch(&jump_key))" you propose above is the right thing to do. Why does the unlikely_ have to be included in the name ? Maybe there is a good reason for it, but it would be nice to have it spelled out. We might consider:

if (unlikely(static_branch(&jump_key)))
...

instead.

Hmm, I see your point here too.

Quoted 33 lines

For the switch statement, from the top of my head the idea would be to get something close to the following:

static __always_inline
int static_switch_{3,4,5,6...}(struct jump_label_key *key);

e.g.:

static __always_inline
int static_switch_3(struct jump_label_key *key)

{
	asm goto("1:"
		JUMP_LABEL_INITIAL_NOP
		".pushsection __switch_table_3,  \"a\" \n\t"
		_ASM_PTR "%c0, 1b, %l[l_1], %l[l_2] \n\t"
		".popsection \n\t"
		: :  "i" (key) : : l_1, l_2 );
	return 0;
l_1:
	return 1;
l_2:
	return 2;
}
switch(static_switch_3(&switch_key)) {
	case 0: .....
		break;
	case 1: .....
		break;
	case 2: .....
		break;
}
Quoted 3 lines

(I have not tried to give that to gcc 4.5.x to see how the resulting assembly looks like. It would be interesting to see if it handles this case well)

Something to think about later (when we need such a thing). But I do agree, perhaps calling it static_branch() instead, is a better idea.

-- Steve