Skip to contentSkip to conversation

tracing: Minor fixes

GIT PULL In review Steven Rostedt 3 participants 9 messages Oct 21, 2010 Tracing

lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

Ingo,

Please pull the latest tip/perf/core tree, which can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/perf/core

Steven Rostedt (2):
tracing: Prevent unloadable modules from using trace_bprintk()
tracing: Do not limit the size of the number of CPU buffers

----

 include/linux/kernel.h      |   21 +++++++++++++++++++--
 kernel/trace/trace.c        |    8 ++------
 kernel/trace/trace_printk.c |    2 ++
 3 files changed, 23 insertions(+), 8 deletions(-)
author maintainer
#1 lore raw

PATCH 1/2 tracing: Prevent unloadable modules from using trace_bprintk()

From: Steven Rostedt <[email protected]>

While debugging a module, I found that unloading the module and then reading the ring buffer can cause strange side effects, including a kernel crash.

This is due to the trace_bprintk(). The trace_bprintk() is a faster version of trace_printk(). The difference is that trace_bprintk() only copies the arguments and a pointer to the format string into the ring buffer.

If a module uses this function and is unloaded, the pointer back to the format string in the module is still around. If the trace file is read, then the pointer is referenced and this can cause a kernel oops.

The simple solution is to not let modules use trace_bprintk() and instead it will use the slower version of this.

When talking with Frederic Weisbecker about it, he suggested not to punish modules that can not be unloaded since they do not have this side effect. Modules that can not be unloaded can still use trace_bprintk(). We added a check for MODVERSIONS to be set to make sure that the module and kernel have the same options. If you run without MODVERSIONS set, and you load a module that was compiled differently, then that's just your tough luck.

Cc Frederic Weisbecker <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/kernel.h      |   21 +++++++++++++++++++--
 kernel/trace/trace_printk.c |    2 ++
 2 files changed, 21 insertions(+), 2 deletions(-)
include/linux/kernel.h +19 -2
index 2b0a35e..1003476 100644
@@ -538,6 +538,23 @@ do {									\
 		____trace_printk_check_format(fmt, ##args);		\
 } while (0)
 
+/*
+ * Module code must not use trace_bprintk, because if it is unloaded
+ * then we leave a pointer back to the module code inside
+ * the ring buffer, and then reading the ring buffer may cause a bug.
+ *
+ * We do allow for modules to use it if the kernel does not allow
+ * unloading of modules, and MODVERSIONS is set (to make sure kernel
+ * and module are the same). If you load modules without MODVERSIONS
+ * set, then you deserve what you get.
+ */
+#if defined(MODULE) &&							\
+	(defined(CONFIG_MODULE_UNLOAD) || !defined(CONFIG_MODVERSIONS))
+# define FORCE_TRACEPRINTK 1
+#else
+# define FORCE_TRACEPRINTK 0
+#endif
+
 /**
  * trace_printk - printf formatting in the ftrace buffer
  * @fmt: the printf format for printing
@@ -558,14 +575,14 @@ do {									\
 #define trace_printk(fmt, args...)					\
 do {									\
 	__trace_printk_check_format(fmt, ##args);			\
-	if (__builtin_constant_p(fmt)) {				\
+	if (__builtin_constant_p(fmt) && !FORCE_TRACEPRINTK) {		\
 		static const char *trace_printk_fmt			\
 		  __attribute__((section("__trace_printk_fmt"))) =	\
 			__builtin_constant_p(fmt) ? fmt : NULL;		\
 									\
 		__trace_bprintk(_THIS_IP_, trace_printk_fmt, ##args);	\
 	} else								\
-		__trace_printk(_THIS_IP_, fmt, ##args);		\
+		__trace_printk(_THIS_IP_, fmt, ##args);			\
 } while (0)
 
 extern int
kernel/trace/trace_printk.c +2 -0
index 2547d88..c4a5db6 100644
@@ -115,7 +115,9 @@ int __trace_bprintk(unsigned long ip, const char *fmt, ...)
 	va_end(ap);
 	return ret;
 }
+#if !FORCE_TRACEPRINTK
 EXPORT_SYMBOL_GPL(__trace_bprintk);
+#endif
 
 int __ftrace_vbprintk(unsigned long ip, const char *fmt, va_list ap)
  {
Signature · 1 line
1.7.1
author maintainer
#8 lore raw

PATCH 2/2 tracing: Do not limit the size of the number of CPU buffers

In reply to Steven Rostedt

From: Steven Rostedt <[email protected]>

The tracing per_cpu buffers were limited to 999 CPUs for a mear savings in stack space of a char array. Up the array to 30 characters which is more than enough to hold a 64 bit number.

---
 kernel/trace/trace.c |    8 ++------
 1 files changed, 2 insertions(+), 6 deletions(-)
kernel/trace/trace.c +2 -6
index 001bcd2..82d9b81 100644
@@ -3996,13 +3996,9 @@ static void tracing_init_debugfs_percpu(long cpu)
 {
 	struct dentry *d_percpu = tracing_dentry_percpu();
 	struct dentry *d_cpu;
-	/* strlen(cpu) + MAX(log10(cpu)) + '\0' */
-	char cpu_dir[7];
+	char cpu_dir[30]; /* 30 characters should be more than enough */
 
-	if (cpu > 999 || cpu < 0)
-		return;
-
-	sprintf(cpu_dir, "cpu%ld", cpu);
+	snprintf(cpu_dir, 30, "cpu%ld", cpu);
 	d_cpu = debugfs_create_dir(cpu_dir, d_percpu);
 	if (!d_cpu) {
 		pr_warning("Could not create debugfs '%s' entry\n", cpu_dir);
Signature · 1 line
1.7.1
#2 lore raw

Re: [PATCH 1/2] tracing: Prevent unloadable modules from using trace_bprintk()

In reply to Steven Rostedt
Quoted 44 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

While debugging a module, I found that unloading the module and then reading the ring buffer can cause strange side effects, including a kernel crash.

This is due to the trace_bprintk(). The trace_bprintk() is a faster version of trace_printk(). The difference is that trace_bprintk() only copies the arguments and a pointer to the format string into the ring buffer.

If a module uses this function and is unloaded, the pointer back to the format string in the module is still around. If the trace file is read, then the pointer is referenced and this can cause a kernel oops.

The simple solution is to not let modules use trace_bprintk() and instead it will use the slower version of this.

When talking with Frederic Weisbecker about it, he suggested not to punish modules that can not be unloaded since they do not have this side effect. Modules that can not be unloaded can still use trace_bprintk(). We added a check for MODVERSIONS to be set to make sure that the module and kernel have the same options. If you run without MODVERSIONS set, and you load a module that was compiled differently, then that's just your tough luck.

Cc Frederic Weisbecker <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/kernel.h      |   21 +++++++++++++++++++--
 kernel/trace/trace_printk.c |    2 ++
 2 files changed, 21 insertions(+), 2 deletions(-)
include/linux/kernel.h +1 -0
index 2b0a35e..1003476 100644
@@ -538,6 +538,23 @@ do {									\
 		____trace_printk_check_format(fmt, ##args);		\
 } while (0)
 
+/*

include/linux/kernel.h do { \ · line 551

Quoted 10 lines
include/linux/kernel.h +10 -0
+ * Module code must not use trace_bprintk, because if it is unloaded
+ * then we leave a pointer back to the module code inside
+ * the ring buffer, and then reading the ring buffer may cause a bug.
+ *
+ * We do allow for modules to use it if the kernel does not allow
+ * unloading of modules, and MODVERSIONS is set (to make sure kernel
+ * and module are the same). If you load modules without MODVERSIONS
+ * set, then you deserve what you get.
+ */
+#if defined(MODULE) &&							\

Did you mean CONFIG_MODULE may be?

Thanks.

author maintainer
#3 lore raw

Re: [PATCH 1/2] tracing: Prevent unloadable modules from using trace_bprintk()

Quoted 7 lines · Frederic Weisbecker
Quoted 7 lines

+/*
+ * Module code must not use trace_bprintk, because if it is unloaded
+ * then we leave a pointer back to the module code inside
+ * the ring buffer, and then reading the ring buffer may cause a bug.
+ *
+ * We do allow for modules to use it if the kernel does not allow
+ * unloading of modules, and MODVERSIONS is set (to make sure kernel

Quoted 8 lines
Quoted 4 lines

+ * and module are the same). If you load modules without MODVERSIONS
+ * set, then you deserve what you get.
+ */
+#if defined(MODULE) && \

Did you mean CONFIG_MODULE may be?

Nope, then the a kernel that allows modules wont use it. I do mean "MODULE", as in defined by the Makefile:

KBUILD_AFLAGS_MODULE := -DMODULE
KBUILD_CFLAGS_MODULE := -DMODULE

-- Steve

#4 lore raw

Re: [PATCH 1/2] tracing: Prevent unloadable modules from using trace_bprintk()

Quoted 18 lines · Steven Rostedt
Quoted 7 lines · Frederic Weisbecker
Quoted 7 lines

+/*
+ * Module code must not use trace_bprintk, because if it is unloaded
+ * then we leave a pointer back to the module code inside
+ * the ring buffer, and then reading the ring buffer may cause a bug.
+ *
+ * We do allow for modules to use it if the kernel does not allow
+ * unloading of modules, and MODVERSIONS is set (to make sure kernel

Quoted 8 lines
Quoted 4 lines

+ * and module are the same). If you load modules without MODVERSIONS
+ * set, then you deserve what you get.
+ */
+#if defined(MODULE) && \

Did you mean CONFIG_MODULE may be?

Quoted 5 lines

Nope, then the a kernel that allows modules wont use it. I do mean "MODULE", as in defined by the Makefile:

KBUILD_AFLAGS_MODULE := -DMODULE
KBUILD_CFLAGS_MODULE := -DMODULE

Ah ok.

Thanks.

#5 lore raw

Re: [PATCH 1/2] tracing: Prevent unloadable modules from using trace_bprintk()

In reply to Steven Rostedt
Quoted 45 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

While debugging a module, I found that unloading the module and then reading the ring buffer can cause strange side effects, including a kernel crash.

This is due to the trace_bprintk(). The trace_bprintk() is a faster version of trace_printk(). The difference is that trace_bprintk() only copies the arguments and a pointer to the format string into the ring buffer.

If a module uses this function and is unloaded, the pointer back to the format string in the module is still around. If the trace file is read, then the pointer is referenced and this can cause a kernel oops.

The simple solution is to not let modules use trace_bprintk() and instead it will use the slower version of this.

When talking with Frederic Weisbecker about it, he suggested not to punish modules that can not be unloaded since they do not have this side effect. Modules that can not be unloaded can still use trace_bprintk(). We added a check for MODVERSIONS to be set to make sure that the module and kernel have the same options. If you run without MODVERSIONS set, and you load a module that was compiled differently, then that's just your tough luck.

Cc Frederic Weisbecker <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/kernel.h      |   21 +++++++++++++++++++--
 kernel/trace/trace_printk.c |    2 ++
 2 files changed, 21 insertions(+), 2 deletions(-)
include/linux/kernel.h +2 -0
index 2b0a35e..1003476 100644
@@ -538,6 +538,23 @@ do {									\
 		____trace_printk_check_format(fmt, ##args);		\
 } while (0)
 
+/*
+ * Module code must not use trace_bprintk, because if it is unloaded

include/linux/kernel.h do { \ · line 552

Quoted 10 lines
include/linux/kernel.h +10 -0
+ * then we leave a pointer back to the module code inside
+ * the ring buffer, and then reading the ring buffer may cause a bug.
+ *
+ * We do allow for modules to use it if the kernel does not allow
+ * unloading of modules, and MODVERSIONS is set (to make sure kernel
+ * and module are the same). If you load modules without MODVERSIONS
+ * set, then you deserve what you get.
+ */
+#if defined(MODULE) &&							\
+	(defined(CONFIG_MODULE_UNLOAD) || !defined(CONFIG_MODVERSIONS))

Erm. Ignore checkpatch when the solution is to mess up the code ...

Quoted 30 lines
include/linux/kernel.h +7 -2
+# define FORCE_TRACEPRINTK 1
+#else
+# define FORCE_TRACEPRINTK 0
+#endif
+
 /**
  * trace_printk - printf formatting in the ftrace buffer
  * @fmt: the printf format for printing
@@ -558,14 +575,14 @@ do {									\
 #define trace_printk(fmt, args...)					\
 do {									\
 	__trace_printk_check_format(fmt, ##args);			\
-	if (__builtin_constant_p(fmt)) {				\
+	if (__builtin_constant_p(fmt) && !FORCE_TRACEPRINTK) {		\
 		static const char *trace_printk_fmt			\
 		  __attribute__((section("__trace_printk_fmt"))) =	\
 			__builtin_constant_p(fmt) ? fmt : NULL;		\
 									\
 		__trace_bprintk(_THIS_IP_, trace_printk_fmt, ##args);	\
 	} else								\
-		__trace_printk(_THIS_IP_, fmt, ##args);		\
+		__trace_printk(_THIS_IP_, fmt, ##args);			\
 } while (0)
 
 extern int
kernel/trace/trace_printk.c +0 -0
index 2547d88..c4a5db6 100644

kernel/trace/trace_printk.c int __trace_bprintk(unsigned long ip, const char *fmt, ...) · line 120

Quoted 6 lines
kernel/trace/trace_printk.c +2 -0
@@ -115,7 +115,9 @@ int __trace_bprintk(unsigned long ip, const char *fmt, ...)
 	va_end(ap);
 	return ret;
 }
+#if !FORCE_TRACEPRINTK
 EXPORT_SYMBOL_GPL(__trace_bprintk);
+#endif

Looks quite ugly all around. Cannot suggest anything better though straight away - so please Cc: it more widely and get an ack from the module folks: Rusty, Linus, akpm.

Thanks,

Ingo

author maintainer
#6 lore raw

Re: [PATCH 1/2] tracing: Prevent unloadable modules from using trace_bprintk()

Quoted 45 lines · Ingo Molnar
Quoted 43 lines · Steven Rostedt

From: Steven Rostedt <[email protected]>

While debugging a module, I found that unloading the module and then reading the ring buffer can cause strange side effects, including a kernel crash.

This is due to the trace_bprintk(). The trace_bprintk() is a faster version of trace_printk(). The difference is that trace_bprintk() only copies the arguments and a pointer to the format string into the ring buffer.

If a module uses this function and is unloaded, the pointer back to the format string in the module is still around. If the trace file is read, then the pointer is referenced and this can cause a kernel oops.

The simple solution is to not let modules use trace_bprintk() and instead it will use the slower version of this.

When talking with Frederic Weisbecker about it, he suggested not to punish modules that can not be unloaded since they do not have this side effect. Modules that can not be unloaded can still use trace_bprintk(). We added a check for MODVERSIONS to be set to make sure that the module and kernel have the same options. If you run without MODVERSIONS set, and you load a module that was compiled differently, then that's just your tough luck.

Cc Frederic Weisbecker <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/kernel.h      |   21 +++++++++++++++++++--
 kernel/trace/trace_printk.c |    2 ++
 2 files changed, 21 insertions(+), 2 deletions(-)
include/linux/kernel.h +0 -0
index 2b0a35e..1003476 100644
@@ -538,6 +538,23 @@ do {									\
 		____trace_printk_check_format(fmt, ##args);		\
 } while (0)
 
Quoted 14 lines

include/linux/kernel.h do { \ · line 552

Quoted 12 lines
include/linux/kernel.h +12 -0
+/*
+ * Module code must not use trace_bprintk, because if it is unloaded
+ * then we leave a pointer back to the module code inside
+ * the ring buffer, and then reading the ring buffer may cause a bug.
+ *
+ * We do allow for modules to use it if the kernel does not allow
+ * unloading of modules, and MODVERSIONS is set (to make sure kernel
+ * and module are the same). If you load modules without MODVERSIONS
+ * set, then you deserve what you get.
+ */
+#if defined(MODULE) &&							\
+	(defined(CONFIG_MODULE_UNLOAD) || !defined(CONFIG_MODVERSIONS))

Erm. Ignore checkpatch when the solution is to mess up the code ...

Heh, OK.

Quoted 25 lines
Quoted 25 lines
include/linux/kernel.h +7 -2
+# define FORCE_TRACEPRINTK 1
+#else
+# define FORCE_TRACEPRINTK 0
+#endif
+
 /**
  * trace_printk - printf formatting in the ftrace buffer
  * @fmt: the printf format for printing
@@ -558,14 +575,14 @@ do {									\
 #define trace_printk(fmt, args...)					\
 do {									\
 	__trace_printk_check_format(fmt, ##args);			\
-	if (__builtin_constant_p(fmt)) {				\
+	if (__builtin_constant_p(fmt) && !FORCE_TRACEPRINTK) {		\
 		static const char *trace_printk_fmt			\
 		  __attribute__((section("__trace_printk_fmt"))) =	\
 			__builtin_constant_p(fmt) ? fmt : NULL;		\
 									\
 		__trace_bprintk(_THIS_IP_, trace_printk_fmt, ##args);	\
 	} else								\
-		__trace_printk(_THIS_IP_, fmt, ##args);		\
+		__trace_printk(_THIS_IP_, fmt, ##args);			\
 } while (0)
 
Quoted 16 lines

kernel/trace/trace_printk.c int __trace_bprintk(unsigned long ip, const char *fmt, ...) · line 120

Quoted 12 lines
include/linux/kernel.h +0 -0
 extern int
kernel/trace/trace_printk.c +2 -0
index 2547d88..c4a5db6 100644
@@ -115,7 +115,9 @@ int __trace_bprintk(unsigned long ip, const char *fmt, ...)
 	va_end(ap);
 	return ret;
 }
+#if !FORCE_TRACEPRINTK
 EXPORT_SYMBOL_GPL(__trace_bprintk);
+#endif

Looks quite ugly all around. Cannot suggest anything better though straight away - so please Cc: it more widely and get an ack from the module folks: Rusty, Linus, akpm.

Just added them.

One thing that bothers me about this patch is the negative of FORCE_TRACEPRINTK. I think it would be a nicer patch if I changed that to a positive TRACE_BPRINTK_ALLOWED, then it would be:

+ if (__builtin_constant_p(fmt) && TRACE_BPRINTK_ALLOWED) {

+#if TRACE_BPRINTK_ALLOWED
EXPORT_SYMBOL_GPL(__trace_bprintk);
+#endif

The patch is still a bit ugly, but this change makes it a little nicer to read.

Thanks,

-- Steve

#7 lore raw

Re: [PATCH 1/2] tracing: Prevent unloadable modules from using trace_bprintk()

Quoted 7 lines · Steven Rostedt
Quoted 5 lines
Quoted 1 line

+#endif

Looks quite ugly all around. Cannot suggest anything better though straight away
- so please Cc: it more widely and get an ack from the module folks: Rusty, Linus, akpm.

Just added them.

Below is the full patch again.

Ingo

----- Forwarded message from Steven Rostedt <[email protected]> -----

Date: Wed, 20 Oct 2010 22:42:34 -0400
From: Steven Rostedt <[email protected]>
To: [email protected]
Cc: Ingo Molnar <[email protected]>, Andrew Morton <[email protected]>,
Frederic Weisbecker <[email protected]>,
Thomas Gleixner <[email protected]>
Subject: [PATCH 1/2] tracing: Prevent unloadable modules from using trace_bprintk()

From: Steven Rostedt <[email protected]>

While debugging a module, I found that unloading the module and then reading the ring buffer can cause strange side effects, including a kernel crash.

This is due to the trace_bprintk(). The trace_bprintk() is a faster version of trace_printk(). The difference is that trace_bprintk() only copies the arguments and a pointer to the format string into the ring buffer.

If a module uses this function and is unloaded, the pointer back to the format string in the module is still around. If the trace file is read, then the pointer is referenced and this can cause a kernel oops.

The simple solution is to not let modules use trace_bprintk() and instead it will use the slower version of this.

When talking with Frederic Weisbecker about it, he suggested not to punish modules that can not be unloaded since they do not have this side effect. Modules that can not be unloaded can still use trace_bprintk(). We added a check for MODVERSIONS to be set to make sure that the module and kernel have the same options. If you run without MODVERSIONS set, and you load a module that was compiled differently, then that's just your tough luck.

Cc Frederic Weisbecker <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/kernel.h      |   21 +++++++++++++++++++--
 kernel/trace/trace_printk.c |    2 ++
 2 files changed, 21 insertions(+), 2 deletions(-)
include/linux/kernel.h +19 -2
index 2b0a35e..1003476 100644
@@ -538,6 +538,23 @@ do {									\
 		____trace_printk_check_format(fmt, ##args);		\
 } while (0)
 
+/*
+ * Module code must not use trace_bprintk, because if it is unloaded
+ * then we leave a pointer back to the module code inside
+ * the ring buffer, and then reading the ring buffer may cause a bug.
+ *
+ * We do allow for modules to use it if the kernel does not allow
+ * unloading of modules, and MODVERSIONS is set (to make sure kernel
+ * and module are the same). If you load modules without MODVERSIONS
+ * set, then you deserve what you get.
+ */
+#if defined(MODULE) &&							\
+	(defined(CONFIG_MODULE_UNLOAD) || !defined(CONFIG_MODVERSIONS))
+# define FORCE_TRACEPRINTK 1
+#else
+# define FORCE_TRACEPRINTK 0
+#endif
+
 /**
  * trace_printk - printf formatting in the ftrace buffer
  * @fmt: the printf format for printing
@@ -558,14 +575,14 @@ do {									\
 #define trace_printk(fmt, args...)					\
 do {									\
 	__trace_printk_check_format(fmt, ##args);			\
-	if (__builtin_constant_p(fmt)) {				\
+	if (__builtin_constant_p(fmt) && !FORCE_TRACEPRINTK) {		\
 		static const char *trace_printk_fmt			\
 		  __attribute__((section("__trace_printk_fmt"))) =	\
 			__builtin_constant_p(fmt) ? fmt : NULL;		\
 									\
 		__trace_bprintk(_THIS_IP_, trace_printk_fmt, ##args);	\
 	} else								\
-		__trace_printk(_THIS_IP_, fmt, ##args);		\
+		__trace_printk(_THIS_IP_, fmt, ##args);			\
 } while (0)
 
 extern int
kernel/trace/trace_printk.c +2 -0
index 2547d88..c4a5db6 100644
@@ -115,7 +115,9 @@ int __trace_bprintk(unsigned long ip, const char *fmt, ...)
 	va_end(ap);
 	return ret;
 }
+#if !FORCE_TRACEPRINTK
 EXPORT_SYMBOL_GPL(__trace_bprintk);
+#endif
 
 int __ftrace_vbprintk(unsigned long ip, const char *fmt, va_list ap)
  {