Skip to contentSkip to conversation

tracing, printk: Force no hashing when trace_printk() is used

RFC PATCH Superseded Steven Rostedt 4 participants 9 messages Apr 3, 2018 – Apr 4, 2018

Subsystems (4)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

RFC PATCH 1/1 tracing, printk: Force no hashing when trace_printk() is used

While debugging an issue I needed to see if the pointers were being processed correctly with trace_printk() and after using "%p" and triggering my bug and trace output, I was disappointed that all my pointers were random garbage and didn't produce anything useful for me. I had to rewrite all the trace_printk()s to use "%lx" instead.

As trace_printk() is not to be used for anything but debugging, and this is enforced by printing in the dmesg:

**********************************************************

 **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
 **                                                      **
 ** trace_printk() being used. Allocating extra memory.  **
 **                                                      **
 ** This means that this is a DEBUG kernel and it is     **
 ** unsafe for production use.                           **
 **                                                      **
 ** If you see this message and you are not debugging    **
 ** the kernel, report this immediately to your vendor!  **
 **                                                      **
 **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **

**********************************************************

on boot up if trace_printk() is used (or when a module is loaded that uses trace_printk()), we can safely assume that the use of trace_printk() is not going to be accidentally added to production code (and if it is, they should be whacked with an overcooked spaghetti noodle).

A new bool is added called "trace_debug" and if it is set, then %p will not be hashed.

Both trace_debug is set and kptr_restrict is set to zero in the same code that produces the above banner. This will allow trace_printk() to not be affected by security code, as trace_printk() should never be run on a machine that needs security of this kind.

---

include/linux/printk.h +1 -0
index e9b603ee9953..7ef6c31d874a 100644
@@ -278,6 +278,7 @@ static inline void printk_safe_flush_on_panic(void)
 #endif
 
 extern int kptr_restrict;
+extern bool trace_debug;
 
 extern asmlinkage void dump_stack(void) __cold;
 
kernel/trace/trace.c +4 -0
index 0f47e653ffd8..cb58cd4ee3a7 100644
@@ -2864,6 +2864,10 @@ void trace_printk_init_buffers(void)
 
 	buffers_allocated = 1;
 
+	/* This is a debug kernel, allow pointers to be shown */
+	trace_debug = true;
+	kptr_restrict = 0;
+
 	/*
 	 * trace_printk_init_buffers() can be called by modules.
 	 * If that happens, then we need to start cmdline recording
lib/vsprintf.c +5 -0
index 89f8a4a4b770..425644ceedcb 100644
@@ -1345,6 +1345,7 @@ char *uuid_string(char *buf, char *end, const u8 *addr,
 }
 
 int kptr_restrict __read_mostly;
+bool trace_debug __read_mostly;
 
 static noinline_for_stack
 char *restricted_pointer(char *buf, char *end, const void *ptr,
@@ -1962,6 +1963,10 @@ char *pointer(const char *fmt, char *buf, char *end, void *ptr,
 		return pointer_string(buf, end, ptr, spec);
 	}
 
+	/* When the kernel is in debugging mode, show all pointers */
+	if (trace_debug)
+		return restricted_pointer(buf, end, ptr, spec);
+
 	/* default is to _not_ leak addresses, hash before printing */
 	return ptr_to_id(buf, end, ptr, spec);
 }
#1 lore raw
Quoted 4 lines · Steven Rostedt

Both trace_debug is set and kptr_restrict is set to zero in the same code that produces the above banner. This will allow trace_printk() to not be affected by security code, as trace_printk() should never be run on a machine that needs security of this kind.

While I think it'd be nice to have a boot-time knob for this (a debate that was unsuccessful in earlier threads), I remain skeptical of having a _runtime_ knob for this, as then it becomes a target (and yes, there are plenty of targets, but why add another).

If this was __ro_after_init, maybe that'd be nicer. CONFIG_TRACING=y is used everywhere, so this is really just the whole knob debate over again. Instead, I've been following Linus's distillation of %p usage in the kernel:

http://lkml.kernel.org/r/CA+55aFwQEd_d40g4mUCSsVRZzrFPUJt74vc6PPpb675hYNXcKw@mail.gmail.com

-Kees

Signature · 2 lines
Kees Cook
Pixel Security
author maintainer
#2 lore raw
Quoted 6 lines · On Tue, 3 Apr 2018 13:07:58 -0700 Kees Cook <[email protected]> wrote:
Quoted 4 lines · Steven Rostedt

Both trace_debug is set and kptr_restrict is set to zero in the same code that produces the above banner. This will allow trace_printk() to not be affected by security code, as trace_printk() should never be run on a machine that needs security of this kind.

Quoted 6 lines

While I think it'd be nice to have a boot-time knob for this (a debate that was unsuccessful in earlier threads), I remain skeptical of having a _runtime_ knob for this, as then it becomes a target (and yes, there are plenty of targets, but why add another).

If this was __ro_after_init, maybe that'd be nicer. CONFIG_TRACING=y

Well, then of course this would need a check to keep modules from setting it. But I think I know of a nice alternative.

Quoted 5 lines

is used everywhere, so this is really just the whole knob debate over again. Instead, I've been following Linus's distillation of %p usage in the kernel:

http://lkml.kernel.org/r/CA+55aFwQEd_d40g4mUCSsVRZzrFPUJt74vc6PPpb675hYNXcKw@mail.gmail.com

Remember, this isn't a printk() that hangs around for production. I was debugging code that modified pointers, and I wanted to make sure that the pointer arithmetic was correct (it wasn't), and randomizing the output made my prints useless.

If you are concerned about attack surface, I could make it a bit more difficult to tweak by malicious software. What about the patch below? It would be much more difficult to modify this knob from an attack vector.

-- Steve

include/linux/printk.h +1 -0
index e9b603ee9953..b624493b3991 100644
@@ -278,6 +278,7 @@ static inline void printk_safe_flush_on_panic(void)
 #endif
 
 extern int kptr_restrict;
+extern struct static_key trace_debug;
 
 extern asmlinkage void dump_stack(void) __cold;
 
kernel/trace/trace.c +4 -0
index 0f47e653ffd8..6c151d00848b 100644
@@ -2864,6 +2864,10 @@ void trace_printk_init_buffers(void)
 
 	buffers_allocated = 1;
 
+	/* This is a debug kernel, allow pointers to be shown */
+	static_key_enable(&trace_debug);
+	kptr_restrict = 0;
+
 	/*
 	 * trace_printk_init_buffers() can be called by modules.
 	 * If that happens, then we need to start cmdline recording
lib/vsprintf.c +5 -0
index 89f8a4a4b770..c3d8eafecb39 100644
@@ -1345,6 +1345,7 @@ char *uuid_string(char *buf, char *end, const u8 *addr,
 }
 
 int kptr_restrict __read_mostly;
+struct static_key trace_debug = STATIC_KEY_INIT_FALSE;
 
 static noinline_for_stack
 char *restricted_pointer(char *buf, char *end, const void *ptr,
@@ -1962,6 +1963,10 @@ char *pointer(const char *fmt, char *buf, char *end, void *ptr,
 		return pointer_string(buf, end, ptr, spec);
 	}
 
+	/* When the kernel is in debugging mode, show all pointers */
+	if (static_key_false(&trace_debug))
+		return restricted_pointer(buf, end, ptr, spec);
+
 	/* default is to _not_ leak addresses, hash before printing */
 	return ptr_to_id(buf, end, ptr, spec);
 }
#3 lore raw
Quoted 78 lines · Steven Rostedt
Quoted 6 lines · On Tue, 3 Apr 2018 13:07:58 -0700 Kees Cook <[email protected]> wrote:
Quoted 4 lines · Steven Rostedt

Both trace_debug is set and kptr_restrict is set to zero in the same code that produces the above banner. This will allow trace_printk() to not be affected by security code, as trace_printk() should never be run on a machine that needs security of this kind.

Quoted 6 lines

While I think it'd be nice to have a boot-time knob for this (a debate that was unsuccessful in earlier threads), I remain skeptical of having a _runtime_ knob for this, as then it becomes a target (and yes, there are plenty of targets, but why add another).

If this was __ro_after_init, maybe that'd be nicer. CONFIG_TRACING=y

Well, then of course this would need a check to keep modules from setting it. But I think I know of a nice alternative.

Quoted 5 lines

is used everywhere, so this is really just the whole knob debate over again. Instead, I've been following Linus's distillation of %p usage in the kernel:

http://lkml.kernel.org/r/CA+55aFwQEd_d40g4mUCSsVRZzrFPUJt74vc6PPpb675hYNXcKw@mail.gmail.com

Remember, this isn't a printk() that hangs around for production. I was debugging code that modified pointers, and I wanted to make sure that the pointer arithmetic was correct (it wasn't), and randomizing the output made my prints useless.

If you are concerned about attack surface, I could make it a bit more difficult to tweak by malicious software. What about the patch below? It would be much more difficult to modify this knob from an attack vector.

-- Steve

include/linux/printk.h +1 -0
index e9b603ee9953..b624493b3991 100644
@@ -278,6 +278,7 @@ static inline void printk_safe_flush_on_panic(void)
 #endif
 
 extern int kptr_restrict;
+extern struct static_key trace_debug;
 
 extern asmlinkage void dump_stack(void) __cold;
 
kernel/trace/trace.c +4 -0
index 0f47e653ffd8..6c151d00848b 100644
@@ -2864,6 +2864,10 @@ void trace_printk_init_buffers(void)
 
 	buffers_allocated = 1;
 
+	/* This is a debug kernel, allow pointers to be shown */
+	static_key_enable(&trace_debug);
+	kptr_restrict = 0;
+
 	/*
 	 * trace_printk_init_buffers() can be called by modules.
 	 * If that happens, then we need to start cmdline recording
lib/vsprintf.c +1 -0
index 89f8a4a4b770..c3d8eafecb39 100644
@@ -1345,6 +1345,7 @@ char *uuid_string(char *buf, char *end, const u8 *addr,
 }
 
 int kptr_restrict __read_mostly;
+struct static_key trace_debug = STATIC_KEY_INIT_FALSE;
 
 static noinline_for_stack
 char *restricted_pointer(char *buf, char *end, const void *ptr,

lib/vsprintf.c char *pointer(const char *fmt, char *buf, char *end, void *ptr, · line 1972

Quoted 10 lines
lib/vsprintf.c +4 -0
@@ -1962,6 +1963,10 @@ char *pointer(const char *fmt, char *buf, char *end, void *ptr,
 		return pointer_string(buf, end, ptr, spec);
 	}
 
+	/* When the kernel is in debugging mode, show all pointers */
+	if (static_key_false(&trace_debug))
+		return restricted_pointer(buf, end, ptr, spec);
+
 	/* default is to _not_ leak addresses, hash before printing */
 	return ptr_to_id(buf, end, ptr, spec);
 }

This uses the deprecated API Steve (I only know because I went to read Documentation/static-keys.txt after seeing this patch).

Hope this helps,
Tobin.

author maintainer
#4 lore raw
Quoted 6 lines · On Wed, 4 Apr 2018 07:43:49 +1000 "Tobin C. Harding" <[email protected]> wrote:
Quoted 6 lines

static noinline_for_stack
char *restricted_pointer(char *buf, char *end, const void *ptr,

lib/vsprintf.c +0 -0
@@ -1962,6 +1963,10 @@ char *pointer(const char *fmt, char *buf, char *end, void *ptr,
 		return pointer_string(buf, end, ptr, spec);
 	}
 
Quoted 10 lines

lib/vsprintf.c char *pointer(const char *fmt, char *buf, char *end, void *ptr, · line 1972

Quoted 7 lines
lib/vsprintf.c +4 -0
+	/* When the kernel is in debugging mode, show all pointers */
+	if (static_key_false(&trace_debug))
+		return restricted_pointer(buf, end, ptr, spec);
+
 	/* default is to _not_ leak addresses, hash before printing */
 	return ptr_to_id(buf, end, ptr, spec);
 }  

This uses the deprecated API Steve (I only know because I went to read Documentation/static-keys.txt after seeing this patch).

Hmm, I've been involved with static keys since it was introduced. I simply copied the code for the original use case (tracepoints). I forgot that we are changing the name (this has been an ongoing bikeshed for some time, I can't keep up. Better than the original suggestion "really_unlikely()" ;)

I need to update tracepoints too.

-- Steve

#5 lore raw
In reply to Steven Rostedt
Quoted 4 lines · Steven Rostedt

If you are concerned about attack surface, I could make it a bit more difficult to tweak by malicious software. What about the patch below? It would be much more difficult to modify this knob from an attack vector.

Not if you build using clang, because that doesn't support asm-goto and thus falls back to a simple runtime variable, which is exactly what Kees didn't want.

author maintainer
#6 lore raw
Quoted 9 lines · On Wed, 4 Apr 2018 09:49:27 +0200 Peter Zijlstra <[email protected]> wrote:
Quoted 4 lines · Steven Rostedt

If you are concerned about attack surface, I could make it a bit more difficult to tweak by malicious software. What about the patch below? It would be much more difficult to modify this knob from an attack vector.

Not if you build using clang, because that doesn't support asm-goto and thus falls back to a simple runtime variable, which is exactly what Kees didn't want.

Fix clang ;-)

-- Steve

#7 lore raw
In reply to Peter Zijlstra
Quoted 9 lines · Peter Zijlstra
Quoted 4 lines · Steven Rostedt

If you are concerned about attack surface, I could make it a bit more difficult to tweak by malicious software. What about the patch below? It would be much more difficult to modify this knob from an attack vector.

Not if you build using clang, because that doesn't support asm-goto and thus falls back to a simple runtime variable, which is exactly what Kees didn't want.

Nah, Clang will get asm-goto soon. I'm not worried about that. Besides the "yay new target issue", I was concerned about Linus yelling about seeing the knob added he specifically said he didn't want. :P

-Kees

Signature · 2 lines
Kees Cook
Pixel Security
author maintainer
#8 lore raw
Quoted 7 lines · On Wed, 4 Apr 2018 09:27:10 -0700 Kees Cook <[email protected]> wrote:
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Steven Rostedt

If you are concerned about attack surface, I could make it a bit more difficult to tweak by malicious software. What about the patch below? It would be much more difficult to modify this knob from an attack vector.

Quoted 8 lines
Quoted 3 lines

Not if you build using clang, because that doesn't support asm-goto and thus falls back to a simple runtime variable, which is exactly what Kees didn't want.

Nah, Clang will get asm-goto soon. I'm not worried about that. Besides the "yay new target issue", I was concerned about Linus yelling about seeing the knob added he specifically said he didn't want. :P

This isn't really a knob. It can only get enabled when trace_printk() is added, which requires a change to the kernel source code. It's just an easy way on doing it yourself and recompiling.

No userspace can enable it, with the exception of adding a kernel module that has a trace_printk() in it. A kernel command line option wont even enable it.

-- Steve