Skip to contentSkip to conversation

tracing: three more updates

GIT PULL In review Steven Rostedt 2 participants 7 messages Dec 5, 2019 – Jan 10, 2020

Subsystems (3)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

Linus,

Two fixes and one patch that was missed:

Fixes:

- Missing __print_hex_dump undef for processing new function in trace events
- Stop WARN_ON messages when lockdown disables tracing on boot up

Enhancement:

- Debug option to inject trace events from userspace (for rasdaemon)

The enhancement has its own config option and is non invasive. It's been discussed for sever months and should have been added to my original push, but I never pulled it into my queue.

Please pull the latest trace-v5.5-2 tree, which can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git
trace-v5.5-2

Tag SHA1: 075c954bfa42bcd2dad254f1fe6fcd81197512de Head SHA1: a356646a56857c2e5ad875beec734d7145ecd49a

Cong Wang (1):
tracing: Introduce trace event injection

Piotr Maziarz (1):
tracing: Fix __print_hex_dump scope

Steven Rostedt (VMware) (1):
tracing: Do not create directories if lockdown is in affect

----

 include/trace/trace_events.h       |   1 +
 kernel/trace/Kconfig               |   9 +
 kernel/trace/Makefile              |   1 +
 kernel/trace/ring_buffer.c         |   6 +
 kernel/trace/trace.c               |  17 ++
 kernel/trace/trace.h               |   1 +
 kernel/trace/trace_events.c        |   6 +
 kernel/trace/trace_events_inject.c | 331 +++++++++++++++++++++++++++++++++++++
 8 files changed, 372 insertions(+)
 create mode 100644 kernel/trace/trace_events_inject.c
author maintainer
#1 lore raw

PATCH 1/3 tracing: Fix __print_hex_dump scope

From: Piotr Maziarz <[email protected]>

undef is needed for parsing __print_hex_dump in traceevent lib.

---
 include/trace/trace_events.h | 1 +
 1 file changed, 1 insertion(+)
include/trace/trace_events.h +1 -0
index 7089760d4c7a..472b33d23a10 100644
@@ -757,6 +757,7 @@ static inline void ftrace_test_probe_##call(void)			\
 #undef __get_str
 #undef __get_bitmask
 #undef __print_array
+#undef __print_hex_dump
 
 #undef TP_printk
 #define TP_printk(fmt, args...) "\"" fmt "\", "  __stringify(args)
Signature · 1 line
2.24.0
author maintainer
#2 lore raw

PATCH 2/3 tracing: Introduce trace event injection

In reply to Steven Rostedt

From: Cong Wang <[email protected]>

We have been trying to use rasdaemon to monitor hardware errors like correctable memory errors. rasdaemon uses trace events to monitor various hardware errors. In order to test it, we have to inject some hardware errors, unfortunately not all of them provide error injections. MCE does provide a way to inject MCE errors, but errors like PCI error and devlink error don't, it is not easy to add error injection to each of them. Instead, it is relatively easier to just allow users to inject trace events in a generic way so that all trace events can be injected.

This patch introduces trace event injection, where a new 'inject' is added to each tracepoint directory. Users could write into this file with key=value pairs to specify the value of each fields of the trace event, all unspecified fields are set to zero values by default.

For example, for the net/net_dev_queue tracepoint, we can inject:

INJECT=/sys/kernel/debug/tracing/events/net/net_dev_queue/inject echo "" > $INJECT
echo "name='test'" > $INJECT
echo "name='test' len=1024" > $INJECT
cat /sys/kernel/debug/tracing/trace
...

   <...>-614   [000] ....    36.571483: net_dev_queue: dev= skbaddr=00000000fbf338c2 len=0
   <...>-614   [001] ....   136.588252: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=0
   <...>-614   [001] .N..   208.431878: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=1024

Triggers could be triggered as usual too:

echo "stacktrace if len == 1025" > /sys/kernel/debug/tracing/events/net/net_dev_queue/trigger echo "len=1025" > $INJECT
cat /sys/kernel/debug/tracing/trace
...

      bash-614   [000] ....    36.571483: net_dev_queue: dev= skbaddr=00000000fbf338c2 len=0
      bash-614   [001] ....   136.588252: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=0
      bash-614   [001] .N..   208.431878: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=1024
      bash-614   [001] .N.1   284.236349: <stack trace>

=> event_inject_write
=> vfs_write
=> ksys_write
=> do_syscall_64
=> entry_SYSCALL_64_after_hwframe

The only thing that can't be injected is string pointers as they require constant string pointers, this can't be done at run time.

---
 kernel/trace/Kconfig               |   9 +
 kernel/trace/Makefile              |   1 +
 kernel/trace/trace.h               |   1 +
 kernel/trace/trace_events.c        |   6 +
 kernel/trace/trace_events_inject.c | 331 +++++++++++++++++++++++++++++
 5 files changed, 348 insertions(+)
 create mode 100644 kernel/trace/trace_events_inject.c
kernel/trace/Kconfig +9 -0
index f67620499faa..29a9c5058b62 100644
@@ -672,6 +672,15 @@ config HIST_TRIGGERS
 	  See Documentation/trace/histogram.rst.
 	  If in doubt, say N.
 
+config TRACE_EVENT_INJECT
+	bool "Trace event injection"
+	depends on TRACING
+	help
+	  Allow user-space to inject a specific trace event into the ring
+	  buffer. This is mainly used for testing purpose.
+
+	  If unsure, say N.
+
 config MMIOTRACE_TEST
 	tristate "Test module for mmiotrace"
 	depends on MMIOTRACE && m
kernel/trace/Makefile +1 -0
index c2b2148bb1d2..0e63db62225f 100644
@@ -69,6 +69,7 @@ obj-$(CONFIG_EVENT_TRACING) += trace_event_perf.o
 endif
 obj-$(CONFIG_EVENT_TRACING) += trace_events_filter.o
 obj-$(CONFIG_EVENT_TRACING) += trace_events_trigger.o
+obj-$(CONFIG_TRACE_EVENT_INJECT) += trace_events_inject.o
 obj-$(CONFIG_HIST_TRIGGERS) += trace_events_hist.o
 obj-$(CONFIG_BPF_EVENTS) += bpf_trace.o
 obj-$(CONFIG_KPROBE_EVENTS) += trace_kprobe.o
kernel/trace/trace.h +1 -0
index ca7fccafbcbb..63bf60f79398 100644
@@ -1601,6 +1601,7 @@ extern struct list_head ftrace_events;
 
 extern const struct file_operations event_trigger_fops;
 extern const struct file_operations event_hist_fops;
+extern const struct file_operations event_inject_fops;
 
 #ifdef CONFIG_HIST_TRIGGERS
 extern int register_trigger_hist_cmd(void);
kernel/trace/trace_events.c +6 -0
index 6b3a69e9aa6a..c6de3cebc127 100644
@@ -2044,6 +2044,12 @@ event_create_dir(struct dentry *parent, struct trace_event_file *file)
 	trace_create_file("format", 0444, file->dir, call,
 			  &ftrace_event_format_fops);
 
+#ifdef CONFIG_TRACE_EVENT_INJECT
+	if (call->event.type && call->class->reg)
+		trace_create_file("inject", 0200, file->dir, file,
+				  &event_inject_fops);
+#endif
+
 	return 0;
 }
 
kernel/trace/trace_events_inject.c new file +331 -0

331 lines not on this page · raw message

Signature · 1 line
2.24.0
author maintainer
#3 lore raw

PATCH 3/3 tracing: Do not create directories if lockdown is in affect

In reply to Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

If lockdown is disabling tracing on boot up, it prevents the tracing files from even bering created. But when that happens, there's several places that will give a warning that the files were not created as that is usually a sign of a bug.

Add in strategic locations where a check is made to see if tracing is disabled by lockdown, and if it is, do not go further, and fail silently (but print that tracing is disabled by lockdown, without doing a WARN_ON()).

Cc Matthew Garrett <[email protected]>
Fixes 17911ff38aa5 ("tracing: Add locked_down checks to the open calls of files created for tracefs")
---
 kernel/trace/ring_buffer.c |  6 ++++++
 kernel/trace/trace.c       | 17 +++++++++++++++++
 2 files changed, 23 insertions(+)
kernel/trace/ring_buffer.c +6 -0
index 66358d66c933..4bf050fcfe3b 100644
@@ -11,6 +11,7 @@
 #include <linux/trace_seq.h>
 #include <linux/spinlock.h>
 #include <linux/irq_work.h>
+#include <linux/security.h>
 #include <linux/uaccess.h>
 #include <linux/hardirq.h>
 #include <linux/kthread.h>	/* for self test */
@@ -5068,6 +5069,11 @@ static __init int test_ringbuffer(void)
 	int cpu;
 	int ret = 0;
 
+	if (security_locked_down(LOCKDOWN_TRACEFS)) {
+		pr_warning("Lockdown is enabled, skipping ring buffer tests\n");
+		return 0;
+	}
+
 	pr_info("Running ring buffer tests...\n");
 
 	buffer = ring_buffer_alloc(RB_TEST_BUFFER_SIZE, RB_FL_OVERWRITE);
kernel/trace/trace.c +17 -0
index 02a23a6e5e00..23459d53d576 100644
@@ -1888,6 +1888,12 @@ int __init register_tracer(struct tracer *type)
 		return -1;
 	}
 
+	if (security_locked_down(LOCKDOWN_TRACEFS)) {
+		pr_warning("Can not register tracer %s due to lockdown\n",
+			   type->name);
+		return -EPERM;
+	}
+
 	mutex_lock(&trace_types_lock);
 
 	tracing_selftest_running = true;
@@ -8789,6 +8795,11 @@ struct dentry *tracing_init_dentry(void)
 {
 	struct trace_array *tr = &global_trace;
 
+	if (security_locked_down(LOCKDOWN_TRACEFS)) {
+		pr_warning("Tracing disabled due to lockdown\n");
+		return ERR_PTR(-EPERM);
+	}
+
 	/* The top level trace array uses  NULL as parent */
 	if (tr->dir)
 		return NULL;
@@ -9231,6 +9242,12 @@ __init static int tracer_alloc_buffers(void)
 	int ring_buf_size;
 	int ret = -ENOMEM;
 
+
+	if (security_locked_down(LOCKDOWN_TRACEFS)) {
+		pr_warning("Tracing disabled due to lockdown\n");
+		return -EPERM;
+	}
+
 	/*
 	 * Make sure we don't accidently add more trace options
 	 * than we have bits for.
Signature · 1 line
2.24.0
author maintainer
#4 lore raw

Re: [PATCH 3/3] tracing: Do not create directories if lockdown is in affect

I should have marked this for stable. The commit it fixes (see Fixes tag) is in 5.4, and it appears this has yet to make it to 5.4 yet.

-- Steve

Quoted 88 lines · Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

If lockdown is disabling tracing on boot up, it prevents the tracing files from even bering created. But when that happens, there's several places that will give a warning that the files were not created as that is usually a sign of a bug.

Add in strategic locations where a check is made to see if tracing is disabled by lockdown, and if it is, do not go further, and fail silently (but print that tracing is disabled by lockdown, without doing a WARN_ON()).

Cc Matthew Garrett <[email protected]>
Fixes 17911ff38aa5 ("tracing: Add locked_down checks to the open calls of files created for tracefs")
---
 kernel/trace/ring_buffer.c |  6 ++++++
 kernel/trace/trace.c       | 17 +++++++++++++++++
 2 files changed, 23 insertions(+)
kernel/trace/ring_buffer.c +6 -0
index 66358d66c933..4bf050fcfe3b 100644
@@ -11,6 +11,7 @@
 #include <linux/trace_seq.h>
 #include <linux/spinlock.h>
 #include <linux/irq_work.h>
+#include <linux/security.h>
 #include <linux/uaccess.h>
 #include <linux/hardirq.h>
 #include <linux/kthread.h>	/* for self test */
@@ -5068,6 +5069,11 @@ static __init int test_ringbuffer(void)
 	int cpu;
 	int ret = 0;
 
+	if (security_locked_down(LOCKDOWN_TRACEFS)) {
+		pr_warning("Lockdown is enabled, skipping ring buffer tests\n");
+		return 0;
+	}
+
 	pr_info("Running ring buffer tests...\n");
 
 	buffer = ring_buffer_alloc(RB_TEST_BUFFER_SIZE, RB_FL_OVERWRITE);
kernel/trace/trace.c +17 -0
index 02a23a6e5e00..23459d53d576 100644
@@ -1888,6 +1888,12 @@ int __init register_tracer(struct tracer *type)
 		return -1;
 	}
 
+	if (security_locked_down(LOCKDOWN_TRACEFS)) {
+		pr_warning("Can not register tracer %s due to lockdown\n",
+			   type->name);
+		return -EPERM;
+	}
+
 	mutex_lock(&trace_types_lock);
 
 	tracing_selftest_running = true;
@@ -8789,6 +8795,11 @@ struct dentry *tracing_init_dentry(void)
 {
 	struct trace_array *tr = &global_trace;
 
+	if (security_locked_down(LOCKDOWN_TRACEFS)) {
+		pr_warning("Tracing disabled due to lockdown\n");
+		return ERR_PTR(-EPERM);
+	}
+
 	/* The top level trace array uses  NULL as parent */
 	if (tr->dir)
 		return NULL;
@@ -9231,6 +9242,12 @@ __init static int tracer_alloc_buffers(void)
 	int ring_buf_size;
 	int ret = -ENOMEM;
 
+
+	if (security_locked_down(LOCKDOWN_TRACEFS)) {
+		pr_warning("Tracing disabled due to lockdown\n");
+		return -EPERM;
+	}
+
 	/*
 	 * Make sure we don't accidently add more trace options
 	 * than we have bits for.
Signature · 1 line
2.24.0
#5 lore raw

Re: [PATCH 3/3] tracing: Do not create directories if lockdown is in affect

Quoted 21 lines · Steven Rostedt

I should have marked this for stable. The commit it fixes (see Fixes tag) is in 5.4, and it appears this has yet to make it to 5.4 yet.

-- Steve

Quoted 14 lines · Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

If lockdown is disabling tracing on boot up, it prevents the tracing files from even bering created. But when that happens, there's several places that will give a warning that the files were not created as that is usually a sign of a bug.

Add in strategic locations where a check is made to see if tracing is disabled by lockdown, and if it is, do not go further, and fail silently (but print that tracing is disabled by lockdown, without doing a WARN_ON()).

Cc Matthew Garrett <[email protected]>
Fixes 17911ff38aa5 ("tracing: Add locked_down checks to the open calls of files created for tracefs")

Relying only on the Fixes: tag to get things picked up by stable is a sure way to get it on the "slow, and maybe eventually, hopefully, it might make it into stable" path :)

I have over 1000 patches right now in that "bucket" that need to be checked to see if they are relevant for stable backporting, just since 5.4 was released. I have automated a lot of it, but still, they require manual review.

I'll go queue this up now, as it's simplest just to ask us to take it after it hits Linus's tree :)

thanks,

greg k-h

author maintainer
#6 lore raw

Re: [PATCH 3/3] tracing: Do not create directories if lockdown is in affect

Quoted 3 lines · On Fri, 10 Jan 2020 17:54:04 +0100 Greg KH <[email protected]> wrote:

Relying only on the Fixes: tag to get things picked up by stable is a sure way to get it on the "slow, and maybe eventually, hopefully, it might make it into stable" path :)

I've been numbed by all the AUTOSEL patches, where I tend to think Fixes is becoming enough. That said, this particular patch, I thought was going in the same release as what it fixed, which is why I never added stable to it. :-/

Quoted 8 lines

I have over 1000 patches right now in that "bucket" that need to be checked to see if they are relevant for stable backporting, just since 5.4 was released. I have automated a lot of it, but still, they require manual review.

I'll go queue this up now, as it's simplest just to ask us to take it after it hits Linus's tree :)

Again, I thought the AUTOSEL would pick it up, as it seems to pick other patches I don't intend on going to stable quickly ;-)

-- Steve