tracing: three more updates
Subsystems (3)
Steven Rostedt
maintainerLinus,
Two fixes and one patch that was missed:
Fixes:
- Missing __print_hex_dump undef for processing new function in trace events
- Stop WARN_ON messages when lockdown disables tracing on boot up
Enhancement:
- Debug option to inject trace events from userspace (for rasdaemon)
The enhancement has its own config option and is non invasive. It's been discussed for sever months and should have been added to my original push, but I never pulled it into my queue.
Please pull the latest trace-v5.5-2 tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace.git
trace-v5.5-2
Tag SHA1: 075c954bfa42bcd2dad254f1fe6fcd81197512de Head SHA1: a356646a56857c2e5ad875beec734d7145ecd49a
Cong Wang (1):
tracing: Introduce trace event injection
Piotr Maziarz (1):
tracing: Fix __print_hex_dump scope
Steven Rostedt (VMware) (1):
tracing: Do not create directories if lockdown is in affect
----
include/trace/trace_events.h | 1 + kernel/trace/Kconfig | 9 + kernel/trace/Makefile | 1 + kernel/trace/ring_buffer.c | 6 + kernel/trace/trace.c | 17 ++ kernel/trace/trace.h | 1 + kernel/trace/trace_events.c | 6 + kernel/trace/trace_events_inject.c | 331 +++++++++++++++++++++++++++++++++++++ 8 files changed, 372 insertions(+) create mode 100644 kernel/trace/trace_events_inject.c
Steven Rostedt
maintainerPATCH 1/3 tracing: Fix __print_hex_dump scope
From: Piotr Maziarz <[email protected]>
undef is needed for parsing __print_hex_dump in traceevent lib.
--- include/trace/trace_events.h | 1 + 1 file changed, 1 insertion(+)
include/trace/trace_events.h +1 -0
@@ -757,6 +757,7 @@ static inline void ftrace_test_probe_##call(void) \#undef __get_str#undef __get_bitmask#undef __print_array +#undef __print_hex_dump#undef TP_printk #define TP_printk(fmt, args...) "\"" fmt "\", " __stringify(args)
Signature · 1 line
2.24.0
Steven Rostedt
maintainerPATCH 2/3 tracing: Introduce trace event injection
In reply to Steven RostedtFrom: Cong Wang <[email protected]>
We have been trying to use rasdaemon to monitor hardware errors like correctable memory errors. rasdaemon uses trace events to monitor various hardware errors. In order to test it, we have to inject some hardware errors, unfortunately not all of them provide error injections. MCE does provide a way to inject MCE errors, but errors like PCI error and devlink error don't, it is not easy to add error injection to each of them. Instead, it is relatively easier to just allow users to inject trace events in a generic way so that all trace events can be injected.
This patch introduces trace event injection, where a new 'inject' is added to each tracepoint directory. Users could write into this file with key=value pairs to specify the value of each fields of the trace event, all unspecified fields are set to zero values by default.
For example, for the net/net_dev_queue tracepoint, we can inject:
INJECT=/sys/kernel/debug/tracing/events/net/net_dev_queue/inject echo "" > $INJECT
echo "name='test'" > $INJECT
echo "name='test' len=1024" > $INJECT
cat /sys/kernel/debug/tracing/trace
...
<...>-614 [000] .... 36.571483: net_dev_queue: dev= skbaddr=00000000fbf338c2 len=0 <...>-614 [001] .... 136.588252: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=0 <...>-614 [001] .N.. 208.431878: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=1024
Triggers could be triggered as usual too:
echo "stacktrace if len == 1025" > /sys/kernel/debug/tracing/events/net/net_dev_queue/trigger echo "len=1025" > $INJECT
cat /sys/kernel/debug/tracing/trace
...
bash-614 [000] .... 36.571483: net_dev_queue: dev= skbaddr=00000000fbf338c2 len=0
bash-614 [001] .... 136.588252: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=0
bash-614 [001] .N.. 208.431878: net_dev_queue: dev=test skbaddr=00000000fbf338c2 len=1024
bash-614 [001] .N.1 284.236349: <stack trace> => event_inject_write
=> vfs_write
=> ksys_write
=> do_syscall_64
=> entry_SYSCALL_64_after_hwframe
The only thing that can't be injected is string pointers as they require constant string pointers, this can't be done at run time.
--- kernel/trace/Kconfig | 9 + kernel/trace/Makefile | 1 + kernel/trace/trace.h | 1 + kernel/trace/trace_events.c | 6 + kernel/trace/trace_events_inject.c | 331 +++++++++++++++++++++++++++++ 5 files changed, 348 insertions(+) create mode 100644 kernel/trace/trace_events_inject.c
kernel/trace/Kconfig +9 -0
@@ -672,6 +672,15 @@ config HIST_TRIGGERS See Documentation/trace/histogram.rst. If in doubt, say N. +config TRACE_EVENT_INJECT + bool "Trace event injection" + depends on TRACING + help + Allow user-space to inject a specific trace event into the ring + buffer. This is mainly used for testing purpose. + + If unsure, say N. + config MMIOTRACE_TEST tristate "Test module for mmiotrace" depends on MMIOTRACE && m
kernel/trace/Makefile +1 -0
@@ -69,6 +69,7 @@ obj-$(CONFIG_EVENT_TRACING) += trace_event_perf.o endif obj-$(CONFIG_EVENT_TRACING) += trace_events_filter.o obj-$(CONFIG_EVENT_TRACING) += trace_events_trigger.o +obj-$(CONFIG_TRACE_EVENT_INJECT) += trace_events_inject.o obj-$(CONFIG_HIST_TRIGGERS) += trace_events_hist.o obj-$(CONFIG_BPF_EVENTS) += bpf_trace.o obj-$(CONFIG_KPROBE_EVENTS) += trace_kprobe.o
kernel/trace/trace.h +1 -0
@@ -1601,6 +1601,7 @@ extern struct list_head ftrace_events;extern const struct file_operations event_trigger_fops;extern const struct file_operations event_hist_fops; +extern const struct file_operations event_inject_fops;#ifdef CONFIG_HIST_TRIGGERS extern int register_trigger_hist_cmd (void );
kernel/trace/trace_events.c +6 -0
@@ -2044,6 +2044,12 @@ event_create_dir(struct dentry *parent, struct trace_event_file *file)trace_create_file ("format" ,0444 , file->dir, call, &ftrace_event_format_fops); +#ifdef CONFIG_TRACE_EVENT_INJECT +if (call->event.type && call->class->reg) +trace_create_file ("inject" ,0200 , file->dir, file, + &event_inject_fops); +#endif +return 0 ; }
kernel/trace/trace_events_inject.c new file +331 -0
331 lines not on this page · raw message
Signature · 1 line
2.24.0
Steven Rostedt
maintainerPATCH 3/3 tracing: Do not create directories if lockdown is in affect
In reply to Steven RostedtFrom: "Steven Rostedt (VMware)" <[email protected]>
If lockdown is disabling tracing on boot up, it prevents the tracing files from even bering created. But when that happens, there's several places that will give a warning that the files were not created as that is usually a sign of a bug.
Add in strategic locations where a check is made to see if tracing is disabled by lockdown, and if it is, do not go further, and fail silently (but print that tracing is disabled by lockdown, without doing a WARN_ON()).
--- kernel/trace/ring_buffer.c | 6 ++++++ kernel/trace/trace.c | 17 +++++++++++++++++ 2 files changed, 23 insertions(+)
kernel/trace/ring_buffer.c +6 -0
@@ -11,6 +11,7 @@#include <linux/trace_seq.h> #include <linux/spinlock.h> #include <linux/irq_work.h> +#include <linux/security.h> #include <linux/uaccess.h> #include <linux/hardirq.h> #include <linux/kthread.h> /* for self test */
@@ -5068,6 +5069,11 @@ static __init int test_ringbuffer(void)int cpu;int ret =0 ; +if (security_locked_down (LOCKDOWN_TRACEFS )) { +pr_warning ("Lockdown is enabled, skipping ring buffer tests\n" ); +return 0 ; + } +pr_info ("Running ring buffer tests...\n" ); buffer =ring_buffer_alloc (RB_TEST_BUFFER_SIZE ,RB_FL_OVERWRITE );
kernel/trace/trace.c +17 -0
@@ -1888,6 +1888,12 @@ int __init register_tracer(struct tracer *type)return -1 ; } +if (security_locked_down (LOCKDOWN_TRACEFS )) { +pr_warning ("Can not register tracer %s due to lockdown\n" , + type->name); +return -EPERM ; + } +mutex_lock (&trace_types_lock); tracing_selftest_running = true;
@@ -8789,6 +8795,11 @@ struct dentry *tracing_init_dentry(void) {struct trace_array *tr = &global_trace; +if (security_locked_down (LOCKDOWN_TRACEFS )) { +pr_warning ("Tracing disabled due to lockdown\n" ); +return ERR_PTR (-EPERM ); + } +/* The top level trace array uses NULL as parent */ if (tr->dir)return NULL ;
@@ -9231,6 +9242,12 @@ __init static int tracer_alloc_buffers(void)int ring_buf_size;int ret = -ENOMEM ; + +if (security_locked_down (LOCKDOWN_TRACEFS )) { +pr_warning ("Tracing disabled due to lockdown\n" ); +return -EPERM ; + } +/* * Make sure we don't accidently add more trace options * than we have bits for.
Signature · 1 line
2.24.0
Steven Rostedt
maintainerRe: [PATCH 3/3] tracing: Do not create directories if lockdown is in affect
I should have marked this for stable. The commit it fixes (see Fixes tag) is in 5.4, and it appears this has yet to make it to 5.4 yet.
-- Steve
Quoted 88 lines · Steven Rostedt
From: "Steven Rostedt (VMware)" <[email protected]>
If lockdown is disabling tracing on boot up, it prevents the tracing files from even bering created. But when that happens, there's several places that will give a warning that the files were not created as that is usually a sign of a bug.
Add in strategic locations where a check is made to see if tracing is disabled by lockdown, and if it is, do not go further, and fail silently (but print that tracing is disabled by lockdown, without doing a
WARN_ON()).Cc Matthew Garrett <[email protected]>Fixes 17911ff38aa5 ("tracing: Add locked_down checks to the open calls of files created for tracefs")Signed-off-by Steven Rostedt (VMware) <[email protected]>--- kernel/trace/ring_buffer.c | 6 ++++++ kernel/trace/trace.c | 17 +++++++++++++++++ 2 files changed, 23 insertions(+)kernel/trace/ring_buffer.c +6
-0@@ -11,6 +11,7 @@#include <linux/trace_seq.h> #include <linux/spinlock.h> #include <linux/irq_work.h> +#include <linux/security.h> #include <linux/uaccess.h> #include <linux/hardirq.h> #include <linux/kthread.h> /* for self test */ @@ -5068,6 +5069,11 @@ static __init int test_ringbuffer(void)int cpu;int ret =0 ; +if (security_locked_down (LOCKDOWN_TRACEFS )) { +pr_warning ("Lockdown is enabled, skipping ring buffer tests\n" ); +return 0 ; + } +pr_info ("Running ring buffer tests...\n" ); buffer =ring_buffer_alloc (RB_TEST_BUFFER_SIZE ,RB_FL_OVERWRITE );kernel/trace/trace.c +17
-0@@ -1888,6 +1888,12 @@ int __init register_tracer(struct tracer *type)return -1 ; } +if (security_locked_down (LOCKDOWN_TRACEFS )) { +pr_warning ("Can not register tracer %s due to lockdown\n" , + type->name); +return -EPERM ; + } +mutex_lock (&trace_types_lock); tracing_selftest_running = true;@@ -8789,6 +8795,11 @@ struct dentry *tracing_init_dentry(void) {struct trace_array *tr = &global_trace; +if (security_locked_down (LOCKDOWN_TRACEFS )) { +pr_warning ("Tracing disabled due to lockdown\n" ); +return ERR_PTR (-EPERM ); + } +/* The top level trace array uses NULL as parent */ if (tr->dir)return NULL ;@@ -9231,6 +9242,12 @@ __init static int tracer_alloc_buffers(void)int ring_buf_size;int ret = -ENOMEM ; + +if (security_locked_down (LOCKDOWN_TRACEFS )) { +pr_warning ("Tracing disabled due to lockdown\n" ); +return -EPERM ; + } +/* * Make sure we don't accidently add more trace options * than we have bits for. Signature · 1 line
2.24.0
Re: [PATCH 3/3] tracing: Do not create directories if lockdown is in affect
Quoted 21 lines · Steven Rostedt
I should have marked this for stable. The commit it fixes (see Fixes tag) is in 5.4, and it appears this has yet to make it to 5.4 yet.
-- Steve
Quoted 14 lines · Steven Rostedt
From: "Steven Rostedt (VMware)" <[email protected]>
If lockdown is disabling tracing on boot up, it prevents the tracing files from even bering created. But when that happens, there's several places that will give a warning that the files were not created as that is usually a sign of a bug.
Add in strategic locations where a check is made to see if tracing is disabled by lockdown, and if it is, do not go further, and fail silently (but print that tracing is disabled by lockdown, without doing a
WARN_ON()).Cc Matthew Garrett <[email protected]>Fixes 17911ff38aa5 ("tracing: Add locked_down checks to the open calls of files created for tracefs")Signed-off-by Steven Rostedt (VMware) <[email protected]>
Relying only on the Fixes: tag to get things picked up by stable is a sure way to get it on the "slow, and maybe eventually, hopefully, it might make it into stable" path :)
I have over 1000 patches right now in that "bucket" that need to be checked to see if they are relevant for stable backporting, just since 5.4 was released. I have automated a lot of it, but still, they require manual review.
I'll go queue this up now, as it's simplest just to ask us to take it after it hits Linus's tree :)
thanks,
greg k-h
Steven Rostedt
maintainerRe: [PATCH 3/3] tracing: Do not create directories if lockdown is in affect
Quoted 3 lines · On Fri, 10 Jan 2020 17:54:04 +0100 Greg KH <[email protected]> wrote:
Relying only on the Fixes: tag to get things picked up by stable is a sure way to get it on the "slow, and maybe eventually, hopefully, it might make it into stable" path :)
I've been numbed by all the AUTOSEL patches, where I tend to think Fixes is becoming enough. That said, this particular patch, I thought was going in the same release as what it fixed, which is why I never added stable to it. :-/
Quoted 8 lines
I have over 1000 patches right now in that "bucket" that need to be checked to see if they are relevant for stable backporting, just since 5.4 was released. I have automated a lot of it, but still, they require manual review.
I'll go queue this up now, as it's simplest just to ask us to take it after it hits Linus's tree :)
Again, I thought the AUTOSEL would pick it up, as it seems to pick other patches I don't intend on going to stable quickly ;-)
-- Steve