tracing: ring-buffer selftest, cleanups, ftrace_dump() fix and document
Subsystems (3)
Steven Rostedt
maintainercover tracing: ring-buffer selftest, cleanups, ftrace_dump() fix and document
One more linux-next patch series.
The first patch is a ring buffer boot time selftest. A bug in my development code looked like the ring buffer was dropping events, and I wrote a test to make sure that wasn't the case. The bug ended up being a stupid mistake in my development code, but the ring buffer test seemed like a good idea to include in the kernel as an option.
Next is a series of clean ups by Jovi.
Jovi also pointed out issues with ftrace_dump(). It's been on my todo list to clean up ftrace_dump() for a long time, and finally decided to do so. It wasn't hard and was quick to do. There was a long standing bug where an NMI triggering a dump while another dump was happening could cause a deadlock. I marked that patch for stable, but since its really just a debugging tool and I don't recall ever hitting the deadlock (I might have), I decided it can wait for 3.10 and I'll do the backports when I get the "Failed to apply" messages from Greg Kroah-Hartman's stable scripts.
Last patch brings the README file in debugfs/tracing/ up to date with more useful information.
Enjoy!
-- Steve
Steven Rostedt (Red Hat) (3):
ring-buffer: Add ring buffer startup selftest
tracing: Fix ftrace_dump()
tracing: Update debugfs README file
zhangwei(Jovi) (6):
tracing: Use pr_warn_once instead of open coded implementation
tracing: Use TRACE_MAX_PRINT instead of constant
tracing: Move find_event_field() into trace_events.c
tracing: Convert trace_destroy_fields() to static
tracing: Fix comment about prefix in arch_syscall_match_sym_name()
tracing: Rename trace_event_mutex to trace_event_sem
----
kernel/trace/Kconfig | 23 +++ kernel/trace/ring_buffer.c | 319 ++++++++++++++++++++++++++++++++++++ kernel/trace/trace.c | 164 +++++++++++------- kernel/trace/trace.h | 6 +- kernel/trace/trace_events.c | 55 +++++-- kernel/trace/trace_events_filter.c | 29 +--- kernel/trace/trace_output.c | 16 +- kernel/trace/trace_output.h | 2 +- kernel/trace/trace_selftest.c | 9 +- kernel/trace/trace_syscalls.c | 2 +- 10 files changed, 506 insertions(+), 119 deletions(-)
Steven Rostedt
maintainerPATCH 1/9 ring-buffer: Add ring buffer startup selftest
From: "Steven Rostedt (Red Hat)" <[email protected]>
When testing my large changes to the ftrace system, there was a bug that looked like the ring buffer was dropping events. I wrote up a quick integrity checker of the ring buffer to see if it was.
Although the bug ended up being something stupid I did in ftrace, and had nothing to do with the ring buffer, I figured if I spent the time to write up this test, I might as well include it in the kernel.
I cleaned it up a bit, as the original version was rather ugly. Not saying this version is pretty, but it's a beauty queen compared to what I original wrote.
To enable the start up test, set CONFIG_RING_BUFFER_STARTUP_TEST.
Note, it runs for 10 seconds, so it will slow your boot time by at least 10 more seconds.
What it does is documented in both the comments and the Kconfig help.
--- kernel/trace/Kconfig | 23 ++++ kernel/trace/ring_buffer.c | 319 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 342 insertions(+)
kernel/trace/Kconfig +23 -0
@@ -565,6 +565,29 @@ config RING_BUFFER_BENCHMARK If unsure, say N. +config RING_BUFFER_STARTUP_TEST + bool "Ring buffer startup self test" + depends on RING_BUFFER + help + Run a simple self test on the ring buffer on boot up. Late in the + kernel boot sequence, the test will start that kicks off + a thread per cpu. Each thread will write various size events + into the ring buffer. Another thread is created to send IPIs + to each of the threads, where the IPI handler will also write + to the ring buffer, to test/stress the nesting ability. + If any anomalies are discovered, a warning will be displayed + and all ring buffers will be disabled. + + The test runs for 10 seconds. This will slow your boot time + by at least 10 more seconds. + + At the end of the test, statics and more checks are done. + It will output the stats of each per cpu buffer. What + was written, the sizes, what was read, what was lost, and + other similar details. + + If unsure, say N + endif # FTRACE endif # TRACING_SUPPORT
kernel/trace/ring_buffer.c +319 -0
332 lines not on this page · raw message
Signature · 1 line
1.7.10.4
Steven Rostedt
maintainerPATCH 2/9 tracing: Use pr_warn_once instead of open coded implementation
In reply to Steven RostedtFrom: "zhangwei(Jovi)" <[email protected]>
Use pr_warn_once, instead of making an open coded implementation.
--- kernel/trace/trace.c | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-)
kernel/trace/trace.c +2 -7
@@ -5205,8 +5205,6 @@ static inline int register_snapshot_cmd(void) { return 0; }struct dentry *tracing_init_dentry_tr (struct trace_array *tr) { -static int once; -if (tr->dir)return tr->dir;
@@ -5216,11 +5214,8 @@ struct dentry *tracing_init_dentry_tr(struct trace_array *tr)if (tr->flags &TRACE_ARRAY_FL_GLOBAL ) tr->dir =debugfs_create_dir ("tracing" ,NULL ); -if (!tr->dir && !once) { - once =1 ; -pr_warning ("Could not create debugfs directory 'tracing'\n" ); -return NULL ; - } +if (!tr->dir) +pr_warn_once ("Could not create debugfs directory 'tracing'\n" );return tr->dir; }
Signature · 1 line
1.7.10.4
Steven Rostedt
maintainerPATCH 3/9 tracing: Use TRACE_MAX_PRINT instead of constant
In reply to Steven RostedtFrom: "zhangwei(Jovi)" <[email protected]>
TRACE_MAX_PRINT macro is defined, but is not used.
--- kernel/trace/trace.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/trace/trace.c +2 -2
@@ -5978,8 +5978,8 @@ void trace_printk_seq(struct trace_seq *s) {/* Probably should print a warning here. */ -if (s->len >=1000 ) - s->len =1000 ; +if (s->len >=TRACE_MAX_PRINT ) + s->len =TRACE_MAX_PRINT ;/* should be zero ended, but we are paranoid. */ s->buffer[s->len] =0 ;
Signature · 1 line
1.7.10.4
Steven Rostedt
maintainerPATCH 4/9 tracing: Move find_event_field() into trace_events.c
In reply to Steven RostedtFrom: "zhangwei(Jovi)" <[email protected]>
By moving find_event_field() and trace_find_field() into trace_events.c, the ftrace_common_fields list and trace_get_fields() can become local to the trace_events.c file.
find_event_field() is renamed to trace_find_event_field() to conform to the tracing global function names.
--- kernel/trace/trace.h | 6 ++---- kernel/trace/trace_events.c | 31 +++++++++++++++++++++++++++++-- kernel/trace/trace_events_filter.c | 29 +---------------------------- 3 files changed, 32 insertions(+), 34 deletions(-)
kernel/trace/trace.h +2 -4
@@ -995,8 +995,6 @@ struct filter_pred {unsigned short right; }; -extern struct list_head ftrace_common_fields; -extern enum regex_type filter_parse_regex (char *buff,int len,char **search,int *not); extern void print_event_filter(struct ftrace_event_call *call,
@@ -1009,8 +1007,8 @@ extern void print_subsystem_event_filter(struct event_subsystem *system, struct trace_seq *s);extern int filter_assign_type (const char *type); -struct list_head * -trace_get_fields (struct ftrace_event_call *event_call); +struct ftrace_event_field * +trace_find_event_field (struct ftrace_event_call *call,char *name); static inline int filter_check_discard(struct ftrace_event_call *call, void *rec,
kernel/trace/trace_events.c +29 -2
@@ -34,7 +34,7 @@ char event_storage[EVENT_STORAGE_SIZE];EXPORT_SYMBOL_GPL (event_storage);LIST_HEAD (ftrace_events); -LIST_HEAD (ftrace_common_fields); +static LIST_HEAD (ftrace_common_fields);#define GFP_TRACE (GFP_KERNEL | __GFP_ZERO)
@@ -54,7 +54,7 @@ static struct kmem_cache *file_cachep;#define while_for_each_event_file () \ } -struct list_head * +static struct list_head *trace_get_fields (struct ftrace_event_call *event_call) {if (!event_call->class->get_fields)
@@ -62,6 +62,33 @@ trace_get_fields(struct ftrace_event_call *event_call)return event_call->class->get_fields (event_call); } +static struct ftrace_event_field * +__find_event_field (struct list_head *head,char *name) +{ +struct ftrace_event_field *field; + +list_for_each_entry (field, head, link) { +if (!strcmp (field->name, name)) +return field; + } + +return NULL ; +} + +struct ftrace_event_field * +trace_find_event_field (struct ftrace_event_call *call,char *name) +{ +struct ftrace_event_field *field; +struct list_head *head; + + field =__find_event_field (&ftrace_common_fields, name); +if (field) +return field; + + head =trace_get_fields (call); +return __find_event_field (head, name); +} +static int __trace_define_field (struct list_head *head,const char *type,const char *name,int offset,int size,int is_signed,int filter_type)
kernel/trace/trace_events_filter.c +1 -28
@@ -658,33 +658,6 @@ void print_subsystem_event_filter(struct event_subsystem *system,mutex_unlock (&event_mutex); } -static struct ftrace_event_field * -__find_event_field (struct list_head *head,char *name) -{ -struct ftrace_event_field *field; - -list_for_each_entry (field, head, link) { -if (!strcmp (field->name, name)) -return field; - } - -return NULL ; -} - -static struct ftrace_event_field * -find_event_field (struct ftrace_event_call *call,char *name) -{ -struct ftrace_event_field *field; -struct list_head *head; - - field =__find_event_field (&ftrace_common_fields, name); -if (field) -return field; - - head =trace_get_fields (call); -return __find_event_field (head, name); -} -static int __alloc_pred_stack (struct pred_stack *stack,int n_preds) { stack->preds =kcalloc (n_preds +1 ,sizeof (*stack->preds),GFP_KERNEL );
@@ -1337,7 +1310,7 @@ static struct filter_pred *create_pred(struct filter_parse_state *ps,return NULL ; } - field =find_event_field (call, operand1); + field =trace_find_event_field (call, operand1);if (!field) {parse_error (ps,FILT_ERR_FIELD_NOT_FOUND ,0 );return NULL ;
Signature · 1 line
1.7.10.4
Steven Rostedt
maintainerPATCH 5/9 tracing: Convert trace_destroy_fields() to static
In reply to Steven RostedtFrom: "zhangwei(Jovi)" <[email protected]>
trace_destroy_fields() is not used outside of the file. It can be a static function.
--- kernel/trace/trace_events.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_events.c +1 -1
@@ -158,7 +158,7 @@ static int trace_define_common_fields(void)return ret; } -void trace_destroy_fields (struct ftrace_event_call *call) +static void trace_destroy_fields (struct ftrace_event_call *call) {struct ftrace_event_field *field, *next;struct list_head *head;
Signature · 1 line
1.7.10.4
Steven Rostedt
maintainerPATCH 6/9 tracing: Fix comment about prefix in arch_syscall_match_sym_name()
In reply to Steven RostedtFrom: "zhangwei(Jovi)" <[email protected]>
ppc64 has its own syscall prefix like ".SyS" or ".sys". Make the comment in arch_syscall_match_sym_name() more understandable.
--- kernel/trace/trace_syscalls.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_syscalls.c +1 -1
@@ -37,7 +37,7 @@ static inline bool arch_syscall_match_sym_name(const char *sym, const char *name/* * Only compare after the "sys" prefix. Archs that use * syscall wrappers may have syscalls symbols aliases prefixed -* with "SyS" instead of "sys", leading to an unwanted +* with ".SyS" or ".sys" instead of "sys", leading to an unwanted * mismatch. */ return !strcmp (sym +3 , name +3 );
Signature · 1 line
1.7.10.4
Steven Rostedt
maintainerPATCH 7/9 tracing: Rename trace_event_mutex to trace_event_sem
In reply to Steven RostedtFrom: "zhangwei(Jovi)" <[email protected]>
trace_event_mutex is an rw semaphore now, not a mutex, change the name.
--- kernel/trace/trace_events.c | 22 +++++++++++----------- kernel/trace/trace_output.c | 16 ++++++++-------- kernel/trace/trace_output.h | 2 +- 3 files changed, 20 insertions(+), 20 deletions(-)
kernel/trace/trace_events.c +11 -11
@@ -1584,7 +1584,7 @@ int trace_add_event_call(struct ftrace_event_call *call) }/* -* Must be called under locking both of event_mutex and trace_event_mutex. +* Must be called under locking both of event_mutex and trace_event_sem. */ static void __trace_remove_event_call (struct ftrace_event_call *call) {
@@ -1597,9 +1597,9 @@ static void __trace_remove_event_call(struct ftrace_event_call *call)void trace_remove_event_call (struct ftrace_event_call *call) {mutex_lock (&event_mutex); -down_write (&trace_event_mutex); +down_write (&trace_event_sem);__trace_remove_event_call (call); -up_write (&trace_event_mutex); +up_write (&trace_event_sem);mutex_unlock (&event_mutex); }
@@ -1707,7 +1707,7 @@ static void trace_module_remove_events(struct module *mod)struct ftrace_event_call *call, *p;bool clear_trace = false; -down_write (&trace_event_mutex); +down_write (&trace_event_sem);list_for_each_entry_safe (call, p, &ftrace_events, list) {if (call->mod == mod) {if (call->flags &TRACE_EVENT_FL_WAS_ENABLED )
@@ -1725,7 +1725,7 @@ static void trace_module_remove_events(struct module *mod)list_del (&file_ops->list);kfree (file_ops); } -up_write (&trace_event_mutex); +up_write (&trace_event_sem);/* * It is safest to reset the ring buffer if the module being unloaded
@@ -2262,9 +2262,9 @@ int event_trace_add_tracer(struct dentry *parent, struct trace_array *tr)if (ret) goto out_unlock; -down_write (&trace_event_mutex); +down_write (&trace_event_sem);__trace_add_event_dirs (tr); -up_write (&trace_event_mutex); +up_write (&trace_event_sem); out_unlock:mutex_unlock (&event_mutex);
@@ -2287,9 +2287,9 @@ early_event_add_tracer(struct dentry *parent, struct trace_array *tr)if (ret) goto out_unlock; -down_write (&trace_event_mutex); +down_write (&trace_event_sem);__trace_early_add_event_dirs (tr); -up_write (&trace_event_mutex); +up_write (&trace_event_sem); out_unlock:mutex_unlock (&event_mutex);
@@ -2304,10 +2304,10 @@ int event_trace_del_tracer(struct trace_array *tr)mutex_lock (&event_mutex); -down_write (&trace_event_mutex); +down_write (&trace_event_sem);__trace_remove_event_dirs (tr);debugfs_remove_recursive (tr->event_dir); -up_write (&trace_event_mutex); +up_write (&trace_event_sem); tr->event_dir =NULL ;
kernel/trace/trace_output.c +8 -8
@@ -14,7 +14,7 @@/* must be a power of 2 */ #define EVENT_HASHSIZE 128 -DECLARE_RWSEM (trace_event_mutex); +DECLARE_RWSEM (trace_event_sem);static struct hlist_head event_hash[EVENT_HASHSIZE ]__read_mostly ;
@@ -826,12 +826,12 @@ static int trace_search_list(struct list_head **list)void trace_event_read_lock (void ) { -down_read (&trace_event_mutex); +down_read (&trace_event_sem); }void trace_event_read_unlock (void ) { -up_read (&trace_event_mutex); +up_read (&trace_event_sem); }/**
@@ -854,7 +854,7 @@ int register_ftrace_event(struct trace_event *event)unsigned key;int ret =0 ; -down_write (&trace_event_mutex); +down_write (&trace_event_sem);if (WARN_ON (!event)) goto out;
@@ -909,14 +909,14 @@ int register_ftrace_event(struct trace_event *event) ret = event->type; out: -up_write (&trace_event_mutex); +up_write (&trace_event_sem);return ret; }EXPORT_SYMBOL_GPL (register_ftrace_event);/* -* Used by module code with the trace_event_mutex held for write. +* Used by module code with the trace_event_sem held for write. */ int __unregister_ftrace_event (struct trace_event *event) {
@@ -931,9 +931,9 @@ int __unregister_ftrace_event(struct trace_event *event)*/ int unregister_ftrace_event (struct trace_event *event) { -down_write (&trace_event_mutex); +down_write (&trace_event_sem);__unregister_ftrace_event (event); -up_write (&trace_event_mutex); +up_write (&trace_event_sem);return 0 ; }
kernel/trace/trace_output.h +1 -1
@@ -33,7 +33,7 @@ trace_print_lat_fmt(struct trace_seq *s, struct trace_entry *entry);/* used by module unregistering */ extern int __unregister_ftrace_event (struct trace_event *event); -extern struct rw_semaphore trace_event_mutex; +extern struct rw_semaphore trace_event_sem;#define MAX_MEMHEX_BYTES 8#define HEX_CHARS (MAX_MEMHEX_BYTES*2 + 1)
Signature · 1 line
1.7.10.4
Steven Rostedt
maintainerPATCH 8/9 tracing: Fix ftrace_dump()
In reply to Steven RostedtFrom: "Steven Rostedt (Red Hat)" <[email protected]>
ftrace_dump() had a lot of issues. What ftrace_dump() does, is when ftrace_dump_on_oops is set (via a kernel parameter or sysctl), it will dump out the ftrace buffers to the console when either a oops, panic, or a sysrq-z occurs.
This was written a long time ago when ftrace was fragile to recursion. But it wasn't written well even for that.
There's a possible deadlock that can occur if a ftrace_dump() is happening and an NMI triggers another dump. This is because it grabs a lock before checking if the dump ran.
It also totally disables ftrace, and tracing for no good reasons.
As the ring_buffer now checks if it is read via a oops or NMI, where there's a chance that the buffer gets corrupted, it will disable itself. No need to have ftrace_dump() do the same.
ftrace_dump() is now cleaned up where it uses an atomic counter to make sure only one dump happens at a time. A simple atomic_inc_return() is enough that is needed for both other CPUs and NMIs. No need for a spinlock, as if one CPU is running the dump, no other CPU needs to do it too.
The tracing_on variable is turned off and not turned on. The original code did this, but it wasn't pretty. By just disabling this variable we get the result of not seeing traces that happen between crashes.
For sysrq-z, it doesn't get turned on, but the user can always write a '1' to the tracing_on file. If they are using sysrq-z, then they should know about tracing_on.
The new code is much easier to read and less error prone. No more deadlock possibility when an NMI triggers here.
--- kernel/trace/trace.c | 62 +++++++++++++++++------------------------ kernel/trace/trace_selftest.c | 9 +++--- 2 files changed, 31 insertions(+), 40 deletions(-)
kernel/trace/trace.c +26 -36
@@ -5997,36 +5997,32 @@ void trace_init_global_iter(struct trace_iterator *iter) iter->trace_buffer = &global_trace.trace_buffer; } -static void -__ftrace_dump (bool disable_tracing,enum ftrace_dump_mode oops_dump_mode) +void ftrace_dump (enum ftrace_dump_mode oops_dump_mode) { -static arch_spinlock_t ftrace_dump_lock = - (arch_spinlock_t )__ARCH_SPIN_LOCK_UNLOCKED;/* use static because iter can be a bit big for the stack */ static struct trace_iterator iter; +static atomic_t dump_running;unsigned int old_userobj; -static int dump_ran;unsigned long flags;int cnt =0 , cpu; -/* only one dump */ -local_irq_save (flags); -arch_spin_lock (&ftrace_dump_lock); -if (dump_ran) - goto out; - - dump_ran =1 ; +/* Only allow one dump user at a time. */ +if (atomic_inc_return (&dump_running) !=1 ) { +atomic_dec (&dump_running); +return ; + } +/* +* Always turn off tracing when we dump. +* We don't need to show trace output of what happens +* between multiple crashes. +* +* If the user does a sysrq-z, then they can re-enable +* tracing with echo 1 > tracing_on. +*/ tracing_off (); -/* Did function tracer already get disabled? */ -if (ftrace_is_dead ()) { -printk ("# WARNING: FUNCTION TRACING IS CORRUPTED\n" ); -printk ("# MAY BE MISSING FUNCTION EVENTS\n" ); - } - -if (disable_tracing) -ftrace_kill (); +local_irq_save (flags);/* Simulate the iterator */ trace_init_global_iter (&iter);
@@ -6056,6 +6052,12 @@ __ftrace_dump(bool disable_tracing, enum ftrace_dump_mode oops_dump_mode)printk (KERN_TRACE "Dumping ftrace buffer:\n" ); +/* Did function tracer already get disabled? */ +if (ftrace_is_dead ()) { +printk ("# WARNING: FUNCTION TRACING IS CORRUPTED\n" ); +printk ("# MAY BE MISSING FUNCTION EVENTS\n" ); + } +/* * We need to stop all tracing on all CPUS to read the * the next buffer. This is a bit expensive, but is
@@ -6095,26 +6097,14 @@ __ftrace_dump(bool disable_tracing, enum ftrace_dump_mode oops_dump_mode)printk (KERN_TRACE "---------------------------------\n" ); out_enable: -/* Re-enable tracing if requested */ -if (!disable_tracing) { - trace_flags |= old_userobj; + trace_flags |= old_userobj; -for_each_tracing_cpu (cpu) { -atomic_dec (&per_cpu_ptr (iter.trace_buffer->data, cpu)->disabled); - } -tracing_on (); +for_each_tracing_cpu (cpu) { +atomic_dec (&per_cpu_ptr (iter.trace_buffer->data, cpu)->disabled); } - - out: -arch_spin_unlock (&ftrace_dump_lock); +atomic_dec (&dump_running);local_irq_restore (flags); } - -/* By default: disable tracing after the dump */ -void ftrace_dump (enum ftrace_dump_mode oops_dump_mode) -{ -__ftrace_dump (true, oops_dump_mode); -}EXPORT_SYMBOL_GPL (ftrace_dump);__init static int tracer_alloc_buffers (void )
kernel/trace/trace_selftest.c +5 -4
@@ -703,8 +703,6 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr)/* Maximum number of functions to trace before diagnosing a hang */ #define GRAPH_MAX_FUNC_TEST 100000000 -static void -__ftrace_dump (bool disable_tracing,enum ftrace_dump_mode oops_dump_mode);static unsigned int graph_hang_thresh;/* Wrap the real function entry probe to avoid possible hanging */
@@ -714,8 +712,11 @@ static int trace_graph_entry_watchdog(struct ftrace_graph_ent *trace)if (unlikely (++graph_hang_thresh >GRAPH_MAX_FUNC_TEST )) {ftrace_graph_stop ();printk (KERN_WARNING "BUG: Function graph tracer hang!\n" ); -if (ftrace_dump_on_oops) -__ftrace_dump (false,DUMP_ALL ); +if (ftrace_dump_on_oops) { +ftrace_dump (DUMP_ALL ); +/* ftrace_dump() disables tracing */ +tracing_on (); + }return 0 ; }
Signature · 1 line
1.7.10.4
Steven Rostedt
maintainerPATCH 9/9 tracing: Update debugfs README file
In reply to Steven RostedtFrom: "Steven Rostedt (Red Hat)" <[email protected]>
Update the README file in debugfs/tracing to something more useful. What's currently in the file is very old and what it shows doesn't have much use. Heck, it tells you how to mount debugfs! But to read this file you would have already needed to mount it.
Replace the file with current up-to-date information. It's rather limited, but what do you expect from a pseudo README file.
--- kernel/trace/trace.c | 89 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 75 insertions(+), 14 deletions(-)
kernel/trace/trace.c +75 -14
@@ -3300,20 +3300,81 @@ static const struct file_operations tracing_iter_fops = {static const char readme_msg[] = "tracing mini-HOWTO:\n\n" -"# mount -t debugfs nodev /sys/kernel/debug\n\n" -"# cat /sys/kernel/debug/tracing/available_tracers\n" -"wakeup wakeup_rt preemptirqsoff preemptoff irqsoff function nop\n\n" -"# cat /sys/kernel/debug/tracing/current_tracer\n" -"nop\n" -"# echo wakeup > /sys/kernel/debug/tracing/current_tracer\n" -"# cat /sys/kernel/debug/tracing/current_tracer\n" -"wakeup\n" -"# cat /sys/kernel/debug/tracing/trace_options\n" -"noprint-parent nosym-offset nosym-addr noverbose\n" -"# echo print-parent > /sys/kernel/debug/tracing/trace_options\n" -"# echo 1 > /sys/kernel/debug/tracing/tracing_on\n" -"# cat /sys/kernel/debug/tracing/trace > /tmp/trace.txt\n" -"# echo 0 > /sys/kernel/debug/tracing/tracing_on\n" + "# echo 0 > tracing_on : quick way to disable tracing\n" + "# echo 1 > tracing_on : quick way to re-enable tracing\n\n" + " Important files:\n" + " trace\t\t\t- The static contents of the buffer\n" + "\t\t\t To clear the buffer write into this file: echo > trace\n" + " trace_pipe\t\t- A consuming read to see the contents of the buffer\n" + " current_tracer\t- function and latency tracers\n" + " available_tracers\t- list of configured tracers for current_tracer\n" + " buffer_size_kb\t- view and modify size of per cpu buffer\n" + " buffer_total_size_kb - view total size of all cpu buffers\n\n" + " trace_clock\t\t-change the clock used to order events\n" + " local: Per cpu clock but may not be synced across CPUS\n" + " global: Synced across CPUs but slows tracing down.\n" + " counter: Not a clock, but just an increment\n" + " uptime: Jiffy counter from time of boot\n" + " perf: Same clock that perf events use\n" +#ifdef CONFIG_X86_64 + " x86-tsc: TSC cycle counter\n" +#endif +"\n trace_marker\t\t- Writes into this file writes into the kernel buffer\n" +" tracing_cpumask\t- Limit which CPUs to trace\n" +" instances\t\t- Make sub-buffers with: mkdir instances/foo\n" +"\t\t\t Remove sub-buffer with rmdir\n" +" trace_options\t\t- Set format or modify how tracing happens\n" +"\t\t\t Disable an option by adding a suffix 'no' to the option name\n" +#ifdef CONFIG_DYNAMIC_FTRACE +"\n available_filter_functions - list of functions that can be filtered on\n" +" set_ftrace_filter\t- echo function name in here to only trace these functions\n" +" accepts: func_full_name, *func_end, func_begin*, *func_middle*\n" +" modules: Can select a group via module\n" +" Format: :mod:<module-name>\n" +" example: echo :mod:ext3 > set_ftrace_filter\n" +" triggers: a command to perform when function is hit\n" +" Format: <function>:<trigger>[:count]\n" +" trigger: traceon, traceoff\n" +" enable_event:<system>:<event>\n" +" disable_event:<system>:<event>\n" +#ifdef CONFIG_STACKTRACE +" stacktrace\n" +#endif +#ifdef CONFIG_TRACER_SNAPSHOT + " snapshot\n" +#endif + " example: echo do_fault:traceoff > set_ftrace_filter\n" + " echo do_trap:traceoff:3 > set_ftrace_filter\n" + " The first one will disable tracing everytime do_fault is hit\n" + " The second will disable tracing at most 3 times whne do_trap is hit\n" + " To remove trigger without count:\n" + " echo '!<function>:<trigger> > set_ftrace_filter\n" + " To remove trigger with a count:\n" + " echo '!<function>:<trigger>:0 > set_ftrace_filter\n" + " set_ftrace_notrace\t- echo function name in here to never trace.\n" + " accepts: func_full_name, *func_end, func_begin*, *func_middle*\n" + " modules: Can select a group via module command :mod:\n" + " Does not accept triggers\n" +#endif /* CONFIG_DYNAMIC_FTRACE */ +#ifdef CONFIG_FUNCTION_TRACER + " set_ftrace_pid\t- Write pid(s) to only function trace those pids (function)\n" +#endif +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +" set_graph_function\t- Trace the nested calls of a function (function_graph)\n" + " max_graph_depth\t- Trace a limited depth of nested calls (0 is unlimited)\n" +#endif +#ifdef CONFIG_TRACER_SNAPSHOT +"\n snashot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n" + "\t\t\t Read the contents for more infromation\n" +#endif +#ifdef CONFIG_STACKTRACE +" stack_trace\t\t- Shows the max stack trace when active\n" +" stack_max_size\t- Shows current max stack size that was traced\n" +"\t\t\t Write into this file to reset the max size (trigger a new trace)\n" +#ifdef CONFIG_DYNAMIC_FTRACE + " stack_trace_filter\t- Like set_ftrace_filter but limits what stack_trace traces\n" +#endif +#endif /* CONFIG_STACKTRACE */ ; static ssize_t
Signature · 1 line
1.7.10.4
Re: [for-next][PATCH 9/9] tracing: Update debugfs README file
Hi Steve,
Quoted 74 lines · Steven Rostedt
From: "Steven Rostedt (Red Hat)" <[email protected]>
Update the README file in debugfs/tracing to something more useful. What's currently in the file is very old and what it shows doesn't have much use. Heck, it tells you how to mount debugfs! But to read this file you would have already needed to mount it.
Replace the file with current up-to-date information. It's rather limited, but what do you expect from a pseudo README file.
Signed-off-by Steven Rostedt <[email protected]>--- kernel/trace/trace.c | 89 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 75 insertions(+), 14 deletions(-)kernel/trace/trace.c +37
-14@@ -3300,20 +3300,81 @@ static const struct file_operations tracing_iter_fops = {static const char readme_msg[] = "tracing mini-HOWTO:\n\n" -"# mount -t debugfs nodev /sys/kernel/debug\n\n" -"# cat /sys/kernel/debug/tracing/available_tracers\n" -"wakeup wakeup_rt preemptirqsoff preemptoff irqsoff function nop\n\n" -"# cat /sys/kernel/debug/tracing/current_tracer\n" -"nop\n" -"# echo wakeup > /sys/kernel/debug/tracing/current_tracer\n" -"# cat /sys/kernel/debug/tracing/current_tracer\n" -"wakeup\n" -"# cat /sys/kernel/debug/tracing/trace_options\n" -"noprint-parent nosym-offset nosym-addr noverbose\n" -"# echo print-parent > /sys/kernel/debug/tracing/trace_options\n" -"# echo 1 > /sys/kernel/debug/tracing/tracing_on\n" -"# cat /sys/kernel/debug/tracing/trace > /tmp/trace.txt\n" -"# echo 0 > /sys/kernel/debug/tracing/tracing_on\n" + "# echo 0 > tracing_on : quick way to disable tracing\n" + "# echo 1 > tracing_on : quick way to re-enable tracing\n\n" + " Important files:\n" + " trace\t\t\t- The static contents of the buffer\n" + "\t\t\t To clear the buffer write into this file: echo > trace\n" + " trace_pipe\t\t- A consuming read to see the contents of the buffer\n" + " current_tracer\t- function and latency tracers\n" + " available_tracers\t- list of configured tracers for current_tracer\n" + " buffer_size_kb\t- view and modify size of per cpu buffer\n" + " buffer_total_size_kb - view total size of all cpu buffers\n\n" + " trace_clock\t\t-change the clock used to order events\n" + " local: Per cpu clock but may not be synced across CPUS\n" + " global: Synced across CPUs but slows tracing down.\n" + " counter: Not a clock, but just an increment\n" + " uptime: Jiffy counter from time of boot\n" + " perf: Same clock that perf events use\n" +#ifdef CONFIG_X86_64 + " x86-tsc: TSC cycle counter\n" +#endif +"\n trace_marker\t\t- Writes into this file writes into the kernel buffer\n" +" tracing_cpumask\t- Limit which CPUs to trace\n" +" instances\t\t- Make sub-buffers with: mkdir instances/foo\n" +"\t\t\t Remove sub-buffer with rmdir\n" +" trace_options\t\t- Set format or modify how tracing happens\n" +"\t\t\t Disable an option by adding a suffix 'no' to the option name\n" +#ifdef CONFIG_DYNAMIC_FTRACE +"\n available_filter_functions - list of functions that can be filtered on\n" +" set_ftrace_filter\t- echo function name in here to only trace these functions\n" +" accepts: func_full_name, *func_end, func_begin*, *func_middle*\n" +" modules: Can select a group via module\n" +" Format: :mod:<module-name>\n" +" example: echo :mod:ext3 > set_ftrace_filter\n" +" triggers: a command to perform when function is hit\n" +" Format: <function>:<trigger>[:count]\n" +" trigger: traceon, traceoff\n" +" enable_event:<system>:<event>\n" +" disable_event:<system>:<event>\n"
kernel/trace/trace.c static const struct file_operations tracing_iter_fops = { · line 3349
Quoted 10 lines
kernel/trace/trace.c +10
-0+#ifdef CONFIG_STACKTRACE + " stacktrace\n" +#endif +#ifdef CONFIG_TRACER_SNAPSHOT + " snapshot\n" +#endif +" example: echo do_fault:traceoff > set_ftrace_filter\n" +" echo do_trap:traceoff:3 > set_ftrace_filter\n" +" The first one will disable tracing everytime do_fault is hit\n" +" The second will disable tracing at most 3 times whne do_trap is hit\n"
s/whne/when/
It's slightly confusing for me to read the "at most" part. Did you mean it by "The second will disable tracing after do_trace is hit 3 times"?
Thanks,
Namhyung
Quoted 31 lines
kernel/trace/trace.c +28
-0+" To remove trigger without count:\n" +" echo '!<function>:<trigger> > set_ftrace_filter\n" +" To remove trigger with a count:\n" +" echo '!<function>:<trigger>:0 > set_ftrace_filter\n" +" set_ftrace_notrace\t- echo function name in here to never trace.\n" +" accepts: func_full_name, *func_end, func_begin*, *func_middle*\n" +" modules: Can select a group via module command :mod:\n" + " Does not accept triggers\n" +#endif /* CONFIG_DYNAMIC_FTRACE */ +#ifdef CONFIG_FUNCTION_TRACER + " set_ftrace_pid\t- Write pid(s) to only function trace those pids (function)\n" +#endif +#ifdef CONFIG_FUNCTION_GRAPH_TRACER +" set_graph_function\t- Trace the nested calls of a function (function_graph)\n" + " max_graph_depth\t- Trace a limited depth of nested calls (0 is unlimited)\n" +#endif +#ifdef CONFIG_TRACER_SNAPSHOT +"\n snashot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n" + "\t\t\t Read the contents for more infromation\n" +#endif +#ifdef CONFIG_STACKTRACE +" stack_trace\t\t- Shows the max stack trace when active\n" +" stack_max_size\t- Shows current max stack size that was traced\n" +"\t\t\t Write into this file to reset the max size (trigger a new trace)\n" +#ifdef CONFIG_DYNAMIC_FTRACE + " stack_trace_filter\t- Like set_ftrace_filter but limits what stack_trace traces\n" +#endif +#endif /* CONFIG_STACKTRACE */ ; static ssize_t
Re: [for-next][PATCH 9/9] tracing: Update debugfs README file
Quoted 109 lines · Namhyung Kim
Hi Steve,
Quoted 74 lines · Steven Rostedt
From: "Steven Rostedt (Red Hat)" <[email protected]>
Update the README file in debugfs/tracing to something more useful. What's currently in the file is very old and what it shows doesn't have much use. Heck, it tells you how to mount debugfs! But to read this file you would have already needed to mount it.
Replace the file with current up-to-date information. It's rather limited, but what do you expect from a pseudo README file.
Signed-off-by Steven Rostedt <[email protected]>--- kernel/trace/trace.c | 89 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 75 insertions(+), 14 deletions(-)kernel/trace/trace.c +37
-14@@ -3300,20 +3300,81 @@ static const struct file_operations tracing_iter_fops = {static const char readme_msg[] = "tracing mini-HOWTO:\n\n" -"# mount -t debugfs nodev /sys/kernel/debug\n\n" -"# cat /sys/kernel/debug/tracing/available_tracers\n" -"wakeup wakeup_rt preemptirqsoff preemptoff irqsoff function nop\n\n" -"# cat /sys/kernel/debug/tracing/current_tracer\n" -"nop\n" -"# echo wakeup > /sys/kernel/debug/tracing/current_tracer\n" -"# cat /sys/kernel/debug/tracing/current_tracer\n" -"wakeup\n" -"# cat /sys/kernel/debug/tracing/trace_options\n" -"noprint-parent nosym-offset nosym-addr noverbose\n" -"# echo print-parent > /sys/kernel/debug/tracing/trace_options\n" -"# echo 1 > /sys/kernel/debug/tracing/tracing_on\n" -"# cat /sys/kernel/debug/tracing/trace > /tmp/trace.txt\n" -"# echo 0 > /sys/kernel/debug/tracing/tracing_on\n" + "# echo 0 > tracing_on : quick way to disable tracing\n" + "# echo 1 > tracing_on : quick way to re-enable tracing\n\n" + " Important files:\n" + " trace\t\t\t- The static contents of the buffer\n" + "\t\t\t To clear the buffer write into this file: echo > trace\n" + " trace_pipe\t\t- A consuming read to see the contents of the buffer\n" + " current_tracer\t- function and latency tracers\n" + " available_tracers\t- list of configured tracers for current_tracer\n" + " buffer_size_kb\t- view and modify size of per cpu buffer\n" + " buffer_total_size_kb - view total size of all cpu buffers\n\n" + " trace_clock\t\t-change the clock used to order events\n" + " local: Per cpu clock but may not be synced across CPUS\n" + " global: Synced across CPUs but slows tracing down.\n" + " counter: Not a clock, but just an increment\n" + " uptime: Jiffy counter from time of boot\n" + " perf: Same clock that perf events use\n" +#ifdef CONFIG_X86_64 + " x86-tsc: TSC cycle counter\n" +#endif +"\n trace_marker\t\t- Writes into this file writes into the kernel buffer\n" +" tracing_cpumask\t- Limit which CPUs to trace\n" +" instances\t\t- Make sub-buffers with: mkdir instances/foo\n" +"\t\t\t Remove sub-buffer with rmdir\n" +" trace_options\t\t- Set format or modify how tracing happens\n" +"\t\t\t Disable an option by adding a suffix 'no' to the option name\n" +#ifdef CONFIG_DYNAMIC_FTRACE +"\n available_filter_functions - list of functions that can be filtered on\n" +" set_ftrace_filter\t- echo function name in here to only trace these functions\n" +" accepts: func_full_name, *func_end, func_begin*, *func_middle*\n" +" modules: Can select a group via module\n" +" Format: :mod:<module-name>\n" +" example: echo :mod:ext3 > set_ftrace_filter\n" +" triggers: a command to perform when function is hit\n" +" Format: <function>:<trigger>[:count]\n" +" trigger: traceon, traceoff\n" +" enable_event:<system>:<event>\n" +" disable_event:<system>:<event>\n" kernel/trace/trace.c static const struct file_operations tracing_iter_fops = { · line 3349
Quoted 10 lines
kernel/trace/trace.c +10
-0+#ifdef CONFIG_STACKTRACE + " stacktrace\n" +#endif +#ifdef CONFIG_TRACER_SNAPSHOT + " snapshot\n" +#endif +" example: echo do_fault:traceoff > set_ftrace_filter\n" +" echo do_trap:traceoff:3 > set_ftrace_filter\n" +" The first one will disable tracing everytime do_fault is hit\n" +" The second will disable tracing at most 3 times whne do_trap is hit\n" s/whne/when/
It's slightly confusing for me to read the "at most" part. Did you mean it by "The second will disable tracing after
do_traceis hit 3 times"?Thanks,
NamhyungQuoted 12 lines
kernel/trace/trace.c +12
-0+" To remove trigger without count:\n" +" echo '!<function>:<trigger> > set_ftrace_filter\n" +" To remove trigger with a count:\n" +" echo '!<function>:<trigger>:0 > set_ftrace_filter\n" +" set_ftrace_notrace\t- echo function name in here to never trace.\n" +" accepts: func_full_name, *func_end, func_begin*, *func_middle*\n" +" modules: Can select a group via module command :mod:\n" + " Does not accept triggers\n" +#endif /* CONFIG_DYNAMIC_FTRACE */ +#ifdef CONFIG_FUNCTION_TRACER + " set_ftrace_pid\t- Write pid(s) to only function trace those pids (function)\n" +#endif
Quoted 8 lines
kernel/trace/trace.c static const struct file_operations tracing_iter_fops = { · line 3369
Quoted 8 lines
kernel/trace/trace.c +8
-0+#ifdef CONFIG_FUNCTION_GRAPH_TRACER +" set_graph_function\t- Trace the nested calls of a function (function_graph)\n" + " max_graph_depth\t- Trace a limited depth of nested calls (0 is unlimited)\n" +#endif +#ifdef CONFIG_TRACER_SNAPSHOT +"\n snashot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n" + "\t\t\t Read the contents for more infromation\n" +#endif
There're two more typos.
s/snashot/snapshot/
s/infromation/information/
Quoted 16 lines
kernel/trace/trace.c static const struct file_operations tracing_iter_fops = { · line 3380
Quoted 11 lines
kernel/trace/trace.c +8
-0+#ifdef CONFIG_STACKTRACE +" stack_trace\t\t- Shows the max stack trace when active\n" +" stack_max_size\t- Shows current max stack size that was traced\n" +"\t\t\t Write into this file to reset the max size (trigger a new trace)\n" +#ifdef CONFIG_DYNAMIC_FTRACE + " stack_trace_filter\t- Like set_ftrace_filter but limits what stack_trace traces\n" +#endif +#endif /* CONFIG_STACKTRACE */ ; static ssize_t--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Steven Rostedt
maintainerRe: [for-next][PATCH 9/9] tracing: Update debugfs README file
Quoted 10 lines · Keun-O Park
Quoted 5 lines
kernel/trace/trace.c static const struct file_operations tracing_iter_fops = { · line 3369
Quoted 5 lines
kernel/trace/trace.c +5
-0+#endif +#ifdef CONFIG_TRACER_SNAPSHOT +"\n snashot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n" + "\t\t\t Read the contents for more infromation\n" +#endif There're two more typos.
s/snashot/snapshot/
s/infromation/information/
spellcheck doesn't work well for patches. Thanks,
-- Steve
Steven Rostedt
maintainerRe: [for-next][PATCH 9/9] tracing: Update debugfs README file
In reply to Keun-O ParkHere's the new version of the patch.
The diff between versions is:
kernel/trace/trace.c +8 -5
@@ -3311,7 +3311,7 @@ static const char readme_msg[] =" buffer_size_kb\t- view and modify size of per cpu buffer\n" " buffer_total_size_kb - view total size of all cpu buffers\n\n" " trace_clock\t\t-change the clock used to order events\n" -" local: Per cpu clock but may not be synced across CPUS\n" +" local: Per cpu clock but may not be synced across CPUs\n" " global: Synced across CPUs but slows tracing down.\n" " counter: Not a clock, but just an increment\n" " uptime: Jiffy counter from time of boot\n"
@@ -3345,8 +3345,11 @@ static const char readme_msg[] =#endif " example: echo do_fault:traceoff > set_ftrace_filter\n" " echo do_trap:traceoff:3 > set_ftrace_filter\n" -" The first one will disable tracing everytime do_fault is hit\n" -" The second will disable tracing at most 3 times whne do_trap is hit\n" +" The first one will disable tracing every time do_fault is hit\n" +" The second will disable tracing at most 3 times when do_trap is hit\n" +" The first time do trap is hit and it disables tracing, the counter\n" +" will decrement to 2. If tracing is already disabled, the counter\n" +" will not decrement. It only decrements when the trigger did work\n" " To remove trigger without count:\n" " echo '!<function>:<trigger> > set_ftrace_filter\n" " To remove trigger with a count:\n"
@@ -3364,8 +3367,8 @@ static const char readme_msg[] = " max_graph_depth\t- Trace a limited depth of nested calls (0 is unlimited)\n"#endif #ifdef CONFIG_TRACER_SNAPSHOT -"\n snashot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n" - "\t\t\t Read the contents for more infromation\n" +"\n snapshot\t\t- Like 'trace' but shows the content of the static snapshot buffer\n" + "\t\t\t Read the contents for more information\n"#endif #ifdef CONFIG_STACKTRACE " stack_trace\t\t- Shows the max stack trace when active\n"
And here's the real patch:
tracing: Update debugfs README file
Update the README file in debugfs/tracing to something more useful. What's currently in the file is very old and what it shows doesn't have much use. Heck, it tells you how to mount debugfs! But to read this file you would have already needed to mount it.
Replace the file with current up-to-date information. It's rather limited, but what do you expect from a pseudo README file.
--- kernel/trace/trace.c | 92 ++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 78 insertions(+), 14 deletions(-)
kernel/trace/trace.c +78 -14
98 lines not on this page · raw message
Signature · 1 line
1.7.10.4
Steven Rostedt
maintainerRe: [for-next][PATCH 9/9] tracing: Update debugfs README file
In reply to Namhyung KimQuoted 9 lines · Namhyung Kim
kernel/trace/trace.c static const struct file_operations tracing_iter_fops = { · line 3349
Quoted 4 lines
kernel/trace/trace.c +4
-0+" example: echo do_fault:traceoff > set_ftrace_filter\n" +" echo do_trap:traceoff:3 > set_ftrace_filter\n" +" The first one will disable tracing everytime do_fault is hit\n" +" The second will disable tracing at most 3 times whne do_trap is hit\n" s/whne/when/
It's slightly confusing for me to read the "at most" part. Did you mean it by "The second will disable tracing after
do_traceis hit 3 times"?
Yeah, I wrote this on little sleep. I'll revise it a little. Thanks!
-- Steve
Quoted 10 lines
Quoted 10 lines
kernel/trace/trace.c +9
-0+" To remove trigger without count:\n" +" echo '!<function>:<trigger> > set_ftrace_filter\n" +" To remove trigger with a count:\n" +" echo '!<function>:<trigger>:0 > set_ftrace_filter\n" +" set_ftrace_notrace\t- echo function name in here to never trace.\n" +" accepts: func_full_name, *func_end, func_begin*, *func_middle*\n" +" modules: Can select a group via module command :mod:\n" + " Does not accept triggers\n" +#endif /* CONFIG_DYNAMIC_FTRACE */