tracing: Dump either the oops's cpu source or all cpus buffers
Subsystems (4)
PATCH 1/1 tracing: Dump either the oops's cpu source or all cpus buffers
The ftrace_dump_on_oops kernel parameter, sysctl and sysrq let one dump every cpu buffers when an oops or panic happens.
It's nice when you have few cpus but it may take ages if have many, plus you miss the real origin of the problem in all the cpu traces.
Sometimes, all you need is to dump the cpu buffer that triggered the opps, most of the time it is our main interest.
This patch modifies ftrace_dump_on_oops to handle this choice.
The ftrace_dump_on_oops kernel parameter, when it comes alone, has the same behaviour than before. But ftrace_dump_on_oops=orig_cpu will only dump the buffer of the cpu that oops'ed.
Similarly, sysctl kernel.ftrace_dump_on_oops=1 and
echo 1 > /proc/sys/kernel/ftrace_dump_on_oops keep their previous behaviour. But setting 2 jumps into cpu origin dump mode.
--- Documentation/kernel-parameters.txt | 6 +++- Documentation/trace/ftrace.txt | 6 ++- drivers/char/sysrq.c | 2 +- include/linux/ftrace.h | 4 ++- include/linux/kernel.h | 11 ++++++- kernel/trace/trace.c | 53 +++++++++++++++++++++++++++-------- 6 files changed, 63 insertions(+), 19 deletions(-)
Documentation/kernel-parameters.txt +5 -1
@@ -789,8 +789,12 @@ and is between 256 and 4096 characters. It is defined in the file as early as possible in order to facilitate early boot debugging. - ftrace_dump_on_oops + ftrace_dump_on_oops[=orig_cpu] [FTRACE] will dump the trace buffers on oops. + If no parameter is passed, ftrace will dump + every cpu buffers, but if you pass orig_cpu, it will + dump only the buffer of the cpu that triggered the + oops. ftrace_filter=[function-list] [FTRACE] Limit the functions traced by the function
Documentation/trace/ftrace.txt +4 -2
@@ -1337,12 +1337,14 @@ ftrace_dump_on_oops must be set. To set ftrace_dump_on_oops, one can either use the sysctl function or set it via the proc system interface. - sysctl kernel.ftrace_dump_on_oops=1 + sysctl kernel.ftrace_dump_on_oops=n or - echo 1 > /proc/sys/kernel/ftrace_dump_on_oops + echo n > /proc/sys/kernel/ftrace_dump_on_oops +If n = 1, ftrace will dump every cpu buffers, if n = 2 ftrace will +only dump the buffer of the cpu that triggered the oops. Here's an example of such a dump after a null pointer dereference in a kernel module:
drivers/char/sysrq.c +1 -1
@@ -289,7 +289,7 @@ static struct sysrq_key_op sysrq_showstate_blocked_op = {static void sysrq_ftrace_dump (int key,struct tty_struct *tty) { -ftrace_dump (); +ftrace_dump (DUMP_ALL ); }static struct sysrq_key_op sysrq_ftrace_dump_op = { .handler = sysrq_ftrace_dump,
include/linux/ftrace.h +3 -1
@@ -492,7 +492,9 @@ static inline int test_tsk_trace_graph(struct task_struct *tsk)return tsk->trace &TSK_TRACE_FL_GRAPH ; } -extern int ftrace_dump_on_oops; +enum ftrace_dump_mode ; + +extern enum ftrace_dump_mode ftrace_dump_on_oops;#ifdef CONFIG_PREEMPT #define INIT_TRACE_RECURSION .trace_recursion = 0,
include/linux/kernel.h +9 -2
@@ -490,6 +490,13 @@ static inline void tracing_off(void) { }static inline void tracing_off_permanent (void ) { }static inline int tracing_is_on (void ) {return 0 ; }#endif + +enum ftrace_dump_mode { +DUMP_NONE , +DUMP_ALL , +DUMP_ORIG , +}; +#ifdef CONFIG_TRACING extern void tracing_start (void );extern void tracing_stop (void );
@@ -571,7 +578,7 @@ __ftrace_vbprintk(unsigned long ip, const char *fmt, va_list ap);extern int __ftrace_vprintk (unsigned long ip,const char *fmt,va_list ap); -extern void ftrace_dump (void ); +extern void ftrace_dump (enum ftrace_dump_mode oops_dump_mode);#else static inline void ftrace_special (unsigned long arg1,unsigned long arg2,unsigned long arg3) { }
@@ -592,7 +599,7 @@ ftrace_vprintk(const char *fmt, va_list ap) {return 0 ; } -static inline void ftrace_dump (void ) { } +static inline void ftrace_dump (enum ftrace_dump_mode oops_dump_mode) { }#endif /* CONFIG_TRACING */ /*
kernel/trace/trace.c +41 -12
108 lines not on this page · raw message
Signature · 1 line
1.6.2.3
PATCH v2 1/1 tracing: Dump either the oops's cpu source or all cpus buffers
The ftrace_dump_on_oops kernel parameter, sysctl and sysrq let one dump every cpu buffers when an oops or panic happens.
It's nice when you have few cpus but it may take ages if you have many, plus you miss the real origin of the problem in all the cpu traces.
Sometimes, all you need is to dump the cpu buffer that triggered the oops, most of the time it is our main interest.
This patch modifies ftrace_dump_on_oops to handle this choice.
The ftrace_dump_on_oops kernel parameter, when it comes alone, has the same behaviour than before. But ftrace_dump_on_oops=orig_cpu will only dump the buffer of the cpu that oops'ed.
Similarly, sysctl kernel.ftrace_dump_on_oops=1 and
echo 1 > /proc/sys/kernel/ftrace_dump_on_oops keep their previous behaviour. But setting 2 jumps into cpu origin dump mode.
v2: Fix double setup
--- Documentation/kernel-parameters.txt | 6 +++- Documentation/trace/ftrace.txt | 6 +++- drivers/char/sysrq.c | 2 +- include/linux/ftrace.h | 4 ++- include/linux/kernel.h | 11 ++++++- kernel/trace/trace.c | 51 ++++++++++++++++++++++++++-------- 6 files changed, 61 insertions(+), 19 deletions(-)
Documentation/kernel-parameters.txt +5 -1
@@ -789,8 +789,12 @@ and is between 256 and 4096 characters. It is defined in the file as early as possible in order to facilitate early boot debugging. - ftrace_dump_on_oops + ftrace_dump_on_oops[=orig_cpu] [FTRACE] will dump the trace buffers on oops. + If no parameter is passed, ftrace will dump + every cpu buffers, but if you pass orig_cpu, it will + dump only the buffer of the cpu that triggered the + oops. ftrace_filter=[function-list] [FTRACE] Limit the functions traced by the function
Documentation/trace/ftrace.txt +4 -2
@@ -1337,12 +1337,14 @@ ftrace_dump_on_oops must be set. To set ftrace_dump_on_oops, one can either use the sysctl function or set it via the proc system interface. - sysctl kernel.ftrace_dump_on_oops=1 + sysctl kernel.ftrace_dump_on_oops=n or - echo 1 > /proc/sys/kernel/ftrace_dump_on_oops + echo n > /proc/sys/kernel/ftrace_dump_on_oops +If n = 1, ftrace will dump every cpu buffers, if n = 2 ftrace will +only dump the buffer of the cpu that triggered the oops. Here's an example of such a dump after a null pointer dereference in a kernel module:
drivers/char/sysrq.c +1 -1
@@ -289,7 +289,7 @@ static struct sysrq_key_op sysrq_showstate_blocked_op = {static void sysrq_ftrace_dump (int key,struct tty_struct *tty) { -ftrace_dump (); +ftrace_dump (DUMP_ALL ); }static struct sysrq_key_op sysrq_ftrace_dump_op = { .handler = sysrq_ftrace_dump,
include/linux/ftrace.h +3 -1
@@ -492,7 +492,9 @@ static inline int test_tsk_trace_graph(struct task_struct *tsk)return tsk->trace &TSK_TRACE_FL_GRAPH ; } -extern int ftrace_dump_on_oops; +enum ftrace_dump_mode ; + +extern enum ftrace_dump_mode ftrace_dump_on_oops;#ifdef CONFIG_PREEMPT #define INIT_TRACE_RECURSION .trace_recursion = 0,
include/linux/kernel.h +9 -2
@@ -490,6 +490,13 @@ static inline void tracing_off(void) { }static inline void tracing_off_permanent (void ) { }static inline int tracing_is_on (void ) {return 0 ; }#endif + +enum ftrace_dump_mode { +DUMP_NONE , +DUMP_ALL , +DUMP_ORIG , +}; +#ifdef CONFIG_TRACING extern void tracing_start (void );extern void tracing_stop (void );
@@ -571,7 +578,7 @@ __ftrace_vbprintk(unsigned long ip, const char *fmt, va_list ap);extern int __ftrace_vprintk (unsigned long ip,const char *fmt,va_list ap); -extern void ftrace_dump (void ); +extern void ftrace_dump (enum ftrace_dump_mode oops_dump_mode);#else static inline void ftrace_special (unsigned long arg1,unsigned long arg2,unsigned long arg3) { }
@@ -592,7 +599,7 @@ ftrace_vprintk(const char *fmt, va_list ap) {return 0 ; } -static inline void ftrace_dump (void ) { } +static inline void ftrace_dump (enum ftrace_dump_mode oops_dump_mode) { }#endif /* CONFIG_TRACING */ /*
kernel/trace/trace.c +39 -12
104 lines not on this page · raw message
Signature · 1 line
1.6.2.3
Quoted 13 lines · On Mon, 19 Apr 2010 03:15:40 +0200 Frederic Weisbecker wrote:
Documentation/kernel-parameters.txt | 6 +++- Documentation/trace/ftrace.txt | 6 +++- drivers/char/sysrq.c | 2 +- include/linux/ftrace.h | 4 ++- include/linux/kernel.h | 11 ++++++- kernel/trace/trace.c | 51 ++++++++++++++++++++++++++-------- 6 files changed, 61 insertions(+), 19 deletions(-)Documentation/kernel-parameters.txt +0
-0
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 795
Quoted 8 lines
Documentation/kernel-parameters.txt +3
-1@@ -789,8 +789,12 @@ and is between 256 and 4096 characters. It is defined in the file as early as possible in order to facilitate early boot debugging. - ftrace_dump_on_oops + ftrace_dump_on_oops[=orig_cpu] [FTRACE] will dump the trace buffers on oops. + If no parameter is passed, ftrace will dump + every cpu buffers, but if you pass orig_cpu, it will
buffers of all CPUs,
Documentation/kernel-parameters.txt and is between 256 and 4096 characters. It is defined in the file · line 796
Quoted 1 line
Documentation/kernel-parameters.txt +1
-0+ dump only the buffer of the cpu that triggered the
CPU
Quoted 11 lines
Documentation/kernel-parameters.txt +1
-0+ oops. ftrace_filter=[function-list] [FTRACE] Limit the functions traced by the functionDocumentation/trace/ftrace.txt +0
-0@@ -1337,12 +1337,14 @@ ftrace_dump_on_oops must be set. To set ftrace_dump_on_oops, one can either use the sysctl function or set it via the proc system interface.
Documentation/trace/ftrace.txt ftrace_dump_on_oops must be set. To set ftrace_dump_on_oops, one · line 1346
Quoted 10 lines
Documentation/trace/ftrace.txt +3
-2- sysctl kernel.ftrace_dump_on_oops=1 + sysctl kernel.ftrace_dump_on_oops=n or - echo 1 > /proc/sys/kernel/ftrace_dump_on_oops + echo n > /proc/sys/kernel/ftrace_dump_on_oops +If n = 1, ftrace will dump every cpu buffers, if n = 2 ftrace will
buffers of all CPUs,
Documentation/trace/ftrace.txt ftrace_dump_on_oops must be set. To set ftrace_dump_on_oops, one · line 1347
Quoted 1 line
Documentation/trace/ftrace.txt +1
-0+only dump the buffer of the cpu that triggered the oops.
CPU
Quoted 8 lines
Documentation/trace/ftrace.txt +0
-0Here's an example of such a dump after a null pointer dereference in a kernel module:kernel/trace/trace.c +0
-0
kernel/trace/trace.c static cpumask_var_t __read_mostly tracing_buffer_mask; · line 121
Quoted 6 lines
kernel/trace/trace.c +2
-1@@ -117,9 +117,12 @@ static cpumask_var_t __read_mostly tracing_buffer_mask;* * It is default off, but you can enable it with either specifying * "ftrace_dump_on_oops" in the kernel command line, or setting -* /proc/sys/kernel/ftrace_dump_on_oops to true. +* /proc/sys/kernel/ftrace_dump_on_oops +* Set 1 if you want to dump every cpu buffers
buffers of all CPUs
kernel/trace/trace.c static cpumask_var_t __read_mostly tracing_buffer_mask; · line 122
Quoted 1 line
kernel/trace/trace.c +1
-0+* Set 2 if you want to dump the buffer of the cpu that triggered oops
CPU
Quoted 19 lines
kernel/trace/trace.c +4
-4*/ -int ftrace_dump_on_oops; + +enum ftrace_dump_mode ftrace_dump_on_oops;static int tracing_set_tracer (const char *buf);@@ -4429,12 +4442,25 @@ static void __ftrace_dump(bool disable_tracing)/* don't look at user memory in panic mode */ trace_flags &= ~TRACE_ITER_SYM_USEROBJ ; -printk (KERN_TRACE "Dumping ftrace buffer:\n" ); -/* Simulate the iterator */ iter.tr = &global_trace; iter.trace = current_trace; - iter.cpu_file =TRACE_PIPE_ALL_CPU ; + +switch (oops_dump_mode) {
kernel/trace/trace.c static void __ftrace_dump(bool disable_tracing) · line 4459
Quoted 10 lines
kernel/trace/trace.c +10
-0+case DUMP_ALL : + iter.cpu_file =TRACE_PIPE_ALL_CPU ; +break ; +case DUMP_ORIG : + iter.cpu_file =raw_smp_processor_id (); +break ; +case DUMP_NONE : + goto out_enable; +default : +printk (KERN_TRACE "Bad dumping mode, switching to all cpu dump\n" );
CPUs
kernel/trace/trace.c static void __ftrace_dump(bool disable_tracing) · line 4466
Quoted 7 lines
kernel/trace/trace.c +4
-0+ iter.cpu_file =TRACE_PIPE_ALL_CPU ; + } + +printk (KERN_TRACE "Dumping ftrace buffer:\n" );/* * We need to stop all tracing on all CPUS to read the
---
~Randy
Quoted 9 lines · Randy Dunlap
Quoted 7 lines · On Mon, 19 Apr 2010 03:15:40 +0200 Frederic Weisbecker wrote:
Documentation/kernel-parameters.txt | 6 +++- Documentation/trace/ftrace.txt | 6 +++- drivers/char/sysrq.c | 2 +- include/linux/ftrace.h | 4 ++- include/linux/kernel.h | 11 ++++++- kernel/trace/trace.c | 51 ++++++++++++++++++++++++++-------- 6 files changed, 61 insertions(+), 19 deletions(-)
Frederic,
Send out v3 with Randy's updates and:
-- Steve
Quoted 10 lines
Quoted 10 lines
Documentation/kernel-parameters.txt +0
-0@@ -789,8 +789,12 @@ and is between 256 and 4096 characters. It is defined in the file as early as possible in order to facilitate early boot debugging.
From: Frederic Weisbecker <[email protected]>
Date: Sun, 18 Apr 2010 22:50:10 +0200
Quoted 15 lines
The
ftrace_dump_on_oopskernel parameter, sysctl and sysrq let one dump every cpu buffers when an oops or panic happens.It's nice when you have few cpus but it may take ages if have many, plus you miss the real origin of the problem in all the cpu traces.
Sometimes, all you need is to dump the cpu buffer that triggered the opps, most of the time it is our main interest.
This patch modifies
ftrace_dump_on_oopsto handle this choice.The
ftrace_dump_on_oopskernel parameter, when it comes alone, has the same behaviour than before. But ftrace_dump_on_oops=orig_cpu will only dump the buffer of the cpu that oops'ed.
Quoted 5 lines
Similarly, sysctl kernel.ftrace_dump_on_oops=1 and
echo 1 >/proc/sys/kernel/ftrace_dump_on_oopskeep their previous behaviour. But setting 2 jumps into cpu origin dump mode.Signed-off-by Frederic Weisbecker <[email protected]>
Thanks I was trying to figure out a way to make ftrace dump do exactly this over the past few days :-)