ftrace: updates for tip
Subsystems (3)
Steven Rostedt
maintainercover ftrace: updates for tip
Ingo,
These patches are fixes for the wakeup tracer.
The first is a fix to the empty output you reported to me earlier.
The second is a fix for the ring buffer to not record updates if it is disabled, otherwised you can get empty traces there too.
The third is the wakeup change to record all tasks and not just realtime tasks. It adds a wakeup_rt tracer to record only rt tasks.
The forth is the addition of the scheduling wakeup and sched switch output to the wakeup tracer. This is very helpful data.
The last is probably the most important, and should go into 29 as well. I never reset the timestamps used to connect the trace stamps in the buffers. This can cause stale stamps to appear and explains some of the really bad timestamps in the buffer I've seen.
-- Steve
The following patches are in:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
branch: tip/devel
Steven Rostedt (5):
trace: do not disable wake up tracer on output of trace
ring-buffer: do not swap if recording is disabled
trace: separate out rt tasks from wakeup tracer
wakeup-tracer: show scheduling data in output
ring-buffer: reset timestamps when ring buffer is reset
----
kernel/trace/ring_buffer.c | 18 ++++++++ kernel/trace/trace.c | 2 +- kernel/trace/trace_sched_wakeup.c | 78 ++++++++++++++++++++----------------- 3 files changed, 61 insertions(+), 37 deletions(-)
Steven Rostedt
maintainerPATCH 1/5 trace: do not disable wake up tracer on output of trace
From: Steven Rostedt <[email protected]>
Impact: fix to erased trace output
To try not to have the outputing of a trace interfere with the wakeup tracer, it would disable tracing while the output was printing. But if a trace had started when it was disabled, it can show a partial trace. To try to solve this, on closing of the tracer, it would clear the trace buffer.
The latency tracers (wakeup and irqsoff) have two buffers. One for recording and one for holding the max trace that is printed. The clearing of the trace above should only affect the recording buffer. But for some reason it would move the erased trace to the print buffer. Probably due to a race with the closing of the trace and the saving ofhe max race.
The above is all pretty useless, and if the user does not want the printing of the trace to be traced itself, then the user can manual disable tracing. This patch removes all the code that tries to keep the output of the tracer from modifying the trace.
--- kernel/trace/trace_sched_wakeup.c | 33 ++------------------------------- 1 files changed, 2 insertions(+), 31 deletions(-)
kernel/trace/trace_sched_wakeup.c +2 -31
@@ -262,12 +262,6 @@ out:atomic_dec (&wakeup_trace->data[cpu]->disabled); } -/* -* save_tracer_enabled is used to save the state of the tracer_enabled -* variable when we disable it when we open a trace output file. -*/ -static int save_tracer_enabled; -static void start_wakeup_tracer (struct trace_array *tr) {int ret;
@@ -306,13 +300,10 @@ static void start_wakeup_tracer(struct trace_array *tr)register_ftrace_function (&trace_ops); -if (tracing_is_enabled ()) { +if (tracing_is_enabled ()) tracer_enabled =1 ; - save_tracer_enabled =1 ; - }else { +else tracer_enabled =0 ; - save_tracer_enabled =0 ; - }return ; fail_deprobe_wake_new:
@@ -324,7 +315,6 @@ fail_deprobe:static void stop_wakeup_tracer (struct trace_array *tr) { tracer_enabled =0 ; - save_tracer_enabled =0 ;unregister_ftrace_function (&trace_ops);unregister_trace_sched_switch (probe_wakeup_sched_switch);unregister_trace_sched_wakeup_new (probe_wakeup);
@@ -350,28 +340,11 @@ static void wakeup_tracer_start(struct trace_array *tr) {wakeup_reset (tr); tracer_enabled =1 ; - save_tracer_enabled =1 ; }static void wakeup_tracer_stop (struct trace_array *tr) { tracer_enabled =0 ; - save_tracer_enabled =0 ; -} - -static void wakeup_tracer_open (struct trace_iterator *iter) -{ -/* stop the trace while dumping */ - tracer_enabled =0 ; -} - -static void wakeup_tracer_close (struct trace_iterator *iter) -{ -/* forget about any processes we were recording */ -if (save_tracer_enabled) { -wakeup_reset (iter->tr); - tracer_enabled =1 ; - } } static struct tracer wakeup_tracer __read_mostly =
@@ -381,8 +354,6 @@ static struct tracer wakeup_tracer __read_mostly = .reset = wakeup_tracer_reset, .start = wakeup_tracer_start, .stop = wakeup_tracer_stop, - .open = wakeup_tracer_open, - .close = wakeup_tracer_close, .print_max =1 , #ifdef CONFIG_FTRACE_SELFTEST .selftest = trace_selftest_startup_wakeup,
Signature · 3 lines
1.5.6.5 --
Re: [PATCH 1/5] trace: do not disable wake up tracer on output of trace
Quoted 10 lines · Steven Rostedt
From: Steven Rostedt <[email protected]>
Impact: fix to erased trace output
To try not to have the outputing of a trace interfere with the wakeup tracer, it would disable tracing while the output was printing. But if a trace had started when it was disabled, it can show a partial trace. To try to solve this, on closing of the tracer, it would clear the trace buffer.
Quoted 6 lines
The latency tracers (wakeup and irqsoff) have two buffers. One for recording and one for holding the max trace that is printed. The clearing of the trace above should only affect the recording buffer. But for some reason it would move the erased trace to the print buffer. Probably due to a race with the closing of the trace and the saving ofhe max race.
hm, that race needs to be fixed then.
Quoted 4 lines
The above is all pretty useless, and if the user does not want the printing of the trace to be traced itself, then the user can manual disable tracing. This patch removes all the code that tries to keep the output of the tracer from modifying the trace.
printing of the trace should not be traced. I cannot imagine any usage mode where that would be interesting - and i can imagine a ton of cases where users would be confused/distracted by the tracer in essence zapping their measurement by replacing it with some uninteresting 'the tracer itself has wakeup delays' data.
auto-disabling latency tracing while the trace is being output is essential. Measurement should never impact the workload that is being measured.
We should fix that race instead.
Ingo
Re: [PATCH 1/5] trace: do not disable wake up tracer on output of trace
Quoted 3 lines · Ingo Molnar
auto-disabling latency tracing while the trace is being output is essential. Measurement should never impact the workload that is being measured.
On the other hand - the latencies caused by tracing output itself should be quite low, correct? If not then that needs fixing too. So i've picked up your change - lets see how useful the result is.
But "if it hurts, disable tracing before reading the tracing result" method you expressed is really not acceptable.
Ingo
Steven Rostedt
maintainerRe: [PATCH 1/5] trace: do not disable wake up tracer on output of trace
In reply to Ingo MolnarQuoted 33 lines · Ingo Molnar
Quoted 10 lines · Steven Rostedt
From: Steven Rostedt <[email protected]>
Impact: fix to erased trace output
To try not to have the outputing of a trace interfere with the wakeup tracer, it would disable tracing while the output was printing. But if a trace had started when it was disabled, it can show a partial trace. To try to solve this, on closing of the tracer, it would clear the trace buffer.
Quoted 6 lines
The latency tracers (wakeup and irqsoff) have two buffers. One for recording and one for holding the max trace that is printed. The clearing of the trace above should only affect the recording buffer. But for some reason it would move the erased trace to the print buffer. Probably due to a race with the closing of the trace and the saving ofhe max race.
hm, that race needs to be fixed then.
Quoted 4 lines
The above is all pretty useless, and if the user does not want the printing of the trace to be traced itself, then the user can manual disable tracing. This patch removes all the code that tries to keep the output of the tracer from modifying the trace.
printing of the trace should not be traced. I cannot imagine any usage mode where that would be interesting - and i can imagine a ton of cases where users would be confused/distracted by the tracer in essence zapping their measurement by replacing it with some uninteresting 'the tracer itself has wakeup delays' data.
Quoted 5 lines
auto-disabling latency tracing while the trace is being output is essential. Measurement should never impact the workload that is being measured.
We should fix that race instead.
Well actually, I don't see the output causing any issues. That's another point. Should we work on adding all this complex code for something that might not every cause an issues? I added it because it did cause an issue with the irqs off traces, although I'm not sure that is true any more either. I just added it to the wakeup tracer just to be consistent.
With the new trace_off switch that disables the ring buffer, and the stopping of the swap buffers with the switch off, I do not think that this code is even necessary anymore.
I think I'll rip out the code from irqsoff and see if it is still an issue there. If not, I think we can get rid of it.
-- Steve
Re: [PATCH 1/5] trace: do not disable wake up tracer on output of trace
Quoted 6 lines · Steven Rostedt
Well actually, I don't see the output causing any issues. That's another point. Should we work on adding all this complex code for something that might not every cause an issues? I added it because it did cause an issue with the irqs off traces, although I'm not sure that is true any more either. I just added it to the wakeup tracer just to be consistent.
Quoted 6 lines
With the new
trace_offswitch that disables the ring buffer, and the stopping of the swap buffers with the switch off, I do not think that this code is even necessary anymore.I think I'll rip out the code from irqsoff and see if it is still an issue there. If not, I think we can get rid of it.
agreed - if the output itself does not disturb max-latency tracing in practice then i'm all for ripping it out.
Ingo
Steven Rostedt
maintainerPATCH 2/5 ring-buffer: do not swap if recording is disabled
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
If the ring buffer recording has been disabled. Do not let swapping of ring buffers occur. Simply return -EAGAIN.
--- kernel/trace/ring_buffer.c | 15 +++++++++++++++ kernel/trace/trace.c | 2 +- 2 files changed, 16 insertions(+), 1 deletions(-)
kernel/trace/ring_buffer.c +15 -0
@@ -2266,9 +2266,24 @@ int ring_buffer_swap_cpu(struct ring_buffer *buffer_a,if (buffer_a->pages != buffer_b->pages)return -EINVAL ; +if (ring_buffer_flags !=RB_BUFFERS_ON ) +return -EAGAIN ; + +if (atomic_read (&buffer_a->record_disabled)) +return -EAGAIN ; + +if (atomic_read (&buffer_b->record_disabled)) +return -EAGAIN ; + cpu_buffer_a = buffer_a->buffers[cpu]; cpu_buffer_b = buffer_b->buffers[cpu]; +if (atomic_read (&cpu_buffer_a->record_disabled)) +return -EAGAIN ; + +if (atomic_read (&cpu_buffer_b->record_disabled)) +return -EAGAIN ; +/* * We can't do a synchronize_sched here because this * function can be called in atomic context.
kernel/trace/trace.c +1 -1
@@ -415,7 +415,7 @@ update_max_tr_single(struct trace_array *tr, struct task_struct *tsk, int cpu)ftrace_enable_cpu (); -WARN_ON_ONCE (ret); +WARN_ON_ONCE (ret && ret != -EAGAIN );__update_max_tr (tr, tsk, cpu);__raw_spin_unlock (&ftrace_max_lock);
Signature · 3 lines
1.5.6.5 --
Steven Rostedt
maintainerPATCH 3/5 trace: separate out rt tasks from wakeup tracer
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Impact: add option to trace all tasks or just RT tasks
The current wakeup tracer only traces RT task wakeups. This is fine for those interested in wake up timings of RT tasks, but it is useless for those that are interested in the causes of long wakeups for non RT tasks.
This patch creates a "wakeup_rt" to implement the tracing of just RT tasks (as the current "wakeup" does). And makes "wakeup" now trace all tasks as an average developer would expect.
--- kernel/trace/trace_sched_wakeup.c | 34 ++++++++++++++++++++++++++++++++-- 1 files changed, 32 insertions(+), 2 deletions(-)
kernel/trace/trace_sched_wakeup.c +32 -2
@@ -25,6 +25,7 @@ static int __read_mostly tracer_enabled;static struct task_struct *wakeup_task;static int wakeup_cpu;static unsigned wakeup_prio =-1 ; +static int wakeup_rt;static raw_spinlock_t wakeup_lock = (raw_spinlock_t )__RAW_SPIN_LOCK_UNLOCKED;
@@ -224,7 +225,7 @@ probe_wakeup(struct rq *rq, struct task_struct *p, int success)tracing_record_cmdline (p);tracing_record_cmdline (current); -if (likely (!rt_task (p)) || +if ((wakeup_rt && !rt_task (p)) || p->prio >= wakeup_prio || p->prio >= current->prio)return ;
@@ -321,7 +322,7 @@ static void stop_wakeup_tracer(struct trace_array *tr)unregister_trace_sched_wakeup (probe_wakeup); } -static int wakeup_tracer_init (struct trace_array *tr) +static int __wakeup_tracer_init (struct trace_array *tr) { tracing_max_latency =0 ; wakeup_trace = tr;
@@ -329,6 +330,18 @@ static int wakeup_tracer_init(struct trace_array *tr)return 0 ; } +static int wakeup_tracer_init (struct trace_array *tr) +{ + wakeup_rt =0 ; +return __wakeup_tracer_init (tr); +} + +static int wakeup_rt_tracer_init (struct trace_array *tr) +{ + wakeup_rt =1 ; +return __wakeup_tracer_init (tr); +} +static void wakeup_tracer_reset (struct trace_array *tr) {stop_wakeup_tracer (tr);
@@ -360,6 +373,19 @@ static struct tracer wakeup_tracer __read_mostly =#endif }; +static struct tracer wakeup_rt_tracer__read_mostly = +{ + .name ="wakeup_rt" , + .init = wakeup_rt_tracer_init, + .reset = wakeup_tracer_reset, + .start = wakeup_tracer_start, + .stop = wakeup_tracer_stop, + .print_max =1 , +#ifdef CONFIG_FTRACE_SELFTEST + .selftest = trace_selftest_startup_wakeup, +#endif +}; +__init static int init_wakeup_tracer (void ) {int ret;
@@ -368,6 +394,10 @@ __init static int init_wakeup_tracer(void)if (ret)return ret; + ret =register_tracer (&wakeup_rt_tracer); +if (ret) +return ret; +return 0 ; }device_initcall (init_wakeup_tracer);
Signature · 3 lines
1.5.6.5 --
Steven Rostedt
maintainerPATCH 4/5 wakeup-tracer: show scheduling data in output
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Impact: better data for wakeup tracer
This patch adds the wakeup and schedule calls that are used by the scheduler tracer to make the wakeup tracer more readable.
--- kernel/trace/trace_sched_wakeup.c | 11 ++++++++--- 1 files changed, 8 insertions(+), 3 deletions(-)
kernel/trace/trace_sched_wakeup.c +8 -3
@@ -153,6 +153,7 @@ probe_wakeup_sched_switch(struct rq *rq, struct task_struct *prev, goto out_unlock;trace_function (wakeup_trace, data,CALLER_ADDR1 ,CALLER_ADDR2 , flags, pc); +tracing_sched_switch_trace (wakeup_trace, data, prev, next, flags, pc);/* * usecs conversion is slow so we try to delay the conversion
@@ -214,6 +215,7 @@ static void wakeup_reset(struct trace_array *tr)static void probe_wakeup (struct rq *rq,struct task_struct *p,int success) { +struct trace_array_cpu *data;int cpu =smp_processor_id ();unsigned long flags;long disabled;
@@ -253,9 +255,12 @@ probe_wakeup(struct rq *rq, struct task_struct *p, int success)local_save_flags (flags); - wakeup_trace->data[wakeup_cpu]->preempt_timestamp =ftrace_now (cpu); -trace_function (wakeup_trace, wakeup_trace->data[wakeup_cpu], -CALLER_ADDR1 ,CALLER_ADDR2 , flags, pc); + data = wakeup_trace->data[wakeup_cpu]; + data->preempt_timestamp =ftrace_now (cpu); +tracing_sched_wakeup_trace (wakeup_trace, data, p, current, + flags, pc); +trace_function (wakeup_trace, data,CALLER_ADDR1 ,CALLER_ADDR2 , + flags, pc); out_locked:__raw_spin_unlock (&wakeup_lock);
Signature · 3 lines
1.5.6.5 --
Steven Rostedt
maintainerPATCH 5/5 ring-buffer: reset timestamps when ring buffer is reset
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
Impact: fix bad times of recent resets
The ring buffer needs to reset its timestamps when reseting of the buffer, otherwise the timestamps are stale and might be used to calculate times in the buffer causing funny timestamps to appear.
--- kernel/trace/ring_buffer.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-)
kernel/trace/ring_buffer.c +3 -0
@@ -2166,6 +2166,9 @@ rb_reset_cpu(struct ring_buffer_per_cpu *cpu_buffer) cpu_buffer->overrun =0 ; cpu_buffer->entries =0 ; + + cpu_buffer->write_stamp =0 ; + cpu_buffer->read_stamp =0 ; }/**
Signature · 3 lines
1.5.6.5 --
Quoted 3 lines · Steven Rostedt
Ingo,
These patches are fixes for the wakeup tracer.
That was quick! :-)
Quoted 8 lines
The first is a fix to the empty output you reported to me earlier.
The second is a fix for the ring buffer to not record updates if it is disabled, otherwised you can get empty traces there too.
The third is the wakeup change to record all tasks and not just realtime tasks. It adds a
wakeup_rttracer to record only rt tasks.
Quoted 7 lines
The forth is the addition of the scheduling wakeup and sched switch output to the wakeup tracer. This is very helpful data.
The last is probably the most important, and should go into 29 as well. I never reset the timestamps used to connect the trace stamps in the buffers. This can cause stale stamps to appear and explains some of the really bad timestamps in the buffer I've seen.
okay, i've pulled your fixes into tip/tracing/ftrace and i've cherry-picked the timestamp fix over into tracing/urgent.
Thanks,
Ingo