tracing/ftrace: add missing wake-up on some callsites
PATCH 1/1 tracing/ftrace: add missing wake-up on some callsites
Impact: fix unwaken pipe
Now that we use a common wakeup infrastructure, we must append a wakeup on few callsites which lack it or tasks reading trace_pipe will not be awaken when events come on few tracers.
--- kernel/trace/trace.c | 2 ++ kernel/trace/trace_branch.c | 2 ++ kernel/trace/trace_hw_branches.c | 2 ++ 3 files changed, 6 insertions(+), 0 deletions(-)
kernel/trace/trace.c +2 -0
@@ -3055,6 +3055,8 @@ int trace_vprintk(unsigned long ip, int depth, const char *fmt, va_list args) out:preempt_enable_notrace (); +trace_wake_up (); +return len; }EXPORT_SYMBOL_GPL (trace_vprintk);
kernel/trace/trace_branch.c +2 -0
@@ -78,6 +78,8 @@ probe_likely_condition(struct ftrace_branch_data *f, int val, int expect) out:atomic_dec (&tr->data[cpu]->disabled);local_irq_restore (flags); + +trace_wake_up (); } static inline
kernel/trace/trace_hw_branches.c +2 -0
@@ -202,6 +202,8 @@ void trace_hw_branch(u64 from, u64 to) out:atomic_dec (&tr->data[cpu]->disabled);local_irq_restore (irq1); + +trace_wake_up (); }static void trace_bts_at (const struct bts_trace *trace,void *at)
Signature · 1 line
1.6.1
Ingo Molnar
AppliedQuoted 6 lines · Frederic Weisbecker
Impact: fix unwaken pipe
Now that we use a common wakeup infrastructure, we must append a wakeup on few callsites which lack it or tasks reading
trace_pipewill not be awaken when events come on few tracers.
Quoted 8 lines
Cc Steven Rostedt <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/trace/trace.c | 2 ++ kernel/trace/trace_branch.c | 2 ++ kernel/trace/trace_hw_branches.c | 2 ++ 3 files changed, 6 insertions(+), 0 deletions(-)
Applied to tip:tracing/ftrace, thanks Frederic!
Ingo
Steven Rostedt
maintainerQuoted 46 lines · Frederic Weisbecker
Impact: fix unwaken pipe
Now that we use a common wakeup infrastructure, we must append a wakeup on few callsites which lack it or tasks reading
trace_pipewill not be awaken when events come on few tracers.Cc Steven Rostedt <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/trace/trace.c | 2 ++ kernel/trace/trace_branch.c | 2 ++ kernel/trace/trace_hw_branches.c | 2 ++ 3 files changed, 6 insertions(+), 0 deletions(-)kernel/trace/trace.c +2
-0@@ -3055,6 +3055,8 @@ int trace_vprintk(unsigned long ip, int depth, const char *fmt, va_list args) out:preempt_enable_notrace (); +trace_wake_up (); +return len; }EXPORT_SYMBOL_GPL (trace_vprintk);kernel/trace/trace_branch.c +2
-0@@ -78,6 +78,8 @@ probe_likely_condition(struct ftrace_branch_data *f, int val, int expect) out:atomic_dec (&tr->data[cpu]->disabled);local_irq_restore (flags); + +trace_wake_up (); } static inlinekernel/trace/trace_hw_branches.c +0
-0
kernel/trace/trace_hw_branches.c void trace_hw_branch(u64 from, u64 to) · line 209
Quoted 8 lines
kernel/trace/trace_hw_branches.c +2
-0@@ -202,6 +202,8 @@ void trace_hw_branch(u64 from, u64 to) out:atomic_dec (&tr->data[cpu]->disabled);local_irq_restore (irq1); + +trace_wake_up (); }static void trace_bts_at (const struct bts_trace *trace,void *at)
Ah, we don't wake up purposely on these three places. ftrace_printk is meant to be called anywhere (including the scheduler). And the branch tracers are also allowed to be called anywhere (they usually are).
Calling "wake_up" from any of these can easily cause a dead lock with the run queue lock, because all three can be called from with in the scheduler.
Sorry, but I have to NACK this change.
-- Steve
Steven Rostedt
maintainerQuoted 63 lines · Steven Rostedt
Quoted 46 lines · Frederic Weisbecker
Impact: fix unwaken pipe
Now that we use a common wakeup infrastructure, we must append a wakeup on few callsites which lack it or tasks reading
trace_pipewill not be awaken when events come on few tracers.Cc Steven Rostedt <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/trace/trace.c | 2 ++ kernel/trace/trace_branch.c | 2 ++ kernel/trace/trace_hw_branches.c | 2 ++ 3 files changed, 6 insertions(+), 0 deletions(-)kernel/trace/trace.c +2
-0@@ -3055,6 +3055,8 @@ int trace_vprintk(unsigned long ip, int depth, const char *fmt, va_list args) out:preempt_enable_notrace (); +trace_wake_up (); +return len; }EXPORT_SYMBOL_GPL (trace_vprintk);kernel/trace/trace_branch.c +2
-0@@ -78,6 +78,8 @@ probe_likely_condition(struct ftrace_branch_data *f, int val, int expect) out:atomic_dec (&tr->data[cpu]->disabled);local_irq_restore (flags); + +trace_wake_up (); } static inlinekernel/trace/trace_hw_branches.c +0
-0kernel/trace/trace_hw_branches.c void trace_hw_branch(u64 from, u64 to) · line 209
Quoted 8 lines
kernel/trace/trace_hw_branches.c +2
-0@@ -202,6 +202,8 @@ void trace_hw_branch(u64 from, u64 to) out:atomic_dec (&tr->data[cpu]->disabled);local_irq_restore (irq1); + +trace_wake_up (); }static void trace_bts_at (const struct bts_trace *trace,void *at)Ah, we don't wake up purposely on these three places.
ftrace_printkis meant to be called anywhere (including the scheduler). And the branch tracers are also allowed to be called anywhere (they usually are).
Quoted 5 lines
Calling "
wake_up" from any of these can easily cause a dead lock with the run queue lock, because all three can be called from with in the scheduler.Sorry, but I have to NACK this change.
Perhaps we could add these callsites back, but we would need to update trace_wake_up.
Have trace_wake_up set a flag instead, and add a tracepoint around the scheduler (outside the grabbing of runqueue locks), that will have a callback to the tracing code. That call back can perform the wakeups.
How does that sound?
-- Steve
Quoted 63 lines · Steven Rostedt
Quoted 46 lines · Frederic Weisbecker
Impact: fix unwaken pipe
Now that we use a common wakeup infrastructure, we must append a wakeup on few callsites which lack it or tasks reading
trace_pipewill not be awaken when events come on few tracers.Cc Steven Rostedt <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/trace/trace.c | 2 ++ kernel/trace/trace_branch.c | 2 ++ kernel/trace/trace_hw_branches.c | 2 ++ 3 files changed, 6 insertions(+), 0 deletions(-)kernel/trace/trace.c +2
-0@@ -3055,6 +3055,8 @@ int trace_vprintk(unsigned long ip, int depth, const char *fmt, va_list args) out:preempt_enable_notrace (); +trace_wake_up (); +return len; }EXPORT_SYMBOL_GPL (trace_vprintk);kernel/trace/trace_branch.c +2
-0@@ -78,6 +78,8 @@ probe_likely_condition(struct ftrace_branch_data *f, int val, int expect) out:atomic_dec (&tr->data[cpu]->disabled);local_irq_restore (flags); + +trace_wake_up (); } static inlinekernel/trace/trace_hw_branches.c +0
-0kernel/trace/trace_hw_branches.c void trace_hw_branch(u64 from, u64 to) · line 209
Quoted 8 lines
kernel/trace/trace_hw_branches.c +2
-0@@ -202,6 +202,8 @@ void trace_hw_branch(u64 from, u64 to) out:atomic_dec (&tr->data[cpu]->disabled);local_irq_restore (irq1); + +trace_wake_up (); }static void trace_bts_at (const struct bts_trace *trace,void *at)Ah, we don't wake up purposely on these three places.
ftrace_printkis meant to be called anywhere (including the scheduler). And the branch tracers are also allowed to be called anywhere (they usually are).
Quoted 5 lines
Calling "
wake_up" from any of these can easily cause a dead lock with the run queue lock, because all three can be called from with in the scheduler.Sorry, but I have to NACK this change.
And fortunately you NACK, I didn't realized how dangerous it can be. I will send a patch to make these branch tracers use the old polling wake up.
But another solution should be found for ftrace_printk() since it doesn't necessarily rely on any tracer.
Quoted 1 line
-- Steve
Quoted 73 lines · Steven Rostedt
Quoted 63 lines · Steven Rostedt
Quoted 46 lines · Frederic Weisbecker
Impact: fix unwaken pipe
Now that we use a common wakeup infrastructure, we must append a wakeup on few callsites which lack it or tasks reading
trace_pipewill not be awaken when events come on few tracers.Cc Steven Rostedt <[email protected]>Cc Arnaldo Carvalho de Melo <[email protected]>Signed-off-by Frederic Weisbecker <[email protected]>--- kernel/trace/trace.c | 2 ++ kernel/trace/trace_branch.c | 2 ++ kernel/trace/trace_hw_branches.c | 2 ++ 3 files changed, 6 insertions(+), 0 deletions(-)kernel/trace/trace.c +2
-0@@ -3055,6 +3055,8 @@ int trace_vprintk(unsigned long ip, int depth, const char *fmt, va_list args) out:preempt_enable_notrace (); +trace_wake_up (); +return len; }EXPORT_SYMBOL_GPL (trace_vprintk);kernel/trace/trace_branch.c +2
-0@@ -78,6 +78,8 @@ probe_likely_condition(struct ftrace_branch_data *f, int val, int expect) out:atomic_dec (&tr->data[cpu]->disabled);local_irq_restore (flags); + +trace_wake_up (); } static inlinekernel/trace/trace_hw_branches.c +0
-0kernel/trace/trace_hw_branches.c void trace_hw_branch(u64 from, u64 to) · line 209
Quoted 8 lines
kernel/trace/trace_hw_branches.c +2
-0@@ -202,6 +202,8 @@ void trace_hw_branch(u64 from, u64 to) out:atomic_dec (&tr->data[cpu]->disabled);local_irq_restore (irq1); + +trace_wake_up (); }static void trace_bts_at (const struct bts_trace *trace,void *at)Ah, we don't wake up purposely on these three places.
ftrace_printkis meant to be called anywhere (including the scheduler). And the branch tracers are also allowed to be called anywhere (they usually are).Quoted 5 lines
Calling "
wake_up" from any of these can easily cause a dead lock with the run queue lock, because all three can be called from with in the scheduler.Sorry, but I have to NACK this change.
Quoted 8 lines
Perhaps we could add these callsites back, but we would need to update
trace_wake_up.Have
trace_wake_upset a flag instead, and add a tracepoint around the scheduler (outside the grabbing of runqueue locks), that will have a callback to the tracing code. That call back can perform the wakeups.How does that sound?
That sounds good but only for these particular tracers I guess.
Quoted 2 lines
-- Steve
Steven Rostedt
maintainerQuoted 4 lines · Frederic Weisbecker
Quoted 4 lines
Perhaps we could add these callsites back, but we would need to update
trace_wake_up.
Quoted 8 lines
Quoted 5 lines
Have
trace_wake_upset a flag instead, and add a tracepoint around the scheduler (outside the grabbing of runqueue locks), that will have a callback to the tracing code. That call back can perform the wakeups.How does that sound?
That sounds good but only for these particular tracers I guess.
OK, what about making a trace_delay_wake_up()?
-- Steve
Quoted 10 lines · Steven Rostedt
Quoted 8 lines · Frederic Weisbecker
Quoted 8 lines
Perhaps we could add these callsites back, but we would need to update
trace_wake_up.Have
trace_wake_upset a flag instead, and add a tracepoint around the scheduler (outside the grabbing of runqueue locks), that will have a callback to the tracing code. That call back can perform the wakeups.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
How does that sound?
That sounds good but only for these particular tracers I guess.
OK, what about making a
trace_delay_wake_up()?
Which would send a delayed work to wake up?
Quoted 2 lines
-- Steve
Steven Rostedt
maintainerQuoted 14 lines · Frederic Weisbecker
Quoted 13 lines · Steven Rostedt
Quoted 11 lines · Frederic Weisbecker
Quoted 9 lines
Perhaps we could add these callsites back, but we would need to update
trace_wake_up.Have
trace_wake_upset a flag instead, and add a tracepoint around the scheduler (outside the grabbing of runqueue locks), that will have a callback to the tracing code. That call back can perform the wakeups.How does that sound?
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
That sounds good but only for these particular tracers I guess.
OK, what about making a
trace_delay_wake_up()?Which would send a delayed work to wake up?
No, I was thinking that trace_delay_wake_up() would be called by these dangerous call sites. Then a per_cpu flag could be set. We could have a trace point in the scheduler code that is outside holding a runqueue lock, and this trace point would call a trace function that will clear the per cpu flag, and then call trace_wake_up().
-- Steve
Quoted 22 lines · Steven Rostedt
Quoted 19 lines · Frederic Weisbecker
Quoted 16 lines · Steven Rostedt
Quoted 4 lines · Frederic Weisbecker
Quoted 4 lines
Perhaps we could add these callsites back, but we would need to update
trace_wake_up.Quoted 8 lines
Quoted 5 lines
Have
trace_wake_upset a flag instead, and add a tracepoint around the scheduler (outside the grabbing of runqueue locks), that will have a callback to the tracing code. That call back can perform the wakeups.How does that sound?
That sounds good but only for these particular tracers I guess.
OK, what about making a
trace_delay_wake_up()?
Quoted 8 lines
Quoted 1 line
Which would send a delayed work to wake up?
No, I was thinking that
trace_delay_wake_up()would be called by these dangerous call sites. Then aper_cpuflag could be set. We could have a trace point in the scheduler code that is outside holding a runqueue lock, and this trace point would call a trace function that will clear the per cpu flag, and then calltrace_wake_up().
Oh yes, sounds nice!
Quoted 2 lines
-- Steve
Quoted 8 lines · Steven Rostedt
Quoted 1 line
Which would send a delayed work to wake up?
No, I was thinking that
trace_delay_wake_up()would be called by these dangerous call sites. Then aper_cpuflag could be set. We could have a trace point in the scheduler code that is outside holding a runqueue lock, and this trace point would call a trace function that will clear the per cpu flag, and then calltrace_wake_up().
No, that's very roundabout and ugly. If we putting a tracepoint there we might as well put real scheduler code there that looks for such a flag. But i'm not convinced we need a flag ...
Ingo
Steven Rostedt
maintainerQuoted 12 lines · Ingo Molnar
Quoted 8 lines · Steven Rostedt
Quoted 1 line
Which would send a delayed work to wake up?
No, I was thinking that
trace_delay_wake_up()would be called by these dangerous call sites. Then aper_cpuflag could be set. We could have a trace point in the scheduler code that is outside holding a runqueue lock, and this trace point would call a trace function that will clear the per cpu flag, and then calltrace_wake_up().
Quoted 3 lines
No, that's very roundabout and ugly. If we putting a tracepoint there we might as well put real scheduler code there that looks for such a flag. But i'm not convinced we need a flag ...
Just a suggestion. I was trying to keep the tracer from being an overhead. But what else would you suggest? Just having the scheduler call trace_wakeup?
-- Steve
Quoted 13 lines · Steven Rostedt
Quoted 11 lines · Ingo Molnar
Quoted 8 lines · Steven Rostedt
Quoted 1 line
Which would send a delayed work to wake up?
No, I was thinking that
trace_delay_wake_up()would be called by these dangerous call sites. Then aper_cpuflag could be set. We could have a trace point in the scheduler code that is outside holding a runqueue lock, and this trace point would call a trace function that will clear the per cpu flag, and then calltrace_wake_up().
Quoted 8 lines
Quoted 4 lines
No, that's very roundabout and ugly. If we putting a tracepoint there we might as well put real scheduler code there that looks for such a flag. But i'm not convinced we need a flag ...
Just a suggestion. I was trying to keep the tracer from being an overhead. But what else would you suggest? Just having the scheduler call
trace_wakeup?
i think we could use a TIF flag to trigger a wakeup at the return-to-userspace (or return-from-IRQ) stage or so?
Ingo
Quoted 11 lines · Steven Rostedt
Quoted 9 lines · Ingo Molnar
Quoted 7 lines · Steven Rostedt
Quoted 1 line
Which would send a delayed work to wake up?
No, I was thinking that
trace_delay_wake_up()would be called by these dangerous call sites. Then aper_cpuflag could be set. We could have a trace point in the scheduler code that is outside holding a runqueue lock, and this trace point would call a trace function that will clear the per cpu flag, and then calltrace_wake_up().
Quoted 7 lines
Quoted 3 lines
No, that's very roundabout and ugly. If we putting a tracepoint there we might as well put real scheduler code there that looks for such a flag. But i'm not convinced we need a flag ...
Just a suggestion. I was trying to keep the tracer from being an overhead. But what else would you suggest? Just having the scheduler call
trace_wakeup?
Actually, Systemtap(LTTng too?) also has same problem. Currently, we're using a periodical timer to wake the reader process up. I assume if we can put a tracepoint at the beginning of schedule(), we can share it.
Thank you,
Quoted 8 lines
-- Steve
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 7 lines
Masami Hiramatsu Software Engineer Hitachi Computer Products (America) Inc. Software Solutions Division e-mail: [email protected]