function-graph: memory leak and race fixes
Steven Rostedt
maintainerIngo,
During testing of the -rt patch, Luis found a bug with the function profiler (not in 30-rc and I sent the patch for you already), that he could trigger a crash when enabling and disabling the function profiler in a loop.
During my investigation of this, I tried enabling and disabling the function graph tracer in a loop and was able to also crash the kernel. This crash was not the same as the profiler crash. Looking into this I found a few problems with the enabling of the function graph tracer.
One was a memory leak, the other two were races on SMP machines. The races were more likely the cause of the crashes I saw. With these patches applied, I no longer can produce the crash.
I've also Cc'd the stable team since these bugs also exist in 2.6.29.
Please pull the latest tip/tracing/urgent tree, which can be found at:
git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-2.6-trace.git
tip/tracing/urgent
Steven Rostedt (3):
function-graph: only allocate init tasks if it was not already done
function-graph: enable the stack after initialization of other variables
function-graph: add memory barriers for accessing task's ret_stack
----
kernel/trace/ftrace.c | 23 +++++++++++++++-------- kernel/trace/trace_functions_graph.c | 6 ++++++ 2 files changed, 21 insertions(+), 8 deletions(-)
Steven Rostedt
maintainerPATCH 1/3 function-graph: only allocate init tasks if it was not already done
From: Steven Rostedt <[email protected]>
When the function graph tracer is enabled, it calls the initialization needed for the init tasks that would be called on all created tasks.
The problem is that this is called every time the function graph tracer is enabled, and the ret_stack is allocated for the idle tasks each time. Thus, the old ret_stack is lost and a memory leak is created.
This is also dangerous because if an interrupt happened on another CPU with the init task and the ret_stack is replaced, we then lose all the return pointers for the interrupt, and a crash would take place.
[ Impact: fix memory leak and possible crash due to race ]
--- kernel/trace/ftrace.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-)
kernel/trace/ftrace.c +4 -2
@@ -2643,8 +2643,10 @@ static int start_graph_tracing(void)return -ENOMEM ;/* The cpu_boot init_task->ret_stack will never be freed */ -for_each_online_cpu (cpu) -ftrace_graph_init_task (idle_task (cpu)); +for_each_online_cpu (cpu) { +if (!idle_task (cpu)->ret_stack) +ftrace_graph_init_task (idle_task (cpu)); + } do { ret = alloc_retstack_tasklist(ret_stack_list);
Signature · 3 lines
1.6.3.1 --
Steven Rostedt
maintainerPATCH 2/3 function-graph: enable the stack after initialization of other variables
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The function graph tracer checks if the task_struct has ret_stack defined to know if it is OK or not to use it. The initialization is done for all tasks by one process, but the idle tasks use the same initialization used by new tasks.
If an interrupt happens on an idle task that just had the ret_stack created, but before the rest of the initialization took place, then we can corrupt the return address of the functions.
This patch moves the setting of the task_struct's ret_stack to after the other variables have been initialized.
[ Impact: prevent kernel panic on idle task when starting function graph ]
--- kernel/trace/ftrace.c | 9 +++++++-- kernel/trace/trace_functions_graph.c | 6 ++++++ 2 files changed, 13 insertions(+), 2 deletions(-)
kernel/trace/ftrace.c +7 -2
@@ -2739,15 +2739,20 @@ void unregister_ftrace_graph(void)void ftrace_graph_init_task (struct task_struct *t) {if (atomic_read (&ftrace_graph_active)) { - t->ret_stack =kmalloc (FTRACE_RETFUNC_DEPTH +struct ftrace_ret_stack *ret_stack; + + ret_stack =kmalloc (FTRACE_RETFUNC_DEPTH *sizeof (struct ftrace_ret_stack ),GFP_KERNEL ); -if (!t->ret_stack) +if (!ret_stack)return ; t->curr_ret_stack =-1 ;atomic_set (&t->tracing_graph_pause,0 );atomic_set (&t->trace_overrun,0 ); t->ftrace_timestamp =0 ; +/* make curr_ret_stack visable before we add the ret_stack */ +smp_wmb (); + t->ret_stack = ret_stack; }else t->ret_stack =NULL ; }
kernel/trace/trace_functions_graph.c +6 -0
@@ -65,6 +65,12 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, int *depth)if (!current->ret_stack)return -EBUSY ; +/* +* We must make sure the ret_stack is tested before we read +* anything else. +*/ +smp_rmb (); +/* The return trace stack is full */ if (current->curr_ret_stack ==FTRACE_RETFUNC_DEPTH -1 ) {atomic_inc (¤t->trace_overrun);
Signature · 3 lines
1.6.3.1 --
Re: [PATCH 2/3] function-graph: enable the stack after initialization of other variables
Quoted 39 lines · Steven Rostedt
From: Steven Rostedt <[email protected]>
The function graph tracer checks if the
task_structhasret_stackdefined to know if it is OK or not to use it. The initialization is done for all tasks by one process, but the idle tasks use the same initialization used by new tasks.If an interrupt happens on an idle task that just had the
ret_stackcreated, but before the rest of the initialization took place, then we can corrupt the return address of the functions.This patch moves the setting of the task_struct's
ret_stackto after the other variables have been initialized.[ Impact: prevent kernel panic on idle task when starting function graph ]
Signed-off-by Steven Rostedt <[email protected]>--- kernel/trace/ftrace.c | 9 +++++++-- kernel/trace/trace_functions_graph.c | 6 ++++++ 2 files changed, 13 insertions(+), 2 deletions(-)kernel/trace/ftrace.c +4
-2@@ -2739,15 +2739,20 @@ void unregister_ftrace_graph(void)void ftrace_graph_init_task (struct task_struct *t) {if (atomic_read (&ftrace_graph_active)) { - t->ret_stack =kmalloc (FTRACE_RETFUNC_DEPTH +struct ftrace_ret_stack *ret_stack; + + ret_stack =kmalloc (FTRACE_RETFUNC_DEPTH *sizeof (struct ftrace_ret_stack ),GFP_KERNEL ); -if (!t->ret_stack) +if (!ret_stack)return ;
kernel/trace/ftrace.c void unregister_ftrace_graph(void) · line 2758
Quoted 10 lines
kernel/trace/ftrace.c +3
-0t->curr_ret_stack =-1 ;atomic_set (&t->tracing_graph_pause,0 );atomic_set (&t->trace_overrun,0 ); t->ftrace_timestamp =0 ; +/* make curr_ret_stack visable before we add the ret_stack */ +smp_wmb (); + t->ret_stack = ret_stack; }else t->ret_stack =NULL ; }
Thanks for fixing this.
kernel/trace/trace_functions_graph.c ftrace_push_return_trace(unsigned long ret, unsigned long func, int *depth) · line 72
Quoted 13 lines
kernel/trace/trace_functions_graph.c +5
-0@@ -65,6 +65,12 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, int *depth)if (!current->ret_stack)return -EBUSY ; +/* +* We must make sure the ret_stack is tested before we read +* anything else. +*/ +smp_rmb ();
Isn't this part a too much costly for very traced function?
Quoted 8 lines
kernel/trace/trace_functions_graph.c +1
-0+/* The return trace stack is full */ if (current->curr_ret_stack ==FTRACE_RETFUNC_DEPTH -1 ) {atomic_inc (¤t->trace_overrun);Signature · 3 lines
1.6.3.1 --
Steven Rostedt
maintainerRe: [PATCH 2/3] function-graph: enable the stack after initialization of other variables
Quoted 8 lines · Frederic Weisbecker
Quoted 8 lines
kernel/trace/trace_functions_graph.c +0
-0@@ -65,6 +65,12 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, int *depth)if (!current->ret_stack)return -EBUSY ;
Quoted 8 lines
kernel/trace/trace_functions_graph.c ftrace_push_return_trace(unsigned long ret, unsigned long func, int *depth) · line 72
Quoted 5 lines
kernel/trace/trace_functions_graph.c +5
-0+/* +* We must make sure the ret_stack is tested before we read +* anything else. +*/ +smp_rmb ();Isn't this part a too much costly for very traced function?
I was thinking that, but otherwise we still have the problem. It is a read barrier which I don't think is as costly as a write or full barrier. But because we have the if statement, perhaps a "read_barrier_depends" can do?
[ added Paul McKenney because he's good with barriers ]
-- Steve
Quoted 12 lines
Quoted 10 lines
kernel/trace/trace_functions_graph.c +1
-0+/* The return trace stack is full */ if (current->curr_ret_stack ==FTRACE_RETFUNC_DEPTH -1 ) {atomic_inc (¤t->trace_overrun);Signature · 3 lines
1.6.3.1 --
Re: [PATCH 2/3] function-graph: enable the stack after initialization of other variables
Quoted 19 lines · Steven Rostedt
Quoted 8 lines · Frederic Weisbecker
Quoted 8 lines
kernel/trace/trace_functions_graph.c +0
-0@@ -65,6 +65,12 @@ ftrace_push_return_trace(unsigned long ret, unsigned long func, int *depth)if (!current->ret_stack)return -EBUSY ;Quoted 8 lines
kernel/trace/trace_functions_graph.c ftrace_push_return_trace(unsigned long ret, unsigned long func, int *depth) · line 72
Quoted 5 lines
kernel/trace/trace_functions_graph.c +5
-0+/* +* We must make sure the ret_stack is tested before we read +* anything else. +*/ +smp_rmb ();Isn't this part a too much costly for very traced function?
Quoted 3 lines
I was thinking that, but otherwise we still have the problem. It is a read barrier which I don't think is as costly as a write or full barrier. But because we have the if statement, perhaps a "
read_barrier_depends" can do?
Although this would work on some CPU architectures (x86, s390, Power, and perhaps a few others), it would break on those weakly ordered systems that do not respect control dependencies.
One approach would be to use another level of indirection. If this new pointer is NULL, you return EBUSY. Make this new pointer reference the updated fields (tracing_graph_pause, trace_overrun, and so on). This could point into the same data structure -- it is not necessary to actually allocate a new chunk of memory.
Then you can use the existing rcu_assign_pointer() and rcu_dereference() primitives.
Does this approach help at all?
Thanx, Paul
Quoted 17 lines
[ added Paul McKenney because he's good with barriers ]
-- Steve
Quoted 12 lines
Quoted 10 lines
kernel/trace/trace_functions_graph.c +1
-0+/* The return trace stack is full */ if (current->curr_ret_stack ==FTRACE_RETFUNC_DEPTH -1 ) {atomic_inc (¤t->trace_overrun);Signature · 3 lines
1.6.3.1 --
Steven Rostedt
maintainerPATCH 3/3 function-graph: add memory barriers for accessing tasks ret_stack
In reply to Steven RostedtFrom: Steven Rostedt <[email protected]>
The code that handles the tasks ret_stack allocation for every task assumes that only an interrupt can cause issues (even though interrupts are disabled).
In reality, the code is allocating the ret_stack for tasks that may be running on other CPUs and there are not efficient memory barriers to handle this case.
[ Impact: prevent crash due to using of uninitialized ret_stack variables ]
--- kernel/trace/ftrace.c | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-)
kernel/trace/ftrace.c +4 -4
@@ -2580,12 +2580,12 @@ static int alloc_retstack_tasklist(struct ftrace_ret_stack **ret_stack_list) }if (t->ret_stack ==NULL ) { - t->curr_ret_stack =-1 ; -/* Make sure IRQs see the -1 first: */ -barrier (); - t->ret_stack = ret_stack_list[start++];atomic_set (&t->tracing_graph_pause,0 );atomic_set (&t->trace_overrun,0 ); + t->curr_ret_stack =-1 ; +/* Make sure the tasks see the -1 first: */ +smp_wmb (); + t->ret_stack = ret_stack_list[start++]; } }while_each_thread (g, t);
Signature · 3 lines
1.6.3.1 --