ftrace: Add 'function-fork' trace option (v2)
cover ftrace: Add 'function-fork' trace option (v2)
Hello,
This patchset add 'function-fork' option to function tracer which makes pid filter to be inherited like 'event-fork' does. During the test, I found a bug of pid filter on an instance directory. The patch 1 fixes it and maybe it should go to the stable tree.
The function-fork option is disabled by default as event-fork does, but we might consider changing the default since it seems to be more natural from an user's perspective IMHO.
v2 changes)
* use ftrace_clear_pids() (Steve)
* add Ack from Masami
The code is also available at 'ftrace/function-fork-v2' branch on
git://git.kernel.org/pub/scm/linux/kernel/git/namhyung/linux-perf.git
Thanks,
Namhyung
Namhyung Kim (4):
ftrace: Fix function pid filter on instances
ftrace: Add 'function-fork' trace option
selftests: ftrace: Add -l/--logdir option
selftests: ftrace: Add a testcase for function PID filter
kernel/trace/ftrace.c | 46 ++++++++++ kernel/trace/trace.c | 6 +- kernel/trace/trace.h | 8 +- tools/testing/selftests/ftrace/ftracetest | 5 ++ .../ftrace/test.d/ftrace/func-filter-pid.tc | 98 ++++++++++++++++++++++ 5 files changed, 161 insertions(+), 2 deletions(-) create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/func-filter-pid.tc
Signature · 1 line
2.12.2
PATCH 1/4 ftrace: Fix function pid filter on instances
When function tracer has a pid filter, it adds a probe to sched_switch to track if current task can be ignored. The probe checks the ftrace_ignore_pid from current tr to filter tasks. But it misses to delete the probe when removing an instance so that it can cause a crash due to the invalid tr pointer (use-after-free).
This is easily reproducible with the following:
# cd /sys/kernel/debug/tracing # mkdir instances/buggy # echo $$ > instances/buggy/set_ftrace_pid # rmdir instances/buggy
============================================================================
BUG: KASAN: use-after-free in ftrace_filter_pid_sched_switch_probe+0x3d/0x90 Read of size 8 by task kworker/0:1/17 CPU: 0 PID: 17 Comm: kworker/0:1 Tainted: G B 4.11.0-rc3 #198 Call Trace: dump_stack+0x68/0x9f kasan_object_err+0x21/0x70 kasan_report.part.1+0x22b/0x500 ? ftrace_filter_pid_sched_switch_probe+0x3d/0x90 kasan_report+0x25/0x30 __asan_load8+0x5e/0x70 ftrace_filter_pid_sched_switch_probe+0x3d/0x90 ? fpid_start+0x130/0x130 __schedule+0x571/0xce0 ...
To fix it, use ftrace_pid_reset() to unregister the probe. As instance_rmdir() already updated ftrace codes, it can just free the filter safely.
--- kernel/trace/ftrace.c | 9 +++++++++ kernel/trace/trace.c | 1 + kernel/trace/trace.h | 2 ++ 3 files changed, 12 insertions(+)
kernel/trace/ftrace.c +9 -0
@@ -5610,6 +5610,15 @@ static void clear_ftrace_pids(struct trace_array *tr)trace_free_pid_list (pid_list); } +void ftrace_clear_pids (struct trace_array *tr) +{ +mutex_lock (&ftrace_lock); + +clear_ftrace_pids (tr); + +mutex_unlock (&ftrace_lock); +} +static void ftrace_pid_reset (struct trace_array *tr) {mutex_lock (&ftrace_lock);
kernel/trace/trace.c +1 -0
@@ -7485,6 +7485,7 @@ static int instance_rmdir(const char *name)tracing_set_nop (tr);event_trace_del_tracer (tr); +ftrace_clear_pids (tr);ftrace_destroy_function_files (tr);tracefs_remove_recursive (tr->dir);free_trace_buffers (tr);
kernel/trace/trace.h +2 -0
@@ -897,6 +897,7 @@ void ftrace_init_tracefs(struct trace_array *tr, struct dentry *d_tracer);void ftrace_init_tracefs_toplevel (struct trace_array *tr,struct dentry *d_tracer);int init_function_trace (void ); +void ftrace_clear_pids (struct trace_array *tr);#else static inline int ftrace_trace_task (struct trace_array *tr) {
@@ -916,6 +917,7 @@ static inline void ftrace_reset_array_ops(struct trace_array *tr) { }static inline void ftrace_init_tracefs (struct trace_array *tr,struct dentry *d) { }static inline void ftrace_init_tracefs_toplevel (struct trace_array *tr,struct dentry *d) { }static inline int init_function_trace (void ) {return 0 ; } +static inline void ftrace_clear_pids (struct trace_array *tr) { }/* ftace_func_t type is not defined, use macro instead of static inline */ #define ftrace_init_array_ops (tr, func) do { } while (0)#endif /* CONFIG_FUNCTION_TRACER */
Signature · 1 line
2.12.2
Re: [PATCH 1/4] ftrace: Fix function pid filter on instances
Quoted 29 lines · On Mon, 17 Apr 2017 11:44:27 +0900 Namhyung Kim <[email protected]> wrote:
When function tracer has a pid filter, it adds a probe to
sched_switchto track if current task can be ignored. The probe checks theftrace_ignore_pidfrom current tr to filter tasks. But it misses to delete the probe when removing an instance so that it can cause a crash due to the invalid tr pointer (use-after-free).This is easily reproducible with the following:
# cd /sys/kernel/debug/tracing # mkdir instances/buggy # echo $$ > instances/buggy/set_ftrace_pid # rmdir instances/buggy============================================================================
BUG: KASAN: use-after-free in ftrace_filter_pid_sched_switch_probe+0x3d/0x90 Read of size 8 by task kworker/0:1/17 CPU: 0 PID: 17 Comm: kworker/0:1 Tainted: G B 4.11.0-rc3 #198 Call Trace: dump_stack+0x68/0x9f kasan_object_err+0x21/0x70 kasan_report.part.1+0x22b/0x500 ? ftrace_filter_pid_sched_switch_probe+0x3d/0x90 kasan_report+0x25/0x30 __asan_load8+0x5e/0x70 ftrace_filter_pid_sched_switch_probe+0x3d/0x90 ? fpid_start+0x130/0x130 __schedule+0x571/0xce0 ...
Quoted 1 line
To fix it, use
ftrace_pid_reset()to unregister the probe. As
^^^^^^^^^^^^^^^^ ftrace_clear_pids()?
Quoted 2 lines
instance_rmdir()already updated ftrace codes, it can just free the filter safely.
And following explanation may not need.
The code looks good to me.
Thanks,
Quoted 64 lines
Fixes 0c8916c34203 ("tracing: Add rmdir to remove multibuffer instances")Signed-off-by Namhyung Kim <[email protected]>--- kernel/trace/ftrace.c | 9 +++++++++ kernel/trace/trace.c | 1 + kernel/trace/trace.h | 2 ++ 3 files changed, 12 insertions(+)kernel/trace/ftrace.c +9
-0@@ -5610,6 +5610,15 @@ static void clear_ftrace_pids(struct trace_array *tr)trace_free_pid_list (pid_list); } +void ftrace_clear_pids (struct trace_array *tr) +{ +mutex_lock (&ftrace_lock); + +clear_ftrace_pids (tr); + +mutex_unlock (&ftrace_lock); +} +static void ftrace_pid_reset (struct trace_array *tr) {mutex_lock (&ftrace_lock);kernel/trace/trace.c +1
-0@@ -7485,6 +7485,7 @@ static int instance_rmdir(const char *name)tracing_set_nop (tr);event_trace_del_tracer (tr); +ftrace_clear_pids (tr);ftrace_destroy_function_files (tr);tracefs_remove_recursive (tr->dir);free_trace_buffers (tr);kernel/trace/trace.h +2
-0@@ -897,6 +897,7 @@ void ftrace_init_tracefs(struct trace_array *tr, struct dentry *d_tracer);void ftrace_init_tracefs_toplevel (struct trace_array *tr,struct dentry *d_tracer);int init_function_trace (void ); +void ftrace_clear_pids (struct trace_array *tr);#else static inline int ftrace_trace_task (struct trace_array *tr) {@@ -916,6 +917,7 @@ static inline void ftrace_reset_array_ops(struct trace_array *tr) { }static inline void ftrace_init_tracefs (struct trace_array *tr,struct dentry *d) { }static inline void ftrace_init_tracefs_toplevel (struct trace_array *tr,struct dentry *d) { }static inline int init_function_trace (void ) {return 0 ; } +static inline void ftrace_clear_pids (struct trace_array *tr) { }/* ftace_func_t type is not defined, use macro instead of static inline */ #define ftrace_init_array_ops (tr, func) do { } while (0)#endif /* CONFIG_FUNCTION_TRACER */ Signature · 1 line
2.12.2
Signature · 1 line
Masami Hiramatsu <[email protected]>
Re: [PATCH 1/4] ftrace: Fix function pid filter on instances
Hi Masami,
Quoted 32 lines · Masami Hiramatsu
Quoted 29 lines · On Mon, 17 Apr 2017 11:44:27 +0900 Namhyung Kim <[email protected]> wrote:
When function tracer has a pid filter, it adds a probe to
sched_switchto track if current task can be ignored. The probe checks theftrace_ignore_pidfrom current tr to filter tasks. But it misses to delete the probe when removing an instance so that it can cause a crash due to the invalid tr pointer (use-after-free).This is easily reproducible with the following:
# cd /sys/kernel/debug/tracing # mkdir instances/buggy # echo $$ > instances/buggy/set_ftrace_pid # rmdir instances/buggy============================================================================
BUG: KASAN: use-after-free in ftrace_filter_pid_sched_switch_probe+0x3d/0x90 Read of size 8 by task kworker/0:1/17 CPU: 0 PID: 17 Comm: kworker/0:1 Tainted: G B 4.11.0-rc3 #198 Call Trace: dump_stack+0x68/0x9f kasan_object_err+0x21/0x70 kasan_report.part.1+0x22b/0x500 ? ftrace_filter_pid_sched_switch_probe+0x3d/0x90 kasan_report+0x25/0x30 __asan_load8+0x5e/0x70 ftrace_filter_pid_sched_switch_probe+0x3d/0x90 ? fpid_start+0x130/0x130 __schedule+0x571/0xce0 ...
Quoted 2 lines
Quoted 1 line
To fix it, use
ftrace_pid_reset()to unregister the probe. As^^^^^^^^^^^^^^^^ ftrace_clear_pids()?
Oops, forgot to update the changelog, sorry.
Quoted 9 lines
Quoted 3 lines
instance_rmdir()already updated ftrace codes, it can just free the filter safely.And following explanation may not need.
The code looks good to me.
Reviewed-by Masami Hiramatsu <[email protected]>
Thanks!
Namhyung
Quoted 68 lines
Quoted 62 lines
Fixes 0c8916c34203 ("tracing: Add rmdir to remove multibuffer instances")Signed-off-by Namhyung Kim <[email protected]>--- kernel/trace/ftrace.c | 9 +++++++++ kernel/trace/trace.c | 1 + kernel/trace/trace.h | 2 ++ 3 files changed, 12 insertions(+)kernel/trace/ftrace.c +9
-0@@ -5610,6 +5610,15 @@ static void clear_ftrace_pids(struct trace_array *tr)trace_free_pid_list (pid_list); } +void ftrace_clear_pids (struct trace_array *tr) +{ +mutex_lock (&ftrace_lock); + +clear_ftrace_pids (tr); + +mutex_unlock (&ftrace_lock); +} +static void ftrace_pid_reset (struct trace_array *tr) {mutex_lock (&ftrace_lock);kernel/trace/trace.c +1
-0@@ -7485,6 +7485,7 @@ static int instance_rmdir(const char *name)tracing_set_nop (tr);event_trace_del_tracer (tr); +ftrace_clear_pids (tr);ftrace_destroy_function_files (tr);tracefs_remove_recursive (tr->dir);free_trace_buffers (tr);kernel/trace/trace.h +2
-1@@ -897,6 +897,7 @@ void ftrace_init_tracefs(struct trace_array *tr, struct dentry *d_tracer);void ftrace_init_tracefs_toplevel (struct trace_array *tr,struct dentry *d_tracer);int init_function_trace (void ); +void ftrace_clear_pids (struct trace_array *tr);#else static inline int ftrace_trace_task (struct trace_array *tr) {@@ -916,6 +917,7 @@ static inline void ftrace_reset_array_ops(struct trace_array *tr) { }static inline void ftrace_init_tracefs (struct trace_array *tr,struct dentry *d) { }static inline void ftrace_init_tracefs_toplevel (struct trace_array *tr,struct dentry *d) { }static inline int init_function_trace (void ) {return 0 ; } +static inline void ftrace_clear_pids (struct trace_array *tr) { }/* ftace_func_t type is not defined, use macro instead of static inline */ #define ftrace_init_array_ops (tr, func) do { } while (0)#endif /* CONFIG_FUNCTION_TRACER */ --Quoted 2 lines
2.12.2
--
Masami Hiramatsu <[email protected]>
Signature · 2 lines
Thanks, Namhyung
PATCH 2/4 ftrace: Add 'function-fork' trace option
In reply to Namhyung KimThe function-fork option is same as event-fork that it tracks task fork/exit and set the pid filter properly. This can be useful if user wants to trace selected tasks including their children only.
--- kernel/trace/ftrace.c | 37 +++++++++++++++++++++++++++++++++++++ kernel/trace/trace.c | 5 ++++- kernel/trace/trace.h | 6 +++++- 3 files changed, 46 insertions(+), 2 deletions(-)
kernel/trace/ftrace.c +37 -0
@@ -5587,6 +5587,43 @@ ftrace_filter_pid_sched_switch_probe(void *data, bool preempt, trace_ignore_this_task(pid_list, next)); } +static void +ftrace_pid_follow_sched_process_fork (void *data, +struct task_struct *self, +struct task_struct *task) +{ +struct trace_pid_list *pid_list; +struct trace_array *tr = data; + + pid_list =rcu_dereference_sched (tr->function_pids); +trace_filter_add_remove_task (pid_list, self, task); +} + +static void +ftrace_pid_follow_sched_process_exit (void *data,struct task_struct *task) +{ +struct trace_pid_list *pid_list; +struct trace_array *tr = data; + + pid_list =rcu_dereference_sched (tr->function_pids); +trace_filter_add_remove_task (pid_list,NULL , task); +} + +void ftrace_pid_follow_fork (struct trace_array *tr,bool enable) +{ +if (enable) { +register_trace_sched_process_fork (ftrace_pid_follow_sched_process_fork, + tr); +register_trace_sched_process_exit (ftrace_pid_follow_sched_process_exit, + tr); + }else { +unregister_trace_sched_process_fork (ftrace_pid_follow_sched_process_fork, + tr); +unregister_trace_sched_process_exit (ftrace_pid_follow_sched_process_exit, + tr); + } +} +static void clear_ftrace_pids (struct trace_array *tr) {struct trace_pid_list *pid_list;
kernel/trace/trace.c +4 -1
@@ -257,7 +257,7 @@ unsigned long long ns2usecs(u64 nsec)/* trace_flags that are default zero for instances */ #define ZEROED_TRACE_FLAGS \ - TRACE_ITER_EVENT_FORK + (TRACE_ITER_EVENT_FORK | TRACE_ITER_FUNC_FORK)/* * The global_trace is the descriptor that holds the top-level tracing
@@ -4205,6 +4205,9 @@ int set_tracer_flag(struct trace_array *tr, unsigned int mask, int enabled)if (mask ==TRACE_ITER_EVENT_FORK )trace_event_follow_fork (tr, enabled); +if (mask ==TRACE_ITER_FUNC_FORK ) +ftrace_pid_follow_fork (tr, enabled); +if (mask ==TRACE_ITER_OVERWRITE ) {ring_buffer_change_overwrite (tr->trace_buffer.buffer, enabled);#ifdef CONFIG_TRACER_MAX_TRACE
kernel/trace/trace.h +5 -1
@@ -898,6 +898,7 @@ void ftrace_init_tracefs_toplevel(struct trace_array *tr, struct dentry *d_tracer);int init_function_trace (void );void ftrace_clear_pids (struct trace_array *tr); +void ftrace_pid_follow_fork (struct trace_array *tr,bool enable);#else static inline int ftrace_trace_task (struct trace_array *tr) {
@@ -918,6 +919,7 @@ static inline void ftrace_init_tracefs(struct trace_array *tr, struct dentry *d)static inline void ftrace_init_tracefs_toplevel (struct trace_array *tr,struct dentry *d) { }static inline int init_function_trace (void ) {return 0 ; }static inline void ftrace_clear_pids (struct trace_array *tr) { } +static inline void ftrace_pid_follow_fork (struct trace_array *tr,bool enable) { }/* ftace_func_t type is not defined, use macro instead of static inline */ #define ftrace_init_array_ops (tr, func) do { } while (0)#endif /* CONFIG_FUNCTION_TRACER */
@@ -991,11 +993,13 @@ extern int trace_get_user(struct trace_parser *parser, const char __user *ubuf,#ifdef CONFIG_FUNCTION_TRACER # define FUNCTION_FLAGS \ - C(FUNCTION, "function-trace"), + C(FUNCTION, "function-trace"), \ + C(FUNC_FORK, "function-fork"),# define FUNCTION_DEFAULT_FLAGS TRACE_ITER_FUNCTION#else # define FUNCTION_FLAGS # define FUNCTION_DEFAULT_FLAGS 0UL +# define TRACE_ITER_FUNC_FORK 0UL#endif #ifdef CONFIG_STACKTRACE
Signature · 1 line
2.12.2
PATCH 3/4 selftests: ftrace: Add -l/--logdir option
In reply to Namhyung KimIn my virtual machine setup, running ftracetest failed on creating LOG_DIR on a read-only filesystem. It'd be convenient to provide an option to specify a different directory as log directory.
--- tools/testing/selftests/ftrace/ftracetest | 5 +++++ 1 file changed, 5 insertions(+)
tools/testing/selftests/ftrace/ftracetest +5 -0
@@ -16,6 +16,7 @@ echo " -k|--keep Keep passed test logs" echo " -v|--verbose Increase verbosity of test messages" echo " -vv Alias of -v -v (Show all results in stdout)" echo " -d|--debug Debug mode (trace all shell commands)" +echo " -l|--logdir <dir> Save logs on the <dir>" exit $1 }
@@ -64,6 +65,10 @@ parse_opts() { # opts DEBUG=1 shift 1 ;; + --logdir|-l) + LOG_DIR=$2 + shift 2 + ;; *.tc) if [ -f "$1" ]; then OPT_TEST_CASES="$OPT_TEST_CASES `abspath $1`"
Signature · 1 line
2.12.2
PATCH 4/4 selftests: ftrace: Add a testcase for function PID filter
In reply to Namhyung KimLike event pid filtering test, add function pid filtering test with the new "function-fork" option. It also tests it on an instance directory so that it can verify the bug related pid filtering on instances.
--- .../ftrace/test.d/ftrace/func-filter-pid.tc | 98 ++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 tools/testing/selftests/ftrace/test.d/ftrace/func-filter-pid.tc
tools/testing/selftests/ftrace/test.d/ftrace/func-filter-pid.tc new file +98 -0
@@ -0,0 +1,98 @@ +#!/bin/sh +# description: ftrace - function pid filters + +# Make sure that function pid matching filter works. +# Also test it on an instance directory + +if ! grep -q function available_tracers; then + echo "no function tracer configured" + exit_unsupported +fi + +if [ ! -f set_ftrace_pid ]; then + echo "set_ftrace_pid not found? Is function tracer not set?" + exit_unsupported +fi + +if [ ! -f set_ftrace_filter ]; then + echo "set_ftrace_filter not found? Is function tracer not set?" + exit_unsupported +fi + +read PID _ < /proc/self/stat + +# default value of function-fork option +orig_value=`grep function-fork trace_options` + +do_reset() { + reset_tracer + clear_trace + enable_tracing + echo > set_ftrace_filter + echo > set_ftrace_pid + + echo $orig_value > trace_options +} + +fail() { # msg + do_reset + echo $1 + exit $FAIL +} + +yield() { + ping localhost -c 1 || sleep .001 || usleep 1 || sleep 1 +} + +do_test() { + disable_tracing + + echo do_execve* > set_ftrace_filter + echo *do_fork >> set_ftrace_filter + + echo $PID > set_ftrace_pid + echo function > current_tracer + + # don't allow children to be traced + echo nofunction-fork > trace_options + + enable_tracing + yield + + count_pid=`cat trace | grep -v ^# | grep $PID | wc -l` + count_other=`cat trace | grep -v ^# | grep -v $PID | wc -l` + + # count_other should be 0 + if [ $count_pid -eq 0 -o $count_other -ne 0 ]; then + fail "PID filtering not working?" + fi + + disable_tracing + clear_trace + + # allow children to be traced + echo function-fork > trace_options + + enable_tracing + yield + + count_pid=`cat trace | grep -v ^# | grep $PID | wc -l` + count_other=`cat trace | grep -v ^# | grep -v $PID | wc -l` + + # count_other should NOT be 0 + if [ $count_pid -eq 0 -o $count_other -eq 0 ]; then + fail "PID filtering not following fork?" + fi +} + +do_test + +mkdir instances/foo +cd instances/foo +do_test +cd ../../ +rmdir instances/foo + +do_reset + +exit 0
Signature · 1 line
2.12.2
Steven Rostedt
maintainerQuoted 6 lines · On Mon, 17 Apr 2017 11:44:26 +0900 Namhyung Kim <[email protected]> wrote:
Hello,
This patchset add 'function-fork' option to function tracer which makes pid filter to be inherited like 'event-fork' does. During the test, I found a bug of pid filter on an instance directory. The patch 1 fixes it and maybe it should go to the stable tree.
Hmm, are the other patches dependent on it?
I think I may just push it separately to Linus now, but the other patches will be on my devel branch which will not be abased off of this fix. Will that break too much? I just cherry-picked a patch from my urgent branch as it required to be on my devel branch and go to Linus.
Hmm, I may be able to make a separate branch with this. I have to see how much it conflicts with my current development.
-- Steve
Hi Steve,
Sorry for little late,
Quoted 11 lines · Steven Rostedt
Quoted 6 lines · On Mon, 17 Apr 2017 11:44:26 +0900 Namhyung Kim <[email protected]> wrote:
Hello,
This patchset add 'function-fork' option to function tracer which makes pid filter to be inherited like 'event-fork' does. During the test, I found a bug of pid filter on an instance directory. The patch 1 fixes it and maybe it should go to the stable tree.
Hmm, are the other patches dependent on it?
Nop, but there will be a small clash on trace.h for the declaration.
Quoted 5 lines
I think I may just push it separately to Linus now, but the other patches will be on my devel branch which will not be abased off of this fix. Will that break too much? I just cherry-picked a patch from my urgent branch as it required to be on my devel branch and go to Linus.
I don't think it breaks much.
Quoted 3 lines
Hmm, I may be able to make a separate branch with this. I have to see how much it conflicts with my current development.
Thanks,
Namhyung
Steven Rostedt
maintainerQuoted 10 lines · On Wed, 19 Apr 2017 09:27:28 +0900 Namhyung Kim <[email protected]> wrote:
Hi Steve,
Sorry for little late,
Quoted 5 lines · Steven Rostedt
Quoted 2 lines · On Mon, 17 Apr 2017 11:44:26 +0900 Namhyung Kim <[email protected]> wrote:
Hello,
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
This patchset add 'function-fork' option to function tracer which makes pid filter to be inherited like 'event-fork' does. During the test, I found a bug of pid filter on an instance directory. The patch 1 fixes it and maybe it should go to the stable tree.
Hmm, are the other patches dependent on it?
Nop, but there will be a small clash on
trace.hfor the declaration.
Yep, I push up a merge with mainline with my linux-next branch to cover the conflicts. I'll let Linus know about it too when I do my pull request in the merge window.
Quoted 8 lines
Quoted 6 lines
I think I may just push it separately to Linus now, but the other patches will be on my devel branch which will not be abased off of this fix. Will that break too much? I just cherry-picked a patch from my urgent branch as it required to be on my devel branch and go to Linus.
I don't think it breaks much.
Except that your test triggers the bug it uncovered ;-)
-- Steve
Quoted 4 lines
Quoted 4 lines
Hmm, I may be able to make a separate branch with this. I have to see how much it conflicts with my current development.