trace: Introduce objtrace trigger to trace the kernel object
Subsystems (3)
v8 cover trace: Introduce objtrace trigger to trace the kernel object
Introduce a method based on function tracer to trace any object and get the value of the object dynamically. the object can be obtained from the dynamic event (kprobe_event/uprobe_event) or the static event(tracepoint).
Usage:
When using the kprobe event, only need to set the objtrace(a new trigger), we can get the value of the object. The object is from the setting of the kprobe event.
For example:
For the function bio_add_page():
int bio_add_page(struct bio *bio, struct page *page,
unsigned int len, unsigned int offset)
Firstly, we can set the base of the object, thus the first string "arg1" stands for the value of the first parameter of this function bio_add_gage(),
# echo 'p bio_add_page arg1=$arg1' > ./kprobe_events
Secondly, we can get the value dynamically based on above object.
find the offset of the bi_size in struct bio:
$ gdb vmlinux (gdb) p &(((struct bio *)0)->bi_iter.bi_size) $1 = (unsigned int *) 0x28
# echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > ./events/kprobes/ \ p_bio_add_page_0/trigger
# cd /sys/kernel/debug/tracing/ # echo 'p bio_add_page arg1=$arg1' > ./kprobe_events # echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > ./events/kprobes/p_bio_add_page_0/trigger
# du -sh /test.txt 12.0K /test.txt
# cat /test.txt > /dev/null
# cat ./trace
# tracer: nop
#
# entries-in-buffer/entries-written: 128/128 #P:4
#
# _-----=> irqs-off/BH-disabled
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| / _-=> migrate-disable
# |||| / delay
# TASK-PID CPU# ||||| TIMESTAMP FUNCTION
# | | | ||||| | |
cat-117 [002] ...1. 1.602243: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x0
cat-117 [002] ...1. 1.602244: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x0
cat-117 [002] ...2. 1.602244: bio_add_page <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x1000
cat-117 [002] ...1. 1.602245: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x1000
cat-117 [002] ...1. 1.602245: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x1000
cat-117 [002] ...2. 1.602245: bio_add_page <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x2000
cat-117 [002] ...1. 1.602245: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x2000
cat-117 [002] ...1. 1.602245: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x2000
cat-117 [002] ...1. 1.602245: submit_bio <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x3000
cat-117 [002] ...1. 1.602245: submit_bio_noacct <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x3000
cat-117 [002] ...1. 1.602246: __submit_bio <-submit_bio_noacct object:0xffff88811bee4000 value:0x3000
cat-117 [002] ...1. 1.602246: submit_bio_checks <-__submit_bio object:0xffff88811bee4000 value:0x3000
cat-117 [002] ...1. 1.602246: __cond_resched <-submit_bio_checks object:0xffff88811bee4000 value:0x3000
cat-117 [002] ...1. 1.602246: should_fail_bio <-submit_bio_checks object:0xffff88811bee4000 value:0x3000
cat-117 [002] ...1. 1.602246: blk_mq_submit_bio <-submit_bio_noacct object:0xffff88811bee4000 value:0x3000
cat-117 [002] ...1. 1.602246: blk_attempt_plug_merge <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000
cat-117 [002] ...1. 1.602246: blk_mq_sched_bio_merge <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000
cat-117 [002] ...1. 1.602247: __rcu_read_lock <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000
cat-117 [002] ...1. 1.602247: __rcu_read_unlock <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000
cat-117 [002] ...1. 1.602247: __blk_mq_alloc_requests <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000
<idle>-0 [002] d..3. 1.602298: bio_endio <-blk_update_request object:0xffff88811bee4000 value:0x0
<idle>-0 [002] d..3. 1.602298: mpage_end_io <-blk_update_request object:0xffff88811bee4000 value:0x0
<idle>-0 [002] d..3. 1.602298: __read_end_io <-blk_update_request object:0xffff88811bee4000 value:0x0
<idle>-0 [002] d..3. 1.602300: bio_put <-blk_update_request object:0xffff88811bee4000 value:0x0
<idle>-0 [002] d..3. 1.602300: bio_free <-blk_update_request object:0xffff88811bee4000 value:0x0
<idle>-0 [002] d..3. 1.602300: mempool_free <-blk_update_request object:0xffff88811bee4000 value:0x0
<idle>-0 [002] d..3. 1.602300: mempool_free_slab <-blk_update_request object:0xffff88811bee4000 value:0x0
<idle>-0 [002] d..3. 1.602300: kmem_cache_free <-blk_update_request object:0xffff88811bee4000 value:0x0
...Almost all changelogs were suggested by Masami([email protected]) and steve([email protected]), thank you all so much.
v8:
- revert to use per-cpu recursion for the function trace_object_events_call
- recover the filter when getting the value of the object
- simplify the implementation for the function get_object_value
- fix the build error
v7:
- use fixed-size array for object pool instead of list structure
- use ftrace_test_recursion_trylock for function trace hook function
- fix trace_object_ref reference count in the init_trace_object
- invoke exit_trace_object no matter whether data->ops->free is null in the unregister_object_trigger
- release private_data of event_trigger_data in the trace_object_trigger_free
- remove [RFC] tag
v6:
- change the objtrace trigger syntax.
- add patchset description
- add <tracefs>/README
v5:
- add testcasts
- add check the field->size
- add lockless to search object
- describe the object trace more clearly in Kconfig
v4:
- please ignore the v4 which is the same as v3
v3:
- change the objfilter to objtrace
- add a command to the objfilter syntax
- change to get the value of the object
- use trace_find_event_field to find the field instead of using argN
- get data from @rec in the event trigger callback funciton
v2:
- adding a "objfilter" trigger to update object
Jeff Xie (4):
trace: Add trace any kernel object
trace/objtrace: get the value of the object
trace/objtrace: Add testcases for objtrace
trace/objtrace: Add documentation for objtrace
Documentation/trace/events.rst | 83 +++ include/linux/trace_events.h | 1 + kernel/trace/Kconfig | 10 + kernel/trace/Makefile | 1 + kernel/trace/trace.c | 3 + kernel/trace/trace.h | 8 + kernel/trace/trace_entries.h | 18 + kernel/trace/trace_events_trigger.c | 1 + kernel/trace/trace_object.c | 631 ++++++++++++++++++ kernel/trace/trace_output.c | 40 ++ .../ftrace/test.d/trigger/trigger-objtrace.tc | 39 ++ 11 files changed, 835 insertions(+) create mode 100644 kernel/trace/trace_object.c create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-objtrace.tc
Signature · 1 line
2.25.1
PATCH v8 1/4 trace: Add trace any kernel object
Introduce objtrace trigger to get the call flow by tracing any kernel object in the function parameter.
The objtrace trigger makes a list of the target object address from the given event parameter, and records all kernel function calls which has the object address in its parameter.
Syntax:
objtrace:add:obj[:count][if <filter>]
Usage:
# echo 'p bio_add_page arg1=$arg1' > kprobe_events # cd events/kprobes/p_bio_add_page_0 # echo 'objtrace:add:arg1:1 if comm == "cat"' > ./trigger # cat /test.txt
--- include/linux/trace_events.h | 1 + kernel/trace/Kconfig | 10 + kernel/trace/Makefile | 1 + kernel/trace/trace.c | 3 + kernel/trace/trace.h | 8 + kernel/trace/trace_entries.h | 17 + kernel/trace/trace_events_trigger.c | 1 + kernel/trace/trace_object.c | 465 ++++++++++++++++++++++++++++ kernel/trace/trace_output.c | 40 +++ 9 files changed, 546 insertions(+) create mode 100644 kernel/trace/trace_object.c
include/linux/trace_events.h +1 -0
@@ -685,6 +685,7 @@ enum event_trigger_type {ETT_EVENT_HIST = (1 <<4 ),ETT_HIST_ENABLE = (1 <<5 ),ETT_EVENT_EPROBE = (1 <<6 ), + ETT_TRACE_OBJECT = (1 << 7), };extern int filter_match_preds (struct event_filter *filter,void *rec);
kernel/trace/Kconfig +10 -0
@@ -250,6 +250,16 @@ config FUNCTION_PROFILER If in doubt, say N. +config TRACE_OBJECT + bool "Trace kernel object in function parameter" + depends on FUNCTION_TRACER + depends on HAVE_FUNCTION_ARG_ACCESS_API + select TRACING + default y + help + You can trace the kernel object in the kernel function parameter. + The kernel object is dynamically specified via event trigger. + config STACK_TRACER bool "Trace max stack" depends on HAVE_FUNCTION_TRACER
kernel/trace/Makefile +1 -0
@@ -67,6 +67,7 @@ obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += trace_functions_graph.o obj-$(CONFIG_TRACE_BRANCH_PROFILING) += trace_branch.o obj-$(CONFIG_BLK_DEV_IO_TRACE) += blktrace.o obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += fgraph.o +obj-$(CONFIG_TRACE_OBJECT) += trace_object.o ifeq ($(CONFIG_BLOCK),y) obj-$(CONFIG_EVENT_TRACING) += blktrace.o endif
kernel/trace/trace.c +3 -0
@@ -5605,6 +5605,9 @@ static const char readme_msg[] ="\t enable_hist:<system>:<event>\n" "\t disable_hist:<system>:<event>\n"#endif +#ifdef CONFIG_TRACE_OBJECT + "\t objtrace:add:obj[:count][if <filter>]\n" +#endif #ifdef CONFIG_STACKTRACE "\t\t stacktrace\n"#endif
kernel/trace/trace.h +8 -0
@@ -54,6 +54,7 @@ enum trace_type {TRACE_TIMERLAT ,TRACE_RAW_DATA ,TRACE_FUNC_REPEATS , +TRACE_OBJECT , __TRACE_LAST_TYPE, };
@@ -472,6 +473,7 @@ extern void __ftrace_bad_type(void); TRACE_GRAPH_RET); \ IF_ASSIGN(var, ent, struct func_repeats_entry, \TRACE_FUNC_REPEATS ); \ + IF_ASSIGN(var, ent, struct trace_object_entry, TRACE_OBJECT);\__ftrace_bad_type (); \ }while (0 )
@@ -1537,6 +1539,12 @@ static inline int register_trigger_hist_cmd(void) { return 0; }static inline int register_trigger_hist_enable_disable_cmds (void ) {return 0 ; }#endif +#ifdef CONFIG_TRACE_OBJECT +extern int register_trigger_object_cmd (void ); +#else +static inline int register_trigger_object_cmd (void ) {return 0 ; } +#endif +extern int register_trigger_cmds (void );extern void clear_event_triggers (struct trace_array *tr);
kernel/trace/trace_entries.h +17 -0
@@ -401,3 +401,20 @@ FTRACE_ENTRY(timerlat, timerlat_entry, __entry->context, __entry->timer_latency) ); + +/* +* trace object entry: +*/ +FTRACE_ENTRY (object, trace_object_entry, + +TRACE_OBJECT , + +F_STRUCT ( + __field( unsigned long, ip ) + __field( unsigned long, parent_ip ) + __field( unsigned long, object ) + ), + +F_printk (" %ps <-- %ps object:%lx\n" , + (void *)__entry->ip, (void *)__entry->parent_ip, __entry->object) +);
kernel/trace/trace_events_trigger.c +1 -0
@@ -2025,6 +2025,7 @@ __init int register_trigger_cmds(void)register_trigger_enable_disable_cmds ();register_trigger_hist_enable_disable_cmds ();register_trigger_hist_cmd (); +register_trigger_object_cmd ();return 0 ; }
kernel/trace/trace_object.c new file +465 -0
465 lines not on this page · raw message
kernel/trace/trace_output.c +40 -0
@@ -1547,6 +1547,45 @@ static struct trace_event trace_func_repeats_event = { .funcs = &trace_func_repeats_funcs, }; +/* TRACE_OBJECT */ +static enum print_line_t trace_object_print (struct trace_iterator *iter,int flags, +struct trace_event *event) +{ +struct trace_object_entry *field; +struct trace_seq *s = &iter->seq; + +trace_assign_type (field, iter->ent); +print_fn_trace (s, field->ip, field->parent_ip, flags); +trace_seq_printf (s," object:0x%lx" , field->object); +trace_seq_putc (s,'\n' ); + +return trace_handle_return (s); +} + +static enum print_line_t trace_object_raw (struct trace_iterator *iter,int flags, +struct trace_event *event) +{ +struct trace_object_entry *field; + +trace_assign_type (field, iter->ent); + +trace_seq_printf (&iter->seq,"%lx %lx\n" , + field->ip, + field->parent_ip); + +return trace_handle_return (&iter->seq); +} + +static struct trace_event_functions trace_object_funcs = { + .trace = trace_object_print, + .raw = trace_object_raw, +}; + +static struct trace_event trace_object_event = { + .type =TRACE_OBJECT , + .funcs = &trace_object_funcs, +}; +static struct trace_event *events[]__initdata = { &trace_fn_event, &trace_ctx_event,
@@ -1561,6 +1600,7 @@ static struct trace_event *events[] __initdata = { &trace_timerlat_event, &trace_raw_data_event, &trace_func_repeats_event, + &trace_object_event, NULL };
Signature · 1 line
2.25.1
PATCH v8 2/4 trace/objtrace: get the value of the object
In reply to Jeff XieUsing objtrace trigger to get the value of the object which from the kernel function parameter.
Syntax:
objtrace:add:obj[,offset][:type][:count][if <filter>]
Usage:
# echo 'p bio_add_page arg1=$arg1' > ./kprobe_events # gdb vmlinux (gdb) p &(((struct bio *)0)->bi_iter.bi_size) $1 = (unsigned int *) 0x28 # echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > ./events/kprobes/ \ p_bio_add_page_0/trigger # cat /test.txt
--- kernel/trace/trace.c | 2 +- kernel/trace/trace_entries.h | 5 +- kernel/trace/trace_object.c | 214 +++++++++++++++++++++++++++++++---- kernel/trace/trace_output.c | 6 +- 4 files changed, 197 insertions(+), 30 deletions(-)
kernel/trace/trace.c +1 -1
@@ -5606,7 +5606,7 @@ static const char readme_msg[] = "\t disable_hist:<system>:<event>\n"#endif #ifdef CONFIG_TRACE_OBJECT - "\t objtrace:add:obj[:count][if <filter>]\n" + "\t objtrace:add:obj[,offset][:type][:count][if <filter>]\n"#endif #ifdef CONFIG_STACKTRACE "\t\t stacktrace\n"
kernel/trace/trace_entries.h +3 -2
@@ -413,8 +413,9 @@ FTRACE_ENTRY(object, trace_object_entry, __field( unsigned long, ip ) __field( unsigned long, parent_ip ) __field( unsigned long, object ) + __field( unsigned long, value ) ), - F_printk(" %ps <-- %ps object:%lx\n", - (void *)__entry->ip, (void *)__entry->parent_ip, __entry->object) + F_printk(" %ps <-- %ps object:%lx value:%lx\n", (void *)__entry->ip, + (void *)__entry->parent_ip, __entry->object, __entry->value) );
kernel/trace/trace_object.c +190 -24
342 lines not on this page · raw message
kernel/trace/trace_output.c +3 -3
@@ -1557,6 +1557,7 @@ static enum print_line_t trace_object_print(struct trace_iterator *iter, int flatrace_assign_type (field, iter->ent);print_fn_trace (s, field->ip, field->parent_ip, flags);trace_seq_printf (s," object:0x%lx" , field->object); +trace_seq_printf (s," value:0x%lx" , field->value);trace_seq_putc (s,'\n' );return trace_handle_return (s);
@@ -1569,9 +1570,8 @@ static enum print_line_t trace_object_raw(struct trace_iterator *iter, int flagstrace_assign_type (field, iter->ent); -trace_seq_printf (&iter->seq,"%lx %lx\n" , - field->ip, - field->parent_ip); +trace_seq_printf (&iter->seq,"%lx %lx %lx %lx\n" , field->ip, + field->parent_ip, field->object, field->value);return trace_handle_return (&iter->seq); }
Signature · 1 line
2.25.1
PATCH v8 3/4 trace/objtrace: Add testcases for objtrace
In reply to Jeff XieAdd a series of testcases to illustrate correct and incorrect usage of objtrace trigger.
--- .../ftrace/test.d/trigger/trigger-objtrace.tc | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-objtrace.tc
tools/testing/selftests/ftrace/test.d/trigger/trigger-objtrace.tc new file +39 -0
@@ -0,0 +1,39 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# description: event trigger - test objtrace-trigger +# requires: kprobe_events "objtrace":README + +fail() { #msg + echo $1 + exit_fail +} + +echo 'p bio_add_page arg1=$arg1' > kprobe_events + +FEATURE=`grep objtrace events/kprobes/p_bio_add_page_0/trigger` +if [ -z "$FEATURE" ]; then + echo "objtrace trigger is not supported" + exit_unsupported +fi + +echo "Test objtrace trigger" +echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > \ + events/kprobes/p_bio_add_page_0/trigger +if [ -z $? ]; then + fail "objtrace trigger syntax error" +fi + +echo "Test objtrace semantic errors" + +# Being lack of objtrace command +! echo 'objtrace:arg1,0x28:u32:1' > events/kprobes/p_bio_add_page_0/trigger +# Bad parameter name +! echo 'objtrace:add:argx:u32:1' > events/kprobes/p_bio_add_page_0/trigger + +echo "reset objtrace trigger" + +echo '!objtrace:add:arg1,0x28:u32' > \ + events/kprobes/p_bio_add_page_0/trigger +echo '-:p_bio_add_page_0' >> ./kprobe_events + +exit 0
Signature · 1 line
2.25.1
PATCH v8 4/4 trace/objtrace: Add documentation for objtrace
In reply to Jeff XieAdded documentation explaining how to use objtrace trigger to get the value of the object.
--- Documentation/trace/events.rst | 83 ++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+)
Documentation/trace/events.rst +83 -0
@@ -546,6 +546,89 @@ The following commands are supported: See Documentation/trace/histogram.rst for details and examples. +- objtrace + + This command provides a way to get the value of any object, The object + can be obtained from the dynamic event(kprobe_event/uprobe_event) or the + static event(tracepoint). + + Usage: + When using the kprobe event, only need to set the objtrace(a new trigger), + we can get the value of the object. The object is from the setting of the + kprobe event. + + For example: + For the function bio_add_page(): + + int bio_add_page(struct bio *bio, struct page *page, + unsigned int len, unsigned int offset) + + Firstly, we can set the base of the object, thus the first string "arg1" + stands for the value of the first parameter of this function bio_add_gage(), + + # echo 'p bio_add_page arg1=$arg1' > ./kprobe_events + + Secondly, we can get the value dynamically based on above object. + + find the offset of the bi_size in struct bio: + $ gdb vmlinux + (gdb) p &(((struct bio *)0)->bi_iter.bi_size) + $1 = (unsigned int *) 0x28 + + # echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > ./events/kprobes/ \ + p_bio_add_page_0/trigger + + # cd /sys/kernel/debug/tracing/ + # echo 'p bio_add_page arg1=$arg1' > ./kprobe_events + # echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > ./events/kprobes/p_bio_add_page_0/trigger + + # du -sh /test.txt + 12.0K /test.txt + + # cat /test.txt > /dev/null + # cat ./trace + # tracer: nop + # + # entries-in-buffer/entries-written: 128/128 #P:4 + # + # _-----=> irqs-off/BH-disabled + # / _----=> need-resched + # | / _---=> hardirq/softirq + # || / _--=> preempt-depth + # ||| / _-=> migrate-disable + # |||| / delay + # TASK-PID CPU# ||||| TIMESTAMP FUNCTION + # | | | ||||| | | + cat-117 [002] ...1. 1.602243: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x0 + cat-117 [002] ...1. 1.602244: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x0 + cat-117 [002] ...2. 1.602244: bio_add_page <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x1000 + cat-117 [002] ...1. 1.602245: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x1000 + cat-117 [002] ...1. 1.602245: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x1000 + cat-117 [002] ...2. 1.602245: bio_add_page <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x2000 + cat-117 [002] ...1. 1.602245: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x2000 + cat-117 [002] ...1. 1.602245: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x2000 + cat-117 [002] ...1. 1.602245: submit_bio <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x3000 + cat-117 [002] ...1. 1.602245: submit_bio_noacct <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x3000 + cat-117 [002] ...1. 1.602246: __submit_bio <-submit_bio_noacct object:0xffff88811bee4000 value:0x3000 + cat-117 [002] ...1. 1.602246: submit_bio_checks <-__submit_bio object:0xffff88811bee4000 value:0x3000 + cat-117 [002] ...1. 1.602246: __cond_resched <-submit_bio_checks object:0xffff88811bee4000 value:0x3000 + cat-117 [002] ...1. 1.602246: should_fail_bio <-submit_bio_checks object:0xffff88811bee4000 value:0x3000 + cat-117 [002] ...1. 1.602246: blk_mq_submit_bio <-submit_bio_noacct object:0xffff88811bee4000 value:0x3000 + cat-117 [002] ...1. 1.602246: blk_attempt_plug_merge <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 + cat-117 [002] ...1. 1.602246: blk_mq_sched_bio_merge <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 + cat-117 [002] ...1. 1.602247: __rcu_read_lock <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 + cat-117 [002] ...1. 1.602247: __rcu_read_unlock <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 + cat-117 [002] ...1. 1.602247: __blk_mq_alloc_requests <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 + <idle>-0 [002] d..3. 1.602298: bio_endio <-blk_update_request object:0xffff88811bee4000 value:0x0 + <idle>-0 [002] d..3. 1.602298: mpage_end_io <-blk_update_request object:0xffff88811bee4000 value:0x0 + <idle>-0 [002] d..3. 1.602298: __read_end_io <-blk_update_request object:0xffff88811bee4000 value:0x0 + <idle>-0 [002] d..3. 1.602300: bio_put <-blk_update_request object:0xffff88811bee4000 value:0x0 + <idle>-0 [002] d..3. 1.602300: bio_free <-blk_update_request object:0xffff88811bee4000 value:0x0 + <idle>-0 [002] d..3. 1.602300: mempool_free <-blk_update_request object:0xffff88811bee4000 value:0x0 + <idle>-0 [002] d..3. 1.602300: mempool_free_slab <-blk_update_request object:0xffff88811bee4000 value:0x0 + <idle>-0 [002] d..3. 1.602300: kmem_cache_free <-blk_update_request object:0xffff88811bee4000 value:0x0 + ... + 7. In-kernel trace event API ============================
Signature · 1 line
2.25.1
Masami Hiramatsu
maintainerHi Jeff,
Thank you for updating the series, I'm testing the series and found some issues. See below;
Quoted 27 lines · On Sun, 30 Jan 2022 22:44:26 +0800 Jeff Xie <[email protected]> wrote:
Introduce a method based on function tracer to trace any object and get the value of the object dynamically. the object can be obtained from the dynamic event (kprobe_event/uprobe_event) or the static event(tracepoint).
Usage:
When using the kprobe event, only need to set the objtrace(a new trigger), we can get the value of the object. The object is from the setting of the kprobe event.For example:
For the functionbio_add_page():int bio_add_page(struct bio *bio, struct page *page,
unsigned int len, unsigned int offset)Firstly, we can set the base of the object, thus the first string "arg1" stands for the value of the first parameter of this function
bio_add_gage(),# echo 'p bio_add_page arg1=$arg1' > ./kprobe_eventsSecondly, we can get the value dynamically based on above object.
find the offset of the
bi_sizein struct bio:$ gdb vmlinux (gdb) p &(((struct bio *)0)->bi_iter.bi_size) $1 = (unsigned int *) 0x28
Quoted 6 lines
# echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > ./events/kprobes/ \ p_bio_add_page_0/trigger# cd /sys/kernel/debug/tracing/ # echo 'p bio_add_page arg1=$arg1' > ./kprobe_events # echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > ./events/kprobes/p_bio_add_page_0/trigger
So doing after this command, I dumped the trigger file.
# cat events/kprobes/p_bio_add_page_0/trigger objtrace:count=1 if comm == "cat"
This is not correct. All the parameters are dropped. Thus I can not remove the trigger without knowing the original command.
# echo '!objtrace:count=1 if comm == "cat"' >> ./events/kprobes/p_bio_add_page_0/trigger sh: write error: Invalid argument
And there are another minor issue is here;
# cd /sys/kernel/debug/tracing/ # echo 'p bio_add_page $arg1 $arg2' > ./kprobe_events # echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > ./events/kprobes/p_bio_add_page_0/trigger # echo '!objtrace:add:arg2' >> events/kprobes/p_bio_add_page_0/trigger
Here, if I passed the "!objtrace:add:<a-field-exists-on-event>" then the objtrace trigger was removed. But as you can see, the parameter is different. Anyway, this is a minor issue, since we don't support multiple objtrace trigger on the same event.
Could you fix these issues (and update the test case to check this?)
Thank you,
Quoted 115 lines
# du -sh /test.txt 12.0K /test.txt# cat /test.txt > /dev/null # cat ./trace # tracer: nop # # entries-in-buffer/entries-written: 128/128 #P:4 # # _-----=> irqs-off/BH-disabled # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay # TASK-PID CPU# ||||| TIMESTAMP FUNCTION # | | | ||||| | | cat-117 [002] ...1. 1.602243: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x0 cat-117 [002] ...1. 1.602244: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x0 cat-117 [002] ...2. 1.602244: bio_add_page <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x1000 cat-117 [002] ...1. 1.602245: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x1000 cat-117 [002] ...1. 1.602245: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x1000 cat-117 [002] ...2. 1.602245: bio_add_page <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x2000 cat-117 [002] ...1. 1.602245: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x2000 cat-117 [002] ...1. 1.602245: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x2000 cat-117 [002] ...1. 1.602245: submit_bio <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602245: submit_bio_noacct <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: __submit_bio <-submit_bio_noacct object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: submit_bio_checks <-__submit_bio object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: __cond_resched <-submit_bio_checks object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: should_fail_bio <-submit_bio_checks object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: blk_mq_submit_bio <-submit_bio_noacct object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: blk_attempt_plug_merge <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: blk_mq_sched_bio_merge <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602247: __rcu_read_lock <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602247: __rcu_read_unlock <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602247: __blk_mq_alloc_requests <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 <idle>-0 [002] d..3. 1.602298: bio_endio <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602298: mpage_end_io <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602298: __read_end_io <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602300: bio_put <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602300: bio_free <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602300: mempool_free <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602300: mempool_free_slab <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602300: kmem_cache_free <-blk_update_request object:0xffff88811bee4000 value:0x0 ...Almost all changelogs were suggested by Masami([email protected]) and steve([email protected]), thank you all so much.
v8:
- revert to use per-cpu recursion for the functiontrace_object_events_call
- recover the filter when getting the value of the object
- simplify the implementation for the functionget_object_value
- fix the build errorv7:
- use fixed-size array for object pool instead of list structure
- useftrace_test_recursion_trylockfor function trace hook function
- fixtrace_object_refreference count in theinit_trace_object
- invokeexit_trace_objectno matter whetherdata->ops->freeis null in theunregister_object_trigger
- releaseprivate_dataofevent_trigger_datain thetrace_object_trigger_free
- remove [RFC] tagv6:
- change the objtrace trigger syntax.
- add patchset description
- add <tracefs>/READMEv5:
- add testcasts
- add check thefield->size
- add lockless to search object
- describe the object trace more clearly inKconfigv4:
- please ignore the v4 which is the same as v3v3:
- change the objfilter to objtrace
- add a command to the objfilter syntax
- change to get the value of the object
- usetrace_find_event_fieldto find the field instead of using argN
- get data from @rec in the event trigger callback funcitonv2:
- adding a "objfilter" trigger to update objectJeff Xie (4):
trace: Add trace any kernel object
trace/objtrace: get the value of the object
trace/objtrace: Add testcases for objtrace
trace/objtrace: Add documentation for objtraceDocumentation/trace/events.rst | 83 +++ include/linux/trace_events.h | 1 + kernel/trace/Kconfig | 10 + kernel/trace/Makefile | 1 + kernel/trace/trace.c | 3 + kernel/trace/trace.h | 8 + kernel/trace/trace_entries.h | 18 + kernel/trace/trace_events_trigger.c | 1 + kernel/trace/trace_object.c | 631 ++++++++++++++++++ kernel/trace/trace_output.c | 40 ++ .../ftrace/test.d/trigger/trigger-objtrace.tc | 39 ++ 11 files changed, 835 insertions(+) create mode 100644 kernel/trace/trace_object.c create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-objtrace.tcSignature · 1 line
2.25.1
Signature · 1 line
Masami Hiramatsu <[email protected]>
Hi Masami,
Quoted 61 lines · Masami Hiramatsu
Hi Jeff,
Thank you for updating the series, I'm testing the series and found some issues. See below;
Quoted 27 lines · On Sun, 30 Jan 2022 22:44:26 +0800 Jeff Xie <[email protected]> wrote:
Introduce a method based on function tracer to trace any object and get the value of the object dynamically. the object can be obtained from the dynamic event (kprobe_event/uprobe_event) or the static event(tracepoint).
Usage:
When using the kprobe event, only need to set the objtrace(a new trigger), we can get the value of the object. The object is from the setting of the kprobe event.For example:
For the functionbio_add_page():int bio_add_page(struct bio *bio, struct page *page,
unsigned int len, unsigned int offset)Firstly, we can set the base of the object, thus the first string "arg1" stands for the value of the first parameter of this function
bio_add_gage(),# echo 'p bio_add_page arg1=$arg1' > ./kprobe_eventsSecondly, we can get the value dynamically based on above object.
find the offset of the
bi_sizein struct bio:$ gdb vmlinux (gdb) p &(((struct bio *)0)->bi_iter.bi_size) $1 = (unsigned int *) 0x28Quoted 6 lines
# echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > ./events/kprobes/ \ p_bio_add_page_0/trigger# cd /sys/kernel/debug/tracing/ # echo 'p bio_add_page arg1=$arg1' > ./kprobe_events # echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > ./events/kprobes/p_bio_add_page_0/triggerSo doing after this command, I dumped the trigger file.
# cat events/kprobes/p_bio_add_page_0/trigger objtrace:count=1 if comm == "cat"This is not correct. All the parameters are dropped. Thus I can not remove the trigger without knowing the original command.
# echo '!objtrace:count=1 if comm == "cat"' >> ./events/kprobes/p_bio_add_page_0/trigger sh: write error: Invalid argumentAnd there are another minor issue is here;
# cd /sys/kernel/debug/tracing/ # echo 'p bio_add_page $arg1 $arg2' > ./kprobe_events # echo 'objtrace:add:arg1,0x28:u32:1 if comm == "cat"' > ./events/kprobes/p_bio_add_page_0/trigger # echo '!objtrace:add:arg2' >> events/kprobes/p_bio_add_page_0/trigger
Quoted 6 lines
Here, if I passed the "!objtrace:add:<a-field-exists-on-event>" then the objtrace trigger was removed. But as you can see, the parameter is different. Anyway, this is a minor issue, since we don't support multiple objtrace trigger on the same event.
Could you fix these issues (and update the test case to check this?)
Thanks for finding these issues, I will fix them.
Quoted 114 lines
Thank you,
Quoted 112 lines
# du -sh /test.txt 12.0K /test.txt# cat /test.txt > /dev/null # cat ./trace # tracer: nop # # entries-in-buffer/entries-written: 128/128 #P:4 # # _-----=> irqs-off/BH-disabled # / _----=> need-resched # | / _---=> hardirq/softirq # || / _--=> preempt-depth # ||| / _-=> migrate-disable # |||| / delay # TASK-PID CPU# ||||| TIMESTAMP FUNCTION # | | | ||||| | | cat-117 [002] ...1. 1.602243: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x0 cat-117 [002] ...1. 1.602244: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x0 cat-117 [002] ...2. 1.602244: bio_add_page <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x1000 cat-117 [002] ...1. 1.602245: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x1000 cat-117 [002] ...1. 1.602245: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x1000 cat-117 [002] ...2. 1.602245: bio_add_page <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x2000 cat-117 [002] ...1. 1.602245: __bio_try_merge_page <-bio_add_page object:0xffff88811bee4000 value:0x2000 cat-117 [002] ...1. 1.602245: __bio_add_page <-bio_add_page object:0xffff88811bee4000 value:0x2000 cat-117 [002] ...1. 1.602245: submit_bio <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602245: submit_bio_noacct <-ext4_mpage_readpages object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: __submit_bio <-submit_bio_noacct object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: submit_bio_checks <-__submit_bio object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: __cond_resched <-submit_bio_checks object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: should_fail_bio <-submit_bio_checks object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: blk_mq_submit_bio <-submit_bio_noacct object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: blk_attempt_plug_merge <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602246: blk_mq_sched_bio_merge <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602247: __rcu_read_lock <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602247: __rcu_read_unlock <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 cat-117 [002] ...1. 1.602247: __blk_mq_alloc_requests <-blk_mq_submit_bio object:0xffff88811bee4000 value:0x3000 <idle>-0 [002] d..3. 1.602298: bio_endio <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602298: mpage_end_io <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602298: __read_end_io <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602300: bio_put <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602300: bio_free <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602300: mempool_free <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602300: mempool_free_slab <-blk_update_request object:0xffff88811bee4000 value:0x0 <idle>-0 [002] d..3. 1.602300: kmem_cache_free <-blk_update_request object:0xffff88811bee4000 value:0x0 ...Almost all changelogs were suggested by Masami([email protected]) and steve([email protected]), thank you all so much.
v8:
- revert to use per-cpu recursion for the functiontrace_object_events_call
- recover the filter when getting the value of the object
- simplify the implementation for the functionget_object_value
- fix the build errorv7:
- use fixed-size array for object pool instead of list structure
- useftrace_test_recursion_trylockfor function trace hook function
- fixtrace_object_refreference count in theinit_trace_object
- invokeexit_trace_objectno matter whetherdata->ops->freeis null in theunregister_object_trigger
- releaseprivate_dataofevent_trigger_datain thetrace_object_trigger_free
- remove [RFC] tagv6:
- change the objtrace trigger syntax.
- add patchset description
- add <tracefs>/READMEv5:
- add testcasts
- add check thefield->size
- add lockless to search object
- describe the object trace more clearly inKconfigv4:
- please ignore the v4 which is the same as v3v3:
- change the objfilter to objtrace
- add a command to the objfilter syntax
- change to get the value of the object
- usetrace_find_event_fieldto find the field instead of using argN
- get data from @rec in the event trigger callback funcitonv2:
- adding a "objfilter" trigger to update objectJeff Xie (4):
trace: Add trace any kernel object
trace/objtrace: get the value of the object
trace/objtrace: Add testcases for objtrace
trace/objtrace: Add documentation for objtraceDocumentation/trace/events.rst | 83 +++ include/linux/trace_events.h | 1 + kernel/trace/Kconfig | 10 + kernel/trace/Makefile | 1 + kernel/trace/trace.c | 3 + kernel/trace/trace.h | 8 + kernel/trace/trace_entries.h | 18 + kernel/trace/trace_events_trigger.c | 1 + kernel/trace/trace_object.c | 631 ++++++++++++++++++ kernel/trace/trace_output.c | 40 ++ .../ftrace/test.d/trigger/trigger-objtrace.tc | 39 ++ 11 files changed, 835 insertions(+) create mode 100644 kernel/trace/trace_object.c create mode 100644 tools/testing/selftests/ftrace/test.d/trigger/trigger-objtrace.tc
Quoted 7 lines
Quoted 3 lines
--
2.25.1--
Masami Hiramatsu <[email protected]>
---
JeffXie