tracing/eprobes: Fixes for unexpected arguments
Subsystems (2)
Steven Rostedt
maintainerv2 cover tracing/eprobes: Fixes for unexpected arguments
While using eprobes, I decided to entertain the thougth of what would happen if I tried to get the instruction pointer "%rip", knowing full well that eprobes do not have access to pt_regs. Well, I found out, and it led me down a rabbit hole of bugs.
This series fixes those bugs, by not allowing register access for eprobes, and also filling the holes of @symbol and @immediate argument.
Changes since v1: https://lore.kernel.org/all/[email protected]/
- Fixed parenthesis warning
- Fixed comment about comm arguments
- Made kprobes and eprobes process "$COMM" as well as "$comm"
- Made filters consistent with histograms with "common_cpu"
Steven Rostedt (Google) (6):
tracing/eprobes: Do not allow eprobes to use $stack, or % for regs tracing/eprobes: Do not hardcode $comm as a string tracing/eprobes: Fix reading of string fields tracing/eprobes: Have event probes be consistent with kprobes and uprobes tracing/probes: Have kprobes and uprobes use $COMM too
tracing: Have filter accept "common_cpu" to be consistent
----
kernel/trace/trace_eprobe.c | 91 ++++++++++++++++++++++++++++++++++++++++++--- kernel/trace/trace_events.c | 1 + kernel/trace/trace_probe.c | 29 +++++++++------ 3 files changed, 104 insertions(+), 17 deletions(-)
Steven Rostedt
maintainerPATCH v2 1/6 tracing/eprobes: Do not allow eprobes to use $stack, or % for regs
From: "Steven Rostedt (Google)" <[email protected]>
While playing with event probes (eprobes), I tried to see what would happen if I attempted to retrieve the instruction pointer (%rip) knowing that event probes do not use pt_regs. The result was:
BUG: kernel NULL pointer dereference, address: 0000000000000024 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: 0000 [#1] PREEMPT SMP PTI CPU: 1 PID: 1847 Comm: trace-cmd Not tainted 5.19.0-rc5-test+ #309 Hardware name: Hewlett-Packard HP Compaq Pro 6300 SFF/339A, BIOS K01 v03.03 07/14/2016 RIP: 0010:get_event_field.isra.0+0x0/0x50 Code: ff 48 c7 c7 c0 8f 74 a1 e8 3d 8b f5 ff e8 88 09 f6 ff 4c 89 e7 e8
50 6a 13 00 48 89 ef 5b 5d 41 5c 41 5d e9 42 6a 13 00 66 90 <48> 63 47 24 8b 57 2c 48 01 c6 8b 47 28 83 f8 02 74 0e 83 f8 04 74
RSP: 0018:ffff916c394bbaf0 EFLAGS: 00010086
RAX: ffff916c854041d8 RBX: ffff916c8d9fbf50 RCX: ffff916c255d2000
RDX: 0000000000000000 RSI: ffff916c255d2008 RDI: 0000000000000000
RBP: 0000000000000000 R08: ffff916c3a2a0c08 R09: ffff916c394bbda8
R10: 0000000000000000 R11: 0000000000000000 R12: ffff916c854041d8
R13: ffff916c854041b0 R14: 0000000000000000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff916c9ea40000(0000)
knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000024 CR3: 000000011b60a002 CR4: 00000000001706e0
Call Trace: <TASK> get_eprobe_size+0xb4/0x640 ? __mod_node_page_state+0x72/0xc0 __eprobe_trace_func+0x59/0x1a0 ? __mod_lruvec_page_state+0xaa/0x1b0 ? page_remove_file_rmap+0x14/0x230 ? page_remove_rmap+0xda/0x170 event_triggers_call+0x52/0xe0 trace_event_buffer_commit+0x18f/0x240 trace_event_raw_event_sched_wakeup_template+0x7a/0xb0 try_to_wake_up+0x260/0x4c0 __wake_up_common+0x80/0x180 __wake_up_common_lock+0x7c/0xc0 do_notify_parent+0x1c9/0x2a0 exit_notify+0x1a9/0x220 do_exit+0x2ba/0x450 do_group_exit+0x2d/0x90 __x64_sys_exit_group+0x14/0x20 do_syscall_64+0x3b/0x90 entry_SYSCALL_64_after_hwframe+0x46/0xb0
Obviously this is not the desired result.
Move the testing for TPARG_FL_TPOINT which is only used for event probes to the top of the "$" variable check, as all the other variables are not used for event probes. Also add a check in the register parsing "%" to fail if an event probe is used.
--- kernel/trace/trace_probe.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
kernel/trace/trace_probe.c +13 -8
@@ -283,7 +283,14 @@ static int parse_probe_vars(char *arg, const struct fetch_type *t,int ret =0 ;int len; -if (strcmp (arg,"retval" ) ==0 ) { +if (flags &TPARG_FL_TPOINT ) { +if (code->data) +return -EFAULT ; + code->data =kstrdup (arg,GFP_KERNEL ); +if (!code->data) +return -ENOMEM ; + code->op =FETCH_OP_TP_ARG ; + }else if (strcmp (arg,"retval" ) ==0 ) {if (flags &TPARG_FL_RETURN ) { code->op =FETCH_OP_RETVAL ; }else {
@@ -323,13 +330,6 @@ static int parse_probe_vars(char *arg, const struct fetch_type *t, code->op =FETCH_OP_ARG ; code->param = (unsigned int )param -1 ;#endif - }else if (flags &TPARG_FL_TPOINT ) { -if (code->data) -return -EFAULT ; - code->data =kstrdup (arg,GFP_KERNEL ); -if (!code->data) -return -ENOMEM ; - code->op =FETCH_OP_TP_ARG ; }else goto inval_var;
@@ -384,6 +384,11 @@ parse_probe_arg(char *arg, const struct fetch_type *type,break ;case '%' :/* named register */ +if (flags &TPARG_FL_TPOINT ) { +/* eprobes do not handle registers */ +trace_probe_log_err (offs,BAD_VAR ); +break ; + } ret =regs_query_register_offset (arg +1 );if (ret >=0 ) { code->op =FETCH_OP_REG ;
Signature · 1 line
2.35.1
Steven Rostedt
maintainerPATCH v2 2/6 tracing/eprobes: Do not hardcode $comm as a string
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
The variable $comm is hard coded as a string, which is true for both kprobes and uprobes, but for event probes (eprobes) it is a field name. In most cases the "comm" field would be a string, but there's no guarantee of that fact.
Do not assume that comm is a string. Not to mention, it currently forces comm fields to fault, as string processing for event probes is currently broken.
--- kernel/trace/trace_probe.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/trace/trace_probe.c +3 -2
@@ -622,9 +622,10 @@ static int traceprobe_parse_probe_arg_body(const char *argv, ssize_t *size,/* * Since $comm and immediate string can not be dereferenced, -* we can find those by strcmp. +* we can find those by strcmp. But ignore for eprobes. */ -if (strcmp (arg,"$comm" ) ==0 ||strncmp (arg,"\\\"" ,2 ) ==0 ) { +if (!(flags &TPARG_FL_TPOINT ) && + (strcmp (arg,"$comm" ) ==0 ||strncmp (arg,"\\\"" ,2 ) ==0 )) {/* The type of $comm must be "string", and not an array. */ if (parg->count || (t &&strcmp (t,"string" ))) goto out;
Signature · 1 line
2.35.1
Steven Rostedt
maintainerPATCH v2 4/6 tracing/eprobes: Have event probes be consistent with kprobes and uprobes
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Currently, if a symbol "@" is attempted to be used with an event probe (eprobes), it will cause a NULL pointer dereference crash.
Both kprobes and uprobes can reference data other than the main registers. Such as immediate address, symbols and the current task name. Have eprobes do the same thing.
For "comm", if "comm" is used and the event being attached to does not have the "comm" field, then make it the "$comm" that kprobes has. This is consistent to the way histograms and filters work.
--- kernel/trace/trace_eprobe.c | 70 +++++++++++++++++++++++++++++++++---- 1 file changed, 64 insertions(+), 6 deletions(-)
kernel/trace/trace_eprobe.c +64 -6
@@ -227,6 +227,7 @@ static int trace_eprobe_tp_arg_update(struct trace_eprobe *ep, int i)struct probe_arg *parg = &ep->tp.args[i];struct ftrace_event_field *field;struct list_head *head; +int ret = -ENOENT ; head =trace_get_fields (ep->event);list_for_each_entry (field, head, link) {
@@ -236,9 +237,20 @@ static int trace_eprobe_tp_arg_update(struct trace_eprobe *ep, int i)return 0 ; } } + +/* +* Argument not found on event. But allow for comm and COMM +* to be used to get the current->comm. +*/ +if (strcmp (parg->code->data,"COMM" ) ==0 || +strcmp (parg->code->data,"comm" ) ==0 ) { + parg->code->op =FETCH_OP_COMM ; + ret =0 ; + } +kfree (parg->code->data); parg->code->data =NULL ; -return -ENOENT ; +return ret; }static int eprobe_event_define_fields (struct trace_event_call *event_call)
@@ -363,16 +375,38 @@ static unsigned long get_event_field(struct fetch_insn *code, void *rec)static int get_eprobe_size (struct trace_probe *tp,void *rec) { +struct fetch_insn *code;struct probe_arg *arg;int i, len, ret =0 ;for (i =0 ; i < tp->nr_args; i++) { arg = tp->args + i; -if (unlikely (arg->dynamic)) { +if (arg->dynamic) {unsigned long val; - val =get_event_field (arg->code, rec); - len =process_fetch_insn_bottom (arg->code +1 , val,NULL ,NULL ); + code = arg->code; + retry: +switch (code->op) { +case FETCH_OP_TP_ARG : + val =get_event_field (code, rec); +break ; +case FETCH_OP_IMM : + val = code->immediate; +break ; +case FETCH_OP_COMM : + val = (unsigned long )current->comm; +break ; +case FETCH_OP_DATA : + val = (unsigned long )code->data; +break ; +case FETCH_NOP_SYMBOL :/* Ignore a place holder */ + code++; + goto retry; +default : +continue ; + } + code++; + len =process_fetch_insn_bottom (code, val,NULL ,NULL );if (len >0 ) ret += len; }
@@ -390,8 +424,28 @@ process_fetch_insn(struct fetch_insn *code, void *rec, void *dest, {unsigned long val; - val =get_event_field (code, rec); -return process_fetch_insn_bottom (code +1 , val, dest, base); + retry: +switch (code->op) { +case FETCH_OP_TP_ARG : + val =get_event_field (code, rec); +break ; +case FETCH_OP_IMM : + val = code->immediate; +break ; +case FETCH_OP_COMM : + val = (unsigned long )current->comm; +break ; +case FETCH_OP_DATA : + val = (unsigned long )code->data; +break ; +case FETCH_NOP_SYMBOL :/* Ignore a place holder */ + code++; + goto retry; +default : +return -EILSEQ ; + } + code++; +return process_fetch_insn_bottom (code, val, dest, base); }NOKPROBE_SYMBOL (process_fetch_insn)
@@ -866,6 +920,10 @@ static int trace_eprobe_tp_update_arg(struct trace_eprobe *ep, const char *argv[trace_probe_log_err (0 ,BAD_ATTACH_ARG ); } +/* Handle symbols "@" */ +if (!ret) + ret =traceprobe_update_arg (&ep->tp.args[i]); +return ret; }
Signature · 1 line
2.35.1
Steven Rostedt
maintainerPATCH v2 3/6 tracing/eprobes: Fix reading of string fields
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Currently when an event probe (eprobe) hooks to a string field, it does not display it as a string, but instead as a number. This makes the field rather useless. Handle the different kinds of strings, dynamic, static, relational/dynamic etc.
Now when a string field is used, the ":string" type can be used to display it:
echo "e:sw sched/sched_switch comm=$next_comm:string" > dynamic_events
--- kernel/trace/trace_eprobe.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
kernel/trace/trace_eprobe.c +21 -0
@@ -311,6 +311,27 @@ static unsigned long get_event_field(struct fetch_insn *code, void *rec) addr = rec + field->offset; +if (is_string_field (field)) { +switch (field->filter_type) { +case FILTER_DYN_STRING : + val = (unsigned long )(rec + (*(unsigned int *)addr &0xffff )); +break ; +case FILTER_RDYN_STRING : + val = (unsigned long )(addr + (*(unsigned int *)addr &0xffff )); +break ; +case FILTER_STATIC_STRING : + val = (unsigned long )addr; +break ; +case FILTER_PTR_STRING : + val = (unsigned long )(*(char *)addr); +break ; +default : +WARN_ON_ONCE (1 ); +return 0 ; + } +return val; + } +switch (field->size) {case 1 :if (field->is_signed)
Signature · 1 line
2.35.1
Steven Rostedt
maintainerPATCH v2 6/6 tracing: Have filter accept "common_cpu" to be consistent
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Make filtering consistent with histograms. As "cpu" can be a field of an event, allow for "common_cpu" to keep it from being confused with the "cpu" field of the event.
--- kernel/trace/trace_events.c | 1 + 1 file changed, 1 insertion(+)
kernel/trace/trace_events.c +1 -0
@@ -176,6 +176,7 @@ static int trace_define_generic_fields(void)__generic_field (int,CPU ,FILTER_CPU );__generic_field (int, cpu,FILTER_CPU ); +__generic_field (int, common_cpu,FILTER_CPU ); __generic_field(char *, COMM, FILTER_COMM); __generic_field(char *, comm, FILTER_COMM);
Signature · 1 line
2.35.1
Steven Rostedt
maintainerPATCH v2 5/6 tracing/probes: Have kprobes and uprobes use $COMM too
In reply to Steven RostedtFrom: "Steven Rostedt (Google)" <[email protected]>
Both $comm and $COMM can be used to get current->comm in eprobes and the filtering and histogram logic. Make kprobes and uprobes consistent in this regard and allow both $comm and $COMM as well. Currently kprobes and uprobes only handle $comm, which is inconsistent with the other utilities, and can be confusing to users.
--- kernel/trace/trace_probe.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
kernel/trace/trace_probe.c +3 -2
@@ -314,7 +314,7 @@ static int parse_probe_vars(char *arg, const struct fetch_type *t, } }else goto inval_var; - }else if (strcmp (arg,"comm" ) ==0 ) { + }else if (strcmp (arg,"comm" ) ==0 ||strcmp (arg,"COMM" ) ==0 ) { code->op =FETCH_OP_COMM ; #ifdef CONFIG_HAVE_FUNCTION_ARG_ACCESS_API } else if (((flags & TPARG_FL_MASK) ==
@@ -625,7 +625,8 @@ static int traceprobe_parse_probe_arg_body(const char *argv, ssize_t *size,* we can find those by strcmp. But ignore for eprobes. */ if (!(flags &TPARG_FL_TPOINT ) && - (strcmp (arg,"$comm" ) ==0 ||strncmp (arg,"\\\"" ,2 ) ==0 )) { + (strcmp (arg,"$comm" ) ==0 ||strcmp (arg,"$COMM" ) ==0 || +strncmp (arg,"\\\"" ,2 ) ==0 )) {/* The type of $comm must be "string", and not an array. */ if (parg->count || (t &&strcmp (t,"string" ))) goto out;
Signature · 1 line
2.35.1
Re: [PATCH v2 6/6] tracing: Have filter accept "common_cpu" to be consistent
In reply to Steven RostedtQuoted 8 lines · On Sat, 20 Aug 2022 09:43:22 -0400 Steven Rostedt <[email protected]> wrote:
From: "Steven Rostedt (Google)" <[email protected]>
Make filtering consistent with histograms. As "cpu" can be a field of an event, allow for "
common_cpu" to keep it from being confused with the "cpu" field of the event.
This looks good to me. Thanks!
Quoted 22 lines
Fixes 1e3bac71c5053 ("tracing/histogram: Rename "cpu" to "common_cpu"")Suggested-by Masami Hiramatsu (Google) <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>--- kernel/trace/trace_events.c | 1 + 1 file changed, 1 insertion(+)kernel/trace/trace_events.c +1
-0@@ -176,6 +176,7 @@ static int trace_define_generic_fields(void)__generic_field (int,CPU ,FILTER_CPU );__generic_field (int, cpu,FILTER_CPU ); +__generic_field (int, common_cpu,FILTER_CPU ); __generic_field(char *, COMM, FILTER_COMM); __generic_field(char *, comm, FILTER_COMM);Signature · 1 line
2.35.1
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Re: [PATCH v2 5/6] tracing/probes: Have kprobes and uprobes use $COMM too
In reply to Steven RostedtQuoted 10 lines · On Sat, 20 Aug 2022 09:43:21 -0400 Steven Rostedt <[email protected]> wrote:
From: "Steven Rostedt (Google)" <[email protected]>
Both $comm and $COMM can be used to get
current->commin eprobes and the filtering and histogram logic. Make kprobes and uprobes consistent in this regard and allow both $comm and $COMM as well. Currently kprobes and uprobes only handle $comm, which is inconsistent with the other utilities, and can be confusing to users.
This looks good to me.
(Note that kprobes/uprobes doesn't need to record cpu/pid, because those
are a part of common field and can be accessed from filter or histogram. Only comm must be recorded as string.)
Thank you,
Quoted 33 lines
Fixes 533059281ee5 ("tracing: probeevent: Introduce new argument fetching code")Suggested-by Masami Hiramatsu (Google) <[email protected]>Signed-off-by Steven Rostedt (Google) <[email protected]>--- kernel/trace/trace_probe.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)kernel/trace/trace_probe.c +3
-2@@ -314,7 +314,7 @@ static int parse_probe_vars(char *arg, const struct fetch_type *t, } }else goto inval_var; - }else if (strcmp (arg,"comm" ) ==0 ) { + }else if (strcmp (arg,"comm" ) ==0 ||strcmp (arg,"COMM" ) ==0 ) { code->op =FETCH_OP_COMM ; #ifdef CONFIG_HAVE_FUNCTION_ARG_ACCESS_API } else if (((flags & TPARG_FL_MASK) ==@@ -625,7 +625,8 @@ static int traceprobe_parse_probe_arg_body(const char *argv, ssize_t *size,* we can find those by strcmp. But ignore for eprobes. */ if (!(flags &TPARG_FL_TPOINT ) && - (strcmp (arg,"$comm" ) ==0 ||strncmp (arg,"\\\"" ,2 ) ==0 )) { + (strcmp (arg,"$comm" ) ==0 ||strcmp (arg,"$COMM" ) ==0 || +strncmp (arg,"\\\"" ,2 ) ==0 )) {/* The type of $comm must be "string", and not an array. */ if (parg->count || (t &&strcmp (t,"string" ))) goto out;Signature · 1 line
2.35.1
Signature · 1 line
Masami Hiramatsu (Google) <[email protected]>
Steven Rostedt
maintainerRe: [PATCH v2 5/6] tracing/probes: Have kprobes and uprobes use $COMM too
Quoted 3 lines · On Mon, 22 Aug 2022 00:19:02 +0900 Masami Hiramatsu (Google)…
This looks good to me.
Thanks Masami. I was holding off sending these in hoping that you would give an ack ;-)
Quoted 4 lines
(Note that kprobes/uprobes doesn't need to record cpu/pid, because those
are a part of common field and can be accessed from filter or histogram. Only comm must be recorded as string.)
Same for eprobes.
-- Steve