Skip to contentSkip to conversation

tracing/probe: Add multi-probes per event support

PATCH In review Masami Hiramatsu 2 participants 19 messages Jun 19, 2019 – Aug 1, 2019

Subsystems (3)
lore
Share & tools
lore

Masami Hiramatsu

author maintainer
#0 lore raw

v2 cover tracing/probe: Add multi-probes per event support

Hello,

This is the 2nd version of multi-probes per event support on ftrace and perf-tools.

Previous version is here;
https://lkml.org/lkml/2019/5/31/573

Quoted 1 line

From this version, I omitted first 9 patches which has been picked

to Steve's tree.
In this version, I've fixed some bugs and hardened some unexpected error cases according to Steve's comment.
Here are changes in this version:

- [1/12] This have below changes.
- Warn if the primary trace_probe does not exist.
- Fix enable_trace_kprobe() to not return error if the any probes are "gone" state. If all probes have gone or any other error reason, the event can not be enabled and return error.
- Fix trace_probe_enable() to roll back all enabled uprobe if any one of uprobe is failed to enable.
- [7/12] Swap the checking order of filename for avoiding unexpected memory access.

====
For trace-event, we can insert same trace-event on several places on the code, and those can record similar information as a same event with same format.

This series implements similar feature on probe-event. Since the probe event is based on the compiled binary, sometimes we find that the target source line is complied into several different addresses, e.g. inlined function, unrolled loop, etc. In those cases, it is useful to put a same probe-event on different addresses.

With this series, we can append multi probes on one event as below

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=%cx >> kprobe_events
  # kprobe_events
  p:kprobes/testevent _do_fork r1=%ax r2=%dx
  p:kprobes/testevent fork_idle r1=%ax r2=%cx

This means testevent is hit on both of _do_fork and fork_idle. As you can see, the appended event must have same number of arguments and those must have same 'type' and 'name' as original one. This is like a function signature, it checks whether the appending event has the same type and name of event arguments and same probe type, but doesn't care about the assignment.

So, below appending commands will be rejected.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax >> kprobe_events
  (No 2nd argument)
  # echo p:testevent fork_idle r1=%ax r2=%ax:x8 >> kprobe_events
  (The type of 2nd argument is different)

If one inlined code has an argument on a register, but another inlined code has fixed value (as a result of optimization), you can also specify the fixed immediate value, e.g.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=\1 >> kprobe_events

Thank you,

---

Masami Hiramatsu (12):
tracing/probe: Split trace_event related data from trace_probe tracing/dynevent: Delete all matched events tracing/dynevent: Pass extra arguments to match operation tracing/kprobe: Add multi-probe per event support tracing/uprobe: Add multi-probe per uprobe event support tracing/kprobe: Add per-probe delete from event tracing/uprobe: Add per-probe delete from event tracing/probe: Add immediate parameter support tracing/probe: Add immediate string parameter support selftests/ftrace: Add a testcase for kprobe multiprobe event selftests/ftrace: Add syntax error test for immediates selftests/ftrace: Add syntax error test for multiprobe

 Documentation/trace/kprobetrace.rst                |    1 
 Documentation/trace/uprobetracer.rst               |    1 
 kernel/trace/trace.c                               |    8 -
 kernel/trace/trace_dynevent.c                      |   10 +
 kernel/trace/trace_dynevent.h                      |    7 -
 kernel/trace/trace_events_hist.c                   |    4 
 kernel/trace/trace_kprobe.c                        |  241 ++++++++++++++----
 kernel/trace/trace_probe.c                         |  176 +++++++++++--
 kernel/trace/trace_probe.h                         |   67 ++++-
 kernel/trace/trace_uprobe.c                        |  263 +++++++++++++++-----
 tools/testing/selftests/ftrace/test.d/functions    |    2 
 .../ftrace/test.d/kprobe/kprobe_multiprobe.tc      |   35 +++
 .../ftrace/test.d/kprobe/kprobe_syntax_errors.tc   |   15 +
 13 files changed, 665 insertions(+), 165 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/kprobe_multiprobe.tc

--
Masami Hiramatsu (Linaro) <[email protected]>

author maintainer
#1 lore raw

PATCH v2 1/12 tracing/probe: Split trace_event related data from trace_probe

Split the trace_event related data from trace_probe data structure and introduce trace_probe_event data structure for its folder. This trace_probe_event data structure can have multiple trace_probe.

---
Changes in v2:
- Warn if the primary trace_probe does not exist according to Steve's comment. This must not happen but is easy to be handled.
- Fix enable_trace_kprobe() to not return error if the any probes are "gone" state. If all probes have gone or any other error reason, the event can not be enabled and return error.
- Fix trace_probe_enable() to roll back all enabled uprobe if any one of uprobe is failed to enable.

---
 kernel/trace/trace_kprobe.c |  157 ++++++++++++++++++++++++++++++-----------
 kernel/trace/trace_probe.c  |   53 +++++++++-----
 kernel/trace/trace_probe.h  |   48 ++++++++++---
 kernel/trace/trace_uprobe.c |  165 ++++++++++++++++++++++++++++++++-----------
 4 files changed, 311 insertions(+), 112 deletions(-)
kernel/trace/trace_kprobe.c +115 -42
index 9d483ad9bb6c..eac6344a2e7c 100644
@@ -180,20 +180,33 @@ unsigned long trace_kprobe_address(struct trace_kprobe *tk)
 	return addr;
 }
 
+static nokprobe_inline struct trace_kprobe *
+trace_kprobe_primary_from_call(struct trace_event_call *call)
+{
+	struct trace_probe *tp;
+
+	tp = trace_probe_primary_from_call(call);
+	if (WARN_ON_ONCE(!tp))
+		return NULL;
+
+	return container_of(tp, struct trace_kprobe, tp);
+}
+
 bool trace_kprobe_on_func_entry(struct trace_event_call *call)
 {
-	struct trace_kprobe *tk = (struct trace_kprobe *)call->data;
+	struct trace_kprobe *tk = trace_kprobe_primary_from_call(call);
 
-	return kprobe_on_func_entry(tk->rp.kp.addr,
+	return tk ? kprobe_on_func_entry(tk->rp.kp.addr,
 			tk->rp.kp.addr ? NULL : tk->rp.kp.symbol_name,
-			tk->rp.kp.addr ? 0 : tk->rp.kp.offset);
+			tk->rp.kp.addr ? 0 : tk->rp.kp.offset) : false;
 }
 
 bool trace_kprobe_error_injectable(struct trace_event_call *call)
 {
-	struct trace_kprobe *tk = (struct trace_kprobe *)call->data;
+	struct trace_kprobe *tk = trace_kprobe_primary_from_call(call);
 
-	return within_error_injection_list(trace_kprobe_address(tk));
+	return tk ? within_error_injection_list(trace_kprobe_address(tk)) :
+	       false;
 }
 
 static int register_kprobe_event(struct trace_kprobe *tk);
@@ -291,32 +304,75 @@ static inline int __enable_trace_kprobe(struct trace_kprobe *tk)
 	return ret;
 }
 
+static void __disable_trace_kprobe(struct trace_probe *tp)
+{
+	struct trace_probe *pos;
+	struct trace_kprobe *tk;
+
+	list_for_each_entry(pos, trace_probe_probe_list(tp), list) {
+		tk = container_of(pos, struct trace_kprobe, tp);
+		if (!trace_kprobe_is_registered(tk))
+			continue;
+		if (trace_kprobe_is_return(tk))
+			disable_kretprobe(&tk->rp);
+		else
+			disable_kprobe(&tk->rp.kp);
+	}
+}
+
 /*
  * Enable trace_probe
  * if the file is NULL, enable "perf" handler, or enable "trace" handler.
  */
-static int
-enable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file)
+static int enable_trace_kprobe(struct trace_event_call *call,
+				struct trace_event_file *file)
 {
-	bool enabled = trace_probe_is_enabled(&tk->tp);
+	struct trace_probe *pos, *tp;
+	struct trace_kprobe *tk;
+	bool enabled;
 	int ret = 0;
 
+	tp = trace_probe_primary_from_call(call);
+	if (WARN_ON_ONCE(!tp))
+		return -ENODEV;
+	enabled = trace_probe_is_enabled(tp);
+
+	/* This also changes "enabled" state */
 	if (file) {
-		ret = trace_probe_add_file(&tk->tp, file);
+		ret = trace_probe_add_file(tp, file);
 		if (ret)
 			return ret;
 	} else
-		trace_probe_set_flag(&tk->tp, TP_FLAG_PROFILE);
+		trace_probe_set_flag(tp, TP_FLAG_PROFILE);
 
 	if (enabled)
 		return 0;
 
-	ret = __enable_trace_kprobe(tk);
-	if (ret) {
+	enabled = false;
+	list_for_each_entry(pos, trace_probe_probe_list(tp), list) {
+		tk = container_of(pos, struct trace_kprobe, tp);
+		if (trace_kprobe_has_gone(tk))
+			continue;
+		ret = __enable_trace_kprobe(tk);
+		if (ret) {
+			if (enabled) {
+				__disable_trace_kprobe(tp);
+				enabled = false;
+			}
+			break;
+		}
+		enabled = true;
+	}
+
+	if (!enabled) {
+		/* No probe is enabled. Roll back */
 		if (file)
-			trace_probe_remove_file(&tk->tp, file);
+			trace_probe_remove_file(tp, file);
 		else
-			trace_probe_clear_flag(&tk->tp, TP_FLAG_PROFILE);
+			trace_probe_clear_flag(tp, TP_FLAG_PROFILE);
+		if (!ret)
+			/* Since all probes are gone, this is not available */
+			ret = -EADDRNOTAVAIL;
 	}
 
 	return ret;
@@ -326,11 +382,14 @@ enable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file)
  * Disable trace_probe
  * if the file is NULL, disable "perf" handler, or disable "trace" handler.
  */
-static int
-disable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file)
+static int disable_trace_kprobe(struct trace_event_call *call,
+				struct trace_event_file *file)
 {
-	struct trace_probe *tp = &tk->tp;
-	int ret = 0;
+	struct trace_probe *tp;
+
+	tp = trace_probe_primary_from_call(call);
+	if (WARN_ON_ONCE(!tp))
+		return -ENODEV;
 
 	if (file) {
 		if (!trace_probe_get_file_link(tp, file))
@@ -341,12 +400,8 @@ disable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file)
 	} else
 		trace_probe_clear_flag(tp, TP_FLAG_PROFILE);
 
-	if (!trace_probe_is_enabled(tp) && trace_kprobe_is_registered(tk)) {
-		if (trace_kprobe_is_return(tk))
-			disable_kretprobe(&tk->rp);
-		else
-			disable_kprobe(&tk->rp.kp);
-	}
+	if (!trace_probe_is_enabled(tp))
+		__disable_trace_kprobe(tp);
 
  out:
 	if (file)
@@ -358,7 +413,7 @@ disable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file)
 		 */
 		trace_probe_remove_file(tp, file);
 
-	return ret;
+	return 0;
 }
 
 #if defined(CONFIG_KPROBES_ON_FTRACE) && \
@@ -1089,7 +1144,10 @@ print_kprobe_event(struct trace_iterator *iter, int flags,
 	struct trace_probe *tp;
 
 	field = (struct kprobe_trace_entry_head *)iter->ent;
-	tp = container_of(event, struct trace_probe, call.event);
+	tp = trace_probe_primary_from_call(
+		container_of(event, struct trace_event_call, event));
+	if (WARN_ON_ONCE(!tp))
+		goto out;
 
 	trace_seq_printf(s, "%s: (", trace_probe_name(tp));
 
@@ -1116,7 +1174,10 @@ print_kretprobe_event(struct trace_iterator *iter, int flags,
 	struct trace_probe *tp;
 
 	field = (struct kretprobe_trace_entry_head *)iter->ent;
-	tp = container_of(event, struct trace_probe, call.event);
+	tp = trace_probe_primary_from_call(
+		container_of(event, struct trace_event_call, event));
+	if (WARN_ON_ONCE(!tp))
+		goto out;
 
 	trace_seq_printf(s, "%s: (", trace_probe_name(tp));
 
@@ -1145,23 +1206,31 @@ static int kprobe_event_define_fields(struct trace_event_call *event_call)
 {
 	int ret;
 	struct kprobe_trace_entry_head field;
-	struct trace_kprobe *tk = (struct trace_kprobe *)event_call->data;
+	struct trace_probe *tp;
+
+	tp = trace_probe_primary_from_call(event_call);
+	if (WARN_ON_ONCE(!tp))
+		return -ENOENT;
 
 	DEFINE_FIELD(unsigned long, ip, FIELD_STRING_IP, 0);
 
-	return traceprobe_define_arg_fields(event_call, sizeof(field), &tk->tp);
+	return traceprobe_define_arg_fields(event_call, sizeof(field), tp);
 }
 
 static int kretprobe_event_define_fields(struct trace_event_call *event_call)
 {
 	int ret;
 	struct kretprobe_trace_entry_head field;
-	struct trace_kprobe *tk = (struct trace_kprobe *)event_call->data;
+	struct trace_probe *tp;
+
+	tp = trace_probe_primary_from_call(event_call);
+	if (WARN_ON_ONCE(!tp))
+		return -ENOENT;
 
 	DEFINE_FIELD(unsigned long, func, FIELD_STRING_FUNC, 0);
 	DEFINE_FIELD(unsigned long, ret_ip, FIELD_STRING_RETIP, 0);
 
-	return traceprobe_define_arg_fields(event_call, sizeof(field), &tk->tp);
+	return traceprobe_define_arg_fields(event_call, sizeof(field), tp);
 }
 
 #ifdef CONFIG_PERF_EVENTS
@@ -1289,20 +1358,19 @@ int bpf_get_kprobe_info(const struct perf_event *event, u32 *fd_type,
 static int kprobe_register(struct trace_event_call *event,
 			   enum trace_reg type, void *data)
 {
-	struct trace_kprobe *tk = (struct trace_kprobe *)event->data;
 	struct trace_event_file *file = data;
 
 	switch (type) {
 	case TRACE_REG_REGISTER:
-		return enable_trace_kprobe(tk, file);
+		return enable_trace_kprobe(event, file);
 	case TRACE_REG_UNREGISTER:
-		return disable_trace_kprobe(tk, file);
+		return disable_trace_kprobe(event, file);
 
 #ifdef CONFIG_PERF_EVENTS
 	case TRACE_REG_PERF_REGISTER:
-		return enable_trace_kprobe(tk, NULL);
+		return enable_trace_kprobe(event, NULL);
 	case TRACE_REG_PERF_UNREGISTER:
-		return disable_trace_kprobe(tk, NULL);
+		return disable_trace_kprobe(event, NULL);
 	case TRACE_REG_PERF_OPEN:
 	case TRACE_REG_PERF_CLOSE:
 	case TRACE_REG_PERF_ADD:
@@ -1369,7 +1437,6 @@ static inline void init_trace_event_call(struct trace_kprobe *tk)
 
 	call->flags = TRACE_EVENT_FL_KPROBE;
 	call->class->reg = kprobe_register;
-	call->data = tk;
 }
 
 static int register_kprobe_event(struct trace_kprobe *tk)
@@ -1432,7 +1499,9 @@ void destroy_local_trace_kprobe(struct trace_event_call *event_call)
 {
 	struct trace_kprobe *tk;
 
-	tk = container_of(event_call, struct trace_kprobe, tp.call);
+	tk = trace_kprobe_primary_from_call(event_call);
+	if (unlikely(!tk))
+		return;
 
 	if (trace_probe_is_enabled(&tk->tp)) {
 		WARN_ON(1);
@@ -1577,7 +1646,8 @@ static __init int kprobe_trace_self_tests_init(void)
 				pr_warn("error on getting probe file.\n");
 				warn++;
 			} else
-				enable_trace_kprobe(tk, file);
+				enable_trace_kprobe(
+					trace_probe_event_call(&tk->tp), file);
 		}
 	}
 
@@ -1598,7 +1668,8 @@ static __init int kprobe_trace_self_tests_init(void)
 				pr_warn("error on getting probe file.\n");
 				warn++;
 			} else
-				enable_trace_kprobe(tk, file);
+				enable_trace_kprobe(
+					trace_probe_event_call(&tk->tp), file);
 		}
 	}
 
@@ -1631,7 +1702,8 @@ static __init int kprobe_trace_self_tests_init(void)
 			pr_warn("error on getting probe file.\n");
 			warn++;
 		} else
-			disable_trace_kprobe(tk, file);
+			disable_trace_kprobe(
+				trace_probe_event_call(&tk->tp), file);
 	}
 
 	tk = find_trace_kprobe("testprobe2", KPROBE_EVENT_SYSTEM);
@@ -1649,7 +1721,8 @@ static __init int kprobe_trace_self_tests_init(void)
 			pr_warn("error on getting probe file.\n");
 			warn++;
 		} else
-			disable_trace_kprobe(tk, file);
+			disable_trace_kprobe(
+				trace_probe_event_call(&tk->tp), file);
 	}
 
 	ret = trace_run_command("-:testprobe", create_or_delete_trace_kprobe);
kernel/trace/trace_probe.c +36 -17

90 lines not on this page · raw message

kernel/trace/trace_probe.h +37 -11

105 lines not on this page · raw message

kernel/trace/trace_uprobe.c +123 -42

271 lines not on this page · raw message

author maintainer
#2 lore raw

PATCH v2 2/12 tracing/dynevent: Delete all matched events

In reply to Masami Hiramatsu

When user gives an event name to delete, delete all matched events instead of the first one.

This means if there are several events which have same name but different group (subsystem) name, those are removed if user passed only the event name, e.g.

  # cat kprobe_events
  p:group1/testevent _do_fork
  p:group2/testevent fork_idle
  # echo -:testevent >> kprobe_events
  # cat kprobe_events
  #
---
 kernel/trace/trace_dynevent.c |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
kernel/trace/trace_dynevent.c +5 -3
index fa100ed3b4de..1cc55c50c491 100644
@@ -61,10 +61,12 @@ int dyn_event_release(int argc, char **argv, struct dyn_event_operations *type)
 	for_each_dyn_event_safe(pos, n) {
 		if (type && type != pos->ops)
 			continue;
-		if (pos->ops->match(system, event, pos)) {
-			ret = pos->ops->free(pos);
+		if (!pos->ops->match(system, event, pos))
+			continue;
+
+		ret = pos->ops->free(pos);
+		if (ret)
 			break;
-		}
 	}
 	mutex_unlock(&event_mutex);
 
author maintainer
#3 lore raw

PATCH v2 3/12 tracing/dynevent: Pass extra arguments to match operation

In reply to Masami Hiramatsu

Pass extra arguments to match operation for checking exact match. If the event doesn't support exact match, it will be ignored.

---
 kernel/trace/trace_dynevent.c    |    4 +++-
 kernel/trace/trace_dynevent.h    |    7 ++++---
 kernel/trace/trace_events_hist.c |    4 ++--
 kernel/trace/trace_kprobe.c      |    4 ++--
 kernel/trace/trace_uprobe.c      |    4 ++--
 5 files changed, 13 insertions(+), 10 deletions(-)
kernel/trace/trace_dynevent.c +3 -1
index 1cc55c50c491..a41fed46c285 100644
@@ -47,6 +47,7 @@ int dyn_event_release(int argc, char **argv, struct dyn_event_operations *type)
 			return -EINVAL;
 		event++;
 	}
+	argc--; argv++;
 
 	p = strchr(event, '/');
 	if (p) {
@@ -61,7 +62,8 @@ int dyn_event_release(int argc, char **argv, struct dyn_event_operations *type)
 	for_each_dyn_event_safe(pos, n) {
 		if (type && type != pos->ops)
 			continue;
-		if (!pos->ops->match(system, event, pos))
+		if (!pos->ops->match(system, event,
+				argc, (const char **)argv, pos))
 			continue;
 
 		ret = pos->ops->free(pos);
kernel/trace/trace_dynevent.h +4 -3
index 8c334064e4d6..46898138d2df 100644
@@ -31,8 +31,9 @@ struct dyn_event;
  * @is_busy: Check whether given event is busy so that it can not be deleted.
  *  Return true if it is busy, otherwides false.
  * @free: Delete the given event. Return 0 if success, otherwides error.
- * @match: Check whether given event and system name match this event.
- *  Return true if it matches, otherwides false.
+ * @match: Check whether given event and system name match this event. The argc
+ *  and argv is used for exact match. Return true if it matches, otherwides
+ *  false.
  *
  * Except for @create, these methods are called under holding event_mutex.
  */
@@ -43,7 +44,7 @@ struct dyn_event_operations {
 	bool (*is_busy)(struct dyn_event *ev);
 	int (*free)(struct dyn_event *ev);
 	bool (*match)(const char *system, const char *event,
-			struct dyn_event *ev);
+		      int argc, const char **argv, struct dyn_event *ev);
 };
 
 /* Register new dyn_event type -- must be called at first */
kernel/trace/trace_events_hist.c +2 -2
index ca6b0dff60c5..65e7d071ed28 100644
@@ -374,7 +374,7 @@ static int synth_event_show(struct seq_file *m, struct dyn_event *ev);
 static int synth_event_release(struct dyn_event *ev);
 static bool synth_event_is_busy(struct dyn_event *ev);
 static bool synth_event_match(const char *system, const char *event,
-			      struct dyn_event *ev);
+			int argc, const char **argv, struct dyn_event *ev);
 
 static struct dyn_event_operations synth_event_ops = {
 	.create = synth_event_create,
@@ -422,7 +422,7 @@ static bool synth_event_is_busy(struct dyn_event *ev)
 }
 
 static bool synth_event_match(const char *system, const char *event,
-			      struct dyn_event *ev)
+			int argc, const char **argv, struct dyn_event *ev)
 {
 	struct synth_event *sev = to_synth_event(ev);
 
kernel/trace/trace_kprobe.c +2 -2
index eac6344a2e7c..e8f72431b866 100644
@@ -39,7 +39,7 @@ static int trace_kprobe_show(struct seq_file *m, struct dyn_event *ev);
 static int trace_kprobe_release(struct dyn_event *ev);
 static bool trace_kprobe_is_busy(struct dyn_event *ev);
 static bool trace_kprobe_match(const char *system, const char *event,
-			       struct dyn_event *ev);
+			int argc, const char **argv, struct dyn_event *ev);
 
 static struct dyn_event_operations trace_kprobe_ops = {
 	.create = trace_kprobe_create,
@@ -138,7 +138,7 @@ static bool trace_kprobe_is_busy(struct dyn_event *ev)
 }
 
 static bool trace_kprobe_match(const char *system, const char *event,
-			       struct dyn_event *ev)
+			int argc, const char **argv, struct dyn_event *ev)
 {
 	struct trace_kprobe *tk = to_trace_kprobe(ev);
 
kernel/trace/trace_uprobe.c +2 -2
index 5b45f0a7ab38..437dddeca8bf 100644
@@ -44,7 +44,7 @@ static int trace_uprobe_show(struct seq_file *m, struct dyn_event *ev);
 static int trace_uprobe_release(struct dyn_event *ev);
 static bool trace_uprobe_is_busy(struct dyn_event *ev);
 static bool trace_uprobe_match(const char *system, const char *event,
-			       struct dyn_event *ev);
+			int argc, const char **argv, struct dyn_event *ev);
 
 static struct dyn_event_operations trace_uprobe_ops = {
 	.create = trace_uprobe_create,
@@ -285,7 +285,7 @@ static bool trace_uprobe_is_busy(struct dyn_event *ev)
 }
 
 static bool trace_uprobe_match(const char *system, const char *event,
-			       struct dyn_event *ev)
+			int argc, const char **argv, struct dyn_event *ev)
 {
 	struct trace_uprobe *tu = to_trace_uprobe(ev);
 
author maintainer
#4 lore raw

PATCH v2 4/12 tracing/kprobe: Add multi-probe per event support

In reply to Masami Hiramatsu

Add multi-probe per one event support to kprobe events. User can define several different probes on one trace event if those events have same "event signature",
e.g.

  # echo p:testevent _do_fork > kprobe_events
  # echo p:testevent fork_idle >> kprobe_events
  # kprobe_events
  p:kprobes/testevent _do_fork
  p:kprobes/testevent fork_idle

The event signature is defined by kprobe type (retprobe or not), the number of args, argument names, and argument types.

Note that this only support appending method. Delete event operation will delete all probes on the event.

---
 kernel/trace/trace.c        |    4 +--
 kernel/trace/trace_kprobe.c |   52 ++++++++++++++++++++++++++++++++++----
 kernel/trace/trace_probe.c  |   59 ++++++++++++++++++++++++++++++++++++-------
 kernel/trace/trace_probe.h  |   14 +++++++++-
 4 files changed, 111 insertions(+), 18 deletions(-)
kernel/trace/trace.c +2 -2
index 77b9c4ca5faa..8f5455046504 100644
@@ -4813,11 +4813,11 @@ static const char readme_msg[] =
 #endif
 #endif /* CONFIG_STACK_TRACER */
 #ifdef CONFIG_DYNAMIC_EVENTS
-	"  dynamic_events\t\t- Add/remove/show the generic dynamic events\n"
+	"  dynamic_events\t\t- Create/append/remove/show the generic dynamic events\n"
 	"\t\t\t  Write into this file to define/undefine new trace events.\n"
 #endif
 #ifdef CONFIG_KPROBE_EVENTS
-	"  kprobe_events\t\t- Add/remove/show the kernel dynamic events\n"
+	"  kprobe_events\t\t- Create/append/remove/show the kernel dynamic events\n"
 	"\t\t\t  Write into this file to define/undefine new trace events.\n"
 #endif
 #ifdef CONFIG_UPROBE_EVENTS
kernel/trace/trace_kprobe.c +46 -6

85 lines not on this page · raw message

kernel/trace/trace_probe.c +50 -9

74 lines not on this page · raw message

kernel/trace/trace_probe.h +13 -1
index 0b84abb884c2..39926e8a344b 100644
@@ -302,6 +302,13 @@ static inline struct list_head *trace_probe_probe_list(struct trace_probe *tp)
 	return &tp->event->probes;
 }
 
+static inline bool trace_probe_has_sibling(struct trace_probe *tp)
+{
+	struct list_head *list = trace_probe_probe_list(tp);
+
+	return !list_empty(list) && !list_is_singular(list);
+}
+
 static inline int trace_probe_unregister_event_call(struct trace_probe *tp)
 {
 	/* tp->event is unregistered in trace_remove_event_call() */
@@ -316,12 +323,15 @@ static inline bool trace_probe_has_single_file(struct trace_probe *tp)
 int trace_probe_init(struct trace_probe *tp, const char *event,
 		     const char *group);
 void trace_probe_cleanup(struct trace_probe *tp);
+int trace_probe_append(struct trace_probe *tp, struct trace_probe *to);
+void trace_probe_unlink(struct trace_probe *tp);
 int trace_probe_register_event_call(struct trace_probe *tp);
 int trace_probe_add_file(struct trace_probe *tp, struct trace_event_file *file);
 int trace_probe_remove_file(struct trace_probe *tp,
 			    struct trace_event_file *file);
 struct event_file_link *trace_probe_get_file_link(struct trace_probe *tp,
 						struct trace_event_file *file);
+int trace_probe_compare_arg_type(struct trace_probe *a, struct trace_probe *b);
 
 #define trace_probe_for_each_link(pos, tp)	\
 	list_for_each_entry(pos, &(tp)->event->files, list)
@@ -419,7 +429,9 @@ extern int traceprobe_define_arg_fields(struct trace_event_call *event_call,
 	C(ARG_TOO_LONG,		"Argument expression is too long"),	\
 	C(NO_ARG_BODY,		"No argument expression"),		\
 	C(BAD_INSN_BNDRY,	"Probe point is not an instruction boundary"),\
-	C(FAIL_REG_PROBE,	"Failed to register probe event"),
+	C(FAIL_REG_PROBE,	"Failed to register probe event"),\
+	C(DIFF_PROBE_TYPE,	"Probe type is different from existing probe"),\
+	C(DIFF_ARG_TYPE,	"Argument type or name is different from existing probe"),
 
 #undef C
 #define C(a, b)		TP_ERR_##a
author maintainer
#5 lore raw

PATCH v2 5/12 tracing/uprobe: Add multi-probe per uprobe event support

In reply to Masami Hiramatsu

Allow user to define several probes on one uprobe event. Note that this only support appending method. So deleting event will delete all probes on the event.

---
 kernel/trace/trace.c        |    2 +
 kernel/trace/trace_uprobe.c |   60 ++++++++++++++++++++++++++++++-------------
 2 files changed, 43 insertions(+), 19 deletions(-)
kernel/trace/trace.c +1 -1
index 8f5455046504..73fbe3b0dd08 100644
@@ -4821,7 +4821,7 @@ static const char readme_msg[] =
 	"\t\t\t  Write into this file to define/undefine new trace events.\n"
 #endif
 #ifdef CONFIG_UPROBE_EVENTS
-	"  uprobe_events\t\t- Add/remove/show the userspace dynamic events\n"
+	"  uprobe_events\t\t- Create/append/remove/show the userspace dynamic events\n"
 	"\t\t\t  Write into this file to define/undefine new trace events.\n"
 #endif
 #if defined(CONFIG_KPROBE_EVENTS) || defined(CONFIG_UPROBE_EVENTS)
kernel/trace/trace_uprobe.c +42 -18

100 lines not on this page · raw message

author maintainer
#6 lore raw

PATCH v2 6/12 tracing/kprobe: Add per-probe delete from event

In reply to Masami Hiramatsu

Allow user to delete a probe from event. This is done by head match. For example, if we have 2 probes on an event

$ cat kprobe_events
p:kprobes/testprobe _do_fork r1=%ax r2=%dx
p:kprobes/testprobe idle_fork r1=%ax r2=%cx

Then you can remove one of them by passing the head of definition which identify the probe.

$ echo "-:kprobes/testprobe idle_fork" >> kprobe_events
---
 kernel/trace/trace_kprobe.c |   25 ++++++++++++++++++++++++-
 kernel/trace/trace_probe.c  |   18 ++++++++++++++++++
 kernel/trace/trace_probe.h  |    2 ++
 3 files changed, 44 insertions(+), 1 deletion(-)
kernel/trace/trace_kprobe.c +24 -1
index f43098bf62dd..18c4175b6585 100644
@@ -137,13 +137,36 @@ static bool trace_kprobe_is_busy(struct dyn_event *ev)
 	return trace_probe_is_enabled(&tk->tp);
 }
 
+static bool trace_kprobe_match_command_head(struct trace_kprobe *tk,
+					    int argc, const char **argv)
+{
+	char buf[MAX_ARGSTR_LEN + 1];
+
+	if (!argc)
+		return true;
+
+	if (!tk->symbol)
+		snprintf(buf, sizeof(buf), "0x%p", tk->rp.kp.addr);
+	else if (tk->rp.kp.offset)
+		snprintf(buf, sizeof(buf), "%s+%u",
+			 trace_kprobe_symbol(tk), tk->rp.kp.offset);
+	else
+		snprintf(buf, sizeof(buf), "%s", trace_kprobe_symbol(tk));
+	if (strcmp(buf, argv[0]))
+		return false;
+	argc--; argv++;
+
+	return trace_probe_match_command_args(&tk->tp, argc, argv);
+}
+
 static bool trace_kprobe_match(const char *system, const char *event,
 			int argc, const char **argv, struct dyn_event *ev)
 {
 	struct trace_kprobe *tk = to_trace_kprobe(ev);
 
 	return strcmp(trace_probe_name(&tk->tp), event) == 0 &&
-	    (!system || strcmp(trace_probe_group_name(&tk->tp), system) == 0);
+	    (!system || strcmp(trace_probe_group_name(&tk->tp), system) == 0) &&
+	    trace_kprobe_match_command_head(tk, argc, argv);
 }
 
 static nokprobe_inline unsigned long trace_kprobe_nhit(struct trace_kprobe *tk)
kernel/trace/trace_probe.c +18 -0
index 651a1449acde..f8c3c65c035d 100644
@@ -1047,3 +1047,21 @@ int trace_probe_compare_arg_type(struct trace_probe *a, struct trace_probe *b)
 
 	return 0;
 }
+
+bool trace_probe_match_command_args(struct trace_probe *tp,
+				    int argc, const char **argv)
+{
+	char buf[MAX_ARGSTR_LEN + 1];
+	int i;
+
+	if (tp->nr_args < argc)
+		return false;
+
+	for (i = 0; i < argc; i++) {
+		snprintf(buf, sizeof(buf), "%s=%s",
+			 tp->args[i].name, tp->args[i].comm);
+		if (strcmp(buf, argv[i]))
+			return false;
+	}
+	return true;
+}
kernel/trace/trace_probe.h +2 -0
index 39926e8a344b..2dcc4e317787 100644
@@ -332,6 +332,8 @@ int trace_probe_remove_file(struct trace_probe *tp,
 struct event_file_link *trace_probe_get_file_link(struct trace_probe *tp,
 						struct trace_event_file *file);
 int trace_probe_compare_arg_type(struct trace_probe *a, struct trace_probe *b);
+bool trace_probe_match_command_args(struct trace_probe *tp,
+				    int argc, const char **argv);
 
 #define trace_probe_for_each_link(pos, tp)	\
 	list_for_each_entry(pos, &(tp)->event->files, list)
author maintainer
#7 lore raw

PATCH v2 7/12 tracing/uprobe: Add per-probe delete from event

In reply to Masami Hiramatsu

Add per-probe delete method from one event passing the head of definition. In other words, the events which match the head N parameters are deleted.

---
Changes in v2:
- Swap the checking order of filename for avoiding unexpected memory access.

---
 kernel/trace/trace_uprobe.c |   31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
kernel/trace/trace_uprobe.c +30 -1
index c568f7b16f5c..878219552177 100644
@@ -284,13 +284,42 @@ static bool trace_uprobe_is_busy(struct dyn_event *ev)
 	return trace_probe_is_enabled(&tu->tp);
 }
 
+static bool trace_uprobe_match_command_head(struct trace_uprobe *tu,
+					    int argc, const char **argv)
+{
+	char buf[MAX_ARGSTR_LEN + 1];
+	int len;
+
+	if (!argc)
+		return true;
+
+	len = strlen(tu->filename);
+	if (strncmp(tu->filename, argv[0], len) || argv[0][len] != ':')
+		return false;
+
+	if (tu->ref_ctr_offset == 0)
+		snprintf(buf, sizeof(buf), "0x%0*lx",
+				(int)(sizeof(void *) * 2), tu->offset);
+	else
+		snprintf(buf, sizeof(buf), "0x%0*lx(0x%lx)",
+				(int)(sizeof(void *) * 2), tu->offset,
+				tu->ref_ctr_offset);
+	if (strcmp(buf, &argv[0][len + 1]))
+		return false;
+
+	argc--; argv++;
+
+	return trace_probe_match_command_args(&tu->tp, argc, argv);
+}
+
 static bool trace_uprobe_match(const char *system, const char *event,
 			int argc, const char **argv, struct dyn_event *ev)
 {
 	struct trace_uprobe *tu = to_trace_uprobe(ev);
 
 	return strcmp(trace_probe_name(&tu->tp), event) == 0 &&
-	    (!system || strcmp(trace_probe_group_name(&tu->tp), system) == 0);
+	   (!system || strcmp(trace_probe_group_name(&tu->tp), system) == 0) &&
+	   trace_uprobe_match_command_head(tu, argc, argv);
 }
 
 static nokprobe_inline struct trace_uprobe *
author maintainer
#8 lore raw

PATCH v2 8/12 tracing/probe: Add immediate parameter support

In reply to Masami Hiramatsu

Add immediate value parameter (\1234) support to probe events. This allows you to specify an immediate (or dummy) parameter instead of fetching from memory or register.

This feature looks odd, but imagine when you put a probe on a code to trace some data. If the code is compiled into 2 instructions and 1 instruction has a value but other has nothing since it is optimized out.
In that case, you can not fold those into one event, even if ftrace supported multiple probes on one event. With this feature, you can set a dummy value like foo=\deadbeef instead of something like foo=%di.

---
 Documentation/trace/kprobetrace.rst  |    1 +
 Documentation/trace/uprobetracer.rst |    1 +
 kernel/trace/trace.c                 |    2 +-
 kernel/trace/trace_probe.c           |   18 ++++++++++++++++++
 kernel/trace/trace_probe.h           |    1 +
 5 files changed, 22 insertions(+), 1 deletion(-)
Documentation/trace/kprobetrace.rst +1 -0
index af776989caca..772467f65a36 100644
@@ -52,6 +52,7 @@ Synopsis of kprobe_events
   $retval	: Fetch return value.(\*2)
   $comm		: Fetch current task comm.
   +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*3)(\*4)
+  \IMM		: Store an immediate value to the argument.
   NAME=FETCHARG : Set NAME as the argument name of FETCHARG.
   FETCHARG:TYPE : Set TYPE as the type of FETCHARG. Currently, basic types
 		  (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
Documentation/trace/uprobetracer.rst +1 -0
index ab13319c66ac..2b4697c0bed7 100644
@@ -45,6 +45,7 @@ Synopsis of uprobe_tracer
    $retval	: Fetch return value.(\*1)
    $comm	: Fetch current task comm.
    +|-[u]OFFS(FETCHARG) : Fetch memory at FETCHARG +|- OFFS address.(\*2)(\*3)
+   \IMM		: Store an immediate value to the argument.
    NAME=FETCHARG     : Set NAME as the argument name of FETCHARG.
    FETCHARG:TYPE     : Set TYPE as the type of FETCHARG. Currently, basic types
 		       (u8/u16/u32/u64/s8/s16/s32/s64), hexadecimal types
kernel/trace/trace.c +1 -1
index 73fbe3b0dd08..3608535f1935 100644
@@ -4846,7 +4846,7 @@ static const char readme_msg[] =
 #else
 	"\t           $stack<index>, $stack, $retval, $comm,\n"
 #endif
-	"\t           +|-[u]<offset>(<fetcharg>)\n"
+	"\t           +|-[u]<offset>(<fetcharg>), \\imm-value\n"
 	"\t     type: s8/16/32/64, u8/16/32/64, x8/16/32/64, string, symbol,\n"
 	"\t           b<bit-width>@<bit-offset>/<container-size>, ustring,\n"
 	"\t           <type>\\[<array-size>\\]\n"
kernel/trace/trace_probe.c +18 -0
index f8c3c65c035d..fb90baec3cd8 100644
@@ -316,6 +316,17 @@ static int parse_probe_vars(char *arg, const struct fetch_type *t,
 	return -EINVAL;
 }
 
+static int str_to_immediate(char *str, unsigned long *imm)
+{
+	if (isdigit(str[0]))
+		return kstrtoul(str, 0, imm);
+	else if (str[0] == '-')
+		return kstrtol(str, 0, (long *)imm);
+	else if (str[0] == '+')
+		return kstrtol(str + 1, 0, (long *)imm);
+	return -EINVAL;
+}
+
 /* Recursive argument parser */
 static int
 parse_probe_arg(char *arg, const struct fetch_type *type,
@@ -444,6 +455,13 @@ parse_probe_arg(char *arg, const struct fetch_type *type,
 			code->offset = offset;
 		}
 		break;
+	case '\\':	/* Immediate value */
+		ret = str_to_immediate(arg + 1, &code->immediate);
+		if (ret)
+			trace_probe_log_err(offs + 1, BAD_IMM);
+		else
+			code->op = FETCH_OP_IMM;
+		break;
 	}
 	if (!ret && code->op == FETCH_OP_NOP) {
 		/* Parsed, but do not find fetch method */
kernel/trace/trace_probe.h +1 -0
index 2dcc4e317787..cc113b82a4ce 100644
@@ -408,6 +408,7 @@ extern int traceprobe_define_arg_fields(struct trace_event_call *event_call,
 	C(BAD_VAR,		"Invalid $-valiable specified"),	\
 	C(BAD_REG_NAME,		"Invalid register name"),		\
 	C(BAD_MEM_ADDR,		"Invalid memory address"),		\
+	C(BAD_IMM,		"Invalid immediate value"),		\
 	C(FILE_ON_KPROBE,	"File offset is not available with kprobe"), \
 	C(BAD_FILE_OFFS,	"Invalid file offset value"),		\
 	C(SYM_ON_UPROBE,	"Symbol is not available with uprobe"),	\
author maintainer
#9 lore raw

PATCH v2 9/12 tracing/probe: Add immediate string parameter support

In reply to Masami Hiramatsu

Add immediate string parameter (\"string") support to probe events. This allows you to specify an immediate (or dummy) parameter instead of fetching a string from memory.

This feature looks odd, but imagine that you put a probe on a code to trace some string data. If the code is compiled into 2 instructions and 1 instruction has a string on memory but other has no string since it is optimized out. In that case, you can not fold those into one event, even if ftrace supported multiple probes on one event. With this feature, you can set a dummy string like foo=\"(optimized)":string instead of something like foo=+0(+0(%bp)):string.

---
 kernel/trace/trace.c        |    2 +-
 kernel/trace/trace_kprobe.c |    3 ++
 kernel/trace/trace_probe.c  |   56 ++++++++++++++++++++++++++++++++-----------
 kernel/trace/trace_probe.h  |    2 ++
 kernel/trace/trace_uprobe.c |    3 ++
 5 files changed, 51 insertions(+), 15 deletions(-)
kernel/trace/trace.c +1 -1
index 3608535f1935..ff9c9e627a87 100644
@@ -4846,7 +4846,7 @@ static const char readme_msg[] =
 #else
 	"\t           $stack<index>, $stack, $retval, $comm,\n"
 #endif
-	"\t           +|-[u]<offset>(<fetcharg>), \\imm-value\n"
+	"\t           +|-[u]<offset>(<fetcharg>), \\imm-value, \\\"imm-string\"\n"
 	"\t     type: s8/16/32/64, u8/16/32/64, x8/16/32/64, string, symbol,\n"
 	"\t           b<bit-width>@<bit-offset>/<container-size>, ustring,\n"
 	"\t           <type>\\[<array-size>\\]\n"
kernel/trace/trace_kprobe.c +3 -0
index 18c4175b6585..7579c53bb053 100644
@@ -1083,6 +1083,9 @@ process_fetch_insn(struct fetch_insn *code, struct pt_regs *regs, void *dest,
 	case FETCH_OP_COMM:
 		val = (unsigned long)current->comm;
 		break;
+	case FETCH_OP_DATA:
+		val = (unsigned long)code->data;
+		break;
 #ifdef CONFIG_HAVE_FUNCTION_ARG_ACCESS_API
 	case FETCH_OP_ARG:
 		val = regs_get_kernel_argument(regs, code->param);
kernel/trace/trace_probe.c +42 -14

104 lines not on this page · raw message

kernel/trace/trace_probe.h +2 -0
index cc113b82a4ce..f805cc4cbe7c 100644
@@ -89,6 +89,7 @@ enum fetch_op {
 	FETCH_OP_COMM,		/* Current comm */
 	FETCH_OP_ARG,		/* Function argument : .param */
 	FETCH_OP_FOFFS,		/* File offset: .immediate */
+	FETCH_OP_DATA,		/* Allocated data: .data */
 	// Stage 2 (dereference) op
 	FETCH_OP_DEREF,		/* Dereference: .offset */
 	FETCH_OP_UDEREF,	/* User-space Dereference: .offset */
@@ -409,6 +410,7 @@ extern int traceprobe_define_arg_fields(struct trace_event_call *event_call,
 	C(BAD_REG_NAME,		"Invalid register name"),		\
 	C(BAD_MEM_ADDR,		"Invalid memory address"),		\
 	C(BAD_IMM,		"Invalid immediate value"),		\
+	C(IMMSTR_NO_CLOSE,	"String is not closed with '\"'"),	\
 	C(FILE_ON_KPROBE,	"File offset is not available with kprobe"), \
 	C(BAD_FILE_OFFS,	"Invalid file offset value"),		\
 	C(SYM_ON_UPROBE,	"Symbol is not available with uprobe"),	\
kernel/trace/trace_uprobe.c +3 -0
index 878219552177..89ab3972fc77 100644
@@ -248,6 +248,9 @@ process_fetch_insn(struct fetch_insn *code, struct pt_regs *regs, void *dest,
 	case FETCH_OP_COMM:
 		val = FETCH_TOKEN_COMM;
 		break;
+	case FETCH_OP_DATA:
+		val = (unsigned long)code->data;
+		break;
 	case FETCH_OP_FOFFS:
 		val = translate_user_vaddr(code->immediate);
 		break;
author maintainer
#10 lore raw

PATCH v2 10/12 selftests/ftrace: Add a testcase for kprobe multiprobe event

In reply to Masami Hiramatsu

Add a testcase for kprobe event with multi-probe.

---
 .../ftrace/test.d/kprobe/kprobe_multiprobe.tc      |   35 ++++++++++++++++++++
 1 file changed, 35 insertions(+)
 create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/kprobe_multiprobe.tc
tools/testing/selftests/ftrace/test.d/kprobe/kprobe_multiprobe.tc new file +35 -0
new file mode 100644
index 000000000000..44494bac86d1
@@ -0,0 +1,35 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: Create/delete multiprobe on kprobe event
+
+[ -f kprobe_events ] || exit_unsupported
+
+grep -q "Create/append/" README || exit_unsupported
+
+# Choose 2 symbols for target
+SYM1=_do_fork
+SYM2=do_exit
+EVENT_NAME=kprobes/testevent
+
+DEF1="p:$EVENT_NAME $SYM1"
+DEF2="p:$EVENT_NAME $SYM2"
+
+:;: "Define an event which has 2 probes" ;:
+echo $DEF1 >> kprobe_events
+echo $DEF2 >> kprobe_events
+cat kprobe_events | grep "$DEF1"
+cat kprobe_events | grep "$DEF2"
+
+:;: "Remove the event by name (should remove both)" ;:
+echo "-:$EVENT_NAME" >> kprobe_events
+test `cat kprobe_events | wc -l` -eq 0
+
+:;: "Remove just 1 event" ;:
+echo $DEF1 >> kprobe_events
+echo $DEF2 >> kprobe_events
+echo "-:$EVENT_NAME $SYM1" >> kprobe_events
+! cat kprobe_events | grep "$DEF1"
+cat kprobe_events | grep "$DEF2"
+
+:;: "Appending different type must fail" ;:
+! echo "$DEF1 \$stack" >> kprobe_events
author maintainer
#11 lore raw

PATCH v2 11/12 selftests/ftrace: Add syntax error test for immediates

In reply to Masami Hiramatsu

Add syntax error test cases for immediate value and immediate string.

---
 .../ftrace/test.d/kprobe/kprobe_syntax_errors.tc   |    5 +++++
 1 file changed, 5 insertions(+)
tools/testing/selftests/ftrace/test.d/kprobe/kprobe_syntax_errors.tc +5 -0
index 29faaec942c6..aa59944bcace 100644
@@ -41,6 +41,11 @@ check_error 'p vfs_read ^%none_reg'	# BAD_REG_NAME
 check_error 'p vfs_read ^@12345678abcde'	# BAD_MEM_ADDR
 check_error 'p vfs_read ^@+10'		# FILE_ON_KPROBE
 
+grep -q "imm-value" README && \
+check_error 'p vfs_read arg1=\^x'	# BAD_IMM
+grep -q "imm-string" README && \
+check_error 'p vfs_read arg1=\"abcd^'	# IMMSTR_NO_CLOSE
+
 check_error 'p vfs_read ^+0@0)'		# DEREF_NEED_BRACE
 check_error 'p vfs_read ^+0ab1(@0)'	# BAD_DEREF_OFFS
 check_error 'p vfs_read +0(+0(@0^)'	# DEREF_OPEN_BRACE
author maintainer
#12 lore raw

PATCH v2 12/12 selftests/ftrace: Add syntax error test for multiprobe

In reply to Masami Hiramatsu

Add syntax error test cases for multiprobe appending errors.

---
 tools/testing/selftests/ftrace/test.d/functions    |    2 +-
 .../ftrace/test.d/kprobe/kprobe_syntax_errors.tc   |   10 ++++++++++
 2 files changed, 11 insertions(+), 1 deletion(-)
tools/testing/selftests/ftrace/test.d/functions +1 -1
index 779ec11f61bd..1cac8cc2cb35 100644
@@ -115,7 +115,7 @@ ftrace_errlog_check() { # err-prefix command-with-error-pos-by-^ command-file
     command=$(echo "$2" | tr -d ^)
     echo "Test command: $command"
     echo > error_log
-    (! echo "$command" > "$3" ) 2> /dev/null
+    (! echo "$command" >> "$3" ) 2> /dev/null
     grep "$1: error:" -A 3 error_log
     N=$(tail -n 1 error_log | wc -c)
     # "  Command: " and "^\n" => 13
tools/testing/selftests/ftrace/test.d/kprobe/kprobe_syntax_errors.tc +10 -0
index aa59944bcace..39ef7ac1f51c 100644
@@ -87,4 +87,14 @@ case $(uname -m) in
     ;;
 esac
 
+# multiprobe errors
+if grep -q "Create/append/" README && grep -q "imm-value" README; then
+echo 'p:kprobes/testevent _do_fork' > kprobe_events
+check_error '^r:kprobes/testevent do_exit'	# DIFF_PROBE_TYPE
+echo 'p:kprobes/testevent _do_fork abcd=\1' > kprobe_events
+check_error 'p:kprobes/testevent _do_fork ^bcd=\1'	# DIFF_ARG_TYPE
+check_error 'p:kprobes/testevent _do_fork ^abcd=\1:u8'	# DIFF_ARG_TYPE
+check_error 'p:kprobes/testevent _do_fork ^abcd=\"foo"'	# DIFF_ARG_TYPE
+fi
+
 exit 0
author maintainer
#13 lore raw
In reply to Masami Hiramatsu

Hi Steve,

Would you have any comment on this?

Thank you,

Quoted 104 lines · On Thu, 20 Jun 2019 00:07:09 +0900 Masami Hiramatsu <[email protected]> wrote:

Hello,

This is the 2nd version of multi-probes per event support on ftrace and perf-tools.

Previous version is here;
https://lkml.org/lkml/2019/5/31/573

Quoted 1 line

From this version, I omitted first 9 patches which has been picked

to Steve's tree.
In this version, I've fixed some bugs and hardened some unexpected error cases according to Steve's comment.
Here are changes in this version:

- [1/12] This have below changes.
- Warn if the primary trace_probe does not exist.
- Fix enable_trace_kprobe() to not return error if the any probes are "gone" state. If all probes have gone or any other error reason, the event can not be enabled and return error.
- Fix trace_probe_enable() to roll back all enabled uprobe if any one of uprobe is failed to enable.
- [7/12] Swap the checking order of filename for avoiding unexpected memory access.

====
For trace-event, we can insert same trace-event on several places on the code, and those can record similar information as a same event with same format.

This series implements similar feature on probe-event. Since the probe event is based on the compiled binary, sometimes we find that the target source line is complied into several different addresses, e.g. inlined function, unrolled loop, etc. In those cases, it is useful to put a same probe-event on different addresses.

With this series, we can append multi probes on one event as below

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=%cx >> kprobe_events
  # kprobe_events
  p:kprobes/testevent _do_fork r1=%ax r2=%dx
  p:kprobes/testevent fork_idle r1=%ax r2=%cx

This means testevent is hit on both of _do_fork and fork_idle. As you can see, the appended event must have same number of arguments and those must have same 'type' and 'name' as original one. This is like a function signature, it checks whether the appending event has the same type and name of event arguments and same probe type, but doesn't care about the assignment.

So, below appending commands will be rejected.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax >> kprobe_events
  (No 2nd argument)
  # echo p:testevent fork_idle r1=%ax r2=%ax:x8 >> kprobe_events
  (The type of 2nd argument is different)

If one inlined code has an argument on a register, but another inlined code has fixed value (as a result of optimization), you can also specify the fixed immediate value, e.g.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=\1 >> kprobe_events

Thank you,

---

Masami Hiramatsu (12):
tracing/probe: Split trace_event related data from trace_probe tracing/dynevent: Delete all matched events tracing/dynevent: Pass extra arguments to match operation tracing/kprobe: Add multi-probe per event support tracing/uprobe: Add multi-probe per uprobe event support tracing/kprobe: Add per-probe delete from event tracing/uprobe: Add per-probe delete from event tracing/probe: Add immediate parameter support tracing/probe: Add immediate string parameter support selftests/ftrace: Add a testcase for kprobe multiprobe event selftests/ftrace: Add syntax error test for immediates selftests/ftrace: Add syntax error test for multiprobe

 Documentation/trace/kprobetrace.rst                |    1 
 Documentation/trace/uprobetracer.rst               |    1 
 kernel/trace/trace.c                               |    8 -
 kernel/trace/trace_dynevent.c                      |   10 +
 kernel/trace/trace_dynevent.h                      |    7 -
 kernel/trace/trace_events_hist.c                   |    4 
 kernel/trace/trace_kprobe.c                        |  241 ++++++++++++++----
 kernel/trace/trace_probe.c                         |  176 +++++++++++--
 kernel/trace/trace_probe.h                         |   67 ++++-
 kernel/trace/trace_uprobe.c                        |  263 +++++++++++++++-----
 tools/testing/selftests/ftrace/test.d/functions    |    2 
 .../ftrace/test.d/kprobe/kprobe_multiprobe.tc      |   35 +++
 .../ftrace/test.d/kprobe/kprobe_syntax_errors.tc   |   15 +
 13 files changed, 665 insertions(+), 165 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/kprobe_multiprobe.tc

--
Masami Hiramatsu (Linaro) <[email protected]>

Signature · 1 line
Masami Hiramatsu <[email protected]>
maintainer
#14 lore raw
Quoted 4 lines · On Thu, 4 Jul 2019 15:39:58 +0900 Masami Hiramatsu <[email protected]> wrote:

Hi Steve,

Would you have any comment on this?

Hi Masami,

It's on my todo list. As today is a US holiday, I'll look at this on Monday.

Thanks for the reminder!

-- Steve

Quoted 111 lines

Thank you,

Quoted 104 lines · On Thu, 20 Jun 2019 00:07:09 +0900 Masami Hiramatsu <[email protected]> wrote:

Hello,

This is the 2nd version of multi-probes per event support on ftrace and perf-tools.

Previous version is here;
https://lkml.org/lkml/2019/5/31/573

Quoted 1 line

From this version, I omitted first 9 patches which has been picked

to Steve's tree.
In this version, I've fixed some bugs and hardened some unexpected error cases according to Steve's comment.
Here are changes in this version:

- [1/12] This have below changes.
- Warn if the primary trace_probe does not exist.
- Fix enable_trace_kprobe() to not return error if the any probes are "gone" state. If all probes have gone or any other error reason, the event can not be enabled and return error.
- Fix trace_probe_enable() to roll back all enabled uprobe if any one of uprobe is failed to enable.
- [7/12] Swap the checking order of filename for avoiding unexpected memory access.

====
For trace-event, we can insert same trace-event on several places on the code, and those can record similar information as a same event with same format.

This series implements similar feature on probe-event. Since the probe event is based on the compiled binary, sometimes we find that the target source line is complied into several different addresses, e.g. inlined function, unrolled loop, etc. In those cases, it is useful to put a same probe-event on different addresses.

With this series, we can append multi probes on one event as below

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=%cx >> kprobe_events
  # kprobe_events
  p:kprobes/testevent _do_fork r1=%ax r2=%dx
  p:kprobes/testevent fork_idle r1=%ax r2=%cx

This means testevent is hit on both of _do_fork and fork_idle. As you can see, the appended event must have same number of arguments and those must have same 'type' and 'name' as original one. This is like a function signature, it checks whether the appending event has the same type and name of event arguments and same probe type, but doesn't care about the assignment.

So, below appending commands will be rejected.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax >> kprobe_events
  (No 2nd argument)
  # echo p:testevent fork_idle r1=%ax r2=%ax:x8 >> kprobe_events
  (The type of 2nd argument is different)

If one inlined code has an argument on a register, but another inlined code has fixed value (as a result of optimization), you can also specify the fixed immediate value, e.g.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=\1 >> kprobe_events

Thank you,

---

Masami Hiramatsu (12):
tracing/probe: Split trace_event related data from trace_probe tracing/dynevent: Delete all matched events tracing/dynevent: Pass extra arguments to match operation tracing/kprobe: Add multi-probe per event support tracing/uprobe: Add multi-probe per uprobe event support tracing/kprobe: Add per-probe delete from event tracing/uprobe: Add per-probe delete from event tracing/probe: Add immediate parameter support tracing/probe: Add immediate string parameter support selftests/ftrace: Add a testcase for kprobe multiprobe event selftests/ftrace: Add syntax error test for immediates selftests/ftrace: Add syntax error test for multiprobe

 Documentation/trace/kprobetrace.rst                |    1 
 Documentation/trace/uprobetracer.rst               |    1 
 kernel/trace/trace.c                               |    8 -
 kernel/trace/trace_dynevent.c                      |   10 +
 kernel/trace/trace_dynevent.h                      |    7 -
 kernel/trace/trace_events_hist.c                   |    4 
 kernel/trace/trace_kprobe.c                        |  241 ++++++++++++++----
 kernel/trace/trace_probe.c                         |  176 +++++++++++--
 kernel/trace/trace_probe.h                         |   67 ++++-
 kernel/trace/trace_uprobe.c                        |  263 +++++++++++++++-----
 tools/testing/selftests/ftrace/test.d/functions    |    2 
 .../ftrace/test.d/kprobe/kprobe_multiprobe.tc      |   35 +++
 .../ftrace/test.d/kprobe/kprobe_syntax_errors.tc   |   15 +
 13 files changed, 665 insertions(+), 165 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/kprobe_multiprobe.tc

--
Masami Hiramatsu (Linaro) <[email protected]>

author maintainer
#15 lore raw

Hi Steve,

Have you already picked this series?
If not yet, should I update and resend this series?

Thank you,

Quoted 130 lines · On Thu, 4 Jul 2019 07:28:33 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 4 lines · On Thu, 4 Jul 2019 15:39:58 +0900 Masami Hiramatsu <[email protected]> wrote:

Hi Steve,

Would you have any comment on this?

Hi Masami,

It's on my todo list. As today is a US holiday, I'll look at this on Monday.

Thanks for the reminder!

-- Steve

Quoted 111 lines

Thank you,

Quoted 104 lines · On Thu, 20 Jun 2019 00:07:09 +0900 Masami Hiramatsu <[email protected]> wrote:

Hello,

This is the 2nd version of multi-probes per event support on ftrace and perf-tools.

Previous version is here;
https://lkml.org/lkml/2019/5/31/573

Quoted 1 line

From this version, I omitted first 9 patches which has been picked

to Steve's tree.
In this version, I've fixed some bugs and hardened some unexpected error cases according to Steve's comment.
Here are changes in this version:

- [1/12] This have below changes.
- Warn if the primary trace_probe does not exist.
- Fix enable_trace_kprobe() to not return error if the any probes are "gone" state. If all probes have gone or any other error reason, the event can not be enabled and return error.
- Fix trace_probe_enable() to roll back all enabled uprobe if any one of uprobe is failed to enable.
- [7/12] Swap the checking order of filename for avoiding unexpected memory access.

====
For trace-event, we can insert same trace-event on several places on the code, and those can record similar information as a same event with same format.

This series implements similar feature on probe-event. Since the probe event is based on the compiled binary, sometimes we find that the target source line is complied into several different addresses, e.g. inlined function, unrolled loop, etc. In those cases, it is useful to put a same probe-event on different addresses.

With this series, we can append multi probes on one event as below

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=%cx >> kprobe_events
  # kprobe_events
  p:kprobes/testevent _do_fork r1=%ax r2=%dx
  p:kprobes/testevent fork_idle r1=%ax r2=%cx

This means testevent is hit on both of _do_fork and fork_idle. As you can see, the appended event must have same number of arguments and those must have same 'type' and 'name' as original one. This is like a function signature, it checks whether the appending event has the same type and name of event arguments and same probe type, but doesn't care about the assignment.

So, below appending commands will be rejected.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax >> kprobe_events
  (No 2nd argument)
  # echo p:testevent fork_idle r1=%ax r2=%ax:x8 >> kprobe_events
  (The type of 2nd argument is different)

If one inlined code has an argument on a register, but another inlined code has fixed value (as a result of optimization), you can also specify the fixed immediate value, e.g.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=\1 >> kprobe_events

Thank you,

---

Masami Hiramatsu (12):
tracing/probe: Split trace_event related data from trace_probe tracing/dynevent: Delete all matched events tracing/dynevent: Pass extra arguments to match operation tracing/kprobe: Add multi-probe per event support tracing/uprobe: Add multi-probe per uprobe event support tracing/kprobe: Add per-probe delete from event tracing/uprobe: Add per-probe delete from event tracing/probe: Add immediate parameter support tracing/probe: Add immediate string parameter support selftests/ftrace: Add a testcase for kprobe multiprobe event selftests/ftrace: Add syntax error test for immediates selftests/ftrace: Add syntax error test for multiprobe

 Documentation/trace/kprobetrace.rst                |    1 
 Documentation/trace/uprobetracer.rst               |    1 
 kernel/trace/trace.c                               |    8 -
 kernel/trace/trace_dynevent.c                      |   10 +
 kernel/trace/trace_dynevent.h                      |    7 -
 kernel/trace/trace_events_hist.c                   |    4 
 kernel/trace/trace_kprobe.c                        |  241 ++++++++++++++----
 kernel/trace/trace_probe.c                         |  176 +++++++++++--
 kernel/trace/trace_probe.h                         |   67 ++++-
 kernel/trace/trace_uprobe.c                        |  263 +++++++++++++++-----
 tools/testing/selftests/ftrace/test.d/functions    |    2 
 .../ftrace/test.d/kprobe/kprobe_multiprobe.tc      |   35 +++
 .../ftrace/test.d/kprobe/kprobe_syntax_errors.tc   |   15 +
 13 files changed, 665 insertions(+), 165 deletions(-)
 create mode 100644 tools/testing/selftests/ftrace/test.d/kprobe/kprobe_multiprobe.tc

--
Masami Hiramatsu (Linaro) <[email protected]>

Signature · 1 line
Masami Hiramatsu <[email protected]>
maintainer
#16 lore raw
Quoted 4 lines · On Tue, 30 Jul 2019 15:23:31 +0900 Masami Hiramatsu <[email protected]> wrote:

Hi Steve,

Have you already picked this series?
If not yet, should I update and resend this series?

I started to, but then got distracted :-/

I'll work with the current series, no need to resend.

Sorry for the delay.

-- Steve

maintainer
#17 lore raw
In reply to Masami Hiramatsu
Quoted 59 lines · On Thu, 20 Jun 2019 00:07:09 +0900 Masami Hiramatsu <[email protected]> wrote:

Hello,

This is the 2nd version of multi-probes per event support on ftrace and perf-tools.

Previous version is here;
https://lkml.org/lkml/2019/5/31/573

Quoted 1 line

From this version, I omitted first 9 patches which has been picked

to Steve's tree.
In this version, I've fixed some bugs and hardened some unexpected error cases according to Steve's comment.
Here are changes in this version:

- [1/12] This have below changes.
- Warn if the primary trace_probe does not exist.
- Fix enable_trace_kprobe() to not return error if the any probes are "gone" state. If all probes have gone or any other error reason, the event can not be enabled and return error.
- Fix trace_probe_enable() to roll back all enabled uprobe if any one of uprobe is failed to enable.
- [7/12] Swap the checking order of filename for avoiding unexpected memory access.

====
For trace-event, we can insert same trace-event on several places on the code, and those can record similar information as a same event with same format.

This series implements similar feature on probe-event. Since the probe event is based on the compiled binary, sometimes we find that the target source line is complied into several different addresses, e.g. inlined function, unrolled loop, etc. In those cases, it is useful to put a same probe-event on different addresses.

With this series, we can append multi probes on one event as below

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=%cx >> kprobe_events
  # kprobe_events
  p:kprobes/testevent _do_fork r1=%ax r2=%dx
  p:kprobes/testevent fork_idle r1=%ax r2=%cx

This means testevent is hit on both of _do_fork and fork_idle. As you can see, the appended event must have same number of arguments and those must have same 'type' and 'name' as original one. This is like a function signature, it checks whether the appending event has the same type and name of event arguments and same probe type, but doesn't care about the assignment.

So, below appending commands will be rejected.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax >> kprobe_events
  (No 2nd argument)
  # echo p:testevent fork_idle r1=%ax r2=%ax:x8 >> kprobe_events
  (The type of 2nd argument is different)
Quoted 8 lines

If one inlined code has an argument on a register, but another inlined code has fixed value (as a result of optimization), you can also specify the fixed immediate value, e.g.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=\1 >> kprobe_events

Hi Masami,

I applied this patch set to my queue. Nice feature! I'll probably be testing it a bit more. I wont be pushing it to my repo until v5.3-rc3 comes out, as I'll plan on rebasing my for-next branch on that.

Thanks!

-- Steve

author maintainer
#18 lore raw
Quoted 72 lines · On Wed, 31 Jul 2019 11:25:39 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 59 lines · On Thu, 20 Jun 2019 00:07:09 +0900 Masami Hiramatsu <[email protected]> wrote:

Hello,

This is the 2nd version of multi-probes per event support on ftrace and perf-tools.

Previous version is here;
https://lkml.org/lkml/2019/5/31/573

Quoted 1 line

From this version, I omitted first 9 patches which has been picked

to Steve's tree.
In this version, I've fixed some bugs and hardened some unexpected error cases according to Steve's comment.
Here are changes in this version:

- [1/12] This have below changes.
- Warn if the primary trace_probe does not exist.
- Fix enable_trace_kprobe() to not return error if the any probes are "gone" state. If all probes have gone or any other error reason, the event can not be enabled and return error.
- Fix trace_probe_enable() to roll back all enabled uprobe if any one of uprobe is failed to enable.
- [7/12] Swap the checking order of filename for avoiding unexpected memory access.

====
For trace-event, we can insert same trace-event on several places on the code, and those can record similar information as a same event with same format.

This series implements similar feature on probe-event. Since the probe event is based on the compiled binary, sometimes we find that the target source line is complied into several different addresses, e.g. inlined function, unrolled loop, etc. In those cases, it is useful to put a same probe-event on different addresses.

With this series, we can append multi probes on one event as below

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=%cx >> kprobe_events
  # kprobe_events
  p:kprobes/testevent _do_fork r1=%ax r2=%dx
  p:kprobes/testevent fork_idle r1=%ax r2=%cx

This means testevent is hit on both of _do_fork and fork_idle. As you can see, the appended event must have same number of arguments and those must have same 'type' and 'name' as original one. This is like a function signature, it checks whether the appending event has the same type and name of event arguments and same probe type, but doesn't care about the assignment.

So, below appending commands will be rejected.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax >> kprobe_events
  (No 2nd argument)
  # echo p:testevent fork_idle r1=%ax r2=%ax:x8 >> kprobe_events
  (The type of 2nd argument is different)
Quoted 8 lines

If one inlined code has an argument on a register, but another inlined code has fixed value (as a result of optimization), you can also specify the fixed immediate value, e.g.

  # echo p:testevent _do_fork r1=%ax r2=%dx > kprobe_events
  # echo p:testevent fork_idle r1=%ax r2=\1 >> kprobe_events
Quoted 5 lines

Hi Masami,

I applied this patch set to my queue. Nice feature! I'll probably be testing it a bit more. I wont be pushing it to my repo until v5.3-rc3 comes out, as I'll plan on rebasing my for-next branch on that.

Thanks! if there is any issue, please let me know. I also start working on perf probe part to support this.

Thank you,

Quoted 4 lines

Thanks!

-- Steve

Signature · 1 line
Masami Hiramatsu <[email protected]>