Skip to contentSkip to conversation

tracing/kprobes: Fix event generation API etc.

PATCH In review Masami Hiramatsu 3 participants 11 messages Apr 25, 2020 – May 21, 2020 Tracing

lore
Share & tools
lore

Masami Hiramatsu

author maintainer
#0 lore raw

cover tracing/kprobes: Fix event generation API etc.

Hello,

Here are bugfix/cleanup patches for the kprobe tracer, [1/3] is a typo fix, [2/3] is fixing boot-time tracer and [3/3] is error checking for the new in-kernel kprobe event API.

Tom, I found that your commit 29a154810546 ("tracing: Change trace_boot to use kprobe_event interface") broke the boot-time tracer's kprobe event because of wrong API usage. Could you review it?

I marked [3/3] as a bugfix, because if the loc == NULL, __kprobe_event_gen_cmd_start() obviously does not work. But it reports actual error at kprobe_event_gen_cmd_end(). That is not good for developers to debug it.

Thank you,

---

Masami Hiramatsu (3):
tracing/kprobes: Fix a double initialization typo tracing/boottime: Fix kprobe event API usage tracing/kprobes: Reject new event if loc is NULL

 kernel/trace/trace_boot.c   |   20 ++++++++------------
 kernel/trace/trace_kprobe.c |    8 +++++++-
 2 files changed, 15 insertions(+), 13 deletions(-)

--
Masami Hiramatsu (Linaro) <[email protected]>

author maintainer
#1 lore raw

PATCH 1/3 tracing/kprobes: Fix a double initialization typo

Fix a typo that resulted in an unnecessary double
initialization to addr.

Fixes c7411a1a126f ("tracing/kprobe: Check whether the non-suffixed symbol is notrace")
---
 kernel/trace/trace_kprobe.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/trace/trace_kprobe.c +1 -1
index d0568af4a0ef..0d9300c3b084 100644
@@ -453,7 +453,7 @@ static bool __within_notrace_func(unsigned long addr)
 
 static bool within_notrace_func(struct trace_kprobe *tk)
 {
-	unsigned long addr = addr = trace_kprobe_address(tk);
+	unsigned long addr = trace_kprobe_address(tk);
 	char symname[KSYM_NAME_LEN], *p;
 
 	if (!__within_notrace_func(addr))
author maintainer
#2 lore raw

PATCH 2/3 tracing/boottime: Fix kprobe event API usage

In reply to Masami Hiramatsu

Fix boottime kprobe events to use API correctly for multiple events.

For example, when we set a multiprobe kprobe events in bootconfig like below,

  ftrace.event.kprobes.myevent {
  	probes = "vfs_read $arg1 $arg2", "vfs_write $arg1 $arg2"
  }

This cause an error;

trace_boot: Failed to add probe: p:kprobes/myevent (null) vfs_read $arg1 $arg2 vfs_write $arg1 $arg2

This shows the 1st argument becomes NULL and multiprobes are merged to 1 probe.

Fixes 29a154810546 ("tracing: Change trace_boot to use kprobe_event interface")
---
 kernel/trace/trace_boot.c |   20 ++++++++------------
 1 file changed, 8 insertions(+), 12 deletions(-)
kernel/trace/trace_boot.c +8 -12
index 06d7feb5255f..9de29bb45a27 100644
@@ -95,24 +95,20 @@ trace_boot_add_kprobe_event(struct xbc_node *node, const char *event)
 	struct xbc_node *anode;
 	char buf[MAX_BUF_LEN];
 	const char *val;
-	int ret;
+	int ret = 0;
 
-	kprobe_event_cmd_init(&cmd, buf, MAX_BUF_LEN);
+	xbc_node_for_each_array_value(node, "probes", anode, val) {
+		kprobe_event_cmd_init(&cmd, buf, MAX_BUF_LEN);
 
-	ret = kprobe_event_gen_cmd_start(&cmd, event, NULL);
-	if (ret)
-		return ret;
+		ret = kprobe_event_gen_cmd_start(&cmd, event, val);
+		if (ret)
+			break;
 
-	xbc_node_for_each_array_value(node, "probes", anode, val) {
-		ret = kprobe_event_add_field(&cmd, val);
+		ret = kprobe_event_gen_cmd_end(&cmd);
 		if (ret)
-			return ret;
+			pr_err("Failed to add probe: %s\n", buf);
 	}
 
-	ret = kprobe_event_gen_cmd_end(&cmd);
-	if (ret)
-		pr_err("Failed to add probe: %s\n", buf);
-
 	return ret;
 }
 #else
maintainer
#3 lore raw

Re: [PATCH 2/3] tracing/boottime: Fix kprobe event API usage

Quoted 35 lines · On Sat, 25 Apr 2020 14:49:17 +0900 Masami Hiramatsu <[email protected]> wrote:

Fix boottime kprobe events to use API correctly for multiple events.

For example, when we set a multiprobe kprobe events in bootconfig like below,

  ftrace.event.kprobes.myevent {
  	probes = "vfs_read $arg1 $arg2", "vfs_write $arg1 $arg2"
  }

This cause an error;

trace_boot: Failed to add probe: p:kprobes/myevent (null) vfs_read $arg1 $arg2 vfs_write $arg1 $arg2

This shows the 1st argument becomes NULL and multiprobes are merged to 1 probe.

Fixes 29a154810546 ("tracing: Change trace_boot to use kprobe_event interface")
---
 kernel/trace/trace_boot.c |   20 ++++++++------------
 1 file changed, 8 insertions(+), 12 deletions(-)
kernel/trace/trace_boot.c +1 -1
index 06d7feb5255f..9de29bb45a27 100644
@@ -95,24 +95,20 @@ trace_boot_add_kprobe_event(struct xbc_node *node, const char *event)
 	struct xbc_node *anode;
 	char buf[MAX_BUF_LEN];
 	const char *val;
-	int ret;
+	int ret = 0;
 

kernel/trace/trace_boot.c trace_boot_add_kprobe_event(struct xbc_node *node, const char *event) · line 105

Quoted 10 lines
kernel/trace/trace_boot.c +5 -4
-	kprobe_event_cmd_init(&cmd, buf, MAX_BUF_LEN);
+	xbc_node_for_each_array_value(node, "probes", anode, val) {
+		kprobe_event_cmd_init(&cmd, buf, MAX_BUF_LEN);
 
-	ret = kprobe_event_gen_cmd_start(&cmd, event, NULL);
-	if (ret)
-		return ret;
+		ret = kprobe_event_gen_cmd_start(&cmd, event, val);
+		if (ret)
+			break;

Should we break here? What about just printing an error message and continuing to the next probe. If I start up something with a typo in the first element, I lose all events. But if I have a typo in the last one, I get all but that one. I rather have it just fail on the ones that don't parse properly.

-- Steve

Quoted 16 lines
kernel/trace/trace_boot.c +2 -7
-	xbc_node_for_each_array_value(node, "probes", anode, val) {
-		ret = kprobe_event_add_field(&cmd, val);
+		ret = kprobe_event_gen_cmd_end(&cmd);
 		if (ret)
-			return ret;
+			pr_err("Failed to add probe: %s\n", buf);
 	}
 
-	ret = kprobe_event_gen_cmd_end(&cmd);
-	if (ret)
-		pr_err("Failed to add probe: %s\n", buf);
-
 	return ret;
 }
 #else
author maintainer
#4 lore raw

Re: [PATCH 2/3] tracing/boottime: Fix kprobe event API usage

Quoted 42 lines · On Sat, 25 Apr 2020 10:00:20 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 39 lines · On Sat, 25 Apr 2020 14:49:17 +0900 Masami Hiramatsu <[email protected]> wrote:

Fix boottime kprobe events to use API correctly for multiple events.

For example, when we set a multiprobe kprobe events in bootconfig like below,

  ftrace.event.kprobes.myevent {
  	probes = "vfs_read $arg1 $arg2", "vfs_write $arg1 $arg2"
  }

This cause an error;

trace_boot: Failed to add probe: p:kprobes/myevent (null) vfs_read $arg1 $arg2 vfs_write $arg1 $arg2

This shows the 1st argument becomes NULL and multiprobes are merged to 1 probe.

Fixes 29a154810546 ("tracing: Change trace_boot to use kprobe_event interface")
---
 kernel/trace/trace_boot.c |   20 ++++++++------------
 1 file changed, 8 insertions(+), 12 deletions(-)
kernel/trace/trace_boot.c +3 -2
index 06d7feb5255f..9de29bb45a27 100644
@@ -95,24 +95,20 @@ trace_boot_add_kprobe_event(struct xbc_node *node, const char *event)
 	struct xbc_node *anode;
 	char buf[MAX_BUF_LEN];
 	const char *val;
-	int ret;
+	int ret = 0;
 
-	kprobe_event_cmd_init(&cmd, buf, MAX_BUF_LEN);
+	xbc_node_for_each_array_value(node, "probes", anode, val) {
+		kprobe_event_cmd_init(&cmd, buf, MAX_BUF_LEN);
 
Quoted 12 lines

kernel/trace/trace_boot.c trace_boot_add_kprobe_event(struct xbc_node *node, const char *event) · line 105

Quoted 6 lines
kernel/trace/trace_boot.c +3 -3
-	ret = kprobe_event_gen_cmd_start(&cmd, event, NULL);
-	if (ret)
-		return ret;
+		ret = kprobe_event_gen_cmd_start(&cmd, event, val);
+		if (ret)
+			break;

Should we break here? What about just printing an error message and continuing to the next probe. If I start up something with a typo in the first element, I lose all events. But if I have a typo in the last one, I get all but that one. I rather have it just fail on the ones that don't parse properly.

This kprobe_event_gen_cmd_start() causes an error only if there is a program bug or out of memory, because it never evaluate given probe definition, but kprobe_event_gen_cmd_end() does. Thus I think this is correct way to handle the error.

IOW, if you typo a probe, it will be handled by kprobe_event_gen_cmd_end() and it shows an error message and continue to process other probe definitions. See below,

Quoted 7 lines

kernel/trace/trace_boot.c trace_boot_add_kprobe_event(struct xbc_node *node, const char *event) · line 110

Quoted 7 lines
kernel/trace/trace_boot.c +2 -3
-	xbc_node_for_each_array_value(node, "probes", anode, val) {
-		ret = kprobe_event_add_field(&cmd, val);
+		ret = kprobe_event_gen_cmd_end(&cmd);
 		if (ret)
-			return ret;
+			pr_err("Failed to add probe: %s\n", buf);
 	}

This continues to next probe ;-)

Thank you,

Quoted 9 lines
Quoted 8 lines
kernel/trace/trace_boot.c +0 -4
-	ret = kprobe_event_gen_cmd_end(&cmd);
-	if (ret)
-		pr_err("Failed to add probe: %s\n", buf);
-
 	return ret;
 }
 #else
Signature · 1 line
Masami Hiramatsu <[email protected]>
Reviewed-by on v1 2/3
#5 lore raw

Re: [PATCH 2/3] tracing/boottime: Fix kprobe event API usage

In reply to Masami Hiramatsu

Hi Masami,

Quoted 15 lines · Masami Hiramatsu

Fix boottime kprobe events to use API correctly for multiple events.

For example, when we set a multiprobe kprobe events in bootconfig like below,

  ftrace.event.kprobes.myevent {
  	probes = "vfs_read $arg1 $arg2", "vfs_write $arg1 $arg2"
  }

This cause an error;

trace_boot: Failed to add probe: p:kprobes/myevent (null) vfs_read
$arg1 $arg2 vfs_write $arg1 $arg2

Quoted 7 lines

This shows the 1st argument becomes NULL and multiprobes are merged to 1 probe.

Fixes 29a154810546 ("tracing: Change trace_boot to use kprobe_event

interface")

Thanks for fixing this!

Quoted 43 lines
---
 kernel/trace/trace_boot.c |   20 ++++++++------------
 1 file changed, 8 insertions(+), 12 deletions(-)
kernel/trace/trace_boot.c +8 -12
index 06d7feb5255f..9de29bb45a27 100644
@@ -95,24 +95,20 @@ trace_boot_add_kprobe_event(struct xbc_node
 *node, const char *event)
 	struct xbc_node *anode;
 	char buf[MAX_BUF_LEN];
 	const char *val;
-	int ret;
+	int ret = 0;
 
-	kprobe_event_cmd_init(&cmd, buf, MAX_BUF_LEN);
+	xbc_node_for_each_array_value(node, "probes", anode, val) {
+		kprobe_event_cmd_init(&cmd, buf, MAX_BUF_LEN);
 
-	ret = kprobe_event_gen_cmd_start(&cmd, event, NULL);
-	if (ret)
-		return ret;
+		ret = kprobe_event_gen_cmd_start(&cmd, event, val);
+		if (ret)
+			break;
 
-	xbc_node_for_each_array_value(node, "probes", anode, val) {
-		ret = kprobe_event_add_field(&cmd, val);
+		ret = kprobe_event_gen_cmd_end(&cmd);
 		if (ret)
-			return ret;
+			pr_err("Failed to add probe: %s\n", buf);
 	}
 
-	ret = kprobe_event_gen_cmd_end(&cmd);
-	if (ret)
-		pr_err("Failed to add probe: %s\n", buf);
-
 	return ret;
 }

#else

author maintainer
#6 lore raw

PATCH 3/3 tracing/kprobes: Reject new event if loc is NULL

In reply to Masami Hiramatsu

Reject the new event which has NULL location for kprobes. For kprobes, user must specify at least the location.

Fixes 2a588dd1d5d6 ("tracing: Add kprobe event command generation functions")
---
 kernel/trace/trace_kprobe.c |    6 ++++++
 1 file changed, 6 insertions(+)
kernel/trace/trace_kprobe.c +6 -0
index 0d9300c3b084..35989383ae11 100644
@@ -940,6 +940,9 @@ EXPORT_SYMBOL_GPL(kprobe_event_cmd_init);
  * complete command or only the first part of it; in the latter case,
  * kprobe_event_add_fields() can be used to add more fields following this.
  *
+ * Unlikely the synth_event_gen_cmd_start(), @loc must be specified. This
+ * returns -EINVAL if @loc == NULL.
+ *
  * Return: 0 if successful, error otherwise.
  */
 int __kprobe_event_gen_cmd_start(struct dynevent_cmd *cmd, bool kretprobe,
@@ -953,6 +956,9 @@ int __kprobe_event_gen_cmd_start(struct dynevent_cmd *cmd, bool kretprobe,
 	if (cmd->type != DYNEVENT_TYPE_KPROBE)
 		return -EINVAL;
 
+	if (!loc)
+		return -EINVAL;
+
 	if (kretprobe)
 		snprintf(buf, MAX_EVENT_NAME_LEN, "r:kprobes/%s", name);
 	else
author maintainer
#7 lore raw
In reply to Masami Hiramatsu

Hi Steve,

It seems this fixes are not picked up yet.
Would you have any consideration?

Thank you,

Quoted 32 lines · On Sat, 25 Apr 2020 14:48:59 +0900 Masami Hiramatsu <[email protected]> wrote:

Hello,

Here are bugfix/cleanup patches for the kprobe tracer, [1/3] is a typo fix, [2/3] is fixing boot-time tracer and [3/3] is error checking for the new in-kernel kprobe event API.

Tom, I found that your commit 29a154810546 ("tracing: Change trace_boot to use kprobe_event interface") broke the boot-time tracer's kprobe event because of wrong API usage. Could you review it?

I marked [3/3] as a bugfix, because if the loc == NULL, __kprobe_event_gen_cmd_start() obviously does not work. But it reports actual error at kprobe_event_gen_cmd_end(). That is not good for developers to debug it.

Thank you,

---

Masami Hiramatsu (3):
tracing/kprobes: Fix a double initialization typo tracing/boottime: Fix kprobe event API usage tracing/kprobes: Reject new event if loc is NULL

 kernel/trace/trace_boot.c   |   20 ++++++++------------
 kernel/trace/trace_kprobe.c |    8 +++++++-
 2 files changed, 15 insertions(+), 13 deletions(-)

--
Masami Hiramatsu (Linaro) <[email protected]>

Signature · 1 line
Masami Hiramatsu <[email protected]>
maintainer
#8 lore raw
Quoted 5 lines · On Wed, 20 May 2020 23:22:20 +0900 Masami Hiramatsu <[email protected]> wrote:

Hi Steve,

It seems this fixes are not picked up yet.
Would you have any consideration?

Ah, I missed your reply to my comment :-/

Yeah, I'll pull that in now and start testing it.

Thanks for the reminder.

-- Steve

maintainer
#9 lore raw
Quoted 10 lines · On Wed, 20 May 2020 10:33:46 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 5 lines · On Wed, 20 May 2020 23:22:20 +0900 Masami Hiramatsu <[email protected]> wrote:

Hi Steve,

It seems this fixes are not picked up yet.
Would you have any consideration?

Quoted 5 lines

Ah, I missed your reply to my comment :-/

Yeah, I'll pull that in now and start testing it.

Thanks for the reminder.

No, it's already in mainline:

Merged: 192ffb7515839b1cc8457e0a8c1e09783de019d3

With commits:

dcbd21c9fca5e954fd4e3d91884907eb6d47187e
da0f1f4167e3af69e1d8b32d6d65195ddd2bfb64
5b4dcd2d201a395ad4054067bfae4a07554fbd65

-- Steve

author maintainer
#10 lore raw
Quoted 21 lines · On Wed, 20 May 2020 10:40:05 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 10 lines · On Wed, 20 May 2020 10:33:46 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 5 lines · On Wed, 20 May 2020 23:22:20 +0900 Masami Hiramatsu <[email protected]> wrote:

Hi Steve,

It seems this fixes are not picked up yet.
Would you have any consideration?

Quoted 5 lines

Ah, I missed your reply to my comment :-/

Yeah, I'll pull that in now and start testing it.

Thanks for the reminder.

No, it's already in mainline:

Quoted 7 lines

Merged: 192ffb7515839b1cc8457e0a8c1e09783de019d3

With commits:

dcbd21c9fca5e954fd4e3d91884907eb6d47187e
da0f1f4167e3af69e1d8b32d6d65195ddd2bfb64
5b4dcd2d201a395ad4054067bfae4a07554fbd65

Oops, I must have checked another branch. Sorry.

Thank you!

Signature · 1 line
Masami Hiramatsu <[email protected]>