Skip to contentSkip to conversation

libbpf: Remove kprobe_event on failed kprobe_open_legacy

PATCH In review Chuang W 4 participants 6 messages Jun 14, 2022 – Jun 20, 2022

Subsystems (4)
lore
Share & tools
lore

Chuang W

author
#0 lore raw

PATCH 1/1 libbpf: Remove kprobe_event on failed kprobe_open_legacy

In a scenario where livepatch and aggrprobe coexist, the creating kprobe_event using tracefs API will succeed, a trace event (e.g. /debugfs/tracing/events/kprobe/XX) will exist, but perf_event_open() will return an error.

---
 tools/lib/bpf/libbpf.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +9 -3
index 0781fae58a06..d0a36350e22a 100644
@@ -10809,10 +10809,11 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
 	}
 	type = determine_kprobe_perf_type_legacy(probe_name, retprobe);
 	if (type < 0) {
+		err = type;
 		pr_warn("failed to determine legacy kprobe event id for '%s+0x%zx': %s\n",
 			kfunc_name, offset,
-			libbpf_strerror_r(type, errmsg, sizeof(errmsg)));
-		return type;
+			libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
+		goto clear_kprobe_event;
 	}
 	attr.size = sizeof(attr);
 	attr.config = type;
@@ -10826,9 +10827,14 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
 		err = -errno;
 		pr_warn("legacy kprobe perf_event_open() failed: %s\n",
 			libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
-		return err;
+		goto clear_kprobe_event;
 	}
 	return pfd;
+
+clear_kprobe_event:
+	/* Clear the newly added kprobe_event */
+	remove_kprobe_event_legacy(probe_name, retprobe);
+	return err;
 }
 
 struct bpf_link *
Signature · 1 line
2.34.1
#1 lore raw
Quoted 4 lines · Chuang W

In a scenario where livepatch and aggrprobe coexist, the creating kprobe_event using tracefs API will succeed, a trace event (e.g. /debugfs/tracing/events/kprobe/XX) will exist, but perf_event_open() will return an error.

This seems a bit strange from API side. I'm not really familiar with livepatch, but I guess this is UAPI now so fixing add_kprobe_event_legacy to fail is not an option?

Quoted 44 lines
---
 tools/lib/bpf/libbpf.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +9 -3
index 0781fae58a06..d0a36350e22a 100644
@@ -10809,10 +10809,11 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
 	}
 	type = determine_kprobe_perf_type_legacy(probe_name, retprobe);
 	if (type < 0) {
+		err = type;
 		pr_warn("failed to determine legacy kprobe event id for '%s+0x%zx': %s\n",
 			kfunc_name, offset,
-			libbpf_strerror_r(type, errmsg, sizeof(errmsg)));
-		return type;
+			libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
+		goto clear_kprobe_event;
 	}
 	attr.size = sizeof(attr);
 	attr.config = type;
@@ -10826,9 +10827,14 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
 		err = -errno;
 		pr_warn("legacy kprobe perf_event_open() failed: %s\n",
 			libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
-		return err;
+		goto clear_kprobe_event;
 	}
 	return pfd;
+
+clear_kprobe_event:
+	/* Clear the newly added kprobe_event */
+	remove_kprobe_event_legacy(probe_name, retprobe);
+	return err;
 }
 
 struct bpf_link *
Signature · 1 line
2.34.1
author
#2 lore raw

Hi John,

Quoted 11 lines · John Fastabend
Quoted 4 lines · Chuang W

In a scenario where livepatch and aggrprobe coexist, the creating kprobe_event using tracefs API will succeed, a trace event (e.g. /debugfs/tracing/events/kprobe/XX) will exist, but perf_event_open() will return an error.

This seems a bit strange from API side. I'm not really familiar with livepatch, but I guess this is UAPI now so fixing add_kprobe_event_legacy to fail is not an option?

The legacy kprobe API (i.e. tracefs API) has two steps:

1) register_kprobe

$ echo 'p:mykprobe XXX' > /sys/kernel/debug/tracing/kprobe_events
This will create a trace event of mykprobe and register a disable
kprobe that waits to be activated.

2) enable_kprobe
2.1) using syscall perf_event_open
as the following code, perf_event_kprobe_open_legacy (file:
tools/lib/bpf/libbpf.c):
---
attr.type = PERF_TYPE_TRACEPOINT;
pfd = syscall(__NR_perf_event_open, &attr,

              pid < 0 ? -1 : pid, /* pid */
              pid == -1 ? 0 : -1, /* cpu */
              -1 /* group_fd */,  PERF_FLAG_FD_CLOEXEC);
---
In the implementation code of perf_event_open, enable_kprobe() will be executed.
2.2) using shell
$ echo 1 > /sys/kernel/debug/tracing/events/kprobes/mykprobe/enable
As with perf_event_open, enable_kprobe() will also be executed.

When using the same function XXX, kprobe and livepatch cannot coexist, that is, step 2) will return an error (ref: arm_kprobe_ftrace()), however, step 1) is ok!
However, the new kprobe API (i.e. perf kprobe API) aggregates register_kprobe and enable_kprobe, internally fixes the issue on failed enable_kprobe.
But above all, for the legacy kprobe API, I think it should remove kprobe_event on failed add_kprobe_event_legacy() in perf_event_kprobe_open_legacy (file: tools/lib/bpf/libbpf.c).

#3 lore raw
Quoted 39 lines · chuang

Hi John,

Quoted 11 lines · John Fastabend
Quoted 4 lines · Chuang W

In a scenario where livepatch and aggrprobe coexist, the creating kprobe_event using tracefs API will succeed, a trace event (e.g. /debugfs/tracing/events/kprobe/XX) will exist, but perf_event_open() will return an error.

This seems a bit strange from API side. I'm not really familiar with livepatch, but I guess this is UAPI now so fixing add_kprobe_event_legacy to fail is not an option?

The legacy kprobe API (i.e. tracefs API) has two steps:

1) register_kprobe

$ echo 'p:mykprobe XXX' > /sys/kernel/debug/tracing/kprobe_events
This will create a trace event of mykprobe and register a disable
kprobe that waits to be activated.

2) enable_kprobe
2.1) using syscall perf_event_open
as the following code, perf_event_kprobe_open_legacy (file:
tools/lib/bpf/libbpf.c):
---
attr.type = PERF_TYPE_TRACEPOINT;
pfd = syscall(__NR_perf_event_open, &attr,

              pid < 0 ? -1 : pid, /* pid */
              pid == -1 ? 0 : -1, /* cpu */
              -1 /* group_fd */,  PERF_FLAG_FD_CLOEXEC);
---
In the implementation code of perf_event_open, enable_kprobe() will be executed.
2.2) using shell
$ echo 1 > /sys/kernel/debug/tracing/events/kprobes/mykprobe/enable
As with perf_event_open, enable_kprobe() will also be executed.
Quoted 2 lines

When using the same function XXX, kprobe and livepatch cannot coexist, that is, step 2) will return an error (ref: arm_kprobe_ftrace()),

just curious.. is that because of ipmodify flag on ftrace_ops? AFAICS that be a poblem just for kretprobes, cc-ing Masami

thanks,
jirka

Quoted 7 lines

however, step 1) is ok!
However, the new kprobe API (i.e. perf kprobe API) aggregates register_kprobe and enable_kprobe, internally fixes the issue on failed enable_kprobe.
But above all, for the legacy kprobe API, I think it should remove kprobe_event on failed add_kprobe_event_legacy() in perf_event_kprobe_open_legacy (file: tools/lib/bpf/libbpf.c).

author
#4 lore raw

Hi Jiri,

Quoted 4 lines · Jiri Olsa

just curious.. is that because of ipmodify flag on ftrace_ops? AFAICS that be a poblem just for kretprobes, cc-ing Masami

Yes, the core reason is caused by ipmodify flag (not only for kretprobes). Before commit 0bc11ed5ab60 ("kprobes: Allow kprobes coexist with livepatch"), it's very easy to trigger this problem.
The kprobe has other problems and is communicating with Masami.

With this fix, whenever an error is returned after add_kprobe_event_legacy(), this guarantees cleanup of the kprobe event.

maintainer
#5 lore raw
Quoted 5 lines · chuang
Quoted 3 lines · Jiri Olsa

just curious.. is that because of ipmodify flag on ftrace_ops? AFAICS that be a poblem just for kretprobes, cc-ing Masami

Quoted 8 lines

Yes, the core reason is caused by ipmodify flag (not only for kretprobes). Before commit 0bc11ed5ab60 ("kprobes: Allow kprobes coexist with livepatch"), it's very easy to trigger this problem.
The kprobe has other problems and is communicating with Masami.

With this fix, whenever an error is returned after add_kprobe_event_legacy(), this guarantees cleanup of the kprobe event.

The details from this follow-up conversation should definitely be part of the commit description, please add them to a v2 as otherwise context is missing if we look at the commit again in say few months from now. Also, would be good if Masami could provide ack given 0bc11ed5ab60.

Thanks,
Daniel