Skip to contentSkip to conversation

libbpf: Cleanup the legacy kprobe_event on failed add/attach_event()

PATCH Superseded Chuang W 2 participants 5 messages Jun 26, 2022 – Jun 28, 2022

Subsystems (4)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Chuang W

author
#0 lore raw

PATCH v3 1/1 libbpf: Cleanup the legacy kprobe_event on failed add/attach_event()

Before the 0bc11ed5ab60 commit ("kprobes: Allow kprobes coexist with livepatch"), in a scenario where livepatch and kprobe coexist on the same function entry, the creation of kprobe_event using add_kprobe_event_legacy() will be successful, at the same time as a trace event (e.g. /debugfs/tracing/events/kprobe/XXX) will exist, but perf_event_open() will return an error because both livepatch and kprobe use FTRACE_OPS_FL_IPMODIFY. As follows:

1) add a livepatch

$ insmod livepatch-XXX.ko

2) add a kprobe using tracefs API (i.e. add_kprobe_event_legacy)

$ echo 'p:mykprobe XXX' > /sys/kernel/debug/tracing/kprobe_events

3) enable this kprobe (i.e. sys_perf_event_open)

This will return an error, -EBUSY.

On Andrii Nakryiko's comment, few error paths in bpf_program__attach_kprobe_opts() which should need to call remove_kprobe_event_legacy().

With this patch, whenever an error is returned after add_kprobe_event_legacy() or bpf_program__attach_perf_event_opts(), this ensures that the created kprobe_event is cleaned.

---
V2->v3:
- add detail commits
- call remove_kprobe_event_legacy() on failed bpf_program__attach_perf_event_opts()

 tools/lib/bpf/libbpf.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +12 -3
index 49e359cd34df..038b0cb3313f 100644
@@ -10811,10 +10811,11 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
 	}
 	type = determine_kprobe_perf_type_legacy(probe_name, retprobe);
 	if (type < 0) {
+		err = type;
 		pr_warn("failed to determine legacy kprobe event id for '%s+0x%zx': %s\n",
 			kfunc_name, offset,
-			libbpf_strerror_r(type, errmsg, sizeof(errmsg)));
-		return type;
+			libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
+		goto clear_kprobe_event;
 	}
 	attr.size = sizeof(attr);
 	attr.config = type;
@@ -10828,9 +10829,14 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
 		err = -errno;
 		pr_warn("legacy kprobe perf_event_open() failed: %s\n",
 			libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
-		return err;
+		goto clear_kprobe_event;
 	}
 	return pfd;
+
+clear_kprobe_event:
+	/* Clear the newly added legacy kprobe_event */
+	remove_kprobe_event_legacy(probe_name, retprobe);
+	return err;
 }
 
 struct bpf_link *
@@ -10899,6 +10905,9 @@ bpf_program__attach_kprobe_opts(const struct bpf_program *prog,
 
 	return link;
 err_out:
+	/* Clear the newly added legacy kprobe_event */
+	if (legacy)
+		remove_kprobe_event_legacy(legacy_probe, retprobe);
 	free(legacy_probe);
 	return libbpf_err_ptr(err);
 }
Signature · 1 line
2.34.1
#1 lore raw
Quoted 25 lines · Chuang W

Before the 0bc11ed5ab60 commit ("kprobes: Allow kprobes coexist with livepatch"), in a scenario where livepatch and kprobe coexist on the same function entry, the creation of kprobe_event using add_kprobe_event_legacy() will be successful, at the same time as a trace event (e.g. /debugfs/tracing/events/kprobe/XXX) will exist, but perf_event_open() will return an error because both livepatch and kprobe use FTRACE_OPS_FL_IPMODIFY. As follows:

1) add a livepatch

$ insmod livepatch-XXX.ko

2) add a kprobe using tracefs API (i.e. add_kprobe_event_legacy)

$ echo 'p:mykprobe XXX' > /sys/kernel/debug/tracing/kprobe_events

3) enable this kprobe (i.e. sys_perf_event_open)

This will return an error, -EBUSY.

On Andrii Nakryiko's comment, few error paths in bpf_program__attach_kprobe_opts() which should need to call remove_kprobe_event_legacy().

Quoted 5 lines

With this patch, whenever an error is returned after add_kprobe_event_legacy() or bpf_program__attach_perf_event_opts(), this ensures that the created kprobe_event is cleaned.

Is this your full name? Signed-off-by is required to have a full name of a person, please update if it's not

Quoted 32 lines

---
V2->v3:
- add detail commits
- call remove_kprobe_event_legacy() on failed bpf_program__attach_perf_event_opts()

 tools/lib/bpf/libbpf.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +3 -3
index 49e359cd34df..038b0cb3313f 100644
@@ -10811,10 +10811,11 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
        }
        type = determine_kprobe_perf_type_legacy(probe_name, retprobe);
        if (type < 0) {
+               err = type;
                pr_warn("failed to determine legacy kprobe event id for '%s+0x%zx': %s\n",
                        kfunc_name, offset,
-                       libbpf_strerror_r(type, errmsg, sizeof(errmsg)));
-               return type;
+                       libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
+               goto clear_kprobe_event;
        }
        attr.size = sizeof(attr);
        attr.config = type;
@@ -10828,9 +10829,14 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
                err = -errno;
                pr_warn("legacy kprobe perf_event_open() failed: %s\n",
                        libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
-               return err;

tools/lib/bpf/libbpf.c static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe, · line 10841

Quoted 10 lines
tools/lib/bpf/libbpf.c +6 -0
+               goto clear_kprobe_event;
        }
        return pfd;
+
+clear_kprobe_event:
+       /* Clear the newly added legacy kprobe_event */
+       remove_kprobe_event_legacy(probe_name, retprobe);
+       return err;
 }
 

this part looks good

tools/lib/bpf/libbpf.c bpf_program__attach_kprobe_opts(const struct bpf_program *prog, · line 10910

Quoted 8 lines
tools/lib/bpf/libbpf.c +3 -0
 struct bpf_link *
@@ -10899,6 +10905,9 @@ bpf_program__attach_kprobe_opts(const struct bpf_program *prog,
 
        return link;
 err_out:
+       /* Clear the newly added legacy kprobe_event */
+       if (legacy)
+               remove_kprobe_event_legacy(legacy_probe, retprobe);

this one will call remove_kprobe_event_legacy() even if we failed to create that kprobe_event in the first place. So let's maybe add

err_clean_legacy:

    if (legacy)
         remove_kprobe_event_legacy(legacy_probe, retprobe);

before err_out: and goto there if we fail to attach (but not if we fail to create pfd)?

Also, looking through libbpf code, I realized that we have exactly the same problem for uprobes, so please add same fixed to perf_event_uprobe_open_legacy and attach_uprobe_opts. Thanks!

Quoted 6 lines
tools/lib/bpf/libbpf.c +0 -1
        free(legacy_probe);
        return libbpf_err_ptr(err);
 }
--

2.34.1

author
#2 lore raw

Hi Andrii,

Quoted 94 lines · Andrii Nakryiko
Quoted 25 lines · Chuang W

Before the 0bc11ed5ab60 commit ("kprobes: Allow kprobes coexist with livepatch"), in a scenario where livepatch and kprobe coexist on the same function entry, the creation of kprobe_event using add_kprobe_event_legacy() will be successful, at the same time as a trace event (e.g. /debugfs/tracing/events/kprobe/XXX) will exist, but perf_event_open() will return an error because both livepatch and kprobe use FTRACE_OPS_FL_IPMODIFY. As follows:

1) add a livepatch

$ insmod livepatch-XXX.ko

2) add a kprobe using tracefs API (i.e. add_kprobe_event_legacy)

$ echo 'p:mykprobe XXX' > /sys/kernel/debug/tracing/kprobe_events

3) enable this kprobe (i.e. sys_perf_event_open)

This will return an error, -EBUSY.

On Andrii Nakryiko's comment, few error paths in bpf_program__attach_kprobe_opts() which should need to call remove_kprobe_event_legacy().

Quoted 5 lines

With this patch, whenever an error is returned after add_kprobe_event_legacy() or bpf_program__attach_perf_event_opts(), this ensures that the created kprobe_event is cleaned.

Is this your full name? Signed-off-by is required to have a full name of a person, please update if it's not

Quoted 32 lines

---
V2->v3:
- add detail commits
- call remove_kprobe_event_legacy() on failed bpf_program__attach_perf_event_opts()

 tools/lib/bpf/libbpf.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +3 -3
index 49e359cd34df..038b0cb3313f 100644
@@ -10811,10 +10811,11 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
        }
        type = determine_kprobe_perf_type_legacy(probe_name, retprobe);
        if (type < 0) {
+               err = type;
                pr_warn("failed to determine legacy kprobe event id for '%s+0x%zx': %s\n",
                        kfunc_name, offset,
-                       libbpf_strerror_r(type, errmsg, sizeof(errmsg)));
-               return type;
+                       libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
+               goto clear_kprobe_event;
        }
        attr.size = sizeof(attr);
        attr.config = type;
@@ -10828,9 +10829,14 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
                err = -errno;
                pr_warn("legacy kprobe perf_event_open() failed: %s\n",
                        libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
-               return err;

tools/lib/bpf/libbpf.c static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe, · line 10841

Quoted 10 lines
tools/lib/bpf/libbpf.c +6 -0
+               goto clear_kprobe_event;
        }
        return pfd;
+
+clear_kprobe_event:
+       /* Clear the newly added legacy kprobe_event */
+       remove_kprobe_event_legacy(probe_name, retprobe);
+       return err;
 }
 

this part looks good

tools/lib/bpf/libbpf.c bpf_program__attach_kprobe_opts(const struct bpf_program *prog, · line 10910

Quoted 8 lines
tools/lib/bpf/libbpf.c +3 -0
 struct bpf_link *
@@ -10899,6 +10905,9 @@ bpf_program__attach_kprobe_opts(const struct bpf_program *prog,
 
        return link;
 err_out:
+       /* Clear the newly added legacy kprobe_event */
+       if (legacy)
+               remove_kprobe_event_legacy(legacy_probe, retprobe);

this one will call remove_kprobe_event_legacy() even if we failed to create that kprobe_event in the first place. So let's maybe add

Quoted 7 lines

err_clean_legacy:

    if (legacy)
         remove_kprobe_event_legacy(legacy_probe, retprobe);

before err_out: and goto there if we fail to attach (but not if we fail to create pfd)?

Nice, I will modify it.

Quoted 5 lines

Also, looking through libbpf code, I realized that we have exactly the same problem for uprobes, so please add same fixed to perf_event_uprobe_open_legacy and attach_uprobe_opts. Thanks!

Oh, yes. I also noticed this problem for uprobes, I was planning to submit a patch for uprobes.
Do you think I should submit another patch for uprobes or combine kprobes and uprobes into one?

Thanks,

Quoted 8 lines
Quoted 8 lines
tools/lib/bpf/libbpf.c +0 -1
        free(legacy_probe);
        return libbpf_err_ptr(err);
 }
--

2.34.1

#3 lore raw
Quoted 115 lines · Chuang W

Hi Andrii,

Quoted 94 lines · Andrii Nakryiko
Quoted 25 lines · Chuang W

Before the 0bc11ed5ab60 commit ("kprobes: Allow kprobes coexist with livepatch"), in a scenario where livepatch and kprobe coexist on the same function entry, the creation of kprobe_event using add_kprobe_event_legacy() will be successful, at the same time as a trace event (e.g. /debugfs/tracing/events/kprobe/XXX) will exist, but perf_event_open() will return an error because both livepatch and kprobe use FTRACE_OPS_FL_IPMODIFY. As follows:

1) add a livepatch

$ insmod livepatch-XXX.ko

2) add a kprobe using tracefs API (i.e. add_kprobe_event_legacy)

$ echo 'p:mykprobe XXX' > /sys/kernel/debug/tracing/kprobe_events

3) enable this kprobe (i.e. sys_perf_event_open)

This will return an error, -EBUSY.

On Andrii Nakryiko's comment, few error paths in bpf_program__attach_kprobe_opts() which should need to call remove_kprobe_event_legacy().

Quoted 5 lines

With this patch, whenever an error is returned after add_kprobe_event_legacy() or bpf_program__attach_perf_event_opts(), this ensures that the created kprobe_event is cleaned.

Is this your full name? Signed-off-by is required to have a full name of a person, please update if it's not

Quoted 32 lines

---
V2->v3:
- add detail commits
- call remove_kprobe_event_legacy() on failed bpf_program__attach_perf_event_opts()

 tools/lib/bpf/libbpf.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +3 -3
index 49e359cd34df..038b0cb3313f 100644
@@ -10811,10 +10811,11 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
        }
        type = determine_kprobe_perf_type_legacy(probe_name, retprobe);
        if (type < 0) {
+               err = type;
                pr_warn("failed to determine legacy kprobe event id for '%s+0x%zx': %s\n",
                        kfunc_name, offset,
-                       libbpf_strerror_r(type, errmsg, sizeof(errmsg)));
-               return type;
+                       libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
+               goto clear_kprobe_event;
        }
        attr.size = sizeof(attr);
        attr.config = type;
@@ -10828,9 +10829,14 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
                err = -errno;
                pr_warn("legacy kprobe perf_event_open() failed: %s\n",
                        libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
-               return err;

tools/lib/bpf/libbpf.c static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe, · line 10841

Quoted 10 lines
tools/lib/bpf/libbpf.c +6 -0
+               goto clear_kprobe_event;
        }
        return pfd;
+
+clear_kprobe_event:
+       /* Clear the newly added legacy kprobe_event */
+       remove_kprobe_event_legacy(probe_name, retprobe);
+       return err;
 }
 

this part looks good

tools/lib/bpf/libbpf.c bpf_program__attach_kprobe_opts(const struct bpf_program *prog, · line 10910

Quoted 8 lines
tools/lib/bpf/libbpf.c +3 -0
 struct bpf_link *
@@ -10899,6 +10905,9 @@ bpf_program__attach_kprobe_opts(const struct bpf_program *prog,
 
        return link;
 err_out:
+       /* Clear the newly added legacy kprobe_event */
+       if (legacy)
+               remove_kprobe_event_legacy(legacy_probe, retprobe);

this one will call remove_kprobe_event_legacy() even if we failed to create that kprobe_event in the first place. So let's maybe add

Quoted 7 lines

err_clean_legacy:

    if (legacy)
         remove_kprobe_event_legacy(legacy_probe, retprobe);

before err_out: and goto there if we fail to attach (but not if we fail to create pfd)?

Nice, I will modify it.

Quoted 5 lines

Also, looking through libbpf code, I realized that we have exactly the same problem for uprobes, so please add same fixed to perf_event_uprobe_open_legacy and attach_uprobe_opts. Thanks!

Quoted 5 lines

Oh, yes. I also noticed this problem for uprobes, I was planning to submit a patch for uprobes.
Do you think I should submit another patch for uprobes or combine kprobes and uprobes into one?

two separate patches make more sense, but send them as a patch series?

Quoted 9 lines

Thanks,

Quoted 8 lines
Quoted 8 lines
tools/lib/bpf/libbpf.c +0 -1
        free(legacy_probe);
        return libbpf_err_ptr(err);
 }
--

2.34.1

author
#4 lore raw

Ok.

Thanks,

Quoted 134 lines · Andrii Nakryiko
Quoted 115 lines · Chuang W

Hi Andrii,

Quoted 94 lines · Andrii Nakryiko
Quoted 25 lines · Chuang W

Before the 0bc11ed5ab60 commit ("kprobes: Allow kprobes coexist with livepatch"), in a scenario where livepatch and kprobe coexist on the same function entry, the creation of kprobe_event using add_kprobe_event_legacy() will be successful, at the same time as a trace event (e.g. /debugfs/tracing/events/kprobe/XXX) will exist, but perf_event_open() will return an error because both livepatch and kprobe use FTRACE_OPS_FL_IPMODIFY. As follows:

1) add a livepatch

$ insmod livepatch-XXX.ko

2) add a kprobe using tracefs API (i.e. add_kprobe_event_legacy)

$ echo 'p:mykprobe XXX' > /sys/kernel/debug/tracing/kprobe_events

3) enable this kprobe (i.e. sys_perf_event_open)

This will return an error, -EBUSY.

On Andrii Nakryiko's comment, few error paths in bpf_program__attach_kprobe_opts() which should need to call remove_kprobe_event_legacy().

Quoted 5 lines

With this patch, whenever an error is returned after add_kprobe_event_legacy() or bpf_program__attach_perf_event_opts(), this ensures that the created kprobe_event is cleaned.

Is this your full name? Signed-off-by is required to have a full name of a person, please update if it's not

Quoted 32 lines

---
V2->v3:
- add detail commits
- call remove_kprobe_event_legacy() on failed bpf_program__attach_perf_event_opts()

 tools/lib/bpf/libbpf.c | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)
tools/lib/bpf/libbpf.c +3 -3
index 49e359cd34df..038b0cb3313f 100644
@@ -10811,10 +10811,11 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
        }
        type = determine_kprobe_perf_type_legacy(probe_name, retprobe);
        if (type < 0) {
+               err = type;
                pr_warn("failed to determine legacy kprobe event id for '%s+0x%zx': %s\n",
                        kfunc_name, offset,
-                       libbpf_strerror_r(type, errmsg, sizeof(errmsg)));
-               return type;
+                       libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
+               goto clear_kprobe_event;
        }
        attr.size = sizeof(attr);
        attr.config = type;
@@ -10828,9 +10829,14 @@ static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe,
                err = -errno;
                pr_warn("legacy kprobe perf_event_open() failed: %s\n",
                        libbpf_strerror_r(err, errmsg, sizeof(errmsg)));
-               return err;

tools/lib/bpf/libbpf.c static int perf_event_kprobe_open_legacy(const char *probe_name, bool retprobe, · line 10841

Quoted 10 lines
tools/lib/bpf/libbpf.c +6 -0
+               goto clear_kprobe_event;
        }
        return pfd;
+
+clear_kprobe_event:
+       /* Clear the newly added legacy kprobe_event */
+       remove_kprobe_event_legacy(probe_name, retprobe);
+       return err;
 }
 

this part looks good

tools/lib/bpf/libbpf.c bpf_program__attach_kprobe_opts(const struct bpf_program *prog, · line 10910

Quoted 8 lines
tools/lib/bpf/libbpf.c +3 -0
 struct bpf_link *
@@ -10899,6 +10905,9 @@ bpf_program__attach_kprobe_opts(const struct bpf_program *prog,
 
        return link;
 err_out:
+       /* Clear the newly added legacy kprobe_event */
+       if (legacy)
+               remove_kprobe_event_legacy(legacy_probe, retprobe);

this one will call remove_kprobe_event_legacy() even if we failed to create that kprobe_event in the first place. So let's maybe add

Quoted 7 lines

err_clean_legacy:

    if (legacy)
         remove_kprobe_event_legacy(legacy_probe, retprobe);

before err_out: and goto there if we fail to attach (but not if we fail to create pfd)?

Nice, I will modify it.

Quoted 5 lines

Also, looking through libbpf code, I realized that we have exactly the same problem for uprobes, so please add same fixed to perf_event_uprobe_open_legacy and attach_uprobe_opts. Thanks!

Quoted 5 lines

Oh, yes. I also noticed this problem for uprobes, I was planning to submit a patch for uprobes.
Do you think I should submit another patch for uprobes or combine kprobes and uprobes into one?

two separate patches make more sense, but send them as a patch series?

Quoted 9 lines

Thanks,

Quoted 8 lines
Quoted 8 lines
tools/lib/bpf/libbpf.c +0 -1
        free(legacy_probe);
        return libbpf_err_ptr(err);
 }
--

2.34.1