Skip to contentSkip to conversation

Upgrading uprobe and kprobe to their `multi` counterparts.

RFC PATCH Superseded Varun R Mallya 4 participants 16 messages Mar 30 – Apr 1

Subsystems (7)
Kselftest BPF Tracing BPF selftests +3 more subsystems

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Upgrading uprobe and kprobe to their `multi` counterparts.

This RFC patch explores auto-upgrading standard uprobes/kprobes to use the multi-uprobe/multi-kprobe infrastructure when applicable.

Background:
The BPF token concept allows privileged operations inside non-privileged user namespaces. However, attaching standard uprobes and kprobes currently relies on the perf_event_open() syscall, which is not BPF token-aware. Multi-uprobes and multi-kprobes bypass perf_event_open() entirely, attaching via the bpf() syscall instead, making them compatible with BPF tokens.

To bridge this gap, the goal is to switch SEC("uprobe") and SEC("kprobe") to use multi-uprobe/kprobe under the hood. To maintain backward compatibility for cases where singular uprobes are explicitly desired, this patch also introduces SEC("uprobe.single") and SEC("kprobe.single").

Current Implementation:
The decision to upgrade is made in bpf_object_prepare_progs() (According to the feedback received in [1].)
If the kernel supports FEAT_UPROBE_MULTI_LINK,
we intercept programs with section names matching "u[ret]probe" and change their expected_attach_type to BPF_TRACE_UPROBE_MULTI.

A similar thing is done with kprobes, but I had to add a new FEAT_KPROBE_MULTI_LINK to the kern_feature_id struct along with it's implementation similar to it's uprobe counterpart.

Just one selftest had to be changed for uprobe but quite a few had to be changed for kprobe. The decision to change them have been explained in the commit descriptions.

Some Observations:
- Earlier, I noted that uprobe and uprobe_multi are equivalent. I have found out that uprobe_multi does not support versioned symbols such as those in tools/testing/selftests/bpf/progs/test_uprobe.c like `SEC("uprobe/./liburandom_read.so: \
urandlib_api_sameoffset@LIBURANDOM_READ_1.0.0")`.
I believe this is something I need to fix as well to be able to support versioned symbols. Right now, these have been excluded from upgradation.

My questions:
- I want know if the conditions I have placed for FEAT_KPROBE_MULTI_LINK to be true in probe_kprobe_multi_link() are correct. I feel like it's incomplete and would need some more things to say definitively that Kprobe-multi works on a particular kernel (especially with respect to the error value like that in it's uprobe counterpart.). I would really appreciate suggestions here.

- I had to exclude sleepable kprobes from being upgraded due to tests failing. I want to know if that was a good desicion.

- I had to change the get_func_ip_test selftest to ?kprobe.single from ?kprobe due to offsets that were added later (after prepare_progs ran). This means that anyone using ?kprobe along with offsets will have to change things which is not ideal. Is it alright if I exclude this class of SEC_DEFs from getting upgraded ?

P.S : Sorry for the incredibly late v2 on the reviews for that patch, I was unsure of the changes I had made and wanted to thoroughly verify things before sending them out.

v1->v2 changes: All suggestions from Andrii's review on v1 were made as well as support for kprobe upgrade was added.

[1]: https://lore.kernel.org/bpf/[email protected]/

Varun R Mallya (3):
libbpf: Auto-upgrade uprobes to multi-uprobes when supported
libbpf: Add FEAT_KPROBE_MULTI_LINK feature probe.
libbpf: Auto-upgrade kprobes to multi-kprobes when supported

 tools/lib/bpf/features.c                      |  37 ++++++
 tools/lib/bpf/libbpf.c                        | 114 ++++++++++++++++--
 tools/lib/bpf/libbpf_internal.h               |   2 +
 .../selftests/bpf/progs/get_func_ip_test.c    |   2 +-
 .../selftests/bpf/progs/missed_kprobe.c       |   4 +-
 .../bpf/progs/test_attach_probe_manual.c      |   4 +-
 .../selftests/bpf/progs/test_fill_link_info.c |   4 +-
 7 files changed, 151 insertions(+), 16 deletions(-)
Signature · 1 line
2.52.0
#1 lore raw

RFC PATCH v2 1/3 libbpf: Auto-upgrade uprobes to multi-uprobes when supported

This patch modifies libbpf to automatically "upgrade" standard SEC("uprobe") and SEC("uretprobe") programs to use the multi-uprobe infrastructure (BPF_TRACE_UPROBE_MULTI) at load time if the kernel supports it, making them compatible with BPF tokens.

To maintain backward compatibility and handle rare cases where singular uprobes are required, new SEC("uprobe.single") and SEC("uretprobe.single") section types are introduced. These force libbpf to use the legacy perf_event_open() attachment path.

tools/testing/selftests/bpf/progs/test_fill_link_info.c has been modified to use SEC("uprobe.single") as it asserts the program type to be BPF_LINK_TYPE_PERF_EVENT and checks properties related to uprobes that use perf.

---
 tools/lib/bpf/libbpf.c                        | 53 +++++++++++++++++--
 .../selftests/bpf/progs/test_fill_link_info.c |  2 +-
 2 files changed, 51 insertions(+), 4 deletions(-)
tools/lib/bpf/libbpf.c +50 -3
index 1eaa7527d4da..bd7b6f486430 100644
@@ -8248,6 +8248,23 @@ static int bpf_object_prepare_progs(struct bpf_object *obj)
 
 	for (i = 0; i < obj->nr_programs; i++) {
 		prog = &obj->programs[i];
+
+		if (kernel_supports(obj, FEAT_UPROBE_MULTI_LINK)) {
+			const char *sec_name = prog->sec_name;
+			/* Here, we filter out for u[ret]probe or "u[ret]probe/"
+			 * but we leave out anything with an '@'
+			 * in it as uprobe_multi does not support versioned
+			 * symbols yet, so we don't upgrade.
+			 */
+			if (((strncmp(sec_name, "uprobe", 6) == 0 &&
+			      (sec_name[6] == '/' || sec_name[6] == '\0')) ||
+			     (strncmp(sec_name, "uretprobe", 9) == 0 &&
+			      (sec_name[9] == '/' || sec_name[9] == '\0'))) &&
+			    !strchr(sec_name, '@')) {
+				prog->expected_attach_type = BPF_TRACE_UPROBE_MULTI;
+			}
+		}
+
 		err = bpf_object__sanitize_prog(obj, prog);
 		if (err)
 			return err;
@@ -9909,9 +9926,11 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("kprobe+",		KPROBE,	0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uprobe+",		KPROBE,	0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uprobe.s+",		KPROBE,	0, SEC_SLEEPABLE, attach_uprobe),
+	SEC_DEF("uprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),
 	SEC_DEF("kretprobe+",		KPROBE, 0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uretprobe+",		KPROBE, 0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uretprobe.s+",		KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
+	SEC_DEF("uretprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),
 	SEC_DEF("kprobe.multi+",	KPROBE,	BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
 	SEC_DEF("kretprobe.multi+",	KPROBE,	BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
 	SEC_DEF("kprobe.session+",	KPROBE,	BPF_TRACE_KPROBE_SESSION, SEC_NONE, attach_kprobe_session),
@@ -12737,6 +12756,32 @@ bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid,
 		func_offset += sym_off;
 	}
 
+	/* This provides backwards compatibility to programs using uprobe, but
+	 * have been auto-upgraded to multi uprobe.
+	 */
+	if (prog->expected_attach_type == BPF_TRACE_UPROBE_MULTI) {
+		LIBBPF_OPTS(bpf_uprobe_multi_opts, multi_opts);
+		unsigned long offsets[1] = {func_offset};
+		__u64 bpf_cookie;
+
+		multi_opts.retprobe = OPTS_GET(opts, retprobe, false);
+		if (offsets[0] || func_name) {
+			multi_opts.offsets = offsets;
+			multi_opts.cnt = 1;
+		}
+		if (ref_ctr_off) {
+			multi_opts.ref_ctr_offsets = &ref_ctr_off;
+			multi_opts.cnt = 1;
+		}
+		bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
+		if (bpf_cookie) {
+			multi_opts.cookies = &bpf_cookie;
+			multi_opts.cnt = 1;
+		}
+
+		return bpf_program__attach_uprobe_multi(prog, pid, binary_path,
+							NULL, &multi_opts);
+	}
 	legacy = determine_uprobe_perf_type() < 0;
 	switch (attach_mode) {
 	case PROBE_ATTACH_MODE_LEGACY:
@@ -12830,6 +12875,7 @@ static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf
 	char *probe_type = NULL, *binary_path = NULL, *func_name = NULL, *func_off;
 	int n, c, ret = -EINVAL;
 	long offset = 0;
+	bool is_retprobe;
 
 	*link = NULL;
 
@@ -12856,13 +12902,14 @@ static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf
 			else
 				offset = 0;
 		}
-		opts.retprobe = strcmp(probe_type, "uretprobe") == 0 ||
-				strcmp(probe_type, "uretprobe.s") == 0;
-		if (opts.retprobe && offset != 0) {
+		is_retprobe = strcmp(probe_type, "uretprobe") == 0 ||
+			      strcmp(probe_type, "uretprobe.s") == 0;
+		if (is_retprobe && offset != 0) {
 			pr_warn("prog '%s': uretprobes do not support offset specification\n",
 				prog->name);
 			break;
 		}
+		opts.retprobe = is_retprobe;
 		opts.func_name = func_name;
 		*link = bpf_program__attach_uprobe_opts(prog, -1, binary_path, offset, &opts);
 		ret = libbpf_get_error(*link);
tools/testing/selftests/bpf/progs/test_fill_link_info.c +1 -1
index fac33a14f200..8e47a818462f 100644
@@ -28,7 +28,7 @@ int BPF_PROG(kprobe_run)
 	return 0;
 }
 
-SEC("uprobe")
+SEC("uprobe.single")
 int BPF_PROG(uprobe_run)
 {
 	return 0;
Signature · 1 line
2.52.0
#2 lore raw

Re: [RFC PATCH bpf-next v2 1/3] libbpf: Auto-upgrade uprobes to multi-uprobes when supported

When func_offset is 0 and func_name is NULL, this condition is false and multi_opts.offsets and multi_opts.cnt are never set. The subsequent call to bpf_program__attach_uprobe_multi() then fails…
Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
index 9ea41f40dc82..056bc50a0bbb 100644

[ ... ]

Quoted 7 lines
tools/lib/bpf/libbpf.c +3 -0
@@ -12783,6 +12802,32 @@ bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid,
 		func_offset += sym_off;
 	}
 
+	/* This provides backwards compatibility to programs using uprobe, but
+	 * have been auto-upgraded to multi uprobe.
+	 */

tools/lib/bpf/libbpf.c bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid, · line 12817

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+	if (prog->expected_attach_type == BPF_TRACE_UPROBE_MULTI) {
+		LIBBPF_OPTS(bpf_uprobe_multi_opts, multi_opts);
+		unsigned long offsets[1] = {func_offset};
+		__u64 bpf_cookie;
+
+		multi_opts.retprobe = OPTS_GET(opts, retprobe, false);
+		if (offsets[0] || func_name) {
+			multi_opts.offsets = offsets;
+			multi_opts.cnt = 1;
+		}

When func_offset is 0 and func_name is NULL, this condition is false and multi_opts.offsets and multi_opts.cnt are never set. The subsequent call to bpf_program__attach_uprobe_multi() then fails with -EINVAL because both func_pattern and cnt are 0:

bpf_program__attach_uprobe_multi() {
    ...
    if (!func_pattern && cnt == 0)
        return libbpf_err_ptr(-EINVAL);
    ...
}

This can happen when callers use bpf_program__attach_uprobe() on an auto-upgraded program with offset 0:

bpf_program__attach_uprobe() {
    DECLARE_LIBBPF_OPTS(bpf_uprobe_opts, opts, .retprobe = retprobe);
    return bpf_program__attach_uprobe_opts(..., func_offset, &opts);
}

Here func_name is never set in opts. The old perf_event path handled offset 0 without issue. Should multi_opts.offsets and multi_opts.cnt always be set unconditionally, since offset 0 is a valid uprobe offset?

Quoted 13 lines
tools/lib/bpf/libbpf.c +13 -0
+		if (ref_ctr_off) {
+			multi_opts.ref_ctr_offsets = &ref_ctr_off;
+			multi_opts.cnt = 1;
+		}
+		bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
+		if (bpf_cookie) {
+			multi_opts.cookies = &bpf_cookie;
+			multi_opts.cnt = 1;
+		}
+
+		return bpf_program__attach_uprobe_multi(prog, pid, binary_path,
+							NULL, &multi_opts);
+	}

[ ... ]

tools/lib/bpf/libbpf.c static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf · line 12952

Quoted 9 lines
tools/lib/bpf/libbpf.c +2 -3
@@ -12902,13 +12948,14 @@ static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf
 			else
 				offset = 0;
 		}
-		opts.retprobe = strcmp(probe_type, "uretprobe") == 0 ||
-				strcmp(probe_type, "uretprobe.s") == 0;
-		if (opts.retprobe && offset != 0) {
+		is_retprobe = strcmp(probe_type, "uretprobe") == 0 ||
+			      strcmp(probe_type, "uretprobe.s") == 0;

Shouldn't this also match "uretprobe.single"? When auto-attaching with SEC("uretprobe.single/binary:func"), sscanf sets probe_type to "uretprobe.single", which doesn't match either "uretprobe" or "uretprobe.s". As a result is_retprobe is false and opts.retprobe stays false, so the program gets attached as a regular uprobe instead of a return probe.

Something like strncmp(probe_type, "uretprobe", 9) == 0 would cover all uretprobe variants ("uretprobe", "uretprobe.s", "uretprobe.single").

tools/lib/bpf/libbpf.c static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf · line 12915

Quoted 9 lines
tools/lib/bpf/libbpf.c +2 -0
+		if (is_retprobe && offset != 0) {
 			pr_warn("prog '%s': uretprobes do not support offset specification\n",
 				prog->name);
 			break;
 		}
+		opts.retprobe = is_retprobe;
 		opts.func_name = func_name;
 		*link = bpf_program__attach_uprobe_opts(prog, -1, binary_path, offset, &opts);
 		ret = libbpf_get_error(*link);

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/23741893093

#3 lore raw

Re: [RFC PATCH bpf-next v2 1/3] libbpf: Auto-upgrade uprobes to multi-uprobes when supported

In reply to Varun R Mallya
Quoted 27 lines · Varun R Mallya

This patch modifies libbpf to automatically "upgrade" standard SEC("uprobe") and SEC("uretprobe") programs to use the multi-uprobe infrastructure (BPF_TRACE_UPROBE_MULTI) at load time if the kernel supports it, making them compatible with BPF tokens.

To maintain backward compatibility and handle rare cases where singular uprobes are required, new SEC("uprobe.single") and SEC("uretprobe.single") section types are introduced. These force libbpf to use the legacy perf_event_open() attachment path.

tools/testing/selftests/bpf/progs/test_fill_link_info.c has been modified to use SEC("uprobe.single") as it asserts the program type to be BPF_LINK_TYPE_PERF_EVENT and checks properties related to uprobes that use perf.

---
 tools/lib/bpf/libbpf.c                        | 53 +++++++++++++++++--
 .../selftests/bpf/progs/test_fill_link_info.c |  2 +-
 2 files changed, 51 insertions(+), 4 deletions(-)
tools/lib/bpf/libbpf.c +0 -0
index 1eaa7527d4da..bd7b6f486430 100644
@@ -8248,6 +8248,23 @@ static int bpf_object_prepare_progs(struct bpf_object *obj)
 

tools/lib/bpf/libbpf.c static int bpf_object_prepare_progs(struct bpf_object *obj) · line 8258

Quoted 10 lines
tools/lib/bpf/libbpf.c +8 -0
 	for (i = 0; i < obj->nr_programs; i++) {
 		prog = &obj->programs[i];
+
+		if (kernel_supports(obj, FEAT_UPROBE_MULTI_LINK)) {
+			const char *sec_name = prog->sec_name;
+			/* Here, we filter out for u[ret]probe or "u[ret]probe/"
+			 * but we leave out anything with an '@'
+			 * in it as uprobe_multi does not support versioned
+			 * symbols yet, so we don't upgrade.
+			 */

nice, I missed that uprobe.multi does not support versioned symbols, I guess we should fix that

tools/lib/bpf/libbpf.c static int bpf_object_prepare_progs(struct bpf_object *obj) · line 8259

Quoted 1 line
tools/lib/bpf/libbpf.c +1 -0
+			if (((strncmp(sec_name, "uprobe", 6) == 0 &&

str_has_pfx ?

Quoted 12 lines
tools/lib/bpf/libbpf.c +8 -0
+			      (sec_name[6] == '/' || sec_name[6] == '\0')) ||
+			     (strncmp(sec_name, "uretprobe", 9) == 0 &&
+			      (sec_name[9] == '/' || sec_name[9] == '\0'))) &&
+			    !strchr(sec_name, '@')) {
+				prog->expected_attach_type = BPF_TRACE_UPROBE_MULTI;
+			}
+		}
+
 		err = bpf_object__sanitize_prog(obj, prog);
 		if (err)
 			return err;

tools/lib/bpf/libbpf.c static const struct bpf_sec_def section_defs[] = { · line 9929

Quoted 4 lines
tools/lib/bpf/libbpf.c +1 -0
@@ -9909,9 +9926,11 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("kprobe+",		KPROBE,	0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uprobe+",		KPROBE,	0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uprobe.s+",		KPROBE,	0, SEC_SLEEPABLE, attach_uprobe),
+	SEC_DEF("uprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),

should we add sleepable counterparts?

tools/lib/bpf/libbpf.c static const struct bpf_sec_def section_defs[] = { · line 9933

Quoted 4 lines
tools/lib/bpf/libbpf.c +1 -0
 	SEC_DEF("kretprobe+",		KPROBE, 0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uretprobe+",		KPROBE, 0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uretprobe.s+",		KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
+	SEC_DEF("uretprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),

just an idea for discussion.. I wonder if it'd be better to add new uprobe section that will upgrade itself to uprobe.multi if it's present, instead of changing the existing (expected) type

but I guess we want existing uprobe programs to benefit from that and there's not really a reason anyone would want perf based uprobe when uprobe_multi is supported

ok I talked myself out of it ;-)

Quoted 8 lines
tools/lib/bpf/libbpf.c +1 -0
 	SEC_DEF("kprobe.multi+",	KPROBE,	BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
 	SEC_DEF("kretprobe.multi+",	KPROBE,	BPF_TRACE_KPROBE_MULTI, SEC_NONE, attach_kprobe_multi),
 	SEC_DEF("kprobe.session+",	KPROBE,	BPF_TRACE_KPROBE_SESSION, SEC_NONE, attach_kprobe_session),
@@ -12737,6 +12756,32 @@ bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid,
 		func_offset += sym_off;
 	}
 
+	/* This provides backwards compatibility to programs using uprobe, but

tools/lib/bpf/libbpf.c bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid, · line 12769

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+	 * have been auto-upgraded to multi uprobe.
+	 */
+	if (prog->expected_attach_type == BPF_TRACE_UPROBE_MULTI) {
+		LIBBPF_OPTS(bpf_uprobe_multi_opts, multi_opts);
+		unsigned long offsets[1] = {func_offset};
+		__u64 bpf_cookie;
+
+		multi_opts.retprobe = OPTS_GET(opts, retprobe, false);
+		if (offsets[0] || func_name) {
+			multi_opts.offsets = offsets;

could we do the same as for ref_ctr_off case and drop the offsets array?

     multi_opts.offsets = &func_offset;

tools/lib/bpf/libbpf.c bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid, · line 12779

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+			multi_opts.cnt = 1;
+		}
+		if (ref_ctr_off) {
+			multi_opts.ref_ctr_offsets = &ref_ctr_off;
+			multi_opts.cnt = 1;
+		}
+		bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
+		if (bpf_cookie) {
+			multi_opts.cookies = &bpf_cookie;
+			multi_opts.cnt = 1;

I think it's better just to set multi_opts.cnt = 1 once outside those if conditions

Quoted 21 lines
tools/lib/bpf/libbpf.c +6 -1
+		}
+
+		return bpf_program__attach_uprobe_multi(prog, pid, binary_path,
+							NULL, &multi_opts);
+	}
 	legacy = determine_uprobe_perf_type() < 0;
 	switch (attach_mode) {
 	case PROBE_ATTACH_MODE_LEGACY:
@@ -12830,6 +12875,7 @@ static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf
 	char *probe_type = NULL, *binary_path = NULL, *func_name = NULL, *func_off;
 	int n, c, ret = -EINVAL;
 	long offset = 0;
+	bool is_retprobe;
 
 	*link = NULL;
 
@@ -12856,13 +12902,14 @@ static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf
 			else
 				offset = 0;
 		}
-		opts.retprobe = strcmp(probe_type, "uretprobe") == 0 ||

tools/lib/bpf/libbpf.c static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf · line 12912

Quoted 10 lines
tools/lib/bpf/libbpf.c +4 -2
-				strcmp(probe_type, "uretprobe.s") == 0;
-		if (opts.retprobe && offset != 0) {
+		is_retprobe = strcmp(probe_type, "uretprobe") == 0 ||
+			      strcmp(probe_type, "uretprobe.s") == 0;
+		if (is_retprobe && offset != 0) {
 			pr_warn("prog '%s': uretprobes do not support offset specification\n",
 				prog->name);
 			break;
 		}
+		opts.retprobe = is_retprobe;

is there any functional change above? looks like just opts.retprobe is replaced with is_retprobe ?

jirka

#4 lore raw

Re: [RFC PATCH bpf-next v2 1/3] libbpf: Auto-upgrade uprobes to multi-uprobes when supported

Quoted 10 lines · Jiri Olsa

tools/lib/bpf/libbpf.c static int bpf_object_prepare_progs(struct bpf_object *obj) · line 8258

Quoted 6 lines · Varun R Mallya
tools/lib/bpf/libbpf.c +6 -0
+			const char *sec_name = prog->sec_name;
+			/* Here, we filter out for u[ret]probe or "u[ret]probe/"
+			 * but we leave out anything with an '@'
+			 * in it as uprobe_multi does not support versioned
+			 * symbols yet, so we don't upgrade.
+			 */

nice, I missed that uprobe.multi does not support versioned symbols, I guess we should fix that

Thanks! I intend to fix that after I am done with this patch.

Quoted 4 lines

tools/lib/bpf/libbpf.c static int bpf_object_prepare_progs(struct bpf_object *obj) · line 8259

Quoted 1 line
tools/lib/bpf/libbpf.c +1 -0
+			if (((strncmp(sec_name, "uprobe", 6) == 0 &&

str_has_pfx ?

Implementing on v3. This looks much cleaner.

Quoted 18 lines

tools/lib/bpf/libbpf.c static const struct bpf_sec_def section_defs[] = { · line 9929

Quoted 5 lines
tools/lib/bpf/libbpf.c +1 -0
@@ -9909,9 +9926,11 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("kprobe+",		KPROBE,	0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uprobe+",		KPROBE,	0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uprobe.s+",		KPROBE,	0, SEC_SLEEPABLE, attach_uprobe),
+	SEC_DEF("uprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),

should we add sleepable counterparts?

tools/lib/bpf/libbpf.c static const struct bpf_sec_def section_defs[] = { · line 9933

Quoted 4 lines
tools/lib/bpf/libbpf.c +1 -0
 	SEC_DEF("kretprobe+",		KPROBE, 0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uretprobe+",		KPROBE, 0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uretprobe.s+",		KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
+	SEC_DEF("uretprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),

just an idea for discussion.. I wonder if it'd be better to add new uprobe section that will upgrade itself to uprobe.multi if it's present, instead of changing the existing (expected) type

Quoted 5 lines

but I guess we want existing uprobe programs to benefit from that and there's not really a reason anyone would want perf based uprobe when uprobe_multi is supported

ok I talked myself out of it ;-)

Yeah, that does seem like it's redundant. I think integrating this into uprobe and kprobe is the best we can do. I have tried my best to ensure that it does not really break any current functionality though.

Quoted 10 lines

tools/lib/bpf/libbpf.c bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid, · line 12769

Quoted 7 lines
tools/lib/bpf/libbpf.c +7 -0
+		LIBBPF_OPTS(bpf_uprobe_multi_opts, multi_opts);
+		unsigned long offsets[1] = {func_offset};
+		__u64 bpf_cookie;
+
+		multi_opts.retprobe = OPTS_GET(opts, retprobe, false);
+		if (offsets[0] || func_name) {
+			multi_opts.offsets = offsets;

could we do the same as for ref_ctr_off case and drop the offsets array?

Quoted 2 lines
     multi_opts.offsets = &func_offset;

An artifact from a previous version. Fixing this.

Quoted 4 lines
Quoted 4 lines
tools/lib/bpf/libbpf.c +4 -0
+			multi_opts.cnt = 1;
+		}
+		if (ref_ctr_off) {
+			multi_opts.ref_ctr_offsets = &ref_ctr_off;
Quoted 8 lines

tools/lib/bpf/libbpf.c bpf_program__attach_uprobe_opts(const struct bpf_program *prog, pid_t pid, · line 12779

Quoted 6 lines
tools/lib/bpf/libbpf.c +6 -0
+			multi_opts.cnt = 1;
+		}
+		bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
+		if (bpf_cookie) {
+			multi_opts.cookies = &bpf_cookie;
+			multi_opts.cnt = 1;

I think it's better just to set multi_opts.cnt = 1 once outside those if conditions

Fixed this as well. I think it solves a part of the AI review as well.

Quoted 35 lines
Quoted 21 lines
tools/lib/bpf/libbpf.c +6 -1
+		}
+
+		return bpf_program__attach_uprobe_multi(prog, pid, binary_path,
+							NULL, &multi_opts);
+	}
 	legacy = determine_uprobe_perf_type() < 0;
 	switch (attach_mode) {
 	case PROBE_ATTACH_MODE_LEGACY:
@@ -12830,6 +12875,7 @@ static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf
 	char *probe_type = NULL, *binary_path = NULL, *func_name = NULL, *func_off;
 	int n, c, ret = -EINVAL;
 	long offset = 0;
+	bool is_retprobe;
 
 	*link = NULL;
 
@@ -12856,13 +12902,14 @@ static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf
 			else
 				offset = 0;
 		}
-		opts.retprobe = strcmp(probe_type, "uretprobe") == 0 ||

tools/lib/bpf/libbpf.c static int attach_uprobe(const struct bpf_program *prog, long cookie, struct bpf · line 12912

Quoted 10 lines
tools/lib/bpf/libbpf.c +4 -2
-				strcmp(probe_type, "uretprobe.s") == 0;
-		if (opts.retprobe && offset != 0) {
+		is_retprobe = strcmp(probe_type, "uretprobe") == 0 ||
+			      strcmp(probe_type, "uretprobe.s") == 0;
+		if (is_retprobe && offset != 0) {
 			pr_warn("prog '%s': uretprobes do not support offset specification\n",
 				prog->name);
 			break;
 		}
+		opts.retprobe = is_retprobe;

is there any functional change above? looks like just opts.retprobe is replaced with is_retprobe ?

Quoted 1 line

jirka

Again, sorry about that. It was an artifact from a previous version. Fixing it in v3.

#5 lore raw

RFC PATCH v2 2/3 libbpf: Add FEAT_KPROBE_MULTI_LINK feature probe.

In reply to Varun R Mallya

Add FEAT_KPROBE_MULTI_LINK, similar to UPROBE_MULTI_LINK by loading and creating a link for a small BPF program with BPF_TRACE_KPROBE_MULTI as the expected attach type, and then checking the success of the operation.

---
 tools/lib/bpf/features.c        | 37 +++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 2 files changed, 39 insertions(+)
tools/lib/bpf/features.c +37 -0
index adcad221c601..13227c9ea69d 100644
@@ -424,6 +424,40 @@ static int probe_uprobe_multi_link(int token_fd)
 	return link_fd < 0 && err == -EINVAL;
 }
 
+static int probe_kprobe_multi_link(int token_fd)
+{
+	LIBBPF_OPTS(bpf_prog_load_opts, load_opts,
+		    .expected_attach_type = BPF_TRACE_KPROBE_MULTI,
+		    .token_fd = token_fd,
+		    .prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	LIBBPF_OPTS(bpf_link_create_opts, link_opts);
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	int prog_fd, link_fd, err;
+	const char *sym = "bpf_map_lookup_elem"; /* stable, always present */
+
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, NULL, "GPL",
+				insns, ARRAY_SIZE(insns), &load_opts);
+	if (prog_fd < 0)
+		return -errno;
+
+	/* attaching to a valid symbol should succeed */
+	link_opts.kprobe_multi.syms = &sym;
+	link_opts.kprobe_multi.cnt = 1;
+	link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_KPROBE_MULTI, &link_opts);
+	err = -errno;
+	if (link_fd >= 0)
+		close(link_fd);
+	close(prog_fd);
+	/* if kprobe_multi is supported, link creation either succeeds or
+	 * fails with something other than -EINVAL due to permissions,
+	 */
+	return link_fd >= 0 || (err != -EINVAL);
+}
+
 static int probe_kern_bpf_cookie(int token_fd)
 {
 	struct bpf_insn insns[] = {
@@ -658,6 +692,9 @@ static struct kern_feature_desc {
 	[FEAT_UPROBE_MULTI_LINK] = {
 		"BPF multi-uprobe link support", probe_uprobe_multi_link,
 	},
+	[FEAT_KPROBE_MULTI_LINK] = {
+		"BPF multi-kprobe link support", probe_kprobe_multi_link,
+	},
 	[FEAT_ARG_CTX_TAG] = {
 		"kernel-side __arg_ctx tag", probe_kern_arg_ctx_tag,
 	},
tools/lib/bpf/libbpf_internal.h +2 -0
index 4bcb6ca69bb1..633106a687c7 100644
@@ -388,6 +388,8 @@ enum kern_feature_id {
 	FEAT_SYSCALL_WRAPPER,
 	/* BPF multi-uprobe link support */
 	FEAT_UPROBE_MULTI_LINK,
+	/* BPF multi-kprobe link support */
+	FEAT_KPROBE_MULTI_LINK,
 	/* Kernel supports arg:ctx tag (__arg_ctx) for global subprogs natively */
 	FEAT_ARG_CTX_TAG,
 	/* Kernel supports '?' at the front of datasec names */
Signature · 1 line
2.52.0
#6 lore raw

Re: [RFC PATCH bpf-next v2 2/3] libbpf: Add FEAT_KPROBE_MULTI_LINK feature probe.

tools/lib/bpf/libbpf_internal.h enum kern_feature_id { · line 392

Quoted 10 lines · Varun R Mallya
tools/lib/bpf/libbpf_internal.h +2 -0
index 4bcb6ca69bb1..633106a687c7 100644
@@ -388,6 +388,8 @@ enum kern_feature_id {
 	FEAT_SYSCALL_WRAPPER,
 	/* BPF multi-uprobe link support */
 	FEAT_UPROBE_MULTI_LINK,
+	/* BPF multi-kprobe link support */
+	FEAT_KPROBE_MULTI_LINK,

Even though it seems good to put FEAT_KPROBE_MULTI_LINK here, better to put it at last to keep backwards compatibility?

When users compile application using old libbpf and run it against such new libbpf.so, will they get incorrect feature support?

Thanks,
Leon

Quoted 3 lines
tools/lib/bpf/libbpf_internal.h +0 -0
 	/* Kernel supports arg:ctx tag (__arg_ctx) for global subprogs natively */
 	FEAT_ARG_CTX_TAG,
 	/* Kernel supports '?' at the front of datasec names */
#7 lore raw

Re: [RFC PATCH bpf-next v2 2/3] libbpf: Add FEAT_KPROBE_MULTI_LINK feature probe.

Quoted 15 lines · Leon Hwang

tools/lib/bpf/libbpf_internal.h enum kern_feature_id { · line 392

Quoted 10 lines · Varun R Mallya
tools/lib/bpf/libbpf_internal.h +2 -0
index 4bcb6ca69bb1..633106a687c7 100644
@@ -388,6 +388,8 @@ enum kern_feature_id {
 	FEAT_SYSCALL_WRAPPER,
 	/* BPF multi-uprobe link support */
 	FEAT_UPROBE_MULTI_LINK,
+	/* BPF multi-kprobe link support */
+	FEAT_KPROBE_MULTI_LINK,

Even though it seems good to put FEAT_KPROBE_MULTI_LINK here, better to put it at last to keep backwards compatibility?

Quoted 5 lines

When users compile application using old libbpf and run it against such new libbpf.so, will they get incorrect feature support?

Thanks,
Leon

Changing in v3. Thanks!

Quoted 4 lines
Quoted 3 lines
tools/lib/bpf/libbpf_internal.h +0 -0
 	/* Kernel supports arg:ctx tag (__arg_ctx) for global subprogs natively */
 	FEAT_ARG_CTX_TAG,
 	/* Kernel supports '?' at the front of datasec names */
#8 lore raw

Re: [RFC PATCH bpf-next v2 2/3] libbpf: Add FEAT_KPROBE_MULTI_LINK feature probe.

In reply to Varun R Mallya
Quoted 32 lines · Varun R Mallya

Add FEAT_KPROBE_MULTI_LINK, similar to UPROBE_MULTI_LINK by loading and creating a link for a small BPF program with BPF_TRACE_KPROBE_MULTI as the expected attach type, and then checking the success of the operation.

---
 tools/lib/bpf/features.c        | 37 +++++++++++++++++++++++++++++++++
 tools/lib/bpf/libbpf_internal.h |  2 ++
 2 files changed, 39 insertions(+)
tools/lib/bpf/features.c +13 -0
index adcad221c601..13227c9ea69d 100644
@@ -424,6 +424,40 @@ static int probe_uprobe_multi_link(int token_fd)
 	return link_fd < 0 && err == -EINVAL;
 }
 
+static int probe_kprobe_multi_link(int token_fd)
+{
+	LIBBPF_OPTS(bpf_prog_load_opts, load_opts,
+		    .expected_attach_type = BPF_TRACE_KPROBE_MULTI,
+		    .token_fd = token_fd,
+		    .prog_flags = token_fd ? BPF_F_TOKEN_FD : 0,
+	);
+	LIBBPF_OPTS(bpf_link_create_opts, link_opts);
+	struct bpf_insn insns[] = {
+		BPF_MOV64_IMM(BPF_REG_0, 0),
+		BPF_EXIT_INSN(),
+	};
+	int prog_fd, link_fd, err;

tools/lib/bpf/features.c static int probe_uprobe_multi_link(int token_fd) · line 449

Quoted 10 lines
tools/lib/bpf/features.c +10 -0
+	const char *sym = "bpf_map_lookup_elem"; /* stable, always present */
+
+	prog_fd = bpf_prog_load(BPF_PROG_TYPE_KPROBE, NULL, "GPL",
+				insns, ARRAY_SIZE(insns), &load_opts);
+	if (prog_fd < 0)
+		return -errno;
+
+	/* attaching to a valid symbol should succeed */
+	link_opts.kprobe_multi.syms = &sym;
+	link_opts.kprobe_multi.cnt = 1;

maybe we could set cnt > MAX_KPROBE_MULTI_CNT and check if the fail is E2BIG ? this could save some cycles, but haven't tried that

jirka

Quoted 11 lines
tools/lib/bpf/features.c +11 -0
+	link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_KPROBE_MULTI, &link_opts);
+	err = -errno;
+	if (link_fd >= 0)
+		close(link_fd);
+	close(prog_fd);
+	/* if kprobe_multi is supported, link creation either succeeds or
+	 * fails with something other than -EINVAL due to permissions,
+	 */
+	return link_fd >= 0 || (err != -EINVAL);
+}
+

SNIP

#9 lore raw

Re: [RFC PATCH bpf-next v2 2/3] libbpf: Add FEAT_KPROBE_MULTI_LINK feature probe.

Quoted 10 lines · Jiri Olsa

tools/lib/bpf/features.c static int probe_uprobe_multi_link(int token_fd) · line 449

Quoted 4 lines
tools/lib/bpf/features.c +4 -0
+
+	/* attaching to a valid symbol should succeed */
+	link_opts.kprobe_multi.syms = &sym;
+	link_opts.kprobe_multi.cnt = 1;

maybe we could set cnt > MAX_KPROBE_MULTI_CNT and check if the fail is E2BIG ? this could save some cycles, but haven't tried that

jirka

Thanks for the suggestion!
I tested this out on an older kernel (<5.8) to see if it fails with a different error code and it did. On newer kernels it always fails with -E2BIG. Implemented this in my latest version (which I will send out in a while).

Quoted 13 lines
Quoted 11 lines
tools/lib/bpf/features.c +11 -0
+	link_fd = bpf_link_create(prog_fd, -1, BPF_TRACE_KPROBE_MULTI, &link_opts);
+	err = -errno;
+	if (link_fd >= 0)
+		close(link_fd);
+	close(prog_fd);
+	/* if kprobe_multi is supported, link creation either succeeds or
+	 * fails with something other than -EINVAL due to permissions,
+	 */
+	return link_fd >= 0 || (err != -EINVAL);
+}
+

SNIP

#10 lore raw

RFC PATCH v2 3/3 libbpf: Auto-upgrade kprobes to multi-kprobes when supported

In reply to Varun R Mallya

This patch modifies libbpf to automatically upgrade standard SEC("kprobe") and SEC("kretprobe") programs to use the multi-kprobe infrastructure (BPF_TRACE_KPROBE_MULTI) at load time if the kernel supports it, making them compatible with BPF tokens.

To maintain backward compatibility and handle cases where singular kprobes are required, new SEC("kprobe.single") and SEC("kretprobe.single") section types are introduced. These force libbpf to use the legacy perf_event_open() attachment path.

The following explain the reasoning for changing selftests:
- test_fill_link_info.c kprobe→kprobe.single:
this test calls bpf_link_get_info_by_fd and asserts BPF_LINK_TYPE_PERF_EVENT and it explicitly needs the perf_event attachment path, which breaks after auto-upgrade to multi.

- test_attach_probe_manual.c kprobe→kprobe.single, kretprobe→kretprobe.single:
this test exercises all four explicit attachment modes (default, legacy, perf, link) and PROBE_ATTACH_MODE_LINK creates a perf_event BPF link which the kernel rejects
for a prog loaded with expected_attach_type = BPF_TRACE_KPROBE_MULTI.

- missed_kprobe.c kprobe->kprobe.single:
The link is explicitly checked in the tests due to which it fails if we do not specifically kprobe.single this.

- get_func_ip_test.c ?kprobe→?kprobe.single: the ? is stripped from sec_name by libbpf at init time so the prog still gets auto-upgraded. It is then manually attached with a non-zero body offset, which kprobe_multi doesn't support.

---
 tools/lib/bpf/libbpf.c                        | 61 +++++++++++++++++--
 .../selftests/bpf/progs/get_func_ip_test.c    |  2 +-
 .../selftests/bpf/progs/missed_kprobe.c       |  4 +-
 .../bpf/progs/test_attach_probe_manual.c      |  4 +-
 .../selftests/bpf/progs/test_fill_link_info.c |  2 +-
 5 files changed, 61 insertions(+), 12 deletions(-)
tools/lib/bpf/libbpf.c +55 -6
index bd7b6f486430..9d0a36f8279a 100644
@@ -8265,6 +8265,24 @@ static int bpf_object_prepare_progs(struct bpf_object *obj)
 			}
 		}
 
+		if (kernel_supports(obj, FEAT_KPROBE_MULTI_LINK)) {
+			const char *sec_name = prog->sec_name;
+			/* Here, we filter out for k[ret]probe or "k[ret]probe/"
+			 * but we leave out anything with an '@'
+			 * in it as kprobe_multi does not support versioned
+			 * symbols, so we don't upgrade. Also for '+' as we do not
+			 * support offsets.
+			 */
+			if (((strncmp(sec_name, "kprobe", 6) == 0 &&
+			      (sec_name[6] == '/' || sec_name[6] == '\0')) ||
+			     (strncmp(sec_name, "kretprobe", 9) == 0 &&
+			      (sec_name[9] == '/' || sec_name[9] == '\0'))) &&
+			    !strchr(sec_name, '@') &&
+			    !strchr(sec_name, '+') &&
+			    !(prog->prog_flags & BPF_F_SLEEPABLE))
+				prog->expected_attach_type = BPF_TRACE_KPROBE_MULTI;
+		}
+
 		err = bpf_object__sanitize_prog(obj, prog);
 		if (err)
 			return err;
@@ -9924,10 +9942,12 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("sk_reuseport/migrate",	SK_REUSEPORT, BPF_SK_REUSEPORT_SELECT_OR_MIGRATE, SEC_ATTACHABLE),
 	SEC_DEF("sk_reuseport",		SK_REUSEPORT, BPF_SK_REUSEPORT_SELECT, SEC_ATTACHABLE),
 	SEC_DEF("kprobe+",		KPROBE,	0, SEC_NONE, attach_kprobe),
+	SEC_DEF("kprobe.single+",	KPROBE,	0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uprobe+",		KPROBE,	0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uprobe.s+",		KPROBE,	0, SEC_SLEEPABLE, attach_uprobe),
 	SEC_DEF("uprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),
 	SEC_DEF("kretprobe+",		KPROBE, 0, SEC_NONE, attach_kprobe),
+	SEC_DEF("kretprobe.single+",	KPROBE, 0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uretprobe+",		KPROBE, 0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uretprobe.s+",		KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
 	SEC_DEF("uretprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),
@@ -11769,6 +11789,25 @@ bpf_program__attach_kprobe_opts(const struct bpf_program *prog,
 	offset = OPTS_GET(opts, offset, 0);
 	pe_opts.bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
 
+	/* This provides backwards compatibility to programs using kprobe, but
+	 * have been auto-upgraded to multi kprobe.
+	 */
+	if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
+	    offset == 0 && attach_mode == PROBE_ATTACH_MODE_DEFAULT) {
+		LIBBPF_OPTS(bpf_kprobe_multi_opts, multi_opts);
+		const char *syms[1] = { func_name };
+		__u64 bpf_cookie;
+
+		multi_opts.retprobe = OPTS_GET(opts, retprobe, false);
+		multi_opts.syms = syms;
+		multi_opts.cnt = 1;
+		bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
+		if (bpf_cookie)
+			multi_opts.cookies = &bpf_cookie;
+
+		return bpf_program__attach_kprobe_multi_opts(prog, NULL, &multi_opts);
+	}
+
 	legacy = determine_kprobe_perf_type() < 0;
 	switch (attach_mode) {
 	case PROBE_ATTACH_MODE_LEGACY:
@@ -12223,14 +12262,24 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf
 	*link = NULL;
 
 	/* no auto-attach for SEC("kprobe") and SEC("kretprobe") */
-	if (strcmp(prog->sec_name, "kprobe") == 0 || strcmp(prog->sec_name, "kretprobe") == 0)
+	if (strcmp(prog->sec_name, "kprobe") == 0 ||
+	    strcmp(prog->sec_name, "kretprobe") == 0 ||
+	    strcmp(prog->sec_name, "kprobe.single") == 0 ||
+	    strcmp(prog->sec_name, "kretprobe.single") == 0)
 		return 0;
 
-	opts.retprobe = str_has_pfx(prog->sec_name, "kretprobe/");
-	if (opts.retprobe)
-		func_name = prog->sec_name + sizeof("kretprobe/") - 1;
-	else
-		func_name = prog->sec_name + sizeof("kprobe/") - 1;
+	if (str_has_pfx(prog->sec_name, "kretprobe/") ||
+	    str_has_pfx(prog->sec_name, "kretprobe.single/")) {
+		opts.retprobe = true;
+		func_name = str_has_pfx(prog->sec_name, "kretprobe/")
+			    ? prog->sec_name + sizeof("kretprobe/") - 1
+			    : prog->sec_name + sizeof("kretprobe.single/") - 1;
+	} else {
+		opts.retprobe = false;
+		func_name = str_has_pfx(prog->sec_name, "kprobe.single/")
+			    ? prog->sec_name + sizeof("kprobe.single/") - 1
+			    : prog->sec_name + sizeof("kprobe/") - 1;
+	}
 
 	n = sscanf(func_name, "%m[a-zA-Z0-9_.]+%li", &func, &offset);
 	if (n < 1) {
tools/testing/selftests/bpf/progs/get_func_ip_test.c +1 -1
index 2011cacdeb18..a039760b8516 100644
@@ -73,7 +73,7 @@ int BPF_PROG(test5, int a, int *b, int ret)
 }
 
 __u64 test6_result = 0;
-SEC("?kprobe")
+SEC("?kprobe.single")
 int test6(struct pt_regs *ctx)
 {
 	__u64 addr = bpf_get_func_ip(ctx);
tools/testing/selftests/bpf/progs/missed_kprobe.c +2 -2
index 51a4fe64c917..5f405888ed0b 100644
@@ -16,14 +16,14 @@ int BPF_PROG(trigger)
 	return 0;
 }
 
-SEC("kprobe/bpf_fentry_test1")
+SEC("kprobe.single/bpf_fentry_test1")
 int test1(struct pt_regs *ctx)
 {
 	bpf_kfunc_common_test();
 	return 0;
 }
 
-SEC("kprobe/bpf_kfunc_common_test")
+SEC("kprobe.single/bpf_kfunc_common_test")
 int test2(struct pt_regs *ctx)
 {
 	return 0;
tools/testing/selftests/bpf/progs/test_attach_probe_manual.c +2 -2
index 7f08bce94596..eb0fa3b39c5f 100644
@@ -14,14 +14,14 @@ int uretprobe_res = 0;
 int uprobe_byname_res = 0;
 void *user_ptr = 0;
 
-SEC("kprobe")
+SEC("kprobe.single")
 int handle_kprobe(struct pt_regs *ctx)
 {
 	kprobe_res = 1;
 	return 0;
 }
 
-SEC("kretprobe")
+SEC("kretprobe.single")
 int handle_kretprobe(struct pt_regs *ctx)
 {
 	kretprobe_res = 2;
tools/testing/selftests/bpf/progs/test_fill_link_info.c +1 -1
index 8e47a818462f..bc6cfed71fb8 100644
@@ -22,7 +22,7 @@ int unused(void)
 			CONFIG_PPC64 ? 0 : 1;
 }
 
-SEC("kprobe")
+SEC("kprobe.single")
 int BPF_PROG(kprobe_run)
 {
 	return 0;
Signature · 1 line
2.52.0
#11 lore raw

Re: [RFC PATCH bpf-next v2 3/3] libbpf: Auto-upgrade kprobes to multi-kprobes when supported

What happens when a program was auto-upgraded to BPF_TRACE_KPROBE_MULTI but the caller passes offset != 0 or attach_mode == PROBE_ATTACH_MODE_LINK?
Quoted 4 lines
tools/lib/bpf/libbpf.c +0 -0
index 056bc50a0bbb..7b5f83ce543e 100644

[ ... ]

Quoted 14 lines
tools/lib/bpf/libbpf.c +10 -0
@@ -11815,6 +11835,25 @@ bpf_program__attach_kprobe_opts(const struct bpf_program *prog,
 	offset = OPTS_GET(opts, offset, 0);
 	pe_opts.bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
 
+	/* This provides backwards compatibility to programs using kprobe, but
+	 * have been auto-upgraded to multi kprobe.
+	 */
+	if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
+	    offset == 0 && attach_mode == PROBE_ATTACH_MODE_DEFAULT) {
+		LIBBPF_OPTS(bpf_kprobe_multi_opts, multi_opts);
+		const char *syms[1] = { func_name };
+		__u64 bpf_cookie;
+
+		multi_opts.retprobe = OPTS_GET(opts, retprobe, false);

tools/lib/bpf/libbpf.c bpf_program__attach_kprobe_opts(const struct bpf_program *prog, · line 11857

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+		multi_opts.syms = syms;
+		multi_opts.cnt = 1;
+		bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
+		if (bpf_cookie)
+			multi_opts.cookies = &bpf_cookie;
+
+		return bpf_program__attach_kprobe_multi_opts(prog, NULL, &multi_opts);
+	}
+
+	legacy = determine_kprobe_perf_type() < 0;

What happens when a program was auto-upgraded to BPF_TRACE_KPROBE_MULTI but the caller passes offset != 0 or attach_mode == PROBE_ATTACH_MODE_LINK?

The redirect above is skipped, and the code falls through to the perf_event path, which eventually calls bpf_link_create() with BPF_PERF_EVENT. The kernel's bpf_prog_attach_check_attach_type() in link_create() rejects this:

kernel/bpf/syscall.c:bpf_prog_attach_check_attach_type() {
    case BPF_PROG_TYPE_KPROBE:
        if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
            attach_type != BPF_TRACE_KPROBE_MULTI)
            return -EINVAL;
}

This affects callers of the public API bpf_program__attach_kprobe_opts() who have a SEC("kprobe") program that was auto-upgraded but then manually attach with offset != 0 (the get_func_ip_test.c case) or with PROBE_ATTACH_MODE_LINK (the test_attach_probe_manual.c case). The selftests were updated to use .single, but external users of the API would get a confusing -EINVAL with no diagnostic.

Would it make sense to add a pr_warn here for the fallthrough case when expected_attach_type == BPF_TRACE_KPROBE_MULTI, suggesting SEC("kprobe.single") as a workaround? Something like:

    if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI) {
        pr_warn("prog '%s': kprobe was auto-upgraded to multi but "
                "offset or attach_mode is incompatible; "
                "use SEC(\"kprobe.single\") instead\n",
                prog->name);
        return libbpf_err_ptr(-ENOTSUP);
    }

[ ... ]

tools/lib/bpf/libbpf.c static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf · line 12315

Quoted 9 lines
tools/lib/bpf/libbpf.c +4 -0
@@ -12269,14 +12308,24 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf
 	*link = NULL;
 
 	/* no auto-attach for SEC("kprobe") and SEC("kretprobe") */
+	if (strcmp(prog->sec_name, "kprobe") == 0 ||
+	    strcmp(prog->sec_name, "kretprobe") == 0 ||
+	    strcmp(prog->sec_name, "kprobe.single") == 0 ||
+	    strcmp(prog->sec_name, "kretprobe.single") == 0)
 		return 0;

This isn't a bug, but the comment above still says SEC("kprobe") and SEC("kretprobe") -- should it also mention the .single variants now that they are handled here?

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/23741893093

#12 lore raw

Re: [RFC PATCH bpf-next v2 3/3] libbpf: Auto-upgrade kprobes to multi-kprobes when supported

Quoted 35 lines · On Mon, Mar 30, 2026 at 11:47:26AM +0000, [email protected] wrote:

tools/lib/bpf/libbpf.c bpf_program__attach_kprobe_opts(const struct bpf_program *prog,

Quoted 6 lines
tools/lib/bpf/libbpf.c +6 -0
+			multi_opts.cookies = &bpf_cookie;
+
+		return bpf_program__attach_kprobe_multi_opts(prog, NULL, &multi_opts);
+	}
+
+	legacy = determine_kprobe_perf_type() < 0;

What happens when a program was auto-upgraded to BPF_TRACE_KPROBE_MULTI but the caller passes offset != 0 or attach_mode == PROBE_ATTACH_MODE_LINK?

The redirect above is skipped, and the code falls through to the perf_event path, which eventually calls bpf_link_create() with BPF_PERF_EVENT. The kernel's bpf_prog_attach_check_attach_type() in link_create() rejects this:

kernel/bpf/syscall.c:bpf_prog_attach_check_attach_type() {
    case BPF_PROG_TYPE_KPROBE:
        if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
            attach_type != BPF_TRACE_KPROBE_MULTI)
            return -EINVAL;
}

This affects callers of the public API bpf_program__attach_kprobe_opts() who have a SEC("kprobe") program that was auto-upgraded but then manually attach with offset != 0 (the get_func_ip_test.c case) or with PROBE_ATTACH_MODE_LINK (the test_attach_probe_manual.c case). The selftests were updated to use .single, but external users of the API would get a confusing -EINVAL with no diagnostic.

Would it make sense to add a pr_warn here for the fallthrough case when expected_attach_type == BPF_TRACE_KPROBE_MULTI, suggesting SEC("kprobe.single") as a workaround? Something like:

Quoted 7 lines
    if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI) {
        pr_warn("prog '%s': kprobe was auto-upgraded to multi but "
                "offset or attach_mode is incompatible; "
                "use SEC(\"kprobe.single\") instead\n",
                prog->name);
        return libbpf_err_ptr(-ENOTSUP);
    }

Nice catch! I've added a check for that now.

Quoted 24 lines

[ ... ]

tools/lib/bpf/libbpf.c static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf · line 12315

Quoted 9 lines
tools/lib/bpf/libbpf.c +4 -0
@@ -12269,14 +12308,24 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf
 	*link = NULL;
 
 	/* no auto-attach for SEC("kprobe") and SEC("kretprobe") */
+	if (strcmp(prog->sec_name, "kprobe") == 0 ||
+	    strcmp(prog->sec_name, "kretprobe") == 0 ||
+	    strcmp(prog->sec_name, "kprobe.single") == 0 ||
+	    strcmp(prog->sec_name, "kretprobe.single") == 0)
 		return 0;

This isn't a bug, but the comment above still says SEC("kprobe") and SEC("kretprobe") -- should it also mention the .single variants now that they are handled here?

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/23741893093

#13 lore raw

Re: [RFC PATCH bpf-next v2 3/3] libbpf: Auto-upgrade kprobes to multi-kprobes when supported

In reply to Varun R Mallya
Quoted 46 lines · Varun R Mallya

This patch modifies libbpf to automatically upgrade standard SEC("kprobe") and SEC("kretprobe") programs to use the multi-kprobe infrastructure (BPF_TRACE_KPROBE_MULTI) at load time if the kernel supports it, making them compatible with BPF tokens.

To maintain backward compatibility and handle cases where singular kprobes are required, new SEC("kprobe.single") and SEC("kretprobe.single") section types are introduced. These force libbpf to use the legacy perf_event_open() attachment path.

The following explain the reasoning for changing selftests:
- test_fill_link_info.c kprobe→kprobe.single:
this test calls bpf_link_get_info_by_fd and asserts BPF_LINK_TYPE_PERF_EVENT and it explicitly needs the perf_event attachment path, which breaks after auto-upgrade to multi.

- test_attach_probe_manual.c kprobe→kprobe.single, kretprobe→kretprobe.single:
this test exercises all four explicit attachment modes (default, legacy, perf, link) and PROBE_ATTACH_MODE_LINK creates a perf_event BPF link which the kernel rejects
for a prog loaded with expected_attach_type = BPF_TRACE_KPROBE_MULTI.

- missed_kprobe.c kprobe->kprobe.single:
The link is explicitly checked in the tests due to which it fails if we do not specifically kprobe.single this.

- get_func_ip_test.c ?kprobe→?kprobe.single: the ? is stripped from sec_name by libbpf at init time so the prog still gets auto-upgraded. It is then manually attached with a non-zero body offset, which kprobe_multi doesn't support.

---
 tools/lib/bpf/libbpf.c                        | 61 +++++++++++++++++--
 .../selftests/bpf/progs/get_func_ip_test.c    |  2 +-
 .../selftests/bpf/progs/missed_kprobe.c       |  4 +-
 .../bpf/progs/test_attach_probe_manual.c      |  4 +-
 .../selftests/bpf/progs/test_fill_link_info.c |  2 +-
 5 files changed, 61 insertions(+), 12 deletions(-)
tools/lib/bpf/libbpf.c +0 -0
index bd7b6f486430..9d0a36f8279a 100644

tools/lib/bpf/libbpf.c static int bpf_object_prepare_progs(struct bpf_object *obj) · line 8273

Quoted 9 lines
tools/lib/bpf/libbpf.c +6 -0
@@ -8265,6 +8265,24 @@ static int bpf_object_prepare_progs(struct bpf_object *obj)
 			}
 		}
 
+		if (kernel_supports(obj, FEAT_KPROBE_MULTI_LINK)) {
+			const char *sec_name = prog->sec_name;
+			/* Here, we filter out for k[ret]probe or "k[ret]probe/"
+			 * but we leave out anything with an '@'
+			 * in it as kprobe_multi does not support versioned
+			 * symbols, so we don't upgrade. Also for '+' as we do not

hum, kprobe versioned symbols?

tools/lib/bpf/libbpf.c static int bpf_object_prepare_progs(struct bpf_object *obj) · line 8276

Quoted 3 lines
tools/lib/bpf/libbpf.c +3 -0
+			 * support offsets.
+			 */
+			if (((strncmp(sec_name, "kprobe", 6) == 0 &&

str_has_pfx ?

tools/lib/bpf/libbpf.c static int bpf_object_prepare_progs(struct bpf_object *obj) · line 8282

Quoted 6 lines
tools/lib/bpf/libbpf.c +6 -0
+			      (sec_name[6] == '/' || sec_name[6] == '\0')) ||
+			     (strncmp(sec_name, "kretprobe", 9) == 0 &&
+			      (sec_name[9] == '/' || sec_name[9] == '\0'))) &&
+			    !strchr(sec_name, '@') &&
+			    !strchr(sec_name, '+') &&
+			    !(prog->prog_flags & BPF_F_SLEEPABLE))

is this check necessary?

tools/lib/bpf/libbpf.c static int bpf_object_prepare_progs(struct bpf_object *obj) · line 8285

Quoted 3 lines
tools/lib/bpf/libbpf.c +3 -0
+				prog->expected_attach_type = BPF_TRACE_KPROBE_MULTI;
+		}
+

maybe add the upgrade logic into separate function, like

static int upgrade_program(struct bpf_program *prog)

Quoted 21 lines
tools/lib/bpf/libbpf.c +3 -0
 		err = bpf_object__sanitize_prog(obj, prog);
 		if (err)
 			return err;
@@ -9924,10 +9942,12 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("sk_reuseport/migrate",	SK_REUSEPORT, BPF_SK_REUSEPORT_SELECT_OR_MIGRATE, SEC_ATTACHABLE),
 	SEC_DEF("sk_reuseport",		SK_REUSEPORT, BPF_SK_REUSEPORT_SELECT, SEC_ATTACHABLE),
 	SEC_DEF("kprobe+",		KPROBE,	0, SEC_NONE, attach_kprobe),
+	SEC_DEF("kprobe.single+",	KPROBE,	0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uprobe+",		KPROBE,	0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uprobe.s+",		KPROBE,	0, SEC_SLEEPABLE, attach_uprobe),
 	SEC_DEF("uprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),
 	SEC_DEF("kretprobe+",		KPROBE, 0, SEC_NONE, attach_kprobe),
+	SEC_DEF("kretprobe.single+",	KPROBE, 0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uretprobe+",		KPROBE, 0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uretprobe.s+",		KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
 	SEC_DEF("uretprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),
@@ -11769,6 +11789,25 @@ bpf_program__attach_kprobe_opts(const struct bpf_program *prog,
 	offset = OPTS_GET(opts, offset, 0);
 	pe_opts.bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
 
+	/* This provides backwards compatibility to programs using kprobe, but

tools/lib/bpf/libbpf.c bpf_program__attach_kprobe_opts(const struct bpf_program *prog, · line 11802

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+	 * have been auto-upgraded to multi kprobe.
+	 */
+	if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
+	    offset == 0 && attach_mode == PROBE_ATTACH_MODE_DEFAULT) {
+		LIBBPF_OPTS(bpf_kprobe_multi_opts, multi_opts);
+		const char *syms[1] = { func_name };
+		__u64 bpf_cookie;
+
+		multi_opts.retprobe = OPTS_GET(opts, retprobe, false);
+		multi_opts.syms = syms;

could we do directly:

		multi_opts.syms = &func_name;

jirka

Quoted 14 lines
tools/lib/bpf/libbpf.c +8 -0
+		multi_opts.cnt = 1;
+		bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
+		if (bpf_cookie)
+			multi_opts.cookies = &bpf_cookie;
+
+		return bpf_program__attach_kprobe_multi_opts(prog, NULL, &multi_opts);
+	}
+
 	legacy = determine_kprobe_perf_type() < 0;
 	switch (attach_mode) {
 	case PROBE_ATTACH_MODE_LEGACY:
@@ -12223,14 +12262,24 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf
 	*link = NULL;
 

SNIP

#14 lore raw

Re: [RFC PATCH bpf-next v2 3/3] libbpf: Auto-upgrade kprobes to multi-kprobes when supported

Quoted 8 lines · Jiri Olsa

tools/lib/bpf/libbpf.c static int bpf_object_prepare_progs(struct bpf_object *obj) · line 8273

Quoted 5 lines · Varun R Mallya
tools/lib/bpf/libbpf.c +5 -0
+			const char *sec_name = prog->sec_name;
+			/* Here, we filter out for k[ret]probe or "k[ret]probe/"
+			 * but we leave out anything with an '@'
+			 * in it as kprobe_multi does not support versioned
+			 * symbols, so we don't upgrade. Also for '+' as we do not

hum, kprobe versioned symbols?

Thanks for catching that! Removed.

Quoted 6 lines

tools/lib/bpf/libbpf.c static int bpf_object_prepare_progs(struct bpf_object *obj) · line 8282

Quoted 4 lines
tools/lib/bpf/libbpf.c +4 -0
+			      (sec_name[9] == '/' || sec_name[9] == '\0'))) &&
+			    !strchr(sec_name, '@') &&
+			    !strchr( sec_name, '+') &&
+			    !(prog->prog_flags & BPF_F_SLEEPABLE))

is this check necessary?

I had to add that check because selftests/bpf/prog_tests/attach_probe.c was failing. Sleepable kprobes are not supposed to attach successfully, but since this was upgraded to multi, it was doing so. So, I had to stop all sleepable kprobes from being upgraded to maintain compatibility.

I'd like to know if kprobe_multi is even allowed to be sleepable. I could not really find any selftests or patches for sleepable kprobe_multi functionality anywhere, so I just want to check if this is not actually intended behaviour and we are missing a check somewhere.

Quoted 7 lines

tools/lib/bpf/libbpf.c static int bpf_object_prepare_progs(struct bpf_object *obj) · line 8285

Quoted 3 lines
tools/lib/bpf/libbpf.c +3 -0
+				prog->expected_attach_type = BPF_TRACE_KPROBE_MULTI;
+		}
+

maybe add the upgrade logic into separate function, like

static int upgrade_program(struct bpf_program *prog)

I am thinking of making the return type void though.
I see that there is int everywhere else, but it does not make sense to me to return an int here since I'm not doing any operation in here that could return an error. Should I keep it as int or make it void ??

Quoted 35 lines
Quoted 22 lines
tools/lib/bpf/libbpf.c +3 -0
 		err = bpf_object__sanitize_prog(obj, prog);
 		if (err)
 			return err;
@@ -9924,10 +9942,12 @@ static const struct bpf_sec_def section_defs[] = {
 	SEC_DEF("sk_reuseport/migrate",	SK_REUSEPORT, BPF_SK_REUSEPORT_SELECT_OR_MIGRATE, SEC_ATTACHABLE),
 	SEC_DEF("sk_reuseport",		SK_REUSEPORT, BPF_SK_REUSEPORT_SELECT, SEC_ATTACHABLE),
 	SEC_DEF("kprobe+",		KPROBE,	0, SEC_NONE, attach_kprobe),
+	SEC_DEF("kprobe.single+",	KPROBE,	0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uprobe+",		KPROBE,	0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uprobe.s+",		KPROBE,	0, SEC_SLEEPABLE, attach_uprobe),
 	SEC_DEF("uprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),
 	SEC_DEF("kretprobe+",		KPROBE, 0, SEC_NONE, attach_kprobe),
+	SEC_DEF("kretprobe.single+",	KPROBE, 0, SEC_NONE, attach_kprobe),
 	SEC_DEF("uretprobe+",		KPROBE, 0, SEC_NONE, attach_uprobe),
 	SEC_DEF("uretprobe.s+",		KPROBE, 0, SEC_SLEEPABLE, attach_uprobe),
 	SEC_DEF("uretprobe.single+",	KPROBE,	0, SEC_NONE, attach_uprobe),
@@ -11769,6 +11789,25 @@ bpf_program__attach_kprobe_opts(const struct bpf_program *prog,
 	offset = OPTS_GET(opts, offset, 0);
 	pe_opts.bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
 
+	/* This provides backwards compatibility to programs using kprobe, but

tools/lib/bpf/libbpf.c bpf_program__attach_kprobe_opts(const struct bpf_program *prog, · line 11802

Quoted 10 lines
tools/lib/bpf/libbpf.c +10 -0
+	 * have been auto-upgraded to multi kprobe.
+	 */
+	if (prog->expected_attach_type == BPF_TRACE_KPROBE_MULTI &&
+	    offset == 0 && attach_mode == PROBE_ATTACH_MODE_DEFAULT) {
+		LIBBPF_OPTS(bpf_kprobe_multi_opts, multi_opts);
+		const char *syms[1] = { func_name };
+		__u64 bpf_cookie;
+
+		multi_opts.retprobe = OPTS_GET(opts, retprobe, false);
+		multi_opts.syms = syms;

could we do directly:

Quoted 3 lines
		multi_opts.syms = &func_name;

jirka

Implemented.

#15 lore raw

Re: [RFC PATCH bpf-next v2 3/3] libbpf: Auto-upgrade kprobes to multi-kprobes when supported

Quoted 10 lines · Varun R Mallya

I had to add that check because selftests/bpf/prog_tests/attach_probe.c was failing. Sleepable kprobes are not supposed to attach successfully, but since this was upgraded to multi, it was doing so. So, I had to stop all sleepable kprobes from being upgraded to maintain compatibility.

I'd like to know if kprobe_multi is even allowed to be sleepable. I could not really find any selftests or patches for sleepable kprobe_multi functionality anywhere, so I just want to check if this is not actually intended behaviour and we are missing a check somewhere.

kernel/trace/bpf_trace.c +4 -0
index 0b040a417442..af7079aa0f36 100644
@@ -2752,6 +2752,10 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
        if (!is_kprobe_multi(prog))
                return -EINVAL;
 
+       /* kprobe_multi is not allowed to be sleepable. */
+       if (prog->sleepable)
+               return -EINVAL;
+
        /* Writing to context is not allowed for kprobes. */
        if (prog->aux->kprobe_write_ctx)
                return -EINVAL;

Adding this removes the need for me to add the sleepable check. Should I send a patch with this or am I making a mistake here ?? Sleepability for kprobe_multi was never checked (If it's meant to not be sleepable, that is.)