Skip to contentSkip to conversation

libbpf: Allow kprobe attach using legacy debugfs interface

PATCH In review Hengqi Chen 3 participants 8 messages Mar 26, 2022 – Mar 31, 2022

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 libbpf: Allow kprobe attach using legacy debugfs interface

On some old kernels, kprobe auto-attach may fail when attach to symbols like udp_send_skb.isra.52 . This is because the kernel has kprobe PMU but don't allow attach to a symbol with '.' ([0]). Add a new option to bpf_kprobe_opts to allow using the legacy kprobe attach directly. This way, users can use bpf_program__attach_kprobe_opts in a dedicated custom sec handler to handle such case.

[0]: https://github.com/torvalds/linux/blob/v4.18/kernel/trace/trace_kprobe.c#L340-L343

---
 tools/lib/bpf/libbpf.c | 9 ++++++++-
 tools/lib/bpf/libbpf.h | 4 +++-
 2 files changed, 11 insertions(+), 2 deletions(-)
tools/lib/bpf/libbpf.c +8 -1
index 809fe209cdcc..9a294bb84bad 100644
@@ -10097,9 +10097,16 @@ static void gen_kprobe_legacy_event_name(char *buf, size_t buf_sz,
 					 const char *kfunc_name, size_t offset)
 {
 	static int index = 0;
+	int i;
 
 	snprintf(buf, buf_sz, "libbpf_%u_%s_0x%zx_%d", getpid(), kfunc_name, offset,
 		 __sync_fetch_and_add(&index, 1));
+
+	/* sanitize .isra.$n symbols */
+	for (i = 0; buf[i]; i++) {
+		if (!isalnum(buf[i]))
+			buf[i] = '_';
+	}
 }
 
 static int add_kprobe_event_legacy(const char *probe_name, bool retprobe,
@@ -10189,7 +10196,7 @@ bpf_program__attach_kprobe_opts(const struct bpf_program *prog,
 	offset = OPTS_GET(opts, offset, 0);
 	pe_opts.bpf_cookie = OPTS_GET(opts, bpf_cookie, 0);
 
-	legacy = determine_kprobe_perf_type() < 0;
+	legacy = OPTS_GET(opts, legacy, false) || determine_kprobe_perf_type() < 0;
 	if (!legacy) {
 		pfd = perf_event_open_probe(false /* uprobe */, retprobe,
 					    func_name, offset,
tools/lib/bpf/libbpf.h +3 -1
index 05dde85e19a6..88a3624e3f15 100644
@@ -413,9 +413,11 @@ struct bpf_kprobe_opts {
 	size_t offset;
 	/* kprobe is return probe */
 	bool retprobe;
+	/* use the legacy debugfs interface */
+	bool legacy;
 	size_t :0;
 };
-#define bpf_kprobe_opts__last_field retprobe
+#define bpf_kprobe_opts__last_field legacy
 
 LIBBPF_API struct bpf_link *
 bpf_program__attach_kprobe(const struct bpf_program *prog, bool retprobe,

--
2.30.2

#1 lore raw
Quoted 8 lines · Hengqi Chen

On some old kernels, kprobe auto-attach may fail when attach to symbols like udp_send_skb.isra.52 . This is because the kernel has kprobe PMU but don't allow attach to a symbol with '.' ([0]). Add a new option to bpf_kprobe_opts to allow using the legacy kprobe attach directly. This way, users can use bpf_program__attach_kprobe_opts in a dedicated custom sec handler to handle such case.

Quoted 4 lines

[0]: https://github.com/torvalds/linux/blob/v4.18/kernel/trace/trace_kprobe.c#L340-L343

---

It's sad, but it makes sense. But, let's have a selftests that validates uses legacy option explicitly (e.g., in prog_tests/attach_probe.c). Also, let's fix this limitation in the kernel? It makes no sense to limit attaching to a proper kallsym symbol.

Quoted 4 lines
 tools/lib/bpf/libbpf.c | 9 ++++++++-
 tools/lib/bpf/libbpf.h | 4 +++-
 2 files changed, 11 insertions(+), 2 deletions(-)

[...]

#2 lore raw

Hello, Andrii

Quoted 9 lines · Andrii Nakryiko
Quoted 8 lines · Hengqi Chen

On some old kernels, kprobe auto-attach may fail when attach to symbols like udp_send_skb.isra.52 . This is because the kernel has kprobe PMU but don't allow attach to a symbol with '.' ([0]). Add a new option to bpf_kprobe_opts to allow using the legacy kprobe attach directly. This way, users can use bpf_program__attach_kprobe_opts in a dedicated custom sec handler to handle such case.

Quoted 8 lines
Quoted 4 lines

[0]: https://github.com/torvalds/linux/blob/v4.18/kernel/trace/trace_kprobe.c#L340-L343

---

It's sad, but it makes sense. But, let's have a selftests that validates uses legacy option explicitly (e.g., in prog_tests/attach_probe.c). Also, let's fix this limitation in the

OK, will add a selftest to exercise the new option.

Quoted 2 lines

kernel? It makes no sense to limit attaching to a proper kallsym symbol.

This limitation is lifted in newer kernel. Kernel v5.4 don't have this issue.

Quoted 7 lines
Quoted 5 lines
 tools/lib/bpf/libbpf.c | 9 ++++++++-
 tools/lib/bpf/libbpf.h | 4 +++-
 2 files changed, 11 insertions(+), 2 deletions(-)

[...]

--
Hengqi

#3 lore raw
Quoted 22 lines · Hengqi Chen

Hello, Andrii

Quoted 9 lines · Andrii Nakryiko
Quoted 8 lines · Hengqi Chen

On some old kernels, kprobe auto-attach may fail when attach to symbols like udp_send_skb.isra.52 . This is because the kernel has kprobe PMU but don't allow attach to a symbol with '.' ([0]). Add a new option to bpf_kprobe_opts to allow using the legacy kprobe attach directly. This way, users can use bpf_program__attach_kprobe_opts in a dedicated custom sec handler to handle such case.

Quoted 8 lines
Quoted 4 lines

[0]: https://github.com/torvalds/linux/blob/v4.18/kernel/trace/trace_kprobe.c#L340-L343

---

It's sad, but it makes sense. But, let's have a selftests that validates uses legacy option explicitly (e.g., in prog_tests/attach_probe.c). Also, let's fix this limitation in the

Quoted 6 lines

OK, will add a selftest to exercise the new option.

Quoted 2 lines

kernel? It makes no sense to limit attaching to a proper kallsym symbol.

This limitation is lifted in newer kernel. Kernel v5.4 don't have this issue.

Oh, ok. So how about another plan of attack then: if kprobe target function has '.' *and* we are on the kernel that doesn't support that, switch to legacy kprobe automatically? No need for a new option, libbpf handles this transparently.

Still need a test for kprobe with '.' in it, though not sure how reliable that will be... We can use kallsyms cache to check if expected xxx.isra.0 (or whatever) is present, and if not - skip subtest?

Quoted 11 lines
Quoted 8 lines
Quoted 6 lines

 tools/lib/bpf/libbpf.c | 9 ++++++++-
 tools/lib/bpf/libbpf.h | 4 +++-
 2 files changed, 11 insertions(+), 2 deletions(-)

[...]

--
Hengqi

#4 lore raw
Quoted 28 lines · Andrii Nakryiko
Quoted 27 lines · Hengqi Chen

Hello, Andrii

Quoted 9 lines · Andrii Nakryiko
Quoted 8 lines · Hengqi Chen

On some old kernels, kprobe auto-attach may fail when attach to symbols like udp_send_skb.isra.52 . This is because the kernel has kprobe PMU but don't allow attach to a symbol with '.' ([0]). Add a new option to bpf_kprobe_opts to allow using the legacy kprobe attach directly. This way, users can use bpf_program__attach_kprobe_opts in a dedicated custom sec handler to handle such case.

Quoted 8 lines
Quoted 4 lines

[0]: https://github.com/torvalds/linux/blob/v4.18/kernel/trace/trace_kprobe.c#L340-L343

---

It's sad, but it makes sense. But, let's have a selftests that validates uses legacy option explicitly (e.g., in prog_tests/attach_probe.c). Also, let's fix this limitation in the

OK, will add a selftest to exercise the new option.

Quoted 2 lines

kernel? It makes no sense to limit attaching to a proper kallsym symbol.

Quoted 7 lines
Quoted 1 line

This limitation is lifted in newer kernel. Kernel v5.4 don't have this issue.

Oh, ok. So how about another plan of attack then: if kprobe target function has '.' *and* we are on the kernel that doesn't support that, switch to legacy kprobe automatically? No need for a new option, libbpf handles this transparently.

That's better, and also eliminate the need for custom SEC() handler.

Quoted 5 lines

Still need a test for kprobe with '.' in it, though not sure how reliable that will be... We can use kallsyms cache to check if expected xxx.isra.0 (or whatever) is present, and if not - skip subtest?

Not sure how to do that. Even if such symbol exists, how to reliably trigger it is another problem.

Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
Quoted 6 lines

 tools/lib/bpf/libbpf.c | 9 ++++++++-
 tools/lib/bpf/libbpf.h | 4 +++-
 2 files changed, 11 insertions(+), 2 deletions(-)

[...]

--
Hengqi

#5 lore raw
Quoted 41 lines · Hengqi Chen
Quoted 28 lines · Andrii Nakryiko
Quoted 27 lines · Hengqi Chen

Hello, Andrii

Quoted 9 lines · Andrii Nakryiko
Quoted 8 lines · Hengqi Chen

On some old kernels, kprobe auto-attach may fail when attach to symbols like udp_send_skb.isra.52 . This is because the kernel has kprobe PMU but don't allow attach to a symbol with '.' ([0]). Add a new option to bpf_kprobe_opts to allow using the legacy kprobe attach directly. This way, users can use bpf_program__attach_kprobe_opts in a dedicated custom sec handler to handle such case.

Quoted 8 lines
Quoted 4 lines

[0]: https://github.com/torvalds/linux/blob/v4.18/kernel/trace/trace_kprobe.c#L340-L343

---

It's sad, but it makes sense. But, let's have a selftests that validates uses legacy option explicitly (e.g., in prog_tests/attach_probe.c). Also, let's fix this limitation in the

OK, will add a selftest to exercise the new option.

Quoted 2 lines

kernel? It makes no sense to limit attaching to a proper kallsym symbol.

Quoted 7 lines
Quoted 1 line

This limitation is lifted in newer kernel. Kernel v5.4 don't have this issue.

Oh, ok. So how about another plan of attack then: if kprobe target function has '.' *and* we are on the kernel that doesn't support that, switch to legacy kprobe automatically? No need for a new option, libbpf handles this transparently.

That's better, and also eliminate the need for custom SEC() handler.

Quoted 9 lines
Quoted 5 lines

Still need a test for kprobe with '.' in it, though not sure how reliable that will be... We can use kallsyms cache to check if expected xxx.isra.0 (or whatever) is present, and if not - skip subtest?

Not sure how to do that. Even if such symbol exists, how to reliably trigger it is another problem.

could we add a function to bpf testmod that is easily triggered and likely to be .isra-ed maybe?

Experimenting, the following function becomes .isra-ed at when compiled with -fipa-sra:

(quoted diff) +0 -0

b/tools/testing/selftests/bpf/bpf
index e585e1c..bb51e21 100644

tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +0 -0
@@ -88,6 +88,17 @@ __weak noinline struct file *bpf_testmod_return_ptr(int 

arg)

        }

}

+struct testisra {

+       int val1;
+       int val2;
+       int val3;

+};
+
+static noinline void bpf_testmod_test_isra(struct testisra *t, int val1, int val2)
+{
+ t->val3 = val1 + val2;
+}
+
noinline ssize_t
bpf_testmod_test_read(struct file *file, struct kobject *kobj,
struct bin_attribute *bin_attr,
@@ -98,8 +109,14 @@ __weak noinline struct file *bpf_testmod_return_ptr(int arg)

                .off = off,
                .len = len,
        };
+       struct testisra t = {
+               .val1 = off,
+               .val2 = len
+       };
        int i = 1;

+ bpf_testmod_test_isra(&t, t.val1, t.val2);
+

        while (bpf_testmod_return_ptr(i))
                i++;

Tested on gcc 9; possibly different results on different versions..

Alan

#6 lore raw

Hello, Alan

Quoted 101 lines · Alan Maguire
Quoted 41 lines · Hengqi Chen
Quoted 28 lines · Andrii Nakryiko
Quoted 27 lines · Hengqi Chen

Hello, Andrii

Quoted 9 lines · Andrii Nakryiko
Quoted 8 lines · Hengqi Chen

On some old kernels, kprobe auto-attach may fail when attach to symbols like udp_send_skb.isra.52 . This is because the kernel has kprobe PMU but don't allow attach to a symbol with '.' ([0]). Add a new option to bpf_kprobe_opts to allow using the legacy kprobe attach directly. This way, users can use bpf_program__attach_kprobe_opts in a dedicated custom sec handler to handle such case.

Quoted 8 lines
Quoted 4 lines

[0]: https://github.com/torvalds/linux/blob/v4.18/kernel/trace/trace_kprobe.c#L340-L343

---

It's sad, but it makes sense. But, let's have a selftests that validates uses legacy option explicitly (e.g., in prog_tests/attach_probe.c). Also, let's fix this limitation in the

OK, will add a selftest to exercise the new option.

Quoted 2 lines

kernel? It makes no sense to limit attaching to a proper kallsym symbol.

Quoted 7 lines
Quoted 1 line

This limitation is lifted in newer kernel. Kernel v5.4 don't have this issue.

Oh, ok. So how about another plan of attack then: if kprobe target function has '.' *and* we are on the kernel that doesn't support that, switch to legacy kprobe automatically? No need for a new option, libbpf handles this transparently.

That's better, and also eliminate the need for custom SEC() handler.

Quoted 9 lines
Quoted 5 lines

Still need a test for kprobe with '.' in it, though not sure how reliable that will be... We can use kallsyms cache to check if expected xxx.isra.0 (or whatever) is present, and if not - skip subtest?

Not sure how to do that. Even if such symbol exists, how to reliably trigger it is another problem.

could we add a function to bpf testmod that is easily triggered and likely to be .isra-ed maybe?

Experimenting, the following function becomes .isra-ed at when compiled with -fipa-sra:

tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +11 -0
index e585e1c..bb51e21 100644
@@ -88,6 +88,17 @@ __weak noinline struct file *bpf_testmod_return_ptr(int 
 arg)
        }
 }
 
+struct testisra {
+       int val1;
+       int val2;
+       int val3;
+};
+
+static noinline void bpf_testmod_test_isra(struct testisra *t, int val1, 
 int val2)
+{
+       t->val3 = val1 + val2;
+}
+
 noinline ssize_t

bpf_testmod_test_read(struct file *file, struct kobject *kobj,
struct bin_attribute *bin_attr,

tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c +6 -0
@@ -98,8 +109,14 @@ __weak noinline struct file 
 *bpf_testmod_return_ptr(int arg)
                .off = off,
                .len = len,
        };
+       struct testisra t = {
+               .val1 = off,
+               .val2 = len
+       };
        int i = 1;
 
+       bpf_testmod_test_isra(&t, t.val1, t.val2);
+
        while (bpf_testmod_return_ptr(i))
                i++;
Quoted 3 lines

Tested on gcc 9; possibly different results on different versions..

Alan

Thanks for the pointer, will give it a try.

Hengqi

#7 lore raw
In reply to Hengqi Chen
Quoted 42 lines · Hengqi Chen
Quoted 28 lines · Andrii Nakryiko
Quoted 27 lines · Hengqi Chen

Hello, Andrii

Quoted 9 lines · Andrii Nakryiko
Quoted 8 lines · Hengqi Chen

On some old kernels, kprobe auto-attach may fail when attach to symbols like udp_send_skb.isra.52 . This is because the kernel has kprobe PMU but don't allow attach to a symbol with '.' ([0]). Add a new option to bpf_kprobe_opts to allow using the legacy kprobe attach directly. This way, users can use bpf_program__attach_kprobe_opts in a dedicated custom sec handler to handle such case.

Quoted 8 lines
Quoted 4 lines

[0]: https://github.com/torvalds/linux/blob/v4.18/kernel/trace/trace_kprobe.c#L340-L343

---

It's sad, but it makes sense. But, let's have a selftests that validates uses legacy option explicitly (e.g., in prog_tests/attach_probe.c). Also, let's fix this limitation in the

OK, will add a selftest to exercise the new option.

Quoted 2 lines

kernel? It makes no sense to limit attaching to a proper kallsym symbol.

Quoted 7 lines
Quoted 1 line

This limitation is lifted in newer kernel. Kernel v5.4 don't have this issue.

Oh, ok. So how about another plan of attack then: if kprobe target function has '.' *and* we are on the kernel that doesn't support that, switch to legacy kprobe automatically? No need for a new option, libbpf handles this transparently.

That's better, and also eliminate the need for custom SEC() handler.

Quoted 8 lines
Quoted 5 lines

Still need a test for kprobe with '.' in it, though not sure how reliable that will be... We can use kallsyms cache to check if expected xxx.isra.0 (or whatever) is present, and if not - skip subtest?

Not sure how to do that. Even if such symbol exists, how to reliably trigger it is another problem.

In addition to what Alan proposed, which relies on compiler to do this whole isra thingy. I wonder if we can just create an alias symbol with dots in its name? I haven't tried, but would be curious to see if that works in bpf_testmod.

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 9 lines
Quoted 7 lines


 tools/lib/bpf/libbpf.c | 9 ++++++++-
 tools/lib/bpf/libbpf.h | 4 +++-
 2 files changed, 11 insertions(+), 2 deletions(-)

[...]

--
Hengqi