Skip to contentSkip to conversation

tracepoint: Add tracepoint_probe_register_may_exist() for BPF tracing

PATCH In review Steven Rostedt 4 participants 11 messages Jun 29, 2021 – Jul 8, 2021

Subsystems (6)
lore
Share & tools
lore

Steven Rostedt

author maintainer
#0 lore raw

PATCH 1/1 tracepoint: Add tracepoint_probe_register_may_exist() for BPF tracing

From: "Steven Rostedt (VMware)" <[email protected]>

All internal use cases for tracepoint_probe_register() is set to not ever be called with the same function and data. If it is, it is considered a bug, as that means the accounting of handling tracepoints is corrupted. If the function and data for a tracepoint is already registered when tracepoint_probe_register() is called, it will call WARN_ON_ONCE() and return with EEXISTS.

The BPF system call can end up calling tracepoint_probe_register() with the same data, which now means that this can trigger the warning because of a user space process. As WARN_ON_ONCE() should not be called because user space called a system call with bad data, there needs to be a way to register a tracepoint without triggering a warning.

Enter tracepoint_probe_register_may_exist(), which can be called, but will not cause a WARN_ON() if the probe already exists. It will still error out with EEXIST, which will then be sent to the user space that performed the BPF system call.

This keeps the previous testing for issues with other users of the tracepoint code, while letting BPF call it with duplicated data and not warn about it.

Fixes c4f6699dfcb85 ("bpf: introduce BPF_RAW_TRACEPOINT")

---

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

 include/linux/tracepoint.h | 10 ++++++++++
 kernel/trace/bpf_trace.c   |  3 ++-
 kernel/tracepoint.c        | 33 ++++++++++++++++++++++++++++++---
 3 files changed, 42 insertions(+), 4 deletions(-)
include/linux/tracepoint.h +10 -0
index 13f65420f188..ab58696d0ddd 100644
@@ -41,7 +41,17 @@ extern int
 tracepoint_probe_register_prio(struct tracepoint *tp, void *probe, void *data,
 			       int prio);
 extern int
+tracepoint_probe_register_prio_may_exist(struct tracepoint *tp, void *probe, void *data,
+					 int prio);
+extern int
 tracepoint_probe_unregister(struct tracepoint *tp, void *probe, void *data);
+static inline int
+tracepoint_probe_register_may_exist(struct tracepoint *tp, void *probe,
+				    void *data)
+{
+	return tracepoint_probe_register_prio_may_exist(tp, probe, data,
+							TRACEPOINT_DEFAULT_PRIO);
+}
 extern void
 for_each_kernel_tracepoint(void (*fct)(struct tracepoint *tp, void *priv),
 		void *priv);
kernel/trace/bpf_trace.c +2 -1
index 7a52bc172841..f0568b3d6bd1 100644
@@ -1840,7 +1840,8 @@ static int __bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *
 	if (prog->aux->max_tp_access > btp->writable_size)
 		return -EINVAL;
 
-	return tracepoint_probe_register(tp, (void *)btp->bpf_func, prog);
+	return tracepoint_probe_register_may_exist(tp, (void *)btp->bpf_func,
+						   prog);
 }
 
 int bpf_probe_register(struct bpf_raw_event_map *btp, struct bpf_prog *prog)
kernel/tracepoint.c +30 -3
index 9f478d29b926..976bf8ce8039 100644
@@ -273,7 +273,8 @@ static void tracepoint_update_call(struct tracepoint *tp, struct tracepoint_func
  * Add the probe function to a tracepoint.
  */
 static int tracepoint_add_func(struct tracepoint *tp,
-			       struct tracepoint_func *func, int prio)
+			       struct tracepoint_func *func, int prio,
+			       bool warn)
 {
 	struct tracepoint_func *old, *tp_funcs;
 	int ret;
@@ -288,7 +289,7 @@ static int tracepoint_add_func(struct tracepoint *tp,
 			lockdep_is_held(&tracepoints_mutex));
 	old = func_add(&tp_funcs, func, prio);
 	if (IS_ERR(old)) {
-		WARN_ON_ONCE(PTR_ERR(old) != -ENOMEM);
+		WARN_ON_ONCE(warn && PTR_ERR(old) != -ENOMEM);
 		return PTR_ERR(old);
 	}
 
@@ -343,6 +344,32 @@ static int tracepoint_remove_func(struct tracepoint *tp,
 	return 0;
 }
 
+/**
+ * tracepoint_probe_register_prio_may_exist -  Connect a probe to a tracepoint with priority
+ * @tp: tracepoint
+ * @probe: probe handler
+ * @data: tracepoint data
+ * @prio: priority of this function over other registered functions
+ *
+ * Same as tracepoint_probe_register_prio() except that it will not warn
+ * if the tracepoint is already registered.
+ */
+int tracepoint_probe_register_prio_may_exist(struct tracepoint *tp, void *probe,
+					     void *data, int prio)
+{
+	struct tracepoint_func tp_func;
+	int ret;
+
+	mutex_lock(&tracepoints_mutex);
+	tp_func.func = probe;
+	tp_func.data = data;
+	tp_func.prio = prio;
+	ret = tracepoint_add_func(tp, &tp_func, prio, false);
+	mutex_unlock(&tracepoints_mutex);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(tracepoint_probe_register_prio_may_exist);
+
 /**
  * tracepoint_probe_register_prio -  Connect a probe to a tracepoint with priority
  * @tp: tracepoint
@@ -366,7 +393,7 @@ int tracepoint_probe_register_prio(struct tracepoint *tp, void *probe,
 	tp_func.func = probe;
 	tp_func.data = data;
 	tp_func.prio = prio;
-	ret = tracepoint_add_func(tp, &tp_func, prio);
+	ret = tracepoint_add_func(tp, &tp_func, prio, true);
 	mutex_unlock(&tracepoints_mutex);
 	return ret;
 }
Signature · 1 line
2.29.2
syzbot
#1 lore raw

Re: [syzbot] WARNING in tracepoint_add_func

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by [email protected]

Tested on:

commit:         c54b245d Merge branch 'for-linus' of git://git.kernel.org/..
git tree:       upstream
kernel config:  https://syzkaller.appspot.com/x/.config?x=b55ee8fdb0113c34
dashboard link: https://syzkaller.appspot.com/bug?extid=721aa903751db87aa244

compiler:
patch: https://syzkaller.appspot.com/x/patch.diff?x=17938e5fd00000

Note: testing is done by a robot and is best-effort only.

#2 lore raw
In reply to Steven Rostedt
Quoted 16 lines · Steven Rostedt

From: "Steven Rostedt (VMware)" <[email protected]>

All internal use cases for tracepoint_probe_register() is set to not ever be called with the same function and data. If it is, it is considered a bug, as that means the accounting of handling tracepoints is corrupted. If the function and data for a tracepoint is already registered when tracepoint_probe_register() is called, it will call WARN_ON_ONCE() and return with EEXISTS.

The BPF system call can end up calling tracepoint_probe_register() with the same data, which now means that this can trigger the warning because of a user space process. As WARN_ON_ONCE() should not be called because user space called a system call with bad data, there needs to be a way to register a tracepoint without triggering a warning.

Quoted 8 lines

Enter tracepoint_probe_register_may_exist(), which can be called, but will not cause a WARN_ON() if the probe already exists. It will still error out with EEXIST, which will then be sent to the user space that performed the BPF system call.

This keeps the previous testing for issues with other users of the tracepoint code, while letting BPF call it with duplicated data and not warn about it.

There doesn't seem to be anything conceptually wrong with attaching the same BPF program twice to the same tracepoint. Is it a hard requirement to have a unique tp+callback combination, or was it done mostly to detect an API misuse? How hard is it to support such use cases?

I was surprised to discover this is not supported (though I never had a use for this, had to construct a test to see the warning).

Quoted 18 lines
Fixes c4f6699dfcb85 ("bpf: introduce BPF_RAW_TRACEPOINT")

---

#syz test: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master

 include/linux/tracepoint.h | 10 ++++++++++
 kernel/trace/bpf_trace.c   |  3 ++-
 kernel/tracepoint.c        | 33 ++++++++++++++++++++++++++++++---
 3 files changed, 42 insertions(+), 4 deletions(-)

[...]

author maintainer
#3 lore raw
Quoted 8 lines · On Wed, 7 Jul 2021 15:12:28 -0700 Andrii Nakryiko <[email protected]>…

There doesn't seem to be anything conceptually wrong with attaching the same BPF program twice to the same tracepoint. Is it a hard requirement to have a unique tp+callback combination, or was it done mostly to detect an API misuse? How hard is it to support such use cases?

I was surprised to discover this is not supported (though I never had a use for this, had to construct a test to see the warning).

The callback is identified by the function and its data combination. If there's two callbacks calling the same function with the same data on the same tracepoint, one question is, why? And the second is how do you differentiate the two?

-- Steve

#4 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Wed, 7 Jul 2021 15:12:28 -0700 Andrii Nakryiko <[email protected]>…

There doesn't seem to be anything conceptually wrong with attaching the same BPF program twice to the same tracepoint. Is it a hard requirement to have a unique tp+callback combination, or was it done mostly to detect an API misuse? How hard is it to support such use cases?

Quoted 7 lines
Quoted 2 lines

I was surprised to discover this is not supported (though I never had a use for this, had to construct a test to see the warning).

The callback is identified by the function and its data combination. If there's two callbacks calling the same function with the same data on the same tracepoint, one question is, why? And the second is how do you differentiate the two?

For places where multiple BPF programs can be attached (kprobes, cgroup programs, etc), we don't put a restriction that all programs have to be unique. It's totally legal to have the same program attached multiple times. So having this for tracepoints will be a one-off behavior.

As for why the user might need that, it's up to the user and I don't want to speculate because it will always sound contrived without a specific production use case. But people are very creative and we try not to dictate how and what can be done if it doesn't break any fundamental assumption and safety.

Quoted 2 lines

-- Steve

author maintainer
#5 lore raw
Quoted 5 lines · On Wed, 7 Jul 2021 16:49:26 -0700 Andrii Nakryiko <[email protected]>…

As for why the user might need that, it's up to the user and I don't want to speculate because it will always sound contrived without a specific production use case. But people are very creative and we try not to dictate how and what can be done if it doesn't break any fundamental assumption and safety.

I guess it doesn't matter, because if they try to do it, the second attachment will simply fail to attach.

-- Steve

#6 lore raw
Quoted 4 lines · Steven Rostedt

On Wed, 7 Jul 2021 16:49:26 -0700
Andrii Nakryiko <[email protected]> wrote:

Quoted 9 lines
Quoted 5 lines

As for why the user might need that, it's up to the user and I don't want to speculate because it will always sound contrived without a specific production use case. But people are very creative and we try not to dictate how and what can be done if it doesn't break any fundamental assumption and safety.

I guess it doesn't matter, because if they try to do it, the second attachment will simply fail to attach.

But not for the kprobe case.

And it might not always be possible to know that the same BPF program is being attached. It could be attached by different processes that re-use pinned program (without being aware of each other). Or it could be done from some generic library that just accepts prog_fd and doesn't really know the exact BPF program and whether it was already attached.

Not sure why it doesn't matter that attachment will fail where it is expected to succeed. The question is rather why such restriction?

Quoted 1 line

-- Steve

author maintainer
#7 lore raw
Quoted 11 lines · On Wed, 7 Jul 2021 17:23:54 -0700 Andrii Nakryiko <[email protected]>…
Quoted 10 lines · Steven Rostedt
Quoted 5 lines · On Wed, 7 Jul 2021 16:49:26 -0700 Andrii Nakryiko <[email protected]>…

As for why the user might need that, it's up to the user and I don't want to speculate because it will always sound contrived without a specific production use case. But people are very creative and we try not to dictate how and what can be done if it doesn't break any fundamental assumption and safety.

Quoted 5 lines
Quoted 3 lines

I guess it doesn't matter, because if they try to do it, the second attachment will simply fail to attach.

But not for the kprobe case.

What do you mean "not for the kprobe case"? What kprobe case?

You attach the same program twice to the same kprobe? Or do you create two kprobes at the same location?

Quoted 10 lines

And it might not always be possible to know that the same BPF program is being attached. It could be attached by different processes that re-use pinned program (without being aware of each other). Or it could be done from some generic library that just accepts prog_fd and doesn't really know the exact BPF program and whether it was already attached.

Not sure why it doesn't matter that attachment will fail where it is expected to succeed. The question is rather why such restriction?

Why is it expected to succeed? It never did. And why such a restriction? Because it complicates the code, and there's no good use case to do so. Why complicate something for little reward?

-- Steve

#8 lore raw
Quoted 19 lines · Steven Rostedt
Quoted 15 lines · On Wed, 7 Jul 2021 17:23:54 -0700 Andrii Nakryiko <[email protected]>…
Quoted 13 lines · Steven Rostedt
Quoted 5 lines · On Wed, 7 Jul 2021 16:49:26 -0700 Andrii Nakryiko <[email protected]>…

As for why the user might need that, it's up to the user and I don't want to speculate because it will always sound contrived without a specific production use case. But people are very creative and we try not to dictate how and what can be done if it doesn't break any fundamental assumption and safety.

I guess it doesn't matter, because if they try to do it, the second attachment will simply fail to attach.

Quoted 7 lines
Quoted 1 line

But not for the kprobe case.

What do you mean "not for the kprobe case"? What kprobe case?

You attach the same program twice to the same kprobe? Or do you create two kprobes at the same location?

I meant attaching the same BPF program twice to the same kernel function through the kprobe mechanism (through perf_event_open() syscall). From user perspective it's one BPF program attached twice to the same kprobe. Not entirely sure if two perf_event_open() calls will create two kprobes or re-use one internally.

Quoted 8 lines
Quoted 8 lines

And it might not always be possible to know that the same BPF program is being attached. It could be attached by different processes that re-use pinned program (without being aware of each other). Or it could be done from some generic library that just accepts prog_fd and doesn't really know the exact BPF program and whether it was already attached.

Quoted 6 lines
Quoted 2 lines

Not sure why it doesn't matter that attachment will fail where it is expected to succeed. The question is rather why such restriction?

Why is it expected to succeed? It never did. And why such a restriction? Because it complicates the code, and there's no good use case to do so. Why complicate something for little reward?

See above about kprobe for why it was my expectation.

But it was my original question whether this causes some complications or it's just an attempt to detect API mis-use. Seems like it's the former, alright.

Quoted 2 lines

-- Steve

#9 lore raw
In reply to Andrii Nakryiko
Quoted 24 lines · ----- On Jul 7, 2021, at 8:23 PM, Andrii Nakryiko [email protected]…
Quoted 4 lines · Steven Rostedt

On Wed, 7 Jul 2021 16:49:26 -0700
Andrii Nakryiko <[email protected]> wrote:

Quoted 9 lines
Quoted 5 lines

As for why the user might need that, it's up to the user and I don't want to speculate because it will always sound contrived without a specific production use case. But people are very creative and we try not to dictate how and what can be done if it doesn't break any fundamental assumption and safety.

I guess it doesn't matter, because if they try to do it, the second attachment will simply fail to attach.

But not for the kprobe case.

And it might not always be possible to know that the same BPF program is being attached. It could be attached by different processes that re-use pinned program (without being aware of each other). Or it could be done from some generic library that just accepts prog_fd and doesn't really know the exact BPF program and whether it was already attached.

Quoted 2 lines

Not sure why it doesn't matter that attachment will fail where it is expected to succeed. The question is rather why such restriction?

Before eBPF came to exist, all in-kernel users of the tracepoint API never required multiple registrations for a given (tracepoint, probe, data) tuple.

This allowed us to expose an API which can consider that the (tracepoint, probe, data) tuple is unique for each registration/unregistration pair, and therefore use that same tuple for unregistration. Refusing multiple registrations for a given tuple allows us to forgo the complexity of reference counting for duplicate registrations, and provide immediate feedback to misbehaving tracers which have duplicate registration or unbalanced registration/unregistration pairs.

From the perspective of a ring buffer tracer, the notion of multiple instances of a given (tracepoint, probe, data) tuple is rather silly: it would mean that a given tracepoint hit would generate many instances of the exact same event into the same trace buffer.

AFAIR, having the WARN_ON_ONCE() within the tracepoint code to highlight this kind of misuse allowed Steven to find a few unbalanced registration/unregistration issues while developing ftrace in the past. I vaguely recall that it triggered for blktrace at some point as well.

Considering that allowing duplicates would add complexity to the tracepoint code, what is the use-case justifying allowing many instances of the exact same callback and data for a given tracepoint ?

One key difference I notice here between eBPF and ring buffer tracers is what eBPF considers a "program". AFAIU (please let me know if I'm mistaken), the "callback" argument provided by eBPF to the tracepoint API is a limited set of trampoline routines. The bulk of the eBPF "program" is provided in the "data" argument. So this means the "program" is both the eBPF code and some context.

So I understand that a given eBPF code could be loaded more than once for a given tracepoint, but I would expect that each registration on a given tracepoint be provided with its own "context", otherwise we end up in a similar situation as the ring buffer's duplicated events scenario I explained above.

Also, we should discuss whether kprobes might benefit from being more strict by rejecting duplicated (instrumentation site, probe, data) tuples.

Thanks,

Mathieu

Signature · 3 lines
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
#10 lore raw
Quoted 45 lines · Mathieu Desnoyers
Quoted 24 lines · ----- On Jul 7, 2021, at 8:23 PM, Andrii Nakryiko [email protected]…
Quoted 4 lines · Steven Rostedt

On Wed, 7 Jul 2021 16:49:26 -0700
Andrii Nakryiko <[email protected]> wrote:

Quoted 9 lines
Quoted 5 lines

As for why the user might need that, it's up to the user and I don't want to speculate because it will always sound contrived without a specific production use case. But people are very creative and we try not to dictate how and what can be done if it doesn't break any fundamental assumption and safety.

I guess it doesn't matter, because if they try to do it, the second attachment will simply fail to attach.

But not for the kprobe case.

And it might not always be possible to know that the same BPF program is being attached. It could be attached by different processes that re-use pinned program (without being aware of each other). Or it could be done from some generic library that just accepts prog_fd and doesn't really know the exact BPF program and whether it was already attached.

Quoted 2 lines

Not sure why it doesn't matter that attachment will fail where it is expected to succeed. The question is rather why such restriction?

Before eBPF came to exist, all in-kernel users of the tracepoint API never required multiple registrations for a given (tracepoint, probe, data) tuple.

This allowed us to expose an API which can consider that the (tracepoint, probe, data) tuple is unique for each registration/unregistration pair, and therefore use that same tuple for unregistration. Refusing multiple registrations for a given tuple allows us to forgo the complexity of reference counting for duplicate registrations, and provide immediate feedback to misbehaving tracers which have duplicate registration or unbalanced registration/unregistration pairs.

From the perspective of a ring buffer tracer, the notion of multiple instances of a given (tracepoint, probe, data) tuple is rather silly: it would mean that a given tracepoint hit would generate many instances of the exact same event into the same trace buffer.

Quoted 7 lines

AFAIR, having the WARN_ON_ONCE() within the tracepoint code to highlight this kind of misuse allowed Steven to find a few unbalanced registration/unregistration issues while developing ftrace in the past. I vaguely recall that it triggered for blktrace at some point as well.

Considering that allowing duplicates would add complexity to the tracepoint code, what is the use-case justifying allowing many instances of the exact same callback and data for a given tracepoint ?

It wasn't clear to me if supporting this would cause any added complexity, which is why I asked.

Quoted 8 lines

One key difference I notice here between eBPF and ring buffer tracers is what eBPF considers a "program". AFAIU (please let me know if I'm mistaken), the "callback" argument provided by eBPF to the tracepoint API is a limited set of trampoline routines. The bulk of the eBPF "program" is provided in the "data" argument. So this means the "program" is both the eBPF code and some context.

So I understand that a given eBPF code could be loaded more than once for a given

No, it turns out it can't, I was just surprised to learn that. Surprised, because AFAIK we don't have such restrictions on uniqueness of attached BPF programs anywhere else where multiple BPF programs are allowed.

Quoted 15 lines

tracepoint, but I would expect that each registration on a given tracepoint be provided with its own "context", otherwise we end up in a similar situation as the ring buffer's duplicated events scenario I explained above.

Also, we should discuss whether kprobes might benefit from being more strict by rejecting duplicated (instrumentation site, probe, data) tuples.

Thanks,

Mathieu

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com