Skip to contentSkip to conversation

Fix bpf_link grace period wait for tracepoints

PATCH Superseded Kumar Kartikeya Dwivedi 4 participants 9 messages Mar 31

Subsystems (3)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Kumar Kartikeya Dwivedi

author maintainer
#0 lore raw

v3 cover Fix bpf_link grace period wait for tracepoints

A recent change to non-faultable tracepoints switched from preempt-disabled critical sections to SRCU-fast, which breaks assumptions in the bpf_link_free() path. Use call_srcu() to fix the breakage. While at it, retire rcu_trace_implies_rcu_gp() and clean up all in-tree users.

Changelog:
----------
v2 -> v3
v2: https://lore.kernel.org/bpf/[email protected]

* Introduce and switch to call_tracepoint_unregister_non_faultable(). (Steven)
* Address Andrii's comment and add Acked-by. (Andrii)
* Drop rcu_trace_implies_rcu_gp() conversion. (Alexei)

v1 -> v2
v1: https://lore.kernel.org/bpf/[email protected]

* Add Reviewed-by tags. (Paul, Puranjay)
* Adjust commit descriptions and comments to clarify intent. (Puranjay)

Kumar Kartikeya Dwivedi (1):
bpf: Fix grace period wait for tracepoint bpf_link

 include/linux/bpf.h        |  4 ++++
 include/linux/tracepoint.h |  8 ++++++++
 kernel/bpf/syscall.c       | 25 +++++++++++++++++++++++--
 3 files changed, 35 insertions(+), 2 deletions(-)

base-commit: c369299895a591d96745d6492d4888259b004a9e

Signature · 1 line
2.52.0
author maintainer
#1 lore raw

PATCH v3 1/1 bpf: Fix grace period wait for tracepoint bpf_link

Recently, tracepoints were switched from using disabled preemption (which acts as RCU read section) to SRCU-fast when they are not faultable. This means that to do a proper grace period wait for programs running in such tracepoints, we must use SRCU's grace period wait. This is only for non-faultable tracepoints, faultable ones continue using RCU Tasks Trace.

However, bpf_link_free() currently does call_rcu() for all cases when the link is non-sleepable (hence, for tracepoints, non-faultable). Fix this by doing a call_srcu() grace period wait.

As far RCU Tasks Trace gp -> RCU gp chaining is concerned, it is deemed unnecessary for tracepoint programs. The link and program are either accessed under RCU Tasks Trace protection, or SRCU-fast protection now.

The earlier logic of chaining both RCU Tasks Trace and RCU gp waits was to generalize the logic, even if it conceded an extra RCU gp wait, however that is unnecessary for tracepoints even before this change. In practice no cost was paid since rcu_trace_implies_rcu_gp() was always true. Hence we need not chaing any RCU gp after the SRCU gp.

For instance, in the non-faultable raw tracepoint, the RCU read section of the program in __bpf_trace_run() is enclosed in the SRCU gp, likewise for faultable raw tracepoint, the program is under the RCU Tasks Trace protection. Hence, the outermost scope can be waited upon to ensure correctness.

Fixes a46023d5616e ("tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast")
---
 include/linux/bpf.h        |  4 ++++
 include/linux/tracepoint.h |  8 ++++++++
 kernel/bpf/syscall.c       | 25 +++++++++++++++++++++++--
 3 files changed, 35 insertions(+), 2 deletions(-)
include/linux/bpf.h +4 -0
index 05b34a6355b0..35b1e25bd104 100644
@@ -1854,6 +1854,10 @@ struct bpf_link_ops {
 	 * target hook is sleepable, we'll go through tasks trace RCU GP and
 	 * then "classic" RCU GP; this need for chaining tasks trace and
 	 * classic RCU GPs is designated by setting bpf_link->sleepable flag
+	 *
+	 * For non-sleepable tracepoint links we go through SRCU gp instead,
+	 * since RCU is not used in that case. Sleepable tracepoints still
+	 * follow the scheme above.
 	 */
 	void (*dealloc_deferred)(struct bpf_link *link);
 	int (*detach)(struct bpf_link *link);
include/linux/tracepoint.h +8 -0
index 22ca1c8b54f3..8d6df1e7b5da 100644
@@ -122,6 +122,11 @@ static inline bool tracepoint_is_faultable(struct tracepoint *tp)
 {
 	return tp->ext && tp->ext->faultable;
 }
+static inline void call_tracepoint_unregister_non_faultable(struct rcu_head *rcu,
+							    rcu_callback_t func)
+{
+	call_srcu(&tracepoint_srcu, rcu, func);
+}
 #else
 static inline void tracepoint_synchronize_unregister(void)
 { }
@@ -129,6 +134,9 @@ static inline bool tracepoint_is_faultable(struct tracepoint *tp)
 {
 	return false;
 }
+static inline void call_tracepoint_unregister_non_faultable(struct rcu_head *rcu,
+							    rcu_callback_t func)
+{  }
 #endif
 
 #ifdef CONFIG_HAVE_SYSCALL_TRACEPOINTS
kernel/bpf/syscall.c +23 -2
index 274039e36465..96c3beebbf89 100644
@@ -3261,6 +3261,18 @@ static void bpf_link_defer_dealloc_rcu_gp(struct rcu_head *rcu)
 	bpf_link_dealloc(link);
 }
 
+static bool bpf_link_is_tracepoint(struct bpf_link *link)
+{
+	/*
+	 * Only these combinations support a tracepoint bpf_link.
+	 * BPF_LINK_TYPE_TRACING raw_tp progs are hardcoded to use
+	 * bpf_raw_tp_link_lops and thus dealloc_deferred(), see
+	 * bpf_raw_tp_link_attach().
+	 */
+	return link->type == BPF_LINK_TYPE_RAW_TRACEPOINT ||
+	       (link->type == BPF_LINK_TYPE_TRACING && link->attach_type == BPF_TRACE_RAW_TP);
+}
+
 static void bpf_link_defer_dealloc_mult_rcu_gp(struct rcu_head *rcu)
 {
 	if (rcu_trace_implies_rcu_gp())
@@ -3279,16 +3291,25 @@ static void bpf_link_free(struct bpf_link *link)
 	if (link->prog)
 		ops->release(link);
 	if (ops->dealloc_deferred) {
-		/* Schedule BPF link deallocation, which will only then
+		/*
+		 * Schedule BPF link deallocation, which will only then
 		 * trigger putting BPF program refcount.
 		 * If underlying BPF program is sleepable or BPF link's target
 		 * attach hookpoint is sleepable or otherwise requires RCU GPs
 		 * to ensure link and its underlying BPF program is not
 		 * reachable anymore, we need to first wait for RCU tasks
-		 * trace sync, and then go through "classic" RCU grace period
+		 * trace sync, and then go through "classic" RCU grace period.
+		 *
+		 * For tracepoint BPF links, we need to go through SRCU grace
+		 * period wait instead when non-faultable tracepoint is used. We
+		 * don't need to chain SRCU grace period waits, however, for the
+		 * faultable case, since it exclusively uses RCU Tasks Trace.
 		 */
 		if (link->sleepable || (link->prog && link->prog->sleepable))
 			call_rcu_tasks_trace(&link->rcu, bpf_link_defer_dealloc_mult_rcu_gp);
+		/* We need to do a SRCU grace period wait for tracepoint-based BPF links. */
+		else if (bpf_link_is_tracepoint(link))
+			call_tracepoint_unregister_non_faultable(&link->rcu, bpf_link_defer_dealloc_rcu_gp);
 		else
 			call_rcu(&link->rcu, bpf_link_defer_dealloc_rcu_gp);
 	} else if (ops->dealloc) {
Signature · 1 line
2.52.0
Reviewed-by on v3 1/1
#2 lore raw

Re: [PATCH bpf v3 1/1] bpf: Fix grace period wait for tracepoint bpf_link

Quoted 1 line · Kumar Kartikeya Dwivedi

As far RCU Tasks Trace gp -> RCU gp chaining is concerned, it is deemed

nit: "As far as" seems better here.

Quoted 8 lines

unnecessary for tracepoint programs. The link and program are either accessed under RCU Tasks Trace protection, or SRCU-fast protection now.

The earlier logic of chaining both RCU Tasks Trace and RCU gp waits was to generalize the logic, even if it conceded an extra RCU gp wait, however that is unnecessary for tracepoints even before this change. In practice no cost was paid since rcu_trace_implies_rcu_gp() was always true. Hence we need not chaing any RCU gp after the SRCU gp.

nit: "chaing" seems to be a typo of chain.

kernel/bpf/syscall.c static void bpf_link_free(struct bpf_link *link) · line 3312

Quoted 6 lines
                if (link->sleepable || (link->prog && link->prog->sleepable))
                        call_rcu_tasks_trace(&link->rcu, bpf_link_defer_dealloc_mult_rcu_gp);
kernel/bpf/syscall.c +3 -0
+               /* We need to do a SRCU grace period wait for tracepoint-based BPF links. */
+               else if (bpf_link_is_tracepoint(link))
+                       call_tracepoint_unregister_non_faultable(&link->rcu, bpf_link_defer_dealloc_rcu_gp);

As non-faultable tracepoints were switched to SRCU-fast, using call_srcu() for deferred
link deallocation looks right to me.

#3 lore raw

Re: [PATCH bpf v3 1/1] bpf: Fix grace period wait for tracepoint bpf_link

In reply to Kumar Kartikeya Dwivedi
Quoted 58 lines · Kumar Kartikeya Dwivedi

Recently, tracepoints were switched from using disabled preemption (which acts as RCU read section) to SRCU-fast when they are not faultable. This means that to do a proper grace period wait for programs running in such tracepoints, we must use SRCU's grace period wait. This is only for non-faultable tracepoints, faultable ones continue using RCU Tasks Trace.

However, bpf_link_free() currently does call_rcu() for all cases when the link is non-sleepable (hence, for tracepoints, non-faultable). Fix this by doing a call_srcu() grace period wait.

As far RCU Tasks Trace gp -> RCU gp chaining is concerned, it is deemed unnecessary for tracepoint programs. The link and program are either accessed under RCU Tasks Trace protection, or SRCU-fast protection now.

The earlier logic of chaining both RCU Tasks Trace and RCU gp waits was to generalize the logic, even if it conceded an extra RCU gp wait, however that is unnecessary for tracepoints even before this change. In practice no cost was paid since rcu_trace_implies_rcu_gp() was always true. Hence we need not chaing any RCU gp after the SRCU gp.

For instance, in the non-faultable raw tracepoint, the RCU read section of the program in __bpf_trace_run() is enclosed in the SRCU gp, likewise for faultable raw tracepoint, the program is under the RCU Tasks Trace protection. Hence, the outermost scope can be waited upon to ensure correctness.

Fixes a46023d5616e ("tracing: Guard __DECLARE_TRACE() use of __DO_TRACE_CALL() with SRCU-fast")
---
 include/linux/bpf.h        |  4 ++++
 include/linux/tracepoint.h |  8 ++++++++
 kernel/bpf/syscall.c       | 25 +++++++++++++++++++++++--
 3 files changed, 35 insertions(+), 2 deletions(-)
include/linux/bpf.h +4 -0
index 05b34a6355b0..35b1e25bd104 100644
@@ -1854,6 +1854,10 @@ struct bpf_link_ops {
         * target hook is sleepable, we'll go through tasks trace RCU GP and
         * then "classic" RCU GP; this need for chaining tasks trace and
         * classic RCU GPs is designated by setting bpf_link->sleepable flag
+        *
+        * For non-sleepable tracepoint links we go through SRCU gp instead,
+        * since RCU is not used in that case. Sleepable tracepoints still
+        * follow the scheme above.
         */
        void (*dealloc_deferred)(struct bpf_link *link);
        int (*detach)(struct bpf_link *link);
include/linux/tracepoint.h +0 -0
index 22ca1c8b54f3..8d6df1e7b5da 100644

include/linux/tracepoint.h static inline bool tracepoint_is_faultable(struct tracepoint *tp) · line 126

Quoted 5 lines
include/linux/tracepoint.h +2 -0
@@ -122,6 +122,11 @@ static inline bool tracepoint_is_faultable(struct tracepoint *tp)
 {
        return tp->ext && tp->ext->faultable;
 }
+static inline void call_tracepoint_unregister_non_faultable(struct rcu_head *rcu,
+                                                           rcu_callback_t func)

The name is too long. imo.
call_tracepoint_srcu() ?

Steven ?

maintainer
#4 lore raw

Re: [PATCH bpf v3 1/1] bpf: Fix grace period wait for tracepoint bpf_link

Quoted 4 lines · On Tue, 31 Mar 2026 08:27:22 -0700 Alexei Starovoitov…
Quoted 4 lines
include/linux/tracepoint.h +0 -0
index 22ca1c8b54f3..8d6df1e7b5da 100644
Quoted 8 lines

include/linux/tracepoint.h static inline bool tracepoint_is_faultable(struct tracepoint *tp) · line 126

Quoted 6 lines
include/linux/tracepoint.h +2 -0
@@ -122,6 +122,11 @@ static inline bool tracepoint_is_faultable(struct tracepoint *tp)
 {
        return tp->ext && tp->ext->faultable;
 }
+static inline void call_tracepoint_unregister_non_faultable(struct rcu_head *rcu,
+                                                           rcu_callback_t func)  

The name is too long. imo.

Don't like typing? ;-)

Quoted 1 line

call_tracepoint_srcu() ?

I don't want "srcu" involved in the API. That's the internal workings and who knows, it may change in the future (like it did going from RCU to SRCU).

-- Steve

#5 lore raw

Re: [PATCH bpf v3 1/1] bpf: Fix grace period wait for tracepoint bpf_link

Quoted 17 lines · Steven Rostedt
Quoted 4 lines · On Tue, 31 Mar 2026 08:27:22 -0700 Alexei Starovoitov…
Quoted 4 lines
include/linux/tracepoint.h +0 -0
index 22ca1c8b54f3..8d6df1e7b5da 100644
Quoted 8 lines

include/linux/tracepoint.h static inline bool tracepoint_is_faultable(struct tracepoint *tp) · line 126

Quoted 6 lines
include/linux/tracepoint.h +2 -0
@@ -122,6 +122,11 @@ static inline bool tracepoint_is_faultable(struct tracepoint *tp)
 {
        return tp->ext && tp->ext->faultable;
 }
+static inline void call_tracepoint_unregister_non_faultable(struct rcu_head *rcu,
+                                                           rcu_callback_t func)

The name is too long. imo.

Quoted 6 lines

Don't like typing? ;-)

Quoted 1 line

call_tracepoint_srcu() ?

I don't want "srcu" involved in the API. That's the internal workings and who knows, it may change in the future (like it did going from RCU to SRCU).

so? When it changes we can rename it to.
I don't think "srcu" part of the name leaking anything.

maintainer
#6 lore raw

Re: [PATCH bpf v3 1/1] bpf: Fix grace period wait for tracepoint bpf_link

Quoted 5 lines · On Tue, 31 Mar 2026 09:04:33 -0700 Alexei Starovoitov…
Quoted 2 lines

I don't want "srcu" involved in the API. That's the internal workings and who knows, it may change in the future (like it did going from RCU to SRCU).

so? When it changes we can rename it to.
I don't think "srcu" part of the name leaking anything.

I prefer to have a API describe what it's doing and not "how" it's doing it.

We can have:

call_tracepoint_unregister_atomic(...)

and

call_tracepoint_unregister_syscall()

The "atomic" describes tracepoints that can be used in atomic sections, where as "syscall" means the tracepoint are safer for syscall context only (faulting, etc).

Yes, it's still long, but I like descriptive names.

-- Steve

#7 lore raw

Re: [PATCH bpf v3 1/1] bpf: Fix grace period wait for tracepoint bpf_link

Quoted 16 lines · Steven Rostedt
Quoted 5 lines · On Tue, 31 Mar 2026 09:04:33 -0700 Alexei Starovoitov…
Quoted 2 lines

I don't want "srcu" involved in the API. That's the internal workings and who knows, it may change in the future (like it did going from RCU to SRCU).

so? When it changes we can rename it to.
I don't think "srcu" part of the name leaking anything.

I prefer to have a API describe what it's doing and not "how" it's doing it.

We can have:

call_tracepoint_unregister_atomic(...)

Quoted 7 lines

and

call_tracepoint_unregister_syscall()

The "atomic" describes tracepoints that can be used in atomic sections, where as "syscall" means the tracepoint are safer for syscall context only (faulting, etc).

These two names work for me.

author maintainer
#8 lore raw

Re: [PATCH bpf v3 1/1] bpf: Fix grace period wait for tracepoint bpf_link

Quoted 20 lines · Alexei Starovoitov
Quoted 18 lines · Steven Rostedt
Quoted 5 lines · On Tue, 31 Mar 2026 09:04:33 -0700 Alexei Starovoitov…
Quoted 2 lines

I don't want "srcu" involved in the API. That's the internal workings and who knows, it may change in the future (like it did going from RCU to SRCU).

so? When it changes we can rename it to.
I don't think "srcu" part of the name leaking anything.

I prefer to have a API describe what it's doing and not "how" it's doing it.

We can have:

call_tracepoint_unregister_atomic(...)

and

Quoted 7 lines
Quoted 5 lines

call_tracepoint_unregister_syscall()

The "atomic" describes tracepoints that can be used in atomic sections, where as "syscall" means the tracepoint are safer for syscall context only (faulting, etc).

These two names work for me.

Just to confirm, do we want to add call_tracepoint_unregister_syscall() without any users?