Skip to contentSkip to conversation

uprobes: Switch uretprobes_srcu to SRCU-fast-updown

PATCH Applied Puranjay Mohan 8 participants 13 messages Jul 6 – Aug 3

Subsystems (6)
BPF perf events Tracing RCU +2 more subsystems
lore
Share & tools
lore
#0 lore raw

cover uprobes: Switch uretprobes_srcu to SRCU-fast-updown

uretprobes_srcu currently uses normal SRCU. Normal SRCU issues an smp_mb() on both srcu_read_lock() and srcu_read_unlock(), i.e. two full memory barriers per read-side critical section. For uretprobes this cost is paid on every uretprobe invocation: prepare_uretprobe() takes the read lock that is later dropped on the normal return path (hprobe_finalize()) or from ri_timer()/dup_utask() (hprobe_expire()).

Switch uretprobes_srcu to the SRCU-fast-updown flavor. SRCU-fast moves the read-side ordering off the reader and onto the (rare) grace-period
side: synchronize_srcu() rides on synchronize_rcu() instead of relying on reader-side smp_mb(). This is a good trade for uretprobes, where reader-side hits vastly outnumber grace periods (uprobe unregistration).

The updown variant (rather than plain SRCU-fast) is required because the read lock is acquired in prepare_uretprobe() on the way out to user space and is released only once the return instance is finalized -- from a different context than it was taken: the normal return path (uprobe_handle_trampoline() -> hprobe_finalize()), the timer callback, or the fork path (ri_timer()/dup_utask() -> hprobe_expire()). srcu_down_read_fast()/srcu_up_read_fast() are designed for this semaphore-like, cross-context pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry lockdep read-side tracking that would warn on it -- which is why the old code had to use the raw __srcu_read_lock() here. For the short, same-context sections in ri_timer() and dup_utask(), guard(srcu_fast_updown) is used instead, giving proper lockdep coverage.

Patch 1 adds the guard(srcu_fast_updown) definition, following the existing guard(srcu)/guard(srcu_fast) pattern.
Patch 2 does the uretprobes_srcu conversion.

Note
----
Only uretprobes_srcu is converted; the main uprobe readers (RB-tree lookup and consumer-list iteration) are deliberately left on RCU Tasks Trace. RCU Tasks Trace is already implemented on top of srcu_read_lock_fast(), so the reader-side cost is identical, and it has a nesting fast path that the uprobe -> sleepable-BPF-program call chain relies on (the BPF trampoline takes rcu_read_lock_trace() while uprobes already holds it; the nested acquire is just a counter bump). Converting those readers to a separate srcu_struct would turn one real + one nested lock into two real locks and lose that optimization for no reader-side gain. uretprobes_srcu is different: it uses normal SRCU (not Tasks Trace), its readers are long-lived and cross-context, and it genuinely benefits from dropping the per-reader barriers.

Puranjay Mohan (2):
srcu: Add lock guard for srcu_fast_updown flavor
uprobes: Switch uretprobes_srcu to SRCU-fast-updown

 include/linux/srcu.h    |  7 +++++++
 include/linux/uprobes.h |  5 +++--
 kernel/events/uprobes.c | 29 +++++++++++++++++------------
 3 files changed, 27 insertions(+), 14 deletions(-)

base-commit: 87bfe634b1193db90e5170e1ddbad04a63ef4501

Signature · 1 line
2.53.0-Meta
#1 lore raw

PATCH 1/2 srcu: Add lock guard for srcu_fast_updown flavor

Add a guard(srcu_fast_updown) definition for scoped SRCU-fast-updown read-side critical sections, following the existing pattern of guard(srcu) and guard(srcu_fast).

---
 include/linux/srcu.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/linux/srcu.h +7 -0
index a54ce9e808b92..72c86d3b23f2e 100644
@@ -638,4 +638,11 @@ DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct,
 DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
 #define class_srcu_fast_notrace_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, _T)
 
+DEFINE_LOCK_GUARD_1(srcu_fast_updown, struct srcu_struct,
+		    _T->scp = srcu_read_lock_fast_updown(_T->lock),
+		    srcu_read_unlock_fast_updown(_T->lock, _T->scp),
+		    struct srcu_ctr __percpu *scp)
+DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_updown, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
+#define class_srcu_fast_updown_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_updown, _T)
+
 #endif
Signature · 1 line
2.53.0-Meta
maintainer Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH bpf-next 1/2] srcu: Add lock guard for srcu_fast_updown flavor

Quoted 5 lines · Puranjay Mohan

Add a guard(srcu_fast_updown) definition for scoped SRCU-fast-updown read-side critical sections, following the existing pattern of guard(srcu) and guard(srcu_fast).

(Or I can take it if you would prefer, but it might be easier and faster for you to send it along with the next patch.)

Quoted 23 lines
---
 include/linux/srcu.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/linux/srcu.h +7 -0
index a54ce9e808b92..72c86d3b23f2e 100644
@@ -638,4 +638,11 @@ DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct,
 DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
 #define class_srcu_fast_notrace_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, _T)
 
+DEFINE_LOCK_GUARD_1(srcu_fast_updown, struct srcu_struct,
+		    _T->scp = srcu_read_lock_fast_updown(_T->lock),
+		    srcu_read_unlock_fast_updown(_T->lock, _T->scp),
+		    struct srcu_ctr __percpu *scp)
+DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_updown, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
+#define class_srcu_fast_updown_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_updown, _T)
+
 #endif
Signature · 1 line
2.53.0-Meta
maintainer Reviewed-by on v1 1/2
#3 lore raw

Re: [PATCH bpf-next 1/2] srcu: Add lock guard for srcu_fast_updown flavor

In reply to Puranjay Mohan
Quoted 15 lines · Puranjay Mohan

Add a guard(srcu_fast_updown) definition for scoped SRCU-fast-updown read-side critical sections, following the existing pattern of guard(srcu) and guard(srcu_fast).

---
 include/linux/srcu.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/linux/srcu.h +0 -0
index a54ce9e808b92..72c86d3b23f2e 100644

include/linux/srcu.h DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct, · line 646

Quoted 9 lines
include/linux/srcu.h +6 -0
@@ -638,4 +638,11 @@ DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct,
 DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
 #define class_srcu_fast_notrace_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, _T)
 
+DEFINE_LOCK_GUARD_1(srcu_fast_updown, struct srcu_struct,
+		    _T->scp = srcu_read_lock_fast_updown(_T->lock),
+		    srcu_read_unlock_fast_updown(_T->lock, _T->scp),
+		    struct srcu_ctr __percpu *scp)
+DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_updown, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
+#define class_srcu_fast_updown_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_updown, _T)

Looks "obviously correct" to me. FWIW,

merge bot Applied
#4 lore raw

[tip: perf/core] srcu: Add lock guard for srcu_fast_updown flavor

In reply to Puranjay Mohan
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     fa8a2d55139fb2e0b09e68c34b730adac92d18d6
Gitweb:        https://git.kernel.org/tip/fa8a2d55139fb2e0b09e68c34b730adac92d18d6
Author:        Puranjay Mohan <[email protected]>
AuthorDate:    Mon, 06 Jul 2026 10:27:41 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 03 Aug 2026 10:09:08 +02:00

srcu: Add lock guard for srcu_fast_updown flavor

Add a guard(srcu_fast_updown) definition for scoped SRCU-fast-updown read-side critical sections, following the existing pattern of guard(srcu) and guard(srcu_fast).

---
 include/linux/srcu.h | 7 +++++++
 1 file changed, 7 insertions(+)
include/linux/srcu.h +7 -0
index a54ce9e..72c86d3 100644
@@ -638,4 +638,11 @@ DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct,
 DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
 #define class_srcu_fast_notrace_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, _T)
 
+DEFINE_LOCK_GUARD_1(srcu_fast_updown, struct srcu_struct,
+		    _T->scp = srcu_read_lock_fast_updown(_T->lock),
+		    srcu_read_unlock_fast_updown(_T->lock, _T->scp),
+		    struct srcu_ctr __percpu *scp)
+DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_updown, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))
+#define class_srcu_fast_updown_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_updown, _T)
+
 #endif
#5 lore raw

PATCH 2/2 uprobes: Switch uretprobes_srcu to SRCU-fast-updown

In reply to Puranjay Mohan

uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit.

Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (synchronize_rcu() instead of smp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.

The updown flavor is required because the SRCU read lock is taken in prepare_uretprobe() when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path via uprobe_handle_trampoline() -> hprobe_finalize(), or from ri_timer() (expiry) or dup_utask() (fork) via hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast() are designed for this acquire-here / release-elsewhere pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry the lockdep read-side tracking that would warn on it.

The short, same-context SRCU sections in ri_timer() and dup_utask() (which guard the uprobe against reuse across the hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown) for proper lockdep coverage.

---
 include/linux/uprobes.h |  5 +++--
 kernel/events/uprobes.c | 29 +++++++++++++++++------------
 2 files changed, 20 insertions(+), 14 deletions(-)
include/linux/uprobes.h +3 -2
index f548fea2adec8..f3b07753c2f3d 100644
@@ -25,6 +25,7 @@ struct mm_struct;
 struct inode;
 struct notifier_block;
 struct page;
+struct srcu_ctr;
 
 /*
  * Allowed return values from uprobe consumer's handler callback
@@ -106,7 +107,7 @@ enum hprobe_state {
  *     underlying uprobe is not guaranteed anymore. __UPROBE_DEAD is just an
  *     internal marker and is handled transparently by hprobe_fetch() helper.
  *
- * When uprobe is SRCU-protected, we also record srcu_idx value, necessary for
+ * When uprobe is SRCU-protected, we also record srcu_scp value, necessary for
  * SRCU unlocking.
  *
  * See hprobe_expire() and hprobe_fetch() for details of race-free uprobe
@@ -115,7 +116,7 @@ enum hprobe_state {
  */
 struct hprobe {
 	enum hprobe_state state;
-	int srcu_idx;
+	struct srcu_ctr __percpu *srcu_scp;
 	struct uprobe *uprobe;
 };
 
kernel/events/uprobes.c +17 -12
index 4084e926e2844..afa491b0bd3f9 100644
@@ -54,7 +54,7 @@ static struct mutex uprobes_mmap_mutex[UPROBES_HASH_SZ];
 DEFINE_STATIC_PERCPU_RWSEM(dup_mmap_sem);
 
 /* Covers return_instance's uprobe lifetime. */
-DEFINE_STATIC_SRCU(uretprobes_srcu);
+DEFINE_STATIC_SRCU_FAST_UPDOWN(uretprobes_srcu);
 
 /* Have a copy of original instruction */
 #define UPROBE_COPY_INSN	0
@@ -707,12 +707,13 @@ static void put_uprobe(struct uprobe *uprobe)
 }
 
 /* Initialize hprobe as SRCU-protected "leased" uprobe */
-static void hprobe_init_leased(struct hprobe *hprobe, struct uprobe *uprobe, int srcu_idx)
+static void hprobe_init_leased(struct hprobe *hprobe, struct uprobe *uprobe,
+			       struct srcu_ctr __percpu *srcu_scp)
 {
 	WARN_ON(!uprobe);
 	hprobe->state = HPROBE_LEASED;
 	hprobe->uprobe = uprobe;
-	hprobe->srcu_idx = srcu_idx;
+	hprobe->srcu_scp = srcu_scp;
 }
 
 /* Initialize hprobe as refcounted ("stable") uprobe (uprobe can be NULL). */
@@ -720,7 +721,7 @@ static void hprobe_init_stable(struct hprobe *hprobe, struct uprobe *uprobe)
 {
 	hprobe->state = uprobe ? HPROBE_STABLE : HPROBE_GONE;
 	hprobe->uprobe = uprobe;
-	hprobe->srcu_idx = -1;
+	hprobe->srcu_scp = NULL;
 }
 
 /*
@@ -757,7 +758,7 @@ static void hprobe_finalize(struct hprobe *hprobe, enum hprobe_state hstate)
 {
 	switch (hstate) {
 	case HPROBE_LEASED:
-		__srcu_read_unlock(&uretprobes_srcu, hprobe->srcu_idx);
+		srcu_up_read_fast(&uretprobes_srcu, hprobe->srcu_scp);
 		break;
 	case HPROBE_STABLE:
 		put_uprobe(hprobe->uprobe);
@@ -829,7 +830,7 @@ static struct uprobe *hprobe_expire(struct hprobe *hprobe, bool get)
 		 */
 		if (try_cmpxchg(&hprobe->state, &hstate, uprobe ? HPROBE_STABLE : HPROBE_GONE)) {
 			/* We won the race, we are the ones to unlock SRCU */
-			__srcu_read_unlock(&uretprobes_srcu, hprobe->srcu_idx);
+			srcu_up_read_fast(&uretprobes_srcu, hprobe->srcu_scp);
 			return get ? get_uprobe(uprobe) : uprobe;
 		}
 
@@ -2045,7 +2046,7 @@ static void ri_timer(struct timer_list *timer)
 	struct return_instance *ri;
 
 	/* SRCU protects uprobe from reuse for the cmpxchg() inside hprobe_expire(). */
-	guard(srcu)(&uretprobes_srcu);
+	guard(srcu_fast_updown)(&uretprobes_srcu);
 	/* RCU protects return_instance from freeing. */
 	guard(rcu)();
 
@@ -2142,7 +2143,7 @@ static int dup_utask(struct task_struct *t, struct uprobe_task *o_utask)
 	t->utask = n_utask;
 
 	/* protect uprobes from freeing, we'll need try_get_uprobe() them */
-	guard(srcu)(&uretprobes_srcu);
+	guard(srcu_fast_updown)(&uretprobes_srcu);
 
 	p = &n_utask->return_instances;
 	for (o = o_utask->return_instances; o; o = o->next) {
@@ -2254,8 +2255,8 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs,
 {
 	struct uprobe_task *utask = current->utask;
 	unsigned long orig_ret_vaddr, trampoline_vaddr;
+	struct srcu_ctr __percpu *srcu_scp;
 	bool chained;
-	int srcu_idx;
 
 	if (!get_xol_area())
 		goto free;
@@ -2293,8 +2294,12 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs,
 		orig_ret_vaddr = utask->return_instances->orig_ret_vaddr;
 	}
 
-	/* __srcu_read_lock() because SRCU lock survives switch to user space */
-	srcu_idx = __srcu_read_lock(&uretprobes_srcu);
+	/*
+	 * Use srcu_down_read_fast() because the SRCU lock survives a switch to
+	 * user space and can be unlocked from a different context by ri_timer()
+	 * or dup_utask().
+	 */
+	srcu_scp = srcu_down_read_fast(&uretprobes_srcu);
 
 	ri->func = instruction_pointer(regs);
 	ri->stack = user_stack_pointer(regs);
@@ -2303,7 +2308,7 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs,
 
 	utask->depth++;
 
-	hprobe_init_leased(&ri->hprobe, uprobe, srcu_idx);
+	hprobe_init_leased(&ri->hprobe, uprobe, srcu_scp);
 	ri->next = utask->return_instances;
 	rcu_assign_pointer(utask->return_instances, ri);
 
Signature · 1 line
2.53.0-Meta
#6 lore raw
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Concurrent hprobe_expire() and hprobe_consume() can cause a use-after-free on uprobe due to misunderstood SRCU semantics. --

commit 86c857f0a70ec453b9e38c36e230b36b91bef460
Author: Puranjay Mohan <[email protected]>

uprobes: Switch uretprobes_srcu to SRCU-fast-updown

uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit.

Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side.

Quoted 4 lines
kernel/events/uprobes.c +0 -0
index 4084e926e2844..afa491b0bd3f9 100644

[ ... ]

kernel/events/uprobes.c static void ri_timer(struct timer_list *timer) · line 2049

Quoted 6 lines
kernel/events/uprobes.c +1 -1
@@ -2045,7 +2046,7 @@ static void ri_timer(struct timer_list *timer)
 	struct return_instance *ri;
 
 	/* SRCU protects uprobe from reuse for the cmpxchg() inside hprobe_expire(). */
-	guard(srcu)(&uretprobes_srcu);
+	guard(srcu_fast_updown)(&uretprobes_srcu);

[Severity: High]
This isn't a bug introduced by this patch, but does the new SRCU read lock acquired here actually protect the uprobe from being freed?

If a uprobe is being unregistered, uprobe_free_deferred() calls call_srcu(&uretprobes_srcu, ...) to free the uprobe. This starts an SRCU grace period that waits only for pre-existing SRCU readers to finish.

Concurrently, if ri_timer() or dup_utask() takes this new SRCU read lock, the grace period will not wait for it because the new lock started after the grace period began.

If the traced task concurrently executes uprobe_handle_trampoline(), calls hprobe_consume() and hprobe_finalize(), and releases the original SRCU read lock, the grace period can complete and free the uprobe (via call_rcu_tasks_trace()).

When ri_timer() then proceeds to call hprobe_expire(), which does:

kernel/events/uprobes.c:hprobe_expire() {
    ...
    case HPROBE_LEASED: {
        struct uprobe *uprobe = try_get_uprobe(hprobe->uprobe);
    ...
}

Could this lead to a use-after-free on the uprobe? The comment in hprobe_expire() seems to assume the new lock prevents the free:

/*
* Even if hprobe_consume() or another hprobe_expire() wins
* the state update race and unlocks SRCU from under us, we
* still have a guarantee that underyling uprobe won't be
* freed due to ongoing caller's SRCU lock region, so we can
* return it regardless.
*/

Is this analysis accurate?

Quoted 3 lines
kernel/events/uprobes.c +0 -0
 	/* RCU protects return_instance from freeing. */
 	guard(rcu)();
 
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Acked-by on v1 2/2
#7 lore raw
In reply to Puranjay Mohan
Quoted 31 lines · Puranjay Mohan

uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit.

Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (synchronize_rcu() instead of smp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.

The updown flavor is required because the SRCU read lock is taken in prepare_uretprobe() when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path via uprobe_handle_trampoline() -> hprobe_finalize(), or from ri_timer() (expiry) or dup_utask() (fork) via hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast() are designed for this acquire-here / release-elsewhere pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry the lockdep read-side tracking that would warn on it.

The short, same-context SRCU sections in ri_timer() and dup_utask() (which guard the uprobe against reuse across the hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown) for proper lockdep coverage.

Quoted 6 lines
---
 include/linux/uprobes.h |  5 +++--
 kernel/events/uprobes.c | 29 +++++++++++++++++------------
 2 files changed, 20 insertions(+), 14 deletions(-)

Very straightforward replacement and will improve performance, thanks! LGTM. Peter or Mingo, I assume you guys will want to route this through your tip tree, right?

[...]

#8 lore raw
Quoted 39 lines · Andrii Nakryiko
Quoted 31 lines · Puranjay Mohan

uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit.

Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (synchronize_rcu() instead of smp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.

The updown flavor is required because the SRCU read lock is taken in prepare_uretprobe() when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path via uprobe_handle_trampoline() -> hprobe_finalize(), or from ri_timer() (expiry) or dup_utask() (fork) via hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast() are designed for this acquire-here / release-elsewhere pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry the lockdep read-side tracking that would warn on it.

The short, same-context SRCU sections in ri_timer() and dup_utask() (which guard the uprobe against reuse across the hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown) for proper lockdep coverage.

Quoted 6 lines
---
 include/linux/uprobes.h |  5 +++--
 kernel/events/uprobes.c | 29 +++++++++++++++++------------
 2 files changed, 20 insertions(+), 14 deletions(-)
Quoted 6 lines

Very straightforward replacement and will improve performance, thanks! LGTM. Peter or Mingo, I assume you guys will want to route this through your tip tree, right?

Hi Peter, Ingo,

Gentle ping for this.

Quoted 1 line

[...]

#9 lore raw
Quoted 45 lines · Kumar Kartikeya Dwivedi
Quoted 43 lines · Andrii Nakryiko
Quoted 31 lines · Puranjay Mohan

uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit.

Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (synchronize_rcu() instead of smp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.

The updown flavor is required because the SRCU read lock is taken in prepare_uretprobe() when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path via uprobe_handle_trampoline() -> hprobe_finalize(), or from ri_timer() (expiry) or dup_utask() (fork) via hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast() are designed for this acquire-here / release-elsewhere pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry the lockdep read-side tracking that would warn on it.

The short, same-context SRCU sections in ri_timer() and dup_utask() (which guard the uprobe against reuse across the hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown) for proper lockdep coverage.

Quoted 6 lines
---
 include/linux/uprobes.h |  5 +++--
 kernel/events/uprobes.c | 29 +++++++++++++++++------------
 2 files changed, 20 insertions(+), 14 deletions(-)

Very straightforward replacement and will improve performance, thanks! LGTM. Peter or Mingo, I assume you guys will want to route this through your tip tree, right?

Quoted 7 lines
Quoted 2 lines

Hi Peter, Ingo,

Gentle ping for this.

Another ping, please consider applying, thanks!

Quoted 2 lines
Quoted 1 line

[...]

maintainer
#10 lore raw
Quoted 50 lines · Andrii Nakryiko
Quoted 48 lines · Kumar Kartikeya Dwivedi
Quoted 45 lines · Andrii Nakryiko
Quoted 31 lines · Puranjay Mohan

uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit.

Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (synchronize_rcu() instead of smp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.

The updown flavor is required because the SRCU read lock is taken in prepare_uretprobe() when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path via uprobe_handle_trampoline() -> hprobe_finalize(), or from ri_timer() (expiry) or dup_utask() (fork) via hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast() are designed for this acquire-here / release-elsewhere pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry the lockdep read-side tracking that would warn on it.

The short, same-context SRCU sections in ri_timer() and dup_utask() (which guard the uprobe against reuse across the hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown) for proper lockdep coverage.

Quoted 6 lines
---
 include/linux/uprobes.h |  5 +++--
 kernel/events/uprobes.c | 29 +++++++++++++++++------------
 2 files changed, 20 insertions(+), 14 deletions(-)

Very straightforward replacement and will improve performance, thanks! LGTM. Peter or Mingo, I assume you guys will want to route this through your tip tree, right?

Quoted 6 lines
Quoted 4 lines

Hi Peter, Ingo,

Gentle ping for this.

Another ping, please consider applying, thanks!

Got them, thanks!

maintainer Reviewed-by on v1 2/2
#11 lore raw
In reply to Puranjay Mohan
Quoted 31 lines · Puranjay Mohan

uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit.

Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (synchronize_rcu() instead of smp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.

The updown flavor is required because the SRCU read lock is taken in prepare_uretprobe() when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path via uprobe_handle_trampoline() -> hprobe_finalize(), or from ri_timer() (expiry) or dup_utask() (fork) via hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast() are designed for this acquire-here / release-elsewhere pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry the lockdep read-side tracking that would warn on it.

The short, same-context SRCU sections in ri_timer() and dup_utask() (which guard the uprobe against reuse across the hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown) for proper lockdep coverage.

Quoted 5 lines
---
 include/linux/uprobes.h |  5 +++--
 kernel/events/uprobes.c | 29 +++++++++++++++++------------
 2 files changed, 20 insertions(+), 14 deletions(-)

And the commit d3f52f53a562 ("srcu: Create an SRCU-fast-updown API") even mentions uretprobes as a likely user.

merge bot Applied
#12 lore raw
In reply to Puranjay Mohan
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     36c8b02c3fe9ba56d6f11ec055cd879bc49871f7
Gitweb:        https://git.kernel.org/tip/36c8b02c3fe9ba56d6f11ec055cd879bc49871f7
Author:        Puranjay Mohan <[email protected]>
AuthorDate:    Mon, 06 Jul 2026 10:27:42 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Mon, 03 Aug 2026 10:09:08 +02:00

uprobes: Switch uretprobes_srcu to SRCU-fast-updown

uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit.

Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (synchronize_rcu() instead of smp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.

The updown flavor is required because the SRCU read lock is taken in prepare_uretprobe() when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path via uprobe_handle_trampoline() -> hprobe_finalize(), or from ri_timer() (expiry) or dup_utask() (fork) via hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast() are designed for this acquire-here / release-elsewhere pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry the lockdep read-side tracking that would warn on it.

The short, same-context SRCU sections in ri_timer() and dup_utask() (which guard the uprobe against reuse across the hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown) for proper lockdep coverage.

---
 include/linux/uprobes.h |  5 +++--
 kernel/events/uprobes.c | 29 +++++++++++++++++------------
 2 files changed, 20 insertions(+), 14 deletions(-)
include/linux/uprobes.h +3 -2
index 18be159..d34dbc0 100644
@@ -25,6 +25,7 @@ struct mm_struct;
 struct inode;
 struct notifier_block;
 struct page;
+struct srcu_ctr;
 
 /*
  * Allowed return values from uprobe consumer's handler callback
@@ -106,7 +107,7 @@ enum hprobe_state {
  *     underlying uprobe is not guaranteed anymore. __UPROBE_DEAD is just an
  *     internal marker and is handled transparently by hprobe_fetch() helper.
  *
- * When uprobe is SRCU-protected, we also record srcu_idx value, necessary for
+ * When uprobe is SRCU-protected, we also record srcu_scp value, necessary for
  * SRCU unlocking.
  *
  * See hprobe_expire() and hprobe_fetch() for details of race-free uprobe
@@ -115,7 +116,7 @@ enum hprobe_state {
  */
 struct hprobe {
 	enum hprobe_state state;
-	int srcu_idx;
+	struct srcu_ctr __percpu *srcu_scp;
 	struct uprobe *uprobe;
 };
 
kernel/events/uprobes.c +17 -12
index e4f526c..a18529a 100644
@@ -54,7 +54,7 @@ static struct mutex uprobes_mmap_mutex[UPROBES_HASH_SZ];
 DEFINE_STATIC_PERCPU_RWSEM(dup_mmap_sem);
 
 /* Covers return_instance's uprobe lifetime. */
-DEFINE_STATIC_SRCU(uretprobes_srcu);
+DEFINE_STATIC_SRCU_FAST_UPDOWN(uretprobes_srcu);
 
 /* Have a copy of original instruction */
 #define UPROBE_COPY_INSN	0
@@ -707,12 +707,13 @@ static void put_uprobe(struct uprobe *uprobe)
 }
 
 /* Initialize hprobe as SRCU-protected "leased" uprobe */
-static void hprobe_init_leased(struct hprobe *hprobe, struct uprobe *uprobe, int srcu_idx)
+static void hprobe_init_leased(struct hprobe *hprobe, struct uprobe *uprobe,
+			       struct srcu_ctr __percpu *srcu_scp)
 {
 	WARN_ON(!uprobe);
 	hprobe->state = HPROBE_LEASED;
 	hprobe->uprobe = uprobe;
-	hprobe->srcu_idx = srcu_idx;
+	hprobe->srcu_scp = srcu_scp;
 }
 
 /* Initialize hprobe as refcounted ("stable") uprobe (uprobe can be NULL). */
@@ -720,7 +721,7 @@ static void hprobe_init_stable(struct hprobe *hprobe, struct uprobe *uprobe)
 {
 	hprobe->state = uprobe ? HPROBE_STABLE : HPROBE_GONE;
 	hprobe->uprobe = uprobe;
-	hprobe->srcu_idx = -1;
+	hprobe->srcu_scp = NULL;
 }
 
 /*
@@ -757,7 +758,7 @@ static void hprobe_finalize(struct hprobe *hprobe, enum hprobe_state hstate)
 {
 	switch (hstate) {
 	case HPROBE_LEASED:
-		__srcu_read_unlock(&uretprobes_srcu, hprobe->srcu_idx);
+		srcu_up_read_fast(&uretprobes_srcu, hprobe->srcu_scp);
 		break;
 	case HPROBE_STABLE:
 		put_uprobe(hprobe->uprobe);
@@ -829,7 +830,7 @@ static struct uprobe *hprobe_expire(struct hprobe *hprobe, bool get)
 		 */
 		if (try_cmpxchg(&hprobe->state, &hstate, uprobe ? HPROBE_STABLE : HPROBE_GONE)) {
 			/* We won the race, we are the ones to unlock SRCU */
-			__srcu_read_unlock(&uretprobes_srcu, hprobe->srcu_idx);
+			srcu_up_read_fast(&uretprobes_srcu, hprobe->srcu_scp);
 			return get && uprobe ? get_uprobe(uprobe) : uprobe;
 		}
 
@@ -2035,7 +2036,7 @@ static void ri_timer(struct timer_list *timer)
 	struct return_instance *ri;
 
 	/* SRCU protects uprobe from reuse for the cmpxchg() inside hprobe_expire(). */
-	guard(srcu)(&uretprobes_srcu);
+	guard(srcu_fast_updown)(&uretprobes_srcu);
 	/* RCU protects return_instance from freeing. */
 	guard(rcu)();
 
@@ -2132,7 +2133,7 @@ static int dup_utask(struct task_struct *t, struct uprobe_task *o_utask)
 	t->utask = n_utask;
 
 	/* protect uprobes from freeing, we'll need try_get_uprobe() them */
-	guard(srcu)(&uretprobes_srcu);
+	guard(srcu_fast_updown)(&uretprobes_srcu);
 
 	p = &n_utask->return_instances;
 	for (o = o_utask->return_instances; o; o = o->next) {
@@ -2244,8 +2245,8 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs,
 {
 	struct uprobe_task *utask = current->utask;
 	unsigned long orig_ret_vaddr, trampoline_vaddr;
+	struct srcu_ctr __percpu *srcu_scp;
 	bool chained;
-	int srcu_idx;
 
 	if (!get_xol_area())
 		goto free;
@@ -2283,8 +2284,12 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs,
 		orig_ret_vaddr = utask->return_instances->orig_ret_vaddr;
 	}
 
-	/* __srcu_read_lock() because SRCU lock survives switch to user space */
-	srcu_idx = __srcu_read_lock(&uretprobes_srcu);
+	/*
+	 * Use srcu_down_read_fast() because the SRCU lock survives a switch to
+	 * user space and can be unlocked from a different context by ri_timer()
+	 * or dup_utask().
+	 */
+	srcu_scp = srcu_down_read_fast(&uretprobes_srcu);
 
 	ri->func = instruction_pointer(regs);
 	ri->stack = user_stack_pointer(regs);
@@ -2293,7 +2298,7 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs,
 
 	utask->depth++;
 
-	hprobe_init_leased(&ri->hprobe, uprobe, srcu_idx);
+	hprobe_init_leased(&ri->hprobe, uprobe, srcu_scp);
 	ri->next = utask->return_instances;
 	rcu_assign_pointer(utask->return_instances, ri);