uprobes: Switch uretprobes_srcu to SRCU-fast-updown
Subsystems (6)
cover uprobes: Switch uretprobes_srcu to SRCU-fast-updown
uretprobes_srcu currently uses normal SRCU. Normal SRCU issues an smp_mb() on both srcu_read_lock() and srcu_read_unlock(), i.e. two full memory barriers per read-side critical section. For uretprobes this cost is paid on every uretprobe invocation: prepare_uretprobe() takes the read lock that is later dropped on the normal return path (hprobe_finalize()) or from ri_timer()/dup_utask() (hprobe_expire()).
Switch uretprobes_srcu to the SRCU-fast-updown flavor. SRCU-fast moves the read-side ordering off the reader and onto the (rare) grace-period
side: synchronize_srcu() rides on synchronize_rcu() instead of relying on reader-side smp_mb(). This is a good trade for uretprobes, where reader-side hits vastly outnumber grace periods (uprobe unregistration).
The updown variant (rather than plain SRCU-fast) is required because the read lock is acquired in prepare_uretprobe() on the way out to user space and is released only once the return instance is finalized -- from a different context than it was taken: the normal return path (uprobe_handle_trampoline() -> hprobe_finalize()), the timer callback, or the fork path (ri_timer()/dup_utask() -> hprobe_expire()). srcu_down_read_fast()/srcu_up_read_fast() are designed for this semaphore-like, cross-context pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry lockdep read-side tracking that would warn on it -- which is why the old code had to use the raw __srcu_read_lock() here. For the short, same-context sections in ri_timer() and dup_utask(), guard(srcu_fast_updown) is used instead, giving proper lockdep coverage.
Patch 1 adds the guard(srcu_fast_updown) definition, following the existing guard(srcu)/guard(srcu_fast) pattern.
Patch 2 does the uretprobes_srcu conversion.
Note
----
Only uretprobes_srcu is converted; the main uprobe readers (RB-tree lookup and consumer-list iteration) are deliberately left on RCU Tasks Trace. RCU Tasks Trace is already implemented on top of srcu_read_lock_fast(), so the reader-side cost is identical, and it has a nesting fast path that the uprobe -> sleepable-BPF-program call chain relies on (the BPF trampoline takes rcu_read_lock_trace() while uprobes already holds it; the nested acquire is just a counter bump). Converting those readers to a separate srcu_struct would turn one real + one nested lock into two real locks and lose that optimization for no reader-side gain. uretprobes_srcu is different: it uses normal SRCU (not Tasks Trace), its readers are long-lived and cross-context, and it genuinely benefits from dropping the per-reader barriers.
Puranjay Mohan (2):
srcu: Add lock guard for srcu_fast_updown flavor
uprobes: Switch uretprobes_srcu to SRCU-fast-updown
include/linux/srcu.h | 7 +++++++ include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 29 +++++++++++++++++------------ 3 files changed, 27 insertions(+), 14 deletions(-)
base-commit: 87bfe634b1193db90e5170e1ddbad04a63ef4501
Signature · 1 line
2.53.0-Meta
PATCH 1/2 srcu: Add lock guard for srcu_fast_updown flavor
Add a guard(srcu_fast_updown) definition for scoped SRCU-fast-updown read-side critical sections, following the existing pattern of guard(srcu) and guard(srcu_fast).
--- include/linux/srcu.h | 7 +++++++ 1 file changed, 7 insertions(+)
include/linux/srcu.h +7 -0
@@ -638,4 +638,11 @@ DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct, DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))#define class_srcu_fast_notrace_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, _T) +DEFINE_LOCK_GUARD_1(srcu_fast_updown, struct srcu_struct, + _T->scp =srcu_read_lock_fast_updown (_T->lock), +srcu_read_unlock_fast_updown (_T->lock, _T->scp), + struct srcu_ctr __percpu *scp) +DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_updown, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T)) +#define class_srcu_fast_updown_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_updown, _T) +#endif
Signature · 1 line
2.53.0-Meta
Re: [PATCH bpf-next 1/2] srcu: Add lock guard for srcu_fast_updown flavor
Quoted 5 lines · Puranjay Mohan
Add a
guard(srcu_fast_updown)definition for scoped SRCU-fast-updown read-side critical sections, following the existing pattern of guard(srcu) andguard(srcu_fast).Signed-off-by Puranjay Mohan <[email protected]>
(Or I can take it if you would prefer, but it might be easier and faster for you to send it along with the next patch.)
Quoted 23 lines
--- include/linux/srcu.h | 7 +++++++ 1 file changed, 7 insertions(+)include/linux/srcu.h +7
-0@@ -638,4 +638,11 @@ DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct, DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))#define class_srcu_fast_notrace_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, _T) +DEFINE_LOCK_GUARD_1(srcu_fast_updown, struct srcu_struct, + _T->scp =srcu_read_lock_fast_updown (_T->lock), +srcu_read_unlock_fast_updown (_T->lock, _T->scp), + struct srcu_ctr __percpu *scp) +DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_updown, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T)) +#define class_srcu_fast_updown_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_updown, _T) +#endif Signature · 1 line
2.53.0-Meta
Re: [PATCH bpf-next 1/2] srcu: Add lock guard for srcu_fast_updown flavor
In reply to Puranjay MohanQuoted 15 lines · Puranjay Mohan
Add a
guard(srcu_fast_updown)definition for scoped SRCU-fast-updown read-side critical sections, following the existing pattern of guard(srcu) andguard(srcu_fast).Signed-off-by Puranjay Mohan <[email protected]>--- include/linux/srcu.h | 7 +++++++ 1 file changed, 7 insertions(+)include/linux/srcu.h +0
-0
include/linux/srcu.h DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct, · line 646
Quoted 9 lines
include/linux/srcu.h +6
-0@@ -638,4 +638,11 @@ DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct, DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))#define class_srcu_fast_notrace_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, _T) +DEFINE_LOCK_GUARD_1(srcu_fast_updown, struct srcu_struct, + _T->scp =srcu_read_lock_fast_updown (_T->lock), +srcu_read_unlock_fast_updown (_T->lock, _T->scp), + struct srcu_ctr __percpu *scp) +DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_updown, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T)) +#define class_srcu_fast_updown_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_updown, _T)
Looks "obviously correct" to me. FWIW,
[tip: perf/core] srcu: Add lock guard for srcu_fast_updown flavor
In reply to Puranjay MohanThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: fa8a2d55139fb2e0b09e68c34b730adac92d18d6 Gitweb: https://git.kernel.org/tip/fa8a2d55139fb2e0b09e68c34b730adac92d18d6 Author: Puranjay Mohan <[email protected]> AuthorDate: Mon, 06 Jul 2026 10:27:41 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 03 Aug 2026 10:09:08 +02:00
srcu: Add lock guard for srcu_fast_updown flavor
Add a guard(srcu_fast_updown) definition for scoped SRCU-fast-updown read-side critical sections, following the existing pattern of guard(srcu) and guard(srcu_fast).
--- include/linux/srcu.h | 7 +++++++ 1 file changed, 7 insertions(+)
include/linux/srcu.h +7 -0
@@ -638,4 +638,11 @@ DEFINE_LOCK_GUARD_1(srcu_fast_notrace, struct srcu_struct, DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T))#define class_srcu_fast_notrace_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_notrace, _T) +DEFINE_LOCK_GUARD_1(srcu_fast_updown, struct srcu_struct, + _T->scp =srcu_read_lock_fast_updown (_T->lock), +srcu_read_unlock_fast_updown (_T->lock, _T->scp), + struct srcu_ctr __percpu *scp) +DECLARE_LOCK_GUARD_1_ATTRS(srcu_fast_updown, __acquires_shared(_T), __releases_shared(*(struct srcu_struct **)_T)) +#define class_srcu_fast_updown_constructor (_T) WITH_LOCK_GUARD_1_ATTRS(srcu_fast_updown, _T) +#endif
PATCH 2/2 uprobes: Switch uretprobes_srcu to SRCU-fast-updown
In reply to Puranjay Mohanuretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit.
Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (synchronize_rcu() instead of smp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.
The updown flavor is required because the SRCU read lock is taken in prepare_uretprobe() when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path via uprobe_handle_trampoline() -> hprobe_finalize(), or from ri_timer() (expiry) or dup_utask() (fork) via hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast() are designed for this acquire-here / release-elsewhere pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry the lockdep read-side tracking that would warn on it.
The short, same-context SRCU sections in ri_timer() and dup_utask() (which guard the uprobe against reuse across the hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown) for proper lockdep coverage.
--- include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 29 +++++++++++++++++------------ 2 files changed, 20 insertions(+), 14 deletions(-)
include/linux/uprobes.h +3 -2
@@ -25,6 +25,7 @@ struct mm_struct;struct inode ;struct notifier_block ;struct page ; +struct srcu_ctr ;/* * Allowed return values from uprobe consumer's handler callback
@@ -106,7 +107,7 @@ enum hprobe_state {* underlying uprobe is not guaranteed anymore. __UPROBE_DEAD is just an * internal marker and is handled transparently by hprobe_fetch() helper. * -* When uprobe is SRCU-protected, we also record srcu_idx value, necessary for +* When uprobe is SRCU-protected, we also record srcu_scp value, necessary for * SRCU unlocking. * * See hprobe_expire() and hprobe_fetch() for details of race-free uprobe
@@ -115,7 +116,7 @@ enum hprobe_state {*/ struct hprobe {enum hprobe_state state; -int srcu_idx; +struct srcu_ctr __percpu *srcu_scp;struct uprobe *uprobe; };
kernel/events/uprobes.c +17 -12
@@ -54,7 +54,7 @@ static struct mutex uprobes_mmap_mutex[UPROBES_HASH_SZ];DEFINE_STATIC_PERCPU_RWSEM (dup_mmap_sem);/* Covers return_instance's uprobe lifetime. */ -DEFINE_STATIC_SRCU (uretprobes_srcu); +DEFINE_STATIC_SRCU_FAST_UPDOWN (uretprobes_srcu);/* Have a copy of original instruction */ #define UPROBE_COPY_INSN 0
@@ -707,12 +707,13 @@ static void put_uprobe(struct uprobe *uprobe) }/* Initialize hprobe as SRCU-protected "leased" uprobe */ -static void hprobe_init_leased (struct hprobe *hprobe,struct uprobe *uprobe,int srcu_idx) +static void hprobe_init_leased (struct hprobe *hprobe,struct uprobe *uprobe, +struct srcu_ctr __percpu *srcu_scp) {WARN_ON (!uprobe); hprobe->state =HPROBE_LEASED ; hprobe->uprobe = uprobe; - hprobe->srcu_idx = srcu_idx; + hprobe->srcu_scp = srcu_scp; }/* Initialize hprobe as refcounted ("stable") uprobe (uprobe can be NULL). */
@@ -720,7 +721,7 @@ static void hprobe_init_stable(struct hprobe *hprobe, struct uprobe *uprobe) { hprobe->state = uprobe ?HPROBE_STABLE :HPROBE_GONE ; hprobe->uprobe = uprobe; - hprobe->srcu_idx =-1 ; + hprobe->srcu_scp =NULL ; }/*
@@ -757,7 +758,7 @@ static void hprobe_finalize(struct hprobe *hprobe, enum hprobe_state hstate) {switch (hstate) {case HPROBE_LEASED : -__srcu_read_unlock (&uretprobes_srcu, hprobe->srcu_idx); +srcu_up_read_fast (&uretprobes_srcu, hprobe->srcu_scp);break ;case HPROBE_STABLE :put_uprobe (hprobe->uprobe);
@@ -829,7 +830,7 @@ static struct uprobe *hprobe_expire(struct hprobe *hprobe, bool get)*/ if (try_cmpxchg (&hprobe->state, &hstate, uprobe ?HPROBE_STABLE :HPROBE_GONE )) {/* We won the race, we are the ones to unlock SRCU */ -__srcu_read_unlock (&uretprobes_srcu, hprobe->srcu_idx); +srcu_up_read_fast (&uretprobes_srcu, hprobe->srcu_scp);return get ?get_uprobe (uprobe) : uprobe; }
@@ -2045,7 +2046,7 @@ static void ri_timer(struct timer_list *timer)struct return_instance *ri;/* SRCU protects uprobe from reuse for the cmpxchg() inside hprobe_expire(). */ -guard (srcu)(&uretprobes_srcu); +guard (srcu_fast_updown)(&uretprobes_srcu);/* RCU protects return_instance from freeing. */ guard (rcu)();
@@ -2142,7 +2143,7 @@ static int dup_utask(struct task_struct *t, struct uprobe_task *o_utask) t->utask = n_utask;/* protect uprobes from freeing, we'll need try_get_uprobe() them */ -guard (srcu)(&uretprobes_srcu); +guard (srcu_fast_updown)(&uretprobes_srcu); p = &n_utask->return_instances;for (o = o_utask->return_instances; o; o = o->next) {
@@ -2254,8 +2255,8 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs, {struct uprobe_task *utask = current->utask;unsigned long orig_ret_vaddr, trampoline_vaddr; +struct srcu_ctr __percpu *srcu_scp;bool chained; -int srcu_idx;if (!get_xol_area ()) goto free;
@@ -2293,8 +2294,12 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs, orig_ret_vaddr = utask->return_instances->orig_ret_vaddr; } -/* __srcu_read_lock() because SRCU lock survives switch to user space */ - srcu_idx =__srcu_read_lock (&uretprobes_srcu); +/* +* Use srcu_down_read_fast() because the SRCU lock survives a switch to +* user space and can be unlocked from a different context by ri_timer() +* or dup_utask(). +*/ + srcu_scp =srcu_down_read_fast (&uretprobes_srcu); ri->func =instruction_pointer (regs); ri->stack =user_stack_pointer (regs);
@@ -2303,7 +2308,7 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs, utask->depth++; -hprobe_init_leased (&ri->hprobe, uprobe, srcu_idx); +hprobe_init_leased (&ri->hprobe, uprobe, srcu_scp); ri->next = utask->return_instances;rcu_assign_pointer (utask->return_instances, ri);
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
Pre-existing issues:
- [High] Concurrent hprobe_expire() and hprobe_consume() can cause a use-after-free on uprobe due to misunderstood SRCU semantics. --
commit 86c857f0a70ec453b9e38c36e230b36b91bef460
Author: Puranjay Mohan <[email protected]>
uprobes: Switch uretprobes_srcu to SRCU-fast-updown
uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit.
Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side.
Quoted 4 lines
kernel/events/uprobes.c +0
-0
[ ... ]
kernel/events/uprobes.c static void ri_timer(struct timer_list *timer) · line 2049
Quoted 6 lines
kernel/events/uprobes.c +1
-1@@ -2045,7 +2046,7 @@ static void ri_timer(struct timer_list *timer)struct return_instance *ri;/* SRCU protects uprobe from reuse for the cmpxchg() inside hprobe_expire(). */ -guard (srcu)(&uretprobes_srcu); +guard (srcu_fast_updown)(&uretprobes_srcu);
[Severity: High]
This isn't a bug introduced by this patch, but does the new SRCU read lock acquired here actually protect the uprobe from being freed?
If a uprobe is being unregistered, uprobe_free_deferred() calls call_srcu(&uretprobes_srcu, ...) to free the uprobe. This starts an SRCU grace period that waits only for pre-existing SRCU readers to finish.
Concurrently, if ri_timer() or dup_utask() takes this new SRCU read lock, the grace period will not wait for it because the new lock started after the grace period began.
If the traced task concurrently executes uprobe_handle_trampoline(), calls hprobe_consume() and hprobe_finalize(), and releases the original SRCU read lock, the grace period can complete and free the uprobe (via call_rcu_tasks_trace()).
When ri_timer() then proceeds to call hprobe_expire(), which does:
kernel/events/uprobes.c:hprobe_expire() {
...
case HPROBE_LEASED: {
struct uprobe *uprobe = try_get_uprobe(hprobe->uprobe);
...
}Could this lead to a use-after-free on the uprobe? The comment in hprobe_expire() seems to assume the new lock prevents the free:
/*
* Even if hprobe_consume() or another hprobe_expire() wins
* the state update race and unlocks SRCU from under us, we
* still have a guarantee that underyling uprobe won't be
* freed due to ongoing caller's SRCU lock region, so we can
* return it regardless.
*/
Is this analysis accurate?
Quoted 3 lines
kernel/events/uprobes.c +0
-0/* RCU protects return_instance from freeing. */ guard (rcu)();
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Quoted 31 lines · Puranjay Mohan
uretprobes_srcucurrently uses normal SRCU, which issues twosmp_mb()per read lock/unlock pair. This overhead is paid on every uretprobe hit.Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (
synchronize_rcu()instead ofsmp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.The updown flavor is required because the SRCU read lock is taken in
prepare_uretprobe()when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path viauprobe_handle_trampoline()->hprobe_finalize(), or fromri_timer()(expiry) ordup_utask()(fork) viahprobe_expire().srcu_down_read_fast()/srcu_up_read_fast()are designed for this acquire-here / release-elsewhere pattern and, unlike the same-contextsrcu_read_lock_fast()variant, do not carry the lockdep read-side tracking that would warn on it.The short, same-context SRCU sections in
ri_timer()anddup_utask()(which guard the uprobe against reuse across thehprobe_expire()cmpxchg) instead useguard(srcu_fast_updown)for proper lockdep coverage.
Quoted 6 lines
Signed-off-by Puranjay Mohan <[email protected]>--- include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 29 +++++++++++++++++------------ 2 files changed, 20 insertions(+), 14 deletions(-)
Very straightforward replacement and will improve performance, thanks! LGTM. Peter or Mingo, I assume you guys will want to route this through your tip tree, right?
[...]
Kumar Kartikeya Dwivedi
maintainerQuoted 39 lines · Andrii Nakryiko
Quoted 31 lines · Puranjay Mohan
uretprobes_srcucurrently uses normal SRCU, which issues twosmp_mb()per read lock/unlock pair. This overhead is paid on every uretprobe hit.Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (
synchronize_rcu()instead ofsmp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.The updown flavor is required because the SRCU read lock is taken in
prepare_uretprobe()when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path viauprobe_handle_trampoline()->hprobe_finalize(), or fromri_timer()(expiry) ordup_utask()(fork) viahprobe_expire().srcu_down_read_fast()/srcu_up_read_fast()are designed for this acquire-here / release-elsewhere pattern and, unlike the same-contextsrcu_read_lock_fast()variant, do not carry the lockdep read-side tracking that would warn on it.The short, same-context SRCU sections in
ri_timer()anddup_utask()(which guard the uprobe against reuse across thehprobe_expire()cmpxchg) instead useguard(srcu_fast_updown)for proper lockdep coverage.Quoted 6 lines
Signed-off-by Puranjay Mohan <[email protected]>--- include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 29 +++++++++++++++++------------ 2 files changed, 20 insertions(+), 14 deletions(-)
Quoted 6 lines
Very straightforward replacement and will improve performance, thanks! LGTM. Peter or Mingo, I assume you guys will want to route this through your tip tree, right?
Acked-by Andrii Nakryiko <[email protected]>
Hi Peter, Ingo,
Gentle ping for this.
Quoted 1 line
[...]
Quoted 45 lines · Kumar Kartikeya Dwivedi
Quoted 43 lines · Andrii Nakryiko
Quoted 31 lines · Puranjay Mohan
uretprobes_srcucurrently uses normal SRCU, which issues twosmp_mb()per read lock/unlock pair. This overhead is paid on every uretprobe hit.Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (
synchronize_rcu()instead ofsmp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.The updown flavor is required because the SRCU read lock is taken in
prepare_uretprobe()when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path viauprobe_handle_trampoline()->hprobe_finalize(), or fromri_timer()(expiry) ordup_utask()(fork) viahprobe_expire().srcu_down_read_fast()/srcu_up_read_fast()are designed for this acquire-here / release-elsewhere pattern and, unlike the same-contextsrcu_read_lock_fast()variant, do not carry the lockdep read-side tracking that would warn on it.The short, same-context SRCU sections in
ri_timer()anddup_utask()(which guard the uprobe against reuse across thehprobe_expire()cmpxchg) instead useguard(srcu_fast_updown)for proper lockdep coverage.Quoted 6 lines
Signed-off-by Puranjay Mohan <[email protected]>--- include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 29 +++++++++++++++++------------ 2 files changed, 20 insertions(+), 14 deletions(-)Very straightforward replacement and will improve performance, thanks! LGTM. Peter or Mingo, I assume you guys will want to route this through your tip tree, right?
Quoted 7 lines
Quoted 2 lines
Acked-by Andrii Nakryiko <[email protected]>Hi Peter, Ingo,
Gentle ping for this.
Another ping, please consider applying, thanks!
Quoted 2 lines
Quoted 1 line
[...]
Peter Zijlstra
maintainerQuoted 50 lines · Andrii Nakryiko
Quoted 48 lines · Kumar Kartikeya Dwivedi
Quoted 45 lines · Andrii Nakryiko
Quoted 31 lines · Puranjay Mohan
uretprobes_srcucurrently uses normal SRCU, which issues twosmp_mb()per read lock/unlock pair. This overhead is paid on every uretprobe hit.Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (
synchronize_rcu()instead ofsmp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.The updown flavor is required because the SRCU read lock is taken in
prepare_uretprobe()when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path viauprobe_handle_trampoline()->hprobe_finalize(), or fromri_timer()(expiry) ordup_utask()(fork) viahprobe_expire().srcu_down_read_fast()/srcu_up_read_fast()are designed for this acquire-here / release-elsewhere pattern and, unlike the same-contextsrcu_read_lock_fast()variant, do not carry the lockdep read-side tracking that would warn on it.The short, same-context SRCU sections in
ri_timer()anddup_utask()(which guard the uprobe against reuse across thehprobe_expire()cmpxchg) instead useguard(srcu_fast_updown)for proper lockdep coverage.Quoted 6 lines
Signed-off-by Puranjay Mohan <[email protected]>--- include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 29 +++++++++++++++++------------ 2 files changed, 20 insertions(+), 14 deletions(-)Very straightforward replacement and will improve performance, thanks! LGTM. Peter or Mingo, I assume you guys will want to route this through your tip tree, right?
Acked-by Andrii Nakryiko <[email protected]>
Quoted 6 lines
Quoted 4 lines
Hi Peter, Ingo,
Gentle ping for this.
Another ping, please consider applying, thanks!
Got them, thanks!
Quoted 31 lines · Puranjay Mohan
uretprobes_srcucurrently uses normal SRCU, which issues twosmp_mb()per read lock/unlock pair. This overhead is paid on every uretprobe hit.Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (
synchronize_rcu()instead ofsmp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.The updown flavor is required because the SRCU read lock is taken in
prepare_uretprobe()when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path viauprobe_handle_trampoline()->hprobe_finalize(), or fromri_timer()(expiry) ordup_utask()(fork) viahprobe_expire().srcu_down_read_fast()/srcu_up_read_fast()are designed for this acquire-here / release-elsewhere pattern and, unlike the same-contextsrcu_read_lock_fast()variant, do not carry the lockdep read-side tracking that would warn on it.The short, same-context SRCU sections in
ri_timer()anddup_utask()(which guard the uprobe against reuse across thehprobe_expire()cmpxchg) instead useguard(srcu_fast_updown)for proper lockdep coverage.
Quoted 5 lines
Signed-off-by Puranjay Mohan <[email protected]>--- include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 29 +++++++++++++++++------------ 2 files changed, 20 insertions(+), 14 deletions(-)
And the commit d3f52f53a562 ("srcu: Create an SRCU-fast-updown API") even mentions uretprobes as a likely user.
The following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 36c8b02c3fe9ba56d6f11ec055cd879bc49871f7 Gitweb: https://git.kernel.org/tip/36c8b02c3fe9ba56d6f11ec055cd879bc49871f7 Author: Puranjay Mohan <[email protected]> AuthorDate: Mon, 06 Jul 2026 10:27:42 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Mon, 03 Aug 2026 10:09:08 +02:00
uprobes: Switch uretprobes_srcu to SRCU-fast-updown
uretprobes_srcu currently uses normal SRCU, which issues two smp_mb() per read lock/unlock pair. This overhead is paid on every uretprobe hit.
Switch to SRCU-fast-updown, which eliminates the per-reader memory barriers by moving the ordering cost to the grace-period side (synchronize_rcu() instead of smp_mb()). This is acceptable because grace periods (uprobe unregistration) are infrequent compared to reader-side uretprobe hits.
The updown flavor is required because the SRCU read lock is taken in prepare_uretprobe() when a return instance is created and is held until that return instance is finalized. The traced thread returns to user space in between, so the lock is inherently released in a different context from where it was acquired: on the normal return path via uprobe_handle_trampoline() -> hprobe_finalize(), or from ri_timer() (expiry) or dup_utask() (fork) via hprobe_expire(). srcu_down_read_fast() / srcu_up_read_fast() are designed for this acquire-here / release-elsewhere pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry the lockdep read-side tracking that would warn on it.
The short, same-context SRCU sections in ri_timer() and dup_utask() (which guard the uprobe against reuse across the hprobe_expire() cmpxchg) instead use guard(srcu_fast_updown) for proper lockdep coverage.
--- include/linux/uprobes.h | 5 +++-- kernel/events/uprobes.c | 29 +++++++++++++++++------------ 2 files changed, 20 insertions(+), 14 deletions(-)
include/linux/uprobes.h +3 -2
@@ -25,6 +25,7 @@ struct mm_struct;struct inode ;struct notifier_block ;struct page ; +struct srcu_ctr ;/* * Allowed return values from uprobe consumer's handler callback
@@ -106,7 +107,7 @@ enum hprobe_state {* underlying uprobe is not guaranteed anymore. __UPROBE_DEAD is just an * internal marker and is handled transparently by hprobe_fetch() helper. * -* When uprobe is SRCU-protected, we also record srcu_idx value, necessary for +* When uprobe is SRCU-protected, we also record srcu_scp value, necessary for * SRCU unlocking. * * See hprobe_expire() and hprobe_fetch() for details of race-free uprobe
@@ -115,7 +116,7 @@ enum hprobe_state {*/ struct hprobe {enum hprobe_state state; -int srcu_idx; +struct srcu_ctr __percpu *srcu_scp;struct uprobe *uprobe; };
kernel/events/uprobes.c +17 -12
@@ -54,7 +54,7 @@ static struct mutex uprobes_mmap_mutex[UPROBES_HASH_SZ];DEFINE_STATIC_PERCPU_RWSEM (dup_mmap_sem);/* Covers return_instance's uprobe lifetime. */ -DEFINE_STATIC_SRCU (uretprobes_srcu); +DEFINE_STATIC_SRCU_FAST_UPDOWN (uretprobes_srcu);/* Have a copy of original instruction */ #define UPROBE_COPY_INSN 0
@@ -707,12 +707,13 @@ static void put_uprobe(struct uprobe *uprobe) }/* Initialize hprobe as SRCU-protected "leased" uprobe */ -static void hprobe_init_leased (struct hprobe *hprobe,struct uprobe *uprobe,int srcu_idx) +static void hprobe_init_leased (struct hprobe *hprobe,struct uprobe *uprobe, +struct srcu_ctr __percpu *srcu_scp) {WARN_ON (!uprobe); hprobe->state =HPROBE_LEASED ; hprobe->uprobe = uprobe; - hprobe->srcu_idx = srcu_idx; + hprobe->srcu_scp = srcu_scp; }/* Initialize hprobe as refcounted ("stable") uprobe (uprobe can be NULL). */
@@ -720,7 +721,7 @@ static void hprobe_init_stable(struct hprobe *hprobe, struct uprobe *uprobe) { hprobe->state = uprobe ?HPROBE_STABLE :HPROBE_GONE ; hprobe->uprobe = uprobe; - hprobe->srcu_idx =-1 ; + hprobe->srcu_scp =NULL ; }/*
@@ -757,7 +758,7 @@ static void hprobe_finalize(struct hprobe *hprobe, enum hprobe_state hstate) {switch (hstate) {case HPROBE_LEASED : -__srcu_read_unlock (&uretprobes_srcu, hprobe->srcu_idx); +srcu_up_read_fast (&uretprobes_srcu, hprobe->srcu_scp);break ;case HPROBE_STABLE :put_uprobe (hprobe->uprobe);
@@ -829,7 +830,7 @@ static struct uprobe *hprobe_expire(struct hprobe *hprobe, bool get)*/ if (try_cmpxchg (&hprobe->state, &hstate, uprobe ?HPROBE_STABLE :HPROBE_GONE )) {/* We won the race, we are the ones to unlock SRCU */ -__srcu_read_unlock (&uretprobes_srcu, hprobe->srcu_idx); +srcu_up_read_fast (&uretprobes_srcu, hprobe->srcu_scp);return get && uprobe ?get_uprobe (uprobe) : uprobe; }
@@ -2035,7 +2036,7 @@ static void ri_timer(struct timer_list *timer)struct return_instance *ri;/* SRCU protects uprobe from reuse for the cmpxchg() inside hprobe_expire(). */ -guard (srcu)(&uretprobes_srcu); +guard (srcu_fast_updown)(&uretprobes_srcu);/* RCU protects return_instance from freeing. */ guard (rcu)();
@@ -2132,7 +2133,7 @@ static int dup_utask(struct task_struct *t, struct uprobe_task *o_utask) t->utask = n_utask;/* protect uprobes from freeing, we'll need try_get_uprobe() them */ -guard (srcu)(&uretprobes_srcu); +guard (srcu_fast_updown)(&uretprobes_srcu); p = &n_utask->return_instances;for (o = o_utask->return_instances; o; o = o->next) {
@@ -2244,8 +2245,8 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs, {struct uprobe_task *utask = current->utask;unsigned long orig_ret_vaddr, trampoline_vaddr; +struct srcu_ctr __percpu *srcu_scp;bool chained; -int srcu_idx;if (!get_xol_area ()) goto free;
@@ -2283,8 +2284,12 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs, orig_ret_vaddr = utask->return_instances->orig_ret_vaddr; } -/* __srcu_read_lock() because SRCU lock survives switch to user space */ - srcu_idx =__srcu_read_lock (&uretprobes_srcu); +/* +* Use srcu_down_read_fast() because the SRCU lock survives a switch to +* user space and can be unlocked from a different context by ri_timer() +* or dup_utask(). +*/ + srcu_scp =srcu_down_read_fast (&uretprobes_srcu); ri->func =instruction_pointer (regs); ri->stack =user_stack_pointer (regs);
@@ -2293,7 +2298,7 @@ static void prepare_uretprobe(struct uprobe *uprobe, struct pt_regs *regs, utask->depth++; -hprobe_init_leased (&ri->hprobe, uprobe, srcu_idx); +hprobe_init_leased (&ri->hprobe, uprobe, srcu_scp); ri->next = utask->return_instances;rcu_assign_pointer (utask->return_instances, ri);