Skip to content

uprobes: Switch uretprobes_srcu to SRCU-fast-updown

Puranjay Mohan bpf linux-perf-users linux-trace-kernel lkml rcu 1 version Applied

Status from this message: The following commit has been merged into the perf/core branch of tip:

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest2 patches 7 repliesApplied

Patches in v1 Applied

#Subject and filesSizeRepliesApplied
1/2srcu: Add lock guard for srcu_fast_updown flavorinclude/linux/srcu.h+7 -02 replies
2/2uprobes: Switch uretprobes_srcu to SRCU-fast-updowninclude/linux/uprobes.h kernel/events/uprobes.c+20 -145 replies

Reviews trailers per patch

All 2 patches have a Reviewed-by or Acked-by.

PatchPaul E. McKenneyAndrii NakryikoOleg Nesterov
1/2R-b R-b
2/2A-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v1 thread

Cover letter [PATCH bpf-next 0/2] uprobes: Switch uretprobes_srcu to SRCU-fast-updown

uretprobes_srcu currently uses normal SRCU. Normal SRCU issues an smp_mb() on both srcu_read_lock() and srcu_read_unlock(), i.e. two full memory barriers per read-side critical section. For uretprobes this cost is paid on every uretprobe invocation: prepare_uretprobe() takes the read lock that is later dropped on the normal return path (hprobe_finalize()) or from ri_timer()/dup_utask() (hprobe_expire()).

Switch uretprobes_srcu to the SRCU-fast-updown flavor. SRCU-fast moves the read-side ordering off the reader and onto the (rare) grace-period
side: synchronize_srcu() rides on synchronize_rcu() instead of relying on reader-side smp_mb(). This is a good trade for uretprobes, where reader-side hits vastly outnumber grace periods (uprobe unregistration).

The updown variant (rather than plain SRCU-fast) is required because the read lock is acquired in prepare_uretprobe() on the way out to user space and is released only once the return instance is finalized -- from a different context than it was taken: the normal return path (uprobe_handle_trampoline() -> hprobe_finalize()), the timer callback, or the fork path (ri_timer()/dup_utask() -> hprobe_expire()). srcu_down_read_fast()/srcu_up_read_fast() are designed for this semaphore-like, cross-context pattern and, unlike the same-context srcu_read_lock_fast() variant, do not carry lockdep read-side tracking that would warn on it -- which is why the old code had to use the raw __srcu_read_lock() here. For the short, same-context sections in ri_timer() and dup_utask(), guard(srcu_fast_updown) is used instead, giving proper lockdep coverage.

Patch 1 adds the guard(srcu_fast_updown) definition, following the existing guard(srcu)/guard(srcu_fast) pattern.
Patch 2 does the uretprobes_srcu conversion.

Note
----
Only uretprobes_srcu is converted; the main uprobe readers (RB-tree lookup and consumer-list iteration) are deliberately left on RCU Tasks Trace. RCU Tasks Trace is already implemented on top of srcu_read_lock_fast(), so the reader-side cost is identical, and it has a nesting fast path that the uprobe -> sleepable-BPF-program call chain relies on (the BPF trampoline takes rcu_read_lock_trace() while uprobes already holds it; the nested acquire is just a counter bump). Converting those readers to a separate srcu_struct would turn one real + one nested lock into two real locks and lose that optimization for no reader-side gain. uretprobes_srcu is different: it uses normal SRCU (not Tasks Trace), its readers are long-lived and cross-context, and it genuinely benefits from dropping the per-reader barriers.

Puranjay Mohan (2):
srcu: Add lock guard for srcu_fast_updown flavor
uprobes: Switch uretprobes_srcu to SRCU-fast-updown

 include/linux/srcu.h    |  7 +++++++
 include/linux/uprobes.h |  5 +++--
 kernel/events/uprobes.c | 29 +++++++++++++++++------------
 3 files changed, 27 insertions(+), 14 deletions(-)

base-commit: 87bfe634b1193db90e5170e1ddbad04a63ef4501

Signature · 1 line
2.53.0-Meta