uprobes,mm: speculative lockless VMA-to-uprobe lookup
Subsystems (10)
Andrii Nakryiko
maintainerv2 cover uprobes,mm: speculative lockless VMA-to-uprobe lookup
Implement speculative (lockless) resolution of VMA to inode to uprobe, bypassing the need to take mmap_lock for reads, if possible. Patch #1 by Suren adds mm_struct helpers that help detect whether mm_struct were changed, which is used by uprobe logic to validate that speculative results can be trusted after all the lookup logic results in a valid uprobe instance. Patch #2 follows to make mm_lock_seq into 64-bit counter (on 64-bit architectures).
Patch #3 adds back RCU-delayed freeing for FMODE_BACKING files, which is necessary to make speculation safe to access struct file's memory in any possible situation.
Patch #4 is a simplification to uprobe VMA flag checking, suggested by Oleg.
And, finally, patch #5 is the speculative VMA-to-uprobe resolution logic. See corresponding patch for details and benchmarking results.
v1->v2:
- adjusted vma_end_write_all() comment to point out it should never be called manually now, but I wasn't sure how ACQUIRE/RELEASE comments should be reworded (previously requested by Jann), so I'd appreciate some help there (Jann);
- int -> long change for mm_lock_seq, as agreed at LPC2024 (Jann, Suren, Liam);
- kfree_rcu_mightsleep() for FMODE_BACKING (Suren, Christian);
- vm_flags simplification in find_active_uprobe_rcu() and find_active_uprobe_speculative() (Oleg);
- guard(rcu)() simplified find_active_uprobe_speculative() implementation.
Andrii Nakryiko (4):
mm: switch to 64-bit mm_lock_seq/vm_lock_seq on 64-bit architectures
fs: add back RCU-delayed freeing of FMODE_BACKING file
uprobes: simplify find_active_uprobe_rcu() VMA checks
uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution
Suren Baghdasaryan (1):
mm: introduce mmap_lock_speculation_{start|end}
fs/file_table.c | 2 +- include/linux/mm.h | 6 ++-- include/linux/mm_types.h | 7 ++-- include/linux/mmap_lock.h | 72 ++++++++++++++++++++++++++++++++------- kernel/events/uprobes.c | 46 ++++++++++++++++++++++++- kernel/fork.c | 3 -- 6 files changed, 114 insertions(+), 22 deletions(-)
Signature · 1 line
2.43.5
Andrii Nakryiko
maintainerPATCH v2 1/5 mm: introduce mmap_lock_speculation_{start|end}
From: Suren Baghdasaryan <[email protected]>
Add helper functions to speculatively perform operations without read-locking mmap_lock, expecting that mmap_lock will not be write-locked and mm is not modified from under us.
--- include/linux/mm_types.h | 3 ++ include/linux/mmap_lock.h | 72 ++++++++++++++++++++++++++++++++------- kernel/fork.c | 3 -- 3 files changed, 63 insertions(+), 15 deletions(-)
include/linux/mm_types.h +3 -0
@@ -876,6 +876,9 @@ struct mm_struct {* Roughly speaking, incrementing the sequence number is * equivalent to releasing locks on VMAs; reading the sequence * number can be part of taking a read lock on a VMA. +* Incremented every time mmap_lock is write-locked/unlocked. +* Initialized to 0, therefore odd values indicate mmap_lock +* is write-locked and even values that it's released. * * Can be modified under write mmap_lock using RELEASE * semantics.
include/linux/mmap_lock.h +60 -12
111 lines not on this page · raw message
kernel/fork.c +0 -3
@@ -1259,9 +1259,6 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,seqcount_init (&mm->write_protect_seq);mmap_init_lock (mm);INIT_LIST_HEAD (&mm->mmlist); -#ifdef CONFIG_PER_VMA_LOCK - mm->mm_lock_seq =0 ; -#endif mm_pgtables_bytes_init (mm); mm->map_count =0 ; mm->locked_vm =0 ;
Signature · 1 line
2.43.5
Re: [PATCH v2 tip/perf/core 1/5] mm: introduce mmap_lock_speculation_{start|end}
+cc Liam, sorry, seems like I forgot to add you to the entire patch set on initial submission.
Quoted 11 lines · Andrii Nakryiko
From: Suren Baghdasaryan <[email protected]>
Add helper functions to speculatively perform operations without read-locking
mmap_lock, expecting thatmmap_lockwill not be write-locked and mm is not modified from under us.Suggested-by Peter Zijlstra <[email protected]>Signed-off-by Suren Baghdasaryan <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>
Quoted 6 lines
--- include/linux/mm_types.h | 3 ++ include/linux/mmap_lock.h | 72 ++++++++++++++++++++++++++++++++------- kernel/fork.c | 3 -- 3 files changed, 63 insertions(+), 15 deletions(-)
Are memory-management folks OK with these changes? It would be nice to get some acks, if so, and I'd include it into respin, fixing minor things in uprobe patches. Thank you!
Note, while this is initially needed for uprobe functionality, having an ability to quickly change whether mm_struct changed inbetween some speculative querying is generally useful functionality, and I believe it would help eliminating mmap_lock usage from /proc/PID/maps code. Which is a great outcome for everyone, as that mmap_lock can be quite disruptive in production workloads.
So please don't see it as some irrelevant uprobe-related requirement, the applicability of this is much wider.
[...]
Andrii Nakryiko
maintainerPATCH v2 2/5 mm: switch to 64-bit mm_lock_seq/vm_lock_seq on 64-bit architectures
In reply to Andrii NakryikoTo increase mm->mm_lock_seq robustness, switch it from int to long, so that it's a 64-bit counter on 64-bit systems and we can stop worrying about it wrapping around in just ~4 billion iterations. Same goes for VMA's matching vm_lock_seq, which is derived from mm_lock_seq.
I didn't use __u64 outright to keep 32-bit architectures unaffected, but if it seems important enough, I have nothing against using __u64.
--- include/linux/mm.h | 6 +++--- include/linux/mm_types.h | 4 ++-- include/linux/mmap_lock.h | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-)
include/linux/mm.h +3 -3
@@ -716,7 +716,7 @@ static inline void vma_end_read(struct vm_area_struct *vma) }/* WARNING! Can only be used if mmap_lock is expected to be write-locked */ -static bool __is_vma_write_locked (struct vm_area_struct *vma,int *mm_lock_seq) +static bool __is_vma_write_locked (struct vm_area_struct *vma,long *mm_lock_seq) {mmap_assert_write_locked (vma->vm_mm);
@@ -735,7 +735,7 @@ static bool __is_vma_write_locked(struct vm_area_struct *vma, int *mm_lock_seq)*/ static inline void vma_start_write (struct vm_area_struct *vma) { -int mm_lock_seq; +long mm_lock_seq;if (__is_vma_write_locked (vma, &mm_lock_seq))return ;
@@ -753,7 +753,7 @@ static inline void vma_start_write(struct vm_area_struct *vma)static inline void vma_assert_write_locked (struct vm_area_struct *vma) { -int mm_lock_seq; +long mm_lock_seq;VM_BUG_ON_VMA (!__is_vma_write_locked (vma, &mm_lock_seq), vma); }
include/linux/mm_types.h +2 -2
@@ -705,7 +705,7 @@ struct vm_area_struct {* counter reuse can only lead to occasional unnecessary use of the * slowpath. */ -int vm_lock_seq; +long vm_lock_seq;struct vma_lock *vm_lock;#endif
@@ -887,7 +887,7 @@ struct mm_struct {* Can be read with ACQUIRE semantics if not holding write * mmap_lock. */ -int mm_lock_seq; +long mm_lock_seq;#endif
include/linux/mmap_lock.h +2 -2
@@ -105,7 +105,7 @@ static inline void inc_mm_lock_seq(struct mm_struct *mm, bool acquire) } } -static inline bool mmap_lock_speculation_start (struct mm_struct *mm,int *seq) +static inline bool mmap_lock_speculation_start (struct mm_struct *mm,long *seq) {/* Pairs with RELEASE semantics in inc_mm_lock_seq(). */ *seq =smp_load_acquire (&mm->mm_lock_seq);
@@ -113,7 +113,7 @@ static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq)return (*seq &1 ) ==0 ; } -static inline bool mmap_lock_speculation_end (struct mm_struct *mm,int seq) +static inline bool mmap_lock_speculation_end (struct mm_struct *mm,long seq) {/* Pairs with ACQUIRE semantics in inc_mm_lock_seq(). */ smp_rmb ();
Signature · 1 line
2.43.5
Andrii Nakryiko
maintainerPATCH v2 3/5 fs: add back RCU-delayed freeing of FMODE_BACKING file
In reply to Andrii Nakryiko6cf41fcfe099 ("backing file: free directly") switched FMODE_BACKING files to direct freeing as back then there were no use cases requiring RCU protected access to such files.
Now, with speculative lockless VMA-to-uprobe lookup logic, we do need to have a guarantee that struct file memory is not going to be freed from under us during speculative check. So add back RCU-delayed freeing logic.
We use headless kfree_rcu_mightsleep() variant, as file_free() is only called for FMODE_BACKING files in might_sleep() context.
--- fs/file_table.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
fs/file_table.c +1 -1
@@ -68,7 +68,7 @@ static inline void file_free(struct file *f)put_cred (f->f_cred);if (unlikely (f->f_mode &FMODE_BACKING )) {path_put (backing_file_user_path (f)); -kfree (backing_file (f)); +kfree_rcu_mightsleep (backing_file (f)); }else {kmem_cache_free (filp_cachep, f); }
Signature · 1 line
2.43.5
Re: [PATCH v2 tip/perf/core 3/5] fs: add back RCU-delayed freeing of FMODE_BACKING file
Quoted 9 lines · Andrii Nakryiko
6cf41fcfe099("backing file: free directly") switchedFMODE_BACKINGfiles to direct freeing as back then there were no use cases requiring RCU protected access to such files.Now, with speculative lockless VMA-to-uprobe lookup logic, we do need to have a guarantee that struct file memory is not going to be freed from under us during speculative check. So add back RCU-delayed freeing logic.
Quoted 8 lines
We use headless
kfree_rcu_mightsleep()variant, asfile_free()is only called forFMODE_BACKINGfiles inmight_sleep()context.Suggested-by Suren Baghdasaryan <[email protected]>Cc Christian Brauner <[email protected]>Cc Amir Goldstein <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>---
Christian Brauner
maintainerRe: [PATCH v2 tip/perf/core 3/5] fs: add back RCU-delayed freeing of FMODE_BACKING file
Quoted 13 lines · Christian Brauner
Quoted 12 lines · Andrii Nakryiko
6cf41fcfe099("backing file: free directly") switchedFMODE_BACKINGfiles to direct freeing as back then there were no use cases requiring RCU protected access to such files.Now, with speculative lockless VMA-to-uprobe lookup logic, we do need to have a guarantee that struct file memory is not going to be freed from under us during speculative check. So add back RCU-delayed freeing logic.
We use headless
kfree_rcu_mightsleep()variant, asfile_free()is only called forFMODE_BACKINGfiles inmight_sleep()context.
Quoted 7 lines
Quoted 5 lines
Suggested-by Suren Baghdasaryan <[email protected]>Cc Christian Brauner <[email protected]>Cc Amir Goldstein <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>---
Reviewed-by Christian Brauner <[email protected]>
Fwiw, I have another patch series for files that I'm testing that will require me to switch FMODE_BACKING to a SLAB_TYPSAFE_BY_RCU cache. That shouldn't matter for your use-case though.
Re: [PATCH v2 tip/perf/core 3/5] fs: add back RCU-delayed freeing of FMODE_BACKING file
Quoted 21 lines · Christian Brauner
Quoted 19 lines · Christian Brauner
Quoted 17 lines · Andrii Nakryiko
6cf41fcfe099("backing file: free directly") switchedFMODE_BACKINGfiles to direct freeing as back then there were no use cases requiring RCU protected access to such files.Now, with speculative lockless VMA-to-uprobe lookup logic, we do need to have a guarantee that struct file memory is not going to be freed from under us during speculative check. So add back RCU-delayed freeing logic.
We use headless
kfree_rcu_mightsleep()variant, asfile_free()is only called forFMODE_BACKINGfiles inmight_sleep()context.Suggested-by Suren Baghdasaryan <[email protected]>Cc Christian Brauner <[email protected]>Cc Amir Goldstein <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>---
Quoted 5 lines
Quoted 1 line
Reviewed-by Christian Brauner <[email protected]>Fwiw, I have another patch series for files that I'm testing that will require me to switch
FMODE_BACKINGto aSLAB_TYPSAFE_BY_RCUcache. That shouldn't matter for your use-case though.
Correct, we assume SLAB_TYPESAFE_BY_RCU semantics for the common case anyways. But hopefully my change won't cause major merge conflicts with your patch set.
Christian Brauner
maintainerRe: [PATCH v2 tip/perf/core 3/5] fs: add back RCU-delayed freeing of FMODE_BACKING file
Quoted 24 lines · Andrii Nakryiko
Quoted 23 lines · Christian Brauner
Quoted 20 lines · Christian Brauner
Quoted 9 lines · Andrii Nakryiko
6cf41fcfe099("backing file: free directly") switchedFMODE_BACKINGfiles to direct freeing as back then there were no use cases requiring RCU protected access to such files.Now, with speculative lockless VMA-to-uprobe lookup logic, we do need to have a guarantee that struct file memory is not going to be freed from under us during speculative check. So add back RCU-delayed freeing logic.
Quoted 8 lines
We use headless
kfree_rcu_mightsleep()variant, asfile_free()is only called forFMODE_BACKINGfiles inmight_sleep()context.Suggested-by Suren Baghdasaryan <[email protected]>Cc Christian Brauner <[email protected]>Cc Amir Goldstein <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>---
Reviewed-by Christian Brauner <[email protected]>
Quoted 7 lines
Quoted 3 lines
Fwiw, I have another patch series for files that I'm testing that will require me to switch
FMODE_BACKINGto aSLAB_TYPSAFE_BY_RCUcache. That shouldn't matter for your use-case though.Correct, we assume
SLAB_TYPESAFE_BY_RCUsemantics for the common case anyways. But hopefully my change won't cause major merge conflicts with your patch set.
Please drop this patch and pull the following tag which adds SLAB_TYPE_SAFE_BY_RCU protection for FMODE_BACKING files aligning them with regular files lifetime (even though not needed). The branch the tag is based on is stable and won't change anymore:
git pull -S git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git tags/vfs-6.13.for-bpf.file
Re: [PATCH v2 tip/perf/core 3/5] fs: add back RCU-delayed freeing of FMODE_BACKING file
Quoted 34 lines · Christian Brauner
Quoted 24 lines · Andrii Nakryiko
Quoted 23 lines · Christian Brauner
Quoted 20 lines · Christian Brauner
Quoted 9 lines · Andrii Nakryiko
6cf41fcfe099("backing file: free directly") switchedFMODE_BACKINGfiles to direct freeing as back then there were no use cases requiring RCU protected access to such files.Now, with speculative lockless VMA-to-uprobe lookup logic, we do need to have a guarantee that struct file memory is not going to be freed from under us during speculative check. So add back RCU-delayed freeing logic.
Quoted 8 lines
We use headless
kfree_rcu_mightsleep()variant, asfile_free()is only called forFMODE_BACKINGfiles inmight_sleep()context.Suggested-by Suren Baghdasaryan <[email protected]>Cc Christian Brauner <[email protected]>Cc Amir Goldstein <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>---
Reviewed-by Christian Brauner <[email protected]>Quoted 7 lines
Quoted 3 lines
Fwiw, I have another patch series for files that I'm testing that will require me to switch
FMODE_BACKINGto aSLAB_TYPSAFE_BY_RCUcache. That shouldn't matter for your use-case though.Correct, we assume
SLAB_TYPESAFE_BY_RCUsemantics for the common case anyways. But hopefully my change won't cause major merge conflicts with your patch set.
Quoted 6 lines
Please drop this patch and pull the following tag which adds
SLAB_TYPE_SAFE_BY_RCUprotection forFMODE_BACKINGfiles aligning them with regular files lifetime (even though not needed). The branch the tag is based on is stable and won't change anymore:git pull -S
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.gittags/vfs-6.13.for-bpf.file
Ok, will drop. It will on Peter to pull this tag into tip/perf/core, but I'll mention all this in the cover letter (and will pull locally for testing, of course). Thanks.
Andrii Nakryiko
maintainerPATCH v2 4/5 uprobes: simplify find_active_uprobe_rcu() VMA checks
In reply to Andrii NakryikoAt the point where find_active_uprobe_rcu() is used we know that VMA in question has triggered software breakpoint, so we don't need to validate vma->vm_flags. Keep only vma->vm_file NULL check.
--- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/uprobes.c +1 -1
@@ -2091,7 +2091,7 @@ static struct uprobe *find_active_uprobe_rcu(unsigned long bp_vaddr, int *is_swbmmap_read_lock (mm); vma =vma_lookup (mm, bp_vaddr);if (vma) { -if (valid_vma (vma, false)) { +if (vma->vm_file) {struct inode *inode =file_inode (vma->vm_file);loff_t offset =vaddr_to_offset (vma, bp_vaddr);
Signature · 1 line
2.43.5
Re: [PATCH v2 tip/perf/core 4/5] uprobes: simplify find_active_uprobe_rcu() VMA checks
Quoted 10 lines · Andrii Nakryiko
At the point where
find_active_uprobe_rcu()is used we know that VMA in question has triggered software breakpoint, so we don't need to validatevma->vm_flags. Keep onlyvma->vm_fileNULL check.Suggested-by Oleg Nesterov <[email protected]>Signed-off-by Andrii Nakryiko <[email protected]>--- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 16 lines
kernel/events/uprobes.c +1
-1@@ -2091,7 +2091,7 @@ static struct uprobe *find_active_uprobe_rcu(unsigned long bp_vaddr, int *is_swbmmap_read_lock (mm); vma =vma_lookup (mm, bp_vaddr);if (vma) { -if (valid_vma (vma, false)) { +if (vma->vm_file) {struct inode *inode =file_inode (vma->vm_file);loff_t offset =vaddr_to_offset (vma, bp_vaddr);Signature · 1 line
2.43.5
Andrii Nakryiko
maintainerPATCH v2 5/5 uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution
In reply to Andrii NakryikoGiven filp_cachep is marked SLAB_TYPESAFE_BY_RCU (and FMODE_BACKING files, a special case, now goes through RCU-delated freeing), we can safely access vma->vm_file->f_inode field locklessly under just rcu_read_lock() protection, which enables looking up uprobe from uprobes_tree completely locklessly and speculatively without the need to acquire mmap_lock for reads. In most cases, anyway, assuming that there are no parallel mm and/or VMA modifications. The underlying struct file's memory won't go away from under us (even if struct file can be reused in the meantime).
We rely on newly added mmap_lock_speculation_{start,end}() helpers to validate that mm_struct stays intact for entire duration of this speculation. If not, we fall back to mmap_lock-protected lookup. The speculative logic is written in such a way that it will safely handle any garbage values that might be read from vma or file structs.
Benchmarking results speak for themselves.
BEFORE (latest tip/perf/core)
=============================
uprobe-nop ( 1 cpus): 3.384 ± 0.004M/s ( 3.384M/s/cpu) uprobe-nop ( 2 cpus): 5.456 ± 0.005M/s ( 2.728M/s/cpu) uprobe-nop ( 3 cpus): 7.863 ± 0.015M/s ( 2.621M/s/cpu) uprobe-nop ( 4 cpus): 9.442 ± 0.008M/s ( 2.360M/s/cpu) uprobe-nop ( 5 cpus): 11.036 ± 0.013M/s ( 2.207M/s/cpu) uprobe-nop ( 6 cpus): 10.884 ± 0.019M/s ( 1.814M/s/cpu) uprobe-nop ( 7 cpus): 7.897 ± 0.145M/s ( 1.128M/s/cpu) uprobe-nop ( 8 cpus): 10.021 ± 0.128M/s ( 1.253M/s/cpu) uprobe-nop (10 cpus): 9.932 ± 0.170M/s ( 0.993M/s/cpu) uprobe-nop (12 cpus): 8.369 ± 0.056M/s ( 0.697M/s/cpu) uprobe-nop (14 cpus): 8.678 ± 0.017M/s ( 0.620M/s/cpu) uprobe-nop (16 cpus): 7.392 ± 0.003M/s ( 0.462M/s/cpu) uprobe-nop (24 cpus): 5.326 ± 0.178M/s ( 0.222M/s/cpu) uprobe-nop (32 cpus): 5.426 ± 0.059M/s ( 0.170M/s/cpu) uprobe-nop (40 cpus): 5.262 ± 0.070M/s ( 0.132M/s/cpu) uprobe-nop (48 cpus): 6.121 ± 0.010M/s ( 0.128M/s/cpu) uprobe-nop (56 cpus): 6.252 ± 0.035M/s ( 0.112M/s/cpu) uprobe-nop (64 cpus): 7.644 ± 0.023M/s ( 0.119M/s/cpu) uprobe-nop (72 cpus): 7.781 ± 0.001M/s ( 0.108M/s/cpu) uprobe-nop (80 cpus): 8.992 ± 0.048M/s ( 0.112M/s/cpu)
AFTER
=====
uprobe-nop ( 1 cpus): 3.534 ± 0.033M/s ( 3.534M/s/cpu) uprobe-nop ( 2 cpus): 6.701 ± 0.007M/s ( 3.351M/s/cpu) uprobe-nop ( 3 cpus): 10.031 ± 0.007M/s ( 3.344M/s/cpu) uprobe-nop ( 4 cpus): 13.003 ± 0.012M/s ( 3.251M/s/cpu) uprobe-nop ( 5 cpus): 16.274 ± 0.006M/s ( 3.255M/s/cpu) uprobe-nop ( 6 cpus): 19.563 ± 0.024M/s ( 3.261M/s/cpu) uprobe-nop ( 7 cpus): 22.696 ± 0.054M/s ( 3.242M/s/cpu) uprobe-nop ( 8 cpus): 24.534 ± 0.010M/s ( 3.067M/s/cpu) uprobe-nop (10 cpus): 30.475 ± 0.117M/s ( 3.047M/s/cpu) uprobe-nop (12 cpus): 33.371 ± 0.017M/s ( 2.781M/s/cpu) uprobe-nop (14 cpus): 38.864 ± 0.004M/s ( 2.776M/s/cpu) uprobe-nop (16 cpus): 41.476 ± 0.020M/s ( 2.592M/s/cpu) uprobe-nop (24 cpus): 64.696 ± 0.021M/s ( 2.696M/s/cpu) uprobe-nop (32 cpus): 85.054 ± 0.027M/s ( 2.658M/s/cpu) uprobe-nop (40 cpus): 101.979 ± 0.032M/s ( 2.549M/s/cpu) uprobe-nop (48 cpus): 110.518 ± 0.056M/s ( 2.302M/s/cpu) uprobe-nop (56 cpus): 117.737 ± 0.020M/s ( 2.102M/s/cpu) uprobe-nop (64 cpus): 124.613 ± 0.079M/s ( 1.947M/s/cpu) uprobe-nop (72 cpus): 133.239 ± 0.032M/s ( 1.851M/s/cpu) uprobe-nop (80 cpus): 142.037 ± 0.138M/s ( 1.775M/s/cpu)
Previously total throughput was maxing out at 11mln/s, and gradually declining past 8 cores. With this change, it now keeps growing with each added CPU, reaching 142mln/s at 80 CPUs (this was measured on a 80-core Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz).
Note, results above assume that commit [0] from linux-trace tree is applied as well, otherwise it will limit scalability to about 10mln/s total throughput.
[0] 10cdb82aa77f ("uprobes: turn trace_uprobe's nhit counter to be per-CPU one")
--- kernel/events/uprobes.c | 44 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+)
kernel/events/uprobes.c +44 -0
@@ -2081,6 +2081,46 @@ static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr)return is_trap_insn (&opcode); } +static struct uprobe *find_active_uprobe_speculative (unsigned long bp_vaddr) +{ +struct mm_struct *mm = current->mm; +struct uprobe *uprobe =NULL ; +struct vm_area_struct *vma; +struct file *vm_file; +loff_t offset; +long seq; + +guard (rcu)(); + +if (!mmap_lock_speculation_start (mm, &seq)) +return NULL ; + + vma =vma_lookup (mm, bp_vaddr); +if (!vma) +return NULL ; + +/* vm_file memory can be reused for another instance of struct file, +* but can't be freed from under us, so it's safe to read fields from +* it, even if the values are some garbage values; ultimately +* find_uprobe_rcu() + mmap_lock_speculation_end() check will ensure +* that whatever we speculatively found is correct +*/ + vm_file =READ_ONCE (vma->vm_file); +if (!vm_file) +return NULL ; + + offset = (loff_t)(vma->vm_pgoff <<PAGE_SHIFT ) + (bp_vaddr - vma->vm_start); + uprobe =find_uprobe_rcu (vm_file->f_inode, offset); +if (!uprobe) +return NULL ; + +/* now double check that nothing about MM changed */ +if (!mmap_lock_speculation_end (mm, seq)) +return NULL ; + +return uprobe; +} +/* assumes being inside RCU protected region */ static struct uprobe *find_active_uprobe_rcu (unsigned long bp_vaddr,int *is_swbp) {
@@ -2088,6 +2128,10 @@ static struct uprobe *find_active_uprobe_rcu(unsigned long bp_vaddr, int *is_swbstruct uprobe *uprobe =NULL ;struct vm_area_struct *vma; + uprobe =find_active_uprobe_speculative (bp_vaddr); +if (uprobe) +return uprobe; +mmap_read_lock (mm); vma =vma_lookup (mm, bp_vaddr);if (vma) {
Signature · 1 line
2.43.5
Oleg Nesterov
maintainerRe: [PATCH v2 tip/perf/core 5/5] uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution
Quoted 20 lines · Andrii Nakryiko
kernel/events/uprobes.c +19
-0+static struct uprobe *find_active_uprobe_speculative (unsigned long bp_vaddr) +{ +struct mm_struct *mm = current->mm; +struct uprobe *uprobe =NULL ; +struct vm_area_struct *vma; +struct file *vm_file; +loff_t offset; +long seq; + +guard (rcu)(); + +if (!mmap_lock_speculation_start (mm, &seq)) +return NULL ; + + vma =vma_lookup (mm, bp_vaddr); +if (!vma) +return NULL ; + +/* vm_file memory can be reused for another instance of struct file,
kernel/events/uprobes.c static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr) · line 2112
Quoted 10 lines
kernel/events/uprobes.c +10
-0+* but can't be freed from under us, so it's safe to read fields from +* it, even if the values are some garbage values; ultimately +* find_uprobe_rcu() + mmap_lock_speculation_end() check will ensure +* that whatever we speculatively found is correct +*/ + vm_file =READ_ONCE (vma->vm_file); +if (!vm_file) +return NULL ; + + offset = (loff_t)(vma->vm_pgoff <<PAGE_SHIFT ) + (bp_vaddr - vma->vm_start);
LGTM. But perhaps vma->vm_pgoff and vma->vm_start need READ_ONCE() as well, if nothing else to shut up KCSAN if this code races with, say, __split_vma() ?
kernel/events/uprobes.c static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr) · line 2113
Quoted 1 line
kernel/events/uprobes.c +1
-0+ uprobe =find_uprobe_rcu (vm_file->f_inode, offset);
OK, I guess vm_file->f_inode is fine without READ_ONCE...
Oleg.
Re: [PATCH v2 tip/perf/core 5/5] uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution
Quoted 27 lines · Oleg Nesterov
Quoted 25 lines · Andrii Nakryiko
kernel/events/uprobes.c +24
-0+static struct uprobe *find_active_uprobe_speculative (unsigned long bp_vaddr) +{ +struct mm_struct *mm = current->mm; +struct uprobe *uprobe =NULL ; +struct vm_area_struct *vma; +struct file *vm_file; +loff_t offset; +long seq; + +guard (rcu)(); + +if (!mmap_lock_speculation_start (mm, &seq)) +return NULL ; + + vma =vma_lookup (mm, bp_vaddr); +if (!vma) +return NULL ; + +/* vm_file memory can be reused for another instance of struct file, +* but can't be freed from under us, so it's safe to read fields from +* it, even if the values are some garbage values; ultimately +* find_uprobe_rcu() + mmap_lock_speculation_end() check will ensure +* that whatever we speculatively found is correct +*/
Quoted 8 lines
kernel/events/uprobes.c static int is_trap_at_addr(struct mm_struct *mm, unsigned long vaddr) · line 2112
Quoted 5 lines
kernel/events/uprobes.c +5
-0+ vm_file =READ_ONCE (vma->vm_file); +if (!vm_file) +return NULL ; + + offset = (loff_t)(vma->vm_pgoff <<PAGE_SHIFT ) + (bp_vaddr - vma->vm_start);LGTM. But perhaps
vma->vm_pgoffandvma->vm_startneedREAD_ONCE()as well, if nothing else to shut up KCSAN if this code races with, say,__split_vma()?
We keep going back and forth between reading directly, using READ_ONCE(), and annotating with data_race(). I don't think it matters in terms of correctness or performance, so I'm happy to add whatever incantations that will make everyone satisfied. Let's see what others think, and I'll incorporate that into the next revision.
Quoted 7 lines
Quoted 2 lines
+ uprobe =
find_uprobe_rcu(vm_file->f_inode, offset);OK, I guess
vm_file->f_inodeis fine withoutREAD_ONCE...Oleg.
Oleg Nesterov
maintainerRe: [PATCH v2 tip/perf/core 5/5] uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution
Quoted 9 lines · Andrii Nakryiko
Quoted 7 lines · Oleg Nesterov
Quoted 6 lines
kernel/events/uprobes.c +5
-0+ vm_file =READ_ONCE (vma->vm_file); +if (!vm_file) +return NULL ; + + offset = (loff_t)(vma->vm_pgoff <<PAGE_SHIFT ) + (bp_vaddr - vma->vm_start);
Quoted 8 lines
Quoted 2 lines
LGTM. But perhaps
vma->vm_pgoffandvma->vm_startneedREAD_ONCE()as well, if nothing else to shut up KCSAN if this code races with, say,__split_vma()?We keep going back and forth between reading directly, using
READ_ONCE(), and annotating withdata_race(). I don't think it matters in terms of correctness or performance, so I'm happy to add whatever incantations that will make everyone satisfied. Let's see what others think, and I'll incorporate that into the next revision.
OK, agreed...
And I guess I was wrong anyway, READ_ONCE() alone won't shutup KCSAN in this case.
Oleg.
kernel test robot
build botRe: [PATCH v2 tip/perf/core 5/5] uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution
In reply to Andrii Nakryikokernel test robot noticed the following build errors:
Hi Andrii,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/perf/core]
url: https://github.com/intel-lab-lkp/linux/commits/Andrii-Nakryiko/mm-introduce-mmap_lock_speculation_-start-end/20241002-065354 base: tip/perf/core patch link: https://lore.kernel.org/r/20241001225207.2215639-6-andrii%40kernel.org patch subject: [PATCH v2 tip/perf/core 5/5] uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution config: loongarch-allyesconfig (https://download.01.org/0day-ci/archive/20241005/[email protected]/config) compiler: loongarch64-linux-gcc (GCC) 14.1.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241005/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
kernel/events/uprobes.c: In function 'find_active_uprobe_speculative':
>> kernel/events/uprobes.c:2098:46: error: passing argument 2 of 'mmap_lock_speculation_start' from incompatible pointer type [-Wincompatible-pointer-types]
2098 | if (!mmap_lock_speculation_start(mm, &seq))
| ^~~~
| |
| long int *
In file included from include/linux/mm.h:16,
from arch/loongarch/include/asm/cacheflush.h:8,
from include/linux/cacheflush.h:5,
from include/linux/highmem.h:8,
from kernel/events/uprobes.c:13:
include/linux/mmap_lock.h:126:75: note: expected 'int *' but argument is of type 'long int *'
126 | static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) { return false; }
| ~~~~~^~~vim +/mmap_lock_speculation_start +2098 kernel/events/uprobes.c
2086
2087 static struct uprobe *find_active_uprobe_speculative(unsigned long bp_vaddr)
2088 {
2089 struct mm_struct *mm = current->mm;
2090 struct uprobe *uprobe = NULL;
2091 struct vm_area_struct *vma;
2092 struct file *vm_file;
2093 loff_t offset;
2094 long seq;
2095
2096 guard(rcu)();
2097
> 2098 if (!mmap_lock_speculation_start(mm, &seq))
2099 return NULL;
2100
2101 vma = vma_lookup(mm, bp_vaddr);
2102 if (!vma)
2103 return NULL;
2104
2105 /* vm_file memory can be reused for another instance of struct file,
2106 * but can't be freed from under us, so it's safe to read fields from
2107 * it, even if the values are some garbage values; ultimately
2108 * find_uprobe_rcu() + mmap_lock_speculation_end() check will ensure
2109 * that whatever we speculatively found is correct
2110 */
2111 vm_file = READ_ONCE(vma->vm_file);
2112 if (!vm_file)
2113 return NULL;
2114
2115 offset = (loff_t)(vma->vm_pgoff << PAGE_SHIFT) + (bp_vaddr - vma->vm_start);
2116 uprobe = find_uprobe_rcu(vm_file->f_inode, offset);
2117 if (!uprobe)
2118 return NULL;
2119
2120 /* now double check that nothing about MM changed */
2121 if (!mmap_lock_speculation_end(mm, seq))
2122 return NULL;
2123
2124 return uprobe;
2125 }
2126
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v2 tip/perf/core 5/5] uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution
In reply to Andrii Nakryikokernel test robot noticed the following build errors:
Hi Andrii,
kernel test robot noticed the following build errors:
[auto build test ERROR on tip/perf/core]
url: https://github.com/intel-lab-lkp/linux/commits/Andrii-Nakryiko/mm-introduce-mmap_lock_speculation_-start-end/20241002-065354 base: tip/perf/core patch link: https://lore.kernel.org/r/20241001225207.2215639-6-andrii%40kernel.org patch subject: [PATCH v2 tip/perf/core 5/5] uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution config: i386-defconfig (https://download.01.org/0day-ci/archive/20241005/[email protected]/config) compiler: clang version 18.1.8 (https://github.com/llvm/llvm-project 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241005/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> kernel/events/uprobes.c:2098:39: error: incompatible pointer types passing 'long *' to parameter of type 'int *' [-Werror,-Wincompatible-pointer-types]
2098 | if (!mmap_lock_speculation_start(mm, &seq))
| ^~~~
include/linux/mmap_lock.h:126:75: note: passing argument to parameter 'seq' here
126 | static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq) { return false; }
| ^1 error generated.
vim +2098 kernel/events/uprobes.c
2086
2087 static struct uprobe *find_active_uprobe_speculative(unsigned long bp_vaddr)
2088 {
2089 struct mm_struct *mm = current->mm;
2090 struct uprobe *uprobe = NULL;
2091 struct vm_area_struct *vma;
2092 struct file *vm_file;
2093 loff_t offset;
2094 long seq;
2095
2096 guard(rcu)();
2097
> 2098 if (!mmap_lock_speculation_start(mm, &seq))
2099 return NULL;
2100
2101 vma = vma_lookup(mm, bp_vaddr);
2102 if (!vma)
2103 return NULL;
2104
2105 /* vm_file memory can be reused for another instance of struct file,
2106 * but can't be freed from under us, so it's safe to read fields from
2107 * it, even if the values are some garbage values; ultimately
2108 * find_uprobe_rcu() + mmap_lock_speculation_end() check will ensure
2109 * that whatever we speculatively found is correct
2110 */
2111 vm_file = READ_ONCE(vma->vm_file);
2112 if (!vm_file)
2113 return NULL;
2114
2115 offset = (loff_t)(vma->vm_pgoff << PAGE_SHIFT) + (bp_vaddr - vma->vm_start);
2116 uprobe = find_uprobe_rcu(vm_file->f_inode, offset);
2117 if (!uprobe)
2118 return NULL;
2119
2120 /* now double check that nothing about MM changed */
2121 if (!mmap_lock_speculation_end(mm, seq))
2122 return NULL;
2123
2124 return uprobe;
2125 }
2126
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Oleg Nesterov
maintainerWell. I am in no position to ack these changes.
But let me say that I personally like this series and I see nothing wrong, except perhaps 5/5 needs some data_race/etc annotations as we have already discussed.
Thanks,
Oleg.
Quoted 48 lines · Andrii Nakryiko
Implement speculative (lockless) resolution of VMA to inode to uprobe, bypassing the need to take
mmap_lockfor reads, if possible. Patch #1 by Suren addsmm_structhelpers that help detect whethermm_structwere changed, which is used by uprobe logic to validate that speculative results can be trusted after all the lookup logic results in a valid uprobe instance. Patch #2 follows to makemm_lock_seqinto 64-bit counter (on 64-bit architectures).Patch #3 adds back RCU-delayed freeing for
FMODE_BACKINGfiles, which is necessary to make speculation safe to access struct file's memory in any possible situation.Patch #4 is a simplification to uprobe VMA flag checking, suggested by Oleg.
And, finally, patch #5 is the speculative VMA-to-uprobe resolution logic. See corresponding patch for details and benchmarking results.
v1->v2:
- adjustedvma_end_write_all()comment to point out it should never be called manually now, but I wasn't sure how ACQUIRE/RELEASE comments should be reworded (previously requested by Jann), so I'd appreciate some help there (Jann);
- int -> long change formm_lock_seq, as agreed at LPC2024 (Jann, Suren, Liam);
-kfree_rcu_mightsleep()forFMODE_BACKING(Suren, Christian);
-vm_flagssimplification infind_active_uprobe_rcu()andfind_active_uprobe_speculative()(Oleg);
- guard(rcu)() simplifiedfind_active_uprobe_speculative()implementation.Andrii Nakryiko (4):
mm: switch to 64-bit mm_lock_seq/vm_lock_seq on 64-bit architectures
fs: add back RCU-delayed freeing ofFMODE_BACKINGfile
uprobes: simplifyfind_active_uprobe_rcu()VMA checks
uprobes: add speculative lockless VMA-to-inode-to-uprobe resolutionSuren Baghdasaryan (1):
mm: introduce mmap_lock_speculation_{start|end}fs/file_table.c | 2 +- include/linux/mm.h | 6 ++-- include/linux/mm_types.h | 7 ++-- include/linux/mmap_lock.h | 72 ++++++++++++++++++++++++++++++++------- kernel/events/uprobes.c | 46 ++++++++++++++++++++++++- kernel/fork.c | 3 -- 6 files changed, 114 insertions(+), 22 deletions(-)--
2.43.5