move per-vma lock into vm_area_struct
Subsystems (12)
Suren Baghdasaryan
maintainerv6 cover move per-vma lock into vm_area_struct
Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3].
Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. This patchset:
1. moves vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned to minimize cacheline sharing;
2. changes vm_area_struct initialization to mark new vma as detached until it is inserted into vma tree;
3. replaces vm_lock and vma->detached flag with a reference counter;
4. changes vm_area_struct cache to SLAB_TYPESAFE_BY_RCU to allow for their reuse and to minimize call_rcu() calls.
Pagefault microbenchmarks show performance improvement:
Hmean faults/cpu-1 507926.5547 ( 0.00%) 506519.3692 * -0.28%* Hmean faults/cpu-4 479119.7051 ( 0.00%) 481333.6802 * 0.46%* Hmean faults/cpu-7 452880.2961 ( 0.00%) 455845.6211 * 0.65%* Hmean faults/cpu-12 347639.1021 ( 0.00%) 352004.2254 * 1.26%* Hmean faults/cpu-21 200061.2238 ( 0.00%) 229597.0317 * 14.76%* Hmean faults/cpu-30 145251.2001 ( 0.00%) 164202.5067 * 13.05%* Hmean faults/cpu-48 106848.4434 ( 0.00%) 120641.5504 * 12.91%* Hmean faults/cpu-56 92472.3835 ( 0.00%) 103464.7916 * 11.89%* Hmean faults/sec-1 507566.1468 ( 0.00%) 506139.0811 * -0.28%* Hmean faults/sec-4 1880478.2402 ( 0.00%) 1886795.6329 * 0.34%* Hmean faults/sec-7 3106394.3438 ( 0.00%) 3140550.7485 * 1.10%* Hmean faults/sec-12 4061358.4795 ( 0.00%) 4112477.0206 * 1.26%* Hmean faults/sec-21 3988619.1169 ( 0.00%) 4577747.1436 * 14.77%* Hmean faults/sec-30 3909839.5449 ( 0.00%) 4311052.2787 * 10.26%* Hmean faults/sec-48 4761108.4691 ( 0.00%) 5283790.5026 * 10.98%* Hmean faults/sec-56 4885561.4590 ( 0.00%) 5415839.4045 * 10.85%*
Changes since v5 [4]
- Added Reviewed-by, per Vlastimil Babka;
- Added replacement of vm_lock and vma->detached flag with vm_refcnt, per Peter Zijlstra and Matthew Wilcox
- Marked vmas detached during exit_mmap;
- Ensureed vmas are in detached state before they are freed;
- Changed SLAB_TYPESAFE_BY_RCU patch to not require ctor, leading to a much simpler code;
- Removed unnecessary patch [5]
- Updated documentation to reflect changes to vm_lock;
Patchset applies over mm-unstable after reverting v5 of this patchset [4] (currently 687e99a5faa5-905ab222508a)
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/
[3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://lore.kernel.org/all/[email protected]/
Suren Baghdasaryan (16):
mm: introduce vma_start_read_locked{_nested} helpers
mm: move per-vma lock into vm_area_struct
mm: mark vma as detached until it's added into vma tree mm/nommu: fix the last places where vma is not locked before being
attached
types: move struct rcuwait into types.h
mm: allow vma_start_read_locked/vma_start_read_locked_nested to fail
mm: move mmap_init_lock() out of the header file
mm: uninline the main body of vma_start_write()
refcount: introduce __refcount_{add|inc}_not_zero_limited
mm: replace vm_lock and detached flag with a reference count
mm: enforce vma to be in detached state before freeing
mm: remove extra vma_numab_state_init() call
mm: introduce vma_ensure_detached()
mm: prepare lock_vma_under_rcu() for vma reuse possibility
mm: make vma cache SLAB_TYPESAFE_BY_RCU
docs/mm: document latest changes to vm_lock
Documentation/mm/process_addrs.rst | 44 ++++---- include/linux/mm.h | 162 +++++++++++++++++++++++------ include/linux/mm_types.h | 37 ++++--- include/linux/mmap_lock.h | 6 -- include/linux/rcuwait.h | 13 +-- include/linux/refcount.h | 20 +++- include/linux/slab.h | 6 -- include/linux/types.h | 12 +++ kernel/fork.c | 88 ++++------------ mm/init-mm.c | 1 + mm/memory.c | 75 +++++++++++-- mm/mmap.c | 8 +- mm/nommu.c | 2 + mm/userfaultfd.c | 31 +++--- mm/vma.c | 15 ++- mm/vma.h | 4 +- tools/testing/vma/linux/atomic.h | 5 + tools/testing/vma/vma_internal.h | 96 +++++++++-------- 18 files changed, 378 insertions(+), 247 deletions(-)
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 1/16 mm: introduce vma_start_read_locked{_nested} helpers
Introduce helper functions which can be used to read-lock a VMA when holding mmap_lock for read. Replace direct accesses to vma->vm_lock with these new helpers.
--- include/linux/mm.h | 24 ++++++++++++++++++++++++ mm/userfaultfd.c | 22 +++++----------------- 2 files changed, 29 insertions(+), 17 deletions(-)
include/linux/mm.h +24 -0
@@ -735,6 +735,30 @@ static inline bool vma_start_read(struct vm_area_struct *vma)return true; } +/* +* Use only while holding mmap read lock which guarantees that locking will not +* fail (nobody can concurrently write-lock the vma). vma_start_read() should +* not be used in such cases because it might fail due to mm_lock_seq overflow. +* This functionality is used to obtain vma read lock and drop the mmap read lock. +*/ +static inline void vma_start_read_locked_nested (struct vm_area_struct *vma,int subclass) +{ +mmap_assert_locked (vma->vm_mm); +down_read_nested (&vma->vm_lock->lock, subclass); +} + +/* +* Use only while holding mmap read lock which guarantees that locking will not +* fail (nobody can concurrently write-lock the vma). vma_start_read() should +* not be used in such cases because it might fail due to mm_lock_seq overflow. +* This functionality is used to obtain vma read lock and drop the mmap read lock. +*/ +static inline void vma_start_read_locked (struct vm_area_struct *vma) +{ +mmap_assert_locked (vma->vm_mm); +down_read (&vma->vm_lock->lock); +} +static inline void vma_end_read (struct vm_area_struct *vma) {rcu_read_lock ();/* keeps vma alive till the end of up_read */
mm/userfaultfd.c +5 -17
@@ -84,16 +84,8 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm,mmap_read_lock (mm); vma =find_vma_and_prepare_anon (mm, address); -if (!IS_ERR (vma)) { -/* -* We cannot use vma_start_read() as it may fail due to -* false locked (see comment in vma_start_read()). We -* can avoid that by directly locking vm_lock under -* mmap_lock, which guarantees that nobody can lock the -* vma for write (vma_start_write()) under us. -*/ -down_read (&vma->vm_lock->lock); - } +if (!IS_ERR (vma)) +vma_start_read_locked (vma);mmap_read_unlock (mm);return vma;
@@ -1491,14 +1483,10 @@ static int uffd_move_lock(struct mm_struct *mm,mmap_read_lock (mm); err =find_vmas_mm_locked (mm, dst_start, src_start, dst_vmap, src_vmap);if (!err) { -/* -* See comment in uffd_lock_vma() as to why not using -* vma_start_read() here. -*/ -down_read (&(*dst_vmap)->vm_lock->lock); +vma_start_read_locked (*dst_vmap);if (*dst_vmap != *src_vmap) -down_read_nested (&(*src_vmap)->vm_lock->lock, -SINGLE_DEPTH_NESTING ); +vma_start_read_locked_nested (*src_vmap, +SINGLE_DEPTH_NESTING ); }mmap_read_unlock (mm);return err;
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 2/16 mm: move per-vma lock into vm_area_struct
In reply to Suren BaghdasaryanBack when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3].
Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. Move vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned as well. With kernel compiled using defconfig, this causes VMA memory consumption to grow from 160 (vm_area_struct) + 40 (vm_lock) bytes to 256 bytes:
slabinfo before:
<name> ... <objsize> <objperslab> <pagesperslab> : ...
vma_lock ... 40 102 1 : ...
vm_area_struct ... 160 51 2 : ...
slabinfo after moving vm_lock:
<name> ... <objsize> <objperslab> <pagesperslab> : ...
vm_area_struct ... 256 32 2 : ...
Aggregate VMA memory consumption per 1000 VMAs grows from 50 to 64 pages, which is 5.5MB per 100000 VMAs. Note that the size of this structure is dependent on the kernel configuration and typically the original size is higher than 160 bytes. Therefore these calculations are close to the worst case scenario. A more realistic vm_area_struct usage before this change is:
<name> ... <objsize> <objperslab> <pagesperslab> : ...
vma_lock ... 40 102 1 : ...
vm_area_struct ... 176 46 2 : ...Aggregate VMA memory consumption per 1000 VMAs grows from 54 to 64 pages, which is 3.9MB per 100000 VMAs. This memory consumption growth can be addressed later by optimizing the vm_lock.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/
[3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/
--- include/linux/mm.h | 28 ++++++++++-------- include/linux/mm_types.h | 6 ++-- kernel/fork.c | 49 ++++---------------------------- tools/testing/vma/vma_internal.h | 33 +++++---------------- 4 files changed, 32 insertions(+), 84 deletions(-)
include/linux/mm.h +16 -12
@@ -697,6 +697,12 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {}#endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_PER_VMA_LOCK +static inline void vma_lock_init (struct vm_area_struct *vma) +{ +init_rwsem (&vma->vm_lock.lock); + vma->vm_lock_seq =UINT_MAX ; +} +/* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to
@@ -714,7 +720,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)if (READ_ONCE (vma->vm_lock_seq) ==READ_ONCE (vma->vm_mm->mm_lock_seq.sequence))return false; -if (unlikely (down_read_trylock (&vma->vm_lock->lock) ==0 )) +if (unlikely (down_read_trylock (&vma->vm_lock.lock) ==0 ))return false;/*
@@ -729,7 +735,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* This pairs with RELEASE semantics in vma_end_write_all(). */ if (unlikely (vma->vm_lock_seq ==raw_read_seqcount (&vma->vm_mm->mm_lock_seq))) { -up_read (&vma->vm_lock->lock); +up_read (&vma->vm_lock.lock);return false; }return true;
@@ -744,7 +750,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)static inline void vma_start_read_locked_nested (struct vm_area_struct *vma,int subclass) {mmap_assert_locked (vma->vm_mm); -down_read_nested (&vma->vm_lock->lock, subclass); +down_read_nested (&vma->vm_lock.lock, subclass); }/*
@@ -756,13 +762,13 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, intstatic inline void vma_start_read_locked (struct vm_area_struct *vma) {mmap_assert_locked (vma->vm_mm); -down_read (&vma->vm_lock->lock); +down_read (&vma->vm_lock.lock); }static inline void vma_end_read (struct vm_area_struct *vma) {rcu_read_lock ();/* keeps vma alive till the end of up_read */ -up_read (&vma->vm_lock->lock); +up_read (&vma->vm_lock.lock);rcu_read_unlock (); }
@@ -791,7 +797,7 @@ static inline void vma_start_write(struct vm_area_struct *vma)if (__is_vma_write_locked (vma, &mm_lock_seq))return ; -down_write (&vma->vm_lock->lock); +down_write (&vma->vm_lock.lock);/* * We should use WRITE_ONCE() here because we can have concurrent reads * from the early lockless pessimistic check in vma_start_read().
@@ -799,7 +805,7 @@ static inline void vma_start_write(struct vm_area_struct *vma)* we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. */ WRITE_ONCE (vma->vm_lock_seq, mm_lock_seq); -up_write (&vma->vm_lock->lock); +up_write (&vma->vm_lock.lock); }static inline void vma_assert_write_locked (struct vm_area_struct *vma)
@@ -811,7 +817,7 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma)static inline void vma_assert_locked (struct vm_area_struct *vma) { -if (!rwsem_is_locked (&vma->vm_lock->lock)) +if (!rwsem_is_locked (&vma->vm_lock.lock))vma_assert_write_locked (vma); }
@@ -844,6 +850,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,#else /* CONFIG_PER_VMA_LOCK */ +static inline void vma_lock_init (struct vm_area_struct *vma) {}static inline bool vma_start_read (struct vm_area_struct *vma) {return false; }static inline void vma_end_read (struct vm_area_struct *vma) {}
@@ -878,10 +885,6 @@ static inline void assert_fault_locked(struct vm_fault *vmf)extern const struct vm_operations_struct vma_dummy_vm_ops; -/* -* WARNING: vma_init does not initialize vma->vm_lock. -* Use vm_area_alloc()/vm_area_free() if vma needs locking. -*/ static inline void vma_init (struct vm_area_struct *vma,struct mm_struct *mm) {memset (vma,0 ,sizeof (*vma));
@@ -890,6 +893,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)INIT_LIST_HEAD (&vma->anon_vma_chain);vma_mark_detached (vma, false);vma_numab_state_init (vma); +vma_lock_init (vma); }/* Use when VMA is not part of the VMA tree and needs no locking */
include/linux/mm_types.h +4 -2
@@ -700,8 +700,6 @@ struct vm_area_struct {* slowpath. */ unsigned int vm_lock_seq; -/* Unstable RCU readers are allowed to read this. */ -struct vma_lock *vm_lock;#endif /*
@@ -754,6 +752,10 @@ struct vm_area_struct {struct vma_numab_state *numab_state;/* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; +#ifdef CONFIG_PER_VMA_LOCK +/* Unstable RCU readers are allowed to read this. */ +struct vma_lock vm_lock____cacheline_aligned_in_smp ; +#endif } __randomize_layout;#ifdef CONFIG_NUMA
kernel/fork.c +5 -44
85 lines not on this page · raw message
tools/testing/vma/vma_internal.h +7 -26
@@ -270,10 +270,10 @@ struct vm_area_struct {/* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) -* - vm_lock->lock (in write mode) +* - vm_lock.lock (in write mode) * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) -* - vm_lock->lock (in read or write mode) +* - vm_lock.lock (in read or write mode) * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). *
@@ -282,7 +282,7 @@ struct vm_area_struct {* slowpath. */ unsigned int vm_lock_seq; -struct vma_lock *vm_lock; +struct vma_lock vm_lock;#endif /*
@@ -459,17 +459,10 @@ static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi)return mas_find (&vmi->mas,ULONG_MAX ); } -static inline bool vma_lock_alloc (struct vm_area_struct *vma) +static inline void vma_lock_init (struct vm_area_struct *vma) { - vma->vm_lock =calloc (1 ,sizeof (struct vma_lock )); - -if (!vma->vm_lock) -return false; - -init_rwsem (&vma->vm_lock->lock); +init_rwsem (&vma->vm_lock.lock); vma->vm_lock_seq =UINT_MAX ; - -return true; }static inline void vma_assert_write_locked (struct vm_area_struct *);
@@ -492,6 +485,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_ops = &vma_dummy_vm_ops;INIT_LIST_HEAD (&vma->anon_vma_chain);vma_mark_detached (vma, false); +vma_lock_init (vma); }static inline struct vm_area_struct *vm_area_alloc (struct mm_struct *mm)
@@ -502,10 +496,6 @@ static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm)return NULL ;vma_init (vma, mm); -if (!vma_lock_alloc (vma)) { -free (vma); -return NULL ; - }return vma; }
@@ -518,10 +508,7 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)return NULL ;memcpy (new, orig,sizeof (*new)); -if (!vma_lock_alloc (new)) { -free (new); -return NULL ; - } +vma_lock_init (new);INIT_LIST_HEAD (&new->anon_vma_chain);return new;
@@ -691,14 +678,8 @@ static inline void mpol_put(struct mempolicy *) { } -static inline void vma_lock_free (struct vm_area_struct *vma) -{ -free (vma->vm_lock); -} -static inline void __vm_area_free (struct vm_area_struct *vma) { -vma_lock_free (vma);free (vma); }
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 3/16 mm: mark vma as detached until it's added into vma tree
In reply to Suren BaghdasaryanCurrent implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree.
Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter.
--- include/linux/mm.h | 27 ++++++++++++++++++++------- kernel/fork.c | 4 ++++ mm/memory.c | 2 +- mm/vma.c | 6 +++--- mm/vma.h | 2 ++ tools/testing/vma/vma_internal.h | 17 ++++++++++++----- 6 files changed, 42 insertions(+), 16 deletions(-)
include/linux/mm.h +20 -7
@@ -821,12 +821,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma)vma_assert_write_locked (vma); } -static inline void vma_mark_detached (struct vm_area_struct *vma,bool detached) +static inline void vma_mark_attached (struct vm_area_struct *vma) +{ + vma->detached = false; +} + +static inline void vma_mark_detached (struct vm_area_struct *vma) {/* When detaching vma should be write-locked */ -if (detached) -vma_assert_write_locked (vma); - vma->detached = detached; +vma_assert_write_locked (vma); + vma->detached = true; +} + +static inline bool is_vma_detached (struct vm_area_struct *vma) +{ +return vma->detached; }static inline void release_fault_lock (struct vm_fault *vmf)
@@ -857,8 +866,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {}static inline void vma_start_write (struct vm_area_struct *vma) {}static inline void vma_assert_write_locked (struct vm_area_struct *vma) {mmap_assert_write_locked (vma->vm_mm); } -static inline void vma_mark_detached (struct vm_area_struct *vma, -bool detached) {} +static inline void vma_mark_attached (struct vm_area_struct *vma) {} +static inline void vma_mark_detached (struct vm_area_struct *vma) {}static inline struct vm_area_struct *lock_vma_under_rcu (struct mm_struct *mm,unsigned long address)
@@ -891,7 +900,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops;INIT_LIST_HEAD (&vma->anon_vma_chain); -vma_mark_detached (vma, false); +#ifdef CONFIG_PER_VMA_LOCK +/* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true; +#endif vma_numab_state_init (vma);vma_lock_init (vma); }
@@ -1086,6 +1098,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi,if (unlikely (mas_is_err (&vmi->mas)))return -ENOMEM ; +vma_mark_attached (vma);return 0 ; }
kernel/fork.c +4 -0
@@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)data_race (memcpy (new, orig,sizeof (*new)));vma_lock_init (new);INIT_LIST_HEAD (&new->anon_vma_chain); +#ifdef CONFIG_PER_VMA_LOCK +/* vma is not locked, can't use vma_mark_detached() */ + new->detached = true; +#endif vma_numab_state_init (new);dup_anon_vma_name (orig, new);
mm/memory.c +1 -1
@@ -6350,7 +6350,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, goto inval;/* Check if the VMA got isolated after we found it */ -if (vma->detached) { +if (is_vma_detached (vma)) {vma_end_read (vma);count_vm_vma_lock_event (VMA_LOCK_MISS );/* The area was replaced with another one */
mm/vma.c +3 -3
@@ -327,7 +327,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi,if (vp->remove) { again: -vma_mark_detached (vp->remove, true); +vma_mark_detached (vp->remove);if (vp->file) {uprobe_munmap (vp->remove, vp->remove->vm_start, vp->remove->vm_end);
@@ -1222,7 +1222,7 @@ static void reattach_vmas(struct ma_state *mas_detach)mas_set (mas_detach,0 );mas_for_each (mas_detach, vma,ULONG_MAX ) -vma_mark_detached (vma, false); +vma_mark_attached (vma);__mt_destroy (mas_detach->tree); }
@@ -1297,7 +1297,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms,if (error) goto munmap_gather_failed; -vma_mark_detached (next, true); +vma_mark_detached (next); nrpages =vma_pages (next); vms->nr_pages += nrpages;
mm/vma.h +2 -0
@@ -157,6 +157,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi,if (unlikely (mas_is_err (&vmi->mas)))return -ENOMEM ; +vma_mark_attached (vma);return 0 ; }
@@ -389,6 +390,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi,__mas_set_range (&vmi->mas, vma->vm_start, vma->vm_end -1 );mas_store_prealloc (&vmi->mas, vma); +vma_mark_attached (vma); }static inline unsigned long vma_iter_addr (struct vma_iterator *vmi)
tools/testing/vma/vma_internal.h +12 -5
@@ -465,13 +465,17 @@ static inline void vma_lock_init(struct vm_area_struct *vma) vma->vm_lock_seq =UINT_MAX ; } +static inline void vma_mark_attached (struct vm_area_struct *vma) +{ + vma->detached = false; +} +static inline void vma_assert_write_locked (struct vm_area_struct *); -static inline void vma_mark_detached (struct vm_area_struct *vma,bool detached) +static inline void vma_mark_detached (struct vm_area_struct *vma) {/* When detaching vma should be write-locked */ -if (detached) -vma_assert_write_locked (vma); - vma->detached = detached; +vma_assert_write_locked (vma); + vma->detached = true; }extern const struct vm_operations_struct vma_dummy_vm_ops;
@@ -484,7 +488,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops;INIT_LIST_HEAD (&vma->anon_vma_chain); -vma_mark_detached (vma, false); +/* vma is not locked, can't use vma_mark_detached() */ + vma->detached = true;vma_lock_init (vma); }
@@ -510,6 +515,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)memcpy (new, orig,sizeof (*new));vma_lock_init (new);INIT_LIST_HEAD (&new->anon_vma_chain); +/* vma is not locked, can't use vma_mark_detached() */ + new->detached = true;return new; }
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 4/16 mm/nommu: fix the last places where vma is not locked before being attached
In reply to Suren Baghdasaryannommu configuration has two places where vma gets attached to the vma tree without write-locking it. Add the missing locks to ensure vma is always locked before it's attached.
--- mm/nommu.c | 2 ++ 1 file changed, 2 insertions(+)
mm/nommu.c +2 -0
@@ -1189,6 +1189,7 @@ unsigned long do_mmap(struct file *file, goto error_just_free;setup_vma_to_mm (vma, current->mm); +vma_start_write (vma); current->mm->map_count++;/* add the VMA to the tree */ vma_iter_store (&vmi, vma);
@@ -1356,6 +1357,7 @@ static int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,setup_vma_to_mm (vma, mm);setup_vma_to_mm (new, mm); +vma_start_write (new);vma_iter_store (vmi, new); mm->map_count++;return 0 ;
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 5/16 types: move struct rcuwait into types.h
In reply to Suren BaghdasaryanMove rcuwait struct definition into types.h so that rcuwait can be used without including rcuwait.h which includes other headers. Without this change mm_types.h can't use rcuwait due to a the following circular dependency:
mm_types.h -> rcuwait.h -> signal.h -> mm_types.h
--- include/linux/rcuwait.h | 13 +------------ include/linux/types.h | 12 ++++++++++++ 2 files changed, 13 insertions(+), 12 deletions(-)
include/linux/rcuwait.h +1 -12
@@ -4,18 +4,7 @@#include <linux/rcupdate.h> #include <linux/sched/signal.h> - -/* -* rcuwait provides a way of blocking and waking up a single -* task in an rcu-safe manner. -* -* The only time @task is non-nil is when a user is blocked (or -* checking if it needs to) on a condition, and reset as soon as we -* know that the condition has succeeded and are awoken. -*/ -struct rcuwait { -struct task_struct __rcu *task; -}; +#include <linux/types.h> #define __RCUWAIT_INITIALIZER (name) \ { .task = NULL, }
include/linux/types.h +12 -0
@@ -248,5 +248,17 @@ typedef void (*swap_func_t)(void *a, void *b, int size);typedef int (*cmp_r_func_t )(const void *a,const void *b,const void *priv);typedef int (*cmp_func_t )(const void *a,const void *b); +/* +* rcuwait provides a way of blocking and waking up a single +* task in an rcu-safe manner. +* +* The only time @task is non-nil is when a user is blocked (or +* checking if it needs to) on a condition, and reset as soon as we +* know that the condition has succeeded and are awoken. +*/ +struct rcuwait { +struct task_struct __rcu *task; +}; +#endif /* __ASSEMBLY__ */ #endif /* _LINUX_TYPES_H */
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 6/16 mm: allow vma_start_read_locked/vma_start_read_locked_nested to fail
In reply to Suren BaghdasaryanWith upcoming replacement of vm_lock with vm_refcnt, we need to handle a possibility of vma_start_read_locked/vma_start_read_locked_nested failing due to refcount overflow. Prepare for such possibility by changing these APIs and adjusting their users.
--- include/linux/mm.h | 6 ++++-- mm/userfaultfd.c | 17 ++++++++++++----- 2 files changed, 16 insertions(+), 7 deletions(-)
include/linux/mm.h +4 -2
@@ -747,10 +747,11 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* not be used in such cases because it might fail due to mm_lock_seq overflow. * This functionality is used to obtain vma read lock and drop the mmap read lock. */ -static inline void vma_start_read_locked_nested (struct vm_area_struct *vma,int subclass) +static inline bool vma_start_read_locked_nested (struct vm_area_struct *vma,int subclass) {mmap_assert_locked (vma->vm_mm);down_read_nested (&vma->vm_lock.lock, subclass); +return true; }/*
@@ -759,10 +760,11 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int* not be used in such cases because it might fail due to mm_lock_seq overflow. * This functionality is used to obtain vma read lock and drop the mmap read lock. */ -static inline void vma_start_read_locked (struct vm_area_struct *vma) +static inline bool vma_start_read_locked (struct vm_area_struct *vma) {mmap_assert_locked (vma->vm_mm);down_read (&vma->vm_lock.lock); +return true; }static inline void vma_end_read (struct vm_area_struct *vma)
mm/userfaultfd.c +12 -5
@@ -85,7 +85,8 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm,mmap_read_lock (mm); vma =find_vma_and_prepare_anon (mm, address);if (!IS_ERR (vma)) -vma_start_read_locked (vma); +if (!vma_start_read_locked (vma)) + vma =ERR_PTR (-EAGAIN );mmap_read_unlock (mm);return vma;
@@ -1483,10 +1484,16 @@ static int uffd_move_lock(struct mm_struct *mm,mmap_read_lock (mm); err =find_vmas_mm_locked (mm, dst_start, src_start, dst_vmap, src_vmap);if (!err) { -vma_start_read_locked (*dst_vmap); -if (*dst_vmap != *src_vmap) -vma_start_read_locked_nested (*src_vmap, -SINGLE_DEPTH_NESTING ); +if (!vma_start_read_locked (*dst_vmap)) { +if (*dst_vmap != *src_vmap) { +if (!vma_start_read_locked_nested (*src_vmap, +SINGLE_DEPTH_NESTING )) { +vma_end_read (*dst_vmap); + err = -EAGAIN ; + } + } + }else + err = -EAGAIN ; }mmap_read_unlock (mm);return err;
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Re: [PATCH v6 06/16] mm: allow vma_start_read_locked/vma_start_read_locked_nested to fail
Quoted 58 lines · Suren Baghdasaryan
With upcoming replacement of
vm_lockwithvm_refcnt, we need to handle a possibility of vma_start_read_locked/vma_start_read_locked_nested failing due to refcount overflow. Prepare for such possibility by changing these APIs and adjusting their users.Signed-off-by Suren Baghdasaryan <[email protected]>Cc Lokesh Gidra <[email protected]>--- include/linux/mm.h | 6 ++++-- mm/userfaultfd.c | 17 ++++++++++++----- 2 files changed, 16 insertions(+), 7 deletions(-)include/linux/mm.h +4
-2@@ -747,10 +747,11 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* not be used in such cases because it might fail due to mm_lock_seq overflow. * This functionality is used to obtain vma read lock and drop the mmap read lock. */ -static inline void vma_start_read_locked_nested (struct vm_area_struct *vma,int subclass) +static inline bool vma_start_read_locked_nested (struct vm_area_struct *vma,int subclass) {mmap_assert_locked (vma->vm_mm);down_read_nested (&vma->vm_lock.lock, subclass); +return true; }/* @@ -759,10 +760,11 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int* not be used in such cases because it might fail due to mm_lock_seq overflow. * This functionality is used to obtain vma read lock and drop the mmap read lock. */ -static inline void vma_start_read_locked (struct vm_area_struct *vma) +static inline bool vma_start_read_locked (struct vm_area_struct *vma) {mmap_assert_locked (vma->vm_mm);down_read (&vma->vm_lock.lock); +return true; }static inline void vma_end_read (struct vm_area_struct *vma)mm/userfaultfd.c +2
-1@@ -85,7 +85,8 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm,mmap_read_lock (mm); vma =find_vma_and_prepare_anon (mm, address);if (!IS_ERR (vma)) -vma_start_read_locked (vma); +if (!vma_start_read_locked (vma)) + vma =ERR_PTR (-EAGAIN );mmap_read_unlock (mm);return vma;
mm/userfaultfd.c static int uffd_move_lock(struct mm_struct *mm, · line 1487
Quoted 8 lines
mm/userfaultfd.c +1
-4@@ -1483,10 +1484,16 @@ static int uffd_move_lock(struct mm_struct *mm,mmap_read_lock (mm); err =find_vmas_mm_locked (mm, dst_start, src_start, dst_vmap, src_vmap);if (!err) { -vma_start_read_locked (*dst_vmap); -if (*dst_vmap != *src_vmap) -vma_start_read_locked_nested (*src_vmap, -SINGLE_DEPTH_NESTING ); +if (!vma_start_read_locked (*dst_vmap)) {
I think you mistakenly reversed the condition. This block should be executed if we manage to lock dst_vma successfully.
Quoted 15 lines
mm/userfaultfd.c +9
-1+if (*dst_vmap != *src_vmap) { +if (!vma_start_read_locked_nested (*src_vmap, +SINGLE_DEPTH_NESTING )) { +vma_end_read (*dst_vmap); + err = -EAGAIN ; + } + } + }else + err = -EAGAIN ; }mmap_read_unlock (mm);return err; --2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerRe: [PATCH v6 06/16] mm: allow vma_start_read_locked/vma_start_read_locked_nested to fail
Quoted 57 lines · Lokesh Gidra
Quoted 55 lines · Suren Baghdasaryan
With upcoming replacement of
vm_lockwithvm_refcnt, we need to handle a possibility of vma_start_read_locked/vma_start_read_locked_nested failing due to refcount overflow. Prepare for such possibility by changing these APIs and adjusting their users.Signed-off-by Suren Baghdasaryan <[email protected]>Cc Lokesh Gidra <[email protected]>--- include/linux/mm.h | 6 ++++-- mm/userfaultfd.c | 17 ++++++++++++----- 2 files changed, 16 insertions(+), 7 deletions(-)include/linux/mm.h +4
-2@@ -747,10 +747,11 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* not be used in such cases because it might fail due to mm_lock_seq overflow. * This functionality is used to obtain vma read lock and drop the mmap read lock. */ -static inline void vma_start_read_locked_nested (struct vm_area_struct *vma,int subclass) +static inline bool vma_start_read_locked_nested (struct vm_area_struct *vma,int subclass) {mmap_assert_locked (vma->vm_mm);down_read_nested (&vma->vm_lock.lock, subclass); +return true; }/* @@ -759,10 +760,11 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int* not be used in such cases because it might fail due to mm_lock_seq overflow. * This functionality is used to obtain vma read lock and drop the mmap read lock. */ -static inline void vma_start_read_locked (struct vm_area_struct *vma) +static inline bool vma_start_read_locked (struct vm_area_struct *vma) {mmap_assert_locked (vma->vm_mm);down_read (&vma->vm_lock.lock); +return true; }static inline void vma_end_read (struct vm_area_struct *vma)mm/userfaultfd.c +2
-1@@ -85,7 +85,8 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm,mmap_read_lock (mm); vma =find_vma_and_prepare_anon (mm, address);if (!IS_ERR (vma)) -vma_start_read_locked (vma); +if (!vma_start_read_locked (vma)) + vma =ERR_PTR (-EAGAIN );
Quoted 14 lines
mm/userfaultfd.c static int uffd_move_lock(struct mm_struct *mm, · line 1487
Quoted 11 lines
mm/userfaultfd.c +1
-4mmap_read_unlock (mm);return vma;@@ -1483,10 +1484,16 @@ static int uffd_move_lock(struct mm_struct *mm,mmap_read_lock (mm); err =find_vmas_mm_locked (mm, dst_start, src_start, dst_vmap, src_vmap);if (!err) { -vma_start_read_locked (*dst_vmap); -if (*dst_vmap != *src_vmap) -vma_start_read_locked_nested (*src_vmap, -SINGLE_DEPTH_NESTING ); +if (!vma_start_read_locked (*dst_vmap)) {I think you mistakenly reversed the condition. This block should be executed if we manage to lock
dst_vmasuccessfully.
Oops. Sorry, you are right. That above condition should have been reversed. I'll fix it in the next revision.
Thanks!
Quoted 15 lines
Quoted 15 lines
mm/userfaultfd.c +9
-1+if (*dst_vmap != *src_vmap) { +if (!vma_start_read_locked_nested (*src_vmap, +SINGLE_DEPTH_NESTING )) { +vma_end_read (*dst_vmap); + err = -EAGAIN ; + } + } + }else + err = -EAGAIN ; }mmap_read_unlock (mm);return err; --2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 7/16 mm: move mmap_init_lock() out of the header file
In reply to Suren Baghdasaryanmmap_init_lock() is used only from mm_init() in fork.c, therefore it does not have to reside in the header file. This move lets us avoid including additional headers in mmap_lock.h later, when mmap_init_lock() needs to initialize rcuwait object.
--- include/linux/mmap_lock.h | 6 ------ kernel/fork.c | 6 ++++++ 2 files changed, 6 insertions(+), 6 deletions(-)
include/linux/mmap_lock.h +0 -6
@@ -122,12 +122,6 @@ static inline bool mmap_lock_speculate_retry(struct mm_struct *mm, unsigned int#endif /* CONFIG_PER_VMA_LOCK */ -static inline void mmap_init_lock (struct mm_struct *mm) -{ -init_rwsem (&mm->mmap_lock); -mm_lock_seqcount_init (mm); -} -static inline void mmap_write_lock (struct mm_struct *mm) {__mmap_lock_trace_start_locking (mm, true);
kernel/fork.c +6 -0
@@ -1224,6 +1224,12 @@ static void mm_init_uprobes_state(struct mm_struct *mm)#endif } +static inline void mmap_init_lock (struct mm_struct *mm) +{ +init_rwsem (&mm->mmap_lock); +mm_lock_seqcount_init (mm); +} +static struct mm_struct *mm_init (struct mm_struct *mm,struct task_struct *p,struct user_namespace *user_ns) {
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 8/16 mm: uninline the main body of vma_start_write()
In reply to Suren Baghdasaryanvma_start_write() is used in many places and will grow in size very soon. It is not used in performance critical paths and uninlining it should limit the future code size growth.
No functional changes.
--- include/linux/mm.h | 12 +++--------- mm/memory.c | 14 ++++++++++++++ 2 files changed, 17 insertions(+), 9 deletions(-)
include/linux/mm.h +3 -9
@@ -787,6 +787,8 @@ static bool __is_vma_write_locked(struct vm_area_struct *vma, unsigned int *mm_lreturn (vma->vm_lock_seq == *mm_lock_seq); } +void __vma_start_write (struct vm_area_struct *vma,unsigned int mm_lock_seq); +/* * Begin writing to a VMA. * Exclude concurrent readers under the per-VMA lock until the currently
@@ -799,15 +801,7 @@ static inline void vma_start_write(struct vm_area_struct *vma)if (__is_vma_write_locked (vma, &mm_lock_seq))return ; -down_write (&vma->vm_lock.lock); -/* -* We should use WRITE_ONCE() here because we can have concurrent reads -* from the early lockless pessimistic check in vma_start_read(). -* We don't really care about the correctness of that early check, but -* we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. -*/ -WRITE_ONCE (vma->vm_lock_seq, mm_lock_seq); -up_write (&vma->vm_lock.lock); +__vma_start_write (vma, mm_lock_seq); }static inline void vma_assert_write_locked (struct vm_area_struct *vma)
mm/memory.c +14 -0
@@ -6329,6 +6329,20 @@ struct vm_area_struct *lock_mm_and_find_vma(struct mm_struct *mm,#endif #ifdef CONFIG_PER_VMA_LOCK +void __vma_start_write (struct vm_area_struct *vma,unsigned int mm_lock_seq) +{ +down_write (&vma->vm_lock.lock); +/* +* We should use WRITE_ONCE() here because we can have concurrent reads +* from the early lockless pessimistic check in vma_start_read(). +* We don't really care about the correctness of that early check, but +* we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. +*/ +WRITE_ONCE (vma->vm_lock_seq, mm_lock_seq); +up_write (&vma->vm_lock.lock); +} +EXPORT_SYMBOL_GPL (__vma_start_write); +/* * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be * stable and not isolated. If the VMA is not found or is being modified the
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 9/16 refcount: introduce __refcount_{add|inc}_not_zero_limited
In reply to Suren BaghdasaryanIntroduce functions to increase refcount but with a top limit above which they will fail to increase. Setting the limit to 0 indicates no limit.
--- include/linux/refcount.h | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
include/linux/refcount.h +19 -1
@@ -137,13 +137,19 @@ static inline unsigned int refcount_read(const refcount_t *r) }static inline __must_check __signed_wrap -bool __refcount_add_not_zero(int i, refcount_t *r, int *oldp) +bool __refcount_add_not_zero_limited(int i, refcount_t *r, int *oldp, +int limit) {int old =refcount_read (r);do {if (!old)break ; +if (limit && old + i > limit) { +if (oldp) + *oldp = old; +return false; + } }while (!atomic_try_cmpxchg_relaxed (&r->refs, &old, old + i));if (oldp)
@@ -155,6 +161,12 @@ bool __refcount_add_not_zero(int i, refcount_t *r, int *oldp)return old; } +static inline __must_check __signed_wrap +bool __refcount_add_not_zero(int i, refcount_t *r, int *oldp) +{ +return __refcount_add_not_zero_limited (i, r, oldp,0 ); +} +/** * refcount_add_not_zero - add a value to a refcount unless it is 0 * @i: the value to add to the refcount
@@ -213,6 +225,12 @@ static inline void refcount_add(int i, refcount_t *r)__refcount_add (i, r,NULL ); } +static inline __must_check bool __refcount_inc_not_zero_limited (refcount_t *r, +int *oldp,int limit) +{ +return __refcount_add_not_zero_limited (1 , r, oldp, limit); +} +static inline __must_check bool __refcount_inc_not_zero (refcount_t *r,int *oldp) {return __refcount_add_not_zero (1 , r, oldp);
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 10/16 mm: replace vm_lock and detached flag with a reference count
In reply to Suren Baghdasaryanrw_semaphore is a sizable structure of 40 bytes and consumes considerable space for each vm_area_struct. However vma_lock has two important specifics which can be used to replace rw_semaphore with a simpler structure:
1. Readers never wait. They try to take the vma_lock and fall back to mmap_lock if that fails.
2. Only one writer at a time will ever try to write-lock a vma_lock because writers first take mmap_lock in write mode.
Because of these requirements, full rw_semaphore functionality is not needed and we can replace rw_semaphore and the vma->detached flag with a refcount (vm_refcnt).
When vma is in detached state, vm_refcnt is 0 and only a call to vma_mark_attached() can take it out of this state. Note that unlike before, now we enforce both vma_mark_attached() and vma_mark_detached() to be done only after vma has been write-locked. vma_mark_attached() changes vm_refcnt to 1 to indicate that it has been attached to the vma tree. When a reader takes read lock, it increments vm_refcnt, unless the top usable bit of vm_refcnt (0x40000000) is set, indicating presence of a writer. When writer takes write lock, it both increments vm_refcnt and sets the top usable bit to indicate its presence. If there are readers, writer will wait using newly introduced mm->vma_writer_wait. Since all writers take mmap_lock in write mode first, there can be only one writer at a time. The last reader to release the lock will signal the writer to wake up.
refcount might overflow if there are many competing readers, in which case read-locking will fail. Readers are expected to handle such failures.
--- include/linux/mm.h | 95 ++++++++++++++++++++++++-------- include/linux/mm_types.h | 23 ++++---- kernel/fork.c | 9 +-- mm/init-mm.c | 1 + mm/memory.c | 33 +++++++---- tools/testing/vma/linux/atomic.h | 5 ++ tools/testing/vma/vma_internal.h | 57 ++++++++++--------- 7 files changed, 147 insertions(+), 76 deletions(-)
include/linux/mm.h +72 -23
@@ -32,6 +32,7 @@#include <linux/memremap.h> #include <linux/slab.h> #include <linux/cacheinfo.h> +#include <linux/rcuwait.h> struct mempolicy ;struct anon_vma ;
@@ -699,10 +700,27 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {}#ifdef CONFIG_PER_VMA_LOCK static inline void vma_lock_init (struct vm_area_struct *vma) { -init_rwsem (&vma->vm_lock.lock); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static struct lock_class_key lockdep_key; + +lockdep_init_map (&vma->vmlock_dep_map,"vm_lock" , &lockdep_key,0 ); +#endif +refcount_set (&vma->vm_refcnt,VMA_STATE_DETACHED ); vma->vm_lock_seq =UINT_MAX ; } +static inline void vma_refcount_put (struct vm_area_struct *vma) +{ +int refcnt; + +if (!__refcount_dec_and_test (&vma->vm_refcnt, &refcnt)) { +rwsem_release (&vma->vmlock_dep_map, _RET_IP_); + +if (refcnt &VMA_STATE_LOCKED ) +rcuwait_wake_up (&vma->vm_mm->vma_writer_wait); + } +} +/* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to
@@ -710,6 +728,8 @@ static inline void vma_lock_init(struct vm_area_struct *vma)*/ static inline bool vma_start_read (struct vm_area_struct *vma) { +int oldcnt; +/* * Check before locking. A race might cause false locked result. * We can use READ_ONCE() for the mm_lock_seq here, and don't need
@@ -720,13 +740,20 @@ static inline bool vma_start_read(struct vm_area_struct *vma)if (READ_ONCE (vma->vm_lock_seq) ==READ_ONCE (vma->vm_mm->mm_lock_seq.sequence))return false; -if (unlikely (down_read_trylock (&vma->vm_lock.lock) ==0 )) + +rwsem_acquire_read (&vma->vmlock_dep_map,0 ,0 , _RET_IP_); +/* Limit at VMA_STATE_LOCKED - 2 to leave one count for a writer */ +if (unlikely (!__refcount_inc_not_zero_limited (&vma->vm_refcnt, &oldcnt, +VMA_STATE_LOCKED -2 ))) { +rwsem_release (&vma->vmlock_dep_map, _RET_IP_);return false; + } +lock_acquired (&vma->vmlock_dep_map, _RET_IP_);/* -* Overflow might produce false locked result. +* Overflow of vm_lock_seq/mm_lock_seq might produce false locked result. * False unlocked result is impossible because we modify and check -* vma->vm_lock_seq under vma->vm_lock protection and mm->mm_lock_seq +* vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq * modification invalidates all existing locks. * * We must use ACQUIRE semantics for the mm_lock_seq so that if we are
@@ -734,10 +761,12 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ -if (unlikely (vma->vm_lock_seq ==raw_read_seqcount (&vma->vm_mm->mm_lock_seq))) { -up_read (&vma->vm_lock.lock); +if (oldcnt &VMA_STATE_LOCKED || +unlikely (vma->vm_lock_seq ==raw_read_seqcount (&vma->vm_mm->mm_lock_seq))) { +vma_refcount_put (vma);return false; } +return true; }
@@ -749,8 +778,17 @@ static inline bool vma_start_read(struct vm_area_struct *vma)*/ static inline bool vma_start_read_locked_nested (struct vm_area_struct *vma,int subclass) { +int oldcnt; +mmap_assert_locked (vma->vm_mm); -down_read_nested (&vma->vm_lock.lock, subclass); +rwsem_acquire_read (&vma->vmlock_dep_map, subclass,0 , _RET_IP_); +/* Limit at VMA_STATE_LOCKED - 2 to leave one count for a writer */ +if (unlikely (!__refcount_inc_not_zero_limited (&vma->vm_refcnt, &oldcnt, +VMA_STATE_LOCKED -2 ))) { +rwsem_release (&vma->vmlock_dep_map, _RET_IP_); +return false; + } +lock_acquired (&vma->vmlock_dep_map, _RET_IP_);return true; }
@@ -762,15 +800,13 @@ static inline bool vma_start_read_locked_nested(struct vm_area_struct *vma, int*/ static inline bool vma_start_read_locked (struct vm_area_struct *vma) { -mmap_assert_locked (vma->vm_mm); -down_read (&vma->vm_lock.lock); -return true; +return vma_start_read_locked_nested (vma,0 ); }static inline void vma_end_read (struct vm_area_struct *vma) {rcu_read_lock ();/* keeps vma alive till the end of up_read */ -up_read (&vma->vm_lock.lock); +vma_refcount_put (vma);rcu_read_unlock (); }
@@ -813,25 +849,42 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma)static inline void vma_assert_locked (struct vm_area_struct *vma) { -if (!rwsem_is_locked (&vma->vm_lock.lock)) +if (refcount_read (&vma->vm_refcnt) <=VMA_STATE_ATTACHED )vma_assert_write_locked (vma); } -static inline void vma_mark_attached (struct vm_area_struct *vma) +/* +* WARNING: to avoid racing with vma_mark_attached(), should be called either +* under mmap_write_lock or when the object has been isolated under +* mmap_write_lock, ensuring no competing writers. +*/ +static inline bool is_vma_detached (struct vm_area_struct *vma) { - vma->detached = false; +return refcount_read (&vma->vm_refcnt) ==VMA_STATE_DETACHED ; } -static inline void vma_mark_detached (struct vm_area_struct *vma) +static inline void vma_mark_attached (struct vm_area_struct *vma) { -/* When detaching vma should be write-locked */ vma_assert_write_locked (vma); - vma->detached = true; + +if (is_vma_detached (vma)) +refcount_set (&vma->vm_refcnt,VMA_STATE_ATTACHED ); } -static inline bool is_vma_detached (struct vm_area_struct *vma) +static inline void vma_mark_detached (struct vm_area_struct *vma) { -return vma->detached; +vma_assert_write_locked (vma); + +if (is_vma_detached (vma)) +return ; + +/* We are the only writer, so no need to use vma_refcount_put(). */ +if (!refcount_dec_and_test (&vma->vm_refcnt)) { +/* +* Reader must have temporarily raised vm_refcnt but it will +* drop it without using the vma since vma is write-locked. +*/ + } }static inline void release_fault_lock (struct vm_fault *vmf)
@@ -896,10 +949,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops;INIT_LIST_HEAD (&vma->anon_vma_chain); -#ifdef CONFIG_PER_VMA_LOCK -/* vma is not locked, can't use vma_mark_detached() */ - vma->detached = true; -#endif vma_numab_state_init (vma);vma_lock_init (vma); }
include/linux/mm_types.h +11 -12
@@ -19,6 +19,7 @@#include <linux/workqueue.h> #include <linux/seqlock.h> #include <linux/percpu_counter.h> +#include <linux/types.h> #include <asm/mmu.h>
@@ -599,9 +600,9 @@ static inline struct anon_vma_name *anon_vma_name_alloc(const char *name) }#endif -struct vma_lock { -struct rw_semaphore lock; -}; +#define VMA_STATE_DETACHED 0x0 +#define VMA_STATE_ATTACHED 0x1 +#define VMA_STATE_LOCKED 0x40000000struct vma_numab_state {/*
@@ -679,19 +680,13 @@ struct vm_area_struct { };#ifdef CONFIG_PER_VMA_LOCK -/* -* Flag to indicate areas detached from the mm->mm_mt tree. -* Unstable RCU readers are allowed to read this. -*/ -bool detached; -/* * Can only be written (using WRITE_ONCE()) while holding both: * - mmap_lock (in write mode) -* - vm_lock->lock (in write mode) +* - vm_refcnt VMA_STATE_LOCKED is set * Can be read reliably while holding one of: * - mmap_lock (in read or write mode) -* - vm_lock->lock (in read or write mode) +* - vm_refcnt VMA_STATE_LOCKED is set or vm_refcnt > VMA_STATE_ATTACHED * Can be read unreliably (using READ_ONCE()) for pessimistic bailout * while holding nothing (except RCU to keep the VMA struct allocated). *
@@ -754,7 +749,10 @@ struct vm_area_struct {struct vm_userfaultfd_ctx vm_userfaultfd_ctx;#ifdef CONFIG_PER_VMA_LOCK /* Unstable RCU readers are allowed to read this. */ -struct vma_lock vm_lock____cacheline_aligned_in_smp ; +refcount_t vm_refcnt____cacheline_aligned_in_smp ; +#ifdef CONFIG_DEBUG_LOCK_ALLOC +struct lockdep_map vmlock_dep_map; +#endif #endif } __randomize_layout;
@@ -889,6 +887,7 @@ struct mm_struct {* by mmlist_lock */ #ifdef CONFIG_PER_VMA_LOCK +struct rcuwait vma_writer_wait;/* * This field has lock-like semantics, meaning it is sometimes * accessed with ACQUIRE/RELEASE semantics.
kernel/fork.c +3 -6
@@ -465,10 +465,6 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)data_race (memcpy (new, orig,sizeof (*new)));vma_lock_init (new);INIT_LIST_HEAD (&new->anon_vma_chain); -#ifdef CONFIG_PER_VMA_LOCK -/* vma is not locked, can't use vma_mark_detached() */ - new->detached = true; -#endif vma_numab_state_init (new);dup_anon_vma_name (orig, new);
@@ -488,8 +484,6 @@ static void vm_area_free_rcu_cb(struct rcu_head *head) struct vm_area_struct *vma = container_of(head, struct vm_area_struct, vm_rcu); -/* The vma should not be locked while being destroyed. */ -VM_BUG_ON_VMA (rwsem_is_locked (&vma->vm_lock.lock), vma); __vm_area_free(vma); }#endif
@@ -1228,6 +1222,9 @@ static inline void mmap_init_lock(struct mm_struct *mm) {init_rwsem (&mm->mmap_lock);mm_lock_seqcount_init (mm); +#ifdef CONFIG_PER_VMA_LOCK +rcuwait_init (&mm->vma_writer_wait); +#endif } static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,
mm/init-mm.c +1 -0
@@ -40,6 +40,7 @@ struct mm_struct init_mm = { .arg_lock = __SPIN_LOCK_UNLOCKED(init_mm.arg_lock), .mmlist =LIST_HEAD_INIT (init_mm.mmlist), #ifdef CONFIG_PER_VMA_LOCK + .vma_writer_wait =__RCUWAIT_INITIALIZER (init_mm.vma_writer_wait), .mm_lock_seq = SEQCNT_ZERO(init_mm.mm_lock_seq), #endif .user_ns = &init_user_ns,
mm/memory.c +23 -10
@@ -6331,7 +6331,25 @@ struct vm_area_struct *lock_mm_and_find_vma(struct mm_struct *mm,#ifdef CONFIG_PER_VMA_LOCK void __vma_start_write (struct vm_area_struct *vma,unsigned int mm_lock_seq) { -down_write (&vma->vm_lock.lock); +bool detached; + +/* +* If vma is detached then only vma_mark_attached() can raise the +* vm_refcnt. mmap_write_lock prevents racing with vma_mark_attached(). +*/ +if (!refcount_inc_not_zero (&vma->vm_refcnt)) { +WRITE_ONCE (vma->vm_lock_seq, mm_lock_seq); +return ; + } + +rwsem_acquire (&vma->vmlock_dep_map,0 ,0 , _RET_IP_); +/* vma is attached, set the writer present bit */ +refcount_add (VMA_STATE_LOCKED , &vma->vm_refcnt); +/* wait until state is VMA_STATE_ATTACHED + (VMA_STATE_LOCKED + 1) */ +rcuwait_wait_event (&vma->vm_mm->vma_writer_wait, +refcount_read (&vma->vm_refcnt) ==VMA_STATE_ATTACHED + (VMA_STATE_LOCKED +1 ), +TASK_UNINTERRUPTIBLE ); +lock_acquired (&vma->vmlock_dep_map, _RET_IP_);/* * We should use WRITE_ONCE() here because we can have concurrent reads * from the early lockless pessimistic check in vma_start_read().
@@ -6339,7 +6357,10 @@ void __vma_start_write(struct vm_area_struct *vma, unsigned int mm_lock_seq)* we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy. */ WRITE_ONCE (vma->vm_lock_seq, mm_lock_seq); -up_write (&vma->vm_lock.lock); + detached =refcount_sub_and_test (VMA_STATE_LOCKED +1 , + &vma->vm_refcnt); +rwsem_release (&vma->vmlock_dep_map, _RET_IP_); +VM_BUG_ON_VMA (detached, vma);/* vma should remain attached */ }EXPORT_SYMBOL_GPL (__vma_start_write);
@@ -6355,7 +6376,6 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,struct vm_area_struct *vma;rcu_read_lock (); -retry: vma =mas_walk (&mas);if (!vma) goto inval;
@@ -6363,13 +6383,6 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,if (!vma_start_read (vma)) goto inval; -/* Check if the VMA got isolated after we found it */ -if (is_vma_detached (vma)) { -vma_end_read (vma); -count_vm_vma_lock_event (VMA_LOCK_MISS ); -/* The area was replaced with another one */ - goto retry; - }/* * At this point, we have a stable reference to a VMA: The VMA is * locked and we know it hasn't already been isolated.
tools/testing/vma/linux/atomic.h +5 -0
@@ -9,4 +9,9 @@#define atomic_set (x, y) do {} while (0)#define U8_MAX UCHAR_MAX +#ifndef atomic_cmpxchg_relaxed +#define atomic_cmpxchg_relaxed uatomic_cmpxchg +#define atomic_cmpxchg_release uatomic_cmpxchg +#endif /* atomic_cmpxchg_relaxed */ +#endif /* _LINUX_ATOMIC_H */
tools/testing/vma/vma_internal.h +32 -25
130 lines not on this page · raw message
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
include/linux/mm.h static inline bool vma_start_read(struct vm_area_struct *vma) · line 765
Quoted 6 lines · Suren Baghdasaryan
include/linux/mm.h +2
-0@@ -734,10 +761,12 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ +if (oldcnt &VMA_STATE_LOCKED || +unlikely (vma->vm_lock_seq ==raw_read_seqcount (&vma->vm_mm->mm_lock_seq))) {
You likely want that unlikely to cover both conditions :-)
Quoted 6 lines
include/linux/mm.h +2
-0+vma_refcount_put (vma);return false; } +return true; }
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 10 lines · Peter Zijlstra
include/linux/mm.h static inline bool vma_start_read(struct vm_area_struct *vma) · line 765
Quoted 6 lines · Suren Baghdasaryan
include/linux/mm.h +2
-0@@ -734,10 +761,12 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ +if (oldcnt &VMA_STATE_LOCKED || +unlikely (vma->vm_lock_seq ==raw_read_seqcount (&vma->vm_mm->mm_lock_seq))) {You likely want that unlikely to cover both conditions :-)
True. VMA_STATE_LOCKED is set only while the writer is updating the vm_lock_seq and that's a narrow window. I'll make that change in the next revision. Thanks!
Quoted 7 lines
Quoted 7 lines
include/linux/mm.h +2
-0+vma_refcount_put (vma);return false; } +return true; }
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Suren BaghdasaryanOn Mon, Dec 16, 2024 at 11:24:13AM -0800, Suren Baghdasaryan wrote:
FWIW, I find the whole VMA_STATE_{A,DE}TATCHED thing awkward. And perhaps s/VMA_STATE_LOCKED/VMA_LOCK_OFFSET/ ?
Also, perhaps:
#define VMA_REF_LIMIT (VMA_LOCK_OFFSET - 2)
include/linux/mm.h static inline void vma_numab_state_free(struct vm_area_struct *vma) {} · line 709
Quoted 12 lines
include/linux/mm.h +6
-1@@ -699,10 +700,27 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {}#ifdef CONFIG_PER_VMA_LOCK static inline void vma_lock_init (struct vm_area_struct *vma) { -init_rwsem (&vma->vm_lock.lock); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static struct lock_class_key lockdep_key; + +lockdep_init_map (&vma->vmlock_dep_map,"vm_lock" , &lockdep_key,0 ); +#endif +refcount_set (&vma->vm_refcnt,VMA_STATE_DETACHED ); vma->vm_lock_seq =UINT_MAX ;
Depending on how you do the actual allocation (GFP_ZERO) you might want to avoid that vm_refcount store entirely.
Perhaps instead write: VM_WARN_ON(refcount_read(&vma->vm_refcnt));
include/linux/mm.h static inline void vma_assert_write_locked(struct vm_area_struct *vma) · line 852
Quoted 6 lines
include/linux/mm.h +1
-1@@ -813,25 +849,42 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma)static inline void vma_assert_locked (struct vm_area_struct *vma) { -if (!rwsem_is_locked (&vma->vm_lock.lock)) +if (refcount_read (&vma->vm_refcnt) <=VMA_STATE_ATTACHED )
if (is_vma_detached(vma))
include/linux/mm.h static inline void vma_assert_write_locked(struct vm_area_struct *vma) · line 863
Quoted 13 lines
include/linux/mm.h +7
-2vma_assert_write_locked (vma); } -static inline void vma_mark_attached (struct vm_area_struct *vma) +/* +* WARNING: to avoid racing with vma_mark_attached(), should be called either +* under mmap_write_lock or when the object has been isolated under +* mmap_write_lock, ensuring no competing writers. +*/ +static inline bool is_vma_detached (struct vm_area_struct *vma) { - vma->detached = false; +return refcount_read (&vma->vm_refcnt) ==VMA_STATE_DETACHED ;
return !refcount_read(&vma->vm_refcnt);
include/linux/mm.h static inline void vma_assert_write_locked(struct vm_area_struct *vma) · line 871
Quoted 11 lines
include/linux/mm.h +4
-3} -static inline void vma_mark_detached (struct vm_area_struct *vma) +static inline void vma_mark_attached (struct vm_area_struct *vma) { -/* When detaching vma should be write-locked */ vma_assert_write_locked (vma); - vma->detached = true; + +if (is_vma_detached (vma)) +refcount_set (&vma->vm_refcnt,VMA_STATE_ATTACHED );
Urgh, so it would be really good to not call this at all them not 0. I've not tried to untangle the mess, but this is really awkward. Surely you don't add it to the mas multiple times either.
Also:
refcount_set(&vma->vm_refcnt, 1);
is so much clearer.
That is, should this not live in vma_iter_store*(), right before mas_store_gfp() ?
Also, ISTR having to set vm_lock_seq right before it?
include/linux/mm.h static inline void vma_assert_write_locked(struct vm_area_struct *vma) · line 879
Quoted 10 lines
include/linux/mm.h +5
-2} -static inline bool is_vma_detached (struct vm_area_struct *vma) +static inline void vma_mark_detached (struct vm_area_struct *vma) { -return vma->detached; +vma_assert_write_locked (vma); + +if (is_vma_detached (vma)) +return ;
Again, this just reads like confusion :/ Surely you don't have the same with mas_detach?
Quoted 9 lines
include/linux/mm.h +8
-0+ +/* We are the only writer, so no need to use vma_refcount_put(). */ +if (!refcount_dec_and_test (&vma->vm_refcnt)) { +/* +* Reader must have temporarily raised vm_refcnt but it will +* drop it without using the vma since vma is write-locked. +*/ + } }
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 4 lines · Peter Zijlstra
On Mon, Dec 16, 2024 at 11:24:13AM -0800, Suren Baghdasaryan wrote:
FWIW, I find the whole VMA_STATE_{A,DE}TATCHED thing awkward. And
I'm bad with naming things, so any better suggestions are welcome. Are you suggesting to drop VMA_STATE_{A,DE}TATCHED nomenclature and use 0/1 values directly?
Quoted 1 line
perhaps s/VMA_STATE_LOCKED/VMA_LOCK_OFFSET/ ?
Sounds good. I'll change it to VMA_LOCK_OFFSET.
Quoted 4 lines
Also, perhaps:
#define
VMA_REF_LIMIT(VMA_LOCK_OFFSET- 2)
Ack.
Quoted 16 lines
include/linux/mm.h static inline void vma_numab_state_free(struct vm_area_struct *vma) {} · line 709
Quoted 13 lines
include/linux/mm.h +6
-1@@ -699,10 +700,27 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {}#ifdef CONFIG_PER_VMA_LOCK static inline void vma_lock_init (struct vm_area_struct *vma) { -init_rwsem (&vma->vm_lock.lock); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static struct lock_class_key lockdep_key; + +lockdep_init_map (&vma->vmlock_dep_map,"vm_lock" , &lockdep_key,0 ); +#endif +refcount_set (&vma->vm_refcnt,VMA_STATE_DETACHED ); vma->vm_lock_seq =UINT_MAX ;Depending on how you do the actual allocation (
GFP_ZERO) you might want to avoid thatvm_refcountstore entirely.
I think we could initialize it to 0 in the slab cache constructor and when vma is freed we already ensure it's 0. So, even when reused it will be in the correct 0 state.
Quoted 2 lines
Perhaps instead write: VM_WARN_ON(refcount_read(&vma->vm_refcnt));
Yes, with the above approach that should work.
Quoted 25 lines
include/linux/mm.h static inline void vma_assert_write_locked(struct vm_area_struct *vma) · line 852
Quoted 7 lines
include/linux/mm.h +1
-1@@ -813,25 +849,42 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma)static inline void vma_assert_locked (struct vm_area_struct *vma) { -if (!rwsem_is_locked (&vma->vm_lock.lock)) +if (refcount_read (&vma->vm_refcnt) <=VMA_STATE_ATTACHED )if (
is_vma_detached(vma))include/linux/mm.h static inline void vma_assert_write_locked(struct vm_area_struct *vma) · line 863
Quoted 13 lines
include/linux/mm.h +7
-2vma_assert_write_locked (vma); } -static inline void vma_mark_attached (struct vm_area_struct *vma) +/* +* WARNING: to avoid racing with vma_mark_attached(), should be called either +* under mmap_write_lock or when the object has been isolated under +* mmap_write_lock, ensuring no competing writers. +*/ +static inline bool is_vma_detached (struct vm_area_struct *vma) { - vma->detached = false; +return refcount_read (&vma->vm_refcnt) ==VMA_STATE_DETACHED ;return !refcount_read(&vma->vm_refcnt);Quoted 2 lines
(quoted diff) +0
-0}
Quoted 13 lines
include/linux/mm.h static inline void vma_assert_write_locked(struct vm_area_struct *vma) · line 871
Quoted 9 lines
include/linux/mm.h +4
-3-static inline void vma_mark_detached (struct vm_area_struct *vma) +static inline void vma_mark_attached (struct vm_area_struct *vma) { -/* When detaching vma should be write-locked */ vma_assert_write_locked (vma); - vma->detached = true; + +if (is_vma_detached (vma)) +refcount_set (&vma->vm_refcnt,VMA_STATE_ATTACHED );Urgh, so it would be really good to not call this at all them not 0. I've not tried to untangle the mess, but this is really awkward. Surely you don't add it to the mas multiple times either.
The issue is that when we merge/split/shrink/grow vmas, we skip on marking them detached while modifying them. Therefore from vma_mark_attached() POV it will look like we are attaching an already attached vma. I can try to clean that up if this is really a concern.
Quoted 6 lines
Also:
refcount_set(&vma->vm_refcnt, 1);is so much clearer.
Ok, IIUC you are in favour of dropping VMA_STATE_ATTACHED/VMA_STATE_DETACHED.
Quoted 3 lines
That is, should this not live in vma_iter_store*(), right before
mas_store_gfp()?
Currently it's done right *after* mas_store_gfp() but I was debating with myself if it indeed should be *before* insertion into the tree...
Quoted 2 lines
Also, ISTR having to set
vm_lock_seqright before it?
Yes, vma_mark_attached() requires vma to be write-locked beforehand, hence the above vma_assert_write_locked(). But oftentimes it's locked not right before vma_mark_attached() because some other modification functions also require vma to be write-locked.
Quoted 14 lines
include/linux/mm.h static inline void vma_assert_write_locked(struct vm_area_struct *vma) · line 879
Quoted 11 lines
include/linux/mm.h +5
-2} -static inline bool is_vma_detached (struct vm_area_struct *vma) +static inline void vma_mark_detached (struct vm_area_struct *vma) { -return vma->detached; +vma_assert_write_locked (vma); + +if (is_vma_detached (vma)) +return ;Again, this just reads like confusion :/ Surely you don't have the same with
mas_detach?
I'll double-check if we ever double-mark vma as detached.
Thanks for the review!
Quoted 10 lines
Quoted 10 lines
include/linux/mm.h +8
-0+ +/* We are the only writer, so no need to use vma_refcount_put(). */ +if (!refcount_dec_and_test (&vma->vm_refcnt)) { +/* +* Reader must have temporarily raised vm_refcnt but it will +* drop it without using the vma since vma is write-locked. +*/ + } }
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 5 lines · Suren Baghdasaryan
Quoted 2 lines
That is, should this not live in vma_iter_store*(), right before
mas_store_gfp()?Currently it's done right *after*
mas_store_gfp()but I was debating with myself if it indeed should be *before* insertion into the tree...
The moment it goes into the tree it becomes visible to RCU lookups, it's a bit weird to have them with !refcnt at that point, but I don't suppose it harms.
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Suren BaghdasaryanQuoted 46 lines · Suren Baghdasaryan
include/linux/mm.h +24
-0+static inline void vma_refcount_put (struct vm_area_struct *vma) +{ +int refcnt; + +if (!__refcount_dec_and_test (&vma->vm_refcnt, &refcnt)) { +rwsem_release (&vma->vmlock_dep_map, _RET_IP_); + +if (refcnt &VMA_STATE_LOCKED ) +rcuwait_wake_up (&vma->vm_mm->vma_writer_wait); + } +} +/* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -710,6 +728,8 @@ static inline void vma_lock_init(struct vm_area_struct *vma)*/ static inline bool vma_start_read (struct vm_area_struct *vma) { +int oldcnt; +/* * Check before locking. A race might cause false locked result. * We can use READ_ONCE() for the mm_lock_seq here, and don't need @@ -720,13 +740,20 @@ static inline bool vma_start_read(struct vm_area_struct *vma)if (READ_ONCE (vma->vm_lock_seq) ==READ_ONCE (vma->vm_mm->mm_lock_seq.sequence))return false; + +rwsem_acquire_read (&vma->vmlock_dep_map,0 ,0 , _RET_IP_); +/* Limit at VMA_STATE_LOCKED - 2 to leave one count for a writer */ +if (unlikely (!__refcount_inc_not_zero_limited (&vma->vm_refcnt, &oldcnt, +VMA_STATE_LOCKED -2 ))) { +rwsem_release (&vma->vmlock_dep_map, _RET_IP_);return false; + } +lock_acquired (&vma->vmlock_dep_map, _RET_IP_);/* +* Overflow of vm_lock_seq/mm_lock_seq might produce false locked result. * False unlocked result is impossible because we modify and check +* vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq * modification invalidates all existing locks. * * We must use ACQUIRE semantics for the mm_lock_seq so that if we are
include/linux/mm.h static inline bool vma_start_read(struct vm_area_struct *vma) · line 766
Quoted 6 lines
include/linux/mm.h +3
-0@@ -734,10 +761,12 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ +if (oldcnt &VMA_STATE_LOCKED || +unlikely (vma->vm_lock_seq ==raw_read_seqcount (&vma->vm_mm->mm_lock_seq))) { +vma_refcount_put (vma);
Suppose we have detach race with a concurrent RCU lookup like:
vma = mas_lookup(); vma_start_write(); mas_detach(); vma_start_read() rwsem_acquire_read() inc // success vma_mark_detach(); dec_and_test // assumes 1->0 // is actually 2->1 if (vm_lock_seq == vma->vm_mm_mm_lock_seq) // true vma_refcount_put dec_and_test() // 1->0 *NO* rwsem_release()
Quoted 5 lines
include/linux/mm.h +1
-0return false; } +return true; }
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 57 lines · Peter Zijlstra
Quoted 46 lines · Suren Baghdasaryan
include/linux/mm.h +24
-0+static inline void vma_refcount_put (struct vm_area_struct *vma) +{ +int refcnt; + +if (!__refcount_dec_and_test (&vma->vm_refcnt, &refcnt)) { +rwsem_release (&vma->vmlock_dep_map, _RET_IP_); + +if (refcnt &VMA_STATE_LOCKED ) +rcuwait_wake_up (&vma->vm_mm->vma_writer_wait); + } +} +/* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -710,6 +728,8 @@ static inline void vma_lock_init(struct vm_area_struct *vma)*/ static inline bool vma_start_read (struct vm_area_struct *vma) { +int oldcnt; +/* * Check before locking. A race might cause false locked result. * We can use READ_ONCE() for the mm_lock_seq here, and don't need @@ -720,13 +740,20 @@ static inline bool vma_start_read(struct vm_area_struct *vma)if (READ_ONCE (vma->vm_lock_seq) ==READ_ONCE (vma->vm_mm->mm_lock_seq.sequence))return false; + +rwsem_acquire_read (&vma->vmlock_dep_map,0 ,0 , _RET_IP_); +/* Limit at VMA_STATE_LOCKED - 2 to leave one count for a writer */ +if (unlikely (!__refcount_inc_not_zero_limited (&vma->vm_refcnt, &oldcnt, +VMA_STATE_LOCKED -2 ))) { +rwsem_release (&vma->vmlock_dep_map, _RET_IP_);return false; + } +lock_acquired (&vma->vmlock_dep_map, _RET_IP_);/* +* Overflow of vm_lock_seq/mm_lock_seq might produce false locked result. * False unlocked result is impossible because we modify and check +* vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq * modification invalidates all existing locks. * * We must use ACQUIRE semantics for the mm_lock_seq so that if we are include/linux/mm.h static inline bool vma_start_read(struct vm_area_struct *vma) · line 766
Quoted 6 lines
include/linux/mm.h +3
-0@@ -734,10 +761,12 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ +if (oldcnt &VMA_STATE_LOCKED || +unlikely (vma->vm_lock_seq ==raw_read_seqcount (&vma->vm_mm->mm_lock_seq))) { +vma_refcount_put (vma);Suppose we have detach race with a concurrent RCU lookup like:
Quoted 16 lines
vma = mas_lookup();
vma_start_write();
mas_detach();
vma_start_read()
rwsem_acquire_read()
inc // success
vma_mark_detach();
dec_and_test // assumes 1->0
// is actually 2->1
if (vm_lock_seq == vma->vm_mm_mm_lock_seq) // true
vma_refcount_put
dec_and_test() // 1->0
*NO* rwsem_release()Yes, this is possible. I think that's not a problem until we start reusing the vmas and I deal with this race later in this patchset. I think what you described here is the same race I mention in the description of this patch:
https://lore.kernel.org/all/[email protected]/
I introduce vma_ensure_detached() in that patch to handle this case and ensure that vmas are detached before they are returned into the slab cache for reuse. Does that make sense?
Quoted 7 lines
Quoted 7 lines
include/linux/mm.h +1
-0return false; } +return true; }
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 75 lines · Suren Baghdasaryan
Quoted 57 lines · Peter Zijlstra
Quoted 46 lines · Suren Baghdasaryan
include/linux/mm.h +24
-0+static inline void vma_refcount_put (struct vm_area_struct *vma) +{ +int refcnt; + +if (!__refcount_dec_and_test (&vma->vm_refcnt, &refcnt)) { +rwsem_release (&vma->vmlock_dep_map, _RET_IP_); + +if (refcnt &VMA_STATE_LOCKED ) +rcuwait_wake_up (&vma->vm_mm->vma_writer_wait); + } +} +/* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -710,6 +728,8 @@ static inline void vma_lock_init(struct vm_area_struct *vma)*/ static inline bool vma_start_read (struct vm_area_struct *vma) { +int oldcnt; +/* * Check before locking. A race might cause false locked result. * We can use READ_ONCE() for the mm_lock_seq here, and don't need @@ -720,13 +740,20 @@ static inline bool vma_start_read(struct vm_area_struct *vma)if (READ_ONCE (vma->vm_lock_seq) ==READ_ONCE (vma->vm_mm->mm_lock_seq.sequence))return false; + +rwsem_acquire_read (&vma->vmlock_dep_map,0 ,0 , _RET_IP_); +/* Limit at VMA_STATE_LOCKED - 2 to leave one count for a writer */ +if (unlikely (!__refcount_inc_not_zero_limited (&vma->vm_refcnt, &oldcnt, +VMA_STATE_LOCKED -2 ))) { +rwsem_release (&vma->vmlock_dep_map, _RET_IP_);return false; + } +lock_acquired (&vma->vmlock_dep_map, _RET_IP_);/* +* Overflow of vm_lock_seq/mm_lock_seq might produce false locked result. * False unlocked result is impossible because we modify and check +* vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq * modification invalidates all existing locks. * * We must use ACQUIRE semantics for the mm_lock_seq so that if we are include/linux/mm.h static inline bool vma_start_read(struct vm_area_struct *vma) · line 766
Quoted 6 lines
include/linux/mm.h +3
-0@@ -734,10 +761,12 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ +if (oldcnt &VMA_STATE_LOCKED || +unlikely (vma->vm_lock_seq ==raw_read_seqcount (&vma->vm_mm->mm_lock_seq))) { +vma_refcount_put (vma);Suppose we have detach race with a concurrent RCU lookup like:
Quoted 16 lines
vma = mas_lookup(); vma_start_write(); mas_detach(); vma_start_read() rwsem_acquire_read() inc // success vma_mark_detach(); dec_and_test // assumes 1->0 // is actually 2->1 if (vm_lock_seq == vma->vm_mm_mm_lock_seq) // true vma_refcount_put dec_and_test() // 1->0 *NO* rwsem_release()
Quoted 8 lines
Yes, this is possible. I think that's not a problem until we start reusing the vmas and I deal with this race later in this patchset. I think what you described here is the same race I mention in the description of this patch:
https://lore.kernel.org/all/[email protected]/
I introducevma_ensure_detached()in that patch to handle this case and ensure that vmas are detached before they are returned into the slab cache for reuse. Does that make sense?
So I just replied there, and no, I don't think it makes sense. Just put the kmem_cache_free() in vma_refcount_put(), to be done on 0.
Anyway, my point was more about the weird entanglement of lockdep and the refcount. Just pull the lockdep annotation out of _put() and put it explicitly in the vma_start_read() error paths and vma_end_read().
Additionally, having vma_end_write() would allow you to put a lockdep annotation in vma_{start,end}_write() -- which was I think the original reason I proposed it a while back, that and having improved clarity when reading the code, since explicitly marking the end of a section is helpful.
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 86 lines · Peter Zijlstra
Quoted 75 lines · Suren Baghdasaryan
Quoted 57 lines · Peter Zijlstra
Quoted 46 lines · Suren Baghdasaryan
include/linux/mm.h +24
-0+static inline void vma_refcount_put (struct vm_area_struct *vma) +{ +int refcnt; + +if (!__refcount_dec_and_test (&vma->vm_refcnt, &refcnt)) { +rwsem_release (&vma->vmlock_dep_map, _RET_IP_); + +if (refcnt &VMA_STATE_LOCKED ) +rcuwait_wake_up (&vma->vm_mm->vma_writer_wait); + } +} +/* * Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to @@ -710,6 +728,8 @@ static inline void vma_lock_init(struct vm_area_struct *vma)*/ static inline bool vma_start_read (struct vm_area_struct *vma) { +int oldcnt; +/* * Check before locking. A race might cause false locked result. * We can use READ_ONCE() for the mm_lock_seq here, and don't need @@ -720,13 +740,20 @@ static inline bool vma_start_read(struct vm_area_struct *vma)if (READ_ONCE (vma->vm_lock_seq) ==READ_ONCE (vma->vm_mm->mm_lock_seq.sequence))return false; + +rwsem_acquire_read (&vma->vmlock_dep_map,0 ,0 , _RET_IP_); +/* Limit at VMA_STATE_LOCKED - 2 to leave one count for a writer */ +if (unlikely (!__refcount_inc_not_zero_limited (&vma->vm_refcnt, &oldcnt, +VMA_STATE_LOCKED -2 ))) { +rwsem_release (&vma->vmlock_dep_map, _RET_IP_);return false; + } +lock_acquired (&vma->vmlock_dep_map, _RET_IP_);/* +* Overflow of vm_lock_seq/mm_lock_seq might produce false locked result. * False unlocked result is impossible because we modify and check +* vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq * modification invalidates all existing locks. * * We must use ACQUIRE semantics for the mm_lock_seq so that if we are include/linux/mm.h static inline bool vma_start_read(struct vm_area_struct *vma) · line 766
Quoted 6 lines
include/linux/mm.h +3
-0@@ -734,10 +761,12 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* after it has been unlocked. * This pairs with RELEASE semantics in vma_end_write_all(). */ +if (oldcnt &VMA_STATE_LOCKED || +unlikely (vma->vm_lock_seq ==raw_read_seqcount (&vma->vm_mm->mm_lock_seq))) { +vma_refcount_put (vma);Suppose we have detach race with a concurrent RCU lookup like:
Quoted 16 lines
vma = mas_lookup(); vma_start_write(); mas_detach(); vma_start_read() rwsem_acquire_read() inc // success vma_mark_detach(); dec_and_test // assumes 1->0 // is actually 2->1 if (vm_lock_seq == vma->vm_mm_mm_lock_seq) // true vma_refcount_put dec_and_test() // 1->0 *NO* rwsem_release()Quoted 8 lines
Yes, this is possible. I think that's not a problem until we start reusing the vmas and I deal with this race later in this patchset. I think what you described here is the same race I mention in the description of this patch:
https://lore.kernel.org/all/[email protected]/
I introducevma_ensure_detached()in that patch to handle this case and ensure that vmas are detached before they are returned into the slab cache for reuse. Does that make sense?
Quoted 2 lines
So I just replied there, and no, I don't think it makes sense. Just put the
kmem_cache_free()invma_refcount_put(), to be done on 0.
That's very appealing indeed and makes things much simpler. The problem I see with that is the case when we detach a vma from the tree to isolate it, then do some cleanup and only then free it. That's done in vms_gather_munmap_vmas() here:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1240 and we even might reattach detached vmas back:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1312. IOW, detached state is not final and we can't destroy the object that reached this state. We could change states to: 0=unused (we can free the object), 1=detached, 2=attached, etc. but then vma_start_read() should do something like refcount_inc_more_than_one() instead of refcount_inc_not_zero(). Would you be ok with such an approach?
Quoted 4 lines
Anyway, my point was more about the weird entanglement of lockdep and the refcount. Just pull the lockdep annotation out of
_put()and put it explicitly in thevma_start_read()error paths andvma_end_read().
Ok, I think that's easy.
Quoted 6 lines
Additionally, having
vma_end_write()would allow you to put a lockdep annotation in vma_{start,end}_write() -- which was I think the original reason I proposed it a while back, that and having improved clarity when reading the code, since explicitly marking the end of a section is helpful.
The vma->vmlock_dep_map is tracking vma->vm_refcnt, not the vma->vm_lock_seq (similar to how today vma->vm_lock has its lockdep tracking that rw_semaphore). If I implement vma_end_write() then it will simply be something like:
void vma_end_write(vma)
{
vma_assert_write_locked(vma);
vma->vm_lock_seq = UINT_MAX;
}so, vmlock_dep_map would not be involved.
If you want to track vma->vm_lock_seq with a separate lockdep, that would be more complicated. Specifically for vma_end_write_all() that would require us to call rwsem_release() on all locked vmas, however we currently do not track individual locked vmas. vma_end_write_all() allows us not to worry about tracking them, knowing that once we do mmap_write_unlock() they all will get unlocked with one increment of mm->mm_lock_seq. If your suggestion is to replace vma_end_write_all() with vma_end_write() and unlock vmas individually across the mm code, that would be a sizable effort. If that is indeed your ultimate goal, I can do that as a separate project: introduce vma_end_write(), gradually add them in required places (not yet sure how complex that would be), then retire vma_end_write_all() and add a lockdep for vma->vm_lock_seq.
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 12 lines · Suren Baghdasaryan
Quoted 2 lines
So I just replied there, and no, I don't think it makes sense. Just put the
kmem_cache_free()invma_refcount_put(), to be done on 0.That's very appealing indeed and makes things much simpler. The problem I see with that is the case when we detach a vma from the tree to isolate it, then do some cleanup and only then free it. That's done in
vms_gather_munmap_vmas()here:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1240 and we even might reattach detached vmas back:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1312. IOW, detached state is not final and we can't destroy the object that reached this state.
Urgh, so that's the munmap() path, but arguably when that fails, the map stays in place.
I think this means you're marking detached too soon; you should only mark detached once you reach the point of no return.
That said, once you've reached the point of no return; and are about to go remove the page-tables, you very much want to ensure a lack of concurrency.
So perhaps waiting for out-standing readers at this point isn't crazy.
Also, I'm having a very hard time reading this maple tree stuff :/ Afaict vms_gather_munmap_vmas() only adds the VMAs to be removed to a second tree, it does not in fact unlink them from the mm yet.
AFAICT it's vma_iter_clear_gfp() that actually wipes the vmas from the mm -- and that being able to fail is mind boggling and I suppose is what gives rise to much of this insanity :/
Anyway, I would expect remove_vma() to be the one that marks it detached (it's already unreachable through vma_lookup() at this point) and there you should wait for concurrent readers to bugger off.
Quoted 4 lines
We could change states to: 0=unused (we can free
the object), 1=detached, 2=attached, etc. but thenvma_start_read()should do something likerefcount_inc_more_than_one()instead ofrefcount_inc_not_zero(). Would you be ok with such an approach?
Urgh, I would strongly suggest ditching refcount_t if we go this route. The thing is; refcount_t should remain a 'simple' straight forward interface and not allow people to do the wrong thing. Its not meant to be the kitchen sink -- we have atomic_t for that.
Anyway, the more common scheme at that point is using -1 for 'free', I think folio->_mapcount uses that even. For that see:
atomic_add_negative*().
Quoted 11 lines
Quoted 5 lines
Additionally, having
vma_end_write()would allow you to put a lockdep annotation in vma_{start,end}_write() -- which was I think the original reason I proposed it a while back, that and having improved clarity when reading the code, since explicitly marking the end of a section is helpful.The
vma->vmlock_dep_mapis trackingvma->vm_refcnt, not thevma->vm_lock_seq(similar to how todayvma->vm_lockhas its lockdep tracking thatrw_semaphore). If I implementvma_end_write()then it will simply be something like:
Quoted 7 lines
void
vma_end_write(vma){ vma_assert_write_locked(vma); vma->vm_lock_seq = UINT_MAX; }so,
vmlock_dep_mapwould not be involved.
That's just weird; why would you not track vma_{start,end}_write() with the exclusive side of the 'rwsem' dep_map ?
Quoted 5 lines
If you want to track
vma->vm_lock_seqwith a separate lockdep, that would be more complicated. Specifically forvma_end_write_all()that would require us to callrwsem_release()on all locked vmas, however we currently do not track individual locked vmas.vma_end_write_all()allows us not to worry about tracking them, knowing that once we do
Quoted 8 lines
mmap_write_unlock()they all will get unlocked with one increment ofmm->mm_lock_seq. If your suggestion is to replacevma_end_write_all()withvma_end_write()and unlock vmas individually across the mm code, that would be a sizable effort. If that is indeed your ultimate goal, I can do that as a separate project: introducevma_end_write(), gradually add them in required places (not yet sure how complex that would be), then retirevma_end_write_all()and add a lockdep forvma->vm_lock_seq.
Yeah, so ultimately I think it would be clearer if you explicitly mark the point where the vma modification is 'done'. But I don't suppose we have to do that here.
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 31 lines · Peter Zijlstra
Quoted 12 lines · Suren Baghdasaryan
Quoted 2 lines
So I just replied there, and no, I don't think it makes sense. Just put the
kmem_cache_free()invma_refcount_put(), to be done on 0.That's very appealing indeed and makes things much simpler. The problem I see with that is the case when we detach a vma from the tree to isolate it, then do some cleanup and only then free it. That's done in
vms_gather_munmap_vmas()here:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1240 and we even might reattach detached vmas back:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1312. IOW, detached state is not final and we can't destroy the object that reached this state.Urgh, so that's the
munmap()path, but arguably when that fails, the map stays in place.I think this means you're marking detached too soon; you should only mark detached once you reach the point of no return.
That said, once you've reached the point of no return; and are about to go remove the page-tables, you very much want to ensure a lack of concurrency.
So perhaps waiting for out-standing readers at this point isn't crazy.
Also, I'm having a very hard time reading this maple tree stuff :/ Afaict
vms_gather_munmap_vmas()only adds the VMAs to be removed to a second tree, it does not in fact unlink them from the mm yet.
Quoted 7 lines
AFAICT it's
vma_iter_clear_gfp()that actually wipes the vmas from the mm -- and that being able to fail is mind boggling and I suppose is what gives rise to much of this insanity :/Anyway, I would expect
remove_vma()to be the one that marks it detached (it's already unreachable throughvma_lookup()at this point) and there you should wait for concurrent readers to bugger off.
Also, I think vma_start_write() in that gather look is too early, you're not actually going to change the VMA yet -- with obvious exception of the split cases.
That too should probably come after you've passes all the fail/unwind spots.
Something like so perhaps? (yeah, I know, I wrecked a bunch)
mm/vma.c +10 -7
@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms,struct vm_area_struct *vma;struct mm_struct *mm; +mas_for_each (mas_detach, vma,ULONG_MAX ) { +vma_start_write (next); +vma_mark_detached (next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count; mm->locked_vm -= vms->locked_vm;
@@ -1219,9 +1224,6 @@ static void reattach_vmas(struct ma_state *mas_detach)struct vm_area_struct *vma;mas_set (mas_detach,0 ); -mas_for_each (mas_detach, vma,ULONG_MAX ) -vma_mark_detached (vma, false); -__mt_destroy (mas_detach->tree); }
@@ -1289,13 +1291,11 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms,if (error) goto end_split_failed; } -vma_start_write (next);mas_set (mas_detach, vms->vma_count++); error =mas_store_gfp (mas_detach, next,GFP_KERNEL );if (error) goto munmap_gather_failed; -vma_mark_detached (next, true); nrpages =vma_pages (next); vms->nr_pages += nrpages;
@@ -1431,14 +1431,17 @@ int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma,struct vma_munmap_struct vms;int error; + error =mas_preallocate (vmi->mas); +if (error) + goto gather_failed; +init_vma_munmap (&vms, vmi, vma, start, end, uf, unlock); error =vms_gather_munmap_vmas (&vms, &mas_detach);if (error) goto gather_failed; error =vma_iter_clear_gfp (vmi, start, end,GFP_KERNEL ); -if (error) - goto clear_tree_failed; +VM_WARN_ON (error);/* Point of no return */ vms_complete_munmap_vmas (&vms, &mas_detach);
Re: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Peter Zijlstra
Quoted 12 lines · Suren Baghdasaryan
Quoted 2 lines
So I just replied there, and no, I don't think it makes sense. Just put the
kmem_cache_free()invma_refcount_put(), to be done on 0.That's very appealing indeed and makes things much simpler. The problem I see with that is the case when we detach a vma from the tree to isolate it, then do some cleanup and only then free it. That's done in
vms_gather_munmap_vmas()here:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1240 and we even might reattach detached vmas back:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1312. IOW, detached state is not final and we can't destroy the object that reached this state.Urgh, so that's the
munmap()path, but arguably when that fails, the map stays in place.I think this means you're marking detached too soon; you should only mark detached once you reach the point of no return.
That said, once you've reached the point of no return; and are about to go remove the page-tables, you very much want to ensure a lack of concurrency.
Quoted 5 lines
Quoted 5 lines
So perhaps waiting for out-standing readers at this point isn't crazy.
Also, I'm having a very hard time reading this maple tree stuff :/ Afaict
vms_gather_munmap_vmas()only adds the VMAs to be removed to a second tree, it does not in fact unlink them from the mm yet.
Yes, that's correct. I tried to make this clear with a gather/complete naming like other areas of the mm. I hope that helped.
Also, the comments for the function state that's what's going on:
* vms_gather_munmap_vmas() - Put all VMAs within a range into a maple tree
* for removal at a later date. Handles splitting first and last if necessary
* and marking the vmas as isolated.
... might be worth updating with new information.
Quoted 4 lines
Quoted 4 lines
AFAICT it's
vma_iter_clear_gfp()that actually wipes the vmas from the mm -- and that being able to fail is mind boggling and I suppose is what gives rise to much of this insanity :/
This is also correct. The maple tree is a b-tree variant that has internal nodes. When you write to it, including nulls, they are tracked and may need to allocate. This is a cost for rcu lookups; we will use the same or less memory in the end but must maintain a consistent view of the ranges.
But to put this into perspective, we get 16 nodes per 4k page, most writes will use 1 or 3 of these from a kmem_cache, so we are talking about a very unlikely possibility. Except when syzbot decides to fail random allocations.
We could preallocate for the write, but this section of the code is GFP_KERNEL, so we don't. Preallocation is an option to simplify the failure path though... which is what you did below.
Quoted 8 lines
Quoted 4 lines
Anyway, I would expect
remove_vma()to be the one that marks it detached (it's already unreachable throughvma_lookup()at this point) and there you should wait for concurrent readers to bugger off.Also, I think
vma_start_write()in that gather look is too early, you're not actually going to change the VMA yet -- with obvious exception of the split cases.
The split needs to start the write on the vma to avoid anyone reading it while it's being altered.
Quoted 3 lines
That too should probably come after you've passes all the fail/unwind spots.
Do you mean the split? I'd like to move the split later as well.. tracking that is a pain and may need an extra vma for when one vma is split twice before removing the middle part.
Actually, I think we need to allocate two (or at least one) vmas in this case and just pass one through to unmap (written only to the mas_detach tree?). It would be nice to find a way to NOT need to do that even.. I had tried to use a vma on the stack years ago, which didn't work out.
mm/vma.c static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, · line 1175
Quoted 11 lines
Something like so perhaps? (yeah, I know, I wrecked a bunch)
mm/vma.c +0
-0@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms,struct vm_area_struct *vma;struct mm_struct *mm;
mas_set(mas_detach, 0);
mm/vma.c static void reattach_vmas(struct ma_state *mas_detach) · line 1227
Quoted 13 lines
mm/vma.c +5
-0+mas_for_each (mas_detach, vma,ULONG_MAX ) { +vma_start_write (next); +vma_mark_detached (next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count; mm->locked_vm -= vms->locked_vm;@@ -1219,9 +1224,6 @@ static void reattach_vmas(struct ma_state *mas_detach)struct vm_area_struct *vma;mas_set (mas_detach,0 );
Drop the mas_set here.
Quoted 21 lines
mm/vma.c +0
-5-mas_for_each (mas_detach, vma,ULONG_MAX ) -vma_mark_detached (vma, false); -__mt_destroy (mas_detach->tree); }@@ -1289,13 +1291,11 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms,if (error) goto end_split_failed; } -vma_start_write (next);mas_set (mas_detach, vms->vma_count++); error =mas_store_gfp (mas_detach, next,GFP_KERNEL );if (error) goto munmap_gather_failed; -vma_mark_detached (next, true); nrpages =vma_pages (next); vms->nr_pages += nrpages;
mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1433
Quoted 3 lines
mm/vma.c +0
-0@@ -1431,14 +1431,17 @@ int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma,struct vma_munmap_struct vms;int error;
The preallocation needs to know the range being stored to know what's going to happen.
vma_iter_config(vmi, start, end);
mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1434
Quoted 1 line
mm/vma.c +1
-0+ error =mas_preallocate (vmi->mas);
We haven't had a need to have a vma iterator preallocate for storing a null, but we can add one for this.
mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1442
Quoted 8 lines
mm/vma.c +3
-0+if (error) + goto gather_failed; +init_vma_munmap (&vms, vmi, vma, start, end, uf, unlock); error =vms_gather_munmap_vmas (&vms, &mas_detach);if (error) goto gather_failed;
Drop this stuff.
mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1444
Quoted 4 lines
mm/vma.c +1
-2error =vma_iter_clear_gfp (vmi, start, end,GFP_KERNEL ); -if (error) - goto clear_tree_failed; +VM_WARN_ON (error);
Do this insteadvma_iter_config(vmi, start, end);vma_iter_clear(vmi);
Quoted 3 lines
mm/vma.c +0
-0/* Point of no return */ vms_complete_munmap_vmas (&vms, &mas_detach);
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 70 lines · Liam R. Howlett
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Peter Zijlstra
Quoted 12 lines · Suren Baghdasaryan
Quoted 2 lines
So I just replied there, and no, I don't think it makes sense. Just put the
kmem_cache_free()invma_refcount_put(), to be done on 0.That's very appealing indeed and makes things much simpler. The problem I see with that is the case when we detach a vma from the tree to isolate it, then do some cleanup and only then free it. That's done in
vms_gather_munmap_vmas()here:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1240 and we even might reattach detached vmas back:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1312. IOW, detached state is not final and we can't destroy the object that reached this state.Urgh, so that's the
munmap()path, but arguably when that fails, the map stays in place.I think this means you're marking detached too soon; you should only mark detached once you reach the point of no return.
That said, once you've reached the point of no return; and are about to go remove the page-tables, you very much want to ensure a lack of concurrency.
Quoted 5 lines
Quoted 5 lines
So perhaps waiting for out-standing readers at this point isn't crazy.
Also, I'm having a very hard time reading this maple tree stuff :/ Afaict
vms_gather_munmap_vmas()only adds the VMAs to be removed to a second tree, it does not in fact unlink them from the mm yet.Yes, that's correct. I tried to make this clear with a gather/complete naming like other areas of the mm. I hope that helped.
Also, the comments for the function state that's what's going on:
*
vms_gather_munmap_vmas()- Put all VMAs within a range into a maple tree
* for removal at a later date. Handles splitting first and last if necessary
* and marking the vmas as isolated.... might be worth updating with new information.
Quoted 4 lines
Quoted 4 lines
AFAICT it's
vma_iter_clear_gfp()that actually wipes the vmas from the mm -- and that being able to fail is mind boggling and I suppose is what gives rise to much of this insanity :/This is also correct. The maple tree is a b-tree variant that has internal nodes. When you write to it, including nulls, they are tracked and may need to allocate. This is a cost for rcu lookups; we will use the same or less memory in the end but must maintain a consistent view of the ranges.
But to put this into perspective, we get 16 nodes per 4k page, most writes will use 1 or 3 of these from a
kmem_cache, so we are talking about a very unlikely possibility. Except when syzbot decides to fail random allocations.We could preallocate for the write, but this section of the code is
GFP_KERNEL, so we don't. Preallocation is an option to simplify the failure path though... which is what you did below.Quoted 5 lines
Quoted 4 lines
Anyway, I would expect
remove_vma()to be the one that marks it detached (it's already unreachable throughvma_lookup()at this point) and there you should wait for concurrent readers to bugger off.
Quoted 6 lines
Quoted 3 lines
Also, I think
vma_start_write()in that gather look is too early, you're not actually going to change the VMA yet -- with obvious exception of the split cases.The split needs to start the write on the vma to avoid anyone reading it while it's being altered.
I think vma_start_write() should be done inside vms_gather_munmap_vmas() for __mmap_prepare() to work correctly:
__mmap_prepare
vms_gather_munmap_vmas
vms_clean_up_area // clears PTEs
...__mmap_complete
vms_complete_munmap_vmas
If we do not write-lock the vmas inside vms_gather_munmap_vmas(), we will be clearing PTEs from under a discoverable vma.
There might be other places like this too but I think we can move vma_mark_detach() like you suggested without moving vma_start_write() and that should be enough.
Quoted 100 lines
Quoted 4 lines
That too should probably come after you've passes all the fail/unwind spots.
Do you mean the split? I'd like to move the split later as well.. tracking that is a pain and may need an extra vma for when one vma is split twice before removing the middle part.
Actually, I think we need to allocate two (or at least one) vmas in this case and just pass one through to unmap (written only to the
mas_detachtree?). It would be nice to find a way to NOT need to do that even.. I had tried to use a vma on the stack years ago, which didn't work out.mm/vma.c static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, · line 1175
Quoted 11 lines
Something like so perhaps? (yeah, I know, I wrecked a bunch)
mm/vma.c +0
-0@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms,struct vm_area_struct *vma;struct mm_struct *mm;
mas_set(mas_detach, 0);mm/vma.c static void reattach_vmas(struct ma_state *mas_detach) · line 1227
Quoted 13 lines
mm/vma.c +5
-0+mas_for_each (mas_detach, vma,ULONG_MAX ) { +vma_start_write (next); +vma_mark_detached (next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count; mm->locked_vm -= vms->locked_vm;@@ -1219,9 +1224,6 @@ static void reattach_vmas(struct ma_state *mas_detach)struct vm_area_struct *vma;mas_set (mas_detach,0 );Drop the
mas_sethere.Quoted 21 lines
mm/vma.c +0
-5-mas_for_each (mas_detach, vma,ULONG_MAX ) -vma_mark_detached (vma, false); -__mt_destroy (mas_detach->tree); }@@ -1289,13 +1291,11 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms,if (error) goto end_split_failed; } -vma_start_write (next);mas_set (mas_detach, vms->vma_count++); error =mas_store_gfp (mas_detach, next,GFP_KERNEL );if (error) goto munmap_gather_failed; -vma_mark_detached (next, true); nrpages =vma_pages (next); vms->nr_pages += nrpages;mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1433
Quoted 3 lines
mm/vma.c +0
-0@@ -1431,14 +1431,17 @@ int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma,struct vma_munmap_struct vms;int error;The preallocation needs to know the range being stored to know what's going to happen.
vma_iter_config(vmi, start, end);mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1434
Quoted 1 line
mm/vma.c +1
-0+ error =mas_preallocate (vmi->mas);We haven't had a need to have a vma iterator preallocate for storing a null, but we can add one for this.
mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1442
Quoted 8 lines
mm/vma.c +3
-0+if (error) + goto gather_failed; +init_vma_munmap (&vms, vmi, vma, start, end, uf, unlock); error =vms_gather_munmap_vmas (&vms, &mas_detach);if (error) goto gather_failed;Drop this stuff.
mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1444
Quoted 4 lines
mm/vma.c +1
-2error =vma_iter_clear_gfp (vmi, start, end,GFP_KERNEL ); -if (error) - goto clear_tree_failed; +VM_WARN_ON (error);Do this instead
vma_iter_config(vmi, start, end);vma_iter_clear(vmi);Quoted 3 lines
mm/vma.c +0
-0/* Point of no return */ vms_complete_munmap_vmas (&vms, &mas_detach);
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 9 lines · Suren Baghdasaryan
I think
vma_start_write()should be done insidevms_gather_munmap_vmas()for__mmap_prepare()to work correctly:
__mmap_prepare
vms_gather_munmap_vmas
vms_clean_up_area// clears PTEs
...__mmap_complete
vms_complete_munmap_vmas
I'm unsure what exactly you mean; __split_vma() will start_write on the one that is broken up and the rest won't actually change until vms_complete_munmap_vmas().
Quoted 2 lines
If we do not write-lock the vmas inside
vms_gather_munmap_vmas(), we will be clearing PTEs from under a discoverable vma.
You will not. vms_complete_munmap_vmas() will call remove_vma() to remove PTEs IIRC, and if you do start_write() and detach() before dropping mmap_lock_write, you should be good.
Quoted 3 lines
There might be other places like this too but I think we can move
vma_mark_detach()like you suggested without movingvma_start_write()and that should be enough.
I really don't see why you can't move vma_start_write() -- note that by moving that after you've unhooked the vmas from the mm (which you have by that point) you get the sync point you wanted.
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 13 lines · Peter Zijlstra
Quoted 9 lines · Suren Baghdasaryan
I think
vma_start_write()should be done insidevms_gather_munmap_vmas()for__mmap_prepare()to work correctly:
__mmap_prepare
vms_gather_munmap_vmas
vms_clean_up_area// clears PTEs
...__mmap_complete
vms_complete_munmap_vmas
Quoted 3 lines
I'm unsure what exactly you mean;
__split_vma()willstart_writeon the one that is broken up and the rest won't actually change untilvms_complete_munmap_vmas().
Ah, sorry, I missed the write-locking in the __split_vma(). Looks like indeed vma_start_write() is not needed in vms_gather_munmap_vmas().
Quoted 7 lines
Quoted 3 lines
If we do not write-lock the vmas inside
vms_gather_munmap_vmas(), we will be clearing PTEs from under a discoverable vma.You will not.
vms_complete_munmap_vmas()will callremove_vma()to remove PTEs IIRC, and if you dostart_write()anddetach()before droppingmmap_lock_write, you should be good.
Ok, I think we will have to move mmap_write_downgrade() inside vms_complete_munmap_vmas() to be called after remove_vma(). vms_clear_ptes() is using vmas, so we can't move remove_vma() before mmap_write_downgrade().
Quoted 10 lines
Quoted 4 lines
There might be other places like this too but I think we can move
vma_mark_detach()like you suggested without movingvma_start_write()and that should be enough.I really don't see why you can't move
vma_start_write()-- note that by moving that after you've unhooked the vmas from the mm (which you have by that point) you get the sync point you wanted.
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 8 lines · Suren Baghdasaryan
Quoted 3 lines
You will not.
vms_complete_munmap_vmas()will callremove_vma()to remove PTEs IIRC, and if you dostart_write()anddetach()before droppingmmap_lock_write, you should be good.Ok, I think we will have to move
mmap_write_downgrade()insidevms_complete_munmap_vmas()to be called afterremove_vma().vms_clear_ptes()is using vmas, so we can't moveremove_vma()beforemmap_write_downgrade().
Why ?!
vms_clear_ptes() and remove_vma() are fine where they are -- there is no concurrency left at this point.
Note that by doing vma_start_write() inside vms_complete_munmap_vmas(), which is *after* the vmas have been unhooked from the mm, you wait for any concurrent user to go away.
And since they're unhooked, there can't be any new users.
So you're the one and only user left, and code is fine the way it is.
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 17 lines · Peter Zijlstra
Quoted 8 lines · Suren Baghdasaryan
Quoted 3 lines
You will not.
vms_complete_munmap_vmas()will callremove_vma()to remove PTEs IIRC, and if you dostart_write()anddetach()before droppingmmap_lock_write, you should be good.Ok, I think we will have to move
mmap_write_downgrade()insidevms_complete_munmap_vmas()to be called afterremove_vma().vms_clear_ptes()is using vmas, so we can't moveremove_vma()beforemmap_write_downgrade().Why ?!
vms_clear_ptes()andremove_vma()are fine where they are -- there is no concurrency left at this point.
Quoted 7 lines
Note that by doing
vma_start_write()insidevms_complete_munmap_vmas(), which is *after* the vmas have been unhooked from the mm, you wait for any concurrent user to go away.And since they're unhooked, there can't be any new users.
So you're the one and only user left, and code is fine the way it is.
Ok, let me make sure I understand this part of your proposal. From your earlier email:
@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms,
struct vm_area_struct *vma;
struct mm_struct *mm;+ mas_for_each(mas_detach, vma, ULONG_MAX) {
+ vma_start_write(next);
+ vma_mark_detached(next, true);
+ }
+
mm = current->mm;
mm->map_count -= vms->vma_count;
mm->locked_vm -= vms->locked_vm;This would mean:
vms_complete_munmap_vmas
vma_start_write
vma_mark_detached
mmap_write_downgrade
vms_clear_ptes
remove_vma
And remove_vma will be just freeing the vmas. Is that correct? I'm a bit confused because the original thinking was that vma_mark_detached() would drop the last refcnt and if it's 0 we would free the vma right there. If that's still what we want to do then I think the above sequence should look like this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_start_write
vma_mark_detached
mmap_write_downgrade
because vma_start_write+vma_mark_detached should be done under mmap_write_lock. Please let me know which way you want to move forward.
Quoted 2 lines
Re: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 65 lines · Suren Baghdasaryan
Quoted 17 lines · Peter Zijlstra
Quoted 8 lines · Suren Baghdasaryan
Quoted 3 lines
You will not.
vms_complete_munmap_vmas()will callremove_vma()to remove PTEs IIRC, and if you dostart_write()anddetach()before droppingmmap_lock_write, you should be good.Ok, I think we will have to move
mmap_write_downgrade()insidevms_complete_munmap_vmas()to be called afterremove_vma().vms_clear_ptes()is using vmas, so we can't moveremove_vma()beforemmap_write_downgrade().Why ?!
vms_clear_ptes()andremove_vma()are fine where they are -- there is no concurrency left at this point.Quoted 7 lines
Note that by doing
vma_start_write()insidevms_complete_munmap_vmas(), which is *after* the vmas have been unhooked from the mm, you wait for any concurrent user to go away.And since they're unhooked, there can't be any new users.
So you're the one and only user left, and code is fine the way it is.
Ok, let me make sure I understand this part of your proposal. From your earlier email:
(quoted diff) +5
-0@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, struct vm_area_struct *vma; struct mm_struct *mm; + mas_for_each(mas_detach, vma, ULONG_MAX) { + vma_start_write(next); + vma_mark_detached(next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count;mm->locked_vm -= vms->locked_vm;This would mean:
vms_complete_munmap_vmas
vma_start_write
vma_mark_detached
mmap_write_downgrade
vms_clear_ptes
remove_vmaAnd
remove_vmawill be just freeing the vmas. Is that correct? I'm a bit confused because the original thinking was thatvma_mark_detached()would drop the last refcnt and if it's 0 we would free the vma right there. If that's still what we want to do then I think the above sequence should look like this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_start_write
vma_mark_detached
mmap_write_downgrade
Quoted 3 lines
because vma_start_write+vma_mark_detached should be done under
mmap_write_lock. Please let me know which way you want to move forward.
Are we sure we're not causing issues with the MAP_FIXED path here?
With the above change, we'd be freeing the PTEs before marking the vmas as detached or vma_start_write().
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 67 lines · Liam R. Howlett' via kernel-team
Quoted 65 lines · Suren Baghdasaryan
Quoted 17 lines · Peter Zijlstra
Quoted 8 lines · Suren Baghdasaryan
Quoted 3 lines
You will not.
vms_complete_munmap_vmas()will callremove_vma()to remove PTEs IIRC, and if you dostart_write()anddetach()before droppingmmap_lock_write, you should be good.Ok, I think we will have to move
mmap_write_downgrade()insidevms_complete_munmap_vmas()to be called afterremove_vma().vms_clear_ptes()is using vmas, so we can't moveremove_vma()beforemmap_write_downgrade().Why ?!
vms_clear_ptes()andremove_vma()are fine where they are -- there is no concurrency left at this point.Quoted 7 lines
Note that by doing
vma_start_write()insidevms_complete_munmap_vmas(), which is *after* the vmas have been unhooked from the mm, you wait for any concurrent user to go away.And since they're unhooked, there can't be any new users.
So you're the one and only user left, and code is fine the way it is.
Ok, let me make sure I understand this part of your proposal. From your earlier email:
(quoted diff) +5
-0@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, struct vm_area_struct *vma; struct mm_struct *mm; + mas_for_each(mas_detach, vma, ULONG_MAX) { + vma_start_write(next); + vma_mark_detached(next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count;mm->locked_vm -= vms->locked_vm;This would mean:
vms_complete_munmap_vmas
vma_start_write
vma_mark_detached
mmap_write_downgrade
vms_clear_ptes
remove_vmaAnd
remove_vmawill be just freeing the vmas. Is that correct? I'm a bit confused because the original thinking was thatvma_mark_detached()would drop the last refcnt and if it's 0 we would free the vma right there. If that's still what we want to do then I think the above sequence should look like this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_start_write
vma_mark_detached
mmap_write_downgrade
Quoted 8 lines
Quoted 3 lines
because vma_start_write+vma_mark_detached should be done under
mmap_write_lock. Please let me know which way you want to move forward.Are we sure we're not causing issues with the
MAP_FIXEDpath here?With the above change, we'd be freeing the PTEs before marking the vmas as detached or
vma_start_write().
IIUC when we call vms_complete_munmap_vmas() all vmas inside mas_detach have been already write-locked, no?
Quoted 3 lines
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 76 lines · Suren Baghdasaryan
Quoted 73 lines · Liam R. Howlett' via kernel-team
Quoted 65 lines · Suren Baghdasaryan
Quoted 17 lines · Peter Zijlstra
Quoted 8 lines · Suren Baghdasaryan
Quoted 3 lines
You will not.
vms_complete_munmap_vmas()will callremove_vma()to remove PTEs IIRC, and if you dostart_write()anddetach()before droppingmmap_lock_write, you should be good.Ok, I think we will have to move
mmap_write_downgrade()insidevms_complete_munmap_vmas()to be called afterremove_vma().vms_clear_ptes()is using vmas, so we can't moveremove_vma()beforemmap_write_downgrade().Why ?!
vms_clear_ptes()andremove_vma()are fine where they are -- there is no concurrency left at this point.Quoted 7 lines
Note that by doing
vma_start_write()insidevms_complete_munmap_vmas(), which is *after* the vmas have been unhooked from the mm, you wait for any concurrent user to go away.And since they're unhooked, there can't be any new users.
So you're the one and only user left, and code is fine the way it is.
Ok, let me make sure I understand this part of your proposal. From your earlier email:
(quoted diff) +5
-0@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, struct vm_area_struct *vma; struct mm_struct *mm; + mas_for_each(mas_detach, vma, ULONG_MAX) { + vma_start_write(next); + vma_mark_detached(next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count;mm->locked_vm -= vms->locked_vm;This would mean:
vms_complete_munmap_vmas
vma_start_write
vma_mark_detached
mmap_write_downgrade
vms_clear_ptes
remove_vmaAnd
remove_vmawill be just freeing the vmas. Is that correct? I'm a bit confused because the original thinking was thatvma_mark_detached()would drop the last refcnt and if it's 0 we would free the vma right there. If that's still what we want to do then I think the above sequence should look like this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_start_write
vma_mark_detached
mmap_write_downgradeQuoted 3 lines
because vma_start_write+vma_mark_detached should be done under
mmap_write_lock. Please let me know which way you want to move forward.Are we sure we're not causing issues with the
MAP_FIXEDpath here?
Quoted 5 lines
Quoted 2 lines
With the above change, we'd be freeing the PTEs before marking the vmas as detached or
vma_start_write().IIUC when we call
vms_complete_munmap_vmas()all vmas insidemas_detachhave been already write-locked, no?
Yeah, I think we can simply do this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_mark_detached
mmap_write_downgrade
If my assumption is incorrect, assertion inside vma_mark_detached() should trigger. I tried a quick test and so far nothing exploded.
Quoted 4 lines
Quoted 4 lines
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Re: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 77 lines · Suren Baghdasaryan
Quoted 76 lines · Suren Baghdasaryan
Quoted 73 lines · Liam R. Howlett' via kernel-team
Quoted 65 lines · Suren Baghdasaryan
Quoted 17 lines · Peter Zijlstra
Quoted 8 lines · Suren Baghdasaryan
Quoted 3 lines
You will not.
vms_complete_munmap_vmas()will callremove_vma()to remove PTEs IIRC, and if you dostart_write()anddetach()before droppingmmap_lock_write, you should be good.Ok, I think we will have to move
mmap_write_downgrade()insidevms_complete_munmap_vmas()to be called afterremove_vma().vms_clear_ptes()is using vmas, so we can't moveremove_vma()beforemmap_write_downgrade().Why ?!
vms_clear_ptes()andremove_vma()are fine where they are -- there is no concurrency left at this point.Quoted 7 lines
Note that by doing
vma_start_write()insidevms_complete_munmap_vmas(), which is *after* the vmas have been unhooked from the mm, you wait for any concurrent user to go away.And since they're unhooked, there can't be any new users.
So you're the one and only user left, and code is fine the way it is.
Ok, let me make sure I understand this part of your proposal. From your earlier email:
(quoted diff) +5
-0@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, struct vm_area_struct *vma; struct mm_struct *mm; + mas_for_each(mas_detach, vma, ULONG_MAX) { + vma_start_write(next); + vma_mark_detached(next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count;mm->locked_vm -= vms->locked_vm;This would mean:
vms_complete_munmap_vmas
vma_start_write
vma_mark_detached
mmap_write_downgrade
vms_clear_ptes
remove_vmaAnd
remove_vmawill be just freeing the vmas. Is that correct? I'm a bit confused because the original thinking was thatvma_mark_detached()would drop the last refcnt and if it's 0 we would free the vma right there. If that's still what we want to do then I think the above sequence should look like this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_start_write
vma_mark_detached
mmap_write_downgradeQuoted 3 lines
because vma_start_write+vma_mark_detached should be done under
mmap_write_lock. Please let me know which way you want to move forward.Are we sure we're not causing issues with the
MAP_FIXEDpath here?
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
With the above change, we'd be freeing the PTEs before marking the vmas as detached or
vma_start_write().IIUC when we call
vms_complete_munmap_vmas()all vmas insidemas_detachhave been already write-locked, no?
That's the way it is today - but I thought you were moving the lock to the complete stage, not adding a new one? (why add a new one otherwise?)
Quoted 12 lines
Yeah, I think we can simply do this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_mark_detached
mmap_write_downgradeIf my assumption is incorrect, assertion inside
vma_mark_detached()should trigger. I tried a quick test and so far nothing exploded.
If they are write locked, then the page faults are not a concern. There is also the rmap race that Jann found in mmap_region() [1]. This is probably also fine since you are keeping the write lock in place earlier on in the gather stage. Note the ptes will already be cleared by the time vms_complete_munmap_vmas() is called in this case.
[1] https://lore.kernel.org/all/CAG48ez0ZpGzxi=-5O_uGQ0xKXOmbjeQ0LjZsRJ1Qtf2X5eOr1w@mail.gmail.com/
Thanks,
Liam
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 79 lines · Liam R. Howlett' via kernel-team
Quoted 77 lines · Suren Baghdasaryan
Quoted 76 lines · Suren Baghdasaryan
Quoted 73 lines · Liam R. Howlett' via kernel-team
Quoted 65 lines · Suren Baghdasaryan
Quoted 17 lines · Peter Zijlstra
Quoted 8 lines · Suren Baghdasaryan
Quoted 3 lines
You will not.
vms_complete_munmap_vmas()will callremove_vma()to remove PTEs IIRC, and if you dostart_write()anddetach()before droppingmmap_lock_write, you should be good.Ok, I think we will have to move
mmap_write_downgrade()insidevms_complete_munmap_vmas()to be called afterremove_vma().vms_clear_ptes()is using vmas, so we can't moveremove_vma()beforemmap_write_downgrade().Why ?!
vms_clear_ptes()andremove_vma()are fine where they are -- there is no concurrency left at this point.Quoted 7 lines
Note that by doing
vma_start_write()insidevms_complete_munmap_vmas(), which is *after* the vmas have been unhooked from the mm, you wait for any concurrent user to go away.And since they're unhooked, there can't be any new users.
So you're the one and only user left, and code is fine the way it is.
Ok, let me make sure I understand this part of your proposal. From your earlier email:
(quoted diff) +5
-0@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, struct vm_area_struct *vma; struct mm_struct *mm; + mas_for_each(mas_detach, vma, ULONG_MAX) { + vma_start_write(next); + vma_mark_detached(next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count;mm->locked_vm -= vms->locked_vm;This would mean:
vms_complete_munmap_vmas
vma_start_write
vma_mark_detached
mmap_write_downgrade
vms_clear_ptes
remove_vmaAnd
remove_vmawill be just freeing the vmas. Is that correct? I'm a bit confused because the original thinking was thatvma_mark_detached()would drop the last refcnt and if it's 0 we would free the vma right there. If that's still what we want to do then I think the above sequence should look like this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_start_write
vma_mark_detached
mmap_write_downgradeQuoted 3 lines
because vma_start_write+vma_mark_detached should be done under
mmap_write_lock. Please let me know which way you want to move forward.Are we sure we're not causing issues with the
MAP_FIXEDpath here?
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
With the above change, we'd be freeing the PTEs before marking the vmas as detached or
vma_start_write().IIUC when we call
vms_complete_munmap_vmas()all vmas insidemas_detachhave been already write-locked, no?That's the way it is today - but I thought you were moving the lock to the complete stage, not adding a new one? (why add a new one otherwise?)
Is my understanding correct that mas_detach is populated by vms_gather_munmap_vmas() only with vmas that went through __split_vma() (and were write-locked there)? I don't see any path that would add any other vma into mas_detach but maybe I'm missing something?
Quoted 27 lines
Quoted 4 lines
Yeah, I think we can simply do this:
Quoted 9 lines
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_mark_detached
mmap_write_downgradeIf my assumption is incorrect, assertion inside
vma_mark_detached()should trigger. I tried a quick test and so far nothing exploded.If they are write locked, then the page faults are not a concern. There is also the rmap race that Jann found in
mmap_region()[1]. This is probably also fine since you are keeping the write lock in place earlier on in the gather stage. Note the ptes will already be cleared by the timevms_complete_munmap_vmas()is called in this case.[1] https://lore.kernel.org/all/CAG48ez0ZpGzxi=-5O_uGQ0xKXOmbjeQ0LjZsRJ1Qtf2X5eOr1w@mail.gmail.com/
Thanks,
LiamTo unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Re: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 87 lines · Suren Baghdasaryan
Quoted 85 lines · Liam R. Howlett' via kernel-team
Quoted 82 lines · Suren Baghdasaryan
Quoted 81 lines · Suren Baghdasaryan
Quoted 67 lines · Liam R. Howlett' via kernel-team
Quoted 65 lines · Suren Baghdasaryan
Quoted 17 lines · Peter Zijlstra
Quoted 8 lines · Suren Baghdasaryan
Quoted 3 lines
You will not.
vms_complete_munmap_vmas()will callremove_vma()to remove PTEs IIRC, and if you dostart_write()anddetach()before droppingmmap_lock_write, you should be good.Ok, I think we will have to move
mmap_write_downgrade()insidevms_complete_munmap_vmas()to be called afterremove_vma().vms_clear_ptes()is using vmas, so we can't moveremove_vma()beforemmap_write_downgrade().Why ?!
vms_clear_ptes()andremove_vma()are fine where they are -- there is no concurrency left at this point.Quoted 7 lines
Note that by doing
vma_start_write()insidevms_complete_munmap_vmas(), which is *after* the vmas have been unhooked from the mm, you wait for any concurrent user to go away.And since they're unhooked, there can't be any new users.
So you're the one and only user left, and code is fine the way it is.
Ok, let me make sure I understand this part of your proposal. From your earlier email:
(quoted diff) +5
-0@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, struct vm_area_struct *vma; struct mm_struct *mm; + mas_for_each(mas_detach, vma, ULONG_MAX) { + vma_start_write(next); + vma_mark_detached(next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count;mm->locked_vm -= vms->locked_vm;This would mean:
vms_complete_munmap_vmas
vma_start_write
vma_mark_detached
mmap_write_downgrade
vms_clear_ptes
remove_vmaAnd
remove_vmawill be just freeing the vmas. Is that correct? I'm a bit confused because the original thinking was thatvma_mark_detached()would drop the last refcnt and if it's 0 we would free the vma right there. If that's still what we want to do then I think the above sequence should look like this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_start_write
vma_mark_detached
mmap_write_downgradeQuoted 8 lines
Quoted 3 lines
because vma_start_write+vma_mark_detached should be done under
mmap_write_lock. Please let me know which way you want to move forward.Are we sure we're not causing issues with the
MAP_FIXEDpath here?With the above change, we'd be freeing the PTEs before marking the vmas as detached or
vma_start_write().IIUC when we call
vms_complete_munmap_vmas()all vmas insidemas_detachhave been already write-locked, no?
Quoted 8 lines
Quoted 2 lines
That's the way it is today - but I thought you were moving the lock to the complete stage, not adding a new one? (why add a new one otherwise?)
Is my understanding correct that
mas_detachis populated byvms_gather_munmap_vmas()only with vmas that went through__split_vma()(and were write-locked there)? I don't see any path that would add any other vma intomas_detachbut maybe I'm missing something?
No, that is not correct.
vms_gather_munmap_vmas() calls split on the first vma, then adds all vmas that are within the range of the munmap() call. Potentially splitting the last vma and adding that in the
"if (next->vm_end > vms->end)" block.
Sometimes this is a single vma that gets split twice, sometimes no splits happen and entire vmas are unmapped, sometimes it's just one vma that isn't split.
My observation is the common case is a single vma, but besides that we see 3, and sometimes 7 at a time, but it could be any number of vmas and not all of them are split.
There is a loop for_each_vma_range() that does:
vma_start_write(next);mas_set(mas_detach, vms->mas_count++);mas_store_gfp(mas_detach, next, GFP_KERNEL);
Quoted 28 lines
Quoted 28 lines
Quoted 5 lines
Yeah, I think we can simply do this:
Quoted 9 lines
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_mark_detached
mmap_write_downgradeIf my assumption is incorrect, assertion inside
vma_mark_detached()should trigger. I tried a quick test and so far nothing exploded.If they are write locked, then the page faults are not a concern. There is also the rmap race that Jann found in
mmap_region()[1]. This is probably also fine since you are keeping the write lock in place earlier on in the gather stage. Note the ptes will already be cleared by the timevms_complete_munmap_vmas()is called in this case.[1] https://lore.kernel.org/all/CAG48ez0ZpGzxi=-5O_uGQ0xKXOmbjeQ0LjZsRJ1Qtf2X5eOr1w@mail.gmail.com/
Thanks,
LiamTo unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 113 lines · Liam R. Howlett
Quoted 87 lines · Suren Baghdasaryan
Quoted 85 lines · Liam R. Howlett' via kernel-team
Quoted 82 lines · Suren Baghdasaryan
Quoted 81 lines · Suren Baghdasaryan
Quoted 67 lines · Liam R. Howlett' via kernel-team
Quoted 65 lines · Suren Baghdasaryan
Quoted 17 lines · Peter Zijlstra
Quoted 8 lines · Suren Baghdasaryan
Quoted 3 lines
You will not.
vms_complete_munmap_vmas()will callremove_vma()to remove PTEs IIRC, and if you dostart_write()anddetach()before droppingmmap_lock_write, you should be good.Ok, I think we will have to move
mmap_write_downgrade()insidevms_complete_munmap_vmas()to be called afterremove_vma().vms_clear_ptes()is using vmas, so we can't moveremove_vma()beforemmap_write_downgrade().Why ?!
vms_clear_ptes()andremove_vma()are fine where they are -- there is no concurrency left at this point.Quoted 7 lines
Note that by doing
vma_start_write()insidevms_complete_munmap_vmas(), which is *after* the vmas have been unhooked from the mm, you wait for any concurrent user to go away.And since they're unhooked, there can't be any new users.
So you're the one and only user left, and code is fine the way it is.
Ok, let me make sure I understand this part of your proposal. From your earlier email:
(quoted diff) +5
-0@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, struct vm_area_struct *vma; struct mm_struct *mm; + mas_for_each(mas_detach, vma, ULONG_MAX) { + vma_start_write(next); + vma_mark_detached(next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count;mm->locked_vm -= vms->locked_vm;This would mean:
vms_complete_munmap_vmas
vma_start_write
vma_mark_detached
mmap_write_downgrade
vms_clear_ptes
remove_vmaAnd
remove_vmawill be just freeing the vmas. Is that correct? I'm a bit confused because the original thinking was thatvma_mark_detached()would drop the last refcnt and if it's 0 we would free the vma right there. If that's still what we want to do then I think the above sequence should look like this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_start_write
vma_mark_detached
mmap_write_downgradeQuoted 8 lines
Quoted 3 lines
because vma_start_write+vma_mark_detached should be done under
mmap_write_lock. Please let me know which way you want to move forward.Are we sure we're not causing issues with the
MAP_FIXEDpath here?With the above change, we'd be freeing the PTEs before marking the vmas as detached or
vma_start_write().IIUC when we call
vms_complete_munmap_vmas()all vmas insidemas_detachhave been already write-locked, no?Quoted 8 lines
Quoted 2 lines
That's the way it is today - but I thought you were moving the lock to the complete stage, not adding a new one? (why add a new one otherwise?)
Is my understanding correct that
mas_detachis populated byvms_gather_munmap_vmas()only with vmas that went through__split_vma()(and were write-locked there)? I don't see any path that would add any other vma intomas_detachbut maybe I'm missing something?No, that is not correct.
vms_gather_munmap_vmas()calls split on the first vma, then adds all vmas that are within the range of themunmap()call. Potentially splitting the last vma and adding that in the
"if (next->vm_end>vms->end)" block.Sometimes this is a single vma that gets split twice, sometimes no splits happen and entire vmas are unmapped, sometimes it's just one vma that isn't split.
My observation is the common case is a single vma, but besides that we see 3, and sometimes 7 at a time, but it could be any number of vmas and not all of them are split.
Quoted 5 lines
There is a loop
for_each_vma_range()that does:
vma_start_write(next);mas_set(mas_detach, vms->mas_count++);mas_store_gfp(mas_detach, next, GFP_KERNEL);
Ah, ok I see now. I completely misunderstood what for_each_vma_range() was doing.
Then I think vma_start_write() should remain inside vms_gather_munmap_vmas() and all vmas in mas_detach should be write-locked, even the ones we are not modifying. Otherwise what would prevent the race I mentioned before?
__mmap_region
__mmap_prepare
vms_gather_munmap_vmas // adds vmas to be unmapped into mas_detach,
// some locked
by __split_vma(), some not locked
lock_vma_under_rcu()
vma = mas_walk // finds
unlocked vma also in mas_detach
vma_start_read(vma) //
succeeds since vma is not locked
// vma->detached, vm_start,
vm_end checks pass
// vma is successfully read-locked
vms_clean_up_area(mas_detach)
vms_clear_ptes
// steps on a cleared PTE
__mmap_new_vma
vma_set_range // installs new vma in the range
__mmap_complete
vms_complete_munmap_vmas // vmas are write-locked and detachedbut it's too late
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 7 lines
Yeah, I think we can simply do this:
Quoted 9 lines
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_mark_detached
mmap_write_downgradeIf my assumption is incorrect, assertion inside
vma_mark_detached()should trigger. I tried a quick test and so far nothing exploded.If they are write locked, then the page faults are not a concern. There is also the rmap race that Jann found in
mmap_region()[1]. This is probably also fine since you are keeping the write lock in place earlier on in the gather stage. Note the ptes will already be cleared by the timevms_complete_munmap_vmas()is called in this case.[1] https://lore.kernel.org/all/CAG48ez0ZpGzxi=-5O_uGQ0xKXOmbjeQ0LjZsRJ1Qtf2X5eOr1w@mail.gmail.com/
Thanks,
LiamTo unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 152 lines · Suren Baghdasaryan
Quoted 113 lines · Liam R. Howlett
Quoted 87 lines · Suren Baghdasaryan
Quoted 85 lines · Liam R. Howlett' via kernel-team
Quoted 82 lines · Suren Baghdasaryan
Quoted 81 lines · Suren Baghdasaryan
Quoted 67 lines · Liam R. Howlett' via kernel-team
Quoted 65 lines · Suren Baghdasaryan
Quoted 17 lines · Peter Zijlstra
Quoted 8 lines · Suren Baghdasaryan
Quoted 3 lines
You will not.
vms_complete_munmap_vmas()will callremove_vma()to remove PTEs IIRC, and if you dostart_write()anddetach()before droppingmmap_lock_write, you should be good.Ok, I think we will have to move
mmap_write_downgrade()insidevms_complete_munmap_vmas()to be called afterremove_vma().vms_clear_ptes()is using vmas, so we can't moveremove_vma()beforemmap_write_downgrade().Why ?!
vms_clear_ptes()andremove_vma()are fine where they are -- there is no concurrency left at this point.Quoted 7 lines
Note that by doing
vma_start_write()insidevms_complete_munmap_vmas(), which is *after* the vmas have been unhooked from the mm, you wait for any concurrent user to go away.And since they're unhooked, there can't be any new users.
So you're the one and only user left, and code is fine the way it is.
Ok, let me make sure I understand this part of your proposal. From your earlier email:
(quoted diff) +5
-0@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, struct vm_area_struct *vma; struct mm_struct *mm; + mas_for_each(mas_detach, vma, ULONG_MAX) { + vma_start_write(next); + vma_mark_detached(next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count;mm->locked_vm -= vms->locked_vm;This would mean:
vms_complete_munmap_vmas
vma_start_write
vma_mark_detached
mmap_write_downgrade
vms_clear_ptes
remove_vmaAnd
remove_vmawill be just freeing the vmas. Is that correct? I'm a bit confused because the original thinking was thatvma_mark_detached()would drop the last refcnt and if it's 0 we would free the vma right there. If that's still what we want to do then I think the above sequence should look like this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_start_write
vma_mark_detached
mmap_write_downgradeQuoted 8 lines
Quoted 3 lines
because vma_start_write+vma_mark_detached should be done under
mmap_write_lock. Please let me know which way you want to move forward.Are we sure we're not causing issues with the
MAP_FIXEDpath here?With the above change, we'd be freeing the PTEs before marking the vmas as detached or
vma_start_write().IIUC when we call
vms_complete_munmap_vmas()all vmas insidemas_detachhave been already write-locked, no?Quoted 8 lines
Quoted 2 lines
That's the way it is today - but I thought you were moving the lock to the complete stage, not adding a new one? (why add a new one otherwise?)
Is my understanding correct that
mas_detachis populated byvms_gather_munmap_vmas()only with vmas that went through__split_vma()(and were write-locked there)? I don't see any path that would add any other vma intomas_detachbut maybe I'm missing something?No, that is not correct.
vms_gather_munmap_vmas()calls split on the first vma, then adds all vmas that are within the range of themunmap()call. Potentially splitting the last vma and adding that in the
"if (next->vm_end>vms->end)" block.Sometimes this is a single vma that gets split twice, sometimes no splits happen and entire vmas are unmapped, sometimes it's just one vma that isn't split.
My observation is the common case is a single vma, but besides that we see 3, and sometimes 7 at a time, but it could be any number of vmas and not all of them are split.
Quoted 5 lines
There is a loop
for_each_vma_range()that does:
vma_start_write(next);mas_set(mas_detach, vms->mas_count++);mas_store_gfp(mas_detach, next, GFP_KERNEL);Ah, ok I see now. I completely misunderstood what
for_each_vma_range()was doing.Then I think
vma_start_write()should remain insidevms_gather_munmap_vmas()and all vmas inmas_detachshould be write-locked, even the ones we are not modifying. Otherwise what would prevent the race I mentioned before?
__mmap_region
__mmap_prepare
vms_gather_munmap_vmas// adds vmas to be unmapped intomas_detach,
// some locked
by__split_vma(), some not locked
lock_vma_under_rcu()
vma =mas_walk// finds
unlocked vma also inmas_detachvma_start_read(vma) // succeeds since vma is not locked // vma->detached, vm_start, vm_end checks pass // vma is successfully read-locked vms_clean_up_area(mas_detach) vms_clear_ptes // steps on a cleared PTE __mmap_new_vma vma_set_range // installs new vma in the range __mmap_complete vms_complete_munmap_vmas // vmas are write-locked and detached
Quoted 1 line
but it's too late
Sorry about the formatting. Without comments should look better:
__mmap_region
__mmap_prepare
vms_gather_munmap_vmas
lock_vma_under_rcu()
vma = mas_walk
vma_start_read(vma)
// vma is still valid and attached
vms_clean_up_area(mas_detach)
vms_clear_ptes
// steps on a cleared PTE
__mmap_new_vma
vma_set_range
__mmap_complete
vms_complete_munmap_vmasQuoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 10 lines
Yeah, I think we can simply do this:
Quoted 9 lines
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_mark_detached
mmap_write_downgradeIf my assumption is incorrect, assertion inside
vma_mark_detached()should trigger. I tried a quick test and so far nothing exploded.If they are write locked, then the page faults are not a concern. There is also the rmap race that Jann found in
mmap_region()[1]. This is probably also fine since you are keeping the write lock in place earlier on in the gather stage. Note the ptes will already be cleared by the timevms_complete_munmap_vmas()is called in this case.[1] https://lore.kernel.org/all/CAG48ez0ZpGzxi=-5O_uGQ0xKXOmbjeQ0LjZsRJ1Qtf2X5eOr1w@mail.gmail.com/
Thanks,
LiamTo unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
Andrew Morton
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Suren BaghdasaryanQuoted 8 lines · On Wed, 18 Dec 2024 13:53:17 -0800 Suren Baghdasaryan <[email protected]> wrote:
Quoted 5 lines
There is a loop
for_each_vma_range()that does:
vma_start_write(next);mas_set(mas_detach, vms->mas_count++);mas_store_gfp(mas_detach, next, GFP_KERNEL);Ah, ok I see now. I completely misunderstood what
for_each_vma_range()was doing.
I'll drop the v6 series from mm-unstable.
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 5 lines · Andrew Morton
Quoted 2 lines · On Wed, 18 Dec 2024 13:53:17 -0800 Suren Baghdasaryan <[email protected]> wrote:
Quoted 2 lines
There is a loop
for_each_vma_range()that does:
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
vma_start_write(next);mas_set(mas_detach, vms->mas_count++);mas_store_gfp(mas_detach, next, GFP_KERNEL);Ah, ok I see now. I completely misunderstood what
for_each_vma_range()was doing.I'll drop the v6 series from mm-unstable.
Sounds good. v7 will be quite different.
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Suren BaghdasaryanQuoted 5 lines · Suren Baghdasaryan
Ah, ok I see now. I completely misunderstood what
for_each_vma_range()was doing.Then I think
vma_start_write()should remain insidevms_gather_munmap_vmas()and all vmas inmas_detachshould be
No, it must not. You really are not modifying anything yet (except the split, which we've already noted mark write themselves).
Quoted 12 lines
write-locked, even the ones we are not modifying. Otherwise what would prevent the race I mentioned before?
__mmap_region
__mmap_prepare
vms_gather_munmap_vmas// adds vmas to be unmapped intomas_detach,
// some locked
by__split_vma(), some not locked
lock_vma_under_rcu()
vma =mas_walk// finds
unlocked vma also inmas_detach
Quoted 9 lines
vma_start_read(vma) //
succeeds since vma is not locked
// vma->detached, vm_start,
vm_end checks pass
// vma is successfully read-locked
vms_clean_up_area(mas_detach)
vms_clear_ptes
// steps on a cleared PTESo here we have the added complexity that the vma is not unhooked at all. Is there anything that would prevent a concurrent gup_fast() from doing the same -- touch a cleared PTE?
AFAICT two threads, one doing overlapping mmap() and the other doing gup_fast() can result in exactly this scenario.
If we don't care about the GUP case, when I'm thinking we should not care about the lockless RCU case either.
Quoted 5 lines
__mmap_new_vma
vma_set_range// installs new vma in the range
__mmap_complete
vms_complete_munmap_vmas// vmas are write-locked and detached
but it's too late
But at this point that old vma really is unhooked, and the vma_write_start() here will ensure readers are gone and it will clear PTEs *again*.
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 37 lines · Peter Zijlstra
Quoted 5 lines · Suren Baghdasaryan
Ah, ok I see now. I completely misunderstood what
for_each_vma_range()was doing.Then I think
vma_start_write()should remain insidevms_gather_munmap_vmas()and all vmas inmas_detachshould beNo, it must not. You really are not modifying anything yet (except the split, which we've already noted mark write themselves).
Quoted 12 lines
write-locked, even the ones we are not modifying. Otherwise what would prevent the race I mentioned before?
__mmap_region
__mmap_prepare
vms_gather_munmap_vmas// adds vmas to be unmapped intomas_detach,
// some locked
by__split_vma(), some not locked
lock_vma_under_rcu()
vma =mas_walk// finds
unlocked vma also inmas_detachQuoted 9 lines
vma_start_read(vma) // succeeds since vma is not locked // vma->detached, vm_start, vm_end checks pass // vma is successfully read-locked vms_clean_up_area(mas_detach) vms_clear_ptes // steps on a cleared PTESo here we have the added complexity that the vma is not unhooked at all. Is there anything that would prevent a concurrent
gup_fast()from doing the same -- touch a cleared PTE?
Quoted 5 lines
AFAICT two threads, one doing overlapping
mmap()and the other doinggup_fast()can result in exactly this scenario.If we don't care about the GUP case, when I'm thinking we should not care about the lockless RCU case either.
Also, at this point we'll just fail to find a page, and that is nothing special. The problem with accessing an unmapped VMA is that the page-table walk will instantiate page-tables.
Given this is an overlapping mmap -- we're going to need to those page-tables anyway, so no harm done.
Only after the VMA is unlinked must we ensure we don't accidentally re-instantiate page-tables.
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 49 lines · Peter Zijlstra
Quoted 37 lines · Peter Zijlstra
Quoted 5 lines · Suren Baghdasaryan
Ah, ok I see now. I completely misunderstood what
for_each_vma_range()was doing.Then I think
vma_start_write()should remain insidevms_gather_munmap_vmas()and all vmas inmas_detachshould beNo, it must not. You really are not modifying anything yet (except the split, which we've already noted mark write themselves).
Quoted 12 lines
write-locked, even the ones we are not modifying. Otherwise what would prevent the race I mentioned before?
__mmap_region
__mmap_prepare
vms_gather_munmap_vmas// adds vmas to be unmapped intomas_detach,
// some locked
by__split_vma(), some not locked
lock_vma_under_rcu()
vma =mas_walk// finds
unlocked vma also inmas_detachQuoted 9 lines
vma_start_read(vma) // succeeds since vma is not locked // vma->detached, vm_start, vm_end checks pass // vma is successfully read-locked vms_clean_up_area(mas_detach) vms_clear_ptes // steps on a cleared PTESo here we have the added complexity that the vma is not unhooked at all. Is there anything that would prevent a concurrent
gup_fast()from doing the same -- touch a cleared PTE?Quoted 5 lines
AFAICT two threads, one doing overlapping
mmap()and the other doinggup_fast()can result in exactly this scenario.If we don't care about the GUP case, when I'm thinking we should not care about the lockless RCU case either.
Also, at this point we'll just fail to find a page, and that is nothing special. The problem with accessing an unmapped VMA is that the page-table walk will instantiate page-tables.
Quoted 5 lines
Given this is an overlapping mmap -- we're going to need to those page-tables anyway, so no harm done.
Only after the VMA is unlinked must we ensure we don't accidentally re-instantiate page-tables.
Got it. I'll need some time to digest all the input but I think I understand more or less the overall direction. Thanks, Peter!
Re: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Peter ZijlstraQuoted 30 lines · Peter Zijlstra
Quoted 29 lines · Peter Zijlstra
Quoted 5 lines · Suren Baghdasaryan
Ah, ok I see now. I completely misunderstood what
for_each_vma_range()was doing.Then I think
vma_start_write()should remain insidevms_gather_munmap_vmas()and all vmas inmas_detachshould beNo, it must not. You really are not modifying anything yet (except the split, which we've already noted mark write themselves).
Quoted 18 lines
write-locked, even the ones we are not modifying. Otherwise what would prevent the race I mentioned before?
__mmap_region
__mmap_prepare
vms_gather_munmap_vmas// adds vmas to be unmapped intomas_detach,
// some locked
by__split_vma(), some not locked
lock_vma_under_rcu()
vma =mas_walk// finds
unlocked vma also inmas_detachvma_start_read(vma) // succeeds since vma is not locked // vma->detached, vm_start, vm_end checks pass // vma is successfully read-locked
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
vms_clean_up_area(mas_detach) vms_clear_ptes // steps on a cleared PTESo here we have the added complexity that the vma is not unhooked at all.
Well, hold on - it is taken out of the rmap/anon vma chain here. It is completely unhooked except the vma tree at this point. We're not adding complexity, we're dealing with it.
Quoted 2 lines
Is there anything that would prevent a concurrent
gup_fast()fromQuoted 1 line
doing the same -- touch a cleared PTE?
Where does gup_fast() install PTEs? Doesn't it bail once a READ_ONCE() on any level returns no PTE?
Quoted 3 lines
Quoted 3 lines
AFAICT two threads, one doing overlapping
mmap()and the other doinggup_fast()can result in exactly this scenario.
The mmap() call will race with the gup_fast(), but either the nr_pinned will be returned from gup_fast() before vms_clean_up_area() removes the page table (or any higher level), or gup_fast() will find nothing.
Quoted 7 lines
Quoted 3 lines
If we don't care about the GUP case, when I'm thinking we should not care about the lockless RCU case either.
Also, at this point we'll just fail to find a page, and that is nothing special. The problem with accessing an unmapped VMA is that the page-table walk will instantiate page-tables.
I think there is a problem if we are reinstalling page tables on a vma that's about to be removed. I think we are avoiding this with our locking though?
Quoted 3 lines
Given this is an overlapping mmap -- we're going to need to those page-tables anyway, so no harm done.
Well, maybe? The mapping may now be an anon vma vs a file backed, or maybe it's PROT_NONE?
Quoted 2 lines
Only after the VMA is unlinked must we ensure we don't accidentally re-instantiate page-tables.
It's not as simple as that, unfortunately. There are vma callbacks for drivers (or hugetlbfs, or whatever) that do other things. So we need to clean up the area before we are able to replace the vma and part of that clean up is the page tables, or anon vma chain, and/or closing a file.
There are other ways of finding the vma as well, besides the vma tree. We are following the locking so that we are safe from those perspectives as well, and so the vma has to be unlinked in a few places in a certain order.
Thanks,
Liam
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 3 lines · Liam R. Howlett
Well, hold on - it is taken out of the rmap/anon vma chain here. It is completely unhooked except the vma tree at this point. We're not adding complexity, we're dealing with it.
So I'm not entirely sure I understand the details here -- this is again about being able to do rollback when things fail?
There is comment above the vms_clean_up_area() call in __mmap_prepare(), but its not making sense atm.
Quoted 5 lines
Quoted 2 lines
Is there anything that would prevent a concurrent
gup_fast()fromQuoted 1 line
doing the same -- touch a cleared PTE?
Where does
gup_fast()install PTEs? Doesn't it bail once aREAD_ONCE()on any level returns no PTE?
I think you're right, GUP doesn't, but any 'normal' page-table walker will.
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
AFAICT two threads, one doing overlapping
mmap()and the other doinggup_fast()can result in exactly this scenario.The
mmap()call will race with thegup_fast(), but either thenr_pinnedwill be returned fromgup_fast()beforevms_clean_up_area()removes the page table (or any higher level), orgup_fast()will find nothing.
Agreed.
Quoted 10 lines
Quoted 6 lines
Quoted 2 lines
If we don't care about the GUP case, when I'm thinking we should not care about the lockless RCU case either.
Also, at this point we'll just fail to find a page, and that is nothing special. The problem with accessing an unmapped VMA is that the page-table walk will instantiate page-tables.
I think there is a problem if we are reinstalling page tables on a vma that's about to be removed. I think we are avoiding this with our locking though?
So this is purely about the overlapping part, right? We need to remove the old pages, install the new mapping and have new pages populate the thing.
But either way around, the range stays valid and page-tables stay needed.
Quoted 5 lines
Quoted 2 lines
Given this is an overlapping mmap -- we're going to need to those page-tables anyway, so no harm done.
Well, maybe? The mapping may now be an anon vma vs a file backed, or maybe it's
PROT_NONE?
The page-tables don't care about all that no? The only thing where it matters is for things like THP, because that affects the level of page-tables, but otherwise it's all page-table content (ptes).
Quoted 8 lines
Quoted 2 lines
Only after the VMA is unlinked must we ensure we don't accidentally re-instantiate page-tables.
It's not as simple as that, unfortunately. There are vma callbacks for drivers (or hugetlbfs, or whatever) that do other things. So we need to clean up the area before we are able to replace the vma and part of that clean up is the page tables, or anon vma chain, and/or closing a file.
Quoted 4 lines
There are other ways of finding the vma as well, besides the vma tree. We are following the locking so that we are safe from those perspectives as well, and so the vma has to be unlinked in a few places in a certain order.
For RCU lookups only the mas tree matters -- and its left present there.
If you really want to block RCU readers, I would suggest punching a hole in the mm_mt. All the traditional code won't notice anyway, this is all with mmap_lock held for writing.
Re: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 8 lines · Peter Zijlstra
Quoted 3 lines · Liam R. Howlett
Well, hold on - it is taken out of the rmap/anon vma chain here. It is completely unhooked except the vma tree at this point. We're not adding complexity, we're dealing with it.
So I'm not entirely sure I understand the details here -- this is again about being able to do rollback when things fail?
no, it's so that things can be ready for the replacement that won't cause issues during insertion. We can't have two VMAs in the rmap for the same area, for instance.
I would like to be able to undo things, but if there's a file that's closed then we can't undo that.
Quoted 4 lines
There is comment above the
vms_clean_up_area()call in__mmap_prepare(), but its not making sense atm.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Is there anything that would prevent a concurrent
gup_fast()fromQuoted 1 line
doing the same -- touch a cleared PTE?
Where does
gup_fast()install PTEs? Doesn't it bail once aREAD_ONCE()on any level returns no PTE?I think you're right, GUP doesn't, but any 'normal' page-table walker will.
Normal page-table walkers will be locked out by the page table lock while the area is cleared, it cannot be re-walked after that because of the vma lock (except rcu walkers, which is why these new locks are needed). Any direct page table walking won't be a problem because we've removed any way to get to it - I think?
Quoted 17 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
AFAICT two threads, one doing overlapping
mmap()and the other doinggup_fast()can result in exactly this scenario.The
mmap()call will race with thegup_fast(), but either thenr_pinnedwill be returned fromgup_fast()beforevms_clean_up_area()removes the page table (or any higher level), orgup_fast()will find nothing.Agreed.
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
If we don't care about the GUP case, when I'm thinking we should not care about the lockless RCU case either.
Also, at this point we'll just fail to find a page, and that is nothing special. The problem with accessing an unmapped VMA is that the page-table walk will instantiate page-tables.
Quoted 7 lines
Quoted 3 lines
I think there is a problem if we are reinstalling page tables on a vma that's about to be removed. I think we are avoiding this with our locking though?
So this is purely about the overlapping part, right? We need to remove the old pages, install the new mapping and have new pages populate the thing.
No. If we allow rcu readers to re-fault, we may hit a race where the page fault has found the vma, but doesn't fault things in before the ptes are removed or the vma is freed/reused. Today that won't happen because we mark the vma as going to be removed before the tree is changed, so anyone reaching the vma will see it's not safe.
If we want to use different locking strategies for munmap() vs MAP_FIXED, then we'd need to be sure the vma that is being freed is isolated for removal and all readers are done before freeing. I think this is what you are trying to convey to Suren?
I don't like the idea of another locking strategy in munmap() vs MAP_FIXED, especially if you think about who would be doing this.. basically no one. There isn't a sane (legitimate) application that's going to try and page fault in something that's being removed.
Quoted 7 lines
But either way around, the range stays valid and page-tables stay needed.
Quoted 3 lines
Quoted 2 lines
Given this is an overlapping mmap -- we're going to need to those page-tables anyway, so no harm done.
Quoted 6 lines
Quoted 2 lines
Well, maybe? The mapping may now be an anon vma vs a file backed, or maybe it's
PROT_NONE?The page-tables don't care about all that no? The only thing where it matters is for things like THP, because that affects the level of page-tables, but otherwise it's all page-table content (ptes).
It sounds racy, couldn't we read the old page table entry attributes and act on it after the new attributes are set?
During the switch, after we drop the page table lock but haven't yet inserted the new vma, then we'd run into a situation where the new mapping may already be occupied if we don't have some sort of locking here. Wouldn't that be an issue as well? It seems like there are a number of things that could go bad?
Quoted 14 lines
Quoted 9 lines
Quoted 3 lines
Only after the VMA is unlinked must we ensure we don't accidentally re-instantiate page-tables.
It's not as simple as that, unfortunately. There are vma callbacks for drivers (or hugetlbfs, or whatever) that do other things. So we need to clean up the area before we are able to replace the vma and part of that clean up is the page tables, or anon vma chain, and/or closing a file.
Quoted 4 lines
There are other ways of finding the vma as well, besides the vma tree. We are following the locking so that we are safe from those perspectives as well, and so the vma has to be unlinked in a few places in a certain order.
Quoted 5 lines
For RCU lookups only the mas tree matters -- and its left present there.
If you really want to block RCU readers, I would suggest punching a hole in the
mm_mt. All the traditional code won't notice anyway, this is all withmmap_lockheld for writing.
We don't want to block all rcu readers, we want to block the rcu readers that would see the problem - that is, anyone trying to read a particular area.
Right now we can page fault in unpopulated vmas while writing other vmas to the tree. We are also moving more users to rcu reading to use the vmas they need without waiting on writes to finish.
Maybe I don't understand your suggestion, but I would think punching a hole would lose this advantage?
Thanks,
Liam
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 10 lines · Liam R. Howlett
Quoted 5 lines
For RCU lookups only the mas tree matters -- and its left present there.
If you really want to block RCU readers, I would suggest punching a hole in the
mm_mt. All the traditional code won't notice anyway, this is all withmmap_lockheld for writing.We don't want to block all rcu readers, we want to block the rcu readers that would see the problem - that is, anyone trying to read a particular area.
Quoted 6 lines
Right now we can page fault in unpopulated vmas while writing other vmas to the tree. We are also moving more users to rcu reading to use the vmas they need without waiting on writes to finish.
Maybe I don't understand your suggestion, but I would think punching a hole would lose this advantage?
My suggestion was to remove the range stuck in mas_detach from mm_mt. That is exactly the affected range, no?
Re: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 16 lines · Peter Zijlstra
Quoted 14 lines · Liam R. Howlett
Quoted 5 lines
For RCU lookups only the mas tree matters -- and its left present there.
If you really want to block RCU readers, I would suggest punching a hole in the
mm_mt. All the traditional code won't notice anyway, this is all withmmap_lockheld for writing.We don't want to block all rcu readers, we want to block the rcu readers that would see the problem - that is, anyone trying to read a particular area.
Right now we can page fault in unpopulated vmas while writing other vmas to the tree. We are also moving more users to rcu reading to use the vmas they need without waiting on writes to finish.
Quoted 5 lines
Quoted 2 lines
Maybe I don't understand your suggestion, but I would think punching a hole would lose this advantage?
My suggestion was to remove the range stuck in
mas_detachfrommm_mt. That is exactly the affected range, no?
Yes.
But then looping over the vmas will show a gap where there should not be a gap.
If we stop rcu readers entirely we lose the advantage.
This is exactly the issue that the locking dance was working around :)
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 24 lines · Liam R. Howlett
Quoted 16 lines · Peter Zijlstra
Quoted 14 lines · Liam R. Howlett
Quoted 5 lines
For RCU lookups only the mas tree matters -- and its left present there.
If you really want to block RCU readers, I would suggest punching a hole in the
mm_mt. All the traditional code won't notice anyway, this is all withmmap_lockheld for writing.We don't want to block all rcu readers, we want to block the rcu readers that would see the problem - that is, anyone trying to read a particular area.
Right now we can page fault in unpopulated vmas while writing other vmas to the tree. We are also moving more users to rcu reading to use the vmas they need without waiting on writes to finish.
Quoted 5 lines
Quoted 2 lines
Maybe I don't understand your suggestion, but I would think punching a hole would lose this advantage?
My suggestion was to remove the range stuck in
mas_detachfrommm_mt. That is exactly the affected range, no?
Quoted 8 lines
Yes.
But then looping over the vmas will show a gap where there should not be a gap.
If we stop rcu readers entirely we lose the advantage.
This is exactly the issue that the locking dance was working around :)
Sorry for not participating in the discussion, folks. I'm down with a terrible flu and my brain is not working well. I'll catch up once I get better.
Quoted 1 line
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Liam R. HowlettQuoted 24 lines · Liam R. Howlett
Quoted 16 lines · Peter Zijlstra
Quoted 14 lines · Liam R. Howlett
Quoted 5 lines
For RCU lookups only the mas tree matters -- and its left present there.
If you really want to block RCU readers, I would suggest punching a hole in the
mm_mt. All the traditional code won't notice anyway, this is all withmmap_lockheld for writing.We don't want to block all rcu readers, we want to block the rcu readers that would see the problem - that is, anyone trying to read a particular area.
Right now we can page fault in unpopulated vmas while writing other vmas to the tree. We are also moving more users to rcu reading to use the vmas they need without waiting on writes to finish.
Quoted 5 lines
Quoted 2 lines
Maybe I don't understand your suggestion, but I would think punching a hole would lose this advantage?
My suggestion was to remove the range stuck in
mas_detachfrommm_mt. That is exactly the affected range, no?
Quoted 8 lines
Yes.
But then looping over the vmas will show a gap where there should not be a gap.
If we stop rcu readers entirely we lose the advantage.
This is exactly the issue that the locking dance was working around :)
IOW we write-lock the entire range before removing any part of it for the whole transaction to be atomic, correct?
Peter, you suggested the following pattern for ensuring vma is detached with no possible readers:
vma_iter_store()vma_start_write()vma_mark_detached()
What do you think about this alternative?
vma_start_write()
...vma_iter_store()vma_mark_detached()
vma_assert_write_locked(vma)
if (unlikely(!refcount_dec_and_test(&vma->vm_refcnt)))
vma_start_write()The second vma_start_write() is unlikely to be executed because the vma is locked, vm_refcnt might be increased only temporarily by readers before they realize the vma is locked and that's a very narrow window. I think performance should not visibly suffer?
OTOH this would let us keep current locking patterns and would guarantee that vma_mark_detached() always exits with a detached and unused vma (less possibilities for someone not following an exact pattern and ending up with a detached but still used vma).
Quoted 1 line
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 57 lines · Suren Baghdasaryan
Quoted 24 lines · Liam R. Howlett
Quoted 16 lines · Peter Zijlstra
Quoted 14 lines · Liam R. Howlett
Quoted 5 lines
For RCU lookups only the mas tree matters -- and its left present there.
If you really want to block RCU readers, I would suggest punching a hole in the
mm_mt. All the traditional code won't notice anyway, this is all withmmap_lockheld for writing.We don't want to block all rcu readers, we want to block the rcu readers that would see the problem - that is, anyone trying to read a particular area.
Right now we can page fault in unpopulated vmas while writing other vmas to the tree. We are also moving more users to rcu reading to use the vmas they need without waiting on writes to finish.
Quoted 5 lines
Quoted 2 lines
Maybe I don't understand your suggestion, but I would think punching a hole would lose this advantage?
My suggestion was to remove the range stuck in
mas_detachfrommm_mt. That is exactly the affected range, no?Quoted 8 lines
Yes.
But then looping over the vmas will show a gap where there should not be a gap.
If we stop rcu readers entirely we lose the advantage.
This is exactly the issue that the locking dance was working around :)
IOW we write-lock the entire range before removing any part of it for the whole transaction to be atomic, correct?
Peter, you suggested the following pattern for ensuring vma is detached with no possible readers:
vma_iter_store()vma_start_write()vma_mark_detached()What do you think about this alternative?
vma_start_write()
...vma_iter_store()vma_mark_detached()vma_assert_write_locked(vma) if (unlikely(!refcount_dec_and_test(&vma->vm_refcnt))) vma_start_write()
Quoted 8 lines
The second
vma_start_write()is unlikely to be executed because the vma is locked,vm_refcntmight be increased only temporarily by readers before they realize the vma is locked and that's a very narrow window. I think performance should not visibly suffer?
OTOH this would let us keep current locking patterns and would guarantee thatvma_mark_detached()always exits with a detached and unused vma (less possibilities for someone not following an exact pattern and ending up with a detached but still used vma).
I posted v7 of this patchset at
https://lore.kernel.org/all/[email protected]/
From the things we discussed, I didn't include the following:
- Changing vma locking patterns
- Changing do_vmi_align_munmap() to avoid reattach_vmas() It seems we need more discussion for the first one and the second one can be done completely independent from this patchset. I feel this patchset is already quite large, so trying to keep its size manageable.
Thanks,
Suren.
Quoted 2 lines
Quoted 2 lines
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Peter ZijlstraQuoted 50 lines · Peter Zijlstra
Quoted 5 lines · Suren Baghdasaryan
Ah, ok I see now. I completely misunderstood what
for_each_vma_range()was doing.Then I think
vma_start_write()should remain insidevms_gather_munmap_vmas()and all vmas inmas_detachshould beNo, it must not. You really are not modifying anything yet (except the split, which we've already noted mark write themselves).
Quoted 12 lines
write-locked, even the ones we are not modifying. Otherwise what would prevent the race I mentioned before?
__mmap_region
__mmap_prepare
vms_gather_munmap_vmas// adds vmas to be unmapped intomas_detach,
// some locked
by__split_vma(), some not locked
lock_vma_under_rcu()
vma =mas_walk// finds
unlocked vma also inmas_detachQuoted 9 lines
vma_start_read(vma) // succeeds since vma is not locked // vma->detached, vm_start, vm_end checks pass // vma is successfully read-locked vms_clean_up_area(mas_detach) vms_clear_ptes // steps on a cleared PTESo here we have the added complexity that the vma is not unhooked at all. Is there anything that would prevent a concurrent
gup_fast()from doing the same -- touch a cleared PTE?AFAICT two threads, one doing overlapping
mmap()and the other doinggup_fast()can result in exactly this scenario.If we don't care about the GUP case, when I'm thinking we should not care about the lockless RCU case either.
Quoted 5 lines
__mmap_new_vma
vma_set_range// installs new vma in the range
__mmap_complete
vms_complete_munmap_vmas// vmas are write-locked and detached
but it's too late
Quoted 3 lines
But at this point that old vma really is unhooked, and the
vma_write_start()here will ensure readers are gone and it will clear PTEs *again*.
So, to summarize, you want vma_start_write() and vma_mark_detached() to be done when we are removing the vma from the tree, right? Something like:
vma_start_write()vma_iter_store()vma_mark_detached()
And the race I described is not a real problem since the vma is still in the tree, so gup_fast() does exactly that and will simply reinstall the ptes.
Quoted 2 lines
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 51 lines · Suren Baghdasaryan
Quoted 50 lines · Peter Zijlstra
Quoted 5 lines · Suren Baghdasaryan
Ah, ok I see now. I completely misunderstood what
for_each_vma_range()was doing.Then I think
vma_start_write()should remain insidevms_gather_munmap_vmas()and all vmas inmas_detachshould beNo, it must not. You really are not modifying anything yet (except the split, which we've already noted mark write themselves).
Quoted 12 lines
write-locked, even the ones we are not modifying. Otherwise what would prevent the race I mentioned before?
__mmap_region
__mmap_prepare
vms_gather_munmap_vmas// adds vmas to be unmapped intomas_detach,
// some locked
by__split_vma(), some not locked
lock_vma_under_rcu()
vma =mas_walk// finds
unlocked vma also inmas_detachQuoted 9 lines
vma_start_read(vma) // succeeds since vma is not locked // vma->detached, vm_start, vm_end checks pass // vma is successfully read-locked vms_clean_up_area(mas_detach) vms_clear_ptes // steps on a cleared PTESo here we have the added complexity that the vma is not unhooked at all. Is there anything that would prevent a concurrent
gup_fast()from doing the same -- touch a cleared PTE?AFAICT two threads, one doing overlapping
mmap()and the other doinggup_fast()can result in exactly this scenario.If we don't care about the GUP case, when I'm thinking we should not care about the lockless RCU case either.
Quoted 5 lines
__mmap_new_vma
vma_set_range// installs new vma in the range
__mmap_complete
vms_complete_munmap_vmas// vmas are write-locked and detached
but it's too late
Quoted 6 lines
Quoted 3 lines
But at this point that old vma really is unhooked, and the
vma_write_start()here will ensure readers are gone and it will clear PTEs *again*.So, to summarize, you want
vma_start_write()andvma_mark_detached()to be done when we are removing the vma from the tree, right?
*after*
Quoted 1 line
Something like:
vma_iter_store()
vma_start_write()
vma_mark_detached()
By having vma_start_write() after being unlinked you get the guarantee of no concurrency. New lookups cannot find you (because of that vma_iter_store()) and existing readers will be waited for.
Quoted 3 lines
And the race I described is not a real problem since the vma is still in the tree, so
gup_fast()does exactly that and will simply reinstall the ptes.
Just so.
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Suren BaghdasaryanQuoted 10 lines · Suren Baghdasaryan
Yeah, I think we can simply do this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_mark_detached
mmap_write_downgradeIf my assumption is incorrect, assertion inside
vma_mark_detached()should trigger. I tried a quick test and so far nothing exploded.
I think that would be unfortunate and could cause regressions. I think we want to keep vms_clear_ptes() under the read-lock.
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 11 lines · Peter Zijlstra
Quoted 8 lines · Suren Baghdasaryan
Yeah, I think we can simply do this:
vms_complete_munmap_vmas
vms_clear_ptes
remove_vma
vma_mark_detached
mmap_write_downgrade
Quoted 5 lines
Quoted 2 lines
If my assumption is incorrect, assertion inside
vma_mark_detached()should trigger. I tried a quick test and so far nothing exploded.I think that would be unfortunate and could cause regressions. I think we want to keep
vms_clear_ptes()under the read-lock.
Ok, I'll stop considering this option.
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Suren BaghdasaryanQuoted 1 line · Suren Baghdasaryan
And
remove_vmawill be just freeing the vmas. Is that correct?
Yep.
Quoted 4 lines
I'm a bit confused because the original thinking was that
vma_mark_detached()would drop the last refcnt and if it's 0 we would free the vma right there. If that's still what we want to do then I think the above sequence should look like this:
Right; sorry about that. So my initial objection to that extra sync was based on the reasons presented -- but having had to look at the unmap path again (my mm-foo is somewhat rusty, I've not done much the past few years) I realized that keeping a VMA alive beyond unmapping PTEs is just plain daft.
So yes, back to your original semantics, but cleaned up to not need that extra sync point -- instead relying on the natural placement of vma_start_write() after unhooking from the mm. And not for reasons of the race, but for reasons of integrity -- VMA without PTEs is asking for more trouble.
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
Quoted 18 lines · Peter Zijlstra
Quoted 1 line · Suren Baghdasaryan
And
remove_vmawill be just freeing the vmas. Is that correct?Yep.
Quoted 4 lines
I'm a bit confused because the original thinking was that
vma_mark_detached()would drop the last refcnt and if it's 0 we would free the vma right there. If that's still what we want to do then I think the above sequence should look like this:Right; sorry about that. So my initial objection to that extra sync was based on the reasons presented -- but having had to look at the unmap path again (my mm-foo is somewhat rusty, I've not done much the past few years) I realized that keeping a VMA alive beyond unmapping PTEs is just plain daft.
Quoted 5 lines
So yes, back to your original semantics, but cleaned up to not need that extra sync point -- instead relying on the natural placement of
vma_start_write()after unhooking from the mm. And not for reasons of the race, but for reasons of integrity -- VMA without PTEs is asking for more trouble.
Ack. Thanks for clarification!
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Liam R. HowlettQuoted 164 lines · Liam R. Howlett
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Peter Zijlstra
Quoted 12 lines · Suren Baghdasaryan
Quoted 2 lines
So I just replied there, and no, I don't think it makes sense. Just put the
kmem_cache_free()invma_refcount_put(), to be done on 0.That's very appealing indeed and makes things much simpler. The problem I see with that is the case when we detach a vma from the tree to isolate it, then do some cleanup and only then free it. That's done in
vms_gather_munmap_vmas()here:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1240 and we even might reattach detached vmas back:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1312. IOW, detached state is not final and we can't destroy the object that reached this state.Urgh, so that's the
munmap()path, but arguably when that fails, the map stays in place.I think this means you're marking detached too soon; you should only mark detached once you reach the point of no return.
That said, once you've reached the point of no return; and are about to go remove the page-tables, you very much want to ensure a lack of concurrency.
Quoted 5 lines
Quoted 5 lines
So perhaps waiting for out-standing readers at this point isn't crazy.
Also, I'm having a very hard time reading this maple tree stuff :/ Afaict
vms_gather_munmap_vmas()only adds the VMAs to be removed to a second tree, it does not in fact unlink them from the mm yet.Yes, that's correct. I tried to make this clear with a gather/complete naming like other areas of the mm. I hope that helped.
Also, the comments for the function state that's what's going on:
*
vms_gather_munmap_vmas()- Put all VMAs within a range into a maple tree
* for removal at a later date. Handles splitting first and last if necessary
* and marking the vmas as isolated.... might be worth updating with new information.
Quoted 4 lines
Quoted 4 lines
AFAICT it's
vma_iter_clear_gfp()that actually wipes the vmas from the mm -- and that being able to fail is mind boggling and I suppose is what gives rise to much of this insanity :/This is also correct. The maple tree is a b-tree variant that has internal nodes. When you write to it, including nulls, they are tracked and may need to allocate. This is a cost for rcu lookups; we will use the same or less memory in the end but must maintain a consistent view of the ranges.
But to put this into perspective, we get 16 nodes per 4k page, most writes will use 1 or 3 of these from a
kmem_cache, so we are talking about a very unlikely possibility. Except when syzbot decides to fail random allocations.We could preallocate for the write, but this section of the code is
GFP_KERNEL, so we don't. Preallocation is an option to simplify the failure path though... which is what you did below.Quoted 8 lines
Quoted 4 lines
Anyway, I would expect
remove_vma()to be the one that marks it detached (it's already unreachable throughvma_lookup()at this point) and there you should wait for concurrent readers to bugger off.Also, I think
vma_start_write()in that gather look is too early, you're not actually going to change the VMA yet -- with obvious exception of the split cases.The split needs to start the write on the vma to avoid anyone reading it while it's being altered.
Quoted 3 lines
That too should probably come after you've passes all the fail/unwind spots.
Do you mean the split? I'd like to move the split later as well.. tracking that is a pain and may need an extra vma for when one vma is split twice before removing the middle part.
Actually, I think we need to allocate two (or at least one) vmas in this case and just pass one through to unmap (written only to the
mas_detachtree?). It would be nice to find a way to NOT need to do that even.. I had tried to use a vma on the stack years ago, which didn't work out.mm/vma.c static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms, · line 1175
Quoted 11 lines
Something like so perhaps? (yeah, I know, I wrecked a bunch)
mm/vma.c +0
-0@@ -1173,6 +1173,11 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms,struct vm_area_struct *vma;struct mm_struct *mm;
mas_set(mas_detach, 0);mm/vma.c static void reattach_vmas(struct ma_state *mas_detach) · line 1227
Quoted 13 lines
mm/vma.c +5
-0+mas_for_each (mas_detach, vma,ULONG_MAX ) { +vma_start_write (next); +vma_mark_detached (next, true); + } + mm = current->mm; mm->map_count -= vms->vma_count; mm->locked_vm -= vms->locked_vm;@@ -1219,9 +1224,6 @@ static void reattach_vmas(struct ma_state *mas_detach)struct vm_area_struct *vma;mas_set (mas_detach,0 );Drop the
mas_sethere.Quoted 21 lines
mm/vma.c +0
-5-mas_for_each (mas_detach, vma,ULONG_MAX ) -vma_mark_detached (vma, false); -__mt_destroy (mas_detach->tree); }@@ -1289,13 +1291,11 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms,if (error) goto end_split_failed; } -vma_start_write (next);mas_set (mas_detach, vms->vma_count++); error =mas_store_gfp (mas_detach, next,GFP_KERNEL );if (error) goto munmap_gather_failed; -vma_mark_detached (next, true); nrpages =vma_pages (next); vms->nr_pages += nrpages;mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1433
Quoted 3 lines
mm/vma.c +0
-0@@ -1431,14 +1431,17 @@ int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma,struct vma_munmap_struct vms;int error;The preallocation needs to know the range being stored to know what's going to happen.
vma_iter_config(vmi, start, end);mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1434
Quoted 1 line
mm/vma.c +1
-0+ error =mas_preallocate (vmi->mas);We haven't had a need to have a vma iterator preallocate for storing a null, but we can add one for this.
mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1442
Quoted 8 lines
mm/vma.c +3
-0+if (error) + goto gather_failed; +init_vma_munmap (&vms, vmi, vma, start, end, uf, unlock); error =vms_gather_munmap_vmas (&vms, &mas_detach);if (error) goto gather_failed;Drop this stuff.
Quoted 8 lines
mm/vma.c int do_vmi_align_munmap(struct vma_iterator *vmi, struct vm_area_struct *vma, · line 1444
Quoted 4 lines
mm/vma.c +1
-2error =vma_iter_clear_gfp (vmi, start, end,GFP_KERNEL ); -if (error) - goto clear_tree_failed; +VM_WARN_ON (error);Do this instead
vma_iter_config(vmi, start, end);vma_iter_clear(vmi);
Thanks for the input, Liam. Let me try to make a patch from these suggestions and see where we end up and what might blow up.
Quoted 4 lines
Quoted 4 lines
mm/vma.c +0
-0/* Point of no return */ vms_complete_munmap_vmas (&vms, &mas_detach);
Peter Zijlstra
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Liam R. HowlettQuoted 2 lines · Liam R. Howlett
This is also correct. The maple tree is a b-tree variant that has internal nodes.
Right, I remembered that much :-)
Quoted 6 lines
Quoted 3 lines
Also, I think
vma_start_write()in that gather look is too early, you're not actually going to change the VMA yet -- with obvious exception of the split cases.The split needs to start the write on the vma to avoid anyone reading it while it's being altered.
__split_vma() does vma_start_write() itself, so that should be good already.
Quoted 4 lines
Quoted 2 lines
That too should probably come after you've passes all the fail/unwind spots.
Do you mean the split?
No, I means the detach muck :-)
Quoted 8 lines
I'd like to move the split later as well..
tracking that is a pain and may need an extra vma for when one vma is split twice before removing the middle part.Actually, I think we need to allocate two (or at least one) vmas in this case and just pass one through to unmap (written only to the
mas_detachtree?). It would be nice to find a way to NOT need to do that even.. I had tried to use a vma on the stack years ago, which didn't work out.
Urgh yeah, vma on stack sounds like utter pain :-)
Suren Baghdasaryan
maintainerRe: [PATCH v6 10/16] mm: replace vm_lock and detached flag with a reference count
In reply to Peter ZijlstraQuoted 26 lines · Peter Zijlstra
Quoted 12 lines · Suren Baghdasaryan
Quoted 2 lines
So I just replied there, and no, I don't think it makes sense. Just put the
kmem_cache_free()invma_refcount_put(), to be done on 0.That's very appealing indeed and makes things much simpler. The problem I see with that is the case when we detach a vma from the tree to isolate it, then do some cleanup and only then free it. That's done in
vms_gather_munmap_vmas()here:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1240 and we even might reattach detached vmas back:
https://elixir.bootlin.com/linux/v6.12.5/source/mm/vma.c#L1312. IOW, detached state is not final and we can't destroy the object that reached this state.Urgh, so that's the
munmap()path, but arguably when that fails, the map stays in place.I think this means you're marking detached too soon; you should only mark detached once you reach the point of no return.
That said, once you've reached the point of no return; and are about to go remove the page-tables, you very much want to ensure a lack of concurrency.
Quoted 5 lines
So perhaps waiting for out-standing readers at this point isn't crazy.
Also, I'm having a very hard time reading this maple tree stuff :/ Afaict
vms_gather_munmap_vmas()only adds the VMAs to be removed to a second tree, it does not in fact unlink them from the mm yet.
Yes, I think you are correct.
Quoted 8 lines
AFAICT it's
vma_iter_clear_gfp()that actually wipes the vmas from the mm -- and that being able to fail is mind boggling and I suppose is what gives rise to much of this insanity :/Anyway, I would expect
remove_vma()to be the one that marks it detached (it's already unreachable throughvma_lookup()at this point) and there you should wait for concurrent readers to bugger off.
There is an issue with that. Note that vms_complete_munmap_vmas() that's calling remove_vma() might drop the mmap write lock, so detaching without a write lock would break current rules.
Quoted 10 lines
Quoted 5 lines
We could change states to: 0=unused (we can free
the object), 1=detached, 2=attached, etc. but thenvma_start_read()should do something likerefcount_inc_more_than_one()instead ofrefcount_inc_not_zero(). Would you be ok with such an approach?Urgh, I would strongly suggest ditching
refcount_tif we go this route. The thing is;refcount_tshould remain a 'simple' straight forward interface and not allow people to do the wrong thing. Its not meant to be the kitchen sink -- we haveatomic_tfor that.
Ack. If we go this route I'll use atomics directly.
Quoted 4 lines
Anyway, the more common scheme at that point is using -1 for 'free', I think
folio->_mapcountuses that even. For that see:
atomic_add_negative*().
Thanks for the reference.
Quoted 37 lines
Quoted 12 lines
Quoted 6 lines
Additionally, having
vma_end_write()would allow you to put a lockdep annotation in vma_{start,end}_write() -- which was I think the original reason I proposed it a while back, that and having improved clarity when reading the code, since explicitly marking the end of a section is helpful.The
vma->vmlock_dep_mapis trackingvma->vm_refcnt, not thevma->vm_lock_seq(similar to how todayvma->vm_lockhas its lockdep tracking thatrw_semaphore). If I implementvma_end_write()then it will simply be something like:Quoted 7 lines
void
vma_end_write(vma){ vma_assert_write_locked(vma); vma->vm_lock_seq = UINT_MAX; }so,
vmlock_dep_mapwould not be involved.That's just weird; why would you not track vma_{start,end}_write() with the exclusive side of the 'rwsem'
dep_map?Quoted 5 lines
If you want to track
vma->vm_lock_seqwith a separate lockdep, that would be more complicated. Specifically forvma_end_write_all()that would require us to callrwsem_release()on all locked vmas, however we currently do not track individual locked vmas.vma_end_write_all()allows us not to worry about tracking them, knowing that once we doQuoted 8 lines
mmap_write_unlock()they all will get unlocked with one increment ofmm->mm_lock_seq. If your suggestion is to replacevma_end_write_all()withvma_end_write()and unlock vmas individually across the mm code, that would be a sizable effort. If that is indeed your ultimate goal, I can do that as a separate project: introducevma_end_write(), gradually add them in required places (not yet sure how complex that would be), then retirevma_end_write_all()and add a lockdep forvma->vm_lock_seq.
Quoted 3 lines
Yeah, so ultimately I think it would be clearer if you explicitly mark the point where the vma modification is 'done'. But I don't suppose we have to do that here.
Ack.
Suren Baghdasaryan
maintainerPATCH v6 11/16 mm: enforce vma to be in detached state before freeing
In reply to Suren Baghdasaryanexit_mmap() frees vmas without detaching them. This will become a problem when we introduce vma reuse. Ensure that vmas are always detached before being freed.
--- kernel/fork.c | 4 ++++ mm/vma.c | 10 ++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-)
kernel/fork.c +4 -0
@@ -473,6 +473,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)void __vm_area_free (struct vm_area_struct *vma) { +#ifdef CONFIG_PER_VMA_LOCK +/* The vma should be detached while being destroyed. */ +VM_BUG_ON_VMA (!is_vma_detached (vma), vma); +#endif vma_numab_state_free (vma);free_anon_vma_name (vma);kmem_cache_free (vm_area_cachep, vma);
mm/vma.c +8 -2
@@ -413,9 +413,15 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable)if (vma->vm_file)fput (vma->vm_file);mpol_put (vma_policy (vma)); -if (unreachable) +if (unreachable) { +#ifdef CONFIG_PER_VMA_LOCK +if (!is_vma_detached (vma)) { +vma_start_write (vma); +vma_mark_detached (vma); + } +#endif __vm_area_free (vma); -else + }else vm_area_free (vma); }
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Peter Zijlstra
maintainerRe: [PATCH v6 11/16] mm: enforce vma to be in detached state before freeing
Quoted 32 lines · Suren Baghdasaryan
exit_mmap()frees vmas without detaching them. This will become a problem when we introduce vma reuse. Ensure that vmas are always detached before being freed.Signed-off-by Suren Baghdasaryan <[email protected]>--- kernel/fork.c | 4 ++++ mm/vma.c | 10 ++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-)kernel/fork.c +4
-0@@ -473,6 +473,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)void __vm_area_free (struct vm_area_struct *vma) { +#ifdef CONFIG_PER_VMA_LOCK +/* The vma should be detached while being destroyed. */ +VM_BUG_ON_VMA (!is_vma_detached (vma), vma); +#endif vma_numab_state_free (vma);free_anon_vma_name (vma);kmem_cache_free (vm_area_cachep, vma);mm/vma.c +0
-0@@ -413,9 +413,15 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable)if (vma->vm_file)fput (vma->vm_file);
mm/vma.c void remove_vma(struct vm_area_struct *vma, bool unreachable) · line 423
Quoted 10 lines
mm/vma.c +7
-1mpol_put (vma_policy (vma)); -if (unreachable) +if (unreachable) { +#ifdef CONFIG_PER_VMA_LOCK +if (!is_vma_detached (vma)) { +vma_start_write (vma); +vma_mark_detached (vma); + } +#endif __vm_area_free (vma);
Again, can't you race with lockess RCU lookups?
Peter Zijlstra
maintainerRe: [PATCH v6 11/16] mm: enforce vma to be in detached state before freeing
Quoted 37 lines · Peter Zijlstra
Quoted 36 lines · Suren Baghdasaryan
exit_mmap()frees vmas without detaching them. This will become a problem when we introduce vma reuse. Ensure that vmas are always detached before being freed.Signed-off-by Suren Baghdasaryan <[email protected]>--- kernel/fork.c | 4 ++++ mm/vma.c | 10 ++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-)kernel/fork.c +4
-0@@ -473,6 +473,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)void __vm_area_free (struct vm_area_struct *vma) { +#ifdef CONFIG_PER_VMA_LOCK +/* The vma should be detached while being destroyed. */ +VM_BUG_ON_VMA (!is_vma_detached (vma), vma); +#endif vma_numab_state_free (vma);free_anon_vma_name (vma);kmem_cache_free (vm_area_cachep, vma);mm/vma.c +2
-1@@ -413,9 +413,15 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable)if (vma->vm_file)fput (vma->vm_file);mpol_put (vma_policy (vma)); -if (unreachable) +if (unreachable) { +#ifdef CONFIG_PER_VMA_LOCK
Quoted 8 lines
mm/vma.c void remove_vma(struct vm_area_struct *vma, bool unreachable) · line 423
Quoted 6 lines
mm/vma.c +5
-0+if (!is_vma_detached (vma)) { +vma_start_write (vma); +vma_mark_detached (vma); + } +#endif __vm_area_free (vma);Again, can't you race with lockess RCU lookups?
Ah, no, removing vma requires holding mmap_lock for writing and having the vma locked, which would ensure preceding RCU readers are complete (per the LOCK_OFFSET waiter thing) and new RCU readers are rejected for the vma sequence thing.
Suren Baghdasaryan
maintainerRe: [PATCH v6 11/16] mm: enforce vma to be in detached state before freeing
Quoted 48 lines · Peter Zijlstra
Quoted 37 lines · Peter Zijlstra
Quoted 36 lines · Suren Baghdasaryan
exit_mmap()frees vmas without detaching them. This will become a problem when we introduce vma reuse. Ensure that vmas are always detached before being freed.Signed-off-by Suren Baghdasaryan <[email protected]>--- kernel/fork.c | 4 ++++ mm/vma.c | 10 ++++++++-- 2 files changed, 12 insertions(+), 2 deletions(-)kernel/fork.c +4
-0@@ -473,6 +473,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)void __vm_area_free (struct vm_area_struct *vma) { +#ifdef CONFIG_PER_VMA_LOCK +/* The vma should be detached while being destroyed. */ +VM_BUG_ON_VMA (!is_vma_detached (vma), vma); +#endif vma_numab_state_free (vma);free_anon_vma_name (vma);kmem_cache_free (vm_area_cachep, vma);mm/vma.c +2
-1@@ -413,9 +413,15 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable)if (vma->vm_file)fput (vma->vm_file);mpol_put (vma_policy (vma)); -if (unreachable) +if (unreachable) { +#ifdef CONFIG_PER_VMA_LOCK Quoted 8 lines
mm/vma.c void remove_vma(struct vm_area_struct *vma, bool unreachable) · line 423
Quoted 6 lines
mm/vma.c +5
-0+if (!is_vma_detached (vma)) { +vma_start_write (vma); +vma_mark_detached (vma); + } +#endif __vm_area_free (vma);Again, can't you race with lockess RCU lookups?
Quoted 4 lines
Ah, no, removing vma requires holding
mmap_lockfor writing and having the vma locked, which would ensure preceding RCU readers are complete (per theLOCK_OFFSETwaiter thing) and new RCU readers are rejected for the vma sequence thing.
Correct. Once vma is detached it can't be found by new readers. Possible existing readers are purged later in this patchset by calling vma_ensure_detached() from vm_area_free(). I don't do that in this patch because those existing temporary readers do not pose issues up until we start reusing the vmas.
Suren Baghdasaryan
maintainerPATCH v6 12/16 mm: remove extra vma_numab_state_init() call
In reply to Suren Baghdasaryanvma_init() already memset's the whole vm_area_struct to 0, so there is no need to an additional vma_numab_state_init().
--- include/linux/mm.h | 1 - 1 file changed, 1 deletion(-)
include/linux/mm.h +0 -1
@@ -949,7 +949,6 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &vma_dummy_vm_ops;INIT_LIST_HEAD (&vma->anon_vma_chain); -vma_numab_state_init (vma);vma_lock_init (vma); }
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 13/16 mm: introduce vma_ensure_detached()
In reply to Suren Baghdasaryanvma_start_read() can temporarily raise vm_refcnt of a write-locked and detached vma:
// vm_refcnt==1 (attached)vma_start_write()
vma->vm_lock_seq = mm->mm_lock_seq
vma_start_read()
vm_refcnt++; // vm_refcnt==2vma_mark_detached()
vm_refcnt--; // vm_refcnt==1
// vma is detached but vm_refcnt!=0 temporarily
if (vma->vm_lock_seq == mm->mm_lock_seq)
vma_refcount_put()
vm_refcnt--; // vm_refcnt==0This is currently not a problem when freeing the vma because RCU grace period should pass before kmem_cache_free(vma) gets called and by that time vma_start_read() should be done and vm_refcnt is 0. However once we introduce possibility of vma reuse before RCU grace period is over, this will become a problem (reused vma might be in non-detached state). Introduce vma_ensure_detached() for the writer to wait for readers until they exit vma_start_read().
--- include/linux/mm.h | 9 ++++++ mm/memory.c | 55 +++++++++++++++++++++++--------- tools/testing/vma/vma_internal.h | 8 +++++ 3 files changed, 57 insertions(+), 15 deletions(-)
include/linux/mm.h +9 -0
@@ -863,6 +863,15 @@ static inline bool is_vma_detached(struct vm_area_struct *vma)return refcount_read (&vma->vm_refcnt) ==VMA_STATE_DETACHED ; } +/* +* WARNING: to avoid racing with vma_mark_attached(), should be called either +* under mmap_write_lock or when the object has been isolated under +* mmap_write_lock, ensuring no competing writers. +* Should be called after marking vma as detached to wait for possible +* readers which temporarily raised vm_refcnt to drop it back and exit. +*/ +void vma_ensure_detached (struct vm_area_struct *vma); +static inline void vma_mark_attached (struct vm_area_struct *vma) {vma_assert_write_locked (vma);
mm/memory.c +40 -15
77 lines not on this page · raw message
tools/testing/vma/vma_internal.h +8 -0
@@ -465,6 +465,14 @@ static inline bool is_vma_detached(struct vm_area_struct *vma)return refcount_read (&vma->vm_refcnt) ==VMA_STATE_DETACHED ; } +static inline void vma_ensure_detached (struct vm_area_struct *vma) +{ +if (is_vma_detached (vma)) +return ; + +refcount_set (&vma->vm_refcnt,VMA_STATE_DETACHED ); +} +static inline void vma_assert_write_locked (struct vm_area_struct *);static inline void vma_mark_attached (struct vm_area_struct *vma) {
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Peter Zijlstra
maintainerRe: [PATCH v6 13/16] mm: introduce vma_ensure_detached()
Quoted 19 lines · Suren Baghdasaryan
vma_start_read()can temporarily raisevm_refcntof a write-locked and detached vma:// vm_refcnt==1 (attached)
vma_start_write()vma->vm_lock_seq = mm->mm_lock_seq vma_start_read() vm_refcnt++; // vm_refcnt==2
vma_mark_detached()vm_refcnt--; // vm_refcnt==1// vma is detached but vm_refcnt!=0 temporarily
if (vma->vm_lock_seq == mm->mm_lock_seq) vma_refcount_put() vm_refcnt--; // vm_refcnt==0
Quoted 7 lines
This is currently not a problem when freeing the vma because RCU grace period should pass before
kmem_cache_free(vma)gets called and by that timevma_start_read()should be done andvm_refcntis 0. However once we introduce possibility of vma reuse before RCU grace period is over, this will become a problem (reused vma might be in non-detached state). Introducevma_ensure_detached()for the writer to wait for readers until they exitvma_start_read().
So aside from the lockdep problem (which I think is fixable), the normal way to fix the above is to make dec_and_test() do the kmem_cache_free().
Then the last user does the free and everything just works.
Suren Baghdasaryan
maintainerRe: [PATCH v6 13/16] mm: introduce vma_ensure_detached()
Quoted 29 lines · Peter Zijlstra
Quoted 19 lines · Suren Baghdasaryan
vma_start_read()can temporarily raisevm_refcntof a write-locked and detached vma:// vm_refcnt==1 (attached)
vma_start_write()vma->vm_lock_seq = mm->mm_lock_seq vma_start_read() vm_refcnt++; // vm_refcnt==2
vma_mark_detached()vm_refcnt--; // vm_refcnt==1// vma is detached but vm_refcnt!=0 temporarily
if (vma->vm_lock_seq == mm->mm_lock_seq) vma_refcount_put() vm_refcnt--; // vm_refcnt==0Quoted 7 lines
This is currently not a problem when freeing the vma because RCU grace period should pass before
kmem_cache_free(vma)gets called and by that timevma_start_read()should be done andvm_refcntis 0. However once we introduce possibility of vma reuse before RCU grace period is over, this will become a problem (reused vma might be in non-detached state). Introducevma_ensure_detached()for the writer to wait for readers until they exitvma_start_read().
Quoted 4 lines
So aside from the lockdep problem (which I think is fixable), the normal way to fix the above is to make
dec_and_test()do thekmem_cache_free().Then the last user does the free and everything just works.
I see your point. Let me reply in the other patch where you have more comments about this.
Suren Baghdasaryan
maintainerPATCH v6 14/16 mm: prepare lock_vma_under_rcu() for vma reuse possibility
In reply to Suren BaghdasaryanOnce we make vma cache SLAB_TYPESAFE_BY_RCU, it will be possible for a vma to be reused and attached to another mm after lock_vma_under_rcu() locks the vma. lock_vma_under_rcu() should ensure that vma_start_read() is using the original mm and after locking the vma it should ensure that vma->vm_mm has not changed from under us.
--- include/linux/mm.h | 10 ++++++---- mm/memory.c | 7 ++++--- 2 files changed, 10 insertions(+), 7 deletions(-)
include/linux/mm.h +6 -4
@@ -725,8 +725,10 @@ static inline void vma_refcount_put(struct vm_area_struct *vma)* Try to read-lock a vma. The function is allowed to occasionally yield false * locked result to avoid performance overhead, in which case we fall back to * using mmap_lock. The function should never yield false unlocked result. +* False locked result is possible if mm_lock_seq overflows or if vma gets +* reused and attached to a different mm before we lock it. */ -static inline bool vma_start_read (struct vm_area_struct *vma) +static inline bool vma_start_read (struct mm_struct *mm,struct vm_area_struct *vma) {int oldcnt;
@@ -737,7 +739,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* we don't rely on for anything - the mm_lock_seq read against which we * need ordering is below. */ -if (READ_ONCE (vma->vm_lock_seq) ==READ_ONCE (vma->vm_mm->mm_lock_seq.sequence)) +if (READ_ONCE (vma->vm_lock_seq) ==READ_ONCE (mm->mm_lock_seq.sequence))return false;
@@ -762,7 +764,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)* This pairs with RELEASE semantics in vma_end_write_all(). */ if (oldcnt &VMA_STATE_LOCKED || -unlikely (vma->vm_lock_seq ==raw_read_seqcount (&vma->vm_mm->mm_lock_seq))) { +unlikely (vma->vm_lock_seq ==raw_read_seqcount (&mm->mm_lock_seq))) {vma_refcount_put (vma);return false; }
@@ -918,7 +920,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,#else /* CONFIG_PER_VMA_LOCK */ static inline void vma_lock_init (struct vm_area_struct *vma) {} -static inline bool vma_start_read (struct vm_area_struct *vma) +static inline bool vma_start_read (struct mm_struct *mm,struct vm_area_struct *vma) {return false; }static inline void vma_end_read (struct vm_area_struct *vma) {}static inline void vma_start_write (struct vm_area_struct *vma) {}
mm/memory.c +4 -3
@@ -6405,7 +6405,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,if (!vma) goto inval; -if (!vma_start_read (vma)) +if (!vma_start_read (mm, vma)) goto inval;/*
@@ -6415,8 +6415,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,* fields are accessible for RCU readers. */ -/* Check since vm_start/vm_end might change before we lock the VMA */ -if (unlikely (address < vma->vm_start || address >= vma->vm_end)) +/* Check if the vma we locked is the right one. */ +if (unlikely (vma->vm_mm != mm || + address < vma->vm_start || address >= vma->vm_end)) goto inval_end_read;rcu_read_unlock ();
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 15/16 mm: make vma cache SLAB_TYPESAFE_BY_RCU
In reply to Suren BaghdasaryanTo enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected by lock_vma_under_rcu(). Current checks are sufficient as long as vma is detached before it is freed. Implement this guarantee by calling vma_ensure_detached() before vma is freed and make vm_area_cachep SLAB_TYPESAFE_BY_RCU. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.
--- include/linux/mm.h | 2 -- include/linux/mm_types.h | 10 +++++++--- include/linux/slab.h | 6 ------ kernel/fork.c | 34 ++++++++++---------------------- mm/mmap.c | 8 +++++++- mm/vma.c | 15 +++----------- mm/vma.h | 2 +- tools/testing/vma/vma_internal.h | 7 +------ 8 files changed, 29 insertions(+), 55 deletions(-)
include/linux/mm.h +0 -2
@@ -258,8 +258,6 @@ void setup_initial_init_mm(void *start_code, void *end_code,struct vm_area_struct *vm_area_alloc (struct mm_struct *);struct vm_area_struct *vm_area_dup (struct vm_area_struct *);void vm_area_free (struct vm_area_struct *); -/* Use only if VMA has no other users */ -void __vm_area_free (struct vm_area_struct *vma);#ifndef CONFIG_MMU extern struct rb_root nommu_region_tree;
include/linux/mm_types.h +7 -3
@@ -544,6 +544,12 @@ static inline void *folio_get_private(struct folio *folio)typedef unsigned long vm_flags_t ; +/* +* freeptr_t represents a SLUB freelist pointer, which might be encoded +* and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. +*/ +typedef struct {unsigned long v; }freeptr_t ; +/* * A region containing a mapping of a non-memory backed file under NOMMU * conditions. These are held in a global tree and are pinned by the VMAs that
@@ -658,9 +664,7 @@ struct vm_area_struct {unsigned long vm_start;unsigned long vm_end; }; -#ifdef CONFIG_PER_VMA_LOCK -struct rcu_head vm_rcu;/* Used for deferred freeing. */ -#endif +freeptr_t vm_freeptr;/* Pointer used by SLAB_TYPESAFE_BY_RCU */ };/*
include/linux/slab.h +0 -6
@@ -234,12 +234,6 @@ enum _slab_flag_bits {#define SLAB_NO_OBJ_EXT __SLAB_FLAG_UNUSED#endif -/* -* freeptr_t represents a SLUB freelist pointer, which might be encoded -* and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. -*/ -typedef struct {unsigned long v; }freeptr_t ; -/* * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests. *
kernel/fork.c +10 -24
@@ -471,36 +471,16 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)return new; } -void __vm_area_free (struct vm_area_struct *vma) +void vm_area_free (struct vm_area_struct *vma) {#ifdef CONFIG_PER_VMA_LOCK -/* The vma should be detached while being destroyed. */ -VM_BUG_ON_VMA (!is_vma_detached (vma), vma); +vma_ensure_detached (vma);#endif vma_numab_state_free (vma);free_anon_vma_name (vma);kmem_cache_free (vm_area_cachep, vma); } -#ifdef CONFIG_PER_VMA_LOCK -static void vm_area_free_rcu_cb (struct rcu_head *head) -{ - struct vm_area_struct *vma = container_of(head, struct vm_area_struct, - vm_rcu); - -__vm_area_free (vma); -} -#endif - -void vm_area_free (struct vm_area_struct *vma) -{ -#ifdef CONFIG_PER_VMA_LOCK -call_rcu (&vma->vm_rcu, vm_area_free_rcu_cb); -#else -__vm_area_free (vma); -#endif -} -static void account_kernel_stack (struct task_struct *tsk,int account) {if (IS_ENABLED (CONFIG_VMAP_STACK )) {
@@ -3147,6 +3127,11 @@ void __init mm_cache_init(void)void __init proc_caches_init (void ) { +struct kmem_cache_args args = { + .use_freeptr_offset = true, + .freeptr_offset = offsetof(struct vm_area_struct , vm_freeptr), + }; + sighand_cachep = kmem_cache_create("sighand_cache", sizeof(struct sighand_struct), 0, SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU|
@@ -3163,8 +3148,9 @@ void __init proc_caches_init(void)sizeof (struct fs_struct ),0 ,SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT , NULL); - vm_area_cachep =KMEM_CACHE (vm_area_struct, -SLAB_HWCACHE_ALIGN |SLAB_NO_MERGE |SLAB_PANIC | + vm_area_cachep =kmem_cache_create ("vm_area_struct" , +sizeof (struct vm_area_struct ), &args, +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_TYPESAFE_BY_RCU |SLAB_ACCOUNT );mmap_init ();nsproxy_cache_init ();
mm/mmap.c +7 -1
@@ -1284,7 +1284,13 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); - remove_vma(vma, /* unreachable = */ true); +#ifdef CONFIG_PER_VMA_LOCK + if (!is_vma_detached(vma)) { + vma_start_write(vma); + vma_mark_detached(vma); + } +#endif + remove_vma(vma); count++; cond_resched(); vma = vma_next(&vmi);
mm/vma.c +3 -12
@@ -406,23 +406,14 @@ static bool can_vma_merge_right(struct vma_merge_struct *vmg,/* * Close a vm structure and free it. */ -void remove_vma (struct vm_area_struct *vma,bool unreachable) +void remove_vma (struct vm_area_struct *vma) {might_sleep ();vma_close (vma);if (vma->vm_file)fput (vma->vm_file);mpol_put (vma_policy (vma)); -if (unreachable) { -#ifdef CONFIG_PER_VMA_LOCK -if (!is_vma_detached (vma)) { -vma_start_write (vma); -vma_mark_detached (vma); - } -#endif -__vm_area_free (vma); - }else -vm_area_free (vma); +vm_area_free (vma); }/*
@@ -1206,7 +1197,7 @@ static void vms_complete_munmap_vmas(struct vma_munmap_struct *vms,/* Remove and clean up vmas */ mas_set (mas_detach,0 );mas_for_each (mas_detach, vma,ULONG_MAX ) -remove_vma (vma,/* unreachable = */ false); +remove_vma (vma);vm_unacct_memory (vms->nr_accounted);validate_mm (mm);
mm/vma.h +1 -1
@@ -170,7 +170,7 @@ int do_vmi_munmap(struct vma_iterator *vmi, struct mm_struct *mm, unsigned long start, size_t len, struct list_head *uf, bool unlock); -void remove_vma (struct vm_area_struct *vma,bool unreachable); +void remove_vma (struct vm_area_struct *vma);void unmap_region (struct ma_state *mas,struct vm_area_struct *vma,struct vm_area_struct *prev,struct vm_area_struct *next);
tools/testing/vma/vma_internal.h +1 -6
@@ -700,14 +700,9 @@ static inline void mpol_put(struct mempolicy *) { } -static inline void __vm_area_free (struct vm_area_struct *vma) -{ -free (vma); -} -static inline void vm_area_free (struct vm_area_struct *vma) { -__vm_area_free (vma); +free (vma); }static inline void lru_add_drain (void )
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerPATCH v6 16/16 docs/mm: document latest changes to vm_lock
In reply to Suren BaghdasaryanChange the documentation to reflect that vm_lock is integrated into vma and replaced with vm_refcnt.
Document newly introduced vma_start_read_locked{_nested} functions.
--- Documentation/mm/process_addrs.rst | 44 ++++++++++++++++++------------ 1 file changed, 26 insertions(+), 18 deletions(-)
Documentation/mm/process_addrs.rst +26 -18
@@ -716,9 +716,14 @@ calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU critical section, then attempts to VMA lock it via :c:func:`!vma_start_read`, before releasing the RCU lock via :c:func:`!rcu_read_unlock`. -VMA read locks hold the read lock on the :c:member:`!vma->vm_lock` semaphore for -their duration and the caller of :c:func:`!lock_vma_under_rcu` must release it -via :c:func:`!vma_end_read`. +In cases when the user already holds mmap read lock, :c:func:`!vma_start_read_locked` +and :c:func:`!vma_start_read_locked_nested` can be used. These functions do not +fail due to lock contention but the caller should still check their return values +in case they fail for other reasons. + +VMA read locks increment :c:member:`!vma.vm_refcnt` reference counter for their +duration and the caller of :c:func:`!lock_vma_under_rcu` must drop it via +:c:func:`!vma_end_read`. VMA **write** locks are acquired via :c:func:`!vma_start_write` in instances where a VMA is about to be modified, unlike :c:func:`!vma_start_read` the lock is always
@@ -726,9 +731,9 @@ acquired. An mmap write lock **must** be held for the duration of the VMA write lock, releasing or downgrading the mmap write lock also releases the VMA write lock so there is no :c:func:`!vma_end_write` function. -Note that a semaphore write lock is not held across a VMA lock. Rather, a -sequence number is used for serialisation, and the write semaphore is only -acquired at the point of write lock to update this. +Note that when write-locking a VMA lock, the :c:member:`!vma.vm_refcnt` is temporarily +modified so that readers can detect the presense of a writer. The reference counter is +restored once the vma sequence number used for serialisation is updated. This ensures the semantics we require - VMA write locks provide exclusive write access to the VMA.
@@ -738,7 +743,7 @@ Implementation details The VMA lock mechanism is designed to be a lightweight means of avoiding the use of the heavily contended mmap lock. It is implemented using a combination of a -read/write semaphore and sequence numbers belonging to the containing +reference counter and sequence numbers belonging to the containing :c:struct:`!struct mm_struct` and the VMA. Read locks are acquired via :c:func:`!vma_start_read`, which is an optimistic
@@ -779,28 +784,31 @@ release of any VMA locks on its release makes sense, as you would never want to keep VMAs locked across entirely separate write operations. It also maintains correct lock ordering. -Each time a VMA read lock is acquired, we acquire a read lock on the -:c:member:`!vma->vm_lock` read/write semaphore and hold it, while checking that -the sequence count of the VMA does not match that of the mm. +Each time a VMA read lock is acquired, we increment :c:member:`!vma.vm_refcnt` +reference counter and check that the sequence count of the VMA does not match +that of the mm. -If it does, the read lock fails. If it does not, we hold the lock, excluding -writers, but permitting other readers, who will also obtain this lock under RCU. +If it does, the read lock fails and :c:member:`!vma.vm_refcnt` is dropped. +If it does not, we keep the reference counter raised, excluding writers, but +permitting other readers, who can also obtain this lock under RCU. Importantly, maple tree operations performed in :c:func:`!lock_vma_under_rcu` are also RCU safe, so the whole read lock operation is guaranteed to function correctly. -On the write side, we acquire a write lock on the :c:member:`!vma->vm_lock` -read/write semaphore, before setting the VMA's sequence number under this lock, -also simultaneously holding the mmap write lock. +On the write side, we set a bit in :c:member:`!vma.vm_refcnt` which can't be +modified by readers and wait for all readers to drop their reference count. +Once there are no readers, VMA's sequence number is set to match that of the +mm. During this entire operation mmap write lock is held. This way, if any read locks are in effect, :c:func:`!vma_start_write` will sleep until these are finished and mutual exclusion is achieved. -After setting the VMA's sequence number, the lock is released, avoiding -complexity with a long-term held write lock. +After setting the VMA's sequence number, the bit in :c:member:`!vma.vm_refcnt` +indicating a writer is cleared. From this point on, VMA's sequence number will +indicate VMA's write-locked state until mmap write lock is dropped or downgraded. -This clever combination of a read/write semaphore and sequence count allows for +This clever combination of a reference counter and sequence count allows for fast RCU-based per-VMA lock acquisition (especially on page fault, though utilised elsewhere) with minimal complexity around lock ordering.
Signature · 1 line
2.47.1.613.gc27f4b7a9f-goog
Suren Baghdasaryan
maintainerQuoted 52 lines · Suren Baghdasaryan
Back when per-vma locks were introduces,
vm_lockwas moved out ofvm_area_structin [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3].
Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging thevm_lockback intovm_area_structalso allowsvm_area_structto useSLAB_TYPESAFE_BY_RCUlater in this patchset. This patchset:
1. movesvm_lockback intovm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned to minimize cacheline sharing;
2. changesvm_area_structinitialization to mark new vma as detached until it is inserted into vma tree;
3. replacesvm_lockandvma->detachedflag with a reference counter;
4. changesvm_area_structcache toSLAB_TYPESAFE_BY_RCUto allow for their reuse and to minimizecall_rcu()calls.Pagefault microbenchmarks show performance improvement:
Hmean faults/cpu-1 507926.5547 ( 0.00%) 506519.3692 * -0.28%* Hmean faults/cpu-4 479119.7051 ( 0.00%) 481333.6802 * 0.46%* Hmean faults/cpu-7 452880.2961 ( 0.00%) 455845.6211 * 0.65%* Hmean faults/cpu-12 347639.1021 ( 0.00%) 352004.2254 * 1.26%* Hmean faults/cpu-21 200061.2238 ( 0.00%) 229597.0317 * 14.76%* Hmean faults/cpu-30 145251.2001 ( 0.00%) 164202.5067 * 13.05%* Hmean faults/cpu-48 106848.4434 ( 0.00%) 120641.5504 * 12.91%* Hmean faults/cpu-56 92472.3835 ( 0.00%) 103464.7916 * 11.89%* Hmean faults/sec-1 507566.1468 ( 0.00%) 506139.0811 * -0.28%* Hmean faults/sec-4 1880478.2402 ( 0.00%) 1886795.6329 * 0.34%* Hmean faults/sec-7 3106394.3438 ( 0.00%) 3140550.7485 * 1.10%* Hmean faults/sec-12 4061358.4795 ( 0.00%) 4112477.0206 * 1.26%* Hmean faults/sec-21 3988619.1169 ( 0.00%) 4577747.1436 * 14.77%* Hmean faults/sec-30 3909839.5449 ( 0.00%) 4311052.2787 * 10.26%* Hmean faults/sec-48 4761108.4691 ( 0.00%) 5283790.5026 * 10.98%* Hmean faults/sec-56 4885561.4590 ( 0.00%) 5415839.4045 * 10.85%*Changes since v5 [4]
- Added Reviewed-by, per Vlastimil Babka;
- Added replacement ofvm_lockandvma->detachedflag withvm_refcnt, per Peter Zijlstra and Matthew Wilcox
- Marked vmas detached duringexit_mmap;
- Ensureed vmas are in detached state before they are freed;
- ChangedSLAB_TYPESAFE_BY_RCUpatch to not require ctor, leading to a much simpler code;
- Removed unnecessary patch [5]
- Updated documentation to reflect changes tovm_lock;
Quoted 2 lines
Patchset applies over mm-unstable after reverting v5 of this patchset [4] (currently 687e99a5faa5-905ab222508a)
^^^
Please be aware of this if trying to apply to a branch. mm-unstable contains an older version of this patchset which needs to be reverted before this one can be applied.
Quoted 49 lines
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/
[3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://lore.kernel.org/all/[email protected]/Suren Baghdasaryan (16):
mm: introduce vma_start_read_locked{_nested} helpers
mm: move per-vma lock intovm_area_struct
mm: mark vma as detached until it's added into vma tree mm/nommu: fix the last places where vma is not locked before being
attached
types: move struct rcuwait intotypes.h
mm: allow vma_start_read_locked/vma_start_read_locked_nested to fail
mm: movemmap_init_lock()out of the header file
mm: uninline the main body ofvma_start_write()
refcount: introduce __refcount_{add|inc}_not_zero_limited
mm: replacevm_lockand detached flag with a reference count
mm: enforce vma to be in detached state before freeing
mm: remove extravma_numab_state_init()call
mm: introducevma_ensure_detached()
mm: preparelock_vma_under_rcu()for vma reuse possibility
mm: make vma cacheSLAB_TYPESAFE_BY_RCU
docs/mm: document latest changes tovm_lockDocumentation/mm/process_addrs.rst | 44 ++++---- include/linux/mm.h | 162 +++++++++++++++++++++++------ include/linux/mm_types.h | 37 ++++--- include/linux/mmap_lock.h | 6 -- include/linux/rcuwait.h | 13 +-- include/linux/refcount.h | 20 +++- include/linux/slab.h | 6 -- include/linux/types.h | 12 +++ kernel/fork.c | 88 ++++------------ mm/init-mm.c | 1 + mm/memory.c | 75 +++++++++++-- mm/mmap.c | 8 +- mm/nommu.c | 2 + mm/userfaultfd.c | 31 +++--- mm/vma.c | 15 ++- mm/vma.h | 4 +- tools/testing/vma/linux/atomic.h | 5 + tools/testing/vma/vma_internal.h | 96 +++++++++-------- 18 files changed, 378 insertions(+), 247 deletions(-)--
2.47.1.613.gc27f4b7a9f-goog
Andrew Morton
maintainerQuoted 7 lines · On Mon, 16 Dec 2024 11:39:16 -0800 Suren Baghdasaryan <[email protected]> wrote:
Quoted 2 lines
Patchset applies over mm-unstable after reverting v5 of this patchset [4] (currently 687e99a5faa5-905ab222508a)
^^^
Please be aware of this if trying to apply to a branch. mm-unstable contains an older version of this patchset which needs to be reverted before this one can be applied.
I quietly updated mm-unstable to v6. I understand that a v7 is expected.
Suren Baghdasaryan
maintainerQuoted 6 lines · Andrew Morton
Quoted 3 lines · On Mon, 16 Dec 2024 11:39:16 -0800 Suren Baghdasaryan <[email protected]> wrote:
Quoted 2 lines
Patchset applies over mm-unstable after reverting v5 of this patchset [4] (currently 687e99a5faa5-905ab222508a)
Quoted 6 lines
Quoted 4 lines
^^^
Please be aware of this if trying to apply to a branch. mm-unstable contains an older version of this patchset which needs to be reverted before this one can be applied.I quietly updated mm-unstable to v6. I understand that a v7 is expected.
Thanks! Yes, I'll post v7 once our discussion with Peter on refcounting is concluded.
Could you please fixup the issue that Lokesh found in
https://lore.kernel.org/all/[email protected]/
?
Instead of
+ if (!vma_start_read_locked(*dst_vmap)) {
it should be:
+ if (vma_start_read_locked(*dst_vmap)) {
That's the only critical issue found in v6 so far. Thanks!