Skip to contentSkip to conversation

move per-vma lock into vm_area_struct

PATCH Superseded Suren Baghdasaryan 7 participants 34 messages Dec 6, 2024 – Dec 11, 2024

Subsystems (10)

v5 of 10 Superseded Latest is v10 →

lore
Share & tools
lore

Suren Baghdasaryan

author maintainer
#0 lore raw

v5 cover move per-vma lock into vm_area_struct

Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3].
Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. This patchset:
1. moves vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned to minimize cacheline sharing;
2. changes vm_area_struct initialization to mark new vma as detached until it is inserted into vma tree;
3. changes vm_area_struct cache to SLAB_TYPESAFE_BY_RCU to allow for their reuse and to minimize call_rcu() calls. To avoid bloating vm_area_struct, we introduce unioned freeptr_t field and allow freeptr_offset to be used with ctor in a separate patch.
Pagefault microbenchmarks do not show noticeable performance change.

Changes since v4 [4]
- Added SOBs, per Lorenzo Stoakes and Shakeel Butt;
- Changed vma_clear() and vma_copy() to set required vma members individually, per Matthew Wilcox
- Added comments in vma_start_read() about new false locked result possibilities, per Vlastimil Babka
- Added back freeptr_t into vm_area_struct, as it's harmless and can be used in the future to shrink the structure, per Vlastimil Babka
- Fixed the race in vm_area_alloc() when vma->detached was temporarily reset by memcpy() before setting it back, per Vlastimil Babka
- Added a patch allowing freeptr_offset to be used with ctor, per Vlastimil Babka

Patch applies over linux-next (due to vm_lock change [5] not in mm tree).

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/
[3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/
[4] https://lore.kernel.org/all/[email protected]/
[5] https://lore.kernel.org/all/[email protected]/

Suren Baghdasaryan (6):
mm: introduce vma_start_read_locked{_nested} helpers
mm: move per-vma lock into vm_area_struct
mm: mark vma as detached until it's added into vma tree
mm: make vma cache SLAB_TYPESAFE_BY_RCU mm/slab: allow freeptr_offset to be used with ctor docs/mm: document latest changes to vm_lock

 Documentation/mm/process_addrs.rst |  10 +-
 include/linux/mm.h                 | 107 ++++++++++++++----
 include/linux/mm_types.h           |  16 ++-
 include/linux/slab.h               |  11 +-
 kernel/fork.c                      | 168 ++++++++++++++++++++---------
 mm/memory.c                        |  17 ++-
 mm/slub.c                          |   2 +-
 mm/userfaultfd.c                   |  22 +---
 mm/vma.c                           |   8 +-
 mm/vma.h                           |   2 +
 tools/testing/vma/vma_internal.h   |  55 ++++------
 11 files changed, 267 insertions(+), 151 deletions(-)

base-commit: ebe1b11614e079c5e366ce9bd3c8f44ca0fbcc1b

Signature · 1 line
2.47.0.338.g60cca15819-goog
author maintainer
#1 lore raw

PATCH v5 1/6 mm: introduce vma_start_read_locked{_nested} helpers

Introduce helper functions which can be used to read-lock a VMA when holding mmap_lock for read. Replace direct accesses to vma->vm_lock with these new helpers.

---
 include/linux/mm.h | 24 ++++++++++++++++++++++++
 mm/userfaultfd.c   | 22 +++++-----------------
 2 files changed, 29 insertions(+), 17 deletions(-)
include/linux/mm.h +24 -0
index 187e42339d8e..c4a001972223 100644
@@ -734,6 +734,30 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
 	return true;
 }
 
+/*
+ * Use only while holding mmap read lock which guarantees that locking will not
+ * fail (nobody can concurrently write-lock the vma). vma_start_read() should
+ * not be used in such cases because it might fail due to mm_lock_seq overflow.
+ * This functionality is used to obtain vma read lock and drop the mmap read lock.
+ */
+static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass)
+{
+	mmap_assert_locked(vma->vm_mm);
+	down_read_nested(&vma->vm_lock->lock, subclass);
+}
+
+/*
+ * Use only while holding mmap read lock which guarantees that locking will not
+ * fail (nobody can concurrently write-lock the vma). vma_start_read() should
+ * not be used in such cases because it might fail due to mm_lock_seq overflow.
+ * This functionality is used to obtain vma read lock and drop the mmap read lock.
+ */
+static inline void vma_start_read_locked(struct vm_area_struct *vma)
+{
+	mmap_assert_locked(vma->vm_mm);
+	down_read(&vma->vm_lock->lock);
+}
+
 static inline void vma_end_read(struct vm_area_struct *vma)
 {
 	rcu_read_lock(); /* keeps vma alive till the end of up_read */
mm/userfaultfd.c +5 -17
index 8e16dc290ddf..bc9a66ec6a6e 100644
@@ -84,16 +84,8 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm,
 
 	mmap_read_lock(mm);
 	vma = find_vma_and_prepare_anon(mm, address);
-	if (!IS_ERR(vma)) {
-		/*
-		 * We cannot use vma_start_read() as it may fail due to
-		 * false locked (see comment in vma_start_read()). We
-		 * can avoid that by directly locking vm_lock under
-		 * mmap_lock, which guarantees that nobody can lock the
-		 * vma for write (vma_start_write()) under us.
-		 */
-		down_read(&vma->vm_lock->lock);
-	}
+	if (!IS_ERR(vma))
+		vma_start_read_locked(vma);
 
 	mmap_read_unlock(mm);
 	return vma;
@@ -1491,14 +1483,10 @@ static int uffd_move_lock(struct mm_struct *mm,
 	mmap_read_lock(mm);
 	err = find_vmas_mm_locked(mm, dst_start, src_start, dst_vmap, src_vmap);
 	if (!err) {
-		/*
-		 * See comment in uffd_lock_vma() as to why not using
-		 * vma_start_read() here.
-		 */
-		down_read(&(*dst_vmap)->vm_lock->lock);
+		vma_start_read_locked(*dst_vmap);
 		if (*dst_vmap != *src_vmap)
-			down_read_nested(&(*src_vmap)->vm_lock->lock,
-					 SINGLE_DEPTH_NESTING);
+			vma_start_read_locked_nested(*src_vmap,
+						SINGLE_DEPTH_NESTING);
 	}
 	mmap_read_unlock(mm);
 	return err;
Signature · 1 line
2.47.0.338.g60cca15819-goog
Reviewed-by on v5 1/6
#2 lore raw

Re: [PATCH v5 1/6] mm: introduce vma_start_read_locked{_nested} helpers

Quoted 8 lines · Suren Baghdasaryan

Introduce helper functions which can be used to read-lock a VMA when holding mmap_lock for read. Replace direct accesses to vma->vm_lock with these new helpers.

author maintainer
#3 lore raw

PATCH v5 2/6 mm: move per-vma lock into vm_area_struct

In reply to Suren Baghdasaryan

Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3].
Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. Move vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned as well. With kernel compiled using defconfig, this causes VMA memory consumption to grow from 160 (vm_area_struct) + 40 (vm_lock) bytes to 256 bytes:

slabinfo before:
<name> ... <objsize> <objperslab> <pagesperslab> : ...
vma_lock ... 40 102 1 : ...
vm_area_struct ... 160 51 2 : ...

slabinfo after moving vm_lock:
<name> ... <objsize> <objperslab> <pagesperslab> : ...
vm_area_struct ... 256 32 2 : ...

Aggregate VMA memory consumption per 1000 VMAs grows from 50 to 64 pages, which is 5.5MB per 100000 VMAs. Note that the size of this structure is dependent on the kernel configuration and typically the original size is higher than 160 bytes. Therefore these calculations are close to the worst case scenario. A more realistic vm_area_struct usage before this change is:

     <name>           ... <objsize> <objperslab> <pagesperslab> : ...
     vma_lock         ...     40  102    1 : ...
     vm_area_struct   ...    176   46    2 : ...

Aggregate VMA memory consumption per 1000 VMAs grows from 54 to 64 pages, which is 3.9MB per 100000 VMAs.
This memory consumption growth can be addressed later by optimizing the vm_lock.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/
[3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/

---
 include/linux/mm.h               | 28 ++++++++++--------
 include/linux/mm_types.h         |  6 ++--
 kernel/fork.c                    | 49 ++++----------------------------
 tools/testing/vma/vma_internal.h | 33 +++++----------------
 4 files changed, 32 insertions(+), 84 deletions(-)
include/linux/mm.h +16 -12
index c4a001972223..ee71a504ef88 100644
@@ -696,6 +696,12 @@ static inline void vma_numab_state_free(struct vm_area_struct *vma) {}
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_PER_VMA_LOCK
+static inline void vma_lock_init(struct vm_area_struct *vma)
+{
+	init_rwsem(&vma->vm_lock.lock);
+	vma->vm_lock_seq = UINT_MAX;
+}
+
 /*
  * Try to read-lock a vma. The function is allowed to occasionally yield false
  * locked result to avoid performance overhead, in which case we fall back to
@@ -713,7 +719,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
 	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence))
 		return false;
 
-	if (unlikely(down_read_trylock(&vma->vm_lock->lock) == 0))
+	if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0))
 		return false;
 
 	/*
@@ -728,7 +734,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
 	 * This pairs with RELEASE semantics in vma_end_write_all().
 	 */
 	if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) {
-		up_read(&vma->vm_lock->lock);
+		up_read(&vma->vm_lock.lock);
 		return false;
 	}
 	return true;
@@ -743,7 +749,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
 static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int subclass)
 {
 	mmap_assert_locked(vma->vm_mm);
-	down_read_nested(&vma->vm_lock->lock, subclass);
+	down_read_nested(&vma->vm_lock.lock, subclass);
 }
 
 /*
@@ -755,13 +761,13 @@ static inline void vma_start_read_locked_nested(struct vm_area_struct *vma, int
 static inline void vma_start_read_locked(struct vm_area_struct *vma)
 {
 	mmap_assert_locked(vma->vm_mm);
-	down_read(&vma->vm_lock->lock);
+	down_read(&vma->vm_lock.lock);
 }
 
 static inline void vma_end_read(struct vm_area_struct *vma)
 {
 	rcu_read_lock(); /* keeps vma alive till the end of up_read */
-	up_read(&vma->vm_lock->lock);
+	up_read(&vma->vm_lock.lock);
 	rcu_read_unlock();
 }
 
@@ -790,7 +796,7 @@ static inline void vma_start_write(struct vm_area_struct *vma)
 	if (__is_vma_write_locked(vma, &mm_lock_seq))
 		return;
 
-	down_write(&vma->vm_lock->lock);
+	down_write(&vma->vm_lock.lock);
 	/*
 	 * We should use WRITE_ONCE() here because we can have concurrent reads
 	 * from the early lockless pessimistic check in vma_start_read().
@@ -798,7 +804,7 @@ static inline void vma_start_write(struct vm_area_struct *vma)
 	 * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy.
 	 */
 	WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq);
-	up_write(&vma->vm_lock->lock);
+	up_write(&vma->vm_lock.lock);
 }
 
 static inline void vma_assert_write_locked(struct vm_area_struct *vma)
@@ -810,7 +816,7 @@ static inline void vma_assert_write_locked(struct vm_area_struct *vma)
 
 static inline void vma_assert_locked(struct vm_area_struct *vma)
 {
-	if (!rwsem_is_locked(&vma->vm_lock->lock))
+	if (!rwsem_is_locked(&vma->vm_lock.lock))
 		vma_assert_write_locked(vma);
 }
 
@@ -843,6 +849,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 
 #else /* CONFIG_PER_VMA_LOCK */
 
+static inline void vma_lock_init(struct vm_area_struct *vma) {}
 static inline bool vma_start_read(struct vm_area_struct *vma)
 		{ return false; }
 static inline void vma_end_read(struct vm_area_struct *vma) {}
@@ -877,10 +884,6 @@ static inline void assert_fault_locked(struct vm_fault *vmf)
 
 extern const struct vm_operations_struct vma_dummy_vm_ops;
 
-/*
- * WARNING: vma_init does not initialize vma->vm_lock.
- * Use vm_area_alloc()/vm_area_free() if vma needs locking.
- */
 static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
 {
 	memset(vma, 0, sizeof(*vma));
@@ -889,6 +892,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
 	INIT_LIST_HEAD(&vma->anon_vma_chain);
 	vma_mark_detached(vma, false);
 	vma_numab_state_init(vma);
+	vma_lock_init(vma);
 }
 
 /* Use when VMA is not part of the VMA tree and needs no locking */
include/linux/mm_types.h +4 -2
index 706b3c926a08..be3551654325 100644
@@ -700,8 +700,6 @@ struct vm_area_struct {
 	 * slowpath.
 	 */
 	unsigned int vm_lock_seq;
-	/* Unstable RCU readers are allowed to read this. */
-	struct vma_lock *vm_lock;
 #endif
 
 	/*
@@ -754,6 +752,10 @@ struct vm_area_struct {
 	struct vma_numab_state *numab_state;	/* NUMA Balancing state */
 #endif
 	struct vm_userfaultfd_ctx vm_userfaultfd_ctx;
+#ifdef CONFIG_PER_VMA_LOCK
+	/* Unstable RCU readers are allowed to read this. */
+	struct vma_lock vm_lock ____cacheline_aligned_in_smp;
+#endif
 } __randomize_layout;
 
 #ifdef CONFIG_NUMA
kernel/fork.c +5 -44

85 lines not on this page · raw message

tools/testing/vma/vma_internal.h +7 -26
index b973b3e41c83..568c18d24d53 100644
@@ -270,10 +270,10 @@ struct vm_area_struct {
 	/*
 	 * Can only be written (using WRITE_ONCE()) while holding both:
 	 *  - mmap_lock (in write mode)
-	 *  - vm_lock->lock (in write mode)
+	 *  - vm_lock.lock (in write mode)
 	 * Can be read reliably while holding one of:
 	 *  - mmap_lock (in read or write mode)
-	 *  - vm_lock->lock (in read or write mode)
+	 *  - vm_lock.lock (in read or write mode)
 	 * Can be read unreliably (using READ_ONCE()) for pessimistic bailout
 	 * while holding nothing (except RCU to keep the VMA struct allocated).
 	 *
@@ -282,7 +282,7 @@ struct vm_area_struct {
 	 * slowpath.
 	 */
 	unsigned int vm_lock_seq;
-	struct vma_lock *vm_lock;
+	struct vma_lock vm_lock;
 #endif
 
 	/*
@@ -459,17 +459,10 @@ static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi)
 	return mas_find(&vmi->mas, ULONG_MAX);
 }
 
-static inline bool vma_lock_alloc(struct vm_area_struct *vma)
+static inline void vma_lock_init(struct vm_area_struct *vma)
 {
-	vma->vm_lock = calloc(1, sizeof(struct vma_lock));
-
-	if (!vma->vm_lock)
-		return false;
-
-	init_rwsem(&vma->vm_lock->lock);
+	init_rwsem(&vma->vm_lock.lock);
 	vma->vm_lock_seq = UINT_MAX;
-
-	return true;
 }
 
 static inline void vma_assert_write_locked(struct vm_area_struct *);
@@ -492,6 +485,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
 	vma->vm_ops = &vma_dummy_vm_ops;
 	INIT_LIST_HEAD(&vma->anon_vma_chain);
 	vma_mark_detached(vma, false);
+	vma_lock_init(vma);
 }
 
 static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm)
@@ -502,10 +496,6 @@ static inline struct vm_area_struct *vm_area_alloc(struct mm_struct *mm)
 		return NULL;
 
 	vma_init(vma, mm);
-	if (!vma_lock_alloc(vma)) {
-		free(vma);
-		return NULL;
-	}
 
 	return vma;
 }
@@ -518,10 +508,7 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 		return NULL;
 
 	memcpy(new, orig, sizeof(*new));
-	if (!vma_lock_alloc(new)) {
-		free(new);
-		return NULL;
-	}
+	vma_lock_init(new);
 	INIT_LIST_HEAD(&new->anon_vma_chain);
 
 	return new;
@@ -691,14 +678,8 @@ static inline void mpol_put(struct mempolicy *)
 {
 }
 
-static inline void vma_lock_free(struct vm_area_struct *vma)
-{
-	free(vma->vm_lock);
-}
-
 static inline void __vm_area_free(struct vm_area_struct *vma)
 {
-	vma_lock_free(vma);
 	free(vma);
 }
 
Signature · 1 line
2.47.0.338.g60cca15819-goog
#4 lore raw

Re: [PATCH v5 2/6] mm: move per-vma lock into vm_area_struct

kernel test robot noticed the following build warnings:

Hi Suren,

kernel test robot noticed the following build warnings:

[auto build test WARNING on ebe1b11614e079c5e366ce9bd3c8f44ca0fbcc1b]

url:    https://github.com/intel-lab-lkp/linux/commits/Suren-Baghdasaryan/mm-introduce-vma_start_read_locked-_nested-helpers/20241207-065517
base:   ebe1b11614e079c5e366ce9bd3c8f44ca0fbcc1b
patch link:    https://lore.kernel.org/r/20241206225204.4008261-3-surenb%40google.com
patch subject: [PATCH v5 2/6] mm: move per-vma lock into vm_area_struct
config: s390-allyesconfig (https://download.01.org/0day-ci/archive/20241207/[email protected]/config)
compiler: s390-linux-gcc (GCC) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241207/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

In file included from mm/damon/vaddr.c:736:
mm/damon/tests/vaddr-kunit.h: In function 'damon_test_three_regions_in_vmas':

>> mm/damon/tests/vaddr-kunit.h:92:1: warning: the frame size of 3424 bytes is larger than 2048 bytes [-Wframe-larger-than=]
      92 | }
         | ^

vim +92 mm/damon/tests/vaddr-kunit.h

17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  38  
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  39  /*
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  40   * Test __damon_va_three_regions() function
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  41   *
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  42   * In case of virtual memory address spaces monitoring, DAMON converts the
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  43   * complex and dynamic memory mappings of each target task to three
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  44   * discontiguous regions which cover every mapped areas.  However, the three
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  45   * regions should not include the two biggest unmapped areas in the original
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  46   * mapping, because the two biggest areas are normally the areas between 1)
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  47   * heap and the mmap()-ed regions, and 2) the mmap()-ed regions and stack.
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  48   * Because these two unmapped areas are very huge but obviously never accessed,
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  49   * covering the region is just a waste.
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  50   *
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  51   * '__damon_va_three_regions() receives an address space of a process.  It
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  52   * first identifies the start of mappings, end of mappings, and the two biggest
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  53   * unmapped areas.  After that, based on the information, it constructs the
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  54   * three regions and returns.  For more detail, refer to the comment of
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  55   * 'damon_init_regions_of()' function definition in 'mm/damon.c' file.
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  56   *
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  57   * For example, suppose virtual address ranges of 10-20, 20-25, 200-210,
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  58   * 210-220, 300-305, and 307-330 (Other comments represent this mappings in
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  59   * more short form: 10-20-25, 200-210-220, 300-305, 307-330) of a process are
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  60   * mapped.  To cover every mappings, the three regions should start with 10,
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  61   * and end with 305.  The process also has three unmapped areas, 25-200,
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  62   * 220-300, and 305-307.  Among those, 25-200 and 220-300 are the biggest two
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  63   * unmapped areas, and thus it should be converted to three regions of 10-25,
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  64   * 200-220, and 300-330.
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  65   */
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  66  static void damon_test_three_regions_in_vmas(struct kunit *test)
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  67  {
d0cf3dd47f0d5d3 mm/damon/vaddr-test.h        Liam R. Howlett 2022-09-06  68  	static struct mm_struct mm;
ba7196e566516f7 mm/damon/tests/vaddr-kunit.h Leo Stone       2024-09-22  69  	struct damon_addr_range regions[3] = {0};
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  70  	/* 10-20-25, 200-210-220, 300-305, 307-330 */
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  71  	struct vm_area_struct vmas[] = {
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  72  		(struct vm_area_struct) {.vm_start = 10, .vm_end = 20},
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  73  		(struct vm_area_struct) {.vm_start = 20, .vm_end = 25},
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  74  		(struct vm_area_struct) {.vm_start = 200, .vm_end = 210},
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  75  		(struct vm_area_struct) {.vm_start = 210, .vm_end = 220},
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  76  		(struct vm_area_struct) {.vm_start = 300, .vm_end = 305},
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  77  		(struct vm_area_struct) {.vm_start = 307, .vm_end = 330},
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  78  	};
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  79  
f0679f9e6d88ae9 mm/damon/tests/vaddr-kunit.h SeongJae Park   2024-09-04  80  	mt_init_flags(&mm.mm_mt, MT_FLAGS_ALLOC_RANGE | MT_FLAGS_USE_RCU);
34403fa579514a6 mm/damon/vaddr-test.h        Liam R. Howlett 2023-01-20  81  	if (__link_vmas(&mm.mm_mt, vmas, ARRAY_SIZE(vmas)))
34403fa579514a6 mm/damon/vaddr-test.h        Liam R. Howlett 2023-01-20  82  		kunit_skip(test, "Failed to create VMA tree");
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  83  
d0cf3dd47f0d5d3 mm/damon/vaddr-test.h        Liam R. Howlett 2022-09-06  84  	__damon_va_three_regions(&mm, regions);
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  85  
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  86  	KUNIT_EXPECT_EQ(test, 10ul, regions[0].start);
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  87  	KUNIT_EXPECT_EQ(test, 25ul, regions[0].end);
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  88  	KUNIT_EXPECT_EQ(test, 200ul, regions[1].start);
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  89  	KUNIT_EXPECT_EQ(test, 220ul, regions[1].end);
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  90  	KUNIT_EXPECT_EQ(test, 300ul, regions[2].start);
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  91  	KUNIT_EXPECT_EQ(test, 330ul, regions[2].end);
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07 @92  }
17ccae8bb5c9289 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  93  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#5 lore raw

Re: [PATCH v5 2/6] mm: move per-vma lock into vm_area_struct

In reply to Suren Baghdasaryan
kernel test robot noticed the following build warnings:

Hi Suren,

kernel test robot noticed the following build warnings:

[auto build test WARNING on ebe1b11614e079c5e366ce9bd3c8f44ca0fbcc1b]

url:    https://github.com/intel-lab-lkp/linux/commits/Suren-Baghdasaryan/mm-introduce-vma_start_read_locked-_nested-helpers/20241207-065517
base:   ebe1b11614e079c5e366ce9bd3c8f44ca0fbcc1b
patch link:    https://lore.kernel.org/r/20241206225204.4008261-3-surenb%40google.com
patch subject: [PATCH v5 2/6] mm: move per-vma lock into vm_area_struct
config: x86_64-allyesconfig (https://download.01.org/0day-ci/archive/20241208/[email protected]/config)
compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241208/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

In file included from mm/damon/vaddr.c:10:
In file included from include/linux/highmem.h:8:
In file included from include/linux/cacheflush.h:5:
In file included from arch/x86/include/asm/cacheflush.h:5:
In file included from include/linux/mm.h:2251:
include/linux/vmstat.h:504:43: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum numa_stat_item') [-Wenum-enum-conversion]

     504 |         return vmstat_text[NR_VM_ZONE_STAT_ITEMS +
         |                            ~~~~~~~~~~~~~~~~~~~~~ ^
     505 |                            item];
         |                            ~~~~
   include/linux/vmstat.h:511:43: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum numa_stat_item') [-Wenum-enum-conversion]
     511 |         return vmstat_text[NR_VM_ZONE_STAT_ITEMS +
         |                            ~~~~~~~~~~~~~~~~~~~~~ ^
     512 |                            NR_VM_NUMA_EVENT_ITEMS +
         |                            ~~~~~~~~~~~~~~~~~~~~~~
   include/linux/vmstat.h:518:36: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
     518 |         return node_stat_name(NR_LRU_BASE + lru) + 3; // skip "nr_"
         |                               ~~~~~~~~~~~ ^ ~~~
   include/linux/vmstat.h:524:43: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum numa_stat_item') [-Wenum-enum-conversion]
     524 |         return vmstat_text[NR_VM_ZONE_STAT_ITEMS +
         |                            ~~~~~~~~~~~~~~~~~~~~~ ^
     525 |                            NR_VM_NUMA_EVENT_ITEMS +
         |                            ~~~~~~~~~~~~~~~~~~~~~~
   In file included from mm/damon/vaddr.c:736:
>> mm/damon/tests/vaddr-kunit.h:66:13: warning: stack frame size (2552) exceeds limit (2048) in 'damon_test_three_regions_in_vmas' [-Wframe-larger-than]

66 | static void damon_test_three_regions_in_vmas(struct kunit *test)
| ^
5 warnings generated.

vim +/damon_test_three_regions_in_vmas +66 mm/damon/tests/vaddr-kunit.h

17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  38  
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  39  /*
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  40   * Test __damon_va_three_regions() function
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  41   *
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  42   * In case of virtual memory address spaces monitoring, DAMON converts the
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  43   * complex and dynamic memory mappings of each target task to three
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  44   * discontiguous regions which cover every mapped areas.  However, the three
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  45   * regions should not include the two biggest unmapped areas in the original
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  46   * mapping, because the two biggest areas are normally the areas between 1)
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  47   * heap and the mmap()-ed regions, and 2) the mmap()-ed regions and stack.
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  48   * Because these two unmapped areas are very huge but obviously never accessed,
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  49   * covering the region is just a waste.
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  50   *
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  51   * '__damon_va_three_regions() receives an address space of a process.  It
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  52   * first identifies the start of mappings, end of mappings, and the two biggest
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  53   * unmapped areas.  After that, based on the information, it constructs the
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  54   * three regions and returns.  For more detail, refer to the comment of
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  55   * 'damon_init_regions_of()' function definition in 'mm/damon.c' file.
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  56   *
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  57   * For example, suppose virtual address ranges of 10-20, 20-25, 200-210,
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  58   * 210-220, 300-305, and 307-330 (Other comments represent this mappings in
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  59   * more short form: 10-20-25, 200-210-220, 300-305, 307-330) of a process are
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  60   * mapped.  To cover every mappings, the three regions should start with 10,
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  61   * and end with 305.  The process also has three unmapped areas, 25-200,
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  62   * 220-300, and 305-307.  Among those, 25-200 and 220-300 are the biggest two
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  63   * unmapped areas, and thus it should be converted to three regions of 10-25,
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  64   * 200-220, and 300-330.
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  65   */
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07 @66  static void damon_test_three_regions_in_vmas(struct kunit *test)
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  67  {
d0cf3dd47f0d5d mm/damon/vaddr-test.h        Liam R. Howlett 2022-09-06  68  	static struct mm_struct mm;
ba7196e566516f mm/damon/tests/vaddr-kunit.h Leo Stone       2024-09-22  69  	struct damon_addr_range regions[3] = {0};
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  70  	/* 10-20-25, 200-210-220, 300-305, 307-330 */
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  71  	struct vm_area_struct vmas[] = {
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  72  		(struct vm_area_struct) {.vm_start = 10, .vm_end = 20},
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  73  		(struct vm_area_struct) {.vm_start = 20, .vm_end = 25},
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  74  		(struct vm_area_struct) {.vm_start = 200, .vm_end = 210},
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  75  		(struct vm_area_struct) {.vm_start = 210, .vm_end = 220},
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  76  		(struct vm_area_struct) {.vm_start = 300, .vm_end = 305},
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  77  		(struct vm_area_struct) {.vm_start = 307, .vm_end = 330},
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  78  	};
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  79  
f0679f9e6d88ae mm/damon/tests/vaddr-kunit.h SeongJae Park   2024-09-04  80  	mt_init_flags(&mm.mm_mt, MT_FLAGS_ALLOC_RANGE | MT_FLAGS_USE_RCU);
34403fa579514a mm/damon/vaddr-test.h        Liam R. Howlett 2023-01-20  81  	if (__link_vmas(&mm.mm_mt, vmas, ARRAY_SIZE(vmas)))
34403fa579514a mm/damon/vaddr-test.h        Liam R. Howlett 2023-01-20  82  		kunit_skip(test, "Failed to create VMA tree");
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  83  
d0cf3dd47f0d5d mm/damon/vaddr-test.h        Liam R. Howlett 2022-09-06  84  	__damon_va_three_regions(&mm, regions);
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  85  
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  86  	KUNIT_EXPECT_EQ(test, 10ul, regions[0].start);
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  87  	KUNIT_EXPECT_EQ(test, 25ul, regions[0].end);
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  88  	KUNIT_EXPECT_EQ(test, 200ul, regions[1].start);
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  89  	KUNIT_EXPECT_EQ(test, 220ul, regions[1].end);
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  90  	KUNIT_EXPECT_EQ(test, 300ul, regions[2].start);
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  91  	KUNIT_EXPECT_EQ(test, 330ul, regions[2].end);
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  92  }
17ccae8bb5c928 mm/damon/vaddr-test.h        SeongJae Park   2021-09-07  93  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
Reviewed-by on v5 2/6
#6 lore raw

Re: [PATCH v5 2/6] mm: move per-vma lock into vm_area_struct

In reply to Suren Baghdasaryan
Quoted 42 lines · Suren Baghdasaryan

Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3].
Splitting single logical structure into multiple ones leads to more complicated management, extra pointer dereferences and overall less maintainable code. When that split-away part is a lock, it complicates things even further. With no performance benefits, there are no reasons for this split. Merging the vm_lock back into vm_area_struct also allows vm_area_struct to use SLAB_TYPESAFE_BY_RCU later in this patchset. Move vm_lock back into vm_area_struct, aligning it at the cacheline boundary and changing the cache to be cacheline-aligned as well. With kernel compiled using defconfig, this causes VMA memory consumption to grow from 160 (vm_area_struct) + 40 (vm_lock) bytes to 256 bytes:

slabinfo before:
<name> ... <objsize> <objperslab> <pagesperslab> : ...
vma_lock ... 40 102 1 : ...
vm_area_struct ... 160 51 2 : ...

slabinfo after moving vm_lock:
<name> ... <objsize> <objperslab> <pagesperslab> : ...
vm_area_struct ... 256 32 2 : ...

Aggregate VMA memory consumption per 1000 VMAs grows from 50 to 64 pages, which is 5.5MB per 100000 VMAs. Note that the size of this structure is dependent on the kernel configuration and typically the original size is higher than 160 bytes. Therefore these calculations are close to the worst case scenario. A more realistic vm_area_struct usage before this change is:

     <name>           ... <objsize> <objperslab> <pagesperslab> : ...
     vma_lock         ...     40  102    1 : ...
     vm_area_struct   ...    176   46    2 : ...

Aggregate VMA memory consumption per 1000 VMAs grows from 54 to 64 pages, which is 3.9MB per 100000 VMAs.
This memory consumption growth can be addressed later by optimizing the vm_lock.

Quoted 7 lines

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/ZsQyI%2F087V34JoIt@xsang-OptiPlex-9020/
[3] https://lore.kernel.org/all/CAJuCfpEisU8Lfe96AYJDZ+OM4NoPmnw9bP53cT_kbfP_pR+-2g@mail.gmail.com/

author maintainer
#7 lore raw

PATCH v5 3/6 mm: mark vma as detached until it's added into vma tree

In reply to Suren Baghdasaryan

Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree.
Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter.

---
 include/linux/mm.h               | 27 ++++++++++++++++++++-------
 kernel/fork.c                    |  4 ++++
 mm/memory.c                      |  2 +-
 mm/vma.c                         |  6 +++---
 mm/vma.h                         |  2 ++
 tools/testing/vma/vma_internal.h | 17 ++++++++++++-----
 6 files changed, 42 insertions(+), 16 deletions(-)
include/linux/mm.h +20 -7
index ee71a504ef88..2bf38c1e9cca 100644
@@ -820,12 +820,21 @@ static inline void vma_assert_locked(struct vm_area_struct *vma)
 		vma_assert_write_locked(vma);
 }
 
-static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached)
+static inline void vma_mark_attached(struct vm_area_struct *vma)
+{
+	vma->detached = false;
+}
+
+static inline void vma_mark_detached(struct vm_area_struct *vma)
 {
 	/* When detaching vma should be write-locked */
-	if (detached)
-		vma_assert_write_locked(vma);
-	vma->detached = detached;
+	vma_assert_write_locked(vma);
+	vma->detached = true;
+}
+
+static inline bool is_vma_detached(struct vm_area_struct *vma)
+{
+	return vma->detached;
 }
 
 static inline void release_fault_lock(struct vm_fault *vmf)
@@ -856,8 +865,8 @@ static inline void vma_end_read(struct vm_area_struct *vma) {}
 static inline void vma_start_write(struct vm_area_struct *vma) {}
 static inline void vma_assert_write_locked(struct vm_area_struct *vma)
 		{ mmap_assert_write_locked(vma->vm_mm); }
-static inline void vma_mark_detached(struct vm_area_struct *vma,
-				     bool detached) {}
+static inline void vma_mark_attached(struct vm_area_struct *vma) {}
+static inline void vma_mark_detached(struct vm_area_struct *vma) {}
 
 static inline struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 		unsigned long address)
@@ -890,7 +899,10 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
 	vma->vm_mm = mm;
 	vma->vm_ops = &vma_dummy_vm_ops;
 	INIT_LIST_HEAD(&vma->anon_vma_chain);
-	vma_mark_detached(vma, false);
+#ifdef CONFIG_PER_VMA_LOCK
+	/* vma is not locked, can't use vma_mark_detached() */
+	vma->detached = true;
+#endif
 	vma_numab_state_init(vma);
 	vma_lock_init(vma);
 }
@@ -1085,6 +1097,7 @@ static inline int vma_iter_bulk_store(struct vma_iterator *vmi,
 	if (unlikely(mas_is_err(&vmi->mas)))
 		return -ENOMEM;
 
+	vma_mark_attached(vma);
 	return 0;
 }
 
kernel/fork.c +4 -0
index 21660a9ad97a..71990f46aa4e 100644
@@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 	data_race(memcpy(new, orig, sizeof(*new)));
 	vma_lock_init(new);
 	INIT_LIST_HEAD(&new->anon_vma_chain);
+#ifdef CONFIG_PER_VMA_LOCK
+	/* vma is not locked, can't use vma_mark_detached() */
+	new->detached = true;
+#endif
 	vma_numab_state_init(new);
 	dup_anon_vma_name(orig, new);
 
mm/memory.c +1 -1
index f823906a4a0f..b252f19b28c9 100644
@@ -6372,7 +6372,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 		goto inval;
 
 	/* Check if the VMA got isolated after we found it */
-	if (vma->detached) {
+	if (is_vma_detached(vma)) {
 		vma_end_read(vma);
 		count_vm_vma_lock_event(VMA_LOCK_MISS);
 		/* The area was replaced with another one */
mm/vma.c +3 -3
index a06747845cac..cdc63728f47f 100644
@@ -327,7 +327,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi,
 
 	if (vp->remove) {
 again:
-		vma_mark_detached(vp->remove, true);
+		vma_mark_detached(vp->remove);
 		if (vp->file) {
 			uprobe_munmap(vp->remove, vp->remove->vm_start,
 				      vp->remove->vm_end);
@@ -1220,7 +1220,7 @@ static void reattach_vmas(struct ma_state *mas_detach)
 
 	mas_set(mas_detach, 0);
 	mas_for_each(mas_detach, vma, ULONG_MAX)
-		vma_mark_detached(vma, false);
+		vma_mark_attached(vma);
 
 	__mt_destroy(mas_detach->tree);
 }
@@ -1295,7 +1295,7 @@ static int vms_gather_munmap_vmas(struct vma_munmap_struct *vms,
 		if (error)
 			goto munmap_gather_failed;
 
-		vma_mark_detached(next, true);
+		vma_mark_detached(next);
 		nrpages = vma_pages(next);
 
 		vms->nr_pages += nrpages;
mm/vma.h +2 -0
index 295d44ea54db..32d99b2963df 100644
@@ -156,6 +156,7 @@ static inline int vma_iter_store_gfp(struct vma_iterator *vmi,
 	if (unlikely(mas_is_err(&vmi->mas)))
 		return -ENOMEM;
 
+	vma_mark_attached(vma);
 	return 0;
 }
 
@@ -385,6 +386,7 @@ static inline void vma_iter_store(struct vma_iterator *vmi,
 
 	__mas_set_range(&vmi->mas, vma->vm_start, vma->vm_end - 1);
 	mas_store_prealloc(&vmi->mas, vma);
+	vma_mark_attached(vma);
 }
 
 static inline unsigned long vma_iter_addr(struct vma_iterator *vmi)
tools/testing/vma/vma_internal.h +12 -5
index 568c18d24d53..0cdc5f8c3d60 100644
@@ -465,13 +465,17 @@ static inline void vma_lock_init(struct vm_area_struct *vma)
 	vma->vm_lock_seq = UINT_MAX;
 }
 
+static inline void vma_mark_attached(struct vm_area_struct *vma)
+{
+	vma->detached = false;
+}
+
 static inline void vma_assert_write_locked(struct vm_area_struct *);
-static inline void vma_mark_detached(struct vm_area_struct *vma, bool detached)
+static inline void vma_mark_detached(struct vm_area_struct *vma)
 {
 	/* When detaching vma should be write-locked */
-	if (detached)
-		vma_assert_write_locked(vma);
-	vma->detached = detached;
+	vma_assert_write_locked(vma);
+	vma->detached = true;
 }
 
 extern const struct vm_operations_struct vma_dummy_vm_ops;
@@ -484,7 +488,8 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
 	vma->vm_mm = mm;
 	vma->vm_ops = &vma_dummy_vm_ops;
 	INIT_LIST_HEAD(&vma->anon_vma_chain);
-	vma_mark_detached(vma, false);
+	/* vma is not locked, can't use vma_mark_detached() */
+	vma->detached = true;
 	vma_lock_init(vma);
 }
 
@@ -510,6 +515,8 @@ static inline struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 	memcpy(new, orig, sizeof(*new));
 	vma_lock_init(new);
 	INIT_LIST_HEAD(&new->anon_vma_chain);
+	/* vma is not locked, can't use vma_mark_detached() */
+	new->detached = true;
 
 	return new;
 }
Signature · 1 line
2.47.0.338.g60cca15819-goog
Reviewed-by on v5 3/6
#8 lore raw

Re: [PATCH v5 3/6] mm: mark vma as detached until it's added into vma tree

Quoted 10 lines · Suren Baghdasaryan

Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree.
Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter.

Quoted 3 lines
#9 lore raw

Re: [PATCH v5 3/6] mm: mark vma as detached until it's added into vma tree

In reply to Suren Baghdasaryan
Quoted 19 lines · Suren Baghdasaryan

Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree.
Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter.

kernel/fork.c +0 -0
index 21660a9ad97a..71990f46aa4e 100644

kernel/fork.c struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) · line 471

Quoted 7 lines
kernel/fork.c +4 -0
@@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 	data_race(memcpy(new, orig, sizeof(*new)));
 	vma_lock_init(new);
 	INIT_LIST_HEAD(&new->anon_vma_chain);
+#ifdef CONFIG_PER_VMA_LOCK
+	/* vma is not locked, can't use vma_mark_detached() */
+	new->detached = true;
+#endif

It occured me we could probably move the detached setting to vma_lock_init() to avoid the #ifdef (also in the ctor in next patch)?

Quoted 3 lines
kernel/fork.c +0 -0
 	vma_numab_state_init(new);
 	dup_anon_vma_name(orig, new);
 
author maintainer
#10 lore raw

Re: [PATCH v5 3/6] mm: mark vma as detached until it's added into vma tree

Quoted 16 lines · Vlastimil Babka
Quoted 13 lines · Suren Baghdasaryan

Current implementation does not set detached flag when a VMA is first allocated. This does not represent the real state of the VMA, which is detached until it is added into mm's VMA tree. Fix this by marking new VMAs as detached and resetting detached flag only after VMA is added into a tree.
Introduce vma_mark_attached() to make the API more readable and to simplify possible future cleanup when vma->vm_mm might be used to indicate detached vma and vma_mark_attached() will need an additional mm parameter.

Quoted 15 lines

kernel/fork.c struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) · line 471

Quoted 12 lines
kernel/fork.c +4 -0
index 21660a9ad97a..71990f46aa4e 100644
@@ -465,6 +465,10 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
      data_race(memcpy(new, orig, sizeof(*new)));
      vma_lock_init(new);
      INIT_LIST_HEAD(&new->anon_vma_chain);
+#ifdef CONFIG_PER_VMA_LOCK
+     /* vma is not locked, can't use vma_mark_detached() */
+     new->detached = true;
+#endif

It occured me we could probably move the detached setting to vma_lock_init() to avoid the #ifdef (also in the ctor in next patch)?

Yes but setting vma->detached inside vma_lock_init() does not seem logical. I went back and forth on this and in the end decided to keep it this way. If the #ifdef-ery is too ugly I can introduce vma_init_detached() or something like that.

Quoted 4 lines
Quoted 4 lines
kernel/fork.c +0 -0
      vma_numab_state_init(new);
      dup_anon_vma_name(orig, new);
 
author maintainer
#11 lore raw

PATCH v5 4/6 mm: make vma cache SLAB_TYPESAFE_BY_RCU

In reply to Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

---
 include/linux/mm.h               |  36 +++++--
 include/linux/mm_types.h         |  10 +-
 include/linux/slab.h             |   6 --
 kernel/fork.c                    | 157 +++++++++++++++++++++++++------
 mm/memory.c                      |  15 ++-
 mm/vma.c                         |   2 +-
 tools/testing/vma/vma_internal.h |   7 +-
 7 files changed, 179 insertions(+), 54 deletions(-)
include/linux/mm.h +29 -7
index 2bf38c1e9cca..3568bcbc7c81 100644
@@ -257,7 +257,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *);
 struct vm_area_struct *vm_area_dup(struct vm_area_struct *);
 void vm_area_free(struct vm_area_struct *);
 /* Use only if VMA has no other users */
-void __vm_area_free(struct vm_area_struct *vma);
+void vm_area_free_unreachable(struct vm_area_struct *vma);
 
 #ifndef CONFIG_MMU
 extern struct rb_root nommu_region_tree;
@@ -706,8 +706,10 @@ static inline void vma_lock_init(struct vm_area_struct *vma)
  * Try to read-lock a vma. The function is allowed to occasionally yield false
  * locked result to avoid performance overhead, in which case we fall back to
  * using mmap_lock. The function should never yield false unlocked result.
+ * False locked result is possible if mm_lock_seq overflows or if vma gets
+ * reused and attached to a different mm before we lock it.
  */
-static inline bool vma_start_read(struct vm_area_struct *vma)
+static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma)
 {
 	/*
 	 * Check before locking. A race might cause false locked result.
@@ -716,7 +718,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
 	 * we don't rely on for anything - the mm_lock_seq read against which we
 	 * need ordering is below.
 	 */
-	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence))
+	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
 		return false;
 
 	if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0))
@@ -733,7 +735,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
 	 * after it has been unlocked.
 	 * This pairs with RELEASE semantics in vma_end_write_all().
 	 */
-	if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) {
+	if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) {
 		up_read(&vma->vm_lock.lock);
 		return false;
 	}
@@ -822,7 +824,15 @@ static inline void vma_assert_locked(struct vm_area_struct *vma)
 
 static inline void vma_mark_attached(struct vm_area_struct *vma)
 {
-	vma->detached = false;
+	/*
+	 * This pairs with smp_rmb() inside is_vma_detached().
+	 * vma is marked attached after all vma modifications are done and it
+	 * got added into the vma tree. All prior vma modifications should be
+	 * made visible before marking the vma attached.
+	 */
+	smp_wmb();
+	/* This pairs with READ_ONCE() in is_vma_detached(). */
+	WRITE_ONCE(vma->detached, false);
 }
 
 static inline void vma_mark_detached(struct vm_area_struct *vma)
@@ -834,7 +844,18 @@ static inline void vma_mark_detached(struct vm_area_struct *vma)
 
 static inline bool is_vma_detached(struct vm_area_struct *vma)
 {
-	return vma->detached;
+	bool detached;
+
+	/* This pairs with WRITE_ONCE() in vma_mark_attached(). */
+	detached = READ_ONCE(vma->detached);
+	/*
+	 * This pairs with smp_wmb() inside vma_mark_attached() to ensure
+	 * vma->detached is read before vma attributes read later inside
+	 * lock_vma_under_rcu().
+	 */
+	smp_rmb();
+
+	return detached;
 }
 
 static inline void release_fault_lock(struct vm_fault *vmf)
@@ -859,7 +880,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 #else /* CONFIG_PER_VMA_LOCK */
 
 static inline void vma_lock_init(struct vm_area_struct *vma) {}
-static inline bool vma_start_read(struct vm_area_struct *vma)
+static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma)
 		{ return false; }
 static inline void vma_end_read(struct vm_area_struct *vma) {}
 static inline void vma_start_write(struct vm_area_struct *vma) {}
@@ -893,6 +914,7 @@ static inline void assert_fault_locked(struct vm_fault *vmf)
 
 extern const struct vm_operations_struct vma_dummy_vm_ops;
 
+/* Use on VMAs not created using vm_area_alloc() */
 static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
 {
 	memset(vma, 0, sizeof(*vma));
include/linux/mm_types.h +7 -3
index be3551654325..5d8779997266 100644
@@ -543,6 +543,12 @@ static inline void *folio_get_private(struct folio *folio)
 
 typedef unsigned long vm_flags_t;
 
+/*
+ * freeptr_t represents a SLUB freelist pointer, which might be encoded
+ * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled.
+ */
+typedef struct { unsigned long v; } freeptr_t;
+
 /*
  * A region containing a mapping of a non-memory backed file under NOMMU
  * conditions.  These are held in a global tree and are pinned by the VMAs that
@@ -657,9 +663,7 @@ struct vm_area_struct {
 			unsigned long vm_start;
 			unsigned long vm_end;
 		};
-#ifdef CONFIG_PER_VMA_LOCK
-		struct rcu_head vm_rcu;	/* Used for deferred freeing. */
-#endif
+		freeptr_t vm_freeptr; /* Pointer used by SLAB_TYPESAFE_BY_RCU */
 	};
 
 	/*
include/linux/slab.h +0 -6
index 10a971c2bde3..681b685b6c4e 100644
@@ -234,12 +234,6 @@ enum _slab_flag_bits {
 #define SLAB_NO_OBJ_EXT		__SLAB_FLAG_UNUSED
 #endif
 
-/*
- * freeptr_t represents a SLUB freelist pointer, which might be encoded
- * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled.
- */
-typedef struct { unsigned long v; } freeptr_t;
-
 /*
  * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests.
  *
kernel/fork.c +127 -30

204 lines not on this page · raw message

mm/memory.c +11 -4
index b252f19b28c9..6f4d4d423835 100644
@@ -6368,10 +6368,16 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 	if (!vma)
 		goto inval;
 
-	if (!vma_start_read(vma))
+	if (!vma_start_read(mm, vma))
 		goto inval;
 
-	/* Check if the VMA got isolated after we found it */
+	/*
+	 * Check if the VMA got isolated after we found it.
+	 * Note: vma we found could have been recycled and is being reattached.
+	 * It's possible to attach a vma while it is read-locked, however a
+	 * read-locked vma can't be detached (detaching requires write-locking).
+	 * Therefore if this check passes, we have an attached and stable vma.
+	 */
 	if (is_vma_detached(vma)) {
 		vma_end_read(vma);
 		count_vm_vma_lock_event(VMA_LOCK_MISS);
@@ -6385,8 +6391,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 	 * fields are accessible for RCU readers.
 	 */
 
-	/* Check since vm_start/vm_end might change before we lock the VMA */
-	if (unlikely(address < vma->vm_start || address >= vma->vm_end))
+	/* Check if the vma we locked is the right one. */
+	if (unlikely(vma->vm_mm != mm ||
+		     address < vma->vm_start || address >= vma->vm_end))
 		goto inval_end_read;
 
 	rcu_read_unlock();
mm/vma.c +1 -1
index cdc63728f47f..648784416833 100644
@@ -414,7 +414,7 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable)
 		fput(vma->vm_file);
 	mpol_put(vma_policy(vma));
 	if (unreachable)
-		__vm_area_free(vma);
+		vm_area_free_unreachable(vma);
 	else
 		vm_area_free(vma);
 }
tools/testing/vma/vma_internal.h +4 -3
index 0cdc5f8c3d60..3eeb1317cc69 100644
@@ -685,14 +685,15 @@ static inline void mpol_put(struct mempolicy *)
 {
 }
 
-static inline void __vm_area_free(struct vm_area_struct *vma)
+static inline void vm_area_free(struct vm_area_struct *vma)
 {
 	free(vma);
 }
 
-static inline void vm_area_free(struct vm_area_struct *vma)
+static inline void vm_area_free_unreachable(struct vm_area_struct *vma)
 {
-	__vm_area_free(vma);
+	vma->detached = true;
+	vm_area_free(vma);
 }
 
 static inline void lru_add_drain(void)
Signature · 1 line
2.47.0.338.g60cca15819-goog
#12 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Hi,

Quoted 27 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check

Quoted 8 lines

vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

This patch (85ad413389aec04cfaaba043caa8128b76c6e491 in next-20241209) seems to cause an oops on a MIPS board of mine (Cavium Octeon III) (abbreviated, full attached):

CPU 2 Unable to handle kernel paging request at virtual address 0000000000000000, epc == ffffffff813a85a0, ra == ffffffff81390438 Oops[#1]:

CPU: 2 UID: 0 PID: 1 Comm: init Not tainted 6.13.0-rc1-00162-g85ad413389ae #156
Call Trace:

unlink_anon_vmas (mm/rmap.c:408)
free_pgtables (mm/memory.c:393)
vms_clear_ptes (mm/vma.c:1143) vms_complete_munmap_vmas (include/linux/mm.h:2737 mm/vma.c:1187) do_vmi_align_munmap (mm/vma.c:1452)
__vm_munmap (mm/vma.c:2892)
sys_munmap (mm/mmap.c:1053)
syscall_common (arch/mips/kernel/scall64-n64.S:62)

I saw that there's already a report, but maybe another arch can be useful for tracking this down.

Please let me know if there's anything else you need.

Regards,
Klara Modin

Quoted 428 lines
---
  include/linux/mm.h               |  36 +++++--
  include/linux/mm_types.h         |  10 +-
  include/linux/slab.h             |   6 --
  kernel/fork.c                    | 157 +++++++++++++++++++++++++------
  mm/memory.c                      |  15 ++-
  mm/vma.c                         |   2 +-
  tools/testing/vma/vma_internal.h |   7 +-
  7 files changed, 179 insertions(+), 54 deletions(-)
include/linux/mm.h +29 -7
index 2bf38c1e9cca..3568bcbc7c81 100644
@@ -257,7 +257,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *);
  struct vm_area_struct *vm_area_dup(struct vm_area_struct *);
  void vm_area_free(struct vm_area_struct *);
  /* Use only if VMA has no other users */
-void __vm_area_free(struct vm_area_struct *vma);
+void vm_area_free_unreachable(struct vm_area_struct *vma);
 
  #ifndef CONFIG_MMU
  extern struct rb_root nommu_region_tree;
@@ -706,8 +706,10 @@ static inline void vma_lock_init(struct vm_area_struct *vma)
   * Try to read-lock a vma. The function is allowed to occasionally yield false
   * locked result to avoid performance overhead, in which case we fall back to
   * using mmap_lock. The function should never yield false unlocked result.
+ * False locked result is possible if mm_lock_seq overflows or if vma gets
+ * reused and attached to a different mm before we lock it.
   */
-static inline bool vma_start_read(struct vm_area_struct *vma)
+static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma)
  {
  	/*
  	 * Check before locking. A race might cause false locked result.
@@ -716,7 +718,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
  	 * we don't rely on for anything - the mm_lock_seq read against which we
  	 * need ordering is below.
  	 */
-	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence))
+	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
  		return false;
 
  	if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0))
@@ -733,7 +735,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
  	 * after it has been unlocked.
  	 * This pairs with RELEASE semantics in vma_end_write_all().
  	 */
-	if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) {
+	if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) {
  		up_read(&vma->vm_lock.lock);
  		return false;
  	}
@@ -822,7 +824,15 @@ static inline void vma_assert_locked(struct vm_area_struct *vma)
 
  static inline void vma_mark_attached(struct vm_area_struct *vma)
  {
-	vma->detached = false;
+	/*
+	 * This pairs with smp_rmb() inside is_vma_detached().
+	 * vma is marked attached after all vma modifications are done and it
+	 * got added into the vma tree. All prior vma modifications should be
+	 * made visible before marking the vma attached.
+	 */
+	smp_wmb();
+	/* This pairs with READ_ONCE() in is_vma_detached(). */
+	WRITE_ONCE(vma->detached, false);
  }
 
  static inline void vma_mark_detached(struct vm_area_struct *vma)
@@ -834,7 +844,18 @@ static inline void vma_mark_detached(struct vm_area_struct *vma)
 
  static inline bool is_vma_detached(struct vm_area_struct *vma)
  {
-	return vma->detached;
+	bool detached;
+
+	/* This pairs with WRITE_ONCE() in vma_mark_attached(). */
+	detached = READ_ONCE(vma->detached);
+	/*
+	 * This pairs with smp_wmb() inside vma_mark_attached() to ensure
+	 * vma->detached is read before vma attributes read later inside
+	 * lock_vma_under_rcu().
+	 */
+	smp_rmb();
+
+	return detached;
  }
 
  static inline void release_fault_lock(struct vm_fault *vmf)
@@ -859,7 +880,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
  #else /* CONFIG_PER_VMA_LOCK */
 
  static inline void vma_lock_init(struct vm_area_struct *vma) {}
-static inline bool vma_start_read(struct vm_area_struct *vma)
+static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma)
  		{ return false; }
  static inline void vma_end_read(struct vm_area_struct *vma) {}
  static inline void vma_start_write(struct vm_area_struct *vma) {}
@@ -893,6 +914,7 @@ static inline void assert_fault_locked(struct vm_fault *vmf)
 
  extern const struct vm_operations_struct vma_dummy_vm_ops;
 
+/* Use on VMAs not created using vm_area_alloc() */
  static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
  {
  	memset(vma, 0, sizeof(*vma));
include/linux/mm_types.h +7 -3
index be3551654325..5d8779997266 100644
@@ -543,6 +543,12 @@ static inline void *folio_get_private(struct folio *folio)
 
  typedef unsigned long vm_flags_t;
 
+/*
+ * freeptr_t represents a SLUB freelist pointer, which might be encoded
+ * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled.
+ */
+typedef struct { unsigned long v; } freeptr_t;
+
  /*
   * A region containing a mapping of a non-memory backed file under NOMMU
   * conditions.  These are held in a global tree and are pinned by the VMAs that
@@ -657,9 +663,7 @@ struct vm_area_struct {
  			unsigned long vm_start;
  			unsigned long vm_end;
  		};
-#ifdef CONFIG_PER_VMA_LOCK
-		struct rcu_head vm_rcu;	/* Used for deferred freeing. */
-#endif
+		freeptr_t vm_freeptr; /* Pointer used by SLAB_TYPESAFE_BY_RCU */
  	};
 
  	/*
include/linux/slab.h +0 -6
index 10a971c2bde3..681b685b6c4e 100644
@@ -234,12 +234,6 @@ enum _slab_flag_bits {
  #define SLAB_NO_OBJ_EXT		__SLAB_FLAG_UNUSED
  #endif
 
-/*
- * freeptr_t represents a SLUB freelist pointer, which might be encoded
- * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled.
- */
-typedef struct { unsigned long v; } freeptr_t;
-
  /*
   * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests.
   *
kernel/fork.c +127 -30
index 71990f46aa4e..e7e76a660e4c 100644
@@ -436,6 +436,98 @@ static struct kmem_cache *vm_area_cachep;
  /* SLAB cache for mm_struct structures (tsk->mm) */
  static struct kmem_cache *mm_cachep;
 
+static void vm_area_ctor(void *data)
+{
+	struct vm_area_struct *vma = (struct vm_area_struct *)data;
+
+#ifdef CONFIG_PER_VMA_LOCK
+	/* vma is not locked, can't use vma_mark_detached() */
+	vma->detached = true;
+#endif
+	INIT_LIST_HEAD(&vma->anon_vma_chain);
+	vma_lock_init(vma);
+}
+
+#ifdef CONFIG_PER_VMA_LOCK
+
+static void vma_clear(struct vm_area_struct *vma, struct mm_struct *mm)
+{
+	vma->vm_mm = mm;
+	vma->vm_ops = &vma_dummy_vm_ops;
+	vma->vm_start = 0;
+	vma->vm_end = 0;
+	vma->anon_vma = NULL;
+	vma->vm_pgoff = 0;
+	vma->vm_file = NULL;
+	vma->vm_private_data = NULL;
+	vm_flags_init(vma, 0);
+	memset(&vma->vm_page_prot, 0, sizeof(vma->vm_page_prot));
+	memset(&vma->shared, 0, sizeof(vma->shared));
+	memset(&vma->vm_userfaultfd_ctx, 0, sizeof(vma->vm_userfaultfd_ctx));
+	vma_numab_state_init(vma);
+#ifdef CONFIG_ANON_VMA_NAME
+	vma->anon_name = NULL;
+#endif
+#ifdef CONFIG_SWAP
+	memset(&vma->swap_readahead_info, 0, sizeof(vma->swap_readahead_info));
+#endif
+#ifndef CONFIG_MMU
+	vma->vm_region = NULL;
+#endif
+#ifdef CONFIG_NUMA
+	vma->vm_policy = NULL;
+#endif
+}
+
+static void vma_copy(const struct vm_area_struct *src, struct vm_area_struct *dest)
+{
+	dest->vm_mm = src->vm_mm;
+	dest->vm_ops = src->vm_ops;
+	dest->vm_start = src->vm_start;
+	dest->vm_end = src->vm_end;
+	dest->anon_vma = src->anon_vma;
+	dest->vm_pgoff = src->vm_pgoff;
+	dest->vm_file = src->vm_file;
+	dest->vm_private_data = src->vm_private_data;
+	vm_flags_init(dest, src->vm_flags);
+	memcpy(&dest->vm_page_prot, &src->vm_page_prot,
+	       sizeof(dest->vm_page_prot));
+	memcpy(&dest->shared, &src->shared, sizeof(dest->shared));
+	memcpy(&dest->vm_userfaultfd_ctx, &src->vm_userfaultfd_ctx,
+	       sizeof(dest->vm_userfaultfd_ctx));
+#ifdef CONFIG_ANON_VMA_NAME
+	dest->anon_name = src->anon_name;
+#endif
+#ifdef CONFIG_SWAP
+	memcpy(&dest->swap_readahead_info, &src->swap_readahead_info,
+	       sizeof(dest->swap_readahead_info));
+#endif
+#ifndef CONFIG_MMU
+	dest->vm_region = src->vm_region;
+#endif
+#ifdef CONFIG_NUMA
+	dest->vm_policy = src->vm_policy;
+#endif
+}
+
+#else /* CONFIG_PER_VMA_LOCK */
+
+static void vma_clear(struct vm_area_struct *vma, struct mm_struct *mm)
+{
+	vma_init(vma, mm);
+}
+
+static void vma_copy(const struct vm_area_struct *src, struct vm_area_struct *dest)
+{
+	/*
+	 * orig->shared.rb may be modified concurrently, but the clone
+	 * will be reinitialized.
+	 */
+	data_race(memcpy(dest, src, sizeof(*dest)));
+}
+
+#endif /* CONFIG_PER_VMA_LOCK */
+
  struct vm_area_struct *vm_area_alloc(struct mm_struct *mm)
  {
  	struct vm_area_struct *vma;
@@ -444,7 +536,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm)
  	if (!vma)
  		return NULL;
 
-	vma_init(vma, mm);
+	vma_clear(vma, mm);
 
  	return vma;
  }
@@ -458,49 +550,46 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 
  	ASSERT_EXCLUSIVE_WRITER(orig->vm_flags);
  	ASSERT_EXCLUSIVE_WRITER(orig->vm_file);
-	/*
-	 * orig->shared.rb may be modified concurrently, but the clone
-	 * will be reinitialized.
-	 */
-	data_race(memcpy(new, orig, sizeof(*new)));
-	vma_lock_init(new);
-	INIT_LIST_HEAD(&new->anon_vma_chain);
-#ifdef CONFIG_PER_VMA_LOCK
-	/* vma is not locked, can't use vma_mark_detached() */
-	new->detached = true;
-#endif
+	vma_copy(orig, new);
  	vma_numab_state_init(new);
  	dup_anon_vma_name(orig, new);
 
  	return new;
  }
 
-void __vm_area_free(struct vm_area_struct *vma)
+static void __vm_area_free(struct vm_area_struct *vma, bool unreachable)
  {
+#ifdef CONFIG_PER_VMA_LOCK
+	/*
+	 * With SLAB_TYPESAFE_BY_RCU, vma can be reused and we need
+	 * vma->detached to be set before vma is returned into the cache.
+	 * This way reused object won't be used by readers until it's
+	 * initialized and reattached.
+	 * If vma is unreachable, there can be no other users and we
+	 * can set vma->detached directly with no risk of a race.
+	 * If vma is reachable, then it should have been already detached
+	 * under vma write-lock or it was never attached.
+	 */
+	if (unreachable)
+		vma->detached = true;
+	else
+		VM_BUG_ON_VMA(!is_vma_detached(vma), vma);
+	vma->vm_lock_seq = UINT_MAX;
+#endif
+	VM_BUG_ON_VMA(!list_empty(&vma->anon_vma_chain), vma);
  	vma_numab_state_free(vma);
  	free_anon_vma_name(vma);
  	kmem_cache_free(vm_area_cachep, vma);
  }
 
-#ifdef CONFIG_PER_VMA_LOCK
-static void vm_area_free_rcu_cb(struct rcu_head *head)
+void vm_area_free(struct vm_area_struct *vma)
  {
-	struct vm_area_struct *vma = container_of(head, struct vm_area_struct,
-						  vm_rcu);
-
-	/* The vma should not be locked while being destroyed. */
-	VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma);
-	__vm_area_free(vma);
+	__vm_area_free(vma, false);
  }
-#endif
 
-void vm_area_free(struct vm_area_struct *vma)
+void vm_area_free_unreachable(struct vm_area_struct *vma)
  {
-#ifdef CONFIG_PER_VMA_LOCK
-	call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb);
-#else
-	__vm_area_free(vma);
-#endif
+	__vm_area_free(vma, true);
  }
 
  static void account_kernel_stack(struct task_struct *tsk, int account)
@@ -3141,6 +3230,12 @@ void __init mm_cache_init(void)
 
  void __init proc_caches_init(void)
  {
+	struct kmem_cache_args args = {
+		.use_freeptr_offset = true,
+		.freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr),
+		.ctor = vm_area_ctor,
+	};
+
  	sighand_cachep = kmem_cache_create("sighand_cache",
  			sizeof(struct sighand_struct), 0,
  			SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU|
@@ -3157,9 +3252,11 @@ void __init proc_caches_init(void)
  			sizeof(struct fs_struct), 0,
  			SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
  			NULL);
-	vm_area_cachep = KMEM_CACHE(vm_area_struct,
-			SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC|
+	vm_area_cachep = kmem_cache_create("vm_area_struct",
+			sizeof(struct vm_area_struct), &args,
+			SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU|
  			SLAB_ACCOUNT);
+
  	mmap_init();
  	nsproxy_cache_init();
  }
mm/memory.c +11 -4
index b252f19b28c9..6f4d4d423835 100644
@@ -6368,10 +6368,16 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
  	if (!vma)
  		goto inval;
 
-	if (!vma_start_read(vma))
+	if (!vma_start_read(mm, vma))
  		goto inval;
 
-	/* Check if the VMA got isolated after we found it */
+	/*
+	 * Check if the VMA got isolated after we found it.
+	 * Note: vma we found could have been recycled and is being reattached.
+	 * It's possible to attach a vma while it is read-locked, however a
+	 * read-locked vma can't be detached (detaching requires write-locking).
+	 * Therefore if this check passes, we have an attached and stable vma.
+	 */
  	if (is_vma_detached(vma)) {
  		vma_end_read(vma);
  		count_vm_vma_lock_event(VMA_LOCK_MISS);
@@ -6385,8 +6391,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
  	 * fields are accessible for RCU readers.
  	 */
 
-	/* Check since vm_start/vm_end might change before we lock the VMA */
-	if (unlikely(address < vma->vm_start || address >= vma->vm_end))
+	/* Check if the vma we locked is the right one. */
+	if (unlikely(vma->vm_mm != mm ||
+		     address < vma->vm_start || address >= vma->vm_end))
  		goto inval_end_read;
 
  	rcu_read_unlock();
mm/vma.c +1 -1
index cdc63728f47f..648784416833 100644
@@ -414,7 +414,7 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable)
  		fput(vma->vm_file);
  	mpol_put(vma_policy(vma));
  	if (unreachable)
-		__vm_area_free(vma);
+		vm_area_free_unreachable(vma);
  	else
  		vm_area_free(vma);
  }
tools/testing/vma/vma_internal.h +1 -1
index 0cdc5f8c3d60..3eeb1317cc69 100644
@@ -685,14 +685,15 @@ static inline void mpol_put(struct mempolicy *)
  {
  }
 
-static inline void __vm_area_free(struct vm_area_struct *vma)
+static inline void vm_area_free(struct vm_area_struct *vma)
  {
  	free(vma);
  }

tools/testing/vma/vma_internal.h static inline void mpol_put(struct mempolicy *) · line 699

Quoted 10 lines
tools/testing/vma/vma_internal.h +3 -2
 
-static inline void vm_area_free(struct vm_area_struct *vma)
+static inline void vm_area_free_unreachable(struct vm_area_struct *vma)
  {
-	__vm_area_free(vma);
+	vma->detached = true;
+	vm_area_free(vma);
  }
 
  static inline void lru_add_drain(void)

# bad: [7e0a79eb1674cbe8916bbc7d7f9f6bfb409dfb4c] commit 0790303ec869 leads to cpu stall without CONFIG_FANOTIFY_ACCESS_PERMISSIONS=y git bisect start 'HEAD'
# status: waiting for good commit(s), bad commit known
# good: [fac04efc5c793dccbd07e2d59af9f90b7fc0dca4] Linux 6.13-rc2
git bisect good fac04efc5c793dccbd07e2d59af9f90b7fc0dca4
# bad: [06036a53ccaf22767bbd5caa491c52bb673604e6] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git
git bisect bad 06036a53ccaf22767bbd5caa491c52bb673604e6
# bad: [ddaabea9ec4322b7d5cdd2017c4cc59b28adb8fb] Merge branch 'linux-next' of git://github.com/c-sky/csky-linux.git
git bisect bad ddaabea9ec4322b7d5cdd2017c4cc59b28adb8fb
# good: [5de9573848aa512bb06e8842faf5da6926e693bd] Merge branch 'tip/urgent' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
git bisect good 5de9573848aa512bb06e8842faf5da6926e693bd
# bad: [8bb435707c07ff7650fe8ef690700845c772d3cf] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/amlogic/linux.git
git bisect bad 8bb435707c07ff7650fe8ef690700845c772d3cf
# good: [7e274b34309bb7884cade55b7cb0b0b8722dcde2] mm: do_zap_pte_range: return any_skipped information to the caller
git bisect good 7e274b34309bb7884cade55b7cb0b0b8722dcde2
# skip: [6e165f54437931f329d09dca6c19d99af08a36e1] mm/page_isolation: fixup isolate_single_pageblock() comment regarding splitting free pages git bisect skip 6e165f54437931f329d09dca6c19d99af08a36e1
# good: [c81cad99b0b13b0bb339a8e30aa13ebc1e5dbff3] minmax.h: update some comments
git bisect good c81cad99b0b13b0bb339a8e30aa13ebc1e5dbff3
# good: [191812969871e9ae447cf3e7426fb72b351f1336] scripts/spelling.txt: add more spellings to spelling.txt
git bisect good 191812969871e9ae447cf3e7426fb72b351f1336
# good: [f635cc53910d4ba7726f925e49ce57c5332cd6a1] ucounts: move kfree() out of critical zone protected by ucounts_lock
git bisect good f635cc53910d4ba7726f925e49ce57c5332cd6a1
# skip: [4f3ce240fd4bd7681335231b8cf21fdc37c4e848] mm/page_alloc: conditionally split > pageblock_order pages in free_one_page() and move_freepages_block_isolate() git bisect skip 4f3ce240fd4bd7681335231b8cf21fdc37c4e848
# bad: [53b2a428e1036e481af0c7fc39cb92128344b689] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
git bisect bad 53b2a428e1036e481af0c7fc39cb92128344b689
# good: [1062a24d0a17862738df0e6202d445ba0a260088] mm: introduce vma_start_read_locked{_nested} helpers
git bisect good 1062a24d0a17862738df0e6202d445ba0a260088
# bad: [c6fcd83658200c74c837c4a2e3f95f7eb487dd7d] mseal: remove can_do_mseal()
git bisect bad c6fcd83658200c74c837c4a2e3f95f7eb487dd7d
# bad: [2684a4244bfeac0c246a2610997dad78b2dc7877] mm/slab: allow freeptr_offset to be used with ctor
git bisect bad 2684a4244bfeac0c246a2610997dad78b2dc7877
# good: [98d5eefb975e46c182fc2de6a543e375481aedaf] mm: mark vma as detached until it's added into vma tree
git bisect good 98d5eefb975e46c182fc2de6a543e375481aedaf
# bad: [85ad413389aec04cfaaba043caa8128b76c6e491] mm: make vma cache SLAB_TYPESAFE_BY_RCU
git bisect bad 85ad413389aec04cfaaba043caa8128b76c6e491

# first bad commit: [85ad413389aec04cfaaba043caa8128b76c6e491] mm: make vma cache SLAB_TYPESAFE_BY_RCU
author maintainer
#13 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Quoted 40 lines · Klara Modin

Hi,

Quoted 27 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check

Quoted 8 lines

vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 3 lines

This patch (85ad413389aec04cfaaba043caa8128b76c6e491 in next-20241209) seems to cause an oops on a MIPS board of mine (Cavium Octeon III) (abbreviated, full attached):

Thanks! This is really helpful because both errors are reported on architectures which don't set CONFIG_PER_VMA_LOCK. This is a great clue.
I'm investigating it and will post a fix asap.
Thanks,
Suren.

Quoted 464 lines

CPU 2 Unable to handle kernel paging request at virtual address 0000000000000000, epc == ffffffff813a85a0, ra == ffffffff81390438 Oops[#1]:

CPU: 2 UID: 0 PID: 1 Comm: init Not tainted
6.13.0-rc1-00162-g85ad413389ae #156
Call Trace:

unlink_anon_vmas (mm/rmap.c:408)
free_pgtables (mm/memory.c:393)
vms_clear_ptes (mm/vma.c:1143) vms_complete_munmap_vmas (include/linux/mm.h:2737 mm/vma.c:1187) do_vmi_align_munmap (mm/vma.c:1452)
__vm_munmap (mm/vma.c:2892)
sys_munmap (mm/mmap.c:1053)
syscall_common (arch/mips/kernel/scall64-n64.S:62)

I saw that there's already a report, but maybe another arch can be useful for tracking this down.

Please let me know if there's anything else you need.

Regards,
Klara Modin

Quoted 438 lines
---
  include/linux/mm.h               |  36 +++++--
  include/linux/mm_types.h         |  10 +-
  include/linux/slab.h             |   6 --
  kernel/fork.c                    | 157 +++++++++++++++++++++++++------
  mm/memory.c                      |  15 ++-
  mm/vma.c                         |   2 +-
  tools/testing/vma/vma_internal.h |   7 +-
  7 files changed, 179 insertions(+), 54 deletions(-)
include/linux/mm.h +29 -7
index 2bf38c1e9cca..3568bcbc7c81 100644
@@ -257,7 +257,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *);
  struct vm_area_struct *vm_area_dup(struct vm_area_struct *);
  void vm_area_free(struct vm_area_struct *);
  /* Use only if VMA has no other users */
-void __vm_area_free(struct vm_area_struct *vma);
+void vm_area_free_unreachable(struct vm_area_struct *vma);
 
  #ifndef CONFIG_MMU
  extern struct rb_root nommu_region_tree;
@@ -706,8 +706,10 @@ static inline void vma_lock_init(struct vm_area_struct *vma)
   * Try to read-lock a vma. The function is allowed to occasionally yield false
   * locked result to avoid performance overhead, in which case we fall back to
   * using mmap_lock. The function should never yield false unlocked result.
+ * False locked result is possible if mm_lock_seq overflows or if vma gets
+ * reused and attached to a different mm before we lock it.
   */
-static inline bool vma_start_read(struct vm_area_struct *vma)
+static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma)
  {
      /*
       * Check before locking. A race might cause false locked result.
@@ -716,7 +718,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
       * we don't rely on for anything - the mm_lock_seq read against which we
       * need ordering is below.
       */
-     if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence))
+     if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
              return false;
 
      if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0))
@@ -733,7 +735,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
       * after it has been unlocked.
       * This pairs with RELEASE semantics in vma_end_write_all().
       */
-     if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) {
+     if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) {
              up_read(&vma->vm_lock.lock);
              return false;
      }
@@ -822,7 +824,15 @@ static inline void vma_assert_locked(struct vm_area_struct *vma)
 
  static inline void vma_mark_attached(struct vm_area_struct *vma)
  {
-     vma->detached = false;
+     /*
+      * This pairs with smp_rmb() inside is_vma_detached().
+      * vma is marked attached after all vma modifications are done and it
+      * got added into the vma tree. All prior vma modifications should be
+      * made visible before marking the vma attached.
+      */
+     smp_wmb();
+     /* This pairs with READ_ONCE() in is_vma_detached(). */
+     WRITE_ONCE(vma->detached, false);
  }
 
  static inline void vma_mark_detached(struct vm_area_struct *vma)
@@ -834,7 +844,18 @@ static inline void vma_mark_detached(struct vm_area_struct *vma)
 
  static inline bool is_vma_detached(struct vm_area_struct *vma)
  {
-     return vma->detached;
+     bool detached;
+
+     /* This pairs with WRITE_ONCE() in vma_mark_attached(). */
+     detached = READ_ONCE(vma->detached);
+     /*
+      * This pairs with smp_wmb() inside vma_mark_attached() to ensure
+      * vma->detached is read before vma attributes read later inside
+      * lock_vma_under_rcu().
+      */
+     smp_rmb();
+
+     return detached;
  }
 
  static inline void release_fault_lock(struct vm_fault *vmf)
@@ -859,7 +880,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
  #else /* CONFIG_PER_VMA_LOCK */
 
  static inline void vma_lock_init(struct vm_area_struct *vma) {}
-static inline bool vma_start_read(struct vm_area_struct *vma)
+static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma)
              { return false; }
  static inline void vma_end_read(struct vm_area_struct *vma) {}
  static inline void vma_start_write(struct vm_area_struct *vma) {}
@@ -893,6 +914,7 @@ static inline void assert_fault_locked(struct vm_fault *vmf)
 
  extern const struct vm_operations_struct vma_dummy_vm_ops;
 
+/* Use on VMAs not created using vm_area_alloc() */
  static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
  {
      memset(vma, 0, sizeof(*vma));
include/linux/mm_types.h +7 -3
index be3551654325..5d8779997266 100644
@@ -543,6 +543,12 @@ static inline void *folio_get_private(struct folio *folio)
 
  typedef unsigned long vm_flags_t;
 
+/*
+ * freeptr_t represents a SLUB freelist pointer, which might be encoded
+ * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled.
+ */
+typedef struct { unsigned long v; } freeptr_t;
+
  /*
   * A region containing a mapping of a non-memory backed file under NOMMU
   * conditions.  These are held in a global tree and are pinned by the VMAs that
@@ -657,9 +663,7 @@ struct vm_area_struct {
                      unsigned long vm_start;
                      unsigned long vm_end;
              };
-#ifdef CONFIG_PER_VMA_LOCK
-             struct rcu_head vm_rcu; /* Used for deferred freeing. */
-#endif
+             freeptr_t vm_freeptr; /* Pointer used by SLAB_TYPESAFE_BY_RCU */
      };
 
      /*
include/linux/slab.h +0 -6
index 10a971c2bde3..681b685b6c4e 100644
@@ -234,12 +234,6 @@ enum _slab_flag_bits {
  #define SLAB_NO_OBJ_EXT             __SLAB_FLAG_UNUSED
  #endif
 
-/*
- * freeptr_t represents a SLUB freelist pointer, which might be encoded
- * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled.
- */
-typedef struct { unsigned long v; } freeptr_t;
-
  /*
   * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests.
   *
kernel/fork.c +127 -30
index 71990f46aa4e..e7e76a660e4c 100644
@@ -436,6 +436,98 @@ static struct kmem_cache *vm_area_cachep;
  /* SLAB cache for mm_struct structures (tsk->mm) */
  static struct kmem_cache *mm_cachep;
 
+static void vm_area_ctor(void *data)
+{
+     struct vm_area_struct *vma = (struct vm_area_struct *)data;
+
+#ifdef CONFIG_PER_VMA_LOCK
+     /* vma is not locked, can't use vma_mark_detached() */
+     vma->detached = true;
+#endif
+     INIT_LIST_HEAD(&vma->anon_vma_chain);
+     vma_lock_init(vma);
+}
+
+#ifdef CONFIG_PER_VMA_LOCK
+
+static void vma_clear(struct vm_area_struct *vma, struct mm_struct *mm)
+{
+     vma->vm_mm = mm;
+     vma->vm_ops = &vma_dummy_vm_ops;
+     vma->vm_start = 0;
+     vma->vm_end = 0;
+     vma->anon_vma = NULL;
+     vma->vm_pgoff = 0;
+     vma->vm_file = NULL;
+     vma->vm_private_data = NULL;
+     vm_flags_init(vma, 0);
+     memset(&vma->vm_page_prot, 0, sizeof(vma->vm_page_prot));
+     memset(&vma->shared, 0, sizeof(vma->shared));
+     memset(&vma->vm_userfaultfd_ctx, 0, sizeof(vma->vm_userfaultfd_ctx));
+     vma_numab_state_init(vma);
+#ifdef CONFIG_ANON_VMA_NAME
+     vma->anon_name = NULL;
+#endif
+#ifdef CONFIG_SWAP
+     memset(&vma->swap_readahead_info, 0, sizeof(vma->swap_readahead_info));
+#endif
+#ifndef CONFIG_MMU
+     vma->vm_region = NULL;
+#endif
+#ifdef CONFIG_NUMA
+     vma->vm_policy = NULL;
+#endif
+}
+
+static void vma_copy(const struct vm_area_struct *src, struct vm_area_struct *dest)
+{
+     dest->vm_mm = src->vm_mm;
+     dest->vm_ops = src->vm_ops;
+     dest->vm_start = src->vm_start;
+     dest->vm_end = src->vm_end;
+     dest->anon_vma = src->anon_vma;
+     dest->vm_pgoff = src->vm_pgoff;
+     dest->vm_file = src->vm_file;
+     dest->vm_private_data = src->vm_private_data;
+     vm_flags_init(dest, src->vm_flags);
+     memcpy(&dest->vm_page_prot, &src->vm_page_prot,
+            sizeof(dest->vm_page_prot));
+     memcpy(&dest->shared, &src->shared, sizeof(dest->shared));
+     memcpy(&dest->vm_userfaultfd_ctx, &src->vm_userfaultfd_ctx,
+            sizeof(dest->vm_userfaultfd_ctx));
+#ifdef CONFIG_ANON_VMA_NAME
+     dest->anon_name = src->anon_name;
+#endif
+#ifdef CONFIG_SWAP
+     memcpy(&dest->swap_readahead_info, &src->swap_readahead_info,
+            sizeof(dest->swap_readahead_info));
+#endif
+#ifndef CONFIG_MMU
+     dest->vm_region = src->vm_region;
+#endif
+#ifdef CONFIG_NUMA
+     dest->vm_policy = src->vm_policy;
+#endif
+}
+
+#else /* CONFIG_PER_VMA_LOCK */
+
+static void vma_clear(struct vm_area_struct *vma, struct mm_struct *mm)
+{
+     vma_init(vma, mm);
+}
+
+static void vma_copy(const struct vm_area_struct *src, struct vm_area_struct *dest)
+{
+     /*
+      * orig->shared.rb may be modified concurrently, but the clone
+      * will be reinitialized.
+      */
+     data_race(memcpy(dest, src, sizeof(*dest)));
+}
+
+#endif /* CONFIG_PER_VMA_LOCK */
+
  struct vm_area_struct *vm_area_alloc(struct mm_struct *mm)
  {
      struct vm_area_struct *vma;
@@ -444,7 +536,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm)
      if (!vma)
              return NULL;
 
-     vma_init(vma, mm);
+     vma_clear(vma, mm);
 
      return vma;
  }
@@ -458,49 +550,46 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 
      ASSERT_EXCLUSIVE_WRITER(orig->vm_flags);
      ASSERT_EXCLUSIVE_WRITER(orig->vm_file);
-     /*
-      * orig->shared.rb may be modified concurrently, but the clone
-      * will be reinitialized.
-      */
-     data_race(memcpy(new, orig, sizeof(*new)));
-     vma_lock_init(new);
-     INIT_LIST_HEAD(&new->anon_vma_chain);
-#ifdef CONFIG_PER_VMA_LOCK
-     /* vma is not locked, can't use vma_mark_detached() */
-     new->detached = true;
-#endif
+     vma_copy(orig, new);
      vma_numab_state_init(new);
      dup_anon_vma_name(orig, new);
 
      return new;
  }
 
-void __vm_area_free(struct vm_area_struct *vma)
+static void __vm_area_free(struct vm_area_struct *vma, bool unreachable)
  {
+#ifdef CONFIG_PER_VMA_LOCK
+     /*
+      * With SLAB_TYPESAFE_BY_RCU, vma can be reused and we need
+      * vma->detached to be set before vma is returned into the cache.
+      * This way reused object won't be used by readers until it's
+      * initialized and reattached.
+      * If vma is unreachable, there can be no other users and we
+      * can set vma->detached directly with no risk of a race.
+      * If vma is reachable, then it should have been already detached
+      * under vma write-lock or it was never attached.
+      */
+     if (unreachable)
+             vma->detached = true;
+     else
+             VM_BUG_ON_VMA(!is_vma_detached(vma), vma);
+     vma->vm_lock_seq = UINT_MAX;
+#endif
+     VM_BUG_ON_VMA(!list_empty(&vma->anon_vma_chain), vma);
      vma_numab_state_free(vma);
      free_anon_vma_name(vma);
      kmem_cache_free(vm_area_cachep, vma);
  }
 
-#ifdef CONFIG_PER_VMA_LOCK
-static void vm_area_free_rcu_cb(struct rcu_head *head)
+void vm_area_free(struct vm_area_struct *vma)
  {
-     struct vm_area_struct *vma = container_of(head, struct vm_area_struct,
-                                               vm_rcu);
-
-     /* The vma should not be locked while being destroyed. */
-     VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma);
-     __vm_area_free(vma);
+     __vm_area_free(vma, false);
  }
-#endif
 
-void vm_area_free(struct vm_area_struct *vma)
+void vm_area_free_unreachable(struct vm_area_struct *vma)
  {
-#ifdef CONFIG_PER_VMA_LOCK
-     call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb);
-#else
-     __vm_area_free(vma);
-#endif
+     __vm_area_free(vma, true);
  }
 
  static void account_kernel_stack(struct task_struct *tsk, int account)
@@ -3141,6 +3230,12 @@ void __init mm_cache_init(void)
 
  void __init proc_caches_init(void)
  {
+     struct kmem_cache_args args = {
+             .use_freeptr_offset = true,
+             .freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr),
+             .ctor = vm_area_ctor,
+     };
+
      sighand_cachep = kmem_cache_create("sighand_cache",
                      sizeof(struct sighand_struct), 0,
                      SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU|
@@ -3157,9 +3252,11 @@ void __init proc_caches_init(void)
                      sizeof(struct fs_struct), 0,
                      SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
                      NULL);
-     vm_area_cachep = KMEM_CACHE(vm_area_struct,
-                     SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC|
+     vm_area_cachep = kmem_cache_create("vm_area_struct",
+                     sizeof(struct vm_area_struct), &args,
+                     SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU|
                      SLAB_ACCOUNT);
+
      mmap_init();
      nsproxy_cache_init();
  }
mm/memory.c +11 -4
index b252f19b28c9..6f4d4d423835 100644
@@ -6368,10 +6368,16 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
      if (!vma)
              goto inval;
 
-     if (!vma_start_read(vma))
+     if (!vma_start_read(mm, vma))
              goto inval;
 
-     /* Check if the VMA got isolated after we found it */
+     /*
+      * Check if the VMA got isolated after we found it.
+      * Note: vma we found could have been recycled and is being reattached.
+      * It's possible to attach a vma while it is read-locked, however a
+      * read-locked vma can't be detached (detaching requires write-locking).
+      * Therefore if this check passes, we have an attached and stable vma.
+      */
      if (is_vma_detached(vma)) {
              vma_end_read(vma);
              count_vm_vma_lock_event(VMA_LOCK_MISS);
@@ -6385,8 +6391,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
       * fields are accessible for RCU readers.
       */
 
-     /* Check since vm_start/vm_end might change before we lock the VMA */
-     if (unlikely(address < vma->vm_start || address >= vma->vm_end))
+     /* Check if the vma we locked is the right one. */
+     if (unlikely(vma->vm_mm != mm ||
+                  address < vma->vm_start || address >= vma->vm_end))
              goto inval_end_read;
 
      rcu_read_unlock();
mm/vma.c +1 -1
index cdc63728f47f..648784416833 100644
@@ -414,7 +414,7 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable)
              fput(vma->vm_file);
      mpol_put(vma_policy(vma));
      if (unreachable)
-             __vm_area_free(vma);
+             vm_area_free_unreachable(vma);
      else
              vm_area_free(vma);
  }
tools/testing/vma/vma_internal.h +4 -3
index 0cdc5f8c3d60..3eeb1317cc69 100644
@@ -685,14 +685,15 @@ static inline void mpol_put(struct mempolicy *)
  {
  }
 
-static inline void __vm_area_free(struct vm_area_struct *vma)
+static inline void vm_area_free(struct vm_area_struct *vma)
  {
      free(vma);
  }
 
-static inline void vm_area_free(struct vm_area_struct *vma)
+static inline void vm_area_free_unreachable(struct vm_area_struct *vma)
  {
-     __vm_area_free(vma);
+     vma->detached = true;
+     vm_area_free(vma);
  }
 
  static inline void lru_add_drain(void)
author maintainer
#14 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Quoted 42 lines · Suren Baghdasaryan
Quoted 40 lines · Klara Modin

Hi,

Quoted 35 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 8 lines
Quoted 3 lines

This patch (85ad413389aec04cfaaba043caa8128b76c6e491 in next-20241209) seems to cause an oops on a MIPS board of mine (Cavium Octeon III) (abbreviated, full attached):

Thanks! This is really helpful because both errors are reported on architectures which don't set CONFIG_PER_VMA_LOCK. This is a great clue.
I'm investigating it and will post a fix asap.

I think https://lore.kernel.org/all/[email protected]/
should fix this.

Quoted 467 lines

Thanks,
Suren.

Quoted 464 lines

CPU 2 Unable to handle kernel paging request at virtual address 0000000000000000, epc == ffffffff813a85a0, ra == ffffffff81390438 Oops[#1]:

CPU: 2 UID: 0 PID: 1 Comm: init Not tainted
6.13.0-rc1-00162-g85ad413389ae #156
Call Trace:

unlink_anon_vmas (mm/rmap.c:408)
free_pgtables (mm/memory.c:393)
vms_clear_ptes (mm/vma.c:1143) vms_complete_munmap_vmas (include/linux/mm.h:2737 mm/vma.c:1187) do_vmi_align_munmap (mm/vma.c:1452)
__vm_munmap (mm/vma.c:2892)
sys_munmap (mm/mmap.c:1053)
syscall_common (arch/mips/kernel/scall64-n64.S:62)

I saw that there's already a report, but maybe another arch can be useful for tracking this down.

Please let me know if there's anything else you need.

Regards,
Klara Modin

Quoted 438 lines
---
  include/linux/mm.h               |  36 +++++--
  include/linux/mm_types.h         |  10 +-
  include/linux/slab.h             |   6 --
  kernel/fork.c                    | 157 +++++++++++++++++++++++++------
  mm/memory.c                      |  15 ++-
  mm/vma.c                         |   2 +-
  tools/testing/vma/vma_internal.h |   7 +-
  7 files changed, 179 insertions(+), 54 deletions(-)
include/linux/mm.h +29 -7
index 2bf38c1e9cca..3568bcbc7c81 100644
@@ -257,7 +257,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *);
  struct vm_area_struct *vm_area_dup(struct vm_area_struct *);
  void vm_area_free(struct vm_area_struct *);
  /* Use only if VMA has no other users */
-void __vm_area_free(struct vm_area_struct *vma);
+void vm_area_free_unreachable(struct vm_area_struct *vma);
 
  #ifndef CONFIG_MMU
  extern struct rb_root nommu_region_tree;
@@ -706,8 +706,10 @@ static inline void vma_lock_init(struct vm_area_struct *vma)
   * Try to read-lock a vma. The function is allowed to occasionally yield false
   * locked result to avoid performance overhead, in which case we fall back to
   * using mmap_lock. The function should never yield false unlocked result.
+ * False locked result is possible if mm_lock_seq overflows or if vma gets
+ * reused and attached to a different mm before we lock it.
   */
-static inline bool vma_start_read(struct vm_area_struct *vma)
+static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma)
  {
      /*
       * Check before locking. A race might cause false locked result.
@@ -716,7 +718,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
       * we don't rely on for anything - the mm_lock_seq read against which we
       * need ordering is below.
       */
-     if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence))
+     if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
              return false;
 
      if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0))
@@ -733,7 +735,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
       * after it has been unlocked.
       * This pairs with RELEASE semantics in vma_end_write_all().
       */
-     if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) {
+     if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) {
              up_read(&vma->vm_lock.lock);
              return false;
      }
@@ -822,7 +824,15 @@ static inline void vma_assert_locked(struct vm_area_struct *vma)
 
  static inline void vma_mark_attached(struct vm_area_struct *vma)
  {
-     vma->detached = false;
+     /*
+      * This pairs with smp_rmb() inside is_vma_detached().
+      * vma is marked attached after all vma modifications are done and it
+      * got added into the vma tree. All prior vma modifications should be
+      * made visible before marking the vma attached.
+      */
+     smp_wmb();
+     /* This pairs with READ_ONCE() in is_vma_detached(). */
+     WRITE_ONCE(vma->detached, false);
  }
 
  static inline void vma_mark_detached(struct vm_area_struct *vma)
@@ -834,7 +844,18 @@ static inline void vma_mark_detached(struct vm_area_struct *vma)
 
  static inline bool is_vma_detached(struct vm_area_struct *vma)
  {
-     return vma->detached;
+     bool detached;
+
+     /* This pairs with WRITE_ONCE() in vma_mark_attached(). */
+     detached = READ_ONCE(vma->detached);
+     /*
+      * This pairs with smp_wmb() inside vma_mark_attached() to ensure
+      * vma->detached is read before vma attributes read later inside
+      * lock_vma_under_rcu().
+      */
+     smp_rmb();
+
+     return detached;
  }
 
  static inline void release_fault_lock(struct vm_fault *vmf)
@@ -859,7 +880,7 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
  #else /* CONFIG_PER_VMA_LOCK */
 
  static inline void vma_lock_init(struct vm_area_struct *vma) {}
-static inline bool vma_start_read(struct vm_area_struct *vma)
+static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma)
              { return false; }
  static inline void vma_end_read(struct vm_area_struct *vma) {}
  static inline void vma_start_write(struct vm_area_struct *vma) {}
@@ -893,6 +914,7 @@ static inline void assert_fault_locked(struct vm_fault *vmf)
 
  extern const struct vm_operations_struct vma_dummy_vm_ops;
 
+/* Use on VMAs not created using vm_area_alloc() */
  static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm)
  {
      memset(vma, 0, sizeof(*vma));
include/linux/mm_types.h +7 -3
index be3551654325..5d8779997266 100644
@@ -543,6 +543,12 @@ static inline void *folio_get_private(struct folio *folio)
 
  typedef unsigned long vm_flags_t;
 
+/*
+ * freeptr_t represents a SLUB freelist pointer, which might be encoded
+ * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled.
+ */
+typedef struct { unsigned long v; } freeptr_t;
+
  /*
   * A region containing a mapping of a non-memory backed file under NOMMU
   * conditions.  These are held in a global tree and are pinned by the VMAs that
@@ -657,9 +663,7 @@ struct vm_area_struct {
                      unsigned long vm_start;
                      unsigned long vm_end;
              };
-#ifdef CONFIG_PER_VMA_LOCK
-             struct rcu_head vm_rcu; /* Used for deferred freeing. */
-#endif
+             freeptr_t vm_freeptr; /* Pointer used by SLAB_TYPESAFE_BY_RCU */
      };
 
      /*
include/linux/slab.h +0 -6
index 10a971c2bde3..681b685b6c4e 100644
@@ -234,12 +234,6 @@ enum _slab_flag_bits {
  #define SLAB_NO_OBJ_EXT             __SLAB_FLAG_UNUSED
  #endif
 
-/*
- * freeptr_t represents a SLUB freelist pointer, which might be encoded
- * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled.
- */
-typedef struct { unsigned long v; } freeptr_t;
-
  /*
   * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests.
   *
kernel/fork.c +127 -30
index 71990f46aa4e..e7e76a660e4c 100644
@@ -436,6 +436,98 @@ static struct kmem_cache *vm_area_cachep;
  /* SLAB cache for mm_struct structures (tsk->mm) */
  static struct kmem_cache *mm_cachep;
 
+static void vm_area_ctor(void *data)
+{
+     struct vm_area_struct *vma = (struct vm_area_struct *)data;
+
+#ifdef CONFIG_PER_VMA_LOCK
+     /* vma is not locked, can't use vma_mark_detached() */
+     vma->detached = true;
+#endif
+     INIT_LIST_HEAD(&vma->anon_vma_chain);
+     vma_lock_init(vma);
+}
+
+#ifdef CONFIG_PER_VMA_LOCK
+
+static void vma_clear(struct vm_area_struct *vma, struct mm_struct *mm)
+{
+     vma->vm_mm = mm;
+     vma->vm_ops = &vma_dummy_vm_ops;
+     vma->vm_start = 0;
+     vma->vm_end = 0;
+     vma->anon_vma = NULL;
+     vma->vm_pgoff = 0;
+     vma->vm_file = NULL;
+     vma->vm_private_data = NULL;
+     vm_flags_init(vma, 0);
+     memset(&vma->vm_page_prot, 0, sizeof(vma->vm_page_prot));
+     memset(&vma->shared, 0, sizeof(vma->shared));
+     memset(&vma->vm_userfaultfd_ctx, 0, sizeof(vma->vm_userfaultfd_ctx));
+     vma_numab_state_init(vma);
+#ifdef CONFIG_ANON_VMA_NAME
+     vma->anon_name = NULL;
+#endif
+#ifdef CONFIG_SWAP
+     memset(&vma->swap_readahead_info, 0, sizeof(vma->swap_readahead_info));
+#endif
+#ifndef CONFIG_MMU
+     vma->vm_region = NULL;
+#endif
+#ifdef CONFIG_NUMA
+     vma->vm_policy = NULL;
+#endif
+}
+
+static void vma_copy(const struct vm_area_struct *src, struct vm_area_struct *dest)
+{
+     dest->vm_mm = src->vm_mm;
+     dest->vm_ops = src->vm_ops;
+     dest->vm_start = src->vm_start;
+     dest->vm_end = src->vm_end;
+     dest->anon_vma = src->anon_vma;
+     dest->vm_pgoff = src->vm_pgoff;
+     dest->vm_file = src->vm_file;
+     dest->vm_private_data = src->vm_private_data;
+     vm_flags_init(dest, src->vm_flags);
+     memcpy(&dest->vm_page_prot, &src->vm_page_prot,
+            sizeof(dest->vm_page_prot));
+     memcpy(&dest->shared, &src->shared, sizeof(dest->shared));
+     memcpy(&dest->vm_userfaultfd_ctx, &src->vm_userfaultfd_ctx,
+            sizeof(dest->vm_userfaultfd_ctx));
+#ifdef CONFIG_ANON_VMA_NAME
+     dest->anon_name = src->anon_name;
+#endif
+#ifdef CONFIG_SWAP
+     memcpy(&dest->swap_readahead_info, &src->swap_readahead_info,
+            sizeof(dest->swap_readahead_info));
+#endif
+#ifndef CONFIG_MMU
+     dest->vm_region = src->vm_region;
+#endif
+#ifdef CONFIG_NUMA
+     dest->vm_policy = src->vm_policy;
+#endif
+}
+
+#else /* CONFIG_PER_VMA_LOCK */
+
+static void vma_clear(struct vm_area_struct *vma, struct mm_struct *mm)
+{
+     vma_init(vma, mm);
+}
+
+static void vma_copy(const struct vm_area_struct *src, struct vm_area_struct *dest)
+{
+     /*
+      * orig->shared.rb may be modified concurrently, but the clone
+      * will be reinitialized.
+      */
+     data_race(memcpy(dest, src, sizeof(*dest)));
+}
+
+#endif /* CONFIG_PER_VMA_LOCK */
+
  struct vm_area_struct *vm_area_alloc(struct mm_struct *mm)
  {
      struct vm_area_struct *vma;
@@ -444,7 +536,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *mm)
      if (!vma)
              return NULL;
 
-     vma_init(vma, mm);
+     vma_clear(vma, mm);
 
      return vma;
  }
@@ -458,49 +550,46 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig)
 
      ASSERT_EXCLUSIVE_WRITER(orig->vm_flags);
      ASSERT_EXCLUSIVE_WRITER(orig->vm_file);
-     /*
-      * orig->shared.rb may be modified concurrently, but the clone
-      * will be reinitialized.
-      */
-     data_race(memcpy(new, orig, sizeof(*new)));
-     vma_lock_init(new);
-     INIT_LIST_HEAD(&new->anon_vma_chain);
-#ifdef CONFIG_PER_VMA_LOCK
-     /* vma is not locked, can't use vma_mark_detached() */
-     new->detached = true;
-#endif
+     vma_copy(orig, new);
      vma_numab_state_init(new);
      dup_anon_vma_name(orig, new);
 
      return new;
  }
 
-void __vm_area_free(struct vm_area_struct *vma)
+static void __vm_area_free(struct vm_area_struct *vma, bool unreachable)
  {
+#ifdef CONFIG_PER_VMA_LOCK
+     /*
+      * With SLAB_TYPESAFE_BY_RCU, vma can be reused and we need
+      * vma->detached to be set before vma is returned into the cache.
+      * This way reused object won't be used by readers until it's
+      * initialized and reattached.
+      * If vma is unreachable, there can be no other users and we
+      * can set vma->detached directly with no risk of a race.
+      * If vma is reachable, then it should have been already detached
+      * under vma write-lock or it was never attached.
+      */
+     if (unreachable)
+             vma->detached = true;
+     else
+             VM_BUG_ON_VMA(!is_vma_detached(vma), vma);
+     vma->vm_lock_seq = UINT_MAX;
+#endif
+     VM_BUG_ON_VMA(!list_empty(&vma->anon_vma_chain), vma);
      vma_numab_state_free(vma);
      free_anon_vma_name(vma);
      kmem_cache_free(vm_area_cachep, vma);
  }
 
-#ifdef CONFIG_PER_VMA_LOCK
-static void vm_area_free_rcu_cb(struct rcu_head *head)
+void vm_area_free(struct vm_area_struct *vma)
  {
-     struct vm_area_struct *vma = container_of(head, struct vm_area_struct,
-                                               vm_rcu);
-
-     /* The vma should not be locked while being destroyed. */
-     VM_BUG_ON_VMA(rwsem_is_locked(&vma->vm_lock.lock), vma);
-     __vm_area_free(vma);
+     __vm_area_free(vma, false);
  }
-#endif
 
-void vm_area_free(struct vm_area_struct *vma)
+void vm_area_free_unreachable(struct vm_area_struct *vma)
  {
-#ifdef CONFIG_PER_VMA_LOCK
-     call_rcu(&vma->vm_rcu, vm_area_free_rcu_cb);
-#else
-     __vm_area_free(vma);
-#endif
+     __vm_area_free(vma, true);
  }
 
  static void account_kernel_stack(struct task_struct *tsk, int account)
@@ -3141,6 +3230,12 @@ void __init mm_cache_init(void)
 
  void __init proc_caches_init(void)
  {
+     struct kmem_cache_args args = {
+             .use_freeptr_offset = true,
+             .freeptr_offset = offsetof(struct vm_area_struct, vm_freeptr),
+             .ctor = vm_area_ctor,
+     };
+
      sighand_cachep = kmem_cache_create("sighand_cache",
                      sizeof(struct sighand_struct), 0,
                      SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU|
@@ -3157,9 +3252,11 @@ void __init proc_caches_init(void)
                      sizeof(struct fs_struct), 0,
                      SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
                      NULL);
-     vm_area_cachep = KMEM_CACHE(vm_area_struct,
-                     SLAB_HWCACHE_ALIGN|SLAB_NO_MERGE|SLAB_PANIC|
+     vm_area_cachep = kmem_cache_create("vm_area_struct",
+                     sizeof(struct vm_area_struct), &args,
+                     SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_TYPESAFE_BY_RCU|
                      SLAB_ACCOUNT);
+
      mmap_init();
      nsproxy_cache_init();
  }
mm/memory.c +11 -4
index b252f19b28c9..6f4d4d423835 100644
@@ -6368,10 +6368,16 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
      if (!vma)
              goto inval;
 
-     if (!vma_start_read(vma))
+     if (!vma_start_read(mm, vma))
              goto inval;
 
-     /* Check if the VMA got isolated after we found it */
+     /*
+      * Check if the VMA got isolated after we found it.
+      * Note: vma we found could have been recycled and is being reattached.
+      * It's possible to attach a vma while it is read-locked, however a
+      * read-locked vma can't be detached (detaching requires write-locking).
+      * Therefore if this check passes, we have an attached and stable vma.
+      */
      if (is_vma_detached(vma)) {
              vma_end_read(vma);
              count_vm_vma_lock_event(VMA_LOCK_MISS);
@@ -6385,8 +6391,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
       * fields are accessible for RCU readers.
       */
 
-     /* Check since vm_start/vm_end might change before we lock the VMA */
-     if (unlikely(address < vma->vm_start || address >= vma->vm_end))
+     /* Check if the vma we locked is the right one. */
+     if (unlikely(vma->vm_mm != mm ||
+                  address < vma->vm_start || address >= vma->vm_end))
              goto inval_end_read;
 
      rcu_read_unlock();
mm/vma.c +1 -1
index cdc63728f47f..648784416833 100644
@@ -414,7 +414,7 @@ void remove_vma(struct vm_area_struct *vma, bool unreachable)
              fput(vma->vm_file);
      mpol_put(vma_policy(vma));
      if (unreachable)
-             __vm_area_free(vma);
+             vm_area_free_unreachable(vma);
      else
              vm_area_free(vma);
  }
tools/testing/vma/vma_internal.h +4 -3
index 0cdc5f8c3d60..3eeb1317cc69 100644
@@ -685,14 +685,15 @@ static inline void mpol_put(struct mempolicy *)
  {
  }
 
-static inline void __vm_area_free(struct vm_area_struct *vma)
+static inline void vm_area_free(struct vm_area_struct *vma)
  {
      free(vma);
  }
 
-static inline void vm_area_free(struct vm_area_struct *vma)
+static inline void vm_area_free_unreachable(struct vm_area_struct *vma)
  {
-     __vm_area_free(vma);
+     vma->detached = true;
+     vm_area_free(vma);
  }
 
  static inline void lru_add_drain(void)
#15 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

In reply to Suren Baghdasaryan
Quoted 82 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

---
 include/linux/mm.h               |  36 +++++--
 include/linux/mm_types.h         |  10 +-
 include/linux/slab.h             |   6 --
 kernel/fork.c                    | 157 +++++++++++++++++++++++++------
 mm/memory.c                      |  15 ++-
 mm/vma.c                         |   2 +-
 tools/testing/vma/vma_internal.h |   7 +-
 7 files changed, 179 insertions(+), 54 deletions(-)
include/linux/mm.h +5 -3
index 2bf38c1e9cca..3568bcbc7c81 100644
@@ -257,7 +257,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *);
 struct vm_area_struct *vm_area_dup(struct vm_area_struct *);
 void vm_area_free(struct vm_area_struct *);
 /* Use only if VMA has no other users */
-void __vm_area_free(struct vm_area_struct *vma);
+void vm_area_free_unreachable(struct vm_area_struct *vma);
 
 #ifndef CONFIG_MMU
 extern struct rb_root nommu_region_tree;
@@ -706,8 +706,10 @@ static inline void vma_lock_init(struct vm_area_struct *vma)
  * Try to read-lock a vma. The function is allowed to occasionally yield false
  * locked result to avoid performance overhead, in which case we fall back to
  * using mmap_lock. The function should never yield false unlocked result.
+ * False locked result is possible if mm_lock_seq overflows or if vma gets
+ * reused and attached to a different mm before we lock it.
  */
-static inline bool vma_start_read(struct vm_area_struct *vma)
+static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma)
 {
 	/*
 	 * Check before locking. A race might cause false locked result.
@@ -716,7 +718,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
 	 * we don't rely on for anything - the mm_lock_seq read against which we
 	 * need ordering is below.
 	 */
-	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence))
+	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
 		return false;
 
 	if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0))

include/linux/mm.h static inline bool vma_start_read(struct vm_area_struct *vma) · line 741

Quoted 8 lines
include/linux/mm.h +1 -1
@@ -733,7 +735,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
 	 * after it has been unlocked.
 	 * This pairs with RELEASE semantics in vma_end_write_all().
 	 */
-	if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) {
+	if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) {
 		up_read(&vma->vm_lock.lock);
 		return false;
 	}

This could have been perhaps another preparatory patch to make this one smaller?

Quoted 32 lines
include/linux/mm.h +31 -0
+static void vm_area_ctor(void *data)
+{
+	struct vm_area_struct *vma = (struct vm_area_struct *)data;
+
+#ifdef CONFIG_PER_VMA_LOCK
+	/* vma is not locked, can't use vma_mark_detached() */
+	vma->detached = true;
+#endif
+	INIT_LIST_HEAD(&vma->anon_vma_chain);
+	vma_lock_init(vma);
+}
+
+#ifdef CONFIG_PER_VMA_LOCK
+
+static void vma_clear(struct vm_area_struct *vma, struct mm_struct *mm)
+{
+	vma->vm_mm = mm;
+	vma->vm_ops = &vma_dummy_vm_ops;
+	vma->vm_start = 0;
+	vma->vm_end = 0;
+	vma->anon_vma = NULL;
+	vma->vm_pgoff = 0;
+	vma->vm_file = NULL;
+	vma->vm_private_data = NULL;
+	vm_flags_init(vma, 0);
+	memset(&vma->vm_page_prot, 0, sizeof(vma->vm_page_prot));
+	memset(&vma->shared, 0, sizeof(vma->shared));
+	memset(&vma->vm_userfaultfd_ctx, 0, sizeof(vma->vm_userfaultfd_ctx));
+	vma_numab_state_init(vma);
+#ifdef CONFIG_ANON_VMA_NAME
+	vma->anon_name = NULL;

include/linux/mm.h

Quoted 10 lines
include/linux/mm.h +10 -0
+#endif
+#ifdef CONFIG_SWAP
+	memset(&vma->swap_readahead_info, 0, sizeof(vma->swap_readahead_info));
+#endif
+#ifndef CONFIG_MMU
+	vma->vm_region = NULL;
+#endif
+#ifdef CONFIG_NUMA
+	vma->vm_policy = NULL;
+#endif

This isn't the ideal pattern I think, now that we have a ctor. Ideally the ctor would do all this (except setting the vm_mm), and then we need to make sure it's also done when freeing the vma, to make sure the freed object is in the same state as a new object after the constructor.

On freeing, things like numab_state and anon_name could be NULL'd (by the respective destructors) only when they are non-NULL and thus freeing the objects pointed to. vm_policy and vm_file could perhaps be handled same way after some refactoring (see remove_vma()), vma_dummy_vm_ops are possibly already reset by vma_close(), etc.

Quoted 34 lines
include/linux/mm.h +34 -0
+}
+
+static void vma_copy(const struct vm_area_struct *src, struct vm_area_struct *dest)
+{
+	dest->vm_mm = src->vm_mm;
+	dest->vm_ops = src->vm_ops;
+	dest->vm_start = src->vm_start;
+	dest->vm_end = src->vm_end;
+	dest->anon_vma = src->anon_vma;
+	dest->vm_pgoff = src->vm_pgoff;
+	dest->vm_file = src->vm_file;
+	dest->vm_private_data = src->vm_private_data;
+	vm_flags_init(dest, src->vm_flags);
+	memcpy(&dest->vm_page_prot, &src->vm_page_prot,
+	       sizeof(dest->vm_page_prot));
+	memcpy(&dest->shared, &src->shared, sizeof(dest->shared));
+	memcpy(&dest->vm_userfaultfd_ctx, &src->vm_userfaultfd_ctx,
+	       sizeof(dest->vm_userfaultfd_ctx));
+#ifdef CONFIG_ANON_VMA_NAME
+	dest->anon_name = src->anon_name;
+#endif
+#ifdef CONFIG_SWAP
+	memcpy(&dest->swap_readahead_info, &src->swap_readahead_info,
+	       sizeof(dest->swap_readahead_info));
+#endif
+#ifndef CONFIG_MMU
+	dest->vm_region = src->vm_region;
+#endif
+#ifdef CONFIG_NUMA
+	dest->vm_policy = src->vm_policy;
+#endif
+}
+
+#else /* CONFIG_PER_VMA_LOCK */
author maintainer
#16 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Quoted 143 lines · Vlastimil Babka
Quoted 82 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

---
 include/linux/mm.h               |  36 +++++--
 include/linux/mm_types.h         |  10 +-
 include/linux/slab.h             |   6 --
 kernel/fork.c                    | 157 +++++++++++++++++++++++++------
 mm/memory.c                      |  15 ++-
 mm/vma.c                         |   2 +-
 tools/testing/vma/vma_internal.h |   7 +-
 7 files changed, 179 insertions(+), 54 deletions(-)
include/linux/mm.h +5 -3
index 2bf38c1e9cca..3568bcbc7c81 100644
@@ -257,7 +257,7 @@ struct vm_area_struct *vm_area_alloc(struct mm_struct *);
 struct vm_area_struct *vm_area_dup(struct vm_area_struct *);
 void vm_area_free(struct vm_area_struct *);
 /* Use only if VMA has no other users */
-void __vm_area_free(struct vm_area_struct *vma);
+void vm_area_free_unreachable(struct vm_area_struct *vma);
 
 #ifndef CONFIG_MMU
 extern struct rb_root nommu_region_tree;
@@ -706,8 +706,10 @@ static inline void vma_lock_init(struct vm_area_struct *vma)
  * Try to read-lock a vma. The function is allowed to occasionally yield false
  * locked result to avoid performance overhead, in which case we fall back to
  * using mmap_lock. The function should never yield false unlocked result.
+ * False locked result is possible if mm_lock_seq overflows or if vma gets
+ * reused and attached to a different mm before we lock it.
  */
-static inline bool vma_start_read(struct vm_area_struct *vma)
+static inline bool vma_start_read(struct mm_struct *mm, struct vm_area_struct *vma)
 {
      /*
       * Check before locking. A race might cause false locked result.
@@ -716,7 +718,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
       * we don't rely on for anything - the mm_lock_seq read against which we
       * need ordering is below.
       */
-     if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(vma->vm_mm->mm_lock_seq.sequence))
+     if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
              return false;
 
      if (unlikely(down_read_trylock(&vma->vm_lock.lock) == 0))

include/linux/mm.h static inline bool vma_start_read(struct vm_area_struct *vma) · line 741

Quoted 8 lines
include/linux/mm.h +1 -1
@@ -733,7 +735,7 @@ static inline bool vma_start_read(struct vm_area_struct *vma)
       * after it has been unlocked.
       * This pairs with RELEASE semantics in vma_end_write_all().
       */
-     if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&vma->vm_mm->mm_lock_seq))) {
+     if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) {
              up_read(&vma->vm_lock.lock);
              return false;
      }

This could have been perhaps another preparatory patch to make this one smaller?

Quoted 32 lines
include/linux/mm.h +31 -0
+static void vm_area_ctor(void *data)
+{
+     struct vm_area_struct *vma = (struct vm_area_struct *)data;
+
+#ifdef CONFIG_PER_VMA_LOCK
+     /* vma is not locked, can't use vma_mark_detached() */
+     vma->detached = true;
+#endif
+     INIT_LIST_HEAD(&vma->anon_vma_chain);
+     vma_lock_init(vma);
+}
+
+#ifdef CONFIG_PER_VMA_LOCK
+
+static void vma_clear(struct vm_area_struct *vma, struct mm_struct *mm)
+{
+     vma->vm_mm = mm;
+     vma->vm_ops = &vma_dummy_vm_ops;
+     vma->vm_start = 0;
+     vma->vm_end = 0;
+     vma->anon_vma = NULL;
+     vma->vm_pgoff = 0;
+     vma->vm_file = NULL;
+     vma->vm_private_data = NULL;
+     vm_flags_init(vma, 0);
+     memset(&vma->vm_page_prot, 0, sizeof(vma->vm_page_prot));
+     memset(&vma->shared, 0, sizeof(vma->shared));
+     memset(&vma->vm_userfaultfd_ctx, 0, sizeof(vma->vm_userfaultfd_ctx));
+     vma_numab_state_init(vma);
+#ifdef CONFIG_ANON_VMA_NAME
+     vma->anon_name = NULL;

include/linux/mm.h

Quoted 10 lines
include/linux/mm.h +10 -0
+#endif
+#ifdef CONFIG_SWAP
+     memset(&vma->swap_readahead_info, 0, sizeof(vma->swap_readahead_info));
+#endif
+#ifndef CONFIG_MMU
+     vma->vm_region = NULL;
+#endif
+#ifdef CONFIG_NUMA
+     vma->vm_policy = NULL;
+#endif

This isn't the ideal pattern I think, now that we have a ctor. Ideally the ctor would do all this (except setting the vm_mm), and then we need to make sure it's also done when freeing the vma, to make sure the freed object is in the same state as a new object after the constructor.

Quoted 5 lines

On freeing, things like numab_state and anon_name could be NULL'd (by the respective destructors) only when they are non-NULL and thus freeing the objects pointed to. vm_policy and vm_file could perhaps be handled same way after some refactoring (see remove_vma()), vma_dummy_vm_ops are possibly already reset by vma_close(), etc.

Ok, let me look some more into it and see if I can improve and simplify the initialization/freeing logic. Thanks!

Quoted 35 lines
Quoted 35 lines
include/linux/mm.h +34 -0
+}
+
+static void vma_copy(const struct vm_area_struct *src, struct vm_area_struct *dest)
+{
+     dest->vm_mm = src->vm_mm;
+     dest->vm_ops = src->vm_ops;
+     dest->vm_start = src->vm_start;
+     dest->vm_end = src->vm_end;
+     dest->anon_vma = src->anon_vma;
+     dest->vm_pgoff = src->vm_pgoff;
+     dest->vm_file = src->vm_file;
+     dest->vm_private_data = src->vm_private_data;
+     vm_flags_init(dest, src->vm_flags);
+     memcpy(&dest->vm_page_prot, &src->vm_page_prot,
+            sizeof(dest->vm_page_prot));
+     memcpy(&dest->shared, &src->shared, sizeof(dest->shared));
+     memcpy(&dest->vm_userfaultfd_ctx, &src->vm_userfaultfd_ctx,
+            sizeof(dest->vm_userfaultfd_ctx));
+#ifdef CONFIG_ANON_VMA_NAME
+     dest->anon_name = src->anon_name;
+#endif
+#ifdef CONFIG_SWAP
+     memcpy(&dest->swap_readahead_info, &src->swap_readahead_info,
+            sizeof(dest->swap_readahead_info));
+#endif
+#ifndef CONFIG_MMU
+     dest->vm_region = src->vm_region;
+#endif
+#ifdef CONFIG_NUMA
+     dest->vm_policy = src->vm_policy;
+#endif
+}
+
+#else /* CONFIG_PER_VMA_LOCK */
#17 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

In reply to Suren Baghdasaryan
Quoted 36 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 1 line

I'm wondering about the vma freeing path. Consider vma_complete():

vma_mark_detached(vp->remove);
vma->detached = true; - plain write
vm_area_free(vp->remove);
vma->vm_lock_seq = UINT_MAX; - plain write
kmem_cache_free(vm_area_cachep)
...
potential reallocation

against:

lock_vma_under_rcu()
- mas_walk finds a stale vma due to race
vma_start_read()
if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
- can be false, the vma was not being locked on the freeing side? down_read_trylock(&vma->vm_lock.lock) - suceeds, wasn't locked this is acquire, but was there any release?
is_vma_detached() - false negative as the write above didn't propagate
here yet; a read barrier but where is the write barrier?
checks for vma->vm_mm, vm_start, vm_end - nobody reset them yet so false
positive, or they got reset on reallocation but writes didn't propagate

Am I missing something that would prevent lock_vma_under_rcu() falsely succeeding here?

author maintainer
#18 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Quoted 52 lines · Vlastimil Babka
Quoted 36 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 1 line

I'm wondering about the vma freeing path. Consider vma_complete():

vma_mark_detached(vp->remove);
vma->detached = true; - plain write
vm_area_free(vp->remove);
vma->vm_lock_seq = UINT_MAX; - plain write
kmem_cache_free(vm_area_cachep)
...
potential reallocation

against:

Quoted 7 lines

lock_vma_under_rcu()
- mas_walk finds a stale vma due to race
vma_start_read()
if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
- can be false, the vma was not being locked on the freeing side? down_read_trylock(&vma->vm_lock.lock) - suceeds, wasn't locked this is acquire, but was there any release?

Yes, there was a release. I think what you missed is that vma_mark_detached() that is called from vma_complete() requires VMA to be write-locked (see vma_assert_write_locked() in vma_mark_detached()). The rule is that a VMA can be attached without write-locking but only a write-locked VMA can be detached. So, after vma_mark_detached() and before down_read_trylock(&vma->vm_lock.lock) in vma_start_read() the VMA write-lock should have been released by mmap_write_unlock() and therefore vma->detached=false should be visible to the reader when it executed lock_vma_under_rcu().

Quoted 8 lines

is_vma_detached() - false negative as the write above didn't propagate
here yet; a read barrier but where is the write barrier?
checks for vma->vm_mm, vm_start, vm_end - nobody reset them yet so false
positive, or they got reset on reallocation but writes didn't propagate

Am I missing something that would prevent lock_vma_under_rcu() falsely succeeding here?

#19 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Quoted 61 lines · Suren Baghdasaryan
Quoted 52 lines · Vlastimil Babka
Quoted 36 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 1 line

I'm wondering about the vma freeing path. Consider vma_complete():

vma_mark_detached(vp->remove);
vma->detached = true; - plain write
vm_area_free(vp->remove);
vma->vm_lock_seq = UINT_MAX; - plain write
kmem_cache_free(vm_area_cachep)
...
potential reallocation

against:

Quoted 7 lines

lock_vma_under_rcu()
- mas_walk finds a stale vma due to race
vma_start_read()
if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
- can be false, the vma was not being locked on the freeing side? down_read_trylock(&vma->vm_lock.lock) - suceeds, wasn't locked this is acquire, but was there any release?

Quoted 5 lines

Yes, there was a release. I think what you missed is that vma_mark_detached() that is called from vma_complete() requires VMA to be write-locked (see vma_assert_write_locked() in vma_mark_detached()). The rule is that a VMA can be attached without write-locking but only a write-locked VMA can be detached. So, after

OK but write unlocking means the mm's seqcount is bumped and becomes non-equal with vma's vma->vm_lock_seq, right?

Yet in the example above we happily set it to UINT_MAX and thus effectively false unlock it for vma_start_read()?

And this is all done before the vma_complete() side would actually reach mmap_write_unlock(), AFAICS.

Quoted 13 lines

vma_mark_detached() and before down_read_trylock(&vma->vm_lock.lock) in vma_start_read() the VMA write-lock should have been released by mmap_write_unlock() and therefore vma->detached=false should be visible to the reader when it executed lock_vma_under_rcu().

Quoted 8 lines

is_vma_detached() - false negative as the write above didn't propagate
here yet; a read barrier but where is the write barrier?
checks for vma->vm_mm, vm_start, vm_end - nobody reset them yet so false
positive, or they got reset on reallocation but writes didn't propagate

Am I missing something that would prevent lock_vma_under_rcu() falsely succeeding here?

author maintainer
#20 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Quoted 69 lines · Vlastimil Babka
Quoted 61 lines · Suren Baghdasaryan
Quoted 52 lines · Vlastimil Babka
Quoted 36 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 1 line

I'm wondering about the vma freeing path. Consider vma_complete():

vma_mark_detached(vp->remove);
vma->detached = true; - plain write
vm_area_free(vp->remove);
vma->vm_lock_seq = UINT_MAX; - plain write
kmem_cache_free(vm_area_cachep)
...
potential reallocation

against:

Quoted 7 lines

lock_vma_under_rcu()
- mas_walk finds a stale vma due to race
vma_start_read()
if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
- can be false, the vma was not being locked on the freeing side? down_read_trylock(&vma->vm_lock.lock) - suceeds, wasn't locked this is acquire, but was there any release?

Quoted 5 lines

Yes, there was a release. I think what you missed is that vma_mark_detached() that is called from vma_complete() requires VMA to be write-locked (see vma_assert_write_locked() in vma_mark_detached()). The rule is that a VMA can be attached without write-locking but only a write-locked VMA can be detached. So, after

Quoted 8 lines

OK but write unlocking means the mm's seqcount is bumped and becomes non-equal with vma's vma->vm_lock_seq, right?

Yet in the example above we happily set it to UINT_MAX and thus effectively false unlock it for vma_start_read()?

And this is all done before the vma_complete() side would actually reach mmap_write_unlock(), AFAICS.

Ah, you are right. With the possibility of reuse, even a freed VMA should be kept write-locked until it is unlocked by mmap_write_unlock(). I think the fix for this is simply to not reset vma->vm_lock_seq inside vm_area_free(). I'll also need to add a comment for vm_lock_seq explaining these requirements.
Do you agree that such a change would resolve the issue?

Quoted 15 lines
Quoted 14 lines

vma_mark_detached() and before down_read_trylock(&vma->vm_lock.lock) in vma_start_read() the VMA write-lock should have been released by mmap_write_unlock() and therefore vma->detached=false should be visible to the reader when it executed lock_vma_under_rcu().

Quoted 8 lines

is_vma_detached() - false negative as the write above didn't propagate
here yet; a read barrier but where is the write barrier?
checks for vma->vm_mm, vm_start, vm_end - nobody reset them yet so false
positive, or they got reset on reallocation but writes didn't propagate

Am I missing something that would prevent lock_vma_under_rcu() falsely succeeding here?

#21 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Quoted 76 lines · Suren Baghdasaryan
Quoted 75 lines · Vlastimil Babka
Quoted 61 lines · Suren Baghdasaryan
Quoted 52 lines · Vlastimil Babka
Quoted 36 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 1 line

I'm wondering about the vma freeing path. Consider vma_complete():

vma_mark_detached(vp->remove);
vma->detached = true; - plain write
vm_area_free(vp->remove);
vma->vm_lock_seq = UINT_MAX; - plain write
kmem_cache_free(vm_area_cachep)
...
potential reallocation

against:

Quoted 7 lines

lock_vma_under_rcu()
- mas_walk finds a stale vma due to race
vma_start_read()
if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
- can be false, the vma was not being locked on the freeing side? down_read_trylock(&vma->vm_lock.lock) - suceeds, wasn't locked this is acquire, but was there any release?

Quoted 5 lines

Yes, there was a release. I think what you missed is that vma_mark_detached() that is called from vma_complete() requires VMA to be write-locked (see vma_assert_write_locked() in vma_mark_detached()). The rule is that a VMA can be attached without write-locking but only a write-locked VMA can be detached. So, after

OK but write unlocking means the mm's seqcount is bumped and becomes non-equal with vma's vma->vm_lock_seq, right?

Yet in the example above we happily set it to UINT_MAX and thus effectively false unlock it for vma_start_read()?

Quoted 7 lines
Quoted 2 lines

And this is all done before the vma_complete() side would actually reach mmap_write_unlock(), AFAICS.

Ah, you are right. With the possibility of reuse, even a freed VMA should be kept write-locked until it is unlocked by mmap_write_unlock(). I think the fix for this is simply to not reset vma->vm_lock_seq inside vm_area_free(). I'll also need to add a

But even if we don't reset vm_lock_seq to UINT_MAX, then whover reallocated it can proceed and end up doing a vma_start_write() and rewrite it there anyway, no?

Quoted 18 lines

comment for vm_lock_seq explaining these requirements. Do you agree that such a change would resolve the issue?

Quoted 15 lines
Quoted 14 lines

vma_mark_detached() and before down_read_trylock(&vma->vm_lock.lock) in vma_start_read() the VMA write-lock should have been released by mmap_write_unlock() and therefore vma->detached=false should be visible to the reader when it executed lock_vma_under_rcu().

Quoted 8 lines

is_vma_detached() - false negative as the write above didn't propagate
here yet; a read barrier but where is the write barrier?
checks for vma->vm_mm, vm_start, vm_end - nobody reset them yet so false
positive, or they got reset on reallocation but writes didn't propagate

Am I missing something that would prevent lock_vma_under_rcu() falsely succeeding here?

author maintainer
#22 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Quoted 81 lines · Vlastimil Babka
Quoted 79 lines · Suren Baghdasaryan
Quoted 77 lines · Vlastimil Babka
Quoted 61 lines · Suren Baghdasaryan
Quoted 52 lines · Vlastimil Babka
Quoted 36 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 1 line

I'm wondering about the vma freeing path. Consider vma_complete():

vma_mark_detached(vp->remove);
vma->detached = true; - plain write
vm_area_free(vp->remove);
vma->vm_lock_seq = UINT_MAX; - plain write
kmem_cache_free(vm_area_cachep)
...
potential reallocation

against:

Quoted 7 lines

lock_vma_under_rcu()
- mas_walk finds a stale vma due to race
vma_start_read()
if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
- can be false, the vma was not being locked on the freeing side? down_read_trylock(&vma->vm_lock.lock) - suceeds, wasn't locked this is acquire, but was there any release?

Quoted 5 lines

Yes, there was a release. I think what you missed is that vma_mark_detached() that is called from vma_complete() requires VMA to be write-locked (see vma_assert_write_locked() in vma_mark_detached()). The rule is that a VMA can be attached without write-locking but only a write-locked VMA can be detached. So, after

OK but write unlocking means the mm's seqcount is bumped and becomes non-equal with vma's vma->vm_lock_seq, right?

Yet in the example above we happily set it to UINT_MAX and thus effectively false unlock it for vma_start_read()?

And this is all done before the vma_complete() side would actually reach mmap_write_unlock(), AFAICS.

Quoted 8 lines
Quoted 4 lines

Ah, you are right. With the possibility of reuse, even a freed VMA should be kept write-locked until it is unlocked by mmap_write_unlock(). I think the fix for this is simply to not reset vma->vm_lock_seq inside vm_area_free(). I'll also need to add a

But even if we don't reset vm_lock_seq to UINT_MAX, then whover reallocated it can proceed and end up doing a vma_start_write() and rewrite it there anyway, no?

Ugh, yes. It looks like we will need a write memory barrier in vma_mark_detached() to make it immediately visible.

Case 1:
lock_vma_under_rcu()
vma_complete()

    vma_mark_detached(vp->remove);
        vma->detached = true;
        smp_wmb();
    vm_area_free(vp->remove);
        vma->vm_lock_seq = UINT_MAX;

                                       vma_start_read()
                                          is_vma_detached() <<- abort

       kmem_cache_free(vm_area_cachep);

mmap_write_unlock()

Case 2:
lock_vma_under_rcu()
vma_complete()

    vma_mark_detached(vp->remove);
        vma->detached = true;
        smp_wmb();
    vm_area_free(vp->remove);
        vma->vm_lock_seq = UINT_MAX;

                                       vma_start_read()

        kmem_cache_free(vm_area_cachep);
mmap_write_unlock() // changes mm->mm_lock_seq but does not matter
                    // reader holds vma->vm_lock, so new writers have to wait
    ...
    vm_area_alloc();
    // sets all vma attributes
    vma_mark_attached();
        smp_wmb();

                                           // if is_vma_detached() called
                                           //  before this point, we will
                                           // abort like in Case 1

        vma->detached = true;
                                           is_vma_detached()
                                           // check vm_mm, vm_start, vm_end
                                           // if all checks pass, this is a
                                           // new attached vma and it's
                                           // read-locked (can't be modified)

Did I miss any other race?

Quoted 20 lines
Quoted 19 lines

comment for vm_lock_seq explaining these requirements. Do you agree that such a change would resolve the issue?

Quoted 15 lines
Quoted 14 lines

vma_mark_detached() and before down_read_trylock(&vma->vm_lock.lock) in vma_start_read() the VMA write-lock should have been released by mmap_write_unlock() and therefore vma->detached=false should be visible to the reader when it executed lock_vma_under_rcu().

Quoted 8 lines

is_vma_detached() - false negative as the write above didn't propagate
here yet; a read barrier but where is the write barrier?
checks for vma->vm_mm, vm_start, vm_end - nobody reset them yet so false
positive, or they got reset on reallocation but writes didn't propagate

Am I missing something that would prevent lock_vma_under_rcu() falsely succeeding here?

author maintainer
#23 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

In reply to Vlastimil Babka
Quoted 81 lines · Vlastimil Babka
Quoted 79 lines · Suren Baghdasaryan
Quoted 77 lines · Vlastimil Babka
Quoted 61 lines · Suren Baghdasaryan
Quoted 52 lines · Vlastimil Babka
Quoted 36 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 1 line

I'm wondering about the vma freeing path. Consider vma_complete():

vma_mark_detached(vp->remove);
vma->detached = true; - plain write
vm_area_free(vp->remove);
vma->vm_lock_seq = UINT_MAX; - plain write
kmem_cache_free(vm_area_cachep)
...
potential reallocation

against:

Quoted 7 lines

lock_vma_under_rcu()
- mas_walk finds a stale vma due to race
vma_start_read()
if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
- can be false, the vma was not being locked on the freeing side? down_read_trylock(&vma->vm_lock.lock) - suceeds, wasn't locked this is acquire, but was there any release?

Quoted 5 lines

Yes, there was a release. I think what you missed is that vma_mark_detached() that is called from vma_complete() requires VMA to be write-locked (see vma_assert_write_locked() in vma_mark_detached()). The rule is that a VMA can be attached without write-locking but only a write-locked VMA can be detached. So, after

OK but write unlocking means the mm's seqcount is bumped and becomes non-equal with vma's vma->vm_lock_seq, right?

Yet in the example above we happily set it to UINT_MAX and thus effectively false unlock it for vma_start_read()?

And this is all done before the vma_complete() side would actually reach mmap_write_unlock(), AFAICS.

Quoted 8 lines
Quoted 4 lines

Ah, you are right. With the possibility of reuse, even a freed VMA should be kept write-locked until it is unlocked by mmap_write_unlock(). I think the fix for this is simply to not reset vma->vm_lock_seq inside vm_area_free(). I'll also need to add a

But even if we don't reset vm_lock_seq to UINT_MAX, then whover reallocated it can proceed and end up doing a vma_start_write() and rewrite it there anyway, no?

Actually, I think with a small change we can simplify these locking rules:

static inline void vma_start_write(struct vm_area_struct *vma)

{
        int mm_lock_seq;

- if (__is_vma_write_locked(vma, &mm_lock_seq))
- return;
+ mmap_assert_write_locked(vma->vm_mm);
+ mm_lock_seq = vma->vm_mm->mm_lock_seq;

        down_write(&vma->vm_lock->lock);
        /*
        * We should use WRITE_ONCE() here because we can have concurrent reads
        * from the early lockless pessimistic check in vma_start_read().
        * We don't really care about the correctness of that early check, but
        * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy.
        */
        WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq);
        up_write(&vma->vm_lock->lock);
}

This will force vma_start_write() to always write-lock vma->vm_lock before changing vma->vm_lock_seq. Since vma->vm_lock survives reuse, the other readers/writers will synchronize on it even if vma got reused.

Quoted 20 lines
Quoted 19 lines

comment for vm_lock_seq explaining these requirements. Do you agree that such a change would resolve the issue?

Quoted 15 lines
Quoted 14 lines

vma_mark_detached() and before down_read_trylock(&vma->vm_lock.lock) in vma_start_read() the VMA write-lock should have been released by mmap_write_unlock() and therefore vma->detached=false should be visible to the reader when it executed lock_vma_under_rcu().

Quoted 8 lines

is_vma_detached() - false negative as the write above didn't propagate
here yet; a read barrier but where is the write barrier?
checks for vma->vm_mm, vm_start, vm_end - nobody reset them yet so false
positive, or they got reset on reallocation but writes didn't propagate

Am I missing something that would prevent lock_vma_under_rcu() falsely succeeding here?

author maintainer
#24 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Quoted 114 lines · Suren Baghdasaryan
Quoted 81 lines · Vlastimil Babka
Quoted 79 lines · Suren Baghdasaryan
Quoted 77 lines · Vlastimil Babka
Quoted 61 lines · Suren Baghdasaryan
Quoted 52 lines · Vlastimil Babka
Quoted 36 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 1 line

I'm wondering about the vma freeing path. Consider vma_complete():

vma_mark_detached(vp->remove);
vma->detached = true; - plain write
vm_area_free(vp->remove);
vma->vm_lock_seq = UINT_MAX; - plain write
kmem_cache_free(vm_area_cachep)
...
potential reallocation

against:

Quoted 7 lines

lock_vma_under_rcu()
- mas_walk finds a stale vma due to race
vma_start_read()
if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
- can be false, the vma was not being locked on the freeing side? down_read_trylock(&vma->vm_lock.lock) - suceeds, wasn't locked this is acquire, but was there any release?

Quoted 5 lines

Yes, there was a release. I think what you missed is that vma_mark_detached() that is called from vma_complete() requires VMA to be write-locked (see vma_assert_write_locked() in vma_mark_detached()). The rule is that a VMA can be attached without write-locking but only a write-locked VMA can be detached. So, after

OK but write unlocking means the mm's seqcount is bumped and becomes non-equal with vma's vma->vm_lock_seq, right?

Yet in the example above we happily set it to UINT_MAX and thus effectively false unlock it for vma_start_read()?

And this is all done before the vma_complete() side would actually reach mmap_write_unlock(), AFAICS.

Quoted 8 lines
Quoted 4 lines

Ah, you are right. With the possibility of reuse, even a freed VMA should be kept write-locked until it is unlocked by mmap_write_unlock(). I think the fix for this is simply to not reset vma->vm_lock_seq inside vm_area_free(). I'll also need to add a

But even if we don't reset vm_lock_seq to UINT_MAX, then whover reallocated it can proceed and end up doing a vma_start_write() and rewrite it there anyway, no?

Actually, I think with a small change we can simplify these locking rules:

static inline void vma_start_write(struct vm_area_struct *vma)

{
        int mm_lock_seq;
(quoted diff) +2 -2
-        if (__is_vma_write_locked(vma, &mm_lock_seq))
-                return;
+        mmap_assert_write_locked(vma->vm_mm);
+        mm_lock_seq = vma->vm_mm->mm_lock_seq;
 
        down_write(&vma->vm_lock->lock);
        /*
        * We should use WRITE_ONCE() here because we can have concurrent reads
        * from the early lockless pessimistic check in vma_start_read().
        * We don't really care about the correctness of that early check, but
        * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy.
        */
        WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq);
        up_write(&vma->vm_lock->lock);

}

Quoted 4 lines

This will force vma_start_write() to always write-lock vma->vm_lock before changing vma->vm_lock_seq. Since vma->vm_lock survives reuse, the other readers/writers will synchronize on it even if vma got reused.

After thinking of all the alternatives, I think the cleanest way to handle vma detaching would be to follow the same pattern as for vma attaching. To attach a vma we do:

vma->vm_mm = xxx;
...
vma_mark_attached()

    smp_wmb();
    WRITE_ONCE(vma->detached, false);

lock_vma_under_rcu() ensures that a vma is attached and still unchanged like this:

lock_vma_under_rcu()

    vma_start_read();
    is_vma_detached()
        detached = READ_ONCE(vma->detached);
        smp_rmb();
    if (vma->vm_mm != mm)

So, vm_area_free() can follow the same pattern to ensure vma reuse gets detected even if lock_vma_under_rcu() succeeds in locking the vma:

vm_area_free()

    vma->vm_mm = NULL;
    smp_wmb();
    WRITE_ONCE(vma->detached, true);

Vlastimil, I think that should address the race you described. WDYT?

Quoted 21 lines
Quoted 21 lines
Quoted 20 lines

comment for vm_lock_seq explaining these requirements. Do you agree that such a change would resolve the issue?

Quoted 15 lines
Quoted 14 lines

vma_mark_detached() and before down_read_trylock(&vma->vm_lock.lock) in vma_start_read() the VMA write-lock should have been released by mmap_write_unlock() and therefore vma->detached=false should be visible to the reader when it executed lock_vma_under_rcu().

Quoted 8 lines

is_vma_detached() - false negative as the write above didn't propagate
here yet; a read barrier but where is the write barrier?
checks for vma->vm_mm, vm_start, vm_end - nobody reset them yet so false
positive, or they got reset on reallocation but writes didn't propagate

Am I missing something that would prevent lock_vma_under_rcu() falsely succeeding here?

#25 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Quoted 145 lines · Suren Baghdasaryan
Quoted 114 lines · Suren Baghdasaryan
Quoted 81 lines · Vlastimil Babka
Quoted 79 lines · Suren Baghdasaryan
Quoted 77 lines · Vlastimil Babka
Quoted 61 lines · Suren Baghdasaryan
Quoted 52 lines · Vlastimil Babka
Quoted 36 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 1 line

I'm wondering about the vma freeing path. Consider vma_complete():

vma_mark_detached(vp->remove);
vma->detached = true; - plain write
vm_area_free(vp->remove);
vma->vm_lock_seq = UINT_MAX; - plain write
kmem_cache_free(vm_area_cachep)
...
potential reallocation

against:

Quoted 7 lines

lock_vma_under_rcu()
- mas_walk finds a stale vma due to race
vma_start_read()
if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
- can be false, the vma was not being locked on the freeing side? down_read_trylock(&vma->vm_lock.lock) - suceeds, wasn't locked this is acquire, but was there any release?

Quoted 5 lines

Yes, there was a release. I think what you missed is that vma_mark_detached() that is called from vma_complete() requires VMA to be write-locked (see vma_assert_write_locked() in vma_mark_detached()). The rule is that a VMA can be attached without write-locking but only a write-locked VMA can be detached. So, after

OK but write unlocking means the mm's seqcount is bumped and becomes non-equal with vma's vma->vm_lock_seq, right?

Yet in the example above we happily set it to UINT_MAX and thus effectively false unlock it for vma_start_read()?

And this is all done before the vma_complete() side would actually reach mmap_write_unlock(), AFAICS.

Quoted 8 lines
Quoted 4 lines

Ah, you are right. With the possibility of reuse, even a freed VMA should be kept write-locked until it is unlocked by mmap_write_unlock(). I think the fix for this is simply to not reset vma->vm_lock_seq inside vm_area_free(). I'll also need to add a

But even if we don't reset vm_lock_seq to UINT_MAX, then whover reallocated it can proceed and end up doing a vma_start_write() and rewrite it there anyway, no?

Actually, I think with a small change we can simplify these locking rules:

static inline void vma_start_write(struct vm_area_struct *vma)

{
        int mm_lock_seq;
(quoted diff) +2 -2
-        if (__is_vma_write_locked(vma, &mm_lock_seq))
-                return;
+        mmap_assert_write_locked(vma->vm_mm);
+        mm_lock_seq = vma->vm_mm->mm_lock_seq;
 
        down_write(&vma->vm_lock->lock);
        /*
        * We should use WRITE_ONCE() here because we can have concurrent reads
        * from the early lockless pessimistic check in vma_start_read().
        * We don't really care about the correctness of that early check, but
        * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy.
        */
        WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq);
        up_write(&vma->vm_lock->lock);

}

Quoted 4 lines

This will force vma_start_write() to always write-lock vma->vm_lock before changing vma->vm_lock_seq. Since vma->vm_lock survives reuse, the other readers/writers will synchronize on it even if vma got reused.

After thinking of all the alternatives, I think the cleanest way to handle vma detaching would be to follow the same pattern as for vma attaching. To attach a vma we do:

vma->vm_mm = xxx;
...
vma_mark_attached()

    smp_wmb();
    WRITE_ONCE(vma->detached, false);

lock_vma_under_rcu() ensures that a vma is attached and still unchanged like this:

lock_vma_under_rcu()

    vma_start_read();
    is_vma_detached()
        detached = READ_ONCE(vma->detached);
        smp_rmb();
    if (vma->vm_mm != mm)

So, vm_area_free() can follow the same pattern to ensure vma reuse gets detected even if lock_vma_under_rcu() succeeds in locking the vma:

Quoted 6 lines

vm_area_free()

    vma->vm_mm = NULL;
    smp_wmb();
    WRITE_ONCE(vma->detached, true);

Vlastimil, I think that should address the race you described. WDYT?

I'm not sure. AFAIU the barriers would ensure that if lock_vma_under_rcu() sees detached, it also sees vm_mm is NULL. But as it doesn't ensure that it will see it detached, so it also doesn't ensure we will see vm_mm as NULL.

I think the main problem is that we unlock the vma by writing to a mm, not the vma, which makes it hard to apply the necessary SLAB_TYPESAFE_BY_RCU validation patterns to it. I thought the direction you were discussing with PeterZ in the other thread would solve this (in addition of getting rid of the rwsem, which we were considering it anyway, but enabling SLAB_TYPESAFE_BY_RCU by that would be a stronger argument).

Perhaps a solution to this that would work with the current rwsem would be that setting detached and vm_mm to NULL would be set under the down_write() of the rwsem. That would make sure that if lock_vma_under_rcu() succeeds the down_read_trylock(), it would be guaranteed to see those assignments?

Quoted 21 lines
Quoted 21 lines
Quoted 21 lines
Quoted 20 lines

comment for vm_lock_seq explaining these requirements. Do you agree that such a change would resolve the issue?

Quoted 15 lines
Quoted 14 lines

vma_mark_detached() and before down_read_trylock(&vma->vm_lock.lock) in vma_start_read() the VMA write-lock should have been released by mmap_write_unlock() and therefore vma->detached=false should be visible to the reader when it executed lock_vma_under_rcu().

Quoted 8 lines

is_vma_detached() - false negative as the write above didn't propagate
here yet; a read barrier but where is the write barrier?
checks for vma->vm_mm, vm_start, vm_end - nobody reset them yet so false
positive, or they got reset on reallocation but writes didn't propagate

Am I missing something that would prevent lock_vma_under_rcu() falsely succeeding here?

author maintainer
#26 lore raw

Re: [PATCH v5 4/6] mm: make vma cache SLAB_TYPESAFE_BY_RCU

Quoted 158 lines · Vlastimil Babka
Quoted 145 lines · Suren Baghdasaryan
Quoted 114 lines · Suren Baghdasaryan
Quoted 81 lines · Vlastimil Babka
Quoted 79 lines · Suren Baghdasaryan
Quoted 77 lines · Vlastimil Babka
Quoted 61 lines · Suren Baghdasaryan
Quoted 52 lines · Vlastimil Babka
Quoted 36 lines · Suren Baghdasaryan

To enable SLAB_TYPESAFE_BY_RCU for vma cache we need to ensure that object reuse before RCU grace period is over will be detected inside lock_vma_under_rcu().
lock_vma_under_rcu() enters RCU read section, finds the vma at the given address, locks the vma and checks if it got detached or remapped to cover a different address range. These last checks are there to ensure that the vma was not modified after we found it but before locking it.
vma reuse introduces several new possibilities:
1. vma can be reused after it was found but before it is locked;
2. vma can be reused and reinitialized (including changing its vm_mm) while being locked in vma_start_read();
3. vma can be reused and reinitialized after it was found but before it is locked, then attached at a new address or to a new mm while read-locked;
For case #1 current checks will help detecting cases when:
- vma was reused but not yet added into the tree (detached check)
- vma was reused at a different address range (address check); We are missing the check for vm_mm to ensure the reused vma was not attached to a different mm. This patch adds the missing check. For case #2, we pass mm to vma_start_read() to prevent access to unstable vma->vm_mm. This might lead to vma_start_read() returning a false locked result but that's not critical if it's rare because it will only lead to a retry under mmap_lock.
For case #3, we ensure the order in which vma->detached flag and vm_start/vm_end/vm_mm are set and checked. vma gets attached after vm_start/vm_end/vm_mm were set and lock_vma_under_rcu() should check vma->detached before checking vm_start/vm_end/vm_mm. This is required because attaching vma happens without vma write-lock, as opposed to vma detaching, which requires vma write-lock. This patch adds memory barriers inside is_vma_detached() and vma_mark_attached() needed to order reads and writes to vma->detached vs vm_start/vm_end/vm_mm. After these provisions, SLAB_TYPESAFE_BY_RCU is added to vm_area_cachep. This will facilitate vm_area_struct reuse and will minimize the number of call_rcu() calls.

Quoted 1 line

I'm wondering about the vma freeing path. Consider vma_complete():

vma_mark_detached(vp->remove);
vma->detached = true; - plain write
vm_area_free(vp->remove);
vma->vm_lock_seq = UINT_MAX; - plain write
kmem_cache_free(vm_area_cachep)
...
potential reallocation

against:

Quoted 7 lines

lock_vma_under_rcu()
- mas_walk finds a stale vma due to race
vma_start_read()
if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
- can be false, the vma was not being locked on the freeing side? down_read_trylock(&vma->vm_lock.lock) - suceeds, wasn't locked this is acquire, but was there any release?

Quoted 5 lines

Yes, there was a release. I think what you missed is that vma_mark_detached() that is called from vma_complete() requires VMA to be write-locked (see vma_assert_write_locked() in vma_mark_detached()). The rule is that a VMA can be attached without write-locking but only a write-locked VMA can be detached. So, after

OK but write unlocking means the mm's seqcount is bumped and becomes non-equal with vma's vma->vm_lock_seq, right?

Yet in the example above we happily set it to UINT_MAX and thus effectively false unlock it for vma_start_read()?

And this is all done before the vma_complete() side would actually reach mmap_write_unlock(), AFAICS.

Quoted 8 lines
Quoted 4 lines

Ah, you are right. With the possibility of reuse, even a freed VMA should be kept write-locked until it is unlocked by mmap_write_unlock(). I think the fix for this is simply to not reset vma->vm_lock_seq inside vm_area_free(). I'll also need to add a

But even if we don't reset vm_lock_seq to UINT_MAX, then whover reallocated it can proceed and end up doing a vma_start_write() and rewrite it there anyway, no?

Actually, I think with a small change we can simplify these locking rules:

static inline void vma_start_write(struct vm_area_struct *vma)

{
        int mm_lock_seq;
(quoted diff) +2 -2
-        if (__is_vma_write_locked(vma, &mm_lock_seq))
-                return;
+        mmap_assert_write_locked(vma->vm_mm);
+        mm_lock_seq = vma->vm_mm->mm_lock_seq;
 
        down_write(&vma->vm_lock->lock);
        /*
        * We should use WRITE_ONCE() here because we can have concurrent reads
        * from the early lockless pessimistic check in vma_start_read().
        * We don't really care about the correctness of that early check, but
        * we should use WRITE_ONCE() for cleanliness and to keep KCSAN happy.
        */
        WRITE_ONCE(vma->vm_lock_seq, mm_lock_seq);
        up_write(&vma->vm_lock->lock);

}

Quoted 4 lines

This will force vma_start_write() to always write-lock vma->vm_lock before changing vma->vm_lock_seq. Since vma->vm_lock survives reuse, the other readers/writers will synchronize on it even if vma got reused.

After thinking of all the alternatives, I think the cleanest way to handle vma detaching would be to follow the same pattern as for vma attaching. To attach a vma we do:

vma->vm_mm = xxx;
...
vma_mark_attached()

    smp_wmb();
    WRITE_ONCE(vma->detached, false);

lock_vma_under_rcu() ensures that a vma is attached and still unchanged like this:

lock_vma_under_rcu()

    vma_start_read();
    is_vma_detached()
        detached = READ_ONCE(vma->detached);
        smp_rmb();
    if (vma->vm_mm != mm)

So, vm_area_free() can follow the same pattern to ensure vma reuse gets detected even if lock_vma_under_rcu() succeeds in locking the vma:

Quoted 6 lines

vm_area_free()

    vma->vm_mm = NULL;
    smp_wmb();
    WRITE_ONCE(vma->detached, true);

Vlastimil, I think that should address the race you described. WDYT?

I'm not sure. AFAIU the barriers would ensure that if lock_vma_under_rcu() sees detached, it also sees vm_mm is NULL. But as it doesn't ensure that it will see it detached, so it also doesn't ensure we will see vm_mm as NULL.

Quoted 6 lines

I think the main problem is that we unlock the vma by writing to a mm, not the vma, which makes it hard to apply the necessary SLAB_TYPESAFE_BY_RCU validation patterns to it. I thought the direction you were discussing with PeterZ in the other thread would solve this (in addition of getting rid of the rwsem, which we were considering it anyway, but enabling SLAB_TYPESAFE_BY_RCU by that would be a stronger argument).

I was hoping to implement SLAB_TYPESAFE_BY_RCU independently from vm_lock change but you are probably right. Incorporating vma->detached flag into the lock itself (which survives reuse) would make things way easier. Let me pivot towards making that change first and see if SLAB_TYPESAFE_BY_RCU becomes simpler.

Quoted 5 lines

Perhaps a solution to this that would work with the current rwsem would be that setting detached and vm_mm to NULL would be set under the down_write() of the rwsem. That would make sure that if lock_vma_under_rcu() succeeds the down_read_trylock(), it would be guaranteed to see those assignments?

Yeah, that would definitely work. I was trying to avoid extra locking but it looks like it's unavoidable.
Anyway, let me try replacing vm_lock first and will see where we end up. Thanks for the input!

Quoted 23 lines
Quoted 22 lines
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines

comment for vm_lock_seq explaining these requirements. Do you agree that such a change would resolve the issue?

Quoted 15 lines
Quoted 14 lines

vma_mark_detached() and before down_read_trylock(&vma->vm_lock.lock) in vma_start_read() the VMA write-lock should have been released by mmap_write_unlock() and therefore vma->detached=false should be visible to the reader when it executed lock_vma_under_rcu().

Quoted 8 lines

is_vma_detached() - false negative as the write above didn't propagate
here yet; a read barrier but where is the write barrier?
checks for vma->vm_mm, vm_start, vm_end - nobody reset them yet so false
positive, or they got reset on reallocation but writes didn't propagate

Am I missing something that would prevent lock_vma_under_rcu() falsely succeeding here?

author maintainer
#27 lore raw

PATCH v5 5/6 mm/slab: allow freeptr_offset to be used with ctor

In reply to Suren Baghdasaryan

There is no real reason to prevent freeptr_offset usage when a slab cache has a ctor. The only real limitation is that any field unioned with the free pointer and initialized by ctor will be overwritten since free pointer is set after @ctor invocation. Document this limitation and enable usage of freeptr_offset with ctor.

---
 include/linux/slab.h | 5 +++--
 mm/slub.c            | 2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/slab.h +3 -2
index 681b685b6c4e..6bad744bef5e 100644
@@ -305,8 +305,9 @@ struct kmem_cache_args {
 	 * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
 	 * is specified, %use_freeptr_offset must be set %true.
 	 *
-	 * Note that @ctor currently isn't supported with custom free pointers
-	 * as a @ctor requires an external free pointer.
+	 * Note that fields unioned with free pointer cannot be initialized by
+	 * @ctor since free pointer is set after @ctor invocation, so those
+	 * values will be overwritten.
 	 */
 	unsigned int freeptr_offset;
 	/**
mm/slub.c +1 -1
index 870a1d95521d..f62c829b7b6b 100644
@@ -5462,7 +5462,7 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s)
 	s->inuse = size;
 
 	if (((flags & SLAB_TYPESAFE_BY_RCU) && !args->use_freeptr_offset) ||
-	    (flags & SLAB_POISON) || s->ctor ||
+	    (flags & SLAB_POISON) || (s->ctor && !args->use_freeptr_offset) ||
 	    ((flags & SLAB_RED_ZONE) &&
 	     (s->object_size < sizeof(void *) || slub_debug_orig_size(s)))) {
 		/*
Signature · 1 line
2.47.0.338.g60cca15819-goog
Reviewed-by on v5 5/6
#28 lore raw

Re: [PATCH v5 5/6] mm/slab: allow freeptr_offset to be used with ctor

Quoted 8 lines · Suren Baghdasaryan

There is no real reason to prevent freeptr_offset usage when a slab cache has a ctor. The only real limitation is that any field unioned with the free pointer and initialized by ctor will be overwritten since free pointer is set after @ctor invocation. Document this limitation and enable usage of freeptr_offset with ctor.

Quoted 34 lines
---
 include/linux/slab.h | 5 +++--
 mm/slub.c            | 2 +-
 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/slab.h +3 -2
index 681b685b6c4e..6bad744bef5e 100644
@@ -305,8 +305,9 @@ struct kmem_cache_args {
 	 * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
 	 * is specified, %use_freeptr_offset must be set %true.
 	 *
-	 * Note that @ctor currently isn't supported with custom free pointers
-	 * as a @ctor requires an external free pointer.
+	 * Note that fields unioned with free pointer cannot be initialized by
+	 * @ctor since free pointer is set after @ctor invocation, so those
+	 * values will be overwritten.
 	 */
 	unsigned int freeptr_offset;
 	/**
mm/slub.c +1 -1
index 870a1d95521d..f62c829b7b6b 100644
@@ -5462,7 +5462,7 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s)
 	s->inuse = size;
 
 	if (((flags & SLAB_TYPESAFE_BY_RCU) && !args->use_freeptr_offset) ||
-	    (flags & SLAB_POISON) || s->ctor ||
+	    (flags & SLAB_POISON) || (s->ctor && !args->use_freeptr_offset) ||
 	    ((flags & SLAB_RED_ZONE) &&
 	     (s->object_size < sizeof(void *) || slub_debug_orig_size(s)))) {
 		/*
author maintainer
#29 lore raw

PATCH v5 6/6 docs/mm: document latest changes to vm_lock

In reply to Suren Baghdasaryan

Change the documentation to reflect that vm_lock is integrated into vma. Document newly introduced vma_start_read_locked{_nested} functions.

---
 Documentation/mm/process_addrs.rst | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
Documentation/mm/process_addrs.rst +7 -3
index 81417fa2ed20..92cf497a9e3c 100644
@@ -716,7 +716,11 @@ calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU
 critical section, then attempts to VMA lock it via :c:func:`!vma_start_read`,
 before releasing the RCU lock via :c:func:`!rcu_read_unlock`.
 
-VMA read locks hold the read lock on the :c:member:`!vma->vm_lock` semaphore for
+In cases when the user already holds mmap read lock, :c:func:`!vma_start_read_locked`
+and :c:func:`!vma_start_read_locked_nested` can be used. These functions always
+succeed in acquiring VMA read lock.
+
+VMA read locks hold the read lock on the :c:member:`!vma.vm_lock` semaphore for
 their duration and the caller of :c:func:`!lock_vma_under_rcu` must release it
 via :c:func:`!vma_end_read`.
 
@@ -780,7 +784,7 @@ keep VMAs locked across entirely separate write operations. It also maintains
 correct lock ordering.
 
 Each time a VMA read lock is acquired, we acquire a read lock on the
-:c:member:`!vma->vm_lock` read/write semaphore and hold it, while checking that
+:c:member:`!vma.vm_lock` read/write semaphore and hold it, while checking that
 the sequence count of the VMA does not match that of the mm.
 
 If it does, the read lock fails. If it does not, we hold the lock, excluding
@@ -790,7 +794,7 @@ Importantly, maple tree operations performed in :c:func:`!lock_vma_under_rcu`
 are also RCU safe, so the whole read lock operation is guaranteed to function
 correctly.
 
-On the write side, we acquire a write lock on the :c:member:`!vma->vm_lock`
+On the write side, we acquire a write lock on the :c:member:`!vma.vm_lock`
 read/write semaphore, before setting the VMA's sequence number under this lock,
 also simultaneously holding the mmap write lock.
 
Signature · 1 line
2.47.0.338.g60cca15819-goog
reviewer
#30 lore raw

Re: [PATCH v5 6/6] docs/mm: document latest changes to vm_lock

Hi,

Can someone explain what the (consistent) usage of '!' does in this file? This is the only file in Documentation/ that uses this syntax.

E.g.:

Documentation/mm/process_addrs.rst calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU · line 721

Quoted 12 lines
Documentation/mm/process_addrs.rst +3 -1
index 81417fa2ed20..92cf497a9e3c 100644
@@ -716,7 +716,11 @@ calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU
 critical section, then attempts to VMA lock it via :c:func:`!vma_start_read`,
 before releasing the RCU lock via :c:func:`!rcu_read_unlock`.
 
-VMA read locks hold the read lock on the :c:member:`!vma->vm_lock` semaphore for
+In cases when the user already holds mmap read lock, :c:func:`!vma_start_read_locked`
+and :c:func:`!vma_start_read_locked_nested` can be used. These functions always
+succeed in acquiring VMA read lock.

thanks.

Signature · 1 line
~Randy
#31 lore raw

Re: [PATCH v5 6/6] docs/mm: document latest changes to vm_lock

Quoted 16 lines · Randy Dunlap

Hi,

Can someone explain what the (consistent) usage of '!' does in this file? This is the only file in Documentation/ that uses this syntax.

E.g.:

Quoted 8 lines
Documentation/mm/process_addrs.rst +0 -0
index 81417fa2ed20..92cf497a9e3c 100644
@@ -716,7 +716,11 @@ calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU
 critical section, then attempts to VMA lock it via :c:func:`!vma_start_read`,
 before releasing the RCU lock via :c:func:`!rcu_read_unlock`.
 
Quoted 5 lines

Documentation/mm/process_addrs.rst calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU · line 721

Quoted 4 lines
Documentation/mm/process_addrs.rst +3 -1
-VMA read locks hold the read lock on the :c:member:`!vma->vm_lock` semaphore for
+In cases when the user already holds mmap read lock, :c:func:`!vma_start_read_locked`
+and :c:func:`!vma_start_read_locked_nested` can be used. These functions always
+succeed in acquiring VMA read lock.

Quoting from https://www.sphinx-doc.org/en/master/usage/referencing.html#syntax

* Suppressed link: Prefixing with an exclamation mark (!) prevents the creation of a link but otherwise keeps the visual output of the role.

For example, writing :py:func:!target displays target(), with no link generated.

This is helpful for cases in which the link target does not exist; e.g. changelog entries that describe removed functionality, or third-party libraries that don’t support intersphinx. Suppressing the link prevents warnings in nitpicky mode.

But in kernel documentation, there is a preferred alternative. Referencing by function names is the way to go. For example:

calls rcu_read_lock() to ensure that the VMA is looked up in an RCU critical section, then attempts to VMA lock it via vma_start_read(), before releasing the RCU lock via rcu_read_unlock().

In cases when the user already holds mmap read lock, vma_start_read_locked() and vma_start_read_locked_nested() can be used. These functions always succeed in acquiring VMA read lock.

They work regardless of link target's existence. Kernel-specific Sphinx extension named "automarkup" does conversions for you.

HTH, Akira

Quoted 3 lines

thanks.

Signature · 1 line
~Randy
author maintainer
#32 lore raw

Re: [PATCH v5 6/6] docs/mm: document latest changes to vm_lock

Quoted 44 lines · Akira Yokosawa
Quoted 16 lines · Randy Dunlap

Hi,

Can someone explain what the (consistent) usage of '!' does in this file? This is the only file in Documentation/ that uses this syntax.

E.g.:

Quoted 8 lines
Documentation/mm/process_addrs.rst +0 -0
index 81417fa2ed20..92cf497a9e3c 100644
@@ -716,7 +716,11 @@ calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU
 critical section, then attempts to VMA lock it via :c:func:`!vma_start_read`,
 before releasing the RCU lock via :c:func:`!rcu_read_unlock`.
 
Quoted 5 lines

Documentation/mm/process_addrs.rst calls :c:func:`!rcu_read_lock` to ensure that the VMA is looked up in an RCU · line 721

Quoted 4 lines
Documentation/mm/process_addrs.rst +3 -1
-VMA read locks hold the read lock on the :c:member:`!vma->vm_lock` semaphore for
+In cases when the user already holds mmap read lock, :c:func:`!vma_start_read_locked`
+and :c:func:`!vma_start_read_locked_nested` can be used. These functions always
+succeed in acquiring VMA read lock.

Quoting from https://www.sphinx-doc.org/en/master/usage/referencing.html#syntax

* Suppressed link: Prefixing with an exclamation mark (!) prevents the creation of a link but otherwise keeps the visual output of the role.

For example, writing :py:func:!target displays target(), with no link generated.

This is helpful for cases in which the link target does not exist; e.g. changelog entries that describe removed functionality, or third-party libraries that don’t support intersphinx. Suppressing the link prevents warnings in nitpicky mode.

But in kernel documentation, there is a preferred alternative. Referencing by function names is the way to go. For example:

calls rcu_read_lock() to ensure that the VMA is looked up in an RCU critical section, then attempts to VMA lock it via vma_start_read(), before releasing the RCU lock via rcu_read_unlock().

Quoted 7 lines

In cases when the user already holds mmap read lock, vma_start_read_locked() and vma_start_read_locked_nested() can be used. These functions always succeed in acquiring VMA read lock.

They work regardless of link target's existence. Kernel-specific Sphinx extension named "automarkup" does conversions for you.

Thanks for the information. I was simply following the same style the document was written in. Sounds like converting it to the preferred alternative in a separate patch would be best. Lorenzo, WDYT?

Quoted 7 lines

HTH, Akira

Quoted 3 lines

thanks.
--
~Randy

maintainer
#33 lore raw
In reply to Suren Baghdasaryan
Quoted 7 lines · On Fri, 6 Dec 2024 14:51:57 -0800 Suren Baghdasaryan <[email protected]> wrote:

Back when per-vma locks were introduces, vm_lock was moved out of vm_area_struct in [1] because of the performance regression caused by false cacheline sharing. Recent investigation [2] revealed that the regressions is limited to a rather old Broadwell microarchitecture and even there it can be mitigated by disabling adjacent cacheline prefetching, see [3].

Quoted 7 lines

...

Patch applies over linux-next (due to vm_lock change [5] not in mm tree).

...

[5] https://lore.kernel.org/all/[email protected]/

Well that's awkward. I added the "seqlock: add raw_seqcount_try_begin" series to mm.git. Peter, please drop your copy from linux-next?