Skip to contentSkip to conversation

mm: limit the scope of vma_start_read()

PATCH Superseded Suren Baghdasaryan 3 participants 8 messages Jul 31, 2025 – Aug 1, 2025

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Suren Baghdasaryan

author maintainer
#0 lore raw

PATCH v2 1/2 mm: limit the scope of vma_start_read()

Limit the scope of vma_start_read() as it is used only as a helper for higher-level locking functions implemented inside mmap_lock.c and we are about to introduce more complex RCU rules for this function. The change is pure code refactoring and has no functional changes.

---
 include/linux/mmap_lock.h | 85 ---------------------------------------
 mm/mmap_lock.c            | 85 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 85 insertions(+), 85 deletions(-)
include/linux/mmap_lock.h +0 -85
index 11a078de9150..2c9fffa58714 100644
@@ -147,91 +147,6 @@ static inline void vma_refcount_put(struct vm_area_struct *vma)
 	}
 }
 
-/*
- * Try to read-lock a vma. The function is allowed to occasionally yield false
- * locked result to avoid performance overhead, in which case we fall back to
- * using mmap_lock. The function should never yield false unlocked result.
- * False locked result is possible if mm_lock_seq overflows or if vma gets
- * reused and attached to a different mm before we lock it.
- * Returns the vma on success, NULL on failure to lock and EAGAIN if vma got
- * detached.
- *
- * WARNING! The vma passed to this function cannot be used if the function
- * fails to lock it because in certain cases RCU lock is dropped and then
- * reacquired. Once RCU lock is dropped the vma can be concurently freed.
- */
-static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
-						    struct vm_area_struct *vma)
-{
-	int oldcnt;
-
-	/*
-	 * Check before locking. A race might cause false locked result.
-	 * We can use READ_ONCE() for the mm_lock_seq here, and don't need
-	 * ACQUIRE semantics, because this is just a lockless check whose result
-	 * we don't rely on for anything - the mm_lock_seq read against which we
-	 * need ordering is below.
-	 */
-	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
-		return NULL;
-
-	/*
-	 * If VMA_LOCK_OFFSET is set, __refcount_inc_not_zero_limited_acquire()
-	 * will fail because VMA_REF_LIMIT is less than VMA_LOCK_OFFSET.
-	 * Acquire fence is required here to avoid reordering against later
-	 * vm_lock_seq check and checks inside lock_vma_under_rcu().
-	 */
-	if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt,
-							      VMA_REF_LIMIT))) {
-		/* return EAGAIN if vma got detached from under us */
-		return oldcnt ? NULL : ERR_PTR(-EAGAIN);
-	}
-
-	rwsem_acquire_read(&vma->vmlock_dep_map, 0, 1, _RET_IP_);
-
-	/*
-	 * If vma got attached to another mm from under us, that mm is not
-	 * stable and can be freed in the narrow window after vma->vm_refcnt
-	 * is dropped and before rcuwait_wake_up(mm) is called. Grab it before
-	 * releasing vma->vm_refcnt.
-	 */
-	if (unlikely(vma->vm_mm != mm)) {
-		/* Use a copy of vm_mm in case vma is freed after we drop vm_refcnt */
-		struct mm_struct *other_mm = vma->vm_mm;
-
-		/*
-		 * __mmdrop() is a heavy operation and we don't need RCU
-		 * protection here. Release RCU lock during these operations.
-		 * We reinstate the RCU read lock as the caller expects it to
-		 * be held when this function returns even on error.
-		 */
-		rcu_read_unlock();
-		mmgrab(other_mm);
-		vma_refcount_put(vma);
-		mmdrop(other_mm);
-		rcu_read_lock();
-		return NULL;
-	}
-
-	/*
-	 * Overflow of vm_lock_seq/mm_lock_seq might produce false locked result.
-	 * False unlocked result is impossible because we modify and check
-	 * vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq
-	 * modification invalidates all existing locks.
-	 *
-	 * We must use ACQUIRE semantics for the mm_lock_seq so that if we are
-	 * racing with vma_end_write_all(), we only start reading from the VMA
-	 * after it has been unlocked.
-	 * This pairs with RELEASE semantics in vma_end_write_all().
-	 */
-	if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) {
-		vma_refcount_put(vma);
-		return NULL;
-	}
-
-	return vma;
-}
-
 /*
  * Use only while holding mmap read lock which guarantees that locking will not
  * fail (nobody can concurrently write-lock the vma). vma_start_read() should
mm/mmap_lock.c +85 -0

91 lines not on this page · raw message

base-commit: 01da54f10fddf3b01c5a3b80f6b16bbad390c302

Signature · 1 line
2.50.1.552.g942d659e1b-goog
author maintainer
#1 lore raw

PATCH v2 2/2 mm: change vma_start_read() to drop RCU lock on failure

vma_start_read() can drop and reacquire RCU lock in certain failure cases. It's not apparent that the RCU session started by the caller of this function might be interrupted when vma_start_read() fails to lock the vma. This might become a source of subtle bugs and to prevent that we change the locking rules for vma_start_read() to drop RCU read lock upon failure. This way it's more obvious that RCU-protected objects are unsafe after vma locking fails.

---
Changes since v1 [1]:
- Fixed missing RCU unlock in lock_vma_under_rcu(), per Lorenzo Stoakes
- Modified comments, per Lorenzo Stoakes

[1] https://lore.kernel.org/all/[email protected]/

 mm/mmap_lock.c | 86 +++++++++++++++++++++++++++-----------------------
 1 file changed, 47 insertions(+), 39 deletions(-)
mm/mmap_lock.c +47 -39
index 10826f347a9f..7ea603f26975 100644
@@ -136,15 +136,16 @@ void vma_mark_detached(struct vm_area_struct *vma)
  * Returns the vma on success, NULL on failure to lock and EAGAIN if vma got
  * detached.
  *
- * WARNING! The vma passed to this function cannot be used if the function
- * fails to lock it because in certain cases RCU lock is dropped and then
- * reacquired. Once RCU lock is dropped the vma can be concurently freed.
+ * IMPORTANT: RCU lock must be held upon entering the function, but upon error
+ *            IT IS RELEASED. The caller must handle this correctly.
  */
 static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
 						    struct vm_area_struct *vma)
 {
+	struct mm_struct *other_mm;
 	int oldcnt;
 
+	RCU_LOCKDEP_WARN(!rcu_read_lock_held(), "no rcu lock held");
 	/*
 	 * Check before locking. A race might cause false locked result.
 	 * We can use READ_ONCE() for the mm_lock_seq here, and don't need
@@ -152,8 +153,10 @@ static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
 	 * we don't rely on for anything - the mm_lock_seq read against which we
 	 * need ordering is below.
 	 */
-	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
-		return NULL;
+	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) {
+		vma = NULL;
+		goto err;
+	}
 
 	/*
 	 * If VMA_LOCK_OFFSET is set, __refcount_inc_not_zero_limited_acquire()
@@ -164,34 +167,14 @@ static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
 	if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt,
 							      VMA_REF_LIMIT))) {
 		/* return EAGAIN if vma got detached from under us */
-		return oldcnt ? NULL : ERR_PTR(-EAGAIN);
+		vma = oldcnt ? NULL : ERR_PTR(-EAGAIN);
+		goto err;
 	}
 
 	rwsem_acquire_read(&vma->vmlock_dep_map, 0, 1, _RET_IP_);
 
-	/*
-	 * If vma got attached to another mm from under us, that mm is not
-	 * stable and can be freed in the narrow window after vma->vm_refcnt
-	 * is dropped and before rcuwait_wake_up(mm) is called. Grab it before
-	 * releasing vma->vm_refcnt.
-	 */
-	if (unlikely(vma->vm_mm != mm)) {
-		/* Use a copy of vm_mm in case vma is freed after we drop vm_refcnt */
-		struct mm_struct *other_mm = vma->vm_mm;
-
-		/*
-		 * __mmdrop() is a heavy operation and we don't need RCU
-		 * protection here. Release RCU lock during these operations.
-		 * We reinstate the RCU read lock as the caller expects it to
-		 * be held when this function returns even on error.
-		 */
-		rcu_read_unlock();
-		mmgrab(other_mm);
-		vma_refcount_put(vma);
-		mmdrop(other_mm);
-		rcu_read_lock();
-		return NULL;
-	}
+	if (unlikely(vma->vm_mm != mm))
+		goto err_unstable;
 
 	/*
 	 * Overflow of vm_lock_seq/mm_lock_seq might produce false locked result.
@@ -206,10 +189,31 @@ static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
 	 */
 	if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) {
 		vma_refcount_put(vma);
-		return NULL;
+		vma = NULL;
+		goto err;
 	}
 
 	return vma;
+err:
+	rcu_read_unlock();
+
+	return vma;
+err_unstable:
+	/*
+	 * If vma got attached to another mm from under us, that mm is not
+	 * stable and can be freed in the narrow window after vma->vm_refcnt
+	 * is dropped and before rcuwait_wake_up(mm) is called. Grab it before
+	 * releasing vma->vm_refcnt.
+	 */
+	other_mm = vma->vm_mm; /* use a copy as vma can be freed after we drop vm_refcnt */
+
+	/* __mmdrop() is a heavy operation, do it after dropping RCU lock. */
+	rcu_read_unlock();
+	mmgrab(other_mm);
+	vma_refcount_put(vma);
+	mmdrop(other_mm);
+
+	return NULL;
 }
 
 /*
@@ -223,11 +227,13 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 	MA_STATE(mas, &mm->mm_mt, address, address);
 	struct vm_area_struct *vma;
 
-	rcu_read_lock();
 retry:
+	rcu_read_lock();
 	vma = mas_walk(&mas);
-	if (!vma)
+	if (!vma) {
+		rcu_read_unlock();
 		goto inval;
+	}
 
 	vma = vma_start_read(mm, vma);
 	if (IS_ERR_OR_NULL(vma)) {
@@ -241,6 +247,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 		/* Failed to lock the VMA */
 		goto inval;
 	}
+
+	rcu_read_unlock();
+
 	/*
 	 * At this point, we have a stable reference to a VMA: The VMA is
 	 * locked and we know it hasn't already been isolated.
@@ -249,16 +258,14 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 	 */
 
 	/* Check if the vma we locked is the right one. */
-	if (unlikely(address < vma->vm_start || address >= vma->vm_end))
-		goto inval_end_read;
+	if (unlikely(address < vma->vm_start || address >= vma->vm_end)) {
+		vma_end_read(vma);
+		goto inval;
+	}
 
-	rcu_read_unlock();
 	return vma;
 
-inval_end_read:
-	vma_end_read(vma);
 inval:
-	rcu_read_unlock();
 	count_vm_vma_lock_event(VMA_LOCK_ABORT);
 	return NULL;
 }
@@ -313,6 +320,7 @@ struct vm_area_struct *lock_next_vma(struct mm_struct *mm,
 		 */
 		if (PTR_ERR(vma) == -EAGAIN) {
 			/* reset to search from the last address */
+			rcu_read_lock();
 			vma_iter_set(vmi, from_addr);
 			goto retry;
 		}
@@ -342,9 +350,9 @@ struct vm_area_struct *lock_next_vma(struct mm_struct *mm,
 	return vma;
 
 fallback_unlock:
+	rcu_read_unlock();
 	vma_end_read(vma);
 fallback:
-	rcu_read_unlock();
 	vma = lock_next_vma_under_mmap_lock(mm, vmi, from_addr);
 	rcu_read_lock();
 	/* Reinitialize the iterator after re-entering rcu read section */
Signature · 1 line
2.50.1.552.g942d659e1b-goog
Reviewed-by on v2 2/2
#2 lore raw

Re: [PATCH v2 2/2] mm: change vma_start_read() to drop RCU lock on failure

Quoted 9 lines · Suren Baghdasaryan

vma_start_read() can drop and reacquire RCU lock in certain failure cases. It's not apparent that the RCU session started by the caller of this function might be interrupted when vma_start_read() fails to lock the vma. This might become a source of subtle bugs and to prevent that we change the locking rules for vma_start_read() to drop RCU read lock upon failure. This way it's more obvious that RCU-protected objects are unsafe after vma locking fails.

IIRC you considered it yourself, I just convinced you to try :)

Quoted 1 line

I thought we didn't need the drop rcu lock for -EAGAIN, but that would just made it more complex for little gain, so this looks good to me.

Nit:

Quoted 17 lines
mm/mmap_lock.c +4 -2
@@ -223,11 +227,13 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 	MA_STATE(mas, &mm->mm_mt, address, address);
 	struct vm_area_struct *vma;
 
-	rcu_read_lock();
 retry:
+	rcu_read_lock();
 	vma = mas_walk(&mas);
-	if (!vma)
+	if (!vma) {
+		rcu_read_unlock();
 		goto inval;
+	}
 
 	vma = vma_start_read(mm, vma);
 	if (IS_ERR_OR_NULL(vma)) {

mm/mmap_lock.c struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, · line 251

Quoted 5 lines
mm/mmap_lock.c +2 -0
@@ -241,6 +247,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 		/* Failed to lock the VMA */
 		goto inval;
 	}
+
+	rcu_read_unlock();

Would it make sense to put this under the comment below?

mm/mmap_lock.c struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, · line 255

Quoted 4 lines
mm/mmap_lock.c +1 -0
+
 	/*
 	 * At this point, we have a stable reference to a VMA: The VMA is
 	 * locked and we know it hasn't already been isolated.

Give it continues like this:

* From here on, we can access the VMA without worrying about which

* fields are accessible for RCU readers.

Quoted 21 lines
mm/mmap_lock.c +4 -6
@@ -249,16 +258,14 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 	 */
 
 	/* Check if the vma we locked is the right one. */
-	if (unlikely(address < vma->vm_start || address >= vma->vm_end))
-		goto inval_end_read;
+	if (unlikely(address < vma->vm_start || address >= vma->vm_end)) {
+		vma_end_read(vma);
+		goto inval;
+	}
 
-	rcu_read_unlock();
 	return vma;
 
-inval_end_read:
-	vma_end_read(vma);
 inval:
-	rcu_read_unlock();
 	count_vm_vma_lock_event(VMA_LOCK_ABORT);
 	return NULL;
 }
author maintainer
#3 lore raw

Re: [PATCH v2 2/2] mm: change vma_start_read() to drop RCU lock on failure

Quoted 10 lines · Vlastimil Babka
Quoted 8 lines · Suren Baghdasaryan

vma_start_read() can drop and reacquire RCU lock in certain failure cases. It's not apparent that the RCU session started by the caller of this function might be interrupted when vma_start_read() fails to lock the vma. This might become a source of subtle bugs and to prevent that we change the locking rules for vma_start_read() to drop RCU read lock upon failure. This way it's more obvious that RCU-protected objects are unsafe after vma locking fails.

Quoted 8 lines
Quoted 1 line

IIRC you considered it yourself, I just convinced you to try :)

Quoted 1 line

I thought we didn't need the drop rcu lock for -EAGAIN, but that would just made it more complex for little gain, so this looks good to me.

Yes, we technically don't but this way it's simpler to explain (drop RCU on any failure).

Quoted 35 lines

Nit:

Quoted 17 lines
mm/mmap_lock.c +4 -2
@@ -223,11 +227,13 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
      MA_STATE(mas, &mm->mm_mt, address, address);
      struct vm_area_struct *vma;
 
-     rcu_read_lock();
 retry:
+     rcu_read_lock();
      vma = mas_walk(&mas);
-     if (!vma)
+     if (!vma) {
+             rcu_read_unlock();
              goto inval;
+     }
 
      vma = vma_start_read(mm, vma);
      if (IS_ERR_OR_NULL(vma)) {

mm/mmap_lock.c struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, · line 251

Quoted 5 lines
mm/mmap_lock.c +2 -0
@@ -241,6 +247,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
              /* Failed to lock the VMA */
              goto inval;
      }
+
+     rcu_read_unlock();

Would it make sense to put this under the comment below?

mm/mmap_lock.c struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, · line 255

Quoted 4 lines
mm/mmap_lock.c +1 -0
+
      /*
       * At this point, we have a stable reference to a VMA: The VMA is
       * locked and we know it hasn't already been isolated.
Quoted 5 lines

Give it continues like this:

* From here on, we can access the VMA without worrying about which

* fields are accessible for RCU readers.

Yep, will change.
Thanks!

Quoted 22 lines
Quoted 22 lines
mm/mmap_lock.c +4 -6
@@ -249,16 +258,14 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
       */
 
      /* Check if the vma we locked is the right one. */
-     if (unlikely(address < vma->vm_start || address >= vma->vm_end))
-             goto inval_end_read;
+     if (unlikely(address < vma->vm_start || address >= vma->vm_end)) {
+             vma_end_read(vma);
+             goto inval;
+     }
 
-     rcu_read_unlock();
      return vma;
 
-inval_end_read:
-     vma_end_read(vma);
 inval:
-     rcu_read_unlock();
      count_vm_vma_lock_event(VMA_LOCK_ABORT);
      return NULL;
 }
Reviewed-by Tested-by on v2 2/2
#4 lore raw

Re: [PATCH v2 2/2] mm: change vma_start_read() to drop RCU lock on failure

In reply to Suren Baghdasaryan
Quoted 10 lines · Suren Baghdasaryan

vma_start_read() can drop and reacquire RCU lock in certain failure cases. It's not apparent that the RCU session started by the caller of this function might be interrupted when vma_start_read() fails to lock the vma. This might become a source of subtle bugs and to prevent that we change the locking rules for vma_start_read() to drop RCU read lock upon failure. This way it's more obvious that RCU-protected objects are unsafe after vma locking fails.

Thanks for iterating on this so quickly - I think this is a good cleanup so well worth having.

The change LGTM, so:

I also confirmed this fixes the issue locally and ran mm self-tests so feel free to also add:

Quoted 15 lines

---
Changes since v1 [1]:
- Fixed missing RCU unlock in lock_vma_under_rcu(), per Lorenzo Stoakes
- Modified comments, per Lorenzo Stoakes

[1] https://lore.kernel.org/all/[email protected]/

 mm/mmap_lock.c | 86 +++++++++++++++++++++++++++-----------------------
 1 file changed, 47 insertions(+), 39 deletions(-)
mm/mmap_lock.c +0 -0
index 10826f347a9f..7ea603f26975 100644

mm/mmap_lock.c void vma_mark_detached(struct vm_area_struct *vma) · line 141

Quoted 9 lines
mm/mmap_lock.c +2 -3
@@ -136,15 +136,16 @@ void vma_mark_detached(struct vm_area_struct *vma)
  * Returns the vma on success, NULL on failure to lock and EAGAIN if vma got
  * detached.
  *
- * WARNING! The vma passed to this function cannot be used if the function
- * fails to lock it because in certain cases RCU lock is dropped and then
- * reacquired. Once RCU lock is dropped the vma can be concurently freed.
+ * IMPORTANT: RCU lock must be held upon entering the function, but upon error
+ *            IT IS RELEASED. The caller must handle this correctly.
  */

Actually maybe this is better replacing the original, as it is super clear and direct.

Quoted 96 lines
mm/mmap_lock.c +32 -27
 static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
 						    struct vm_area_struct *vma)
 {
+	struct mm_struct *other_mm;
 	int oldcnt;
 
+	RCU_LOCKDEP_WARN(!rcu_read_lock_held(), "no rcu lock held");
 	/*
 	 * Check before locking. A race might cause false locked result.
 	 * We can use READ_ONCE() for the mm_lock_seq here, and don't need
@@ -152,8 +153,10 @@ static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
 	 * we don't rely on for anything - the mm_lock_seq read against which we
 	 * need ordering is below.
 	 */
-	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
-		return NULL;
+	if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence)) {
+		vma = NULL;
+		goto err;
+	}
 
 	/*
 	 * If VMA_LOCK_OFFSET is set, __refcount_inc_not_zero_limited_acquire()
@@ -164,34 +167,14 @@ static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
 	if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt,
 							      VMA_REF_LIMIT))) {
 		/* return EAGAIN if vma got detached from under us */
-		return oldcnt ? NULL : ERR_PTR(-EAGAIN);
+		vma = oldcnt ? NULL : ERR_PTR(-EAGAIN);
+		goto err;
 	}
 
 	rwsem_acquire_read(&vma->vmlock_dep_map, 0, 1, _RET_IP_);
 
-	/*
-	 * If vma got attached to another mm from under us, that mm is not
-	 * stable and can be freed in the narrow window after vma->vm_refcnt
-	 * is dropped and before rcuwait_wake_up(mm) is called. Grab it before
-	 * releasing vma->vm_refcnt.
-	 */
-	if (unlikely(vma->vm_mm != mm)) {
-		/* Use a copy of vm_mm in case vma is freed after we drop vm_refcnt */
-		struct mm_struct *other_mm = vma->vm_mm;
-
-		/*
-		 * __mmdrop() is a heavy operation and we don't need RCU
-		 * protection here. Release RCU lock during these operations.
-		 * We reinstate the RCU read lock as the caller expects it to
-		 * be held when this function returns even on error.
-		 */
-		rcu_read_unlock();
-		mmgrab(other_mm);
-		vma_refcount_put(vma);
-		mmdrop(other_mm);
-		rcu_read_lock();
-		return NULL;
-	}
+	if (unlikely(vma->vm_mm != mm))
+		goto err_unstable;
 
 	/*
 	 * Overflow of vm_lock_seq/mm_lock_seq might produce false locked result.
@@ -206,10 +189,31 @@ static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
 	 */
 	if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) {
 		vma_refcount_put(vma);
-		return NULL;
+		vma = NULL;
+		goto err;
 	}
 
 	return vma;
+err:
+	rcu_read_unlock();
+
+	return vma;
+err_unstable:
+	/*
+	 * If vma got attached to another mm from under us, that mm is not
+	 * stable and can be freed in the narrow window after vma->vm_refcnt
+	 * is dropped and before rcuwait_wake_up(mm) is called. Grab it before
+	 * releasing vma->vm_refcnt.
+	 */
+	other_mm = vma->vm_mm; /* use a copy as vma can be freed after we drop vm_refcnt */
+
+	/* __mmdrop() is a heavy operation, do it after dropping RCU lock. */
+	rcu_read_unlock();
+	mmgrab(other_mm);
+	vma_refcount_put(vma);
+	mmdrop(other_mm);
+
+	return NULL;
 }
 
 /*

mm/mmap_lock.c struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm, · line 234

Quoted 10 lines
mm/mmap_lock.c +3 -2
@@ -223,11 +227,13 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 	MA_STATE(mas, &mm->mm_mt, address, address);
 	struct vm_area_struct *vma;
 
-	rcu_read_lock();
 retry:
+	rcu_read_lock();
 	vma = mas_walk(&mas);
-	if (!vma)
+	if (!vma) {
+		rcu_read_unlock();

Good :>)

Again, very easy thing to miss.

Quoted 58 lines
mm/mmap_lock.c +10 -8
 		goto inval;
+	}
 
 	vma = vma_start_read(mm, vma);
 	if (IS_ERR_OR_NULL(vma)) {
@@ -241,6 +247,9 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 		/* Failed to lock the VMA */
 		goto inval;
 	}
+
+	rcu_read_unlock();
+
 	/*
 	 * At this point, we have a stable reference to a VMA: The VMA is
 	 * locked and we know it hasn't already been isolated.
@@ -249,16 +258,14 @@ struct vm_area_struct *lock_vma_under_rcu(struct mm_struct *mm,
 	 */
 
 	/* Check if the vma we locked is the right one. */
-	if (unlikely(address < vma->vm_start || address >= vma->vm_end))
-		goto inval_end_read;
+	if (unlikely(address < vma->vm_start || address >= vma->vm_end)) {
+		vma_end_read(vma);
+		goto inval;
+	}
 
-	rcu_read_unlock();
 	return vma;
 
-inval_end_read:
-	vma_end_read(vma);
 inval:
-	rcu_read_unlock();
 	count_vm_vma_lock_event(VMA_LOCK_ABORT);
 	return NULL;
 }
@@ -313,6 +320,7 @@ struct vm_area_struct *lock_next_vma(struct mm_struct *mm,
 		 */
 		if (PTR_ERR(vma) == -EAGAIN) {
 			/* reset to search from the last address */
+			rcu_read_lock();
 			vma_iter_set(vmi, from_addr);
 			goto retry;
 		}
@@ -342,9 +350,9 @@ struct vm_area_struct *lock_next_vma(struct mm_struct *mm,
 	return vma;
 
 fallback_unlock:
+	rcu_read_unlock();
 	vma_end_read(vma);
 fallback:
-	rcu_read_unlock();
 	vma = lock_next_vma_under_mmap_lock(mm, vmi, from_addr);
 	rcu_read_lock();
 	/* Reinitialize the iterator after re-entering rcu read section */
--

2.50.1.552.g942d659e1b-goog

author maintainer Reviewed-by Lorenzo Stoakes on v2 1/2
#5 lore raw
In reply to Suren Baghdasaryan
Quoted 8 lines · Suren Baghdasaryan

Limit the scope of vma_start_read() as it is used only as a helper for higher-level locking functions implemented inside mmap_lock.c and we are about to introduce more complex RCU rules for this function. The change is pure code refactoring and has no functional changes.

Forgot to add Lorenzo's

Thanks!

Quoted 202 lines
---
 include/linux/mmap_lock.h | 85 ---------------------------------------
 mm/mmap_lock.c            | 85 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 85 insertions(+), 85 deletions(-)
include/linux/mmap_lock.h +0 -85
index 11a078de9150..2c9fffa58714 100644
@@ -147,91 +147,6 @@ static inline void vma_refcount_put(struct vm_area_struct *vma)
        }
 }
 
-/*
- * Try to read-lock a vma. The function is allowed to occasionally yield false
- * locked result to avoid performance overhead, in which case we fall back to
- * using mmap_lock. The function should never yield false unlocked result.
- * False locked result is possible if mm_lock_seq overflows or if vma gets
- * reused and attached to a different mm before we lock it.
- * Returns the vma on success, NULL on failure to lock and EAGAIN if vma got
- * detached.
- *
- * WARNING! The vma passed to this function cannot be used if the function
- * fails to lock it because in certain cases RCU lock is dropped and then
- * reacquired. Once RCU lock is dropped the vma can be concurently freed.
- */
-static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
-                                                   struct vm_area_struct *vma)
-{
-       int oldcnt;
-
-       /*
-        * Check before locking. A race might cause false locked result.
-        * We can use READ_ONCE() for the mm_lock_seq here, and don't need
-        * ACQUIRE semantics, because this is just a lockless check whose result
-        * we don't rely on for anything - the mm_lock_seq read against which we
-        * need ordering is below.
-        */
-       if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
-               return NULL;
-
-       /*
-        * If VMA_LOCK_OFFSET is set, __refcount_inc_not_zero_limited_acquire()
-        * will fail because VMA_REF_LIMIT is less than VMA_LOCK_OFFSET.
-        * Acquire fence is required here to avoid reordering against later
-        * vm_lock_seq check and checks inside lock_vma_under_rcu().
-        */
-       if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt,
-                                                             VMA_REF_LIMIT))) {
-               /* return EAGAIN if vma got detached from under us */
-               return oldcnt ? NULL : ERR_PTR(-EAGAIN);
-       }
-
-       rwsem_acquire_read(&vma->vmlock_dep_map, 0, 1, _RET_IP_);
-
-       /*
-        * If vma got attached to another mm from under us, that mm is not
-        * stable and can be freed in the narrow window after vma->vm_refcnt
-        * is dropped and before rcuwait_wake_up(mm) is called. Grab it before
-        * releasing vma->vm_refcnt.
-        */
-       if (unlikely(vma->vm_mm != mm)) {
-               /* Use a copy of vm_mm in case vma is freed after we drop vm_refcnt */
-               struct mm_struct *other_mm = vma->vm_mm;
-
-               /*
-                * __mmdrop() is a heavy operation and we don't need RCU
-                * protection here. Release RCU lock during these operations.
-                * We reinstate the RCU read lock as the caller expects it to
-                * be held when this function returns even on error.
-                */
-               rcu_read_unlock();
-               mmgrab(other_mm);
-               vma_refcount_put(vma);
-               mmdrop(other_mm);
-               rcu_read_lock();
-               return NULL;
-       }
-
-       /*
-        * Overflow of vm_lock_seq/mm_lock_seq might produce false locked result.
-        * False unlocked result is impossible because we modify and check
-        * vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq
-        * modification invalidates all existing locks.
-        *
-        * We must use ACQUIRE semantics for the mm_lock_seq so that if we are
-        * racing with vma_end_write_all(), we only start reading from the VMA
-        * after it has been unlocked.
-        * This pairs with RELEASE semantics in vma_end_write_all().
-        */
-       if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) {
-               vma_refcount_put(vma);
-               return NULL;
-       }
-
-       return vma;
-}
-
 /*
  * Use only while holding mmap read lock which guarantees that locking will not
  * fail (nobody can concurrently write-lock the vma). vma_start_read() should
mm/mmap_lock.c +85 -0
index b006cec8e6fe..10826f347a9f 100644
@@ -127,6 +127,91 @@ void vma_mark_detached(struct vm_area_struct *vma)
        }
 }
 
+/*
+ * Try to read-lock a vma. The function is allowed to occasionally yield false
+ * locked result to avoid performance overhead, in which case we fall back to
+ * using mmap_lock. The function should never yield false unlocked result.
+ * False locked result is possible if mm_lock_seq overflows or if vma gets
+ * reused and attached to a different mm before we lock it.
+ * Returns the vma on success, NULL on failure to lock and EAGAIN if vma got
+ * detached.
+ *
+ * WARNING! The vma passed to this function cannot be used if the function
+ * fails to lock it because in certain cases RCU lock is dropped and then
+ * reacquired. Once RCU lock is dropped the vma can be concurently freed.
+ */
+static inline struct vm_area_struct *vma_start_read(struct mm_struct *mm,
+                                                   struct vm_area_struct *vma)
+{
+       int oldcnt;
+
+       /*
+        * Check before locking. A race might cause false locked result.
+        * We can use READ_ONCE() for the mm_lock_seq here, and don't need
+        * ACQUIRE semantics, because this is just a lockless check whose result
+        * we don't rely on for anything - the mm_lock_seq read against which we
+        * need ordering is below.
+        */
+       if (READ_ONCE(vma->vm_lock_seq) == READ_ONCE(mm->mm_lock_seq.sequence))
+               return NULL;
+
+       /*
+        * If VMA_LOCK_OFFSET is set, __refcount_inc_not_zero_limited_acquire()
+        * will fail because VMA_REF_LIMIT is less than VMA_LOCK_OFFSET.
+        * Acquire fence is required here to avoid reordering against later
+        * vm_lock_seq check and checks inside lock_vma_under_rcu().
+        */
+       if (unlikely(!__refcount_inc_not_zero_limited_acquire(&vma->vm_refcnt, &oldcnt,
+                                                             VMA_REF_LIMIT))) {
+               /* return EAGAIN if vma got detached from under us */
+               return oldcnt ? NULL : ERR_PTR(-EAGAIN);
+       }
+
+       rwsem_acquire_read(&vma->vmlock_dep_map, 0, 1, _RET_IP_);
+
+       /*
+        * If vma got attached to another mm from under us, that mm is not
+        * stable and can be freed in the narrow window after vma->vm_refcnt
+        * is dropped and before rcuwait_wake_up(mm) is called. Grab it before
+        * releasing vma->vm_refcnt.
+        */
+       if (unlikely(vma->vm_mm != mm)) {
+               /* Use a copy of vm_mm in case vma is freed after we drop vm_refcnt */
+               struct mm_struct *other_mm = vma->vm_mm;
+
+               /*
+                * __mmdrop() is a heavy operation and we don't need RCU
+                * protection here. Release RCU lock during these operations.
+                * We reinstate the RCU read lock as the caller expects it to
+                * be held when this function returns even on error.
+                */
+               rcu_read_unlock();
+               mmgrab(other_mm);
+               vma_refcount_put(vma);
+               mmdrop(other_mm);
+               rcu_read_lock();
+               return NULL;
+       }
+
+       /*
+        * Overflow of vm_lock_seq/mm_lock_seq might produce false locked result.
+        * False unlocked result is impossible because we modify and check
+        * vma->vm_lock_seq under vma->vm_refcnt protection and mm->mm_lock_seq
+        * modification invalidates all existing locks.
+        *
+        * We must use ACQUIRE semantics for the mm_lock_seq so that if we are
+        * racing with vma_end_write_all(), we only start reading from the VMA
+        * after it has been unlocked.
+        * This pairs with RELEASE semantics in vma_end_write_all().
+        */
+       if (unlikely(vma->vm_lock_seq == raw_read_seqcount(&mm->mm_lock_seq))) {
+               vma_refcount_put(vma);
+               return NULL;
+       }
+
+       return vma;
+}
+
 /*
  * Lookup and lock a VMA under RCU protection. Returned VMA is guaranteed to be
  * stable and not isolated. If the VMA is not found or is being modified the

base-commit: 01da54f10fddf3b01c5a3b80f6b16bbad390c302 --
2.50.1.552.g942d659e1b-goog

#6 lore raw
Quoted 7 lines · Suren Baghdasaryan
Quoted 6 lines · Suren Baghdasaryan

Limit the scope of vma_start_read() as it is used only as a helper for higher-level locking functions implemented inside mmap_lock.c and we are about to introduce more complex RCU rules for this function. The change is pure code refactoring and has no functional changes.

Quoted 8 lines
Quoted 2 lines

Forgot to add Lorenzo's

Thanks!

:>)

Reviewed-by on v2 1/2
#7 lore raw
In reply to Suren Baghdasaryan
Quoted 7 lines · Suren Baghdasaryan

Limit the scope of vma_start_read() as it is used only as a helper for higher-level locking functions implemented inside mmap_lock.c and we are about to introduce more complex RCU rules for this function. The change is pure code refactoring and has no functional changes.