Skip to contentSkip to conversation

mm/userfaultfd: detect VMA type change after copy retry in mfill_copy_folio_retry()

PATCH Superseded David Carlier 6 participants 8 messages Apr 9 – Apr 12

Subsystems (2)

v5 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

PATCH v5 1/1 mm/userfaultfd: detect VMA type change after copy retry in mfill_copy_folio_retry()

mfill_copy_folio_retry() drops mmap_lock for the copy_from_user() call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.

Pass the caller's ops into mfill_copy_folio_retry() and compare against the current vma_uffd_ops() after re-acquiring the lock. Return -EAGAIN if they differ so the operation can be retried.

Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")
---
 mm/userfaultfd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
mm/userfaultfd.c +12 -2
index 481ec7eb4442..214923a411c1 100644
@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
 	return ret;
 }
 
-static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio)
+static int mfill_copy_folio_retry(struct mfill_state *state,
+				  const struct vm_uffd_ops *ops,
+				  struct folio *folio)
 {
 	unsigned long src_addr = state->src_addr;
 	void *kaddr;
@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio
 	if (err)
 		return err;
 
+	/*
+	 * The VMA type may have changed while the lock was dropped
+	 * (e.g. replaced with a hugetlb mapping), making the caller's
+	 * ops pointer stale.
+	 */
+	if (vma_uffd_ops(state->vma) != ops)
+		return -EAGAIN;
+
 	err = mfill_establish_pmd(state);
 	if (err)
 		return err;
@@ -495,7 +505,7 @@ static int __mfill_atomic_pte(struct mfill_state *state,
 		 * will take care of unlocking if needed.
 		 */
 		if (unlikely(ret)) {
-			ret = mfill_copy_folio_retry(state, folio);
+			ret = mfill_copy_folio_retry(state, ops, folio);
 			if (ret)
 				goto err_folio_put;
 		}
Signature · 1 line
2.53.0
maintainer Acked-by on v5 1/1
#1 lore raw
Quoted 11 lines · David Carlier

mfill_copy_folio_retry() drops mmap_lock for the copy_from_user() call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.

Pass the caller's ops into mfill_copy_folio_retry() and compare against the current vma_uffd_ops() after re-acquiring the lock. Return -EAGAIN if they differ so the operation can be retried.

Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")
Quoted 46 lines
---
 mm/userfaultfd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
mm/userfaultfd.c +12 -2
index 481ec7eb4442..214923a411c1 100644
@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
 	return ret;
 }
 
-static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio)
+static int mfill_copy_folio_retry(struct mfill_state *state,
+				  const struct vm_uffd_ops *ops,
+				  struct folio *folio)
 {
 	unsigned long src_addr = state->src_addr;
 	void *kaddr;
@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio
 	if (err)
 		return err;
 
+	/*
+	 * The VMA type may have changed while the lock was dropped
+	 * (e.g. replaced with a hugetlb mapping), making the caller's
+	 * ops pointer stale.
+	 */
+	if (vma_uffd_ops(state->vma) != ops)
+		return -EAGAIN;
+
 	err = mfill_establish_pmd(state);
 	if (err)
 		return err;
@@ -495,7 +505,7 @@ static int __mfill_atomic_pte(struct mfill_state *state,
 		 * will take care of unlocking if needed.
 		 */
 		if (unlikely(ret)) {
-			ret = mfill_copy_folio_retry(state, folio);
+			ret = mfill_copy_folio_retry(state, ops, folio);
 			if (ret)
 				goto err_folio_put;
 		}
Signature · 1 line
2.53.0
Signature · 2 lines
Sincerely yours,
Mike.
#2 lore raw
In reply to David Carlier
Quoted 10 lines · David Carlier

mfill_copy_folio_retry() drops mmap_lock for the copy_from_user() call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.

Pass the caller's ops into mfill_copy_folio_retry() and compare against the current vma_uffd_ops() after re-acquiring the lock. Return -EAGAIN if they differ so the operation can be retried.

Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")

I don't have such sha1, is it a stale mm-unstable commit? Seems to be 4974a6aaa768 ("userfaultfd: mfill_atomic(): remove retry logic") now in mm-unstable (and can further change)

Quoted 44 lines
---
 mm/userfaultfd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
mm/userfaultfd.c +12 -2
index 481ec7eb4442..214923a411c1 100644
@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
 	return ret;
 }
 
-static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio)
+static int mfill_copy_folio_retry(struct mfill_state *state,
+				  const struct vm_uffd_ops *ops,
+				  struct folio *folio)
 {
 	unsigned long src_addr = state->src_addr;
 	void *kaddr;
@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio
 	if (err)
 		return err;
 
+	/*
+	 * The VMA type may have changed while the lock was dropped
+	 * (e.g. replaced with a hugetlb mapping), making the caller's
+	 * ops pointer stale.
+	 */
+	if (vma_uffd_ops(state->vma) != ops)
+		return -EAGAIN;
+
 	err = mfill_establish_pmd(state);
 	if (err)
 		return err;
@@ -495,7 +505,7 @@ static int __mfill_atomic_pte(struct mfill_state *state,
 		 * will take care of unlocking if needed.
 		 */
 		if (unlikely(ret)) {
-			ret = mfill_copy_folio_retry(state, folio);
+			ret = mfill_copy_folio_retry(state, ops, folio);
 			if (ret)
 				goto err_folio_put;
 		}
maintainer
#3 lore raw
Quoted 10 lines · On Thu, 9 Apr 2026 19:04:49 +0200 "Vlastimil Babka (SUSE)" <[email protected]>…
Quoted 9 lines · David Carlier

mfill_copy_folio_retry() drops mmap_lock for the copy_from_user() call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.

Pass the caller's ops into mfill_copy_folio_retry() and compare against the current vma_uffd_ops() after re-acquiring the lock. Return -EAGAIN if they differ so the operation can be retried.

Quoted 5 lines
Quoted 1 line
Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")

I don't have such sha1, is it a stale mm-unstable commit? Seems to be 4974a6aaa768 ("userfaultfd: mfill_atomic(): remove retry logic") now in mm-unstable (and can further change)

Yup. I queued this as a squashable fix "userfaultfd-mfill_atomic-remove-retry-logic-fix.patch". Against Mike's "userfaultfd: mfill_atomic(): remove retry logic", queued for 2nd week of he merge window.

I'll remove that Fixes: tag - it changes every day and results in "invalid Fixes:" emails from Mark.

reviewer
#4 lore raw
In reply to David Carlier
Quoted 31 lines · David Carlier

mfill_copy_folio_retry() drops mmap_lock for the copy_from_user() call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.

Pass the caller's ops into mfill_copy_folio_retry() and compare against the current vma_uffd_ops() after re-acquiring the lock. Return -EAGAIN if they differ so the operation can be retried.

Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")
---
 mm/userfaultfd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
mm/userfaultfd.c +3 -1
index 481ec7eb4442..214923a411c1 100644
@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
 	return ret;
 }
 
-static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio)
+static int mfill_copy_folio_retry(struct mfill_state *state,
+				  const struct vm_uffd_ops *ops,
+				  struct folio *folio)
 {
 	unsigned long src_addr = state->src_addr;
 	void *kaddr;

mm/userfaultfd.c static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio · line 476

Quoted 10 lines
mm/userfaultfd.c +7 -0
@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio
 	if (err)
 		return err;
 
+	/*
+	 * The VMA type may have changed while the lock was dropped
+	 * (e.g. replaced with a hugetlb mapping), making the caller's
+	 * ops pointer stale.
+	 */
+	if (vma_uffd_ops(state->vma) != ops)
+		return -EAGAIN;

I agree with -EAGAIN here, but we discussed over all the things on possible inode change and I don't know why we don't consider that.

I still think those should be considered.

If the vma snapshot idea is not welcomed, fine. We need to think of something to cover those too. Current patch won't cover "ops unchaged" but "inode changed", or offset changed, for example.

Thanks,

Quoted 16 lines
mm/userfaultfd.c +2 -1
+
 	err = mfill_establish_pmd(state);
 	if (err)
 		return err;
@@ -495,7 +505,7 @@ static int __mfill_atomic_pte(struct mfill_state *state,
 		 * will take care of unlocking if needed.
 		 */
 		if (unlikely(ret)) {
-			ret = mfill_copy_folio_retry(state, folio);
+			ret = mfill_copy_folio_retry(state, ops, folio);
 			if (ret)
 				goto err_folio_put;
 		}
Signature · 1 line
2.53.0
Signature · 1 line
Peter Xu
maintainer
#5 lore raw
Quoted 16 lines · Peter Xu

mm/userfaultfd.c static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio · line 476

Quoted 11 lines · David Carlier
mm/userfaultfd.c +7 -0
@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio
 	if (err)
 		return err;
 
+	/*
+	 * The VMA type may have changed while the lock was dropped
+	 * (e.g. replaced with a hugetlb mapping), making the caller's
+	 * ops pointer stale.
+	 */
+	if (vma_uffd_ops(state->vma) != ops)
+		return -EAGAIN;

I agree with -EAGAIN here, but we discussed over all the things on possible inode change and I don't know why we don't consider that.

Quoted 5 lines

I still think those should be considered.

If the vma snapshot idea is not welcomed, fine. We need to think of something to cover those too. Current patch won't cover "ops unchaged" but "inode changed", or offset changed, for example.

This patch is enough to fix the regression introduced by my refactoring.

The inode/file/vma_snapshot checks are needed to solve the issue that existed roughly for a decade.

This should be a separate patch and it's really not urgent.

Quoted 4 lines

Thanks,

Signature · 1 line
Peter Xu
Signature · 2 lines
Sincerely yours,
Mike.
#6 lore raw
In reply to David Carlier
Quoted 32 lines · On Thu, 9 Apr 2026 13:06:53 +0100 David Carlier <[email protected]> wrote:

mfill_copy_folio_retry() drops mmap_lock for the copy_from_user() call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.

Pass the caller's ops into mfill_copy_folio_retry() and compare against the current vma_uffd_ops() after re-acquiring the lock. Return -EAGAIN if they differ so the operation can be retried.

Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")
---
 mm/userfaultfd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
mm/userfaultfd.c +3 -1
index 481ec7eb4442..214923a411c1 100644
@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
 	return ret;
 }
 
-static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio)
+static int mfill_copy_folio_retry(struct mfill_state *state,
+				  const struct vm_uffd_ops *ops,
+				  struct folio *folio)
 {
 	unsigned long src_addr = state->src_addr;
 	void *kaddr;
@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio
 	if (err)

mm/userfaultfd.c static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio · line 477

Quoted 10 lines
mm/userfaultfd.c +8 -0
 		return err;
 
+	/*
+	 * The VMA type may have changed while the lock was dropped
+	 * (e.g. replaced with a hugetlb mapping), making the caller's
+	 * ops pointer stale.
+	 */
+	if (vma_uffd_ops(state->vma) != ops)
+		return -EAGAIN;
+

hmm I am not sure if this is correct for shmem MAP_PRIVATE.

mfill_atomic_pte_copy() overrides ops to &anon_uffd_ops for MAP_PRIVATE mappings:

    if (!(state->vma->vm_flags & VM_SHARED))
        ops = &anon_uffd_ops;

This overridden ops pointer propagates through __mfill_atomic_pte() into mfill_copy_folio_retry(). But the new check here calls vma_uffd_ops() which returns the original file-backed ops (e.g. &shmem_uffd_ops). For shmem MAP_PRIVATE VMAs, the comparison always fails even when the VMA type has not changed.

Maybe save the original (non-overridden) ops before the MAP_PRIVATE override and compare against that?

maintainer
#7 lore raw

Hi Usama,

Quoted 50 lines · Usama Arif
Quoted 32 lines · On Thu, 9 Apr 2026 13:06:53 +0100 David Carlier <[email protected]> wrote:

mfill_copy_folio_retry() drops mmap_lock for the copy_from_user() call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.

Pass the caller's ops into mfill_copy_folio_retry() and compare against the current vma_uffd_ops() after re-acquiring the lock. Return -EAGAIN if they differ so the operation can be retried.

Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")
---
 mm/userfaultfd.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)
mm/userfaultfd.c +3 -1
index 481ec7eb4442..214923a411c1 100644
@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)
 	return ret;
 }
 
-static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio)
+static int mfill_copy_folio_retry(struct mfill_state *state,
+				  const struct vm_uffd_ops *ops,
+				  struct folio *folio)
 {
 	unsigned long src_addr = state->src_addr;
 	void *kaddr;
@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio
 	if (err)

mm/userfaultfd.c static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio · line 477

Quoted 10 lines
mm/userfaultfd.c +8 -0
 		return err;
 
+	/*
+	 * The VMA type may have changed while the lock was dropped
+	 * (e.g. replaced with a hugetlb mapping), making the caller's
+	 * ops pointer stale.
+	 */
+	if (vma_uffd_ops(state->vma) != ops)
+		return -EAGAIN;
+

hmm I am not sure if this is correct for shmem MAP_PRIVATE.

mfill_atomic_pte_copy() overrides ops to &anon_uffd_ops for MAP_PRIVATE mappings:

Quoted 8 lines
    if (!(state->vma->vm_flags & VM_SHARED))
        ops = &anon_uffd_ops;

This overridden ops pointer propagates through __mfill_atomic_pte() into mfill_copy_folio_retry(). But the new check here calls vma_uffd_ops() which returns the original file-backed ops (e.g. &shmem_uffd_ops). For shmem MAP_PRIVATE VMAs, the comparison always fails even when the VMA type has not changed.

Good catch.

@Andrew, can you please drop the patch for now?

Quoted 2 lines

Maybe save the original (non-overridden) ops before the MAP_PRIVATE override and compare against that?

Signature · 2 lines
Sincerely yours,
Mike.