mm/userfaultfd: detect VMA type change after copy retry in mfill_copy_folio_retry()
Subsystems (2)
PATCH v5 1/1 mm/userfaultfd: detect VMA type change after copy retry in mfill_copy_folio_retry()
mfill_copy_folio_retry() drops mmap_lock for the copy_from_user() call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.
Pass the caller's ops into mfill_copy_folio_retry() and compare against the current vma_uffd_ops() after re-acquiring the lock. Return -EAGAIN if they differ so the operation can be retried.
--- mm/userfaultfd.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)
mm/userfaultfd.c +12 -2
@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } -static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +static int mfill_copy_folio_retry (struct mfill_state *state, +const struct vm_uffd_ops *ops, +struct folio *folio) {unsigned long src_addr = state->src_addr;void *kaddr;
@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folioif (err)return err; +/* +* The VMA type may have changed while the lock was dropped +* (e.g. replaced with a hugetlb mapping), making the caller's +* ops pointer stale. +*/ +if (vma_uffd_ops (state->vma) != ops) +return -EAGAIN ; + err =mfill_establish_pmd (state);if (err)return err;
@@ -495,7 +505,7 @@ static int __mfill_atomic_pte(struct mfill_state *state,* will take care of unlocking if needed. */ if (unlikely (ret)) { - ret =mfill_copy_folio_retry (state, folio); + ret =mfill_copy_folio_retry (state, ops, folio);if (ret) goto err_folio_put; }
Signature · 1 line
2.53.0
Quoted 11 lines · David Carlier
mfill_copy_folio_retry()dropsmmap_lockfor thecopy_from_user()call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.Pass the caller's ops into
mfill_copy_folio_retry()and compare against the currentvma_uffd_ops()after re-acquiring the lock. Return-EAGAINif they differ so the operation can be retried.Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")Signed-off-by David Carlier <[email protected]>
Quoted 46 lines
--- mm/userfaultfd.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)mm/userfaultfd.c +12
-2@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } -static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +static int mfill_copy_folio_retry (struct mfill_state *state, +const struct vm_uffd_ops *ops, +struct folio *folio) {unsigned long src_addr = state->src_addr;void *kaddr;@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folioif (err)return err; +/* +* The VMA type may have changed while the lock was dropped +* (e.g. replaced with a hugetlb mapping), making the caller's +* ops pointer stale. +*/ +if (vma_uffd_ops (state->vma) != ops) +return -EAGAIN ; + err =mfill_establish_pmd (state);if (err)return err;@@ -495,7 +505,7 @@ static int __mfill_atomic_pte(struct mfill_state *state,* will take care of unlocking if needed. */ if (unlikely (ret)) { - ret =mfill_copy_folio_retry (state, folio); + ret =mfill_copy_folio_retry (state, ops, folio);if (ret) goto err_folio_put; }Signature · 1 line
2.53.0
Signature · 2 lines
Sincerely yours, Mike.
Quoted 10 lines · David Carlier
mfill_copy_folio_retry()dropsmmap_lockfor thecopy_from_user()call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.Pass the caller's ops into
mfill_copy_folio_retry()and compare against the currentvma_uffd_ops()after re-acquiring the lock. Return-EAGAINif they differ so the operation can be retried.Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")
I don't have such sha1, is it a stale mm-unstable commit? Seems to be 4974a6aaa768 ("userfaultfd: mfill_atomic(): remove retry logic") now in mm-unstable (and can further change)
Quoted 44 lines
Signed-off-by David Carlier <[email protected]>--- mm/userfaultfd.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)mm/userfaultfd.c +12
-2@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } -static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +static int mfill_copy_folio_retry (struct mfill_state *state, +const struct vm_uffd_ops *ops, +struct folio *folio) {unsigned long src_addr = state->src_addr;void *kaddr;@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folioif (err)return err; +/* +* The VMA type may have changed while the lock was dropped +* (e.g. replaced with a hugetlb mapping), making the caller's +* ops pointer stale. +*/ +if (vma_uffd_ops (state->vma) != ops) +return -EAGAIN ; + err =mfill_establish_pmd (state);if (err)return err;@@ -495,7 +505,7 @@ static int __mfill_atomic_pte(struct mfill_state *state,* will take care of unlocking if needed. */ if (unlikely (ret)) { - ret =mfill_copy_folio_retry (state, folio); + ret =mfill_copy_folio_retry (state, ops, folio);if (ret) goto err_folio_put; }
Peter Xu
reviewerQuoted 31 lines · David Carlier
mfill_copy_folio_retry()dropsmmap_lockfor thecopy_from_user()call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.Pass the caller's ops into
mfill_copy_folio_retry()and compare against the currentvma_uffd_ops()after re-acquiring the lock. Return-EAGAINif they differ so the operation can be retried.Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")Signed-off-by David Carlier <[email protected]>--- mm/userfaultfd.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)mm/userfaultfd.c +3
-1@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } -static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +static int mfill_copy_folio_retry (struct mfill_state *state, +const struct vm_uffd_ops *ops, +struct folio *folio) {unsigned long src_addr = state->src_addr;void *kaddr;
mm/userfaultfd.c static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio · line 476
Quoted 10 lines
mm/userfaultfd.c +7
-0@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folioif (err)return err; +/* +* The VMA type may have changed while the lock was dropped +* (e.g. replaced with a hugetlb mapping), making the caller's +* ops pointer stale. +*/ +if (vma_uffd_ops (state->vma) != ops) +return -EAGAIN ;
I agree with -EAGAIN here, but we discussed over all the things on possible inode change and I don't know why we don't consider that.
I still think those should be considered.
If the vma snapshot idea is not welcomed, fine. We need to think of something to cover those too. Current patch won't cover "ops unchaged" but "inode changed", or offset changed, for example.
Thanks,
Quoted 16 lines
mm/userfaultfd.c +2
-1+ err =mfill_establish_pmd (state);if (err)return err;@@ -495,7 +505,7 @@ static int __mfill_atomic_pte(struct mfill_state *state,* will take care of unlocking if needed. */ if (unlikely (ret)) { - ret =mfill_copy_folio_retry (state, folio); + ret =mfill_copy_folio_retry (state, ops, folio);if (ret) goto err_folio_put; }Signature · 1 line
2.53.0
Signature · 1 line
Peter Xu
Andrew Morton
maintainerQuoted 10 lines · On Thu, 9 Apr 2026 19:04:49 +0200 "Vlastimil Babka (SUSE)" <[email protected]>…
Quoted 9 lines · David Carlier
mfill_copy_folio_retry()dropsmmap_lockfor thecopy_from_user()call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.Pass the caller's ops into
mfill_copy_folio_retry()and compare against the currentvma_uffd_ops()after re-acquiring the lock. Return-EAGAINif they differ so the operation can be retried.
Quoted 5 lines
Quoted 1 line
Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")I don't have such sha1, is it a stale mm-unstable commit? Seems to be
4974a6aaa768("userfaultfd:mfill_atomic(): remove retry logic") now in mm-unstable (and can further change)
Yup. I queued this as a squashable fix "userfaultfd-mfill_atomic-remove-retry-logic-fix.patch". Against Mike's "userfaultfd: mfill_atomic(): remove retry logic", queued for 2nd week of he merge window.
I'll remove that Fixes: tag - it changes every day and results in "invalid Fixes:" emails from Mark.
Mike Rapoport
maintainerQuoted 16 lines · Peter Xu
mm/userfaultfd.c static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio · line 476
Quoted 11 lines · David Carlier
mm/userfaultfd.c +7
-0@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folioif (err)return err; +/* +* The VMA type may have changed while the lock was dropped +* (e.g. replaced with a hugetlb mapping), making the caller's +* ops pointer stale. +*/ +if (vma_uffd_ops (state->vma) != ops) +return -EAGAIN ;I agree with
-EAGAINhere, but we discussed over all the things on possible inode change and I don't know why we don't consider that.
Quoted 5 lines
I still think those should be considered.
If the vma snapshot idea is not welcomed, fine. We need to think of something to cover those too. Current patch won't cover "ops unchaged" but "inode changed", or offset changed, for example.
This patch is enough to fix the regression introduced by my refactoring.
The inode/file/vma_snapshot checks are needed to solve the issue that existed roughly for a decade.
This should be a separate patch and it's really not urgent.
Quoted 4 lines
Thanks,
Signature · 1 line
Peter Xu
Signature · 2 lines
Sincerely yours, Mike.
Quoted 32 lines · On Thu, 9 Apr 2026 13:06:53 +0100 David Carlier <[email protected]> wrote:
mfill_copy_folio_retry()dropsmmap_lockfor thecopy_from_user()call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.Pass the caller's ops into
mfill_copy_folio_retry()and compare against the currentvma_uffd_ops()after re-acquiring the lock. Return-EAGAINif they differ so the operation can be retried.Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")Signed-off-by David Carlier <[email protected]>--- mm/userfaultfd.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)mm/userfaultfd.c +3
-1@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } -static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +static int mfill_copy_folio_retry (struct mfill_state *state, +const struct vm_uffd_ops *ops, +struct folio *folio) {unsigned long src_addr = state->src_addr;void *kaddr;@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folioif (err)
mm/userfaultfd.c static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio · line 477
Quoted 10 lines
mm/userfaultfd.c +8
-0return err; +/* +* The VMA type may have changed while the lock was dropped +* (e.g. replaced with a hugetlb mapping), making the caller's +* ops pointer stale. +*/ +if (vma_uffd_ops (state->vma) != ops) +return -EAGAIN ; +
hmm I am not sure if this is correct for shmem MAP_PRIVATE.
mfill_atomic_pte_copy() overrides ops to &anon_uffd_ops for MAP_PRIVATE mappings:
if (!(state->vma->vm_flags & VM_SHARED))
ops = &anon_uffd_ops;This overridden ops pointer propagates through __mfill_atomic_pte() into mfill_copy_folio_retry(). But the new check here calls vma_uffd_ops() which returns the original file-backed ops (e.g. &shmem_uffd_ops). For shmem MAP_PRIVATE VMAs, the comparison always fails even when the VMA type has not changed.
Maybe save the original (non-overridden) ops before the MAP_PRIVATE override and compare against that?
Mike Rapoport
maintainerHi Usama,
Quoted 50 lines · Usama Arif
Quoted 32 lines · On Thu, 9 Apr 2026 13:06:53 +0100 David Carlier <[email protected]> wrote:
mfill_copy_folio_retry()dropsmmap_lockfor thecopy_from_user()call. During this window, the VMA can be replaced with a different type (e.g. hugetlb), making the caller's ops pointer stale. Subsequent use of the stale ops can lead to incorrect folio handling or a kernel crash.Pass the caller's ops into
mfill_copy_folio_retry()and compare against the currentvma_uffd_ops()after re-acquiring the lock. Return-EAGAINif they differ so the operation can be retried.Fixes 59da5c32ffa3 ("userfaultfd: mfill_atomic(): remove retry logic")Signed-off-by David Carlier <[email protected]>--- mm/userfaultfd.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-)mm/userfaultfd.c +3
-1@@ -443,7 +443,9 @@ static int mfill_copy_folio_locked(struct folio *folio, unsigned long src_addr)return ret; } -static int mfill_copy_folio_retry (struct mfill_state *state,struct folio *folio) +static int mfill_copy_folio_retry (struct mfill_state *state, +const struct vm_uffd_ops *ops, +struct folio *folio) {unsigned long src_addr = state->src_addr;void *kaddr;@@ -465,6 +467,14 @@ static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folioif (err)mm/userfaultfd.c static int mfill_copy_folio_retry(struct mfill_state *state, struct folio *folio · line 477
Quoted 10 lines
mm/userfaultfd.c +8
-0return err; +/* +* The VMA type may have changed while the lock was dropped +* (e.g. replaced with a hugetlb mapping), making the caller's +* ops pointer stale. +*/ +if (vma_uffd_ops (state->vma) != ops) +return -EAGAIN ; +hmm I am not sure if this is correct for shmem
MAP_PRIVATE.
mfill_atomic_pte_copy()overrides ops to&anon_uffd_opsforMAP_PRIVATEmappings:
Quoted 8 lines
if (!(state->vma->vm_flags & VM_SHARED)) ops = &anon_uffd_ops;This overridden ops pointer propagates through
__mfill_atomic_pte()intomfill_copy_folio_retry(). But the new check here callsvma_uffd_ops()which returns the original file-backed ops (e.g.&shmem_uffd_ops). For shmemMAP_PRIVATEVMAs, the comparison always fails even when the VMA type has not changed.
Good catch.
@Andrew, can you please drop the patch for now?
Quoted 2 lines
Maybe save the original (non-overridden) ops before the
MAP_PRIVATEoverride and compare against that?
Signature · 2 lines
Sincerely yours, Mike.