Skip to contentSkip to conversation

userfaultfd: assorted fixes (part 1)

PATCH Superseded Tal Zussman 7 participants 58 messages Nov 11 – Nov 16

Subsystems (11)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover userfaultfd: assorted fixes (part 1)

This is a first batch of userfaultfd fixes, with more to come.

These were found by a mix of regular old LLM review and by having an LLM write a TLA+ model of uffd interactions with PTE bits and other state and seeing what issues fell out.

Most of these fix places where mm code loses uffd-wp or RWP protection while moving, migrating or zapping an entry. Userspace then either misses writes and accesses, or gets pages reported that nobody touched. There are also two hangs (UFFDIO_MOVE and MADV_GUARD_INSTALL), and UFFDIO_ZEROPAGE modifying a file through a private mapping.

Patches 1-2:   UFFDIO_ZEROPAGE and UFFDIO_MOVE fixes, cc stable
Patches 3-5:   PAGEMAP_SCAN write-protect vs. migration and
               device-private entries
Patch 6:       UFFDIO_MOVE of the huge zero page from an RWP source
Patch 7:       mremap() losing uffd state when a move fails halfway
Patch 8:       migrate_vma replacing a uffd-protected zero page
Patches 9-11:  guard regions vs. uffd-wp
Patch 12:      UFFDIO_UNREGISTER with a misaligned hugetlb end

A few of the fixes change what userspace sees:

- UFFDIO_ZEROPAGE on a private file mapping no longer fails with -EEXIST where the file already has a page, on an mm that cannot use the zero page (patch 1)
- UFFDIO_MOVE returns -EEXIST for any huge destination PMD, not only for a present THP (patch 2)
- MADV_GUARD_INSTALL sends UFFD_EVENT_REMOVE, and can return -ENOMEM or -EINVAL if the VMA goes away or changes while the lock is dropped (patch 11)
- UFFDIO_UNREGISTER with a misaligned hugetlb end fails before unregistering anything (patch 12)

Patch 6 has little user impact on its own. It is included because, together with patch 7, it allows removing clear_uffd_wp_pmd().

Patch 7 adds spurious minor faults on shmem and hugetlb after mremap():
when dropping uffd protection there, change_protection() makes writable pages that were never protected read-only. UFFDIO_UNREGISTER and UFFDIO_WRITEPROTECT already behave this way and lead to the same spurious faults. This will be fixed in change_protection() in a separate series.

Based on mm-new.

---
Tal Zussman (12):
userfaultfd: use anonymous memory for UFFDIO_ZEROPAGE on private file mappings
userfaultfd: refuse UFFDIO_MOVE into a non-present huge destination PMD fs/proc/task_mmu: downgrade writable migration entries in PAGEMAP_SCAN write-protect mm/migrate: keep the uffd bit when restoring a writable migration entry fs/proc/task_mmu: write-protect device-private THPs in PAGEMAP_SCAN mm/huge_memory: build the moved huge zero PMD from the destination VMA mm/mremap: resolve uffd state when the context is dropped, not during the move mm/migrate_device: do not migrate a uffd-protected zero page mm/madvise: let a guard marker carry the uffd-wp bit
mm: set the uffd-wp bit on a guard marker when write-protecting mm/madvise: send UFFD_EVENT_REMOVE when installing guard regions mm/userfaultfd: check hugetlb end alignment in UFFDIO_UNREGISTER pre-scan

 fs/proc/task_mmu.c      | 64 +++++++++++++++++++++++++++++++++++++++++---
 include/linux/leafops.h | 18 +++++++++++++
 mm/huge_memory.c        | 31 +++-------------------
 mm/hugetlb.c            | 27 +------------------
 mm/madvise.c            | 62 +++++++++++++++++++++++++++++++++++--------
 mm/migrate.c            |  2 +-
 mm/migrate_device.c     | 23 ++++++++++++++--
 mm/mprotect.c           | 26 ++++++++++++++++--
 mm/mremap.c             | 46 +-------------------------------
 mm/userfaultfd.c        | 70 ++++++++++++++++++++++++-------------------------
 10 files changed, 215 insertions(+), 154 deletions(-)

---
base-commit: 2dc0136b1fdac153434ab02778fcdb32db5d0dae
change-id: 20261106-uffd-state-fixes-d2111e2abd48

Best regards,

Signature · 1 line
Tal Zussman <[email protected]>
#1 lore raw

PATCH 1/12 userfaultfd: use anonymous memory for UFFDIO_ZEROPAGE on private file mappings

If the mm cannot map the shared zero page, UFFDIO_ZEROPAGE falls back to mfill_atomic_pte_zeroed_folio(), which allocates a zeroed folio and installs it through vma_uffd_ops(), no matter what kind of mapping it is. An s390 KVM guest with storage keys or protected virtualization is such an mm.

For a MAP_PRIVATE shmem mapping, that adds the zeroed folio to the page cache of the file. That is wrong: a private mapping must never modify the file it maps, and the file might even be mapped from a read-only fd.

The ioctl can also fail where it should succeed. A missing fault is only delivered for a hole in the file, but UFFDIO_ZEROPAGE can be called on an address that did not fault, or can race with a write to the file. If the file has a page at that offset by then, inserting the zeroed folio fails and the ioctl returns -EEXIST although nothing is mapped at the destination address.

What we want is what UFFDIO_ZEROPAGE does on a private mapping when the shared zero page can be used, and what UFFDIO_COPY does: only check that nothing is mapped at the destination address, map zeroed memory there, and leave the page cache alone. The zeroed folio path did that before it was moved to vm_uffd_ops.

To fix it, pick the ops in the common folio path, as mfill_atomic_pte_copy() did for UFFDIO_COPY: anon_uffd_ops for a private mapping, the ops of the VMA otherwise. That makes mfill_atomic_pte_copy() and mfill_atomic_pte_zeroed_folio() redundant, so remove them and rename __mfill_atomic_pte() to mfill_atomic_pte_folio().

UFFDIO_ZEROPAGE no longer modifies the file, and only fails with -EEXIST if something is already mapped at the destination address.

Fixes f74991b4e383 ("shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops")
---
 mm/userfaultfd.c | 46 ++++++++++++++++------------------------------
 1 file changed, 16 insertions(+), 30 deletions(-)
mm/userfaultfd.c +16 -30
index d65184d9a480..61a712e1f9f5 100644
@@ -531,15 +531,27 @@ static int mfill_copy_folio_retry(struct mfill_state *mfill_state,
 	return 0;
 }
 
-static int __mfill_atomic_pte(struct mfill_state *state,
-			      const struct vm_uffd_ops *ops)
+static int mfill_atomic_pte_folio(struct mfill_state *state)
 {
+	const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma);
 	unsigned long dst_addr = state->dst_addr;
 	unsigned long src_addr = state->src_addr;
 	uffd_flags_t flags = state->flags;
 	struct folio *folio;
 	int ret;
 
+	/*
+	 * The normal page fault path for a MAP_PRIVATE mapping in a
+	 * file-backed VMA will invoke the fault, fill the hole in the file and
+	 * COW it right away. The result generates plain anonymous memory.
+	 * So when we are asked to fill a hole in a MAP_PRIVATE mapping, we'll
+	 * generate anonymous memory directly without actually filling the
+	 * hole. For the MAP_PRIVATE case the robustness check only happens in
+	 * the pagetable (to verify it's still none) and not in the page cache.
+	 */
+	if (!(state->vma->vm_flags & VM_SHARED))
+		ops = &anon_uffd_ops;
+
 	if (!ops) {
 		VM_WARN_ONCE(1, "UFFDIO_COPY for unsupported VMA");
 		return -EOPNOTSUPP;
@@ -597,32 +609,6 @@ static int __mfill_atomic_pte(struct mfill_state *state,
 	return ret;
 }
 
-static int mfill_atomic_pte_copy(struct mfill_state *state)
-{
-	const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma);
-
-	/*
-	 * The normal page fault path for a MAP_PRIVATE mapping in a
-	 * file-backed VMA will invoke the fault, fill the hole in the file and
-	 * COW it right away. The result generates plain anonymous memory.
-	 * So when we are asked to fill a hole in a MAP_PRIVATE mapping, we'll
-	 * generate anonymous memory directly without actually filling the
-	 * hole. For the MAP_PRIVATE case the robustness check only happens in
-	 * the pagetable (to verify it's still none) and not in the page cache.
-	 */
-	if (!(state->vma->vm_flags & VM_SHARED))
-		ops = &anon_uffd_ops;
-
-	return __mfill_atomic_pte(state, ops);
-}
-
-static int mfill_atomic_pte_zeroed_folio(struct mfill_state *state)
-{
-	const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma);
-
-	return __mfill_atomic_pte(state, ops);
-}
-
 static int mfill_atomic_pte_zeropage(struct mfill_state *state)
 {
 	struct vm_area_struct *dst_vma = state->vma;
@@ -634,7 +620,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)
 
 	if (mm_forbids_zeropage(dst_vma->vm_mm) ||
 	    (dst_vma->vm_flags & VM_SHARED))
-		return mfill_atomic_pte_zeroed_folio(state);
+		return mfill_atomic_pte_folio(state);
 
 	_dst_pte = pte_mkspecial(pfn_pte(zero_pfn(dst_addr),
 					 dst_vma->vm_page_prot));
@@ -927,7 +913,7 @@ static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state)
 	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON))
 		return mfill_atomic_pte_poison(state);
 	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY))
-		return mfill_atomic_pte_copy(state);
+		return mfill_atomic_pte_folio(state);
 	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE))
 		return mfill_atomic_pte_zeropage(state);
 
Signature · 1 line
2.39.5
maintainer Acked-by on v1 1/12
#2 lore raw

Re: [PATCH 01/12] userfaultfd: use anonymous memory for UFFDIO_ZEROPAGE on private file mappings

Quoted 30 lines · Tal Zussman

If the mm cannot map the shared zero page, UFFDIO_ZEROPAGE falls back to mfill_atomic_pte_zeroed_folio(), which allocates a zeroed folio and installs it through vma_uffd_ops(), no matter what kind of mapping it is. An s390 KVM guest with storage keys or protected virtualization is such an mm.

For a MAP_PRIVATE shmem mapping, that adds the zeroed folio to the page cache of the file. That is wrong: a private mapping must never modify the file it maps, and the file might even be mapped from a read-only fd.

The ioctl can also fail where it should succeed. A missing fault is only delivered for a hole in the file, but UFFDIO_ZEROPAGE can be called on an address that did not fault, or can race with a write to the file. If the file has a page at that offset by then, inserting the zeroed folio fails and the ioctl returns -EEXIST although nothing is mapped at the destination address.

What we want is what UFFDIO_ZEROPAGE does on a private mapping when the shared zero page can be used, and what UFFDIO_COPY does: only check that nothing is mapped at the destination address, map zeroed memory there, and leave the page cache alone. The zeroed folio path did that before it was moved to vm_uffd_ops.

To fix it, pick the ops in the common folio path, as mfill_atomic_pte_copy() did for UFFDIO_COPY: anon_uffd_ops for a private mapping, the ops of the VMA otherwise. That makes mfill_atomic_pte_copy() and mfill_atomic_pte_zeroed_folio() redundant, so remove them and rename __mfill_atomic_pte() to mfill_atomic_pte_folio().

Quoted 6 lines

UFFDIO_ZEROPAGE no longer modifies the file, and only fails with -EEXIST if something is already mapped at the destination address.

Fixes f74991b4e383 ("shmem, userfaultfd: implement shmem uffd operations using vm_uffd_ops")

Looks sensible (also another cleanup in disguise which I like ;):

Quoted 91 lines
---
 mm/userfaultfd.c | 46 ++++++++++++++++------------------------------
 1 file changed, 16 insertions(+), 30 deletions(-)
mm/userfaultfd.c +16 -31
index d65184d9a480..61a712e1f9f5 100644
@@ -531,15 +531,27 @@ static int mfill_copy_folio_retry(struct mfill_state *mfill_state,
 	return 0;
 }
 
-static int __mfill_atomic_pte(struct mfill_state *state,
-			      const struct vm_uffd_ops *ops)
+static int mfill_atomic_pte_folio(struct mfill_state *state)
 {
+	const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma);
 	unsigned long dst_addr = state->dst_addr;
 	unsigned long src_addr = state->src_addr;
 	uffd_flags_t flags = state->flags;
 	struct folio *folio;
 	int ret;
 
+	/*
+	 * The normal page fault path for a MAP_PRIVATE mapping in a
+	 * file-backed VMA will invoke the fault, fill the hole in the file and
+	 * COW it right away. The result generates plain anonymous memory.
+	 * So when we are asked to fill a hole in a MAP_PRIVATE mapping, we'll
+	 * generate anonymous memory directly without actually filling the
+	 * hole. For the MAP_PRIVATE case the robustness check only happens in
+	 * the pagetable (to verify it's still none) and not in the page cache.
+	 */
+	if (!(state->vma->vm_flags & VM_SHARED))
+		ops = &anon_uffd_ops;
+
 	if (!ops) {
 		VM_WARN_ONCE(1, "UFFDIO_COPY for unsupported VMA");
 		return -EOPNOTSUPP;
@@ -597,32 +609,6 @@ static int __mfill_atomic_pte(struct mfill_state *state,
 	return ret;
 }
 
-static int mfill_atomic_pte_copy(struct mfill_state *state)
-{
-	const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma);
-
-	/*
-	 * The normal page fault path for a MAP_PRIVATE mapping in a
-	 * file-backed VMA will invoke the fault, fill the hole in the file and
-	 * COW it right away. The result generates plain anonymous memory.
-	 * So when we are asked to fill a hole in a MAP_PRIVATE mapping, we'll
-	 * generate anonymous memory directly without actually filling the
-	 * hole. For the MAP_PRIVATE case the robustness check only happens in
-	 * the pagetable (to verify it's still none) and not in the page cache.
-	 */
-	if (!(state->vma->vm_flags & VM_SHARED))
-		ops = &anon_uffd_ops;
-
-	return __mfill_atomic_pte(state, ops);
-}
-
-static int mfill_atomic_pte_zeroed_folio(struct mfill_state *state)
-{
-	const struct vm_uffd_ops *ops = vma_uffd_ops(state->vma);
-
-	return __mfill_atomic_pte(state, ops);
-}
-
 static int mfill_atomic_pte_zeropage(struct mfill_state *state)
 {
 	struct vm_area_struct *dst_vma = state->vma;
@@ -634,7 +620,7 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)
 
 	if (mm_forbids_zeropage(dst_vma->vm_mm) ||
 	    (dst_vma->vm_flags & VM_SHARED))
-		return mfill_atomic_pte_zeroed_folio(state);
+		return mfill_atomic_pte_folio(state);
 
 	_dst_pte = pte_mkspecial(pfn_pte(zero_pfn(dst_addr),
 					 dst_vma->vm_page_prot));
@@ -927,7 +913,7 @@ static __always_inline ssize_t mfill_atomic_pte(struct mfill_state *state)
 	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_POISON))
 		return mfill_atomic_pte_poison(state);
 	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_COPY))
-		return mfill_atomic_pte_copy(state);
+		return mfill_atomic_pte_folio(state);
 	if (uffd_flags_mode_is(flags, MFILL_ATOMIC_ZEROPAGE))
 		return mfill_atomic_pte_zeropage(state);
 
 
--
Quoted 2 lines

2.39.5

--
Cheers, Lorenzo

#3 lore raw

PATCH 2/12 userfaultfd: refuse UFFDIO_MOVE into a non-present huge destination PMD

In reply to Tal Zussman

move_pages() refuses to move into a destination PMD that already maps a THP, but checks for that with pmd_trans_huge(), which is only true for a present THP. A device-private THP or a THP under migration is a non-present huge PMD leaf and is not refused.

The destination PMD is then treated as if it pointed to a PTE table. It is not empty, so pte_alloc() leaves it alone, and move_pages_ptes() fails to map a PTE table from it and returns -EAGAIN. move_pages() retries on -EAGAIN, with both VMA read locks held. If the source is a THP, it is split first for nothing.

For a THP under migration, the loop ends once the migration finishes. A device-private THP can remain in device memory indefinitely, so the caller of UFFDIO_MOVE hangs until it receives a fatal signal, and anything in the mm that needs to write-lock one of the VMAs blocks along with it. This requires a driver that migrates THPs to device memory.

To fix it, check the destination with pmd_is_huge(), which is true for any huge leaf, present or not. UFFDIO_MOVE now fails with -EEXIST in these cases, as it already does for a present THP.

Fixes adef440691ba ("userfaultfd: UFFDIO_MOVE uABI")
Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")
---
 mm/userfaultfd.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
mm/userfaultfd.c +6 -5
index 61a712e1f9f5..ad9d243dc64b 100644
@@ -2021,12 +2021,13 @@ static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 
 		dst_pmdval = pmdp_get_lockless(dst_pmd);
 		/*
-		 * If the dst_pmd is mapped as THP don't override it and just
-		 * be strict. If dst_pmd changes into TPH after this check, the
-		 * move_pages_huge_pmd() will detect the change and retry
-		 * while move_pages_pte() will detect the change and fail.
+		 * If the dst_pmd is a huge leaf, present or not, don't
+		 * override it and just be strict. If dst_pmd changes into a
+		 * leaf after this check, move_pages_huge_pmd() will detect the
+		 * change and retry while move_pages_ptes() will fail to map
+		 * the PTE table and the next iteration will see the leaf.
 		 */
-		if (unlikely(pmd_trans_huge(dst_pmdval))) {
+		if (unlikely(pmd_is_huge(dst_pmdval))) {
 			err = -EEXIST;
 			break;
 		}
Signature · 1 line
2.39.5
#4 lore raw

Re: [PATCH 02/12] userfaultfd: refuse UFFDIO_MOVE into a non-present huge destination PMD

Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce: - [Medium] userfaultfd: spurious -EFAULT for WP…

Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce:
- [Medium] userfaultfd: spurious -EFAULT for WP markers in move_pages_ptes() [1]
--

commit cae4a6b3bdb227f2a82bf72d15644fbdf0a7b4b5
Author: Tal Zussman <[email protected]>

userfaultfd: refuse UFFDIO_MOVE into a non-present huge destination PMD

This commit changes move_pages() to check the destination PMD with pmd_is_huge() instead of pmd_trans_huge() so that UFFDIO_MOVE returns -EEXIST for non-present huge PMD leaves such as device-private THPs or THPs under migration.

Quoted 13 lines
mm/userfaultfd.c +1 -4
index 61a712e1f9f53..ad9d243dc64b0 100644
@@ -2021,12 +2021,13 @@ static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 
 		dst_pmdval = pmdp_get_lockless(dst_pmd);
 		/*
-		 * If the dst_pmd is mapped as THP don't override it and just
-		 * be strict. If dst_pmd changes into TPH after this check, the
-		 * move_pages_huge_pmd() will detect the change and retry
-		 * while move_pages_pte() will detect the change and fail.
+		 * If the dst_pmd is a huge leaf, present or not, don't

mm/userfaultfd.c static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start, · line 2033

Quoted 10 lines
mm/userfaultfd.c +5 -1
+		 * override it and just be strict. If dst_pmd changes into a
+		 * leaf after this check, move_pages_huge_pmd() will detect the
+		 * change and retry while move_pages_ptes() will fail to map
+		 * the PTE table and the next iteration will see the leaf.
 		 */
-		if (unlikely(pmd_trans_huge(dst_pmdval))) {
+		if (unlikely(pmd_is_huge(dst_pmdval))) {
 			err = -EEXIST;
 			break;
 		}

[Severity: Medium]
This problem wasn't introduced by this patch, but does move_pages_ptes() return a spurious -EFAULT [1] when UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES encounters an unpopulated PTE carrying a UFFD write-protect marker?

mm/userfaultfd.c:move_pages_ptes() {
	...
	if (pte_none(orig_src_pte)) {
		if (!(mode & UFFDIO_MOVE_MODE_ALLOW_SRC_HOLES))
			ret = -ENOENT;
		else /* nothing to do to move a hole */
			ret = PAGE_SIZE;
		goto out;
	}
	...
	if (pte_present(orig_src_pte)) {
		...
	} else { /* !pte_present() */
		struct folio *folio = NULL;
		const softleaf_t entry = softleaf_from_pte(orig_src_pte);

		if (softleaf_is_migration(entry)) {
			...
		} else if (!softleaf_is_swap(entry)) {
			ret = -EFAULT;
			goto out;
		}
	...
}

Because a UFFD WP marker is a non-zero PTE entry, pte_none(orig_src_pte) is false and execution falls through to the non-present branch. PTE markers are not swap entries, so softleaf_is_swap(entry) is also false and move_pages_ptes() aborts the move with -EFAULT instead of handling the write-protected hole.

[1] https://sashiko.dev/#/bug/linux-5f33b673-0c7c-4b59-b85e-71d45e428896

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
maintainer Acked-by on v1 2/12
#5 lore raw

Re: [PATCH 02/12] userfaultfd: refuse UFFDIO_MOVE into a non-present huge destination PMD

In reply to Tal Zussman
Quoted 4 lines · Tal Zussman

move_pages() refuses to move into a destination PMD that already maps a THP, but checks for that with pmd_trans_huge(), which is only true for a present THP. A device-private THP or a THP under migration is a non-present huge PMD leaf and is not refused.

non-present huge PMD softleaf :)

Quoted 13 lines

The destination PMD is then treated as if it pointed to a PTE table. It is not empty, so pte_alloc() leaves it alone, and move_pages_ptes() fails to map a PTE table from it and returns -EAGAIN. move_pages() retries on -EAGAIN, with both VMA read locks held. If the source is a THP, it is split first for nothing.

For a THP under migration, the loop ends once the migration finishes. A device-private THP can remain in device memory indefinitely, so the caller of UFFDIO_MOVE hangs until it receives a fatal signal, and anything in the mm that needs to write-lock one of the VMAs blocks along with it. This requires a driver that migrates THPs to device memory.

Quoted 8 lines

To fix it, check the destination with pmd_is_huge(), which is true for any huge leaf, present or not. UFFDIO_MOVE now fails with -EEXIST in these cases, as it already does for a present THP.

Fixes adef440691ba ("userfaultfd: UFFDIO_MOVE uABI")
Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")

Looks sensible to me so:

Quoted 29 lines
---
 mm/userfaultfd.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)
mm/userfaultfd.c +6 -6
index 61a712e1f9f5..ad9d243dc64b 100644
@@ -2021,12 +2021,13 @@ static ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
 
 		dst_pmdval = pmdp_get_lockless(dst_pmd);
 		/*
-		 * If the dst_pmd is mapped as THP don't override it and just
-		 * be strict. If dst_pmd changes into TPH after this check, the
-		 * move_pages_huge_pmd() will detect the change and retry
-		 * while move_pages_pte() will detect the change and fail.
+		 * If the dst_pmd is a huge leaf, present or not, don't
+		 * override it and just be strict. If dst_pmd changes into a
+		 * leaf after this check, move_pages_huge_pmd() will detect the
+		 * change and retry while move_pages_ptes() will fail to map
+		 * the PTE table and the next iteration will see the leaf.
 		 */
-		if (unlikely(pmd_trans_huge(dst_pmdval))) {
+		if (unlikely(pmd_is_huge(dst_pmdval))) {
 			err = -EEXIST;
 			break;
 		}
 
--
Quoted 2 lines

2.39.5

--
Cheers, Lorenzo

reviewer Tested-by Reviewed-by on v1 2/12
#6 lore raw

Re: [PATCH 02/12] userfaultfd: refuse UFFDIO_MOVE into a non-present huge destination PMD

In reply to Tal Zussman
Quoted 17 lines · Tal Zussman

move_pages() refuses to move into a destination PMD that already maps a THP, but checks for that with pmd_trans_huge(), which is only true for a present THP. A device-private THP or a THP under migration is a non-present huge PMD leaf and is not refused.

The destination PMD is then treated as if it pointed to a PTE table. It is not empty, so pte_alloc() leaves it alone, and move_pages_ptes() fails to map a PTE table from it and returns -EAGAIN. move_pages() retries on -EAGAIN, with both VMA read locks held. If the source is a THP, it is split first for nothing.

For a THP under migration, the loop ends once the migration finishes. A device-private THP can remain in device memory indefinitely, so the caller of UFFDIO_MOVE hangs until it receives a fatal signal, and anything in the mm that needs to write-lock one of the VMAs blocks along with it. This requires a driver that migrates THPs to device memory.

Quoted 8 lines

To fix it, check the destination with pmd_is_huge(), which is true for any huge leaf, present or not. UFFDIO_MOVE now fails with -EEXIST in these cases, as it already does for a present THP.

Fixes adef440691ba ("userfaultfd: UFFDIO_MOVE uABI")
Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")
Signature · 1 line
~Gregory
#7 lore raw

PATCH 3/12 fs/proc/task_mmu: downgrade writable migration entries in PAGEMAP_SCAN write-protect

In reply to Tal Zussman

PAGEMAP_SCAN with PM_SCAN_WP_MATCHING write-protects pages in a VMA registered for async uffd-wp, and sets the uffd bit on non-present entries as they are. An entry with the uffd bit set must not be writable, so that the page is write-protected once it is mapped again. change_protection() therefore downgrades writable migration and device-private entries to readable ones first, but PAGEMAP_SCAN does not.

For a writable migration entry at the PTE level, remove_migration_pte() restores a writable PTE without the uffd bit. So if a page is under migration while PAGEMAP_SCAN write-protects it, the write protection is lost. A user tracking writes with async uffd-wp sees the page reported as written on the next scan although nothing wrote to it.

With CONFIG_PAGE_TABLE_CHECK, setting the uffd bit on a writable non-present PTE or PMD also triggers a warning.

To fix it, downgrade writable migration and device-private entries before setting the uffd bit, as change_protection() does, at the PTE, PMD, and hugetlb levels.

Fixes 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs")
---
 fs/proc/task_mmu.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 51 insertions(+), 3 deletions(-)
fs/proc/task_mmu.c +51 -3
index 22361607020d..53780a454432 100644
@@ -2395,6 +2395,38 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
 	return categories;
 }
 
+/* Downgrade a writable migration or device-private entry to a readable one. */
+static softleaf_t make_readable_softleaf(softleaf_t entry)
+{
+	if (softleaf_is_device_private_write(entry))
+		return make_readable_device_private_entry(swp_offset(entry));
+	if (!softleaf_is_migration_write(entry))
+		return entry;
+	if (folio_test_anon(softleaf_to_folio(entry)))
+		return make_readable_exclusive_migration_entry(swp_offset(entry));
+	return make_readable_migration_entry(swp_offset(entry));
+}
+
+/*
+ * An armed entry must not come back writable, so downgrade a writable
+ * migration or device-private entry before giving it the bit, as
+ * change_softleaf_pte() does.
+ */
+static pte_t set_uffd_wp_swp_pte(pte_t pte)
+{
+	softleaf_t entry = softleaf_from_pte(pte);
+	pte_t newpte;
+
+	if (!softleaf_is_migration_write(entry) &&
+	    !softleaf_is_device_private_write(entry))
+		return pte_swp_mkuffd(pte);
+
+	newpte = softleaf_to_pte(make_readable_softleaf(entry));
+	if (pte_swp_soft_dirty(pte))
+		newpte = pte_swp_mksoft_dirty(newpte);
+	return pte_swp_mkuffd(newpte);
+}
+
 static void make_uffd_wp_pte(struct vm_area_struct *vma,
 			     unsigned long addr, pte_t *pte, pte_t ptent)
 {
@@ -2408,7 +2440,7 @@ static void make_uffd_wp_pte(struct vm_area_struct *vma,
 		set_pte_at(vma->vm_mm, addr, pte,
 			   make_pte_marker(PTE_MARKER_UFFD_WP));
 	} else {
-		ptent = pte_swp_mkuffd(ptent);
+		ptent = set_uffd_wp_swp_pte(ptent);
 		set_pte_at(vma->vm_mm, addr, pte, ptent);
 	}
 }
@@ -2467,6 +2499,22 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
 	return categories;
 }
 
+/* Same as set_uffd_wp_swp_pte(), for a non-present PMD. */
+static pmd_t set_uffd_wp_swp_pmd(pmd_t pmd)
+{
+	softleaf_t entry = softleaf_from_pmd(pmd);
+	pmd_t newpmd;
+
+	if (!softleaf_is_migration_write(entry) &&
+	    !softleaf_is_device_private_write(entry))
+		return pmd_swp_mkuffd(pmd);
+
+	newpmd = softleaf_to_pmd(make_readable_softleaf(entry));
+	if (pmd_swp_soft_dirty(pmd))
+		newpmd = pmd_swp_mksoft_dirty(newpmd);
+	return pmd_swp_mkuffd(newpmd);
+}
+
 static void make_uffd_wp_pmd(struct vm_area_struct *vma,
 			     unsigned long addr, pmd_t *pmdp)
 {
@@ -2477,7 +2525,7 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma,
 		pmd = pmd_mkuffd(old);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
 	} else if (pmd_is_migration_entry(pmd)) {
-		pmd = pmd_swp_mkuffd(pmd);
+		pmd = set_uffd_wp_swp_pmd(pmd);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
 	}
 }
@@ -2547,7 +2595,7 @@ static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
 
 	if (softleaf_is_migration(entry)) {
 		set_huge_pte_at(vma->vm_mm, addr, ptep,
-				pte_swp_mkuffd(ptent), psize);
+				set_uffd_wp_swp_pte(ptent), psize);
 	} else {
 		pte_t old_pte, new_pte;
 
Signature · 1 line
2.39.5
#8 lore raw

Re: [PATCH 03/12] fs/proc/task_mmu: downgrade writable migration entries in PAGEMAP_SCAN write-protect

Quoted 34 lines · Tal Zussman

PAGEMAP_SCAN with PM_SCAN_WP_MATCHING write-protects pages in a VMA registered for async uffd-wp, and sets the uffd bit on non-present entries as they are. An entry with the uffd bit set must not be writable, so that the page is write-protected once it is mapped again. change_protection() therefore downgrades writable migration and device-private entries to readable ones first, but PAGEMAP_SCAN does not.

For a writable migration entry at the PTE level, remove_migration_pte() restores a writable PTE without the uffd bit. So if a page is under migration while PAGEMAP_SCAN write-protects it, the write protection is lost. A user tracking writes with async uffd-wp sees the page reported as written on the next scan although nothing wrote to it.

With CONFIG_PAGE_TABLE_CHECK, setting the uffd bit on a writable non-present PTE or PMD also triggers a warning.

To fix it, downgrade writable migration and device-private entries before setting the uffd bit, as change_protection() does, at the PTE, PMD, and hugetlb levels.

Fixes 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs")
---
 fs/proc/task_mmu.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 51 insertions(+), 3 deletions(-)
fs/proc/task_mmu.c +0 -0
index 22361607020d..53780a454432 100644
@@ -2395,6 +2395,38 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
 	return categories;

fs/proc/task_mmu.c static unsigned long pagemap_page_category(struct pagemap_scan_private *p, · line 2405

Quoted 10 lines
fs/proc/task_mmu.c +8 -0
 }
 
+/* Downgrade a writable migration or device-private entry to a readable one. */
+static softleaf_t make_readable_softleaf(softleaf_t entry)
+{
+	if (softleaf_is_device_private_write(entry))
+		return make_readable_device_private_entry(swp_offset(entry));
+	if (!softleaf_is_migration_write(entry))
+		return entry;
+	if (folio_test_anon(softleaf_to_folio(entry)))

Hmm _seems_ a bit dubious. You identify a readable exclusive migration entry by checking whether its folio is anon?

But I guess because a writable migration entry can only come from a writable PTE, that implies anonexclusive. But you do need to explicitly say that.

Also we really do have to break this out as a helper. I see it's open coded in change_softleaf_pte(), change_non_present_huge_pmd() and hugetlb_change_protection().

But maybe that can be a follow up...

fs/proc/task_mmu.c static unsigned long pagemap_page_category(struct pagemap_scan_private *p, · line 2407

Quoted 2 lines
fs/proc/task_mmu.c +2 -0
+		return make_readable_exclusive_migration_entry(swp_offset(entry));
+	return make_readable_migration_entry(swp_offset(entry));

This is all super compressed.

I'd say add to leafops.h but break it up:

	static inline softleaf_t softleaf_migration_mkreadable(softleaf_t entry)
	{
		const pgoff_t offset = swp_offset(entry);

		VM_WARN_ON_ONCE(!softleaf_is_migration_write(entry));

		/* A writable anon migration entry implies AnonExclusive. */
		if (folio_test_anon(softleaf_to_folio(entry)))
			return make_readable_exclusive_migration_entry(offset);
		return make_readable_migration_entry(offset);
	}

	static inline softleaf_t softleaf_mkreadable(softleaf_t entry)
	{
		switch (softleaf_type(entry)) {
		case SOFTLEAF_DEVICE_PRIVATE_WRITE:
			return make_readable_device_private_entry(swp_offset(entry));
		case SOFTLEAF_MIGRATION_WRITE:
			return softleaf_migration_mkreadable(entry);
		default:
			return entry;
		}
	}

fs/proc/task_mmu.c static unsigned long pagemap_page_category(struct pagemap_scan_private *p, · line 2415

Quoted 8 lines
fs/proc/task_mmu.c +8 -0
+}
+
+/*
+ * An armed entry must not come back writable, so downgrade a writable
+ * migration or device-private entry before giving it the bit, as
+ * change_softleaf_pte() does.
+ */
+static pte_t set_uffd_wp_swp_pte(pte_t pte)

This is a contradictory name, you're interacting with softleaves, there's no such thing as a uffd swap entry :)

I/we obviously have to go and fixup the naming (I didn't _quite_ finish the softleaf conversions I guess) but let's not propagate it.

Quoted 27 lines
fs/proc/task_mmu.c +15 -1
+{
+	softleaf_t entry = softleaf_from_pte(pte);
+	pte_t newpte;
+
+	if (!softleaf_is_migration_write(entry) &&
+	    !softleaf_is_device_private_write(entry))
+		return pte_swp_mkuffd(pte);
+
+	newpte = softleaf_to_pte(make_readable_softleaf(entry));
+	if (pte_swp_soft_dirty(pte))
+		newpte = pte_swp_mksoft_dirty(newpte);
+	return pte_swp_mkuffd(newpte);
+}
+
 static void make_uffd_wp_pte(struct vm_area_struct *vma,
 			     unsigned long addr, pte_t *pte, pte_t ptent)
 {
@@ -2408,7 +2440,7 @@ static void make_uffd_wp_pte(struct vm_area_struct *vma,
 		set_pte_at(vma->vm_mm, addr, pte,
 			   make_pte_marker(PTE_MARKER_UFFD_WP));
 	} else {
-		ptent = pte_swp_mkuffd(ptent);
+		ptent = set_uffd_wp_swp_pte(ptent);
 		set_pte_at(vma->vm_mm, addr, pte, ptent);
 	}
 }

fs/proc/task_mmu.c static unsigned long pagemap_thp_category(struct pagemap_scan_private *p, · line 2502

Quoted 4 lines
fs/proc/task_mmu.c +1 -0
@@ -2467,6 +2499,22 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
 	return categories;
 }
 
+/* Same as set_uffd_wp_swp_pte(), for a non-present PMD. */

No need to say non-present. Naming it with 'softleaf' somewhere in there implies that. E.g. set_u

fs/proc/task_mmu.c static unsigned long pagemap_thp_category(struct pagemap_scan_private *p, · line 2503

Quoted 1 line
fs/proc/task_mmu.c +1 -0
+static pmd_t set_uffd_wp_swp_pmd(pmd_t pmd)

Same comment here, this is just hugely confusing.

Quoted 37 lines
fs/proc/task_mmu.c +16 -3
+{
+	softleaf_t entry = softleaf_from_pmd(pmd);
+	pmd_t newpmd;
+
+	if (!softleaf_is_migration_write(entry) &&
+	    !softleaf_is_device_private_write(entry))
+		return pmd_swp_mkuffd(pmd);
+
+	newpmd = softleaf_to_pmd(make_readable_softleaf(entry));
+	if (pmd_swp_soft_dirty(pmd))
+		newpmd = pmd_swp_mksoft_dirty(newpmd);
+	return pmd_swp_mkuffd(newpmd);
+}
+
 static void make_uffd_wp_pmd(struct vm_area_struct *vma,
 			     unsigned long addr, pmd_t *pmdp)
 {
@@ -2477,7 +2525,7 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma,
 		pmd = pmd_mkuffd(old);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
 	} else if (pmd_is_migration_entry(pmd)) {
-		pmd = pmd_swp_mkuffd(pmd);
+		pmd = set_uffd_wp_swp_pmd(pmd);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
 	}
 }
@@ -2547,7 +2595,7 @@ static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
 
 	if (softleaf_is_migration(entry)) {
 		set_huge_pte_at(vma->vm_mm, addr, ptep,
-				pte_swp_mkuffd(ptent), psize);
+				set_uffd_wp_swp_pte(ptent), psize);
 	} else {
 		pte_t old_pte, new_pte;
 
 
--
Quoted 2 lines

2.39.5

--
Cheers, Lorenzo

reviewer Tested-by on v1 3/12
#9 lore raw

Re: [PATCH 03/12] fs/proc/task_mmu: downgrade writable migration entries in PAGEMAP_SCAN write-protect

In reply to Tal Zussman
Quoted 18 lines · Tal Zussman

PAGEMAP_SCAN with PM_SCAN_WP_MATCHING write-protects pages in a VMA registered for async uffd-wp, and sets the uffd bit on non-present entries as they are. An entry with the uffd bit set must not be writable, so that the page is write-protected once it is mapped again. change_protection() therefore downgrades writable migration and device-private entries to readable ones first, but PAGEMAP_SCAN does not.

For a writable migration entry at the PTE level, remove_migration_pte() restores a writable PTE without the uffd bit. So if a page is under migration while PAGEMAP_SCAN write-protects it, the write protection is lost. A user tracking writes with async uffd-wp sees the page reported as written on the next scan although nothing wrote to it.

With CONFIG_PAGE_TABLE_CHECK, setting the uffd bit on a writable non-present PTE or PMD also triggers a warning.

Quoted 6 lines

To fix it, downgrade writable migration and device-private entries before setting the uffd bit, as change_protection() does, at the PTE, PMD, and hugetlb levels.

Fixes 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs")

Reproduced and managed to emit a page-table-check warning and demonstrate the lost WP during migration.

Will leave the review notes to Lorenzo, the overall idea looks sound, if painfully verbose / special. But then uffd appears to create this pattern everywhere it touches something :\

Signature · 1 line
~Gregory
#10 lore raw

PATCH 4/12 mm/migrate: keep the uffd bit when restoring a writable migration entry

In reply to Tal Zussman

remove_migration_pte() makes the restored PTE writable for a writable migration entry, and only looks at the uffd bit of the entry otherwise. A writable migration entry that has the uffd bit set is therefore restored as a writable PTE without the uffd bit, and the page is no longer write-protected.

An entry with the uffd bit set must be restored write-protected, whether the entry is writable or not. remove_migration_pmd() gets this right: it makes the PMD writable and sets the uffd bit as two independent steps, and pmd_mkuffd() write-protects the result. To fix it, do the same at the PTE level, where pte_mkuffd() write-protects the PTE as well.

The only known path that created such entries is PAGEMAP_SCAN. Now that PAGEMAP_SCAN downgrades them like change_protection() does, we should no longer see them, so this change alone should not be visible to userspace. If another path ever creates a writable migration entry with the uffd bit set, the page now stays write-protected instead of losing the uffd bit.

Fixes 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs")
---
 mm/migrate.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/migrate.c +1 -1
index 28a8d7734735..d200f64b5f98 100644
@@ -399,7 +399,7 @@ static bool remove_migration_pte(struct folio *folio,
 
 		if (softleaf_is_migration_write(entry))
 			pte = pte_mkwrite(pte, vma);
-		else if (pte_swp_uffd(old_pte))
+		if (pte_swp_uffd(old_pte))
 			pte = pte_mkuffd(pte);
 
 		/* See do_swap_page(): restore PAGE_NONE for RWP */
Signature · 1 line
2.39.5
maintainer Acked-by on v1 4/12
#11 lore raw

Re: [PATCH 04/12] mm/migrate: keep the uffd bit when restoring a writable migration entry

Quoted 5 lines · Tal Zussman

remove_migration_pte() makes the restored PTE writable for a writable migration entry, and only looks at the uffd bit of the entry otherwise. A writable migration entry that has the uffd bit set is therefore restored as a writable PTE without the uffd bit, and the page is no longer write-protected.

The UFFD WP bit is such a terrible, invasive, broken piece of... engineering :)

Notice, again, the hallmark of an utterly broken abstraction - every user just has to happen to know to handle each of your weird edge cases every single time they happen to interact with anything that touches it.

I do hope we never merge something as bad as this again.

Quoted 3 lines

An entry with the uffd bit set must be restored write-protected, whether the entry is writable or not. remove_migration_pmd() gets this right: it

Sigh :)

Quoted 5 lines

makes the PMD writable and sets the uffd bit as two independent steps, and pmd_mkuffd() write-protects the result. To fix it, do the same at the PTE level, where pte_mkuffd() write-protects the PTE as well.

The only known path that created such entries is PAGEMAP_SCAN. Now that

Wait what? :)

How does a read-only scan of entries create softleaf entries?

I think this should be explained in the commit message :)

Quoted 7 lines

PAGEMAP_SCAN downgrades them like change_protection() does, we should no longer see them, so this change alone should not be visible to userspace. If another path ever creates a writable migration entry with the uffd bit set, the page now stays write-protected instead of losing the uffd bit.

Fixes 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs")

This seems like a super-weird fixes target.

BTW I've noticed you've not been cc'ing stable, is that on purpose? No autosel in mm so we do need to do that on stuff intended to get backported. :)

Quoted 1 line

The change itself seems reasonable, as cursed as the reason for it is, so with the fixes target confirmed + commit message updated:

Quoted 19 lines
---
 mm/migrate.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/migrate.c +1 -2
index 28a8d7734735..d200f64b5f98 100644
@@ -399,7 +399,7 @@ static bool remove_migration_pte(struct folio *folio,
 
 		if (softleaf_is_migration_write(entry))
 			pte = pte_mkwrite(pte, vma);
-		else if (pte_swp_uffd(old_pte))
+		if (pte_swp_uffd(old_pte))
 			pte = pte_mkuffd(pte);
 
 		/* See do_swap_page(): restore PAGE_NONE for RWP */
 
--
Quoted 2 lines

2.39.5

--
Cheers, Lorenzo

#12 lore raw

Re: [PATCH 04/12] mm/migrate: keep the uffd bit when restoring a writable migration entry

Quoted 26 lines · Lorenzo Stoakes (ARM)
Quoted 5 lines · Tal Zussman

remove_migration_pte() makes the restored PTE writable for a writable migration entry, and only looks at the uffd bit of the entry otherwise. A writable migration entry that has the uffd bit set is therefore restored as a writable PTE without the uffd bit, and the page is no longer write-protected.

The UFFD WP bit is such a terrible, invasive, broken piece of... engineering :)

Notice, again, the hallmark of an utterly broken abstraction - every user just has to happen to know to handle each of your weird edge cases every single time they happen to interact with anything that touches it.

I do hope we never merge something as bad as this again.

Quoted 3 lines

An entry with the uffd bit set must be restored write-protected, whether the entry is writable or not. remove_migration_pmd() gets this right: it

Sigh :)

Quoted 4 lines

makes the PMD writable and sets the uffd bit as two independent steps, and pmd_mkuffd() write-protects the result. To fix it, do the same at the PTE level, where pte_mkuffd() write-protects the PTE as well.

Quoted 8 lines
Quoted 1 line

The only known path that created such entries is PAGEMAP_SCAN. Now that

Wait what? :)

How does a read-only scan of entries create softleaf entries?

I think this should be explained in the commit message :)

Will add a sentence... unfortunately not a read-only scan...

Quoted 10 lines
Quoted 7 lines

PAGEMAP_SCAN downgrades them like change_protection() does, we should no longer see them, so this change alone should not be visible to userspace. If another path ever creates a writable migration entry with the uffd bit set, the page now stays write-protected instead of losing the uffd bit.

Fixes 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs")

This seems like a super-weird fixes target.

So this piece of code was added in f45ec5ff16a7, but the issue derives from PAGEMAP_SCAN, which is the commit above. I think 52526ca7fdb9 is the right target, but can change it.

Quoted 4 lines

BTW I've noticed you've not been cc'ing stable, is that on purpose? No autosel in mm so we do need to do that on stuff intended to get backported. :)

I added stable for patches 1 and 2, but for this one and the previous patch the impact is PAGEMAP_SCAN reporting a page as written when it wasn't, which over-reports but doesn't lose anything, so I left it off. Maybe I'm being too conservative though.

Quoted 32 lines
Quoted 1 line

The change itself seems reasonable, as cursed as the reason for it is, so with the fixes target confirmed + commit message updated:

Quoted 19 lines
---
 mm/migrate.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/migrate.c +1 -2
index 28a8d7734735..d200f64b5f98 100644
@@ -399,7 +399,7 @@ static bool remove_migration_pte(struct folio *folio,
 
 		if (softleaf_is_migration_write(entry))
 			pte = pte_mkwrite(pte, vma);
-		else if (pte_swp_uffd(old_pte))
+		if (pte_swp_uffd(old_pte))
 			pte = pte_mkuffd(pte);
 
 		/* See do_swap_page(): restore PAGE_NONE for RWP */
 
--
Quoted 2 lines

2.39.5

--
Cheers, Lorenzo

#13 lore raw

Re: [PATCH 04/12] mm/migrate: keep the uffd bit when restoring a writable migration entry

Quoted 29 lines · Tal Zussman
Quoted 28 lines · Lorenzo Stoakes (ARM)
Quoted 5 lines · Tal Zussman

remove_migration_pte() makes the restored PTE writable for a writable migration entry, and only looks at the uffd bit of the entry otherwise. A writable migration entry that has the uffd bit set is therefore restored as a writable PTE without the uffd bit, and the page is no longer write-protected.

The UFFD WP bit is such a terrible, invasive, broken piece of... engineering :)

Notice, again, the hallmark of an utterly broken abstraction - every user just has to happen to know to handle each of your weird edge cases every single time they happen to interact with anything that touches it.

I do hope we never merge something as bad as this again.

Quoted 3 lines

An entry with the uffd bit set must be restored write-protected, whether the entry is writable or not. remove_migration_pmd() gets this right: it

Sigh :)

Quoted 5 lines

makes the PMD writable and sets the uffd bit as two independent steps, and pmd_mkuffd() write-protects the result. To fix it, do the same at the PTE level, where pte_mkuffd() write-protects the PTE as well.

The only known path that created such entries is PAGEMAP_SCAN. Now that

Quoted 8 lines
Quoted 6 lines

Wait what? :)

How does a read-only scan of entries create softleaf entries?

I think this should be explained in the commit message :)

Will add a sentence... unfortunately not a read-only scan...

Thanks! Yeah I realised later the true horrors of what PAGEMAP_SCAN can do with uffd flags (ugh).

Quoted 21 lines
Quoted 11 lines
Quoted 8 lines

PAGEMAP_SCAN downgrades them like change_protection() does, we should no longer see them, so this change alone should not be visible to userspace. If another path ever creates a writable migration entry with the uffd bit set, the page now stays write-protected instead of losing the uffd bit.

Fixes 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs")

This seems like a super-weird fixes target.

So this piece of code was added in f45ec5ff16a7, but the issue derives from PAGEMAP_SCAN, which is the commit above. I think 52526ca7fdb9 is the right target, but can change it.

Quoted 4 lines

BTW I've noticed you've not been cc'ing stable, is that on purpose? No autosel in mm so we do need to do that on stuff intended to get backported. :)

Quoted 4 lines

I added stable for patches 1 and 2, but for this one and the previous patch the impact is PAGEMAP_SCAN reporting a page as written when it wasn't, which over-reports but doesn't lose anything, so I left it off. Maybe I'm being too conservative though.

No that's fine, if it's sensible not to then let's not.

Was just checking to make sure it was intentional ;)

Quoted 26 lines
Quoted 26 lines
Quoted 2 lines

The change itself seems reasonable, as cursed as the reason for it is, so with the fixes target confirmed + commit message updated:

Quoted 18 lines
---
 mm/migrate.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/migrate.c +1 -1
index 28a8d7734735..d200f64b5f98 100644
@@ -399,7 +399,7 @@ static bool remove_migration_pte(struct folio *folio,
 
 		if (softleaf_is_migration_write(entry))
 			pte = pte_mkwrite(pte, vma);
-		else if (pte_swp_uffd(old_pte))
+		if (pte_swp_uffd(old_pte))
 			pte = pte_mkuffd(pte);
 
 		/* See do_swap_page(): restore PAGE_NONE for RWP */
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

--
2.39.5

--
Cheers, Lorenzo

--
Cheers, Lorenzo

#14 lore raw

Re: [PATCH 04/12] mm/migrate: keep the uffd bit when restoring a writable migration entry

Quoted 63 lines · Lorenzo Stoakes (ARM)
Quoted 29 lines · Tal Zussman
Quoted 28 lines · Lorenzo Stoakes (ARM)
Quoted 5 lines · Tal Zussman

remove_migration_pte() makes the restored PTE writable for a writable migration entry, and only looks at the uffd bit of the entry otherwise. A writable migration entry that has the uffd bit set is therefore restored as a writable PTE without the uffd bit, and the page is no longer write-protected.

The UFFD WP bit is such a terrible, invasive, broken piece of... engineering :)

Notice, again, the hallmark of an utterly broken abstraction - every user just has to happen to know to handle each of your weird edge cases every single time they happen to interact with anything that touches it.

I do hope we never merge something as bad as this again.

Quoted 3 lines

An entry with the uffd bit set must be restored write-protected, whether the entry is writable or not. remove_migration_pmd() gets this right: it

Sigh :)

Quoted 5 lines

makes the PMD writable and sets the uffd bit as two independent steps, and pmd_mkuffd() write-protects the result. To fix it, do the same at the PTE level, where pte_mkuffd() write-protects the PTE as well.

The only known path that created such entries is PAGEMAP_SCAN. Now that

Quoted 8 lines
Quoted 6 lines

Wait what? :)

How does a read-only scan of entries create softleaf entries?

I think this should be explained in the commit message :)

Will add a sentence... unfortunately not a read-only scan...

Thanks! Yeah I realised later the true horrors of what PAGEMAP_SCAN can do with uffd flags (ugh).

Quoted 21 lines
Quoted 11 lines
Quoted 8 lines

PAGEMAP_SCAN downgrades them like change_protection() does, we should no longer see them, so this change alone should not be visible to userspace. If another path ever creates a writable migration entry with the uffd bit set, the page now stays write-protected instead of losing the uffd bit.

Fixes 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs")

This seems like a super-weird fixes target.

So this piece of code was added in f45ec5ff16a7, but the issue derives from PAGEMAP_SCAN, which is the commit above. I think 52526ca7fdb9 is the right target, but can change it.

Quoted 4 lines

BTW I've noticed you've not been cc'ing stable, is that on purpose? No autosel in mm so we do need to do that on stuff intended to get backported. :)

Quoted 9 lines
Quoted 4 lines

I added stable for patches 1 and 2, but for this one and the previous patch the impact is PAGEMAP_SCAN reporting a page as written when it wasn't, which over-reports but doesn't lose anything, so I left it off. Maybe I'm being too conservative though.

No that's fine, if it's sensible not to then let's not.

Was just checking to make sure it was intentional ;)

Looks like Breno hit the warning fixed by patch 3:

https://lore.kernel.org/linux-mm/[email protected]/

So I think Cc stable it is.

Quoted 38 lines
Quoted 26 lines
Quoted 26 lines
Quoted 2 lines

The change itself seems reasonable, as cursed as the reason for it is, so with the fixes target confirmed + commit message updated:

Quoted 18 lines
---
 mm/migrate.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/migrate.c +1 -1
index 28a8d7734735..d200f64b5f98 100644
@@ -399,7 +399,7 @@ static bool remove_migration_pte(struct folio *folio,
 
 		if (softleaf_is_migration_write(entry))
 			pte = pte_mkwrite(pte, vma);
-		else if (pte_swp_uffd(old_pte))
+		if (pte_swp_uffd(old_pte))
 			pte = pte_mkuffd(pte);
 
 		/* See do_swap_page(): restore PAGE_NONE for RWP */
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

--
2.39.5

--
Cheers, Lorenzo

--
Cheers, Lorenzo

reviewer Tested-by Reviewed-by on v1 4/12
#15 lore raw

Re: [PATCH 04/12] mm/migrate: keep the uffd bit when restoring a writable migration entry

In reply to Tal Zussman
Quoted 19 lines · Tal Zussman

remove_migration_pte() makes the restored PTE writable for a writable migration entry, and only looks at the uffd bit of the entry otherwise. A writable migration entry that has the uffd bit set is therefore restored as a writable PTE without the uffd bit, and the page is no longer write-protected.

An entry with the uffd bit set must be restored write-protected, whether the entry is writable or not. remove_migration_pmd() gets this right: it makes the PMD writable and sets the uffd bit as two independent steps, and pmd_mkuffd() write-protects the result. To fix it, do the same at the PTE level, where pte_mkuffd() write-protects the PTE as well.

The only known path that created such entries is PAGEMAP_SCAN. Now that PAGEMAP_SCAN downgrades them like change_protection() does, we should no longer see them, so this change alone should not be visible to userspace. If another path ever creates a writable migration entry with the uffd bit set, the page now stays write-protected instead of losing the uffd bit.

Quoted 2 lines
Fixes 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs")

yeesh, agreed with Lorenzo, this is painful.

I wonder if we can fix this abstraction somehow or if we're stuck with it forever. The nuance here suggests we're going to find more of these corner conditions over time.

I reproduced the issue and showed this patch also closed the problem by itself (without the prior patch).

Signature · 1 line
~Gregory
reviewer Reviewed-by on v1 4/12
#16 lore raw

Re: [PATCH 04/12] mm/migrate: keep the uffd bit when restoring a writable migration entry

In reply to Tal Zussman

On 2026-11-12 at 04:55 +1100, Tal Zussman <[email protected]> wrote...

Quoted 19 lines

remove_migration_pte() makes the restored PTE writable for a writable migration entry, and only looks at the uffd bit of the entry otherwise. A writable migration entry that has the uffd bit set is therefore restored as a writable PTE without the uffd bit, and the page is no longer write-protected.

An entry with the uffd bit set must be restored write-protected, whether the entry is writable or not. remove_migration_pmd() gets this right: it makes the PMD writable and sets the uffd bit as two independent steps, and pmd_mkuffd() write-protects the result. To fix it, do the same at the PTE level, where pte_mkuffd() write-protects the PTE as well.

The only known path that created such entries is PAGEMAP_SCAN. Now that PAGEMAP_SCAN downgrades them like change_protection() does, we should no longer see them, so this change alone should not be visible to userspace. If another path ever creates a writable migration entry with the uffd bit set, the page now stays write-protected instead of losing the uffd bit.

Quoted 2 lines
Fixes 52526ca7fdb9 ("fs/proc/task_mmu: implement IOCTL to get and optionally clear info about PTEs")

The code change looks correct to me, so assuming you address Lorenzo's feedback on the commit message:

Quoted 21 lines
---
 mm/migrate.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/migrate.c +1 -1
index 28a8d7734735..d200f64b5f98 100644
@@ -399,7 +399,7 @@ static bool remove_migration_pte(struct folio *folio,
 
 		if (softleaf_is_migration_write(entry))
 			pte = pte_mkwrite(pte, vma);
-		else if (pte_swp_uffd(old_pte))
+		if (pte_swp_uffd(old_pte))
 			pte = pte_mkuffd(pte);
 
 		/* See do_swap_page(): restore PAGE_NONE for RWP */
Signature · 2 lines
-- 
2.39.5
#17 lore raw

PATCH 5/12 fs/proc/task_mmu: write-protect device-private THPs in PAGEMAP_SCAN

In reply to Tal Zussman

PAGEMAP_SCAN with PM_SCAN_WP_MATCHING write-protects a huge PMD through make_uffd_wp_pmd(), which handles a present THP and a migration entry but does nothing for a device-private THP. The PMD never gets the uffd bit.

pagemap_thp_category() reports a non-present PMD without the uffd bit as written. A user tracking writes with async uffd-wp therefore sees every page of a device-private THP as written on every scan, whether it was written to or not. This requires a driver that migrates THPs to device memory.

A device-private PMD must be write-protected like any other huge PMD. To fix it, pass it to set_uffd_wp_swp_pmd() as well, which downgrades a writable entry and sets the uffd bit. When the THP migrates back, set_pmd_migration_entry() carries the uffd bit over to the migration entry, so the write protection is kept.

Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")
---
 fs/proc/task_mmu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/proc/task_mmu.c +2 -1
index 53780a454432..752f1b13d47a 100644
@@ -2524,7 +2524,8 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma,
 		old = pmdp_invalidate_ad(vma, addr, pmdp);
 		pmd = pmd_mkuffd(old);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
-	} else if (pmd_is_migration_entry(pmd)) {
+	} else if (pmd_is_migration_entry(pmd) ||
+		   pmd_is_device_private_entry(pmd)) {
 		pmd = set_uffd_wp_swp_pmd(pmd);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
 	}
Signature · 1 line
2.39.5
#18 lore raw

Re: [PATCH 05/12] fs/proc/task_mmu: write-protect device-private THPs in PAGEMAP_SCAN

Quoted 2 lines · Tal Zussman

PAGEMAP_SCAN with PM_SCAN_WP_MATCHING write-protects a huge PMD through make_uffd_wp_pmd(), which handles a present THP and a migration entry

It's crazy to me that we write-protect page tables through a procfs interface like this. But I guess we have to live with it now.

I suppose that answers my q about this on a previous commit...

Quoted 2 lines

but does nothing for a device-private THP. The PMD never gets the uffd bit.

Sigh yeah. Device-private feels like another 'just have to happen to know to update X million open-coded parts of the code'.

We really need to find a way to abstract this stuff.

Quoted 24 lines

pagemap_thp_category() reports a non-present PMD without the uffd bit as written. A user tracking writes with async uffd-wp therefore sees every page of a device-private THP as written on every scan, whether it was written to or not. This requires a driver that migrates THPs to device memory.

A device-private PMD must be write-protected like any other huge PMD. To fix it, pass it to set_uffd_wp_swp_pmd() as well, which downgrades a writable entry and sets the uffd bit. When the THP migrates back, set_pmd_migration_entry() carries the uffd bit over to the migration entry, so the write protection is kept.

Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")
---
 fs/proc/task_mmu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/proc/task_mmu.c +0 -0
index 53780a454432..752f1b13d47a 100644

fs/proc/task_mmu.c static void make_uffd_wp_pmd(struct vm_area_struct *vma, · line 2528

Quoted 6 lines
fs/proc/task_mmu.c +2 -1
@@ -2524,7 +2524,8 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma,
 		old = pmdp_invalidate_ad(vma, addr, pmdp);
 		pmd = pmd_mkuffd(old);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
-	} else if (pmd_is_migration_entry(pmd)) {
+	} else if (pmd_is_migration_entry(pmd) ||
+		   pmd_is_device_private_entry(pmd)) {

Could this not just be !pmd_none() + cover all softleaf entries? Or is that not valid?

It seems completely insane to have to specifically spell out all the possibilities, open-coded, everywhere.

Quoted 7 lines
fs/proc/task_mmu.c +0 -1
 		pmd = set_uffd_wp_swp_pmd(pmd);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
 	}
 
--

2.39.5

--
Cheers, Lorenzo

#19 lore raw

Re: [PATCH 05/12] fs/proc/task_mmu: write-protect device-private THPs in PAGEMAP_SCAN

Quoted 9 lines · Lorenzo Stoakes (ARM)
Quoted 2 lines · Tal Zussman

but does nothing for a device-private THP. The PMD never gets the uffd bit.

Sigh yeah. Device-private feels like another 'just have to happen to know to update X million open-coded parts of the code'.

We really need to find a way to abstract this stuff.

I've poked at it a bit, but zone device creates this issue in many places. The core issue is really that softleaf can mean 20 things, and the number of things has been piled on heavily.

I don't know how we solve this, it's just a rats nest of complexity at this point.

Signature · 1 line
~Gregory
#20 lore raw

Re: [PATCH 05/12] fs/proc/task_mmu: write-protect device-private THPs in PAGEMAP_SCAN

Quoted 11 lines · Gregory Price (Meta)
Quoted 9 lines · Lorenzo Stoakes (ARM)
Quoted 2 lines · Tal Zussman

but does nothing for a device-private THP. The PMD never gets the uffd bit.

Sigh yeah. Device-private feels like another 'just have to happen to know to update X million open-coded parts of the code'.

We really need to find a way to abstract this stuff.

Quoted 6 lines

I've poked at it a bit, but zone device creates this issue in many places. The core issue is really that softleaf can mean 20 things, and the number of things has been piled on heavily.

I don't know how we solve this, it's just a rats nest of complexity at this point.

And you are our Jack Russell? ;)

TBH we need some generic handlers for softleaf cases. The fact it's abstracted in a form that we can describe clearly it might be possible to pull useful things out of that.

Like:

softleaf_can_xxx()
softleaf_should_yyy()

etc.

Ideally we'd have entirely shared page table code but that seems like a far off goal esp. in the face of the schloppgeddon.

But in the meantime maybe we can use semantic helpers to at least generalise the glue between softleaf X requires handling Y?

Quoted 4 lines

--
~Gregory

--
Cheers, Lorenzo

#21 lore raw

Re: [PATCH 05/12] fs/proc/task_mmu: write-protect device-private THPs in PAGEMAP_SCAN

In reply to Lorenzo Stoakes (ARM)
Quoted 51 lines · Lorenzo Stoakes (ARM)
Quoted 2 lines · Tal Zussman

PAGEMAP_SCAN with PM_SCAN_WP_MATCHING write-protects a huge PMD through make_uffd_wp_pmd(), which handles a present THP and a migration entry

It's crazy to me that we write-protect page tables through a procfs interface like this. But I guess we have to live with it now.

I suppose that answers my q about this on a previous commit...

Quoted 2 lines

but does nothing for a device-private THP. The PMD never gets the uffd bit.

Sigh yeah. Device-private feels like another 'just have to happen to know to update X million open-coded parts of the code'.

We really need to find a way to abstract this stuff.

Quoted 24 lines

pagemap_thp_category() reports a non-present PMD without the uffd bit as written. A user tracking writes with async uffd-wp therefore sees every page of a device-private THP as written on every scan, whether it was written to or not. This requires a driver that migrates THPs to device memory.

A device-private PMD must be write-protected like any other huge PMD. To fix it, pass it to set_uffd_wp_swp_pmd() as well, which downgrades a writable entry and sets the uffd bit. When the THP migrates back, set_pmd_migration_entry() carries the uffd bit over to the migration entry, so the write protection is kept.

Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")
---
 fs/proc/task_mmu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/proc/task_mmu.c +0 -0
index 53780a454432..752f1b13d47a 100644

fs/proc/task_mmu.c static void make_uffd_wp_pmd(struct vm_area_struct *vma, · line 2528

Quoted 6 lines
fs/proc/task_mmu.c +2 -1
@@ -2524,7 +2524,8 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma,
 		old = pmdp_invalidate_ad(vma, addr, pmdp);
 		pmd = pmd_mkuffd(old);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
-	} else if (pmd_is_migration_entry(pmd)) {
+	} else if (pmd_is_migration_entry(pmd) ||
+		   pmd_is_device_private_entry(pmd)) {

Could this not just be !pmd_none() + cover all softleaf entries? Or is that not valid?

Quoted 3 lines

It seems completely insane to have to specifically spell out all the possibilities, open-coded, everywhere.

So we actually have a helper for exactly this that I didn't notice before... pmd_is_valid_softleaf() is equivalent. !pmd_none() should work but I think the existing helper is better.

Quoted 11 lines
Quoted 7 lines
fs/proc/task_mmu.c +0 -1
 		pmd = set_uffd_wp_swp_pmd(pmd);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
 	}
 
--

2.39.5

--
Cheers, Lorenzo

#22 lore raw

Re: [PATCH 05/12] fs/proc/task_mmu: write-protect device-private THPs in PAGEMAP_SCAN

Quoted 52 lines · Tal Zussman
Quoted 51 lines · Lorenzo Stoakes (ARM)
Quoted 2 lines · Tal Zussman

PAGEMAP_SCAN with PM_SCAN_WP_MATCHING write-protects a huge PMD through make_uffd_wp_pmd(), which handles a present THP and a migration entry

It's crazy to me that we write-protect page tables through a procfs interface like this. But I guess we have to live with it now.

I suppose that answers my q about this on a previous commit...

Quoted 2 lines

but does nothing for a device-private THP. The PMD never gets the uffd bit.

Sigh yeah. Device-private feels like another 'just have to happen to know to update X million open-coded parts of the code'.

We really need to find a way to abstract this stuff.

Quoted 24 lines

pagemap_thp_category() reports a non-present PMD without the uffd bit as written. A user tracking writes with async uffd-wp therefore sees every page of a device-private THP as written on every scan, whether it was written to or not. This requires a driver that migrates THPs to device memory.

A device-private PMD must be write-protected like any other huge PMD. To fix it, pass it to set_uffd_wp_swp_pmd() as well, which downgrades a writable entry and sets the uffd bit. When the THP migrates back, set_pmd_migration_entry() carries the uffd bit over to the migration entry, so the write protection is kept.

Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")
---
 fs/proc/task_mmu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
fs/proc/task_mmu.c +0 -0
index 53780a454432..752f1b13d47a 100644

fs/proc/task_mmu.c static void make_uffd_wp_pmd(struct vm_area_struct *vma, · line 2528

Quoted 6 lines
fs/proc/task_mmu.c +2 -1
@@ -2524,7 +2524,8 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma,
 		old = pmdp_invalidate_ad(vma, addr, pmdp);
 		pmd = pmd_mkuffd(old);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
-	} else if (pmd_is_migration_entry(pmd)) {
+	} else if (pmd_is_migration_entry(pmd) ||
+		   pmd_is_device_private_entry(pmd)) {

Could this not just be !pmd_none() + cover all softleaf entries? Or is that not valid?

Quoted 7 lines
Quoted 3 lines

It seems completely insane to have to specifically spell out all the possibilities, open-coded, everywhere.

So we actually have a helper for exactly this that I didn't notice before... pmd_is_valid_softleaf() is equivalent. !pmd_none() should work but I think the existing helper is better.

Hmm yeah that could work actually!

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
fs/proc/task_mmu.c +0 -0
 		pmd = set_uffd_wp_swp_pmd(pmd);
 		set_pmd_at(vma->vm_mm, addr, pmdp, pmd);
 	}
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

--
2.39.5

--
Cheers, Lorenzo

--
Cheers, Lorenzo

reviewer Tested-by on v1 5/12
#23 lore raw

Re: [PATCH 05/12] fs/proc/task_mmu: write-protect device-private THPs in PAGEMAP_SCAN

In reply to Tal Zussman
Quoted 11 lines · Tal Zussman

PAGEMAP_SCAN with PM_SCAN_WP_MATCHING write-protects a huge PMD through make_uffd_wp_pmd(), which handles a present THP and a migration entry but does nothing for a device-private THP. The PMD never gets the uffd bit.

pagemap_thp_category() reports a non-present PMD without the uffd bit as written. A user tracking writes with async uffd-wp therefore sees every page of a device-private THP as written on every scan, whether it was written to or not. This requires a driver that migrates THPs to device memory.

Quoted 8 lines

A device-private PMD must be write-protected like any other huge PMD. To fix it, pass it to set_uffd_wp_swp_pmd() as well, which downgrades a writable entry and sets the uffd bit. When the THP migrates back, set_pmd_migration_entry() carries the uffd bit over to the migration entry, so the write protection is kept.

Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")

reproduced the issue by observing we fail to carry back the write protect bit after migration.

Implementation details i'll leave to Lorenzo's feedback, i don't feel strongly about it, it's gonna look bad either way i think :\

Signature · 1 line
~Gregory
#24 lore raw

PATCH 6/12 mm/huge_memory: build the moved huge zero PMD from the destination VMA

In reply to Tal Zussman

move_pages_huge_pmd() builds the destination PMD for the huge zero page by copying the source PMD and clearing its uffd bit, so the destination keeps the protection of the source.

A source VMA registered for RWP can map the huge zero page with a PAGE_NONE PMD, and UFFDIO_MOVE does not check how the source is registered. The destination then gets a PAGE_NONE PMD without the uffd bit, although nothing asked for that in the destination VMA.

The next access fixes it up through the NUMA hinting fault path. Until then, the first access to the destination takes a spurious fault and GUP has to take the fault path. The impact on userspace is small.

The moved entry must have the protection of the destination VMA. To fix it, build the entry from the vm_page_prot of the destination VMA, as move_zeropage_pte() does for the small zero page, and keep marking it soft-dirty. A destination VMA registered for RWP still gets its PMD re-armed afterwards.

Fixes 763f43865fc4 ("mm: preserve RWP marker across PTE rewrites")
---
 mm/huge_memory.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
mm/huge_memory.c +3 -3
index 0d589be136d2..09176b290f76 100644
@@ -2981,9 +2981,9 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm
 		/* Follow mremap() behavior and treat the entry dirty after the move */
 		_dst_pmd = pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
 	} else {
-		src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd);
-		_dst_pmd = move_soft_dirty_pmd(src_pmdval);
-		_dst_pmd = clear_uffd_wp_pmd(_dst_pmd);
+		pmdp_huge_clear_flush(src_vma, src_addr, src_pmd);
+		_dst_pmd = folio_mk_pmd(huge_zero_folio, dst_vma->vm_page_prot);
+		_dst_pmd = move_soft_dirty_pmd(pmd_mkspecial(_dst_pmd));
 	}
 
 	/* Re-arm RWP on the moved PMD if dst_vma is RWP-registered. */
Signature · 1 line
2.39.5
reviewer Reviewed-by on v1 6/12
#25 lore raw

Re: [PATCH 06/12] mm/huge_memory: build the moved huge zero PMD from the destination VMA

Quoted 13 lines · Tal Zussman

move_pages_huge_pmd() builds the destination PMD for the huge zero page by copying the source PMD and clearing its uffd bit, so the destination keeps the protection of the source.

A source VMA registered for RWP can map the huge zero page with a PAGE_NONE PMD, and UFFDIO_MOVE does not check how the source is registered. The destination then gets a PAGE_NONE PMD without the uffd bit, although nothing asked for that in the destination VMA.

The next access fixes it up through the NUMA hinting fault path. Until then, the first access to the destination takes a spurious fault and GUP has to take the fault path. The impact on userspace is small.

Quoted 8 lines

The moved entry must have the protection of the destination VMA. To fix it, build the entry from the vm_page_prot of the destination VMA, as move_zeropage_pte() does for the small zero page, and keep marking it soft-dirty. A destination VMA registered for RWP still gets its PMD re-armed afterwards.

Fixes 763f43865fc4 ("mm: preserve RWP marker across PTE rewrites")
Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#26 lore raw

Re: [PATCH 06/12] mm/huge_memory: build the moved huge zero PMD from the destination VMA

In reply to Tal Zussman
Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce: - [Critical] mm: clean+writable PTEs and UFFD_WP…

Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce:
- [Critical] mm: clean+writable PTEs and UFFD_WP bypass in move_present_ptes() [1]
--

commit a31e8bf5652b13454bb968641961e319c67afcd0
Author: Tal Zussman <[email protected]>

mm/huge_memory: build the moved huge zero PMD from the destination VMA

This commit builds the destination PMD for a moved huge zero page from dst_vma->vm_page_prot instead of copying the source PMD.

Quoted 5 lines
mm/huge_memory.c +0 -0
index 0d589be136d21..09176b290f768 100644

mm/huge_memory.c int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm · line 2987

Quoted 10 lines
mm/huge_memory.c +3 -3
@@ -2981,9 +2981,9 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm
 		/* Follow mremap() behavior and treat the entry dirty after the move */
 		_dst_pmd = pmd_mkwrite(pmd_mkdirty(_dst_pmd), dst_vma);
 	} else {
-		src_pmdval = pmdp_huge_clear_flush(src_vma, src_addr, src_pmd);
-		_dst_pmd = move_soft_dirty_pmd(src_pmdval);
-		_dst_pmd = clear_uffd_wp_pmd(_dst_pmd);
+		pmdp_huge_clear_flush(src_vma, src_addr, src_pmd);
+		_dst_pmd = folio_mk_pmd(huge_zero_folio, dst_vma->vm_page_prot);
+		_dst_pmd = move_soft_dirty_pmd(pmd_mkspecial(_dst_pmd));
 	}

[Severity: Critical]
This problem wasn't introduced by this patch, but in the src_folio branch above and in move_present_ptes() [1], doesn't unconditionally calling pmd_mkwrite() and pte_mkwrite() on the destination entry bypass userfaultfd write-protection and create clean-but-writable page table entries?

mm/userfaultfd.c:move_present_ptes() {
	...
		orig_dst_pte = folio_mk_pte(src_folio, dst_vma->vm_page_prot);
		/* Set soft dirty bit so userspace can notice the pte was moved */
		if (pgtable_supports_soft_dirty())
			orig_dst_pte = pte_mksoft_dirty(orig_dst_pte);
		if (pte_dirty(orig_src_pte))
			orig_dst_pte = pte_mkdirty(orig_dst_pte);
		orig_dst_pte = pte_mkwrite(orig_dst_pte, dst_vma);
	...
}

Both functions drop userfaultfd write-protection from the source entry and make the destination entry writable, allowing userspace to bypass write-protection tracking by moving protected pages. In addition, when move_present_ptes() moves a clean page, calling pte_mkwrite() without pte_mkdirty() leaves orig_dst_pte clean and writable, so later writes can go untracked and be lost on reclaim.

[1] https://sashiko.dev/#/bug/linux-6a04b397-aeb2-44a3-b8f5-9f235e068739

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
reviewer Tested-by Reviewed-by on v1 6/12
#27 lore raw

Re: [PATCH 06/12] mm/huge_memory: build the moved huge zero PMD from the destination VMA

In reply to Tal Zussman
Quoted 13 lines · Tal Zussman

move_pages_huge_pmd() builds the destination PMD for the huge zero page by copying the source PMD and clearing its uffd bit, so the destination keeps the protection of the source.

A source VMA registered for RWP can map the huge zero page with a PAGE_NONE PMD, and UFFDIO_MOVE does not check how the source is registered. The destination then gets a PAGE_NONE PMD without the uffd bit, although nothing asked for that in the destination VMA.

The next access fixes it up through the NUMA hinting fault path. Until then, the first access to the destination takes a spurious fault and GUP has to take the fault path. The impact on userspace is small.

Quoted 8 lines

The moved entry must have the protection of the destination VMA. To fix it, build the entry from the vm_page_prot of the destination VMA, as move_zeropage_pte() does for the small zero page, and keep marking it soft-dirty. A destination VMA registered for RWP still gets its PMD re-armed afterwards.

Fixes 763f43865fc4 ("mm: preserve RWP marker across PTE rewrites")

Reproduced the spurious fault. Change looks reasonable.

Signature · 1 line
~Gregory
#28 lore raw

PATCH 7/12 mm/mremap: resolve uffd state when the context is dropped, not during the move

In reply to Tal Zussman

Without UFFD_FEATURE_EVENT_REMAP, the new VMA of an mremap() is not registered with userfaultfd: mremap_userfaultfd_prep() drops the context from it once the move succeeded. The uffd-wp bits, uffd-wp markers, and RWP PAGE_NONE entries in the new VMA must be cleared as well.

However, move_ptes(), move_huge_pmd(), and move_huge_pte() clear them from each entry while moving it, before we know whether the move will succeed. If a page table allocation fails halfway through, mremap() moves the entries back into the old VMA, which is still registered, without their uffd state.

Writes to a WP range are then no longer reported, and an RWP range becomes accessible without a fault. Nothing restores the protection afterwards. This requires a page table allocation to fail in the middle of mremap(), so it is rare.

The uffd state must only be cleared once the move can no longer fail. To fix it, leave the entries alone during the move and clear the uffd state of the new VMA in mremap_userfaultfd_prep(), right before dropping the context, using userfaultfd_clear_ptes() like unregistering does. A failed move then puts the entries back as they were.

This adds a second walk over the page tables of the new VMA, for VMAs registered for WP or RWP without UFFD_FEATURE_EVENT_REMAP.

Like unregistering, userfaultfd_clear_ptes() also asks for the PTEs to be made writable again where that is safe (MM_CP_TRY_CHANGE_WRITABLE). The old code only cleared the uffd bit and left the PTE read-only. For private anonymous memory this saves the minor fault on the first write to each page that was protected.

However, this adds spurious minor faults on shmem and hugetlb. When resolving, change_protection() rewrites every present entry in the range with vma->vm_page_prot, so a page that was writable and not protected is made read-only, and its first write after mremap() faults. UFFDIO_UNREGISTER and UFFDIO_WRITEPROTECT already behave this way and lead to the same spurious faults. This needs to be fixed in change_protection() and will be done in a separate change.

The entries no longer need per-entry handling during the move, so drop uffd_supports_page_table_move(). Where the old and new addresses are suitably aligned, move_normal_pmd() and move_normal_pud() can move whole page tables of these VMAs again.

clear_uffd_wp_pmd() has no users left, so remove it.

Fixes 0cef0bb836e3 ("mm: clear uffd-wp PTE/PMD state on mremap()")
Fixes 763f43865fc4 ("mm: preserve RWP marker across PTE rewrites")
---
 mm/huge_memory.c | 25 -------------------------
 mm/hugetlb.c     | 27 +--------------------------
 mm/mremap.c      | 46 +---------------------------------------------
 mm/userfaultfd.c |  1 +
 4 files changed, 3 insertions(+), 96 deletions(-)
mm/huge_memory.c +0 -25
index 09176b290f76..468a49041b95 100644
@@ -2616,18 +2616,6 @@ static pmd_t move_soft_dirty_pmd(pmd_t pmd)
 	return pmd;
 }
 
-static pmd_t clear_uffd_wp_pmd(pmd_t pmd)
-{
-	if (pmd_none(pmd))
-		return pmd;
-	if (pmd_present(pmd))
-		pmd = pmd_clear_uffd(pmd);
-	else
-		pmd = pmd_swp_clear_uffd(pmd);
-
-	return pmd;
-}
-
 bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 		  unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd)
 {
@@ -2663,19 +2651,6 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 			pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
 		}
 		pmd = move_soft_dirty_pmd(pmd);
-		if (vma_has_uffd_without_event_remap(vma)) {
-			/*
-			 * See __copy_present_ptes(): normalise the RWP marker
-			 * so the destination starts accessible instead of
-			 * taking a numa-hinting fault on first access. Only the
-			 * marker (protnone + uffd) needs it; leave other present
-			 * PMDs in the VMA untouched.
-			 */
-			if (pmd_present(pmd) && userfaultfd_rwp(vma) &&
-			    pmd_uffd(pmd))
-				pmd = pmd_modify(pmd, vma->vm_page_prot);
-			pmd = clear_uffd_wp_pmd(pmd);
-		}
 		set_pmd_at(mm, new_addr, new_pmd, pmd);
 		if (force_flush)
 			flush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
mm/hugetlb.c +1 -26
index cb2de24affdc..97bd29925476 100644
@@ -5168,7 +5168,6 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
 			  unsigned long new_addr, pte_t *src_pte, pte_t *dst_pte,
 			  unsigned long sz)
 {
-	bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma);
 	struct hstate *h = hstate_vma(vma);
 	struct mm_struct *mm = vma->vm_mm;
 	spinlock_t *src_ptl, *dst_ptl;
@@ -5179,31 +5178,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
 	double_pt_lock(dst_ptl, src_ptl);
 
 	pte = huge_ptep_get_and_clear(mm, old_addr, src_pte, sz);
-
-	if (need_clear_uffd_wp && pte_is_uffd_wp_marker(pte)) {
-		huge_pte_clear(mm, new_addr, dst_pte, sz);
-	} else {
-		if (need_clear_uffd_wp) {
-			if (pte_present(pte)) {
-				/*
-				 * See __copy_present_ptes(): normalise the RWP
-				 * marker so the destination starts accessible
-				 * instead of taking a numa-hinting fault on
-				 * first access. Only the marker (protnone + uffd)
-				 * needs it; leave other present PTEs untouched.
-				 */
-				if (userfaultfd_rwp(vma) && huge_pte_uffd(pte)) {
-					pte = huge_pte_modify(pte, vma->vm_page_prot);
-					pte = arch_make_huge_pte(pte, huge_page_shift(h),
-								 vma->vm_flags);
-				}
-				pte = huge_pte_clear_uffd(pte);
-			} else {
-				pte = pte_swp_clear_uffd(pte);
-			}
-		}
-		set_huge_pte_at(mm, new_addr, dst_pte, pte, sz);
-	}
+	set_huge_pte_at(mm, new_addr, dst_pte, pte, sz);
 
 	double_pt_unlock(dst_ptl, src_ptl);
 }
mm/mremap.c +1 -45

76 lines not on this page · raw message

mm/userfaultfd.c +1 -0
index ad9d243dc64b..bf42fabb5afd 100644
@@ -3161,6 +3161,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma,
 		up_write(&ctx->map_changing_lock);
 	} else {
 		/* Drop uffd context if remap feature not enabled */
+		userfaultfd_clear_ptes(vma, vma->vm_start, vma->vm_end);
 		userfaultfd_reset_ctx(vma);
 	}
 }
Signature · 1 line
2.39.5
reviewer Reviewed-by on v1 7/12
#29 lore raw

Re: [PATCH 07/12] mm/mremap: resolve uffd state when the context is dropped, not during the move

Quoted 1 line · Tal Zussman

However, this adds spurious minor faults on shmem and hugetlb.

"Minor faults" reads as UFFDIO_REGISTER_MODE_MINOR in a uffd changelog. Maybe "write faults"?

The RWP side keeps its behaviour: vm_page_prot does not depend on RWP, so resolving with change_protection() gives the same PTEs as the per-entry code did.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#30 lore raw

Re: [PATCH 07/12] mm/mremap: resolve uffd state when the context is dropped, not during the move

Quoted 6 lines · Kiryl Shutsemau
Quoted 1 line · Tal Zussman

However, this adds spurious minor faults on shmem and hugetlb.

"Minor faults" reads as UFFDIO_REGISTER_MODE_MINOR in a uffd changelog. Maybe "write faults"?

Yep, will change for v2. Thanks!

Quoted 9 lines

The RWP side keeps its behaviour: vm_page_prot does not depend on RWP, so resolving with change_protection() gives the same PTEs as the per-entry code did.

Signature · 1 line
   Kiryl Shutsemau / Kirill A. Shutemov
maintainer Reviewed-by on v1 7/12
#31 lore raw

Re: [PATCH 07/12] mm/mremap: resolve uffd state when the context is dropped, not during the move

In reply to Tal Zussman
Quoted 39 lines · Tal Zussman

Without UFFD_FEATURE_EVENT_REMAP, the new VMA of an mremap() is not registered with userfaultfd: mremap_userfaultfd_prep() drops the context from it once the move succeeded. The uffd-wp bits, uffd-wp markers, and RWP PAGE_NONE entries in the new VMA must be cleared as well.

However, move_ptes(), move_huge_pmd(), and move_huge_pte() clear them from each entry while moving it, before we know whether the move will succeed. If a page table allocation fails halfway through, mremap() moves the entries back into the old VMA, which is still registered, without their uffd state.

Writes to a WP range are then no longer reported, and an RWP range becomes accessible without a fault. Nothing restores the protection afterwards. This requires a page table allocation to fail in the middle of mremap(), so it is rare.

The uffd state must only be cleared once the move can no longer fail. To fix it, leave the entries alone during the move and clear the uffd state of the new VMA in mremap_userfaultfd_prep(), right before dropping the context, using userfaultfd_clear_ptes() like unregistering does. A failed move then puts the entries back as they were.

This adds a second walk over the page tables of the new VMA, for VMAs registered for WP or RWP without UFFD_FEATURE_EVENT_REMAP.

Like unregistering, userfaultfd_clear_ptes() also asks for the PTEs to be made writable again where that is safe (MM_CP_TRY_CHANGE_WRITABLE). The old code only cleared the uffd bit and left the PTE read-only. For private anonymous memory this saves the minor fault on the first write to each page that was protected.

However, this adds spurious minor faults on shmem and hugetlb. When resolving, change_protection() rewrites every present entry in the range with vma->vm_page_prot, so a page that was writable and not protected is made read-only, and its first write after mremap() faults. UFFDIO_UNREGISTER and UFFDIO_WRITEPROTECT already behave this way and lead to the same spurious faults. This needs to be fixed in change_protection() and will be done in a separate change.

Quoted 6 lines

The entries no longer need per-entry handling during the move, so drop uffd_supports_page_table_move(). Where the old and new addresses are suitably aligned, move_normal_pmd() and move_normal_pud() can move whole page tables of these VMAs again.

clear_uffd_wp_pmd() has no users left, so remove it.

This reads a little schloppy, I wonder if it coudl be cut down a bit?

Quoted 4 lines
Fixes 0cef0bb836e3 ("mm: clear uffd-wp PTE/PMD state on mremap()")
Fixes 763f43865fc4 ("mm: preserve RWP marker across PTE rewrites")

This LGTM and is secretly a cleanup which I love :) So:

Quoted 194 lines
---
 mm/huge_memory.c | 25 -------------------------
 mm/hugetlb.c     | 27 +--------------------------
 mm/mremap.c      | 46 +---------------------------------------------
 mm/userfaultfd.c |  1 +
 4 files changed, 3 insertions(+), 96 deletions(-)
mm/huge_memory.c +0 -25
index 09176b290f76..468a49041b95 100644
@@ -2616,18 +2616,6 @@ static pmd_t move_soft_dirty_pmd(pmd_t pmd)
 	return pmd;
 }
 
-static pmd_t clear_uffd_wp_pmd(pmd_t pmd)
-{
-	if (pmd_none(pmd))
-		return pmd;
-	if (pmd_present(pmd))
-		pmd = pmd_clear_uffd(pmd);
-	else
-		pmd = pmd_swp_clear_uffd(pmd);
-
-	return pmd;
-}
-
 bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 		  unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd)
 {
@@ -2663,19 +2651,6 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 			pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
 		}
 		pmd = move_soft_dirty_pmd(pmd);
-		if (vma_has_uffd_without_event_remap(vma)) {
-			/*
-			 * See __copy_present_ptes(): normalise the RWP marker
-			 * so the destination starts accessible instead of
-			 * taking a numa-hinting fault on first access. Only the
-			 * marker (protnone + uffd) needs it; leave other present
-			 * PMDs in the VMA untouched.
-			 */
-			if (pmd_present(pmd) && userfaultfd_rwp(vma) &&
-			    pmd_uffd(pmd))
-				pmd = pmd_modify(pmd, vma->vm_page_prot);
-			pmd = clear_uffd_wp_pmd(pmd);
-		}
 		set_pmd_at(mm, new_addr, new_pmd, pmd);
 		if (force_flush)
 			flush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
mm/hugetlb.c +1 -26
index cb2de24affdc..97bd29925476 100644
@@ -5168,7 +5168,6 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
 			  unsigned long new_addr, pte_t *src_pte, pte_t *dst_pte,
 			  unsigned long sz)
 {
-	bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma);
 	struct hstate *h = hstate_vma(vma);
 	struct mm_struct *mm = vma->vm_mm;
 	spinlock_t *src_ptl, *dst_ptl;
@@ -5179,31 +5178,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
 	double_pt_lock(dst_ptl, src_ptl);
 
 	pte = huge_ptep_get_and_clear(mm, old_addr, src_pte, sz);
-
-	if (need_clear_uffd_wp && pte_is_uffd_wp_marker(pte)) {
-		huge_pte_clear(mm, new_addr, dst_pte, sz);
-	} else {
-		if (need_clear_uffd_wp) {
-			if (pte_present(pte)) {
-				/*
-				 * See __copy_present_ptes(): normalise the RWP
-				 * marker so the destination starts accessible
-				 * instead of taking a numa-hinting fault on
-				 * first access. Only the marker (protnone + uffd)
-				 * needs it; leave other present PTEs untouched.
-				 */
-				if (userfaultfd_rwp(vma) && huge_pte_uffd(pte)) {
-					pte = huge_pte_modify(pte, vma->vm_page_prot);
-					pte = arch_make_huge_pte(pte, huge_page_shift(h),
-								 vma->vm_flags);
-				}
-				pte = huge_pte_clear_uffd(pte);
-			} else {
-				pte = pte_swp_clear_uffd(pte);
-			}
-		}
-		set_huge_pte_at(mm, new_addr, dst_pte, pte, sz);
-	}
+	set_huge_pte_at(mm, new_addr, dst_pte, pte, sz);
 
 	double_pt_unlock(dst_ptl, src_ptl);
 }
mm/mremap.c +1 -45
index fff0465f5e06..4564416a6994 100644
@@ -198,7 +198,6 @@ static int move_ptes(struct pagetable_move_control *pmc,
 		unsigned long extent, pmd_t *old_pmd, pmd_t *new_pmd)
 {
 	struct vm_area_struct *vma = pmc->old;
-	bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma);
 	struct mm_struct *mm = vma->vm_mm;
 	pte_t *old_ptep, *new_ptep;
 	pte_t old_pte, pte;
@@ -290,27 +289,7 @@ static int move_ptes(struct pagetable_move_control *pmc,
 		pte = get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes);
 		pte = move_pte(pte, old_addr, new_addr);
 		pte = move_soft_dirty_pte(pte);
-
-		if (need_clear_uffd_wp && pte_is_uffd_wp_marker(pte))
-			pte_clear(mm, new_addr, new_ptep);
-		else {
-			if (need_clear_uffd_wp) {
-				if (pte_present(pte)) {
-					/*
-					 * See __copy_present_ptes(): normalise
-					 * RWP PTEs so the destination starts
-					 * accessible instead of taking a
-					 * numa-hinting fault on first access.
-					 */
-					if (userfaultfd_rwp(vma) && pte_uffd(pte))
-						pte = pte_modify(pte, vma->vm_page_prot);
-					pte = pte_clear_uffd(pte);
-				} else {
-					pte = pte_swp_clear_uffd(pte);
-				}
-			}
-			set_ptes(mm, new_addr, new_ptep, pte, nr_ptes);
-		}
+		set_ptes(mm, new_addr, new_ptep, pte, nr_ptes);
 	}
 
 	lazy_mmu_mode_disable();
@@ -334,25 +313,6 @@ static inline bool arch_supports_page_table_move(void)
 }
 #endif
 
-static inline bool uffd_supports_page_table_move(struct pagetable_move_control *pmc)
-{
-	/*
-	 * If we are moving a VMA that has uffd-wp registered but with
-	 * remap events disabled (new VMA will not be registered with uffd), we
-	 * need to ensure that the uffd-wp state is cleared from all pgtables.
-	 * This means recursing into lower page tables in move_page_tables().
-	 *
-	 * We might get called with VMAs reversed when recovering from a
-	 * failed page table move. In that case, the
-	 * "old"-but-actually-"originally new" VMA during recovery will not have
-	 * a uffd context. Recursing into lower page tables during the original
-	 * move but not during the recovery move will cause trouble, because we
-	 * run into already-existing page tables. So check both VMAs.
-	 */
-	return !vma_has_uffd_without_event_remap(pmc->old) &&
-	       !vma_has_uffd_without_event_remap(pmc->new);
-}
-
 #ifdef CONFIG_HAVE_MOVE_PMD
 static bool move_normal_pmd(struct pagetable_move_control *pmc,
 			pmd_t *old_pmd, pmd_t *new_pmd)
@@ -365,8 +325,6 @@ static bool move_normal_pmd(struct pagetable_move_control *pmc,
 
 	if (!arch_supports_page_table_move())
 		return false;
-	if (!uffd_supports_page_table_move(pmc))
-		return false;
 	/*
 	 * The destination pmd shouldn't be established, free_pgtables()
 	 * should have released it.
@@ -434,8 +392,6 @@ static bool move_normal_pud(struct pagetable_move_control *pmc,
 
 	if (!arch_supports_page_table_move())
 		return false;
-	if (!uffd_supports_page_table_move(pmc))
-		return false;
 	/*
 	 * The destination pud shouldn't be established, free_pgtables()
 	 * should have released it.
mm/userfaultfd.c +1 -1
index ad9d243dc64b..bf42fabb5afd 100644
@@ -3161,6 +3161,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma,
 		up_write(&ctx->map_changing_lock);
 	} else {
 		/* Drop uffd context if remap feature not enabled */
+		userfaultfd_clear_ptes(vma, vma->vm_start, vma->vm_end);
 		userfaultfd_reset_ctx(vma);
 	}
 }
 
--
Quoted 2 lines

2.39.5

--
Cheers, Lorenzo

#32 lore raw

Re: [PATCH 07/12] mm/mremap: resolve uffd state when the context is dropped, not during the move

Quoted 40 lines · Lorenzo Stoakes (ARM)
Quoted 39 lines · Tal Zussman

Without UFFD_FEATURE_EVENT_REMAP, the new VMA of an mremap() is not registered with userfaultfd: mremap_userfaultfd_prep() drops the context from it once the move succeeded. The uffd-wp bits, uffd-wp markers, and RWP PAGE_NONE entries in the new VMA must be cleared as well.

However, move_ptes(), move_huge_pmd(), and move_huge_pte() clear them from each entry while moving it, before we know whether the move will succeed. If a page table allocation fails halfway through, mremap() moves the entries back into the old VMA, which is still registered, without their uffd state.

Writes to a WP range are then no longer reported, and an RWP range becomes accessible without a fault. Nothing restores the protection afterwards. This requires a page table allocation to fail in the middle of mremap(), so it is rare.

The uffd state must only be cleared once the move can no longer fail. To fix it, leave the entries alone during the move and clear the uffd state of the new VMA in mremap_userfaultfd_prep(), right before dropping the context, using userfaultfd_clear_ptes() like unregistering does. A failed move then puts the entries back as they were.

This adds a second walk over the page tables of the new VMA, for VMAs registered for WP or RWP without UFFD_FEATURE_EVENT_REMAP.

Like unregistering, userfaultfd_clear_ptes() also asks for the PTEs to be made writable again where that is safe (MM_CP_TRY_CHANGE_WRITABLE). The old code only cleared the uffd bit and left the PTE read-only. For private anonymous memory this saves the minor fault on the first write to each page that was protected.

However, this adds spurious minor faults on shmem and hugetlb. When resolving, change_protection() rewrites every present entry in the range with vma->vm_page_prot, so a page that was writable and not protected is made read-only, and its first write after mremap() faults. UFFDIO_UNREGISTER and UFFDIO_WRITEPROTECT already behave this way and lead to the same spurious faults. This needs to be fixed in change_protection() and will be done in a separate change.

Quoted 9 lines
Quoted 6 lines

The entries no longer need per-entry handling during the move, so drop uffd_supports_page_table_move(). Where the old and new addresses are suitably aligned, move_normal_pmd() and move_normal_pud() can move whole page tables of these VMAs again.

clear_uffd_wp_pmd() has no users left, so remove it.

This reads a little schloppy, I wonder if it coudl be cut down a bit?

I'll trim it a little. I was trying to be precise about all the side effects, and the spurious fault issue popped up along the way and it got kind of long...

Quoted 9 lines
Quoted 4 lines
Fixes 0cef0bb836e3 ("mm: clear uffd-wp PTE/PMD state on mremap()")
Fixes 763f43865fc4 ("mm: preserve RWP marker across PTE rewrites")

This LGTM and is secretly a cleanup which I love :) So:

Thanks!

Quoted 200 lines
Quoted 194 lines
---
 mm/huge_memory.c | 25 -------------------------
 mm/hugetlb.c     | 27 +--------------------------
 mm/mremap.c      | 46 +---------------------------------------------
 mm/userfaultfd.c |  1 +
 4 files changed, 3 insertions(+), 96 deletions(-)
mm/huge_memory.c +0 -25
index 09176b290f76..468a49041b95 100644
@@ -2616,18 +2616,6 @@ static pmd_t move_soft_dirty_pmd(pmd_t pmd)
 	return pmd;
 }
 
-static pmd_t clear_uffd_wp_pmd(pmd_t pmd)
-{
-	if (pmd_none(pmd))
-		return pmd;
-	if (pmd_present(pmd))
-		pmd = pmd_clear_uffd(pmd);
-	else
-		pmd = pmd_swp_clear_uffd(pmd);
-
-	return pmd;
-}
-
 bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 		  unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd)
 {
@@ -2663,19 +2651,6 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
 			pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
 		}
 		pmd = move_soft_dirty_pmd(pmd);
-		if (vma_has_uffd_without_event_remap(vma)) {
-			/*
-			 * See __copy_present_ptes(): normalise the RWP marker
-			 * so the destination starts accessible instead of
-			 * taking a numa-hinting fault on first access. Only the
-			 * marker (protnone + uffd) needs it; leave other present
-			 * PMDs in the VMA untouched.
-			 */
-			if (pmd_present(pmd) && userfaultfd_rwp(vma) &&
-			    pmd_uffd(pmd))
-				pmd = pmd_modify(pmd, vma->vm_page_prot);
-			pmd = clear_uffd_wp_pmd(pmd);
-		}
 		set_pmd_at(mm, new_addr, new_pmd, pmd);
 		if (force_flush)
 			flush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
mm/hugetlb.c +1 -26
index cb2de24affdc..97bd29925476 100644
@@ -5168,7 +5168,6 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
 			  unsigned long new_addr, pte_t *src_pte, pte_t *dst_pte,
 			  unsigned long sz)
 {
-	bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma);
 	struct hstate *h = hstate_vma(vma);
 	struct mm_struct *mm = vma->vm_mm;
 	spinlock_t *src_ptl, *dst_ptl;
@@ -5179,31 +5178,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
 	double_pt_lock(dst_ptl, src_ptl);
 
 	pte = huge_ptep_get_and_clear(mm, old_addr, src_pte, sz);
-
-	if (need_clear_uffd_wp && pte_is_uffd_wp_marker(pte)) {
-		huge_pte_clear(mm, new_addr, dst_pte, sz);
-	} else {
-		if (need_clear_uffd_wp) {
-			if (pte_present(pte)) {
-				/*
-				 * See __copy_present_ptes(): normalise the RWP
-				 * marker so the destination starts accessible
-				 * instead of taking a numa-hinting fault on
-				 * first access. Only the marker (protnone + uffd)
-				 * needs it; leave other present PTEs untouched.
-				 */
-				if (userfaultfd_rwp(vma) && huge_pte_uffd(pte)) {
-					pte = huge_pte_modify(pte, vma->vm_page_prot);
-					pte = arch_make_huge_pte(pte, huge_page_shift(h),
-								 vma->vm_flags);
-				}
-				pte = huge_pte_clear_uffd(pte);
-			} else {
-				pte = pte_swp_clear_uffd(pte);
-			}
-		}
-		set_huge_pte_at(mm, new_addr, dst_pte, pte, sz);
-	}
+	set_huge_pte_at(mm, new_addr, dst_pte, pte, sz);
 
 	double_pt_unlock(dst_ptl, src_ptl);
 }
mm/mremap.c +1 -45
index fff0465f5e06..4564416a6994 100644
@@ -198,7 +198,6 @@ static int move_ptes(struct pagetable_move_control *pmc,
 		unsigned long extent, pmd_t *old_pmd, pmd_t *new_pmd)
 {
 	struct vm_area_struct *vma = pmc->old;
-	bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma);
 	struct mm_struct *mm = vma->vm_mm;
 	pte_t *old_ptep, *new_ptep;
 	pte_t old_pte, pte;
@@ -290,27 +289,7 @@ static int move_ptes(struct pagetable_move_control *pmc,
 		pte = get_and_clear_ptes(mm, old_addr, old_ptep, nr_ptes);
 		pte = move_pte(pte, old_addr, new_addr);
 		pte = move_soft_dirty_pte(pte);
-
-		if (need_clear_uffd_wp && pte_is_uffd_wp_marker(pte))
-			pte_clear(mm, new_addr, new_ptep);
-		else {
-			if (need_clear_uffd_wp) {
-				if (pte_present(pte)) {
-					/*
-					 * See __copy_present_ptes(): normalise
-					 * RWP PTEs so the destination starts
-					 * accessible instead of taking a
-					 * numa-hinting fault on first access.
-					 */
-					if (userfaultfd_rwp(vma) && pte_uffd(pte))
-						pte = pte_modify(pte, vma->vm_page_prot);
-					pte = pte_clear_uffd(pte);
-				} else {
-					pte = pte_swp_clear_uffd(pte);
-				}
-			}
-			set_ptes(mm, new_addr, new_ptep, pte, nr_ptes);
-		}
+		set_ptes(mm, new_addr, new_ptep, pte, nr_ptes);
 	}
 
 	lazy_mmu_mode_disable();
@@ -334,25 +313,6 @@ static inline bool arch_supports_page_table_move(void)
 }
 #endif
 
-static inline bool uffd_supports_page_table_move(struct pagetable_move_control *pmc)
-{
-	/*
-	 * If we are moving a VMA that has uffd-wp registered but with
-	 * remap events disabled (new VMA will not be registered with uffd), we
-	 * need to ensure that the uffd-wp state is cleared from all pgtables.
-	 * This means recursing into lower page tables in move_page_tables().
-	 *
-	 * We might get called with VMAs reversed when recovering from a
-	 * failed page table move. In that case, the
-	 * "old"-but-actually-"originally new" VMA during recovery will not have
-	 * a uffd context. Recursing into lower page tables during the original
-	 * move but not during the recovery move will cause trouble, because we
-	 * run into already-existing page tables. So check both VMAs.
-	 */
-	return !vma_has_uffd_without_event_remap(pmc->old) &&
-	       !vma_has_uffd_without_event_remap(pmc->new);
-}
-
 #ifdef CONFIG_HAVE_MOVE_PMD
 static bool move_normal_pmd(struct pagetable_move_control *pmc,
 			pmd_t *old_pmd, pmd_t *new_pmd)
@@ -365,8 +325,6 @@ static bool move_normal_pmd(struct pagetable_move_control *pmc,
 
 	if (!arch_supports_page_table_move())
 		return false;
-	if (!uffd_supports_page_table_move(pmc))
-		return false;
 	/*
 	 * The destination pmd shouldn't be established, free_pgtables()
 	 * should have released it.
@@ -434,8 +392,6 @@ static bool move_normal_pud(struct pagetable_move_control *pmc,
 
 	if (!arch_supports_page_table_move())
 		return false;
-	if (!uffd_supports_page_table_move(pmc))
-		return false;
 	/*
 	 * The destination pud shouldn't be established, free_pgtables()
 	 * should have released it.
mm/userfaultfd.c +1 -1
index ad9d243dc64b..bf42fabb5afd 100644
@@ -3161,6 +3161,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma,
 		up_write(&ctx->map_changing_lock);
 	} else {
 		/* Drop uffd context if remap feature not enabled */
+		userfaultfd_clear_ptes(vma, vma->vm_start, vma->vm_end);
 		userfaultfd_reset_ctx(vma);
 	}
 }
 
--
Quoted 2 lines

2.39.5

--
Cheers, Lorenzo

#33 lore raw

Re: [PATCH 07/12] mm/mremap: resolve uffd state when the context is dropped, not during the move

In reply to Tal Zussman
Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce: - [High] userfaultfd: missing vma_start_write()…

Thank you for your contribution! Sashiko AI review found no new issues in this patch, but noticed 1 related pre-existing issue it does not introduce:
- [High] userfaultfd: missing vma_start_write() in userfaultfd_clear_vma() [1]
--

commit 4d41e88f9230b819ec9df4cf29791deea1bb44b7
Author: Tal Zussman <[email protected]>

mm/mremap: resolve uffd state when the context is dropped, not during the move

This commit leaves userfaultfd PTE and PMD state untouched while moving page table entries during mremap() and instead resolves it via userfaultfd_clear_ptes() in mremap_userfaultfd_prep() once the move succeeds.

mm/userfaultfd.c void mremap_userfaultfd_prep(struct vm_area_struct *vma, · line 3167

Quoted 12 lines
mm/userfaultfd.c +1 -0
index ad9d243dc64b0..bf42fabb5afd5 100644
@@ -3161,6 +3161,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma,
 		up_write(&ctx->map_changing_lock);
 	} else {
 		/* Drop uffd context if remap feature not enabled */
+		userfaultfd_clear_ptes(vma, vma->vm_start, vma->vm_end);
 		userfaultfd_reset_ctx(vma);
 	}
 }

[Severity: High]
This problem wasn't introduced by this patch, but in the existing unregister and release paths (userfaultfd_clear_vma() and userfaultfd_release_new()), can calling userfaultfd_clear_ptes() before vma_start_write() allow a concurrent UFFDIO_COPY, UFFDIO_CONTINUE, or UFFDIO_MOVE to leave stale uffd PTE bits in the unregistered VMA [1]?

CPU0 CPU1
userfaultfd_clear_vma()
userfaultfd_clear_ptes()

    change_protection()
      // scans past addr
                                      mfill_atomic()
                                        uffd_lock_vma()
                                          lock_vma_under_rcu()
                                        mfill_atomic_install_pte()
                                          // sees VM_UFFD_WP
                                          // installs pte_mkuffd() at addr
                                        uffd_unlock_vma()
  vma_modify_flags_uffd()
  userfaultfd_reset_ctx()
    userfaultfd_set_ctx()
      vma_start_write(vma)
      // clears VM_UFFD_* and ctx

Neither userfaultfd_unregister() nor userfaultfd_release() increments ctx->mmap_changing, and lock_vma_under_rcu() succeeds until vma_start_write() is called in userfaultfd_set_ctx() (or inside vma_modify_flags_uffd() when the VMA is split or merged). When a later write fault reuses the exclusive page via wp_page_reuse() and makes the PTE writable while the uffd bit is still set, page_table_check_pte_flags() triggers WARN_ON_ONCE(pte_uffd(pte) && pte_write(pte)).

[1] https://sashiko.dev/#/bug/linux-5cddfd13-146a-47fc-97f8-feca4b2339b4

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
reviewer Tested-by Reviewed-by on v1 7/12
#34 lore raw

Re: [PATCH 07/12] mm/mremap: resolve uffd state when the context is dropped, not during the move

In reply to Tal Zussman
Quoted 44 lines · Tal Zussman

Without UFFD_FEATURE_EVENT_REMAP, the new VMA of an mremap() is not registered with userfaultfd: mremap_userfaultfd_prep() drops the context from it once the move succeeded. The uffd-wp bits, uffd-wp markers, and RWP PAGE_NONE entries in the new VMA must be cleared as well.

However, move_ptes(), move_huge_pmd(), and move_huge_pte() clear them from each entry while moving it, before we know whether the move will succeed. If a page table allocation fails halfway through, mremap() moves the entries back into the old VMA, which is still registered, without their uffd state.

Writes to a WP range are then no longer reported, and an RWP range becomes accessible without a fault. Nothing restores the protection afterwards. This requires a page table allocation to fail in the middle of mremap(), so it is rare.

The uffd state must only be cleared once the move can no longer fail. To fix it, leave the entries alone during the move and clear the uffd state of the new VMA in mremap_userfaultfd_prep(), right before dropping the context, using userfaultfd_clear_ptes() like unregistering does. A failed move then puts the entries back as they were.

This adds a second walk over the page tables of the new VMA, for VMAs registered for WP or RWP without UFFD_FEATURE_EVENT_REMAP.

Like unregistering, userfaultfd_clear_ptes() also asks for the PTEs to be made writable again where that is safe (MM_CP_TRY_CHANGE_WRITABLE). The old code only cleared the uffd bit and left the PTE read-only. For private anonymous memory this saves the minor fault on the first write to each page that was protected.

However, this adds spurious minor faults on shmem and hugetlb. When resolving, change_protection() rewrites every present entry in the range with vma->vm_page_prot, so a page that was writable and not protected is made read-only, and its first write after mremap() faults. UFFDIO_UNREGISTER and UFFDIO_WRITEPROTECT already behave this way and lead to the same spurious faults. This needs to be fixed in change_protection() and will be done in a separate change.

The entries no longer need per-entry handling during the move, so drop uffd_supports_page_table_move(). Where the old and new addresses are suitably aligned, move_normal_pmd() and move_normal_pud() can move whole page tables of these VMAs again.

Quoted 5 lines

clear_uffd_wp_pmd() has no users left, so remove it.

Fixes 0cef0bb836e3 ("mm: clear uffd-wp PTE/PMD state on mremap()")
Fixes 763f43865fc4 ("mm: preserve RWP marker across PTE rewrites")

Is this a fair summary:

"""
A page-table move clears uffd state before the entire operation succeeds. If an allocation or later operation fails, the protection bits cannot be recovered.

Transport the page table state unchanged in both directions. Resolve the protection bits only after the move and VMA callback succeed.

... details about spurious faults ...
... remove clear_uffd_wp_pmd ...
"""

Anyway, I was able to reproduce this reliably by injecting some ENOMEM failures, and the change looks reasonable.

Signature · 1 line
~Gregory
#35 lore raw

PATCH 8/12 mm/migrate_device: do not migrate a uffd-protected zero page

In reply to Tal Zussman

migrate_vma_setup() installs a migration entry for each page it collects, which carries the uffd bit over to the new PTE. However, a PTE that maps the zero page is left in place without a migration entry. migrate_vma_insert_page() later overwrites it with a new PTE that is built only from the VMA flags, so the uffd-wp bit or the RWP PAGE_NONE protection of the old PTE is lost. The same applies to the huge zero PMD.

Once a driver has migrated such a zero page, writes and accesses to it no longer fault into userfaultfd, so the manager misses them. With async tracking, PAGEMAP_SCAN unconditionally reports the page as written or accessed.

This requires a driver that migrates memory to a device.

To fix it, do not mark a zero page for migration if its entry has the uffd bit set in a VMA registered for WP or RWP. The driver sees an address that was not migrated, which drivers already have to deal with.

The uffd bit can also be set after the zero page was collected, as write-protecting only takes the mmap lock for read. So check again under the page table lock in migrate_vma_insert_page() and migrate_vma_insert_huge_pmd_page().

The alternative would be to carry the uffd state over in migrate_vma_insert_page() and migrate_vma_insert_huge_pmd_page(). Not migrating the zero page is simpler, and a zero page that is protected by userfaultfd in a range that a driver migrates should be rare.

Fixes 292924b26024 ("userfaultfd: wp: apply _PAGE_UFFD_WP bit")
Fixes a30b48bf1b24 ("mm/migrate_device: implement THP migration of zone device pages")
---
 mm/migrate_device.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)
mm/migrate_device.c +21 -2
index 7919e04abc1c..ace216ea834c 100644
@@ -159,7 +159,16 @@ static int migrate_vma_collect_huge_pmd(pmd_t *pmdp, unsigned long start,
 
 		folio = pmd_folio(*pmdp);
 		if (is_huge_zero_folio(folio)) {
+			/*
+			 * Nothing carries the uffd state of the zero page
+			 * over, so skip it if its uffd bit is set.
+			 */
+			bool protected = userfaultfd_protected(walk->vma) &&
+					 pmd_uffd(*pmdp);
+
 			spin_unlock(ptl);
+			if (protected)
+				return migrate_vma_collect_skip(start, end, walk);
 			return migrate_vma_collect_hole(start, end, -1, walk);
 		}
 		if (pmd_write(*pmdp))
@@ -337,6 +346,9 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,
 			pfn = pte_pfn(pte);
 			if (is_zero_pfn(pfn) &&
 			    (migrate->flags & MIGRATE_VMA_SELECT_SYSTEM)) {
+				/* See migrate_vma_collect_huge_pmd(). */
+				if (userfaultfd_protected(vma) && pte_uffd(pte))
+					goto next;
 				mpfn = MIGRATE_PFN_MIGRATE;
 				migrate->cpages++;
 				goto next;
@@ -878,10 +890,14 @@ static int migrate_vma_insert_huge_pmd_page(struct migrate_vma *migrate,
 	if (userfaultfd_missing(vma))
 		goto unlock_abort;
 
-	if (is_huge_zero_pmd(*pmdp))
+	if (is_huge_zero_pmd(*pmdp)) {
+		/* The uffd bit can have been set since the collection. */
+		if (userfaultfd_protected(vma) && pmd_uffd(*pmdp))
+			goto unlock_abort;
 		flush = true;
-	else if (!pmd_none(*pmdp))
+	} else if (!pmd_none(*pmdp)) {
 		goto unlock_abort;
+	}
 
 	add_mm_counter(vma->vm_mm, MM_ANONPAGES, HPAGE_PMD_NR);
 	folio_add_new_anon_rmap(folio, vma, addr, RMAP_EXCLUSIVE);
@@ -1081,6 +1097,9 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,
 
 		if (!is_zero_pfn(pfn))
 			goto unlock_abort;
+		/* The uffd bit can have been set since the collection. */
+		if (userfaultfd_protected(vma) && pte_uffd(orig_pte))
+			goto unlock_abort;
 		flush = true;
 	} else if (!pte_none(orig_pte))
 		goto unlock_abort;
Signature · 1 line
2.39.5
#36 lore raw

Re: [PATCH 08/12] mm/migrate_device: do not migrate a uffd-protected zero page

Quoted 15 lines · Tal Zussman

migrate_vma_setup() installs a migration entry for each page it collects, which carries the uffd bit over to the new PTE. However, a PTE that maps the zero page is left in place without a migration entry. migrate_vma_insert_page() later overwrites it with a new PTE that is built only from the VMA flags, so the uffd-wp bit or the RWP PAGE_NONE protection of the old PTE is lost. The same applies to the huge zero PMD.

Once a driver has migrated such a zero page, writes and accesses to it no longer fault into userfaultfd, so the manager misses them. With async tracking, PAGEMAP_SCAN unconditionally reports the page as written or accessed.

This requires a driver that migrates memory to a device.

Quoted 9 lines

To fix it, do not mark a zero page for migration if its entry has the uffd bit set in a VMA registered for WP or RWP. The driver sees an address that was not migrated, which drivers already have to deal with.

The uffd bit can also be set after the zero page was collected, as write-protecting only takes the mmap lock for read. So check again under the page table lock in migrate_vma_insert_page() and migrate_vma_insert_huge_pmd_page().

Similar to last patch, this changelog could be improved. My best attempt at summarizing:

"""
Device migration replaces a zero-page mapping with a fresh entry, losing its userfaultfd WP or RWP protection.

Don't migrate the uffd-protected zero pages.

Skip protected zero pages during collection. Check again under the page-table lock before insertion, because another thread can apply protection after collection finishes.

The alternative ...
"""

I have concerns with this:

Quoted 4 lines

The alternative would be to carry the uffd state over in migrate_vma_insert_page() and migrate_vma_insert_huge_pmd_page(). Not migrating the zero page is simpler, and a zero page that is protected by userfaultfd in a range that a driver migrates should be rare.

This basically force-fails a migration of a uffd protected zero page.

Wouldn't we expect this to cause drivers to fail to make progress if they require that page to be resident on the device? Not sure we want to force drivers to try to handle this special case (if they even can).

We'll need some input from the migrate_device/GPU folks.

Signature · 1 line
~Gregory
#37 lore raw

Re: [PATCH 08/12] mm/migrate_device: do not migrate a uffd-protected zero page

Quoted 33 lines · Gregory Price (Meta)
Quoted 15 lines · Tal Zussman

migrate_vma_setup() installs a migration entry for each page it collects, which carries the uffd bit over to the new PTE. However, a PTE that maps the zero page is left in place without a migration entry. migrate_vma_insert_page() later overwrites it with a new PTE that is built only from the VMA flags, so the uffd-wp bit or the RWP PAGE_NONE protection of the old PTE is lost. The same applies to the huge zero PMD.

Once a driver has migrated such a zero page, writes and accesses to it no longer fault into userfaultfd, so the manager misses them. With async tracking, PAGEMAP_SCAN unconditionally reports the page as written or accessed.

This requires a driver that migrates memory to a device.

Quoted 9 lines

To fix it, do not mark a zero page for migration if its entry has the uffd bit set in a VMA registered for WP or RWP. The driver sees an address that was not migrated, which drivers already have to deal with.

The uffd bit can also be set after the zero page was collected, as write-protecting only takes the mmap lock for read. So check again under the page table lock in migrate_vma_insert_page() and migrate_vma_insert_huge_pmd_page().

Similar to last patch, this changelog could be improved. My best attempt at summarizing:

"""
Device migration replaces a zero-page mapping with a fresh entry, losing its userfaultfd WP or RWP protection.

Quoted 9 lines

Don't migrate the uffd-protected zero pages.

Skip protected zero pages during collection. Check again under the page-table lock before insertion, because another thread can apply protection after collection finishes.

The alternative ...
"""

Thanks, I'll trim it along these lines.

Quoted 7 lines

I have concerns with this:

Quoted 4 lines

The alternative would be to carry the uffd state over in migrate_vma_insert_page() and migrate_vma_insert_huge_pmd_page(). Not migrating the zero page is simpler, and a zero page that is protected by userfaultfd in a range that a driver migrates should be rare.

Quoted 8 lines

This basically force-fails a migration of a uffd protected zero page.

Wouldn't we expect this to cause drivers to fail to make progress if they require that page to be resident on the device? Not sure we want to force drivers to try to handle this special case (if they even can).

We'll need some input from the migrate_device/GPU folks.

Yeah, makes sense. I can implement the alternative approach, but testing for this is a little tricky without the right devices... I might drop this patch and send it separately with the GPU folks on Cc.

Quoted 3 lines
Signature · 1 line
~Gregory
reviewer Reviewed-by
#38 lore raw

Re: [PATCH 08/12] mm/migrate_device: do not migrate a uffd-protected zero page

On 2026-11-14 at 09:26 +1100, Tal Zussman <[email protected]> wrote...

Quoted 40 lines · Gregory Price (Meta)
Quoted 39 lines · Tal Zussman
Quoted 15 lines

migrate_vma_setup() installs a migration entry for each page it collects, which carries the uffd bit over to the new PTE. However, a PTE that maps the zero page is left in place without a migration entry. migrate_vma_insert_page() later overwrites it with a new PTE that is built only from the VMA flags, so the uffd-wp bit or the RWP PAGE_NONE protection of the old PTE is lost. The same applies to the huge zero PMD.

Once a driver has migrated such a zero page, writes and accesses to it no longer fault into userfaultfd, so the manager misses them. With async tracking, PAGEMAP_SCAN unconditionally reports the page as written or accessed.

This requires a driver that migrates memory to a device.

Quoted 9 lines

To fix it, do not mark a zero page for migration if its entry has the uffd bit set in a VMA registered for WP or RWP. The driver sees an address that was not migrated, which drivers already have to deal with.

The uffd bit can also be set after the zero page was collected, as write-protecting only takes the mmap lock for read. So check again under the page table lock in migrate_vma_insert_page() and migrate_vma_insert_huge_pmd_page().

Similar to last patch, this changelog could be improved. My best attempt at summarizing:

"""
Device migration replaces a zero-page mapping with a fresh entry, losing its userfaultfd WP or RWP protection.

Don't migrate the uffd-protected zero pages.

Skip protected zero pages during collection. Check again under the page-table lock before insertion, because another thread can apply protection after collection finishes.

Quoted 20 lines
Quoted 3 lines

The alternative ...
"""

Thanks, I'll trim it along these lines.

Quoted 14 lines

I have concerns with this:

Quoted 4 lines

The alternative would be to carry the uffd state over in migrate_vma_insert_page() and migrate_vma_insert_huge_pmd_page(). Not migrating the zero page is simpler, and a zero page that is protected by userfaultfd in a range that a driver migrates should be rare.

This basically force-fails a migration of a uffd protected zero page.

Wouldn't we expect this to cause drivers to fail to make progress if they require that page to be resident on the device? Not sure we want to force drivers to try to handle this special case (if they even can).

We'll need some input from the migrate_device/GPU folks.

Thanks, I was halfway through responding last week but Microslop had other ideas.

Failing the migration is fine - drivers can never *require* a page to be resident on the device, as migration can always fail for a host of reasons. In these cases drivers should fallback to remote mapping pages (eg. using hmm_range_fault()).

Now if userfaultfd+device memory was a commonly used feature then it would be worth fixing this as always failing would have a large performance impact. However I've never tested uffd+device memory and have never been confident the implementation was correct anyway (both features were merged around the same time). So I think the simple fix is fine:

Quoted 3 lines

Yeah, makes sense. I can implement the alternative approach, but testing for this is a little tricky without the right devices... I might drop this patch and send it separately with the GPU folks on Cc.

It probably is worth copying other GPU folk as well though in case they have different use-cases. Matthew Brost (Intel) has done a lot of work in this area in DRM, but I notice he is already on CC.

- Alistair

Quoted 4 lines
Quoted 3 lines
Signature · 1 line
~Gregory
#39 lore raw

Re: [PATCH 08/12] mm/migrate_device: do not migrate a uffd-protected zero page

Quoted 70 lines · Alistair Popple

On 2026-11-14 at 09:26 +1100, Tal Zussman <[email protected]> wrote...

Quoted 40 lines · Gregory Price (Meta)
Quoted 39 lines · Tal Zussman
Quoted 15 lines

migrate_vma_setup() installs a migration entry for each page it collects, which carries the uffd bit over to the new PTE. However, a PTE that maps the zero page is left in place without a migration entry. migrate_vma_insert_page() later overwrites it with a new PTE that is built only from the VMA flags, so the uffd-wp bit or the RWP PAGE_NONE protection of the old PTE is lost. The same applies to the huge zero PMD.

Once a driver has migrated such a zero page, writes and accesses to it no longer fault into userfaultfd, so the manager misses them. With async tracking, PAGEMAP_SCAN unconditionally reports the page as written or accessed.

This requires a driver that migrates memory to a device.

Quoted 9 lines

To fix it, do not mark a zero page for migration if its entry has the uffd bit set in a VMA registered for WP or RWP. The driver sees an address that was not migrated, which drivers already have to deal with.

The uffd bit can also be set after the zero page was collected, as write-protecting only takes the mmap lock for read. So check again under the page table lock in migrate_vma_insert_page() and migrate_vma_insert_huge_pmd_page().

Similar to last patch, this changelog could be improved. My best attempt at summarizing:

"""
Device migration replaces a zero-page mapping with a fresh entry, losing its userfaultfd WP or RWP protection.

Don't migrate the uffd-protected zero pages.

Skip protected zero pages during collection. Check again under the page-table lock before insertion, because another thread can apply protection after collection finishes.

Quoted 20 lines
Quoted 3 lines

The alternative ...
"""

Thanks, I'll trim it along these lines.

Quoted 14 lines

I have concerns with this:

Quoted 4 lines

The alternative would be to carry the uffd state over in migrate_vma_insert_page() and migrate_vma_insert_huge_pmd_page(). Not migrating the zero page is simpler, and a zero page that is protected by userfaultfd in a range that a driver migrates should be rare.

This basically force-fails a migration of a uffd protected zero page.

Wouldn't we expect this to cause drivers to fail to make progress if they require that page to be resident on the device? Not sure we want to force drivers to try to handle this special case (if they even can).

We'll need some input from the migrate_device/GPU folks.

Thanks, I was halfway through responding last week but Microslop had other ideas.

Failing the migration is fine - drivers can never *require* a page to be resident on the device, as migration can always fail for a host of reasons. In these cases drivers should fallback to remote mapping pages (eg. using hmm_range_fault()).

Quoted 8 lines

Now if userfaultfd+device memory was a commonly used feature then it would be worth fixing this as always failing would have a large performance impact. However I've never tested uffd+device memory and have never been confident the implementation was correct anyway (both features were merged around the same time). So I think the simple fix is fine:

Thanks! In that case I'll leave this in with a trimmed commit message and add some more GPU people to Cc.

Quoted 15 lines
Quoted 3 lines

Yeah, makes sense. I can implement the alternative approach, but testing for this is a little tricky without the right devices... I might drop this patch and send it separately with the GPU folks on Cc.

It probably is worth copying other GPU folk as well though in case they have different use-cases. Matthew Brost (Intel) has done a lot of work in this area in DRM, but I notice he is already on CC.

- Alistair

Quoted 4 lines
Quoted 3 lines
Signature · 1 line
~Gregory
#40 lore raw

PATCH 9/12 mm/madvise: let a guard marker carry the uffd-wp bit

In reply to Tal Zussman

MADV_GUARD_INSTALL can only install guard markers in a range without populated entries. It zaps the range and retries, restarting the syscall if the range still isn't clear. On a file-backed VMA registered for uffd-wp, the zap keeps uffd-wp markers and converts write-protected PTEs into new markers, as the pages are still in the page cache and must stay write-protected. The range never becomes clear, and installing a guard region in write-protected shmem keeps restarting the madvise() call. Signal handlers still run, but the call does not return until the task is killed.

An entry that is write-protected must stay write-protected while a guard is installed on it and after the guard is removed. PTE markers are a bit mask, so a marker can be a guard marker and a uffd-wp marker at the same time. To fix it, let a guard marker carry the uffd-wp bit:

- guard install adds the guard bit to an existing uffd-wp marker
- guard removal clears only the guard bit
- unprotecting or unregistering clears the uffd-wp bit of a guard marker

The fault path, zap, fork, and pagemap already handle a marker with both bits set. However, pte_is_uffd_marker() would let UFFDIO_COPY overwrite the guard, so make it ignore guard markers. Anonymous VMAs drop uffd-wp markers on zap and never get the combined marker.

MADV_GUARD_INSTALL on a uffd-wp shmem range now succeeds instead of hanging. When the guard region is removed again, its pages are still write-protected. While the guard is in place, pagemap reports both PM_GUARD_REGION and PM_UFFD_WP for them.

There are two simpler alternatives, but neither works. Zapping with ZAP_FLAG_DROP_MARKER fixes the hang but drops the write protection: the page is still in the page cache once the guard is removed, and writes to it are not reported. Refusing guard install on uffd-wp file VMAs in is_valid_guard_vma() would break a combination that works on anonymous memory, by adding a new -EINVAL for it.

Fixes f807123d578d ("mm: allow guard regions in file-backed and read-only mappings")
---
 include/linux/leafops.h |  4 ++++
 mm/madvise.c            | 28 ++++++++++++++++++++++++++--
 mm/mprotect.c           | 23 +++++++++++++++++++++--
 3 files changed, 51 insertions(+), 4 deletions(-)
include/linux/leafops.h +4 -0
index 4c1476ae3234..add4968ded44 100644
@@ -578,6 +578,10 @@ static inline bool pte_is_uffd_marker(pte_t pte)
 	if (!softleaf_is_marker(entry))
 		return false;
 
+	/* A guard marker stays a guard, even with the uffd-wp bit set. */
+	if (softleaf_is_guard_marker(entry))
+		return false;
+
 	/* UFFD WP, poisoned swap entries are UFFD-handled. */
 	if (softleaf_is_uffd_wp_marker(entry))
 		return true;
mm/madvise.c +26 -2
index 0214efd2a014..e2c6b0f26672 100644
@@ -1123,6 +1123,23 @@ static int guard_install_pte_entry(pte_t *pte, unsigned long addr,
 		return 0;
 	}
 
+	/*
+	 * Zapping keeps uffd-wp markers on file-backed VMAs, so add the guard
+	 * to the marker instead. The slot stays write-protected once the guard
+	 * is removed.
+	 */
+	if (!vma_is_anonymous(walk->vma) && pte_is_uffd_wp_marker(pteval)) {
+		const softleaf_t entry = softleaf_from_pte(pteval);
+
+		set_pte_at(walk->mm, addr, pte,
+			   make_pte_marker(softleaf_to_marker(entry) |
+					   PTE_MARKER_GUARD));
+		update_mmu_cache(walk->vma, addr, pte);
+		(*nr_pages)++;
+
+		return 0;
+	}
+
 	/* If populated return >0 so we abort the operation + zap. */
 	return 1;
 }
@@ -1259,8 +1276,15 @@ static int guard_remove_pte_entry(pte_t *pte, unsigned long addr,
 	pte_t ptent = ptep_get(pte);
 
 	if (is_guard_pte_marker(ptent)) {
-		/* Simply clear the PTE marker. */
-		pte_clear(walk->mm, addr, pte);
+		const softleaf_t entry = softleaf_from_pte(ptent);
+		const pte_marker marker = softleaf_to_marker(entry) &
+					  ~PTE_MARKER_GUARD;
+
+		/* Clear the guard bit and keep the uffd-wp bit, if set. */
+		if (marker)
+			set_pte_at(walk->mm, addr, pte, make_pte_marker(marker));
+		else
+			pte_clear(walk->mm, addr, pte);
 		update_mmu_cache(walk->vma, addr, pte);
 	}
 
mm/mprotect.c +21 -2
index ef4e7188e24d..6cc5b99b4185 100644
@@ -249,9 +249,28 @@ static long change_softleaf_pte(struct vm_area_struct *vma,
 		 * because any access should sigbus/sigsegv
 		 * anyway.
 		 */
-		if (softleaf_is_poison_marker(entry) ||
-		    softleaf_is_guard_marker(entry))
+		if (softleaf_is_poison_marker(entry))
 			return 0;
+
+		/*
+		 * A guard marker on a file-backed VMA carries the
+		 * uffd-wp state of the slot for when the guard is
+		 * removed.
+		 */
+		if (softleaf_is_guard_marker(entry)) {
+			const pte_marker old = softleaf_to_marker(entry);
+			pte_marker marker = old;
+
+			if (uffd_prot_resolve)
+				marker &= ~PTE_MARKER_UFFD_WP;
+
+			if (marker == old)
+				return 0;
+			set_pte_at(vma->vm_mm, addr, pte,
+				   make_pte_marker(marker));
+			return 1;
+		}
+
 		/*
 		 * If this is uffd-wp pte marker and we'd like
 		 * to unprotect it, drop it; the next page
Signature · 1 line
2.39.5
#41 lore raw

Re: [PATCH 09/12] mm/madvise: let a guard marker carry the uffd-wp bit

Quoted 37 lines · Tal Zussman

MADV_GUARD_INSTALL can only install guard markers in a range without populated entries. It zaps the range and retries, restarting the syscall if the range still isn't clear. On a file-backed VMA registered for uffd-wp, the zap keeps uffd-wp markers and converts write-protected PTEs into new markers, as the pages are still in the page cache and must stay write-protected. The range never becomes clear, and installing a guard region in write-protected shmem keeps restarting the madvise() call. Signal handlers still run, but the call does not return until the task is killed.

An entry that is write-protected must stay write-protected while a guard is installed on it and after the guard is removed. PTE markers are a bit mask, so a marker can be a guard marker and a uffd-wp marker at the same time. To fix it, let a guard marker carry the uffd-wp bit:

- guard install adds the guard bit to an existing uffd-wp marker
- guard removal clears only the guard bit
- unprotecting or unregistering clears the uffd-wp bit of a guard marker

The fault path, zap, fork, and pagemap already handle a marker with both bits set. However, pte_is_uffd_marker() would let UFFDIO_COPY overwrite the guard, so make it ignore guard markers. Anonymous VMAs drop uffd-wp markers on zap and never get the combined marker.

MADV_GUARD_INSTALL on a uffd-wp shmem range now succeeds instead of hanging. When the guard region is removed again, its pages are still write-protected. While the guard is in place, pagemap reports both PM_GUARD_REGION and PM_UFFD_WP for them.

There are two simpler alternatives, but neither works. Zapping with ZAP_FLAG_DROP_MARKER fixes the hang but drops the write protection: the page is still in the page cache once the guard is removed, and writes to it are not reported. Refusing guard install on uffd-wp file VMAs in is_valid_guard_vma() would break a combination that works on anonymous memory, by adding a new -EINVAL for it.

Quoted 2 lines
Fixes f807123d578d ("mm: allow guard regions in file-backed and read-only mappings")

Please drop this patch too.

With my patch disallowing guard marker install on uffd-armed VMAs this goes away (and doesn't make sense anyway.)

I really feel these guard region ones should have been sent separately, btw.

You're establishing a bunch of interlocking dependencies of different changes in this series that makes life hard for reviewers, especially given backports etc.

And now I'm having to essentially inverse cherry pick stuff :)

Quoted 104 lines
---
 include/linux/leafops.h |  4 ++++
 mm/madvise.c            | 28 ++++++++++++++++++++++++++--
 mm/mprotect.c           | 23 +++++++++++++++++++++--
 3 files changed, 51 insertions(+), 4 deletions(-)
include/linux/leafops.h +4 -0
index 4c1476ae3234..add4968ded44 100644
@@ -578,6 +578,10 @@ static inline bool pte_is_uffd_marker(pte_t pte)
 	if (!softleaf_is_marker(entry))
 		return false;
 
+	/* A guard marker stays a guard, even with the uffd-wp bit set. */
+	if (softleaf_is_guard_marker(entry))
+		return false;
+
 	/* UFFD WP, poisoned swap entries are UFFD-handled. */
 	if (softleaf_is_uffd_wp_marker(entry))
 		return true;
mm/madvise.c +26 -2
index 0214efd2a014..e2c6b0f26672 100644
@@ -1123,6 +1123,23 @@ static int guard_install_pte_entry(pte_t *pte, unsigned long addr,
 		return 0;
 	}
 
+	/*
+	 * Zapping keeps uffd-wp markers on file-backed VMAs, so add the guard
+	 * to the marker instead. The slot stays write-protected once the guard
+	 * is removed.
+	 */
+	if (!vma_is_anonymous(walk->vma) && pte_is_uffd_wp_marker(pteval)) {
+		const softleaf_t entry = softleaf_from_pte(pteval);
+
+		set_pte_at(walk->mm, addr, pte,
+			   make_pte_marker(softleaf_to_marker(entry) |
+					   PTE_MARKER_GUARD));
+		update_mmu_cache(walk->vma, addr, pte);
+		(*nr_pages)++;
+
+		return 0;
+	}
+
 	/* If populated return >0 so we abort the operation + zap. */
 	return 1;
 }
@@ -1259,8 +1276,15 @@ static int guard_remove_pte_entry(pte_t *pte, unsigned long addr,
 	pte_t ptent = ptep_get(pte);
 
 	if (is_guard_pte_marker(ptent)) {
-		/* Simply clear the PTE marker. */
-		pte_clear(walk->mm, addr, pte);
+		const softleaf_t entry = softleaf_from_pte(ptent);
+		const pte_marker marker = softleaf_to_marker(entry) &
+					  ~PTE_MARKER_GUARD;
+
+		/* Clear the guard bit and keep the uffd-wp bit, if set. */
+		if (marker)
+			set_pte_at(walk->mm, addr, pte, make_pte_marker(marker));
+		else
+			pte_clear(walk->mm, addr, pte);
 		update_mmu_cache(walk->vma, addr, pte);
 	}
 
mm/mprotect.c +21 -3
index ef4e7188e24d..6cc5b99b4185 100644
@@ -249,9 +249,28 @@ static long change_softleaf_pte(struct vm_area_struct *vma,
 		 * because any access should sigbus/sigsegv
 		 * anyway.
 		 */
-		if (softleaf_is_poison_marker(entry) ||
-		    softleaf_is_guard_marker(entry))
+		if (softleaf_is_poison_marker(entry))
 			return 0;
+
+		/*
+		 * A guard marker on a file-backed VMA carries the
+		 * uffd-wp state of the slot for when the guard is
+		 * removed.
+		 */
+		if (softleaf_is_guard_marker(entry)) {
+			const pte_marker old = softleaf_to_marker(entry);
+			pte_marker marker = old;
+
+			if (uffd_prot_resolve)
+				marker &= ~PTE_MARKER_UFFD_WP;
+
+			if (marker == old)
+				return 0;
+			set_pte_at(vma->vm_mm, addr, pte,
+				   make_pte_marker(marker));
+			return 1;
+		}
+
 		/*
 		 * If this is uffd-wp pte marker and we'd like
 		 * to unprotect it, drop it; the next page
 
--
Quoted 2 lines

2.39.5

--
Cheers, Lorenzo

#42 lore raw

Re: [PATCH 09/12] mm/madvise: let a guard marker carry the uffd-wp bit

Quoted 46 lines · Lorenzo Stoakes (ARM)
Quoted 37 lines · Tal Zussman

MADV_GUARD_INSTALL can only install guard markers in a range without populated entries. It zaps the range and retries, restarting the syscall if the range still isn't clear. On a file-backed VMA registered for uffd-wp, the zap keeps uffd-wp markers and converts write-protected PTEs into new markers, as the pages are still in the page cache and must stay write-protected. The range never becomes clear, and installing a guard region in write-protected shmem keeps restarting the madvise() call. Signal handlers still run, but the call does not return until the task is killed.

An entry that is write-protected must stay write-protected while a guard is installed on it and after the guard is removed. PTE markers are a bit mask, so a marker can be a guard marker and a uffd-wp marker at the same time. To fix it, let a guard marker carry the uffd-wp bit:

- guard install adds the guard bit to an existing uffd-wp marker
- guard removal clears only the guard bit
- unprotecting or unregistering clears the uffd-wp bit of a guard marker

The fault path, zap, fork, and pagemap already handle a marker with both bits set. However, pte_is_uffd_marker() would let UFFDIO_COPY overwrite the guard, so make it ignore guard markers. Anonymous VMAs drop uffd-wp markers on zap and never get the combined marker.

MADV_GUARD_INSTALL on a uffd-wp shmem range now succeeds instead of hanging. When the guard region is removed again, its pages are still write-protected. While the guard is in place, pagemap reports both PM_GUARD_REGION and PM_UFFD_WP for them.

There are two simpler alternatives, but neither works. Zapping with ZAP_FLAG_DROP_MARKER fixes the hang but drops the write protection: the page is still in the page cache once the guard is removed, and writes to it are not reported. Refusing guard install on uffd-wp file VMAs in is_valid_guard_vma() would break a combination that works on anonymous memory, by adding a new -EINVAL for it.

Quoted 2 lines
Fixes f807123d578d ("mm: allow guard regions in file-backed and read-only mappings")

Please drop this patch too.

With my patch disallowing guard marker install on uffd-armed VMAs this goes away (and doesn't make sense anyway.)

Quoted 7 lines

I really feel these guard region ones should have been sent separately, btw.

You're establishing a bunch of interlocking dependencies of different changes in this series that makes life hard for reviewers, especially given backports etc.

And now I'm having to essentially inverse cherry pick stuff :)

Sorry about that. These started off as one fix and then grew a couple more on further rounds of AI review...

I'll drop this and the other two guard patches.

Quoted 110 lines
Quoted 104 lines
---
 include/linux/leafops.h |  4 ++++
 mm/madvise.c            | 28 ++++++++++++++++++++++++++--
 mm/mprotect.c           | 23 +++++++++++++++++++++--
 3 files changed, 51 insertions(+), 4 deletions(-)
include/linux/leafops.h +4 -0
index 4c1476ae3234..add4968ded44 100644
@@ -578,6 +578,10 @@ static inline bool pte_is_uffd_marker(pte_t pte)
 	if (!softleaf_is_marker(entry))
 		return false;
 
+	/* A guard marker stays a guard, even with the uffd-wp bit set. */
+	if (softleaf_is_guard_marker(entry))
+		return false;
+
 	/* UFFD WP, poisoned swap entries are UFFD-handled. */
 	if (softleaf_is_uffd_wp_marker(entry))
 		return true;
mm/madvise.c +26 -2
index 0214efd2a014..e2c6b0f26672 100644
@@ -1123,6 +1123,23 @@ static int guard_install_pte_entry(pte_t *pte, unsigned long addr,
 		return 0;
 	}
 
+	/*
+	 * Zapping keeps uffd-wp markers on file-backed VMAs, so add the guard
+	 * to the marker instead. The slot stays write-protected once the guard
+	 * is removed.
+	 */
+	if (!vma_is_anonymous(walk->vma) && pte_is_uffd_wp_marker(pteval)) {
+		const softleaf_t entry = softleaf_from_pte(pteval);
+
+		set_pte_at(walk->mm, addr, pte,
+			   make_pte_marker(softleaf_to_marker(entry) |
+					   PTE_MARKER_GUARD));
+		update_mmu_cache(walk->vma, addr, pte);
+		(*nr_pages)++;
+
+		return 0;
+	}
+
 	/* If populated return >0 so we abort the operation + zap. */
 	return 1;
 }
@@ -1259,8 +1276,15 @@ static int guard_remove_pte_entry(pte_t *pte, unsigned long addr,
 	pte_t ptent = ptep_get(pte);
 
 	if (is_guard_pte_marker(ptent)) {
-		/* Simply clear the PTE marker. */
-		pte_clear(walk->mm, addr, pte);
+		const softleaf_t entry = softleaf_from_pte(ptent);
+		const pte_marker marker = softleaf_to_marker(entry) &
+					  ~PTE_MARKER_GUARD;
+
+		/* Clear the guard bit and keep the uffd-wp bit, if set. */
+		if (marker)
+			set_pte_at(walk->mm, addr, pte, make_pte_marker(marker));
+		else
+			pte_clear(walk->mm, addr, pte);
 		update_mmu_cache(walk->vma, addr, pte);
 	}
 
mm/mprotect.c +21 -3
index ef4e7188e24d..6cc5b99b4185 100644
@@ -249,9 +249,28 @@ static long change_softleaf_pte(struct vm_area_struct *vma,
 		 * because any access should sigbus/sigsegv
 		 * anyway.
 		 */
-		if (softleaf_is_poison_marker(entry) ||
-		    softleaf_is_guard_marker(entry))
+		if (softleaf_is_poison_marker(entry))
 			return 0;
+
+		/*
+		 * A guard marker on a file-backed VMA carries the
+		 * uffd-wp state of the slot for when the guard is
+		 * removed.
+		 */
+		if (softleaf_is_guard_marker(entry)) {
+			const pte_marker old = softleaf_to_marker(entry);
+			pte_marker marker = old;
+
+			if (uffd_prot_resolve)
+				marker &= ~PTE_MARKER_UFFD_WP;
+
+			if (marker == old)
+				return 0;
+			set_pte_at(vma->vm_mm, addr, pte,
+				   make_pte_marker(marker));
+			return 1;
+		}
+
 		/*
 		 * If this is uffd-wp pte marker and we'd like
 		 * to unprotect it, drop it; the next page
 
--
Quoted 2 lines

2.39.5

--
Cheers, Lorenzo

#43 lore raw

PATCH 10/12 mm: set the uffd-wp bit on a guard marker when write-protecting

In reply to Tal Zussman

UFFDIO_WRITEPROTECT and PAGEMAP_SCAN with PM_SCAN_WP_MATCHING do not write-protect an entry that holds a guard marker. change_softleaf_pte() ignores guard markers, and make_uffd_wp_pte() only sets the swap uffd bit on the marker PTE, which guard removal clears along with the marker.

On a file-backed VMA, the page is still in the page cache once the guard is removed. It faults back in writable, and writes to it are not reported.

To fix it, set the uffd-wp bit on a guard marker of a file-backed VMA in both paths. A guard marker can carry that bit, and guard removal then keeps the write protection by leaving a uffd-wp marker in its place.

Move is_guard_pte_marker() to leafops.h as pte_is_guard_marker(), next to pte_is_uffd_wp_marker(), so that make_uffd_wp_pte() can use it.

A page that is write-protected while a guard is in place is now still write-protected after the guard is removed.

Anonymous memory is left as it is. With UFFD_FEATURE_WP_UNPOPULATED, an entry that is write-protected while a guard is in place is still unprotected after the guard is removed.

Fixes f807123d578d ("mm: allow guard regions in file-backed and read-only mappings")
---
 fs/proc/task_mmu.c      |  7 +++++++
 include/linux/leafops.h | 14 ++++++++++++++
 mm/madvise.c            | 11 ++---------
 mm/mprotect.c           |  5 ++++-
 4 files changed, 27 insertions(+), 10 deletions(-)
fs/proc/task_mmu.c +7 -0
index 752f1b13d47a..7b1c843a4fd0 100644
@@ -2439,6 +2439,13 @@ static void make_uffd_wp_pte(struct vm_area_struct *vma,
 	} else if (pte_none(ptent)) {
 		set_pte_at(vma->vm_mm, addr, pte,
 			   make_pte_marker(PTE_MARKER_UFFD_WP));
+	} else if (!vma_is_anonymous(vma) && pte_is_guard_marker(ptent)) {
+		/* Keep the protection in the marker, as mprotect does. */
+		const softleaf_t entry = softleaf_from_pte(ptent);
+
+		set_pte_at(vma->vm_mm, addr, pte,
+			   make_pte_marker(softleaf_to_marker(entry) |
+					   PTE_MARKER_UFFD_WP));
 	} else {
 		ptent = set_uffd_wp_swp_pte(ptent);
 		set_pte_at(vma->vm_mm, addr, pte, ptent);
include/linux/leafops.h +14 -0
index add4968ded44..831b85a184c4 100644
@@ -547,6 +547,20 @@ static inline bool pte_is_marker(pte_t pte)
 	return softleaf_is_marker(softleaf_from_pte(pte));
 }
 
+/**
+ * pte_is_guard_marker() - Does this PTE entry encode a guard region marker
+ * leaf entry?
+ * @pte: PTE entry.
+ *
+ * Returns: true if this PTE is a guard marker leaf entry, otherwise false.
+ */
+static inline bool pte_is_guard_marker(pte_t pte)
+{
+	const softleaf_t entry = softleaf_from_pte(pte);
+
+	return softleaf_is_guard_marker(entry);
+}
+
 /**
  * pte_is_uffd_wp_marker() - Does this PTE entry encode a userfaultfd write
  * protect marker leaf entry?
mm/madvise.c +2 -9
index e2c6b0f26672..e8b05dc899f1 100644
@@ -1085,13 +1085,6 @@ static bool is_valid_guard_vma(const struct vm_area_struct *vma,
 	return true;
 }
 
-static bool is_guard_pte_marker(pte_t ptent)
-{
-	const softleaf_t entry = softleaf_from_pte(ptent);
-
-	return softleaf_is_guard_marker(entry);
-}
-
 static int guard_install_pud_entry(pud_t *pud, unsigned long addr,
 				   unsigned long next, struct mm_walk *walk)
 {
@@ -1117,7 +1110,7 @@ static int guard_install_pte_entry(pte_t *pte, unsigned long addr,
 	unsigned long *nr_pages = (unsigned long *)walk->private;
 
 	/* If there is already a guard page marker, we have nothing to do. */
-	if (is_guard_pte_marker(pteval)) {
+	if (pte_is_guard_marker(pteval)) {
 		(*nr_pages)++;
 
 		return 0;
@@ -1275,7 +1268,7 @@ static int guard_remove_pte_entry(pte_t *pte, unsigned long addr,
 {
 	pte_t ptent = ptep_get(pte);
 
-	if (is_guard_pte_marker(ptent)) {
+	if (pte_is_guard_marker(ptent)) {
 		const softleaf_t entry = softleaf_from_pte(ptent);
 		const pte_marker marker = softleaf_to_marker(entry) &
 					  ~PTE_MARKER_GUARD;
mm/mprotect.c +4 -1
index 6cc5b99b4185..3f2d6e3aca93 100644
@@ -261,7 +261,10 @@ static long change_softleaf_pte(struct vm_area_struct *vma,
 			const pte_marker old = softleaf_to_marker(entry);
 			pte_marker marker = old;
 
-			if (uffd_prot_resolve)
+			if ((cp_flags & MM_CP_UFFD_WP) &&
+			    !vma_is_anonymous(vma))
+				marker |= PTE_MARKER_UFFD_WP;
+			else if (uffd_prot_resolve)
 				marker &= ~PTE_MARKER_UFFD_WP;
 
 			if (marker == old)
Signature · 1 line
2.39.5
#44 lore raw

Re: [PATCH 10/12] mm: set the uffd-wp bit on a guard marker when write-protecting

Quoted 4 lines · Tal Zussman

UFFDIO_WRITEPROTECT and PAGEMAP_SCAN with PM_SCAN_WP_MATCHING do not write-protect an entry that holds a guard marker. change_softleaf_pte() ignores guard markers, and make_uffd_wp_pte() only sets the swap uffd bit on the marker PTE, which guard removal clears along with the marker.

There's no 'swap uffd' bit, softleaf uffd bit :) (I need to rename some stuff still...)

Quoted 15 lines

On a file-backed VMA, the page is still in the page cache once the guard is removed. It faults back in writable, and writes to it are not reported.

To fix it, set the uffd-wp bit on a guard marker of a file-backed VMA in both paths. A guard marker can carry that bit, and guard removal then keeps the write protection by leaving a uffd-wp marker in its place.

Move is_guard_pte_marker() to leafops.h as pte_is_guard_marker(), next to pte_is_uffd_wp_marker(), so that make_uffd_wp_pte() can use it.

A page that is write-protected while a guard is in place is now still write-protected after the guard is removed.

Quoted 6 lines

Anonymous memory is left as it is. With UFFD_FEATURE_WP_UNPOPULATED, an entry that is write-protected while a guard is in place is still unprotected after the guard is removed.

Fixes f807123d578d ("mm: allow guard regions in file-backed and read-only mappings")

Similarly, please drop this patch.

This does leave the opposite order (guard region _then_ uffd register etc.) unaddressed, but guard regions behave as if the range were unmapped, and uffd-wp state does not survive an unmap/remap either, so I think it's reasonable for WP not to propagate there.

Quoted 104 lines
---
 fs/proc/task_mmu.c      |  7 +++++++
 include/linux/leafops.h | 14 ++++++++++++++
 mm/madvise.c            | 11 ++---------
 mm/mprotect.c           |  5 ++++-
 4 files changed, 27 insertions(+), 10 deletions(-)
fs/proc/task_mmu.c +7 -0
index 752f1b13d47a..7b1c843a4fd0 100644
@@ -2439,6 +2439,13 @@ static void make_uffd_wp_pte(struct vm_area_struct *vma,
 	} else if (pte_none(ptent)) {
 		set_pte_at(vma->vm_mm, addr, pte,
 			   make_pte_marker(PTE_MARKER_UFFD_WP));
+	} else if (!vma_is_anonymous(vma) && pte_is_guard_marker(ptent)) {
+		/* Keep the protection in the marker, as mprotect does. */
+		const softleaf_t entry = softleaf_from_pte(ptent);
+
+		set_pte_at(vma->vm_mm, addr, pte,
+			   make_pte_marker(softleaf_to_marker(entry) |
+					   PTE_MARKER_UFFD_WP));
 	} else {
 		ptent = set_uffd_wp_swp_pte(ptent);
 		set_pte_at(vma->vm_mm, addr, pte, ptent);
include/linux/leafops.h +14 -0
index add4968ded44..831b85a184c4 100644
@@ -547,6 +547,20 @@ static inline bool pte_is_marker(pte_t pte)
 	return softleaf_is_marker(softleaf_from_pte(pte));
 }
 
+/**
+ * pte_is_guard_marker() - Does this PTE entry encode a guard region marker
+ * leaf entry?
+ * @pte: PTE entry.
+ *
+ * Returns: true if this PTE is a guard marker leaf entry, otherwise false.
+ */
+static inline bool pte_is_guard_marker(pte_t pte)
+{
+	const softleaf_t entry = softleaf_from_pte(pte);
+
+	return softleaf_is_guard_marker(entry);
+}
+
 /**
  * pte_is_uffd_wp_marker() - Does this PTE entry encode a userfaultfd write
  * protect marker leaf entry?
mm/madvise.c +2 -9
index e2c6b0f26672..e8b05dc899f1 100644
@@ -1085,13 +1085,6 @@ static bool is_valid_guard_vma(const struct vm_area_struct *vma,
 	return true;
 }
 
-static bool is_guard_pte_marker(pte_t ptent)
-{
-	const softleaf_t entry = softleaf_from_pte(ptent);
-
-	return softleaf_is_guard_marker(entry);
-}
-
 static int guard_install_pud_entry(pud_t *pud, unsigned long addr,
 				   unsigned long next, struct mm_walk *walk)
 {
@@ -1117,7 +1110,7 @@ static int guard_install_pte_entry(pte_t *pte, unsigned long addr,
 	unsigned long *nr_pages = (unsigned long *)walk->private;
 
 	/* If there is already a guard page marker, we have nothing to do. */
-	if (is_guard_pte_marker(pteval)) {
+	if (pte_is_guard_marker(pteval)) {
 		(*nr_pages)++;
 
 		return 0;
@@ -1275,7 +1268,7 @@ static int guard_remove_pte_entry(pte_t *pte, unsigned long addr,
 {
 	pte_t ptent = ptep_get(pte);
 
-	if (is_guard_pte_marker(ptent)) {
+	if (pte_is_guard_marker(ptent)) {
 		const softleaf_t entry = softleaf_from_pte(ptent);
 		const pte_marker marker = softleaf_to_marker(entry) &
 					  ~PTE_MARKER_GUARD;
mm/mprotect.c +4 -2
index 6cc5b99b4185..3f2d6e3aca93 100644
@@ -261,7 +261,10 @@ static long change_softleaf_pte(struct vm_area_struct *vma,
 			const pte_marker old = softleaf_to_marker(entry);
 			pte_marker marker = old;
 
-			if (uffd_prot_resolve)
+			if ((cp_flags & MM_CP_UFFD_WP) &&
+			    !vma_is_anonymous(vma))
+				marker |= PTE_MARKER_UFFD_WP;
+			else if (uffd_prot_resolve)
 				marker &= ~PTE_MARKER_UFFD_WP;
 
 			if (marker == old)
 
--
Quoted 2 lines

2.39.5

--
Cheers, Lorenzo

#45 lore raw

PATCH 11/12 mm/madvise: send UFFD_EVENT_REMOVE when installing guard regions

In reply to Tal Zussman

MADV_GUARD_INSTALL zaps whatever is mapped in the range before installing the guard markers. However, unlike MADV_DONTNEED and MADV_REMOVE, madvise_guard_install() does not call userfaultfd_remove(), so the userfaultfd manager is not told that the pages are gone.

On an anonymous VMA, the zap drops uffd-wp and RWP protection along with the pages. Writes and accesses to the range after the guard is removed are then not reported, and the manager cannot know that it has to protect the range again.

To fix it, send UFFD_EVENT_REMOVE before zapping. Guard install on a VMA registered with userfaultfd already takes the mmap lock instead of the VMA lock, so we can drop the lock for the event and look the VMA up again afterwards.

As with MADV_DONTNEED, the call can now return -ENOMEM if the VMA goes away while the lock is dropped, or -EINVAL if it is replaced by a VMA that does not support guard regions.

Fixes 662df3e5c376 ("mm: madvise: implement lightweight guard page mechanism")
---
 mm/madvise.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
mm/madvise.c +23 -0
index e8b05dc899f1..6413f0e65381 100644
@@ -1166,6 +1166,29 @@ static long madvise_guard_install(struct madvise_behavior *madv_behavior)
 	if (!is_valid_guard_vma(vma, /* allow_locked = */false))
 		return -EINVAL;
 
+	/*
+	 * Installing guards zaps the range, so report it to a userfaultfd
+	 * manager like MADV_DONTNEED does. The VMA can change while the mmap
+	 * lock is dropped for the event, so look it up again.
+	 */
+	if (!userfaultfd_remove(vma, range->start, range->end)) {
+		mark_mmap_lock_dropped(madv_behavior);
+		mmap_read_lock(madv_behavior->mm);
+		madv_behavior->vma = vma = vma_lookup(madv_behavior->mm,
+						      range->start);
+		if (!vma)
+			return -ENOMEM;
+		if (!is_valid_guard_vma(vma, /* allow_locked = */false))
+			return -EINVAL;
+		/*
+		 * The VMA may have been split while the lock was dropped.
+		 * Handle the part in this VMA. The remainder is handled by the
+		 * next iteration of madvise_walk_vmas().
+		 */
+		if (range->end > vma->vm_end)
+			range->end = vma->vm_end;
+	}
+
 	/*
 	 * Set atomically under read lock. All pertinent readers will need to
 	 * acquire an mmap/VMA write lock to read it. All remaining readers may
Signature · 1 line
2.39.5
#46 lore raw

Re: [PATCH 11/12] mm/madvise: send UFFD_EVENT_REMOVE when installing guard regions

Quoted 4 lines · Tal Zussman

On an anonymous VMA, the zap drops uffd-wp and RWP protection along with the pages. Writes and accesses to the range after the guard is removed are then not reported, and the manager cannot know that it has to protect the range again.

For RWP this only holds if the range is registered without MISSING.

The zap frees the pages, so with MISSING the first access after the guard is removed is reported as a missing fault. And there is nothing to protect again: RWP only covers present PTEs, and a page faulted back in shows up as accessed in PAGEMAP_SCAN.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#47 lore raw

Re: [PATCH 11/12] mm/madvise: send UFFD_EVENT_REMOVE when installing guard regions

+ Jose

Quoted 6 lines · Kiryl Shutsemau
Quoted 4 lines · Tal Zussman

On an anonymous VMA, the zap drops uffd-wp and RWP protection along with the pages. Writes and accesses to the range after the guard is removed are then not reported, and the manager cannot know that it has to protect the range again.

Quoted 7 lines

For RWP this only holds if the range is registered without MISSING.

The zap frees the pages, so with MISSING the first access after the guard is removed is reported as a missing fault. And there is nothing to protect again: RWP only covers present PTEs, and a page faulted back in shows up as accessed in PAGEMAP_SCAN.

So this makes sense, but I think this gets at the documentation point raised in the other thread. It seems that to use sync RWP "properly" it needs to be combined with MINOR for shmem and hugetlb (and MISSING for holes) and with MISSING for anonymous memory. Is that accurate? If so, this should be stated much more explicitly in the documentation + man page. I see that co-registering with MISSING is mentioned, but not the rest of the points (reclaim, collapse, etc.) The example in userfaultfd.rst also registers a MAP_SHARED mapping only with RWP and MISSING, no MINOR, so that should be clarified too. Hopefully this will make it into Jose's documentation patch.

In any case, this change is also needed for MISSING faults. If a guard discards the contents of a page registered with MISSING, the manager isn't told. Once the guard is removed and the page is faulted in again, the manager may fill it with the old contents, when instead it should have discarded them and filled it with zeroes. This is similar to MADV_DONTNEED's handling. I'll update the commit message to emphasize this point rather than the WP/RWP aspect.

Thanks,
Tal

#48 lore raw

Re: [PATCH 11/12] mm/madvise: send UFFD_EVENT_REMOVE when installing guard regions

Quoted 16 lines · Tal Zussman

+ Jose

Quoted 6 lines · Kiryl Shutsemau
Quoted 4 lines · Tal Zussman

On an anonymous VMA, the zap drops uffd-wp and RWP protection along with the pages. Writes and accesses to the range after the guard is removed are then not reported, and the manager cannot know that it has to protect the range again.

Quoted 7 lines

For RWP this only holds if the range is registered without MISSING.

The zap frees the pages, so with MISSING the first access after the guard is removed is reported as a missing fault. And there is nothing to protect again: RWP only covers present PTEs, and a page faulted back in shows up as accessed in PAGEMAP_SCAN.

Quoted 10 lines

So this makes sense, but I think this gets at the documentation point raised in the other thread. It seems that to use sync RWP "properly" it needs to be combined with MINOR for shmem and hugetlb (and MISSING for holes) and with MISSING for anonymous memory. Is that accurate? If so, this should be stated much more explicitly in the documentation + man page. I see that co-registering with MISSING is mentioned, but not the rest of the points (reclaim, collapse, etc.) The example in userfaultfd.rst also registers a MAP_SHARED mapping only with RWP and MISSING, no MINOR, so that should be clarified too. Hopefully this will make it into Jose's documentation patch.

okay, if Kiryl is okay with that I will add it to my upcoming patch.

Signature · 2 lines
cheers,
jose a. p-a
#49 lore raw

Re: [PATCH 11/12] mm/madvise: send UFFD_EVENT_REMOVE when installing guard regions

Quoted 17 lines · Jose A. Perez de Azpillaga
Quoted 16 lines · Tal Zussman

+ Jose

Quoted 13 lines · Kiryl Shutsemau
Quoted 4 lines · Tal Zussman

On an anonymous VMA, the zap drops uffd-wp and RWP protection along with the pages. Writes and accesses to the range after the guard is removed are then not reported, and the manager cannot know that it has to protect the range again.

For RWP this only holds if the range is registered without MISSING.

The zap frees the pages, so with MISSING the first access after the guard is removed is reported as a missing fault. And there is nothing to protect again: RWP only covers present PTEs, and a page faulted back in shows up as accessed in PAGEMAP_SCAN.

Quoted 4 lines
Quoted 4 lines

So this makes sense, but I think this gets at the documentation point raised in the other thread. It seems that to use sync RWP "properly" it needs to be combined with MINOR for shmem and hugetlb (and MISSING for holes) and with MISSING for anonymous memory. Is that accurate?

Yes. Async mode needs neither: a page faulted back in is mapped without the uffd bit and PAGEMAP_SCAN reports it as accessed.

Quoted 6 lines
Quoted 6 lines

If so, this should be stated much more explicitly in the documentation + man page. I see that co-registering with MISSING is mentioned, but not the rest of the points (reclaim, collapse, etc.) The example in userfaultfd.rst also registers a MAP_SHARED mapping only with RWP and MISSING, no MINOR, so that should be clarified too.

Without MINOR the example can lose data. Eviction runs in sync mode, but an offset that is not mapped in guest_mem, either pre-populated or zapped by shmem reclaim or collapse, is mapped back on access without a fault. A vCPU write between the pwrite() and the hole punch is lost. With MINOR the access faults and waits for the handler, like an RWP fault.

Quoted 3 lines
Quoted 1 line

Hopefully this will make it into Jose's documentation patch.

okay, if Kiryl is okay with that I will add it to my upcoming patch.

Yes, please.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#50 lore raw

Re: [PATCH 11/12] mm/madvise: send UFFD_EVENT_REMOVE when installing guard regions

In reply to Tal Zussman
Quoted 15 lines · Tal Zussman

MADV_GUARD_INSTALL zaps whatever is mapped in the range before installing the guard markers. However, unlike MADV_DONTNEED and MADV_REMOVE, madvise_guard_install() does not call userfaultfd_remove(), so the userfaultfd manager is not told that the pages are gone.

On an anonymous VMA, the zap drops uffd-wp and RWP protection along with the pages. Writes and accesses to the range after the guard is removed are then not reported, and the manager cannot know that it has to protect the range again.

To fix it, send UFFD_EVENT_REMOVE before zapping. Guard install on a VMA registered with userfaultfd already takes the mmap lock instead of the VMA lock, so we can drop the lock for the event and look the VMA up again afterwards.

Quoted 6 lines

As with MADV_DONTNEED, the call can now return -ENOMEM if the VMA goes away while the lock is dropped, or -EINVAL if it is replaced by a VMA that does not support guard regions.

Fixes 662df3e5c376 ("mm: madvise: implement lightweight guard page mechanism")

This report was super-synthetic AI slop

Quoted 1 line

Sorry I hate this.

Since this was already silently broken, I think the right solution is to simply disallow guard region installation for uffd-armed VMAs.

I had the AI go look for users, and it seems nobody is relying on this (glibc, android, CRIU, etc.), and there's really no sensible world in which somebody's combining the two. So I think fine to do that.

It was simply an oversight (reflex action is to disable new features for hugetlb
+ uffd)

I'll send a patch separate from this, please drop it from the series.

Quoted 25 lines
---
 mm/madvise.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
mm/madvise.c +13 -0
index e8b05dc899f1..6413f0e65381 100644
@@ -1166,6 +1166,29 @@ static long madvise_guard_install(struct madvise_behavior *madv_behavior)
 	if (!is_valid_guard_vma(vma, /* allow_locked = */false))
 		return -EINVAL;
 
+	/*
+	 * Installing guards zaps the range, so report it to a userfaultfd
+	 * manager like MADV_DONTNEED does. The VMA can change while the mmap
+	 * lock is dropped for the event, so look it up again.
+	 */
+	if (!userfaultfd_remove(vma, range->start, range->end)) {
+		mark_mmap_lock_dropped(madv_behavior);
+		mmap_read_lock(madv_behavior->mm);
+		madv_behavior->vma = vma = vma_lookup(madv_behavior->mm,
+						      range->start);
+		if (!vma)
+			return -ENOMEM;
+		if (!is_valid_guard_vma(vma, /* allow_locked = */false))

mm/madvise.c static long madvise_guard_install(struct madvise_behavior *madv_behavior) · line 1191

Quoted 10 lines
mm/madvise.c +10 -0
+			return -EINVAL;
+		/*
+		 * The VMA may have been split while the lock was dropped.
+		 * Handle the part in this VMA. The remainder is handled by the
+		 * next iteration of madvise_walk_vmas().
+		 */
+		if (range->end > vma->vm_end)
+			range->end = vma->vm_end;
+	}
+

(separate rant - that we have to a. know to do it and b. write duplicative open-coded crap like this is a perfect example of why uffd is such an utterly broken mm feature that should never have been merged...)

Quoted 7 lines
mm/madvise.c +0 -1
 	/*
 	 * Set atomically under read lock. All pertinent readers will need to
 	 * acquire an mmap/VMA write lock to read it. All remaining readers may
 
--

2.39.5

--
Cheers, Lorenzo

#51 lore raw

Re: [PATCH 11/12] mm/madvise: send UFFD_EVENT_REMOVE when installing guard regions

Quoted 5 lines · Lorenzo Stoakes (ARM)
Quoted 2 lines
Fixes 662df3e5c376 ("mm: madvise: implement lightweight guard page mechanism")

This report was super-synthetic AI slop

Oops, let me finish this thought... it was super-synthetic AI slop to the point that I am not sure it's even worth referencing.

--
Cheers, Lorenzo

#52 lore raw

Re: [PATCH 11/12] mm/madvise: send UFFD_EVENT_REMOVE when installing guard regions

In reply to Lorenzo Stoakes (ARM)
Quoted 20 lines · Lorenzo Stoakes (ARM)
Quoted 19 lines · Tal Zussman

MADV_GUARD_INSTALL zaps whatever is mapped in the range before installing the guard markers. However, unlike MADV_DONTNEED and MADV_REMOVE, madvise_guard_install() does not call userfaultfd_remove(), so the userfaultfd manager is not told that the pages are gone.

On an anonymous VMA, the zap drops uffd-wp and RWP protection along with the pages. Writes and accesses to the range after the guard is removed are then not reported, and the manager cannot know that it has to protect the range again.

To fix it, send UFFD_EVENT_REMOVE before zapping. Guard install on a VMA registered with userfaultfd already takes the mmap lock instead of the VMA lock, so we can drop the lock for the event and look the VMA up again afterwards.

As with MADV_DONTNEED, the call can now return -ENOMEM if the VMA goes away while the lock is dropped, or -EINVAL if it is replaced by a VMA that does not support guard regions.

Quoted 7 lines
Quoted 2 lines
Fixes 662df3e5c376 ("mm: madvise: implement lightweight guard page mechanism")

mm/[email protected]/__;!!BDUfV1Et5lrpZQ!
T5BLvK-D7GgO7Y87n7ULpXAEzR6V57uNMU7saDR9ShW05mDBxMyqFA4DD22Nbycn_r7rhjPtEZp8gBAN$ <https://urldefense.com/v3/__https://lore.kernel.org/linux-mm/[email protected]/__;!!BDUfV1Et5lrpZQ!T5BLvK-D7GgO7Y87n7ULpXAEzR6V57uNMU7saDR9ShW05mDBxMyqFA4DD22Nbycn_r7rhjPtEZp8gBAN$>

This report was super-synthetic AI slop

Will drop it

Quoted 4 lines
Quoted 1 line

Sorry I hate this.

Don't worry, me too...

Quoted 7 lines

Since this was already silently broken, I think the right solution is to simply disallow guard region installation for uffd-armed VMAs.

I had the AI go look for users, and it seems nobody is relying on this (glibc, android, CRIU, etc.), and there's really no sensible world in which somebody's combining the two. So I think fine to do that.

What about CRIU restoring an application that uses guard regions? CRIU registers the task's memory with uffd MISSING mode for lazy restore, and if the application (e.g., glibc) is using guard regions, this could break. I believe glibc falls back to mprotect() after guard installation failure, but I'm not sure about other users. I agree that there's no sensible reason for a single application to combine these two, but uffd manager mode is where we get these fun combinations...

Re: "silently broken", WP and RWP are more broken here than MISSING and MINOR modes (due to the issues in the other two guard patches). MINOR and MISSING mode already work today, modulo this event notification. If the CRIU issue above matters, disallowing guard regions on WP/RWP VMAs could be a good middle ground (and of course this should all be documented somewhere...). Otherwise disallowing guard regions on uffd entirely sounds reasonable to me.

Quoted 5 lines

It was simply an oversight (reflex action is to disable new features for hugetlb
+ uffd)

I'll send a patch separate from this, please drop it from the series.

Will do!

Quoted 31 lines
Quoted 31 lines
---
 mm/madvise.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
mm/madvise.c +19 -0
index e8b05dc899f1..6413f0e65381 100644
@@ -1166,6 +1166,29 @@ static long madvise_guard_install(struct madvise_behavior *madv_behavior)
 	if (!is_valid_guard_vma(vma, /* allow_locked = */false))
 		return -EINVAL;
 
+	/*
+	 * Installing guards zaps the range, so report it to a userfaultfd
+	 * manager like MADV_DONTNEED does. The VMA can change while the mmap
+	 * lock is dropped for the event, so look it up again.
+	 */
+	if (!userfaultfd_remove(vma, range->start, range->end)) {
+		mark_mmap_lock_dropped(madv_behavior);
+		mmap_read_lock(madv_behavior->mm);
+		madv_behavior->vma = vma = vma_lookup(madv_behavior->mm,
+						      range->start);
+		if (!vma)
+			return -ENOMEM;
+		if (!is_valid_guard_vma(vma, /* allow_locked = */false))
+			return -EINVAL;
+		/*
+		 * The VMA may have been split while the lock was dropped.
+		 * Handle the part in this VMA. The remainder is handled by the
+		 * next iteration of madvise_walk_vmas().
+		 */
Quoted 9 lines

mm/madvise.c static long madvise_guard_install(struct madvise_behavior *madv_behavior) · line 1191

Quoted 4 lines
mm/madvise.c +4 -0
+		if (range->end > vma->vm_end)
+			range->end = vma->vm_end;
+	}
+

(separate rant - that we have to a. know to do it and b. write duplicative open-coded crap like this is a perfect example of why uffd is such an utterly broken mm feature that should never have been merged...)

But where would be the fun in that... ;)

Quoted 11 lines
Quoted 7 lines
mm/madvise.c +0 -1
 	/*
 	 * Set atomically under read lock. All pertinent readers will need to
 	 * acquire an mmap/VMA write lock to read it. All remaining readers may
 
--

2.39.5

--
Cheers, Lorenzo

#53 lore raw

Re: [PATCH 11/12] mm/madvise: send UFFD_EVENT_REMOVE when installing guard regions

In reply to Tal Zussman
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] mm/madvise: missing can_madvise_modify() recheck in madvise_guard_install() --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] mm/madvise: missing can_madvise_modify() recheck in madvise_guard_install()
--

commit 8be4fedce36f0b45069f1f630f0d9241d579512f
Author: Tal Zussman <[email protected]>

mm/madvise: send UFFD_EVENT_REMOVE when installing guard regions

This commit updates madvise_guard_install() to call userfaultfd_remove() before zapping the target range and re-look up the VMA if the mmap lock was dropped.

Quoted 12 lines
mm/madvise.c +4 -0
index e8b05dc899f10..6413f0e65381b 100644
@@ -1166,6 +1166,29 @@ static long madvise_guard_install(struct madvise_behavior *madv_behavior)
 	if (!is_valid_guard_vma(vma, /* allow_locked = */false))
 		return -EINVAL;
 
+	/*
+	 * Installing guards zaps the range, so report it to a userfaultfd
+	 * manager like MADV_DONTNEED does. The VMA can change while the mmap
+	 * lock is dropped for the event, so look it up again.

mm/madvise.c static long madvise_guard_install(struct madvise_behavior *madv_behavior) · line 1182

Quoted 10 lines
mm/madvise.c +10 -0
+	 */
+	if (!userfaultfd_remove(vma, range->start, range->end)) {
+		mark_mmap_lock_dropped(madv_behavior);
+		mmap_read_lock(madv_behavior->mm);
+		madv_behavior->vma = vma = vma_lookup(madv_behavior->mm,
+						      range->start);
+		if (!vma)
+			return -ENOMEM;
+		if (!is_valid_guard_vma(vma, /* allow_locked = */false))
+			return -EINVAL;

[Severity: Medium]
Should this also recheck can_madvise_modify(madv_behavior) after looking up the VMA again?

madvise_vma_behavior() checks can_madvise_modify() before calling madvise_guard_install(). While mm->mmap_lock is dropped inside userfaultfd_remove(), another thread can make the VMA read-only via mprotect(PROT_READ) and seal it with mseal(). Because is_valid_guard_vma() only checks VMA_LOCKED_BIT, vma_is_mm_backed() and vma_is_hugetlb(), madvise_guard_install() will then proceed to zap pages and install guard markers on the sealed read-only VMA instead of returning -EPERM.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
#54 lore raw

PATCH 12/12 mm/userfaultfd: check hugetlb end alignment in UFFDIO_UNREGISTER pre-scan

In reply to Tal Zussman

UFFDIO_REGISTER and UFFDIO_UNREGISTER both scan the VMAs of the range before changing anything. For a hugetlb VMA, userfaultfd_register() rejects a range whose start or end is not huge page aligned there. However, userfaultfd_unregister() only checks the start.

An unaligned end only fails later, when vma_modify_flags_uffd() cannot split the VMA there. At that point, userfaultfd_clear_vma() already cleared the WP or RWP protection. The ioctl returns -EINVAL and the VMA stays registered, but accesses to its pages are no longer reported. This requires a caller passing a misaligned range.

To fix it, check the end of a registered VMA in the pre-scan as well. The ioctl still returns -EINVAL, but no longer unregisters the VMAs before the misaligned one first.

Fixes cab350afcbc9 ("userfaultfd: hugetlbfs: allow registration of ranges containing huge pages")
Fixes f369b07c8614 ("mm/uffd: reset write protection when unregister with wp-mode")
---
 mm/userfaultfd.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
mm/userfaultfd.c +12 -0
index bf42fabb5afd..c0076c8b4e9d 100644
@@ -3983,6 +3983,18 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
 		if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
 			goto out_unlock;
 
+		/*
+		 * A range ending inside a registered hugetlb vma must end
+		 * huge page aligned, as in userfaultfd_register().
+		 */
+		if (cur->vm_userfaultfd_ctx.ctx && vma_is_hugetlb(cur) &&
+		    end <= cur->vm_end && end > cur->vm_start) {
+			unsigned long vma_hpagesize = vma_kernel_pagesize(cur);
+
+			if (end & (vma_hpagesize - 1))
+				goto out_unlock;
+		}
+
 		found = true;
 	} for_each_vma_range(vmi, cur, end);
 	VM_WARN_ON_ONCE(!found);
Signature · 1 line
2.39.5
#55 lore raw

Re: [PATCH 12/12] mm/userfaultfd: check hugetlb end alignment in UFFDIO_UNREGISTER pre-scan

hugetlb vs userfaultfd is a bit like the alien vs. predator of worst mm features of all time (TM)

Quoted 27 lines · Tal Zussman

UFFDIO_REGISTER and UFFDIO_UNREGISTER both scan the VMAs of the range before changing anything. For a hugetlb VMA, userfaultfd_register() rejects a range whose start or end is not huge page aligned there. However, userfaultfd_unregister() only checks the start.

An unaligned end only fails later, when vma_modify_flags_uffd() cannot split the VMA there. At that point, userfaultfd_clear_vma() already cleared the WP or RWP protection. The ioctl returns -EINVAL and the VMA stays registered, but accesses to its pages are no longer reported. This requires a caller passing a misaligned range.

To fix it, check the end of a registered VMA in the pre-scan as well. The ioctl still returns -EINVAL, but no longer unregisters the VMAs before the misaligned one first.

Fixes cab350afcbc9 ("userfaultfd: hugetlbfs: allow registration of ranges containing huge pages")
Fixes f369b07c8614 ("mm/uffd: reset write protection when unregister with wp-mode")
---
 mm/userfaultfd.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
mm/userfaultfd.c +0 -0
index bf42fabb5afd..c0076c8b4e9d 100644

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3989

Quoted 7 lines
mm/userfaultfd.c +4 -0
@@ -3983,6 +3983,18 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
 		if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
 			goto out_unlock;
 
+		/*
+		 * A range ending inside a registered hugetlb vma must end
+		 * huge page aligned, as in userfaultfd_register().
+		 */

Hmm I think this is more or less saying what the code is doing :) so I think unnecessary. Maybe worth using the words 'cursed' and 'eldrich' here to describe what's going on? Or possibly just e.g.:

	/* See check in userfaultfd_register(). */

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3990

Quoted 1 line
mm/userfaultfd.c +1 -0
+		if (cur->vm_userfaultfd_ctx.ctx && vma_is_hugetlb(cur) &&

This first check is pretty horrid to spell out loud, maybe do something like:

	do {
		const struct userfaultfd_ctx *vma_ctx = cur->vm_userfaultfd_ctx.ctx;
		...
		if (vma_ctx && vma_is_hugetlb(cur) && end <= cur->vm_end &&
		    end > cur->vm_start) {
		    ...
		}

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3992

Quoted 2 lines
mm/userfaultfd.c +2 -0
+		    end <= cur->vm_end && end > cur->vm_start) {
+			unsigned long vma_hpagesize = vma_kernel_pagesize(cur);

Could be const?

Also is vma_kernel_pagesize() right? If the VMA is hugetlb, then that'll end up as hugetlb_vm_op_pagesize() which is huge_page_size(hstate_vma()), but it seems userfaultfd is full of this crap and yeah ok. Fine :)

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3994

Quoted 2 lines
mm/userfaultfd.c +2 -0
+
+			if (end & (vma_hpagesize - 1))

Could do:
if (!IS_ALIGNED(end, vma_hpagesize))

Instead?

Quoted 10 lines
mm/userfaultfd.c +3 -1
+				goto out_unlock;
+		}
+
 		found = true;
 	} for_each_vma_range(vmi, cur, end);
 	VM_WARN_ON_ONCE(!found);
 
--

2.39.5

--
Cheers, Lorenzo

#56 lore raw

Re: [PATCH 12/12] mm/userfaultfd: check hugetlb end alignment in UFFDIO_UNREGISTER pre-scan

Quoted 4 lines · Lorenzo Stoakes (ARM)

hugetlb vs userfaultfd is a bit like the alien vs. predator of worst mm features of all time (TM)

Alien vs. Predator had a sequel... let's see what the mm version of that will be :)

Quoted 40 lines · Tal Zussman
Quoted 27 lines

UFFDIO_REGISTER and UFFDIO_UNREGISTER both scan the VMAs of the range before changing anything. For a hugetlb VMA, userfaultfd_register() rejects a range whose start or end is not huge page aligned there. However, userfaultfd_unregister() only checks the start.

An unaligned end only fails later, when vma_modify_flags_uffd() cannot split the VMA there. At that point, userfaultfd_clear_vma() already cleared the WP or RWP protection. The ioctl returns -EINVAL and the VMA stays registered, but accesses to its pages are no longer reported. This requires a caller passing a misaligned range.

To fix it, check the end of a registered VMA in the pre-scan as well. The ioctl still returns -EINVAL, but no longer unregisters the VMAs before the misaligned one first.

Fixes cab350afcbc9 ("userfaultfd: hugetlbfs: allow registration of ranges containing huge pages")
Fixes f369b07c8614 ("mm/uffd: reset write protection when unregister with wp-mode")
---
 mm/userfaultfd.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
mm/userfaultfd.c +0 -0
index bf42fabb5afd..c0076c8b4e9d 100644

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3989

Quoted 7 lines
mm/userfaultfd.c +4 -0
@@ -3983,6 +3983,18 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
 		if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
 			goto out_unlock;
 
+		/*
+		 * A range ending inside a registered hugetlb vma must end
+		 * huge page aligned, as in userfaultfd_register().
+		 */

Hmm I think this is more or less saying what the code is doing :) so I think unnecessary. Maybe worth using the words 'cursed' and 'eldrich' here to describe what's going on? Or possibly just e.g.:

Quoted 2 lines
	/* See check in userfaultfd_register(). */

Will cut it down.

Quoted 21 lines

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3990

Quoted 1 line
mm/userfaultfd.c +1 -0
+		if (cur->vm_userfaultfd_ctx.ctx && vma_is_hugetlb(cur) &&

This first check is pretty horrid to spell out loud, maybe do something like:

	do {
		const struct userfaultfd_ctx *vma_ctx = cur->vm_userfaultfd_ctx.ctx;
		...
		if (vma_ctx && vma_is_hugetlb(cur) && end <= cur->vm_end &&
		    end > cur->vm_start) {
		    ...
		}

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3992

Quoted 2 lines
mm/userfaultfd.c +2 -0
+		    end <= cur->vm_end && end > cur->vm_start) {
+			unsigned long vma_hpagesize = vma_kernel_pagesize(cur);

Could be const?

Also is vma_kernel_pagesize() right? If the VMA is hugetlb, then that'll end up as hugetlb_vm_op_pagesize() which is huge_page_size(hstate_vma()), but it seems userfaultfd is full of this crap and yeah ok. Fine :)

Quoted 9 lines

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3994

Quoted 2 lines
mm/userfaultfd.c +2 -0
+
+			if (end & (vma_hpagesize - 1))

Could do:
if (!IS_ALIGNED(end, vma_hpagesize))

Instead?

I just took this chunk straight from userfaultfd_register() and added the ctx check, but I can clean this up while here. userfaultfd_register() probably deserves a follow-up too.

Quoted 14 lines
Quoted 10 lines
mm/userfaultfd.c +3 -1
+				goto out_unlock;
+		}
+
 		found = true;
 	} for_each_vma_range(vmi, cur, end);
 	VM_WARN_ON_ONCE(!found);
 
--

2.39.5

--
Cheers, Lorenzo

#57 lore raw

Re: [PATCH 12/12] mm/userfaultfd: check hugetlb end alignment in UFFDIO_UNREGISTER pre-scan

Quoted 8 lines · Tal Zussman
Quoted 4 lines · Lorenzo Stoakes (ARM)

hugetlb vs userfaultfd is a bit like the alien vs. predator of worst mm features of all time (TM)

Alien vs. Predator had a sequel... let's see what the mm version of that will be :)

AI vs. Maintainer? :P

Quoted 41 lines
Quoted 41 lines · Tal Zussman
Quoted 27 lines

UFFDIO_REGISTER and UFFDIO_UNREGISTER both scan the VMAs of the range before changing anything. For a hugetlb VMA, userfaultfd_register() rejects a range whose start or end is not huge page aligned there. However, userfaultfd_unregister() only checks the start.

An unaligned end only fails later, when vma_modify_flags_uffd() cannot split the VMA there. At that point, userfaultfd_clear_vma() already cleared the WP or RWP protection. The ioctl returns -EINVAL and the VMA stays registered, but accesses to its pages are no longer reported. This requires a caller passing a misaligned range.

To fix it, check the end of a registered VMA in the pre-scan as well. The ioctl still returns -EINVAL, but no longer unregisters the VMAs before the misaligned one first.

Fixes cab350afcbc9 ("userfaultfd: hugetlbfs: allow registration of ranges containing huge pages")
Fixes f369b07c8614 ("mm/uffd: reset write protection when unregister with wp-mode")
---
 mm/userfaultfd.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
mm/userfaultfd.c +0 -0
index bf42fabb5afd..c0076c8b4e9d 100644

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3989

Quoted 7 lines
mm/userfaultfd.c +4 -0
@@ -3983,6 +3983,18 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
 		if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
 			goto out_unlock;
 
+		/*
+		 * A range ending inside a registered hugetlb vma must end
+		 * huge page aligned, as in userfaultfd_register().
+		 */

Hmm I think this is more or less saying what the code is doing :) so I think unnecessary. Maybe worth using the words 'cursed' and 'eldrich' here to describe what's going on? Or possibly just e.g.:

Quoted 4 lines
Quoted 2 lines
	/* See check in userfaultfd_register(). */

Will cut it down.

Thanks!

Quoted 28 lines
Quoted 28 lines

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3990

Quoted 2 lines
mm/userfaultfd.c +1 -0
+		if (cur->vm_userfaultfd_ctx.ctx && vma_is_hugetlb(cur) &&

This first check is pretty horrid to spell out loud, maybe do something like:

	do {
		const struct userfaultfd_ctx *vma_ctx = cur->vm_userfaultfd_ctx.ctx;
		...
		if (vma_ctx && vma_is_hugetlb(cur) && end <= cur->vm_end &&
		    end > cur->vm_start) {
		    ...
		}

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3992

Quoted 2 lines
mm/userfaultfd.c +2 -0
+		    end <= cur->vm_end && end > cur->vm_start) {
+			unsigned long vma_hpagesize = vma_kernel_pagesize(cur);

Could be const?

Also is vma_kernel_pagesize() right? If the VMA is hugetlb, then that'll end up as hugetlb_vm_op_pagesize() which is huge_page_size(hstate_vma()), but it seems userfaultfd is full of this crap and yeah ok. Fine :)

mm/userfaultfd.c static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, · line 3994

Quoted 2 lines
mm/userfaultfd.c +2 -0
+
+			if (end & (vma_hpagesize - 1))

Could do:
if (!IS_ALIGNED(end, vma_hpagesize))

Quoted 7 lines
Quoted 3 lines

Instead?

I just took this chunk straight from userfaultfd_register() and added the ctx check, but I can clean this up while here. userfaultfd_register() probably deserves a follow-up too.

Thanks!

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
mm/userfaultfd.c +3 -0
+				goto out_unlock;
+		}
+
 		found = true;
 	} for_each_vma_range(vmi, cur, end);
 	VM_WARN_ON_ONCE(!found);
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

--
2.39.5

--
Cheers, Lorenzo

--
Cheers, Lorenzo