Skip to contentSkip to conversation

mm/uffd: Add feature bit UFFD_FEATURE_WP_UNPOPULATED

PATCH Applied Peter Xu 2 participants 9 messages Mar 9, 2023 – Mar 24, 2023

Subsystems (10)
lore
Share & tools
lore

Peter Xu

author reviewer
#0 lore raw

v4 cover mm/uffd: Add feature bit UFFD_FEATURE_WP_UNPOPULATED

v1: https://lore.kernel.org/r/[email protected]
v2: https://lore.kernel.org/r/[email protected]
v3: https://lore.kernel.org/r/[email protected]

v4:
- s/handle_pte_missing/do_pte_missing/, fix spellings, etc. [David]
- Add a helper userfaultfd_wp_use_markers() [David]
- Update userfaultfd.rst describing the new feature bit

The new feature bit will make anonymous memory acts the same like file memory on userfaultfd-wp in that it'll also wr-protect none ptes.

It can be useful in two cases:

(1) Uffd-wp app that needs to wr-protect none ptes like QEMU snapshot, so pre-fault can be replaced by enabling this flag and speed up protections

(2) It helps to implement async uffd-wp mode that Muhammad is working on [1]

It's debateable whether this is the most ideal solution because with the new feature bit set, wr-protect none pte needs to pre-populate the pgtables to the last level (PAGE_SIZE). But it seems fine so far to service either purpose above, so we can leave optimizations for later.

The series brings pte markers to anonymous memory too. There's some change in the common mm code path in the 1st patch, great to have some eye looking at it, but hopefully they're still relatively straightforward.

Thanks,

[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

Peter Xu (2):
mm/uffd: UFFD_FEATURE_WP_UNPOPULATED selftests/mm: Smoke test UFFD_FEATURE_WP_UNPOPULATED

 Documentation/admin-guide/mm/userfaultfd.rst | 17 ++++++
 fs/userfaultfd.c                             | 16 ++++++
 include/linux/mm_inline.h                    |  6 +++
 include/linux/userfaultfd_k.h                | 23 ++++++++
 include/uapi/linux/userfaultfd.h             | 10 +++-
 mm/memory.c                                  | 56 +++++++++++++++-----
 mm/mprotect.c                                | 51 ++++++++++++++----
 tools/testing/selftests/mm/userfaultfd.c     | 45 +++++++++++++++-
 8 files changed, 197 insertions(+), 27 deletions(-)
Signature · 1 line
2.39.1
author reviewer
#1 lore raw

PATCH v4 1/2 mm/uffd: UFFD_FEATURE_WP_UNPOPULATED

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

With WP_UNPOPUATED, application like QEMU can avoid pre-read faults all the memory before wr-protect during taking a live snapshot. Quotting from Muhammad's test result here [3] based on a simple program [4]:

(1) With huge page disabled
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 1111453 (pre-fault 1101011)
Test MADVISE: 278276 (pre-fault 266378)
Test WP-UNPOPULATE: 11712

(2) With Huge page enabled
echo always > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 22521 (pre-fault 22348)
Test MADVISE: 4909 (pre-fault 4743)
Test WP-UNPOPULATE: 14448

There'll be a great perf boost for no-thp case, while for thp enabled with extreme case of all-thp-zero WP_UNPOPULATED can be slower than MADVISE, but that's low possibility in reality, also the overhead was not reduced but postponed until a follow up write on any huge zero thp, so potentially it is faster by making the follow up writes slower.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://github.com/xzpeter/clibs/blob/master/uffd-test/uffd-wp-perf.c

---
 Documentation/admin-guide/mm/userfaultfd.rst | 17 ++++++
 fs/userfaultfd.c                             | 16 ++++++
 include/linux/mm_inline.h                    |  6 +++
 include/linux/userfaultfd_k.h                | 23 ++++++++
 include/uapi/linux/userfaultfd.h             | 10 +++-
 mm/memory.c                                  | 56 +++++++++++++++-----
 mm/mprotect.c                                | 51 ++++++++++++++----
 7 files changed, 154 insertions(+), 25 deletions(-)
Documentation/admin-guide/mm/userfaultfd.rst +17 -0
index 7dc823b56ca4..c86b56c95ea6 100644
@@ -219,6 +219,23 @@ former will have ``UFFD_PAGEFAULT_FLAG_WP`` set, the latter
 you still need to supply a page when ``UFFDIO_REGISTER_MODE_MISSING`` was
 used.
 
+Userfaultfd write-protect mode currently behave differently on none ptes
+(when e.g. page is missing) over different types of memories.
+
+For anonymous memory, ``ioctl(UFFDIO_WRITEPROTECT)`` will ignore none ptes
+(e.g. when pages are missing and not populated).  For file-backed memories
+like shmem and hugetlbfs, none ptes will be write protected just like a
+present pte.  In other words, there will be a userfaultfd write fault
+message generated when writting to a missing page on file typed memories,
+as long as the page range was write-protected before.  Such a message will
+not be generated on anonymous memories by default.
+
+If the application wants to be able to write protect none ptes on anonymous
+memory, one can pre-populate the memory with e.g. MADV_POPULATE_READ.  On
+newer kernels, one can also detect the feature UFFD_FEATURE_WP_UNPOPULATED
+and set the feature bit in advance to make sure none ptes will also be
+write protected even upon anonymous memory.
+
 QEMU/KVM
 ========
 
fs/userfaultfd.c +16 -0
index 44d1ee429eb0..881e9c82b9d1 100644
@@ -108,6 +108,21 @@ static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
 	return ctx->features & UFFD_FEATURE_INITIALIZED;
 }
 
+/*
+ * Whether WP_UNPOPULATED is enabled on the uffd context.  It is only
+ * meaningful when userfaultfd_wp()==true on the vma and when it's
+ * anonymous.
+ */
+bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
+{
+	struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
+
+	if (!ctx)
+		return false;
+
+	return ctx->features & UFFD_FEATURE_WP_UNPOPULATED;
+}
+
 static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,
 				     vm_flags_t flags)
 {
@@ -1971,6 +1986,7 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx,
 #endif
 #ifndef CONFIG_PTE_MARKER_UFFD_WP
 	uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
+	uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
 #endif
 	uffdio_api.ioctls = UFFD_API_IOCTLS;
 	ret = -EFAULT;
include/linux/mm_inline.h +6 -0
index de1e622dd366..0e1d239a882c 100644
@@ -557,6 +557,12 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
 	/* The current status of the pte should be "cleared" before calling */
 	WARN_ON_ONCE(!pte_none(*pte));
 
+	/*
+	 * NOTE: userfaultfd_wp_unpopulated() doesn't need this whole
+	 * thing, because when zapping either it means it's dropping the
+	 * page, or in TTU where the present pte will be quickly replaced
+	 * with a swap pte.  There's no way of leaking the bit.
+	 */
 	if (vma_is_anonymous(vma) || !userfaultfd_wp(vma))
 		return;
 
include/linux/userfaultfd_k.h +23 -0
index 3767f18114ef..0cf8880219da 100644
@@ -179,6 +179,7 @@ extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start,
 				  unsigned long end, struct list_head *uf);
 extern void userfaultfd_unmap_complete(struct mm_struct *mm,
 				       struct list_head *uf);
+extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
 
 #else /* CONFIG_USERFAULTFD */
 
@@ -274,8 +275,30 @@ static inline bool uffd_disable_fault_around(struct vm_area_struct *vma)
 	return false;
 }
 
+static inline bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
+{
+	return false;
+}
+
 #endif /* CONFIG_USERFAULTFD */
 
+static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
+{
+	/* Only wr-protect mode uses pte markers */
+	if (!userfaultfd_wp(vma))
+		return false;
+
+	/* File-based uffd-wp always need markers */
+	if (!vma_is_anonymous(vma))
+		return true;
+
+	/*
+	 * Anonymous uffd-wp only needs the markers if WP_UNPOPULATED
+	 * enabled (to apply markers on zero pages).
+	 */
+	return userfaultfd_wp_unpopulated(vma);
+}
+
 static inline bool pte_marker_entry_uffd_wp(swp_entry_t entry)
 {
 #ifdef CONFIG_PTE_MARKER_UFFD_WP
include/uapi/linux/userfaultfd.h +9 -1
index 005e5e306266..90c958952bfc 100644
@@ -38,7 +38,8 @@
 			   UFFD_FEATURE_MINOR_HUGETLBFS |	\
 			   UFFD_FEATURE_MINOR_SHMEM |		\
 			   UFFD_FEATURE_EXACT_ADDRESS |		\
-			   UFFD_FEATURE_WP_HUGETLBFS_SHMEM)
+			   UFFD_FEATURE_WP_HUGETLBFS_SHMEM |	\
+			   UFFD_FEATURE_WP_UNPOPULATED)
 #define UFFD_API_IOCTLS				\
 	((__u64)1 << _UFFDIO_REGISTER |		\
 	 (__u64)1 << _UFFDIO_UNREGISTER |	\
@@ -203,6 +204,12 @@ struct uffdio_api {
 	 *
 	 * UFFD_FEATURE_WP_HUGETLBFS_SHMEM indicates that userfaultfd
 	 * write-protection mode is supported on both shmem and hugetlbfs.
+	 *
+	 * UFFD_FEATURE_WP_UNPOPULATED indicates that userfaultfd
+	 * write-protection mode will always apply to unpopulated pages
+	 * (i.e. empty ptes).  This will be the default behavior for shmem
+	 * & hugetlbfs, so this flag only affects anonymous memory behavior
+	 * when userfault write-protection mode is registered.
 	 */
 #define UFFD_FEATURE_PAGEFAULT_FLAG_WP		(1<<0)
 #define UFFD_FEATURE_EVENT_FORK			(1<<1)
@@ -217,6 +224,7 @@ struct uffdio_api {
 #define UFFD_FEATURE_MINOR_SHMEM		(1<<10)
 #define UFFD_FEATURE_EXACT_ADDRESS		(1<<11)
 #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM		(1<<12)
+#define UFFD_FEATURE_WP_UNPOPULATED		(1<<13)
 	__u64 features;
 
 	__u64 ioctls;
mm/memory.c +42 -14

124 lines not on this page · raw message

mm/mprotect.c +41 -10

85 lines not on this page · raw message

Signature · 1 line
2.39.1
Acked-by on v4 1/2
#2 lore raw

Re: [PATCH v4 1/2] mm/uffd: UFFD_FEATURE_WP_UNPOPULATED

Quoted 7 lines

(1) With huge page disabled
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 1111453 (pre-fault 1101011)
Test MADVISE: 278276 (pre-fault 266378)

Thinking about it, I guess the biggest slowdown here is the "one fake pagefault at a time" handling.

Quoted 38 lines
   Test WP-UNPOPULATE: 11712

   (2) With Huge page enabled
   echo always > /sys/kernel/mm/transparent_hugepage/enabled
   ./uffd_wp_perf
   Test DEFAULT: 4
   Test PRE-READ: 22521 (pre-fault 22348)
   Test MADVISE: 4909 (pre-fault 4743)
   Test WP-UNPOPULATE: 14448

There'll be a great perf boost for no-thp case, while for thp enabled with extreme case of all-thp-zero WP_UNPOPULATED can be slower than MADVISE, but that's low possibility in reality, also the overhead was not reduced but postponed until a follow up write on any huge zero thp, so potentially it is faster by making the follow up writes slower.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://github.com/xzpeter/clibs/blob/master/uffd-test/uffd-wp-perf.c

---
  Documentation/admin-guide/mm/userfaultfd.rst | 17 ++++++
  fs/userfaultfd.c                             | 16 ++++++
  include/linux/mm_inline.h                    |  6 +++
  include/linux/userfaultfd_k.h                | 23 ++++++++
  include/uapi/linux/userfaultfd.h             | 10 +++-
  mm/memory.c                                  | 56 +++++++++++++++-----
  mm/mprotect.c                                | 51 ++++++++++++++----
  7 files changed, 154 insertions(+), 25 deletions(-)
Documentation/admin-guide/mm/userfaultfd.rst +0 -0
index 7dc823b56ca4..c86b56c95ea6 100644
@@ -219,6 +219,23 @@ former will have ``UFFD_PAGEFAULT_FLAG_WP`` set, the latter
  you still need to supply a page when ``UFFDIO_REGISTER_MODE_MISSING`` was

Documentation/admin-guide/mm/userfaultfd.rst former will have ``UFFD_PAGEFAULT_FLAG_WP`` set, the latter · line 229

Quoted 10 lines
Documentation/admin-guide/mm/userfaultfd.rst +8 -0
  used.
 
+Userfaultfd write-protect mode currently behave differently on none ptes
+(when e.g. page is missing) over different types of memories.
+
+For anonymous memory, ``ioctl(UFFDIO_WRITEPROTECT)`` will ignore none ptes
+(e.g. when pages are missing and not populated).  For file-backed memories
+like shmem and hugetlbfs, none ptes will be write protected just like a
+present pte.  In other words, there will be a userfaultfd write fault
+message generated when writting to a missing page on file typed memories,

s/writting/writing/

Quoted 9 lines
Documentation/admin-guide/mm/userfaultfd.rst +9 -0
+as long as the page range was write-protected before.  Such a message will
+not be generated on anonymous memories by default.
+
+If the application wants to be able to write protect none ptes on anonymous
+memory, one can pre-populate the memory with e.g. MADV_POPULATE_READ.  On
+newer kernels, one can also detect the feature UFFD_FEATURE_WP_UNPOPULATED
+and set the feature bit in advance to make sure none ptes will also be
+write protected even upon anonymous memory.
+

[...]

Quoted 14 lines

/*
* A number of key systems in x86 including ioremap() rely on the assumption

Documentation/admin-guide/mm/userfaultfd.rst +4 -0
@@ -1350,6 +1364,10 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma,
  			      unsigned long addr, pte_t *pte,
  			      struct zap_details *details, pte_t pteval)
  {
+	/* Zap on anonymous always means dropping everything */
+	if (vma_is_anonymous(vma))
+		return;
+
  	if (zap_drop_file_uffd_wp(details))
  		return;
 

Documentation/admin-guide/mm/userfaultfd.rst static unsigned long zap_pte_range(struct mmu_gather *tlb, · line 1480

Quoted 9 lines
Documentation/admin-guide/mm/userfaultfd.rst +4 -2
@@ -1456,8 +1474,12 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
  				continue;
  			rss[mm_counter(page)]--;
  		} else if (pte_marker_entry_uffd_wp(entry)) {
-			/* Only drop the uffd-wp marker if explicitly requested */
-			if (!zap_drop_file_uffd_wp(details))
+			/*
+			 * For anon: always drop the marker; for file: only
+			 * drop the marker if explicitly requested.
+			 */

So MADV_DONTNEED a pte marker in an anonymous VMA will always remove that marker. Is that the same handling as for MADV_DONTNEED on shmem or on fallocate(PUNCHHOLE) on shmem?

Documentation/admin-guide/mm/userfaultfd.rst static unsigned long zap_pte_range(struct mmu_gather *tlb, · line 1483

Quoted 3 lines
Documentation/admin-guide/mm/userfaultfd.rst +2 -0
+			if (!vma_is_anonymous(vma) &&
+			    !zap_drop_file_uffd_wp(details))
  				continue;

Maybe it would be nicer to have a zap_drop_uffd_wp_marker(vma, details) and have the comment in there. Especially because of the other hunk above.

So zap_drop_file_uffd_wp(details) -> zap_drop_uffd_wp_marker(vma, details) and move the anon handling + comment in there.

Documentation/admin-guide/mm/userfaultfd.rst static vm_fault_t pte_marker_clear(struct vm_fault *vmf) · line 3654

Quoted 12 lines
Documentation/admin-guide/mm/userfaultfd.rst +6 -0
  		} else if (is_hwpoison_entry(entry) ||
  			   is_swapin_error_entry(entry)) {
@@ -3624,6 +3646,14 @@ static vm_fault_t pte_marker_clear(struct vm_fault *vmf)
  	return 0;
  }
 
+static vm_fault_t do_pte_missing(struct vm_fault *vmf)
+{
+	if (vma_is_anonymous(vmf->vma))
+		return do_anonymous_page(vmf);
+	else
+		return do_fault(vmf);

No need for the "else" statement.

Quoted 18 lines
Documentation/admin-guide/mm/userfaultfd.rst +4 -3
+}
+
  /*
   * This is actually a page-missing access, but with uffd-wp special pte
   * installed.  It means this pte was wr-protected before being unmapped.
@@ -3634,11 +3664,10 @@ static vm_fault_t pte_marker_handle_uffd_wp(struct vm_fault *vmf)
  	 * Just in case there're leftover special ptes even after the region
  	 * got unregistered - we can simply clear them.
  	 */
-	if (unlikely(!userfaultfd_wp(vmf->vma) || vma_is_anonymous(vmf->vma)))
+	if (unlikely(!userfaultfd_wp(vmf->vma)))
  		return pte_marker_clear(vmf);
 
-	/* do_fault() can handle pte markers too like none pte */
-	return do_fault(vmf);
+	return do_pte_missing(vmf);
  }
 

[...]

Quoted 22 lines
mm/mprotect.c +9 -1
index 231929f119d9..455f7051098f 100644
@@ -276,7 +276,15 @@ static long change_pte_range(struct mmu_gather *tlb,
  		} else {
  			/* It must be an none page, or what else?.. */
  			WARN_ON_ONCE(!pte_none(oldpte));
-			if (unlikely(uffd_wp && !vma_is_anonymous(vma))) {
+
+			/*
+			 * Nobody plays with any none ptes besides
+			 * userfaultfd when applying the protections.
+			 */
+			if (likely(!uffd_wp))
+				continue;
+
+			if (userfaultfd_wp_use_markers(vma)) {
  				/*
  				 * For file-backed mem, we need to be able to
  				 * wr-protect a none pte, because even if the

mm/mprotect.c static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) · line 332

Quoted 6 lines
mm/mprotect.c +2 -1
@@ -320,23 +328,46 @@ static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd)
  	return 0;
  }
 
-/* Return true if we're uffd wr-protecting file-backed memory, or false */
+/*
+ * Return true if we want to split huge thps in change protection

"huge thps" sounds redundant. "if we want to PTE-map a huge PMD" ?

mm/mprotect.c static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) · line 333

Quoted 1 line
mm/mprotect.c +1 -0
+ * procedure, false otherwise.

In general,

Signature · 3 lines
Thanks,

David / dhildenb
author reviewer
#3 lore raw

Re: [PATCH v4 1/2] mm/uffd: UFFD_FEATURE_WP_UNPOPULATED

Quoted 10 lines · David Hildenbrand
Quoted 7 lines

(1) With huge page disabled
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 1111453 (pre-fault 1101011)
Test MADVISE: 278276 (pre-fault 266378)

Thinking about it, I guess the biggest slowdown here is the "one fake pagefault at a time" handling.

I think so, though I assume the idea here is to avoid any faulting.

Quoted 80 lines
Quoted 38 lines
   Test WP-UNPOPULATE: 11712

   (2) With Huge page enabled
   echo always > /sys/kernel/mm/transparent_hugepage/enabled
   ./uffd_wp_perf
   Test DEFAULT: 4
   Test PRE-READ: 22521 (pre-fault 22348)
   Test MADVISE: 4909 (pre-fault 4743)
   Test WP-UNPOPULATE: 14448

There'll be a great perf boost for no-thp case, while for thp enabled with extreme case of all-thp-zero WP_UNPOPULATED can be slower than MADVISE, but that's low possibility in reality, also the overhead was not reduced but postponed until a follow up write on any huge zero thp, so potentially it is faster by making the follow up writes slower.

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/
[3] https://lore.kernel.org/all/[email protected]/
[4] https://github.com/xzpeter/clibs/blob/master/uffd-test/uffd-wp-perf.c

---
  Documentation/admin-guide/mm/userfaultfd.rst | 17 ++++++
  fs/userfaultfd.c                             | 16 ++++++
  include/linux/mm_inline.h                    |  6 +++
  include/linux/userfaultfd_k.h                | 23 ++++++++
  include/uapi/linux/userfaultfd.h             | 10 +++-
  mm/memory.c                                  | 56 +++++++++++++++-----
  mm/mprotect.c                                | 51 ++++++++++++++----
  7 files changed, 154 insertions(+), 25 deletions(-)
Documentation/admin-guide/mm/userfaultfd.rst +0 -0
index 7dc823b56ca4..c86b56c95ea6 100644

Documentation/admin-guide/mm/userfaultfd.rst former will have ``UFFD_PAGEFAULT_FLAG_WP`` set, the latter · line 228

Quoted 10 lines
Documentation/admin-guide/mm/userfaultfd.rst +8 -0
@@ -219,6 +219,23 @@ former will have ``UFFD_PAGEFAULT_FLAG_WP`` set, the latter
  you still need to supply a page when ``UFFDIO_REGISTER_MODE_MISSING`` was
  used.
+Userfaultfd write-protect mode currently behave differently on none ptes
+(when e.g. page is missing) over different types of memories.
+
+For anonymous memory, ``ioctl(UFFDIO_WRITEPROTECT)`` will ignore none ptes
+(e.g. when pages are missing and not populated).  For file-backed memories
+like shmem and hugetlbfs, none ptes will be write protected just like a
+present pte.  In other words, there will be a userfaultfd write fault
+message generated when writting to a missing page on file typed memories,

s/writting/writing/

Quoted 9 lines
Documentation/admin-guide/mm/userfaultfd.rst +9 -0
+as long as the page range was write-protected before.  Such a message will
+not be generated on anonymous memories by default.
+
+If the application wants to be able to write protect none ptes on anonymous
+memory, one can pre-populate the memory with e.g. MADV_POPULATE_READ.  On
+newer kernels, one can also detect the feature UFFD_FEATURE_WP_UNPOPULATED
+and set the feature bit in advance to make sure none ptes will also be
+write protected even upon anonymous memory.
+

[...]

Quoted 17 lines

/*
* A number of key systems in x86 including ioremap() rely on the assumption

Documentation/admin-guide/mm/userfaultfd.rst +4 -1
@@ -1350,6 +1364,10 @@ zap_install_uffd_wp_if_needed(struct vm_area_struct *vma,
  			      unsigned long addr, pte_t *pte,
  			      struct zap_details *details, pte_t pteval)
  {
+	/* Zap on anonymous always means dropping everything */
+	if (vma_is_anonymous(vma))
+		return;
+
  	if (zap_drop_file_uffd_wp(details))
  		return;
@@ -1456,8 +1474,12 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
  				continue;
  			rss[mm_counter(page)]--;
  		} else if (pte_marker_entry_uffd_wp(entry)) {
-			/* Only drop the uffd-wp marker if explicitly requested */
Quoted 8 lines

Documentation/admin-guide/mm/userfaultfd.rst static unsigned long zap_pte_range(struct mmu_gather *tlb, · line 1480

Quoted 5 lines
Documentation/admin-guide/mm/userfaultfd.rst +4 -1
-			if (!zap_drop_file_uffd_wp(details))
+			/*
+			 * For anon: always drop the marker; for file: only
+			 * drop the marker if explicitly requested.
+			 */

So MADV_DONTNEED a pte marker in an anonymous VMA will always remove that marker.

Yes.

Quoted 2 lines

Is that the same handling as for MADV_DONTNEED on shmem or on fallocate(PUNCHHOLE) on shmem?

Same as PUNCHHOLE for shmem, while DONTNEED for shmem will retain the marker. Here the idea is we drop the marker if the user wants to drop the page, no matter what type of memory is underneath.

Quoted 10 lines

Documentation/admin-guide/mm/userfaultfd.rst static unsigned long zap_pte_range(struct mmu_gather *tlb, · line 1483

Quoted 4 lines
Documentation/admin-guide/mm/userfaultfd.rst +2 -0
+			if (!vma_is_anonymous(vma) &&
+			    !zap_drop_file_uffd_wp(details))
  				continue;

Maybe it would be nicer to have a zap_drop_uffd_wp_marker(vma, details) and have the comment in there. Especially because of the other hunk above.

So zap_drop_file_uffd_wp(details) -> zap_drop_uffd_wp_marker(vma, details) and move the anon handling + comment in there.

Yes we can.

Actually here I always thought DROP_MARKER is too specific and the caller will be confused on when to pass it in.

After introduction of ZAP_FLAG_UNMAP for hugetlb, I think we can also have another more generic flag ZAP_FLAG_TRUNCATE only set during truncations, then here the old DROP_MARKER can be replaced by "TRUNCATE | UNMAP".

Quoted 15 lines

Documentation/admin-guide/mm/userfaultfd.rst static vm_fault_t pte_marker_clear(struct vm_fault *vmf) · line 3653

Quoted 13 lines
Documentation/admin-guide/mm/userfaultfd.rst +6 -0
  		} else if (is_hwpoison_entry(entry) ||
  			   is_swapin_error_entry(entry)) {
@@ -3624,6 +3646,14 @@ static vm_fault_t pte_marker_clear(struct vm_fault *vmf)
  	return 0;
  }
+static vm_fault_t do_pte_missing(struct vm_fault *vmf)
+{
+	if (vma_is_anonymous(vmf->vma))
+		return do_anonymous_page(vmf);
+	else
+		return do_fault(vmf);

No need for the "else" statement.

I don't see much difference in this specific context, but I'm fine to drop it too.

Quoted 41 lines
Quoted 17 lines
Documentation/admin-guide/mm/userfaultfd.rst +4 -3
+}
+
  /*
   * This is actually a page-missing access, but with uffd-wp special pte
   * installed.  It means this pte was wr-protected before being unmapped.
@@ -3634,11 +3664,10 @@ static vm_fault_t pte_marker_handle_uffd_wp(struct vm_fault *vmf)
  	 * Just in case there're leftover special ptes even after the region
  	 * got unregistered - we can simply clear them.
  	 */
-	if (unlikely(!userfaultfd_wp(vmf->vma) || vma_is_anonymous(vmf->vma)))
+	if (unlikely(!userfaultfd_wp(vmf->vma)))
  		return pte_marker_clear(vmf);
-	/* do_fault() can handle pte markers too like none pte */
-	return do_fault(vmf);
+	return do_pte_missing(vmf);
  }

[...]

Quoted 21 lines
mm/mprotect.c +9 -1
index 231929f119d9..455f7051098f 100644
@@ -276,7 +276,15 @@ static long change_pte_range(struct mmu_gather *tlb,
  		} else {
  			/* It must be an none page, or what else?.. */
  			WARN_ON_ONCE(!pte_none(oldpte));
-			if (unlikely(uffd_wp && !vma_is_anonymous(vma))) {
+
+			/*
+			 * Nobody plays with any none ptes besides
+			 * userfaultfd when applying the protections.
+			 */
+			if (likely(!uffd_wp))
+				continue;
+
+			if (userfaultfd_wp_use_markers(vma)) {
  				/*
  				 * For file-backed mem, we need to be able to
  				 * wr-protect a none pte, because even if the
Quoted 8 lines

mm/mprotect.c static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) · line 331

Quoted 6 lines
mm/mprotect.c +2 -1
@@ -320,23 +328,46 @@ static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd)
  	return 0;
  }
-/* Return true if we're uffd wr-protecting file-backed memory, or false */
+/*
+ * Return true if we want to split huge thps in change protection

"huge thps" sounds redundant. "if we want to PTE-map a huge PMD" ?

Sure.

Quoted 7 lines

mm/mprotect.c static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) · line 333

Quoted 2 lines
mm/mprotect.c +1 -0
+ * procedure, false otherwise.

In general,

Thanks,

Signature · 1 line
Peter Xu
author reviewer
#4 lore raw

Re: [PATCH v4 1/2] mm/uffd: UFFD_FEATURE_WP_UNPOPULATED

In reply to Peter Xu
Quoted 3 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

[...]

Hi, Andrew,

Could you add a fixup to this patch as attached? It contains two comment changes suggested by David, and also a oneliner fix to khugepaged (to bail out anon thp collapsing when seeing pte markers). The latter one was something I spot only later on.

Thanks,

Signature · 1 line
Peter Xu

From 085596bc8913349cbeb3ec3303815f71f1a34d89 Mon Sep 17 00:00:00 2001
From: Peter Xu <[email protected]>
Date: Tue, 21 Mar 2023 16:09:26 -0400
Subject: [PATCH] fixup! mm/uffd: UFFD_FEATURE_WP_UNPOPULATED

---
 Documentation/admin-guide/mm/userfaultfd.rst | 2 +-
 mm/khugepaged.c                              | 2 +-
 mm/mprotect.c                                | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)
Documentation/admin-guide/mm/userfaultfd.rst +1 -1
index 16843d5a4f65..7c304e432205 100644
@@ -226,7 +226,7 @@ For anonymous memory, ``ioctl(UFFDIO_WRITEPROTECT)`` will ignore none ptes
 (e.g. when pages are missing and not populated).  For file-backed memories
 like shmem and hugetlbfs, none ptes will be write protected just like a
 present pte.  In other words, there will be a userfaultfd write fault
-message generated when writting to a missing page on file typed memories,
+message generated when writing to a missing page on file typed memories,
 as long as the page range was write-protected before.  Such a message will
 not be generated on anonymous memories by default.
 
mm/khugepaged.c +1 -1
index bdde0a02811b..2a5372c49b82 100644
@@ -1283,7 +1283,7 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,
 				 * enabled swap entries.  Please see
 				 * comment below for pte_uffd_wp().
 				 */
-				if (pte_swp_uffd_wp(pteval)) {
+				if (pte_swp_uffd_wp_any(pteval)) {
 					result = SCAN_PTE_UFFD_WP;
 					goto out_unmap;
 				}
mm/mprotect.c +2 -2
index e27bbd0fde6f..b9da9a5f87fe 100644
@@ -329,8 +329,8 @@ static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd)
 }
 
 /*
- * Return true if we want to split huge thps in change protection
- * procedure, false otherwise.
+ * Return true if we want to split THPs into PTE mappings in change
+ * protection procedure, false otherwise.
  */
 static inline bool
 pgtable_split_needed(struct vm_area_struct *vma, unsigned long cp_flags)
Signature · 1 line
2.39.1
author reviewer
#5 lore raw

PATCH v4 2/2 selftests/mm: Smoke test UFFD_FEATURE_WP_UNPOPULATED

In reply to Peter Xu

Enable it by default on the stress test, and add some smoke tests for the pte markers on anonymous.

---
 tools/testing/selftests/mm/userfaultfd.c | 45 ++++++++++++++++++++++--
 1 file changed, 43 insertions(+), 2 deletions(-)
tools/testing/selftests/mm/userfaultfd.c +43 -2
index 7f22844ed704..e030d63c031a 100644
@@ -1444,6 +1444,43 @@ static int pagemap_test_fork(bool present)
 	return result;
 }
 
+static void userfaultfd_wp_unpopulated_test(int pagemap_fd)
+{
+	uint64_t value;
+
+	/* Test applying pte marker to anon unpopulated */
+	wp_range(uffd, (uint64_t)area_dst, page_size, true);
+	value = pagemap_read_vaddr(pagemap_fd, area_dst);
+	pagemap_check_wp(value, true);
+
+	/* Test unprotect on anon pte marker */
+	wp_range(uffd, (uint64_t)area_dst, page_size, false);
+	value = pagemap_read_vaddr(pagemap_fd, area_dst);
+	pagemap_check_wp(value, false);
+
+	/* Test zap on anon marker */
+	wp_range(uffd, (uint64_t)area_dst, page_size, true);
+	if (madvise(area_dst, page_size, MADV_DONTNEED))
+		err("madvise(MADV_DONTNEED) failed");
+	value = pagemap_read_vaddr(pagemap_fd, area_dst);
+	pagemap_check_wp(value, false);
+
+	/* Test fault in after marker removed */
+	*area_dst = 1;
+	value = pagemap_read_vaddr(pagemap_fd, area_dst);
+	pagemap_check_wp(value, false);
+	/* Drop it to make pte none again */
+	if (madvise(area_dst, page_size, MADV_DONTNEED))
+		err("madvise(MADV_DONTNEED) failed");
+
+	/* Test read-zero-page upon pte marker */
+	wp_range(uffd, (uint64_t)area_dst, page_size, true);
+	*(volatile char *)area_dst;
+	/* Drop it to make pte none again */
+	if (madvise(area_dst, page_size, MADV_DONTNEED))
+		err("madvise(MADV_DONTNEED) failed");
+}
+
 static void userfaultfd_pagemap_test(unsigned int test_pgsize)
 {
 	struct uffdio_register uffdio_register;
@@ -1462,7 +1499,7 @@ static void userfaultfd_pagemap_test(unsigned int test_pgsize)
 	/* Flush so it doesn't flush twice in parent/child later */
 	fflush(stdout);
 
-	uffd_test_ctx_init(0);
+	uffd_test_ctx_init(UFFD_FEATURE_WP_UNPOPULATED);
 
 	if (test_pgsize > page_size) {
 		/* This is a thp test */
@@ -1482,6 +1519,10 @@ static void userfaultfd_pagemap_test(unsigned int test_pgsize)
 
 	pagemap_fd = pagemap_open();
 
+	/* Smoke test WP_UNPOPULATED first when it's still empty */
+	if (test_pgsize == page_size)
+		userfaultfd_wp_unpopulated_test(pagemap_fd);
+
 	/* Touch the page */
 	*area_dst = 1;
 	wp_range(uffd, (uint64_t)area_dst, test_pgsize, true);
@@ -1526,7 +1567,7 @@ static int userfaultfd_stress(void)
 	struct uffdio_register uffdio_register;
 	struct uffd_stats uffd_stats[nr_cpus];
 
-	uffd_test_ctx_init(0);
+	uffd_test_ctx_init(UFFD_FEATURE_WP_UNPOPULATED);
 
 	if (posix_memalign(&area, page_size, page_size))
 		err("out of memory");
Signature · 1 line
2.39.1
#6 lore raw

Re: [PATCH v4 2/2] selftests/mm: Smoke test UFFD_FEATURE_WP_UNPOPULATED

Quoted 2 lines · Peter Xu

Enable it by default on the stress test, and add some smoke tests for the pte markers on anonymous.

Would it make sense to make kernel support optional and test both paths -- once with the feature enabled (if available on the kernel we're testing) and once with the feature disabled?

Signature · 3 lines
Thanks,

David / dhildenb
author reviewer
#7 lore raw

Re: [PATCH v4 2/2] selftests/mm: Smoke test UFFD_FEATURE_WP_UNPOPULATED

Quoted 7 lines · David Hildenbrand
Quoted 2 lines · Peter Xu

Enable it by default on the stress test, and add some smoke tests for the pte markers on anonymous.

Would it make sense to make kernel support optional and test both paths -- once with the feature enabled (if available on the kernel we're testing) and once with the feature disabled?

Yeah, I think the current uffd selftest is not friendly to old kernels so I made it simple - IOW the test should fail already on old kernels AFAIK. Maybe I can prepare some patches this week or next to cleanup some parts of it.

Thanks,

Signature · 1 line
Peter Xu
#8 lore raw

Re: [PATCH v4 2/2] selftests/mm: Smoke test UFFD_FEATURE_WP_UNPOPULATED

Quoted 5 lines · Peter Xu
Quoted 4 lines · David Hildenbrand
Quoted 2 lines · Peter Xu

Enable it by default on the stress test, and add some smoke tests for the pte markers on anonymous.

Quoted 8 lines
Quoted 3 lines

Would it make sense to make kernel support optional and test both paths -- once with the feature enabled (if available on the kernel we're testing) and once with the feature disabled?

Yeah, I think the current uffd selftest is not friendly to old kernels so I made it simple - IOW the test should fail already on old kernels AFAIK. Maybe I can prepare some patches this week or next to cleanup some parts of it.

Some people to tend to run upstream selftests on older kernels, but I guess this is not high priority. Can always be added on top.

Signature · 3 lines
Thanks,

David / dhildenb