Skip to contentSkip to conversation

mm/uffd: UFFD_FEATURE_WP_UNPOPULATED

PATCH Superseded Peter Xu 3 participants 20 messages Feb 27, 2023 – Mar 6, 2023

Subsystems (8)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Peter Xu

author reviewer
#0 lore raw

PATCH v2 1/1 mm/uffd: UFFD_FEATURE_WP_UNPOPULATED

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

---
 fs/userfaultfd.c                         | 14 ++++++
 include/linux/mm_inline.h                |  6 +++
 include/linux/userfaultfd_k.h            |  6 +++
 include/uapi/linux/userfaultfd.h         | 10 +++-
 mm/memory.c                              | 54 ++++++++++++++++------
 mm/mprotect.c                            | 59 ++++++++++++++++++++----
 tools/testing/selftests/mm/userfaultfd.c | 22 ++++++++-
 7 files changed, 146 insertions(+), 25 deletions(-)
fs/userfaultfd.c +14 -0
index 44d1ee429eb0..a2499908985a 100644
@@ -108,6 +108,19 @@ static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
 	return ctx->features & UFFD_FEATURE_INITIALIZED;
 }
 
+bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
+{
+	struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
+
+	if (!userfaultfd_wp(vma))
+		return false;
+
+	if (!ctx)
+		return false;
+
+	return ctx->features & UFFD_FEATURE_WP_UNPOPULATED;
+}
+
 static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,
 				     vm_flags_t flags)
 {
@@ -1971,6 +1984,7 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx,
 #endif
 #ifndef CONFIG_PTE_MARKER_UFFD_WP
 	uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM;
+	uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED;
 #endif
 	uffdio_api.ioctls = UFFD_API_IOCTLS;
 	ret = -EFAULT;
include/linux/mm_inline.h +6 -0
index de1e622dd366..0e1d239a882c 100644
@@ -557,6 +557,12 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr,
 	/* The current status of the pte should be "cleared" before calling */
 	WARN_ON_ONCE(!pte_none(*pte));
 
+	/*
+	 * NOTE: userfaultfd_wp_unpopulated() doesn't need this whole
+	 * thing, because when zapping either it means it's dropping the
+	 * page, or in TTU where the present pte will be quickly replaced
+	 * with a swap pte.  There's no way of leaking the bit.
+	 */
 	if (vma_is_anonymous(vma) || !userfaultfd_wp(vma))
 		return;
 
include/linux/userfaultfd_k.h +6 -0
index 3767f18114ef..9d43c1e88175 100644
@@ -179,6 +179,7 @@ extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start,
 				  unsigned long end, struct list_head *uf);
 extern void userfaultfd_unmap_complete(struct mm_struct *mm,
 				       struct list_head *uf);
+extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
 
 #else /* CONFIG_USERFAULTFD */
 
@@ -274,6 +275,11 @@ static inline bool uffd_disable_fault_around(struct vm_area_struct *vma)
 	return false;
 }
 
+static inline bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
+{
+	return false;
+}
+
 #endif /* CONFIG_USERFAULTFD */
 
 static inline bool pte_marker_entry_uffd_wp(swp_entry_t entry)
include/uapi/linux/userfaultfd.h +9 -1
index 005e5e306266..90c958952bfc 100644
@@ -38,7 +38,8 @@
 			   UFFD_FEATURE_MINOR_HUGETLBFS |	\
 			   UFFD_FEATURE_MINOR_SHMEM |		\
 			   UFFD_FEATURE_EXACT_ADDRESS |		\
-			   UFFD_FEATURE_WP_HUGETLBFS_SHMEM)
+			   UFFD_FEATURE_WP_HUGETLBFS_SHMEM |	\
+			   UFFD_FEATURE_WP_UNPOPULATED)
 #define UFFD_API_IOCTLS				\
 	((__u64)1 << _UFFDIO_REGISTER |		\
 	 (__u64)1 << _UFFDIO_UNREGISTER |	\
@@ -203,6 +204,12 @@ struct uffdio_api {
 	 *
 	 * UFFD_FEATURE_WP_HUGETLBFS_SHMEM indicates that userfaultfd
 	 * write-protection mode is supported on both shmem and hugetlbfs.
+	 *
+	 * UFFD_FEATURE_WP_UNPOPULATED indicates that userfaultfd
+	 * write-protection mode will always apply to unpopulated pages
+	 * (i.e. empty ptes).  This will be the default behavior for shmem
+	 * & hugetlbfs, so this flag only affects anonymous memory behavior
+	 * when userfault write-protection mode is registered.
 	 */
 #define UFFD_FEATURE_PAGEFAULT_FLAG_WP		(1<<0)
 #define UFFD_FEATURE_EVENT_FORK			(1<<1)
@@ -217,6 +224,7 @@ struct uffdio_api {
 #define UFFD_FEATURE_MINOR_SHMEM		(1<<10)
 #define UFFD_FEATURE_EXACT_ADDRESS		(1<<11)
 #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM		(1<<12)
+#define UFFD_FEATURE_WP_UNPOPULATED		(1<<13)
 	__u64 features;
 
 	__u64 ioctls;
mm/memory.c +41 -13

116 lines not on this page · raw message

mm/mprotect.c +49 -10

93 lines not on this page · raw message

tools/testing/selftests/mm/userfaultfd.c +21 -1
index 7f22844ed704..122695a63cf1 100644
@@ -1462,7 +1462,7 @@ static void userfaultfd_pagemap_test(unsigned int test_pgsize)
 	/* Flush so it doesn't flush twice in parent/child later */
 	fflush(stdout);
 
-	uffd_test_ctx_init(0);
+	uffd_test_ctx_init(UFFD_FEATURE_WP_UNPOPULATED);
 
 	if (test_pgsize > page_size) {
 		/* This is a thp test */
@@ -1482,6 +1482,26 @@ static void userfaultfd_pagemap_test(unsigned int test_pgsize)
 
 	pagemap_fd = pagemap_open();
 
+	/* Smoke test WP_UNPOPULATED first when it's still empty */
+	if (test_pgsize == page_size) {
+		/* Test applying pte marker to anon unpopulated */
+		wp_range(uffd, (uint64_t)area_dst, test_pgsize, true);
+		value = pagemap_read_vaddr(pagemap_fd, area_dst);
+		pagemap_check_wp(value, true);
+
+		/* Test unprotect with anon pte marker */
+		wp_range(uffd, (uint64_t)area_dst, test_pgsize, false);
+		value = pagemap_read_vaddr(pagemap_fd, area_dst);
+		pagemap_check_wp(value, false);
+
+		/* Re-apply, test zap on anon which should drop the marker */
+		wp_range(uffd, (uint64_t)area_dst, test_pgsize, true);
+		if (madvise(area_dst, test_pgsize, MADV_DONTNEED))
+			err("madvise(MADV_DONTNEED) failed");
+		value = pagemap_read_vaddr(pagemap_fd, area_dst);
+		pagemap_check_wp(value, false);
+	}
+
 	/* Touch the page */
 	*area_dst = 1;
 	wp_range(uffd, (uint64_t)area_dst, test_pgsize, true);
Signature · 1 line
2.39.1
author reviewer
#1 lore raw
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Results:

        Test DEFAULT: 2
        Test PRE-READ: 3277099 (pre-fault 3253826)
        Test MADVISE: 2250361 (pre-fault 2226310)
        Test WP-UNPOPULATE: 20850

I'll add these information into the commit message when there's a new version.

[1] https://github.com/xzpeter/clibs/blob/master/uffd-test/uffd-wp-perf.c

Signature · 1 line
Peter Xu
#2 lore raw

Hi Peter,

Thank you so much for sending.

Quoted 75 lines · Peter Xu
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Results:

Quoted 7 lines
        Test DEFAULT: 2
        Test PRE-READ: 3277099 (pre-fault 3253826)
        Test MADVISE: 2250361 (pre-fault 2226310)
        Test WP-UNPOPULATE: 20850

I'll add these information into the commit message when there's a new version.

I'm hitting a bug where I'm unable to write to the memory after adding this patch and wp the memory. I'm hitting this case in your test and my tests as well. Please apply the following diff to your test to reproduce on your end:

uffd_wp_perf.c 2023-02-28 12:13:11.077827160 +0500 renamed from uffd_wp_perf.c.orig 2023-02-28 12:09:38.971820791 +0500 +1 -0
@@ -114,6 +114,7 @@
         start1 = get_usec();
     }
     wp_range(uffd, buffer, SIZE, true);
+    buffer[0] = 'a';
     if (start1 == 0)
          printf("%"PRIu64"\n", get_usec() - start);
     else
Quoted 3 lines

[1] https://github.com/xzpeter/clibs/blob/master/uffd-test/uffd-wp-perf.c

Signature · 2 lines
BR,
Muhammad Usama Anjum
author reviewer
#3 lore raw
Quoted 1 line · Muhammad Usama Anjum

Hi Peter,

Hi, Muhammad,

Quoted 93 lines

Thank you so much for sending.

Quoted 75 lines · Peter Xu
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Results:

Quoted 7 lines
        Test DEFAULT: 2
        Test PRE-READ: 3277099 (pre-fault 3253826)
        Test MADVISE: 2250361 (pre-fault 2226310)
        Test WP-UNPOPULATE: 20850

I'll add these information into the commit message when there's a new version.

I'm hitting a bug where I'm unable to write to the memory after adding this patch and wp the memory. I'm hitting this case in your test and my tests as well. Please apply the following diff to your test to reproduce on your end:

uffd_wp_perf.c 2023-02-28 12:13:11.077827160 +0500 line 120

Quoted 7 lines
uffd_wp_perf.c 2023-02-28 12:13:11.077827160 +0500 renamed from uffd_wp_perf.c.orig 2023-02-28 12:09:38.971820791 +0500 +1 -0
@@ -114,6 +114,7 @@
         start1 = get_usec();
     }
     wp_range(uffd, buffer, SIZE, true);
+    buffer[0] = 'a';
     if (start1 == 0)
          printf("%"PRIu64"\n", get_usec() - start);
     else

This is expected, because the test didn't start any fault resolving thread, so the write will block until someone unprotects the page.

But it shouldn't happen to your use case if you applied both WP_UNPOPULATED & WP_ASYNC.

Could you try "cat /proc/$PID/stack" to see where does your thread stuck at?

Signature · 1 line
Peter Xu
#4 lore raw
Quoted 95 lines · Peter Xu
Quoted 1 line · Muhammad Usama Anjum

Hi Peter,

Hi, Muhammad,

Quoted 90 lines

Thank you so much for sending.

Quoted 75 lines · Peter Xu
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Results:

Quoted 7 lines
        Test DEFAULT: 2
        Test PRE-READ: 3277099 (pre-fault 3253826)
        Test MADVISE: 2250361 (pre-fault 2226310)
        Test WP-UNPOPULATE: 20850

I'll add these information into the commit message when there's a new version.

I'm hitting a bug where I'm unable to write to the memory after adding this patch and wp the memory. I'm hitting this case in your test and my tests as well. Please apply the following diff to your test to reproduce on your end:

Quoted 13 lines

uffd_wp_perf.c 2023-02-28 12:13:11.077827160 +0500 line 120

Quoted 10 lines
uffd_wp_perf.c 2023-02-28 12:13:11.077827160 +0500 renamed from uffd_wp_perf.c.orig 2023-02-28 12:09:38.971820791 +0500 +1 -0
@@ -114,6 +114,7 @@
         start1 = get_usec();
     }
     wp_range(uffd, buffer, SIZE, true);
+    buffer[0] = 'a';
     if (start1 == 0)
          printf("%"PRIu64"\n", get_usec() - start);
     else

This is expected, because the test didn't start any fault resolving thread, so the write will block until someone unprotects the page.

Ohh.. sorry. Wrong reproducer.

Quoted 3 lines

But it shouldn't happen to your use case if you applied both WP_UNPOPULATED & WP_ASYNC.

I'm using both WP_UNPOPULATED and ASYNC. The program gets stuck at right time:

1..57
ok 1 sanity_tests_sd wrong flag specified
ok 2 sanity_tests_sd wrong mask specified
ok 3 sanity_tests_sd wrong return mask specified
ok 4 sanity_tests_sd mixture of correct and wrong flag ok 5 sanity_tests_sd PM_SCAN_OP_WP cannot be used without get ok 6 sanity_tests_sd Clear area with larger vec size
^C
Program received signal SIGINT, Interrupt. 0x000000000040220c in sanity_tests_sd () at pagemap_ioctl.c:198
198 mem[i]++;
(gdb) bt
#0 0x000000000040220c in sanity_tests_sd () at pagemap_ioctl.c:198 #1 0x0000000000404e14 in main () at pagemap_ioctl.c:846 ()

/proc/$PID/stack is empty. Not sure why. I can see stack trace of other applications, but not this one's.

Let me send better reproducer for you.

Quoted 4 lines

Could you try "cat /proc/$PID/stack" to see where does your thread stuck at?

Signature · 2 lines
BR,
Muhammad Usama Anjum
#5 lore raw

Hi Peter,

Finally I've found the bug.

Quoted 96 lines · Muhammad Usama Anjum
Quoted 95 lines · Peter Xu
Quoted 1 line · Muhammad Usama Anjum

Hi Peter,

Hi, Muhammad,

Quoted 90 lines

Thank you so much for sending.

Quoted 75 lines · Peter Xu
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Results:

Quoted 7 lines
        Test DEFAULT: 2
        Test PRE-READ: 3277099 (pre-fault 3253826)
        Test MADVISE: 2250361 (pre-fault 2226310)
        Test WP-UNPOPULATE: 20850

I'll add these information into the commit message when there's a new version.

I'm hitting a bug where I'm unable to write to the memory after adding this patch and wp the memory. I'm hitting this case in your test and my tests as well. Please apply the following diff to your test to reproduce on your end:

Quoted 7 lines
Quoted 7 lines

uffd_wp_perf.c 2023-02-28 12:13:11.077827160 +0500 line 117

Quoted 7 lines
uffd_wp_perf.c 2023-02-28 12:13:11.077827160 +0500 renamed from uffd_wp_perf.c.orig 2023-02-28 12:09:38.971820791 +0500 +1 -0
@@ -114,6 +114,7 @@
         start1 = get_usec();
     }
     wp_range(uffd, buffer, SIZE, true);
+    buffer[0] = 'a';

While using WP_UNPOPULATED, we get stuck if newly allocated memory is read without initialization. This can be reproduced by either of the following statements:

    printf("%c", buffer[0]);
    buffer[0]++;

This bug has start to appear on this patch. How are you handling reading newly allocated memory when WP_UNPOPULATED is defined?

Running my pagemap_ioctl selftest as benchmark in a VM:
without zeropage / wp_unpopulated (decide from pte_none() if page is dirty or not, buggy and wrong implementation, just for reference) 26.608 seconds
with zeropage
39.203 seconds
with wp_unpopulated
62.907 seconds

136% worse performance overall
60% worse performance of unpopulated than zeropage

Quoted 40 lines
Quoted 6 lines

uffd_wp_perf.c 2023-02-28 12:13:11.077827160 +0500 line 120

Quoted 3 lines
uffd_wp_perf.c 2023-02-28 12:13:11.077827160 +0500 renamed from uffd_wp_perf.c.orig 2023-02-28 12:09:38.971820791 +0500 +0 -0
     if (start1 == 0)
          printf("%"PRIu64"\n", get_usec() - start);
     else

This is expected, because the test didn't start any fault resolving thread, so the write will block until someone unprotects the page.

Ohh.. sorry. Wrong reproducer.

Quoted 3 lines

But it shouldn't happen to your use case if you applied both WP_UNPOPULATED & WP_ASYNC.

I'm using both WP_UNPOPULATED and ASYNC. The program gets stuck at right time:

1..57
ok 1 sanity_tests_sd wrong flag specified
ok 2 sanity_tests_sd wrong mask specified
ok 3 sanity_tests_sd wrong return mask specified
ok 4 sanity_tests_sd mixture of correct and wrong flag ok 5 sanity_tests_sd PM_SCAN_OP_WP cannot be used without get ok 6 sanity_tests_sd Clear area with larger vec size
^C
Program received signal SIGINT, Interrupt. 0x000000000040220c in sanity_tests_sd () at pagemap_ioctl.c:198
198 mem[i]++;
(gdb) bt
#0 0x000000000040220c in sanity_tests_sd () at pagemap_ioctl.c:198 #1 0x0000000000404e14 in main () at pagemap_ioctl.c:846 ()

/proc/$PID/stack is empty. Not sure why. I can see stack trace of other applications, but not this one's.

Let me send better reproducer for you.

Quoted 4 lines

Could you try "cat /proc/$PID/stack" to see where does your thread stuck at?

Signature · 2 lines
BR,
Muhammad Usama Anjum
author reviewer
#6 lore raw
Quoted 1 line · Muhammad Usama Anjum

Hi Peter,

Hi, Muhammad,

Quoted 8 lines

While using WP_UNPOPULATED, we get stuck if newly allocated memory is read without initialization. This can be reproduced by either of the following statements:

    printf("%c", buffer[0]);
    buffer[0]++;

This bug has start to appear on this patch. How are you handling reading newly allocated memory when WP_UNPOPULATED is defined?

Yes it's a bug, thanks for the reproducer. You're right I missed a trivial but important detail. Could you try apply below on top?

---8<---

mm/memory.c +1 -1
index 46934133bd0b..2f4b3892948b 100644
@@ -4062,7 +4062,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
                                                vma->vm_page_prot));
                vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
                                vmf->address, &vmf->ptl);
-               if (!pte_none(*vmf->pte)) {
+               if (vmf_pte_changed(vmf)) {
                        update_mmu_tlb(vma, vmf->address, vmf->pte);
                        goto unlock;
                }

---8<---

I can send a new version after you confirmed it at least works on your side. I'll also add some more test to cover that in the next version.

The current smoke test within this patch is really light; I somehow rely on you on this patch on the testing side, and thanks for that.

Quoted 11 lines

Running my pagemap_ioctl selftest as benchmark in a VM:
without zeropage / wp_unpopulated (decide from pte_none() if page is dirty or not, buggy and wrong implementation, just for reference) 26.608 seconds
with zeropage
39.203 seconds
with wp_unpopulated
62.907 seconds

136% worse performance overall
60% worse performance of unpopulated than zeropage

Yes this is unfortunate, because we're protecting more things than before when with WP_ZEROPAGE / WP_UNPOPULATED but that's what it is for (when we want to make sure that accuracy on the holes).

I didn't look closer to your whole test suite yet, but my pure test on protection above should mean that it's still much better for such a use case than either (1) pre-read or (2) MADV_POPULATE_READ.

Again, I hope the performance result is not a concern to you. If it is, please let us know.

Thanks,

Signature · 1 line
Peter Xu
#7 lore raw
Quoted 23 lines · Peter Xu
Quoted 1 line · Muhammad Usama Anjum

Hi Peter,

Hi, Muhammad,

Quoted 8 lines

While using WP_UNPOPULATED, we get stuck if newly allocated memory is read without initialization. This can be reproduced by either of the following statements:

    printf("%c", buffer[0]);
    buffer[0]++;

This bug has start to appear on this patch. How are you handling reading newly allocated memory when WP_UNPOPULATED is defined?

Yes it's a bug, thanks for the reproducer. You're right I missed a trivial but important detail. Could you try apply below on top?

---8<---

mm/memory.c +0 -0
index 46934133bd0b..2f4b3892948b 100644

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf)

Quoted 9 lines
mm/memory.c +1 -2
@@ -4062,7 +4062,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
                                                vma->vm_page_prot));
                vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
                                vmf->address, &vmf->ptl);
-               if (!pte_none(*vmf->pte)) {
+               if (vmf_pte_changed(vmf)) {
                        update_mmu_tlb(vma, vmf->address, vmf->pte);
                        goto unlock;
                }
---8<---

This patch works. Thank you so much!

Quoted 19 lines

I can send a new version after you confirmed it at least works on your side. I'll also add some more test to cover that in the next version.

The current smoke test within this patch is really light; I somehow rely on you on this patch on the testing side, and thanks for that.

Quoted 11 lines

Running my pagemap_ioctl selftest as benchmark in a VM:
without zeropage / wp_unpopulated (decide from pte_none() if page is dirty or not, buggy and wrong implementation, just for reference) 26.608 seconds
with zeropage
39.203 seconds
with wp_unpopulated
62.907 seconds

136% worse performance overall
60% worse performance of unpopulated than zeropage

Quoted 7 lines

Yes this is unfortunate, because we're protecting more things than before when with WP_ZEROPAGE / WP_UNPOPULATED but that's what it is for (when we want to make sure that accuracy on the holes).

I didn't look closer to your whole test suite yet, but my pure test on protection above should mean that it's still much better for such a use case than either (1) pre-read or (2) MADV_POPULATE_READ.

Ohh... I should stop comparing UNPOPULATE with buggy implementation and compare with pre-read. I've compared apples with oranges.

I'll do better benchmark for the comparison sake. I'll let you know if the performance is becoming an issue. Overall we need pagemap_ioctl + UFFD to correctly emulate Windows syscall. Secondly we also need good performance (more the better).

Quoted 6 lines

Again, I hope the performance result is not a concern to you. If it is, please let us know.

Thanks,

Signature · 2 lines
BR,
Muhammad Usama Anjum
#8 lore raw
Quoted 58 lines · Muhammad Usama Anjum
Quoted 23 lines · Peter Xu
Quoted 1 line · Muhammad Usama Anjum

Hi Peter,

Hi, Muhammad,

Quoted 8 lines

While using WP_UNPOPULATED, we get stuck if newly allocated memory is read without initialization. This can be reproduced by either of the following statements:

     printf("%c", buffer[0]);
     buffer[0]++;

This bug has start to appear on this patch. How are you handling reading newly allocated memory when WP_UNPOPULATED is defined?

Yes it's a bug, thanks for the reproducer. You're right I missed a trivial but important detail. Could you try apply below on top?

---8<---

mm/memory.c +0 -0
index 46934133bd0b..2f4b3892948b 100644

mm/memory.c static vm_fault_t do_anonymous_page(struct vm_fault *vmf)

Quoted 9 lines
mm/memory.c +1 -2
@@ -4062,7 +4062,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf)
                                                 vma->vm_page_prot));
                 vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
                                 vmf->address, &vmf->ptl);
-               if (!pte_none(*vmf->pte)) {
+               if (vmf_pte_changed(vmf)) {
                         update_mmu_tlb(vma, vmf->address, vmf->pte);
                         goto unlock;
                 }
---8<---

This patch works. Thank you so much!

Quoted 23 lines

I can send a new version after you confirmed it at least works on your side. I'll also add some more test to cover that in the next version.

The current smoke test within this patch is really light; I somehow rely on you on this patch on the testing side, and thanks for that.

Quoted 11 lines

Running my pagemap_ioctl selftest as benchmark in a VM:
without zeropage / wp_unpopulated (decide from pte_none() if page is dirty or not, buggy and wrong implementation, just for reference) 26.608 seconds
with zeropage
39.203 seconds
with wp_unpopulated
62.907 seconds

136% worse performance overall
60% worse performance of unpopulated than zeropage

Yes this is unfortunate, because we're protecting more things than before when with WP_ZEROPAGE / WP_UNPOPULATED but that's what it is for (when we want to make sure that accuracy on the holes).

Quoted 5 lines
Quoted 3 lines

I didn't look closer to your whole test suite yet, but my pure test on protection above should mean that it's still much better for such a use case than either (1) pre-read or (2) MADV_POPULATE_READ.

Ohh... I should stop comparing UNPOPULATE with buggy implementation and compare with pre-read. I've compared apples with oranges.

Note that I think there are ways to avoid that overhead (as raised in reply to Peter's reply), so IMHO it's still valuable to know which benefit we could have without allocating pagetables and placing the shared zeropages.

Quoted 5 lines

I'll do better benchmark for the comparison sake. I'll let you know if the performance is becoming an issue. Overall we need pagemap_ioctl + UFFD to correctly emulate Windows syscall. Secondly we also need good performance (more the better).

I'm curious, are you eventually applying UFFD to possibly large (sparse) VMAs, that are eventually even only PROT_READ or PROT_WRITE?

Especially for such large sparse VMAs, the current way of allocating pagetables to place markers/zeropages is far from optimal.

Signature · 3 lines
Thanks,

David / dhildenb
author reviewer
#9 lore raw
Quoted 2 lines · David Hildenbrand

Especially for such large sparse VMAs, the current way of allocating pagetables to place markers/zeropages is far from optimal.

IMHO that's not a generic workload. As mentioned in the reply there, I would suggest we go with simple then we have space to optimize it in the future if necessary, because the API will be the same.

If I want to monitor a super sparse VMA on dirty, one thing I can already try is register with MISSING + WP (!WP_UNPOPULATED). It may be a good intermediate solution to avoid any marker ovearheads.

Signature · 1 line
Peter Xu
#10 lore raw
Quoted 8 lines · Peter Xu
Quoted 2 lines · David Hildenbrand

Especially for such large sparse VMAs, the current way of allocating pagetables to place markers/zeropages is far from optimal.

IMHO that's not a generic workload. As mentioned in the reply there, I would suggest we go with simple then we have space to optimize it in the future if necessary, because the API will be the same.

I disagree with "generic workload", we use sparse mmaps all over the place, and when blindly used by e.g., CRIU, we'll simply end up wasting memory and time.

But I already agreed that this optimization that is a separate thing to implement.

Signature · 3 lines
Thanks,

David / dhildenb
#11 lore raw
Quoted 8 lines · David Hildenbrand
Quoted 7 lines · Peter Xu
Quoted 2 lines · David Hildenbrand

Especially for such large sparse VMAs, the current way of allocating pagetables to place markers/zeropages is far from optimal.

IMHO that's not a generic workload.  As mentioned in the reply there, I would suggest we go with simple then we have space to optimize it in the future if necessary, because the API will be the same.

This is a good idea.

I'm trying to understand why aren't we going with most optimized implementation. Why aren't we targeting it at this point in time?

Quoted 5 lines
Quoted 1 line

I disagree with "generic workload", we use sparse mmaps all over the place, and when blindly used by e.g., CRIU, we'll simply end up wasting memory and time.

I've heard about a use case where a file of size 10s of GBs can be mapped to the memory and then accessed off and on. We need to handle this correctly and efficiently.

Quoted 4 lines

But I already agreed that this optimization that is a separate thing to implement.

Signature · 2 lines
BR,
Muhammad Usama Anjum
author reviewer
#12 lore raw
Quoted 3 lines · Muhammad Usama Anjum

I've heard about a use case where a file of size 10s of GBs can be mapped to the memory and then accessed off and on. We need to handle this correctly and efficiently.

Note again that besides any slowness it'll stop working for generic FSs.

One way is we can consider enabling uffd-wp async for all archs, but I need to think more about it, and it will not be compatible with any other modes.

Signature · 1 line
Peter Xu
#13 lore raw
In reply to Peter Xu
Quoted 73 lines · Peter Xu
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Quoted 6 lines

Results:

        Test DEFAULT: 2
        Test PRE-READ: 3277099 (pre-fault 3253826)
        Test MADVISE: 2250361 (pre-fault 2226310)
        Test WP-UNPOPULATE: 20850

In your case:
Default < WP-UNPOPULATE < MADVISE < PRE-READ

In my testing on next-20230228 with this patch and my uffd async patch:

Test DEFAULT: 6
Test PRE-READ: 37157 (pre-fault 37006)
Test MADVISE: 4884 (pre-fault 4465)
Test WP-UNPOPULATE: 17794

DEFAULT < MADVISE < WP-UNPOPULATE < PRE-READ

On my setup, MADVISE is performing better than WP-UNPOPULATE consistently. I'm not sure why I'm getting this discrepancy here. I've liked your results to be honest where we perform better with WP-UNPOPULATE than MADVISE. What can be done to get consistent benchmarks over your and my side?

Quoted 6 lines

I'll add these information into the commit message when there's a new version.

[1] https://github.com/xzpeter/clibs/blob/master/uffd-test/uffd-wp-perf.c

Signature · 2 lines
BR,
Muhammad Usama Anjum
#14 lore raw
Quoted 91 lines · Muhammad Usama Anjum
Quoted 73 lines · Peter Xu
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Quoted 6 lines

Results:

         Test DEFAULT: 2
         Test PRE-READ: 3277099 (pre-fault 3253826)
         Test MADVISE: 2250361 (pre-fault 2226310)
         Test WP-UNPOPULATE: 20850

In your case:
Default < WP-UNPOPULATE < MADVISE < PRE-READ

In my testing on next-20230228 with this patch and my uffd async patch:

Test DEFAULT: 6
Test PRE-READ: 37157 (pre-fault 37006)
Test MADVISE: 4884 (pre-fault 4465)
Test WP-UNPOPULATE: 17794

Quoted 6 lines

DEFAULT < MADVISE < WP-UNPOPULATE < PRE-READ

On my setup, MADVISE is performing better than WP-UNPOPULATE consistently. I'm not sure why I'm getting this discrepancy here. I've liked your results to be honest where we perform better with WP-UNPOPULATE than MADVISE. What can be done to get consistent benchmarks over your and my side?

Probably because the current approach from Peter uses uffd-wp markers, and these markers can currently only reside on the PTE level, not on the PMD level yet.

With MADVISE you get a huge zeropage and avoid dealing with PTEs.

Signature · 3 lines
Thanks,

David / dhildenb
author reviewer
#15 lore raw
Quoted 99 lines · David Hildenbrand
Quoted 91 lines · Muhammad Usama Anjum
Quoted 73 lines · Peter Xu
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Quoted 6 lines

Results:

         Test DEFAULT: 2
         Test PRE-READ: 3277099 (pre-fault 3253826)
         Test MADVISE: 2250361 (pre-fault 2226310)
         Test WP-UNPOPULATE: 20850

In your case:
Default < WP-UNPOPULATE < MADVISE < PRE-READ

In my testing on next-20230228 with this patch and my uffd async patch:

Test DEFAULT: 6
Test PRE-READ: 37157 (pre-fault 37006)
Test MADVISE: 4884 (pre-fault 4465)
Test WP-UNPOPULATE: 17794

Quoted 6 lines

DEFAULT < MADVISE < WP-UNPOPULATE < PRE-READ

On my setup, MADVISE is performing better than WP-UNPOPULATE consistently. I'm not sure why I'm getting this discrepancy here. I've liked your results to be honest where we perform better with WP-UNPOPULATE than MADVISE. What can be done to get consistent benchmarks over your and my side?

Quoted 5 lines

Probably because the current approach from Peter uses uffd-wp markers, and these markers can currently only reside on the PTE level, not on the PMD level yet.

With MADVISE you get a huge zeropage and avoid dealing with PTEs.

Yes, probably. But then when write happens it'll be done there when split, so the overhead was delayed.

Meanwhile I'll retest again (probably tomorrow..) with bare metals with THP on/off to double check.

Muhammad, do you think the current performance will work for you?

Especially I want to double check with you again on whether XFS/EXT4/... will be needed for the tracking purpose so you can reply here together. We shouldn't merge anything that doesn't have at least one existing good use case, and we may need to rethink if it's not.

For performance, one approach is probably making uffd-wp async separate from other features, where we can revert the meaning of uffd-wp bit to mimic what soft-dirty does (I think this will look closer to what David mentioned in the other thread), by defining uffd-wp=1 as "written" and uffd-wp=0 as clean.

IIUC that'll make it one bit and work as fast as soft-dirty, meanwhile all uffd-wp marker things can hopefully still be maintained. However I really don't like it to violate a lot of things, e.g., when UFFDIO_WRITEPROTECT another round we'll need to DROP uffd-wp if async, but APPLY uffd-wp if sync... So in general it'll need more thoughts and slower to do.

Signature · 1 line
Peter Xu
#16 lore raw
Quoted 104 lines · Peter Xu
Quoted 103 lines · David Hildenbrand
Quoted 91 lines · Muhammad Usama Anjum
Quoted 73 lines · Peter Xu
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Quoted 6 lines

Results:

         Test DEFAULT: 2
         Test PRE-READ: 3277099 (pre-fault 3253826)
         Test MADVISE: 2250361 (pre-fault 2226310)
         Test WP-UNPOPULATE: 20850

In your case:
Default < WP-UNPOPULATE < MADVISE < PRE-READ

In my testing on next-20230228 with this patch and my uffd async patch:

Test DEFAULT: 6
Test PRE-READ: 37157 (pre-fault 37006)
Test MADVISE: 4884 (pre-fault 4465)
Test WP-UNPOPULATE: 17794

Quoted 6 lines

DEFAULT < MADVISE < WP-UNPOPULATE < PRE-READ

On my setup, MADVISE is performing better than WP-UNPOPULATE consistently. I'm not sure why I'm getting this discrepancy here. I've liked your results to be honest where we perform better with WP-UNPOPULATE than MADVISE. What can be done to get consistent benchmarks over your and my side?

Probably because the current approach from Peter uses uffd-wp markers, and these markers can currently only reside on the PTE level, not on the PMD level yet.

Quoted 7 lines
Quoted 1 line

With MADVISE you get a huge zeropage and avoid dealing with PTEs.

Yes, probably. But then when write happens it'll be done there when split, so the overhead was delayed.

Meanwhile I'll retest again (probably tomorrow..) with bare metals with THP on/off to double check.

Turning on/off THP has effect.

(1) With huge page disabled
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 1111453 (pre-fault 1101011)
Test MADVISE: 278276 (pre-fault 266378)
Test WP-UNPOPULATE: 11712

(2) With Huge page enabled
echo always > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 22521 (pre-fault 22348)
Test MADVISE: 4909 (pre-fault 4743)
Test WP-UNPOPULATE: 14448

Quoted 2 lines

Muhammad, do you think the current performance will work for you?

Yes, the current performance is good enough for now. We'll be using normal pages for memory on which PAGEMAP_IOCTL will be used for emulating Windows syscall. When normal size pages are used, (1) show that WP-UNPOPULATE has really good performance (278276 --> 11712). Lets proceed with it.

Quoted 5 lines

Especially I want to double check with you again on whether XFS/EXT4/... will be needed for the tracking purpose so you can reply here together. We shouldn't merge anything that doesn't have at least one existing good use case, and we may need to rethink if it's not.

We have been trying to find soft dirty pages correctly on anon memory. Right now with UN_POPULATED, ASYNC and PAGEMAP_IOCTL, we'll be able to achieve this correctly.

We want to track the file mapped memory the same way for another Windows api translation in which file memory will be tracked to find out soft-dirty pages. I believe once it is supported in the mm, filesystem doesn't matter as we'll be tracking memory mapped files.

So we have use cases for both soft-dirty tracking on anon and file backed memories.

Quoted 6 lines

For performance, one approach is probably making uffd-wp async separate from other features, where we can revert the meaning of uffd-wp bit to mimic what soft-dirty does (I think this will look closer to what David mentioned in the other thread), by defining uffd-wp=1 as "written" and uffd-wp=0 as clean.

I'm happy with current state of things. WP Async is just a feature where faults are resolved automatically. The page is WP or not perfectly translates if page is written (dirty) or not with just not (!) operation. If we go and change the meanings of uffd_wp=1 as written, it would cause confusion and probably we'll need a lot more changes to add async feature which is right now quite small changeset.

Quoted 6 lines

IIUC that'll make it one bit and work as fast as soft-dirty, meanwhile all uffd-wp marker things can hopefully still be maintained. However I really don't like it to violate a lot of things, e.g., when UFFDIO_WRITEPROTECT another round we'll need to DROP uffd-wp if async, but APPLY uffd-wp if sync... So in general it'll need more thoughts and slower to do.

Sorry, I'm unable to understand your example here. Can you please elaborate?

Quoted 1 line
Signature · 2 lines
BR,
Muhammad Usama Anjum
author reviewer
#17 lore raw
Quoted 122 lines · Muhammad Usama Anjum
Quoted 104 lines · Peter Xu
Quoted 103 lines · David Hildenbrand
Quoted 91 lines · Muhammad Usama Anjum
Quoted 73 lines · Peter Xu
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Quoted 6 lines

Results:

         Test DEFAULT: 2
         Test PRE-READ: 3277099 (pre-fault 3253826)
         Test MADVISE: 2250361 (pre-fault 2226310)
         Test WP-UNPOPULATE: 20850

In your case:
Default < WP-UNPOPULATE < MADVISE < PRE-READ

In my testing on next-20230228 with this patch and my uffd async patch:

Test DEFAULT: 6
Test PRE-READ: 37157 (pre-fault 37006)
Test MADVISE: 4884 (pre-fault 4465)
Test WP-UNPOPULATE: 17794

Quoted 6 lines

DEFAULT < MADVISE < WP-UNPOPULATE < PRE-READ

On my setup, MADVISE is performing better than WP-UNPOPULATE consistently. I'm not sure why I'm getting this discrepancy here. I've liked your results to be honest where we perform better with WP-UNPOPULATE than MADVISE. What can be done to get consistent benchmarks over your and my side?

Probably because the current approach from Peter uses uffd-wp markers, and these markers can currently only reside on the PTE level, not on the PMD level yet.

Quoted 7 lines
Quoted 1 line

With MADVISE you get a huge zeropage and avoid dealing with PTEs.

Yes, probably. But then when write happens it'll be done there when split, so the overhead was delayed.

Meanwhile I'll retest again (probably tomorrow..) with bare metals with THP on/off to double check.

Turning on/off THP has effect.

(1) With huge page disabled
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 1111453 (pre-fault 1101011)
Test MADVISE: 278276 (pre-fault 266378)
Test WP-UNPOPULATE: 11712

Quoted 7 lines

(2) With Huge page enabled
echo always > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 22521 (pre-fault 22348)
Test MADVISE: 4909 (pre-fault 4743)
Test WP-UNPOPULATE: 14448

Thanks. Yes then this is explaine-able to me. As mentioned previously, thp will drastically speedup everything during prefault+ioctl(UFFDIO_WP), though that overhead will be applied when the 1st write triggers later, afaik.

Quoted 7 lines
Quoted 3 lines

Muhammad, do you think the current performance will work for you?

Yes, the current performance is good enough for now. We'll be using normal pages for memory on which PAGEMAP_IOCTL will be used for emulating Windows syscall. When normal size pages are used, (1) show that WP-UNPOPULATE has really good performance (278276 --> 11712). Lets proceed with it.

OK.

Quoted 6 lines
Quoted 6 lines

Especially I want to double check with you again on whether XFS/EXT4/... will be needed for the tracking purpose so you can reply here together. We shouldn't merge anything that doesn't have at least one existing good use case, and we may need to rethink if it's not.

Quoted 8 lines

We have been trying to find soft dirty pages correctly on anon memory. Right now with UN_POPULATED, ASYNC and PAGEMAP_IOCTL, we'll be able to achieve this correctly.

We want to track the file mapped memory the same way for another Windows api translation in which file memory will be tracked to find out soft-dirty pages. I believe once it is supported in the mm, filesystem doesn't matter as we'll be tracking memory mapped files.

Some more elaboration would be great here.

For example, what is the other windows API? What will you do with the dirty information? What is the impact of supporting only shmem/hugetlbfs for "!anon"?

Quoted 10 lines

So we have use cases for both soft-dirty tracking on anon and file backed memories.

Quoted 6 lines

For performance, one approach is probably making uffd-wp async separate from other features, where we can revert the meaning of uffd-wp bit to mimic what soft-dirty does (I think this will look closer to what David mentioned in the other thread), by defining uffd-wp=1 as "written" and uffd-wp=0 as clean.

Quoted 6 lines

I'm happy with current state of things. WP Async is just a feature where faults are resolved automatically. The page is WP or not perfectly translates if page is written (dirty) or not with just not (!) operation. If we go and change the meanings of uffd_wp=1 as written, it would cause confusion and probably we'll need a lot more changes to add async feature which is right now quite small changeset.

Yes, that's also why I wanted to keep it like this unless necessary. It makes all uffd features still more aligned with everything so more maintainable.

Quoted 8 lines
Quoted 7 lines

IIUC that'll make it one bit and work as fast as soft-dirty, meanwhile all uffd-wp marker things can hopefully still be maintained. However I really don't like it to violate a lot of things, e.g., when UFFDIO_WRITEPROTECT another round we'll need to DROP uffd-wp if async, but APPLY uffd-wp if sync... So in general it'll need more thoughts and slower to do.

Sorry, I'm unable to understand your example here. Can you please elaborate?

I think it's mostly what you stated above, the changeset will just get more involved because of either the redefinition of uffd-wp depending on feature enabled, or using yet another bit (e.g., if you still remember the very original reply I was planning to reuse soft-dirty bit but used together with VM_UFFD_WP; I think that resolves the degraded perf on unpopulated memory too because it'll work more like soft-dirty).

This is some detail which, IMHO, we can leave that for later. To me I'm more curious on knowing more details of your use case, and how uffd could suite that the best way it could, assuming that (or part of it like the unpopulate work here) can also be leveraged by others. Your answers to above questions may help.

Thanks,

Signature · 1 line
Peter Xu
#18 lore raw
Quoted 115 lines · Peter Xu
Quoted 114 lines · Muhammad Usama Anjum
Quoted 104 lines · Peter Xu
Quoted 103 lines · David Hildenbrand
Quoted 91 lines · Muhammad Usama Anjum
Quoted 73 lines · Peter Xu
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Quoted 6 lines

Results:

         Test DEFAULT: 2
         Test PRE-READ: 3277099 (pre-fault 3253826)
         Test MADVISE: 2250361 (pre-fault 2226310)
         Test WP-UNPOPULATE: 20850

In your case:
Default < WP-UNPOPULATE < MADVISE < PRE-READ

In my testing on next-20230228 with this patch and my uffd async patch:

Test DEFAULT: 6
Test PRE-READ: 37157 (pre-fault 37006)
Test MADVISE: 4884 (pre-fault 4465)
Test WP-UNPOPULATE: 17794

Quoted 6 lines

DEFAULT < MADVISE < WP-UNPOPULATE < PRE-READ

On my setup, MADVISE is performing better than WP-UNPOPULATE consistently. I'm not sure why I'm getting this discrepancy here. I've liked your results to be honest where we perform better with WP-UNPOPULATE than MADVISE. What can be done to get consistent benchmarks over your and my side?

Probably because the current approach from Peter uses uffd-wp markers, and these markers can currently only reside on the PTE level, not on the PMD level yet.

Quoted 7 lines
Quoted 1 line

With MADVISE you get a huge zeropage and avoid dealing with PTEs.

Yes, probably. But then when write happens it'll be done there when split, so the overhead was delayed.

Meanwhile I'll retest again (probably tomorrow..) with bare metals with THP on/off to double check.

Turning on/off THP has effect.

Quoted 7 lines
Quoted 7 lines

(1) With huge page disabled
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 1111453 (pre-fault 1101011)
Test MADVISE: 278276 (pre-fault 266378)
Test WP-UNPOPULATE: 11712

Hi Paul,

We were reading, touching or faulting memory before engaging wp. This reading is called pre-read in this benchmark. We can fault the memory through madvise as well, it is called madvise here. WP-Unpopulated is being added in this patch which will remove the need to pre-fault the memory and it is way faster as shown by the numbers.

Quoted 34 lines
Quoted 8 lines

(2) With Huge page enabled
echo always > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 22521 (pre-fault 22348)
Test MADVISE: 4909 (pre-fault 4743)
Test WP-UNPOPULATE: 14448

Thanks. Yes then this is explaine-able to me. As mentioned previously, thp will drastically speedup everything during prefault+ioctl(UFFDIO_WP), though that overhead will be applied when the 1st write triggers later, afaik.

Quoted 7 lines
Quoted 3 lines

Muhammad, do you think the current performance will work for you?

Yes, the current performance is good enough for now. We'll be using normal pages for memory on which PAGEMAP_IOCTL will be used for emulating Windows syscall. When normal size pages are used, (1) show that WP-UNPOPULATE has really good performance (278276 --> 11712). Lets proceed with it.

OK.

Quoted 10 lines
Quoted 6 lines

Especially I want to double check with you again on whether XFS/EXT4/... will be needed for the tracking purpose so you can reply here together. We shouldn't merge anything that doesn't have at least one existing good use case, and we may need to rethink if it's not.

We have been trying to find soft dirty pages correctly on anon memory. Right now with UN_POPULATED, ASYNC and PAGEMAP_IOCTL, we'll be able to achieve this correctly.

Quoted 6 lines
Quoted 4 lines

We want to track the file mapped memory the same way for another Windows api translation in which file memory will be tracked to find out soft-dirty pages. I believe once it is supported in the mm, filesystem doesn't matter as we'll be tracking memory mapped files.

Some more elaboration would be great here.

There is something called "writecopy" in Windows. Paul has given me some details.

@Paul can you share more details that:
1) What is writecopy?
2) If there is no documentation or documented behavior, Can you please share the current reference implementation in Wine even though it is buggy at the time? (I can implement correct implementation with this patch and my wip v11 pagemap_ioctl patches for testing purpose.)

Quoted 49 lines

For example, what is the other windows API? What will you do with the dirty information? What is the impact of supporting only shmem/hugetlbfs for "!anon"?

Quoted 10 lines

So we have use cases for both soft-dirty tracking on anon and file backed memories.

Quoted 6 lines

For performance, one approach is probably making uffd-wp async separate from other features, where we can revert the meaning of uffd-wp bit to mimic what soft-dirty does (I think this will look closer to what David mentioned in the other thread), by defining uffd-wp=1 as "written" and uffd-wp=0 as clean.

Quoted 6 lines

I'm happy with current state of things. WP Async is just a feature where faults are resolved automatically. The page is WP or not perfectly translates if page is written (dirty) or not with just not (!) operation. If we go and change the meanings of uffd_wp=1 as written, it would cause confusion and probably we'll need a lot more changes to add async feature which is right now quite small changeset.

Yes, that's also why I wanted to keep it like this unless necessary. It makes all uffd features still more aligned with everything so more maintainable.

Quoted 8 lines
Quoted 7 lines

IIUC that'll make it one bit and work as fast as soft-dirty, meanwhile all uffd-wp marker things can hopefully still be maintained. However I really don't like it to violate a lot of things, e.g., when UFFDIO_WRITEPROTECT another round we'll need to DROP uffd-wp if async, but APPLY uffd-wp if sync... So in general it'll need more thoughts and slower to do.

Sorry, I'm unable to understand your example here. Can you please elaborate?

I think it's mostly what you stated above, the changeset will just get more involved because of either the redefinition of uffd-wp depending on feature enabled, or using yet another bit (e.g., if you still remember the very original reply I was planning to reuse soft-dirty bit but used together with VM_UFFD_WP; I think that resolves the degraded perf on unpopulated memory too because it'll work more like soft-dirty).

This is some detail which, IMHO, we can leave that for later. To me I'm more curious on knowing more details of your use case, and how uffd could suite that the best way it could, assuming that (or part of it like the unpopulate work here) can also be leveraged by others. Your answers to above questions may help.

Thanks,

Signature · 2 lines
BR,
Muhammad Usama Anjum
#19 lore raw
Quoted 174 lines · Muhammad Usama Anjum
Quoted 115 lines · Peter Xu
Quoted 114 lines · Muhammad Usama Anjum
Quoted 104 lines · Peter Xu
Quoted 103 lines · David Hildenbrand
Quoted 91 lines · Muhammad Usama Anjum
Quoted 73 lines · Peter Xu
Quoted 51 lines · Peter Xu

This is a new feature that controls how uffd-wp handles none ptes. When it's set, the kernel will handle anonymous memory the same way as file memory, by allowing the user to wr-protect unpopulated ptes.

File memories handles none ptes consistently by allowing wr-protecting of none ptes because of the unawareness of page cache being exist or not. For anonymous it was not as persistent because we used to assume that we don't need protections on none ptes or known zero pages.

One use case of such a feature bit was VM live snapshot, where if without wr-protecting empty ptes the snapshot can contain random rubbish in the holes of the anonymous memory, which can cause misbehave of the guest when the guest OS assumes the pages should be all zeros.

QEMU worked it around by pre-populate the section with reads to fill in zero page entries before starting the whole snapshot process [1].

Recently there's another need raised on using userfaultfd wr-protect for detecting dirty pages (to replace soft-dirty in some cases) [2]. In that case if without being able to wr-protect none ptes by default, the dirty info can get lost, since we cannot treat every none pte to be dirty (the current design is identify a page dirty based on uffd-wp bit being cleared).

In general, we want to be able to wr-protect empty ptes too even for anonymous.

This patch implements UFFD_FEATURE_WP_UNPOPULATED so that it'll make uffd-wp handling on none ptes being consistent no matter what the memory type is underneath. It doesn't have any impact on file memories so far because we already have pte markers taking care of that. So it only affects anonymous.

The feature bit is by default off, so the old behavior will be maintained. Sometimes it may be wanted because the wr-protect of none ptes will contain overheads not only during UFFDIO_WRITEPROTECT (by applying pte markers to anonymous), but also on creating the pgtables to store the pte markers. So there's potentially less chance of using thp on the first fault for a none pmd or larger than a pmd.

The major implementation part is teaching the whole kernel to understand pte markers even for anonymously mapped ranges, meanwhile allowing the UFFDIO_WRITEPROTECT ioctl to apply pte markers for anonymous too when the new feature bit is set.

Note that even if the patch subject starts with mm/uffd, there're a few small refactors to major mm path of handling anonymous page faults. But they should be straightforward.

So far, add a very light smoke test within the userfaultfd kselftest pagemap unit test to make sure anon pte markers work.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/Y+v2HJ8+3i%2FKzDBu@x1n/

---
v1->v2:
- Use pte markers rather than populate zero pages when protect [David]
- Rename WP_ZEROPAGE to WP_UNPOPULATED [David]

Some very initial performance numbers (I only ran in a VM but it should be similar, unit is "us") below as requested. The measurement is about time spent when wr-protecting 10G range of empty but mapped memory. It's done in a VM, assuming we'll get similar results on bare metal.

Four test cases:

- default UFFDIO_WP
- pre-read the memory, then UFFDIO_WP (what QEMU does right now)
- pre-fault using MADV_POPULATE_READ, then default UFFDIO_WP
- UFFDIO_WP with WP_UNPOPULATED

Quoted 6 lines

Results:

         Test DEFAULT: 2
         Test PRE-READ: 3277099 (pre-fault 3253826)
         Test MADVISE: 2250361 (pre-fault 2226310)
         Test WP-UNPOPULATE: 20850

In your case:
Default < WP-UNPOPULATE < MADVISE < PRE-READ

In my testing on next-20230228 with this patch and my uffd async patch:

Test DEFAULT: 6
Test PRE-READ: 37157 (pre-fault 37006)
Test MADVISE: 4884 (pre-fault 4465)
Test WP-UNPOPULATE: 17794

Quoted 6 lines

DEFAULT < MADVISE < WP-UNPOPULATE < PRE-READ

On my setup, MADVISE is performing better than WP-UNPOPULATE consistently. I'm not sure why I'm getting this discrepancy here. I've liked your results to be honest where we perform better with WP-UNPOPULATE than MADVISE. What can be done to get consistent benchmarks over your and my side?

Probably because the current approach from Peter uses uffd-wp markers, and these markers can currently only reside on the PTE level, not on the PMD level yet.

Quoted 7 lines
Quoted 1 line

With MADVISE you get a huge zeropage and avoid dealing with PTEs.

Yes, probably. But then when write happens it'll be done there when split, so the overhead was delayed.

Meanwhile I'll retest again (probably tomorrow..) with bare metals with THP on/off to double check.

Turning on/off THP has effect.

Quoted 7 lines
Quoted 7 lines

(1) With huge page disabled
echo madvise > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 1111453 (pre-fault 1101011)
Test MADVISE: 278276 (pre-fault 266378)
Test WP-UNPOPULATE: 11712

Hi Paul,

We were reading, touching or faulting memory before engaging wp. This reading is called pre-read in this benchmark. We can fault the memory through madvise as well, it is called madvise here. WP-Unpopulated is being added in this patch which will remove the need to pre-fault the memory and it is way faster as shown by the numbers.

Quoted 34 lines
Quoted 8 lines

(2) With Huge page enabled
echo always > /sys/kernel/mm/transparent_hugepage/enabled ./uffd_wp_perf
Test DEFAULT: 4
Test PRE-READ: 22521 (pre-fault 22348)
Test MADVISE: 4909 (pre-fault 4743)
Test WP-UNPOPULATE: 14448

Thanks. Yes then this is explaine-able to me. As mentioned previously, thp will drastically speedup everything during prefault+ioctl(UFFDIO_WP), though that overhead will be applied when the 1st write triggers later, afaik.

Quoted 7 lines
Quoted 3 lines

Muhammad, do you think the current performance will work for you?

Yes, the current performance is good enough for now. We'll be using normal pages for memory on which PAGEMAP_IOCTL will be used for emulating Windows syscall. When normal size pages are used, (1) show that WP-UNPOPULATE has really good performance (278276 --> 11712). Lets proceed with it.

OK.

Quoted 10 lines
Quoted 6 lines

Especially I want to double check with you again on whether XFS/EXT4/... will be needed for the tracking purpose so you can reply here together. We shouldn't merge anything that doesn't have at least one existing good use case, and we may need to rethink if it's not.

We have been trying to find soft dirty pages correctly on anon memory. Right now with UN_POPULATED, ASYNC and PAGEMAP_IOCTL, we'll be able to achieve this correctly.

Quoted 6 lines
Quoted 4 lines

We want to track the file mapped memory the same way for another Windows api translation in which file memory will be tracked to find out soft-dirty pages. I believe once it is supported in the mm, filesystem doesn't matter as we'll be tracking memory mapped files.

Some more elaboration would be great here.

There is something called "writecopy" in Windows. Paul has given me some details.

Quoted 6 lines

@Paul can you share more details that:
1) What is writecopy?
2) If there is no documentation or documented behavior, Can you please share the current reference implementation in Wine even though it is buggy at the time? (I can implement correct implementation with this patch and my wip v11 pagemap_ioctl patches for testing purpose.)

Hi Peter,

Just had some offline discussion with Paul, lets leave this second use case for future. We don't want to make discussions more entangled. There isn't much clarity on it right now. Lets only complete the first use case.

The current patch seems good. Please can you post the next version by including the missing part. Then I'll also be able to send the v11 of pagemap_ioctl by basing on it.

Quoted 51 lines
Quoted 50 lines

For example, what is the other windows API? What will you do with the dirty information? What is the impact of supporting only shmem/hugetlbfs for "!anon"?

Quoted 10 lines

So we have use cases for both soft-dirty tracking on anon and file backed memories.

Quoted 6 lines

For performance, one approach is probably making uffd-wp async separate from other features, where we can revert the meaning of uffd-wp bit to mimic what soft-dirty does (I think this will look closer to what David mentioned in the other thread), by defining uffd-wp=1 as "written" and uffd-wp=0 as clean.

Quoted 6 lines

I'm happy with current state of things. WP Async is just a feature where faults are resolved automatically. The page is WP or not perfectly translates if page is written (dirty) or not with just not (!) operation. If we go and change the meanings of uffd_wp=1 as written, it would cause confusion and probably we'll need a lot more changes to add async feature which is right now quite small changeset.

Yes, that's also why I wanted to keep it like this unless necessary. It makes all uffd features still more aligned with everything so more maintainable.

Quoted 8 lines
Quoted 7 lines

IIUC that'll make it one bit and work as fast as soft-dirty, meanwhile all uffd-wp marker things can hopefully still be maintained. However I really don't like it to violate a lot of things, e.g., when UFFDIO_WRITEPROTECT another round we'll need to DROP uffd-wp if async, but APPLY uffd-wp if sync... So in general it'll need more thoughts and slower to do.

Sorry, I'm unable to understand your example here. Can you please elaborate?

I think it's mostly what you stated above, the changeset will just get more involved because of either the redefinition of uffd-wp depending on feature enabled, or using yet another bit (e.g., if you still remember the very original reply I was planning to reuse soft-dirty bit but used together with VM_UFFD_WP; I think that resolves the degraded perf on unpopulated memory too because it'll work more like soft-dirty).

This is some detail which, IMHO, we can leave that for later. To me I'm more curious on knowing more details of your use case, and how uffd could suite that the best way it could, assuming that (or part of it like the unpopulate work here) can also be leveraged by others. Your answers to above questions may help.

Thanks,

Signature · 2 lines
BR,
Muhammad Usama Anjum