Skip to contentSkip to conversation

userfaultfd: working set tracking for VM guest memory

RFC PATCH In review Kiryl Shutsemau (Meta) 5 participants 52 messages Apr 14 – Apr 27

Subsystems (16)
lore
Share & tools
lore

Kiryl Shutsemau (Meta)

author reviewer
#0 lore raw

cover userfaultfd: working set tracking for VM guest memory

This series adds userfaultfd support for tracking the working set of VM guest memory, enabling VMMs to identify cold pages and evict them to tiered or remote storage.

== Problem ==

VMMs managing guest memory need to:
1. Track which pages are actively used (working set detection)
2. Safely evict cold pages to slower storage
3. Fetch pages back on demand when accessed again

For shmem-backed guest memory, working set tracking partially works
today: MADV_DONTNEED zaps PTEs while pages stay in page cache, and re-access auto-resolves from cache. But safe eviction still requires synchronous fault interception to prevent data loss races.

For anonymous guest memory (needed for KSM cross-VM deduplication), there is no mechanism at all — clearing a PTE loses the page.

== Solution ==

The series introduces a unified userfaultfd interface that works across both anonymous and shmem-backed memory:

UFFD_FEATURE_MINOR_ANON: extends MODE_MINOR registration to anonymous private memory. Uses the PROT_NONE hinting mechanism (same as NUMA balancing) to make pages inaccessible without freeing them.

UFFD_FEATURE_MINOR_ASYNC: auto-resolves minor faults without handler involvement. The kernel restores PTE permissions immediately and the faulting thread continues. Works for anonymous, shmem, and hugetlbfs.

UFFDIO_DEACTIVATE: marks pages as deactivated. For anonymous memory, sets PROT_NONE on PTEs (pages stay resident). For shmem/hugetlbfs, zaps PTEs (pages stay in page cache).

UFFDIO_SET_MODE: toggles MINOR_ASYNC at runtime, synchronized via mmap_write_lock. Enables the VMM workflow: async mode for lightweight detection, sync mode for race-free eviction.

PAGE_IS_UFFD_DEACTIVATED: PAGEMAP_SCAN category flag for efficient batch detection of cold (still-deactivated) anonymous pages.

== VMM Workflow ==

UFFDIO_DEACTIVATE(all) -- async, no vCPU stalls
sleep(interval)
PAGEMAP_SCAN -- find cold pages
UFFDIO_SET_MODE(sync) -- block faults for eviction
pwrite + MADV_DONTNEED cold pages -- safe, faults block
UFFDIO_SET_MODE(async) -- resume tracking

The same workflow applies to shmem, with a different PAGEMAP_SCAN mask (!PAGE_IS_PRESENT instead of PAGE_IS_UFFD_DEACTIVATED).

== NUMA Balancing ==

NUMA balancing scanning is skipped on anonymous VM_UFFD_MINOR VMAs to avoid protnone conflicts. NUMA locality stats are fed from the uffd fault path via task_numa_fault() so the scheduler retains placement data. Shmem VMAs are unaffected (UFFDIO_DEACTIVATE zaps PTEs there, no protnone involved).

== Testing ==

The series includes 6 new selftests covering async/sync modes, PAGEMAP_SCAN cold detection, GUP through protnone, UFFDIO_SET_MODE toggling, and cleanup on close. All 73 uffd unit tests pass (including hugetlb) across defconfig, allnoconfig, allmodconfig, and randomized configs.

Kiryl Shutsemau (Meta) (12):
userfaultfd: define UAPI constants for anonymous minor faults
userfaultfd: add UFFD_FEATURE_MINOR_ANON registration support
userfaultfd: implement UFFDIO_DEACTIVATE ioctl
userfaultfd: UFFDIO_CONTINUE for anonymous memory
mm: intercept protnone faults on VM_UFFD_MINOR anonymous VMAs
userfaultfd: auto-resolve shmem and hugetlbfs minor faults in async
mode
sched/numa: skip scanning anonymous VM_UFFD_MINOR VMAs
userfaultfd: enable UFFD_FEATURE_MINOR_ANON
mm/pagemap: add PAGE_IS_UFFD_DEACTIVATED to PAGEMAP_SCAN
userfaultfd: add UFFDIO_SET_MODE for runtime sync/async toggle selftests/mm: add userfaultfd anonymous minor fault tests Documentation/userfaultfd: document working set tracking

 Documentation/admin-guide/mm/userfaultfd.rst | 141 ++++-
 fs/proc/task_mmu.c                           |  11 +-
 fs/userfaultfd.c                             | 184 +++++-
 include/linux/huge_mm.h                      |   6 +
 include/linux/mm.h                           |   2 +
 include/linux/sched/numa_balancing.h         |   1 +
 include/linux/userfaultfd_k.h                |  21 +-
 include/trace/events/sched.h                 |   3 +-
 include/uapi/linux/fs.h                      |   1 +
 include/uapi/linux/userfaultfd.h             |  40 +-
 kernel/sched/fair.c                          |  13 +
 mm/huge_memory.c                             |  33 +-
 mm/hugetlb.c                                 |   3 +-
 mm/memory.c                                  |  51 +-
 mm/mprotect.c                                |   9 +-
 mm/shmem.c                                   |   3 +-
 mm/userfaultfd.c                             | 164 +++++-
 tools/testing/selftests/mm/uffd-unit-tests.c | 458 +++++++++++++++
 18 files changed, 1096 insertions(+), 48 deletions(-)

Kiryl Shutsemau (Meta) (12):
userfaultfd: define UAPI constants for anonymous minor faults
userfaultfd: add UFFD_FEATURE_MINOR_ANON registration support
userfaultfd: implement UFFDIO_DEACTIVATE ioctl
userfaultfd: UFFDIO_CONTINUE for anonymous memory
mm: intercept protnone faults on VM_UFFD_MINOR anonymous VMAs
userfaultfd: auto-resolve shmem and hugetlbfs minor faults in async
mode
sched/numa: skip scanning anonymous VM_UFFD_MINOR VMAs
userfaultfd: enable UFFD_FEATURE_MINOR_ANON
mm/pagemap: add PAGE_IS_UFFD_DEACTIVATED to PAGEMAP_SCAN
userfaultfd: add UFFDIO_SET_MODE for runtime sync/async toggle selftests/mm: add userfaultfd anonymous minor fault tests Documentation/userfaultfd: document working set tracking

 Documentation/admin-guide/mm/userfaultfd.rst | 141 +++++-
 fs/proc/task_mmu.c                           |  11 +-
 fs/userfaultfd.c                             | 184 +++++++-
 include/linux/huge_mm.h                      |   6 +
 include/linux/mm.h                           |   2 +
 include/linux/sched/numa_balancing.h         |   1 +
 include/linux/userfaultfd_k.h                |  21 +-
 include/trace/events/sched.h                 |   3 +-
 include/uapi/linux/fs.h                      |   1 +
 include/uapi/linux/userfaultfd.h             |  40 +-
 kernel/sched/fair.c                          |  13 +
 mm/huge_memory.c                             |  33 +-
 mm/hugetlb.c                                 |   3 +-
 mm/memory.c                                  |  51 ++-
 mm/mprotect.c                                |   9 +-
 mm/shmem.c                                   |   3 +-
 mm/userfaultfd.c                             | 164 ++++++-
 tools/testing/selftests/mm/uffd-unit-tests.c | 458 +++++++++++++++++++
 18 files changed, 1096 insertions(+), 48 deletions(-)
Signature · 1 line
2.51.2
author reviewer
#1 lore raw

RFC PATCH 1/12 userfaultfd: define UAPI constants for anonymous minor faults

Add UAPI definitions for userfaultfd working set tracking on anonymous memory:

- UFFD_FEATURE_MINOR_ANON: minor fault support for anonymous memory
- UFFD_FEATURE_MINOR_ASYNC: auto-resolve minor faults without handler
- UFFDIO_DEACTIVATE: mark pages as deactivated (protnone or PTE zap)

Not yet added to UFFD_API_FEATURES or UFFD_API_RANGE_IOCTLS.

Assisted-by Claude:claude-opus-4-6
---
 include/uapi/linux/userfaultfd.h | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
include/uapi/linux/userfaultfd.h +17 -0
index 2841e4ea8f2c..336d07e1b6de 100644
@@ -79,6 +79,7 @@
 #define _UFFDIO_WRITEPROTECT		(0x06)
 #define _UFFDIO_CONTINUE		(0x07)
 #define _UFFDIO_POISON			(0x08)
+#define _UFFDIO_DEACTIVATE		(0x09)
 #define _UFFDIO_API			(0x3F)
 
 /* userfaultfd ioctl ids */
@@ -103,6 +104,8 @@
 				      struct uffdio_continue)
 #define UFFDIO_POISON		_IOWR(UFFDIO, _UFFDIO_POISON, \
 				      struct uffdio_poison)
+#define UFFDIO_DEACTIVATE	_IOR(UFFDIO, _UFFDIO_DEACTIVATE,	\
+				     struct uffdio_range)
 
 /* read() structure */
 struct uffd_msg {
@@ -230,6 +233,18 @@ struct uffdio_api {
 	 *
 	 * UFFD_FEATURE_MOVE indicates that the kernel supports moving an
 	 * existing page contents from userspace.
+	 *
+	 * UFFD_FEATURE_MINOR_ANON indicates that minor fault interception
+	 * is supported for anonymous private memory.  Pages are made
+	 * inaccessible via UFFDIO_DEACTIVATE (sets PROT_NONE while
+	 * preserving the page) and faults are delivered when the pages
+	 * are re-accessed.
+	 *
+	 * UFFD_FEATURE_MINOR_ASYNC indicates asynchronous minor fault
+	 * mode.  When set, faults on deactivated pages are auto-resolved
+	 * by the kernel (PTE permissions restored immediately) without
+	 * delivering a message to the userfaultfd handler.  Use
+	 * PAGEMAP_SCAN to find pages that were not re-accessed.
 	 */
 #define UFFD_FEATURE_PAGEFAULT_FLAG_WP		(1<<0)
 #define UFFD_FEATURE_EVENT_FORK			(1<<1)
@@ -248,6 +263,8 @@ struct uffdio_api {
 #define UFFD_FEATURE_POISON			(1<<14)
 #define UFFD_FEATURE_WP_ASYNC			(1<<15)
 #define UFFD_FEATURE_MOVE			(1<<16)
+#define UFFD_FEATURE_MINOR_ANON			(1<<17)
+#define UFFD_FEATURE_MINOR_ASYNC		(1<<18)
 	__u64 features;
 
 	__u64 ioctls;
Signature · 1 line
2.51.2
author reviewer
#2 lore raw

RFC PATCH 2/12 userfaultfd: add UFFD_FEATURE_MINOR_ANON registration support

In reply to Kiryl Shutsemau (Meta)

Allow UFFDIO_REGISTER_MODE_MINOR on anonymous VMAs when the UFFD_FEATURE_MINOR_ANON feature is enabled.

Replace the bool wp_async parameter in vma_can_userfault() and userfaultfd_register_range() with an extensible ctx_flags bitmap. Add UFFD_CTX_WP_ASYNC and UFFD_CTX_MINOR_ANON flags, and userfaultfd_ctx_flags() to build the bitmap from ctx->features.

Add userfaultfd_minor_async() helper for checking async minor mode from the fault path.

Gate UFFD_FEATURE_MINOR_ANON and UFFD_FEATURE_MINOR_ASYNC on CONFIG_HAVE_ARCH_USERFAULTFD_MINOR. Validate that MINOR_ASYNC requires at least one minor feature.

Not yet visible to userspace (not in UFFD_API_FEATURES).

Assisted-by Claude:claude-opus-4-6
---
 fs/userfaultfd.c              | 49 ++++++++++++++++++++++++++++++-----
 include/linux/userfaultfd_k.h | 19 +++++++++++---
 mm/userfaultfd.c              |  4 +--
 3 files changed, 59 insertions(+), 13 deletions(-)
fs/userfaultfd.c +42 -7
index bdc84e5219cd..8d508ad19e89 100644
@@ -89,6 +89,27 @@ static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
 	return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
 }
 
+static bool userfaultfd_minor_anon_ctx(struct userfaultfd_ctx *ctx)
+{
+	return ctx && (ctx->features & UFFD_FEATURE_MINOR_ANON);
+}
+
+static bool userfaultfd_minor_async_ctx(struct userfaultfd_ctx *ctx)
+{
+	return ctx && (ctx->features & UFFD_FEATURE_MINOR_ASYNC);
+}
+
+static unsigned int userfaultfd_ctx_flags(struct userfaultfd_ctx *ctx)
+{
+	unsigned int flags = 0;
+
+	if (userfaultfd_wp_async_ctx(ctx))
+		flags |= UFFD_CTX_WP_ASYNC;
+	if (userfaultfd_minor_anon_ctx(ctx))
+		flags |= UFFD_CTX_MINOR_ANON;
+	return flags;
+}
+
 /*
  * Whether WP_UNPOPULATED is enabled on the uffd context.  It is only
  * meaningful when userfaultfd_wp()==true on the vma and when it's
@@ -1271,7 +1292,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
 	bool basic_ioctls;
 	unsigned long start, end;
 	struct vma_iterator vmi;
-	bool wp_async = userfaultfd_wp_async_ctx(ctx);
+	unsigned int ctx_flags = userfaultfd_ctx_flags(ctx);
 
 	user_uffdio_register = (struct uffdio_register __user *) arg;
 
@@ -1345,7 +1366,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
 
 		/* check not compatible vmas */
 		ret = -EINVAL;
-		if (!vma_can_userfault(cur, vm_flags, wp_async))
+		if (!vma_can_userfault(cur, vm_flags, ctx_flags))
 			goto out_unlock;
 
 		/*
@@ -1398,7 +1419,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
 	VM_WARN_ON_ONCE(!found);
 
 	ret = userfaultfd_register_range(ctx, vma, vm_flags, start, end,
-					 wp_async);
+					 ctx_flags);
 
 out_unlock:
 	mmap_write_unlock(mm);
@@ -1443,7 +1464,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
 	unsigned long start, end, vma_end;
 	const void __user *buf = (void __user *)arg;
 	struct vma_iterator vmi;
-	bool wp_async = userfaultfd_wp_async_ctx(ctx);
+	unsigned int ctx_flags = userfaultfd_ctx_flags(ctx);
 
 	ret = -EFAULT;
 	if (copy_from_user(&uffdio_unregister, buf, sizeof(uffdio_unregister)))
@@ -1505,7 +1526,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
 		 * provides for more strict behavior to notice
 		 * unregistration errors.
 		 */
-		if (!vma_can_userfault(cur, cur->vm_flags, wp_async))
+		if (!vma_can_userfault(cur, cur->vm_flags, ctx_flags))
 			goto out_unlock;
 
 		found = true;
@@ -1526,7 +1547,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx,
 			goto skip;
 
 		VM_WARN_ON_ONCE(vma->vm_userfaultfd_ctx.ctx != ctx);
-		VM_WARN_ON_ONCE(!vma_can_userfault(vma, vma->vm_flags, wp_async));
+		VM_WARN_ON_ONCE(!vma_can_userfault(vma, vma->vm_flags, ctx_flags));
 		VM_WARN_ON_ONCE(!(vma->vm_flags & VM_MAYWRITE));
 
 		if (vma->vm_start > start)
@@ -1890,6 +1911,11 @@ bool userfaultfd_wp_async(struct vm_area_struct *vma)
 	return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx);
 }
 
+bool userfaultfd_minor_async(struct vm_area_struct *vma)
+{
+	return userfaultfd_minor_async_ctx(vma->vm_userfaultfd_ctx.ctx);
+}
+
 static inline unsigned int uffd_ctx_features(__u64 user_features)
 {
 	/*
@@ -1993,11 +2019,20 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx,
 	if (features & UFFD_FEATURE_WP_ASYNC)
 		features |= UFFD_FEATURE_WP_UNPOPULATED;
 
+	ret = -EINVAL;
+	/* MINOR_ASYNC requires at least one minor feature */
+	if ((features & UFFD_FEATURE_MINOR_ASYNC) &&
+	    !(features & (UFFD_FEATURE_MINOR_ANON |
+			  UFFD_FEATURE_MINOR_HUGETLBFS |
+			  UFFD_FEATURE_MINOR_SHMEM)))
+		goto err_out;
+
 	/* report all available features and ioctls to userland */
 	uffdio_api.features = UFFD_API_FEATURES;
 #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
 	uffdio_api.features &=
-		~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM);
+		~(UFFD_FEATURE_MINOR_HUGETLBFS | UFFD_FEATURE_MINOR_SHMEM |
+		  UFFD_FEATURE_MINOR_ANON | UFFD_FEATURE_MINOR_ASYNC);
 #endif
 	if (!pgtable_supports_uffd_wp())
 		uffdio_api.features &= ~UFFD_FEATURE_PAGEFAULT_FLAG_WP;
include/linux/userfaultfd_k.h +15 -4
index fd5f42765497..d1d4ed4a08b0 100644
@@ -208,9 +208,13 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
 	return vma->vm_flags & __VM_UFFD_FLAGS;
 }
 
+/* Flags for vma_can_userfault() describing uffd context capabilities */
+#define UFFD_CTX_WP_ASYNC	(1 << 0)
+#define UFFD_CTX_MINOR_ANON	(1 << 1)
+
 static inline bool vma_can_userfault(struct vm_area_struct *vma,
 				     vm_flags_t vm_flags,
-				     bool wp_async)
+				     unsigned int ctx_flags)
 {
 	vm_flags &= __VM_UFFD_FLAGS;
 
@@ -218,14 +222,15 @@ static inline bool vma_can_userfault(struct vm_area_struct *vma,
 		return false;
 
 	if ((vm_flags & VM_UFFD_MINOR) &&
-	    (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma)))
+	    !is_vm_hugetlb_page(vma) && !vma_is_shmem(vma) &&
+	    !(vma_is_anonymous(vma) && (ctx_flags & UFFD_CTX_MINOR_ANON)))
 		return false;
 
 	/*
 	 * If wp async enabled, and WP is the only mode enabled, allow any
 	 * memory type.
 	 */
-	if (wp_async && (vm_flags == VM_UFFD_WP))
+	if ((ctx_flags & UFFD_CTX_WP_ASYNC) && (vm_flags == VM_UFFD_WP))
 		return true;
 
 	/*
@@ -270,6 +275,7 @@ extern void userfaultfd_unmap_complete(struct mm_struct *mm,
 				       struct list_head *uf);
 extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
 extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
+extern bool userfaultfd_minor_async(struct vm_area_struct *vma);
 
 void userfaultfd_reset_ctx(struct vm_area_struct *vma);
 
@@ -283,7 +289,7 @@ int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
 			       struct vm_area_struct *vma,
 			       vm_flags_t vm_flags,
 			       unsigned long start, unsigned long end,
-			       bool wp_async);
+			       unsigned int ctx_flags);
 
 void userfaultfd_release_new(struct userfaultfd_ctx *ctx);
 
@@ -446,6 +452,11 @@ static inline bool userfaultfd_wp_async(struct vm_area_struct *vma)
 	return false;
 }
 
+static inline bool userfaultfd_minor_async(struct vm_area_struct *vma)
+{
+	return false;
+}
+
 static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
 {
 	return false;
mm/userfaultfd.c +2 -2
index 927086bb4a3c..dba1ea26fdfe 100644
@@ -2008,7 +2008,7 @@ int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
 			       struct vm_area_struct *vma,
 			       vm_flags_t vm_flags,
 			       unsigned long start, unsigned long end,
-			       bool wp_async)
+			       unsigned int ctx_flags)
 {
 	VMA_ITERATOR(vmi, ctx->mm, start);
 	struct vm_area_struct *prev = vma_prev(&vmi);
@@ -2021,7 +2021,7 @@ int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
 	for_each_vma_range(vmi, vma, end) {
 		cond_resched();
 
-		VM_WARN_ON_ONCE(!vma_can_userfault(vma, vm_flags, wp_async));
+		VM_WARN_ON_ONCE(!vma_can_userfault(vma, vm_flags, ctx_flags));
 		VM_WARN_ON_ONCE(vma->vm_userfaultfd_ctx.ctx &&
 				vma->vm_userfaultfd_ctx.ctx != ctx);
 		VM_WARN_ON_ONCE(!(vma->vm_flags & VM_MAYWRITE));
Signature · 1 line
2.51.2
author reviewer
#3 lore raw

RFC PATCH 3/12 userfaultfd: implement UFFDIO_DEACTIVATE ioctl

In reply to Kiryl Shutsemau (Meta)

UFFDIO_DEACTIVATE marks pages as deactivated within a VM_UFFD_MINOR range:

- Anonymous memory: set protnone via change_protection(MM_CP_UFFD_DEACTIVATE). Pages stay resident with PFNs preserved, only permissions removed. MM_CP_UFFD_DEACTIVATE is handled independently from MM_CP_PROT_NUMA, bypassing folio_can_map_prot_numa() and CONFIG_NUMA_BALANCING guards.

- Shared shmem/hugetlbfs: zap PTEs via zap_page_range_single(). Pages stay in page cache.

- Private hugetlb: rejected with -EINVAL (zapping would destroy content).

Cleanup on unregister/close: restore protnone PTEs to normal permissions in userfaultfd_clear_vma(), preventing permanently inaccessible pages.

Assisted-by Claude:claude-opus-4-6
---
 fs/userfaultfd.c              | 35 ++++++++++++++++
 include/linux/mm.h            |  2 +
 include/linux/userfaultfd_k.h |  2 +
 mm/huge_memory.c              |  9 ++--
 mm/mprotect.c                 |  9 +++-
 mm/userfaultfd.c              | 78 +++++++++++++++++++++++++++++++++--
 6 files changed, 127 insertions(+), 8 deletions(-)
fs/userfaultfd.c +35 -0
index 8d508ad19e89..b317c9854b86 100644
@@ -1441,6 +1441,10 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx,
 		if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR))
 			ioctls_out &= ~((__u64)1 << _UFFDIO_CONTINUE);
 
+		/* DEACTIVATE is only supported for MINOR ranges. */
+		if (!(uffdio_register.mode & UFFDIO_REGISTER_MODE_MINOR))
+			ioctls_out &= ~((__u64)1 << _UFFDIO_DEACTIVATE);
+
 		/*
 		 * Now that we scanned all vmas we can already tell
 		 * userland which ioctls methods are guaranteed to
@@ -1788,6 +1792,34 @@ static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx,
 	return ret;
 }
 
+static int userfaultfd_deactivate(struct userfaultfd_ctx *ctx,
+				  unsigned long arg)
+{
+	int ret;
+	struct uffdio_range uffdio_range;
+
+	if (atomic_read(&ctx->mmap_changing))
+		return -EAGAIN;
+
+	if (copy_from_user(&uffdio_range, (void __user *)arg,
+			   sizeof(uffdio_range)))
+		return -EFAULT;
+
+	ret = validate_range(ctx->mm, uffdio_range.start, uffdio_range.len);
+	if (ret)
+		return ret;
+
+	if (mmget_not_zero(ctx->mm)) {
+		ret = mdeactivate_range(ctx, uffdio_range.start,
+					uffdio_range.len);
+		mmput(ctx->mm);
+	} else {
+		return -ESRCH;
+	}
+
+	return ret;
+}
+
 static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
 {
 	__s64 ret;
@@ -2108,6 +2140,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd,
 	case UFFDIO_POISON:
 		ret = userfaultfd_poison(ctx, arg);
 		break;
+	case UFFDIO_DEACTIVATE:
+		ret = userfaultfd_deactivate(ctx, arg);
+		break;
 	}
 	return ret;
 }
include/linux/mm.h +2 -0
index abb4963c1f06..fc2841264d56 100644
@@ -3036,6 +3036,8 @@ int get_cmdline(struct task_struct *task, char *buffer, int buflen);
 #define  MM_CP_UFFD_WP_RESOLVE             (1UL << 3) /* Resolve wp */
 #define  MM_CP_UFFD_WP_ALL                 (MM_CP_UFFD_WP | \
 					    MM_CP_UFFD_WP_RESOLVE)
+/* Whether this change is for uffd deactivation */
+#define  MM_CP_UFFD_DEACTIVATE             (1UL << 4)
 
 bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,
 			     pte_t pte);
include/linux/userfaultfd_k.h +2 -0
index d1d4ed4a08b0..c94b5c5b5f24 100644
@@ -130,6 +130,8 @@ extern int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
 			       unsigned long len, bool enable_wp);
 extern long uffd_wp_range(struct vm_area_struct *vma,
 			  unsigned long start, unsigned long len, bool enable_wp);
+extern int mdeactivate_range(struct userfaultfd_ctx *ctx, unsigned long start,
+			     unsigned long len);
 
 /* move_pages */
 void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
mm/huge_memory.c +5 -4
index b298cba853ab..2ad736ff007c 100644
@@ -2563,6 +2563,7 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	spinlock_t *ptl;
 	pmd_t oldpmd, entry;
 	bool prot_numa = cp_flags & MM_CP_PROT_NUMA;
+	bool uffd_deactivate = cp_flags & MM_CP_UFFD_DEACTIVATE;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
 	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
 	int ret = 1;
@@ -2582,8 +2583,11 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		goto unlock;
 	}
 
-	if (prot_numa) {
+	/* Already protnone — nothing to do for either NUMA or uffd */
+	if ((prot_numa || uffd_deactivate) && pmd_protnone(*pmd))
+		goto unlock;
 
+	if (prot_numa) {
 		/*
 		 * Avoid trapping faults against the zero page. The read-only
 		 * data is likely to be read-cached on the local CPU and
@@ -2592,9 +2596,6 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		if (is_huge_zero_pmd(*pmd))
 			goto unlock;
 
-		if (pmd_protnone(*pmd))
-			goto unlock;
-
 		if (!folio_can_map_prot_numa(pmd_folio(*pmd), vma,
 					     vma_is_single_threaded_private(vma)))
 			goto unlock;
mm/mprotect.c +8 -1
index c0571445bef7..7c612a680014 100644
@@ -220,6 +220,7 @@ static long change_pte_range(struct mmu_gather *tlb,
 	long pages = 0;
 	bool is_private_single_threaded;
 	bool prot_numa = cp_flags & MM_CP_PROT_NUMA;
+	bool uffd_deactivate = cp_flags & MM_CP_UFFD_DEACTIVATE;
 	bool uffd_wp = cp_flags & MM_CP_UFFD_WP;
 	bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE;
 	int nr_ptes;
@@ -245,7 +246,8 @@ static long change_pte_range(struct mmu_gather *tlb,
 			pte_t ptent;
 
 			/* Already in the desired state. */
-			if (prot_numa && pte_protnone(oldpte))
+			if ((prot_numa || uffd_deactivate) &&
+			    pte_protnone(oldpte))
 				continue;
 
 			page = vm_normal_page(vma, addr, oldpte);
@@ -255,6 +257,8 @@ static long change_pte_range(struct mmu_gather *tlb,
 			/*
 			 * Avoid trapping faults against the zero or KSM
 			 * pages. See similar comment in change_huge_pmd.
+			 * Skip this filter for uffd deactivation which
+			 * must set protnone regardless of NUMA placement.
 			 */
 			if (prot_numa &&
 			    !folio_can_map_prot_numa(folio, vma,
@@ -651,6 +655,9 @@ long change_protection(struct mmu_gather *tlb,
 	WARN_ON_ONCE(cp_flags & MM_CP_PROT_NUMA);
 #endif
 
+	if (cp_flags & MM_CP_UFFD_DEACTIVATE)
+		newprot = PAGE_NONE;
+
 	if (is_vm_hugetlb_page(vma))
 		pages = hugetlb_change_protection(vma, start, end, newprot,
 						  cp_flags);
mm/userfaultfd.c +75 -3

107 lines not on this page · raw message

Signature · 1 line
2.51.2
author reviewer
#4 lore raw

RFC PATCH 4/12 userfaultfd: UFFDIO_CONTINUE for anonymous memory

In reply to Kiryl Shutsemau (Meta)

Allow UFFDIO_CONTINUE on anonymous VMAs with VM_UFFD_MINOR. For shmem, CONTINUE installs a PTE from page cache. For anonymous memory, the page is already mapped via a protnone PTE — CONTINUE restores the original VMA permissions.

PTE level: mfill_atomic_pte_continue_anon() walks to the PTE, verifies protnone, restores permissions. Rename the shmem path to mfill_atomic_pte_continue_shmem() for clarity.

PMD/THP level: mfill_atomic_pmd_continue_anon() restores protnone PMD permissions in place without splitting. Handles PMD races with EAGAIN retry in the mfill_atomic loop.

Add protnone PTE/PMD checks in userfaultfd_must_wait() so sync minor faults properly block until resolved.

Assisted-by Claude:claude-opus-4-6
---
 fs/userfaultfd.c |  9 +++++-
 mm/userfaultfd.c | 82 ++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 84 insertions(+), 7 deletions(-)
fs/userfaultfd.c +8 -1
index b317c9854b86..43064238fd8d 100644
@@ -340,8 +340,11 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
 	if (!pmd_present(_pmd))
 		return false;
 
-	if (pmd_trans_huge(_pmd))
+	if (pmd_trans_huge(_pmd)) {
+		if (pmd_protnone(_pmd) && (reason & VM_UFFD_MINOR))
+			return true;
 		return !pmd_write(_pmd) && (reason & VM_UFFD_WP);
+	}
 
 	pte = pte_offset_map(pmd, address);
 	if (!pte)
@@ -366,6 +369,9 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,
 	 */
 	if (!pte_write(ptent) && (reason & VM_UFFD_WP))
 		goto out;
+	/* PTE is still protnone (deactivated), wait for userspace to resolve. */
+	if (pte_protnone(ptent) && (reason & VM_UFFD_MINOR))
+		goto out;
 
 	ret = false;
 out:
@@ -1820,6 +1826,7 @@ static int userfaultfd_deactivate(struct userfaultfd_ctx *ctx,
 	return ret;
 }
 
+
 static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
 {
 	__s64 ret;
mm/userfaultfd.c +76 -6

102 lines not on this page · raw message

Signature · 1 line
2.51.2
author reviewer
#5 lore raw

RFC PATCH 5/12 mm: intercept protnone faults on VM_UFFD_MINOR anonymous VMAs

In reply to Kiryl Shutsemau (Meta)

When a protnone PTE/PMD fault occurs on a VMA with VM_UFFD_MINOR, dispatch to the userfaultfd minor fault path instead of NUMA balancing.
Async: restore permissions inline. Sync: deliver via handle_userfault().

Feed NUMA locality stats from the fault path via task_numa_fault() so the scheduler retains placement data even though NUMA scanning is skipped on these VMAs.

Assisted-by Claude:claude-opus-4-6
---
 include/linux/huge_mm.h |  6 +++++
 mm/huge_memory.c        | 24 +++++++++++++++++++
 mm/memory.c             | 51 +++++++++++++++++++++++++++++++++++++++--
 3 files changed, 79 insertions(+), 2 deletions(-)
include/linux/huge_mm.h +6 -0
index a4d9f964dfde..a900bb530998 100644
@@ -519,6 +519,7 @@ static inline bool folio_test_pmd_mappable(struct folio *folio)
 }
 
 vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf);
+vm_fault_t do_huge_pmd_uffd_minor(struct vm_fault *vmf);
 
 vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf);
 
@@ -707,6 +708,11 @@ static inline vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	return 0;
 }
 
+static inline vm_fault_t do_huge_pmd_uffd_minor(struct vm_fault *vmf)
+{
+	return 0;
+}
+
 static inline vm_fault_t do_huge_pmd_device_private(struct vm_fault *vmf)
 {
 	return 0;
mm/huge_memory.c +24 -0
index 2ad736ff007c..264c646a8573 100644
@@ -2181,6 +2181,30 @@ static inline bool can_change_pmd_writable(struct vm_area_struct *vma,
 	return pmd_dirty(pmd);
 }
 
+vm_fault_t do_huge_pmd_uffd_minor(struct vm_fault *vmf)
+{
+	struct vm_area_struct *vma = vmf->vma;
+
+	if (userfaultfd_minor_async(vma)) {
+		pmd_t pmd;
+
+		vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
+		if (unlikely(!pmd_same(pmdp_get(vmf->pmd), vmf->orig_pmd))) {
+			spin_unlock(vmf->ptl);
+			return 0;
+		}
+		pmd = pmd_modify(vmf->orig_pmd, vma->vm_page_prot);
+		pmd = pmd_mkyoung(pmd);
+		set_pmd_at(vma->vm_mm, vmf->address & HPAGE_PMD_MASK,
+			   vmf->pmd, pmd);
+		update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
+		spin_unlock(vmf->ptl);
+		return 0;
+	}
+
+	return handle_userfault(vmf, VM_UFFD_MINOR);
+}
+
 /* NUMA hinting page fault entry point for trans huge pmds */
 vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 {
mm/memory.c +49 -2

71 lines not on this page · raw message

Signature · 1 line
2.51.2
author reviewer
#6 lore raw

RFC PATCH 6/12 userfaultfd: auto-resolve shmem and hugetlbfs minor faults in async mode

In reply to Kiryl Shutsemau (Meta)

When UFFD_FEATURE_MINOR_ASYNC is enabled, skip handle_userfault() in the shmem and hugetlbfs minor fault paths. The normal fault path installs the PTE from page cache directly.

Assisted-by Claude:claude-opus-4-6
---
 mm/hugetlb.c | 3 ++-
 mm/shmem.c   | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)
mm/hugetlb.c +2 -1
index 327eaa4074d3..c10d2432768c 100644
@@ -5847,7 +5847,8 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping,
 		}
 
 		/* Check for page in userfault range. */
-		if (userfaultfd_minor(vma)) {
+		if (userfaultfd_minor(vma) &&
+		    !userfaultfd_minor_async(vma)) {
 			folio_unlock(folio);
 			folio_put(folio);
 			/* See comment in userfaultfd_missing() block above */
mm/shmem.c +2 -1
index b40f3cd48961..ce47e77fc090 100644
@@ -2489,7 +2489,8 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
 	fault_mm = vma ? vma->vm_mm : NULL;
 
 	folio = filemap_get_entry(inode->i_mapping, index);
-	if (folio && vma && userfaultfd_minor(vma)) {
+	if (folio && vma && userfaultfd_minor(vma) &&
+	    !userfaultfd_minor_async(vma)) {
 		if (!xa_is_value(folio))
 			folio_put(folio);
 		*fault_type = handle_userfault(vmf, VM_UFFD_MINOR);
Signature · 1 line
2.51.2
author reviewer
#7 lore raw

RFC PATCH 7/12 sched/numa: skip scanning anonymous VM_UFFD_MINOR VMAs

In reply to Kiryl Shutsemau (Meta)

Avoid protnone conflict on anonymous VMAs. Shmem unaffected. NUMA stats fed from uffd fault path instead.
Add NUMAB_SKIP_UFFD_MINOR trace reason.

Assisted-by Claude:claude-opus-4-6
---
 include/linux/sched/numa_balancing.h |  1 +
 include/trace/events/sched.h         |  3 ++-
 kernel/sched/fair.c                  | 13 +++++++++++++
 3 files changed, 16 insertions(+), 1 deletion(-)
include/linux/sched/numa_balancing.h +1 -0
index 52b22c5c396d..5668074a4271 100644
@@ -23,6 +23,7 @@ enum numa_vmaskip_reason {
 	NUMAB_SKIP_PID_INACTIVE,
 	NUMAB_SKIP_IGNORE_PID,
 	NUMAB_SKIP_SEQ_COMPLETED,
+	NUMAB_SKIP_UFFD_MINOR,
 };
 
 #ifdef CONFIG_NUMA_BALANCING
include/trace/events/sched.h +2 -1
index 7b2645b50e78..02e79b56db28 100644
@@ -728,7 +728,8 @@ DEFINE_EVENT(sched_numa_pair_template, sched_swap_numa,
 	EM( NUMAB_SKIP_SCAN_DELAY,		"scan_delay" )	\
 	EM( NUMAB_SKIP_PID_INACTIVE,		"pid_inactive" )	\
 	EM( NUMAB_SKIP_IGNORE_PID,		"ignore_pid_inactive" )		\
-	EMe(NUMAB_SKIP_SEQ_COMPLETED,		"seq_completed" )
+	EM( NUMAB_SKIP_SEQ_COMPLETED,		"seq_completed" )	\
+	EMe(NUMAB_SKIP_UFFD_MINOR,		"uffd_minor" )
 
 /* Redefine for export. */
 #undef EM
kernel/sched/fair.c +13 -0
index ab4114712be7..57beb04562cf 100644
@@ -25,6 +25,7 @@
 #include <linux/hugetlb_inline.h>
 #include <linux/jiffies.h>
 #include <linux/mm_api.h>
+#include <linux/userfaultfd_k.h>
 #include <linux/highmem.h>
 #include <linux/spinlock_api.h>
 #include <linux/cpumask_api.h>
@@ -3459,6 +3460,18 @@ static void task_numa_work(struct callback_head *work)
 			continue;
 		}
 
+		/*
+		 * Skip anonymous VMAs registered for userfaultfd minor faults.
+		 * Both NUMA balancing and uffd use protnone PTEs on anonymous
+		 * memory — let uffd own the hinting. For shmem, UFFDIO_DEACTIVATE
+		 * zaps PTEs entirely (no protnone conflict), so NUMA scanning
+		 * can proceed normally.
+		 */
+		if (vma_is_anonymous(vma) && userfaultfd_minor(vma)) {
+			trace_sched_skip_vma_numa(mm, vma, NUMAB_SKIP_UFFD_MINOR);
+			continue;
+		}
+
 		/*
 		 * Shared library pages mapped by multiple processes are not
 		 * migrated as it is expected they are cache replicated. Avoid
Signature · 1 line
2.51.2
author reviewer
#8 lore raw

RFC PATCH 8/12 userfaultfd: enable UFFD_FEATURE_MINOR_ANON

In reply to Kiryl Shutsemau (Meta)

Add UFFD_FEATURE_MINOR_ANON, UFFD_FEATURE_MINOR_ASYNC to UFFD_API_FEATURES and UFFDIO_DEACTIVATE to UFFD_API_RANGE_IOCTLS. The feature is now available to userspace.

Assisted-by Claude:claude-opus-4-6
---
 include/uapi/linux/userfaultfd.h | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)
include/uapi/linux/userfaultfd.h +7 -3
index 336d07e1b6de..775825da2596 100644
@@ -42,7 +42,9 @@
 			   UFFD_FEATURE_WP_UNPOPULATED |	\
 			   UFFD_FEATURE_POISON |		\
 			   UFFD_FEATURE_WP_ASYNC |		\
-			   UFFD_FEATURE_MOVE)
+			   UFFD_FEATURE_MOVE |			\
+			   UFFD_FEATURE_MINOR_ANON |		\
+			   UFFD_FEATURE_MINOR_ASYNC)
 #define UFFD_API_IOCTLS				\
 	((__u64)1 << _UFFDIO_REGISTER |		\
 	 (__u64)1 << _UFFDIO_UNREGISTER |	\
@@ -54,13 +56,15 @@
 	 (__u64)1 << _UFFDIO_MOVE |		\
 	 (__u64)1 << _UFFDIO_WRITEPROTECT |	\
 	 (__u64)1 << _UFFDIO_CONTINUE |		\
-	 (__u64)1 << _UFFDIO_POISON)
+	 (__u64)1 << _UFFDIO_POISON |		\
+	 (__u64)1 << _UFFDIO_DEACTIVATE)
 #define UFFD_API_RANGE_IOCTLS_BASIC		\
 	((__u64)1 << _UFFDIO_WAKE |		\
 	 (__u64)1 << _UFFDIO_COPY |		\
 	 (__u64)1 << _UFFDIO_WRITEPROTECT |	\
 	 (__u64)1 << _UFFDIO_CONTINUE |		\
-	 (__u64)1 << _UFFDIO_POISON)
+	 (__u64)1 << _UFFDIO_POISON |		\
+	 (__u64)1 << _UFFDIO_DEACTIVATE)
 
 /*
  * Valid ioctl command number range with this API is from 0x00 to
Signature · 1 line
2.51.2
author reviewer
#9 lore raw

RFC PATCH 9/12 mm/pagemap: add PAGE_IS_UFFD_DEACTIVATED to PAGEMAP_SCAN

In reply to Kiryl Shutsemau (Meta)

Report deactivated anonymous pages in PAGEMAP_SCAN results. Only set on anonymous VMAs (shmem cold = !PAGE_IS_PRESENT). Both PTE and PMD (THP) levels handled.

Assisted-by Claude:claude-opus-4-6
---
 fs/proc/task_mmu.c      | 11 ++++++++++-
 include/uapi/linux/fs.h |  1 +
 2 files changed, 11 insertions(+), 1 deletion(-)
fs/proc/task_mmu.c +10 -1
index e091931d7ca1..fc42cfd5720a 100644
@@ -2329,7 +2329,7 @@ static int pagemap_release(struct inode *inode, struct file *file)
 				 PAGE_IS_FILE |	PAGE_IS_PRESENT |	\
 				 PAGE_IS_SWAPPED | PAGE_IS_PFNZERO |	\
 				 PAGE_IS_HUGE | PAGE_IS_SOFT_DIRTY |	\
-				 PAGE_IS_GUARD)
+				 PAGE_IS_GUARD | PAGE_IS_UFFD_DEACTIVATED)
 #define PM_SCAN_FLAGS		(PM_SCAN_WP_MATCHING | PM_SCAN_CHECK_WPASYNC)
 
 struct pagemap_scan_private {
@@ -2354,6 +2354,10 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p,
 
 		categories = PAGE_IS_PRESENT;
 
+		if (pte_protnone(pte) && vma_is_accessible(vma) &&
+		    vma_is_anonymous(vma) && userfaultfd_minor(vma))
+			categories |= PAGE_IS_UFFD_DEACTIVATED;
+
 		if (!pte_uffd_wp(pte))
 			categories |= PAGE_IS_WRITTEN;
 
@@ -2422,6 +2426,11 @@ static unsigned long pagemap_thp_category(struct pagemap_scan_private *p,
 		struct page *page;
 
 		categories |= PAGE_IS_PRESENT;
+
+		if (pmd_protnone(pmd) && vma_is_accessible(vma) &&
+		    vma_is_anonymous(vma) && userfaultfd_minor(vma))
+			categories |= PAGE_IS_UFFD_DEACTIVATED;
+
 		if (!pmd_uffd_wp(pmd))
 			categories |= PAGE_IS_WRITTEN;
 
include/uapi/linux/fs.h +1 -0
index 70b2b661f42c..af5b28901800 100644
@@ -455,6 +455,7 @@ typedef int __bitwise __kernel_rwf_t;
 #define PAGE_IS_HUGE		(1 << 6)
 #define PAGE_IS_SOFT_DIRTY	(1 << 7)
 #define PAGE_IS_GUARD		(1 << 8)
+#define PAGE_IS_UFFD_DEACTIVATED (1 << 9)
 
 /*
  * struct page_region - Page region with flags
Signature · 1 line
2.51.2
author reviewer
#10 lore raw

RFC PATCH 10/12 userfaultfd: add UFFDIO_SET_MODE for runtime sync/async toggle

In reply to Kiryl Shutsemau (Meta)

Add UFFDIO_SET_MODE ioctl to toggle UFFD_FEATURE_MINOR_ASYNC at runtime. Takes mmap_write_lock for serialization against all in-flight faults. On sync-to-async transition, wake threads blocked in handle_userfault() so they retry and auto-resolve.

Since ctx->features can now be modified concurrently, add userfaultfd_features() helper that wraps READ_ONCE() and convert all ctx->features reads to use it.

Assisted-by Claude:claude-opus-4-6
---
 fs/userfaultfd.c                 | 95 ++++++++++++++++++++++++++++----
 include/uapi/linux/userfaultfd.h | 13 +++++
 2 files changed, 96 insertions(+), 12 deletions(-)
fs/userfaultfd.c +83 -12
index 43064238fd8d..0edb33599491 100644
@@ -79,24 +79,33 @@ struct userfaultfd_wake_range {
 /* internal indication that UFFD_API ioctl was successfully executed */
 #define UFFD_FEATURE_INITIALIZED		(1u << 31)
 
+/*
+ * Read ctx->features with READ_ONCE() since UFFDIO_SET_MODE can
+ * modify it concurrently.
+ */
+static unsigned int userfaultfd_features(struct userfaultfd_ctx *ctx)
+{
+	return READ_ONCE(ctx->features);
+}
+
 static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
 {
-	return ctx->features & UFFD_FEATURE_INITIALIZED;
+	return userfaultfd_features(ctx) & UFFD_FEATURE_INITIALIZED;
 }
 
 static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
 {
-	return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
+	return ctx && (userfaultfd_features(ctx) & UFFD_FEATURE_WP_ASYNC);
 }
 
 static bool userfaultfd_minor_anon_ctx(struct userfaultfd_ctx *ctx)
 {
-	return ctx && (ctx->features & UFFD_FEATURE_MINOR_ANON);
+	return ctx && (userfaultfd_features(ctx) & UFFD_FEATURE_MINOR_ANON);
 }
 
 static bool userfaultfd_minor_async_ctx(struct userfaultfd_ctx *ctx)
 {
-	return ctx && (ctx->features & UFFD_FEATURE_MINOR_ASYNC);
+	return ctx && (userfaultfd_features(ctx) & UFFD_FEATURE_MINOR_ASYNC);
 }
 
 static unsigned int userfaultfd_ctx_flags(struct userfaultfd_ctx *ctx)
@@ -122,7 +131,7 @@ bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
 	if (!ctx)
 		return false;
 
-	return ctx->features & UFFD_FEATURE_WP_UNPOPULATED;
+	return userfaultfd_features(ctx) & UFFD_FEATURE_WP_UNPOPULATED;
 }
 
 static int userfaultfd_wake_function(wait_queue_entry_t *wq, unsigned mode,
@@ -435,7 +444,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)
 	/* 0 or > 1 flags set is a bug; we expect exactly 1. */
 	VM_WARN_ON_ONCE(!reason || (reason & (reason - 1)));
 
-	if (ctx->features & UFFD_FEATURE_SIGBUS)
+	if (userfaultfd_features(ctx) & UFFD_FEATURE_SIGBUS)
 		goto out;
 	if (!(vmf->flags & FAULT_FLAG_USER) && (ctx->flags & UFFD_USER_MODE_ONLY))
 		goto out;
@@ -506,7 +515,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)
 	init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
 	uwq.wq.private = current;
 	uwq.msg = userfault_msg(vmf->address, vmf->real_address, vmf->flags,
-				reason, ctx->features);
+				reason, userfaultfd_features(ctx));
 	uwq.ctx = ctx;
 	uwq.waken = false;
 
@@ -668,7 +677,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs)
 	if (!octx)
 		return 0;
 
-	if (!(octx->features & UFFD_FEATURE_EVENT_FORK)) {
+	if (!(userfaultfd_features(octx) & UFFD_FEATURE_EVENT_FORK)) {
 		userfaultfd_reset_ctx(vma);
 		return 0;
 	}
@@ -774,7 +783,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma,
 	if (!ctx)
 		return;
 
-	if (ctx->features & UFFD_FEATURE_EVENT_REMAP) {
+	if (userfaultfd_features(ctx) & UFFD_FEATURE_EVENT_REMAP) {
 		vm_ctx->ctx = ctx;
 		userfaultfd_ctx_get(ctx);
 		down_write(&ctx->map_changing_lock);
@@ -824,7 +833,7 @@ bool userfaultfd_remove(struct vm_area_struct *vma,
 	struct userfaultfd_wait_queue ewq;
 
 	ctx = vma->vm_userfaultfd_ctx.ctx;
-	if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_REMOVE))
+	if (!ctx || !(userfaultfd_features(ctx) & UFFD_FEATURE_EVENT_REMOVE))
 		return true;
 
 	userfaultfd_ctx_get(ctx);
@@ -863,7 +872,7 @@ int userfaultfd_unmap_prep(struct vm_area_struct *vma, unsigned long start,
 	struct userfaultfd_unmap_ctx *unmap_ctx;
 	struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
 
-	if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_UNMAP) ||
+	if (!ctx || !(userfaultfd_features(ctx) & UFFD_FEATURE_EVENT_UNMAP) ||
 	    has_unmap_ctx(ctx, unmaps, start, end))
 		return 0;
 
@@ -1826,6 +1835,65 @@ static int userfaultfd_deactivate(struct userfaultfd_ctx *ctx,
 	return ret;
 }
 
+/*
+ * Features that can be toggled at runtime via UFFDIO_SET_MODE.
+ * Only async features that were enabled at UFFDIO_API time may be toggled.
+ */
+#define UFFD_FEATURE_TOGGLEABLE	(UFFD_FEATURE_MINOR_ASYNC)
+
+static int userfaultfd_set_mode(struct userfaultfd_ctx *ctx,
+				  unsigned long arg)
+{
+	struct uffdio_set_mode mode;
+	struct mm_struct *mm = ctx->mm;
+
+	if (copy_from_user(&mode, (void __user *)arg, sizeof(mode)))
+		return -EFAULT;
+
+	/* enable and disable must not overlap */
+	if (mode.enable & mode.disable)
+		return -EINVAL;
+
+	/* only toggleable features are allowed */
+	if ((mode.enable | mode.disable) & ~UFFD_FEATURE_TOGGLEABLE)
+		return -EINVAL;
+
+	if (!mmget_not_zero(mm))
+		return -ESRCH;
+
+	/*
+	 * mmap_write_lock serializes against all page faults.
+	 * After we release, no in-flight faults from the old mode exist.
+	 */
+	{
+		unsigned int new_features;
+
+		mmap_write_lock(mm);
+		new_features = userfaultfd_features(ctx);
+		new_features |= mode.enable;
+		new_features &= ~mode.disable;
+		WRITE_ONCE(ctx->features, new_features);
+		mmap_write_unlock(mm);
+	}
+
+	/*
+	 * If switching to async, wake threads blocked in handle_userfault().
+	 * They will retry the fault and auto-resolve under the new mode.
+	 * len=0 means wake all pending faults on this context.
+	 */
+	if (mode.enable & UFFD_FEATURE_MINOR_ASYNC) {
+		struct userfaultfd_wake_range range = { .len = 0 };
+
+		spin_lock_irq(&ctx->fault_pending_wqh.lock);
+		__wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL,
+				     &range);
+		__wake_up(&ctx->fault_wqh, TASK_NORMAL, 1, &range);
+		spin_unlock_irq(&ctx->fault_pending_wqh.lock);
+	}
+
+	mmput(mm);
+	return 0;
+}
 
 static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
 {
@@ -2150,6 +2218,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd,
 	case UFFDIO_DEACTIVATE:
 		ret = userfaultfd_deactivate(ctx, arg);
 		break;
+	case UFFDIO_SET_MODE:
+		ret = userfaultfd_set_mode(ctx, arg);
+		break;
 	}
 	return ret;
 }
@@ -2177,7 +2248,7 @@ static void userfaultfd_show_fdinfo(struct seq_file *m, struct file *f)
 	 *	protocols: aa:... bb:...
 	 */
 	seq_printf(m, "pending:\t%lu\ntotal:\t%lu\nAPI:\t%Lx:%x:%Lx\n",
-		   pending, total, UFFD_API, ctx->features,
+		   pending, total, UFFD_API, userfaultfd_features(ctx),
 		   UFFD_API_IOCTLS|UFFD_API_RANGE_IOCTLS);
 }
 #endif
include/uapi/linux/userfaultfd.h +13 -0
index 775825da2596..f0f14f9db06c 100644
@@ -84,6 +84,7 @@
 #define _UFFDIO_CONTINUE		(0x07)
 #define _UFFDIO_POISON			(0x08)
 #define _UFFDIO_DEACTIVATE		(0x09)
+#define _UFFDIO_SET_MODE		(0x0A)
 #define _UFFDIO_API			(0x3F)
 
 /* userfaultfd ioctl ids */
@@ -110,6 +111,8 @@
 				      struct uffdio_poison)
 #define UFFDIO_DEACTIVATE	_IOR(UFFDIO, _UFFDIO_DEACTIVATE,	\
 				     struct uffdio_range)
+#define UFFDIO_SET_MODE		_IOW(UFFDIO, _UFFDIO_SET_MODE,	\
+				     struct uffdio_set_mode)
 
 /* read() structure */
 struct uffd_msg {
@@ -395,6 +398,16 @@ struct uffdio_move {
 	__s64 move;
 };
 
+struct uffdio_set_mode {
+	/*
+	 * Toggle async mode for features at runtime.
+	 * Supported: UFFD_FEATURE_MINOR_ASYNC.
+	 * Setting a bit in both enable and disable is invalid.
+	 */
+	__u64 enable;
+	__u64 disable;
+};
+
 /*
  * Flags for the userfaultfd(2) system call itself.
  */
Signature · 1 line
2.51.2
reviewer
#11 lore raw

Re: [RFC, PATCH 10/12] userfaultfd: add UFFDIO_SET_MODE for runtime sync/async toggle

Quoted 137 lines · On Tue, 14 Apr 2026 15:23:44 +0100 "Kiryl Shutsemau (Meta)" <[email protected]>…

Add UFFDIO_SET_MODE ioctl to toggle UFFD_FEATURE_MINOR_ASYNC at runtime. Takes mmap_write_lock for serialization against all in-flight faults. On sync-to-async transition, wake threads blocked in handle_userfault() so they retry and auto-resolve.

Since ctx->features can now be modified concurrently, add userfaultfd_features() helper that wraps READ_ONCE() and convert all ctx->features reads to use it.

Assisted-by Claude:claude-opus-4-6
---
 fs/userfaultfd.c                 | 95 ++++++++++++++++++++++++++++----
 include/uapi/linux/userfaultfd.h | 13 +++++
 2 files changed, 96 insertions(+), 12 deletions(-)
fs/userfaultfd.c +32 -11
index 43064238fd8d..0edb33599491 100644
@@ -79,24 +79,33 @@ struct userfaultfd_wake_range {
 /* internal indication that UFFD_API ioctl was successfully executed */
 #define UFFD_FEATURE_INITIALIZED		(1u << 31)
 
+/*
+ * Read ctx->features with READ_ONCE() since UFFDIO_SET_MODE can
+ * modify it concurrently.
+ */
+static unsigned int userfaultfd_features(struct userfaultfd_ctx *ctx)
+{
+	return READ_ONCE(ctx->features);
+}
+
 static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
 {
-	return ctx->features & UFFD_FEATURE_INITIALIZED;
+	return userfaultfd_features(ctx) & UFFD_FEATURE_INITIALIZED;
 }
 
 static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx)
 {
-	return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC);
+	return ctx && (userfaultfd_features(ctx) & UFFD_FEATURE_WP_ASYNC);
 }
 
 static bool userfaultfd_minor_anon_ctx(struct userfaultfd_ctx *ctx)
 {
-	return ctx && (ctx->features & UFFD_FEATURE_MINOR_ANON);
+	return ctx && (userfaultfd_features(ctx) & UFFD_FEATURE_MINOR_ANON);
 }
 
 static bool userfaultfd_minor_async_ctx(struct userfaultfd_ctx *ctx)
 {
-	return ctx && (ctx->features & UFFD_FEATURE_MINOR_ASYNC);
+	return ctx && (userfaultfd_features(ctx) & UFFD_FEATURE_MINOR_ASYNC);
 }
 
 static unsigned int userfaultfd_ctx_flags(struct userfaultfd_ctx *ctx)
@@ -122,7 +131,7 @@ bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma)
 	if (!ctx)
 		return false;
 
-	return ctx->features & UFFD_FEATURE_WP_UNPOPULATED;
+	return userfaultfd_features(ctx) & UFFD_FEATURE_WP_UNPOPULATED;
 }
 
 static int userfaultfd_wake_function(wait_queue_entry_t *wq, unsigned mode,
@@ -435,7 +444,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)
 	/* 0 or > 1 flags set is a bug; we expect exactly 1. */
 	VM_WARN_ON_ONCE(!reason || (reason & (reason - 1)));
 
-	if (ctx->features & UFFD_FEATURE_SIGBUS)
+	if (userfaultfd_features(ctx) & UFFD_FEATURE_SIGBUS)
 		goto out;
 	if (!(vmf->flags & FAULT_FLAG_USER) && (ctx->flags & UFFD_USER_MODE_ONLY))
 		goto out;
@@ -506,7 +515,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason)
 	init_waitqueue_func_entry(&uwq.wq, userfaultfd_wake_function);
 	uwq.wq.private = current;
 	uwq.msg = userfault_msg(vmf->address, vmf->real_address, vmf->flags,
-				reason, ctx->features);
+				reason, userfaultfd_features(ctx));
 	uwq.ctx = ctx;
 	uwq.waken = false;
 
@@ -668,7 +677,7 @@ int dup_userfaultfd(struct vm_area_struct *vma, struct list_head *fcs)
 	if (!octx)
 		return 0;
 
-	if (!(octx->features & UFFD_FEATURE_EVENT_FORK)) {
+	if (!(userfaultfd_features(octx) & UFFD_FEATURE_EVENT_FORK)) {
 		userfaultfd_reset_ctx(vma);
 		return 0;
 	}
@@ -774,7 +783,7 @@ void mremap_userfaultfd_prep(struct vm_area_struct *vma,
 	if (!ctx)
 		return;
 
-	if (ctx->features & UFFD_FEATURE_EVENT_REMAP) {
+	if (userfaultfd_features(ctx) & UFFD_FEATURE_EVENT_REMAP) {
 		vm_ctx->ctx = ctx;
 		userfaultfd_ctx_get(ctx);
 		down_write(&ctx->map_changing_lock);
@@ -824,7 +833,7 @@ bool userfaultfd_remove(struct vm_area_struct *vma,
 	struct userfaultfd_wait_queue ewq;
 
 	ctx = vma->vm_userfaultfd_ctx.ctx;
-	if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_REMOVE))
+	if (!ctx || !(userfaultfd_features(ctx) & UFFD_FEATURE_EVENT_REMOVE))
 		return true;
 
 	userfaultfd_ctx_get(ctx);
@@ -863,7 +872,7 @@ int userfaultfd_unmap_prep(struct vm_area_struct *vma, unsigned long start,
 	struct userfaultfd_unmap_ctx *unmap_ctx;
 	struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx;
 
-	if (!ctx || !(ctx->features & UFFD_FEATURE_EVENT_UNMAP) ||
+	if (!ctx || !(userfaultfd_features(ctx) & UFFD_FEATURE_EVENT_UNMAP) ||
 	    has_unmap_ctx(ctx, unmaps, start, end))
 		return 0;
 
@@ -1826,6 +1835,65 @@ static int userfaultfd_deactivate(struct userfaultfd_ctx *ctx,
 	return ret;
 }
 
+/*
+ * Features that can be toggled at runtime via UFFDIO_SET_MODE.
+ * Only async features that were enabled at UFFDIO_API time may be toggled.
+ */
+#define UFFD_FEATURE_TOGGLEABLE	(UFFD_FEATURE_MINOR_ASYNC)
+
+static int userfaultfd_set_mode(struct userfaultfd_ctx *ctx,
+				  unsigned long arg)
+{
+	struct uffdio_set_mode mode;
+	struct mm_struct *mm = ctx->mm;
+

fs/userfaultfd.c static int userfaultfd_deactivate(struct userfaultfd_ctx *ctx, · line 1859

Quoted 10 lines
fs/userfaultfd.c +10 -0
+	if (copy_from_user(&mode, (void __user *)arg, sizeof(mode)))
+		return -EFAULT;
+
+	/* enable and disable must not overlap */
+	if (mode.enable & mode.disable)
+		return -EINVAL;
+
+	/* only toggleable features are allowed */
+	if ((mode.enable | mode.disable) & ~UFFD_FEATURE_TOGGLEABLE)
+		return -EINVAL;

The commit message states "Only async features that were enabled at UFFDIO_API time may be toggled." However, the code only checks that the requested feature is in UFFD_FEATURE_TOGGLEABLE.

Is it intentional that a user who opened a uffd without UFFD_FEATURE_MINOR_ASYNC can still enable it later via UFFDIO_SET_MODE?

Quoted 101 lines
fs/userfaultfd.c +41 -1
+
+	if (!mmget_not_zero(mm))
+		return -ESRCH;
+
+	/*
+	 * mmap_write_lock serializes against all page faults.
+	 * After we release, no in-flight faults from the old mode exist.
+	 */
+	{
+		unsigned int new_features;
+
+		mmap_write_lock(mm);
+		new_features = userfaultfd_features(ctx);
+		new_features |= mode.enable;
+		new_features &= ~mode.disable;
+		WRITE_ONCE(ctx->features, new_features);
+		mmap_write_unlock(mm);
+	}
+
+	/*
+	 * If switching to async, wake threads blocked in handle_userfault().
+	 * They will retry the fault and auto-resolve under the new mode.
+	 * len=0 means wake all pending faults on this context.
+	 */
+	if (mode.enable & UFFD_FEATURE_MINOR_ASYNC) {
+		struct userfaultfd_wake_range range = { .len = 0 };
+
+		spin_lock_irq(&ctx->fault_pending_wqh.lock);
+		__wake_up_locked_key(&ctx->fault_pending_wqh, TASK_NORMAL,
+				     &range);
+		__wake_up(&ctx->fault_wqh, TASK_NORMAL, 1, &range);
+		spin_unlock_irq(&ctx->fault_pending_wqh.lock);
+	}
+
+	mmput(mm);
+	return 0;
+}
 
 static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg)
 {
@@ -2150,6 +2218,9 @@ static long userfaultfd_ioctl(struct file *file, unsigned cmd,
 	case UFFDIO_DEACTIVATE:
 		ret = userfaultfd_deactivate(ctx, arg);
 		break;
+	case UFFDIO_SET_MODE:
+		ret = userfaultfd_set_mode(ctx, arg);
+		break;
 	}
 	return ret;
 }
@@ -2177,7 +2248,7 @@ static void userfaultfd_show_fdinfo(struct seq_file *m, struct file *f)
 	 *	protocols: aa:... bb:...
 	 */
 	seq_printf(m, "pending:\t%lu\ntotal:\t%lu\nAPI:\t%Lx:%x:%Lx\n",
-		   pending, total, UFFD_API, ctx->features,
+		   pending, total, UFFD_API, userfaultfd_features(ctx),
 		   UFFD_API_IOCTLS|UFFD_API_RANGE_IOCTLS);
 }
 #endif
include/uapi/linux/userfaultfd.h +13 -0
index 775825da2596..f0f14f9db06c 100644
@@ -84,6 +84,7 @@
 #define _UFFDIO_CONTINUE		(0x07)
 #define _UFFDIO_POISON			(0x08)
 #define _UFFDIO_DEACTIVATE		(0x09)
+#define _UFFDIO_SET_MODE		(0x0A)
 #define _UFFDIO_API			(0x3F)
 
 /* userfaultfd ioctl ids */
@@ -110,6 +111,8 @@
 				      struct uffdio_poison)
 #define UFFDIO_DEACTIVATE	_IOR(UFFDIO, _UFFDIO_DEACTIVATE,	\
 				     struct uffdio_range)
+#define UFFDIO_SET_MODE		_IOW(UFFDIO, _UFFDIO_SET_MODE,	\
+				     struct uffdio_set_mode)
 
 /* read() structure */
 struct uffd_msg {
@@ -395,6 +398,16 @@ struct uffdio_move {
 	__s64 move;
 };
 
+struct uffdio_set_mode {
+	/*
+	 * Toggle async mode for features at runtime.
+	 * Supported: UFFD_FEATURE_MINOR_ASYNC.
+	 * Setting a bit in both enable and disable is invalid.
+	 */
+	__u64 enable;
+	__u64 disable;
+};
+
 /*
  * Flags for the userfaultfd(2) system call itself.
  */
Signature · 1 line
2.51.2
author reviewer
#12 lore raw

Re: [RFC, PATCH 10/12] userfaultfd: add UFFDIO_SET_MODE for runtime sync/async toggle

Quoted 11 lines · Usama Arif

fs/userfaultfd.c static int userfaultfd_deactivate(struct userfaultfd_ctx *ctx, · line 1859

Quoted 3 lines
fs/userfaultfd.c +3 -0
+	/* only toggleable features are allowed */
+	if ((mode.enable | mode.disable) & ~UFFD_FEATURE_TOGGLEABLE)
+		return -EINVAL;

The commit message states "Only async features that were enabled at UFFDIO_API time may be toggled." However, the code only checks that the requested feature is in UFFD_FEATURE_TOGGLEABLE.

Is it intentional that a user who opened a uffd without UFFD_FEATURE_MINOR_ASYNC can still enable it later via UFFDIO_SET_MODE?

The comment is stale. Will fix.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author reviewer
#13 lore raw

RFC PATCH 11/12 selftests/mm: add userfaultfd anonymous minor fault tests

In reply to Kiryl Shutsemau (Meta)

Add tests for UFFD_FEATURE_MINOR_ANON, UFFD_FEATURE_MINOR_ASYNC, UFFDIO_DEACTIVATE, UFFDIO_SET_MODE, and PAGE_IS_UFFD_DEACTIVATED:

- minor-anon-async: populate pages, register MODE_MINOR with MINOR_ASYNC, deactivate via UFFDIO_DEACTIVATE, re-access and verify content is preserved with no faults delivered to the handler.

- minor-anon-sync: same setup but without MINOR_ASYNC. Verify that each deactivated page access delivers a MINOR fault to the handler, and UFFDIO_CONTINUE resolves it. Exercises both PTE and THP paths.

- minor-anon-pagemap: deactivate a range, touch first half, use PAGEMAP_SCAN with PAGE_IS_UFFD_DEACTIVATED to verify the untouched second half is reported as cold.

- minor-anon-gup: write() from a deactivated page into a pipe to exercise GUP resolution through protnone PTEs via async auto-restore.

- minor-anon-async-toggle: full detection-to-eviction cycle using UFFDIO_SET_MODE. Start async (detection), flip to sync (eviction of cold pages), flip back to async.

- minor-anon-close: deactivate pages, close the uffd fd, verify all pages are accessible again (protnone PTEs restored on cleanup).

Assisted-by Claude:claude-opus-4-6
---
 tools/testing/selftests/mm/uffd-unit-tests.c | 458 +++++++++++++++++++
 1 file changed, 458 insertions(+)
tools/testing/selftests/mm/uffd-unit-tests.c +458 -0
index 6f5e404a446c..8bd5a642bd5a 100644
@@ -7,6 +7,7 @@
 
 #include "uffd-common.h"
 
+#include <linux/fs.h>
 #include "../../../../mm/gup_test.h"
 
 #ifdef __NR_userfaultfd
@@ -623,6 +624,423 @@ void uffd_minor_collapse_test(uffd_global_test_opts_t *gopts, uffd_test_args_t *
 	uffd_minor_test_common(gopts, true, false);
 }
 
+static void deactivate_range(int uffd, __u64 start, __u64 len)
+{
+	struct uffdio_range range = { .start = start, .len = len };
+
+	if (ioctl(uffd, UFFDIO_DEACTIVATE, &range))
+		err("UFFDIO_DEACTIVATE failed");
+}
+
+static void set_async_mode(int uffd, bool enable)
+{
+	struct uffdio_set_mode mode = { };
+
+	if (enable)
+		mode.enable = UFFD_FEATURE_MINOR_ASYNC;
+	else
+		mode.disable = UFFD_FEATURE_MINOR_ASYNC;
+
+	if (ioctl(uffd, UFFDIO_SET_MODE, &mode))
+		err("UFFDIO_SET_MODE failed");
+}
+
+/*
+ * Test async minor faults on anonymous memory.
+ * Populate pages, register MODE_MINOR with MINOR_ASYNC,
+ * deactivate, re-access, verify content preserved and no faults delivered.
+ */
+static void uffd_minor_anon_async_test(uffd_global_test_opts_t *gopts,
+				       uffd_test_args_t *args)
+{
+	unsigned long nr_pages = gopts->nr_pages;
+	unsigned long page_size = gopts->page_size;
+	unsigned long p;
+
+	/* Populate all pages with known content */
+	for (p = 0; p < nr_pages; p++)
+		memset(gopts->area_dst + p * page_size, p % 255 + 1, page_size);
+
+	/* Register MODE_MINOR (uffd was opened with MINOR_ANON | MINOR_ASYNC) */
+	if (uffd_register(gopts->uffd, gopts->area_dst,
+			  nr_pages * page_size,
+			  false, false, true))
+		err("register failure");
+
+	/* Deactivate all pages — sets protnone */
+	deactivate_range(gopts->uffd, (uint64_t)gopts->area_dst,
+			 nr_pages * page_size);
+
+	/* Access all pages — should auto-resolve, no faults */
+	for (p = 0; p < nr_pages; p++) {
+		unsigned char *page = (unsigned char *)gopts->area_dst +
+				      p * page_size;
+		unsigned char expected = p % 255 + 1;
+
+		if (page[0] != expected) {
+			uffd_test_fail("page %lu content mismatch: %u != %u",
+				       p, page[0], expected);
+			return;
+		}
+	}
+
+	uffd_test_pass();
+}
+
+/*
+ * Custom fault handler for anon minor — just UFFDIO_CONTINUE, no content
+ * modification (the page is protnone so we can't access it from here).
+ */
+static void uffd_handle_minor_anon(uffd_global_test_opts_t *gopts,
+				   struct uffd_msg *msg,
+				   struct uffd_args *uargs)
+{
+	struct uffdio_continue req;
+
+	if (!(msg->arg.pagefault.flags & UFFD_PAGEFAULT_FLAG_MINOR))
+		err("expected minor fault, got 0x%llx",
+		    msg->arg.pagefault.flags);
+
+	req.range.start = msg->arg.pagefault.address;
+	req.range.len = gopts->page_size;
+	req.mode = 0;
+	if (ioctl(gopts->uffd, UFFDIO_CONTINUE, &req)) {
+		/*
+		 * THP races with khugepaged collapse/split:
+		 * EAGAIN: PMD changed under us
+		 * EEXIST: THP present but already resolved
+		 * In both cases the page is accessible — the faulting
+		 * thread retries and succeeds.
+		 */
+		if (errno != EEXIST && errno != EAGAIN)
+			err("UFFDIO_CONTINUE failed");
+	}
+
+	uargs->minor_faults++;
+}
+
+/*
+ * Test sync minor faults on anonymous memory.
+ * Populate pages, register MODE_MINOR (sync), deactivate,
+ * access from worker thread, verify fault delivered, UFFDIO_CONTINUE resolves.
+ */
+static void uffd_minor_anon_sync_test(uffd_global_test_opts_t *gopts,
+				      uffd_test_args_t *args)
+{
+	unsigned long nr_pages = gopts->nr_pages;
+	unsigned long page_size = gopts->page_size;
+	pthread_t uffd_mon;
+	struct uffd_args uargs = { };
+	char c = '\0';
+	unsigned long p;
+
+	uargs.gopts = gopts;
+	uargs.handle_fault = uffd_handle_minor_anon;
+
+	/* Populate all pages */
+	for (p = 0; p < nr_pages; p++)
+		memset(gopts->area_dst + p * page_size, p % 255 + 1, page_size);
+
+	/* Register MODE_MINOR (uffd opened with MINOR_ANON, no MINOR_ASYNC) */
+	if (uffd_register(gopts->uffd, gopts->area_dst,
+			  nr_pages * page_size,
+			  false, false, true))
+		err("register failure");
+
+	/* Deactivate all pages */
+	deactivate_range(gopts->uffd, (uint64_t)gopts->area_dst,
+			 nr_pages * page_size);
+
+	/* Start fault handler thread */
+	if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &uargs))
+		err("uffd_poll_thread create");
+
+	/* Access all pages — triggers sync minor faults, handler does CONTINUE */
+	for (p = 0; p < nr_pages; p++) {
+		unsigned char *page = (unsigned char *)gopts->area_dst +
+				      p * page_size;
+
+		if (page[0] != (p % 255 + 1)) {
+			uffd_test_fail("page %lu content mismatch", p);
+			goto out;
+		}
+	}
+
+	if (uargs.minor_faults == 0) {
+		uffd_test_fail("expected minor faults, got 0");
+		goto out;
+	}
+
+	uffd_test_pass();
+out:
+	if (write(gopts->pipefd[1], &c, sizeof(c)) != sizeof(c))
+		err("pipe write");
+	if (pthread_join(uffd_mon, NULL))
+		err("join() failed");
+}
+
+/*
+ * Test PAGEMAP_SCAN detection of deactivated (cold) pages.
+ */
+static void uffd_minor_anon_pagemap_test(uffd_global_test_opts_t *gopts,
+					  uffd_test_args_t *args)
+{
+	unsigned long nr_pages = gopts->nr_pages;
+	unsigned long page_size = gopts->page_size;
+	unsigned long p;
+	struct page_region regions[16];
+	struct pm_scan_arg pm_arg;
+	int pagemap_fd;
+	long ret;
+
+	/* Need at least 4 pages */
+	if (nr_pages < 4) {
+		uffd_test_skip("need at least 4 pages");
+		return;
+	}
+
+	/* Populate all pages */
+	for (p = 0; p < nr_pages; p++)
+		memset(gopts->area_dst + p * page_size, 0xab, page_size);
+
+	/* Register and deactivate */
+	if (uffd_register(gopts->uffd, gopts->area_dst,
+			  nr_pages * page_size,
+			  false, false, true))
+		err("register failure");
+
+	deactivate_range(gopts->uffd, (uint64_t)gopts->area_dst,
+			 nr_pages * page_size);
+
+	/* Touch first half of pages to re-activate them (async auto-resolve) */
+	for (p = 0; p < nr_pages / 2; p++) {
+		volatile char *page = gopts->area_dst + p * page_size;
+		(void)*page;
+	}
+
+	/* Scan for cold (still deactivated) pages */
+	pagemap_fd = open("/proc/self/pagemap", O_RDONLY);
+	if (pagemap_fd < 0)
+		err("open pagemap");
+
+	memset(&pm_arg, 0, sizeof(pm_arg));
+	pm_arg.size = sizeof(pm_arg);
+	pm_arg.start = (uint64_t)gopts->area_dst;
+	pm_arg.end = (uint64_t)gopts->area_dst + nr_pages * page_size;
+	pm_arg.vec = (uint64_t)regions;
+	pm_arg.vec_len = 16;
+	pm_arg.category_mask = PAGE_IS_UFFD_DEACTIVATED;
+	pm_arg.return_mask = PAGE_IS_UFFD_DEACTIVATED;
+
+	ret = ioctl(pagemap_fd, PAGEMAP_SCAN, &pm_arg);
+	close(pagemap_fd);
+
+	if (ret < 0) {
+		uffd_test_fail("PAGEMAP_SCAN failed: %s", strerror(errno));
+		return;
+	}
+
+	/*
+	 * The second half of pages should be reported as deactivated.
+	 * They may be coalesced into one region.
+	 */
+	if (ret < 1) {
+		uffd_test_fail("expected cold pages, got %ld regions", ret);
+		return;
+	}
+
+	/* Verify the cold region covers the second half */
+	uint64_t cold_start = regions[0].start;
+	uint64_t expected_start = (uint64_t)gopts->area_dst +
+				  (nr_pages / 2) * page_size;
+
+	if (cold_start != expected_start) {
+		uffd_test_fail("cold region starts at 0x%lx, expected 0x%lx",
+			       (unsigned long)cold_start,
+			       (unsigned long)expected_start);
+		return;
+	}
+
+	uffd_test_pass();
+}
+
+/*
+ * Test that GUP resolves through protnone PTEs (async mode).
+ * Deactivate pages, then use a pipe to exercise GUP on the deactivated
+ * memory. write() from deactivated pages triggers GUP which must fault
+ * through the protnone PTE.
+ */
+static void uffd_minor_anon_gup_test(uffd_global_test_opts_t *gopts,
+				     uffd_test_args_t *args)
+{
+	unsigned long page_size = gopts->page_size;
+	char *buf;
+	int pipefd[2];
+
+	buf = malloc(page_size);
+	if (!buf)
+		err("malloc");
+
+	/* Populate first page with known content */
+	memset(gopts->area_dst, 0xCD, page_size);
+
+	if (uffd_register(gopts->uffd, gopts->area_dst, page_size,
+			  false, false, true))
+		err("register failure");
+
+	deactivate_range(gopts->uffd, (uint64_t)gopts->area_dst, page_size);
+
+	if (pipe(pipefd))
+		err("pipe");
+
+	/*
+	 * write() from the deactivated page into the pipe.
+	 * This triggers GUP on the protnone PTE. In async mode the
+	 * kernel auto-restores permissions and GUP succeeds.
+	 */
+	if (write(pipefd[1], gopts->area_dst, page_size) != page_size) {
+		uffd_test_fail("write from deactivated page failed: %s",
+			       strerror(errno));
+		goto out;
+	}
+
+	if (read(pipefd[0], buf, page_size) != page_size) {
+		uffd_test_fail("read from pipe failed");
+		goto out;
+	}
+
+	if (memcmp(buf, "\xCD", 1) != 0) {
+		uffd_test_fail("content mismatch: got 0x%02x, expected 0xCD",
+			       (unsigned char)buf[0]);
+		goto out;
+	}
+
+	uffd_test_pass();
+out:
+	close(pipefd[0]);
+	close(pipefd[1]);
+	free(buf);
+}
+
+/*
+ * Test runtime toggle between async and sync modes.
+ * Start in async mode (detection), flip to sync (eviction), verify faults
+ * block, resolve them, flip back to async.
+ */
+static void uffd_minor_anon_async_toggle_test(uffd_global_test_opts_t *gopts,
+					      uffd_test_args_t *args)
+{
+	unsigned long nr_pages = gopts->nr_pages;
+	unsigned long page_size = gopts->page_size;
+	struct uffd_args uargs = { };
+	pthread_t uffd_mon;
+	char c = '\0';
+	unsigned long p;
+
+	uargs.gopts = gopts;
+	uargs.handle_fault = uffd_handle_minor_anon;
+
+	/* Populate */
+	for (p = 0; p < nr_pages; p++)
+		memset(gopts->area_dst + p * page_size, p % 255 + 1, page_size);
+
+	if (uffd_register(gopts->uffd, gopts->area_dst,
+			  nr_pages * page_size,
+			  false, false, true))
+		err("register failure");
+
+	/* Phase 1: async detection — deactivate, access first half */
+	deactivate_range(gopts->uffd, (uint64_t)gopts->area_dst,
+			 nr_pages * page_size);
+
+	for (p = 0; p < nr_pages / 2; p++) {
+		volatile char *page = gopts->area_dst + p * page_size;
+		(void)*page;  /* auto-resolves in async mode */
+	}
+
+	/* Phase 2: flip to sync for eviction */
+	set_async_mode(gopts->uffd, false);
+
+	/* Start handler — will receive faults for cold pages */
+	if (pthread_create(&uffd_mon, NULL, uffd_poll_thread, &uargs))
+		err("uffd_poll_thread create");
+
+	/* Access second half (cold pages) — should trigger sync faults */
+	for (p = nr_pages / 2; p < nr_pages; p++) {
+		unsigned char *page = (unsigned char *)gopts->area_dst +
+				      p * page_size;
+		if (page[0] != (p % 255 + 1)) {
+			uffd_test_fail("page %lu content mismatch", p);
+			goto out;
+		}
+	}
+
+	if (uargs.minor_faults == 0) {
+		uffd_test_fail("expected sync faults, got 0");
+		goto out;
+	}
+
+	/* Phase 3: flip back to async */
+	set_async_mode(gopts->uffd, true);
+
+	/* Deactivate and access again — should auto-resolve */
+	deactivate_range(gopts->uffd, (uint64_t)gopts->area_dst,
+			 nr_pages * page_size);
+
+	for (p = 0; p < nr_pages; p++) {
+		volatile char *page = gopts->area_dst + p * page_size;
+		(void)*page;
+	}
+
+	uffd_test_pass();
+out:
+	if (write(gopts->pipefd[1], &c, sizeof(c)) != sizeof(c))
+		err("pipe write");
+	if (pthread_join(uffd_mon, NULL))
+		err("join() failed");
+}
+
+/*
+ * Test that deactivated pages become accessible after closing uffd.
+ */
+static void uffd_minor_anon_close_test(uffd_global_test_opts_t *gopts,
+				       uffd_test_args_t *args)
+{
+	unsigned long nr_pages = gopts->nr_pages;
+	unsigned long page_size = gopts->page_size;
+	unsigned long p;
+
+	/* Populate */
+	for (p = 0; p < nr_pages; p++)
+		memset(gopts->area_dst + p * page_size, p % 255 + 1, page_size);
+
+	if (uffd_register(gopts->uffd, gopts->area_dst,
+			  nr_pages * page_size,
+			  false, false, true))
+		err("register failure");
+
+	deactivate_range(gopts->uffd, (uint64_t)gopts->area_dst,
+			 nr_pages * page_size);
+
+	/* Close uffd — should restore protnone PTEs */
+	close(gopts->uffd);
+	gopts->uffd = -1;
+
+	/* All pages should be accessible with original content */
+	for (p = 0; p < nr_pages; p++) {
+		unsigned char *page = (unsigned char *)gopts->area_dst +
+				      p * page_size;
+		unsigned char expected = p % 255 + 1;
+
+		if (page[0] != expected) {
+			uffd_test_fail("page %lu not accessible after close", p);
+			return;
+		}
+	}
+
+	uffd_test_pass();
+}
+
 static sigjmp_buf jbuf, *sigbuf;
 
 static void sighndl(int sig, siginfo_t *siginfo, void *ptr)
@@ -1625,6 +2043,46 @@ uffd_test_case_t uffd_tests[] = {
 		/* We can't test MADV_COLLAPSE, so try our luck */
 		.uffd_feature_required = UFFD_FEATURE_MINOR_SHMEM,
 	},
+	{
+		.name = "minor-anon-async",
+		.uffd_fn = uffd_minor_anon_async_test,
+		.mem_targets = MEM_ANON,
+		.uffd_feature_required =
+		UFFD_FEATURE_MINOR_ANON | UFFD_FEATURE_MINOR_ASYNC,
+	},
+	{
+		.name = "minor-anon-sync",
+		.uffd_fn = uffd_minor_anon_sync_test,
+		.mem_targets = MEM_ANON,
+		.uffd_feature_required = UFFD_FEATURE_MINOR_ANON,
+	},
+	{
+		.name = "minor-anon-pagemap",
+		.uffd_fn = uffd_minor_anon_pagemap_test,
+		.mem_targets = MEM_ANON,
+		.uffd_feature_required =
+		UFFD_FEATURE_MINOR_ANON | UFFD_FEATURE_MINOR_ASYNC,
+	},
+	{
+		.name = "minor-anon-gup",
+		.uffd_fn = uffd_minor_anon_gup_test,
+		.mem_targets = MEM_ANON,
+		.uffd_feature_required =
+		UFFD_FEATURE_MINOR_ANON | UFFD_FEATURE_MINOR_ASYNC,
+	},
+	{
+		.name = "minor-anon-async-toggle",
+		.uffd_fn = uffd_minor_anon_async_toggle_test,
+		.mem_targets = MEM_ANON,
+		.uffd_feature_required =
+		UFFD_FEATURE_MINOR_ANON | UFFD_FEATURE_MINOR_ASYNC,
+	},
+	{
+		.name = "minor-anon-close",
+		.uffd_fn = uffd_minor_anon_close_test,
+		.mem_targets = MEM_ANON,
+		.uffd_feature_required = UFFD_FEATURE_MINOR_ANON,
+	},
 	{
 		.name = "sigbus",
 		.uffd_fn = uffd_sigbus_test,
Signature · 1 line
2.51.2
author reviewer
#14 lore raw

RFC PATCH 12/12 Documentation/userfaultfd: document working set tracking

In reply to Kiryl Shutsemau (Meta)

Document the new userfaultfd capabilities for VM working set tracking:

- UFFD_FEATURE_MINOR_ANON and UFFD_FEATURE_MINOR_ASYNC for anonymous minor fault interception using the PROT_NONE hinting mechanism.
- UFFDIO_DEACTIVATE for marking pages as inaccessible while keeping them resident.
- Sync and async fault resolution modes, and UFFDIO_SET_MODE for runtime toggling between them.
- PAGEMAP_SCAN with PAGE_IS_UFFD_DEACTIVATED for cold page detection.
- Cleanup semantics on unregister and close.
- NUMA balancing interaction on anonymous VMAs.
- Complete VMM workflow example for the cold page eviction lifecycle, with a note on shmem applicability.

Update the feature flag descriptions at the top of the guide to reference the new section.

Assisted-by Claude:claude-opus-4-6
---
 Documentation/admin-guide/mm/userfaultfd.rst | 141 ++++++++++++++++++-
 1 file changed, 140 insertions(+), 1 deletion(-)
Documentation/admin-guide/mm/userfaultfd.rst +140 -1
index e5cc8848dcb3..fc89e029060c 100644
@@ -111,7 +111,11 @@ events, except page fault notifications, may be generated:
 - ``UFFD_FEATURE_MINOR_HUGETLBFS`` indicates that the kernel supports
   ``UFFDIO_REGISTER_MODE_MINOR`` registration for hugetlbfs virtual memory
   areas. ``UFFD_FEATURE_MINOR_SHMEM`` is the analogous feature indicating
-  support for shmem virtual memory areas.
+  support for shmem virtual memory areas. ``UFFD_FEATURE_MINOR_ANON``
+  extends minor fault support to anonymous private memory using
+  PROT_NONE hinting; see the `Anonymous Minor Faults`_ section.
+  ``UFFD_FEATURE_MINOR_ASYNC`` enables asynchronous auto-resolution for
+  anonymous minor faults (requires ``UFFD_FEATURE_MINOR_ANON``).
 
 - ``UFFD_FEATURE_MOVE`` indicates that the kernel supports moving an
   existing page contents from userspace.
@@ -297,6 +301,141 @@ transparent to the guest, we want that same address range to act as if it was
 still poisoned, even though it's on a new physical host which ostensibly
 doesn't have a memory error in the exact same spot.
 
+Anonymous Minor Faults
+----------------------
+
+``UFFD_FEATURE_MINOR_ANON`` enables ``UFFDIO_REGISTER_MODE_MINOR`` on
+anonymous private memory. Unlike shmem/hugetlbfs minor faults (where a page
+exists in the page cache but has no PTE), anonymous minor faults use the
+PROT_NONE hinting mechanism: pages remain resident in memory with their PFNs
+preserved in the PTEs, but access permissions are removed so the next access
+triggers a fault.
+
+This is designed for VM memory managers that need to track the working set of
+anonymous guest memory for cold page eviction to tiered or remote storage.
+
+**Setup:**
+
+1. Open a userfaultfd and enable ``UFFD_FEATURE_MINOR_ANON`` (and optionally
+   ``UFFD_FEATURE_MINOR_ASYNC``) via ``UFFDIO_API``.
+
+2. Register the guest memory range with ``UFFDIO_REGISTER_MODE_MINOR``
+   (and ``UFFDIO_REGISTER_MODE_MISSING`` if evicted pages will need to be
+   fetched back from storage).
+
+**Deactivation:**
+
+Use ``UFFDIO_DEACTIVATE`` to mark pages as inaccessible. This ioctl takes a
+``struct uffdio_range`` and sets PROT_NONE on all present PTEs in the range,
+using the same mechanism as NUMA balancing. Pages stay resident and their
+physical frames are preserved — only access permissions are removed.
+
+**Fault Handling:**
+
+When a deactivated page is accessed:
+
+- **Sync mode** (default): The faulting thread blocks and a
+  ``UFFD_PAGEFAULT_FLAG_MINOR`` message is delivered to the userfaultfd
+  handler. The handler resolves the fault with ``UFFDIO_CONTINUE``, which
+  restores the PTE permissions and wakes the faulting thread.
+
+- **Async mode** (``UFFD_FEATURE_MINOR_ASYNC``): The kernel automatically
+  restores PTE permissions and the thread continues without blocking. No
+  message is delivered to the handler.
+
+**Cold Page Detection with PAGEMAP_SCAN:**
+
+After deactivating a range and letting the application run, use the
+``PAGEMAP_SCAN`` ioctl on ``/proc/pid/pagemap`` with the
+``PAGE_IS_UFFD_DEACTIVATED`` category flag to efficiently find pages that were
+never re-accessed (cold pages)::
+
+    struct pm_scan_arg arg = {
+        .size = sizeof(arg),
+        .start = guest_mem_start,
+        .end = guest_mem_end,
+        .vec = (uint64_t)regions,
+        .vec_len = regions_len,
+        .category_mask = PAGE_IS_UFFD_DEACTIVATED,
+        .return_mask = PAGE_IS_UFFD_DEACTIVATED,
+    };
+    long n = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg);
+
+The returned ``page_region`` array contains contiguous cold ranges that can
+then be evicted.
+
+**Cleanup:**
+
+When the userfaultfd is closed or the range is unregistered, all protnone
+PTEs are automatically restored to their normal VMA permissions. This
+prevents pages from becoming permanently inaccessible.
+
+**Interaction with NUMA Balancing:**
+
+NUMA balancing is automatically disabled on anonymous VMAs registered with
+``UFFDIO_REGISTER_MODE_MINOR``, since both mechanisms use PROT_NONE PTEs
+as access hints and would interfere with each other. Shmem VMAs are not
+affected since ``UFFDIO_DEACTIVATE`` zaps PTEs there instead of using
+PROT_NONE.
+
+**VMM Working Set Tracking Workflow:**
+
+A typical VMM lifecycle for cold page eviction to tiered storage::
+
+    /* One-time setup */
+    uffd = userfaultfd(O_CLOEXEC | O_NONBLOCK);
+    ioctl(uffd, UFFDIO_API, &(struct uffdio_api){
+        .api = UFFD_API,
+        .features = UFFD_FEATURE_MINOR_ANON |
+                    UFFD_FEATURE_MINOR_ASYNC,
+    });
+    ioctl(uffd, UFFDIO_REGISTER, &(struct uffdio_register){
+        .range = { guest_mem, guest_size },
+        .mode = UFFDIO_REGISTER_MODE_MINOR |
+                UFFDIO_REGISTER_MODE_MISSING,
+    });
+
+    /* Tracking loop */
+    while (vm_running) {
+        /* 1. Detection phase (async — no vCPU stalls) */
+        ioctl(uffd, UFFDIO_DEACTIVATE, &full_range);
+        sleep(tracking_interval);
+
+        /* 2. Find cold pages */
+        ioctl(pagemap_fd, PAGEMAP_SCAN, &(struct pm_scan_arg){
+            .category_mask = PAGE_IS_UFFD_DEACTIVATED,
+            ...
+        });
+
+        /* 3. Switch to sync for safe eviction */
+        ioctl(uffd, UFFDIO_SET_MODE,
+              &(struct uffdio_set_mode){
+                  .disable = UFFD_FEATURE_MINOR_ASYNC });
+
+        /* 4. Evict cold pages (vCPU faults block in handler) */
+        for each cold range:
+            pwrite(storage_fd, cold_addr, len, offset);
+            madvise(cold_addr, len, MADV_DONTNEED);
+
+        /* 5. Resume async tracking */
+        ioctl(uffd, UFFDIO_SET_MODE,
+              &(struct uffdio_set_mode){
+                  .enable = UFFD_FEATURE_MINOR_ASYNC });
+    }
+
+During step 4, if a vCPU accesses a cold page being evicted, it blocks
+with a ``UFFD_PAGEFAULT_FLAG_MINOR`` fault. The handler can either let it
+wait (the eviction completes, ``MADV_DONTNEED`` fires, the fault retries as
+``MISSING`` and is resolved with ``UFFDIO_COPY`` from storage) or resolve
+it immediately with ``UFFDIO_CONTINUE``.
+
+The same workflow applies to shmem-backed guest memory
+(``UFFD_FEATURE_MINOR_SHMEM``). The only difference is the
+``PAGEMAP_SCAN`` mask for cold page detection: use
+``!PAGE_IS_PRESENT`` instead of ``PAGE_IS_UFFD_DEACTIVATED``, since
+``UFFDIO_DEACTIVATE`` zaps PTEs on shmem (pages stay in page cache)
+rather than setting PROT_NONE.
+
 QEMU/KVM
 ========
 
Signature · 1 line
2.51.2
reviewer
#15 lore raw
In reply to Kiryl Shutsemau (Meta)

Hi, Kiryl,

Quoted 3 lines · Kiryl Shutsemau (Meta)

This series adds userfaultfd support for tracking the working set of VM guest memory, enabling VMMs to identify cold pages and evict them to tiered or remote storage.

Thanks for sharing this work, it looks very interesting to me.

Personally I am also looking at some kind of VMM memtiering issues. I'm not sure if you saw my lsfmm proposal, it mentioned the challenge we're facing, it's slightly different but still a bit relevant:

https://lore.kernel.org/all/[email protected]/

Unfortunately, that proposal was rejected upstream.

For us, it's so far more about migration and how migration process introduce zero impact to guest workloads especially on hotness. I'm not sure if we have any shared goals over that aspect.

Quoted 33 lines

== Problem ==

VMMs managing guest memory need to:
1. Track which pages are actively used (working set detection)
2. Safely evict cold pages to slower storage
3. Fetch pages back on demand when accessed again

For shmem-backed guest memory, working set tracking partially works
today: MADV_DONTNEED zaps PTEs while pages stay in page cache, and re-access auto-resolves from cache. But safe eviction still requires synchronous fault interception to prevent data loss races.

For anonymous guest memory (needed for KSM cross-VM deduplication), there is no mechanism at all — clearing a PTE loses the page.

== Solution ==

The series introduces a unified userfaultfd interface that works across both anonymous and shmem-backed memory:

UFFD_FEATURE_MINOR_ANON: extends MODE_MINOR registration to anonymous private memory. Uses the PROT_NONE hinting mechanism (same as NUMA balancing) to make pages inaccessible without freeing them.

UFFD_FEATURE_MINOR_ASYNC: auto-resolves minor faults without handler involvement. The kernel restores PTE permissions immediately and the faulting thread continues. Works for anonymous, shmem, and hugetlbfs.

UFFDIO_DEACTIVATE: marks pages as deactivated. For anonymous memory, sets PROT_NONE on PTEs (pages stay resident). For shmem/hugetlbfs, zaps PTEs (pages stay in page cache).

Quoted 8 lines

UFFDIO_SET_MODE: toggles MINOR_ASYNC at runtime, synchronized via mmap_write_lock. Enables the VMM workflow: async mode for lightweight detection, sync mode for race-free eviction.

PAGE_IS_UFFD_DEACTIVATED: PAGEMAP_SCAN category flag for efficient batch detection of cold (still-deactivated) anonymous pages.

== VMM Workflow ==

AFAIU, this workflow provides two functionalities:

Quoted 4 lines

UFFDIO_DEACTIVATE(all) -- async, no vCPU stalls
sleep(interval)
PAGEMAP_SCAN -- find cold pages

Until here it's only about page hotness tracking. I am curious whether you evaluated idle page tracking. Is it because of perf overheads on rmap? To me, your solution (until here.. on the hotness sampling) reads more like a more efficient way to do idle page tracking but only per-mm, not per-folio.

That will also be something I would like to benefit if QEMU will decide to do full userspace swap. I think that's our last resort, I'll likely start with something that makes QEMU work together with Linux on swapping (e.g. we're happy to make MGLRU or any reclaim logic that Linux mm currently uses, as long as efficient) then QEMU only cares about the rest, which is what the migration problem is about.

The other issue about idle page tracking to us is, I believe MGLRU currently doesn't work well with it (due to ignoring IDLE bits) where the old LRU algo works. I'm not sure how much you evaluated above, so it'll be great to share from that perspective too. I also mentioned some of these challenges in the lsfmm proposal link above.

Quoted 3 lines

UFFDIO_SET_MODE(sync) -- block faults for eviction
pwrite + MADV_DONTNEED cold pages -- safe, faults block
UFFDIO_SET_MODE(async) -- resume tracking

These operations are the 2nd function. It's, IMHO, a full userspace swap system based on userfaultfd.

Have you thought about directly relying on userfaultfd-wp to do this work? The relevant question is, why do we need to block guest reads on pages being evicted by the userapp? Can we still allow that to happen, which seems to be more efficient? IIUC, only writes / updates matters in such swap system.

Also, I'm not sure if you're aware of LLNL's umap library:

https://github.com/llnl/umap

That implemnted the swap system using userfaultfd wr-protect mode only, so no new kernel API needed.

Thanks,

Quoted 94 lines

The same workflow applies to shmem, with a different PAGEMAP_SCAN mask (!PAGE_IS_PRESENT instead of PAGE_IS_UFFD_DEACTIVATED).

== NUMA Balancing ==

NUMA balancing scanning is skipped on anonymous VM_UFFD_MINOR VMAs to avoid protnone conflicts. NUMA locality stats are fed from the uffd fault path via task_numa_fault() so the scheduler retains placement data. Shmem VMAs are unaffected (UFFDIO_DEACTIVATE zaps PTEs there, no protnone involved).

== Testing ==

The series includes 6 new selftests covering async/sync modes, PAGEMAP_SCAN cold detection, GUP through protnone, UFFDIO_SET_MODE toggling, and cleanup on close. All 73 uffd unit tests pass (including hugetlb) across defconfig, allnoconfig, allmodconfig, and randomized configs.

Kiryl Shutsemau (Meta) (12):
userfaultfd: define UAPI constants for anonymous minor faults
userfaultfd: add UFFD_FEATURE_MINOR_ANON registration support
userfaultfd: implement UFFDIO_DEACTIVATE ioctl
userfaultfd: UFFDIO_CONTINUE for anonymous memory
mm: intercept protnone faults on VM_UFFD_MINOR anonymous VMAs
userfaultfd: auto-resolve shmem and hugetlbfs minor faults in async
mode
sched/numa: skip scanning anonymous VM_UFFD_MINOR VMAs
userfaultfd: enable UFFD_FEATURE_MINOR_ANON
mm/pagemap: add PAGE_IS_UFFD_DEACTIVATED to PAGEMAP_SCAN
userfaultfd: add UFFDIO_SET_MODE for runtime sync/async toggle selftests/mm: add userfaultfd anonymous minor fault tests Documentation/userfaultfd: document working set tracking

 Documentation/admin-guide/mm/userfaultfd.rst | 141 ++++-
 fs/proc/task_mmu.c                           |  11 +-
 fs/userfaultfd.c                             | 184 +++++-
 include/linux/huge_mm.h                      |   6 +
 include/linux/mm.h                           |   2 +
 include/linux/sched/numa_balancing.h         |   1 +
 include/linux/userfaultfd_k.h                |  21 +-
 include/trace/events/sched.h                 |   3 +-
 include/uapi/linux/fs.h                      |   1 +
 include/uapi/linux/userfaultfd.h             |  40 +-
 kernel/sched/fair.c                          |  13 +
 mm/huge_memory.c                             |  33 +-
 mm/hugetlb.c                                 |   3 +-
 mm/memory.c                                  |  51 +-
 mm/mprotect.c                                |   9 +-
 mm/shmem.c                                   |   3 +-
 mm/userfaultfd.c                             | 164 +++++-
 tools/testing/selftests/mm/uffd-unit-tests.c | 458 +++++++++++++++
 18 files changed, 1096 insertions(+), 48 deletions(-)

Kiryl Shutsemau (Meta) (12):
userfaultfd: define UAPI constants for anonymous minor faults
userfaultfd: add UFFD_FEATURE_MINOR_ANON registration support
userfaultfd: implement UFFDIO_DEACTIVATE ioctl
userfaultfd: UFFDIO_CONTINUE for anonymous memory
mm: intercept protnone faults on VM_UFFD_MINOR anonymous VMAs
userfaultfd: auto-resolve shmem and hugetlbfs minor faults in async
mode
sched/numa: skip scanning anonymous VM_UFFD_MINOR VMAs
userfaultfd: enable UFFD_FEATURE_MINOR_ANON
mm/pagemap: add PAGE_IS_UFFD_DEACTIVATED to PAGEMAP_SCAN
userfaultfd: add UFFDIO_SET_MODE for runtime sync/async toggle selftests/mm: add userfaultfd anonymous minor fault tests Documentation/userfaultfd: document working set tracking

 Documentation/admin-guide/mm/userfaultfd.rst | 141 +++++-
 fs/proc/task_mmu.c                           |  11 +-
 fs/userfaultfd.c                             | 184 +++++++-
 include/linux/huge_mm.h                      |   6 +
 include/linux/mm.h                           |   2 +
 include/linux/sched/numa_balancing.h         |   1 +
 include/linux/userfaultfd_k.h                |  21 +-
 include/trace/events/sched.h                 |   3 +-
 include/uapi/linux/fs.h                      |   1 +
 include/uapi/linux/userfaultfd.h             |  40 +-
 kernel/sched/fair.c                          |  13 +
 mm/huge_memory.c                             |  33 +-
 mm/hugetlb.c                                 |   3 +-
 mm/memory.c                                  |  51 ++-
 mm/mprotect.c                                |   9 +-
 mm/shmem.c                                   |   3 +-
 mm/userfaultfd.c                             | 164 ++++++-
 tools/testing/selftests/mm/uffd-unit-tests.c | 458 +++++++++++++++++++
 18 files changed, 1096 insertions(+), 48 deletions(-)
Signature · 1 line
2.51.2
Signature · 1 line
Peter Xu
author reviewer
#16 lore raw
Quoted 7 lines · Peter Xu

Hi, Kiryl,

Quoted 3 lines · Kiryl Shutsemau (Meta)

This series adds userfaultfd support for tracking the working set of VM guest memory, enabling VMMs to identify cold pages and evict them to tiered or remote storage.

Quoted 7 lines

Thanks for sharing this work, it looks very interesting to me.

Personally I am also looking at some kind of VMM memtiering issues. I'm not sure if you saw my lsfmm proposal, it mentioned the challenge we're facing, it's slightly different but still a bit relevant:

https://lore.kernel.org/all/[email protected]/

Thanks will read up. I didn't follow userfultfd work until recently.

Quoted 1 line

Unfortunately, that proposal was rejected upstream.

Sorry about that. We can chat about in hall track, if you are there :)

Quoted 11 lines
Quoted 1 line

== VMM Workflow ==

AFAIU, this workflow provides two functionalities:

Quoted 4 lines

UFFDIO_DEACTIVATE(all) -- async, no vCPU stalls
sleep(interval)
PAGEMAP_SCAN -- find cold pages

Until here it's only about page hotness tracking. I am curious whether you evaluated idle page tracking. Is it because of perf overheads on rmap?

I didn't gave idle page tracking much thought. I needed uffd faults to serialize reclaim against memory accesses. If use it for one thing we can as well try to use it for tracking as well. And it seems to be fitting together nicely with sync/async mode flipping.

Quoted 17 lines

To
me, your solution (until here.. on the hotness sampling) reads more like a more efficient way to do idle page tracking but only per-mm, not per-folio.

That will also be something I would like to benefit if QEMU will decide to do full userspace swap. I think that's our last resort, I'll likely start with something that makes QEMU work together with Linux on swapping (e.g. we're happy to make MGLRU or any reclaim logic that Linux mm currently uses, as long as efficient) then QEMU only cares about the rest, which is what the migration problem is about.

The other issue about idle page tracking to us is, I believe MGLRU currently doesn't work well with it (due to ignoring IDLE bits) where the old LRU algo works. I'm not sure how much you evaluated above, so it'll be great to share from that perspective too. I also mentioned some of these challenges in the lsfmm proposal link above.

Quoted 6 lines
Quoted 3 lines

UFFDIO_SET_MODE(sync) -- block faults for eviction
pwrite + MADV_DONTNEED cold pages -- safe, faults block
UFFDIO_SET_MODE(async) -- resume tracking

These operations are the 2nd function. It's, IMHO, a full userspace swap system based on userfaultfd.

Right. And we want to decide where to put cold pages from userspace.

Quoted 5 lines

Have you thought about directly relying on userfaultfd-wp to do this work? The relevant question is, why do we need to block guest reads on pages being evicted by the userapp? Can we still allow that to happen, which seems to be more efficient? IIUC, only writes / updates matters in such swap system.

But we do care about about read accesses. We don't want to swap out pages that got read-touched. And we cannot in practice switch to WP mode after PAGEMAP_SCAN: it would require a lot of UFFDIO_WRITEPROTECT calls with TLB flushing each.

With my approach switching tracking and reclaiming is single bit flip under mmap lock.

Quoted 6 lines

Also, I'm not sure if you're aware of LLNL's umap library:

https://github.com/llnl/umap

That implemnted the swap system using userfaultfd wr-protect mode only, so no new kernel API needed.

Will look into it. Thanks.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
reviewer
#17 lore raw
Quoted 10 lines · Kiryl Shutsemau
Quoted 9 lines · Peter Xu

Hi, Kiryl,

Quoted 3 lines · Kiryl Shutsemau (Meta)

This series adds userfaultfd support for tracking the working set of VM guest memory, enabling VMMs to identify cold pages and evict them to tiered or remote storage.

Thanks for sharing this work, it looks very interesting to me.

Quoted 7 lines
Quoted 5 lines

Personally I am also looking at some kind of VMM memtiering issues. I'm not sure if you saw my lsfmm proposal, it mentioned the challenge we're facing, it's slightly different but still a bit relevant:

https://lore.kernel.org/all/[email protected]/

Thanks will read up. I didn't follow userfultfd work until recently.

Thanks. Note that the proposal doesn't have much with userfaultfd. You'll see when you start reading.

Quoted 4 lines
Quoted 2 lines

Unfortunately, that proposal was rejected upstream.

Sorry about that. We can chat about in hall track, if you are there :)

I won't be there (as it's rejected.. hence not invited). But I'm always happy to discuss on this topic on the list or elsewhere. Alone the way I believe it'll also help us to know what is the most acceptable path forward as it's still very relevant.

Quoted 10 lines
Quoted 10 lines
Quoted 2 lines

== VMM Workflow ==

AFAIU, this workflow provides two functionalities:

Quoted 4 lines

UFFDIO_DEACTIVATE(all) -- async, no vCPU stalls
sleep(interval)
PAGEMAP_SCAN -- find cold pages

Quoted 7 lines
Quoted 2 lines

Until here it's only about page hotness tracking. I am curious whether you evaluated idle page tracking. Is it because of perf overheads on rmap?

I didn't gave idle page tracking much thought. I needed uffd faults to serialize reclaim against memory accesses. If use it for one thing we can as well try to use it for tracking as well. And it seems to be fitting together nicely with sync/async mode flipping.

Yes, I get your point.

It's just that it'll still partly done what access bit has already been doing for mm core in general on tracking hotness. So I wonder if we should still try to see if we can separate the two problems.

One other quick thought is maybe we could also report hotness from kernel directly rather than relying on async faults, you can refer to "(2) Hotness Information API" in my above proposal. Here when it's only about knowing which page is less frequently used, it's only a READ interface.

Quoted 27 lines
Quoted 18 lines

To
me, your solution (until here.. on the hotness sampling) reads more like a more efficient way to do idle page tracking but only per-mm, not per-folio.

That will also be something I would like to benefit if QEMU will decide to do full userspace swap. I think that's our last resort, I'll likely start with something that makes QEMU work together with Linux on swapping (e.g. we're happy to make MGLRU or any reclaim logic that Linux mm currently uses, as long as efficient) then QEMU only cares about the rest, which is what the migration problem is about.

The other issue about idle page tracking to us is, I believe MGLRU currently doesn't work well with it (due to ignoring IDLE bits) where the old LRU algo works. I'm not sure how much you evaluated above, so it'll be great to share from that perspective too. I also mentioned some of these challenges in the lsfmm proposal link above.

Quoted 6 lines
Quoted 3 lines

UFFDIO_SET_MODE(sync) -- block faults for eviction
pwrite + MADV_DONTNEED cold pages -- safe, faults block
UFFDIO_SET_MODE(async) -- resume tracking

These operations are the 2nd function. It's, IMHO, a full userspace swap system based on userfaultfd.

Right. And we want to decide where to put cold pages from userspace.

Quoted 8 lines
Quoted 5 lines

Have you thought about directly relying on userfaultfd-wp to do this work? The relevant question is, why do we need to block guest reads on pages being evicted by the userapp? Can we still allow that to happen, which seems to be more efficient? IIUC, only writes / updates matters in such swap system.

But we do care about about read accesses. We don't want to swap out pages that got read-touched. And we cannot in practice switch to WP mode

This is a good point.

When it's considered on top of your above "async trapping to collect hotness with userfaultfd" idea, it flows naturally with this idea indeed.

However, IMHO that should really be an extremely small window, and the major part the userapp should rely on is the larger window sampling whether, in your current case, PROT_NONE (or PTE_NONE for shmem) switched back to a accessable PTE.

It means using RW protection v.s. WR-ONLY protection will only differ very slightly if by accident some page got read-only during evicting. For example, if the mgmt app monitors PROT_NONE state for 30 seconds, make a decision to evict, evicting takes 5ms, then within 5ms someone read the page. It means it only misses the 5ms/30sec access pattern of guest.

So far I don't yet know if this would justify a new kernel API just for that small false postive reporting some page is cold but actually it's hot. To me it's still fine to consider using WP-ONLY and just allow that trivial window to get refaulted later, because it shouldn't be the majority.

Quoted 2 lines

after PAGEMAP_SCAN: it would require a lot of UFFDIO_WRITEPROTECT calls with TLB flushing each.

This is indeed a concern, maybe a bigger one. I don't know how much benefit we can get from avoiding one extra TLB flush when evicting. IMHO some numbers might be more than great to justify this part.

While at this, I do have a pure question that is relevant on the full protection scheme (and it can be naive; please bare with me on not yet reading the whole series): if you change anon mappings to PROT_NONE in pgtables, then how do the mgmt app reads this page before dumping it to anywhere? It's not like shmem where you can have a separate mapping.

Do you need to fork(), for example?

Quoted 4 lines

With my approach switching tracking and reclaiming is single bit flip under mmap lock.

Quoted 8 lines
Quoted 6 lines

Also, I'm not sure if you're aware of LLNL's umap library:

https://github.com/llnl/umap

That implemnted the swap system using userfaultfd wr-protect mode only, so no new kernel API needed.

Will look into it. Thanks.

Thanks,

Signature · 1 line
Peter Xu
#18 lore raw
In reply to Kiryl Shutsemau (Meta)
Quoted 19 lines · Kiryl Shutsemau (Meta)

This series adds userfaultfd support for tracking the working set of VM guest memory, enabling VMMs to identify cold pages and evict them to tiered or remote storage.

== Problem ==

VMMs managing guest memory need to:
1. Track which pages are actively used (working set detection)
2. Safely evict cold pages to slower storage
3. Fetch pages back on demand when accessed again

For shmem-backed guest memory, working set tracking partially works
today: MADV_DONTNEED zaps PTEs while pages stay in page cache, and re-access auto-resolves from cache. But safe eviction still requires synchronous fault interception to prevent data loss races.

For anonymous guest memory (needed for KSM cross-VM deduplication), there is no mechanism at all — clearing a PTE loses the page.

Quoted 8 lines

== Solution ==

The series introduces a unified userfaultfd interface that works across both anonymous and shmem-backed memory:

UFFD_FEATURE_MINOR_ANON: extends MODE_MINOR registration to anonymous private memory. Uses the PROT_NONE hinting mechanism (same as NUMA balancing) to make pages inaccessible without freeing them.

I would rather tackle this from the other direction: it's another form of protection (like WP), not really a "minor" mode.

Could we add a UFFDIO_REGISTER_MODE_RWP (or however we would call it) and support it for anon+shmem, avoiding the zapping for shmem completely?

Signature · 3 lines
Cheers,

David
author reviewer
#19 lore raw
Quoted 29 lines · David Hildenbrand (Arm)
Quoted 19 lines · Kiryl Shutsemau (Meta)

This series adds userfaultfd support for tracking the working set of VM guest memory, enabling VMMs to identify cold pages and evict them to tiered or remote storage.

== Problem ==

VMMs managing guest memory need to:
1. Track which pages are actively used (working set detection)
2. Safely evict cold pages to slower storage
3. Fetch pages back on demand when accessed again

For shmem-backed guest memory, working set tracking partially works
today: MADV_DONTNEED zaps PTEs while pages stay in page cache, and re-access auto-resolves from cache. But safe eviction still requires synchronous fault interception to prevent data loss races.

For anonymous guest memory (needed for KSM cross-VM deduplication), there is no mechanism at all — clearing a PTE loses the page.

Quoted 8 lines

== Solution ==

The series introduces a unified userfaultfd interface that works across both anonymous and shmem-backed memory:

UFFD_FEATURE_MINOR_ANON: extends MODE_MINOR registration to anonymous private memory. Uses the PROT_NONE hinting mechanism (same as NUMA balancing) to make pages inaccessible without freeing them.

Quoted 5 lines

I would rather tackle this from the other direction: it's another form of protection (like WP), not really a "minor" mode.

Could we add a UFFDIO_REGISTER_MODE_RWP (or however we would call it) and support it for anon+shmem, avoiding the zapping for shmem completely?

I like this idea.

It should be functionally equivalent, but your interface idea fits better with the rest.

Thanks! Will give it a try.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author reviewer
#20 lore raw
Quoted 36 lines · Kiryl Shutsemau
Quoted 29 lines · David Hildenbrand (Arm)
Quoted 19 lines · Kiryl Shutsemau (Meta)

This series adds userfaultfd support for tracking the working set of VM guest memory, enabling VMMs to identify cold pages and evict them to tiered or remote storage.

== Problem ==

VMMs managing guest memory need to:
1. Track which pages are actively used (working set detection)
2. Safely evict cold pages to slower storage
3. Fetch pages back on demand when accessed again

For shmem-backed guest memory, working set tracking partially works
today: MADV_DONTNEED zaps PTEs while pages stay in page cache, and re-access auto-resolves from cache. But safe eviction still requires synchronous fault interception to prevent data loss races.

For anonymous guest memory (needed for KSM cross-VM deduplication), there is no mechanism at all — clearing a PTE loses the page.

Quoted 8 lines

== Solution ==

The series introduces a unified userfaultfd interface that works across both anonymous and shmem-backed memory:

UFFD_FEATURE_MINOR_ANON: extends MODE_MINOR registration to anonymous private memory. Uses the PROT_NONE hinting mechanism (same as NUMA balancing) to make pages inaccessible without freeing them.

Quoted 5 lines

I would rather tackle this from the other direction: it's another form of protection (like WP), not really a "minor" mode.

Could we add a UFFDIO_REGISTER_MODE_RWP (or however we would call it) and support it for anon+shmem, avoiding the zapping for shmem completely?

Quoted 6 lines

I like this idea.

It should be functionally equivalent, but your interface idea fits better with the rest.

Thanks! Will give it a try.

Here is an updated version:

https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git/log/?h=uffd/rfc-v2

will post after -rc1 is tagged.

I like it more. It got substantially cleaner.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#21 lore raw
Quoted 16 lines · Kiryl Shutsemau
Quoted 8 lines · Kiryl Shutsemau
Quoted 6 lines · David Hildenbrand (Arm)

I would rather tackle this from the other direction: it's another form of protection (like WP), not really a "minor" mode.

Could we add a UFFDIO_REGISTER_MODE_RWP (or however we would call it) and support it for anon+shmem, avoiding the zapping for shmem completely?

Quoted 6 lines

I like this idea.

It should be functionally equivalent, but your interface idea fits better with the rest.

Thanks! Will give it a try.

Quoted 7 lines

Here is an updated version:

https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git/log/?h=uffd/rfc-v2

will post after -rc1 is tagged.

I like it more. It got substantially cleaner.

I don't have time to look into the details just yet, but my thinking was that

a) It would avoid the zap+refault

b) We could reuse the uffd-wp PTE bit + marker to indicate/remember the protection, making it co-exist with NUMA hinting naturally.

b) obviously means that we cannot use uffd-wp and uffd-rwp at the same time in the same uffd area. I guess that should be acceptable for the use cases we you should have in mind?

But I also haven't taken a closer look at this patch set, whether you would already be using a PTE bit somehow (I suspect not :) )

Signature · 3 lines
Cheers,

David
author reviewer
#22 lore raw
Quoted 21 lines · David Hildenbrand (Arm)
Quoted 20 lines · Kiryl Shutsemau
Quoted 8 lines · Kiryl Shutsemau
Quoted 6 lines · David Hildenbrand (Arm)

I would rather tackle this from the other direction: it's another form of protection (like WP), not really a "minor" mode.

Could we add a UFFDIO_REGISTER_MODE_RWP (or however we would call it) and support it for anon+shmem, avoiding the zapping for shmem completely?

Quoted 6 lines

I like this idea.

It should be functionally equivalent, but your interface idea fits better with the rest.

Thanks! Will give it a try.

Here is an updated version:

https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git/log/?h=uffd/rfc-v2

Quoted 8 lines
Quoted 3 lines

will post after -rc1 is tagged.

I like it more. It got substantially cleaner.

I don't have time to look into the details just yet, but my thinking was that

a) It would avoid the zap+refault

Yep.

Quoted 6 lines

b) We could reuse the uffd-wp PTE bit + marker to indicate/remember the protection, making it co-exist with NUMA hinting naturally.

b) obviously means that we cannot use uffd-wp and uffd-rwp at the same time in the same uffd area. I guess that should be acceptable for the use cases we you should have in mind?

I took a different path: I still use PROT_NONE PTEs, so it cannot co-exist with NUMA balancing [fully], but WP + RWP should be fine. I need to add a test for this.

I didn't give up on NUMA balancing completely. task_numa_fault() is called on RWP fault. So it should help scheduler decisions somewhat.

I think an RWP user might want to use WP too.

Do you see this trade-off as reasonable?

Quoted 2 lines

But I also haven't taken a closer look at this patch set, whether you would already be using a PTE bit somehow (I suspect not :) )

No. I didn't want to allocate a new bit or invent some arch-specific trick for this. This functionality is available everywhere where PAGE_NONE exists.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author reviewer
#23 lore raw
Quoted 7 lines · Kiryl Shutsemau
Quoted 3 lines

b) obviously means that we cannot use uffd-wp and uffd-rwp at the same time in the same uffd area. I guess that should be acceptable for the use cases we you should have in mind?

I took a different path: I still use PROT_NONE PTEs, so it cannot co-exist with NUMA balancing [fully], but WP + RWP should be fine. I need to add a test for this.

WP + RWP works. Test case added.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#24 lore raw
In reply to Kiryl Shutsemau
Quoted 29 lines · Kiryl Shutsemau
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 5 lines · Kiryl Shutsemau

Here is an updated version:

https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git/log/?h=uffd/rfc-v2

Quoted 8 lines
Quoted 3 lines

will post after -rc1 is tagged.

I like it more. It got substantially cleaner.

I don't have time to look into the details just yet, but my thinking was that

a) It would avoid the zap+refault

Yep.

Quoted 6 lines

b) We could reuse the uffd-wp PTE bit + marker to indicate/remember the protection, making it co-exist with NUMA hinting naturally.

b) obviously means that we cannot use uffd-wp and uffd-rwp at the same time in the same uffd area. I guess that should be acceptable for the use cases we you should have in mind?

I took a different path: I still use PROT_NONE PTEs, so it cannot co-exist with NUMA balancing [fully], but WP + RWP should be fine. I need to add a test for this.

Quoted 6 lines

I didn't give up on NUMA balancing completely. task_numa_fault() is called on RWP fault. So it should help scheduler decisions somewhat.

I think an RWP user might want to use WP too.

Do you see this trade-off as reasonable?

One reason why the PTE bit was added for the WP case was to distinguish it from other write faults.

I assume without a dedicated PTE bit your design will always suffer from false positive notifications.

Leaving NUMA-balancing aside, a simple mprotect(PROT_NONE)+mprotect(PROT_READ) would already be problematic to distinguish both cases. Zap+refault for shmem would likely have similar problems (we'd need a marker).

I don't think a design that allows for false positives is what we really want, especially as it would diverge from what we already have for WP.

Yes, using the PTE bit (that we already have) implies that we could, for now, not allow the combination of WP + RWP.

Signature · 3 lines
Cheers,

David
author reviewer
#25 lore raw
Quoted 40 lines · David Hildenbrand (Arm)
Quoted 29 lines · Kiryl Shutsemau
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 5 lines · Kiryl Shutsemau

Here is an updated version:

https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git/log/?h=uffd/rfc-v2

Quoted 8 lines
Quoted 3 lines

will post after -rc1 is tagged.

I like it more. It got substantially cleaner.

I don't have time to look into the details just yet, but my thinking was that

a) It would avoid the zap+refault

Yep.

Quoted 6 lines

b) We could reuse the uffd-wp PTE bit + marker to indicate/remember the protection, making it co-exist with NUMA hinting naturally.

b) obviously means that we cannot use uffd-wp and uffd-rwp at the same time in the same uffd area. I guess that should be acceptable for the use cases we you should have in mind?

I took a different path: I still use PROT_NONE PTEs, so it cannot co-exist with NUMA balancing [fully], but WP + RWP should be fine. I need to add a test for this.

Quoted 6 lines

I didn't give up on NUMA balancing completely. task_numa_fault() is called on RWP fault. So it should help scheduler decisions somewhat.

I think an RWP user might want to use WP too.

Do you see this trade-off as reasonable?

One reason why the PTE bit was added for the WP case was to distinguish it from other write faults.

Quoted 6 lines

I assume without a dedicated PTE bit your design will always suffer from false positive notifications.

Leaving NUMA-balancing aside, a simple mprotect(PROT_NONE)+mprotect(PROT_READ) would already be problematic to distinguish both cases.

Hm. I didn't consider this case (miss some uffd lore). Will rework to reuse existing PTE bit.

Thanks for the feedback!

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author reviewer
#26 lore raw
Quoted 6 lines · Kiryl Shutsemau
Quoted 3 lines

Leaving NUMA-balancing aside, a simple mprotect(PROT_NONE)+mprotect(PROT_READ) would already be problematic to distinguish both cases.

Hm. I didn't consider this case (miss some uffd lore). Will rework to reuse existing PTE bit.

See https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git uffd/rfc-v3

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#27 lore raw
Quoted 10 lines · Kiryl Shutsemau
Quoted 6 lines · Kiryl Shutsemau
Quoted 3 lines

Leaving NUMA-balancing aside, a simple mprotect(PROT_NONE)+mprotect(PROT_READ) would already be problematic to distinguish both cases.

Hm. I didn't consider this case (miss some uffd lore). Will rework to reuse existing PTE bit.

See https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git uffd/rfc-v3

Quick feedback from skimming over it:

1) ARCH_SUPPORTS_PROT_NONE needs some thought, because I am pretty sure all architectures support something like mprotect(PROT_NONE), and the config option might be misleading.

So you very likely want to express different semantics here. You want to know whether pte_protnone()/pmd_protnone() works.

2) The other stuff is really just an extension of existing WP handling. I suspect we want to have some reasonable cleanups to not end up in common code with

@@ -1841,7 +1841,7 @@ static void copy_huge_non_present_pmd(
add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
- if (!userfaultfd_wp(dst_vma))
+ if (!userfaultfd_wp(dst_vma) && !userfaultfd_rwp(dst_vma))
pmd = pmd_swp_clear_uffd_wp(pmd);
set_pmd_at(dst_mm, addr, dst_pmd, pmd);

All the uffd handling should be better isolated (i.e., a single vma check?), and likely the uffd bit should be abstracted away from being called "wp" to something more generic.

Maybe it's simply a "uffd" flag which's semantics depend on the vma flags.

Maybe something like:

@@ -1841,7 +1841,7 @@ static void copy_huge_non_present_pmd(
add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR); mm_inc_nr_ptes(dst_mm); pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable); if (!userfaultfd_uses_pte_bit(dst_vma))
pmd = pmd_swp_clear_uffd(pmd);
set_pmd_at(dst_mm, addr, dst_pmd, pmd);

Not sure, needs another thought. But I think there are some decent cleanups to be had.

3) Some other stuff needs a second thought, like

mm/gup.c +2 -1
index 8e7dc2c6ee738..08fc18f1290d4 100644
@@ -695,7 +695,8 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page,
 	/* ... and a write-fault isn't required for other reasons. */
 	if (pmd_needs_soft_dirty_wp(vma, pmd))
 		return false;
-	return !userfaultfd_huge_pmd_wp(vma, pmd);
+	return !userfaultfd_huge_pmd_wp(vma, pmd) &&
+	       !userfaultfd_huge_pmd_rwp(vma, pmd);
 }
 

How can a pte be writable and prot_none at the same time? Maybe just confused AI output that you should carefully double check before sending that out officially.

4) How do we want to handle PM_UFFD_WP?

We are pretty much out of flags soon. Overloading PM_UFFD_WP means that we will not be able to easily support using a separate bit.

But our internal design will not easily allow that either, and I am not really sure we want to go down that path any time soon.

Maybe we could document this for now as "In WP VMAs, indicated WP PTEs. Otherwise, in RWP VMAs, indicates RWP.". Whenever we would allow both at the same time, we could change the semantics. User space would fail to create one with both protection types for now either way.

Signature · 3 lines
Cheers,

David
author reviewer
#28 lore raw
Quoted 15 lines · David Hildenbrand (Arm)
Quoted 10 lines · Kiryl Shutsemau
Quoted 6 lines · Kiryl Shutsemau
Quoted 3 lines

Leaving NUMA-balancing aside, a simple mprotect(PROT_NONE)+mprotect(PROT_READ) would already be problematic to distinguish both cases.

Hm. I didn't consider this case (miss some uffd lore). Will rework to reuse existing PTE bit.

See https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git uffd/rfc-v3

Quick feedback from skimming over it:

Quoted 6 lines

1) ARCH_SUPPORTS_PROT_NONE needs some thought, because I am pretty sure all architectures support something like mprotect(PROT_NONE), and the config option might be misleading.

So you very likely want to express different semantics here. You want to know whether pte_protnone()/pmd_protnone() works.

We do support mprotect(PROT_NONE) everywhere, but we don't always have a way to distinguish such entries from others without VMA in hands. Like, there are other PTEs that don't have present bit set. In my and NUMA balancing context we cannot rely on VMA, because we want to install PAGE_NONE entires into accessible VMA.

So we need two things; pte/pmd_protnone() checks and PAGE_NONE itself. The first to test PTE for PAGE_NONE, the second for pte/pmd_modify() to make the entry protnone.

Currently, generic code only use this functionality for NUMA balancing and gated by NUMA balancing config option. So I moved it under separate config option.

Do you want it to be named differently?

Quoted 30 lines

2) The other stuff is really just an extension of existing WP handling. I suspect we want to have some reasonable cleanups to not end up in common code with

(quoted diff) +1 -1
@@ -1841,7 +1841,7 @@ static void copy_huge_non_present_pmd(
 	add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
 	mm_inc_nr_ptes(dst_mm);
 	pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
-	if (!userfaultfd_wp(dst_vma))
+	if (!userfaultfd_wp(dst_vma) && !userfaultfd_rwp(dst_vma))
 		pmd = pmd_swp_clear_uffd_wp(pmd);
 	set_pmd_at(dst_mm, addr, dst_pmd, pmd);
 

All the uffd handling should be better isolated (i.e., a single vma check?), and likely the uffd bit should be abstracted away from being called "wp" to something more generic.

Maybe it's simply a "uffd" flag which's semantics depend on the vma flags.

Maybe something like:

(quoted diff) +0 -0
@@ -1841,7 +1841,7 @@ static void copy_huge_non_present_pmd(
 	add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
 	mm_inc_nr_ptes(dst_mm);
 	pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
 	if (!userfaultfd_uses_pte_bit(dst_vma))
 		pmd = pmd_swp_clear_uffd(pmd);
 	set_pmd_at(dst_mm, addr, dst_pmd, pmd);
 
Quoted 2 lines

Not sure, needs another thought. But I think there are some decent cleanups to be had.

That's fair. Maybe userfaultfd_protected() name is better for the VMA check?

And about UFFD_WP bit name. Maybe we can just drop _WP: _PAGE_UFFD_WP -> _PAGE_UFFD, pte_uffd_wp() -> pte_uffd()?

But it is a lot of changes. Can I do the bit rename as a follow up patchset?

Quoted 15 lines

3) Some other stuff needs a second thought, like

mm/gup.c +2 -1
index 8e7dc2c6ee738..08fc18f1290d4 100644
@@ -695,7 +695,8 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page,
 	/* ... and a write-fault isn't required for other reasons. */
 	if (pmd_needs_soft_dirty_wp(vma, pmd))
 		return false;
-	return !userfaultfd_huge_pmd_wp(vma, pmd);
+	return !userfaultfd_huge_pmd_wp(vma, pmd) &&
+	       !userfaultfd_huge_pmd_rwp(vma, pmd);
 }
 
Quoted 2 lines

How can a pte be writable and prot_none at the same time? Maybe just confused AI output that you should carefully double check before sending that out officially.

Note that this path is for !pmd_write() case to begin with. It serves FOLL_FORCE case. I believe this check is correct: we don't want to allow to write to such pages even with FOLL_FORCE.

But looking around, I missed gup_can_follow_protnone() modification. It has to return false for RWP.

Quoted 5 lines

4) How do we want to handle PM_UFFD_WP?

We are pretty much out of flags soon. Overloading PM_UFFD_WP means that we will not be able to easily support using a separate bit.

Quoted 7 lines

But our internal design will not easily allow that either, and I am not really sure we want to go down that path any time soon.

Maybe we could document this for now as "In WP VMAs, indicated WP PTEs. Otherwise, in RWP VMAs, indicates RWP.". Whenever we would allow both at the same time, we could change the semantics. User space would fail to create one with both protection types for now either way.

Yeah. I think about doing documentation-only update for PM_UFFD_WP for now.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
author reviewer
#29 lore raw
Quoted 22 lines · Kiryl Shutsemau
Quoted 15 lines

3) Some other stuff needs a second thought, like

mm/gup.c +2 -1
index 8e7dc2c6ee738..08fc18f1290d4 100644
@@ -695,7 +695,8 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page,
 	/* ... and a write-fault isn't required for other reasons. */
 	if (pmd_needs_soft_dirty_wp(vma, pmd))
 		return false;
-	return !userfaultfd_huge_pmd_wp(vma, pmd);
+	return !userfaultfd_huge_pmd_wp(vma, pmd) &&
+	       !userfaultfd_huge_pmd_rwp(vma, pmd);
 }
 
Quoted 2 lines

How can a pte be writable and prot_none at the same time? Maybe just confused AI output that you should carefully double check before sending that out officially.

Note that this path is for !pmd_write() case to begin with. It serves FOLL_FORCE case. I believe this check is correct: we don't want to allow to write to such pages even with FOLL_FORCE.

Quoted 2 lines

But looking around, I missed gup_can_follow_protnone() modification. It has to return false for RWP.

With gup_can_follow_protnone() fixed, the checks in can_follow_write_pmd/pte() are redundant. Will drop them.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#30 lore raw
Quoted 23 lines · Kiryl Shutsemau
Quoted 22 lines · Kiryl Shutsemau
Quoted 15 lines

3) Some other stuff needs a second thought, like

mm/gup.c +2 -1
index 8e7dc2c6ee738..08fc18f1290d4 100644
@@ -695,7 +695,8 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page,
 	/* ... and a write-fault isn't required for other reasons. */
 	if (pmd_needs_soft_dirty_wp(vma, pmd))
 		return false;
-	return !userfaultfd_huge_pmd_wp(vma, pmd);
+	return !userfaultfd_huge_pmd_wp(vma, pmd) &&
+	       !userfaultfd_huge_pmd_rwp(vma, pmd);
 }
 
Quoted 2 lines

How can a pte be writable and prot_none at the same time? Maybe just confused AI output that you should carefully double check before sending that out officially.

Note that this path is for !pmd_write() case to begin with. It serves FOLL_FORCE case. I believe this check is correct: we don't want to allow to write to such pages even with FOLL_FORCE.

Quoted 5 lines
Quoted 2 lines

But looking around, I missed gup_can_follow_protnone() modification. It has to return false for RWP.

With gup_can_follow_protnone() fixed, the checks in can_follow_write_pmd/pte() are redundant. Will drop them.

Yes, that sounds better.

Signature · 3 lines
Cheers,

David
#31 lore raw
In reply to Kiryl Shutsemau
Quoted 13 lines · Kiryl Shutsemau
Quoted 12 lines · David Hildenbrand (Arm)
Quoted 3 lines · Kiryl Shutsemau

See https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git uffd/rfc-v3

Quick feedback from skimming over it:

1) ARCH_SUPPORTS_PROT_NONE needs some thought, because I am pretty sure all architectures support something like mprotect(PROT_NONE), and the config option might be misleading.

Quoted 8 lines
Quoted 2 lines

So you very likely want to express different semantics here. You want to know whether pte_protnone()/pmd_protnone() works.

We do support mprotect(PROT_NONE) everywhere, but we don't always have a way to distinguish such entries from others without VMA in hands. Like, there are other PTEs that don't have present bit set. In my and NUMA balancing context we cannot rely on VMA, because we want to install PAGE_NONE entires into accessible VMA.

Exactly. So it's not ARCH_SUPPORTS_PROT_NONE.

Quoted 10 lines

So we need two things; pte/pmd_protnone() checks and PAGE_NONE itself. The first to test PTE for PAGE_NONE, the second for pte/pmd_modify() to make the entry protnone.

Currently, generic code only use this functionality for NUMA balancing and gated by NUMA balancing config option. So I moved it under separate config option.

Do you want it to be named differently?

Would ARCH_SUPPORTS_PXX_PROTNONE or sth. like that better describe that pte_protnone()/pmd_protnone() do what we want?

Quoted 31 lines
Quoted 31 lines

2) The other stuff is really just an extension of existing WP handling. I suspect we want to have some reasonable cleanups to not end up in common code with

(quoted diff) +1 -1
@@ -1841,7 +1841,7 @@ static void copy_huge_non_present_pmd(
 	add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
 	mm_inc_nr_ptes(dst_mm);
 	pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
-	if (!userfaultfd_wp(dst_vma))
+	if (!userfaultfd_wp(dst_vma) && !userfaultfd_rwp(dst_vma))
 		pmd = pmd_swp_clear_uffd_wp(pmd);
 	set_pmd_at(dst_mm, addr, dst_pmd, pmd);
 

All the uffd handling should be better isolated (i.e., a single vma check?), and likely the uffd bit should be abstracted away from being called "wp" to something more generic.

Maybe it's simply a "uffd" flag which's semantics depend on the vma flags.

Maybe something like:

(quoted diff) +0 -0
@@ -1841,7 +1841,7 @@ static void copy_huge_non_present_pmd(
 	add_mm_counter(dst_mm, MM_ANONPAGES, HPAGE_PMD_NR);
 	mm_inc_nr_ptes(dst_mm);
 	pgtable_trans_huge_deposit(dst_mm, dst_pmd, pgtable);
 	if (!userfaultfd_uses_pte_bit(dst_vma))
 		pmd = pmd_swp_clear_uffd(pmd);
 	set_pmd_at(dst_mm, addr, dst_pmd, pmd);
 
Quoted 5 lines
Quoted 2 lines

Not sure, needs another thought. But I think there are some decent cleanups to be had.

That's fair. Maybe userfaultfd_protected() name is better for the VMA check?

Yes, something like that could also work.

Quoted 3 lines

And about UFFD_WP bit name. Maybe we can just drop _WP: _PAGE_UFFD_WP -> _PAGE_UFFD, pte_uffd_wp() -> pte_uffd()?

Yes, I hinted at the above with pmd_swp_clear_uffd().

Quoted 3 lines

But it is a lot of changes. Can I do the bit rename as a follow up patchset?

Let's get this clean. There is no need to rush that in ;)

I suspect it's a fairly mechanical change.

Quoted 23 lines
Quoted 16 lines

3) Some other stuff needs a second thought, like

mm/gup.c +2 -1
index 8e7dc2c6ee738..08fc18f1290d4 100644
@@ -695,7 +695,8 @@ static inline bool can_follow_write_pmd(pmd_t pmd, struct page *page,
 	/* ... and a write-fault isn't required for other reasons. */
 	if (pmd_needs_soft_dirty_wp(vma, pmd))
 		return false;
-	return !userfaultfd_huge_pmd_wp(vma, pmd);
+	return !userfaultfd_huge_pmd_wp(vma, pmd) &&
+	       !userfaultfd_huge_pmd_rwp(vma, pmd);
 }
 
Quoted 2 lines

How can a pte be writable and prot_none at the same time? Maybe just confused AI output that you should carefully double check before sending that out officially.

Note that this path is for !pmd_write() case to begin with. It serves FOLL_FORCE case. I believe this check is correct: we don't want to allow to write to such pages even with FOLL_FORCE.

Quoted 2 lines

But looking around, I missed gup_can_follow_protnone() modification. It has to return false for RWP.

Right, read-permission checks come before the write-permission checks.

Quoted 9 lines
Quoted 9 lines

4) How do we want to handle PM_UFFD_WP?

We are pretty much out of flags soon. Overloading PM_UFFD_WP means that we will not be able to easily support using a separate bit.

But our internal design will not easily allow that either, and I am not really sure we want to go down that path any time soon.

Quoted 7 lines
Quoted 4 lines

Maybe we could document this for now as "In WP VMAs, indicated WP PTEs. Otherwise, in RWP VMAs, indicates RWP.". Whenever we would allow both at the same time, we could change the semantics. User space would fail to create one with both protection types for now either way.

Yeah. I think about doing documentation-only update for PM_UFFD_WP for now.

Ok, good!

Signature · 3 lines
Cheers,

David
author reviewer
#32 lore raw
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 5 lines · Kiryl Shutsemau
Quoted 4 lines · David Hildenbrand (Arm)
Quoted 3 lines · Kiryl Shutsemau

See https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git uffd/rfc-v3

RFCv4 addresses all your feedback plus more :)

https://git.kernel.org/pub/scm/linux/kernel/git/kas/linux.git uffd/rfc-v4

Still plan to post it after v7.1-rc1 (unless you want it earlier).

The patchet is pretty good shape in my eyes and will probably drop RFC tag.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
reviewer
#33 lore raw

Hello, Kiryl,

Quoted 2 lines · Kiryl Shutsemau

The patchet is pretty good shape in my eyes and will probably drop RFC tag.

I still have some high level questions not yet got answered. Do you want to answer them?

https://lore.kernel.org/all/[email protected]/

In summary, it's about:

- Whether we have explored other approaches on page hotness tracking

- Whether read protection is required for an userspace swap system (e.g. did you get time to have a look at umap?)

Thanks,

Signature · 1 line
Peter Xu
author reviewer
#34 lore raw
Quoted 10 lines · Peter Xu

Hello, Kiryl,

Quoted 2 lines · Kiryl Shutsemau

The patchet is pretty good shape in my eyes and will probably drop RFC tag.

I still have some high level questions not yet got answered. Do you want to answer them?

https://lore.kernel.org/all/[email protected]/

Sorry, reply to this got lost in my TODO list.

Quoted 3 lines

In summary, it's about:

- Whether we have explored other approaches on page hotness tracking

So, for read/write tracking we have clear_refs=1, page_idle and DAMON. Did I miss something?

clear_refs is process-wide hammer. And you can miss a hot page if it races with LRU rotation.

page_idle needs rmap. It will not scale.

DAMON is built around sampling. It is good for working set estimation, but I don't think it is directly useful for eviction decision. It can miss hot pages. LRU rotation will also loose info.

None of them gives comparable capabilities.

We also need a mechanism to atomically evict pages.

Quoted 2 lines

- Whether read protection is required for an userspace swap system (e.g. did you get time to have a look at umap?)

I looked at it briefly, so I can miss details.

IIUC, in absence of read tracking it doesn't collect hotness information at all. The eviction is based on fault-in time: the oldest faulted-in page gets evicted first. I guess it is fine if you don't care much about refault cost. Like, if your workload fits into memory completely and refaults are rare.

That's not my case.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
reviewer
#35 lore raw
Quoted 7 lines · Kiryl Shutsemau
Quoted 6 lines · Peter Xu

Hello, Kiryl,

Quoted 2 lines · Kiryl Shutsemau

The patchet is pretty good shape in my eyes and will probably drop RFC tag.

Quoted 6 lines
Quoted 4 lines

I still have some high level questions not yet got answered. Do you want to answer them?

https://lore.kernel.org/all/[email protected]/

Sorry, reply to this got lost in my TODO list.

No worries.

Quoted 5 lines
Quoted 4 lines

In summary, it's about:

- Whether we have explored other approaches on page hotness tracking

Quoted 7 lines

So, for read/write tracking we have clear_refs=1, page_idle and DAMON. Did I miss something?

clear_refs is process-wide hammer. And you can miss a hot page if it races with LRU rotation.

page_idle needs rmap. It will not scale.

Yes. If you would benefit from a per-mm page_idle, then it may apply to us too if we will be enforced to implement full-userspace swap in QEMU.

That's also why I suggested (in my previous reply) that we split the
requirement: one is for hotness tracking, the other is about read-inclusive trapping (v.s. wr-protect only traps).

Quoted 4 lines

DAMON is built around sampling. It is good for working set estimation, but I don't think it is directly useful for eviction decision. It can miss hot pages. LRU rotation will also loose info.

Exactly. If we need to collect ACCESS bit (or anything similar) for eviction accuracy pusrpose, IIUC we need per-page info, we can't estimate by sampling.

Quoted 2 lines

None of them gives comparable capabilities.

I want to see if some of your work can be generalized so we can use too, and we can also work together.

Quoted 2 lines

We also need a mechanism to atomically evict pages.

Yes, this is the 2nd question below, and btw uffd-wp can also achieve this.

Quoted 8 lines
Quoted 3 lines

- Whether read protection is required for an userspace swap system (e.g. did you get time to have a look at umap?)

I looked at it briefly, so I can miss details.

IIUC, in absence of read tracking it doesn't collect hotness information at all. The eviction is based on fault-in time: the oldest faulted-in

For example, let's imagine if we can have a per-mm idle page tracker, would it work for you to collect hotness info?

The other idea is, no matter whether we use MGLRU or legacy LRU, if we can expose a better interface to share hotness info from kernel to userspace, would it be possible?

Quoted 3 lines

page gets evicted first. I guess it is fine if you don't care much about refault cost. Like, if your workload fits into memory completely and refaults are rare.

One thing to mention is, if we have any hotness tracking facility ready above (e.g. per-mm idle page tracking) we _will_ trap read faults too; it's just that it'll be much faster (when it's hardware ACCESS bit).

So if I'm not wrong, what I am trying to discuss as a full userspace swap system will always trap read too for most of the cases.

The difference is only about that 5ms (in case of 30s+5ms example I gave in the other email). Your RW protection will also trap that 5ms, what I described won't: when a decision is made, we wr-protect the page, any read on top of it will still go through so it will trigger a refault. My point is, that 5ms missing over 30s (in reality maybe more than 30s) sampling window (which covered read accesses) isn't a major issue, and IMHO it's not a strong enough reason to include the whole RW feature.

The other thing is, as I mentioned in the other email, I still don't know how the current RW protection would work for anonymous. I don't yet think the user swapper can read the anon page with RW-protected pgtables. So far my understanding is maybe you only care about shmem so it's fine, but it'll always be great to confirm with you.

Thanks,

Quoted 6 lines

That's not my case.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
Signature · 1 line
Peter Xu
#36 lore raw
Quoted 6 lines

The other thing is, as I mentioned in the other email, I still don't know how the current RW protection would work for anonymous. I don't yet think the user swapper can read the anon page with RW-protected pgtables. So far my understanding is maybe you only care about shmem so it's fine, but it'll always be great to confirm with you.

I wonder if uffdio_move could be used for a swapper implementation instead?

If we ever have to read from a protnone page, maybe we could teach ptrace access to do it, or have something that can read from prot_none areas -- like uffdio_copy, which can write to prot-none areas.

Signature · 3 lines
Cheers,

David
reviewer
#37 lore raw
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 6 lines

The other thing is, as I mentioned in the other email, I still don't know how the current RW protection would work for anonymous. I don't yet think the user swapper can read the anon page with RW-protected pgtables. So far my understanding is maybe you only care about shmem so it's fine, but it'll always be great to confirm with you.

I wonder if uffdio_move could be used for a swapper implementation instead?

If RW is justified to be useful first, maybe.

I had a gut feeling Kirill's use case doesn't use anon at all, then if nobody needs it we can still decide to not support anon.

Quoted 4 lines

If we ever have to read from a protnone page, maybe we could teach ptrace access to do it, or have something that can read from prot_none areas -- like uffdio_copy, which can write to prot-none areas.

Somethinig like swap_access() in my proposal can also partly achieve that.

https://lore.kernel.org/all/[email protected]/

There, it was only about reading from swap so far, though. But that one might be easier to be extended to read PROT_NONE and directly put data into buffer user specified (ps: in my local tree impl I named it maccess() to pair with mincore(), but it doesn't really matter; it doesn't even need to be a syscall..).

To me, the interfacing is not a major issue. The major question I have is why RW protection can help in swap system impl when we already have uffd-wp.

So I want to make sure the use case can't be implemented by uffd-wp already. Because that's really what we might do for QEMU.

The other thing is I want to see possibilities of reusing any new kernel feature that can provide hotness. Currently idle page tracking has issue, not only about perf and rmap, but also on not working with mglru (defaults for Fedora/RHEL). If we can split the requirements and solve the hotness issue, then it'll also be very helpful.

Thanks,

Signature · 1 line
Peter Xu
author reviewer
#38 lore raw
Quoted 7 lines · Peter Xu
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 6 lines

The other thing is, as I mentioned in the other email, I still don't know how the current RW protection would work for anonymous. I don't yet think the user swapper can read the anon page with RW-protected pgtables. So far my understanding is maybe you only care about shmem so it's fine, but it'll always be great to confirm with you.

That's true. We use vhost and therefore shmem in our setup.

One idea I had about how to make atomic eviction for anon is extending process_vm_read() and process_madvise():

- Add a flag to process_vm_read() to bypass the protnone check on accessible (or only RWP?) VMAs.

- Allow process_madvise(MADV_DONTNEED) when the caller already has ptrace write access to the target.

The standing objection to remote DONTNEED has been "destructive", but process_vm_writev() already lets a ptrace-capable caller overwrite arbitrary anon with attacker-chosen content. DONTNEED is strictly weaker — it zeroes, it does not inject — so the trust model is already established.

Quoted 1 line
Quoted 1 line

I wonder if uffdio_move could be used for a swapper implementation instead?

I considered it. UFFDIO_MOVE can in principle relocate the cold folio into a staging VMA inside the VMM, which then reads it and drops it. The downside is the VMM has to maintain a second address range and serialise eviction through it. A purpose-built primitive — something like UFFDIO_EVICT that zaps the PTE and returns the folio contents (optionally to an fd for io_uring) — seems cleaner.

Quoted 6 lines

If RW is justified to be useful first, maybe.

I had a gut feeling Kirill's use case doesn't use anon at all, then if nobody needs it we can still decide to not support anon.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

If we ever have to read from a protnone page, maybe we could teach ptrace access to do it, or have something that can read from prot_none areas -- like uffdio_copy, which can write to prot-none areas.

Somethinig like swap_access() in my proposal can also partly achieve that.

https://lore.kernel.org/all/[email protected]/

A maccess()-style primitive that reads through PROT_NONE is a reasonable building block and overlaps with part of what UFFDIO_EVICT would need.

Quoted 11 lines

There, it was only about reading from swap so far, though. But that one might be easier to be extended to read PROT_NONE and directly put data into buffer user specified (ps: in my local tree impl I named it maccess() to pair with mincore(), but it doesn't really matter; it doesn't even need to be a syscall..).

To me, the interfacing is not a major issue. The major question I have is why RW protection can help in swap system impl when we already have uffd-wp.

So I want to make sure the use case can't be implemented by uffd-wp already. Because that's really what we might do for QEMU.

Race-free eviction can definitely be implemented with uffd-wp already. But not proper working set discovery.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
reviewer
#39 lore raw
Quoted 11 lines · Kiryl Shutsemau
Quoted 7 lines · Peter Xu
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 6 lines

The other thing is, as I mentioned in the other email, I still don't know how the current RW protection would work for anonymous. I don't yet think the user swapper can read the anon page with RW-protected pgtables. So far my understanding is maybe you only care about shmem so it's fine, but it'll always be great to confirm with you.

That's true. We use vhost and therefore shmem in our setup.

I see, thanks for confirming.

Side note: I believe host works for anon too since GUP works for anon, but it doesn't matter as long as we know anon isn't a must.

Quoted 16 lines

One idea I had about how to make atomic eviction for anon is extending process_vm_read() and process_madvise():

- Add a flag to process_vm_read() to bypass the protnone check on accessible (or only RWP?) VMAs.

- Allow process_madvise(MADV_DONTNEED) when the caller already has ptrace write access to the target.

The standing objection to remote DONTNEED has been "destructive", but process_vm_writev() already lets a ptrace-capable caller overwrite arbitrary anon with attacker-chosen content. DONTNEED is strictly weaker — it zeroes, it does not inject — so the trust model is already established.

Quoted 8 lines
Quoted 1 line
Quoted 1 line

I wonder if uffdio_move could be used for a swapper implementation instead?

I considered it. UFFDIO_MOVE can in principle relocate the cold folio into a staging VMA inside the VMM, which then reads it and drops it. The downside is the VMM has to maintain a second address range and serialise eviction through it. A purpose-built primitive — something like UFFDIO_EVICT that zaps the PTE and returns the folio contents (optionally to an fd for io_uring) — seems cleaner.

Right, the other thing is unnecessary overhead on the extra pgtable operations when moving to the staging VMA (e.g. tlb flush).

Quoted 25 lines
Quoted 8 lines

If RW is justified to be useful first, maybe.

I had a gut feeling Kirill's use case doesn't use anon at all, then if nobody needs it we can still decide to not support anon.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

If we ever have to read from a protnone page, maybe we could teach ptrace access to do it, or have something that can read from prot_none areas -- like uffdio_copy, which can write to prot-none areas.

Somethinig like swap_access() in my proposal can also partly achieve that.

https://lore.kernel.org/all/[email protected]/

A maccess()-style primitive that reads through PROT_NONE is a reasonable building block and overlaps with part of what UFFDIO_EVICT would need.

Quoted 6 lines

There, it was only about reading from swap so far, though. But that one might be easier to be extended to read PROT_NONE and directly put data into buffer user specified (ps: in my local tree impl I named it maccess() to pair with mincore(), but it doesn't really matter; it doesn't even need to be a syscall..).

Quoted 8 lines
Quoted 5 lines

To me, the interfacing is not a major issue. The major question I have is why RW protection can help in swap system impl when we already have uffd-wp.

So I want to make sure the use case can't be implemented by uffd-wp already. Because that's really what we might do for QEMU.

Race-free eviction can definitely be implemented with uffd-wp already. But not proper working set discovery.

Good. Then we can focus the discussion on hotness tracking with RWP and its benefits, and compare it with a pure access bit focused tracking system (as I mentioned in the other reply).

Thanks,

Signature · 1 line
Peter Xu
#40 lore raw
In reply to Peter Xu
Quoted 16 lines · Peter Xu
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 6 lines

The other thing is, as I mentioned in the other email, I still don't know how the current RW protection would work for anonymous. I don't yet think the user swapper can read the anon page with RW-protected pgtables. So far my understanding is maybe you only care about shmem so it's fine, but it'll always be great to confirm with you.

I wonder if uffdio_move could be used for a swapper implementation instead?

If RW is justified to be useful first, maybe.

I had a gut feeling Kirill's use case doesn't use anon at all, then if nobody needs it we can still decide to not support anon.

Quoted 1 line
Quoted 5 lines
Quoted 3 lines

If we ever have to read from a protnone page, maybe we could teach ptrace access to do it, or have something that can read from prot_none areas -- like uffdio_copy, which can write to prot-none areas.

Somethinig like swap_access() in my proposal can also partly achieve that.

Looks more like the hammer for the nail here: we could fault the page in just fine, while keeping it mapped prot_none and keeping the uffd-rwp pte bit set.

I was rather thinking of some uffd-specific thing that can read from a uffd-rwp protected pte without trigger uffd.

Quoted 9 lines

https://lore.kernel.org/all/[email protected]/

There, it was only about reading from swap so far, though. But that one might be easier to be extended to read PROT_NONE and directly put data into buffer user specified (ps: in my local tree impl I named it maccess() to pair with mincore(), but it doesn't really matter; it doesn't even need to be a syscall..).

Quoted 5 lines

To me, the interfacing is not a major issue. The major question I have is why RW protection can help in swap system impl when we already have uffd-wp.

So I want to make sure the use case can't be implemented by uffd-wp already. Because that's really what we might do for QEMU.

There has to be some added value indeed.

Signature · 3 lines
Cheers,

David
#41 lore raw
In reply to Peter Xu
Quoted 2 lines · On Thu, 23 Apr 2026 14:57:34 -0400 Peter Xu <[email protected]> wrote:
Quoted 1 line · Kiryl Shutsemau

On Thu, Apr 23, 2026 at 10:50:06AM -0400, Peter Xu wrote:

[...]

Quoted 1 line
Quoted 1 line
Quoted 1 line

- Whether we have explored other approaches on page hotness tracking

[...]

Quoted 7 lines
Quoted 3 lines

DAMON is built around sampling. It is good for working set estimation, but I don't think it is directly useful for eviction decision. It can miss hot pages. LRU rotation will also loose info.

Exactly. If we need to collect ACCESS bit (or anything similar) for eviction accuracy pusrpose, IIUC we need per-page info, we can't estimate by sampling.

That's a fair argument.

Nonetheless, there are some companies who use DAMON [1] for a similar eviction purpose on their products.

Also, page level accuracy issue was indeed concerns from many people. DAMON therefore provides page level DAMOS filter [2]. The idea is finding a large region of cold pages in low overhead first, then do page level access recheck on page of the region using the filter, just before doing the eviction.

DAMON-based memory tiering also uses it [3], to avoid wrongly promoting/demoting cold/hot pages in DAMON-claimed hot/cold regions. The evaluation result was not very bad, and a few more users reported positive test results.

Also, DAMON can be used for page level monitoring [5] and open to changes for users. Actually a work [6] for making DAMON-based page level monitoring more lightweight is ongoing.

I understand no one fits all and the decision is up to each user :) Nevertheless, I will be happy to help if you have any question or request for DAMON.

[1] https://cdn.amazon.science/ee/a4/41ff11374f2f865e5e24de11bd17/resource-management-in-aurora-serverless.pdf
[2] https://origin.kernel.org/doc/html/latest/mm/damon/design.html#filters
[3] https://github.com/damonitor/damo/blob/next/scripts/mem_tier.sh#L40
[4] https://www.phoronix.com/news/DAMON-Self-Tuned-Memory-Tiering
[5] https://origin.kernel.org/doc/html/latest/mm/damon/faq.html#can-i-simply-monitor-page-granularity
[6] https://lore.kernel.org/[email protected]

Thanks,
SJ

[...]

reviewer
#42 lore raw
Quoted 25 lines · SeongJae Park
Quoted 2 lines · On Thu, 23 Apr 2026 14:57:34 -0400 Peter Xu <[email protected]> wrote:
Quoted 1 line · Kiryl Shutsemau

On Thu, Apr 23, 2026 at 10:50:06AM -0400, Peter Xu wrote:

[...]

Quoted 1 line
Quoted 1 line
Quoted 1 line

- Whether we have explored other approaches on page hotness tracking

[...]

Quoted 7 lines
Quoted 3 lines

DAMON is built around sampling. It is good for working set estimation, but I don't think it is directly useful for eviction decision. It can miss hot pages. LRU rotation will also loose info.

Exactly. If we need to collect ACCESS bit (or anything similar) for eviction accuracy pusrpose, IIUC we need per-page info, we can't estimate by sampling.

That's a fair argument.

Nonetheless, there are some companies who use DAMON [1] for a similar eviction purpose on their products.

Also, page level accuracy issue was indeed concerns from many people. DAMON therefore provides page level DAMOS filter [2]. The idea is finding a large region of cold pages in low overhead first, then do page level access recheck on page of the region using the filter, just before doing the eviction.

Quoted 8 lines

DAMON-based memory tiering also uses it [3], to avoid wrongly promoting/demoting cold/hot pages in DAMON-claimed hot/cold regions. The evaluation result was not very bad, and a few more users reported positive test results.

Also, DAMON can be used for page level monitoring [5] and open to changes for users. Actually a work [6] for making DAMON-based page level monitoring more lightweight is ongoing.

Good to know that, thanks for the info, SJ. I'll add a note and try to explore all these at some point.

I recall I read a paper describing damon tracking overheads when granularity is small and when the memory scope is large (in VM's case, it can be e.g. 1TB or more). Would there be quick answer on whether this one still suffers (or maybe it was never a problem)?

Quoted 4 lines

I understand no one fits all and the decision is up to each user :) Nevertheless, I will be happy to help if you have any question or request for DAMON.

I'll definitely ask after digging more into that, thanks for the offer!

Quoted 14 lines

[1] https://cdn.amazon.science/ee/a4/41ff11374f2f865e5e24de11bd17/resource-management-in-aurora-serverless.pdf
[2] https://origin.kernel.org/doc/html/latest/mm/damon/design.html#filters
[3] https://github.com/damonitor/damo/blob/next/scripts/mem_tier.sh#L40
[4] https://www.phoronix.com/news/DAMON-Self-Tuned-Memory-Tiering
[5] https://origin.kernel.org/doc/html/latest/mm/damon/faq.html#can-i-simply-monitor-page-granularity
[6] https://lore.kernel.org/[email protected]

Thanks,
SJ

[...]

Signature · 1 line
Peter Xu
#43 lore raw
Quoted 35 lines · On Fri, 24 Apr 2026 07:55:11 -0400 Peter Xu <[email protected]> wrote:
Quoted 25 lines · SeongJae Park
Quoted 2 lines · On Thu, 23 Apr 2026 14:57:34 -0400 Peter Xu <[email protected]> wrote:
Quoted 1 line · Kiryl Shutsemau

On Thu, Apr 23, 2026 at 10:50:06AM -0400, Peter Xu wrote:

[...]

Quoted 1 line
Quoted 1 line
Quoted 1 line

- Whether we have explored other approaches on page hotness tracking

[...]

Quoted 7 lines
Quoted 3 lines

DAMON is built around sampling. It is good for working set estimation, but I don't think it is directly useful for eviction decision. It can miss hot pages. LRU rotation will also loose info.

Exactly. If we need to collect ACCESS bit (or anything similar) for eviction accuracy pusrpose, IIUC we need per-page info, we can't estimate by sampling.

That's a fair argument.

Nonetheless, there are some companies who use DAMON [1] for a similar eviction purpose on their products.

Also, page level accuracy issue was indeed concerns from many people. DAMON therefore provides page level DAMOS filter [2]. The idea is finding a large region of cold pages in low overhead first, then do page level access recheck on page of the region using the filter, just before doing the eviction.

Quoted 8 lines

DAMON-based memory tiering also uses it [3], to avoid wrongly promoting/demoting cold/hot pages in DAMON-claimed hot/cold regions. The evaluation result was not very bad, and a few more users reported positive test results.

Also, DAMON can be used for page level monitoring [5] and open to changes for users. Actually a work [6] for making DAMON-based page level monitoring more lightweight is ongoing.

Quoted 7 lines

Good to know that, thanks for the info, SJ. I'll add a note and try to explore all these at some point.

I recall I read a paper describing damon tracking overheads when granularity is small and when the memory scope is large (in VM's case, it can be e.g. 1TB or more). Would there be quick answer on whether this one still suffers (or maybe it was never a problem)?

I think that should still be same. In case of fixed granularity monitoring, the overhead is inherently proportional to the memory size. And we didn't make many effort on making the overhead lower. We have two ongoing works [1,2] for that, though.

Nonetheless, whether the overhead is too high or not would depend on the use case, I'd say. That is, if the system has hundreds of CPUs, letting DAMON occupying one CPU might be no real problem. Rather, there were users who willing to give more than one CPUs to DAMON if DAMON can provide more accurate monitoring results or work faster. That kind of scaling is possible, by using multiple kdamonds that monitors different partitions of the address ranges.

Quoted 7 lines
Quoted 5 lines

I understand no one fits all and the decision is up to each user :) Nevertheless, I will be happy to help if you have any question or request for DAMON.

I'll definitely ask after digging more into that, thanks for the offer!

The pleasure is mine! :)

Quoted 8 lines
Quoted 8 lines

[1] https://cdn.amazon.science/ee/a4/41ff11374f2f865e5e24de11bd17/resource-management-in-aurora-serverless.pdf
[2] https://origin.kernel.org/doc/html/latest/mm/damon/design.html#filters
[3] https://github.com/damonitor/damo/blob/next/scripts/mem_tier.sh#L40
[4] https://www.phoronix.com/news/DAMON-Self-Tuned-Memory-Tiering
[5] https://origin.kernel.org/doc/html/latest/mm/damon/faq.html#can-i-simply-monitor-page-granularity
[6] https://lore.kernel.org/[email protected]

[1] https://lore.kernel.org/[email protected]
[2] https://lore.kernel.org/[email protected]

Thanks,
SJ

[...]

author reviewer
#44 lore raw
In reply to Peter Xu
Quoted 9 lines · Peter Xu
Quoted 7 lines · Kiryl Shutsemau
Quoted 2 lines

- Whether read protection is required for an userspace swap system (e.g. did you get time to have a look at umap?)

I looked at it briefly, so I can miss details.

IIUC, in absence of read tracking it doesn't collect hotness information at all. The eviction is based on fault-in time: the oldest faulted-in

Quoted 6 lines

For example, let's imagine if we can have a per-mm idle page tracker, would it work for you to collect hotness info?

The other idea is, no matter whether we use MGLRU or legacy LRU, if we can expose a better interface to share hotness info from kernel to userspace, would it be possible?

I don't see how either fits our problem.

Both page_idle and the LRUs (legacy or MGLRU) track accesses on physical memory. We need visibility in the virtual address space domain.

We don't care which physical page backs a given guest address at any moment. We want to know which piece of the user's dataset is cold, and the answer has to be indifferent to kernel actions underneath: the tracking must survive migration and swap-out. RWP gives us that — the uffd-wp bit is preserved across swap PTEs and migration entries, so the "this VA was declared cold" marker stays attached to the VA. A physical-side tracker loses its state the moment the folio is freed or
replaced: a refaulted folio is a fresh object with no history.

Scaling goes the same way. Per-mm tracking of the form RWP does can scale with the working set. A physical-side tracker scales with all folios on the LRU/memcg, then needs an rmap walk per folio to map back to a VA — which is exactly the reason page_idle doesn't scale for this use case today.

There is also a cgroup-level confound: memcg hotness mixes guest memory with the VMM's own (worker threads, I/O buffers, vhost-user rings). VMA-scoped tracking is the natural unit regardless of the migration story.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
reviewer
#45 lore raw
Quoted 13 lines · Kiryl Shutsemau
Quoted 12 lines · Peter Xu
Quoted 7 lines · Kiryl Shutsemau
Quoted 2 lines

- Whether read protection is required for an userspace swap system (e.g. did you get time to have a look at umap?)

I looked at it briefly, so I can miss details.

IIUC, in absence of read tracking it doesn't collect hotness information at all. The eviction is based on fault-in time: the oldest faulted-in

For example, let's imagine if we can have a per-mm idle page tracker, would it work for you to collect hotness info?

Quoted 8 lines
Quoted 3 lines

The other idea is, no matter whether we use MGLRU or legacy LRU, if we can expose a better interface to share hotness info from kernel to userspace, would it be possible?

I don't see how either fits our problem.

Both page_idle and the LRUs (legacy or MGLRU) track accesses on physical memory. We need visibility in the virtual address space domain.

Yes they are, but ACCESS bit isn't. ACCESS bit is only about virtual mapping or any similar mapping (like EPT's access bit).

What I described with per-mm tracking (either we call it per-mm idle page tracking or using other interface) is about relying on ACCESS bit, not pgtable changes using RWP. IMHO It's more efficient and it will also achieve your goal of VA tracking.

In your case (and also ours), if you're looking for VMs running virtual machines, I think you need both pgtable's ACCESS bit and EPT-similar ACCESS bit. Here what's redundant is rmap, not ACCESS bit tracking. When both MMU and secondary MMU supports hardware access tracking, AFAIU it's faster than RWP.

Quoted 5 lines

We don't care which physical page backs a given guest address at any moment. We want to know which piece of the user's dataset is cold, and the answer has to be indifferent to kernel actions underneath: the tracking must survive migration and swap-out. RWP gives us that — the

This is exactly what we hit... that's the reason why I was trying to propose a new API to read directly from swap (swap_access) or similar.

Btw, from another perspective, I believe we could also persist ACCESS bit across migration or swap out.

For migration, see e.g. remove_migration_pte() has:

		if (!softleaf_is_migration_young(entry))
			pte = pte_mkold(pte);

For swap, it's different. Normally, if an userapp would manage page hotness, it will record the hotness within the userspace with whatever algorithm it wants. Then it will also survive host swap happening because that hotness is per-VA. It should be deduced from any hotness tracking system it previously used to sample (and it still can be idle page tracking, even if not efficient enough; when the VM page isn't mapped anywhere else, rmap is pure overhead, it doesn't introduce false positives).

Quoted 11 lines

uffd-wp bit is preserved across swap PTEs and migration entries, so the "this VA was declared cold" marker stays attached to the VA. A physical-side tracker loses its state the moment the folio is freed or
replaced: a refaulted folio is a fresh object with no history.

Scaling goes the same way. Per-mm tracking of the form RWP does can scale with the working set. A physical-side tracker scales with all folios on the LRU/memcg, then needs an rmap walk per folio to map back to a VA — which is exactly the reason page_idle doesn't scale for this use case today.

Quoted 4 lines

There is also a cgroup-level confound: memcg hotness mixes guest memory with the VMM's own (worker threads, I/O buffers, vhost-user rings). VMA-scoped tracking is the natural unit regardless of the migration story.

This kind of further proved you're using shmem and you have separate mappings.

Again, when with a per-mm idle page tracking these issue should all be gone. That per-mm idle page tracking needs to:

- Ignore rmap so it's VA based
- Still consider secondary MMUs, hence mmu young notifier needs to present
- Work based on ACCESS bit (to leverage hardware tracking accelerations), rather than relying on a kernel fault to set the access mark, which should be more efficient.

The other thing is, could you please still answer why RWP is required for swap impl in general? It's not yet mentioned in the reply.

Personally I really feel like we're looking at very similar problems. It is a great news to me, because if you can convince me on the new api it means our use case may likely also adopt the approach, vice versa.

It would be great to share the new interface no matter what it is, instead of trying to push different ones.

Thanks,

Signature · 1 line
Peter Xu
author reviewer
#46 lore raw
Quoted 5 lines · Peter Xu
Quoted 2 lines · Kiryl Shutsemau

Both page_idle and the LRUs (legacy or MGLRU) track accesses on physical memory. We need visibility in the virtual address space domain.

Yes they are, but ACCESS bit isn't.

A-bit is not a reliable signal for userspace working-set tracking because the kernel itself is a concurrent consumer. It is exactly why page_idle needs PG_young on top of the A-bit: PG_young is the "kernel ate the A-bit but the page was actually touched" escape hatch. And bringing PG_young into the picture puts us right back into physical-side tracking.

Quoted 4 lines

For migration, see e.g. remove_migration_pte() has:

		if (!softleaf_is_migration_young(entry))
			pte = pte_mkold(pte);

remove_migration_pte() only propagates young-at-unmap. It does not cover the common case: A-bit cleared by reclaim before migration started. The concurrent-consumer problem is what breaks the signal, not the migration boundary.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
reviewer
#47 lore raw
Quoted 10 lines · Kiryl Shutsemau
Quoted 5 lines · Peter Xu
Quoted 2 lines · Kiryl Shutsemau

Both page_idle and the LRUs (legacy or MGLRU) track accesses on physical memory. We need visibility in the virtual address space domain.

Yes they are, but ACCESS bit isn't.

A-bit is not a reliable signal for userspace working-set tracking because the kernel itself is a concurrent consumer. It is exactly why page_idle needs PG_young on top of the A-bit: PG_young is the "kernel

I assume you meant PG_idle. I actually don't know whether PG_young is still actively used anywhere in the current code base.

Quoted 6 lines

ate the A-bit but the page was actually touched" escape hatch. And bringing PG_young into the picture puts us right back into physical-side tracking.

Quoted 2 lines

For migration, see e.g. remove_migration_pte() has:

Quoted 7 lines
Quoted 2 lines
		if (!softleaf_is_migration_young(entry))
			pte = pte_mkold(pte);

remove_migration_pte() only propagates young-at-unmap. It does not cover the common case: A-bit cleared by reclaim before migration started. The concurrent-consumer problem is what breaks the signal, not the migration boundary.

IMHO it's a separate problem, and AFAIU it was well solved at least with old LRUs with PG_idle. It's just slightly unfortunate it doesn't yet work with MGLRU. Also, when the extra bit is in folio->flags, it only works if both the consumers are reporting per-folio, not per-mm.

I'm actually curious whether there're numbers or solid proof showing that in your case the per-folio perf is too bad already to justify a new per-mm API, like RWP. It's because currently this proposal is still so far very much about "let's implement a swap system". It really doesn't yet have a lot to prove on hotness tracking POV.

Not asking for a time-consuming test immediately, but IMHO these should really be solid clues to first justify the overhead with current rmap in production.

For us, we know the overhead in theory, but we never really measured how much.

Even if so, I don't think it's unsolvable.

I want to explore if there's something that can still be generic and work for per-mm tracking. I believe if we can have some bit in the ptes, then when mm reclaim code walks clearing ACCESS bit and sees some vma is being tracked, then instead of setting PG_idle, it can just move the access bit over to that special pte bit, and only to this vma this pte. IIUC that'll benefit from both worlds: fast HW-accelerated access bit, and no minor faults.

Would something like that worth exploring?

Signature · 1 line
Peter Xu
reviewer
#48 lore raw
Quoted 2 lines · Peter Xu

For us, we know the overhead in theory, but we never really measured how much.

I think I didn't discuss the other side of things, where hypervisor can touch some pages and mark it accessed, even if you don't want to.

IMHO we should track most of such accesses, except special cases.

One example is when the access happened because of emulated DMAs, then it should be marked alongside to the guest access: that's when you mentioned vhost, and I believe whatever vhost touched on the pages it should be marked hot even the guest didn't touch it.

The special case is really migration that I can think so.

When at this, I also want to know whether you can benefit from something like swap_access() / maccess() that I can propose when it's ready. It means we can shift some corner case accesses from hypervisor to that API then it won't cause false-positive even if rmap is involved. Then rmap will be a pure perf issue, if it will still be an issue at all.

I can also share at least my plan on our side, in case it helps to find shared goals. Basically we at least have two ways to go:

Plan A: propose maccess() then use it in qemu for migration purpose.
migration is so far the only thing we want to rule out. We want to keep DMAs access promote pages, for example. This will keep relying on kernel to do hotness tracking and evictions. IMHO the best and simplest solution so far taking both user+kernel into account.

Plan B: if above didn't work, we can implement swap in qemu. We need some
API like what you're looking for, except that the major challenge to us is MGLRU compatibility.

So IIUC if you're looking for completely flexible swap backends, then Plan A won't work for you: that'll always stick with kernel's swapfile managements and it's not always flexible enough. Then I want to look if we can share goal in plan B. But if we can find some shared goal in plan A / maccess(), then I'll also be more than willing to know. For that part I can definitely see what I can do.

Thanks,

Signature · 1 line
Peter Xu
author reviewer
#49 lore raw
In reply to Peter Xu
Quoted 29 lines · Peter Xu
Quoted 10 lines · Kiryl Shutsemau
Quoted 5 lines · Peter Xu
Quoted 2 lines · Kiryl Shutsemau

Both page_idle and the LRUs (legacy or MGLRU) track accesses on physical memory. We need visibility in the virtual address space domain.

Yes they are, but ACCESS bit isn't.

A-bit is not a reliable signal for userspace working-set tracking because the kernel itself is a concurrent consumer. It is exactly why page_idle needs PG_young on top of the A-bit: PG_young is the "kernel

I assume you meant PG_idle. I actually don't know whether PG_young is still actively used anywhere in the current code base.

Quoted 6 lines

ate the A-bit but the page was actually touched" escape hatch. And bringing PG_young into the picture puts us right back into physical-side tracking.

Quoted 2 lines

For migration, see e.g. remove_migration_pte() has:

Quoted 7 lines
Quoted 2 lines
		if (!softleaf_is_migration_young(entry))
			pte = pte_mkold(pte);

remove_migration_pte() only propagates young-at-unmap. It does not cover the common case: A-bit cleared by reclaim before migration started. The concurrent-consumer problem is what breaks the signal, not the migration boundary.

Quoted 8 lines

IMHO it's a separate problem, and AFAIU it was well solved at least with old LRUs with PG_idle. It's just slightly unfortunate it doesn't yet work with MGLRU. Also, when the extra bit is in folio->flags, it only works if both the consumers are reporting per-folio, not per-mm.

I'm actually curious whether there're numbers or solid proof showing that in your case the per-folio perf is too bad already to justify a new per-mm API, like RWP.

Fair ask, and I don't have numbers I can point to right now. But I'd flag that the case for RWP doesn't rest only on cost:

- LRU-agnostic. Per-folio approaches are bound to the current

    reclaim backend (legacy, MGLRU, whatever is next);

- Race-free against reclaim's A-bit consumption;
- Deterministic preservation across swap and migration.

Numbers would strengthen the cost story but they don't change those structural points.

Quoted 9 lines

I want to explore if there's something that can still be generic and work for per-mm tracking. I believe if we can have some bit in the ptes, then when mm reclaim code walks clearing ACCESS bit and sees some vma is being tracked, then instead of setting PG_idle, it can just move the access bit over to that special pte bit, and only to this vma this pte. IIUC that'll benefit from both worlds: fast HW-accelerated access bit, and no minor faults.

Would something like that worth exploring?

This can be interesting. But a spare pte bit is high ask.

And when you start tracking, you need to clear A-bit. Where do you move it? Activate folio?

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov
#50 lore raw
In reply to Kiryl Shutsemau
Quoted 9 lines · Kiryl Shutsemau
Quoted 5 lines · Peter Xu
Quoted 2 lines · Kiryl Shutsemau

Both page_idle and the LRUs (legacy or MGLRU) track accesses on physical memory. We need visibility in the virtual address space domain.

Yes they are, but ACCESS bit isn't.

A-bit is not a reliable signal for userspace working-set tracking because the kernel itself is a concurrent consumer.

Right, I don't think we want to rely on either the A bit just like we don't want to rely on the Dirty bit in other code. (and even SoftDirty bit is a flawed concept)

I do see some value in a reliable RWP mechanism based on uffd. The real question is, how much benefit it would bring (which other use cases could benefit from it).

Signature · 3 lines
Cheers,

David
author reviewer
#51 lore raw
Quoted 8 lines · David Hildenbrand (Arm)
Quoted 7 lines · Kiryl Shutsemau
Quoted 5 lines · Peter Xu
Quoted 2 lines · Kiryl Shutsemau

Both page_idle and the LRUs (legacy or MGLRU) track accesses on physical memory. We need visibility in the virtual address space domain.

Yes they are, but ACCESS bit isn't.

Quoted 7 lines
Quoted 2 lines

A-bit is not a reliable signal for userspace working-set tracking because the kernel itself is a concurrent consumer.

Right, I don't think we want to rely on either the A bit just like we don't want to rely on the Dirty bit in other code. (and even SoftDirty bit is a flawed concept)

I do see some value in a reliable RWP mechanism based on uffd. The real question is, how much benefit it would bring (which other use cases could benefit from it).

I have not put much thought into use-cases beyond VMs, but I think other mmap-heavy loads (databases, GC'd runtimes, etc) can make a use in a similar way: workset tracking and organizing userspace-driven memory tiering based on the workset data.

Signature · 1 line
  Kiryl Shutsemau / Kirill A. Shutemov