Skip to contentSkip to conversation

introduce PGTY_mgt_entry page_type

RFC PATCH In review Huan Yang 4 participants 25 messages Jul 24, 2025 – Jul 25, 2025

Subsystems (5)
lore
Share & tools
lore

Huan Yang

author
#0 lore raw

cover introduce PGTY_mgt_entry page_type

Summary
==
This patchset reuses page_type to store migrate entry count during the period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to early termination when folio is unmapped in try_to_migrate.

In my self-constructed test scenario, the migration time can be reduced from over 150+ms to around 30+ms, achieving nearly a 70% performance improvement. Additionally, the flame graph shows that the proportion of remove_migration_ptes can be reduced from 80%+ to 60%+.

Notice: migrate entry specifically refers to migrate PTE entry, as large folio are not supported page type and 0 mapcount reuse.

Principle
==
When a page removes all PTEs in try_to_migrate and sets up a migrate PTE entry, we can determine whether the traversal of remaining VMAs can be terminated early by checking if mapcount is zero. This optimization helps improve performance during migration.

However, when removing migrate PTE entries and setting up PTEs for the destination folio in remove_migration_ptes, there is no such information available to assist in deciding whether the traversal of remaining VMAs can be ended early. Therefore, it is necessary to traversal all VMAs associated with this folio.

In reality, when a folio is fully unmapped and before all migrate PTE entries are removed, the mapcount will always be zero. Since page_type and mapcount share a union, and referring to folio_mapcount, we can reuse page_type to record the number of migrate PTE entries of the current folio in the system as long as it's not a large folio. This reuse does not affect calls to folio_mapcount, which will always return zero.

Therefore, we can set the folio's page_type to PGTY_mgt_entry when try_to_migrate completes, the folio is already unmapped, and it's not a large folio. The remaining 24 bits can then be used to record the number of migrate PTE entries generated by try_to_migrate.

Then, in remove_migration_ptes, when the nr_mgt_entry count drops to zero, we can terminate the VMA traversal early.

It's important to note that we need to initialize the folio's page_type to PGTY_mgt_entry and set the migrate entry count only while holding the rmap walk lock.This is because during the lock period, we can prevent new VMA fork (which would increase migrate entries) and VMA unmap (which would decrease migrate entries).

However, I doubt there is actually an additional critical section here, for example anon:

Process Parent fork
try_to_migrate
anon_vma_clone
write_lock
avc_inster_tree tail
....
folio_lock_anon_vma_read copy_pte_range
vma_iter pte_lock
.... pte_present copy
...
pte_lock
new forked pte clean
....
remove_migration_ptes
rmap_walk_anon_lock

If my understanding is correct and such a critical section exists, it shouldn't cause any issues—newly added PTEs can still be properly removed and converted into migrate entries.

But in this:

Process Parent fork
try_to_migrate

                                        anon_vma_clone
                                            write_lock
                                                avc_inster_tree
                                        ....
    folio_lock_anon_vma_read             copy_pte_range
        vma_iter
                pte_lock
                    migrate entry set
                ....                        pte_lock
                                                pte_nonpresent copy
                                            ....

....
remove_migration_ptes
rmap_walk_anon_lock

If the parent process first acquires the pte_lock to set a migrate entry, the child process will then directly copy the non-present migrate entry, resulting in an increase in migrate entries. However, since the newly added VMA is positioned later in the rb tree of the folio's anon_vma, when we traverse to this child-process-added migrate entry, the count of migrate entries will still be correctly recorded, and this will not cause any issues.

If I misunderstand, please correct me. :)

After a folio exits try_to_migrate and before remove_migration_ptes acquires the rmap lock, the system can perform normal fork and unmap operations. Therefore, we need to increment or decrement the migrate entry count recorded in the folio (if it's of type PGTY_mgt_entry) when handling copy/zap_nonpresent_pte.

When performing remove_migration_ptes during migration to start removing migrate entries, we need to dynamically decrement the recorded migrate entry count. Once this count reaches zero, it indicates there are no remaining migrate entries in the associated VMAs that need to be cleared and replaced with the destination PFN. This allows us to safely terminate the VMA traversal early.

However, it's important to note that if issues occur during migration requiring an undo operation, PGTY_mgt_entry can no longer be used. This is because the dst needs to be set back to the src, and the presence of PGTY_mgt_entry would interfere with the normal usage of mapcount when setup rmap info.

Test
==
I set up a 2-node test environment using QEMU, and used mbind to trigger page migration between nodes for the specified VMA.

The core idea of the test scenario is to create a situation where the number of VMAs that need to be itered in the anon_vma is significantly larger than the folio's mapcount.

To achieve this, I constructed an exaggerated scenario: the parent process allocates 5MB of memory and binds it to node0, then immediately forks 1000 child processes. Each child process runs and immediately memset all this memory to complete COW-ed. Afterwards, the parent process calls mbind to migrate the memory from node0 to node1, while recording the time consumed during this period.
Additionally, perf is used to capture a flame graph during the mbind execution.

The time cost results are as follows:
Patch1-9 Normal(f817b6d)
18ms 197ms
58ms 152ms
40ms 120ms

The hot path show in fireflame:

    Patch1-9
      move_to_new_folio        38.89%
      remove_migration_ptes    61.11%
      ---------------------
      move_to_new_folio        32.76%
      remove_migration_ptes    67.24%
      ---------------------
      move_to_new_folio        37.50%
      remove_migration_ptes    62.50%

    Normal(f817b6d)
      move_to_new_folio        11.43%
      remove_migration_ptes    87.43%
      ---------------------
      move_to_new_folio        13.91%
      remove_migration_ptes    86.09%
      ---------------------
      move_to_new_folio        12.50%
      remove_migration_ptes    85.83%

Can easy see that cost time optimized by approximately 75.3%. And the proportion of the remove_migration_ptes function path has decreased by approximately 20%.

Simplify Test Code:

```c
#define size (5 << 20)
#define CHILD_COUNT 1000

int *buffer = (int *)mmap(NULL, size, PROT_READ | PROT_WRITE,
                        MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);

unsigned long mask = 1UL << 0;
mbind(buffer, size, MPOL_BIND, &mask, 2, 0);
// let all page-faulted in node 0
memset(buffer, 0, size);

// fork child.
pid_t children[CHILD_COUNT];
for (int i = 0; i < CHILD_COUNT; i++) {
    pid_t pid = fork();
    if (pid == 0) {
        // let all child process COW-ed
        memset(buffer, 0, size);
        sleep(100000);
    } else {
        children[i] = pid;
    }
}

// maybe you need sleep to wait child process COW-ed
sleep(10);


// You can use perf watch here
mask = 1UL << 1;
// migrate this buffer from node 0 -> node 1
mbind(buffer, size, MPOL_BIND, &mask, 4, MPOL_MF_MOVE);

```

Notice: this code removed many error assert and resource clean action, time record ...

Why RFC
==
Memory migration is one of the most general-purpose modules. My own tests cannot cover all system scenarios, and there may be omissions or misunderstandings in the code modifications.

If good enough, I will send the formal patch.

Patch 1-7 do some code clean work.
Patch 8 prepare for PGTY_mgt_entry.
Patch 9 apply it.

Huan Yang (9):
mm: introduce PAGE_TYPE_SHIFT
mm: add page_type value helper
mm/rmap: simplify rmap_walk invoke
mm/rmap: add args in rmap_walk_control done hook mm/rmap: introduce exit hook
mm/rmap: introduce migrate_walk_arg mm/migrate: rename rmap_walk_arg folio mm/migrate: infrastructure for migrate entry page_type. mm/migrate: apply migrate entry page_type

 include/linux/page-flags.h | 106 +++++++++++++++++++++++++++++++++++--
 include/linux/rmap.h       |   7 ++-
 mm/ksm.c                   |   2 +-
 mm/memory.c                |   2 +
 mm/migrate.c               |  38 ++++++++++---
 mm/rmap.c                  |  85 ++++++++++++++++++-----------
 6 files changed, 193 insertions(+), 47 deletions(-)

--
2.34.1

author
#1 lore raw

RFC PATCH 1/9 mm: introduce PAGE_TYPE_SHIFT

The current shift value for page_type is 24. To avoid hardcode, define a macro called PAGE_TYPE_SHIFT and PAGE_TYPE_MASK.

No functional change.

---
 include/linux/page-flags.h | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
include/linux/page-flags.h +12 -5
index 4fe5ee67535b..3c7103c2eee4 100644
@@ -961,9 +961,12 @@ enum pagetype {
 	PGTY_mapcount_underflow = 0xff
 };
 
+#define PAGE_TYPE_SHIFT		24
+#define PAGE_TYPE_MASK		((1 << PAGE_TYPE_SHIFT) - 1)
+
 static inline bool page_type_has_type(int page_type)
 {
-	return page_type < (PGTY_mapcount_underflow << 24);
+	return page_type < (PGTY_mapcount_underflow << PAGE_TYPE_SHIFT);
 }
 
 /* This takes a mapcount which is one more than page->_mapcount */
@@ -980,7 +983,8 @@ static inline bool page_has_type(const struct page *page)
 #define FOLIO_TYPE_OPS(lname, fname)					\
 static __always_inline bool folio_test_##fname(const struct folio *folio) \
 {									\
-	return data_race(folio->page.page_type >> 24) == PGTY_##lname;	\
+	return data_race(folio->page.page_type >> PAGE_TYPE_SHIFT)      \
+	       == PGTY_##lname;						\
 }									\
 static __always_inline void __folio_set_##fname(struct folio *folio)	\
 {									\
@@ -988,7 +992,8 @@ static __always_inline void __folio_set_##fname(struct folio *folio)	\
 		return;							\
 	VM_BUG_ON_FOLIO(data_race(folio->page.page_type) != UINT_MAX,	\
 			folio);						\
-	folio->page.page_type = (unsigned int)PGTY_##lname << 24;	\
+	folio->page.page_type = (unsigned int)PGTY_##lname		\
+					      << PAGE_TYPE_SHIFT;	\
 }									\
 static __always_inline void __folio_clear_##fname(struct folio *folio)	\
 {									\
@@ -1002,14 +1007,16 @@ static __always_inline void __folio_clear_##fname(struct folio *folio)	\
 FOLIO_TYPE_OPS(lname, fname)						\
 static __always_inline int Page##uname(const struct page *page)		\
 {									\
-	return data_race(page->page_type >> 24) == PGTY_##lname;	\
+	return data_race(page->page_type >> PAGE_TYPE_SHIFT)		\
+	       == PGTY_##lname;						\
 }									\
 static __always_inline void __SetPage##uname(struct page *page)		\
 {									\
 	if (Page##uname(page))						\
 		return;							\
 	VM_BUG_ON_PAGE(data_race(page->page_type) != UINT_MAX, page);	\
-	page->page_type = (unsigned int)PGTY_##lname << 24;		\
+	page->page_type = (unsigned int)PGTY_##lname			\
+					<< PAGE_TYPE_SHIFT;		\
 }									\
 static __always_inline void __ClearPage##uname(struct page *page)	\
 {									\
Signature · 1 line
2.34.1
author
#2 lore raw

RFC PATCH 2/9 mm: add page_type value helper

In reply to Huan Yang

Add two helper functions __SetPageXXXValue and __GetPageXXXValue to assist in assigning values to the specified page_type. Since the current page_type value is not supported on large folio, only the page version of the helper functions is provided.

---
 include/linux/page-flags.h | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
include/linux/page-flags.h +17 -0
index 3c7103c2eee4..52c9435079d5 100644
@@ -1024,6 +1024,23 @@ static __always_inline void __ClearPage##uname(struct page *page)	\
 		return;							\
 	VM_BUG_ON_PAGE(!Page##uname(page), page);			\
 	page->page_type = UINT_MAX;					\
+}									\
+static __always_inline void __PageSet##uname##Value(struct page *page,	\
+						    unsigned int value) \
+{									\
+	if (!Page##uname(page))						\
+		return;							\
+	if (unlikely(value > (PAGE_TYPE_MASK)))				\
+		return;							\
+	WRITE_ONCE(page->page_type, (unsigned int)PGTY_##lname		\
+				    << PAGE_TYPE_SHIFT | value);	\
+}									\
+static __always_inline unsigned int __PageGet##uname##Value(		\
+				    struct page *page)			\
+{									\
+	if (!Page##uname(page))						\
+		return 0;						\
+	return READ_ONCE(page->page_type) & PAGE_TYPE_MASK;		\
 }
 
 /*
Signature · 1 line
2.34.1
author
#3 lore raw

RFC PATCH 3/9 mm/rmap: simplify rmap_walk invoke

In reply to Huan Yang

Currently, the rmap walk is split into two functions: rmap_walk_locked and rmap_walk, but their implementation functionality is very similar.

This patch simplifies the rmap walk function and moves the locked parameter to rmap walk control.

No functional change.

---
 include/linux/rmap.h |  3 ++-
 mm/migrate.c         |  6 ++----
 mm/rmap.c            | 43 ++++++++++++++++---------------------------
 3 files changed, 20 insertions(+), 32 deletions(-)
include/linux/rmap.h +2 -1
index 45904ff413ab..f0d17c971a20 100644
@@ -996,6 +996,7 @@ void remove_migration_ptes(struct folio *src, struct folio *dst, int flags);
  * arg: passed to rmap_one() and invalid_vma()
  * try_lock: bail out if the rmap lock is contended
  * contended: indicate the rmap traversal bailed out due to lock contention
+ * locked: already locked before invoke rmap_walk
  * rmap_one: executed on each vma where page is mapped
  * done: for checking traversing termination condition
  * anon_lock: for getting anon_lock by optimized way rather than default
@@ -1005,6 +1006,7 @@ struct rmap_walk_control {
 	void *arg;
 	bool try_lock;
 	bool contended;
+	bool locked;
 	/*
 	 * Return false if page table scanning in rmap_walk should be stopped.
 	 * Otherwise, return true.
@@ -1018,7 +1020,6 @@ struct rmap_walk_control {
 };
 
 void rmap_walk(struct folio *folio, struct rmap_walk_control *rwc);
-void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc);
 struct anon_vma *folio_lock_anon_vma_read(const struct folio *folio,
 					  struct rmap_walk_control *rwc);
 
mm/migrate.c +2 -4
index 8cf0f9c9599d..a5a49af7857a 100644
@@ -355,15 +355,13 @@ void remove_migration_ptes(struct folio *src, struct folio *dst, int flags)
 
 	struct rmap_walk_control rwc = {
 		.rmap_one = remove_migration_pte,
+		.locked = flags & RMP_LOCKED,
 		.arg = &rmap_walk_arg,
 	};
 
 	VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src != dst), src);
 
-	if (flags & RMP_LOCKED)
-		rmap_walk_locked(dst, &rwc);
-	else
-		rmap_walk(dst, &rwc);
+	rmap_walk(dst, &rwc);
 }
 
 /*
mm/rmap.c +16 -27

100 lines not on this page · raw message

Signature · 1 line
2.34.1
author
#4 lore raw

RFC PATCH 4/9 mm/rmap: add args in rmap_walk_control done hook

In reply to Huan Yang

In the done hook that determines whether rmap_walk can terminate the traversal early, we may need to read parameters from rmap_walk_control to assist in this decision.

This patch adds the rmap_walk_control as a parameter to the done hook.

---
 include/linux/rmap.h | 2 +-
 mm/ksm.c             | 2 +-
 mm/rmap.c            | 6 +++---
 3 files changed, 5 insertions(+), 5 deletions(-)
include/linux/rmap.h +1 -1
index f0d17c971a20..a305811d6310 100644
@@ -1013,7 +1013,7 @@ struct rmap_walk_control {
 	 */
 	bool (*rmap_one)(struct folio *folio, struct vm_area_struct *vma,
 					unsigned long addr, void *arg);
-	int (*done)(struct folio *folio);
+	int (*done)(struct folio *folio, struct rmap_walk_control *rwc);
 	struct anon_vma *(*anon_lock)(const struct folio *folio,
 				      struct rmap_walk_control *rwc);
 	bool (*invalid_vma)(struct vm_area_struct *vma, void *arg);
mm/ksm.c +1 -1
index ef73b25fd65a..635fe402af91 100644
@@ -3072,7 +3072,7 @@ void rmap_walk_ksm(struct folio *folio, struct rmap_walk_control *rwc)
 				anon_vma_unlock_read(anon_vma);
 				return;
 			}
-			if (rwc->done && rwc->done(folio)) {
+			if (rwc->done && rwc->done(folio, rwc)) {
 				anon_vma_unlock_read(anon_vma);
 				return;
 			}
mm/rmap.c +3 -3
index bae9f79c7dc9..e590f4071eca 100644
@@ -2232,7 +2232,7 @@ static bool invalid_migration_vma(struct vm_area_struct *vma, void *arg)
 	return vma_is_temporary_stack(vma);
 }
 
-static int folio_not_mapped(struct folio *folio)
+static int folio_not_mapped(struct folio *folio, struct rmap_walk_control *rwc)
 {
 	return !folio_mapped(folio);
 }
@@ -2828,7 +2828,7 @@ static void rmap_walk_anon(struct folio *folio, struct rmap_walk_control *rwc)
 
 		if (!rwc->rmap_one(folio, vma, address, rwc->arg))
 			break;
-		if (rwc->done && rwc->done(folio))
+		if (rwc->done && rwc->done(folio, rwc))
 			break;
 	}
 
@@ -2891,7 +2891,7 @@ static void __rmap_walk_file(struct folio *folio, struct address_space *mapping,
 
 		if (!rwc->rmap_one(folio, vma, address, rwc->arg))
 			goto done;
-		if (rwc->done && rwc->done(folio))
+		if (rwc->done && rwc->done(folio, rwc))
 			goto done;
 	}
 done:
Signature · 1 line
2.34.1
author
#5 lore raw

RFC PATCH 5/9 mm/rmap: introduce exit hook

In reply to Huan Yang

When approaching the end of an rmap_walk traversal, we may need to perform some resource statistics or cleanup work, and these operations need to be done under lock.

This patch adds a new exit hook to rmap_walk_control.

Note that currently, the exit hook only be used in the anon and file rmap_walk implementations, as ksm no need it.

---
 include/linux/rmap.h | 2 ++
 mm/rmap.c            | 6 ++++++
 2 files changed, 8 insertions(+)
include/linux/rmap.h +2 -0
index a305811d6310..594dc4df3bfb 100644
@@ -999,6 +999,7 @@ void remove_migration_ptes(struct folio *src, struct folio *dst, int flags);
  * locked: already locked before invoke rmap_walk
  * rmap_one: executed on each vma where page is mapped
  * done: for checking traversing termination condition
+ * exit: do some clean work below lock before leave rmap_walk
  * anon_lock: for getting anon_lock by optimized way rather than default
  * invalid_vma: for skipping uninterested vma
  */
@@ -1014,6 +1015,7 @@ struct rmap_walk_control {
 	bool (*rmap_one)(struct folio *folio, struct vm_area_struct *vma,
 					unsigned long addr, void *arg);
 	int (*done)(struct folio *folio, struct rmap_walk_control *rwc);
+	void (*exit)(struct folio *folio, struct rmap_walk_control *rwc);
 	struct anon_vma *(*anon_lock)(const struct folio *folio,
 				      struct rmap_walk_control *rwc);
 	bool (*invalid_vma)(struct vm_area_struct *vma, void *arg);
mm/rmap.c +6 -0
index e590f4071eca..66b48ab192f5 100644
@@ -2832,6 +2832,9 @@ static void rmap_walk_anon(struct folio *folio, struct rmap_walk_control *rwc)
 			break;
 	}
 
+	if (rwc->exit)
+		rwc->exit(folio, rwc);
+
 	if (!locked)
 		anon_vma_unlock_read(anon_vma);
 }
@@ -2895,6 +2898,9 @@ static void __rmap_walk_file(struct folio *folio, struct address_space *mapping,
 			goto done;
 	}
 done:
+	if (rwc->exit)
+		rwc->exit(folio, rwc);
+
 	if (!locked)
 		i_mmap_unlock_read(mapping);
 }
Signature · 1 line
2.34.1
author
#6 lore raw

RFC PATCH 6/9 mm/rmap: introduce migrate_walk_arg

In reply to Huan Yang

In try_to_migrate, rmap_one as well as the done and exit hooks may require more information from try_to_migrate to assist in performing migration-related operations.

This patch introduces a new migrate_walk_arg structure to serve as the arg parameter for rmap_walk_control in try_to_migrate.

---
 mm/rmap.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
mm/rmap.c +11 -2
index 66b48ab192f5..2433e12c131d 100644
@@ -2261,6 +2261,10 @@ void try_to_unmap(struct folio *folio, enum ttu_flags flags)
 	rmap_walk(folio, &rwc);
 }
 
+struct migrate_walk_arg {
+	enum ttu_flags flags;
+};
+
 /*
  * @arg: enum ttu_flags will be passed to this argument.
  *
@@ -2276,7 +2280,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 	pte_t pteval;
 	struct page *subpage;
 	struct mmu_notifier_range range;
-	enum ttu_flags flags = (enum ttu_flags)(long)arg;
+	struct migrate_walk_arg *mwa = (struct migrate_walk_arg *)arg;
+	enum ttu_flags flags = mwa->flags;
 	unsigned long pfn;
 	unsigned long hsz = 0;
 
@@ -2575,9 +2580,13 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
  */
 void try_to_migrate(struct folio *folio, enum ttu_flags flags)
 {
+	struct migrate_walk_arg arg = {
+		.flags = flags,
+	};
+
 	struct rmap_walk_control rwc = {
 		.rmap_one = try_to_migrate_one,
-		.arg = (void *)flags,
+		.arg = (void *)&arg,
 		.done = folio_not_mapped,
 		.locked = flags & TTU_RMAP_LOCKED,
 		.anon_lock = folio_lock_anon_vma_read,
Signature · 1 line
2.34.1
author
#7 lore raw

RFC PATCH 7/9 mm/migrate: rename rmap_walk_arg folio

In reply to Huan Yang

The current naming of "folio" in rmap_walk_arg does not clearly reflect the actual role of this parameter. This patch renames it to "src" to distinguish it from the "folio" parameter in remove_migration_pte.

No functional change.

---
 mm/migrate.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
mm/migrate.c +3 -3
index a5a49af7857a..a5ea8fba2997 100644
@@ -230,7 +230,7 @@ static bool try_to_map_unused_to_zeropage(struct page_vma_mapped_walk *pvmw,
 }
 
 struct rmap_walk_arg {
-	struct folio *folio;
+	struct folio *src;
 	bool map_unused_to_zeropage;
 };
 
@@ -241,7 +241,7 @@ static bool remove_migration_pte(struct folio *folio,
 		struct vm_area_struct *vma, unsigned long addr, void *arg)
 {
 	struct rmap_walk_arg *rmap_walk_arg = arg;
-	DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->folio, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
+	DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->src, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
 
 	while (page_vma_mapped_walk(&pvmw)) {
 		rmap_t rmap_flags = RMAP_NONE;
@@ -349,7 +349,7 @@ static bool remove_migration_pte(struct folio *folio,
 void remove_migration_ptes(struct folio *src, struct folio *dst, int flags)
 {
 	struct rmap_walk_arg rmap_walk_arg = {
-		.folio = src,
+		.src = src,
 		.map_unused_to_zeropage = flags & RMP_USE_SHARED_ZEROPAGE,
 	};
 
Signature · 1 line
2.34.1
author
#8 lore raw

RFC PATCH 8/9 mm/migrate: infrastructure for migrate entry page_type.

In reply to Huan Yang

When a page removes all PTEs in try_to_migrate and sets up a migrate PTE entry, we can determine whether the traversal of remaining VMAs can be terminated early by checking if mapcount is zero. This optimization helps improve performance during migration.

However, when removing migrate PTE entries and setting up PTEs for the destination folio in remove_migration_ptes, there is no such information available to assist in deciding whether the traversal of remaining VMAs can be ended early. Therefore, it is necessary to traversal all VMAs associated with this folio.

In reality, when a folio is fully unmapped and before all migrate PTE entries are removed, the mapcount will always be zero. Since page_type and mapcount share a union, and referring to folio_mapcount, we can reuse page_type to record the number of migrate PTE entries of the current folio in the system as long as it's not a large folio. This reuse does not affect calls to folio_mapcount, which will always return zero.

Therefore, we can set the folio's page_type to PGTY_mgt_entry when try_to_migrate completes, the folio is already unmapped, and it's not a large folio. The remaining 24 bits can then be used to record the number of migrate PTE entries generated by try_to_migrate.

Then, in remove_migration_ptes, when the nr_mgt_entry count drops to zero, we can terminate the VMA traversal early.

This patch solely completes the infrastructure setup for PGTY_mgt_entry, with no actual usage implemented, the next patch will do it.

---
 include/linux/page-flags.h | 72 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)
include/linux/page-flags.h +72 -0
index 52c9435079d5..d8e80d5ae1f8 100644
@@ -957,6 +957,7 @@ enum pagetype {
 	PGTY_zsmalloc		= 0xf6,
 	PGTY_unaccepted		= 0xf7,
 	PGTY_large_kmalloc	= 0xf8,
+	PGTY_mgt_entry		= 0xf9,
 
 	PGTY_mapcount_underflow = 0xff
 };
@@ -1125,6 +1126,77 @@ PAGE_TYPE_OPS(Zsmalloc, zsmalloc, zsmalloc)
 PAGE_TYPE_OPS(Unaccepted, unaccepted, unaccepted)
 FOLIO_TYPE_OPS(large_kmalloc, large_kmalloc)
 
+
+PAGE_TYPE_OPS(Mentry, mgt_entry, mgt_entry)
+
+#define PAGE_MGT_ENTRY_TYPE_MAX		PAGE_TYPE_MASK
+
+static inline void folio_remove_mgte(struct folio *folio)
+{
+	if (!folio_test_mgt_entry(folio))
+		return;
+
+	__folio_clear_mgt_entry(folio);
+}
+
+static inline void folio_init_mgte(struct folio *folio, int nr_mgt_entry)
+{
+	// PAGE_TYPE value currently do not support large folio.
+	VM_BUG_ON(folio_test_large(folio));
+	VM_BUG_ON(folio_test_mgt_entry(folio));
+
+	if (unlikely(nr_mgt_entry > PAGE_MGT_ENTRY_TYPE_MAX))
+		return;
+
+	__folio_set_mgt_entry(folio);
+	__PageSetMentryValue(&folio->page, nr_mgt_entry);
+}
+
+static inline int folio_get_mgte_count(struct folio *folio)
+{
+	if (!folio_test_mgt_entry(folio))
+		return 0;
+
+	VM_BUG_ON(folio_test_large(folio));
+
+	return __PageGetMentryValue(&folio->page);
+}
+
+static inline void folio_dec_mgte_count(struct folio *folio)
+{
+	unsigned int nr_mgte, old;
+
+	do {
+		old = READ_ONCE(folio->page.page_type);
+
+		if ((old >> PAGE_TYPE_SHIFT) != PGTY_mgt_entry)
+			return;
+
+		nr_mgte = old & PAGE_TYPE_MASK;
+		BUG_ON(nr_mgte == 0);
+	} while (cmpxchg(&folio->page.page_type, old, old - 1) != old);
+}
+
+static inline void folio_inc_mgte_count(struct folio *folio)
+{
+	unsigned int nr_mgte, old;
+
+	do {
+		old = READ_ONCE(folio->page.page_type);
+
+		if ((old >> PAGE_TYPE_SHIFT) != PGTY_mgt_entry)
+			return;
+
+		nr_mgte = old & PAGE_TYPE_MASK;
+
+		if (unlikely(nr_mgte == PAGE_MGT_ENTRY_TYPE_MAX)) {
+			// overflow, can't reuse PAGE_TYPE here.
+			folio_remove_mgte(folio);
+			break;
+		}
+	} while (cmpxchg(&folio->page.page_type, old, old + 1) != old);
+}
+
 /**
  * PageHuge - Determine if the page belongs to hugetlbfs
  * @page: The page to test.
Signature · 1 line
2.34.1
author
#9 lore raw

RFC PATCH 9/9 mm/migrate: apply migrate entry page_type

In reply to Huan Yang

When a single-page folio is already unmapped, we can reuse the page_type field by setting it to PGTY_mgt_entry.This indicates that the folio is in a critical state where no PFNs for this folio exist in the system, with migrate entries used instead.

The lower 24 bits of this field can be utilized to store the count of migrate entries during try_to_migrate.

It's important to note that we need to initialize the folio's page_type to PGTY_mgt_entry and set the migrate entry count only while holding the rmap walk lock.This is because during the lock period, we can prevent new VMA fork (which would increase migrate entries) and VMA unmap (which would decrease migrate entries).

After a folio exits try_to_migrate and before remove_migration_ptes acquires the rmap lock, the system can perform normal fork and unmap operations. Therefore, we need to increment or decrement the migrate entry count recorded in the folio (if it's of type PGTY_mgt_entry) when handling copy/zap_nonpresent_pte.

When performing remove_migration_ptes during migration to start removing migrate entries, we need to dynamically decrement the recorded migrate entry count. Once this count reaches zero, it indicates there are no remaining migrate entries in the associated VMAs that need to be cleared and replaced with the destination PFN. This allows us to safely terminate the VMA traversal early.

However, it's important to note that if issues occur during migration requiring an undo operation, PGTY_mgt_entry can no longer be used. This is because the dst needs to be set back to the src, and the presence of PGTY_mgt_entry would interfere with the normal usage of mapcount when setup rmap info.

---
 mm/memory.c  |  2 ++
 mm/migrate.c | 28 +++++++++++++++++++++++++++-
 mm/rmap.c    | 17 +++++++++++++++++
 3 files changed, 46 insertions(+), 1 deletion(-)
mm/memory.c +2 -0
index b4a7695b1e31..f9d71b118c11 100644
@@ -861,6 +861,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 				pte = pte_swp_mkuffd_wp(pte);
 			set_pte_at(src_mm, addr, src_pte, pte);
 		}
+		folio_inc_mgte_count(folio);
 	} else if (is_device_private_entry(entry)) {
 		page = pfn_swap_entry_to_page(entry);
 		folio = page_folio(page);
@@ -1651,6 +1652,7 @@ static inline int zap_nonpresent_ptes(struct mmu_gather *tlb,
 		if (!should_zap_folio(details, folio))
 			return 1;
 		rss[mm_counter(folio)]--;
+		folio_dec_mgte_count(folio);
 	} else if (pte_marker_entry_uffd_wp(entry)) {
 		/*
 		 * For anon: always drop the marker; for file: only
mm/migrate.c +27 -1
index a5ea8fba2997..fc2fac1559bd 100644
@@ -241,7 +241,8 @@ static bool remove_migration_pte(struct folio *folio,
 		struct vm_area_struct *vma, unsigned long addr, void *arg)
 {
 	struct rmap_walk_arg *rmap_walk_arg = arg;
-	DEFINE_FOLIO_VMA_WALK(pvmw, rmap_walk_arg->src, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
+	struct folio *src = rmap_walk_arg->src;
+	DEFINE_FOLIO_VMA_WALK(pvmw, src, vma, addr, PVMW_SYNC | PVMW_MIGRATION);
 
 	while (page_vma_mapped_walk(&pvmw)) {
 		rmap_t rmap_flags = RMAP_NONE;
@@ -334,6 +335,7 @@ static bool remove_migration_pte(struct folio *folio,
 
 		trace_remove_migration_pte(pvmw.address, pte_val(pte),
 					   compound_order(new));
+		folio_dec_mgte_count(src);
 
 		/* No need to invalidate - it was non-present before */
 		update_mmu_cache(vma, pvmw.address, pvmw.pte);
@@ -342,12 +344,27 @@ static bool remove_migration_pte(struct folio *folio,
 	return true;
 }
 
+static int folio_removed_all_migrate_entry(struct folio *folio,
+					   struct rmap_walk_control *rwc)
+{
+	struct rmap_walk_arg *arg = (struct rmap_walk_arg *)rwc->arg;
+	struct folio *src = arg->src;
+
+	VM_BUG_ON(!folio_test_mgt_entry(src));
+
+	if (!folio_get_mgte_count(src))
+		return true;
+	return false;
+}
+
 /*
  * Get rid of all migration entries and replace them by
  * references to the indicated page.
  */
 void remove_migration_ptes(struct folio *src, struct folio *dst, int flags)
 {
+	bool undo = src == dst;
+
 	struct rmap_walk_arg rmap_walk_arg = {
 		.src = src,
 		.map_unused_to_zeropage = flags & RMP_USE_SHARED_ZEROPAGE,
@@ -356,12 +373,21 @@ void remove_migration_ptes(struct folio *src, struct folio *dst, int flags)
 	struct rmap_walk_control rwc = {
 		.rmap_one = remove_migration_pte,
 		.locked = flags & RMP_LOCKED,
+		.done = !undo && folio_test_mgt_entry(src) ?
+				folio_removed_all_migrate_entry :
+				NULL,
 		.arg = &rmap_walk_arg,
 	};
 
 	VM_BUG_ON_FOLIO((flags & RMP_USE_SHARED_ZEROPAGE) && (src != dst), src);
 
+	if (undo)
+		folio_remove_mgte(src);
+
 	rmap_walk(dst, &rwc);
+
+	if (!undo)
+		folio_remove_mgte(src);
 }
 
 /*
mm/rmap.c +17 -0
index 2433e12c131d..f3911491b2d9 100644
@@ -2263,6 +2263,7 @@ void try_to_unmap(struct folio *folio, enum ttu_flags flags)
 
 struct migrate_walk_arg {
 	enum ttu_flags flags;
+	unsigned int nr_migrate_entry;
 };
 
 /*
@@ -2282,6 +2283,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 	struct mmu_notifier_range range;
 	struct migrate_walk_arg *mwa = (struct migrate_walk_arg *)arg;
 	enum ttu_flags flags = mwa->flags;
+	unsigned int nr_migrate_entry = 0;
 	unsigned long pfn;
 	unsigned long hsz = 0;
 
@@ -2548,6 +2550,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 						hsz);
 			else
 				set_pte_at(mm, address, pvmw.pte, swp_pte);
+			nr_migrate_entry++;
 			trace_set_migration_pte(address, pte_val(swp_pte),
 						folio_order(folio));
 			/*
@@ -2565,11 +2568,24 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 		folio_put(folio);
 	}
 
+	mwa->nr_migrate_entry += nr_migrate_entry;
+
 	mmu_notifier_invalidate_range_end(&range);
 
 	return ret;
 }
 
+static void folio_set_migrate_entry_type(struct folio *folio,
+					 struct rmap_walk_control *rwc)
+{
+	struct migrate_walk_arg *mwa = (struct migrate_walk_arg *)rwc->arg;
+	unsigned int nr_migrate_entry = mwa->nr_migrate_entry;
+
+	if (nr_migrate_entry && !folio_test_large(folio) &&
+	    !folio_mapped(folio))
+		folio_init_mgte(folio, nr_migrate_entry);
+}
+
 /**
  * try_to_migrate - try to replace all page table mappings with swap entries
  * @folio: the folio to replace page table entries for
@@ -2588,6 +2604,7 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags)
 		.rmap_one = try_to_migrate_one,
 		.arg = (void *)&arg,
 		.done = folio_not_mapped,
+		.exit = folio_set_migrate_entry_type,
 		.locked = flags & TTU_RMAP_LOCKED,
 		.anon_lock = folio_lock_anon_vma_read,
 	};
Signature · 1 line
2.34.1
#10 lore raw
In reply to Huan Yang
Quoted 6 lines · Huan Yang

Summary
==
This patchset reuses page_type to store migrate entry count during the period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to early termination when folio is unmapped in try_to_migrate.

I absolutely detest (ab)using page types for that, so no from my side unless I am missing something important.

Quoted 2 lines

In my self-constructed test scenario, the migration time can be reduced

How relevant is that in practice?

Quoted 14 lines

from over 150+ms to around 30+ms, achieving nearly a 70% performance improvement. Additionally, the flame graph shows that the proportion of remove_migration_ptes can be reduced from 80%+ to 60%+.

Notice: migrate entry specifically refers to migrate PTE entry, as large folio are not supported page type and 0 mapcount reuse.

Principle
==
When a page removes all PTEs in try_to_migrate and sets up a migrate PTE entry, we can determine whether the traversal of remaining VMAs can be terminated early by checking if mapcount is zero. This optimization helps improve performance during migration.

Quoted 5 lines

However, when removing migrate PTE entries and setting up PTEs for the destination folio in remove_migration_ptes, there is no such information available to assist in deciding whether the traversal of remaining VMAs can be ended early. Therefore, it is necessary to traversal all VMAs associated with this folio.

Yes, we don't know how many migration entries are still pointing at the page.

Quoted 8 lines

In reality, when a folio is fully unmapped and before all migrate PTE entries are removed, the mapcount will always be zero. Since page_type and mapcount share a union, and referring to folio_mapcount, we can reuse page_type to record the number of migrate PTE entries of the current folio in the system as long as it's not a large folio. This reuse does not affect calls to folio_mapcount, which will always return zero.

Quoted 4 lines
Quoted 1 line

Therefore, we can set the folio's page_type to PGTY_mgt_entry when

try_to_migrate completes, the folio is already unmapped, and it's not a large folio. The remaining 24 bits can then be used to record the number of migrate PTE entries generated by try_to_migrate.

In the future the page type will no longer overlay the mapcount and, consequently, be sticky.

Quoted 9 lines

Then, in remove_migration_ptes, when the nr_mgt_entry count drops to zero, we can terminate the VMA traversal early.

It's important to note that we need to initialize the folio's page_type to PGTY_mgt_entry and set the migrate entry count only while holding the rmap walk lock.This is because during the lock period, we can prevent new VMA fork (which would increase migrate entries) and VMA unmap (which would decrease migrate entries).

The more I read about PGTY_mgt_entry, the more I hate it.

Quoted 39 lines

However, I doubt there is actually an additional critical section here, for example anon:

Process Parent fork
try_to_migrate
anon_vma_clone
write_lock
avc_inster_tree tail
....
folio_lock_anon_vma_read copy_pte_range
vma_iter pte_lock
.... pte_present copy
...
pte_lock
new forked pte clean
....
remove_migration_ptes
rmap_walk_anon_lock

If my understanding is correct and such a critical section exists, it shouldn't cause any issues—newly added PTEs can still be properly removed and converted into migrate entries.

But in this:

Process Parent fork
try_to_migrate

                                         anon_vma_clone
                                             write_lock
                                                 avc_inster_tree
                                         ....
     folio_lock_anon_vma_read             copy_pte_range
         vma_iter
                 pte_lock
                     migrate entry set
                 ....                        pte_lock
                                                 pte_nonpresent copy
                                             ....
Quoted 3 lines

....
remove_migration_ptes
rmap_walk_anon_lock

Just a note: migration entries also apply to non-anon folios.

Signature · 3 lines
Cheers,

David / dhildenb
author
#11 lore raw
Quoted 8 lines · David Hildenbrand
Quoted 6 lines · Huan Yang

Summary
==
This patchset reuses page_type to store migrate entry count during the period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to early termination when folio is unmapped in try_to_migrate.

Quoted 6 lines

I absolutely detest (ab)using page types for that, so no from my side unless I am missing something important.

Quoted 2 lines

In my self-constructed test scenario, the migration time can be reduced

How relevant is that in practice?

IMO, any folio mapped < nr vma in mapping(anon_vma, addresss_space), will benefit from this.

So, all pages that have been COW-ed by child processes can be skipped.

Quoted 93 lines
Quoted 15 lines

from over 150+ms to around 30+ms, achieving nearly a 70% performance improvement. Additionally, the flame graph shows that the proportion of remove_migration_ptes can be reduced from 80%+ to 60%+.

Notice: migrate entry specifically refers to migrate PTE entry, as large folio are not supported page type and 0 mapcount reuse.

Principle
==
When a page removes all PTEs in try_to_migrate and sets up a migrate PTE entry, we can determine whether the traversal of remaining VMAs can be terminated early by checking if mapcount is zero. This optimization helps improve performance during migration.

Quoted 5 lines

However, when removing migrate PTE entries and setting up PTEs for the destination folio in remove_migration_ptes, there is no such information available to assist in deciding whether the traversal of remaining VMAs can be ended early. Therefore, it is necessary to traversal all VMAs associated with this folio.

Yes, we don't know how many migration entries are still pointing at the page.

Quoted 8 lines

In reality, when a folio is fully unmapped and before all migrate PTE entries are removed, the mapcount will always be zero. Since page_type and mapcount share a union, and referring to folio_mapcount, we can reuse page_type to record the number of migrate PTE entries of the current folio in the system as long as it's not a large folio. This reuse does not affect calls to folio_mapcount, which will always return zero.

Quoted 4 lines
Quoted 1 line

Therefore, we can set the folio's page_type to PGTY_mgt_entry when

try_to_migrate completes, the folio is already unmapped, and it's not a large folio. The remaining 24 bits can then be used to record the number of migrate PTE entries generated by try_to_migrate.

In the future the page type will no longer overlay the mapcount and, consequently, be sticky.

Quoted 9 lines

Then, in remove_migration_ptes, when the nr_mgt_entry count drops to zero, we can terminate the VMA traversal early.

It's important to note that we need to initialize the folio's page_type to PGTY_mgt_entry and set the migrate entry count only while holding the rmap walk lock.This is because during the lock period, we can prevent new VMA fork (which would increase migrate entries) and VMA unmap (which would decrease migrate entries).

The more I read about PGTY_mgt_entry, the more I hate it.

Quoted 26 lines

However, I doubt there is actually an additional critical section here, for example anon:

Process Parent                          fork try_to_migrate
                                         anon_vma_clone write_lock avc_inster_tree tail
                                         ....
     folio_lock_anon_vma_read             copy_pte_range vma_iter                            pte_lock
                 ....                           pte_present copy
                                             ...
                 pte_lock
                     new forked pte clean
....
remove_migration_ptes
     rmap_walk_anon_lock

If my understanding is correct and such a critical section exists, it shouldn't cause any issues—newly added PTEs can still be properly removed and converted into migrate entries.

But in this:

Quoted 16 lines

Process Parent                          fork try_to_migrate
                                         anon_vma_clone write_lock avc_inster_tree
                                         ....
     folio_lock_anon_vma_read             copy_pte_range vma_iter
                 pte_lock
                     migrate entry set
                 ....                        pte_lock pte_nonpresent copy
                                             ....
....
remove_migration_ptes
     rmap_walk_anon_lock

Quoted 1 line

Just a note: migration entries also apply to non-anon folios.

Yes, just example.

#12 lore raw
Quoted 13 lines · Huan Yang
Quoted 11 lines · David Hildenbrand
Quoted 6 lines · Huan Yang

Summary
==
This patchset reuses page_type to store migrate entry count during the period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to early termination when folio is unmapped in try_to_migrate.

I absolutely detest (ab)using page types for that, so no from my side unless I am missing something important.

Quoted 1 line
Quoted 7 lines
Quoted 3 lines
Quoted 1 line

In my self-constructed test scenario, the migration time can be reduced

How relevant is that in practice?

IMO, any folio mapped < nr vma in mapping(anon_vma, addresss_space), will benefit from this.

So, all pages that have been COW-ed by child processes can be skipped.

For small anon folios, you could use the anon-exclusive marker to derive "there can only be a single mapping".

It's stored alongside the migration entry.

So once you restored that single migration entry, you can just stop the walk.

Signature · 3 lines
Cheers,

David / dhildenb
#13 lore raw
Quoted 25 lines · David Hildenbrand
Quoted 15 lines · Huan Yang
Quoted 13 lines · David Hildenbrand
Quoted 7 lines · Huan Yang

Summary
==
This patchset reuses page_type to store migrate entry count during the period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to
early termination when folio is unmapped in try_to_migrate.

I absolutely detest (ab)using page types for that, so no from my side unless I am missing something important.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

In my self-constructed test scenario, the migration time can be reduced

How relevant is that in practice?

IMO, any folio mapped < nr vma in mapping(anon_vma, addresss_space), will benefit from this.

So, all pages that have been COW-ed by child processes can be skipped.

Quoted 7 lines

For small anon folios, you could use the anon-exclusive marker to derive "there can only be a single mapping".

It's stored alongside the migration entry.

So once you restored that single migration entry, you can just stop the walk.

Essentially, something (untested) like this:

mm/migrate.c +9 -0
index 425401b2d4e14..aa5bf96b1daee 100644
@@ -421,6 +421,15 @@ static bool remove_migration_pte(struct folio *folio,
 
                /* No need to invalidate - it was non-present before */
                update_mmu_cache(vma, pvmw.address, pvmw.pte);
+
+               /*
+                * If the small anon folio is exclusive, here can be exactly one
+                * page mapping -- the one we just restored.
+                */
+               if (!folio_test_large(folio) && (rmap_flags & RMAP_EXCLUSIVE)) {
+                       page_vma_mapped_walk_done(&pvmw);
+                       break;
+               }
        }
 
        return true;
Signature · 3 lines
Cheers,

David / dhildenb
#14 lore raw
Quoted 46 lines · David Hildenbrand
Quoted 25 lines · David Hildenbrand
Quoted 15 lines · Huan Yang
Quoted 13 lines · David Hildenbrand
Quoted 7 lines · Huan Yang

Summary
==
This patchset reuses page_type to store migrate entry count during the period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to
early termination when folio is unmapped in try_to_migrate.

I absolutely detest (ab)using page types for that, so no from my side unless I am missing something important.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

In my self-constructed test scenario, the migration time can be reduced

How relevant is that in practice?

IMO, any folio mapped < nr vma in mapping(anon_vma, addresss_space), will benefit from this.

So, all pages that have been COW-ed by child processes can be skipped.

Quoted 7 lines

For small anon folios, you could use the anon-exclusive marker to derive "there can only be a single mapping".

It's stored alongside the migration entry.

So once you restored that single migration entry, you can just stop the walk.

Essentially, something (untested) like this:

mm/migrate.c +2 -0
index 425401b2d4e14..aa5bf96b1daee 100644
@@ -421,6 +421,15 @@ static bool remove_migration_pte(struct folio *folio,
 
                  /* No need to invalidate - it was non-present before */
                  update_mmu_cache(vma, pvmw.address, pvmw.pte);
+
+               /*

mm/migrate.c static bool remove_migration_pte(struct folio *folio, · line 435

Quoted 10 lines
mm/migrate.c +7 -0
+                * If the small anon folio is exclusive, here can be exactly one
+                * page mapping -- the one we just restored.
+                */
+               if (!folio_test_large(folio) && (rmap_flags & RMAP_EXCLUSIVE)) {
+                       page_vma_mapped_walk_done(&pvmw);
+                       break;
+               }
          }
 
          return true;

Probably that won't really help I assume, because __folio_set_anon() will move the new anon folio under vma->anon_vma, not vma->anon_vma->root.

So I assume you mean that we had a COW-shared folio now mapped only into some VMAs (some mappings in other processes removed due to CoW or similar).

In that case aborting early can help.

Not in all cases though, just imagine that the very last VMA we're iterating maps the page. You have to iterate through all of them either way ... no way around that, really.

Signature · 3 lines
Cheers,

David / dhildenb
author
#15 lore raw
Quoted 61 lines · David Hildenbrand
Quoted 49 lines · David Hildenbrand
Quoted 25 lines · David Hildenbrand
Quoted 15 lines · Huan Yang
Quoted 13 lines · David Hildenbrand
Quoted 7 lines · Huan Yang

Summary
==
This patchset reuses page_type to store migrate entry count during the period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to
early termination when folio is unmapped in try_to_migrate.

I absolutely detest (ab)using page types for that, so no from my side unless I am missing something important.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Quoted 1 line

In my self-constructed test scenario, the migration time can be reduced

How relevant is that in practice?

IMO, any folio mapped < nr vma in mapping(anon_vma, addresss_space), will benefit from this.

So, all pages that have been COW-ed by child processes can be skipped.

Quoted 7 lines

For small anon folios, you could use the anon-exclusive marker to derive "there can only be a single mapping".

It's stored alongside the migration entry.

So once you restored that single migration entry, you can just stop the walk.

Essentially, something (untested) like this:

mm/migrate.c +3 -0
index 425401b2d4e14..aa5bf96b1daee 100644
@@ -421,6 +421,15 @@ static bool remove_migration_pte(struct folio 
 *folio,
                      /* No need to invalidate - it was non-present before */
                   update_mmu_cache(vma, pvmw.address, pvmw.pte);
+
+               /*
+                * If the small anon folio is exclusive, here can be 

exactly one
+                * page mapping -- the one we just restored.

Quoted 8 lines

+                */
+               if (!folio_test_large(folio) && (rmap_flags & RMAP_EXCLUSIVE)) {
+                       page_vma_mapped_walk_done(&pvmw); +                       break;
+               }
          }
             return true;

Probably that won't really help I assume, because __folio_set_anon() will move the new anon folio under vma->anon_vma, not vma->anon_vma->root.

Quoted 5 lines

So I assume you mean that we had a COW-shared folio now mapped only into some VMAs (some mappings in other processes removed due to CoW or similar).

In that case aborting early can help.

Not in all cases though, just imagine that the very last VMA we're iterating maps the page. You have to iterate through all of them either way ... no way around that, really.

Indeed, whether we can exit the loop early depends on the position of the terminating VMA in the tree.

I think a better approach would be to remove the fully COW-ed VMAs and their associated AVCs from the anon_vma's tree.

I've been researching this aspect, but haven't made any progress yet.(I have some ideas, but the specific implementation is still challenging.)

#16 lore raw
Quoted 79 lines · Huan Yang
Quoted 71 lines · David Hildenbrand
Quoted 53 lines · David Hildenbrand
Quoted 27 lines · David Hildenbrand
Quoted 19 lines · Huan Yang
Quoted 17 lines · David Hildenbrand
Quoted 8 lines · Huan Yang

Summary
==
This patchset reuses page_type to store migrate
entry count during the
period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to
early termination when folio is unmapped in try_to_migrate.

I absolutely detest (ab)using page types for that, so no from my side unless I am missing something important.

Quoted 3 lines

In my self-constructed test scenario, the migration time can be reduced

Quoted 6 lines
Quoted 1 line

How relevant is that in practice?

IMO, any folio mapped < nr vma in mapping(anon_vma, addresss_space), will benefit from this.

So, all pages that have been COW-ed by child processes can be skipped.

Quoted 8 lines

For small anon folios, you could use the anon-exclusive marker to derive
"there can only be a single mapping".

It's stored alongside the migration entry.

So once you restored that single migration entry, you can just stop the walk.

Essentially, something (untested) like this:

mm/migrate.c +0 -0
index 425401b2d4e14..aa5bf96b1daee 100644
@@ -421,6 +421,15 @@ static bool remove_migration_pte(struct folio
 *folio,
                      /* No need to invalidate - it was non-present

before */
                  update_mmu_cache(vma, pvmw.address, pvmw.pte); +
+               /*
+                * If the small anon folio is exclusive, here can be exactly one
+                * page mapping -- the one we just restored.

Quoted 8 lines

+                */
+               if (!folio_test_large(folio) && (rmap_flags & RMAP_EXCLUSIVE)) {
+                       page_vma_mapped_walk_done(&pvmw); +                       break;
+               }
          }
             return true;

Probably that won't really help I assume, because __folio_set_anon() will move the new anon folio under vma->anon_vma, not vma->anon_vma->root.

So I assume you mean that we had a COW-shared folio now mapped only into some VMAs (some mappings in other processes removed due to CoW or similar).

Quoted 5 lines

In that case aborting early can help.

Not in all cases though, just imagine that the very last VMA we're iterating maps the page. You have to iterate through all of them either way ... no way around that, really.

Quoted 9 lines

Indeed, whether we can exit the loop early depends on the position of the terminating VMA in the tree.

I think a better approach would be to remove the fully COW-ed VMAs and their associated AVCs from the anon_vma's tree.

I've been researching this aspect, but haven't made any progress yet.(I have some ideas, but the specific implementation is still challenging.)

Please leave this alone, I'm in the midst of trying to make fundamental changes to the anon rmap logic and it's really very subtle and indeed challenging (as you've seen).

Since I intend to change the whole mechanism around this, efforts to adjust the existing behaviour are going to strictly conflict with that.

We are 'lazy' in actually properly accounting for fully CoW'd VMAs and so can only know 'maybe' if it has, I mean as from above you've noticed.

The CoW hierarchy also makes life challenging, see vma_had_uncowed_parents() for an example of the subtlty.

Having looked at anon rmap in detail, I have come to think the only sensible way forward is something fairly bold.

Thanks!

author
#17 lore raw
Quoted 76 lines · Lorenzo Stoakes
Quoted 75 lines · Huan Yang
Quoted 63 lines · David Hildenbrand
Quoted 46 lines · David Hildenbrand
Quoted 21 lines · David Hildenbrand
Quoted 13 lines · Huan Yang
Quoted 12 lines · David Hildenbrand
Quoted 8 lines · Huan Yang

Summary
==
This patchset reuses page_type to store migrate
entry count during the
period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to
early termination when folio is unmapped in try_to_migrate.

I absolutely detest (ab)using page types for that, so no from my side unless I am missing something important.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

In my self-constructed test scenario, the migration time can be reduced

How relevant is that in practice?

IMO, any folio mapped < nr vma in mapping(anon_vma, addresss_space), will benefit from this.

So, all pages that have been COW-ed by child processes can be skipped.

Quoted 8 lines

For small anon folios, you could use the anon-exclusive marker to derive
"there can only be a single mapping".

It's stored alongside the migration entry.

So once you restored that single migration entry, you can just stop the walk.

Essentially, something (untested) like this:

mm/migrate.c +0 -0
index 425401b2d4e14..aa5bf96b1daee 100644
@@ -421,6 +421,15 @@ static bool remove_migration_pte(struct folio
 *folio,
                      /* No need to invalidate - it was non-present

before */
                  update_mmu_cache(vma, pvmw.address, pvmw.pte);
+
+               /*
+                * If the small anon folio is exclusive, here can be exactly one
+                * page mapping -- the one we just restored.

Quoted 8 lines

+                */
+               if (!folio_test_large(folio) && (rmap_flags & RMAP_EXCLUSIVE)) {
+                       page_vma_mapped_walk_done(&pvmw); +                       break;
+               }
          }
             return true;

Probably that won't really help I assume, because __folio_set_anon() will move the new anon folio under vma->anon_vma, not vma->anon_vma->root.

So I assume you mean that we had a COW-shared folio now mapped only into some VMAs (some mappings in other processes removed due to CoW or similar).

Quoted 5 lines

In that case aborting early can help.

Not in all cases though, just imagine that the very last VMA we're iterating maps the page. You have to iterate through all of them either way ... no way around that, really.

Indeed, whether we can exit the loop early depends on the position of the terminating VMA in the tree.

I think a better approach would be to remove the fully COW-ed VMAs and their associated AVCs from the anon_vma's tree.

Quoted 6 lines
Quoted 3 lines

I've been researching this aspect, but haven't made any progress yet.(I have some ideas, but the specific implementation is still challenging.)

Please leave this alone, I'm in the midst of trying to make fundamental changes to the anon rmap logic and it's really very subtle and indeed challenging (as you've seen).

OK, but I still try to research it, not only for upstream or something.

Quoted 4 lines

Since I intend to change the whole mechanism around this, efforts to adjust the existing behaviour are going to strictly conflict with that.

Quoted 8 lines

We are 'lazy' in actually properly accounting for fully CoW'd VMAs and so can only know 'maybe' if it has, I mean as from above you've noticed.

The CoW hierarchy also makes life challenging, see vma_had_uncowed_parents() for an example of the subtlty.

Having looked at anon rmap in detail, I have come to think the only sensible way forward is something fairly bold.

Woo, that's really interesting. I'm looking forward to this.

Thanks.

Quoted 3 lines

Thanks!

#18 lore raw
Quoted 6 lines · Huan Yang
Quoted 2 lines · Lorenzo Stoakes

Having looked at anon rmap in detail, I have come to think the only sensible way forward is something fairly bold.

Woo, that's really interesting. I'm looking forward to this.

Haha thanks, I found the one person in the world who's excited by anon rmap :P

Or rather, the only OTHER person ;)

Cheers, Lorenzo

author
#19 lore raw
Quoted 8 lines · Lorenzo Stoakes
Quoted 4 lines · Huan Yang
Quoted 2 lines · Lorenzo Stoakes

Having looked at anon rmap in detail, I have come to think the only sensible way forward is something fairly bold.

Woo, that's really interesting. I'm looking forward to this.

Haha thanks, I found the one person in the world who's excited by anon rmap :P

Or rather, the only OTHER person ;)

Not only is this interesting, but the efficiency of anonymous page reclamation also frequently affects the performance of our products.

In our flame graphs, we can see that many anonymous page reverse mapping paths are among the hotspots.

Therefore, please continue your work—I’m really looking forward to the results.

Additionally, if there’s anything I can do to help, I’d be happy to participate.

Thanks.

Quoted 3 lines

Cheers, Lorenzo

#20 lore raw
In reply to Huan Yang

NAK. This series is completely un-upstreamable in any form.

David has responded to you already, but to underline.

The lesson here is that you really ought to discuss things with people in the subsystem you are changing in advance of spending a lot of time doing work like this which you intend to upstream.

Quoted 11 lines · Huan Yang

Summary
==
This patchset reuses page_type to store migrate entry count during the period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to early termination when folio is unmapped in try_to_migrate.

In my self-constructed test scenario, the migration time can be reduced from over 150+ms to around 30+ms, achieving nearly a 70% performance improvement. Additionally, the flame graph shows that the proportion of remove_migration_ptes can be reduced from 80%+ to 60%+.

This sounds completely contrived. I don't even know if you have a use case here.

Quoted 17 lines

Notice: migrate entry specifically refers to migrate PTE entry, as large folio are not supported page type and 0 mapcount reuse.

Principle
==
When a page removes all PTEs in try_to_migrate and sets up a migrate PTE entry, we can determine whether the traversal of remaining VMAs can be terminated early by checking if mapcount is zero. This optimization helps improve performance during migration.

However, when removing migrate PTE entries and setting up PTEs for the destination folio in remove_migration_ptes, there is no such information available to assist in deciding whether the traversal of remaining VMAs can be ended early. Therefore, it is necessary to traversal all VMAs associated with this folio.

Quoted 7 lines

In reality, when a folio is fully unmapped and before all migrate PTE entries are removed, the mapcount will always be zero. Since page_type and mapcount share a union, and referring to folio_mapcount, we can reuse page_type to record the number of migrate PTE entries of the current folio in the system as long as it's not a large folio. This reuse does not affect calls to folio_mapcount, which will always return zero.

OK so - if you ever find yourself thinking this way, please stop. We are in the midst of fundamentally changing how folios and pages work.

There is absolutely ZERO room for reusing arbitrary fields in this way. Any series that attempts to do this will be rejected.

Again, I must say - if you had raised this ahead of time we could have saved you some effort.

Quoted 5 lines

Therefore, we can set the folio's page_type to PGTY_mgt_entry when try_to_migrate completes, the folio is already unmapped, and it's not a large folio. The remaining 24 bits can then be used to record the number of migrate PTE entries generated by try_to_migrate.

I mean there's so much wrong here. The future is large folios. Making some fundamental change that relies on not-large folio is a mistake. 24 bits... I mean no.

Quoted 9 lines

Then, in remove_migration_ptes, when the nr_mgt_entry count drops to zero, we can terminate the VMA traversal early.

It's important to note that we need to initialize the folio's page_type to PGTY_mgt_entry and set the migrate entry count only while holding the rmap walk lock.This is because during the lock period, we can prevent new VMA fork (which would increase migrate entries) and VMA unmap (which would decrease migrate entries).

No, no no. NO.

You are not introducing new locking complexity for this.

I could go on, but there's no point.

This series is not upstreamable, NAK.

author
#21 lore raw
Quoted 7 lines · Lorenzo Stoakes

NAK. This series is completely un-upstreamable in any form.

David has responded to you already, but to underline.

The lesson here is that you really ought to discuss things with people in the subsystem you are changing in advance of spending a lot of time doing work like this which you intend to upstream.

Yes, this is a very useful lesson.:)

In the future, when I have ideas in this area, I will bring them up for discussion first, especially when

they involve folios or pages.

Quoted 9 lines · Huan Yang
Quoted 7 lines

Summary
==
This patchset reuses page_type to store migrate entry count during the period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to early termination when folio is unmapped in try_to_migrate.

Quoted 6 lines
Quoted 4 lines

In my self-constructed test scenario, the migration time can be reduced from over 150+ms to around 30+ms, achieving nearly a 70% performance improvement. Additionally, the flame graph shows that the proportion of remove_migration_ptes can be reduced from 80%+ to 60%+.

This sounds completely contrived. I don't even know if you have a use case here.

The test case I provided does have an amplified effect, but the optimization it demonstrates is real. It's just that when scaled up to the system level, the effect becomes difficult to observe.

Quoted 33 lines
Quoted 17 lines

Notice: migrate entry specifically refers to migrate PTE entry, as large folio are not supported page type and 0 mapcount reuse.

Principle
==
When a page removes all PTEs in try_to_migrate and sets up a migrate PTE entry, we can determine whether the traversal of remaining VMAs can be terminated early by checking if mapcount is zero. This optimization helps improve performance during migration.

However, when removing migrate PTE entries and setting up PTEs for the destination folio in remove_migration_ptes, there is no such information available to assist in deciding whether the traversal of remaining VMAs can be ended early. Therefore, it is necessary to traversal all VMAs associated with this folio.

Quoted 7 lines

In reality, when a folio is fully unmapped and before all migrate PTE entries are removed, the mapcount will always be zero. Since page_type and mapcount share a union, and referring to folio_mapcount, we can reuse page_type to record the number of migrate PTE entries of the current folio in the system as long as it's not a large folio. This reuse does not affect calls to folio_mapcount, which will always return zero.

OK so - if you ever find yourself thinking this way, please stop. We are in the midst of fundamentally changing how folios and pages work.

There is absolutely ZERO room for reusing arbitrary fields in this way. Any series that attempts to do this will be rejected.

Again, I must say - if you had raised this ahead of time we could have saved you some effort.

Quoted 7 lines
Quoted 4 lines

Therefore, we can set the folio's page_type to PGTY_mgt_entry when try_to_migrate completes, the folio is already unmapped, and it's not a large folio. The remaining 24 bits can then be used to record the number of migrate PTE entries generated by try_to_migrate.

I mean there's so much wrong here. The future is large folios. Making some fundamental change that relies on not-large folio is a mistake. 24 bits... I mean no.

Thanks, I understand it.

Quoted 17 lines
Quoted 9 lines

Then, in remove_migration_ptes, when the nr_mgt_entry count drops to zero, we can terminate the VMA traversal early.

It's important to note that we need to initialize the folio's page_type to PGTY_mgt_entry and set the migrate entry count only while holding the rmap walk lock.This is because during the lock period, we can prevent new VMA fork (which would increase migrate entries) and VMA unmap (which would decrease migrate entries).

No, no no. NO.

You are not introducing new locking complexity for this.

I could go on, but there's no point.

This series is not upstreamable, NAK.

reviewer
#22 lore raw
Quoted 1 line · Huan Yang

在 2025/7/24 17:15, Lorenzo Stoakes 写道:

[snip]

Quoted 12 lines
Quoted 12 lines · Huan Yang
Quoted 11 lines

Summary
==
This patchset reuses page_type to store migrate entry count during the period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to early termination when folio is unmapped in try_to_migrate.

In my self-constructed test scenario, the migration time can be reduced from over 150+ms to around 30+ms, achieving nearly a 70% performance improvement. Additionally, the flame graph shows that the proportion of remove_migration_ptes can be reduced from 80%+ to 60%+.

Quoted 7 lines
Quoted 2 lines

This sounds completely contrived. I don't even know if you have a use case here.

The test case I provided does have an amplified effect, but the optimization it demonstrates is real. It's just that when scaled up to the system level, the effect becomes difficult to observe.

It's more important to sell your problems than selling your code :-)

If you cannot prove that the optimization has some practical effect, it's hard to persuade others for increased complexity.

---
Best Regards,
Huang, Ying

author
#23 lore raw
Quoted 17 lines · Ying
Quoted 1 line · Huan Yang

在 2025/7/24 17:15, Lorenzo Stoakes 写道:

[snip]

Quoted 12 lines
Quoted 12 lines · Huan Yang
Quoted 11 lines

Summary
==
This patchset reuses page_type to store migrate entry count during the period from migrate entry setup to removal, enabling accelerated VMA traversal when removing migrate entries, following a similar principle to early termination when folio is unmapped in try_to_migrate.

In my self-constructed test scenario, the migration time can be reduced from over 150+ms to around 30+ms, achieving nearly a 70% performance improvement. Additionally, the flame graph shows that the proportion of remove_migration_ptes can be reduced from 80%+ to 60%+.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

This sounds completely contrived. I don't even know if you have a use case here.

The test case I provided does have an amplified effect, but the optimization it demonstrates is real. It's just that when scaled up to the system level, the effect becomes difficult to observe.

It's more important to sell your problems than selling your code :-)

I'll remember it. Thanks. :)

Quoted 3 lines

If you cannot prove that the optimization has some practical effect, it's hard to persuade others for increased complexity.

To be honest, this patch stems from an issue I noticed during code review.

When this patchset was completed, I did put in some effort to find its benefits, and it was only

under such an exaggeratedly constructed test scenario that the effect could be demonstrated. :(

The actual problem I'm facing has been described in other replies.

It's actually about some anonymous pages and fully COW-ed pages, but their avcs haven't been

removed from the anon_vma's RB tree, resulting in inefficient traversal.

Lorenzo has mentioned that he has some bold ideas regarding this, let's look forward it. :)

Thanks.

Quoted 4 lines

---
Best Regards,
Huang, Ying

#24 lore raw
Quoted 10 lines
Quoted 2 lines

If you cannot prove that the optimization has some practical effect, it's hard to persuade others for increased complexity.

To be honest, this patch stems from an issue I noticed during code review.

When this patchset was completed, I did put in some effort to find its benefits, and it was only

under such an exaggeratedly constructed test scenario that the effect could be demonstrated. :(

I mean, thanks for looking into that and trying to find a way to improve it. :)

That VMA walk is the real problem, stopping earlier is just an optimization that works in some cases. I guess on average it will improve things, although probably really hard to quantify in reality.

I think tracking the #migration entries might be a very good debugging tool.

A cleaner and more reliably solution regarding what you tried to implement would be able to

(a) track it in a separate counter, at the time we establish/remove a migration entry, not once the mapcount is already 0. With "struct folio" getting allocated separately in the future this could maybe be feasible (and putting it under a config knob).

(b) doing it also for large folios as well

(b) might be tricky with migration entries being used for THP splits, but probable it could be special-cased somehow, I am sure.

Signature · 3 lines
Cheers,

David / dhildenb