mm/migrate: move NUMA hinting fault folio isolation + checks under PTL

Let's just return 0 on success, which is less confusing.

... especially because we got it wrong in the migrate.h stub where we have "return -EAGAIN; /* can't migrate now */" instead of "return 0;". Likely this wrong return value doesn't currently matter, but it certainly adds confusion.

We'll add migrate_misplaced_folio_prepare() next, where we want to use the same "return 0 on success" approach, so let's just clean this up.

Signed-off-by David Hildenbrand <[email protected]>

---
 mm/huge_memory.c | 5 ++---
 mm/memory.c      | 2 +-
 mm/migrate.c     | 4 ++--
 3 files changed, 5 insertions(+), 6 deletions(-)

Let's just return 0 on success, which is less confusing.

... especially because we got it wrong in the migrate.h stub where we have "return -EAGAIN; /* can't migrate now */" instead of "return 0;". Likely this wrong return value doesn't currently matter, but it certainly adds confusion.

We'll add migrate_misplaced_folio_prepare() next, where we want to use the same "return 0 on success" approach, so let's just clean this up

Signed-off-by David Hildenbrand <[email protected]>

---
  mm/huge_memory.c | 5 ++---
  mm/memory.c      | 2 +-
  mm/migrate.c     | 4 ++--
  3 files changed, 5 insertions(+), 6 deletions(-)

mm/huge_memory.c +2 -3

index 0fffaa58a47a..fc27dabcd8e3 100644
@@ -1652,7 +1652,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
  	unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
  	int nid = NUMA_NO_NODE;
  	int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK);
-	bool migrated = false, writable = false;
+	bool writable = false;
  	int flags = 0;
 
  	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);

@@ -1696,8 +1696,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
  	spin_unlock(vmf->ptl);
  	writable = false;
 
-	migrated = migrate_misplaced_folio(folio, vma, target_nid);
-	if (migrated) {
+	if (!migrate_misplaced_folio(folio, vma, target_nid)) {
  		flags |= TNF_MIGRATED;
  		nid = target_nid;
  	} else {

mm/memory.c +1 -1

index 00728ea95583..118660de5bcc 100644
@@ -5354,7 +5354,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
  	ignore_writable = true;
 
  	/* Migrate to the requested node */
-	if (migrate_misplaced_folio(folio, vma, target_nid)) {
+	if (!migrate_misplaced_folio(folio, vma, target_nid)) {
  		nid = target_nid;
  		flags |= TNF_MIGRATED;
  	} else {

mm/migrate.c +2 -2

index 781979567f64..0307b54879a0 100644
@@ -2629,11 +2629,11 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
  					    nr_succeeded);
  	}
  	BUG_ON(!list_empty(&migratepages));
-	return isolated;
+	return isolated ? 0 : -EAGAIN;
 
  out:
  	folio_put(folio);
-	return 0;
+	return -EAGAIN;
  }
  #endif /* CONFIG_NUMA_BALANCING */
  #endif /* CONFIG_NUMA */

Let's just return 0 on success, which is less confusing.

... especially because we got it wrong in the migrate.h stub where we have "return -EAGAIN; /* can't migrate now */" instead of "return 0;". Likely this wrong return value doesn't currently matter, but it certainly adds confusion.

We'll add migrate_misplaced_folio_prepare() next, where we want to use the same "return 0 on success" approach, so let's just clean this up.

Signed-off-by David Hildenbrand <[email protected]>

---
 mm/huge_memory.c | 5 ++---
 mm/memory.c      | 2 +-
 mm/migrate.c     | 4 ++--
 3 files changed, 5 insertions(+), 6 deletions(-)

mm/huge_memory.c +2 -3

index 0fffaa58a47a..fc27dabcd8e3 100644
@@ -1652,7 +1652,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
 	int nid = NUMA_NO_NODE;
 	int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK);
-	bool migrated = false, writable = false;
+	bool writable = false;
 	int flags = 0;
 
 	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);

@@ -1696,8 +1696,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	spin_unlock(vmf->ptl);
 	writable = false;
 
-	migrated = migrate_misplaced_folio(folio, vma, target_nid);
-	if (migrated) {
+	if (!migrate_misplaced_folio(folio, vma, target_nid)) {
 		flags |= TNF_MIGRATED;
 		nid = target_nid;
 	} else {

mm/memory.c +1 -1

index 00728ea95583..118660de5bcc 100644
@@ -5354,7 +5354,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	ignore_writable = true;
 
 	/* Migrate to the requested node */
-	if (migrate_misplaced_folio(folio, vma, target_nid)) {
+	if (!migrate_misplaced_folio(folio, vma, target_nid)) {
 		nid = target_nid;
 		flags |= TNF_MIGRATED;
 	} else {

mm/migrate.c +0 -0

index 781979567f64..0307b54879a0 100644

Quoted 58 lines · David Hildenbrand

Let's just return 0 on success, which is less confusing.

... especially because we got it wrong in the migrate.h stub where we have "return -EAGAIN; /* can't migrate now */" instead of "return 0;". Likely this wrong return value doesn't currently matter, but it certainly adds confusion.

We'll add migrate_misplaced_folio_prepare() next, where we want to use the same "return 0 on success" approach, so let's just clean this up.

Signed-off-by David Hildenbrand <[email protected]>

---
  mm/huge_memory.c | 5 ++---
  mm/memory.c      | 2 +-
  mm/migrate.c     | 4 ++--
  3 files changed, 5 insertions(+), 6 deletions(-)

mm/huge_memory.c +2 -3

index 0fffaa58a47a..fc27dabcd8e3 100644
@@ -1652,7 +1652,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
  	unsigned long haddr = vmf->address & HPAGE_PMD_MASK;
  	int nid = NUMA_NO_NODE;
  	int target_nid, last_cpupid = (-1 & LAST_CPUPID_MASK);
-	bool migrated = false, writable = false;
+	bool writable = false;
  	int flags = 0;
 
  	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);

@@ -1696,8 +1696,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
  	spin_unlock(vmf->ptl);
  	writable = false;
 
-	migrated = migrate_misplaced_folio(folio, vma, target_nid);
-	if (migrated) {
+	if (!migrate_misplaced_folio(folio, vma, target_nid)) {
  		flags |= TNF_MIGRATED;
  		nid = target_nid;
  	} else {

mm/memory.c +1 -1

index 00728ea95583..118660de5bcc 100644
@@ -5354,7 +5354,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
  	ignore_writable = true;
 
  	/* Migrate to the requested node */
-	if (migrate_misplaced_folio(folio, vma, target_nid)) {
+	if (!migrate_misplaced_folio(folio, vma, target_nid)) {
  		nid = target_nid;
  		flags |= TNF_MIGRATED;
  	} else {

mm/migrate.c +0 -0

index 781979567f64..0307b54879a0 100644
@@ -2629,11 +2629,11 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma, · line 2632

Quoted 5 lines

mm/migrate.c +1 -1

  					    nr_succeeded);
  	}
  	BUG_ON(!list_empty(&migratepages));
-	return isolated;
+	return isolated ? 0 : -EAGAIN;

Is it good to use -EAGAIN as error code always? At least if nr_remaining < 0, we can use that?

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Signed-off-by David Hildenbrand <[email protected]>

---
 include/linux/migrate.h |  7 ++++
 mm/huge_memory.c        |  8 ++--
 mm/memory.c             |  9 +++--
 mm/migrate.c            | 81 +++++++++++++++++++----------------------
 4 files changed, 55 insertions(+), 50 deletions(-)

include/linux/migrate.h +7 -0

index f9d92482d117..644be30b69c8 100644
@@ -139,9 +139,16 @@ const struct movable_operations *page_movable_ops(struct page *page)
 }
 
 #ifdef CONFIG_NUMA_BALANCING
+int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
 			   int node);
 #else
+static inline int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 static inline int migrate_misplaced_folio(struct folio *folio,
 					 struct vm_area_struct *vma, int node)
 {

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	if (node_is_toptier(nid))
 		last_cpupid = folio_last_cpupid(folio);
 	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
 		goto out_map;
 	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
 	spin_unlock(vmf->ptl);
 	writable = false;
 

mm/memory.c +0 -0

index 118660de5bcc..4fd1ecfced4d 100644

Quoted 81 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Signed-off-by David Hildenbrand <[email protected]>

---
  include/linux/migrate.h |  7 ++++
  mm/huge_memory.c        |  8 ++--
  mm/memory.c             |  9 +++--
  mm/migrate.c            | 81 +++++++++++++++++++----------------------
  4 files changed, 55 insertions(+), 50 deletions(-)

include/linux/migrate.h +7 -0

index f9d92482d117..644be30b69c8 100644
@@ -139,9 +139,16 @@ const struct movable_operations *page_movable_ops(struct page *page)
  }
 
  #ifdef CONFIG_NUMA_BALANCING
+int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node);
  int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
  			   int node);
  #else
+static inline int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
  static inline int migrate_misplaced_folio(struct folio *folio,
  					 struct vm_area_struct *vma, int node)
  {

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
  	if (node_is_toptier(nid))
  		last_cpupid = folio_last_cpupid(folio);
  	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
  		goto out_map;
  	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
  	spin_unlock(vmf->ptl);
  	writable = false;
 

mm/memory.c +0 -0

index 118660de5bcc..4fd1ecfced4d 100644
@@ -5207,8 +5207,6 @@ int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf,
  {
  	struct vm_area_struct *vma = vmf->vma;
 

mm/memory.c int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf, · line 5211

Quoted 4 lines

mm/memory.c +0 -2

-	folio_get(folio);
-
  	/* Record the current PID acceesing VMA */
  	vma_set_access_pid_bit(vma);

Without taking a reference here, is it safe to call mpol_misplaced() on the folio at the return? Can this folio be freed?

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Signed-off-by David Hildenbrand <[email protected]>

---
  include/linux/migrate.h |  7 ++++
  mm/huge_memory.c        |  8 ++--
  mm/memory.c             |  9 +++--
  mm/migrate.c            | 81 +++++++++++++++++++----------------------
  4 files changed, 55 insertions(+), 50 deletions(-)

include/linux/migrate.h +7 -0

index f9d92482d117..644be30b69c8 100644
@@ -139,9 +139,16 @@ const struct movable_operations *page_movable_ops(struct page *page)
  }
 
  #ifdef CONFIG_NUMA_BALANCING
+int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node);
  int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
  			   int node);
  #else
+static inline int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
  static inline int migrate_misplaced_folio(struct folio *folio,
  					 struct vm_area_struct *vma, int node)
  {

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
  	if (node_is_toptier(nid))
  		last_cpupid = folio_last_cpupid(folio);
  	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
  		goto out_map;
  	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
  	spin_unlock(vmf->ptl);
  	writable = false;
 

mm/memory.c +5 -4

index 118660de5bcc..4fd1ecfced4d 100644
@@ -5207,8 +5207,6 @@ int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf,
  {
  	struct vm_area_struct *vma = vmf->vma;
 
-	folio_get(folio);
-
  	/* Record the current PID acceesing VMA */
  	vma_set_access_pid_bit(vma);
 

@@ -5345,10 +5343,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
  	else
  		last_cpupid = folio_last_cpupid(folio);
  	target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
  		goto out_map;
  	}
+	/* The folio is isolated and isolation code holds a folio reference. */
  	pte_unmap_unlock(vmf->pte, vmf->ptl);
  	writable = false;
  	ignore_writable = true;

mm/migrate.c +32 -4

index 0307b54879a0..27f070f64f27 100644
@@ -2530,9 +2530,37 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,
  	return __folio_alloc_node(gfp, order, nid);
  }
 
-static int numamigrate_isolate_folio(pg_data_t *pgdat, struct folio *folio)
+/*
+ * Prepare for calling migrate_misplaced_folio() by isolating the folio if
+ * permitted. Must be called with the PTL still held.
+ */
+int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node)
  {
  	int nr_pages = folio_nr_pages(folio);
+	pg_data_t *pgdat = NODE_DATA(node);
+
+	if (folio_is_file_lru(folio)) {
+		/*
+		 * Do not migrate file folios that are mapped in multiple
+		 * processes with execute permissions as they are probably
+		 * shared libraries.
+		 *
+		 * See folio_likely_mapped_shared() on possible imprecision
+		 * when we cannot easily detect if a folio is shared.
+		 */
+		if ((vma->vm_flags & VM_EXEC) &&
+		    folio_likely_mapped_shared(folio))
+			return -EACCES;
+
+		/*
+		 * Do not migrate dirty folios as not all filesystems can move
+		 * dirty folios in MIGRATE_ASYNC mode which is a waste of
+		 * cycles.
+		 */
+		if (folio_test_dirty(folio))
+			return -EAGAIN;
+	}
 
  	/* Avoid migrating to a node that is nearly full */
  	if (!migrate_balanced_pgdat(pgdat, nr_pages)) {

@@ -2550,65 +2578,37 @@ static int numamigrate_isolate_folio(pg_data_t *pgdat, struct folio *folio)
  		 * further.
  		 */
  		if (z < 0)
-			return 0;
+			return -EAGAIN;
 
  		wakeup_kswapd(pgdat->node_zones + z, 0,
  			      folio_order(folio), ZONE_MOVABLE);
-		return 0;
+		return -EAGAIN;
  	}
 
  	if (!folio_isolate_lru(folio))
-		return 0;
+		return -EAGAIN;
 
  	node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio),
  			    nr_pages);

mm/migrate.c +5 -31

  /*
   * Attempt to migrate a misplaced folio to the specified destination
- * node. Caller is expected to have an elevated reference count on
- * the folio that will be dropped by this function before returning.
+ * node. Caller is expected to have isolated the folio by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio. This function will un-isolate the
+ * folio, dereferencing the folio before returning.
   */
  int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
  			    int node)
  {
  	pg_data_t *pgdat = NODE_DATA(node);
-	int isolated;
  	int nr_remaining;
  	unsigned int nr_succeeded;
  	LIST_HEAD(migratepages);
  	int nr_pages = folio_nr_pages(folio);
 
-	/*
-	 * Don't migrate file folios that are mapped in multiple processes
-	 * with execute permissions as they are probably shared libraries.
-	 *
-	 * See folio_likely_mapped_shared() on possible imprecision when we
-	 * cannot easily detect if a folio is shared.
-	 */
-	if (folio_likely_mapped_shared(folio) && folio_is_file_lru(folio) &&
-	    (vma->vm_flags & VM_EXEC))
-		goto out;
-
-	/*
-	 * Also do not migrate dirty folios as not all filesystems can move
-	 * dirty folios in MIGRATE_ASYNC mode which is a waste of cycles.
-	 */
-	if (folio_is_file_lru(folio) && folio_test_dirty(folio))
-		goto out;
-
-	isolated = numamigrate_isolate_folio(pgdat, folio);
-	if (!isolated)
-		goto out;
-
  	list_add(&folio->lru, &migratepages);
  	nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
  				     NULL, node, MIGRATE_ASYNC,

@@ -2620,7 +2620,6 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
  					folio_is_file_lru(folio), -nr_pages);
  			folio_putback_lru(folio);
  		}
-		isolated = 0;
  	}
  	if (nr_succeeded) {
  		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);

@@ -2629,11 +2628,7 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
  					    nr_succeeded);
  	}
  	BUG_ON(!list_empty(&migratepages));
-	return isolated ? 0 : -EAGAIN;
-
-out:
-	folio_put(folio);
-	return -EAGAIN;
+	return nr_remaining ? -EAGAIN : 0;
  }
  #endif /* CONFIG_NUMA_BALANCING */
  #endif /* CONFIG_NUMA */

Quoted 16 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

Quoted 4 lines

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

s/golio/folio

Make sense to me. Feel free to add:

Reviewed-by Baolin Wang <[email protected]>

Quoted 6 lines

node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio),
nr_pages);

mm/migrate.c +0 -4

-
-	/*
-	 * Isolating the folio has taken another reference, so the
-	 * caller's reference can be safely dropped without the folio

mm/migrate.c static int numamigrate_isolate_folio(pg_data_t *pgdat, struct folio *folio) · line 2594

Quoted 6 lines

mm/migrate.c +1 -4

-	 * disappearing underneath us during migration.
-	 */
-	folio_put(folio);
-	return 1;
+	return 0;
   }

(just a nit: returning boolean seems more readable)

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Signed-off-by David Hildenbrand <[email protected]>

---
 include/linux/migrate.h |  7 ++++
 mm/huge_memory.c        |  8 ++--
 mm/memory.c             |  9 +++--
 mm/migrate.c            | 81 +++++++++++++++++++----------------------
 4 files changed, 55 insertions(+), 50 deletions(-)

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	if (node_is_toptier(nid))
 		last_cpupid = folio_last_cpupid(folio);
 	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
 		goto out_map;
 	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
 	spin_unlock(vmf->ptl);
 	writable = false;
 

mm/memory.c +0 -0

index 118660de5bcc..4fd1ecfced4d 100644

Quoted 23 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Quoted 7 lines

Signed-off-by David Hildenbrand <[email protected]>

---
  include/linux/migrate.h |  7 ++++
  mm/huge_memory.c        |  8 ++--
  mm/memory.c             |  9 +++--
  mm/migrate.c            | 81 +++++++++++++++++++----------------------
  4 files changed, 55 insertions(+), 50 deletions(-)

LGTM. Reviewed-by: Zi Yan <[email protected]>

One nit below:

<snip>

Quoted 25 lines

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
  	if (node_is_toptier(nid))
  		last_cpupid = folio_last_cpupid(folio);
  	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
  		goto out_map;
  	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
  	spin_unlock(vmf->ptl);
  	writable = false;
 

mm/memory.c +0 -0

index 118660de5bcc..4fd1ecfced4d 100644

<snip>

mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 5351

Quoted 12 lines

mm/memory.c +4 -2

@@ -5345,10 +5343,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
  	else
  		last_cpupid = folio_last_cpupid(folio);
  	target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
  		goto out_map;
  	}

These two locations are repeated code, maybe just merge the ifs into numa_migrate_prep(). Feel free to ignore if you are not going to send another version. :)

--
Best Regards,
Yan, Zi

Quoted 67 lines · Zi Yan

Quoted 23 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Quoted 7 lines

Signed-off-by David Hildenbrand <[email protected]>

---
  include/linux/migrate.h |  7 ++++
  mm/huge_memory.c        |  8 ++--
  mm/memory.c             |  9 +++--
  mm/migrate.c            | 81 +++++++++++++++++++----------------------
  4 files changed, 55 insertions(+), 50 deletions(-)

LGTM. Reviewed-by: Zi Yan <[email protected]>

One nit below:

<snip>

Quoted 25 lines

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
  	if (node_is_toptier(nid))
  		last_cpupid = folio_last_cpupid(folio);
  	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
  		goto out_map;
  	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
  	spin_unlock(vmf->ptl);
  	writable = false;
 

mm/memory.c +0 -0

index 118660de5bcc..4fd1ecfced4d 100644

<snip>

Quoted 16 lines

mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 5351

Quoted 12 lines

mm/memory.c +4 -2

@@ -5345,10 +5343,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
  	else
  		last_cpupid = folio_last_cpupid(folio);
  	target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
  		goto out_map;
  	}

These two locations are repeated code, maybe just merge the ifs into numa_migrate_prep(). Feel free to ignore if you are not going to send another version. :)

I went back and forth a couple of times and

a) Didn't want to move numa_migrate_prep() into migrate_misplaced_folio_prepare(), because having that code in mm/migrate.c felt a bit odd.

b) Didn't want to move migrate_misplaced_folio_prepare() because I enjoy seeing the migrate_misplaced_folio_prepare() and migrate_misplaced_folio() calls in the same callercontext.

I also considered renaming numa_migrate_prep(), but wasn't really able to come up with a good name.

But maybe a) is not too bad?

We'd have migrate_misplaced_folio_prepare() consume &flags and &target_nid, and perform the "flags |= TNF_MIGRATE_FAIL;" internally.

What would be your take?

Quoted 85 lines · David Hildenbrand

Quoted 67 lines · Zi Yan

Quoted 23 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Quoted 7 lines

Signed-off-by David Hildenbrand <[email protected]>

---
   include/linux/migrate.h |  7 ++++
   mm/huge_memory.c        |  8 ++--
   mm/memory.c             |  9 +++--
   mm/migrate.c            | 81 +++++++++++++++++++----------------------
   4 files changed, 55 insertions(+), 50 deletions(-)

LGTM. Reviewed-by: Zi Yan <[email protected]>

One nit below:

<snip>

Quoted 25 lines

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
   	if (node_is_toptier(nid))
   		last_cpupid = folio_last_cpupid(folio);
   	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
   		goto out_map;
   	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
   	spin_unlock(vmf->ptl);
   	writable = false;
 

mm/memory.c +0 -0

index 118660de5bcc..4fd1ecfced4d 100644

<snip>

Quoted 16 lines

mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 5351

Quoted 12 lines

mm/memory.c +4 -2

@@ -5345,10 +5343,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
   	else
   		last_cpupid = folio_last_cpupid(folio);
   	target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
   		goto out_map;
   	}

These two locations are repeated code, maybe just merge the ifs into numa_migrate_prep(). Feel free to ignore if you are not going to send another version. :)

Quoted 5 lines

I went back and forth a couple of times and

a) Didn't want to move numa_migrate_prep() into migrate_misplaced_folio_prepare(), because having that code in mm/migrate.c felt a bit odd.

I agree after checking the actual code, since the code is just updating NUMA fault stats and checking where the folio should be.

Quoted 6 lines

b) Didn't want to move migrate_misplaced_folio_prepare() because I enjoy seeing the migrate_misplaced_folio_prepare() and migrate_misplaced_folio() calls in the same callercontext.

I also considered renaming numa_migrate_prep(), but wasn't really able to come up with a good name.

How about numa_migrate_check()? Since it tells whether a folio should be migrated or not.

Quoted 3 lines

But maybe a) is not too bad?

Quoted 3 lines

We'd have migrate_misplaced_folio_prepare() consume &flags and &target_nid, and perform the "flags |= TNF_MIGRATE_FAIL;" internally.

What would be your take?

I would either rename numa_migrate_prep() or just do nothing. I have to admit that the "prep" and "prepare" in both function names motivated me to propose the merge, but now the actual code tells me they should be separate.

Quoted 109 lines · Zi Yan

Quoted 85 lines · David Hildenbrand

Quoted 67 lines · Zi Yan

Quoted 23 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Quoted 7 lines

Signed-off-by David Hildenbrand <[email protected]>

---
   include/linux/migrate.h |  7 ++++
   mm/huge_memory.c        |  8 ++--
   mm/memory.c             |  9 +++--
   mm/migrate.c            | 81 +++++++++++++++++++----------------------
   4 files changed, 55 insertions(+), 50 deletions(-)

LGTM. Reviewed-by: Zi Yan <[email protected]>

One nit below:

<snip>

Quoted 25 lines

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
   	if (node_is_toptier(nid))
   		last_cpupid = folio_last_cpupid(folio);
   	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
   		goto out_map;
   	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
   	spin_unlock(vmf->ptl);
   	writable = false;
 

mm/memory.c +0 -0

index 118660de5bcc..4fd1ecfced4d 100644

<snip>

Quoted 16 lines

mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 5351

Quoted 12 lines

mm/memory.c +4 -2

@@ -5345,10 +5343,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
   	else
   		last_cpupid = folio_last_cpupid(folio);
   	target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
   		goto out_map;
   	}

These two locations are repeated code, maybe just merge the ifs into numa_migrate_prep(). Feel free to ignore if you are not going to send another version. :)

Quoted 5 lines

I went back and forth a couple of times and

a) Didn't want to move numa_migrate_prep() into migrate_misplaced_folio_prepare(), because having that code in mm/migrate.c felt a bit odd.

I agree after checking the actual code, since the code is just updating NUMA fault stats and checking where the folio should be.

Quoted 6 lines

b) Didn't want to move migrate_misplaced_folio_prepare() because I enjoy seeing the migrate_misplaced_folio_prepare() and migrate_misplaced_folio() calls in the same callercontext.

I also considered renaming numa_migrate_prep(), but wasn't really able to come up with a good name.

How about numa_migrate_check()? Since it tells whether a folio should be migrated or not.

Quoted 3 lines

But maybe a) is not too bad?

Quoted 7 lines

Quoted 3 lines

We'd have migrate_misplaced_folio_prepare() consume &flags and &target_nid, and perform the "flags |= TNF_MIGRATE_FAIL;" internally.

What would be your take?

I would either rename numa_migrate_prep() or just do nothing. I have to admit that the "prep" and "prepare" in both function names motivated me to propose the merge, but now the actual code tells me they should be separate.

Let's leave it like that for now. Renaming to numa_migrate_check() makes sense, and likely moving more numa handling stuff in there.

Bit I yet have to figure out why some of the memory.c vs. huge_memory.c code differences exist, so we can unify them.

For example, why did 33024536bafd9 introduce slightly different last_cpupid handling in do_huge_pmd_numa_page(), whereby it seems like some subtle difference in handling NUMA_BALANCING_MEMORY_TIERING? Maybe I am missing something obvious. :)

Quoted 121 lines · David Hildenbrand

Quoted 109 lines · Zi Yan

Quoted 85 lines · David Hildenbrand

Quoted 67 lines · Zi Yan

Quoted 23 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Quoted 7 lines

Signed-off-by David Hildenbrand <[email protected]>

---
   include/linux/migrate.h |  7 ++++
   mm/huge_memory.c        |  8 ++--
   mm/memory.c             |  9 +++--
   mm/migrate.c            | 81 +++++++++++++++++++----------------------
   4 files changed, 55 insertions(+), 50 deletions(-)

LGTM. Reviewed-by: Zi Yan <[email protected]>

One nit below:

<snip>

Quoted 25 lines

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
   	if (node_is_toptier(nid))
   		last_cpupid = folio_last_cpupid(folio);
   	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
   		goto out_map;
   	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
   	spin_unlock(vmf->ptl);
   	writable = false;
 

mm/memory.c +0 -0

index 118660de5bcc..4fd1ecfced4d 100644

<snip>

Quoted 16 lines

mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 5351

Quoted 12 lines

mm/memory.c +4 -2

@@ -5345,10 +5343,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
   	else
   		last_cpupid = folio_last_cpupid(folio);
   	target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
   		goto out_map;
   	}

These two locations are repeated code, maybe just merge the ifs into numa_migrate_prep(). Feel free to ignore if you are not going to send another version. :)

Quoted 5 lines

I went back and forth a couple of times and

a) Didn't want to move numa_migrate_prep() into migrate_misplaced_folio_prepare(), because having that code in mm/migrate.c felt a bit odd.

I agree after checking the actual code, since the code is just updating NUMA fault stats and checking where the folio should be.

Quoted 6 lines

b) Didn't want to move migrate_misplaced_folio_prepare() because I enjoy seeing the migrate_misplaced_folio_prepare() and migrate_misplaced_folio() calls in the same callercontext.

I also considered renaming numa_migrate_prep(), but wasn't really able to come up with a good name.

How about numa_migrate_check()? Since it tells whether a folio should be migrated or not.

Quoted 3 lines

But maybe a) is not too bad?

Quoted 7 lines

Quoted 3 lines

We'd have migrate_misplaced_folio_prepare() consume &flags and &target_nid, and perform the "flags |= TNF_MIGRATE_FAIL;" internally.

What would be your take?

I would either rename numa_migrate_prep() or just do nothing. I have to admit that the "prep" and "prepare" in both function names motivated me to propose the merge, but now the actual code tells me they should be separate.

Let's leave it like that for now. Renaming to numa_migrate_check() makes sense, and likely moving more numa handling stuff in there.

Quoted 7 lines

Bit I yet have to figure out why some of the memory.c vs. huge_memory.c code differences exist, so we can unify them.

For example, why did 33024536bafd9 introduce slightly different last_cpupid handling in do_huge_pmd_numa_page(), whereby it seems like some subtle difference in handling NUMA_BALANCING_MEMORY_TIERING? Maybe I am missing something obvious. :)

It seems to me that a sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING check is missing in do_huge_pmd_numa_page(). So the

if (node_is_toptier(nid))

should be

if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || node_is_toptier(nid))

to be consistent with other checks. Add Ying to confirm.

I also think a function like

bool folio_has_cpupid(folio)
{
return !(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || node_is_toptier(folio_nid(folio));
}

would be better than the existing checks.

Quoted 135 lines · Zi Yan

Quoted 121 lines · David Hildenbrand

Quoted 109 lines · Zi Yan

Quoted 85 lines · David Hildenbrand

Quoted 67 lines · Zi Yan

Quoted 23 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Quoted 7 lines

Signed-off-by David Hildenbrand <[email protected]>

---
   include/linux/migrate.h |  7 ++++
   mm/huge_memory.c        |  8 ++--
   mm/memory.c             |  9 +++--
   mm/migrate.c            | 81 +++++++++++++++++++----------------------
   4 files changed, 55 insertions(+), 50 deletions(-)

LGTM. Reviewed-by: Zi Yan <[email protected]>

One nit below:

<snip>

Quoted 25 lines

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
   	if (node_is_toptier(nid))
   		last_cpupid = folio_last_cpupid(folio);
   	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
   		goto out_map;
   	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
   	spin_unlock(vmf->ptl);
   	writable = false;
 

mm/memory.c +0 -0

index 118660de5bcc..4fd1ecfced4d 100644

<snip>

Quoted 16 lines

mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 5351

Quoted 12 lines

mm/memory.c +4 -2

@@ -5345,10 +5343,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
   	else
   		last_cpupid = folio_last_cpupid(folio);
   	target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
   		goto out_map;
   	}

These two locations are repeated code, maybe just merge the ifs into numa_migrate_prep(). Feel free to ignore if you are not going to send another version. :)

Quoted 5 lines

I went back and forth a couple of times and

a) Didn't want to move numa_migrate_prep() into migrate_misplaced_folio_prepare(), because having that code in mm/migrate.c felt a bit odd.

I agree after checking the actual code, since the code is just updating NUMA fault stats and checking where the folio should be.

Quoted 6 lines

b) Didn't want to move migrate_misplaced_folio_prepare() because I enjoy seeing the migrate_misplaced_folio_prepare() and migrate_misplaced_folio() calls in the same callercontext.

I also considered renaming numa_migrate_prep(), but wasn't really able to come up with a good name.

How about numa_migrate_check()? Since it tells whether a folio should be migrated or not.

Quoted 3 lines

But maybe a) is not too bad?

Quoted 7 lines

Quoted 3 lines

We'd have migrate_misplaced_folio_prepare() consume &flags and &target_nid, and perform the "flags |= TNF_MIGRATE_FAIL;" internally.

What would be your take?

I would either rename numa_migrate_prep() or just do nothing. I have to admit that the "prep" and "prepare" in both function names motivated me to propose the merge, but now the actual code tells me they should be separate.

Let's leave it like that for now. Renaming to numa_migrate_check() makes sense, and likely moving more numa handling stuff in there.

Quoted 7 lines

Bit I yet have to figure out why some of the memory.c vs. huge_memory.c code differences exist, so we can unify them.

For example, why did 33024536bafd9 introduce slightly different last_cpupid handling in do_huge_pmd_numa_page(), whereby it seems like some subtle difference in handling NUMA_BALANCING_MEMORY_TIERING? Maybe I am missing something obvious. :)

It seems to me that a sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING check is missing in do_huge_pmd_numa_page(). So the

if (node_is_toptier(nid))

Quoted 6 lines

should be

if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || node_is_toptier(nid))

to be consistent with other checks. Add Ying to confirm.

Yes. It should be so. Sorry for my mistake and confusing.

Quoted 9 lines

I also think a function like

bool folio_has_cpupid(folio)
{
return !(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || node_is_toptier(folio_nid(folio));
}

Quoted 1 line

would be better than the existing checks.

Yes. This looks better. Even better, we can add some comments to the function too.

Quoted 137 lines · Ying

Quoted 135 lines · Zi Yan

Quoted 121 lines · David Hildenbrand

Quoted 109 lines · Zi Yan

Quoted 85 lines · David Hildenbrand

Quoted 67 lines · Zi Yan

Quoted 23 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Quoted 7 lines

Signed-off-by David Hildenbrand <[email protected]>

---
    include/linux/migrate.h |  7 ++++
    mm/huge_memory.c        |  8 ++--
    mm/memory.c             |  9 +++--
    mm/migrate.c            | 81 +++++++++++++++++++----------------------
    4 files changed, 55 insertions(+), 50 deletions(-)

LGTM. Reviewed-by: Zi Yan <[email protected]>

One nit below:

<snip>

Quoted 25 lines

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
    	if (node_is_toptier(nid))
    		last_cpupid = folio_last_cpupid(folio);
    	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
    		goto out_map;
    	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
    	spin_unlock(vmf->ptl);
    	writable = false;
 

mm/memory.c +0 -0

index 118660de5bcc..4fd1ecfced4d 100644

<snip>

Quoted 16 lines

mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 5351

Quoted 12 lines

mm/memory.c +4 -2

@@ -5345,10 +5343,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
    	else
    		last_cpupid = folio_last_cpupid(folio);
    	target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
    		goto out_map;
    	}

These two locations are repeated code, maybe just merge the ifs into numa_migrate_prep(). Feel free to ignore if you are not going to send another version. :)

Quoted 5 lines

I went back and forth a couple of times and

a) Didn't want to move numa_migrate_prep() into migrate_misplaced_folio_prepare(), because having that code in mm/migrate.c felt a bit odd.

I agree after checking the actual code, since the code is just updating NUMA fault stats and checking where the folio should be.

Quoted 6 lines

b) Didn't want to move migrate_misplaced_folio_prepare() because I enjoy seeing the migrate_misplaced_folio_prepare() and migrate_misplaced_folio() calls in the same callercontext.

I also considered renaming numa_migrate_prep(), but wasn't really able to come up with a good name.

How about numa_migrate_check()? Since it tells whether a folio should be migrated or not.

Quoted 3 lines

But maybe a) is not too bad?

Quoted 7 lines

Quoted 3 lines

We'd have migrate_misplaced_folio_prepare() consume &flags and &target_nid, and perform the "flags |= TNF_MIGRATE_FAIL;" internally.

What would be your take?

I would either rename numa_migrate_prep() or just do nothing. I have to admit that the "prep" and "prepare" in both function names motivated me to propose the merge, but now the actual code tells me they should be separate.

Let's leave it like that for now. Renaming to numa_migrate_check() makes sense, and likely moving more numa handling stuff in there.

Quoted 7 lines

Bit I yet have to figure out why some of the memory.c vs. huge_memory.c code differences exist, so we can unify them.

For example, why did 33024536bafd9 introduce slightly different last_cpupid handling in do_huge_pmd_numa_page(), whereby it seems like some subtle difference in handling NUMA_BALANCING_MEMORY_TIERING? Maybe I am missing something obvious. :)

It seems to me that a sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING check is missing in do_huge_pmd_numa_page(). So the

if (node_is_toptier(nid))

Quoted 8 lines

Quoted 6 lines

should be

if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || node_is_toptier(nid))

to be consistent with other checks. Add Ying to confirm.

Yes. It should be so. Sorry for my mistake and confusing.

Thank you for the confirmation.

Quoted 9 lines

Quoted 9 lines

I also think a function like

bool folio_has_cpupid(folio)
{
return !(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || node_is_toptier(folio_nid(folio));
}

Quoted 4 lines

Quoted 1 line

would be better than the existing checks.

Yes. This looks better. Even better, we can add some comments to the function too.

I will prepare a patch about it.

Quoted 159 lines · Zi Yan

Quoted 137 lines · Ying

Quoted 135 lines · Zi Yan

Quoted 121 lines · David Hildenbrand

Quoted 109 lines · Zi Yan

Quoted 85 lines · David Hildenbrand

Quoted 67 lines · Zi Yan

Quoted 23 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Quoted 7 lines

Signed-off-by David Hildenbrand <[email protected]>

---
    include/linux/migrate.h |  7 ++++
    mm/huge_memory.c        |  8 ++--
    mm/memory.c             |  9 +++--
    mm/migrate.c            | 81 +++++++++++++++++++----------------------
    4 files changed, 55 insertions(+), 50 deletions(-)

LGTM. Reviewed-by: Zi Yan <[email protected]>

One nit below:

<snip>

Quoted 25 lines

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
    	if (node_is_toptier(nid))
    		last_cpupid = folio_last_cpupid(folio);
    	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
    		goto out_map;
    	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
    	spin_unlock(vmf->ptl);
    	writable = false;
 

mm/memory.c +0 -0

index 118660de5bcc..4fd1ecfced4d 100644

<snip>

Quoted 16 lines

mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 5351

Quoted 12 lines

mm/memory.c +4 -2

@@ -5345,10 +5343,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
    	else
    		last_cpupid = folio_last_cpupid(folio);
    	target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
    		goto out_map;
    	}

These two locations are repeated code, maybe just merge the ifs into numa_migrate_prep(). Feel free to ignore if you are not going to send another version. :)

Quoted 5 lines

I went back and forth a couple of times and

a) Didn't want to move numa_migrate_prep() into migrate_misplaced_folio_prepare(), because having that code in mm/migrate.c felt a bit odd.

I agree after checking the actual code, since the code is just updating NUMA fault stats and checking where the folio should be.

Quoted 6 lines

b) Didn't want to move migrate_misplaced_folio_prepare() because I enjoy seeing the migrate_misplaced_folio_prepare() and migrate_misplaced_folio() calls in the same callercontext.

I also considered renaming numa_migrate_prep(), but wasn't really able to come up with a good name.

How about numa_migrate_check()? Since it tells whether a folio should be migrated or not.

Quoted 3 lines

But maybe a) is not too bad?

Quoted 7 lines

Quoted 3 lines

We'd have migrate_misplaced_folio_prepare() consume &flags and &target_nid, and perform the "flags |= TNF_MIGRATE_FAIL;" internally.

What would be your take?

I would either rename numa_migrate_prep() or just do nothing. I have to admit that the "prep" and "prepare" in both function names motivated me to propose the merge, but now the actual code tells me they should be separate.

Let's leave it like that for now. Renaming to numa_migrate_check() makes sense, and likely moving more numa handling stuff in there.

Quoted 7 lines

Bit I yet have to figure out why some of the memory.c vs. huge_memory.c code differences exist, so we can unify them.

For example, why did 33024536bafd9 introduce slightly different last_cpupid handling in do_huge_pmd_numa_page(), whereby it seems like some subtle difference in handling NUMA_BALANCING_MEMORY_TIERING? Maybe I am missing something obvious. :)

It seems to me that a sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING check is missing in do_huge_pmd_numa_page(). So the

if (node_is_toptier(nid))

Quoted 8 lines

Quoted 6 lines

should be

if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || node_is_toptier(nid))

to be consistent with other checks. Add Ying to confirm.

Yes. It should be so. Sorry for my mistake and confusing.

Thank you for the confirmation.

Quoted 9 lines

Quoted 9 lines

I also think a function like

bool folio_has_cpupid(folio)
{
return !(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || node_is_toptier(folio_nid(folio));
}

Quoted 6 lines

Quoted 4 lines

Quoted 1 line

would be better than the existing checks.

Yes. This looks better. Even better, we can add some comments to the function too.

I will prepare a patch about it.

Do you have capacity to further consolidate the logic, maybe moving more stuff into the numa_migrate_prep (and renaming it? :)).

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Signed-off-by David Hildenbrand <[email protected]>

---
  include/linux/migrate.h |  7 ++++
  mm/huge_memory.c        |  8 ++--
  mm/memory.c             |  9 +++--
  mm/migrate.c            | 81 +++++++++++++++++++----------------------
  4 files changed, 55 insertions(+), 50 deletions(-)

include/linux/migrate.h +7 -0

index f9d92482d117..644be30b69c8 100644
@@ -139,9 +139,16 @@ const struct movable_operations *page_movable_ops(struct page *page)
  }
 
  #ifdef CONFIG_NUMA_BALANCING
+int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node);
  int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
  			   int node);
  #else
+static inline int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
  static inline int migrate_misplaced_folio(struct folio *folio,
  					 struct vm_area_struct *vma, int node)
  {

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
  	if (node_is_toptier(nid))
  		last_cpupid = folio_last_cpupid(folio);
  	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
  		goto out_map;
  	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
  	spin_unlock(vmf->ptl);
  	writable = false;
 

mm/memory.c +5 -4

index 118660de5bcc..4fd1ecfced4d 100644
@@ -5207,8 +5207,6 @@ int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf,
  {
  	struct vm_area_struct *vma = vmf->vma;
 
-	folio_get(folio);
-
  	/* Record the current PID acceesing VMA */
  	vma_set_access_pid_bit(vma);
 

@@ -5345,10 +5343,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
  	else
  		last_cpupid = folio_last_cpupid(folio);
  	target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
  		goto out_map;
  	}
+	/* The folio is isolated and isolation code holds a folio reference. */
  	pte_unmap_unlock(vmf->pte, vmf->ptl);
  	writable = false;
  	ignore_writable = true;

mm/migrate.c +37 -38

index 0307b54879a0..27f070f64f27 100644
@@ -2530,9 +2530,37 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,
  	return __folio_alloc_node(gfp, order, nid);
  }
 
-static int numamigrate_isolate_folio(pg_data_t *pgdat, struct folio *folio)
+/*
+ * Prepare for calling migrate_misplaced_folio() by isolating the folio if
+ * permitted. Must be called with the PTL still held.
+ */
+int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node)
  {
  	int nr_pages = folio_nr_pages(folio);
+	pg_data_t *pgdat = NODE_DATA(node);
+
+	if (folio_is_file_lru(folio)) {
+		/*
+		 * Do not migrate file folios that are mapped in multiple
+		 * processes with execute permissions as they are probably
+		 * shared libraries.
+		 *
+		 * See folio_likely_mapped_shared() on possible imprecision
+		 * when we cannot easily detect if a folio is shared.
+		 */
+		if ((vma->vm_flags & VM_EXEC) &&
+		    folio_likely_mapped_shared(folio))
+			return -EACCES;
+
+		/*
+		 * Do not migrate dirty folios as not all filesystems can move
+		 * dirty folios in MIGRATE_ASYNC mode which is a waste of
+		 * cycles.
+		 */
+		if (folio_test_dirty(folio))
+			return -EAGAIN;
+	}
 
  	/* Avoid migrating to a node that is nearly full */
  	if (!migrate_balanced_pgdat(pgdat, nr_pages)) {

@@ -2550,65 +2578,37 @@ static int numamigrate_isolate_folio(pg_data_t *pgdat, struct folio *folio)
  		 * further.
  		 */
  		if (z < 0)
-			return 0;
+			return -EAGAIN;
 
  		wakeup_kswapd(pgdat->node_zones + z, 0,
  			      folio_order(folio), ZONE_MOVABLE);
-		return 0;
+		return -EAGAIN;
  	}
 
  	if (!folio_isolate_lru(folio))
-		return 0;
+		return -EAGAIN;
 
  	node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio),
  			    nr_pages);
-
-	/*
-	 * Isolating the folio has taken another reference, so the
-	 * caller's reference can be safely dropped without the folio
-	 * disappearing underneath us during migration.
-	 */
-	folio_put(folio);
-	return 1;
+	return 0;
  }
 
  /*
   * Attempt to migrate a misplaced folio to the specified destination
- * node. Caller is expected to have an elevated reference count on
- * the folio that will be dropped by this function before returning.
+ * node. Caller is expected to have isolated the folio by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio. This function will un-isolate the
+ * folio, dereferencing the folio before returning.
   */
  int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
  			    int node)
  {
  	pg_data_t *pgdat = NODE_DATA(node);
-	int isolated;
  	int nr_remaining;
  	unsigned int nr_succeeded;
  	LIST_HEAD(migratepages);
  	int nr_pages = folio_nr_pages(folio);
 
-	/*
-	 * Don't migrate file folios that are mapped in multiple processes
-	 * with execute permissions as they are probably shared libraries.
-	 *
-	 * See folio_likely_mapped_shared() on possible imprecision when we
-	 * cannot easily detect if a folio is shared.
-	 */
-	if (folio_likely_mapped_shared(folio) && folio_is_file_lru(folio) &&
-	    (vma->vm_flags & VM_EXEC))
-		goto out;
-
-	/*
-	 * Also do not migrate dirty folios as not all filesystems can move
-	 * dirty folios in MIGRATE_ASYNC mode which is a waste of cycles.
-	 */
-	if (folio_is_file_lru(folio) && folio_test_dirty(folio))
-		goto out;
-
-	isolated = numamigrate_isolate_folio(pgdat, folio);
-	if (!isolated)
-		goto out;
-
  	list_add(&folio->lru, &migratepages);
  	nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
  				     NULL, node, MIGRATE_ASYNC,

@@ -2620,7 +2620,6 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
  					folio_is_file_lru(folio), -nr_pages);
  			folio_putback_lru(folio);
  		}
-		isolated = 0;
  	}
  	if (nr_succeeded) {
  		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);

@@ -2629,11 +2628,7 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
  					    nr_succeeded);
  	}

Quoted 230 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Signed-off-by David Hildenbrand <[email protected]>

---
   include/linux/migrate.h |  7 ++++
   mm/huge_memory.c        |  8 ++--
   mm/memory.c             |  9 +++--
   mm/migrate.c            | 81 +++++++++++++++++++----------------------
   4 files changed, 55 insertions(+), 50 deletions(-)

include/linux/migrate.h +7 -0

index f9d92482d117..644be30b69c8 100644
@@ -139,9 +139,16 @@ const struct movable_operations *page_movable_ops(struct page *page)
   }
 
   #ifdef CONFIG_NUMA_BALANCING
+int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node);
   int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
   			   int node);
   #else
+static inline int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
   static inline int migrate_misplaced_folio(struct folio *folio,
   					 struct vm_area_struct *vma, int node)
   {

mm/huge_memory.c +5 -3

index fc27dabcd8e3..4b2817bb2c7d 100644
@@ -1688,11 +1688,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
   	if (node_is_toptier(nid))
   		last_cpupid = folio_last_cpupid(folio);
   	target_nid = numa_migrate_prep(folio, vmf, haddr, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
   		goto out_map;
   	}
-
+	/* The folio is isolated and isolation code holds a folio reference. */
   	spin_unlock(vmf->ptl);
   	writable = false;
 

mm/memory.c +5 -4

index 118660de5bcc..4fd1ecfced4d 100644
@@ -5207,8 +5207,6 @@ int numa_migrate_prep(struct folio *folio, struct vm_fault *vmf,
   {
   	struct vm_area_struct *vma = vmf->vma;
 
-	folio_get(folio);
-
   	/* Record the current PID acceesing VMA */
   	vma_set_access_pid_bit(vma);
 

@@ -5345,10 +5343,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
   	else
   		last_cpupid = folio_last_cpupid(folio);
   	target_nid = numa_migrate_prep(folio, vmf, vmf->address, nid, &flags);
-	if (target_nid == NUMA_NO_NODE) {
-		folio_put(folio);
+	if (target_nid == NUMA_NO_NODE)
+		goto out_map;
+	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+		flags |= TNF_MIGRATE_FAIL;
   		goto out_map;
   	}
+	/* The folio is isolated and isolation code holds a folio reference. */
   	pte_unmap_unlock(vmf->pte, vmf->ptl);
   	writable = false;
   	ignore_writable = true;

mm/migrate.c +37 -38

index 0307b54879a0..27f070f64f27 100644
@@ -2530,9 +2530,37 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,
   	return __folio_alloc_node(gfp, order, nid);
   }
 
-static int numamigrate_isolate_folio(pg_data_t *pgdat, struct folio *folio)
+/*
+ * Prepare for calling migrate_misplaced_folio() by isolating the folio if
+ * permitted. Must be called with the PTL still held.
+ */
+int migrate_misplaced_folio_prepare(struct folio *folio,
+		struct vm_area_struct *vma, int node)
   {
   	int nr_pages = folio_nr_pages(folio);
+	pg_data_t *pgdat = NODE_DATA(node);
+
+	if (folio_is_file_lru(folio)) {
+		/*
+		 * Do not migrate file folios that are mapped in multiple
+		 * processes with execute permissions as they are probably
+		 * shared libraries.
+		 *
+		 * See folio_likely_mapped_shared() on possible imprecision
+		 * when we cannot easily detect if a folio is shared.
+		 */
+		if ((vma->vm_flags & VM_EXEC) &&
+		    folio_likely_mapped_shared(folio))
+			return -EACCES;
+
+		/*
+		 * Do not migrate dirty folios as not all filesystems can move
+		 * dirty folios in MIGRATE_ASYNC mode which is a waste of
+		 * cycles.
+		 */
+		if (folio_test_dirty(folio))
+			return -EAGAIN;
+	}
 
   	/* Avoid migrating to a node that is nearly full */
   	if (!migrate_balanced_pgdat(pgdat, nr_pages)) {

@@ -2550,65 +2578,37 @@ static int numamigrate_isolate_folio(pg_data_t *pgdat, struct folio *folio)
   		 * further.
   		 */
   		if (z < 0)
-			return 0;
+			return -EAGAIN;
 
   		wakeup_kswapd(pgdat->node_zones + z, 0,
   			      folio_order(folio), ZONE_MOVABLE);
-		return 0;
+		return -EAGAIN;
   	}
 
   	if (!folio_isolate_lru(folio))
-		return 0;
+		return -EAGAIN;
 
   	node_stat_mod_folio(folio, NR_ISOLATED_ANON + folio_is_file_lru(folio),
   			    nr_pages);
-
-	/*
-	 * Isolating the folio has taken another reference, so the
-	 * caller's reference can be safely dropped without the folio
-	 * disappearing underneath us during migration.
-	 */
-	folio_put(folio);
-	return 1;
+	return 0;
   }
 
   /*
    * Attempt to migrate a misplaced folio to the specified destination
- * node. Caller is expected to have an elevated reference count on
- * the folio that will be dropped by this function before returning.
+ * node. Caller is expected to have isolated the folio by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio. This function will un-isolate the
+ * folio, dereferencing the folio before returning.
    */
   int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
   			    int node)
   {
   	pg_data_t *pgdat = NODE_DATA(node);
-	int isolated;
   	int nr_remaining;
   	unsigned int nr_succeeded;
   	LIST_HEAD(migratepages);
   	int nr_pages = folio_nr_pages(folio);
 
-	/*
-	 * Don't migrate file folios that are mapped in multiple processes
-	 * with execute permissions as they are probably shared libraries.
-	 *
-	 * See folio_likely_mapped_shared() on possible imprecision when we
-	 * cannot easily detect if a folio is shared.
-	 */
-	if (folio_likely_mapped_shared(folio) && folio_is_file_lru(folio) &&
-	    (vma->vm_flags & VM_EXEC))
-		goto out;
-
-	/*
-	 * Also do not migrate dirty folios as not all filesystems can move
-	 * dirty folios in MIGRATE_ASYNC mode which is a waste of cycles.
-	 */
-	if (folio_is_file_lru(folio) && folio_test_dirty(folio))
-		goto out;
-
-	isolated = numamigrate_isolate_folio(pgdat, folio);
-	if (!isolated)
-		goto out;
-
   	list_add(&folio->lru, &migratepages);
   	nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
   				     NULL, node, MIGRATE_ASYNC,

@@ -2620,7 +2620,6 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
   					folio_is_file_lru(folio), -nr_pages);
   			folio_putback_lru(folio);
   		}
-		isolated = 0;
   	}
   	if (nr_succeeded) {
   		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);

@@ -2629,11 +2628,7 @@ int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma,
   					    nr_succeeded);
   	}

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, struct vm_area_struct *vma, · line 2634

Quoted 10 lines

mm/migrate.c +1 -5

   	BUG_ON(!list_empty(&migratepages));
-	return isolated ? 0 : -EAGAIN;
-
-out:
-	folio_put(folio);
-	return -EAGAIN;
+	return nr_remaining ? -EAGAIN : 0;
   }
   #endif /* CONFIG_NUMA_BALANCING */
   #endif /* CONFIG_NUMA */

Hi David,

I have tested these patches on my system. My system has 2 DRAM nodes and 1 PMEM node.
I tested page migration between DRAM nodes and page promotion from PMEM to DRAM.
Both are working fine.

below are the results.

Migration results
=============
numa_pte_updates 18977
numa_huge_pte_updates 0
numa_hint_faults 18504
numa_hint_faults_local 2116
numa_pages_migrated 16388

Promotion Results
===============
nr_sec_page_table_pages 0
nr_iommu_pages 0
nr_swapcached 0
pgpromote_success 16386
pgpromote_candidate 0

Tested-by Donet Tom <[email protected]>

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

While at it, combine the folio_is_file_lru() checks.

Signed-off-by David Hildenbrand <[email protected]>

---

Quoted 21 lines · David Hildenbrand

Currently we always take a folio reference even if migration will not even be tried or isolation failed, requiring us to grab+drop an additional reference.

Further, we end up calling folio_likely_mapped_shared() while the folio might have already been unmapped, because after we dropped the PTL, that can easily happen. We want to stop touching mapcounts and friends from such context, and only call folio_likely_mapped_shared() while the folio is still mapped: mapcount information is pretty much stale and unreliable otherwise.

So let's move checks into numamigrate_isolate_folio(), rename that function to migrate_misplaced_folio_prepare(), and call that function from callsites where we call migrate_misplaced_folio(), but still with the PTL held.

We can now stop taking temporary folio references, and really only take a reference if folio isolation succeeded. Doing the folio_likely_mapped_shared() + golio isolation under PT lock is now similar to how we handle MADV_PAGEOUT.

Quoted 4 lines

While at it, combine the folio_is_file_lru() checks.

Signed-off-by David Hildenbrand <[email protected]>

---

Donet just reported an issue. I suspect this fixes it -- in any case, this is the right thing to do.

From 0833b9896e98c8d88c521609c811a220d14a2e7e Mon Sep 17 00:00:00 2001
From: David Hildenbrand <[email protected]>
Date: Wed, 26 Jun 2024 18:14:44 +0200
Subject: [PATCH] Fixup: mm/migrate: move NUMA hinting fault folio isolation + checks under PTL

Donet reports an issue during NUMA migration we haven't seen previously:

    [   71.422804] list_del corruption, c00c00000061b3c8->next is LIST_POISON1 (5deadbeef0000100)
    [   71.422839] ------------[ cut here ]------------ [   71.422843] kernel BUG at lib/list_debug.c:56!
    [   71.422850] Oops: Exception in kernel mode, sig: 5 [#1]

We forgot to convert one "return 0;" to return an error instead from migrate_misplaced_folio_prepare() in case the target node is nearly full.

Signed-off-by David Hildenbrand <[email protected]>

---
 mm/migrate.c | 2 +-

 1 file changed, 1 insertion(+), 1 deletion(-)

mm/migrate.c +1 -1

index 8beedbb42a93..9ed43c1eea5e 100644
@@ -2564,7 +2564,7 @@ int migrate_misplaced_folio_prepare(struct folio 
 *folio,
          int z;
 
          if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING))
-            return 0;
+            return -EAGAIN;
          for (z = pgdat->nr_zones - 1; z >= 0; z--) {
              if (managed_zone(pgdat->node_zones + z))

                 break;

base-commit: 4b17ce353e02b47b00e2fe87b862f09e8b9a47e6