Migrate Pages in lieu of discard

From: Dave Hansen <[email protected]>

When memory fills up on a node, memory contents can be automatically migrated to another node. The biggest problems are knowing when to migrate and to where the migration should be targeted.

The most straightforward way to generate the "to where" list would be to follow the page allocator fallback lists. Those lists already tell us if memory is full where to look next. It would also be logical to move memory in that order.

But, the allocator fallback lists have a fatal flaw: most nodes appear in all the lists. This would potentially lead to migration cycles (A->B, B->A, A->B, ...).

Instead of using the allocator fallback lists directly, keep a separate node migration ordering. But, reuse the same data used to generate page allocator fallback in the first place:
find_next_best_node().

This means that the firmware data used to populate node distances essentially dictates the ordering for now. It should also be architecture-neutral since all NUMA architectures have a working find_next_best_node().

The protocol for node_demotion[] access and writing is not standard. It has no specific locking and is intended to be read locklessly. Readers must take care to avoid observing changes that appear incoherent. This was done so that node_demotion[] locking has no chance of becoming a bottleneck on large systems with lots of CPUs in direct reclaim.

This code is unused for now. It will be called later in the series.

Signed-off-by Dave Hansen <[email protected]>

Cc Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

---

 b/mm/internal.h   |    1
 b/mm/migrate.c    |  137 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 b/mm/page_alloc.c |    2
 3 files changed, 138 insertions(+), 2 deletions(-)

diff -puN mm/internal.h~auto-setup-default-migration-path-from-firmware mm/internal.h

mm/internal.h 2020-08-18 11:36:48.968583188 -0700 renamed from mm/internal.h~auto-setup-default-migration-path-from-firmware 2020-08-18 11:36:48.960583188 -0700 +1 -0

@@ -203,6 +203,7 @@ extern int user_min_free_kbytes;
 
 extern void zone_pcp_update(struct zone *zone);
 extern void zone_pcp_reset(struct zone *zone);
+extern int find_next_best_node(int node, nodemask_t *used_node_mask);
 
 #if defined CONFIG_COMPACTION || defined CONFIG_CMA
 

diff -puN mm/migrate.c~auto-setup-default-migration-path-from-firmware mm/migrate.c

mm/migrate.c 2020-08-18 11:36:48.970583188 -0700 renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware 2020-08-18 11:36:48.962583188 -0700 +85 -1

@@ -1160,6 +1160,10 @@ out:
        return rc;
 }
 
+/*
+ * Writes to this array occur without locking.  READ_ONCE()
+ * is recommended for readers to ensure consistent reads.
+ */
 static int node_demotion[MAX_NUMNODES] = {[0 ...  MAX_NUMNODES - 1] = NUMA_NO_NODE};
 
 /**

@@ -1173,7 +1177,13 @@ static int node_demotion[MAX_NUMNODES] =
  */
 int next_demotion_node(int node)
 {
-       return node_demotion[node];
+       /*
+        * node_demotion[] is updated without excluding
+        * this function from running.  READ_ONCE() avoids
+        * reading multiple, inconsistent 'node' values
+        * during an update.
+        */
+       return READ_ONCE(node_demotion[node]);
 }
 
 /*

@@ -3001,3 +3011,128 @@ void migrate_vma_finalize(struct migrate
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
 #endif /* CONFIG_DEVICE_PRIVATE */
+
+/* Disable reclaim-based migration. */
+static void disable_all_migrate_targets(void)
+{
+       int node;
+
+       for_each_online_node(node)
+               node_demotion[node] = NUMA_NO_NODE;
+}
+
+/*
+ * Find an automatic demotion target for 'node'.
+ * Failing here is OK.  It might just indicate
+ * being at the end of a chain.
+ */
+static int establish_migrate_target(int node, nodemask_t *used)
+{
+       int migration_target;
+
+       /*
+        * Can not set a migration target on a
+        * node with it already set.
+        *
+        * No need for READ_ONCE() here since this
+        * in the write path for node_demotion[].
+        * This should be the only thread writing.
+        */
+       if (node_demotion[node] != NUMA_NO_NODE)
+               return NUMA_NO_NODE;
+
+       migration_target = find_next_best_node(node, used);
+       if (migration_target == NUMA_NO_NODE)
+               return NUMA_NO_NODE;
+
+       node_demotion[node] = migration_target;
+
+       return migration_target;
+}
+
+/*
+ * When memory fills up on a node, memory contents can be
+ * automatically migrated to another node instead of
+ * discarded at reclaim.
+ *
+ * Establish a "migration path" which will start at nodes
+ * with CPUs and will follow the priorities used to build the
+ * page allocator zonelists.
+ *
+ * The difference here is that cycles must be avoided.  If
+ * node0 migrates to node1, then neither node1, nor anything
+ * node1 migrates to can migrate to node0.
+ *
+ * This function can run simultaneously with readers of
+ * node_demotion[].  However, it can not run simultaneously
+ * with itself.  Exclusion is provided by memory hotplug events
+ * being single-threaded.
+ */
+void __set_migration_target_nodes(void)
+{
+       nodemask_t next_pass    = NODE_MASK_NONE;
+       nodemask_t this_pass    = NODE_MASK_NONE;
+       nodemask_t used_targets = NODE_MASK_NONE;
+       int node;
+
+       /*
+        * Avoid any oddities like cycles that could occur
+        * from changes in the topology.  This will leave
+        * a momentary gap when migration is disabled.
+        */
+       disable_all_migrate_targets();
+
+       /*
+        * Ensure that the "disable" is visible across the system.
+        * Readers will see either a combination of before+disable

mm/migrate.c 2020-08-18 11:36:48.970583188 -0700 renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware 2020-08-18 11:36:48.962583188 -0700 +41 -0

+
+       /*
+        * Allocations go close to CPUs, first.  Assume that
+        * the migration path starts at the nodes with CPUs.
+        */
+       next_pass = node_states[N_CPU];
+again:
+       this_pass = next_pass;
+       next_pass = NODE_MASK_NONE;
+       /*
+        * To avoid cycles in the migration "graph", ensure
+        * that migration sources are not future targets by
+        * setting them in 'used_targets'.  Do this only
+        * once per pass so that multiple source nodes can
+        * share a target node.
+        *
+        * 'used_targets' will become unavailable in future
+        * passes.  This limits some opportunities for
+        * multiple source nodes to share a desintation.
+        */
+       nodes_or(used_targets, used_targets, this_pass);
+       for_each_node_mask(node, this_pass) {
+               int target_node = establish_migrate_target(node, &used_targets);
+
+               if (target_node == NUMA_NO_NODE)
+                       continue;
+
+               /* Visit targets from this pass in the next pass: */
+               node_set(target_node, next_pass);
+       }
+       /* Is another pass necessary? */
+       if (!nodes_empty(next_pass))
+               goto again;
+}
+
+void set_migration_target_nodes(void)
+{
+       get_online_mems();
+       __set_migration_target_nodes();
+       put_online_mems();
+}

diff -puN mm/page_alloc.c~auto-setup-default-migration-path-from-firmware mm/page_alloc.c

mm/page_alloc.c 2020-08-18 11:36:48.972583188 -0700 renamed from mm/page_alloc.c~auto-setup-default-migration-path-from-firmware 2020-08-18 11:36:48.964583188 -0700 +1 -1

@@ -5607,7 +5607,7 @@ static int node_load[MAX_NUMNODES];
  *
  * Return: node id of the found node or %NUMA_NO_NODE if no node is found.
  */
-static int find_next_best_node(int node, nodemask_t *used_node_mask)
+int find_next_best_node(int node, nodemask_t *used_node_mask)
 {
        int n, val;
        int min_val = INT_MAX;

_

From: Dave Hansen <[email protected]>

Reclaim-based migration is attempting to optimize data placement in memory based on the system topology. If the system changes, so must the migration ordering.

The implementation here is pretty simple and entirely unoptimized. On any memory or CPU hotplug events, assume that a node was added or removed and recalculate all migration targets. This ensures that the node_demotion[] array is always ready to be used in case the new reclaim mode is enabled.

This recalculation is far from optimal, most glaringly that it does not even attempt to figure out if nodes are actually coming or going. But, given the expected paucity of hotplug events, this should be fine.

Signed-off-by Dave Hansen <[email protected]>

Cc Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

---

 b/mm/migrate.c |   93 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)

diff -puN mm/migrate.c~enable-numa-demotion mm/migrate.c

mm/migrate.c 2020-08-18 11:36:50.197583185 -0700 renamed from mm/migrate.c~enable-numa-demotion 2020-08-18 11:36:50.193583185 -0700 +3 -0

@@ -49,6 +49,7 @@
 #include <linux/sched/mm.h>
 #include <linux/ptrace.h>
 #include <linux/oom.h>
+#include <linux/memory.h>
 
 #include <asm/tlbflush.h>
 

@@ -3130,9 +3131,101 @@ again:
                goto again;
 }
 
+/*
+ * For callers that do not hold get_online_mems() already.

mm/migrate.c 2020-08-18 11:36:50.197583185 -0700 renamed from mm/migrate.c~enable-numa-demotion 2020-08-18 11:36:50.193583185 -0700 +11 -0

+ * like events that online or offline NUMA nodes.
+ *
+ * The ordering is also currently dependent on which nodes have
+ * CPUs.  That means we need CPU on/offline notification too.
+ */
+static int migration_online_cpu(unsigned int cpu)
+{
+       set_migration_target_nodes();
+       return 0;
+}
+

mm/migrate.c 2020-08-18 11:36:50.197583185 -0700 renamed from mm/migrate.c~enable-numa-demotion 2020-08-18 11:36:50.193583185 -0700 +63 -0

+ * migration targets.
+ *
+ * These callbacks already hold get_online_mems().  That
+ * is why __set_migration_target_nodes() can be used as
+ * opposed to set_migration_target_nodes().
+ */
+#if defined(CONFIG_MEMORY_HOTPLUG)
+static int __meminit migrate_on_reclaim_callback(struct notifier_block *self,
+                                                unsigned long action, void *arg)
+{
+       switch (action) {
+       case MEM_GOING_OFFLINE:
+               /*
+                * Make sure there are not transient states where
+                * an offline node is a migration target.  This
+                * will leave migration disabled until the offline
+                * completes and the MEM_OFFLINE case below runs.
+                */
+               disable_all_migrate_targets();
+               break;
+       case MEM_OFFLINE:
+       case MEM_ONLINE:
+               /*
+                * Recalculate the target nodes once the node
+                * reaches its final state (online or offline).
+                */
+               __set_migration_target_nodes();
+               break;
+       case MEM_CANCEL_OFFLINE:
+               /*
+                * MEM_GOING_OFFLINE disabled all the migration
+                * targets.  Reenable them.
+                */
+               __set_migration_target_nodes();
+               break;
+       case MEM_GOING_ONLINE:
+       case MEM_CANCEL_ONLINE:
+               break;
+       }
+
+       return notifier_from_errno(0);
+}
+
+static int __init migrate_on_reclaim_init(void)
+{
+       int ret;
+
+       ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "migrate on reclaim",
+                               migration_online_cpu,
+                               migration_offline_cpu);
+       /*
+        * In the unlikely case that this fails, the automatic
+        * migration targets may become suboptimal for nodes
+        * where N_CPU changes.  With such a small impact in a
+        * rare case, do not bother trying to do anything special.
+        */
+       WARN_ON(ret < 0);
+
+       hotplug_memory_notifier(migrate_on_reclaim_callback, 100);
+       return 0;
+}
+late_initcall(migrate_on_reclaim_init);
+#endif /* CONFIG_MEMORY_HOTPLUG */

_

diff -puN mm/migrate.c~migrate_pages-add-success-return mm/migrate.c

mm/migrate.c renamed from mm/migrate.c~migrate_pages-add-success-return +2 -2

@@ -1432,6 +1432,7 @@ out:
  * @mode:		The migration mode that specifies the constraints for
  *			page migration, if any.
  * @reason:		The reason for page migration.
+ * @nr_succeeded:	The number of pages migrated successfully.
  *
  * The function returns after 10 attempts or if no pages are movable any more
  * because the list has become empty or no retryable pages exist any more.

@@ -1442,11 +1443,10 @@ out:
  */
 int migrate_pages(struct list_head *from, new_page_t get_new_page,
 		free_page_t put_new_page, unsigned long private,
-		enum migrate_mode mode, int reason)
+		enum migrate_mode mode, int reason, unsigned int *nr_succeeded)
 {
 	int retry = 1;
 	int nr_failed = 0;
-	int nr_succeeded = 0;
 	int pass = 0;
 	struct page *page;
 	struct page *page2;

mm/migrate.c renamed from mm/migrate.c~migrate_pages-add-success-return +3 -3

 				break;
 			default:
 				/*

@@ -1517,11 +1517,11 @@ retry:
 	nr_failed += retry;
 	rc = nr_failed;
 out:
-	if (nr_succeeded)
-		count_vm_events(PGMIGRATE_SUCCESS, nr_succeeded);
+	if (*nr_succeeded)
+		count_vm_events(PGMIGRATE_SUCCESS, *nr_succeeded);
 	if (nr_failed)
 		count_vm_events(PGMIGRATE_FAIL, nr_failed);
-	trace_mm_migrate_pages(nr_succeeded, nr_failed, mode, reason);
+	trace_mm_migrate_pages(*nr_succeeded, nr_failed, mode, reason);
 
 	if (!swapwrite)
 		current->flags &= ~PF_SWAPWRITE;

diff -puN mm/vmscan.c~demote-with-migrate_pages mm/vmscan.c

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +93 -0

@@ -43,6 +43,7 @@
 #include <linux/kthread.h>
 #include <linux/freezer.h>
 #include <linux/memcontrol.h>
+#include <linux/migrate.h>
 #include <linux/delayacct.h>
 #include <linux/sysctl.h>
 #include <linux/oom.h>

@@ -1040,6 +1041,24 @@ static enum page_references page_check_r
 	return PAGEREF_RECLAIM;
 }
 
+bool migrate_demote_page_ok(struct page *page, struct scan_control *sc)
+{
+	int next_nid = next_demotion_node(page_to_nid(page));
+
+	VM_BUG_ON_PAGE(!PageLocked(page), page);
+	VM_BUG_ON_PAGE(PageHuge(page), page);
+	VM_BUG_ON_PAGE(PageLRU(page), page);
+
+	if (next_nid == NUMA_NO_NODE)
+		return false;
+	if (PageTransHuge(page) && !thp_migration_supported())
+		return false;
+
+	// FIXME: actually enable this later in the series
+	return false;
+}
+
+
 /* Check if a page is dirty or under writeback */
 static void page_check_dirty_writeback(struct page *page,
 				       bool *dirty, bool *writeback)

@@ -1070,6 +1089,66 @@ static void page_check_dirty_writeback(s
 		mapping->a_ops->is_dirty_writeback(page, dirty, writeback);
 }
 
+static struct page *alloc_demote_page(struct page *page, unsigned long node)
+{
+	/*
+	 * Try to fail quickly if memory on the target node is not
+	 * available.  Leaving out __GFP_IO and __GFP_FS helps with
+	 * this.  If the desintation node is full, we want kswapd to
+	 * run there so that its pages will get reclaimed and future
+	 * migration attempts may succeed.
+	 */
+	gfp_t flags = (__GFP_HIGHMEM | __GFP_MOVABLE | __GFP_NORETRY |
+		       __GFP_NOMEMALLOC | __GFP_NOWARN | __GFP_THISNODE |
+		       __GFP_KSWAPD_RECLAIM);
+	/* HugeTLB pages should not be on the LRU */
+	WARN_ON_ONCE(PageHuge(page));
+
+	if (PageTransHuge(page)) {
+		struct page *thp;
+
+		flags |= __GFP_COMP;
+
+		thp = alloc_pages_node(node, flags, HPAGE_PMD_ORDER);
+		if (!thp)
+			return NULL;
+		prep_transhuge_page(thp);
+		return thp;
+	}
+
+	return __alloc_pages_node(node, flags, 0);
+}
+
+/*
+ * Take pages on @demote_list and attempt to demote them to
+ * another node.  Pages which are not demoted are added to
+ * @ret_pages.
+ */
+static unsigned int demote_page_list(struct list_head *ret_list,
+				     struct list_head *demote_pages,
+				     struct pglist_data *pgdat,
+				     struct scan_control *sc)
+{
+	int target_nid = next_demotion_node(pgdat->node_id);
+	unsigned int nr_succeeded = 0;
+	int err;
+
+	if (list_empty(demote_pages))
+		return 0;
+
+	/* Demotion ignores all cpuset and mempolicy settings */
+	err = migrate_pages(demote_pages, alloc_demote_page, NULL,
+			    target_nid, MIGRATE_ASYNC, MR_DEMOTION,
+			    &nr_succeeded);
+
+	if (err) {
+		putback_movable_pages(demote_pages);
+		list_splice(ret_list, demote_pages);
+	}
+
+	return nr_succeeded;
+}
+
 /*
  * shrink_page_list() returns the number of reclaimed pages
  */

@@ -1082,6 +1161,7 @@ static unsigned int shrink_page_list(str
 {
 	LIST_HEAD(ret_pages);
 	LIST_HEAD(free_pages);
+	LIST_HEAD(demote_pages);
 	unsigned int nr_reclaimed = 0;
 	unsigned int pgactivate = 0;
 

@@ -1237,6 +1317,16 @@ static unsigned int shrink_page_list(str
 		}
 
 		/*
+		 * Before reclaiming the page, try to relocate
+		 * its contents to another node.
+		 */
+		if (migrate_demote_page_ok(page, sc)) {
+			list_add(&page->lru, &demote_pages);
+			unlock_page(page);
+			continue;
+		}
+
+		/*
 		 * Anonymous process memory has backing store?
 		 * Try to allocate it some swap space here.
 		 * Lazyfree page could be freed directly

@@ -1484,6 +1574,9 @@ keep:
 		VM_BUG_ON_PAGE(PageLRU(page) || PageUnevictable(page), page);
 	}
 
+	/* Migrate pages selected for demotion */
+	nr_reclaimed += demote_page_list(&ret_pages, &demote_pages, pgdat, sc);
+
 	pgactivate = stat->nr_activate[0] + stat->nr_activate[1];
 
 	mem_cgroup_uncharge_list(&free_pages);

_

Quoted 5 lines

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +3 -0

+	/* Migrate pages selected for demotion */
+	nr_reclaimed += demote_page_list(&ret_pages, &demote_pages, pgdat, sc);
+
 	pgactivate = stat->nr_activate[0] + stat->nr_activate[1];

Quoted 2 lines

mem_cgroup_uncharge_list(&free_pages);
_

Generally, it's good to batch the page migration. But one side effect is that, if the pages are failed to be migrated, they will be placed back to the LRU list instead of falling back to be reclaimed really. This may cause some issue in some situation. For example, if there's no enough space in the PMEM (slow) node, so the page migration fails, OOM may be triggered, because the direct reclaiming on the DRAM (fast) node may make no progress, while it can reclaim some pages really before.

Quoted 7 lines · Ying

Quoted 7 lines

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +3 -0

+    /* Migrate pages selected for demotion */
+    nr_reclaimed += demote_page_list(&ret_pages, &demote_pages, pgdat, sc);
+
     pgactivate = stat->nr_activate[0] + stat->nr_activate[1];
 
     mem_cgroup_uncharge_list(&free_pages);

_

Quoted 7 lines

Generally, it's good to batch the page migration. But one side effect is that, if the pages are failed to be migrated, they will be placed back to the LRU list instead of falling back to be reclaimed really. This may cause some issue in some situation. For example, if there's no enough space in the PMEM (slow) node, so the page migration fails, OOM may be triggered, because the direct reclaiming on the DRAM (fast) node may make no progress, while it can reclaim some pages really before.

Yes, agreed.

There are a couple of ways we could fix this. Instead of splicing 'demote_pages' back into 'ret_pages', we could try to get them back on 'page_list' and goto the beginning on shrink_page_list(). This will probably yield the best behavior, but might be a bit ugly.

We could also add a field to 'struct scan_control' and just stop trying to migrate after it has failed one or more times. The trick will be picking a threshold that doesn't mess with either the normal reclaim rate or the migration rate.

This is on my list to fix up next.

Quoted 17 lines · Dave Hansen

Quoted 7 lines · Ying

Quoted 7 lines

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +3 -0

+    /* Migrate pages selected for demotion */
+    nr_reclaimed += demote_page_list(&ret_pages, &demote_pages, pgdat, sc);
+
     pgactivate = stat->nr_activate[0] + stat->nr_activate[1];
 
     mem_cgroup_uncharge_list(&free_pages);

_

Quoted 7 lines

Generally, it's good to batch the page migration. But one side effect is that, if the pages are failed to be migrated, they will be placed back to the LRU list instead of falling back to be reclaimed really. This may cause some issue in some situation. For example, if there's no enough space in the PMEM (slow) node, so the page migration fails, OOM may be triggered, because the direct reclaiming on the DRAM (fast) node may make no progress, while it can reclaim some pages really before.

Quoted 1 line

Yes, agreed.

Kind of. But I think that should be transient and very rare. The kswapd on pmem nodes will be waken up to drop pages when we try to allocate migration target pages. It should be very rare that there is not reclaimable page on pmem nodes.

Quoted 10 lines

There are a couple of ways we could fix this. Instead of splicing 'demote_pages' back into 'ret_pages', we could try to get them back on 'page_list' and goto the beginning on shrink_page_list(). This will probably yield the best behavior, but might be a bit ugly.

We could also add a field to 'struct scan_control' and just stop trying to migrate after it has failed one or more times. The trick will be picking a threshold that doesn't mess with either the normal reclaim rate or the migration rate.

In my patchset I implemented a fallback mechanism via adding a new PGDAT_CONTENDED node flag. Please check this out:
https://patchwork.kernel.org/patch/10993839/.

Basically the PGDAT_CONTENDED flag will be set once migrate_pages() return -ENOMEM which indicates the target pmem node is under memory pressure, then it would fallback to regular reclaim path. The flag would be cleared by clear_pgdat_congested() once the pmem node memory pressure is gone.

We already use node flags to indicate the state of node in reclaim code, i.e. PGDAT_WRITEBACK, PGDAT_DIRTY, etc. So, adding a new flag sounds more straightforward to me IMHO.

Quoted 3 lines

This is on my list to fix up next.

Quoted 40 lines · Yang Shi

Quoted 17 lines · Dave Hansen

Quoted 7 lines · Ying

Quoted 7 lines

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +3 -0

+    /* Migrate pages selected for demotion */
+    nr_reclaimed += demote_page_list(&ret_pages, &demote_pages, pgdat, sc);
+
     pgactivate = stat->nr_activate[0] + stat->nr_activate[1];
 
     mem_cgroup_uncharge_list(&free_pages);

_

Quoted 7 lines

Generally, it's good to batch the page migration. But one side effect is that, if the pages are failed to be migrated, they will be placed back to the LRU list instead of falling back to be reclaimed really. This may cause some issue in some situation. For example, if there's no enough space in the PMEM (slow) node, so the page migration fails, OOM may be triggered, because the direct reclaiming on the DRAM (fast) node may make no progress, while it can reclaim some pages really before.

Quoted 1 line

Yes, agreed.

Kind of. But I think that should be transient and very rare. The kswapd on pmem nodes will be waken up to drop pages when we try to allocate migration target pages. It should be very rare that there is not reclaimable page on pmem nodes.

Quoted 10 lines

There are a couple of ways we could fix this. Instead of splicing 'demote_pages' back into 'ret_pages', we could try to get them back on 'page_list' and goto the beginning on shrink_page_list(). This will probably yield the best behavior, but might be a bit ugly.

We could also add a field to 'struct scan_control' and just stop trying to migrate after it has failed one or more times. The trick will be picking a threshold that doesn't mess with either the normal reclaim rate or the migration rate.

In my patchset I implemented a fallback mechanism via adding a new PGDAT_CONTENDED node flag. Please check this out:
https://patchwork.kernel.org/patch/10993839/.

Quoted 5 lines

Basically the PGDAT_CONTENDED flag will be set once migrate_pages() return -ENOMEM which indicates the target pmem node is under memory pressure, then it would fallback to regular reclaim path. The flag would be cleared by clear_pgdat_congested() once the pmem node memory pressure is gone.

There may be some races between the flag set and clear. For example,

- try to migrate some pages from DRAM node to PMEM node

- no enough free pages on the PMEM node, so wakeup kswapd

- kswapd on PMEM node reclaimed some page and try to clear PGDAT_CONTENDED on DRAM node

- set PGDAT_CONTENDED on DRAM node

This may be resolvable. But I still prefer to fallback to real page reclaiming directly for the pages failed to be migrated. That looks more robust.

Best Regards,
Huang, Ying

Quoted 7 lines

We already use node flags to indicate the state of node in reclaim code, i.e. PGDAT_WRITEBACK, PGDAT_DIRTY, etc. So, adding a new flag sounds more straightforward to me IMHO.

Quoted 3 lines

This is on my list to fix up next.

From: Dave Hansen <[email protected]>

This is mostly derived from a patch from Yang Shi:

        https://lore.kernel.org/linux-mm/[email protected]/

Add code to the reclaim path (shrink_page_list()) to "demote" data to another NUMA node instead of discarding the data. This always avoids the cost of I/O needed to read the page back in and sometimes avoids the writeout cost when the pagee is dirty.

Note: This just adds the start of infratructure for migration. It is actually disabled next to the FIXME in migrate_demote_page_ok().

Signed-off-by Dave Hansen <[email protected]>

Cc Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

---

 b/include/linux/migrate.h |    2
 b/mm/vmscan.c             |   93 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 95 insertions(+)

diff -puN include/linux/migrate.h~demote-with-migrate_pages include/linux/migrate.h

include/linux/migrate.h 2020-08-18 11:36:52.923583179 -0700 renamed from include/linux/migrate.h~demote-with-migrate_pages 2020-08-18 11:36:52.916583179 -0700 +2 -0

@@ -25,6 +25,7 @@ enum migrate_reason {
        MR_MEMPOLICY_MBIND,
        MR_NUMA_MISPLACED,
        MR_CONTIG_RANGE,
+       MR_DEMOTION,
        MR_TYPES
 };
 

@@ -210,6 +211,7 @@ struct migrate_vma {
 int migrate_vma_setup(struct migrate_vma *args);
 void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
+int next_demotion_node(int node);
 
 #endif /* CONFIG_MIGRATION */
 

diff -puN mm/vmscan.c~demote-with-migrate_pages mm/vmscan.c

mm/vmscan.c 2020-08-18 11:36:52.924583179 -0700 renamed from mm/vmscan.c~demote-with-migrate_pages 2020-08-18 11:36:52.919583179 -0700 +1 -0

@@ -43,6 +43,7 @@
 #include <linux/kthread.h>
 #include <linux/freezer.h>
 #include <linux/memcontrol.h>
+#include <linux/migrate.h>
 #include <linux/delayacct.h>
 #include <linux/sysctl.h>
 #include <linux/oom.h>

@@ -1040,6 +1041,24 @@ static enum page_references page_check_r
        return PAGEREF_RECLAIM;

mm/vmscan.c 2020-08-18 11:36:52.924583179 -0700 renamed from mm/vmscan.c~demote-with-migrate_pages 2020-08-18 11:36:52.919583179 -0700 +84 -0

+       if (next_nid == NUMA_NO_NODE)
+               return false;
+       if (PageTransHuge(page) && !thp_migration_supported())
+               return false;
+
+       // FIXME: actually enable this later in the series
+       return false;
+}
+
+
 /* Check if a page is dirty or under writeback */
 static void page_check_dirty_writeback(struct page *page,
                                       bool *dirty, bool *writeback)

@@ -1070,6 +1089,66 @@ static void page_check_dirty_writeback(s
                mapping->a_ops->is_dirty_writeback(page, dirty, writeback);
 }
 
+static struct page *alloc_demote_page(struct page *page, unsigned long node)
+{
+       /*
+        * Try to fail quickly if memory on the target node is not
+        * available.  Leaving out __GFP_IO and __GFP_FS helps with
+        * this.  If the desintation node is full, we want kswapd to
+        * run there so that its pages will get reclaimed and future
+        * migration attempts may succeed.
+        */
+       gfp_t flags = (__GFP_HIGHMEM | __GFP_MOVABLE | __GFP_NORETRY |
+                      __GFP_NOMEMALLOC | __GFP_NOWARN | __GFP_THISNODE |
+                      __GFP_KSWAPD_RECLAIM);
+       /* HugeTLB pages should not be on the LRU */
+       WARN_ON_ONCE(PageHuge(page));
+
+       if (PageTransHuge(page)) {
+               struct page *thp;
+
+               flags |= __GFP_COMP;
+
+               thp = alloc_pages_node(node, flags, HPAGE_PMD_ORDER);
+               if (!thp)
+                       return NULL;
+               prep_transhuge_page(thp);
+               return thp;
+       }
+
+       return __alloc_pages_node(node, flags, 0);
+}
+
+/*
+ * Take pages on @demote_list and attempt to demote them to
+ * another node.  Pages which are not demoted are added to
+ * @ret_pages.
+ */
+static unsigned int demote_page_list(struct list_head *ret_list,
+                                    struct list_head *demote_pages,
+                                    struct pglist_data *pgdat,
+                                    struct scan_control *sc)
+{
+       int target_nid = next_demotion_node(pgdat->node_id);
+       unsigned int nr_succeeded = 0;
+       int err;
+
+       if (list_empty(demote_pages))
+               return 0;
+
+       /* Demotion ignores all cpuset and mempolicy settings */
+       err = migrate_pages(demote_pages, alloc_demote_page, NULL,
+                           target_nid, MIGRATE_ASYNC, MR_DEMOTION,
+                           &nr_succeeded);
+
+       if (err) {
+               putback_movable_pages(demote_pages);
+               list_splice(ret_list, demote_pages);
+       }
+
+       return nr_succeeded;
+}
+
 /*
  * shrink_page_list() returns the number of reclaimed pages
  */

@@ -1082,6 +1161,7 @@ static unsigned int shrink_page_list(str
 {
        LIST_HEAD(ret_pages);
        LIST_HEAD(free_pages);
+       LIST_HEAD(demote_pages);
        unsigned int nr_reclaimed = 0;
        unsigned int pgactivate = 0;
 

@@ -1237,6 +1317,16 @@ static unsigned int shrink_page_list(str
                }
 
                /*
+                * Before reclaiming the page, try to relocate
+                * its contents to another node.
+                */
+               if (migrate_demote_page_ok(page, sc)) {
+                       list_add(&page->lru, &demote_pages);
+                       unlock_page(page);
+                       continue;
+               }
+
+               /*
                 * Anonymous process memory has backing store?
                 * Try to allocate it some swap space here.
                 * Lazyfree page could be freed directly

@@ -1484,6 +1574,9 @@ keep:
                VM_BUG_ON_PAGE(PageLRU(page) || PageUnevictable(page), page);
        }
 
+       /* Migrate pages selected for demotion */
+       nr_reclaimed += demote_page_list(&ret_pages, &demote_pages, pgdat, sc);
+
        pgactivate = stat->nr_activate[0] + stat->nr_activate[1];
 
        mem_cgroup_uncharge_list(&free_pages);

_

From: Yang Shi <[email protected]>

Account the number of demoted pages into reclaim_state->nr_demoted.

Add pgdemote_kswapd and pgdemote_direct VM counters showed in /proc/vmstat.

[ daveh:
- __count_vm_events() a bit, and made them look at the THP size directly rather than getting data from migrate_pages()
]

Signed-off-by Yang Shi <[email protected]>

Signed-off-by Dave Hansen <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

---

 b/include/linux/vm_event_item.h |    2 ++
 b/mm/vmscan.c                   |    6 ++++++
 b/mm/vmstat.c                   |    2 ++
 3 files changed, 10 insertions(+)

diff -puN include/linux/vm_event_item.h~mm-vmscan-add-page-demotion-counter include/linux/vm_event_item.h

include/linux/vm_event_item.h 2020-08-18 11:36:54.070583176 -0700 renamed from include/linux/vm_event_item.h~mm-vmscan-add-page-demotion-counter 2020-08-18 11:36:54.062583176 -0700 +2 -0

@@ -32,6 +32,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PS
                PGREFILL,
                PGSTEAL_KSWAPD,
                PGSTEAL_DIRECT,
+               PGDEMOTE_KSWAPD,
+               PGDEMOTE_DIRECT,
                PGSCAN_KSWAPD,
                PGSCAN_DIRECT,
                PGSCAN_DIRECT_THROTTLE,

diff -puN mm/vmscan.c~mm-vmscan-add-page-demotion-counter mm/vmscan.c

mm/vmscan.c 2020-08-18 11:36:54.072583176 -0700 renamed from mm/vmscan.c~mm-vmscan-add-page-demotion-counter 2020-08-18 11:36:54.064583176 -0700 +6 -0

@@ -147,6 +147,7 @@ struct scan_control {
                unsigned int immediate;
                unsigned int file_taken;
                unsigned int taken;
+               unsigned int demoted;
        } nr;
 
        /* for recording the reclaimed slab by now */

@@ -1146,6 +1147,11 @@ static unsigned int demote_page_list(str
                list_splice(ret_list, demote_pages);
        }
 
+       if (current_is_kswapd())
+               __count_vm_events(PGDEMOTE_KSWAPD, nr_succeeded);
+       else
+               __count_vm_events(PGDEMOTE_DIRECT, nr_succeeded);
+
        return nr_succeeded;
 }
 

diff -puN mm/vmstat.c~mm-vmscan-add-page-demotion-counter mm/vmstat.c

mm/vmstat.c 2020-08-18 11:36:54.072583176 -0700 renamed from mm/vmstat.c~mm-vmscan-add-page-demotion-counter 2020-08-18 11:36:54.067583176 -0700 +2 -0

@@ -1200,6 +1200,8 @@ const char * const vmstat_text[] = {
        "pgrefill",
        "pgsteal_kswapd",
        "pgsteal_direct",
+       "pgdemote_kswapd",
+       "pgdemote_direct",
        "pgscan_kswapd",
        "pgscan_direct",
        "pgscan_direct_throttle",

_

Quoted 6 lines · Dave Hansen

From: Yang Shi <[email protected]>

Account the number of demoted pages into reclaim_state->nr_demoted.

Quoted 7 lines

Add pgdemote_kswapd and pgdemote_direct VM counters showed in /proc/vmstat.

[ daveh:
- __count_vm_events() a bit, and made them look at the THP size directly rather than getting data from migrate_pages()
]

This reminds me that we'd better to make migrate_pages() return the number of base pages. Now both nr_failed and nr_succeeded is inc'ed regardless of whether it is THP.

A quick look finds the callers of migrate_pages() seem not care about the exact returned number, they just care if it is positive or not.

And reclaim code depends on correct nr_succeeded to avoid over reclaim.

I will come up with a patch to post to the mailing list.

Quoted 62 lines

Signed-off-by Yang Shi <[email protected]>

Signed-off-by Dave Hansen <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

---

 b/include/linux/vm_event_item.h |    2 ++
 b/mm/vmscan.c                   |    6 ++++++
 b/mm/vmstat.c                   |    2 ++
 3 files changed, 10 insertions(+)

diff -puN include/linux/vm_event_item.h~mm-vmscan-add-page-demotion-counter include/linux/vm_event_item.h

include/linux/vm_event_item.h 2020-08-18 11:36:54.070583176 -0700 renamed from include/linux/vm_event_item.h~mm-vmscan-add-page-demotion-counter 2020-08-18 11:36:54.062583176 -0700 +2 -0

@@ -32,6 +32,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PS
                PGREFILL,
                PGSTEAL_KSWAPD,
                PGSTEAL_DIRECT,
+               PGDEMOTE_KSWAPD,
+               PGDEMOTE_DIRECT,
                PGSCAN_KSWAPD,
                PGSCAN_DIRECT,
                PGSCAN_DIRECT_THROTTLE,

diff -puN mm/vmscan.c~mm-vmscan-add-page-demotion-counter mm/vmscan.c

mm/vmscan.c 2020-08-18 11:36:54.072583176 -0700 renamed from mm/vmscan.c~mm-vmscan-add-page-demotion-counter 2020-08-18 11:36:54.064583176 -0700 +6 -0

@@ -147,6 +147,7 @@ struct scan_control {
                unsigned int immediate;
                unsigned int file_taken;
                unsigned int taken;
+               unsigned int demoted;
        } nr;
 
        /* for recording the reclaimed slab by now */

@@ -1146,6 +1147,11 @@ static unsigned int demote_page_list(str
                list_splice(ret_list, demote_pages);
        }
 
+       if (current_is_kswapd())
+               __count_vm_events(PGDEMOTE_KSWAPD, nr_succeeded);
+       else
+               __count_vm_events(PGDEMOTE_DIRECT, nr_succeeded);
+
        return nr_succeeded;
 }
 

diff -puN mm/vmstat.c~mm-vmscan-add-page-demotion-counter mm/vmstat.c

mm/vmstat.c 2020-08-18 11:36:54.072583176 -0700 renamed from mm/vmstat.c~mm-vmscan-add-page-demotion-counter 2020-08-18 11:36:54.067583176 -0700 +2 -0

@@ -1200,6 +1200,8 @@ const char * const vmstat_text[] = {
        "pgrefill",
        "pgsteal_kswapd",
        "pgsteal_direct",
+       "pgdemote_kswapd",
+       "pgdemote_direct",
        "pgscan_kswapd",
        "pgscan_direct",
        "pgscan_direct_throttle",

_

From: Yang Shi <[email protected]>

Account the number of demoted pages into reclaim_state->nr_demoted.

Add pgdemote_kswapd and pgdemote_direct VM counters showed in /proc/vmstat.

[ daveh:
- __count_vm_events() a bit, and made them look at the THP size directly rather than getting data from migrate_pages()
]

Signed-off-by Yang Shi <[email protected]>

Signed-off-by Dave Hansen <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

---

 b/include/linux/vm_event_item.h |    2 ++
 b/mm/vmscan.c                   |    6 ++++++
 b/mm/vmstat.c                   |    2 ++
 3 files changed, 10 insertions(+)

diff -puN include/linux/vm_event_item.h~mm-vmscan-add-page-demotion-counter include/linux/vm_event_item.h

include/linux/vm_event_item.h 2020-08-18 11:36:54.070583176 -0700 renamed from include/linux/vm_event_item.h~mm-vmscan-add-page-demotion-counter 2020-08-18 11:36:54.062583176 -0700 +2 -0

@@ -32,6 +32,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PS
                PGREFILL,
                PGSTEAL_KSWAPD,
                PGSTEAL_DIRECT,
+               PGDEMOTE_KSWAPD,
+               PGDEMOTE_DIRECT,
                PGSCAN_KSWAPD,
                PGSCAN_DIRECT,
                PGSCAN_DIRECT_THROTTLE,

diff -puN mm/vmscan.c~mm-vmscan-add-page-demotion-counter mm/vmscan.c

mm/vmscan.c 2020-08-18 11:36:54.072583176 -0700 renamed from mm/vmscan.c~mm-vmscan-add-page-demotion-counter 2020-08-18 11:36:54.064583176 -0700 +6 -0

@@ -147,6 +147,7 @@ struct scan_control {
                unsigned int immediate;
                unsigned int file_taken;
                unsigned int taken;
+               unsigned int demoted;
        } nr;
 
        /* for recording the reclaimed slab by now */

@@ -1146,6 +1147,11 @@ static unsigned int demote_page_list(str
                list_splice(ret_list, demote_pages);
        }
 
+       if (current_is_kswapd())
+               __count_vm_events(PGDEMOTE_KSWAPD, nr_succeeded);
+       else
+               __count_vm_events(PGDEMOTE_DIRECT, nr_succeeded);
+
        return nr_succeeded;
 }
 

diff -puN mm/vmstat.c~mm-vmscan-add-page-demotion-counter mm/vmstat.c

mm/vmstat.c 2020-08-18 11:36:54.072583176 -0700 renamed from mm/vmstat.c~mm-vmscan-add-page-demotion-counter 2020-08-18 11:36:54.067583176 -0700 +2 -0

@@ -1200,6 +1200,8 @@ const char * const vmstat_text[] = {
        "pgrefill",
        "pgsteal_kswapd",
        "pgsteal_direct",
+       "pgdemote_kswapd",
+       "pgdemote_direct",
        "pgscan_kswapd",
        "pgscan_direct",
        "pgscan_direct_throttle",

_

From: Dave Hansen <[email protected]>

Global reclaim aims to reduce the amount of memory used on a given node or set of nodes. Migrating pages to another node serves this purpose.

memcg reclaim is different. Its goal is to reduce the total memory consumption of the entire memcg, across all nodes. Migration does not assist memcg reclaim because it just moves page contents between nodes rather than actually reducing memory consumption.

Signed-off-by Dave Hansen <[email protected]>

Suggested-by Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

---

 b/mm/vmscan.c |   33 +++++++++++++++++++++++++--------
 1 file changed, 25 insertions(+), 8 deletions(-)

diff -puN mm/vmscan.c~never-demote-for-memcg-reclaim mm/vmscan.c

mm/vmscan.c 2020-08-18 11:36:56.447583170 -0700 renamed from mm/vmscan.c~never-demote-for-memcg-reclaim 2020-08-18 11:36:56.442583170 -0700 +4 -1

@@ -296,8 +296,11 @@ static bool writeback_throttling_sane(st
 #endif
 
 static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
-                                         int node_id)
+                                         int node_id,
+                                         struct scan_control *sc)
 {
+       bool in_cgroup_reclaim = false;
+
        /* Always age anon pages when we have swap */
        if (memcg == NULL) {
                if (get_nr_swap_pages() > 0)

mm/vmscan.c 2020-08-18 11:36:56.447583170 -0700 renamed from mm/vmscan.c~never-demote-for-memcg-reclaim 2020-08-18 11:36:56.442583170 -0700 +13 -5

+        * memcg so should not be performed when in memcg
+        * reclaim.
+        */
+       if (!in_cgroup_reclaim && (next_demotion_node(node_id) >= 0))
                return true;
 
        /* No way to reclaim anon pages */

@@ -326,7 +339,7 @@ unsigned long zone_reclaimable_pages(str
 
        nr = zone_page_state_snapshot(zone, NR_ZONE_INACTIVE_FILE) +
                zone_page_state_snapshot(zone, NR_ZONE_ACTIVE_FILE);
-       if (can_reclaim_anon_pages(NULL, zone_to_nid(zone)))
+       if (can_reclaim_anon_pages(NULL, zone_to_nid(zone), NULL))
                nr += zone_page_state_snapshot(zone, NR_ZONE_INACTIVE_ANON) +
                        zone_page_state_snapshot(zone, NR_ZONE_ACTIVE_ANON);
 

@@ -1070,6 +1083,10 @@ bool migrate_demote_page_ok(struct page
        VM_BUG_ON_PAGE(PageHuge(page), page);
        VM_BUG_ON_PAGE(PageLRU(page), page);
 
+       /* It is pointless to do demotion in memcg reclaim */
+       if (cgroup_reclaim(sc))
+               return false;
+
        if (next_nid == NUMA_NO_NODE)
                return false;
        if (PageTransHuge(page) && !thp_migration_supported())

@@ -2369,7 +2386,7 @@ static void get_scan_count(struct lruvec
        enum lru_list lru;
 
        /* If we have no swap space, do not bother scanning anon pages. */
-       if (!sc->may_swap || !can_reclaim_anon_pages(memcg, pgdat->node_id)) {
+       if (!sc->may_swap || !can_reclaim_anon_pages(memcg, pgdat->node_id, sc)) {
                scan_balance = SCAN_FILE;
                goto out;
        }

@@ -2653,7 +2670,7 @@ static void shrink_lruvec(struct lruvec
         * rebalance the anon lru active/inactive ratio.
         */
        if (can_reclaim_anon_pages(lruvec_memcg(lruvec),
-                              lruvec_pgdat(lruvec)->node_id) &&
+                              lruvec_pgdat(lruvec)->node_id, sc) &&
            inactive_is_low(lruvec, LRU_INACTIVE_ANON))
                shrink_active_list(SWAP_CLUSTER_MAX, lruvec,
                                   sc, LRU_ACTIVE_ANON);

@@ -2724,7 +2741,7 @@ static inline bool should_continue_recla
         */
        pages_for_compaction = compact_gap(sc->order);
        inactive_lru_pages = node_page_state(pgdat, NR_INACTIVE_FILE);
-       if (can_reclaim_anon_pages(NULL, pgdat->node_id))
+       if (can_reclaim_anon_pages(NULL, pgdat->node_id, sc))
                inactive_lru_pages += node_page_state(pgdat, NR_INACTIVE_ANON);
 
        return inactive_lru_pages > pages_for_compaction;

@@ -3485,7 +3502,7 @@ static void age_active_anon(struct pglis
        struct mem_cgroup *memcg;
        struct lruvec *lruvec;
 
-       if (!can_reclaim_anon_pages(NULL, pgdat->node_id))
+       if (!can_reclaim_anon_pages(NULL, pgdat->node_id, sc))
                return;
 
        lruvec = mem_cgroup_lruvec(NULL, pgdat);

_

todo:
Changes since (https://lwn.net/Articles/824830/):
* Use higher-level migrate_pages() API approach from Yang Shi's earlier patches.

* made sure to actually check node_reclaim_mode's new bit
* disabled migration entirely before introducing RECLAIM_MIGRATE
* Replace GFP_NOWAIT with explicit __GFP_KSWAPD_RECLAIM and comment why we want that.
* Comment on effects of that keep multiple source nodes from sharing target nodes

The full series is also available here:

        https://github.com/hansendc/linux/tree/automigrate-20200818

--

We're starting to see systems with more and more kinds of memory such as Intel's implementation of persistent memory.

Let's say you have a system with some DRAM and some persistent memory. Today, once DRAM fills up, reclaim will start and some of the DRAM contents will be thrown out. Allocations will, at some point, start falling over to the slower persistent memory.

That has two nasty properties. First, the newer allocations can end up in the slower persistent memory. Second, reclaimed data in DRAM are just discarded even if there are gobs of space in persistent memory that could be used.

This set implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.

While I've talked about a DRAM/PMEM pairing, this approach would function in any environment where memory tiers exist.

This is not perfect. It "strands" pages in slower memory and never brings them back to fast DRAM. Other things need to be built to promote hot pages back to DRAM.

This is also all based on an upstream mechanism that allows persistent memory to be onlined and used as if it were volatile:

        http://lkml.kernel.org/r/[email protected]

== Open Issues ==

* For cpusets and memory policies that restrict allocations to PMEM, is it OK to demote to PMEM? Do we need a cgroup- level API to opt-in or opt-out of these migrations?

* Migration failures will result in pages being unreclaimable. Need to be able to fall back to normal reclaim.

Cc Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

--

Dave Hansen (5):
mm/numa: node demotion data structure and lookup mm/vmscan: Attempt to migrate page in lieu of discard mm/numa: automatically generate node migration order mm/vmscan: never demote for memcg reclaim mm/numa: new reclaim mode to enable reclaim-based migration

Keith Busch (2):
mm/migrate: Defer allocating new page until needed mm/vmscan: Consider anonymous pages without swap

Yang Shi (1):
mm/vmscan: add page demotion counter

 Documentation/admin-guide/sysctl/vm.rst |    9
 include/linux/migrate.h                 |    6
 include/linux/node.h                    |    9
 include/linux/vm_event_item.h           |    2
 include/trace/events/migrate.h          |    3
 mm/debug.c                              |    1
 mm/internal.h                           |    1
 mm/migrate.c                            |  400 ++++++++++++++++++++++++++------
 mm/page_alloc.c                         |    2
 mm/vmscan.c                             |   88 ++++++-
 mm/vmstat.c                             |    2
 11 files changed, 439 insertions(+), 84 deletions(-)

Dave Hansen (5):
mm/numa: node demotion data structure and lookup mm/vmscan: Attempt to migrate page in lieu of discard mm/numa: automatically generate node migration order mm/vmscan: never demote for memcg reclaim mm/numa: new reclaim mode to enable reclaim-based migration

Keith Busch (2):
mm/migrate: Defer allocating new page until needed mm/vmscan: Consider anonymous pages without swap

Yang Shi (1):
mm/vmscan: add page demotion counter

 Documentation/admin-guide/sysctl/vm.rst |    9
 include/linux/migrate.h                 |    6
 include/linux/node.h                    |    9
 include/linux/vm_event_item.h           |    2
 include/trace/events/migrate.h          |    3
 mm/debug.c                              |    1
 mm/internal.h                           |    1
 mm/migrate.c                            |  400 ++++++++++++++++++++++++++------
 mm/page_alloc.c                         |    2
 mm/vmscan.c                             |   88 ++++++-
 mm/vmstat.c                             |    2
 11 files changed, 439 insertions(+), 84 deletions(-)