Migrate Pages in lieu of discard

From: Dave Hansen <[email protected]>

I went to go add a new RECLAIM_* mode for the zone_reclaim_mode sysctl. Like a good kernel developer, I also went to go update the documentation. I noticed that the bits in the documentation didn't match the bits in the #defines.

The VM never explicitly checks the RECLAIM_ZONE bit. The bit is, however implicitly checked when checking 'node_reclaim_mode==0'. The RECLAIM_ZONE #define was removed in a cleanup. That, by itself is fine.

But, when the bit was removed (bit 0) the _other_ bit locations also got changed. That's not OK because the bit values are documented to mean one specific thing and users surely rely on them meaning that one thing and not changing from kernel to kernel. The end result is that if someone had a script that did:

sysctl vm.zone_reclaim_mode=1

This script would have gone from enalbing node reclaim for clean unmapped pages to writing out pages during node reclaim after the commit in question. That's not great.

Put the bits back the way they were and add a comment so something like this is a bit harder to do again. Update the documentation to make it clear that the first bit is ignored.

Signed-off-by Dave Hansen <[email protected]>

Fixes 648b5cf368e0 ("mm/vmscan: remove unused RECLAIM_OFF/RECLAIM_ZONE")

Reviewed-by Ben Widawsky <[email protected]>

Acked-by David Rientjes <[email protected]>

Acked-by Christoph Lameter <[email protected]>

Cc Alex Shi <[email protected]>

Cc Daniel Wagner <[email protected]>

Cc "Tobin C. Harding" <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Andrew Morton <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc Qian Cai <[email protected]>

Cc Daniel Wagner <[email protected]>

Cc osalvador <[email protected]>

Cc [email protected]

--

Changes from v2:
* Update description to indicate that bit0 was used for clean unmapped page node reclaim.
---

 b/Documentation/admin-guide/sysctl/vm.rst |   10 +++++-----
 b/mm/vmscan.c                             |    9 +++++++--
 2 files changed, 12 insertions(+), 7 deletions(-)

diff -puN Documentation/admin-guide/sysctl/vm.rst~mm-vmscan-restore-old-zone_reclaim_mode-abi Documentation/admin-guide/sysctl/vm.rst

Documentation/admin-guide/sysctl/vm.rst renamed from Documentation/admin-guide/sysctl/vm.rst~mm-vmscan-restore-old-zone_reclaim_mode-abi +5 -5

@@ -978,11 +978,11 @@ that benefit from having their data cach
 left disabled as the caching effect is likely to be more important than
 data locality.
 
-zone_reclaim may be enabled if it's known that the workload is partitioned
-such that each partition fits within a NUMA node and that accessing remote
-memory would cause a measurable performance reduction.  The page allocator
-will then reclaim easily reusable pages (those page cache pages that are
-currently not used) before allocating off node pages.
+Consider enabling one or more zone_reclaim mode bits if it's known that the
+workload is partitioned such that each partition fits within a NUMA node
+and that accessing remote memory would cause a measurable performance
+reduction.  The page allocator will take additional actions before
+allocating off node pages.
 
 Allowing zone reclaim to write out pages stops processes that are
 writing large amounts of data from dirtying pages on other nodes. Zone

diff -puN mm/vmscan.c~mm-vmscan-restore-old-zone_reclaim_mode-abi mm/vmscan.c

mm/vmscan.c renamed from mm/vmscan.c~mm-vmscan-restore-old-zone_reclaim_mode-abi +7 -2

@@ -4086,8 +4086,13 @@ module_init(kswapd_init)
  */
 int node_reclaim_mode __read_mostly;
 
-#define RECLAIM_WRITE (1<<0)	/* Writeout pages during reclaim */
-#define RECLAIM_UNMAP (1<<1)	/* Unmap pages during reclaim */
+/*
+ * These bit locations are exposed in the vm.zone_reclaim_mode sysctl
+ * ABI.  New bits are OK, but existing bits can never change.
+ */
+#define RECLAIM_ZONE  (1<<0)   /* Run shrink_inactive_list on the zone */
+#define RECLAIM_WRITE (1<<1)   /* Writeout pages during reclaim */
+#define RECLAIM_UNMAP (1<<2)   /* Unmap pages during reclaim */
 
 /*
  * Priority for NODE_RECLAIM. This determines the fraction of pages

_

From: Dave Hansen <[email protected]>

It is currently not obvious that the RECLAIM_* bits are part of the uapi since they are defined in vmscan.c. Move them to a uapi header to make it obvious.

This should have no functional impact.

Signed-off-by Dave Hansen <[email protected]>

Reviewed-by Ben Widawsky <[email protected]>

Acked-by David Rientjes <[email protected]>

Acked-by Christoph Lameter <[email protected]>

Cc Alex Shi <[email protected]>

Cc Daniel Wagner <[email protected]>

Cc "Tobin C. Harding" <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Andrew Morton <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc Qian Cai <[email protected]>

Cc Daniel Wagner <[email protected]>

Cc osalvador <[email protected]>

From: Dave Hansen <[email protected]>

RECLAIM_ZONE was assumed to be unused because it was never explicitly used in the kernel. However, there were a number of places where it was checked implicitly by checking 'node_reclaim_mode' for a zero value.

These zero checks are not great because it is not obvious what a zero mode *means* in the code. Replace them with a helper which makes it more obvious: node_reclaim_enabled().

This helper also provides a handy place to explicitly check the RECLAIM_ZONE bit itself. Check it explicitly there to make it more obvious where the bit can affect behavior.

This should have no functional impact.

Signed-off-by Dave Hansen <[email protected]>

Reviewed-by Ben Widawsky <[email protected]>

Acked-by Christoph Lameter <[email protected]>

Cc Alex Shi <[email protected]>

Cc "Tobin C. Harding" <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Andrew Morton <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc Qian Cai <[email protected]>

Cc Daniel Wagner <[email protected]>

Cc osalvador <[email protected]>

From: Dave Hansen <[email protected]>

RECLAIM_ZONE was assumed to be unused because it was never explicitly used in the kernel. However, there were a number of places where it was checked implicitly by checking 'node_reclaim_mode' for a zero value.

These zero checks are not great because it is not obvious what a zero mode *means* in the code. Replace them with a helper which makes it more obvious: node_reclaim_enabled().

This helper also provides a handy place to explicitly check the RECLAIM_ZONE bit itself. Check it explicitly there to make it more obvious where the bit can affect behavior.

This should have no functional impact.

Signed-off-by Dave Hansen <[email protected]>

Reviewed-by Ben Widawsky <[email protected]>

Acked-by Christoph Lameter <[email protected]>

Cc Alex Shi <[email protected]>

Cc "Tobin C. Harding" <[email protected]>

Cc Christoph Lameter <[email protected]>

Cc Andrew Morton <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc Qian Cai <[email protected]>

Cc Daniel Wagner <[email protected]>

Cc osalvador <[email protected]>

mm/migrate.c renamed from mm/migrate.c~0006-node-Define-and-export-memory-migration-path +15 -0

+
+/**
+ * next_demotion_node() - Get the next node in the demotion path
+ * @node: The starting node to lookup the next node
+ *
+ * @returns: node id for next memory node in the demotion path hierarchy
+ * from @node; NUMA_NO_NODE if @node is terminal.  This does not keep
+ * @node online or guarantee that it *continues* to be the next demotion
+ * target.
+ */
+int next_demotion_node(int node)
+{
+	return node_demotion[node];
+}
+
 /*
  * Obtain the lock on page, remove all ptes and migrate the page
  * to the newly allocated page in newpage.

_

Quoted 7 lines · Dave Hansen

diff -puN mm/migrate.c~0006-node-Define-and-export-memory-migration-path mm/migrate.c

mm/migrate.c renamed from mm/migrate.c~0006-node-Define-and-export-memory-migration-path +0 -0

@@ -1161,6 +1161,22 @@ out:
 	return rc;
 }
 

mm/migrate.c out: · line 1164

Quoted 1 line

mm/migrate.c renamed from mm/migrate.c~0006-node-Define-and-export-memory-migration-path +1 -0

+static int node_demotion[MAX_NUMNODES] = {[0 ...  MAX_NUMNODES - 1] = NUMA_NO_NODE};

__read_mostly?

From: Dave Hansen <[email protected]>

When memory fills up on a node, memory contents can be automatically migrated to another node. The biggest problems are knowing when to migrate and to where the migration should be targeted.

The most straightforward way to generate the "to where" list would be to follow the page allocator fallback lists. Those lists already tell us if memory is full where to look next. It would also be logical to move memory in that order.

But, the allocator fallback lists have a fatal flaw: most nodes appear in all the lists. This would potentially lead to migration cycles (A->B, B->A, A->B, ...).

Instead of using the allocator fallback lists directly, keep a separate node migration ordering. But, reuse the same data used to generate page allocator fallback in the first place:
find_next_best_node().

This means that the firmware data used to populate node distances essentially dictates the ordering for now. It should also be architecture-neutral since all NUMA architectures have a working find_next_best_node().

The protocol for node_demotion[] access and writing is not standard. It has no specific locking and is intended to be read locklessly. Readers must take care to avoid observing changes that appear incoherent. This was done so that node_demotion[] locking has no chance of becoming a bottleneck on large systems with lots of CPUs in direct reclaim.

This code is unused for now. It will be called later in the series.

Signed-off-by Dave Hansen <[email protected]>

Cc Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc osalvador <[email protected]>

---

 b/mm/internal.h   |    5 +
 b/mm/migrate.c    |  137 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
 b/mm/page_alloc.c |    2
 3 files changed, 142 insertions(+), 2 deletions(-)

diff -puN mm/internal.h~auto-setup-default-migration-path-from-firmware mm/internal.h

mm/internal.h 2021-01-25 16:23:10.616866706 -0800 renamed from mm/internal.h~auto-setup-default-migration-path-from-firmware 2021-01-25 16:23:10.607866706 -0800 +5 -0

@@ -515,12 +515,17 @@ static inline void mminit_validate_memmo
 
 #ifdef CONFIG_NUMA
 extern int node_reclaim(struct pglist_data *, gfp_t, unsigned int);
+extern int find_next_best_node(int node, nodemask_t *used_node_mask);
 #else
 static inline int node_reclaim(struct pglist_data *pgdat, gfp_t mask,
                                unsigned int order)
 {
        return NODE_RECLAIM_NOSCAN;
 }
+static inline int find_next_best_node(int node, nodemask_t *used_node_mask)
+{
+       return NUMA_NO_NODE;
+}
 #endif
 
 extern int hwpoison_filter(struct page *p);

diff -puN mm/migrate.c~auto-setup-default-migration-path-from-firmware mm/migrate.c

mm/migrate.c 2021-01-25 16:23:10.617866706 -0800 renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware 2021-01-25 16:23:10.609866706 -0800 +4 -0

@@ -1161,6 +1161,10 @@ out:
        return rc;
 }
 
+/*
+ * Writes to this array occur without locking.  READ_ONCE()
+ * is recommended for readers to ensure consistent reads.
+ */
 static int node_demotion[MAX_NUMNODES] = {[0 ...  MAX_NUMNODES - 1] = NUMA_NO_NODE};
 
 /**

mm/migrate.c 2021-01-25 16:23:10.617866706 -0800 renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware 2021-01-25 16:23:10.609866706 -0800 +46 -0

 }
 
 /*

@@ -3124,3 +3134,128 @@ void migrate_vma_finalize(struct migrate
 }
 EXPORT_SYMBOL(migrate_vma_finalize);
 #endif /* CONFIG_DEVICE_PRIVATE */
+
+/* Disable reclaim-based migration. */
+static void disable_all_migrate_targets(void)
+{
+       int node;
+
+       for_each_online_node(node)
+               node_demotion[node] = NUMA_NO_NODE;
+}
+
+/*
+ * Find an automatic demotion target for 'node'.
+ * Failing here is OK.  It might just indicate
+ * being at the end of a chain.
+ */
+static int establish_migrate_target(int node, nodemask_t *used)
+{
+       int migration_target;
+
+       /*
+        * Can not set a migration target on a
+        * node with it already set.
+        *
+        * No need for READ_ONCE() here since this
+        * in the write path for node_demotion[].
+        * This should be the only thread writing.
+        */
+       if (node_demotion[node] != NUMA_NO_NODE)
+               return NUMA_NO_NODE;
+
+       migration_target = find_next_best_node(node, used);
+       if (migration_target == NUMA_NO_NODE)
+               return NUMA_NO_NODE;
+
+       node_demotion[node] = migration_target;
+
+       return migration_target;
+}
+
+/*
+ * When memory fills up on a node, memory contents can be
+ * automatically migrated to another node instead of
+ * discarded at reclaim.
+ *
+ * Establish a "migration path" which will start at nodes
+ * with CPUs and will follow the priorities used to build the

mm/migrate.c 2021-01-25 16:23:10.617866706 -0800 renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware 2021-01-25 16:23:10.609866706 -0800 +37 -0

+ */
+void __set_migration_target_nodes(void)
+{
+       nodemask_t next_pass    = NODE_MASK_NONE;
+       nodemask_t this_pass    = NODE_MASK_NONE;
+       nodemask_t used_targets = NODE_MASK_NONE;
+       int node;
+
+       /*
+        * Avoid any oddities like cycles that could occur
+        * from changes in the topology.  This will leave
+        * a momentary gap when migration is disabled.
+        */
+       disable_all_migrate_targets();
+
+       /*
+        * Ensure that the "disable" is visible across the system.
+        * Readers will see either a combination of before+disable
+        * state or disable+after.  They will never see before and
+        * after state together.
+        *
+        * The before+after state together might have cycles and
+        * could cause readers to do things like loop until this
+        * function finishes.  This ensures they can only see a
+        * single "bad" read and would, for instance, only loop
+        * once.
+        */
+       smp_wmb();
+
+       /*
+        * Allocations go close to CPUs, first.  Assume that
+        * the migration path starts at the nodes with CPUs.
+        */
+       next_pass = node_states[N_CPU];
+again:
+       this_pass = next_pass;
+       next_pass = NODE_MASK_NONE;

mm/migrate.c 2021-01-25 16:23:10.617866706 -0800 renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware 2021-01-25 16:23:10.609866706 -0800 +7 -0

+        */
+       nodes_or(used_targets, used_targets, this_pass);
+       for_each_node_mask(node, this_pass) {
+               int target_node = establish_migrate_target(node, &used_targets);
+
+               if (target_node == NUMA_NO_NODE)
+                       continue;

mm/migrate.c 2021-01-25 16:23:10.617866706 -0800 renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware 2021-01-25 16:23:10.609866706 -0800 +5 -0

+{
+       get_online_mems();
+       __set_migration_target_nodes();
+       put_online_mems();
+}

diff -puN mm/page_alloc.c~auto-setup-default-migration-path-from-firmware mm/page_alloc.c

mm/page_alloc.c 2021-01-25 16:23:10.619866706 -0800 renamed from mm/page_alloc.c~auto-setup-default-migration-path-from-firmware 2021-01-25 16:23:10.612866706 -0800 +1 -1

@@ -5704,7 +5704,7 @@ static int node_load[MAX_NUMNODES];
  *
  * Return: node id of the found node or %NUMA_NO_NODE if no node is found.
  */
-static int find_next_best_node(int node, nodemask_t *used_node_mask)
+int find_next_best_node(int node, nodemask_t *used_node_mask)
 {
        int n, val;
        int min_val = INT_MAX;

_

mm/migrate.c static int node_demotion[MAX_NUMNODES] = · line 1186

Quoted 9 lines

int next_demotion_node(int node)
{

mm/migrate.c renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware +6 -1

-       return node_demotion[node];
+       /*
+        * node_demotion[] is updated without excluding
+        * this function from running.  READ_ONCE() avoids
+        * reading multiple, inconsistent 'node' values
+        * during an update.
+        */

Don't we need a smp_rmb() here? The single write barrier might be not enough in migration target set. Typically a write barrier should be used in pairs with a read barrier.

Quoted 1 line

+ return READ_ONCE(node_demotion[node]);

Why not consolidate the patch #4 in this patch? The patch #4 just add the definition of node_demotion and the function, then the function is changed in this patch, and the function is not used by anyone between the adding and changing.

Quoted 9 lines

+/*
+ * When memory fills up on a node, memory contents can be
+ * automatically migrated to another node instead of
+ * discarded at reclaim.
+ *
+ * Establish a "migration path" which will start at nodes
+ * with CPUs and will follow the priorities used to build the
+ * page allocator zonelists.
+ *

Quoted 8 lines

+ * The difference here is that cycles must be avoided. If
+ * node0 migrates to node1, then neither node1, nor anything
+ * node1 migrates to can migrate to node0.
+ *
+ * This function can run simultaneously with readers of
+ * node_demotion[]. However, it can not run simultaneously
+ * with itself. Exclusion is provided by memory hotplug events
+ * being single-threaded.

Maybe an example diagram for the physical topology and how the migration target is generated in the comment seems helpful to understand the code.

mm/migrate.c void migrate_vma_finalize(struct migrate · line 3239

Quoted 13 lines

mm/migrate.c renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware +13 -0

+again:
+       this_pass = next_pass;
+       next_pass = NODE_MASK_NONE;
+       /*
+        * To avoid cycles in the migration "graph", ensure
+        * that migration sources are not future targets by
+        * setting them in 'used_targets'.  Do this only
+        * once per pass so that multiple source nodes can
+        * share a target node.
+        *
+        * 'used_targets' will become unavailable in future
+        * passes.  This limits some opportunities for
+        * multiple source nodes to share a desintation.

s/desination/destination

Quoted 11 lines

mm/migrate.c renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware +11 -0

+        */
+       nodes_or(used_targets, used_targets, this_pass);
+       for_each_node_mask(node, this_pass) {
+               int target_node = establish_migrate_target(node, &used_targets);
+
+               if (target_node == NUMA_NO_NODE)
+                       continue;
+
+               /* Visit targets from this pass in the next pass: */
+               node_set(target_node, next_pass);
+       }

mm/migrate.c void migrate_vma_finalize(struct migrate · line 3256

Quoted 6 lines

mm/migrate.c renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware +6 -0

+       /* Is another pass necessary? */
+       if (!nodes_empty(next_pass))
+               goto again;
+}
+
+void set_migration_target_nodes(void)

It seems this function is not called outside migrate.c, so it should be static.

Sure. Were you thinking of a code comment, or enhanced changelog?

Let's say there's a system with two sockets each with the same three classes of memory: fast, medium and slow. Each memory class is placed in its own NUMA node and the CPUs are attached to the fast memory. That leaves 6 NUMA nodes (0-5):

	Socket A: 0, 1, 2
	Socket B: 3, 4, 5

The migration path for this configuration path would start on the nodes with the processors and fast memory, progress through medium and end with the slow memory:

	0 -> 1 -> 2 -> stop
	3 -> 4 -> 5 -> stop

This is represented in the node_demotion[] like this:

{ 1, // Node 0 migrates to 1
2, // Node 1 migrates to 2
-1, // Node 2 does not migrate
4, // Node 3 migrates to 1
5, // Node 4 migrates to 2
-1} // Node 5 does not migrate

Is that what you were thinking of?

On 1/29/21 12:46 PM, Yang Shi wrote:
...

Quoted 10 lines

mm/migrate.c static int node_demotion[MAX_NUMNODES] = · line 1186

Quoted 9 lines

int next_demotion_node(int node)
{

mm/migrate.c renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware +6 -1

-       return node_demotion[node];
+       /*
+        * node_demotion[] is updated without excluding
+        * this function from running.  READ_ONCE() avoids
+        * reading multiple, inconsistent 'node' values
+        * during an update.
+        */

Quoted 3 lines

Don't we need a smp_rmb() here? The single write barrier might be not enough in migration target set. Typically a write barrier should be used in pairs with a read barrier.

I don't think we need one, practically.

Since there is no locking against node_demotion[] updates, although a smp_rmb() would ensure that this read is up-to-date, it could change freely after the smp_rmb().

In other words, smp_rmb() would shrink the window where a "stale" read could occur but would not eliminate it.

Quoted 6 lines

Quoted 1 line

+ return READ_ONCE(node_demotion[node]);

Why not consolidate the patch #4 in this patch? The patch #4 just add the definition of node_demotion and the function, then the function is changed in this patch, and the function is not used by anyone between the adding and changing.

I really wanted to highlight that the locking scheme and the READ_ONCE() (or lack thereof) was specifically due to how node_demotion[] was being updated.

The READ_ONCE() is not, for instance, inherent to the data structure.

...

Quoted 18 lines

Quoted 17 lines

+/*
+ * When memory fills up on a node, memory contents can be
+ * automatically migrated to another node instead of
+ * discarded at reclaim.
+ *
+ * Establish a "migration path" which will start at nodes
+ * with CPUs and will follow the priorities used to build the
+ * page allocator zonelists.
+ *
+ * The difference here is that cycles must be avoided. If
+ * node0 migrates to node1, then neither node1, nor anything
+ * node1 migrates to can migrate to node0.
+ *
+ * This function can run simultaneously with readers of
+ * node_demotion[]. However, it can not run simultaneously
+ * with itself. Exclusion is provided by memory hotplug events
+ * being single-threaded.

Quoted 3 lines

Maybe an example diagram for the physical topology and how the migration target is generated in the comment seems helpful to understand the code.

Sure. Were you thinking of a code comment, or enhanced changelog?

Let's say there's a system with two sockets each with the same three classes of memory: fast, medium and slow. Each memory class is placed in its own NUMA node and the CPUs are attached to the fast memory. That leaves 6 NUMA nodes (0-5):

        Socket A: 0, 1, 2
        Socket B: 3, 4, 5

The migration path for this configuration path would start on the nodes with the processors and fast memory, progress through medium and end with the slow memory:

        0 -> 1 -> 2 -> stop
        3 -> 4 -> 5 -> stop

This is represented in the node_demotion[] like this:

{ 1, // Node 0 migrates to 1
2, // Node 1 migrates to 2
-1, // Node 2 does not migrate
4, // Node 3 migrates to 1
5, // Node 4 migrates to 2
-1} // Node 5 does not migrate

Is that what you were thinking of?

...

Quoted 14 lines

mm/migrate.c void migrate_vma_finalize(struct migrate · line 3239

Quoted 13 lines

mm/migrate.c renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware +13 -0

+again:
+       this_pass = next_pass;
+       next_pass = NODE_MASK_NONE;
+       /*
+        * To avoid cycles in the migration "graph", ensure
+        * that migration sources are not future targets by
+        * setting them in 'used_targets'.  Do this only
+        * once per pass so that multiple source nodes can
+        * share a target node.
+        *
+        * 'used_targets' will become unavailable in future
+        * passes.  This limits some opportunities for
+        * multiple source nodes to share a desintation.

Quoted 1 line

s/desination/destination

Fixed, thanks.

Quoted 11 lines

Quoted 11 lines

mm/migrate.c renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware +11 -0

+        */
+       nodes_or(used_targets, used_targets, this_pass);
+       for_each_node_mask(node, this_pass) {
+               int target_node = establish_migrate_target(node, &used_targets);
+
+               if (target_node == NUMA_NO_NODE)
+                       continue;
+
+               /* Visit targets from this pass in the next pass: */
+               node_set(target_node, next_pass);
+       }

Quoted 8 lines

mm/migrate.c void migrate_vma_finalize(struct migrate · line 3256

Quoted 6 lines

mm/migrate.c renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware +6 -0

+       /* Is another pass necessary? */
+       if (!nodes_empty(next_pass))
+               goto again;
+}
+
+void set_migration_target_nodes(void)

It seems this function is not called outside migrate.c, so it should be static.

Fixed, thanks.

Quoted 14 lines · Dave Hansen

On 1/29/21 12:46 PM, Yang Shi wrote:
...

Quoted 12 lines

mm/migrate.c static int node_demotion[MAX_NUMNODES] = · line 1186

Quoted 9 lines

int next_demotion_node(int node)
{

mm/migrate.c renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware +6 -1

-       return node_demotion[node];
+       /*
+        * node_demotion[] is updated without excluding
+        * this function from running.  READ_ONCE() avoids
+        * reading multiple, inconsistent 'node' values
+        * during an update.
+        */

Don't we need a smp_rmb() here? The single write barrier might be not enough in migration target set. Typically a write barrier should be used in pairs with a read barrier.

Quoted 5 lines

I don't think we need one, practically.

Since there is no locking against node_demotion[] updates, although a smp_rmb() would ensure that this read is up-to-date, it could change freely after the smp_rmb().

Yes, but this should be able to guarantee we see "disable + after" state. Isn't it more preferred?

Quoted 15 lines · Yang Shi

Quoted 14 lines · Dave Hansen

On 1/29/21 12:46 PM, Yang Shi wrote:
...

Quoted 12 lines

mm/migrate.c static int node_demotion[MAX_NUMNODES] = · line 1186

Quoted 9 lines

int next_demotion_node(int node)
{

mm/migrate.c renamed from mm/migrate.c~auto-setup-default-migration-path-from-firmware +6 -1

-       return node_demotion[node];
+       /*
+        * node_demotion[] is updated without excluding
+        * this function from running.  READ_ONCE() avoids
+        * reading multiple, inconsistent 'node' values
+        * during an update.
+        */

Don't we need a smp_rmb() here? The single write barrier might be not enough in migration target set. Typically a write barrier should be used in pairs with a read barrier.

Quoted 7 lines

Quoted 5 lines

I don't think we need one, practically.

Since there is no locking against node_demotion[] updates, although a smp_rmb() would ensure that this read is up-to-date, it could change freely after the smp_rmb().

Yes, but this should be able to guarantee we see "disable + after" state. Isn't it more preferred?

I'm debating how much of this is theoretical versus actually applicable to what we have in the kernel. But, I'm generally worried about code like this that *looks* innocuous:

        int terminal_node = start_node;
        int next_node = next_demotion_node(start_node);
        while (next_node != NUMA_NO_NODE) {
                next_node = terminal_node;
                terminal_node = next_demotion_node(terminal_node);
        }

That could loop forever if it doesn't go out to memory during each loop.

However, if node_demotion[] *is* read on every trip through the loop, it will eventually terminate. READ_ONCE() can guarantee that, as could compiler barriers like smp_rmb().

But, after staring at it for a while, I think RCU may be the most clearly correct way to solve the problem. Or, maybe just throw in the towel and do a spinlock like a normal human being. :)

Anyway, here's what I was thinking I'd do with RCU:

1. node_demotion[] starts off in a "before" state
2. Writers to node_demotion[] first set the whole array such that it will not induce cycles, like setting every member to NUMA_NO_NODE. (the "disable" state)
3. Writer calls synchronize_rcu(). After it returns, no readers can observe the "before" values.
4. Writer sets the actual values it wants. (the "after" state)
5. Readers use rcu_read_lock() over any critical section where they read the array. They are guaranteed to only see one of the two adjacent states (before+disabled, or disabled+after), but never before+after within one RCU read-side critical section.
6. Readers use READ_ONCE() or some other compiler directive to ensure the compiler does not reorder or combine reads from multiple, adjacent RCU read-side critical sections.

Although, after writing this, plain old locks are sounding awfully tempting.

From: Dave Hansen <[email protected]>

Reclaim-based migration is attempting to optimize data placement in memory based on the system topology. If the system changes, so must the migration ordering.

The implementation here is pretty simple and entirely unoptimized. On any memory or CPU hotplug events, assume that a node was added or removed and recalculate all migration targets. This ensures that the node_demotion[] array is always ready to be used in case the new reclaim mode is enabled.

This recalculation is far from optimal, most glaringly that it does not even attempt to figure out if nodes are actually coming or going. But, given the expected paucity of hotplug events, this should be fine.

Signed-off-by Dave Hansen <[email protected]>

Cc Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc osalvador <[email protected]>

---

 b/mm/migrate.c |   97 +++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 95 insertions(+), 2 deletions(-)

diff -puN mm/migrate.c~enable-numa-demotion mm/migrate.c

mm/migrate.c 2021-01-25 16:23:11.855866703 -0800 renamed from mm/migrate.c~enable-numa-demotion 2021-01-25 16:23:11.850866703 -0800 +3 -1

@@ -49,6 +49,7 @@
 #include <linux/sched/mm.h>
 #include <linux/ptrace.h>
 #include <linux/oom.h>
+#include <linux/memory.h>
 
 #include <asm/tlbflush.h>
 

@@ -3135,6 +3136,7 @@ void migrate_vma_finalize(struct migrate
 EXPORT_SYMBOL(migrate_vma_finalize);
 #endif /* CONFIG_DEVICE_PRIVATE */
 
+#if defined(CONFIG_MEMORY_HOTPLUG)
 /* Disable reclaim-based migration. */
 static void disable_all_migrate_targets(void)
 {

@@ -3191,7 +3193,7 @@ static int establish_migrate_target(int
  * with itself.  Exclusion is provided by memory hotplug events
  * being single-threaded.
  */
-void __set_migration_target_nodes(void)
+static void __set_migration_target_nodes(void)
 {
        nodemask_t next_pass    = NODE_MASK_NONE;
        nodemask_t this_pass    = NODE_MASK_NONE;

mm/migrate.c 2021-01-25 16:23:11.855866703 -0800 renamed from mm/migrate.c~enable-numa-demotion 2021-01-25 16:23:11.850866703 -0800 +11 -0

+ * like events that online or offline NUMA nodes.
+ *
+ * The ordering is also currently dependent on which nodes have
+ * CPUs.  That means we need CPU on/offline notification too.
+ */
+static int migration_online_cpu(unsigned int cpu)
+{
+       set_migration_target_nodes();
+       return 0;
+}
+

mm/migrate.c 2021-01-25 16:23:11.855866703 -0800 renamed from mm/migrate.c~enable-numa-demotion 2021-01-25 16:23:11.850866703 -0800 +8 -0

+ * migration targets.
+ *
+ * These callbacks already hold get_online_mems().  That
+ * is why __set_migration_target_nodes() can be used as
+ * opposed to set_migration_target_nodes().
+ */
+static int __meminit migrate_on_reclaim_callback(struct notifier_block *self,
+                                                unsigned long action, void *arg)

mm/migrate.c 2021-01-25 16:23:11.855866703 -0800 renamed from mm/migrate.c~enable-numa-demotion 2021-01-25 16:23:11.850866703 -0800 +44 -0

+               break;
+       case MEM_OFFLINE:
+       case MEM_ONLINE:
+               /*
+                * Recalculate the target nodes once the node
+                * reaches its final state (online or offline).
+                */
+               __set_migration_target_nodes();
+               break;
+       case MEM_CANCEL_OFFLINE:
+               /*
+                * MEM_GOING_OFFLINE disabled all the migration
+                * targets.  Reenable them.
+                */
+               __set_migration_target_nodes();
+               break;
+       case MEM_GOING_ONLINE:
+       case MEM_CANCEL_ONLINE:
+               break;
+       }
+
+       return notifier_from_errno(0);
+}
+
+static int __init migrate_on_reclaim_init(void)
+{
+       int ret;
+
+       ret = cpuhp_setup_state(CPUHP_AP_ONLINE_DYN, "migrate on reclaim",
+                               migration_online_cpu,
+                               migration_offline_cpu);
+       /*
+        * In the unlikely case that this fails, the automatic
+        * migration targets may become suboptimal for nodes
+        * where N_CPU changes.  With such a small impact in a
+        * rare case, do not bother trying to do anything special.
+        */
+       WARN_ON(ret < 0);
+
+       hotplug_memory_notifier(migrate_on_reclaim_callback, 100);
+       return 0;
+}
+late_initcall(migrate_on_reclaim_init);
+#endif /* CONFIG_MEMORY_HOTPLUG */

_

From: Dave Hansen <[email protected]>

Reclaim-based migration is attempting to optimize data placement in memory based on the system topology. If the system changes, so must the migration ordering.

The implementation here is pretty simple and entirely unoptimized. On any memory or CPU hotplug events, assume that a node was added or removed and recalculate all migration targets. This ensures that the node_demotion[] array is always ready to be used in case the new reclaim mode is enabled.

This recalculation is far from optimal, most glaringly that it does not even attempt to figure out if nodes are actually coming or going. But, given the expected paucity of hotplug events, this should be fine.

Signed-off-by Dave Hansen <[email protected]>

Cc Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc osalvador <[email protected]>

---

mm/migrate.c renamed from mm/migrate.c~enable-numa-demotion +57 -0

+
+/*
+ * React to hotplug events that might affect the migration targes
+ * like events that online or offline NUMA nodes.
+ *
+ * The ordering is also currently dependent on which nodes have
+ * CPUs.  That means we need CPU on/offline notification too.
+ */
+static int migration_online_cpu(unsigned int cpu)
+{
+	set_migration_target_nodes();
+	return 0;
+}
+
+static int migration_offline_cpu(unsigned int cpu)
+{
+	set_migration_target_nodes();
+	return 0;
+}
+
+/*
+ * This leaves migrate-on-reclaim transiently disabled
+ * between the MEM_GOING_OFFLINE and MEM_OFFLINE events.
+ * This runs reclaim-based micgration is enabled or not.
+ * This ensures that the user can turn reclaim-based
+ * migration at any time without needing to recalcuate
+ * migration targets.
+ *
+ * These callbacks already hold get_online_mems().  That
+ * is why __set_migration_target_nodes() can be used as
+ * opposed to set_migration_target_nodes().
+ */
+static int __meminit migrate_on_reclaim_callback(struct notifier_block *self,
+						 unsigned long action, void *arg)
+{
+	switch (action) {
+	case MEM_GOING_OFFLINE:
+		/*
+		 * Make sure there are not transient states where
+		 * an offline node is a migration target.  This
+		 * will leave migration disabled until the offline
+		 * completes and the MEM_OFFLINE case below runs.
+		 */
+		disable_all_migrate_targets();
+		break;
+	case MEM_OFFLINE:
+	case MEM_ONLINE:
+		/*
+		 * Recalculate the target nodes once the node
+		 * reaches its final state (online or offline).
+		 */
+		__set_migration_target_nodes();
+		break;
+	case MEM_CANCEL_OFFLINE:
+		/*
+		 * MEM_GOING_OFFLINE disabled all the migration
+		 * targets.  Reenable them.

Quoted 33 lines

mm/migrate.c renamed from mm/migrate.c~enable-numa-demotion +33 -0

+static int __meminit migrate_on_reclaim_callback(struct notifier_block *self,
+						 unsigned long action, void *arg)
+{
+	switch (action) {
+	case MEM_GOING_OFFLINE:
+		/*
+		 * Make sure there are not transient states where
+		 * an offline node is a migration target.  This
+		 * will leave migration disabled until the offline
+		 * completes and the MEM_OFFLINE case below runs.
+		 */
+		disable_all_migrate_targets();
+		break;
+	case MEM_OFFLINE:
+	case MEM_ONLINE:
+		/*
+		 * Recalculate the target nodes once the node
+		 * reaches its final state (online or offline).
+		 */
+		__set_migration_target_nodes();
+		break;
+	case MEM_CANCEL_OFFLINE:
+		/*
+		 * MEM_GOING_OFFLINE disabled all the migration
+		 * targets.  Reenable them.
+		 */
+		__set_migration_target_nodes();
+		break;
+	case MEM_GOING_ONLINE:
+	case MEM_CANCEL_ONLINE:
+		break;
+	}
+

Review

Quoted 2 lines

(quoted diff) +2 -0

+	return notifier_from_errno(0);
+}

This looks good, and I kinda like it.
But in this case, all we care about is whether NUMA node does or does not have memory, so we have to remove/added into the demotion list. So, would make more sense to have a kinda helper in node_states_{set,clear}_node that calls the respective functions (disable_all_migrate_targets and __set_migration_target_nodes)?

Quoted 33 lines · Oscar Salvador

Quoted 33 lines

mm/migrate.c renamed from mm/migrate.c~enable-numa-demotion +33 -0

+static int __meminit migrate_on_reclaim_callback(struct notifier_block *self,
+						 unsigned long action, void *arg)
+{
+	switch (action) {
+	case MEM_GOING_OFFLINE:
+		/*
+		 * Make sure there are not transient states where
+		 * an offline node is a migration target.  This
+		 * will leave migration disabled until the offline
+		 * completes and the MEM_OFFLINE case below runs.
+		 */
+		disable_all_migrate_targets();
+		break;
+	case MEM_OFFLINE:
+	case MEM_ONLINE:
+		/*
+		 * Recalculate the target nodes once the node
+		 * reaches its final state (online or offline).
+		 */
+		__set_migration_target_nodes();
+		break;
+	case MEM_CANCEL_OFFLINE:
+		/*
+		 * MEM_GOING_OFFLINE disabled all the migration
+		 * targets.  Reenable them.
+		 */
+		__set_migration_target_nodes();
+		break;
+	case MEM_GOING_ONLINE:
+	case MEM_CANCEL_ONLINE:
+		break;
+	}
+

Quoted 8 lines

Review

Quoted 2 lines

(quoted diff) +2 -0

+	return notifier_from_errno(0);
+}

This looks good, and I kinda like it.
But in this case, all we care about is whether NUMA node does or does not have memory, so we have to remove/added into the demotion list. So, would make more sense to have a kinda helper in node_states_{set,clear}_node that calls the respective functions (disable_all_migrate_targets and __set_migration_target_nodes)?

Of, you're saying that we could do this in the hotplug code itself instead of from a notifier? I agree, we *could*. That would be more efficient. But, I do like the idea of doing this from a notifier because it's a bit less brittle.

Do you feel strongly about this one?

From: Yang Shi <[email protected]>

The migrate_pages() returns the number of pages that were not migrated, or an error code. When returning an error code, there is no way to know how many pages were migrated or not migrated.

In the following patch, migrate_pages() is used to demote pages to PMEM node, we need account how many pages are reclaimed (demoted) since page reclaim behavior depends on this. Add *nr_succeeded parameter to make migrate_pages() return how many pages are demoted successfully for all cases.

Signed-off-by Yang Shi <[email protected]>

Signed-off-by Dave Hansen <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc osalvador <[email protected]>

---

 b/include/linux/migrate.h |    5 +++--
 b/mm/compaction.c         |    3 ++-
 b/mm/gup.c                |    4 +++-
 b/mm/memory-failure.c     |    4 +++-
 b/mm/memory_hotplug.c     |    4 +++-
 b/mm/mempolicy.c          |    8 ++++++--
 b/mm/migrate.c            |   17 ++++++++++-------
 b/mm/page_alloc.c         |    9 ++++++---
 8 files changed, 36 insertions(+), 18 deletions(-)

diff -puN include/linux/migrate.h~migrate_pages-add-success-return include/linux/migrate.h

include/linux/migrate.h 2021-01-25 16:23:12.954866701 -0800 renamed from include/linux/migrate.h~migrate_pages-add-success-return 2021-01-25 16:23:12.931866701 -0800 +3 -2

@@ -40,7 +40,8 @@ extern int migrate_page(struct address_s
                        struct page *newpage, struct page *page,
                        enum migrate_mode mode);
 extern int migrate_pages(struct list_head *l, new_page_t new, free_page_t free,
-               unsigned long private, enum migrate_mode mode, int reason);
+               unsigned long private, enum migrate_mode mode, int reason,
+               unsigned int *nr_succeeded);
 extern struct page *alloc_migration_target(struct page *page, unsigned long private);
 extern int isolate_movable_page(struct page *page, isolate_mode_t mode);
 extern void putback_movable_page(struct page *page);

@@ -58,7 +59,7 @@ extern int migrate_page_move_mapping(str
 static inline void putback_movable_pages(struct list_head *l) {}
 static inline int migrate_pages(struct list_head *l, new_page_t new,
                free_page_t free, unsigned long private, enum migrate_mode mode,
-               int reason)
+               int reason, unsigned int *nr_succeeded)
        { return -ENOSYS; }
 static inline struct page *alloc_migration_target(struct page *page,
                unsigned long private)

diff -puN mm/compaction.c~migrate_pages-add-success-return mm/compaction.c

mm/compaction.c 2021-01-25 16:23:12.956866701 -0800 renamed from mm/compaction.c~migrate_pages-add-success-return 2021-01-25 16:23:12.933866701 -0800 +2 -1

@@ -2199,6 +2199,7 @@ compact_zone(struct compact_control *cc,
        unsigned long last_migrated_pfn;
        const bool sync = cc->mode != MIGRATE_ASYNC;
        bool update_cached;
+       unsigned int nr_succeeded = 0;
 
        /*
         * These counters track activities during zone compaction.  Initialize

@@ -2317,7 +2318,7 @@ compact_zone(struct compact_control *cc,
 
                err = migrate_pages(&cc->migratepages, compaction_alloc,
                                compaction_free, (unsigned long)cc, cc->mode,
-                               MR_COMPACTION);
+                               MR_COMPACTION, &nr_succeeded);
 
                trace_mm_compaction_migratepages(cc->nr_migratepages, err,
                                                        &cc->migratepages);

diff -puN mm/gup.c~migrate_pages-add-success-return mm/gup.c

mm/gup.c 2021-01-25 16:23:12.957866701 -0800 renamed from mm/gup.c~migrate_pages-add-success-return 2021-01-25 16:23:12.935866701 -0800 +3 -1

@@ -1599,6 +1599,7 @@ static long check_and_migrate_cma_pages(
        unsigned long step;
        bool drain_allow = true;
        bool migrate_allow = true;
+       unsigned int nr_succeeded = 0;
        LIST_HEAD(cma_page_list);
        long ret = nr_pages;
        struct migration_target_control mtc = {

@@ -1654,7 +1655,8 @@ check_again:
                                put_page(pages[i]);
 
                if (migrate_pages(&cma_page_list, alloc_migration_target, NULL,
-                       (unsigned long)&mtc, MIGRATE_SYNC, MR_CONTIG_RANGE)) {
+                       (unsigned long)&mtc, MIGRATE_SYNC, MR_CONTIG_RANGE,
+                                 &nr_succeeded)) {
                        /*
                         * some of the pages failed migration. Do get_user_pages
                         * without migration.

diff -puN mm/memory-failure.c~migrate_pages-add-success-return mm/memory-failure.c

mm/memory-failure.c 2021-01-25 16:23:12.959866701 -0800 renamed from mm/memory-failure.c~migrate_pages-add-success-return 2021-01-25 16:23:12.939866701 -0800 +3 -1

@@ -1783,6 +1783,7 @@ static int __soft_offline_page(struct pa
        unsigned long pfn = page_to_pfn(page);
        struct page *hpage = compound_head(page);
        char const *msg_page[] = {"page", "hugepage"};
+       unsigned int nr_succeeded = 0;
        bool huge = PageHuge(page);
        LIST_HEAD(pagelist);
        struct migration_target_control mtc = {

@@ -1826,7 +1827,8 @@ static int __soft_offline_page(struct pa
 
        if (isolate_page(hpage, &pagelist)) {
                ret = migrate_pages(&pagelist, alloc_migration_target, NULL,
-                       (unsigned long)&mtc, MIGRATE_SYNC, MR_MEMORY_FAILURE);
+                       (unsigned long)&mtc, MIGRATE_SYNC, MR_MEMORY_FAILURE,
+                       &nr_succeeded);
                if (!ret) {
                        bool release = !huge;
 

diff -puN mm/memory_hotplug.c~migrate_pages-add-success-return mm/memory_hotplug.c

mm/memory_hotplug.c 2021-01-25 16:23:12.959866701 -0800 renamed from mm/memory_hotplug.c~migrate_pages-add-success-return 2021-01-25 16:23:12.941866701 -0800 +3 -1

@@ -1278,6 +1278,7 @@ do_migrate_range(unsigned long start_pfn
        unsigned long pfn;
        struct page *page, *head;
        int ret = 0;
+       unsigned int nr_succeeded = 0;
        LIST_HEAD(source);
 
        for (pfn = start_pfn; pfn < end_pfn; pfn++) {

@@ -1352,7 +1353,8 @@ do_migrate_range(unsigned long start_pfn
                if (nodes_empty(nmask))
                        node_set(mtc.nid, nmask);
                ret = migrate_pages(&source, alloc_migration_target, NULL,
-                       (unsigned long)&mtc, MIGRATE_SYNC, MR_MEMORY_HOTPLUG);
+                       (unsigned long)&mtc, MIGRATE_SYNC, MR_MEMORY_HOTPLUG,
+                       &nr_succeeded);
                if (ret) {
                        list_for_each_entry(page, &source, lru) {
                                pr_warn("migrating pfn %lx failed ret:%d ",

diff -puN mm/mempolicy.c~migrate_pages-add-success-return mm/mempolicy.c

mm/mempolicy.c 2021-01-25 16:23:12.960866701 -0800 renamed from mm/mempolicy.c~migrate_pages-add-success-return 2021-01-25 16:23:12.944866701 -0800 +6 -2

@@ -1071,6 +1071,7 @@ static int migrate_page_add(struct page
 static int migrate_to_node(struct mm_struct *mm, int source, int dest,
                           int flags)
 {
+       unsigned int nr_succeeded = 0;
        nodemask_t nmask;
        LIST_HEAD(pagelist);
        int err = 0;

@@ -1093,7 +1094,8 @@ static int migrate_to_node(struct mm_str
 
        if (!list_empty(&pagelist)) {
                err = migrate_pages(&pagelist, alloc_migration_target, NULL,
-                               (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL);
+                               (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL,
+                               &nr_succeeded);
                if (err)
                        putback_movable_pages(&pagelist);
        }

@@ -1270,6 +1272,7 @@ static long do_mbind(unsigned long start
                     nodemask_t *nmask, unsigned long flags)
 {
        struct mm_struct *mm = current->mm;
+       unsigned int nr_succeeded = 0;
        struct mempolicy *new;
        unsigned long end;
        int err;

@@ -1349,7 +1352,8 @@ static long do_mbind(unsigned long start
                if (!list_empty(&pagelist)) {
                        WARN_ON_ONCE(flags & MPOL_MF_LAZY);
                        nr_failed = migrate_pages(&pagelist, new_page, NULL,
-                               start, MIGRATE_SYNC, MR_MEMPOLICY_MBIND);
+                               start, MIGRATE_SYNC, MR_MEMPOLICY_MBIND,
+                               &nr_succeeded);
                        if (nr_failed)
                                putback_movable_pages(&pagelist);
                }

diff -puN mm/migrate.c~migrate_pages-add-success-return mm/migrate.c

mm/migrate.c 2021-01-25 16:23:12.964866701 -0800 renamed from mm/migrate.c~migrate_pages-add-success-return 2021-01-25 16:23:12.947866701 -0800 +2 -2

@@ -1432,6 +1432,7 @@ out:
  * @mode:              The migration mode that specifies the constraints for
  *                     page migration, if any.
  * @reason:            The reason for page migration.
+ * @nr_succeeded:      The number of pages migrated successfully.
  *
  * The function returns after 10 attempts or if no pages are movable any more
  * because the list has become empty or no retryable pages exist any more.

@@ -1442,12 +1443,11 @@ out:
  */
 int migrate_pages(struct list_head *from, new_page_t get_new_page,
                free_page_t put_new_page, unsigned long private,
-               enum migrate_mode mode, int reason)
+               enum migrate_mode mode, int reason, unsigned int *nr_succeeded)
 {
        int retry = 1;
        int thp_retry = 1;
        int nr_failed = 0;
-       int nr_succeeded = 0;
        int nr_thp_succeeded = 0;
        int nr_thp_failed = 0;
        int nr_thp_split = 0;

mm/migrate.c 2021-01-25 16:23:12.964866701 -0800 renamed from mm/migrate.c~migrate_pages-add-success-return 2021-01-25 16:23:12.947866701 -0800 +8 -5

                                        break;
                                }
-                               nr_succeeded++;
+                               (*nr_succeeded)++;
                                break;
                        default:
                                /*

@@ -1550,12 +1550,12 @@ retry:
        nr_thp_failed += thp_retry;
        rc = nr_failed;
 out:
-       count_vm_events(PGMIGRATE_SUCCESS, nr_succeeded);
+       count_vm_events(PGMIGRATE_SUCCESS, *nr_succeeded);
        count_vm_events(PGMIGRATE_FAIL, nr_failed);
        count_vm_events(THP_MIGRATION_SUCCESS, nr_thp_succeeded);
        count_vm_events(THP_MIGRATION_FAIL, nr_thp_failed);
        count_vm_events(THP_MIGRATION_SPLIT, nr_thp_split);
-       trace_mm_migrate_pages(nr_succeeded, nr_failed, nr_thp_succeeded,
+       trace_mm_migrate_pages(*nr_succeeded, nr_failed, nr_thp_succeeded,
                               nr_thp_failed, nr_thp_split, mode, reason);
 
        if (!swapwrite)

@@ -1623,6 +1623,7 @@ static int store_status(int __user *stat
 static int do_move_pages_to_node(struct mm_struct *mm,
                struct list_head *pagelist, int node)
 {
+       unsigned int nr_succeeded = 0;
        int err;
        struct migration_target_control mtc = {
                .nid = node,

@@ -1630,7 +1631,8 @@ static int do_move_pages_to_node(struct
        };
 
        err = migrate_pages(pagelist, alloc_migration_target, NULL,
-                       (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL);
+                       (unsigned long)&mtc, MIGRATE_SYNC, MR_SYSCALL,
+                       &nr_succeeded);
        if (err)
                putback_movable_pages(pagelist);
        return err;

@@ -2103,6 +2105,7 @@ int migrate_misplaced_page(struct page *
        pg_data_t *pgdat = NODE_DATA(node);
        int isolated;
        int nr_remaining;
+       unsigned int nr_succeeded = 0;
        LIST_HEAD(migratepages);
 
        /*

@@ -2127,7 +2130,7 @@ int migrate_misplaced_page(struct page *
        list_add(&page->lru, &migratepages);
        nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_page,
                                     NULL, node, MIGRATE_ASYNC,
-                                    MR_NUMA_MISPLACED);
+                                    MR_NUMA_MISPLACED, &nr_succeeded);
        if (nr_remaining) {
                if (!list_empty(&migratepages)) {
                        list_del(&page->lru);

diff -puN mm/page_alloc.c~migrate_pages-add-success-return mm/page_alloc.c

mm/page_alloc.c 2021-01-25 16:23:12.968866701 -0800 renamed from mm/page_alloc.c~migrate_pages-add-success-return 2021-01-25 16:23:12.950866701 -0800 +6 -3

@@ -8401,7 +8401,8 @@ static unsigned long pfn_max_align_up(un
 
 /* [start, end) must belong to a single zone. */
 static int __alloc_contig_migrate_range(struct compact_control *cc,
-                                       unsigned long start, unsigned long end)
+                                       unsigned long start, unsigned long end,
+                                       unsigned int *nr_succeeded)
 {
        /* This function is based on compact_zone() from compaction.c. */
        unsigned int nr_reclaimed;

@@ -8439,7 +8440,8 @@ static int __alloc_contig_migrate_range(
                cc->nr_migratepages -= nr_reclaimed;
 
                ret = migrate_pages(&cc->migratepages, alloc_migration_target,
-                               NULL, (unsigned long)&mtc, cc->mode, MR_CONTIG_RANGE);
+                               NULL, (unsigned long)&mtc, cc->mode, MR_CONTIG_RANGE,
+                               nr_succeeded);
        }
        if (ret < 0) {
                putback_movable_pages(&cc->migratepages);

@@ -8475,6 +8477,7 @@ int alloc_contig_range(unsigned long sta
        unsigned long outer_start, outer_end;
        unsigned int order;
        int ret = 0;
+       unsigned int nr_succeeded = 0;
 
        struct compact_control cc = {
                .nr_migratepages = 0,

@@ -8527,7 +8530,7 @@ int alloc_contig_range(unsigned long sta
         * allocated.  So, if we fall through be sure to clear ret so that
         * -EBUSY is not accidentally used or returned to caller.
         */
-       ret = __alloc_contig_migrate_range(&cc, start, end);
+       ret = __alloc_contig_migrate_range(&cc, start, end, &nr_succeeded);
        if (ret && ret != -EBUSY)
                goto done;
        ret =0;

_

mm/migrate.c retry: · line 1527

Quoted 2 lines

mm/migrate.c renamed from mm/migrate.c~migrate_pages-add-success-return +0 -0

@@ -1527,7 +1527,7 @@ retry:
                                        nr_succeeded += nr_subpages;

It seems the above line is missed. The THP accounting change was merged in v5.9 before I submitted this patch.

From: Dave Hansen <[email protected]>

This is mostly derived from a patch from Yang Shi:

	https://lore.kernel.org/linux-mm/[email protected]/

Add code to the reclaim path (shrink_page_list()) to "demote" data to another NUMA node instead of discarding the data. This always avoids the cost of I/O needed to read the page back in and sometimes avoids the writeout cost when the pagee is dirty.

A second pass through shrink_page_list() will be made if any demotions fail. This essentally falls back to normal reclaim behavior in the case that demotions fail. Previous versions of this patch may have simply failed to reclaim pages which were eligible for demotion but were unable to be demoted in practice.

Note: This just adds the start of infratructure for migration. It is actually disabled next to the FIXME in migrate_demote_page_ok().

Signed-off-by Dave Hansen <[email protected]>

Cc Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc osalvador <[email protected]>

--

changes from 202010:
* add MR_NUMA_MISPLACED to trace MIGRATE_REASON define
* make migrate_demote_page_ok() static, remove 'sc' arg until later patch
* remove unnecessary alloc_demote_page() hugetlb warning
* Simplify alloc_demote_page() gfp mask. Depend on __GFP_NORETRY to make it lightweight instead of fancier stuff like leaving out __GFP_IO/FS.
* Allocate migration page with alloc_migration_target() instead of allocating directly.
changes from 20200730:
* Add another pass through shrink_page_list() when demotion fails.
---

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +28 -0

+
+/*
+ * Take pages on @demote_list and attempt to demote them to
+ * another node.  Pages which are not demoted are left on
+ * @demote_pages.
+ */
+static unsigned int demote_page_list(struct list_head *demote_pages,
+				     struct pglist_data *pgdat,
+				     struct scan_control *sc)
+{
+	int target_nid = next_demotion_node(pgdat->node_id);
+	unsigned int nr_succeeded = 0;
+	int err;
+
+	if (list_empty(demote_pages))
+		return 0;
+
+	/* Demotion ignores all cpuset and mempolicy settings */
+	err = migrate_pages(demote_pages, alloc_demote_page, NULL,
+			    target_nid, MIGRATE_ASYNC, MR_DEMOTION,
+			    &nr_succeeded);
+
+	return nr_succeeded;
+}
+
 /*
  * shrink_page_list() returns the number of reclaimed pages
  */

@@ -1078,12 +1135,15 @@ static unsigned int shrink_page_list(str
 {
 	LIST_HEAD(ret_pages);
 	LIST_HEAD(free_pages);
+	LIST_HEAD(demote_pages);
 	unsigned int nr_reclaimed = 0;
 	unsigned int pgactivate = 0;
+	bool do_demote_pass = true;
 
 	memset(stat, 0, sizeof(*stat));
 	cond_resched();
 
+retry:
 	while (!list_empty(page_list)) {
 		struct address_space *mapping;
 		struct page *page;

@@ -1233,6 +1293,16 @@ static unsigned int shrink_page_list(str
 		}

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +13 -0

+
+		/*
 		 * Anonymous process memory has backing store?
 		 * Try to allocate it some swap space here.
 		 * Lazyfree page could be freed directly

@@ -1479,6 +1549,17 @@ keep:
 		list_add(&page->lru, &ret_pages);
 		VM_BUG_ON_PAGE(PageLRU(page) || PageUnevictable(page), page);
 	}
+	/* 'page_list' is always empty here */
+
+	/* Migrate pages selected for demotion */
+	nr_reclaimed += demote_page_list(&demote_pages, pgdat, sc);
+	/* Pages that could not be demoted are still in @demote_pages */
+	if (!list_empty(&demote_pages)) {
+		/* Pages which failed to demoted go back on on @page_list for retry: */
+		list_splice_init(&demote_pages, page_list);
+		do_demote_pass = false;
+		goto retry;
+	}
 
 	pgactivate = stat->nr_activate[0] + stat->nr_activate[1];
 

_

Quoted 44 lines · Dave Hansen

From: Dave Hansen <[email protected]>

This is mostly derived from a patch from Yang Shi:

      https://lore.kernel.org/linux-mm/[email protected]/

Add code to the reclaim path (shrink_page_list()) to "demote" data to another NUMA node instead of discarding the data. This always avoids the cost of I/O needed to read the page back in and sometimes avoids the writeout cost when the pagee is dirty.

A second pass through shrink_page_list() will be made if any demotions fail. This essentally falls back to normal reclaim behavior in the case that demotions fail. Previous versions of this patch may have simply failed to reclaim pages which were eligible for demotion but were unable to be demoted in practice.

Note: This just adds the start of infratructure for migration. It is actually disabled next to the FIXME in migrate_demote_page_ok().

Signed-off-by Dave Hansen <[email protected]>

Cc Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc osalvador <[email protected]>

--

changes from 202010:
* add MR_NUMA_MISPLACED to trace MIGRATE_REASON define
* make migrate_demote_page_ok() static, remove 'sc' arg until later patch
* remove unnecessary alloc_demote_page() hugetlb warning
* Simplify alloc_demote_page() gfp mask. Depend on __GFP_NORETRY to make it lightweight instead of fancier stuff like leaving out __GFP_IO/FS.
* Allocate migration page with alloc_migration_target() instead of allocating directly.
changes from 20200730:
* Add another pass through shrink_page_list() when demotion fails.
---

[...]

mm/vmscan.c static void page_check_dirty_writeback(s · line 1100

Quoted 13 lines

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +13 -0

+static struct page *alloc_demote_page(struct page *page, unsigned long node)
+{
+        struct migration_target_control mtc = {
+             /*
+              * Fail quickly and quietly.  Page will likely
+              * just be discarded instead of migrated.
+              */
+             .gfp_mask = GFP_HIGHUSER | __GFP_NORETRY | __GFP_NOWARN,
+             .nid = node
+     };
+
+        return alloc_migration_target(page, (unsigned long)&mtc);
+}

Migration for THP pages will set direct reclaim. I guess that is fine right? AFAIK, direct reclaim will only be tried once with GFP_NORETRY.

Quoted 47 lines

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +28 -0

+
+/*
+ * Take pages on @demote_list and attempt to demote them to
+ * another node.  Pages which are not demoted are left on
+ * @demote_pages.
+ */
+static unsigned int demote_page_list(struct list_head *demote_pages,
+                                  struct pglist_data *pgdat,
+                                  struct scan_control *sc)
+{
+     int target_nid = next_demotion_node(pgdat->node_id);
+     unsigned int nr_succeeded = 0;
+     int err;
+
+     if (list_empty(demote_pages))
+             return 0;
+
+     /* Demotion ignores all cpuset and mempolicy settings */
+     err = migrate_pages(demote_pages, alloc_demote_page, NULL,
+                         target_nid, MIGRATE_ASYNC, MR_DEMOTION,
+                         &nr_succeeded);
+
+     return nr_succeeded;
+}
+
 /*
  * shrink_page_list() returns the number of reclaimed pages
  */

@@ -1078,12 +1135,15 @@ static unsigned int shrink_page_list(str
 {
      LIST_HEAD(ret_pages);
      LIST_HEAD(free_pages);
+     LIST_HEAD(demote_pages);
      unsigned int nr_reclaimed = 0;
      unsigned int pgactivate = 0;
+     bool do_demote_pass = true;
 
      memset(stat, 0, sizeof(*stat));
      cond_resched();
 
+retry:
      while (!list_empty(page_list)) {
              struct address_space *mapping;
              struct page *page;

@@ -1233,6 +1293,16 @@ static unsigned int shrink_page_list(str
              }
 

mm/vmscan.c static unsigned int shrink_page_list(str · line 1303

Quoted 9 lines

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +8 -0

              /*
+              * Before reclaiming the page, try to relocate
+              * its contents to another node.
+              */
+             if (do_demote_pass && migrate_demote_page_ok(page)) {
+                     list_add(&page->lru, &demote_pages);
+                     unlock_page(page);
+                     continue;
+             }

Should we keep it simple for now and only try to demote those pages that are free of cpusets and memory policies?
Actually, demoting those pages to a CPU or a NUMA node that does not fall into their set, would violate those constraints right?

So I think we should leave those pages alone for now.

Quoted 23 lines

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +13 -0

+
+             /*
               * Anonymous process memory has backing store?
               * Try to allocate it some swap space here.
               * Lazyfree page could be freed directly

@@ -1479,6 +1549,17 @@ keep:
              list_add(&page->lru, &ret_pages);
              VM_BUG_ON_PAGE(PageLRU(page) || PageUnevictable(page), page);
      }
+     /* 'page_list' is always empty here */
+
+     /* Migrate pages selected for demotion */
+     nr_reclaimed += demote_page_list(&demote_pages, pgdat, sc);
+     /* Pages that could not be demoted are still in @demote_pages */
+     if (!list_empty(&demote_pages)) {
+             /* Pages which failed to demoted go back on on @page_list for retry: */
+             list_splice_init(&demote_pages, page_list);
+             do_demote_pass = false;
+             goto retry;
+     }
 
      pgactivate = stat->nr_activate[0] + stat->nr_activate[1];
 

Quoted 2 lines

_

--
Oscar Salvador
SUSE L3

Quoted 4 lines

Should we keep it simple for now and only try to demote those pages that are free of cpusets and memory policies?
Actually, demoting those pages to a CPU or a NUMA node that does not fall into their set, would violate those constraints right?

Yes, this has been discussed since the very beginning. There is not an easy way to figure out the memory placement policy (cpuset and mempolicy) from "page". I think this also prevents "demote those pages that are free of cpusets and memory policies".

The conclusion was the violation should be fine for now. And the demotion feature is opt'ed in by a new node reclaim mode.

From: Dave Hansen <[email protected]>

This is mostly derived from a patch from Yang Shi:

        https://lore.kernel.org/linux-mm/[email protected]/

Add code to the reclaim path (shrink_page_list()) to "demote" data to another NUMA node instead of discarding the data. This always avoids the cost of I/O needed to read the page back in and sometimes avoids the writeout cost when the pagee is dirty.

A second pass through shrink_page_list() will be made if any demotions fail. This essentally falls back to normal reclaim behavior in the case that demotions fail. Previous versions of this patch may have simply failed to reclaim pages which were eligible for demotion but were unable to be demoted in practice.

Note: This just adds the start of infratructure for migration. It is actually disabled next to the FIXME in migrate_demote_page_ok().

Signed-off-by Dave Hansen <[email protected]>

Cc Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc osalvador <[email protected]>

--

changes from 202010:
* add MR_NUMA_MISPLACED to trace MIGRATE_REASON define
* make migrate_demote_page_ok() static, remove 'sc' arg until later patch
* remove unnecessary alloc_demote_page() hugetlb warning
* Simplify alloc_demote_page() gfp mask. Depend on __GFP_NORETRY to make it lightweight instead of fancier stuff like leaving out __GFP_IO/FS.
* Allocate migration page with alloc_migration_target() instead of allocating directly.
changes from 20200730:
* Add another pass through shrink_page_list() when demotion fails.
---

 b/include/linux/migrate.h        |    9 ++++
 b/include/trace/events/migrate.h |    3 -
 b/mm/vmscan.c                    |   81 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 92 insertions(+), 1 deletion(-)

diff -puN include/linux/migrate.h~demote-with-migrate_pages include/linux/migrate.h

include/linux/migrate.h 2021-01-25 16:23:14.599866696 -0800 renamed from include/linux/migrate.h~demote-with-migrate_pages 2021-01-25 16:23:14.591866696 -0800 +9 -0

@@ -27,6 +27,7 @@ enum migrate_reason {
        MR_MEMPOLICY_MBIND,
        MR_NUMA_MISPLACED,
        MR_CONTIG_RANGE,
+       MR_DEMOTION,
        MR_TYPES
 };
 

@@ -196,6 +197,14 @@ struct migrate_vma {
 int migrate_vma_setup(struct migrate_vma *args);
 void migrate_vma_pages(struct migrate_vma *migrate);
 void migrate_vma_finalize(struct migrate_vma *migrate);
+int next_demotion_node(int node);
+
+#else /* CONFIG_MIGRATION disabled: */
+
+static inline int next_demotion_node(int node)
+{
+       return NUMA_NO_NODE;
+}
 
 #endif /* CONFIG_MIGRATION */
 

diff -puN include/trace/events/migrate.h~demote-with-migrate_pages include/trace/events/migrate.h

include/trace/events/migrate.h 2021-01-25 16:23:14.599866696 -0800 renamed from include/trace/events/migrate.h~demote-with-migrate_pages 2021-01-25 16:23:14.593866696 -0800 +2 -1

@@ -20,7 +20,8 @@
        EM( MR_SYSCALL,         "syscall_or_cpuset")            \
        EM( MR_MEMPOLICY_MBIND, "mempolicy_mbind")              \
        EM( MR_NUMA_MISPLACED,  "numa_misplaced")               \
-       EMe(MR_CONTIG_RANGE,    "contig_range")
+       EM( MR_CONTIG_RANGE,    "contig_range")                 \
+       EMe(MR_DEMOTION,        "demotion")
 
 /*
  * First define the enums in the above macros to be exported to userspace

diff -puN mm/vmscan.c~demote-with-migrate_pages mm/vmscan.c

mm/vmscan.c 2021-01-25 16:23:14.601866696 -0800 renamed from mm/vmscan.c~demote-with-migrate_pages 2021-01-25 16:23:14.595866696 -0800 +21 -0

@@ -43,6 +43,7 @@
 #include <linux/kthread.h>
 #include <linux/freezer.h>
 #include <linux/memcontrol.h>
+#include <linux/migrate.h>
 #include <linux/delayacct.h>
 #include <linux/sysctl.h>
 #include <linux/oom.h>

@@ -1036,6 +1037,24 @@ static enum page_references page_check_r
        return PAGEREF_RECLAIM;
 }
 
+static bool migrate_demote_page_ok(struct page *page)
+{
+       int next_nid = next_demotion_node(page_to_nid(page));
+
+       VM_BUG_ON_PAGE(!PageLocked(page), page);
+       VM_BUG_ON_PAGE(PageHuge(page), page);
+       VM_BUG_ON_PAGE(PageLRU(page), page);
+
+       if (next_nid == NUMA_NO_NODE)
+               return false;
+       if (PageTransHuge(page) && !thp_migration_supported())
+               return false;
+
+       // FIXME: actually enable this later in the series
+       return false;
+}
+
+
 /* Check if a page is dirty or under writeback */
 static void page_check_dirty_writeback(struct page *page,
                                       bool *dirty, bool *writeback)

@@ -1066,6 +1085,44 @@ static void page_check_dirty_writeback(s
                mapping->a_ops->is_dirty_writeback(page, dirty, writeback);
 }
 
+static struct page *alloc_demote_page(struct page *page, unsigned long node)
+{

mm/vmscan.c 2021-01-25 16:23:14.601866696 -0800 renamed from mm/vmscan.c~demote-with-migrate_pages 2021-01-25 16:23:14.595866696 -0800 +50 -0

+}
+
+/*
+ * Take pages on @demote_list and attempt to demote them to
+ * another node.  Pages which are not demoted are left on
+ * @demote_pages.
+ */
+static unsigned int demote_page_list(struct list_head *demote_pages,
+                                    struct pglist_data *pgdat,
+                                    struct scan_control *sc)
+{
+       int target_nid = next_demotion_node(pgdat->node_id);
+       unsigned int nr_succeeded = 0;
+       int err;
+
+       if (list_empty(demote_pages))
+               return 0;
+
+       /* Demotion ignores all cpuset and mempolicy settings */
+       err = migrate_pages(demote_pages, alloc_demote_page, NULL,
+                           target_nid, MIGRATE_ASYNC, MR_DEMOTION,
+                           &nr_succeeded);
+
+       return nr_succeeded;
+}
+
 /*
  * shrink_page_list() returns the number of reclaimed pages
  */

@@ -1078,12 +1135,15 @@ static unsigned int shrink_page_list(str
 {
        LIST_HEAD(ret_pages);
        LIST_HEAD(free_pages);
+       LIST_HEAD(demote_pages);
        unsigned int nr_reclaimed = 0;
        unsigned int pgactivate = 0;
+       bool do_demote_pass = true;
 
        memset(stat, 0, sizeof(*stat));
        cond_resched();
 
+retry:
        while (!list_empty(page_list)) {
                struct address_space *mapping;
                struct page *page;

@@ -1233,6 +1293,16 @@ static unsigned int shrink_page_list(str
                }
 
                /*
+                * Before reclaiming the page, try to relocate
+                * its contents to another node.
+                */
+               if (do_demote_pass && migrate_demote_page_ok(page)) {
+                       list_add(&page->lru, &demote_pages);
+                       unlock_page(page);
+                       continue;
+               }
+
+               /*
                 * Anonymous process memory has backing store?
                 * Try to allocate it some swap space here.
                 * Lazyfree page could be freed directly

@@ -1479,6 +1549,17 @@ keep:
                list_add(&page->lru, &ret_pages);
                VM_BUG_ON_PAGE(PageLRU(page) || PageUnevictable(page), page);
        }
+       /* 'page_list' is always empty here */
+
+       /* Migrate pages selected for demotion */
+       nr_reclaimed += demote_page_list(&demote_pages, pgdat, sc);
+       /* Pages that could not be demoted are still in @demote_pages */
+       if (!list_empty(&demote_pages)) {
+               /* Pages which failed to demoted go back on on @page_list for retry: */
+               list_splice_init(&demote_pages, page_list);
+               do_demote_pass = false;
+               goto retry;
+       }
 
        pgactivate = stat->nr_activate[0] + stat->nr_activate[1];
 

_

mm/vmscan.c static void page_check_dirty_writeback(s · line 1099

Quoted 12 lines

mm/vmscan.c renamed from mm/vmscan.c~demote-with-migrate_pages +12 -0

+static struct page *alloc_demote_page(struct page *page, unsigned long node)
+{
+        struct migration_target_control mtc = {
+               /*
+                * Fail quickly and quietly.  Page will likely
+                * just be discarded instead of migrated.
+                */
+               .gfp_mask = GFP_HIGHUSER | __GFP_NORETRY | __GFP_NOWARN,
+               .nid = node
+       };
+
+        return alloc_migration_target(page, (unsigned long)&mtc);

Other than the gfp flag question raised by Oscar, I'm wondering how we guarantee the demotion allocation happens on the designated node. In the previous version __GFP_THISNODE is set to guarantee this. In this version you switched to use alloc_migration_target() API but without having nodemask or __GFP_THISNODE. If nodemask is NULL the allocation may fall back to an unexpected node.

And GFP_HIGHUSER does respect cpuset, so if the demotion target node is excluded by the cpuset which the task belongs to, the migration would fail. This might be a way to respect cpuset, but it should just work for direct reclaimer. So, is this change really expected?

From: Keith Busch <[email protected]>

Reclaim anonymous pages if a migration path is available now that demotion provides a non-swap recourse for reclaiming anon pages.

Note that this check is subtly different from the anon_should_be_aged() checks. This mechanism checks whether a specific page in a specific context *can* actually be reclaimed, given current swap space and cgroup limits

anon_should_be_aged() is a much simpler and more prelimiary check which just says whether there is a possibility of future reclaim.

#Signed-off-by: Keith Busch <[email protected]>

Cc Keith Busch <[email protected]>

[vishal: fixup the migration->demotion rename]

Signed-off-by Vishal Verma <[email protected]>

Signed-off-by Dave Hansen <[email protected]>

Cc Yang Shi <[email protected]>

Cc David Rientjes <[email protected]>

Cc Huang Ying <[email protected]>

Cc Dan Williams <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc osalvador <[email protected]>

--

Changes from Dave 10/2020:
* remove 'total_swap_pages' modification

Changes from Dave 06/2020:
* rename reclaim_anon_pages()->can_reclaim_anon_pages()

Note: Keith's Intel SoB is commented out because he is no longer at Intel and his @intel.com mail will bouncee
---

 b/mm/vmscan.c |   35 ++++++++++++++++++++++++++++++++---
 1 file changed, 32 insertions(+), 3 deletions(-)

diff -puN mm/vmscan.c~0009-mm-vmscan-Consider-anonymous-pages-without-swap mm/vmscan.c

mm/vmscan.c 2021-01-25 16:23:18.111866688 -0800 renamed from mm/vmscan.c~0009-mm-vmscan-Consider-anonymous-pages-without-swap 2021-01-25 16:23:18.106866688 -0800 +30 -1

@@ -289,6 +289,34 @@ static bool writeback_throttling_sane(st
 }
 #endif
 
+static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
+                                         int node_id)
+{
+       if (memcg == NULL) {
+               /*
+                * For non-memcg reclaim, is there
+                * space in any swap device?
+                */
+               if (get_nr_swap_pages() > 0)
+                       return true;
+       } else {
+               /* Is the memcg below its swap limit? */
+               if (mem_cgroup_get_nr_swap_pages(memcg) > 0)
+                       return true;
+       }
+
+       /*
+        * The page can not be swapped.
+        *
+        * Can it be reclaimed from this node via demotion?
+        */
+       if (next_demotion_node(node_id) >= 0)
+               return true;
+
+       /* No way to reclaim anon pages */
+       return false;
+}
+
 /*
  * This misses isolated pages which are not accounted for to save counters.
  * As the data only determines if reclaim or compaction continues, it is

@@ -300,7 +328,7 @@ unsigned long zone_reclaimable_pages(str
 
        nr = zone_page_state_snapshot(zone, NR_ZONE_INACTIVE_FILE) +
                zone_page_state_snapshot(zone, NR_ZONE_ACTIVE_FILE);
-       if (get_nr_swap_pages() > 0)
+       if (can_reclaim_anon_pages(NULL, zone_to_nid(zone)))
                nr += zone_page_state_snapshot(zone, NR_ZONE_INACTIVE_ANON) +
                        zone_page_state_snapshot(zone, NR_ZONE_ACTIVE_ANON);
 

@@ -2323,6 +2351,7 @@ enum scan_balance {
 static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc,
                           unsigned long *nr)
 {
+       struct pglist_data *pgdat = lruvec_pgdat(lruvec);
        struct mem_cgroup *memcg = lruvec_memcg(lruvec);
        unsigned long anon_cost, file_cost, total_cost;
        int swappiness = mem_cgroup_swappiness(memcg);

mm/vmscan.c 2021-01-25 16:23:18.111866688 -0800 renamed from mm/vmscan.c~0009-mm-vmscan-Consider-anonymous-pages-without-swap 2021-01-25 16:23:18.106866688 -0800 +1 -1

                scan_balance = SCAN_FILE;
                goto out;
        }

@@ -2708,7 +2737,7 @@ static inline bool should_continue_recla
         */
        pages_for_compaction = compact_gap(sc->order);
        inactive_lru_pages = node_page_state(pgdat, NR_INACTIVE_FILE);
-       if (get_nr_swap_pages() > 0)
+       if (can_reclaim_anon_pages(NULL, pgdat->node_id))
                inactive_lru_pages += node_page_state(pgdat, NR_INACTIVE_ANON);
 
        return inactive_lru_pages > pages_for_compaction;

_

mm/vmscan.c static void get_scan_count(struct lruvec · line 2365

Quoted 4 lines

        /* If we have no swap space, do not bother scanning anon pages. */

mm/vmscan.c renamed from mm/vmscan.c~0009-mm-vmscan-Consider-anonymous-pages-without-swap +1 -1

-       if (!sc->may_swap || mem_cgroup_get_nr_swap_pages(memcg) <= 0) {
+       if (!sc->may_swap || !can_reclaim_anon_pages(memcg, pgdat->node_id)) {

Just one minor thing about may_swap. It may be cleared by nr_boost_reclaim. But demotion should be fine for boost_reclaim.

Quoted 6 lines · Yang Shi

mm/vmscan.c static void get_scan_count(struct lruvec · line 2365

Quoted 4 lines

        /* If we have no swap space, do not bother scanning anon pages. */

mm/vmscan.c renamed from mm/vmscan.c~0009-mm-vmscan-Consider-anonymous-pages-without-swap +1 -1

-       if (!sc->may_swap || mem_cgroup_get_nr_swap_pages(memcg) <= 0) {
+       if (!sc->may_swap || !can_reclaim_anon_pages(memcg, pgdat->node_id)) {

Just one minor thing about may_swap. It may be cleared by nr_boost_reclaim. But demotion should be fine for boost_reclaim.

In other words, this if() is here is to avoid generating suboptimal I/O during boost_reclaim. But, since demotion doesn't generate any I/O, it *should* be fine for boost_reclaim.

I agree with that in theory. Although, I'm tempted to put it in the TODO list as something to look at in the future. Do you think it's something that's immediately necessary?

This also contains a few prerequisite patches that fix up an issue with the vm.zone_reclaim_mode sysctl ABI.