Skip to contentSkip to conversation

Migrate Pages in lieu of discard

RFC PATCH In review Dave Hansen 8 participants 30 messages Oct 16, 2019 – Oct 22, 2019

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

cover Migrate Pages in lieu of discard

We're starting to see systems with more and more kinds of memory such as Intel's implementation of persistent memory.

Let's say you have a system with some DRAM and some persistent memory. Today, once DRAM fills up, reclaim will start and some of the DRAM contents will be thrown out. Allocations will, at some point, start falling over to the slower persistent memory.

That has two nasty properties. First, the newer allocations can end up in the slower persistent memory. Second, reclaimed data in DRAM are just discarded even if there are gobs of space in persistent memory that could be used.

This set implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.

While I've talked about a DRAM/PMEM pairing, this approach would function in any environment where memory tiers exist.

This is not perfect. It "strands" pages in slower memory and never brings them back to fast DRAM. Other things need to be built to promote hot pages back to DRAM.

This is part of a larger patch set. If you want to apply these or play with them, I'd suggest using the tree from here. It includes autonuma-based hot page promotion back to DRAM:

	http://lkml.kernel.org/r/[email protected]

This is also all based on an upstream mechanism that allows persistent memory to be onlined and used as if it were volatile:

	http://lkml.kernel.org/r/[email protected]
#1 lore raw

PATCH 1/4 node: Define and export memory migration path

From: Keith Busch <[email protected]>

Prepare for the kernel to auto-migrate pages to other memory nodes with a user defined node migration table. This allows creating single migration target for each NUMA node to enable the kernel to do NUMA page migrations instead of simply reclaiming colder pages. A node with no target is a "terminal node", so reclaim acts normally there. The migration target does not fundamentally _need_ to be a single node, but this implementation starts there to limit complexity.

If you consider the migration path as a graph, cycles (loops) in the graph are disallowed. This avoids wasting resources by constantly migrating (A->B, B->A, A->B ...). The expectation is that cycles will never be allowed, and this rule is enforced if the user tries to make such a cycle.

---

 b/drivers/base/node.c  |   73 +++++++++++++++++++++++++++++++++++++++++++++++++
 b/include/linux/node.h |    6 ++++
 2 files changed, 79 insertions(+)
drivers/base/node.c renamed from drivers/base/node.c~0003-node-Define-and-export-memory-migration-path +73 -0
diff -puN drivers/base/node.c~0003-node-Define-and-export-memory-migration-path drivers/base/node.c
@@ -101,6 +101,10 @@ static const struct attribute_group *nod
 	NULL,
 };
 
+#define TERMINAL_NODE -1
+static int node_migration[MAX_NUMNODES] = {[0 ...  MAX_NUMNODES - 1] = TERMINAL_NODE};
+static DEFINE_SPINLOCK(node_migration_lock);
+
 static void node_remove_accesses(struct node *node)
 {
 	struct node_access_nodes *c, *cnext;
@@ -530,6 +534,74 @@ static ssize_t node_read_distance(struct
 }
 static DEVICE_ATTR(distance, S_IRUGO, node_read_distance, NULL);
 
+static ssize_t migration_path_show(struct device *dev,
+				   struct device_attribute *attr,
+				   char *buf)
+{
+	return sprintf(buf, "%d\n", node_migration[dev->id]);
+}
+
+static ssize_t migration_path_store(struct device *dev,
+				    struct device_attribute *attr,
+				    const char *buf, size_t count)
+{
+	int i, err, nid = dev->id;
+	nodemask_t visited = NODE_MASK_NONE;
+	long next;
+
+	err = kstrtol(buf, 0, &next);
+	if (err)
+		return -EINVAL;
+
+	if (next < 0) {
+		spin_lock(&node_migration_lock);
+		WRITE_ONCE(node_migration[nid], TERMINAL_NODE);
+		spin_unlock(&node_migration_lock);
+		return count;
+	}
+	if (next >= MAX_NUMNODES || !node_online(next))
+		return -EINVAL;
+
+	/*
+	 * Follow the entire migration path from 'nid' through the point where
+	 * we hit a TERMINAL_NODE.
+	 *
+	 * Don't allow loops migration cycles in the path.
+	 */
+	node_set(nid, visited);
+	spin_lock(&node_migration_lock);
+	for (i = next; node_migration[i] != TERMINAL_NODE;
+	     i = node_migration[i]) {
+		/* Fail if we have visited this node already */
+		if (node_test_and_set(i, visited)) {
+			spin_unlock(&node_migration_lock);
+			return -EINVAL;
+		}
+	}
+	WRITE_ONCE(node_migration[nid], next);
+	spin_unlock(&node_migration_lock);
+
+	return count;
+}
+static DEVICE_ATTR_RW(migration_path);
+
+/**
+ * next_migration_node() - Get the next node in the migration path
+ * @current_node: The starting node to lookup the next node
+ *
+ * @returns: node id for next memory node in the migration path hierarchy from
+ * 	     @current_node; -1 if @current_node is terminal or its migration
+ * 	     node is not online.
+ */
+int next_migration_node(int current_node)
+{
+	int nid = READ_ONCE(node_migration[current_node]);
+
+	if (nid >= 0 && node_online(nid))
+		return nid;
+	return TERMINAL_NODE;
+}
+
 static struct attribute *node_dev_attrs[] = {
 	&dev_attr_cpumap.attr,
 	&dev_attr_cpulist.attr,
@@ -537,6 +609,7 @@ static struct attribute *node_dev_attrs[
 	&dev_attr_numastat.attr,
 	&dev_attr_distance.attr,
 	&dev_attr_vmstat.attr,
+	&dev_attr_migration_path.attr,
 	NULL
 };
 ATTRIBUTE_GROUPS(node_dev);
include/linux/node.h renamed from include/linux/node.h~0003-node-Define-and-export-memory-migration-path +6 -0
diff -puN include/linux/node.h~0003-node-Define-and-export-memory-migration-path include/linux/node.h
@@ -134,6 +134,7 @@ static inline int register_one_node(int
 	return error;
 }
 
+extern int next_migration_node(int current_node);
 extern void unregister_one_node(int nid);
 extern int register_cpu_under_node(unsigned int cpu, unsigned int nid);
 extern int unregister_cpu_under_node(unsigned int cpu, unsigned int nid);
@@ -186,6 +187,11 @@ static inline void register_hugetlbfs_wi
 						node_registration_func_t unreg)
 {
 }
+
+static inline int next_migration_node(int current_node)
+{
+	return -1;
+}
 #endif
 
 #define to_node(device) container_of(device, struct node, dev)

_

#2 lore raw

Re: [PATCH 1/4] node: Define and export memory migration path

Quoted 29 lines · Dave Hansen

From: Keith Busch <[email protected]>

Prepare for the kernel to auto-migrate pages to other memory nodes with a user defined node migration table. This allows creating single migration target for each NUMA node to enable the kernel to do NUMA page migrations instead of simply reclaiming colder pages. A node with no target is a "terminal node", so reclaim acts normally there. The migration target does not fundamentally _need_ to be a single node, but this implementation starts there to limit complexity.

If you consider the migration path as a graph, cycles (loops) in the graph are disallowed. This avoids wasting resources by constantly migrating (A->B, B->A, A->B ...). The expectation is that cycles will never be allowed, and this rule is enforced if the user tries to make such a cycle.

---

 b/drivers/base/node.c  |   73 +++++++++++++++++++++++++++++++++++++++++++++++++
 b/include/linux/node.h |    6 ++++
 2 files changed, 79 insertions(+)

diff -puN drivers/base/node.c~0003-node-Define-and-export-memory-migration-path drivers/base/node.c

drivers/base/node.c static const struct attribute_group *nod · line 104

Quoted 4 lines
drivers/base/node.c renamed from drivers/base/node.c~0003-node-Define-and-export-memory-migration-path +1 -0
@@ -101,6 +101,10 @@ static const struct attribute_group *nod
 	NULL,
 };
 
+#define TERMINAL_NODE -1

Wouldn't we have a confusion with NUMA_NO_NODE, which is also -1?

drivers/base/node.c static const struct attribute_group *nod · line 105

Quoted 1 line
drivers/base/node.c renamed from drivers/base/node.c~0003-node-Define-and-export-memory-migration-path +1 -0
+static int node_migration[MAX_NUMNODES] = {[0 ...  MAX_NUMNODES - 1] = TERMINAL_NODE};

This is the first time is see range initializer in kernel code. It is GCC extension. Do we use it anywhere already?

Many distributions compile kernel with NODES_SHIFT==10, which means this array will take 4k even on single node machine.

Should it be dynamic?

Quoted 19 lines
drivers/base/node.c renamed from drivers/base/node.c~0003-node-Define-and-export-memory-migration-path +12 -0
+static DEFINE_SPINLOCK(node_migration_lock);
+
 static void node_remove_accesses(struct node *node)
 {
 	struct node_access_nodes *c, *cnext;
@@ -530,6 +534,74 @@ static ssize_t node_read_distance(struct
 }
 static DEVICE_ATTR(distance, S_IRUGO, node_read_distance, NULL);
 
+static ssize_t migration_path_show(struct device *dev,
+				   struct device_attribute *attr,
+				   char *buf)
+{
+	return sprintf(buf, "%d\n", node_migration[dev->id]);
+}
+
+static ssize_t migration_path_store(struct device *dev,
+				    struct device_attribute *attr,
+				    const char *buf, size_t count)

drivers/base/node.c static ssize_t node_read_distance(struct · line 556

Quoted 10 lines
drivers/base/node.c renamed from drivers/base/node.c~0003-node-Define-and-export-memory-migration-path +10 -0
+{
+	int i, err, nid = dev->id;
+	nodemask_t visited = NODE_MASK_NONE;
+	long next;
+
+	err = kstrtol(buf, 0, &next);
+	if (err)
+		return -EINVAL;
+
+	if (next < 0) {

Any negative number to set it to terminal node? Why not limit it to -1? We may find use for user negative numbers later.

drivers/base/node.c static ssize_t node_read_distance(struct · line 563

Quoted 7 lines
drivers/base/node.c renamed from drivers/base/node.c~0003-node-Define-and-export-memory-migration-path +7 -0
+		spin_lock(&node_migration_lock);
+		WRITE_ONCE(node_migration[nid], TERMINAL_NODE);
+		spin_unlock(&node_migration_lock);
+		return count;
+	}
+	if (next >= MAX_NUMNODES || !node_online(next))
+		return -EINVAL;

What prevents offlining after the check?

Quoted 76 lines
drivers/base/node.c renamed from drivers/base/node.c~0003-node-Define-and-export-memory-migration-path +41 -0
+	/*
+	 * Follow the entire migration path from 'nid' through the point where
+	 * we hit a TERMINAL_NODE.
+	 *
+	 * Don't allow loops migration cycles in the path.
+	 */
+	node_set(nid, visited);
+	spin_lock(&node_migration_lock);
+	for (i = next; node_migration[i] != TERMINAL_NODE;
+	     i = node_migration[i]) {
+		/* Fail if we have visited this node already */
+		if (node_test_and_set(i, visited)) {
+			spin_unlock(&node_migration_lock);
+			return -EINVAL;
+		}
+	}
+	WRITE_ONCE(node_migration[nid], next);
+	spin_unlock(&node_migration_lock);
+
+	return count;
+}
+static DEVICE_ATTR_RW(migration_path);
+
+/**
+ * next_migration_node() - Get the next node in the migration path
+ * @current_node: The starting node to lookup the next node
+ *
+ * @returns: node id for next memory node in the migration path hierarchy from
+ * 	     @current_node; -1 if @current_node is terminal or its migration
+ * 	     node is not online.
+ */
+int next_migration_node(int current_node)
+{
+	int nid = READ_ONCE(node_migration[current_node]);
+
+	if (nid >= 0 && node_online(nid))
+		return nid;
+	return TERMINAL_NODE;
+}
+
 static struct attribute *node_dev_attrs[] = {
 	&dev_attr_cpumap.attr,
 	&dev_attr_cpulist.attr,
@@ -537,6 +609,7 @@ static struct attribute *node_dev_attrs[
 	&dev_attr_numastat.attr,
 	&dev_attr_distance.attr,
 	&dev_attr_vmstat.attr,
+	&dev_attr_migration_path.attr,
 	NULL
 };
 ATTRIBUTE_GROUPS(node_dev);

diff -puN include/linux/node.h~0003-node-Define-and-export-memory-migration-path include/linux/node.h

include/linux/node.h renamed from include/linux/node.h~0003-node-Define-and-export-memory-migration-path +6 -0
@@ -134,6 +134,7 @@ static inline int register_one_node(int
 	return error;
 }
 
+extern int next_migration_node(int current_node);
 extern void unregister_one_node(int nid);
 extern int register_cpu_under_node(unsigned int cpu, unsigned int nid);
 extern int unregister_cpu_under_node(unsigned int cpu, unsigned int nid);
@@ -186,6 +187,11 @@ static inline void register_hugetlbfs_wi
 						node_registration_func_t unreg)
 {
 }
+
+static inline int next_migration_node(int current_node)
+{
+	return -1;
+}
 #endif
 
 #define to_node(device) container_of(device, struct node, dev)

_

Signature · 1 line
 Kirill A. Shutemov
#3 lore raw

Re: [PATCH 1/4] node: Define and export memory migration path

Quoted 9 lines · Kirill A. Shutemov

drivers/base/node.c static ssize_t node_read_distance(struct · line 563

Quoted 7 lines
drivers/base/node.c renamed from drivers/base/node.c~0003-node-Define-and-export-memory-migration-path +7 -0
+		spin_lock(&node_migration_lock);
+		WRITE_ONCE(node_migration[nid], TERMINAL_NODE);
+		spin_unlock(&node_migration_lock);
+		return count;
+	}
+	if (next >= MAX_NUMNODES || !node_online(next))
+		return -EINVAL;

What prevents offlining after the check?

And what is story with memory hotplug interaction? I don't see any hooks into memory hotplug to adjust migration path on offlining. Hm?

Signature · 1 line
 Kirill A. Shutemov
#4 lore raw

PATCH 2/4 mm/migrate: Defer allocating new page until needed

In reply to Dave Hansen

From: Keith Busch <[email protected]>

Migrating pages had been allocating the new page before it was actually needed. Subsequent operations may still fail, which would have to handle cleaning up the newly allocated page when it was never used.

Defer allocating the page until we are actually ready to make use of it, after locking the original page. This simplifies error handling, but should not have any functional change in behavior. This is just refactoring page migration so the main part can more easily be reused by other code.

---

 b/mm/migrate.c |  154 ++++++++++++++++++++++++++++-----------------------------
 1 file changed, 76 insertions(+), 78 deletions(-)
mm/migrate.c renamed from mm/migrate.c~0004-mm-migrate-Defer-allocating-new-page-until-needed +76 -78
diff -puN mm/migrate.c~0004-mm-migrate-Defer-allocating-new-page-until-needed mm/migrate.c
@@ -1005,56 +1005,17 @@ out:
 	return rc;
 }
 
-static int __unmap_and_move(struct page *page, struct page *newpage,
-				int force, enum migrate_mode mode)
+static int __unmap_and_move(new_page_t get_new_page,
+			    free_page_t put_new_page,
+			    unsigned long private, struct page *page,
+			    enum migrate_mode mode,
+			    enum migrate_reason reason)
 {
 	int rc = -EAGAIN;
 	int page_was_mapped = 0;
 	struct anon_vma *anon_vma = NULL;
 	bool is_lru = !__PageMovable(page);
-
-	if (!trylock_page(page)) {
-		if (!force || mode == MIGRATE_ASYNC)
-			goto out;
-
-		/*
-		 * It's not safe for direct compaction to call lock_page.
-		 * For example, during page readahead pages are added locked
-		 * to the LRU. Later, when the IO completes the pages are
-		 * marked uptodate and unlocked. However, the queueing
-		 * could be merging multiple pages for one bio (e.g.
-		 * mpage_readpages). If an allocation happens for the
-		 * second or third page, the process can end up locking
-		 * the same page twice and deadlocking. Rather than
-		 * trying to be clever about what pages can be locked,
-		 * avoid the use of lock_page for direct compaction
-		 * altogether.
-		 */
-		if (current->flags & PF_MEMALLOC)
-			goto out;
-
-		lock_page(page);
-	}
-
-	if (PageWriteback(page)) {
-		/*
-		 * Only in the case of a full synchronous migration is it
-		 * necessary to wait for PageWriteback. In the async case,
-		 * the retry loop is too short and in the sync-light case,
-		 * the overhead of stalling is too much
-		 */
-		switch (mode) {
-		case MIGRATE_SYNC:
-		case MIGRATE_SYNC_NO_COPY:
-			break;
-		default:
-			rc = -EBUSY;
-			goto out_unlock;
-		}
-		if (!force)
-			goto out_unlock;
-		wait_on_page_writeback(page);
-	}
+	struct page *newpage;
 
 	/*
 	 * By try_to_unmap(), page->mapcount goes down to 0 here. In this case,
@@ -1073,6 +1034,12 @@ static int __unmap_and_move(struct page
 	if (PageAnon(page) && !PageKsm(page))
 		anon_vma = page_get_anon_vma(page);
 
+	newpage = get_new_page(page, private);
+	if (!newpage) {
+		rc = -ENOMEM;
+		goto out;
+	}
+
 	/*
 	 * Block others from accessing the new page when we get around to
 	 * establishing additional references. We are usually the only one
@@ -1082,11 +1049,11 @@ static int __unmap_and_move(struct page
 	 * This is much like races on refcount of oldpage: just don't BUG().
 	 */
 	if (unlikely(!trylock_page(newpage)))
-		goto out_unlock;
+		goto out_put;
 
 	if (unlikely(!is_lru)) {
 		rc = move_to_new_page(newpage, page, mode);
-		goto out_unlock_both;
+		goto out_unlock;
 	}
 
 	/*
@@ -1105,7 +1072,7 @@ static int __unmap_and_move(struct page
 		VM_BUG_ON_PAGE(PageAnon(page), page);
 		if (page_has_private(page)) {
 			try_to_free_buffers(page);
-			goto out_unlock_both;
+			goto out_unlock;
 		}
 	} else if (page_mapped(page)) {
 		/* Establish migration ptes */
@@ -1122,15 +1089,9 @@ static int __unmap_and_move(struct page
 	if (page_was_mapped)
 		remove_migration_ptes(page,
 			rc == MIGRATEPAGE_SUCCESS ? newpage : page, false);
-
-out_unlock_both:
-	unlock_page(newpage);
 out_unlock:
-	/* Drop an anon_vma reference if we took one */
-	if (anon_vma)
-		put_anon_vma(anon_vma);
-	unlock_page(page);
-out:
+	unlock_page(newpage);
+out_put:
 	/*
 	 * If migration is successful, decrease refcount of the newpage
 	 * which will not free the page because new page owner increased
@@ -1141,12 +1102,20 @@ out:
 	 * state.
 	 */
 	if (rc == MIGRATEPAGE_SUCCESS) {
+		set_page_owner_migrate_reason(newpage, reason);
 		if (unlikely(!is_lru))
 			put_page(newpage);
 		else
 			putback_lru_page(newpage);
+	} else if (put_new_page) {
+		put_new_page(newpage, private);
+	} else {
+		put_page(newpage);
 	}
-
+out:
+	/* Drop an anon_vma reference if we took one */
+	if (anon_vma)
+		put_anon_vma(anon_vma);
 	return rc;
 }
 
@@ -1171,16 +1140,11 @@ static ICE_noinline int unmap_and_move(n
 				   int force, enum migrate_mode mode,
 				   enum migrate_reason reason)
 {
-	int rc = MIGRATEPAGE_SUCCESS;
-	struct page *newpage;
+	int rc = -EAGAIN;
 
 	if (!thp_migration_supported() && PageTransHuge(page))
 		return -ENOMEM;
 
-	newpage = get_new_page(page, private);
-	if (!newpage)
-		return -ENOMEM;
-
 	if (page_count(page) == 1) {
 		/* page was freed from under us. So we are done. */
 		ClearPageActive(page);
@@ -1191,17 +1155,57 @@ static ICE_noinline int unmap_and_move(n
 				__ClearPageIsolated(page);
 			unlock_page(page);
 		}
-		if (put_new_page)
-			put_new_page(newpage, private);
-		else
-			put_page(newpage);
+		rc = MIGRATEPAGE_SUCCESS;
 		goto out;
 	}
 
-	rc = __unmap_and_move(page, newpage, force, mode);
-	if (rc == MIGRATEPAGE_SUCCESS)
-		set_page_owner_migrate_reason(newpage, reason);
+	if (!trylock_page(page)) {
+		if (!force || mode == MIGRATE_ASYNC)
+			return rc;
+
+		/*
+		 * It's not safe for direct compaction to call lock_page.
+		 * For example, during page readahead pages are added locked
+		 * to the LRU. Later, when the IO completes the pages are
+		 * marked uptodate and unlocked. However, the queueing
+		 * could be merging multiple pages for one bio (e.g.
+		 * mpage_readpages). If an allocation happens for the
+		 * second or third page, the process can end up locking
+		 * the same page twice and deadlocking. Rather than
+		 * trying to be clever about what pages can be locked,
+		 * avoid the use of lock_page for direct compaction
+		 * altogether.
+		 */
+		if (current->flags & PF_MEMALLOC)
+			return rc;
 
+		lock_page(page);
+	}
+
+	if (PageWriteback(page)) {
+		/*
+		 * Only in the case of a full synchronous migration is it
+		 * necessary to wait for PageWriteback. In the async case,
+		 * the retry loop is too short and in the sync-light case,
+		 * the overhead of stalling is too much
+		 */
+		switch (mode) {
+		case MIGRATE_SYNC:
+		case MIGRATE_SYNC_NO_COPY:
+			break;
+		default:
+			rc = -EBUSY;
+			goto out_unlock;
+		}
+		if (!force)
+			goto out_unlock;
+		wait_on_page_writeback(page);
+	}
+	rc = __unmap_and_move(get_new_page, put_new_page, private,
+			      page, mode, reason);
+
+out_unlock:
+	unlock_page(page);
 out:
 	if (rc != -EAGAIN) {
 		/*
@@ -1242,9 +1246,8 @@ out:
 		if (rc != -EAGAIN) {
 			if (likely(!__PageMovable(page))) {
 				putback_lru_page(page);
-				goto put_new;
+				goto done;
 			}
-
 			lock_page(page);
 			if (PageMovable(page))
 				putback_movable_page(page);
@@ -1253,13 +1256,8 @@ out:
 			unlock_page(page);
 			put_page(page);
 		}
-put_new:
-		if (put_new_page)
-			put_new_page(newpage, private);
-		else
-			put_page(newpage);
 	}
-
+done:
 	return rc;
 }
 

_

#5 lore raw

Re: [PATCH 2/4] mm/migrate: Defer allocating new page until needed

Quoted 7 lines · Dave Hansen

From: Keith Busch <[email protected]>

Migrating pages had been allocating the new page before it was actually needed. Subsequent operations may still fail, which would have to handle cleaning up the newly allocated page when it was never used.

Quoted 5 lines

Defer allocating the page until we are actually ready to make use of it, after locking the original page. This simplifies error handling, but should not have any functional change in behavior. This is just refactoring page migration so the main part can more easily be reused by other code.

Well, the functional change I see is that now we allocate a new page under page lock of old page.

It *should* be fine, but it has to be call out in the commit message.

Signature · 1 line
 Kirill A. Shutemov
#6 lore raw

PATCH 3/4 mm/vmscan: Attempt to migrate page in lieu of discard

In reply to Dave Hansen

From: Keith Busch <[email protected]>

If a memory node has a preferred migration path to demote cold pages, attempt to move those inactive pages to that migration node before reclaiming. This will better utilize available memory, provide a faster tier than swapping or discarding, and allow such pages to be reused immediately without IO to retrieve the data.

Much like swap, this is an opt-in feature that requires user defining where to send pages when reclaiming them. When handling anonymous pages, this will be considered before swap if enabled. Should the demotion fail for any reason, the page reclaim will proceed as if the demotion feature was not enabled.

Some places we would like to see this used:

1. Persistent memory being as a slower, cheaper DRAM replacement
2. Remote memory-only "expansion" NUMA nodes
3. Resolving memory imbalances where one NUMA node is seeing more allocation activity than another. This helps keep more recent allocations closer to the CPUs on the node doing the allocating.

---

 b/include/linux/migrate.h        |    6 ++++
 b/include/trace/events/migrate.h |    3 +-
 b/mm/debug.c                     |    1 
 b/mm/migrate.c                   |   51 +++++++++++++++++++++++++++++++++++++++
 b/mm/vmscan.c                    |   27 ++++++++++++++++++++
 5 files changed, 87 insertions(+), 1 deletion(-)
include/linux/migrate.h renamed from include/linux/migrate.h~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard +6 -0
diff -puN include/linux/migrate.h~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard include/linux/migrate.h
@@ -25,6 +25,7 @@ enum migrate_reason {
 	MR_MEMPOLICY_MBIND,
 	MR_NUMA_MISPLACED,
 	MR_CONTIG_RANGE,
+	MR_DEMOTION,
 	MR_TYPES
 };
 
@@ -79,6 +80,7 @@ extern int migrate_huge_page_move_mappin
 extern int migrate_page_move_mapping(struct address_space *mapping,
 		struct page *newpage, struct page *page, enum migrate_mode mode,
 		int extra_count);
+extern int migrate_demote_mapping(struct page *page);
 #else
 
 static inline void putback_movable_pages(struct list_head *l) {}
@@ -105,6 +107,10 @@ static inline int migrate_huge_page_move
 	return -ENOSYS;
 }
 
+static inline int migrate_demote_mapping(struct page *page)
+{
+	return -ENOSYS;
+}
 #endif /* CONFIG_MIGRATION */
 
 #ifdef CONFIG_COMPACTION
include/trace/events/migrate.h renamed from include/trace/events/migrate.h~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard +2 -1
diff -puN include/trace/events/migrate.h~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard include/trace/events/migrate.h
@@ -20,7 +20,8 @@
 	EM( MR_SYSCALL,		"syscall_or_cpuset")		\
 	EM( MR_MEMPOLICY_MBIND,	"mempolicy_mbind")		\
 	EM( MR_NUMA_MISPLACED,	"numa_misplaced")		\
-	EMe(MR_CONTIG_RANGE,	"contig_range")
+	EM( MR_CONTIG_RANGE,	"contig_range")			\
+	EMe(MR_DEMOTION,	"demotion")
 
 /*
  * First define the enums in the above macros to be exported to userspace
mm/debug.c renamed from mm/debug.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard +1 -0
diff -puN mm/debug.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard mm/debug.c
@@ -25,6 +25,7 @@ const char *migrate_reason_names[MR_TYPE
 	"mempolicy_mbind",
 	"numa_misplaced",
 	"cma",
+	"demotion",
 };
 
 const struct trace_print_flags pageflag_names[] = {
mm/migrate.c renamed from mm/migrate.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard +51 -0

57 lines not on this page · raw message

mm/vmscan.c renamed from mm/vmscan.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard +27 -0
diff -puN mm/vmscan.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard mm/vmscan.c
@@ -1262,6 +1262,33 @@ static unsigned long shrink_page_list(st
 			; /* try to reclaim the page below */
 		}
 
+		if (!PageHuge(page)) {
+			int rc = migrate_demote_mapping(page);
+
+			/*
+			 * -ENOMEM on a THP may indicate either migration is
+			 * unsupported or there was not enough contiguous
+			 * space. Split the THP into base pages and retry the
+			 * head immediately. The tail pages will be considered
+			 * individually within the current loop's page list.
+			 */
+			if (rc == -ENOMEM && PageTransHuge(page) &&
+			    !split_huge_page_to_list(page, page_list))
+				rc = migrate_demote_mapping(page);
+
+			if (rc == MIGRATEPAGE_SUCCESS) {
+				unlock_page(page);
+				if (likely(put_page_testzero(page)))
+					goto free_it;
+				/*
+				 * Speculative reference will free this page,
+				 * so leave it off the LRU.
+				 */
+				nr_reclaimed++;
+				continue;
+			}
+		}
+
 		/*
 		 * Anonymous process memory has backing store?
 		 * Try to allocate it some swap space here.

_

#7 lore raw

Re: [PATCH 3/4] mm/vmscan: Attempt to migrate page in lieu of discard

Quoted 161 lines · Dave Hansen

From: Keith Busch <[email protected]>

If a memory node has a preferred migration path to demote cold pages, attempt to move those inactive pages to that migration node before reclaiming. This will better utilize available memory, provide a faster tier than swapping or discarding, and allow such pages to be reused immediately without IO to retrieve the data.

Much like swap, this is an opt-in feature that requires user defining where to send pages when reclaiming them. When handling anonymous pages, this will be considered before swap if enabled. Should the demotion fail for any reason, the page reclaim will proceed as if the demotion feature was not enabled.

Some places we would like to see this used:

1. Persistent memory being as a slower, cheaper DRAM replacement
2. Remote memory-only "expansion" NUMA nodes
3. Resolving memory imbalances where one NUMA node is seeing more allocation activity than another. This helps keep more recent allocations closer to the CPUs on the node doing the allocating.

---

 b/include/linux/migrate.h        |    6 ++++
 b/include/trace/events/migrate.h |    3 +-
 b/mm/debug.c                     |    1
 b/mm/migrate.c                   |   51 +++++++++++++++++++++++++++++++++++++++
 b/mm/vmscan.c                    |   27 ++++++++++++++++++++
 5 files changed, 87 insertions(+), 1 deletion(-)

diff -puN include/linux/migrate.h~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard include/linux/migrate.h

include/linux/migrate.h 2019-10-16 15:06:58.103952593 -0700 renamed from include/linux/migrate.h~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard 2019-10-16 15:06:58.090952593 -0700 +6 -0
@@ -25,6 +25,7 @@ enum migrate_reason {
        MR_MEMPOLICY_MBIND,
        MR_NUMA_MISPLACED,
        MR_CONTIG_RANGE,
+       MR_DEMOTION,
        MR_TYPES
 };
 
@@ -79,6 +80,7 @@ extern int migrate_huge_page_move_mappin
 extern int migrate_page_move_mapping(struct address_space *mapping,
                struct page *newpage, struct page *page, enum migrate_mode mode,
                int extra_count);
+extern int migrate_demote_mapping(struct page *page);
 #else
 
 static inline void putback_movable_pages(struct list_head *l) {}
@@ -105,6 +107,10 @@ static inline int migrate_huge_page_move
        return -ENOSYS;
 }
 
+static inline int migrate_demote_mapping(struct page *page)
+{
+       return -ENOSYS;
+}
 #endif /* CONFIG_MIGRATION */
 
 #ifdef CONFIG_COMPACTION

diff -puN include/trace/events/migrate.h~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard include/trace/events/migrate.h

include/trace/events/migrate.h 2019-10-16 15:06:58.103952593 -0700 renamed from include/trace/events/migrate.h~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard 2019-10-16 15:06:58.092952593 -0700 +2 -1
@@ -20,7 +20,8 @@
        EM( MR_SYSCALL,         "syscall_or_cpuset")            \
        EM( MR_MEMPOLICY_MBIND, "mempolicy_mbind")              \
        EM( MR_NUMA_MISPLACED,  "numa_misplaced")               \
-       EMe(MR_CONTIG_RANGE,    "contig_range")
+       EM( MR_CONTIG_RANGE,    "contig_range")                 \
+       EMe(MR_DEMOTION,        "demotion")
 
 /*
  * First define the enums in the above macros to be exported to userspace

diff -puN mm/debug.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard mm/debug.c

mm/debug.c 2019-10-16 15:06:58.103952593 -0700 renamed from mm/debug.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard 2019-10-16 15:06:58.094952593 -0700 +1 -0
@@ -25,6 +25,7 @@ const char *migrate_reason_names[MR_TYPE
        "mempolicy_mbind",
        "numa_misplaced",
        "cma",
+       "demotion",
 };
 
 const struct trace_print_flags pageflag_names[] = {

diff -puN mm/migrate.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard mm/migrate.c

mm/migrate.c 2019-10-16 15:06:58.104952593 -0700 renamed from mm/migrate.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard 2019-10-16 15:06:58.097952593 -0700 +51 -0
@@ -1119,6 +1119,57 @@ out:
        return rc;
 }
 
+static struct page *alloc_demote_node_page(struct page *page, unsigned long node)
+{
+       /*
+        * The flags are set to allocate only on the desired node in the
+        * migration path, and to fail fast if not immediately available. We
+        * are already doing memory reclaim, we don't want heroic efforts to
+        * get a page.
+        */
+       gfp_t mask = GFP_NOWAIT | __GFP_NOWARN | __GFP_NORETRY |
+                       __GFP_NOMEMALLOC | __GFP_THISNODE | __GFP_MOVABLE;
+       struct page *newpage;
+
+       if (PageTransHuge(page)) {
+               mask |= __GFP_COMP;
+               newpage = alloc_pages_node(node, mask, HPAGE_PMD_ORDER);
+               if (newpage)
+                       prep_transhuge_page(newpage);
+       } else
+               newpage = alloc_pages_node(node, mask, 0);
+
+       return newpage;
+}
+
+/**
+ * migrate_demote_mapping() - Migrate this page and its mappings to its
+ *                            demotion node.
+ * @page: A locked, isolated, non-huge page that should migrate to its current
+ *        node's demotion target, if available. Since this is intended to be
+ *        called during memory reclaim, all flag options are set to fail fast.
+ *
+ * @returns: MIGRATEPAGE_SUCCESS if successful, -errno otherwise.
+ */
+int migrate_demote_mapping(struct page *page)
+{
+       int next_nid = next_migration_node(page_to_nid(page));
+
+       VM_BUG_ON_PAGE(!PageLocked(page), page);
+       VM_BUG_ON_PAGE(PageHuge(page), page);
+       VM_BUG_ON_PAGE(PageLRU(page), page);
+
+       if (next_nid < 0)
+               return -ENOSYS;
+       if (PageTransHuge(page) && !thp_migration_supported())
+               return -ENOMEM;
+
+       /* MIGRATE_ASYNC is the most light weight and never blocks.*/
+       return __unmap_and_move(alloc_demote_node_page, NULL, next_nid,
+                               page, MIGRATE_ASYNC, MR_DEMOTION);
+}
+
+
 /*
  * gcc 4.7 and 4.8 on arm get an ICEs when inlining unmap_and_move().  Work
  * around it.

diff -puN mm/vmscan.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard mm/vmscan.c

mm/vmscan.c 2019-10-16 15:06:58.105952593 -0700 renamed from mm/vmscan.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard 2019-10-16 15:06:58.099952593 -0700 +3 -0
@@ -1262,6 +1262,33 @@ static unsigned long shrink_page_list(st
                        ; /* try to reclaim the page below */
                }
 
+               if (!PageHuge(page)) {
+                       int rc = migrate_demote_mapping(page);
+

mm/vmscan.c 2019-10-16 15:06:58.105952593 -0700 static unsigned long shrink_page_list(st · line 1277

Quoted 10 lines
mm/vmscan.c 2019-10-16 15:06:58.105952593 -0700 renamed from mm/vmscan.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard 2019-10-16 15:06:58.099952593 -0700 +10 -0
+                       /*
+                        * -ENOMEM on a THP may indicate either migration is
+                        * unsupported or there was not enough contiguous
+                        * space. Split the THP into base pages and retry the
+                        * head immediately. The tail pages will be considered
+                        * individually within the current loop's page list.
+                        */
+                       if (rc == -ENOMEM && PageTransHuge(page) &&
+                           !split_huge_page_to_list(page, page_list))
+                               rc = migrate_demote_mapping(page);

I recalled when Keith posted the patch at the first time, I raised question about why not just migrating THP in a whole? The migrate_pages() could handle this. If it fails, it just fallbacks to base page.

Since the most optimistic gfp flags are used, it should not trap into nested direct reclaim. The migrate_pages() should just return failure then fallback to base page.

Quoted 19 lines
mm/vmscan.c 2019-10-16 15:06:58.105952593 -0700 renamed from mm/vmscan.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard 2019-10-16 15:06:58.099952593 -0700 +14 -0
+
+                       if (rc == MIGRATEPAGE_SUCCESS) {
+                               unlock_page(page);
+                               if (likely(put_page_testzero(page)))
+                                       goto free_it;
+                               /*
+                                * Speculative reference will free this page,
+                                * so leave it off the LRU.
+                                */
+                               nr_reclaimed++;
+                               continue;
+                       }
+               }
+
                /*
                 * Anonymous process memory has backing store?
                 * Try to allocate it some swap space here.

_

#8 lore raw

Re: [PATCH 3/4] mm/vmscan: Attempt to migrate page in lieu of discard

Quoted 13 lines · Yang Shi

mm/vmscan.c static unsigned long shrink_page_list(st · line 1277

Quoted 13 lines
mm/vmscan.c renamed from mm/vmscan.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard +13 -0
+               if (!PageHuge(page)) {
+                       int rc = migrate_demote_mapping(page);
+
+                       /*
+                        * -ENOMEM on a THP may indicate either migration is
+                        * unsupported or there was not enough contiguous
+                        * space. Split the THP into base pages and retry the
+                        * head immediately. The tail pages will be considered
+                        * individually within the current loop's page list.
+                        */
+                       if (rc == -ENOMEM && PageTransHuge(page) &&
+                           !split_huge_page_to_list(page, page_list))
+                               rc = migrate_demote_mapping(page);
Quoted 4 lines

I recalled when Keith posted the patch at the first time, I raised question about why not just migrating THP in a whole? The migrate_pages() could handle this. If it fails, it just fallbacks to base page.

There's a pair of migrate_demote_mapping()s in there. I expected that the first will migrate the whole THP and the second plus the split is only used if fails the whole migration.

Am I reading it wrong?

#9 lore raw

Re: [PATCH 3/4] mm/vmscan: Attempt to migrate page in lieu of discard

Quoted 15 lines · Dave Hansen
Quoted 13 lines · Yang Shi
Quoted 13 lines
mm/vmscan.c renamed from mm/vmscan.c~0005-mm-vmscan-Attempt-to-migrate-page-in-lieu-of-discard +13 -0
+               if (!PageHuge(page)) {
+                       int rc = migrate_demote_mapping(page);
+
+                       /*
+                        * -ENOMEM on a THP may indicate either migration is
+                        * unsupported or there was not enough contiguous
+                        * space. Split the THP into base pages and retry the
+                        * head immediately. The tail pages will be considered
+                        * individually within the current loop's page list.
+                        */
+                       if (rc == -ENOMEM && PageTransHuge(page) &&
+                           !split_huge_page_to_list(page, page_list))
+                               rc = migrate_demote_mapping(page);
Quoted 8 lines
Quoted 4 lines

I recalled when Keith posted the patch at the first time, I raised question about why not just migrating THP in a whole? The migrate_pages() could handle this. If it fails, it just fallbacks to base page.

There's a pair of migrate_demote_mapping()s in there. I expected that the first will migrate the whole THP and the second plus the split is only used if fails the whole migration.

Ah, that's right. I mis-read the first migrate_demote_mapping(). But, why calling migrate_demote_mapping() twice for THP and base page (if THP is failed) instead of calling migrate_pages() that does deal with all the cases.

Quoted 2 lines

Am I reading it wrong?

#10 lore raw

PATCH 4/4 mm/vmscan: Consider anonymous pages without swap

In reply to Dave Hansen

From: Keith Busch <[email protected]>

Age and reclaim anonymous pages if a migration path is available. The node has other recourses for inactive anonymous pages beyond swap,

---

 b/include/linux/swap.h |   20 ++++++++++++++++++++
 b/mm/vmscan.c          |   10 +++++-----
 2 files changed, 25 insertions(+), 5 deletions(-)
include/linux/swap.h renamed from include/linux/swap.h~0006-mm-vmscan-Consider-anonymous-pages-without-swap +20 -0
diff -puN include/linux/swap.h~0006-mm-vmscan-Consider-anonymous-pages-without-swap include/linux/swap.h
@@ -680,5 +680,25 @@ static inline bool mem_cgroup_swap_full(
 }
 #endif
 
+static inline bool reclaim_anon_pages(struct mem_cgroup *memcg,
+				      int node_id)
+{
+	/* Always age anon pages when we have swap */
+	if (memcg == NULL) {
+		if (get_nr_swap_pages() > 0)
+			return true;
+	} else {
+		if (mem_cgroup_get_nr_swap_pages(memcg) > 0)
+			return true;
+	}
+
+	/* Also age anon pages if we can auto-migrate them */
+	if (next_migration_node(node_id) >= 0)
+		return true;
+
+	/* No way to reclaim anon pages */
+	return false;
+}
+
 #endif /* __KERNEL__*/
 #endif /* _LINUX_SWAP_H */
mm/vmscan.c renamed from mm/vmscan.c~0006-mm-vmscan-Consider-anonymous-pages-without-swap +5 -5
diff -puN mm/vmscan.c~0006-mm-vmscan-Consider-anonymous-pages-without-swap mm/vmscan.c
@@ -327,7 +327,7 @@ unsigned long zone_reclaimable_pages(str
 
 	nr = zone_page_state_snapshot(zone, NR_ZONE_INACTIVE_FILE) +
 		zone_page_state_snapshot(zone, NR_ZONE_ACTIVE_FILE);
-	if (get_nr_swap_pages() > 0)
+	if (reclaim_anon_pages(NULL, zone_to_nid(zone)))
 		nr += zone_page_state_snapshot(zone, NR_ZONE_INACTIVE_ANON) +
 			zone_page_state_snapshot(zone, NR_ZONE_ACTIVE_ANON);
 
@@ -2166,7 +2166,7 @@ static bool inactive_list_is_low(struct
 	 * If we don't have swap space, anonymous page deactivation
 	 * is pointless.
 	 */
-	if (!file && !total_swap_pages)
+	if (!file && !reclaim_anon_pages(NULL, pgdat->node_id))
 		return false;
 
 	inactive = lruvec_lru_size(lruvec, inactive_lru, sc->reclaim_idx);
@@ -2241,7 +2241,7 @@ static void get_scan_count(struct lruvec
 	enum lru_list lru;
 
 	/* If we have no swap space, do not bother scanning anon pages. */
-	if (!sc->may_swap || mem_cgroup_get_nr_swap_pages(memcg) <= 0) {
+	if (!sc->may_swap || !reclaim_anon_pages(memcg, pgdat->node_id)) {
 		scan_balance = SCAN_FILE;
 		goto out;
 	}
@@ -2604,7 +2604,7 @@ static inline bool should_continue_recla
 	 */
 	pages_for_compaction = compact_gap(sc->order);
 	inactive_lru_pages = node_page_state(pgdat, NR_INACTIVE_FILE);
-	if (get_nr_swap_pages() > 0)
+	if (!reclaim_anon_pages(NULL, pgdat->node_id))
 		inactive_lru_pages += node_page_state(pgdat, NR_INACTIVE_ANON);
 	if (sc->nr_reclaimed < pages_for_compaction &&
 			inactive_lru_pages > pages_for_compaction)
@@ -3289,7 +3289,7 @@ static void age_active_anon(struct pglis
 {
 	struct mem_cgroup *memcg;
 
-	if (!total_swap_pages)
+	if (!reclaim_anon_pages(NULL, pgdat->node_id))
 		return;
 
 	memcg = mem_cgroup_iter(NULL, NULL, NULL);

_

#11 lore raw
In reply to Dave Hansen
Quoted 30 lines · Dave Hansen

We're starting to see systems with more and more kinds of memory such as Intel's implementation of persistent memory.

Let's say you have a system with some DRAM and some persistent memory. Today, once DRAM fills up, reclaim will start and some of the DRAM contents will be thrown out. Allocations will, at some point, start falling over to the slower persistent memory.

That has two nasty properties. First, the newer allocations can end up in the slower persistent memory. Second, reclaimed data in DRAM are just discarded even if there are gobs of space in persistent memory that could be used.

This set implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.

While I've talked about a DRAM/PMEM pairing, this approach would function in any environment where memory tiers exist.

This is not perfect. It "strands" pages in slower memory and never brings them back to fast DRAM. Other things need to be built to promote hot pages back to DRAM.

This is part of a larger patch set. If you want to apply these or play with them, I'd suggest using the tree from here. It includes autonuma-based hot page promotion back to DRAM:

Quoted 6 lines
        http://lkml.kernel.org/r/[email protected]

This is also all based on an upstream mechanism that allows persistent memory to be onlined and used as if it were volatile:

        http://lkml.kernel.org/r/[email protected]

The memory cgroup part of the story is missing here. Since PMEM is treated as slow DRAM, shouldn't its usage be accounted to the corresponding memcg's memory/memsw counters and the migration should not happen for memcg limit reclaim? Otherwise some jobs can hog the whole PMEM.

Also what happens when PMEM is full? Can the memory migrated to PMEM be reclaimed (or discarded)?

Shakeel

#12 lore raw
Quoted 5 lines · Shakeel Butt
Quoted 4 lines · Dave Hansen

This set implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.

..> The memory cgroup part of the story is missing here. Since PMEM is

Quoted 4 lines

treated as slow DRAM, shouldn't its usage be accounted to the corresponding memcg's memory/memsw counters and the migration should not happen for memcg limit reclaim? Otherwise some jobs can hog the whole PMEM.

My expectation (and I haven't confirmed this) is that the any memory use is accounted to the owning cgroup, whether it is DRAM or PMEM. memcg limit reclaim and global reclaim both end up doing migrations and neither should have a net effect on the counters.

There is certainly a problem here because DRAM is a more valuable resource vs. PMEM, and memcg accounts for them as if they were equally valuable. I really want to see memcg account for this cost discrepancy at some point, but I'm not quite sure what form it would take. Any feedback from you heavy memcg users out there would be much appreciated.

Quoted 2 lines

Also what happens when PMEM is full? Can the memory migrated to PMEM be reclaimed (or discarded)?

Yep. The "migration path" can be as long as you want, but once the data hits a "terminal node" it will stop getting migrated and normal discard at the end of reclaim happens.

#13 lore raw
Quoted 13 lines · Dave Hansen
Quoted 5 lines · Shakeel Butt
Quoted 4 lines · Dave Hansen

This set implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.

..> The memory cgroup part of the story is missing here. Since PMEM is

Quoted 4 lines

treated as slow DRAM, shouldn't its usage be accounted to the corresponding memcg's memory/memsw counters and the migration should not happen for memcg limit reclaim? Otherwise some jobs can hog the whole PMEM.

Quoted 5 lines

My expectation (and I haven't confirmed this) is that the any memory use is accounted to the owning cgroup, whether it is DRAM or PMEM. memcg limit reclaim and global reclaim both end up doing migrations and neither should have a net effect on the counters.

Hmm I didn't see the memcg charge migration in the code on demotion. So, in the code [patch 3] the counters are being decremented as DRAM is freed but not incremented for PMEM.

Quoted 6 lines

There is certainly a problem here because DRAM is a more valuable resource vs. PMEM, and memcg accounts for them as if they were equally valuable. I really want to see memcg account for this cost discrepancy at some point, but I'm not quite sure what form it would take. Any feedback from you heavy memcg users out there would be much appreciated.

There are two apparent use-cases i.e. explicit (apps moving their pages to PMEM to reduce cost) and implicit (admin moves cold pages to PMEM transparently to the apps) for the PMEM. In the implicit case, I see both DRAM and PMEM as same resource from the perspective of memcg limits i.e. same memcg counter, something like cgroup v1's memsw). For the explicit case, maybe separate counters make sense like cgroup v2's memory and swap.

Quoted 6 lines
Quoted 2 lines

Also what happens when PMEM is full? Can the memory migrated to PMEM be reclaimed (or discarded)?

Yep. The "migration path" can be as long as you want, but once the data hits a "terminal node" it will stop getting migrated and normal discard at the end of reclaim happens.

I might have missed it but I didn't see the migrated pages inserted back to LRUs. If they are not in LRU, the reclaimer will never see them.

#14 lore raw
Quoted 8 lines · Shakeel Butt
Quoted 5 lines

My expectation (and I haven't confirmed this) is that the any memory use is accounted to the owning cgroup, whether it is DRAM or PMEM. memcg limit reclaim and global reclaim both end up doing migrations and neither should have a net effect on the counters.

Hmm I didn't see the memcg charge migration in the code on demotion. So, in the code [patch 3] the counters are being decremented as DRAM is freed but not incremented for PMEM.

I had assumed that the migration code was doing this for me. I'll go make sure either way.

#15 lore raw
In reply to Dave Hansen
Quoted 13 lines · Dave Hansen
Quoted 5 lines · Shakeel Butt
Quoted 4 lines · Dave Hansen

This set implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.

..> The memory cgroup part of the story is missing here. Since PMEM is

Quoted 4 lines

treated as slow DRAM, shouldn't its usage be accounted to the corresponding memcg's memory/memsw counters and the migration should not happen for memcg limit reclaim? Otherwise some jobs can hog the whole PMEM.

Quoted 4 lines

My expectation (and I haven't confirmed this) is that the any memory use is accounted to the owning cgroup, whether it is DRAM or PMEM. memcg limit reclaim and global reclaim both end up doing migrations and neither should have a net effect on the counters.

Yes, your expectation is correct. As long as PMEM is a NUMA node, it is treated as regular memory by memcg. But, I don't think memcg limit reclaim should do migration since limit reclaim is used to reduce memory usage, but migration doesn't reduce usage, it just moves memory from one node to the other.

In my implementation, I just skip migration for memcg limit reclaim,
please see: https://lore.kernel.org/linux-mm/[email protected]/

Quoted 6 lines

There is certainly a problem here because DRAM is a more valuable resource vs. PMEM, and memcg accounts for them as if they were equally valuable. I really want to see memcg account for this cost discrepancy at some point, but I'm not quite sure what form it would take. Any feedback from you heavy memcg users out there would be much appreciated.

We did have some demands to control the ratio between DRAM and PMEM as I mentioned in LSF/MM. Mel Gorman did suggest make memcg account DRAM and PMEM respectively or something similar.

Quoted 7 lines
Quoted 3 lines

Also what happens when PMEM is full? Can the memory migrated to PMEM be reclaimed (or discarded)?

Yep. The "migration path" can be as long as you want, but once the data hits a "terminal node" it will stop getting migrated and normal discard at the end of reclaim happens.

I recalled I had a hallway conversation with Keith about this in LSF/MM. We all agree there should be not a cycle. But, IMHO, I don't think exporting migration path to userspace (or letting user to define migration path) and having multiple migration stops are good ideas in general.

Quoted 1 line
#16 lore raw
Quoted 6 lines · Yang Shi
Quoted 4 lines · Dave Hansen

My expectation (and I haven't confirmed this) is that the any memory use is accounted to the owning cgroup, whether it is DRAM or PMEM. memcg limit reclaim and global reclaim both end up doing migrations and neither should have a net effect on the counters.

Quoted 8 lines

Yes, your expectation is correct. As long as PMEM is a NUMA node, it is treated as regular memory by memcg. But, I don't think memcg limit reclaim should do migration since limit reclaim is used to reduce memory usage, but migration doesn't reduce usage, it just moves memory from one node to the other.

In my implementation, I just skip migration for memcg limit reclaim,
please see: https://lore.kernel.org/linux-mm/[email protected]/

Ahh, got it. That does make sense. I might have to steal your implementation.

#17 lore raw
In reply to Yang Shi
Quoted 36 lines · Yang Shi
Quoted 13 lines · Dave Hansen
Quoted 5 lines · Shakeel Butt
Quoted 4 lines · Dave Hansen

This set implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.

..> The memory cgroup part of the story is missing here. Since PMEM is

Quoted 4 lines

treated as slow DRAM, shouldn't its usage be accounted to the corresponding memcg's memory/memsw counters and the migration should not happen for memcg limit reclaim? Otherwise some jobs can hog the whole PMEM.

Quoted 4 lines

My expectation (and I haven't confirmed this) is that the any memory use is accounted to the owning cgroup, whether it is DRAM or PMEM. memcg limit reclaim and global reclaim both end up doing migrations and neither should have a net effect on the counters.

Yes, your expectation is correct. As long as PMEM is a NUMA node, it is treated as regular memory by memcg. But, I don't think memcg limit reclaim should do migration since limit reclaim is used to reduce memory usage, but migration doesn't reduce usage, it just moves memory from one node to the other.

In my implementation, I just skip migration for memcg limit reclaim,
please see: https://lore.kernel.org/linux-mm/[email protected]/

Quoted 6 lines

There is certainly a problem here because DRAM is a more valuable resource vs. PMEM, and memcg accounts for them as if they were equally valuable. I really want to see memcg account for this cost discrepancy at some point, but I'm not quite sure what form it would take. Any feedback from you heavy memcg users out there would be much appreciated.

Quoted 4 lines

We did have some demands to control the ratio between DRAM and PMEM as I mentioned in LSF/MM. Mel Gorman did suggest make memcg account DRAM and PMEM respectively or something similar.

Can you please describe how you plan to use this ratio? Are applications supposed to use this ratio or the admins will be adjusting this ratio? Also should it dynamically updated based on the workload i.e. as the working set or hot pages grows we want more DRAM and as cold pages grows we want more PMEM? Basically I am trying to see if we have something like smart auto-numa balancing to fulfill your use-case.

Quoted 15 lines
Quoted 7 lines
Quoted 3 lines

Also what happens when PMEM is full? Can the memory migrated to PMEM be reclaimed (or discarded)?

Yep. The "migration path" can be as long as you want, but once the data hits a "terminal node" it will stop getting migrated and normal discard at the end of reclaim happens.

I recalled I had a hallway conversation with Keith about this in LSF/MM. We all agree there should be not a cycle. But, IMHO, I don't think exporting migration path to userspace (or letting user to define migration path) and having multiple migration stops are good ideas in general.

Quoted 1 line
#18 lore raw
Quoted 43 lines · Shakeel Butt
Quoted 36 lines · Yang Shi
Quoted 13 lines · Dave Hansen
Quoted 5 lines · Shakeel Butt
Quoted 4 lines · Dave Hansen

This set implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.

..> The memory cgroup part of the story is missing here. Since PMEM is

Quoted 4 lines

treated as slow DRAM, shouldn't its usage be accounted to the corresponding memcg's memory/memsw counters and the migration should not happen for memcg limit reclaim? Otherwise some jobs can hog the whole PMEM.

Quoted 4 lines

My expectation (and I haven't confirmed this) is that the any memory use is accounted to the owning cgroup, whether it is DRAM or PMEM. memcg limit reclaim and global reclaim both end up doing migrations and neither should have a net effect on the counters.

Yes, your expectation is correct. As long as PMEM is a NUMA node, it is treated as regular memory by memcg. But, I don't think memcg limit reclaim should do migration since limit reclaim is used to reduce memory usage, but migration doesn't reduce usage, it just moves memory from one node to the other.

In my implementation, I just skip migration for memcg limit reclaim,
please see: https://lore.kernel.org/linux-mm/[email protected]/

Quoted 6 lines

There is certainly a problem here because DRAM is a more valuable resource vs. PMEM, and memcg accounts for them as if they were equally valuable. I really want to see memcg account for this cost discrepancy at some point, but I'm not quite sure what form it would take. Any feedback from you heavy memcg users out there would be much appreciated.

Quoted 4 lines

We did have some demands to control the ratio between DRAM and PMEM as I mentioned in LSF/MM. Mel Gorman did suggest make memcg account DRAM and PMEM respectively or something similar.

Quoted 7 lines

Can you please describe how you plan to use this ratio? Are applications supposed to use this ratio or the admins will be adjusting this ratio? Also should it dynamically updated based on the workload i.e. as the working set or hot pages grows we want more DRAM and as cold pages grows we want more PMEM? Basically I am trying to see if we have something like smart auto-numa balancing to fulfill your use-case.

We thought it should be controlled by admins and transparent to the end users. The ratio is fixed, but the memory could be moved between DRAM and PMEM dynamically as long as it doesn't exceed the ratio so that we could keep warmer data in DRAM and colder data in PMEM.

I talked this about in LSF/MM, please check this out:
https://lwn.net/Articles/787418/

Quoted 16 lines
Quoted 16 lines
Quoted 8 lines
Quoted 4 lines

Also what happens when PMEM is full? Can the memory migrated to PMEM be reclaimed (or discarded)?

Yep. The "migration path" can be as long as you want, but once the data hits a "terminal node" it will stop getting migrated and normal discard at the end of reclaim happens.

I recalled I had a hallway conversation with Keith about this in LSF/MM. We all agree there should be not a cycle. But, IMHO, I don't think exporting migration path to userspace (or letting user to define migration path) and having multiple migration stops are good ideas in general.

Quoted 1 line
#19 lore raw
In reply to Dave Hansen
Quoted 30 lines · Dave Hansen

We're starting to see systems with more and more kinds of memory such as Intel's implementation of persistent memory.

Let's say you have a system with some DRAM and some persistent memory. Today, once DRAM fills up, reclaim will start and some of the DRAM contents will be thrown out. Allocations will, at some point, start falling over to the slower persistent memory.

That has two nasty properties. First, the newer allocations can end up in the slower persistent memory. Second, reclaimed data in DRAM are just discarded even if there are gobs of space in persistent memory that could be used.

This set implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.

While I've talked about a DRAM/PMEM pairing, this approach would function in any environment where memory tiers exist.

This is not perfect. It "strands" pages in slower memory and never brings them back to fast DRAM. Other things need to be built to promote hot pages back to DRAM.

This is part of a larger patch set. If you want to apply these or play with them, I'd suggest using the tree from here. It includes autonuma-based hot page promotion back to DRAM:

Quoted 7 lines
        http://lkml.kernel.org/r/[email protected]

This is also all based on an upstream mechanism that allows persistent memory to be onlined and used as if it were volatile:

        http://lkml.kernel.org/r/[email protected]

We prototyped something very similar to this patch series in the past.

One problem that came up is that if you get into direct reclaim, because persistent memory can have pretty low write throughput, you can end up stalling users for a pretty long time while migrating pages.

To mitigate that, we tried changing background reclaim to start migrating much earlier (but not otherwise reclaiming), however it drastically increased the code complexity and still had the chance of not being able to catch up with pressure.

Because of that, we moved to a solution based on the proactive reclaim of idle pages, that was presented at LSFMM earlier this year:
https://lwn.net/Articles/787611/ .

-- Suleiman

#20 lore raw
Quoted 4 lines · Suleiman Souhlal

One problem that came up is that if you get into direct reclaim, because persistent memory can have pretty low write throughput, you can end up stalling users for a pretty long time while migrating pages.

Basically, you're saying that memory load spikes turn into latency spikes?

FWIW, we have been benchmarking this sucker with benchmarks that claim to care about latency. In general, compared to DRAM, we do see worse latency, but nothing catastrophic yet. I'd be interested if you have any workloads that act as reasonable proxies for your latency requirements.

Quoted 3 lines

Because of that, we moved to a solution based on the proactive reclaim of idle pages, that was presented at LSFMM earlier this year:
https://lwn.net/Articles/787611/ .

I saw the presentation. The feedback in the room as I remember it was that proactive reclaim essentially replaced the existing reclaim mechanism, to which the audience was not receptive. Have folks opinions changed on that, or are you looking for other solutions?

#21 lore raw
Quoted 14 lines · Dave Hansen
Quoted 4 lines · Suleiman Souhlal

One problem that came up is that if you get into direct reclaim, because persistent memory can have pretty low write throughput, you can end up stalling users for a pretty long time while migrating pages.

Basically, you're saying that memory load spikes turn into latency spikes?

FWIW, we have been benchmarking this sucker with benchmarks that claim to care about latency. In general, compared to DRAM, we do see worse latency, but nothing catastrophic yet. I'd be interested if you have any workloads that act as reasonable proxies for your latency requirements.

Quoted 9 lines
Quoted 3 lines

Because of that, we moved to a solution based on the proactive reclaim of idle pages, that was presented at LSFMM earlier this year:
https://lwn.net/Articles/787611/ .

I saw the presentation. The feedback in the room as I remember it was that proactive reclaim essentially replaced the existing reclaim mechanism, to which the audience was not receptive. Have folks opinions changed on that, or are you looking for other solutions?

I am currently working on a solution which shares the mechanisms between regular and proactive reclaim. The interested users/admins can setup proactive reclaim otherwise the regular reclaim will work on low memory. I will have something in one/two months and will post the patches.

Shakeel

#22 lore raw
In reply to Dave Hansen
Quoted 8 lines · Dave Hansen
Quoted 4 lines · Suleiman Souhlal

One problem that came up is that if you get into direct reclaim, because persistent memory can have pretty low write throughput, you can end up stalling users for a pretty long time while migrating pages.

Basically, you're saying that memory load spikes turn into latency spikes?

Yes, exactly.

Quoted 4 lines

FWIW, we have been benchmarking this sucker with benchmarks that claim to care about latency. In general, compared to DRAM, we do see worse latency, but nothing catastrophic yet. I'd be interested if you have any workloads that act as reasonable proxies for your latency requirements.

Sorry, I don't know of any specific workloads I can share. :-( Maybe Jonathan or Shakeel have something more.

I realize it's not very useful without giving specific examples, but even disregarding persistent memory, we've had latency issues with direct reclaim when using zswap. It's been such a problem that we're conducting experiments with not doing zswap compression in direct reclaim (but still doing it proactively).
The low write throughput of persistent memory would make this worse.

I think the case where we're most likely to run into this is when the machine is close to OOM situation and we end up thrashing rather than OOM killing.

Somewhat related, I noticed that this patch series ratelimits migrations from persistent memory to DRAM, but it might also make sense to ratelimit migrations from DRAM to persistent memory. If all the write bandwidth is taken by migrations, there might not be any more available for applications accessing pages in persistent memory, resulting in higher latency.

Another issue we ran into, that I think might also apply to this patch series, is that because kernel memory can't be allocated on persistent memory, it's possible for all of DRAM to get filled by user memory and have kernel allocations fail even though there is still a lot of free persistent memory. This is easy to trigger, just start an application that is bigger than DRAM.
To mitigate that, we introduced a new watermark for DRAM zones above which user memory can't be allocated, to leave some space for kernel allocations.

-- Suleiman

#23 lore raw
Quoted 6 lines · Suleiman Souhlal

Another issue we ran into, that I think might also apply to this patch series, is that because kernel memory can't be allocated on persistent memory, it's possible for all of DRAM to get filled by user memory and have kernel allocations fail even though there is still a lot of free persistent memory. This is easy to trigger, just start an application that is bigger than DRAM.

Why doesn't this happen on everyone's laptops where DRAM is contended between userspace and kernel allocations? Does the OOM killer trigger fast enough to save us?

Quoted 3 lines

To mitigate that, we introduced a new watermark for DRAM zones above which user memory can't be allocated, to leave some space for kernel allocations.

I'd be curious why the existing users of ZONE_MOVABLE don't have to do this? Are there just no users of ZONE_MOVABLE?

#24 lore raw
Quoted 9 lines · Dave Hansen
Quoted 6 lines · Suleiman Souhlal

Another issue we ran into, that I think might also apply to this patch series, is that because kernel memory can't be allocated on persistent memory, it's possible for all of DRAM to get filled by user memory and have kernel allocations fail even though there is still a lot of free persistent memory. This is easy to trigger, just start an application that is bigger than DRAM.

Quoted 3 lines

Why doesn't this happen on everyone's laptops where DRAM is contended between userspace and kernel allocations? Does the OOM killer trigger fast enough to save us?

Well in this case, there is plenty of free persistent memory on the machine, but not any free DRAM to allocate kernel memory. In the situation I'm describing, we end up OOMing when we, in my opinion, shouldn't.

Quoted 6 lines
Quoted 3 lines

To mitigate that, we introduced a new watermark for DRAM zones above which user memory can't be allocated, to leave some space for kernel allocations.

I'd be curious why the existing users of ZONE_MOVABLE don't have to do this? Are there just no users of ZONE_MOVABLE?

That's an excellent question for which I don't currently have an answer.

I haven't had the chance to test your patch series, and it's possible that it doesn't suffer from the issue.

-- Suleiman

reviewer
#25 lore raw
In reply to Dave Hansen
Quoted 29 lines · Dave Hansen

We're starting to see systems with more and more kinds of memory such as Intel's implementation of persistent memory.

Let's say you have a system with some DRAM and some persistent memory. Today, once DRAM fills up, reclaim will start and some of the DRAM contents will be thrown out. Allocations will, at some point, start falling over to the slower persistent memory.

That has two nasty properties. First, the newer allocations can end up in the slower persistent memory. Second, reclaimed data in DRAM are just discarded even if there are gobs of space in persistent memory that could be used.

This set implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.

While I've talked about a DRAM/PMEM pairing, this approach would function in any environment where memory tiers exist.

This is not perfect. It "strands" pages in slower memory and never brings them back to fast DRAM. Other things need to be built to promote hot pages back to DRAM.

This is part of a larger patch set. If you want to apply these or play with them, I'd suggest using the tree from here. It includes autonuma-based hot page promotion back to DRAM:

Quoted 6 lines
	http://lkml.kernel.org/r/[email protected]

This is also all based on an upstream mechanism that allows persistent memory to be onlined and used as if it were volatile:

	http://lkml.kernel.org/r/[email protected]

How does this compare to
http://lkml.kernel.org/r/[email protected]?

Signature · 2 lines
Michal Hocko
SUSE Labs
#26 lore raw
Quoted 2 lines · Michal Hocko

How does this compare to
http://lkml.kernel.org/r/[email protected]

It's a _bit_ more tied to persistent memory and it appears a bit more tied to two tiers rather something arbitrarily deep. They're pretty similar conceptually although there are quite a few differences.

For instance, what I posted has a static mapping for the migration path.
If node A is in reclaim, we always try to allocate pages on node B.
There are no restrictions on what those nodes can be. In Yang Shi's apporach, there's a dynamic search for a target migration node on each migration that follows the normal alloc fallback path. This ends up making migration nodes special.

There are also some different choices that are pretty arbitrary. For instance, when you allocation a migration target page, should you cause memory pressure on the target?

To be honest, though, I don't see anything fatally flawed with it. It's probably a useful exercise to factor out the common bits from the two sets and see what we can agree on being absolutely necessary.

#27 lore raw
Quoted 8 lines · Dave Hansen
Quoted 2 lines · Michal Hocko

How does this compare to
http://lkml.kernel.org/r/[email protected]

It's a _bit_ more tied to persistent memory and it appears a bit more tied to two tiers rather something arbitrarily deep. They're pretty similar conceptually although there are quite a few differences.

My patches do assume two tiers for now but it is not hard to extend to multiple tiers. Since it is a RFC so I didn't make it that complicated.

However, IMHO I really don't think supporting multiple tiers by making the migration path configurable to admins or users is a good choice. Memory migration caused by compaction or reclaim (not via syscall) should be transparent to the users, it is the kernel internal activity. It shouldn't be exposed to the end users.

I prefer firmware or OS build the migration path personally.

Quoted 7 lines

For instance, what I posted has a static mapping for the migration path.
If node A is in reclaim, we always try to allocate pages on node B.
There are no restrictions on what those nodes can be. In Yang Shi's apporach, there's a dynamic search for a target migration node on each migration that follows the normal alloc fallback path. This ends up making migration nodes special.

The reason that I didn't pursue static mapping is that the node might be offlined or onlined, so you have to keep the mapping right every time the node state is changed. Dynamic search just returns the closest migration target node no matter what the topology is. It should be not time consuming.

Actually, my patches don't restrict the migration target node has to be PMEM, it could be any memory lower than DRAM, but it just happens PMEM is the only available media. My patch's commit log explains this point. Again I really prefer the firmware or HMAT or ACPI driver could build the migration path in kernel.

In addition, DRAM node is definitely excluded from migration target since I don't think doing such migration between DRAM nodes is a good idea in general.

Quoted 4 lines

There are also some different choices that are pretty arbitrary. For instance, when you allocation a migration target page, should you cause memory pressure on the target?

Yes, those are definitely arbitrary. We do need sort of a lot of details in the future by figuring out how real life workload work.

Quoted 4 lines

To be honest, though, I don't see anything fatally flawed with it. It's probably a useful exercise to factor out the common bits from the two sets and see what we can agree on being absolutely necessary.

Sure, that definitely would help us move forward.

Quoted 1 line
#28 lore raw
Quoted 11 lines · Yang Shi
Quoted 8 lines · Dave Hansen
Quoted 2 lines · Michal Hocko

How does this compare to
http://lkml.kernel.org/r/[email protected]

It's a _bit_ more tied to persistent memory and it appears a bit more tied to two tiers rather something arbitrarily deep. They're pretty similar conceptually although there are quite a few differences.

Quoted 6 lines

My patches do assume two tiers for now but it is not hard to extend to multiple tiers. Since it is a RFC so I didn't make it that complicated.

However, IMHO I really don't think supporting multiple tiers by making the migration path configurable to admins or users is a good choice.

It's an optional override not a user requirement.

Quoted 5 lines

Memory migration caused by compaction or reclaim (not via syscall) should be transparent to the users, it is the kernel internal activity. It shouldn't be exposed to the end users.

I prefer firmware or OS build the migration path personally.

The OS can't, it can only trust platform firmware to tell it the memory properties.

The BIOS likely gets the tables right most of the time, and the OS can assume they are correct, but when things inevitably go wrong a user override is needed. That override is more usable as an explicit migration path rather than requiring users to manually craft and inject custom ACPI tables. I otherwise do not see the substance behind this objection to a migration path override.

reviewer
#29 lore raw
In reply to Dave Hansen
Quoted 8 lines · Dave Hansen
Quoted 2 lines · Michal Hocko

How does this compare to
http://lkml.kernel.org/r/[email protected]

It's a _bit_ more tied to persistent memory and it appears a bit more tied to two tiers rather something arbitrarily deep. They're pretty similar conceptually although there are quite a few differences.

Quoted 6 lines

For instance, what I posted has a static mapping for the migration path.
If node A is in reclaim, we always try to allocate pages on node B.
There are no restrictions on what those nodes can be. In Yang Shi's apporach, there's a dynamic search for a target migration node on each migration that follows the normal alloc fallback path. This ends up making migration nodes special.

As we have discussed at LSFMM this year and there seemed to be a goog consensus on that, the resulting implementation should be as pmem neutral as possible. After all node migration mode sounds like a reasonable feature even without pmem. So I would be more inclined to the normal alloc fallback path rather than a very specific and static migration fallback path. If that turns out impractical then sure let's come up with something more specific but I think there is quite a long route there because we do not really have much of an experience with this so far.

Quoted 3 lines

There are also some different choices that are pretty arbitrary. For instance, when you allocation a migration target page, should you cause memory pressure on the target?

Those are details to really sort out and they require some experimentation to.

Quoted 3 lines

To be honest, though, I don't see anything fatally flawed with it. It's probably a useful exercise to factor out the common bits from the two sets and see what we can agree on being absolutely necessary.

Makes sense. What would that be? Is there a real consensus on having the new node_reclaim mode to be the configuration mechanism? Do we want to support generic NUMA without any PMEM in place as well for starter?

Thanks!

Signature · 2 lines
Michal Hocko
SUSE Labs