Migrate Pages in lieu of discard
Subsystems (12)
cover Migrate Pages in lieu of discard
The full series is also available here:
https://github.com/hying-caritas/linux/tree/automigrate-20210715
The changes since the last post are as follows,
* Reduce code duplication via move common demotion condition code into can_demote_anon_pages() per Wei's comments.
--
We're starting to see systems with more and more kinds of memory such as Intel's implementation of persistent memory.
Let's say you have a system with some DRAM and some persistent memory. Today, once DRAM fills up, reclaim will start and some of the DRAM contents will be thrown out. Allocations will, at some point, start falling over to the slower persistent memory.
That has two nasty properties. First, the newer allocations can end up in the slower persistent memory. Second, reclaimed data in DRAM are just discarded even if there are gobs of space in persistent memory that could be used.
This set implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.
While I've talked about a DRAM/PMEM pairing, this approach would function in any environment where memory tiers exist.
This is not perfect. It "strands" pages in slower memory and never brings them back to fast DRAM. Huang Ying has follow-on work which repurposes autonuma to promote hot pages back to DRAM.
This is also all based on an upstream mechanism that allows persistent memory to be onlined and used as if it were volatile:
http://lkml.kernel.org/r/[email protected]
We have tested the patchset with the postgresql and pgbench. On a 2-socket server machine with DRAM and PMEM, the kernel with the patchset can improve the score of pgbench up to 22.1% compared with that of the DRAM only + disk case. This comes from the reduced disk read throughput (which reduces up to 70.8%).
== Open Issues ==
* Memory policies and cpusets that, for instance, restrict allocations to DRAM can be demoted to PMEM whenever they opt in to this new mechanism. A cgroup-level API to opt-in or opt-out of these migrations will likely be required as a follow-on.
* Could be more aggressive about where anon LRU scanning occurs since it no longer necessarily involves I/O. get_scan_count() for instance says: "If we have no swap space, do not bother scanning anon pages"
--
Changes since (automigrate-20210625):
* Reduce code duplication via move common demotion condition code into can_demote_anon_pages() per Wei's comments.
Changes since (automigrate-20210618):
* Squash the original 01/10 and 02/10 and move the RCU protection from the original 03/10 to the squashed 1/9.
* Make the newly added migrate_pages() parameter optional per Oscar's comments.
* Restore the original behavior of MADV_PAGEOUT per Zi's comments.
* Guard next_demotion_node() with numa_demotion_enabled per Wei's comments.
Changes since (automigrate-20210331):
* Change the page allocation flags per Michal's comments.
* Change the user interface to enable the feature.
Changes since (automigrate-20210304):
* Add ack/review tags
* Remove duplicate synchronize_rcu() call
Changes since (automigrate-20210122):
* move from GFP_HIGHUSER -> GFP_HIGHUSER_MOVABLE since pages *are* movable.
* Separate out helpers that check for being able to relaim anonymous pages versus being able to meaningfully scan the anon LRU.
Changes since (automigrate-20200818):
* Fall back to normal reclaim when demotion fails
* Fix some compile issues, when page migration and NUMA are off
Changes since (automigrate-20201007):
* separate out checks for "can scan anon LRU" from "can actually swap anon pages right now". Previous series conflated them and may have been overly aggressive scanning LRU
* add MR_DEMOTION to tracepoint header
* remove unnecessary hugetlb page check
Changes since (https://lwn.net/Articles/824830/):
* Use higher-level migrate_pages() API approach from Yang Shi's earlier patches.
* made sure to actually check node_reclaim_mode's new bit
* disabled migration entirely before introducing RECLAIM_MIGRATE
* Replace GFP_NOWAIT with explicit __GFP_KSWAPD_RECLAIM and comment why we want that.
* Comment on effects of that keep multiple source nodes from sharing target nodes
PATCH 1/9 mm/numa: automatically generate node migration order
From: Dave Hansen <[email protected]>
Prepare for the kernel to auto-migrate pages to other memory nodes with a node migration table. This allows creating single migration target for each NUMA node to enable the kernel to do NUMA page migrations instead of simply discarding colder pages. A node with no target is a "terminal node", so reclaim acts normally there. The migration target does not fundamentally _need_ to be a single node, but this implementation starts there to limit complexity.
When memory fills up on a node, memory contents can be automatically migrated to another node. The biggest problems are knowing when to migrate and to where the migration should be targeted.
The most straightforward way to generate the "to where" list would be to follow the page allocator fallback lists. Those lists already tell us if memory is full where to look next. It would also be logical to move memory in that order.
But, the allocator fallback lists have a fatal flaw: most nodes appear in all the lists. This would potentially lead to migration cycles (A->B, B->A, A->B, ...).
Instead of using the allocator fallback lists directly, keep a separate node migration ordering. But, reuse the same data used to generate page allocator fallback in the first place:find_next_best_node().
This means that the firmware data used to populate node distances essentially dictates the ordering for now. It should also be architecture-neutral since all NUMA architectures have a working find_next_best_node().
RCU is used to allow lock-less read of node_demotion[] and prevent demotion cycles been observed. If multiple reads of node_demotion[] are performed, a single rcu_read_lock() must be held over all reads to ensure no cycles are observed. Details are as follows.
=== What does RCU provide? ===
Imaginge a simple loop which walks down the demotion path looking for the last node:
terminal_node = start_node;
while (node_demotion[terminal_node] != NUMA_NO_NODE) {
terminal_node = node_demotion[terminal_node];
}The initial values are:
node_demotion[0] = 1;
node_demotion[1] = NUMA_NO_NODE;and are updated to:
node_demotion[0] = NUMA_NO_NODE;
node_demotion[1] = 0;What guarantees that the cycle is not observed:
node_demotion[0] = 1;
node_demotion[1] = 0;and would loop forever?
With RCU, a rcu_read_lock/unlock() can be placed around the loop. Since the write side does a synchronize_rcu(), the loop that observed the old contents is known to be complete before the synchronize_rcu() has completed.
RCU, combined with disable_all_migrate_targets(), ensures that the old migration state is not visible by the time __set_migration_target_nodes() is called.
=== What does READ_ONCE() provide? ===
READ_ONCE() forbids the compiler from merging or reordering successive reads of node_demotion[]. This ensures that any updates are *eventually* observed.
Consider the above loop again. The compiler could theoretically read the entirety of node_demotion[] into local storage (registers) and never go back to memory, and *permanently* observe bad values for node_demotion[].
Note: RCU does not provide any universal compiler-ordering guarantees:
https://lore.kernel.org/lkml/[email protected]/
This code is unused for now. It will be called later in the series.
--
Changes from 20210618:
* Merge patches for data structure definition and initialization
* Move RCU usage from the next patch in series per Zi's comments
Changes from 20210302:
* Fix typo in node_demotion[] comment
Changes since 20200122:
* Make node_demotion[] __read_mostly
* Add big node_demotion[] comment
Changes in July 2020:
- Remove loop from next_demotion_node() and get_online_mems(). This means that the node returned by next_demotion_node() might now be offline, but the worst case is that the allocation fails. That's fine since it is transient.
--- mm/internal.h | 5 ++ mm/migrate.c | 216 ++++++++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 2 +- 3 files changed, 222 insertions(+), 1 deletion(-)
mm/internal.h +5 -0
@@ -539,12 +539,17 @@ static inline void mminit_validate_memmodel_limits(unsigned long *start_pfn,#ifdef CONFIG_NUMA extern int node_reclaim (struct pglist_data *,gfp_t ,unsigned int ); +extern int find_next_best_node (int node,nodemask_t *used_node_mask);#else static inline int node_reclaim (struct pglist_data *pgdat,gfp_t mask,unsigned int order) {return NODE_RECLAIM_NOSCAN ; } +static inline int find_next_best_node (int node,nodemask_t *used_node_mask) +{ +return NUMA_NO_NODE ; +}#endif extern int hwpoison_filter (struct page *p);
mm/migrate.c +216 -0
225 lines not on this page · raw message
mm/page_alloc.c +1 -1
@@ -6139,7 +6139,7 @@ static int node_load[MAX_NUMNODES];* * Return: node id of the found node or %NUMA_NO_NODE if no node is found. */ -static int find_next_best_node (int node,nodemask_t *used_node_mask) +int find_next_best_node (int node,nodemask_t *used_node_mask) {int n, val;int min_val =INT_MAX ;
Signature · 1 line
2.30.2
Zi Yan
reviewerRe: [PATCH -V10 1/9] mm/numa: automatically generate node migration order
Quoted 34 lines · Huang Ying
From: Dave Hansen <[email protected]>
Prepare for the kernel to auto-migrate pages to other memory nodes with a node migration table. This allows creating single migration target for each NUMA node to enable the kernel to do NUMA page migrations instead of simply discarding colder pages. A node with no target is a "terminal node", so reclaim acts normally there. The migration target does not fundamentally _need_ to be a single node, but this implementation starts there to limit complexity.
When memory fills up on a node, memory contents can be automatically migrated to another node. The biggest problems are knowing when to migrate and to where the migration should be targeted.
The most straightforward way to generate the "to where" list would be to follow the page allocator fallback lists. Those lists already tell us if memory is full where to look next. It would also be logical to move memory in that order.
But, the allocator fallback lists have a fatal flaw: most nodes appear in all the lists. This would potentially lead to migration cycles (
A->B,B->A,A->B, ...).Instead of using the allocator fallback lists directly, keep a separate node migration ordering. But, reuse the same data used to generate page allocator fallback in the first place:
find_next_best_node().This means that the firmware data used to populate node distances essentially dictates the ordering for now. It should also be architecture-neutral since all NUMA architectures have a working
find_next_best_node().
Quoted 8 lines
RCU is used to allow lock-less read of node_demotion[] and prevent demotion cycles been observed. If multiple reads of node_demotion[] are performed, a single
rcu_read_lock()must be held over all reads to ensure no cycles are observed. Details are as follows.=== What does RCU provide? ===
Imaginge a simple loop which walks down the demotion path looking
s/Imaginge/Imagine
Quoted 81 lines
for the last node:
terminal_node = start_node; while (node_demotion[terminal_node] != NUMA_NO_NODE) { terminal_node = node_demotion[terminal_node]; }The initial values are:
node_demotion[0] = 1; node_demotion[1] = NUMA_NO_NODE;and are updated to:
node_demotion[0] = NUMA_NO_NODE; node_demotion[1] = 0;What guarantees that the cycle is not observed:
node_demotion[0] = 1; node_demotion[1] = 0;and would loop forever?
With RCU, a rcu_read_lock/unlock() can be placed around the loop. Since the write side does a
synchronize_rcu(), the loop that observed the old contents is known to be complete before thesynchronize_rcu()has completed.RCU, combined with
disable_all_migrate_targets(), ensures that the old migration state is not visible by the time__set_migration_target_nodes()is called.=== What does
READ_ONCE()provide? ===
READ_ONCE()forbids the compiler from merging or reordering successive reads of node_demotion[]. This ensures that any updates are *eventually* observed.Consider the above loop again. The compiler could theoretically read the entirety of node_demotion[] into local storage (registers) and never go back to memory, and *permanently* observe bad values for node_demotion[].
Note: RCU does not provide any universal compiler-ordering guarantees:
https://lore.kernel.org/lkml/[email protected]/This code is unused for now. It will be called later in the series.
Signed-off-by Dave Hansen <[email protected]>Signed-off-by "Huang, Ying" <[email protected]>Reviewed-by Yang Shi <[email protected]>Reviewed-by Oscar Salvador <[email protected]>Cc Michal Hocko <[email protected]>Cc Wei Xu <[email protected]>Cc Zi Yan <[email protected]>Cc David Rientjes <[email protected]>Cc Dan Williams <[email protected]>Cc David Hildenbrand <[email protected]>--
Changes from 20210618:
* Merge patches for data structure definition and initialization
* Move RCU usage from the next patch in series per Zi's commentsChanges from 20210302:
* Fix typo in node_demotion[] commentChanges since 20200122:
* Make node_demotion[]__read_mostly
* Add big node_demotion[] commentChanges in July 2020:
- Remove loop fromnext_demotion_node()andget_online_mems(). This means that the node returned bynext_demotion_node()might now be offline, but the worst case is that the allocation fails. That's fine since it is transient.
Quoted 5 lines
--- mm/internal.h | 5 ++ mm/migrate.c | 216 ++++++++++++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 2 +- 3 files changed, 222 insertions(+), 1 deletion(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
—
Best Regards,
Yan, Zi
PATCH 2/9 mm/migrate: update node demotion order on hotplug events
In reply to Huang YingFrom: Dave Hansen <[email protected]>
Reclaim-based migration is attempting to optimize data placement in memory based on the system topology. If the system changes, so must the migration ordering.
The implementation is conceptually simple and entirely unoptimized. On any memory or CPU hotplug events, assume that a node was added or removed and recalculate all migration targets. This ensures that the node_demotion[] array is always ready to be used in case the new reclaim mode is enabled.
This recalculation is far from optimal, most glaringly that it does not even attempt to figure out the hotplug event would have some *actual* effect on the demotion order. But, given the expected paucity of hotplug events, this should be fine.
--
Changes since 20210618:
* moved RCU part to the prev patch in series.
Changes since 20210302:
* remove duplicate synchronize_rcu()
--- mm/migrate.c | 90 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 89 insertions(+), 1 deletion(-)
mm/migrate.c +89 -1
@@ -49,6 +49,7 @@#include <linux/sched/mm.h> #include <linux/ptrace.h> #include <linux/oom.h> +#include <linux/memory.h> #include <asm/tlbflush.h>
@@ -3057,6 +3058,7 @@ void migrate_vma_finalize(struct migrate_vma *migrate)EXPORT_SYMBOL (migrate_vma_finalize);#endif /* CONFIG_DEVICE_PRIVATE */ +#if defined(CONFIG_MEMORY_HOTPLUG )/* Disable reclaim-based migration. */ static void __disable_all_migrate_targets (void ) {
@@ -3191,10 +3193,96 @@ static void __set_migration_target_nodes(void)/* * For callers that do not hold get_online_mems() already. */ -__maybe_unused // <- temporay to prevent warnings during bisects static void set_migration_target_nodes (void ) {get_online_mems ();__set_migration_target_nodes ();put_online_mems (); } + +/* +* React to hotplug events that might affect the migration targets +* like events that online or offline NUMA nodes. +* +* The ordering is also currently dependent on which nodes have +* CPUs. That means we need CPU on/offline notification too. +*/ +static int migration_online_cpu (unsigned int cpu) +{ +set_migration_target_nodes (); +return 0 ; +} + +static int migration_offline_cpu (unsigned int cpu) +{ +set_migration_target_nodes (); +return 0 ; +} + +/* +* This leaves migrate-on-reclaim transiently disabled between +* the MEM_GOING_OFFLINE and MEM_OFFLINE events. This runs +* whether reclaim-based migration is enabled or not, which +* ensures that the user can turn reclaim-based migration at +* any time without needing to recalculate migration targets. +* +* These callbacks already hold get_online_mems(). That is why +* __set_migration_target_nodes() can be used as opposed to +* set_migration_target_nodes(). +*/ +static int __meminit migrate_on_reclaim_callback (struct notifier_block *self, +unsigned long action,void *arg) +{ +switch (action) { +case MEM_GOING_OFFLINE : +/* +* Make sure there are not transient states where +* an offline node is a migration target. This +* will leave migration disabled until the offline +* completes and the MEM_OFFLINE case below runs. +*/ +disable_all_migrate_targets (); +break ; +case MEM_OFFLINE : +case MEM_ONLINE : +/* +* Recalculate the target nodes once the node +* reaches its final state (online or offline). +*/ +__set_migration_target_nodes (); +break ; +case MEM_CANCEL_OFFLINE : +/* +* MEM_GOING_OFFLINE disabled all the migration +* targets. Reenable them. +*/ +__set_migration_target_nodes (); +break ; +case MEM_GOING_ONLINE : +case MEM_CANCEL_ONLINE : +break ; + } + +return notifier_from_errno (0 ); +} + +static int __init migrate_on_reclaim_init (void ) +{ +int ret; + + ret =cpuhp_setup_state (CPUHP_AP_ONLINE_DYN ,"migrate on reclaim" , + migration_online_cpu, + migration_offline_cpu); +/* +* In the unlikely case that this fails, the automatic +* migration targets may become suboptimal for nodes +* where N_CPU changes. With such a small impact in a +* rare case, do not bother trying to do anything special. +*/ +WARN_ON (ret <0 ); + +hotplug_memory_notifier (migrate_on_reclaim_callback,100 ); +return 0 ; +} +late_initcall (migrate_on_reclaim_init); +#endif /* CONFIG_MEMORY_HOTPLUG */
Signature · 1 line
2.30.2
Zi Yan
reviewerRe: [PATCH -V10 2/9] mm/migrate: update node demotion order on hotplug events
Quoted 30 lines · Huang Ying
From: Dave Hansen <[email protected]>
Reclaim-based migration is attempting to optimize data placement in memory based on the system topology. If the system changes, so must the migration ordering.
The implementation is conceptually simple and entirely unoptimized. On any memory or CPU hotplug events, assume that a node was added or removed and recalculate all migration targets. This ensures that the node_demotion[] array is always ready to be used in case the new reclaim mode is enabled.
This recalculation is far from optimal, most glaringly that it does not even attempt to figure out the hotplug event would have some *actual* effect on the demotion order. But, given the expected paucity of hotplug events, this should be fine.
Signed-off-by Dave Hansen <[email protected]>Signed-off-by "Huang, Ying" <[email protected]>Reviewed-by Yang Shi <[email protected]>Cc Michal Hocko <[email protected]>Cc Wei Xu <[email protected]>Cc Zi Yan <[email protected]>Cc osalvador <[email protected]>Cc David Rientjes <[email protected]>Cc Dan Williams <[email protected]>Cc David Hildenbrand <[email protected]>--
Quoted 8 lines
Changes since 20210618:
* moved RCU part to the prev patch in series.Changes since 20210302:
* remove duplicatesynchronize_rcu()--- mm/migrate.c | 90 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 89 insertions(+), 1 deletion(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
—
Best Regards,
Yan, Zi
PATCH 3/9 mm/migrate: enable returning precise migrate_pages() success count
In reply to Huang YingFrom: Yang Shi <[email protected]>
Under normal circumstances, migrate_pages() returns the number of pages migrated. In error conditions, it returns an error code. When returning an error code, there is no way to know how many pages were migrated or not migrated.
Make migrate_pages() return how many pages are demoted successfully for all cases, including when encountering errors. Page reclaim behavior will depend on this in subsequent patches.
--
Note: Yang Shi originally wrote the patch, thus the SoB. There was also a Reviewed-by provided since there were some modifications made to this after the original work.
Changes since 20210618:
* Reword some bits of the changelog/subject
* Allow callers to pass a NULL 'ret_succeeded'. This ensures that callers don't have to declare and pass a dummy variable.
Changes since 20210302:
* Fix definition of CONFIG_MIGRATION=n stub migrate_pages(). Its parameters were wrong, but oddly enough did not generate any compile errors.
Changes since 20200122:
* Fix migrate_pages() to manipulate nr_succeeded *value* rather than the pointer.
--- include/linux/migrate.h | 5 +++-- mm/compaction.c | 2 +- mm/gup.c | 2 +- mm/memory-failure.c | 2 +- mm/memory_hotplug.c | 2 +- mm/mempolicy.c | 4 ++-- mm/migrate.c | 11 ++++++++--- mm/page_alloc.c | 2 +- 8 files changed, 18 insertions(+), 12 deletions(-)
include/linux/migrate.h +3 -2
@@ -41,7 +41,8 @@ extern int migrate_page(struct address_space *mapping, struct page *newpage, struct page *page, enum migrate_mode mode);extern int migrate_pages (struct list_head *l,new_page_t new,free_page_t free, -unsigned long private,enum migrate_mode mode,int reason); +unsigned long private,enum migrate_mode mode,int reason, +unsigned int *ret_succeeded);extern struct page *alloc_migration_target (struct page *page,unsigned long private);extern int isolate_movable_page (struct page *page,isolate_mode_t mode);
@@ -56,7 +57,7 @@ extern int migrate_page_move_mapping(struct address_space *mapping,static inline void putback_movable_pages (struct list_head *l) {}static inline int migrate_pages (struct list_head *l,new_page_t new,free_page_t free,unsigned long private,enum migrate_mode mode, -int reason) +int reason,unsigned int *ret_succeeded) {return -ENOSYS ; }static inline struct page *alloc_migration_target (struct page *page,unsigned long private)
mm/compaction.c +1 -1
@@ -2398,7 +2398,7 @@ compact_zone(struct compact_control *cc, struct capture_control *capc) err =migrate_pages (&cc->migratepages, compaction_alloc, compaction_free, (unsigned long )cc, cc->mode, -MR_COMPACTION ); +MR_COMPACTION ,NULL );trace_mm_compaction_migratepages (cc->nr_migratepages, err, &cc->migratepages);
mm/gup.c +1 -1
@@ -1772,7 +1772,7 @@ static long check_and_migrate_movable_pages(unsigned long nr_pages,if (!list_empty (&movable_page_list)) { ret =migrate_pages (&movable_page_list, alloc_migration_target,NULL , (unsigned long )&mtc,MIGRATE_SYNC , -MR_LONGTERM_PIN ); +MR_LONGTERM_PIN ,NULL );if (ret && !list_empty (&movable_page_list))putback_movable_pages (&movable_page_list); }
mm/memory-failure.c +1 -1
@@ -2093,7 +2093,7 @@ static int __soft_offline_page(struct page *page)if (isolate_page (hpage, &pagelist)) { ret =migrate_pages (&pagelist, alloc_migration_target,NULL , - (unsigned long )&mtc,MIGRATE_SYNC ,MR_MEMORY_FAILURE ); + (unsigned long )&mtc,MIGRATE_SYNC ,MR_MEMORY_FAILURE ,NULL );if (!ret) {bool release = !huge;
mm/memory_hotplug.c +1 -1
@@ -1469,7 +1469,7 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)if (nodes_empty (nmask))node_set (mtc.nid, nmask); ret =migrate_pages (&source, alloc_migration_target,NULL , - (unsigned long )&mtc,MIGRATE_SYNC ,MR_MEMORY_HOTPLUG ); + (unsigned long )&mtc,MIGRATE_SYNC ,MR_MEMORY_HOTPLUG ,NULL );if (ret) {list_for_each_entry (page, &source, lru) {if (__ratelimit (&migrate_rs)) {
mm/mempolicy.c +2 -2
@@ -1084,7 +1084,7 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest,if (!list_empty (&pagelist)) { err =migrate_pages (&pagelist, alloc_migration_target,NULL , - (unsigned long )&mtc,MIGRATE_SYNC ,MR_SYSCALL ); + (unsigned long )&mtc,MIGRATE_SYNC ,MR_SYSCALL ,NULL );if (err)putback_movable_pages (&pagelist); }
@@ -1338,7 +1338,7 @@ static long do_mbind(unsigned long start, unsigned long len,if (!list_empty (&pagelist)) {WARN_ON_ONCE (flags &MPOL_MF_LAZY ); nr_failed =migrate_pages (&pagelist, new_page,NULL , - start,MIGRATE_SYNC ,MR_MEMPOLICY_MBIND ); + start,MIGRATE_SYNC ,MR_MEMPOLICY_MBIND ,NULL );if (nr_failed)putback_movable_pages (&pagelist); }
mm/migrate.c +8 -3
@@ -1429,6 +1429,8 @@ static inline int try_split_thp(struct page *page, struct page **page2,* @mode: The migration mode that specifies the constraints for * page migration, if any. * @reason: The reason for page migration. +* @ret_succeeded: Set to the number of pages migrated successfully if +* the caller passes a non-NULL pointer. * * The function returns after 10 attempts or if no pages are movable any more * because the list has become empty or no retryable pages exist any more.
@@ -1439,7 +1441,7 @@ static inline int try_split_thp(struct page *page, struct page **page2,*/ int migrate_pages (struct list_head *from,new_page_t get_new_page,free_page_t put_new_page,unsigned long private, -enum migrate_mode mode,int reason) +enum migrate_mode mode,int reason,unsigned int *ret_succeeded) {int retry =1 ;int thp_retry =1 ;
@@ -1594,6 +1596,9 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page,if (!swapwrite) current->flags &= ~PF_SWAPWRITE ; +if (ret_succeeded) + *ret_succeeded = nr_succeeded; +return rc; }
@@ -1663,7 +1668,7 @@ static int do_move_pages_to_node(struct mm_struct *mm, }; err =migrate_pages (pagelist, alloc_migration_target,NULL , - (unsigned long )&mtc,MIGRATE_SYNC ,MR_SYSCALL ); + (unsigned long )&mtc,MIGRATE_SYNC ,MR_SYSCALL ,NULL );if (err)putback_movable_pages (pagelist);return err;
@@ -2178,7 +2183,7 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,list_add (&page->lru, &migratepages); nr_remaining =migrate_pages (&migratepages, *new,NULL , node, -MIGRATE_ASYNC ,MR_NUMA_MISPLACED ); +MIGRATE_ASYNC ,MR_NUMA_MISPLACED ,NULL );if (nr_remaining) {if (!list_empty (&migratepages)) {list_del (&page->lru);
mm/page_alloc.c +1 -1
@@ -8960,7 +8960,7 @@ static int __alloc_contig_migrate_range(struct compact_control *cc, cc->nr_migratepages -= nr_reclaimed; ret =migrate_pages (&cc->migratepages, alloc_migration_target, -NULL , (unsigned long )&mtc, cc->mode,MR_CONTIG_RANGE ); +NULL , (unsigned long )&mtc, cc->mode,MR_CONTIG_RANGE ,NULL );/* * On -ENOMEM, migrate_pages() bails out right away. It is pointless
Signature · 1 line
2.30.2
Zi Yan
reviewerRe: [PATCH -V10 3/9] mm/migrate: enable returning precise migrate_pages() success count
Quoted 41 lines · Huang Ying
From: Yang Shi <[email protected]>
Under normal circumstances,
migrate_pages()returns the number of pages migrated. In error conditions, it returns an error code. When returning an error code, there is no way to know how many pages were migrated or not migrated.Make
migrate_pages()return how many pages are demoted successfully for all cases, including when encountering errors. Page reclaim behavior will depend on this in subsequent patches.Signed-off-by Yang Shi <[email protected]>Signed-off-by Dave Hansen <[email protected]>Signed-off-by "Huang, Ying" <[email protected]>Suggested-by osalvador <[email protected]> [optional parameter]Reviewed-by Yang Shi <[email protected]>Cc Michal Hocko <[email protected]>Cc Wei Xu <[email protected]>Cc Zi Yan <[email protected]>Cc Dan Williams <[email protected]>Cc David Hildenbrand <[email protected]>--
Note: Yang Shi originally wrote the patch, thus the SoB. There was also a Reviewed-by provided since there were some modifications made to this after the original work.
Changes since 20210618:
* Reword some bits of the changelog/subject
* Allow callers to pass a NULL 'ret_succeeded'. This ensures that callers don't have to declare and pass a dummy variable.Changes since 20210302:
* Fix definition ofCONFIG_MIGRATION=nstubmigrate_pages(). Its parameters were wrong, but oddly enough did not generate any compile errors.Changes since 20200122:
* Fixmigrate_pages()to manipulatenr_succeeded*value* rather than the pointer.
Quoted 11 lines
--- include/linux/migrate.h | 5 +++-- mm/compaction.c | 2 +- mm/gup.c | 2 +- mm/memory-failure.c | 2 +- mm/memory_hotplug.c | 2 +- mm/mempolicy.c | 4 ++-- mm/migrate.c | 11 ++++++++--- mm/page_alloc.c | 2 +- 8 files changed, 18 insertions(+), 12 deletions(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
—
Best Regards,
Yan, Zi
PATCH 4/9 mm/migrate: demote pages during reclaim
In reply to Huang YingFrom: Dave Hansen <[email protected]>
This is mostly derived from a patch from Yang Shi:
https://lore.kernel.org/linux-mm/[email protected]/
Add code to the reclaim path (shrink_page_list()) to "demote" data to another NUMA node instead of discarding the data. This always avoids the cost of I/O needed to read the page back in and sometimes avoids the writeout cost when the page is dirty.
A second pass through shrink_page_list() will be made if any demotions fail. This essentially falls back to normal reclaim behavior in the case that demotions fail. Previous versions of this patch may have simply failed to reclaim pages which were eligible for demotion but were unable to be demoted in practice.
For some cases, for example, MADV_PAGEOUT, the pages are always discarded instead of demoted to follow the kernel API definition. Because MADV_PAGEOUT is defined as freeing specified pages regardless in which tier they are.
Note: This just adds the start of infrastructure for migration. It is actually disabled next to the FIXME in migrate_demote_page_ok().
--
changes from 20210625:
* Move common demotion condition into can_demote_anon_pages() and call it out of loop in shrink_page_list() per Wei's comments.
changes from 20210618:
* Fix some typos in comments and patch description.
* Fix MADV_PAGEOUT behavior per Zi's comments.
changes from 20210122:
* move from GFP_HIGHUSER -> GFP_HIGHUSER_MOVABLE (Ying)
changes from 202010:
* add MR_NUMA_MISPLACED to trace MIGRATE_REASON define
* make migrate_demote_page_ok() static, remove 'sc' arg until later patch
* remove unnecessary alloc_demote_page() hugetlb warning
* Simplify alloc_demote_page() gfp mask. Depend on __GFP_NORETRY to make it lightweight instead of fancier stuff like leaving out __GFP_IO/FS.
* Allocate migration page with alloc_migration_target() instead of allocating directly.
changes from 20200730:
* Add another pass through shrink_page_list() when demotion fails.
changes from 20210302:
* Use __GFP_THISNODE and revise the comment explaining the GFP mask constructionn
--- include/linux/migrate.h | 9 ++++ include/trace/events/migrate.h | 3 +- mm/vmscan.c | 85 ++++++++++++++++++++++++++++++++++ 3 files changed, 96 insertions(+), 1 deletion(-)
include/linux/migrate.h +9 -0
@@ -28,6 +28,7 @@ enum migrate_reason {MR_NUMA_MISPLACED ,MR_CONTIG_RANGE ,MR_LONGTERM_PIN , +MR_DEMOTION , MR_TYPES };
@@ -167,6 +168,14 @@ struct migrate_vma {int migrate_vma_setup (struct migrate_vma *args);void migrate_vma_pages (struct migrate_vma *migrate);void migrate_vma_finalize (struct migrate_vma *migrate); +int next_demotion_node (int node); + +#else /* CONFIG_MIGRATION disabled: */ + +static inline int next_demotion_node (int node) +{ +return NUMA_NO_NODE ; +}#endif /* CONFIG_MIGRATION */
include/trace/events/migrate.h +2 -1
@@ -21,7 +21,8 @@ EM( MR_MEMPOLICY_MBIND, "mempolicy_mbind") \ EM( MR_NUMA_MISPLACED, "numa_misplaced") \ EM( MR_CONTIG_RANGE, "contig_range") \ -EMe (MR_LONGTERM_PIN ,"longterm_pin" ) + EM( MR_LONGTERM_PIN, "longterm_pin") \ +EMe (MR_DEMOTION ,"demotion" )/* * First define the enums in the above macros to be exported to userspace
mm/vmscan.c +85 -0
139 lines not on this page · raw message
Signature · 1 line
2.30.2
PATCH 5/9 mm/vmscan: add page demotion counter
In reply to Huang YingFrom: Yang Shi <[email protected]>
Account the number of demoted pages.
Add pgdemote_kswapd and pgdemote_direct VM counters showed in /proc/vmstat.
[ daveh:
- __count_vm_events() a bit, and made them look at the THP size directly rather than getting data from migrate_pages()
]
--
Changes since 202010:
* remove unused scan-control 'demoted' field
--- include/linux/vm_event_item.h | 2 ++ mm/vmscan.c | 5 +++++ mm/vmstat.c | 2 ++ 3 files changed, 9 insertions(+)
include/linux/vm_event_item.h +2 -0
@@ -33,6 +33,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,PGREUSE ,PGSTEAL_KSWAPD ,PGSTEAL_DIRECT , +PGDEMOTE_KSWAPD , +PGDEMOTE_DIRECT ,PGSCAN_KSWAPD ,PGSCAN_DIRECT , PGSCAN_DIRECT_THROTTLE,
mm/vmscan.c +5 -0
@@ -1316,6 +1316,11 @@ static unsigned int demote_page_list(struct list_head *demote_pages, target_nid,MIGRATE_ASYNC ,MR_DEMOTION , &nr_succeeded); +if (current_is_kswapd ()) +__count_vm_events (PGDEMOTE_KSWAPD , nr_succeeded); +else +__count_vm_events (PGDEMOTE_DIRECT , nr_succeeded); +return nr_succeeded; }
mm/vmstat.c +2 -0
@@ -1217,6 +1217,8 @@ const char * const vmstat_text[] = {"pgreuse" ,"pgsteal_kswapd" ,"pgsteal_direct" , +"pgdemote_kswapd" , +"pgdemote_direct" ,"pgscan_kswapd" ,"pgscan_direct" , "pgscan_direct_throttle",
Signature · 1 line
2.30.2
PATCH 6/9 mm/vmscan: add helper for querying ability to age anonymous pages
In reply to Huang YingFrom: Dave Hansen <[email protected]>
Anonymous pages are kept on their own LRU(s). These lists could theoretically always be scanned and maintained. But, without swap, there is currently nothing the kernel can *do* with the results of a scanned, sorted LRU for anonymous pages.
A check for '!total_swap_pages' currently serves as a valid check as to whether anonymous LRUs should be maintained. However, another method will be added shortly: page demotion.
Abstract out the 'total_swap_pages' checks into a helper, give it a logically significant name, and check for the possibility of page demotion.
Changes since 20210625:
* Rename function to be more consistent with other similar functions.
Changes since 20210618:
* Rename function per Oscar's comments.
* Change parameter per Wei's comments.
* Make the function static.
* Consider whether demotion is disabled.
--- mm/vmscan.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-)
mm/vmscan.c +18 -2
@@ -2724,6 +2724,21 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, } } +/* +* Anonymous LRU management is a waste if there is +* ultimately no way to reclaim the memory. +*/ +static bool can_age_anon_pages (struct pglist_data *pgdat, +struct scan_control *sc) +{ +/* Aging the anon LRU is valuable if swap is present: */ +if (total_swap_pages >0 ) +return true; + +/* Also valuable if anon pages can be demoted: */ +return can_demote_anon_pages (pgdat->node_id, sc); +} +static void shrink_lruvec (struct lruvec *lruvec,struct scan_control *sc) {unsigned long nr[NR_LRU_LISTS ];
@@ -2833,7 +2848,8 @@ static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc)* Even if we did not try to evict anon pages at all, we want to * rebalance the anon lru active/inactive ratio. */ -if (total_swap_pages &&inactive_is_low (lruvec,LRU_INACTIVE_ANON )) +if (can_age_anon_pages (lruvec_pgdat (lruvec), sc) && +inactive_is_low (lruvec,LRU_INACTIVE_ANON ))shrink_active_list (SWAP_CLUSTER_MAX , lruvec, sc,LRU_ACTIVE_ANON ); }
@@ -3662,7 +3678,7 @@ static void age_active_anon(struct pglist_data *pgdat,struct mem_cgroup *memcg;struct lruvec *lruvec; -if (!total_swap_pages) +if (!can_age_anon_pages (pgdat, sc))return ; lruvec =mem_cgroup_lruvec (NULL , pgdat);
Signature · 1 line
2.30.2
PATCH 7/9 mm/vmscan: Consider anonymous pages without swap
In reply to Huang YingFrom: Keith Busch <[email protected]>
Reclaim anonymous pages if a migration path is available now that demotion provides a non-swap recourse for reclaiming anon pages.
Note that this check is subtly different from the can_age_anon_pages() checks. This mechanism checks whether a specific page in a specific context can actually be reclaimed, given current swap space and cgroup limits.
can_age_anon_pages() is a much simpler and more preliminary check which just says whether there is a possibility of future reclaim.
--
Changes since 20210618:
* Consider whether demotion is disabled
Changes from Dave 202010:
* remove 'total_swap_pages' modification
Changes from Dave 202006:
* rename reclaim_anon_pages()->can_reclaim_anon_pages()
Note: Keith's Intel SoB is commented out because he is no longer at Intel and his @intel.com mail will bounce.
--- mm/vmscan.c | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-)
mm/vmscan.c +30 -4
@@ -521,7 +521,7 @@ static long add_nr_deferred(long nr, struct shrinker *shrinker,static bool can_demote_anon_pages (int nid,struct scan_control *sc) { -if (sc->no_demotion) +if (sc && sc->no_demotion)return false;if (next_demotion_node (nid) ==NUMA_NO_NODE )return false;
@@ -530,6 +530,31 @@ static bool can_demote_anon_pages(int nid, struct scan_control *sc)return false; } +static inline bool can_reclaim_anon_pages (struct mem_cgroup *memcg, +int nid, +struct scan_control *sc) +{ +if (memcg ==NULL ) { +/* +* For non-memcg reclaim, is there +* space in any swap device? +*/ +if (get_nr_swap_pages () >0 ) +return true; + }else { +/* Is the memcg below its swap limit? */ +if (mem_cgroup_get_nr_swap_pages (memcg) >0 ) +return true; + } + +/* +* The page can not be swapped. +* +* Can it be reclaimed from this node via demotion? +*/ +return can_demote_anon_pages (nid, sc); +} +/* * This misses isolated pages which are not accounted for to save counters. * As the data only determines if reclaim or compaction continues, it is
@@ -541,7 +566,7 @@ unsigned long zone_reclaimable_pages(struct zone *zone) nr =zone_page_state_snapshot (zone,NR_ZONE_INACTIVE_FILE ) +zone_page_state_snapshot (zone,NR_ZONE_ACTIVE_FILE ); -if (get_nr_swap_pages () >0 ) +if (can_reclaim_anon_pages (NULL ,zone_to_nid (zone),NULL )) nr +=zone_page_state_snapshot (zone,NR_ZONE_INACTIVE_ANON ) +zone_page_state_snapshot (zone,NR_ZONE_ACTIVE_ANON );
@@ -2539,6 +2564,7 @@ enum scan_balance {static void get_scan_count (struct lruvec *lruvec,struct scan_control *sc,unsigned long *nr) { +struct pglist_data *pgdat =lruvec_pgdat (lruvec);struct mem_cgroup *memcg =lruvec_memcg (lruvec);unsigned long anon_cost, file_cost, total_cost;int swappiness =mem_cgroup_swappiness (memcg);
@@ -2549,7 +2575,7 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc,enum lru_list lru;/* If we have no swap space, do not bother scanning anon pages. */ -if (!sc->may_swap ||mem_cgroup_get_nr_swap_pages (memcg) <=0 ) { +if (!sc->may_swap || !can_reclaim_anon_pages (memcg, pgdat->node_id, sc)) { scan_balance =SCAN_FILE ; goto out; }
@@ -2919,7 +2945,7 @@ static inline bool should_continue_reclaim(struct pglist_data *pgdat,*/ pages_for_compaction =compact_gap (sc->order); inactive_lru_pages =node_page_state (pgdat,NR_INACTIVE_FILE ); -if (get_nr_swap_pages () >0 ) +if (can_reclaim_anon_pages (NULL , pgdat->node_id, sc)) inactive_lru_pages +=node_page_state (pgdat,NR_INACTIVE_ANON );return inactive_lru_pages > pages_for_compaction;
Signature · 1 line
2.30.2
PATCH 8/9 mm/vmscan: never demote for memcg reclaim
In reply to Huang YingFrom: Dave Hansen <[email protected]>
Global reclaim aims to reduce the amount of memory used on a given node or set of nodes. Migrating pages to another node serves this purpose.
memcg reclaim is different. Its goal is to reduce the total memory consumption of the entire memcg, across all nodes. Migration does not assist memcg reclaim because it just moves page contents between nodes rather than actually reducing memory consumption.
--
changes from 20210618:
* Avoid to scan anon lists for demotion for cgroup reclaim.
--- mm/vmscan.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
mm/vmscan.c +7 -2
@@ -521,8 +521,13 @@ static long add_nr_deferred(long nr, struct shrinker *shrinker,static bool can_demote_anon_pages (int nid,struct scan_control *sc) { -if (sc && sc->no_demotion) -return false; +if (sc) { +if (sc->no_demotion) +return false; +/* It is pointless to do demotion in memcg reclaim */ +if (cgroup_reclaim (sc)) +return false; + }if (next_demotion_node (nid) ==NUMA_NO_NODE )return false;
Signature · 1 line
2.30.2
PATCH 9/9 mm/migrate: add sysfs interface to enable reclaim migration
In reply to Huang YingSome method is obviously needed to enable reclaim-based migration.
Just like traditional autonuma, there will be some workloads that will benefit like workloads with more "static" configurations where hot pages stay hot and cold pages stay cold. If pages come and go from the hot and cold sets, the benefits of this approach will be more limited.
The benefits are truly workload-based and *not* hardware-based. We do not believe that there is a viable threshold where certain hardware configurations should have this mechanism enabled while others do not.
To be conservative, earlier work defaulted to disable reclaim- based migration and did not include a mechanism to enable it. This proposes add a new sysfs file
/sys/kernel/mm/numa/demotion_enabled
as a method to enable it.
We are open to any alternative that allows end users to enable this mechanism or disable it if workload harm is detected (just like traditional autonuma).
Once this is enabled page demotion may move data to a NUMA node that does not fall into the cpuset of the allocating process. This could be construed to violate the guarantees of cpusets. However, since this is an opt-in mechanism, the assumption is that anyone enabling it is content to relax the guarantees.
Changes since 20210618:
* Guard next_demotion_node() with numa_demotion_enabled if necessary per Wei's comments.
Changes since 20210331:
* Use sysfs interface separated from the zone_reclaim sysctl.
Changes since 20210304:
* Add Documentation/ material about relaxing cpuset constraints
Changes since 20200122:
* Changelog material about relaxing cpuset constraints
--- .../ABI/testing/sysfs-kernel-mm-numa | 24 ++++++++ include/linux/mempolicy.h | 4 ++ mm/mempolicy.c | 61 +++++++++++++++++++ mm/vmscan.c | 5 +- 4 files changed, 92 insertions(+), 2 deletions(-) create mode 100644 Documentation/ABI/testing/sysfs-kernel-mm-numa
Documentation/ABI/testing/sysfs-kernel-mm-numa new file +24 -0
@@ -0,0 +1,24 @@ +What: /sys/kernel/mm/numa/ +Date: June 2021 +Contact: Linux memory management mailing list <[email protected]> +Description: Interface for NUMA + +What: /sys/kernel/mm/numa/demotion_enabled +Date: June 2021 +Contact: Linux memory management mailing list <[email protected]> +Description: Enable/disable demoting pages during reclaim + + Page migration during reclaim is intended for systems + with tiered memory configurations. These systems have + multiple types of memory with varied performance + characteristics instead of plain NUMA systems where + the same kind of memory is found at varied distances. + Allowing page migration during reclaim enables these + systems to migrate pages from fast tiers to slow tiers + when the fast tier is under pressure. This migration + is performed before swap. It may move data to a NUMA + node that does not fall into the cpuset of the + allocating process which might be construed to violate + the guarantees of cpusets. This should not be enabled + on systems which need strict cpuset location + guarantees.
include/linux/mempolicy.h +4 -0
@@ -184,6 +184,8 @@ extern bool vma_migratable(struct vm_area_struct *vma);extern int mpol_misplaced (struct page *,struct vm_area_struct *,unsigned long );extern void mpol_put_task_policy (struct task_struct *); +extern bool numa_demotion_enabled; +#else struct mempolicy {};
@@ -292,5 +294,7 @@ static inline nodemask_t *policy_nodemask_current(gfp_t gfp) {return NULL ; } + +#define numa_demotion_enabled false#endif /* CONFIG_NUMA */ #endif
mm/mempolicy.c +61 -0
64 lines not on this page · raw message
mm/vmscan.c +3 -2
@@ -521,6 +521,8 @@ static long add_nr_deferred(long nr, struct shrinker *shrinker,static bool can_demote_anon_pages (int nid,struct scan_control *sc) { +if (!numa_demotion_enabled) +return false;if (sc) {if (sc->no_demotion)return false;
@@ -531,8 +533,7 @@ static bool can_demote_anon_pages(int nid, struct scan_control *sc)if (next_demotion_node (nid) ==NUMA_NO_NODE )return false; -// FIXME: actually enable this later in the series -return false; +return true; } static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
Signature · 1 line
2.30.2
Andrew Morton
maintainerQuoted 39 lines · On Thu, 15 Jul 2021 13:51:36 +0800 Huang Ying <[email protected]> wrote:
The full series is also available here:
https://github.com/hying-caritas/linux/tree/automigrate-20210715The changes since the last post are as follows,
* Reduce code duplication via move common demotion condition code into
can_demote_anon_pages()per Wei's comments.--
We're starting to see systems with more and more kinds of memory such as Intel's implementation of persistent memory.
Let's say you have a system with some DRAM and some persistent memory. Today, once DRAM fills up, reclaim will start and some of the DRAM contents will be thrown out. Allocations will, at some point, start falling over to the slower persistent memory.
That has two nasty properties. First, the newer allocations can end up in the slower persistent memory. Second, reclaimed data in DRAM are just discarded even if there are gobs of space in persistent memory that could be used.
This set implements a solution to these problems. At the end of the reclaim process in
shrink_page_list()just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.While I've talked about a DRAM/PMEM pairing, this approach would function in any environment where memory tiers exist.
This is not perfect. It "strands" pages in slower memory and never brings them back to fast DRAM. Huang Ying has follow-on work which repurposes autonuma to promote hot pages back to DRAM.
This is also all based on an upstream mechanism that allows persistent memory to be onlined and used as if it were volatile:
Quoted 7 lines
http://lkml.kernel.org/r/[email protected]We have tested the patchset with the postgresql and pgbench. On a 2-socket server machine with DRAM and PMEM, the kernel with the patchset can improve the score of pgbench up to 22.1% compared with that of the DRAM only + disk case. This comes from the reduced disk read throughput (which reduces up to 70.8%).
The [0/n] description talks a lot about PMEM, but the patches themselves are all about NUMA nodes. I assume that what ties this together is that the PMEM tends to be organized as a NUMA node on its own, and that by enabling migrate-to-remote-node-during-reclaim, we get this PMEM behaviour as a desired side-effect?
IOW, perhaps this [0/n] description could explain the linkage between PMEM and NUMA nodes more explicitly.
Secondly... at some point it just won't make sense to migrate pages to a remote node because that remote node isn't remote enough. I mean, if accessing the remote node is 2% slower than the local node, it's best to just reclaim the page rather than going to the effort of migrating it around? And I assume the patchset doesn't (yet) attempt to make this judgement?
Thirdly, the final patch which makes the feature off-by-default is a concern. I expect this will cause the new feature to have much less testing (functional and performance). We could make it default-on for now, then flip that to default-off after 5.14-rc5 (for example). That will help, but is a bit lame. Is it not possible for the kernel to look at the overall system layout and make an educated guess as to whether this system will benefit?
And I see this, from the [9/9] changelog:
: The benefits are truly workload-based and *not* hardware-based. We do : not believe that there is a viable threshold where certain hardware : configurations should have this mechanism enabled while others do not.
so... why not simply enable the thing for all systems and be done with it? What sort of downsides are you anticipating? We could of course include an emergency-off knob, but hoping that it won't need to be used.
Finally, having a feature which is good for some workloads, bad for others and which contains a system-wide enable knob is really quite sad. It requires a lot of work from each and every operator in the world! They must experimentally run their workloads both with and without, and determine which is best. And they should rerun this testing periodically as userspace and the kernel evolve, to determine whether their earlier experimenting holds true. And what happens if workload #1 wins and workload #2 loses?
And of course, many operators simply won't do all of this and they'll run slower, or will miss out of benefits. What can we do to relieve our users of all of this?
Quoted 7 lines · Andrew Morton
Quoted 5 lines · On Thu, 15 Jul 2021 13:51:36 +0800 Huang Ying <[email protected]> wrote:
We have tested the patchset with the postgresql and pgbench. On a 2-socket server machine with DRAM and PMEM, the kernel with the patchset can improve the score of pgbench up to 22.1% compared with that of the DRAM only + disk case. This comes from the reduced disk read throughput (which reduces up to 70.8%).
Quoted 5 lines
The [0/n] description talks a lot about PMEM, but the patches themselves are all about NUMA nodes. I assume that what ties this together is that the PMEM tends to be organized as a NUMA node on its own, and that by enabling migrate-to-remote-node-during-reclaim, we get this PMEM behaviour as a desired side-effect?
Yes, an assumption of this whole thing is that there are uniform performance capabilities within a NUMA node. ACPI systems that implement the HMAT table require that each proximity domain (PXM) have these uniform capabilities. This works out great since PXMs are so closely translated to NUMA nodes.
Quoted 2 lines
IOW, perhaps this [0/n] description could explain the linkage between PMEM and NUMA nodes more explicitly.
Sure, we can do that.
Quoted 5 lines
Secondly... at some point it just won't make sense to migrate pages to a remote node because that remote node isn't remote enough. I mean, if accessing the remote node is 2% slower than the local node, it's best to just reclaim the page rather than going to the effort of migrating it around?
One thing to keep in mind is that these migrations are *cheap*. Because of the point where the migration is done near the end of the reclaim process the page should be unmapped. The expensive part like TLB shootdowns don't need to happen.
But, yes, you're right. There are going to be situations where it would have been better to reclaim the page.
Quoted 1 line
And I assume the patchset doesn't (yet) attempt to make this judgement?
Correct. The assumption is that, on average, having something in slower memory is better than having it out of memory.
Quoted 7 lines
Thirdly, the final patch which makes the feature off-by-default is a concern. I expect this will cause the new feature to have much less testing (functional and performance). We could make it default-on for now, then flip that to default-off after 5.14-rc5 (for example). That will help, but is a bit lame. Is it not possible for the kernel to look at the overall system layout and make an educated guess as to whether this system will benefit?
Unfortunately, no. It's highly workload dependent. I see it a lot like autonuma or khugepaged/THP: there are going to be some big upsides, but we're under no illusions that the wins will be universal.
I'd actually be quite happy to flip it to be default-on all the time. But, I think I'll be a lot more confident about it once we have the "promotion" side of the equation merged as well:
https://lwn.net/Articles/835402/
Quoted 10 lines
And I see this, from the [9/9] changelog:
: The benefits are truly workload-based and *not* hardware-based. We do : not believe that there is a viable threshold where certain hardware : configurations should have this mechanism enabled while others do not.
so... why not simply enable the thing for all systems and be done with it? What sort of downsides are you anticipating? We could of course include an emergency-off knob, but hoping that it won't need to be used.
The worst case scenario is that an important workload starts up, eats a bunch of RAM and then goes idle, maybe during off-peak workload hours. During the night, a bunch of batch jobs kick in and evict some of that workload's data to PMEM. The workload spools up again the next morning and is hampered by the slower PMEM when there's lots of DRAM available.
This will be helped by the autonuma-based promotion once it's in place.
Quoted 9 lines
Finally, having a feature which is good for some workloads, bad for others and which contains a system-wide enable knob is really quite sad. It requires a lot of work from each and every operator in the world! They must experimentally run their workloads both with and without, and determine which is best. And they should rerun this testing periodically as userspace and the kernel evolve, to determine whether their earlier experimenting holds true. And what happens if workload #1 wins and workload #2 loses?
Quoted 3 lines
And of course, many operators simply won't do all of this and they'll run slower, or will miss out of benefits. What can we do to relieve our users of all of this?
I think this is a much bigger question than this patch set. It's basically the tale of /proc/sys/vm/* or /sys/kernel/mm. Some workloads win with transparent_hugepage/enabled=always, some lose. Some win with zone_reclaim_mode=7, some prefer zone_reclaim_mode=0.
The end game for auto-migration is to on by default, wherever these hardware configurations show up. Just like THP, I expect this to be broadly useful. But, given the breadth of our user base, I also expect it to be nasty for a minority of users. They'll need a way to turn it off, and hopefully they'll also provide feedback so that, if possible, we can improve the default behavior and bring them back into the fold.
Quoted 6 lines · Dave Hansen
The end game for auto-migration is to on by default, wherever these hardware configurations show up. Just like THP, I expect this to be broadly useful. But, given the breadth of our user base, I also expect it to be nasty for a minority of users. They'll need a way to turn it off, and hopefully they'll also provide feedback so that, if possible, we can improve the default behavior and bring them back into the fold.
... or if the THP experience is anything to go by, they continue to shout loudly about how borken it is for a decade ;-(
Quoted 10 lines · Andrew Morton
On Thu, 15 Jul 2021 13:51:36 +0800 Huang Ying <[email protected]> wrote:
The [0/n] description talks a lot about PMEM, but the patches themselves are all about NUMA nodes. I assume that what ties this together is that the PMEM tends to be organized as a NUMA node on its own, and that by enabling migrate-to-remote-node-during-reclaim, we get this PMEM behaviour as a desired side-effect?
IOW, perhaps this [0/n] description could explain the linkage between PMEM and NUMA nodes more explicitly.
Hi, Andrew,
I have added some words in the [0/9] description to link PMEM and NUMA nodes. The updated description is as below. Can you take a look at it?
Best Regards,
Huang, Ying
--------------------------8<-----------------------------------
We're starting to see systems with more and more kinds of memory such as Intel's implementation of persistent memory.
Let's say you have a system with some DRAM and some persistent memory. Today, once DRAM fills up, reclaim will start and some of the DRAM contents will be thrown out. Allocations will, at some point, start falling over to the slower persistent memory.
That has two nasty properties. First, the newer allocations can end up in the slower persistent memory. Second, reclaimed data in DRAM are just discarded even if there are gobs of space in persistent memory that could be used.
This patchset implements a solution to these problems. At the end of the reclaim process in shrink_page_list() just before the last page refcount is dropped, the page is migrated to persistent memory instead of being dropped.
While I've talked about a DRAM/PMEM pairing, this approach would function in any environment where memory tiers exist.
This is not perfect. It "strands" pages in slower memory and never brings them back to fast DRAM. Huang Ying has follow-on work which repurposes autonuma to promote hot pages back to DRAM.
This is also all based on an upstream mechanism that allows persistent memory to be onlined and used as if it were volatile:
http://lkml.kernel.org/r/[email protected]
With that, the DRAM and PMEM in each socket will be represented as 2 separate NUMA nodes, with the CPUs sit in the DRAM node. So the general inter-NUMA demotion mechanism introduced in the patchset can migrate the cold DRAM pages to the PMEM node.
We have tested the patchset with the postgresql and pgbench. On a 2-socket server machine with DRAM and PMEM, the kernel with the patchset can improve the score of pgbench up to 22.1% compared with that of the DRAM only + disk case. This comes from the reduced disk read throughput (which reduces up to 70.8%).
== Open Issues ==
* Memory policies and cpusets that, for instance, restrict allocations to DRAM can be demoted to PMEM whenever they opt in to this new mechanism. A cgroup-level API to opt-in or opt-out of these migrations will likely be required as a follow-on.
* Could be more aggressive about where anon LRU scanning occurs since it no longer necessarily involves I/O. get_scan_count() for instance says: "If we have no swap space, do not bother scanning anon pages"
Andrew Morton
maintainerQuoted 18 lines · Ying
Quoted 10 lines · Andrew Morton
On Thu, 15 Jul 2021 13:51:36 +0800 Huang Ying <[email protected]> wrote:
The [0/n] description talks a lot about PMEM, but the patches themselves are all about NUMA nodes. I assume that what ties this together is that the PMEM tends to be organized as a NUMA node on its own, and that by enabling migrate-to-remote-node-during-reclaim, we get this PMEM behaviour as a desired side-effect?
IOW, perhaps this [0/n] description could explain the linkage between PMEM and NUMA nodes more explicitly.
Hi, Andrew,
I have added some words in the [0/9] description to link PMEM and NUMA nodes. The updated description is as below. Can you take a look at it?
Quoted 7 lines
...
With that, the DRAM and PMEM in each socket will be represented as 2 separate NUMA nodes, with the CPUs sit in the DRAM node. So the general inter-NUMA demotion mechanism introduced in the patchset can migrate the cold DRAM pages to the PMEM node.
Bingo, thanks.