Transparent Page Placement for Tiered-Memory
Subsystems (13)
cover Transparent Page Placement for Tiered-Memory
[resend in proper format]
With the advent of new memory types and technologies, we can see different types of memory together, e.g. DRAM, PMEM, CXL-enabled memory, etc. In recent future, we can see CXL-Memory be available in the physical address- space as a CPU-less NUMA node along with the native DDR memory channels. As different types of memory have different level of performance impact, how we manage pages across the NUMA nodes should be a matter of concern.
Dave Hansen's patchset on "Migrate Pages in lieu of discard" demotes toptier pages to a slow tier node during the reclamation process.
https://lwn.net/Articles/860215/
However, that patchset does not include the features to promote pages on slow tier memory node to the toptier one. As a result, pages demoted or newly allocated on the slow tier node, experiences NUMA latency and hurt application performance. In this patch set, we augment existing AutoNUMA mechanism to promote pages from slow tier nodes to toptier nodes.
We decouple reclamation and allocation logics for the toptier node so that reclamation gets triggered at a higher watermark and demotes colder pages to the slow-tier memory. As a result, toptier nodes can maintain some free space to accept both new allocation and promotion from slowtier nodes. During promotion, we add hysteresis to page and only promote pages that are less likely to be demoted within a short period of time. This reduces the chance for a page being ping-ponged across the NUMA nodes due to frequent demotion and promotion within a short period of time.
We tested this patchset on systems with CXL-enabled DRAM and PMEM tiers. We find this patchset can bring hotter pages to the toptier node while moving the colder pages to the slow-tier nodes for a good range of Meta production workloads with live traffic. As a result, toptier nodes serve more hot pages and the application performance improves.
Case Study of a Meta cache application with two NUMA nodes
==========================================================
Toptier node: DRAM directly attached to the CPU Slowtier node: DRAM attached through CXL
Toptier vs Slowtier memory capacity ratio is 1:4
With default page placement policy, file caches fills up the toptier node and anons get trapped in the slowtier node. Only 14% of the total anons reside in toptier node. Remote NUMA read bandwidth is 80%. Throughput regression is 18% compared to all memory being served from toptier node.
This patchset brings 80% of the anons to the toptier node. Anons on the slowtier memory is mostly cold anons. As the toptier node can not host all the hot memory, some hot files still remain on the slowtier node. Even though, remote NUMA read bandwidth reduces from 80% to 40%. With this patchset, throughput regression is only 5% compared to the baseline of toptier node serving the whole working set.
Hasan Al Maruf (5):
Promotion and demotion related statistics NUMA balancing for tiered-memory system Decouple reclaim and allocation for toptier nodes Reclaim to satisfy WMARK_DEMOTE on toptier nodes active LRU-based promotion to avoid ping-pong
Documentation/admin-guide/sysctl/kernel.rst | 18 +++++ Documentation/admin-guide/sysctl/vm.rst | 12 ++++ include/linux/mempolicy.h | 11 ++- include/linux/mm.h | 4 ++ include/linux/mmzone.h | 5 ++ include/linux/node.h | 7 ++ include/linux/page-flags.h | 9 +++ include/linux/page_ext.h | 3 + include/linux/sched/numa_balancing.h | 63 ++++++++++++++++- include/linux/sched/sysctl.h | 6 ++ include/linux/vm_event_item.h | 13 ++++ include/trace/events/mmflags.h | 10 ++- kernel/sched/core.c | 36 ++++++++-- kernel/sched/fair.c | 23 ++++++- kernel/sched/sched.h | 2 + kernel/sysctl.c | 19 ++++-- mm/huge_memory.c | 29 +++++--- mm/memory.c | 15 +++- mm/mempolicy.c | 30 +++++++- mm/migrate.c | 48 ++++++++++--- mm/mprotect.c | 8 ++- mm/page_alloc.c | 34 ++++++++- mm/vmscan.c | 76 +++++++++++++++++++-- mm/vmstat.c | 20 +++++- 24 files changed, 451 insertions(+), 50 deletions(-)
--
2.30.2
PATCH 1/5 Promotion and demotion related statistics
Promotion and demotion related statistics can help better understand the effectiveness of the page placement mechanism. we want to know among the promoted/demoted pages what is the distribution of anon and file pages. how much of the demoted pages become promotion candidate can give us insight on whether the page placement mechanism is thrashing among the NUMA nodes. we can also use this information to rate limit the migration across the NUMA nodes.
Promotion can fail for many reasons, e.g., target node having low memory, page refcount being abnormal, whole system being low on memory etc. Adding counters to track the failure reasons will give the detailed info about why and where it fails, and help debugging the system.
To track the demoted pages, PG_demoted bit is introduced for pages that get demoted. Upon demotion, PG_demoted bit is set in thepage flag. upon promotion, the bit gets reset for that page.
promotion related statistics:
=============================pgpromote_candidate - candidates that get selected for promotion pgpromote_candidate_demoted - promotion candidate that got demoted earlier pgpromote_candidate_anon - promotion candidate that are anon pgpromote_candidate_file - promotion candidate that are file pgpromote_tried - pages that had a try to migrate via NUMA Balancing pgpromote_file- successfully promoted file pages pgpromote_anon - successfully promoted anon pages
promotion failure related statistics:
=====================================pgmigrate_fail_dst_node_full - failed as the target node is full pgmigrate_fail_numa_isolate - failed in isolating numa page pgmigrate_fail_nomem - failed as no memory left in the system pgmigrate_fail_refcount - failed as ref count mismatched
demotion related statistics:
============================pgdemote_file - successfully demoted file pagespgdemote_anon - successfully demoted anon pages
--- include/linux/mempolicy.h | 4 +- include/linux/page-flags.h | 9 ++++ include/linux/page_ext.h | 3 ++ include/linux/sched/numa_balancing.h | 63 +++++++++++++++++++++++++++- include/linux/vm_event_item.h | 13 ++++++ include/trace/events/mmflags.h | 10 ++++- kernel/sched/fair.c | 12 +++++- kernel/sched/sched.h | 1 + mm/huge_memory.c | 2 +- mm/memory.c | 2 +- mm/mempolicy.c | 7 +++- mm/migrate.c | 48 ++++++++++++++++----- mm/vmscan.c | 8 ++++ mm/vmstat.c | 13 ++++++ 14 files changed, 174 insertions(+), 21 deletions(-)
include/linux/mempolicy.h +2 -2
@@ -184,7 +184,7 @@ extern void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol);/* Check if a vma is migratable */ extern bool vma_migratable (struct vm_area_struct *vma); -extern int mpol_misplaced (struct page *,struct vm_area_struct *,unsigned long ); +extern int mpol_misplaced (struct page *,struct vm_area_struct *,unsigned long ,int );extern void mpol_put_task_policy (struct task_struct *);extern bool numa_demotion_enabled;
@@ -284,7 +284,7 @@ static inline int mpol_parse_str(char *str, struct mempolicy **mpol)#endif static inline int mpol_misplaced (struct page *page,struct vm_area_struct *vma, -unsigned long address) +unsigned long address,int flags) {return -1 ;/* no node preference */ }
include/linux/page-flags.h +9 -0
@@ -137,6 +137,9 @@ enum pageflags {#endif #ifdef CONFIG_64BIT PG_arch_2, +#ifdef CONFIG_NUMA_BALANCING + PG_demoted, +#endif #endif __NR_PAGEFLAGS,
@@ -443,6 +446,12 @@ TESTCLEARFLAG(Young, young, PF_ANY) PAGEFLAG(Idle, idle, PF_ANY)#endif +#if defined(CONFIG_NUMA_BALANCING ) && defined(CONFIG_64BIT ) +TESTPAGEFLAG(Demoted, demoted, PF_NO_TAIL) +SETPAGEFLAG(Demoted, demoted, PF_NO_TAIL) +TESTCLEARFLAG(Demoted, demoted, PF_NO_TAIL) +#endif +/* * PageReported() is used to track reported free pages within the Buddy * allocator. We can use the non-atomic version of the test and set
include/linux/page_ext.h +3 -0
@@ -23,6 +23,9 @@ enum page_ext_flags { PAGE_EXT_YOUNG, PAGE_EXT_IDLE,#endif +#if defined(CONFIG_NUMA_BALANCING ) && !defined(CONFIG_64BIT ) + PAGE_EXT_DEMOTED, +#endif };/*
include/linux/sched/numa_balancing.h +61 -2
90 lines not on this page · raw message
include/linux/vm_event_item.h +13 -0
@@ -35,6 +35,8 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,PGSTEAL_DIRECT ,PGDEMOTE_KSWAPD ,PGDEMOTE_DIRECT , +PGDEMOTE_FILE , +PGDEMOTE_ANON ,PGSCAN_KSWAPD ,PGSCAN_DIRECT , PGSCAN_DIRECT_THROTTLE,
@@ -56,9 +58,20 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,NUMA_HINT_FAULTS ,NUMA_HINT_FAULTS_LOCAL ,NUMA_PAGE_MIGRATE , +PGPROMOTE_CANDIDATE ,/* candidates get selected for promotion */ +PGPROMOTE_CANDIDATE_DEMOTED ,/* promotion candidate that got demoted earlier */ +PGPROMOTE_CANDIDATE_ANON ,/* promotion candidate that are anon */ +PGPROMOTE_CANDIDATE_FILE ,/* promotion candidate that are file */ +PGPROMOTE_TRIED ,/* tried to migrate via NUMA balancing */ +PGPROMOTE_FILE ,/* successfully promoted file pages */ +PGPROMOTE_ANON ,/* successfully promoted anon pages */ #endif #ifdef CONFIG_MIGRATIONPGMIGRATE_SUCCESS ,PGMIGRATE_FAIL , +PGMIGRATE_DST_NODE_FULL_FAIL ,/* failed as the target node is full */ +PGMIGRATE_NUMA_ISOLATE_FAIL ,/* failed in isolating numa page */ +PGMIGRATE_NOMEM_FAIL ,/* failed as no memory left */ +PGMIGRATE_REFCOUNT_FAIL ,/* failed in ref count */ THP_MIGRATION_SUCCESS ,THP_MIGRATION_FAIL , THP_MIGRATION_SPLIT,
include/trace/events/mmflags.h +9 -1
@@ -85,6 +85,13 @@#define IF_HAVE_PG_ARCH_2 (flag,string)#endif +#if defined(CONFIG_NUMA_BALANCING ) && defined(CONFIG_64BIT ) +#define IF_HAVE_PG_DEMOTED (flag, string) ,{1UL << flag, string} +#else +#define IF_HAVE_PG_DEMOTED (flag, string) +#endif + +#define __def_pageflag_names \ {1UL << PG_locked, "locked" }, \ {1UL << PG_waiters, "waiters" }, \
@@ -112,7 +119,8 @@ IF_HAVE_PG_UNCACHED(PG_uncached, "uncached" ) \ IF_HAVE_PG_HWPOISON(PG_hwpoison, "hwpoison" ) \ IF_HAVE_PG_IDLE(PG_young, "young" ) \ IF_HAVE_PG_IDLE(PG_idle, "idle" ) \ -IF_HAVE_PG_ARCH_2(PG_arch_2, "arch_2" ) +IF_HAVE_PG_ARCH_2(PG_arch_2, "arch_2") \ +IF_HAVE_PG_DEMOTED(PG_demoted, "demoted")#define show_page_flags (flags) \ (flags) ? __print_flags(flags, "|", \
kernel/sched/fair.c +11 -1
@@ -1416,12 +1416,22 @@ static inline unsigned long group_weight(struct task_struct *p, int nid, }bool should_numa_migrate_memory (struct task_struct *p,struct page * page, -int src_nid,int dst_cpu) +int src_nid,int dst_cpu,int flags) {struct numa_group *ng =deref_curr_numa_group (p);int dst_nid =cpu_to_node (dst_cpu);int last_cpupid, this_cpupid; +count_vm_numa_event (PGPROMOTE_CANDIDATE ); + +if (flags &TNF_DEMOTED ) +count_vm_numa_event (PGPROMOTE_CANDIDATE_DEMOTED ); + +if (page_is_file_lru (page)) +count_vm_numa_event (PGPROMOTE_CANDIDATE_FILE ); +else +count_vm_numa_event (PGPROMOTE_CANDIDATE_ANON ); + this_cpupid =cpu_pid_to_cpupid (dst_cpu, current->pid); last_cpupid =page_cpupid_xchg_last (page, this_cpupid);
kernel/sched/sched.h +1 -0
@@ -51,6 +51,7 @@#include <linux/kthread.h> #include <linux/membarrier.h> #include <linux/migrate.h> +#include <linux/mm_inline.h> #include <linux/mmu_context.h> #include <linux/nmi.h> #include <linux/proc_fs.h>
mm/huge_memory.c +1 -1
@@ -1475,7 +1475,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t pmd)* page_table_lock if at all possible */ page_locked =trylock_page (page); - target_nid =mpol_misplaced (page, vma, haddr); + target_nid =mpol_misplaced (page, vma, haddr, flags);if (target_nid ==NUMA_NO_NODE ) {/* If the page was locked, there are no parallel migrations */ if (page_locked)
mm/memory.c +1 -1
@@ -4131,7 +4131,7 @@ static int numa_migrate_prep(struct page *page, struct vm_area_struct *vma, *flags |=TNF_FAULT_LOCAL ; } -return mpol_misplaced (page, vma, addr); +return mpol_misplaced (page, vma, addr, *flags); }static vm_fault_t do_numa_page (struct vm_fault *vmf)
mm/mempolicy.c +5 -2
@@ -2466,7 +2466,7 @@ static void sp_free(struct sp_node *n)* Policy determination "mimics" alloc_page_vma(). * Called from fault path where we know the vma and faulting address. */ -int mpol_misplaced (struct page *page,struct vm_area_struct *vma,unsigned long addr) +int mpol_misplaced (struct page *page,struct vm_area_struct *vma,unsigned long addr,int flags) {struct mempolicy *pol;struct zoneref *z;
@@ -2477,6 +2477,9 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned longint polnid =NUMA_NO_NODE ;int ret =-1 ; +if (test_and_clear_page_demoted (page)) + flags |=TNF_DEMOTED ; + pol =get_vma_policy (vma, addr);if (!(pol->flags &MPOL_F_MOF )) goto out;
@@ -2526,7 +2529,7 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned longif (pol->flags &MPOL_F_MORON ) { polnid = thisnid; -if (!should_numa_migrate_memory (current, page, curnid, thiscpu)) +if (!should_numa_migrate_memory (current, page, curnid, thiscpu, flags)) goto out; }
mm/migrate.c +37 -11
120 lines not on this page · raw message
mm/vmscan.c +8 -0
@@ -1142,6 +1142,7 @@ static unsigned int demote_page_list(struct list_head *demote_pages,int target_nid =next_demotion_node (pgdat->node_id);unsigned int nr_succeeded;int err; +bool file_lru;if (list_empty (demote_pages))return 0 ;
@@ -1149,6 +1150,8 @@ static unsigned int demote_page_list(struct list_head *demote_pages,if (target_nid ==NUMA_NO_NODE )return 0 ; + file_lru =page_is_file_lru (lru_to_page (demote_pages)); +/* Demotion ignores all cpuset and mempolicy settings */ err = migrate_pages(demote_pages, alloc_demote_page, NULL, target_nid, MIGRATE_ASYNC, MR_DEMOTION,
@@ -1159,6 +1162,11 @@ static unsigned int demote_page_list(struct list_head *demote_pages,else __count_vm_events (PGDEMOTE_DIRECT , nr_succeeded); +if (file_lru) +__count_vm_events (PGDEMOTE_FILE , nr_succeeded); +else +__count_vm_events (PGDEMOTE_ANON , nr_succeeded); +return nr_succeeded; }
mm/vmstat.c +13 -0
@@ -1261,6 +1261,8 @@ const char * const vmstat_text[] = {"pgsteal_direct" ,"pgdemote_kswapd" ,"pgdemote_direct" , +"pgdemote_file" , +"pgdemote_anon" ,"pgscan_kswapd" ,"pgscan_direct" , "pgscan_direct_throttle",
@@ -1291,10 +1293,21 @@ const char * const vmstat_text[] = {"numa_hint_faults" ,"numa_hint_faults_local" ,"numa_pages_migrated" , +"pgpromote_candidate" , +"pgpromote_candidate_demoted" , +"pgpromote_candidate_anon" , +"pgpromote_candidate_file" , +"pgpromote_tried" , +"pgpromote_file" , +"pgpromote_anon" , #endif #ifdef CONFIG_MIGRATION"pgmigrate_success" ,"pgmigrate_fail" , +"pgmigrate_fail_dst_node_full" , +"pgmigrate_fail_numa_isolate" , +"pgmigrate_fail_nomem" , +"pgmigrate_fail_refcount" ,"thp_migration_success" ,"thp_migration_fail" , "thp_migration_split",
Signature · 1 line
2.30.2
PATCH 2/5 NUMA balancing for tiered-memory system
In reply to Hasan Al MarufWith the advent of new memory types and technologies, a server may have different types of memory, e.g. DRAM, PMEM, CXL-enabled memory, etc. As different types of memory usually have different level of performance impact, such a system can be called as a tiered-memory system.
In a tiered-memory system, NUMA memory nodes can be CPU-less nodes. For such a system, memory with CPU are considered as the toptier node while memory without CPU are non-toptier nodes.
In default NUMA Balancing, NUMA hint faults are generate on both toptier and non-toptier nodes. However, in a tiered-memory system, hot memories in toptier memory nodes may not need to be migrated around. In such cases, it's unnecessary to scan the pages in the toptier memory nodes. We disable unnecessary scannings in the toptier nodes for a tiered-memory system.
To support NUMA balancing for a tiered-memory system, the existing sysctl user-space interface for enabling numa_balancing is extended in a backward compatible way, so that users can enable/disable these functionalities individually. The sysctl is converted from a Boolean value to a bits field. Current definition for '/proc/sys/kernel/numa_balancing' is as follow:
- 0x0: NUMA_BALANCING_DISABLED
- 0x1: NUMA_BALANCING_NORMAL
- 0x2: NUMA_BALANCING_TIERED_MEMORY
If a system has single toptier node online, default NUMA balancing will automatically be downgraded to the tiered-memory mode to avoid the unnecessary scanning in the toptier node mentioned above.
--- Documentation/admin-guide/sysctl/kernel.rst | 18 +++++++++++ include/linux/mempolicy.h | 2 ++ include/linux/node.h | 7 ++++ include/linux/sched/sysctl.h | 6 ++++ kernel/sched/core.c | 36 +++++++++++++++++---- kernel/sched/fair.c | 10 +++++- kernel/sched/sched.h | 1 + kernel/sysctl.c | 7 ++-- mm/huge_memory.c | 27 ++++++++++------ mm/mprotect.c | 8 ++++- 10 files changed, 101 insertions(+), 21 deletions(-)
Documentation/admin-guide/sysctl/kernel.rst +18 -0
@@ -608,6 +608,24 @@ numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb`_, and numa_balancing_settle_count sysctls. +By default, NUMA hinting faults are generate on both toptier and non-toptier +nodes. However, in a tiered-memory system, hot memories in toptier memory nodes +may not need to be migrated around. In such cases, it's unnecessary to scan the +pages in the toptier memory nodes. For a tiered-memory system, unnecessary scannings +and hinting faults in the toptier nodes are disabled. + +This interface takes bits field as input. Supported values and corresponding modes are +as follow: + +- 0x0: NUMA_BALANCING_DISABLED +- 0x1: NUMA_BALANCING_NORMAL +- 0x2: NUMA_BALANCING_TIERED_MEMORY + +If a system has single toptier node online, then default NUMA balancing will +automatically be downgraded to the tiered-memory mode to avoid the unnecessary scanning +and hinting faults. + + numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb ===============================================================================================================================
include/linux/mempolicy.h +2 -0
@@ -188,6 +188,7 @@ extern int mpol_misplaced(struct page *, struct vm_area_struct *, unsigned long,extern void mpol_put_task_policy (struct task_struct *);extern bool numa_demotion_enabled; +extern bool numa_promotion_tiered_enabled;#else
@@ -299,5 +300,6 @@ static inline nodemask_t *policy_nodemask_current(gfp_t gfp) }#define numa_demotion_enabled false +#define numa_promotion_tiered_enabled false#endif /* CONFIG_NUMA */ #endif
include/linux/node.h +7 -0
@@ -181,4 +181,11 @@ static inline void register_hugetlbfs_with_node(node_registration_func_t reg,#define to_node (device) container_of(device, struct node, dev) +static inline bool node_is_toptier (int node) +{ +// ideally, toptier nodes should be the memory with CPU. +// for now, just assume node0 is the toptier memory +// return node_state(node, N_CPU); +return (node ==0 ); +}#endif /* _LINUX_NODE_H_ */
include/linux/sched/sysctl.h +6 -0
@@ -39,6 +39,12 @@ enum sched_tunable_scaling { };extern enum sched_tunable_scaling sysctl_sched_tunable_scaling; +#define NUMA_BALANCING_DISABLED 0x0 +#define NUMA_BALANCING_NORMAL 0x1 +#define NUMA_BALANCING_TIERED_MEMORY 0x2 + +extern int sysctl_numa_balancing_mode; +extern unsigned int sysctl_numa_balancing_scan_delay;extern unsigned int sysctl_numa_balancing_scan_period_min;extern unsigned int sysctl_numa_balancing_scan_period_max;
kernel/sched/core.c +29 -7
@@ -3596,9 +3596,29 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) }DEFINE_STATIC_KEY_FALSE (sched_numa_balancing); +int sysctl_numa_balancing_mode; +bool numa_promotion_tiered_enabled;#ifdef CONFIG_NUMA_BALANCING +/* +* If there is only one toptier node available, pages on that +* node can not be promotrd to anywhere. In that case, downgrade +* to numa_promotion_tiered_enabled mode +*/ +static void check_numa_promotion_mode (void ) +{ +int node, toptier_node_count =0 ; + +for_each_online_node (node) { +if (node_is_toptier (node)) + ++toptier_node_count; + } +if (toptier_node_count ==1 ) { + numa_promotion_tiered_enabled = true; + } +} +void set_numabalancing_state (bool enabled) {if (enabled)
@@ -3611,20 +3631,22 @@ void set_numabalancing_state(bool enabled)int sysctl_numa_balancing (struct ctl_table *table,int write,void *buffer,size_t *lenp,loff_t *ppos) { -struct ctl_table t;int err; -int state =static_branch_likely (&sched_numa_balancing);if (write && !capable (CAP_SYS_ADMIN ))return -EPERM ; - t = *table; - t.data = &state; - err =proc_dointvec_minmax (&t, write, buffer, lenp, ppos); + err =proc_dointvec_minmax (table, write, buffer, lenp, ppos);if (err <0 )return err; -if (write) -set_numabalancing_state (state); +if (write) { +if (sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) +check_numa_promotion_mode (); +else if (sysctl_numa_balancing_mode &NUMA_BALANCING_TIERED_MEMORY ) + numa_promotion_tiered_enabled = true; + +set_numabalancing_state (*(int *)table->data); + }return err; }#endif
kernel/sched/fair.c +9 -1
@@ -1424,7 +1424,7 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page,count_vm_numa_event (PGPROMOTE_CANDIDATE ); -if (flags &TNF_DEMOTED ) +if (numa_demotion_enabled && (flags &TNF_DEMOTED ))count_vm_numa_event (PGPROMOTE_CANDIDATE_DEMOTED );if (page_is_file_lru (page))
@@ -1435,6 +1435,14 @@ bool should_numa_migrate_memory(struct task_struct *p, struct page * page, this_cpupid =cpu_pid_to_cpupid (dst_cpu, current->pid); last_cpupid =page_cpupid_xchg_last (page, this_cpupid); +/* +* The pages in non-toptier memory node should be migrated +* according to hot/cold instead of accessing CPU node. +*/ +if (numa_promotion_tiered_enabled && !node_is_toptier (src_nid)) +return true; + +/* * Allow first faults or private faults to migrate immediately early in * the lifetime of a task. The magic number 4 is based on waiting for
kernel/sched/sched.h +1 -0
@@ -51,6 +51,7 @@#include <linux/kthread.h> #include <linux/membarrier.h> #include <linux/migrate.h> +#include <linux/mempolicy.h> #include <linux/mm_inline.h> #include <linux/mmu_context.h> #include <linux/nmi.h>
kernel/sysctl.c +4 -3
@@ -113,6 +113,7 @@ static int sixty = 60;static int __maybe_unused neg_one =-1 ;static int __maybe_unused two =2 ; +static int __maybe_unused three =3 ;static int __maybe_unused four =4 ;static unsigned long zero_ul;static unsigned long one_ul =1 ;
@@ -1840,12 +1841,12 @@ static struct ctl_table kern_table[] = { }, { .procname = "numa_balancing", - .data = NULL, /* filled in by handler */ - .maxlen = sizeof(unsigned int), + .data = &sysctl_numa_balancing_mode, + .maxlen = sizeof(int), .mode = 0644, .proc_handler = sysctl_numa_balancing, .extra1 = SYSCTL_ZERO, - .extra2 = SYSCTL_ONE, + .extra2 = &three, },#endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_SCHED_DEBUG */
mm/huge_memory.c +18 -9
@@ -22,6 +22,7 @@#include <linux/freezer.h> #include <linux/pfn_t.h> #include <linux/mman.h> +#include <linux/mempolicy.h> #include <linux/memremap.h> #include <linux/pagemap.h> #include <linux/debugfs.h>
@@ -1849,16 +1850,24 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, }#endif -/* -* Avoid trapping faults against the zero page. The read-only -* data is likely to be read-cached on the local CPU and -* local/remote hits to the zero page are not interesting. -*/ -if (prot_numa &&is_huge_zero_pmd (*pmd)) - goto unlock; +if (prot_numa) { +struct page *page; +/* +* Avoid trapping faults against the zero page. The read-only +* data is likely to be read-cached on the local CPU and +* local/remote hits to the zero page are not interesting. +*/ +if (is_huge_zero_pmd (*pmd)) + goto unlock; -if (prot_numa &&pmd_protnone (*pmd)) - goto unlock; +if (pmd_protnone (*pmd)) + goto unlock; + +/* skip scanning toptier node */ + page =pmd_page (*pmd); +if (numa_promotion_tiered_enabled &&node_is_toptier (page_to_nid (page))) + goto unlock; + }/* * In case prot_numa, we are under mmap_read_lock(mm). It's critical
mm/mprotect.c +7 -1
@@ -83,6 +83,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,*/ if (prot_numa) {struct page *page; +int nid;/* Avoid TLB flush if possible */ if (pte_protnone (oldpte))
@@ -109,7 +110,12 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,* Don't mess with PTEs if page is already on the node * a single-threaded process is running on. */ -if (target_node ==page_to_nid (page)) + nid =page_to_nid (page); +if (target_node == nid) +continue ; + +/* skip scanning toptier node */ +if (numa_promotion_tiered_enabled &&node_is_toptier (nid))continue ; }
Signature · 1 line
2.30.2
PATCH 3/5 Decouple reclaim and allocation for toptier nodes
In reply to Hasan Al MarufWith a tight memory constraint, we need to proactively keep some free memory in toptier node, such that 1) new allocation which is mainly for request processing can be directly put in the toptier node and 2) toptier node is able to accept hot pages promoted from non-toptier node. To achieve that, we decouple the reclamation and allocation mechanism, i.e. reclamation gets triggered at a different watermark -- WMARK_DEMOTE, while allocation checks for the traditional WMARK_HIGH. In this way, toptier nodes can maintain some free space to accept both new allocation and promotion from non-toptier nodes.
On each toptier memory node, kswapd daemon is woken up to demote memory when free memory on the node falls below the following fraction
demote_scale_factor/10000
The default value of demote_scale_factor is 200 , (i.e. 2%) so kswapd will be woken up when available free memory on a node falls below 2%. The demote_scale_factor can be adjusted higher if we need kswapd to keep more free memory around by updating the sysctl variable
/proc/sys/vm/demote_scale_factor
--- Documentation/admin-guide/sysctl/vm.rst | 12 +++++++++ include/linux/mempolicy.h | 5 ++++ include/linux/mm.h | 4 +++ include/linux/mmzone.h | 5 ++++ kernel/sched/fair.c | 3 +++ kernel/sysctl.c | 12 ++++++++- mm/mempolicy.c | 23 +++++++++++++++++ mm/page_alloc.c | 34 ++++++++++++++++++++++++- mm/vmscan.c | 26 +++++++++++++++++++ mm/vmstat.c | 7 ++++- 10 files changed, 128 insertions(+), 3 deletions(-)
Documentation/admin-guide/sysctl/vm.rst +12 -0
@@ -74,6 +74,7 @@ Currently, these files are in /proc/sys/vm: - vfs_cache_pressure - watermark_boost_factor - watermark_scale_factor +- demote_scale_factor - zone_reclaim_mode
@@ -961,6 +962,17 @@ that the number of free pages kswapd maintains for latency reasons is too small for the allocation bursts occurring in the system. This knob can then be used to tune kswapd aggressiveness accordingly. +demote_scale_factor +=================== + +This factor controls when kswapd wakes up to demote pages from toptier +nodes. It defines the amount of memory left in a toptier node/system +before kswapd is woken up and how much memory needs to be free from those +nodes before kswapd goes back to sleep. + +The unit is in fractions of 10,000. The default value of 200 means if there +are less than 2% of free toptier memory in a node/system, we will start to +demote pages from that node. zone_reclaim_mode =================
include/linux/mempolicy.h +5 -0
@@ -145,6 +145,7 @@ extern void numa_default_policy(void);extern void numa_policy_init (void );extern void mpol_rebind_task (struct task_struct *tsk,const nodemask_t *new);extern void mpol_rebind_mm (struct mm_struct *mm,nodemask_t *new); +extern void check_toptier_balanced (void ); extern int huge_node(struct vm_area_struct *vma, unsigned long addr, gfp_t gfp_flags,
@@ -299,6 +300,10 @@ static inline nodemask_t *policy_nodemask_current(gfp_t gfp) return NULL; } +static inline void check_toptier_balanced (void ) +{ +} +#define numa_demotion_enabled false#define numa_promotion_tiered_enabled false#endif /* CONFIG_NUMA */
include/linux/mm.h +4 -0
@@ -3153,6 +3153,10 @@ static inline bool debug_guardpage_enabled(void) { return false; }static inline bool page_is_guard (struct page *page) {return false; }#endif /* CONFIG_DEBUG_PAGEALLOC */ +#ifdef CONFIG_MIGRATION +extern int demote_scale_factor; +#endif +#if MAX_NUMNODES >1 void __init setup_nr_node_ids (void );#else
include/linux/mmzone.h +5 -0
@@ -329,12 +329,14 @@ enum zone_watermarks {WMARK_MIN ,WMARK_LOW ,WMARK_HIGH , +WMARK_DEMOTE ,NR_WMARK };#define min_wmark_pages (z) (z->_watermark[WMARK_MIN] + z->watermark_boost)#define low_wmark_pages (z) (z->_watermark[WMARK_LOW] + z->watermark_boost)#define high_wmark_pages (z) (z->_watermark[WMARK_HIGH] + z->watermark_boost) +#define demote_wmark_pages (z) (z->_watermark[WMARK_DEMOTE] + z->watermark_boost)#define wmark_pages (z, i) (z->_watermark[i] + z->watermark_boost) struct per_cpu_pages {
@@ -884,6 +886,7 @@ bool zone_watermark_ok(struct zone *z, unsigned int order, unsigned int alloc_flags);bool zone_watermark_ok_safe (struct zone *z,unsigned int order,unsigned long mark,int highest_zoneidx); +bool pgdat_toptier_balanced (pg_data_t *pgdat,int order,int zone_idx);/* * Memory initialization context, use to differentiate memory added by * the platform statically or via memory hotplug interface.
@@ -1011,6 +1014,8 @@ int min_free_kbytes_sysctl_handler(struct ctl_table *, int, void *, size_t *, loff_t *);int watermark_scale_factor_sysctl_handler (struct ctl_table *,int ,void *,size_t *,loff_t *); +int demote_scale_factor_sysctl_handler (struct ctl_table *,int ,void __user *, +size_t *,loff_t *);extern int sysctl_lowmem_reserve_ratio[MAX_NR_ZONES ];int lowmem_reserve_ratio_sysctl_handler (struct ctl_table *,int ,void *,size_t *,loff_t *);
kernel/sched/fair.c +3 -0
@@ -21,6 +21,8 @@* Copyright (C) 2007 Red Hat, Inc., Peter Zijlstra */ #include "sched.h" +#include <trace/events/sched.h> +#include <linux/mempolicy.h> /* * Targeted preemption latency for CPU-bound tasks:
@@ -10802,6 +10804,7 @@ void trigger_load_balance(struct rq *rq)raise_softirq (SCHED_SOFTIRQ );nohz_balancer_kick (rq); +check_toptier_balanced (); }static void rq_online_fair (struct rq *rq)
kernel/sysctl.c +11 -1
@@ -112,6 +112,7 @@ static int sixty = 60;#endif static int __maybe_unused neg_one =-1 ; +static int __maybe_unused one =1 ;static int __maybe_unused two =2 ;static int __maybe_unused three =3 ;static int __maybe_unused four =4 ;
@@ -121,8 +122,8 @@ static unsigned long long_max = LONG_MAX;static int one_hundred =100 ;static int two_hundred =200 ;static int one_thousand =1000 ; -#ifdef CONFIG_PRINTK static int ten_thousand =10000 ; +#ifdef CONFIG_PRINTK #endif #ifdef CONFIG_PERF_EVENTS static int six_hundred_forty_kb =640 *1024 ;
@@ -3000,6 +3001,15 @@ static struct ctl_table vm_table[] = { .extra1 = SYSCTL_ONE, .extra2 = &one_thousand, }, + { + .procname = "demote_scale_factor", + .data = &demote_scale_factor, + .maxlen = sizeof(demote_scale_factor), + .mode = 0644, + .proc_handler = demote_scale_factor_sysctl_handler, + .extra1 = &one, + .extra2 = &ten_thousand, + }, { .procname = "percpu_pagelist_fraction", .data = &percpu_pagelist_fraction,
mm/mempolicy.c +23 -0
@@ -1042,6 +1042,29 @@ static long do_get_mempolicy(int *policy, nodemask_t *nmask,return err; } +void check_toptier_balanced (void ) +{ +int nid; +int balanced; + +if (!numa_promotion_tiered_enabled) +return ; + +for_each_node_state (nid,N_MEMORY ) { +pg_data_t *pgdat =NODE_DATA (nid); + +if (!node_is_toptier (nid)) +continue ; + + balanced =pgdat_toptier_balanced (pgdat,0 ,ZONE_MOVABLE ); +if (!balanced) { + pgdat->kswapd_order =0 ; + pgdat->kswapd_highest_zoneidx =ZONE_NORMAL ; +wakeup_kswapd (pgdat->node_zones +ZONE_NORMAL ,0 ,1 ,ZONE_NORMAL ); + } + } +} +#ifdef CONFIG_MIGRATION /* * page migration, thp tail pages can be passed.
mm/page_alloc.c +33 -1
@@ -3599,7 +3599,8 @@ struct page *rmqueue(struct zone *preferred_zone,if (test_bit (ZONE_BOOSTED_WATERMARK , &zone->flags)) {clear_bit (ZONE_BOOSTED_WATERMARK , &zone->flags);wakeup_kswapd (zone,0 ,0 ,zone_idx (zone)); - } + }else if (!pgdat_toptier_balanced (zone->zone_pgdat, order,zone_idx (zone))) +wakeup_kswapd (zone,0 ,0 ,zone_idx (zone));VM_BUG_ON_PAGE (page &&bad_range (zone, page), page);return page;
@@ -8047,6 +8048,22 @@ static void __setup_per_zone_wmarks(void) zone->_watermark[WMARK_LOW ] =min_wmark_pages (zone) + tmp; zone->_watermark[WMARK_HIGH ] =min_wmark_pages (zone) + tmp *2 ; +if (numa_promotion_tiered_enabled) { + tmp =mult_frac (zone_managed_pages (zone), demote_scale_factor,10000 ); + +/* +* Clamp demote watermark between twice high watermark +* and max managed pages. +*/ +if (tmp <2 * zone->_watermark[WMARK_HIGH ]) + tmp =2 * zone->_watermark[WMARK_HIGH ]; +if (tmp >zone_managed_pages (zone)) + tmp =zone_managed_pages (zone); + zone->_watermark[WMARK_DEMOTE ] = tmp; + + zone->watermark_boost =0 ; + } +spin_unlock_irqrestore (&zone->lock, flags); }
@@ -8163,6 +8180,21 @@ int watermark_scale_factor_sysctl_handler(struct ctl_table *table, int write,return 0 ; } +int demote_scale_factor_sysctl_handler (struct ctl_table *table,int write, +void __user *buffer,size_t *length,loff_t *ppos) +{ +int rc; + + rc =proc_dointvec_minmax (table, write, buffer, length, ppos); +if (rc) +return rc; + +if (write) +setup_per_zone_wmarks (); + +return 0 ; +} +#ifdef CONFIG_NUMA static void setup_min_unmapped_ratio (void ) {
mm/vmscan.c +26 -0
@@ -41,6 +41,7 @@#include <linux/kthread.h> #include <linux/freezer.h> #include <linux/memcontrol.h> +#include <linux/mempolicy.h> #include <linux/migrate.h> #include <linux/delayacct.h> #include <linux/sysctl.h>
@@ -190,6 +191,7 @@ static void set_task_reclaim_state(struct task_struct *task,static LIST_HEAD (shrinker_list);static DECLARE_RWSEM (shrinker_rwsem); +int demote_scale_factor =200 ;#ifdef CONFIG_MEMCG /*
@@ -3598,6 +3600,30 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)return false; } +bool pgdat_toptier_balanced (pg_data_t *pgdat,int order,int zone_idx) +{ +unsigned long mark; +struct zone *zone; + +if (!node_is_toptier (pgdat->node_id) || + !numa_promotion_tiered_enabled || + order >0 || zone_idx <ZONE_NORMAL ) { +return true; + } + + zone = pgdat->node_zones +ZONE_NORMAL ; + +if (!managed_zone (zone)) +return true; + + mark =min (demote_wmark_pages (zone),zone_managed_pages (zone)); + +if (zone_page_state (zone,NR_FREE_PAGES ) < mark) +return false; + +return true; +} +/* Clear pgdat state for congested, dirty or under writeback. */ static void clear_pgdat_congested (pg_data_t *pgdat) {
mm/vmstat.c +6 -1
@@ -28,6 +28,7 @@#include <linux/mm_inline.h> #include <linux/page_ext.h> #include <linux/page_owner.h> +#include <linux/migrate.h> #include "internal.h"
@@ -1649,7 +1650,9 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat, struct zone *zone) {int i; -seq_printf (m,"Node %d, zone %8s" , pgdat->node_id, zone->name); +seq_printf (m,"Node %d, zone %8s, toptier %d next_demotion_node %d" , + pgdat->node_id, zone->name,node_is_toptier (pgdat->node_id), +next_demotion_node (pgdat->node_id));if (is_zone_first_populated (pgdat, zone)) {seq_printf (m,"\n per-node stats" );for (i =0 ; i <NR_VM_NODE_STAT_ITEMS ; i++) {
@@ -1666,6 +1669,7 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,"\n min %lu" "\n low %lu" "\n high %lu" +"\n demote %lu" "\n spanned %lu" "\n present %lu" "\n managed %lu"
@@ -1674,6 +1678,7 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,min_wmark_pages (zone),low_wmark_pages (zone),high_wmark_pages (zone), +node_is_toptier (pgdat->node_id) ?demote_wmark_pages (zone) :0 , zone->spanned_pages, zone->present_pages, zone_managed_pages(zone),
Signature · 1 line
2.30.2
PATCH 4/5 Reclaim to satisfy WMARK_DEMOTE on toptier nodes
In reply to Hasan Al MarufWhen kswapd is wakenup on a toptier node in a tiered-memory NUMA balancing mode, it reclaims pages until the toptier node is balanced and the number of free pages on toptier node satisfies WMARK_DEMOTE.
When THP (Transparent Huge Page) is enabled, sometimes demotion/promotion between the memory nodes may pause for several hundreds of seconds as the pages in the toptier node may sometimes become so hot, that kswapd fails to reclaim any page. Finally, the kswapd failure count (pgdat->kswapd_failures) reaches its max value and kswapd will not be waken up until a successful direct reclaiming. For general use case, this isn't a big problem as the memory users will do direct reclaim finally and trigger successful direct reclaiming or OOM to fix the issue. But in memory tiering system, the demotion and promotion will avoid to create too much memory pressure on the fast memory node, so direct reclaiming will not be triggered to resolve the issue. To resolve this, when promotion enabled, kswapd will be waken up every 10 seconds to try to free some pages to recover kswapd failures.
--- mm/vmscan.c | 42 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 36 insertions(+), 6 deletions(-)
mm/vmscan.c +36 -6
@@ -2386,8 +2386,14 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc,unsigned long ap, fp;enum lru_list lru; -/* If we have no swap space, do not bother scanning anon pages. */ -if (!sc->may_swap || !can_reclaim_anon_pages (memcg, pgdat->node_id, sc)) { +/* +* If we have no swap space, do not bother scanning anon pages. +* However, anon pages on toptier node can be demoted via reclaim +* when numa promotion is enabled. Disable the check to prevent +* demotion for no swap space when numa promotion is enabled. +*/ +if (!numa_promotion_tiered_enabled && + (!sc->may_swap || !can_reclaim_anon_pages (memcg, pgdat->node_id, sc))) { scan_balance =SCAN_FILE ; goto out; }
@@ -2916,7 +2922,10 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)if (!managed_zone (zone))continue ; - total_high_wmark +=high_wmark_pages (zone); +if (numa_promotion_tiered_enabled &&node_is_toptier (pgdat->node_id)) + total_high_wmark +=demote_wmark_pages (zone); +else + total_high_wmark +=high_wmark_pages (zone); }/*
@@ -3574,6 +3583,9 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)unsigned long mark =-1 ;struct zone *zone; +if (numa_promotion_tiered_enabled &&node_is_toptier (pgdat->node_id) && + highest_zoneidx >=ZONE_NORMAL ) +return pgdat_toptier_balanced (pgdat,0 , highest_zoneidx);/* * Check watermarks bottom-up as lower zones are more likely to * meet watermarks.
@@ -3692,7 +3704,10 @@ static bool kswapd_shrink_node(pg_data_t *pgdat,if (!managed_zone (zone))continue ; - sc->nr_to_reclaim +=max (high_wmark_pages (zone),SWAP_CLUSTER_MAX ); +if (numa_promotion_tiered_enabled &&node_is_toptier (pgdat->node_id)) + sc->nr_to_reclaim +=max (demote_wmark_pages (zone),SWAP_CLUSTER_MAX ); +else + sc->nr_to_reclaim +=max (high_wmark_pages (zone),SWAP_CLUSTER_MAX ); }/*
@@ -4021,8 +4036,23 @@ static void kswapd_try_to_sleep(pg_data_t *pgdat, int alloc_order, int reclaim_o*/ set_pgdat_percpu_threshold (pgdat, calculate_normal_threshold); -if (!kthread_should_stop ()) -schedule (); +if (!kthread_should_stop ()) { +/* +* In numa promotion modes, try harder to recover from +* kswapd failures, because direct reclaiming may be +* not triggered. +*/ +if (numa_promotion_tiered_enabled && +node_is_toptier (pgdat->node_id) && + pgdat->kswapd_failures >=MAX_RECLAIM_RETRIES ) { + remaining =schedule_timeout (10 *HZ ); +if (!remaining) { + pgdat->kswapd_highest_zoneidx =ZONE_MOVABLE ; + pgdat->kswapd_order =0 ; + } + }else +schedule (); + }set_pgdat_percpu_threshold (pgdat, calculate_pressure_threshold); }else {
Signature · 1 line
2.30.2
PATCH 5/5 active LRU-based promotion to avoid ping-pong
In reply to Hasan Al MarufWhenever a remote hint-fault happens on a page, the default NUMA balancing promotes the page without checking its state. As a result, cold pages with very infrequent accesses, can still be the promotion candidates. Once promoted to the local node, these type of pages may shortly become the demotion candidate if the toptier nodes are always under pressure. Thus, promotion traffics generated from infrequently accessed pages can easily fill up the toptier node's reclaimed free spaces and eventually generate a higher demotion traffic for the non- toptier node. This demotion-promotion ping-pong causes unnecessary traffic over the memory nodes and can negatively impact on the performance of memory bound applications.
To solve this ping-pong issue, instead of instant promotion, we check a page's age through its position in the LRU list. If the faulted page is in inactive LRU, then we don't instantly consider the page as the promotion candidate as it might be an infrequently accessed pages. We only consider the faulted pages that are in the active LRUs (either of anon or file active LRU) as the promotion candidate. This approach significantly reduce the promotion traffic and always maintain a satisfactory amount of free memory on the toptier node to support both new allocations and promotion from non-tortier nodes.
--- mm/memory.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
mm/memory.c +13 -0
@@ -4202,6 +4202,19 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) last_cpupid =page_cpupid_last (page); page_nid =page_to_nid (page); + +/* Only migrate pages that are active on non-toptier node */ +if (numa_promotion_tiered_enabled && + !node_is_toptier (page_nid) && + !PageActive (page)) { +count_vm_numa_event (NUMA_HINT_FAULTS ); +if (page_nid ==numa_node_id ()) +count_vm_numa_event (NUMA_HINT_FAULTS_LOCAL ); +mark_page_accessed (page); +pte_unmap_unlock (vmf->pte, vmf->ptl); + goto out; + } + target_nid =numa_migrate_prep (page, vma, vmf->address, page_nid, &flags);pte_unmap_unlock (vmf->pte, vmf->ptl);
Signature · 1 line
2.30.2
Quoted 47 lines · Hasan Al Maruf
[resend in proper format]
With the advent of new memory types and technologies, we can see different types of memory together, e.g. DRAM, PMEM, CXL-enabled memory, etc. In recent future, we can see CXL-Memory be available in the physical address- space as a CPU-less NUMA node along with the native DDR memory channels. As different types of memory have different level of performance impact, how we manage pages across the NUMA nodes should be a matter of concern.
Dave Hansen's patchset on "Migrate Pages in lieu of discard" demotes toptier pages to a slow tier node during the reclamation process.
https://lwn.net/Articles/860215/However, that patchset does not include the features to promote pages on slow tier memory node to the toptier one. As a result, pages demoted or newly allocated on the slow tier node, experiences NUMA latency and hurt application performance. In this patch set, we augment existing AutoNUMA mechanism to promote pages from slow tier nodes to toptier nodes.
We decouple reclamation and allocation logics for the toptier node so that reclamation gets triggered at a higher watermark and demotes colder pages to the slow-tier memory. As a result, toptier nodes can maintain some free space to accept both new allocation and promotion from slowtier nodes. During promotion, we add hysteresis to page and only promote pages that are less likely to be demoted within a short period of time. This reduces the chance for a page being ping-ponged across the NUMA nodes due to frequent demotion and promotion within a short period of time.
We tested this patchset on systems with CXL-enabled DRAM and PMEM tiers. We find this patchset can bring hotter pages to the toptier node while moving the colder pages to the slow-tier nodes for a good range of Meta production workloads with live traffic. As a result, toptier nodes serve more hot pages and the application performance improves.
Case Study of a Meta cache application with two NUMA nodes
==========================================================
Toptier node: DRAM directly attached to the CPU Slowtier node: DRAM attached through CXLToptier vs Slowtier memory capacity ratio is 1:4
With default page placement policy, file caches fills up the toptier node and anons get trapped in the slowtier node. Only 14% of the total anons reside in toptier node. Remote NUMA read bandwidth is 80%. Throughput regression is 18% compared to all memory being served from toptier node.
Quoted 6 lines
This patchset brings 80% of the anons to the toptier node. Anons on the slowtier memory is mostly cold anons. As the toptier node can not host all the hot memory, some hot files still remain on the slowtier node. Even though, remote NUMA read bandwidth reduces from 80% to 40%. With this patchset, throughput regression is only 5% compared to the baseline of toptier node serving the whole working set.
Hi, Hasan,
I found that quite some code in your patchset is exactly same as that in my patchset as follows,
https://lore.kernel.org/lkml/[email protected]/
and patches in the following repo we used to publish some patchset that hasn't been sent to community for review,
https://git.kernel.org/pub/scm/linux/kernel/git/vishal/tiering.git/log/?h=tiering-0.72
I am glad that more people have interest and worked on optimizing page placement for tiering memory system. How about we merge instead of duplicate our effort?
Because I tried to make the patches above as simple as possible (at least first 3), can you comment the most basic patches there to help them to be improved. And then we can build our more complex/advanced patches on top of that?
Best Regards,
Huang, Ying
Hi Huang,
We find the patches in the tiering series are well thought and helpful. For our workloads, we initially started with that series and we find the whole series is too complex and some features do not benefit as expected. Therefore, we have come up with the current basic patches which are essential and help achieve most of the intended behaviors while reducing complexity as much as possible.
As we started with your tiering series (with 72 patches), there are overlaps between our patches and the tiering series. We adopt the functionalities from the tiering series, modify, and extend them to make page placement mechanism simpler but workable. Here is the key points for each of the patches in our Transparent Page Placement series.
Patch #1:
We combine all the promotion and demotion related statistics in this patch Having statistics on both promotion, demotion, and failures help observe the systems behavior and reason about performance behavior. Besides, anon vs file breakdown in both promotion and demotion path help understand application behavior on a tiered memory systems. As applications may have different sensitivity toward the anon and file placements, this breakdown in the migration path is often helpful to assess the effectiveness of the page placement policy.
Patch #2:
This patch largely overlaps with your current series on NUMA Balancing.
https://lore.kernel.org/lkml/[email protected]/
This patch is a combination of your Patch #2 and Patch #3 except the static 10MB free space in the top-tier node to maintain a free headroom for new allocation and promotion. Rather, we find having a user defined demote watermark would make it more generic that we include in our patch#3
Patch #3:
This patch has the logic for having a separate demote watermark per node. In the tiering series, that demote watermark is somewhat bound to the cgroup and triggered on per-application basis. Besides, It only supports cgroup-v1. However, we think, instead of cgroup based soft reclamation, a global per-node demote watermark is more meaningful and should be the basic one to start with. In that case, the user does not have to think about per-application setup.
Patch #4:
This patch includes the code for kswapd based reclamation. As I mentioned earlier, instead of cgroup-based reclamation, here we look whether a node is balanced during each kswapd invocation. For top-tier node, we check whether kswapd reclaimed till DEMOTE_WMARK is satisfied, for other nodes the default mechanism continues. The differences between tiering series and this patch is the cgroup based reclamation vs per-node reclamation.
Patch #5:
In your patches for promotion, you consider re-fault time for promotion candidate selection. Although the hot-threshold is tunable, from our experiments, we find this not helpful to some extent. For example, if different subset of pages have different re-access time, time-based promotion should not be able to distinguish between them. If you make the time window long enough, then any infrequently accessed pages will also become the promotion candidate, and later be a candidate for the demotion.
In this patch, we propose LRU based promotion, which would give anon and files different promotion paths. If pages are used sporadically at high frequency, irregular pages would be eventually moved from the active LRU list. We find that our LRU based approach can reduce up to 11x promotion traffic while retaining the same application throughput for multiple workloads.
Besides, with promotion rate limit, if files largely get promoted to top-tier, anon promotion rate often gets hampered as files are taking the large portion of the total rate (which often happen for applications that generates huge caches). In our LRU-based approach, each type has their own separate LRU to check. So for workloads with smaller anons and large file usage, with LRU-based approach, we can see more anons are being promoted rather than the files.
I don't mind this patchset being merged to your current patchset under discussion or any later ones. But, I think this series contains the very basic functionalities to have a workable page placement mechanism for tiered-memory. This can obviously be augmented by the other features in you future tiering series.
Best,
Hasan
Hi, Hasan,
Quoted 12 lines · Hasan Al Maruf
Hi Huang,
We find the patches in the tiering series are well thought and helpful. For our workloads, we initially started with that series and we find the whole series is too complex and some features do not benefit as expected. Therefore, we have come up with the current basic patches which are essential and help achieve most of the intended behaviors while reducing complexity as much as possible.
As we started with your tiering series (with 72 patches), there are overlaps between our patches and the tiering series. We adopt the functionalities from the tiering series, modify, and extend them to make page placement mechanism simpler but workable.
Thanks for the background!
Quoted 63 lines
Here is the key points for
each of the patches in our Transparent Page Placement series.Patch #1:
We combine all the promotion and demotion related statistics in this patch Having statistics on both promotion, demotion, and failures help observe the systems behavior and reason about performance behavior. Besides, anon vs file breakdown in both promotion and demotion path help understand application behavior on a tiered memory systems. As applications may have different sensitivity toward the anon and file placements, this breakdown in the migration path is often helpful to assess the effectiveness of the page placement policy.Patch #2:
This patch largely overlaps with your current series on NUMA Balancing.
https://lore.kernel.org/lkml/[email protected]/
This patch is a combination of your Patch #2 and Patch #3 except the static 10MB free space in the top-tier node to maintain a free headroom for new allocation and promotion. Rather, we find having a user defined demote watermark would make it more generic that we include in our patch#3Patch #3:
This patch has the logic for having a separate demote watermark per node. In the tiering series, that demote watermark is somewhat bound to the cgroup and triggered on per-application basis. Besides, It only supports cgroup-v1. However, we think, instead of cgroup based soft reclamation, a global per-node demote watermark is more meaningful and should be the basic one to start with. In that case, the user does not have to think about per-application setup.Patch #4:
This patch includes the code for kswapd based reclamation. As I mentioned earlier, instead of cgroup-based reclamation, here we look whether a node is balanced during each kswapd invocation. For top-tier node, we check whether kswapd reclaimed tillDEMOTE_WMARKis satisfied, for other nodes the default mechanism continues. The differences between tiering series and this patch is the cgroup based reclamation vs per-node reclamation.Patch #5:
In your patches for promotion, you consider re-fault time for promotion candidate selection. Although the hot-threshold is tunable, from our experiments, we find this not helpful to some extent. For example, if different subset of pages have different re-access time, time-based promotion should not be able to distinguish between them. If you make the time window long enough, then any infrequently accessed pages will also become the promotion candidate, and later be a candidate for the demotion.In this patch, we propose LRU based promotion, which would give anon and files different promotion paths. If pages are used sporadically at high frequency, irregular pages would be eventually moved from the active LRU list. We find that our LRU based approach can reduce up to 11x promotion traffic while retaining the same application throughput for multiple workloads.
Besides, with promotion rate limit, if files largely get promoted to top-tier, anon promotion rate often gets hampered as files are taking the large portion of the total rate (which often happen for applications that generates huge caches). In our LRU-based approach, each type has their own separate LRU to check. So for workloads with smaller anons and large file usage, with LRU-based approach, we can see more anons are being promoted rather than the files.
Quoted 5 lines
I don't mind this patchset being merged to your current patchset under discussion or any later ones. But, I think this series contains the very basic functionalities to have a workable page placement mechanism for tiered-memory. This can obviously be augmented by the other features in you future tiering series.
Thanks for detailed description! After reading your patchset and the description above, I found that the basic part ([1/6] - [3/6]) of the promotion patchset as follows can be the base for your patchset too.
https://lore.kernel.org/lkml/[email protected]/
The main problem of that basic patchset is lacking review. Can I ask you to help to review that patchset, especially the common base [1/6] - [3/6]? If you think the rest of the patchset isn't good enough for you, we can try to merge just [1/6] - [3/6] firstly. Do you agree?
Best Regards,
Huang, Ying
Hi Huang,
Quoted 5 lines
Thanks for detailed description! After reading your patchset and the description above, I found that the basic part ([1/6] - [3/6]) of the promotion patchset as follows can be the base for your patchset too.
Yeah, I agree with you. The first 3 patches of the above series are pretty much the base of this series. We can add patch [3/5]-[5/5] after your 3 patches. Then add the patch [1/5] to include the statistics counters.
Quoted 4 lines
The main problem of that basic patchset is lacking review. Can I ask you to help to review that patchset, especially the common base [1/6] - [3/6]? If you think the rest of the patchset isn't good enough for you, we can try to merge just [1/6] - [3/6] firstly. Do you agree?
Sounds good to me. I would add my reviews within the next couple of days.
Thanks,
Hasan
Thanks,
Lu, Zhixing
ask for commit hash
In reply to Hasan Al MarufHi, I'm trying to apply these patches in linux kernel v5.15 but meet some conflicts, could you please tell me which commit hash did you start?
Thanks,
Lu, Zhixing