NUMA balancing: optimize memory placement for memory tiering system
Subsystems (11)
cover NUMA balancing: optimize memory placement for memory tiering system
The changes since the last post are as follows,
- Rebased on v5.17-rc1
- Remove [4-6] from the original patchset to make it easier to be reviewed.
- Change the additional promotion watermark to be the high watermark / 4.
--
With the advent of various new memory types, some machines will have multiple types of memory, e.g. DRAM and PMEM (persistent memory). The memory subsystem of these machines can be called memory tiering system, because the performance of the different types of memory are different.
After commit c221c0b0308f ("device-dax: "Hotplug" persistent memory for use like normal RAM"), the PMEM could be used as the cost-effective volatile memory in separate NUMA nodes. In a typical memory tiering system, there are CPUs, DRAM and PMEM in each physical NUMA node. The CPUs and the DRAM will be put in one logical node, while the PMEM will be put in another (faked) logical node.
To optimize the system overall performance, the hot pages should be placed in DRAM node. To do that, we need to identify the hot pages in the PMEM node and migrate them to DRAM node via NUMA migration.
In the original NUMA balancing, there are already a set of existing mechanisms to identify the pages recently accessed by the CPUs in a node and migrate the pages to the node. So we can reuse these mechanisms to build the mechanisms to optimize the page placement in the memory tiering system. This is implemented in this patchset.
At the other hand, the cold pages should be placed in PMEM node. So, we also need to identify the cold pages in the DRAM node and migrate them to PMEM node.
In commit 26aa2d199d6f ("mm/migrate: demote pages during reclaim"), a mechanism to demote the cold DRAM pages to PMEM node under memory pressure is implemented. Based on that, the cold DRAM pages can be demoted to PMEM node proactively to free some memory space on DRAM node to accommodate the promoted hot PMEM pages. This is implemented in this patchset too.
We have tested the solution with the pmbench memory accessing benchmark with the 80:20 read/write ratio and the Gauss access address distribution on a 2 socket Intel server with Optane DC Persistent Memory Model. The test results shows that the pmbench score can improve up to 95.9%.
Changelog:
v11:
- Rebased on v5.17-rc1
- Remove [4-6] from the original patchset to make it easier to be reviewed.
- Change the additional promotion watermark to be the high watermark / 4.
v10:
- Rebased on v5.16-rc1
- Revise error processing for [1/6] (promotion counter) per Yang's comments
- Add sysctl document for [2/6] (optimize page placement)
- Reset threshold adjustment state when disable/enable tiering mode
- Reset threshold when workload transition is detected.
v9:
- Rebased on v5.15-rc4
- Make "add promotion counter" the first patch per Yang's comments
v8:
- Rebased on v5.15-rc1
- Make user-specified threshold take effect sooner
v7:
- Rebased on the mmots tree of 2021-07-15.
- Some minor fixes.
v6:
- Rebased on the latest page demotion patchset. (which bases on v5.11)
v5:
- Rebased on the latest page demotion patchset. (which bases on v5.10)
v4:
- Rebased on the latest page demotion patchset. (which bases on v5.9-rc6)
- Add page promotion counter.
v3:
- Move the rate limit control as late as possible per Mel Gorman's comments.
- Revise the hot page selection implementation to store page scan time in struct page.
- Code cleanup.
- Rebased on the latest page demotion patchset.
v2:
- Addressed comments for V1.
- Rebased on v5.5.
Best Regards,
Huang, Ying
PATCH 1/3 NUMA Balancing: add page promotion counter
In a system with multiple memory types, e.g. DRAM and PMEM, the CPU and DRAM in one socket will be put in one NUMA node as before, while the PMEM will be put in another NUMA node as described in the description of the commit c221c0b0308f ("device-dax: "Hotplug" persistent memory for use like normal RAM"). So, the NUMA balancing mechanism will identify all PMEM accesses as remote access and try to promote the PMEM pages to DRAM.
To distinguish the number of the inter-type promoted pages from that of the inter-socket migrated pages. A new vmstat count is added. The counter is per-node (count in the target node). So this can be used to identify promotion imbalance among the NUMA nodes.
--- include/linux/mmzone.h | 3 +++ include/linux/node.h | 5 +++++ mm/migrate.c | 13 ++++++++++--- mm/vmstat.c | 3 +++ 4 files changed, 21 insertions(+), 3 deletions(-)
include/linux/mmzone.h +3 -0
@@ -210,6 +210,9 @@ enum node_stat_item { NR_PAGETABLE, /* used for pagetables */#ifdef CONFIG_SWAP NR_SWAPCACHE, +#endif +#ifdef CONFIG_NUMA_BALANCING +PGPROMOTE_SUCCESS ,/* promote successfully */ #endifNR_VM_NODE_STAT_ITEMS };
include/linux/node.h +5 -0
@@ -181,4 +181,9 @@ static inline void register_hugetlbfs_with_node(node_registration_func_t reg,#define to_node (device) container_of(device, struct node, dev) +static inline bool node_is_toptier (int node) +{ +return node_state (node,N_CPU ); +} +#endif /* _LINUX_NODE_H_ */
mm/migrate.c +10 -3
@@ -2072,6 +2072,7 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,pg_data_t *pgdat =NODE_DATA (node);int isolated;int nr_remaining; +int nr_succeeded;LIST_HEAD (migratepages);new_page_t *new;bool compound;
@@ -2110,7 +2111,8 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,list_add (&page->lru, &migratepages); nr_remaining =migrate_pages (&migratepages, *new,NULL , node, -MIGRATE_ASYNC ,MR_NUMA_MISPLACED ,NULL ); +MIGRATE_ASYNC ,MR_NUMA_MISPLACED , + &nr_succeeded);if (nr_remaining) {if (!list_empty (&migratepages)) {list_del (&page->lru);
@@ -2119,8 +2121,13 @@ int migrate_misplaced_page(struct page *page, struct vm_area_struct *vma,putback_lru_page (page); } isolated =0 ; - }else -count_vm_numa_events (NUMA_PAGE_MIGRATE , nr_pages); + } +if (nr_succeeded) { +count_vm_numa_events (NUMA_PAGE_MIGRATE , nr_succeeded); +if (!node_is_toptier (page_to_nid (page)) &&node_is_toptier (node)) +mod_node_page_state (NODE_DATA (node),PGPROMOTE_SUCCESS , + nr_succeeded); + }BUG_ON (!list_empty (&migratepages));return isolated;
mm/vmstat.c +3 -0
@@ -1242,6 +1242,9 @@ const char * const vmstat_text[] = {#ifdef CONFIG_SWAP "nr_swapcached",#endif +#ifdef CONFIG_NUMA_BALANCING +"pgpromote_success" , +#endif/* enum writeback_stat_item counters */ "nr_dirty_threshold",
Signature · 1 line
2.30.2
Re: [PATCH -V11 1/3] NUMA Balancing: add page promotion counter
Quoted 22 lines · Huang Ying
In a system with multiple memory types, e.g. DRAM and PMEM, the CPU and DRAM in one socket will be put in one NUMA node as before, while the PMEM will be put in another NUMA node as described in the description of the commit
c221c0b0308f("device-dax: "Hotplug" persistent memory for use like normal RAM"). So, the NUMA balancing mechanism will identify all PMEM accesses as remote access and try to promote the PMEM pages to DRAM.To distinguish the number of the inter-type promoted pages from that of the inter-socket migrated pages. A new vmstat count is added. The counter is per-node (count in the target node). So this can be used to identify promotion imbalance among the NUMA nodes.
Signed-off-by "Huang, Ying" <[email protected]>Reviewed-by Yang Shi <[email protected]>Tested-by Baolin Wang <[email protected]>Reviewed-by Baolin Wang <[email protected]>Cc Andrew Morton <[email protected]>Cc Michal Hocko <[email protected]>Cc Rik van Riel <[email protected]>Cc Mel Gorman <[email protected]>Cc Peter Zijlstra <[email protected]>
Quoted 8 lines
Cc Dave Hansen <[email protected]>Cc Zi Yan <[email protected]>Cc Wei Xu <[email protected]>Cc osalvador <[email protected]>Cc Shakeel Butt <[email protected]>Cc zhongjiang-ali <[email protected]>
PATCH 2/3 NUMA balancing: optimize page placement for memory tiering system
In reply to Huang YingWith the advent of various new memory types, some machines will have multiple types of memory, e.g. DRAM and PMEM (persistent memory). The memory subsystem of these machines can be called memory tiering system, because the performance of the different types of memory are usually different.
In such system, because of the memory accessing pattern changing etc, some pages in the slow memory may become hot globally. So in this patch, the NUMA balancing mechanism is enhanced to optimize the page placement among the different memory types according to hot/cold dynamically.
In a typical memory tiering system, there are CPUs, fast memory and slow memory in each physical NUMA node. The CPUs and the fast memory will be put in one logical node (called fast memory node), while the slow memory will be put in another (faked) logical node (called slow memory node). That is, the fast memory is regarded as local while the slow memory is regarded as remote. So it's possible for the recently accessed pages in the slow memory node to be promoted to the fast memory node via the existing NUMA balancing mechanism.
The original NUMA balancing mechanism will stop to migrate pages if the free memory of the target node becomes below the high watermark. This is a reasonable policy if there's only one memory type. But this makes the original NUMA balancing mechanism almost do not work to optimize page placement among different memory types. Details are as follows.
It's the common cases that the working-set size of the workload is larger than the size of the fast memory nodes. Otherwise, it's unnecessary to use the slow memory at all. So, there are almost always no enough free pages in the fast memory nodes, so that the globally hot pages in the slow memory node cannot be promoted to the fast memory node. To solve the issue, we have 2 choices as follows,
a. Ignore the free pages watermark checking when promoting hot pages
from the slow memory node to the fast memory node. This will create some memory pressure in the fast memory node, thus trigger the memory reclaiming. So that, the cold pages in the fast memory node will be demoted to the slow memory node.
b. Make kswapd of the fast memory node to reclaim pages until the
free pages are a little (for example, high_watermark / 4) more than the high watermark. Then, if the free pages of the fast memory node reaches high watermark, and some hot pages need to be promoted, kswapd of the fast memory node will be waken up to demote more cold pages in the fast memory node to the slow memory node. This will free some extra space in the fast memory node, so the hot pages in the slow memory node can be promoted to the fast memory node.
The choice "a" may create high memory pressure in the fast memory node. If the memory pressure of the workload is high, the memory pressure may become so high that the memory allocation latency of the workload is influenced, e.g. the direct reclaiming may be triggered.
The choice "b" works much better at this aspect. If the memory pressure of the workload is high, the hot pages promotion will stop earlier because its allocation watermark is higher than that of the normal memory allocation. So in this patch, choice "b" is implemented.
In addition to the original page placement optimization among sockets, the NUMA balancing mechanism is extended to be used to optimize page placement according to hot/cold among different memory types. So the sysctl user space interface (numa_balancing) is extended in a backward compatible way as follow, so that the users can enable/disable these functionality individually.
The sysctl is converted from a Boolean value to a bits field. The definition of the flags is,
- 0x0: NUMA_BALANCING_DISABLED
- 0x1: NUMA_BALANCING_NORMAL
- 0x2: NUMA_BALANCING_MEMORY_TIERING
We have tested the patch with the pmbench memory accessing benchmark with the 80:20 read/write ratio and the Gauss access address distribution on a 2 socket Intel server with Optane DC Persistent Memory Model. The test results shows that the pmbench score can improve up to 95.9%.
--- Documentation/admin-guide/sysctl/kernel.rst | 29 ++++++++++++++------- include/linux/sched/sysctl.h | 10 +++++++ kernel/sched/core.c | 21 ++++++++++++--- kernel/sysctl.c | 2 +- mm/migrate.c | 19 ++++++++++++-- mm/vmscan.c | 17 ++++++++++++ 6 files changed, 82 insertions(+), 16 deletions(-)
Documentation/admin-guide/sysctl/kernel.rst +20 -9
@@ -595,16 +595,23 @@ Documentation/admin-guide/kernel-parameters.rst). numa_balancing ============== -Enables/disables automatic page fault based NUMA memory -balancing. Memory is moved automatically to nodes -that access it often. +Enables/disables and configure automatic page fault based NUMA memory +balancing. Memory is moved automatically to nodes that access it +often. The value to set can be the result to OR the following, -Enables/disables automatic NUMA memory balancing. On NUMA machines, there -is a performance penalty if remote memory is accessed by a CPU. When this -feature is enabled the kernel samples what task thread is accessing memory -by periodically unmapping pages and later trapping a page fault. At the -time of the page fault, it is determined if the data being accessed should -be migrated to a local memory node. += ================================= +0x0 NUMA_BALANCING_DISABLED +0x1 NUMA_BALANCING_NORMAL +0x2 NUMA_BALANCING_MEMORY_TIERING += ================================= + +Or NUMA_BALANCING_NORMAL to optimize page placement among different +NUMA nodes to reduce remote accessing. On NUMA machines, there is a +performance penalty if remote memory is accessed by a CPU. When this +feature is enabled the kernel samples what task thread is accessing +memory by periodically unmapping pages and later trapping a page +fault. At the time of the page fault, it is determined if the data +being accessed should be migrated to a local memory node. The unmapping of pages and trapping faults incur additional overhead that ideally is offset by improved memory locality but there is no universal
@@ -615,6 +622,10 @@ faults may be controlled by the `numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb`_, and numa_balancing_settle_count sysctls. +Or NUMA_BALANCING_MEMORY_TIERING to optimize page placement among +different types of memory (represented as different NUMA nodes) to +place the hot pages in the fast memory. This is implemented based on +unmapping and page fault too. numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb ===============================================================================================================================
include/linux/sched/sysctl.h +10 -0
@@ -23,6 +23,16 @@ enum sched_tunable_scaling { SCHED_TUNABLESCALING_END, }; +#define NUMA_BALANCING_DISABLED 0x0 +#define NUMA_BALANCING_NORMAL 0x1 +#define NUMA_BALANCING_MEMORY_TIERING 0x2 + +#ifdef CONFIG_NUMA_BALANCING +extern int sysctl_numa_balancing_mode; +#else +#define sysctl_numa_balancing_mode 0 +#endif +/* * control realtime throttling: *
kernel/sched/core.c +17 -4
@@ -4279,7 +4279,9 @@ DEFINE_STATIC_KEY_FALSE(sched_numa_balancing);#ifdef CONFIG_NUMA_BALANCING -void set_numabalancing_state (bool enabled) +int sysctl_numa_balancing_mode; + +static void __set_numabalancing_state (bool enabled) {if (enabled)static_branch_enable (&sched_numa_balancing);
@@ -4287,13 +4289,22 @@ void set_numabalancing_state(bool enabled)static_branch_disable (&sched_numa_balancing); } +void set_numabalancing_state (bool enabled) +{ +if (enabled) + sysctl_numa_balancing_mode =NUMA_BALANCING_NORMAL ; +else + sysctl_numa_balancing_mode =NUMA_BALANCING_DISABLED ; +__set_numabalancing_state (enabled); +} +#ifdef CONFIG_PROC_SYSCTL int sysctl_numa_balancing (struct ctl_table *table,int write,void *buffer,size_t *lenp,loff_t *ppos) {struct ctl_table t;int err; -int state =static_branch_likely (&sched_numa_balancing); +int state = sysctl_numa_balancing_mode;if (write && !capable (CAP_SYS_ADMIN ))return -EPERM ;
@@ -4303,8 +4314,10 @@ int sysctl_numa_balancing(struct ctl_table *table, int write, err =proc_dointvec_minmax (&t, write, buffer, lenp, ppos);if (err <0 )return err; -if (write) -set_numabalancing_state (state); +if (write) { + sysctl_numa_balancing_mode = state; +__set_numabalancing_state (state); + } return err; }#endif
kernel/sysctl.c +1 -1
@@ -1689,7 +1689,7 @@ static struct ctl_table kern_table[] = { .mode = 0644, .proc_handler = sysctl_numa_balancing, .extra1 = SYSCTL_ZERO, - .extra2 = SYSCTL_ONE, + .extra2 = SYSCTL_FOUR, },#endif /* CONFIG_NUMA_BALANCING */ {
mm/migrate.c +17 -2
@@ -51,6 +51,7 @@#include <linux/oom.h> #include <linux/memory.h> #include <linux/random.h> +#include <linux/sched/sysctl.h> #include <asm/tlbflush.h>
@@ -2034,16 +2035,30 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) {int page_lru;int nr_pages =thp_nr_pages (page); +int order =compound_order (page); -VM_BUG_ON_PAGE (compound_order (page) && !PageTransHuge (page), page); +VM_BUG_ON_PAGE (order && !PageTransHuge (page), page);/* Do not migrate THP mapped by multiple processes */ if (PageTransHuge (page) &&total_mapcount (page) >1 )return 0 ;/* Avoid migrating to a node that is nearly full */ -if (!migrate_balanced_pgdat (pgdat, nr_pages)) +if (!migrate_balanced_pgdat (pgdat, nr_pages)) { +int z; + +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_MEMORY_TIERING ) || + !numa_demotion_enabled) +return 0 ; +if (next_demotion_node (pgdat->node_id) ==NUMA_NO_NODE ) +return 0 ; +for (z = pgdat->nr_zones -1 ; z >=0 ; z--) { +if (populated_zone (pgdat->node_zones + z)) +break ; + } +wakeup_kswapd (pgdat->node_zones + z,0 , order,ZONE_MOVABLE );return 0 ; + }if (isolate_lru_page (page))return 0 ;
mm/vmscan.c +17 -0
@@ -56,6 +56,7 @@#include <linux/swapops.h> #include <linux/balloon_compaction.h> +#include <linux/sched/sysctl.h> #include "internal.h"
@@ -3966,6 +3967,13 @@ static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx)return false; } +/* +* Keep the free pages on fast memory node a little more than the high +* watermark to accommodate the promoted pages. +*/ +#define NUMA_BALANCING_PROMOTE_WATERMARK_DIV 4 +#define NUMA_BALANCING_PROMOTE_WATERMARK_MIN (10UL * 1024 * 1024 >> PAGE_SHIFT) +/* * Returns true if there is an eligible zone balanced for the request order * and highest_zoneidx
@@ -3987,6 +3995,15 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx)continue ; mark =high_wmark_pages (zone); +if (sysctl_numa_balancing_mode &NUMA_BALANCING_MEMORY_TIERING && + numa_demotion_enabled && +next_demotion_node (pgdat->node_id) !=NUMA_NO_NODE ) { +unsigned long promote_mark; + + promote_mark =max (NUMA_BALANCING_PROMOTE_WATERMARK_MIN , + mark /NUMA_BALANCING_PROMOTE_WATERMARK_DIV ); + mark += promote_mark; + }if (zone_watermark_ok_safe (zone, order, mark, highest_zoneidx))return true; }
Signature · 1 line
2.30.2
Johannes Weiner
maintainerRe: [PATCH -V11 2/3] NUMA balancing: optimize page placement for memory tiering system
Hi Huang,
Quoted 28 lines · Huang Ying
It's the common cases that the working-set size of the workload is larger than the size of the fast memory nodes. Otherwise, it's unnecessary to use the slow memory at all. So, there are almost always no enough free pages in the fast memory nodes, so that the globally hot pages in the slow memory node cannot be promoted to the fast memory node. To solve the issue, we have 2 choices as follows,
a. Ignore the free pages watermark checking when promoting hot pages
from the slow memory node to the fast memory node. This will create some memory pressure in the fast memory node, thus trigger the memory reclaiming. So that, the cold pages in the fast memory node will be demoted to the slow memory node.b. Make kswapd of the fast memory node to reclaim pages until the
free pages are a little (for example,high_watermark/ 4) more than the high watermark. Then, if the free pages of the fast memory node reaches high watermark, and some hot pages need to be promoted, kswapd of the fast memory node will be waken up to demote more cold pages in the fast memory node to the slow memory node. This will free some extra space in the fast memory node, so the hot pages in the slow memory node can be promoted to the fast memory node.The choice "a" may create high memory pressure in the fast memory node. If the memory pressure of the workload is high, the memory pressure may become so high that the memory allocation latency of the workload is influenced, e.g. the direct reclaiming may be triggered.
Quoted 5 lines
The choice "b" works much better at this aspect. If the memory pressure of the workload is high, the hot pages promotion will stop earlier because its allocation watermark is higher than that of the normal memory allocation. So in this patch, choice "b" is implemented.
I agree with that choice.
It's conceivable we'd eventually want a mix of both, where promotions boost kswapd watermarks on-demand, such that no fast memory is wasted if there are no references into slow memory. But that can be done when it proves necessary, IMO.
Quoted 34 lines
(quoted diff) +16
-9@@ -595,16 +595,23 @@ Documentation/admin-guide/kernel-parameters.rst). numa_balancing ============== -Enables/disables automatic page fault based NUMA memory -balancing. Memory is moved automatically to nodes -that access it often. +Enables/disables and configure automatic page fault based NUMA memory +balancing. Memory is moved automatically to nodes that access it +often. The value to set can be the result to OR the following, -Enables/disables automatic NUMA memory balancing. On NUMA machines, there -is a performance penalty if remote memory is accessed by a CPU. When this -feature is enabled the kernel samples what task thread is accessing memory -by periodically unmapping pages and later trapping a page fault. At the -time of the page fault, it is determined if the data being accessed should -be migrated to a local memory node. += ================================= +0x0 NUMA_BALANCING_DISABLED +0x1 NUMA_BALANCING_NORMAL +0x2 NUMA_BALANCING_MEMORY_TIERING += ================================= + +Or NUMA_BALANCING_NORMAL to optimize page placement among different +NUMA nodes to reduce remote accessing. On NUMA machines, there is a +performance penalty if remote memory is accessed by a CPU. When this +feature is enabled the kernel samples what task thread is accessing +memory by periodically unmapping pages and later trapping a page +fault. At the time of the page fault, it is determined if the data +being accessed should be migrated to a local memory node. The unmapping of pages and trapping faults incur additional overhead that ideally is offset by improved memory locality but there is no universal
Review faults may be controlled by the `numa_balancing_scan_period_min_ms, · line 628
Quoted 7 lines
(quoted diff) +4
-0@@ -615,6 +622,10 @@ faults may be controlled by the `numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb`_, and numa_balancing_settle_count sysctls. +Or NUMA_BALANCING_MEMORY_TIERING to optimize page placement among +different types of memory (represented as different NUMA nodes) to +place the hot pages in the fast memory. This is implemented based on +unmapping and page fault too.
NORMAL | TIERING appears to be a non-sensical combination.
Would it be better to have a tristate (disabled, normal, tiering) rather than a mask?
Quoted 13 lines
(quoted diff) +2
-1@@ -2034,16 +2035,30 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) { int page_lru; int nr_pages = thp_nr_pages(page); + int order = compound_order(page); - VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page); + VM_BUG_ON_PAGE(order && !PageTransHuge(page), page); /* Do not migrate THP mapped by multiple processes */ if (PageTransHuge(page) && total_mapcount(page) > 1) return 0;
Review static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) · line 2054
Quoted 10 lines
(quoted diff) +8
-1/* Avoid migrating to a node that is nearly full */ - if (!migrate_balanced_pgdat(pgdat, nr_pages)) + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { + int z; + + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || + !numa_demotion_enabled) + return 0; + if (next_demotion_node(pgdat->node_id) == NUMA_NO_NODE) + return 0;
The encoded behavior doesn't seem very user-friendly: Unless the user enables numa demotion in a separate flag, enabling numa balancing in tiered mode will silently do nothing.
Would it make more sense to have a central flag for the operation of tiered memory systems that will enable both promotion and demotion?
Alternatively, it could also ignore the state of demotion and promote anyway if asked to, resulting in regular reclaim to make room. It might not be the most popular combination, but would be in line with the zone_reclaim_mode policy of preferring reclaim over remote accesses. It would make the knobs behave more as expected and it's less convoluted than having flags select other user-visible flags.
Quoted 17 lines
(quoted diff) +7
-0@@ -3966,6 +3967,13 @@ static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx) return false; } +/* + * Keep the free pages on fast memory node a little more than the high + * watermark to accommodate the promoted pages. + */ +#define NUMA_BALANCING_PROMOTE_WATERMARK_DIV 4 +#define NUMA_BALANCING_PROMOTE_WATERMARK_MIN (10UL * 1024 * 1024 >> PAGE_SHIFT) + /* * Returns true if there is an eligible zone balanced for the request order * and highest_zoneidx@@ -3987,6 +3995,15 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx) continue;
Review static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx) · line 4006
Quoted 10 lines
(quoted diff) +9
-0mark = high_wmark_pages(zone); + if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && + numa_demotion_enabled && + next_demotion_node(pgdat->node_id) != NUMA_NO_NODE) { + unsigned long promote_mark; + + promote_mark = max(NUMA_BALANCING_PROMOTE_WATERMARK_MIN, + mark / NUMA_BALANCING_PROMOTE_WATERMARK_DIV); + mark += promote_mark; + }
I'm not sure about this formula.
The high watermark straddles both the atomic allocation pool (min_free_kbytes) as well as the direct reclaim latency buffer (watermark_scale_factor) on top of it. Making the promotion buffer a quarter of the raw total number conflates the two settings.
E.g. if the user grows the atomic pool to support bursty network rx, the two direct reclaim buffers between min, low and high stay the same, but the tier-promotion buffer grows. This is unexpected, and not what the user intended.
I'm thinking it would be better to do this:
mm/page_alloc.c +2 -1
@@ -8573,7 +8573,8 @@ static void __setup_per_zone_wmarks(void) zone->watermark_boost =0 ; zone->_watermark[WMARK_LOW ] =min_wmark_pages (zone) + tmp; - zone->_watermark[WMARK_HIGH ] =min_wmark_pages (zone) + tmp *2 ; + zone->_watermark[WMARK_HIGH ] =low_wmark_pages (zone) + tmp; + zone->_watermark[WMARK_PROMO ] =high_wmark_pages (zone) + tmp;spin_unlock_irqrestore (&zone->lock, flags); }
and then have kswapd choose between the high and promo watermarks depending on the numa balancing mode.
On my 32G desktop with default settings this produces the same ballpark of values as your code - high/4=7424, tmp=7311.
This would behave more intuitively with the existing watermark tuning knobs, and I think would make the code overall simpler as well.
Re: [PATCH -V11 2/3] NUMA balancing: optimize page placement for memory tiering system
Hi, Johannes,
Thanks a lot for your review!
Quoted 37 lines · Johannes Weiner
Hi Huang,
Quoted 28 lines · Huang Ying
It's the common cases that the working-set size of the workload is larger than the size of the fast memory nodes. Otherwise, it's unnecessary to use the slow memory at all. So, there are almost always no enough free pages in the fast memory nodes, so that the globally hot pages in the slow memory node cannot be promoted to the fast memory node. To solve the issue, we have 2 choices as follows,
a. Ignore the free pages watermark checking when promoting hot pages
from the slow memory node to the fast memory node. This will create some memory pressure in the fast memory node, thus trigger the memory reclaiming. So that, the cold pages in the fast memory node will be demoted to the slow memory node.b. Make kswapd of the fast memory node to reclaim pages until the
free pages are a little (for example,high_watermark/ 4) more than the high watermark. Then, if the free pages of the fast memory node reaches high watermark, and some hot pages need to be promoted, kswapd of the fast memory node will be waken up to demote more cold pages in the fast memory node to the slow memory node. This will free some extra space in the fast memory node, so the hot pages in the slow memory node can be promoted to the fast memory node.The choice "a" may create high memory pressure in the fast memory node. If the memory pressure of the workload is high, the memory pressure may become so high that the memory allocation latency of the workload is influenced, e.g. the direct reclaiming may be triggered.
Quoted 5 lines
The choice "b" works much better at this aspect. If the memory pressure of the workload is high, the hot pages promotion will stop earlier because its allocation watermark is higher than that of the normal memory allocation. So in this patch, choice "b" is implemented.
Quoted 6 lines
I agree with that choice.
It's conceivable we'd eventually want a mix of both, where promotions boost kswapd watermarks on-demand, such that no fast memory is wasted if there are no references into slow memory. But that can be done when it proves necessary, IMO.
Sure.
Quoted 44 lines
Quoted 34 lines
(quoted diff) +16
-9@@ -595,16 +595,23 @@ Documentation/admin-guide/kernel-parameters.rst). numa_balancing ============== -Enables/disables automatic page fault based NUMA memory -balancing. Memory is moved automatically to nodes -that access it often. +Enables/disables and configure automatic page fault based NUMA memory +balancing. Memory is moved automatically to nodes that access it +often. The value to set can be the result to OR the following, -Enables/disables automatic NUMA memory balancing. On NUMA machines, there -is a performance penalty if remote memory is accessed by a CPU. When this -feature is enabled the kernel samples what task thread is accessing memory -by periodically unmapping pages and later trapping a page fault. At the -time of the page fault, it is determined if the data being accessed should -be migrated to a local memory node. += ================================= +0x0 NUMA_BALANCING_DISABLED +0x1 NUMA_BALANCING_NORMAL +0x2 NUMA_BALANCING_MEMORY_TIERING += ================================= + +Or NUMA_BALANCING_NORMAL to optimize page placement among different +NUMA nodes to reduce remote accessing. On NUMA machines, there is a +performance penalty if remote memory is accessed by a CPU. When this +feature is enabled the kernel samples what task thread is accessing +memory by periodically unmapping pages and later trapping a page +fault. At the time of the page fault, it is determined if the data +being accessed should be migrated to a local memory node. The unmapping of pages and trapping faults incur additional overhead that ideally is offset by improved memory locality but there is no universalReview faults may be controlled by the `numa_balancing_scan_period_min_ms, · line 628
Quoted 7 lines
(quoted diff) +4
-0@@ -615,6 +622,10 @@ faults may be controlled by the `numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb`_, and numa_balancing_settle_count sysctls. +Or NUMA_BALANCING_MEMORY_TIERING to optimize page placement among +different types of memory (represented as different NUMA nodes) to +place the hot pages in the fast memory. This is implemented based on +unmapping and page fault too.NORMAL | TIERING appears to be a non-sensical combination.
Quoted 2 lines
Would it be better to have a tristate (disabled, normal, tiering) rather than a mask?
NORMAL is for balancing cross-socket memory accessing among DRAM nodes. TIERING is for optimizing page placement between DRAM and PMEM in one socket. We think it's possible to do both.
For example, with [3/3] of the patchset,
- TIERING: because DRAM pages aren't made PROT_NONE, it's disabled to balance among DRAM nodes.
- NORMAL | TIERING: both cross-socket balancing among DRAM nodes and page placement optimizing between DRAM and PMEM are enabled.
Quoted 13 lines
Quoted 13 lines
(quoted diff) +2
-1@@ -2034,16 +2035,30 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) { int page_lru; int nr_pages = thp_nr_pages(page); + int order = compound_order(page); - VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page); + VM_BUG_ON_PAGE(order && !PageTransHuge(page), page); /* Do not migrate THP mapped by multiple processes */ if (PageTransHuge(page) && total_mapcount(page) > 1) return 0;
Quoted 14 lines
Review static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) · line 2054
Quoted 10 lines
(quoted diff) +8
-1/* Avoid migrating to a node that is nearly full */ - if (!migrate_balanced_pgdat(pgdat, nr_pages)) + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { + int z; + + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || + !numa_demotion_enabled) + return 0; + if (next_demotion_node(pgdat->node_id) == NUMA_NO_NODE) + return 0;The encoded behavior doesn't seem very user-friendly: Unless the user enables numa demotion in a separate flag, enabling numa balancing in tiered mode will silently do nothing.
In theory, TIERING still does something even with numa_demotion_enabled == false. Where it works more like the original NUMA balancing. If there's some free space in DRAM node (for example, some programs exit), some PMEM pages will be promoted to DRAM. But as in the change log, this isn't good enough for page placement optimizing.
Quoted 2 lines
Would it make more sense to have a central flag for the operation of tiered memory systems that will enable both promotion and demotion?
IMHO, it may be possible for people to enable demotion alone. For example, if some people want to use a user space page placement optimizing solution based on PMU counters, they may disable TIERING, but still use demotion as a way to avoid swapping in some situation. Do you think this makes sense?
Quoted 6 lines
Alternatively, it could also ignore the state of demotion and promote anyway if asked to, resulting in regular reclaim to make room. It might not be the most popular combination, but would be in line with the
zone_reclaim_modepolicy of preferring reclaim over remote accesses. It would make the knobs behave more as expected and it's less convoluted than having flags select other user-visible flags.
Sorry, I don't get your idea here. Do you suggest to add another knob like zone_relcaim_mode? Then we can define some bit to control demotion and promotion there? If so, I still don't know how to fit this into the existing NUMA balancing framework.
Quoted 57 lines
Quoted 17 lines
(quoted diff) +7
-0@@ -3966,6 +3967,13 @@ static bool pgdat_watermark_boosted(pg_data_t *pgdat, int highest_zoneidx) return false; } +/* + * Keep the free pages on fast memory node a little more than the high + * watermark to accommodate the promoted pages. + */ +#define NUMA_BALANCING_PROMOTE_WATERMARK_DIV 4 +#define NUMA_BALANCING_PROMOTE_WATERMARK_MIN (10UL * 1024 * 1024 >> PAGE_SHIFT) + /* * Returns true if there is an eligible zone balanced for the request order * and highest_zoneidx@@ -3987,6 +3995,15 @@ static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx) continue;Review static bool pgdat_balanced(pg_data_t *pgdat, int order, int highest_zoneidx) · line 4006
Quoted 10 lines
(quoted diff) +9
-0mark = high_wmark_pages(zone); + if (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING && + numa_demotion_enabled && + next_demotion_node(pgdat->node_id) != NUMA_NO_NODE) { + unsigned long promote_mark; + + promote_mark = max(NUMA_BALANCING_PROMOTE_WATERMARK_MIN, + mark / NUMA_BALANCING_PROMOTE_WATERMARK_DIV); + mark += promote_mark; + }I'm not sure about this formula.
The high watermark straddles both the atomic allocation pool (
min_free_kbytes) as well as the direct reclaim latency buffer (watermark_scale_factor) on top of it. Making the promotion buffer a quarter of the raw total number conflates the two settings.E.g. if the user grows the atomic pool to support bursty network rx, the two direct reclaim buffers between min, low and high stay the same, but the tier-promotion buffer grows. This is unexpected, and not what the user intended.
I'm thinking it would be better to do this:
mm/page_alloc.c +2
-1@@ -8573,7 +8573,8 @@ static void __setup_per_zone_wmarks(void) zone->watermark_boost =0 ; zone->_watermark[WMARK_LOW ] =min_wmark_pages (zone) + tmp; - zone->_watermark[WMARK_HIGH ] =min_wmark_pages (zone) + tmp *2 ; + zone->_watermark[WMARK_HIGH ] =low_wmark_pages (zone) + tmp; + zone->_watermark[WMARK_PROMO ] =high_wmark_pages (zone) + tmp;spin_unlock_irqrestore (&zone->lock, flags); }
Quoted 8 lines
and then have kswapd choose between the high and promo watermarks depending on the numa balancing mode.
On my 32G desktop with default settings this produces the same ballpark of values as your code - high/4=7424, tmp=7311.
This would behave more intuitively with the existing watermark tuning knobs, and I think would make the code overall simpler as well.
Yes! This looks great! Will do this in the next version, Thanks!
Best Regards,
Huang, Ying
Johannes Weiner
maintainerRe: [PATCH -V11 2/3] NUMA balancing: optimize page placement for memory tiering system
Hi Huang,
Sorry, I didn't see this reply until you sent out the new version already :( Apologies.
Quoted 19 lines · Ying
Quoted 12 lines · Huang Ying
Review faults may be controlled by the `numa_balancing_scan_period_min_ms, · line 628
Quoted 8 lines
(quoted diff) +4
-0@@ -615,6 +622,10 @@ faults may be controlled by the `numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb`_, and numa_balancing_settle_count sysctls. +Or NUMA_BALANCING_MEMORY_TIERING to optimize page placement among +different types of memory (represented as different NUMA nodes) to +place the hot pages in the fast memory. This is implemented based on +unmapping and page fault too.NORMAL | TIERING appears to be a non-sensical combination.
Quoted 2 lines
Would it be better to have a tristate (disabled, normal, tiering) rather than a mask?
NORMAL is for balancing cross-socket memory accessing among DRAM nodes. TIERING is for optimizing page placement between DRAM and PMEM in one socket. We think it's possible to do both.
Quoted 7 lines
For example, with [3/3] of the patchset,
- TIERING: because DRAM pages aren't made
PROT_NONE, it's disabled to balance among DRAM nodes.- NORMAL | TIERING: both cross-socket balancing among DRAM nodes and page placement optimizing between DRAM and PMEM are enabled.
Ok, I get it. So NORMAL would enable PROT_NONE sampling on all nodes, and TIERING would additionally raise the watermarks on DRAM nodes.
Thanks!
Quoted 28 lines
Quoted 13 lines
Quoted 13 lines
(quoted diff) +2
-1@@ -2034,16 +2035,30 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) { int page_lru; int nr_pages = thp_nr_pages(page); + int order = compound_order(page); - VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page); + VM_BUG_ON_PAGE(order && !PageTransHuge(page), page); /* Do not migrate THP mapped by multiple processes */ if (PageTransHuge(page) && total_mapcount(page) > 1) return 0;Quoted 14 lines
Review static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) · line 2054
Quoted 10 lines
(quoted diff) +8
-1/* Avoid migrating to a node that is nearly full */ - if (!migrate_balanced_pgdat(pgdat, nr_pages)) + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { + int z; + + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || + !numa_demotion_enabled) + return 0; + if (next_demotion_node(pgdat->node_id) == NUMA_NO_NODE) + return 0;The encoded behavior doesn't seem very user-friendly: Unless the user enables numa demotion in a separate flag, enabling numa balancing in tiered mode will silently do nothing.
Quoted 5 lines
In theory, TIERING still does something even with
numa_demotion_enabled== false. Where it works more like the original NUMA balancing. If there's some free space in DRAM node (for example, some programs exit), some PMEM pages will be promoted to DRAM. But as in the change log, this isn't good enough for page placement optimizing.
Right, so it's a behavior that likely isn't going to be useful.
Quoted 8 lines
Quoted 2 lines
Would it make more sense to have a central flag for the operation of tiered memory systems that will enable both promotion and demotion?
IMHO, it may be possible for people to enable demotion alone. For example, if some people want to use a user space page placement optimizing solution based on PMU counters, they may disable TIERING, but still use demotion as a way to avoid swapping in some situation. Do you think this makes sense?
Yes, it does.
Quoted 11 lines
Quoted 6 lines
Alternatively, it could also ignore the state of demotion and promote anyway if asked to, resulting in regular reclaim to make room. It might not be the most popular combination, but would be in line with the
zone_reclaim_modepolicy of preferring reclaim over remote accesses. It would make the knobs behave more as expected and it's less convoluted than having flags select other user-visible flags.Sorry, I don't get your idea here. Do you suggest to add another knob like
zone_relcaim_mode? Then we can define some bit to control demotion and promotion there? If so, I still don't know how to fit this into the existing NUMA balancing framework.
No, I'm just suggesting to remove the !numa_demotion_disabled check from the promotion path on unbalanced nodes. Keep the switches independent from each other.
Like you said, demotion without promotion can be a valid config with a userspace promoter.
And I'm saying promotion without demotion can be a valid config in a zone_reclaim_mode type of setup.
We also seem to agree degraded promotion when demotion enabled likely isn't very useful to anybody. So maybe it should be removed?
It just comes down to user expectations. There is no masterswitch that says "do the right thing on tiered systems", so absent of that I think it would be best to keep the semantics of each of the two knobs simple and predictable, without tricky interdependencies - like quietly degrading promotion behavior when demotion is disabled.
Does that make sense?
Thanks!
Johannes
Re: [PATCH -V11 2/3] NUMA balancing: optimize page placement for memory tiering system
Quoted 4 lines · Johannes Weiner
Hi Huang,
Sorry, I didn't see this reply until you sent out the new version already :( Apologies.
Never mind!
Quoted 105 lines · Ying
Quoted 19 lines
Quoted 12 lines · Huang Ying
Review faults may be controlled by the `numa_balancing_scan_period_min_ms, · line 628
Quoted 8 lines
(quoted diff) +4
-0@@ -615,6 +622,10 @@ faults may be controlled by the `numa_balancing_scan_period_min_ms, numa_balancing_scan_delay_ms, numa_balancing_scan_period_max_ms, numa_balancing_scan_size_mb`_, and numa_balancing_settle_count sysctls. +Or NUMA_BALANCING_MEMORY_TIERING to optimize page placement among +different types of memory (represented as different NUMA nodes) to +place the hot pages in the fast memory. This is implemented based on +unmapping and page fault too.NORMAL | TIERING appears to be a non-sensical combination.
Quoted 2 lines
Would it be better to have a tristate (disabled, normal, tiering) rather than a mask?
NORMAL is for balancing cross-socket memory accessing among DRAM nodes. TIERING is for optimizing page placement between DRAM and PMEM in one socket. We think it's possible to do both.
Quoted 7 lines
For example, with [3/3] of the patchset,
- TIERING: because DRAM pages aren't made
PROT_NONE, it's disabled to balance among DRAM nodes.- NORMAL | TIERING: both cross-socket balancing among DRAM nodes and page placement optimizing between DRAM and PMEM are enabled.
Ok, I get it. So NORMAL would enable
PROT_NONEsampling on all nodes, and TIERING would additionally raise the watermarks on DRAM nodes.Thanks!
Quoted 28 lines
Quoted 13 lines
Quoted 13 lines
(quoted diff) +2
-1@@ -2034,16 +2035,30 @@ static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) { int page_lru; int nr_pages = thp_nr_pages(page); + int order = compound_order(page); - VM_BUG_ON_PAGE(compound_order(page) && !PageTransHuge(page), page); + VM_BUG_ON_PAGE(order && !PageTransHuge(page), page); /* Do not migrate THP mapped by multiple processes */ if (PageTransHuge(page) && total_mapcount(page) > 1) return 0;Quoted 14 lines
Review static int numamigrate_isolate_page(pg_data_t *pgdat, struct page *page) · line 2054
Quoted 10 lines
(quoted diff) +8
-1/* Avoid migrating to a node that is nearly full */ - if (!migrate_balanced_pgdat(pgdat, nr_pages)) + if (!migrate_balanced_pgdat(pgdat, nr_pages)) { + int z; + + if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) || + !numa_demotion_enabled) + return 0; + if (next_demotion_node(pgdat->node_id) == NUMA_NO_NODE) + return 0;The encoded behavior doesn't seem very user-friendly: Unless the user enables numa demotion in a separate flag, enabling numa balancing in tiered mode will silently do nothing.
Quoted 5 lines
In theory, TIERING still does something even with
numa_demotion_enabled== false. Where it works more like the original NUMA balancing. If there's some free space in DRAM node (for example, some programs exit), some PMEM pages will be promoted to DRAM. But as in the change log, this isn't good enough for page placement optimizing.Right, so it's a behavior that likely isn't going to be useful.
Quoted 8 lines
Quoted 2 lines
Would it make more sense to have a central flag for the operation of tiered memory systems that will enable both promotion and demotion?
IMHO, it may be possible for people to enable demotion alone. For example, if some people want to use a user space page placement optimizing solution based on PMU counters, they may disable TIERING, but still use demotion as a way to avoid swapping in some situation. Do you think this makes sense?
Yes, it does.
Quoted 11 lines
Quoted 6 lines
Alternatively, it could also ignore the state of demotion and promote anyway if asked to, resulting in regular reclaim to make room. It might not be the most popular combination, but would be in line with the
zone_reclaim_modepolicy of preferring reclaim over remote accesses. It would make the knobs behave more as expected and it's less convoluted than having flags select other user-visible flags.Sorry, I don't get your idea here. Do you suggest to add another knob like
zone_relcaim_mode? Then we can define some bit to control demotion and promotion there? If so, I still don't know how to fit this into the existing NUMA balancing framework.No, I'm just suggesting to remove the !numa_demotion_disabled check from the promotion path on unbalanced nodes. Keep the switches independent from each other.
Like you said, demotion without promotion can be a valid config with a userspace promoter.
And I'm saying promotion without demotion can be a valid config in a
zone_reclaim_modetype of setup.We also seem to agree degraded promotion when demotion enabled likely isn't very useful to anybody. So maybe it should be removed?
Quoted 7 lines
It just comes down to user expectations. There is no masterswitch that says "do the right thing on tiered systems", so absent of that I think it would be best to keep the semantics of each of the two knobs simple and predictable, without tricky interdependencies - like quietly degrading promotion behavior when demotion is disabled.
Does that make sense?
Yes. It does. I will do that in the next version!
Best Regards,
Huang, Ying
PATCH 3/3 memory tiering: skip to scan fast memory
In reply to Huang YingIf the NUMA balancing isn't used to optimize the page placement among sockets but only among memory types, the hot pages in the fast memory node couldn't be migrated (promoted) to anywhere. So it's unnecessary to scan the pages in the fast memory node via changing their PTE/PMD mapping to be PROT_NONE. So that the page faults could be avoided too.
In the test, if only the memory tiering NUMA balancing mode is enabled, the number of the NUMA balancing hint faults for the DRAM node is reduced to almost 0 with the patch. While the benchmark score doesn't change visibly.
--- mm/huge_memory.c | 30 +++++++++++++++++++++--------- mm/mprotect.c | 13 ++++++++++++- 2 files changed, 33 insertions(+), 10 deletions(-)
mm/huge_memory.c +21 -9
@@ -34,6 +34,7 @@#include <linux/oom.h> #include <linux/numa.h> #include <linux/page_owner.h> +#include <linux/sched/sysctl.h> #include <asm/tlb.h> #include <asm/pgalloc.h>
@@ -1766,17 +1767,28 @@ int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, }#endif -/* -* Avoid trapping faults against the zero page. The read-only -* data is likely to be read-cached on the local CPU and -* local/remote hits to the zero page are not interesting. -*/ -if (prot_numa &&is_huge_zero_pmd (*pmd)) - goto unlock; +if (prot_numa) { +struct page *page; +/* +* Avoid trapping faults against the zero page. The read-only +* data is likely to be read-cached on the local CPU and +* local/remote hits to the zero page are not interesting. +*/ +if (is_huge_zero_pmd (*pmd)) + goto unlock; -if (prot_numa &&pmd_protnone (*pmd)) - goto unlock; +if (pmd_protnone (*pmd)) + goto unlock; + page =pmd_page (*pmd); +/* +* Skip scanning top tier node if normal numa +* balancing is disabled +*/ +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && +node_is_toptier (page_to_nid (page))) + goto unlock; + }/* * In case prot_numa, we are under mmap_read_lock(mm). It's critical * to not clear pmd intermittently to avoid race with MADV_DONTNEED
mm/mprotect.c +12 -1
@@ -29,6 +29,7 @@#include <linux/uaccess.h> #include <linux/mm_inline.h> #include <linux/pgtable.h> +#include <linux/sched/sysctl.h> #include <asm/cacheflush.h> #include <asm/mmu_context.h> #include <asm/tlbflush.h>
@@ -83,6 +84,7 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,*/ if (prot_numa) {struct page *page; +int nid;/* Avoid TLB flush if possible */ if (pte_protnone (oldpte))
@@ -109,7 +111,16 @@ static unsigned long change_pte_range(struct vm_area_struct *vma, pmd_t *pmd,* Don't mess with PTEs if page is already on the node * a single-threaded process is running on. */ -if (target_node ==page_to_nid (page)) + nid =page_to_nid (page); +if (target_node == nid) +continue ; + +/* +* Skip scanning top tier node if normal numa +* balancing is disabled +*/ +if (!(sysctl_numa_balancing_mode &NUMA_BALANCING_NORMAL ) && +node_is_toptier (nid))continue ; }
Signature · 1 line
2.30.2
Re: [PATCH -V11 3/3] memory tiering: skip to scan fast memory
Quoted 21 lines · Huang Ying
If the NUMA balancing isn't used to optimize the page placement among sockets but only among memory types, the hot pages in the fast memory node couldn't be migrated (promoted) to anywhere. So it's unnecessary to scan the pages in the fast memory node via changing their PTE/PMD mapping to be
PROT_NONE. So that the page faults could be avoided too.In the test, if only the memory tiering NUMA balancing mode is enabled, the number of the NUMA balancing hint faults for the DRAM node is reduced to almost 0 with the patch. While the benchmark score doesn't change visibly.
Signed-off-by "Huang, Ying" <[email protected]>Suggested-by Dave Hansen <[email protected]>Tested-by Baolin Wang <[email protected]>Reviewed-by Baolin Wang <[email protected]>Cc Andrew Morton <[email protected]>Cc Michal Hocko <[email protected]>Cc Rik van Riel <[email protected]>Cc Mel Gorman <[email protected]>Cc Peter Zijlstra <[email protected]>
Quoted 8 lines
Cc Yang Shi <[email protected]>Cc Zi Yan <[email protected]>Cc Wei Xu <[email protected]>Cc osalvador <[email protected]>Cc Shakeel Butt <[email protected]>Cc zhongjiang-ali <[email protected]>
Looks good to me.