mm: Hot page tracking and promotion infrastructure
Subsystems (12)
v3 cover mm: Hot page tracking and promotion infrastructure
[If someone wants to be off the CC-list, pls drop me a note. Will remove from the next iteration]
Hi,
This patchset introduces a new subsystem for hot page tracking and promotion (pghot) with the following goals:
- Unify hot page detection from multiple sources like hint faults, page table scans, hardware hints (IBS).
- Decouple detection from migration.
- Centralize promotion logic via per-lowertier-node kernel thread.
- Move migration rate limiting and associated logic in NUMAB=2 (current NUMA Balancing based hot page promotion) from scheduler to pghot sub-system to enable broader reuse.
Currently, multiple kernel subsystems detect page accesses independently. This patchset consolidates accesses from these mechanisms by providing:
- A common API for reporting page accesses
- Shared infrastructure for tracking hotness at PFN granularity
- Per-lowertier-node kernel threads for promoting pages.
Here is a brief summary of how this subsystem works:
- Tracks frequency, last access time and accessing node for each recorded access.
- These hotness parameters are maintained on a per-PFN in an unsigned long variable within the existing mem_section data structure.
Bits 0-31 are used to store nid, frequency and time.
Bits 32-62 are unused now.
Bit 63 is used to indicate the page is ready for migration.
- Classifies pages as hot based on configurable thresholds.
- Pages classified as hot are marked as ready for migration using the ready bit.
- Per-lowertier-node kmigrated threads periodically scan the PFNs of lower tier nodes, checking for the migration-ready bit to perform batched migrations.
Three page hotness sources have been integrated with pghot subsystem on experimental basis:
1. IBS
2. klruscand (based on MGLRU page table walks)
3. NUMA Balancing (mode 2).
Major change in v3
==================
The major design change in this version is to move away from the hash and heap based hot page records management and instead use statically allocated per-PFN unsigned long variable for storing hotness parameters. This was the approach that I had used in what was called the kmigrated patchset [1]. While earlier I had used extended page flags, here mem_section data structure is used to store per-PFN hotness information for PFNs spanning the section.
Advantages of this approach:
- Eliminates the need for dynamic allocation and deallocation of hot page records. Also, no more atomic context allocations.
- Removes the requirement for special data structures (like hash lists and heap) to manage hot page records.
- Considerable space savings per hot page record (Just an unsigned long now instead of 40 bytes per record in the earlier approach)
- Fixed complexity for looking up the hot page record of a PFN.
- No locking complexity but just atomic updates to per-PFN record.
Downsides:
- Not easily possible to obtain top N hot pages list but a kernel thread will periodically scan the hotness records of its corresponding lower tier to obtain the hot pages for promotion.
- A page may become cold by the time kmigrated gets to act on it.
Space overhead:
- One pointer overhead for each memory section to store hotness array pointer. With a section size of 128MB resulting in 8192 sections per TB of node memory, there will be 64KB of memory used per TB. Currently I am using mem_section to store the hotness array pointer instead of creating a parallel data structure. If the latter method is preferred, then hotness array pointers are required only for the lower tier nodes.
- With 4K PFNs, there can be 32768 PFNs in a section and hence with 8 bytes (unsigned long) per PFN, hotness array will consume 2GB per TB of node memory. This will be for lower tier nodes only.
Other changes in v3
===================
- Migration thread is renamed to kmigrated (earlier called kpromoted).
- Most code cleanups as suggested by Jonathan Cameron.
- NUMAB mode 2 is now fully enabled as hotness source to pghot sub-system with off-loading of large pages migration to kmigrated.
- Sysctl knobs to enable access recording from different sources independently.
Results
=======
System details
--------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)
$ numactl -H available: 3 nodes (0-2) node 0 cpus: 0-95,192-287 node 0 size: 128460 MB node 1 cpus: 96-191,288-383 node 1 size: 128893 MB node 2 cpus: node 2 size: 257993 MB node distances: node 0 1 2 0: 10 32 50 1: 32 10 60 2: 255 255 10
Microbenchmark details
----------------------
Multi-threaded application with 64 threads that access memory at 4K granularity repetitively and randomly. The number of accesses per thread and the randomness pattern for each thread are fixed beforehand. The accesses are divided into stores and loads.
Benchmark threads run on Node 0, while memory is initially provisioned on CXL node 2 before the accesses start. There are three modes in which the benchmark is run:
Mode 1: Regular 4K page accesses. The memory is provisioned on CXL node using mmap(MAP_POPULATE). 50% loads and 50% stores.
Mode 2: mmapped file 4K accesses. The memory is provisioned on CXL node using mmap(fd, MAP_POPULATE|MAP_SHARED). 100% loads.
Mode 3: 2M THP page accesses. The memory is provisioned on CXL node using mmap, madvise(MADV_HUGEPAGE) and move_pages(to cxl node). 50% loads and 50% stores.
Repetitive accesses results in lowertier pages becoming hot and kmigrated detecting and migrating them. The benchmark score is the time taken to finish the accesses in microseconds. The sooner it finishes the better it is. All the numbers shown below are average of 3 runs.
Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the patched case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the patched case.
pgtscan - Klruscand (MGLRU based PTE A bit scanning) source hwhints - IBS as source
Results summary
---------------
Performance Impact:
- NUMAB2: 4.5% regression in Mode 1 and 19.8% regression in Mode 2.
- Hardware hints (IBS): Shows close to original NUMAB2 performance.
- Page table scanning: Good performance, comprehensive migration.
Migration Effectiveness:
- NUMAB2 and pgtscan achieve similar migration counts to baseline.
- THP migration significantly improved with new sources.
- Hardware hints show some sampling limitations.
Mode 1 - Time taken (microseconds, lower is better)
------------------------------------------------------
Source Base Patched Change
------------------------------------------------------
NUMAB0 115,668,771 117,775,032 +1.8%
NUMAB2 102,894,589 107,576,615 +4.5%
pgtscan NA 111,399,698 NA
hwhints NA 103,232,152 NA
------------------------------------------------------
Mode 1 - Pages migrated (pgpromote_success)
------------------------------------------------------
Source Base Patched Change
------------------------------------------------------
NUMAB0 0 0 0%
NUMAB2 2097144 2097152 +0.0%
pgtscan NA 2097152 NA
hwhints NA 1269467 NA
------------------------------------------------------
Mode 2 - Time taken (microseconds, lower is better)
------------------------------------------------------
Source Base Patched Change
------------------------------------------------------
NUMAB0 110,273,416 113,801,899 +3.2%
NUMAB2 71,859,123 86,098,560 +19.8%
pgtscan NA 71,545,031 NA
hwhints NA 71,857,476 NA
------------------------------------------------------
Mode 2 - Pages migrated (pgpromote_success)
------------------------------------------------------
Source Base Patched Change
------------------------------------------------------
NUMAB0 0 0 0%
NUMAB2 2097152 2080128 -0.8%
pgtscan NA 2097152 NA
hwhints NA 2097115 NA
------------------------------------------------------
Mode 3 - Time taken (microseconds, lower is better)
------------------------------------------------------
Source Base Patched Change
------------------------------------------------------
NUMAB0 30,944,794 30,537,137 -1.3%
NUMAB2 29,773,930 31,184,442 +4.7%
pgtscan NA 28,580,878 NA
hwhints NA 28,732,128 NA
------------------------------------------------------
Mode 3 - Pages migrated (thp_migration_success)
------------------------------------------------------
Source Base Patched Change
------------------------------------------------------
NUMAB0 0 0 0
NUMAB2 3754 1278 -65.9%
pgtscan NA 33032 NA
hwhints NA 32768 NA
------------------------------------------------------
Results Analysis TODO
---------------------
- Regression in NUMAB2 needs further analysis. The overhead of pghot path and effect of batched migration needs to be identified. It is seen that migrations get kicked off a bit later in kmigrated-NUMAB2 case compared to base-NUMAB2 case. This also needs further investigation.
This v3 patchset applies on top of upstream commit e53642b87a4f and can be fetched from:
https://github.com/AMDESE/linux-mm/tree/bharata/pghot-rfcv3
v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/
v0: https://lore.kernel.org/linux-mm/[email protected]/
TODOs
=====
- Check if the page is still within the hotness time window when kmigrated gets to it.
- Per-zone or per-section indicators to walk only zones or sections that have hot PFNs instead of kmigrated walking all the PFNs of the lower tier node.
- Bulk access reporting may be desirable for sources like IBS.
- Take care of memory hotplug for allocation/freeing of mem_section->hot_map.
- Currently I am defaulting to node 0 if target NID isn't specified by the source. The best fallback target node may have to determined dynamically.
- Provide compatibility alias for the sysctls moved from sched to pghot.
- Wider testing and benchmark coverage.
- Address Ying Huang's comment about merging migrate_misplaced_folio() and migrate_misplaced_folios_batch() and correctly handling memcg stats counting properly in the latter.
[1] kmigrated approach: https://lore.kernel.org/linux-mm/[email protected]/
Bharata B Rao (5):
mm: migrate: Allow misplaced migration without VMA too
mm: Hot page tracking and promotion
x86: ibs: In-kernel IBS driver for memory access profiling
x86: ibs: Enable IBS profiling for memory accesses
mm: sched: Move hot page promotion from NUMAB=2 to pghot tracking
Gregory Price (1):
migrate: implement migrate_misplaced_folios_batch
Kinsey Ho (2):
mm: mglru: generalize page table walk
mm: klruscand: use mglru scanning for page promotion
arch/x86/events/amd/ibs.c | 11 + arch/x86/include/asm/entry-common.h | 3 + arch/x86/include/asm/hardirq.h | 2 + arch/x86/include/asm/ibs.h | 9 + arch/x86/include/asm/msr-index.h | 16 + arch/x86/mm/Makefile | 3 +- arch/x86/mm/ibs.c | 343 +++++++++++++++++ include/linux/migrate.h | 6 + include/linux/mmzone.h | 19 + include/linux/pghot.h | 55 +++ include/linux/vm_event_item.h | 21 + kernel/sched/debug.c | 1 - kernel/sched/fair.c | 152 +------- mm/Kconfig | 19 + mm/Makefile | 2 + mm/huge_memory.c | 26 +- mm/internal.h | 4 + mm/klruscand.c | 110 ++++++ mm/memory.c | 31 +- mm/migrate.c | 41 +- mm/mm_init.c | 10 + mm/page_ext.c | 11 + mm/pghot.c | 571 ++++++++++++++++++++++++++++ mm/vmscan.c | 181 ++++++--- mm/vmstat.c | 21 + 25 files changed, 1427 insertions(+), 241 deletions(-) create mode 100644 arch/x86/include/asm/ibs.h create mode 100644 arch/x86/mm/ibs.c create mode 100644 include/linux/pghot.h create mode 100644 mm/klruscand.c create mode 100644 mm/pghot.c
Signature · 1 line
2.34.1
RFC PATCH v3 1/8 mm: migrate: Allow misplaced migration without VMA too
We want isolation of misplaced folios to work in contexts where VMA isn't available, typically when performing migrations from a kernel thread context. In order to prepare for that allow migrate_misplaced_folio_prepare() to be called with a NULL VMA.
When migrate_misplaced_folio_prepare() is called with non-NULL VMA, it will check if the folio is mapped shared and that requires holding PTL lock. This path isn't taken when the function is invoked with NULL VMA (migration outside of process context). Hence for such cases, it is not necessary this function be called with PTL lock held.
--- mm/migrate.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/migrate.c +3 -2
@@ -2644,7 +2644,8 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,/* * Prepare for calling migrate_misplaced_folio() by isolating the folio if -* permitted. Must be called with the PTL still held. +* permitted. Must be called with the PTL still held if called with a non-NULL +* vma. */ int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)
@@ -2661,7 +2662,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,* See folio_maybe_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ -if ((vma->vm_flags &VM_EXEC ) &&folio_maybe_mapped_shared (folio)) +if (vma && (vma->vm_flags &VM_EXEC ) &&folio_maybe_mapped_shared (folio))return -EACCES ;/*
Signature · 1 line
2.34.1
RFC PATCH v3 2/8 migrate: implement migrate_misplaced_folios_batch
In reply to Bharata B RaoFrom: Gregory Price <[email protected]>
A common operation in tiering is to migrate multiple pages at once. The migrate_misplaced_folio function requires one call for each individual folio. Expose a batch-variant of the same call for use when doing batch migrations.
--- include/linux/migrate.h | 6 ++++++ mm/migrate.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+)
include/linux/migrate.h +6 -0
@@ -103,6 +103,7 @@ static inline int set_movable_ops(const struct movable_operations *ops, enum pagint migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node);int migrate_misplaced_folio (struct folio *folio,int node); +int migrate_misplaced_folios_batch (struct list_head *foliolist,int node);#else static inline int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)
@@ -113,6 +114,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node) {return -EAGAIN ;/* can't migrate now */ } +static inline int migrate_misplaced_folios_batch (struct list_head *foliolist, +int node) +{ +return -EAGAIN ;/* can't migrate now */ +}#endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_MIGRATION
mm/migrate.c +36 -0
@@ -2739,5 +2739,41 @@ int migrate_misplaced_folio(struct folio *folio, int node)BUG_ON (!list_empty (&migratepages));return nr_remaining ? -EAGAIN :0 ; } + +/** +* migrate_misplaced_folios_batch - Batch variant of migrate_misplaced_folio. +* Attempts to migrate a folio list to the specified destination. +* @folio_list - Isolated list of folios to be batch-migrated. +* @node - The NUMA node ID to where the folios should be migrated. +* +* Caller is expected to have isolated the folios by calling +* migrate_misplaced_folio_prepare(), which will result in an +* elevated reference count on the folio. +* +* This function will un-isolate the folios, dereference them, and +* remove them from the list before returning. +* +* Return: 0 on success and -EAGAIN on failure or partial migration. +* On return, @folio_list will be empty regardless of success/failure. +*/ +int migrate_misplaced_folios_batch (struct list_head *folio_list,int node) +{ +pg_data_t *pgdat =NODE_DATA (node); +unsigned int nr_succeeded; +int nr_remaining; + + nr_remaining =migrate_pages (folio_list, alloc_misplaced_dst_folio, +NULL , node,MIGRATE_ASYNC , +MR_NUMA_MISPLACED , &nr_succeeded); +if (nr_remaining) +putback_movable_pages (folio_list); + +if (nr_succeeded) { +count_vm_numa_events (NUMA_PAGE_MIGRATE , nr_succeeded); +mod_node_page_state (pgdat,PGPROMOTE_SUCCESS , nr_succeeded); + } +BUG_ON (!list_empty (folio_list)); +return nr_remaining ? -EAGAIN :0 ; +}#endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_NUMA */
Signature · 1 line
2.34.1
RFC PATCH v3 3/8 mm: Hot page tracking and promotion
In reply to Bharata B RaoThis introduces a sub-system for collecting memory access information from different sources. It maintains the hotness information based on the access history and time of access.
Additionally, it provides per-lowertier-node kernel threads (named kmigrated) that periodically promote the pages that are eligible for promotion.
Sub-systems that generate hot page access info can report that using this API:
int pghot_record_access(unsigned long pfn, int nid, int src,
unsigned long time)
@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (sub-system) that generated the
access info
@time: The access time in jiffies
Some temperature sources may not provide the nid from which the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. For such sources, the default toptier node to which such pages should be promoted is hard coded.
Also, the access time provided some sources may at best be considered approximate. This is especially true for hot pages detected by PTE A bit scanning.
The hotness information is stored for every page of lower tier memory in an unsigned long variable that is part of mem_section data structure.
kmigrated is a per-lowertier-node kernel thread that migrates the folios marked for migration in batches. Each kmigrated thread walks the PFN range spanning its node and checks for potential migration candidates.
--- include/linux/mmzone.h | 14 ++ include/linux/pghot.h | 52 ++++ include/linux/vm_event_item.h | 4 + mm/Kconfig | 11 + mm/Makefile | 1 + mm/mm_init.c | 10 + mm/page_ext.c | 11 + mm/pghot.c | 446 ++++++++++++++++++++++++++++++++++ mm/vmstat.c | 4 + 9 files changed, 553 insertions(+) create mode 100644 include/linux/pghot.h create mode 100644 mm/pghot.c
include/linux/mmzone.h +14 -0
@@ -1068,6 +1068,7 @@ enum pgdat_flags {* many pages under writeback */ PGDAT_RECLAIM_LOCKED ,/* prevents concurrent reclaim */ + PGDAT_KMIGRATED_ACTIVATE, /* activates kmigrated */ }; enum zone_flags {
@@ -1522,6 +1523,10 @@ typedef struct pglist_data {#ifdef CONFIG_MEMORY_FAILURE struct memory_failure_stats mf_stats;#endif +#ifdef CONFIG_PGHOT +struct task_struct *kmigrated; +wait_queue_head_t kmigrated_wait; +#endif } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)
@@ -1920,12 +1925,21 @@ struct mem_section {unsigned long section_mem_map;struct mem_section_usage *usage; +#ifdef CONFIG_PGHOT +/* +* Per-PFN hotness data for this section. +*/ +unsigned long *hot_map; +#endif #ifdef CONFIG_PAGE_EXTENSION /* * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use * section. (see page_ext.h about this.) */ struct page_ext *page_ext; +#endif +#if (defined(CONFIG_PGHOT ) && !defined(CONFIG_PAGE_EXTENSION )) || \ + (!defined(CONFIG_PGHOT ) && defined(CONFIG_PAGE_EXTENSION ))unsigned long pad;#endif /*
include/linux/pghot.h new file +52 -0
52 lines not on this page · raw message
include/linux/vm_event_item.h +4 -0
@@ -188,6 +188,10 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, KSTACK_REST,#endif #endif /* CONFIG_DEBUG_STACK_USAGE */ +PGHOT_RECORDED_ACCESSES , +PGHOT_RECORD_HWHINTS , +PGHOT_RECORD_PGTSCANS , +PGHOT_RECORD_HINTFAULTS , NR_VM_EVENT_ITEMS };
mm/Kconfig +11 -0
@@ -1372,6 +1372,17 @@ config PT_RECLAIM config FIND_NORMAL_PAGE def_bool n +config PGHOT + bool "Hot page tracking and promotion" + def_bool y + depends on NUMA && MIGRATION && SPARSEMEM && MMU + help + A sub-system to track page accesses in lower tier memory and + maintain hot page information. Promotes hot pages from lower + tiers to top tier by using the memory access information provided + by various sources. Asynchronous promotion is done by per-node + kernel threads. + source "mm/damon/Kconfig" endmenu
mm/Makefile +1 -0
@@ -146,3 +146,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o +obj-$(CONFIG_PGHOT) += pghot.o
mm/mm_init.c +10 -0
@@ -1401,6 +1401,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)static void pgdat_init_kcompactd (struct pglist_data *pgdat) {}#endif +#ifdef CONFIG_PGHOT +static void pgdat_init_kmigrated (struct pglist_data *pgdat) +{ +init_waitqueue_head (&pgdat->kmigrated_wait); +} +#else +static inline void pgdat_init_kmigrated (struct pglist_data *pgdat) {} +#endif +static void __meminit pgdat_init_internals (struct pglist_data *pgdat) {int i;
@@ -1410,6 +1419,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)pgdat_init_split_queue (pgdat);pgdat_init_kcompactd (pgdat); +pgdat_init_kmigrated (pgdat);init_waitqueue_head (&pgdat->kswapd_wait);init_waitqueue_head (&pgdat->pfmemalloc_wait);
mm/page_ext.c +11 -0
@@ -76,6 +76,16 @@ static struct page_ext_operations page_idle_ops __initdata = { };#endif +static bool need_page_mig (void ) +{ +return true; +} + +static struct page_ext_operations page_mig_ops__initdata = { + .need = need_page_mig, + .need_shared_flags = true, +}; +static struct page_ext_operations *page_ext_ops[]__initdata = { #ifdef CONFIG_PAGE_OWNER &page_owner_ops,
@@ -89,6 +99,7 @@ static struct page_ext_operations *page_ext_ops[] __initdata = {#ifdef CONFIG_PAGE_TABLE_CHECK &page_table_check_ops, #endif + &page_mig_ops, };unsigned long page_ext_size;
mm/pghot.c new file +446 -0
446 lines not on this page · raw message
mm/vmstat.c +4 -0
@@ -1496,6 +1496,10 @@ const char * const vmstat_text[] = { [I(KSTACK_REST)] = "kstack_rest",#endif #endif + [I(PGHOT_RECORDED_ACCESSES)] = "pghot_recorded_accesses", + [I(PGHOT_RECORD_HWHINTS)] = "pghot_recorded_hwhints", + [I(PGHOT_RECORD_PGTSCANS)] = "pghot_recorded_pgtscans", + [I (PGHOT_RECORD_HINTFAULTS )] ="pghot_recorded_hintfaults" , #undef I#endif /* CONFIG_VM_EVENT_COUNTERS */ };
Signature · 1 line
2.34.1
Re: [RFC PATCH v3 3/8] mm: Hot page tracking and promotion
Quoted 55 lines · Bharata B Rao
This introduces a sub-system for collecting memory access information from different sources. It maintains the hotness information based on the access history and time of access.
Additionally, it provides per-lowertier-node kernel threads (named kmigrated) that periodically promote the pages that are eligible for promotion.
Sub-systems that generate hot page access info can report that using this API:
int pghot_record_access(unsigned long pfn, int nid, int src,
unsigned long time)@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (sub-system) that generated the
access info
@time: The access time in jiffiesSome temperature sources may not provide the nid from which the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. For such sources, the default toptier node to which such pages should be promoted is hard coded.
Also, the access time provided some sources may at best be considered approximate. This is especially true for hot pages detected by PTE A bit scanning.
The hotness information is stored for every page of lower tier memory in an unsigned long variable that is part of
mem_sectiondata structure.kmigrated is a per-lowertier-node kernel thread that migrates the folios marked for migration in batches. Each kmigrated thread walks the PFN range spanning its node and checks for potential migration candidates.
Signed-off-by Bharata B Rao <[email protected]>---
include/linux/mmzone.h | 14 ++ include/linux/pghot.h | 52 ++++ include/linux/vm_event_item.h | 4 + mm/Kconfig | 11 + mm/Makefile | 1 + mm/mm_init.c | 10 + mm/page_ext.c | 11 + mm/pghot.c | 446 ++++++++++++++++++++++++++++++++++ mm/vmstat.c | 4 + 9 files changed, 553 insertions(+)create mode 100644
include/linux/pghot.h
create mode 100644mm/pghot.c+
<snip>
Quoted 23 lines
mm/pghot.c new file +23
-0+/* +* Walks the PFNs of the zone, isolates and migrates them in batches. +*/ +static void kmigrated_walk_zone (unsigned long start_pfn,unsigned long end_pfn, +int src_nid) +{ +int cur_nid =NUMA_NO_NODE ; +LIST_HEAD (migrate_list); +int batch_count =0 ; +struct folio *folio; +struct page *page; +unsigned long pfn; + + pfn = start_pfn; + do { + unsigned long nid = NUMA_NO_NODE, freq = 0, time = 0, nr = 1; + + if (!pfn_valid(pfn)) + goto out_next; + + page = pfn_to_online_page(pfn); + if (!page) + goto out_next;
mm/pghot.c line 281
Quoted 10 lines
mm/pghot.c new file +10
-0+ + folio =page_folio (page); + nr =folio_nr_pages (folio); +if (folio_nid (folio) != src_nid) + goto out_next; + +if (!folio_test_lru (folio)) + goto out_next; + +if (pghot_get_hotness (pfn, &nid, &freq, &time))
Better to remove freq value, it’s not used later.
Regards,
Alok Rathore
Re: [RFC PATCH v3 3/8] mm: Hot page tracking and promotion
Quoted 27 lines · Alok Rathore
Quoted 27 lines
mm/pghot.c new file +27
-0+/* +* Walks the PFNs of the zone, isolates and migrates them in batches. +*/ +static void kmigrated_walk_zone (unsigned long start_pfn,unsigned long end_pfn, + int src_nid) +{ + int cur_nid = NUMA_NO_NODE; + LIST_HEAD(migrate_list); + int batch_count = 0; + struct folio *folio; + struct page *page; + unsigned long pfn; + + pfn = start_pfn; + do { + unsigned long nid = NUMA_NO_NODE, freq = 0, time = 0, nr = 1; + + if (!pfn_valid(pfn)) + goto out_next; + + page = pfn_to_online_page(pfn); + if (!page) + goto out_next; + + folio = page_folio(page); + nr = folio_nr_pages(folio); + if (folio_nid(folio) != src_nid)
Quoted 8 lines
mm/pghot.c line 281
Quoted 6 lines
+ goto
out_next;mm/pghot.c new file +5
-0+ + if (!folio_test_lru(folio)) + goto out_next; + + if (pghot_get_hotness(pfn, &nid, &freq, &time))Better to remove freq value, it’s not used later.
Yes, can be removed. I am leaving it here just for a future possibility of having kmigrated being driven to migrate hottest pages on priority compared to hot pages at which point frequency value may come in handy.
Regards,
Bharata.
RFC PATCH v3 4/8 x86: ibs: In-kernel IBS driver for memory access profiling
In reply to Bharata B RaoUse IBS (Instruction Based Sampling) feature present in AMD processors for memory access tracking. The access information obtained from IBS via NMI is fed to pghot sub-system for futher action.
In addition to many other information related to the memory access, IBS provides physical (and virtual) address of the access and indicates if the access came from slower tier. Only memory accesses originating from slower tiers are further acted upon by this driver.
The samples are initially accumulated in percpu buffers which are flushed to pghot hot page tracking mechanism using irq_work.
TODO: Many counters are added to vmstat just as debugging aid for now.
About IBS
---------
IBS can be programmed to provide data about instruction execution periodically. This is done by programming a desired sample count (number of ops) in a control register. When the programmed number of ops are dispatched, a micro-op gets tagged, various information about the tagged micro-op's execution is populated in IBS execution MSRs and an interrupt is raised. While IBS provides a lot of data for each sample, for the purpose of memory access profiling, we are interested in linear and physical address of the memory access that reached DRAM. Recent AMD processors provide further filtering where it is possible to limit the sampling to those ops that had an L3 miss which greately reduces the non-useful samples.
While IBS provides capability to sample instruction fetch and execution, only IBS execution sampling is used here to collect data about memory accesses that occur during the instruction execution.
More information about IBS is available in Sec 13.3 of AMD64 Architecture Programmer's Manual, Volume 2:System Programming which is present at:
https://bugzilla.kernel.org/attachment.cgi?id=288923
Information about MSRs used for programming IBS can be found in Sec 2.1.14.4 of PPR Vol 1 for AMD Family 19h Model 11h B1 which is currently present at:
https://www.amd.com/system/files/TechDocs/55901_0.25.zip
--- arch/x86/events/amd/ibs.c | 11 ++ arch/x86/include/asm/ibs.h | 7 + arch/x86/include/asm/msr-index.h | 16 ++ arch/x86/mm/Makefile | 3 +- arch/x86/mm/ibs.c | 311 +++++++++++++++++++++++++++++++ include/linux/vm_event_item.h | 17 ++ mm/vmstat.c | 17 ++ 7 files changed, 381 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/ibs.h create mode 100644 arch/x86/mm/ibs.c
arch/x86/events/amd/ibs.c +11 -0
@@ -13,9 +13,11 @@#include <linux/ptrace.h> #include <linux/syscore_ops.h> #include <linux/sched/clock.h> +#include <linux/pghot.h> #include <asm/apic.h> #include <asm/msr.h> +#include <asm/ibs.h> #include "../perf_event.h"
@@ -1756,6 +1758,15 @@ static __init int amd_ibs_init(void) {u32 caps; +/* +* TODO: Find a clean way to disable perf IBS so that IBS +* can be used for memory access profiling. +*/ +if (arch_hw_access_profiling) { +pr_info ("IBS isn't available for perf use\n" ); +return 0 ; + } + caps =__get_ibs_caps ();if (!caps)return -ENODEV ;/* ibs not supported by the cpu */
arch/x86/include/asm/ibs.h new file +7 -0
@@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_IBS_H +#define _ASM_X86_IBS_H + +extern bool arch_hw_access_profiling; + +#endif /* _ASM_X86_IBS_H */
arch/x86/include/asm/msr-index.h +16 -0
@@ -760,6 +760,22 @@/* AMD Last Branch Record MSRs */ #define MSR_AMD64_LBR_SELECT 0xc000010e +/* AMD IBS MSR bits */ +#define MSR_AMD64_IBSOPDATA2_DATASRC 0x7 +#define MSR_AMD64_IBSOPDATA2_DATASRC_LCL_CACHE 0x1 +#define MSR_AMD64_IBSOPDATA2_DATASRC_PEER_CACHE_NEAR 0x2 +#define MSR_AMD64_IBSOPDATA2_DATASRC_DRAM 0x3 +#define MSR_AMD64_IBSOPDATA2_DATASRC_FAR_CCX_CACHE 0x5 +#define MSR_AMD64_IBSOPDATA2_DATASRC_EXT_MEM 0x8 +#define MSR_AMD64_IBSOPDATA2_RMTNODE 0x10 + +#define MSR_AMD64_IBSOPDATA3_LDOP BIT_ULL(0) +#define MSR_AMD64_IBSOPDATA3_STOP BIT_ULL(1) +#define MSR_AMD64_IBSOPDATA3_DCMISS BIT_ULL(7) +#define MSR_AMD64_IBSOPDATA3_LADDR_VALID BIT_ULL(17) +#define MSR_AMD64_IBSOPDATA3_PADDR_VALID BIT_ULL(18) +#define MSR_AMD64_IBSOPDATA3_L2MISS BIT_ULL(20) +/* Zen4 */ #define MSR_ZEN4_BP_CFG 0xc001102e#define MSR_ZEN4_BP_CFG_BP_SPEC_REDUCE_BIT 4
arch/x86/mm/Makefile +2 -1
@@ -22,7 +22,8 @@ CFLAGS_REMOVE_pgprot.o = -pg endif obj-y := init.o init_$(BITS).o fault.o ioremap.o extable.o mmap.o \ - pgtable.o physaddr.o tlb.o cpu_entry_area.o maccess.o pgprot.o + pgtable.o physaddr.o tlb.o cpu_entry_area.o maccess.o pgprot.o \ + ibs.o obj-y += pat/
arch/x86/mm/ibs.c new file +311 -0
311 lines not on this page · raw message
include/linux/vm_event_item.h +17 -0
@@ -192,6 +192,23 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,PGHOT_RECORD_HWHINTS ,PGHOT_RECORD_PGTSCANS ,PGHOT_RECORD_HINTFAULTS , +HWHINT_NR_EVENTS , +HWHINT_KERNEL , +HWHINT_KTHREAD , +HWHINT_NON_LOAD_STORES , +HWHINT_DC_L2_HITS , +HWHINT_LOCAL_L3L1L2 , +HWHINT_LOCAL_PEER_CACHE_NEAR , +HWHINT_FAR_CACHE_HITS , +HWHINT_DRAM_ACCESSES , +HWHINT_CXL_ACCESSES , +HWHINT_REMOTE_NODE , +HWHINT_LADDR_INVALID , +HWHINT_KERNEL_ADDR , +HWHINT_PADDR_INVALID , +HWHINT_NON_LRU , +HWHINT_BUFFER_FULL , +HWHINT_USEFUL_SAMPLES ,NR_VM_EVENT_ITEMS };
mm/vmstat.c +17 -0
@@ -1500,6 +1500,23 @@ const char * const vmstat_text[] = { [I(PGHOT_RECORD_HWHINTS)] = "pghot_recorded_hwhints", [I(PGHOT_RECORD_PGTSCANS)] = "pghot_recorded_pgtscans", [I(PGHOT_RECORD_HINTFAULTS)] = "pghot_recorded_hintfaults", + [I(HWHINT_NR_EVENTS)] = "hwhint_nr_events", + [I(HWHINT_KERNEL)] = "hwhint_kernel", + [I(HWHINT_KTHREAD)] = "hwhint_kthread", + [I(HWHINT_NON_LOAD_STORES)] = "hwhint_non_load_stores", + [I(HWHINT_DC_L2_HITS)] = "hwhint_dc_l2_hits", + [I(HWHINT_LOCAL_L3L1L2)] = "hwhint_local_l3l1l2", + [I(HWHINT_LOCAL_PEER_CACHE_NEAR)] = "hwhint_local_peer_cache_near", + [I(HWHINT_FAR_CACHE_HITS)] = "hwhint_far_cache_hits", + [I(HWHINT_DRAM_ACCESSES)] = "hwhint_dram_accesses", + [I(HWHINT_CXL_ACCESSES)] = "hwhint_cxl_accesses", + [I(HWHINT_REMOTE_NODE)] = "hwhint_remote_node", + [I(HWHINT_LADDR_INVALID)] = "hwhint_invalid_laddr", + [I(HWHINT_KERNEL_ADDR)] = "hwhint_kernel_addr", + [I(HWHINT_PADDR_INVALID)] = "hwhint_invalid_paddr", + [I(HWHINT_NON_LRU)] = "hwhint_non_lru", + [I(HWHINT_BUFFER_FULL)] = "hwhint_buffer_full", + [I (HWHINT_USEFUL_SAMPLES )] ="hwhint_useful_samples" , #undef I#endif /* CONFIG_VM_EVENT_COUNTERS */ };
Signature · 1 line
2.34.1
RFC PATCH v3 5/8 x86: ibs: Enable IBS profiling for memory accesses
In reply to Bharata B RaoEnable IBS memory access data collection for user memory accesses by programming the required MSRs. The profiling is turned ON only for user mode execution and turned OFF for kernel mode execution. Profiling is explicitly disabled for NMI handler too.
TODOs:
- IBS sampling rate is kept fixed for now.
- Arch/vendor separation/isolation of the code needs relook.
--- arch/x86/include/asm/entry-common.h | 3 +++ arch/x86/include/asm/hardirq.h | 2 ++ arch/x86/include/asm/ibs.h | 2 ++ arch/x86/mm/ibs.c | 32 +++++++++++++++++++++++++++++ 4 files changed, 39 insertions(+)
arch/x86/include/asm/entry-common.h +3 -0
@@ -9,10 +9,12 @@#include <asm/io_bitmap.h> #include <asm/fpu/api.h> #include <asm/fred.h> +#include <asm/ibs.h> /* Check that the stack and regs on entry from user mode are sane. */ static __always_inline void arch_enter_from_user_mode(struct pt_regs *regs) { +hw_access_profiling_stop ();if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) {/* * Make sure that the entry code gave us a sensible EFLAGS
@@ -106,6 +108,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, static __always_inline void arch_exit_to_user_mode(void) {amd_clear_divider (); +hw_access_profiling_start (); }#define arch_exit_to_user_mode arch_exit_to_user_mode
arch/x86/include/asm/hardirq.h +2 -0
@@ -91,4 +91,6 @@ static __always_inline bool kvm_get_cpu_l1tf_flush_l1d(void) static __always_inline void kvm_set_cpu_l1tf_flush_l1d(void) { }#endif /* IS_ENABLED(CONFIG_KVM_INTEL) */ +#define arch_nmi_enter () hw_access_profiling_stop() +#define arch_nmi_exit () hw_access_profiling_start()#endif /* _ASM_X86_HARDIRQ_H */
arch/x86/include/asm/ibs.h +2 -0
@@ -2,6 +2,8 @@#ifndef _ASM_X86_IBS_H#define _ASM_X86_IBS_H +void hw_access_profiling_start (void ); +void hw_access_profiling_stop (void );extern bool arch_hw_access_profiling;#endif /* _ASM_X86_IBS_H */
arch/x86/mm/ibs.c +32 -0
@@ -16,6 +16,7 @@ static u64 ibs_config __read_mostly;static u32 ibs_caps;#define IBS_NR_SAMPLES 150 +#define IBS_SAMPLE_PERIOD 10000/* * Basic access info captured for each memory access.
@@ -98,6 +99,36 @@ static void ibs_irq_handler(struct irq_work *i)schedule_work_on (smp_processor_id (), &ibs_work); } +void hw_access_profiling_stop (void ) +{ +u64 ops_ctl; + +if (!arch_hw_access_profiling) +return ; + +rdmsrl (MSR_AMD64_IBSOPCTL , ops_ctl); +wrmsrl (MSR_AMD64_IBSOPCTL , ops_ctl & ~IBS_OP_ENABLE ); +} + +void hw_access_profiling_start (void ) +{ +u64 config =0 ; +unsigned int period =IBS_SAMPLE_PERIOD ; + +if (!arch_hw_access_profiling) +return ; + +/* Disable IBS for kernel thread */ +if (!current->mm) + goto out; + + config = (period >>4 ) &IBS_OP_MAX_CNT ; + config |= (period &IBS_OP_MAX_CNT_EXT_MASK ); + config |= ibs_config; +out: +wrmsrl (MSR_AMD64_IBSOPCTL , config); +} +/* * IBS NMI handler: Process the memory access info reported by IBS. *
@@ -304,6 +335,7 @@ static int __init ibs_access_profiling_init(void) x86_amd_ibs_access_profile_startup, x86_amd_ibs_access_profile_teardown); + arch_hw_access_profiling = true;pr_info ("IBS setup for memory access profiling\n" );return 0 ; }
Signature · 1 line
2.34.1
RFC PATCH v3 6/8 mm: mglru: generalize page table walk
In reply to Bharata B RaoFrom: Kinsey Ho <[email protected]>
Refactor the existing MGLRU page table walking logic to make it resumable.
Additionally, introduce two hooks into the MGLRU page table walk:
accessed callback and flush callback. The accessed callback is called for each accessed page detected via the scanned accessed bit. The flush callback is called when the accessed callback reports that a flush is required. This allows for processing pages in batches for efficiency.
With a generalised page table walk, introduce a new scan function which repeatedly scans on the same young generation and does not add a new young generation.
--- include/linux/mmzone.h | 5 ++ mm/internal.h | 4 + mm/vmscan.c | 181 +++++++++++++++++++++++++++++++---------- 3 files changed, 145 insertions(+), 45 deletions(-)
include/linux/mmzone.h +5 -0
@@ -548,6 +548,8 @@ struct lru_gen_mm_walk {unsigned long seq;/* the next address within an mm to scan */ unsigned long next_addr; +/* called for each accessed pte/pmd */ +bool (*accessed_cb)(unsigned long pfn);/* to batch promoted pages */ int nr_pages[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ];/* to batch the mm stats */
@@ -555,6 +557,9 @@ struct lru_gen_mm_walk {/* total batched items */ int batched;int swappiness; +/* for the pmd under scanning */ +int nr_young_pte; +int nr_total_pte;bool force_scan; };
mm/internal.h +4 -0
@@ -538,6 +538,10 @@ extern unsigned long highest_memmap_pfn;bool folio_isolate_lru (struct folio *folio);void folio_putback_lru (struct folio *folio);extern void reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason); +void set_task_reclaim_state (struct task_struct *task, +struct reclaim_state *rs); +void lru_gen_scan_lruvec (struct lruvec *lruvec,unsigned long seq, +bool (*accessed_cb)(unsigned long ),void (*flush_cb)(void ));#ifdef CONFIG_NUMA int user_proactive_reclaim (char *buf,struct mem_cgroup *memcg,pg_data_t *pgdat);
mm/vmscan.c +136 -45
361 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC PATCH v3 7/8 mm: klruscand: use mglru scanning for page promotion
In reply to Bharata B RaoFrom: Kinsey Ho <[email protected]>
Introduce a new kernel daemon, klruscand, that periodically invokes the MGLRU page table walk. It leverages the new callbacks to gather access information and forwards it to pghot sub-system for promotion decisions.
This benefits from reusing the existing MGLRU page table walk infrastructure, which is optimized with features such as hierarchical scanning and bloom filters to reduce CPU overhead.
As an additional optimization to be added in the future, we can tune the scan intervals for each memcg.
--- mm/Kconfig | 8 ++++ mm/Makefile | 1 + mm/klruscand.c | 110 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 119 insertions(+) create mode 100644 mm/klruscand.c
mm/Kconfig +8 -0
@@ -1383,6 +1383,14 @@ config PGHOT by various sources. Asynchronous promotion is done by per-node kernel threads. +config KLRUSCAND + bool "Kernel lower tier access scan daemon" + default n + depends on PGHOT && LRU_GEN_WALKS_MMU + help + Scan for accesses from lower tiers by invoking MGLRU to perform + page table walks. + source "mm/damon/Kconfig" endmenu
mm/Makefile +1 -0
@@ -147,3 +147,4 @@ obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o obj-$(CONFIG_PGHOT) += pghot.o +obj-$(CONFIG_KLRUSCAND) += klruscand.o
mm/klruscand.c new file +110 -0
110 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC PATCH v3 8/8 mm: sched: Move hot page promotion from NUMAB=2 to pghot tracking
In reply to Bharata B RaoCurrently hot page promotion (NUMA_BALANCING_MEMORY_TIERING mode of NUMA Balancing) does hot page detection (via hint faults), hot page classification and eventual promotion, all by itself and sits within the scheduler.
With the new hot page tracking and promotion mechanism being available, NUMA Balancing can limit itself to detection of hot pages (via hint faults) and off-load rest of the functionality to the common hot page tracking system.
pghot_record_access(PGHOT_HINT_FAULT) API is used to feed the hot page info. In addition, the migration rate limiting and dynamic threshold logic are moved to kmigrated so that the same can be used for hot pages reported by other sources too.
--- include/linux/pghot.h | 3 + kernel/sched/debug.c | 1 - kernel/sched/fair.c | 152 ++---------------------------------------- mm/huge_memory.c | 26 ++------ mm/memory.c | 31 ++------- mm/pghot.c | 129 ++++++++++++++++++++++++++++++++++- 6 files changed, 147 insertions(+), 195 deletions(-)
include/linux/pghot.h +3 -0
@@ -42,6 +42,9 @@ enum pghot_src {#define PGHOT_FREQ_MAX (1 << PGHOT_FREQ_WIDTH)#define PGHOT_TIME_MAX (1 << PGHOT_TIME_WIDTH) +#define KMIGRATED_MIGRATION_ADJUST_STEPS 16 +#define KMIGRATED_PROMOTION_THRESHOLD_WINDOW 60000 +int pghot_record_access (unsigned long pfn,int nid,int src,unsigned long now);#else static inline int pghot_record_access (unsigned long pfn,int nid,int src,unsigned long now)
kernel/sched/debug.c +0 -1
@@ -520,7 +520,6 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("scan_period_min_ms" ,0644 , numa, &sysctl_numa_balancing_scan_period_min);debugfs_create_u32 ("scan_period_max_ms" ,0644 , numa, &sysctl_numa_balancing_scan_period_max);debugfs_create_u32 ("scan_size_mb" ,0644 , numa, &sysctl_numa_balancing_scan_size); -debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);
kernel/sched/fair.c +5 -147
182 lines not on this page · raw message
mm/huge_memory.c +6 -20
@@ -39,6 +39,7 @@#include <linux/compat.h> #include <linux/pgalloc_tag.h> #include <linux/pagewalk.h> +#include <linux/pghot.h> #include <asm/tlb.h> #include <asm/pgalloc.h>
@@ -2050,29 +2051,12 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) target_nid =numa_migrate_check (folio, vmf, haddr, &flags, writable, &last_cpupid); + nid = target_nid;if (target_nid ==NUMA_NO_NODE ) goto out_map; -if (migrate_misplaced_folio_prepare (folio, vma, target_nid)) { - flags |=TNF_MIGRATE_FAIL ; - goto out_map; - } -/* The folio is isolated and isolation code holds a folio reference. */ -spin_unlock (vmf->ptl); - writable = false; -if (!migrate_misplaced_folio (folio, target_nid)) { - flags |=TNF_MIGRATED ; - nid = target_nid; -task_numa_fault (last_cpupid, nid,HPAGE_PMD_NR , flags); -return 0 ; - } + writable = false; - flags |=TNF_MIGRATE_FAIL ; - vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd); -if (unlikely (!pmd_same (pmdp_get (vmf->pmd), vmf->orig_pmd))) { -spin_unlock (vmf->ptl); -return 0 ; - } out_map:/* Restore the PMD */ pmd =pmd_modify (pmdp_get (vmf->pmd), vma->vm_page_prot);
@@ -2083,8 +2067,10 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)update_mmu_cache_pmd (vma, vmf->address, vmf->pmd);spin_unlock (vmf->ptl); -if (nid !=NUMA_NO_NODE ) +if (nid !=NUMA_NO_NODE ) { +pghot_record_access (folio_pfn (folio), nid,PGHOT_HINT_FAULT , jiffies);task_numa_fault (last_cpupid, nid,HPAGE_PMD_NR , flags); + }return 0 ; }
mm/memory.c +6 -25
@@ -74,6 +74,7 @@#include <linux/perf_event.h> #include <linux/ptrace.h> #include <linux/vmalloc.h> +#include <linux/pghot.h> #include <linux/sched/sysctl.h> #include <trace/events/kmem.h>
@@ -5989,34 +5990,12 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) target_nid =numa_migrate_check (folio, vmf, vmf->address, &flags, writable, &last_cpupid); + nid = target_nid;if (target_nid ==NUMA_NO_NODE ) goto out_map; -if (migrate_misplaced_folio_prepare (folio, vma, target_nid)) { - flags |=TNF_MIGRATE_FAIL ; - goto out_map; - } -/* The folio is isolated and isolation code holds a folio reference. */ -pte_unmap_unlock (vmf->pte, vmf->ptl); + writable = false; ignore_writable = true; - -/* Migrate to the requested node */ -if (!migrate_misplaced_folio (folio, target_nid)) { - nid = target_nid; - flags |=TNF_MIGRATED ; -task_numa_fault (last_cpupid, nid, nr_pages, flags); -return 0 ; - } - - flags |=TNF_MIGRATE_FAIL ; - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); -if (unlikely (!vmf->pte)) -return 0 ; -if (unlikely (!pte_same (ptep_get (vmf->pte), vmf->orig_pte))) { -pte_unmap_unlock (vmf->pte, vmf->ptl); -return 0 ; - } out_map:/* * Make it present again, depending on how arch implements
@@ -6030,8 +6009,10 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) writable);pte_unmap_unlock (vmf->pte, vmf->ptl); -if (nid !=NUMA_NO_NODE ) +if (nid !=NUMA_NO_NODE ) { +pghot_record_access (folio_pfn (folio), nid,PGHOT_HINT_FAULT , jiffies);task_numa_fault (last_cpupid, nid, nr_pages, flags); + }return 0 ; }
mm/pghot.c +127 -2
165 lines not on this page · raw message
Signature · 1 line
2.34.1
On 10-Nov-25 10:53 AM, Bharata B Rao wrote:
<snip>
Quoted 2 lines
Results
=======
Earlier I included results from the scenario where there was enough free memory in the toptier node and hence demotions weren't getting triggered. Here I am including results from a similar microbenchmark that results in demotion too.
System details
--------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)
$ numactl -H available: 3 nodes (0-2) node 0 cpus: 0-95,192-287 node 0 size: 128460 MB node 1 cpus: 96-191,288-383 node 1 size: 128893 MB node 2 cpus: node 2 size: 257993 MB node distances: node 0 1 2 0: 10 32 50 1: 32 10 60 2: 255 255 10
Microbenchmark details
----------------------
Single threaded application that allocates memory on both DRAM and CXL nodes using mmap(MAP_POPULATE). Every 1G region of allocated memory on CXL node is accessed at 4K granularity randomly and repetitively to build up the notion of hotness in the 1GB region that is under access. This should drive promotion. For promotion to work successfully, the DRAM memory that has been provisioned (and not being accessed) should be demoted first. There is enough free memory in the CXL node to for demotions.
In summary, this benchmark creates a memory pressure on DRAM node and does CXL memory accesses to drive both demotion and promotion.
The number of accesses are fixed and hence, the quicker the accessed pages get promoted to DRAM, the sooner the benchmark is expected to finish.
DRAM-node = 1
CXL-node = 2
Initial DRAM alloc ratio = 75%
Allocation-size = 171798691840
Initial DRAM Alloc-size = 128849018880
Initial CXL Alloc-size = 42949672960
Hot-region-size = 1073741824
Nr-regions = 160
Nr-regions DRAM = 120 (provisioned but not accessed)
Nr-hot-regions CXL = 40
Access pattern = random
Access granularity = 4096
Delay b/n accesses = 0
Load/store ratio = 50l50s
THP used = no
Nr accesses = 42949672960
Nr repetitions = 1024
Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the patched case. No migrations.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the patched case.
pgtscan - Klruscand (MGLRU based PTE A bit scanning) source hwhints - IBS as source
Time taken (microseconds, lower is better)
----------------------------------------------
Source Base Patched Change
----------------------------------------------
NUMAB0 63,036,030 64,441,675 +2.2%
NUMAB2 62,286,691 68,786,394 +10.4%(#)
pgtscan NA 68,702,226
hwhints NA 67,455,607
----------------------------------------------
Pages migrated (pgpromote_success)
----------------------------------------------
Source Base Patched
----------------------------------------------
NUMAB0 0 0
NUMAB2 82134(*) 0(#)
pgtscan NA 6,561,136
hwhints NA 3,293($)
----------------------------------------------
(#) Unlike base NUMAB2, pghot migrates after 2 accesses.
Getting two successive accesses within the observation window is hard with NUMA hint faults. The default sysctl_numa_balancing_scan_size of 256MB is too less to obtain significant number of hint faults.
(*) High run-to-run variation, so the average isn't really representative.
Hint fault latency comes out higher than the default 1s threshold mostly, preventing migrations.
($) Sampling limitation
Pages demoted (pgdemote_kswapd+pgdemote_direct)
(This data is not really a comparision point but just providing these numbers to show that the workload results in both promotion and demotion)
----------------------------------------------
Source Base Patched
----------------------------------------------
NUMAB0 5,222,366 5,341,502
NUMAB2 5,256,310 5,325,845
pgtscan NA 5,317,709
hwhints NA 5,287,091
----------------------------------------------
Promotion candidate pages (pgpromote_candidate)
----------------------------------------------
Source Base Patched
----------------------------------------------
NUMAB0 0 0
NUMAB2 82,848 0
pgtscan NA 0
hwhints NA 0
----------------------------------------------
Non-rate limited Promotion candidate pages (pgpromote_candidate_nrl)
----------------------------------------------
Source Base Patched
----------------------------------------------
NUMAB0 0 0
NUMAB2 0 0
pgtscan NA 6,561,147
hwhints NA 3,292
----------------------------------------------