mm: Hot page tracking and promotion infrastructure
Subsystems (13)
v4 cover mm: Hot page tracking and promotion infrastructure
Hi,
This is v4 of page hotness tracking and promotion sub-system pghot.
This patchset introduces a new subsystem for hot page tracking and promotion (pghot) with the following goals:
- Unify hot page detection from multiple sources like hint faults, page table scans, hardware hints (IBS).
- Decouple detection from migration.
- Centralize promotion logic via per-lowertier-node kernel thread.
- Move migration rate limiting and associated logic in NUMAB=2 (current NUMA Balancing based hot page promotion) from scheduler to pghot sub-system to enable broader reuse.
Currently, multiple kernel subsystems detect page accesses independently. This patchset consolidates accesses from these mechanisms by providing:
- A common API for reporting page accesses
- Shared infrastructure for tracking hotness at PFN granularity
- Per-lowertier-node kernel threads for promoting pages.
Here is a brief summary of how this subsystem works:
- Tracks frequency, last access time and accessing node for each recorded access.
- These hotness parameters are maintained on a per-PFN in an unsigned long variable within the existing mem_section data structure.
Bits 0-31 are used to store nid, frequency and time.
Bits 32-62 are unused now.
Bit 63 is used to indicate the page is ready for migration.
- Classifies pages as hot based on configurable thresholds.
- Pages classified as hot are marked as ready for migration using the ready bit.
- Per-lowertier-node kmigrated threads periodically scan the PFNs of lower tier nodes, checking for the migration-ready bit to perform batched migrations.
Four page hotness sources have been integrated with pghot subsystem on experimental basis:
1. IBS
2. klruscand (based on MGLRU page table walks)
3. NUMA Balancing (mode 2).
4. folio_mark_accessed()
Changes in v4
=============
- Addition of folio_mark_accessed() as source to track and promote unmapped page cache pages.
- Per-section indicator for hotness based on which a section is taken up for scanning. This should considerably reduce the scanning effort by kmigrated. The LSB of the pointer used to store the hotness data for each section is reprovisioned as section hotness indicator.
- Added a file under admin-guide to document the usage of pghot sub-system.
- HWhint source IBS is under its own config option now.
- All vmstat counters are under appropriate config options now.
- Most tunables are moved to a dedicated debugfs dir.
- Some code cleanup.
Results
=======
System details
--------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)
$ numactl -H available: 3 nodes (0-2) node 0 cpus: 0-95,192-287 node 0 size: 128460 MB node 1 cpus: 96-191,288-383 node 1 size: 128893 MB node 2 cpus: node 2 size: 257993 MB node distances: node 0 1 2 0: 10 32 50 1: 32 10 60 2: 255 255 10
Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the patched case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the patched case.
pgtscan - Klruscand (MGLRU based PTE A bit scanning) source hwhints - IBS as source
FMA - folio_mark_accessed()
==============================================================
Scenario 1 - Enough memory in toptier and hence only promotion
==============================================================
Microbenchmark details
----------------------
Multi-threaded application with 64 threads that access memory at 4K granularity repetitively and randomly. The number of accesses per thread and the randomness pattern for each thread are fixed beforehand. The accesses are divided into stores and loads.
Benchmark threads run on Node 0, while memory is initially provisioned on CXL node 2 before the accesses start. There are three modes in which the benchmark is run:
Mode 1: Regular 4K page accesses. The memory is provisioned on CXL node using mmap(MAP_POPULATE). 50% loads and 50% stores.
Mode 2: mmapped file 4K accesses. The memory is provisioned on CXL node using mmap(fd, MAP_POPULATE|MAP_SHARED). 100% loads.
Repetitive accesses results in lowertier pages becoming hot and kmigrated detecting and migrating them. The benchmark score is the time taken to finish the accesses in microseconds. The sooner it finishes the better it is. All the numbers shown below are average of 3 runs.
Mode 1 - Time taken (microseconds, lower is better)
------------------------------------------------------
Source Base Patched Change
------------------------------------------------------
NUMAB0 118,986,471 116,240,187 -2.3%
NUMAB2 104,025,651 105,636,591 +1.5%
pgtscan NA 110,800,511 NA
hwhints NA 100,442,082 NA
------------------------------------------------------
Mode 1 - Pages migrated (pgpromote_success)
---------------------------------------
Source Base Patched
---------------------------------------
NUMAB0 0 0
NUMAB2 2097152 2097152
pgtscan NA 2097152
hwhints NA 1232876
---------------------------------------
Mode 2 - Time taken (microseconds, lower is better)
------------------------------------------------------
Source Base Patched Change
------------------------------------------------------
NUMAB0 113,352,595 110,053,021 -2.9%
NUMAB2 72,339,008 84,999,971 +17.5%
pgtscan NA 66,189,266 NA
hwhints NA 71,644,577 NA
------------------------------------------------------
Mode 2 - Pages migrated (pgpromote_success)
---------------------------------------
Source Base Patched
---------------------------------------
NUMAB0 0 0
NUMAB2 2097152 2095978
pgtscan NA 1993077
hwhints NA 2097129
---------------------------------------
==============================================================
Scenario 2 - Toptier memory overcommited, promotion + demotion
==============================================================
Single threaded application that allocates memory on both DRAM and CXL nodes using mmap(MAP_POPULATE). Every 1G region of allocated memory on CXL node is accessed at 4K granularity randomly and repetitively to build up the notion of hotness in the 1GB region that is under access. This should drive promotion. For promotion to work successfully, the DRAM memory that has been provisioned (and not being accessed) should be demoted first. There is enough free memory in the CXL node to for demotions.
In summary, this benchmark creates a memory pressure on DRAM node and does CXL memory accesses to drive both demotion and promotion.
The number of accesses are fixed and hence, the quicker the accessed pages get promoted to DRAM, the sooner the benchmark is expected to finish.
DRAM-node = 1
CXL-node = 2
Initial DRAM alloc ratio = 75%
Allocation-size = 171798691840
Initial DRAM Alloc-size = 128849018880
Initial CXL Alloc-size = 42949672960
Hot-region-size = 1073741824
Nr-regions = 160
Nr-regions DRAM = 120 (provisioned but not accessed)
Nr-hot-regions CXL = 40
Access pattern = random
Access granularity = 4096
Delay b/n accesses = 0
Load/store ratio = 50l50s
THP used = no
Nr accesses = 42949672960
Nr repetitions = 1024
Time taken (microseconds, lower is better)
------------------------------------------------------
Source Base Patched Change
------------------------------------------------------
NUMAB0 61,537,418 59,165,269 -3.8%
NUMAB2 62,070,563 63,087,940 +1.6%
pgtscan NA 66,886,552 NA
hwhints NA 63,35,4394 NA
------------------------------------------------------
Pages migrated (pgpromote_success)
---------------------------------------
Source Base Patched
---------------------------------------
NUMAB0 0 0
NUMAB2 0 0
pgtscan NA 6481483
hwhints NA 304
---------------------------------------
===============================================================
Scenario 3 - Numbers from folio_mark_accessed() (FMA) as source
===============================================================
Single threaded microbenchmark that provisions a file of 2G size on CXL node initially, runs on Node 0 and reads random file pages at 4k granularity iteratively and repetitively. FMA source detects the reads on unmapped page cache pages residing on CXL node and mark them for promotion.
------------------------------------------------------------------
Base Patched Patched
FMA source FMA cource
Disabled Enabled
------------------------------------------------------------------
Time taken(us) 96,511,260 119,332,436 82,807,865pgpromote_success 0 0 524242
------------------------------------------------------------------
Results summary
===============
- The observations from v3 pretty much remain the same for 1st and 2nd scenarios.
- FMA source: Compared to the base kernel, the time taken to complete the file accesses decreases with promotion of file pages in the patched version. However when the FMA source isn't enabled we see a regression compared to base that needs to be investigated.
This v4 patchset applies on top of upstream commit 4941a17751c9 and can be fetched from:
https://github.com/AMDESE/linux-mm/tree/bharata/pghot-rfcv4
v3: https://lore.kernel.org/linux-mm/[email protected]/
v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/
v0: https://lore.kernel.org/linux-mm/[email protected]/
TODOs
=====
- Check if the page is still within the hotness time window when kmigrated gets to it.
- Bulk access reporting may be desirable for sources like IBS.
- Take care of memory hotplug for allocation/freeing of mem_section->hot_map.
- Currently I am defaulting to node 0 if target NID isn't specified by the source. The best fallback target node may have to determined dynamically.
- Provide compatibility alias for the sysctls moved from sched to pghot.
- Wider testing and benchmark coverage.
- Address Ying Huang's comment about merging migrate_misplaced_folio() and migrate_misplaced_folios_batch() and correctly handling memcg stats counting properly in the latter.
Bharata B Rao (6):
mm: migrate: Allow misplaced migration without VMA too
mm: Hot page tracking and promotion
x86: ibs: In-kernel IBS driver for memory access profiling
x86: ibs: Enable IBS profiling for memory accesses
mm: sched: Move hot page promotion from NUMAB=2 to pghot tracking
mm: pghot: Add folio_mark_accessed() as hotness source
Gregory Price (1):
migrate: implement migrate_misplaced_folios_batch
Kinsey Ho (2):
mm: mglru: generalize page table walk
mm: klruscand: use mglru scanning for page promotion
Documentation/admin-guide/mm/pghot.txt | 64 +++ arch/x86/events/amd/ibs.c | 10 + arch/x86/include/asm/entry-common.h | 3 + arch/x86/include/asm/hardirq.h | 2 + arch/x86/include/asm/msr-index.h | 16 + arch/x86/mm/Makefile | 1 + arch/x86/mm/ibs.c | 348 ++++++++++++++++ include/linux/migrate.h | 6 + include/linux/mmzone.h | 19 + include/linux/pghot.h | 87 ++++ include/linux/vm_event_item.h | 26 ++ kernel/sched/debug.c | 1 - kernel/sched/fair.c | 152 +------ mm/Kconfig | 32 ++ mm/Makefile | 2 + mm/huge_memory.c | 26 +- mm/internal.h | 4 + mm/klruscand.c | 110 +++++ mm/memory.c | 31 +- mm/migrate.c | 41 +- mm/mm_init.c | 10 + mm/pghot-debug.c | 187 +++++++++ mm/pghot.c | 533 +++++++++++++++++++++++++ mm/swap.c | 8 + mm/vmscan.c | 181 ++++++--- mm/vmstat.c | 26 ++ 26 files changed, 1686 insertions(+), 240 deletions(-) create mode 100644 Documentation/admin-guide/mm/pghot.txt create mode 100644 arch/x86/mm/ibs.c create mode 100644 include/linux/pghot.h create mode 100644 mm/klruscand.c create mode 100644 mm/pghot-debug.c create mode 100644 mm/pghot.c
Signature · 1 line
2.34.1
RFC PATCH v4 1/9 mm: migrate: Allow misplaced migration without VMA too
We want isolation of misplaced folios to work in contexts where VMA isn't available, typically when performing migrations from a kernel thread context. In order to prepare for that allow migrate_misplaced_folio_prepare() to be called with a NULL VMA.
When migrate_misplaced_folio_prepare() is called with non-NULL VMA, it will check if the folio is mapped shared and that requires holding PTL lock. This path isn't taken when the function is invoked with NULL VMA (migration outside of process context). Hence for such cases, it is not necessary this function be called with PTL lock held.
--- mm/migrate.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
mm/migrate.c +3 -2
@@ -2644,7 +2644,8 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,/* * Prepare for calling migrate_misplaced_folio() by isolating the folio if -* permitted. Must be called with the PTL still held. +* permitted. Must be called with the PTL still held if called with a non-NULL +* vma. */ int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)
@@ -2661,7 +2662,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,* See folio_maybe_mapped_shared() on possible imprecision * when we cannot easily detect if a folio is shared. */ -if ((vma->vm_flags &VM_EXEC ) &&folio_maybe_mapped_shared (folio)) +if (vma && (vma->vm_flags &VM_EXEC ) &&folio_maybe_mapped_shared (folio))return -EACCES ;/*
Signature · 1 line
2.34.1
RFC PATCH v4 2/9 migrate: implement migrate_misplaced_folios_batch
In reply to Bharata B RaoFrom: Gregory Price <[email protected]>
Tiered memory systems often require migrating multiple folios at once. Currently, migrate_misplaced_folio() handles only one folio per call, which is inefficient for batch operations. This patch introduces migrate_misplaced_folios_batch(), a batch variant that leverages migrate_pages() internally for improved performance.
The caller must isolate folios beforehand using migrate_misplaced_folio_prepare(). On return, the folio list will be empty regardless of success or failure.
--- include/linux/migrate.h | 6 ++++++ mm/migrate.c | 36 ++++++++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+)
include/linux/migrate.h +6 -0
@@ -103,6 +103,7 @@ static inline int set_movable_ops(const struct movable_operations *ops, enum pagint migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node);int migrate_misplaced_folio (struct folio *folio,int node); +int migrate_misplaced_folios_batch (struct list_head *foliolist,int node);#else static inline int migrate_misplaced_folio_prepare (struct folio *folio,struct vm_area_struct *vma,int node)
@@ -113,6 +114,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node) {return -EAGAIN ;/* can't migrate now */ } +static inline int migrate_misplaced_folios_batch (struct list_head *foliolist, +int node) +{ +return -EAGAIN ;/* can't migrate now */ +}#endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_MIGRATION
mm/migrate.c +36 -0
@@ -2739,5 +2739,41 @@ int migrate_misplaced_folio(struct folio *folio, int node)BUG_ON (!list_empty (&migratepages));return nr_remaining ? -EAGAIN :0 ; } + +/** +* migrate_misplaced_folios_batch - Batch variant of migrate_misplaced_folio. +* Attempts to migrate a folio list to the specified destination. +* @folio_list - Isolated list of folios to be batch-migrated. +* @node - The NUMA node ID to where the folios should be migrated. +* +* Caller is expected to have isolated the folios by calling +* migrate_misplaced_folio_prepare(), which will result in an +* elevated reference count on the folio. +* +* This function will un-isolate the folios, dereference them, and +* remove them from the list before returning. +* +* Return: 0 on success and -EAGAIN on failure or partial migration. +* On return, @folio_list will be empty regardless of success/failure. +*/ +int migrate_misplaced_folios_batch (struct list_head *folio_list,int node) +{ +pg_data_t *pgdat =NODE_DATA (node); +unsigned int nr_succeeded; +int nr_remaining; + + nr_remaining =migrate_pages (folio_list, alloc_misplaced_dst_folio, +NULL , node,MIGRATE_ASYNC , +MR_NUMA_MISPLACED , &nr_succeeded); +if (nr_remaining) +putback_movable_pages (folio_list); + +if (nr_succeeded) { +count_vm_numa_events (NUMA_PAGE_MIGRATE , nr_succeeded); +mod_node_page_state (pgdat,PGPROMOTE_SUCCESS , nr_succeeded); + } +BUG_ON (!list_empty (folio_list)); +return nr_remaining ? -EAGAIN :0 ; +}#endif /* CONFIG_NUMA_BALANCING */ #endif /* CONFIG_NUMA */
Signature · 1 line
2.34.1
RFC PATCH v4 3/9 mm: Hot page tracking and promotion
In reply to Bharata B RaoThis introduces a sub-system for collecting memory access information from different sources. It maintains the hotness information based on the access history and time of access.
Additionally, it provides per-lowertier-node kernel threads (named kmigrated) that periodically promote the pages that are eligible for promotion.
Sub-systems that generate hot page access info can report that using this API:
int pghot_record_access(unsigned long pfn, int nid, int src,
unsigned long time)
@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (sub-system) that generated the
access info
@time: The access time in jiffies
Some temperature sources may not provide the nid from which the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. For such sources, the default toptier node to which such pages should be promoted is hard coded.
The hotness information is stored for every page of lower tier memory in an unsigned long variable that is part of mem_section data structure.
kmigrated is a per-lowertier-node kernel thread that migrates the folios marked for migration in batches. Each kmigrated thread walks the PFN range spanning its node and checks for potential migration candidates.
A bunch of tunables for enabling different hotness sources, setting target_nid, frequency threshold are provided in debugfs.
--- Documentation/admin-guide/mm/pghot.txt | 63 ++++ include/linux/mmzone.h | 14 + include/linux/pghot.h | 71 +++++ include/linux/vm_event_item.h | 6 + mm/Kconfig | 11 + mm/Makefile | 1 + mm/mm_init.c | 10 + mm/pghot-debug.c | 180 +++++++++++ mm/pghot.c | 402 +++++++++++++++++++++++++ mm/vmstat.c | 6 + 10 files changed, 764 insertions(+) create mode 100644 Documentation/admin-guide/mm/pghot.txt create mode 100644 include/linux/pghot.h create mode 100644 mm/pghot-debug.c create mode 100644 mm/pghot.c
Documentation/admin-guide/mm/pghot.txt new file +63 -0
@@ -0,0 +1,63 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================================= +PGHOT: Hot Page Tracking Tunables +================================= + +Overview +======== +The PGHOT subsystem tracks frequently accessed pages in lower-tier memory and +promotes them to faster tiers. It uses per-PFN hotness metadata and asynchronous +migration via per-node kernel threads (kmigrated). + +This document describes tunables available via **debugfs** and **sysctl** for +PGHOT. + +Debugfs Interface +================= +Path: /sys/kernel/debug/pghot/ + +1. **enabled_sources** + - Bitmask to enable/disable hotness sources. + - Bits: + - 0: Hardware hints (value 0x1) + - 1: Page table scan (value 0x2) + - 2: Hint faults (value 0x4) + - Default: 0 (disabled) + - Example: + # echo 0x7 > /sys/kernel/debug/pghot/enabled_sources + Enables all sources. + +2. **target_nid** + - NUMA node ID to which hot pages should be promoted when source does not provide nid. + - Default: 0 + - Example: + # echo 1 > /sys/kernel/debug/pghot/target_nid + +3. **freq_threshold** + - Minimum access frequency before a page is marked ready for promotion. + - Range: 1 to 8 + - Default: 2 + - Example: + # echo 3 > /sys/kernel/debug/pghot/freq_threshold + +4. **kmigrated_sleep_ms** + - Sleep interval (ms) for kmigrated thread between scans. + - Default: 100 + +5. **kmigrated_batch_nr** + - Maximum number of folios migrated in one batch. + - Default: 512 + +Sysctl Interface +================ +1. pghot_promote_freq_window_ms + +Path: /proc/sys/vm/pghot_promote_freq_window_ms + +- Controls the time window (in ms) for counting access frequency. A page is + considered hot only when **freq_threshold** number of accesses occur with + this time period. +- Default: 5000 (5 seconds) +- Example: + # sysctl vm.pghot_promote_freq_window_ms=3000
include/linux/mmzone.h +14 -0
@@ -1068,6 +1068,7 @@ enum pgdat_flags {* many pages under writeback */ PGDAT_RECLAIM_LOCKED ,/* prevents concurrent reclaim */ + PGDAT_KMIGRATED_ACTIVATE, /* activates kmigrated */ }; enum zone_flags {
@@ -1522,6 +1523,10 @@ typedef struct pglist_data {#ifdef CONFIG_MEMORY_FAILURE struct memory_failure_stats mf_stats;#endif +#ifdef CONFIG_PGHOT +struct task_struct *kmigrated; +wait_queue_head_t kmigrated_wait; +#endif } pg_data_t;#define node_present_pages (nid) (NODE_DATA(nid)->node_present_pages)
@@ -1920,12 +1925,21 @@ struct mem_section {unsigned long section_mem_map;struct mem_section_usage *usage; +#ifdef CONFIG_PGHOT +/* +* Per-PFN hotness data for this section. +*/ +unsigned long *hot_map; +#endif #ifdef CONFIG_PAGE_EXTENSION /* * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use * section. (see page_ext.h about this.) */ struct page_ext *page_ext; +#endif +#if (defined(CONFIG_PGHOT ) && !defined(CONFIG_PAGE_EXTENSION )) || \ + (!defined(CONFIG_PGHOT ) && defined(CONFIG_PAGE_EXTENSION ))unsigned long pad;#endif /*
include/linux/pghot.h new file +71 -0
71 lines not on this page · raw message
include/linux/vm_event_item.h +6 -0
@@ -188,6 +188,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, KSTACK_REST,#endif #endif /* CONFIG_DEBUG_STACK_USAGE */ +#ifdef CONFIG_PGHOT +PGHOT_RECORDED_ACCESSES , +PGHOT_RECORD_HWHINTS , +PGHOT_RECORD_PGTSCANS , +PGHOT_RECORD_HINTFAULTS , +#endif /* CONFIG_PGHOT */NR_VM_EVENT_ITEMS };
mm/Kconfig +11 -0
@@ -1379,6 +1379,17 @@ config PT_RECLAIM config FIND_NORMAL_PAGE def_bool n +config PGHOT + bool "Hot page tracking and promotion" + def_bool n + depends on NUMA && MIGRATION && SPARSEMEM && MMU + help + A sub-system to track page accesses in lower tier memory and + maintain hot page information. Promotes hot pages from lower + tiers to top tier by using the memory access information provided + by various sources. Asynchronous promotion is done by per-node + kernel threads. + source "mm/damon/Kconfig" endmenu
mm/Makefile +1 -0
@@ -146,3 +146,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o +obj-$(CONFIG_PGHOT) += pghot.o
mm/mm_init.c +10 -0
@@ -1401,6 +1401,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)static void pgdat_init_kcompactd (struct pglist_data *pgdat) {}#endif +#ifdef CONFIG_PGHOT +static void pgdat_init_kmigrated (struct pglist_data *pgdat) +{ +init_waitqueue_head (&pgdat->kmigrated_wait); +} +#else +static inline void pgdat_init_kmigrated (struct pglist_data *pgdat) {} +#endif +static void __meminit pgdat_init_internals (struct pglist_data *pgdat) {int i;
@@ -1410,6 +1419,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)pgdat_init_split_queue (pgdat);pgdat_init_kcompactd (pgdat); +pgdat_init_kmigrated (pgdat);init_waitqueue_head (&pgdat->kswapd_wait);init_waitqueue_head (&pgdat->pfmemalloc_wait);
mm/pghot-debug.c new file +180 -0
180 lines not on this page · raw message
mm/pghot.c new file +402 -0
402 lines not on this page · raw message
mm/vmstat.c +6 -0
@@ -1496,6 +1496,12 @@ const char * const vmstat_text[] = { [I(KSTACK_REST)] = "kstack_rest",#endif #endif +#ifdef CONFIG_PGHOT + [I(PGHOT_RECORDED_ACCESSES)] = "pghot_recorded_accesses", + [I(PGHOT_RECORD_HWHINTS)] = "pghot_recorded_hwhints", + [I(PGHOT_RECORD_PGTSCANS)] = "pghot_recorded_pgtscans", + [I(PGHOT_RECORD_HINTFAULTS)] = "pghot_recorded_hintfaults", +#endif /* CONFIG_PGHOT */ #undef I#endif /* CONFIG_VM_EVENT_COUNTERS */ };
Signature · 1 line
2.34.1
Re: [RFC PATCH v4 3/9] mm: Hot page tracking and promotion
Quoted 39 lines · Bharata B Rao
This introduces a sub-system for collecting memory access information from different sources. It maintains the hotness information based on the access history and time of access.
Additionally, it provides per-lowertier-node kernel threads (named kmigrated) that periodically promote the pages that are eligible for promotion.
Sub-systems that generate hot page access info can report that using this API:
int pghot_record_access(unsigned long pfn, int nid, int src,
unsigned long time)@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (sub-system) that generated the
access info
@time: The access time in jiffiesSome temperature sources may not provide the nid from which the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. For such sources, the default toptier node to which such pages should be promoted is hard coded.
The hotness information is stored for every page of lower tier memory in an unsigned long variable that is part of
mem_sectiondata structure.kmigrated is a per-lowertier-node kernel thread that migrates the folios marked for migration in batches. Each kmigrated thread walks the PFN range spanning its node and checks for potential migration candidates.
A bunch of tunables for enabling different hotness sources, setting
target_nid, frequency threshold are provided in debugfs.Signed-off-by Bharata B Rao <[email protected]>
<snip>
Quoted 38 lines
+++
b/include/linux/pghot.hinclude/linux/pghot.h new file +36
-0@@ -0,0 +1,71 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_PGHOT_H +#define _LINUX_PGHOT_H + +/* Page hotness temperature sources */ +enum pghot_src { +PGHOT_HW_HINTS , +PGHOT_PGTABLE_SCAN , +PGHOT_HINT_FAULT , +}; + +#ifdef CONFIG_PGHOT +/* +* Bit positions to enable individual sources in pghot/records_enabled +* of debugfs. +*/ +enum pghot_src_enabed { +PGHOT_HWHINTS_BIT =0 , +PGHOT_PGTSCAN_BIT , +PGHOT_HINTFAULT_BIT , +PGHOT_MAX_BIT +}; + +#define PGHOT_HWHINTS_ENABLED BIT(PGHOT_HWHINTS_BIT) +#define PGHOT_PGTSCAN_ENABLED BIT(PGHOT_PGTSCAN_BIT) +#define PGHOT_HINTFAULT_ENABLED BIT(PGHOT_HINTFAULT_BIT) +#define PGHOT_SRC_ENABLED_MASK GENMASK(PGHOT_MAX_BIT - 1, 0) + +#define PGHOT_DEFAULT_FREQ_WINDOW (5 * MSEC_PER_SEC) +#define PGHOT_DEFAULT_FREQ_THRESHOLD 2 + +#define KMIGRATED_DEFAULT_SLEEP_MS 100 +#define KMIGRATED_DEFAULT_BATCH_NR 512 + +#define PGHOT_DEFAULT_NODE 0 +
include/linux/pghot.h line 46
Quoted 10 lines
include/linux/pghot.h new file +10
-0+/* +* Bits 0-31 are used to store nid, frequency and time. +* Bits 32-62 are unused now. +* Bit 63 is used to indicate the page is ready for migration. +*/ +#define PGHOT_MIGRATE_READY 63 + +#define PGHOT_NID_WIDTH 10 +#define PGHOT_FREQ_WIDTH 3 +/* time is stored in 19 bits which can represent up to 8.73s with HZ=1000 */
If we consider HZ = 1000 then using 19 bit time is coming 8.73 mins. I think by mistake you commented as 8.73 secs.
Suggetion:
If we are targeting to promote page in ~8 secs then 13 bits would be enough, that way we can handle hotness using 32 bits per pfn insead of 64 bits.
#define PGHOT_MIGRATE_READY 31
#define PGHOT_NID_WIDTH 10
#define PGHOT_FREQ_WIDTH 3
/* time is stored in 13 bits which can represent up to 8.19s with HZ=1000 */
#define PGHOT_TIME_WIDTH 13
include/linux/pghot.h line 55
Quoted 9 lines
include/linux/pghot.h new file +9
-0+#define PGHOT_TIME_WIDTH 19 + +#define PGHOT_NID_SHIFT 0 +#define PGHOT_FREQ_SHIFT (PGHOT_NID_SHIFT + PGHOT_NID_WIDTH) +#define PGHOT_TIME_SHIFT (PGHOT_FREQ_SHIFT + PGHOT_FREQ_WIDTH) + +#define PGHOT_NID_MASK ((1UL << PGHOT_NID_SHIFT) - 1) +#define PGHOT_FREQ_MASK ((1UL << PGHOT_FREQ_SHIFT) - 1) +#define PGHOT_TIME_MASK ((1UL << PGHOT_TIME_SHIFT) - 1)
Mask generation of freq, nid and time seems not correct. It should be
#define PGHOT_NID_MASK ((1UL << PGHOT_NID_WIDTH) - 1) #define PGHOT_FREQ_MASK ((1UL << PGHOT_FREQ_WIDTH) - 1) #define PGHOT_TIME_MASK ((1UL << PGHOT_TIME_WIDTH) - 1)
Can you please have a look?
Regards,
Alok Rathore
Re: [RFC PATCH v4 3/9] mm: Hot page tracking and promotion
On 22-Dec-25 3:47 PM, Alok Rathore wrote:
<snip>
Quoted 7 lines
include/linux/pghot.h line 46
Quoted 4 lines
include/linux/pghot.h new file +4
-0+ +#define PGHOT_NID_WIDTH 10 +#define PGHOT_FREQ_WIDTH 3 +/* time is stored in 19 bits which can represent up to 8.73s with HZ=1000 */ If we consider HZ = 1000 then using 19 bit time is coming 8.73 mins. I think by mistake you commented as 8.73 secs.
Yes you are right.
Quoted 4 lines
Suggetion:
If we are targeting to promote page in ~8 secs then 13 bits would be enough, that way we can handle hotness using 32 bits per pfn insead of 64 bits.
Right, we will have to figure out how many seconds would be ideal and it needn't be in minutes and hence we should be able to fit everything within 32 bits.
Based on the discussions in Dec 18 Linux Memory Hotness and Promotion call, I will check if I can reduce the overhead further and have a configurable precision mode.
Quoted 23 lines
#define PGHOT_MIGRATE_READY 31
#define PGHOT_NID_WIDTH 10
#define PGHOT_FREQ_WIDTH 3
/* time is stored in 13 bits which can represent up to 8.19s with HZ=1000 */ #define PGHOT_TIME_WIDTH 13include/linux/pghot.h line 55
Quoted 9 lines
include/linux/pghot.h new file +9
-0+#define PGHOT_TIME_WIDTH 19 + +#define PGHOT_NID_SHIFT 0 +#define PGHOT_FREQ_SHIFT (PGHOT_NID_SHIFT + PGHOT_NID_WIDTH) +#define PGHOT_TIME_SHIFT (PGHOT_FREQ_SHIFT + PGHOT_FREQ_WIDTH) + +#define PGHOT_NID_MASK ((1UL << PGHOT_NID_SHIFT) - 1) +#define PGHOT_FREQ_MASK ((1UL << PGHOT_FREQ_SHIFT) - 1) +#define PGHOT_TIME_MASK ((1UL << PGHOT_TIME_SHIFT) - 1)Mask generation of freq, nid and time seems not correct. It should be
#define PGHOT_NID_MASK ((1UL <<PGHOT_NID_WIDTH) - 1) #define PGHOT_FREQ_MASK ((1UL <<PGHOT_FREQ_WIDTH) - 1) #define PGHOT_TIME_MASK ((1UL <<PGHOT_TIME_WIDTH) - 1)
Quoted 1 line
Can you please have a look?
Thanks for catching this. Will fix.
Regards,
Bharata.
RFC PATCH v4 4/9 x86: ibs: In-kernel IBS driver for memory access profiling
In reply to Bharata B RaoUse IBS (Instruction Based Sampling) feature present in AMD processors for memory access tracking. The access information obtained from IBS via NMI is fed to pghot sub-system for futher action.
In addition to many other information related to the memory access, IBS provides physical (and virtual) address of the access and indicates if the access came from slower tier. Only memory accesses originating from slower tiers are further acted upon by this driver.
The samples are initially accumulated in percpu buffers which are flushed to pghot hot page tracking mechanism using irq_work.
TODO: Many counters are added to vmstat just as debugging aid for now.
About IBS
---------
IBS can be programmed to provide data about instruction execution periodically. This is done by programming a desired sample count (number of ops) in a control register. When the programmed number of ops are dispatched, a micro-op gets tagged, various information about the tagged micro-op's execution is populated in IBS execution MSRs and an interrupt is raised. While IBS provides a lot of data for each sample, for the purpose of memory access profiling, we are interested in linear and physical address of the memory access that reached DRAM. Recent AMD processors provide further filtering where it is possible to limit the sampling to those ops that had an L3 miss which greately reduces the non-useful samples.
While IBS provides capability to sample instruction fetch and execution, only IBS execution sampling is used here to collect data about memory accesses that occur during the instruction execution.
More information about IBS is available in Sec 13.3 of AMD64 Architecture Programmer's Manual, Volume 2:System Programming which is present at:
https://bugzilla.kernel.org/attachment.cgi?id=288923
Information about MSRs used for programming IBS can be found in Sec 2.1.14.4 of PPR Vol 1 for AMD Family 19h Model 11h B1 which is currently present at:
https://www.amd.com/system/files/TechDocs/55901_0.25.zip
--- arch/x86/events/amd/ibs.c | 10 + arch/x86/include/asm/msr-index.h | 16 ++ arch/x86/mm/Makefile | 1 + arch/x86/mm/ibs.c | 316 +++++++++++++++++++++++++++++++ include/linux/pghot.h | 6 + include/linux/vm_event_item.h | 19 ++ mm/Kconfig | 13 ++ mm/vmstat.c | 19 ++ 8 files changed, 400 insertions(+) create mode 100644 arch/x86/mm/ibs.c
arch/x86/events/amd/ibs.c +10 -0
@@ -13,6 +13,7 @@#include <linux/ptrace.h> #include <linux/syscore_ops.h> #include <linux/sched/clock.h> +#include <linux/pghot.h> #include <asm/apic.h> #include <asm/msr.h>
@@ -1756,6 +1757,15 @@ static __init int amd_ibs_init(void) {u32 caps; +/* +* TODO: Find a clean way to disable perf IBS so that IBS +* can be used for memory access profiling. +*/ +if (hwmem_access_profiler_inuse ()) { +pr_info ("IBS isn't available for perf use\n" ); +return 0 ; + } + caps =__get_ibs_caps ();if (!caps)return -ENODEV ;/* ibs not supported by the cpu */
arch/x86/include/asm/msr-index.h +16 -0
@@ -760,6 +760,22 @@/* AMD Last Branch Record MSRs */ #define MSR_AMD64_LBR_SELECT 0xc000010e +/* AMD IBS MSR bits */ +#define MSR_AMD64_IBSOPDATA2_DATASRC 0x7 +#define MSR_AMD64_IBSOPDATA2_DATASRC_LCL_CACHE 0x1 +#define MSR_AMD64_IBSOPDATA2_DATASRC_PEER_CACHE_NEAR 0x2 +#define MSR_AMD64_IBSOPDATA2_DATASRC_DRAM 0x3 +#define MSR_AMD64_IBSOPDATA2_DATASRC_FAR_CCX_CACHE 0x5 +#define MSR_AMD64_IBSOPDATA2_DATASRC_EXT_MEM 0x8 +#define MSR_AMD64_IBSOPDATA2_RMTNODE 0x10 + +#define MSR_AMD64_IBSOPDATA3_LDOP BIT_ULL(0) +#define MSR_AMD64_IBSOPDATA3_STOP BIT_ULL(1) +#define MSR_AMD64_IBSOPDATA3_DCMISS BIT_ULL(7) +#define MSR_AMD64_IBSOPDATA3_LADDR_VALID BIT_ULL(17) +#define MSR_AMD64_IBSOPDATA3_PADDR_VALID BIT_ULL(18) +#define MSR_AMD64_IBSOPDATA3_L2MISS BIT_ULL(20) +/* Zen4 */ #define MSR_ZEN4_BP_CFG 0xc001102e#define MSR_ZEN4_BP_CFG_BP_SPEC_REDUCE_BIT 4
arch/x86/mm/Makefile +1 -0
@@ -57,3 +57,4 @@ obj-$(CONFIG_X86_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_amd.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_boot.o +obj-$(CONFIG_HWMEM_PROFILER) += ibs.o
arch/x86/mm/ibs.c new file +316 -0
316 lines not on this page · raw message
include/linux/pghot.h +6 -0
@@ -2,6 +2,12 @@#ifndef _LINUX_PGHOT_H#define _LINUX_PGHOT_H +#ifdef CONFIG_HWMEM_PROFILER +bool hwmem_access_profiler_inuse (void ); +#else +static inline bool hwmem_access_profiler_inuse (void ) {return false; } +#endif +/* Page hotness temperature sources */ enum pghot_src { PGHOT_HW_HINTS,
include/linux/vm_event_item.h +19 -0
@@ -193,6 +193,25 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,PGHOT_RECORD_HWHINTS ,PGHOT_RECORD_PGTSCANS ,PGHOT_RECORD_HINTFAULTS , +#ifdef CONFIG_HWMEM_PROFILER +HWHINT_NR_EVENTS , +HWHINT_KERNEL , +HWHINT_KTHREAD , +HWHINT_NON_LOAD_STORES , +HWHINT_DC_L2_HITS , +HWHINT_LOCAL_L3L1L2 , +HWHINT_LOCAL_PEER_CACHE_NEAR , +HWHINT_FAR_CACHE_HITS , +HWHINT_DRAM_ACCESSES , +HWHINT_CXL_ACCESSES , +HWHINT_REMOTE_NODE , +HWHINT_LADDR_INVALID , +HWHINT_KERNEL_ADDR , +HWHINT_PADDR_INVALID , +HWHINT_NON_LRU , +HWHINT_BUFFER_FULL , +HWHINT_USEFUL_SAMPLES , +#endif /* CONFIG_HWMEM_PROFILER */ #endif /* CONFIG_PGHOT */NR_VM_EVENT_ITEMS };
mm/Kconfig +13 -0
@@ -1390,6 +1390,19 @@ config PGHOT by various sources. Asynchronous promotion is done by per-node kernel threads. +config HWMEM_PROFILER + bool "HW based memory access profiling" + def_bool n + depends on PGHOT + depends on X86_64 + help + Some hardware platforms are capable of providing memory access + information in direct and actionable manner. Instruction Based + Sampling (IBS) present on AMD Zen CPUs in one such example. + Memory accesses obtained via such HW based mechanisms are + rolled up to PGHOT sub-system for further action like hot page + promotion or NUMA Balancing + source "mm/damon/Kconfig" endmenu
mm/vmstat.c +19 -0
@@ -1501,6 +1501,25 @@ const char * const vmstat_text[] = { [I(PGHOT_RECORD_HWHINTS)] = "pghot_recorded_hwhints", [I(PGHOT_RECORD_PGTSCANS)] = "pghot_recorded_pgtscans", [I(PGHOT_RECORD_HINTFAULTS)] = "pghot_recorded_hintfaults", +#ifdef CONFIG_HWMEM_PROFILER + [I(HWHINT_NR_EVENTS)] = "hwhint_nr_events", + [I(HWHINT_KERNEL)] = "hwhint_kernel", + [I(HWHINT_KTHREAD)] = "hwhint_kthread", + [I(HWHINT_NON_LOAD_STORES)] = "hwhint_non_load_stores", + [I(HWHINT_DC_L2_HITS)] = "hwhint_dc_l2_hits", + [I(HWHINT_LOCAL_L3L1L2)] = "hwhint_local_l3l1l2", + [I(HWHINT_LOCAL_PEER_CACHE_NEAR)] = "hwhint_local_peer_cache_near", + [I(HWHINT_FAR_CACHE_HITS)] = "hwhint_far_cache_hits", + [I(HWHINT_DRAM_ACCESSES)] = "hwhint_dram_accesses", + [I(HWHINT_CXL_ACCESSES)] = "hwhint_cxl_accesses", + [I(HWHINT_REMOTE_NODE)] = "hwhint_remote_node", + [I(HWHINT_LADDR_INVALID)] = "hwhint_invalid_laddr", + [I(HWHINT_KERNEL_ADDR)] = "hwhint_kernel_addr", + [I(HWHINT_PADDR_INVALID)] = "hwhint_invalid_paddr", + [I(HWHINT_NON_LRU)] = "hwhint_non_lru", + [I(HWHINT_BUFFER_FULL)] = "hwhint_buffer_full", + [I(HWHINT_USEFUL_SAMPLES)] = "hwhint_useful_samples", +#endif /* CONFIG_HWMEM_PROFILER */ #endif /* CONFIG_PGHOT */ #undef I #endif /* CONFIG_VM_EVENT_COUNTERS */
Signature · 1 line
2.34.1
RFC PATCH v4 5/9 x86: ibs: Enable IBS profiling for memory accesses
In reply to Bharata B RaoEnable IBS memory access data collection for user memory accesses by programming the required MSRs. The profiling is turned ON only for user mode execution and turned OFF for kernel mode execution. Profiling is explicitly disabled for NMI handler too.
TODOs:
- IBS sampling rate is kept fixed for now.
- Arch/vendor separation/isolation of the code needs relook.
--- arch/x86/include/asm/entry-common.h | 3 +++ arch/x86/include/asm/hardirq.h | 2 ++ arch/x86/mm/ibs.c | 32 +++++++++++++++++++++++++++++ include/linux/pghot.h | 4 ++++ 4 files changed, 41 insertions(+)
arch/x86/include/asm/entry-common.h +3 -0
@@ -4,6 +4,7 @@#include <linux/randomize_kstack.h> #include <linux/user-return-notifier.h> +#include <linux/pghot.h> #include <asm/nospec-branch.h> #include <asm/io_bitmap.h>
@@ -13,6 +14,7 @@/* Check that the stack and regs on entry from user mode are sane. */ static __always_inline void arch_enter_from_user_mode(struct pt_regs *regs) { +hwmem_access_profiling_stop ();if (IS_ENABLED (CONFIG_DEBUG_ENTRY )) {/* * Make sure that the entry code gave us a sensible EFLAGS
@@ -106,6 +108,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs, static __always_inline void arch_exit_to_user_mode(void) {amd_clear_divider (); +hwmem_access_profiling_start (); }#define arch_exit_to_user_mode arch_exit_to_user_mode
arch/x86/include/asm/hardirq.h +2 -0
@@ -91,4 +91,6 @@ static __always_inline bool kvm_get_cpu_l1tf_flush_l1d(void) static __always_inline void kvm_set_cpu_l1tf_flush_l1d(void) { }#endif /* IS_ENABLED(CONFIG_KVM_INTEL) */ +#define arch_nmi_enter () hwmem_access_profiling_stop() +#define arch_nmi_exit () hwmem_access_profiling_start()#endif /* _ASM_X86_HARDIRQ_H */
arch/x86/mm/ibs.c +32 -0
@@ -16,6 +16,7 @@ static u64 ibs_config __read_mostly;static u32 ibs_caps;#define IBS_NR_SAMPLES 150 +#define IBS_SAMPLE_PERIOD 10000/* * Basic access info captured for each memory access.
@@ -43,6 +44,36 @@ struct ibs_sample_pcpu __percpu *ibs_s;static struct work_struct ibs_work;static struct irq_work ibs_irq_work; +void hwmem_access_profiling_stop (void ) +{ +u64 ops_ctl; + +if (!hwmem_access_profiling) +return ; + +rdmsrl (MSR_AMD64_IBSOPCTL , ops_ctl); +wrmsrl (MSR_AMD64_IBSOPCTL , ops_ctl & ~IBS_OP_ENABLE ); +} + +void hwmem_access_profiling_start (void ) +{ +u64 config =0 ; +unsigned int period =IBS_SAMPLE_PERIOD ; + +if (!hwmem_access_profiling) +return ; + +/* Disable IBS for kernel thread */ +if (!current->mm) + goto out; + + config = (period >>4 ) &IBS_OP_MAX_CNT ; + config |= (period &IBS_OP_MAX_CNT_EXT_MASK ); + config |= ibs_config; +out: +wrmsrl (MSR_AMD64_IBSOPCTL , config); +} +bool hwmem_access_profiler_inuse (void ) {return hwmem_access_profiling;
@@ -309,6 +340,7 @@ static int __init ibs_access_profiling_init(void) x86_amd_ibs_access_profile_startup, x86_amd_ibs_access_profile_teardown); + hwmem_access_profiling = true;pr_info ("IBS setup for memory access profiling\n" );return 0 ; }
include/linux/pghot.h +4 -0
@@ -4,8 +4,12 @@#ifdef CONFIG_HWMEM_PROFILER bool hwmem_access_profiler_inuse (void ); +void hwmem_access_profiling_start (void ); +void hwmem_access_profiling_stop (void );#else static inline bool hwmem_access_profiler_inuse (void ) {return false; } +static inline void hwmem_access_profiling_start (void ) {} +static inline void hwmem_access_profiling_stop (void ) {}#endif /* Page hotness temperature sources */
Signature · 1 line
2.34.1
RFC PATCH v4 6/9 mm: mglru: generalize page table walk
In reply to Bharata B RaoFrom: Kinsey Ho <[email protected]>
Refactor the existing MGLRU page table walking logic to make it resumable.
Additionally, introduce two hooks into the MGLRU page table walk:
accessed callback and flush callback. The accessed callback is called for each accessed page detected via the scanned accessed bit. The flush callback is called when the accessed callback reports that a flush is required. This allows for processing pages in batches for efficiency.
With a generalised page table walk, introduce a new scan function which repeatedly scans on the same young generation and does not add a new young generation.
--- include/linux/mmzone.h | 5 ++ mm/internal.h | 4 + mm/vmscan.c | 181 +++++++++++++++++++++++++++++++---------- 3 files changed, 145 insertions(+), 45 deletions(-)
include/linux/mmzone.h +5 -0
@@ -548,6 +548,8 @@ struct lru_gen_mm_walk {unsigned long seq;/* the next address within an mm to scan */ unsigned long next_addr; +/* called for each accessed pte/pmd */ +bool (*accessed_cb)(unsigned long pfn);/* to batch promoted pages */ int nr_pages[MAX_NR_GENS ][ANON_AND_FILE ][MAX_NR_ZONES ];/* to batch the mm stats */
@@ -555,6 +557,9 @@ struct lru_gen_mm_walk {/* total batched items */ int batched;int swappiness; +/* for the pmd under scanning */ +int nr_young_pte; +int nr_total_pte;bool force_scan; };
mm/internal.h +4 -0
@@ -538,6 +538,10 @@ extern unsigned long highest_memmap_pfn;bool folio_isolate_lru (struct folio *folio);void folio_putback_lru (struct folio *folio);extern void reclaim_throttle (pg_data_t *pgdat,enum vmscan_throttle_state reason); +void set_task_reclaim_state (struct task_struct *task, +struct reclaim_state *rs); +void lru_gen_scan_lruvec (struct lruvec *lruvec,unsigned long seq, +bool (*accessed_cb)(unsigned long ),void (*flush_cb)(void ));#ifdef CONFIG_NUMA int user_proactive_reclaim (char *buf,struct mem_cgroup *memcg,pg_data_t *pgdat);
mm/vmscan.c +136 -45
361 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC PATCH v4 7/9 mm: klruscand: use mglru scanning for page promotion
In reply to Bharata B RaoFrom: Kinsey Ho <[email protected]>
Introduce a new kernel daemon, klruscand, that periodically invokes the MGLRU page table walk. It leverages the new callbacks to gather access information and forwards it to pghot sub-system for promotion decisions.
This benefits from reusing the existing MGLRU page table walk infrastructure, which is optimized with features such as hierarchical scanning and bloom filters to reduce CPU overhead.
As an additional optimization to be added in the future, we can tune the scan intervals for each memcg.
--- mm/Kconfig | 8 ++++ mm/Makefile | 1 + mm/klruscand.c | 110 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 119 insertions(+) create mode 100644 mm/klruscand.c
mm/Kconfig +8 -0
@@ -1403,6 +1403,14 @@ config HWMEM_PROFILER rolled up to PGHOT sub-system for further action like hot page promotion or NUMA Balancing +config KLRUSCAND + bool "Kernel lower tier access scan daemon" + default n + depends on PGHOT && LRU_GEN_WALKS_MMU + help + Scan for accesses from lower tiers by invoking MGLRU to perform + page table walks. + source "mm/damon/Kconfig" endmenu
mm/Makefile +1 -0
@@ -147,3 +147,4 @@ obj-$(CONFIG_EXECMEM) += execmem.o obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o obj-$(CONFIG_PGHOT) += pghot.o +obj-$(CONFIG_KLRUSCAND) += klruscand.o
mm/klruscand.c new file +110 -0
110 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC PATCH v4 8/9 mm: sched: Move hot page promotion from NUMAB=2 to pghot tracking
In reply to Bharata B RaoCurrently hot page promotion (NUMA_BALANCING_MEMORY_TIERING mode of NUMA Balancing) does hot page detection (via hint faults), hot page classification and eventual promotion, all by itself and sits within the scheduler.
With the new hot page tracking and promotion mechanism being available, NUMA Balancing can limit itself to detection of hot pages (via hint faults) and off-load rest of the functionality to the common hot page tracking system.
pghot_record_access(PGHOT_HINT_FAULT) API is used to feed the hot page info. In addition, the migration rate limiting and dynamic threshold logic are moved to kmigrated so that the same can be used for hot pages reported by other sources too.
--- include/linux/pghot.h | 3 + kernel/sched/debug.c | 1 - kernel/sched/fair.c | 152 ++---------------------------------------- mm/huge_memory.c | 26 ++------ mm/memory.c | 31 ++------- mm/pghot.c | 129 ++++++++++++++++++++++++++++++++++- 6 files changed, 147 insertions(+), 195 deletions(-)
include/linux/pghot.h +3 -0
@@ -71,6 +71,9 @@ enum pghot_src_enabed {#define PGHOT_SECTION_HOT_BIT BIT(0)#define PGHOT_SECTION_HOT_MASK GENMASK(PGHOT_SECTION_HOT_BIT - 1, 0) +#define KMIGRATED_MIGRATION_ADJUST_STEPS 16 +#define KMIGRATED_PROMOTION_THRESHOLD_WINDOW 60000 +int pghot_record_access (unsigned long pfn,int nid,int src,unsigned long now);#else static inline int pghot_record_access (unsigned long pfn,int nid,int src,unsigned long now)
kernel/sched/debug.c +0 -1
@@ -520,7 +520,6 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("scan_period_min_ms" ,0644 , numa, &sysctl_numa_balancing_scan_period_min);debugfs_create_u32 ("scan_period_max_ms" ,0644 , numa, &sysctl_numa_balancing_scan_period_max);debugfs_create_u32 ("scan_size_mb" ,0644 , numa, &sysctl_numa_balancing_scan_size); -debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);
kernel/sched/fair.c +5 -147
182 lines not on this page · raw message
mm/huge_memory.c +6 -20
@@ -39,6 +39,7 @@#include <linux/compat.h> #include <linux/pgalloc_tag.h> #include <linux/pagewalk.h> +#include <linux/pghot.h> #include <asm/tlb.h> #include <asm/pgalloc.h>
@@ -2051,29 +2052,12 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) target_nid =numa_migrate_check (folio, vmf, haddr, &flags, writable, &last_cpupid); + nid = target_nid;if (target_nid ==NUMA_NO_NODE ) goto out_map; -if (migrate_misplaced_folio_prepare (folio, vma, target_nid)) { - flags |=TNF_MIGRATE_FAIL ; - goto out_map; - } -/* The folio is isolated and isolation code holds a folio reference. */ -spin_unlock (vmf->ptl); - writable = false; -if (!migrate_misplaced_folio (folio, target_nid)) { - flags |=TNF_MIGRATED ; - nid = target_nid; -task_numa_fault (last_cpupid, nid,HPAGE_PMD_NR , flags); -return 0 ; - } + writable = false; - flags |=TNF_MIGRATE_FAIL ; - vmf->ptl =pmd_lock (vma->vm_mm, vmf->pmd); -if (unlikely (!pmd_same (pmdp_get (vmf->pmd), vmf->orig_pmd))) { -spin_unlock (vmf->ptl); -return 0 ; - } out_map:/* Restore the PMD */ pmd =pmd_modify (pmdp_get (vmf->pmd), vma->vm_page_prot);
@@ -2084,8 +2068,10 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)update_mmu_cache_pmd (vma, vmf->address, vmf->pmd);spin_unlock (vmf->ptl); -if (nid !=NUMA_NO_NODE ) +if (nid !=NUMA_NO_NODE ) { +pghot_record_access (folio_pfn (folio), nid,PGHOT_HINT_FAULT , jiffies);task_numa_fault (last_cpupid, nid,HPAGE_PMD_NR , flags); + }return 0 ; }
mm/memory.c +6 -25
@@ -75,6 +75,7 @@#include <linux/perf_event.h> #include <linux/ptrace.h> #include <linux/vmalloc.h> +#include <linux/pghot.h> #include <linux/sched/sysctl.h> #include <trace/events/kmem.h>
@@ -6007,34 +6008,12 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) target_nid =numa_migrate_check (folio, vmf, vmf->address, &flags, writable, &last_cpupid); + nid = target_nid;if (target_nid ==NUMA_NO_NODE ) goto out_map; -if (migrate_misplaced_folio_prepare (folio, vma, target_nid)) { - flags |=TNF_MIGRATE_FAIL ; - goto out_map; - } -/* The folio is isolated and isolation code holds a folio reference. */ -pte_unmap_unlock (vmf->pte, vmf->ptl); + writable = false; ignore_writable = true; - -/* Migrate to the requested node */ -if (!migrate_misplaced_folio (folio, target_nid)) { - nid = target_nid; - flags |=TNF_MIGRATED ; -task_numa_fault (last_cpupid, nid, nr_pages, flags); -return 0 ; - } - - flags |=TNF_MIGRATE_FAIL ; - vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); -if (unlikely (!vmf->pte)) -return 0 ; -if (unlikely (!pte_same (ptep_get (vmf->pte), vmf->orig_pte))) { -pte_unmap_unlock (vmf->pte, vmf->ptl); -return 0 ; - } out_map:/* * Make it present again, depending on how arch implements
@@ -6048,8 +6027,10 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) writable);pte_unmap_unlock (vmf->pte, vmf->ptl); -if (nid !=NUMA_NO_NODE ) +if (nid !=NUMA_NO_NODE ) { +pghot_record_access (folio_pfn (folio), nid,PGHOT_HINT_FAULT , jiffies);task_numa_fault (last_cpupid, nid, nr_pages, flags); + }return 0 ; }
mm/pghot.c +127 -2
165 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [RFC PATCH v4 8/9] mm: sched: Move hot page promotion from NUMAB=2 to pghot tracking
Quoted 16 lines · Bharata B Rao
Currently hot page promotion (
NUMA_BALANCING_MEMORY_TIERINGmode of NUMA Balancing) does hot page detection (via hint faults), hot page classification and eventual promotion, all by itself and sits within the scheduler.With the new hot page tracking and promotion mechanism being available, NUMA Balancing can limit itself to detection of hot pages (via hint faults) and off-load rest of the functionality to the common hot page tracking system.
pghot_record_access(PGHOT_HINT_FAULT)API is used to feed the hot page info. In addition, the migration rate limiting and dynamic threshold logic are moved to kmigrated so that the same can be used for hot pages reported by other sources too.Signed-off-by Bharata B Rao <[email protected]>
<snip>
Quoted 129 lines
mm/pghot.c +93
-2@@ -12,6 +12,9 @@* the hot pages. kmigrated runs for each lower tier node. It iterates * over the node's PFNs and migrates pages marked for migration into * their targeted nodes. +* +* Migration rate-limiting and dynamic threshold logic implementations +* were moved from NUMA Balancing mode 2. */ #include <linux/mm.h> #include <linux/migrate.h> @@ -25,6 +28,8 @@ static unsigned int pghot_freq_threshold = PGHOT_DEFAULT_FREQ_THRESHOLD;static unsigned int kmigrated_sleep_ms =KMIGRATED_DEFAULT_SLEEP_MS ;static unsigned int kmigrated_batch_nr =KMIGRATED_DEFAULT_BATCH_NR ; +/* Restrict the NUMA promotion throughput (MB/s) for each target node. */ +static unsigned int sysctl_pghot_promote_rate_limit =65536 ;static unsigned int sysctl_pghot_freq_window =PGHOT_DEFAULT_FREQ_WINDOW ;static DEFINE_STATIC_KEY_FALSE (pghot_src_hwhints);@@ -43,6 +48,14 @@ static const struct ctl_table pghot_sysctls[] = { .proc_handler = proc_dointvec_minmax, .extra1 = SYSCTL_ZERO, }, + { + .procname = "pghot_promote_rate_limit_MBps", + .data = &sysctl_pghot_promote_rate_limit, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + }, };#endif @@ -137,8 +150,13 @@ int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now) old_freq = (hotness >>PGHOT_FREQ_SHIFT ) &PGHOT_FREQ_MASK ; old_time = (hotness >>PGHOT_TIME_SHIFT ) &PGHOT_TIME_MASK ; -if (((time - old_time) >msecs_to_jiffies (sysctl_pghot_freq_window)) - || (nid !=NUMA_NO_NODE && old_nid != nid)) +/* +* Bypass the new window logic for NUMA hint fault source +* as it is too slow in reporting accesses. +* TODO: Fix this. +*/ +if ((((time - old_time) >msecs_to_jiffies (sysctl_pghot_freq_window)) + && (src !=PGHOT_HINT_FAULT )) || (nid !=NUMA_NO_NODE && old_nid != nid)) new_window = true;if (new_window)@@ -166,6 +184,110 @@ int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)return 0 ; } +/* +* For memory tiering mode, if there are enough free pages (more than +* enough watermark defined here) in fast memory node, to take full +* advantage of fast memory capacity, all recently accessed slow +* memory pages will be migrated to fast memory node without +* considering hot threshold. +*/ +static bool pgdat_free_space_enough (struct pglist_data *pgdat) +{ +int z; +unsigned long enough_wmark; + + enough_wmark =max (1UL *1024 *1024 *1024 >>PAGE_SHIFT , + pgdat->node_present_pages >>4 ); +for (z = pgdat->nr_zones -1 ; z >=0 ; z--) { +struct zone *zone = pgdat->node_zones + z; + +if (!populated_zone (zone)) +continue ; + +if (zone_watermark_ok (zone,0 , +promo_wmark_pages (zone) + enough_wmark, +ZONE_MOVABLE ,0 )) +return true; + } +return false; +} + +/* +* For memory tiering mode, too high promotion/demotion throughput may +* hurt application latency. So we provide a mechanism to rate limit +* the number of pages that are tried to be promoted. +*/ +static bool kmigrated_promotion_rate_limit (struct pglist_data *pgdat,unsigned long rate_limit, +int nr,unsigned long now_ms) +{ +unsigned long nr_cand; +unsigned int start; + +mod_node_page_state (pgdat,PGPROMOTE_CANDIDATE , nr); + nr_cand =node_page_state (pgdat,PGPROMOTE_CANDIDATE ); + start = pgdat->nbp_rl_start; +if (now_ms - start >MSEC_PER_SEC && +cmpxchg (&pgdat->nbp_rl_start, start, now_ms) == start) + pgdat->nbp_rl_nr_cand = nr_cand; +if (nr_cand - pgdat->nbp_rl_nr_cand >= rate_limit) +return true; +return false; +} + +static void kmigrated_promotion_adjust_threshold (struct pglist_data *pgdat, +unsigned long rate_limit,unsigned int ref_th, +unsigned long now_ms) +{ +unsigned int start, th_period, unit_th, th; +unsigned long nr_cand, ref_cand, diff_cand; + + th_period =KMIGRATED_PROMOTION_THRESHOLD_WINDOW ; + start = pgdat->nbp_th_start; +if (now_ms - start > th_period && +cmpxchg (&pgdat->nbp_th_start, start, now_ms) == start) { + ref_cand = rate_limit * +KMIGRATED_PROMOTION_THRESHOLD_WINDOW /MSEC_PER_SEC ; + nr_cand =node_page_state (pgdat,PGPROMOTE_CANDIDATE ); + diff_cand = nr_cand - pgdat->nbp_th_nr_cand; + unit_th = ref_th *2 /KMIGRATED_MIGRATION_ADJUST_STEPS ; + th = pgdat->nbp_threshold ? : ref_th; +if (diff_cand > ref_cand *11 /10 ) + th =max (th - unit_th, unit_th); +else if (diff_cand < ref_cand *9 /10 ) + th =min (th + unit_th, ref_th *2 ); + pgdat->nbp_th_nr_cand = nr_cand; + pgdat->nbp_threshold = th;
mm/pghot.c int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now) · line 269
Quoted 10 lines
mm/pghot.c +10
-0+ } +} + +static bool kmigrated_should_migrate_memory (unsigned long nr_pages,unsigned long nid, +unsigned long time) +{ +struct pglist_data *pgdat; +unsigned long rate_limit; +unsigned int th, def_th; +unsigned long now = jiffies;
now = jiffies & PGHOT_TIME_MASK;
Quoted 6 lines
mm/pghot.c +6
-0+unsigned long now_ms =jiffies_to_msecs (now); + + pgdat =NODE_DATA (nid); +if (pgdat_free_space_enough (pgdat)) { +/* workload changed, reset hot threshold */ + pgdat->nbp_threshold =0 ;
mm/pghot.c int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now) · line 285
Quoted 10 lines
mm/pghot.c +10
-0+mod_node_page_state (pgdat,PGPROMOTE_CANDIDATE_NRL , nr_pages); +return true; + } + + def_th = sysctl_pghot_freq_window; + rate_limit =MB_TO_PAGES (sysctl_pghot_promote_rate_limit); +kmigrated_promotion_adjust_threshold (pgdat, rate_limit, def_th, now_ms); + + th = pgdat->nbp_threshold ? : def_th; +if (jiffies_to_msecs (now - time) >= th)
Setting time in pfn hotness using PGHOT_TIME_MASK in pghot_record_access(). Therefore here also it should be calculated using PGHOT_TIME_MASK. Then it'll be right comparision.
Regards,
Alok Rathore
Re: [RFC PATCH v4 8/9] mm: sched: Move hot page promotion from NUMAB=2 to pghot tracking
On 22-Dec-25 3:56 PM, Alok Rathore wrote:
<snip>
Quoted 23 lines
Quoted 8 lines
+static bool kmigrated_should_migrate_memory(unsigned long
nr_pages, unsigned long nid,
+ unsigned long time)
+{
+ structpglist_data*pgdat;
+ unsigned longrate_limit;
+ unsigned int th,def_th;
+ unsigned long now = jiffies;now = jiffies &
PGHOT_TIME_MASK;Quoted 13 lines
+ unsigned long
now_ms=jiffies_to_msecs(now);mm/pghot.c +12
-0+ + pgdat = NODE_DATA(nid); + if (pgdat_free_space_enough(pgdat)) { + /* workload changed, reset hot threshold */ + pgdat->nbp_threshold = 0; + mod_node_page_state(pgdat, PGPROMOTE_CANDIDATE_NRL, nr_pages); + return true; + } + + def_th = sysctl_pghot_freq_window; + rate_limit = MB_TO_PAGES(sysctl_pghot_promote_rate_limit); + kmigrated_promotion_adjust_threshold(pgdat, rate_limit, def_th, now_ms);
Quoted 8 lines
Quoted 3 lines
+
+ th =pgdat->nbp_threshold? :def_th;
+ if (jiffies_to_msecs(now - time)>= th)Setting time in pfn hotness using
PGHOT_TIME_MASKinpghot_record_access(). Therefore
here also it should be calculated usingPGHOT_TIME_MASK. Then it'll be right comparision.
That's correct, will fix this in next iteration.
Regards,
Bharataa.
RFC PATCH v4 9/9 mm: pghot: Add folio_mark_accessed() as hotness source
In reply to Bharata B RaoUnmapped page cache pages that end up in lower tiers don't get promoted easily. There were attempts to identify such pages and get them promoted as part of NUMA Balancing earlier [1]. The same idea is taken forward here by using folio_mark_accessed() as a source of hotness.
Lower tier accesses from folio_mark_accessed() are reported to pghot sub-system for hotness tracking and subsequent promotion.
TODO: Need a better naming for this hotness source. Need to better understand/evaluate the overhead of hotness info collection from this path.
[1] https://lore.kernel.org/linux-mm/[email protected]/
--- Documentation/admin-guide/mm/pghot.txt | 3 ++- include/linux/pghot.h | 3 +++ include/linux/vm_event_item.h | 1 + mm/pghot-debug.c | 7 +++++++ mm/pghot.c | 6 ++++++ mm/swap.c | 8 ++++++++ mm/vmstat.c | 1 + 7 files changed, 28 insertions(+), 1 deletion(-)
Documentation/admin-guide/mm/pghot.txt +2 -1
@@ -23,9 +23,10 @@ Path: /sys/kernel/debug/pghot/ - 0: Hardware hints (value 0x1) - 1: Page table scan (value 0x2) - 2: Hint faults (value 0x4) + - 2: folio_mark_accessed (value 0x8) - Default: 0 (disabled) - Example: - # echo 0x7 > /sys/kernel/debug/pghot/enabled_sources + # echo 0xf > /sys/kernel/debug/pghot/enabled_sources Enables all sources. 2. **target_nid**
include/linux/pghot.h +3 -0
@@ -17,6 +17,7 @@ enum pghot_src {PGHOT_HW_HINTS ,PGHOT_PGTABLE_SCAN ,PGHOT_HINT_FAULT , + PGHOT_FMA, };#ifdef CONFIG_PGHOT
@@ -28,12 +29,14 @@ enum pghot_src_enabed {PGHOT_HWHINTS_BIT =0 ,PGHOT_PGTSCAN_BIT ,PGHOT_HINTFAULT_BIT , +PGHOT_FMA_BIT , PGHOT_MAX_BIT };#define PGHOT_HWHINTS_ENABLED BIT(PGHOT_HWHINTS_BIT)#define PGHOT_PGTSCAN_ENABLED BIT(PGHOT_PGTSCAN_BIT)#define PGHOT_HINTFAULT_ENABLED BIT(PGHOT_HINTFAULT_BIT) +#define PGHOT_FMA_ENABLED BIT(PGHOT_FMA_BIT)#define PGHOT_SRC_ENABLED_MASK GENMASK(PGHOT_MAX_BIT - 1, 0)#define PGHOT_DEFAULT_FREQ_WINDOW (5 * MSEC_PER_SEC)
include/linux/vm_event_item.h +1 -0
@@ -193,6 +193,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,PGHOT_RECORD_HWHINTS ,PGHOT_RECORD_PGTSCANS ,PGHOT_RECORD_HINTFAULTS , +PGHOT_RECORD_FMA , #ifdef CONFIG_HWMEM_PROFILERHWHINT_NR_EVENTS , HWHINT_KERNEL,
mm/pghot-debug.c +7 -0
@@ -117,6 +117,13 @@ static void pghot_src_enabled_update(unsigned int enabled)else static_branch_disable (&pghot_src_hintfaults); } + +if (changed &PGHOT_FMA_ENABLED ) { +if (enabled &PGHOT_FMA_ENABLED ) +static_branch_enable (&pghot_src_fma); +else +static_branch_disable (&pghot_src_fma); + } } static ssize_t pghot_src_enabled_write(struct file *filp, const char __user *ubuf,
mm/pghot.c +6 -0
@@ -35,6 +35,7 @@ static unsigned int sysctl_pghot_freq_window = PGHOT_DEFAULT_FREQ_WINDOW;static DEFINE_STATIC_KEY_FALSE (pghot_src_hwhints);static DEFINE_STATIC_KEY_FALSE (pghot_src_pghtscans);static DEFINE_STATIC_KEY_FALSE (pghot_src_hintfaults); +static DEFINE_STATIC_KEY_FALSE (pghot_src_fma);#include "pghot-debug.c"
@@ -110,6 +111,11 @@ int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)return -EINVAL ;count_vm_event (PGHOT_RECORD_HINTFAULTS );break ; +case PGHOT_FMA : +if (!static_branch_likely (&pghot_src_fma)) +return -EINVAL ; +count_vm_event (PGHOT_RECORD_FMA ); +break ;default :return -EINVAL ; }
mm/swap.c +8 -0
@@ -37,6 +37,8 @@#include <linux/page_idle.h> #include <linux/local_lock.h> #include <linux/buffer_head.h> +#include <linux/pghot.h> +#include <linux/memory-tiers.h> #include "internal.h"
@@ -454,8 +456,14 @@ static bool lru_gen_clear_refs(struct folio *folio)*/ void folio_mark_accessed (struct folio *folio) { +unsigned long pfn =folio_pfn (folio); +if (folio_test_dropbehind (folio))return ; + +if (!node_is_toptier (pfn_to_nid (pfn))) +pghot_record_access (pfn,NUMA_NO_NODE ,PGHOT_FMA , jiffies); +if (lru_gen_enabled ()) {lru_gen_inc_refs (folio);return ;
mm/vmstat.c +1 -0
@@ -1501,6 +1501,7 @@ const char * const vmstat_text[] = { [I(PGHOT_RECORD_HWHINTS)] = "pghot_recorded_hwhints", [I(PGHOT_RECORD_PGTSCANS)] = "pghot_recorded_pgtscans", [I(PGHOT_RECORD_HINTFAULTS)] = "pghot_recorded_hintfaults", + [I (PGHOT_RECORD_FMA )] ="pghot_recorded_fma" , #ifdef CONFIG_HWMEM_PROFILER [I(HWHINT_NR_EVENTS)] = "hwhint_nr_events", [I(HWHINT_KERNEL)] = "hwhint_kernel",
Signature · 1 line
2.34.1
Gregory Price
reviewerQuoted 4 lines · Bharata B Rao
Hi,
This is v4 of page hotness tracking and promotion sub-system pghot.
Just fyi, we haven't started real workload testing, but we're setting up the process soon. Just wanted to let you know it's on the radar.
We want to test this at the same time with some fixes we've been working on for multi-tenant fairness in the demotion/reclaim path.
~Gregory
Quoted 11 lines · Gregory Price
Quoted 4 lines · Bharata B Rao
Hi,
This is v4 of page hotness tracking and promotion sub-system pghot.
Just fyi, we haven't started real workload testing, but we're setting up the process soon. Just wanted to let you know it's on the radar.
We want to test this at the same time with some fixes we've been working on for multi-tenant fairness in the demotion/reclaim path.
Thanks Gregory for taking this up for testing.
I am working on v5 with following enhancements:
1. Default mode: 1 byte per PFN for storing hotness info. NID isn't stored but promotion will be done to a default toptier node(specifiable via sysfs/debugs). Frequency and time are stored but due to paucity of bits, I will have to use some sort of bucketing scheme for storing time like how current hot page promotion logic does.
2. Precision mode: 4 bytes or 32bits per PFN where NID is stored in addition to time and frequency.
2 is completed and 1 is in the works. Once I get it working correctly, I will post v5.
Since there were a few bugs in v4, I think it makes sense to start testing on v5 itself.
Regards,
Bharata.
Gregory Price
reviewerQuoted 23 lines · Bharata B Rao
Quoted 11 lines · Gregory Price
Quoted 4 lines · Bharata B Rao
Hi,
This is v4 of page hotness tracking and promotion sub-system pghot.
Just fyi, we haven't started real workload testing, but we're setting up the process soon. Just wanted to let you know it's on the radar.
We want to test this at the same time with some fixes we've been working on for multi-tenant fairness in the demotion/reclaim path.
Thanks Gregory for taking this up for testing.
I am working on v5 with following enhancements:
1. Default mode: 1 byte per PFN for storing hotness info. NID isn't stored but promotion will be done to a default toptier node(specifiable via sysfs/debugs). Frequency and time are stored but due to paucity of bits, I will have to use some sort of bucketing scheme for storing time like how current hot page promotion logic does.
Quoted 9 lines
2. Precision mode: 4 bytes or 32bits per PFN where NID is stored in addition to time and frequency.
2 is completed and 1 is in the works. Once I get it working correctly, I will post v5.
Since there were a few bugs in v4, I think it makes sense to start testing on v5 itself.
Can do, thanks for the heads up, we'll wait for v5.
Cheers,
Gregory