Skip to contentSkip to conversation

mm: Hot page tracking and promotion infrastructure

PATCH Superseded Bharata B Rao 6 participants 29 messages May 4 – May 13

Subsystems (16)

v7 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v7 cover mm: Hot page tracking and promotion infrastructure

Hi,

This is v7 of pghot, a hot-page tracking and promotion subsystem. The main change in this version is to add support for IBS Memory Profiler as page hotness source(PGHOT_HWHINTS). IBS Memory Profiler is a facility that will be present in future AMD processors. It provides memory access information and is independent of the existing IBS instance that is primarily used by the perf subsystem.

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) with the following goals:

- Unify hot page detection from multiple sources like hint faults, page table scans, hardware hints (AMD IBS).
- Decouple detection from migration.
- Centralize promotion logic via per-lower-tier-node kmigrated kernel thread.
- Move promotion rate‑limiting and related logic used by numa_balancing=2 (NUMAB2, the current NUMA balancing–based promotion) from the scheduler to pghot for broader reuse.

Currently, multiple kernel subsystems detect page accesses independently. This patchset consolidates accesses from these mechanisms by providing:

- A common API for reporting page accesses.
- Shared infrastructure for tracking hotness at PFN granularity.
- Per-lower-tier-node kernel threads for promoting pages.

Here is a brief summary of how this subsystem works:

- Tracks frequency and last access time.
- Additionally, the accessing NUMA node ID (NID) for each recorded access is also tracked in the precision mode.
- These hotness parameters are maintained in a per-PFN hotness record within the existing mem_section data structure.
- In default mode, one byte (u8) is used for hotness record. 5 bits are used to store time and bucketing scheme is used to represent a total access time up to 4s with HZ=1000. Default toptier NID (0) is used as the target for promotion which can be changed via debugfs tunable.
- In precision mode, 4 bytes (u32) are used for each hotness record. 14 bits are used to store time which can represent around 16s with HZ=1000.
- Classifies pages as hot based on configurable thresholds.
- Pages classified as hot are marked as ready for migration using the ready bit. Both modes use MSB of the hotness record as ready bit.
- Per-lower-tier-node kmigrated threads periodically scan the PFNs of lower-tier nodes, checking for the migration-ready bit to perform batched migrations. Interval between successive scans and batching value are configurable via debugfs tunables.

Memory overhead
---------------
Default mode: 1 byte per lower-tier PFN. For a 1TB lower-tier memory this amounts to 256MB overhead (assuming 4K pages)

Precision mode: 4 bytes per lower-tier PFN. For a 1TB of lower memory this amounts to 1G overhead.

Bit layout of hotness record
----------------------------
Default mode
- Bits 0-1: Frequency (2bits, 4 access samples)
- Bits 2-6: Bucketed time (5bits, up to 4s with HZ=1000)
- Bit 7: Migration ready bit

Precision mode
- Bits 0-9: Target NID (10bits)
- Bits 10-12: Frequency (3bits, 8 access samples)
- Bits 13-26: Time (14bits, up to 16s with HZ=1000)
- Bits 27-30: Reserved
- Bit 31: Migration ready bit

Potential hotness sources
-------------------------
1. NUMA Balancing (NUMAB2, Tiering mode) - included in this patchset.
2. AMD IBS Memory Profiler: HW based access profiler - included in this patchset.
3. klruscand - PTE‑A bit scanning built on MGLRU’s walk helpers - was showcased in previous versions but not part of this version.
4. folio_mark_accessed() - Page cache access tracking (unmapped page cache pages) - was showcased in previous versions but not part of this patchset.

Changes in v7
-------------
- Added AMD IBS Memory Profiler as page hotness source.
- Addressed review comments from v6 (Thanks to Shashiko AI, Gregory and Donet)
- Early exit from batched migration routine if input
list is empty
- Changed the name of batched migration routine to indicate that it handles "promotion" of batched "memcg" folios.
- Debug code in batched migration routine to check if all
the folios in the input list belong to the same memcg.
- Kconfig dependency cleanups.
- Fix one-off-regression in nid check in pghot-precise.
- More checks to validate nid in pghot-precise.
- Early check to not call kmigrated_run() for lower tier nodes.
- Handling PTE writable and ignore_writable conditions correctly in hint fault handler.
- Using unsigned int instead of unsigned long for representing time in ms.
- Misc cleanups.

Results
=======
Posted as replies to this mail thread.

This v7 patchset applies on top of upstream commit c1f49dea2b8f and can be fetched from:

https://github.com/AMDESE/linux-mm/tree/bharata/pghot-v7

v6: https://lore.kernel.org/linux-mm/[email protected]/
v5: https://lore.kernel.org/linux-mm/[email protected]/
v4: https://lore.kernel.org/linux-mm/[email protected]/
v3: https://lore.kernel.org/linux-mm/[email protected]/
v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/
v0: https://lore.kernel.org/linux-mm/[email protected]/

Bharata B Rao (6):
mm: migrate: Allow misplaced migration without VMA
mm: Hot page tracking and promotion - pghot
mm: pghot: Precision mode for pghot
mm: sched: move NUMA balancing tiering promotion to pghot x86/ibs: Move IBS caps definitions into its own header x86/mm/ibs: In-kernel driver for AMD IBS Memory Profiler

Gregory Price (1):
mm: migrate: Add promote_misplaced_memcg_folios()

 Documentation/admin-guide/mm/index.rst |   1 +
 Documentation/admin-guide/mm/pghot.rst |  80 ++++
 arch/x86/Kconfig                       |  16 +
 arch/x86/include/asm/ibs-caps.h        |  93 ++++
 arch/x86/include/asm/ibs-mprof.h       |  46 ++
 arch/x86/include/asm/msr-index.h       |   8 +
 arch/x86/include/asm/perf_event.h      |  81 +---
 arch/x86/mm/Makefile                   |   1 +
 arch/x86/mm/ibs-mprof.c                | 308 ++++++++++++
 include/linux/cpuhotplug.h             |   1 +
 include/linux/migrate.h                |   9 +-
 include/linux/mm.h                     |  35 +-
 include/linux/mmzone.h                 |  24 +-
 include/linux/pghot.h                  | 113 +++++
 include/linux/vm_event_item.h          |  11 +
 init/Kconfig                           |  13 +
 kernel/sched/core.c                    |   7 +
 kernel/sched/debug.c                   |   1 -
 kernel/sched/fair.c                    | 177 +------
 kernel/sched/sched.h                   |   1 -
 mm/Kconfig                             |  34 ++
 mm/Makefile                            |   6 +
 mm/huge_memory.c                       |  24 +-
 mm/memcontrol.c                        |   6 +-
 mm/memory-tiers.c                      |  15 +-
 mm/memory.c                            |  28 +-
 mm/mempolicy.c                         |   3 -
 mm/migrate.c                           |  98 +++-
 mm/mm_init.c                           |  10 +
 mm/pghot-default.c                     |  79 +++
 mm/pghot-precise.c                     |  81 ++++
 mm/pghot-tunables.c                    | 182 +++++++
 mm/pghot.c                             | 633 +++++++++++++++++++++++++
 mm/vmstat.c                            |  13 +-
 34 files changed, 1922 insertions(+), 316 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/pghot.rst
 create mode 100644 arch/x86/include/asm/ibs-caps.h
 create mode 100644 arch/x86/include/asm/ibs-mprof.h
 create mode 100644 arch/x86/mm/ibs-mprof.c
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot-precise.c
 create mode 100644 mm/pghot-tunables.c
 create mode 100644 mm/pghot.c

base-commit: c1f49dea2b8f335813d3b348fd39117fb8efb428

IBS Memory Profiler driver part of this patchset depends on the patchset that increases the number of APIC EILVT registers -
https://lore.kernel.org/lkml/[email protected]/

Signature · 1 line
2.34.1
#1 lore raw

PATCH v7 1/7 mm: migrate: Allow misplaced migration without VMA

We want isolation of misplaced folios to work in contexts where VMA isn't available, typically when performing migrations from a kernel thread context. In order to prepare for that, allow migrate_misplaced_folio_prepare() to be called with a NULL VMA.

When migrate_misplaced_folio_prepare() is called with non-NULL VMA, it will check if the folio is mapped shared and that requires holding PTL lock. This path isn't taken when the function is invoked with NULL VMA (migration outside of process context). Therefore, when VMA == NULL, migrate_misplaced_folio_prepare() does not require the caller to hold the PTL.

---
 mm/migrate.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
mm/migrate.c +7 -2
index 8a64291ab5b4..eb21a02fade0 100644
@@ -2671,7 +2671,12 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,
 
 /*
  * Prepare for calling migrate_misplaced_folio() by isolating the folio if
- * permitted. Must be called with the PTL still held.
+ * permitted. Must be called with the PTL still held if called with a non-NULL
+ * vma.
+ *
+ * When called with a NULL vma (e.g., kernel thread initiated migration),
+ * migrate_misplaced_folio_prepare() will allow shared executable folios
+ * to be migrated.
  */
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -2688,7 +2693,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 		 * See folio_maybe_mapped_shared() on possible imprecision
 		 * when we cannot easily detect if a folio is shared.
 		 */
-		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+		if (vma && (vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
 			return -EACCES;
 
 		/*
Signature · 1 line
2.34.1
#2 lore raw

PATCH v7 2/7 mm: migrate: Add promote_misplaced_memcg_folios()

In reply to Bharata B Rao

From: Gregory Price <[email protected]>

Tiered memory systems often require migrating multiple folios at once. Currently, migrate_misplaced_folio() handles only one folio per call, which is inefficient for batch operations. This patch introduces promote_misplaced_memcg_folios(), a batch variant that leverages migrate_pages() internally for improved performance.

The caller must isolate folios beforehand using migrate_misplaced_folio_prepare(). Additionally all the folios in the isolated list must belong to the same memcg. On return, the folio list will be empty regardless of success or failure.

This function will be used by pghot kmigrated thread.

[Rewrote commit description, memcg awareness]
---
 include/linux/migrate.h |  5 ++++
 mm/migrate.c            | 57 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 62 insertions(+)
include/linux/migrate.h +5 -0
index d5af2b7f577b..d136612eef9d 100644
@@ -111,6 +111,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
+int promote_misplaced_memcg_folios(struct list_head *folio_list, int node);
 #else
 static inline int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -121,6 +122,10 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
+static inline int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_MIGRATION
mm/migrate.c +57 -0

61 lines not on this page · raw message

Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH v7 2/7] mm: migrate: Add promote_misplaced_memcg_folios()

Hi Bharata

Quoted 6 lines · Bharata B Rao
mm/migrate.c +6 -0
+int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
+{
+	struct mem_cgroup *memcg = NULL;
+	unsigned int nr_succeeded = 0;
+	struct folio *first;
+	int nr_remaining;

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2808

Quoted 10 lines
mm/migrate.c +10 -0
+
+	if (list_empty(folio_list))
+		return 0;
+
+	first = list_first_entry(folio_list, struct folio, lru);
+#ifdef CONFIG_DEBUG_VM
+	{
+		struct folio *f;
+		list_for_each_entry(f, folio_list, lru)
+		VM_WARN_ON_ONCE(folio_memcg(f) != folio_memcg(first));

It looks like the indentation might be off here.

Quoted 22 lines
mm/migrate.c +21 -0
+	}
+#endif
+	memcg = get_mem_cgroup_from_folio(first);
+
+	nr_remaining = migrate_pages(folio_list, alloc_misplaced_dst_folio,
+				     NULL, node, MIGRATE_ASYNC,
+				     MR_NUMA_MISPLACED, &nr_succeeded);
+	if (nr_remaining)
+		putback_movable_pages(folio_list);
+
+	if (nr_succeeded) {
+		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
+		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(node)),
+				 PGPROMOTE_SUCCESS, nr_succeeded);
+	}
+
+	mem_cgroup_put(memcg);
+	WARN_ON(!list_empty(folio_list));
+	return nr_remaining ? -EAGAIN : 0;
+}
  #endif /* CONFIG_NUMA_BALANCING */
#4 lore raw

Re: [PATCH v7 2/7] mm: migrate: Add promote_misplaced_memcg_folios()

Quoted 14 lines · Donet Tom

Hi Bharata

Quoted 11 lines · Bharata B Rao
mm/migrate.c +11 -0
+int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
+{
+    struct mem_cgroup *memcg = NULL;
+    unsigned int nr_succeeded = 0;
+    struct folio *first;
+    int nr_remaining;
+
+    if (list_empty(folio_list))
+        return 0;
+
+    first = list_first_entry(folio_list, struct folio, lru);
Quoted 8 lines
Quoted 5 lines

+#ifdef CONFIG_DEBUG_VM
+    {
+        struct folio *f;
+        list_for_each_entry(f, folio_list, lru)
+        VM_WARN_ON_ONCE(folio_memcg(f) != folio_memcg(first));

It looks like the indentation might be off here.

Yeah looks like. Will fix.

Regards,
Bharata.

#5 lore raw

PATCH v7 3/7 mm: Hot page tracking and promotion - pghot

In reply to Bharata B Rao

pghot is a subsystem that collects memory access information from multiple sources, classifies hot pages resident in lower-tier memory, and promotes them to faster tiers. It stores per-PFN hotness metadata and performs asynchronous, batched promotion via a per-lower-tier-node kernel thread (kmigrated).

This change introduces the default (compact) mode of pghot:

- Per-PFN hotness record (phi_t = u8) embedded via mem_section:
- 2 bits: access frequency (4 levels)
- 5 bits: time bucket (≈4s window with HZ=1000, bucketed jiffies)
- 1 bit : migration-ready flag (MSB)
The LSB of mem_section->hot_map pointer is used as a per-section "hot" flag to gate scanning.

- Event recording API:
int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now) @pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffies
- Sources (e.g., NUMA hint faults, HW hints) call this to report accesses.
- In default mode, the nid is not stored/used for targeting; promotion goes to a configurable toptier node (pghot_target_nid).

- Promotion engine:
- One kmigrated thread per lower-tier node.
- Scans only sections whose "hot" flag was raised, iterates PFNs, and batches candidates by destination node.
- Uses migrate_misplaced_folios_batch() to move batched folios.

- Tunables & stats:
- debugfs: enabled_sources, target_nid, freq_threshold, kmigrated_sleep_ms, kmigrated_batch_nr
- sysctl : vm.pghot_promote_freq_window_ms
- vmstat : pghot_recorded_accesses, pghot_recorded_hintfaults, pghot_recorded_hwhints

Memory overhead
---------------
Default mode uses 1 byte of hotness metadata per PFN on lower-tier nodes.

Behavior & policy
-----------------
- Default mode promotion target:
The nid passed by sources is not stored; hot pages promote to pghot_target_nid (toptier). Precision mode (added later in the series) changes this.

- Record consumption:
kmigrated consumes (clears) the "migration-ready" bit before attempting isolation. Additionally the hotness record is reset. If isolation/migration fails, the folio is not re-queued automatically; subsequent accesses will re-arm it. This avoids retry storms and keeps batching stable.

- Wakeups:
kmigrated wakeups are intentionally timeout-driven. We set the per-pgdat "activate" flag on access, and kmigrated checks this flag on its next sleep interval. This keeps the first cut simple and avoids potential wake storms; active wakeups can be considered in a follow-up.

---
 Documentation/admin-guide/mm/index.rst |   1 +
 Documentation/admin-guide/mm/pghot.rst |  80 ++++
 include/linux/migrate.h                |   4 +-
 include/linux/mmzone.h                 |  20 +
 include/linux/pghot.h                  |  82 ++++
 include/linux/vm_event_item.h          |   5 +
 mm/Kconfig                             |  14 +
 mm/Makefile                            |   1 +
 mm/migrate.c                           |  16 +-
 mm/mm_init.c                           |  10 +
 mm/pghot-default.c                     |  79 ++++
 mm/pghot-tunables.c                    | 182 +++++++++
 mm/pghot.c                             | 494 +++++++++++++++++++++++++
 mm/vmstat.c                            |   5 +
 14 files changed, 986 insertions(+), 7 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/pghot.rst
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot-tunables.c
 create mode 100644 mm/pghot.c
Documentation/admin-guide/mm/index.rst +1 -0
index bbb563cba5d2..4d6810b02365 100644
@@ -43,3 +43,4 @@ the Linux memory management.
    userfaultfd
    zswap
    kho
+   pghot
Documentation/admin-guide/mm/pghot.rst new file +80 -0

80 lines not on this page · raw message

include/linux/migrate.h +2 -2
index d136612eef9d..53bae80d11ae 100644
@@ -107,7 +107,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
 
 #endif /* CONFIG_MIGRATION */
 
-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
@@ -126,7 +126,7 @@ static inline int promote_misplaced_memcg_folios(struct list_head *folio_list, i
 {
 	return -EAGAIN; /* can't migrate now */
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */
 
 #ifdef CONFIG_MIGRATION
 
include/linux/mmzone.h +20 -0
index 9adb2ad21da5..eb08431dc9fb 100644
@@ -1155,6 +1155,7 @@ enum pgdat_flags {
 					 * many pages under writeback
 					 */
 	PGDAT_RECLAIM_LOCKED,		/* prevents concurrent reclaim */
+	PGDAT_KMIGRATED_ACTIVATE,	/* activates kmigrated */
 };
 
 enum zone_flags {
@@ -1609,6 +1610,10 @@ typedef struct pglist_data {
 #ifdef CONFIG_MEMORY_FAILURE
 	struct memory_failure_stats mf_stats;
 #endif
+#ifdef CONFIG_PGHOT
+	struct task_struct *kmigrated;
+	wait_queue_head_t kmigrated_wait;
+#endif
 } pg_data_t;
 
 #define node_present_pages(nid)	(NODE_DATA(nid)->node_present_pages)
@@ -2019,12 +2024,27 @@ struct mem_section {
 	unsigned long section_mem_map;
 
 	struct mem_section_usage *usage;
+#ifdef CONFIG_PGHOT
+	/*
+	 * Per-PFN hotness data for this section.
+	 * Array of phi_t (u8 in default mode).
+	 * LSB is used as PGHOT_SECTION_HOT_BIT flag.
+	 */
+	void *hot_map;
+#endif
 #ifdef CONFIG_PAGE_EXTENSION
 	/*
 	 * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use
 	 * section. (see page_ext.h about this.)
 	 */
 	struct page_ext *page_ext;
+#endif
+	/*
+	 * Padding to maintain consistent mem_section size when exactly
+	 * one of PGHOT or PAGE_EXTENSION is enabled. This ensures
+	 * optimal alignment regardless of configuration.
+	 */
+#if (defined(CONFIG_PGHOT) ^ defined(CONFIG_PAGE_EXTENSION))
 	unsigned long pad;
 #endif
 	/*
include/linux/pghot.h new file +82 -0

82 lines not on this page · raw message

include/linux/vm_event_item.h +5 -0
index 03fe95f5a020..58d510711bd4 100644
@@ -175,6 +175,11 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		KSTACK_REST,
 #endif
 #endif /* CONFIG_DEBUG_STACK_USAGE */
+#ifdef CONFIG_PGHOT
+		PGHOT_RECORDED_ACCESSES,
+		PGHOT_RECORDED_HINTFAULTS,
+		PGHOT_RECORDED_HWHINTS,
+#endif /* CONFIG_PGHOT */
 		NR_VM_EVENT_ITEMS
 };
 
mm/Kconfig +14 -0
index 0a43bb80df4f..ebfa149d8123 100644
@@ -1469,6 +1469,20 @@ config LAZY_MMU_MODE_KUNIT_TEST
 
 	  If unsure, say N.
 
+config PGHOT
+	bool "Hot page tracking and promotion"
+	default n
+	depends on NUMA_MIGRATION && SPARSEMEM
+	help
+	  A sub-system to track page accesses in lower tier memory and
+	  maintain hot page information. Promotes hot pages from lower
+	  tiers to top tier by using the memory access information provided
+	  by various sources. Asynchronous promotion is done by per-node
+	  kernel threads.
+
+	  This adds 1 byte of metadata overhead per page in lower-tier
+	  memory nodes.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +1 -0
index 8ad2ab08244e..33014de43acc 100644
@@ -150,3 +150,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
+obj-$(CONFIG_PGHOT) += pghot.o pghot-tunables.o pghot-default.o
mm/migrate.c +11 -5
index 747277aadf19..726d27b61a46 100644
@@ -2625,7 +2625,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long, nr_pages,
 }
 #endif /* CONFIG_NUMA_MIGRATION */
 
-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
 /*
  * Returns true if this is a safe migration target node for misplaced NUMA
  * pages. Currently it only checks the watermarks which is crude.
@@ -2745,12 +2745,10 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
  */
 int migrate_misplaced_folio(struct folio *folio, int node)
 {
-	pg_data_t *pgdat = NODE_DATA(node);
 	int nr_remaining;
 	unsigned int nr_succeeded;
 	LIST_HEAD(migratepages);
 	struct mem_cgroup *memcg = get_mem_cgroup_from_folio(folio);
-	struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
 
 	list_add(&folio->lru, &migratepages);
 	nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
@@ -2759,12 +2757,18 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	if (nr_remaining && !list_empty(&migratepages))
 		putback_movable_pages(&migratepages);
 	if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
 		if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
 		    && !node_is_toptier(folio_nid(folio))
-		    && node_is_toptier(node))
+		    && node_is_toptier(node)) {
+			pg_data_t *pgdat = NODE_DATA(node);
+			struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
+
 			mod_lruvec_state(lruvec, PGPROMOTE_SUCCESS, nr_succeeded);
+		}
+#endif
 	}
 	mem_cgroup_put(memcg);
 	BUG_ON(!list_empty(&migratepages));
@@ -2817,14 +2821,16 @@ int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
 		putback_movable_pages(folio_list);
 
 	if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
 		mod_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(node)),
 				 PGPROMOTE_SUCCESS, nr_succeeded);
+#endif
 	}
 
 	mem_cgroup_put(memcg);
 	WARN_ON(!list_empty(folio_list));
 	return nr_remaining ? -EAGAIN : 0;
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */
mm/mm_init.c +10 -0
index f9f8e1af921c..2396c42028ae 100644
@@ -1384,6 +1384,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)
 static void pgdat_init_kcompactd(struct pglist_data *pgdat) {}
 #endif
 
+#ifdef CONFIG_PGHOT
+static void pgdat_init_kmigrated(struct pglist_data *pgdat)
+{
+	init_waitqueue_head(&pgdat->kmigrated_wait);
+}
+#else
+static inline void pgdat_init_kmigrated(struct pglist_data *pgdat) {}
+#endif
+
 static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 {
 	int i;
@@ -1393,6 +1402,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 
 	pgdat_init_split_queue(pgdat);
 	pgdat_init_kcompactd(pgdat);
+	pgdat_init_kmigrated(pgdat);
 
 	init_waitqueue_head(&pgdat->kswapd_wait);
 	init_waitqueue_head(&pgdat->pfmemalloc_wait);
mm/pghot-default.c new file +79 -0

79 lines not on this page · raw message

mm/pghot-tunables.c new file +182 -0

182 lines not on this page · raw message

mm/pghot.c new file +494 -0

494 lines not on this page · raw message

mm/vmstat.c +5 -0
index f534972f517d..4064ead568cc 100644
@@ -1489,6 +1489,11 @@ const char * const vmstat_text[] = {
 	[I(KSTACK_REST)]			= "kstack_rest",
 #endif
 #endif
+#ifdef CONFIG_PGHOT
+	[I(PGHOT_RECORDED_ACCESSES)]		= "pghot_recorded_accesses",
+	[I(PGHOT_RECORDED_HINTFAULTS)]		= "pghot_recorded_hintfaults",
+	[I(PGHOT_RECORDED_HWHINTS)]		= "pghot_recorded_hwhints",
+#endif /* CONFIG_PGHOT */
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
 };
Signature · 1 line
2.34.1
#6 lore raw

PATCH v7 4/7 mm: pghot: Precision mode for pghot

In reply to Bharata B Rao

Default pghot stores hotness in a 1‑byte record per PFN, limiting frequency to 2 bits, time to a 5‑bit bucket, and preventing storage of per‑PFN toptier NID. This restricts time granularity and forces all promotions to use the global pghot_target_nid.

This patch adds an optional precision mode (CONFIG_PGHOT_PRECISE) that expands the hotness record to 4 bytes (u32) and provides:

- 10‑bit NID field for per‑PFN promotion target,
- 3‑bit frequency field (freq_threshold range 1–7),
- 14‑bit time field offering finer recency tracking,
- MSB migrate‑ready bit.

Precision mode improves placement accuracy on systems with multiple toptier nodes and provides higher‑resolution hotness tracking, at the cost of increasing metadata to 4 bytes per PFN.

Documentation, tunables, and the record layout are updated accordingly.

---
 Documentation/admin-guide/mm/pghot.rst |  4 +-
 include/linux/mmzone.h                 |  2 +-
 include/linux/pghot.h                  | 31 ++++++++++
 mm/Kconfig                             | 11 ++++
 mm/Makefile                            |  7 ++-
 mm/pghot-precise.c                     | 81 ++++++++++++++++++++++++++
 mm/pghot.c                             | 13 +++--
 7 files changed, 141 insertions(+), 8 deletions(-)
 create mode 100644 mm/pghot-precise.c
Documentation/admin-guide/mm/pghot.rst +2 -2
index 5f51dd1d4d45..7b84e911afe7 100644
@@ -37,7 +37,7 @@ Path: /sys/kernel/debug/pghot/
 
 3. **freq_threshold**
    - Minimum access frequency before a page is marked ready for promotion.
-   - Range: 1 to 3
+   - Range: 1 to 3 in default mode, 1 to 7 in precision mode.
    - Default: 2
    - Example:
      # echo 3 > /sys/kernel/debug/pghot/freq_threshold
@@ -59,7 +59,7 @@ Path: /proc/sys/vm/pghot_promote_freq_window_ms
 - Controls the time window (in ms) for counting access frequency. A page is
   considered hot only when **freq_threshold** number of accesses occur with
   this time period.
-- Default: 3000 (3 seconds)
+- Default: 3000 (3 seconds) in default mode and 5000 (5s) in precision mode.
 - Example:
   # sysctl vm.pghot_promote_freq_window_ms=3000
 
include/linux/mmzone.h +1 -1
index eb08431dc9fb..9577bdc575d9 100644
@@ -2027,7 +2027,7 @@ struct mem_section {
 #ifdef CONFIG_PGHOT
 	/*
 	 * Per-PFN hotness data for this section.
-	 * Array of phi_t (u8 in default mode).
+	 * Array of phi_t (u8 in default mode, u32 in precision mode).
 	 * LSB is used as PGHOT_SECTION_HOT_BIT flag.
 	 */
 	void *hot_map;
include/linux/pghot.h +31 -0
index 525d4dd28fc1..2e1742b8caee 100644
@@ -35,6 +35,36 @@ DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
 
 #define PGHOT_DEFAULT_NODE		0
 
+#if defined(CONFIG_PGHOT_PRECISE)
+#define PGHOT_DEFAULT_FREQ_WINDOW	(5 * MSEC_PER_SEC)
+
+/*
+ * Bits 0-26 are used to store nid, frequency and time.
+ * Bits 27-30 are unused now.
+ * Bit 31 is used to indicate the page is ready for migration.
+ */
+#define PGHOT_MIGRATE_READY		31
+
+#define PGHOT_NID_WIDTH			10
+#define PGHOT_FREQ_WIDTH		3
+/* time is stored in 14 bits which can represent up to 16s with HZ=1000 */
+#define PGHOT_TIME_WIDTH		14
+
+#define PGHOT_NID_SHIFT			0
+#define PGHOT_FREQ_SHIFT		(PGHOT_NID_SHIFT + PGHOT_NID_WIDTH)
+#define PGHOT_TIME_SHIFT		(PGHOT_FREQ_SHIFT + PGHOT_FREQ_WIDTH)
+
+#define PGHOT_NID_MASK			GENMASK(PGHOT_NID_WIDTH - 1, 0)
+#define PGHOT_FREQ_MASK			GENMASK(PGHOT_FREQ_WIDTH - 1, 0)
+#define PGHOT_TIME_MASK			GENMASK(PGHOT_TIME_WIDTH - 1, 0)
+
+#define PGHOT_NID_MAX			((1 << PGHOT_NID_WIDTH) - 1)
+#define PGHOT_FREQ_MAX			((1 << PGHOT_FREQ_WIDTH) - 1)
+#define PGHOT_TIME_MAX			((1 << PGHOT_TIME_WIDTH) - 1)
+
+typedef u32 phi_t;
+
+#else	/* !CONFIG_PGHOT_PRECISE */
 #define PGHOT_DEFAULT_FREQ_WINDOW	(3 * MSEC_PER_SEC)
 
 /*
@@ -61,6 +91,7 @@ DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
 #define PGHOT_TIME_MAX			((1 << PGHOT_TIME_WIDTH) - 1)
 
 typedef u8 phi_t;
+#endif /* CONFIG_PGHOT_PRECISE */
 
 #define PGHOT_RECORD_SIZE		sizeof(phi_t)
 
mm/Kconfig +11 -0
index ebfa149d8123..cc4b5685ecd4 100644
@@ -1483,6 +1483,17 @@ config PGHOT
 	  This adds 1 byte of metadata overhead per page in lower-tier
 	  memory nodes.
 
+config PGHOT_PRECISE
+	bool "Hot page tracking precision mode"
+	default n
+	depends on PGHOT
+	help
+	  Enables precision mode for tracking hot pages with pghot sub-system.
+	  Adds fine-grained access time tracking and explicit toptier target
+	  NID tracking. Precise hot page tracking comes at the cost of using
+	  4 bytes per page against the default one byte per page. Preferable
+	  to enable this on systems with multiple nodes in toptier.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +6 -1
index 33014de43acc..dc61f4d955f8 100644
@@ -150,4 +150,9 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
-obj-$(CONFIG_PGHOT) += pghot.o pghot-tunables.o pghot-default.o
+obj-$(CONFIG_PGHOT) += pghot.o pghot-tunables.o
+ifdef CONFIG_PGHOT_PRECISE
+obj-$(CONFIG_PGHOT) += pghot-precise.o
+else
+obj-$(CONFIG_PGHOT) += pghot-default.o
+endif
mm/pghot-precise.c new file +81 -0

81 lines not on this page · raw message

mm/pghot.c +9 -4
index 02e6959b647a..0b31d5917833 100644
@@ -10,6 +10,9 @@
  * the frequency of access and last access time. Promotions are done
  * to a default toptier NID.
  *
+ * In the precision mode, 4 bytes are used to store the frequency
+ * of access, last access time and the accessing NID.
+ *
  * A kernel thread named kmigrated is provided to migrate or promote
  * the hot pages. kmigrated runs for each lower tier node. It iterates
  * over the node's PFNs and  migrates pages marked for migration into
@@ -52,13 +55,15 @@ static bool kmigrated_started __ro_after_init;
  * for the purpose of tracking page hotness and subsequent promotion.
  *
  * @pfn: PFN of the page
- * @nid: Unused
+ * @nid: Target NID to where the page needs to be migrated in precision
+ *       mode but unused in default mode
  * @src: The identifier of the sub-system that reports the access
  * @now: Access time in jiffies
  *
- * Updates the frequency and time of access and marks the page as
- * ready for migration if the frequency crosses a threshold. The pages
- * marked for migration are migrated by kmigrated kernel thread.
+ * Updates the NID (in precision mode only), frequency and time of access
+ * and marks the page as ready for migration if the frequency crosses a
+ * threshold. The pages marked for migration are migrated by kmigrated
+ * kernel thread.
  *
  * Return: 0 on success and -EINVAL on failure to record the access.
  */
Signature · 1 line
2.34.1
#7 lore raw

Re: [PATCH v7 4/7] mm: pghot: Precision mode for pghot

Hi Bharata

mm/pghot-precise.c line 13

Quoted 5 lines · Bharata B Rao
mm/pghot-precise.c new file +5 -0
+#include <linux/pghot.h>
+#include <linux/jiffies.h>
+#include <linux/memory-tiers.h>
+
+bool pghot_nid_valid(int nid)

I might be missing something, but since pghot_nid_valid() exists in both pghot-default.c and pghot-precise.c, would it make sense to move it to a header file as a static inline function?

-Donet

Quoted 16 lines
mm/pghot-precise.c new file +16 -0
+{
+	if (nid != NUMA_NO_NODE &&
+	    (!numa_valid_node(nid) || nid > PGHOT_NID_MAX ||
+	     !node_online(nid) || !node_is_toptier(nid)))
+		return false;
+
+	return true;
+}
+
+unsigned long pghot_access_latency(unsigned long old_time, unsigned long time)
+{
+	return jiffies_to_msecs((time - old_time) & PGHOT_TIME_MASK);
+}
+
+bool pghot_update_record(phi_t *phi, int nid, unsigned long now)
+{
#8 lore raw

Re: [PATCH v7 4/7] mm: pghot: Precision mode for pghot

Quoted 12 lines · Donet Tom

Hi Bharata

mm/pghot-precise.c line 13

Quoted 5 lines · Bharata B Rao
mm/pghot-precise.c new file +5 -0
+#include <linux/pghot.h>
+#include <linux/jiffies.h>
+#include <linux/memory-tiers.h>
+
+bool pghot_nid_valid(int nid)

I might be missing something, but since pghot_nid_valid() exists in both pghot- default.c and pghot-precise.c, would it make sense to move it to a header file as a static inline function?

It exists in both modes of pghot but the implementations differ. Hence it can't reside as static inline function in pghot.h.

Regards,
Bharata.

#9 lore raw

PATCH v7 5/7 mm: sched: move NUMA balancing tiering promotion to pghot

In reply to Bharata B Rao

Currently hot page promotion (NUMA_BALANCING_MEMORY_TIERING mode of NUMA Balancing) does hot page detection (via hint faults), hot page classification and eventual promotion, all by itself and sits within the scheduler.

With pghot, the new hot page tracking and promotion mechanism being available, NUMA Balancing can limit itself to detection of hot pages (via hint faults) and off-load rest of the functionality to pghot.

To achieve this, pghot_record_access(PGHOT_HINTFAULTS) API is used to feed the hot page info to pghot. In addition, the migration rate limiting and dynamic threshold logic are moved to kmigrated so that the same can be used for hot pages reported by other sources too. Hence it becomes necessary to introduce a new config option CONFIG_NUMA_BALANCING_TIERING to control the hint faults souce for hot page promotion. This option controls the NUMA_BALANCING_MEMORY_TIERING mode of kernel.numa_balancing

This movement of hot page promotion to pghot results in the following changes to the behaviour of hint faults based hot page promotion:

1. Promotion is no longer done in the fault path but instead is deferred to kmigrated and happens in batches.
2. NUMA_BALANCING_MEMORY_TIERING mode used to promote on first access. Pghot by default, promotes on second access though this can be changed by setting /sys/kernel/debug/pghot/freq_threshold. hot_threshold_ms debugfs tunable now gets replaced by pghot's freq_threshold.
3. In NUMA_BALANCING_MEMORY_TIERING mode, hint fault latency is the difference between the PTE update time (during scanning) and the access time (hint fault). However with pghot, a single latency threshold is used for two purposes:
a) If the time difference between successive accesses are within the threshold, the page is marked as hot.
b) Later when kmigrated picks up the page for migration, it will migrate only if the difference between the current time and the time when the page was marked hot is with the threshold.
4. Batch migration of misplaced folios is done from non-process context where VMA info is not readily available. Without VMA and the exec check on that, it will not be possible to filter out exec pages during migration prep stage. Hence shared executable pages also will be subjected to misplaced migration.
5. The max scan period which is used in dynamic threshold logic was a debugfs tunable. However this has been converted to a scalar metric in pghot.
6. In the uncommon case of using NUMA_BALANCING_NORMAL mode to balance between lower and higher tier nodes, we end up waking the kswapd when there is no headroom in the toptier.

Key code changes due to this movement are detailed below to help easy understanding of the restructuring.

1. Scanning and access times are no longer tracked in last_cpupid field of folio flags. Hence all code related to this (like folio_xchg_access_time(), cpupid_valid()) are removed.
2. The misplaced migration routines become conditional to CONFIG_PGHOT in addition to CONFIG_NUMA_BALANCING.
3. The promotion related stats (like PGPROMOTE_SUCCESS etc) are now moved to under CONFIG_PGHOT as these stats are part of promotion engine which will be used for other hotness sources as well.
4. Routines that are responsibile for migration rate limiting dynamic thresholding, pgdat balancing during promotion etc are moved to pghot with appropriate renaming.

---
 include/linux/mm.h     |  35 ++------
 include/linux/mmzone.h |   4 +-
 init/Kconfig           |  13 +++
 kernel/sched/core.c    |   7 ++
 kernel/sched/debug.c   |   1 -
 kernel/sched/fair.c    | 177 ++---------------------------------------
 kernel/sched/sched.h   |   1 -
 mm/huge_memory.c       |  24 +++++-
 mm/memcontrol.c        |   6 +-
 mm/memory-tiers.c      |  15 ++--
 mm/memory.c            |  28 +++++--
 mm/mempolicy.c         |   3 -
 mm/migrate.c           |  16 +++-
 mm/pghot.c             | 134 +++++++++++++++++++++++++++++++
 mm/vmstat.c            |   2 +-
 15 files changed, 239 insertions(+), 227 deletions(-)
include/linux/mm.h +7 -28
index 0b776907152e..3b237946b322 100644
@@ -2271,17 +2271,6 @@ static inline int folio_nid(const struct folio *folio)
 }
 
 #ifdef CONFIG_NUMA_BALANCING
-/* page access time bits needs to hold at least 4 seconds */
-#define PAGE_ACCESS_TIME_MIN_BITS	12
-#if LAST_CPUPID_SHIFT < PAGE_ACCESS_TIME_MIN_BITS
-#define PAGE_ACCESS_TIME_BUCKETS				\
-	(PAGE_ACCESS_TIME_MIN_BITS - LAST_CPUPID_SHIFT)
-#else
-#define PAGE_ACCESS_TIME_BUCKETS	0
-#endif
-
-#define PAGE_ACCESS_TIME_MASK				\
-	(LAST_CPUPID_MASK << PAGE_ACCESS_TIME_BUCKETS)
 
 static inline int cpu_pid_to_cpupid(int cpu, int pid)
 {
@@ -2347,15 +2336,6 @@ static inline void page_cpupid_reset_last(struct page *page)
 }
 #endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */
 
-static inline int folio_xchg_access_time(struct folio *folio, int time)
-{
-	int last_time;
-
-	last_time = folio_xchg_last_cpupid(folio,
-					   time >> PAGE_ACCESS_TIME_BUCKETS);
-	return last_time << PAGE_ACCESS_TIME_BUCKETS;
-}
-
 static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 {
 	unsigned int pid_bit;
@@ -2366,18 +2346,12 @@ static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 	}
 }
 
-bool folio_use_access_time(struct folio *folio);
 #else /* !CONFIG_NUMA_BALANCING */
 static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid)
 {
 	return folio_nid(folio); /* XXX */
 }
 
-static inline int folio_xchg_access_time(struct folio *folio, int time)
-{
-	return 0;
-}
-
 static inline int folio_last_cpupid(struct folio *folio)
 {
 	return folio_nid(folio); /* XXX */
@@ -2420,11 +2394,16 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid)
 static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 {
 }
-static inline bool folio_use_access_time(struct folio *folio)
+#endif /* CONFIG_NUMA_BALANCING */
+
+#ifdef CONFIG_NUMA_BALANCING_TIERING
+bool folio_is_promo_candidate(struct folio *folio);
+#else
+static inline bool folio_is_promo_candidate(struct folio *folio)
 {
 	return false;
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING_TIERING */
 
 #if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
 
include/linux/mmzone.h +2 -2
index 9577bdc575d9..b29d06168826 100644
@@ -287,7 +287,7 @@ enum node_stat_item {
 #ifdef CONFIG_SWAP
 	NR_SWAPCACHE,
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	PGPROMOTE_SUCCESS,	/* promote successfully */
 	/**
 	 * Candidate pages for promotion based on hint fault latency.  This
@@ -1566,7 +1566,7 @@ typedef struct pglist_data {
 	struct deferred_split deferred_split_queue;
 #endif
 
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	/* start time in ms of current promote rate limit period */
 	unsigned int nbp_rl_start;
 	/* number of promote candidate pages at start time of current rate limit period */
init/Kconfig +13 -0
index 2937c4d308ae..7624be1c739a 100644
@@ -1027,6 +1027,19 @@ config NUMA_BALANCING_DEFAULT_ENABLED
 	  If set, automatic NUMA balancing will be enabled if running on a NUMA
 	  machine.
 
+config NUMA_BALANCING_TIERING
+	bool "NUMA balancing memory tiering promotion"
+	depends on NUMA_BALANCING && PGHOT
+	help
+	  Enable NUMA balancing mode 2 (memory tiering). This allows
+	  automatic promotion of hot pages from slower memory tiers to
+	  faster tiers using the pghot subsystem.
+
+	  This requires CONFIG_PGHOT for the hot page tracking engine.
+	  This option is required for kernel.numa_balancing=2.
+
+	  If unsure, say N.
+
 config SLAB_OBJ_EXT
 	bool
 
kernel/sched/core.c +7 -0
index da20fb6ea25a..46ce75f00b40 100644
@@ -4523,6 +4523,7 @@ void set_numabalancing_state(bool enabled)
 }
 
 #ifdef CONFIG_PROC_SYSCTL
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 static void reset_memory_tiering(void)
 {
 	struct pglist_data *pgdat;
@@ -4533,6 +4534,7 @@ static void reset_memory_tiering(void)
 		pgdat->nbp_th_start = jiffies_to_msecs(jiffies);
 	}
 }
+#endif
 
 static int sysctl_numa_balancing(const struct ctl_table *table, int write,
 			  void *buffer, size_t *lenp, loff_t *ppos)
@@ -4550,9 +4552,14 @@ static int sysctl_numa_balancing(const struct ctl_table *table, int write,
 	if (err < 0)
 		return err;
 	if (write) {
+		if ((state & NUMA_BALANCING_MEMORY_TIERING) &&
+		    !IS_ENABLED(CONFIG_NUMA_BALANCING_TIERING))
+			return -EOPNOTSUPP;
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 		    (state & NUMA_BALANCING_MEMORY_TIERING))
 			reset_memory_tiering();
+#endif
 		sysctl_numa_balancing_mode = state;
 		__set_numabalancing_state(state);
 	}
kernel/sched/debug.c +0 -1
index 74c1617cf652..abf53f3071ea 100644
@@ -623,7 +623,6 @@ static __init int sched_init_debug(void)
 	debugfs_create_u32("scan_period_min_ms", 0644, numa, &sysctl_numa_balancing_scan_period_min);
 	debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max);
 	debugfs_create_u32("scan_size_mb", 0644, numa, &sysctl_numa_balancing_scan_size);
-	debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold);
 #endif /* CONFIG_NUMA_BALANCING */
 
 	debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);
kernel/sched/fair.c +5 -172

217 lines not on this page · raw message

kernel/sched/sched.h +0 -1
index 9f63b15d309d..f176643516b5 100644
@@ -3066,7 +3066,6 @@ extern unsigned int sysctl_numa_balancing_scan_delay;
 extern unsigned int sysctl_numa_balancing_scan_period_min;
 extern unsigned int sysctl_numa_balancing_scan_period_max;
 extern unsigned int sysctl_numa_balancing_scan_size;
-extern unsigned int sysctl_numa_balancing_hot_threshold;
 
 #ifdef CONFIG_SCHED_HRTICK
 
mm/huge_memory.c +21 -3
index 970e077019b7..1890b1e534a4 100644
@@ -40,6 +40,7 @@
 #include <linux/pgalloc.h>
 #include <linux/pgalloc_tag.h>
 #include <linux/pagewalk.h>
+#include <linux/pghot.h>
 
 #include <asm/tlb.h>
 #include "internal.h"
@@ -2267,7 +2268,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	int nid = NUMA_NO_NODE;
 	int target_nid, last_cpupid;
 	pmd_t pmd, old_pmd;
-	bool writable = false;
+	bool writable = false, needs_promotion = false;
 	int flags = 0;
 
 	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
@@ -2294,11 +2295,23 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 
 	target_nid = numa_migrate_check(folio, vmf, haddr, &flags, writable,
 					&last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
+		 * Isolation and migration are handled by pghot.
+		 */
+		nid = target_nid;
+		goto out_map;
+	}
+
+	/* Balancing b/n toptier nodes, mode=NUMA_BALANCING_NORMAL */
 	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
 		flags |= TNF_MIGRATE_FAIL;
 		goto out_map;
@@ -2330,8 +2343,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
 	spin_unlock(vmf->ptl);
 
-	if (nid != NUMA_NO_NODE)
-		task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags);
+	if (nid != NUMA_NO_NODE) {
+		if (needs_promotion)
+			pghot_record_access(folio_pfn(folio), nid,
+					    PGHOT_HINTFAULTS, jiffies);
+		else
+			task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags);
+	}
 	return 0;
 }
 
mm/memcontrol.c +3 -3
index c3d98ab41f1f..033b80ad248e 100644
@@ -400,7 +400,7 @@ static const unsigned int memcg_node_stat_items[] = {
 #ifdef CONFIG_SWAP
 	NR_SWAPCACHE,
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	PGPROMOTE_SUCCESS,
 #endif
 	PGDEMOTE_KSWAPD,
@@ -1594,7 +1594,7 @@ static const struct memory_stat memory_stats[] = {
 	{ "pgscan_khugepaged",		PGSCAN_KHUGEPAGED	},
 	{ "pgscan_proactive",		PGSCAN_PROACTIVE	},
 	{ "pgrefill",			PGREFILL		},
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	{ "pgpromote_success",		PGPROMOTE_SUCCESS	},
 #endif
 };
@@ -1646,7 +1646,7 @@ static int memcg_page_state_output_unit(int item)
 	case PGSCAN_KHUGEPAGED:
 	case PGSCAN_PROACTIVE:
 	case PGREFILL:
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	case PGPROMOTE_SUCCESS:
 #endif
 		return 1;
mm/memory-tiers.c +8 -7
index 54851d8a195b..be134a32f5bf 100644
@@ -51,18 +51,19 @@ static const struct bus_type memory_tier_subsys = {
 	.dev_name = "memory_tier",
 };
 
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 /**
- * folio_use_access_time - check if a folio reuses cpupid for page access time
+ * folio_is_promo_candidate - check if the folio qualifies for promotion
+ *
  * @folio: folio to check
  *
- * folio's _last_cpupid field is repurposed by memory tiering. In memory
- * tiering mode, cpupid of slow memory folio (not toptier memory) is used to
- * record page access time.
+ * Checks if NUMA Balancing tiering mode is set and the folio belongs
+ * to lower tier. If so, it qualifies for promotion to toptier when
+ * it is categorized as hot.
  *
- * Return: the folio _last_cpupid is used to record page access time
+ * Return: True if the above condition is met, else False.
  */
-bool folio_use_access_time(struct folio *folio)
+bool folio_is_promo_candidate(struct folio *folio)
 {
 	return (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 	       !node_is_toptier(folio_nid(folio));
mm/memory.c +23 -5
index ea6568571131..17ea31750573 100644
@@ -75,6 +75,7 @@
 #include <linux/perf_event.h>
 #include <linux/ptrace.h>
 #include <linux/vmalloc.h>
+#include <linux/pghot.h>
 #include <linux/sched/sysctl.h>
 #include <linux/pgalloc.h>
 #include <linux/uaccess.h>
@@ -6062,10 +6063,9 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,
 	if (folio_maybe_mapped_shared(folio) && (vma->vm_flags & VM_SHARED))
 		*flags |= TNF_SHARED;
 	/*
-	 * For memory tiering mode, cpupid of slow memory page is used
-	 * to record page access time.  So use default value.
+	 * For memory tiering mode, last_cpupid is unused. So use default value.
 	 */
-	if (folio_use_access_time(folio))
+	if (folio_is_promo_candidate(folio))
 		*last_cpupid = (-1 & LAST_CPUPID_MASK);
 	else
 		*last_cpupid = folio_last_cpupid(folio);
@@ -6146,6 +6146,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	int nid = NUMA_NO_NODE;
 	bool writable = false, ignore_writable = false;
 	bool pte_write_upgrade = vma_wants_manual_pte_write_upgrade(vma);
+	bool needs_promotion = false;
 	int last_cpupid;
 	int target_nid;
 	pte_t pte, old_pte;
@@ -6180,12 +6181,24 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 	nr_pages = folio_nr_pages(folio);
 
 	target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
 					writable, &last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
+		 * Isolation and migration are handled by pghot.
+		 */
+		nid = target_nid;
+		goto out_map;
+	}
+
+	/* Balancing b/n toptier nodes, mode=NUMA_BALANCING_NORMAL */
 	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
 		flags |= TNF_MIGRATE_FAIL;
 		goto out_map;
@@ -6225,8 +6238,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 					    writable);
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
 
-	if (nid != NUMA_NO_NODE)
-		task_numa_fault(last_cpupid, nid, nr_pages, flags);
+	if (nid != NUMA_NO_NODE) {
+		if (needs_promotion)
+			pghot_record_access(folio_pfn(folio), nid,
+					    PGHOT_HINTFAULTS, jiffies);
+		else
+			task_numa_fault(last_cpupid, nid, nr_pages, flags);
+	}
 	return 0;
 }
 
mm/mempolicy.c +0 -3
index 4e4421b22b59..aef9bb8a6cd4 100644
@@ -872,9 +872,6 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 	    node_is_toptier(nid))
 		return false;
 
-	if (folio_use_access_time(folio))
-		folio_xchg_access_time(folio, jiffies_to_msecs(jiffies));
-
 	return true;
 }
 
mm/migrate.c +15 -1
index 726d27b61a46..a468fa4f7963 100644
@@ -2709,8 +2709,18 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 	if (!migrate_balanced_pgdat(pgdat, nr_pages)) {
 		int z;
 
-		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING))
+		/*
+		 * Kswapd wakeup for creating headroom in toptier is done only
+		 * for hot page promotion case and not for misplaced migrations
+		 * between toptier nodes.
+		 *
+		 * In the uncommon case of using NUMA_BALANCING_NORMAL mode
+		 * to balance between lower and higher tier nodes, we end up
+		 * waking the kswapd.
+		 */
+		if (node_is_toptier(folio_nid(folio)))
 			return -EAGAIN;
+
 		for (z = pgdat->nr_zones - 1; z >= 0; z--) {
 			if (managed_zone(pgdat->node_zones + z))
 				break;
@@ -2760,6 +2770,8 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 #ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+#endif
+#ifdef CONFIG_PGHOT
 		if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
 		    && !node_is_toptier(folio_nid(folio))
 		    && node_is_toptier(node)) {
@@ -2824,6 +2836,8 @@ int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
 #ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+#endif
+#ifdef CONFIG_PGHOT
 		mod_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(node)),
 				 PGPROMOTE_SUCCESS, nr_succeeded);
 #endif
mm/pghot.c +134 -0

164 lines not on this page · raw message

mm/vmstat.c +1 -1
index 4064ead568cc..da668ff05032 100644
@@ -1268,7 +1268,7 @@ const char * const vmstat_text[] = {
 #ifdef CONFIG_SWAP
 	[I(NR_SWAPCACHE)]			= "nr_swapcached",
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	[I(PGPROMOTE_SUCCESS)]			= "pgpromote_success",
 	[I(PGPROMOTE_CANDIDATE)]		= "pgpromote_candidate",
 	[I(PGPROMOTE_CANDIDATE_NRL)]		= "pgpromote_candidate_nrl",
Signature · 1 line
2.34.1
#10 lore raw

Re: [PATCH v7 5/7] mm: sched: move NUMA balancing tiering promotion to pghot

Hi Bharata

Quoted 4 lines · Bharata B Rao
mm/pghot.c +3 -0
+/*
+ * For memory tiering mode, if there are enough free pages (more than
+ * enough watermark defined here) in fast memory node, to take full

mm/pghot.c int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now) · line 186

Quoted 10 lines
mm/pghot.c +10 -0
+ * advantage of fast memory capacity, all recently accessed slow
+ * memory pages will be migrated to fast memory node without
+ * considering hot threshold.
+ */
+static bool pgdat_free_space_enough(struct pglist_data *pgdat)
+{
+	int z;
+	unsigned long enough_wmark;
+
+	enough_wmark = max(1UL * 1024 * 1024 * 1024 >> PAGE_SHIFT,

Just a thought—would it be better to use #define for these hardcoded values?

-Donet

Quoted 2 lines
mm/pghot.c +2 -0
+			   pgdat->node_present_pages >> 4);
+	for (z = pgdat->nr_zones - 1; z >= 0; z--) {
#11 lore raw

Re: [PATCH v7 5/7] mm: sched: move NUMA balancing tiering promotion to pghot

Quoted 18 lines · Donet Tom

Hi Bharata

mm/pghot.c int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now) · line 186

Quoted 13 lines · Bharata B Rao

  +/*
+ * For memory tiering mode, if there are enough free pages (more than
+ * enough watermark defined here) in fast memory node, to take full
+ * advantage of fast memory capacity, all recently accessed slow
+ * memory pages will be migrated to fast memory node without
+ * considering hot threshold.
+ */

mm/pghot.c +6 -0
+static bool pgdat_free_space_enough(struct pglist_data *pgdat)
+{
+    int z;
+    unsigned long enough_wmark;
+
+    enough_wmark = max(1UL * 1024 * 1024 * 1024 >> PAGE_SHIFT,

Just a thought—would it be better to use #define for these hardcoded values?

We could. It was a code movement, hence left it untouched.

Regards,
Bharata.

#12 lore raw

RFC PATCH v7 6/7 x86/ibs: Move IBS caps definitions into its own header

In reply to Bharata B Rao

Subsequent patch adds IBS Memory Profiler driver that is independent of the perf subsystem but needs the CPUID 0x8000001B capability bits. Hence move those bit definitions out of asm/perf_event.h into a dedicated header so the new driver can consume them without pulling in perf.

---
 arch/x86/include/asm/ibs-caps.h   | 85 +++++++++++++++++++++++++++++++
 arch/x86/include/asm/perf_event.h | 81 +----------------------------
 2 files changed, 86 insertions(+), 80 deletions(-)
 create mode 100644 arch/x86/include/asm/ibs-caps.h
arch/x86/include/asm/ibs-caps.h new file +85 -0
new file mode 100644
index 000000000000..ddf6c512c8f9
@@ -0,0 +1,85 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_IBS_CAPS_H
+#define _ASM_X86_IBS_CAPS_H
+
+/*
+ * IBS cpuid feature detection
+ */
+
+#define IBS_CPUID_FEATURES		0x8000001b
+
+/*
+ * Same bit mask as for IBS cpuid feature flags (Fn8000_001B_EAX), but
+ * bit 0 is used to indicate the existence of IBS.
+ */
+#define IBS_CAPS_AVAIL			(1U<<0)
+#define IBS_CAPS_FETCHSAM		(1U<<1)
+#define IBS_CAPS_OPSAM			(1U<<2)
+#define IBS_CAPS_RDWROPCNT		(1U<<3)
+#define IBS_CAPS_OPCNT			(1U<<4)
+#define IBS_CAPS_BRNTRGT		(1U<<5)
+#define IBS_CAPS_OPCNTEXT		(1U<<6)
+#define IBS_CAPS_RIPINVALIDCHK		(1U<<7)
+#define IBS_CAPS_OPBRNFUSE		(1U<<8)
+#define IBS_CAPS_FETCHCTLEXTD		(1U<<9)
+#define IBS_CAPS_OPDATA4		(1U<<10)
+#define IBS_CAPS_ZEN4			(1U<<11)
+#define IBS_CAPS_OPLDLAT		(1U<<12)
+#define IBS_CAPS_DIS			(1U<<13)
+#define IBS_CAPS_FETCHLAT		(1U<<14)
+#define IBS_CAPS_BIT63_FILTER		(1U<<15)
+#define IBS_CAPS_STRMST_RMTSOCKET	(1U<<16)
+#define IBS_CAPS_OPDTLBPGSIZE		(1U<<19)
+
+#define IBS_CAPS_DEFAULT		(IBS_CAPS_AVAIL		\
+					 | IBS_CAPS_FETCHSAM	\
+					 | IBS_CAPS_OPSAM)
+
+/*
+ * IBS APIC setup
+ */
+#define IBSCTL				0x1cc
+#define IBSCTL_LVT_OFFSET_VALID		(1ULL<<8)
+#define IBSCTL_LVT_OFFSET_MASK		0x0F
+
+/* IBS fetch bits/masks */
+#define IBS_FETCH_L3MISSONLY		      (1ULL << 59)
+#define IBS_FETCH_RAND_EN		      (1ULL << 57)
+#define IBS_FETCH_VAL			      (1ULL << 49)
+#define IBS_FETCH_ENABLE		      (1ULL << 48)
+#define IBS_FETCH_CNT			     0xFFFF0000ULL
+#define IBS_FETCH_MAX_CNT		     0x0000FFFFULL
+
+#define IBS_FETCH_2_DIS			      (1ULL <<  0)
+#define IBS_FETCH_2_FETCHLAT_FILTER	    (0xFULL <<  1)
+#define IBS_FETCH_2_FETCHLAT_FILTER_SHIFT	       (1)
+#define IBS_FETCH_2_EXCL_RIP_63_EQ_1	      (1ULL <<  5)
+#define IBS_FETCH_2_EXCL_RIP_63_EQ_0	      (1ULL <<  6)
+
+/*
+ * IBS op bits/masks
+ * The lower 7 bits of the current count are random bits
+ * preloaded by hardware and ignored in software
+ */
+#define IBS_OP_LDLAT_EN			      (1ULL << 63)
+#define IBS_OP_LDLAT_THRSH		    (0xFULL << 59)
+#define IBS_OP_LDLAT_THRSH_SHIFT		      (59)
+#define IBS_OP_CUR_CNT			(0xFFF80ULL << 32)
+#define IBS_OP_CUR_CNT_RAND		(0x0007FULL << 32)
+#define IBS_OP_CUR_CNT_EXT_MASK		   (0x7FULL << 52)
+#define IBS_OP_CNT_CTL			      (1ULL << 19)
+#define IBS_OP_VAL			      (1ULL << 18)
+#define IBS_OP_ENABLE			      (1ULL << 17)
+#define IBS_OP_L3MISSONLY		      (1ULL << 16)
+#define IBS_OP_MAX_CNT			     0x0000FFFFULL
+#define IBS_OP_MAX_CNT_EXT		     0x007FFFFFULL	/* not a register bit mask */
+#define IBS_OP_MAX_CNT_EXT_MASK		   (0x7FULL << 20)	/* separate upper 7 bits */
+#define IBS_RIP_INVALID			      (1ULL << 38)
+
+#define IBS_OP_2_DIS			      (1ULL <<  0)
+#define IBS_OP_2_EXCL_RIP_63_EQ_0	      (1ULL <<  1)
+#define IBS_OP_2_EXCL_RIP_63_EQ_1	      (1ULL <<  2)
+#define IBS_OP_2_STRM_ST_FILTER		      (1ULL <<  3)
+#define IBS_OP_2_STRM_ST_FILTER_SHIFT		       (3)
+
+#endif /*  _ASM_X86_IBS_CAPS_H */
arch/x86/include/asm/perf_event.h +1 -80

93 lines not on this page · raw message

Signature · 1 line
2.34.1
#13 lore raw

RFC PATCH v7 7/7 x86/mm/ibs: In-kernel driver for AMD IBS Memory Profiler

In reply to Bharata B Rao

Use IBS (Instruction Based Sampling) Memory Profiler feature present in AMD Zen6 processors for memory access tracking. The access information obtained from IBS Memory Profiler is fed to pghot sub-system for further action using pghot_record_access(PGHOT_HWHINTS, ...) API.

IBS Memory Profiler as page hotness source is enabled by the new config option HWMEM_PROFILER and is also gated by the existing pghot_src_hwhints static key set via debugfs.

More details about IBS Memory Profiler can be obtained from the AMD document titled "AMD64 Zen6 Instruction Based Sampling (IBS) Extensions and Features".

---
 arch/x86/Kconfig                 |  16 ++
 arch/x86/include/asm/ibs-caps.h  |   8 +
 arch/x86/include/asm/ibs-mprof.h |  46 +++++
 arch/x86/include/asm/msr-index.h |   8 +
 arch/x86/mm/Makefile             |   1 +
 arch/x86/mm/ibs-mprof.c          | 308 +++++++++++++++++++++++++++++++
 include/linux/cpuhotplug.h       |   1 +
 include/linux/vm_event_item.h    |   6 +
 mm/Kconfig                       |   9 +
 mm/vmstat.c                      |   6 +
 10 files changed, 409 insertions(+)
 create mode 100644 arch/x86/include/asm/ibs-mprof.h
 create mode 100644 arch/x86/mm/ibs-mprof.c
arch/x86/Kconfig +16 -0
index 99bb5217649a..f06c0c44ecce 100644
@@ -1514,6 +1514,22 @@ config AMD_MEM_ENCRYPT
 	  This requires an AMD processor that supports Secure Memory
 	  Encryption (SME).
 
+config AMD_IBS_MEMPROF
+	bool "AMD IBS Memory Profiler"
+	depends on X86_64 && CPU_SUP_AMD
+	depends on PGHOT
+	select HWMEM_PROFILER
+	help
+	  Use the AMD Instruction Based Sampling (IBS) Memory Profiler
+	  facility (present on Zen6 and later AMD CPUs) to feed
+	  hardware-observed memory accesses into the pghot subsystem
+	  for hot-page detection and promotion.
+
+	  When disabled, no IBS Memory Profiler MSRs are programmed and
+	  the corresponding NMI handler is not installed.
+
+	  If unsure, say N.
+
 # Common NUMA Features
 config NUMA
 	bool "NUMA Memory Allocation and Scheduler Support"
arch/x86/include/asm/ibs-caps.h +8 -0
index ddf6c512c8f9..1f6c4058a0e3 100644
@@ -29,6 +29,7 @@
 #define IBS_CAPS_FETCHLAT		(1U<<14)
 #define IBS_CAPS_BIT63_FILTER		(1U<<15)
 #define IBS_CAPS_STRMST_RMTSOCKET	(1U<<16)
+#define IBS_CAPS_MEM_PROFILER		(1U<<18)
 #define IBS_CAPS_OPDTLBPGSIZE		(1U<<19)
 
 #define IBS_CAPS_DEFAULT		(IBS_CAPS_AVAIL		\
@@ -42,6 +43,13 @@
 #define IBSCTL_LVT_OFFSET_VALID		(1ULL<<8)
 #define IBSCTL_LVT_OFFSET_MASK		0x0F
 
+/*
+ * IBS Memprofiler setup
+ */
+#define IBSCTL_MPROF_LVT_OFFSET_VALID	(1ULL << 24)
+#define IBSCTL_MPROF_LVT_OFFSET_SHIFT	16
+#define IBSCTL_MPROF_LVT_OFFSET_MASK	(0xFULL << IBSCTL_MPROF_LVT_OFFSET_SHIFT)
+
 /* IBS fetch bits/masks */
 #define IBS_FETCH_L3MISSONLY		      (1ULL << 59)
 #define IBS_FETCH_RAND_EN		      (1ULL << 57)
arch/x86/include/asm/ibs-mprof.h new file +46 -0

46 lines not on this page · raw message

arch/x86/include/asm/msr-index.h +8 -0
index a14a0f43e04a..c44b68940f43 100644
@@ -1315,4 +1315,12 @@
 						* a #GP
 						*/
 
+/* AMD IBS Memory Profiler MSRs */
+#define MSR_AMD64_IBS_MPROF_CTL		0xc0010380
+#define MSR_AMD64_IBS_MPROF_CTL2	0xc0010381
+#define MSR_AMD64_IBS_MPROF_DATA2	0xc0010382
+#define MSR_AMD64_IBS_MPROF_DATA3	0xc0010383
+#define MSR_AMD64_IBS_MPROF_LINADDR	0xc0010384
+#define MSR_AMD64_IBS_MPROF_PHYADDR	0xc0010385
+
 #endif /* _ASM_X86_MSR_INDEX_H */
arch/x86/mm/Makefile +1 -0
index 3a5364853eab..050a7379d9f7 100644
@@ -59,3 +59,4 @@ obj-$(CONFIG_X86_MEM_ENCRYPT)	+= mem_encrypt.o
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_amd.o
 
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_boot.o
+obj-$(CONFIG_AMD_IBS_MEMPROF)	+= ibs-mprof.o
arch/x86/mm/ibs-mprof.c new file +308 -0

308 lines not on this page · raw message

include/linux/cpuhotplug.h +1 -0
index 22ba327ec227..feaa3f571726 100644
@@ -150,6 +150,7 @@ enum cpuhp_state {
 	CPUHP_AP_PERF_X86_AMD_UNCORE_STARTING,
 	CPUHP_AP_PERF_X86_STARTING,
 	CPUHP_AP_PERF_X86_AMD_IBS_STARTING,
+	CPUHP_AP_MM_AMD_IBS_MEMPROF_STARTING,
 	CPUHP_AP_PERF_XTENSA_STARTING,
 	CPUHP_AP_ARM_VFP_STARTING,
 	CPUHP_AP_ARM64_DEBUG_MONITORS_STARTING,
include/linux/vm_event_item.h +6 -0
index 58d510711bd4..a9c04a9735c6 100644
@@ -179,6 +179,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		PGHOT_RECORDED_ACCESSES,
 		PGHOT_RECORDED_HINTFAULTS,
 		PGHOT_RECORDED_HWHINTS,
+#ifdef CONFIG_HWMEM_PROFILER
+		HWHINT_TOTAL_EVENTS,
+		HWHINT_DRAM_ACCESSES,
+		HWHINT_EXTMEM_ACCESSES,
+		HWHINT_USEFUL_EVENTS,
+#endif /* CONFIG_HWMEM_PROFILER */
 #endif /* CONFIG_PGHOT */
 		NR_VM_EVENT_ITEMS
 };
mm/Kconfig +9 -0
index cc4b5685ecd4..674cfcea7bb0 100644
@@ -1494,6 +1494,15 @@ config PGHOT_PRECISE
 	  4 bytes per page against the default one byte per page. Preferable
 	  to enable this on systems with multiple nodes in toptier.
 
+config HWMEM_PROFILER
+	bool
+	depends on PGHOT
+	help
+	  Umbrella symbol enabled by any in-kernel driver that forwards
+	  hardware-observed memory accesses to the pghot subsystem (for
+	  example AMD_IBS_MEMPROF on x86_64). Drivers select this; users
+	  do not enable it directly.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/vmstat.c +6 -0
index da668ff05032..06e7ae06519e 100644
@@ -1493,6 +1493,12 @@ const char * const vmstat_text[] = {
 	[I(PGHOT_RECORDED_ACCESSES)]		= "pghot_recorded_accesses",
 	[I(PGHOT_RECORDED_HINTFAULTS)]		= "pghot_recorded_hintfaults",
 	[I(PGHOT_RECORDED_HWHINTS)]		= "pghot_recorded_hwhints",
+#ifdef CONFIG_HWMEM_PROFILER
+	[I(HWHINT_TOTAL_EVENTS)]		= "hwhint_total_events",
+	[I(HWHINT_DRAM_ACCESSES)]		= "hwhint_dram_accesses",
+	[I(HWHINT_EXTMEM_ACCESSES)]		= "hwhint_extmem_accesses",
+	[I(HWHINT_USEFUL_EVENTS)]		= "hwhint_useful_events",
+#endif /* CONFIG_HWMEM_PROFILER */
 #endif /* CONFIG_PGHOT */
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
Signature · 1 line
2.34.1
#14 lore raw
In reply to Bharata B Rao
Quoted 4 lines · Bharata B Rao

Results
=======
Posted as replies to this mail thread.

Micro-benchmark numbers for IBS Memory Profiler pghot source

Test system details
-------------------
2 node AMD system with 1 regular NUMA node (0) and a CXL node (1)

$ numactl -H
available: 3 nodes (0-1)
node 0 cpus: 0-255
node 0 size: 515563 MB
node 1 cpus:
node 1 size: 258034 MB
node distances:
node distances:
node   0   1
  0:  10  50
  1:  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the patched case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case HWHINTS - IBS Memory Profiler as source for pghot

Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

==============================================================
Scenario 1 - Enough memory in toptier and hence only promotion
==============================================================
Multi-threaded application with 64 threads that access memory(8G) at 4K granularity repetitively and randomly. The number of accesses per thread and the randomness pattern for each thread are fixed beforehand. The accesses are divided into stores and loads in the ratio of 50:50.

Benchmark threads run on Node 0, while memory is initially provisioned on CXL node 1 before the accesses start.

Repetitive accesses results in lowertier pages becoming hot and kmigrated detecting and migrating them. The benchmark score is the time taken to finish the accesses in microseconds. The sooner it finishes the better it is. All the numbers shown below are average of 3 runs.

Time taken (microseconds, lower is better)
---------------------------------------------------------
Source Base Pghot-default
---------------------------------------------------------

NUMAB0          181,393,365     184,331,381
NUMAB2          42,287,528
HWHINTS         NA              50,422,862

---------------------------------------------------------

Stats comparision b/n base-NUMAB2 and pghot-default-hwhints
---------------------------------------------------------------------
Base-NUMAB2 Pghot-default-hwhints
---------------------------------------------------------------------

pgpromote_success               2097152         1961087
numa_hint_faults                2358069         0
pghot_recorded_accesses         NA              1962696
pghot_recorded_hintfaults       NA              0
pghot_recorded_hwhints          NA              5532979
hwhint_total_events             NA              5532979

---------------------------------------------------------------------

#16 lore raw
Quoted 5 lines · Matthew Wilcox
Quoted 1 line · Bharata B Rao

This is v7 of pghot, a hot-page tracking and promotion subsystem. The

I continue to think we should not do this.

I am unclear about the benefits of the patchset, I have not tested it or reviewed the latest revision. My big concern was that top-tier might not always be suitable.

I see that there are some numbers posted, but I find this weird "After the graph creation, the processes are stopped and data is migrated to CXL node 2 before continuing so that BFS phase starts accessing lower tier memory." Why not allocate everything on CXL node 2?

Balbir

#17 lore raw
Quoted 10 lines · Balbir Singh
Quoted 5 lines · Matthew Wilcox
Quoted 1 line · Bharata B Rao

This is v7 of pghot, a hot-page tracking and promotion subsystem. The

I continue to think we should not do this.

I am unclear about the benefits of the patchset, I have not tested it or reviewed the latest revision. My big concern was that top-tier might not always be suitable.

So you are saying that we should have a capability to promote accessed pages from lower tier to an other tier that is not classified as top tier? Is that non-top tier node the one which generates accesses?

Quoted 5 lines

I see that there are some numbers posted, but I find this weird "After the graph creation, the processes are stopped and data is migrated to CXL node 2 before continuing so that BFS phase starts accessing lower tier memory." Why not allocate everything on CXL node 2?

In the ideal scenario, the benefit is to see if any pages that land up on lower tier get identified as hot and get promoted. That means we need to create an over-committed scenario where the pages get demoted first. I have provided numbers from such cases in my previous versions. The problem with this case is that the base hot page promotion (NUMAB2) hasn't shown any benefit at all with my micro-benchmark - Ref:
https://lore.kernel.org/linux-mm/[email protected]/

Same has been observed with redis-memtier benchmark -
https://lore.kernel.org/linux-mm/[email protected]/

Instead what I am doing here is to take out demotion from the scenario but still retain the access pattern of the benchmark by pushing out the data to lower tier when the benchmark reaches steady allocation state.

Regards,
Bharata.

#18 lore raw
Quoted 8 lines · Bharata B Rao
Quoted 7 lines · Balbir Singh
Quoted 5 lines · Matthew Wilcox
Quoted 1 line · Bharata B Rao

This is v7 of pghot, a hot-page tracking and promotion subsystem. The

I continue to think we should not do this.

Quoted 8 lines
Quoted 3 lines

I am unclear about the benefits of the patchset, I have not tested it or reviewed the latest revision. My big concern was that top-tier might not always be suitable.

So you are saying that we should have a capability to promote accessed pages from lower tier to an other tier that is not classified as top tier? Is that non-top tier node the one which generates accesses?

Yes, a top tier node could be CPU less for example.

Quoted 9 lines
Quoted 5 lines

I see that there are some numbers posted, but I find this weird "After the graph creation, the processes are stopped and data is migrated to CXL node 2 before continuing so that BFS phase starts accessing lower tier memory." Why not allocate everything on CXL node 2?

In the ideal scenario, the benefit is to see if any pages that land up on lower tier get identified as hot and get promoted. That means we need to create an over-committed scenario where the pages get demoted first. I have provided

Why do the pages need to get demoted? Why not allocate them from the lower tier to show that promotion upwards is helpful

Quoted 5 lines

numbers from such cases in my previous versions. The problem with this case is that the base hot page promotion (NUMAB2) hasn't shown any benefit at all with my micro-benchmark - Ref:
https://lore.kernel.org/linux-mm/[email protected]/

Quoted 7 lines

Same has been observed with redis-memtier benchmark -
https://lore.kernel.org/linux-mm/[email protected]/

Instead what I am doing here is to take out demotion from the scenario but still retain the access pattern of the benchmark by pushing out the data to lower tier when the benchmark reaches steady allocation state.

Balbir

#19 lore raw
Quoted 10 lines · Balbir Singh
Quoted 8 lines
Quoted 3 lines

I am unclear about the benefits of the patchset, I have not tested it or reviewed the latest revision. My big concern was that top-tier might not always be suitable.

So you are saying that we should have a capability to promote accessed pages from lower tier to an other tier that is not classified as top tier? Is that non-top tier node the one which generates accesses?

Yes, a top tier node could be CPU less for example.

Currently kmigrated thread in pghot doesn't explicitly prevent promotion to non-toptier nodes. Here is how this works for the two modes of operation in pghot:

pghot-default: In this mode, the target NID isn't explicitly tracked and hence kmigrated relies on the user-configurable pghot_target_nid. Though there is a !node_is_toptier(nid) check in the helper routine that populates pghot_target_nid, that can be relaxed if required.

pghot-precise: In this mode, the accessing CPU's node is tracked as the target nid and promotion is done to that node. Note that pghot_target_nid isn't used here.

Hence I don't see any major issues in this patchset to cover your use case. Let me know if I miss anything here.

BTW, does the existing hot page promotion cover the use case you are targeting?

Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

I see that there are some numbers posted, but I find this weird "After the graph creation, the processes are stopped and data is migrated to CXL node 2 before continuing so that BFS phase starts accessing lower tier memory." Why not allocate everything on CXL node 2?

Quoted 6 lines
Quoted 3 lines

In the ideal scenario, the benefit is to see if any pages that land up on lower tier get identified as hot and get promoted. That means we need to create an over-committed scenario where the pages get demoted first. I have provided

Why do the pages need to get demoted? Why not allocate them from the lower tier to show that promotion upwards is helpful

As you can see, these are controlled experiments to measure the effectiveness of hot page detection and promotion and the benefits from promotion. It can be done in the way you are suggesting; just that I found it a bit simpler to pause the benchmark, migrate all pages to lower tier memory before the benchmark starts accessing them rather than relying on setting memory policies to achieve the same effect.

Regards,
Bharata.

maintainer
#20 lore raw
In reply to Matthew Wilcox
Quoted 4 lines · Matthew Wilcox
Quoted 1 line · Bharata B Rao

This is v7 of pghot, a hot-page tracking and promotion subsystem. The

I continue to think we should not do this.

My only pushback on the general "we should not do this" is that we need something to counter-balance the demotion bit in vmscan.c, and the current implementation (prot_none faults) is rather :[

I think this series needs to greatly limit its complexity and provide some gentle correction for LRU inversions, and I think they're making a decent attempt at that.

But then I think local memory expansion on CXL is going pretty swimmingly in our datacenters :], others may not feel the same.

~Gregory

#21 lore raw
Quoted 9 lines · Gregory Price
Quoted 4 lines · Matthew Wilcox
Quoted 1 line · Bharata B Rao

This is v7 of pghot, a hot-page tracking and promotion subsystem. The

I continue to think we should not do this.

My only pushback on the general "we should not do this" is that we need something to counter-balance the demotion bit in vmscan.c, and the current implementation (prot_none faults) is rather :[

So you are saying pghot subsystem currently does hot page detection and promotion only, which is fine. But the current implementation of demotion is not very optimal and hence we should spend effort in fine-tuning demotion first?

In this series itself I have shown via benchmark numbers that for over-committed cases (involving both demotion and promotion), the workload isn't really showing real benefit due to demotion and promotion. Are you specifically referring to this problem?

Quoted 4 lines

I think this series needs to greatly limit its complexity and provide some gentle correction for LRU inversions, and I think they're making a decent attempt at that.

Regarding complexity, I agree that the initial version of this patchset was quite complicated in the way it maintained hot page information. But the later versions including this one have greatly reduced the complexity with one byte of hot page information per PFN, atomic updates to hotness data without any locks, per-lowertier kmigrated threads for promotion and reuse of existing hot page promotion engine.

Did you have anything else in mind wrt complexity?

Can you provide more context about the LRU inversion problem?

Regards,
Bharata.

maintainer
#22 lore raw
Quoted 9 lines · Bharata B Rao
Quoted 6 lines · Gregory Price
Quoted 4 lines · Matthew Wilcox
Quoted 1 line · Bharata B Rao

This is v7 of pghot, a hot-page tracking and promotion subsystem. The

I continue to think we should not do this.

Quoted 8 lines
Quoted 3 lines

My only pushback on the general "we should not do this" is that we need something to counter-balance the demotion bit in vmscan.c, and the current implementation (prot_none faults) is rather :[

So you are saying pghot subsystem currently does hot page detection and promotion only, which is fine. But the current implementation of demotion is not very optimal and hence we should spend effort in fine-tuning demotion first?

I'm saying because of demotion and fallbacks, we need a mechanism to handle promotions. I'm not convinced a hotness will extend to coldness
- at least any better than LRU/MGLRU.

Quoted 5 lines

In this series itself I have shown via benchmark numbers that for over-committed cases (involving both demotion and promotion), the workload isn't really showing real benefit due to demotion and promotion. Are you specifically referring to this problem?

If over-committed means over-subscribed hot-tier (more hot memory than available top tier memory), then yeah that result is intuitive. I haven't pointed to any specific issue, as of yet, still taking time to consider some of the results.

Quoted 3 lines

Can you provide more context about the LRU inversion problem?

I've been tracking some data around shrink_folio_list and alloc_migrate_folio behavior when a low tier node is full.

The result is we end up just swapping memory from high tier straight to swap and skip demotion, resulting in a bunch of file and anon refaults.

Hardware: Single Socket, 768GB DRAM, 256GB CXL Expander

In this workload, we see swap usage after the full 1TB of memory is utilized, and as a result we see swap spillage.

second_chance = second alloc attempt in alloc_migrate_folio succeeds swap_fallback = second chance fails, we swap directly from top tier

Sample data:

pgdemote_kswapd 333052779
pgdemote_direct 3181480482
pgdemote_second_chance 31017629
pgdemote_swap_fallback 335759535
workingset_refault_anon 30106868
workingset_refault_file 2343035341

(note here: swap fallback is number of occurances, while the others are
number of pages. As a result, the actual number of swapped pages is likely much closer to the pgdemote_direct number)

As a result: LRU is just broken on CXL systems, LRU inverts by design.

In a sane world we would just see the second tier as an extention of the LRU, but that doesn't necessarily mean we can gleen hotness data from it (it's still largely a coldness tracking mechanism).

I have patches I haven't RFC'd yet that try to address this, but I need more time to test it.

I don't think this is something to address with PGHot.

---

mm/vmscan.c +6 -1
index 112983b42559..ccdd698c5937 100644
@@ -1043,7 +1043,10 @@ struct folio *alloc_migrate_folio(struct folio *src, unsigned long private)
        mtc->gfp_mask &= ~__GFP_THISNODE;
        mtc->nmask = allowed_mask;
 
-       return alloc_migration_target(src, (unsigned long)mtc);
+       dst = alloc_migration_target(src, (unsigned long)mtc);
+       if (dst)
+               count_vm_events(PGDEMOTE_SECOND_CHANCE, folio_nr_pages(src));
+       return dst;
 }
 
 /*
@@ -1616,6 +1619,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
        /* Folios that could not be demoted are still in @demote_folios */
        if (!list_empty(&demote_folios)) {
                /* Folios which weren't demoted go back on @folio_list */
+               if (!sc->proactive)
+                       count_vm_event(PGDEMOTE_SWAP_FALLBACK);
                list_splice_init(&demote_folios, folio_list);
 
                /*
#23 lore raw
In reply to Bharata B Rao
Quoted 3 lines · Bharata B Rao

Results
=======
Posted as replies to this mail thread.

Graph500 benchmark results:

Test system details
-------------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0-95,192-287
node 0 size: 128460 MB
node 1 cpus: 96-191,288-383
node 1 size: 128893 MB
node 2 cpus:
node 2 size: 257993 MB
node distances:
node   0   1   2
  0:  10  32  50
  1:  32  10  60
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the pghot case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the pghot case.
NUMAB3 - Enabled both regular and tiering mode of NUMA Balancing
(kernel.numa_balancing=3)

Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

Graph500 details
----------------
Command: mpirun -n 128 --bind-to core --map-by core graph500/src/graph500_reference_bfs 28 16

After the graph creation, the processes are stopped and data is migrated to CXL node 2 before continuing so that BFS phase starts accessing lower tier memory.

Total memory usage is slightly over 100GB and will fit within Node 0 and 1. Hence there is no memory pressure to induce demotions.

harmonic_mean_TEPS - Higher is better
=====================================================================================
Base Base pghot-default
pghot-precise
NUMAB0 NUMAB2 NUMAB2 NUMAB2
=====================================================================================

harmonic_mean_TEPS      5.08026e+08     7.48633e+08     5.46257e+08     7.45101e+08
mean_time               8.45413         5.73702         7.86245         5.76421
median_TEPS             5.09236e+08     7.25058e+08     5.40525e+08     7.63752e+08
max_TEPS                5.15244e+08     1.03391e+09     8.51317e+08     9.7552e+08
pgpromote_success       0               13809474        13763582        13763155
numa_pte_updates        0               26746117        39502157        36368086
numa_hint_faults        0               13811769        24248272        21172314

=====================================================================================
pghot-default NUMAB3
=====================================================================================

harmonic_mean_TEPS                                      7.00515e+08
mean_time                                               6.13109
median_TEPS                                             7.06813e+08
max_TEPS                                                7.63164e+08
pgpromote_success                                       13762087
numa_pte_updates                                        93632490
numa_hint_faults                                        70566306

=====================================================================================
- The base case shows a good improvement with NUMAB2 in harmonic_mean_TEPS.
- The same improvement gets maintained with pghot-precise too.
- pghot-default mode doesn't show benefit even when achieving similar page promotion numbers. This mode doesn't track accessing NID and by default promotes to NID=0 which probably isn't all that beneficial as processes are running on both Node 0 and Node 1.
- pghot-default recovers the performance when balancing between toptier nodes 0 and 1 is enabled in addition to hot page promotion.

maintainer
#24 lore raw
Quoted 6 lines · On Tue, 5 May 2026 16:11:43 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 3 lines · Bharata B Rao

Results
=======
Posted as replies to this mail thread.

Graph500 benchmark results:

Please include (and maintain) the testing results in the formal changelogs (perhaps in the [0/N], in a condensed summary form).

I mean, the entire point of the whole patchset is to improve performance (yes?), so this contribution lives or dies by its performance testing results.

The first thing your audience will want to know is "how good is this for our users". So tell us! Up front, within the first paragraphs!

The better the results, the more motivated people will be to help get your work upstream.

#25 lore raw
Quoted 11 lines · Andrew Morton
Quoted 6 lines · On Tue, 5 May 2026 16:11:43 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 3 lines · Bharata B Rao

Results
=======
Posted as replies to this mail thread.

Graph500 benchmark results:

Please include (and maintain) the testing results in the formal changelogs (perhaps in the [0/N], in a condensed summary form).

The results and associated description were getting too long and hence I was hesitating to make it part of 0/N. But then as you say, I shall include a condensed summary from next time.

Quoted 4 lines

I mean, the entire point of the whole patchset is to improve performance (yes?), so this contribution lives or dies by its performance testing results.

The entire point of this patchset is not just about improving the performance. It is mainly about adding a new dedicated infrastructure for detecting and promoting hot pages. It is about having a subsystem that can act as a single source of truth page hotness in the kernel. Though we aren't there yet, we have started by having a minimal infrastructure that centralizes the hot page promotion and associated heuristics that currently sits in scheduler so that the same can be used with other page hotness sources as well.

The first source is the hintfaults based hot page promotion. Here the address space scanning and introduction of hint faults still remains like earlier. But the promotion engine is part of pghot. Hence the comparison numbers with base this source is about meeting the current level of performance and ensuring that the workloads don't suffer due to batched migration.

There are other sources as well with primary one being the IBS Memory Profiler which provides memory access information directly from the hardware. I have some numbers for this source as well. Initial results look encouraging and more tests can tell us if this source can be an independent one or complements the existing one.

Then the earlier versions of this patchset had another source - PTE A bit based scanning where the idea was to completely replace the hint fault based mechanism by PTE A bit based accesses thereby taking out both the detection and promotion parts out of the process context. I have temporarily removed this from this patchset for two reasons: a) to simplify the patchset so that we can get some consensus on the infrastructure part first. b) to explore the commonality with another PTE A bit scanning approach (called klruscand) that used MGLRU's scanning mechanism.

Also on the horizon is to use hot page info that CXL Hotness Monitoring Unit (CHMU) can provide.

Quoted 6 lines

The first thing your audience will want to know is "how good is this for our users". So tell us! Up front, within the first paragraphs!

The better the results, the more motivated people will be to help get your work upstream.

So currently it is a multi-step approach with first step of building a common hotness infrastructure and moving existing mechanism to make use of it w/o any regression. Then follow up with more sources.

Regards,
Bharata.

maintainer
#26 lore raw
Quoted 9 lines · Bharata B Rao

The entire point of this patchset is not just about improving the performance. It is mainly about adding a new dedicated infrastructure for detecting and promoting hot pages. It is about having a subsystem that can act as a single source of truth page hotness in the kernel. Though we aren't there yet, we have started by having a minimal infrastructure that centralizes the hot page promotion and associated heuristics that currently sits in scheduler so that the same can be used with other page hotness sources as well.

The goal of hotness tracking in general is to improve performance.

The goal of PGHot should be a reasonable baseline for the kernel to course-correct LRU inversions across tiers over time, because LRU threads only scan invidiual nodes and don't compare across nodes.

I would hazard against trying to wholesale state it "Shall be the single source of truth", as we will inevitably discover some condition which is not covered / cannot be captured / we will simply get it wrong.

Plus, intuitively, counter-balancing LRU/MGLRU aging is probably as good good as we can get without having to inject per-workload information into the system - at which point the users should use DAMON.

~Gregory

#27 lore raw
Quoted 11 lines · Gregory Price
Quoted 9 lines · Bharata B Rao

The entire point of this patchset is not just about improving the performance. It is mainly about adding a new dedicated infrastructure for detecting and promoting hot pages. It is about having a subsystem that can act as a single source of truth page hotness in the kernel. Though we aren't there yet, we have started by having a minimal infrastructure that centralizes the hot page promotion and associated heuristics that currently sits in scheduler so that the same can be used with other page hotness sources as well.

Quoted 1 line

The goal of hotness tracking in general is to improve performance.

Agreed. As I have mentioned elsewhere in the thread, right now we have just moved the existing promotion mechanism to pghot, hence the initial concern has been to ensure the earlier performance levels are still met with centralized promotion engine that does batched promotions from non-process context.

Quoted 4 lines

The goal of PGHot should be a reasonable baseline for the kernel to course-correct LRU inversions across tiers over time, because LRU threads only scan invidiual nodes and don't compare across nodes.

Right.

Quoted 4 lines

I would hazard against trying to wholesale state it "Shall be the single source of truth", as we will inevitably discover some condition which is not covered / cannot be captured / we will simply get it wrong.

Yeah. The ideal goal of single source of truth may be a bit far fetched but pghot is definitely a subsystem that can work with multiple page hotness sources, aggregate hot signals from them and provide a single unified promotion mechanism.

Regards,
Bharata.

#28 lore raw
In reply to Bharata B Rao
Quoted 3 lines · Bharata B Rao

Results
=======
Posted as replies to this mail thread.

Initial Graph500 benchmark numbers for IBS Memory Profiler source:

Test system details
-------------------
3 node AMD system with 2 regular NUMA nodes (0, 1) in NPS2 mode and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node distances:
node 0 cpus: 0-63,128-191
node 0 size: 257715 MB
node 1 cpus: 64-127,192-255
node 1 size: 257845 MB
node 2 cpus:
node 2 size: 258032 MB
node distances:
node   0   1   2
  0:  10  12  50
  1:  12  10  50
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the pghot case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the pghot case.
HWHINTS - IBS Memory Profiler as source for pghot

Pghot by default promotes after two accesses but for NUMAB2 and HWHINTS sources, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

Graph500 details
----------------
Command: mpirun -n 128 --bind-to core --map-by core graph500/src/graph500_reference_bfs 28 16

After the graph creation, the processes are stopped and data is migrated to CXL node 2 before continuing so that BFS phase starts accessing lower tier memory.

Total memory usage is slightly over 100GB and will fit within Node 0 and 1. Hence there is no memory pressure to induce demotions.

harmonic_mean_TEPS - Higher is better
=============================================================================

                                Base            Base            pghot-default
                                NUMAB0          NUMAB2          NUMAB2
=============================================================================
harmonic_mean_TEPS              4.09614e+08     1.28401e+09     1.47926e+09
mean_time                       10.4853         3.34492         2.90342
median_TEPS                     4.10086e+08     1.44584e+09     1.85957e+09
max_TEPS                        4.1661e+08      1.79773e+09     1.99242e+09
pgpromote_success               0               13746029        13412213
numa_hint_faults                0               13753808        26669823
pghot_recorded_accesses         NA              NA              26669551
pghot_recorded_hintfaults       NA              NA              26669823
pghot_recorded_hwhints          NA              NA              0
hwhint_total_events             NA              NA              0

=============================================================================
pghot-default HWHINTS
=============================================================================

harmonic_mean_TEPS                                              1.52334e+09
mean_time                                                       2.81941
median_TEPS                                                     1.57446e+09
max_TEPS                                                        1.72014e+09

pgpromote_success 3415599
numa_hint_faults 0

pghot_recorded_accesses                                         3440912
pghot_recorded_hintfaults                                       0
pghot_recorded_hwhints                                          24475210
hwhint_total_events                                             24475244

=============================================================================
While no migration (NUMAB0) at all hurts Graph500, HWHINTS with pghot is able to provide similar benchmark numbers even when not migrating as aggressively as base NUMAB2.