Skip to contentSkip to conversation

mm: Hot page tracking and promotion infrastructure

RFC PATCH Superseded Bharata B Rao 9 participants 53 messages Sep 10, 2025 – Oct 23, 2025

Subsystems (10)

v2 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v2 cover mm: Hot page tracking and promotion infrastructure

Hi,

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) that consolidates memory access information from various sources and enables centralized promotion of hot pages across memory tiers.

Currently, multiple kernel subsystems detect page accesses independently. For eg.

- NUMA Balancing via hint faults
- MGLRU via page table scanning for PTE A bit

This patchset consolidates the accesses from these mechanisms by providing a common API for reporting page accesses and a shared infrastructure for tracking hotness at PFN granularity and per-node kernel threads for promoting pages.

Here is a brief summary of how this subsystem works:

- Tracks frequency, last access time and accessing node as part of each access record.
- Maintains per-PFN access records in hash lists.
- Classifies pages as hot based on configurable thresholds.
- Uses per-toptier-node max-heaps to prioritize hot pages for promotion.
- Launches per-toptier-node kpromoted threads to perform batched migrations.

When different subsystems report page accesses via the API introduced by this new subsystem, a record for each such page is stored in hash lists (hashed by PFN value). In addition to the PFN and target_nid, the hotness record includes parameters like frequency and time of access from which the hotness is derived. Repeated reporting of access on the same PFN will result in updating of hotness information. When the hotness of a record (as updated during reporting of access) crosses a threshold, the record becomes part of a max heap data structure. Records in the max heap are arranged based on the hotness and hence the top elements of the heap will correspond to the hottest pages. There will be one such heap for each toptier node so that per-toptier-node kpromoted thread can easily extract the top N records from its own heap and perform batched migration.

Three page hotness sources have been integrated with pghot subsystem on experimental basis:

1. IBS
2. klruscand (based on MGLRU page table walks)
3. NUMA Balancing (mode 2).

Changes in v2
=============
- Moved migration rate-limiting and dynamic threshold logic from NUMA Balancing subsystem to pghot. With this, the logic to classify a page as hot resembles more closely to the existing mechanism.
- Converted NUMA Balancing mode 2 to just detect accesses through NUMA hint faults and delegate rest of the processing (hot page classification and promotion) to pghot.
- Packed the three parameters required for hot page tracking (nid, frequency and timestamp) into a single u32 for space efficiency.
- Misc cleanups and refactoring.

This v2 patchset applies on top of upstream commit 8742b2d8935f and can be fetched from:
https://github.com/AMDESE/linux-mm/tree/bharata/kpromoted-rfcv2

v1: https://lore.kernel.org/linux-mm/[email protected]/
v0: https://lore.kernel.org/linux-mm/[email protected]/

TODOs
=====
- Memory allocation: High volume of allocations and frees (millions) from atomic context needs evaluation.
- Memory overhead: The amount of data needed for tracking hotness is also a concern.
- Integrate Kscand[1], the PTE A bit based approach that Raghavendra KT is working upon, so that Kscand acts as temperature sources and uses pghot for hot page heuristics and promotion.
- Heap pruning: Consider adding heap pruning mechanism for periodic cleaning of cold records.
- Address Ying Huang's comment about merging migrate_misplaced_folio() and migrate_misplaced_folios_batch() and correctly handling memcg stats counting properly in the latter.
- Testing: Light functional testing done; performance benchmarking and stress testing will follow in the next iterations.

Any feedback is welcome!

Bharata B Rao (5):
mm: migrate: Allow misplaced migration without VMA too
mm: Hot page tracking and promotion
x86: ibs: In-kernel IBS driver for memory access profiling
x86: ibs: Enable IBS profiling for memory accesses
mm: sched: Move hot page promotion from NUMAB=2 to kpromoted

Gregory Price (1):
migrate: implement migrate_misplaced_folios_batch

Kinsey Ho (2):
mm: mglru: generalize page table walk
mm: klruscand: use mglru scanning for page promotion

 arch/x86/events/amd/ibs.c           |  11 +
 arch/x86/include/asm/entry-common.h |   3 +
 arch/x86/include/asm/hardirq.h      |   2 +
 arch/x86/include/asm/ibs.h          |   9 +
 arch/x86/include/asm/msr-index.h    |  16 +
 arch/x86/mm/Makefile                |   3 +-
 arch/x86/mm/ibs.c                   | 343 +++++++++++++++
 include/linux/migrate.h             |   6 +
 include/linux/mmzone.h              |  16 +
 include/linux/pghot.h               |  98 +++++
 include/linux/vm_event_item.h       |  26 ++
 kernel/sched/fair.c                 | 149 +------
 mm/Kconfig                          |  19 +
 mm/Makefile                         |   2 +
 mm/internal.h                       |   4 +
 mm/klruscand.c                      | 118 +++++
 mm/memory.c                         |  32 +-
 mm/migrate.c                        |  36 +-
 mm/mm_init.c                        |  10 +
 mm/pghot.c                          | 648 ++++++++++++++++++++++++++++
 mm/vmscan.c                         | 176 ++++++--
 mm/vmstat.c                         |  26 ++
 22 files changed, 1535 insertions(+), 218 deletions(-)
 create mode 100644 arch/x86/include/asm/ibs.h
 create mode 100644 arch/x86/mm/ibs.c
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/klruscand.c
 create mode 100644 mm/pghot.c

[1] Kscand - https://lore.kernel.org/linux-mm/[email protected]/

Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH v2 1/8 mm: migrate: Allow misplaced migration without VMA too

We want isolation of misplaced folios to work in contexts where VMA isn't available. In order to prepare for that allow migrate_misplaced_folio_prepare() to be called with a NULL VMA.

When migrate_misplaced_folio_prepare() is called with non-NULL VMA, it will check if the folio is mapped shared and that requires holding PTL lock. This path isn't taken when the function is invoked with NULL VMA (migration outside of process context). Hence for such cases, it is not necessary this function be called with PTL lock held.

---
 mm/migrate.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/migrate.c +3 -2
index 425401b2d4e1..7e356c0b1b5a 100644
@@ -2619,7 +2619,8 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,
 
 /*
  * Prepare for calling migrate_misplaced_folio() by isolating the folio if
- * permitted. Must be called with the PTL still held.
+ * permitted. Must be called with the PTL still held if called with a non-NULL
+ * vma.
  */
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -2636,7 +2637,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 		 * See folio_maybe_mapped_shared() on possible imprecision
 		 * when we cannot easily detect if a folio is shared.
 		 */
-		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+		if (vma && (vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
 			return -EACCES;
 
 		/*
Signature · 1 line
2.34.1
#2 lore raw

RFC PATCH v2 2/8 migrate: implement migrate_misplaced_folios_batch

In reply to Bharata B Rao

From: Gregory Price <[email protected]>

A common operation in tiering is to migrate multiple pages at once. The migrate_misplaced_folio function requires one call for each individual folio. Expose a batch-variant of the same call for use when doing batch migrations.

---
 include/linux/migrate.h |  6 ++++++
 mm/migrate.c            | 31 +++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
include/linux/migrate.h +6 -0
index acadd41e0b5c..0593f5869be8 100644
@@ -107,6 +107,7 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
+int migrate_misplaced_folios_batch(struct list_head *foliolist, int node);
 #else
 static inline int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -117,6 +118,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
+static inline int migrate_misplaced_folios_batch(struct list_head *foliolist,
+						 int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_MIGRATION
mm/migrate.c +31 -0
index 7e356c0b1b5a..1268a95eda0e 100644
@@ -2714,5 +2714,36 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	BUG_ON(!list_empty(&migratepages));
 	return nr_remaining ? -EAGAIN : 0;
 }
+
+/*
+ * Batch variant of migrate_misplaced_folio. Attempts to migrate
+ * a folio list to the specified destination.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio.
+ *
+ * This function will un-isolate the folios, dereference them, and
+ * remove them from the list before returning.
+ */
+int migrate_misplaced_folios_batch(struct list_head *folio_list, int node)
+{
+	pg_data_t *pgdat = NODE_DATA(node);
+	unsigned int nr_succeeded;
+	int nr_remaining;
+
+	nr_remaining = migrate_pages(folio_list, alloc_misplaced_dst_folio,
+				     NULL, node, MIGRATE_ASYNC,
+				     MR_NUMA_MISPLACED, &nr_succeeded);
+	if (nr_remaining)
+		putback_movable_pages(folio_list);
+
+	if (nr_succeeded) {
+		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded);
+	}
+	BUG_ON(!list_empty(folio_list));
+	return nr_remaining ? -EAGAIN : 0;
+}
 #endif /* CONFIG_NUMA_BALANCING */
 #endif /* CONFIG_NUMA */
Signature · 1 line
2.34.1
#3 lore raw

Re: [RFC PATCH v2 2/8] migrate: implement migrate_misplaced_folios_batch

Quoted 7 lines · On Wed, 10 Sep 2025 20:16:47 +0530 Bharata B Rao <[email protected]> wrote:

From: Gregory Price <[email protected]>

A common operation in tiering is to migrate multiple pages at once. The migrate_misplaced_folio function requires one call for each individual folio. Expose a batch-variant of the same call for use when doing batch migrations.

I probably missed an earlier discussion of this but what does the _batch postfix add over the plural (folios)?

Quoted 36 lines
---
 include/linux/migrate.h |  6 ++++++
 mm/migrate.c            | 31 +++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
include/linux/migrate.h +6 -0
index acadd41e0b5c..0593f5869be8 100644
@@ -107,6 +107,7 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
+int migrate_misplaced_folios_batch(struct list_head *foliolist, int node);
 #else
 static inline int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -117,6 +118,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
+static inline int migrate_misplaced_folios_batch(struct list_head *foliolist,
+						 int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_MIGRATION
mm/migrate.c +0 -0
index 7e356c0b1b5a..1268a95eda0e 100644

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2718

Quoted 5 lines
mm/migrate.c +2 -0
@@ -2714,5 +2714,36 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	BUG_ON(!list_empty(&migratepages));
 	return nr_remaining ? -EAGAIN : 0;
 }
+
+/*

Kernel-doc perhaps appropriate?

Quoted 31 lines
mm/migrate.c +29 -0
+ * Batch variant of migrate_misplaced_folio. Attempts to migrate
+ * a folio list to the specified destination.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio.
+ *
+ * This function will un-isolate the folios, dereference them, and
+ * remove them from the list before returning.
+ */
+int migrate_misplaced_folios_batch(struct list_head *folio_list, int node)
+{
+	pg_data_t *pgdat = NODE_DATA(node);
+	unsigned int nr_succeeded;
+	int nr_remaining;
+
+	nr_remaining = migrate_pages(folio_list, alloc_misplaced_dst_folio,
+				     NULL, node, MIGRATE_ASYNC,
+				     MR_NUMA_MISPLACED, &nr_succeeded);
+	if (nr_remaining)
+		putback_movable_pages(folio_list);
+
+	if (nr_succeeded) {
+		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded);
+	}
+	BUG_ON(!list_empty(folio_list));
+	return nr_remaining ? -EAGAIN : 0;
+}
 #endif /* CONFIG_NUMA_BALANCING */
 #endif /* CONFIG_NUMA */
#4 lore raw

Re: [RFC PATCH v2 2/8] migrate: implement migrate_misplaced_folios_batch

Quoted 5 lines · Jonathan Cameron
Quoted 2 lines · On Wed, 10 Sep 2025 20:16:47 +0530 Bharata B Rao <[email protected]> wrote:

From: Gregory Price <[email protected]>

Quoted 7 lines
Quoted 5 lines

A common operation in tiering is to migrate multiple pages at once. The migrate_misplaced_folio function requires one call for each individual folio. Expose a batch-variant of the same call for use when doing batch migrations.

I probably missed an earlier discussion of this but what does the _batch postfix add over the plural (folios)?

https://lore.kernel.org/linux-mm/[email protected]/

Quoted 31 lines
Quoted 31 lines
---
 include/linux/migrate.h |  6 ++++++
 mm/migrate.c            | 31 +++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+)
include/linux/migrate.h +6 -0
index acadd41e0b5c..0593f5869be8 100644
@@ -107,6 +107,7 @@ static inline int migrate_huge_page_move_mapping(struct address_space *mapping,
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
+int migrate_misplaced_folios_batch(struct list_head *foliolist, int node);
 #else
 static inline int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -117,6 +118,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
+static inline int migrate_misplaced_folios_batch(struct list_head *foliolist,
+						 int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 #endif /* CONFIG_NUMA_BALANCING */
 
Quoted 13 lines

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2718

Quoted 11 lines
include/linux/migrate.h +0 -0
 #ifdef CONFIG_MIGRATION
mm/migrate.c +2 -0
index 7e356c0b1b5a..1268a95eda0e 100644
@@ -2714,5 +2714,36 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	BUG_ON(!list_empty(&migratepages));
 	return nr_remaining ? -EAGAIN : 0;
 }
+
+/*

Kernel-doc perhaps appropriate?

Probably yes, will take care in next iteration.

Thanks for looking into this patchset.

Regards,
Bharata.

#5 lore raw

RFC PATCH v2 3/8 mm: Hot page tracking and promotion

In reply to Bharata B Rao

This introduces a sub-system for collecting memory access information from different sources. It maintains the hotness information based on the access history and time of access.

Additionally, it provides per-lowertier-node kernel threads (named kpromoted) that periodically promote the pages that are eligible for promotion.

Sub-systems that generate hot page access info can report that using this API:

int pghot_record_access(u64 pfn, int nid, int src,
unsigned long time)

@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (sub-system) that generated the
access info
@time: The access time in jiffies

Some temperature sources may not provide the nid from which the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. For such sources, the default toptier node to which such pages should be promoted is hard coded.

Also, the access time provided some sources may at best be considered approximate. This is especially true for hot pages detected by PTE A bit scanning.

The hot PFN records are stored in hash lists hashed by PFN value. The PFN records that are categorized as hot enough to be promoted are maintained in a per-lowertier-node max heap from which kpromoted extracts and promotes them.

---
 include/linux/mmzone.h        |  11 +
 include/linux/pghot.h         |  96 +++++++
 include/linux/vm_event_item.h |   9 +
 mm/Kconfig                    |  11 +
 mm/Makefile                   |   1 +
 mm/mm_init.c                  |  10 +
 mm/pghot.c                    | 524 ++++++++++++++++++++++++++++++++++
 mm/vmstat.c                   |   9 +
 8 files changed, 671 insertions(+)
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot.c
include/linux/mmzone.h +11 -0
index 0c5da9141983..f7094babed10 100644
@@ -1349,6 +1349,10 @@ struct memory_failure_stats {
 };
 #endif
 
+#ifdef CONFIG_PGHOT
+#include <linux/pghot.h>
+#endif
+
 /*
  * On NUMA machines, each NUMA node would have a pg_data_t to describe
  * it's memory layout. On UMA machines there is a single pglist_data which
@@ -1497,6 +1501,13 @@ typedef struct pglist_data {
 #ifdef CONFIG_MEMORY_FAILURE
 	struct memory_failure_stats mf_stats;
 #endif
+#ifdef CONFIG_PGHOT
+	struct task_struct *kpromoted;
+	wait_queue_head_t kpromoted_wait;
+	struct pghot_info **phi_buf;
+	struct max_heap heap;
+	spinlock_t heap_lock;
+#endif
 } pg_data_t;
 
 #define node_present_pages(nid)	(NODE_DATA(nid)->node_present_pages)
include/linux/pghot.h new file +96 -0

96 lines not on this page · raw message

include/linux/vm_event_item.h +9 -0
index 9e15a088ba38..a996fa9df785 100644
@@ -186,6 +186,15 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		KSTACK_REST,
 #endif
 #endif /* CONFIG_DEBUG_STACK_USAGE */
+		PGHOT_RECORDED_ACCESSES,
+		PGHOT_RECORD_HWHINTS,
+		PGHOT_RECORD_PGTSCANS,
+		PGHOT_RECORD_HINTFAULTS,
+		PGHOT_RECORDS_HASH,
+		PGHOT_RECORDS_HEAP,
+		KPROMOTED_RIGHT_NODE,
+		KPROMOTED_NON_LRU,
+		KPROMOTED_DROPPED,
 		NR_VM_EVENT_ITEMS
 };
 
mm/Kconfig +11 -0
index e443fe8cd6cf..8b236eb874cf 100644
@@ -1381,6 +1381,17 @@ config PT_RECLAIM
 
 	  Note: now only empty user PTE page table pages will be reclaimed.
 
+config PGHOT
+	bool "Hot page tracking and promotion"
+	def_bool y
+	depends on NUMA && MIGRATION && MMU
+	select MIN_HEAP
+	help
+	  A sub-system to track page accesses in lower tier memory and
+	  maintain hot page information. Promotes hot pages from lower
+	  tiers to top tier by using the memory access information provided
+	  by various sources. Asynchronous promotion is done by per-node
+	  kernel threads.
 
 source "mm/damon/Kconfig"
 
mm/Makefile +1 -0
index ef54aa615d9d..ecdd5241bea8 100644
@@ -147,3 +147,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o
+obj-$(CONFIG_PGHOT) += pghot.o
mm/mm_init.c +10 -0
index 5c21b3af216b..f7992be3ff7f 100644
@@ -1402,6 +1402,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)
 static void pgdat_init_kcompactd(struct pglist_data *pgdat) {}
 #endif
 
+#ifdef CONFIG_PGHOT
+static void pgdat_init_kpromoted(struct pglist_data *pgdat)
+{
+	init_waitqueue_head(&pgdat->kpromoted_wait);
+}
+#else
+static void pgdat_init_kpromoted(struct pglist_data *pgdat) {}
+#endif
+
 static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 {
 	int i;
@@ -1411,6 +1420,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 
 	pgdat_init_split_queue(pgdat);
 	pgdat_init_kcompactd(pgdat);
+	pgdat_init_kpromoted(pgdat);
 
 	init_waitqueue_head(&pgdat->kswapd_wait);
 	init_waitqueue_head(&pgdat->pfmemalloc_wait);
mm/pghot.c new file +524 -0

524 lines not on this page · raw message

mm/vmstat.c +9 -0
index 71cd1ceba191..ee122c2cd137 100644
@@ -1494,6 +1494,15 @@ const char * const vmstat_text[] = {
 	[I(KSTACK_REST)]			= "kstack_rest",
 #endif
 #endif
+	[I(PGHOT_RECORDED_ACCESSES)]		= "pghot_recorded_accesses",
+	[I(PGHOT_RECORD_HWHINTS)]		= "pghot_recorded_hwhints",
+	[I(PGHOT_RECORD_PGTSCANS)]		= "pghot_recorded_pgtscans",
+	[I(PGHOT_RECORD_HINTFAULTS)]		= "pghot_recorded_hintfaults",
+	[I(PGHOT_RECORDS_HASH)]			= "pghot_records_hash",
+	[I(PGHOT_RECORDS_HEAP)]			= "pghot_records_heap",
+	[I(KPROMOTED_RIGHT_NODE)]		= "kpromoted_right_node",
+	[I(KPROMOTED_NON_LRU)]			= "kpromoted_non_lru",
+	[I(KPROMOTED_DROPPED)]			= "kpromoted_dropped",
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
 };
Signature · 1 line
2.34.1
#6 lore raw

Re: [RFC PATCH v2 3/8] mm: Hot page tracking and promotion

Quoted 30 lines · On Wed, 10 Sep 2025 20:16:48 +0530 Bharata B Rao <[email protected]> wrote:

This introduces a sub-system for collecting memory access information from different sources. It maintains the hotness information based on the access history and time of access.

Additionally, it provides per-lowertier-node kernel threads (named kpromoted) that periodically promote the pages that are eligible for promotion.

Sub-systems that generate hot page access info can report that using this API:

int pghot_record_access(u64 pfn, int nid, int src,
unsigned long time)

@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (sub-system) that generated the
access info
@time: The access time in jiffies

Some temperature sources may not provide the nid from which the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. For such sources, the default toptier node to which such pages should be promoted is hard coded.

Also, the access time provided some sources may at best be considered approximate. This is especially true for hot pages detected by PTE A bit scanning.

Quoted 6 lines

The hot PFN records are stored in hash lists hashed by PFN value. The PFN records that are categorized as hot enough to be promoted are maintained in a per-lowertier-node max heap from which kpromoted extracts and promotes them.

A fairly superficial review only of this. At some point I'll aim to take a closer look at the heap bit.

Quoted 26 lines
---
 include/linux/mmzone.h        |  11 +
 include/linux/pghot.h         |  96 +++++++
 include/linux/vm_event_item.h |   9 +
 mm/Kconfig                    |  11 +
 mm/Makefile                   |   1 +
 mm/mm_init.c                  |  10 +
 mm/pghot.c                    | 524 ++++++++++++++++++++++++++++++++++
 mm/vmstat.c                   |   9 +
 8 files changed, 671 insertions(+)
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot.c
include/linux/mmzone.h +0 -0
index 0c5da9141983..f7094babed10 100644
include/linux/pghot.h new file +2 -0
new file mode 100644
index 000000000000..1443643aab13
 
+
+struct pghot_info {

include/linux/pghot.h line 70

Quoted 10 lines
include/linux/pghot.h new file +10 -0
+	unsigned long pfn;
+
+	/*
+	 * The following three fundamental parameters
+	 * required to track the hotness of page/PFN are
+	 * packed within a single u32.
+	 */
+	u32 frequency:PGHOT_FREQ_BITS; /* Number of accesses within current window */
+	u32 nid:PGHOT_NID_BITS; /* Most recent access from this node */
+	u32 last_update:PGHOT_TIME_BITS; /* Most recent access time */

Add spaces around : I think to help the eye parse those.

include/linux/pghot.h line 80

Quoted 10 lines
include/linux/pghot.h new file +10 -0
+
+	struct hlist_node hnode;
+	size_t heap_idx; /* Position in max heap for quick retreival */
+};
+
+struct max_heap {
+	size_t nr;
+	size_t size;
+	struct pghot_info **data;
+	DECLARE_FLEX_ARRAY(struct pghot_info *, preallocated);

That macro is all about use in unions rather than generally being needed. Do you need that here rather than

	struct pg_hot_info *preallocated[];

Can you add a __counted_by() marking?

Quoted 13 lines
include/linux/pghot.h new file +1 -0
+};
mm/pghot.c new file +3 -0
new file mode 100644
index 000000000000..9f7581818b8f
@@ -0,0 +1,524 @@
 
+
+static struct folio *kpromoted_isolate_folio(struct pghot_info *phi)
+{

mm/pghot.c line 130

Quoted 10 lines
mm/pghot.c new file +10 -0
+	struct page *page = pfn_to_page(phi->pfn);
+	struct folio *folio;
+
+	if (!page)
+		return NULL;
+
+	folio = page_folio(page);
+	if (migrate_misplaced_folio_prepare(folio, NULL, phi->nid))
+		return NULL;
+	else

else not needed.

Quoted 69 lines
mm/pghot.c new file +68 -0
+		return folio;
+}
 
+static int phi_heap_extract(pg_data_t *pgdat, int batch_count, int freq_th,
+			    struct list_head *migrate_list, int *count)
+{
+	spinlock_t *phi_heap_lock = &pgdat->heap_lock;
+	struct max_heap *phi_heap = &pgdat->heap;
+	int max_retries = 10;
+	int bkt, i = 0;
+
+	if (batch_count < 0 || !migrate_list || !count || freq_th < 1 ||
+	    freq_th > KPROMOTED_FREQ_THRESHOLD)
+		return -EINVAL;
+
+	*count = 0;
+	for (i = 0; i < batch_count; i++) {
+		struct pghot_info *top = NULL;
+		bool should_continue = false;
+		struct folio *folio;
+		int retries = 0;
+
+		while (retries < max_retries) {
+			spin_lock(phi_heap_lock);
+			if (phi_heap->nr > 0 && phi_heap->data[0]->frequency >= freq_th) {
+				should_continue = true;
+				bkt = hash_min(phi_heap->data[0]->pfn, phi_hash_order);
+				top = phi_heap->data[0];
+			}
+			spin_unlock(phi_heap_lock);
+
+			if (!should_continue)
+				goto done;
+
+			spin_lock(&phi_hash[bkt].lock);
+			spin_lock(phi_heap_lock);
+			if (phi_heap->nr == 0 || phi_heap->data[0] != top ||
+			    phi_heap->data[0]->frequency < freq_th) {
+				spin_unlock(phi_heap_lock);
+				spin_unlock(&phi_hash[bkt].lock);
+				retries++;
+				continue;
+			}
+
+			top = phi_heap->data[0];
+			hlist_del_init(&top->hnode);
+
+			phi_heap->nr--;
+			if (phi_heap->nr > 0) {
+				phi_heap->data[0] = phi_heap->data[phi_heap->nr];
+				phi_heap->data[0]->heap_idx = 0;
+				min_heap_sift_down(phi_heap, 0, &phi_heap_cb,
+						   phi_heap->data);
+			}
+
+			spin_unlock(phi_heap_lock);
+			spin_unlock(&phi_hash[bkt].lock);
+
+			if (!phi_is_pfn_hot(top)) {
+				count_vm_event(KPROMOTED_DROPPED);
+				goto skip;
+			}
+
+			folio = kpromoted_isolate_folio(top);
+			if (folio) {
+				list_add(&folio->lru, migrate_list);
+				(*count)++;
+			}
+skip:

mm/pghot.c line 227

Quoted 10 lines
mm/pghot.c new file +10 -0
+			phi_free(top);
+			break;
+		}
+		if (retries >= max_retries) {
+			pr_warn("%s: Too many retries\n", __func__);
+			break;
+		}
+
+	}
+done:

If that is all there is, I'd use an early return as tends to give simpler code.

mm/pghot.c line 236

Quoted 9 lines
mm/pghot.c new file +9 -0
+	return 0;
+}
+
+static void phi_heap_add_or_adjust(struct pghot_info *phi)
+{
+	pg_data_t *pgdat = NODE_DATA(phi->nid);
+	struct max_heap *phi_heap = &pgdat->heap;
+
+	spin_lock(&pgdat->heap_lock);

guard() perhaps.

Quoted 86 lines
mm/pghot.c new file +85 -0
+	if (phi->heap_idx >= 0 && phi->heap_idx < phi_heap->nr &&
+	    phi_heap->data[phi->heap_idx] == phi) {
+		/* Entry exists in heap */
+		if (phi->frequency < KPROMOTED_FREQ_THRESHOLD) {
+			/* Below threshold, remove from the heap */
+			phi_heap->nr--;
+			if (phi->heap_idx < phi_heap->nr) {
+				phi_heap->data[phi->heap_idx] =
+					phi_heap->data[phi_heap->nr];
+				phi_heap->data[phi->heap_idx]->heap_idx =
+					phi->heap_idx;
+				min_heap_sift_down(phi_heap, phi->heap_idx,
+						   &phi_heap_cb, phi_heap->data);
+			}
+			phi->heap_idx = -1;
+
+		} else {
+			/* Update position in heap */
+			phi_heap_update_entry(phi_heap, phi);
+		}
+	} else if (phi->frequency >= KPROMOTED_FREQ_THRESHOLD) {
+		/*
+		 * Add to the heap. If heap is full we will have
+		 * to wait for the next access reporting to elevate
+		 * it to heap.
+		 */
+		if (phi_heap_insert(phi_heap, phi))
+			count_vm_event(PGHOT_RECORDS_HEAP);
+	}
+	spin_unlock(&pgdat->heap_lock);
+}
 
+
+/*
+ * Called by subsystems that generate page hotness/access information.
+ *
+ *  @pfn: The PFN of the memory accessed
+ *  @nid: The accessing NUMA node ID
+ *  @src: The temperature source (sub-system) that generated the
+ *        access info
+ *  @time: The access time in jiffies
+ *
+ * Maintains the access records per PFN, classifies them as
+ * hot based on subsequent accesses and finally hands over
+ * them to kpromoted for migration.
+ */
+int pghot_record_access(u64 pfn, int nid, int src, unsigned long now)
+{
+	struct pghot_info *phi;
+	struct page *page;
+	struct folio *folio;
+	int bkt;
+	bool new_entry = false, new_window = false;
+	u32 cur_time = now & PGHOT_TIME_MASK;
+
+	if (!kpromoted_started)
+		return -EINVAL;
+
+	if (nid >= PGHOT_NID_MAX)
+		return -EINVAL;
+
+	count_vm_event(PGHOT_RECORDED_ACCESSES);
+
+	switch (src) {
+	case PGHOT_HW_HINTS:
+		count_vm_event(PGHOT_RECORD_HWHINTS);
+		break;
+	case PGHOT_PGTABLE_SCAN:
+		count_vm_event(PGHOT_RECORD_PGTSCANS);
+		break;
+	case PGHOT_HINT_FAULT:
+		count_vm_event(PGHOT_RECORD_HINTFAULTS);
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	/*
+	 * Record only accesses from lower tiers.
+	 */
+	if (node_is_toptier(pfn_to_nid(pfn)))
+		return 0;
+
+	/*
+	 * Reject the non-migratable pages right away.
+	 */

mm/pghot.c line 342

Quoted 10 lines
mm/pghot.c new file +10 -0
+	page = pfn_to_online_page(pfn);
+	if (!page || is_zone_device_page(page))
+		return 0;
+
+	folio = page_folio(page);
+	if (!folio_test_lru(folio))
+		return 0;
+
+	bkt = hash_min(pfn, phi_hash_order);
+	spin_lock(&phi_hash[bkt].lock);

If this doesn't get more complex later, use guard() and early returns on error.

mm/pghot.c line 347

Quoted 5 lines
mm/pghot.c new file +5 -0
+	phi = phi_lookup(pfn, bkt);
+	if (!phi) {
+		phi = phi_alloc(pfn);
+		if (!phi)
+			goto out;

Not an error? Add a comment on why not perhaps.

Quoted 25 lines
mm/pghot.c new file +25 -0
+		new_entry = true;
+	}
+
+	/*
+	 * If the previous access was beyond the threshold window
+	 * start frequency tracking afresh.
+	 */
+	if (((cur_time - phi->last_update) > msecs_to_jiffies(sysctl_pghot_freq_window)) ||
+	    (nid != NUMA_NO_NODE && phi->nid != nid))
+		new_window = true;
+
+	if (new_entry || new_window) {
+		/* New window */
+		phi->frequency = 1; /* TODO: Factor in the history */
+	} else if (phi->frequency < PGHOT_FREQ_MAX)
+		phi->frequency++;
+	phi->last_update = cur_time;
+	phi->nid = (nid == NUMA_NO_NODE) ? KPROMOTED_DEFAULT_NODE : nid;
+
+	if (new_entry) {
+		/* Insert the new entry into hash table */
+		hlist_add_head(&phi->hnode, &phi_hash[bkt].hash);
+		count_vm_event(PGHOT_RECORDS_HASH);
+	} else {
+		/* Add/update the position in heap */

mm/pghot.c line 382

Quoted 10 lines
mm/pghot.c new file +10 -0
+		phi_heap_add_or_adjust(phi);
+	}
+out:
+	spin_unlock(&phi_hash[bkt].lock);
+	return 0;
+}
+
+/*
+ * Extract the hot page records and batch-migrate the
+ * hot pages.

Wrap comments to 80 chars.

Quoted 18 lines

+ */

mm/pghot.c new file +17 -0
+static void kpromoted_migrate(pg_data_t *pgdat)
+{
+	int count, ret;
+	LIST_HEAD(migrate_list);
+
+	/*
+	 * Extract the top N elements from the heap that match
+	 * the requested hotness threshold.
+	 *
+	 * PFNs ineligible from migration standpoint are removed
+	 * from the heap and hash.
+	 *
+	 * Folios eligible for migration are isolated and returned
+	 * in @migrate_list.
+	 */
+	ret = phi_heap_extract(pgdat, KPROMOTED_MIGRATE_BATCH,
+			       KPROMOTED_FREQ_THRESHOLD, &migrate_list, &count);

mm/pghot.c line 410

Quoted 10 lines
mm/pghot.c new file +10 -0
+	if (ret)
+		return;
+
+	if (!list_empty(&migrate_list))
+		migrate_misplaced_folios_batch(&migrate_list, pgdat->node_id);
+}
+
+static int kpromoted(void *p)
+{
+	pg_data_t *pgdat = (pg_data_t *)p;

Cast not needed.

	pg_data_t *pgdat = p;
Quoted 10 lines
mm/pghot.c new file +10 -0
+
+	while (!kthread_should_stop()) {
+		wait_event_timeout(pgdat->kpromoted_wait, false,
+				   msecs_to_jiffies(KPROMOTE_DELAY));
+		kpromoted_migrate(pgdat);
+	}
+	return 0;
+}
+
+static int kpromoted_run(int nid)

mm/pghot.c line 430

Quoted 10 lines
mm/pghot.c new file +10 -0
+{
+	pg_data_t *pgdat = NODE_DATA(nid);
+	int ret = 0;
+
+	if (!node_is_toptier(nid))
+		return 0;
+
+	if (!pgdat->phi_buf) {
+		pgdat->phi_buf = vzalloc_node(phi_heap_entries * sizeof(struct pghot_info *),
+					      nid);

I'd use sizeof(*pgdat->phi_buf) here to avoid need to check types match when reading the code. Sadly there isn't a vcalloc_node().

Quoted 4 lines
mm/pghot.c new file +4 -0
+		if (!pgdat->phi_buf)
+			return -ENOMEM;
+
+		min_heap_init(&pgdat->heap, pgdat->phi_buf, phi_heap_entries);

mm/pghot.c line 444

Quoted 10 lines
mm/pghot.c new file +10 -0
+		spin_lock_init(&pgdat->heap_lock);
+	}
+
+	if (!pgdat->kpromoted)
+		pgdat->kpromoted = kthread_create_on_node(kpromoted, pgdat, nid,
+							  "kpromoted%d", nid);
+	if (IS_ERR(pgdat->kpromoted)) {
+		ret = PTR_ERR(pgdat->kpromoted);
+		pgdat->kpromoted = NULL;
+		pr_info("Failed to start kpromoted%d, ret %d\n", nid, ret);

Unless there is going to be more in later patches that prevents it. Just return here.

mm/pghot.c line 448

Quoted 4 lines
mm/pghot.c new file +4 -0
+	} else {
+		wake_up_process(pgdat->kpromoted);
+	}
+	return ret;

return 0; //after change suggested above.

Quoted 6 lines
mm/pghot.c new file +6 -0
+}
+
+/*
+ * TODO: Handle cleanup during node offline.
+ */
+static int __init pghot_init(void)

mm/pghot.c line 464

Quoted 10 lines
mm/pghot.c new file +10 -0
+{
+	unsigned int hash_size;
+	size_t hash_entries;
+	size_t nr_pages = 0;
+	pg_data_t *pgdat;
+	int i, nid, ret;
+
+	/*
+	 * Arrive at the hash and heap sizes based on the
+	 * number of pages present in the lower tier nodes.

Trivial: Wrap closer to 80 chars.

Quoted 4 lines
mm/pghot.c new file +4 -0
+	 */
+	for_each_node_state(nid, N_MEMORY) {
+		if (!node_is_toptier(nid))
+			nr_pages += NODE_DATA(nid)->node_present_pages;

mm/pghot.c line 478

Quoted 10 lines
mm/pghot.c new file +10 -0
+	}
+
+	if (!nr_pages)
+		return 0;
+
+	hash_entries = nr_pages * PGHOT_HASH_PCT / 100;
+	hash_size = hash_entries / PGHOT_HASH_ENTRIES;
+	phi_hash_order = ilog2(hash_size);
+
+	phi_hash = vmalloc(sizeof(struct pghot_hash) * hash_size);

Prefer sizeof(*phy_hash) so I don't need to check types match :)

vcalloc() probably appropriate here.

mm/pghot.c line 481

Quoted 3 lines
mm/pghot.c new file +3 -0
+	if (!phi_hash) {
+		ret = -ENOMEM;
+		goto out;

Out label isn't clearly an 'error' which is a little confusing.

mm/pghot.c line 492

Quoted 11 lines
mm/pghot.c new file +11 -0
+	}
+
+	for (i = 0; i < hash_size; i++) {
+		INIT_HLIST_HEAD(&phi_hash[i].hash);
+		spin_lock_init(&phi_hash[i].lock);
+	}
+
+	phi_cache = KMEM_CACHE(pghot_info, 0);
+	if (unlikely(!phi_cache)) {
+		ret = -ENOMEM;
+		goto out;

Whilst not strictly necessary I'd add multiple labels so only things that have been allocated are free rather than relying on them being NULL otherwise. Whilst not a correctness thing it makes it slightly easier to check tear down paths are correct.

Quoted 32 lines
mm/pghot.c new file +32 -0
+	}
+
+	phi_heap_entries = hash_entries * PGHOT_HEAP_PCT / 100;
+	for_each_node_state(nid, N_CPU) {
+		ret = kpromoted_run(nid);
+		if (ret)
+			goto out_stop_kthread;
+	}
+
+	register_sysctl_init("vm", pghot_sysctls);
+	kpromoted_started = true;
+	pr_info("pghot: Started page hotness monitoring and promotion thread\n");
+	pr_info("pghot: nr_pages %ld hash_size %d hash_entries %ld hash_order %d heap_entries %d\n",
+	       nr_pages, hash_size, hash_entries, phi_hash_order, phi_heap_entries);
+	return 0;
+
+out_stop_kthread:
+	for_each_node_state(nid, N_CPU) {
+		pgdat = NODE_DATA(nid);
+		if (pgdat->kpromoted) {
+			kthread_stop(pgdat->kpromoted);
+			pgdat->kpromoted = NULL;
+			vfree(pgdat->phi_buf);
+		}
+	}
+out:
+	kmem_cache_destroy(phi_cache);
+	vfree(phi_hash);
+	return ret;
+}
+
+late_initcall(pghot_init)
#7 lore raw

Re: [RFC PATCH v2 3/8] mm: Hot page tracking and promotion

Quoted 11 lines · Jonathan Cameron

include/linux/pghot.h line 80

Quoted 6 lines · On Wed, 10 Sep 2025 20:16:48 +0530 Bharata B Rao <[email protected]> wrote:
include/linux/pghot.h new file +6 -0
+
+struct max_heap {
+	size_t nr;
+	size_t size;
+	struct pghot_info **data;
+	DECLARE_FLEX_ARRAY(struct pghot_info *, preallocated);

That macro is all about use in unions rather than generally being needed. Do you need that here rather than

Quoted 3 lines
	struct pg_hot_info *preallocated[];

Can you add a __counted_by() marking?

I was using DEFINE_MIN_HEAP macro earlier which gave problems when I had to define per-node instance of the heap with the same name. The workaround for that resulted in the use of above flex array.

Not needed, I will revert back to using array of pointers with __counted_by() marking.

Quoted 6 lines
Quoted 6 lines
mm/pghot.c new file +6 -0
+
+	for (i = 0; i < hash_size; i++) {
+		INIT_HLIST_HEAD(&phi_hash[i].hash);
+		spin_lock_init(&phi_hash[i].lock);
+	}
+
Quoted 8 lines

mm/pghot.c line 492

Quoted 4 lines
mm/pghot.c new file +4 -0
+	phi_cache = KMEM_CACHE(pghot_info, 0);
+	if (unlikely(!phi_cache)) {
+		ret = -ENOMEM;
+		goto out;

Whilst not strictly necessary I'd add multiple labels so only things that have been allocated are free rather than relying on them being NULL otherwise. Whilst not a correctness thing it makes it slightly easier to check tear down paths are correct.

In general I agree but for freeing with a loop exit, the current method appeared much simpler.

I will take care of rest of the review comments.

Regards,
Bharata.

#8 lore raw

RFC PATCH v2 4/8 x86: ibs: In-kernel IBS driver for memory access profiling

In reply to Bharata B Rao

Use IBS (Instruction Based Sampling) feature present in AMD processors for memory access tracking. The access information obtained from IBS via NMI is fed to kpromoted daemon for futher action.

In addition to many other information related to the memory access, IBS provides physical (and virtual) address of the access and indicates if the access came from slower tier. Only memory accesses originating from slower tiers are further acted upon by this driver.

The samples are initially accumulated in percpu buffers which are flushed to pghot hot page tracking mechanism using irq_work.

TODO: Many counters are added to vmstat just as debugging aid for now.

About IBS
---------
IBS can be programmed to provide data about instruction execution periodically. This is done by programming a desired sample count (number of ops) in a control register. When the programmed number of ops are dispatched, a micro-op gets tagged, various information about the tagged micro-op's execution is populated in IBS execution MSRs and an interrupt is raised. While IBS provides a lot of data for each sample, for the purpose of memory access profiling, we are interested in linear and physical address of the memory access that reached DRAM. Recent AMD processors provide further filtering where it is possible to limit the sampling to those ops that had an L3 miss which greately reduces the non-useful samples.

While IBS provides capability to sample instruction fetch and execution, only IBS execution sampling is used here to collect data about memory accesses that occur during the instruction execution.

More information about IBS is available in Sec 13.3 of AMD64 Architecture Programmer's Manual, Volume 2:System Programming which is present at:
https://bugzilla.kernel.org/attachment.cgi?id=288923

Information about MSRs used for programming IBS can be found in Sec 2.1.14.4 of PPR Vol 1 for AMD Family 19h Model 11h B1 which is currently present at:
https://www.amd.com/system/files/TechDocs/55901_0.25.zip

---
 arch/x86/events/amd/ibs.c        |  11 ++
 arch/x86/include/asm/ibs.h       |   7 +
 arch/x86/include/asm/msr-index.h |  16 ++
 arch/x86/mm/Makefile             |   3 +-
 arch/x86/mm/ibs.c                | 311 +++++++++++++++++++++++++++++++
 include/linux/vm_event_item.h    |  17 ++
 mm/vmstat.c                      |  17 ++
 7 files changed, 381 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/ibs.h
 create mode 100644 arch/x86/mm/ibs.c
arch/x86/events/amd/ibs.c +11 -0
index 112f43b23ebf..1498dc9caeb2 100644
@@ -13,9 +13,11 @@
 #include <linux/ptrace.h>
 #include <linux/syscore_ops.h>
 #include <linux/sched/clock.h>
+#include <linux/pghot.h>
 
 #include <asm/apic.h>
 #include <asm/msr.h>
+#include <asm/ibs.h>
 
 #include "../perf_event.h"
 
@@ -1756,6 +1758,15 @@ static __init int amd_ibs_init(void)
 {
 	u32 caps;
 
+	/*
+	 * TODO: Find a clean way to disable perf IBS so that IBS
+	 * can be used for memory access profiling.
+	 */
+	if (arch_hw_access_profiling) {
+		pr_info("IBS isn't available for perf use\n");
+		return 0;
+	}
+
 	caps = __get_ibs_caps();
 	if (!caps)
 		return -ENODEV;	/* ibs not supported by the cpu */
arch/x86/include/asm/ibs.h new file +7 -0
new file mode 100644
index 000000000000..b5a4f2ca6330
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_IBS_H
+#define _ASM_X86_IBS_H
+
+extern bool arch_hw_access_profiling;
+
+#endif /* _ASM_X86_IBS_H */
arch/x86/include/asm/msr-index.h +16 -0
index b65c3ba5fa14..55d26380550c 100644
@@ -742,6 +742,22 @@
 /* AMD Last Branch Record MSRs */
 #define MSR_AMD64_LBR_SELECT			0xc000010e
 
+/* AMD IBS MSR bits */
+#define MSR_AMD64_IBSOPDATA2_DATASRC			0x7
+#define MSR_AMD64_IBSOPDATA2_DATASRC_LCL_CACHE		0x1
+#define MSR_AMD64_IBSOPDATA2_DATASRC_PEER_CACHE_NEAR	0x2
+#define MSR_AMD64_IBSOPDATA2_DATASRC_DRAM		0x3
+#define MSR_AMD64_IBSOPDATA2_DATASRC_FAR_CCX_CACHE	0x5
+#define MSR_AMD64_IBSOPDATA2_DATASRC_EXT_MEM		0x8
+#define	MSR_AMD64_IBSOPDATA2_RMTNODE			0x10
+
+#define MSR_AMD64_IBSOPDATA3_LDOP		BIT_ULL(0)
+#define MSR_AMD64_IBSOPDATA3_STOP		BIT_ULL(1)
+#define MSR_AMD64_IBSOPDATA3_DCMISS		BIT_ULL(7)
+#define MSR_AMD64_IBSOPDATA3_LADDR_VALID	BIT_ULL(17)
+#define MSR_AMD64_IBSOPDATA3_PADDR_VALID	BIT_ULL(18)
+#define MSR_AMD64_IBSOPDATA3_L2MISS		BIT_ULL(20)
+
 /* Zen4 */
 #define MSR_ZEN4_BP_CFG                 0xc001102e
 #define MSR_ZEN4_BP_CFG_BP_SPEC_REDUCE_BIT 4
arch/x86/mm/Makefile +2 -1
index 5b9908f13dcf..967e5af9eba9 100644
@@ -22,7 +22,8 @@ CFLAGS_REMOVE_pgprot.o			= -pg
 endif
 
 obj-y				:=  init.o init_$(BITS).o fault.o ioremap.o extable.o mmap.o \
-				    pgtable.o physaddr.o tlb.o cpu_entry_area.o maccess.o pgprot.o
+				    pgtable.o physaddr.o tlb.o cpu_entry_area.o maccess.o pgprot.o \
+				    ibs.o
 
 obj-y				+= pat/
 
arch/x86/mm/ibs.c new file +311 -0

311 lines not on this page · raw message

include/linux/vm_event_item.h +17 -0
index a996fa9df785..bca57b05766d 100644
@@ -195,6 +195,23 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		KPROMOTED_RIGHT_NODE,
 		KPROMOTED_NON_LRU,
 		KPROMOTED_DROPPED,
+		HWHINT_NR_EVENTS,
+		HWHINT_KERNEL,
+		HWHINT_KTHREAD,
+		HWHINT_NON_LOAD_STORES,
+		HWHINT_DC_L2_HITS,
+		HWHINT_LOCAL_L3L1L2,
+		HWHINT_LOCAL_PEER_CACHE_NEAR,
+		HWHINT_FAR_CACHE_HITS,
+		HWHINT_DRAM_ACCESSES,
+		HWHINT_CXL_ACCESSES,
+		HWHINT_REMOTE_NODE,
+		HWHINT_LADDR_INVALID,
+		HWHINT_KERNEL_ADDR,
+		HWHINT_PADDR_INVALID,
+		HWHINT_NON_LRU,
+		HWHINT_BUFFER_FULL,
+		HWHINT_USEFUL_SAMPLES,
 		NR_VM_EVENT_ITEMS
 };
 
mm/vmstat.c +17 -0
index ee122c2cd137..aa743708c79b 100644
@@ -1503,6 +1503,23 @@ const char * const vmstat_text[] = {
 	[I(KPROMOTED_RIGHT_NODE)]		= "kpromoted_right_node",
 	[I(KPROMOTED_NON_LRU)]			= "kpromoted_non_lru",
 	[I(KPROMOTED_DROPPED)]			= "kpromoted_dropped",
+	[I(HWHINT_NR_EVENTS)]			= "hwhint_nr_events",
+	[I(HWHINT_KERNEL)]			= "hwhint_kernel",
+	[I(HWHINT_KTHREAD)]			= "hwhint_kthread",
+	[I(HWHINT_NON_LOAD_STORES)]		= "hwhint_non_load_stores",
+	[I(HWHINT_DC_L2_HITS)]			= "hwhint_dc_l2_hits",
+	[I(HWHINT_LOCAL_L3L1L2)]		= "hwhint_local_l3l1l2",
+	[I(HWHINT_LOCAL_PEER_CACHE_NEAR)]	= "hwhint_local_peer_cache_near",
+	[I(HWHINT_FAR_CACHE_HITS)]		= "hwhint_far_cache_hits",
+	[I(HWHINT_DRAM_ACCESSES)]		= "hwhint_dram_accesses",
+	[I(HWHINT_CXL_ACCESSES)]		= "hwhint_cxl_accesses",
+	[I(HWHINT_REMOTE_NODE)]			= "hwhint_remote_node",
+	[I(HWHINT_LADDR_INVALID)]		= "hwhint_invalid_laddr",
+	[I(HWHINT_KERNEL_ADDR)]			= "hwhint_kernel_addr",
+	[I(HWHINT_PADDR_INVALID)]		= "hwhint_invalid_paddr",
+	[I(HWHINT_NON_LRU)]			= "hwhint_non_lru",
+	[I(HWHINT_BUFFER_FULL)]			= "hwhint_buffer_full",
+	[I(HWHINT_USEFUL_SAMPLES)]		= "hwhint_useful_samples",
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
 };
Signature · 1 line
2.34.1
#9 lore raw

Re: [RFC PATCH v2 4/8] x86: ibs: In-kernel IBS driver for memory access profiling

Quoted 77 lines · On Wed, 10 Sep 2025 20:16:49 +0530 Bharata B Rao <[email protected]> wrote:

Use IBS (Instruction Based Sampling) feature present in AMD processors for memory access tracking. The access information obtained from IBS via NMI is fed to kpromoted daemon for futher action.

In addition to many other information related to the memory access, IBS provides physical (and virtual) address of the access and indicates if the access came from slower tier. Only memory accesses originating from slower tiers are further acted upon by this driver.

The samples are initially accumulated in percpu buffers which are flushed to pghot hot page tracking mechanism using irq_work.

TODO: Many counters are added to vmstat just as debugging aid for now.

About IBS
---------
IBS can be programmed to provide data about instruction execution periodically. This is done by programming a desired sample count (number of ops) in a control register. When the programmed number of ops are dispatched, a micro-op gets tagged, various information about the tagged micro-op's execution is populated in IBS execution MSRs and an interrupt is raised. While IBS provides a lot of data for each sample, for the purpose of memory access profiling, we are interested in linear and physical address of the memory access that reached DRAM. Recent AMD processors provide further filtering where it is possible to limit the sampling to those ops that had an L3 miss which greately reduces the non-useful samples.

While IBS provides capability to sample instruction fetch and execution, only IBS execution sampling is used here to collect data about memory accesses that occur during the instruction execution.

More information about IBS is available in Sec 13.3 of AMD64 Architecture Programmer's Manual, Volume 2:System Programming which is present at:
https://bugzilla.kernel.org/attachment.cgi?id=288923

Information about MSRs used for programming IBS can be found in Sec 2.1.14.4 of PPR Vol 1 for AMD Family 19h Model 11h B1 which is currently present at:
https://www.amd.com/system/files/TechDocs/55901_0.25.zip

---
 arch/x86/events/amd/ibs.c        |  11 ++
 arch/x86/include/asm/ibs.h       |   7 +
 arch/x86/include/asm/msr-index.h |  16 ++
 arch/x86/mm/Makefile             |   3 +-
 arch/x86/mm/ibs.c                | 311 +++++++++++++++++++++++++++++++
 include/linux/vm_event_item.h    |  17 ++
 mm/vmstat.c                      |  17 ++
 7 files changed, 381 insertions(+), 1 deletion(-)
 create mode 100644 arch/x86/include/asm/ibs.h
 create mode 100644 arch/x86/mm/ibs.c
arch/x86/events/amd/ibs.c +2 -0
index 112f43b23ebf..1498dc9caeb2 100644
@@ -13,9 +13,11 @@
 #include <linux/ptrace.h>
 #include <linux/syscore_ops.h>
 #include <linux/sched/clock.h>
+#include <linux/pghot.h>
 
 #include <asm/apic.h>
 #include <asm/msr.h>
+#include <asm/ibs.h>
 
 #include "../perf_event.h"
 

arch/x86/events/amd/ibs.c static __init int amd_ibs_init(void) · line 1763

Quoted 6 lines
arch/x86/events/amd/ibs.c +3 -0
@@ -1756,6 +1758,15 @@ static __init int amd_ibs_init(void)
 {
 	u32 caps;
 
+	/*
+	 * TODO: Find a clean way to disable perf IBS so that IBS
+	 * can be used for memory access profiling.

Agreed on this being a key thing. This applies to quite a few other sources of data so finding a generally acceptable solution to this would be great. Davidlohr mentioned on the CXL sync that he has something tackling this for the CHMU driver around this.

Quoted 16 lines
arch/x86/events/amd/ibs.c +6 -0
+	 */
+	if (arch_hw_access_profiling) {
+		pr_info("IBS isn't available for perf use\n");
+		return 0;
+	}
+
 	caps = __get_ibs_caps();
 	if (!caps)
 		return -ENODEV;	/* ibs not supported by the cpu */
arch/x86/mm/ibs.c new file +0 -0
new file mode 100644
index 000000000000..6669710dd35b
@@ -0,0 +1,311 @@

...

arch/x86/mm/ibs.c line 76

Quoted 11 lines
arch/x86/mm/ibs.c new file +11 -0
+
+static int ibs_pop_sample(struct ibs_sample *s)
+{
+	struct ibs_sample_pcpu *ibs_pcpu = raw_cpu_ptr(ibs_s);
+
+	int next = ibs_pcpu->tail + 1;
+
+	if (ibs_pcpu->head == ibs_pcpu->tail)
+		return 0;
+
+	if (next >= IBS_NR_SAMPLES)

== seems more appropriate to me. If it's > then something went wrong and we lost data.

Quoted 11 lines
arch/x86/mm/ibs.c new file +9 -0
+		next = 0;
+
+	*s = ibs_pcpu->samples[ibs_pcpu->tail];
+	ibs_pcpu->tail = next;
+	return 1;
+}
 
 
+static void setup_APIC_ibs(void)
+{
+	int offset;

arch/x86/mm/ibs.c line 256

Quoted 10 lines
arch/x86/mm/ibs.c new file +10 -0
+
+	offset = get_ibs_lvt_offset();
+	if (offset < 0)
+		goto failed;
+
+	if (!setup_APIC_eilvt(offset, 0, APIC_EILVT_MSG_NMI, 0))
+		return;
+failed:
+	pr_warn("IBS APIC setup failed on cpu #%d\n",
+		smp_processor_id());

Unless this is going to get more complex, move that up to the if () block above and return directly there.

arch/x86/mm/ibs.c line 287

Quoted 10 lines
arch/x86/mm/ibs.c new file +9 -0
+}
 
+static int __init ibs_access_profiling_init(void)
+{
+	if (!boot_cpu_has(X86_FEATURE_IBS)) {
+		pr_info("IBS capability is unavailable for access profiling\n");
+		return 0;
+	}
+
+	ibs_s = alloc_percpu_gfp(struct ibs_sample_pcpu, GFP_KERNEL | __GFP_ZERO);

sizeof(*ibs_s).
Same as in other cases. It's nice to avoid having to check types when reviewing code.

arch/x86/mm/ibs.c line 298

Quoted 11 lines
arch/x86/mm/ibs.c new file +11 -0
+	if (!ibs_s)
+		return 0;
+
+	INIT_WORK(&ibs_work, ibs_work_handler);
+	init_irq_work(&ibs_irq_work, ibs_irq_handler);
+
+	/* Uses IBS Op sampling */
+	ibs_config = IBS_OP_CNT_CTL | IBS_OP_ENABLE;
+	ibs_caps = cpuid_eax(IBS_CPUID_FEATURES);
+	if (ibs_caps & IBS_CAPS_ZEN4)
+		ibs_config |= IBS_OP_L3MISSONLY;

ibs_config seems to only be used locally so the global seems unnecessary. You'll need to pass it in to the one user in the next patch though.

Quoted 13 lines
arch/x86/mm/ibs.c new file +13 -0
+
+	register_nmi_handler(NMI_LOCAL, ibs_overflow_handler, 0, "ibs");
+
+	cpuhp_setup_state(CPUHP_AP_PERF_X86_AMD_IBS_STARTING,
+			  "x86/amd/ibs_access_profile:starting",
+			  x86_amd_ibs_access_profile_startup,
+			  x86_amd_ibs_access_profile_teardown);
+
+	pr_info("IBS setup for memory access profiling\n");
+	return 0;
+}
+
+arch_initcall(ibs_access_profiling_init);
#10 lore raw

Re: [RFC PATCH v2 4/8] x86: ibs: In-kernel IBS driver for memory access profiling

Quoted 7 lines · Jonathan Cameron
Quoted 5 lines · On Wed, 10 Sep 2025 20:16:49 +0530 Bharata B Rao <[email protected]> wrote:
arch/x86/events/amd/ibs.c +0 -0
@@ -1756,6 +1758,15 @@ static __init int amd_ibs_init(void)
 {
 	u32 caps;
 
Quoted 8 lines

arch/x86/events/amd/ibs.c static __init int amd_ibs_init(void) · line 1763

Quoted 3 lines
arch/x86/events/amd/ibs.c +3 -0
+	/*
+	 * TODO: Find a clean way to disable perf IBS so that IBS
+	 * can be used for memory access profiling.

Agreed on this being a key thing. This applies to quite a few other sources of data so finding a generally acceptable solution to this would be great. Davidlohr mentioned on the CXL sync that he has something tackling this for the CHMU driver around this.

Okay, will wait to check that.

Quoted 21 lines
Quoted 18 lines
arch/x86/events/amd/ibs.c +6 -0
+	 */
+	if (arch_hw_access_profiling) {
+		pr_info("IBS isn't available for perf use\n");
+		return 0;
+	}
+
 	caps = __get_ibs_caps();
 	if (!caps)
 		return -ENODEV;	/* ibs not supported by the cpu */
arch/x86/mm/ibs.c new file +0 -0
new file mode 100644
index 000000000000..6669710dd35b
@@ -0,0 +1,311 @@

...

Quoted 14 lines

arch/x86/mm/ibs.c line 76

Quoted 11 lines
arch/x86/mm/ibs.c new file +11 -0
+
+static int ibs_pop_sample(struct ibs_sample *s)
+{
+	struct ibs_sample_pcpu *ibs_pcpu = raw_cpu_ptr(ibs_s);
+
+	int next = ibs_pcpu->tail + 1;
+
+	if (ibs_pcpu->head == ibs_pcpu->tail)
+		return 0;
+
+	if (next >= IBS_NR_SAMPLES)

== seems more appropriate to me. If it's > then something went wrong and we lost data.

Makes sense, will try that.

Quoted 9 lines
Quoted 7 lines
arch/x86/mm/ibs.c new file +6 -0
+		next = 0;
+
+	*s = ibs_pcpu->samples[ibs_pcpu->tail];
+	ibs_pcpu->tail = next;
+	return 1;
+}
Quoted 16 lines

arch/x86/mm/ibs.c line 256

Quoted 13 lines
arch/x86/mm/ibs.c new file +13 -0
+static void setup_APIC_ibs(void)
+{
+	int offset;
+
+	offset = get_ibs_lvt_offset();
+	if (offset < 0)
+		goto failed;
+
+	if (!setup_APIC_eilvt(offset, 0, APIC_EILVT_MSG_NMI, 0))
+		return;
+failed:
+	pr_warn("IBS APIC setup failed on cpu #%d\n",
+		smp_processor_id());

Unless this is going to get more complex, move that up to the if () block above and return directly there.

I want to print a warning for both the cases: when LVT offset couldn't be obtained and also when LVT entry couldn't be setup.

Regards,
Bharata.

#11 lore raw

RFC PATCH v2 5/8 x86: ibs: Enable IBS profiling for memory accesses

In reply to Bharata B Rao

Enable IBS memory access data collection for user memory accesses by programming the required MSRs. The profiling is turned ON only for user mode execution and turned OFF for kernel mode execution. Profiling is explicitly disabled for NMI handler too.

TODOs:

- IBS sampling rate is kept fixed for now.
- Arch/vendor separation/isolation of the code needs relook.

---
 arch/x86/include/asm/entry-common.h |  3 +++
 arch/x86/include/asm/hardirq.h      |  2 ++
 arch/x86/include/asm/ibs.h          |  2 ++
 arch/x86/mm/ibs.c                   | 32 +++++++++++++++++++++++++++++
 4 files changed, 39 insertions(+)
arch/x86/include/asm/entry-common.h +3 -0
index d535a97c7284..7144b57d209b 100644
@@ -9,10 +9,12 @@
 #include <asm/io_bitmap.h>
 #include <asm/fpu/api.h>
 #include <asm/fred.h>
+#include <asm/ibs.h>
 
 /* Check that the stack and regs on entry from user mode are sane. */
 static __always_inline void arch_enter_from_user_mode(struct pt_regs *regs)
 {
+	hw_access_profiling_stop();
 	if (IS_ENABLED(CONFIG_DEBUG_ENTRY)) {
 		/*
 		 * Make sure that the entry code gave us a sensible EFLAGS
@@ -99,6 +101,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs,
 static __always_inline void arch_exit_to_user_mode(void)
 {
 	amd_clear_divider();
+	hw_access_profiling_start();
 }
 #define arch_exit_to_user_mode arch_exit_to_user_mode
 
arch/x86/include/asm/hardirq.h +2 -0
index f00c09ffe6a9..0752cb6ebd7a 100644
@@ -91,4 +91,6 @@ static __always_inline bool kvm_get_cpu_l1tf_flush_l1d(void)
 static __always_inline void kvm_set_cpu_l1tf_flush_l1d(void) { }
 #endif /* IS_ENABLED(CONFIG_KVM_INTEL) */
 
+#define arch_nmi_enter()	hw_access_profiling_stop()
+#define arch_nmi_exit()		hw_access_profiling_start()
 #endif /* _ASM_X86_HARDIRQ_H */
arch/x86/include/asm/ibs.h +2 -0
index b5a4f2ca6330..6b480958534e 100644
@@ -2,6 +2,8 @@
 #ifndef _ASM_X86_IBS_H
 #define _ASM_X86_IBS_H
 
+void hw_access_profiling_start(void);
+void hw_access_profiling_stop(void);
 extern bool arch_hw_access_profiling;
 
 #endif /* _ASM_X86_IBS_H */
arch/x86/mm/ibs.c +32 -0
index 6669710dd35b..3128e8fa5f39 100644
@@ -16,6 +16,7 @@ static u64 ibs_config __read_mostly;
 static u32 ibs_caps;
 
 #define IBS_NR_SAMPLES	150
+#define IBS_SAMPLE_PERIOD      10000
 
 /*
  * Basic access info captured for each memory access.
@@ -98,6 +99,36 @@ static void ibs_irq_handler(struct irq_work *i)
 	schedule_work_on(smp_processor_id(), &ibs_work);
 }
 
+void hw_access_profiling_stop(void)
+{
+	u64 ops_ctl;
+
+	if (!arch_hw_access_profiling)
+		return;
+
+	rdmsrl(MSR_AMD64_IBSOPCTL, ops_ctl);
+	wrmsrl(MSR_AMD64_IBSOPCTL, ops_ctl & ~IBS_OP_ENABLE);
+}
+
+void hw_access_profiling_start(void)
+{
+	u64 config = 0;
+	unsigned int period = IBS_SAMPLE_PERIOD;
+
+	if (!arch_hw_access_profiling)
+		return;
+
+	/* Disable IBS for kernel thread */
+	if (!current->mm)
+		goto out;
+
+	config = (period >> 4)  & IBS_OP_MAX_CNT;
+	config |= (period & IBS_OP_MAX_CNT_EXT_MASK);
+	config |= ibs_config;
+out:
+	wrmsrl(MSR_AMD64_IBSOPCTL, config);
+}
+
 /*
  * IBS NMI handler: Process the memory access info reported by IBS.
  *
@@ -304,6 +335,7 @@ static int __init ibs_access_profiling_init(void)
 			  x86_amd_ibs_access_profile_startup,
 			  x86_amd_ibs_access_profile_teardown);
 
+	arch_hw_access_profiling = true;
 	pr_info("IBS setup for memory access profiling\n");
 	return 0;
 }
Signature · 1 line
2.34.1
#12 lore raw

Re: [RFC PATCH v2 5/8] x86: ibs: Enable IBS profiling for memory accesses

Quoted 6 lines · On Wed, 10 Sep 2025 20:16:50 +0530 Bharata B Rao <[email protected]> wrote:

Enable IBS memory access data collection for user memory accesses by programming the required MSRs. The profiling is turned ON only for user mode execution and turned OFF for kernel mode execution. Profiling is explicitly disabled for NMI handler too.

Quoted 6 lines

TODOs:

- IBS sampling rate is kept fixed for now.
- Arch/vendor separation/isolation of the code needs relook.

One "oops I misread it" wrt to review of previous patch.
+ a really trivial thing.

J

Quoted 37 lines
---
 arch/x86/include/asm/entry-common.h |  3 +++
 arch/x86/include/asm/hardirq.h      |  2 ++
 arch/x86/include/asm/ibs.h          |  2 ++
 arch/x86/mm/ibs.c                   | 32 +++++++++++++++++++++++++++++
 4 files changed, 39 insertions(+)
arch/x86/mm/ibs.c +15 -0
index 6669710dd35b..3128e8fa5f39 100644
@@ -16,6 +16,7 @@ static u64 ibs_config __read_mostly;
 static u32 ibs_caps;
 
 #define IBS_NR_SAMPLES	150
+#define IBS_SAMPLE_PERIOD      10000
 
 /*
  * Basic access info captured for each memory access.
@@ -98,6 +99,36 @@ static void ibs_irq_handler(struct irq_work *i)
 	schedule_work_on(smp_processor_id(), &ibs_work);
 }
 
+void hw_access_profiling_stop(void)
+{
+	u64 ops_ctl;
+
+	if (!arch_hw_access_profiling)
+		return;
+
+	rdmsrl(MSR_AMD64_IBSOPCTL, ops_ctl);
+	wrmsrl(MSR_AMD64_IBSOPCTL, ops_ctl & ~IBS_OP_ENABLE);
+}
+
+void hw_access_profiling_start(void)
+{
+	u64 config = 0;

arch/x86/mm/ibs.c static void ibs_irq_handler(struct irq_work *i) · line 125

Quoted 10 lines
arch/x86/mm/ibs.c +10 -0
+	unsigned int period = IBS_SAMPLE_PERIOD;
+
+	if (!arch_hw_access_profiling)
+		return;
+
+	/* Disable IBS for kernel thread */
+	if (!current->mm)
+		goto out;
+
+	config = (period >> 4)  & IBS_OP_MAX_CNT;

Bonus space though before & that can go.

arch/x86/mm/ibs.c static void ibs_irq_handler(struct irq_work *i) · line 127

Quoted 2 lines
arch/x86/mm/ibs.c +2 -0
+	config |= (period & IBS_OP_MAX_CNT_EXT_MASK);
+	config |= ibs_config;

Ah. Ignore comment in previous patch on this not being global. Clearly it needs to be. Oops I misread this earlier.

Quoted 15 lines
arch/x86/mm/ibs.c +5 -0
+out:
+	wrmsrl(MSR_AMD64_IBSOPCTL, config);
+}
+
 /*
  * IBS NMI handler: Process the memory access info reported by IBS.
  *
@@ -304,6 +335,7 @@ static int __init ibs_access_profiling_init(void)
 			  x86_amd_ibs_access_profile_startup,
 			  x86_amd_ibs_access_profile_teardown);
 
+	arch_hw_access_profiling = true;
 	pr_info("IBS setup for memory access profiling\n");
 	return 0;
 }
#13 lore raw

RFC PATCH v2 6/8 mm: mglru: generalize page table walk

In reply to Bharata B Rao

From: Kinsey Ho <[email protected]>

Refactor the existing MGLRU page table walking logic to make it resumable.

Additionally, introduce two hooks into the MGLRU page table walk:
accessed callback and flush callback. The accessed callback is called for each accessed page detected via the scanned accessed bit. The flush callback is called when the accessed callback reports an out of space error. This allows for processing pages in batches for efficiency.

With a generalised page table walk, introduce a new scan function which repeatedly scans on the same young generation and does not add a new young generation.

---
 include/linux/mmzone.h |   5 ++
 mm/internal.h          |   4 +
 mm/vmscan.c            | 176 ++++++++++++++++++++++++++++++-----------
 3 files changed, 139 insertions(+), 46 deletions(-)
include/linux/mmzone.h +5 -0
index f7094babed10..4ad15490aff6 100644
@@ -533,6 +533,8 @@ struct lru_gen_mm_walk {
 	unsigned long seq;
 	/* the next address within an mm to scan */
 	unsigned long next_addr;
+	/* called for each accessed pte/pmd */
+	int (*accessed_cb)(unsigned long pfn);
 	/* to batch promoted pages */
 	int nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
 	/* to batch the mm stats */
@@ -540,6 +542,9 @@ struct lru_gen_mm_walk {
 	/* total batched items */
 	int batched;
 	int swappiness;
+	/* for the pmd under scanning */
+	int nr_young_pte;
+	int nr_total_pte;
 	bool force_scan;
 };
 
mm/internal.h +4 -0
index 45b725c3dc03..6c2c86abfde2 100644
@@ -548,6 +548,10 @@ static inline int user_proactive_reclaim(char *buf,
 	return 0;
 }
 #endif
+void set_task_reclaim_state(struct task_struct *task,
+				   struct reclaim_state *rs);
+void lru_gen_scan_lruvec(struct lruvec *lruvec, unsigned long seq,
+			 int (*accessed_cb)(unsigned long), void (*flush_cb)(void));
 
 /*
  * in mm/rmap.c:
mm/vmscan.c +130 -46

346 lines not on this page · raw message

Signature · 1 line
2.34.1
#14 lore raw

RFC PATCH v2 7/8 mm: klruscand: use mglru scanning for page promotion

In reply to Bharata B Rao

From: Kinsey Ho <[email protected]>

Introduce a new kernel daemon, klruscand, that periodically invokes the MGLRU page table walk. It leverages the new callbacks to gather access information and forwards it to the pghot hot page tracking sub-system for promotion decisions.

This benefits from reusing the existing MGLRU page table walk infrastructure, which is optimized with features such as hierarchical scanning and bloom filters to reduce CPU overhead.

As an additional optimization to be added in the future, we can tune the scan intervals for each memcg.

[Reduced the scan interval to 100ms, pfn_t to unsigned long]
---
 mm/Kconfig     |   8 ++++
 mm/Makefile    |   1 +
 mm/klruscand.c | 118 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 127 insertions(+)
 create mode 100644 mm/klruscand.c
mm/Kconfig +8 -0
index 8b236eb874cf..6d53c1208729 100644
@@ -1393,6 +1393,14 @@ config PGHOT
 	  by various sources. Asynchronous promotion is done by per-node
 	  kernel threads.
 
+config KLRUSCAND
+	bool "Kernel lower tier access scan daemon"
+	default y
+	depends on PGHOT && LRU_GEN_WALKS_MMU
+	help
+	  Scan for accesses from lower tiers by invoking MGLRU to perform
+	  page table walks.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +1 -0
index ecdd5241bea8..05a96ec35aa3 100644
@@ -148,3 +148,4 @@ obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o
 obj-$(CONFIG_PGHOT) += pghot.o
+obj-$(CONFIG_KLRUSCAND) += klruscand.o
mm/klruscand.c new file +118 -0

118 lines not on this page · raw message

Signature · 1 line
2.34.1
#15 lore raw

Re: [RFC PATCH v2 7/8] mm: klruscand: use mglru scanning for page promotion

Quoted 11 lines · On Wed, 10 Sep 2025 20:16:52 +0530 Bharata B Rao <[email protected]> wrote:

From: Kinsey Ho <[email protected]>

Introduce a new kernel daemon, klruscand, that periodically invokes the MGLRU page table walk. It leverages the new callbacks to gather access information and forwards it to the pghot hot page tracking sub-system for promotion decisions.

This benefits from reusing the existing MGLRU page table walk infrastructure, which is optimized with features such as hierarchical scanning and bloom filters to reduce CPU overhead.

Quoted 7 lines

As an additional optimization to be added in the future, we can tune the scan intervals for each memcg.

[Reduced the scan interval to 100ms, pfn_t to unsigned long]

Some very minor comments inline. I know even less about the stuff this is using than IBS (and I don't know much about that ;)

J

Quoted 12 lines
---
 mm/Kconfig     |   8 ++++
 mm/Makefile    |   1 +
 mm/klruscand.c | 118 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 127 insertions(+)
 create mode 100644 mm/klruscand.c
mm/Kconfig +0 -0
index 8b236eb874cf..6d53c1208729 100644

mm/Kconfig config PGHOT · line 1398

Quoted 6 lines
mm/Kconfig +3 -0
@@ -1393,6 +1393,14 @@ config PGHOT
 	  by various sources. Asynchronous promotion is done by per-node
 	  kernel threads.
 
+config KLRUSCAND
+	bool "Kernel lower tier access scan daemon"
+	default y

Why default to y? That's very rarely done for new features.

mm/klruscand.c line 2

Quoted 13 lines
mm/Kconfig +4 -0
+	depends on PGHOT && LRU_GEN_WALKS_MMU
+	help
+	  Scan for accesses from lower tiers by invoking MGLRU to perform
+	  page table walks.
 
mm/klruscand.c new file +2 -0
new file mode 100644
index 000000000000..1a51aab29bd9
@@ -0,0 +1,118 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include <linux/memcontrol.h>

Probably pick some ordering scheme for includes.
I'm not spotting what is currently used here.

Quoted 18 lines
mm/klruscand.c new file +18 -0
+#include <linux/kthread.h>
+#include <linux/module.h>
+#include <linux/vmalloc.h>
+#include <linux/random.h>
+#include <linux/migrate.h>
+#include <linux/mm_inline.h>
+#include <linux/slab.h>
+#include <linux/sched/clock.h>
+#include <linux/memory-tiers.h>
+#include <linux/sched/mm.h>
+#include <linux/sched.h>
+#include <linux/pghot.h>
+
+#include "internal.h"
+
+#define KLRUSCAND_INTERVAL_MS 100
+#define BATCH_SIZE (2 << 16)
+

mm/klruscand.c line 30

Quoted 10 lines
mm/klruscand.c new file +10 -0
+static struct task_struct *scan_thread;
+static unsigned long pfn_batch[BATCH_SIZE];
+static int batch_index;
+
+static void flush_cb(void)
+{
+	int i = 0;
+
+	for (; i < batch_index; i++) {
+		u64 pfn = pfn_batch[i];

Why dance through types? pfn_batch is unsigned long and it is cast back to that below.

mm/klruscand.c line 35

Quoted 5 lines
mm/klruscand.c new file +5 -0
+
+		pghot_record_access((unsigned long)pfn, NUMA_NO_NODE,
+					PGHOT_PGTABLE_SCAN, jiffies);
+
+		if (i % 16 == 0)

No problem with this, but maybe a comment on why 16?

mm/klruscand.c line 55

Quoted 11 lines
mm/klruscand.c new file +10 -0
+			cond_resched();
+	}
+	batch_index = 0;
+}
 
+static int klruscand_run(void *unused)
+{
+	struct lru_gen_mm_walk *walk;
+
+	walk = kzalloc(sizeof(*walk),
+		       __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN);

Maybe use __free() magic so we can forget about having to clear this up on exit. Entirely up to you though as doesn't simplify code much in this case.

Quoted 47 lines
mm/klruscand.c new file +47 -0
+	if (!walk)
+		return -ENOMEM;
+
+	while (!kthread_should_stop()) {
+		unsigned long next_wake_time;
+		long sleep_time;
+		struct mem_cgroup *memcg;
+		int flags;
+		int nid;
+
+		next_wake_time = jiffies + msecs_to_jiffies(KLRUSCAND_INTERVAL_MS);
+
+		for_each_node_state(nid, N_MEMORY) {
+			pg_data_t *pgdat = NODE_DATA(nid);
+			struct reclaim_state rs = { 0 };
+
+			if (node_is_toptier(nid))
+				continue;
+
+			rs.mm_walk = walk;
+			set_task_reclaim_state(current, &rs);
+			flags = memalloc_noreclaim_save();
+
+			memcg = mem_cgroup_iter(NULL, NULL, NULL);
+			do {
+				struct lruvec *lruvec =
+					mem_cgroup_lruvec(memcg, pgdat);
+				unsigned long max_seq =
+					READ_ONCE((lruvec)->lrugen.max_seq);
+
+				lru_gen_scan_lruvec(lruvec, max_seq,
+						    accessed_cb, flush_cb);
+				cond_resched();
+			} while ((memcg = mem_cgroup_iter(NULL, memcg, NULL)));
+
+			memalloc_noreclaim_restore(flags);
+			set_task_reclaim_state(current, NULL);
+			memset(walk, 0, sizeof(*walk));
+		}
+
+		sleep_time = next_wake_time - jiffies;
+		if (sleep_time > 0 && sleep_time != MAX_SCHEDULE_TIMEOUT)
+			schedule_timeout_idle(sleep_time);
+	}
+	kfree(walk);
+	return 0;
+}
#16 lore raw

RFC PATCH v2 8/8 mm: sched: Move hot page promotion from NUMAB=2 to kpromoted

In reply to Bharata B Rao

Currently hot page promotion (NUMA_BALANCING_MEMORY_TIERING mode of NUMA Balancing) does hot page detection (via hint faults), hot page classification and eventual promotion, all by itself and sits within the scheduler.

With the new hot page tracking and promotion mechanism being available, NUMA Balancing can limit itself to detection of hot pages (via hint faults) and off-load rest of the functionality to the common hot page tracking system.

pghot_record_access(PGHOT_HINT_FAULT) API is used to feed the hot page info. In addition, the migration rate limiting and dynamic threshold logic are moved to kpromoted so that the same can be used for hot pages reported by other sources too.

---
 include/linux/pghot.h |   2 +
 kernel/sched/fair.c   | 149 ++----------------------------------------
 mm/memory.c           |  32 ++-------
 mm/pghot.c            | 132 +++++++++++++++++++++++++++++++++++--
 4 files changed, 142 insertions(+), 173 deletions(-)
include/linux/pghot.h +2 -0
index 1443643aab13..98a72e01bdd6 100644
@@ -47,6 +47,8 @@ enum pghot_src {
 #define PGHOT_HEAP_PCT		25
 
 #define KPROMOTED_MIGRATE_BATCH	1024
+#define KPROMOTED_MIGRATION_ADJUST_STEPS	16
+#define KPROMOTED_PROMOTION_THRESHOLD_WINDOW	60000
 
 /*
  * If target NID isn't available, kpromoted promotes to node 0
kernel/sched/fair.c +5 -144

173 lines not on this page · raw message

mm/memory.c +7 -25
index 0ba4f6b71847..eeb34e8d9b8e 100644
@@ -75,6 +75,7 @@
 #include <linux/ptrace.h>
 #include <linux/vmalloc.h>
 #include <linux/sched/sysctl.h>
+#include <linux/pghot.h>
 
 #include <trace/events/kmem.h>
 
@@ -5864,34 +5865,12 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 
 	target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
 					writable, &last_cpupid);
+	nid = target_nid;
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
-	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
-		flags |= TNF_MIGRATE_FAIL;
-		goto out_map;
-	}
-	/* The folio is isolated and isolation code holds a folio reference. */
-	pte_unmap_unlock(vmf->pte, vmf->ptl);
+
 	writable = false;
 	ignore_writable = true;
-
-	/* Migrate to the requested node */
-	if (!migrate_misplaced_folio(folio, target_nid)) {
-		nid = target_nid;
-		flags |= TNF_MIGRATED;
-		task_numa_fault(last_cpupid, nid, nr_pages, flags);
-		return 0;
-	}
-
-	flags |= TNF_MIGRATE_FAIL;
-	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
-				       vmf->address, &vmf->ptl);
-	if (unlikely(!vmf->pte))
-		return 0;
-	if (unlikely(!pte_same(ptep_get(vmf->pte), vmf->orig_pte))) {
-		pte_unmap_unlock(vmf->pte, vmf->ptl);
-		return 0;
-	}
 out_map:
 	/*
 	 * Make it present again, depending on how arch implements
@@ -5905,8 +5884,11 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 					    writable);
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
 
-	if (nid != NUMA_NO_NODE)
+	if (nid != NUMA_NO_NODE) {
+		pghot_record_access(folio_pfn(folio), nid, PGHOT_HINT_FAULT,
+				    jiffies);
 		task_numa_fault(last_cpupid, nid, nr_pages, flags);
+	}
 	return 0;
 }
 
mm/pghot.c +128 -4

175 lines not on this page · raw message

Signature · 1 line
2.34.1
#17 lore raw

Re: [RFC PATCH v2 8/8] mm: sched: Move hot page promotion from NUMAB=2 to kpromoted

Quoted 10 lines · On Wed, 10 Sep 2025 20:16:53 +0530 Bharata B Rao <[email protected]> wrote:

Currently hot page promotion (NUMA_BALANCING_MEMORY_TIERING mode of NUMA Balancing) does hot page detection (via hint faults), hot page classification and eventual promotion, all by itself and sits within the scheduler.

With the new hot page tracking and promotion mechanism being available, NUMA Balancing can limit itself to detection of hot pages (via hint faults) and off-load rest of the functionality to the common hot page tracking system.

Quoted 6 lines

pghot_record_access(PGHOT_HINT_FAULT) API is used to feed the hot page info. In addition, the migration rate limiting and dynamic threshold logic are moved to kpromoted so that the same can be used for hot pages reported by other sources too.

Making a direct replacement without any fallback to previous method is going to need a lot of data to show there are no important regressions.

So bold move if that's the intent!

J

Quoted 23 lines
---
 include/linux/pghot.h |   2 +
 kernel/sched/fair.c   | 149 ++----------------------------------------
 mm/memory.c           |  32 ++-------
 mm/pghot.c            | 132 +++++++++++++++++++++++++++++++++++--
 4 files changed, 142 insertions(+), 173 deletions(-)
mm/pghot.c +3 -0
index 9f7581818b8f..9f5746892bce 100644
@@ -9,6 +9,9 @@
  *
  * kpromoted is a kernel thread that runs on each toptier node and
  * promotes pages from max_heap.
+ *
+ * Migration rate-limiting and dynamic threshold logic implementations
+ * were moved from NUMA Balancing mode 2.
  */
 #include <linux/pghot.h>
 #include <linux/kthread.h>

mm/pghot.c static bool kpromoted_started __ro_after_init; · line 41

Quoted 5 lines
mm/pghot.c +2 -0
@@ -34,6 +37,9 @@ static bool kpromoted_started __ro_after_init;
 
 static unsigned int sysctl_pghot_freq_window = KPROMOTED_FREQ_WINDOW;
 
+/* Restrict the NUMA promotion throughput (MB/s) for each target node. */
+static unsigned int sysctl_pghot_promote_rate_limit = 65536;

If the comment correlates with the value, this is 64 GiB/s? That seems unlikely if I guess possible.

Quoted 9 lines
mm/pghot.c +2 -0
+
 #ifdef CONFIG_SYSCTL
 static const struct ctl_table pghot_sysctls[] = {
 	{
@@ -44,8 +50,17 @@ static const struct ctl_table pghot_sysctls[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= SYSCTL_ZERO,
 	},
+	{

mm/pghot.c static const struct ctl_table pghot_sysctls[] = { · line 63

Quoted 10 lines
mm/pghot.c +8 -0
+		.procname	= "pghot_promote_rate_limit_MBps",
+		.data		= &sysctl_pghot_promote_rate_limit,
+		.maxlen		= sizeof(unsigned int),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+	},
 };
 #endif
+

Put that in earlier patch to reduce noise here.

mm/pghot.c static bool phi_heap_insert(struct max_heap *phi_heap, struct pghot_info *phi) · line 114

Quoted 10 lines
mm/pghot.c +3 -0
 static bool phi_heap_less(const void *lhs, const void *rhs, void *args)
 {
 	return (*(struct pghot_info **)lhs)->frequency >
@@ -94,11 +109,99 @@ static bool phi_heap_insert(struct max_heap *phi_heap, struct pghot_info *phi)
 	return true;
 }
 
+/*
+ * For memory tiering mode, if there are enough free pages (more than
+ * enough watermark defined here) in fast memory node, to take full

I'd use enough_wmark Just because "more than enough" is a common
English phrase and I at least tripped over that sentence as a result!

Quoted 17 lines
mm/pghot.c +17 -0
+ * advantage of fast memory capacity, all recently accessed slow
+ * memory pages will be migrated to fast memory node without
+ * considering hot threshold.
+ */
+static bool pgdat_free_space_enough(struct pglist_data *pgdat)
+{
+	int z;
+	unsigned long enough_wmark;
+
+	enough_wmark = max(1UL * 1024 * 1024 * 1024 >> PAGE_SHIFT,
+			   pgdat->node_present_pages >> 4);
+	for (z = pgdat->nr_zones - 1; z >= 0; z--) {
+		struct zone *zone = pgdat->node_zones + z;
+
+		if (!populated_zone(zone))
+			continue;
+

mm/pghot.c static bool phi_heap_insert(struct max_heap *phi_heap, struct pghot_info *phi) · line 164

Quoted 10 lines
mm/pghot.c +9 -0
+		if (zone_watermark_ok(zone, 0,
+				      promo_wmark_pages(zone) + enough_wmark,
+				      ZONE_MOVABLE, 0))
+			return true;
+	}
+	return false;
+}
 
+
+static void kpromoted_promotion_adjust_threshold(struct pglist_data *pgdat,

Needs documentation of the algorithm and the reasons for various choices.

I see it is a code move though so maybe that's a job for another day.

Quoted 16 lines
mm/pghot.c +16 -0
+						 unsigned long rate_limit,
+						 unsigned int ref_th,
+						 unsigned long now)
+{
+	unsigned int start, th_period, unit_th, th;
+	unsigned long nr_cand, ref_cand, diff_cand;
+
+	now = jiffies_to_msecs(now);
+	th_period = KPROMOTED_PROMOTION_THRESHOLD_WINDOW;
+	start = pgdat->nbp_th_start;
+	if (now - start > th_period &&
+	    cmpxchg(&pgdat->nbp_th_start, start, now) == start) {
+		ref_cand = rate_limit *
+			KPROMOTED_PROMOTION_THRESHOLD_WINDOW / MSEC_PER_SEC;
+		nr_cand = node_page_state(pgdat, PGPROMOTE_CANDIDATE);
+		diff_cand = nr_cand - pgdat->nbp_th_nr_cand;

mm/pghot.c static bool phi_heap_insert(struct max_heap *phi_heap, struct pghot_info *phi) · line 190

Quoted 10 lines
mm/pghot.c +10 -0
+		unit_th = ref_th * 2 / KPROMOTED_MIGRATION_ADJUST_STEPS;
+		th = pgdat->nbp_threshold ? : ref_th;
+		if (diff_cand > ref_cand * 11 / 10)
+			th = max(th - unit_th, unit_th);
+		else if (diff_cand < ref_cand * 9 / 10)
+			th = min(th + unit_th, ref_th * 2);
+		pgdat->nbp_th_nr_cand = nr_cand;
+		pgdat->nbp_threshold = th;
+	}
+}

+

mm/pghot.c static bool phi_heap_insert(struct max_heap *phi_heap, struct pghot_info *phi) · line 205

Quoted 10 lines
mm/pghot.c +4 -1
 static bool phi_is_pfn_hot(struct pghot_info *phi)
 {
 	struct page *page = pfn_to_online_page(phi->pfn);
-	unsigned long now = jiffies;
 	struct folio *folio;
+	struct pglist_data *pgdat;
+	unsigned long rate_limit;
+	unsigned int latency, th, def_th;
+	unsigned long now = jiffies;
 

Avoid the reorder. Just put it here in first place if you prefer this.

#18 lore raw

Re: [RFC PATCH v2 8/8] mm: sched: Move hot page promotion from NUMAB=2 to kpromoted

Quoted 18 lines · Jonathan Cameron
Quoted 15 lines · On Wed, 10 Sep 2025 20:16:53 +0530 Bharata B Rao <[email protected]> wrote:

Currently hot page promotion (NUMA_BALANCING_MEMORY_TIERING mode of NUMA Balancing) does hot page detection (via hint faults), hot page classification and eventual promotion, all by itself and sits within the scheduler.

With the new hot page tracking and promotion mechanism being available, NUMA Balancing can limit itself to detection of hot pages (via hint faults) and off-load rest of the functionality to the common hot page tracking system.

pghot_record_access(PGHOT_HINT_FAULT) API is used to feed the hot page info. In addition, the migration rate limiting and dynamic threshold logic are moved to kpromoted so that the same can be used for hot pages reported by other sources too.

Quoted 6 lines
Quoted 1 line

Making a direct replacement without any fallback to previous method is going to need a lot of data to show there are no important regressions.

So bold move if that's the intent!

Firstly I am only moving the existing hot page heuristics that is part of NUMAB=2 to kpromoted so that the same can be applied to hot pages being identified by other sources. So the hint fault mechanism that is inherent to NUMAB=2 still remains.

In fact, kscand effort started as a potential replacement for the existing hot page promotion mechanism by getting rid of hint faults and moving the page table scanning out of process context.

In any case, I will start including numbers from the next post.

Quoted 8 lines

mm/pghot.c static bool kpromoted_started __ro_after_init; · line 41

Quoted 5 lines

static unsigned int sysctl_pghot_freq_window = KPROMOTED_FREQ_WINDOW;

mm/pghot.c +2 -0
+/* Restrict the NUMA promotion throughput (MB/s) for each target node. */
+static unsigned int sysctl_pghot_promote_rate_limit = 65536;

If the comment correlates with the value, this is 64 GiB/s? That seems unlikely if I guess possible.

IIUC, the existing logic tries to limit promotion rate to 64 GiB/s by limiting the number of candidate pages that are promoted within the 1s observation interval.

Are you saying that achieving the rate of 64 GiB/s is not possible or unlikely?

Quoted 13 lines
Quoted 13 lines
mm/pghot.c +5 -0
+
 #ifdef CONFIG_SYSCTL
 static const struct ctl_table pghot_sysctls[] = {
 	{
@@ -44,8 +50,17 @@ static const struct ctl_table pghot_sysctls[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= SYSCTL_ZERO,
 	},
+	{
+		.procname	= "pghot_promote_rate_limit_MBps",
+		.data		= &sysctl_pghot_promote_rate_limit,
+		.maxlen		= sizeof(unsigned int),
Quoted 8 lines

mm/pghot.c static const struct ctl_table pghot_sysctls[] = { · line 63

Quoted 7 lines
mm/pghot.c +5 -0
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+	},
 };
 #endif
+

Put that in earlier patch to reduce noise here.

This patch moves the hot page heuristics to kpromoted and hence this related sysctl is also being moved in this patch.

Quoted 8 lines
Quoted 8 lines
mm/pghot.c +0 -0
 static bool phi_heap_less(const void *lhs, const void *rhs, void *args)
 {
 	return (*(struct pghot_info **)lhs)->frequency >
@@ -94,11 +109,99 @@ static bool phi_heap_insert(struct max_heap *phi_heap, struct pghot_info *phi)
 	return true;
 }
 
Quoted 6 lines

mm/pghot.c static bool phi_heap_insert(struct max_heap *phi_heap, struct pghot_info *phi) · line 114

Quoted 3 lines
mm/pghot.c +3 -0
+/*
+ * For memory tiering mode, if there are enough free pages (more than
+ * enough watermark defined here) in fast memory node, to take full

I'd use enough_wmark Just because "more than enough" is a common
English phrase and I at least tripped over that sentence as a result!

Ah I see that, but as you note later, I am currently only doing the movement.

Quoted 26 lines
Quoted 25 lines
mm/pghot.c +24 -0
+ * advantage of fast memory capacity, all recently accessed slow
+ * memory pages will be migrated to fast memory node without
+ * considering hot threshold.
+ */
+static bool pgdat_free_space_enough(struct pglist_data *pgdat)
+{
+	int z;
+	unsigned long enough_wmark;
+
+	enough_wmark = max(1UL * 1024 * 1024 * 1024 >> PAGE_SHIFT,
+			   pgdat->node_present_pages >> 4);
+	for (z = pgdat->nr_zones - 1; z >= 0; z--) {
+		struct zone *zone = pgdat->node_zones + z;
+
+		if (!populated_zone(zone))
+			continue;
+
+		if (zone_watermark_ok(zone, 0,
+				      promo_wmark_pages(zone) + enough_wmark,
+				      ZONE_MOVABLE, 0))
+			return true;
+	}
+	return false;
+}
Quoted 6 lines

Review static bool phi_heap_insert(struct max_heap *phi_heap, struct pghot_info *phi) · line 140

Quoted 2 lines
(quoted diff) +2 -0
+
+static void kpromoted_promotion_adjust_threshold(struct pglist_data *pgdat,

Needs documentation of the algorithm and the reasons for various choices.

I see it is a code move though so maybe that's a job for another day.

Sure.

Regards,
Bharata.

#19 lore raw

Re: [RFC PATCH v2 8/8] mm: sched: Move hot page promotion from NUMAB=2 to kpromoted

Quoted 24 lines · On Mon, 6 Oct 2025 11:27:21 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 23 lines · Jonathan Cameron
Quoted 10 lines · On Wed, 10 Sep 2025 20:16:53 +0530 Bharata B Rao <[email protected]> wrote:

Currently hot page promotion (NUMA_BALANCING_MEMORY_TIERING mode of NUMA Balancing) does hot page detection (via hint faults), hot page classification and eventual promotion, all by itself and sits within the scheduler.

With the new hot page tracking and promotion mechanism being available, NUMA Balancing can limit itself to detection of hot pages (via hint faults) and off-load rest of the functionality to the common hot page tracking system.

Quoted 6 lines

pghot_record_access(PGHOT_HINT_FAULT) API is used to feed the hot page info. In addition, the migration rate limiting and dynamic threshold logic are moved to kpromoted so that the same can be used for hot pages reported by other sources too.

Making a direct replacement without any fallback to previous method is going to need a lot of data to show there are no important regressions.

Quoted 6 lines
Quoted 1 line

So bold move if that's the intent!

Firstly I am only moving the existing hot page heuristics that is part of NUMAB=2 to kpromoted so that the same can be applied to hot pages being identified by other sources. So the hint fault mechanism that is inherent to NUMAB=2 still remains.

That makes sense.

Quoted 4 lines

In fact, kscand effort started as a potential replacement for the existing hot page promotion mechanism by getting rid of hint faults and moving the page table scanning out of process context.

Understood and I'm in favor of the that approach but not sure it will be a fit for all workloads.

Quoted 2 lines

In any case, I will start including numbers from the next post.

Great.

Quoted 9 lines
Quoted 8 lines

mm/pghot.c static bool kpromoted_started __ro_after_init; · line 41

Quoted 5 lines

static unsigned int sysctl_pghot_freq_window = KPROMOTED_FREQ_WINDOW;

mm/pghot.c +2 -0
+/* Restrict the NUMA promotion throughput (MB/s) for each target node. */
+static unsigned int sysctl_pghot_promote_rate_limit = 65536;  

If the comment correlates with the value, this is 64 GiB/s? That seems unlikely if I guess possible.

Quoted 6 lines

IIUC, the existing logic tries to limit promotion rate to 64 GiB/s by limiting the number of candidate pages that are promoted within the 1s observation interval.

Are you saying that achieving the rate of 64 GiB/s is not possible or unlikely?

Seem rather too high to me, but maybe I just have the wrong mental model of what we should be moving.

Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
mm/pghot.c +5 -0
+
 #ifdef CONFIG_SYSCTL
 static const struct ctl_table pghot_sysctls[] = {
 	{
@@ -44,8 +50,17 @@ static const struct ctl_table pghot_sysctls[] = {
 		.proc_handler	= proc_dointvec_minmax,
 		.extra1		= SYSCTL_ZERO,
 	},
+	{
+		.procname	= "pghot_promote_rate_limit_MBps",
+		.data		= &sysctl_pghot_promote_rate_limit,
+		.maxlen		= sizeof(unsigned int),
Quoted 8 lines

mm/pghot.c static const struct ctl_table pghot_sysctls[] = { · line 63

Quoted 7 lines
mm/pghot.c +5 -0
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+	},
 };
 #endif
+  

Put that in earlier patch to reduce noise here.

Quoted 2 lines

This patch moves the hot page heuristics to kpromoted and hence this related sysctl is also being moved in this patch.

I just mean the blank line - not the block above.
This is just a patch set tidying up comment.

Jonathan

#20 lore raw
In reply to Bharata B Rao
Quoted 4 lines · Bharata B Rao

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) that consolidates memory access information from various sources and enables centralized promotion of hot pages across memory tiers.

Just to be clear, I continue to believe this is a terrible idea and we should not do this. If systems will be built with CXL (and given the horrendous performance, I cannot see why they would be), the kernel should not be migrating memory around like this.

reviewer
#21 lore raw
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Bharata B Rao

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) that consolidates memory access information from various sources and enables centralized promotion of hot pages across memory tiers.

Just to be clear, I continue to believe this is a terrible idea and we should not do this. If systems will be built with CXL (and given the horrendous performance, I cannot see why they would be), the kernel should not be migrating memory around like this.

I've been considered this problem from the opposite approach since LSFMM.

Rather than decide how to move stuff around, what if instead we just decide not to ever put certain classes of memory on CXL. Right now, so long as CXL is in the page allocator, it's the wild west - any page can end up anywhere.

I have enough data now from ZONE_MOVABLE-only CXL deployments on real workloads to show local CXL expansion is valuable and performant enough to be worth deploying - but the key piece for me is that ZONE_MOVABLE disallows GFP_KERNEL. For example: this keeps SLAB meta-data out of CXL, but allows any given user-driven page allocation (including page cache, file, and anon mappings) to land there.

I'm hoping to share some of this data in the coming months.

I've yet to see any strong indication that a complex hotness/movement system is warranted (yet) - but that may simply be because we have local cards with no switching involved. So far LRU-based promotion and demotion has been sufficient.

It seems the closer to random-access the access pattern, the less valuable ANY movement is. Which should be intuitive. But, having CXL beats touching disk every day of the week.

So I've become conflicted on this work - but only because I haven't seen the data to suggest such complexity is warranted.

~Gregory

#22 lore raw
Quoted 19 lines · Gregory Price
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Bharata B Rao

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) that consolidates memory access information from various sources and enables centralized promotion of hot pages across memory tiers.

Just to be clear, I continue to believe this is a terrible idea and we should not do this. If systems will be built with CXL (and given the horrendous performance, I cannot see why they would be), the kernel should not be migrating memory around like this.

I've been considered this problem from the opposite approach since LSFMM.

Rather than decide how to move stuff around, what if instead we just decide not to ever put certain classes of memory on CXL. Right now, so long as CXL is in the page allocator, it's the wild west - any page can end up anywhere.

Quoted 7 lines

I have enough data now from ZONE_MOVABLE-only CXL deployments on real workloads to show local CXL expansion is valuable and performant enough to be worth deploying - but the key piece for me is that ZONE_MOVABLE disallows GFP_KERNEL. For example: this keeps SLAB meta-data out of CXL, but allows any given user-driven page allocation (including page cache, file, and anon mappings) to land there.

This is similar to our use case, although the direct allocation can be controlled by cpusets or mempolicies as needed depending on the memory access latency required for the workload; nothing new there, though, it's the same argument as NUMA in general and the abstraction of these far memory nodes as separate NUMA nodes makes this very straightforward.

Quoted 7 lines

I'm hoping to share some of this data in the coming months.

I've yet to see any strong indication that a complex hotness/movement system is warranted (yet) - but that may simply be because we have local cards with no switching involved. So far LRU-based promotion and demotion has been sufficient.

To me, this is a key point. As we've discussed in meetings, we're in the early days here. The CHMU does provide a lot of flexibility, both to create very good and very bad hotness trackers. But I think the key point is that we have multiple sources of hotness information depending on the platform and some of these sources only make sense for the kernel (or a BPF offload) to maintain as the source of truth. Some of these sources will be clear-on-read so only one entity would be possible to have as the source of truth of page hotness.

I've been pretty focused on the promotion story here rather than demotion because of how responsive it needs to be. Harvesting the page table accessed bits or waiting on a sliding window through NUMA Balancing (even NUMAB=2) is not as responsive as needed for very fast promotion to top tier memory, hence things like the CHMU (or PEBS or IBS etc).

A few things that I think we need to discuss and align on:

- the kernel as the source of truth for all memory hotness information, which can then be abstracted and used for multiple downstream purposes, memory tiering only being one of them

- the long-term plan for NUMAB=2 and memory tiering support in the kernel in general, are we planning on supporting this through NUMA hint faults forever despite their drawbacks (too slow, too much overhead for KVM)

- the role of the kernel vs userspace in driving the memory migration; lots of discussion on hardware assists that can be leveraged for memory migration but today the balancing is driven in process context. The kthread as the driver of migration is yet to be a sold argument, but are where a number of companies are currently looking

There's also some feature support that is possible with these CXL memory expansion devices that have started to pop up in labs that can also drastically reduce overall TCO. Perhaps Wei Xu, cc'd, will be able to chime in as well.

This topic seems due for an alignment session as well, so will look to get that scheduled in the coming weeks if people are up for it.

Quoted 9 lines

It seems the closer to random-access the access pattern, the less valuable ANY movement is. Which should be intuitive. But, having CXL beats touching disk every day of the week.

So I've become conflicted on this work - but only because I haven't seen the data to suggest such complexity is warranted.

~Gregory

reviewer
#23 lore raw
Quoted 7 lines · David Rientjes
Quoted 7 lines

I'm hoping to share some of this data in the coming months.

I've yet to see any strong indication that a complex hotness/movement system is warranted (yet) - but that may simply be because we have local cards with no switching involved. So far LRU-based promotion and demotion has been sufficient.

...

Quoted 7 lines

I've been pretty focused on the promotion story here rather than demotion because of how responsive it needs to be. Harvesting the page table accessed bits or waiting on a sliding window through NUMA Balancing (even NUMAB=2) is not as responsive as needed for very fast promotion to top tier memory, hence things like the CHMU (or PEBS or IBS etc).

I feel the need to throw out there that we need to set some kind of baseline for comparison that isn't simply comparing new hotness tracking stuff against "Doing Nothing".

For example, if we assume MGLRU is the default, we probably want to compare against some kind of simplistic system that is essentially:

if tier0 has bandwidth room, and
if tier1 is bandwidth pressured, then
promote a chunk from tier1 youngest generation LRU ("hottest") and demote a chunk from tier0 older LRU ("coldest") [if there's no space available].

Active bandwidth utilization numbers are still a little hard to come by, but a system like above could be implemented largely in userland with a few tweaks to reclaim.

~Gregory

reviewer
#24 lore raw
In reply to David Rientjes
Quoted 75 lines · David Rientjes
Quoted 19 lines · Gregory Price
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Bharata B Rao

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) that consolidates memory access information from various sources and enables centralized promotion of hot pages across memory tiers.

Just to be clear, I continue to believe this is a terrible idea and we should not do this. If systems will be built with CXL (and given the horrendous performance, I cannot see why they would be), the kernel should not be migrating memory around like this.

I've been considered this problem from the opposite approach since LSFMM.

Rather than decide how to move stuff around, what if instead we just decide not to ever put certain classes of memory on CXL. Right now, so long as CXL is in the page allocator, it's the wild west - any page can end up anywhere.

Quoted 7 lines

I have enough data now from ZONE_MOVABLE-only CXL deployments on real workloads to show local CXL expansion is valuable and performant enough to be worth deploying - but the key piece for me is that ZONE_MOVABLE disallows GFP_KERNEL. For example: this keeps SLAB meta-data out of CXL, but allows any given user-driven page allocation (including page cache, file, and anon mappings) to land there.

This is similar to our use case, although the direct allocation can be controlled by cpusets or mempolicies as needed depending on the memory access latency required for the workload; nothing new there, though, it's the same argument as NUMA in general and the abstraction of these far memory nodes as separate NUMA nodes makes this very straightforward.

Quoted 7 lines

I'm hoping to share some of this data in the coming months.

I've yet to see any strong indication that a complex hotness/movement system is warranted (yet) - but that may simply be because we have local cards with no switching involved. So far LRU-based promotion and demotion has been sufficient.

To me, this is a key point. As we've discussed in meetings, we're in the early days here. The CHMU does provide a lot of flexibility, both to create very good and very bad hotness trackers. But I think the key point is that we have multiple sources of hotness information depending on the platform and some of these sources only make sense for the kernel (or a BPF offload) to maintain as the source of truth. Some of these sources will be clear-on-read so only one entity would be possible to have as the source of truth of page hotness.

I've been pretty focused on the promotion story here rather than demotion because of how responsive it needs to be. Harvesting the page table accessed bits or waiting on a sliding window through NUMA Balancing (even NUMAB=2) is not as responsive as needed for very fast promotion to top tier memory, hence things like the CHMU (or PEBS or IBS etc).

A few things that I think we need to discuss and align on:

- the kernel as the source of truth for all memory hotness information, which can then be abstracted and used for multiple downstream purposes, memory tiering only being one of them

- the long-term plan for NUMAB=2 and memory tiering support in the kernel in general, are we planning on supporting this through NUMA hint faults forever despite their drawbacks (too slow, too much overhead for KVM)

- the role of the kernel vs userspace in driving the memory migration; lots of discussion on hardware assists that can be leveraged for memory migration but today the balancing is driven in process context. The kthread as the driver of migration is yet to be a sold argument, but are where a number of companies are currently looking

Quoted 7 lines

There's also some feature support that is possible with these CXL memory expansion devices that have started to pop up in labs that can also drastically reduce overall TCO. Perhaps Wei Xu, cc'd, will be able to chime in as well.

This topic seems due for an alignment session as well, so will look to get that scheduled in the coming weeks if people are up for it.

Our experience is that workloads in hyper-scalar data centers such as Google often have significant cold memory. Offloading this to CXL memory devices, backed by cheaper, lower-performance media (e.g. DRAM with hardware compression), can be a practical approach to reduce overall TCO. Page promotion and demotion are then critical for such a tiered memory system.

A kernel thread to drive hot page collection and promotion seems logical, especially since hot page data from new sources (e.g. CHMU) are collected outside the process execution context and in the form of physical addresses.

I do agree that we need to balance the complexity and benefits of any new data structures for hotness tracking.

Quoted 9 lines
Quoted 9 lines

It seems the closer to random-access the access pattern, the less valuable ANY movement is. Which should be intuitive. But, having CXL beats touching disk every day of the week.

So I've become conflicted on this work - but only because I haven't seen the data to suggest such complexity is warranted.

~Gregory

#25 lore raw
Quoted 91 lines · Wei Xu
Quoted 75 lines · David Rientjes
Quoted 19 lines · Gregory Price
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Bharata B Rao

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) that consolidates memory access information from various sources and enables centralized promotion of hot pages across memory tiers.

Just to be clear, I continue to believe this is a terrible idea and we should not do this. If systems will be built with CXL (and given the horrendous performance, I cannot see why they would be), the kernel should not be migrating memory around like this.

I've been considered this problem from the opposite approach since LSFMM.

Rather than decide how to move stuff around, what if instead we just decide not to ever put certain classes of memory on CXL. Right now, so long as CXL is in the page allocator, it's the wild west - any page can end up anywhere.

Quoted 7 lines

I have enough data now from ZONE_MOVABLE-only CXL deployments on real workloads to show local CXL expansion is valuable and performant enough to be worth deploying - but the key piece for me is that ZONE_MOVABLE disallows GFP_KERNEL. For example: this keeps SLAB meta-data out of CXL, but allows any given user-driven page allocation (including page cache, file, and anon mappings) to land there.

This is similar to our use case, although the direct allocation can be controlled by cpusets or mempolicies as needed depending on the memory access latency required for the workload; nothing new there, though, it's the same argument as NUMA in general and the abstraction of these far memory nodes as separate NUMA nodes makes this very straightforward.

Quoted 7 lines

I'm hoping to share some of this data in the coming months.

I've yet to see any strong indication that a complex hotness/movement system is warranted (yet) - but that may simply be because we have local cards with no switching involved. So far LRU-based promotion and demotion has been sufficient.

To me, this is a key point. As we've discussed in meetings, we're in the early days here. The CHMU does provide a lot of flexibility, both to create very good and very bad hotness trackers. But I think the key point is that we have multiple sources of hotness information depending on the platform and some of these sources only make sense for the kernel (or a BPF offload) to maintain as the source of truth. Some of these sources will be clear-on-read so only one entity would be possible to have as the source of truth of page hotness.

I've been pretty focused on the promotion story here rather than demotion because of how responsive it needs to be. Harvesting the page table accessed bits or waiting on a sliding window through NUMA Balancing (even NUMAB=2) is not as responsive as needed for very fast promotion to top tier memory, hence things like the CHMU (or PEBS or IBS etc).

A few things that I think we need to discuss and align on:

- the kernel as the source of truth for all memory hotness information, which can then be abstracted and used for multiple downstream purposes, memory tiering only being one of them

- the long-term plan for NUMAB=2 and memory tiering support in the kernel in general, are we planning on supporting this through NUMA hint faults forever despite their drawbacks (too slow, too much overhead for KVM)

- the role of the kernel vs userspace in driving the memory migration; lots of discussion on hardware assists that can be leveraged for memory migration but today the balancing is driven in process context. The kthread as the driver of migration is yet to be a sold argument, but are where a number of companies are currently looking

Quoted 7 lines

There's also some feature support that is possible with these CXL memory expansion devices that have started to pop up in labs that can also drastically reduce overall TCO. Perhaps Wei Xu, cc'd, will be able to chime in as well.

This topic seems due for an alignment session as well, so will look to get that scheduled in the coming weeks if people are up for it.

Our experience is that workloads in hyper-scalar data centers such as Google often have significant cold memory. Offloading this to CXL memory devices, backed by cheaper, lower-performance media (e.g. DRAM with hardware compression), can be a practical approach to reduce overall TCO. Page promotion and demotion are then critical for such a tiered memory system.

Quoted 7 lines

A kernel thread to drive hot page collection and promotion seems logical, especially since hot page data from new sources (e.g. CHMU) are collected outside the process execution context and in the form of physical addresses.

I do agree that we need to balance the complexity and benefits of any new data structures for hotness tracking.

I think there is a mismatch in the tiering structure and the patches. If you see the example in memory tiering

/*
* ...
* Example 3:
*
* Node 0 is CPU + DRAM nodes, Node 1 is HBM node, node 2 is PMEM node. *
* node distances:
* node 0 1 2

 *    0  10   20   30
 *    1  20   10   40
 *    2  30   40   10

*
* memory_tiers0 = 1
* memory_tiers1 = 0
* memory_tiers2 = 2
*..
*/

The topmost tier need not be DRAM, patch 3 states

"
[..]
* kpromoted is a kernel thread that runs on each toptier node and
* promotes pages from max_heap.
"

Also, there is no data in the cover letter to indicate what workloads benefit from migration to top-tier and by how much?

Balbir

#26 lore raw
Quoted 122 lines · Balbir Singh
Quoted 91 lines · Wei Xu
Quoted 75 lines · David Rientjes
Quoted 19 lines · Gregory Price
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Bharata B Rao

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) that consolidates memory access information from various sources and enables centralized promotion of hot pages across memory tiers.

Just to be clear, I continue to believe this is a terrible idea and we should not do this. If systems will be built with CXL (and given the horrendous performance, I cannot see why they would be), the kernel should not be migrating memory around like this.

I've been considered this problem from the opposite approach since LSFMM.

Rather than decide how to move stuff around, what if instead we just decide not to ever put certain classes of memory on CXL. Right now, so long as CXL is in the page allocator, it's the wild west - any page can end up anywhere.

Quoted 7 lines

I have enough data now from ZONE_MOVABLE-only CXL deployments on real workloads to show local CXL expansion is valuable and performant enough to be worth deploying - but the key piece for me is that ZONE_MOVABLE disallows GFP_KERNEL. For example: this keeps SLAB meta-data out of CXL, but allows any given user-driven page allocation (including page cache, file, and anon mappings) to land there.

This is similar to our use case, although the direct allocation can be controlled by cpusets or mempolicies as needed depending on the memory access latency required for the workload; nothing new there, though, it's the same argument as NUMA in general and the abstraction of these far memory nodes as separate NUMA nodes makes this very straightforward.

Quoted 7 lines

I'm hoping to share some of this data in the coming months.

I've yet to see any strong indication that a complex hotness/movement system is warranted (yet) - but that may simply be because we have local cards with no switching involved. So far LRU-based promotion and demotion has been sufficient.

To me, this is a key point. As we've discussed in meetings, we're in the early days here. The CHMU does provide a lot of flexibility, both to create very good and very bad hotness trackers. But I think the key point is that we have multiple sources of hotness information depending on the platform and some of these sources only make sense for the kernel (or a BPF offload) to maintain as the source of truth. Some of these sources will be clear-on-read so only one entity would be possible to have as the source of truth of page hotness.

I've been pretty focused on the promotion story here rather than demotion because of how responsive it needs to be. Harvesting the page table accessed bits or waiting on a sliding window through NUMA Balancing (even NUMAB=2) is not as responsive as needed for very fast promotion to top tier memory, hence things like the CHMU (or PEBS or IBS etc).

A few things that I think we need to discuss and align on:

- the kernel as the source of truth for all memory hotness information, which can then be abstracted and used for multiple downstream purposes, memory tiering only being one of them

- the long-term plan for NUMAB=2 and memory tiering support in the kernel in general, are we planning on supporting this through NUMA hint faults forever despite their drawbacks (too slow, too much overhead for KVM)

- the role of the kernel vs userspace in driving the memory migration; lots of discussion on hardware assists that can be leveraged for memory migration but today the balancing is driven in process context. The kthread as the driver of migration is yet to be a sold argument, but are where a number of companies are currently looking

Quoted 7 lines

There's also some feature support that is possible with these CXL memory expansion devices that have started to pop up in labs that can also drastically reduce overall TCO. Perhaps Wei Xu, cc'd, will be able to chime in as well.

This topic seems due for an alignment session as well, so will look to get that scheduled in the coming weeks if people are up for it.

Our experience is that workloads in hyper-scalar data centers such as Google often have significant cold memory. Offloading this to CXL memory devices, backed by cheaper, lower-performance media (e.g. DRAM with hardware compression), can be a practical approach to reduce overall TCO. Page promotion and demotion are then critical for such a tiered memory system.

Quoted 7 lines

A kernel thread to drive hot page collection and promotion seems logical, especially since hot page data from new sources (e.g. CHMU) are collected outside the process execution context and in the form of physical addresses.

I do agree that we need to balance the complexity and benefits of any new data structures for hotness tracking.

I think there is a mismatch in the tiering structure and the patches. If you see the example in memory tiering

/*
* ...
* Example 3:
*
* Node 0 is CPU + DRAM nodes, Node 1 is HBM node, node 2 is PMEM node. *
* node distances:
* node 0 1 2

 *    0  10   20   30
 *    1  20   10   40
 *    2  30   40   10

*
* memory_tiers0 = 1
* memory_tiers1 = 0
* memory_tiers2 = 2
*..
*/

Quoted 6 lines

The topmost tier need not be DRAM, patch 3 states

"
[..]
* kpromoted is a kernel thread that runs on each toptier node and
* promotes pages from max_heap.

That comment is not accurate, will reword it next time.

Currently I am using kthread_create_on_node() to create one kernel thread for each toptier node. I haven't tried this patchset with HBM but it should end up creating a kthread for HBM node too.

However unlike for regular DRAM nodes, the kthread for HBM node can't be bound to any CPU.

Quoted 3 lines

Also, there is no data in the cover letter to indicate what workloads benefit from migration to top-tier and by how much?

I have been trying to get the tracking infrastructure up and hoping to get some review on that. I will start including numbers from the next iteration.

Regards,
Bharata.

#27 lore raw
In reply to Wei Xu
Quoted 84 lines · On Tue, 16 Sep 2025 17:30:46 -0700 Wei Xu <[email protected]> wrote:
Quoted 75 lines · David Rientjes
Quoted 19 lines · Gregory Price
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Bharata B Rao

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) that consolidates memory access information from various sources and enables centralized promotion of hot pages across memory tiers.

Just to be clear, I continue to believe this is a terrible idea and we should not do this. If systems will be built with CXL (and given the horrendous performance, I cannot see why they would be), the kernel should not be migrating memory around like this.

I've been considered this problem from the opposite approach since LSFMM.

Rather than decide how to move stuff around, what if instead we just decide not to ever put certain classes of memory on CXL. Right now, so long as CXL is in the page allocator, it's the wild west - any page can end up anywhere.

Quoted 7 lines

I have enough data now from ZONE_MOVABLE-only CXL deployments on real workloads to show local CXL expansion is valuable and performant enough to be worth deploying - but the key piece for me is that ZONE_MOVABLE disallows GFP_KERNEL. For example: this keeps SLAB meta-data out of CXL, but allows any given user-driven page allocation (including page cache, file, and anon mappings) to land there.

This is similar to our use case, although the direct allocation can be controlled by cpusets or mempolicies as needed depending on the memory access latency required for the workload; nothing new there, though, it's the same argument as NUMA in general and the abstraction of these far memory nodes as separate NUMA nodes makes this very straightforward.

Quoted 7 lines

I'm hoping to share some of this data in the coming months.

I've yet to see any strong indication that a complex hotness/movement system is warranted (yet) - but that may simply be because we have local cards with no switching involved. So far LRU-based promotion and demotion has been sufficient.

To me, this is a key point. As we've discussed in meetings, we're in the early days here. The CHMU does provide a lot of flexibility, both to create very good and very bad hotness trackers. But I think the key point is that we have multiple sources of hotness information depending on the platform and some of these sources only make sense for the kernel (or a BPF offload) to maintain as the source of truth. Some of these sources will be clear-on-read so only one entity would be possible to have as the source of truth of page hotness.

I've been pretty focused on the promotion story here rather than demotion because of how responsive it needs to be. Harvesting the page table accessed bits or waiting on a sliding window through NUMA Balancing (even NUMAB=2) is not as responsive as needed for very fast promotion to top tier memory, hence things like the CHMU (or PEBS or IBS etc).

A few things that I think we need to discuss and align on:

- the kernel as the source of truth for all memory hotness information, which can then be abstracted and used for multiple downstream purposes, memory tiering only being one of them

- the long-term plan for NUMAB=2 and memory tiering support in the kernel in general, are we planning on supporting this through NUMA hint faults forever despite their drawbacks (too slow, too much overhead for KVM)

- the role of the kernel vs userspace in driving the memory migration; lots of discussion on hardware assists that can be leveraged for memory migration but today the balancing is driven in process context. The kthread as the driver of migration is yet to be a sold argument, but are where a number of companies are currently looking

Quoted 7 lines

There's also some feature support that is possible with these CXL memory expansion devices that have started to pop up in labs that can also drastically reduce overall TCO. Perhaps Wei Xu, cc'd, will be able to chime in as well.

This topic seems due for an alignment session as well, so will look to get that scheduled in the coming weeks if people are up for it.

Quoted 6 lines

Our experience is that workloads in hyper-scalar data centers such as Google often have significant cold memory. Offloading this to CXL memory devices, backed by cheaper, lower-performance media (e.g. DRAM with hardware compression), can be a practical approach to reduce overall TCO. Page promotion and demotion are then critical for such a tiered memory system.

For the hardware compression devices how are you dealing with capacity variation / overcommit? Whilst there have been some discussions on that but without a backing store of flash or similar it seems to be challenging to use compressed memory in a tiering system (so as 'normalish' memory) unless you don't mind occasionally and unexpectedly running out of memory (in nasty async ways as dirty cache lines get written back).

Or do you mean zswap type use with a hardware offload of the actual compression?

Quoted 19 lines

A kernel thread to drive hot page collection and promotion seems logical, especially since hot page data from new sources (e.g. CHMU) are collected outside the process execution context and in the form of physical addresses.

I do agree that we need to balance the complexity and benefits of any new data structures for hotness tracking.

Quoted 9 lines
Quoted 9 lines

It seems the closer to random-access the access pattern, the less valuable ANY movement is. Which should be intuitive. But, having CXL beats touching disk every day of the week.

So I've become conflicted on this work - but only because I haven't seen the data to suggest such complexity is warranted.

~Gregory

#28 lore raw
Quoted 38 lines · Jonathan Cameron
Quoted 31 lines · On Tue, 16 Sep 2025 17:30:46 -0700 Wei Xu <[email protected]> wrote:
Quoted 30 lines · David Rientjes
Quoted 26 lines · Gregory Price
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Bharata B Rao

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) that consolidates memory access information from various sources and enables centralized promotion of hot pages across memory tiers.

Just to be clear, I continue to believe this is a terrible idea and we should not do this. If systems will be built with CXL (and given the horrendous performance, I cannot see why they would be), the kernel should not be migrating memory around like this.

I've been considered this problem from the opposite approach since LSFMM.

Rather than decide how to move stuff around, what if instead we just decide not to ever put certain classes of memory on CXL. Right now, so long as CXL is in the page allocator, it's the wild west - any page can end up anywhere.

I have enough data now from ZONE_MOVABLE-only CXL deployments on real workloads to show local CXL expansion is valuable and performant enough to be worth deploying - but the key piece for me is that ZONE_MOVABLE disallows GFP_KERNEL. For example: this keeps SLAB meta-data out of CXL, but allows any given user-driven page allocation (including page cache, file, and anon mappings) to land there.

[snip]

Quoted 15 lines
Quoted 8 lines
Quoted 7 lines

There's also some feature support that is possible with these CXL memory expansion devices that have started to pop up in labs that can also drastically reduce overall TCO. Perhaps Wei Xu, cc'd, will be able to chime in as well.

This topic seems due for an alignment session as well, so will look to get that scheduled in the coming weeks if people are up for it.

Quoted 6 lines

Our experience is that workloads in hyper-scalar data centers such as Google often have significant cold memory. Offloading this to CXL memory devices, backed by cheaper, lower-performance media (e.g. DRAM with hardware compression), can be a practical approach to reduce overall TCO. Page promotion and demotion are then critical for such a tiered memory system.

Quoted 2 lines

For the hardware compression devices how are you dealing with capacity variation / overcommit?

I understand that this is indeed one of the key questions from the upstream kernel’s perspective.
So, I am jumping in to answer w.r.t. what we do in ZeroPoint; obviously I can not speak of other solutions/deployments. However, our HW interface follows existing open specifications from OCP​ [1], so what I am describing below is more widely applicable.

At a very high level, the way our HW works is that the DPA is indeed overcommitted. Then, there is a control plane over CXL.io (PCIe) which exposes the real remaining capacity, as well as some configurable MSI-X interrupts that raise warnings when the capacity crosses over certain configurable thresholds.

Last year I presented this interface in LSF/MM [2]. Based on the feedback I got there, we have an early prototype that acts as the *last* memory tier before reclaim (kind of "compressed tier in lieu of discard" as was suggested to me by Dan).

What is different from standard tiering is that the control plane is checked on demotion to make sure there is still capacity left. If not, the demotion fails. While this seems stable so far, a missing piece is to ensure that this tier is mainly written by demotions and not arbitrary kernel allocations (at least as a starting point). I want to explore how mempolicies can help there, or something of the sort that Gregory described.

This early prototype still needs quite some work in order to find the right abstractions. Hopefully, I will be able to push an RFC in the near future (a couple of months).

Quoted 5 lines

Whilst there have been some discussions on that but without a backing store of flash or similar it seems to be challenging to use compressed memory in a tiering system (so as 'normalish' memory) unless you don't mind occasionally and unexpectedly running out of memory (in nasty async ways as dirty cache lines get written back).

There ​are several things that may be done on the device side. For now, I think the kernel should be unaware of these. But with what I described above, the goal is to have the capacity thresholds configured in a way that we can absorb the occasional dirty cache lines that are written back.

Quoted 3 lines

Or do you mean zswap type use with a hardware offload of the actual compression?

I would categorize this as a completely different discussion (and product line for us).

[1] https://www.opencompute.org/documents/hyperscale-tiered-memory-expander-specification-for-compute-express-link-cxl-1-pdf
[2] https://www.youtube.com/watch?v=tXWEbaJmZ_s

Thanks,
Yiannis

PS: Sending from a personal email address to avoid issues with confidentiality footers of the corporate domain.

reviewer
#29 lore raw
Quoted 3 lines · Yiannis Nikolakopoulos
Quoted 3 lines

For the hardware compression devices how are you dealing with capacity variation / overcommit?

...

Quoted 7 lines

What is different from standard tiering is that the control plane is checked on demotion to make sure there is still capacity left. If not, the demotion fails. While this seems stable so far, a missing piece is to ensure that this tier is mainly written by demotions and not arbitrary kernel allocations (at least as a starting point). I want to explore how mempolicies can help there, or something of the sort that Gregory described.

Writing back the description as i understand it:

1) The intent is to only have this memory allocable via demotion (i.e. no fault or direct allocation from userland possible)

2) The intent is to still have this memory accessible directly (DMA), while compressed, not trigger a fault/promotion on access (i.e. no zswap faults)

3) The intent is to have an external monitoring software handle outrunning run-away decompression/hotness by promoting that data.

So basically we want a zswap-like interface for allocation, but to retain the struct page in page tables such that no faults are incurred on access. Then if the page becomes hot, depend on some kind of HMU tiering system to get it off the device.

I think we all understand there's some bear we have to outrun to deal with problem #3 - and many of us are skeptical that the bear won't catch up with our pants down. Let's ignore this for the moment.

If such a device's memory is added to the default page allocator, then the question becomes one of *isolation* - such that the kernel will provide some "Capital-G Guarantee" that certain NUMA nodes will NEVER be used except under very explicit scenarios.

There are only 3 mechanisms with which to restrict this (presently):

1) ZONE membership (to disallow GFP_KERNEL)
2) cgroups->cpusets->mems_allowed
3) task/vma mempolicy
(obvious #4: Don't put it in the default page allocator)

cpusets and mempolicy are not sufficient to provide full isolation
- cgroups have the opposite hierarchical relationship than desired. The parent cgroup will lock out all children cgroups from using nodes not present in the parent mems_allowed. e.g. if you lock out access from the root cgroup, no cgroup on the entire system is eligible to allocate the memory. If you don't lock out the root cgroup - any root cgroup task is eligible. This isn't tractible.

- task/vma mempolicy gets ignored in many cases and is closer to a suggestion than enforcible. It's also subject to rebinding as a task's cgroups.cpuset.mems_allowed changes.

I haven't read up enough on ZONE_DEVICE to understand the implications of membership there, but have you explored this as an option? I don't see the work i'm doing intersecting well with your efforts - except maybe on the vmscan.c work around allocation on demotion.

The work i'm doing is more aligned with - hey, filesystems are a global resource, why are we using cgroup/task/vma policies to dictate whether a filesystem's cache is eligible to land in remote nodes? i.e. drawing better boundaries and controls around what can land in some set of remote nodes "by default". You're looking for *strong isolation* controls, which implies a different kind of allocator interface.

~Gregory

#30 lore raw

Hi Gregory,

Thanks for all the feedback. I am finally getting some time to come back to this.

Quoted 14 lines · Gregory Price
Quoted 3 lines · Yiannis Nikolakopoulos
Quoted 3 lines

For the hardware compression devices how are you dealing with capacity variation / overcommit?

...

Quoted 7 lines

What is different from standard tiering is that the control plane is checked on demotion to make sure there is still capacity left. If not, the demotion fails. While this seems stable so far, a missing piece is to ensure that this tier is mainly written by demotions and not arbitrary kernel allocations (at least as a starting point). I want to explore how mempolicies can help there, or something of the sort that Gregory described.

Quoted 4 lines

Writing back the description as i understand it:

1) The intent is to only have this memory allocable via demotion (i.e. no fault or direct allocation from userland possible)

Yes that is what looks to me like the "safe" way to begin with. In theory you could have userland apps/middleware that is aware of this memory and its quirks and are ok to use it but I guess we can leave that for later and it feels like it could be provided by a separate driver.

Quoted 4 lines

2) The intent is to still have this memory accessible directly (DMA), while compressed, not trigger a fault/promotion on access (i.e. no zswap faults)

Correct. One of the big advantages of CXL.mem is the cache-line access granularity and our customers don't want to lose that.

Quoted 3 lines

3) The intent is to have an external monitoring software handle outrunning run-away decompression/hotness by promoting that data.

External is not strictly necessary. E.g. it could be an additional source of input to the kpromote/kmigrate solution.

Quoted 2 lines

So basically we want a zswap-like interface for allocation, but to

If by "zswap-like interface" you mean something that can reject the demote (or store according to the zswap semantics) then yes. I just want to be careful when comparing with zswap.

Quoted 3 lines

retain the struct page in page tables such that no faults are incurred on access. Then if the page becomes hot, depend on some kind of HMU tiering system to get it off the device.

Correct.

Quoted 4 lines

I think we all understand there's some bear we have to outrun to deal with problem #3 - and many of us are skeptical that the bear won't catch up with our pants down. Let's ignore this for the moment.

Agreed.

Quoted 21 lines

If such a device's memory is added to the default page allocator, then the question becomes one of *isolation* - such that the kernel will provide some "Capital-G Guarantee" that certain NUMA nodes will NEVER be used except under very explicit scenarios.

There are only 3 mechanisms with which to restrict this (presently):

1) ZONE membership (to disallow GFP_KERNEL)
2) cgroups->cpusets->mems_allowed
3) task/vma mempolicy
(obvious #4: Don't put it in the default page allocator)

cpusets and mempolicy are not sufficient to provide full isolation
- cgroups have the opposite hierarchical relationship than desired. The parent cgroup will lock out all children cgroups from using nodes not present in the parent mems_allowed. e.g. if you lock out access from the root cgroup, no cgroup on the entire system is eligible to allocate the memory. If you don't lock out the root cgroup - any root cgroup task is eligible. This isn't tractible.

Quoted 8 lines

- task/vma mempolicy gets ignored in many cases and is closer to a suggestion than enforcible. It's also subject to rebinding as a task's cgroups.cpuset.mems_allowed changes.

I haven't read up enough on ZONE_DEVICE to understand the implications of membership there, but have you explored this as an option? I don't see the work i'm doing intersecting well with your efforts - except maybe on the vmscan.c work around allocation on demotion.

Thanks for the very helpful breakdown. Your take on #2 & #3 seems reasonable. About #1, I've skimmed through the rest of the thread and I'll continue addressing your responses there.

Yiannis

Quoted 9 lines

The work i'm doing is more aligned with - hey, filesystems are a global resource, why are we using cgroup/task/vma policies to dictate whether a filesystem's cache is eligible to land in remote nodes? i.e. drawing better boundaries and controls around what can land in some set of remote nodes "by default". You're looking for *strong isolation* controls, which implies a different kind of allocator interface.

~Gregory

#31 lore raw
In reply to Yiannis Nikolakopoulos

On Thu, 25 Sep 2025 16:03:46 +0200
Yiannis Nikolakopoulos <[email protected]> wrote:

Hi Yiannis,

Quoted 88 lines
Quoted 36 lines · Jonathan Cameron
Quoted 31 lines · On Tue, 16 Sep 2025 17:30:46 -0700 Wei Xu <[email protected]> wrote:
Quoted 30 lines · David Rientjes
Quoted 26 lines · Gregory Price
Quoted 10 lines · Matthew Wilcox
Quoted 4 lines · Bharata B Rao

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) that consolidates memory access information from various sources and enables centralized promotion of hot pages across memory tiers.

Just to be clear, I continue to believe this is a terrible idea and we should not do this. If systems will be built with CXL (and given the horrendous performance, I cannot see why they would be), the kernel should not be migrating memory around like this.

I've been considered this problem from the opposite approach since LSFMM.

Rather than decide how to move stuff around, what if instead we just decide not to ever put certain classes of memory on CXL. Right now, so long as CXL is in the page allocator, it's the wild west - any page can end up anywhere.

I have enough data now from ZONE_MOVABLE-only CXL deployments on real workloads to show local CXL expansion is valuable and performant enough to be worth deploying - but the key piece for me is that ZONE_MOVABLE disallows GFP_KERNEL. For example: this keeps SLAB meta-data out of CXL, but allows any given user-driven page allocation (including page cache, file, and anon mappings) to land there.

[snip]

Quoted 15 lines
Quoted 8 lines
Quoted 7 lines

There's also some feature support that is possible with these CXL memory expansion devices that have started to pop up in labs that can also drastically reduce overall TCO. Perhaps Wei Xu, cc'd, will be able to chime in as well.

This topic seems due for an alignment session as well, so will look to get that scheduled in the coming weeks if people are up for it.

Quoted 6 lines

Our experience is that workloads in hyper-scalar data centers such as Google often have significant cold memory. Offloading this to CXL memory devices, backed by cheaper, lower-performance media (e.g. DRAM with hardware compression), can be a practical approach to reduce overall TCO. Page promotion and demotion are then critical for such a tiered memory system.

Quoted 2 lines

For the hardware compression devices how are you dealing with capacity variation / overcommit?

I understand that this is indeed one of the key questions from the upstream kernel’s perspective.
So, I am jumping in to answer w.r.t. what we do in ZeroPoint; obviously I can not speak of other solutions/deployments. However, our HW interface follows existing open specifications from OCP​ [1], so what I am describing below is more widely applicable.

At a very high level, the way our HW works is that the DPA is indeed overcommitted. Then, there is a control plane over CXL.io (PCIe) which exposes the real remaining capacity, as well as some configurable MSI-X interrupts that raise warnings when the capacity crosses over certain configurable thresholds.

Last year I presented this interface in LSF/MM [2]. Based on the feedback I got there, we have an early prototype that acts as the *last* memory tier before reclaim (kind of "compressed tier in lieu of discard" as was suggested to me by Dan).

What is different from standard tiering is that the control plane is checked on demotion to make sure there is still capacity left. If not, the demotion fails. While this seems stable so far, a missing piece is to ensure that this tier is mainly written by demotions and not arbitrary kernel allocations (at least as a starting point). I want to explore how mempolicies can help there, or something of the sort that Gregory described.

This early prototype still needs quite some work in order to find the right abstractions. Hopefully, I will be able to push an RFC in the near future (a couple of months).

Quoted 5 lines

Whilst there have been some discussions on that but without a backing store of flash or similar it seems to be challenging to use compressed memory in a tiering system (so as 'normalish' memory) unless you don't mind occasionally and unexpectedly running out of memory (in nasty async ways as dirty cache lines get written back).

Quoted 4 lines

There ​are several things that may be done on the device side. For now, I think the kernel should be unaware of these. But with what I described above, the goal is to have the capacity thresholds configured in a way that we can absorb the occasional dirty cache lines that are written back.

In worst case they are far from occasional. It's not hard to imagine a malicious program that ensures that all L3 in a system (say 256MiB+) is full of cache lines from the far compressed memory all of which are changed in a fashion that makes the allocation much less compressible. If you are doing compression at cache line granularity that's not so bad because it would only be 256MiB margin needed. If the system in question is doing large block side compression, say 4KiB. Then we have a 64x write amplification multiplier. If the virus is streaming over memory the evictions we are seeing at the result of new lines being fetched to be made much less compressible.

Add a accelerator (say DPDK or other zero copy into userspace buffers) into the mix and you have a mess. You'll need to be extremely careful with what goes in this compressed memory or hold enormous buffer capacity against fast changes in compressability.

Key is that all software is potentially malicious (sometimes accidentally so ;)

Now, if we can put this into a special pool where it is acceptable to drop the writes and return poison (so the application crashes) then that may be fine.

Or block writes. Running compressed memory as read only CoW is one way to
avoid this problem.

Quoted 14 lines
Quoted 3 lines

Or do you mean zswap type use with a hardware offload of the actual compression?

I would categorize this as a completely different discussion (and product line for us).

[1] https://www.opencompute.org/documents/hyperscale-tiered-memory-expander-specification-for-compute-express-link-cxl-1-pdf
[2] https://www.youtube.com/watch?v=tXWEbaJmZ_s

Thanks,
Yiannis

PS: Sending from a personal email address to avoid issues with confidentiality footers of the corporate domain.

reviewer
#32 lore raw
Quoted 6 lines · Jonathan Cameron

Now, if we can put this into a special pool where it is acceptable to drop the writes and return poison (so the application crashes) then that may be fine.

Or block writes. Running compressed memory as read only CoW is one way to
avoid this problem.

This is an interesting thought. If you drop a write and return poison, can you instead handle the poison message as a fault and promote on fault? Then you might just be able to turn this whole thing into a zswap backend that promotes on write.

Then you don't particular care about stronger isolation controls (except maybe keeping kernel memory out of those regions).

~Gregory

reviewer
#33 lore raw
Quoted 4 lines · Gregory Price
Quoted 3 lines · Jonathan Cameron

Now, if we can put this into a special pool where it is acceptable to drop the writes and return poison (so the application crashes) then that may be fine.

Quoted 9 lines
Quoted 3 lines

Or block writes. Running compressed memory as read only CoW is one way to
avoid this problem.

This is an interesting thought. If you drop a write and return poison, can you instead handle the poison message as a fault and promote on fault? Then you might just be able to turn this whole thing into a zswap backend that promotes on write.

I just realized this would require some mechanism to re-issue the write.

So yeah, you'd have to do this via some some heroic page table enforcement. The key observation here is that zswap hacks off all the page table entries - rather than leave them present and readable. In this design, you want to leave them present and readable, and therefore need some way to prevent entries from changing out from under you.

Quoted 4 lines

Then you don't particular care about stronger isolation controls (except maybe keeping kernel memory out of those regions).

~Gregory

#34 lore raw
In reply to Gregory Price
Quoted 4 lines · On Thu, 25 Sep 2025 11:08:59 -0400 Gregory Price <[email protected]> wrote:
Quoted 3 lines · Jonathan Cameron

Now, if we can put this into a special pool where it is acceptable to drop the writes and return poison (so the application crashes) then that may be fine.

Quoted 8 lines
Quoted 3 lines

Or block writes. Running compressed memory as read only CoW is one way to
avoid this problem.

This is an interesting thought. If you drop a write and return poison, can you instead handle the poison message as a fault and promote on fault? Then you might just be able to turn this whole thing into a zswap backend that promotes on write.

Poison only comes on subsequent read so you don't see anything at write (which are inherently asynchronous due to cache write back). There are only few ways to do writes that are allowed to fail (the 64 byte atomic deferrable write stuff) and I think on all architectures where they can even be pointed at main memory, they only defer if on uncacheable memory.

Seeing poison on subsequent read is far too late to promote the page, you've lost the data. The poison only works as ultimate safety gate. Also once you've tripped it the device probably needs to drop all write and return poison on all reads, not just the problem one (otherwise things might fail much later).

The CoW thing only works because it's a permissions fault at point of asking for permission to write (so way before it goes into the cache). Then you can check margins to make sure you can still sink all outstanding writes if they become uncompressible and only let the write through if safe
- if not promote some stuff before letting it proceed.
Or you just promote on write and rely on the demotion path performing those careful checks later.

Jonathan

Quoted 5 lines

Then you don't particular care about stronger isolation controls (except maybe keeping kernel memory out of those regions).

~Gregory

reviewer
#35 lore raw
Quoted 8 lines · Jonathan Cameron

The CoW thing only works because it's a permissions fault at point of asking for permission to write (so way before it goes into the cache). Then you can check margins to make sure you can still sink all outstanding writes if they become uncompressible and only let the write through if safe
- if not promote some stuff before letting it proceed.
Or you just promote on write and rely on the demotion path performing those careful checks later.

Agreed. The question is now whether you can actually enforce page table bits not changing. I think you'd need your own fault handling infrastructure / driver for these pages.

This does smell a lot like a kernel-internal dax allocation interface. There was a bunch of talk about virtualizing zswap backends, so that might be a nice place to look to insert this kind of hook.

Then the device driver (which it will definitely need) would have to field page faults accordingly.

It feels much more natural to put this as a zswap/zram backend.

~Gregory

#36 lore raw
Quoted 14 lines · On Thu, 25 Sep 2025 12:06:28 -0400 Gregory Price <[email protected]> wrote:
Quoted 8 lines · Jonathan Cameron

The CoW thing only works because it's a permissions fault at point of asking for permission to write (so way before it goes into the cache). Then you can check margins to make sure you can still sink all outstanding writes if they become uncompressible and only let the write through if safe
- if not promote some stuff before letting it proceed.
Or you just promote on write and rely on the demotion path performing those careful checks later.

Agreed. The question is now whether you can actually enforce page table bits not changing. I think you'd need your own fault handling infrastructure / driver for these pages.

Quoted 9 lines

This does smell a lot like a kernel-internal dax allocation interface. There was a bunch of talk about virtualizing zswap backends, so that might be a nice place to look to insert this kind of hook.

Then the device driver (which it will definitely need) would have to field page faults accordingly.

It feels much more natural to put this as a zswap/zram backend.

Agreed. I currently see two paths that are generic (ish).

1. zswap route - faulting as you describe on writes.
2. Fail safe route - Map compressible memory it into a VM (or application) you don't mind killing if we loose that promotion race due to pathological application. The attacker only disturbs memory allocated to that application / VM so the blast radius is contained.

Jonathan

Quoted 1 line

~Gregory

reviewer
#37 lore raw
Quoted 8 lines · Jonathan Cameron
Quoted 2 lines · On Thu, 25 Sep 2025 12:06:28 -0400 Gregory Price <[email protected]> wrote:

It feels much more natural to put this as a zswap/zram backend.

Agreed. I currently see two paths that are generic (ish).

1. zswap route - faulting as you describe on writes.

aaaaaaaaaaaaaaaaaaaaaaah but therein lies the rub

The interposition point for zswap/zram is the PTE present bit being hacked off to generate access faults.

If you want any random VMA to be eligible for demotion into the tier, then you need to override that VMA's fault/protect hooks in its vm_area_struct. This is idea is a non-starter.

What you'd have to do is have those particular vm_area_struct's be provided by some special allocator that says the memory is eligible for demotion to compressed memory, and to route all faults through it.

That looks a lot like hacking up mm internals to support a single hardware case. Hard to justify.

This may quite literally only be possible to do for for unmapped pages, which would limit the application to things like mm/filemap.c and making IO (read/write) calls faster.

which - hey - maybe that's the best use-case anyway. Have all the read-only compressible filecache you want. At least you avoid touching disk.

~Gregory

reviewer
#38 lore raw
Quoted 6 lines · Gregory Price
Quoted 5 lines · Jonathan Cameron
Quoted 2 lines · On Thu, 25 Sep 2025 12:06:28 -0400 Gregory Price <[email protected]> wrote:

It feels much more natural to put this as a zswap/zram backend.

Quoted 9 lines
Quoted 3 lines

Agreed. I currently see two paths that are generic (ish).

1. zswap route - faulting as you describe on writes.

aaaaaaaaaaaaaaaaaaaaaaah but therein lies the rub

The interposition point for zswap/zram is the PTE present bit being hacked off to generate access faults.

I went digging around a bit.

Not only this, but the PTE is used to store the swap entry ID, so you can't just use a swap backend and keep the mapping. It's just not a compatible abstraction - so as a zswap-backend this is DOA.

Even if you could figure out a way to re-use the abstraction and just take a hard-fault to fault it back in as read-only, you lose the swap entry on fault. That just gets nasty trying to reconcile the differences between this interface and swap at that point.

So here's a fun proposal. I'm not sure of how NUMA nodes for devices get determined -

1. Carve out an explicit proximity domain (NUMA node) for the compressed region via SRAT.
https://docs.kernel.org/driver-api/cxl/platform/acpi/srat.html

2. Make sure this proximity domain (NUMA node) has separate data in the HMAT so it can be an explicit demotion target for higher tiers
https://docs.kernel.org/driver-api/cxl/platform/acpi/hmat.html

3. Create a node-to-zone-allocator registration and retrieval function device_folio_alloc = nid_to_alloc(nid)

4. Create a DAX extension that registers the above allocator interface

5. in alloc_migration_target() mm/migrate.c
Since nid is not a valid buddy-allocator target, everything here will fail. So we can simply append the following to the bottom

   device_folio_alloc = nid_to_alloc(nid, DEVICE_FOLIO_ALLOC);
   if (device_folio_alloc)
       folio = device_folio_alloc(...)
   return folio;

6. in struct migration_target_control add a new .no_writable value
- This will say the new mapping replacements should have the writable bit chopped off.

7. On write-fault, extent mm/memory.c:do_numa_page to detect this and simply promote the page to allow writes. Write faults will be expensive, but you'll have pretty strong guarantees around not unexpectedly running out of space.

You can then loosen the .no_writable restriction with settings if you have high confidence that your system will outrun your ability to promote/evict/whatever if device memory becomes hot.

The only thing I don't know off hand is how shared pages will work in this setup. For VMAs with a mapping that exist at demotion time, this all works wonderfully - less so if the mapping doesn't exist or a new VMA is created after a demotion has occurred.

I don't know what will happen there.

I think this would also sate the desire for a "separate CXL allocator" for integration into other paths as well.

~Gregory

#39 lore raw
Quoted 32 lines · Gregory Price
Quoted 6 lines · Gregory Price
Quoted 5 lines · Jonathan Cameron
Quoted 2 lines · On Thu, 25 Sep 2025 12:06:28 -0400 Gregory Price <[email protected]> wrote:

It feels much more natural to put this as a zswap/zram backend.

Quoted 9 lines
Quoted 3 lines

Agreed. I currently see two paths that are generic (ish).

1. zswap route - faulting as you describe on writes.

aaaaaaaaaaaaaaaaaaaaaaah but therein lies the rub

The interposition point for zswap/zram is the PTE present bit being hacked off to generate access faults.

I went digging around a bit.

Not only this, but the PTE is used to store the swap entry ID, so you can't just use a swap backend and keep the mapping. It's just not a compatible abstraction - so as a zswap-backend this is DOA.

Even if you could figure out a way to re-use the abstraction and just take a hard-fault to fault it back in as read-only, you lose the swap entry on fault. That just gets nasty trying to reconcile the differences between this interface and swap at that point.

So here's a fun proposal. I'm not sure of how NUMA nodes for devices get determined -

Quoted 7 lines

1. Carve out an explicit proximity domain (NUMA node) for the compressed region via SRAT.
https://docs.kernel.org/driver-api/cxl/platform/acpi/srat.html

2. Make sure this proximity domain (NUMA node) has separate data in the HMAT so it can be an explicit demotion target for higher tiers
https://docs.kernel.org/driver-api/cxl/platform/acpi/hmat.html

This makes sense. I've done a dirty hardcoding trick in my prototype so that my node is always the last target. I'll have a look on how to make this right.

Quoted 6 lines

3. Create a node-to-zone-allocator registration and retrieval function device_folio_alloc = nid_to_alloc(nid)

4. Create a DAX extension that registers the above allocator interface

Quoted 8 lines

5. in alloc_migration_target() mm/migrate.c
Since nid is not a valid buddy-allocator target, everything here will fail. So we can simply append the following to the bottom

   device_folio_alloc = nid_to_alloc(nid, DEVICE_FOLIO_ALLOC);
   if (device_folio_alloc)
       folio = device_folio_alloc(...)
   return folio;

In my current prototype alloc_migration_target was working (naively). Steps 3, 4 and 5 seem like an interesting thing to try after all this discussion.

Quoted 5 lines

6. in struct migration_target_control add a new .no_writable value
- This will say the new mapping replacements should have the writable bit chopped off.

Quoted 8 lines

7. On write-fault, extent mm/memory.c:do_numa_page to detect this and simply promote the page to allow writes. Write faults will be expensive, but you'll have pretty strong guarantees around not unexpectedly running out of space.

You can then loosen the .no_writable restriction with settings if you have high confidence that your system will outrun your ability to promote/evict/whatever if device memory becomes hot.

That looks modular enough that will allow me to test both writable and no_writable and being able to compare.

Quoted 5 lines

The only thing I don't know off hand is how shared pages will work in this setup. For VMAs with a mapping that exist at demotion time, this all works wonderfully - less so if the mapping doesn't exist or a new VMA is created after a demotion has occurred.

I'll keep that in mind.

Quoted 7 lines

I don't know what will happen there.

I think this would also sate the desire for a "separate CXL allocator" for integration into other paths as well.

~Gregory

Thanks a lot for all the discussion and the input. I can move my prototype towards this direction and will get back with what I 've learned and an RFC if it makes sense. Please keep me in the loop in any related discussions.

Best,
/Yiannis

reviewer
#40 lore raw
Quoted 11 lines · Yiannis Nikolakopoulos
Quoted 7 lines · Gregory Price

1. Carve out an explicit proximity domain (NUMA node) for the compressed region via SRAT.
https://docs.kernel.org/driver-api/cxl/platform/acpi/srat.html

2. Make sure this proximity domain (NUMA node) has separate data in the HMAT so it can be an explicit demotion target for higher tiers
https://docs.kernel.org/driver-api/cxl/platform/acpi/hmat.html

This makes sense. I've done a dirty hardcoding trick in my prototype so that my node is always the last target. I'll have a look on how to make this right.

I think it's probably a CEDT/CDAT/HMAT/SRAT/etc negotiation.

Essentially the platform needs to allow a single device to expose multiple numa nodes based on different expected performance. From those ranges. Then software needs to program the HDM decoders appropriately.

Quoted 4 lines
Quoted 4 lines

5. in alloc_migration_target() mm/migrate.c
Since nid is not a valid buddy-allocator target, everything here will fail. So we can simply append the following to the bottom

Quoted 8 lines
Quoted 4 lines
   device_folio_alloc = nid_to_alloc(nid, DEVICE_FOLIO_ALLOC);
   if (device_folio_alloc)
       folio = device_folio_alloc(...)
   return folio;

In my current prototype alloc_migration_target was working (naively). Steps 3, 4 and 5 seem like an interesting thing to try after all this discussion.

Quoted 1 line

Right because the memory is directly accessible to the buddy allocator. What i'm proposing would remove this memory from the buddy allocator and force more explicit integration (in this case with this function).

more explicitly: in this design __folio_alloc can never access this
memory.

~Gregory

#41 lore raw
Quoted 13 lines · On Fri, 17 Oct 2025 10:15:57 -0400 Gregory Price <[email protected]> wrote:
Quoted 11 lines · Yiannis Nikolakopoulos
Quoted 7 lines · Gregory Price

1. Carve out an explicit proximity domain (NUMA node) for the compressed region via SRAT.
https://docs.kernel.org/driver-api/cxl/platform/acpi/srat.html

2. Make sure this proximity domain (NUMA node) has separate data in the HMAT so it can be an explicit demotion target for higher tiers
https://docs.kernel.org/driver-api/cxl/platform/acpi/hmat.html

This makes sense. I've done a dirty hardcoding trick in my prototype so that my node is always the last target. I'll have a look on how to make this right.

Quoted 6 lines

I think it's probably a CEDT/CDAT/HMAT/SRAT/etc negotiation.

Essentially the platform needs to allow a single device to expose multiple numa nodes based on different expected performance. From those ranges. Then software needs to program the HDM decoders appropriately.

It's a bit 'fuzzy' to justify but maybe (for CXL) a CFWMS flag (so CEDT as you mention) to say this host memory region may be backed by compressed memory?

Might be able to justify it from spec point of view by arguing that compression is a QoS related characteristic. Always possible host hardware will want to handle it differently before it even hits the bus even if it's just a case throttling writing differently.

That then ends up in it's own NUMA node. Whether we take on the splitting CFMWS entries into multiple NUMA nodes depending on what backing devices end up in them is something we kicked into the long grass originally, but that can definitely be revisited. That doesn't matter for initial support of compressed memory though if we can do it via a seperate CXL Fixed Memory Window Structure (CFMWS) in CEDT.

Quoted 22 lines
Quoted 5 lines
Quoted 5 lines

5. in alloc_migration_target() mm/migrate.c
Since nid is not a valid buddy-allocator target, everything here will fail. So we can simply append the following to the bottom

Quoted 8 lines
Quoted 4 lines
   device_folio_alloc = nid_to_alloc(nid, DEVICE_FOLIO_ALLOC);
   if (device_folio_alloc)
       folio = device_folio_alloc(...)
   return folio;  

In my current prototype alloc_migration_target was working (naively). Steps 3, 4 and 5 seem like an interesting thing to try after all this discussion.

Quoted 1 line

Right because the memory is directly accessible to the buddy allocator. What i'm proposing would remove this memory from the buddy allocator and force more explicit integration (in this case with this function).

more explicitly: in this design __folio_alloc can never access this
memory.

~Gregory

reviewer
#42 lore raw
Quoted 8 lines · Jonathan Cameron
Quoted 5 lines · On Fri, 17 Oct 2025 10:15:57 -0400 Gregory Price <[email protected]> wrote:

Essentially the platform needs to allow a single device to expose multiple numa nodes based on different expected performance. From those ranges. Then software needs to program the HDM decoders appropriately.

Quoted 9 lines

It's a bit 'fuzzy' to justify but maybe (for CXL) a CFWMS flag (so CEDT as you mention) to say this host memory region may be backed by compressed memory?

Might be able to justify it from spec point of view by arguing that compression is a QoS related characteristic. Always possible host hardware will want to handle it differently before it even hits the bus even if it's just a case throttling writing differently.

That's a Consortium discussion to have (and I am not of the consortium :P), but yeah you could do it that way.

More generally could have a "Not-for-general-consumption bit" instead of specifically a compressed bit. Maybe both a "No-Consume" and a "Special Node" bit would be useful separately.

Of course then platforms need to be made to understand all these:

"No-Consume" -> force EFI_MEMORY_SP or leave it reserved "Special Node" -> allocate its own PXM / Provide discrete CFMWS

Naming obviously non-instructive here, may as well call them Nancy and Bob bits.

Quoted 8 lines

That then ends up in it's own NUMA node. Whether we take on the splitting CFMWS entries into multiple NUMA nodes depending on what backing devices end up in them is something we kicked into the long grass originally, but that can definitely be revisited. That doesn't matter for initial support of compressed memory though if we can do it via a seperate CXL Fixed Memory Window Structure (CFMWS) in CEDT.

This is the way I would initially approach it tbh - but i'm also not a hardware/firmware person, so i don't know exactly what bits a device would set to tell BIOS/EFI "Hey, give this chunk its own CFMWS", or if that lies solely with BIOS/EFI.

~Gregory

#43 lore raw
Quoted 13 lines · On Fri, 17 Oct 2025 10:59:01 -0400 Gregory Price <[email protected]> wrote:
Quoted 12 lines · Jonathan Cameron
Quoted 5 lines · On Fri, 17 Oct 2025 10:15:57 -0400 Gregory Price <[email protected]> wrote:

Essentially the platform needs to allow a single device to expose multiple numa nodes based on different expected performance. From those ranges. Then software needs to program the HDM decoders appropriately.

It's a bit 'fuzzy' to justify but maybe (for CXL) a CFWMS flag (so CEDT as you mention) to say this host memory region may be backed by compressed memory?

Quoted 8 lines
Quoted 5 lines

Might be able to justify it from spec point of view by arguing that compression is a QoS related characteristic. Always possible host hardware will want to handle it differently before it even hits the bus even if it's just a case throttling writing differently.

That's a Consortium discussion to have (and I am not of the consortium :P), but yeah you could do it that way.

The moment I know it's raised there I (and others involved in consortium) can't talk about it in public. (I love standards org IP rules!) So it's useful to have a pre discussion before that happens. We've done this before for other topics and it can be very productive.

Quoted 5 lines

More generally could have a "Not-for-general-consumption bit" instead of specifically a compressed bit. Maybe both a "No-Consume" and a "Special Node" bit would be useful separately.

Quoted 7 lines

Of course then platforms need to be made to understand all these:

"No-Consume" -> force EFI_MEMORY_SP or leave it reserved "Special Node" -> allocate its own PXM / Provide discrete CFMWS

Naming obviously non-instructive here, may as well call them Nancy and Bob bits.

For compression specifically I think there is value in making it explicitly compression because the host hardware might handle that differently. The other bits might be worth having as well though. SPM was all about 'you could' use it as normal memory but someone put it there for something else. This more a case of SPOM. Specific Purpose Only Memory - eats babies if you don't know the extra rules for each instance of that.

Quoted 10 lines
Quoted 9 lines

That then ends up in it's own NUMA node. Whether we take on the splitting CFMWS entries into multiple NUMA nodes depending on what backing devices end up in them is something we kicked into the long grass originally, but that can definitely be revisited. That doesn't matter for initial support of compressed memory though if we can do it via a seperate CXL Fixed Memory Window Structure (CFMWS) in CEDT.

Quoted 4 lines

This is the way I would initially approach it tbh - but i'm also not a hardware/firmware person, so i don't know exactly what bits a device would set to tell BIOS/EFI "Hey, give this chunk its own CFMWS", or if that lies solely with BIOS/EFI.

It's not a device thing wrt to nodes today (and there are good reasons why it should not be at that granularity e.g. node explosion has costs). The BIOS might pre setup the decoders and even lock them, but I'd expect we'll move away from that to fully OS managed over time (to get flexibility)
- exception to that being when confidential compute is making its usual mess of things.

Maybe the BIOS would have a look at devices and decide to enable a compressed memory CFMWS if it finds devices that need it and not do so otherwise, though not doing so breaks hotplug of compressed memory devices.

So my guess is either we need to fix Linux to allow splitting a fixed memory window up into multiple NUMA nodes, or platforms have to spin extra fixed memory windows (host side PA ranges with a NUMA node for each).

Which option depends a bit on whether we expect host hardware to either handle compressed differently from normal ram, or at least separate it for QoS reasons.

What fun.

J

Quoted 2 lines

~Gregory

reviewer
#44 lore raw
Quoted 12 lines · Jonathan Cameron
Quoted 11 lines

More generally could have a "Not-for-general-consumption bit" instead of specifically a compressed bit. Maybe both a "No-Consume" and a "Special Node" bit would be useful separately.

Of course then platforms need to be made to understand all these:

"No-Consume" -> force EFI_MEMORY_SP or leave it reserved "Special Node" -> allocate its own PXM / Provide discrete CFMWS

Naming obviously non-instructive here, may as well call them Nancy and Bob bits.

Quoted 8 lines

For compression specifically I think there is value in making it explicitly compression because the host hardware might handle that differently. The other bits might be worth having as well though. SPM was all about 'you could' use it as normal memory but someone put it there for something else. This more a case of SPOM. Specific Purpose Only Memory - eats babies if you don't know the extra rules for each instance of that.

This is a fair point. Something like a SPOM bit that says some other bit-field is valid and you get to add new extensions about how the memory should be used? :shrug: probably sufficiently extensible but maybe never used for anything more than compression.

Quoted 8 lines

Maybe the BIOS would have a look at devices and decide to enable a compressed memory CFMWS if it finds devices that need it and not do so otherwise, though not doing so breaks hotplug of compressed memory devices.

So my guess is either we need to fix Linux to allow splitting a fixed memory window up into multiple NUMA nodes, or platforms have to spin extra fixed memory windows (host side PA ranges with a NUMA node for each).

I don't think splitting a CFMW into multiple nodes is feasible, but I also haven't looked at that region of ACPI code since i finished the docs. I can look into that.

I would prefer the former, since this is already what's done for hostbridge interleave vs non-interleave setups, where the host may expose multiple CFMW for the same devices depending on how the OS.

Quoted 4 lines

Which option depends a bit on whether we expect host hardware to either handle compressed differently from normal ram, or at least separate it for QoS reasons.

There's only a handful of folks discussing this at the moment, but so far we've all be consistent in our gut telling us it should be handled differently for reliability reasons. But also, more opinions always welcome :]

~Gregory

reviewer
#45 lore raw
Quoted 3 lines · Gregory Price

I would prefer the former, since this is already what's done for hostbridge interleave vs non-interleave setups, where the host may expose multiple CFMW for the same devices depending on how the OS.

bah, got distracted

"Depending on how the OS may choose to configure things at some unknown point in the future"

~Gregory

#46 lore raw
Quoted 9 lines · On Tue, 21 Oct 2025 14:57:26 -0400 Gregory Price <[email protected]> wrote:
Quoted 3 lines · Gregory Price

I would prefer the former, since this is already what's done for hostbridge interleave vs non-interleave setups, where the host may expose multiple CFMW for the same devices depending on how the OS.

bah, got distracted

"Depending on how the OS may choose to configure things at some unknown point in the future"

My gut feeling is the need to do dynamic NUMA nodes will not be driven but this but more by large scale fabrics (if that ever happens) and trade offs of host PA space vs QoS in the hardware. Those trade offs might put memory with very different performance characteristics behind one window.

Maybe it'll become a thing that can be used for compression. Otherwise compression from host hardware point of view might be like the question of share or separate fixed memory windows for persistent / volatile. Ideally they'd be separate but if Host PA space is limited, someone might build a system where a single fixed memory window is used to support both.

Possible virtualization of some of this stuff will make it more complex again. Any crazy mess can share a fake fixed memory window as the QoS is all behind some page tables.

Meh. Let's suggest people burn host PA space for now. If anyone hits that limit they can solve it (crosses fingers it's not my lot :)

Jonathan

Quoted 3 lines

~Gregory

reviewer
#47 lore raw
Quoted 7 lines · Jonathan Cameron

My gut feeling is the need to do dynamic NUMA nodes will not be driven but this but more by large scale fabrics (if that ever happens) and trade offs of host PA space vs QoS in the hardware. Those trade offs might put memory with very different performance characteristics behind one window.

I can't believe we live in a world where "We have to think about the scenario where we actually need all 256 TB of 48-bit phys-addressing" is not a tongue in cheek joke o_o

That's a paltry 2048 128GB DIMMs... and whatever monstrosity you have to build to host it all but that's at least a fun engineering problem :V

Bring on the 128-bit CPUs!

What do we name those x86 registers though? Slap the E back on for ERAX?

Quoted 3 lines

Meh. Let's suggest people burn host PA space for now. If anyone hits that limit they can solve it (crosses fingers it's not my lot :)

+1

~Gregory

#48 lore raw
Quoted 9 lines · On Wed, 22 Oct 2025 11:05:16 -0400 Gregory Price <[email protected]> wrote:
Quoted 7 lines · Jonathan Cameron

My gut feeling is the need to do dynamic NUMA nodes will not be driven but this but more by large scale fabrics (if that ever happens) and trade offs of host PA space vs QoS in the hardware. Those trade offs might put memory with very different performance characteristics behind one window.

Quoted 3 lines

I can't believe we live in a world where "We have to think about the scenario where we actually need all 256 TB of 48-bit phys-addressing" is not a tongue in cheek joke o_o

You think everyone wires all the 48 bits? Certainly not everyone does.

Quoted 15 lines

That's a paltry 2048 128GB DIMMs... and whatever monstrosity you have to build to host it all but that's at least a fun engineering problem :V

Bring on the 128-bit CPUs!

What do we name those x86 registers though? Slap the E back on for ERAX?

Quoted 3 lines

Meh. Let's suggest people burn host PA space for now. If anyone hits that limit they can solve it (crosses fingers it's not my lot :)

+1

~Gregory

#49 lore raw
In reply to Jonathan Cameron
Quoted 5 lines · Jonathan Cameron

On Thu, 25 Sep 2025 16:03:46 +0200
Yiannis Nikolakopoulos <[email protected]> wrote:

Hi Yiannis,

Hi Jonathan! Thanks for your response!

[snip]

Quoted 6 lines
Quoted 4 lines

There are several things that may be done on the device side. For now, I think the kernel should be unaware of these. But with what I described above, the goal is to have the capacity thresholds configured in a way that we can absorb the occasional dirty cache lines that are written back.

In worst case they are far from occasional. It's not hard to imagine a malicious

This is correct. Any simplification on my end is mainly based on the empirical evidence of the use cases we are testing for (tiering). But I fully respect that we need to be proactive and assume the worst case scenario.

Quoted 6 lines

program that ensures that all L3 in a system (say 256MiB+) is full of cache lines from the far compressed memory all of which are changed in a fashion that makes the allocation much less compressible. If you are doing compression at cache line granularity that's not so bad because it would only be 256MiB margin needed. If the system in question is doing large block side compression, say 4KiB. Then we have a 64x write amplification multiplier. If the virus is streaming over

This is insightful indeed :). However, even in the case of the 64x amplification, you implicitly assume that each of the cachelines in the L3 belongs to a different page. But then one cache-line would not deteriorate the compressed size of the entire page that much (the bandwidth amplification on the device is a different -performance- story). So even in the 4K case the two ends of the spectrum are to either have big amplification with low compression ratio impact, or small amplification with higher compression ratio impact. Another practical assumption here, is that the different HMU mechanisms would help promote the contended pages before this becomes a big issue. Which of course might still not be enough on the malicious streaming writes workload.
Overall, I understand these are heuristics and I do see your point that this needs to be robust even for the maliciously behaving programs.

Quoted 5 lines

memory the evictions we are seeing at the result of new lines being fetched to be made much less compressible.

Add a accelerator (say DPDK or other zero copy into userspace buffers) into the mix and you have a mess. You'll need to be extremely careful with what goes

Good point about the zero copy stuff.

Quoted 2 lines

in this compressed memory or hold enormous buffer capacity against fast changes in compressability.

To my experience the factor of buffer capacity would be closer to the benefit that you get from the compression (e.g. 2x the cache size in your example).
But I understand the burden of proof is on our end. As we move further with this I will try to provide data as well.

Quoted 8 lines

Key is that all software is potentially malicious (sometimes accidentally so ;)

Now, if we can put this into a special pool where it is acceptable to drop the writes and return poison (so the application crashes) then that may be fine.

Or block writes. Running compressed memory as read only CoW is one way to
avoid this problem.

These could be good starting points, as I see in the rest of the thread.

Thanks,
Yiannis

#50 lore raw
Quoted 9 lines · On Thu, 16 Oct 2025 18:16:31 +0200 Yiannis Nikolakopoulos…
Quoted 5 lines · Jonathan Cameron

On Thu, 25 Sep 2025 16:03:46 +0200
Yiannis Nikolakopoulos <[email protected]> wrote:

Hi Yiannis,

Hi Jonathan! Thanks for your response!

Hi Yiannis,

This is way more fun than doing real work ;)

Quoted 17 lines

[snip]

Quoted 6 lines
Quoted 4 lines

There are several things that may be done on the device side. For now, I think the kernel should be unaware of these. But with what I described above, the goal is to have the capacity thresholds configured in a way that we can absorb the occasional dirty cache lines that are written back.

In worst case they are far from occasional. It's not hard to imagine a malicious

This is correct. Any simplification on my end is mainly based on the empirical evidence of the use cases we are testing for (tiering). But I fully respect that we need to be proactive and assume the worst case scenario.

Quoted 6 lines

program that ensures that all L3 in a system (say 256MiB+) is full of cache lines from the far compressed memory all of which are changed in a fashion that makes the allocation much less compressible. If you are doing compression at cache line granularity that's not so bad because it would only be 256MiB margin needed. If the system in question is doing large block side compression, say 4KiB. Then we have a 64x write amplification multiplier. If the virus is streaming over

Quoted 6 lines

This is insightful indeed :). However, even in the case of the 64x amplification, you implicitly assume that each of the cachelines in the L3 belongs to a different page. But then one cache-line would not deteriorate the compressed size of the entire page that much (the bandwidth amplification on the device is a different -performance- story).

This is putting limits on what compression algorithm is used. We could do that but then we'd have to never support anything different. Maybe if the device itself provided the worse case amplification numbers that would do Any device that gets this wrong is buggy - but it might be hard to detect that if people don't publish their compression algs and the proofs of worst case blow up of compression blocks.

I guess we could do the maths on what the device manufacturer says and if we don't believe them or they haven't provided enough info to check, double it :)

Quoted 7 lines

So even in the 4K case the two ends of the spectrum are to either have big amplification with low compression ratio impact, or small amplification with higher compression ratio impact. Another practical assumption here, is that the different HMU mechanisms would help promote the contended pages before this becomes a big issue. Which of course might still not be enough on the malicious streaming writes workload.

Using promotion to get you out of this is a non starter unless you have a backstop because we'll have annoying things like pinning going on or bandwidth bottlenecks at the promotion target.
Promotion might massively reduce the performance impact of course under normal conditions.

Quoted 8 lines

Overall, I understand these are heuristics and I do see your point that this needs to be robust even for the maliciously behaving programs.

Quoted 5 lines

memory the evictions we are seeing at the result of new lines being fetched to be made much less compressible.

Add a accelerator (say DPDK or other zero copy into userspace buffers) into the mix and you have a mess. You'll need to be extremely careful with what goes

Quoted 8 lines

Good point about the zero copy stuff.

Quoted 2 lines

in this compressed memory or hold enormous buffer capacity against fast changes in compressability.

To my experience the factor of buffer capacity would be closer to the benefit that you get from the compression (e.g. 2x the cache size in your example).
But I understand the burden of proof is on our end. As we move further with this I will try to provide data as well.

If we are aiming for generality the nasty problem is that either we have to write rules on what Linux will cope with, or design it to cope with the worse possible implementation :(

I can think of lots of plausible sounding cases that have horrendous multiplication factors if done in a naive fashion.
* De-duplication
* Metadata flag for all 0s
* Some general purpose compression algs are very vulnerable to the tails of the probability distributions. Some will flip between multiple modes with very different characteristics, perhaps to meet latency guarantees.

Would be fun to ask an information theorist / compression expert to lay out an algorithm with the worst possible tail performance but with good average.

Quoted 10 lines
Quoted 8 lines

Key is that all software is potentially malicious (sometimes accidentally so ;)

Now, if we can put this into a special pool where it is acceptable to drop the writes and return poison (so the application crashes) then that may be fine.

Or block writes. Running compressed memory as read only CoW is one way to
avoid this problem.

These could be good starting points, as I see in the rest of the thread.

Fun problems. Maybe we start with very conservative handling and then argue for relaxations later.

Jonathan

Quoted 2 lines

Thanks,
Yiannis

reviewer
#51 lore raw
Quoted 8 lines · Jonathan Cameron
Quoted 2 lines · On Thu, 16 Oct 2025 18:16:31 +0200 Yiannis Nikolakopoulos…

These could be good starting points, as I see in the rest of the thread.

Fun problems. Maybe we start with very conservative handling and then argue for relaxations later.

Not to pile on, but if we can't even manage the conservative handling due to other design issues - then it doesn't bode well for the rest. So getting that right should be the priority - not a maybe.

~Gregory

#52 lore raw
In reply to David Rientjes

Hi David,

Quoted 6 lines · David Rientjes

I've been pretty focused on the promotion story here rather than demotion because of how responsive it needs to be. Harvesting the page table accessed bits or waiting on a sliding window through NUMA Balancing (even NUMAB=2) is not as responsive as needed for very fast promotion to top tier memory, hence things like the CHMU (or PEBS or IBS etc).

First, thanks for sharing your thoughts on the promotion responsiveness challenges, definitely a critical aspect for tiering strategies.

We recently put together a preliminary report using our experimental HW that I believe could be relevant to the ongoing discussions:
A Limits Study of Memory-side Tiering Telemetry:
https://arxiv.org/abs/2508.09351

It's essentially an initial step toward quantifying the benefits of HMU on the memory side, aiming to compare promotion quality (e.g., hotness coverage and accuracy) across HMU, PEBS-based promotion, and NUMA balancing (promotion path).
Hopefully, this kind of work can help us better understand some of the trade-offs being discussed, support more data-driven comparisons, and spark more fruitful discussions...

Best,
Vinicius