Skip to contentSkip to conversation

mm: Hot page tracking and promotion infrastructure

RFC PATCH Superseded Bharata B Rao 4 participants 27 messages Mar 23 – Apr 30

Subsystems (11)

v6 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v6 cover mm: Hot page tracking and promotion infrastructure

Hi,

This is v6 of pghot, a hot-page tracking and promotion subsystem. The main changes in this version are to retain only hintfault as source and a lot of cleanups, fixes and restructuring.

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) with the following goals:

- Unify hot page detection from multiple sources like hint faults, page table scans, hardware hints (AMD IBS).
- Decouple detection from migration.
- Centralize promotion logic via per-lower-tier-node kmigrated kernel thread.
- Move promotion rate‑limiting and related logic used by numa_balancing=2 (NUMAB2, the current NUMA balancing–based promotion) from the scheduler to pghot for broader reuse.

Currently, multiple kernel subsystems detect page accesses independently. This patchset consolidates accesses from these mechanisms by providing:

- A common API for reporting page accesses.
- Shared infrastructure for tracking hotness at PFN granularity.
- Per-lower-tier-node kernel threads for promoting pages.

Here is a brief summary of how this subsystem works:

- Tracks frequency and last access time.
- Additionally, the accessing NUMA node ID (NID) for each recorded access is also tracked in the precision mode.
- These hotness parameters are maintained in a per-PFN hotness record within the existing mem_section data structure.
- In default mode, one byte (u8) is used for hotness record. 5 bits are used to store time and bucketing scheme is used to represent a total access time up to 4s with HZ=1000. Default toptier NID (0) is used as the target for promotion which can be changed via debugfs tunable.
- In precision mode, 4 bytes (u32) are used for each hotness record. 14 bits are used to store time which can represent around 16s with HZ=1000.
- Classifies pages as hot based on configurable thresholds.
- Pages classified as hot are marked as ready for migration using the ready bit. Both modes use MSB of the hotness record as ready bit.
- Per-lower-tier-node kmigrated threads periodically scan the PFNs of lower-tier nodes, checking for the migration-ready bit to perform batched migrations. Interval between successive scans and batching value are configurable via debugfs tunables.

Memory overhead
---------------
Default mode: 1 byte per lower-tier PFN. For a 1TB lower-tier memory this amounts to 256MB overhead (assuming 4K pages)

Precision mode: 4 bytes per lower-tier PFN. For a 1TB of lower memory this amounts to 1G overhead.

Bit layout of hotness record
----------------------------
Default mode
- Bits 0-1: Frequency (2bits, 4 access samples)
- Bits 2-6: Bucketed time (5bits, up to 4s with HZ=1000)
- Bit 7: Migration ready bit

Precision mode
- Bits 0-9: Target NID (10 bits)
- Bits 10-12: Frequency (3bits, 8 access samples)
- Bits 13-26: Time (14bits, up to 16s with HZ=1000)
- Bits 27-30: Reserved
- Bit 31: Migration ready bit

Potential hotness sources
-------------------------
1. NUMA Balancing (NUMAB2, Tiering mode)
2. IBS - Instruction Based Sampling, hardware based sampling mechanism present on AMD CPUs.
3. klruscand - PTE‑A bit scanning built on MGLRU’s walk helpers.
4. folio_mark_accessed() - Page cache access tracking (unmapped page cache pages)

Changes in v6
=============
- While earlier versions included sample implementation for all the hotness sources listed above, I have retained only NUMAB2 in this iteration and will include the hardened versions of them in subsequent iterations.
- Cleaned up NUMAB2 implementation by removing the unused code (like access time tracking code etc).
- Ensured that NUMAB1 mode works as before. (Earlier versions made NUMA hint fault handler work only for NUMAB2 version)
- NUMA Balancing tiering mode is moved to its own new config CONFIG_NUMA_BALANCING_TIERING to make code sharing between NUMA Balancing and pghot easier.
- A bunch of hot page promotion related stats now depend upon CONFIG_PGHOT as the promotion engine is part of pghot.
- Fixed kmigrated to take a reference on the folio when walking the PFNs checking for migrate-ready folios.
- Fixed speculative folio access issue reported by Chris Mason's review-prompts.
- Added per-memcg NUMA_PAGE_MIGRATE stats accounting for batch migration API too.
- Added support for initializing hot_maps in the newly added sections during memory hotplug.
- Default hotness threshold window changed from 4s to 3s as the maximum time representable in default mode is 3.9s only.
- Lot of cleanups and code restructuring.

Results
=======
Posted as replies to this mail thread.

This v6 patchset applies on top of upstream commit a989fde763f4f and can be fetched from:

https://github.com/AMDESE/linux-mm/tree/bharata/pghot-rfcv6

v5: https://lore.kernel.org/linux-mm/[email protected]/
v4: https://lore.kernel.org/linux-mm/[email protected]/
v3: https://lore.kernel.org/linux-mm/[email protected]/
v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/
v0: https://lore.kernel.org/linux-mm/[email protected]/

Bharata B Rao (4):
mm: migrate: Allow misplaced migration without VMA
mm: Hot page tracking and promotion - pghot
mm: pghot: Precision mode for pghot
mm: sched: move NUMA balancing tiering promotion to pghot

Gregory Price (1):
mm: migrate: Add migrate_misplaced_folios_batch()

 Documentation/admin-guide/mm/pghot.txt |  80 ++++
 include/linux/migrate.h                |  10 +-
 include/linux/mm.h                     |  35 +-
 include/linux/mmzone.h                 |  24 +-
 include/linux/pghot.h                  | 113 +++++
 include/linux/vm_event_item.h          |   5 +
 init/Kconfig                           |  13 +
 kernel/sched/core.c                    |   7 +
 kernel/sched/debug.c                   |   1 -
 kernel/sched/fair.c                    | 177 +------
 kernel/sched/sched.h                   |   1 -
 mm/Kconfig                             |  25 +
 mm/Makefile                            |   6 +
 mm/huge_memory.c                       |  27 +-
 mm/memcontrol.c                        |   6 +-
 mm/memory-tiers.c                      |  15 +-
 mm/memory.c                            |  36 +-
 mm/mempolicy.c                         |   3 -
 mm/migrate.c                           |  88 +++-
 mm/mm_init.c                           |  10 +
 mm/pghot-default.c                     |  79 ++++
 mm/pghot-precise.c                     |  81 ++++
 mm/pghot-tunables.c                    | 182 ++++++++
 mm/pghot.c                             | 618 +++++++++++++++++++++++++
 mm/vmstat.c                            |   7 +-
 25 files changed, 1411 insertions(+), 238 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/pghot.txt
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot-precise.c
 create mode 100644 mm/pghot-tunables.c
 create mode 100644 mm/pghot.c

base-commit: a989fde763f4f24209e4702f50a45be572340e68

Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH v6 1/5 mm: migrate: Allow misplaced migration without VMA

We want isolation of misplaced folios to work in contexts where VMA isn't available, typically when performing migrations from a kernel thread context. In order to prepare for that, allow migrate_misplaced_folio_prepare() to be called with a NULL VMA.

When migrate_misplaced_folio_prepare() is called with non-NULL VMA, it will check if the folio is mapped shared and that requires holding PTL lock. This path isn't taken when the function is invoked with NULL VMA (migration outside of process context). Therefore, when VMA == NULL, migrate_misplaced_folio_prepare() does not require the caller to hold the PTL.

---
 mm/migrate.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
mm/migrate.c +7 -2
index 2c3d489ecf51..a15184950e65 100644
@@ -2652,7 +2652,12 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,
 
 /*
  * Prepare for calling migrate_misplaced_folio() by isolating the folio if
- * permitted. Must be called with the PTL still held.
+ * permitted. Must be called with the PTL still held if called with a non-NULL
+ * vma.
+ *
+ * When called with a NULL vma (e.g., kernel thread initiated migration),
+ * migrate_misplaced_folio_prepare() will allow shared executable folios
+ * to be migrated.
  */
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -2669,7 +2674,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 		 * See folio_maybe_mapped_shared() on possible imprecision
 		 * when we cannot easily detect if a folio is shared.
 		 */
-		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+		if (vma && (vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
 			return -EACCES;
 
 		/*
Signature · 1 line
2.34.1
#2 lore raw

RFC PATCH v6 2/5 mm: migrate: Add migrate_misplaced_folios_batch()

In reply to Bharata B Rao

From: Gregory Price <[email protected]>

Tiered memory systems often require migrating multiple folios at once. Currently, migrate_misplaced_folio() handles only one folio per call, which is inefficient for batch operations. This patch introduces migrate_misplaced_folios_batch(), a batch variant that leverages migrate_pages() internally for improved performance.

The caller must isolate folios beforehand using migrate_misplaced_folio_prepare(). On return, the folio list will be empty regardless of success or failure.

This function will be used by pghot kmigrated thread.

[Rewrote commit description]
---
 include/linux/migrate.h |  6 ++++++
 mm/migrate.c            | 48 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+)
include/linux/migrate.h +6 -0
index d5af2b7f577b..5c1e2691cec2 100644
@@ -111,6 +111,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
+int migrate_misplaced_folios_batch(struct list_head *folio_list, int node);
 #else
 static inline int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -121,6 +122,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
+static inline int migrate_misplaced_folios_batch(struct list_head *folio_list,
+						 int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_MIGRATION
mm/migrate.c +48 -0

53 lines not on this page · raw message

Signature · 1 line
2.34.1
#3 lore raw

Re: [RFC PATCH v6 2/5] mm: migrate: Add migrate_misplaced_folios_batch()

Replies to review from Shashiko
[https://sashiko.dev/#/patchset/20260323095104.238982-1-bharata%40amd.com]

Quoted 6 lines
Quoted 3 lines
mm/migrate.c +0 -0

[ ... ]

Quoted 1 line
mm/migrate.c +0 -0
@@ -2751,5 +2751,53 @@ int migrate_misplaced_folio(struct folio *folio, int node)

[ ... ]

Quoted 11 lines

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2794

Quoted 6 lines
mm/migrate.c +6 -0
+	if (nr_remaining)
+		putback_movable_pages(folio_list);
+
+	if (nr_succeeded) {
+		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded);

Should this use mod_lruvec_state() instead of mod_node_page_state()? Using mod_node_page_state() updates the global NUMA node statistic but bypasses the memcg-specific lruvec statistic. This can cause the pgpromote_success metric in memory.stat to undercount for batched promotions.

That's correct. Earlier, memcg wasn't available in this routine but that is no longer the case in v6. Hence I can get lruvec and should be able to use mod_lruvec_state() itself for PGPROMOTE_SUCCESS accounting. This will be part of next iteration.

Quoted 4 lines

Are there missing topology and sysctl checks before incrementing PGPROMOTE_SUCCESS here?

Quoted 8 lines

In the single-folio variant, this statistic is only incremented when:
if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)

	    && !node_is_toptier(folio_nid(folio))
	    && node_is_toptier(node))
		mod_lruvec_state(lruvec, PGPROMOTE_SUCCESS, nr_succeeded);

Unconditionally recording the stat inside the migration helper could corrupt promotion statistics if the function is ever used for lateral NUMA migrations in the future.

This routine is called only for hot page promotion by pghot kmigrated thread and has been documented as such. So unconditional call should be fine.

Regards,
Bharata.

#4 lore raw

Re: [RFC PATCH v6 2/5] mm: migrate: Add migrate_misplaced_folios_batch()

In reply to Bharata B Rao

Hi Bharata

Quoted 73 lines · Bharata B Rao

From: Gregory Price <[email protected]>

Tiered memory systems often require migrating multiple folios at once. Currently, migrate_misplaced_folio() handles only one folio per call, which is inefficient for batch operations. This patch introduces migrate_misplaced_folios_batch(), a batch variant that leverages migrate_pages() internally for improved performance.

The caller must isolate folios beforehand using migrate_misplaced_folio_prepare(). On return, the folio list will be empty regardless of success or failure.

This function will be used by pghot kmigrated thread.

[Rewrote commit description]
---
  include/linux/migrate.h |  6 ++++++
  mm/migrate.c            | 48 +++++++++++++++++++++++++++++++++++++++++
  2 files changed, 54 insertions(+)
include/linux/migrate.h +6 -0
index d5af2b7f577b..5c1e2691cec2 100644
@@ -111,6 +111,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
  int migrate_misplaced_folio_prepare(struct folio *folio,
  		struct vm_area_struct *vma, int node);
  int migrate_misplaced_folio(struct folio *folio, int node);
+int migrate_misplaced_folios_batch(struct list_head *folio_list, int node);
  #else
  static inline int migrate_misplaced_folio_prepare(struct folio *folio,
  		struct vm_area_struct *vma, int node)
@@ -121,6 +122,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
  {
  	return -EAGAIN; /* can't migrate now */
  }
+static inline int migrate_misplaced_folios_batch(struct list_head *folio_list,
+						 int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
  #endif /* CONFIG_NUMA_BALANCING */
 
  #ifdef CONFIG_MIGRATION
mm/migrate.c +19 -0
index a15184950e65..94daec0f49ef 100644
@@ -2751,5 +2751,53 @@ int migrate_misplaced_folio(struct folio *folio, int node)
  	BUG_ON(!list_empty(&migratepages));
  	return nr_remaining ? -EAGAIN : 0;
  }
+
+/**
+ * migrate_misplaced_folios_batch() - Batch variant of migrate_misplaced_folio
+ * Attempts to migrate a folio list to the specified destination.
+ * @folio_list: Isolated list of folios to be batch-migrated.
+ * @node: The NUMA node ID to where the folios should be migrated.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio. All the isolated folios
+ * in the list must belong to the same memcg so that NUMA_PAGE_MIGRATE
+ * stat can be attributed correctly to the memcg.
+ *
+ * This function will un-isolate the folios, drop the elevated reference
+ * and remove them from the list before returning. This is called
+ * only for batched promotion of hot pages from lower tier nodes.
+ *
+ * Return: 0 on success and -EAGAIN on failure or partial migration.
+ *         On return, @folio_list will be empty regardless of success/failure.

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2782

Quoted 10 lines
mm/migrate.c +9 -0
+ */
+int migrate_misplaced_folios_batch(struct list_head *folio_list, int node)
+{
+	pg_data_t *pgdat = NODE_DATA(node);
+	struct mem_cgroup *memcg = NULL;
+	unsigned int nr_succeeded = 0;
+	int nr_remaining;
+
+	if (!list_empty(folio_list)) {
 

We seem to proceed even when the list is empty. Should we instead return early in that case?

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2783

Quoted 2 lines
mm/migrate.c +2 -0
+		struct folio *first = list_first_entry(folio_list, struct folio, lru);
+		memcg = get_mem_cgroup_from_folio(first);

I had a small question—are we ensuring that a single list contains folios from the same memcg?

-Donet

Quoted 20 lines
mm/migrate.c +18 -0
+	}
+
+	nr_remaining = migrate_pages(folio_list, alloc_misplaced_dst_folio,
+				     NULL, node, MIGRATE_ASYNC,
+				     MR_NUMA_MISPLACED, &nr_succeeded);
+	if (nr_remaining)
+		putback_movable_pages(folio_list);
+
+	if (nr_succeeded) {
+		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded);
+		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+	}
+
+	mem_cgroup_put(memcg);
+	WARN_ON(!list_empty(folio_list));
+	return nr_remaining ? -EAGAIN : 0;
+}
  #endif /* CONFIG_NUMA_BALANCING */
  #endif /* CONFIG_NUMA */
maintainer
#5 lore raw

Re: [RFC PATCH v6 2/5] mm: migrate: Add migrate_misplaced_folios_batch()

Quoted 80 lines · Donet Tom

Hi Bharata

Quoted 76 lines · Bharata B Rao

From: Gregory Price <[email protected]>

Tiered memory systems often require migrating multiple folios at once. Currently, migrate_misplaced_folio() handles only one folio per call, which is inefficient for batch operations. This patch introduces migrate_misplaced_folios_batch(), a batch variant that leverages migrate_pages() internally for improved performance.

The caller must isolate folios beforehand using migrate_misplaced_folio_prepare(). On return, the folio list will be empty regardless of success or failure.

This function will be used by pghot kmigrated thread.

[Rewrote commit description]
---
  include/linux/migrate.h |  6 ++++++
  mm/migrate.c            | 48 +++++++++++++++++++++++++++++++++++++++++
  2 files changed, 54 insertions(+)
include/linux/migrate.h +6 -0
index d5af2b7f577b..5c1e2691cec2 100644
@@ -111,6 +111,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
  int migrate_misplaced_folio_prepare(struct folio *folio,
  		struct vm_area_struct *vma, int node);
  int migrate_misplaced_folio(struct folio *folio, int node);
+int migrate_misplaced_folios_batch(struct list_head *folio_list, int node);
  #else
  static inline int migrate_misplaced_folio_prepare(struct folio *folio,
  		struct vm_area_struct *vma, int node)
@@ -121,6 +122,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
  {
  	return -EAGAIN; /* can't migrate now */
  }
+static inline int migrate_misplaced_folios_batch(struct list_head *folio_list,
+						 int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
  #endif /* CONFIG_NUMA_BALANCING */
  #ifdef CONFIG_MIGRATION
mm/migrate.c +23 -0
index a15184950e65..94daec0f49ef 100644
@@ -2751,5 +2751,53 @@ int migrate_misplaced_folio(struct folio *folio, int node)
  	BUG_ON(!list_empty(&migratepages));
  	return nr_remaining ? -EAGAIN : 0;
  }
+
+/**
+ * migrate_misplaced_folios_batch() - Batch variant of migrate_misplaced_folio
+ * Attempts to migrate a folio list to the specified destination.
+ * @folio_list: Isolated list of folios to be batch-migrated.
+ * @node: The NUMA node ID to where the folios should be migrated.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio. All the isolated folios
+ * in the list must belong to the same memcg so that NUMA_PAGE_MIGRATE
+ * stat can be attributed correctly to the memcg.
+ *
+ * This function will un-isolate the folios, drop the elevated reference
+ * and remove them from the list before returning. This is called
+ * only for batched promotion of hot pages from lower tier nodes.
+ *
+ * Return: 0 on success and -EAGAIN on failure or partial migration.
+ *         On return, @folio_list will be empty regardless of success/failure.
+ */
+int migrate_misplaced_folios_batch(struct list_head *folio_list, int node)
+{
+	pg_data_t *pgdat = NODE_DATA(node);
Quoted 9 lines

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node)

Quoted 6 lines
mm/migrate.c +5 -0
+	struct mem_cgroup *memcg = NULL;
+	unsigned int nr_succeeded = 0;
+	int nr_remaining;
+
+	if (!list_empty(folio_list)) {
 

We seem to proceed even when the list is empty. Should we instead return early in that case?

Well that seems utterly reasonable, yes you are right.

Quoted 7 lines

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2783

Quoted 2 lines
mm/migrate.c +2 -0
+		struct folio *first = list_first_entry(folio_list, struct folio, lru);
+		memcg = get_mem_cgroup_from_folio(first);

I had a small question—are we ensuring that a single list contains folios from the same memcg?

It has been a long while since i originally wrote this commit.

I believe I originally wrote this I used it in the context of folio_mark_accessed() driven promotions - trying to get some semblance of NUMA balancing for unmapped page cache pages.

These folios got put into a task workqueue that then got processed on the way out of the kernel.

I think I made the assumption at the time that the folios would all belong to the same memcg - I have since learned that this almost certainly is not the case.

That means a bulk migration may have to first process the folios into lists by memcg before migrating them.

So this commit likely needs to be redone.

~Gregory

#6 lore raw

Re: [RFC PATCH v6 2/5] mm: migrate: Add migrate_misplaced_folios_batch()

Quoted 12 lines · Gregory Price
Quoted 11 lines · Donet Tom
Quoted 11 lines

+/**
+ * migrate_misplaced_folios_batch() - Batch variant of migrate_misplaced_folio
+ * Attempts to migrate a folio list to the specified destination.
+ * @folio_list: Isolated list of folios to be batch-migrated.
+ * @node: The NUMA node ID to where the folios should be migrated.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio. All the isolated folios
+ * in the list must belong to the same memcg so that NUMA_PAGE_MIGRATE
+ * stat can be attributed correctly to the memcg.

^ This is the expectation from the caller. pghot caller ensures this. I could add a debug check in this routine to verify if that is indeed so.

Quoted 21 lines
Quoted 11 lines
Quoted 11 lines

+ *
+ * This function will un-isolate the folios, drop the elevated reference
+ * and remove them from the list before returning. This is called
+ * only for batched promotion of hot pages from lower tier nodes.
+ *
+ * Return: 0 on success and -EAGAIN on failure or partial migration.
+ * On return, @folio_list will be empty regardless of success/failure.
+ */

mm/migrate.c +3 -0
+int migrate_misplaced_folios_batch(struct list_head *folio_list, int node)
+{
+	pg_data_t *pgdat = NODE_DATA(node);
Quoted 9 lines

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2781

Quoted 6 lines
mm/migrate.c +5 -0
+	struct mem_cgroup *memcg = NULL;
+	unsigned int nr_succeeded = 0;
+	int nr_remaining;
+
+	if (!list_empty(folio_list)) {

We seem to proceed even when the list is empty. Should we instead return early in that case?

Quoted 1 line

Well that seems utterly reasonable, yes you are right.

Right. Earlier migrate_pages() may have handled empty source list, but with memcg-awareness in this routine, it is better to return right away.

Quoted 18 lines
Quoted 8 lines

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2783

Quoted 3 lines
mm/migrate.c +2 -0
+		struct folio *first = list_first_entry(folio_list, struct folio, lru);
+		memcg = get_mem_cgroup_from_folio(first);

I had a small question—are we ensuring that a single list contains folios from the same memcg?

It has been a long while since i originally wrote this commit.

I believe I originally wrote this I used it in the context of folio_mark_accessed() driven promotions - trying to get some semblance of NUMA balancing for unmapped page cache pages.

These folios got put into a task workqueue that then got processed on the way out of the kernel.

Quoted 8 lines

I think I made the assumption at the time that the folios would all belong to the same memcg - I have since learned that this almost certainly is not the case.

That means a bulk migration may have to first process the folios into lists by memcg before migrating them.

So this commit likely needs to be redone.

See above.

Regards,
Bharata.

maintainer
#7 lore raw

Re: [RFC PATCH v6 2/5] mm: migrate: Add migrate_misplaced_folios_batch()

Quoted 14 lines · Bharata B Rao
Quoted 12 lines · Gregory Price
Quoted 11 lines · Donet Tom
Quoted 11 lines

+/**
+ * migrate_misplaced_folios_batch() - Batch variant of migrate_misplaced_folio
+ * Attempts to migrate a folio list to the specified destination.
+ * @folio_list: Isolated list of folios to be batch-migrated.
+ * @node: The NUMA node ID to where the folios should be migrated.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio. All the isolated folios
+ * in the list must belong to the same memcg so that NUMA_PAGE_MIGRATE
+ * stat can be attributed correctly to the memcg.

Quoted 3 lines

^ This is the expectation from the caller. pghot caller ensures this. I could add a debug check in this routine to verify if that is indeed so.

Wondering if the interface name should somehow reflect this as well, to make it a bit clearer.

~Gregory

#8 lore raw

Re: [RFC PATCH v6 2/5] mm: migrate: Add migrate_misplaced_folios_batch()

Hi Gregory

Quoted 14 lines · Gregory Price
Quoted 13 lines · Bharata B Rao
Quoted 12 lines · Gregory Price
Quoted 11 lines · Donet Tom
Quoted 11 lines

+/**
+ * migrate_misplaced_folios_batch() - Batch variant of migrate_misplaced_folio
+ * Attempts to migrate a folio list to the specified destination.
+ * @folio_list: Isolated list of folios to be batch-migrated.
+ * @node: The NUMA node ID to where the folios should be migrated.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio. All the isolated folios
+ * in the list must belong to the same memcg so that NUMA_PAGE_MIGRATE
+ * stat can be attributed correctly to the memcg.

Quoted 5 lines
Quoted 3 lines

^ This is the expectation from the caller. pghot caller ensures this. I could add a debug check in this routine to verify if that is indeed so.

Wondering if the interface name should somehow reflect this as well, to make it a bit clearer.

I agree—this would make the interface clearer.

-Donet

Quoted 2 lines

~Gregory

#9 lore raw

Re: [RFC PATCH v6 2/5] mm: migrate: Add migrate_misplaced_folios_batch()

Quoted 18 lines · Donet Tom

Hi Gregory

Quoted 15 lines · Gregory Price
Quoted 14 lines · Bharata B Rao
Quoted 13 lines · Gregory Price
Quoted 12 lines · Donet Tom
Quoted 12 lines

+/**
+ * migrate_misplaced_folios_batch() - Batch variant of migrate_misplaced_folio
+ * Attempts to migrate a folio list to the specified destination.
+ * @folio_list: Isolated list of folios to be batch-migrated.
+ * @node: The NUMA node ID to where the folios should be migrated.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio. All the isolated folios
+ * in the list must belong to the same memcg so that NUMA_PAGE_MIGRATE
+ * stat can be attributed correctly to the memcg.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

^ This is the expectation from the caller. pghot caller ensures this. I could add a debug check in this routine to verify if that is indeed so.

Wondering if the interface name should somehow reflect this as well, to make it a bit clearer.

I agree—this would make the interface clearer.

Sure, will change.

Regards,
Bharata.

#10 lore raw

RFC PATCH v6 3/5 mm: Hot page tracking and promotion - pghot

In reply to Bharata B Rao

pghot is a subsystem that collects memory access information from multiple sources, classifies hot pages resident in lower-tier memory, and promotes them to faster tiers. It stores per-PFN hotness metadata and performs asynchronous, batched promotion via a per-lower-tier-node kernel thread (kmigrated).

This change introduces the default (compact) mode of pghot:

- Per-PFN hotness record (phi_t = u8) embedded via mem_section:
- 2 bits: access frequency (4 levels)
- 5 bits: time bucket (≈4s window with HZ=1000, bucketed jiffies)
- 1 bit : migration-ready flag (MSB)
The LSB of mem_section->hot_map pointer is used as a per-section "hot" flag to gate scanning.

- Event recording API:
int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now) @pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffies
- Sources (e.g., NUMA hint faults, HW hints) call this to report accesses.
- In default mode, the nid is not stored/used for targeting; promotion goes to a configurable toptier node (pghot_target_nid).

- Promotion engine:
- One kmigrated thread per lower-tier node.
- Scans only sections whose "hot" flag was raised, iterates PFNs, and batches candidates by destination node.
- Uses migrate_misplaced_folios_batch() to move batched folios.

- Tunables & stats:
- debugfs: enabled_sources, target_nid, freq_threshold, kmigrated_sleep_ms, kmigrated_batch_nr
- sysctl : vm.pghot_promote_freq_window_ms
- vmstat : pghot_recorded_accesses, pghot_recorded_hintfaults, pghot_recorded_hwhints

Memory overhead
---------------
Default mode uses 1 byte of hotness metadata per PFN on lower-tier nodes.

Behavior & policy
-----------------
- Default mode promotion target:
The nid passed by sources is not stored; hot pages promote to pghot_target_nid (toptier). Precision mode (added later in the series) changes this.

- Record consumption:
kmigrated consumes (clears) the "migration-ready" bit before attempting isolation. If isolation/migration fails, the folio is not re-queued automatically; subsequent accesses will re-arm it. This avoids retry storms and keeps batching stable.

- Wakeups:
kmigrated wakeups are intentionally timeout-driven in v6. We set the per-pgdat "activate" flag on access, and kmigrated checks this flag on its next sleep interval. This keeps the first cut simple and avoids potential wake storms; active wakeups can be considered in a follow-up.

---
 Documentation/admin-guide/mm/pghot.txt |  80 +++++
 include/linux/migrate.h                |   4 +-
 include/linux/mmzone.h                 |  20 ++
 include/linux/pghot.h                  |  82 +++++
 include/linux/vm_event_item.h          |   5 +
 mm/Kconfig                             |  14 +
 mm/Makefile                            |   1 +
 mm/migrate.c                           |  19 +-
 mm/mm_init.c                           |  10 +
 mm/pghot-default.c                     |  79 ++++
 mm/pghot-tunables.c                    | 182 ++++++++++
 mm/pghot.c                             | 479 +++++++++++++++++++++++++
 mm/vmstat.c                            |   5 +
 13 files changed, 971 insertions(+), 9 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/pghot.txt
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot-tunables.c
 create mode 100644 mm/pghot.c
Documentation/admin-guide/mm/pghot.txt new file +80 -0
new file mode 100644
index 000000000000..5f51dd1d4d45
@@ -0,0 +1,80 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=================================
+PGHOT: Hot Page Tracking Tunables
+=================================
+
+Overview
+========
+The PGHOT subsystem tracks frequently accessed pages in lower-tier memory and
+promotes them to faster tiers. It uses per-PFN hotness metadata and asynchronous
+migration via per-node kernel threads (kmigrated).
+
+This document describes tunables available via **debugfs** and **sysctl** for
+PGHOT.
+
+Debugfs Interface
+=================
+Path: /sys/kernel/debug/pghot/
+
+1. **enabled_sources**
+   - Bitmask to enable/disable hotness sources.
+   - Bits:
+     - 0: Hint faults (value 0x1)
+     - 1: Hardware hints (value 0x2)
+   - Default: 0 (disabled)
+   - Example:
+     # echo 0x3 > /sys/kernel/debug/pghot/enabled_sources
+     Enables all sources.
+
+2. **target_nid**
+   - Toptier NUMA node ID to which hot pages should be promoted when source
+     does not provide nid. Used when hotness source can't provide accessing
+     NID or when the tracking mode is default.
+   - Default: 0
+   - Example:
+     # echo 1 > /sys/kernel/debug/pghot/target_nid
+
+3. **freq_threshold**
+   - Minimum access frequency before a page is marked ready for promotion.
+   - Range: 1 to 3
+   - Default: 2
+   - Example:
+     # echo 3 > /sys/kernel/debug/pghot/freq_threshold
+
+4. **kmigrated_sleep_ms**
+   - Sleep interval (ms) for kmigrated thread between scans.
+   - Default: 100
+
+5. **kmigrated_batch_nr**
+   - Maximum number of folios migrated in one batch.
+   - Default: 512
+
+Sysctl Interface
+================
+1. pghot_promote_freq_window_ms
+
+Path: /proc/sys/vm/pghot_promote_freq_window_ms
+
+- Controls the time window (in ms) for counting access frequency. A page is
+  considered hot only when **freq_threshold** number of accesses occur with
+  this time period.
+- Default: 3000 (3 seconds)
+- Example:
+  # sysctl vm.pghot_promote_freq_window_ms=3000
+
+Vmstat Counters
+===============
+Following vmstat counters provide some stats about pghot subsystem.
+
+Path: /proc/vmstat
+
+1. **pghot_recorded_accesses**
+   - Number of total hot page accesses recorded by pghot.
+
+2. **pghot_recorded_hintfaults**
+   - Number of recorded accesses reported by NUMA Balancing based
+     hotness source.
+
+3. **pghot_recorded_hwhints**
+   - Number of recorded accesses reported by hwhints source.
include/linux/migrate.h +2 -2
index 5c1e2691cec2..7f912b6ebf02 100644
@@ -107,7 +107,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
 
 #endif /* CONFIG_MIGRATION */
 
-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
@@ -127,7 +127,7 @@ static inline int migrate_misplaced_folios_batch(struct list_head *folio_list,
 {
 	return -EAGAIN; /* can't migrate now */
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */
 
 #ifdef CONFIG_MIGRATION
 
include/linux/mmzone.h +20 -0
index 3e51190a55e4..d7ed60956543 100644
@@ -1064,6 +1064,7 @@ enum pgdat_flags {
 					 * many pages under writeback
 					 */
 	PGDAT_RECLAIM_LOCKED,		/* prevents concurrent reclaim */
+	PGDAT_KMIGRATED_ACTIVATE,	/* activates kmigrated */
 };
 
 enum zone_flags {
@@ -1518,6 +1519,10 @@ typedef struct pglist_data {
 #ifdef CONFIG_MEMORY_FAILURE
 	struct memory_failure_stats mf_stats;
 #endif
+#ifdef CONFIG_PGHOT
+	struct task_struct *kmigrated;
+	wait_queue_head_t kmigrated_wait;
+#endif
 } pg_data_t;
 
 #define node_present_pages(nid)	(NODE_DATA(nid)->node_present_pages)
@@ -1930,12 +1935,27 @@ struct mem_section {
 	unsigned long section_mem_map;
 
 	struct mem_section_usage *usage;
+#ifdef CONFIG_PGHOT
+	/*
+	 * Per-PFN hotness data for this section.
+	 * Array of phi_t (u8 in default mode).
+	 * LSB is used as PGHOT_SECTION_HOT_BIT flag.
+	 */
+	void *hot_map;
+#endif
 #ifdef CONFIG_PAGE_EXTENSION
 	/*
 	 * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use
 	 * section. (see page_ext.h about this.)
 	 */
 	struct page_ext *page_ext;
+#endif
+	/*
+	 * Padding to maintain consistent mem_section size when exactly
+	 * one of PGHOT or PAGE_EXTENSION is enabled. This ensures
+	 * optimal alignment regardless of configuration.
+	 */
+#if (defined(CONFIG_PGHOT) ^ defined(CONFIG_PAGE_EXTENSION))
 	unsigned long pad;
 #endif
 	/*
include/linux/pghot.h new file +82 -0

82 lines not on this page · raw message

include/linux/vm_event_item.h +5 -0
index 22a139f82d75..4ce670c1bb02 100644
@@ -188,6 +188,11 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		KSTACK_REST,
 #endif
 #endif /* CONFIG_DEBUG_STACK_USAGE */
+#ifdef CONFIG_PGHOT
+		PGHOT_RECORDED_ACCESSES,
+		PGHOT_RECORDED_HINTFAULTS,
+		PGHOT_RECORDED_HWHINTS,
+#endif /* CONFIG_PGHOT */
 		NR_VM_EVENT_ITEMS
 };
 
mm/Kconfig +14 -0
index ebd8ea353687..4aeab6aee535 100644
@@ -1471,6 +1471,20 @@ config LAZY_MMU_MODE_KUNIT_TEST
 
 	  If unsure, say N.
 
+config PGHOT
+	bool "Hot page tracking and promotion"
+	def_bool n
+	depends on NUMA && MIGRATION && SPARSEMEM && MMU
+	help
+	  A sub-system to track page accesses in lower tier memory and
+	  maintain hot page information. Promotes hot pages from lower
+	  tiers to top tier by using the memory access information provided
+	  by various sources. Asynchronous promotion is done by per-node
+	  kernel threads.
+
+	  This adds 1 byte of metadata overhead per page in lower-tier
+	  memory nodes.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +1 -0
index 8ad2ab08244e..33014de43acc 100644
@@ -150,3 +150,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
+obj-$(CONFIG_PGHOT) += pghot.o pghot-tunables.o pghot-default.o
mm/migrate.c +12 -7
index 94daec0f49ef..a5f48984ed3e 100644
@@ -2606,7 +2606,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long, nr_pages,
 	return kernel_move_pages(pid, nr_pages, pages, nodes, status, flags);
 }
 
-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
 /*
  * Returns true if this is a safe migration target node for misplaced NUMA
  * pages. Currently it only checks the watermarks which is crude.
@@ -2726,12 +2726,10 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
  */
 int migrate_misplaced_folio(struct folio *folio, int node)
 {
-	pg_data_t *pgdat = NODE_DATA(node);
 	int nr_remaining;
 	unsigned int nr_succeeded;
 	LIST_HEAD(migratepages);
 	struct mem_cgroup *memcg = get_mem_cgroup_from_folio(folio);
-	struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
 
 	list_add(&folio->lru, &migratepages);
 	nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
@@ -2740,12 +2738,18 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	if (nr_remaining && !list_empty(&migratepages))
 		putback_movable_pages(&migratepages);
 	if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
 		if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
 		    && !node_is_toptier(folio_nid(folio))
-		    && node_is_toptier(node))
+		    && node_is_toptier(node)) {
+			pg_data_t *pgdat = NODE_DATA(node);
+			struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
+
 			mod_lruvec_state(lruvec, PGPROMOTE_SUCCESS, nr_succeeded);
+		}
+#endif
 	}
 	mem_cgroup_put(memcg);
 	BUG_ON(!list_empty(&migratepages));
@@ -2773,7 +2777,6 @@ int migrate_misplaced_folio(struct folio *folio, int node)
  */
 int migrate_misplaced_folios_batch(struct list_head *folio_list, int node)
 {
-	pg_data_t *pgdat = NODE_DATA(node);
 	struct mem_cgroup *memcg = NULL;
 	unsigned int nr_succeeded = 0;
 	int nr_remaining;
@@ -2790,14 +2793,16 @@ int migrate_misplaced_folios_batch(struct list_head *folio_list, int node)
 		putback_movable_pages(folio_list);
 
 	if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
-		mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_node_page_state(NODE_DATA(node), PGPROMOTE_SUCCESS, nr_succeeded);
+#endif
 	}
 
 	mem_cgroup_put(memcg);
 	WARN_ON(!list_empty(folio_list));
 	return nr_remaining ? -EAGAIN : 0;
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */
 #endif /* CONFIG_NUMA */
mm/mm_init.c +10 -0
index df34797691bd..c777c54cfe69 100644
@@ -1398,6 +1398,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)
 static void pgdat_init_kcompactd(struct pglist_data *pgdat) {}
 #endif
 
+#ifdef CONFIG_PGHOT
+static void pgdat_init_kmigrated(struct pglist_data *pgdat)
+{
+	init_waitqueue_head(&pgdat->kmigrated_wait);
+}
+#else
+static inline void pgdat_init_kmigrated(struct pglist_data *pgdat) {}
+#endif
+
 static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 {
 	int i;
@@ -1407,6 +1416,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 
 	pgdat_init_split_queue(pgdat);
 	pgdat_init_kcompactd(pgdat);
+	pgdat_init_kmigrated(pgdat);
 
 	init_waitqueue_head(&pgdat->kswapd_wait);
 	init_waitqueue_head(&pgdat->pfmemalloc_wait);
mm/pghot-default.c new file +79 -0

79 lines not on this page · raw message

mm/pghot-tunables.c new file +182 -0

182 lines not on this page · raw message

mm/pghot.c new file +479 -0

479 lines not on this page · raw message

mm/vmstat.c +5 -0
index 86b14b0f77b5..d3fbe2a5d0e6 100644
@@ -1486,6 +1486,11 @@ const char * const vmstat_text[] = {
 	[I(KSTACK_REST)]			= "kstack_rest",
 #endif
 #endif
+#ifdef CONFIG_PGHOT
+	[I(PGHOT_RECORDED_ACCESSES)]		= "pghot_recorded_accesses",
+	[I(PGHOT_RECORDED_HINTFAULTS)]		= "pghot_recorded_hintfaults",
+	[I(PGHOT_RECORDED_HWHINTS)]		= "pghot_recorded_hwhints",
+#endif /* CONFIG_PGHOT */
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
 };
Signature · 1 line
2.34.1
#11 lore raw

Re: [RFC PATCH v6 3/5] mm: Hot page tracking and promotion - pghot

kernel test robot noticed the following build errors:

Hi Bharata,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on a989fde763f4f24209e4702f50a45be572340e68]

url:    https://github.com/intel-lab-lkp/linux/commits/Bharata-B-Rao/mm-migrate-Allow-misplaced-migration-without-VMA/20260325-092849
base:   a989fde763f4f24209e4702f50a45be572340e68
patch link:    https://lore.kernel.org/r/20260323095104.238982-4-bharata%40amd.com
patch subject: [RFC PATCH v6 3/5] mm: Hot page tracking and promotion - pghot
config: powerpc-allmodconfig (https://download.01.org/0day-ci/archive/20260327/[email protected]/config)
compiler: powerpc64-linux-gcc (GCC) 15.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260327/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from mm/pghot-default.c:10:
mm/pghot-default.c: In function 'pghot_access_latency':

>> include/linux/pghot.h:56:41: error: implicit declaration of function 'GENMASK' [-Wimplicit-function-declaration]
      56 | #define PGHOT_TIME_MASK                 GENMASK(PGHOT_TIME_WIDTH - 1, 0)
         |                                         ^~~~~~~
   include/linux/pghot.h:57:42: note: in expansion of macro 'PGHOT_TIME_MASK'
      57 | #define PGHOT_TIME_BUCKETS_MASK         (PGHOT_TIME_MASK << PGHOT_TIME_BUCKETS_SHIFT)
         |                                          ^~~~~~~~~~~~~~~
   mm/pghot-default.c:24:17: note: in expansion of macro 'PGHOT_TIME_BUCKETS_MASK'
      24 |         time &= PGHOT_TIME_BUCKETS_MASK;
         |                 ^~~~~~~~~~~~~~~~~~~~~~~
   mm/pghot-default.c: In function 'pghot_update_record':
   mm/pghot-default.c:60:36: error: implicit declaration of function 'BIT' [-Wimplicit-function-declaration]
      60 |                         hotness |= BIT(PGHOT_MIGRATE_READY);
         |                                    ^~~
   In file included from include/linux/jump_label.h:77,
                    from include/linux/static_key.h:1,
                    from include/linux/pghot.h:13:
   mm/pghot-default.c:61:28: error: implicit declaration of function 'try_cmpxchg' [-Wimplicit-function-declaration]
      61 |         } while (unlikely(!try_cmpxchg(phi, &old_hotness, hotness)));
         |                            ^~~~~~~~~~~
   include/linux/compiler.h:77:45: note: in definition of macro 'unlikely'
      77 | # define unlikely(x)    __builtin_expect(!!(x), 0)
         |                                             ^

vim +/GENMASK +56 include/linux/pghot.h

    54	
    55	#define PGHOT_FREQ_MASK			GENMASK(PGHOT_FREQ_WIDTH - 1, 0)
  > 56	#define PGHOT_TIME_MASK			GENMASK(PGHOT_TIME_WIDTH - 1, 0)

57 #define PGHOT_TIME_BUCKETS_MASK (PGHOT_TIME_MASK << PGHOT_TIME_BUCKETS_SHIFT)
58

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#12 lore raw

Re: [RFC PATCH v6 3/5] mm: Hot page tracking and promotion - pghot

In reply to Bharata B Rao

Hi Bharata

Quoted 851 lines · Bharata B Rao

pghot is a subsystem that collects memory access information from multiple sources, classifies hot pages resident in lower-tier memory, and promotes them to faster tiers. It stores per-PFN hotness metadata and performs asynchronous, batched promotion via a per-lower-tier-node kernel thread (kmigrated).

This change introduces the default (compact) mode of pghot:

- Per-PFN hotness record (phi_t = u8) embedded via mem_section:
- 2 bits: access frequency (4 levels)
- 5 bits: time bucket (≈4s window with HZ=1000, bucketed jiffies)
- 1 bit : migration-ready flag (MSB)
The LSB of mem_section->hot_map pointer is used as a per-section "hot" flag to gate scanning.

- Event recording API:
int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now) @pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffies
- Sources (e.g., NUMA hint faults, HW hints) call this to report accesses.
- In default mode, the nid is not stored/used for targeting; promotion goes to a configurable toptier node (pghot_target_nid).

- Promotion engine:
- One kmigrated thread per lower-tier node.
- Scans only sections whose "hot" flag was raised, iterates PFNs, and batches candidates by destination node.
- Uses migrate_misplaced_folios_batch() to move batched folios.

- Tunables & stats:
- debugfs: enabled_sources, target_nid, freq_threshold, kmigrated_sleep_ms, kmigrated_batch_nr
- sysctl : vm.pghot_promote_freq_window_ms
- vmstat : pghot_recorded_accesses, pghot_recorded_hintfaults, pghot_recorded_hwhints

Memory overhead
---------------
Default mode uses 1 byte of hotness metadata per PFN on lower-tier nodes.

Behavior & policy
-----------------
- Default mode promotion target:
The nid passed by sources is not stored; hot pages promote to pghot_target_nid (toptier). Precision mode (added later in the series) changes this.

- Record consumption:
kmigrated consumes (clears) the "migration-ready" bit before attempting isolation. If isolation/migration fails, the folio is not re-queued automatically; subsequent accesses will re-arm it. This avoids retry storms and keeps batching stable.

- Wakeups:
kmigrated wakeups are intentionally timeout-driven in v6. We set the per-pgdat "activate" flag on access, and kmigrated checks this flag on its next sleep interval. This keeps the first cut simple and avoids potential wake storms; active wakeups can be considered in a follow-up.

---
  Documentation/admin-guide/mm/pghot.txt |  80 +++++
  include/linux/migrate.h                |   4 +-
  include/linux/mmzone.h                 |  20 ++
  include/linux/pghot.h                  |  82 +++++
  include/linux/vm_event_item.h          |   5 +
  mm/Kconfig                             |  14 +
  mm/Makefile                            |   1 +
  mm/migrate.c                           |  19 +-
  mm/mm_init.c                           |  10 +
  mm/pghot-default.c                     |  79 ++++
  mm/pghot-tunables.c                    | 182 ++++++++++
  mm/pghot.c                             | 479 +++++++++++++++++++++++++
  mm/vmstat.c                            |   5 +
  13 files changed, 971 insertions(+), 9 deletions(-)

create mode 100644 Documentation/admin-guide/mm/pghot.txt create mode 100644 include/linux/pghot.h
create mode 100644 mm/pghot-default.c
create mode 100644 mm/pghot-tunables.c
create mode 100644 mm/pghot.c

Documentation/admin-guide/mm/pghot.txt new file +80 -0
new file mode 100644
index 000000000000..5f51dd1d4d45
@@ -0,0 +1,80 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=================================
+PGHOT: Hot Page Tracking Tunables
+=================================
+
+Overview
+========
+The PGHOT subsystem tracks frequently accessed pages in lower-tier memory and
+promotes them to faster tiers. It uses per-PFN hotness metadata and asynchronous
+migration via per-node kernel threads (kmigrated).
+
+This document describes tunables available via **debugfs** and **sysctl** for
+PGHOT.
+
+Debugfs Interface
+=================
+Path: /sys/kernel/debug/pghot/
+
+1. **enabled_sources**
+   - Bitmask to enable/disable hotness sources.
+   - Bits:
+     - 0: Hint faults (value 0x1)
+     - 1: Hardware hints (value 0x2)
+   - Default: 0 (disabled)
+   - Example:
+     # echo 0x3 > /sys/kernel/debug/pghot/enabled_sources
+     Enables all sources.
+
+2. **target_nid**
+   - Toptier NUMA node ID to which hot pages should be promoted when source
+     does not provide nid. Used when hotness source can't provide accessing
+     NID or when the tracking mode is default.
+   - Default: 0
+   - Example:
+     # echo 1 > /sys/kernel/debug/pghot/target_nid
+
+3. **freq_threshold**
+   - Minimum access frequency before a page is marked ready for promotion.
+   - Range: 1 to 3
+   - Default: 2
+   - Example:
+     # echo 3 > /sys/kernel/debug/pghot/freq_threshold
+
+4. **kmigrated_sleep_ms**
+   - Sleep interval (ms) for kmigrated thread between scans.
+   - Default: 100
+
+5. **kmigrated_batch_nr**
+   - Maximum number of folios migrated in one batch.
+   - Default: 512
+
+Sysctl Interface
+================
+1. pghot_promote_freq_window_ms
+
+Path: /proc/sys/vm/pghot_promote_freq_window_ms
+
+- Controls the time window (in ms) for counting access frequency. A page is
+  considered hot only when **freq_threshold** number of accesses occur with
+  this time period.
+- Default: 3000 (3 seconds)
+- Example:
+  # sysctl vm.pghot_promote_freq_window_ms=3000
+
+Vmstat Counters
+===============
+Following vmstat counters provide some stats about pghot subsystem.
+
+Path: /proc/vmstat
+
+1. **pghot_recorded_accesses**
+   - Number of total hot page accesses recorded by pghot.
+
+2. **pghot_recorded_hintfaults**
+   - Number of recorded accesses reported by NUMA Balancing based
+     hotness source.
+
+3. **pghot_recorded_hwhints**
+   - Number of recorded accesses reported by hwhints source.
include/linux/migrate.h +2 -2
index 5c1e2691cec2..7f912b6ebf02 100644
@@ -107,7 +107,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
 
  #endif /* CONFIG_MIGRATION */
 
-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
  int migrate_misplaced_folio_prepare(struct folio *folio,
  		struct vm_area_struct *vma, int node);
  int migrate_misplaced_folio(struct folio *folio, int node);
@@ -127,7 +127,7 @@ static inline int migrate_misplaced_folios_batch(struct list_head *folio_list,
  {
  	return -EAGAIN; /* can't migrate now */
  }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */
 
  #ifdef CONFIG_MIGRATION
 
include/linux/mmzone.h +20 -0
index 3e51190a55e4..d7ed60956543 100644
@@ -1064,6 +1064,7 @@ enum pgdat_flags {
  					 * many pages under writeback
  					 */
  	PGDAT_RECLAIM_LOCKED,		/* prevents concurrent reclaim */
+	PGDAT_KMIGRATED_ACTIVATE,	/* activates kmigrated */
  };
 
  enum zone_flags {
@@ -1518,6 +1519,10 @@ typedef struct pglist_data {
  #ifdef CONFIG_MEMORY_FAILURE
  	struct memory_failure_stats mf_stats;
  #endif
+#ifdef CONFIG_PGHOT
+	struct task_struct *kmigrated;
+	wait_queue_head_t kmigrated_wait;
+#endif
  } pg_data_t;
 
  #define node_present_pages(nid)	(NODE_DATA(nid)->node_present_pages)
@@ -1930,12 +1935,27 @@ struct mem_section {
  	unsigned long section_mem_map;
 
  	struct mem_section_usage *usage;
+#ifdef CONFIG_PGHOT
+	/*
+	 * Per-PFN hotness data for this section.
+	 * Array of phi_t (u8 in default mode).
+	 * LSB is used as PGHOT_SECTION_HOT_BIT flag.
+	 */
+	void *hot_map;
+#endif
  #ifdef CONFIG_PAGE_EXTENSION
  	/*
  	 * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use
  	 * section. (see page_ext.h about this.)
  	 */
  	struct page_ext *page_ext;
+#endif
+	/*
+	 * Padding to maintain consistent mem_section size when exactly
+	 * one of PGHOT or PAGE_EXTENSION is enabled. This ensures
+	 * optimal alignment regardless of configuration.
+	 */
+#if (defined(CONFIG_PGHOT) ^ defined(CONFIG_PAGE_EXTENSION))
  	unsigned long pad;
  #endif
  	/*
include/linux/pghot.h new file +82 -0
new file mode 100644
index 000000000000..525d4dd28fc1
@@ -0,0 +1,82 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_PGHOT_H
+#define _LINUX_PGHOT_H
+
+/* Page hotness temperature sources */
+enum pghot_src {
+	PGHOT_HINTFAULTS = 0,
+	PGHOT_HWHINTS,
+	PGHOT_SRC_MAX
+};
+
+#ifdef CONFIG_PGHOT
+#include <linux/static_key.h>
+
+extern unsigned int pghot_target_nid;
+extern unsigned int pghot_src_enabled;
+extern unsigned int pghot_freq_threshold;
+extern unsigned int kmigrated_sleep_ms;
+extern unsigned int kmigrated_batch_nr;
+extern unsigned int sysctl_pghot_freq_window;
+
+void pghot_debug_init(void);
+
+DECLARE_STATIC_KEY_FALSE(pghot_src_hintfaults);
+DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
+
+#define PGHOT_HINTFAULTS_ENABLED	BIT(PGHOT_HINTFAULTS)
+#define PGHOT_HWHINTS_ENABLED		BIT(PGHOT_HWHINTS)
+#define PGHOT_SRC_ENABLED_MASK		GENMASK(PGHOT_SRC_MAX - 1, 0)
+
+#define PGHOT_DEFAULT_FREQ_THRESHOLD	2
+
+#define KMIGRATED_DEFAULT_SLEEP_MS	100
+#define KMIGRATED_DEFAULT_BATCH_NR	512
+
+#define PGHOT_DEFAULT_NODE		0
+
+#define PGHOT_DEFAULT_FREQ_WINDOW	(3 * MSEC_PER_SEC)
+
+/*
+ * Bits 0-6 are used to store frequency and time.
+ * Bit 7 is used to indicate the page is ready for migration.
+ */
+#define PGHOT_MIGRATE_READY		7
+
+#define PGHOT_FREQ_WIDTH		2
+/* Bucketed time is stored in 5 bits which can represent up to 3.9s with HZ=1000 */
+#define PGHOT_TIME_BUCKETS_SHIFT	7
+#define PGHOT_TIME_WIDTH		5
+#define PGHOT_NID_WIDTH			10
+
+#define PGHOT_FREQ_SHIFT		0
+#define PGHOT_TIME_SHIFT		(PGHOT_FREQ_SHIFT + PGHOT_FREQ_WIDTH)
+
+#define PGHOT_FREQ_MASK			GENMASK(PGHOT_FREQ_WIDTH - 1, 0)
+#define PGHOT_TIME_MASK			GENMASK(PGHOT_TIME_WIDTH - 1, 0)
+#define PGHOT_TIME_BUCKETS_MASK		(PGHOT_TIME_MASK << PGHOT_TIME_BUCKETS_SHIFT)
+
+#define PGHOT_NID_MAX			((1 << PGHOT_NID_WIDTH) - 1)
+#define PGHOT_FREQ_MAX			((1 << PGHOT_FREQ_WIDTH) - 1)
+#define PGHOT_TIME_MAX			((1 << PGHOT_TIME_WIDTH) - 1)
+
+typedef u8 phi_t;
+
+#define PGHOT_RECORD_SIZE		sizeof(phi_t)
+
+#define PGHOT_SECTION_HOT_BIT		0
+#define PGHOT_SECTION_HOT_MASK		BIT(PGHOT_SECTION_HOT_BIT)
+
+bool pghot_nid_valid(int nid);
+unsigned long pghot_access_latency(unsigned long old_time, unsigned long time);
+bool pghot_update_record(phi_t *phi, int nid, unsigned long now);
+int pghot_get_record(phi_t *phi, int *nid, int *freq, unsigned long *time);
+
+int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now);
+#else
+static inline int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
+{
+	return 0;
+}
+#endif /* CONFIG_PGHOT */
+#endif /* _LINUX_PGHOT_H */
include/linux/vm_event_item.h +5 -0
index 22a139f82d75..4ce670c1bb02 100644
@@ -188,6 +188,11 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
  		KSTACK_REST,
  #endif
  #endif /* CONFIG_DEBUG_STACK_USAGE */
+#ifdef CONFIG_PGHOT
+		PGHOT_RECORDED_ACCESSES,
+		PGHOT_RECORDED_HINTFAULTS,
+		PGHOT_RECORDED_HWHINTS,
+#endif /* CONFIG_PGHOT */
  		NR_VM_EVENT_ITEMS
  };
 
mm/Kconfig +14 -0
index ebd8ea353687..4aeab6aee535 100644
@@ -1471,6 +1471,20 @@ config LAZY_MMU_MODE_KUNIT_TEST
 
  	  If unsure, say N.
 
+config PGHOT
+	bool "Hot page tracking and promotion"
+	def_bool n
+	depends on NUMA && MIGRATION && SPARSEMEM && MMU
+	help
+	  A sub-system to track page accesses in lower tier memory and
+	  maintain hot page information. Promotes hot pages from lower
+	  tiers to top tier by using the memory access information provided
+	  by various sources. Asynchronous promotion is done by per-node
+	  kernel threads.
+
+	  This adds 1 byte of metadata overhead per page in lower-tier
+	  memory nodes.
+
  source "mm/damon/Kconfig"
 
  endmenu
mm/Makefile +1 -0
index 8ad2ab08244e..33014de43acc 100644
@@ -150,3 +150,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
  obj-$(CONFIG_EXECMEM) += execmem.o
  obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
  obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
+obj-$(CONFIG_PGHOT) += pghot.o pghot-tunables.o pghot-default.o
mm/migrate.c +12 -7
index 94daec0f49ef..a5f48984ed3e 100644
@@ -2606,7 +2606,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long, nr_pages,
  	return kernel_move_pages(pid, nr_pages, pages, nodes, status, flags);
  }
 
-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
  /*
   * Returns true if this is a safe migration target node for misplaced NUMA
   * pages. Currently it only checks the watermarks which is crude.
@@ -2726,12 +2726,10 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
   */
  int migrate_misplaced_folio(struct folio *folio, int node)
  {
-	pg_data_t *pgdat = NODE_DATA(node);
  	int nr_remaining;
  	unsigned int nr_succeeded;
  	LIST_HEAD(migratepages);
  	struct mem_cgroup *memcg = get_mem_cgroup_from_folio(folio);
-	struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
 
  	list_add(&folio->lru, &migratepages);
  	nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
@@ -2740,12 +2738,18 @@ int migrate_misplaced_folio(struct folio *folio, int node)
  	if (nr_remaining && !list_empty(&migratepages))
  		putback_movable_pages(&migratepages);
  	if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
  		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
  		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
  		if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
  		    && !node_is_toptier(folio_nid(folio))
-		    && node_is_toptier(node))
+		    && node_is_toptier(node)) {
+			pg_data_t *pgdat = NODE_DATA(node);
+			struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
+
  			mod_lruvec_state(lruvec, PGPROMOTE_SUCCESS, nr_succeeded);
+		}
+#endif
  	}
  	mem_cgroup_put(memcg);
  	BUG_ON(!list_empty(&migratepages));
@@ -2773,7 +2777,6 @@ int migrate_misplaced_folio(struct folio *folio, int node)
   */
  int migrate_misplaced_folios_batch(struct list_head *folio_list, int node)
  {
-	pg_data_t *pgdat = NODE_DATA(node);
  	struct mem_cgroup *memcg = NULL;
  	unsigned int nr_succeeded = 0;
  	int nr_remaining;
@@ -2790,14 +2793,16 @@ int migrate_misplaced_folios_batch(struct list_head *folio_list, int node)
  		putback_movable_pages(folio_list);
 
  	if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
  		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
-		mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded);
  		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_node_page_state(NODE_DATA(node), PGPROMOTE_SUCCESS, nr_succeeded);
+#endif
  	}
 
  	mem_cgroup_put(memcg);
  	WARN_ON(!list_empty(folio_list));
  	return nr_remaining ? -EAGAIN : 0;
  }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */
  #endif /* CONFIG_NUMA */
mm/mm_init.c +10 -0
index df34797691bd..c777c54cfe69 100644
@@ -1398,6 +1398,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)
  static void pgdat_init_kcompactd(struct pglist_data *pgdat) {}
  #endif
 
+#ifdef CONFIG_PGHOT
+static void pgdat_init_kmigrated(struct pglist_data *pgdat)
+{
+	init_waitqueue_head(&pgdat->kmigrated_wait);
+}
+#else
+static inline void pgdat_init_kmigrated(struct pglist_data *pgdat) {}
+#endif
+
  static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
  {
  	int i;
@@ -1407,6 +1416,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 
  	pgdat_init_split_queue(pgdat);
  	pgdat_init_kcompactd(pgdat);
+	pgdat_init_kmigrated(pgdat);
 
  	init_waitqueue_head(&pgdat->kswapd_wait);
  	init_waitqueue_head(&pgdat->pfmemalloc_wait);
mm/pghot-default.c new file +79 -0
new file mode 100644
index 000000000000..e610062345e4
@@ -0,0 +1,79 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pghot: Default mode
+ *
+ * 1 byte hotness record per PFN.
+ * Bucketed time and frequency tracked as part of the record.
+ * Promotion to @pghot_target_nid by default.
+ */
+
+#include <linux/pghot.h>
+#include <linux/jiffies.h>
+
+/* pghot-default doesn't store and hence no NID validation is required */
+bool pghot_nid_valid(int nid)
+{
+	return true;
+}
+
+/*
+ * @time is regular time, @old_time is bucketed time.
+ */
+unsigned long pghot_access_latency(unsigned long old_time, unsigned long time)
+{
+	time &= PGHOT_TIME_BUCKETS_MASK;
+	old_time <<= PGHOT_TIME_BUCKETS_SHIFT;
+
+	return jiffies_to_msecs((time - old_time) & PGHOT_TIME_BUCKETS_MASK);
+}
+
+bool pghot_update_record(phi_t *phi, int nid, unsigned long now)
+{
+	phi_t freq, old_freq, hotness, old_hotness, old_time;
+	phi_t time = now >> PGHOT_TIME_BUCKETS_SHIFT;
+
+	old_hotness = READ_ONCE(*phi);
+	do {
+		bool new_window = false;
+
+		hotness = old_hotness;
+		old_freq = (hotness >> PGHOT_FREQ_SHIFT) & PGHOT_FREQ_MASK;
+		old_time = (hotness >> PGHOT_TIME_SHIFT) & PGHOT_TIME_MASK;
+
+		if (pghot_access_latency(old_time, now) > sysctl_pghot_freq_window)
+			new_window = true;
+
+		if (new_window)
+			freq = 1;
+		else if (old_freq < PGHOT_FREQ_MAX)
+			freq = old_freq + 1;
+		else
+			freq = old_freq;
+
+		hotness &= ~(PGHOT_FREQ_MASK << PGHOT_FREQ_SHIFT);
+		hotness &= ~(PGHOT_TIME_MASK << PGHOT_TIME_SHIFT);
+
+		hotness |= (freq & PGHOT_FREQ_MASK) << PGHOT_FREQ_SHIFT;
+		hotness |= (time & PGHOT_TIME_MASK) << PGHOT_TIME_SHIFT;
+
+		if (freq >= pghot_freq_threshold)
+			hotness |= BIT(PGHOT_MIGRATE_READY);
+	} while (unlikely(!try_cmpxchg(phi, &old_hotness, hotness)));
+	return !!(hotness & BIT(PGHOT_MIGRATE_READY));
+}
+
+int pghot_get_record(phi_t *phi, int *nid, int *freq, unsigned long *time)
+{
+	phi_t old_hotness, hotness = 0;
+
+	old_hotness = READ_ONCE(*phi);
+	do {
+		if (!(old_hotness & BIT(PGHOT_MIGRATE_READY)))
+			return -EINVAL;
+	} while (unlikely(!try_cmpxchg(phi, &old_hotness, hotness)));
+
+	*nid = pghot_target_nid;
+	*freq = (old_hotness >> PGHOT_FREQ_SHIFT) & PGHOT_FREQ_MASK;
+	*time = (old_hotness >> PGHOT_TIME_SHIFT) & PGHOT_TIME_MASK;
+	return 0;
+}
mm/pghot-tunables.c new file +182 -0
new file mode 100644
index 000000000000..f04e2137309e
@@ -0,0 +1,182 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pghot tunables in debugfs
+ */
+#include <linux/pghot.h>
+#include <linux/memory-tiers.h>
+#include <linux/debugfs.h>
+
+static struct dentry *debugfs_pghot;
+static DEFINE_MUTEX(pghot_tunables_lock);
+
+static ssize_t pghot_freq_th_write(struct file *filp, const char __user *ubuf,
+				   size_t cnt, loff_t *ppos)
+{
+	char buf[16];
+	unsigned int freq;
+
+	if (cnt > 15)
+		cnt = 15;
+
+	if (copy_from_user(&buf, ubuf, cnt))
+		return -EFAULT;
+	buf[cnt] = '\0';
+
+	if (kstrtouint(buf, 10, &freq))
+		return -EINVAL;
+
+	if (!freq || freq > PGHOT_FREQ_MAX)
+		return -EINVAL;
+
+	mutex_lock(&pghot_tunables_lock);
+	pghot_freq_threshold = freq;
+	mutex_unlock(&pghot_tunables_lock);
+
+	*ppos += cnt;
+	return cnt;
+}
+
+static int pghot_freq_th_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", pghot_freq_threshold);
+	return 0;
+}
+
+static int pghot_freq_th_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, pghot_freq_th_show, NULL);
+}
+
+static const struct file_operations pghot_freq_th_fops = {
+	.open		= pghot_freq_th_open,
+	.write		= pghot_freq_th_write,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release,
+};
+
+static ssize_t pghot_target_nid_write(struct file *filp, const char __user *ubuf,
+				      size_t cnt, loff_t *ppos)
+{
+	char buf[16];
+	unsigned int nid;
+
+	if (cnt > 15)
+		cnt = 15;
+
+	if (copy_from_user(&buf, ubuf, cnt))
+		return -EFAULT;
+	buf[cnt] = '\0';
+
+	if (kstrtouint(buf, 10, &nid))
+		return -EINVAL;
+
+	if (nid > PGHOT_NID_MAX || !node_online(nid) || !node_is_toptier(nid))
+		return -EINVAL;
+	mutex_lock(&pghot_tunables_lock);
+	pghot_target_nid = nid;
+	mutex_unlock(&pghot_tunables_lock);
+
+	*ppos += cnt;
+	return cnt;
+}
+
+static int pghot_target_nid_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%d\n", pghot_target_nid);
+	return 0;
+}
+
+static int pghot_target_nid_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, pghot_target_nid_show, NULL);
+}
+
+static const struct file_operations pghot_target_nid_fops = {
+	.open		= pghot_target_nid_open,
+	.write		= pghot_target_nid_write,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release,
+};
+
+static void pghot_src_enabled_update(unsigned int enabled)
+{
+	unsigned int changed = pghot_src_enabled ^ enabled;
+
+	if (changed & PGHOT_HINTFAULTS_ENABLED) {
+		if (enabled & PGHOT_HINTFAULTS_ENABLED)
+			static_branch_enable(&pghot_src_hintfaults);
+		else
+			static_branch_disable(&pghot_src_hintfaults);
+	}
+
+	if (changed & PGHOT_HWHINTS_ENABLED) {
+		if (enabled & PGHOT_HWHINTS_ENABLED)
+			static_branch_enable(&pghot_src_hwhints);
+		else
+			static_branch_disable(&pghot_src_hwhints);
+	}
+}
+
+static ssize_t pghot_src_enabled_write(struct file *filp, const char __user *ubuf,
+					   size_t cnt, loff_t *ppos)
+{
+	char buf[16];
+	unsigned int enabled;
+
+	if (cnt > 15)
+		cnt = 15;
+
+	if (copy_from_user(&buf, ubuf, cnt))
+		return -EFAULT;
+	buf[cnt] = '\0';
+
+	if (kstrtouint(buf, 0, &enabled))
+		return -EINVAL;
+
+	if (enabled & ~PGHOT_SRC_ENABLED_MASK)
+		return -EINVAL;
+
+	mutex_lock(&pghot_tunables_lock);
+	pghot_src_enabled_update(enabled);
+	pghot_src_enabled = enabled;
+	mutex_unlock(&pghot_tunables_lock);
+
+	*ppos += cnt;
+	return cnt;
+}
+
+static int pghot_src_enabled_show(struct seq_file *m, void *v)
+{
+	seq_printf(m, "%u\n", pghot_src_enabled);
+	return 0;
+}
+
+static int pghot_src_enabled_open(struct inode *inode, struct file *filp)
+{
+	return single_open(filp, pghot_src_enabled_show, NULL);
+}
+
+static const struct file_operations pghot_src_enabled_fops = {
+	.open		= pghot_src_enabled_open,
+	.write		= pghot_src_enabled_write,
+	.read		= seq_read,
+	.llseek		= seq_lseek,
+	.release	= seq_release,
+};
+
+void pghot_debug_init(void)
+{
+	debugfs_pghot = debugfs_create_dir("pghot", NULL);
+	debugfs_create_file("enabled_sources", 0644, debugfs_pghot, NULL,
+			    &pghot_src_enabled_fops);
+	debugfs_create_file("target_nid", 0644, debugfs_pghot, NULL,
+			    &pghot_target_nid_fops);
+	debugfs_create_file("freq_threshold", 0644, debugfs_pghot, NULL,
+			    &pghot_freq_th_fops);
+	debugfs_create_u32("kmigrated_sleep_ms", 0644, debugfs_pghot,
+			    &kmigrated_sleep_ms);
+	debugfs_create_u32("kmigrated_batch_nr", 0644, debugfs_pghot,
+			    &kmigrated_batch_nr);
+}
mm/pghot.c new file +87 -0
new file mode 100644
index 000000000000..dac9e6f3b61e
@@ -0,0 +1,479 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Maintains information about hot pages from slower tier nodes and
+ * promotes them.
+ *
+ * Per-PFN hotness information is stored for lower tier nodes in
+ * mem_section.
+ *
+ * In the default mode, a single byte (u8) is used to store
+ * the frequency of access and last access time. Promotions are done
+ * to a default toptier NID.
+ *
+ * A kernel thread named kmigrated is provided to migrate or promote
+ * the hot pages. kmigrated runs for each lower tier node. It iterates
+ * over the node's PFNs and  migrates pages marked for migration into
+ * their targeted nodes.
+ */
+#include <linux/mm.h>
+#include <linux/migrate.h>
+#include <linux/memory.h>
+#include <linux/memory-tiers.h>
+#include <linux/pghot.h>
+
+unsigned int pghot_target_nid = PGHOT_DEFAULT_NODE;
+unsigned int pghot_src_enabled;
+unsigned int pghot_freq_threshold = PGHOT_DEFAULT_FREQ_THRESHOLD;
+unsigned int kmigrated_sleep_ms = KMIGRATED_DEFAULT_SLEEP_MS;
+unsigned int kmigrated_batch_nr = KMIGRATED_DEFAULT_BATCH_NR;
+
+unsigned int sysctl_pghot_freq_window = PGHOT_DEFAULT_FREQ_WINDOW;
+
+DEFINE_STATIC_KEY_FALSE(pghot_src_hwhints);
+DEFINE_STATIC_KEY_FALSE(pghot_src_hintfaults);
+
+#ifdef CONFIG_SYSCTL
+static const struct ctl_table pghot_sysctls[] = {
+	{
+		.procname       = "pghot_promote_freq_window_ms",
+		.data           = &sysctl_pghot_freq_window,
+		.maxlen         = sizeof(unsigned int),
+		.mode           = 0644,
+		.proc_handler   = proc_dointvec_minmax,
+		.extra1         = SYSCTL_ZERO,
+	},
+};
+#endif
+
+static bool kmigrated_started __ro_after_init;
+
+/**
+ * pghot_record_access() - Record page accesses from lower tier memory
+ * for the purpose of tracking page hotness and subsequent promotion.
+ *
+ * @pfn: PFN of the page
+ * @nid: Unused
+ * @src: The identifier of the sub-system that reports the access
+ * @now: Access time in jiffies
+ *
+ * Updates the frequency and time of access and marks the page as
+ * ready for migration if the frequency crosses a threshold. The pages
+ * marked for migration are migrated by kmigrated kernel thread.
+ *
+ * Return: 0 on success and -EINVAL on failure to record the access.
+ */
+int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
+{
+	struct mem_section *ms;
+	struct folio *folio;
+	phi_t *phi, *hot_map;
+	struct page *page;
+
+	if (!kmigrated_started)
+		return 0;
+
+	if (!pghot_nid_valid(nid))
+		return -EINVAL;
+
+	switch (src) {
+	case PGHOT_HINTFAULTS:
+		if (!static_branch_unlikely(&pghot_src_hintfaults))
+			return 0;
+		count_vm_event(PGHOT_RECORDED_HINTFAULTS);
+		break;
+	case PGHOT_HWHINTS:
+		if (!static_branch_unlikely(&pghot_src_hwhints))
+			return 0;
+		count_vm_event(PGHOT_RECORDED_HWHINTS);

mm/pghot.c line 97

Quoted 10 lines
mm/pghot.c new file +10 -0
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	/*
+	 * Record only accesses from lower tiers.
+	 */
+	if (node_is_toptier(pfn_to_nid(pfn)))
+		return 0;

Just a thought—could we check this at the beginning of the function, before the switch case?

Quoted 154 lines
mm/pghot.c new file +154 -0
+
+	/*
+	 * Reject the non-migratable pages right away.
+	 */
+	page = pfn_to_online_page(pfn);
+	if (!page || is_zone_device_page(page))
+		return 0;
+
+	folio = page_folio(page);
+	if (!folio_try_get(folio))
+		return 0;
+
+	if (unlikely(page_folio(page) != folio))
+		goto out;
+
+	if (!folio_test_lru(folio))
+		goto out;
+
+	/* Get the hotness slot corresponding to the 1st PFN of the folio */
+	pfn = folio_pfn(folio);
+	ms = __pfn_to_section(pfn);
+	if (!ms || !ms->hot_map)
+		goto out;
+
+	hot_map = (phi_t *)(((unsigned long)(ms->hot_map)) & ~PGHOT_SECTION_HOT_MASK);
+	phi = &hot_map[pfn % PAGES_PER_SECTION];
+
+	count_vm_event(PGHOT_RECORDED_ACCESSES);
+
+	/*
+	 * Update the hotness parameters.
+	 */
+	if (pghot_update_record(phi, nid, now)) {
+		set_bit(PGHOT_SECTION_HOT_BIT, (unsigned long *)&ms->hot_map);
+		set_bit(PGDAT_KMIGRATED_ACTIVATE, &page_pgdat(page)->flags);
+	}
+out:
+	folio_put(folio);
+	return 0;
+}
+
+static int pghot_get_hotness(unsigned long pfn, int *nid, int *freq,
+			     unsigned long *time)
+{
+	phi_t *phi, *hot_map;
+	struct mem_section *ms;
+
+	ms = __pfn_to_section(pfn);
+	if (!ms || !ms->hot_map)
+		return -EINVAL;
+
+	hot_map = (phi_t *)(((unsigned long)(ms->hot_map)) & ~PGHOT_SECTION_HOT_MASK);
+	phi = &hot_map[pfn % PAGES_PER_SECTION];
+
+	return pghot_get_record(phi, nid, freq, time);
+}
+
+/*
+ * Walks the PFNs of the zone, isolates and migrates them in batches.
+ */
+static void kmigrated_walk_zone(unsigned long start_pfn, unsigned long end_pfn,
+				int src_nid)
+{
+	struct mem_cgroup *cur_memcg = NULL;
+	int cur_nid = NUMA_NO_NODE;
+	LIST_HEAD(migrate_list);
+	int batch_count = 0;
+	struct folio *folio;
+	struct page *page;
+	unsigned long pfn;
+
+	pfn = start_pfn;
+	do {
+		int nid = NUMA_NO_NODE, nr = 1;
+		struct mem_cgroup *memcg;
+		unsigned long time = 0;
+		int freq = 0;
+
+		if (!pfn_valid(pfn))
+			goto out_next;
+
+		page = pfn_to_online_page(pfn);
+		if (!page)
+			goto out_next;
+
+		folio = page_folio(page);
+		if (!folio_try_get(folio))
+			goto out_next;
+
+		if (unlikely(page_folio(page) != folio)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		nr = folio_nr_pages(folio);
+		if (folio_nid(folio) != src_nid) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (!folio_test_lru(folio)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (pghot_get_hotness(pfn, &nid, &freq, &time)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (nid == NUMA_NO_NODE)
+			nid = pghot_target_nid;
+
+		if (folio_nid(folio) == nid) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (migrate_misplaced_folio_prepare(folio, NULL, nid)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		memcg = folio_memcg(folio);
+		if (cur_nid == NUMA_NO_NODE) {
+			cur_nid = nid;
+			cur_memcg = memcg;
+		}
+
+		/* If NID or memcg changed, flush the previous batch first */
+		if (cur_nid != nid || cur_memcg != memcg) {
+			if (!list_empty(&migrate_list))
+				migrate_misplaced_folios_batch(&migrate_list, cur_nid);
+			cur_nid = nid;
+			cur_memcg = memcg;
+			batch_count = 0;
+			cond_resched();
+		}
+
+		list_add(&folio->lru, &migrate_list);
+		folio_put(folio);
+
+		if (++batch_count > kmigrated_batch_nr) {
+			migrate_misplaced_folios_batch(&migrate_list, cur_nid);
+			batch_count = 0;
+			cond_resched();
+		}
+out_next:
+		pfn += nr;
+	} while (pfn < end_pfn);
+	if (!list_empty(&migrate_list))
+		migrate_misplaced_folios_batch(&migrate_list, cur_nid);
+}
+

mm/pghot.c line 261

Quoted 10 lines
mm/pghot.c new file +10 -0
+static void kmigrated_do_work(pg_data_t *pgdat)
+{
+	unsigned long section_nr, s_begin, start_pfn;
+	struct mem_section *ms;
+	int nid;
+
+	clear_bit(PGDAT_KMIGRATED_ACTIVATE, &pgdat->flags);
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
+		start_pfn = section_nr_to_pfn(section_nr);

I may be missing something, but in pghot_setup_hot_map() and kmigrated_do_work() we seem to iterate over all memory sections. On large memory systems, could this become a bottleneck right?

Since hot_map is allocated only for lower-tier memory and the hotness information is primarily used there, would it make sense to skip scanning higher-tier sections?

for_each_online_node(nid) {
        if (node_is_toptier(nid))
            continue;

        start_pfn = node_start_pfn(nid);
        end_pfn = node_end_pfn(nid);

        s_begin = pfn_to_section_nr(start_pfn); for_each_present_section_nr(s_begin, section_nr) { }
}

Would this approach be reasonable, or am I overlooking something?

Quoted 37 lines
mm/pghot.c new file +37 -0
+		ms = __nr_to_section(section_nr);
+
+		if (!pfn_valid(start_pfn))
+			continue;
+
+		nid = pfn_to_nid(start_pfn);
+		if (node_is_toptier(nid) || nid != pgdat->node_id)
+			continue;
+
+		if (!test_and_clear_bit(PGHOT_SECTION_HOT_BIT, (unsigned long *)&ms->hot_map))
+			continue;
+
+		kmigrated_walk_zone(start_pfn, start_pfn + PAGES_PER_SECTION,
+				    pgdat->node_id);
+	}
+}
+
+static inline bool kmigrated_work_requested(pg_data_t *pgdat)
+{
+	return test_bit(PGDAT_KMIGRATED_ACTIVATE, &pgdat->flags);
+}
+
+/*
+ * Per-node kthread that iterates over its PFNs and migrates the
+ * pages that have been marked for migration.
+ */
+static int kmigrated(void *p)
+{
+	pg_data_t *pgdat = p;
+
+	while (!kthread_should_stop()) {
+		long timeout = msecs_to_jiffies(READ_ONCE(kmigrated_sleep_ms));
+
+		if (wait_event_timeout(pgdat->kmigrated_wait, kmigrated_work_requested(pgdat),
+				       timeout))
+			kmigrated_do_work(pgdat);
+	}

mm/pghot.c line 308

Quoted 10 lines
mm/pghot.c new file +10 -0
+	return 0;
+}
+
+static int kmigrated_run(int nid)
+{
+	pg_data_t *pgdat = NODE_DATA(nid);
+	int ret;
+
+	if (node_is_toptier(nid))
+		return 0;

I might be missing something, but since this function is only called from pghot_init(), would it make sense to check the condition before calling kmigrated_run() to avoid the function call overhead?

Quoted 56 lines
mm/pghot.c new file +56 -0
+
+	if (!pgdat->kmigrated) {
+		pgdat->kmigrated = kthread_create_on_node(kmigrated, pgdat, nid,
+							  "kmigrated%d", nid);
+		if (IS_ERR(pgdat->kmigrated)) {
+			ret = PTR_ERR(pgdat->kmigrated);
+			pgdat->kmigrated = NULL;
+			pr_err("Failed to start kmigrated%d, ret %d\n", nid, ret);
+			return ret;
+		}
+		pr_info("pghot: Started kmigrated thread for node %d\n", nid);
+	}
+	wake_up_process(pgdat->kmigrated);
+	return 0;
+}
+
+static void pghot_free_hot_map(struct mem_section *ms)
+{
+	kfree((void *)((unsigned long)ms->hot_map & ~PGHOT_SECTION_HOT_MASK));
+	ms->hot_map = NULL;
+}
+
+static int pghot_alloc_hot_map(struct mem_section *ms, int nid)
+{
+	ms->hot_map = kcalloc_node(PAGES_PER_SECTION, PGHOT_RECORD_SIZE, GFP_KERNEL,
+				   nid);
+	if (!ms->hot_map)
+		return -ENOMEM;
+	return 0;
+}
+
+static void pghot_offline_sec_hotmap(unsigned long start_pfn,
+				     unsigned long nr_pages)
+{
+	unsigned long start, end, pfn;
+	struct mem_section *ms;
+
+	start = SECTION_ALIGN_DOWN(start_pfn);
+	end = SECTION_ALIGN_UP(start_pfn + nr_pages);
+
+	for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {
+		ms = __pfn_to_section(pfn);
+		if (!ms || !ms->hot_map)
+			continue;
+
+		pghot_free_hot_map(ms);
+	}
+}
+
+static int pghot_online_sec_hotmap(unsigned long start_pfn,
+				   unsigned long nr_pages)
+{
+	int nid = pfn_to_nid(start_pfn);
+	unsigned long start, end, pfn;
+	struct mem_section *ms;
+	int fail = 0;

mm/pghot.c line 374

Quoted 10 lines
mm/pghot.c new file +10 -0
+
+	start = SECTION_ALIGN_DOWN(start_pfn);
+	end = SECTION_ALIGN_UP(start_pfn + nr_pages);
+
+	for (pfn = start; !fail && pfn < end; pfn += PAGES_PER_SECTION) {
+		ms = __pfn_to_section(pfn);
+		if (!ms || ms->hot_map)
+			continue;
+
+		fail = pghot_alloc_hot_map(ms, nid);

I may be missing something, but after pghot_alloc_hot_map fails, we continue the loop. Would it make sense to break and go to the cleanup logic instead?

-Donet

Quoted 121 lines
mm/pghot.c new file +105 -0
+	}
+
+	if (!fail)
+		return 0;
+
+	/* rollback */
+	end = pfn - PAGES_PER_SECTION;
+	for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {
+		ms = __pfn_to_section(pfn);
+		if (ms && ms->hot_map)
+			pghot_free_hot_map(ms);
+	}
+	return -ENOMEM;
+}
+
+static int pghot_memhp_callback(struct notifier_block *self,
+				unsigned long action, void *arg)
+{
+	struct memory_notify *mn = arg;
+	int ret = 0;
+
+	switch (action) {
+	case MEM_GOING_ONLINE:
+		ret = pghot_online_sec_hotmap(mn->start_pfn, mn->nr_pages);
+		break;
+	case MEM_OFFLINE:
+	case MEM_CANCEL_ONLINE:
+		pghot_offline_sec_hotmap(mn->start_pfn, mn->nr_pages);
+		break;
+	}
+
+	return notifier_from_errno(ret);
+}
+
+static void pghot_destroy_hot_map(void)
+{
+	unsigned long section_nr, s_begin;
+	struct mem_section *ms;
+
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
+		ms = __nr_to_section(section_nr);
+		pghot_free_hot_map(ms);
+	}
+}
+
+static int pghot_setup_hot_map(void)
+{
+	unsigned long section_nr, s_begin, start_pfn;
+	struct mem_section *ms;
+	int nid;
+
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
+		ms = __nr_to_section(section_nr);
+		start_pfn = section_nr_to_pfn(section_nr);
+		nid = pfn_to_nid(start_pfn);
+
+		if (node_is_toptier(nid) || !pfn_valid(start_pfn))
+			continue;
+
+		if (pghot_alloc_hot_map(ms, nid))
+			goto out_free_hot_map;
+	}
+	hotplug_memory_notifier(pghot_memhp_callback, DEFAULT_CALLBACK_PRI);
+	return 0;
+
+out_free_hot_map:
+	pghot_destroy_hot_map();
+	return -ENOMEM;
+}
+
+static int __init pghot_init(void)
+{
+	pg_data_t *pgdat;
+	int nid, ret;
+
+	ret = pghot_setup_hot_map();
+	if (ret)
+		return ret;
+
+	for_each_node_state(nid, N_MEMORY) {
+		ret = kmigrated_run(nid);
+		if (ret)
+			goto out_stop_kthread;
+	}
+	register_sysctl_init("vm", pghot_sysctls);
+	pghot_debug_init();
+
+	kmigrated_started = true;
+	return 0;
+
+out_stop_kthread:
+	for_each_node_state(nid, N_MEMORY) {
+		pgdat = NODE_DATA(nid);
+		if (pgdat->kmigrated) {
+			kthread_stop(pgdat->kmigrated);
+			pgdat->kmigrated = NULL;
+		}
+	}
+	pghot_destroy_hot_map();
+	return ret;
+}
+
+late_initcall_sync(pghot_init)
mm/vmstat.c +5 -0
index 86b14b0f77b5..d3fbe2a5d0e6 100644
@@ -1486,6 +1486,11 @@ const char * const vmstat_text[] = {
  	[I(KSTACK_REST)]			= "kstack_rest",
  #endif
  #endif
+#ifdef CONFIG_PGHOT
+	[I(PGHOT_RECORDED_ACCESSES)]		= "pghot_recorded_accesses",
+	[I(PGHOT_RECORDED_HINTFAULTS)]		= "pghot_recorded_hintfaults",
+	[I(PGHOT_RECORDED_HWHINTS)]		= "pghot_recorded_hwhints",
+#endif /* CONFIG_PGHOT */
  #undef I
  #endif /* CONFIG_VM_EVENT_COUNTERS */
  };
maintainer
#13 lore raw

Re: [RFC PATCH v6 3/5] mm: Hot page tracking and promotion - pghot

Quoted 12 lines · Donet Tom

mm/pghot.c line 97

Quoted 6 lines
mm/pghot.c new file +6 -0
+
+	/*
+	 * Record only accesses from lower tiers.
+	 */
+	if (node_is_toptier(pfn_to_nid(pfn)))
+		return 0;

Just a thought—could we check this at the beginning of the function, before the switch case?

Can you please trim responses to omit un-related code? Not all of us use mail clients that auto-collapse quoted regions and it takes a lot of scrolling to reach here n_n;;;

Quoted 7 lines
Quoted 7 lines
mm/pghot.c new file +7 -0
+static void kmigrated_do_work(pg_data_t *pgdat)
+{
+	unsigned long section_nr, s_begin, start_pfn;
+	struct mem_section *ms;
+	int nid;
+
+	clear_bit(PGDAT_KMIGRATED_ACTIVATE, &pgdat->flags);
Quoted 9 lines

mm/pghot.c line 261

Quoted 3 lines
mm/pghot.c new file +3 -0
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
+		start_pfn = section_nr_to_pfn(section_nr);

I may be missing something, but in pghot_setup_hot_map() and kmigrated_do_work() we seem to iterate over all memory sections. On large memory systems, could this become a bottleneck right?

I think this is going to end up being a function of page / section size.

I would hesitate to over-optimize on this early-on, otherwise we'll find ourselves re-building a bunch of DAMON functionality that chunks memory into regions and such.

But it is a concern.

Quoted 12 lines

Since hot_map is allocated only for lower-tier memory and the hotness information is primarily used there, would it make sense to skip scanning higher-tier sections?

for_each_online_node(nid) {
        if (node_is_toptier(nid))
            continue;

        start_pfn = node_start_pfn(nid);
        end_pfn = node_end_pfn(nid);

Quoted 7 lines

        s_begin = pfn_to_section_nr(start_pfn); for_each_present_section_nr(s_begin, section_nr) { }
}

Would this approach be reasonable, or am I overlooking something?

re: your note later on - kmigrated doesn't run for top tier nodes

kmigrated_run() { if (node_is_toptier(nid)) return 0; }

Quoted 4 lines
Quoted 4 lines
mm/pghot.c new file +4 -0
+
+static int kmigrated_run(int nid)
+{
+	pg_data_t *pgdat = NODE_DATA(nid);
Quoted 9 lines

mm/pghot.c line 308

Quoted 4 lines
mm/pghot.c new file +4 -0
+	int ret;
+
+	if (node_is_toptier(nid))
+		return 0;

I might be missing something, but since this function is only called from pghot_init(), would it make sense to check the condition before calling kmigrated_run() to avoid the function call overhead?

this gets optimized out when static is inlined and optimized.

~Gregory

#14 lore raw

Re: [RFC PATCH v6 3/5] mm: Hot page tracking and promotion - pghot

Hi Gregory

Quoted 12 lines · Gregory Price
Quoted 11 lines · Donet Tom

mm/pghot.c line 97

Quoted 6 lines
mm/pghot.c new file +6 -0
+
+	/*
+	 * Record only accesses from lower tiers.
+	 */
+	if (node_is_toptier(pfn_to_nid(pfn)))
+		return 0;

Just a thought—could we check this at the beginning of the function, before the switch case?

Quoted 3 lines

Can you please trim responses to omit un-related code? Not all of us use mail clients that auto-collapse quoted regions and it takes a lot of scrolling to reach here n_n;;;

Apologies for that, and thanks for pointing it out. I’ll keep the responses concise and include only the relevant parts going forward.

Quoted 16 lines
Quoted 16 lines

mm/pghot.c line 261

Quoted 11 lines
mm/pghot.c new file +10 -0
+static void kmigrated_do_work(pg_data_t *pgdat)
+{
+	unsigned long section_nr, s_begin, start_pfn;
+	struct mem_section *ms;
+	int nid;
+
+	clear_bit(PGDAT_KMIGRATED_ACTIVATE, &pgdat->flags);
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
+		start_pfn = section_nr_to_pfn(section_nr);

I may be missing something, but in pghot_setup_hot_map() and kmigrated_do_work() we seem to iterate over all memory sections. On large memory systems, could this become a bottleneck right?

Quoted 3 lines

I think this is going to end up being a function of page / section size.

I would hesitate to over-optimize on this early-on, otherwise we'll find

Sure.

Quoted 39 lines

ourselves re-building a bunch of DAMON functionality that chunks memory into regions and such.

But it is a concern.

Quoted 11 lines

Since hot_map is allocated only for lower-tier memory and the hotness information is primarily used there, would it make sense to skip scanning higher-tier sections?

for_each_online_node(nid) {
        if (node_is_toptier(nid))
            continue;

        start_pfn = node_start_pfn(nid);
        end_pfn = node_end_pfn(nid);

Quoted 7 lines

        s_begin = pfn_to_section_nr(start_pfn); for_each_present_section_nr(s_begin, section_nr) { }
}

Would this approach be reasonable, or am I overlooking something?

re: your note later on - kmigrated doesn't run for top tier nodes

kmigrated_run() { if (node_is_toptier(nid)) return 0; }

Quoted 12 lines

mm/pghot.c line 308

Quoted 8 lines
mm/pghot.c new file +8 -0
+
+static int kmigrated_run(int nid)
+{
+	pg_data_t *pgdat = NODE_DATA(nid);
+	int ret;
+
+	if (node_is_toptier(nid))
+		return 0;

I might be missing something, but since this function is only called from pghot_init(), would it make sense to check the condition before calling kmigrated_run() to avoid the function call overhead?

Quoted 2 lines

this gets optimized out when static is inlined and optimized.

Thanks, Gregory.

I might be missing something, but since kmigrated_run() is not marked inline (static int kmigrated_run()), will it still be optimized or inlined by the compiler without the inline keyword?- Donet

Quoted 3 lines

~Gregory

#15 lore raw

Re: [RFC PATCH v6 3/5] mm: Hot page tracking and promotion - pghot

In reply to Donet Tom
Quoted 29 lines · Donet Tom
Quoted 29 lines
mm/pghot.c new file +29 -0
+int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
+{
+    struct mem_section *ms;
+    struct folio *folio;
+    phi_t *phi, *hot_map;
+    struct page *page;
+
+    if (!kmigrated_started)
+        return 0;
+
+    if (!pghot_nid_valid(nid))
+        return -EINVAL;
+
+    switch (src) {
+    case PGHOT_HINTFAULTS:
+        if (!static_branch_unlikely(&pghot_src_hintfaults))
+            return 0;
+        count_vm_event(PGHOT_RECORDED_HINTFAULTS);
+        break;
+    case PGHOT_HWHINTS:
+        if (!static_branch_unlikely(&pghot_src_hwhints))
+            return 0;
+        count_vm_event(PGHOT_RECORDED_HWHINTS);
+        break;
+    default:
+        return -EINVAL;
+    }
+
+    /*
Quoted 8 lines
Quoted 4 lines

+     * Record only accesses from lower tiers.
+     */
+    if (node_is_toptier(pfn_to_nid(pfn)))
+        return 0;

Just a thought—could we check this at the beginning of the function, before the switch case?

I am accumulating two stats here: How many hot page intimations pghot obtained that are attributable to different sources

and

out of them, how many turned out to be actionable

Quoted 23 lines
Quoted 23 lines
mm/pghot.c new file +21 -0
+
+    /*
+     * Reject the non-migratable pages right away.
+     */
+    page = pfn_to_online_page(pfn);
+    if (!page || is_zone_device_page(page))
+        return 0;
+
+    folio = page_folio(page);
+    if (!folio_try_get(folio))
+        return 0;
+
+    if (unlikely(page_folio(page) != folio))
+        goto out;
+
+    if (!folio_test_lru(folio))
+        goto out;
+
+    /* Get the hotness slot corresponding to the 1st PFN of the folio */
+    pfn = folio_pfn(folio);
+    ms = __pfn_to_section(pfn);
Quoted 8 lines
Quoted 8 lines

+    if (!ms || !ms->hot_map)
+        goto out;
+
+    hot_map = (phi_t *)(((unsigned long)(ms->hot_map)) & ~PGHOT_SECTION_HOT_MASK);
+    phi = &hot_map[pfn % PAGES_PER_SECTION];
+
+    count_vm_event(PGHOT_RECORDED_ACCESSES);

which is this ^

Quoted 27 lines

mm/pghot.c line 261

Quoted 10 lines
mm/pghot.c new file +10 -0
+static void kmigrated_do_work(pg_data_t *pgdat)
+{
+    unsigned long section_nr, s_begin, start_pfn;
+    struct mem_section *ms;
+    int nid;
+
+    clear_bit(PGDAT_KMIGRATED_ACTIVATE, &pgdat->flags);
+    s_begin = next_present_section_nr(-1);
+    for_each_present_section_nr(s_begin, section_nr) {
+        start_pfn = section_nr_to_pfn(section_nr);

I may be missing something, but in pghot_setup_hot_map() and kmigrated_do_work() we seem to iterate over all memory sections. On large memory systems, could this become a bottleneck right?

Since hot_map is allocated only for lower-tier memory and the hotness information is primarily used there, would it make sense to skip scanning higher-tier sections?

for_each_online_node(nid) {
        if (node_is_toptier(nid))
            continue;

        start_pfn = node_start_pfn(nid);
        end_pfn = node_end_pfn(nid);

Quoted 6 lines

        s_begin = pfn_to_section_nr(start_pfn); for_each_present_section_nr(s_begin, section_nr) { }
}

Would this approach be reasonable, or am I overlooking something?

I didn't just yet optimize the walk. Since there is one kmigrated thread per lower tier, this routine already is aware of which node to scan. We can limit the section walk to that node instead. Something like this:

static void kmigrated_do_work(pg_data_t *pgdat)

{
	unsigned long section_nr, s_begin, start_pfn, end_pfn;
	struct mem_section *ms;
	int nid = pgdat->node_id;

	start_pfn = SECTION_ALIGN_DOWN(node_start_pfn(nid));
	end_pfn = SECTION_ALIGN_UP(start_pfn + node_end_pfn(nid));

	for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
		section_nr = pfn_to_section_nr(pfn);
		
		if (!present_section_nr(section_nr))
			continue;

		ms = __nr_to_section(section_nr);
		
		...

		kmigrated_walk_zone(pfn, pfn + PAGES_PER_SECTION, nid);
	}
}
Quoted 12 lines
Quoted 12 lines
mm/pghot.c new file +12 -0
+static int pghot_online_sec_hotmap(unsigned long start_pfn,
+                   unsigned long nr_pages)
+{
+    int nid = pfn_to_nid(start_pfn);
+    unsigned long start, end, pfn;
+    struct mem_section *ms;
+    int fail = 0;
+
+    start = SECTION_ALIGN_DOWN(start_pfn);
+    end = SECTION_ALIGN_UP(start_pfn + nr_pages);
+
+    for (pfn = start; !fail && pfn < end; pfn += PAGES_PER_SECTION) {
Quoted 8 lines
Quoted 5 lines

+        ms = __pfn_to_section(pfn);
+        if (!ms || ms->hot_map)
+            continue;
+
+        fail = pghot_alloc_hot_map(ms, nid);

I may be missing something, but after pghot_alloc_hot_map fails, we continue the loop. Would it make sense to break and go to the cleanup logic instead?

There is a !fail check in the for-loop due to which we break when alloc fails.

Regards,
Bharata.

#16 lore raw

Re: [RFC PATCH v6 3/5] mm: Hot page tracking and promotion - pghot

Hi Bharata

Quoted 37 lines · Bharata B Rao
Quoted 28 lines · Donet Tom
Quoted 28 lines
mm/pghot.c new file +28 -0
+int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
+{
+    struct mem_section *ms;
+    struct folio *folio;
+    phi_t *phi, *hot_map;
+    struct page *page;
+
+    if (!kmigrated_started)
+        return 0;
+
+    if (!pghot_nid_valid(nid))
+        return -EINVAL;
+
+    switch (src) {
+    case PGHOT_HINTFAULTS:
+        if (!static_branch_unlikely(&pghot_src_hintfaults))
+            return 0;
+        count_vm_event(PGHOT_RECORDED_HINTFAULTS);
+        break;
+    case PGHOT_HWHINTS:
+        if (!static_branch_unlikely(&pghot_src_hwhints))
+            return 0;
+        count_vm_event(PGHOT_RECORDED_HWHINTS);
+        break;
+    default:
+        return -EINVAL;
+    }
+
Quoted 8 lines
Quoted 5 lines

+    /*
+     * Record only accesses from lower tiers.
+     */
+    if (node_is_toptier(pfn_to_nid(pfn)))
+        return 0;

Just a thought—could we check this at the beginning of the function, before the switch case?

Quoted 7 lines

I am accumulating two stats here: How many hot page intimations pghot obtained that are attributable to different sources

and

out of them, how many turned out to be actionable

Understood. Thanks for the clarification.

Quoted 84 lines
Quoted 23 lines
Quoted 23 lines
mm/pghot.c new file +21 -0
+
+    /*
+     * Reject the non-migratable pages right away.
+     */
+    page = pfn_to_online_page(pfn);
+    if (!page || is_zone_device_page(page))
+        return 0;
+
+    folio = page_folio(page);
+    if (!folio_try_get(folio))
+        return 0;
+
+    if (unlikely(page_folio(page) != folio))
+        goto out;
+
+    if (!folio_test_lru(folio))
+        goto out;
+
+    /* Get the hotness slot corresponding to the 1st PFN of the folio */
+    pfn = folio_pfn(folio);
+    ms = __pfn_to_section(pfn);
Quoted 8 lines
Quoted 8 lines

+    if (!ms || !ms->hot_map)
+        goto out;
+
+    hot_map = (phi_t *)(((unsigned long)(ms->hot_map)) & ~PGHOT_SECTION_HOT_MASK);
+    phi = &hot_map[pfn % PAGES_PER_SECTION];
+
+    count_vm_event(PGHOT_RECORDED_ACCESSES);

which is this ^

Quoted 26 lines

mm/pghot.c line 261

Quoted 10 lines
mm/pghot.c new file +10 -0
+static void kmigrated_do_work(pg_data_t *pgdat)
+{
+    unsigned long section_nr, s_begin, start_pfn;
+    struct mem_section *ms;
+    int nid;
+
+    clear_bit(PGDAT_KMIGRATED_ACTIVATE, &pgdat->flags);
+    s_begin = next_present_section_nr(-1);
+    for_each_present_section_nr(s_begin, section_nr) {
+        start_pfn = section_nr_to_pfn(section_nr);

I may be missing something, but in pghot_setup_hot_map() and kmigrated_do_work() we seem to iterate over all memory sections. On large memory systems, could this become a bottleneck right?

Since hot_map is allocated only for lower-tier memory and the hotness information is primarily used there, would it make sense to skip scanning higher-tier sections?

for_each_online_node(nid) {
        if (node_is_toptier(nid))
            continue;

        start_pfn = node_start_pfn(nid);
        end_pfn = node_end_pfn(nid);

Quoted 6 lines

        s_begin = pfn_to_section_nr(start_pfn); for_each_present_section_nr(s_begin, section_nr) { }
}

Would this approach be reasonable, or am I overlooking something?

I didn't just yet optimize the walk. Since there is one kmigrated thread per lower tier, this routine already is aware of which node to scan. We can limit the section walk to that node instead. Something like this:

static void kmigrated_do_work(pg_data_t *pgdat)

{
	unsigned long section_nr, s_begin, start_pfn, end_pfn;
	struct mem_section *ms;
	int nid = pgdat->node_id;

	start_pfn = SECTION_ALIGN_DOWN(node_start_pfn(nid));
	end_pfn = SECTION_ALIGN_UP(start_pfn + node_end_pfn(nid));

	for (pfn = start_pfn; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
		section_nr = pfn_to_section_nr(pfn);
		
		if (!present_section_nr(section_nr))
			continue;
Quoted 7 lines
		ms = __nr_to_section(section_nr);
		
		...

		kmigrated_walk_zone(pfn, pfn + PAGES_PER_SECTION, nid);
	}
}

Thanks. This looks good to me.

Quoted 20 lines
Quoted 12 lines
Quoted 12 lines
mm/pghot.c new file +11 -0
+static int pghot_online_sec_hotmap(unsigned long start_pfn,
+                   unsigned long nr_pages)
+{
+    int nid = pfn_to_nid(start_pfn);
+    unsigned long start, end, pfn;
+    struct mem_section *ms;
+    int fail = 0;
+
+    start = SECTION_ALIGN_DOWN(start_pfn);
+    end = SECTION_ALIGN_UP(start_pfn + nr_pages);
+
Quoted 8 lines
Quoted 6 lines

+    for (pfn = start; !fail && pfn < end; pfn += PAGES_PER_SECTION) { +        ms = __pfn_to_section(pfn);
+        if (!ms || ms->hot_map)
+            continue;
+
+        fail = pghot_alloc_hot_map(ms, nid);

I may be missing something, but after pghot_alloc_hot_map fails, we continue the loop. Would it make sense to break and go to the cleanup logic instead?

Quoted 1 line

There is a !fail check in the for-loop due to which we break when alloc fails.

My bad, I missed it.

-Donet

Quoted 3 lines

Regards,
Bharata.

#17 lore raw

RFC PATCH v6 4/5 mm: pghot: Precision mode for pghot

In reply to Bharata B Rao

Default pghot stores hotness in a 1‑byte record per PFN, limiting frequency to 2 bits, time to a 5‑bit bucket, and preventing storage of per‑PFN toptier NID. This restricts time granularity and forces all promotions to use the global pghot_target_nid.

This patch adds an optional precision mode (CONFIG_PGHOT_PRECISE) that expands the hotness record to 4 bytes (u32) and provides:

- 10‑bit NID field for per‑PFN promotion target,
- 3‑bit frequency field (freq_threshold range 1–7),
- 14‑bit time field offering finer recency tracking,
- MSB migrate‑ready bit.

Precision mode improves placement accuracy on systems with multiple toptier nodes and provides higher‑resolution hotness tracking, at the cost of increasing metadata to 4 bytes per PFN.

Documentation, tunables, and the record layout are updated accordingly.

---
 Documentation/admin-guide/mm/pghot.txt |  4 +-
 include/linux/mmzone.h                 |  2 +-
 include/linux/pghot.h                  | 31 ++++++++++
 mm/Kconfig                             | 11 ++++
 mm/Makefile                            |  7 ++-
 mm/pghot-precise.c                     | 81 ++++++++++++++++++++++++++
 mm/pghot.c                             | 13 +++--
 7 files changed, 141 insertions(+), 8 deletions(-)
 create mode 100644 mm/pghot-precise.c
Documentation/admin-guide/mm/pghot.txt +2 -2
index 5f51dd1d4d45..7b84e911afe7 100644
@@ -37,7 +37,7 @@ Path: /sys/kernel/debug/pghot/
 
 3. **freq_threshold**
    - Minimum access frequency before a page is marked ready for promotion.
-   - Range: 1 to 3
+   - Range: 1 to 3 in default mode, 1 to 7 in precision mode.
    - Default: 2
    - Example:
      # echo 3 > /sys/kernel/debug/pghot/freq_threshold
@@ -59,7 +59,7 @@ Path: /proc/sys/vm/pghot_promote_freq_window_ms
 - Controls the time window (in ms) for counting access frequency. A page is
   considered hot only when **freq_threshold** number of accesses occur with
   this time period.
-- Default: 3000 (3 seconds)
+- Default: 3000 (3 seconds) in default mode and 5000 (5s) in precision mode.
 - Example:
   # sysctl vm.pghot_promote_freq_window_ms=3000
 
include/linux/mmzone.h +1 -1
index d7ed60956543..61fd259d9897 100644
@@ -1938,7 +1938,7 @@ struct mem_section {
 #ifdef CONFIG_PGHOT
 	/*
 	 * Per-PFN hotness data for this section.
-	 * Array of phi_t (u8 in default mode).
+	 * Array of phi_t (u8 in default mode, u32 in precision mode).
 	 * LSB is used as PGHOT_SECTION_HOT_BIT flag.
 	 */
 	void *hot_map;
include/linux/pghot.h +31 -0
index 525d4dd28fc1..2e1742b8caee 100644
@@ -35,6 +35,36 @@ DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
 
 #define PGHOT_DEFAULT_NODE		0
 
+#if defined(CONFIG_PGHOT_PRECISE)
+#define PGHOT_DEFAULT_FREQ_WINDOW	(5 * MSEC_PER_SEC)
+
+/*
+ * Bits 0-26 are used to store nid, frequency and time.
+ * Bits 27-30 are unused now.
+ * Bit 31 is used to indicate the page is ready for migration.
+ */
+#define PGHOT_MIGRATE_READY		31
+
+#define PGHOT_NID_WIDTH			10
+#define PGHOT_FREQ_WIDTH		3
+/* time is stored in 14 bits which can represent up to 16s with HZ=1000 */
+#define PGHOT_TIME_WIDTH		14
+
+#define PGHOT_NID_SHIFT			0
+#define PGHOT_FREQ_SHIFT		(PGHOT_NID_SHIFT + PGHOT_NID_WIDTH)
+#define PGHOT_TIME_SHIFT		(PGHOT_FREQ_SHIFT + PGHOT_FREQ_WIDTH)
+
+#define PGHOT_NID_MASK			GENMASK(PGHOT_NID_WIDTH - 1, 0)
+#define PGHOT_FREQ_MASK			GENMASK(PGHOT_FREQ_WIDTH - 1, 0)
+#define PGHOT_TIME_MASK			GENMASK(PGHOT_TIME_WIDTH - 1, 0)
+
+#define PGHOT_NID_MAX			((1 << PGHOT_NID_WIDTH) - 1)
+#define PGHOT_FREQ_MAX			((1 << PGHOT_FREQ_WIDTH) - 1)
+#define PGHOT_TIME_MAX			((1 << PGHOT_TIME_WIDTH) - 1)
+
+typedef u32 phi_t;
+
+#else	/* !CONFIG_PGHOT_PRECISE */
 #define PGHOT_DEFAULT_FREQ_WINDOW	(3 * MSEC_PER_SEC)
 
 /*
@@ -61,6 +91,7 @@ DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
 #define PGHOT_TIME_MAX			((1 << PGHOT_TIME_WIDTH) - 1)
 
 typedef u8 phi_t;
+#endif /* CONFIG_PGHOT_PRECISE */
 
 #define PGHOT_RECORD_SIZE		sizeof(phi_t)
 
mm/Kconfig +11 -0
index 4aeab6aee535..14383bb1d890 100644
@@ -1485,6 +1485,17 @@ config PGHOT
 	  This adds 1 byte of metadata overhead per page in lower-tier
 	  memory nodes.
 
+config PGHOT_PRECISE
+	bool "Hot page tracking precision mode"
+	def_bool n
+	depends on PGHOT
+	help
+	  Enables precision mode for tracking hot pages with pghot sub-system.
+	  Adds fine-grained access time tracking and explicit toptier target
+	  NID tracking. Precise hot page tracking comes at the cost of using
+	  4 bytes per page against the default one byte per page. Preferable
+	  to enable this on systems with multiple nodes in toptier.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +6 -1
index 33014de43acc..dc61f4d955f8 100644
@@ -150,4 +150,9 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
-obj-$(CONFIG_PGHOT) += pghot.o pghot-tunables.o pghot-default.o
+obj-$(CONFIG_PGHOT) += pghot.o pghot-tunables.o
+ifdef CONFIG_PGHOT_PRECISE
+obj-$(CONFIG_PGHOT) += pghot-precise.o
+else
+obj-$(CONFIG_PGHOT) += pghot-default.o
+endif
mm/pghot-precise.c new file +81 -0

81 lines not on this page · raw message

mm/pghot.c +9 -4
index dac9e6f3b61e..7d7ef0800ae2 100644
@@ -10,6 +10,9 @@
  * the frequency of access and last access time. Promotions are done
  * to a default toptier NID.
  *
+ * In the precision mode, 4 bytes are used to store the frequency
+ * of access, last access time and the accessing NID.
+ *
  * A kernel thread named kmigrated is provided to migrate or promote
  * the hot pages. kmigrated runs for each lower tier node. It iterates
  * over the node's PFNs and  migrates pages marked for migration into
@@ -52,13 +55,15 @@ static bool kmigrated_started __ro_after_init;
  * for the purpose of tracking page hotness and subsequent promotion.
  *
  * @pfn: PFN of the page
- * @nid: Unused
+ * @nid: Target NID to where the page needs to be migrated in precision
+ *       mode but unused in default mode
  * @src: The identifier of the sub-system that reports the access
  * @now: Access time in jiffies
  *
- * Updates the frequency and time of access and marks the page as
- * ready for migration if the frequency crosses a threshold. The pages
- * marked for migration are migrated by kmigrated kernel thread.
+ * Updates the NID (in precision mode only), frequency and time of access
+ * and marks the page as ready for migration if the frequency crosses a
+ * threshold. The pages marked for migration are migrated by kmigrated
+ * kernel thread.
  *
  * Return: 0 on success and -EINVAL on failure to record the access.
  */
Signature · 1 line
2.34.1
#18 lore raw

Re: [RFC PATCH v6 4/5] mm: pghot: Precision mode for pghot

Replies to review from Shashiko
[https://sashiko.dev/#/patchset/20260323095104.238982-1-bharata%40amd.com]

Quoted 8 lines
Quoted 8 lines
mm/Kconfig +0 -0
index 4aeab6aee535..14383bb1d890 100644
@@ -1485,6 +1485,17 @@ config PGHOT
 	  This adds 1 byte of metadata overhead per page in lower-tier
 	  memory nodes.
 
Quoted 6 lines

mm/Kconfig config PGHOT · line 1490

Quoted 3 lines
mm/Kconfig +3 -0
+config PGHOT_PRECISE
+	bool "Hot page tracking precision mode"
+	def_bool n

This isn't a bug, but having both bool with a prompt and def_bool is redundant and can be confusing. Should this use default n instead?

Sure.

Quoted 23 lines
Quoted 3 lines
mm/Kconfig +3 -0
+	depends on PGHOT
+	help
+	  Enables precision mode for tracking hot pages with pghot sub-system.

[ ... ]

Quoted 19 lines
mm/pghot-precise.c new file +13 -0
new file mode 100644
index 000000000000..9e8007adfff9
@@ -0,0 +1,81 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pghot: Precision mode
+ *
+ * 4 byte hotness record per PFN (u32)
+ * NID, time and frequency tracked as part of the record.
+ */
+
+#include <linux/pghot.h>
+#include <linux/jiffies.h>
+
+bool pghot_nid_valid(int nid)
+{
Quoted 8 lines

mm/pghot-precise.c line 18

Quoted 5 lines
mm/pghot-precise.c new file +5 -0
+	/*
+	 * TODO: Add node_online() and node_is_toptier() checks?
+	 */
+	if (nid != NUMA_NO_NODE && (nid < 0 || nid >= PGHOT_NID_MAX))
+		return false;

Could this validation allow node IDs that exceed the system's actual MAX_NUMNODES?

Unlikely as the nid gets passed down from other sub-systems like NUMA Balancing etc.

Quoted 6 lines

If PGHOT_NID_MAX is 1023 but the system supports fewer nodes, could passing a large node ID bypass this check and cause out-of-bounds accesses in NUMA node arrays like node_data or node_states?
Also, is there an off-by-one regression here? PGHOT_NID_MAX is the maximum value that fits in the bitfield, so should this check be nid > PGHOT_NID_MAX to allow the maximum node ID?

Yes, will adjust this in the next iteration.

Quoted 6 lines
Quoted 6 lines
mm/pghot-precise.c new file +6 -0
+
+	return true;
+}
+
+unsigned long pghot_access_latency(unsigned long old_time, unsigned long time)
+{
Quoted 8 lines

mm/pghot-precise.c line 26

Quoted 2 lines
mm/pghot-precise.c new file +2 -0
+	return jiffies_to_msecs((time - old_time) & PGHOT_TIME_MASK);
+}

Does this timestamp wraparound allow cold pages to be improperly promoted? If a page is accessed exactly at intervals matching the wraparound of the 14-bit time mask (e.g., every 16.384 seconds at HZ=1000), the latency would evaluate to 0. Would this bypass the reset of the frequency counter and falsely trigger migration?

The hotness timestamp is stored in a limited number of bits (14 bits in precise mode, 5 bucketed bits in default mode), which means it can only represent a finite window of time before wrapping around — approximately 16 seconds for precise mode and 4 seconds for default mode at HZ=1000. Latency calculations rely on modular subtraction of these truncated timestamps. This produces the correct elapsed time as long as the true interval between two events is shorter than the wrap period. When the interval exceeds the wrap period, there could be approximations.

In practice, this should be acceptable because the system has two independent gates that a page must pass through before migration actually occurs:

1. Access frequency gate (recording time): pghot_update_record() determines whether consecutive accesses to a page fall within the same frequency-counting window. A page must be accessed at least pghot_freq_threshold times within sysctl_pghot_freq_window (default 5 seconds) to be marked migration-ready.

2. Recency gate (migration time): kmigrated_should_migrate_memory() checks whether the page's last recorded access is recent enough relative to the current time before actually initiating migration.

Quoted 16 lines
Quoted 16 lines
mm/pghot-precise.c new file +16 -0
+
+bool pghot_update_record(phi_t *phi, int nid, unsigned long now)
+{
+	phi_t freq, old_freq, hotness, old_hotness, old_time;
+	phi_t time = now & PGHOT_TIME_MASK;
+
+	nid = (nid == NUMA_NO_NODE) ? pghot_target_nid : nid;
+	old_hotness = READ_ONCE(*phi);
+
+	do {
+		bool new_window = false;
+
+		hotness = old_hotness;
+		old_freq = (hotness >> PGHOT_FREQ_SHIFT) & PGHOT_FREQ_MASK;
+		old_time = (hotness >> PGHOT_TIME_SHIFT) & PGHOT_TIME_MASK;
+
Quoted 8 lines

mm/pghot-precise.c line 44

Quoted 2 lines
mm/pghot-precise.c new file +2 -0
+		if (pghot_access_latency(old_time, time) > sysctl_pghot_freq_window)
+			new_window = true;

Is there a missing upper bound on the sysctl_pghot_freq_window configuration? The time is tracked in a 14-bit field, which represents a maximum latency of 16,383 jiffies. If a user configures a window larger than this maximum representable latency, could this condition permanently evaluate to false, causing sparse accesses to accumulate over arbitrarily long periods?

I can add an upper bound check.

Regards,
Bharata.

#19 lore raw

Re: [RFC PATCH v6 4/5] mm: pghot: Precision mode for pghot

In reply to Bharata B Rao
kernel test robot noticed the following build errors:

Hi Bharata,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on a989fde763f4f24209e4702f50a45be572340e68]

url:    https://github.com/intel-lab-lkp/linux/commits/Bharata-B-Rao/mm-migrate-Allow-misplaced-migration-without-VMA/20260325-092849
base:   a989fde763f4f24209e4702f50a45be572340e68
patch link:    https://lore.kernel.org/r/20260323095104.238982-5-bharata%40amd.com
patch subject: [RFC PATCH v6 4/5] mm: pghot: Precision mode for pghot
config: s390-allyesconfig (https://download.01.org/0day-ci/archive/20260327/[email protected]/config)
compiler: s390-linux-gcc (GCC) 15.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260327/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

mm/pghot-precise.c: In function 'pghot_nid_valid':

>> mm/pghot-precise.c:17:20: error: 'NUMA_NO_NODE' undeclared (first use in this function)
      17 |         if (nid != NUMA_NO_NODE && (nid < 0 || nid >= PGHOT_NID_MAX))
         |                    ^~~~~~~~~~~~
   mm/pghot-precise.c:17:20: note: each undeclared identifier is reported only once for each function it appears in
   mm/pghot-precise.c: In function 'pghot_update_record':
   mm/pghot-precise.c:33:23: error: 'NUMA_NO_NODE' undeclared (first use in this function)
      33 |         nid = (nid == NUMA_NO_NODE) ? pghot_target_nid : nid;
         |                       ^~~~~~~~~~~~

vim +/NUMA_NO_NODE +17 mm/pghot-precise.c

    11	
    12	bool pghot_nid_valid(int nid)
    13	{
    14		/*
    15		 * TODO: Add node_online() and node_is_toptier() checks?
    16		 */
  > 17		if (nid != NUMA_NO_NODE && (nid < 0 || nid >= PGHOT_NID_MAX))
    18			return false;
    19	
    20		return true;
    21	}
    22	
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#20 lore raw

Re: [RFC PATCH v6 4/5] mm: pghot: Precision mode for pghot

In reply to Bharata B Rao
kernel test robot noticed the following build errors:

Hi Bharata,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on a989fde763f4f24209e4702f50a45be572340e68]

url:    https://github.com/intel-lab-lkp/linux/commits/Bharata-B-Rao/mm-migrate-Allow-misplaced-migration-without-VMA/20260325-092849
base:   a989fde763f4f24209e4702f50a45be572340e68
patch link:    https://lore.kernel.org/r/20260323095104.238982-5-bharata%40amd.com
patch subject: [RFC PATCH v6 4/5] mm: pghot: Precision mode for pghot
config: powerpc-allmodconfig (https://download.01.org/0day-ci/archive/20260327/[email protected]/config)
compiler: powerpc64-linux-gcc (GCC) 15.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260327/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

mm/pghot-precise.c: In function 'pghot_nid_valid':
mm/pghot-precise.c:17:20: error: 'NUMA_NO_NODE' undeclared (first use in this function)

      17 |         if (nid != NUMA_NO_NODE && (nid < 0 || nid >= PGHOT_NID_MAX))
         |                    ^~~~~~~~~~~~
   mm/pghot-precise.c:17:20: note: each undeclared identifier is reported only once for each function it appears in
   In file included from mm/pghot-precise.c:9:
   mm/pghot-precise.c: In function 'pghot_access_latency':
   include/linux/pghot.h:59:41: error: implicit declaration of function 'GENMASK' [-Wimplicit-function-declaration]
      59 | #define PGHOT_TIME_MASK                 GENMASK(PGHOT_TIME_WIDTH - 1, 0)
         |                                         ^~~~~~~
   mm/pghot-precise.c:25:53: note: in expansion of macro 'PGHOT_TIME_MASK'
      25 |         return jiffies_to_msecs((time - old_time) & PGHOT_TIME_MASK);
         |                                                     ^~~~~~~~~~~~~~~
   mm/pghot-precise.c: In function 'pghot_update_record':
   mm/pghot-precise.c:33:23: error: 'NUMA_NO_NODE' undeclared (first use in this function)
      33 |         nid = (nid == NUMA_NO_NODE) ? pghot_target_nid : nid;
         |                       ^~~~~~~~~~~~
>> mm/pghot-precise.c:62:36: error: implicit declaration of function 'BIT' [-Wimplicit-function-declaration]
      62 |                         hotness |= BIT(PGHOT_MIGRATE_READY);
         |                                    ^~~
   In file included from include/linux/jump_label.h:77,
                    from include/linux/static_key.h:1,
                    from include/linux/pghot.h:13:
>> mm/pghot-precise.c:63:28: error: implicit declaration of function 'try_cmpxchg' [-Wimplicit-function-declaration]
      63 |         } while (unlikely(!try_cmpxchg(phi, &old_hotness, hotness)));
         |                            ^~~~~~~~~~~
   include/linux/compiler.h:77:45: note: in definition of macro 'unlikely'
      77 | # define unlikely(x)    __builtin_expect(!!(x), 0)
         |                                             ^

vim +/BIT +62 mm/pghot-precise.c

    27	
    28	bool pghot_update_record(phi_t *phi, int nid, unsigned long now)
    29	{
    30		phi_t freq, old_freq, hotness, old_hotness, old_time;
    31		phi_t time = now & PGHOT_TIME_MASK;
    32	
    33		nid = (nid == NUMA_NO_NODE) ? pghot_target_nid : nid;
    34		old_hotness = READ_ONCE(*phi);
    35	
    36		do {
    37			bool new_window = false;
    38	
    39			hotness = old_hotness;
    40			old_freq = (hotness >> PGHOT_FREQ_SHIFT) & PGHOT_FREQ_MASK;
    41			old_time = (hotness >> PGHOT_TIME_SHIFT) & PGHOT_TIME_MASK;
    42	
    43			if (pghot_access_latency(old_time, time) > sysctl_pghot_freq_window)
    44				new_window = true;
    45	
    46			if (new_window)
    47				freq = 1;
    48			else if (old_freq < PGHOT_FREQ_MAX)
    49				freq = old_freq + 1;
    50			else
    51				freq = old_freq;
    52	
    53			hotness &= ~(PGHOT_NID_MASK << PGHOT_NID_SHIFT);
    54			hotness &= ~(PGHOT_FREQ_MASK << PGHOT_FREQ_SHIFT);
    55			hotness &= ~(PGHOT_TIME_MASK << PGHOT_TIME_SHIFT);
    56	
    57			hotness |= (nid & PGHOT_NID_MASK) << PGHOT_NID_SHIFT;
    58			hotness |= (freq & PGHOT_FREQ_MASK) << PGHOT_FREQ_SHIFT;
    59			hotness |= (time & PGHOT_TIME_MASK) << PGHOT_TIME_SHIFT;
    60	
    61			if (freq >= pghot_freq_threshold)
  > 62				hotness |= BIT(PGHOT_MIGRATE_READY);
  > 63		} while (unlikely(!try_cmpxchg(phi, &old_hotness, hotness)));
    64		return !!(hotness & BIT(PGHOT_MIGRATE_READY));
    65	}
    66	
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#21 lore raw

RFC PATCH v6 5/5 mm: sched: move NUMA balancing tiering promotion to pghot

In reply to Bharata B Rao

Currently hot page promotion (NUMA_BALANCING_MEMORY_TIERING mode of NUMA Balancing) does hot page detection (via hint faults), hot page classification and eventual promotion, all by itself and sits within the scheduler.

With pghot, the new hot page tracking and promotion mechanism being available, NUMA Balancing can limit itself to detection of hot pages (via hint faults) and off-load rest of the functionality to pghot.

To achieve this, pghot_record_access(PGHOT_HINT_FAULT) API is used to feed the hot page info to pghot. In addition, the migration rate limiting and dynamic threshold logic are moved to kmigrated so that the same can be used for hot pages reported by other sources too. Hence it becomes necessary to introduce a new config option CONFIG_NUMA_BALANCING_TIERING to control the hint faults souce for hot page promotion. This option controls the NUMA_BALANCING_MEMORY_TIERING mode of kernel.numa_balancing

This movement of hot page promotion to pghot results in the following changes to the behaviour of hint faults based hot page promotion:

1. Promotion is no longer done in the fault path but instead is deferred to kmigrated and happens in batches.
2. NUMA_BALANCING_MEMORY_TIERING mode used to promote on first access. Pghot by default, promotes on second access though this can be changed by setting /sys/kernel/debug/pghot/freq_threshold. hot_threshold_ms debugfs tunable now gets replaced by pghot's freq_threshold.
3. In NUMA_BALANCING_MEMORY_TIERING mode, hint fault latency is the difference between the PTE update time (during scanning) and the access time (hint fault). However with pghot, a single latency threshold is used for two purposes:
a) If the time difference between successive accesses are within the threshold, the page is marked as hot.
b) Later when kmigrated picks up the page for migration, it will migrate only if the difference between the current time and the time when the page was marked hot is with the threshold.
4. Batch migration of misplaced folios is done from non-process context where VMA info is not readily available. Without VMA and the exec check on that, it will not be possible to filter out exec pages during migration prep stage. Hence shared executable pages also will be subjected to misplaced migration.
5. The max scan period which is used in dynamic threshold logic was a debugfs tunable. However this has been converted to a scalar metric in pghot.

Key code changes due to this movement are detailed below to help easy understanding of the restructuring.

1. Scanning and access times are no longer tracked in last_cpupid field of folio flags. Hence all code related to this (like folio_xchg_access_time(), cpupid_valid()) are removed.
2. The misplaced migration routines become conditional to CONFIG_PGHOT in addition to CONFIG_NUMA_BALANCING.
3. The promotion related stats (like PGPROMOTE_SUCCESS etc) are now moved to under CONFIG_PGHOT as these stats are part of promotion engine which will be used for other hotness sources as well.
4. Routines that are responsibile for migration rate limiting dynamic thresholding, pgdat balancing during promotion etc are moved to pghot with appropriate renaming.

---
 include/linux/mm.h     |  35 ++------
 include/linux/mmzone.h |   4 +-
 init/Kconfig           |  13 +++
 kernel/sched/core.c    |   7 ++
 kernel/sched/debug.c   |   1 -
 kernel/sched/fair.c    | 177 ++---------------------------------------
 kernel/sched/sched.h   |   1 -
 mm/huge_memory.c       |  27 ++++++-
 mm/memcontrol.c        |   6 +-
 mm/memory-tiers.c      |  15 ++--
 mm/memory.c            |  36 +++++++--
 mm/mempolicy.c         |   3 -
 mm/migrate.c           |  16 +++-
 mm/pghot.c             | 134 +++++++++++++++++++++++++++++++
 mm/vmstat.c            |   2 +-
 15 files changed, 248 insertions(+), 229 deletions(-)
include/linux/mm.h +7 -28
index abb4963c1f06..81249a06dfeb 100644
@@ -1998,17 +1998,6 @@ static inline int folio_nid(const struct folio *folio)
 }
 
 #ifdef CONFIG_NUMA_BALANCING
-/* page access time bits needs to hold at least 4 seconds */
-#define PAGE_ACCESS_TIME_MIN_BITS	12
-#if LAST_CPUPID_SHIFT < PAGE_ACCESS_TIME_MIN_BITS
-#define PAGE_ACCESS_TIME_BUCKETS				\
-	(PAGE_ACCESS_TIME_MIN_BITS - LAST_CPUPID_SHIFT)
-#else
-#define PAGE_ACCESS_TIME_BUCKETS	0
-#endif
-
-#define PAGE_ACCESS_TIME_MASK				\
-	(LAST_CPUPID_MASK << PAGE_ACCESS_TIME_BUCKETS)
 
 static inline int cpu_pid_to_cpupid(int cpu, int pid)
 {
@@ -2074,15 +2063,6 @@ static inline void page_cpupid_reset_last(struct page *page)
 }
 #endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */
 
-static inline int folio_xchg_access_time(struct folio *folio, int time)
-{
-	int last_time;
-
-	last_time = folio_xchg_last_cpupid(folio,
-					   time >> PAGE_ACCESS_TIME_BUCKETS);
-	return last_time << PAGE_ACCESS_TIME_BUCKETS;
-}
-
 static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 {
 	unsigned int pid_bit;
@@ -2093,18 +2073,12 @@ static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 	}
 }
 
-bool folio_use_access_time(struct folio *folio);
 #else /* !CONFIG_NUMA_BALANCING */
 static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid)
 {
 	return folio_nid(folio); /* XXX */
 }
 
-static inline int folio_xchg_access_time(struct folio *folio, int time)
-{
-	return 0;
-}
-
 static inline int folio_last_cpupid(struct folio *folio)
 {
 	return folio_nid(folio); /* XXX */
@@ -2147,11 +2121,16 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid)
 static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 {
 }
-static inline bool folio_use_access_time(struct folio *folio)
+#endif /* CONFIG_NUMA_BALANCING */
+
+#ifdef CONFIG_NUMA_BALANCING_TIERING
+bool folio_is_promo_candidate(struct folio *folio);
+#else
+static inline bool folio_is_promo_candidate(struct folio *folio)
 {
 	return false;
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING_TIERING */
 
 #if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
 
include/linux/mmzone.h +2 -2
index 61fd259d9897..bfaaa757b19c 100644
@@ -232,7 +232,7 @@ enum node_stat_item {
 #ifdef CONFIG_SWAP
 	NR_SWAPCACHE,
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	PGPROMOTE_SUCCESS,	/* promote successfully */
 	/**
 	 * Candidate pages for promotion based on hint fault latency.  This
@@ -1475,7 +1475,7 @@ typedef struct pglist_data {
 	struct deferred_split deferred_split_queue;
 #endif
 
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	/* start time in ms of current promote rate limit period */
 	unsigned int nbp_rl_start;
 	/* number of promote candidate pages at start time of current rate limit period */
init/Kconfig +13 -0
index 444ce811ea67..56ef148487fa 100644
@@ -1013,6 +1013,19 @@ config NUMA_BALANCING_DEFAULT_ENABLED
 	  If set, automatic NUMA balancing will be enabled if running on a NUMA
 	  machine.
 
+config NUMA_BALANCING_TIERING
+	bool "NUMA balancing memory tiering promotion"
+	depends on NUMA_BALANCING && PGHOT
+	help
+	  Enable NUMA balancing mode 2 (memory tiering). This allows
+	  automatic promotion of hot pages from slower memory tiers to
+	  faster tiers using the pghot subsystem.
+
+	  This requires CONFIG_PGHOT for the hot page tracking engine.
+	  This option is required for kernel.numa_balancing=2.
+
+	  If unsure, say N.
+
 config SLAB_OBJ_EXT
 	bool
 
kernel/sched/core.c +7 -0
index 496dff740dca..f8ca5dff9cad 100644
@@ -4463,6 +4463,7 @@ void set_numabalancing_state(bool enabled)
 }
 
 #ifdef CONFIG_PROC_SYSCTL
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 static void reset_memory_tiering(void)
 {
 	struct pglist_data *pgdat;
@@ -4473,6 +4474,7 @@ static void reset_memory_tiering(void)
 		pgdat->nbp_th_start = jiffies_to_msecs(jiffies);
 	}
 }
+#endif
 
 static int sysctl_numa_balancing(const struct ctl_table *table, int write,
 			  void *buffer, size_t *lenp, loff_t *ppos)
@@ -4490,9 +4492,14 @@ static int sysctl_numa_balancing(const struct ctl_table *table, int write,
 	if (err < 0)
 		return err;
 	if (write) {
+		if ((state & NUMA_BALANCING_MEMORY_TIERING) &&
+		    !IS_ENABLED(CONFIG_NUMA_BALANCING_TIERING))
+			return -EOPNOTSUPP;
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 		    (state & NUMA_BALANCING_MEMORY_TIERING))
 			reset_memory_tiering();
+#endif
 		sysctl_numa_balancing_mode = state;
 		__set_numabalancing_state(state);
 	}
kernel/sched/debug.c +0 -1
index b24f40f05019..c6a3325ebbd2 100644
@@ -622,7 +622,6 @@ static __init int sched_init_debug(void)
 	debugfs_create_u32("scan_period_min_ms", 0644, numa, &sysctl_numa_balancing_scan_period_min);
 	debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max);
 	debugfs_create_u32("scan_size_mb", 0644, numa, &sysctl_numa_balancing_scan_size);
-	debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold);
 #endif /* CONFIG_NUMA_BALANCING */
 
 	debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);
kernel/sched/fair.c +5 -172

217 lines not on this page · raw message

kernel/sched/sched.h +0 -1
index 43bbf0693cca..a47f7e3d51a6 100644
@@ -3021,7 +3021,6 @@ extern unsigned int sysctl_numa_balancing_scan_delay;
 extern unsigned int sysctl_numa_balancing_scan_period_min;
 extern unsigned int sysctl_numa_balancing_scan_period_max;
 extern unsigned int sysctl_numa_balancing_scan_size;
-extern unsigned int sysctl_numa_balancing_hot_threshold;
 
 #ifdef CONFIG_SCHED_HRTICK
 
mm/huge_memory.c +24 -3
index b298cba853ab..fe957ff91df9 100644
@@ -40,6 +40,7 @@
 #include <linux/pgalloc.h>
 #include <linux/pgalloc_tag.h>
 #include <linux/pagewalk.h>
+#include <linux/pghot.h>
 
 #include <asm/tlb.h>
 #include "internal.h"
@@ -2190,7 +2191,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	int nid = NUMA_NO_NODE;
 	int target_nid, last_cpupid;
 	pmd_t pmd, old_pmd;
-	bool writable = false;
+	bool writable = false, needs_promotion = false;
 	int flags = 0;
 
 	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
@@ -2217,11 +2218,26 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 
 	target_nid = numa_migrate_check(folio, vmf, haddr, &flags, writable,
 					&last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
+		 * Isolation and migration are handled by pghot.
+		 *
+		 * TODO: mode2 check
+		 */
+		writable = false;
+		nid = target_nid;
+		goto out_map;
+	}
+
+	/* Balancing b/n toptier nodes, mode=NUMA_BALANCING_NORMAL */
 	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
 		flags |= TNF_MIGRATE_FAIL;
 		goto out_map;
@@ -2253,8 +2269,13 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
 	spin_unlock(vmf->ptl);
 
-	if (nid != NUMA_NO_NODE)
-		task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags);
+	if (nid != NUMA_NO_NODE) {
+		if (needs_promotion)
+			pghot_record_access(folio_pfn(folio), nid,
+					    PGHOT_HINTFAULTS, jiffies);
+		else
+			task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags);
+	}
 	return 0;
 }
 
mm/memcontrol.c +3 -3
index 772bac21d155..fcd92f2ffd0c 100644
@@ -323,7 +323,7 @@ static const unsigned int memcg_node_stat_items[] = {
 #ifdef CONFIG_SWAP
 	NR_SWAPCACHE,
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	PGPROMOTE_SUCCESS,
 #endif
 	PGDEMOTE_KSWAPD,
@@ -1400,7 +1400,7 @@ static const struct memory_stat memory_stats[] = {
 	{ "pgdemote_direct",		PGDEMOTE_DIRECT		},
 	{ "pgdemote_khugepaged",	PGDEMOTE_KHUGEPAGED	},
 	{ "pgdemote_proactive",		PGDEMOTE_PROACTIVE	},
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	{ "pgpromote_success",		PGPROMOTE_SUCCESS	},
 #endif
 };
@@ -1443,7 +1443,7 @@ static int memcg_page_state_output_unit(int item)
 	case PGDEMOTE_DIRECT:
 	case PGDEMOTE_KHUGEPAGED:
 	case PGDEMOTE_PROACTIVE:
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	case PGPROMOTE_SUCCESS:
 #endif
 		return 1;
mm/memory-tiers.c +8 -7
index 986f809376eb..7303dc10035c 100644
@@ -51,18 +51,19 @@ static const struct bus_type memory_tier_subsys = {
 	.dev_name = "memory_tier",
 };
 
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 /**
- * folio_use_access_time - check if a folio reuses cpupid for page access time
+ * folio_is_promo_candidate - check if the folio qualifies for promotion
+ *
  * @folio: folio to check
  *
- * folio's _last_cpupid field is repurposed by memory tiering. In memory
- * tiering mode, cpupid of slow memory folio (not toptier memory) is used to
- * record page access time.
+ * Checks if NUMA Balancing tiering mode is set and the folio belongs
+ * to lower tier. If so, it qualifies for promotion to toptier when
+ * it is categorized as hot.
  *
- * Return: the folio _last_cpupid is used to record page access time
+ * Return: True if the above condition is met, else False.
  */
-bool folio_use_access_time(struct folio *folio)
+bool folio_is_promo_candidate(struct folio *folio)
 {
 	return (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 	       !node_is_toptier(folio_nid(folio));
mm/memory.c +29 -7
index 2f815a34d924..289fa6c07a42 100644
@@ -75,6 +75,7 @@
 #include <linux/perf_event.h>
 #include <linux/ptrace.h>
 #include <linux/vmalloc.h>
+#include <linux/pghot.h>
 #include <linux/sched/sysctl.h>
 #include <linux/pgalloc.h>
 #include <linux/uaccess.h>
@@ -5968,10 +5969,9 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,
 	if (folio_maybe_mapped_shared(folio) && (vma->vm_flags & VM_SHARED))
 		*flags |= TNF_SHARED;
 	/*
-	 * For memory tiering mode, cpupid of slow memory page is used
-	 * to record page access time.  So use default value.
+	 * For memory tiering mode, last_cpupid is unused. So use default value.
 	 */
-	if (folio_use_access_time(folio))
+	if (folio_is_promo_candidate(folio))
 		*last_cpupid = (-1 & LAST_CPUPID_MASK);
 	else
 		*last_cpupid = folio_last_cpupid(folio);
@@ -6052,6 +6052,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	int nid = NUMA_NO_NODE;
 	bool writable = false, ignore_writable = false;
 	bool pte_write_upgrade = vma_wants_manual_pte_write_upgrade(vma);
+	bool needs_promotion = false;
 	int last_cpupid;
 	int target_nid;
 	pte_t pte, old_pte;
@@ -6086,16 +6087,31 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 	nr_pages = folio_nr_pages(folio);
 
 	target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
 					writable, &last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
-	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
+		 * Isolation and migration are handled by pghot.
+		 */
+		writable = false;
+		ignore_writable = true;
+		nid = target_nid;
+		goto out_map;
+	}
+
+	/* Balancing b/n toptier nodes, mode=NUMA_BALANCING_NORMAL */
+	if (migrate_misplaced_folio_prepare(folio, vmf->vma, target_nid)) {
 		flags |= TNF_MIGRATE_FAIL;
 		goto out_map;
 	}
+
 	/* The folio is isolated and isolation code holds a folio reference. */
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
 	writable = false;
@@ -6110,7 +6126,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	}
 
 	flags |= TNF_MIGRATE_FAIL;
-	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
+	vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
 				       vmf->address, &vmf->ptl);
 	if (unlikely(!vmf->pte))
 		return 0;
@@ -6118,6 +6134,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 		pte_unmap_unlock(vmf->pte, vmf->ptl);
 		return 0;
 	}
+
 out_map:
 	/*
 	 * Make it present again, depending on how arch implements
@@ -6131,8 +6148,13 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 					    writable);
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
 
-	if (nid != NUMA_NO_NODE)
-		task_numa_fault(last_cpupid, nid, nr_pages, flags);
+	if (nid != NUMA_NO_NODE) {
+		if (needs_promotion)
+			pghot_record_access(folio_pfn(folio), nid,
+					    PGHOT_HINTFAULTS, jiffies);
+		else
+			task_numa_fault(last_cpupid, nid, nr_pages, flags);
+	}
 	return 0;
 }
 
mm/mempolicy.c +0 -3
index 0e5175f1c767..6eed217a5917 100644
@@ -866,9 +866,6 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 	    node_is_toptier(nid))
 		return false;
 
-	if (folio_use_access_time(folio))
-		folio_xchg_access_time(folio, jiffies_to_msecs(jiffies));
-
 	return true;
 }
 
mm/migrate.c +15 -1
index a5f48984ed3e..db6832b4b95b 100644
@@ -2690,8 +2690,18 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 	if (!migrate_balanced_pgdat(pgdat, nr_pages)) {
 		int z;
 
-		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING))
+		/*
+		 * Kswapd wakeup for creating headroom in toptier is done only
+		 * for hot page promotion case and not for misplaced migrations
+		 * between toptier nodes.
+		 *
+		 * In the uncommon case of using NUMA_BALANCING_NORMAL mode
+		 * to balance between lower and higher tier nodes, we end up
+		 * up waking up kswapd.
+		 */
+		if (node_is_toptier(folio_nid(folio)))
 			return -EAGAIN;
+
 		for (z = pgdat->nr_zones - 1; z >= 0; z--) {
 			if (managed_zone(pgdat->node_zones + z))
 				break;
@@ -2741,6 +2751,8 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 #ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+#endif
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 		if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
 		    && !node_is_toptier(folio_nid(folio))
 		    && node_is_toptier(node)) {
@@ -2796,6 +2808,8 @@ int migrate_misplaced_folios_batch(struct list_head *folio_list, int node)
 #ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+#endif
+#ifdef CONFIG_PGHOT
 		mod_node_page_state(NODE_DATA(node), PGPROMOTE_SUCCESS, nr_succeeded);
 #endif
 	}
mm/pghot.c +134 -0

164 lines not on this page · raw message

mm/vmstat.c +1 -1
index d3fbe2a5d0e6..f28f786f8931 100644
@@ -1267,7 +1267,7 @@ const char * const vmstat_text[] = {
 #ifdef CONFIG_SWAP
 	[I(NR_SWAPCACHE)]			= "nr_swapcached",
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	[I(PGPROMOTE_SUCCESS)]			= "pgpromote_success",
 	[I(PGPROMOTE_CANDIDATE)]		= "pgpromote_candidate",
 	[I(PGPROMOTE_CANDIDATE_NRL)]		= "pgpromote_candidate_nrl",
Signature · 1 line
2.34.1
#22 lore raw

Re: [RFC PATCH v6 5/5] mm: sched: move NUMA balancing tiering promotion to pghot

Replies to review from Shashiko
[https://sashiko.dev/#/patchset/20260323095104.238982-1-bharata%40amd.com]

Quoted 11 lines
Quoted 4 lines
mm/huge_memory.c +0 -0
index b298cba853ab..fe957ff91df9 100644

[ ... ]

Quoted 6 lines
mm/huge_memory.c +1 -0
@@ -2217,11 +2218,26 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 
Quoted 11 lines

mm/huge_memory.c vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · line 2226

Quoted 4 lines
mm/huge_memory.c +0 -0
 	target_nid = numa_migrate_check(folio, vmf, haddr, &flags, writable,
 					&last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;

Does this code inadvertently record pages forbidden from migrating in pghot? If numa_migrate_check() returns NUMA_NO_NODE due to memory policy, the code jumps to out_map. At that point, nid retains its initialized value of folio_nid(folio). Since nid != NUMA_NO_NODE evaluates to true, it looks like pghot_record_access() will still be called at the end of the function, which could pollute the tracking map and trigger useless daemon sweeps.

While for NUMAB1, we need to call task_numa_fault() to just record the fault stats, the equivalent of it for NUMAB2 isn't necessary. I will make the call to pghot_record_access() conditional to this scenario.

Quoted 15 lines

mm/huge_memory.c vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · line 2235

Quoted 9 lines
mm/huge_memory.c +9 -0
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
+		 * Isolation and migration are handled by pghot.
+		 *
+		 * TODO: mode2 check
+		 */
+		writable = false;

Can this cause a regression involving spurious write-protect faults? By forcing writable = false here before jumping to out_map, the write permission appears to be stripped from the restored mapping even if the page was previously writable. If the application access was a write, it might immediately suffer a spurious write-protect fault when it retries the instruction.

Quoted 3 lines

mm/huge_memory.c vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) · line 2238

Quoted 3 lines
mm/huge_memory.c +3 -0
+		nid = target_nid;
+		goto out_map;
+	}

Since there is no migration from the fault context for NUMAB2, we don't release PTL lock at all. Hence I realize that I don't have to modify writable and ignore_writable at all for the hot page promotion case.

Quoted 13 lines

[ ... ]

Quoted 4 lines
mm/memory.c +0 -0
index 2f815a34d924..289fa6c07a42 100644

[ ... ]

Quoted 7 lines
mm/memory.c +1 -0
@@ -6086,16 +6087,31 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 	nr_pages = folio_nr_pages(folio);
 
Quoted 8 lines

mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 6096

Quoted 4 lines
mm/memory.c +0 -0
 	target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
 					writable, &last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;

Similarly to do_huge_pmd_numa_page(), could this also inadvertently record pages forbidden from migrating when jumping to out_map with an unmodified nid?

Same as above.

Quoted 5 lines
Quoted 5 lines
mm/memory.c +4 -1
-	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
Quoted 8 lines

mm/memory.c static vm_fault_t do_numa_page(struct vm_fault *vmf) · line 6104

Quoted 4 lines
mm/memory.c +4 -0
+		 * Isolation and migration are handled by pghot.
+		 */
+		writable = false;
+		ignore_writable = true;

Would forcing writable = false and ignore_writable = true here also strip permissions and cause a performance regression from spurious write-protect faults for previously writable pages?

Same as above.

Quoted 14 lines
Quoted 13 lines
mm/memory.c +6 -0
+		nid = target_nid;
+		goto out_map;
+	}
+
+	/* Balancing b/n toptier nodes, mode=NUMA_BALANCING_NORMAL */
+	if (migrate_misplaced_folio_prepare(folio, vmf->vma, target_nid)) {
 		flags |= TNF_MIGRATE_FAIL;
 		goto out_map;
 	}
mm/migrate.c +0 -0
index a5f48984ed3e..db6832b4b95b 100644

[ ... ]

Quoted 1 line

mm/migrate.c int migrate_misplaced_folios_batch(struct list_head

Quoted 1 line
mm/migrate.c +0 -0
@@ -2796,6 +2808,8 @@ int migrate_misplaced_folios_batch(struct list_head

*folio_list, int node)

Quoted 7 lines
Quoted 7 lines
mm/migrate.c +3 -0
 #ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+#endif
+#ifdef CONFIG_PGHOT
+		mod_node_page_state(NODE_DATA(node), PGPROMOTE_SUCCESS, nr_succeeded);
 #endif
Quoted 8 lines

mm/migrate.c int migrate_misplaced_folios_batch(struct list_head · line 2815

Quoted 1 line
mm/migrate.c +0 -0
 	}

Should this use mod_lruvec_state() instead of mod_node_page_state() to accurately update both the node and memcg counters? PGPROMOTE_SUCCESS is defined in memcg_node_stat_items, so it is exported to the cgroup memory.stat interface. Using mod_node_page_state() bypasses memcg accounting, which might cause the cgroup-level promotion statistics to silently underflow or remain at zero.

Replied to this in one of the previous threads.

Quoted 4 lines
Quoted 4 lines
mm/pghot.c +0 -0
index 7d7ef0800ae2..3c0ba254ad4c 100644

[ ... ]

Quoted 1 line

mm/pghot.c int pghot_record_access(unsigned long pfn, int nid, int

Quoted 1 line
mm/pghot.c +0 -0
@@ -141,6 +166,110 @@ int pghot_record_access(unsigned long pfn, int nid, int

src, unsigned long now)

Quoted 10 lines
Quoted 2 lines
mm/pghot.c +0 -0
 	return 0;
 }

[ ... ]

mm/pghot.c int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)

Quoted 7 lines
mm/pghot.c +7 -0
+static bool kmigrated_should_migrate_memory(unsigned long nr_pages, int nid,
+					    unsigned long time)
+{
+	struct pglist_data *pgdat;
+	unsigned long rate_limit;
+	unsigned int th, def_th;
+	unsigned long now_ms = jiffies_to_msecs(jiffies); /* Based on full-width

jiffies */

Quoted 7 lines

mm/pghot.c int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now) · line 252

Quoted 1 line
mm/pghot.c +1 -0
+	unsigned long now = jiffies;

Could this cause spurious threshold adjustments during millisecond wrap-arounds? jiffies_to_msecs() returns a 32-bit unsigned int which is zero-extended into the 64-bit unsigned long now_ms. When computing time differences like now_ms - start (where start is an unsigned int) in kmigrated_promotion_adjust_threshold(), the subtraction is performed in 64-bit

arithmetic.

Quoted 3 lines

When the 32-bit counter wraps around, the subtraction could yield a very large value that falsely satisfies conditions like > MSEC_PER_SEC or > th_period, which could cause a regression in rate-limiting.

It could wrap around but due to cmpxchg updating pgdat->nbp_rl_start to post-wrap value, it should be self-correcting. However I can change now_ms unsigned int to be perfect.

Quoted 7 lines
Quoted 3 lines
mm/pghot.c +3 -0
+
+	pgdat = NODE_DATA(nid);
+	if (pgdat_free_space_enough(pgdat)) {

[ ... ]

Quoted 3 lines
mm/pghot.c +3 -0
+
+	return !kmigrated_promotion_rate_limit(pgdat, rate_limit, nr_pages, now_ms);
+}

[ ... ]

Quoted 1 line

mm/pghot.c static void kmigrated_walk_zone(unsigned long start_pfn,

Quoted 1 line
mm/pghot.c +0 -0
@@ -218,6 +347,11 @@ static void kmigrated_walk_zone(unsigned long start_pfn,

unsigned long end_pfn,

Quoted 19 lines

mm/pghot.c static void kmigrated_walk_zone(unsigned long start_pfn, unsigned long end_pfn, · line 354

Quoted 8 lines
mm/pghot.c +5 -0
 			goto out_next;
 		}
 
+		if (!kmigrated_should_migrate_memory(nr, nid, time)) {
+			folio_put(folio);
+			goto out_next;
+		}
+

Does this correctly advance the PFN when encountering large folio tail pages? Looking at the rest of kmigrated_walk_zone(), the loop iterator pfn is advanced by nr = folio_nr_pages(folio) at the out_next label.
If the loop lands on a tail page of a large folio (for example, if a previous iteration failed a check and incremented by 1), folio_nr_pages() returns the size of the entire large folio. Adding the full folio size to a tail page's PFN overshoots the end of the folio, potentially skipping valid pages of subsequent allocations.
Would it be safer to advance by
folio_nr_pages(folio) - folio_page_idx(folio, page)?

We could end up on tail pages leading to skipping of some folios but I think they will be reached in the next pass. Anyway I will check if your suggestion can be incorporated without any additional overhead.

Regards,
Bharata.

#23 lore raw
In reply to Bharata B Rao

Microbenchmark results

Test system details
-------------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0-95,192-287
node 0 size: 128460 MB
node 1 cpus: 96-191,288-383
node 1 size: 128893 MB
node 2 cpus:
node 2 size: 257993 MB
node distances:
node   0   1   2
  0:  10  32  50
  1:  32  10  60
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the patched case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the patched case.

Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

==============================================================
Scenario 1 - Enough memory in toptier and hence only promotion
==============================================================
Multi-threaded application with 64 threads that access memory at 4K granularity repetitively and randomly. The number of accesses per thread and the randomness pattern for each thread are fixed beforehand. The accesses are divided into stores and loads in the ratio of 50:50.

Benchmark threads run on Node 0, while memory is initially provisioned on CXL node 2 before the accesses start.

Repetitive accesses results in lowertier pages becoming hot and kmigrated detecting and migrating them. The benchmark score is the time taken to finish the accesses in microseconds. The sooner it finishes the better it is. All the numbers shown below are average of 3 runs.

Default mode - Time taken (microseconds, lower is better)
---------------------------------------------------------
Source Base Pghot
---------------------------------------------------------

NUMAB0          119,658,562     118,037,791
NUMAB2          104,205,571     102,705,330

---------------------------------------------------------

Default mode - Pages migrated (pgpromote_success)
---------------------------------------------------------
Source Base Pghot
---------------------------------------------------------

NUMAB0          0               0
NUMAB2          2097152         2097152

---------------------------------------------------------

Precision mode - Time taken (microseconds, lower is better)
-----------------------------------------------------------
Source Base Pghot
-----------------------------------------------------------

NUMAB0          119,658,562     115,173,151
NUMAB2          104,205,571     102,194,435

-----------------------------------------------------------

Precision mode - Pages migrated (pgpromote_success)
---------------------------------------------------
Source Base Pghot
---------------------------------------------------

NUMAB0          0               0
NUMAB2          2097152         2097152

---------------------------------------------------

Rate of migration (pgpromote_success)
-----------------------------------------
Time(s) Base Pghot
-----------------------------------------

0               0               0
28              0               0
32              262144          262144
36              524288          469012
40              786432          720896
44              1048576         983040
48              1310720         1245184
52              1572864         1507328
56              1835008         1769472
60              2097152         2031616
64              2097152         2097152

-----------------------------------------

==============================================================
Scenario 2 - Toptier memory overcommited, promotion + demotion
==============================================================
Single threaded application that allocates memory on both DRAM and CXL nodes using mmap(MAP_POPULATE). Every 1G region of allocated memory on CXL node is accessed at 4K granularity randomly and repetitively to build up the notion of hotness in the 1GB region that is under access. This should drive promotion. For promotion to work successfully, the DRAM memory that has been provisioned (and not being accessed) should be demoted first. There is enough free memory in the CXL node to for demotions.

In summary, this benchmark creates a memory pressure on DRAM node and does CXL memory accesses to drive both demotion and promotion.

The number of accesses are fixed and hence, the quicker the accessed pages get promoted to DRAM, the sooner the benchmark is expected to finish. All the numbers shown below are average of 3 runs.

DRAM-node                       = 1
CXL-node                        = 2
Initial DRAM alloc ratio        = 75%
Allocation-size                 = 171798691840
Initial DRAM Alloc-size         = 128849018880
Initial CXL Alloc-size          = 42949672960
Hot-region-size                 = 1073741824
Nr-regions                      = 160
Nr-regions DRAM                 = 120 (provisioned but not accessed)
Nr-hot-regions CXL              = 40
Access pattern                  = random
Access granularity              = 4096
Delay b/n accesses              = 0
Load/store ratio                = 50l50s
THP used                        = no
Nr accesses                     = 42949672960
Nr repetitions                  = 1024

Default mode - Time taken (microseconds, lower is better)
------------------------------------------------------
Source Base Pghot
------------------------------------------------------

NUMAB0          61,028,534      59,432,137
NUMAB2          63,070,998      61,375,763

------------------------------------------------------

Default mode - Pages migrated (pgpromote_success)
-------------------------------------------------
Source Base Pghot
-------------------------------------------------

NUMAB0          0               0
NUMAB2          26546           1070842 (High R2R variation in Base)

---------------------------------------

Precision mode - Time taken (microseconds, lower is better)
------------------------------------------------------
Source Base Pghot
------------------------------------------------------

NUMAB0          61,028,534      60,354,547
NUMAB2          63,070,998      60,199,147

------------------------------------------------------

Precision mode - Pages migrated (pgpromote_success)
---------------------------------------------------
Source Base Pghot
---------------------------------------------------

NUMAB0          0               0
NUMAB2          26546           1088621 (High R2R variation in Base)

---------------------------------------------------

- The base case itself doesn't show any improvement in benchmark numbers due to hot page promotion. The same pattern is seen in pghot case with all the sources except hwhints. The benchmark itself may need tuning so that promotion helps.
- There is a high run to run variation in the number of pages promoted in base case.
- Most promotion attempts in base case fail because the NUMA hint fault latency is found to exceed the threshold value (default threshold is 1000ms) in majority of the promotion attempts.
- Unlike base NUMAB2 where the hint fault latency is the difference between the PTE update time (during scanning) and the access time (hint fault), pghot uses a single latency threshold (3000ms in pghot-default and 5000ms in pghot-precise) for two purposes.
1. If the time difference between successive accesses are within the threshold, the page is marked as hot.
2. Later when kmigrated picks up the page for migration, it will migrate only if the difference between the current time and the time when the page was marked hot is with the threshold.

#24 lore raw
In reply to Bharata B Rao

Redis-memtier results

Test system details
-------------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0-95,192-287
node 0 size: 128460 MB
node 1 cpus: 96-191,288-383
node 1 size: 128893 MB
node 2 cpus:
node 2 size: 257993 MB
node distances:
node   0   1   2
  0:  10  32  50
  1:  32  10  60
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the patched case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the patched case.

Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

==============================================================
Scenario 1 - Enough memory in toptier and hence only promotion
==============================================================
In the setup phase, 64GB database is provisioned and explicitly moved to Node 2 by migrating redis-server's memory to Node 2. Memtier is run on Node 1.

Parallel distribution, 50% of the keys accessed, each 4 times.

16        Threads
100       Connections per thread
77808     Requests per client

==================================================================================================
Type Ops/sec Avg. Latency p50 Latency p99 Latency p99.9
Latency KB/sec
--------------------------------------------------------------------------------------------------
Base, NUMAB0
Totals 226611.42 225.92873 224.25500 423.93500
454.65500 514886.68
--------------------------------------------------------------------------------------------------
Base, NUMAB2
Totals 257211.48 204.99755 216.06300 370.68700
454.65500 584413.47
--------------------------------------------------------------------------------------------------
pghot-default, NUMAB2
Totals 255631.78 209.20335 216.06300 378.87900
450.55900 580824.22
--------------------------------------------------------------------------------------------------
pghot-precise, NUMAB2
Totals 249494.46 209.31820 212.99100 380.92700
448.51100 566879.53
==================================================================================================

pgpromote_success
==================================

Base, NUMAB0            0
Base, NUMAB2            10,435,176
pghot-default, NUMAB2   10,435,235
pghot-precise, NUMAB2   10,435,294

==================================

- There is a clear benefit of hot page promotion seen. Both base and pghot show similar benefits.
- The number of pages promoted in both cases are more or less same.

==============================================================
Scenario 2 - Toptier memory overcommited, promotion + demotion
==============================================================
In the setup phase, 192GB database is provisioned. The database occupies Node 1 entirely(~128GB) and spills over to Node 2 (~64GB). Memtier is run on Node 1.

Parallel distribution, 50% of the keys accessed, each 4 times.

16        Threads
100       Connections per thread
233424    Requests per client

==================================================================================================
Type Ops/sec Avg. Latency p50 Latency p99 Latency p99.9
Latency KB/sec
--------------------------------------------------------------------------------------------------
Base, NUMAB0
Totals 237743.40 217.72842 201.72700 395.26300
440.31900 540389.78
--------------------------------------------------------------------------------------------------
Base, NUMAB2
Totals 235935.72 219.36544 210.94300 411.64700
477.18300 536280.93
--------------------------------------------------------------------------------------------------
pghot-default, NUMAB2
Totals 248283.99 219.74875 211.96700 413.69500
509.95100 564348.49
--------------------------------------------------------------------------------------------------
pghot-precise, NUMAB2
Totals 240529.35 222.11878 215.03900 411.64700
464.89500 546722.22
==================================================================================================
pgpromote_success pgdemote_kswapd
===============================================================

Base, NUMAB0            0                       672,591
Base, NUMAB2            350,632                 689,751
pghot-default, NUMAB2   17,118,987              17,421,474
pghot-precise, NUMAB2   24,030,292              24,342,569

===============================================================

- No clear benefit is seen with hot page promotion both in base and pghot case.
- Most promotion attempts in base case fail because the NUMA hint fault latency is found to exceed the threshold value (default threshold of 1000ms) in majority of the promotion attempts.
- Unlike base NUMAB2 where the hint fault latency is the difference between the PTE update time (during scanning) and the access time (hint fault), pghot uses a single latency threshold (3000ms in pghot-default and 5000ms in pghot-precise) for two purposes.
1. If the time difference between successive accesses are within the threshold, the page is marked as hot.
2. Later when kmigrated picks up the page for migration, it will migrate only if the difference between the current time and the time when the page was marked hot is with the threshold.
Because of the above difference in behaviour, more number of pages get qualified for promotion compared to base NUMAB2.

#25 lore raw
In reply to Bharata B Rao

Graph500 results

Test system details
-------------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0-95,192-287
node 0 size: 128460 MB
node 1 cpus: 96-191,288-383
node 1 size: 128893 MB
node 2 cpus:
node 2 size: 257993 MB
node distances:
node   0   1   2
  0:  10  32  50
  1:  32  10  60
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the pghot case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the pghot case.
NUMAB3 - Enabled both regular and tiering mode of NUMA Balancing
(kernel.numa_balancing=3)

Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

Graph500 details
----------------
Command: mpirun -n 128 --bind-to core --map-by core graph500/src/graph500_reference_bfs 28 16

After the graph creation, the processes are stopped and data is migrated to CXL node 2 before continuing so that BFS phase starts accessing lower tier memory.

Total memory usage is slightly over 100GB and will fit within Node 0 and 1. Hence there is no memory pressure to induce demotions.

harmonic_mean_TEPS - Higher is better
=====================================================================================
Base Base pghot-default
pghot-precise
NUMAB0 NUMAB2 NUMAB2 NUMAB2
=====================================================================================

harmonic_mean_TEPS      5.07693e+08     7.08679e+08     5.56854e+08     7.39417e+08
mean_time               8.45968         6.06046         7.71283         5.80853
median_TEPS             5.08914e+08     7.23181e+08     5.51614e+08     7.58993e+08
max_TEPS                5.15226e+08     1.01654e+09     7.75233e+08     9.69136e+08
pgpromote_success       0               13797978        13746431        13752523
numa_pte_updates        0               26727341        39998363        48374479
numa_hint_faults        0               13798301        24459996        32728927

=====================================================================================
pghot-default NUMAB3
=====================================================================================

harmonic_mean_TEPS                                      7.18678e+08
mean_time                                               5.97614
median_TEPS                                             7.376e+08
max_TEPS                                                7.47337e+08
pgpromote_success                                       13821625
numa_pte_updates                                        93534398
numa_hint_faults                                        69164048

=====================================================================================
- The base case shows a good improvement with NUMAB2 in harmonic_mean_TEPS.
- The same improvement gets maintained with pghot-precise too.
- pghot-default mode doesn't show benefit even when achieving similar page promotion numbers. This mode doesn't track accessing NID and by default promotes to NID=0 which probably isn't all that beneficial as processes are running on both Node 0 and Node 1.
- pghot-default recovers the performance when balancing between toptier nodes 0 and 1 is enabled in addition to hot page promotion.

#26 lore raw
In reply to Bharata B Rao

NAS Parallel Benchmarks - BT results

Test system details
-------------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0-95,192-287
node 0 size: 128460 MB
node 1 cpus: 96-191,288-383
node 1 size: 128893 MB
node 2 cpus:
node 2 size: 257993 MB
node distances:
node   0   1   2
  0:  10  32  50
  1:  32  10  60
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the pghot case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the pghot case. Both promotion and demotion are enabled in this case.
NUMAB3 - Enabled both regular and tiering mode of NUMA Balancing
(kernel.numa_balancing=3)

Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

NAS-BT details
--------------
Command: mpirun -np 16 /usr/bin/numactl --cpunodebind=0,1 NPB3.4.4/NPB3.4-MPI/bin/bt.F.x

While class D uses around 24G of memory (which is too less to show the benefit of promotion), class E results in around 368G of memory which overflows my toptier. Hence I wanted something in between these classes. So I have modified class F to the problem size of 768 which results in around 160GB of memory.

After the memory consumption stabilizes, all the rank PIDs are paused and their memory is moved to CXL node using migratepages command. This simulates the situation of memory residing on lower tier node and access by BT processes leading to promotion.

Time in seconds - Lower is better
Mop/s total - Higher is better
=====================================================================================
Base Base pghot-default
pghot-precise
NUMAB0 NUMAB2 NUMAB2 NUMAB2
=====================================================================================

Time in seconds         7321.79         4333.85         6498.78         4386.27
Mop/s total             53451.77        90303.780       60221.01        89224.51
pgpromote_success       0               41971151        423163051       41957809
pgpromote_candidate     0               0               1870949786      0
pgpromote_candidate_nrl 0               41971151        29360089        41957809
pgdemote_kswapd         0               0               391179763       0
numa_pte_updates        0               42041312        1919944389      2568923206
numa_hint_faults        0               41972330        1911683592      2562729196

=====================================================================================
pghot-default NUMAB3
=====================================================================================
Time in seconds 4425.84
Mop/s total 88426.77

pgpromote_success                                       41957442
pgpromote_candidate                                     0
pgpromote_candidate_nrl                                 41957442
pgdemote_kswapd                                         0
numa_pte_updates                                        2588634775
numa_hint_faults                                        2581645889

=====================================================================================

- In the base case, the benchmark numbers improve significantly due to hot page promotion.
- Though the benchmark runs for hundreds of minutes, the pages get promoted within the first few mins.
- pghot-precise is able to match the base case numbers.
- The benchmark suffers in pghot-default case due to promotion being limited to the default NID (0) only. This leads to excessive PTE updates, hint faults, demotion and promotion churn.
- With NUMAB3, pghot-default case recovers the performce as in this mode misplaced hot page migrations get right placed due to NUMA balancing mode=1 being active.