Skip to contentSkip to conversation

mm: Hot page tracking and promotion infrastructure

RFC PATCH Superseded Bharata B Rao 3 participants 30 messages Jan 29 – Feb 27

Subsystems (13)

v5 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

v5 cover mm: Hot page tracking and promotion infrastructure

Hi,

This is v5 of pghot, a hot-page tracking and promotion subsystem. The major change in v5 is reducing the default hotness record size to 1 byte per PFN and adding an optional precision mode (CONFIG_PGHOT_PRECISE) that uses 4 bytes per PFN.

This patchset introduces a new subsystem for hot page tracking and promotion (pghot) with the following goals:

- Unify hot page detection from multiple sources like hint faults, page table scans, hardware hints (AMD IBS).
- Decouple detection from migration.
- Centralize promotion logic via per-lower-tier-node kmigrated kernel thread.
- Move promotion rate‑limiting and related logic used by numa_balancing=2 (current NUMA balancing–based promotion) from the scheduler to pghot for broader reuse.

Currently, multiple kernel subsystems detect page accesses independently. This patchset consolidates accesses from these mechanisms by providing:

- A common API for reporting page accesses.
- Shared infrastructure for tracking hotness at PFN granularity.
- Per-lower-tier-node kernel threads for promoting pages.

Here is a brief summary of how this subsystem works:

- Tracks frequency and last access time.
- Additionally, the accessing NUMA node ID (NID) for each recorded access is also tracked in the precision mode.
- These hotness parameters are maintained in a per-PFN hotness record within the existing mem_section data structure.
- In default mode, one byte (u8) is used for hotness record. 5 bits are used to store time and bucketing scheme is used to represent a total access time up to 4s with HZ=1000. Default toptier NID (0) is used as the target for promotion which can be changed via debugfs tunable.
- In precision mode, 4 bytes (u32) are used for each hotness record. 14 bits are used to store time which can represent around 16s with HZ=1000.
- Classifies pages as hot based on configurable thresholds.
- Pages classified as hot are marked as ready for migration using the ready bit. Both modes use MSB of the hotness record as ready bit.
- Per-lower-tier-node kmigrated threads periodically scan the PFNs of lower-tier nodes, checking for the migration-ready bit to perform batched migrations. Interval between successive scans and batching value are configurable via debugfs tunables.

Memory overhead
---------------
Default mode: 1 byte per lower-tier PFN. For a 1TB lower-tier memory this amounts to 256MB overhead (assuming 4K pages)

Precision mode: 4 bytes per lower-tier PFN. For a 1TB of lower memory this amounts to 1G overhead.

Bit layout of hotness record
----------------------------
Default mode
- Bits 0-1: Frequency (2bits, 4 access samples)
- Bits 2-6: Bucketed time (5bits, up to 4s with HZ=1000)
- Bit 7: Migration ready bit

Precision mode
- Bits 0-9: Target NID (10 bits)
- Bits 10-12: Frequency (3bits, 8 access samples)
- Bits 13-26: Time (14bits, up to 16s with HZ=1000)
- Bits 27-30: Reserved
- Bit 31: Migration ready bit

Integrated sources
------------------
1. IBS - Instruction Based Sampling, hardware based sampling mechanism present on AMD CPUs.
2. klruscand - PTE‑A bit scanning built on MGLRU’s walk helpers.
3. NUMA Balancing (Tiering mode)
4. folio_mark_accessed() - Page cache access tracking (unmapped page cache pages)

Changes in v5
=============
- Significant reduction in memory overhead for storing per-PFN hotness data
- Two modes of operation (default and precision mode). The code which is specific to each implementation is moved to its own individual file.
- Many bug fixes, code cleanups and code reorganization.

Results
=======
TODO: Will post benchmark nubmers as reply to this patchset soon.

This v5 patchset applies on top of upstream commit 4941a17751c9 and can be fetched from:

https://github.com/AMDESE/linux-mm/tree/bharata/pghot-rfcv5

v4: https://lore.kernel.org/linux-mm/[email protected]/
v3: https://lore.kernel.org/linux-mm/[email protected]/
v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/
v0: https://lore.kernel.org/linux-mm/[email protected]/

Bharata B Rao (7):
mm: migrate: Allow misplaced migration without VMA
mm: Hot page tracking and promotion
mm: pghot: Precision mode for pghot
mm: sched: move NUMA balancing tiering promotion to pghot
x86: ibs: In-kernel IBS driver for memory access profiling
x86: ibs: Enable IBS profiling for memory accesses
mm: pghot: Add folio_mark_accessed() as hotness source

Gregory Price (1):
migrate: Add migrate_misplaced_folios_batch()

Kinsey Ho (2):
mm: mglru: generalize page table walk
mm: klruscand: use mglru scanning for page promotion

 Documentation/admin-guide/mm/pghot.txt |  89 +++++
 arch/x86/events/amd/ibs.c              |  10 +
 arch/x86/include/asm/entry-common.h    |   3 +
 arch/x86/include/asm/hardirq.h         |   2 +
 arch/x86/include/asm/msr-index.h       |  16 +
 arch/x86/mm/Makefile                   |   1 +
 arch/x86/mm/ibs.c                      | 349 +++++++++++++++++
 include/linux/migrate.h                |   6 +
 include/linux/mmzone.h                 |  26 ++
 include/linux/pghot.h                  | 142 +++++++
 include/linux/vm_event_item.h          |  26 ++
 kernel/sched/debug.c                   |   1 -
 kernel/sched/fair.c                    | 152 +-------
 mm/Kconfig                             |  46 +++
 mm/Makefile                            |   7 +
 mm/huge_memory.c                       |  26 +-
 mm/internal.h                          |   4 +
 mm/klruscand.c                         | 110 ++++++
 mm/memory.c                            |  31 +-
 mm/migrate.c                           |  41 +-
 mm/mm_init.c                           |  10 +
 mm/pghot-default.c                     |  73 ++++
 mm/pghot-precise.c                     |  70 ++++
 mm/pghot-tunables.c                    | 196 ++++++++++
 mm/pghot.c                             | 505 +++++++++++++++++++++++++
 mm/swap.c                              |   8 +
 mm/vmscan.c                            | 181 ++++++---
 mm/vmstat.c                            |  26 ++
 28 files changed, 1917 insertions(+), 240 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/pghot.txt
 create mode 100644 arch/x86/mm/ibs.c
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/klruscand.c
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot-precise.c
 create mode 100644 mm/pghot-tunables.c
 create mode 100644 mm/pghot.c
Signature · 1 line
2.34.1
#1 lore raw

RFC PATCH v5 1/10 mm: migrate: Allow misplaced migration without VMA

We want isolation of misplaced folios to work in contexts where VMA isn't available, typically when performing migrations from a kernel thread context. In order to prepare for that, allow migrate_misplaced_folio_prepare() to be called with a NULL VMA.

When migrate_misplaced_folio_prepare() is called with non-NULL VMA, it will check if the folio is mapped shared and that requires holding PTL lock. This path isn't taken when the function is invoked with NULL VMA (migration outside of process context). Therefore, when VMA == NULL, migrate_misplaced_folio_prepare() does not require the caller to hold the PTL.

---
 mm/migrate.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/migrate.c +3 -2
index 5169f9717f60..70f8f3ad4fd8 100644
@@ -2652,7 +2652,8 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,
 
 /*
  * Prepare for calling migrate_misplaced_folio() by isolating the folio if
- * permitted. Must be called with the PTL still held.
+ * permitted. Must be called with the PTL still held if called with a non-NULL
+ * vma.
  */
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -2669,7 +2670,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 		 * See folio_maybe_mapped_shared() on possible imprecision
 		 * when we cannot easily detect if a folio is shared.
 		 */
-		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+		if (vma && (vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
 			return -EACCES;
 
 		/*
Signature · 1 line
2.34.1
#2 lore raw

RFC PATCH v5 2/10 migrate: Add migrate_misplaced_folios_batch()

In reply to Bharata B Rao

From: Gregory Price <[email protected]>

Tiered memory systems often require migrating multiple folios at once. Currently, migrate_misplaced_folio() handles only one folio per call, which is inefficient for batch operations. This patch introduces migrate_misplaced_folios_batch(), a batch variant that leverages migrate_pages() internally for improved performance.

The caller must isolate folios beforehand using migrate_misplaced_folio_prepare(). On return, the folio list will be empty regardless of success or failure.

This function will be used by pghot kmigrated thread.

[Rewrote commit description]
---
 include/linux/migrate.h |  6 ++++++
 mm/migrate.c            | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+)
include/linux/migrate.h +6 -0
index 26ca00c325d9..f28326b88592 100644
@@ -103,6 +103,7 @@ static inline int set_movable_ops(const struct movable_operations *ops, enum pag
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
+int migrate_misplaced_folios_batch(struct list_head *folio_list, int node);
 #else
 static inline int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -113,6 +114,11 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
+static inline int migrate_misplaced_folios_batch(struct list_head *folio_list,
+						 int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_MIGRATION
mm/migrate.c +36 -0
index 70f8f3ad4fd8..4a3a9a4ff435 100644
@@ -2747,5 +2747,41 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	BUG_ON(!list_empty(&migratepages));
 	return nr_remaining ? -EAGAIN : 0;
 }
+
+/**
+ * migrate_misplaced_folios_batch() - Batch variant of migrate_misplaced_folio.
+ * Attempts to migrate a folio list to the specified destination.
+ * @folio_list: Isolated list of folios to be batch-migrated.
+ * @node: The NUMA node ID to where the folios should be migrated.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folio.
+ *
+ * This function will un-isolate the folios, drop the elevated reference
+ * and remove them from the list before returning.
+ *
+ * Return: 0 on success and -EAGAIN on failure or partial migration.
+ *         On return, @folio_list will be empty regardless of success/failure.
+ */
+int migrate_misplaced_folios_batch(struct list_head *folio_list, int node)
+{
+	pg_data_t *pgdat = NODE_DATA(node);
+	unsigned int nr_succeeded = 0;
+	int nr_remaining;
+
+	nr_remaining = migrate_pages(folio_list, alloc_misplaced_dst_folio,
+				     NULL, node, MIGRATE_ASYNC,
+				     MR_NUMA_MISPLACED, &nr_succeeded);
+	if (nr_remaining)
+		putback_movable_pages(folio_list);
+
+	if (nr_succeeded) {
+		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded);
+	}
+	WARN_ON(!list_empty(folio_list));
+	return nr_remaining ? -EAGAIN : 0;
+}
 #endif /* CONFIG_NUMA_BALANCING */
 #endif /* CONFIG_NUMA */
Signature · 1 line
2.34.1
reviewer
#3 lore raw

Re: [RFC PATCH v5 02/10] migrate: Add migrate_misplaced_folios_batch()

Quoted 18 lines · On Thu, 29 Jan 2026 20:10:35 +0530 Bharata B Rao <[email protected]> wrote:

From: Gregory Price <[email protected]>

Tiered memory systems often require migrating multiple folios at once. Currently, migrate_misplaced_folio() handles only one folio per call, which is inefficient for batch operations. This patch introduces migrate_misplaced_folios_batch(), a batch variant that leverages migrate_pages() internally for improved performance.

The caller must isolate folios beforehand using migrate_misplaced_folio_prepare(). On return, the folio list will be empty regardless of success or failure.

This function will be used by pghot kmigrated thread.

[Rewrote commit description]

---

[...snip...]

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2777

Quoted 2 lines
mm/migrate.c +2 -0
+	if (nr_remaining)
+		putback_movable_pages(folio_list);

Hello Bharata, I hope you are doing well! Thank you for the series. I saw the numbers and they look great. I'm hoping to do some more testing myself as well : -) I'm also going through the series as well!!

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2779

Quoted 1 line
mm/migrate.c +1 -0
+	if (nr_succeeded) {

The single-folio case, migrate_misplaced_folio, has a guard here to check that the function performs more than just a migration, but a promotion. Specifically, it checks that the folio's node is not toptier, and the destination node is toptier. Should that also be included here?

When this is called in kmigrated_walk_zone in the next patch, there is no check to make sure that the folios are actually on a lower tier, and the destination is on a higher tier. Maybe I'm missing something? But it wasn't entirely obvious to me that the migration is always a promotion.

I want to note that we're also skipping the count_memcg_events, which I understand is much harder to do here becuase each folio might belong to a different memcg. Ying also noted this in his reply from v1 [1] but I don't think it ever got addressed.

Anyways, thank you! I hope you have a great day!
Joshua

Quoted 11 lines
mm/migrate.c +6 -0
+		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
+		mod_node_page_state(pgdat, PGPROMOTE_SUCCESS, nr_succeeded);
+	}
+	WARN_ON(!list_empty(folio_list));
+	return nr_remaining ? -EAGAIN : 0;
+}
 #endif /* CONFIG_NUMA_BALANCING */
 #endif /* CONFIG_NUMA */
Signature · 1 line
2.34.1

[1] https://lore.kernel.org/linux-mm/87a541e51s.fsf@DESKTOP-5N7EMDA/

#4 lore raw

Re: [RFC PATCH v5 02/10] migrate: Add migrate_misplaced_folios_batch()

Quoted 4 lines · Joshua Hahn

Hello Bharata, I hope you are doing well! Thank you for the series. I saw the numbers and they look great. I'm hoping to do some more testing myself as well : -) I'm also going through the series as well!!

Thanks Joshua for looking at the patchset and for your testing offer!

Quoted 7 lines

mm/migrate.c int migrate_misplaced_folio(struct folio *folio, int node) · line 2779

Quoted 2 lines
mm/migrate.c +1 -0
+	if (nr_succeeded) {

The single-folio case, migrate_misplaced_folio, has a guard here to check that the function performs more than just a migration, but a promotion. Specifically, it checks that the folio's node is not toptier, and the destination node is toptier. Should that also be included here?

Ideally yes, but right now the batch variant gets called only for promotion case.

Quoted 5 lines

When this is called in kmigrated_walk_zone in the next patch, there is no check to make sure that the folios are actually on a lower tier, and the destination is on a higher tier. Maybe I'm missing something? But it wasn't entirely obvious to me that the migration is always a promotion.

Firstly the hotness is tracked only for lower tier pages. pghot_record_access() ensures this.

Next, there is one kmigrated thread for each lower tier node and it looks at its own PFNs. This ensures that only lower tier PFNs are considered for promotion.

Quoted 5 lines

I want to note that we're also skipping the count_memcg_events, which I understand is much harder to do here becuase each folio might belong to a different memcg. Ying also noted this in his reply from v1 [1] but I don't think it ever got addressed.

Ying's suggestion about unifying single and batch versions of misplaced migration routines is in my TODO list. memcg accounting looks harder, I will give it a try.

Regards,
Bharata.

#5 lore raw

RFC PATCH v5 3/10 mm: Hot page tracking and promotion

In reply to Bharata B Rao

This introduces a subsystem for collecting memory access information from different sources. It maintains the hotness information based on the access history and time of access.

Additionally, it provides per-lower-tier-node kernel threads (named kmigrated) that periodically promote the pages that are eligible for promotion.

Sub-systems that generate hot page access info can report that using this API:

int pghot_record_access(unsigned long pfn, int nid, int src,
unsigned long time)

@pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffies

Some temperature sources may not provide the nid from which the page was accessed. This is true for sources that use page table scanning for PTE Accessed bit. For such sources, a configurable/default toptier node is used as promotion target.

The hotness information is stored for every page of lower tier memory in a u8 variable (1 byte) that is part of mem_section data structure.

kmigrated is a per-lower-tier-node kernel thread that migrates the folios marked for migration in batches. Each kmigrated thread walks the PFN range spanning its node and checks for potential migration candidates.

A bunch of tunables for enabling different hotness sources, setting target_nid, frequency threshold are provided in debugfs.

---
 Documentation/admin-guide/mm/pghot.txt |  84 ++++++
 include/linux/mmzone.h                 |  21 ++
 include/linux/pghot.h                  |  94 +++++++
 include/linux/vm_event_item.h          |   6 +
 mm/Kconfig                             |  14 +
 mm/Makefile                            |   1 +
 mm/mm_init.c                           |  10 +
 mm/pghot-default.c                     |  73 +++++
 mm/pghot-tunables.c                    | 189 +++++++++++++
 mm/pghot.c                             | 370 +++++++++++++++++++++++++
 mm/vmstat.c                            |   6 +
 11 files changed, 868 insertions(+)
 create mode 100644 Documentation/admin-guide/mm/pghot.txt
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot-tunables.c
 create mode 100644 mm/pghot.c
Documentation/admin-guide/mm/pghot.txt new file +84 -0
new file mode 100644
index 000000000000..01291b72e7ab
@@ -0,0 +1,84 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=================================
+PGHOT: Hot Page Tracking Tunables
+=================================
+
+Overview
+========
+The PGHOT subsystem tracks frequently accessed pages in lower-tier memory and
+promotes them to faster tiers. It uses per-PFN hotness metadata and asynchronous
+migration via per-node kernel threads (kmigrated).
+
+This document describes tunables available via **debugfs** and **sysctl** for
+PGHOT.
+
+Debugfs Interface
+=================
+Path: /sys/kernel/debug/pghot/
+
+1. **enabled_sources**
+   - Bitmask to enable/disable hotness sources.
+   - Bits:
+     - 0: Hardware hints (value 0x1)
+     - 1: Page table scan (value 0x2)
+     - 2: Hint faults (value 0x4)
+   - Default: 0 (disabled)
+   - Example:
+     # echo 0x7 > /sys/kernel/debug/pghot/enabled_sources
+     Enables all sources.
+
+2. **target_nid**
+   - Toptier NUMA node ID to which hot pages should be promoted when source
+     does not provide nid. Used when hotness source can't provide accessing
+     NID or when the tracking mode is default.
+   - Default: 0
+   - Example:
+     # echo 1 > /sys/kernel/debug/pghot/target_nid
+
+3. **freq_threshold**
+   - Minimum access frequency before a page is marked ready for promotion.
+   - Range: 1 to 3
+   - Default: 2
+   - Example:
+     # echo 3 > /sys/kernel/debug/pghot/freq_threshold
+
+4. **kmigrated_sleep_ms**
+   - Sleep interval (ms) for kmigrated thread between scans.
+   - Default: 100
+
+5. **kmigrated_batch_nr**
+   - Maximum number of folios migrated in one batch.
+   - Default: 512
+
+Sysctl Interface
+================
+1. pghot_promote_freq_window_ms
+
+Path: /proc/sys/vm/pghot_promote_freq_window_ms
+
+- Controls the time window (in ms) for counting access frequency. A page is
+  considered hot only when **freq_threshold** number of accesses occur with
+  this time period.
+- Default: 4000 (4 seconds)
+- Example:
+  # sysctl vm.pghot_promote_freq_window_ms=3000
+
+Vmstat Counters
+===============
+Following vmstat counters provide some stats about pghot subsystem.
+
+Path: /proc/vmstat
+
+1. **pghot_recorded_accesses**
+   - Number of total hot page accesses recorded by pghot.
+
+2. **pghot_recorded_hwhints**
+   - Number of recorded accesses reported by hwhints source.
+
+3. **pghot_recorded_pgtscans**
+   - Number of recorded accesses reported by PTE A-bit based source.
+
+4. **pghot_recorded_hintfaults**
+   - Number of recorded accesses reported by NUMA Balancing based
+     hotness source.
include/linux/mmzone.h +21 -0
index 75ef7c9f9307..22e08befb096 100644
@@ -1064,6 +1064,7 @@ enum pgdat_flags {
 					 * many pages under writeback
 					 */
 	PGDAT_RECLAIM_LOCKED,		/* prevents concurrent reclaim */
+	PGDAT_KMIGRATED_ACTIVATE,	/* activates kmigrated */
 };
 
 enum zone_flags {
@@ -1518,6 +1519,10 @@ typedef struct pglist_data {
 #ifdef CONFIG_MEMORY_FAILURE
 	struct memory_failure_stats mf_stats;
 #endif
+#ifdef CONFIG_PGHOT
+	struct task_struct *kmigrated;
+	wait_queue_head_t kmigrated_wait;
+#endif
 } pg_data_t;
 
 #define node_present_pages(nid)	(NODE_DATA(nid)->node_present_pages)
@@ -1916,12 +1921,28 @@ struct mem_section {
 	unsigned long section_mem_map;
 
 	struct mem_section_usage *usage;
+#ifdef CONFIG_PGHOT
+	/*
+	 * Per-PFN hotness data for this section.
+	 * Array of phi_t (u8 in default mode).
+	 * LSB is used as PGHOT_SECTION_HOT_BIT flag.
+	 */
+	void *hot_map;
+#endif
 #ifdef CONFIG_PAGE_EXTENSION
 	/*
 	 * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use
 	 * section. (see page_ext.h about this.)
 	 */
 	struct page_ext *page_ext;
+#endif
+	/*
+	 * Padding to maintain consistent mem_section size when exactly
+	 * one of PGHOT or PAGE_EXTENSION is enabled. This ensures
+	 * optimal alignment regardless of configuration.
+	 */
+#if (defined(CONFIG_PGHOT) && !defined(CONFIG_PAGE_EXTENSION)) || \
+		(!defined(CONFIG_PGHOT) && defined(CONFIG_PAGE_EXTENSION))
 	unsigned long pad;
 #endif
 	/*
include/linux/pghot.h new file +94 -0

94 lines not on this page · raw message

include/linux/vm_event_item.h +6 -0
index 92f80b4d69a6..5b8fd93b55fd 100644
@@ -188,6 +188,12 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		KSTACK_REST,
 #endif
 #endif /* CONFIG_DEBUG_STACK_USAGE */
+#ifdef CONFIG_PGHOT
+		PGHOT_RECORDED_ACCESSES,
+		PGHOT_RECORD_HWHINTS,
+		PGHOT_RECORD_PGTSCANS,
+		PGHOT_RECORD_HINTFAULTS,
+#endif /* CONFIG_PGHOT */
 		NR_VM_EVENT_ITEMS
 };
 
mm/Kconfig +14 -0
index bd0ea5454af8..f4f0147faac5 100644
@@ -1464,6 +1464,20 @@ config PT_RECLAIM
 config FIND_NORMAL_PAGE
 	def_bool n
 
+config PGHOT
+	bool "Hot page tracking and promotion"
+	def_bool n
+	depends on NUMA && MIGRATION && SPARSEMEM && MMU
+	help
+	  A sub-system to track page accesses in lower tier memory and
+	  maintain hot page information. Promotes hot pages from lower
+	  tiers to top tier by using the memory access information provided
+	  by various sources. Asynchronous promotion is done by per-node
+	  kernel threads.
+
+	  This adds 1 byte of metadata overhead per page in lower-tier
+	  memory nodes.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +1 -0
index 2d0570a16e5b..655a27f3a215 100644
@@ -147,3 +147,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o
+obj-$(CONFIG_PGHOT) += pghot.o pghot-tunables.o pghot-default.o
mm/mm_init.c +10 -0
index fc2a6f1e518f..64109feaa1c3 100644
@@ -1401,6 +1401,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)
 static void pgdat_init_kcompactd(struct pglist_data *pgdat) {}
 #endif
 
+#ifdef CONFIG_PGHOT
+static void pgdat_init_kmigrated(struct pglist_data *pgdat)
+{
+	init_waitqueue_head(&pgdat->kmigrated_wait);
+}
+#else
+static inline void pgdat_init_kmigrated(struct pglist_data *pgdat) {}
+#endif
+
 static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 {
 	int i;
@@ -1410,6 +1419,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 
 	pgdat_init_split_queue(pgdat);
 	pgdat_init_kcompactd(pgdat);
+	pgdat_init_kmigrated(pgdat);
 
 	init_waitqueue_head(&pgdat->kswapd_wait);
 	init_waitqueue_head(&pgdat->pfmemalloc_wait);
mm/pghot-default.c new file +73 -0

73 lines not on this page · raw message

mm/pghot-tunables.c new file +189 -0

189 lines not on this page · raw message

mm/pghot.c new file +370 -0

370 lines not on this page · raw message

mm/vmstat.c +6 -0
index 65de88cdf40e..f6f91b9dd887 100644
@@ -1501,6 +1501,12 @@ const char * const vmstat_text[] = {
 	[I(KSTACK_REST)]			= "kstack_rest",
 #endif
 #endif
+#ifdef CONFIG_PGHOT
+	[I(PGHOT_RECORDED_ACCESSES)]		= "pghot_recorded_accesses",
+	[I(PGHOT_RECORD_HWHINTS)]		= "pghot_recorded_hwhints",
+	[I(PGHOT_RECORD_PGTSCANS)]		= "pghot_recorded_pgtscans",
+	[I(PGHOT_RECORD_HINTFAULTS)]		= "pghot_recorded_hintfaults",
+#endif /* CONFIG_PGHOT */
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
 };
Signature · 1 line
2.34.1
#6 lore raw

Re: [RFC PATCH v5 03/10] mm: Hot page tracking and promotion

Quoted 36 lines · Bharata B Rao
mm/pghot.c new file +36 -0
+
+/*
+ * Walks the PFNs of the zone, isolates and migrates them in batches.
+ */
+static void kmigrated_walk_zone(unsigned long start_pfn, unsigned long end_pfn,
+				int src_nid)
+{
+	int cur_nid = NUMA_NO_NODE;
+	LIST_HEAD(migrate_list);
+	int batch_count = 0;
+	struct folio *folio;
+	struct page *page;
+	unsigned long pfn;
+
+	pfn = start_pfn;
+	do {
+		int nid = NUMA_NO_NODE, nr = 1;
+		int freq = 0;
+		unsigned long time = 0;
+
+		if (!pfn_valid(pfn))
+			goto out_next;
+
+		page = pfn_to_online_page(pfn);
+		if (!page)
+			goto out_next;
+
+		folio = page_folio(page);
+		nr = folio_nr_pages(folio);
+		if (folio_nid(folio) != src_nid)
+			goto out_next;
+
+		if (!folio_test_lru(folio))
+			goto out_next;
+
+		if (pghot_get_hotness(pfn, &nid, &freq, &time))

mm/pghot.c line 196

Quoted 10 lines
mm/pghot.c new file +10 -0
+			goto out_next;
+
+		if (nid == NUMA_NO_NODE)
+			nid = pghot_target_nid;
+
+		if (folio_nid(folio) == nid)
+			goto out_next;
+
+		if (migrate_misplaced_folio_prepare(folio, NULL, nid))
+			goto out_next;

We should hold a folio reference before the above call which will isolate the folio from LRU. Otherwise we may hit

VM_BUG_ON_FOLIO(!folio_ref_count(folio), folio)

in folio_isolate_lru().

I hit this only when running Graph500 benchmark and have fixed it in the github at: https://github.com/AMDESE/linux-mm/tree/bharata/pghot-rfcv6-pre

The numbers that I have posted for micro-benchmarks and redis-memtier are without this fix while Graph500 numbers are with this fix.

Regards,
Bharata.

reviewer
#7 lore raw

Re: [RFC PATCH v5 03/10] mm: Hot page tracking and promotion

Quoted 42 lines · Bharata B Rao
Quoted 41 lines · Bharata B Rao
mm/pghot.c new file +41 -0
+
+/*
+ * Walks the PFNs of the zone, isolates and migrates them in batches.
+ */
+static void kmigrated_walk_zone(unsigned long start_pfn, unsigned long end_pfn,
+				int src_nid)
+{
+	int cur_nid = NUMA_NO_NODE;
+	LIST_HEAD(migrate_list);
+	int batch_count = 0;
+	struct folio *folio;
+	struct page *page;
+	unsigned long pfn;
+
+	pfn = start_pfn;
+	do {
+		int nid = NUMA_NO_NODE, nr = 1;
+		int freq = 0;
+		unsigned long time = 0;
+
+		if (!pfn_valid(pfn))
+			goto out_next;
+
+		page = pfn_to_online_page(pfn);
+		if (!page)
+			goto out_next;
+
+		folio = page_folio(page);
+		nr = folio_nr_pages(folio);
+		if (folio_nid(folio) != src_nid)
+			goto out_next;
+
+		if (!folio_test_lru(folio))
+			goto out_next;
+
+		if (pghot_get_hotness(pfn, &nid, &freq, &time))
+			goto out_next;
+
+		if (nid == NUMA_NO_NODE)
+			nid = pghot_target_nid;
+
Quoted 9 lines

mm/pghot.c line 196

Quoted 5 lines
mm/pghot.c new file +5 -0
+		if (folio_nid(folio) == nid)
+			goto out_next;
+
+		if (migrate_misplaced_folio_prepare(folio, NULL, nid))
+			goto out_next;

We should hold a folio reference before the above call which will isolate the folio from LRU. Otherwise we may hit

Also relevant note from other work I'm doing, we may want a fast-out for zone-device folios here. We should not bother tracking those at all.

(this may also become relevant for private-node memory as well, but I may try to generalize zone_device & private-node checks as the conditions are very similar).

~Gregory

#8 lore raw

Re: [RFC PATCH v5 03/10] mm: Hot page tracking and promotion

Quoted 53 lines · Gregory Price
Quoted 42 lines · Bharata B Rao
Quoted 41 lines · Bharata B Rao
mm/pghot.c new file +41 -0
+
+/*
+ * Walks the PFNs of the zone, isolates and migrates them in batches.
+ */
+static void kmigrated_walk_zone(unsigned long start_pfn, unsigned long end_pfn,
+				int src_nid)
+{
+	int cur_nid = NUMA_NO_NODE;
+	LIST_HEAD(migrate_list);
+	int batch_count = 0;
+	struct folio *folio;
+	struct page *page;
+	unsigned long pfn;
+
+	pfn = start_pfn;
+	do {
+		int nid = NUMA_NO_NODE, nr = 1;
+		int freq = 0;
+		unsigned long time = 0;
+
+		if (!pfn_valid(pfn))
+			goto out_next;
+
+		page = pfn_to_online_page(pfn);
+		if (!page)
+			goto out_next;
+
+		folio = page_folio(page);
+		nr = folio_nr_pages(folio);
+		if (folio_nid(folio) != src_nid)
+			goto out_next;
+
+		if (!folio_test_lru(folio))
+			goto out_next;
+
+		if (pghot_get_hotness(pfn, &nid, &freq, &time))
+			goto out_next;
+
+		if (nid == NUMA_NO_NODE)
+			nid = pghot_target_nid;
+
Quoted 9 lines

mm/pghot.c line 196

Quoted 5 lines
mm/pghot.c new file +5 -0
+		if (folio_nid(folio) == nid)
+			goto out_next;
+
+		if (migrate_misplaced_folio_prepare(folio, NULL, nid))
+			goto out_next;

We should hold a folio reference before the above call which will isolate the folio from LRU. Otherwise we may hit

Quoted 2 lines

Also relevant note from other work I'm doing, we may want a fast-out for zone-device folios here. We should not bother tracking those at all.

Yes, zone device folios aren't not tracked by pghot. They get discarded by pghot_record_access() itself.

Quoted 4 lines

(this may also become relevant for private-node memory as well, but I may try to generalize zone_device & private-node checks as the conditions are very similar).

Good.

Regards,
Bharata.

#9 lore raw

RFC PATCH v5 4/10 mm: pghot: Precision mode for pghot

In reply to Bharata B Rao

By default, one byte per PFN is used to store hotness information. Limited number of bits are used to store the access time leading to coarse-grained time tracking. Also there aren't enough bits to track the toptier NID explicitly and hence the default target_nid is used for promotion.

This precise mode relaxes the above situation by storing the hotness information in 4 bytes per PFN. More fine-grained access time tracking and toptier NID tracking becomes possible in this mode.

Typically useful when toptier consists of more than one node.

---
 Documentation/admin-guide/mm/pghot.txt |  4 +-
 include/linux/mmzone.h                 |  2 +-
 include/linux/pghot.h                  | 31 ++++++++++++
 mm/Kconfig                             | 11 ++++
 mm/Makefile                            |  7 ++-
 mm/pghot-precise.c                     | 70 ++++++++++++++++++++++++++
 mm/pghot.c                             | 13 +++--
 7 files changed, 130 insertions(+), 8 deletions(-)
 create mode 100644 mm/pghot-precise.c
Documentation/admin-guide/mm/pghot.txt +2 -2
index 01291b72e7ab..b329e692ef89 100644
@@ -38,7 +38,7 @@ Path: /sys/kernel/debug/pghot/
 
 3. **freq_threshold**
    - Minimum access frequency before a page is marked ready for promotion.
-   - Range: 1 to 3
+   - Range: 1 to 3 in default mode, 1 to 7 in precision mode.
    - Default: 2
    - Example:
      # echo 3 > /sys/kernel/debug/pghot/freq_threshold
@@ -60,7 +60,7 @@ Path: /proc/sys/vm/pghot_promote_freq_window_ms
 - Controls the time window (in ms) for counting access frequency. A page is
   considered hot only when **freq_threshold** number of accesses occur with
   this time period.
-- Default: 4000 (4 seconds)
+- Default: 4000 (4 seconds) in default mode and 5000 (5s) in precision mode.
 - Example:
   # sysctl vm.pghot_promote_freq_window_ms=3000
 
include/linux/mmzone.h +1 -1
index 22e08befb096..49c374064fc2 100644
@@ -1924,7 +1924,7 @@ struct mem_section {
 #ifdef CONFIG_PGHOT
 	/*
 	 * Per-PFN hotness data for this section.
-	 * Array of phi_t (u8 in default mode).
+	 * Array of phi_t (u8 in default mode, u32 in precision mode).
 	 * LSB is used as PGHOT_SECTION_HOT_BIT flag.
 	 */
 	void *hot_map;
include/linux/pghot.h +31 -0
index 88e57aab697b..d3d59b0c0cf6 100644
@@ -48,6 +48,36 @@ enum pghot_src_enabled {
 
 #define PGHOT_DEFAULT_NODE		0
 
+#if defined(CONFIG_PGHOT_PRECISE)
+#define PGHOT_DEFAULT_FREQ_WINDOW	(5 * MSEC_PER_SEC)
+
+/*
+ * Bits 0-26 are used to store nid, frequency and time.
+ * Bits 27-30 are unused now.
+ * Bit 31 is used to indicate the page is ready for migration.
+ */
+#define PGHOT_MIGRATE_READY		31
+
+#define PGHOT_NID_WIDTH			10
+#define PGHOT_FREQ_WIDTH		3
+/* time is stored in 14 bits which can represent up to 16s with HZ=1000 */
+#define PGHOT_TIME_WIDTH		14
+
+#define PGHOT_NID_SHIFT			0
+#define PGHOT_FREQ_SHIFT		(PGHOT_NID_SHIFT + PGHOT_NID_WIDTH)
+#define PGHOT_TIME_SHIFT		(PGHOT_FREQ_SHIFT + PGHOT_FREQ_WIDTH)
+
+#define PGHOT_NID_MASK			GENMASK(PGHOT_NID_WIDTH - 1, 0)
+#define PGHOT_FREQ_MASK			GENMASK(PGHOT_FREQ_WIDTH - 1, 0)
+#define PGHOT_TIME_MASK			GENMASK(PGHOT_TIME_WIDTH - 1, 0)
+
+#define PGHOT_NID_MAX			((1 << PGHOT_NID_WIDTH) - 1)
+#define PGHOT_FREQ_MAX			((1 << PGHOT_FREQ_WIDTH) - 1)
+#define PGHOT_TIME_MAX			((1 << PGHOT_TIME_WIDTH) - 1)
+
+typedef u32 phi_t;
+
+#else	/* !CONFIG_PGHOT_PRECISE */
 #define PGHOT_DEFAULT_FREQ_WINDOW	(4 * MSEC_PER_SEC)
 
 /*
@@ -74,6 +104,7 @@ enum pghot_src_enabled {
 #define PGHOT_TIME_MAX			((1 << PGHOT_TIME_WIDTH) - 1)
 
 typedef u8 phi_t;
+#endif /* CONFIG_PGHOT_PRECISE */
 
 #define PGHOT_RECORD_SIZE		sizeof(phi_t)
 
mm/Kconfig +11 -0
index f4f0147faac5..fde5aee3e16f 100644
@@ -1478,6 +1478,17 @@ config PGHOT
 	  This adds 1 byte of metadata overhead per page in lower-tier
 	  memory nodes.
 
+config PGHOT_PRECISE
+	bool "Hot page tracking precision mode"
+	def_bool n
+	depends on PGHOT
+	help
+	  Enables precision mode for tracking hot pages with pghot sub-system.
+	  Adds fine-grained access time tracking and explicit toptier target
+	  NID tracking. Precise hot page tracking comes at the cost of using
+	  4 bytes per page against the default one byte per page. Preferable
+	  to enable this on systems with multiple nodes in toptier.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +6 -1
index 655a27f3a215..89f999647752 100644
@@ -147,4 +147,9 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_PT_RECLAIM) += pt_reclaim.o
-obj-$(CONFIG_PGHOT) += pghot.o pghot-tunables.o pghot-default.o
+obj-$(CONFIG_PGHOT) += pghot.o pghot-tunables.o
+ifdef CONFIG_PGHOT_PRECISE
+obj-$(CONFIG_PGHOT) += pghot-precise.o
+else
+obj-$(CONFIG_PGHOT) += pghot-default.o
+endif
mm/pghot-precise.c new file +70 -0

70 lines not on this page · raw message

mm/pghot.c +9 -4
index 95b5012d5b99..bf1d9029cbaa 100644
@@ -10,6 +10,9 @@
  * the frequency of access and last access time. Promotions are done
  * to a default toptier NID.
  *
+ * In the precision mode, 4 bytes are used to store the frequency
+ * of access, last access time and the accessing NID.
+ *
  * A kernel thread named kmigrated is provided to migrate or promote
  * the hot pages. kmigrated runs for each lower tier node. It iterates
  * over the node's PFNs and  migrates pages marked for migration into
@@ -52,13 +55,15 @@ static bool kmigrated_started __ro_after_init;
  * for the purpose of tracking page hotness and subsequent promotion.
  *
  * @pfn: PFN of the page
- * @nid: Unused
+ * @nid: Target NID to where the page needs to be migrated in precision
+ *       mode but unused in default mode
  * @src: The identifier of the sub-system that reports the access
  * @now: Access time in jiffies
  *
- * Updates the frequency and time of access and marks the page as
- * ready for migration if the frequency crosses a threshold. The pages
- * marked for migration are migrated by kmigrated kernel thread.
+ * Updates the NID (in precision mode only), frequency and time of access
+ * and marks the page as ready for migration if the frequency crosses a
+ * threshold. The pages marked for migration are migrated by kmigrated
+ * kernel thread.
  *
  * Return: 0 on success and -EINVAL on failure to record the access.
  */
Signature · 1 line
2.34.1
#10 lore raw

RFC PATCH v5 5/10 mm: sched: move NUMA balancing tiering promotion to pghot

In reply to Bharata B Rao

Currently hot page promotion (NUMA_BALANCING_MEMORY_TIERING mode of NUMA Balancing) does hot page detection (via hint faults), hot page classification and eventual promotion, all by itself and sits within the scheduler.

With pghot, the new hot page tracking and promotion mechanism being available, NUMA Balancing can limit itself to detection of hot pages (via hint faults) and off-load rest of the functionality to the common hot page tracking system.

pghot_record_access(PGHOT_HINT_FAULT) API is used to feed the hot page info to pghot. In addition, the migration rate limiting and dynamic threshold logic are moved to kmigrated so that the same can be used for hot pages reported by other sources too.

---
 kernel/sched/debug.c |   1 -
 kernel/sched/fair.c  | 152 ++-----------------------------------------
 mm/huge_memory.c     |  26 ++------
 mm/memory.c          |  31 ++-------
 mm/pghot.c           | 124 +++++++++++++++++++++++++++++++++++
 5 files changed, 141 insertions(+), 193 deletions(-)
kernel/sched/debug.c +0 -1
index 41caa22e0680..02931902a9c6 100644
@@ -520,7 +520,6 @@ static __init int sched_init_debug(void)
 	debugfs_create_u32("scan_period_min_ms", 0644, numa, &sysctl_numa_balancing_scan_period_min);
 	debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max);
 	debugfs_create_u32("scan_size_mb", 0644, numa, &sysctl_numa_balancing_scan_size);
-	debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold);
 #endif /* CONFIG_NUMA_BALANCING */
 
 	debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);
kernel/sched/fair.c +5 -147

182 lines not on this page · raw message

mm/huge_memory.c +6 -20
index 40cf59301c21..f52587e70b3c 100644
@@ -40,6 +40,7 @@
 #include <linux/pgalloc.h>
 #include <linux/pgalloc_tag.h>
 #include <linux/pagewalk.h>
+#include <linux/pghot.h>
 
 #include <asm/tlb.h>
 #include "internal.h"
@@ -2217,29 +2218,12 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 
 	target_nid = numa_migrate_check(folio, vmf, haddr, &flags, writable,
 					&last_cpupid);
+	nid = target_nid;
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
-	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
-		flags |= TNF_MIGRATE_FAIL;
-		goto out_map;
-	}
-	/* The folio is isolated and isolation code holds a folio reference. */
-	spin_unlock(vmf->ptl);
-	writable = false;
 
-	if (!migrate_misplaced_folio(folio, target_nid)) {
-		flags |= TNF_MIGRATED;
-		nid = target_nid;
-		task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags);
-		return 0;
-	}
+	writable = false;
 
-	flags |= TNF_MIGRATE_FAIL;
-	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
-	if (unlikely(!pmd_same(pmdp_get(vmf->pmd), vmf->orig_pmd))) {
-		spin_unlock(vmf->ptl);
-		return 0;
-	}
 out_map:
 	/* Restore the PMD */
 	pmd = pmd_modify(pmdp_get(vmf->pmd), vma->vm_page_prot);
@@ -2250,8 +2234,10 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
 	spin_unlock(vmf->ptl);
 
-	if (nid != NUMA_NO_NODE)
+	if (nid != NUMA_NO_NODE) {
+		pghot_record_access(folio_pfn(folio), nid, PGHOT_HINT_FAULT, jiffies);
 		task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags);
+	}
 	return 0;
 }
 
mm/memory.c +6 -25
index 2a55edc48a65..98a9a3b675a0 100644
@@ -75,6 +75,7 @@
 #include <linux/perf_event.h>
 #include <linux/ptrace.h>
 #include <linux/vmalloc.h>
+#include <linux/pghot.h>
 #include <linux/sched/sysctl.h>
 #include <linux/pgalloc.h>
 #include <linux/uaccess.h>
@@ -6046,34 +6047,12 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 
 	target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
 					writable, &last_cpupid);
+	nid = target_nid;
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
-	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
-		flags |= TNF_MIGRATE_FAIL;
-		goto out_map;
-	}
-	/* The folio is isolated and isolation code holds a folio reference. */
-	pte_unmap_unlock(vmf->pte, vmf->ptl);
+
 	writable = false;
 	ignore_writable = true;
-
-	/* Migrate to the requested node */
-	if (!migrate_misplaced_folio(folio, target_nid)) {
-		nid = target_nid;
-		flags |= TNF_MIGRATED;
-		task_numa_fault(last_cpupid, nid, nr_pages, flags);
-		return 0;
-	}
-
-	flags |= TNF_MIGRATE_FAIL;
-	vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
-				       vmf->address, &vmf->ptl);
-	if (unlikely(!vmf->pte))
-		return 0;
-	if (unlikely(!pte_same(ptep_get(vmf->pte), vmf->orig_pte))) {
-		pte_unmap_unlock(vmf->pte, vmf->ptl);
-		return 0;
-	}
 out_map:
 	/*
 	 * Make it present again, depending on how arch implements
@@ -6087,8 +6066,10 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 					    writable);
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
 
-	if (nid != NUMA_NO_NODE)
+	if (nid != NUMA_NO_NODE) {
+		pghot_record_access(folio_pfn(folio), nid, PGHOT_HINT_FAULT, jiffies);
 		task_numa_fault(last_cpupid, nid, nr_pages, flags);
+	}
 	return 0;
 }
 
mm/pghot.c +124 -0

154 lines not on this page · raw message

Signature · 1 line
2.34.1
#11 lore raw

RFC PATCH v5 6/10 x86: ibs: In-kernel IBS driver for memory access profiling

In reply to Bharata B Rao

Use IBS (Instruction Based Sampling) feature present in AMD processors for memory access tracking. The access information obtained from IBS via NMI is fed to pghot sub-system for futher action.

In addition to many other information related to the memory access, IBS provides physical (and virtual) address of the access and indicates if the access came from slower tier. Only memory accesses originating from slower tiers are further acted upon by this driver.

The samples are initially accumulated in percpu buffers which are flushed to pghot hot page tracking mechanism using irq_work.

TODO: Many counters are added to vmstat just as debugging aid for now.

About IBS
---------
IBS can be programmed to provide data about instruction execution periodically. This is done by programming a desired sample count (number of ops) in a control register. When the programmed number of ops are dispatched, a micro-op gets tagged, various information about the tagged micro-op's execution is populated in IBS execution MSRs and an interrupt is raised. While IBS provides a lot of data for each sample, for the purpose of memory access profiling, we are interested in linear and physical address of the memory access that reached DRAM. Recent AMD processors provide further filtering where it is possible to limit the sampling to those ops that had an L3 miss which greately reduces the non-useful samples.

While IBS provides capability to sample instruction fetch and execution, only IBS execution sampling is used here to collect data about memory accesses that occur during the instruction execution.

More information about IBS is available in Sec 13.3 of AMD64 Architecture Programmer's Manual, Volume 2:System Programming which is present at:
https://bugzilla.kernel.org/attachment.cgi?id=288923

Information about MSRs used for programming IBS can be found in Sec 2.1.14.4 of PPR Vol 1 for AMD Family 19h Model 11h B1 which is currently present at:
https://www.amd.com/system/files/TechDocs/55901_0.25.zip

---
 arch/x86/events/amd/ibs.c        |  10 +
 arch/x86/include/asm/msr-index.h |  16 ++
 arch/x86/mm/Makefile             |   1 +
 arch/x86/mm/ibs.c                | 317 +++++++++++++++++++++++++++++++
 include/linux/pghot.h            |   8 +
 include/linux/vm_event_item.h    |  19 ++
 mm/Kconfig                       |  13 ++
 mm/vmstat.c                      |  19 ++
 8 files changed, 403 insertions(+)
 create mode 100644 arch/x86/mm/ibs.c
arch/x86/events/amd/ibs.c +10 -0
index aca89f23d2e0..dc544d084c17 100644
@@ -13,6 +13,7 @@
 #include <linux/ptrace.h>
 #include <linux/syscore_ops.h>
 #include <linux/sched/clock.h>
+#include <linux/pghot.h>
 
 #include <asm/apic.h>
 #include <asm/msr.h>
@@ -1760,6 +1761,15 @@ static __init int amd_ibs_init(void)
 {
 	u32 caps;
 
+	/*
+	 * TODO: Find a clean way to disable perf IBS so that IBS
+	 * can be used for memory access profiling.
+	 */
+	if (hwmem_access_profiler_inuse()) {
+		pr_info("IBS isn't available for perf use\n");
+		return 0;
+	}
+
 	caps = __get_ibs_caps();
 	if (!caps)
 		return -ENODEV;	/* ibs not supported by the cpu */
arch/x86/include/asm/msr-index.h +16 -0
index 3d0a0950d20a..3c5d69ec83a2 100644
@@ -784,6 +784,22 @@
 /* AMD Last Branch Record MSRs */
 #define MSR_AMD64_LBR_SELECT			0xc000010e
 
+/* AMD IBS MSR bits */
+#define MSR_AMD64_IBSOPDATA2_DATASRC			0x7
+#define MSR_AMD64_IBSOPDATA2_DATASRC_LCL_CACHE		0x1
+#define MSR_AMD64_IBSOPDATA2_DATASRC_PEER_CACHE_NEAR	0x2
+#define MSR_AMD64_IBSOPDATA2_DATASRC_DRAM		0x3
+#define MSR_AMD64_IBSOPDATA2_DATASRC_FAR_CCX_CACHE	0x5
+#define MSR_AMD64_IBSOPDATA2_DATASRC_EXT_MEM		0x8
+#define	MSR_AMD64_IBSOPDATA2_RMTNODE			0x10
+
+#define MSR_AMD64_IBSOPDATA3_LDOP		BIT_ULL(0)
+#define MSR_AMD64_IBSOPDATA3_STOP		BIT_ULL(1)
+#define MSR_AMD64_IBSOPDATA3_DCMISS		BIT_ULL(7)
+#define MSR_AMD64_IBSOPDATA3_LADDR_VALID	BIT_ULL(17)
+#define MSR_AMD64_IBSOPDATA3_PADDR_VALID	BIT_ULL(18)
+#define MSR_AMD64_IBSOPDATA3_L2MISS		BIT_ULL(20)
+
 /* Zen4 */
 #define MSR_ZEN4_BP_CFG                 0xc001102e
 #define MSR_ZEN4_BP_CFG_BP_SPEC_REDUCE_BIT 4
arch/x86/mm/Makefile +1 -0
index 5b9908f13dcf..361a456582e9 100644
@@ -57,3 +57,4 @@ obj-$(CONFIG_X86_MEM_ENCRYPT)	+= mem_encrypt.o
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_amd.o
 
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_boot.o
+obj-$(CONFIG_HWMEM_PROFILER)	+= ibs.o
arch/x86/mm/ibs.c new file +317 -0

317 lines not on this page · raw message

include/linux/pghot.h +8 -0
index d3d59b0c0cf6..20ea9767dbdd 100644
@@ -2,6 +2,14 @@
 #ifndef _LINUX_PGHOT_H
 #define _LINUX_PGHOT_H
 
+#include <linux/types.h>
+
+#ifdef CONFIG_HWMEM_PROFILER
+bool hwmem_access_profiler_inuse(void);
+#else
+static inline bool hwmem_access_profiler_inuse(void) { return false; }
+#endif
+
 /* Page hotness temperature sources */
 enum pghot_src {
 	PGHOT_HW_HINTS,
include/linux/vm_event_item.h +19 -0
index 5b8fd93b55fd..67efbca9051c 100644
@@ -193,6 +193,25 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		PGHOT_RECORD_HWHINTS,
 		PGHOT_RECORD_PGTSCANS,
 		PGHOT_RECORD_HINTFAULTS,
+#ifdef CONFIG_HWMEM_PROFILER
+		HWHINT_NR_EVENTS,
+		HWHINT_KERNEL,
+		HWHINT_KTHREAD,
+		HWHINT_NON_LOAD_STORES,
+		HWHINT_DC_L2_HITS,
+		HWHINT_LOCAL_L3L1L2,
+		HWHINT_LOCAL_PEER_CACHE_NEAR,
+		HWHINT_FAR_CACHE_HITS,
+		HWHINT_DRAM_ACCESSES,
+		HWHINT_CXL_ACCESSES,
+		HWHINT_REMOTE_NODE,
+		HWHINT_LADDR_INVALID,
+		HWHINT_KERNEL_ADDR,
+		HWHINT_PADDR_INVALID,
+		HWHINT_NON_LRU,
+		HWHINT_BUFFER_FULL,
+		HWHINT_USEFUL_SAMPLES,
+#endif /* CONFIG_HWMEM_PROFILER */
 #endif /* CONFIG_PGHOT */
 		NR_VM_EVENT_ITEMS
 };
mm/Kconfig +13 -0
index fde5aee3e16f..07b16aece877 100644
@@ -1489,6 +1489,19 @@ config PGHOT_PRECISE
 	  4 bytes per page against the default one byte per page. Preferable
 	  to enable this on systems with multiple nodes in toptier.
 
+config HWMEM_PROFILER
+	bool "HW based memory access profiling"
+	def_bool n
+	depends on PGHOT
+	depends on X86_64
+	help
+	  Some hardware platforms are capable of providing memory access
+	  information in direct and actionable manner. Instruction Based
+	  Sampling (IBS) present on AMD Zen CPUs in one such example.
+	  Memory accesses obtained via such HW based mechanisms are
+	  rolled up to PGHOT sub-system for further action like hot page
+	  promotion or NUMA Balancing
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/vmstat.c +19 -0
index f6f91b9dd887..62c47f44edf0 100644
@@ -1506,6 +1506,25 @@ const char * const vmstat_text[] = {
 	[I(PGHOT_RECORD_HWHINTS)]		= "pghot_recorded_hwhints",
 	[I(PGHOT_RECORD_PGTSCANS)]		= "pghot_recorded_pgtscans",
 	[I(PGHOT_RECORD_HINTFAULTS)]		= "pghot_recorded_hintfaults",
+#ifdef CONFIG_HWMEM_PROFILER
+	[I(HWHINT_NR_EVENTS)]			= "hwhint_nr_events",
+	[I(HWHINT_KERNEL)]			= "hwhint_kernel",
+	[I(HWHINT_KTHREAD)]			= "hwhint_kthread",
+	[I(HWHINT_NON_LOAD_STORES)]		= "hwhint_non_load_stores",
+	[I(HWHINT_DC_L2_HITS)]			= "hwhint_dc_l2_hits",
+	[I(HWHINT_LOCAL_L3L1L2)]		= "hwhint_local_l3l1l2",
+	[I(HWHINT_LOCAL_PEER_CACHE_NEAR)]	= "hwhint_local_peer_cache_near",
+	[I(HWHINT_FAR_CACHE_HITS)]		= "hwhint_far_cache_hits",
+	[I(HWHINT_DRAM_ACCESSES)]		= "hwhint_dram_accesses",
+	[I(HWHINT_CXL_ACCESSES)]		= "hwhint_cxl_accesses",
+	[I(HWHINT_REMOTE_NODE)]			= "hwhint_remote_node",
+	[I(HWHINT_LADDR_INVALID)]		= "hwhint_invalid_laddr",
+	[I(HWHINT_KERNEL_ADDR)]			= "hwhint_kernel_addr",
+	[I(HWHINT_PADDR_INVALID)]		= "hwhint_invalid_paddr",
+	[I(HWHINT_NON_LRU)]			= "hwhint_non_lru",
+	[I(HWHINT_BUFFER_FULL)]			= "hwhint_buffer_full",
+	[I(HWHINT_USEFUL_SAMPLES)]		= "hwhint_useful_samples",
+#endif /* CONFIG_HWMEM_PROFILER */
 #endif /* CONFIG_PGHOT */
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
Signature · 1 line
2.34.1
#12 lore raw

RFC PATCH v5 7/10 x86: ibs: Enable IBS profiling for memory accesses

In reply to Bharata B Rao

Enable IBS memory access data collection for user memory accesses by programming the required MSRs. The profiling is turned ON only for user mode execution and turned OFF for kernel mode execution. Profiling is explicitly disabled for NMI handler too.

TODOs:

- IBS sampling rate is kept fixed for now.
- Arch/vendor separation/isolation of the code needs relook.

---
 arch/x86/include/asm/entry-common.h |  3 +++
 arch/x86/include/asm/hardirq.h      |  2 ++
 arch/x86/mm/ibs.c                   | 32 +++++++++++++++++++++++++++++
 include/linux/pghot.h               |  4 ++++
 4 files changed, 41 insertions(+)
arch/x86/include/asm/entry-common.h +3 -0
index ce3eb6d5fdf9..0f381a63669e 100644
@@ -4,6 +4,7 @@
 
 #include <linux/randomize_kstack.h>
 #include <linux/user-return-notifier.h>
+#include <linux/pghot.h>
 
 #include <asm/nospec-branch.h>
 #include <asm/io_bitmap.h>
@@ -13,6 +14,7 @@
 /* Check that the stack and regs on entry from user mode are sane. */
 static __always_inline void arch_enter_from_user_mode(struct pt_regs *regs)
 {
+	hwmem_access_profiling_stop();
 	if (IS_ENABLED(CONFIG_DEBUG_ENTRY)) {
 		/*
 		 * Make sure that the entry code gave us a sensible EFLAGS
@@ -106,6 +108,7 @@ static inline void arch_exit_to_user_mode_prepare(struct pt_regs *regs,
 static __always_inline void arch_exit_to_user_mode(void)
 {
 	amd_clear_divider();
+	hwmem_access_profiling_start();
 }
 #define arch_exit_to_user_mode arch_exit_to_user_mode
 
arch/x86/include/asm/hardirq.h +2 -0
index 6b6d472baa0b..e80c305c17d1 100644
@@ -91,4 +91,6 @@ static __always_inline bool kvm_get_cpu_l1tf_flush_l1d(void)
 static __always_inline void kvm_set_cpu_l1tf_flush_l1d(void) { }
 #endif /* IS_ENABLED(CONFIG_KVM_INTEL) */
 
+#define arch_nmi_enter()	hwmem_access_profiling_stop()
+#define arch_nmi_exit()		hwmem_access_profiling_start()
 #endif /* _ASM_X86_HARDIRQ_H */
arch/x86/mm/ibs.c +32 -0
index 752f688375f9..d0d93f09432d 100644
@@ -16,6 +16,7 @@ static u64 ibs_config __read_mostly;
 static u32 ibs_caps;
 
 #define IBS_NR_SAMPLES	150
+#define IBS_SAMPLE_PERIOD      10000
 
 /*
  * Basic access info captured for each memory access.
@@ -43,6 +44,36 @@ struct ibs_sample_pcpu __percpu *ibs_s;
 static struct work_struct ibs_work;
 static struct irq_work ibs_irq_work;
 
+void hwmem_access_profiling_stop(void)
+{
+	u64 ops_ctl;
+
+	if (!hwmem_access_profiling)
+		return;
+
+	rdmsrl(MSR_AMD64_IBSOPCTL, ops_ctl);
+	wrmsrl(MSR_AMD64_IBSOPCTL, ops_ctl & ~IBS_OP_ENABLE);
+}
+
+void hwmem_access_profiling_start(void)
+{
+	u64 config = 0;
+	unsigned int period = IBS_SAMPLE_PERIOD;
+
+	if (!hwmem_access_profiling)
+		return;
+
+	/* Disable IBS for kernel thread */
+	if (!current->mm)
+		goto out;
+
+	config = (period >> 4) & IBS_OP_MAX_CNT;
+	config |= (period & IBS_OP_MAX_CNT_EXT_MASK);
+	config |= ibs_config;
+out:
+	wrmsrl(MSR_AMD64_IBSOPCTL, config);
+}
+
 bool hwmem_access_profiler_inuse(void)
 {
 	return hwmem_access_profiling;
@@ -310,6 +341,7 @@ static int __init ibs_access_profiling_init(void)
 			  x86_amd_ibs_access_profile_startup,
 			  x86_amd_ibs_access_profile_teardown);
 
+	hwmem_access_profiling = true;
 	pr_info("IBS setup for memory access profiling\n");
 	return 0;
 }
include/linux/pghot.h +4 -0
index 20ea9767dbdd..603791183102 100644
@@ -6,8 +6,12 @@
 
 #ifdef CONFIG_HWMEM_PROFILER
 bool hwmem_access_profiler_inuse(void);
+void hwmem_access_profiling_start(void);
+void hwmem_access_profiling_stop(void);
 #else
 static inline bool hwmem_access_profiler_inuse(void) { return false; }
+static inline void hwmem_access_profiling_start(void) {}
+static inline void hwmem_access_profiling_stop(void) {}
 #endif
 
 /* Page hotness temperature sources */
Signature · 1 line
2.34.1
#13 lore raw

RFC PATCH v5 8/10 mm: mglru: generalize page table walk

In reply to Bharata B Rao

From: Kinsey Ho <[email protected]>

Refactor the existing MGLRU page table walking logic to make it resumable.

Additionally, introduce two hooks into the MGLRU page table walk:
accessed callback and flush callback. The accessed callback is called for each accessed page detected via the scanned accessed bit. The flush callback is called when the accessed callback reports that a flush is required. This allows for processing pages in batches for efficiency.

With a generalised page table walk, introduce a new scan function which repeatedly scans on the same young generation and does not add a new young generation.

---
 include/linux/mmzone.h |   5 ++
 mm/internal.h          |   4 +
 mm/vmscan.c            | 181 +++++++++++++++++++++++++++++++----------
 3 files changed, 145 insertions(+), 45 deletions(-)
include/linux/mmzone.h +5 -0
index 49c374064fc2..26350a4951ff 100644
@@ -548,6 +548,8 @@ struct lru_gen_mm_walk {
 	unsigned long seq;
 	/* the next address within an mm to scan */
 	unsigned long next_addr;
+	/* called for each accessed pte/pmd */
+	bool (*accessed_cb)(unsigned long pfn);
 	/* to batch promoted pages */
 	int nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
 	/* to batch the mm stats */
@@ -555,6 +557,9 @@ struct lru_gen_mm_walk {
 	/* total batched items */
 	int batched;
 	int swappiness;
+	/* for the pmd under scanning */
+	int nr_young_pte;
+	int nr_total_pte;
 	bool force_scan;
 };
 
mm/internal.h +4 -0
index e430da900430..426db1ae286f 100644
@@ -538,6 +538,10 @@ extern unsigned long highest_memmap_pfn;
 bool folio_isolate_lru(struct folio *folio);
 void folio_putback_lru(struct folio *folio);
 extern void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason);
+void set_task_reclaim_state(struct task_struct *task,
+				   struct reclaim_state *rs);
+void lru_gen_scan_lruvec(struct lruvec *lruvec, unsigned long seq,
+			 bool (*accessed_cb)(unsigned long), void (*flush_cb)(void));
 #ifdef CONFIG_NUMA
 int user_proactive_reclaim(char *buf,
 			   struct mem_cgroup *memcg, pg_data_t *pgdat);
mm/vmscan.c +136 -45

361 lines not on this page · raw message

Signature · 1 line
2.34.1
#14 lore raw

RFC PATCH v5 9/10 mm: klruscand: use mglru scanning for page promotion

In reply to Bharata B Rao

From: Kinsey Ho <[email protected]>

Introduce a new kernel daemon, klruscand, that periodically invokes the MGLRU page table walk. It leverages the new callbacks to gather access information and forwards it to pghot sub-system for promotion decisions.

This benefits from reusing the existing MGLRU page table walk infrastructure, which is optimized with features such as hierarchical scanning and bloom filters to reduce CPU overhead.

As an additional optimization to be added in the future, we can tune the scan intervals for each memcg.

[Reduced the scan interval to 500ms, KLRUSCAND to default n in config]
---
 mm/Kconfig     |   8 ++++
 mm/Makefile    |   1 +
 mm/klruscand.c | 110 +++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 119 insertions(+)
 create mode 100644 mm/klruscand.c
mm/Kconfig +8 -0
index 07b16aece877..9e9eca8db8bf 100644
@@ -1502,6 +1502,14 @@ config HWMEM_PROFILER
 	  rolled up to PGHOT sub-system for further action like hot page
 	  promotion or NUMA Balancing
 
+config KLRUSCAND
+	bool "Kernel lower tier access scan daemon"
+	default n
+	depends on PGHOT && LRU_GEN_WALKS_MMU
+	help
+	  Scan for accesses from lower tiers by invoking MGLRU to perform
+	  page table walks.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +1 -0
index 89f999647752..c68df497a063 100644
@@ -153,3 +153,4 @@ obj-$(CONFIG_PGHOT) += pghot-precise.o
 else
 obj-$(CONFIG_PGHOT) += pghot-default.o
 endif
+obj-$(CONFIG_KLRUSCAND) += klruscand.o
mm/klruscand.c new file +110 -0

110 lines not on this page · raw message

Signature · 1 line
2.34.1
#15 lore raw

RFC PATCH v5 10/10 mm: pghot: Add folio_mark_accessed() as hotness source

In reply to Bharata B Rao

Unmapped page cache pages that end up in lower tiers don't get promoted easily. There were attempts to identify such pages and get them promoted as part of NUMA Balancing earlier [1]. The same idea is taken forward here by using folio_mark_accessed() as a source of hotness.

Lower tier accesses from folio_mark_accessed() are reported to pghot sub-system for hotness tracking and subsequent promotion.

TODO: Need a better naming for this hotness source. Need to better understand/evaluate the overhead of hotness info collection from this path.

[1] https://lore.kernel.org/linux-mm/[email protected]/

---
 Documentation/admin-guide/mm/pghot.txt | 7 ++++++-
 include/linux/pghot.h                  | 5 +++++
 include/linux/vm_event_item.h          | 1 +
 mm/pghot-tunables.c                    | 7 +++++++
 mm/pghot.c                             | 6 ++++++
 mm/swap.c                              | 8 ++++++++
 mm/vmstat.c                            | 1 +
 7 files changed, 34 insertions(+), 1 deletion(-)
Documentation/admin-guide/mm/pghot.txt +6 -1
index b329e692ef89..c8eb61064247 100644
@@ -23,9 +23,10 @@ Path: /sys/kernel/debug/pghot/
      - 0: Hardware hints (value 0x1)
      - 1: Page table scan (value 0x2)
      - 2: Hint faults (value 0x4)
+     - 3: folio_mark_accessed (value 0x8)
    - Default: 0 (disabled)
    - Example:
-     # echo 0x7 > /sys/kernel/debug/pghot/enabled_sources
+     # echo 0xf > /sys/kernel/debug/pghot/enabled_sources
      Enables all sources.
 
 2. **target_nid**
@@ -82,3 +83,7 @@ Path: /proc/vmstat
 4. **pghot_recorded_hintfaults**
    - Number of recorded accesses reported by NUMA Balancing based
      hotness source.
+
+5. **pghot_recorded_fma**
+   - Number of recorded accesses reported by folio_mark_accessed()
+     hotness source.
include/linux/pghot.h +5 -0
index 603791183102..8cf9dfb5365a 100644
@@ -19,6 +19,7 @@ enum pghot_src {
 	PGHOT_HW_HINTS,
 	PGHOT_PGTABLE_SCAN,
 	PGHOT_HINT_FAULT,
+	PGHOT_FMA,
 };
 
 #ifdef CONFIG_PGHOT
@@ -36,6 +37,7 @@ void pghot_debug_init(void);
 DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
 DECLARE_STATIC_KEY_FALSE(pghot_src_pgtscans);
 DECLARE_STATIC_KEY_FALSE(pghot_src_hintfaults);
+DECLARE_STATIC_KEY_FALSE(pghot_src_fma);
 
 /*
  * Bit positions to enable individual sources in pghot/records_enabled
@@ -45,6 +47,7 @@ enum pghot_src_enabled {
 	PGHOT_HWHINTS_BIT = 0,
 	PGHOT_PGTSCAN_BIT,
 	PGHOT_HINTFAULT_BIT,
+	PGHOT_FMA_BIT,
 	PGHOT_MAX_BIT
 };
 
@@ -52,6 +55,8 @@ enum pghot_src_enabled {
 #define PGHOT_PGTSCAN_ENABLED		BIT(PGHOT_PGTSCAN_BIT)
 #define PGHOT_HINTFAULT_ENABLED		BIT(PGHOT_HINTFAULT_BIT)
 #define PGHOT_SRC_ENABLED_MASK		GENMASK(PGHOT_MAX_BIT - 1, 0)
+#define PGHOT_FMA_ENABLED		BIT(PGHOT_FMA_BIT)
+#define PGHOT_SRC_ENABLED_MASK		GENMASK(PGHOT_MAX_BIT - 1, 0)
 
 #define PGHOT_DEFAULT_FREQ_THRESHOLD	2
 
include/linux/vm_event_item.h +1 -0
index 67efbca9051c..ac1f28646b9c 100644
@@ -193,6 +193,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		PGHOT_RECORD_HWHINTS,
 		PGHOT_RECORD_PGTSCANS,
 		PGHOT_RECORD_HINTFAULTS,
+		PGHOT_RECORD_FMA,
 #ifdef CONFIG_HWMEM_PROFILER
 		HWHINT_NR_EVENTS,
 		HWHINT_KERNEL,
mm/pghot-tunables.c +7 -0
index 79afbcb1e4f0..11c7f742a1be 100644
@@ -124,6 +124,13 @@ static void pghot_src_enabled_update(unsigned int enabled)
 		else
 			static_branch_disable(&pghot_src_hintfaults);
 	}
+
+	if (changed & PGHOT_FMA_ENABLED) {
+		if (enabled & PGHOT_FMA_ENABLED)
+			static_branch_enable(&pghot_src_fma);
+		else
+			static_branch_disable(&pghot_src_fma);
+	}
 }
 
 static ssize_t pghot_src_enabled_write(struct file *filp, const char __user *ubuf,
mm/pghot.c +6 -0
index 6fc76c1eaff8..537f4af816ff 100644
@@ -43,6 +43,7 @@ static unsigned int sysctl_pghot_promote_rate_limit = 65536;
 DEFINE_STATIC_KEY_FALSE(pghot_src_hwhints);
 DEFINE_STATIC_KEY_FALSE(pghot_src_pgtscans);
 DEFINE_STATIC_KEY_FALSE(pghot_src_hintfaults);
+DEFINE_STATIC_KEY_FALSE(pghot_src_fma);
 
 #ifdef CONFIG_SYSCTL
 static const struct ctl_table pghot_sysctls[] = {
@@ -113,6 +114,11 @@ int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
 			return -EINVAL;
 		count_vm_event(PGHOT_RECORD_HINTFAULTS);
 		break;
+	case PGHOT_FMA:
+		if (!static_branch_likely(&pghot_src_fma))
+			return -EINVAL;
+		count_vm_event(PGHOT_RECORD_FMA);
+		break;
 	default:
 		return -EINVAL;
 	}
mm/swap.c +8 -0
index 2260dcd2775e..31a654b19844 100644
@@ -37,6 +37,8 @@
 #include <linux/page_idle.h>
 #include <linux/local_lock.h>
 #include <linux/buffer_head.h>
+#include <linux/pghot.h>
+#include <linux/memory-tiers.h>
 
 #include "internal.h"
 
@@ -454,8 +456,14 @@ static bool lru_gen_clear_refs(struct folio *folio)
  */
 void folio_mark_accessed(struct folio *folio)
 {
+	unsigned long pfn = folio_pfn(folio);
+
 	if (folio_test_dropbehind(folio))
 		return;
+
+	if (!node_is_toptier(pfn_to_nid(pfn)))
+		pghot_record_access(pfn, NUMA_NO_NODE, PGHOT_FMA, jiffies);
+
 	if (lru_gen_enabled()) {
 		lru_gen_inc_refs(folio);
 		return;
mm/vmstat.c +1 -0
index 62c47f44edf0..c4d90baf440b 100644
@@ -1506,6 +1506,7 @@ const char * const vmstat_text[] = {
 	[I(PGHOT_RECORD_HWHINTS)]		= "pghot_recorded_hwhints",
 	[I(PGHOT_RECORD_PGTSCANS)]		= "pghot_recorded_pgtscans",
 	[I(PGHOT_RECORD_HINTFAULTS)]		= "pghot_recorded_hintfaults",
+	[I(PGHOT_RECORD_FMA)]			= "pghot_recorded_fma",
 #ifdef CONFIG_HWMEM_PROFILER
 	[I(HWHINT_NR_EVENTS)]			= "hwhint_nr_events",
 	[I(HWHINT_KERNEL)]			= "hwhint_kernel",
Signature · 1 line
2.34.1
#16 lore raw
In reply to Bharata B Rao
Quoted 4 lines · Bharata B Rao

Results
=======
TODO: Will post benchmark nubmers as reply to this patchset soon.

Here is the first set of results from a microbenchmark:

Test system details
-------------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0-95,192-287
node 0 size: 128460 MB
node 1 cpus: 96-191,288-383
node 1 size: 128893 MB
node 2 cpus:
node 2 size: 257993 MB
node distances:
node   0   1   2
  0:  10  32  50
  1:  32  10  60
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the patched case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the patched case.
pgtscan - Klruscand (MGLRU based PTE A bit scanning) source hwhints - IBS as source

Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

==============================================================
Scenario 1 - Enough memory in toptier and hence only promotion
==============================================================
Multi-threaded application with 64 threads that access memory at 4K granularity repetitively and randomly. The number of accesses per thread and the randomness pattern for each thread are fixed beforehand. The accesses are divided into stores and loads in the ratio of 50:50.

Benchmark threads run on Node 0, while memory is initially provisioned on CXL node 2 before the accesses start.

Repetitive accesses results in lowertier pages becoming hot and kmigrated detecting and migrating them. The benchmark score is the time taken to finish the accesses in microseconds. The sooner it finishes the better it is. All the numbers shown below are average of 3 runs.

Default mode - Time taken (microseconds, lower is better)
---------------------------------------------------------
Source Base Pghot
---------------------------------------------------------

NUMAB0          117,069,417     115,802,776
NUMAB2          102,918,471     103,378,828
pgtscan         NA              110,203,286
hwhints         NA              92,880,388

---------------------------------------------------------

Default mode - Pages migrated (pgpromote_success)
---------------------------------------------------------
Source Base Pghot
---------------------------------------------------------

NUMAB0          0               0
NUMAB2          2097147         2097131
pgtscan         NA              2097130
hwhints         NA              1706556

---------------------------------------------------------

Precision mode - Time taken (microseconds, lower is better)
-----------------------------------------------------------
Source Base Pghot
-----------------------------------------------------------

NUMAB0          117,069,417     115,078,527
NUMAB2          102,918,471     101,742,985
pgtscan         NA              110,024,513     NA
hwhints         NA              101,163,603     NA

-----------------------------------------------------------

Precision mode - Pages migrated (pgpromote_success)
---------------------------------------------------
Source Base Pghot
---------------------------------------------------

NUMAB0          0               0
NUMAB2          2097147         2097144
pgtscan         NA              2097129
hwhints         NA              1144304

---------------------------------------------------

- The NUMAB2 benchmark numbers and pgpromote_success numbers more or less match in base and patched case.
- Though the pgtscan case promotes all possible pages, the benchmark number suffers. This source needs tuning.
- Hwhints case is able to provide benchmark numbers similar to base NUMAB2 even with less number of migrations.
- With both default and precision modes of pghot the benchmark behaves more or less similarly.

==============================================================
Scenario 2 - Toptier memory overcommited, promotion + demotion
==============================================================
Single threaded application that allocates memory on both DRAM and CXL nodes using mmap(MAP_POPULATE). Every 1G region of allocated memory on CXL node is accessed at 4K granularity randomly and repetitively to build up the notion of hotness in the 1GB region that is under access. This should drive promotion. For promotion to work successfully, the DRAM memory that has been provisioned (and not being accessed) should be demoted first. There is enough free memory in the CXL node to for demotions.

In summary, this benchmark creates a memory pressure on DRAM node and does CXL memory accesses to drive both demotion and promotion.

The number of accesses are fixed and hence, the quicker the accessed pages get promoted to DRAM, the sooner the benchmark is expected to finish. All the numbers shown below are average of 3 runs.

DRAM-node                       = 1
CXL-node                        = 2
Initial DRAM alloc ratio        = 75%
Allocation-size                 = 171798691840
Initial DRAM Alloc-size         = 128849018880
Initial CXL Alloc-size          = 42949672960
Hot-region-size                 = 1073741824
Nr-regions                      = 160
Nr-regions DRAM                 = 120 (provisioned but not accessed)
Nr-hot-regions CXL              = 40
Access pattern                  = random
Access granularity              = 4096
Delay b/n accesses              = 0
Load/store ratio                = 50l50s
THP used                        = no
Nr accesses                     = 42949672960
Nr repetitions                  = 1024

Default mode - Time taken (microseconds, lower is better)
------------------------------------------------------
Source Base Pghot
------------------------------------------------------

NUMAB0          63,809,267      60,794,786
NUMAB2          67,541,601      62,376,991
pgtscan         NA              67,902,126
hwhints         NA              59,872,525

------------------------------------------------------

Default mode - Pages migrated (pgpromote_success)
-------------------------------------------------
Source Base Pghot
-------------------------------------------------

NUMAB0          0               0
NUMAB2          179635          932693  (High R2R variation in base)
pgtscan         NA              27487
hwhints         NA              274

---------------------------------------

Precision mode - Time taken (microseconds, lower is better)
------------------------------------------------------
Source Base Pghot
------------------------------------------------------

NUMAB0          63,809,267      64,553,914
NUMAB2          67,541,601      62,148,082
pgtscan         NA              65,073,396
hwhints         NA              59,958,655

------------------------------------------------------

Precision mode - Pages migrated (pgpromote_success)
---------------------------------------------------
Source Base Pghot
---------------------------------------------------

NUMAB0          0               0
NUMAB2          179635          988360  (High R2R variaion in base)
pgtscan         NA              21418   (High R2R variation in patched)
hwhints         NA              174     (High R2R variation in patched)

---------------------------------------------------

- The base case itself doesn't show any improvement in benchmark numbers due to hot page promotion. The same pattern is seen in pghot case with all the sources except hwhints. The benchmark itself may need tuning so that promotion helps.
- There is a high run to run variation in the number of pages promoted in base case.
- Most promotion attempts in base case fail because the NUMA hint fault latency is found to exceed the threshold value (default threshold is 1000ms) in majority of the promotion attempts.
- Unlike base NUMAB2 where the hint fault latency is the difference between the PTE update time (during scanning) and the access time (hint fault), pghot uses a single latency threshold (4000ms in pghot-default and 5000ms in pghot-precise) for two purposes.
1. If the time difference between successive accesses are within the threshold, the page is marked as hot.
2. Later when kmigrated picks up the page for migration, it will migrate only if the difference between the current time and the time when the page was marked hot is with the threshold.
Because of the above difference in behaviour, more number of pages get qualified for promotion compared to base NUMAB2.

#17 lore raw
In reply to Bharata B Rao
Quoted 3 lines · Bharata B Rao

Results
=======
TODO: Will post benchmark nubmers as reply to this patchset soon.

Numbers from redis-memtier benchmark:

Test system details
-------------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0-95,192-287
node 0 size: 128460 MB
node 1 cpus: 96-191,288-383
node 1 size: 128893 MB
node 2 cpus:
node 2 size: 257993 MB
node distances:
node   0   1   2
  0:  10  32  50
  1:  32  10  60
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the patched case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the patched case.

Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

==============================================================
Scenario 1 - Enough memory in toptier and hence only promotion
==============================================================
In the setup phase, 64GB database is provisioned and explicitly moved to Node 2 by migrating redis-server's memory to Node 2. Memtier is run on Node 1.

Parallel distribution, 50% of the keys accessed, each 4 times.

16        Threads
100       Connections per thread
77808     Requests per client

==================================================================================================
Type Ops/sec Avg. Latency p50 Latency p99 Latency p99.9
Latency KB/sec
--------------------------------------------------------------------------------------------------
Base, NUMAB0
Totals 225827.75 226.49746 225.27900 425.98300
454.65500 513106.09
--------------------------------------------------------------------------------------------------
Base, NUMAB2
Totals 254869.29 205.61759 216.06300 399.35900
454.65500 579091.74
--------------------------------------------------------------------------------------------------
pghot-default, NUMAB2
Totals 264229.35 202.81411 215.03900 393.21500
446.46300 600358.86
--------------------------------------------------------------------------------------------------
pghot-precise, NUMAB2
Totals 261136.17 203.32692 215.03900 391.16700
446.46300 593330.81
==================================================================================================

pgpromote_success
==================================

Base, NUMAB0            0
Base, NUMAB2            10,435,178
pghot-default, NUMAB2   10,435,031
pghot-precise, NUMAB2   10,435,245

==================================

- There is a clear benefit of hot page promotion seen. Both base and pghot show similar benefits.
- The number of pages promoted in both cases are more or less same.

==============================================================
Scenario 2 - Toptier memory overcommited, promotion + demotion
==============================================================
In the setup phase, 192GB database is provisioned. The database occupies Node 1 entirely(~128GB) and spills over to Node 2 (~64GB). Memtier is run on Node 1.

Parallel distribution, 50% of the keys accessed, each 4 times.

16        Threads
100       Connections per thread
233424    Requests per client

==================================================================================================
Type Ops/sec Avg. Latency p50 Latency p99 Latency p99.9
Latency KB/sec
--------------------------------------------------------------------------------------------------
Base, NUMAB0
Totals 246474.55 211.90623 192.51100 370.68700
448.51100 560235.63
--------------------------------------------------------------------------------------------------
Base, NUMAB2
Totals 232790.88 221.18604 214.01500 419.83900
509.95100 529132.72
--------------------------------------------------------------------------------------------------
pghot-default, NUMAB2
Totals 241615.60 216.12761 210.94300 391.16700
475.13500 549191.27
--------------------------------------------------------------------------------------------------
pghot-precise, NUMAB2
Totals 238557.37 217.57630 207.87100 395.26300
471.03900 542239.92
==================================================================================================
pgpromote_success pgdemote_kswapd
===============================================================

Base, NUMAB0            0                       832,494
Base, NUMAB2            352,075                 720,409
pghot-default, NUMAB2   25,865,321              26,154,984
pghot-precise, NUMAB2   25,525,429              25,838,095

===============================================================

- No clear benefit is seen with hot page promotion both in base and pghot case.
- Most promotion attempts in base case fail because the NUMA hint fault latency is found to exceed the threshold value (default threshold of 1000ms) in majority of the promotion attempts.
- Unlike base NUMAB2 where the hint fault latency is the difference between the PTE update time (during scanning) and the access time (hint fault), pghot uses a single latency threshold (4000ms in pghot-default and 5000ms in pghot-precise) for two purposes.
1. If the time difference between successive accesses are within the threshold, the page is marked as hot.
2. Later when kmigrated picks up the page for migration, it will migrate only if the difference between the current time and the time when the page was marked hot is with the threshold.
Because of the above difference in behaviour, more number of pages get qualified for promotion compared to base NUMAB2.

#18 lore raw
In reply to Bharata B Rao
Quoted 4 lines · Bharata B Rao

Results
=======
TODO: Will post benchmark nubmers as reply to this patchset soon.

Here are Graph500 numbers for the hint fault source:

Test system details
-------------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0-95,192-287
node 0 size: 128460 MB
node 1 cpus: 96-191,288-383
node 1 size: 128893 MB
node 2 cpus:
node 2 size: 257993 MB
node distances:
node   0   1   2
  0:  10  32  50
  1:  32  10  60
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the pghot case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the pghot case.

Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

Graph500 details
----------------
Command: mpirun -n 128 --bind-to core --map-by core graph500/src/graph500_reference_bfs 28 16

After the graph creation, the processes are stopped and data is migrated to CXL node 2 before continuing so that BFS phase starts accessing lower tier memory.

Total memory usage is slightly over 100GB and will fit within Node 0 and 1. Hence there is no memory pressure to induce demotions.

=====================================================================================
Base Base pghot-default
pghot-precise
NUMAB0 NUMAB2 NUMAB2 NUMAB2
=====================================================================================

harmonic_mean_TEPS      5.10676e+08     7.56804e+08     5.92473e+08     7.47091e+08
mean_time               8.41027         5.67508         7.24915         5.74886
median_TEPS             5.11535e+08     7.24252e+08     5.63155e+08     7.71638e+08
max_TEPS                5.1785e+08      1.06051e+09     7.88018e+08     1.0504e+09
pgpromote_success       0               13557718        13737730        13734469
numa_pte_updates        0               26491591        26848847        26726856
numa_hint_faults        0               13558077        13882743        13798024

=====================================================================================

- The base case shows a good improvement with NUMAB2(48%) in harmonic_mean_TEPS.
- The same improvement gets maintained with pghot-precise too (46%).
- pghot-default mode doesn't show benefit even when achieving similar page promotion numbers. This mode doesn't track accessing NID and by default promotes to NID=0 which probably isn't all that beneficial as processes are running on both Node 0 and Node 1.

reviewer
#19 lore raw
Quoted 10 lines · Bharata B Rao

=====================================================================================
Base Base pghot-default
pghot-precise
NUMAB0 NUMAB2 NUMAB2 NUMAB2
=====================================================================================

harmonic_mean_TEPS      5.10676e+08     7.56804e+08     5.92473e+08     7.47091e+08
mean_time               8.41027         5.67508         7.24915         5.74886
median_TEPS             5.11535e+08     7.24252e+08     5.63155e+08     7.71638e+08
max_TEPS                5.1785e+08      1.06051e+09     7.88018e+08     1.0504e+09
Quoted 5 lines
pgpromote_success       0               13557718        13737730        13734469
numa_pte_updates        0               26491591        26848847        26726856
numa_hint_faults        0               13558077        13882743        13798024

=====================================================================================

Can you contextualize TEPS? Higher better? Higher worse? etc. Unfamiliar with this benchmark.

~Gregory

#20 lore raw
Quoted 11 lines · Gregory Price
Quoted 10 lines · Bharata B Rao

=====================================================================================
Base Base pghot-default
pghot-precise
NUMAB0 NUMAB2 NUMAB2 NUMAB2
=====================================================================================

harmonic_mean_TEPS      5.10676e+08     7.56804e+08     5.92473e+08     7.47091e+08
mean_time               8.41027         5.67508         7.24915         5.74886
median_TEPS             5.11535e+08     7.24252e+08     5.63155e+08     7.71638e+08
max_TEPS                5.1785e+08      1.06051e+09     7.88018e+08     1.0504e+09
Quoted 7 lines
Quoted 5 lines
pgpromote_success       0               13557718        13737730        13734469
numa_pte_updates        0               26491591        26848847        26726856
numa_hint_faults        0               13558077        13882743        13798024

=====================================================================================

Can you contextualize TEPS? Higher better? Higher worse? etc.

In the Graph500 benchmark, higher TEPS (Traversed Edges Per Second) values are better.

Regards,
Bharata.

reviewer
#21 lore raw
In reply to Bharata B Rao
Quoted 5 lines · Bharata B Rao

- pghot-default mode doesn't show benefit even when achieving similar page promotion numbers. This mode doesn't track accessing NID and by default promotes to NID=0 which probably isn't all that beneficial as processes are running on both Node 0 and Node 1.

Lacking access-nid data, maybe it's better to select a random (or round-robin) node in the upper tier? That would at least approach 1/N accuracy in promotion for most access patterns.

~Gregory

#22 lore raw
In reply to Bharata B Rao
Quoted 33 lines · Bharata B Rao
Quoted 4 lines · Bharata B Rao

Results
=======
TODO: Will post benchmark nubmers as reply to this patchset soon.

Here are Graph500 numbers for the hint fault source:

Test system details
-------------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0-95,192-287
node 0 size: 128460 MB
node 1 cpus: 96-191,288-383
node 1 size: 128893 MB
node 2 cpus:
node 2 size: 257993 MB
node distances:
node   0   1   2
  0:  10  32  50
  1:  32  10  60
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the pghot case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the pghot case.

Quoted 4 lines

Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

These numbers are from scenario where demotion is present:

=============================================
Over-committed scenario, promotion + demotion
=============================================
Command: mpirun -n 128 --bind-to core --map-by core /home/bharata/benchmarks/graph500/src/graph500_reference_bfs 30 16

The scale factor of 30 results in around 400GB of memory being provisioned resulting in the data spilling over to CXL node. No explicit migration of data is done in this case unlike the previous case.

=====================================================================================
Base Base pghot-default
pghot-precise
NUMAB0 NUMAB2 NUMAB2 NUMAB2
=====================================================================================

harmonic_mean_TEPS      9.28713e+08     7.90431e+08     7.32193e+08     7.81051e+08
mean_time               18.4984         21.7346         23.4634         21.9956
median_TEPS             9.25707e+08     7.86684e+08     7.27053e+08     7.82823e+08
max_TEPS                9.57632e+08     8.4758e+08      8.22172e+08     7.9889e+08
pgpromote_success       0               22846743        22807167        25994988
pgpromote_candidate     0               24628924        29436044        27029173
pgpromote_candidate_nrl 0               140921          220             38387
pgdemote_kswapd         0               41523110        45121134        50042594
numa_pte_updates        0               121904763       71503891        68779424
numa_hint_faults        0               81708126        29583391        27176332

=====================================================================================

- In the base case, the benchmark suffers when promotion and demotion are enabled (NUMAB2 case).
- Same behaviour is seen with both modes of pghot.
- Though the overall benchmark numbers remain more or less same with base and pghot NUMAB2 cases, the number of pte updates and hint faults are seen to spike up during some runs. Yet to understand the exact reason for this.

reviewer
#23 lore raw
In reply to Bharata B Rao
Quoted 7 lines · Bharata B Rao

Hi,

This is v5 of pghot, a hot-page tracking and promotion subsystem. The major change in v5 is reducing the default hotness record size to 1 byte per PFN and adding an optional precision mode (CONFIG_PGHOT_PRECISE) that uses 4 bytes per PFN.

In the future can you add a

base-commit:

for the series? Make's it easier to automate pulling it in for testing and backports etc.

~Gregory

#24 lore raw
Quoted 9 lines · Gregory Price
Quoted 7 lines · Bharata B Rao

Hi,

This is v5 of pghot, a hot-page tracking and promotion subsystem. The major change in v5 is reducing the default hotness record size to 1 byte per PFN and adding an optional precision mode (CONFIG_PGHOT_PRECISE) that uses 4 bytes per PFN.

Quoted 6 lines

In the future can you add a

base-commit:

for the series? Make's it easier to automate pulling it in for testing and backports etc.

Good suggestion, will do thanks.

BTW this series applies on f0b9d8eb98df.
Latest github branch:
https://github.com/AMDESE/linux-mm/tree/bharata/pghot-rfcv6-pre

Regards,
Bharata.

#25 lore raw
In reply to Bharata B Rao
Quoted 5 lines · Bharata B Rao

Results
=======
TODO: Will post benchmark nubmers as reply to this patchset soon.

Here are some numbers from NAS Parallel Benchmark (NPB) with BT application:

Test system details
-------------------
3 node AMD Zen5 system with 2 regular NUMA nodes (0, 1) and a CXL node (2)

$ numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0-95,192-287
node 0 size: 128460 MB
node 1 cpus: 96-191,288-383
node 1 size: 128893 MB
node 2 cpus:
node 2 size: 257993 MB
node distances:
node   0   1   2
  0:  10  32  50
  1:  32  10  60
  2:  255  255  10

Hotness sources
---------------
NUMAB0 - Without NUMA Balancing in base case and with no source enabled
in the pghot case. No migrations occur.
NUMAB2 - Existing hot page promotion for the base case and
use of hint faults as source in the pghot case. Both promotion and demotion are enabled in this case.

Pghot by default promotes after two accesses but for NUMAB2 source, promotion is done after one access to match the base behaviour. (/sys/kernel/debug/pghot/freq_threshold=1)

NAS-BT details
--------------
Command: mpirun -np 16 /usr/bin/numactl --cpunodebind=0,1 NPB3.4.4/NPB3.4-MPI/bin/bt.F.x

While class D uses around 24G of memory (which is too less to show the benefit of promition), class E results in around 368G of memory which overflows my toptier. Hence I wanted something in between these classes. So I have modified class F to the problem size of 768 which results in around 160GB of memory.

After the memory consumption stabilizes, all the rank PIDs are paused and their memory is moved to CXL node using migratepages command. This simulates the situation of memory residing on lower tier node and access by BT processes leading to promotion.

Time in seconds - Lower is better
Mop/s total - Higher is better
=====================================================================================
Base Base pghot-default
pghot-precise
NUMAB0 NUMAB2 NUMAB2 NUMAB2
=====================================================================================

Time in seconds         7349.86         4422.50         6219.71         4113.56
Mop/s total             53247.66        88493.630       62923.030       95139.810
pgpromote_success       0               42181834        248503390       41955718
pgpromote_candidate     0               0               577086192       0
pgpromote_candidate_nrl 0               42181834        29410329        41956171
pgdemote_kswapd         0               0               216489010       0
numa_pte_updates        0               42252749        607470975       42037882
numa_hint_faults        0               42183772        606540729       41968150

=====================================================================================

- In the base case, the benchmark numbers improve significantly due to hot page promotion.
- Though the benchmark runs for hundreds of minutes, the pages get promoted within the first few mins.
- pghot-precise is able to match the base case numbers.
- The benchmark suffers in pghot-default case due to promotion being limited to the default NID (0) only. This leads to excessive PTE updates, hint faults, demotion and promotion churn.

reviewer
#26 lore raw
Quoted 19 lines · Bharata B Rao

Time in seconds - Lower is better
Mop/s total - Higher is better
=====================================================================================
Base Base pghot-default
pghot-precise
NUMAB0 NUMAB2 NUMAB2 NUMAB2
=====================================================================================

Time in seconds         7349.86         4422.50         6219.71         4113.56
Mop/s total             53247.66        88493.630       62923.030       95139.810
pgpromote_success       0               42181834        248503390       41955718
pgpromote_candidate     0               0               577086192       0
pgpromote_candidate_nrl 0               42181834        29410329        41956171
pgdemote_kswapd         0               0               216489010       0
numa_pte_updates        0               42252749        607470975       42037882
numa_hint_faults        0               42183772        606540729       41968150

=====================================================================================

Quoted 8 lines

- In the base case, the benchmark numbers improve significantly due to hot page promotion.
- Though the benchmark runs for hundreds of minutes, the pages get promoted within the first few mins.
- pghot-precise is able to match the base case numbers.
- The benchmark suffers in pghot-default case due to promotion being limited to the default NID (0) only. This leads to excessive PTE updates, hint faults, demotion and promotion churn.

Wow, this really seems to justify the extra memory usage.

Is it possible for you to change pghot-default to move the page to a random (or round-robin) node on the top tier instead of NID(0) by default?

At least then pghot-default would be correct 1/N % of the time (in theory). I'd be curious to see how close it gets to NUMAB2 with that.

~Gregory

#27 lore raw
Quoted 29 lines · Gregory Price
Quoted 19 lines · Bharata B Rao

Time in seconds - Lower is better
Mop/s total - Higher is better
=====================================================================================
Base Base pghot-default
pghot-precise
NUMAB0 NUMAB2 NUMAB2 NUMAB2
=====================================================================================

Time in seconds         7349.86         4422.50         6219.71         4113.56
Mop/s total             53247.66        88493.630       62923.030       95139.810
pgpromote_success       0               42181834        248503390       41955718
pgpromote_candidate     0               0               577086192       0
pgpromote_candidate_nrl 0               42181834        29410329        41956171
pgdemote_kswapd         0               0               216489010       0
numa_pte_updates        0               42252749        607470975       42037882
numa_hint_faults        0               42183772        606540729       41968150

=====================================================================================

Quoted 8 lines

- In the base case, the benchmark numbers improve significantly due to hot page promotion.
- Though the benchmark runs for hundreds of minutes, the pages get promoted within the first few mins.
- pghot-precise is able to match the base case numbers.
- The benchmark suffers in pghot-default case due to promotion being limited to the default NID (0) only. This leads to excessive PTE updates, hint faults, demotion and promotion churn.

Quoted 7 lines

Wow, this really seems to justify the extra memory usage.

Is it possible for you to change pghot-default to move the page to a random (or round-robin) node on the top tier instead of NID(0) by default?

At least then pghot-default would be correct 1/N % of the time (in theory). I'd be curious to see how close it gets to NUMAB2 with that.

For pghot-default, with target_nid alternating between the available toptier nodes 0 and 1, the numbers catch up with pghot-precise and base NUMAB2 case as seen below:
================================
Time in seconds 4337.98
Mop/s total 90217.86

pgpromote_success 42170085
pgpromote_candidate 0
pgpromote_candidate_nrl 42171963

pgdemote_kswapd         0
numa_pte_updates        42338538
numa_hint_faults        42185662

================================

Regards,
Bharata.

reviewer
#28 lore raw
Quoted 25 lines · Bharata B Rao
Quoted 10 lines
Quoted 10 lines

Time in seconds - Lower is better
Mop/s total - Higher is better
=====================================================================================
Base Base pghot-default
pghot-precise
NUMAB0 NUMAB2 NUMAB2 NUMAB2
=====================================================================================

Time in seconds         7349.86         4422.50         6219.71         4113.56
Mop/s total             53247.66        88493.630       62923.030       95139.810
Quoted 7 lines
Quoted 7 lines
pgpromote_success       0               42181834        248503390       41955718
pgpromote_candidate     0               0               577086192       0
pgpromote_candidate_nrl 0               42181834        29410329        41956171
pgdemote_kswapd         0               0               216489010       0
numa_pte_updates        0               42252749        607470975       42037882
numa_hint_faults        0               42183772        606540729       41968150

=====================================================================================

For pghot-default, with target_nid alternating between the available toptier nodes 0 and 1, the numbers catch up with pghot-precise and base NUMAB2 case as seen below:
================================
Time in seconds 4337.98
Mop/s total 90217.86

Quoted 8 lines

pgpromote_success 42170085
pgpromote_candidate 0
pgpromote_candidate_nrl 42171963

pgdemote_kswapd         0
numa_pte_updates        42338538
numa_hint_faults        42185662

================================

Fascinating! Thank you for the quick follow up.

I wonder if this was a lucky run, it almost seems *too* perfect.

~Gregory

#29 lore raw
Quoted 17 lines · Gregory Price
Quoted 8 lines

For pghot-default, with target_nid alternating between the available toptier nodes 0 and 1, the numbers catch up with pghot-precise and base NUMAB2 case as seen below:
================================
Time in seconds 4337.98
Mop/s total 90217.86

Quoted 8 lines

pgpromote_success 42170085
pgpromote_candidate 0
pgpromote_candidate_nrl 42171963

pgdemote_kswapd         0
numa_pte_updates        42338538
numa_hint_faults        42185662

================================

Quoted 3 lines

Fascinating! Thank you for the quick follow up.

I wonder if this was a lucky run, it almost seems *too* perfect.

It consistently performs that way. Here are the numbers from another run:

================================
Time in seconds 4329.22
Mop/s total 90400.27

pgpromote_success 41967282
pgpromote_candidate 0
pgpromote_candidate_nrl 41968339

pgdemote_kswapd         0
numa_pte_updates        42253854
numa_hint_faults        42019449

================================

grep -E "pgpromote|pgdemote" /sys/devices/system/node/node0/vmstat pgpromote_success 20996597
pgpromote_candidate 0
pgpromote_candidate_nrl 41968339 (*)
pgdemote_kswapd 0
pgdemote_direct 0
pgdemote_khugepaged 0
pgdemote_proactive 0

grep -E "pgpromote|pgdemote" /sys/devices/system/node/node1/vmstat pgpromote_success 20970685
pgpromote_candidate 0
pgpromote_candidate_nrl 0
pgdemote_kswapd 0
pgdemote_direct 0
pgdemote_khugepaged 0
pgdemote_proactive 0

(*) The round-robin b/n nodes 0 and 1 happens after this metric is attributed to the original default target_nid. Hence nrl metric gets populated for node 0 only.