Skip to contentSkip to conversation

mm: Hot page tracking and promotion infrastructure

PATCH In review Bharata B Rao 12 participants 59 messages Jul 28 – Oct 29

Subsystems (18)
lore
Share & tools
lore
#0 lore raw

v8 cover mm: Hot page tracking and promotion infrastructure

Hi,

This patchset introduces pghot, a subsystem for hot page tracking and promotion. It collects memory access information from multiple sources, classifies hot pages resident in lower-tier memory, and promotes them to faster tiers via a per-lower-tier-node kernel thread (kmigrated).

The main changes in v8 are a rework of the per-section hotness map to be RCU-protected, a move of the pghot tunables from debugfs to sysctl, and a rework of the AMD IBS Memory Profiler driver that now uses a dedicated system interrupt vector instead of an NMI handler. The IBS driver is also split into core infrastructure and a separate patch that adds the runtime controls and arms the profiler. Also there are a few fixes to the hint faults source.

Goals of this patchset:

- Unify hot page detection from multiple sources like hint faults, page table scans, hardware hints (AMD IBS).
- Decouple detection from migration.
- Centralize promotion logic via per-lower-tier-node kmigrated kernel thread.
- Move promotion rate-limiting and related logic used by numa_balancing=2 (NUMAB2, the current NUMA balancing-based promotion) from the scheduler to pghot for broader reuse.

Currently, multiple kernel subsystems detect page accesses independently. This patchset consolidates accesses from these mechanisms by providing:

- A common API for reporting page accesses.
- Shared infrastructure for tracking hotness at PFN granularity.
- Per-lower-tier-node kernel threads for promoting pages.

Performance summary
===================
All results are on 3-node tiered systems (DRAM top tier + CPU-less CXL lower tier). Speedups below are normalized to the base kernel with no tiering (NUMAB=0). Columns compare mainline hint-fault tiering (base NUMAB=2) against pghot in two modes: hint-fault driven (pghot-hf, NUMAB=2) and HW-hint driven via the AMD IBS memory profiler (pghot-hw, NUMAB=0, no NUMA scanning). Single run per config (abench is avg of 3).

Benchmark Metric (higher=better) base-NUMAB2 pghot-hf pghot-hw
-------------------------------------------------------------------------

NAS BT (MPI)   Mop/s total                  2.26x     2.26x     2.13x
Graph500 BFS   harmonic-mean TEPS           2.64x     2.49x     2.68x
llama.cpp      decode tok/s (tg128)         1.09x     1.41x     1.21x
Redis+memtier  ops/sec                      1.05x     1.05x     1.00x
Microbench     completion time (1/t)        2.41x     2.40x     2.35x

-------------------------------------------------------------------------
(baseline = base kernel, no tiering = 1.00x)

Summary: pghot hint-faults reproduces mainline NUMAB=2 with no regression on every workload; pghot HW-hints/IBS matches it mostly.

- NAS BT (MPI, Class F modified, 16 ranks): full footprint pre-staged on the CXL lower tier, then measured cold; isolates hot-page promotion.
- Graph500 (reference BFS, SCALE=28, edgefactor=16, 128 ranks):
working set on the CXL node; figure of merit is harmonic-mean TEPS.
- llama.cpp (llama-bench, Mixtral-8x22B Q4_K_M): ~1/3 of the model demoted to CXL under memory pressure; decode (tg128) is tier-sensitive.
- Redis + memtier (~64 GB dataset, 62.2M x 1 KB keys): dataset migrated to CXL, GET-heavy traffic over 50% of the keyspace.
- Microbenchmark: 64 threads, random 4K access over 8G on
the CXL node; latency-bound.

Detailed per-benchmark tables (throughput/latency + vmstat and pghot promotion counters) will be sent as replies to this patchset post.

How pghot works
===============
- Tracks frequency and last access time.
- Additionally, the accessing NUMA node ID (NID) for each recorded access is tracked in precision mode.
- These hotness parameters are maintained in a per-PFN hotness record within a per-section map hanging off the existing mem_section data structure. The map is RCU-protected and also carries a section-level "hot" flag used to gate scanning by kmigrated.
- In default mode, one byte (u8) is used for the hotness record. 5 bits store time using a bucketing scheme representing up to 4s with HZ=1000. The default toptier NID (0) is used as the promotion target and can be changed via the vm.pghot_target_nid sysctl.
- In precision mode, 4 bytes (u32) are used for each hotness record. 14 bits store time, representing around 16s with HZ=1000, and the accessing NID is stored per-PFN as the promotion target.
- Classifies pages as hot based on configurable thresholds.
- Pages classified as hot are marked ready for migration using the ready bit (MSB of the hotness record in both modes).
- Per-lower-tier-node kmigrated threads periodically scan the PFNs of lower-tier nodes, checking the migration-ready bit to perform batched migrations. The scan interval and batch size are configurable via debugfs tunables.

Memory overhead
===============
Default mode: 1 byte per lower-tier PFN. For 1TB of lower-tier memory this amounts to 256MB overhead (assuming 4K pages).

Precision mode: 4 bytes per lower-tier PFN. For 1TB of lower-tier memory this amounts to 1GB overhead.

Bit layout of hotness record
============================
Default mode
- Bits 0-1: Frequency (2 bits, 4 access samples)
- Bits 2-6: Bucketed time (5 bits, up to 4s with HZ=1000)
- Bit 7: Migration ready bit

Precision mode
- Bits 0-9: Target NID (10 bits)
- Bits 10-12: Frequency (3 bits, 8 access samples)
- Bits 13-26: Time (14 bits, up to 16s with HZ=1000)
- Bits 27-30: Reserved
- Bit 31: Migration ready bit

Tunables
========
sysctl (/proc/sys/vm/):
- pghot_enabled_sources - bitmask of hotness sources (0x1 hint

                               faults, 0x2 hardware hints)

- pghot_target_nid - toptier target NID (default mode)
- pghot_freq_threshold - accesses before a page is promotable
- pghot_promote_freq_window_ms - access-frequency counting window
- pghot_promote_rate_limit_MBps - per-target promotion rate limit (numa_balancing_promote_rate_limit_MBps is retained as a compat alias)

debugfs (/sys/kernel/debug/pghot/):
- kmigrated_sleep_ms, kmigrated_batch_nr

Potential hotness sources
=========================
1. NUMA Balancing (NUMAB2, Tiering mode) - included in this patchset.
2. AMD IBS Memory Profiler: HW based access profiler - included in this patchset.
3. klruscand - PTE-A bit scanning built on MGLRU's walk helpers - was showcased in previous versions but not part of this patchset.
4. folio_mark_accessed() - Page cache access tracking (unmapped page cache pages) - was showcased in previous versions but not part of this patchset.

Changes in v8
=============
pghot-core
- Augmented documentation with per-source information
- Reworked the per-section hotness map (mem_section->hot_map) into an RCU-protected structure (struct pghot_hot_map) that holds the per-PFN records and section-level flags, for safe concurrent access and teardown. The section "hot" flag now lives in this structure instead of the pointer's LSB.
- Cleanup and fixes in pghot_update_record().
- Moved pghot tunables enabled_sources, target_nid and freq_threshold from debugfs to sysctl (/proc/sys/vm/). Only kmigrated_sleep_ms and kmigrated_batch_nr remain in debugfs.
- Kmigrated walk: Protect folio_memcg() under RCU readlock

pghot-hintfaults
- Retained kernel.numa_balancing_promote_rate_limit_MBps for compatibility
- Filter shared executable pages at NUMA hint fault time in do_numa_page()/do_huge_pmd_numa_page(), since the VMA is not available to kmigrated's batched migration from non-process context.
- Fixed NUMAB2 to continue feeding to task_numa_fault() so that scan period updates happens for pghot-NUMAB2.
- Now hint faults source is enabled by default in pghot so that setting kernel.numa_balancing=2 is enough for tiering mode to get activated as before.
- To avoid a regression for existing configs, default PGHOT to y when NUMA_BALANCING is enabled and default NUMA_BALANCING_TIERING to y. Since the tiering promotion engine now lives in pghot, an existing CONFIG_NUMA_BALANCING=y config would otherwise silently lose the ability to use kernel.numa_balancing=2 (writes returning -EOPNOTSUPP) after an oldconfig upgrade.
Note: PGHOT depends on SPARSEMEM, so a NUMA_BALANCING=y && !SPARSEMEM (e.g. FLATMEM) config can no longer use kernel.numa_balancing=2.

pghot-hwhints
- Split the IBS Memory Profiler driver into core infrastructure and a separate patch that adds runtime controls (debugfs/sysfs and the ibs-memprof= cmdline parameter) and arms the profiler.
- Changed IBS Memory Profiler source to use regular interrupts rather than NMI. Unlike the primary IBS which needs to obtain samples from kernel and interrupt contexts for performance profiling, with memory access profiling only user space samples are obtained. Hence NMI isn't required. Also this simplifies the code by getting rid of irq_work hander.
- Fixed lost wakeup issue in work handler that drains the samples by draining samples in a loop until the ring buffer is observably empty.
- Drain and discard the samples of the CPU that is going offline.
- Fix the issue where the work item queued for a dying CPU can execute on a different online CPU leading to draining of that CPU's ring.
- Added ibs-memprof kernel cmdline option to enable hwhint source at boot time itself.
- Added vmstat counter hwhint_dropped_events that provides the count of those samples which were dropped due to sample buffer overrun.
- Added a sysfs tunable /sys/devices/system/cpu/ibs-mprof/enabled to dynamically enable or disable IBS Memory Profiler.
- Added debugfs tunables (l3miss-only, period, lat-filter, lat-thresh) to support configuration changes to IBS Memory profiler.

Compile test summary
====================
The series was compile-tested across the full range of relevant Kconfig combinations on x86_64 (defconfig-based) and cross-compiled on arm64. Combinations of PGHOT, PGHOT_PRECISE, NUMA_BALANCING, NUMA_BALANCING_TIERING, NUMA_MIGRATION, HWMEM_PROFILER/AMD_IBS_MEMPROF, DEBUG_FS and SYSCTL were verified, including pghot disabled, pghot standalone (no NUMA balancing), NUMA balancing mode 1 and mode 2 (tiering), precise mode, the AMD IBS memory profiler, NUMA_MIGRATION=n, and pghot with DEBUG_FS and/or SYSCTL disabled. Kconfig guards were confirmed (NUMA_BALANCING_TIERING without PGHOT, and HWMEM_PROFILER without a selecting driver, are dropped). Since PGHOT defaults to y when NUMA_BALANCING is set and NUMA_BALANCING_TIERING defaults to y, an existing config with NUMA_BALANCING=y but no pghot symbols resolves, via olddefconfig, to PGHOT=y and NUMA_BALANCING_TIERING=y and builds cleanly, while NUMA_BALANCING=n leaves PGHOT off. allnoconfig, tinyconfig, allyesconfig and allmodconfig were also built. All configurations build with no new warnings.

This v8 patchset applies on top of upstream commit 8cdeaa50eae8 and can be fetched from:

https://github.com/AMDESE/linux-mm/tree/bharata/pghot-v8

v7: https://lore.kernel.org/linux-mm/[email protected]/
v6: https://lore.kernel.org/linux-mm/[email protected]/
v5: https://lore.kernel.org/linux-mm/[email protected]/
v4: https://lore.kernel.org/linux-mm/[email protected]/
v3: https://lore.kernel.org/linux-mm/[email protected]/
v2: https://lore.kernel.org/linux-mm/[email protected]/
v1: https://lore.kernel.org/linux-mm/[email protected]/
v0: https://lore.kernel.org/linux-mm/[email protected]/

base-commit: 8cdeaa50eae8dad34885515f62559ee83e7e8dda

IBS Memory Profiler part of this patchset depends on the patchset that increases the number of APIC EILVT registers -
https://lore.kernel.org/lkml/[email protected]/

AI assistance was taken to run and summarize the benchmark results.

Bharata B Rao (7):
mm: migrate: Allow misplaced migration without VMA
mm: Hot page tracking and promotion - pghot
mm: pghot: Precision mode for pghot
mm: sched: move NUMA balancing tiering promotion to pghot x86/ibs: Move IBS caps definitions into its own header x86/mm/ibs: In-kernel driver for AMD IBS Memory Profiler x86/mm/ibs: Add runtime controls for IBS memprofiler

Gregory Price (1):
mm: migrate: Add promote_misplaced_memcg_folios()

 .../admin-guide/kernel-parameters.txt         |   5 +
 Documentation/admin-guide/mm/index.rst        |   1 +
 Documentation/admin-guide/mm/pghot.rst        | 196 ++++
 arch/x86/Kconfig                              |  16 +
 arch/x86/entry/entry_fred.c                   |   1 +
 arch/x86/include/asm/hardirq.h                |   3 +
 arch/x86/include/asm/ibs-caps.h               |  93 ++
 arch/x86/include/asm/ibs-mprof.h              |  61 ++
 arch/x86/include/asm/idtentry.h               |   6 +
 arch/x86/include/asm/irq_vectors.h            |   4 +-
 arch/x86/include/asm/msr-index.h              |   8 +
 arch/x86/include/asm/perf_event.h             |  81 +-
 arch/x86/kernel/idt.c                         |   3 +
 arch/x86/kernel/irq.c                         |   3 +
 arch/x86/mm/Makefile                          |   1 +
 arch/x86/mm/ibs-mprof.c                       | 704 ++++++++++++++
 include/linux/cpuhotplug.h                    |   1 +
 include/linux/migrate.h                       |   9 +-
 include/linux/mm.h                            |  35 +-
 include/linux/mmzone.h                        |  25 +-
 include/linux/pghot.h                         | 140 +++
 include/linux/vm_event_item.h                 |  12 +
 init/Kconfig                                  |  14 +
 kernel/sched/core.c                           |   7 +
 kernel/sched/debug.c                          |   1 -
 kernel/sched/fair.c                           | 171 +---
 kernel/sched/sched.h                          |   1 -
 mm/Kconfig                                    |  34 +
 mm/Makefile                                   |   6 +
 mm/huge_memory.c                              |  32 +-
 mm/memcontrol.c                               |   6 +-
 mm/memory-tiers.c                             |  15 +-
 mm/memory.c                                   |  36 +-
 mm/mempolicy.c                                |   3 -
 mm/migrate.c                                  |  99 +-
 mm/mm_init.c                                  |  10 +
 mm/pghot-default.c                            |  76 ++
 mm/pghot-precise.c                            |  77 ++
 mm/pghot.c                                    | 891 ++++++++++++++++++
 mm/vmstat.c                                   |  14 +-
 40 files changed, 2593 insertions(+), 308 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/pghot.rst
 create mode 100644 arch/x86/include/asm/ibs-caps.h
 create mode 100644 arch/x86/include/asm/ibs-mprof.h
 create mode 100644 arch/x86/mm/ibs-mprof.c
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot-precise.c
 create mode 100644 mm/pghot.c
Signature · 1 line
2.34.1
#1 lore raw

PATCH v8 1/8 mm: migrate: Allow misplaced migration without VMA

We want isolation of misplaced folios to work in contexts where VMA isn't available, typically when performing migrations from a kernel thread context. In order to prepare for that, allow migrate_misplaced_folio_prepare() to be called with a NULL VMA.

When migrate_misplaced_folio_prepare() is called with non-NULL VMA, it will check if the folio is mapped shared and that requires holding PTL lock. This path isn't taken when the function is invoked with NULL VMA (migration outside of process context). Therefore, when VMA == NULL, migrate_misplaced_folio_prepare() does not require the caller to hold the PTL.

---
 mm/migrate.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
mm/migrate.c +7 -2
index d9b23909d716..ab7227376757 100644
@@ -2674,7 +2674,12 @@ static struct folio *alloc_misplaced_dst_folio(struct folio *src,
 
 /*
  * Prepare for calling migrate_misplaced_folio() by isolating the folio if
- * permitted. Must be called with the PTL still held.
+ * permitted. Must be called with the PTL still held if called with a non-NULL
+ * vma.
+ *
+ * When called with a NULL vma (e.g., kernel thread initiated migration),
+ * migrate_misplaced_folio_prepare() will allow shared executable folios
+ * to be migrated.
  */
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -2691,7 +2696,7 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 		 * See folio_maybe_mapped_shared() on possible imprecision
 		 * when we cannot easily detect if a folio is shared.
 		 */
-		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+		if (vma && (vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
 			return -EACCES;
 
 		/*
Signature · 1 line
2.34.1
Reviewed-by on v8 1/8
#2 lore raw

Re: [PATCH v8 1/8] mm: migrate: Allow misplaced migration without VMA

Quoted 6 lines · Bharata B Rao

We want isolation of misplaced folios to work in contexts where VMA isn't available, typically when performing migrations from a kernel thread context. In order to prepare for that, allow migrate_misplaced_folio_prepare() to be called with a NULL VMA.

Quoted 8 lines

When migrate_misplaced_folio_prepare() is called with non-NULL VMA, it will check if the folio is mapped shared and that requires holding PTL lock. This path isn't taken when the function is invoked with NULL VMA (migration outside of process context). Therefore, when VMA == NULL, migrate_misplaced_folio_prepare() does not require the caller to hold the PTL.

#3 lore raw

PATCH v8 2/8 mm: migrate: Add promote_misplaced_memcg_folios()

In reply to Bharata B Rao

From: Gregory Price <[email protected]>

Tiered memory systems often require migrating multiple folios at once. Currently, migrate_misplaced_folio() handles only one folio per call, which is inefficient for batch operations. This patch introduces promote_misplaced_memcg_folios(), a batch variant that leverages migrate_pages() internally for improved performance.

The caller must isolate folios beforehand using migrate_misplaced_folio_prepare(). Additionally all the folios in the isolated list must belong to the same memcg. On return, the folio list will be empty regardless of success or failure.

This function will be used by pghot kmigrated thread.

[Rewrote commit description, memcg awareness]
---
 include/linux/migrate.h |  5 ++++
 mm/migrate.c            | 58 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 63 insertions(+)
include/linux/migrate.h +5 -0
index d5af2b7f577b..d136612eef9d 100644
@@ -111,6 +111,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
+int promote_misplaced_memcg_folios(struct list_head *folio_list, int node);
 #else
 static inline int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node)
@@ -121,6 +122,10 @@ static inline int migrate_misplaced_folio(struct folio *folio, int node)
 {
 	return -EAGAIN; /* can't migrate now */
 }
+static inline int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
+{
+	return -EAGAIN; /* can't migrate now */
+}
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_MIGRATION
mm/migrate.c +58 -0

62 lines not on this page · raw message

Signature · 1 line
2.34.1
#4 lore raw

Re: [PATCH v8 2/8] mm: migrate: Add promote_misplaced_memcg_folios()

[Reply to Shashiko review]

Quoted 48 lines · Bharata B Rao
mm/migrate.c +37 -0
index ab7227376757..58a8a0cf6fa3 100644
@@ -2773,4 +2773,62 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	BUG_ON(!list_empty(&migratepages));
 	return nr_remaining ? -EAGAIN : 0;
 }
+
+/**
+ * promote_misplaced_memcg_folios() - Batch variant of migrate_misplaced_folio
+ * Attempts to promote a folio list to the specified destination.
+ * @folio_list: Isolated list of folios to be batch-promoted.
+ * @node: The NUMA node ID to where the folios should be promoted.
+ *
+ * Caller is expected to have isolated the folios by calling
+ * migrate_misplaced_folio_prepare(), which will result in an
+ * elevated reference count on the folios. All the isolated folios
+ * in the list must belong to the same memcg so that NUMA_PAGE_MIGRATE
+ * stat can be attributed correctly to the memcg.
+ *
+ * This function will un-isolate the folios, drop the elevated reference
+ * and remove them from the list before returning. This should be called
+ * only for batched promotion of hot pages from lower tier nodes.
+ *
+ * Return: 0 on success and -EAGAIN on failure or partial promotion.
+ *         On return, @folio_list will be empty regardless of success/failure.
+ */
+int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
+{
+	struct mem_cgroup *memcg = NULL;
+	unsigned int nr_succeeded = 0;
+	struct folio *first;
+	int nr_remaining;
+
+	if (list_empty(folio_list))
+		return 0;
+
+	first = list_first_entry(folio_list, struct folio, lru);
+#ifdef CONFIG_DEBUG_VM
+	{
+		struct folio *f;
+
+		list_for_each_entry(f, folio_list, lru)
+			VM_WARN_ON_ONCE(folio_memcg(f) != folio_memcg(first));
 

Does this iteration need to hold the RCU read lock?
Looking at folio_memcg(), it calls obj_cgroup_memcg(), which executes a

Quoted 8 lines

lockdep assertion requiring either rcu_read_lock or cgroup_mutex. Calling it without these locks will cause a lockdep splat when CONFIG_PROVE_LOCKING is enabled.
Additionally, without the RCU lock, can the returned memcg pointer be concurrently updated due to memcg reparenting? If so, this race might cause folio_memcg(f) and folio_memcg(first) to evaluate to different pointers even if both folios belong to the same original memcg, potentially triggering a spurious VM_WARN_ON_ONCE().

That list walk needs RCU read lock protection. Will fix.

Regards,
Bharata.

Reviewed-by on v8 2/8
#5 lore raw

Re: [PATCH v8 2/8] mm: migrate: Add promote_misplaced_memcg_folios()

In reply to Bharata B Rao
Quoted 13 lines · Bharata B Rao

From: Gregory Price <[email protected]>

Tiered memory systems often require migrating multiple folios at once. Currently, migrate_misplaced_folio() handles only one folio per call, which is inefficient for batch operations. This patch introduces promote_misplaced_memcg_folios(), a batch variant that leverages migrate_pages() internally for improved performance.

The caller must isolate folios beforehand using migrate_misplaced_folio_prepare(). Additionally all the folios in the isolated list must belong to the same memcg. On return, the folio list will be empty regardless of success or failure.

Quoted 5 lines

This function will be used by pghot kmigrated thread.

[Rewrote commit description, memcg awareness]
#6 lore raw

PATCH v8 3/8 mm: Hot page tracking and promotion - pghot

In reply to Bharata B Rao

pghot is a subsystem that collects memory access information from multiple sources, classifies hot pages resident in lower-tier memory, and promotes them to faster tiers. It stores per-PFN hotness metadata and performs asynchronous, batched promotion via a per-lower-tier-node kernel thread (kmigrated).

This change introduces the default (compact) mode of pghot:

- Per-PFN hotness record (phi_t = u8) embedded via mem_section:
- 2 bits: access frequency (4 levels)
- 5 bits: time bucket (≈4s window with HZ=1000, bucketed jiffies)
- 1 bit : migration-ready flag (MSB)

- Event recording API:
int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now) @pfn: The PFN of the memory accessed
@nid: The accessing NUMA node ID
@src: The temperature source (subsystem) that generated the
access info
@time: The access time in jiffies
- Sources (e.g., NUMA hint faults, HW hints) call this to report accesses.
- In default mode, the nid is not stored/used for targeting; promotion goes to a configurable toptier node (pghot_target_nid).

- Promotion engine:
- One kmigrated thread per lower-tier node.
- Scans only sections whose "hot" flag was raised, iterates PFNs, and batches candidates by destination node.
- Uses promote_misplaced_memcg_folios() to move batched folios.

- Tunables & stats:
- debugfs: kmigrated_sleep_ms, kmigrated_batch_nr
- sysctl : vm.pghot_promote_freq_window_ms, vm.pghot_freq_threshold, vm.pghot_enabled_sources, vm.pghot_target_nid
- vmstat : pghot_recorded_accesses, pghot_recorded_hintfaults, pghot_recorded_hwhints

Memory overhead
---------------
Default mode uses 1 byte of hotness metadata per PFN on lower-tier nodes.

Behavior & policy
-----------------
- Default mode promotion target:
The nid passed by sources is not stored; hot pages promote to pghot_target_nid (toptier). Precision mode (added later in the series) changes this.

- Record consumption:
kmigrated consumes (clears) the "migration-ready" bit before attempting isolation. Additionally the hotness record is reset. If isolation/migration fails, the folio is not re-queued automatically; subsequent accesses will re-arm it. This avoids retry storms and keeps batching stable.

- Wakeups:
kmigrated wakeups are intentionally timeout-driven. We set the per-pgdat "activate" flag on access, and kmigrated checks this flag on its next sleep interval. This keeps the first cut simple and avoids potential wake storms; active wakeups can be considered in a follow-up.

---
 Documentation/admin-guide/mm/index.rst |   1 +
 Documentation/admin-guide/mm/pghot.rst |  89 +++
 include/linux/migrate.h                |   4 +-
 include/linux/mmzone.h                 |  21 +
 include/linux/pghot.h                  | 103 ++++
 include/linux/vm_event_item.h          |   5 +
 mm/Kconfig                             |  14 +
 mm/Makefile                            |   1 +
 mm/migrate.c                           |  16 +-
 mm/mm_init.c                           |  10 +
 mm/pghot-default.c                     |  76 +++
 mm/pghot.c                             | 725 +++++++++++++++++++++++++
 mm/vmstat.c                            |   5 +
 13 files changed, 1063 insertions(+), 7 deletions(-)
 create mode 100644 Documentation/admin-guide/mm/pghot.rst
 create mode 100644 include/linux/pghot.h
 create mode 100644 mm/pghot-default.c
 create mode 100644 mm/pghot.c
Documentation/admin-guide/mm/index.rst +1 -0
index bbb563cba5d2..4d6810b02365 100644
@@ -43,3 +43,4 @@ the Linux memory management.
    userfaultfd
    zswap
    kho
+   pghot
Documentation/admin-guide/mm/pghot.rst new file +89 -0

89 lines not on this page · raw message

include/linux/migrate.h +2 -2
index d136612eef9d..53bae80d11ae 100644
@@ -107,7 +107,7 @@ static inline void softleaf_entry_wait_on_locked(softleaf_t entry, spinlock_t *p
 
 #endif /* CONFIG_MIGRATION */
 
-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
 int migrate_misplaced_folio_prepare(struct folio *folio,
 		struct vm_area_struct *vma, int node);
 int migrate_misplaced_folio(struct folio *folio, int node);
@@ -126,7 +126,7 @@ static inline int promote_misplaced_memcg_folios(struct list_head *folio_list, i
 {
 	return -EAGAIN; /* can't migrate now */
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */
 
 #ifdef CONFIG_MIGRATION
 
include/linux/mmzone.h +21 -0
index ca2712187147..11c547a51f43 100644
@@ -1156,6 +1156,7 @@ enum pgdat_flags {
 					 * many pages under writeback
 					 */
 	PGDAT_RECLAIM_LOCKED,		/* prevents concurrent reclaim */
+	PGDAT_KMIGRATED_ACTIVATE,	/* activates kmigrated */
 };
 
 enum zone_flags {
@@ -1598,6 +1599,10 @@ typedef struct pglist_data {
 #ifdef CONFIG_MEMORY_FAILURE
 	struct memory_failure_stats mf_stats;
 #endif
+#ifdef CONFIG_PGHOT
+	struct task_struct *kmigrated;
+	wait_queue_head_t kmigrated_wait;
+#endif
 } pg_data_t;
 
 #define node_present_pages(nid)	(NODE_DATA(nid)->node_present_pages)
@@ -1992,6 +1997,7 @@ struct mem_section_usage {
 
 struct page;
 struct page_ext;
+struct pghot_hot_map;
 struct mem_section {
 	/*
 	 * This is, logically, a pointer to an array of struct
@@ -2008,12 +2014,27 @@ struct mem_section {
 	unsigned long section_mem_map;
 
 	struct mem_section_usage *usage;
+#ifdef CONFIG_PGHOT
+	/*
+	 * RCU-protected pointer to this section's struct pghot_hot_map,
+	 * which holds the per-PFN hotness records and the section-level
+	 * flags.
+	 */
+	struct pghot_hot_map __rcu *hot_map;
+#endif
 #ifdef CONFIG_PAGE_EXTENSION
 	/*
 	 * If SPARSEMEM, pgdat doesn't have page_ext pointer. We use
 	 * section. (see page_ext.h about this.)
 	 */
 	struct page_ext *page_ext;
+#endif
+	/*
+	 * Padding to maintain consistent mem_section size when exactly
+	 * one of PGHOT or PAGE_EXTENSION is enabled. This ensures
+	 * optimal alignment regardless of configuration.
+	 */
+#if (defined(CONFIG_PGHOT) ^ defined(CONFIG_PAGE_EXTENSION))
 	unsigned long pad;
 #endif
 	/*
include/linux/pghot.h new file +103 -0

103 lines not on this page · raw message

include/linux/vm_event_item.h +5 -0
index 03fe95f5a020..58d510711bd4 100644
@@ -175,6 +175,11 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		KSTACK_REST,
 #endif
 #endif /* CONFIG_DEBUG_STACK_USAGE */
+#ifdef CONFIG_PGHOT
+		PGHOT_RECORDED_ACCESSES,
+		PGHOT_RECORDED_HINTFAULTS,
+		PGHOT_RECORDED_HWHINTS,
+#endif /* CONFIG_PGHOT */
 		NR_VM_EVENT_ITEMS
 };
 
mm/Kconfig +14 -0
index 9e0ca4824905..0a5bcd5d45ed 100644
@@ -1509,6 +1509,20 @@ config LAZY_MMU_MODE_KUNIT_TEST
 
 	  If unsure, say N.
 
+config PGHOT
+	bool "Hot page tracking and promotion"
+	default n
+	depends on NUMA_MIGRATION && SPARSEMEM
+	help
+	  A sub-system to track page accesses in lower tier memory and
+	  maintain hot page information. Promotes hot pages from lower
+	  tiers to top tier by using the memory access information provided
+	  by various sources. Asynchronous promotion is done by per-node
+	  kernel threads.
+
+	  This adds 1 byte of metadata overhead per page in lower-tier
+	  memory nodes.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +1 -0
index eff9f9e7e061..4939a1a74c1d 100644
@@ -147,3 +147,4 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
+obj-$(CONFIG_PGHOT) += pghot.o pghot-default.o
mm/migrate.c +11 -5
index 58a8a0cf6fa3..7ab15780226e 100644
@@ -2628,7 +2628,7 @@ SYSCALL_DEFINE6(move_pages, pid_t, pid, unsigned long, nr_pages,
 }
 #endif /* CONFIG_NUMA_MIGRATION */
 
-#ifdef CONFIG_NUMA_BALANCING
+#if defined(CONFIG_NUMA_BALANCING) || defined(CONFIG_PGHOT)
 /*
  * Returns true if this is a safe migration target node for misplaced NUMA
  * pages. Currently it only checks the watermarks which is crude.
@@ -2748,12 +2748,10 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
  */
 int migrate_misplaced_folio(struct folio *folio, int node)
 {
-	pg_data_t *pgdat = NODE_DATA(node);
 	int nr_remaining;
 	unsigned int nr_succeeded;
 	LIST_HEAD(migratepages);
 	struct mem_cgroup *memcg = get_mem_cgroup_from_folio(folio);
-	struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
 
 	list_add(&folio->lru, &migratepages);
 	nr_remaining = migrate_pages(&migratepages, alloc_misplaced_dst_folio,
@@ -2762,12 +2760,18 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 	if (nr_remaining && !list_empty(&migratepages))
 		putback_movable_pages(&migratepages);
 	if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
 		if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
 		    && !node_is_toptier(folio_nid(folio))
-		    && node_is_toptier(node))
+		    && node_is_toptier(node)) {
+			pg_data_t *pgdat = NODE_DATA(node);
+			struct lruvec *lruvec = mem_cgroup_lruvec(memcg, pgdat);
+
 			mod_lruvec_state(lruvec, PGPROMOTE_SUCCESS, nr_succeeded);
+		}
+#endif
 	}
 	mem_cgroup_put(memcg);
 	BUG_ON(!list_empty(&migratepages));
@@ -2821,14 +2825,16 @@ int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
 		putback_movable_pages(folio_list);
 
 	if (nr_succeeded) {
+#ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
 		mod_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(node)),
 				 PGPROMOTE_SUCCESS, nr_succeeded);
+#endif
 	}
 
 	mem_cgroup_put(memcg);
 	WARN_ON(!list_empty(folio_list));
 	return nr_remaining ? -EAGAIN : 0;
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING || CONFIG_PGHOT */
mm/mm_init.c +10 -0
index 0f64909e8d20..13ef3cf1e48c 100644
@@ -1386,6 +1386,15 @@ static void pgdat_init_kcompactd(struct pglist_data *pgdat)
 static void pgdat_init_kcompactd(struct pglist_data *pgdat) {}
 #endif
 
+#ifdef CONFIG_PGHOT
+static void pgdat_init_kmigrated(struct pglist_data *pgdat)
+{
+	init_waitqueue_head(&pgdat->kmigrated_wait);
+}
+#else
+static inline void pgdat_init_kmigrated(struct pglist_data *pgdat) {}
+#endif
+
 static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 {
 	int i;
@@ -1393,6 +1402,7 @@ static void __meminit pgdat_init_internals(struct pglist_data *pgdat)
 	pgdat_resize_init(pgdat);
 	pgdat_kswapd_lock_init(pgdat);
 	pgdat_init_kcompactd(pgdat);
+	pgdat_init_kmigrated(pgdat);
 
 	init_waitqueue_head(&pgdat->kswapd_wait);
 	init_waitqueue_head(&pgdat->pfmemalloc_wait);
mm/pghot-default.c new file +76 -0

76 lines not on this page · raw message

mm/pghot.c new file +725 -0

725 lines not on this page · raw message

mm/vmstat.c +5 -0
index f534972f517d..4064ead568cc 100644
@@ -1489,6 +1489,11 @@ const char * const vmstat_text[] = {
 	[I(KSTACK_REST)]			= "kstack_rest",
 #endif
 #endif
+#ifdef CONFIG_PGHOT
+	[I(PGHOT_RECORDED_ACCESSES)]		= "pghot_recorded_accesses",
+	[I(PGHOT_RECORDED_HINTFAULTS)]		= "pghot_recorded_hintfaults",
+	[I(PGHOT_RECORDED_HWHINTS)]		= "pghot_recorded_hwhints",
+#endif /* CONFIG_PGHOT */
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
 };
Signature · 1 line
2.34.1
#7 lore raw

Re: [PATCH v8 3/8] mm: Hot page tracking and promotion - pghot

[Reply to Shashiko review]

Quoted 13 lines · Bharata B Rao
mm/Kconfig +4 -0
index 9e0ca4824905..0a5bcd5d45ed 100644
@@ -1509,6 +1509,20 @@ config LAZY_MMU_MODE_KUNIT_TEST
 
 	  If unsure, say N.
 
+config PGHOT
+	bool "Hot page tracking and promotion"
+	default n
+	depends on NUMA_MIGRATION && SPARSEMEM
 
Quoted 4 lines

Will this fail to compile if CONFIG_PGHOT is enabled but CONFIG_NUMA_BALANCING is disabled? The kmigrated_promotion_rate_limit() function unconditionally accesses fields like pgdat->nbp_rl_start which are gated by CONFIG_NUMA_BALANCING.

kmigrated_promotion_rate_limit() and the pgdat->nbp_* fields aren't part of this patch (3/8) -- they're introduced in 5/8. That same patch moves the nbp_* fields (and the PGPROMOTE_* node stats) from #ifdef CONFIG_NUMA_BALANCING to #ifdef CONFIG_PGHOT.

So with CONFIG_PGHOT=y the fields exist regardless of CONFIG_NUMA_BALANCING. PGHOT depends only on NUMA_MIGRATION && SPARSEMEM, so PGHOT=y with NUMA_BALANCING=n is a supported configuration and builds cleanly.

Quoted 35 lines
mm/pghot-default.c new file +28 -0
new file mode 100644
index 000000000000..ed66a867a644
@@ -0,0 +1,76 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pghot: Default mode
+ *
+ * 1 byte hotness record per PFN.
+ * Bucketed time and frequency tracked as part of the record.
+ * Promotion to @sysctl_pghot_target_nid by default.
+ */
+
+#include <linux/pghot.h>
+#include <linux/jiffies.h>
+
+/* pghot-default doesn't store and hence no NID validation is required */
+bool pghot_nid_valid(int nid)
+{
+	return true;
+}
+
+/*
+ * @time is regular time, @old_time is bucketed time.
+ */
+unsigned long pghot_access_latency(unsigned long old_time, unsigned long time)
+{
+	time &= PGHOT_TIME_BUCKETS_MASK;
+	old_time <<= PGHOT_TIME_BUCKETS_SHIFT;
+
+	return jiffies_to_msecs((time - old_time) & PGHOT_TIME_BUCKETS_MASK);
+}
Quoted 5 lines

Does this time calculation allow cold pages to be falsely promoted? Since the time bucket uses 5 bits, PGHOT_TIME_BUCKETS_MASK wraps around approximately every 4 seconds at HZ=1000. If an access occurs hours later, the modulo arithmetic might yield a small random latency, tricking the threshold check and causing widespread false promotions.

This is the same scheme that the existing NUMAB=2 has always used:
numa_hint_fault_latency() computes (time - last_time) & PAGE_ACCESS_TIME_MASK. pghot's default (1-byte) record simply keeps that accuracy-vs-space tradeoff; precise mode (CONFIG_PGHOT_PRECISE) widens the window to ~16s.

This is not the only check that allows promotion. There are others as well like frequency threshold (by default 2 accesses), pgdat_free_space_enough() check, the dynamic hot-threshold adjustment, and pghot_promote_rate_limit.

A genuinely cold page may get promoted (and will eventually get demoted too under pressure) but I don't think it will lead to widespread false promotions. I'll add a comment to pghot_access_latency() documenting the bounded-window tradeoff so it's explicit.

Quoted 95 lines
mm/pghot-default.c new file +1 -0
+}
mm/pghot.c +92 -0
+/*
+ * Walks the PFNs of the zone, isolates and migrates them in batches.
+ */
+static void kmigrated_walk_zone(unsigned long start_pfn, unsigned long end_pfn,
+				int src_nid)
+{
+	struct mem_cgroup *cur_memcg = NULL;
+	int cur_nid = NUMA_NO_NODE;
+	LIST_HEAD(migrate_list);
+	int batch_count = 0;
+	struct folio *folio;
+	struct page *page;
+	unsigned long pfn;
+
+	pfn = start_pfn;
+	do {
+		int nid = NUMA_NO_NODE, nr = 1;
+		struct mem_cgroup *memcg;
+		unsigned long time = 0;
+		int freq = 0;
+
+		if (!pfn_valid(pfn))
+			goto out_next;
+
+		page = pfn_to_online_page(pfn);
+		if (!page)
+			goto out_next;
+
+		folio = page_folio(page);
+		if (!folio_try_get(folio))
+			goto out_next;
+
+		if (unlikely(page_folio(page) != folio)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		nr = folio_nr_pages(folio);
+		if (folio_nid(folio) != src_nid) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (!folio_test_lru(folio)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (pghot_get_hotness(pfn, &nid, &freq, &time)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (folio_nid(folio) == nid) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		if (migrate_misplaced_folio_prepare(folio, NULL, nid)) {
+			folio_put(folio);
+			goto out_next;
+		}
+
+		rcu_read_lock();
+		memcg = folio_memcg(folio);
+		rcu_read_unlock();
+		if (cur_nid == NUMA_NO_NODE) {
+			cur_nid = nid;
+			cur_memcg = memcg;
+		}
+
+		/* If NID or memcg changed, flush the previous batch first */
+		if (cur_nid != nid || cur_memcg != memcg) {
+			if (!list_empty(&migrate_list))
+				promote_misplaced_memcg_folios(&migrate_list, cur_nid);
+			cur_nid = nid;
+			cur_memcg = memcg;
+			batch_count = 0;
+			cond_resched();
+		}
+
+		list_add(&folio->lru, &migrate_list);
+		folio_put(folio);
+
+		if (++batch_count > READ_ONCE(kmigrated_batch_nr)) {
+			promote_misplaced_memcg_folios(&migrate_list, cur_nid);
+			batch_count = 0;
+			cond_resched();
+		}
+out_next:
+		pfn += nr;
+	} while (pfn < end_pfn);
Quoted 8 lines

Is there a risk of unbounded PFN scanning causing soft lockups here? cond_resched() is only called when a batch is successfully flushed. If a section's hot bit is set but no pages qualify for migration and the code takes the goto out_next path, the loop might spin through millions of PFNs without ever yielding the CPU.

Is there a risk of unbounded PFN scanning causing soft lockups here? ... the loop might spin through millions of PFNs without ever yielding

kmigrated_walk_zone() is called per-section, so a single walk scans at most PAGES_PER_SECTION PFNs (32768 on x86_64) not millions. The worst case can occur if kmigrated_do_work()'s outer loop walks every hot section of the node back-to-back without resulting in anything to migrate.

Hence let me add a resched point once per section in that outer loop.

Quoted 5 lines

Could this PFN advancement skip valid folios if the loop starts on a tail page? If folio_try_get() fails on a head page, nr remains 1 and pfn advances to a tail page. On the next iteration, resolving the head folio succeeds, setting nr to folio_nr_pages(folio). Adding this full length to the tail page's PFN will overshoot the actual end of the folio and skip subsequent pages.

Theoretically yes, but not sure if that can happen for standard THPs. And gigantic (1G) folios are anyway skipped as they aren't part of LRU.

But then I will fix it to ensure that it won't ever happen

Quoted 16 lines
mm/pghot.c +15 -0
+	if (!list_empty(&migrate_list))
+		promote_misplaced_memcg_folios(&migrate_list, cur_nid);
+}
+
+static void kmigrated_do_work(pg_data_t *pgdat)
+{
+	unsigned long section_nr, s_begin, start_pfn;
+	struct pghot_hot_map *hot_map;
+	struct mem_section *ms;
+	bool hot;
+	int nid;
+
+	clear_bit(PGDAT_KMIGRATED_ACTIVATE, &pgdat->flags);
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
Quoted 4 lines

Does this skip the very first present memory section? The macro for_each_present_section_nr() internally calls next_present_section_nr() on its start parameter, which means it will immediately advance past s_begin and never process the first section for promotion.

No. next_present_section_nr() is called with (start - 1) which guarantees that the first present section is considered in the loop.

Quoted 10 lines
mm/pghot.c +5 -0
+		start_pfn = section_nr_to_pfn(section_nr);
+		ms = __nr_to_section(section_nr);
+
+		if (!pfn_valid(start_pfn))
+			continue;

What happens if a memory section is only partially populated and starts with a memory hole? This check appears to skip the entire section based solely on the first PFN, which would ignore any valid hot folios located further inside the section.

Hmm what kind of sections are those which start with a hole? Device memory sections? Those are already excluded from pghot's tracking. pghot tracks only on-LRU RAM folios whose sections won't get skipped with this check, I believe.

Quoted 32 lines
mm/pghot.c +30 -0
+
+		nid = pfn_to_nid(start_pfn);
+		if (node_is_toptier(nid) || nid != pgdat->node_id)
+			continue;
+
+		rcu_read_lock();
+		hot_map = rcu_dereference(ms->hot_map);
+		hot = hot_map &&
+		      test_and_clear_bit(PGHOT_SECTION_HOT_BIT, &hot_map->flags);
+		rcu_read_unlock();
+
+		if (!hot)
+			continue;
+
+		kmigrated_walk_zone(start_pfn, start_pfn + PAGES_PER_SECTION,
+				    pgdat->node_id);
+	}
+}
+
 
+
+static void pghot_free_hot_map(struct mem_section *ms)
+{
+	struct pghot_hot_map *hot_map;
+
+	hot_map = rcu_dereference_protected(ms->hot_map, 1);
+	if (!hot_map)
+		return;
+
+	RCU_INIT_POINTER(ms->hot_map, NULL);
+	kfree_rcu(hot_map, rcu);
Quoted 3 lines

Can two racing threads double-free the hot_map pointer here? Because rcu_dereference_protected() is used without a lock, concurrent calls might both read the same non-NULL pointer and call kfree_rcu() on it.

pghot_free_hot_map() is called for three places:

1. pghot_offline_sec_hotmap() — from the notifier on MEM_OFFLINE / MEM_CANCEL_ONLINE.
2. pghot_online_sec_hotmap() rollback — from the notifier on MEM_GOING_ONLINE.
3. pghot_destroy_hot_map() — only from pghot_init()'s failure path.

First two paths are already serialized by mem_hotplug_lock. Regarding the third one, pghot_destroy_hot_map() needs to run under get_online_mems() to address your next two comments. That would close the window for path 3 as well.

Quoted 71 lines
mm/pghot.c +70 -0
+}
+
+static int pghot_alloc_hot_map(struct mem_section *ms, int nid)
+{
+	struct pghot_hot_map *hot_map;
+
+	hot_map = kzalloc_node(struct_size(hot_map, phi, PAGES_PER_SECTION),
+			       GFP_KERNEL, nid);
+	if (!hot_map)
+		return -ENOMEM;
+
+	rcu_assign_pointer(ms->hot_map, hot_map);
+	return 0;
+}
+
+static void pghot_offline_sec_hotmap(unsigned long start_pfn,
+				     unsigned long nr_pages)
+{
+	unsigned long start, end, pfn;
+	struct mem_section *ms;
+
+	start = SECTION_ALIGN_DOWN(start_pfn);
+	end = SECTION_ALIGN_UP(start_pfn + nr_pages);
+
+	for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {
+		ms = __pfn_to_section(pfn);
+		if (!ms)
+			continue;
+
+		pghot_free_hot_map(ms);
+	}
+}
+
+static int pghot_online_sec_hotmap(unsigned long start_pfn,
+				   unsigned long nr_pages)
+{
+	int nid = pfn_to_nid(start_pfn);
+	unsigned long start, end, pfn;
+	struct mem_section *ms;
+	int fail = 0;
+
+	/*
+	 * No section hot maps for toptier sections.
+	 */
+	if (node_is_toptier(nid))
+		return 0;
+
+	start = SECTION_ALIGN_DOWN(start_pfn);
+	end = SECTION_ALIGN_UP(start_pfn + nr_pages);
+
+	for (pfn = start; !fail && pfn < end; pfn += PAGES_PER_SECTION) {
+		ms = __pfn_to_section(pfn);
+		if (!ms || rcu_access_pointer(ms->hot_map))
+			continue;
+
+		fail = pghot_alloc_hot_map(ms, nid);
+	}
+
+	if (!fail)
+		return 0;
+
+	/* rollback: free the sections allocated before the failure */
+	end = pfn - PAGES_PER_SECTION;
+	for (pfn = start; pfn < end; pfn += PAGES_PER_SECTION) {
+		ms = __pfn_to_section(pfn);
+		if (ms)
+			pghot_free_hot_map(ms);
+	}
+	return -ENOMEM;
+}
Quoted 4 lines

Could this rollback loop inadvertently destroy hotness metadata for pre-existing online sections? The loop iterates from start to the failure point and frees every hot_map without checking if this specific function invocation was the one that originally allocated it.

Though there is a check to skip allocation if the hot_map already exists, it is not really possible to hit that as I think memory hotplugs are section aligned and we free hot_maps of those sections (ranging from start to failure pfn) for which allocation succeded during this onlining.

The only way the section being onlined could have a map is due to the setup-vs-notifier race, which I will fix as explained in the previous and the next review comments.

Quoted 38 lines
mm/pghot.c +37 -0
+
+static int pghot_memhp_callback(struct notifier_block *self,
+				unsigned long action, void *arg)
+{
+	struct memory_notify *mn = arg;
+	int ret = 0;
+
+	switch (action) {
+	case MEM_GOING_ONLINE:
+		ret = pghot_online_sec_hotmap(mn->start_pfn, mn->nr_pages);
+		break;
+	case MEM_OFFLINE:
+	case MEM_CANCEL_ONLINE:
+		pghot_offline_sec_hotmap(mn->start_pfn, mn->nr_pages);
+		break;
+	}
+
+	return notifier_from_errno(ret);
+}
+
+static struct notifier_block pghot_mem_notifier = {
+	.notifier_call = pghot_memhp_callback,
+	.priority = DEFAULT_CALLBACK_PRI,
+};
+
+static void pghot_destroy_hot_map(void)
+{
+	unsigned long section_nr, s_begin;
+	struct mem_section *ms;
+
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
+		ms = __nr_to_section(section_nr);
+		pghot_free_hot_map(ms);
+	}
+
+	unregister_memory_notifier(&pghot_mem_notifier);
Quoted 4 lines

Is there a race condition with the memory hotplug notifier during teardown? Because the maps are freed before the notifier is unregistered, a concurrent memory offline event could trigger the callback and attempt to free the same map again.

See the reply to next comment.

Quoted 26 lines
mm/pghot.c +25 -0
+}
+
+static int pghot_setup_hot_map(void)
+{
+	unsigned long section_nr, s_begin, start_pfn;
+	struct mem_section *ms;
+	int nid, ret;
+
+	ret = register_memory_notifier(&pghot_mem_notifier);
+	if (ret)
+		return ret;
+
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
+		ms = __nr_to_section(section_nr);
+		start_pfn = section_nr_to_pfn(section_nr);
+		nid = pfn_to_nid(start_pfn);
+
+		if (node_is_toptier(nid) || !pfn_valid(start_pfn))
+			continue;
+
+		if (pghot_alloc_hot_map(ms, nid))
+			goto out_free_hot_map;
+	}
+	return 0;
Quoted 4 lines

Can a concurrent memory hotplug event leak hot_map structures during setup? The notifier is registered before the loop allocates maps for pre-existing sections. If a memory online event triggers the callback concurrently, it could allocate a map that this loop subsequently overwrites and leaks.

Both the above are locking issues. They can be fixed by holding get_online_mems() in both setup and destroy paths.

Regards,
Bharata.

#8 lore raw

Re: [PATCH v8 3/8] mm: Hot page tracking and promotion - pghot

In reply to Bharata B Rao
Quoted 17 lines · Bharata B Rao
mm/pghot.c new file +17 -0
+/**
+ * pghot_record_access() - Record page accesses from lower tier memory
+ * for the purpose of tracking page hotness and subsequent promotion.
+ *
+ * @pfn: PFN of the page
+ * @nid: Unused
+ * @src: The identifier of the sub-system that reports the access
+ * @now: Access time in jiffies
+ *
+ * Updates the frequency and time of access and marks the page as
+ * ready for migration if the frequency crosses a threshold. The pages
+ * marked for migration are migrated by kmigrated kernel thread.
+ *
+ * Return: 0 on success and -EINVAL on failure to record the access.
+ */
+int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
+{

[...]

Quoted 1 line

+}

This needs a EXPORT_SYMBOL_GPL(pghot_record_access) for any module based sources.

Thanks,
Davidlohr

#9 lore raw

Re: [PATCH v8 3/8] mm: Hot page tracking and promotion - pghot

Quoted 20 lines · Davidlohr Bueso
Quoted 17 lines · Bharata B Rao
mm/pghot.c new file +17 -0
+/**
+ * pghot_record_access() - Record page accesses from lower tier memory
+ * for the purpose of tracking page hotness and subsequent promotion.
+ *
+ * @pfn: PFN of the page
+ * @nid: Unused
+ * @src: The identifier of the sub-system that reports the access
+ * @now: Access time in jiffies
+ *
+ * Updates the frequency and time of access and marks the page as
+ * ready for migration if the frequency crosses a threshold. The pages
+ * marked for migration are migrated by kmigrated kernel thread.
+ *
+ * Return: 0 on success and -EINVAL on failure to record the access.
+ */
+int pghot_record_access(unsigned long pfn, int nid, int src, unsigned long now)
+{

[...]

Quoted 4 lines
Quoted 1 line

+}

This needs a EXPORT_SYMBOL_GPL(pghot_record_access) for any module based sources.

Ah I didn't see that till now. CXL driver would need that. Thanks will add.

Regards,
Bharata.

#10 lore raw

PATCH v8 4/8 mm: pghot: Precision mode for pghot

In reply to Bharata B Rao

Default pghot stores hotness in a 1‑byte record per PFN, limiting frequency to 2 bits, time to a 5‑bit bucket, and preventing storage of per‑PFN toptier NID. This restricts time granularity and forces all promotions to use the global pghot_target_nid.

This patch adds an optional precision mode (CONFIG_PGHOT_PRECISE) that expands the hotness record to 4 bytes (u32) and provides:

- 10‑bit NID field for per‑PFN promotion target,
- 3‑bit frequency field (freq_threshold range 1–7),
- 14‑bit time field offering finer recency tracking,
- MSB migrate‑ready bit.

Precision mode improves placement accuracy on systems with multiple toptier nodes and provides higher‑resolution hotness tracking, at the cost of increasing metadata to 4 bytes per PFN.

Documentation, tunables, and the record layout are updated accordingly.

---
 Documentation/admin-guide/mm/pghot.rst |  4 +-
 include/linux/pghot.h                  | 39 ++++++++++++-
 mm/Kconfig                             | 11 ++++
 mm/Makefile                            |  7 ++-
 mm/pghot-precise.c                     | 77 ++++++++++++++++++++++++++
 mm/pghot.c                             | 13 +++--
 6 files changed, 143 insertions(+), 8 deletions(-)
 create mode 100644 mm/pghot-precise.c
Documentation/admin-guide/mm/pghot.rst +2 -2
index 0edbe0082816..6d9b3d9522d1 100644
@@ -56,7 +56,7 @@ Path: /proc/sys/vm/pghot_target_nid
 Path: /proc/sys/vm/pghot_freq_threshold
 
 - Minimum access frequency before a page is marked ready for promotion.
-  Range is 1 to 3 in default mode.
+  Range is 1 to 3 in default mode and 1 to 7 in precision mode.
 - Default: 2
 - Example:
   # sysctl vm.pghot_freq_threshold=1
@@ -68,7 +68,7 @@ Path: /proc/sys/vm/pghot_promote_freq_window_ms
 - Controls the time window (in ms) for counting access frequency. A page is
   considered hot only when **pghot_freq_threshold** number of accesses occur
   with this time period.
-- Default: 3000 (3 seconds)
+- Default: 3000 (3 seconds) in default mode and 5000 (5s) in precision mode.
 - Example:
   # sysctl vm.pghot_promote_freq_window_ms=3000
 
include/linux/pghot.h +38 -1
index 7b85d717f410..313e5a6973e4 100644
@@ -37,8 +37,40 @@ DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
 
 #define PGHOT_DEFAULT_NODE		0
 
+#if defined(CONFIG_PGHOT_PRECISE)
 #define PGHOT_FREQ_WINDOW_MIN		(1 * MSEC_PER_SEC)
-#define PGHOT_FREQ_WINDOW_DEFAULT	(3 * MSEC_PER_SEC)
+#define PGHOT_FREQ_WINDOW_DEFAULT	(5 * MSEC_PER_SEC)
+
+/*
+ * Bits 0-26 are used to store nid, frequency and time.
+ * Bits 27-30 are unused now.
+ * Bit 31 is used to indicate the page is ready for migration.
+ */
+#define PGHOT_MIGRATE_READY		31
+
+#define PGHOT_NID_WIDTH			10
+#define PGHOT_FREQ_WIDTH		3
+/* time is stored in 14 bits which can represent up to 16s with HZ=1000 */
+#define PGHOT_TIME_WIDTH		14
+
+#define PGHOT_NID_SHIFT			0
+#define PGHOT_FREQ_SHIFT		(PGHOT_NID_SHIFT + PGHOT_NID_WIDTH)
+#define PGHOT_TIME_SHIFT		(PGHOT_FREQ_SHIFT + PGHOT_FREQ_WIDTH)
+
+#define PGHOT_NID_MASK			GENMASK(PGHOT_NID_WIDTH - 1, 0)
+#define PGHOT_FREQ_MASK			GENMASK(PGHOT_FREQ_WIDTH - 1, 0)
+#define PGHOT_TIME_MASK			GENMASK(PGHOT_TIME_WIDTH - 1, 0)
+
+#define PGHOT_NID_MAX			((1 << PGHOT_NID_WIDTH) - 1)
+#define PGHOT_FREQ_MAX			((1 << PGHOT_FREQ_WIDTH) - 1)
+#define PGHOT_TIME_MAX			((1 << PGHOT_TIME_WIDTH) - 1)
+#define PGHOT_FREQ_WINDOW_MAX		PGHOT_TIME_MAX
+
+typedef u32 phi_t;
+
+#else	/* !CONFIG_PGHOT_PRECISE */
+#define PGHOT_FREQ_WINDOW_MIN		(1 * MSEC_PER_SEC)
+#define PGHOT_FREQ_WINDOW_DEFAULT       (3 * MSEC_PER_SEC)
 
 /*
  * Bits 0-6 are used to store frequency and time.
@@ -66,6 +98,11 @@ DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
 
 typedef u8 phi_t;
 
+static_assert(MAX_NUMNODES <= (1 << PGHOT_NID_WIDTH),
+	      "pghot precise nid field too narrow for MAX_NUMNODES");
+
+#endif /* CONFIG_PGHOT_PRECISE */
+
 #define PGHOT_RECORD_SIZE		sizeof(phi_t)
 
 #define PGHOT_SECTION_HOT_BIT		0
mm/Kconfig +11 -0
index 0a5bcd5d45ed..955a826ecfe9 100644
@@ -1523,6 +1523,17 @@ config PGHOT
 	  This adds 1 byte of metadata overhead per page in lower-tier
 	  memory nodes.
 
+config PGHOT_PRECISE
+	bool "Hot page tracking precision mode"
+	default n
+	depends on PGHOT
+	help
+	  Enables precision mode for tracking hot pages with pghot sub-system.
+	  Adds fine-grained access time tracking and explicit toptier target
+	  NID tracking. Precise hot page tracking comes at the cost of using
+	  4 bytes per page against the default one byte per page. Preferable
+	  to enable this on systems with multiple nodes in toptier.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +6 -1
index 4939a1a74c1d..83bac64cab26 100644
@@ -147,4 +147,9 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
-obj-$(CONFIG_PGHOT) += pghot.o pghot-default.o
+obj-$(CONFIG_PGHOT) += pghot.o
+ifdef CONFIG_PGHOT_PRECISE
+obj-$(CONFIG_PGHOT) += pghot-precise.o
+else
+obj-$(CONFIG_PGHOT) += pghot-default.o
+endif
mm/pghot-precise.c new file +77 -0

77 lines not on this page · raw message

mm/pghot.c +9 -4
index 19cc023091f4..f66baacb2862 100644
@@ -10,6 +10,9 @@
  * the frequency of access and last access time. Promotions are done
  * to a default toptier NID.
  *
+ * In the precision mode, 4 bytes are used to store the frequency
+ * of access, last access time and the accessing NID.
+ *
  * A kernel thread named kmigrated is provided to migrate or promote
  * the hot pages. kmigrated runs for each lower tier node. It iterates
  * over the node's PFNs and  migrates pages marked for migration into
@@ -158,13 +161,15 @@ static bool kmigrated_started __ro_after_init;
  * for the purpose of tracking page hotness and subsequent promotion.
  *
  * @pfn: PFN of the page
- * @nid: Unused
+ * @nid: Target NID to where the page needs to be migrated in precision
+ *       mode but unused in default mode
  * @src: The identifier of the sub-system that reports the access
  * @now: Access time in jiffies
  *
- * Updates the frequency and time of access and marks the page as
- * ready for migration if the frequency crosses a threshold. The pages
- * marked for migration are migrated by kmigrated kernel thread.
+ * Updates the NID (in precision mode only), frequency and time of access
+ * and marks the page as ready for migration if the frequency crosses a
+ * threshold. The pages marked for migration are migrated by kmigrated
+ * kernel thread.
  *
  * Return: 0 on success and -EINVAL on failure to record the access.
  */
Signature · 1 line
2.34.1
#11 lore raw

Re: [PATCH v8 4/8] mm: pghot: Precision mode for pghot

[Reply to Shashiko review]

Quoted 96 lines · Bharata B Rao

Default pghot stores hotness in a 1‑byte record per PFN, limiting frequency to 2 bits, time to a 5‑bit bucket, and preventing storage of per‑PFN toptier NID. This restricts time granularity and forces all promotions to use the global pghot_target_nid.

This patch adds an optional precision mode (CONFIG_PGHOT_PRECISE) that expands the hotness record to 4 bytes (u32) and provides:

- 10‑bit NID field for per‑PFN promotion target,
- 3‑bit frequency field (freq_threshold range 1–7),
- 14‑bit time field offering finer recency tracking,
- MSB migrate‑ready bit.

Precision mode improves placement accuracy on systems with multiple toptier nodes and provides higher‑resolution hotness tracking, at the cost of increasing metadata to 4 bytes per PFN.

Documentation, tunables, and the record layout are updated accordingly.

---
 Documentation/admin-guide/mm/pghot.rst |  4 +-
 include/linux/pghot.h                  | 39 ++++++++++++-
 mm/Kconfig                             | 11 ++++
 mm/Makefile                            |  7 ++-
 mm/pghot-precise.c                     | 77 ++++++++++++++++++++++++++
 mm/pghot.c                             | 13 +++--
 6 files changed, 143 insertions(+), 8 deletions(-)
 create mode 100644 mm/pghot-precise.c
Documentation/admin-guide/mm/pghot.rst +2 -2
index 0edbe0082816..6d9b3d9522d1 100644
@@ -56,7 +56,7 @@ Path: /proc/sys/vm/pghot_target_nid
 Path: /proc/sys/vm/pghot_freq_threshold
 
 - Minimum access frequency before a page is marked ready for promotion.
-  Range is 1 to 3 in default mode.
+  Range is 1 to 3 in default mode and 1 to 7 in precision mode.
 - Default: 2
 - Example:
   # sysctl vm.pghot_freq_threshold=1
@@ -68,7 +68,7 @@ Path: /proc/sys/vm/pghot_promote_freq_window_ms
 - Controls the time window (in ms) for counting access frequency. A page is
   considered hot only when **pghot_freq_threshold** number of accesses occur
   with this time period.
-- Default: 3000 (3 seconds)
+- Default: 3000 (3 seconds) in default mode and 5000 (5s) in precision mode.
 - Example:
   # sysctl vm.pghot_promote_freq_window_ms=3000
 
include/linux/pghot.h +29 -1
index 7b85d717f410..313e5a6973e4 100644
@@ -37,8 +37,40 @@ DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
 
 #define PGHOT_DEFAULT_NODE		0
 
+#if defined(CONFIG_PGHOT_PRECISE)
 #define PGHOT_FREQ_WINDOW_MIN		(1 * MSEC_PER_SEC)
-#define PGHOT_FREQ_WINDOW_DEFAULT	(3 * MSEC_PER_SEC)
+#define PGHOT_FREQ_WINDOW_DEFAULT	(5 * MSEC_PER_SEC)
+
+/*
+ * Bits 0-26 are used to store nid, frequency and time.
+ * Bits 27-30 are unused now.
+ * Bit 31 is used to indicate the page is ready for migration.
+ */
+#define PGHOT_MIGRATE_READY		31
+
+#define PGHOT_NID_WIDTH			10
+#define PGHOT_FREQ_WIDTH		3
+/* time is stored in 14 bits which can represent up to 16s with HZ=1000 */
+#define PGHOT_TIME_WIDTH		14
+
+#define PGHOT_NID_SHIFT			0
+#define PGHOT_FREQ_SHIFT		(PGHOT_NID_SHIFT + PGHOT_NID_WIDTH)
+#define PGHOT_TIME_SHIFT		(PGHOT_FREQ_SHIFT + PGHOT_FREQ_WIDTH)
+
+#define PGHOT_NID_MASK			GENMASK(PGHOT_NID_WIDTH - 1, 0)
+#define PGHOT_FREQ_MASK			GENMASK(PGHOT_FREQ_WIDTH - 1, 0)
+#define PGHOT_TIME_MASK			GENMASK(PGHOT_TIME_WIDTH - 1, 0)
+
+#define PGHOT_NID_MAX			((1 << PGHOT_NID_WIDTH) - 1)
+#define PGHOT_FREQ_MAX			((1 << PGHOT_FREQ_WIDTH) - 1)
+#define PGHOT_TIME_MAX			((1 << PGHOT_TIME_WIDTH) - 1)
+#define PGHOT_FREQ_WINDOW_MAX		PGHOT_TIME_MAX
+
+typedef u32 phi_t;
 

This is a pre-existing issue, but expanding the size of phi_t to 4 bytes significantly worsens the reliability of hot_map allocations. In mm/pghot.c, pghot_alloc_hot_map() allocates the tracking structure:
hot_map = kzalloc_node(struct_size(hot_map, phi, PAGES_PER_SECTION),

Quoted 8 lines
                          GFP_KERNEL, nid);

With 4 bytes per PFN, the allocation size for a standard section grows to over 131KB, requiring an order-6 contiguous physical allocation. Since high-order GFP_KERNEL allocations frequently fail under memory fragmentation, could this cause memory hotplug operations to silently fail to track hot pages for newly onlined memory sections?
Would it be better to convert this to use kvzalloc_node() and kvfree_rcu() to avoid the contiguous physical memory requirement?

Agreed. Will switch to kvzalloc_node() and kvfree_rcu().

Quoted 14 lines
include/linux/pghot.h +6 -0
+
+#else	/* !CONFIG_PGHOT_PRECISE */
+#define PGHOT_FREQ_WINDOW_MIN		(1 * MSEC_PER_SEC)
+#define PGHOT_FREQ_WINDOW_DEFAULT       (3 * MSEC_PER_SEC)
 
 /*
  * Bits 0-6 are used to store frequency and time.
@@ -66,6 +98,11 @@ DECLARE_STATIC_KEY_FALSE(pghot_src_hwhints);
 
 typedef u8 phi_t;
 
+static_assert(MAX_NUMNODES <= (1 << PGHOT_NID_WIDTH),
+	      "pghot precise nid field too narrow for MAX_NUMNODES");
 
Quoted 7 lines

Is this static_assert placed in the correct block?
It appears to be inside the #else block for CONFIG_PGHOT_PRECISE, which means it will be disabled precisely when precision mode is enabled.
If a kernel is compiled with CONFIG_PGHOT_PRECISE=y and MAX_NUMNODES > 1024, the compiler won't catch the narrow NID field. Could this lead to NIDs being silently truncated during runtime, causing pages to be migrated to incorrect NUMA nodes?

Yes, I got the placement wrong, it should have been within CONFIG_PGHOT_PRECISE.

Quoted 93 lines
include/linux/pghot.h +3 -0
+
+#endif /* CONFIG_PGHOT_PRECISE */
+
 #define PGHOT_RECORD_SIZE		sizeof(phi_t)
 
 #define PGHOT_SECTION_HOT_BIT		0
mm/Kconfig +11 -0
index 0a5bcd5d45ed..955a826ecfe9 100644
@@ -1523,6 +1523,17 @@ config PGHOT
 	  This adds 1 byte of metadata overhead per page in lower-tier
 	  memory nodes.
 
+config PGHOT_PRECISE
+	bool "Hot page tracking precision mode"
+	default n
+	depends on PGHOT
+	help
+	  Enables precision mode for tracking hot pages with pghot sub-system.
+	  Adds fine-grained access time tracking and explicit toptier target
+	  NID tracking. Precise hot page tracking comes at the cost of using
+	  4 bytes per page against the default one byte per page. Preferable
+	  to enable this on systems with multiple nodes in toptier.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/Makefile +6 -1
index 4939a1a74c1d..83bac64cab26 100644
@@ -147,4 +147,9 @@ obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o
 obj-$(CONFIG_EXECMEM) += execmem.o
 obj-$(CONFIG_TMPFS_QUOTA) += shmem_quota.o
 obj-$(CONFIG_LAZY_MMU_MODE_KUNIT_TEST) += tests/lazy_mmu_mode_kunit.o
-obj-$(CONFIG_PGHOT) += pghot.o pghot-default.o
+obj-$(CONFIG_PGHOT) += pghot.o
+ifdef CONFIG_PGHOT_PRECISE
+obj-$(CONFIG_PGHOT) += pghot-precise.o
+else
+obj-$(CONFIG_PGHOT) += pghot-default.o
+endif
mm/pghot-precise.c new file +43 -0
new file mode 100644
index 000000000000..df82f2068a99
@@ -0,0 +1,77 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * pghot: Precision mode
+ *
+ * 4 byte hotness record per PFN (u32)
+ * NID, time and frequency tracked as part of the record.
+ */
+
+#include <linux/pghot.h>
+#include <linux/jiffies.h>
+#include <linux/memory-tiers.h>
+
+bool pghot_nid_valid(int nid)
+{
+	if (nid != NUMA_NO_NODE &&
+	    (!numa_valid_node(nid) || nid > PGHOT_NID_MAX ||
+	     !node_online(nid) || !node_is_toptier(nid)))
+		return false;
+
+	return true;
+}
+
+unsigned long pghot_access_latency(unsigned long old_time, unsigned long time)
+{
+	return jiffies_to_msecs((time - old_time) & PGHOT_TIME_MASK);
+}
+
+bool pghot_update_record(phi_t *phi, int nid, unsigned long now)
+{
+	phi_t freq, old_freq, hotness, old_hotness, old_time;
+	phi_t time = now & PGHOT_TIME_MASK;
+
+	nid = (nid == NUMA_NO_NODE) ? sysctl_pghot_target_nid : nid;
+	old_hotness = READ_ONCE(*phi);
+
+	do {
+		bool new_window = false;
+
+		old_freq = (old_hotness >> PGHOT_FREQ_SHIFT) & PGHOT_FREQ_MASK;
+		old_time = (old_hotness >> PGHOT_TIME_SHIFT) & PGHOT_TIME_MASK;
+
+		if (pghot_access_latency(old_time, time) > sysctl_pghot_freq_window)
+			new_window = true;
Quoted 8 lines

Can the 14-bit time mask cause cold pages to be incorrectly evaluated as hot? Since PGHOT_TIME_MASK is 14 bits, the tracked time value wraps around roughly every 16.38 seconds (assuming HZ=1000).
If a cold page is accessed once, left completely idle for a long period, and then accessed again, is it possible that the elapsed time calculated using the wrapped mask randomly falls within the default 5-second sysctl window? Could this lead to spurious hotness promotions and unnecessary memory migrations for pages that aren't actually accessed frequently?

For the reasons explained in the similar path in pghot-default, this is okay. This is in fact better than pghot-default as 14bits give a better range of ~16s.

Regards,
Bharata.

#12 lore raw

PATCH v8 5/8 mm: sched: move NUMA balancing tiering promotion to pghot

In reply to Bharata B Rao

Currently hot page promotion (NUMA_BALANCING_MEMORY_TIERING mode of NUMA Balancing) does hot page detection (via hint faults), hot page classification and eventual promotion, all by itself and sits within the scheduler.

With pghot, the new hot page tracking and promotion mechanism being available, NUMA Balancing can limit itself to detection of hot pages (via hint faults) and off-load rest of the functionality to pghot.

To achieve this, pghot_record_access(PGHOT_HINTFAULTS) API is used to feed the hot page info to pghot. In addition, the migration rate limiting and dynamic threshold logic are moved to kmigrated so that the same can be used for hot pages reported by other sources too. Hence it becomes necessary to introduce a new config option CONFIG_NUMA_BALANCING_TIERING to control the hint faults source for hot page promotion. This option controls the NUMA_BALANCING_MEMORY_TIERING mode of kernel.numa_balancing

This movement of hot page promotion to pghot results in the following changes to the behaviour of hint faults based hot page promotion:

1. Promotion is no longer done in the fault path but instead is deferred to kmigrated and happens in batches.
2. NUMA_BALANCING_MEMORY_TIERING mode used to promote on first access. Pghot by default, promotes on second access though this can be changed by setting /proc/sys/vm/pghot_freq_threshold tunable.
3. hot_threshold_ms debugfs tunable now gets replaced by /proc/sys/vm/pghot_promote_freq_window_ms with default value of 1000ms changed to 3000ms in pghot-default and 5000ms pghot-precise.
4. In NUMA_BALANCING_MEMORY_TIERING mode, hint fault latency is the difference between the PTE update time (during scanning) and the access time (hint fault). However with pghot, a single latency threshold is used for two purposes:
a) If the time difference between successive accesses are within the threshold, the page is marked as hot.
b) Later when kmigrated picks up the page for migration, it will migrate only if the difference between the current time and the time when the page was marked hot is with the threshold.
5. The max scan period which is used in dynamic threshold logic was a debugfs tunable (sysctl_numa_balancing_scan_period_max). However this has been converted to a scalar metric in pghot.
6. In the uncommon case of using NUMA_BALANCING_NORMAL mode to balance between lower and higher tier nodes, we end up waking the kswapd when there is no headroom in the toptier.

Key code changes due to this movement are detailed below to help easy understanding of the restructuring.

1. Scanning and access times are no longer tracked in last_cpupid field of folio flags. Hence all code related to this (like folio_xchg_access_time(), cpupid_valid()) are removed.
2. The misplaced migration routines become conditional to CONFIG_PGHOT in addition to CONFIG_NUMA_BALANCING.
3. The promotion related stats (like PGPROMOTE_SUCCESS etc) are now moved to under CONFIG_PGHOT as these stats are part of promotion engine which will be used for other hotness sources as well.
4. Routines that are responsibile for migration rate limiting dynamic thresholding, pgdat balancing during promotion etc are moved to pghot with appropriate renaming.

---
 Documentation/admin-guide/mm/pghot.rst |  23 +++-
 include/linux/mm.h                     |  35 +----
 include/linux/mmzone.h                 |   4 +-
 init/Kconfig                           |  14 ++
 kernel/sched/core.c                    |   7 +
 kernel/sched/debug.c                   |   1 -
 kernel/sched/fair.c                    | 171 +------------------------
 kernel/sched/sched.h                   |   1 -
 mm/Kconfig                             |   2 +-
 mm/huge_memory.c                       |  32 ++++-
 mm/memcontrol.c                        |   6 +-
 mm/memory-tiers.c                      |  15 ++-
 mm/memory.c                            |  36 +++++-
 mm/mempolicy.c                         |   3 -
 mm/migrate.c                           |  16 ++-
 mm/pghot.c                             | 163 ++++++++++++++++++++++-
 mm/vmstat.c                            |   2 +-
 17 files changed, 310 insertions(+), 221 deletions(-)
Documentation/admin-guide/mm/pghot.rst +22 -1
index 6d9b3d9522d1..accf78d95d02 100644
@@ -35,7 +35,11 @@ Path: /proc/sys/vm/pghot_enabled_sources
 - Bits:
   - 0: Hint faults (value 0x1)
   - 1: Hardware hints (value 0x2)
-- Default: 0 (disabled)
+- Default: 0x1 (hint faults enabled, hardware hints disabled). Hint faults
+  are enabled by default so that selecting the NUMA Balancing memory
+  tiering mode (kernel.numa_balancing=2) promotes hot pages without any
+  additional opt-in. Hint faults are fed to pghot only while tiering mode
+  is selected, so this default has no effect otherwise.
 - Example:
   # echo 0x3 > /proc/sys/vm/pghot_enabled_sources
   Enables both hint faults and hwhints sources
@@ -87,3 +91,20 @@ Path: /proc/vmstat
 
 3. **pghot_recorded_hwhints**
    - Number of recorded accesses reported by hwhints source.
+
+NUMA Hint Faults Source
+=======================
+The "hint faults" source is the tiering mode of NUMA Balancing
+which acts as a source of page hotness to pghot.
+
+It is controlled by the NUMA_BALANCING_TIERING config option, which
+gates the memory tiering mode (NUMA_BALANCING_MEMORY_TIERING) of NUMA
+Balancing. At runtime that mode must additionally be selected through
+the kernel.numa_balancing sysctl.
+
+This source is enabled by default through the hint faults bit (0x1) of
+**pghot_enabled_sources**, so selecting the tiering mode alone is enough
+to promote hot pages. It can be disabled at runtime by clearing that bit:
+
+# echo 0x0 > /proc/sys/vm/pghot_enabled_sources
+
include/linux/mm.h +7 -28
index 485df9c2dbdd..1fdb0e143c65 100644
@@ -2305,17 +2305,6 @@ static inline int folio_nid(const struct folio *folio)
 }
 
 #ifdef CONFIG_NUMA_BALANCING
-/* page access time bits needs to hold at least 4 seconds */
-#define PAGE_ACCESS_TIME_MIN_BITS	12
-#if LAST_CPUPID_SHIFT < PAGE_ACCESS_TIME_MIN_BITS
-#define PAGE_ACCESS_TIME_BUCKETS				\
-	(PAGE_ACCESS_TIME_MIN_BITS - LAST_CPUPID_SHIFT)
-#else
-#define PAGE_ACCESS_TIME_BUCKETS	0
-#endif
-
-#define PAGE_ACCESS_TIME_MASK				\
-	(LAST_CPUPID_MASK << PAGE_ACCESS_TIME_BUCKETS)
 
 static inline int cpu_pid_to_cpupid(int cpu, int pid)
 {
@@ -2381,15 +2370,6 @@ static inline void page_cpupid_reset_last(struct page *page)
 }
 #endif /* LAST_CPUPID_NOT_IN_PAGE_FLAGS */
 
-static inline int folio_xchg_access_time(struct folio *folio, int time)
-{
-	int last_time;
-
-	last_time = folio_xchg_last_cpupid(folio,
-					   time >> PAGE_ACCESS_TIME_BUCKETS);
-	return last_time << PAGE_ACCESS_TIME_BUCKETS;
-}
-
 static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 {
 	unsigned int pid_bit;
@@ -2400,18 +2380,12 @@ static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 	}
 }
 
-bool folio_use_access_time(struct folio *folio);
 #else /* !CONFIG_NUMA_BALANCING */
 static inline int folio_xchg_last_cpupid(struct folio *folio, int cpupid)
 {
 	return folio_nid(folio); /* XXX */
 }
 
-static inline int folio_xchg_access_time(struct folio *folio, int time)
-{
-	return 0;
-}
-
 static inline int folio_last_cpupid(struct folio *folio)
 {
 	return folio_nid(folio); /* XXX */
@@ -2454,11 +2428,16 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid)
 static inline void vma_set_access_pid_bit(struct vm_area_struct *vma)
 {
 }
-static inline bool folio_use_access_time(struct folio *folio)
+#endif /* CONFIG_NUMA_BALANCING */
+
+#ifdef CONFIG_NUMA_BALANCING_TIERING
+bool folio_is_promo_candidate(struct folio *folio);
+#else
+static inline bool folio_is_promo_candidate(struct folio *folio)
 {
 	return false;
 }
-#endif /* CONFIG_NUMA_BALANCING */
+#endif /* CONFIG_NUMA_BALANCING_TIERING */
 
 #if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS)
 
include/linux/mmzone.h +2 -2
index 11c547a51f43..c5a9237eedd8 100644
@@ -288,7 +288,7 @@ enum node_stat_item {
 #ifdef CONFIG_SWAP
 	NR_SWAPCACHE,
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	PGPROMOTE_SUCCESS,	/* promote successfully */
 	/**
 	 * Candidate pages for promotion based on hint fault latency.  This
@@ -1555,7 +1555,7 @@ typedef struct pglist_data {
 	unsigned long first_deferred_pfn;
 #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
 
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	/* start time in ms of current promote rate limit period */
 	unsigned int nbp_rl_start;
 	/* number of promote candidate pages at start time of current rate limit period */
init/Kconfig +14 -0
index 5230d4879b1c..75f5a0cb9065 100644
@@ -1038,6 +1038,20 @@ config NUMA_BALANCING_DEFAULT_ENABLED
 	  If set, automatic NUMA balancing will be enabled if running on a NUMA
 	  machine.
 
+config NUMA_BALANCING_TIERING
+	bool "NUMA balancing memory tiering promotion"
+	depends on NUMA_BALANCING && PGHOT
+	default y
+	help
+	  Enable NUMA balancing mode 2 (memory tiering). This allows
+	  automatic promotion of hot pages from slower memory tiers to
+	  faster tiers using the pghot subsystem.
+
+	  This requires CONFIG_PGHOT for the hot page tracking engine.
+	  This option is required for kernel.numa_balancing=2.
+
+	  If unsure, say N.
+
 config SLAB_OBJ_EXT
 	bool
 
kernel/sched/core.c +7 -0
index 96226707c2f6..10863460fed2 100644
@@ -4639,6 +4639,7 @@ void set_numabalancing_state(bool enabled)
 }
 
 #ifdef CONFIG_PROC_SYSCTL
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 static void reset_memory_tiering(void)
 {
 	struct pglist_data *pgdat;
@@ -4649,6 +4650,7 @@ static void reset_memory_tiering(void)
 		pgdat->nbp_th_start = jiffies_to_msecs(jiffies);
 	}
 }
+#endif
 
 static int sysctl_numa_balancing(const struct ctl_table *table, int write,
 			  void *buffer, size_t *lenp, loff_t *ppos)
@@ -4666,9 +4668,14 @@ static int sysctl_numa_balancing(const struct ctl_table *table, int write,
 	if (err < 0)
 		return err;
 	if (write) {
+		if ((state & NUMA_BALANCING_MEMORY_TIERING) &&
+		    !IS_ENABLED(CONFIG_NUMA_BALANCING_TIERING))
+			return -EOPNOTSUPP;
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 		    (state & NUMA_BALANCING_MEMORY_TIERING))
 			reset_memory_tiering();
+#endif
 		sysctl_numa_balancing_mode = state;
 		__set_numabalancing_state(state);
 	}
kernel/sched/debug.c +0 -1
index 40584b27ea0c..7918de84abcf 100644
@@ -665,7 +665,6 @@ static __init int sched_init_debug(void)
 	debugfs_create_u32("scan_period_min_ms", 0644, numa, &sysctl_numa_balancing_scan_period_min);
 	debugfs_create_u32("scan_period_max_ms", 0644, numa, &sysctl_numa_balancing_scan_period_max);
 	debugfs_create_u32("scan_size_mb", 0644, numa, &sysctl_numa_balancing_scan_size);
-	debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold);
 #endif /* CONFIG_NUMA_BALANCING */
 
 #ifdef CONFIG_SCHED_CACHE
kernel/sched/fair.c +6 -165

211 lines not on this page · raw message

kernel/sched/sched.h +0 -1
index 56acf502ba26..84b94f143a23 100644
@@ -3120,7 +3120,6 @@ extern unsigned int sysctl_numa_balancing_scan_delay;
 extern unsigned int sysctl_numa_balancing_scan_period_min;
 extern unsigned int sysctl_numa_balancing_scan_period_max;
 extern unsigned int sysctl_numa_balancing_scan_size;
-extern unsigned int sysctl_numa_balancing_hot_threshold;
 
 #ifdef CONFIG_SCHED_HRTICK
 
mm/Kconfig +1 -1
index 955a826ecfe9..61ab18f4aebb 100644
@@ -1511,7 +1511,7 @@ config LAZY_MMU_MODE_KUNIT_TEST
 
 config PGHOT
 	bool "Hot page tracking and promotion"
-	default n
+	default y if NUMA_BALANCING
 	depends on NUMA_MIGRATION && SPARSEMEM
 	help
 	  A sub-system to track page accesses in lower tier memory and
mm/huge_memory.c +30 -2
index 2bccb0a53a0a..c36092ebca42 100644
@@ -41,6 +41,7 @@
 #include <linux/pgalloc.h>
 #include <linux/pgalloc_tag.h>
 #include <linux/pagewalk.h>
+#include <linux/pghot.h>
 
 #include <asm/tlb.h>
 #include "internal.h"
@@ -2205,7 +2206,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	int nid = NUMA_NO_NODE;
 	int target_nid, last_cpupid;
 	pmd_t pmd, old_pmd;
-	bool writable = false;
+	bool writable = false, needs_promotion = false;
 	int flags = 0;
 
 	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
@@ -2232,11 +2233,29 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 
 	target_nid = numa_migrate_check(folio, vmf, haddr, &flags, writable,
 					&last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
+		 *
+		 * Isolation and migration are handled by pghot. Since VMA
+		 * won't be available to kmigrated which does batched migration
+		 * from non-process context, filter shared EXEC pages here itself.
+		 */
+		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+			nid = NUMA_NO_NODE;
+		else
+			nid = target_nid;
+		goto out_map;
+	}
+
+	/* Balancing b/n toptier nodes, mode=NUMA_BALANCING_NORMAL */
 	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
 		flags |= TNF_MIGRATE_FAIL;
 		goto out_map;
@@ -2268,8 +2287,17 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	update_mmu_cache_pmd(vma, vmf->address, vmf->pmd);
 	spin_unlock(vmf->ptl);
 
-	if (nid != NUMA_NO_NODE)
+	if (nid != NUMA_NO_NODE) {
+		if (needs_promotion)
+			pghot_record_access(folio_pfn(folio), nid,
+					    PGHOT_HINTFAULTS, jiffies);
+		/*
+		 * Even when the hint fault is handed off to pghot for
+		 * promotion, keep feeding NUMA Balancing per-task fault
+		 * statistics for scan-period adjustment.
+		 */
 		task_numa_fault(last_cpupid, nid, HPAGE_PMD_NR, flags);
+	}
 	return 0;
 }
 
mm/memcontrol.c +3 -3
index 6dc4888a90f3..295f41f1ee16 100644
@@ -400,7 +400,7 @@ static const unsigned int memcg_node_stat_items[] = {
 #ifdef CONFIG_SWAP
 	NR_SWAPCACHE,
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	PGPROMOTE_SUCCESS,
 #endif
 	PGDEMOTE_KSWAPD,
@@ -1603,7 +1603,7 @@ static const struct memory_stat memory_stats[] = {
 	{ "pgscan_khugepaged",		PGSCAN_KHUGEPAGED	},
 	{ "pgscan_proactive",		PGSCAN_PROACTIVE	},
 	{ "pgrefill",			PGREFILL		},
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	{ "pgpromote_success",		PGPROMOTE_SUCCESS	},
 #endif
 };
@@ -1655,7 +1655,7 @@ static int memcg_page_state_output_unit(int item)
 	case PGSCAN_KHUGEPAGED:
 	case PGSCAN_PROACTIVE:
 	case PGREFILL:
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	case PGPROMOTE_SUCCESS:
 #endif
 		return 1;
mm/memory-tiers.c +8 -7
index 54851d8a195b..be134a32f5bf 100644
@@ -51,18 +51,19 @@ static const struct bus_type memory_tier_subsys = {
 	.dev_name = "memory_tier",
 };
 
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_NUMA_BALANCING_TIERING
 /**
- * folio_use_access_time - check if a folio reuses cpupid for page access time
+ * folio_is_promo_candidate - check if the folio qualifies for promotion
+ *
  * @folio: folio to check
  *
- * folio's _last_cpupid field is repurposed by memory tiering. In memory
- * tiering mode, cpupid of slow memory folio (not toptier memory) is used to
- * record page access time.
+ * Checks if NUMA Balancing tiering mode is set and the folio belongs
+ * to lower tier. If so, it qualifies for promotion to toptier when
+ * it is categorized as hot.
  *
- * Return: the folio _last_cpupid is used to record page access time
+ * Return: True if the above condition is met, else False.
  */
-bool folio_use_access_time(struct folio *folio)
+bool folio_is_promo_candidate(struct folio *folio)
 {
 	return (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
 	       !node_is_toptier(folio_nid(folio));
mm/memory.c +32 -4
index ff338c2abe92..91daba4c2bc9 100644
@@ -75,6 +75,7 @@
 #include <linux/perf_event.h>
 #include <linux/ptrace.h>
 #include <linux/vmalloc.h>
+#include <linux/pghot.h>
 #include <linux/sched/sysctl.h>
 #include <linux/pgalloc.h>
 #include <linux/uaccess.h>
@@ -6029,10 +6030,9 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,
 	if (folio_maybe_mapped_shared(folio) && (vma->vm_flags & VM_SHARED))
 		*flags |= TNF_SHARED;
 	/*
-	 * For memory tiering mode, cpupid of slow memory page is used
-	 * to record page access time.  So use default value.
+	 * For memory tiering mode, last_cpupid is unused. So use default value.
 	 */
-	if (folio_use_access_time(folio))
+	if (folio_is_promo_candidate(folio))
 		*last_cpupid = (-1 & LAST_CPUPID_MASK);
 	else
 		*last_cpupid = folio_last_cpupid(folio);
@@ -6113,6 +6113,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	int nid = NUMA_NO_NODE;
 	bool writable = false, ignore_writable = false;
 	bool pte_write_upgrade = vma_wants_manual_pte_write_upgrade(vma);
+	bool needs_promotion = false;
 	int last_cpupid;
 	int target_nid;
 	pte_t pte, old_pte;
@@ -6147,12 +6148,30 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 	nr_pages = folio_nr_pages(folio);
 
 	target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
 					writable, &last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
+		 *
+		 * Isolation and migration are handled by pghot. Since VMA
+		 * won't be available to kmigrated which does batched migration
+		 * from non-process context, filter shared EXEC pages here itself.
+		 */
+		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+			nid = NUMA_NO_NODE;
+		else
+			nid = target_nid;
+		goto out_map;
+	}
+
+	/* Balancing b/n toptier nodes, mode=NUMA_BALANCING_NORMAL */
 	if (migrate_misplaced_folio_prepare(folio, vma, target_nid)) {
 		flags |= TNF_MIGRATE_FAIL;
 		goto out_map;
@@ -6192,8 +6211,17 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 					    writable);
 	pte_unmap_unlock(vmf->pte, vmf->ptl);
 
-	if (nid != NUMA_NO_NODE)
+	if (nid != NUMA_NO_NODE) {
+		if (needs_promotion)
+			pghot_record_access(folio_pfn(folio), nid,
+					    PGHOT_HINTFAULTS, jiffies);
+		/*
+		 * Even when the hint fault is handed off to pghot for
+		 * promotion, keep feeding NUMA Balancing per-task fault
+		 * statistics for scan-period adjustment.
+		 */
 		task_numa_fault(last_cpupid, nid, nr_pages, flags);
+	}
 	return 0;
 }
 
mm/mempolicy.c +0 -3
index 36699fabd3c2..111ac78c2b23 100644
@@ -872,9 +872,6 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 	    node_is_toptier(nid))
 		return false;
 
-	if (folio_use_access_time(folio))
-		folio_xchg_access_time(folio, jiffies_to_msecs(jiffies));
-
 	return true;
 }
 
mm/migrate.c +15 -1
index 7ab15780226e..4b97d6997f4b 100644
@@ -2712,8 +2712,18 @@ int migrate_misplaced_folio_prepare(struct folio *folio,
 	if (!migrate_balanced_pgdat(pgdat, nr_pages)) {
 		int z;
 
-		if (!(sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING))
+		/*
+		 * Kswapd wakeup for creating headroom in toptier is done only
+		 * for hot page promotion case and not for misplaced migrations
+		 * between toptier nodes.
+		 *
+		 * In the uncommon case of using NUMA_BALANCING_NORMAL mode
+		 * to balance between lower and higher tier nodes, we end up
+		 * waking the kswapd.
+		 */
+		if (node_is_toptier(folio_nid(folio)))
 			return -EAGAIN;
+
 		for (z = pgdat->nr_zones - 1; z >= 0; z--) {
 			if (managed_zone(pgdat->node_zones + z))
 				break;
@@ -2763,6 +2773,8 @@ int migrate_misplaced_folio(struct folio *folio, int node)
 #ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+#endif
+#ifdef CONFIG_PGHOT
 		if ((sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING)
 		    && !node_is_toptier(folio_nid(folio))
 		    && node_is_toptier(node)) {
@@ -2828,6 +2840,8 @@ int promote_misplaced_memcg_folios(struct list_head *folio_list, int node)
 #ifdef CONFIG_NUMA_BALANCING
 		count_vm_numa_events(NUMA_PAGE_MIGRATE, nr_succeeded);
 		count_memcg_events(memcg, NUMA_PAGE_MIGRATE, nr_succeeded);
+#endif
+#ifdef CONFIG_PGHOT
 		mod_lruvec_state(mem_cgroup_lruvec(memcg, NODE_DATA(node)),
 				 PGPROMOTE_SUCCESS, nr_succeeded);
 #endif
mm/pghot.c +162 -1

214 lines not on this page · raw message

mm/vmstat.c +1 -1
index 4064ead568cc..da668ff05032 100644
@@ -1268,7 +1268,7 @@ const char * const vmstat_text[] = {
 #ifdef CONFIG_SWAP
 	[I(NR_SWAPCACHE)]			= "nr_swapcached",
 #endif
-#ifdef CONFIG_NUMA_BALANCING
+#ifdef CONFIG_PGHOT
 	[I(PGPROMOTE_SUCCESS)]			= "pgpromote_success",
 	[I(PGPROMOTE_CANDIDATE)]		= "pgpromote_candidate",
 	[I(PGPROMOTE_CANDIDATE_NRL)]		= "pgpromote_candidate_nrl",
Signature · 1 line
2.34.1
#13 lore raw

Re: [PATCH v8 5/8] mm: sched: move NUMA balancing tiering promotion to pghot

[Reply to Shashiko review]

Quoted 49 lines · Bharata B Rao
mm/huge_memory.c +18 -1
index 2bccb0a53a0a..c36092ebca42 100644
@@ -41,6 +41,7 @@
 #include <linux/pgalloc.h>
 #include <linux/pgalloc_tag.h>
 #include <linux/pagewalk.h>
+#include <linux/pghot.h>
 
 #include <asm/tlb.h>
 #include "internal.h"
@@ -2205,7 +2206,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 	int nid = NUMA_NO_NODE;
 	int target_nid, last_cpupid;
 	pmd_t pmd, old_pmd;
-	bool writable = false;
+	bool writable = false, needs_promotion = false;
 	int flags = 0;
 
 	vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd);
@@ -2232,11 +2233,29 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 
 	target_nid = numa_migrate_check(folio, vmf, haddr, &flags, writable,
 					&last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
+		 *
+		 * Isolation and migration are handled by pghot. Since VMA
+		 * won't be available to kmigrated which does batched migration
+		 * from non-process context, filter shared EXEC pages here itself.
+		 */
+		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+			nid = NUMA_NO_NODE;
+		else
+			nid = target_nid;
+		goto out_map;
+	}
 

Does this drop NUMA fault statistics for shared EXEC pages?

Quoted 2 lines

If nid is set to NUMA_NO_NODE here, the out_map path skips task_numa_fault() entirely, which seems like it would disrupt scan-period adjustment.

It does drop the call to task_numa_fault() for shared EXEC pages but then base kernel also has a memory-tiering gate that resulted in shared EXEC pages not being considered for scan period updates. So this doesn't really drop the NUMA fault stats for shared exec pages. However I can make it structurally similar to base so that it is the memory-tiering gate that rejects the scan period update for shared exec pages.

Quoted 3 lines

Additionally, for non-shared pages, nid is set to target_nid, which might prematurely account the fault to the target node before the migration actually happens.

Accounting the fault to target node is deliberate here because in pghot the promotion is delegated to kmigrated. So reporting mem_node=target_nid (top tier) is what feeds the scan period update in task_numa_fault(). Since pghot does async migration, it feeds all promo candidates rather than only the successfully migrated ones like in base.

Quoted 62 lines
mm/memory.c +20 -3
index ff338c2abe92..91daba4c2bc9 100644
@@ -75,6 +75,7 @@
 #include <linux/perf_event.h>
 #include <linux/ptrace.h>
 #include <linux/vmalloc.h>
+#include <linux/pghot.h>
 #include <linux/sched/sysctl.h>
 #include <linux/pgalloc.h>
 #include <linux/uaccess.h>
@@ -6029,10 +6030,9 @@ int numa_migrate_check(struct folio *folio, struct vm_fault *vmf,
 	if (folio_maybe_mapped_shared(folio) && (vma->vm_flags & VM_SHARED))
 		*flags |= TNF_SHARED;
 	/*
-	 * For memory tiering mode, cpupid of slow memory page is used
-	 * to record page access time.  So use default value.
+	 * For memory tiering mode, last_cpupid is unused. So use default value.
 	 */
-	if (folio_use_access_time(folio))
+	if (folio_is_promo_candidate(folio))
 		*last_cpupid = (-1 & LAST_CPUPID_MASK);
 	else
 		*last_cpupid = folio_last_cpupid(folio);
@@ -6113,6 +6113,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 	int nid = NUMA_NO_NODE;
 	bool writable = false, ignore_writable = false;
 	bool pte_write_upgrade = vma_wants_manual_pte_write_upgrade(vma);
+	bool needs_promotion = false;
 	int last_cpupid;
 	int target_nid;
 	pte_t pte, old_pte;
@@ -6147,12 +6148,30 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf)
 		goto out_map;
 
 	nid = folio_nid(folio);
+	needs_promotion = folio_is_promo_candidate(folio);
 	nr_pages = folio_nr_pages(folio);
 
 	target_nid = numa_migrate_check(folio, vmf, vmf->address, &flags,
 					writable, &last_cpupid);
 	if (target_nid == NUMA_NO_NODE)
 		goto out_map;
+
+	if (needs_promotion) {
+		/*
+		 * Hot page promotion, mode=NUMA_BALANCING_MEMORY_TIERING.
+		 *
+		 * Isolation and migration are handled by pghot. Since VMA
+		 * won't be available to kmigrated which does batched migration
+		 * from non-process context, filter shared EXEC pages here itself.
+		 */
+		if ((vma->vm_flags & VM_EXEC) && folio_maybe_mapped_shared(folio))
+			nid = NUMA_NO_NODE;
+		else
+			nid = target_nid;
+		goto out_map;
+	}
 

Is the same accounting disruption happening here in do_numa_page() as well?

Quoted 2 lines

Assigning NUMA_NO_NODE skips task_numa_fault() later in the function, and assigning target_nid credits the target node prior to the actual migration.

Reply to this same as above.

The other two comments for this patch actually belong to patch 3/8 and have been addressed there.

Regards,
Bharata.

#14 lore raw

PATCH v8 6/8 x86/ibs: Move IBS caps definitions into its own header

In reply to Bharata B Rao

Subsequent patch adds IBS Memory Profiler driver that is independent of the perf subsystem but needs the CPUID 0x8000001B capability bits. Hence move those bit definitions out of asm/perf_event.h into a dedicated header so the new driver can consume them without pulling in perf.

---
 arch/x86/include/asm/ibs-caps.h   | 85 +++++++++++++++++++++++++++++++
 arch/x86/include/asm/perf_event.h | 81 +----------------------------
 2 files changed, 86 insertions(+), 80 deletions(-)
 create mode 100644 arch/x86/include/asm/ibs-caps.h
arch/x86/include/asm/ibs-caps.h new file +85 -0
new file mode 100644
index 000000000000..ddf6c512c8f9
@@ -0,0 +1,85 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_IBS_CAPS_H
+#define _ASM_X86_IBS_CAPS_H
+
+/*
+ * IBS cpuid feature detection
+ */
+
+#define IBS_CPUID_FEATURES		0x8000001b
+
+/*
+ * Same bit mask as for IBS cpuid feature flags (Fn8000_001B_EAX), but
+ * bit 0 is used to indicate the existence of IBS.
+ */
+#define IBS_CAPS_AVAIL			(1U<<0)
+#define IBS_CAPS_FETCHSAM		(1U<<1)
+#define IBS_CAPS_OPSAM			(1U<<2)
+#define IBS_CAPS_RDWROPCNT		(1U<<3)
+#define IBS_CAPS_OPCNT			(1U<<4)
+#define IBS_CAPS_BRNTRGT		(1U<<5)
+#define IBS_CAPS_OPCNTEXT		(1U<<6)
+#define IBS_CAPS_RIPINVALIDCHK		(1U<<7)
+#define IBS_CAPS_OPBRNFUSE		(1U<<8)
+#define IBS_CAPS_FETCHCTLEXTD		(1U<<9)
+#define IBS_CAPS_OPDATA4		(1U<<10)
+#define IBS_CAPS_ZEN4			(1U<<11)
+#define IBS_CAPS_OPLDLAT		(1U<<12)
+#define IBS_CAPS_DIS			(1U<<13)
+#define IBS_CAPS_FETCHLAT		(1U<<14)
+#define IBS_CAPS_BIT63_FILTER		(1U<<15)
+#define IBS_CAPS_STRMST_RMTSOCKET	(1U<<16)
+#define IBS_CAPS_OPDTLBPGSIZE		(1U<<19)
+
+#define IBS_CAPS_DEFAULT		(IBS_CAPS_AVAIL		\
+					 | IBS_CAPS_FETCHSAM	\
+					 | IBS_CAPS_OPSAM)
+
+/*
+ * IBS APIC setup
+ */
+#define IBSCTL				0x1cc
+#define IBSCTL_LVT_OFFSET_VALID		(1ULL<<8)
+#define IBSCTL_LVT_OFFSET_MASK		0x0F
+
+/* IBS fetch bits/masks */
+#define IBS_FETCH_L3MISSONLY		      (1ULL << 59)
+#define IBS_FETCH_RAND_EN		      (1ULL << 57)
+#define IBS_FETCH_VAL			      (1ULL << 49)
+#define IBS_FETCH_ENABLE		      (1ULL << 48)
+#define IBS_FETCH_CNT			     0xFFFF0000ULL
+#define IBS_FETCH_MAX_CNT		     0x0000FFFFULL
+
+#define IBS_FETCH_2_DIS			      (1ULL <<  0)
+#define IBS_FETCH_2_FETCHLAT_FILTER	    (0xFULL <<  1)
+#define IBS_FETCH_2_FETCHLAT_FILTER_SHIFT	       (1)
+#define IBS_FETCH_2_EXCL_RIP_63_EQ_1	      (1ULL <<  5)
+#define IBS_FETCH_2_EXCL_RIP_63_EQ_0	      (1ULL <<  6)
+
+/*
+ * IBS op bits/masks
+ * The lower 7 bits of the current count are random bits
+ * preloaded by hardware and ignored in software
+ */
+#define IBS_OP_LDLAT_EN			      (1ULL << 63)
+#define IBS_OP_LDLAT_THRSH		    (0xFULL << 59)
+#define IBS_OP_LDLAT_THRSH_SHIFT		      (59)
+#define IBS_OP_CUR_CNT			(0xFFF80ULL << 32)
+#define IBS_OP_CUR_CNT_RAND		(0x0007FULL << 32)
+#define IBS_OP_CUR_CNT_EXT_MASK		   (0x7FULL << 52)
+#define IBS_OP_CNT_CTL			      (1ULL << 19)
+#define IBS_OP_VAL			      (1ULL << 18)
+#define IBS_OP_ENABLE			      (1ULL << 17)
+#define IBS_OP_L3MISSONLY		      (1ULL << 16)
+#define IBS_OP_MAX_CNT			     0x0000FFFFULL
+#define IBS_OP_MAX_CNT_EXT		     0x007FFFFFULL	/* not a register bit mask */
+#define IBS_OP_MAX_CNT_EXT_MASK		   (0x7FULL << 20)	/* separate upper 7 bits */
+#define IBS_RIP_INVALID			      (1ULL << 38)
+
+#define IBS_OP_2_DIS			      (1ULL <<  0)
+#define IBS_OP_2_EXCL_RIP_63_EQ_0	      (1ULL <<  1)
+#define IBS_OP_2_EXCL_RIP_63_EQ_1	      (1ULL <<  2)
+#define IBS_OP_2_STRM_ST_FILTER		      (1ULL <<  3)
+#define IBS_OP_2_STRM_ST_FILTER_SHIFT		       (3)
+
+#endif /*  _ASM_X86_IBS_CAPS_H */
arch/x86/include/asm/perf_event.h +1 -80

93 lines not on this page · raw message

Signature · 1 line
2.34.1
#15 lore raw

PATCH v8 7/8 x86/mm/ibs: In-kernel driver for AMD IBS Memory Profiler

In reply to Bharata B Rao

Use IBS (Instruction Based Sampling) Memory Profiler feature present in AMD Zen6 processors for memory access tracking. The access information obtained from IBS Memory Profiler is fed to pghot sub-system for further action using pghot_record_access(PGHOT_HWHINTS, ...) API.

IBS Memory Profiler as page hotness source is enabled by the new config option HWMEM_PROFILER and is also gated by the existing pghot_src_hwhints static key set via sysctl tunable vm.pghot_enabled_sources.

New system vector 0xea is used for memprofiler interrupt.

This adds the driver infrastructure (interrupt setup, per-CPU ring, workqueue, cpuhp state, pghot call site etc) but the profiler isn't armed yet and that is done in the next commit.

More details about IBS Memory Profiler can be obtained from the AMD document titled "AMD64 Zen6 Instruction Based Sampling (IBS) Extensions and Features".

---
 Documentation/admin-guide/mm/pghot.rst |  44 ++-
 arch/x86/Kconfig                       |  16 ++
 arch/x86/entry/entry_fred.c            |   1 +
 arch/x86/include/asm/hardirq.h         |   3 +
 arch/x86/include/asm/ibs-caps.h        |   8 +
 arch/x86/include/asm/ibs-mprof.h       |  47 ++++
 arch/x86/include/asm/idtentry.h        |   6 +
 arch/x86/include/asm/irq_vectors.h     |   4 +-
 arch/x86/include/asm/msr-index.h       |   8 +
 arch/x86/kernel/idt.c                  |   3 +
 arch/x86/kernel/irq.c                  |   3 +
 arch/x86/mm/Makefile                   |   1 +
 arch/x86/mm/ibs-mprof.c                | 363 +++++++++++++++++++++++++
 include/linux/cpuhotplug.h             |   1 +
 include/linux/vm_event_item.h          |   7 +
 mm/Kconfig                             |   9 +
 mm/vmstat.c                            |   7 +
 17 files changed, 529 insertions(+), 2 deletions(-)
 create mode 100644 arch/x86/include/asm/ibs-mprof.h
 create mode 100644 arch/x86/mm/ibs-mprof.c
Documentation/admin-guide/mm/pghot.rst +43 -1
index accf78d95d02..b802b80e5c24 100644
@@ -87,7 +87,7 @@ Path: /proc/vmstat
 
 2. **pghot_recorded_hintfaults**
    - Number of recorded accesses reported by NUMA Balancing based
-     hotness source.
+     hint faults source.
 
 3. **pghot_recorded_hwhints**
    - Number of recorded accesses reported by hwhints source.
@@ -108,3 +108,45 @@ to promote hot pages. It can be disabled at runtime by clearing that bit:
 
 # echo 0x0 > /proc/sys/vm/pghot_enabled_sources
 
+Hardware Hints Source
+=====================
+pghot can consume memory access samples reported by hardware profilers.
+This "hardware hints" (hwhints) source feeds hardware-observed accesses
+into pghot for hot page detection and promotion.
+
+Generic in-kernel support for such profilers is controlled by the
+HWMEM_PROFILER config option. It is not enabled directly by the user;
+an in-kernel driver that forwards hardware-observed accesses to pghot
+selects it.
+
+The AMD IBS Memory Profiler (config AMD_IBS_MEMPROF, available on Zen6
+and later AMD CPUs) is one such driver. It uses the AMD Instruction
+Based Sampling (IBS) Memory Profiler facility to sample user memory
+accesses and record them with pghot.
+
+This source can be activated at runtime through the hardware
+hints bit (0x2) of **pghot_enabled_sources**.
+
+# echo 0x2 > /proc/sys/vm/pghot_enabled_sources
+
+HWHINTS Vmstat Counters
+=======================
+Following vmstat counters provide some stats about hardware hints source.
+
+Path: /proc/vmstat
+
+1. **hwhint_total_events**
+   - Number of total hwhint events recorded by hwhints source.
+
+2. **hwhint_useful_events**
+   - Number of actionable events from hwhints source.
+
+3. **hwhint_dropped_events**
+   - Number of events that were dropped due to buffer overrun.
+
+4. **hwhint_dram_accesses**
+   - Number of DRAM accesses reported by hwhints source.
+
+5. **hwhint_extmem_accesses**
+   - Number of external memory (like CXL) accesses reported by hwhints source.
+
arch/x86/Kconfig +16 -0
index bdad90f210e4..c6c5ed83506a 100644
@@ -1501,6 +1501,22 @@ config AMD_MEM_ENCRYPT
 	  This requires an AMD processor that supports Secure Memory
 	  Encryption (SME).
 
+config AMD_IBS_MEMPROF
+	bool "AMD IBS Memory Profiler"
+	depends on X86_64 && CPU_SUP_AMD
+	depends on PGHOT
+	select HWMEM_PROFILER
+	help
+	  Use the AMD Instruction Based Sampling (IBS) Memory Profiler
+	  facility (present on Zen6 and later AMD CPUs) to feed
+	  hardware-observed memory accesses into the pghot subsystem
+	  for hot-page detection and promotion.
+
+	  When disabled, no IBS Memory Profiler MSRs are programmed and
+	  the corresponding interrupt handler is not installed.
+
+	  If unsure, say N.
+
 # Common NUMA Features
 config NUMA
 	bool "NUMA Memory Allocation and Scheduler Support"
arch/x86/entry/entry_fred.c +1 -0
index fb3594ddf731..f52099e3fa85 100644
@@ -120,6 +120,7 @@ static idtentry_t sysvec_table[NR_SYSTEM_VECTORS] __ro_after_init = {
 	SYSVEC(POSTED_INTR_NESTED_VECTOR,	kvm_posted_intr_nested_ipi),
 
 	SYSVEC(POSTED_MSI_NOTIFICATION_VECTOR,	posted_msi_notification),
+	SYSVEC(IBS_MEMPROF_VECTOR,		ibs_memprof),
 };
 
 static bool fred_setup_done __initdata;
arch/x86/include/asm/hardirq.h +3 -0
index dea60d66d976..2da4930ea399 100644
@@ -49,6 +49,9 @@ enum irq_stat_counts {
 #endif
 #ifdef CONFIG_X86_POSTED_MSI
 	IRQ_COUNT_POSTED_MSI_NOTIFICATION,
+#endif
+#ifdef CONFIG_AMD_IBS_MEMPROF
+	IRQ_COUNT_IBS_MEMPROF,
 #endif
 	IRQ_COUNT_PIC_APIC_ERROR,
 #ifdef CONFIG_X86_IO_APIC
arch/x86/include/asm/ibs-caps.h +8 -0
index ddf6c512c8f9..1f6c4058a0e3 100644
@@ -29,6 +29,7 @@
 #define IBS_CAPS_FETCHLAT		(1U<<14)
 #define IBS_CAPS_BIT63_FILTER		(1U<<15)
 #define IBS_CAPS_STRMST_RMTSOCKET	(1U<<16)
+#define IBS_CAPS_MEM_PROFILER		(1U<<18)
 #define IBS_CAPS_OPDTLBPGSIZE		(1U<<19)
 
 #define IBS_CAPS_DEFAULT		(IBS_CAPS_AVAIL		\
@@ -42,6 +43,13 @@
 #define IBSCTL_LVT_OFFSET_VALID		(1ULL<<8)
 #define IBSCTL_LVT_OFFSET_MASK		0x0F
 
+/*
+ * IBS Memprofiler setup
+ */
+#define IBSCTL_MPROF_LVT_OFFSET_VALID	(1ULL << 24)
+#define IBSCTL_MPROF_LVT_OFFSET_SHIFT	16
+#define IBSCTL_MPROF_LVT_OFFSET_MASK	(0xFULL << IBSCTL_MPROF_LVT_OFFSET_SHIFT)
+
 /* IBS fetch bits/masks */
 #define IBS_FETCH_L3MISSONLY		      (1ULL << 59)
 #define IBS_FETCH_RAND_EN		      (1ULL << 57)
arch/x86/include/asm/ibs-mprof.h new file +47 -0

47 lines not on this page · raw message

arch/x86/include/asm/idtentry.h +6 -0
index 20f548702404..4c82e5c6d759 100644
@@ -735,6 +735,12 @@ DECLARE_IDTENTRY_SYSVEC(POSTED_MSI_NOTIFICATION_VECTOR,	sysvec_posted_msi_notifi
 # define fred_sysvec_posted_msi_notification		NULL
 # endif
 
+# ifdef CONFIG_AMD_IBS_MEMPROF
+DECLARE_IDTENTRY_SYSVEC(IBS_MEMPROF_VECTOR,		sysvec_ibs_memprof);
+#else
+# define fred_sysvec_ibs_memprof			NULL
+#endif
+
 #if IS_ENABLED(CONFIG_HYPERV)
 DECLARE_IDTENTRY_SYSVEC(HYPERVISOR_CALLBACK_VECTOR,	sysvec_hyperv_callback);
 DECLARE_IDTENTRY_SYSVEC(HYPERV_REENLIGHTENMENT_VECTOR,	sysvec_hyperv_reenlightenment);
arch/x86/include/asm/irq_vectors.h +3 -1
index 85253fc8e384..575f880cb147 100644
@@ -105,10 +105,12 @@
  */
 #define POSTED_MSI_NOTIFICATION_VECTOR	0xeb
 
+#define IBS_MEMPROF_VECTOR		0xea
+
 #define NR_VECTORS			 256
 
 #ifdef CONFIG_X86_LOCAL_APIC
-#define FIRST_SYSTEM_VECTOR		POSTED_MSI_NOTIFICATION_VECTOR
+#define FIRST_SYSTEM_VECTOR		IBS_MEMPROF_VECTOR
 #else
 #define FIRST_SYSTEM_VECTOR		NR_VECTORS
 #endif
arch/x86/include/asm/msr-index.h +8 -0
index 18c4be75e927..58d1bd7ebe72 100644
@@ -1317,4 +1317,12 @@
 						* a #GP
 						*/
 
+/* AMD IBS Memory Profiler MSRs */
+#define MSR_AMD64_IBS_MPROF_CTL		0xc0010380
+#define MSR_AMD64_IBS_MPROF_CTL2	0xc0010381
+#define MSR_AMD64_IBS_MPROF_DATA2	0xc0010382
+#define MSR_AMD64_IBS_MPROF_DATA3	0xc0010383
+#define MSR_AMD64_IBS_MPROF_LINADDR	0xc0010384
+#define MSR_AMD64_IBS_MPROF_PHYADDR	0xc0010385
+
 #endif /* _ASM_X86_MSR_INDEX_H */
arch/x86/kernel/idt.c +3 -0
index 90a22e24a9eb..50ec5fc82619 100644
@@ -169,6 +169,9 @@ static const __initconst struct idt_data apic_idts[] = {
 # ifdef CONFIG_X86_POSTED_MSI
 	INTG(POSTED_MSI_NOTIFICATION_VECTOR,	asm_sysvec_posted_msi_notification),
 # endif
+#ifdef CONFIG_AMD_IBS_MEMPROF
+	INTG(IBS_MEMPROF_VECTOR,		asm_sysvec_ibs_memprof),
+#endif
 #endif
 };
 
arch/x86/kernel/irq.c +3 -0
index 30122f0b3af9..c59a474d9202 100644
@@ -121,6 +121,9 @@ static const struct irq_stat_info irq_stat_info[IRQ_COUNT_MAX] = {
 #endif
 #ifdef CONFIG_X86_POSTED_MSI
 	ISS(POSTED_MSI_NOTIFICATION,	"PMN",	"  Posted MSI notification event\n"),
+#endif
+#ifdef CONFIG_AMD_IBS_MEMPROF
+	ISS(IBS_MEMPROF,		"IMP",	"  IBS Memory Profiler interrupts\n"),
 #endif
 	IDS(PIC_APIC_ERROR,		"ERR",	"  PIC/APIC error interrupts\n"),
 #ifdef CONFIG_X86_IO_APIC
arch/x86/mm/Makefile +1 -0
index 3a5364853eab..050a7379d9f7 100644
@@ -59,3 +59,4 @@ obj-$(CONFIG_X86_MEM_ENCRYPT)	+= mem_encrypt.o
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_amd.o
 
 obj-$(CONFIG_AMD_MEM_ENCRYPT)	+= mem_encrypt_boot.o
+obj-$(CONFIG_AMD_IBS_MEMPROF)	+= ibs-mprof.o
arch/x86/mm/ibs-mprof.c new file +363 -0

363 lines not on this page · raw message

include/linux/cpuhotplug.h +1 -0
index 0fb3a2a62eb0..2cd232b3bfcc 100644
@@ -150,6 +150,7 @@ enum cpuhp_state {
 	CPUHP_AP_PERF_X86_AMD_UNCORE_STARTING,
 	CPUHP_AP_PERF_X86_STARTING,
 	CPUHP_AP_PERF_X86_AMD_IBS_STARTING,
+	CPUHP_AP_MM_AMD_IBS_MEMPROF_STARTING,
 	CPUHP_AP_PERF_XTENSA_STARTING,
 	CPUHP_AP_ARM_VFP_STARTING,
 	CPUHP_AP_ARM64_DEBUG_MONITORS_STARTING,
include/linux/vm_event_item.h +7 -0
index 58d510711bd4..39409aa95482 100644
@@ -179,6 +179,13 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		PGHOT_RECORDED_ACCESSES,
 		PGHOT_RECORDED_HINTFAULTS,
 		PGHOT_RECORDED_HWHINTS,
+#ifdef CONFIG_HWMEM_PROFILER
+		HWHINT_TOTAL_EVENTS,
+		HWHINT_DRAM_ACCESSES,
+		HWHINT_EXTMEM_ACCESSES,
+		HWHINT_USEFUL_EVENTS,
+		HWHINT_DROPPED_EVENTS,
+#endif /* CONFIG_HWMEM_PROFILER */
 #endif /* CONFIG_PGHOT */
 		NR_VM_EVENT_ITEMS
 };
mm/Kconfig +9 -0
index 61ab18f4aebb..6a70e81b371e 100644
@@ -1534,6 +1534,15 @@ config PGHOT_PRECISE
 	  4 bytes per page against the default one byte per page. Preferable
 	  to enable this on systems with multiple nodes in toptier.
 
+config HWMEM_PROFILER
+	bool
+	depends on PGHOT
+	help
+	  Umbrella symbol enabled by any in-kernel driver that forwards
+	  hardware-observed memory accesses to the pghot subsystem (for
+	  example AMD_IBS_MEMPROF on x86_64). Drivers select this; users
+	  do not enable it directly.
+
 source "mm/damon/Kconfig"
 
 endmenu
mm/vmstat.c +7 -0
index da668ff05032..0d0ea5e664e3 100644
@@ -1493,6 +1493,13 @@ const char * const vmstat_text[] = {
 	[I(PGHOT_RECORDED_ACCESSES)]		= "pghot_recorded_accesses",
 	[I(PGHOT_RECORDED_HINTFAULTS)]		= "pghot_recorded_hintfaults",
 	[I(PGHOT_RECORDED_HWHINTS)]		= "pghot_recorded_hwhints",
+#ifdef CONFIG_HWMEM_PROFILER
+	[I(HWHINT_TOTAL_EVENTS)]		= "hwhint_total_events",
+	[I(HWHINT_DRAM_ACCESSES)]		= "hwhint_dram_accesses",
+	[I(HWHINT_EXTMEM_ACCESSES)]		= "hwhint_extmem_accesses",
+	[I(HWHINT_USEFUL_EVENTS)]		= "hwhint_useful_events",
+	[I(HWHINT_DROPPED_EVENTS)]		= "hwhint_dropped_events",
+#endif /* CONFIG_HWMEM_PROFILER */
 #endif /* CONFIG_PGHOT */
 #undef I
 #endif /* CONFIG_VM_EVENT_COUNTERS */
Signature · 1 line
2.34.1
#16 lore raw

Re: [PATCH v8 7/8] x86/mm/ibs: In-kernel driver for AMD IBS Memory Profiler

[Reply to Shashiko review]

Quoted 31 lines · Bharata B Rao
arch/x86/mm/ibs-mprof.c new file +22 -0
new file mode 100644
index 000000000000..923fb8f99552
+
+static bool mprof_pop_sample(struct mprof_sample_pcpu *pcpu, struct mprof_sample *s)
+{
+	int tail = READ_ONCE(pcpu->tail);
+	/*
+	 * Pairs with the smp_store_release() of head in mprof_push_sample();
+	 * ensures the sample slot stores are visible before we read the slot.
+	 */
+	int head = smp_load_acquire(&pcpu->head);
+	int next = tail + 1;
+
+	if (head == tail)
+		return false;
+
+	if (next >= IBS_NR_SAMPLES)
+		next = 0;
+
+	*s = pcpu->samples[tail];
+
+	WRITE_ONCE(pcpu->tail, next);
+	return true;
+}

Could the compiler reorder the non-volatile struct assignment of the sample after the volatile store to tail?

Quoted 7 lines

If a hardware interrupt preempts the worker thread exactly after the tail update is published but before the sample copy is complete, the interrupt handler could observe the advanced tail and overwrite the sample data being read.

Would it be safer to use smp_store_release(&pcpu->tail, next) here to ensure memory reads are strictly ordered before the tail update is published?

In this sample buffer SPSC ring, there is always one empty slot which keeps the producer at least one slot behind the consumer. So writes to the same slot that is being read is not possible even if the tail update is observed early.

Having said that, making the tail publish an explicit release is clearer and symmetric with the head handling. Hence will update this tail publish using smp_store_release().

Quoted 31 lines
arch/x86/mm/ibs-mprof.c new file +30 -0
+static void mprof_overflow_handler(void)
+{
+	u64 mem_ctl, mem_data3, mem_data2, paddr, data_src;
+	struct work_struct *w = &this_cpu_ptr(&mprof_work)->work;
+	unsigned long pfn;
+	struct page *page;
+
+	rdmsrq(MSR_AMD64_IBS_MPROF_CTL, mem_ctl);
+	if (!(mem_ctl & IBS_MPROF_CTL_VAL))
+		return;
+
+	mprof_disable_profiling(mem_ctl);
+	count_vm_event(HWHINT_TOTAL_EVENTS);
+
+	rdmsrq(MSR_AMD64_IBS_MPROF_DATA3, mem_data3);
+	rdmsrq(MSR_AMD64_IBS_MPROF_DATA2, mem_data2);
+
+	data_src = mem_data2 & IBS_MPROF_DATA2_DATASRC_MASK;
+	data_src |= ((mem_data2 & IBS_MPROF_DATA2_DATASRC_MASK_HIGH) >>
+			IBS_MPROF_DATA2_DATASRC_MASK_HIGH_SHIFT);
+
+	switch (data_src) {
+	case IBS_MPROF_DATA2_DATASRC_DRAM:
+		count_vm_event(HWHINT_DRAM_ACCESSES);
+		break;
+	case IBS_MPROF_DATA2_DATASRC_EXT_MEM:
+		count_vm_event(HWHINT_EXTMEM_ACCESSES);
+		break;
+	}
+
Quoted 4 lines

Does this switch intentionally omit the IBS_MPROF_DATA2_DATASRC_DRAM_FAR case? It appears that if remote DRAM accesses occur, they are currently silently dropped from the statistics rather than being counted under HWHINT_DRAM_ACCESSES.

No it was an oversight, remote DRAM access is still a DRAM access. It is just a stats reporting mismatch and promotion behaviour isn't affected. Will include remote DRAM access into HWHINT_DRAM_ACCESSES counter.

Quoted 44 lines
arch/x86/mm/ibs-mprof.c new file +24 -0
+
+static int x86_amd_ibs_mprof_teardown(unsigned int cpu)
+{
+	int offset = get_mprof_lvt_offset();
+	u64 mem_ctl;
+
+	if (offset >= 0)
+		setup_APIC_eilvt(offset, IBS_MEMPROF_VECTOR, APIC_DELIVERY_MODE_FIXED, 1);
+
+	rdmsrq(MSR_AMD64_IBS_MPROF_CTL, mem_ctl);
+	mprof_disable_profiling(mem_ctl);
+
+	/*
+	 * The producer is now silenced and this CPU's worker is gone. Drop
+	 * any unconsumed samples (see mprof_drain_cpu) and clear the pending
+	 * flag so a subsequent re-online of this CPU starts from a clean
+	 * state.
+	 */
+	mprof_drain_cpu(cpu);
+	per_cpu(mprof_work_pending, cpu) = false;
+
+	return 0;
+}
+

Is the assumption that "this CPU's worker is gone" always true here?

When a CPU goes offline, its per-CPU workqueues are unbound during the CPUHP_AP_WORKQUEUE_ONLINE phase, meaning pending work items like mprof_work_handler can migrate to another CPU and continue executing.

Because x86_amd_ibs_mprof_teardown runs later in the STARTING phase, mprof_drain_cpu() modifies the ring buffer unsynchronized. If the migrated worker is concurrently popping a sample, this could happen:

Worker CPU (migrated):
mprof_pop_sample() reads tail

Dying CPU:
mprof_drain_cpu() zeroes head and tail

Worker CPU (migrated):
mprof_pop_sample() updates tail = next

Quoted 6 lines

Could this desync the tail pointer, potentially causing the worker loop to read up to 149 uninitialized or stale samples and falsely report them to pghot_record_access?

Would it be safer to register the teardown at an ONLINE-level hotplug state so it can use cancel_work_sync() after disabling profiling?

That comment/assumption was inaccurate: after the workqueue unbind at offline, the worker can still be running on another CPU, so mprof_drain_cpu()'s ring reset could race it.

The migrated worker operates on its own ring buffer, so it is a single consumer which completely drains the ring on its own. The teardown reset was therefore both redundant and the only thing racing the worker. I will drop it in the next version.

Regards,
Bharata.

maintainer
#17 lore raw

RFC 1/1 x86: Remove CALL_FUNCTION_VECTOR

In reply to Bharata B Rao
Quoted 8 lines · Bharata B Rao
arch/x86/include/asm/irq_vectors.h +0 -0
index 85253fc8e384..575f880cb147 100644
@@ -105,10 +105,12 @@
  */
 #define POSTED_MSI_NOTIFICATION_VECTOR	0xeb
 

arch/x86/include/asm/irq_vectors.h line 116

Quoted 10 lines
arch/x86/include/asm/irq_vectors.h +3 -1
+#define IBS_MEMPROF_VECTOR		0xea
+
 #define NR_VECTORS			 256
 
 #ifdef CONFIG_X86_LOCAL_APIC
-#define FIRST_SYSTEM_VECTOR		POSTED_MSI_NOTIFICATION_VECTOR
+#define FIRST_SYSTEM_VECTOR		IBS_MEMPROF_VECTOR
 #else
 #define FIRST_SYSTEM_VECTOR		NR_VECTORS
 #endif

Right, so while looking at this, I noticed a few things; and I think we can do the below.

---
Subject: x86: Remove CALL_FUNCTION_VECTOR

Per the lines in linux/smp.h:

#define generic_smp_call_function_interrupt \
generic_smp_call_function_single_interrupt

The CALL_FUNCTION_VECTOR and CALL_FUNCTION_SINGLE_VECTOR have the exact same handler and are thus identical.

Boot tested with vng; Xen parts completely untested.

---
Note: we could also move HYPERV_*_VECTOR to the bottom and compress FIRST_SYSTEM_VECTOR depending on CONFIG_HYPERV, giving 2 interrupts back to !HYPERV peeps.

 arch/x86/entry/entry_fred.c        |  1 -
 arch/x86/include/asm/idtentry.h    |  1 -
 arch/x86/include/asm/irq_vectors.h | 39 +++++++++++++++++++-------------------
 arch/x86/include/asm/xen/events.h  |  1 -
 arch/x86/kernel/apic/ipi.c         |  6 +++---
 arch/x86/kernel/idt.c              |  1 -
 arch/x86/kernel/smp.c              |  9 ---------
 arch/x86/xen/smp.c                 | 22 +--------------------
 8 files changed, 23 insertions(+), 57 deletions(-)
arch/x86/entry/entry_fred.c +0 -1
index 854899bfec5d..661e125ed936 100644
@@ -114,7 +114,6 @@ static idtentry_t sysvec_table[NR_SYSTEM_VECTORS] __ro_after_init = {
 
 	SYSVEC(RESCHEDULE_VECTOR,		reschedule_ipi),
 	SYSVEC(CALL_FUNCTION_SINGLE_VECTOR,	call_function_single),
-	SYSVEC(CALL_FUNCTION_VECTOR,		call_function),
 	SYSVEC(REBOOT_VECTOR,			reboot),
 
 	SYSVEC(THRESHOLD_APIC_VECTOR,		threshold),
arch/x86/include/asm/idtentry.h +0 -1
index f400cfac69a6..d429a2720e8e 100644
@@ -679,7 +679,6 @@ DECLARE_IDTENTRY_SYSVEC(X86_PLATFORM_IPI_VECTOR,	sysvec_x86_platform_ipi);
 DECLARE_IDTENTRY(RESCHEDULE_VECTOR,			sysvec_reschedule_ipi);
 DECLARE_IDTENTRY_SYSVEC(REBOOT_VECTOR,			sysvec_reboot);
 DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_SINGLE_VECTOR,	sysvec_call_function_single);
-DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_VECTOR,		sysvec_call_function);
 #else
 # define fred_sysvec_reschedule_ipi			NULL
 # define fred_sysvec_reboot				NULL
arch/x86/include/asm/irq_vectors.h +19 -20
index 85253fc8e384..ca8f2045f915 100644
@@ -44,10 +44,10 @@
 #define ISA_IRQ_VECTOR(irq)		(((FIRST_EXTERNAL_VECTOR + 16) & ~15) + irq)
 
 /*
- * Special IRQ vectors used by the SMP architecture, 0xf0-0xff
+ * Special IRQ vectors used by the SMP architecture, 0xec-0xff
  *
  *  some of the following vectors are 'rare', they are merged
- *  into a single vector (CALL_FUNCTION_VECTOR) to save vector space.
+ *  into a single vector to save vector space.
  *  TLB, reschedule and local APIC vectors are performance-critical.
  */
 
@@ -61,49 +61,48 @@
 
 #define ERROR_APIC_VECTOR		0xfe
 #define RESCHEDULE_VECTOR		0xfd
-#define CALL_FUNCTION_VECTOR		0xfc
-#define CALL_FUNCTION_SINGLE_VECTOR	0xfb
-#define THERMAL_APIC_VECTOR		0xfa
-#define THRESHOLD_APIC_VECTOR		0xf9
-#define REBOOT_VECTOR			0xf8
+#define CALL_FUNCTION_SINGLE_VECTOR	0xfc
+#define THERMAL_APIC_VECTOR		0xfb
+#define THRESHOLD_APIC_VECTOR		0xfa
+#define REBOOT_VECTOR			0xf9
 
 /*
  * Generic system vector for platform specific use
  */
-#define X86_PLATFORM_IPI_VECTOR		0xf7
+#define X86_PLATFORM_IPI_VECTOR		0xf8
 
 /*
  * IRQ work vector:
  */
-#define IRQ_WORK_VECTOR			0xf6
+#define IRQ_WORK_VECTOR			0xf7
 
 /* IRQ vector for PMIs when running a guest with a mediated PMU. */
-#define PERF_GUEST_MEDIATED_PMI_VECTOR	0xf5
+#define PERF_GUEST_MEDIATED_PMI_VECTOR	0xf6
 
-#define DEFERRED_ERROR_VECTOR		0xf4
+#define DEFERRED_ERROR_VECTOR		0xf5
 
 /* Vector on which hypervisor callbacks will be delivered */
-#define HYPERVISOR_CALLBACK_VECTOR	0xf3
+#define HYPERVISOR_CALLBACK_VECTOR	0xf4
 
 /* Vector for KVM to deliver posted interrupt IPI */
-#define POSTED_INTR_VECTOR		0xf2
-#define POSTED_INTR_WAKEUP_VECTOR	0xf1
-#define POSTED_INTR_NESTED_VECTOR	0xf0
+#define POSTED_INTR_VECTOR		0xf3
+#define POSTED_INTR_WAKEUP_VECTOR	0xf2
+#define POSTED_INTR_NESTED_VECTOR	0xf1
 
-#define MANAGED_IRQ_SHUTDOWN_VECTOR	0xef
+#define MANAGED_IRQ_SHUTDOWN_VECTOR	0xf0
 
 #if IS_ENABLED(CONFIG_HYPERV)
-#define HYPERV_REENLIGHTENMENT_VECTOR	0xee
-#define HYPERV_STIMER0_VECTOR		0xed
+#define HYPERV_REENLIGHTENMENT_VECTOR	0xef
+#define HYPERV_STIMER0_VECTOR		0xee
 #endif
 
-#define LOCAL_TIMER_VECTOR		0xec
+#define LOCAL_TIMER_VECTOR		0xed
 
 /*
  * Posted interrupt notification vector for all device MSIs delivered to
  * the host kernel.
  */
-#define POSTED_MSI_NOTIFICATION_VECTOR	0xeb
+#define POSTED_MSI_NOTIFICATION_VECTOR	0xec
 
 #define NR_VECTORS			 256
 
arch/x86/include/asm/xen/events.h +0 -1
index 62bdceb594f1..11eeceb91fd6 100644
@@ -6,7 +6,6 @@
 
 enum ipi_vector {
 	XEN_RESCHEDULE_VECTOR,
-	XEN_CALL_FUNCTION_VECTOR,
 	XEN_CALL_FUNCTION_SINGLE_VECTOR,
 	XEN_SPIN_UNLOCK_VECTOR,
 	XEN_IRQ_WORK_VECTOR,
arch/x86/kernel/apic/ipi.c +3 -3
index c627bee3b14f..499bb3b8803b 100644
@@ -88,14 +88,14 @@ void native_send_call_func_ipi(const struct cpumask *mask)
 			goto sendmask;
 
 		if (cpumask_test_cpu(cpu, mask))
-			__apic_send_IPI_all(CALL_FUNCTION_VECTOR);
+			__apic_send_IPI_all(CALL_FUNCTION_SINGLE_VECTOR);
 		else if (num_online_cpus() > 1)
-			__apic_send_IPI_allbutself(CALL_FUNCTION_VECTOR);
+			__apic_send_IPI_allbutself(CALL_FUNCTION_SINGLE_VECTOR);
 		return;
 	}
 
 sendmask:
-	__apic_send_IPI_mask(mask, CALL_FUNCTION_VECTOR);
+	__apic_send_IPI_mask(mask, CALL_FUNCTION_SINGLE_VECTOR);
 }
 
 void apic_send_nmi_to_offline_cpu(unsigned int cpu)
arch/x86/kernel/idt.c +0 -1
index 90a22e24a9eb..b48e0e33ff03 100644
@@ -133,7 +133,6 @@ static const struct idt_data ia32_idt[] __initconst = {
 static const __initconst struct idt_data apic_idts[] = {
 #ifdef CONFIG_SMP
 	INTG(RESCHEDULE_VECTOR,			asm_sysvec_reschedule_ipi),
-	INTG(CALL_FUNCTION_VECTOR,		asm_sysvec_call_function),
 	INTG(CALL_FUNCTION_SINGLE_VECTOR,	asm_sysvec_call_function_single),
 	INTG(REBOOT_VECTOR,			asm_sysvec_reboot),
 #endif
arch/x86/kernel/smp.c +0 -9
index 985103cab16c..c7397816b311 100644
@@ -255,15 +255,6 @@ DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_reschedule_ipi)
 	trace_reschedule_exit(RESCHEDULE_VECTOR);
 }
 
-DEFINE_IDTENTRY_SYSVEC(sysvec_call_function)
-{
-	apic_eoi();
-	trace_call_function_entry(CALL_FUNCTION_VECTOR);
-	inc_irq_stat(CALL_FUNCTION);
-	generic_smp_call_function_interrupt();
-	trace_call_function_exit(CALL_FUNCTION_VECTOR);
-}
-
 DEFINE_IDTENTRY_SYSVEC(sysvec_call_function_single)
 {
 	apic_eoi();
arch/x86/xen/smp.c +1 -21
index 05ee0d3b0874..d5544026ede5 100644
@@ -11,7 +11,6 @@
 #include "xen-ops.h"
 
 static DEFINE_PER_CPU(struct xen_common_irq, xen_resched_irq) = { .irq = -1 };
-static DEFINE_PER_CPU(struct xen_common_irq, xen_callfunc_irq) = { .irq = -1 };
 static DEFINE_PER_CPU(struct xen_common_irq, xen_callfuncsingle_irq) = { .irq = -1 };
 static DEFINE_PER_CPU(struct xen_common_irq, xen_debug_irq) = { .irq = -1 };
 
@@ -37,12 +36,6 @@ void xen_smp_intr_free(unsigned int cpu)
 		unbind_from_irqhandler(per_cpu(xen_resched_irq, cpu).irq, NULL);
 		per_cpu(xen_resched_irq, cpu).irq = -1;
 	}
-	kfree(per_cpu(xen_callfunc_irq, cpu).name);
-	per_cpu(xen_callfunc_irq, cpu).name = NULL;
-	if (per_cpu(xen_callfunc_irq, cpu).irq >= 0) {
-		unbind_from_irqhandler(per_cpu(xen_callfunc_irq, cpu).irq, NULL);
-		per_cpu(xen_callfunc_irq, cpu).irq = -1;
-	}
 	kfree(per_cpu(xen_debug_irq, cpu).name);
 	per_cpu(xen_debug_irq, cpu).name = NULL;
 	if (per_cpu(xen_debug_irq, cpu).irq >= 0) {
@@ -80,16 +73,6 @@ int xen_smp_intr_init(unsigned int cpu)
 	callfunc_name = kasprintf(GFP_KERNEL, "callfunc%d", cpu);
 	if (!callfunc_name)
 		goto fail_mem;
-	per_cpu(xen_callfunc_irq, cpu).name = callfunc_name;
-	rc = bind_ipi_to_irqhandler(XEN_CALL_FUNCTION_VECTOR,
-				    cpu,
-				    xen_call_function_interrupt,
-				    IRQF_PERCPU|IRQF_NOBALANCING,
-				    callfunc_name,
-				    NULL);
-	if (rc < 0)
-		goto fail;
-	per_cpu(xen_callfunc_irq, cpu).irq = rc;
 
 	if (!xen_fifo_events) {
 		debug_name = kasprintf(GFP_KERNEL, "debug%d", cpu);
@@ -154,7 +137,7 @@ void xen_smp_send_call_function_ipi(const struct cpumask *mask)
 {
 	int cpu;
 
-	__xen_send_IPI_mask(mask, XEN_CALL_FUNCTION_VECTOR);
+	__xen_send_IPI_mask(mask, XEN_CALL_FUNCTION_SINGLE_VECTOR);
 
 	/* Make sure other vcpus get a chance to run if they need to. */
 	for_each_cpu(cpu, mask) {
@@ -179,9 +162,6 @@ static inline int xen_map_vector(int vector)
 	case RESCHEDULE_VECTOR:
 		xen_vector = XEN_RESCHEDULE_VECTOR;
 		break;
-	case CALL_FUNCTION_VECTOR:
-		xen_vector = XEN_CALL_FUNCTION_VECTOR;
-		break;
 	case CALL_FUNCTION_SINGLE_VECTOR:
 		xen_vector = XEN_CALL_FUNCTION_SINGLE_VECTOR;
 		break;
maintainer
#18 lore raw

Re: [RFC] x86: Remove CALL_FUNCTION_VECTOR

Quoted 32 lines · Peter Zijlstra
Quoted 8 lines · Bharata B Rao
arch/x86/include/asm/irq_vectors.h +0 -0
index 85253fc8e384..575f880cb147 100644
@@ -105,10 +105,12 @@
   */
  #define POSTED_MSI_NOTIFICATION_VECTOR	0xeb
 

arch/x86/include/asm/irq_vectors.h line 116

Quoted 10 lines
arch/x86/include/asm/irq_vectors.h +3 -1
+#define IBS_MEMPROF_VECTOR		0xea
+
  #define NR_VECTORS			 256
 
  #ifdef CONFIG_X86_LOCAL_APIC
-#define FIRST_SYSTEM_VECTOR		POSTED_MSI_NOTIFICATION_VECTOR
+#define FIRST_SYSTEM_VECTOR		IBS_MEMPROF_VECTOR
  #else
  #define FIRST_SYSTEM_VECTOR		NR_VECTORS
  #endif

Right, so while looking at this, I noticed a few things; and I think we can do the below.

---
Subject: x86: Remove CALL_FUNCTION_VECTOR

Per the lines in linux/smp.h:

   #define generic_smp_call_function_interrupt \
	  generic_smp_call_function_single_interrupt
Quoted 6 lines

The CALL_FUNCTION_VECTOR and CALL_FUNCTION_SINGLE_VECTOR have the exact same handler and are thus identical.

Boot tested with vng; Xen parts completely untested.

The Xen parts look fine.

Juergen

Quoted 250 lines

---
Note: we could also move HYPERV_*_VECTOR to the bottom and compress FIRST_SYSTEM_VECTOR depending on CONFIG_HYPERV, giving 2 interrupts back to !HYPERV peeps.

  arch/x86/entry/entry_fred.c        |  1 -
  arch/x86/include/asm/idtentry.h    |  1 -
  arch/x86/include/asm/irq_vectors.h | 39 +++++++++++++++++++-------------------
  arch/x86/include/asm/xen/events.h  |  1 -
  arch/x86/kernel/apic/ipi.c         |  6 +++---
  arch/x86/kernel/idt.c              |  1 -
  arch/x86/kernel/smp.c              |  9 ---------
  arch/x86/xen/smp.c                 | 22 +--------------------
  8 files changed, 23 insertions(+), 57 deletions(-)
arch/x86/entry/entry_fred.c +0 -1
index 854899bfec5d..661e125ed936 100644
@@ -114,7 +114,6 @@ static idtentry_t sysvec_table[NR_SYSTEM_VECTORS] __ro_after_init = {
 
  	SYSVEC(RESCHEDULE_VECTOR,		reschedule_ipi),
  	SYSVEC(CALL_FUNCTION_SINGLE_VECTOR,	call_function_single),
-	SYSVEC(CALL_FUNCTION_VECTOR,		call_function),
  	SYSVEC(REBOOT_VECTOR,			reboot),
 
  	SYSVEC(THRESHOLD_APIC_VECTOR,		threshold),
arch/x86/include/asm/idtentry.h +0 -1
index f400cfac69a6..d429a2720e8e 100644
@@ -679,7 +679,6 @@ DECLARE_IDTENTRY_SYSVEC(X86_PLATFORM_IPI_VECTOR,	sysvec_x86_platform_ipi);
  DECLARE_IDTENTRY(RESCHEDULE_VECTOR,			sysvec_reschedule_ipi);
  DECLARE_IDTENTRY_SYSVEC(REBOOT_VECTOR,			sysvec_reboot);
  DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_SINGLE_VECTOR,	sysvec_call_function_single);
-DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_VECTOR,		sysvec_call_function);
  #else
  # define fred_sysvec_reschedule_ipi			NULL
  # define fred_sysvec_reboot				NULL
arch/x86/include/asm/irq_vectors.h +19 -20
index 85253fc8e384..ca8f2045f915 100644
@@ -44,10 +44,10 @@
  #define ISA_IRQ_VECTOR(irq)		(((FIRST_EXTERNAL_VECTOR + 16) & ~15) + irq)
 
  /*
- * Special IRQ vectors used by the SMP architecture, 0xf0-0xff
+ * Special IRQ vectors used by the SMP architecture, 0xec-0xff
   *
   *  some of the following vectors are 'rare', they are merged
- *  into a single vector (CALL_FUNCTION_VECTOR) to save vector space.
+ *  into a single vector to save vector space.
   *  TLB, reschedule and local APIC vectors are performance-critical.
   */
 
@@ -61,49 +61,48 @@
 
  #define ERROR_APIC_VECTOR		0xfe
  #define RESCHEDULE_VECTOR		0xfd
-#define CALL_FUNCTION_VECTOR		0xfc
-#define CALL_FUNCTION_SINGLE_VECTOR	0xfb
-#define THERMAL_APIC_VECTOR		0xfa
-#define THRESHOLD_APIC_VECTOR		0xf9
-#define REBOOT_VECTOR			0xf8
+#define CALL_FUNCTION_SINGLE_VECTOR	0xfc
+#define THERMAL_APIC_VECTOR		0xfb
+#define THRESHOLD_APIC_VECTOR		0xfa
+#define REBOOT_VECTOR			0xf9
 
  /*
   * Generic system vector for platform specific use
   */
-#define X86_PLATFORM_IPI_VECTOR		0xf7
+#define X86_PLATFORM_IPI_VECTOR		0xf8
 
  /*
   * IRQ work vector:
   */
-#define IRQ_WORK_VECTOR			0xf6
+#define IRQ_WORK_VECTOR			0xf7
 
  /* IRQ vector for PMIs when running a guest with a mediated PMU. */
-#define PERF_GUEST_MEDIATED_PMI_VECTOR	0xf5
+#define PERF_GUEST_MEDIATED_PMI_VECTOR	0xf6
 
-#define DEFERRED_ERROR_VECTOR		0xf4
+#define DEFERRED_ERROR_VECTOR		0xf5
 
  /* Vector on which hypervisor callbacks will be delivered */
-#define HYPERVISOR_CALLBACK_VECTOR	0xf3
+#define HYPERVISOR_CALLBACK_VECTOR	0xf4
 
  /* Vector for KVM to deliver posted interrupt IPI */
-#define POSTED_INTR_VECTOR		0xf2
-#define POSTED_INTR_WAKEUP_VECTOR	0xf1
-#define POSTED_INTR_NESTED_VECTOR	0xf0
+#define POSTED_INTR_VECTOR		0xf3
+#define POSTED_INTR_WAKEUP_VECTOR	0xf2
+#define POSTED_INTR_NESTED_VECTOR	0xf1
 
-#define MANAGED_IRQ_SHUTDOWN_VECTOR	0xef
+#define MANAGED_IRQ_SHUTDOWN_VECTOR	0xf0
 
  #if IS_ENABLED(CONFIG_HYPERV)
-#define HYPERV_REENLIGHTENMENT_VECTOR	0xee
-#define HYPERV_STIMER0_VECTOR		0xed
+#define HYPERV_REENLIGHTENMENT_VECTOR	0xef
+#define HYPERV_STIMER0_VECTOR		0xee
  #endif
 
-#define LOCAL_TIMER_VECTOR		0xec
+#define LOCAL_TIMER_VECTOR		0xed
 
  /*
   * Posted interrupt notification vector for all device MSIs delivered to
   * the host kernel.
   */
-#define POSTED_MSI_NOTIFICATION_VECTOR	0xeb
+#define POSTED_MSI_NOTIFICATION_VECTOR	0xec
 
  #define NR_VECTORS			 256
 
arch/x86/include/asm/xen/events.h +0 -1
index 62bdceb594f1..11eeceb91fd6 100644
@@ -6,7 +6,6 @@
 
  enum ipi_vector {
  	XEN_RESCHEDULE_VECTOR,
-	XEN_CALL_FUNCTION_VECTOR,
  	XEN_CALL_FUNCTION_SINGLE_VECTOR,
  	XEN_SPIN_UNLOCK_VECTOR,
  	XEN_IRQ_WORK_VECTOR,
arch/x86/kernel/apic/ipi.c +3 -3
index c627bee3b14f..499bb3b8803b 100644
@@ -88,14 +88,14 @@ void native_send_call_func_ipi(const struct cpumask *mask)
  			goto sendmask;
 
  		if (cpumask_test_cpu(cpu, mask))
-			__apic_send_IPI_all(CALL_FUNCTION_VECTOR);
+			__apic_send_IPI_all(CALL_FUNCTION_SINGLE_VECTOR);
  		else if (num_online_cpus() > 1)
-			__apic_send_IPI_allbutself(CALL_FUNCTION_VECTOR);
+			__apic_send_IPI_allbutself(CALL_FUNCTION_SINGLE_VECTOR);
  		return;
  	}
 
  sendmask:
-	__apic_send_IPI_mask(mask, CALL_FUNCTION_VECTOR);
+	__apic_send_IPI_mask(mask, CALL_FUNCTION_SINGLE_VECTOR);
  }
 
  void apic_send_nmi_to_offline_cpu(unsigned int cpu)
arch/x86/kernel/idt.c +0 -1
index 90a22e24a9eb..b48e0e33ff03 100644
@@ -133,7 +133,6 @@ static const struct idt_data ia32_idt[] __initconst = {
  static const __initconst struct idt_data apic_idts[] = {
  #ifdef CONFIG_SMP
  	INTG(RESCHEDULE_VECTOR,			asm_sysvec_reschedule_ipi),
-	INTG(CALL_FUNCTION_VECTOR,		asm_sysvec_call_function),
  	INTG(CALL_FUNCTION_SINGLE_VECTOR,	asm_sysvec_call_function_single),
  	INTG(REBOOT_VECTOR,			asm_sysvec_reboot),
  #endif
arch/x86/kernel/smp.c +0 -9
index 985103cab16c..c7397816b311 100644
@@ -255,15 +255,6 @@ DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_reschedule_ipi)
  	trace_reschedule_exit(RESCHEDULE_VECTOR);
  }
 
-DEFINE_IDTENTRY_SYSVEC(sysvec_call_function)
-{
-	apic_eoi();
-	trace_call_function_entry(CALL_FUNCTION_VECTOR);
-	inc_irq_stat(CALL_FUNCTION);
-	generic_smp_call_function_interrupt();
-	trace_call_function_exit(CALL_FUNCTION_VECTOR);
-}
-
  DEFINE_IDTENTRY_SYSVEC(sysvec_call_function_single)
  {
  	apic_eoi();
arch/x86/xen/smp.c +1 -21
index 05ee0d3b0874..d5544026ede5 100644
@@ -11,7 +11,6 @@
  #include "xen-ops.h"
 
  static DEFINE_PER_CPU(struct xen_common_irq, xen_resched_irq) = { .irq = -1 };
-static DEFINE_PER_CPU(struct xen_common_irq, xen_callfunc_irq) = { .irq = -1 };
  static DEFINE_PER_CPU(struct xen_common_irq, xen_callfuncsingle_irq) = { .irq = -1 };
  static DEFINE_PER_CPU(struct xen_common_irq, xen_debug_irq) = { .irq = -1 };
 
@@ -37,12 +36,6 @@ void xen_smp_intr_free(unsigned int cpu)
  		unbind_from_irqhandler(per_cpu(xen_resched_irq, cpu).irq, NULL);
  		per_cpu(xen_resched_irq, cpu).irq = -1;
  	}
-	kfree(per_cpu(xen_callfunc_irq, cpu).name);
-	per_cpu(xen_callfunc_irq, cpu).name = NULL;
-	if (per_cpu(xen_callfunc_irq, cpu).irq >= 0) {
-		unbind_from_irqhandler(per_cpu(xen_callfunc_irq, cpu).irq, NULL);
-		per_cpu(xen_callfunc_irq, cpu).irq = -1;
-	}
  	kfree(per_cpu(xen_debug_irq, cpu).name);
  	per_cpu(xen_debug_irq, cpu).name = NULL;
  	if (per_cpu(xen_debug_irq, cpu).irq >= 0) {
@@ -80,16 +73,6 @@ int xen_smp_intr_init(unsigned int cpu)
  	callfunc_name = kasprintf(GFP_KERNEL, "callfunc%d", cpu);
  	if (!callfunc_name)
  		goto fail_mem;
-	per_cpu(xen_callfunc_irq, cpu).name = callfunc_name;
-	rc = bind_ipi_to_irqhandler(XEN_CALL_FUNCTION_VECTOR,
-				    cpu,
-				    xen_call_function_interrupt,
-				    IRQF_PERCPU|IRQF_NOBALANCING,
-				    callfunc_name,
-				    NULL);
-	if (rc < 0)
-		goto fail;
-	per_cpu(xen_callfunc_irq, cpu).irq = rc;
 
  	if (!xen_fifo_events) {
  		debug_name = kasprintf(GFP_KERNEL, "debug%d", cpu);
@@ -154,7 +137,7 @@ void xen_smp_send_call_function_ipi(const struct cpumask *mask)
  {
  	int cpu;
 
-	__xen_send_IPI_mask(mask, XEN_CALL_FUNCTION_VECTOR);
+	__xen_send_IPI_mask(mask, XEN_CALL_FUNCTION_SINGLE_VECTOR);
 
  	/* Make sure other vcpus get a chance to run if they need to. */
  	for_each_cpu(cpu, mask) {
@@ -179,9 +162,6 @@ static inline int xen_map_vector(int vector)
  	case RESCHEDULE_VECTOR:
  		xen_vector = XEN_RESCHEDULE_VECTOR;
  		break;
-	case CALL_FUNCTION_VECTOR:
-		xen_vector = XEN_CALL_FUNCTION_VECTOR;
-		break;
  	case CALL_FUNCTION_SINGLE_VECTOR:
  		xen_vector = XEN_CALL_FUNCTION_SINGLE_VECTOR;
  		break;
#19 lore raw

Re: [RFC] x86: Remove CALL_FUNCTION_VECTOR

In reply to Peter Zijlstra
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [Medium] x86/xen: memory leak of callfunc_name in xen_smp_intr_init() - [Low] x86/xen: unused…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [Medium] x86/xen: memory leak of callfunc_name in xen_smp_intr_init()
- [Low] x86/xen: unused xen_call_function_interrupt() causes build failure
- [Low] x86: dead call_function tracepoints and fred_sysvec_call_function macro --

commit a87f5828c6202eb98c865ff7619fe6fc331a81ac
Author: Peter Zijlstra <[email protected]>

x86: Remove CALL_FUNCTION_VECTOR

This commit removes CALL_FUNCTION_VECTOR and routes multi-CPU function call IPIs through CALL_FUNCTION_SINGLE_VECTOR across native x86 APIC, FRED, IDT, and Xen SMP paths.

Quoted 12 lines
arch/x86/include/asm/idtentry.h +0 -1
index f400cfac69a6b..d429a2720e8e7 100644
@@ -679,7 +679,6 @@ DECLARE_IDTENTRY_SYSVEC(X86_PLATFORM_IPI_VECTOR,	sysvec_x86_platform_ipi);
 DECLARE_IDTENTRY(RESCHEDULE_VECTOR,			sysvec_reschedule_ipi);
 DECLARE_IDTENTRY_SYSVEC(REBOOT_VECTOR,			sysvec_reboot);
 DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_SINGLE_VECTOR,	sysvec_call_function_single);
-DECLARE_IDTENTRY_SYSVEC(CALL_FUNCTION_VECTOR,		sysvec_call_function);
 #else
 # define fred_sysvec_reschedule_ipi			NULL
 # define fred_sysvec_reboot				NULL

[ ... ]

Quoted 10 lines
arch/x86/kernel/smp.c +0 -2
index 985103cab16c4..c7397816b311a 100644
@@ -255,15 +255,6 @@ DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_reschedule_ipi)
 	trace_reschedule_exit(RESCHEDULE_VECTOR);
 }
 
-DEFINE_IDTENTRY_SYSVEC(sysvec_call_function)
-{

arch/x86/kernel/smp.c DEFINE_IDTENTRY_SYSVEC_SIMPLE(sysvec_reschedule_ipi) · line 260

Quoted 10 lines
arch/x86/kernel/smp.c +0 -7
-	apic_eoi();
-	trace_call_function_entry(CALL_FUNCTION_VECTOR);
-	inc_irq_stat(CALL_FUNCTION);
-	generic_smp_call_function_interrupt();
-	trace_call_function_exit(CALL_FUNCTION_VECTOR);
-}
-
 DEFINE_IDTENTRY_SYSVEC(sysvec_call_function_single)
 {
 	apic_eoi();

[Severity: Low]
Removing sysvec_call_function() in arch/x86/kernel/smp.c drops the only call sites of trace_call_function_entry() and trace_call_function_exit(), while DEFINE_IRQ_VECTOR_EVENT(call_function) remains in arch/x86/include/asm/trace/irq_vectors.h:

    DEFINE_IRQ_VECTOR_EVENT(call_function);

Those dead tracepoints are also still registered in arch/x86/kernel/trace.c:osnoise_arch_register() and osnoise_arch_unregister():

    ret = register_trace_call_function_entry(trace_intel_irq_entry, NULL);
    if (ret)
            goto out_call_function_single_exit;

    ret = register_trace_call_function_exit(trace_intel_irq_exit, "call_function");

as well as in kernel/trace/rv/monitors/nrp/nrp.c:attach_vector_irq() and kernel/trace/rv/monitors/sts/sts.c:attach_vector_irq() (and their corresponding detach_vector_irq() functions):

    rv_attach_trace_probe("nrp", call_function_entry, handle_vector_irq_entry);
    rv_attach_trace_probe("sts", call_function_entry, handle_vector_irq_entry);

In addition, in arch/x86/include/asm/idtentry.h, the !CONFIG_SMP branch still defines:

    # define fred_sysvec_call_function			NULL

Should the call_function tracepoint definitions, their osnoise and RV monitor registrations, and fred_sysvec_call_function also be removed so the unused tracepoints do not remain exported and trigger warnings when building with UT=1?

Quoted 4 lines
arch/x86/xen/smp.c +0 -0
index 05ee0d3b0874e..d5544026ede5a 100644

[ ... ]

Quoted 4 lines
arch/x86/xen/smp.c +0 -0
@@ -80,16 +73,6 @@ int xen_smp_intr_init(unsigned int cpu)
 	callfunc_name = kasprintf(GFP_KERNEL, "callfunc%d", cpu);
 	if (!callfunc_name)
 		goto fail_mem;

arch/x86/xen/smp.c int xen_smp_intr_init(unsigned int cpu) · line 92

Quoted 10 lines
arch/x86/xen/smp.c +0 -10
-	per_cpu(xen_callfunc_irq, cpu).name = callfunc_name;
-	rc = bind_ipi_to_irqhandler(XEN_CALL_FUNCTION_VECTOR,
-				    cpu,
-				    xen_call_function_interrupt,
-				    IRQF_PERCPU|IRQF_NOBALANCING,
-				    callfunc_name,
-				    NULL);
-	if (rc < 0)
-		goto fail;
-	per_cpu(xen_callfunc_irq, cpu).irq = rc;

[Severity: Medium]
Does xen_smp_intr_init() in arch/x86/xen/smp.c leak callfunc_name on every CPU initialization under Xen?

The kasprintf() allocation for "callfunc%d" is left in place:

    callfunc_name = kasprintf(GFP_KERNEL, "callfunc%d", cpu);
    if (!callfunc_name)
            goto fail_mem;

Because per_cpu(xen_callfunc_irq, cpu).name = callfunc_name was removed, that buffer is never saved, so it is leaked if the subsequent VIRQ_DEBUG setup jumps to fail_mem or fail, and is unconditionally overwritten shortly after by the "callfuncsingle%d" allocation:

    callfunc_name = kasprintf(GFP_KERNEL, "callfuncsingle%d", cpu);
    if (!callfunc_name)
            goto fail_mem;

    per_cpu(xen_callfuncsingle_irq, cpu).name = callfunc_name;

[Severity: Low]
Also in arch/x86/xen/smp.c, removing bind_ipi_to_irqhandler() for XEN_CALL_FUNCTION_VECTOR in xen_smp_intr_init() leaves xen_call_function_interrupt() without any callers:

    static irqreturn_t xen_call_function_interrupt(int irq, void *dev_id);
    ...
    static irqreturn_t xen_call_function_interrupt(int irq, void *dev_id)
    {
            generic_smp_call_function_interrupt();
            inc_irq_stat(CALL_FUNCTION);

            return IRQ_HANDLED;
    }

Won't leaving this static function defined in arch/x86/xen/smp.c trigger an unused-function warning and break the build when CONFIG_XEN=y, CONFIG_SMP=y, and CONFIG_WERROR=y?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#20 lore raw

PATCH v8 8/8 x86/mm/ibs: Add runtime controls for IBS memprofiler

In reply to Bharata B Rao

Expose runtime tunables for IBS Memory profiler and arm the profiler.

<debugfs>/ibs-mprof/l3miss-only  - IbsMemL3MissOnly filter
<debugfs>/ibs-mprof/period       - op sample period (IbsMemMaxCnt)
<debugfs>/ibs-mprof/lat-filter   - IbsMemLatFltEn load-latency filter
<debugfs>/ibs-mprof/lat-thresh   - IbsMemLatThrsh (0x0..0xf)

In addition, a kernel cmdline parameter (ibs-memprof) is introduced to enable profiling right from boot time.

/sys/devices/system/cpu/ibs-mprof/enabled can be used to turn the profiling on or off.

Enabling the profiler will result in IBS Memory Profiler samples to be reported to pghot subsystem. pghot will act on these samples only if hwhints source is enabled explicilty using pghot/enabled_sources debugfs tunable.

---
 .../admin-guide/kernel-parameters.txt         |   5 +
 Documentation/admin-guide/mm/pghot.rst        |  44 +++
 arch/x86/include/asm/ibs-mprof.h              |  14 +
 arch/x86/mm/ibs-mprof.c                       | 371 +++++++++++++++++-
 4 files changed, 419 insertions(+), 15 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +5 -0
index b5493a7f8f22..c25bf35767eb 100644
@@ -2216,6 +2216,11 @@ Kernel parameters
 			syscalls, essentially overriding IA32_EMULATION_DEFAULT_DISABLED at
 			boot time. When false, unconditionally disables IA32 emulation.
 
+	ibs-memprof=	[X86-64]
+			Format: <bool>
+			When true, IBS Memory Profiler will be enabled at boottime
+			for memory access profiling. It can be changed at runtime too
+			using /sys/devices/system/cpu/ibs-mprof/enabled.
 
 	idle=		[X86,EARLY]
 			Format: idle=poll, idle=halt, idle=nomwait
Documentation/admin-guide/mm/pghot.rst +44 -0

47 lines not on this page · raw message

arch/x86/include/asm/ibs-mprof.h +14 -0
index 3de1880c4126..8b91c0ff1c33 100644
@@ -30,6 +30,9 @@
 #define IBS_MPROF_DATA3_SW_PREFETCH	BIT_ULL(21)
 
 /* MSR_AMD64_IBS_MPROF_CTL bits */
+#define IBS_MPROF_CTL_LATFLTEN		BIT_ULL(63)	/* IbsMemLatFltEn */
+#define IBS_MPROF_CTL_LATTHRSH_SHIFT	59
+#define IBS_MPROF_CTL_LATTHRSH_MASK	(0xFULL << IBS_MPROF_CTL_LATTHRSH_SHIFT)
 #define IBS_MPROF_CTL_CNT_CTL		BIT_ULL(19)
 #define IBS_MPROF_CTL_VAL		BIT_ULL(18)
 #define IBS_MPROF_CTL_ENABLE		BIT_ULL(17)
@@ -37,6 +40,17 @@
 #define IBS_MPROF_CTL_MAXCNT_MASK	0x0000FFFFULL
 #define IBS_MPROF_CTL_MAXCNT_EXT_MASK	(0x7FULL << 20)	/* separate upper 7 bits */
 
+/*
+ * IbsMemMaxCnt is a 27-bit op count; the low 4 bits are always zero.
+ * The hardware minimum is 16, but such a short interval is impractical
+ * (interrupt storm), so restrict the sample period to a sensible floor.
+ */
+#define IBS_MPROF_MAXCNT_MIN		5000
+#define IBS_MPROF_MAXCNT_MAX		((1U << 27) - 1)
+
+/* IbsMemLatThrsh is a 4-bit field. */
+#define IBS_MPROF_LATTHRSH_MAX		0xF
+
 /* MSR_AMD64_IBS_MPROF_CTL2 bits */
 #define IBS_MPROF_CTL2_DISABLE		BIT_ULL(0)
 #define IBS_MPROF_CTL2_EXCLUDE_USER	BIT_ULL(1)
arch/x86/mm/ibs-mprof.c +356 -15

426 lines not on this page · raw message

Signature · 1 line
2.34.1
#21 lore raw

Re: [PATCH v8 8/8] x86/mm/ibs: Add runtime controls for IBS memprofiler

[Reply to Shashiko review]

Quoted 13 lines · Bharata B Rao
arch/x86/mm/ibs-mprof.c +0 -0
index 923fb8f99552..07e0516db2ee 100644
 /*
  * Record the IBS-reported access sample in percpu buffer.
  * Called from IBS interrupt handler.
@@ -159,28 +199,55 @@ static inline void mprof_drain_cpu(unsigned int cpu)
 

Does this code safely handle CPU hotplug teardown?
During a CPU hotplug offline event, the high-priority stop_machine thread can preempt the mprof_work_handler worker mid-execution (for example, after reading head and tail).

Quoted 8 lines

x86_amd_ibs_mprof_teardown() then locklessly zeroes head and tail via mprof_drain_cpu().
When the CPU goes offline, the preempted worker thread is migrated to another CPU and resumes. Could it compute and write back its old tail value, desynchronizing tail from the newly reset head?
This might cause the ring buffer to appear full of old or uninitialized samples, which are then passed to pghot_record_access(), potentially leading to wild page promotions or memory corruption in the pghot subsystem.

This comment is same as the one provided in 7/8 which has been replied-to there.

Quoted 21 lines
arch/x86/mm/ibs-mprof.c +14 -0
+/*
+ * Publish a new config snapshot and push it to every online CPU
+ * immediately. Must be called with mprof_cfg_lock held.
+ */
+static void mprof_publish(const struct mprof_config *newcfg)
+{
+	struct mprof_config *slot;
+
+	lockdep_assert_held(&mprof_cfg_lock);
+
+	/* Fill the slot that is not currently published, then flip to it. */
+	slot = (mprof_cfg == &mprof_cfg_slots[0]) ?
+		&mprof_cfg_slots[1] : &mprof_cfg_slots[0];
+	*slot = *newcfg;

Can this struct copy cause a data race with sysfs and debugfs readers?

Because sysfs and debugfs attribute readers run in standard preemptible process context without locking, a reader task (like enabled_show) can execute smp_load_acquire() to get a pointer to the active slot and then be preempted.

Quoted 7 lines

If a concurrent writer updates the configuration twice, it flips the active pointer back to the reused slot and directly overwrites its contents here via a plain struct copy.

When the preempted reader resumes, it performs a plain C read concurrently with this struct assignment, which might result in a data race and torn reads of configuration fields.

Will fix this by adding mutex protection in the read side.

Quoted 17 lines
arch/x86/mm/ibs-mprof.c +10 -0
+	mprof_compose(slot);
+	/* Publish the fully composed slot; pairs with smp_load_acquire() in readers. */
+	smp_store_release(&mprof_cfg, slot);
+
+	/*
+	 * on_each_cpu() with wait serializes against any in-flight interrupt
+	 * handler on each CPU, so the previously published slot has no readers
+	 * once this returns and can be safely reused by the next writer.
+	 */
+	on_each_cpu(mprof_reprogram_this_cpu, NULL, 1);

Does this unconditionally arm the hardware profiler on all CPUs?

In x86_amd_ibs_mprof_startup(), if setup_APIC_eilvt() fails to configure the APIC interrupt vector, the function logs a warning and returns 0, allowing the CPU to successfully come online without configuring its APIC routing for IBS.

Quoted 7 lines

If the user enables the profiler via sysfs, this on_each_cpu() call executes mprof_enable_profiling() on all online CPUs, including any that failed setup.

When the hardware sample overflows, will the CPU deliver the interrupt using the previously configured (or uninitialized) EILVT vector, potentially resulting in spurious interrupts delivered to other subsystems or a system panic?

I will prevent this situation (where failure from EILVT setup is observed from online handler) by validating the EILVT offset at init time like how primary IBS does.

Quoted 18 lines
arch/x86/mm/ibs-mprof.c +17 -0
+static ssize_t enabled_show(struct device *dev,
+			    struct device_attribute *attr, char *buf)
+{
+	/* Acquire the snapshot; pairs with smp_store_release() in the writers. */
+	struct mprof_config *cfg = smp_load_acquire(&mprof_cfg);
+
+	return sysfs_emit(buf, "%s\n", str_enabled_disabled(cfg->enabled));
+}
+
+static ssize_t enabled_store(struct device *dev, struct device_attribute *attr,
+			     const char *buf, size_t count)
+{
+	struct mprof_config new;
+	bool enabled;
+	int ret;
+
+	ret = kstrtobool(buf, &enabled);
Quoted 8 lines

Does this break standard sysfs read-modify-write interactions?

The enabled_show() function outputs the state using str_enabled_disabled(), which prints "enabled\n" or "disabled\n".

However, kstrtobool() does not recognize "enabled" or "disabled" as valid boolean strings. As a result, standard boolean flag interactions like echo $(cat enabled) > enabled will fail with -EINVAL.

Doesn't look like.

kstrtobool() matches the first character, with the accepted set 'EeYyTt1DdNnFf0' plus "on"/"off". So enabled with pass. Same with disabled.

Regards,
Bharata.

#22 lore raw

Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - microbenchmark numbers

In reply to Bharata B Rao

Microbenchmark
==============
Multi-threaded application with 64 threads that access memory(8G) at 4K granularity repetitively and randomly. The number of accesses per thread and the randomness pattern for each thread are fixed beforehand. The accesses are divided into stores and loads in the ratio of 50:50.

Benchmark threads run on Node 0, while memory is initially provisioned on CXL node 2 before the accesses start.

Repetitive accesses results in lowertier pages becoming hot and kmigrated detecting and migrating them. The benchmark score is the time taken to finish the accesses in microseconds. The sooner it finishes the better it is. All the numbers shown below are average of 3 runs.

Table 1: Benchmark completion time (lower is better), avg of 3 runs 64 threads, random 4K access over 8G, mem on CXL node 2, cpu node 0

Config Time (us) Time (s) Speedup
---------------------------------------

C1       97,165,273       97.2    1.00x
C2       40,378,842       40.4    2.41x
C3       95,431,823       95.4    1.02x
C4       40,455,945       40.5    2.40x
C5       41,289,646       41.3    2.35x
C6       64,260,194       64.3    1.51x

Speedup = C1 (base-NUMAB0) time / config time (>1.00x is faster)

Legend:
C1 = base kernel, NUMAB0 (hot page promotion disabled)
C2 = base kernel, NUMAB2 (hot page promotion enabled)
C3 = pghot kernel, NUMAB0 (hint faults source disabled)
C4 = pghot kernel, NUMAB2 (hint faults source enabled)
C5 = pghot kernel, IBS Memory profiler, pghot_freq_threshold=1 (NUMAB0) C6 = pghot kernel, IBS Memory profiler, pghot_freq_threshold=2 (NUMAB0)

Table 2: Page migration / hotness vmstat counters (avg of 3 runs) '-' = counter not present for that kernel/config

ctr C1 C2 C3 C4 C5 C6
-----------------------------------------------------------------------

M1             0  2,097,152          0  2,097,152  1,966,188  1,444,783
M2             0  2,283,787          0  2,097,152  1,966,188  1,444,783
M3             0  2,097,152          0  2,097,152  1,966,188  1,444,783
M4             0  2,097,152          0  2,097,152          0          0
M5             0  2,283,787          0  2,097,152          0          0
M6             -          -          0  2,097,152  1,968,079  3,508,998
M7             -          -          0  2,097,152          0          0
M8             -          -          0          0  5,654,097  5,422,763
M9             -          -          0          0  5,654,097  5,422,763
M10            -          -          0          0  3,439,001  1,701,889
M11            -          -          0          0  1,879,453  3,105,182
M12            -          -          0          0  5,654,097  5,422,763
M13            -          -          0          0          0          0

Counter legend:
M1 = pgpromote_success
M2 = pgpromote_candidate_nrl
M3 = numa_pages_migrated
M4 = numa_pte_updates
M5 = numa_hint_faults
M6 = pghot_recorded_accesses
M7 = pghot_recorded_hintfaults
M8 = pghot_recorded_hwhints
M9 = hwhint_total_events
M10 = hwhint_dram_accesses
M11 = hwhint_extmem_accesses
M12 = hwhint_useful_events
M13 = hwhint_dropped_events

Key observations
================
- The base NUMAB0 vs NUMAB2 numbers clearly show the benefit of promotion.
- pghot-hintfault matches the base tiering case both in speedup and number of promottions.
- pghot-hwhint also matches the base tiering case both in speedup and number of promotions.

#23 lore raw

Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - NAS BT

In reply to Bharata B Rao

NAS Parallel Benchmark BT (Block Tri-diagonal solver), MPI version, Class F modified, run with 16 ranks. Compute threads are bound to node 1 via:

  /usr/bin/mpirun -np 16 /usr/bin/numactl --cpunodebind=1 \
      NPB3.4.4/NPB3.4-MPI/bin/bt.F.x

While class D uses around 24G of memory (which is too less to show the benefit of promition), class E results in around 368G of memory which overflows my toptier. Hence I wanted something in between these classes. So I have modified class F to the problem size of 768 which results in around 160GB of memory.

System is a 3-node tiered setup: node 0 and node 1 are DRAM (CPUs), node 2 is a CPU-less CXL lower tier (~252 GiB per node). Before the measurement phase all provisioned memory (~161.4 GiB across ranks) is migrated to the lower-tier CXL node 2. The ranks are then resumed and measurement begins, so the run starts fully cold on slow memory. This isolates the effect of hot-page promotion (node 2 -> node 1) on a bandwidth/latency-sensitive workload.

Legend (kernel / promotion source)
----------------------------------------------------------------------
A = base-numab0 Base kernel, NO tiering (NUMAB=0); reference
B = base-numab2 Base kernel, tiering via NUMA Balancing
mode 2 (hint-fault driven promotion)
C = pghot-hintfaults pghot kernel, hint-fault source (NUMAB=2)
D = pghot-hwhints pghot kernel, HW-hint source (IBS memory
profiler); does NOT need NUMA Balancing

Table 1a - Runtime (lower is better)
----------------------------------------------------------------------
Metric A B C D
----------------------------------------------------------------------
NPB Time (s) 7482.25 3306.81 3313.82 3506.55
Speedup vs A (time) 1.00 2.26 2.26 2.13

Table 1b - Throughput (higher is better)
----------------------------------------------------------------------
Metric A B C D
----------------------------------------------------------------------
NPB Mop/s total 52305.50 118350.48 118100.13 111608.96

Mop/s ratio vs A             1.00       2.26       2.26       2.13
Mop/s ratio vs B             0.44       1.00       1.00       0.94

Peak prov. mem (GiB) 161.38 161.38 161.38 161.38

Table 2 - vmstat counter deltas (after - before measurement window)
----------------------------------------------------------------------
Counter A B C D
----------------------------------------------------------------------

pgpromote_success                   0  42180536  42169996  39898316
pgpromote_candidate                 0         0         0         0
pgpromote_candidate_nrl             0  42180536  42169996  39899553
pgdemote_kswapd                     0         0         0         0
pgdemote_direct                     0         0         0         0
numa_pte_updates                    0  42250308  45027035         0
numa_hint_faults                    0  42182545  44119239         0
numa_pages_migrated                 0  42182543  42169996  39898316
pgmigrate_success                   0  42182543  42169996  39898316
pghot_recorded_accesses             0         0  44119239  40004334
pghot_recorded_hintfaults           0         0  44119239         0
pghot_recorded_hwhints              0         0         0 584209652
hwhint_total_events                 0         0         0 584209975
hwhint_dram_accesses                0         0         0 544195074
hwhint_extmem_accesses              0         0         0  40005074
hwhint_useful_events                0         0         0 584209652

Key observations
----------------------------------------------------------------------
Bottom line: hint-fault pghot reproduces mainline NUMAB=2 exactly, while HW-hint pghot delivers ~94% of that performance with zero NUMA-balancing overhead - a good trade-off where IBS is available.

#24 lore raw

Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - Graph500

In reply to Bharata B Rao

Benchmark : Graph500 reference BFS, SCALE=28, edgefactor=16, 128 ranks Topology : top-tier NUMA node=1 (CPUs+DRAM), lower-tier node=2 (mem-only/CXL)

Config legend
-------------
R1 : base kernel, no tiering (numa_balancing=0)
R2 : base kernel, tiering (numa_balancing=2 / NUMAB2)
R3 : pghot kernel, hintfaults source (pghot_enabled_sources=1,
pghot_freq_threshold=1, numa_balancing=2, promotion=on)
R4 : pghot kernel, hwhints source (pghot_enabled_sources=2,
pghot_freq_threshold=1, IBS mem-profiler on, numa_balancing=0, promotion=off)

Column legend
-------------
hmean : harmonic_mean_TEPS (primary Graph500 metric)
hstddev : harmonic_stddev_TEPS
median : median_TEPS
bfs_t : mean BFS time (seconds)
spdup : speedup of hmean vs R1 baseline

Table 1: Performance
--------------------

+--------+------------+----------+------------+---------+--------+
| Config |   hmean    | hstddev  |   median   |  bfs_t  | spdup  |
|        |   (TEPS)   |  (TEPS)  |   (TEPS)   |  (sec)  |        |
+--------+------------+----------+------------+---------+--------+
| R1     | 5.469e+08  | 4.55e+05 | 5.468e+08  |  7.853  | 1.00x  |
| R2     | 1.442e+09  | 8.04e+07 | 1.491e+09  |  2.979  | 2.64x  |
| R3     | 1.362e+09  | 7.92e+07 | 1.515e+09  |  3.153  | 2.49x  |
| R4     | 1.463e+09  | 1.35e+07 | 1.487e+09  |  2.935  | 2.68x  |
+--------+------------+----------+------------+---------+--------+

Table 2: Relevant kernel counters (/proc/vmstat deltas over the run)
--------------------------------------------------------------------
Values are accumulated deltas (before -> after) for the whole run.

+----------------------------+-----------+-----------+-----------+-----------+
| Counter                    |    R1     |    R2     |    R3     |    R4     |
+----------------------------+-----------+-----------+-----------+-----------+
| numa_pte_updates           |         0 |  26675234 |  28872628 |         0 |
| numa_hint_faults           |         0 |  13747383 |  15501854 |         0 |
| numa_pages_migrated        |         0 |  13747376 |  13726437 |   3683579 |
| pgpromote_success          |         0 |  13747254 |  13726437 |   3683579 |
| pghot_recorded_accesses    |         0 |         0 |  15501778 |   3682112 |
| pghot_recorded_hintfaults  |         0 |         0 |  15501854 |         0 |
| pghot_recorded_hwhints     |         0 |         0 |         0 |  24291151 |
| hwhint_total_events        |         0 |         0 |         0 |  24291198 |
| hwhint_dram_accesses       |         0 |         0 |         0 |  19588681 |
| hwhint_extmem_accesses     |         0 |         0 |         0 |   3663926 |
| pgmigrate_success          |  26840188 |  40586709 |  40570940 |  30521808 |
+----------------------------+-----------+-----------+-----------+-----------+

Key findings
------------
1. Tiering helps Graph500.
2. pghot-hintfaults is on par with base NUMAB2 tiering.
3. pghot-hwhints (R4) matches NUMAB=2 performance with zero NUMA hint faults.

#25 lore raw

Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - redis-memtier

In reply to Bharata B Rao

Redis-memtier

In-memory Redis server is loaded with a ~64 GB dataset
(62.2M keys x 1 KB) whose pages are then explicitly migrated to the CXL node (lower tier), while the Redis server and the memtier client run on a top-tier DRAM node. The measurement phase drives GET traffic with memtier (16 threads x 100 conns, PASSES=12) over 50% of the keyspace so that repeatedly-accessed lower-tier pages become promotion candidates. We compare throughput/latency and the kernel promotion activity across the base kernel and the pghot kernel, and across hotness sources (NUMA- balancing hint faults vs. the IBS hardware memory profiler).

System: AMD Zen6, Nodes 0,1 = DRAM (top tier); node 2 = CXL (lower tier).

Cases:

  C1 base/NUMAB0     base kernel, promotion OFF (baseline)
  C2 base/NUMAB2     base kernel, NUMAB tiering promotion ON
  C3 pghot/NUMAB2    pghot kernel, promotion via pghot, source =

NUMA hint faults (kmigrated batched migration)
C4 pghot/IBS-10000 pghot, source = IBS profiler (period 10000), NUMAB=0 C5 pghot/IBS-5000 same as C4 but IBS period=5000 (2x sampling)

=========================================================================
Table 1: Benchmark metrics (memtier)
=========================================================================
Case NUMAB Ops/sec vs base Avg p50 p99 p99.9

                                                          latency (ms) ->
------------------------------------------------------------------------
C1 base/NUMAB0         0   281,424      ref 181.62 180.22 348.16  362.50
C2 base/NUMAB2         2   295,551    +5.0% 173.18 168.96 307.20  354.30
C3 pghot/NUMAB2        2   296,269    +5.3% 172.42 169.98 327.68  364.54
C4 pghot/IBS-10000     0   281,645    +0.1% 181.67 179.20 350.21  362.50
C5 pghot/IBS-5000      0   281,665    +0.1% 181.72 180.22 348.16  362.50

------------------------------------------------------------------------

=========================================================================
Table 2: Page-migration metrics (vmstat delta over measurement)
=========================================================================
Legend: C1=base/NUMAB0 C2=base/NUMAB2 C3=pghot/NUMAB2

        C4=pghot/IBS-10000  C5=pghot/IBS-5000  ('-'=counter absent)

metric C1 C2 C3 C4 C5
-------------------------------------------------------------------------

pgpromote_success             0 10,433,576 10,433,484   594,762 1,128,469
pgpromote_candidate_nrl       0 10,433,576 10,433,484   594,762 1,128,469
numa_pte_updates              0 20,333,714 20,344,211         0         0
numa_hint_faults              0 10,433,576 10,433,533         0         0
numa_pages_migrated           0 10,433,576 10,433,484   594,762 1,128,463
pgmigrate_success             0 10,433,576 10,433,484   594,762 1,128,463
pghot_recorded_accesses       -          - 10,433,533   595,426 1,130,753
pghot_recorded_hintfaults     -          - 10,433,533         0         0
pghot_recorded_hwhints        -          -          0 1,000,187 2,026,558
pgdemote_kswapd               0          0          0         0         0

-------------------------------------------------------------------------

=========================================================================
Key observations
=========================================================================
1. Promotion helps this workload: enabling NUMAB tiering promotion lifts throughput from 281,424 to 295,551 ops/sec (+5.0%) and cuts average latency 181.6 -> 173.2 ms.
2. pghot-hintfault has no regression vs base as it matches the base tiering case.
3. pghot-hwhint sources promotes far fewer pages than hint faults and it barely shows any improvement over base notier case.
4. Halving the period (10000->5000) ~doubles hw hints (1.00M->2.03M) and promotions (0.59M->1.13M) as expected, but is still far short of the hot set, so throughput does not yet move.

#26 lore raw

Re: [PATCH v8 0/8] mm: Hot page tracking and promotion infrastructure - llama-bench

In reply to Bharata B Rao

llama-bench is the micro-benchmark shipped with llama.cpp. It loads a GGUF model through the production inference path and reports two throughput numbers per run: pp512 (one batched 512-token prefill, compute-bound) and tg128 (128 sequential autoregressive decode steps, bandwidth-bound).

Workload: llama-bench (llama.cpp), Mixtral-8x22B-Instruct Q4_K_M (140.6B params, 79.7 GiB), -t 64 -p 512 -n 128 -r 5 --mmap 0. pp512 = prefill (compute-bound); tg128 = decode (memory-bandwidth-bound).

Setup: 3-node box, N0/N1 DRAM (256 GiB each), N2 CXL (distances 10/12/50). Bench pinned to N1 CPUs with MPOL_PREFERRED_MANY({1}); a 200 GiB hot hog on N1 forces kswapd to naturally demote ~1/3 of the model to CXL; each mode is then measured while the hog keeps N1 under pressure. pghot target_nid=1, freq_threshold=1, freq_window=3000ms, rate_limit=65536 MBps, kmigrated 100ms/512. hwhints runs arm AMD IBS (l3miss-only=1, period=10000).

Legend (columns)
----------------

  r1 = base / notier        7.2.0-rc2-base   nb=0 src=-
  r2 = base / tier (NUMAB2) 7.2.0-rc2-base   nb=2 src=-
  r3 = pghot / hintfaults   7.2.0-rc2-pghot  nb=2 src=0x1
  r4 = pghot / hwhints(IBS) 7.2.0-rc2-pghot  nb=0 src=0x2
  r5 = pghot / both         7.2.0-rc2-pghot  nb=2 src=0x3

(nb = kernel.numa_balancing; src = pghot_enabled_sources)

Table 1 - Throughput (llama-bench, tokens/s over 5 reps)
-------------------------------------------------------
metric r1 r2 r3 r4 r5
-------------------------------------------------------------------

pp512 t/s                 69.31    62.46    54.53    53.95    51.49
  stddev                   2.70     6.65    10.41     8.12     4.79
tg128 t/s                 4.262    4.625    5.993    5.161    6.851
  stddev                  0.004    0.313    1.255    0.535    1.807
tg128 vs r1               1.00x    1.09x    1.41x    1.21x    1.61x
pp512 vs r1               1.00x    0.90x    0.79x    0.78x    0.74x

Table 1b - tg128 per-iteration (t/s), shows convergence
-------------------------------------------------------
iter r1 r2 r3 r4 r5
-------------------------------------------------------------------

iter1                     4.268    4.183    4.393    4.462    4.415
iter2                     4.261    4.457    5.336    4.820    5.602
iter3                     4.260    4.678    5.995    5.218    7.322
iter4                     4.261    4.841    6.501    5.494    8.152
iter5                     4.259    4.969    7.740    5.813    8.763

Table 2 - Key vmstat counters, run-phase deltas (millions of pages/events)
--------------------------------------------------------------------------
metric r1 r2 r3 r4 r5
------------------------------------------------------------------

pgpromote_success             0.00    3.05    6.95    2.17    6.53
pgpromote_candidate           0.00   14.34   24.12   13.83   21.09
pgdemote_kswapd               0.69    5.02   10.84    3.89    8.05
pgmigrate_success             0.69    8.07   17.80    6.06   14.59
numa_pte_updates              0.00   28.77   24.56    0.00   12.04
numa_hint_faults              0.00   27.77   24.12    0.00   11.39
pghot_recorded_accesses       0.00    0.00   24.12   13.83   21.28
pghot_recorded_hintfaults     0.00    0.00   24.12    0.00   11.39
pghot_recorded_hwhints        0.00    0.00    0.00   34.03   32.42
hwhint_total_events           0.00    0.00    0.00   34.03   32.42
hwhint_dram_accesses          0.00    0.00    0.00   20.03   22.34
hwhint_extmem_accesses        0.00    0.00    0.00   13.83    9.89

Key observations
----------------
1. pghot-both (both hintfaults and hwhints enabled) improves decode over both base baselines.
2. Prefill (pp512) suffers from tiering promotion (0.79x hintfaults, 0.78x hwhints, 0.74x both vs notier); pghot/hf and /hw pp512 are noisy (stddev ~8-10).

#28 lore raw
Quoted 8 lines · Andrew Morton
Quoted 2 lines · On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <[email protected]> wrote:

This patchset introduces pghot, a subsystem for hot page tracking and promotion.

Can DAMON be used to do this sort of thing adequately?

Thanks, I'll await review input on this.

I don't think we should do this. I don't think it's a use-case that customers actually want; I think it's something CPU vendors think their customers want. I've said this repeatedly over the months this patchset has been in development. I say no.

maintainer
#29 lore raw
Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · Andrew Morton
Quoted 2 lines · On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <[email protected]> wrote:

This patchset introduces pghot, a subsystem for hot page tracking and promotion.

Quoted 9 lines
Quoted 3 lines

Can DAMON be used to do this sort of thing adequately?

Thanks, I'll await review input on this.

I don't think we should do this. I don't think it's a use-case that customers actually want; I think it's something CPU vendors think their customers want. I've said this repeatedly over the months this patchset has been in development. I say no.

I've been consistent about saying we (Meta) have been actively interested in this work and helping to drive the design of it.

So it's not simply something born out of a CPU vendor think tank, we're actively working on solving real problems with it.

I don't have data I can share now, but I think it's fair to wait for that data before faith-merging this on the hopes it's impactful. It does make it more difficult for us to do fleet-scale testing without it being upstream, but we'll get something.

Tiered memory is being deployed aggressively [0] for considerable value, and is here to stay - like it or not.

~Gregory

[0] https://jovans2.github.io/files/vistara_camera_ready.pdf

#30 lore raw
Quoted 20 lines · Gregory Price
Quoted 12 lines · Matthew Wilcox
Quoted 5 lines · Andrew Morton

Can DAMON be used to do this sort of thing adequately?

Thanks, I'll await review input on this.

I don't think we should do this. I don't think it's a use-case that customers actually want; I think it's something CPU vendors think their customers want. I've said this repeatedly over the months this patchset has been in development. I say no.

I've been consistent about saying we (Meta) have been actively interested in this work and helping to drive the design of it.

So it's not simply something born out of a CPU vendor think tank, we're actively working on solving real problems with it.

Quoted 7 lines

I don't have data I can share now, but I think it's fair to wait for that data before faith-merging this on the hopes it's impactful. It does make it more difficult for us to do fleet-scale testing without it being upstream, but we'll get something.

Tiered memory is being deployed aggressively [0] for considerable value, and is here to stay - like it or not.

The diffstat is scary, to say the least. And the x86 stuff certainly needs x86 maintainer acks.

So far there is no real review -- we're in v8 without a single tag.

So this is far away from being mergeable.

I'm afraid this will need a lot more discussions, and likely RFC would have been still appropriate at this point.

It's a shame this work was going on for quite a while without feedback (blame me if you want :) but there are only so many hours in a day), but the main problem is that this work in particular was not perceived to be high or even medium priority so far.

Signature · 3 lines
Cheers,

David
maintainer
#31 lore raw
Quoted 11 lines · David Hildenbrand (Arm)

The diffstat is scary, to say the least. And the x86 stuff certainly needs x86 maintainer acks.

So far there is no real review -- we're in v8 without a single tag.

So this is far away from being mergeable.

I'm afraid this will need a lot more discussions, and likely RFC would have been still appropriate at this point.

No disagreement here, my lament of not being able to get fleet-scale data is more of a personal one, not a cry to merge. We're working on it.

Quoted 5 lines

It's a shame this work was going on for quite a while without feedback (blame me if you want :) but there are only so many hours in a day), but the main problem is that this work in particular was not perceived to be high or even medium priority so far.

Should note that we hold a bi-weekly working group on this to discuss the design and micro-benchmarks - so it hasn't been developed in a complete vacuum. We (I) owe that group some explicit on-list reviews and testing data - that's my fault (life has been busy).

~Gregory

#32 lore raw
In reply to David Hildenbrand (Arm)
Quoted 3 lines · David Hildenbrand (Arm)

The diffstat is scary, to say the least. And the x86 stuff certainly needs x86 maintainer acks.

The diffstat can be logically grouped into the following 3 sets:

1. IBS Memory Profiler: A bunch of x86 specific files got included in this version when I switched from NMI to using fixed interrupt for IBS Memory Profiler.

2. Many of the files in mm and all in sched are touched due to moving of NUMA Balancing tiering mode (kernel.numa_balancing=2) from scheduler to pghot as hint fault source.

3. Then rest of the changes are due to pghot core.

Quoted 7 lines

So far there is no real review -- we're in v8 without a single tag.

So this is far away from being mergeable.

I'm afraid this will need a lot more discussions, and likely RFC would have been still appropriate at this point.

Noted.

Regards,
Bharata.

#33 lore raw
In reply to Matthew Wilcox
Quoted 6 lines · Matthew Wilcox
Quoted 5 lines · Andrew Morton
Quoted 2 lines · On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <[email protected]> wrote:

This patchset introduces pghot, a subsystem for hot page tracking and promotion.

Quoted 9 lines
Quoted 3 lines

Can DAMON be used to do this sort of thing adequately?

Thanks, I'll await review input on this.

I don't think we should do this. I don't think it's a use-case that customers actually want; I think it's something CPU vendors think their customers want. I've said this repeatedly over the months this patchset has been in development. I say no.

From our perspective, this is not merely a hypothetical use case. We are actively developing and evaluating CXL tiered-memory systems, and expect promoting CXL-resident pages that become hot back to DRAM to be a practical requirement.

NUMA balancing mode 2 already provides such a promotion path. What we are looking for is a better hot-page detection mechanism than NUMA hint faults. AMD IBS is an interesting option, but the v8 results do not yet show a clear performance advantage over hint faults.

IBS is not the only possible hardware source. CXL 3.2 defines CHMU for device-side hotness monitoring, and CHMU has also been discussed in the DAMON and CXL hotness-monitoring discussions.

CHMU does not normally provide the CPU NUMA node that generated an access, which complicates target-node selection on multi-socket systems. However, single-socket systems are also common across data-center CPU generations. On such systems, the missing requester NUMA node is much less of a limitation.

This suggests that a configurable framework capable of consuming different hotness sources, depending on the platform topology and available hardware, would be useful.

We would also be interested in helping test and compare the different approaches as implementations become available.

Thanks,
Yongting Lin

#34 lore raw
Quoted 10 lines · Yongting Lin
Quoted 4 lines · Matthew Wilcox

I don't think we should do this. I don't think it's a use-case that customers actually want; I think it's something CPU vendors think their customers want. I've said this repeatedly over the months this patchset has been in development. I say no.

From our perspective, this is not merely a hypothetical use case. We are actively developing and evaluating CXL tiered-memory systems, and expect promoting CXL-resident pages that become hot back to DRAM to be a practical requirement.

But what *is* your use case? Truly hot data ends up in L1/L2/L3 no matter whether it came across the ridiculously high latency CXL link or from regular DRAM. So we're talking about "warm" data that's presumably measured in gigabytes (since current server CPUs have about half a gigabyte of LLC)

I see the basis for saying "this task is low priority, it gets its memory allocated on CXL" and "this task is high priority, it gets its memory allocated on DRAM". I don't see the use case where we're trying to figure out that region A of this task is sufficiently warmer than region B, and so we want to demote region B back to CXL and promote region A back to DRAM.

I particularly don't see a case for trying to do it on single-page granularity. There's just too much damn information to track. Maybe at a 2MB or 1GB boundary, but not per page.

#35 lore raw
Quoted 24 lines · Matthew Wilcox
Quoted 10 lines · Yongting Lin
Quoted 4 lines · Matthew Wilcox

I don't think we should do this. I don't think it's a use-case that customers actually want; I think it's something CPU vendors think their customers want. I've said this repeatedly over the months this patchset has been in development. I say no.

From our perspective, this is not merely a hypothetical use case. We are actively developing and evaluating CXL tiered-memory systems, and expect promoting CXL-resident pages that become hot back to DRAM to be a practical requirement.

But what *is* your use case? Truly hot data ends up in L1/L2/L3 no matter whether it came across the ridiculously high latency CXL link or from regular DRAM. So we're talking about "warm" data that's presumably measured in gigabytes (since current server CPUs have about half a gigabyte of LLC)

I see the basis for saying "this task is low priority, it gets its memory allocated on CXL" and "this task is high priority, it gets its memory allocated on DRAM". I don't see the use case where we're trying to figure out that region A of this task is sufficiently warmer than region B, and so we want to demote region B back to CXL and promote region A back to DRAM.

Quoted 3 lines

I particularly don't see a case for trying to do it on single-page granularity. There's just too much damn information to track. Maybe at a 2MB or 1GB boundary, but not per page.

Replying to the observation about per-page tracking information...

Since pghot has seen several versions and implementations over time, things have changed multiple times and it is also easy to miss the information about what gets tracked per-page and how it gets used from my long cover note. Hence pasting some info here if it helps.

How pghot works
===============
- Tracks frequency and last access time.
- Additionally, the accessing NUMA node ID (NID) for each recorded access is tracked in precision mode.
- These hotness parameters are maintained in a per-PFN hotness record within a per-section map hanging off the existing mem_section data structure. The map is RCU-protected and also carries a section-level "hot" flag used to gate scanning by kmigrated.
- In default mode, one byte (u8) is used for the hotness record. 5 bits store time using a bucketing scheme representing up to 4s with HZ=1000. The default toptier NID (0) is used as the promotion target and can be changed via the vm.pghot_target_nid sysctl.
- In precision mode, 4 bytes (u32) are used for each hotness record. 14 bits store time, representing around 16s with HZ=1000, and the accessing NID is stored per-PFN as the promotion target.
- Classifies pages as hot based on configurable thresholds.
- Pages classified as hot are marked ready for migration using the ready bit (MSB of the hotness record in both modes).
- Per-lower-tier-node kmigrated threads periodically scan the PFNs of lower-tier nodes, checking the migration-ready bit to perform batched migrations. The scan interval and batch size are configurable via debugfs tunables.

Memory overhead
===============
Default mode: 1 byte per lower-tier PFN. For 1TB of lower-tier memory this amounts to 256MB overhead (assuming 4K pages).

Precision mode: 4 bytes per lower-tier PFN. For 1TB of lower-tier memory this amounts to 1GB overhead.

Regards,
Bharata.

#36 lore raw
In reply to Matthew Wilcox
Quoted 12 lines · Matthew Wilcox
Quoted 10 lines · Yongting Lin
Quoted 4 lines · Matthew Wilcox

I don't think we should do this. I don't think it's a use-case that customers actually want; I think it's something CPU vendors think their customers want. I've said this repeatedly over the months this patchset has been in development. I say no.

From our perspective, this is not merely a hypothetical use case. We are actively developing and evaluating CXL tiered-memory systems, and expect promoting CXL-resident pages that become hot back to DRAM to be a practical requirement.

Quoted 5 lines

But what *is* your use case? Truly hot data ends up in L1/L2/L3 no matter whether it came across the ridiculously high latency CXL link or from regular DRAM. So we're talking about "warm" data that's presumably measured in gigabytes (since current server CPUs have about half a gigabyte of LLC)

I think some context may have been lost in my wording. Ignoring CPU caches for the moment, my original point was that the hot portion of a workload should reside in DRAM, while its cold portion can be placed on CXL. If some CXL-resident memory later becomes frequently accessed, it becomes hot from this placement perspective and should be considered for promotion back to DRAM.

In the terminology of your cache-related distinction, what I called "hot" memory here would more accurately be called "warm" memory.

Although LLCs have become fairly large, for the working set that does not fit in the LLC, placing it in DRAM rather than CXL still makes a clear performance difference, at least for the workloads we have evaluated.

Quoted 5 lines

I see the basis for saying "this task is low priority, it gets its memory allocated on CXL" and "this task is high priority, it gets its memory allocated on DRAM". I don't see the use case where we're trying to figure out that region A of this task is sufficiently warmer than region B, and so we want to demote region B back to CXL and promote region A back to DRAM.

Our motivation for using CXL is not related to task priority. Many of our services have clearly distinguishable warm and cold memory within the same workload, and we would like to demote the cold portion to CXL.

Although these workloads usually exhibit access locality, their access patterns can also change abruptly: memory that was cold and demoted to CXL can become actively accessed again. In that case, we may need to promote it back to DRAM. A mechanism that can identify such changes accurately and promptly would allow those pages to be moved back to DRAM sooner and reduce the resulting performance loss.

Thanks,
Yongting

maintainer
#37 lore raw
In reply to Matthew Wilcox
Quoted 8 lines · Matthew Wilcox
Quoted 6 lines · Yongting Lin

On Tue, Jul 28, 2026 at 07:24:46PM +0100, Matthew Wilcox wrote:

From our perspective, this is not merely a hypothetical use case. We are actively developing and evaluating CXL tiered-memory systems, and expect promoting CXL-resident pages that become hot back to DRAM to be a practical requirement.

Quoted 6 lines

But what *is* your use case? Truly hot data ends up in L1/L2/L3 no matter whether it came across the ridiculously high latency CXL link or from regular DRAM. So we're talking about "warm" data that's presumably measured in gigabytes (since current server CPUs have about half a gigabyte of LLC)

Speaking for myself (not Yongting Lin) - our use case is pretty straight forward and deployed at scale today, with plans to expand on it in future generations.

We have systems with 1TB+ of memory, with a 768GB/256GB DRAM/CXL split.

We're expecting these numbers to grow, along with density of work.

On these systems we use both scaled and stacked workloads (single and multi-container) where we do have a decent amount of warm data landing on CXL. In fact, we're looking to make that a more explicit scenario to reduce per-workload variance (higher floors, lower ceilings).

See Joshua's Tiered Memcg Limits work [0]
[0] https://lore.kernel.org/all/[email protected]/

Right now this works when mixing workloads which largely use mapped memory, but for workloads using unmapped pagecache, they end up permanently demoted to CXL because NUMA Balancing can't promote page cache (but reclaim can demote it, and the page allocator can fallback to CXL).

PGHot at least gives us the *full* solution that NUMA balancing does not.

Quoted 6 lines

I see the basis for saying "this task is low priority, it gets its memory allocated on CXL" and "this task is high priority, it gets its memory allocated on DRAM". I don't see the use case where we're trying to figure out that region A of this task is sufficiently warmer than region B, and so we want to demote region B back to CXL and promote region A back to DRAM.

We find the vast, vast majority of "low priority" work ends up pushed out to swap / fully invalidated anyway, and not even taking up much of any memory in the first place. This work is not the problem - it's usually things like system updates or monitors etc.

Actual user work wanting X-GB of memory is almost never willing to take the latency hit of 100% CXL - and even if you did that, you actually cause bandwidth issues for the rest of the system due to limited bandwidth compared to local DRAM.

Something to remember: Demotion does not respect cpusets or mempolicy, and page cache is owned by the first-toucher, so cpuset limitations break for page cache in general. (This is one of many reasons I've been exploring the private nodes series).

===

For actual tangible problems:

We've found fallback allocations landing on CXL is preferable to forcing reclaim to run and demote to CXL before a new DRAM allocation can be made. e.g. zone_reclaim_mode=0x0 is better than 0xf. ZRM induced stalls are just always worse than eating a CXL page and moving it later.

This directly results in hot memory (in fact many gbs of it) on CXL.

This is especially prevelent when workloads spinup after the top tier is already / near full. The tier-aware memcg stuff should help with this, but it still requires both demotion and promotion to level out the usage between tiers to an acceptable ratio.

I'm not looking for a system that can predict the future and maximize performance. At best I'd like a system that nudges placement in the right direction over a long period for better workload consistency.

Quoted 3 lines

I particularly don't see a case for trying to do it on single-page granularity. There's just too much damn information to track. Maybe at a 2MB or 1GB boundary, but not per page.

I largely agree with you here. If you look at the other work my colleagues have been doing - it's been heavily focused on making THPs more reliable and even pushing towards 1GB THPs. We're getting to a point where 4kb just doesn't make sense anymore for many things.

~Gregory

#38 lore raw
In reply to Yongting Lin

On Mon, 10 Aug 2026 11:38:14 +0800 "Yongting Lin" <[email protected]> wrote:

[...]

Quoted 3 lines

IBS is not the only possible hardware source. CXL 3.2 defines CHMU for device-side hotness monitoring, and CHMU has also been discussed in the DAMON and CXL hotness-monitoring discussions.

Yes, we discussed DAMON could be extended for access check primitives other than page table accessed bit, and CHMU could be one of such primitives. The extension is in progress [1].

[...]

Quoted 2 lines

We would also be interested in helping test and compare the different approaches as implementations become available.

Please feel free to let me know if you want to test DAMON-based approaches. I will be happy to help :)

[1] https://lore.kernel.org/all/[email protected]/

Thanks,
SJ

[...]

#39 lore raw
Quoted 8 lines · SJ Park

[...]

Quoted 3 lines

IBS is not the only possible hardware source. CXL 3.2 defines CHMU for device-side hotness monitoring, and CHMU has also been discussed in the DAMON and CXL hotness-monitoring discussions.

Yes, we discussed DAMON could be extended for access check primitives other than page table accessed bit, and CHMU could be one of such primitives. The extension is in progress [1].

Thanks for the pointer. I will take a closer look at the series.

Quoted 6 lines

[...]

Quoted 2 lines

We would also be interested in helping test and compare the different approaches as implementations become available.

Please feel free to let me know if you want to test DAMON-based approaches. I will be happy to help :)

Yes, we would be interested in testing this on our CXL tiered-memory systems.

I previously built a Redis workload with controlled hot and cold memory regions, and compared DAMON's results with those reported by a WSS tool. The estimated hot and cold memory sizes differed substantially. I understand that DAMON and the WSS tool have different semantics, and it is also possible that my DAMON parameters were not optimally tuned.

Once hardware-based monitoring becomes available, I would also like to compare DAMON's page-table accessed-bit monitoring with PMU- and CHMU-based access monitoring.

Thanks,
Yongting

#40 lore raw
In reply to Andrew Morton
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <[email protected]> wrote:

This patchset introduces pghot, a subsystem for hot page tracking and promotion.

Can DAMON be used to do this sort of thing adequately?

That really depends on the goal I believe, let me expand...

DAMON can indeed migrate hot pages today via DAMOS_MIGRATE_HOT, and IIUC it can even do promotion and demotion together that auto-tunes to a target node utilization. So for someone who wants to write a per-workload, per-process tiering policy from user space, DAMON is a good fit and I don't think pghot competes with it there.

But the intent behind pghot isn't really to be another migration policy. It is to build a common in-kernel infrastructure that can act as a single promotion subsystem, fed by multiple sources, with the promotion engine and its heuristics (rate limiting, dynamic threshold) living in one place rather than in the scheduler. Hint faults are just the first source. The IBS Memory Profiler, which gives access information directly from the hardware, is the other one. CXL HMU can be another potential source. Hence the design centers on an exact, per-PFN record that any of these sources can update, rather than on sampling.

The per-page, source-driven model is where the two differ the most. DAMON estimates hotness by sampling one page per region per interval, which is low overhead and self-contained, but it may miss a hot page inside a region. pghot records every reported access at PFN granularity, so it does not miss, but in turn it relies on a source to report the access.

The other practical difference is that pghot works with no user-space agent at all. With numa_balancing=2 the promotion just happens, and in fact pghot takes over the mode-2 promotion path that currently sits in the scheduler. DAMON, outside of its built-in modules, expects a controller in user space to drive it, IIUC.

So I would see DAMON and pghot as complementary rather than one replacing the other. In fact DAMON could well become one of the sources feeding pghot, and conversely pghot's batched, rate-limited engine could serve as a migration backend.

Quoted 5 lines

Thanks, I'll await review input on this.

AI review had much to say:

	https://sashiko.dev/#/patchset/[email protected]

Thanks, will address those comments.

Regards,
Bharata.

#41 lore raw
Quoted 9 lines · On Wed, 29 Jul 2026 15:05:10 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <[email protected]> wrote:

This patchset introduces pghot, a subsystem for hot page tracking and promotion.

Can DAMON be used to do this sort of thing adequately?

That really depends on the goal I believe, let me expand...

Thank you Bharata, this clearly explains me your view. I find a few little gaps between yours and mine, though. Let me clarify below.

Quoted 17 lines

DAMON can indeed migrate hot pages today via DAMOS_MIGRATE_HOT, and IIUC it can even do promotion and demotion together that auto-tunes to a target node utilization. So for someone who wants to write a per-workload, per-process tiering policy from user space, DAMON is a good fit and I don't think pghot competes with it there.

But the intent behind pghot isn't really to be another migration policy. It is to build a common in-kernel infrastructure that can act as a single promotion subsystem, fed by multiple sources, with the promotion engine and its heuristics (rate limiting, dynamic threshold) living in one place rather than in the scheduler. Hint faults are just the first source. The IBS Memory Profiler, which gives access information directly from the hardware, is the other one. CXL HMU can be another potential source. Hence the design centers on an exact, per-PFN record that any of these sources can update, rather than on sampling.

Quoted 4 lines

The per-page, source-driven model is where the two differ the most. DAMON estimates hotness by sampling one page per region per interval, which is low overhead and self-contained, but it may miss a hot page inside a region.

DAMON can do [1] fixed granularity monitoring, down to page size. Of course it increases overhead. A few people are using DAMON in this way, though, and we are working together to optimize it.

Quoted 2 lines

pghot records every reported access at PFN granularity,
so it does not miss, but in turn it relies on a source to report the access.

DAMON is not designed to use only the single source. We started working [2] on extending DAMON for multiple sources including AMD IBS. A few working prototpyes are shared. We plan to share the first version at next LSFMMBPF.

Quoted 6 lines

The other practical difference is that pghot works with no user-space agent at all. With numa_balancing=2 the promotion just happens, and in fact pghot takes over the mode-2 promotion path that currently sits in the scheduler. DAMON, outside of its built-in modules, expects a controller in user space to drive it, IIUC.

For a level of auto-tuned hot/cold pages promotion/demotion, DAMON-based memory tiering doesn't require user-space agent, either. Only a number of simple sysfs file writes are needed at the beginning.

Quoted 3 lines

So I would see DAMON and pghot as complementary rather than one replacing the other.

I have no strong opinion.

Quoted 3 lines

In fact DAMON could well become one of the sources feeding pghot, and conversely pghot's batched, rate-limited engine could serve as a migration backend.

Makes sense to me. If there are needs, we are open to add new DAMON APIs.

[1] https://origin.kernel.org/doc/html/latest/mm/damon/faq.html#can-i-simply-monitor-page-granularity
[2] https://lore.kernel.org/all/[email protected]/

Thanks,
SJ

[...]

#42 lore raw
Quoted 60 lines · On Wed, 29 Jul 2026 06:54:18 -0700 SJ Park <[email protected]> wrote:
Quoted 9 lines · On Wed, 29 Jul 2026 15:05:10 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <[email protected]> wrote:

This patchset introduces pghot, a subsystem for hot page tracking and promotion.

Can DAMON be used to do this sort of thing adequately?

That really depends on the goal I believe, let me expand...

Thank you Bharata, this clearly explains me your view. I find a few little gaps between yours and mine, though. Let me clarify below.

Quoted 17 lines

DAMON can indeed migrate hot pages today via DAMOS_MIGRATE_HOT, and IIUC it can even do promotion and demotion together that auto-tunes to a target node utilization. So for someone who wants to write a per-workload, per-process tiering policy from user space, DAMON is a good fit and I don't think pghot competes with it there.

But the intent behind pghot isn't really to be another migration policy. It is to build a common in-kernel infrastructure that can act as a single promotion subsystem, fed by multiple sources, with the promotion engine and its heuristics (rate limiting, dynamic threshold) living in one place rather than in the scheduler. Hint faults are just the first source. The IBS Memory Profiler, which gives access information directly from the hardware, is the other one. CXL HMU can be another potential source. Hence the design centers on an exact, per-PFN record that any of these sources can update, rather than on sampling.

Quoted 4 lines

The per-page, source-driven model is where the two differ the most. DAMON estimates hotness by sampling one page per region per interval, which is low overhead and self-contained, but it may miss a hot page inside a region.

DAMON can do [1] fixed granularity monitoring, down to page size. Of course it increases overhead. A few people are using DAMON in this way, though, and we are working together to optimize it.

Quoted 2 lines

pghot records every reported access at PFN granularity,
so it does not miss, but in turn it relies on a source to report the access.

DAMON is not designed to use only the single source. We started working [2] on extending DAMON for multiple sources including AMD IBS. A few working prototpyes are shared. We plan to share the first version at next LSFMMBPF.

Quoted 6 lines

The other practical difference is that pghot works with no user-space agent at all. With numa_balancing=2 the promotion just happens, and in fact pghot takes over the mode-2 promotion path that currently sits in the scheduler. DAMON, outside of its built-in modules, expects a controller in user space to drive it, IIUC.

For a level of auto-tuned hot/cold pages promotion/demotion, DAMON-based memory tiering doesn't require user-space agent, either. Only a number of simple sysfs file writes are needed at the beginning.

Quoted 1 line
Quoted 8 lines
Quoted 2 lines

So I would see DAMON and pghot as complementary rather than one replacing the other.

I have no strong opinion.

Quoted 3 lines

In fact DAMON could well become one of the sources feeding pghot, and conversely pghot's batched, rate-limited engine could serve as a migration backend.

One thing to add for clarification. DAMON is not only for monitoring. It also provides a general access pattern based operation engine, a.k.a DAMOS. It provides auto-tuned rate limiting (a.k.a DAMOS quota) and flexible filtering.

Hence, yet another imaginable way of the collaboration is plugging things in the opposite way. pghot's multi-source access monitoring becomes additional access check primitives feeding DAMON, and DAMON serves as a migration backend with its auto-tuned rate limiting and filtering. I proposed damon_report_access() in LSFMMBPF'25, suggesting this kind of collaboration as one of possible use cases.

Quoted 12 lines

Makes sense to me. If there are needs, we are open to add new DAMON APIs.

[1] https://origin.kernel.org/doc/html/latest/mm/damon/faq.html#can-i-simply-monitor-page-granularity
[2] https://lore.kernel.org/all/[email protected]/

Thanks,
SJ

[...]

Thanks,
SJ

#43 lore raw
In reply to Andrew Morton
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <[email protected]> wrote:

This patchset introduces pghot, a subsystem for hot page tracking and promotion.

Can DAMON be used to do this sort of thing adequately?

Hi SJ,

I started comparing hot page detection and promotion aspects of DAMON and pghot through benchmark runs. Here is the first set of results from pointer chase workload.

Since I wasn't very familiar with DAMON tunables and settings, I let the AI to chose some default followed by other combinations. The test harness is developed and run by AI. Request you to go through this and let me know you feedback about the combinations and configurations tried here. Based on that I can adapt the settings for the future runs.

======================================================================
DAMON vs pghot: hot-page detection and promotion, measured (single-threaded and 64-thread workloads)
======================================================================

Summary
-------
This note compares DAMON and pghot on how well each detects a 16 GB hot set on a slow-tier node and promotes it to the fast tier, using the same pointer-chase workload on a 3-tier NUMA machine. Here DAMON refers to the in-kernel access sampling with the DAMOS migrate_hot/migrate_cold actions, and pghot refers to per-PFN hotness tracking with the kmigrated promotion thread. All runs are from one uniform harness with the same metric set.

In short, pghot promotes exactly the hot set with only a couple of sysctls, and it behaves the same way irrespective of the thread count. DAMON can reach a comparable steady-state accuracy, but only with workload-specific tuning, and (to bound the fast-tier residency) only with a full bidirectional promote-plus-demote setup. Even then, it reaches the right amount only after first over-promoting and then settling back down. Two further pitfalls are worth noting: one seemingly reasonable tuning promoted nothing at all, and the promote-only configurations over-promote heavily. The difference largely reflects region-granular sampling on the DAMON side versus per-page tracking on the pghot side.

Test setup
----------
Machine : 3 NUMA nodes. node0,node1 = DRAM w/ CPUs, 256 GB each (top
tier). node2 = CPU-less slow tier, 256 GB, SLIT distance 50 (asymmetric 255). phys 0x8100000000..0xc100000000.
Kernels : base = 7.2.0-rc6 (DAMON; NUMAB tiering)

           pghot = 7.2.0-rc6 + pghot v8+

Workload : pointer chase (dependent loads, latency bound) over a 64 GB anon
buffer; 16 GB hot subset, rest cold.
Single-threaded: one thread chases the whole 16 GB.
Multi-threaded: the buffer is split into 64 chunks of 1 GB, each thread chasing its own 256 MB hot sub-region (16 GB hot total across 64 CPUs).
Timed phase = 240 s.
Placement: buffer relocated to node2 via move_pages()
Task pinned to node0 CPUs, Target = node0.

Legend
------
lat : mean access latency, ns/access, shown first->steady (last 25%).
conv : time to within 10% of steady latency (s). '-' = never / flat. node0 : fast-tier resident of the workload at end (GB). Hot set = 16 GB. over : over-promotion relative to the 16 GB hot set.
regs : DAMON monitoring regions actually created, measured via the
damon_aggregated tracepoint (DAMON runs only).
sample/aggr : DAMON sampling interval / aggregation interval (Table B). acc/reg/filt : DAMON min_nr_accesses / max_regions / YOUNG ops-filter.
thr/NB : pghot_freq_threshold / kernel.numa_balancing.
Note : DAMON promotion is counted in pgmigrate_success (and the scheme's
sz_applied); pghot promotion in pgpromote_success and pghot_recorded_*. So placement (numastat) and latency are used for the cross-mechanism comparison. pgdemote_kswapd/pgdemote_direct are 0 in every run (no reclaim-path demotion).

======================================================================
PART 1 - SINGLE-THREADED (64 GB total / 16 GB hot)
======================================================================
(base-kernel baseline steady latency 278 ns; pghot-kernel baseline 295 ns)

Table A. Headline: best of each mechanism
-----------------------------------------

Mechanism                          lat        conv  node0  over
                                   (ns)       (s)   (GB)   (%)
---------------------------------  ---------  ----  -----  ----
baseline, no promotion              320->278     -    0.0     -
DAMON promote-only best (row D0)    290->133    33   22.4   +40
DAMON bidirectional     (row D5)    299->138    47   15.3    -4
pghot, hint-fault source (row P1)   166->129    16   16.0     0

Table B. DAMON tuning sweep (base kernel)
-----------------------------------------

id  sample/aggr  acc  reg      filt  lat       conv node0  over  regs
                                     (ns)      (s)  (GB)   (%)
--  -----------  ---  -------  ----  --------  ---- -----  ----  ----
D0  5ms/100ms     1     1,000  no    290->133    33  22.4   +40    12
D1  1ms/ 20ms     2   100,000  yes   311->279     -   0.0   n/a    14
D2  1ms/ 20ms     2 4,000,000  yes   325->280     -   0.0   n/a    13
D3  5ms/100ms     1   100,000  yes   298->123    85  34.4  +115    13
D4  5ms/100ms     1 4,000,000  yes   315->136    96  27.0   +69    14
D5  5ms/100ms     1   100,000  yes   299->138    47  15.3    -4    14  <-bidir

Notes on Table B:
- D0/D3/D4 show that raising max_regions does NOT reduce over-promotion; the over-promotion is high and varies run-to-run (+40 / +115 / +69) with no trend from the cap. The measured region count stays ~12-14 in every case (see conclusion 3).
- D1/D2 (1 ms / 20 ms window, min_nr_accesses=2) promote nothing at all.
- D5 is bidirectional (promote + a migrate_cold demote scheme on a second kdamond over node0). It ends at 15.3 GB (about the hot set) but only after first over-promoting and then demoting the excess.
- kdamond CPU over the 240 s run: D0 7 s, D1/D2 2 s, D3 12 s, D4 9 s, D5 10 s.

Table C. pghot hint-fault source (pghot kernel)
-----------------------------------------------

id  source (mask)         thr NB  lat       conv node0 over
                                  (ns)      (s)  (GB)  (%)
--  --------------------  --- --  --------  ---- ----- ----
P1  hint-fault (0x1)       1   2  166->129    16  16.0    0
P2  hint-fault (0x1)       2   2  388->129    85  16.0    0

vmstat deltas (pghot runs):
- P1 pgpromote_success=1,195,791 (*partial - see below).
- P2 pgpromote_success=4,194,304 (=16 GB exactly),

     pghot_recorded_hintfaults=5,307,760, numa_hint_faults=5,307,760.

* P1 (thr=1) promotes so quickly that most of it happens during the load phase, before the vmstat baseline snapshot, so its delta is partial; placement (16 GB exact) is the reliable figure.

======================================================================
PART 2 - MULTITHREADED (64 threads, 64 GB / 16 GB hot)
======================================================================
(base-kernel baseline steady latency 251 ns; pghot-kernel baseline 282 ns)

Here latency does not track placement linearly, so it does not rank the configs on its own: partial promotion can yield most of the latency gain, and the two baselines differ by kernel. The mechanism was not measured (bandwidth and/or working-set/TLB effects are both plausible), so placement is used as the accuracy metric.

Table D. Multithreaded results
------------------------------
id config kern lat node0 over regs

                                   (ns) (GB)
--  ----------------------  -----  ---  -----  -----  ------
M0  baseline                 base   251   0.0     -        -
M1  DAMON promo-only 1k      base   124  63.4  +296%     351
M2  DAMON promo-only 100k+y  base   120  61.3  +283%      52
M3  DAMON bidir 1k           base   124  31.1   +95%     588
M4  DAMON bidir 100k+y       base   136  17.1    +7%  21,101
M5  pghot hintfault thr1     pghot  129  16.0     0%       -

Notes on Table D:
- Promotion (vmstat): M1 pgmig 14.2M, M2 15.2M, M3 14.9M, M4 18.4M; M5 pgpromote_success 4,112,464 (pghot_recorded_hintfaults 2,490,368).
- M1/M2 promote-only over-promote to nearly the whole 64 GB buffer. With no demotion, one-way promotion plus region-boundary false positives keep accumulating over the run.
- M3/M4 add a migrate_cold demote scheme (a second kdamond over node0). M4 (fine regions) ends at 17.1 GB (about the hot set) after a mid-run overshoot and a long settle.
- Region counts are much higher than single-threaded, likely because the 64 per-thread hot chunks create access-rate gradients that DAMON splits on. M4 reached 21,101 regions and consequently used 146 s of kdamond CPU over the run (vs 17-25 s for M1-M3); that is still far coarser than page size and it still over-promotes (+7%).
- M5 behaves the same as the single-threaded pghot run (not sensitive to the thread count).

Table E. DAMON migration & demotion counters (all DAMON runs)
-------------------------------------------------------------
pgdemote_kswapd and pgdemote_direct are 0 in every DAMON run: DAMON's demotion (the bidirectional migrate_cold scheme) is explicit migration, counted in pgmigrate_success and the scheme's sz_applied, not reclaim-path demotion. The reclaim counters would move only under fast-tier memory pressure (the overcommit case, left to a separate test).

run               pgmigrate_success  pgdem_kswapd  pgdem_direct  demote GB
----------------  -----------------  ------------  ------------  ---------
st-d0 (po,1k)             6,293,366             0             0          -
st-d1 (nil)                      0             0             0          -
st-d2 (nil)                      0             0             0          -
st-d3 (po,100k)          8,573,820             0             0          -
st-d4 (po,4M)            7,081,247             0             0          -
st-d5 (bidir)            8,607,955             0             0        9.6
mt-d0 (po,1k)           14,182,546             0             0          -
mt-fine (po,100k)       15,160,909             0             0          -
mt-bidir-coarse         14,889,977             0             0       13.7
mt-bidir-fine           18,356,299             0             0       26.5

(po = promote-only; demote GB = demote scheme sz_applied, bidir only. For
promote-only, pgmigrate_success is the promotion count; for bidir it is promotion + demotion combined. E.g. mt-bidir-fine promoted 43.6 GB and demoted 26.5 GB to net-place 17.1 GB.)

======================================================================
Conclusions
======================================================================

1. DAMON needs considerably more, and less obvious, tuning than pghot. A seemingly reasonable "faster and stricter" DAMON (1 ms sample, 20 ms aggr, min_nr_accesses=2) promoted nothing at all (D1/D2). This is consistent with the per-page reuse distance (~1 s for the single-threaded chase) far exceeding the 20 ms aggregation window, so few regions reach 2 accesses within a window and DAMON treats them as cold. Working operation needed a 100 ms window and min_nr_accesses=1.
pghot worked in every configuration with numa_balancing=2 (+ optional pghot_freq_threshold).

2. Region sampling over-promotes; per-PFN tracking is exact. Single-threaded, every promote-only DAMON config over-promoted, by a variable amount with no trend from the region cap (D0/1k=+40%, D3/100k=+115%, D4/4M=+69%); pghot placed exactly 16 GB in every hint-fault run. Adding a demote scheme (D5, bidirectional) brings single-thread DAMON to about 16 GB (15.3 GB), but it gets there by promoting too much and then demoting the excess. Multithreaded, promote-only DAMON over-promotes to ~62-63 GB (+283-296%); only the bidirectional fine config bounds it near the hot set (17.1 GB, +7%).

3. Setting the region size to the page size does not close the gap in practice. I measured the number of monitoring regions DAMON actually created, using the damon_aggregated tracepoint. Single-threaded, DAMON created only about 12-15 regions in every case (max_regions = 1,000 / 100,000 / 4,000,000), so the cap is never the binding constraint, and kdamond used only 2-12 s of CPU. Multithreaded, DAMON splits more - 52 to 588 regions for most configs, and 21,101 for the bidirectional-fine run (likely because the per-thread hot chunks create access-rate gradients) - but that is still far coarser than page size, and it cost 146 s of kdamond CPU while still over-promoting (+7%). Reaching page-sized regions over the 256 GB slow tier would need about 67 M regions (the 16 GB hot set alone ~4 M), i.e. around 4 GB of region structures walked by a single kdamond every aggregation. DAMON does not create anywhere near that many, and the rising kdamond CPU (146 s at 21,101 regions versus 2-25 s otherwise) is consistent with the cost of pushing toward that count.

4. Using many threads helps DAMON's detection speed but not its accuracy. The higher access density lets DAMON detect quickly (latency converges in ~10-15 s vs 33-96 s single-threaded). But promote-only still over-promotes to nearly the whole buffer, and bounding the residency still requires the bidirectional promote-plus-demote design. Even then M4 reaches 17.1 GB only after an overshoot and a long settle, with heavy churn (pgmigrate_success 18.4 M, far more than the 4 M pages of the hot set) and 146 s of kdamond CPU. pghot hint-fault places exactly 16 GB with the same behaviour as single-threaded, i.e. it is not sensitive to the thread count.

Overall, pghot provides exact, prompt promotion with essentially no tuning, at the per-page granularity that its sources give it, and it is not sensitive to the thread count. DAMON can approach the same steady-state accuracy, but only with workload-specific tuning and a full bidirectional setup; it can easily be mis-tuned into doing nothing, promote-only over-promotes heavily, it cannot in practice be pushed down to page-sized regions, and in the multithreaded case it reaches the right answer only after an over-promote-and-settle transient with substantial migration churn and kdamond CPU.

======================================================================
Methodology caveats
======================================================================
- Latency is the primary metric for this pointer-chase workload, but in the multithreaded case it does not track placement linearly (baselines differ by kernel; partial promotion can lower the latency out of proportion to how much was promoted). This was not root-caused - bandwidth and/or working-set/TLB effects are both plausible and were not measured - so it must be read alongside placement (numastat).
- Over-promotion figures vary by several GB run-to-run; the conclusions are stable but the exact GB values are approximate.
- DAMON and pghot increment different vmstat counters; the cross-mechanism comparison relies on placement and latency, which are mechanism-independent.
- vmstat counters are snapshotted around the 240 s timed phase (after the move_pages load phase), and hence exclude the relocation migrations.
- The overcommit case (hot set larger than the fast-tier capacity, which would exercise demotion under pressure) is left to a separate test.

Regards,
Bharata.

#44 lore raw
Quoted 14 lines · On Thu, 6 Aug 2026 11:19:22 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <[email protected]> wrote:

This patchset introduces pghot, a subsystem for hot page tracking and promotion.

Can DAMON be used to do this sort of thing adequately?

Hi SJ,

I started comparing hot page detection and promotion aspects of DAMON and pghot through benchmark runs. Here is the first set of results from pointer chase workload.

Quoted 5 lines

Since I wasn't very familiar with DAMON tunables and settings, I let the AI to chose some default followed by other combinations. The test harness is developed and run by AI. Request you to go through this and let me know you feedback about the combinations and configurations tried here. Based on that I can adapt the settings for the future runs.

This sounds like you asked AI to do the comparison because you don't know DAMON. And you still don't know DAMON, so you cannot really explain the results. So asking me to review what your AI did and guide your AI. Sigh.

Correct me if I'm wrong. If I'm not wrong, no, I'm not an AI slop reviewer. I will not review that until this is clarified. I call it AI slop regardless of the output when it is something that generated and shared without full understanding from the human user.

If you want me to review something, please send me what you really understand. To compare/discuss DAMON, please learn it first. Spend time and ask question. I will be more than happy to help that.

Thanks,
SJ

[...]

#45 lore raw
Quoted 26 lines · SJ Park
Quoted 14 lines · On Thu, 6 Aug 2026 11:19:22 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <[email protected]> wrote:

This patchset introduces pghot, a subsystem for hot page tracking and promotion.

Can DAMON be used to do this sort of thing adequately?

Hi SJ,

I started comparing hot page detection and promotion aspects of DAMON and pghot through benchmark runs. Here is the first set of results from pointer chase workload.

Quoted 5 lines

Since I wasn't very familiar with DAMON tunables and settings, I let the AI to chose some default followed by other combinations. The test harness is developed and run by AI. Request you to go through this and let me know you feedback about the combinations and configurations tried here. Based on that I can adapt the settings for the future runs.

This sounds like you asked AI to do the comparison because you don't know DAMON. And you still don't know DAMON, so you cannot really explain the results. So asking me to review what your AI did and guide your AI. Sigh.

Quoted 8 lines

Correct me if I'm wrong. If I'm not wrong, no, I'm not an AI slop reviewer. I will not review that until this is clarified. I call it AI slop regardless of the output when it is something that generated and shared without full understanding from the human user.

If you want me to review something, please send me what you really understand. To compare/discuss DAMON, please learn it first. Spend time and ask question. I will be more than happy to help that.

I was seeking feedback about the combinations and configurations that were tried so that I can adapt the setting for future runs. I spent time looking at the rich set of configuration possibilities [1] that DAMON provides and experimented with something very basic to start with, followed by a few changes in each step. Now I can see more meaningful numbers from DAMON without the need to enable migrate_cold.

Metric notes
------------
- lat : steady access latency, ns/access, mean of last 25% of samples (lower is better). Baselines: base 337 ns, pghot 348 ns (within 3%, so absolute latency is broadly comparable here).
- node0: end-of-run per-process placement on node 0 (numastat -p).
- CPU : cumulative cputime of the monitor/promoter thread over the 240 s run, and its peak instantaneous %cpu (ps). This is the concentrated cost only; see caveat 2 on pghot's distributed cost.
- promoted: bytes actually migrated (DAMON scheme sz_applied; pghot pgpromote_success).

DAMON configuration method (default vs manually set)
----------------------------------------------------
The aim was to start from DAMON's own sysfs defaults and set only what a promote scheme needs, then change one thing at a time.

Left at the DAMON default in every run below:
sample=5ms, aggr=100ms, update=60s, apply_interval=0 (=aggr), intervals_goal off, watermarks none, quota goals none, filters none. min_nr_regions=10 (except the page-size runs).

Set manually in every run (the minimum for migrate_hot to act):
operations=paddr; one target region = node 2 physical range; action=migrate_hot; target_nid=0; access pattern nr_accesses>=1 (sz>=1 page, any age), i.e. "promote anything accessed at least once in the aggregation window".

Per-run additions (the single variable that changes each row):
D0 default : nothing extra (pure default + migrate_hot).
D1 page-size : min=max nr_regions = 67,108,864 (one region per
4 KiB page over the 256 GiB target).
D2 +quota : quota bytes=128 MiB, reset=1000 ms (~128 MB/s),
weights sz/acc/age = 0/1000/0 (hottest first).
D3 fine+quota : max_nr_regions=100,000 (min default 10) + D2 quota.
D4 pagesz+q : page-size regions (as D1) + a quota (1 GiB/1000 ms)
+ the same weights.

Table 1. DAMON, base kernel (each row changes one variable)
------------------------------------------------------------
id config regions quota lat node0 over kdamond

                        (settled/pin) (MB/s)  (ns)  (GB)          CPU
--  -----------------  -------------  ------  ----  -----  -----  ---------
B0  baseline           - (no DAMON)   none    337   0.0    -      -
D0  default (coarse)   11             none    121   64.0  +300%  25s/10%
D1  page-size          64M pin        none    129   33.0  +106%  277s/100%
D2  default + quota    14             128     105   23.3   +46%  9s/3.7%
D3  fine + quota       14-38          128      88   23.6   +47%  14s/5.6%

D4 page-size + quota 64M pin ~1000 296 1.0 -94% 270s/100%

B0 is the no-promotion control on the base kernel (numa_balancing=0, no DAMON), and is the reference for the lat column.

The "over" column is measured against the 16 GiB HOT SET, not against
B0: over = (node0 - 16 GiB) / 16 GiB. B0 promotes nothing, so its node0 is 0.0 and its over is n/a.

Regions column:
- A plain number (D0/D2/D3) is the SETTLED count DAMON adapted to, measured via the damon_aggregated tracepoint. These rows leave min_nr_regions at the default 10; D3 only raises the max_nr_regions ceiling to 100,000 (a cap, not a target), so DAMON splits/merges freely and settles at 14-38.
- "64M pin" (D1/D4) means min_nr_regions = max_nr_regions = 67,108,864 (one 4 KiB region per page of the 256 GiB target). Setting min=max PINS the count: DAMON cannot merge/split, so it is forced, not settled. The actual count was unmeasurable (NA) because the pegged kdamond never completed an aggregation pass to emit the tracepoint.

Table 2. pghot, pghot kernel
-----------------------------

id  config            thr  lat  node0  over  promoted  kmigrated
                           (ns) (GB)         (GiB)     CPU
--  ----------------  ---  ---  -----  ----  --------  ---------
P0  baseline          -    348   0.0    -      -       0s
P2  hint-fault (def)  2    124  16.0    0%   16.0      6s/0.7%
P1  hint-fault        1    112  16.0    0%   15.9      13s/1.1%

Notes on Table 2:
- numa_balancing=2, and pghot tunables with the default values (pghot_enabled_sources=1 (hint-fault source), pghot_target_nid=0 and pghot_promote_rate_limit_MBps)
- Both thresholds place exactly the 16 GiB hot set (0% over):

    P2 pgpromote_success 4,194,304 = 16.0 GiB exactly;
    P1 pgpromote_success 4,157,158 (15.9 GiB);

- thr=1 converges faster (first interval already 136 ns; it promotes during the load phase) and is slightly quicker (112 vs 124 ns) at marginally higher kmigrated CPU. thr=2 starts at 395 ns and converges over the run.

Caveats
-------
1. Two kernels: DAMON on base, pghot on pghot. The baselines are within 3% (337 vs 348 ns), so absolute latency is broadly comparable, but this is not a single-kernel A/B.
2. The CPU columns are the concentrated monitor/promoter thread only. pghot's hint-fault source also imposes distributed overhead on the application threads (NUMA scanning + hint-fault handling:
numa_pte_updates 10-15M, numa_hint_faults 3-8M), which is NOT in the kmigrated figure. DAMON's cost is concentrated in kdamond and is captured. A fair total-cost comparison must add pghot's NUMA- balancing tax.
3. Latency does not track placement linearly for this workload and the mechanism was not isolated; placement/over-promotion is the primary accuracy metric.
4. DAMON and pghot increment different counters (scheme sz_applied vs pgpromote_success); cross-mechanism promotion counts are not directly comparable, so placement (numastat) is used.

[1] DAMON config directory - tree view
--------------------------------------

root@host:/sys/kernel/mm/damon/admin# tree
.
└── kdamonds
    ├── 0
    │   ├── contexts
    │   │   ├── 0
    │   │   │   ├── addr_unit
    │   │   │   ├── avail_operations
    │   │   │   ├── monitoring_attrs
    │   │   │   │   ├── intervals
    │   │   │   │   │   ├── aggr_us
    │   │   │   │   │   ├── intervals_goal
    │   │   │   │   │   │   ├── access_bp
    │   │   │   │   │   │   ├── aggrs
    │   │   │   │   │   │   ├── max_sample_us
    │   │   │   │   │   │   └── min_sample_us
    │   │   │   │   │   ├── sample_us
    │   │   │   │   │   └── update_us
    │   │   │   │   └── nr_regions
    │   │   │   │       ├── max
    │   │   │   │       └── min
    │   │   │   ├── operations
    │   │   │   ├── schemes
    │   │   │   │   ├── 0
    │   │   │   │   │   ├── access_pattern
    │   │   │   │   │   │   ├── age
    │   │   │   │   │   │   │   ├── max
    │   │   │   │   │   │   │   └── min
    │   │   │   │   │   │   ├── nr_accesses
    │   │   │   │   │   │   │   ├── max
    │   │   │   │   │   │   │   └── min
    │   │   │   │   │   │   └── sz
    │   │   │   │   │   │       ├── max
    │   │   │   │   │   │       └── min
    │   │   │   │   │   ├── action
    │   │   │   │   │   ├── apply_interval_us
    │   │   │   │   │   ├── core_filters
    │   │   │   │   │   │   └── nr_filters
    │   │   │   │   │   ├── dests
    │   │   │   │   │   │   └── nr_dests
    │   │   │   │   │   ├── filters
    │   │   │   │   │   │   └── nr_filters
    │   │   │   │   │   ├── ops_filters
    │   │   │   │   │   │   └── nr_filters
    │   │   │   │   │   ├── quotas
    │   │   │   │   │   │   ├── bytes
    │   │   │   │   │   │   ├── effective_bytes
    │   │   │   │   │   │   ├── goals
    │   │   │   │   │   │   │   └── nr_goals
    │   │   │   │   │   │   ├── goal_tuner
    │   │   │   │   │   │   ├── ms
    │   │   │   │   │   │   ├── reset_interval_ms
    │   │   │   │   │   │   └── weights
    │   │   │   │   │   │       ├── age_permil
    │   │   │   │   │   │       ├── nr_accesses_permil
    │   │   │   │   │   │       └── sz_permil
    │   │   │   │   │   ├── stats
    │   │   │   │   │   │   ├── max_nr_snapshots
    │   │   │   │   │   │   ├── nr_applied
    │   │   │   │   │   │   ├── nr_snapshots
    │   │   │   │   │   │   ├── nr_tried
    │   │   │   │   │   │   ├── qt_exceeds
    │   │   │   │   │   │   ├── sz_applied
    │   │   │   │   │   │   ├── sz_ops_filter_passed
    │   │   │   │   │   │   └── sz_tried
    │   │   │   │   │   ├── target_nid
    │   │   │   │   │   ├── tried_regions
    │   │   │   │   │   │   └── total_bytes
    │   │   │   │   │   └── watermarks
    │   │   │   │   │       ├── high
    │   │   │   │   │       ├── interval_us
    │   │   │   │   │       ├── low
    │   │   │   │   │       ├── metric
    │   │   │   │   │       └── mid
    │   │   │   │   └── nr_schemes
    │   │   │   └── targets
    │   │   │       ├── 0
    │   │   │       │   ├── obsolete_target
    │   │   │       │   ├── pid_target
    │   │   │       │   └── regions
    │   │   │       │       ├── 0
    │   │   │       │       │   ├── end
    │   │   │       │       │   └── start
    │   │   │       │       └── nr_regions
    │   │   │       └── nr_targets
    │   │   └── nr_contexts
    │   ├── pid
    │   ├── refresh_ms
    │   └── state
    └── nr_kdamonds

29 directories, 60 files

#46 lore raw
Quoted 145 lines · On Mon, 10 Aug 2026 10:16:37 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 26 lines · SJ Park
Quoted 14 lines · On Thu, 6 Aug 2026 11:19:22 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 2 lines · On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <[email protected]> wrote:

This patchset introduces pghot, a subsystem for hot page tracking and promotion.

Can DAMON be used to do this sort of thing adequately?

Hi SJ,

I started comparing hot page detection and promotion aspects of DAMON and pghot through benchmark runs. Here is the first set of results from pointer chase workload.

Quoted 5 lines

Since I wasn't very familiar with DAMON tunables and settings, I let the AI to chose some default followed by other combinations. The test harness is developed and run by AI. Request you to go through this and let me know you feedback about the combinations and configurations tried here. Based on that I can adapt the settings for the future runs.

This sounds like you asked AI to do the comparison because you don't know DAMON. And you still don't know DAMON, so you cannot really explain the results. So asking me to review what your AI did and guide your AI. Sigh.

Quoted 8 lines

Correct me if I'm wrong. If I'm not wrong, no, I'm not an AI slop reviewer. I will not review that until this is clarified. I call it AI slop regardless of the output when it is something that generated and shared without full understanding from the human user.

If you want me to review something, please send me what you really understand. To compare/discuss DAMON, please learn it first. Spend time and ask question. I will be more than happy to help that.

I was seeking feedback about the combinations and configurations that were tried so that I can adapt the setting for future runs. I spent time looking at the rich set of configuration possibilities [1] that DAMON provides and experimented with something very basic to start with, followed by a few changes in each step. Now I can see more meaningful numbers from DAMON without the need to enable migrate_cold.

Metric notes
------------
- lat : steady access latency, ns/access, mean of last 25% of samples (lower is better). Baselines: base 337 ns, pghot 348 ns (within 3%, so absolute latency is broadly comparable here).
- node0: end-of-run per-process placement on node 0 (numastat -p).
- CPU : cumulative cputime of the monitor/promoter thread over the 240 s run, and its peak instantaneous %cpu (ps). This is the concentrated cost only; see caveat 2 on pghot's distributed cost.
- promoted: bytes actually migrated (DAMON scheme sz_applied; pghot pgpromote_success).

DAMON configuration method (default vs manually set)
----------------------------------------------------
The aim was to start from DAMON's own sysfs defaults and set only what a promote scheme needs, then change one thing at a time.

Left at the DAMON default in every run below:
sample=5ms, aggr=100ms, update=60s, apply_interval=0 (=aggr), intervals_goal off, watermarks none, quota goals none, filters none. min_nr_regions=10 (except the page-size runs).

Set manually in every run (the minimum for migrate_hot to act):
operations=paddr; one target region = node 2 physical range; action=migrate_hot; target_nid=0; access pattern nr_accesses>=1 (sz>=1 page, any age), i.e. "promote anything accessed at least once in the aggregation window".

Per-run additions (the single variable that changes each row):
D0 default : nothing extra (pure default + migrate_hot).
D1 page-size : min=max nr_regions = 67,108,864 (one region per
4 KiB page over the 256 GiB target).
D2 +quota : quota bytes=128 MiB, reset=1000 ms (~128 MB/s),
weights sz/acc/age = 0/1000/0 (hottest first).
D3 fine+quota : max_nr_regions=100,000 (min default 10) + D2 quota.
D4 pagesz+q : page-size regions (as D1) + a quota (1 GiB/1000 ms)
+ the same weights.

Table 1. DAMON, base kernel (each row changes one variable)
------------------------------------------------------------
id config regions quota lat node0 over kdamond

                        (settled/pin) (MB/s)  (ns)  (GB)          CPU
--  -----------------  -------------  ------  ----  -----  -----  ---------
B0  baseline           - (no DAMON)   none    337   0.0    -      -
D0  default (coarse)   11             none    121   64.0  +300%  25s/10%
D1  page-size          64M pin        none    129   33.0  +106%  277s/100%
D2  default + quota    14             128     105   23.3   +46%  9s/3.7%
D3  fine + quota       14-38          128      88   23.6   +47%  14s/5.6%

D4 page-size + quota 64M pin ~1000 296 1.0 -94% 270s/100%

B0 is the no-promotion control on the base kernel (numa_balancing=0, no DAMON), and is the reference for the lat column.

The "over" column is measured against the 16 GiB HOT SET, not against
B0: over = (node0 - 16 GiB) / 16 GiB. B0 promotes nothing, so its node0 is 0.0 and its over is n/a.

Regions column:
- A plain number (D0/D2/D3) is the SETTLED count DAMON adapted to, measured via the damon_aggregated tracepoint. These rows leave min_nr_regions at the default 10; D3 only raises the max_nr_regions ceiling to 100,000 (a cap, not a target), so DAMON splits/merges freely and settles at 14-38.
- "64M pin" (D1/D4) means min_nr_regions = max_nr_regions = 67,108,864 (one 4 KiB region per page of the 256 GiB target). Setting min=max PINS the count: DAMON cannot merge/split, so it is forced, not settled. The actual count was unmeasurable (NA) because the pegged kdamond never completed an aggregation pass to emit the tracepoint.

Table 2. pghot, pghot kernel
-----------------------------

id  config            thr  lat  node0  over  promoted  kmigrated
                           (ns) (GB)         (GiB)     CPU
--  ----------------  ---  ---  -----  ----  --------  ---------
P0  baseline          -    348   0.0    -      -       0s
P2  hint-fault (def)  2    124  16.0    0%   16.0      6s/0.7%
P1  hint-fault        1    112  16.0    0%   15.9      13s/1.1%

Notes on Table 2:
- numa_balancing=2, and pghot tunables with the default values (pghot_enabled_sources=1 (hint-fault source), pghot_target_nid=0 and pghot_promote_rate_limit_MBps)
- Both thresholds place exactly the 16 GiB hot set (0% over):

    P2 pgpromote_success 4,194,304 = 16.0 GiB exactly;
    P1 pgpromote_success 4,157,158 (15.9 GiB);

- thr=1 converges faster (first interval already 136 ns; it promotes during the load phase) and is slightly quicker (112 vs 124 ns) at marginally higher kmigrated CPU. thr=2 starts at 395 ns and converges over the run.

Caveats
-------
1. Two kernels: DAMON on base, pghot on pghot. The baselines are within 3% (337 vs 348 ns), so absolute latency is broadly comparable, but this is not a single-kernel A/B.
2. The CPU columns are the concentrated monitor/promoter thread only. pghot's hint-fault source also imposes distributed overhead on the application threads (NUMA scanning + hint-fault handling:
numa_pte_updates 10-15M, numa_hint_faults 3-8M), which is NOT in the kmigrated figure. DAMON's cost is concentrated in kdamond and is

Quoted 8 lines

captured. A fair total-cost comparison must add pghot's NUMA- balancing tax.
3. Latency does not track placement linearly for this workload and the mechanism was not isolated; placement/over-promotion is the primary accuracy metric.
4. DAMON and pghot increment different counters (scheme sz_applied vs pgpromote_success); cross-mechanism promotion counts are not directly comparable, so placement (numastat) is used.

Very nice data, thank you for sharing!

A few things would better to be clarified in my humble opinion.

What are the intentions and expectations behind the different configurations? What is the conclusion you get from the test results? Depending on this, the test workload and machine setup would also have rooms to better be clarified.

Also it is not very clear if you still asking my feedback or not. If you still want my feedback, could you pleae andd the clarifications to questions I added above? That would be helpful for providing more meaningful feedback for you.

Quoted 94 lines

[1] DAMON config directory - tree view
--------------------------------------

root@host:/sys/kernel/mm/damon/admin# tree
.
└── kdamonds
    ├── 0
    │   ├── contexts
    │   │   ├── 0
    │   │   │   ├── addr_unit
    │   │   │   ├── avail_operations
    │   │   │   ├── monitoring_attrs
    │   │   │   │   ├── intervals
    │   │   │   │   │   ├── aggr_us
    │   │   │   │   │   ├── intervals_goal
    │   │   │   │   │   │   ├── access_bp
    │   │   │   │   │   │   ├── aggrs
    │   │   │   │   │   │   ├── max_sample_us
    │   │   │   │   │   │   └── min_sample_us
    │   │   │   │   │   ├── sample_us
    │   │   │   │   │   └── update_us
    │   │   │   │   └── nr_regions
    │   │   │   │       ├── max
    │   │   │   │       └── min
    │   │   │   ├── operations
    │   │   │   ├── schemes
    │   │   │   │   ├── 0
    │   │   │   │   │   ├── access_pattern
    │   │   │   │   │   │   ├── age
    │   │   │   │   │   │   │   ├── max
    │   │   │   │   │   │   │   └── min
    │   │   │   │   │   │   ├── nr_accesses
    │   │   │   │   │   │   │   ├── max
    │   │   │   │   │   │   │   └── min
    │   │   │   │   │   │   └── sz
    │   │   │   │   │   │       ├── max
    │   │   │   │   │   │       └── min
    │   │   │   │   │   ├── action
    │   │   │   │   │   ├── apply_interval_us
    │   │   │   │   │   ├── core_filters
    │   │   │   │   │   │   └── nr_filters
    │   │   │   │   │   ├── dests
    │   │   │   │   │   │   └── nr_dests
    │   │   │   │   │   ├── filters
    │   │   │   │   │   │   └── nr_filters
    │   │   │   │   │   ├── ops_filters
    │   │   │   │   │   │   └── nr_filters
    │   │   │   │   │   ├── quotas
    │   │   │   │   │   │   ├── bytes
    │   │   │   │   │   │   ├── effective_bytes
    │   │   │   │   │   │   ├── goals
    │   │   │   │   │   │   │   └── nr_goals
    │   │   │   │   │   │   ├── goal_tuner
    │   │   │   │   │   │   ├── ms
    │   │   │   │   │   │   ├── reset_interval_ms
    │   │   │   │   │   │   └── weights
    │   │   │   │   │   │       ├── age_permil
    │   │   │   │   │   │       ├── nr_accesses_permil
    │   │   │   │   │   │       └── sz_permil
    │   │   │   │   │   ├── stats
    │   │   │   │   │   │   ├── max_nr_snapshots
    │   │   │   │   │   │   ├── nr_applied
    │   │   │   │   │   │   ├── nr_snapshots
    │   │   │   │   │   │   ├── nr_tried
    │   │   │   │   │   │   ├── qt_exceeds
    │   │   │   │   │   │   ├── sz_applied
    │   │   │   │   │   │   ├── sz_ops_filter_passed
    │   │   │   │   │   │   └── sz_tried
    │   │   │   │   │   ├── target_nid
    │   │   │   │   │   ├── tried_regions
    │   │   │   │   │   │   └── total_bytes
    │   │   │   │   │   └── watermarks
    │   │   │   │   │       ├── high
    │   │   │   │   │       ├── interval_us
    │   │   │   │   │       ├── low
    │   │   │   │   │       ├── metric
    │   │   │   │   │       └── mid
    │   │   │   │   └── nr_schemes
    │   │   │   └── targets
    │   │   │       ├── 0
    │   │   │       │   ├── obsolete_target
    │   │   │       │   ├── pid_target
    │   │   │       │   └── regions
    │   │   │       │       ├── 0
    │   │   │       │       │   ├── end
    │   │   │       │       │   └── start
    │   │   │       │       └── nr_regions
    │   │   │       └── nr_targets
    │   │   └── nr_contexts
    │   ├── pid
    │   ├── refresh_ms
    │   └── state
    └── nr_kdamonds
Quoted 1 line

29 directories, 60 files

I'd suggest referencing the document [1] instead.

[1] https://origin.kernel.org/doc/html/latest/admin-guide/mm/damon/usage.html

Thanks,
SJ

[...]

reviewer
#47 lore raw
In reply to Bharata B Rao
Quoted 26 lines · On Tue, 28 Jul 2026 11:13:48 +0530 Bharata B Rao <[email protected]> wrote:

Hi,

This patchset introduces pghot, a subsystem for hot page tracking and promotion. It collects memory access information from multiple sources, classifies hot pages resident in lower-tier memory, and promotes them to faster tiers via a per-lower-tier-node kernel thread (kmigrated).

The main changes in v8 are a rework of the per-section hotness map to be RCU-protected, a move of the pghot tunables from debugfs to sysctl, and a rework of the AMD IBS Memory Profiler driver that now uses a dedicated system interrupt vector instead of an NMI handler. The IBS driver is also split into core infrastructure and a separate patch that adds the runtime controls and arms the profiler. Also there are a few fixes to the hint faults source.

Goals of this patchset:

- Unify hot page detection from multiple sources like hint faults, page table scans, hardware hints (AMD IBS).
- Decouple detection from migration.
- Centralize promotion logic via per-lower-tier-node kmigrated kernel thread.
- Move promotion rate-limiting and related logic used by numa_balancing=2 (NUMAB2, the current NUMA balancing-based promotion) from the scheduler to pghot for broader reuse.

Quoted 6 lines

Currently, multiple kernel subsystems detect page accesses independently. This patchset consolidates accesses from these mechanisms by providing:

- A common API for reporting page accesses.
- Shared infrastructure for tracking hotness at PFN granularity.
- Per-lower-tier-node kernel threads for promoting pages.

Hi Bharata,

We've been testing pghot in a Meta kernel with worklaods to try and see what the performance characteristics look like. I wanted to report some of the findings that we've found in our testing. Note that these results are from a modified kernel that has my tiered memcg series [1] backported, as well as Gregory's numa balancing scanning fixes [2] as well.

- Without tuning and using the default tunables that pghot comes with, we've found around a slight performance regression against raw NUMAB2. This was being run on a host with 750G DRAM and 250G CXL.
- However, we've found lower pgscan_direct, pgpromote, and pgdemote rates as well.
- We've seen PSI dramatically drop on the host as well as the promotion happens asynchronously.
- I also found a bug where the kmigrated thread isn't created for nodes that are onlined later after boot, like CXL nodes that get onlined with memory later on. I've been using a small hack commit in pghot_memhp_callback that just checks for the MEM_ONLINE case and spawns a kmigrated thread.

Finally, we've seen some issues with pghot causing userspace crashes, which leads us to believe that there might be some bugs that manifest in userspace as well (we have not been able to find out where in pghot these bugs are introduced).

Thanks for working on pghot!
Joshua

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

#48 lore raw
Quoted 8 lines · Joshua Hahn

Hi Bharata,

We've been testing pghot in a Meta kernel with worklaods to try and see what the performance characteristics look like. I wanted to report some of the findings that we've found in our testing. Note that these results are from a modified kernel that has my tiered memcg series [1] backported, as well as Gregory's numa balancing scanning fixes [2] as well.

Hi Joshua,

Thanks for testing pghot with your workload!

Quoted 4 lines

- Without tuning and using the default tunables that pghot comes with, we've found around a slight performance regression against raw NUMAB2. This was being run on a host with 750G DRAM and 250G CXL.

pghot promotes on 2nd access by default since pghot_freq_threshold is 2 by default. This will result in slower promotions compared to NUMAB2 which promotes on 1st access. Could you please try with setting pghot_freq_threshold to 1? I think I should make the default as 1 going forward.

In any case, will it be possible to provide more information on what your workloads are, and the exact performance metrics? Any representative workload that can be substituted to reproduce this behaviour can also help.

And when you say slight perf regression, how much is it exactly?

Quoted 4 lines

- However, we've found lower pgscan_direct, pgpromote, and pgdemote rates as well.
- We've seen PSI dramatically drop on the host as well as the promotion happens asynchronously.

That's good to see.

Quoted 5 lines

- I also found a bug where the kmigrated thread isn't created for nodes that are onlined later after boot, like CXL nodes that get onlined with memory later on. I've been using a small hack commit in pghot_memhp_callback that just checks for the MEM_ONLINE case and spawns a kmigrated thread.

I will take care of this scenario (CXL node online after boot) in my next posting.

Quoted 5 lines

Finally, we've seen some issues with pghot causing userspace crashes, which leads us to believe that there might be some bugs that manifest in userspace as well (we have not been able to find out where in pghot these bugs are introduced).

That's something I have never seen till now with any of the benchmarks that I have run. Any kernel messages attributing the failure to pghot? Anything attributable to access reporting or kmigrated threads?

Any clues from coredump or gdb debug of userspace crash?

Were you running your workloads as part of a non-root memcg?

Do you see these failures without [1] and/or [2] ?

Quoted 6 lines

Thanks for working on pghot!
Joshua

[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/

Regards,
Bharata.

reviewer
#49 lore raw
Quoted 19 lines · On Wed, 16 Sep 2026 08:38:17 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 8 lines · Joshua Hahn

Hi Bharata,

We've been testing pghot in a Meta kernel with worklaods to try and see what the performance characteristics look like. I wanted to report some of the findings that we've found in our testing. Note that these results are from a modified kernel that has my tiered memcg series [1] backported, as well as Gregory's numa balancing scanning fixes [2] as well.

Hi Joshua,

Thanks for testing pghot with your workload!

Quoted 4 lines

- Without tuning and using the default tunables that pghot comes with, we've found around a slight performance regression against raw NUMAB2. This was being run on a host with 750G DRAM and 250G CXL.

Quoted 5 lines

pghot promotes on 2nd access by default since pghot_freq_threshold is 2 by default. This will result in slower promotions compared to NUMAB2 which promotes on 1st access. Could you please try with setting pghot_freq_threshold to 1? I think I should make the default as 1 going forward.

Gotcha. Out of curiosity, I'm wondering why you decided to make this a tunable. Specifically, I'm wondering if there are any workloads that benefit from making it require a second access. To me keeping it at first access makes a lot of sense. Obviously you could say that scanning workloads would want to use 2 accesses, but there should be other ways to prevent that from happening rather than counting the number of accesses before promotion. Just my 2c.

Quoted 5 lines

In any case, will it be possible to provide more information on what your workloads are, and the exact performance metrics? Any representative workload that can be substituted to reproduce this behaviour can also help.

And when you say slight perf regression, how much is it exactly?

I'm seeing that average latency is about 20% higher and p99 latency is about 5% higher.

Quoted 7 lines
Quoted 4 lines

- However, we've found lower pgscan_direct, pgpromote, and pgdemote rates as well.
- We've seen PSI dramatically drop on the host as well as the promotion happens asynchronously.

That's good to see.

Quoted 7 lines
Quoted 5 lines

- I also found a bug where the kmigrated thread isn't created for nodes that are onlined later after boot, like CXL nodes that get onlined with memory later on. I've been using a small hack commit in pghot_memhp_callback that just checks for the MEM_ONLINE case and spawns a kmigrated thread.

I will take care of this scenario (CXL node online after boot) in my next posting.

Thanks, it will make it easier for us to use on our end as well.

Quoted 8 lines
Quoted 4 lines

Finally, we've seen some issues with pghot causing userspace crashes, which leads us to believe that there might be some bugs that manifest in userspace as well (we have not been able to find out where in pghot these bugs are introduced).

That's something I have never seen till now with any of the benchmarks that I have run. Any kernel messages attributing the failure to pghot? Anything attributable to access reporting or kmigrated threads?

We're doing A/B testing with NUMAB2 only and pghot, and we are seeing the userspace crashes in the pghot host only. It seems like there are some segfaults trying to reference ip 000... and a warning that references "Too many tries to read user space".

I don't have anything in particular that would direct us right to pghot, but this crash has repeatedly come up, only on pghot (across reboots) and not on our NUMAB2 worklods.

Quoted 3 lines

Any clues from coredump or gdb debug of userspace crash?

Were you running your workloads as part of a non-root memcg?

They were part of a non-root memcg.

Quoted 1 line

Do you see these failures without [1] and/or [2] ?

Yup, I think these are unrelated to those changes.

Thanks again Bharata!
Joshua

#50 lore raw
Quoted 27 lines · Joshua Hahn
Quoted 19 lines · On Wed, 16 Sep 2026 08:38:17 +0530 Bharata B Rao <[email protected]> wrote:
Quoted 8 lines · Joshua Hahn

Hi Bharata,

We've been testing pghot in a Meta kernel with worklaods to try and see what the performance characteristics look like. I wanted to report some of the findings that we've found in our testing. Note that these results are from a modified kernel that has my tiered memcg series [1] backported, as well as Gregory's numa balancing scanning fixes [2] as well.

Hi Joshua,

Thanks for testing pghot with your workload!

Quoted 4 lines

- Without tuning and using the default tunables that pghot comes with, we've found around a slight performance regression against raw NUMAB2. This was being run on a host with 750G DRAM and 250G CXL.

Quoted 5 lines

pghot promotes on 2nd access by default since pghot_freq_threshold is 2 by default. This will result in slower promotions compared to NUMAB2 which promotes on 1st access. Could you please try with setting pghot_freq_threshold to 1? I think I should make the default as 1 going forward.

Quoted 7 lines

Gotcha. Out of curiosity, I'm wondering why you decided to make this a tunable. Specifically, I'm wondering if there are any workloads that benefit from making it require a second access. To me keeping it at first access makes a lot of sense. Obviously you could say that scanning workloads would want to use 2 accesses, but there should be other ways to prevent that from happening rather than counting the number of accesses before promotion. Just my 2c.

The basic idea is to consider two factors for determining the page hotness: the frequency of access and the recency of access. Hence I have kept both of them as configurable so that users can tune them for their workloads.

However since the immediate comparision of pghot-hintfaults source is with existing NUMAB2, it makes sense to have the default freq as 1 to get closer to apples-to-apples comparision by default.

Quoted 9 lines
Quoted 6 lines

In any case, will it be possible to provide more information on what your workloads are, and the exact performance metrics? Any representative workload that can be substituted to reproduce this behaviour can also help.

And when you say slight perf regression, how much is it exactly?

I'm seeing that average latency is about 20% higher and p99 latency is about 5% higher.

I hope this gap should get closed with pghot_freq_threshold=1.

Quoted 10 lines
Quoted 9 lines
Quoted 5 lines

Finally, we've seen some issues with pghot causing userspace crashes, which leads us to believe that there might be some bugs that manifest in userspace as well (we have not been able to find out where in pghot these bugs are introduced).

That's something I have never seen till now with any of the benchmarks that I have run. Any kernel messages attributing the failure to pghot? Anything attributable to access reporting or kmigrated threads?

Quoted 8 lines

We're doing A/B testing with NUMAB2 only and pghot, and we are seeing the userspace crashes in the pghot host only. It seems like there are some segfaults trying to reference ip 000... and a warning that references "Too many tries to read user space".

I don't have anything in particular that would direct us right to pghot, but this crash has repeatedly come up, only on pghot (across reboots) and not on our NUMAB2 worklods.

When you find out more about this failure, please let me know.

Regards,
Bharata.

#51 lore raw
In reply to Bharata B Rao
Quoted 8 lines · Bharata B Rao
Quoted 5 lines · Joshua Hahn

- I also found a bug where the kmigrated thread isn't created for nodes that are onlined later after boot, like CXL nodes that get onlined with memory later on. I've been using a small hack commit in pghot_memhp_callback that just checks for the MEM_ONLINE case and spawns a kmigrated thread.

I will take care of this scenario (CXL node online after boot) in my next posting.

I was also running into this for chmu integration. I have been carrying the below in my own testing, fwiw.

Thanks,
Davidlohr

-------------8<------------------------------------------

mm/pghot.c +69 -0
index c7f523a61683..2f1e16e5091d 100644
@@ -602,6 +609,18 @@ static int kmigrated_run(int nid)
 	return 0;
 }
 
+static void kmigrated_stop(int nid)
+{
+	pg_data_t *pgdat = NODE_DATA(nid);
+	struct task_struct *t;
+
+	t = pgdat->kmigrated;
+	if (t) {
+		kthread_stop(t);
+		pgdat->kmigrated = NULL;
+	}
+}
+
 static void pghot_free_hot_map(struct mem_section *ms)
 {
 	struct pghot_hot_map *hot_map;
@@ -707,6 +726,58 @@ static struct notifier_block pghot_mem_notifier = {
 	.priority = DEFAULT_CALLBACK_PRI,
 };
 
+/*
+ * The first memory block of a node is onlined before the node gets
+ * assigned to a memory tier (memory tiers react to
+ * NODE_ADDED_FIRST_MEMORY at MEMTIER_HOTPLUG_PRI), so at
+ * MEM_GOING_ONLINE time for that block node_is_toptier() still
+ * returns true and pghot_online_sec_hotmap() skips it. Backfill the
+ * hot maps once the node tier is known.
+ */
+static void pghot_alloc_node_hotmaps(int nid)
+{
+	unsigned long section_nr, s_begin, start_pfn;
+	struct mem_section *ms;
+	struct page *page;
+
+	s_begin = next_present_section_nr(-1);
+	for_each_present_section_nr(s_begin, section_nr) {
+		ms = __nr_to_section(section_nr);
+		start_pfn = section_nr_to_pfn(section_nr);
+
+		page = pfn_to_online_page(start_pfn);
+		if (!page || page_to_nid(page) != nid)
+			continue;
+
+		if (rcu_access_pointer(ms->hot_map))
+			continue;
+
+		if (pghot_alloc_hot_map(ms, nid))
+			pr_warn("pghot: no hot map for section %lu on node %d\n",
+				section_nr, nid);
+	}
+}
+
+static int pghot_node_callback(struct notifier_block *self,
+			       unsigned long action, void *arg)
+{
+	struct node_notify *nn = arg;
+
+	switch (action) {
+	case NODE_ADDED_FIRST_MEMORY:
+		if (node_is_toptier(nn->nid))
+			break;
+		pghot_alloc_node_hotmaps(nn->nid);
+		kmigrated_run(nn->nid);
+		break;
+	case NODE_REMOVED_LAST_MEMORY:
+		kmigrated_stop(nn->nid);
+		break;
+	}
+
+	return NOTIFY_OK;
+}
+
 static void pghot_destroy_hot_map(void)
 {
 	unsigned long section_nr, s_begin;
@@ -891,6 +962,11 @@ static int __init pghot_init(void)
 		if (ret)
 			goto out_stop_kthread;
 	}
+
+	ret = hotplug_node_notifier(pghot_node_callback, 0);
+	if (ret)
+		goto out_stop_kthread;
+
 	pghot_sysctl_init();
 	pghot_debug_init();
 	pghot_src_enabled_init();
Signature · 1 line
2.39.5
#52 lore raw
Quoted 88 lines · Davidlohr Bueso
Quoted 9 lines · Bharata B Rao
Quoted 5 lines · Joshua Hahn

- I also found a bug where the kmigrated thread isn't created for nodes that are onlined later after boot, like CXL nodes that get onlined with memory later on. I've been using a small hack commit in pghot_memhp_callback that just checks for the MEM_ONLINE case and spawns a kmigrated thread.

I will take care of this scenario (CXL node online after boot) in my next posting.

I was also running into this for chmu integration. I have been carrying the below in my own testing, fwiw.

Thanks,
Davidlohr

-------------8<------------------------------------------

mm/pghot.c +0 -0
index c7f523a61683..2f1e16e5091d 100644
@@ -602,6 +609,18 @@ static int kmigrated_run(int nid)
      return 0;
  }

mm/pghot.c +12 -0
+static void kmigrated_stop(int nid)
+{
+    pg_data_t *pgdat = NODE_DATA(nid);
+    struct task_struct *t;
+
+    t = pgdat->kmigrated;
+    if (t) {
+        kthread_stop(t);
+        pgdat->kmigrated = NULL;
+    }
+}
+

 static void pghot_free_hot_map(struct mem_section *ms) {
     struct pghot_hot_map *hot_map;

mm/pghot.c +0 -0
@@ -707,6 +726,58 @@ static struct notifier_block pghot_mem_notifier = {
      .priority = DEFAULT_CALLBACK_PRI,
  };

mm/pghot.c +42 -0
+/*
+ * The first memory block of a node is onlined before the node gets
+ * assigned to a memory tier (memory tiers react to
+ * NODE_ADDED_FIRST_MEMORY at MEMTIER_HOTPLUG_PRI), so at
+ * MEM_GOING_ONLINE time for that block node_is_toptier() still
+ * returns true and pghot_online_sec_hotmap() skips it. Backfill the
+ * hot maps once the node tier is known.
+ */
+static void pghot_alloc_node_hotmaps(int nid)
+{
+    unsigned long section_nr, s_begin, start_pfn;
+    struct mem_section *ms;
+    struct page *page;
+
+    s_begin = next_present_section_nr(-1);
+    for_each_present_section_nr(s_begin, section_nr) {
+        ms = __nr_to_section(section_nr);
+        start_pfn = section_nr_to_pfn(section_nr);
+
+        page = pfn_to_online_page(start_pfn);
+        if (!page || page_to_nid(page) != nid)
+            continue;
+
+        if (rcu_access_pointer(ms->hot_map))
+            continue;
+
+        if (pghot_alloc_hot_map(ms, nid))
+            pr_warn("pghot: no hot map for section %lu on node %d\n",
+                section_nr, nid);
+    }
+}
+
+static int pghot_node_callback(struct notifier_block *self,
+                   unsigned long action, void *arg)
+{
+    struct node_notify *nn = arg;
+
+    switch (action) {
+    case NODE_ADDED_FIRST_MEMORY:
+        if (node_is_toptier(nn->nid))
+            break;
+        pghot_alloc_node_hotmaps(nn->nid);

mm/pghot.c static struct notifier_block pghot_mem_notifier = { · line 780

Quoted 10 lines
mm/pghot.c +10 -0
+        kmigrated_run(nn->nid);
+        break;
+    case NODE_REMOVED_LAST_MEMORY:
+        kmigrated_stop(nn->nid);
+        break;
+    }
+
+    return NOTIFY_OK;
+}
+

I don't think we should allocate section hotmaps from node hotplug notifier callback and let it get allocated from the already existing memory hotplug notifier callback.

I have already taken care of node hotplug in my latest tree from which I posted only the IBS Memory Profiler part.

https://github.com/AMDESE/linux-mm/tree/bharata/pghot-mprof-rfcv0

Regards,
Bharata.

#53 lore raw
In reply to Bharata B Rao
Quoted 16 lines · Bharata B Rao

Performance summary
===================
All results are on 3-node tiered systems (DRAM top tier + CPU-less CXL lower tier). Speedups below are normalized to the base kernel with no tiering (NUMAB=0). Columns compare mainline hint-fault tiering (base NUMAB=2) against pghot in two modes: hint-fault driven (pghot-hf, NUMAB=2) and HW-hint driven via the AMD IBS memory profiler (pghot-hw, NUMAB=0, no NUMA scanning). Single run per config (abench is avg of 3).

Benchmark Metric (higher=better) base-NUMAB2 pghot-hf pghot-hw
-------------------------------------------------------------------------

NAS BT (MPI)   Mop/s total                  2.26x     2.26x     2.13x
Graph500 BFS   harmonic-mean TEPS           2.64x     2.49x     2.68x
llama.cpp      decode tok/s (tg128)         1.09x     1.41x     1.21x
Redis+memtier  ops/sec                      1.05x     1.05x     1.00x
Microbench     completion time (1/t)        2.41x     2.40x     2.35x
Quoted 5 lines

-------------------------------------------------------------------------
(baseline = base kernel, no tiering = 1.00x)

Summary: pghot hint-faults reproduces mainline NUMAB=2 with no regression on every workload; pghot HW-hints/IBS matches it mostly.

The IBS doesn't seem to bring anything to the table, other than just serve as an example - we've discussed this in the past this. And for NUMAB2 I would expect this to be the very best case in that you are not suffering from loosing the locality info.

This is a lot of added complexity and code just to break even. And hardware sources should really outperform NUMAB2 for this to be worthwhile, methinks.

I know Joshua mentioned some improved PSI by doing the promotion async; which makes me wonder that perhaps just a simpler promoting kthread is enough, so NUMAB2 just kicks the migration to it and not done in the accessing tasks' context etc. And Gregory also recently improved this path vs NUMAB1.

I mention this because I have integrated the chmu as a source of hotness for pghot and this was tested on a real device vs NUMAB3. This was based on Jonathan's rfc for the perf driver, only that I carve out the first chmu instance for this, and leave the rest of them (if any for perf).
For example, 'XSBench' improved by a factor of ~2.7x (I would like to report a wider range of benchmarks at some point, but this is what I have now). But a lot of this bypasses most of this framework and just piggy backs on kmigrated moving memory. chmu does not do well with pghot_record_access(), which is designed for sampling... so I just set MIGRATE_READY right away. I also added a pghot call/api to make use of the regions chmu deals with instead of per access/page. The cxl driver just takes the dpa range given by the chmu hotlist, coalesces into adjacent hpa ranges (just 1 for the case of no interleaving) and feeds that to the pghot subsystem as a list of these contiguous physical chunks ranked/sorted by access density (accesses per page per second, so a large lukewarm region and a smaller hotter one can be differentiated and promote the latter first).

So perhaps any mm interface for this stuff should just not be designed around sampling, and instead proper hardware sources?
A lot of the complexity in pghot can be removed without this imo (per-pfn metadata, accumulate+decay phase). Of course we still have the problem to evaluate the cost of replacing a chunk from the top tier(s) with the what the low tier is reporting has "hot". And this is particularly true with chmu with limited full system memory visibility (as opposed to IBS, for example). I don't really have a good answer for this, other than the user could use proactive reclaim along with the per-device tunables (ie unit sizes and thresholds for chmu) to configure things realistically for their use cases - prob. easier said than done.

Anyway these are some thoughts prior to LPC.

Thanks,
Davidlohr

#54 lore raw

Hello Davidlohr,

On Thu, 24 Sep 2026 19:12:26 -0700 Davidlohr Bueso <[email protected]> wrote:
[...]

Quoted 2 lines

So perhaps any mm interface for this stuff should just not be designed around sampling, and instead proper hardware sources?

Makes sense to me. I proposed [1] damos_add_folios() in LSFMMBPF'25 for a reason that is similar to your points in my humble view. To quote from the slide [2],

+/**
+ * damos_add_folios - Add a list of folios as highest priority target for given
+ * damos.
+ * @scheme: DAMOS scheme for the specific access-aware operation.
+ * @folios: List of folios to apply the access-aware operation.
+ *
+ * If a kernel component finds folios that eligible for a specific memory
+ * management operation (e.g., CXL-promotion or demotion), execution of the
+ * operation can be requested to be done by DAMOS using this function. The
+ * execution of the operation will be asynchronously done by DAMOS worker
+ * thread. DAMOS features for resource control (DAMOS quotas) will also be
+ * applied.
+ */
+void damos_add_folios(struct damos *scheme, struct list_head *folios)

Quoted 9 lines

A lot of the complexity in pghot can be removed without this imo (per-pfn metadata, accumulate+decay phase). Of course we still have the problem to evaluate the cost of replacing a chunk from the top tier(s) with the what the low tier is reporting has "hot". And this is particularly true with chmu with limited full system memory visibility (as opposed to IBS, for example). I don't really have a good answer for this, other than the user could use proactive reclaim along with the per-device tunables (ie unit sizes and thresholds for chmu) to configure things realistically for their use cases - prob. easier said than done.

As the above quote says, the API caller can still use DAMOS features for fine controls such as quotas, filtrers and quota auto-tuning. We could add more optimized features for the API callers. My rough idea was that such operation-focused features could help serving this kind of additional requirements.

We didn' find a real request for damos_add_folios() so far, though. And hence no progress has made for that since LSFMMBPF'25, and I have no plan to work on it for now.

[1] https://lwn.net/Articles/1016525/
[2] https://github.com/damonitor/talks/blob/master/2025/lsfmmbpf/damon_requirements_lsfmmbpf_2025.pdf

Thanks,
SJ

[...]

#55 lore raw
Quoted 11 lines · SJ Park

Hello Davidlohr,

On Thu, 24 Sep 2026 19:12:26 -0700 Davidlohr Bueso <[email protected]> wrote:
[...]

Quoted 2 lines

So perhaps any mm interface for this stuff should just not be designed around sampling, and instead proper hardware sources?

Makes sense to me. I proposed [1] damos_add_folios() in LSFMMBPF'25 for a reason that is similar to your points in my humble view. To quote from the slide [2],

Quoted 14 lines

+/**
+ * damos_add_folios - Add a list of folios as highest priority target for given
+ * damos.
+ * @scheme: DAMOS scheme for the specific access-aware operation.
+ * @folios: List of folios to apply the access-aware operation.
+ *
+ * If a kernel component finds folios that eligible for a specific memory
+ * management operation (e.g., CXL-promotion or demotion), execution of the
+ * operation can be requested to be done by DAMOS using this function. The
+ * execution of the operation will be asynchronously done by DAMOS worker
+ * thread. DAMOS features for resource control (DAMOS quotas) will also be
+ * applied.
+ */
+void damos_add_folios(struct damos *scheme, struct list_head *folios)

Considering chmu alone would not justify a generic mm interface, I am more and more inclined to go with the chmu <-> DAMON option (I recall your presentation) + do NUMAB async (1 and 2) and that would solve most problems in a practical manner.

Thanks,
Davidlohr

maintainer
#56 lore raw
Quoted 15 lines · Davidlohr Bueso
Quoted 6 lines

+/**
+ * damos_add_folios - Add a list of folios as highest priority target for given
+ * damos.
+ * @scheme: DAMOS scheme for the specific access-aware operation.
+ * @folios: List of folios to apply the access-aware operation.
+ *

Quoted 8 lines

+ * If a kernel component finds folios that eligible for a specific memory
+ * management operation (e.g., CXL-promotion or demotion), execution of the
+ * operation can be requested to be done by DAMOS using this function. The
+ * execution of the operation will be asynchronously done by DAMOS worker
+ * thread. DAMOS features for resource control (DAMOS quotas) will also be
+ * applied.
+ */
+void damos_add_folios(struct damos *scheme, struct list_head *folios)

Quoted 4 lines

Considering chmu alone would not justify a generic mm interface, I am more and more inclined to go with the chmu <-> DAMON option (I recall your presentation) + do NUMAB async (1 and 2) and that would solve most problems in a practical manner.

The major pushback on NUMAB async has been that there is no good way to charge this work to the cgroup.

Shakeel recently posted something that might help:
https://lore.kernel.org/linux-mm/[email protected]/#t

But the approach does not generalize for everyone (see Tejun's response, net rx). May be worth a look for folks.

~Gregory

#57 lore raw
In reply to Davidlohr Bueso

Thanks Davidlohr for using pghot and providing your comments. The timing is also perfect as we can continue the discussions in LPC too. Let me try to address your concerns by with top-posting first where I cover other general concerns as well and would reply inline for your XSBench observation.

Complexity of pghot
===================
Let me start with lines-of-code and functionality breakup.

1135 mm/pghot.c
86 mm/pghot-default.c
85 mm/pghot-precise.c
208 include/linux/pghot.h

    45 mm/migrate.c

1559 total plus some small bits in mm/vmstat.c, mm/init.c, Kconfig, Makefile etc That includes around 160 lines of NUMAB2 engine code moved from kernel/sched/fair.c

The above includes the following distinct functionalities and parts:

Hotness recording mechanism (pghot_record_access())
Hotness promotion engine (which is moved from kernel/sched/fair.c) PFN walk and async migration (kmigrated)
CPU, memory and node hotplug handling
Sysctl and debugfs tunables, vmstat counters, enabling/disabling parts etc

Memory overhead
---------------
Default mode: 1 byte per lower-tier PFN => 256MB overhead per 1TB of lower-tier memory.
Precision mode: 4 bytes per lower-tier PFN => 1GB for 1TB of lower-tier memory.

In not-yet-posted version, I am tracking hotness at sub-section granularity (in addition to the existing per-section bit) which makes PFN walk more targeted and reduces the CPU overhead of kmigrated thread. This adds 64KB per 1TB of lower tier memory overhead for 2M sub-section tracking.

Runtime overhead
----------------
Recording hotness is mostly a constant time operation function. Hotness record lookup involves PFN to section, PFN to section->hot_map. Hotness record update involves a few atomic bit operations and an atomic cmpxchg operation.

Kmigrated walks only those sections that are marked hot. In the next version, it would even skip those sub-section ranges that aren't marked hot. Within a sub-section of 2M it walks all the PFNs (512 for x86). Hotness record consumption again involves an atomic cmpxchg operation.

migrate_pages() already supports batched migrations, kmigrated just feeds it with batched hot folio list extracted during the PFN walk.

No fancy data structures to track hotness, no complicated synchronization mechanisms between producer (pghot_record_access()) and consumer (kmigrated)

Hotness promotion engine
------------------------
This is not something that was written for pghot from scratch, but instead it is the same hot page promotion engine that is part of NUMAB2 which is now generalized and moved to pghot. So this is not the complexity that pghot introduces afresh.

So considering all these, I see pghot as a light-weight and low-overhead mechanism to track per-PFN hotness and do async batch migration. Initial versions had fancy double data structures; a large hash a small binary tree of promotion-ready records and associated synchronization mechanism, but that is all past now.

pghot interface for sampling
============================
pghot_record_access() interface was designed keeping the existing NUMAB2 source in mind. It fits that and it fits other sources like IBS Memory Profiler. So sampling sources report an access and the shared promotion engine acts upon it.

But for sources like CHMU, from what you describe, I gather that a bulk reporting interface plus an indication to bypass the engine to treat the PFNs as migrate-ready, is what is required. Should those migrate-ready PFNs go through the regular pghot tracking (getting into section hotmaps to be picked up by kmigrated) or even that should be bypassed?

However, in the context of PTE A bit based source, I have often thought about extending the interface for

- bulk reporting where more than one PFN gets reported.
- indicating the bypass options (frequency check bypass, recency check bypass etc)

NUMAB2 has to perform better in pghot
=====================================
pghot is about a sub-system that makes it possible to have multiple sources to coexist with reuse of common hot page promotion engine.

NUMAB2 source resides within the scheduler and the promotion engine is also part of the scheduler. Through pghot, I am separating the source (NUMA hint faults) from the engine and moving that existing engine into pghot, to a common place where it gets reused for other sources as well.

It is the same NUMA hint faults and more or less the same engine and hence my main objective is to ensure that there is no regression during this move. Additional performance optimizations can be done to the engine itself separately but that shouldn't be the baseline expectation from pghot.

Is moving hot page promotion out of scheduler into a dedicated system, a good thing in general? I believe so as scheduler isn't the right place for it to reside. However I would like to hear from scheduler folks on this.

Do we even need a centralized hot page promotion engine?
========================================================
NUMAB2 is good and serves as a good baseline for any new source that comes up. But with different kinds of sources becoming available, do we want all of them to duplicate the hot page heuristics and promote hot pages on their own? I thought that may not be preferable and hence started this effort.

CXL HMU may not need the promotion engine, but IBS Memory Profiler needs. It needs a promotion engine with full recency and frequency considerations before promoting. I don't think an arch driver like IBS Memory Profiler should be doing hotness heuristics within itself but instead be using the existing engine. In fact in my early posts, the driver based on primary IBS instance was feeding the "access sample" as "NUMA hint fault" to NUMAB1/B2 so that rest of NUMA Balancing/Hot page promotion engine just worked. But I think pghot is a better approach than that.

Why full pghot? Isn't async batch migration enough?
===================================================
Some of the above reasons apply but during the course of iterations, I have had implementations of just the migrator (kmigrated [1]).

If every sub-system/source has intelligence of its own and just wants to handover a list of pages to async migrator thread, that's not much of an effort as this implementation showed.

But then if some source needs rate-limiting and another source needs only hottest pages to be promoted, then again we overlap with the existing NUMAB2 engine.

Then if we want to be slightly generic and want two sources to complement each other or the promoter to differentiate between lukewarm vs hot pages/regions then we may have to maintain hotness records and may soon end up with something similar to pghot's hotness tracking and reporting mechanism.

Hardware sources have to out-perform NUMAB2
===========================================
Different sources will have different characteristics and capabilities and will help different workloads differently. So it is the choice that one could provide. Sometimes sources can complement each other as well.

What IBS Memory Profiler has shown is that it can match and/or exceed (for Graph500, ptr-chase, llama) NUMAB2 with no hint faults overhead [2]. Also please check the initial XSBench numbers in my inline reply.

Cost of an unused source
========================
Not all the sources are required for every situation. Sources can be disabled at compile time or not enabled at run time with no cost or effect on other sources. But hotmap allocations would remain as a static cost even when no source is enabled (built out at compile time, the map is gone; sources off at runtime, the map remains)

Tracking granularity: per-PFN vs region
=======================================
Often times this question comes up when pghot is compared with DAMON.

Firstly, the baseline that we have (NUMAB2), tracks hotness at page granularity. To support this, pghot started with per-PFN granularity. Naturally two concerns come up:

1. Memory overhead: I have shown the numbers above. It is lower-tier only and not much IMHO.

2. Scan/CPU overhead: pghot started with scanning all PFNs, that was expensive. Then it introduced hotness bit per memory section so that only those sections which are marked hot are scanned. Now I have added (yet to be posted) a sub-section level hotness tracking where a hotness bit is maintained for each fixed 2M region within a section. This has considerably reduced the CPU overhead for kmigrated thread. While the ptr-chase numbers that I shared with the separated out IBS RFC v0 post of IBS Memory Profiler [2] does show the kmigrated utilization numbers with sub-section tracking, I plan to have some more numbers ready for LPC.

I am beginning to feel that this may be a good middle ground between real region-level tracking (where all pages of region are promoted irrespective of their real hotness) vs scanning at sub-section granularity but performing per-PFN precise promotion.

Losing cache locality due to async migration
============================================
I see that async batched migration is important to free up the heavy migration related activities from the process context. Losing cache locality is inherent to that mechanism I believe. Given that pghot-NUMAB2 didn't show any significant regression with the benchmark numbers I presented, it would be good to find other benchmarks that show up this locality cost in a tiered memory setup.

Quoted 49 lines · Davidlohr Bueso
Quoted 19 lines · Bharata B Rao

Performance summary
===================
All results are on 3-node tiered systems (DRAM top tier + CPU-less CXL lower tier). Speedups below are normalized to the base kernel with no tiering (NUMAB=0). Columns compare mainline hint-fault tiering (base NUMAB=2) against pghot in two modes: hint-fault driven (pghot-hf, NUMAB=2) and HW-hint driven via the AMD IBS memory profiler (pghot-hw, NUMAB=0, no NUMA scanning). Single run per config (abench is avg of 3).

Benchmark      Metric (higher=better)   base-NUMAB2  pghot-hf  pghot-hw
-------------------------------------------------------------------------
NAS BT (MPI)   Mop/s total                  2.26x     2.26x     2.13x Graph500 BFS   harmonic-mean TEPS           2.64x     2.49x     2.68x llama.cpp      decode tok/s (tg128)         1.09x     1.41x     1.21x Redis+memtier  ops/sec                      1.05x     1.05x     1.00x Microbench     completion time (1/t)        2.41x     2.40x     2.35x
-------------------------------------------------------------------------
(baseline = base kernel, no tiering = 1.00x)

Quoted 2 lines

Summary: pghot hint-faults reproduces mainline NUMAB=2 with no regression on every workload; pghot HW-hints/IBS matches it mostly.

The IBS doesn't seem to bring anything to the table, other than just serve as an example - we've discussed this in the past this. And for NUMAB2 I would expect this to be the very best case in that you are not suffering from loosing the locality info.

This is a lot of added complexity and code just to break even. And hardware sources should really outperform NUMAB2 for this to be worthwhile, methinks.

I know Joshua mentioned some improved PSI by doing the promotion async; which makes me wonder that perhaps just a simpler promoting kthread is enough, so NUMAB2 just kicks the migration to it and not done in the accessing tasks' context etc. And Gregory also recently improved this path vs NUMAB1.

I mention this because I have integrated the chmu as a source of hotness for pghot and this was tested on a real device vs NUMAB3. This was based on Jonathan's rfc for the perf driver, only that I carve out the first chmu instance for this, and leave the rest of them (if any for perf).
For example, 'XSBench' improved by a factor of ~2.7x (I would like to report a wider range of benchmarks at some point, but this is what I have now). But a lot of this bypasses most of this framework and just piggy backs on kmigrated moving memory. chmu does not do well with pghot_record_access(), which is designed for sampling... so I just set

Quoted 8 lines

MIGRATE_READY right away. I also added a pghot call/api to make use of the regions chmu deals with instead of per access/page. The cxl driver just takes the dpa range given by the chmu hotlist, coalesces into adjacent hpa ranges (just 1 for the case of no interleaving) and feeds that to the pghot subsystem as a list of these contiguous physical chunks ranked/sorted by access density (accesses per page per second, so a large lukewarm region and a smaller hotter one can be differentiated and promote the latter first).

I would be interested to understand more about how you ran XSBench, the parameters used, the promotion stats, if demotion stats etc. Do share when you get time.

I did a round of testing to compare base-NUMAB2, pghot-hintfaults(NUMAB2) and pghot-hwhints (IBS Memory Profiler). I am still experimenting with options, placement etc but here are my initial numbers:

Non-Overcommitted case: XSBench working set fits fully within toptier but starts on lower tier before the measurement phase
(XSBench -s XL -g 238847 -t 64 -p 60000000 -l 34, footprint ~116.56 GiB)

Runtime (s) lookups/s Promotions (pages)
base-NUMAB0 498.6 4.09M 0
base-NUMAB2 422.4 4.83M 30.5M
pghot-hintfaults 329.6 6.20M 30.5M
pghot-hwhints 109.2 18.69M 2.7M

Same amount of pages promoted by both base-NUMAB2 and pghot-hintfaults but runtime is better with pghot. This could be the benefit of async batched migration showing and no adverse effect of losing cache locality.

pghot-hwhints shows good results. As I said this is just a first peek to the experimental numbers, I should have more concrete numbers and conclusion in LPC.

[1] Kmigrated -
https://lore.kernel.org/linux-mm/[email protected]/#t
[2] IBS Memory Profiler RFC v0 -
https://lore.kernel.org/linux-mm/[email protected]/

Regards,
Bharata.

#58 lore raw
Quoted 7 lines · Bharata B Rao

Hotness promotion engine
------------------------
This is not something that was written for pghot from scratch, but instead it is the same hot page promotion engine that is part of NUMAB2 which is now generalized and moved to pghot. So this is not the complexity that pghot introduces afresh.

Quoted 5 lines

So considering all these, I see pghot as a light-weight and low-overhead mechanism to track per-PFN hotness and do async batch migration. Initial versions had fancy double data structures; a large hash a small binary tree of promotion-ready records and associated synchronization mechanism, but that is all past now.

I think we are all in agreement that the async batch migration is wanted.

Quoted 29 lines

pghot interface for sampling
============================
pghot_record_access() interface was designed keeping the existing NUMAB2 source in mind. It fits that and it fits other sources like IBS Memory Profiler. So sampling sources report an access and the shared promotion engine acts upon it.

But for sources like CHMU, from what you describe, I gather that a bulk reporting interface plus an indication to bypass the engine to treat the PFNs as migrate-ready, is what is required. Should those migrate-ready PFNs go through the regular pghot tracking (getting into section hotmaps to be picked up by kmigrated) or even that should be bypassed?

However, in the context of PTE A bit based source, I have often thought about extending the interface for

- bulk reporting where more than one PFN gets reported.
- indicating the bypass options (frequency check bypass, recency check bypass etc)

NUMAB2 has to perform better in pghot
=====================================
pghot is about a sub-system that makes it possible to have multiple sources to coexist with reuse of common hot page promotion engine.

NUMAB2 source resides within the scheduler and the promotion engine is also part of the scheduler. Through pghot, I am separating the source (NUMA hint faults) from the engine and moving that existing engine into pghot, to a common place where it gets reused for other sources as well.

Quoted 8 lines

It is the same NUMA hint faults and more or less the same engine and hence my main objective is to ensure that there is no regression during this move. Additional performance optimizations can be done to the engine itself separately but that shouldn't be the baseline expectation from pghot.

Is moving hot page promotion out of scheduler into a dedicated system, a good thing in general? I believe so as scheduler isn't the right place for it to reside. However I would like to hear from scheduler folks on this.

So two of the autonuma balancing og authors are scheduler experts - and iirc *the* reason back then was locality. And Peter has already nacked the IBS stuff in the past. But indeed the batch async part would be good to get nack/ack; albeit the cgroup charging situation.

Quoted 7 lines

Do we even need a centralized hot page promotion engine?
========================================================
NUMAB2 is good and serves as a good baseline for any new source that comes up. But with different kinds of sources becoming available, do we want all of them to duplicate the hot page heuristics and promote hot pages on their own? I thought that may not be preferable and hence started this effort.

What are these sources that will become available?

Quoted 8 lines

CXL HMU may not need the promotion engine, but IBS Memory Profiler needs. It needs a promotion engine with full recency and frequency considerations before promoting. I don't think an arch driver like IBS Memory Profiler should be doing hotness heuristics within itself but instead be using the existing engine. In fact in my early posts, the driver based on primary IBS instance was feeding the "access sample" as "NUMA hint fault" to NUMAB1/B2 so that rest of NUMA Balancing/Hot page promotion engine just worked. But I think pghot is a better approach than that.

I agree that the IBS driver should not be doing hotness heuristics, it's the hw that should.

Quoted 24 lines

Why full pghot? Isn't async batch migration enough?
===================================================
Some of the above reasons apply but during the course of iterations, I have had implementations of just the migrator (kmigrated [1]).

If every sub-system/source has intelligence of its own and just wants to handover a list of pages to async migrator thread, that's not much of an effort as this implementation showed.

But then if some source needs rate-limiting and another source needs only hottest pages to be promoted, then again we overlap with the existing NUMAB2 engine.

Then if we want to be slightly generic and want two sources to complement each other or the promoter to differentiate between lukewarm vs hot pages/regions then we may have to maintain hotness records and may soon end up with something similar to pghot's hotness tracking and reporting mechanism.

Hardware sources have to out-perform NUMAB2
===========================================
Different sources will have different characteristics and capabilities and will help different workloads differently. So it is the choice that one could provide. Sometimes sources can complement each other as well.

Quoted 3 lines

What IBS Memory Profiler has shown is that it can match and/or exceed (for Graph500, ptr-chase, llama) NUMAB2 with no hint faults overhead [2]. Also please check the initial XSBench numbers in my inline reply.

It's all about the numbers, and the ones you have just don't really sell - which is one of the reasons this has been going on for years. If numa balancing didn't exist, then maybe adding all this would make sense. For the XSBench I don't think making decisions based on the non-overcommitted case is worthwhile.

Quoted 20 lines

Cost of an unused source
========================
Not all the sources are required for every situation. Sources can be disabled at compile time or not enabled at run time with no cost or effect on other sources. But hotmap allocations would remain as a static cost even when no source is enabled (built out at compile time, the map is gone; sources off at runtime, the map remains)

Tracking granularity: per-PFN vs region
=======================================
Often times this question comes up when pghot is compared with DAMON.

Firstly, the baseline that we have (NUMAB2), tracks hotness at page granularity. To support this, pghot started with per-PFN granularity. Naturally two concerns come up:

1. Memory overhead: I have shown the numbers above. It is lower-tier only and not much IMHO.

Quoted 9 lines

2. Scan/CPU overhead: pghot started with scanning all PFNs, that was expensive. Then it introduced hotness bit per memory section so that only those sections which are marked hot are scanned. Now I have added (yet to be posted) a sub-section level hotness tracking where a hotness bit is maintained for each fixed 2M region within a section. This has considerably reduced the CPU overhead for kmigrated thread. While the ptr-chase numbers that I shared with the separated out IBS RFC v0 post of IBS Memory Profiler [2] does show the kmigrated utilization numbers with sub-section tracking, I plan to have some more numbers ready for LPC.

I look forward to seeing any new numbers you have. Region granularity is certainly more aligned with willy as well as chmu.

Quoted 4 lines

I am beginning to feel that this may be a good middle ground between real region-level tracking (where all pages of region are promoted irrespective of their real hotness) vs scanning at sub-section granularity but performing per-PFN precise promotion.

[...]

Quoted 3 lines

I would be interested to understand more about how you ran XSBench, the parameters used, the promotion stats, if demotion stats etc. Do share when you get time.

The actual command is 'XSBench -g 90424' so only set the gridpoints, everything else is default, so total ~44Gb footprint. I don't have the vmstats currently (I do not run these benchmarks) but will share them once I get them - I can affirm that demotion is in fact enabled, so full TPP up and down.

For the chmu: 32GB device (1:1 dram and cxl), this is with a 4k unit size, 1s epoch, reporting mode is always on, threshold value is 1024.

The numa balancing mode was set to 3, the rest used the default values:
pghot_freq_threshold=2, pghot_promote_freq_window_ms=3000, pghot_promote_rage_limit_MBps=65536, kmigrated_sleep_ms=100, kmigrated_batch_nr=512.

I also have numbers for two more benchmarks with a real chmu, with basically the same numab parameters:

(i) TaoBench almost 2x throughput, going from ~250 qps to ~480 qps, vs NUMAB3.

    (dram:cxl is 16:32Gb with a 32Gb memsize, num_clients=2, clients_per_thread=75)

(ii) Graph500 only shows a smaller ~20% improvement vs NUMAB3 (bfs mean time drops
from 5.0 to 4.15 secs). This was for a memory ratio dram:cxl as 64:32Gb. The algorithm is BFS, edge factor 512, 16 mpi processes. (mpiexec.openmpi -n 16 ./graph500_reference_bfs 22 512)

Quoted 7 lines

I did a round of testing to compare base-NUMAB2, pghot-hintfaults(NUMAB2) and pghot-hwhints (IBS Memory Profiler). I am still experimenting with options, placement etc but here are my initial numbers:

Non-Overcommitted case: XSBench working set fits fully within toptier but starts on lower tier before the measurement phase

Those are nice numbers, but I don't think this is the methodology to use... It is more representative for the workload's working set to be > total dram and therefore spill into slower tier(s), instead of artificially starting in the slow memory and moving up. Do you have data for the over committed case?

Thanks,
Davidlohr

Quoted 22 lines

(XSBench -s XL -g 238847 -t 64 -p 60000000 -l 34, footprint ~116.56 GiB)

Runtime (s) lookups/s Promotions (pages)
base-NUMAB0 498.6 4.09M 0
base-NUMAB2 422.4 4.83M 30.5M
pghot-hintfaults 329.6 6.20M 30.5M
pghot-hwhints 109.2 18.69M 2.7M

Same amount of pages promoted by both base-NUMAB2 and pghot-hintfaults but runtime is better with pghot. This could be the benefit of async batched migration showing and no adverse effect of losing cache locality.

pghot-hwhints shows good results. As I said this is just a first peek to the experimental numbers, I should have more concrete numbers and conclusion in LPC.

[1] Kmigrated -
https://lore.kernel.org/linux-mm/[email protected]/#t
[2] IBS Memory Profiler RFC v0 -
https://lore.kernel.org/linux-mm/[email protected]/

Regards,
Bharata.