Skip to contentSkip to conversation

mm/damon: add node_eligible_mem_bp goal metric

PATCH In review Ravi Jonnalagadda 3 participants 5 messages Apr 26 – Apr 27

Subsystems (3)
Memory management Documentation DAMON
lore
Share & tools
lore

Ravi Jonnalagadda

author
#0 lore raw

PATCH v9 1/1 mm/damon: add node_eligible_mem_bp goal metric

Background and Motivation
=========================

In heterogeneous memory systems, controlling memory distribution across NUMA nodes is essential for performance optimization. This patch enables system-wide page distribution with target-state goals such as "maintain 60% of scheme-eligible memory on DRAM" using PA-mode DAMON schemes.

Rather than using absolute thresholds, this metric tracks the ratio of memory that matches each scheme's access pattern filters on a target node, enabling the quota system to automatically adjust migration aggressiveness to maintain the desired distribution.

What This Metric Measures
=========================

node_eligible_mem_bp:
scheme_eligible_bytes_on_node / total_scheme_eligible_bytes * 10000

Two-Scheme Setup for Hot Page Distribution
==========================================

For maintaining 60% of hot memory on DRAM (node 0) and 40% on CXL (node 1):

PULL scheme: migrate_hot to node 0
goal: node_eligible_mem_bp, nid=0, target=6000
addr filter: node 1 address range (only migrate FROM CXL) "Move hot pages to DRAM if less than 60% of hot data is in DRAM"

PUSH scheme: migrate_hot to node 1
goal: node_eligible_mem_bp, nid=1, target=4000
addr filter: node 0 address range (only migrate FROM DRAM) "Move hot pages to CXL if less than 40% of hot data is in CXL"

Each scheme independently measures its own eligible memory and adjusts its quota to achieve its target ratio. The schemes work in concert through DAMON's unified monitoring context, with the quota autotuner balancing their relative aggressiveness.

Implementation Details
======================

The implementation adds a new quota goal metric type DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP to the existing DAMOS quota goal framework. When this metric is configured for a scheme:

1. During each quota adjustment cycle, damos_get_node_eligible_mem_bp() is called to calculate the current memory distribution.

2. The function iterates through all regions that match the scheme's access pattern (via __damos_valid_target()) and calculates:
- Total eligible bytes across all nodes
- Eligible bytes specifically on the target node (goal->nid)

3. For each eligible region, damos_calc_eligible_bytes() walks through the physical address range, using damon_get_folio() to look up each folio and determine its NUMA node via folio_nid().

4. Large folios are handled by calculating the exact overlap between the region boundaries and folio boundaries, ensuring accurate byte counts even when regions partially span folios.

5. The ratio (node_eligible / total_eligible * 10000) is returned as basis points, which the quota autotuner uses to adjust the scheme's effective quota size (esz).

The implementation requires CONFIG_DAMON_PADDR since damon_get_folio() is only available for physical address space monitoring.

Testing Results
===============

Functionally tested on a two-node heterogeneous memory system with DRAM (node 0) and CXL memory (node 1). A PUSH+PULL scheme configuration using migrate_hot actions was used to reach a target hot memory ratio between the two tiers.

With the TEMPORAL tuner, the system converges quickly to the target distribution. The tuner drives esz to maximum when under goal and to zero once the goal is met, forming a simple on/off feedback loop that stabilizes at the desired ratio.

With the CONSIST tuner, the scheme still converges but more slowly, as it migrates and then throttles itself based on quota feedback. The time to reach the goal varies depending on workload intensity.

Note: This metric works with both TEMPORAL and CONSIST goal tuners.

Suggested-by SeongJae Park <[email protected]>
Signed-off-by Ravi Jonnalagadda <[email protected]>
Reviewed-by SeongJae Park <[email protected]>

---
Changes since v8:
=================
https://lore.kernel.org/linux-mm/[email protected]/

- Removed unnecessary casts in mult_frac() call to fix potential divide-by-zero on 32-bit PAE systems (Sashiko/SJ review)
- Kept original parameter name 'ctx' in damos_set_effective_quota() to avoid unnecessary diff (SJ review)

Changes since v7:
=================
https://lore.kernel.org/linux-mm/[email protected]/

- Wrapped if condition for goal->metric check to fix 80-column violation
- Used PAGE_ALIGN_DOWN(addr + PAGE_SIZE) instead of addr += PAGE_SIZE for proper page-aligned advancement
- Wrapped damos_goal_tune_esz_bp_temporal() function signature for 80-column compliance
- Removed unintended damos_trace_esz() call from first charge window
- Added addr filter usage note in two-scheme setup documentation

Changes since v6:
=================
https://lore.kernel.org/linux-mm/[email protected]/

- Dropped node_ineligible_mem_bp metric per maintainer feedback
- Updated two-scheme setup to use 60%/40% split with both schemes using node_eligible_mem_bp (per SeongJae Park's suggestion)

Changes since v5:
=================
https://lore.kernel.org/linux-mm/[email protected]/

- Rebased onto mm-new instead of damon/next for sashiko review
- Removed Reported-by/Closes tags per maintainer feedback (not needed for bugs found before merge)

Changes since v4:
=================
https://lore.kernel.org/linux-mm/[email protected]/

- Fixed commit message description for DAMOS_QUOTA_NODE_INELIGIBLE_MEM_BP per review feedback
- Added clarifying comment for ops-common.h include (for damon_get_folio())
- Fixed build error when CONFIG_DAMON_PADDR is disabled by adding #ifdef CONFIG_DAMON_PADDR guards around functions using damon_get_folio()
- Dropped RFC tag per maintainer feedback

Changes since RFC v3:
=====================
https://lore.kernel.org/linux-mm/[email protected]/

- Fixed phys_addr_t overflow on 32-bit PAE systems by using phys_addr_t for intermediate byte calculations
- Added cond_resched() per region to prevent CPU soft lockups
- Improved folio overlap calculation for accurate byte counting

Changes since RFC v2:
=====================
https://lore.kernel.org/linux-mm/[email protected]/

- Added node_ineligible_mem_bp complementary metric
- Fixed DAMON_OPS_PADDR validation in damon_commit_ctx()
- Improved commit message with two-scheme setup documentation

Changes since RFC v1:
=====================
https://lore.kernel.org/linux-mm/[email protected]/

- Initial implementation based on SeongJae Park's suggestion
- Added basic node_eligible_mem_bp metric

---
 include/linux/damon.h    |   3 +
 mm/damon/core.c          | 168 +++++++++++++++++++++++++++++++++++----
 mm/damon/sysfs-schemes.c |   7 ++
 3 files changed, 163 insertions(+), 15 deletions(-)
include/linux/damon.h +3 -0
index f2cdb7c3f5e6..986b8c902585 100644
@@ -159,6 +159,8 @@ enum damos_action {
  * @DAMOS_QUOTA_NODE_MEMCG_FREE_BP:	MemFree ratio of a node for a cgroup.
  * @DAMOS_QUOTA_ACTIVE_MEM_BP:		Active to total LRU memory ratio.
  * @DAMOS_QUOTA_INACTIVE_MEM_BP:	Inactive to total LRU memory ratio.
+ * @DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP:	Scheme-eligible memory ratio of a
+ *					node in basis points (0-10000).
  * @NR_DAMOS_QUOTA_GOAL_METRICS:	Number of DAMOS quota goal metrics.
  *
  * Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported.

@@ -172,6 +174,7 @@ enum damos_quota_goal_metric {
 	DAMOS_QUOTA_NODE_MEMCG_FREE_BP,
 	DAMOS_QUOTA_ACTIVE_MEM_BP,
 	DAMOS_QUOTA_INACTIVE_MEM_BP,
+	DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP,
 	NR_DAMOS_QUOTA_GOAL_METRICS,
 };
mm/damon/core.c +153 -15

263 lines not on this page · raw message

mm/damon/sysfs-schemes.c +7 -0
index 04746cbb3327..e0a4e309b15d 100644
@@ -1094,6 +1094,10 @@ struct damos_sysfs_qgoal_metric_name damos_sysfs_qgoal_metric_names[] = {
 		.metric = DAMOS_QUOTA_INACTIVE_MEM_BP,
 		.name = "inactive_mem_bp",
 	},
+	{
+		.metric = DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP,
+		.name = "node_eligible_mem_bp",
+	},
 };
 
 static ssize_t target_metric_show(struct kobject *kobj,

@@ -2686,6 +2690,9 @@ static int damos_sysfs_add_quota_score(
 			}
 			goal->nid = sysfs_goal->nid;
 			break;
+		case DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP:
+			goal->nid = sysfs_goal->nid;
+			break;
 		default:
 			break;
 		}

base-commit: 4f7b2cc1ad8719d55a8de9675a0aca65cb17c364

Signature · 1 line
2.43.0
AI review
#1 lore raw
Sashiko AI review found 4 potential issue(s): - [High] Missing `cond_resched()` inside the inner address-scanning loop causes CPU soft lockups. - [High] Direct use of `mult_frac()` with 64-bit…

Sashiko AI review found 4 potential issue(s):
- [High] Missing cond_resched() inside the inner address-scanning loop causes CPU soft lockups.
- [High] Direct use of mult_frac() with 64-bit phys_addr_t causes a linker error on 32-bit architectures with PAE.
- [High] Integer overflow in mult_frac() on 32-bit systems without PAE leads to incorrect metric calculations.
- [High] Unbounded O(N) page-by-page memory scan in quota evaluation breaks DAMON's low-overhead design. --

commit 078c85e7912c268fbbe87930fc285713a5b57d05
Author: Ravi Jonnalagadda <[email protected]>

mm/damon: add node_eligible_mem_bp goal metric

This commit introduces a new DAMON quota goal metric, DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP, which tracks the ratio of scheme-eligible memory on a target NUMA node compared to total system-wide eligible memory. It operates by exhaustively scanning the physical address ranges of eligible regions to determine folio NUMA locality, returning the ratio in basis points.

Quoted 4 lines
mm/damon/core.c +0 -0
index 3dbbbfdeff71..b23b6bd53aba 100644

[ ... ]

Quoted 1 line
mm/damon/core.c +0 -0
@@ -2287,7 +2306,111 @@ static unsigned long damos_get_node_memcg_used_bp(

[ ... ]

Quoted 9 lines
mm/damon/core.c +9 -0
+static phys_addr_t damos_calc_eligible_bytes(struct damon_ctx *c,
+		struct damos *s, int nid, phys_addr_t *total)
+{
+	struct damon_target *t;
+	struct damon_region *r;
+	phys_addr_t total_eligible = 0;
+	phys_addr_t node_eligible = 0;
+
+	damon_for_each_target(t, c) {

mm/damon/core.c static unsigned long damos_get_node_memcg_used_bp( · line 2344

Quoted 10 lines
mm/damon/core.c +10 -0
+		damon_for_each_region(r, t) {
+			phys_addr_t addr, end_addr;
+
+			if (!__damos_valid_target(r, s))
+				continue;
+
+			/* Convert from core address units to physical bytes */
+			addr = (phys_addr_t)r->ar.start * c->addr_unit;
+			end_addr = (phys_addr_t)r->ar.end * c->addr_unit;
+			while (addr < end_addr) {

Does this unbounded page-by-page scan break DAMON's low-overhead design?

This inner loop iterates through every single page in all eligible regions during the frequent quota adjustment cycle. Scanning massive memory pools (e.g., hundreds of GBs) requires hundreds of millions of synchronous atomic refcount operations via damon_get_folio().

Should the NUMA distribution be estimated using sampling or region sizes instead?

Quoted 25 lines
mm/damon/core.c +25 -0
+				struct folio *folio;
+				phys_addr_t folio_start, folio_end;
+				phys_addr_t overlap_start, overlap_end;
+				phys_addr_t counted;
+
+				folio = damon_get_folio(PHYS_PFN(addr));
+				if (!folio) {
+					addr = PAGE_ALIGN_DOWN(addr + PAGE_SIZE);
+					continue;
+				}
+
+				/*
+				 * Calculate exact overlap between the region
+				 * [addr, end_addr) and the folio range.
+				 * The folio may start before addr if addr is
+				 * in the middle of a large folio.
+				 */
+				folio_start = PFN_PHYS(folio_pfn(folio));
+				folio_end = folio_start + folio_size(folio);
+
+				overlap_start = max(addr, folio_start);
+				overlap_end = min(end_addr, folio_end);
+
+				if (overlap_end > overlap_start) {
+					counted = overlap_end - overlap_start;

mm/damon/core.c static unsigned long damos_get_node_memcg_used_bp( · line 2379

Quoted 10 lines
mm/damon/core.c +10 -0
+					total_eligible += counted;
+					if (folio_nid(folio) == nid)
+						node_eligible += counted;
+				}
+
+				/* Advance past the entire folio */
+				addr = folio_end;
+				folio_put(folio);
+			}
+			cond_resched();

Can missing cond_resched() inside the while loop cause CPU soft lockups?

Because cond_resched() is placed outside the inner loop, and DAMON regions can span hundreds of gigabytes, the kdamond thread will monopolize the CPU for millions of iterations without yielding. Should a cond_resched() be added inside the address-scanning loop?

Quoted 15 lines
mm/damon/core.c +15 -0
+		}
+	}
+
+	*total = total_eligible;
+	return node_eligible;
+}
+
+static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
+		struct damos *s, int nid)
+{
+	phys_addr_t total_eligible = 0;
+	phys_addr_t node_eligible;
+
+	if (c->ops.id != DAMON_OPS_PADDR)
+		return 0;

mm/damon/core.c static unsigned long damos_get_node_memcg_used_bp( · line 2404

Quoted 10 lines
mm/damon/core.c +10 -0
+
+	if (nid < 0 || nid >= MAX_NUMNODES || !node_online(nid))
+		return 0;
+
+	node_eligible = damos_calc_eligible_bytes(c, s, nid, &total_eligible);
+
+	if (!total_eligible)
+		return 0;
+
+	return mult_frac(node_eligible, 10000, total_eligible);

Can this cause a linker error on 32-bit architectures with PAE?

On 32-bit systems with CONFIG_PHYS_ADDR_T_64BIT enabled, phys_addr_t is a 64-bit integer. The mult_frac() macro performs native division using the type of its arguments. Since the kernel does not link 64-bit division routines (__udivdi3) on 32-bit platforms, this will cause a build failure.

Could this also cause an integer overflow on 32-bit systems without PAE?

When phys_addr_t is a 32-bit unsigned integer, the multiplication node_eligible * 10000 is performed in 32-bit arithmetic. If node_eligible exceeds roughly 429 KB (UINT_MAX / 10000), the multiplication will silently overflow, producing an incorrect ratio.

Quoted 1 line

+}

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#2 lore raw
Quoted 2 lines · On Sun, 26 Apr 2026 22:04:08 +0000 [email protected] wrote:

Sashiko AI review found 4 potential issue(s):
- [High] Missing cond_resched() inside the inner address-scanning loop causes CPU soft lockups.

Unlikely.

Quoted 1 line

- [High] Direct use of mult_frac() with 64-bit phys_addr_t causes a linker error on 32-bit architectures with PAE.

Good finding. Read below for detailed answer.

Quoted 1 line

- [High] Integer overflow in mult_frac() on 32-bit systems without PAE leads to incorrect metric calculations.

No real issue.

Quoted 1 line

- [High] Unbounded O(N) page-by-page memory scan in quota evaluation breaks DAMON's low-overhead design.

Intended behavior.

[...]

Quoted 21 lines
Quoted 8 lines
mm/damon/core.c +8 -0
+static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
+		struct damos *s, int nid)
+{
+	phys_addr_t total_eligible = 0;
+	phys_addr_t node_eligible;
+
+	if (c->ops.id != DAMON_OPS_PADDR)
+		return 0;

mm/damon/core.c static unsigned long damos_get_node_memcg_used_bp( · line 2404

Quoted 10 lines
mm/damon/core.c +10 -0
+
+	if (nid < 0 || nid >= MAX_NUMNODES || !node_online(nid))
+		return 0;
+
+	node_eligible = damos_calc_eligible_bytes(c, s, nid, &total_eligible);
+
+	if (!total_eligible)
+		return 0;
+
+	return mult_frac(node_eligible, 10000, total_eligible);

Can this cause a linker error on 32-bit architectures with PAE?

Quoted 4 lines

On 32-bit systems with CONFIG_PHYS_ADDR_T_64BIT enabled, phys_addr_t is a 64-bit integer. The mult_frac() macro performs native division using the type of its arguments. Since the kernel does not link 64-bit division routines (__udivdi3) on 32-bit platforms, this will cause a build failure.

Heh, good finding. I suggested a fixup on my reply [1] to the patch.

[1] https://lore.kernel.org/[email protected]

Thanks,
SJ

[...]

maintainer
#3 lore raw
In reply to Ravi Jonnalagadda

[...]

Quoted 8 lines

Changes since v8:
=================
https://lore.kernel.org/linux-mm/[email protected]/

- Removed unnecessary casts in mult_frac() call to fix potential divide-by-zero on 32-bit PAE systems (Sashiko/SJ review)
- Kept original parameter name 'ctx' in damos_set_effective_quota() to avoid unnecessary diff (SJ review)

[...]

Quoted 9 lines
(quoted diff) +9 -0
+static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
+		struct damos *s, int nid)
+{
+	phys_addr_t total_eligible = 0;
+	phys_addr_t node_eligible;
+
+	if (c->ops.id != DAMON_OPS_PADDR)
+		return 0;
+

Review

Quoted 10 lines
(quoted diff) +10 -0
+	if (nid < 0 || nid >= MAX_NUMNODES || !node_online(nid))
+		return 0;
+
+	node_eligible = damos_calc_eligible_bytes(c, s, nid, &total_eligible);
+
+	if (!total_eligible)
+		return 0;
+
+	return mult_frac(node_eligible, 10000, total_eligible);
+}

Sashiko found [1] this can cause __udivdi3 linking issue on 32bit machine with CONFIG_PHYS_ADDR_T_64BIT. Seems that is correct to me. I believe below attaching fixup can fix it?

Ravi, if my fixup looks good to you, could you please post another version (say, v9.1) for another Sashiko review round?

[1] https://lore.kernel.org/[email protected]

Thanks,
SJ

[...]

=== >8 ===
From 828ddfdfa3a996efc4620919a3cc77d55088c466 Mon Sep 17 00:00:00 2001
From: SeongJae Park <[email protected]>
Date: Sun, 26 Apr 2026 15:54:48 -0700
Subject: [PATCH] mm/damon/core: avoid __udivdi3 linking issue

On 32bit machines having PHYS_ADDR_T_64BIT, mult_frac() with phys_addr_t will cause __udivdi3 linking issue. Cast values to 'unsigned long' before calling mult_frac(). This could cause divide-by-zero if the denominator becomes zero by casting. Avoid it by checking the zero case with the casted value.

Signed-off-by SeongJae Park <[email protected]>
---
 mm/damon/core.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/damon/core.c +3 -2
index 0ea747d487465..aec43314fc4a9 100644
@@ -2921,10 +2921,11 @@ static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
 
        node_eligible = damos_calc_eligible_bytes(c, s, nid, &total_eligible);
 
-       if (!total_eligible)
+       if (!(unsigned long)total_eligible)
                return 0;
 
-       return mult_frac(node_eligible, 10000, total_eligible);
+       return mult_frac((unsigned long)node_eligible, 10000,
+                       (unsigned long)total_eligible);
 }
 #else /* CONFIG_DAMON_PADDR */
 static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,

--
2.47.3

author
#4 lore raw
Quoted 35 lines · SeongJae Park

[...]

Quoted 8 lines

Changes since v8:
=================
https://lore.kernel.org/linux-mm/[email protected]/

- Removed unnecessary casts in mult_frac() call to fix potential divide-by-zero on 32-bit PAE systems (Sashiko/SJ review)
- Kept original parameter name 'ctx' in damos_set_effective_quota() to avoid unnecessary diff (SJ review)

[...]

Quoted 9 lines
(quoted diff) +9 -0
+static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
+             struct damos *s, int nid)
+{
+     phys_addr_t total_eligible = 0;
+     phys_addr_t node_eligible;
+
+     if (c->ops.id != DAMON_OPS_PADDR)
+             return 0;
+

Review

Quoted 10 lines
(quoted diff) +10 -0
+     if (nid < 0 || nid >= MAX_NUMNODES || !node_online(nid))
+             return 0;
+
+     node_eligible = damos_calc_eligible_bytes(c, s, nid, &total_eligible);
+
+     if (!total_eligible)
+             return 0;
+
+     return mult_frac(node_eligible, 10000, total_eligible);
+}

Sashiko found [1] this can cause __udivdi3 linking issue on 32bit machine with CONFIG_PHYS_ADDR_T_64BIT. Seems that is correct to me. I believe below attaching fixup can fix it?

Quoted 3 lines

Ravi, if my fixup looks good to you, could you please post another version (say, v9.1) for another Sashiko review round?

Makes sense SJ. will send v9.1 shortly with the fixup.

Quoted 46 lines

[1] https://lore.kernel.org/[email protected]

Thanks,
SJ

[...]

=== >8 ===
From 828ddfdfa3a996efc4620919a3cc77d55088c466 Mon Sep 17 00:00:00 2001
From: SeongJae Park <[email protected]>
Date: Sun, 26 Apr 2026 15:54:48 -0700
Subject: [PATCH] mm/damon/core: avoid __udivdi3 linking issue

On 32bit machines having PHYS_ADDR_T_64BIT, mult_frac() with phys_addr_t will cause __udivdi3 linking issue. Cast values to 'unsigned long' before calling mult_frac(). This could cause divide-by-zero if the denominator becomes zero by casting. Avoid it by checking the zero case with the casted value.

Signed-off-by SeongJae Park <[email protected]>
---
 mm/damon/core.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/damon/core.c +3 -3
index 0ea747d487465..aec43314fc4a9 100644
@@ -2921,10 +2921,11 @@ static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
 
        node_eligible = damos_calc_eligible_bytes(c, s, nid, &total_eligible);
 
-       if (!total_eligible)
+       if (!(unsigned long)total_eligible)
                return 0;
 
-       return mult_frac(node_eligible, 10000, total_eligible);
+       return mult_frac((unsigned long)node_eligible, 10000,
+                       (unsigned long)total_eligible);
 }
 #else /* CONFIG_DAMON_PADDR */
 static unsigned long damos_get_node_eligible_mem_bp(struct damon_ctx *c,
--

2.47.3