mm/damon: add node_eligible_mem_bp goal metric
Subsystems (3)
PATCH 1/1 mm/damon: add node_eligible_mem_bp goal metric
Background and Motivation
=========================
In heterogeneous memory systems, controlling memory distribution across NUMA nodes is essential for performance optimization. This patch enables system-wide page distribution with target-state goals such as "maintain 60% of scheme-eligible memory on DRAM" using PA-mode DAMON schemes.
Rather than using absolute thresholds, this metric tracks the ratio of memory that matches each scheme's access pattern filters on a target node, enabling the quota system to automatically adjust migration aggressiveness to maintain the desired distribution.
What This Metric Measures
=========================
node_eligible_mem_bp:
scheme_eligible_bytes_on_node / total_scheme_eligible_bytes * 10000
Two-Scheme Setup for Hot Page Distribution
==========================================
For maintaining 60% of hot memory on DRAM (node 0) and 40% on CXL (node 1):
PULL scheme: migrate_hot to node 0
goal: node_eligible_mem_bp, nid=0, target=6000
addr filter: node 1 address range (only migrate FROM CXL) "Move hot pages to DRAM if less than 60% of hot data is in DRAM"
PUSH scheme: migrate_hot to node 1
goal: node_eligible_mem_bp, nid=1, target=4000
addr filter: node 0 address range (only migrate FROM DRAM) "Move hot pages to CXL if less than 40% of hot data is in CXL"
Each scheme independently measures its own eligible memory and adjusts its quota to achieve its target ratio. The schemes work in concert through DAMON's unified monitoring context, with the quota autotuner balancing their relative aggressiveness.
Implementation Details
======================
The implementation adds a new quota goal metric type DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP to the existing DAMOS quota goal framework. When this metric is configured for a scheme:
1. During each quota adjustment cycle, damos_get_node_eligible_mem_bp() is called to calculate the current memory distribution.
2. The function iterates through all regions that match the scheme's access pattern (via __damos_valid_target()) and calculates:
- Total eligible bytes across all nodes
- Eligible bytes specifically on the target node (goal->nid)
3. For each eligible region, damos_calc_eligible_bytes() walks through the physical address range, using damon_get_folio() to look up each folio and determine its NUMA node via folio_nid().
4. Large folios are handled by calculating the exact overlap between the region boundaries and folio boundaries, ensuring accurate byte counts even when regions partially span folios.
5. The ratio (node_eligible / total_eligible * 10000) is returned as basis points, which the quota autotuner uses to adjust the scheme's effective quota size (esz).
The implementation requires CONFIG_DAMON_PADDR since damon_get_folio() is only available for physical address space monitoring.
Testing Results
===============
Functionally tested on a two-node heterogeneous memory system with DRAM (node 0) and CXL memory (node 1). A PUSH+PULL scheme configuration using migrate_hot actions was used to reach a target hot memory ratio between the two tiers.
With the TEMPORAL tuner, the system converges quickly to the target distribution. The tuner drives esz to maximum when under goal and to zero once the goal is met, forming a simple on/off feedback loop that stabilizes at the desired ratio.
With the CONSIST tuner, the scheme still converges but more slowly, as it migrates and then throttles itself based on quota feedback. The time to reach the goal varies depending on workload intensity.
Note: This metric works with both TEMPORAL and CONSIST goal tuners.
---
Changes since v9:
=================
https://lore.kernel.org/linux-mm/[email protected]/
- Cast phys_addr_t to unsigned long before mult_frac() to avoid __udivdi3 linking error on 32-bit systems with CONFIG_PHYS_ADDR_T_64BIT (SJ's fixup suggestion based on Sashiko review)
- Check !(unsigned long)total_eligible to avoid divide-by-zero after casting (SJ review)
Changes since v8:
=================
https://lore.kernel.org/linux-mm/[email protected]/
- Removed unnecessary casts in mult_frac() call to fix potential divide-by-zero on 32-bit PAE systems (Sashiko/SJ review)
- Kept original parameter name 'ctx' in damos_set_effective_quota() to avoid unnecessary diff (SJ review)
Changes since v7:
=================
https://lore.kernel.org/linux-mm/[email protected]/
- Wrapped if condition for goal->metric check to fix 80-column violation
- Used PAGE_ALIGN_DOWN(addr + PAGE_SIZE) instead of addr += PAGE_SIZE for proper page-aligned advancement
- Wrapped damos_goal_tune_esz_bp_temporal() function signature for 80-column compliance
- Removed unintended damos_trace_esz() call from first charge window
- Added addr filter usage note in two-scheme setup documentation
Changes since v6:
=================
https://lore.kernel.org/linux-mm/[email protected]/
- Dropped node_ineligible_mem_bp metric per maintainer feedback
- Updated two-scheme setup to use 60%/40% split with both schemes using node_eligible_mem_bp (per SeongJae Park's suggestion)
Changes since v5:
=================
https://lore.kernel.org/linux-mm/[email protected]/
- Rebased onto mm-new instead of damon/next for sashiko review
- Removed Reported-by/Closes tags per maintainer feedback (not needed for bugs found before merge)
Changes since v4:
=================
https://lore.kernel.org/linux-mm/[email protected]/
- Fixed commit message description for DAMOS_QUOTA_NODE_INELIGIBLE_MEM_BP per review feedback
- Added clarifying comment for ops-common.h include (for damon_get_folio())
- Fixed build error when CONFIG_DAMON_PADDR is disabled by adding #ifdef CONFIG_DAMON_PADDR guards around functions using damon_get_folio()
- Dropped RFC tag per maintainer feedback
Changes since RFC v3:
=====================
https://lore.kernel.org/linux-mm/[email protected]/
- Fixed phys_addr_t overflow on 32-bit PAE systems by using phys_addr_t for intermediate byte calculations
- Added cond_resched() per region to prevent CPU soft lockups
- Improved folio overlap calculation for accurate byte counting
Changes since RFC v2:
=====================
https://lore.kernel.org/linux-mm/[email protected]/
- Added node_ineligible_mem_bp complementary metric
- Fixed DAMON_OPS_PADDR validation in damon_commit_ctx()
- Improved commit message with two-scheme setup documentation
Changes since RFC v1:
=====================
https://lore.kernel.org/linux-mm/[email protected]/
- Initial implementation based on SeongJae Park's suggestion
- Added basic node_eligible_mem_bp metric
--- include/linux/damon.h | 3 + mm/damon/core.c | 169 +++++++++++++++++++++++++++++++++++---- mm/damon/sysfs-schemes.c | 7 ++ 3 files changed, 164 insertions(+), 15 deletions(-)
include/linux/damon.h +3 -0
@@ -159,6 +159,8 @@ enum damos_action {* @DAMOS_QUOTA_NODE_MEMCG_FREE_BP: MemFree ratio of a node for a cgroup. * @DAMOS_QUOTA_ACTIVE_MEM_BP: Active to total LRU memory ratio. * @DAMOS_QUOTA_INACTIVE_MEM_BP: Inactive to total LRU memory ratio. +* @DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP: Scheme-eligible memory ratio of a +* node in basis points (0-10000). * @NR_DAMOS_QUOTA_GOAL_METRICS: Number of DAMOS quota goal metrics. * * Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported.
@@ -172,6 +174,7 @@ enum damos_quota_goal_metric {DAMOS_QUOTA_NODE_MEMCG_FREE_BP ,DAMOS_QUOTA_ACTIVE_MEM_BP ,DAMOS_QUOTA_INACTIVE_MEM_BP , +DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP , NR_DAMOS_QUOTA_GOAL_METRICS, };
mm/damon/core.c +154 -15
264 lines not on this page · raw message
mm/damon/sysfs-schemes.c +7 -0
@@ -1094,6 +1094,10 @@ struct damos_sysfs_qgoal_metric_name damos_sysfs_qgoal_metric_names[] = { .metric = DAMOS_QUOTA_INACTIVE_MEM_BP, .name = "inactive_mem_bp", }, + { + .metric = DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP, + .name = "node_eligible_mem_bp", + }, }; static ssize_t target_metric_show(struct kobject *kobj,
@@ -2686,6 +2690,9 @@ static int damos_sysfs_add_quota_score( } goal->nid = sysfs_goal->nid;break ; +case DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP : + goal->nid = sysfs_goal->nid; +break ;default :break ; }
base-commit: 4f7b2cc1ad8719d55a8de9675a0aca65cb17c364
Signature · 1 line
2.43.0
[email protected]
AI reviewSashiko AI review found 3 potential issue(s): - [High] Data truncation and arithmetic overflow in metric calculation on 32-bit systems. - [High] Missing `cond_resched()` inside the inner PFN-walking…
Sashiko AI review found 3 potential issue(s):
- [High] Data truncation and arithmetic overflow in metric calculation on 32-bit systems.
- [High] Missing cond_resched() inside the inner PFN-walking loop causes CPU soft lockups.
- [High] Address advancement logic is vulnerable to wrap-around infinite loops. --
commit 10100659661304515a414f3a15324e8278667f42
Author: Ravi Jonnalagadda <[email protected]>
mm/damon: add node_eligible_mem_bp goal metric
This commit introduces the DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP goal metric to track the ratio of scheme-eligible memory on a specific NUMA node, allowing the quota autotuner to adjust based on spatial memory distribution.
Quoted 4 lines
mm/damon/core.c +0
-0
[ ... ]
Quoted 3 lines
mm/damon/core.c +3
-0+static phys_addr_t damos_calc_eligible_bytes (struct damon_ctx *c, +struct damos *s,int nid,phys_addr_t *total) +{
[ ... ]
mm/damon/core.c static unsigned long damos_get_node_memcg_used_bp( · line 2352
Quoted 12 lines
mm/damon/core.c +12
-0+/* Convert from core address units to physical bytes */ + addr = (phys_addr_t )r->ar.start * c->addr_unit; + end_addr = (phys_addr_t )r->ar.end * c->addr_unit; +while (addr < end_addr) { +struct folio *folio; +phys_addr_t folio_start, folio_end; +phys_addr_t overlap_start, overlap_end; +phys_addr_t counted; + + folio =damon_get_folio (PHYS_PFN (addr)); +if (!folio) { + addr =PAGE_ALIGN_DOWN (addr +PAGE_SIZE );
If a DAMON region extends to the upper boundary of the physical address space, could addr + PAGE_SIZE overflow and wrap around to 0?
If addr wraps to 0, addr < end_addr would remain true, which might cause an unrecoverable infinite loop.
Quoted 2 lines
mm/damon/core.c +2
-0+continue ; + }
[ ... ]
mm/damon/core.c static unsigned long damos_get_node_memcg_used_bp( · line 2378
Quoted 4 lines
mm/damon/core.c +4
-0+/* Advance past the entire folio */ + addr = folio_end; +folio_put (folio); + }
Because DAMON regions can span tens of gigabytes, could this inner loop execute millions of times sequentially without yielding?
The cond_resched() is placed outside this loop, which might trigger CPU soft lockups if a very large region is processed. Would it be better to move cond_resched() inside the while loop?
Quoted 18 lines
mm/damon/core.c +18
-0+cond_resched (); + } + } + + *total = total_eligible; +return node_eligible; +} + +static unsigned long damos_get_node_eligible_mem_bp (struct damon_ctx *c, +struct damos *s,int nid) +{ +phys_addr_t total_eligible =0 ; +phys_addr_t node_eligible; + +if (c->ops.id !=DAMON_OPS_PADDR ) +return 0 ; + +if (nid <0 || nid >=MAX_NUMNODES || !node_online (nid))
mm/damon/core.c static unsigned long damos_get_node_memcg_used_bp( · line 2406
Quoted 10 lines
mm/damon/core.c +10
-0+return 0 ; + + node_eligible =damos_calc_eligible_bytes (c, s, nid, &total_eligible); + +if (!(unsigned long )total_eligible) +return 0 ; + +return mult_frac ((unsigned long )node_eligible,10000 , + (unsigned long )total_eligible); +}
On 32-bit PAE systems, phys_addr_t is 64-bit while unsigned long is 32-bit. Does casting total_eligible and node_eligible to unsigned long discard the upper 32 bits?
If total_eligible happens to be exactly a multiple of 4 GB, the truncation could result in 0, causing the function to unexpectedly bypass the check.
Furthermore, mult_frac() performs intermediate multiplication. Even if the sizes fit in 32 bits, could the intermediate calculation overflow the 32-bit maximum when node_eligible exceeds ~429 KB?
Would it be safer to evaluate the byte counts using 64-bit arithmetic, such as div64_u64(), to prevent truncation and overflow?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
SeongJae Park
maintainerQuoted 3 lines · On Mon, 27 Apr 2026 01:24:29 +0000 [email protected] wrote:
Sashiko AI review found 3 potential issue(s):
- [High] Data truncation and arithmetic overflow in metric calculation on 32-bit systems.
- [High] Missingcond_resched()inside the inner PFN-walking loop causes CPU soft lockups.
Not new Sashiko findings. Above two issues are not real problems or better to see if the issue can really happen. I explained detailed reason on previous Sashiko reviews.
Quoted 1 line
- [High] Address advancement logic is vulnerable to wrap-around infinite loops.
This is a new and good finding. I asked a fix on my reply to the patch.
Quoted 36 lines
--
commit
10100659661304515a414f3a15324e8278667f42
Author: Ravi Jonnalagadda <[email protected]>mm/damon: add
node_eligible_mem_bpgoal metricThis commit introduces the
DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BPgoal metric to track the ratio of scheme-eligible memory on a specific NUMA node, allowing the quota autotuner to adjust based on spatial memory distribution.Quoted 4 lines
mm/damon/core.c +0
-0[ ... ]
Quoted 3 lines
mm/damon/core.c +3
-0+static phys_addr_t damos_calc_eligible_bytes (struct damon_ctx *c, +struct damos *s,int nid,phys_addr_t *total) +{[ ... ]
mm/damon/core.c static unsigned long damos_get_node_memcg_used_bp( · line 2352
Quoted 12 lines
mm/damon/core.c +12
-0+/* Convert from core address units to physical bytes */ + addr = (phys_addr_t )r->ar.start * c->addr_unit; + end_addr = (phys_addr_t )r->ar.end * c->addr_unit; +while (addr < end_addr) { +struct folio *folio; +phys_addr_t folio_start, folio_end; +phys_addr_t overlap_start, overlap_end; +phys_addr_t counted; + + folio =damon_get_folio (PHYS_PFN (addr)); +if (!folio) { + addr =PAGE_ALIGN_DOWN (addr +PAGE_SIZE );If a DAMON region extends to the upper boundary of the physical address space, could addr +
PAGE_SIZEoverflow and wrap around to 0?
Quoted 2 lines
If addr wraps to 0, addr <
end_addrwould remain true, which might cause an unrecoverable infinite loop.
Good finding. I asked another version for fix of this, on my reply to the original patch.
Thanks,
SJ
[...]
SeongJae Park
maintainerPATCH 1/1 mm/damon/core: (fixup) break calc_eligible_bytes() loop if addr overflows
As Sashiko pointed out, it could result in infinite loop. Check the overflow and break the loop in the case.
Also, wrap the addr setup line to keep the 80 columns limit.
---
NOTE:
Writing entire patches for minor Sashiko finding is tiresome. This is an experimental posting of fixup patch, wishing Sashiko to review this kind of incremental fixup.
--- mm/damon/core.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
mm/damon/core.c +4 -1
@@ -2349,7 +2349,10 @@ static phys_addr_t damos_calc_eligible_bytes(struct damon_ctx *c, folio =damon_get_folio (PHYS_PFN (addr));if (!folio) { - addr =PAGE_ALIGN_DOWN (addr +PAGE_SIZE ); + addr =PAGE_ALIGN_DOWN (addr + +PAGE_SIZE ); +if (!addr) +break ;continue ; }
Signature · 1 line
2.47.3
SeongJae Park
maintainerOn Sun, 26 Apr 2026 18:02:09 -0700 Ravi Jonnalagadda <[email protected]> wrote:
[...]
Quoted 19 lines
(quoted diff) +19
-0+static phys_addr_t damos_calc_eligible_bytes(struct damon_ctx *c, + struct damos *s, int nid, phys_addr_t *total) +{ + struct damon_target *t; + struct damon_region *r; + phys_addr_t total_eligible = 0; + phys_addr_t node_eligible = 0; + + damon_for_each_target(t, c) { + damon_for_each_region(r, t) { + phys_addr_t addr, end_addr; + + if (!__damos_valid_target(r, s)) + continue; + + /* Convert from core address units to physical bytes */ + addr = (phys_addr_t)r->ar.start * c->addr_unit; + end_addr = (phys_addr_t)r->ar.end * c->addr_unit; + while (addr < end_addr) {
Review
Quoted 10 lines
(quoted diff) +10
-0+ struct folio *folio; + phys_addr_t folio_start, folio_end; + phys_addr_t overlap_start, overlap_end; + phys_addr_t counted; + + folio = damon_get_folio(PHYS_PFN(addr)); + if (!folio) { + addr = PAGE_ALIGN_DOWN(addr + PAGE_SIZE); + continue; + }
Sashiko found this could result in an infinite loop if 'addr + PAGE_SIZE' overflows. The probability is quite low, but could we fix this? Maybe like below?
'''
mm/damon/core.c +4 -1
@@ -2349,7 +2349,10 @@ static phys_addr_t damos_calc_eligible_bytes(struct damon_ctx *c, folio =damon_get_folio (PHYS_PFN (addr));if (!folio) { - addr =PAGE_ALIGN_DOWN (addr +PAGE_SIZE ); + addr =PAGE_ALIGN_DOWN (addr + +PAGE_SIZE ); +if (!addr) +break ;continue ; }
'''
Ravi, could you please post one more version (maybe v9.2) with the abive fix if it looks good to you?
[1] https://lore.kernel.org/[email protected]
Thanks,
SJ
[...]