mm/damon: hardware-sampled access reports

This series lets DAMON take its access information from a hardware sampler instead of from a page-table scan, and lets a scheme's score be weighted by what that sampler reported.

The change from v1 [1] is that it is now built on the data attribute probe infrastructure that has since landed in mm-new: a PMU is expressed as one more probe on a context, with its own weight and probe hits, rather than as a mechanism of its own beside the ops set. The rest of this posting is the same substrate reworked onto that, plus two consumers.

This is not a merge request. The series is based on damon/next at the base-commit below, which moves, so the tree it was built and tested from is also on

https://github.com/ravis-opensrc/linux/tree/damon/perf-rfc-v2-09-06-26

It is posted for design feedback, on the substrate and on where it belongs in the roadmap for extending DAMON beyond the pte-accessed bit [2]. That roadmap's second milestone, now open, is a first data attribute monitored through damon_report_access(), and that is what a sampling PMU is here. So this series keeps that function and its callers and replaces its body: the reporting path a hardware sampler needs cannot take a mutex, and the drain has to reach a virtual-address context as well as a physical one. The shape of the ring, the drain and the sysfs surface are what is most useful to review.

This series lets DAMON take its access information from a hardware sampler instead of from a page-table scan, and lets a scheme's score be weighted by what that sampler reported.

The change from v1 [1] is that it is now built on the data attribute probe infrastructure that has since landed in mm-new: a PMU is expressed as one more probe on a context, with its own weight and probe hits, rather than as a mechanism of its own beside the ops set. The rest of this posting is the same substrate reworked onto that, plus two consumers.

This is not a merge request. The series is based on damon/next at the base-commit below, which moves, so the tree it was built and tested from is also on

https://github.com/ravis-opensrc/linux/tree/damon/perf-rfc-v2-09-06-26

It is posted for design feedback, on the substrate and on where it belongs in the roadmap for extending DAMON beyond the pte-accessed bit [2]. That roadmap's second milestone, now open, is a first data attribute monitored through damon_report_access(), and that is what a sampling PMU is here. So this series keeps that function and its callers and replaces its body: the reporting path a hardware sampler needs cannot take a mutex, and the drain has to reach a virtual-address context as well as a physical one. The shape of the ring, the drain and the sysfs surface are what is most useful to review.

## Why a unified perf-event substrate

DAMON derives its access information from the PTE Accessed bit. A sampling PMU carries what that bit cannot: which addresses the hardware went to, and how often it went there. Many machines already have such a unit, and more than one kind of it, so what this series is after is letting DAMON's regions be tuned from whichever perf-based hardware source a machine offers rather than from the Accessed bit alone. A sampler does not arrive on a kdamond's terms, though: it delivers an address when the hardware decides to, in NMI context, with no relation to the sampling interval and no mm to walk.

The alternative is a backend per PMU vendor, each owning its own configuration, sysfs knobs and lifecycle. The perf-event direction [3] avoids that: let DAMON register kernel-counter perf events and consume samples from any sampling PMU the perf core already knows about. This series follows it, adding one substrate below the ops sets rather than an ops set per PMU -- a report ring that any in-kernel access source can push into, and a drain that folds those reports into region probe hits on the aggregation boundary the kdamond already has.

What running it
across vendors needed on top of that direction is:

- per-CPU lockless rings between the NMI sample handler and the kdamond drain,

- a ring partitioned by probe class, so the page-fault primitive and a PMU can populate one context without either seeing the other's reports,

- per-CPU events that follow CPU hotplug, armed when the kdamond starts and disarmed and drained when it stops,
- a per-PMU owner, so two contexts cannot claim the same PMU type,
- whichever address a PMU does report carried on the report and matched against the context's own address space, so one source serves a paddr or a vaddr context without a backend per address space.

This is tested with PEBS on Intel and IBS on AMD, both configured as perf_event attributes on a probe and using the perf core's event plumbing rather than per-vendor MSR code. A third source has already been written against the same
ring: Kunwu Chan's ARM SPE backend [5], which reaches it through an AUX buffer drained in process context instead of an overflow callback, and which the roadmap [2] places in its third milestone.

The partitioning is what lets promotion and demotion run in one context. A sampler says which regions are hot; it says nothing about which are cold, because a sampler that reports nothing about a page cannot distinguish untouched from unsampled.

Region age is what a demotion scheme matches on, and age comes from the page-fault primitive.

With the ring partitioned by class both are live at
once: the probe supplies hotness, the primitive supplies age, and two schemes over the same regions can move memory in both directions under one kdamond.

The two classes are not symmetric, and the asymmetry is in where a report comes from rather than in what it carries. A page fault has no monitoring context at report time, so a per-context ring is not expressible for it: those reports go to one global set of per-CPU rings, drained by the single context whose page-fault primitive is enabled. A PMU overflow handler does carry the context that armed the event, so those reports go to that context's own per-CPU rings and each context drains only what it owns. Both classes share the ring layout and the producer side, so the barrier pairing cannot drift between them.

```
  page-fault report                    PMU sample, NMI context
  no ctx at report time                handler carries the owning ctx
          |                                     |
          v                                     v
  +----------------------------+       +----------------------------+
  | one global per-CPU ring    |       | per-CPU rings per context  |
  | set, shared by every ctx   |       | that armed a perf event    |
  +----------------------------+       +----------------------------+
  | drained by the one ctx     |       | each ctx drains only the   |
  | with the pf primitive on   |       | rings it owns              |
  +----------------------------+       +----------------------------+
          |                                     |
          +------------------+------------------+
                             |

                 drain on the aggregation boundary,
                 each report matched to a region
                             |
             +---------------+---------------+
             |                               |
        region age                    region probe hits
        demotion scheme               promotion scheme
```

That is the property this series exists to enable, and three things follow from it:

- One measurement drives both directions, at whichever scope the mode gives. Bandwidth comes from resctrl MBM, which is where the discussion on a platform-independent bandwidth signal for tiering has pointed [6]: a virtual-address context reads a monitoring group created for the processes it monitors, and a physical-address context reads the root monitoring group, so the same controller is scoped to a set of processes or to the whole machine without changing what it does. Reading it there also means a bandwidth allocation reserved for an application bounds what the controller optimises within. It spreads the hot set across nodes in the ratio that maximises achieved bandwidth when bandwidth is what the workload is short of, and it demotes cold pages off the near node in the same context at the same time -- either proactively under a fixed quota or only once DRAM is under memory pressure. Neither direction is configured as a target ratio; the proportion or weight the controller writes bounds how much memory moves while the score decides which pages move, and a hardware sample makes that score proportional to the traffic a region generates rather than one bit per page per scan. Earlier work on bandwidth-driven interleaving [4] measured the same effect from userspace.

- Bandwidth and capacity are expanded at once rather than one being chosen. Static weighted interleaving expands bandwidth, but it is a placement rule rather than a decision about any particular page, so cold pages keep their share of DRAM. NUMA balancing expands usable DRAM by promoting whatever is hot, but it does so without reference to what the far node can deliver, so it keeps promoting after the near node has stopped being the better place to read from. Here the hot set's distribution comes from a bandwidth measurement and the cold set is demoted on region age, in one context, so the far node's bandwidth is used and DRAM is not held by memory nothing is touching. The decision itself is a quota goal the kernel already carries:
node_eligible_mem_bp [8] bounds how much of a node a scheme may fill, so what the controller supplies is the bandwidth measurement, not the placement.

- When bandwidth is not the constraint it reverts to latency-first tiering on its own. The search starts with everything on the near node and moves memory outward only when the measurement improves, so on a workload that does not saturate that node no step outward improves anything: the hot set stays in DRAM for the lower access latency and cold pages keep being demoted. The same configuration covers both cases, with nothing to switch on and no ratio to revise.

## What the series adds

1. mm/damon/vaddr: support page fault access check primitive -- implements the page-fault primitive for a virtual address space context; it was available for physical addresses only. The marker whose fault reports the access is installed through the target mm and the vma of the sampling address directly, and prepare_access_checks() dispatches on the enabled primitive as the physical address space one does.

2. mm/damon/core: read the CPU number with preemption disabled -- the report path runs in fault context, which is preemptible. Read the CPU number with preemption disabled and keep it disabled across the report, so the recorded CPU stays consistent with the per-CPU ring the report is queued to.

3. mm/damon/paddr: lock the folio for the page fault primitive rmap walk -- take the folio lock unconditionally around the walk, as the other reverse mapping walks in DAMON do, so every mapping type reaches it locked, and release the reference the lookup took on each exit path.

4. mm/damon: add damos_node_eligible_mem_bp tracepoint -- a per-tick tracepoint over the node-eligible-memory quota goal evaluation, exposing the goal's target and current values, so the loop a bandwidth-driven controller steers is visible to a tracer.

5. mm/damon/core: add per-probe-class report rings and unified drain -- the substrate. Per-CPU SPSC rings an NMI-context source can publish into, partitioned by probe index into a global page-fault ring and a per-context perf ring, plus the kdamond-side drain that matches each report to a region by binary search over a per-target snapshot and credits it to that region's probe hits on the aggregation boundary. The address space of the target selects which address of a report is matched, and pid targets are filtered by thread group id.

6. mm/damon: add perf-event overflow handler feeding the report ring -- an ops-agnostic perf-event source whose overflow handler turns a PMU sample into a report, setting each address field the PMU reported as valid. A PMU that reports a virtual address only can therefore drive a virtual-address context, one that reports a physical address a physical-address context, and the same source serves either without a backend per address space. Per-CPU events are armed and released through cpuhp callbacks, and a per-PMU owner keeps two contexts from claiming the same PMU type.

7. mm/damon/ops-common: use probe-weighted score when probe weights are set -- lets a scheme's frequency subscore come from the probe hits, weighted per probe class, so what the sampler reported reaches the tiering decision. With no weights set the subscore comes from the access rate as before.

8. mm/damon: add perf_event prep for PMU-driven hotness probes -- the sysfs surface and the event lifecycle: a perf_event prep action carrying the PMU type, the event config and the sample attributes per probe, with per-CPU or single-instance arming depending on how many counters the PMU needs. Arming is deferred on a context built for a commit, so a weight-only commit leaves the running event untouched.

9. mm/damon/tests/drain-kunit: kunit for report rings and unified drain -- unit tests for the rings and the drain: inject and drain, overflow on wrap, producer routing by probe index, ring partitioning, pf-ring ownership, thread-group filtering, the address space match, and both primitives live on one context.

Patches 1-4 apply standalone on damon/next and are useful without the rest; 5-8 are the substrate and its first two consumers, and 9 covers them. If the first four read right, they need not wait on the rest of this series -- happy to repost them on their own for damon/next if that is easier to take. Two of them, 2 and 3, correct code that is already in mm-new rather than adding anything new, so they could instead go as a small fixes series carrying
Fixes: tags; guidance on which of the two routes is preferred is welcome.

## Userspace setup model

The runs were driven by an auto_tier subcommand added to damo on the branch below, which reads achieved bandwidth from resctrl MBM and searches for the hot set's distribution that maximises it, while the demotion scheme runs alongside it in the same context. That tooling is not part of this posting; it is on

https://github.com/ravis-opensrc/damo/tree/damo/auto-tier-bw-2026-09-08

- --bw_source resctrl is the option this adds: the controller reads achieved bandwidth from an MBM monitoring group and searches, rather than being given a distribution to install.
- Everything else -- intervals, schemes, filters, the probe's perf_event attributes -- is what the generator writes from the command line shown: 5 ms sampling, 100 ms aggregation, a 1 s ops update.
- Both configurations select proactive demotion, so they demote continuously rather than waiting for pressure, and both were left closed loop throughout.
- The search is the algorithm described in [4].

### Configuration A: AMD IBS Op, paddr ops, system-wide

```
$ sudo mount -t resctrl resctrl /sys/fs/resctrl
$ sudo python3 tools/damon_tier_gen.py --hotness ibs \
        --near_node 0 --far_node 4 \
        --cold_demote --cold_demote_mode proactive \
        -o tier.yaml
$ sudo damo auto_tier tier.yaml --bw_source resctrl --verbose
```

- Scope is the machine, not a process set. The distribution is steered through node_eligible_mem_bp quota goals over each node's own physical ranges, which the generator reads from /proc/iomem.
- AMD Turin, DRAM on node 0 and a CXL node. IBS Op at a sample_period of 262144 with sample_phys_addr set, which is what a physical-address context needs from the sample.
- Workload is a hot-and-cold mix so both schemes have something to act on:
multiload drives the bandwidth-hungry hot set, and a second process allocates on the near node, touches it once and goes idle, leaving pages that age out for the demotion scheme to find.

### Configuration B: Intel PEBS L3-miss, vaddr ops, per-PID

```
$ sudo mount -t resctrl resctrl /sys/fs/resctrl
$ sudo python3 tools/damon_tier_gen.py --hotness pebs \
        --pid $HOT_PID --pid $COLD_PID \
        --near_node 0 --far_node 1 \
        --cold_demote --cold_demote_mode proactive \
        --min_nr_regions 1000 --max_nr_regions 20000 \
        -o tier.yaml
$ sudo damo auto_tier tier.yaml --bw_source resctrl --verbose
```

- Scope is the processes the monitoring group names and nothing else. The distribution is steered through the hot scheme's DamosDest weights [9].
- Intel Granite Rapids, DRAM on node 0 and CXL on node 1. PEBS L3-miss at

    `sample_freq` 5003 with `precise_ip` 2.

- Same workload shape.

## What the runs show

Per-node reference, each figure measured by binding the same workload to one node:

```
  Granite Rapids, 64 threads x 8 GiB
    node 0   1.0 TB DRAM      DRAM only   269,885 MB/s
    node 1   2.0 TB CXL       CXL only    249,903 MB/s

  Turin, 32 threads x 4 GiB
    node 0   386 GB DRAM      DRAM only   113,596 MB/s
    node 4   1.0 TB CXL       CXL only     33,253 MB/s
```

- The two machines differ in the way that matters: on the Turin the far node is 3.4x slower than the near one, on the Granite Rapids the two are within 8%.
- Each configuration gets one graph, the climb: the run itself, one mark per decision taken from the bandwidth just measured, with the share it moved to under each mark. The settled line below it is the median over the samples the run went on to hold at that share.

### Configuration B, virtual-address mode

```
   MB/s                                                       climb
   460k |                                                          *
   440k |                                                      *
   420k |                                                  *
   400k |
   380k |                                             *
   360k |                                       *
   340k |                             *    *
   320k |                        *
   300k |              *    *
   280k |          *
   260k | *   *
         --------+------------+--------------+---------------+------------+--
                 1            4              7              10           13
                 92           80             68              56           44
                     decision index, near-node share (%)

   settled        share 44, 430,547 MB/s held
```

- Thirteen decisions to settle at 44%, no target given: about 60% more than the better of the two nodes on its own, and the share does not change again over the 81 samples that follow.
- Cold demotion runs over the same interval, in the same context, actuated by the same kdamond: 34 regions totalling 15.4 GiB applied, and the idle process ends with 4,099,980 pages -- 15.6 GiB -- on the CXL node having started on DRAM.
- The promotion scheme's applied byte count is flat across the demotion ramp, 291.1 GiB before against 291.3 GiB after, so the recovered capacity is the demotion scheme's.

### Configuration A, physical-address mode

```
   MB/s                                                   climb
   136k |
   134k |                                        *     *
   132k |                            *     *
   130k |                      *
   128k |
   126k |                *
   124k |
   122k |          *
   120k |
   118k |    *
        +----+-----+-----+-----+-----+-----+-----+-----+--
             1     2     3     4     5     6     7     8
             92    88    84    80    76    78    80    78
          decision index, near-node share (%)

   settled        share 78, 134,107 MB/s held
```

- Eight decisions to settle at 78, from a start with almost everything on the near node: the same algorithm reaches the optimum share on its own from the bandwidth reading alone, and cold pages are demoted to the far node while it does so.
- A distribution beats either end because both nodes supply at once: at the settled share the near node is supplying 104,618 MB/s against the 113,596 it reaches with the whole working set bound to it, so the further 37,669 MB/s arriving over CXL is bandwidth it could not have supplied at any share.
- The idle process ends with 2,863,156 pages -- 10.9 GiB -- on the CXL node, out of the 32 GiB it allocated on DRAM and stopped touching. Residency is the attributable figure in this mode rather than a per-scheme applied count, because the schemes share physical ranges.

### What the runs are meant to establish

Not the settled share and not the absolute bandwidth; those are properties of these machines and these workloads. What reproduces is:

- A controller reading achieved bandwidth converges unattended, from a configuration naming no target, to a distribution that beats either node on its own, and holds it once found.
- A demotion scheme recovers near-node capacity in the same context while that happens.
- One code path does both, steering node_eligible_mem_bp over physical ranges system-wide on one machine and DamosDest weights over a named process group on the other.

## Beyond a CPU PMU

Nothing above is specific to PEBS or IBS. A source qualifies if it can report an accessed address to the ring, which is the only thing the substrate asks of it. A CXL device's Hotness Monitoring Unit, whose r3.2 perf driver has been posted [7] exposing its hotlist through an AUX buffer, or a custom monitoring unit on an accelerator or a memory expander, reports exactly that, and a backend delivering those reports through a perf event reaches the same drain, the same probe hits and the same schemes already in the tree. Such a unit sees the traffic that reaches its own device rather than a system-wide view, so what it contributes is the hot set resident on that device: the tiering logic can promote those pages to DRAM without the host having to detect that hotness itself, and without depending on what produced the reports. That posting names driving tiering from such a unit as the intent and the in-kernel step as future work, and consuming it as a DAMON access-check primitive was raised there as one way to get it; a backend on this substrate is that path.

[1] v1 of this series

    https://lore.kernel.org/damon/[email protected]/
[2] Roadmap for extending DAMON beyond pte-accessed bit
    https://lore.kernel.org/damon/[email protected]/
[3] mm/damon: introduce perf event based access check
    https://lore.kernel.org/damon/[email protected]/

[4] B. Tabatabai, R. Jonnalagadda et al., "Bandwidth Speaks, We Listen:
Dynamic Interleaving for Tiered Memory", ISMM 2026.
https://dl.acm.org/doi/10.1145/3814942.3816137
[5] mm/damon/perf: add ARM SPE AUX backend

    https://lore.kernel.org/damon/[email protected]/

[6] A platform-independent subsystem for bandwidth information, and resctrl as that source
https://lore.kernel.org/linux-mm/[email protected]/
[7] CXL Hotness Monitoring Unit perf driver

    https://lore.kernel.org/linux-mm/[email protected]/
[8] mm/damon: add node_eligible_mem_bp goal metric, merged for v7.2
    https://lore.kernel.org/linux-mm/[email protected]/
[9] mm/damon/vaddr: allow interleaving in migrate_{hot,cold} actions, merged
    for v6.17
    https://lore.kernel.org/linux-mm/[email protected]/

Ravi Jonnalagadda (9):
mm/damon/vaddr: support page fault access check primitive mm/damon/core: read the CPU number with preemption disabled mm/damon/paddr: lock the folio for the page fault primitive rmap walk mm/damon: add damos_node_eligible_mem_bp tracepoint mm/damon/core: add per-probe-class report rings and unified drain mm/damon: add perf-event overflow handler feeding the report ring mm/damon/ops-common: use probe-weighted score when probe weights are
set
mm/damon: add perf_event prep for PMU-driven hotness probes mm/damon/tests/drain-kunit: kunit for report rings and unified drain

 include/linux/damon.h        |  136 ++++-
 include/trace/events/damon.h |   32 +
 mm/damon/Kconfig             |   18 +
 mm/damon/Makefile            |    1 +
 mm/damon/core.c              |  995 +++++++++++++++++++++++++++----
 mm/damon/ops-common.c        |   21 +-
 mm/damon/paddr.c             |   37 +-
 mm/damon/perf_source.c       |  489 +++++++++++++++
 mm/damon/perf_source.h       |   54 ++
 mm/damon/sysfs.c             |  266 ++++++++-
 mm/damon/tests/.kunitconfig  |    4 +
 mm/damon/tests/core-kunit.h  |    2 +-
 mm/damon/tests/drain-kunit.h | 1091 ++++++++++++++++++++++++++++++++++
 mm/damon/tests/perf-kunit.h  |  133 +++++
 mm/damon/vaddr.c             |   83 ++-
 15 files changed, 3220 insertions(+), 142 deletions(-)
 create mode 100644 mm/damon/perf_source.c
 create mode 100644 mm/damon/perf_source.h
 create mode 100644 mm/damon/tests/drain-kunit.h
 create mode 100644 mm/damon/tests/perf-kunit.h

base-commit: e1f34dce183a96fc93bf9a42dcdc0ec0bf82a3d2

Signature · 1 line

2.43.0

Hello Ravi,

Quoted 16 lines · On Thu, 10 Sep 2026 10:16:14 -0700 Ravi Jonnalagadda <[email protected]>…

This series lets DAMON take its access information from a hardware sampler instead of from a page-table scan, and lets a scheme's score be weighted by what that sampler reported.

The change from v1 [1] is that it is now built on the data attribute probe infrastructure that has since landed in mm-new: a PMU is expressed as one more probe on a context, with its own weight and probe hits, rather than as a mechanism of its own beside the ops set. The rest of this posting is the same substrate reworked onto that, plus two consumers.

This is not a merge request. The series is based on damon/next at the base-commit below, which moves, so the tree it was built and tested from is also on

https://github.com/ravis-opensrc/linux/tree/damon/perf-rfc-v2-09-06-26

Quoted 8 lines

It is posted for design feedback, on the substrate and on where it belongs in the roadmap for extending DAMON beyond the pte-accessed bit [2]. That roadmap's second milestone, now open, is a first data attribute monitored through damon_report_access(), and that is what a sampling PMU is here. So this series keeps that function and its callers and replaces its body: the reporting path a hardware sampler needs cannot take a mutex, and the drain has to reach a virtual-address context as well as a physical one. The shape of the ring, the drain and the sysfs surface are what is most useful to review.

First of all, thank you very much for sharing this great series.

I think the high level direction, especially utilizing data attributes infrastructure and interface is good and aligned with our beyond-page table accessed-bit plan.

Nevertheless, this version is highly ambitious, optimized and big. We will land only essential parts in milestone 2. The goal of milestone 2 is only essential functionality. We will further extend its functionality and optimize its performance, in multiple and parallel steps, in the phase for the milestone 3 (or, simply phase 3).

I understand that's also what you are aligned with. And you had to post this big series mainly because we just started milestone 2, and therefore you have nothing to really base on.

Let me further provide some comments including what looks especially big and optimized to me, and how those could be split into small pieces.

Quoted 12 lines

## Why a unified perf-event substrate

DAMON derives its access information from the PTE Accessed bit. A sampling PMU carries what that bit cannot: which addresses the hardware went to, and how often it went there. Many machines already have such a unit, and more than one kind of it, so what this series is after is letting DAMON's regions be tuned from whichever perf-based hardware source a machine offers rather than from the Accessed bit alone. A sampler does not arrive on a kdamond's terms, though: it delivers an address when the hardware decides to, in NMI context, with no relation to the sampling interval and no mm to walk.

Quoted 7 lines

The alternative is a backend per PMU vendor, each owning its own configuration, sysfs knobs and lifecycle. The perf-event direction [3] avoids that: let DAMON register kernel-counter perf events and consume samples from any sampling PMU the perf core already knows about. This series follows it, adding one substrate below the ops sets rather than an ops set per PMU -- a report ring that any in-kernel access source can push into, and a drain that folds those reports into region probe hits on the aggregation boundary the kdamond already has.

This perfectly aligns with the plan. We will use perf event abstraction for AMD IBS-like h/w features based access monitoring primitives.

Quoted 3 lines

What running it
across vendors needed on top of that direction is:

Quoted 1 line

- per-CPU lockless rings between the NMI sample handler and the kdamond drain,

I understand we need to make it lockless. I wonder if we have to make it per-CPU. I understand it will be better in terms of performance, especially on machines having many CPUs. That said, this feels like somewhat we can discuss in phase 3. And it would deserve to have sufficient discussions and performance evaluations.

Quoted 3 lines

- a ring partitioned by probe class, so the page-fault primitive and a PMU can populate one context without either seeing the other's reports,

This also feels like something that we can discuss later in phase 3.

Particularly, page-fault primitive is out of the scope of milestone 2. The path forward and the timeline for it are quite unclear as of now. It requires alignments with other subsystems that would be challenging and taking time. I have a very rough plan to try it again with read-write protection, though. I think it might be safer to just keep it out of the scope of your project for now.

Quoted 7 lines

- per-CPU events that follow CPU hotplug, armed when the kdamond starts and disarmed and drained when it stops,
- a per-PMU owner, so two contexts cannot claim the same PMU type,
- whichever address a PMU does report carried on the report and matched against the context's own address space, so one source serves a paddr or a vaddr context without a backend per address space.

These all soudns making sense to me. Nonetheless, I think we can scope milestone 2 to support only physical address and defer these things to the phase 3.

Quoted 7 lines

This is tested with PEBS on Intel and IBS on AMD, both configured as perf_event attributes on a probe and using the perf core's event plumbing rather than per-vendor MSR code. A third source has already been written against the same
ring: Kunwu Chan's ARM SPE backend [5], which reaches it through an AUX buffer drained in process context instead of an overflow callback, and which the roadmap [2] places in its third milestone.

Awesome, appreciate your huge effort on this!

Quoted 5 lines

The partitioning is what lets promotion and demotion run in one context. A sampler says which regions are hot; it says nothing about which are cold, because a sampler that reports nothing about a page cannot distinguish untouched from unsampled.

I'm not really sure. I think absence of samples for an address range can also mean the address range is cold? Actually the page table accessed bit based monitoring also use a sort of sampling, so I don't show real distinction.

Maybe you're right, but I think this deserves sufficient discussions and testing that we could defer to the phase 3.

Quoted 2 lines

Region age is what a demotion scheme matches on, and age comes from the page-fault primitive.

We would have age in perf event based mode, too. Isn't it?

Quoted 3 lines

With the ring partitioned by class both are live at
once: the probe supplies hotness, the primitive supplies age, and two schemes over the same regions can move memory in both directions under one kdamond.

Unless the needs are clearly confirmed, I'd prefer having single class for simplicity.

Quoted 25 lines

The two classes are not symmetric, and the asymmetry is in where a report comes from rather than in what it carries. A page fault has no monitoring context at report time, so a per-context ring is not expressible for it: those reports go to one global set of per-CPU rings, drained by the single context whose page-fault primitive is enabled. A PMU overflow handler does carry the context that armed the event, so those reports go to that context's own per-CPU rings and each context drains only what it owns. Both classes share the ring layout and the producer side, so the barrier pairing cannot drift between them.

```
  page-fault report                    PMU sample, NMI context
  no ctx at report time                handler carries the owning ctx
          |                                     |
          v                                     v
  +----------------------------+       +----------------------------+
  | one global per-CPU ring    |       | per-CPU rings per context  |
  | set, shared by every ctx   |       | that armed a perf event    |
  +----------------------------+       +----------------------------+
  | drained by the one ctx     |       | each ctx drains only the   |
  | with the pf primitive on   |       | rings it owns              |
  +----------------------------+       +----------------------------+
          |                                     |
          +------------------+------------------+
                             |

Quoted 8 lines

                 drain on the aggregation boundary,
                 each report matched to a region
                             |
             +---------------+---------------+
             |                               |
        region age                    region probe hits
        demotion scheme               promotion scheme
```

Again, I'd suggest to simply ignore page-fault report for now.

Quoted 37 lines

That is the property this series exists to enable, and three things follow from it:

- One measurement drives both directions, at whichever scope the mode gives. Bandwidth comes from resctrl MBM, which is where the discussion on a platform-independent bandwidth signal for tiering has pointed [6]: a virtual-address context reads a monitoring group created for the processes it monitors, and a physical-address context reads the root monitoring group, so the same controller is scoped to a set of processes or to the whole machine without changing what it does. Reading it there also means a bandwidth allocation reserved for an application bounds what the controller optimises within. It spreads the hot set across nodes in the ratio that maximises achieved bandwidth when bandwidth is what the workload is short of, and it demotes cold pages off the near node in the same context at the same time -- either proactively under a fixed quota or only once DRAM is under memory pressure. Neither direction is configured as a target ratio; the proportion or weight the controller writes bounds how much memory moves while the score decides which pages move, and a hardware sample makes that score proportional to the traffic a region generates rather than one bit per page per scan. Earlier work on bandwidth-driven interleaving [4] measured the same effect from userspace.

- Bandwidth and capacity are expanded at once rather than one being chosen. Static weighted interleaving expands bandwidth, but it is a placement rule rather than a decision about any particular page, so cold pages keep their share of DRAM. NUMA balancing expands usable DRAM by promoting whatever is hot, but it does so without reference to what the far node can deliver, so it keeps promoting after the near node has stopped being the better place to read from. Here the hot set's distribution comes from a bandwidth measurement and the cold set is demoted on region age, in one context, so the far node's bandwidth is used and DRAM is not held by memory nothing is touching. The decision itself is a quota goal the kernel already carries:
node_eligible_mem_bp [8] bounds how much of a node a scheme may fill, so what the controller supplies is the bandwidth measurement, not the placement.

Quoted 7 lines

- When bandwidth is not the constraint it reverts to latency-first tiering on its own. The search starts with everything on the near node and moves memory outward only when the measurement improves, so on a workload that does not saturate that node no step outward improves anything: the hot set stays in DRAM for the lower access latency and cold pages keep being demoted. The same configuration covers both cases, with nothing to switch on and no ratio to revise.

Sounds very interesting.

Quoted 9 lines

## What the series adds

1. mm/damon/vaddr: support page fault access check primitive -- implements the page-fault primitive for a virtual address space context; it was available for physical addresses only. The marker whose fault reports the access is installed through the target mm and the vma of the sampling address directly, and prepare_access_checks() dispatches on the enabled primitive as the physical address space one does.

As I mentioned above, I'd suggest to keep page fault primitive out of the scope for now.

Quoted 6 lines

2. mm/damon/core: read the CPU number with preemption disabled -- the report path runs in fault context, which is preemptible. Read the CPU number with preemption disabled and keep it disabled across the report, so the recorded CPU stays consistent with the per-CPU ring the report is queued to.

Sounds good and will be needed for milestone 2, too. I will reuse or reference this in my milestone 2 implemenation. This will be very helpful, thank you for sharing.

Quoted 6 lines

3. mm/damon/paddr: lock the folio for the page fault primitive rmap walk -- take the folio lock unconditionally around the walk, as the other reverse mapping walks in DAMON do, so every mapping type reaches it locked, and release the reference the lookup took on each exit path.

Again, let's keep page fault-based monitoring out of the scope for now.

Quoted 5 lines

4. mm/damon: add damos_node_eligible_mem_bp tracepoint -- a per-tick tracepoint over the node-eligible-memory quota goal evaluation, exposing the goal's target and current values, so the loop a bandwidth-driven controller steers is visible to a tracer.

This seems doesn't need to wait anything. If this turned out to be helpful, please feel free to separately send patches for this.

Quoted 9 lines

5. mm/damon/core: add per-probe-class report rings and unified drain -- the substrate. Per-CPU SPSC rings an NMI-context source can publish into, partitioned by probe index into a global page-fault ring and a per-context perf ring, plus the kdamond-side drain that matches each report to a region by binary search over a per-target snapshot and credits it to that region's probe hits on the aggregation boundary. The address space of the target selects which address of a report is matched, and pid targets are filtered by thread group id.

Sounds like good optimizations that we can discuss with tests in the phase 3.

Quoted 10 lines

6. mm/damon: add perf-event overflow handler feeding the report ring -- an ops-agnostic perf-event source whose overflow handler turns a PMU sample into a report, setting each address field the PMU reported as valid. A PMU that reports a virtual address only can therefore drive a virtual-address context, one that reports a physical address a physical-address context, and the same source serves either without a backend per address space. Per-CPU events are armed and released through cpuhp callbacks, and a per-PMU owner keeps two contexts from claiming the same PMU type.

This should be what also essential for milestone 2. I will reuse or reference this in my milestone 2 implemenation. This will be very helpful, thank you for sharing.

Nonetheless, virtual-address context considered parts may be skipped in milestone 2. We can discuss such things in the phase 3.

Quoted 5 lines

7. mm/damon/ops-common: use probe-weighted score when probe weights are set -- lets a scheme's frequency subscore come from the probe hits, weighted per probe class, so what the sampler reported reaches the tiering decision. With no weights set the subscore comes from the access rate as before.

I'm working on adding more attributes-only mode (probe weights are set) support to DAMOS. Hopefully we can converge on this. This is not a part of our planned milestone 2, but I will keep doing this in parallel, so hopefully full DAMOS support for attributes-only mode will be done around end of milestone 2. Let's keep discussion on the works.

Quoted 7 lines

8. mm/damon: add perf_event prep for PMU-driven hotness probes -- the sysfs surface and the event lifecycle: a perf_event prep action carrying the PMU type, the event config and the sample attributes per probe, with per-CPU or single-instance arming depending on how many counters the PMU needs. Arming is deferred on a context built for a commit, so a weight-only commit leaves the running event untouched.

This is also what would be essential for milestone 2. I will reuse or reference this in my milestone 2 implemenation. This will be very helpful, thank you for sharing.

Quoted 6 lines

9. mm/damon/tests/drain-kunit: kunit for report rings and unified drain -- unit tests for the rings and the drain: inject and drain, overflow on wrap, producer routing by probe index, ring partitioning, pf-ring ownership, thread-group filtering, the address space match, and both primitives live on one context.

Testing is always important and nice. Thank you for doing this.

Quoted 166 lines

Patches 1-4 apply standalone on damon/next and are useful without the rest; 5-8 are the substrate and its first two consumers, and 9 covers them. If the first four read right, they need not wait on the rest of this series -- happy to repost them on their own for damon/next if that is easier to take. Two of them, 2 and 3, correct code that is already in mm-new rather than adding anything new, so they could instead go as a small fixes series carrying
Fixes: tags; guidance on which of the two routes is preferred is welcome.

## Userspace setup model

The runs were driven by an auto_tier subcommand added to damo on the branch below, which reads achieved bandwidth from resctrl MBM and searches for the hot set's distribution that maximises it, while the demotion scheme runs alongside it in the same context. That tooling is not part of this posting; it is on

https://github.com/ravis-opensrc/damo/tree/damo/auto-tier-bw-2026-09-08

- --bw_source resctrl is the option this adds: the controller reads achieved bandwidth from an MBM monitoring group and searches, rather than being given a distribution to install.
- Everything else -- intervals, schemes, filters, the probe's perf_event attributes -- is what the generator writes from the command line shown: 5 ms sampling, 100 ms aggregation, a 1 s ops update.
- Both configurations select proactive demotion, so they demote continuously rather than waiting for pressure, and both were left closed loop throughout.
- The search is the algorithm described in [4].

### Configuration A: AMD IBS Op, paddr ops, system-wide

```
$ sudo mount -t resctrl resctrl /sys/fs/resctrl
$ sudo python3 tools/damon_tier_gen.py --hotness ibs \
        --near_node 0 --far_node 4 \
        --cold_demote --cold_demote_mode proactive \
        -o tier.yaml
$ sudo damo auto_tier tier.yaml --bw_source resctrl --verbose
```

- Scope is the machine, not a process set. The distribution is steered through node_eligible_mem_bp quota goals over each node's own physical ranges, which the generator reads from /proc/iomem.
- AMD Turin, DRAM on node 0 and a CXL node. IBS Op at a sample_period of 262144 with sample_phys_addr set, which is what a physical-address context needs from the sample.
- Workload is a hot-and-cold mix so both schemes have something to act on:
multiload drives the bandwidth-hungry hot set, and a second process allocates on the near node, touches it once and goes idle, leaving pages that age out for the demotion scheme to find.

### Configuration B: Intel PEBS L3-miss, vaddr ops, per-PID

```
$ sudo mount -t resctrl resctrl /sys/fs/resctrl
$ sudo python3 tools/damon_tier_gen.py --hotness pebs \
        --pid $HOT_PID --pid $COLD_PID \
        --near_node 0 --far_node 1 \
        --cold_demote --cold_demote_mode proactive \
        --min_nr_regions 1000 --max_nr_regions 20000 \
        -o tier.yaml
$ sudo damo auto_tier tier.yaml --bw_source resctrl --verbose
```

- Scope is the processes the monitoring group names and nothing else. The distribution is steered through the hot scheme's DamosDest weights [9].
- Intel Granite Rapids, DRAM on node 0 and CXL on node 1. PEBS L3-miss at

    `sample_freq` 5003 with `precise_ip` 2.

- Same workload shape.

## What the runs show

Per-node reference, each figure measured by binding the same workload to one node:

```
  Granite Rapids, 64 threads x 8 GiB
    node 0   1.0 TB DRAM      DRAM only   269,885 MB/s
    node 1   2.0 TB CXL       CXL only    249,903 MB/s

  Turin, 32 threads x 4 GiB
    node 0   386 GB DRAM      DRAM only   113,596 MB/s
    node 4   1.0 TB CXL       CXL only     33,253 MB/s
```

- The two machines differ in the way that matters: on the Turin the far node is 3.4x slower than the near one, on the Granite Rapids the two are within 8%.
- Each configuration gets one graph, the climb: the run itself, one mark per decision taken from the bandwidth just measured, with the share it moved to under each mark. The settled line below it is the median over the samples the run went on to hold at that share.

### Configuration B, virtual-address mode

```
   MB/s                                                       climb
   460k |                                                          *
   440k |                                                      *
   420k |                                                  *
   400k |
   380k |                                             *
   360k |                                       *
   340k |                             *    *
   320k |                        *
   300k |              *    *
   280k |          *
   260k | *   *
         --------+------------+--------------+---------------+------------+--
                 1            4              7              10           13
                 92           80             68              56           44
                     decision index, near-node share (%)

   settled        share 44, 430,547 MB/s held
```

- Thirteen decisions to settle at 44%, no target given: about 60% more than the better of the two nodes on its own, and the share does not change again over the 81 samples that follow.
- Cold demotion runs over the same interval, in the same context, actuated by the same kdamond: 34 regions totalling 15.4 GiB applied, and the idle process ends with 4,099,980 pages -- 15.6 GiB -- on the CXL node having started on DRAM.
- The promotion scheme's applied byte count is flat across the demotion ramp, 291.1 GiB before against 291.3 GiB after, so the recovered capacity is the demotion scheme's.

### Configuration A, physical-address mode

```
   MB/s                                                   climb
   136k |
   134k |                                        *     *
   132k |                            *     *
   130k |                      *
   128k |
   126k |                *
   124k |
   122k |          *
   120k |
   118k |    *
        +----+-----+-----+-----+-----+-----+-----+-----+--
             1     2     3     4     5     6     7     8
             92    88    84    80    76    78    80    78
          decision index, near-node share (%)

   settled        share 78, 134,107 MB/s held
```

- Eight decisions to settle at 78, from a start with almost everything on the near node: the same algorithm reaches the optimum share on its own from the bandwidth reading alone, and cold pages are demoted to the far node while it does so.
- A distribution beats either end because both nodes supply at once: at the settled share the near node is supplying 104,618 MB/s against the 113,596 it reaches with the whole working set bound to it, so the further 37,669 MB/s arriving over CXL is bandwidth it could not have supplied at any share.
- The idle process ends with 2,863,156 pages -- 10.9 GiB -- on the CXL node, out of the 32 GiB it allocated on DRAM and stopped touching. Residency is the attributable figure in this mode rather than a per-scheme applied count, because the schemes share physical ranges.

### What the runs are meant to establish

Not the settled share and not the absolute bandwidth; those are properties of these machines and these workloads. What reproduces is:

Quoted 8 lines

- A controller reading achieved bandwidth converges unattended, from a configuration naming no target, to a distribution that beats either node on its own, and holds it once found.
- A demotion scheme recovers near-node capacity in the same context while that happens.
- One code path does both, steering node_eligible_mem_bp over physical ranges system-wide on one machine and DamosDest weights over a named process group on the other.

Nice testing and measurement, thank you for sharing! I'm happy the future DAMON could help this!

Quoted 17 lines

## Beyond a CPU PMU

Nothing above is specific to PEBS or IBS. A source qualifies if it can report an accessed address to the ring, which is the only thing the substrate asks of it. A CXL device's Hotness Monitoring Unit, whose r3.2 perf driver has been posted [7] exposing its hotlist through an AUX buffer, or a custom monitoring unit on an accelerator or a memory expander, reports exactly that, and a backend delivering those reports through a perf event reaches the same drain, the same probe hits and the same schemes already in the tree. Such a unit sees the traffic that reaches its own device rather than a system-wide view, so what it contributes is the hot set resident on that device: the tiering logic can promote those pages to DRAM without the host having to detect that hotness itself, and without depending on what produced the reports. That posting names driving tiering from such a unit as the intent and the in-kernel step as future work, and consuming it as a DAMON access-check primitive was raised there as one way to get it; a backend on this substrate is that path.

That makes sense to me. Thank you for sharing this nice survey and theories!

Quoted 24 lines

[1] v1 of this series

    https://lore.kernel.org/damon/[email protected]/
[2] Roadmap for extending DAMON beyond pte-accessed bit
    https://lore.kernel.org/damon/[email protected]/
[3] mm/damon: introduce perf event based access check
    https://lore.kernel.org/damon/[email protected]/

[4] B. Tabatabai, R. Jonnalagadda et al., "Bandwidth Speaks, We Listen:
Dynamic Interleaving for Tiered Memory", ISMM 2026.
https://dl.acm.org/doi/10.1145/3814942.3816137
[5] mm/damon/perf: add ARM SPE AUX backend

    https://lore.kernel.org/damon/[email protected]/

[6] A platform-independent subsystem for bandwidth information, and resctrl as that source
https://lore.kernel.org/linux-mm/[email protected]/
[7] CXL Hotness Monitoring Unit perf driver

    https://lore.kernel.org/linux-mm/[email protected]/
[8] mm/damon: add node_eligible_mem_bp goal metric, merged for v7.2
    https://lore.kernel.org/linux-mm/[email protected]/
[9] mm/damon/vaddr: allow interleaving in migrate_{hot,cold} actions, merged
    for v6.17
    https://lore.kernel.org/linux-mm/[email protected]/

Quoted 11 lines

Ravi Jonnalagadda (9):
mm/damon/vaddr: support page fault access check primitive mm/damon/core: read the CPU number with preemption disabled mm/damon/paddr: lock the folio for the page fault primitive rmap walk mm/damon: add damos_node_eligible_mem_bp tracepoint mm/damon/core: add per-probe-class report rings and unified drain mm/damon: add perf-event overflow handler feeding the report ring mm/damon/ops-common: use probe-weighted score when probe weights are
set
mm/damon: add perf_event prep for PMU-driven hotness probes mm/damon/tests/drain-kunit: kunit for report rings and unified drain

As this series is for high level discussion, and the code will significantly changed after rebasing on milestone 2 output, I wouldn't go deep into each line of the code for now. Feel free to let me know if you want.

Quoted 23 lines

 include/linux/damon.h        |  136 ++++-
 include/trace/events/damon.h |   32 +
 mm/damon/Kconfig             |   18 +
 mm/damon/Makefile            |    1 +
 mm/damon/core.c              |  995 +++++++++++++++++++++++++++----
 mm/damon/ops-common.c        |   21 +-
 mm/damon/paddr.c             |   37 +-
 mm/damon/perf_source.c       |  489 +++++++++++++++
 mm/damon/perf_source.h       |   54 ++
 mm/damon/sysfs.c             |  266 ++++++++-
 mm/damon/tests/.kunitconfig  |    4 +
 mm/damon/tests/core-kunit.h  |    2 +-
 mm/damon/tests/drain-kunit.h | 1091 ++++++++++++++++++++++++++++++++++
 mm/damon/tests/perf-kunit.h  |  133 +++++
 mm/damon/vaddr.c             |   83 ++-
 15 files changed, 3220 insertions(+), 142 deletions(-)
 create mode 100644 mm/damon/perf_source.c
 create mode 100644 mm/damon/perf_source.h
 create mode 100644 mm/damon/tests/drain-kunit.h
 create mode 100644 mm/damon/tests/perf-kunit.h

Quoted 5 lines

base-commit: e1f34dce183a96fc93bf9a42dcdc0ec0bf82a3d2 --
2.43.0

Thanks,
SJ

Quoted 8 lines · SJ Park

Hello Ravi,

Quoted 3 lines · On Thu, 10 Sep 2026 10:16:14 -0700 Ravi Jonnalagadda <[email protected]>…

This series lets DAMON take its access information from a hardware sampler instead of from a page-table scan, and lets a scheme's score be weighted by what that sampler reported.

Quoted 2 lines

First of all, thank you very much for sharing this great series.

Thanks SJ for the detailed review and the milestone 2 / phase 3 clarifications.

Quoted 10 lines

Quoted 4 lines

What running it
across vendors needed on top of that direction is:

- per-CPU lockless rings between the NMI sample handler and the kdamond drain,

I understand we need to make it lockless. I wonder if we have to make it per-CPU. I understand it will be better in terms of performance, especially on machines having many CPUs. That said, this feels like somewhat we can discuss in phase 3. And it would deserve to have sufficient discussions and performance evaluations.

The per-CPU structure is not a performance optimization we can defer -- it is required by the calling context. A perf-event overflow handler runs in NMI context, which cannot take a mutex or any sleeping lock. damon_report_access() takes a mutex, so it cannot be called from there.

Understood. I included the page-fault source in v2 because the December 2025 RFC [1] that introduced damon_report_access() had it as the primary consumer, and I wanted to carry forward that ability. Since it is out of scope for milestone 2, I will drop it from v3.

Patches 1, 2, and 3 will be dropped from the v3 submission since they are all tied to the page fault path.

Quoted 9 lines

Quoted 8 lines

- per-CPU events that follow CPU hotplug, armed when the kdamond starts and disarmed and drained when it stops,
- a per-PMU owner, so two contexts cannot claim the same PMU type,
- whichever address a PMU does report carried on the report and matched against the context's own address space, so one source serves a paddr or a vaddr context without a backend per address space.

Quoted 4 lines

These all soudns making sense to me. Nonetheless, I think we can scope milestone 2 to support only physical address and defer these things to the phase 3.

Got it. Will scope v3 to PA only. so included results for v3 would be based on AMD IBS testing.

Quoted 21 lines

Quoted 8 lines

This is tested with PEBS on Intel and IBS on AMD, both configured as perf_event attributes on a probe and using the perf core's event plumbing rather than per-vendor MSR code. A third source has already been written against the same
ring: Kunwu Chan's ARM SPE backend [5], which reaches it through an AUX buffer drained in process context instead of an overflow callback, and which the roadmap [2] places in its third milestone.

Awesome, appreciate your huge effort on this!

Quoted 5 lines

The partitioning is what lets promotion and demotion run in one context. A sampler says which regions are hot; it says nothing about which are cold, because a sampler that reports nothing about a page cannot distinguish untouched from unsampled.

I'm not really sure. I think absence of samples for an address range can also mean the address range is cold? Actually the page table accessed bit based monitoring also use a sort of sampling, so I don't show real distinction.

Quoted 8 lines

Maybe you're right, but I think this deserves sufficient discussions and testing that we could defer to the phase 3.

Quoted 2 lines

Region age is what a demotion scheme matches on, and age comes from the page-fault primitive.

We would have age in perf event based mode, too. Isn't it?

You are right. Region age accumulates whenever nr_accesses stays at zero across aggregation boundaries, and that holds whether the zero comes from a PTE scan or from an empty perf-event drain.

I was initially concerned that a sparse sampling PMU might not cover every cold region in every aggregation window, leaving silence that could be mistaken for cold,
whereas page faults provide higher spatial coverage for confirming first-access. Having both together was intended as a defence against that gap.

Based on your observation I retested this on hardware. On AMD Turin the cold demotion scheme found and demoted the idle working set correctly using only the hardware-sampled IBS signal -- nr_accesses aged to zero for regions that genuinely had no traffic, and the scheme acted on age as expected. The combined design may still be worth exploring later when page fault
is considered to be reintroduced in phase 3.

Quoted 8 lines

Quoted 4 lines

With the ring partitioned by class both are live at
once: the probe supplies hotness, the primitive supplies age, and two schemes over the same regions can move memory in both directions under one kdamond.

Unless the needs are clearly confirmed, I'd prefer having single class for simplicity.

With the page fault primitive out of scope there is no case for two ring classes. I will prepare v3 with a single class.

Quoted 8 lines

Quoted 4 lines

4. mm/damon: add damos_node_eligible_mem_bp tracepoint -- a per-tick tracepoint over the node-eligible-memory quota goal evaluation, exposing the goal's target and current values, so the loop a bandwidth-driven controller steers is visible to a tracer.

This seems doesn't need to wait anything. If this turned out to be helpful, please feel free to separately send patches for this.

Yes. It is quite useful to track goal convergence. Will send a single patch targeting mm-new.

Summary of what v3 will contain (5 patches, former patches 5-9 of v2):
- paddr-only, single ring class, no page fault primitive
- per-CPU rings retained (required by NMI calling context, not an optimization)
- perf-event overflow handler (paddr path, vaddr parts deferred to phase 3)
- sysfs/lifecycle surface
- probe-weighted score
- kunit tests updated for the simplified scope

Separate send targeting mm-new:
- tracepoint (former patch 4)

[1] RFC PATCH v3 00/37: mm/damon: introduce
per-CPUs/threads/write/read monitoring

    https://lore.kernel.org/damon/[email protected]/

This series lets DAMON take its access information from a hardware sampler instead of from a page-table scan, and lets a scheme's score be weighted by what that sampler reported.

From: "Lian Wang (ProcessMission)" <[email protected]>

Hi Ravi and SJ,

Thank you for sharing this work. The report path looks potentially useful for a DAMON observation-granularity problem that we are investigating.

The scenario originated from an issue reported by SXF, and we are planning to share and discuss it at LPC. We have now reproduced the underlying observation difference on both a local PC and an x86 server. The server run below is our larger, controlled reproduction.

Our test monitors a QEMU process whose guest memory is backed by a shared tmpfs file. The guest repeatedly writes one 4 KiB page in every 2 MiB range of a resident 48 GiB data set. The unique 4 KiB spatial coverage of one sweep is about 96 MiB, or 0.195% of that data set.

Using vaddr DAMON with a 500 ms sampling interval, 20 second aggregation interval, and 500--2000 regions, two opposite-order runs reported a mean hot ratio of 1.751% without shmem PMD mappings and 75.836% with them. A control that writes the full 2 MiB ranges reported 75.366% with PMD mappings. Hence, in this workload the current observation does not distinguish spatially sparse 4 KiB activity from dense activity once the large mappings are present.

We are testing existing DAMON strategy mitigations first. As a next step, we plan to replay recorded access addresses through the report/probe decision path, independently of a live PMU, and check whether DAMOS can distinguish the sparse and dense cases. In that experiment, raw report frequency and unique 4 KiB spatial coverage will remain separate quantities; lack of reports will remain unknown when source coverage or loss is insufficient.

If the report evidence makes the decision reliable, the response we want to evaluate is an internal, bounded PMD mapping demotion (splitting the mapping while retaining the underlying THP folio), rather than exposing physical folio split as a DAMOS action. We will first keep the response stat-only and archive the would-act ranges. After the replay result is repeatable, we can replace the replay producer with an appropriate hardware source and share the results.

This seems like a useful consumer experiment for the report interface, even if the vaddr and optimized transport parts belong to the later phase. Please let us know if this direction conflicts with the intended report semantics.

We will share the strategy, replay, and mapping-decision results once they are repeatable. We are also reviewing and testing this series, and will follow up with the appropriate tag(s) for the exact revision if that work justifies them.

The observation makes sense. However, does the sparse access pattern realistic? If so, what is the purpose or expected benefit of using huge page for workloads having such access pattern?

I'm not very sure if this is the right direction. PMD mapping demotion sounds like you just don't want to use huge pages. If so, you could disable huge pages.

technically speaking, it is not the report semantics. Reporting allows any information to be reported. Page faults like information could be coarse grained, same to the current page table accessed-bit based one. Only finer grained access primitive reports, like those from perf events, would increase the accuracy for the sparse access pattern monitoring.

Hi SJ,

Thank you for the questions. I mixed the original field scenario and our controlled reproducer in my previous mail. I also jumped from the observation problem to PMD mapping demotion before explaining why the deployment needs both huge pages and finer-grained observation. That made it sound as if our goal was simply to stop using huge pages.

Quoted 3 lines

The observation makes sense. However, does the sparse access pattern realistic? If so, what is the purpose or expected benefit of using huge page for workloads having such access pattern?

The original case is not a lab-created request to use huge pages for a sparse microbenchmark. It was reported by SXF from a KVM/QEMU deployment where the VM has a large memory allocation backed by a shared tmpfs file, host THP is enabled, and Oracle runs inside the guest. They use host-side DAMON to measure the VM's hot-memory proportion. With the same business memory use, they observed a much larger hot proportion when THP was enabled.

The guest tmpfs 4 KiB/2 MiB writer is their controlled diagnostic case for isolating that observation; it is not a claim that the Oracle workload is exactly a one-page-per-2-MiB loop. We then reproduced and quantified the same effect on a PC and on the x86 server.

For this deployment, keeping large pages is a real requirement at the host virtualization layer. QEMU owns a large resident guest-RAM mapping. Host THP allows that mapping to use PMDs and allows KVM to use large secondary mappings, reducing host page-table memory, TLB and nested-page-walk pressure, and KVM mapping/fault overhead. These benefits are independent of whether every 4 KiB page inside a particular 2 MiB range is hot at one observation time.

The performance requirement and monitoring use case come from the deployment, not from our lab model. We will also report the measured Oracle/VM benefit and the production access distribution when those data are ready. The diagnostic result establishes the observation mechanism, but it should not substitute for those workload-level measurements.

Quoted 4 lines

I'm not very sure if this is the right direction. PMD mapping demotion sounds like you just don't want to use huge pages. If so, you could disable huge pages.

I agree. My reasoning in the previous mail was too jumpy: I went from a coarse observation directly to a possible MM response, and that obscured the actual goal. PMD mapping demotion is not part of our current proposed solution. The current direction is to keep both the huge folio and PMD mapping unchanged and use a genuinely fine-grained access primitive to improve the observation and the stat-only decision. We should first find out whether that is sufficient before discussing any mapping change at all.

With this context clarified, we will return to the immediate work: use the existing IBS/perf-event proposal as the concrete strategy under test, and see whether its fine-grained evidence can improve the observation and DAMOS decision for this case. We will use the results to test and review this series, report both improvements and remaining gaps, and not assume in advance that it is the final solution.

I hope this clarifies why keeping huge pages and requesting finer-grained observation are not contradictory requirements in this case.

If I am still misunderstanding any part of your questions, or if any part of this explanation remains unclear, please continue to correct me. We will keep sharing our findings and would like to make sure we are aligned before going further.

Quoted 12 lines · On Wed, 16 Sep 2026 11:44:30 +0800 "Lian Wang (ProcessMission)"…

Hi SJ,

Thank you for the questions. I mixed the original field scenario and our controlled reproducer in my previous mail. I also jumped from the observation problem to PMD mapping demotion before explaining why the deployment needs both huge pages and finer-grained observation. That made it sound as if our goal was simply to stop using huge pages.

Quoted 3 lines

The observation makes sense. However, does the sparse access pattern realistic? If so, what is the purpose or expected benefit of using huge page for workloads having such access pattern?

Quoted 6 lines

The original case is not a lab-created request to use huge pages for a sparse microbenchmark. It was reported by SXF from a KVM/QEMU deployment where the VM has a large memory allocation backed by a shared tmpfs file, host THP is enabled, and Oracle runs inside the guest. They use host-side DAMON to measure the VM's hot-memory proportion. With the same business memory use, they observed a much larger hot proportion when THP was enabled.

Thank you for clarifying this. So I understand it is an issue from a real world workload. I'm still curious how sparse the real access pattern in the real world workload is, and if there is a low haning fruit. There might be rooms to improve in their DAMON usage.

Could you share more details about the real world setup and findings? More specifically, how they define the hot proportion and how they collect the number? What is DAMON setup? What are the real numbers they saw?

Also, I understand they want to understand real working set size of guests. Could you further share what they are planning to do with the monitored working set size? The required level of accuracy of the monitoring could depend on the purpose.

Quoted 6 lines

The guest tmpfs 4 KiB/2 MiB writer is their controlled diagnostic case for isolating that observation; it is not a claim that the Oracle workload is exactly a one-page-per-2-MiB loop. We then reproduced and quantified the same effect on a PC and on the x86 server.

Quoted 6 lines

For this deployment, keeping large pages is a real requirement at the host virtualization layer. QEMU owns a large resident guest-RAM mapping. Host THP allows that mapping to use PMDs and allows KVM to use large secondary mappings, reducing host page-table memory, TLB and nested-page-walk pressure, and KVM mapping/fault overhead. These benefits are independent of whether every 4 KiB page inside a particular 2 MiB range is hot at one observation time.

Thank you for clarifying this. Now I understand why they use THP, and I fully agree they should keep using THP.

[...]

Quoted 5 lines

The performance requirement and monitoring use case come from the deployment, not from our lab model. We will also report the measured Oracle/VM benefit and the production access distribution when those data are ready. The diagnostic result establishes the observation mechanism, but it should not substitute for those workload-level measurements.

Thanks to your detailed and nice explanation, I don't doubt about the needs to use huge pages. So I wouldn't insist on Oracle/VM benefit from huge pages.

I'm still curious the real access distribution, though. I understand you don't have the data yet. It would be very helpful if you could share that in future.

Quoted 5 lines

Quoted 4 lines

I'm not very sure if this is the right direction. PMD mapping demotion sounds like you just don't want to use huge pages. If so, you could disable huge pages.

Quoted 7 lines

I agree. My reasoning in the previous mail was too jumpy: I went from a coarse observation directly to a possible MM response, and that obscured the actual goal. PMD mapping demotion is not part of our current proposed solution. The current direction is to keep both the huge folio and PMD mapping unchanged and use a genuinely fine-grained access primitive to improve the observation and the stat-only decision. We should first find out whether that is sufficient before discussing any mapping change at all.

Thank you for clarifying. Yes, if you really need sub-pmd access, I believe the perf event based DAMON could be more useful. Nonetheless, as I requested above, knowing the DAMON usage and purpose in more detail might let us finding a low hanging fruit.

[...]

Quoted 6 lines

With this context clarified, we will return to the immediate work: use the existing IBS/perf-event proposal as the concrete strategy under test, and see whether its fine-grained evidence can improve the observation and DAMOS decision for this case. We will use the results to test and review this series, report both improvements and remaining gaps, and not assume in advance that it is the final solution.

That makes sense. Because Ravi thankfully already implemented this nice working version prototype, there is no reason to not test it right now.

Quoted 3 lines

I hope this clarifies why keeping huge pages and requesting finer-grained observation are not contradictory requirements in this case.

Yes, it is now clear to me. Thank you for patiently englightening me, Lian.

Quoted 5 lines

If I am still misunderstanding any part of your questions, or if any part of this explanation remains unclear, please continue to correct me. We will keep sharing our findings and would like to make sure we are aligned before going further.

I believe testing the perf event based monitoring with Ravi's implementation is a good path forward.

I'm curious about the real setup and results your clients got from their real world wokload. But that's not necessarily a blocker of the perf event based monitoring test.

Thanks,
SJ

[...]

I still want to better understand the real world use case before digging into the specific solution. But, I was thinking perf event based monitoring might not feasible for the case.

That is, we can request DAMON to do some preparation action for the data attribute sampling, using damon_prep. At the moment, set_pgidle prep action is supported. Maybe we could add a new prep action, say, split_pmd? Then, the following sampling memory access check probe (would be a kind of "allow pidle_unset" probe filter) will be able to show the access on exactly the pte accessed bit, not the pmd accessed bit.

Breaking pmd would cause the overhead, but it would be capped by the max_nr_regions. So the user could at least consider about the tradeoff. I think this is simpler and more intuitive than DAMOS-based pmd breaking, at least.

Hi SJ,

Thank you for thinking further about this and for suggesting the damon_prep direction. Let me first clarify the real-world use case and correct one expectation that I created in my previous mail.

SXF develops a hyperconverged infrastructure platform. The workloads running in the guests are customer workloads, and their types, memory sizes and access distributions are not controlled by us. Oracle is the workload in the case that exposed the problem, but we cannot define one Oracle access distribution as representative of the platform or provide one generally applicable workload-specific THP performance number.

The platform has two requirements:

1. Host THP is enabled by default. Large mappings generally provide useful translation and virtualization benefits, so disabling THP merely to make monitoring easier is not an acceptable general solution.

2. We also need an estimate of each VM's active memory as an input to later memory-tiering configuration. If a small number of distributed 4 KiB accesses makes many 2 MiB mappings appear fully active, the hot-tier requirement can be substantially overestimated. VM sizes vary, so the absolute error can become large on a large VM when accesses are spread over many huge mappings.

The reported setup is a KVM/QEMU VM whose large guest-RAM allocation is backed by a shared tmpfs file. Oracle runs in the guest, host THP is enabled, and host DAMON vaddr monitoring targets the QEMU process. The field observation is that, for the same business memory-use case, the measured hot proportion is much higher with THP. The guest-side tmpfs 4 KiB/2 MiB writer is a controlled diagnostic for isolating this mechanism, not a model of the complete Oracle access distribution.

The DAMON setup is:

operations=vaddr monitoring_attrs/nr_regions/min=500 monitoring_attrs/nr_regions/max=2000 monitoring_attrs/intervals/sample_us=500000 monitoring_attrs/intervals/aggr_us=20000000 monitoring_attrs/intervals/update_us=60000000 schemes/nr_schemes=1
schemes/0/action=stat schemes/0/access_pattern/age/min=0 schemes/0/access_pattern/age/max=18446744073709551615 schemes/0/access_pattern/nr_accesses/min=1 schemes/0/access_pattern/nr_accesses/max=18446744073709551615 schemes/0/access_pattern/sz/min=0 schemes/0/access_pattern/sz/max=18446744073709551615

The hot proportion is derived from the bytes selected by this stat-only scheme (at least one observed access in a 20-second aggregation window) relative to the monitored QEMU scope. It is used for later configuration, not as a request for DAMOS to split or migrate memory.

Quoted 3 lines

I still want to better understand the real world use case before digging into the specific solution. But, I was thinking perf event based monitoring might not feasible for the case.

We agree that perf-event feasibility should be established rather than assumed. We will test the address space reported while a KVM vCPU is running, per-VM attribution, coverage and loss before relying on that path. If you had a specific feasibility concern in mind, please let us know so that we can include it in the test.

Quoted 8 lines

That is, we can request DAMON to do some preparation action for the data attribute sampling, using damon_prep. At the moment, set_pgidle prep action is supported. Maybe we could add a new prep action, say, split_pmd? Then, the following sampling memory access check probe (would be a kind of "allow pidle_unset" probe filter) will be able to show the access on exactly the pte accessed bit, not the pmd accessed bit.

Quoted 4 lines

Breaking pmd would cause the overhead, but it would be capped by the max_nr_regions. So the user could at least consider about the tradeoff. I think this is simpler and more intuitive than DAMOS-based pmd breaking, at least.

This placement makes sense to us. A split_pmd preparation for access sampling is clearer than exposing PMD breaking as a DAMOS action. We understand it as refining the mapping used for observation, without splitting the underlying huge folio, and then letting the following probe observe PTE accessed bits.

We will follow this idea and validate it. In particular, we will measure the active-memory result and mapping cost, check whether max_nr_regions also bounds the cumulative number of mappings that remain PTE-mapped over time, and verify whether splitting the host process PMD is sufficient in the QEMU case or KVM's large secondary mapping remains the limiting layer. We will compare it with the perf-event path, stat-only first, without proposing a user-visible DAMOS split action or physical folio split.

Please rest assured that we see this as a long-term technical discussion and collaboration, not an attempt to rush one mechanism into the tree. We will continue sharing both positive and negative results. I also plan to present the scenario, evidence and tradeoffs in more detail at LPC. Please feel free to share any concerns or ideas at any time; we are happy to keep iterating on them together.

This topic has now become broader than Ravi's hardware-sampled report series. If you think it would keep the discussions clearer, I can start a separate thread focused on huge-page observation and the damon_prep approach, while keeping perf-event implementation and testing comments on Ravi's thread.

Please let me know whether this clarifies the intended use and whether our reading of the split_pmd damon_prep proposal matches what you had in mind.