DAMON Requirements for Access-aware MM of Future
Subsystems (2)
SeongJae Park
maintainerHi all,
I find a few interesting and promising projects that aim to do efficient access pattern-aware memory management of near future, including below (alphabetically sorted).
- CXL hotness monitoring unit
(https://lore.kernel.org/[email protected])
- Memory tiering fainess by per-cgroup control of promotion and demotion
(https://lore.kernel.org/[email protected])
- Promotion of unmapped page cache folios
(https://lore.kernel.org/[email protected])
- Slow-tier page promotion based on PTE A bit
(https://lore.kernel.org/[email protected])
- Workingset reporting
(https://lore.kernel.org/[email protected])
The goal of DAMON is to help accelerating such developments by being a framework that can reduce fundamental efforts for monitoring memory access patterns and managing memory using the information. AWS Aurora Serverless v2 and SK hynix are successfully using DAMON in the way for proactive memory reclamation[1] and CXL memory tiering[2].
To further deliver such benefits for the ongoing and future projects, we need to better understand what the projects really need, how DAMON can provide those now or in future, and if there are alternatives better than DAMON. Regardless of the conclusion about DAMON, the works apparently have common parts, so the discussion will benefit all.
I propose to have the discussion at LSF/MM/BPF. In the session, I will briefly introduce the works and possible DAMON usages, and continue the open discussion for better understanding each other. The discussion will not be limited to DAMON and abovely mentioned projects but possible alternatives and general access-aware memory management projects. After the discussion, we will hopefully find ways to efficiently collaborate, or at least do not disturb each other.
[1] https://assets.amazon.science/ee/a4/41ff11374f2f865e5e24de11bd17/resource-management-in-aurora-serverless.pdf
[2] https://github.com/skhynix/hmsdk/wiki/Capacity-Expansion
Thanks,
SJ
Quoted 7 lines · SeongJae Park
Hi all,
I find a few interesting and promising projects that aim to do efficient access pattern-aware memory management of near future, including below (alphabetically sorted).
Quoted 6 lines
- CXL hotness monitoring unit
(https://lore.kernel.org/[email protected])
- Memory tiering fainess by per-cgroup control of promotion and demotion
(https://lore.kernel.org/[email protected])
- Promotion of unmapped page cache folios
(https://lore.kernel.org/[email protected])
I'm not sure how DAMON can help with this one. As I understand DAMON, it monitors accesses to user addresses. This patchset is trying to solve the problem for file pages which aren't mapped to userspace at all. ie only accessed through read() and write().
Quoted 15 lines · Matthew Wilcox
Quoted 7 lines · SeongJae Park
Hi all,
I find a few interesting and promising projects that aim to do efficient access pattern-aware memory management of near future, including below (alphabetically sorted).
Quoted 6 lines
- CXL hotness monitoring unit
(https://lore.kernel.org/[email protected])
- Memory tiering fainess by per-cgroup control of promotion and demotion
(https://lore.kernel.org/[email protected])
- Promotion of unmapped page cache folios
(https://lore.kernel.org/[email protected])
Quoted 4 lines
I'm not sure how DAMON can help with this one. As I understand DAMON, it monitors accesses to user addresses. This patchset is trying to solve the problem for file pages which aren't mapped to userspace at all. ie only accessed through
read()andwrite().
DAMON can monitor physical addresses to, though the mechanism is different. I haven't assessed this as a solution, yet.
~Gregory
SeongJae Park
maintainerQuoted 16 lines · On Thu, 2 Jan 2025 10:22:14 -0500 Gregory Price <[email protected]> wrote:
Quoted 15 lines · Matthew Wilcox
Quoted 13 lines · SeongJae Park
Hi all,
I find a few interesting and promising projects that aim to do efficient access pattern-aware memory management of near future, including below (alphabetically sorted).
- CXL hotness monitoring unit
(https://lore.kernel.org/[email protected])
- Memory tiering fainess by per-cgroup control of promotion and demotion
(https://lore.kernel.org/[email protected])
- Promotion of unmapped page cache folios
(https://lore.kernel.org/[email protected])
Quoted 7 lines
Quoted 4 lines
I'm not sure how DAMON can help with this one. As I understand DAMON, it monitors accesses to user addresses. This patchset is trying to solve the problem for file pages which aren't mapped to userspace at all. ie only accessed through
read()andwrite().DAMON can monitor physical addresses to, though the mechanism is different.
Thank you for answering this, Gregory. As Gregory explained, users can use physical address monitoring mode of DAMON for this. For unmapped pages, DAMON sets and reads PG_idle to check if it is accessed or not. Since PG_idle is respected by read() and write() use case to my understanding, DAMON should be able to check accesses to unmapped pages.
Quoted 1 line
I haven't assessed this as a solution, yet.
To quickly see this, I ran below simple test.
First, I start DAMON in physical address space monitoring mode, wait for one minutes to let it monitor the accesses of the system, and show the access pattern on the system in access temperature histogram format.
$ sudo ./damo start $ sleep 60 $ sudo ./damo report access --style temperature-sz-hist <temperature> <total size> [-7,480,000,000, -7,479,999,999) 59.868 GiB | | total size: 59.868 GiB
The access temperature histogram format shows size of memory of given access temperature range. Access temperature is a metric that represents the access hotness. If any access to the region is continuously found, the value increases. If no access to the region is found, the temperature becomes zero. If it continues showing no access, the temperature further decreases (goes to minus). Refer to the document[1] for more details.
So from the above output, we can show all memory of the system is not accessed at all for the last minute.
Now I start a program that continuously overwrites 10 GiB file in background. Attaching the source code (Attachment 0, dd_like.c) at the bottom of this mail.
After a few seconds, I show the temperature histogram again.
$ sudo ./damo report access --style temperature-sz-hist <temperature> <total size> [-12,590,000,000, -11,699,000,000) 42.038 GiB |********************| [-11,699,000,000, -10,808,000,000) 0 B | | [-10,808,000,000, -9,917,000,000) 0 B | | [-9,917,000,000, -9,026,000,000) 0 B | | [-9,026,000,000, -8,135,000,000) 5.986 GiB |*** | [-8,135,000,000, -7,244,000,000) 0 B | | [-7,244,000,000, -6,353,000,000) 0 B | | [-6,353,000,000, -5,462,000,000) 0 B | | [-5,462,000,000, -4,571,000,000) 0 B | | [-4,571,000,000, -3,680,000,000) 5.951 GiB |*** | [-3,680,000,000, -2,789,000,000) 5.893 GiB |*** | total size: 59.868 GiB
We can show DAMON found about 10 GiB relatively hot regions.
This is a very simple test that not well tuned. Maybe because of that, there are details to investigate, including why the 10 GiB regions are having different and negative access temperature. I'll skip those for now, since the point of this test is that DAMON at least somehow react to accesses for unmapped pages.
[1] https://github.com/damonitor/damo/blob/next/USAGE.md#access-temperature
Thanks,
SJ
Quoted 2 lines
~Gregory
==== Attachment 0 (dd_like.c) ====
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
#include <errno.h>
int main(int argc, char *argv[]) {
int block_size, count;
char *dest_path;
if (argc != 4) {
printf("Usage: ./dd_simulator <block_size> <count> <destination_file>\n");
return 1;
}
block_size = atoi(argv[1]);
count = atoi(argv[2]);
dest_path = argv[3];
// Validate input parameters
if (block_size <= 0 || count <= 0) {
fprintf(stderr, "Invalid block size or count\n");
return 1;
}
// Write block size of zeroes 'count' times
char *zeroes = calloc(block_size, sizeof(char));
if (!zeroes) {
fprintf(stderr, "Memory allocation failed\n");
return 1;
}
while (1) {
// Open destination file in write mode
FILE *dest_file = fopen(dest_path, "w");
if (!dest_file) {
fprintf(stderr, "Failed to open %s for writing: %s\n", dest_path, strerror(errno));
return 1;
}
// Write block size of zeroes 'count' times
for (int i = 0; i < count; i++)
fwrite(zeroes, block_size, 1, dest_file);
fclose(dest_file);
printf("one pass");
}
free(zeroes);
return 0;
}SeongJae Park
maintainerQuoted 35 lines · On Thu, 2 Jan 2025 10:00:19 -0800 SeongJae Park <[email protected]> wrote:
Quoted 16 lines · On Thu, 2 Jan 2025 10:22:14 -0500 Gregory Price <[email protected]> wrote:
Quoted 15 lines · Matthew Wilcox
Quoted 13 lines · SeongJae Park
Hi all,
I find a few interesting and promising projects that aim to do efficient access pattern-aware memory management of near future, including below (alphabetically sorted).
- CXL hotness monitoring unit
(https://lore.kernel.org/[email protected])
- Memory tiering fainess by per-cgroup control of promotion and demotion
(https://lore.kernel.org/[email protected])
- Promotion of unmapped page cache folios
(https://lore.kernel.org/[email protected])Quoted 7 lines
Quoted 4 lines
I'm not sure how DAMON can help with this one. As I understand DAMON, it monitors accesses to user addresses. This patchset is trying to solve the problem for file pages which aren't mapped to userspace at all. ie only accessed through
read()andwrite().DAMON can monitor physical addresses to, though the mechanism is different.
Thank you for answering this, Gregory. As Gregory explained, users can use physical address monitoring mode of DAMON for this. For unmapped pages, DAMON sets and reads
PG_idleto check if it is accessed or not. SincePG_idleis respected byread()andwrite()use case to my understanding, DAMON should be able to check accesses to unmapped pages.Quoted 1 line
I haven't assessed this as a solution, yet.
To quickly see this, I ran below simple test.
[...]
Quoted 2 lines
the point of this test is that DAMON at least somehow react to accesses for unmapped pages.
Forgot clarifying this point, sorry. My test shows DAMON can detect accesses to unmapped pages, but not asseses if it is feasible as the unmapped pages promotion solution. More works and discussions would be needed for that.
Thanks,
SJ
[...]
Quoted 9 lines · SeongJae Park
Hi all,
I find a few interesting and promising projects that aim to do efficient access pattern-aware memory management of near future, including below (alphabetically sorted).
- Promotion of unmapped page cache folios
(https://lore.kernel.org/[email protected])
I'll break down a few observations I made while hacking on unmapped page cache promotion - and my concerns for a leveraging DAMON here.
Additionally some other concerns I've seen raised about duplicating promotion logic across various kernel components.
Latest RFC:
https://lore.kernel.org/linux-mm/[email protected]/
Basic Premise:
Use folio_mark_accessed() as a measure of hotness for promotion. Defer promotion to task_work due to locking complexities.
My major concerns / lessons learned from this exercise include:
1) The cost of checking promotion candidacy can be problematic
In my microbenchmark in the last RFC version, I showed that while the performance upside (~22-25%) is substantial, there was a non-trivial cost associated with injecting even a single global boolean check in the file_read() path. This was unexpected.
I can probably optimize the disabled case with a likely() clause, but I did not expect such sensitivity. This tells me injecting an unconditional call into DAMON may be too much overhead.
I would need to explore this further - including whether it is feasible to inject such a large dependency into swap.c
This may not affect all cases, but it does affect at least this one.
2) The complexity of "when it is safe" to promote a folio is subtle at best, and "actively hostile" at worst.
I learned in v1 of the RFC that promotion inline with fma() is not feasible due to a few contexts (task dying in particular) in which migration is not safe. I deferred to task work because I noticed prior attempts (in development notes) had seen similar issues.
Adding a folio reference and/or page flag to defer that migration to another context (i.g. async kthread) solves this at the expensive of implementation complexity. (leaked folios if done wrong)
I'd have to look at whether it's worth the increased complexity to aggregate this (particular) identification mechanism - but I think there is clear value to aggregating promotion.
I could see some value in pumping tracking bits into DAMON - but I also see value is making tasks handle promotion as a form of fairness.
3) There were expressed opinions on runtime fairness WRT to promotion.
There's two competing thoughts:
A) Making accessing tasks eat inline promotion cost captures that cost in their runtime slice, promoting fairness in scheduling.
B) Aggregating promotion to an external thread can reduce inline faults and tail latencies, but may hides per-task cost. This is a concern if one task drives all the promotions, effectingly stealing an entire core by nature of the async design.
I don't have a good answer to this, just an observation that charging promotion time to the identifying task was a concern that was raised.
4) TPP and Unmapped Page Promotion may affect each other.
There is a rate-limiting mechanism in the migration path that was intended to prevent over-pressuring bandwidth with aggressive migrations - prevent major memory stalls.
By adding more pressure on this limit from an additional source, we're obviously increasing the time it takes to converge.
This is probably the greatest argument for creating a new, aggregated promotion mechanism to serve all of these identification mechanism.
This would make it easier for us to determine whether/what identification mechanisms can be aggregated while enabling forward progress on each of them separately.
5) Scarce resources
We need to be careful not to consume excessive amounts of resources in an attempt to track all these identifying mechanisms. Even 1 byte per folio is 256MB on a 1TB machine. This gets out of hand quick.
With task-work, I was able to add no additional resource consumption, but deferring to a fully async scenario and needing to track things like last-accessing CPU, timestamps, and etc.
We'll need to examine this closely if we decide to aggregate either of these mechanisms.
~Gregory
SeongJae Park
maintainerHello Gregory,
Quoted 8 lines · On Mon, 13 Jan 2025 22:06:09 -0500 Gregory Price <[email protected]> wrote:
Quoted 7 lines · SeongJae Park
Hi all,
I find a few interesting and promising projects that aim to do efficient access pattern-aware memory management of near future, including below (alphabetically sorted).
Quoted 6 lines
Quoted 2 lines
- Promotion of unmapped page cache folios
(https://lore.kernel.org/[email protected])I'll break down a few observations I made while hacking on unmapped page cache promotion - and my concerns for a leveraging DAMON here.
Thank you for sharing this!
Quoted 16 lines
Additionally some other concerns I've seen raised about duplicating promotion logic across various kernel components.
Latest RFC:
https://lore.kernel.org/linux-mm/[email protected]/Basic Premise:
Usefolio_mark_accessed()as a measure of hotness for promotion. Defer promotion totask_workdue to locking complexities.My major concerns / lessons learned from this exercise include:
1) The cost of checking promotion candidacy can be problematic
Quoted 8 lines
In my microbenchmark in the last RFC version, I showed that while the performance upside (~22-25%) is substantial, there was a non-trivial cost associated with injecting even a single global boolean check in the
file_read()path. This was unexpected.I can probably optimize the disabled case with a
likely()clause, but I did not expect such sensitivity. This tells me injecting an unconditional call into DAMON may be too much overhead.
I cannot agree more with you about the point that the mechanism for finding the promotion/demotion (and any access-aware system operation) candidates should induce only modest or at least controllable overhead. Actually it was the one of biggest motivations of DAMON design, and I haven't imagined adding unconditional calls to DAMON here.
Nonetheless, injecting an unconditional call here should be avoided for not only DAMON calls but any expensive calls? I'm also not pretty sure what DAMON call you are thinking about.
Quoted 3 lines
I would need to explore this further - including whether it is feasible to inject such a large dependency into
swap.c
I understand DAMON is not small in terms of the code size, and has many limitations that makes it unusable in many use cases. But, again, I'm not pretty sure what kind of DAMON usage in swap.c you're thinking about, and therefore not easy to understyand what part of DAMON is considered as a large dependency that concerns you. It would be great if we can make more concrete example as a result of this topic session at LSFMMBPF.
FYI, I also not having specific idea for helping unmapped pages promotion for now. That's my assignment that I will do by LSFMMBPF. But, a few things that I naively thinking DAMON might be able to help unmapped promotions are,
1. Using DAMON for profiling how much hot and cold unmapped pages are in which tier, and use the information for unmapped pages promotion optimization.
2. Using DAMOS to target-promote hot unmapped pages while using page faults-based promotion for mapped pages.
3. Using DAMOS to promote both mapped and unmapped hot pages.
For the first and second ideas, DAMON need to target unmapped pages. I think DAMOS filters can be extended for that, and I posted an RFC before:
https://lore.kernel.org/[email protected]
Using the RFC-applied kernel and a version of DAMON user-space tool that adds the support, idea one could be done like below.
$ sudo ./damo report access --snapshot_damos_filter reject none unmapped --style recency-sz-hist
# damos filters (df): reject none unmapped
<last accessed time (us)> <df-passed size>
[-36.300 s, -32.670 s) 10.297 MiB |* |
[-32.670 s, -29.040 s) 7.297 MiB |* |
[-29.040 s, -25.410 s) 0 B | |
[-25.410 s, -21.780 s) 0 B | |
[-21.780 s, -18.150 s) 0 B | |
[-18.150 s, -14.520 s) 0 B | |
[-14.520 s, -10.890 s) 0 B | |
[-10.890 s, -7.260 s) 0 B | |
[-7.260 s, -3.630 s) 3.088 GiB |********************|
[-3.630 s, -0 ns) 80.000 KiB |* |
[-0 ns, --3630000000 ns) 16.000 KiB |* | <last accessed time (us)> <total size>
[-36.300 s, -32.670 s) 24.493 GiB |********************|
[-32.670 s, -29.040 s) 5.869 GiB |***** |
[-29.040 s, -25.410 s) 5.568 GiB |***** |
[-25.410 s, -21.780 s) 0 B | |
[-21.780 s, -18.150 s) 5.899 GiB |***** |
[-18.150 s, -14.520 s) 5.807 GiB |***** |
[-14.520 s, -10.890 s) 0 B | |
[-10.890 s, -7.260 s) 0 B | |
[-7.260 s, -3.630 s) 12.231 GiB |********** |
[-3.630 s, -0 ns) 356.000 KiB |* |
[-0 ns, --3630000000 ns) 396.000 KiB |* |
total size: 59.868 GiBThe above output was retrieved while a kernel build is running in background, and says among 24.493 GiB cold memory that last accessed more than 32.67 seconds before, 10.297 MiB are unmapped pages.
For the third idea, whether and how to collaborate with page faults-based promotion of mapped pages could be something to discuss. Some ideas off the my head is that we can simply make them exclusive, or use DAMOS for proactive promotion under peaceful situation, but uses page faults based promotion for more urgent situation, somewhat like kswapd and direct reclaims.
For all three ideas, DAMON will do the monitoring and promotions on DAMON thread, so no change to swap.c or file io path would be required.
Again, these are just not-yet-settled brainstorming level ideas, and I will try to make these more specific and settled by LSFMMBPF. Please feel free to add comments on this thread rather than waiting for LSFMMBPF, though!
Quoted 15 lines
This may not affect all cases, but it does affect at least this one.
2) The complexity of "when it is safe" to promote a folio is subtle at best, and "actively hostile" at worst.
I learned in v1 of the RFC that promotion inline with
fma()is not feasible due to a few contexts (task dying in particular) in which migration is not safe. I deferred to task work because I noticed prior attempts (in development notes) had seen similar issues.Adding a folio reference and/or page flag to defer that migration to another context (i.g. async kthread) solves this at the expensive of implementation complexity. (leaked folios if done wrong)
Quoted 5 lines
I'd have to look at whether it's worth the increased complexity to aggregate this (particular) identification mechanism - but I think there is clear value to aggregating promotion.
I could see some value in pumping tracking bits into DAMON -
I agree to all the points and willing to make DAMON well serve the purpose.
Quoted 2 lines
but I
also see value is making tasks handle promotion as a form of fairness.
I agree that could be good in terms of fairness. I want to learn more about the significance of it, though.
Quoted 7 lines
3) There were expressed opinions on runtime fairness WRT to promotion.
There's two competing thoughts:
A) Making accessing tasks eat inline promotion cost captures that cost in their runtime slice, promoting fairness in scheduling.
Quoted 7 lines
B) Aggregating promotion to an external thread can reduce inline faults and tail latencies, but may hides per-task cost. This is a concern if one task drives all the promotions, effectingly stealing an entire core by nature of the async design.
I don't have a good answer to this, just an observation that charging promotion time to the identifying task was a concern that was raised.
I think we might be able to pursue two ways in parallel? Using asynchronous external thread in more peaceful situation, and let tasks do inline promotion with fairness under more urgent situation, like kswapd and direct reclaims.
DAMON may fit well for the proactive solutions under less urgent situation. DAMON_RECLAIM was made in the direction, and working without significant issues on products for years.
Quoted 11 lines
4) TPP and Unmapped Page Promotion may affect each other.
There is a rate-limiting mechanism in the migration path that was intended to prevent over-pressuring bandwidth with aggressive migrations - prevent major memory stalls.
By adding more pressure on this limit from an additional source, we're obviously increasing the time it takes to converge.
Quoted 6 lines
This is probably the greatest argument for creating a new, aggregated promotion mechanism to serve all of these identification mechanism.
This would make it easier for us to determine whether/what identification mechanisms can be aggregated while enabling forward progress on each of them separately.
I agree. DAMON allows combining multiple different mechanisms with its core logic, so I beleive it migt be a place that can aggregate the different identification mechanisms.
DAMON's access monitoring results based system operations feature, namely DAMOS, also has its own aggressiveness control logic, and resides in the core layer, so could be used consistently with different promotion candidates identification mechanisms.
Quoted 7 lines
5) Scarce resources
We need to be careful not to consume excessive amounts of resources in an attempt to track all these identifying mechanisms. Even 1 byte per folio is 256MB on a 1TB machine. This gets out of hand quick.
Quoted 6 lines
With task-work, I was able to add no additional resource consumption, but deferring to a fully async scenario and needing to track things like last-accessing CPU, timestamps, and etc.
We'll need to examine this closely if we decide to aggregate either of these mechanisms.
Agreed again. In case of DAMON, it tries to keep the resources in its own data structure. The resource consumption with the own data structure can also be problematic, but it at least allows setting the upper-bound, regardless of the system size. So it is controllable and scalable.
I wish to continue more detailed discussions on LSFMMBPF and this thread!
Thank you again sharing your experiences and thoughts on this topic. I show those are making the discussion much more informative and helpful.
Thanks,
SJ
Quoted 2 lines
~Gregory
Quoted 19 lines · SeongJae Park
Hello Gregory,
Quoted 10 lines
1) The cost of checking promotion candidacy can be problematic
In my microbenchmark in the last RFC version, I showed that while the performance upside (~22-25%) is substantial, there was a non-trivial cost associated with injecting even a single global boolean check in the
file_read()path. This was unexpected.I can probably optimize the disabled case with a
likely()clause, but I did not expect such sensitivity. This tells me injecting an unconditional call into DAMON may be too much overhead.I cannot agree more with you about the point that the mechanism for finding the promotion/demotion (and any access-aware system operation) candidates should induce only modest or at least controllable overhead. Actually it was the one of biggest motivations of DAMON design, and I haven't imagined adding unconditional calls to DAMON here.
Quoted 4 lines
Nonetheless, injecting an unconditional call here should be avoided for not only DAMON calls but any expensive calls? I'm also not pretty sure what DAMON call you are thinking about.
Just any call, DAMON or otherwise. The explicit check injecting ~2-3% overhead on my microbench was a simple
+ } else if (!folio_test_isolated(folio) &&
+ (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
If this is causing additional overhead, call me skeptical that trying anything more complicated will turn out better.
Quoted 11 lines
Quoted 3 lines
I would need to explore this further - including whether it is feasible to inject such a large dependency into
swap.cI understand DAMON is not small in terms of the code size, and has many limitations that makes it unusable in many use cases. But, again, I'm not pretty sure what kind of DAMON usage in
swap.cyou're thinking about, and therefore not easy to understyand what part of DAMON is considered as a large dependency that concerns you. It would be great if we can make more concrete example as a result of this topic session at LSFMMBPF.
It's not a matter of code size - it's a matter of tightly coupling core components of the kernel to extraneous ones. Adding additional dependencies between components increases overall system complexity and makes it hard to reason about the behavior of the system.
For example, in the prior snippet:
+ } else if (!folio_test_isolated(folio) &&
+ (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) &&
+ numa_pagecache_promotion_enabled) {
+ promotion_candidate(folio);
This amounts to
if (some condition && feature enabled)
mark a folio as a candidate for promotion
The promotion_candidate() function is contained within migrate.c and uses (mostly) migrate.c mechanisms (from a task_work).
All you need to understand the behavior is between swap.c and migrate.c.
If instead you aggregate this to DAMON, understanding the behavior of swap can require you to understand what DAMON is actually doing with this information.
Now you need to understand swap.c, migrate.c, AND DAMON.
It makes it more difficult to reason about the system when something goes wrong. This increases the maintenance burden for maintainers (and onboarding complexity for anyone new to the kernel, for that matter).
That doesn't mean we shouldn't consider doing this - it just means that benefit needs to outweight the complexity/maintenance cost.
Quoted 10 lines
FYI, I also not having specific idea for helping unmapped pages promotion for now. That's my assignment that I will do by LSFMMBPF. But, a few things that I naively thinking DAMON might be able to help unmapped promotions are,
1. Using DAMON for profiling how much hot and cold unmapped pages are in which tier, and use the information for unmapped pages promotion optimization.
2. Using DAMOS to target-promote hot unmapped pages while using page faults-based promotion for mapped pages.
3. Using DAMOS to promote both mapped and unmapped hot pages.
This missing the scenario where DAMOS/DAMON is not suitible for deployment in someone's environment. The kernel should still do *something*.
And that is kind of the point - we can expose more complexity to the users with DAMON, but the kernel should be able to do some reasonable promotion action without this additional system.
Quoted 6 lines
Quoted 2 lines
but I
also see value is making tasks handle promotion as a form of fairness.I agree that could be good in terms of fairness. I want to learn more about the significance of it, though.
Fairness in this scenario is simple.
If one task is causing an outsizes number of promotions to occur, and it causes some ASYNC system to handle those promotions, it is effectively acquiring more CPU time via that ASYNC system than other residents.
Trying to charge this time back to the noisey task is harder than just having the task incur the cost of migration. But doing it inline can cause the task to slow down.
So it's difficult to predict how it's going to pan out. Need evidence.
Quoted 9 lines
I agree. DAMON allows combining multiple different mechanisms with its core logic, so I beleive it migt be a place that can aggregate the different identification mechanisms.
DAMON's access monitoring results based system operations feature, namely DAMOS, also has its own aggressiveness control logic, and resides in the core layer, so could be used consistently with different promotion candidates identification mechanisms.
Without data this is a nice thought, but we have existing mechanisms that work and can be improved - lets not disrupt that.
Finding an aggregated promotion solution helps everyone move forward without disrupting development in these areas (and makes the different indentification mechanisms play nice with each other).
Trying to also create a voltron "one indentification system to rule them all" is a nice thought, but it's heavy-weight compared to adding a folio flag check and a call to mpol_migrate_misplaced(). We need to respect that reality and not regress the existing mechanisms by trying to over-engineer a generalized solution.
~Gregory
SeongJae Park
maintainerQuoted 1 line · On Fri, 24 Jan 2025 12:21:31 -0500 Gregory Price <[email protected]> wrote:
On Thu, Jan 23, 2025 at 06:11:53PM -0800, SeongJae Park wrote:
[...]
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This tells me injecting
an unconditional call into DAMON may be too much overhead.
[...]
Quoted 5 lines
Quoted 3 lines
I'm also not pretty sure what DAMON
call you are thinking about.Just any call, DAMON or otherwise.
Thanks for clarifying.
[...]
Quoted 4 lines
It's not a matter of code size - it's a matter of tightly coupling core components of the kernel to extraneous ones. Adding additional dependencies between components increases overall system complexity and makes it hard to reason about the behavior of the system.
[...]
Quoted 5 lines
Now you need to understand
swap.c,migrate.c, AND DAMON.It makes it more difficult to reason about the system when something goes wrong. This increases the maintenance burden for maintainers (and onboarding complexity for anyone new to the kernel, for that matter).
Thank you for this kind clarification. This is very helpful at better understanding your point. I cannot agree more on your point that tightly coupling multiple components makes things compelx. Let me emphasize your points from other side, too. This doesn't mean we should avoid using multiple components together. If the interface is well designed and being used correctly, using multiple components together rather reduce the complexity and maintenance burden.
In the example, swap.c maintainer should easily know something in migrate.c that being used by swap.c is not working as documented or expected, and ask migrate.c maintainer to fix it. I'm trying to make DAMON be designed and used in such a way. I'm proposing this LSFMMBPF to help that by discussing in depth, including specific examples of current or potential DAMON usages and DAMON interfaces that not well designed for those.
Quoted 3 lines
That doesn't mean we shouldn't consider doing this - it just means that benefit needs to outweight the complexity/maintenance cost.
I agree this too, of course :)
[...]
Quoted 2 lines
This missing the scenario where DAMOS/DAMON is not suitible for deployment in someone's environment.
I understand that you are saying a scenario that deploying out-of-kernel components such as DAMON user-space tool is impossible, while those are essential for a given usage. And I agree that such case can be in real.
Quoted 6 lines
The kernel should still do
*something*.And that is kind of the point - we can expose more complexity to the users with DAMON, but the kernel should be able to do some reasonable promotion action without this additional system.
I understand that you mean using DAMON for promotion requires users controls using additional systems such as DAMON user-space tool (damo). That's correct, at least for today's DAMON usages for CXL memory tiering. HMSDK[1] is such an additional system.
Nevertheless, that's not necessarily the case in future. DAMON aims to allow flexible custom usages, while also just transparently works fairly well. I shared the humble ambition at last year's LPC[2]. We will pursue the direction for memory tiering-purpose DAMON usage, too.
[...]
Quoted 13 lines
Quoted 6 lines
Quoted 2 lines
but I
also see value is making tasks handle promotion as a form of fairness.I agree that could be good in terms of fairness. I want to learn more about the significance of it, though.
Fairness in this scenario is simple.
If one task is causing an outsizes number of promotions to occur, and it causes some ASYNC system to handle those promotions, it is effectively acquiring more CPU time via that ASYNC system than other residents.
Quoted 5 lines
Trying to charge this time back to the noisey task is harder than just having the task incur the cost of migration. But doing it inline can cause the task to slow down.
So it's difficult to predict how it's going to pan out. Need evidence.
Yes, I agree that we need more data to say more about this topic. Nonetheless, I understand you are saying that's something better to have in future, and need to aware of its potential risk, not a strict blocker of async approach exploration.
Quoted 5 lines
Quoted 5 lines
I agree. DAMON allows combining multiple different mechanisms with its core logic, so I beleive it migt be a place that can aggregate the different identification mechanisms.
Quoted 8 lines
Quoted 5 lines
DAMON's access monitoring results based system operations feature, namely DAMOS, also has its own aggressiveness control logic, and resides in the core layer, so could be used consistently with different promotion candidates identification mechanisms.
Without data this is a nice thought, but we have existing mechanisms that work and can be improved - lets not disrupt that.
Cannot agree more. My intention is not to disrubpt that but ensuring people who looking into such improvments are on the same page regarding available current and future options.
Quoted 10 lines
Finding an aggregated promotion solution helps everyone move forward without disrupting development in these areas (and makes the different indentification mechanisms play nice with each other).
Trying to also create a voltron "one indentification system to rule them all" is a nice thought, but it's heavy-weight compared to adding a folio flag check and a call to
mpol_migrate_misplaced(). We need to respect that reality and not regress the existing mechanisms by trying to over-engineer a generalized solution.
100% agreed. This point is, and should, always be in DAMON hackers' mind.
Thank you for kindly clarifying your points and nice advice :)
[1] https://github.com/skhynix/hmsdk/wiki/Capacity-Expansion
[2] https://lpc.events/event/18/contributions/1768/
Thanks,
SJ
Quoted 6 lines · Gregory Price
5) Scarce resources
We need to be careful not to consume excessive amounts of resources in an attempt to track all these identifying mechanisms. Even 1 byte per folio is 256MB on a 1TB machine. This gets out of hand quick.
Quoted 6 lines
With task-work, I was able to add no additional resource consumption, but deferring to a fully async scenario and needing to track things like last-accessing CPU, timestamps, and etc.
We'll need to examine this closely if we decide to aggregate either of these mechanisms.
My concern with physical address space monitoring is fragmentation. I ran some numbers on a few prod machines. Grouping by regions with the same memcg and ignoring any unmapped memory to be generous, machines with higher utilization can have a region/total pages ratio of ~40%, and even those with lower utilization (<50%) can also reach 20%. Accurately tracking these regions would require quite the region metadata, on the order of GBs.
SeongJae Park
maintainerHi Yuanchu,
Quoted 13 lines · On Wed, 29 Jan 2025 18:15:08 -0800 Yuanchu Xie <[email protected]> wrote:
Quoted 6 lines · Gregory Price
5) Scarce resources
We need to be careful not to consume excessive amounts of resources in an attempt to track all these identifying mechanisms. Even 1 byte per folio is 256MB on a 1TB machine. This gets out of hand quick.
Quoted 6 lines
With task-work, I was able to add no additional resource consumption, but deferring to a fully async scenario and needing to track things like last-accessing CPU, timestamps, and etc.
We'll need to examine this closely if we decide to aggregate either of these mechanisms.
Quoted 7 lines
My concern with physical address space monitoring is fragmentation. I ran some numbers on a few prod machines. Grouping by regions with the same memcg and ignoring any unmapped memory to be generous, machines with higher utilization can have a region/total pages ratio of ~40%, and even those with lower utilization (<50%) can also reach 20%. Accurately tracking these regions would require quite the region metadata, on the order of GBs.
You're right, if we need page level accuracy access monitoring and want to use DAMON with its regions based mechanism for that, the memory overhead of damon_region could be high. That's mainly because DAMON's regions-based mechanism has not designed for such usage. It is more for a best-effort tradeoff between the overhead and the accuracy.
Regions-based mechanism is not necessarily the only mechanism of future DAMON, though. If there are use cases that regions-based best-effort accuracy cannot be used while exactly the page level accuracy is really required, we can think about optimizing regions based mechanism or developing new one.
But, IMHO, the page level accurate access pattern is not always essential. In many cases, being able to distinguish some amount of regions agains others based on access pattern is practical enough. Indeed, DAMON has been used on real-world products with physical address based moitoring mode for years with no significant problem. Also I think physical address space based monitoring results[1] on a real server workload that I shared recently seems not very bad.
Of course your use case could be different from what I have experienced so far. I'm curious if and why you really need page level accuracy.
[1] https://lore.kernel.org/[email protected]
Thanks,
SJ
Quoted 24 lines · On Wed, 29 Jan 2025 19:47:49 -0800 SeongJae Park <[email protected]> wrote:
Hi Yuanchu,
Quoted 20 lines · On Wed, 29 Jan 2025 18:15:08 -0800 Yuanchu Xie <[email protected]> wrote:
Quoted 6 lines · Gregory Price
5) Scarce resources
We need to be careful not to consume excessive amounts of resources in an attempt to track all these identifying mechanisms. Even 1 byte per folio is 256MB on a 1TB machine. This gets out of hand quick.
Quoted 6 lines
With task-work, I was able to add no additional resource consumption, but deferring to a fully async scenario and needing to track things like last-accessing CPU, timestamps, and etc.
We'll need to examine this closely if we decide to aggregate either of these mechanisms.
My concern with physical address space monitoring is fragmentation. I ran some numbers on a few prod machines. Grouping by regions with the same memcg and ignoring any unmapped memory to be generous, machines with higher utilization can have a region/total pages ratio of ~40%, and even those with lower utilization (<50%) can also reach 20%. Accurately tracking these regions would require quite the region metadata, on the order of GBs.
I'd second this. Some cases are reasonably well behaved and regions 'kind of work' for PA based tracking some very much not. Add anything like overcommitted VMs on top and contiguity of 'hotness' beyond very small regions goes out the window very quickly (unfortunately I'm not able to share specific data).
So there are definitely cases where I'd expect something else to be needed.
There are a plenty of approximate tracking methods in the literature that might be good enough with much lower overhead than precise tracking (sketches etc) if we can feed them the right data.
Typically we don't need the answer on how hot all memory is, just some info on 'this lot are particularly hot' and 'this lot are reasonably' cold.
Damon (as it currently stands) can sometimes give this info so to me it's a possible producer of data for another layer that focuses on abstracting the data to what we want only. Hopefully we can make that work for all the forms of tracking temperature that people are looking at. I'm biased in favor of hardware units but no everyone will have those toys available for a while yet :)
Quoted 27 lines
You're right, if we need page level accuracy access monitoring and want to use DAMON with its regions based mechanism for that, the memory overhead of
damon_regioncould be high. That's mainly because DAMON's regions-based mechanism has not designed for such usage. It is more for a best-effort tradeoff between the overhead and the accuracy.Regions-based mechanism is not necessarily the only mechanism of future DAMON, though. If there are use cases that regions-based best-effort accuracy cannot be used while exactly the page level accuracy is really required, we can think about optimizing regions based mechanism or developing new one.
But, IMHO, the page level accurate access pattern is not always essential. In many cases, being able to distinguish some amount of regions agains others based on access pattern is practical enough. Indeed, DAMON has been used on real-world products with physical address based moitoring mode for years with no significant problem. Also I think physical address space based monitoring results[1] on a real server workload that I shared recently seems not very bad.
Of course your use case could be different from what I have experienced so far. I'm curious if and why you really need page level accuracy.
[1] https://lore.kernel.org/[email protected]
Thanks,
SJ
Quoted 7 lines · On Wed, 1 Jan 2025 14:20:39 -0800 SeongJae Park <[email protected]> wrote:
Hi all,
I find a few interesting and promising projects that aim to do efficient access pattern-aware memory management of near future, including below (alphabetically sorted).
Hi SJ,
Quoted 2 lines
- CXL hotness monitoring unit
(https://lore.kernel.org/[email protected])
For hardware hotness monitors the type of data has relatively little connection to what I understand Damon provides and the control schemes are somewhat different. Hotness tracking units should provide a simple list of hot fixed size granuals (hot 'pages') to whatever is using the hotness engine. Damon and other in kernel schemes might also be able to provide such outputs, but the underlying schemes seem very different as the outputs of these trackers neither map to Damon regions, or to dense sets of page counters.
So to me the commonality looks to be one layer up: We get lists of stuff to consider moving and control paths to whatever is providing those lists to indicate:
* More or fewer suggestions please (bandwidth controls etc)
* Minimum 'hotness' below which it should not suggest moving them.
For CXL Hotness monitoring units, there are open questions about how to get good data given a limited resources likely to be found on devices. Simplest sense can be thought of as a fixed set of counters, but typically it will be more complex than that with statistical accuracy tradeoffs rather than did we count it or not.
We need to do some work to find out what works best across many workloads considering options (depending on hardware capabilities) such as
a) coarse to fine
b) random subsampling of 256MiB chunks of PA space.
c) scanning across PA space looking at a smallish region (16Gig maybe) at a time.
Also need to be flexible to use multiple parallel trackers if available on a given device or time slices on a single tracker.
I'm not yet seeing enough different engines to figure out if there is commonality in that control scheme between CXL style interfaces and those that we may see from other places etc. If anyone is in a position to share info on other hotness monitoring offloaded units that are targeting real products + their interfaces that would be great. For now I think we are going to end up with something specific in the CXL HMU driver with the rest of the kernel just seeing a list of 'hot PA address chunks / pages in PA space'.
Given we will need a virtualized solution as well for guests that are running on a fixed mix of tiers, I'd expect a "virtio-hotness" or similar that only provides these sorts of generalized controls leaving the host to figure out how to control the particular hotness trackers. The controls to that would be inline with what I'd expect to be exposed to other layers of the kernel from a given hotness tracker.
For me it feels like we are a bit early wrt to hardware trackers to come to firm conclusions, but perhaps others are further ahead with answering some of the precursor questions. I am keen that we don't end up with a solution that doesn't work with them so this discussion if of interest to me.
Quoted 21 lines
- Memory tiering fainess by per-cgroup control of promotion and demotion
(https://lore.kernel.org/[email protected])
- Promotion of unmapped page cache folios
(https://lore.kernel.org/[email protected])
- Slow-tier page promotion based on PTE A bit
(https://lore.kernel.org/[email protected])
- Workingset reporting
(https://lore.kernel.org/[email protected])The goal of DAMON is to help accelerating such developments by being a framework that can reduce fundamental efforts for monitoring memory access patterns and managing memory using the information. AWS Aurora Serverless v2 and SK hynix are successfully using DAMON in the way for proactive memory reclamation[1] and CXL memory tiering[2].
To further deliver such benefits for the ongoing and future projects, we need to better understand what the projects really need, how DAMON can provide those now or in future, and if there are alternatives better than DAMON. Regardless of the conclusion about DAMON, the works apparently have common parts, so the discussion will benefit all.
Quoted 7 lines
I propose to have the discussion at LSF/MM/BPF. In the session, I will briefly introduce the works and possible DAMON usages, and continue the open discussion for better understanding each other. The discussion will not be limited to DAMON and abovely mentioned projects but possible alternatives and general access-aware memory management projects. After the discussion, we will hopefully find ways to efficiently collaborate, or at least do not disturb each other.
I like that last comment :)
Jonathan
SeongJae Park
maintainerHello,
Quoted 38 lines · On Wed, 1 Jan 2025 14:20:39 -0800 SeongJae Park <[email protected]> wrote:
Hi all,
I find a few interesting and promising projects that aim to do efficient access pattern-aware memory management of near future, including below (alphabetically sorted).
- CXL hotness monitoring unit
(https://lore.kernel.org/[email protected])
- Memory tiering fainess by per-cgroup control of promotion and demotion
(https://lore.kernel.org/[email protected])
- Promotion of unmapped page cache folios
(https://lore.kernel.org/[email protected])
- Slow-tier page promotion based on PTE A bit
(https://lore.kernel.org/[email protected])
- Workingset reporting
(https://lore.kernel.org/[email protected])The goal of DAMON is to help accelerating such developments by being a framework that can reduce fundamental efforts for monitoring memory access patterns and managing memory using the information. AWS Aurora Serverless v2 and SK hynix are successfully using DAMON in the way for proactive memory reclamation[1] and CXL memory tiering[2].
To further deliver such benefits for the ongoing and future projects, we need to better understand what the projects really need, how DAMON can provide those now or in future, and if there are alternatives better than DAMON. Regardless of the conclusion about DAMON, the works apparently have common parts, so the discussion will benefit all.
I propose to have the discussion at LSF/MM/BPF. In the session, I will briefly introduce the works and possible DAMON usages, and continue the open discussion for better understanding each other. The discussion will not be limited to DAMON and abovely mentioned projects but possible alternatives and general access-aware memory management projects. After the discussion, we will hopefully find ways to efficiently collaborate, or at least do not disturb each other.
Quoted 2 lines
[1] https://assets.amazon.science/ee/a4/41ff11374f2f865e5e24de11bd17/resource-management-in-aurora-serverless.pdf
[2] https://github.com/skhynix/hmsdk/wiki/Capacity-Expansion
A draft of the slides for this session is now available at
https://github.com/damonitor/talks/blob/master/2025/lsfmmbpf/damon_requirements_lsfmmbpf_2025.pdf
I may make more last time changes to the slides, but the final version should also be available on the same URL.
Thanks,
SJ
[...]