Skip to contentSkip to conversation

mm/damon/sysfs-schemes: add use_nodes_of_tier on sysfs-schemes

PATCH Simon Wang (王传国) 3 participants 6 messages May 30, 2025 – Jun 9, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw
Quoted 38 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

This patch adds use_nodes_of_tier under

/sys/kernel/mm/damon/admin/kdamonds/<N>/contexts/<N>/schemes/<N>/

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines

The 'use_nodes_of_tier' can be used to select nodes within the same memory tier of target_nid for DAMOS actions such as

DAMOS_MIGRATE_{HOT,COLD}.

Could you please elaborate in what setup you think this option is useful, and measurement of the usefulness if you have?

Quoted 7 lines
Quoted 7 lines

I'm asking the above question because of below reasons. My anticiapted usage of DAMOS_MIGRATE_{HOT,COLD} is for not only memory tiering but generic NUMA node management. And my proposed usage of these for memory tiering is making per-node promotion/demotion for gradually promoting and demoting pages step by step between node. It could be slow but I anticipate such slow but continued promotion/demotion is more important for reliable performance on

production systems of large time scale.

Quoted 9 lines
Quoted 5 lines

And I believe the approach can be applied to general NUMA nodes management, once DAMON is extended for per-CPU access monitoring.

I'm not saying this change is not useful, but asking you to give me a chance to learn your changes, better.

I believe some users may want to ​​use only the target node's memory​​ and reserve other nodes in the same tier for specific applications. Therefore, I added a switch file use_nodes_of_tier.

Thank you for clarifying, Simon.

Because this is an ABI change that difficult to revert and therefore we may need to support for long term, I'd like to have more clear theory and/or data if possible. In my humble opinion, above clarification doesn't sound like a strong enough justification for ABI change.

Quoted 6 lines

More specifically, it would be better if you could answer below questions. Who would be such users, how common the use case would be, and what are the benefit of doing so? Is that only theory? Or, a real existing use case? Can you share measurement of the benefit from this change that measured from real workloads or benchmarks? Is there an alternative way to do this without ABI change?

Your concern is that adding the bool use_nodes_of_tier variable and introducing an additional parameter to multiple functions would cause ABI changes, correct?​​

​​I propose avoiding the creation of the 'use_nodes_of_tier' sysfs file. Instead, we can modify the __damon_pa_migrate_folio_list() function to change the allowed_mask from NODE_MASK_NONE to the full node mask of the entire tier where the target_nid resides. This approach would be similar to the implementation in commit 320080272892
('mm/demotion: demote pages according to allocation fallback order').

I'd like to confirm two modification points with you:
​​1.Regarding alloc_migrate_folio()​​:
Restoring the original nodemask and gfp_mask in this function is the correct approach, correct? ​​2.Regarding DAMON's migration logic​​:
The target scope should be expanded from a single specified node to the entire memory tier
(where the target node resides), correct?
​​Can we confirm these two points are agreed upon?​

Quoted 48 lines
Quoted 3 lines

I think it might be better to set the default value of use_nodes_of_tier to true (i.e., allow using fallback nodes). What do you think

In my humble opinion, we can consider setting it true by default, if we agree the benefit of the change is significant. With only currently given information, I cannot easily say if I think this can really be useful. As asked abovely, more clear thoery and/or real data would be helpful.

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
---
 include/linux/damon.h        |  9 ++++++++-
 include/linux/memory-tiers.h |  5 +++++
 mm/damon/core.c              |  6 ++++--
 mm/damon/lru_sort.c          |  3 ++-
 mm/damon/paddr.c             | 19 ++++++++++++-------
 mm/damon/reclaim.c           |  3 ++-
 mm/damon/sysfs-schemes.c     | 31
Quoted 8 lines
Quoted 8 lines

++++++++++++++++++++++++++++++-

Quoted 4 lines
 mm/memory-tiers.c            | 13 +++++++++++++
 samples/damon/mtier.c        |  3 ++-
 samples/damon/prcl.c         |  3 ++-
 10 files changed, 80 insertions(+), 15 deletions(-)

Can we please make this change more separated? Maybe we can split the change for memory-tiers.c, DAMON core layer, and DAMON sysfs

interface.

Quoted 3 lines
Quoted 1 line

That will make review much easier.

Yes,I'll split this patch to be 2 patches.

Thank you for accepting my suggestion. But I think it deserves 3 patches, each for

- memory-tiers.c,
- DAMON core layer, and
- and DAMON sysfs interface.

But, let's further discuss on the high level topic (if this change is really beneficial enough to make ABI change).

Thanks,
SJ

[...]

maintainer
#1 lore raw

Hi Simon,

Thank you for continuing this important discussion.

Before starting, though, seems your mail client is not setting 'In-Reply-To' field of your mails. For people who uses 'In-Reply-To' field based threads displaying tools, ths thread could be difficult to read the whole contents. Please consider using tools that set the field correctly if possible.

You could get more information about available mailing tools from
https://docs.kernel.org/process/email-clients.html

Btw, I use hkml
(https://docs.kernel.org/process/email-clients.html#hackermail-tui) ;)

On Fri, 30 May 2025 08:04:42 +0000 Simon Wang (王传国) <[email protected]> wrote:

[...]

Quoted 2 lines

Your concern is that adding the bool use_nodes_of_tier variable and introducing an additional parameter to multiple functions would cause ABI changes, correct?​​

You are correct.

Quoted 7 lines

​​I propose avoiding the creation of the 'use_nodes_of_tier' sysfs file. Instead, we can modify the __damon_pa_migrate_folio_list() function to change the allowed_mask from NODE_MASK_NONE to the full node mask of the entire tier where the target_nid resides. This approach would be similar to the implementation in commit 320080272892 ('mm/demotion: demote pages according to allocation fallback order').

Then, this causes a behavior change, which we should not allow if it can be considered a regression. In other words, we could do this if it is a clear improvement.

So, let's think about if your proposed change is an improvement. As the commit 320080272892 is nicely explaining, I think that it is an improved behavior for demotion. Actually it seems good behavior for promotion, too. But, the behavior we are discussing here is not for the demotion but general migration (specifically, DAMOS_MIGRATE_{HOT,COLD}).

In my opinion, DAMOS_MIGRATE_{HOT,COLD} behavior should be somewhat similar to that of move_pages() syscall, to make its behavior easy to expect. So I think having commit 320080272892's behavior improvement to DAMOS_MIGRATE_{HOT,COLD} is not a right thing to do.

And this asks me a question. Is current DAMOS_MIGRATE_{HOT,COLD} behavior similar to move_pages() syscall? Not really, since do_move_pages_to_node(), which is called from move_pages() syscall and calls migrate_pages() is setting mtc->nmask as NULL, while DAMOS_MIGRATE_{HOT,COLD} set it as NODE_MASK_NONE. Also, do_move_pages_to_node() uses alloc_migration_target() while DAMOS_MIGRATE_{HOT,COLD} uses alloc_migrate_folio().

I overlooked this different behavior while reviewing this code, sorry. And I don't think this difference is what we need to keep, unless there are good rasons that well documented. Thank you for let us find this, Simon.

So I suggest to set mtc->nmask as NULL, and use alloc_migration_target() from __damon_pa_migrate_folio_list(), same to move_pages() system call. To use alloc_migrate_folio() from __damon_pa_migrate_folio_list(), we renamed it from alloc_demote_folio(), and made it none-static. If we use alloc_migration_target() from __damon_pa_migrate_folio_list(), there is no reason to keep the changes. Let's revert those too.

Cc-ing Honggyu, who originally implemented the current behavior of __damon_pa_migrate(). Honggyu, could you please let us know if the above suggested changes are not ok for you?

If Honggyu has no problem at the suggested change, Simon, would you mind doing that? I can also make the patches. I don't really care who do that. I just think someone should do that. This shouldn't be urgent real issue, in my opinion, though.

Quoted 4 lines

I'd like to confirm two modification points with you:
​​1.Regarding alloc_migrate_folio()​​:
Restoring the original nodemask and gfp_mask in this function is the correct approach, correct?

I think that's correct, but let's discuss about the patch on the patch's thread.

Quoted 3 lines

​​2.Regarding DAMON's migration logic​​:
The target scope should be expanded from a single specified node to the entire memory tier
(where the target node resides), correct?

I don't think so, as abovely explained.

Quoted 1 line

​​Can we confirm these two points are agreed upon?​

I believe hope this is answered above.

Thanks,
SJ

[...]

#2 lore raw
Quoted 90 lines · On Fri, 30 May 2025 12:40:16 -0700 SeongJae Park <[email protected]> wrote:

Hi Simon,

Thank you for continuing this important discussion.

Before starting, though, seems your mail client is not setting 'In-Reply-To' field of your mails. For people who uses 'In-Reply-To' field based threads displaying tools, ths thread could be difficult to read the whole contents. Please consider using tools that set the field correctly if possible.

You could get more information about available mailing tools from
https://docs.kernel.org/process/email-clients.html

Btw, I use hkml
(https://docs.kernel.org/process/email-clients.html#hackermail-tui) ;)

On Fri, 30 May 2025 08:04:42 +0000 Simon Wang (王传国) <[email protected]> wrote:

[...]

Quoted 2 lines

Your concern is that adding the bool use_nodes_of_tier variable and introducing an additional parameter to multiple functions would cause ABI changes, correct?​​

You are correct.

Quoted 7 lines

​​I propose avoiding the creation of the 'use_nodes_of_tier' sysfs file. Instead, we can modify the __damon_pa_migrate_folio_list() function to change the allowed_mask from NODE_MASK_NONE to the full node mask of the entire tier where the target_nid resides. This approach would be similar to the implementation in commit 320080272892 ('mm/demotion: demote pages according to allocation fallback order').

Then, this causes a behavior change, which we should not allow if it can be considered a regression. In other words, we could do this if it is a clear improvement.

So, let's think about if your proposed change is an improvement. As the commit 320080272892 is nicely explaining, I think that it is an improved behavior for demotion. Actually it seems good behavior for promotion, too. But, the behavior we are discussing here is not for the demotion but general migration (specifically, DAMOS_MIGRATE_{HOT,COLD}).

In my opinion, DAMOS_MIGRATE_{HOT,COLD} behavior should be somewhat similar to that of move_pages() syscall, to make its behavior easy to expect. So I think having commit 320080272892's behavior improvement to DAMOS_MIGRATE_{HOT,COLD} is not a right thing to do.

And this asks me a question. Is current DAMOS_MIGRATE_{HOT,COLD} behavior similar to move_pages() syscall? Not really, since do_move_pages_to_node(), which is called from move_pages() syscall and calls migrate_pages() is setting mtc->nmask as NULL, while DAMOS_MIGRATE_{HOT,COLD} set it as NODE_MASK_NONE. Also, do_move_pages_to_node() uses alloc_migration_target() while DAMOS_MIGRATE_{HOT,COLD} uses alloc_migrate_folio().

I overlooked this different behavior while reviewing this code, sorry. And I don't think this difference is what we need to keep, unless there are good rasons that well documented. Thank you for let us find this, Simon.

So I suggest to set mtc->nmask as NULL, and use alloc_migration_target() from __damon_pa_migrate_folio_list(), same to move_pages() system call. To use alloc_migrate_folio() from __damon_pa_migrate_folio_list(), we renamed it from alloc_demote_folio(), and made it none-static. If we use alloc_migration_target() from __damon_pa_migrate_folio_list(), there is no reason to keep the changes. Let's revert those too.

Cc-ing Honggyu, who originally implemented the current behavior of __damon_pa_migrate(). Honggyu, could you please let us know if the above suggested changes are not ok for you?

If Honggyu has no problem at the suggested change, Simon, would you mind doing that? I can also make the patches. I don't really care who do that. I just think someone should do that. This shouldn't be urgent real issue, in my opinion, though.

Quoted 4 lines

I'd like to confirm two modification points with you:
​​1.Regarding alloc_migrate_folio()​​:
Restoring the original nodemask and gfp_mask in this function is the correct approach, correct?

I think that's correct, but let's discuss about the patch on the patch's thread.

Quoted 3 lines

​​2.Regarding DAMON's migration logic​​:
The target scope should be expanded from a single specified node to the entire memory tier
(where the target node resides), correct?

I don't think so, as abovely explained.

Quoted 1 line

​​Can we confirm these two points are agreed upon?​

Quoted 7 lines

I believe hope this is answered above.

Thanks,
SJ

[...]

Sent using hkml (https://github.com/sjp38/hackermail)

maintainer
#3 lore raw
In reply to SeongJae Park

On Fri, 30 May 2025 12:40:16 -0700 SeongJae Park <[email protected]> wrote:
[...]

Quoted 7 lines

So I suggest to set mtc->nmask as NULL, and use alloc_migration_target() from __damon_pa_migrate_folio_list(), same to move_pages() system call. To use alloc_migrate_folio() from __damon_pa_migrate_folio_list(), we renamed it from alloc_demote_folio(), and made it none-static. If we use alloc_migration_target() from __damon_pa_migrate_folio_list(), there is no reason to keep the changes. Let's revert those too.

Quoted 8 lines

Cc-ing Honggyu, who originally implemented the current behavior of __damon_pa_migrate(). Honggyu, could you please let us know if the above suggested changes are not ok for you?

If Honggyu has no problem at the suggested change, Simon, would you mind doing that? I can also make the patches. I don't really care who do that. I just think someone should do that. This shouldn't be urgent real issue, in my opinion, though.

This is a friendly reminder of the above. Please let me know if you have any opinion. I will proceed to make the patches if I don't get other inputs by this week.

Thanks,
SJ

[...]

#4 lore raw
In reply to SeongJae Park

Hi SeongJae and Simon,

Quoted 9 lines · SeongJae Park

Hi Simon,

Thank you for continuing this important discussion.

Before starting, though, seems your mail client is not setting 'In-Reply-To' field of your mails. For people who uses 'In-Reply-To' field based threads displaying tools, ths thread could be difficult to read the whole contents. Please consider using tools that set the field correctly if possible.

Sorry for the late response, I also had some difficulty to find its original patch and I just found it and replied at the following links.
https://lore.kernel.org/linux-mm/[email protected]
https://lore.kernel.org/linux-mm/[email protected]

Quoted 23 lines

You could get more information about available mailing tools from
https://docs.kernel.org/process/email-clients.html

Btw, I use hkml
(https://docs.kernel.org/process/email-clients.html#hackermail-tui) ;)

On Fri, 30 May 2025 08:04:42 +0000 Simon Wang (王传国) <[email protected]> wrote:

[...]

Quoted 2 lines

Your concern is that adding the bool use_nodes_of_tier variable and introducing an additional parameter to multiple functions would cause ABI changes, correct?​​

You are correct.

Quoted 7 lines

​​I propose avoiding the creation of the 'use_nodes_of_tier' sysfs file. Instead, we can modify the __damon_pa_migrate_folio_list() function to change the allowed_mask from NODE_MASK_NONE to the full node mask of the entire tier where the target_nid resides. This approach would be similar to the implementation in commit 320080272892 ('mm/demotion: demote pages according to allocation fallback order').

Quoted 3 lines

Then, this causes a behavior change, which we should not allow if it can be considered a regression. In other words, we could do this if it is a clear improvement.

I agree this is a behavior change.

Quoted 11 lines

So, let's think about if your proposed change is an improvement. As the commit 320080272892 is nicely explaining, I think that it is an improved behavior for demotion. Actually it seems good behavior for promotion, too. But, the behavior we are discussing here is not for the demotion but general migration (specifically, DAMOS_MIGRATE_{HOT,COLD}).

In my opinion, DAMOS_MIGRATE_{HOT,COLD} behavior should be somewhat similar to that of move_pages() syscall, to make its behavior easy to expect. So I think having commit 320080272892's behavior improvement to DAMOS_MIGRATE_{HOT,COLD} is not a right thing to do.

Quoted 7 lines

And this asks me a question. Is current DAMOS_MIGRATE_{HOT,COLD} behavior similar to move_pages() syscall? Not really, since do_move_pages_to_node(), which is called from move_pages() syscall and calls migrate_pages() is setting mtc->nmask as NULL, while DAMOS_MIGRATE_{HOT,COLD} set it as NODE_MASK_NONE.

Also, do_move_pages_to_node() uses alloc_migration_target() while DAMOS_MIGRATE_{HOT,COLD} uses alloc_migrate_folio().

I can see alloc_migrate_folio() also calls alloc_migration_target(), but do you mean alloc_migrate_folio() setting mtc->nmask to NULL is the difference?

Quoted 12 lines

I overlooked this different behavior while reviewing this code, sorry. And I don't think this difference is what we need to keep, unless there are good rasons that well documented. Thank you for let us find this, Simon.

So I suggest to set mtc->nmask as NULL, and use alloc_migration_target() from __damon_pa_migrate_folio_list(), same to move_pages() system call. To use alloc_migrate_folio() from __damon_pa_migrate_folio_list(), we renamed it from alloc_demote_folio(), and made it none-static. If we use alloc_migration_target() from __damon_pa_migrate_folio_list(), there is no reason to keep the changes. Let's revert those too.

Quoted 13 lines

Cc-ing Honggyu, who originally implemented the current behavior of __damon_pa_migrate(). Honggyu, could you please let us know if the above suggested changes are not ok for you?

If Honggyu has no problem at the suggested change, Simon, would you mind doing that? I can also make the patches. I don't really care who do that. I just think someone should do that. This shouldn't be urgent real issue, in my opinion, though.

Quoted 4 lines

I'd like to confirm two modification points with you:
​​1.Regarding alloc_migrate_folio()​​:
Restoring the original nodemask and gfp_mask in this function is the correct approach, correct?

I also think restoring the both mtc->nmask and mtc->gfp_mask are needed.

Quoted 8 lines

I think that's correct, but let's discuss about the patch on the patch's thread.

Quoted 3 lines

​​2.Regarding DAMON's migration logic​​:
The target scope should be expanded from a single specified node to the entire memory tier
(where the target node resides), correct?

I don't think so, as abovely explained.

I also think this makes our use case unexpected and cannot prevent migration is done beyond other side of socket.

Quoted 10 lines
Quoted 2 lines

​​Can we confirm these two points are agreed upon?​

I believe hope this is answered above.

Thanks,
SJ

[...]

Thanks,
Honggyu

maintainer
#5 lore raw
Quoted 3 lines · On Mon, 9 Jun 2025 21:39:50 +0900 Honggyu Kim <[email protected]> wrote:

Hi SeongJae and Simon,

On 5/31/2025 4:40 AM, SeongJae Park wrote:

[...]

Quoted 1 line
Quoted 1 line · On Fri, 30 May 2025 08:04:42 +0000 Simon Wang (王传国) <[email protected]>…

[...]

Quoted 16 lines
Quoted 16 lines

So, let's think about if your proposed change is an improvement. As the commit 320080272892 is nicely explaining, I think that it is an improved behavior for demotion. Actually it seems good behavior for promotion, too. But, the behavior we are discussing here is not for the demotion but general migration (specifically, DAMOS_MIGRATE_{HOT,COLD}).

In my opinion, DAMOS_MIGRATE_{HOT,COLD} behavior should be somewhat similar to that of move_pages() syscall, to make its behavior easy to expect. So I think having commit 320080272892's behavior improvement to DAMOS_MIGRATE_{HOT,COLD} is not a right thing to do.

And this asks me a question. Is current DAMOS_MIGRATE_{HOT,COLD} behavior similar to move_pages() syscall? Not really, since do_move_pages_to_node(), which is called from move_pages() syscall and calls migrate_pages() is setting mtc->nmask as NULL, while DAMOS_MIGRATE_{HOT,COLD} set it as NODE_MASK_NONE.

Quoted 5 lines
Quoted 2 lines

Also, do_move_pages_to_node() uses alloc_migration_target() while DAMOS_MIGRATE_{HOT,COLD} uses alloc_migrate_folio().

I can see alloc_migrate_folio() also calls alloc_migration_target(), but do you mean alloc_migrate_folio() setting mtc->nmask to NULL is the difference?

Yes, and also alloc_migration_target()'s internal optimizations for demotion use case.

Nonetheless, I'm saying about the differences between DAMOS_MIGRATE_{HOT,COLD} and move_pages() behaviors in the bigger context.

Quoted 13 lines
Quoted 13 lines

I overlooked this different behavior while reviewing this code, sorry. And I don't think this difference is what we need to keep, unless there are good rasons that well documented. Thank you for let us find this, Simon.

So I suggest to set mtc->nmask as NULL, and use alloc_migration_target() from __damon_pa_migrate_folio_list(), same to move_pages() system call. To use alloc_migrate_folio() from __damon_pa_migrate_folio_list(), we renamed it from alloc_demote_folio(), and made it none-static. If we use alloc_migration_target() from __damon_pa_migrate_folio_list(), there is no reason to keep the changes. Let's revert those too.

Quoted 8 lines
Quoted 8 lines

Cc-ing Honggyu, who originally implemented the current behavior of __damon_pa_migrate(). Honggyu, could you please let us know if the above suggested changes are not ok for you?

If Honggyu has no problem at the suggested change, Simon, would you mind doing that? I can also make the patches. I don't really care who do that. I just think someone should do that. This shouldn't be urgent real issue, in my opinion, though.

I will send an RFC for this soon, to make discussions easier and unblocked.

Thanks,
SJ

[...]