mm/vmscan: respect mems_effective in demote_folio_list()
Subsystems (8)
PATCH 1/1 mm/vmscan: respect mems_effective in demote_folio_list()
Commit 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim") introduces the cpuset.mems_effective check and applies it to can_demote(). However, it does not apply this check in demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.
To address the issue that demotion targets do not respect cpuset.mem_effective in demote_folio_list(), implement a new function get_demotion_targets(), which returns a preferred demotion target and all allowed (fallback) nodes against mems_effective, and update demote_folio_list() and can_demote() accordingly to use get_demotion_targets().
Furthermore, update some supporting functions:
- Add a parameter for next_demotion_node() to return a copy of node_demotion[]->preferred, allowing get_demotion_targets() to select the next-best node for demotion.
- Change the parameters for cpuset_node_allowed() and mem_cgroup_node_allowed() from nid to nodemask * to allow for direct logic-and operations with mems_effective.
--- include/linux/cpuset.h | 5 +-- include/linux/memcontrol.h | 6 +-- include/linux/memory-tiers.h | 6 +-- kernel/cgroup/cpuset.c | 14 +++---- mm/memcontrol.c | 5 ++- mm/memory-tiers.c | 8 +++- mm/vmscan.c | 77 +++++++++++++++++++++++++++++------- 7 files changed, 87 insertions(+), 34 deletions(-)
include/linux/cpuset.h +2 -3
@@ -174,7 +174,7 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern void cpuset_node_allowed (struct cgroup *cgroup,nodemask_t *nodes);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }
@@ -301,9 +301,8 @@ static inline bool read_mems_allowed_retry(unsigned int seq) return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline void cpuset_node_allowed (struct cgroup *cgroup,nodemask_t *nodes) { -return true; }#endif /* !CONFIG_CPUSETS */
include/linux/memcontrol.h +3 -3
@@ -1740,7 +1740,7 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +void mem_cgroup_node_allowed (struct mem_cgroup *memcg,nodemask_t *nodes);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);
@@ -1811,9 +1811,9 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +static inline void mem_cgroup_node_allowed (struct mem_cgroup *memcg, +nodemask_t *nodes) { -return true; }static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
include/linux/memory-tiers.h +3 -3
@@ -53,11 +53,11 @@ struct memory_dev_type *mt_find_alloc_memory_type(int adist, struct list_head *memory_types);void mt_put_memory_types (struct list_head *memory_types);#ifdef CONFIG_MIGRATION -int next_demotion_node (int node); +int next_demotion_node (int node,nodemask_t *mask);void node_get_allowed_targets (pg_data_t *pgdat,nodemask_t *targets);bool node_is_toptier (int node);#else -static inline int next_demotion_node (int node) +static inline int next_demotion_node (int node,nodemask_t *mask) {return NUMA_NO_NODE ; }
@@ -101,7 +101,7 @@ static inline void clear_node_memory_type(int node, struct memory_dev_type *memt } -static inline int next_demotion_node (int node) +static inline int next_demotion_node (int node,nodemask_t *mask) {return NUMA_NO_NODE ; }
kernel/cgroup/cpuset.c +6 -8
@@ -4416,11 +4416,10 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)return allowed; } -bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +void cpuset_node_allowed (struct cgroup *cgroup,nodemask_t *nodes) {struct cgroup_subsys_state *css;struct cpuset *cs; -bool allowed;/* * In v1, mem_cgroup and cpuset are unlikely in the same hierarchy
@@ -4428,16 +4427,16 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)* so return true to avoid taking a global lock on the empty check. */ if (!cpuset_v2 ()) -return true; +return ; css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys);if (!css) -return true; +return ;/* * Normally, accessing effective_mems would require the cpuset_mutex -* or callback_lock - but node_isset is atomic and the reference -* taken via cgroup_get_e_css is sufficient to protect css. +* or callback_lock - but the reference taken via cgroup_get_e_css +* is sufficient to protect css. * * Since this interface is intended for use by migration paths, we * relax locking here to avoid taking global locks - while accepting
@@ -4447,9 +4446,8 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)* cannot make strong isolation guarantees, so this is acceptable. */ cs = container_of(css, struct cpuset, css); - allowed =node_isset (nid, cs->effective_mems); +nodes_and (*nodes, *nodes, cs->effective_mems);css_put (css); -return allowed; }/**
mm/memcontrol.c +3 -2
@@ -5597,9 +5597,10 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +void mem_cgroup_node_allowed (struct mem_cgroup *memcg,nodemask_t *nodes) { -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +if (memcg) +cpuset_node_allowed (memcg->css.cgroup, nodes); }void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
mm/memory-tiers.c +7 -1
@@ -320,13 +320,14 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)/** * next_demotion_node() - Get the next node in the demotion path * @node: The starting node to lookup the next node +* @mask: The preferred nodemask copy to be returned * * Return: node id for next memory node in the demotion path hierarchy * from @node; NUMA_NO_NODE if @node is terminal. This does not keep * @node online or guarantee that it *continues* to be the next demotion * target. */ -int next_demotion_node (int node) +int next_demotion_node (int node,nodemask_t *mask) {struct demotion_nodes *nd;int target;
@@ -355,7 +356,12 @@ int next_demotion_node(int node)* last target node. Or introducing per-cpu data to avoid * caching issue, which seems more complicated. So selecting * target node randomly seems better until now. +* +* Copy preferred nodes as the fallback if the returned one +* does not satisify some constraints like cpuset. */ +if (mask) +nodes_copy (*mask, nd->preferred); target =node_random (&nd->preferred);rcu_read_unlock ();
mm/vmscan.c +63 -14
118 lines not on this page · raw message
--
2.52.0.351.gbe84eed79e-goog
Andrew Morton
maintainerQuoted 3 lines · On Sat, 20 Dec 2025 06:10:21 +0000 Bing Jiao <[email protected]> wrote:
Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote().
So we'll want
in the changelog.
Quoted 10 lines
However, it does not apply this check in
demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.To address the issue that demotion targets do not respect cpuset.mem_effective in
demote_folio_list(), implement a new functionget_demotion_targets(), which returns a preferred demotion target and all allowed (fallback) nodes againstmems_effective, and updatedemote_folio_list()andcan_demote()accordingly to useget_demotion_targets().
7d709f49babc fist appeared in 6.16, so we must decide whether to backport this fix into -stable kernels, via a Cc:
<[email protected]>.
To make this decision it's best to have a clear understanding of the userspace visible impact of the bug. Putting pages into improper nodes is undesirable, but how much does it affect real-world workloads? Please include in the changelog some words about this to help others understand why we should backport the fix.
Quoted 7 lines
Furthermore, update some supporting functions:
- Add a parameter fornext_demotion_node()to return a copy of node_demotion[]->preferred, allowingget_demotion_targets()to select the next-best node for demotion.
- Change the parameters forcpuset_node_allowed()andmem_cgroup_node_allowed()from nid to nodemask * to allow for direct logic-and operations withmems_effective.
If we do decide to backport the fix into earlier kernels then it's best to keep the patch as small and as simple as possible. So non-bugfix changes such as these are best made via a second followup patch which can be merged via the normal -rc staging process.
Quoted 41 lines · Andrew Morton
Quoted 3 lines · On Sat, 20 Dec 2025 06:10:21 +0000 Bing Jiao <[email protected]> wrote:
Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote().So we'll want
Fixes 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim")in the changelog.
Quoted 10 lines
However, it does not apply this check in
demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.To address the issue that demotion targets do not respect cpuset.mem_effective in
demote_folio_list(), implement a new functionget_demotion_targets(), which returns a preferred demotion target and all allowed (fallback) nodes againstmems_effective, and updatedemote_folio_list()andcan_demote()accordingly to useget_demotion_targets().
7d709f49babcfist appeared in 6.16, so we must decide whether to backport this fix into -stable kernels, via a Cc:
<[email protected]>.To make this decision it's best to have a clear understanding of the userspace visible impact of the bug. Putting pages into improper nodes is undesirable, but how much does it affect real-world workloads? Please include in the changelog some words about this to help others understand why we should backport the fix.
Quoted 7 lines
Furthermore, update some supporting functions:
- Add a parameter fornext_demotion_node()to return a copy of node_demotion[]->preferred, allowingget_demotion_targets()to select the next-best node for demotion.
- Change the parameters forcpuset_node_allowed()andmem_cgroup_node_allowed()from nid to nodemask * to allow for direct logic-and operations withmems_effective.
Quoted 5 lines
If we do decide to backport the fix into earlier kernels then it's best to keep the patch as small and as simple as possible. So non-bugfix changes such as these are best made via a second followup patch which can be merged via the normal -rc staging process.
Hi Andrew, thank you for the review and suggestions.
I hvae sent a patch v2 for the backport. However, I forgot to add the CC:stable line. I will fix it in v3.
Best,
Bing
Gregory Price
maintainerI think this patch can be done without as many changes as proposed here.
Review
Quoted 11 lines
(quoted diff) +4
-4-bool mem_cgroup_node_allowed(struct mem_cgroup *memcg, int nid); +void mem_cgroup_node_allowed(struct mem_cgroup *memcg, nodemask_t *nodes); -static inline bool mem_cgroup_node_allowed(struct mem_cgroup *memcg, int nid) +static inline void mem_cgroup_node_allowed(struct mem_cgroup *memcg, -int next_demotion_node(int node); +int next_demotion_node(int node, nodemask_t *mask); -bool cpuset_node_allowed(struct cgroup *cgroup, int nid) +void cpuset_node_allowed(struct cgroup *cgroup, nodemask_t *nodes)
These are some fairly major contract changes, and the names don't make much sense as a result.
Would be better to just make something like
/* Filter the given nmask based on cpuset.mems.allowed */ mem_cgroup_filter_mems_allowed(memg, nmask);
(or some other, better name)
separate of the existing interfaces, and operate on one scratch-mask if possible.
Quoted 29 lines
(quoted diff) +29
-0+static int get_demotion_targets(nodemask_t *targets, struct pglist_data *pgdat, + struct mem_cgroup *memcg) +{ + nodemask_t allowed_mask; + nodemask_t preferred_mask; + int preferred_node; + + if (!pgdat) + return NUMA_NO_NODE; + + preferred_node = next_demotion_node(pgdat->node_id, &preferred_mask); + if (preferred_node == NUMA_NO_NODE) + return NUMA_NO_NODE; + + node_get_allowed_targets(pgdat, &allowed_mask); + mem_cgroup_node_allowed(memcg, &allowed_mask); + if (nodes_empty(allowed_mask)) + return NUMA_NO_NODE; + + if (targets) + nodes_copy(*targets, allowed_mask); + + do { + if (node_isset(preferred_node, allowed_mask)) + return preferred_node; + + nodes_and(preferred_mask, preferred_mask, allowed_mask); + if (!nodes_empty(preferred_mask)) + return node_random(&preferred_mask);
Review
Quoted 10 lines
(quoted diff) +10
-0+ + /* + * Hop to the next tier of preferred nodes. Even if + * preferred_node is not set in allowed_mask, still can use it + * to query the nest-best demotion nodes. + */ + preferred_node = next_demotion_node(preferred_node, + &preferred_mask); + } while (preferred_node != NUMA_NO_NODE); +
What you're implementing here is effectively a new feature - allowing demotion to jump nodes rather than just target the next demotion node.
This is nice, but it should be a separate patch proposal (I think Andrew said something as much already) - not as part of a fix.
Review
Quoted 3 lines
(quoted diff) +3
-0+ /* + * Should not reach here, as a non-empty allowed_mask ensures + * there must have a target node for demotion.
Does it? What if preferred_node is online when calling next_demotion_node(), but then is offline when node_get_allowed_targets() is called?
Review
Quoted 7 lines
(quoted diff) +7
-0+ * Otherwise, it suggests something wrong in node_demotion[]->preferred, + * where the same-tier nodes have different preferred targets. + * E.g., if node 0 identifies both nodes 2 and 3 as preferred targets, + * but nodes 2 and 3 themselves have different preferred nodes. + */ + WARN_ON_ONCE(1); + return node_random(&allowed_mask);
Just returning a random allowed node seems like an objectively poor result and we should just not demote if we reach this condition. It likesly means hotplug was happening and node states changed.
Review static unsigned int demote_folio_list(struct list_head *demote_folios, · line 1047
Quoted 9 lines
(quoted diff) +1
-2@@ -1041,10 +1090,10 @@ static unsigned int demote_folio_list(struct list_head *demote_folios, if (list_empty(demote_folios)) return 0; + target_nid = get_demotion_targets(&allowed_mask, pgdat, memcg); if (target_nid == NUMA_NO_NODE) return 0; - - node_get_allowed_targets(pgdat, &allowed_mask);
in the immediate fixup patch, it seems more expedient to just add the function i described above
/* Filter the given nmask based on cpuset.mems.allowed */ mem_cgroup_filter_mems_allowed(memg, nmask);
and then add that immediate after the node_get_allowed_targets() call.
Then come back around afterwards to add the tier/node-skip functionality from above in a separate feature patch.
~Gregory
---
mm/vmscan.c +5 -0
@@ -1046,6 +1046,11 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,node_get_allowed_targets (pgdat, &allowed_mask); +/* Filter based on mems_allowed, fail if the result is empty */ +mem_cgroup_filter_nodemask (memcg, &allowed_mask); +if (nodes_empty (allowed_mask)) +return 0 ; +/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL, (unsigned long)&mtc, MIGRATE_ASYNC, MR_DEMOTION,
Quoted 20 lines · Gregory Price
I think this patch can be done without as many changes as proposed here.
Review
Quoted 11 lines
(quoted diff) +4
-4-bool mem_cgroup_node_allowed(struct mem_cgroup *memcg, int nid); +void mem_cgroup_node_allowed(struct mem_cgroup *memcg, nodemask_t *nodes); -static inline bool mem_cgroup_node_allowed(struct mem_cgroup *memcg, int nid) +static inline void mem_cgroup_node_allowed(struct mem_cgroup *memcg, -int next_demotion_node(int node); +int next_demotion_node(int node, nodemask_t *mask); -bool cpuset_node_allowed(struct cgroup *cgroup, int nid) +void cpuset_node_allowed(struct cgroup *cgroup, nodemask_t *nodes)These are some fairly major contract changes, and the names don't make much sense as a result.
Would be better to just make something like
Quoted 8 lines
/* Filter the given nmask based on cpuset.mems.allowed */
mem_cgroup_filter_mems_allowed(memg, nmask);(or some other, better name)
separate of the existing interfaces, and operate on one scratch-mask if possible.
Hi Gregory, thank you for the review and suggestions.
I have divided these changes into 2 patches based on your suggestions. Since mem_cgroup_node_allowed() and cpuset_node_allowed() are dangling, they are removed in v2 2/2.
Quoted 43 lines
Quoted 29 lines
(quoted diff) +29
-0+static int get_demotion_targets(nodemask_t *targets, struct pglist_data *pgdat, + struct mem_cgroup *memcg) +{ + nodemask_t allowed_mask; + nodemask_t preferred_mask; + int preferred_node; + + if (!pgdat) + return NUMA_NO_NODE; + + preferred_node = next_demotion_node(pgdat->node_id, &preferred_mask); + if (preferred_node == NUMA_NO_NODE) + return NUMA_NO_NODE; + + node_get_allowed_targets(pgdat, &allowed_mask); + mem_cgroup_node_allowed(memcg, &allowed_mask); + if (nodes_empty(allowed_mask)) + return NUMA_NO_NODE; + + if (targets) + nodes_copy(*targets, allowed_mask); + + do { + if (node_isset(preferred_node, allowed_mask)) + return preferred_node; + + nodes_and(preferred_mask, preferred_mask, allowed_mask); + if (!nodes_empty(preferred_mask)) + return node_random(&preferred_mask);Review
Quoted 10 lines
(quoted diff) +10
-0+ + /* + * Hop to the next tier of preferred nodes. Even if + * preferred_node is not set in allowed_mask, still can use it + * to query the nest-best demotion nodes. + */ + preferred_node = next_demotion_node(preferred_node, + &preferred_mask); + } while (preferred_node != NUMA_NO_NODE); +What you're implementing here is effectively a new feature - allowing demotion to jump nodes rather than just target the next demotion node.
Quoted 3 lines
This is nice, but it should be a separate patch proposal (I think Andrew said something as much already) - not as part of a fix.
Thanks for the suggestion.
I sent a v2 patch series for fixes and backport. This function (jump node) will be sent in another thread for distinguishing between fixes and features.
Quoted 37 lines
Review
Quoted 3 lines
(quoted diff) +3
-0+ /* + * Should not reach here, as a non-empty allowed_mask ensures + * there must have a target node for demotion.Does it? What if
preferred_nodeis online when callingnext_demotion_node(), but then is offline whennode_get_allowed_targets()is called?Review
Quoted 7 lines
(quoted diff) +7
-0+ * Otherwise, it suggests something wrong in node_demotion[]->preferred, + * where the same-tier nodes have different preferred targets. + * E.g., if node 0 identifies both nodes 2 and 3 as preferred targets, + * but nodes 2 and 3 themselves have different preferred nodes. + */ + WARN_ON_ONCE(1); + return node_random(&allowed_mask);Just returning a random allowed node seems like an objectively poor result and we should just not demote if we reach this condition. It likesly means hotplug was happening and node states changed.
Review static unsigned int demote_folio_list(struct list_head *demote_folios, · line 1047
Quoted 9 lines
(quoted diff) +1
-2@@ -1041,10 +1090,10 @@ static unsigned int demote_folio_list(struct list_head *demote_folios, if (list_empty(demote_folios)) return 0; + target_nid = get_demotion_targets(&allowed_mask, pgdat, memcg); if (target_nid == NUMA_NO_NODE) return 0; - - node_get_allowed_targets(pgdat, &allowed_mask);in the immediate fixup patch, it seems more expedient to just add the function i described above
/* Filter the given nmask based on cpuset.mems.allowed */
mem_cgroup_filter_mems_allowed(memg, nmask);
Quoted 7 lines
and then add that immediate after the
node_get_allowed_targets()call.Then come back around afterwards to add the tier/node-skip functionality from above in a separate feature patch.
~Gregory
Thanks for the hit. I had never considered hot-swapping before.
Quoted 10 lines
---
mm/vmscan.c +0
-0@@ -1046,6 +1046,11 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,
mm/vmscan.c static unsigned int demote_folio_list(struct list_head *demote_folios, · line 1056
Quoted 10 lines
mm/vmscan.c +5
-0node_get_allowed_targets (pgdat, &allowed_mask); +/* Filter based on mems_allowed, fail if the result is empty */ +mem_cgroup_filter_nodemask (memcg, &allowed_mask); +if (nodes_empty (allowed_mask)) +return 0 ; +/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL, (unsigned long)&mtc, MIGRATE_ASYNC, MR_DEMOTION,
Thanks for the code. My v2 1/2 is based on your suggestion with some changes.
Best,
Bing
v2 cover fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoThis patch series addresses two issues in demote_folio_list() and can_demote() in reclaim/demotion.
Commit 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim") introduces the cpuset.mems_effective check and applies it to can_demote(). However:
1. It does not apply this check in demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.
2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set in mems_effective.
To address these bugs, implement a new function mem_cgroup_filter_mems_allowed() to filter out nodes that are not set in mems_effective, and update demote_folio_list() and can_demote() accordingly.
Reproduct Bug 1:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.
Test script to reproduct:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Should respect node 0-2 limit.
# Observation: Node 3 shows significant allocation (MemFree drops) stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1
Reproduct Bug 2:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Pages are demoted to Nodes 4-5
# Observation: No pages are demoted before oom.
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2Bing Jiao (2):
mm/vmscan: respect mems_effective in demote_folio_list() mm/vmscan: check all allowed targets in can_demote()
include/linux/cpuset.h | 6 +++--- include/linux/memcontrol.h | 6 +++--- kernel/cgroup/cpuset.c | 12 +++++------- mm/memcontrol.c | 5 +++-- mm/vmscan.c | 27 ++++++++++++++++++--------- 5 files changed, 32 insertions(+), 24 deletions(-)
--
2.52.0.351.gbe84eed79e-goog
PATCH v2 1/2 mm/vmscan: respect mems_effective in demote_folio_list()
Commit 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim") introduces the cpuset.mems_effective check and applies it to can_demote(). However, it does not apply this check in demote_folio_list().
This omission leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems. The impact is two-fold:
1. Resource Isolation: This bug breaks resource isolation provided by cpuset.mems. It allows pages to be demoted to nodes that are dedicated to other tasks or are intended for hot-unplugging.
2. Performance Issue: In multi-tier systems, users use cpuset.mems to bind tasks to different performed-far tiers (e.g., avoiding the slowest tiers for latency-sensitive data). This bug can cause unexpected latency spikes if pages are demoted to the farthest nodes.
To address the bug, implement a new function mem_cgroup_filter_mems_allowed() to filter out nodes that are not set in mems_effective, and update demote_folio_list() to utilize this filtering logic. This ensures that demotions target respect task's memory placement constraints.
--- include/linux/cpuset.h | 6 ++++++ include/linux/memcontrol.h | 7 +++++++ kernel/cgroup/cpuset.c | 18 ++++++++++++++++++ mm/memcontrol.c | 6 ++++++ mm/vmscan.c | 13 ++++++++++--- 5 files changed, 47 insertions(+), 3 deletions(-)
include/linux/cpuset.h +6 -0
@@ -175,6 +175,7 @@ static inline void set_mems_allowed(nodemask_t nodemask) }extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }
@@ -305,6 +306,11 @@ static inline bool cpuset_node_allowed(struct cgroup *cgroup, int nid) {return true; } + +static inline void cpuset_node_filter_allowed (struct cgroup *cgroup, +nodemask_t *mask) +{ +}#endif /* !CONFIG_CPUSETS */ #endif /* _LINUX_CPUSET_H */
include/linux/memcontrol.h +7 -0
@@ -1742,6 +1742,8 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +void mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg,nodemask_t *mask); +void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);static inline bool memcg_is_dying (struct mem_cgroup *memcg)
@@ -1816,6 +1818,11 @@ static inline bool mem_cgroup_node_allowed(struct mem_cgroup *memcg, int nid)return true; } +static inline bool mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg, +nodemask_t *mask) +{ +} +static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg) { }
kernel/cgroup/cpuset.c +18 -0
@@ -4452,6 +4452,24 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)return allowed; } +void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) +{ +struct cgroup_subsys_state *css; +struct cpuset *cs; + +if (!cpuset_v2 ()) +return ; + + css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys); +if (!css) +return ; + +/* Follows the same assumption in cpuset_node_allowed() */ + cs = container_of(css, struct cpuset, css); +nodes_and (*mask, *mask, cs->effective_mems); +css_put (css); +} +/** * cpuset_spread_node() - On which node to begin search for a page * @rotor: round robin rotor
mm/memcontrol.c +6 -0
@@ -5602,6 +5602,12 @@ bool mem_cgroup_node_allowed(struct mem_cgroup *memcg, int nid)return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; } +void mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg,nodemask_t *mask) +{ +if (memcg) +cpuset_node_filter_allowed (memcg->css.cgroup, mask); +} +void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg) {if (mem_cgroup_disabled () || !cgroup_subsys_on_dfl (memory_cgrp_subsys))
mm/vmscan.c +10 -3
@@ -1018,7 +1018,8 @@ static struct folio *alloc_demote_folio(struct folio *src,* Folios which are not demoted are left on @demote_folios. */ static unsigned int demote_folio_list (struct list_head *demote_folios, -struct pglist_data *pgdat) +struct pglist_data *pgdat, +struct mem_cgroup *memcg) {int target_nid =next_demotion_node (pgdat->node_id);unsigned int nr_succeeded;
@@ -1032,7 +1033,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,*/ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, .nmask = &allowed_mask, .reason = MR_DEMOTION, };
@@ -1044,6 +1044,13 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,return 0 ;node_get_allowed_targets (pgdat, &allowed_mask); +/* Filter the given nmask based on cpuset.mems.allowed */ +mem_cgroup_filter_mems_allowed (memcg, &allowed_mask); +if (nodes_empty (allowed_mask)) +return 0 ; +if (!node_isset (target_nid, allowed_mask)) + target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL,
@@ -1565,7 +1572,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,/* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_demoted =demote_folio_list (&demote_folios, pgdat); + nr_demoted =demote_folio_list (&demote_folios, pgdat, memcg); nr_reclaimed += nr_demoted; stat->nr_demoted += nr_demoted;/* Folios that could not be demoted are still in @demote_folios */
Signature · 1 line
2.52.0.351.gbe84eed79e-goog
kernel test robot
build botFYI: kernel test robot notices the stable kernel rule is not satisfied.
Hi,
Thanks for your patch.
FYI: kernel test robot notices the stable kernel rule is not satisfied.
The check is based on https://www.kernel.org/doc/html/latest/process/stable-kernel-rules.html#option-1
Rule: add the tag "Cc: [email protected]" in the sign-off area to have the patch automatically included in the stable tree.
Subject: [PATCH v2 1/2] mm/vmscan: respect mems_effective in demote_folio_list()
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Quoted 100 lines · Bing Jiao
Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote(). However, it does not apply this check indemote_folio_list().This omission leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems. The impact is two-fold:
1. Resource Isolation: This bug breaks resource isolation provided by cpuset.mems. It allows pages to be demoted to nodes that are dedicated to other tasks or are intended for hot-unplugging.
2. Performance Issue: In multi-tier systems, users use cpuset.mems to bind tasks to different performed-far tiers (e.g., avoiding the slowest tiers for latency-sensitive data). This bug can cause unexpected latency spikes if pages are demoted to the farthest nodes.
To address the bug, implement a new function
mem_cgroup_filter_mems_allowed()to filter out nodes that are not set inmems_effective, and updatedemote_folio_list()to utilize this filtering logic. This ensures that demotions target respect task's memory placement constraints.Fixes 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim")Signed-off-by Bing Jiao <[email protected]>--- include/linux/cpuset.h | 6 ++++++ include/linux/memcontrol.h | 7 +++++++ kernel/cgroup/cpuset.c | 18 ++++++++++++++++++ mm/memcontrol.c | 6 ++++++ mm/vmscan.c | 13 ++++++++++--- 5 files changed, 47 insertions(+), 3 deletions(-)include/linux/cpuset.h +6
-0@@ -175,6 +175,7 @@ static inline void set_mems_allowed(nodemask_t nodemask) }extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }@@ -305,6 +306,11 @@ static inline bool cpuset_node_allowed(struct cgroup *cgroup, int nid) {return true; } + +static inline void cpuset_node_filter_allowed (struct cgroup *cgroup, +nodemask_t *mask) +{ +}#endif /* !CONFIG_CPUSETS */ #endif /* _LINUX_CPUSET_H */ include/linux/memcontrol.h +7
-0@@ -1742,6 +1742,8 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +void mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg,nodemask_t *mask); +void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);static inline bool memcg_is_dying (struct mem_cgroup *memcg)@@ -1816,6 +1818,11 @@ static inline bool mem_cgroup_node_allowed(struct mem_cgroup *memcg, int nid)return true; } +static inline bool mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg, +nodemask_t *mask) +{ +} +static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg) { }kernel/cgroup/cpuset.c +8
-0@@ -4452,6 +4452,24 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)return allowed; } +void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) +{ +struct cgroup_subsys_state *css; +struct cpuset *cs; + +if (!cpuset_v2 ()) +return ; +
kernel/cgroup/cpuset.c bool cpuset_node_allowed(struct cgroup *cgroup, int nid) · line 4472
Quoted 10 lines
kernel/cgroup/cpuset.c +10
-0+ css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys); +if (!css) +return ; + +/* Follows the same assumption in cpuset_node_allowed() */ + cs = container_of(css, struct cpuset, css); +nodes_and (*mask, *mask, cs->effective_mems); +css_put (css); +} +
The functions cpuset_node_filter_allowed and cpuset_node_allowed are similar. We should create a helper function to obtain cs->effective_mems, which can then be used by both cpuset_node_filter_allowed and cpuset_node_allowed.
For example:
nodemask_t *mask cpuset_get_mem_allowed(struct cgroup *cgroup) {
}
bool cpuset_node_allowed(struct cgroup *cgroup, int nid) {e_mask = cpuset_node_allowed(cgroup);
return allowed = node_isset(nid, mask);
}
void cpuset_node_filter_allowed(struct cgroup *cgroup, nodemask_t mask) {e_mask = cpuset_node_allowed(cgroup);nodes_and(mask, *mask, e_mask);
}
Previously, I did not think we should distinguish between cgroup v1 and v2 here. This should be a common function; at least based on its name, it should not be solely for v2.
Quoted 65 lines
kernel/cgroup/cpuset.c +0
-0/** * cpuset_spread_node() - On which node to begin search for a page * @rotor: round robin rotor mm/memcontrol.c +6
-0@@ -5602,6 +5602,12 @@ bool mem_cgroup_node_allowed(struct mem_cgroup *memcg, int nid)return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; } +void mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg,nodemask_t *mask) +{ +if (memcg) +cpuset_node_filter_allowed (memcg->css.cgroup, mask); +} +void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg) {if (mem_cgroup_disabled () || !cgroup_subsys_on_dfl (memory_cgrp_subsys))mm/vmscan.c +10
-3@@ -1018,7 +1018,8 @@ static struct folio *alloc_demote_folio(struct folio *src,* Folios which are not demoted are left on @demote_folios. */ static unsigned int demote_folio_list (struct list_head *demote_folios, -struct pglist_data *pgdat) +struct pglist_data *pgdat, +struct mem_cgroup *memcg) {int target_nid =next_demotion_node (pgdat->node_id);unsigned int nr_succeeded;@@ -1032,7 +1033,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,*/ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, .nmask = &allowed_mask, .reason = MR_DEMOTION, };@@ -1044,6 +1044,13 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,return 0 ;node_get_allowed_targets (pgdat, &allowed_mask); +/* Filter the given nmask based on cpuset.mems.allowed */ +mem_cgroup_filter_mems_allowed (memcg, &allowed_mask); +if (nodes_empty (allowed_mask)) +return 0 ; +if (!node_isset (target_nid, allowed_mask)) + target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL,@@ -1565,7 +1572,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,/* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_demoted =demote_folio_list (&demote_folios, pgdat); + nr_demoted =demote_folio_list (&demote_folios, pgdat, memcg); nr_reclaimed += nr_demoted; stat->nr_demoted += nr_demoted;/* Folios that could not be demoted are still in @demote_folios */
Signature · 2 lines
Best regards, Ridong
kernel test robot
build botkernel test robot noticed the following build warnings:
Hi Bing,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on tj-cgroup/for-next linus/master v6.19-rc2 next-20251219] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Bing-Jiao/mm-vmscan-respect-mems_effective-in-demote_folio_list/20251222-074143 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20251221233635.3761887-2-bingjiao%40google.com patch subject: [PATCH v2 1/2] mm/vmscan: respect mems_effective in demote_folio_list() config: arm-allnoconfig (https://download.01.org/0day-ci/archive/20251223/[email protected]/config) compiler: clang version 22.0.0git (https://github.com/llvm/llvm-project 185f5fd5ce4c65116ca8cf6df467a682ef090499) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251223/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
In file included from kernel/sched/rq-offsets.c:5:
In file included from kernel/sched/sched.h:61:
In file included from include/linux/syscalls_api.h:1:
In file included from include/linux/syscalls.h:96:
In file included from include/trace/syscall.h:7:
In file included from include/linux/trace_events.h:10:
In file included from include/linux/perf_event.h:53:
In file included from include/linux/security.h:35:
In file included from include/linux/bpf.h:32:
>> include/linux/memcontrol.h:1824:1: warning: non-void function does not return a value [-Wreturn-type]
1824 | }
| ^ 1 warning generated.
--
In file included from arch/arm/kernel/signal.c:12:
In file included from include/linux/resume_user_mode.h:8:
>> include/linux/memcontrol.h:1824:1: warning: non-void function does not return a value [-Wreturn-type]
1824 | }
| ^
arch/arm/kernel/signal.c:143:15: warning: variable 'aux' set but not used [-Wunused-but-set-variable]
143 | char __user *aux;
| ^ 2 warnings generated.
--
In file included from kernel/sched/rq-offsets.c:5:
In file included from kernel/sched/sched.h:61:
In file included from include/linux/syscalls_api.h:1:
In file included from include/linux/syscalls.h:96:
In file included from include/trace/syscall.h:7:
In file included from include/linux/trace_events.h:10:
In file included from include/linux/perf_event.h:53:
In file included from include/linux/security.h:35:
In file included from include/linux/bpf.h:32:
>> include/linux/memcontrol.h:1824:1: warning: non-void function does not return a value [-Wreturn-type]
1824 | }
| ^1 warning generated.
vim +1824 include/linux/memcontrol.h
1820
1821 static inline bool mem_cgroup_filter_mems_allowed(struct mem_cgroup *memcg,
1822 nodemask_t *mask)
1823 {
> 1824 }
1825
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botkernel test robot noticed the following build warnings:
Hi Bing,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on tj-cgroup/for-next linus/master v6.19-rc2 next-20251219] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Bing-Jiao/mm-vmscan-respect-mems_effective-in-demote_folio_list/20251222-074143 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/20251221233635.3761887-2-bingjiao%40google.com patch subject: [PATCH v2 1/2] mm/vmscan: respect mems_effective in demote_folio_list() config: alpha-allnoconfig (https://download.01.org/0day-ci/archive/20251223/[email protected]/config) compiler: alpha-linux-gcc (GCC) 15.1.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251223/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
In file included from include/linux/bpf.h:32,
from include/linux/security.h:35,
from include/linux/perf_event.h:53,
from include/linux/trace_events.h:10,
from include/trace/syscall.h:7,
from include/linux/syscalls.h:96,
from include/linux/syscalls_api.h:1,
from kernel/sched/sched.h:61,
from kernel/sched/rq-offsets.c:5:
include/linux/memcontrol.h: In function 'mem_cgroup_filter_mems_allowed':
>> include/linux/memcontrol.h:1824:1: warning: no return statement in function returning non-void [-Wreturn-type]
1824 | }
| ^--
In file included from include/linux/bpf.h:32,
from include/linux/security.h:35,
from include/linux/perf_event.h:53,
from include/linux/trace_events.h:10,
from include/trace/syscall.h:7,
from include/linux/syscalls.h:96,
from include/linux/syscalls_api.h:1,
from kernel/sched/sched.h:61,
from kernel/sched/rq-offsets.c:5:
include/linux/memcontrol.h: In function 'mem_cgroup_filter_mems_allowed':
>> include/linux/memcontrol.h:1824:1: warning: no return statement in function returning non-void [-Wreturn-type]
1824 | }
| ^vim +1824 include/linux/memcontrol.h
1820
1821 static inline bool mem_cgroup_filter_mems_allowed(struct mem_cgroup *memcg,
1822 nodemask_t *mask)
1823 {
> 1824 }
1825
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v2 2/2 mm/vmscan: check all allowed targets in can_demote()
In reply to Bing JiaoCommit 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim") introduces the cpuset.mems_effective check and applies it to can_demote(). However, it checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets. This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set in mems_effective.
To address the bug, use mem_cgroup_filter_mems_allowed() to filter out allowed targets obtained from node_get_allowed_targets(). Also remove some unused functions.
--- include/linux/cpuset.h | 6 ------ include/linux/memcontrol.h | 7 ------- kernel/cgroup/cpuset.c | 28 ++++------------------------ mm/memcontrol.c | 5 ----- mm/vmscan.c | 14 ++++++++------ 5 files changed, 12 insertions(+), 48 deletions(-)
include/linux/cpuset.h +0 -6
@@ -174,7 +174,6 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid);extern void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask);#else /* !CONFIG_CPUSETS */
@@ -302,11 +301,6 @@ static inline bool read_mems_allowed_retry(unsigned int seq)return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) -{ -return true; -} -static inline void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) {
include/linux/memcontrol.h +0 -7
@@ -1740,8 +1740,6 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); -void mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg,nodemask_t *mask);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);
@@ -1813,11 +1811,6 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) -{ -return true; -} -static inline bool mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg,nodemask_t *mask) {
kernel/cgroup/cpuset.c +4 -24
@@ -4416,11 +4416,10 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)return allowed; } -bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) {struct cgroup_subsys_state *css;struct cpuset *cs; -bool allowed;/* * In v1, mem_cgroup and cpuset are unlikely in the same hierarchy
@@ -4428,15 +4427,15 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)* so return true to avoid taking a global lock on the empty check. */ if (!cpuset_v2 ()) -return true; +return ; css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys);if (!css) -return true; +return ;/* * Normally, accessing effective_mems would require the cpuset_mutex -* or callback_lock - but node_isset is atomic and the reference +* or callback_lock - but it is acceptable and the reference * taken via cgroup_get_e_css is sufficient to protect css. * * Since this interface is intended for use by migration paths, we
@@ -4447,25 +4446,6 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)* cannot make strong isolation guarantees, so this is acceptable. */ cs = container_of(css, struct cpuset, css); - allowed =node_isset (nid, cs->effective_mems); -css_put (css); -return allowed; -} - -void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) -{ -struct cgroup_subsys_state *css; -struct cpuset *cs; - -if (!cpuset_v2 ()) -return ; - - css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys); -if (!css) -return ; - -/* Follows the same assumption in cpuset_node_allowed() */ - cs = container_of(css, struct cpuset, css);nodes_and (*mask, *mask, cs->effective_mems);css_put (css); }
mm/memcontrol.c +0 -5
@@ -5597,11 +5597,6 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) -{ -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; -} -void mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg,nodemask_t *mask) {if (memcg)
mm/vmscan.c +8 -6
@@ -344,19 +344,21 @@ static void flush_reclaim_state(struct scan_control *sc)static bool can_demote (int nid,struct scan_control *sc,struct mem_cgroup *memcg) { -int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask; -if (!numa_demotion_enabled) +if (!pgdat || !numa_demotion_enabled)return false;if (sc && sc->no_demotion)return false; - demotion_nid =next_demotion_node (nid); -if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask))return false; -/* If demotion node isn't in the cgroup's mems_allowed, fall back */ -return mem_cgroup_node_allowed (memcg, demotion_nid); +/* Filter the given nmask based on cpuset.mems.allowed */ +mem_cgroup_filter_mems_allowed (memcg, &allowed_mask); +return !nodes_empty (allowed_mask); } static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
Signature · 1 line
2.52.0.351.gbe84eed79e-goog
Re: [PATCH v2 2/2] mm/vmscan: check all allowed targets in can_demote()
Quoted 122 lines · Bing Jiao
Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote(). However, it checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets. This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set inmems_effective.To address the bug, use
mem_cgroup_filter_mems_allowed()to filter out allowed targets obtained fromnode_get_allowed_targets(). Also remove some unused functions.Fixes 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim")Signed-off-by Bing Jiao <[email protected]>--- include/linux/cpuset.h | 6 ------ include/linux/memcontrol.h | 7 ------- kernel/cgroup/cpuset.c | 28 ++++------------------------ mm/memcontrol.c | 5 ----- mm/vmscan.c | 14 ++++++++------ 5 files changed, 12 insertions(+), 48 deletions(-)include/linux/cpuset.h +0
-6@@ -174,7 +174,6 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid);extern void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask);#else /* !CONFIG_CPUSETS */ @@ -302,11 +301,6 @@ static inline bool read_mems_allowed_retry(unsigned int seq)return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) -{ -return true; -} -static inline void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) {include/linux/memcontrol.h +0
-7@@ -1740,8 +1740,6 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); -void mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg,nodemask_t *mask);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);@@ -1813,11 +1811,6 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) -{ -return true; -} -static inline bool mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg,nodemask_t *mask) {kernel/cgroup/cpuset.c +4
-17@@ -4416,11 +4416,10 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)return allowed; } -bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) {struct cgroup_subsys_state *css;struct cpuset *cs; -bool allowed;/* * In v1, mem_cgroup and cpuset are unlikely in the same hierarchy @@ -4428,15 +4427,15 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)* so return true to avoid taking a global lock on the empty check. */ if (!cpuset_v2 ()) -return true; +return ; css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys);if (!css) -return true; +return ;/* * Normally, accessing effective_mems would require the cpuset_mutex -* or callback_lock - but node_isset is atomic and the reference +* or callback_lock - but it is acceptable and the reference * taken via cgroup_get_e_css is sufficient to protect css. * * Since this interface is intended for use by migration paths, we @@ -4447,25 +4446,6 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)* cannot make strong isolation guarantees, so this is acceptable. */ cs = container_of(css, struct cpuset, css); - allowed =node_isset (nid, cs->effective_mems); -css_put (css); -return allowed; -} - -void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) -{ -struct cgroup_subsys_state *css; -struct cpuset *cs; - -if (!cpuset_v2 ()) -return ;
kernel/cgroup/cpuset.c bool cpuset_node_allowed(struct cgroup *cgroup, int nid) · line 4451
Quoted 10 lines
kernel/cgroup/cpuset.c +0
-7- - css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys); -if (!css) -return ; - -/* Follows the same assumption in cpuset_node_allowed() */ - cs = container_of(css, struct cpuset, css);nodes_and (*mask, *mask, cs->effective_mems);css_put (css); }
Oh, I see you merged these two functions here.
However, I think cpuset_get_mem_allowed would be more versatile in general use.
You can then check whether the returned nodemask intersects with your target mask. In the future, there may be scenarios where users simply want to retrieve the effective masks directly.
Quoted 48 lines
mm/memcontrol.c +0
-5@@ -5597,11 +5597,6 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) -{ -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; -} -void mem_cgroup_filter_mems_allowed (struct mem_cgroup *memcg,nodemask_t *mask) {if (memcg)mm/vmscan.c +8
-6@@ -344,19 +344,21 @@ static void flush_reclaim_state(struct scan_control *sc)static bool can_demote (int nid,struct scan_control *sc,struct mem_cgroup *memcg) { -int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask; -if (!numa_demotion_enabled) +if (!pgdat || !numa_demotion_enabled)return false;if (sc && sc->no_demotion)return false; - demotion_nid =next_demotion_node (nid); -if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask))return false; -/* If demotion node isn't in the cgroup's mems_allowed, fall back */ -return mem_cgroup_node_allowed (memcg, demotion_nid); +/* Filter the given nmask based on cpuset.mems.allowed */ +mem_cgroup_filter_mems_allowed (memcg, &allowed_mask); +return !nodes_empty (allowed_mask); } static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
Signature · 2 lines
Best regards, Ridong
Re: [PATCH v2 2/2] mm/vmscan: check all allowed targets in can_demote()
Quoted 23 lines · Chen Ridong
Quoted 7 lines · Bing Jiao
kernel/cgroup/cpuset.c +0
-7-void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) -{ -struct cgroup_subsys_state *css; -struct cpuset *cs; - -if (!cpuset_v2 ()) -return ;kernel/cgroup/cpuset.c bool cpuset_node_allowed(struct cgroup *cgroup, int nid) · line 4451
Quoted 10 lines
kernel/cgroup/cpuset.c +0
-7- - css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys); -if (!css) -return ; - -/* Follows the same assumption in cpuset_node_allowed() */ - cs = container_of(css, struct cpuset, css);nodes_and (*mask, *mask, cs->effective_mems);css_put (css); }Oh, I see you merged these two functions here.
Quoted 5 lines
However, I think
cpuset_get_mem_allowedwould be more versatile in general use.You can then check whether the returned nodemask intersects with your target mask. In the future, there may be scenarios where users simply want to retrieve the effective masks directly.
Hi Ridong, thank you for the suggestions.
I agree that returning a nodemask would provide greater versatility.
I think cpuset_get_mem_allowed_relax() would be a better name, since we do not need the locking and online mem guarantees compared to an similar function cpuset_mems_allowed().
Best,
Bing
Re: [PATCH v2 2/2] mm/vmscan: check all allowed targets in can_demote()
Quoted 30 lines · Bing Jiao
Quoted 23 lines · Chen Ridong
Quoted 7 lines · Bing Jiao
kernel/cgroup/cpuset.c +0
-7-void cpuset_node_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) -{ -struct cgroup_subsys_state *css; -struct cpuset *cs; - -if (!cpuset_v2 ()) -return ;kernel/cgroup/cpuset.c bool cpuset_node_allowed(struct cgroup *cgroup, int nid) · line 4451
Quoted 10 lines
kernel/cgroup/cpuset.c +0
-7- - css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys); -if (!css) -return ; - -/* Follows the same assumption in cpuset_node_allowed() */ - cs = container_of(css, struct cpuset, css);nodes_and (*mask, *mask, cs->effective_mems);css_put (css); }Oh, I see you merged these two functions here.
Quoted 5 lines
However, I think
cpuset_get_mem_allowedwould be more versatile in general use.You can then check whether the returned nodemask intersects with your target mask. In the future, there may be scenarios where users simply want to retrieve the effective masks directly.
Quoted 8 lines
Hi Ridong, thank you for the suggestions.
I agree that returning a nodemask would provide greater versatility.
I think
cpuset_get_mem_allowed_relax()would be a better name, since we do not need the locking and online mem guarantees compared to an similar functioncpuset_mems_allowed().
I think the key difference between cpuset_mems_allowed and the helper you intend to implement lies not in locking or online memory guarantees, but in the input parameter: you want to retrieve cpuset->effective_mems for a cgroup from another subsystem.
The cs->effective_mems should typically only include online nodes, except during brief transitional periods such as hotplug operations. Similarly, node migration logic also requires online nodes.
Therefore, cpuset_get_mem_allowed seems acceptable to me.
Additionally, you may consider calling guarantee_online_mems inside your new helper to ensure consistency.
Signature · 2 lines
Best regards, Ridong
PATCH v3 1/1 mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoFix two bugs in demote_folio_list() and can_demote() due to incorrect demotion target checks in reclaim/demotion.
Commit 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim") introduces the cpuset.mems_effective check and applies it to can_demote(). However:
1. It does not apply this check in demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.
2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set in mems_effective.
These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update cpuset_node_allowed() and mem_cgroup_node_allowed() to return effective_mems, allowing directly logic-and operation against demotion targets. Also update can_demote() and demote_folio_list() accordingly.
Reproduct Bug 1:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Should respect node 0-2 limit.
# Observation: Node 3 shows significant allocation (MemFree drops)
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Reproduct Bug 2:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Pages are demoted to Nodes 4-5
# Observation: No pages are demoted before oom.
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2--- include/linux/cpuset.h | 6 +++--- include/linux/memcontrol.h | 6 +++--- kernel/cgroup/cpuset.c | 16 ++++++++-------- mm/memcontrol.c | 6 ++++-- mm/vmscan.c | 35 +++++++++++++++++++++++------------ 5 files changed, 41 insertions(+), 28 deletions(-)
include/linux/cpuset.h +3 -3
@@ -174,7 +174,7 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern nodemask_t cpuset_node_get_allowed (struct cgroup *cgroup);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }
@@ -301,9 +301,9 @@ static inline bool read_mems_allowed_retry(unsigned int seq) return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline nodemask_t cpuset_node_get_allowed (struct cgroup *cgroup) { -return true; +return node_possible_map; }#endif /* !CONFIG_CPUSETS */
include/linux/memcontrol.h +3 -3
@@ -1740,7 +1740,7 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +nodemask_t mem_cgroup_node_get_allowed (struct mem_cgroup *memcg);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);
@@ -1811,9 +1811,9 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +static inline nodemask_t mem_cgroup_node_get_allowed (struct mem_cgroup *memcg) { -return true; +return node_possible_map; }static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
kernel/cgroup/cpuset.c +8 -8
@@ -4416,23 +4416,23 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)return allowed; } -bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +nodemask_t cpuset_node_get_allowed (struct cgroup *cgroup) { +nodemask_t nodes = node_possible_map;struct cgroup_subsys_state *css;struct cpuset *cs; -bool allowed;/* * In v1, mem_cgroup and cpuset are unlikely in the same hierarchy * and mems_allowed is likely to be empty even if we could get to it, -* so return true to avoid taking a global lock on the empty check. +* so return directly to avoid taking a global lock on the empty check. */ -if (!cpuset_v2 ()) -return true; +if (!cgroup || !cpuset_v2 ()) +return nodes; css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys);if (!css) -return true; +return nodes;/* * Normally, accessing effective_mems would require the cpuset_mutex
@@ -4447,9 +4447,9 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)* cannot make strong isolation guarantees, so this is acceptable. */ cs = container_of(css, struct cpuset, css); - allowed =node_isset (nid, cs->effective_mems); +nodes_copy (nodes, cs->effective_mems);css_put (css); -return allowed; +return nodes; }/**
mm/memcontrol.c +4 -2
@@ -5597,9 +5597,11 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +nodemask_t mem_cgroup_node_get_allowed (struct mem_cgroup *memcg) { -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +if (memcg) +return cpuset_node_get_allowed (memcg->css.cgroup); +return node_possible_map; }void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
mm/vmscan.c +23 -12
@@ -345,18 +345,24 @@ static bool can_demote(int nid, struct scan_control *sc, struct mem_cgroup *memcg) {int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask, allowed_mems; -if (!numa_demotion_enabled) +if (!pgdat || !numa_demotion_enabled)return false;if (sc && sc->no_demotion)return false; - demotion_nid =next_demotion_node (nid); -if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask)) +return false; + + allowed_mems =mem_cgroup_node_get_allowed (memcg); +nodes_and (allowed_mask, allowed_mask, allowed_mems); +if (nodes_empty (allowed_mask))return false; -/* If demotion node isn't in the cgroup's mems_allowed, fall back */ -if (mem_cgroup_node_allowed (memcg, demotion_nid)) { +for_each_node_mask (demotion_nid, allowed_mask) {int z;struct zone *zone;struct pglist_data *pgdat =NODE_DATA (demotion_nid);
@@ -1029,11 +1035,12 @@ static struct folio *alloc_demote_folio(struct folio *src,* Folios which are not demoted are left on @demote_folios. */ static unsigned int demote_folio_list (struct list_head *demote_folios, -struct pglist_data *pgdat) +struct pglist_data *pgdat, +struct mem_cgroup *memcg) {int target_nid =next_demotion_node (pgdat->node_id);unsigned int nr_succeeded; -nodemask_t allowed_mask; +nodemask_t allowed_mask, allowed_mems;struct migration_target_control mtc = {/*
@@ -1043,7 +1050,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,*/ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, .nmask = &allowed_mask, .reason = MR_DEMOTION, };
@@ -1051,10 +1057,15 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (list_empty (demote_folios))return 0 ; -if (target_nid ==NUMA_NO_NODE ) -return 0 ; -node_get_allowed_targets (pgdat, &allowed_mask); + allowed_mems =mem_cgroup_node_get_allowed (memcg); +nodes_and (allowed_mask, allowed_mask, allowed_mems); +if (nodes_empty (allowed_mask)) +return false; + +if (target_nid ==NUMA_NO_NODE || !node_isset (target_nid, allowed_mask)) + target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL,
@@ -1576,7 +1587,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,/* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_demoted =demote_folio_list (&demote_folios, pgdat); + nr_demoted =demote_folio_list (&demote_folios, pgdat, memcg); nr_reclaimed += nr_demoted; stat->nr_demoted += nr_demoted;/* Folios that could not be demoted are still in @demote_folios */
--
2.52.0.358.g0dd7633a29-goog
Re: [PATCH v3] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 3 lines · Bing Jiao
Fix two bugs in
demote_folio_list()andcan_demote()due to incorrect demotion target checks in reclaim/demotion.
Considering these bugs are introduced from one commit, I think it is better to fix it in one patch rather than split it.
v3: Rename cpuset_node_allowed() as cpuset_node_get_allowed() to
return effective_mems directly, providing better versatility (thanks Ridong for the suggestion).
Gregory Price
maintainerRe: [PATCH v3] mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoQuoted 5 lines · Bing Jiao
include/linux/cpuset.h +2
-2-static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline nodemask_t cpuset_node_get_allowed (struct cgroup *cgroup) { -return true; +return node_possible_map;
...
Quoted 6 lines
include/linux/memcontrol.h +2
-2-static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +static inline nodemask_t mem_cgroup_node_get_allowed (struct mem_cgroup *memcg) { -return true; +return node_possible_map; }
...
Quoted 4 lines
kernel/cgroup/cpuset.c +2
-1-bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +nodemask_t cpuset_node_get_allowed (struct cgroup *cgroup) { +nodemask_t nodes = node_possible_map;
...
mm/memcontrol.c subsys_initcall(mem_cgroup_swap_init); · line 5605
Quoted 8 lines
mm/memcontrol.c +4
-2-bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +nodemask_t mem_cgroup_node_get_allowed (struct mem_cgroup *memcg) { -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +if (memcg) +return cpuset_node_get_allowed (memcg->css.cgroup); +return node_possible_map; }
node_possible_map or node_states[N_MEMORY]?
The latter seems more appropriate to me since node_possible_map will include offline nodes.
kernel/cgroup/cpuset.c bool cpuset_node_allowed(struct cgroup *cgroup, int nid) · line 4453
Quoted 6 lines
kernel/cgroup/cpuset.c +2
-2- allowed =node_isset (nid, cs->effective_mems); +nodes_copy (nodes, cs->effective_mems);css_put (css); -return allowed; +return nodes; }
I saw in vmscan you check for returning an empty nodemask, may want to at least add a comment to the function definition that says this needs to be checked.
mm/vmscan.c static bool can_demote(int nid, struct scan_control *sc, · line 349
Quoted 10 lines
mm/vmscan.c +2
-0@@ -345,18 +345,24 @@ static bool can_demote(int nid, struct scan_control *sc, struct mem_cgroup *memcg) {int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask, allowed_mems;
Only other concern i have is the number of nodemasks being added to the stack. Should be <512 bytes, but I've run into situations where builds have screamed at me for adding one nodemask to the stack, let alone 3+.
Have you run this through klp?
~Gregory
Re: [PATCH v3] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 13 lines · Gregory Price
mm/memcontrol.c subsys_initcall(mem_cgroup_swap_init); · line 5605
Quoted 8 lines · Bing Jiao
mm/memcontrol.c +4
-2-bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +nodemask_t mem_cgroup_node_get_allowed (struct mem_cgroup *memcg) { -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +if (memcg) +return cpuset_node_get_allowed (memcg->css.cgroup); +return node_possible_map; }
node_possible_mapor node_states[N_MEMORY]?
Quoted 2 lines
The latter seems more appropriate to me since
node_possible_mapwill include offline nodes.
Yes, I agree that node_states[N_MEMORY] would be better.
Quoted 10 lines
kernel/cgroup/cpuset.c bool cpuset_node_allowed(struct cgroup *cgroup, int nid) · line 4453
Quoted 6 lines
kernel/cgroup/cpuset.c +2
-2- allowed =node_isset (nid, cs->effective_mems); +nodes_copy (nodes, cs->effective_mems);css_put (css); -return allowed; +return nodes; }I saw in vmscan you check for returning an empty nodemask, may want to at least add a comment to the function definition that says this needs to be checked.
Will add a comment to say that it may return an empty nodemask.
Quoted 6 lines
Quoted 6 lines
mm/vmscan.c +0
-0@@ -345,18 +345,24 @@ static bool can_demote(int nid, struct scan_control *sc, struct mem_cgroup *memcg)
Quoted 8 lines
mm/vmscan.c static bool can_demote(int nid, struct scan_control *sc, · line 349
Quoted 4 lines
mm/vmscan.c +2
-0{int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask, allowed_mems;Only other concern i have is the number of nodemasks being added to the stack. Should be <512 bytes, but I've run into situations where builds have screamed at me for adding one nodemask to the stack, let alone 3+.
While having 3+ nodemasks presents a risk, utilizing two nodemasks should be acceptable. Given that the maximum number of nodes is 1024 (2^10), two nodemasks would require 256 bytes, which should be okay.
Otherwise, we can keep to use mem_node_filter_allowed(). Only update it to return a nodemask when subsequent features need.
Quoted 1 line
Have you run this through klp?
I have not run it through klp. Will do it then.
Thanks,
Bing
Re: [PATCH v3] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 7 lines · Bing Jiao
Quoted 1 line · Gregory Price
Have you run this through klp?
I have not run it through klp. Will do it then.
Thanks,
Bing
Hi Gregory,
I tried to build a klp against kernel v6.16, but it ran into a segfault during create-diff-object using kpatch-build v0.9.11.
I do not think it was caused by this patch. But I will try it again at a later time.
Best,
Bing
Re: [PATCH v3] mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoQuoted 123 lines · Bing Jiao
Fix two bugs in
demote_folio_list()andcan_demote()due to incorrect demotion target checks in reclaim/demotion.Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote(). However:1. It does not apply this check in
demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in
can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set inmems_effective.These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update
cpuset_node_allowed()andmem_cgroup_node_allowed()to returneffective_mems, allowing directly logic-and operation against demotion targets. Also updatecan_demote()anddemote_folio_list()accordingly.Reproduct Bug 1:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Should respect node 0-2 limit. # Observation: Node 3 shows significant allocation (MemFree drops) stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Reproduct Bug 2:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Pages are demoted to Nodes 4-5 # Observation: No pages are demoted before oom. stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2Fixes 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim")Cc <[email protected]>Signed-off-by Bing Jiao <[email protected]>--- include/linux/cpuset.h | 6 +++--- include/linux/memcontrol.h | 6 +++--- kernel/cgroup/cpuset.c | 16 ++++++++-------- mm/memcontrol.c | 6 ++++-- mm/vmscan.c | 35 +++++++++++++++++++++++------------ 5 files changed, 41 insertions(+), 28 deletions(-)include/linux/cpuset.h +3
-3@@ -174,7 +174,7 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern nodemask_t cpuset_node_get_allowed (struct cgroup *cgroup);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }@@ -301,9 +301,9 @@ static inline bool read_mems_allowed_retry(unsigned int seq) return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline nodemask_t cpuset_node_get_allowed (struct cgroup *cgroup) { -return true; +return node_possible_map; }#endif /* !CONFIG_CPUSETS */ include/linux/memcontrol.h +3
-3@@ -1740,7 +1740,7 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +nodemask_t mem_cgroup_node_get_allowed (struct mem_cgroup *memcg);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);@@ -1811,9 +1811,9 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +static inline nodemask_t mem_cgroup_node_get_allowed (struct mem_cgroup *memcg) { -return true; +return node_possible_map; }static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)kernel/cgroup/cpuset.c +0
-0
kernel/cgroup/cpuset.c bool cpuset_current_node_allowed(int node, gfp_t gfp_mask) · line 4420
Quoted 6 lines
kernel/cgroup/cpuset.c +1
-1@@ -4416,23 +4416,23 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)return allowed; } -bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +nodemask_t cpuset_node_get_allowed (struct cgroup *cgroup) {
Could we define it as:
void cpuset_node_get_allowed(struct cgroup *cgroup, nodemask_t *node)
to align with the naming style of node_get_allowed_targets?
Quoted 40 lines
kernel/cgroup/cpuset.c +7
-7+nodemask_t nodes = node_possible_map;struct cgroup_subsys_state *css;struct cpuset *cs; -bool allowed;/* * In v1, mem_cgroup and cpuset are unlikely in the same hierarchy * and mems_allowed is likely to be empty even if we could get to it, -* so return true to avoid taking a global lock on the empty check. +* so return directly to avoid taking a global lock on the empty check. */ -if (!cpuset_v2 ()) -return true; +if (!cgroup || !cpuset_v2 ()) +return nodes; css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys);if (!css) -return true; +return nodes;/* * Normally, accessing effective_mems would require the cpuset_mutex @@ -4447,9 +4447,9 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)* cannot make strong isolation guarantees, so this is acceptable. */ cs = container_of(css, struct cpuset, css); - allowed =node_isset (nid, cs->effective_mems); +nodes_copy (nodes, cs->effective_mems);css_put (css); -return allowed; +return nodes; }/** mm/memcontrol.c +0
-0
mm/memcontrol.c subsys_initcall(mem_cgroup_swap_init); · line 5600
Quoted 5 lines
mm/memcontrol.c +1
-1@@ -5597,9 +5597,11 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +nodemask_t mem_cgroup_node_get_allowed (struct mem_cgroup *memcg)
void mem_cgroup_node_get_allowed(struct mem_cgroup *memcg, nodemask_t *node)
Quoted 26 lines
mm/memcontrol.c +3
-1{ -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +if (memcg) +return cpuset_node_get_allowed (memcg->css.cgroup); +return node_possible_map; }void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)mm/vmscan.c +3
-2@@ -345,18 +345,24 @@ static bool can_demote(int nid, struct scan_control *sc, struct mem_cgroup *memcg) {int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask, allowed_mems; -if (!numa_demotion_enabled) +if (!pgdat || !numa_demotion_enabled)return false;if (sc && sc->no_demotion)return false; - demotion_nid =next_demotion_node (nid);
mm/vmscan.c static bool can_demote(int nid, struct scan_control *sc, · line 364
Quoted 10 lines
mm/vmscan.c +7
-1-if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask)) +return false; + + allowed_mems =mem_cgroup_node_get_allowed (memcg); +nodes_and (allowed_mask, allowed_mask, allowed_mems); +if (nodes_empty (allowed_mask))return false;
node_get_allowed_targets(pgdat, &allowed_mask); mem_cgroup_node_get_allowed(memcg, allowed_mems); if (!nodes_intersects(allowed_mask, allowed_mems)) return false; Would it look better?
Quoted 59 lines
mm/vmscan.c +13
-10-/* If demotion node isn't in the cgroup's mems_allowed, fall back */ -if (mem_cgroup_node_allowed (memcg, demotion_nid)) { +for_each_node_mask (demotion_nid, allowed_mask) {int z;struct zone *zone;struct pglist_data *pgdat =NODE_DATA (demotion_nid);@@ -1029,11 +1035,12 @@ static struct folio *alloc_demote_folio(struct folio *src,* Folios which are not demoted are left on @demote_folios. */ static unsigned int demote_folio_list (struct list_head *demote_folios, -struct pglist_data *pgdat) +struct pglist_data *pgdat, +struct mem_cgroup *memcg) {int target_nid =next_demotion_node (pgdat->node_id);unsigned int nr_succeeded; -nodemask_t allowed_mask; +nodemask_t allowed_mask, allowed_mems;struct migration_target_control mtc = {/* @@ -1043,7 +1050,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,*/ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, .nmask = &allowed_mask, .reason = MR_DEMOTION, };@@ -1051,10 +1057,15 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (list_empty (demote_folios))return 0 ; -if (target_nid ==NUMA_NO_NODE ) -return 0 ; -node_get_allowed_targets (pgdat, &allowed_mask); + allowed_mems =mem_cgroup_node_get_allowed (memcg); +nodes_and (allowed_mask, allowed_mask, allowed_mems); +if (nodes_empty (allowed_mask)) +return false; + +if (target_nid ==NUMA_NO_NODE || !node_isset (target_nid, allowed_mask)) + target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL,@@ -1576,7 +1587,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,/* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_demoted =demote_folio_list (&demote_folios, pgdat); + nr_demoted =demote_folio_list (&demote_folios, pgdat, memcg); nr_reclaimed += nr_demoted; stat->nr_demoted += nr_demoted;/* Folios that could not be demoted are still in @demote_folios */ --2.52.0.358.g0dd7633a29-goog
Signature · 2 lines
Best regards, Ridong
Re: [PATCH v3] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 5 lines · Chen Ridong
Quoted 2 lines
+nodemask_t
cpuset_node_get_allowed(struct cgroup *cgroup)
{Could we define it as:
Quoted 8 lines
void
cpuset_node_get_allowed(struct cgroup *cgroup, nodemask_t *node)to align with the naming style of
node_get_allowed_targets?mm/memcontrol.c subsys_initcall(mem_cgroup_swap_init); · line 5600
Quoted 2 lines
mm/memcontrol.c +1
-1-bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +nodemask_t mem_cgroup_node_get_allowed (struct mem_cgroup *memcg)void
mem_cgroup_node_get_allowed(struct mem_cgroup *memcg, nodemask_t *node)
Thanks for the suggestion. Pass a pointer is better.
Also, Gregory mentioned that the stack size may be an issue if systems have many nodes.
Do you think it is better to use mem_cgroup_node_filter_allowed() to keep the stack size smaller?
Quoted 5 lines
mm/vmscan.c static bool can_demote(int nid, struct scan_control *sc, · line 358
Quoted 5 lines
mm/vmscan.c +3
-2- demotion_nid =next_demotion_node (nid); -if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask)) +return false;
This is a fast-fail path. When the queried node is the farthest node, allowed_mask will be empty. Thus, I would like to keep this check before mem_cgroup_node_get_allowed().
Quoted 4 lines
Quoted 4 lines
mm/vmscan.c +4
-0+ + allowed_mems =mem_cgroup_node_get_allowed (memcg); +nodes_and (allowed_mask, allowed_mask, allowed_mems); +if (nodes_empty (allowed_mask))
Quoted 8 lines
Quoted 2 lines
return false;
node_get_allowed_targets(pgdat, &allowed_mask); mem_cgroup_node_get_allowed(memcg, allowed_mems); if (!nodes_intersects(allowed_mask, allowed_mems)) return false; Would it look better?
Yes, nodes_intersects() is better than logic-and.
Will update in v3.
Best,
Bing
Re: [PATCH v3] mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoQuoted 21 lines · Bing Jiao
Fix two bugs in
demote_folio_list()andcan_demote()due to incorrect demotion target checks in reclaim/demotion.Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote(). However:1. It does not apply this check in
demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in
can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set inmems_effective.These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
Quoted 6 lines
To address these bugs, update
cpuset_node_allowed()andmem_cgroup_node_allowed()to returneffective_mems, allowing directly logic-and operation against demotion targets. Also updatecan_demote()anddemote_folio_list()accordingly.Reproduct Bug 1:
"Reproduct" is not an English word. Use either "Reproduce" or "Reproduction".
Cheers,
Longman
Re: [PATCH v3] mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoQuoted 82 lines · Bing Jiao
Fix two bugs in
demote_folio_list()andcan_demote()due to incorrect demotion target checks in reclaim/demotion.Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote(). However:1. It does not apply this check in
demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in
can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set inmems_effective.These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update
cpuset_node_allowed()andmem_cgroup_node_allowed()to returneffective_mems, allowing directly logic-and operation against demotion targets. Also updatecan_demote()anddemote_folio_list()accordingly.Reproduct Bug 1:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Should respect node 0-2 limit. # Observation: Node 3 shows significant allocation (MemFree drops) stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Reproduct Bug 2:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Pages are demoted to Nodes 4-5 # Observation: No pages are demoted before oom. stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2Fixes 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim")Cc <[email protected]>Signed-off-by Bing Jiao <[email protected]>--- include/linux/cpuset.h | 6 +++--- include/linux/memcontrol.h | 6 +++--- kernel/cgroup/cpuset.c | 16 ++++++++-------- mm/memcontrol.c | 6 ++++-- mm/vmscan.c | 35 +++++++++++++++++++++++------------ 5 files changed, 41 insertions(+), 28 deletions(-)include/linux/cpuset.h +1
-1@@ -174,7 +174,7 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern nodemask_t cpuset_node_get_allowed (struct cgroup *cgroup);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }
include/linux/cpuset.h static inline bool read_mems_allowed_retry(unsigned int seq) · line 307
Quoted 9 lines
include/linux/cpuset.h +2
-2@@ -301,9 +301,9 @@ static inline bool read_mems_allowed_retry(unsigned int seq)return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline nodemask_t cpuset_node_get_allowed (struct cgroup *cgroup) { -return true; +return node_possible_map; }
The nodemask_t type can be large depending on the setting of CONFIG_NODES_SHIFT. Passing a large data structure on stack may not be a good idea. You can return a pointer to nodemask_t instead. In that case, you will have a add a "const" qualifier to the return type to make sure that the node mask won't get accidentally modified. Alternatively, you can pass a nodemask_t pointer as an output parameter and copy out the nodemask_t data.
The name "cpuset_node_get_allowed" doesn't fit the cpuset naming convention. There is a "cpuset_mems_allowed(struct task_struct *)" to return "mems_allowed" of a task. This new helper is for returning the mems_allowed defined in the cpuset. Perhaps we could just use "cpuset_nodes_allowed(struct cgroup *)".
Cheers,
Longman
Re: [PATCH v3] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 7 lines · Waiman Long
The
nodemask_ttype can be large depending on the setting ofCONFIG_NODES_SHIFT. Passing a large data structure on stack may not be a good idea. You can return a pointer tonodemask_tinstead. In that case, you will have a add a "const" qualifier to the return type to make sure that the node mask won't get accidentally modified. Alternatively, you can pass anodemask_tpointer as an output parameter and copy out thenodemask_tdata.
Quoted 8 lines
The name "
cpuset_node_get_allowed" doesn't fit the cpuset naming convention. There is a "cpuset_mems_allowed(struct task_struct *)" to return "mems_allowed" of a task. This new helper is for returning themems_alloweddefined in the cpuset. Perhaps we could just use "cpuset_nodes_allowed(struct cgroup *)".Cheers,
Longman
Thank you for the explanation and suggestion.
I have updated v4, which updates the functions to filter out nodes rather than returning mems_allowed. In v3, the caller need to declare a temp nodemask_t to store mems_allowed and then intersect with lower-tier nodemask, which is unnecessarily increase the stack size.
The function name is updated as "cpuset_nodes_filter_allowed". Do you think it is still better to use "cpuset_nodes_allowed" when doing the filtering thing?
Thanks,
Bing
PATCH v4 1/1 mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoFix two bugs in demote_folio_list() and can_demote() due to incorrect demotion target checks in reclaim/demotion.
Commit 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim") introduces the cpuset.mems_effective check and applies it to can_demote(). However:
1. It does not apply this check in demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.
2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set in mems_effective.
These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update cpuset_node_allowed() and mem_cgroup_node_allowed() to return effective_mems, allowing directly logic-and operation against demotion targets. Also update can_demote() and demote_folio_list() accordingly.
Bug 1 reproduction:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Should respect node 0-2 limit.
# Observation: Node 3 shows significant allocation (MemFree drops)
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Bug 2 reproduction:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Pages are demoted to Nodes 4-5
# Observation: No pages are demoted before oom.
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2---
v3 -> v4:
Update functions to filter out nodes that are not in cgroup's mems_allowed rather than returning mems_allowed directly. It minimizes stack usage and remains versatile enough to return mems_allowed when all possible nodes are set in the passed nodemask_t pointer.
---
include/linux/cpuset.h | 7 ++++--- include/linux/memcontrol.h | 6 +++--- kernel/cgroup/cpuset.c | 24 +++++++++++++++--------- mm/memcontrol.c | 5 +++-- mm/vmscan.c | 32 +++++++++++++++++++++----------- 5 files changed, 46 insertions(+), 28 deletions(-)
include/linux/cpuset.h +4 -3
@@ -174,7 +174,8 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern void cpuset_nodes_filter_allowed (struct cgroup *cgroup, +nodemask_t *mask);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }
@@ -301,9 +302,9 @@ static inline bool read_mems_allowed_retry(unsigned int seq) return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline void cpuset_nodes_filter_allowed (struct cgroup *cgroup, +nodemask_t *mask) { -return true; }#endif /* !CONFIG_CPUSETS */
include/linux/memcontrol.h +3 -3
@@ -1740,7 +1740,7 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);
@@ -1811,9 +1811,9 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +static inline void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg, +nodemask_t *mask) { -return true; }static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
kernel/cgroup/cpuset.c +15 -9
@@ -4416,27 +4416,34 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)return allowed; } -bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +/** +* cpuset_nodes_filter_allowed - filter out nodes not in cgroup's mems_allowed. +* @cgroup: pointer to struct cgroup. +* @mask: pointer to struct nodemask_t to be filtered. +* +* Description: Filters out nodes that are not in mems_allowed for @cgroup. +* Nodes returned in @mask are not guaranteed to be online. +**/ +void cpuset_nodes_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) {struct cgroup_subsys_state *css;struct cpuset *cs; -bool allowed;/* * In v1, mem_cgroup and cpuset are unlikely in the same hierarchy * and mems_allowed is likely to be empty even if we could get to it, -* so return true to avoid taking a global lock on the empty check. +* so return directly to avoid taking a global lock on the empty check. */ -if (!cpuset_v2 ()) -return true; +if (!cgroup || !cpuset_v2 ()) +return ; css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys);if (!css) -return true; +return ;/* * Normally, accessing effective_mems would require the cpuset_mutex -* or callback_lock - but node_isset is atomic and the reference +* or callback_lock - but not doing so is acceptable and the reference * taken via cgroup_get_e_css is sufficient to protect css. * * Since this interface is intended for use by migration paths, we
@@ -4447,9 +4454,8 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)* cannot make strong isolation guarantees, so this is acceptable. */ cs = container_of(css, struct cpuset, css); - allowed =node_isset (nid, cs->effective_mems); +nodes_and (*mask, *mask, cs->effective_mems);css_put (css); -return allowed; }/**
mm/memcontrol.c +3 -2
@@ -5597,9 +5597,10 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask) { -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +if (memcg) +cpuset_nodes_filter_allowed (memcg->css.cgroup, mask); }void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
mm/vmscan.c +21 -11
@@ -345,18 +345,24 @@ static bool can_demote(int nid, struct scan_control *sc, struct mem_cgroup *memcg) {int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask; -if (!numa_demotion_enabled) +if (!pgdat || !numa_demotion_enabled)return false;if (sc && sc->no_demotion)return false; - demotion_nid =next_demotion_node (nid); -if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask)) +return false; + +/* Filter out nodes that are not in cgroup's mems_allowed. */ +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +if (nodes_empty (allowed_mask))return false; -/* If demotion node isn't in the cgroup's mems_allowed, fall back */ -if (mem_cgroup_node_allowed (memcg, demotion_nid)) { +for_each_node_mask (demotion_nid, allowed_mask) {int z;struct zone *zone;struct pglist_data *pgdat =NODE_DATA (demotion_nid);
@@ -1029,7 +1035,8 @@ static struct folio *alloc_demote_folio(struct folio *src,* Folios which are not demoted are left on @demote_folios. */ static unsigned int demote_folio_list (struct list_head *demote_folios, -struct pglist_data *pgdat) +struct pglist_data *pgdat, +struct mem_cgroup *memcg) {int target_nid =next_demotion_node (pgdat->node_id);unsigned int nr_succeeded;
@@ -1043,7 +1050,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,*/ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, .nmask = &allowed_mask, .reason = MR_DEMOTION, };
@@ -1051,10 +1057,14 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (list_empty (demote_folios))return 0 ; -if (target_nid ==NUMA_NO_NODE ) -return 0 ; -node_get_allowed_targets (pgdat, &allowed_mask); +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +if (nodes_empty (allowed_mask)) +return false; + +if (!node_isset (target_nid, allowed_mask)) + target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL,
@@ -1576,7 +1586,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,/* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_demoted =demote_folio_list (&demote_folios, pgdat); + nr_demoted =demote_folio_list (&demote_folios, pgdat, memcg); nr_reclaimed += nr_demoted; stat->nr_demoted += nr_demoted;/* Folios that could not be demoted are still in @demote_folios */
--
2.52.0.358.g0dd7633a29-goog
Andrew Morton
maintainerRe: [PATCH v4] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 2 lines · On Sun, 4 Jan 2026 08:54:05 +0000 Bing Jiao <[email protected]> wrote:
Fix two bugs in
demote_folio_list()andcan_demote()due to incorrect demotion target checks in reclaim/demotion.
Thanks.
Quoted 54 lines
Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote(). However:1. It does not apply this check in
demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in
can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set inmems_effective.These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update
cpuset_node_allowed()andmem_cgroup_node_allowed()to returneffective_mems, allowing directly logic-and operation against demotion targets. Also updatecan_demote()anddemote_folio_list()accordingly.Bug 1 reproduction:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Should respect node 0-2 limit. # Observation: Node 3 shows significant allocation (MemFree drops) stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Bug 2 reproduction:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Pages are demoted to Nodes 4-5 # Observation: No pages are demoted before oom. stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2
Quoted 2 lines
Fixes 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim")Cc <[email protected]>
We'll want to fix these things in 6.16.X and later, but you've prepared this patch against "mm/vmscan: don't demote if there is not enough free memory in the lower memory tier", which is presently under test/review in mm.git's mm-unstable branch.
This seems to be incorrect ordering - this fix should go ahead of Akinobu Mita's series "mm: fix oom-killer not being invoked when demotion is enabled v2".
So can you please redo this patch against current mainline? And please also review the "mm: fix oom-killer not being invoked when demotion is enabled" series to ensure that things will work together nicely when that time comes.
Re: [PATCH v4] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 5 lines · Andrew Morton
We'll want to fix these things in 6.16.X and later, but you've prepared this patch against "mm/vmscan: don't demote if there is not enough free memory in the lower memory tier", which is presently under test/review in mm.git's mm-unstable branch.
Quoted 8 lines
This seems to be incorrect ordering - this fix should go ahead of Akinobu Mita's series "mm: fix oom-killer not being invoked when demotion is enabled v2".
So can you please redo this patch against current mainline? And please also review the "mm: fix oom-killer not being invoked when demotion is enabled" series to ensure that things will work together nicely when that time comes.
Hi Andrew,
Thank you for the detailed explanation. Patch v4 has been sent.
V4 is patched against the mainline. Tested on both the mainline and mm-everything (after Akinobu Mita's series "mm: fix oom-killer not being invoked when demotion is enabled v2"), and passed.
Best,
Bing
Re: [PATCH v4] mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoQuoted 175 lines · Bing Jiao
Fix two bugs in
demote_folio_list()andcan_demote()due to incorrect demotion target checks in reclaim/demotion.Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote(). However:1. It does not apply this check in
demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in
can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set inmems_effective.These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update
cpuset_node_allowed()andmem_cgroup_node_allowed()to returneffective_mems, allowing directly logic-and operation against demotion targets. Also updatecan_demote()anddemote_folio_list()accordingly.Bug 1 reproduction:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Should respect node 0-2 limit. # Observation: Node 3 shows significant allocation (MemFree drops) stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Bug 2 reproduction:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Pages are demoted to Nodes 4-5 # Observation: No pages are demoted before oom. stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2Fixes 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim")Cc <[email protected]>Signed-off-by Bing Jiao <[email protected]>---
v3 -> v4:
Update functions to filter out nodes that are not in cgroup'smems_allowedrather than returningmems_alloweddirectly. It minimizes stack usage and remains versatile enough to returnmems_allowedwhen all possible nodes are set in the passednodemask_tpointer.
---include/linux/cpuset.h | 7 ++++--- include/linux/memcontrol.h | 6 +++--- kernel/cgroup/cpuset.c | 24 +++++++++++++++--------- mm/memcontrol.c | 5 +++-- mm/vmscan.c | 32 +++++++++++++++++++++----------- 5 files changed, 46 insertions(+), 28 deletions(-)include/linux/cpuset.h +4
-3@@ -174,7 +174,8 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern void cpuset_nodes_filter_allowed (struct cgroup *cgroup, +nodemask_t *mask);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }@@ -301,9 +302,9 @@ static inline bool read_mems_allowed_retry(unsigned int seq) return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline void cpuset_nodes_filter_allowed (struct cgroup *cgroup, +nodemask_t *mask) { -return true; }#endif /* !CONFIG_CPUSETS */ include/linux/memcontrol.h +3
-3@@ -1740,7 +1740,7 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);@@ -1811,9 +1811,9 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +static inline void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg, +nodemask_t *mask) { -return true; }static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)kernel/cgroup/cpuset.c +14
-7@@ -4416,27 +4416,34 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)return allowed; } -bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +/** +* cpuset_nodes_filter_allowed - filter out nodes not in cgroup's mems_allowed. +* @cgroup: pointer to struct cgroup. +* @mask: pointer to struct nodemask_t to be filtered. +* +* Description: Filters out nodes that are not in mems_allowed for @cgroup. +* Nodes returned in @mask are not guaranteed to be online. +**/ +void cpuset_nodes_filter_allowed (struct cgroup *cgroup,nodemask_t *mask) {struct cgroup_subsys_state *css;struct cpuset *cs; -bool allowed;/* * In v1, mem_cgroup and cpuset are unlikely in the same hierarchy * and mems_allowed is likely to be empty even if we could get to it, -* so return true to avoid taking a global lock on the empty check. +* so return directly to avoid taking a global lock on the empty check. */ -if (!cpuset_v2 ()) -return true; +if (!cgroup || !cpuset_v2 ()) +return ; css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys);if (!css) -return true; +return ;/* * Normally, accessing effective_mems would require the cpuset_mutex -* or callback_lock - but node_isset is atomic and the reference +* or callback_lock - but not doing so is acceptable and the reference * taken via cgroup_get_e_css is sufficient to protect css. * * Since this interface is intended for use by migration paths, we
kernel/cgroup/cpuset.c bool cpuset_node_allowed(struct cgroup *cgroup, int nid) · line 4457
Quoted 5 lines
kernel/cgroup/cpuset.c +1
-1@@ -4447,9 +4454,8 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)* cannot make strong isolation guarantees, so this is acceptable. */ cs = container_of(css, struct cpuset, css); - allowed =node_isset (nid, cs->effective_mems); +nodes_and (*mask, *mask, cs->effective_mems);
Why do we need the and operation? Can't we just copy cs->effective_mems to mask directly?
Per Longman's suggestion, name it cpuset_nodes_allowed and handle the filtering in mem_cgroup_node_filter_allowed. Please keep the allowed nodes retrieval logic common.
Quoted 103 lines
kernel/cgroup/cpuset.c +0
-1css_put (css); -return allowed; }/** mm/memcontrol.c +3
-2@@ -5597,9 +5597,10 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask) { -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +if (memcg) +cpuset_nodes_filter_allowed (memcg->css.cgroup, mask); }void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)mm/vmscan.c +21
-12@@ -345,18 +345,24 @@ static bool can_demote(int nid, struct scan_control *sc, struct mem_cgroup *memcg) {int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask; -if (!numa_demotion_enabled) +if (!pgdat || !numa_demotion_enabled)return false;if (sc && sc->no_demotion)return false; - demotion_nid =next_demotion_node (nid); -if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask)) +return false; + +/* Filter out nodes that are not in cgroup's mems_allowed. */ +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +if (nodes_empty (allowed_mask))return false; -/* If demotion node isn't in the cgroup's mems_allowed, fall back */ -if (mem_cgroup_node_allowed (memcg, demotion_nid)) { +for_each_node_mask (demotion_nid, allowed_mask) {int z;struct zone *zone;struct pglist_data *pgdat =NODE_DATA (demotion_nid);@@ -1029,7 +1035,8 @@ static struct folio *alloc_demote_folio(struct folio *src,* Folios which are not demoted are left on @demote_folios. */ static unsigned int demote_folio_list (struct list_head *demote_folios, -struct pglist_data *pgdat) +struct pglist_data *pgdat, +struct mem_cgroup *memcg) {int target_nid =next_demotion_node (pgdat->node_id);unsigned int nr_succeeded;@@ -1043,7 +1050,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,*/ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, .nmask = &allowed_mask, .reason = MR_DEMOTION, };@@ -1051,10 +1057,14 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (list_empty (demote_folios))return 0 ; -if (target_nid ==NUMA_NO_NODE ) -return 0 ; -node_get_allowed_targets (pgdat, &allowed_mask); +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +if (nodes_empty (allowed_mask)) +return false; + +if (!node_isset (target_nid, allowed_mask)) + target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL,@@ -1576,7 +1586,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,/* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_demoted =demote_folio_list (&demote_folios, pgdat); + nr_demoted =demote_folio_list (&demote_folios, pgdat, memcg); nr_reclaimed += nr_demoted; stat->nr_demoted += nr_demoted;/* Folios that could not be demoted are still in @demote_folios */ --2.52.0.358.g0dd7633a29-goog
Signature · 2 lines
Best regards, Ridong
Re: [PATCH v4] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 11 lines · Chen Ridong
kernel/cgroup/cpuset.c bool cpuset_node_allowed(struct cgroup *cgroup, int nid) · line 4457
Quoted 3 lines
kernel/cgroup/cpuset.c +1
-1cs = container_of(css, struct cpuset, css); - allowed =node_isset (nid, cs->effective_mems); +nodes_and (*mask, *mask, cs->effective_mems);Why do we need the and operation? Can't we just copy
cs->effective_memsto mask directly?Per Longman's suggestion, name it
cpuset_nodes_allowedand handle the filtering inmem_cgroup_node_filter_allowed. Please keep the allowed nodes retrieval logic common.Best regards,
Ridong
Thank you for the explanation and suggestions.
Patch v5 has been sent with the corresponding updates.
Best,
Bing
PATCH v5 1/1 mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoFix two bugs in demote_folio_list() and can_demote() due to incorrect demotion target checks in reclaim/demotion.
Commit 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim") introduces the cpuset.mems_effective check and applies it to can_demote(). However:
1. It does not apply this check in demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.
2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set in mems_effective.
These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update cpuset_node_allowed() and mem_cgroup_node_allowed() to return effective_mems, allowing directly logic-and operation against demotion targets. Also update can_demote() and demote_folio_list() accordingly.
Bug 1 reproduction:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Should respect node 0-2 limit.
# Observation: Node 3 shows significant allocation (MemFree drops)
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Bug 2 reproduction:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Pages are demoted to Nodes 4-5
# Observation: No pages are demoted before oom.
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2---
Patch against the linux mainline.
Tested on the mainline and passed. Pages can be demoted to correct nodes on a VM with emulated 3-tiers far memory nodes.
Tested on mm-everyting, after Akinobu Mita's series "mm: fix oom-killer not being invoked when demotion is enabled v2", and passed. OOM can be triggered properly when far nodes are oom.
--- include/linux/cpuset.h | 6 +++--- include/linux/memcontrol.h | 6 +++--- kernel/cgroup/cpuset.c | 33 +++++++++++++++++++++++---------- mm/memcontrol.c | 10 ++++++++-- mm/vmscan.c | 30 ++++++++++++++++++------------ 5 files changed, 55 insertions(+), 30 deletions(-)
include/linux/cpuset.h +3 -3
@@ -174,7 +174,7 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }
@@ -301,9 +301,9 @@ static inline bool read_mems_allowed_retry(unsigned int seq) return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask) { -return true; +nodes_copy (*mask, node_states[N_MEMORY ]); }#endif /* !CONFIG_CPUSETS */
include/linux/memcontrol.h +3 -3
@@ -1744,7 +1744,7 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);
@@ -1815,9 +1815,9 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +static inline void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg, +nodemask_t *mask) { -return true; }static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
kernel/cgroup/cpuset.c +23 -10
@@ -4427,27 +4427,41 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)return allowed; } -bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +/** +* cpuset_nodes_allowed - return mems_allowed mask from a cgroup cpuset. +* @cgroup: pointer to struct cgroup. +* @mask: pointer to struct nodemask_t to be returned. +* +* Returns mems_allowed mask from a cgroup cpuset if it is cgroup v2 and +* has cpuset subsys. Otherwise, returns node_states[N_MEMORY]. +* +* Returned @mask may be empty, and nodes in @mask are not guaranteed +* to be online. +**/ +void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask) {struct cgroup_subsys_state *css;struct cpuset *cs; -bool allowed;/* * In v1, mem_cgroup and cpuset are unlikely in the same hierarchy * and mems_allowed is likely to be empty even if we could get to it, -* so return true to avoid taking a global lock on the empty check. +* so return directly to avoid taking a global lock on the empty check. */ -if (!cpuset_v2 ()) -return true; +if (!cgroup || !cpuset_v2 ()) { +nodes_copy (*mask, node_states[N_MEMORY ]); +return ; + } css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys); -if (!css) -return true; +if (!css) { +nodes_copy (*mask, node_states[N_MEMORY ]); +return ; + }/* * Normally, accessing effective_mems would require the cpuset_mutex -* or callback_lock - but node_isset is atomic and the reference +* or callback_lock - but not doing so is acceptable and the reference * taken via cgroup_get_e_css is sufficient to protect css. * * Since this interface is intended for use by migration paths, we
@@ -4458,9 +4472,8 @@ bool cpuset_node_allowed(struct cgroup *cgroup, int nid)* cannot make strong isolation guarantees, so this is acceptable. */ cs = container_of(css, struct cpuset, css); - allowed =node_isset (nid, cs->effective_mems); +nodes_copy (*mask, cs->effective_mems);css_put (css); -return allowed; }/**
mm/memcontrol.c +8 -2
@@ -5624,9 +5624,15 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask) { -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +nodemask_t allowed; + +if (!memcg) +return ; + +cpuset_nodes_allowed (memcg->css.cgroup, &allowed); +nodes_and (*mask, *mask, allowed); }void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
mm/vmscan.c +18 -12
@@ -344,19 +344,21 @@ static void flush_reclaim_state(struct scan_control *sc)static bool can_demote (int nid,struct scan_control *sc,struct mem_cgroup *memcg) { -int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask; -if (!numa_demotion_enabled) +if (!pgdat || !numa_demotion_enabled)return false;if (sc && sc->no_demotion)return false; - demotion_nid =next_demotion_node (nid); -if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask))return false; -/* If demotion node isn't in the cgroup's mems_allowed, fall back */ -return mem_cgroup_node_allowed (memcg, demotion_nid); +/* Filter out nodes that are not in cgroup's mems_allowed. */ +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +return !nodes_empty (allowed_mask); } static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
@@ -1019,7 +1021,8 @@ static struct folio *alloc_demote_folio(struct folio *src,* Folios which are not demoted are left on @demote_folios. */ static unsigned int demote_folio_list (struct list_head *demote_folios, -struct pglist_data *pgdat) +struct pglist_data *pgdat, +struct mem_cgroup *memcg) {int target_nid =next_demotion_node (pgdat->node_id);unsigned int nr_succeeded;
@@ -1033,7 +1036,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,*/ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, .nmask = &allowed_mask, .reason = MR_DEMOTION, };
@@ -1041,10 +1043,14 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (list_empty (demote_folios))return 0 ; -if (target_nid ==NUMA_NO_NODE ) -return 0 ; -node_get_allowed_targets (pgdat, &allowed_mask); +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +if (nodes_empty (allowed_mask)) +return false; + +if (!node_isset (target_nid, allowed_mask)) + target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL,
@@ -1566,7 +1572,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,/* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_demoted =demote_folio_list (&demote_folios, pgdat); + nr_demoted =demote_folio_list (&demote_folios, pgdat, memcg); nr_reclaimed += nr_demoted; stat->nr_demoted += nr_demoted;/* Folios that could not be demoted are still in @demote_folios */
--
2.52.0.358.g0dd7633a29-goog
Gregory Price
maintainerRe: [PATCH v5] mm/vmscan: fix demotion targets checks in reclaim/demotion
On Mon, Jan 05, 2026 at 05:01:52AM +0000, Bing Jiao wrote:
... snip ...
Quoted 14 lines
kernel/cgroup/cpuset.c +13
-0+/** +* cpuset_nodes_allowed - return mems_allowed mask from a cgroup cpuset. +* @cgroup: pointer to struct cgroup. +* @mask: pointer to struct nodemask_t to be returned. +* +* Returns mems_allowed mask from a cgroup cpuset if it is cgroup v2 and +* has cpuset subsys. Otherwise, returns node_states[N_MEMORY]. +* +* Returned @mask may be empty, and nodes in @mask are not guaranteed +* to be online. +**/ +void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask) +void cpuset_nodes_allowed(struct cgroup *cgroup, nodemask_t *mask) {
... snip ...
kernel/cgroup/cpuset.c bool cpuset_current_node_allowed(int node, gfp_t gfp_mask) · line 4464
Quoted 4 lines
kernel/cgroup/cpuset.c +1
-1/* * Normally, accessing effective_mems would require the cpuset_mutex -* or callback_lock - but node_isset is atomic and the reference +* or callback_lock - but not doing so is acceptable and the reference
"node_isset is atomic" is an argument that not taking cpuset_mutex is acceptable since it's a singular operation against a nodemask (one bit it checked) - and therefore for a moment in time the node is either allowed or not (and we make no absolute guarantee of corrected when this race occurs, we just note that we're corrected).
nodes_copy is not atomic, and in fact this can result in returning an empty nodemask if cs->effective_mems is being recalculated at the time this copy occurs.
Rather than just saying "not doing so is acceptable" - can you please change this comment to explain the implications of not acquiring the mutex a little more clearly?
Example:
``` We do not acquire cpuset_mutex during this check because the correctness of this information is stale immediately after the query anyway - this saves lock contention in exchange for racing against mems_allowed rebinds. As a result, @mask may be empty because cs->effective_mems can be rebound during this call. Callers must check the mask for validity on return. ```
The rest of the comments in the function explains a about this, but I think with this update the comments need a little more rework.
~Gregory
Re: [PATCH v5] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 47 lines · Gregory Price
On Mon, Jan 05, 2026 at 05:01:52AM +0000, Bing Jiao wrote:
... snip ...Quoted 14 lines
kernel/cgroup/cpuset.c +13
-0+/** +* cpuset_nodes_allowed - return mems_allowed mask from a cgroup cpuset. +* @cgroup: pointer to struct cgroup. +* @mask: pointer to struct nodemask_t to be returned. +* +* Returns mems_allowed mask from a cgroup cpuset if it is cgroup v2 and +* has cpuset subsys. Otherwise, returns node_states[N_MEMORY]. +* +* Returned @mask may be empty, and nodes in @mask are not guaranteed +* to be online. +**/ +void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask) +void cpuset_nodes_allowed(struct cgroup *cgroup, nodemask_t *mask) {... snip ...
kernel/cgroup/cpuset.c bool cpuset_current_node_allowed(int node, gfp_t gfp_mask) · line 4464
Quoted 4 lines
kernel/cgroup/cpuset.c +1
-1/* * Normally, accessing effective_mems would require the cpuset_mutex -* or callback_lock - but node_isset is atomic and the reference +* or callback_lock - but not doing so is acceptable and the reference "
node_issetis atomic" is an argument that not takingcpuset_mutexis acceptable since it's a singular operation against a nodemask (one bit it checked) - and therefore for a moment in time the node is either allowed or not (and we make no absolute guarantee of corrected when this race occurs, we just note that we're corrected).
nodes_copyis not atomic, and in fact this can result in returning an empty nodemask ifcs->effective_memsis being recalculated at the time this copy occurs.Rather than just saying "not doing so is acceptable" - can you please change this comment to explain the implications of not acquiring the mutex a little more clearly?
Example:
``` We do not acquire cpuset_mutex during this check because the correctness of this information is stale immediately after the query anyway - this saves lock contention in exchange for racing against mems_allowed rebinds. As a result, @mask may be empty because cs->effective_mems can be rebound during this call. Callers must check the mask for validity on return. ```
Quoted 4 lines
The rest of the comments in the function explains a about this, but I think with this update the comments need a little more rework.
~Gregory
Thanks for the suggestions. I will reword the comment in V6.
Best,
Bing
PATCH v6 1/1 mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoFix two bugs in demote_folio_list() and can_demote() due to incorrect demotion target checks in reclaim/demotion.
Commit 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim") introduces the cpuset.mems_effective check and applies it to can_demote(). However:
1. It does not apply this check in demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.
2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set in mems_effective.
These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update cpuset_node_allowed() and mem_cgroup_node_allowed() to return effective_mems, allowing directly logic-and operation against demotion targets. Also update can_demote() and demote_folio_list() accordingly.
Bug 1 reproduction:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Should respect node 0-2 limit.
# Observation: Node 3 shows significant allocation (MemFree drops)
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Bug 2 reproduction:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Pages are demoted to Nodes 4-5
# Observation: No pages are demoted before oom.
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2---
Patch against the linux mainline.
Tested on the mainline and passed.
Tested on mm-everyting, after Akinobu Mita's series "mm: fix oom-killer not being invoked when demotion is enabled v2", and passed.
v5 -> v6: update cpuset_nodes_allowed()'s comments; move some comments from cpuset_nodes_allowed() to mem_cgroup_node_filter_allowed().
--- include/linux/cpuset.h | 6 ++--- include/linux/memcontrol.h | 6 ++--- kernel/cgroup/cpuset.c | 54 +++++++++++++++++++++++++------------- mm/memcontrol.c | 16 +++++++++-- mm/vmscan.c | 30 ++++++++++++--------- 5 files changed, 74 insertions(+), 38 deletions(-)
include/linux/cpuset.h +3 -3
@@ -174,7 +174,7 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }
@@ -301,9 +301,9 @@ static inline bool read_mems_allowed_retry(unsigned int seq) return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask) { -return true; +nodes_copy (*mask, node_states[N_MEMORY ]); }#endif /* !CONFIG_CPUSETS */
include/linux/memcontrol.h +3 -3
@@ -1744,7 +1744,7 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);
@@ -1815,9 +1815,9 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +static inline void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg, +nodemask_t *mask) { -return true; }static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
kernel/cgroup/cpuset.c +36 -18
76 lines not on this page · raw message
mm/memcontrol.c +14 -2
@@ -5624,9 +5624,21 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask) { -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +nodemask_t allowed; + +if (!memcg) +return ; + +/* +* Since this interface is intended for use by migration paths, and +* reclaim and migration are subject to race conditions such as changes +* in effective_mems and hot-unpluging of nodes, inaccurate allowed +* mask is acceptable. +*/ +cpuset_nodes_allowed (memcg->css.cgroup, &allowed); +nodes_and (*mask, *mask, allowed); }void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
mm/vmscan.c +18 -12
@@ -344,19 +344,21 @@ static void flush_reclaim_state(struct scan_control *sc)static bool can_demote (int nid,struct scan_control *sc,struct mem_cgroup *memcg) { -int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask; -if (!numa_demotion_enabled) +if (!pgdat || !numa_demotion_enabled)return false;if (sc && sc->no_demotion)return false; - demotion_nid =next_demotion_node (nid); -if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask))return false; -/* If demotion node isn't in the cgroup's mems_allowed, fall back */ -return mem_cgroup_node_allowed (memcg, demotion_nid); +/* Filter out nodes that are not in cgroup's mems_allowed. */ +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +return !nodes_empty (allowed_mask); } static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
@@ -1019,7 +1021,8 @@ static struct folio *alloc_demote_folio(struct folio *src,* Folios which are not demoted are left on @demote_folios. */ static unsigned int demote_folio_list (struct list_head *demote_folios, -struct pglist_data *pgdat) +struct pglist_data *pgdat, +struct mem_cgroup *memcg) {int target_nid =next_demotion_node (pgdat->node_id);unsigned int nr_succeeded;
@@ -1033,7 +1036,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,*/ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, .nmask = &allowed_mask, .reason = MR_DEMOTION, };
@@ -1041,10 +1043,14 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (list_empty (demote_folios))return 0 ; -if (target_nid ==NUMA_NO_NODE ) -return 0 ; -node_get_allowed_targets (pgdat, &allowed_mask); +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +if (nodes_empty (allowed_mask)) +return false; + +if (!node_isset (target_nid, allowed_mask)) + target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL,
@@ -1566,7 +1572,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,/* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_demoted =demote_folio_list (&demote_folios, pgdat); + nr_demoted =demote_folio_list (&demote_folios, pgdat, memcg); nr_reclaimed += nr_demoted; stat->nr_demoted += nr_demoted;/* Folios that could not be demoted are still in @demote_folios */
--
2.52.0.358.g0dd7633a29-goog
Re: [PATCH v6] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 32 lines · Bing Jiao
Bug 1 reproduction:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Should respect node 0-2 limit. # Observation: Node 3 shows significant allocation (MemFree drops) stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Bug 2 reproduction:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Pages are demoted to Nodes 4-5 # Observation: No pages are demoted before oom. stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2
Quoted 3 lines
Fixes 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim")Cc <[email protected]>Signed-off-by Bing Jiao <[email protected]>
This looks ok now, haven't tested it myself yet, but looks good.
Thank you for the fix.
~Gregory
Andrew Morton
maintainerRe: [PATCH v6] mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoQuoted 46 lines · On Tue, 6 Jan 2026 07:56:54 +0000 Bing Jiao <[email protected]> wrote:
Fix two bugs in
demote_folio_list()andcan_demote()due to incorrect demotion target checks in reclaim/demotion.Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote(). However:1. It does not apply this check in
demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in
can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set inmems_effective.These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update
cpuset_node_allowed()andmem_cgroup_node_allowed()to returneffective_mems, allowing directly logic-and operation against demotion targets. Also updatecan_demote()anddemote_folio_list()accordingly.Bug 1 reproduction:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Should respect node 0-2 limit. # Observation: Node 3 shows significant allocation (MemFree drops) stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Bug 2 reproduction:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.
Quoted 10 lines
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Pages are demoted to Nodes 4-5
# Observation: No pages are demoted before oom.
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2Thanks.
I'm not confident in my attempts to resolve Akinobu Mita's "mm/vmscan:
don't demote if there is not enough free memory in the lower memory tier" against this. In can_demote(). So I'll drop Akinobu's series, sorry.
Akinobu, can you please redo that series against tomorrow's linux-next? it looks like it needs a resend anyway to try to create some reviewer input.
Re: [PATCH v6] mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoQuoted 226 lines · Bing Jiao
Fix two bugs in
demote_folio_list()andcan_demote()due to incorrect demotion target checks in reclaim/demotion.Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote(). However:1. It does not apply this check in
demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in
can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set inmems_effective.These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update
cpuset_node_allowed()andmem_cgroup_node_allowed()to returneffective_mems, allowing directly logic-and operation against demotion targets. Also updatecan_demote()anddemote_folio_list()accordingly.Bug 1 reproduction:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Should respect node 0-2 limit. # Observation: Node 3 shows significant allocation (MemFree drops) stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Bug 2 reproduction:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Pages are demoted to Nodes 4-5 # Observation: No pages are demoted before oom. stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2Fixes 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim")Cc <[email protected]>Signed-off-by Bing Jiao <[email protected]>---
Patch against the linux mainline.
Tested on the mainline and passed.
Tested on mm-everyting, after Akinobu Mita's series "mm: fix oom-killer not being invoked when demotion is enabled v2", and passed.v5 -> v6: update cpuset_nodes_allowed()'s comments; move some comments from
cpuset_nodes_allowed()tomem_cgroup_node_filter_allowed().--- include/linux/cpuset.h | 6 ++--- include/linux/memcontrol.h | 6 ++--- kernel/cgroup/cpuset.c | 54 +++++++++++++++++++++++++------------- mm/memcontrol.c | 16 +++++++++-- mm/vmscan.c | 30 ++++++++++++--------- 5 files changed, 74 insertions(+), 38 deletions(-)include/linux/cpuset.h +3
-3@@ -174,7 +174,7 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }@@ -301,9 +301,9 @@ static inline bool read_mems_allowed_retry(unsigned int seq) return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask) { -return true; +nodes_copy (*mask, node_states[N_MEMORY ]); }#endif /* !CONFIG_CPUSETS */ include/linux/memcontrol.h +3
-3@@ -1744,7 +1744,7 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);@@ -1815,9 +1815,9 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +static inline void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg, +nodemask_t *mask) { -return true; }static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)kernel/cgroup/cpuset.c +36
-18@@ -4427,40 +4427,58 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)return allowed; } -bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +/** +* cpuset_nodes_allowed - return effective_mems mask from a cgroup cpuset. +* @cgroup: pointer to struct cgroup. +* @mask: pointer to struct nodemask_t to be returned. +* +* Returns effective_mems mask from a cgroup cpuset if it is cgroup v2 and +* has cpuset subsys. Otherwise, returns node_states[N_MEMORY]. +* +* This function intentionally avoids taking the cpuset_mutex or callback_lock +* when accessing effective_mems. This is because the obtained effective_mems +* is stale immediately after the query anyway (e.g., effective_mems is updated +* immediately after releasing the lock but before returning). +* +* As a result, returned @mask may be empty because cs->effective_mems can be +* rebound during this call. Besides, nodes in @mask are not guaranteed to be +* online due to hot plugins. Callers should check the mask for validity on +* return based on its subsequent use. +**/ +void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask) {struct cgroup_subsys_state *css;struct cpuset *cs; -bool allowed;/* * In v1, mem_cgroup and cpuset are unlikely in the same hierarchy * and mems_allowed is likely to be empty even if we could get to it, -* so return true to avoid taking a global lock on the empty check. +* so return directly to avoid taking a global lock on the empty check. */ -if (!cpuset_v2 ()) -return true; +if (!cgroup || !cpuset_v2 ()) { +nodes_copy (*mask, node_states[N_MEMORY ]); +return ; + } css =cgroup_get_e_css (cgroup, &cpuset_cgrp_subsys); -if (!css) -return true; +if (!css) { +nodes_copy (*mask, node_states[N_MEMORY ]); +return ; + }/* -* Normally, accessing effective_mems would require the cpuset_mutex -* or callback_lock - but node_isset is atomic and the reference -* taken via cgroup_get_e_css is sufficient to protect css. -* -* Since this interface is intended for use by migration paths, we -* relax locking here to avoid taking global locks - while accepting -* there may be rare scenarios where the result may be innaccurate. +* The reference taken via cgroup_get_e_css is sufficient to +* protect css, but it does not imply safe accesses to effective_mems. * -* Reclaim and migration are subject to these same race conditions, and -* cannot make strong isolation guarantees, so this is acceptable. +* Normally, accessing effective_mems would require the cpuset_mutex +* or callback_lock - but the correctness of this information is stale +* immediately after the query anyway. We do not acquire the lock +* during this process to save lock contention in exchange for racing +* against mems_allowed rebinds. */ cs = container_of(css, struct cpuset, css); - allowed =node_isset (nid, cs->effective_mems); +nodes_copy (*mask, cs->effective_mems);css_put (css); -return allowed; }/** mm/memcontrol.c +6
-2@@ -5624,9 +5624,21 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask) { -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +nodemask_t allowed; + +if (!memcg) +return ; +
mm/memcontrol.c subsys_initcall(mem_cgroup_swap_init); · line 5643
Quoted 10 lines
mm/memcontrol.c +8
-0+/* +* Since this interface is intended for use by migration paths, and +* reclaim and migration are subject to race conditions such as changes +* in effective_mems and hot-unpluging of nodes, inaccurate allowed +* mask is acceptable. +*/ +cpuset_nodes_allowed (memcg->css.cgroup, &allowed); +nodes_and (*mask, *mask, allowed); }
If this is acceptable, it looks good to me.
Quoted 79 lines
mm/memcontrol.c +0
-0void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)mm/vmscan.c +18
-13@@ -344,19 +344,21 @@ static void flush_reclaim_state(struct scan_control *sc)static bool can_demote (int nid,struct scan_control *sc,struct mem_cgroup *memcg) { -int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask; -if (!numa_demotion_enabled) +if (!pgdat || !numa_demotion_enabled)return false;if (sc && sc->no_demotion)return false; - demotion_nid =next_demotion_node (nid); -if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask))return false; -/* If demotion node isn't in the cgroup's mems_allowed, fall back */ -return mem_cgroup_node_allowed (memcg, demotion_nid); +/* Filter out nodes that are not in cgroup's mems_allowed. */ +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +return !nodes_empty (allowed_mask); } static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,@@ -1019,7 +1021,8 @@ static struct folio *alloc_demote_folio(struct folio *src,* Folios which are not demoted are left on @demote_folios. */ static unsigned int demote_folio_list (struct list_head *demote_folios, -struct pglist_data *pgdat) +struct pglist_data *pgdat, +struct mem_cgroup *memcg) {int target_nid =next_demotion_node (pgdat->node_id);unsigned int nr_succeeded;@@ -1033,7 +1036,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,*/ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, .nmask = &allowed_mask, .reason = MR_DEMOTION, };@@ -1041,10 +1043,14 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (list_empty (demote_folios))return 0 ; -if (target_nid ==NUMA_NO_NODE ) -return 0 ; -node_get_allowed_targets (pgdat, &allowed_mask); +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +if (nodes_empty (allowed_mask)) +return false; + +if (!node_isset (target_nid, allowed_mask)) + target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL,@@ -1566,7 +1572,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,/* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_demoted =demote_folio_list (&demote_folios, pgdat); + nr_demoted =demote_folio_list (&demote_folios, pgdat, memcg); nr_reclaimed += nr_demoted; stat->nr_demoted += nr_demoted;/* Folios that could not be demoted are still in @demote_folios */ --
Quoted 1 line
2.52.0.358.g0dd7633a29-goog
Signature · 2 lines
Best regards, Ridong
v7 cover mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoHi Andrew,
I am sorry for issuing a new patch version after v6 has been merged into mm-hotfixes-unstable.
Main updates in v7:
1. Fixed a bug in v6.
Specifically, next_demotion_node() may return NUMA_NO_NODE if nodes were hot-unplugged. V6 directly checks node_isset(target_nid, allowed_mask), which will cause out-of-boundary bug if target_nid is NUMA_NO_NODE (-1).
2. Preferred node selection.
[Patch 1/2] originally implemented a random selection from allowed nodes if the preferred node from next_demotion_node() was missing from mems_allowed. This behavior contradicts the purpose of migration_target_control.nid, which is intended to identify the preferred node nearest to the source.
To resolve this inconsistency, incorporat the preferred node selection patch into this series.
If there is a consensus among reviewers to backport Patch 2/2 alongside Patch 1/2, they can be combined. Otherwise, I will post Patch 2/2 in another series.
Many thanks!
Best regards,
Bing
Bing Jiao (2):
mm/vmscan: fix demotion targets checks in reclaim/demotion mm/vmscan: select the closest preferred node in demote_folio_list()
include/linux/cpuset.h | 6 ++-- include/linux/memcontrol.h | 6 ++-- include/linux/memory-tiers.h | 6 ++-- kernel/cgroup/cpuset.c | 54 ++++++++++++++++++++++++------------ mm/memcontrol.c | 16 +++++++++-- mm/memory-tiers.c | 11 +++++--- mm/vmscan.c | 49 +++++++++++++++++++++++++------- 7 files changed, 105 insertions(+), 43 deletions(-)
--
2.52.0.457.g6b5491de43-goog
PATCH v7 1/2 mm/vmscan: fix demotion targets checks in reclaim/demotion
Fix two bugs in demote_folio_list() and can_demote() due to incorrect demotion target checks in reclaim/demotion.
Commit 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim") introduces the cpuset.mems_effective check and applies it to can_demote(). However:
1. It does not apply this check in demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.
2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set in mems_effective.
These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update cpuset_node_allowed() to return effective_mems and mem_cgroup_node_allowed() to filter out nodes that are not set in effective_mems. Also update can_demote() and demote_folio_list() accordingly.
Bug 1 reproduction:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Should respect node 0-2 limit.
# Observation: Node 3 shows significant allocation (MemFree drops)
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Bug 2 reproduction:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Pages are demoted to Nodes 4-5
# Observation: No pages are demoted before oom.
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2---
v6 -> v7: fix buggy code in v6.
Specifically, next_demotion_node() may return NUMA_NO_NODE if nodes were hot-unplugged. V6 directly checks node_isset(target_nid, allowed_mask), which will cause out-of-boundary bug if target_nid is NUMA_NO_NODE (-1).
--- include/linux/cpuset.h | 6 ++--- include/linux/memcontrol.h | 6 ++--- kernel/cgroup/cpuset.c | 54 +++++++++++++++++++++++++------------- mm/memcontrol.c | 16 +++++++++-- mm/vmscan.c | 28 +++++++++++++------- 5 files changed, 75 insertions(+), 35 deletions(-)
include/linux/cpuset.h +3 -3
@@ -174,7 +174,7 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }
@@ -301,9 +301,9 @@ static inline bool read_mems_allowed_retry(unsigned int seq) return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask) { -return true; +nodes_copy (*mask, node_states[N_MEMORY ]); }#endif /* !CONFIG_CPUSETS */
include/linux/memcontrol.h +3 -3
@@ -1744,7 +1744,7 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);
@@ -1815,9 +1815,9 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +static inline void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg, +nodemask_t *mask) { -return true; }static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
kernel/cgroup/cpuset.c +36 -18
76 lines not on this page · raw message
mm/memcontrol.c +14 -2
@@ -5624,9 +5624,21 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask) { -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +nodemask_t allowed; + +if (!memcg) +return ; + +/* +* Since this interface is intended for use by migration paths, and +* reclaim and migration are subject to race conditions such as changes +* in effective_mems and hot-unpluging of nodes, inaccurate allowed +* mask is acceptable. +*/ +cpuset_nodes_allowed (memcg->css.cgroup, &allowed); +nodes_and (*mask, *mask, allowed); }void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
mm/vmscan.c +19 -9
@@ -344,19 +344,21 @@ static void flush_reclaim_state(struct scan_control *sc)static bool can_demote (int nid,struct scan_control *sc,struct mem_cgroup *memcg) { -int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask; -if (!numa_demotion_enabled) +if (!pgdat || !numa_demotion_enabled)return false;if (sc && sc->no_demotion)return false; - demotion_nid =next_demotion_node (nid); -if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask))return false; -/* If demotion node isn't in the cgroup's mems_allowed, fall back */ -return mem_cgroup_node_allowed (memcg, demotion_nid); +/* Filter out nodes that are not in cgroup's mems_allowed. */ +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +return !nodes_empty (allowed_mask); } static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
@@ -1019,7 +1021,8 @@ static struct folio *alloc_demote_folio(struct folio *src,* Folios which are not demoted are left on @demote_folios. */ static unsigned int demote_folio_list (struct list_head *demote_folios, -struct pglist_data *pgdat) +struct pglist_data *pgdat, +struct mem_cgroup *memcg) {int target_nid =next_demotion_node (pgdat->node_id);unsigned int nr_succeeded;
@@ -1033,7 +1036,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,*/ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, .nmask = &allowed_mask, .reason = MR_DEMOTION, };
@@ -1042,9 +1044,17 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,return 0 ;if (target_nid ==NUMA_NO_NODE ) +/* No lower-tier nodes or nodes were hot-unplugged. */ return 0 ;node_get_allowed_targets (pgdat, &allowed_mask); +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +if (nodes_empty (allowed_mask)) +return 0 ; + +if (!node_isset (target_nid, allowed_mask)) + target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL,
@@ -1566,7 +1576,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,/* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_demoted =demote_folio_list (&demote_folios, pgdat); + nr_demoted =demote_folio_list (&demote_folios, pgdat, memcg); nr_reclaimed += nr_demoted; stat->nr_demoted += nr_demoted;/* Folios that could not be demoted are still in @demote_folios */
--
2.52.0.457.g6b5491de43-goog
PATCH v7 2/2 mm/vmscan: select the closest preferred node in demote_folio_list()
In reply to Bing JiaoThe preferred demotion node (migration_target_control.nid) should be the one closest to the source node to minimize migration latency. Currently, a discrepancy exists where demote_folio_list() randomly selects an allowed node if the preferred node from next_demotion_node() is not set in mems_allowed.
To address it, update next_demotion_node() to return preferred nodes, allowing the caller to select the preferred one.
Also update demote_folio_list() to traverse the demotion targets hierarchically until a preferred node within mems_allowed is found. It ensures that the selected demotion target is consistently the closest available node.
--- include/linux/memory-tiers.h | 6 +++--- mm/memory-tiers.c | 11 +++++++---- mm/vmscan.c | 25 ++++++++++++++++++++++--- 3 files changed, 32 insertions(+), 10 deletions(-)
include/linux/memory-tiers.h +3 -3
@@ -53,11 +53,11 @@ struct memory_dev_type *mt_find_alloc_memory_type(int adist, struct list_head *memory_types);void mt_put_memory_types (struct list_head *memory_types);#ifdef CONFIG_MIGRATION -int next_demotion_node (int node); +int next_demotion_node (int node,nodemask_t *preferred_nodes);void node_get_allowed_targets (pg_data_t *pgdat,nodemask_t *targets);bool node_is_toptier (int node);#else -static inline int next_demotion_node (int node) +static inline int next_demotion_node (int node,nodemask_t *preferred_nodes) {return NUMA_NO_NODE ; }
@@ -101,7 +101,7 @@ static inline void clear_node_memory_type(int node, struct memory_dev_type *memt } -static inline int next_demotion_node (int node) +static inline int next_demotion_node (int node,nodemask_t *preferred_nodes) {return NUMA_NO_NODE ; }
mm/memory-tiers.c +7 -4
@@ -320,13 +320,14 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)/** * next_demotion_node() - Get the next node in the demotion path * @node: The starting node to lookup the next node +* @preferred_nodes: The pointer to nodemask of all preferred nodes to return * * Return: node id for next memory node in the demotion path hierarchy -* from @node; NUMA_NO_NODE if @node is terminal. This does not keep -* @node online or guarantee that it *continues* to be the next demotion -* target. +* from @node; NUMA_NO_NODE if @node is terminal. Also returns all preferred +* nodes in @preferred_nodes. This does not keep @node online or guarantee +* that it *continues* to be the next demotion target. */ -int next_demotion_node (int node) +int next_demotion_node (int node,nodemask_t *preferred_nodes) {struct demotion_nodes *nd;int target;
@@ -357,6 +358,8 @@ int next_demotion_node(int node)* target node randomly seems better until now. */ target =node_random (&nd->preferred); +if (preferred_nodes) +nodes_copy (*preferred_nodes, nd->preferred);rcu_read_unlock ();return target;
mm/vmscan.c +22 -3
@@ -1024,9 +1024,10 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,struct pglist_data *pgdat, struct mem_cgroup *memcg) { - int target_nid = next_demotion_node(pgdat->node_id); + int target_nid;unsigned int nr_succeeded;nodemask_t allowed_mask; +nodemask_t preferred; struct migration_target_control mtc = { /*
@@ -1052,8 +1053,26 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (nodes_empty (allowed_mask))return 0 ; -if (!node_isset (target_nid, allowed_mask)) - target_nid =node_random (&allowed_mask); + target_nid =next_demotion_node (pgdat->node_id, &preferred); +while (target_nid !=NUMA_NO_NODE && + !node_isset (target_nid, allowed_mask)) { +/* Filter out preferred nodes that are not in allowed. */ +nodes_and (preferred, preferred, allowed_mask); +if (!nodes_empty (preferred)) { +/* Randomly select one node from preferred. */ + target_nid =node_random (&preferred); +break ; + } +/* +* Preferred nodes in the lower tier are not set in allowed. +* Recursively get preferred from the next lower tier. +*/ + target_nid =next_demotion_node (target_nid, &preferred); + } + +if (target_nid ==NUMA_NO_NODE ) +/* Nodes are gone (e.g., hot-unplugged). */ +return 0 ; mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */
--
2.52.0.457.g6b5491de43-goog
PATCH 1/1 mm/vmscan: fix uninitialized variable in demote_folio_list()
Commit d92027a5f30a6c ("mm/vmscan: select the closest preferred node in demote_folio_list()") refactored the selection logic for demotion target nodes. However, it removed the initial assignment of 'target_nid' while leaving a premature check for 'NUMA_NO_NODE' in place.
Since 'target_nid' is now assigned further down in the function based on preferred and allowed node masks, the initial check at line 1046 is redundant.
Remove the redundant check to resolve the compiler warning.
---
Hi Andrew,
Sorry for bother you again in my patch series.
There is a broken commit in mm-new branch, which is introduced by this patch series, Commit d92027a5f30a6c ("mm/vmscan: select the closest preferred node in demote_folio_list()").
As the broken commit is currently only in *mm-new*, could you please squash the fix into the broken commit before it is pulled into the mainline?
Thanks for your time!
Best regards,
Bing
--- mm/vmscan.c | 4 ---- 1 file changed, 4 deletions(-)
mm/vmscan.c +0 -4
@@ -1044,10 +1044,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (list_empty (demote_folios))return 0 ; -if (target_nid ==NUMA_NO_NODE ) -/* No lower-tier nodes or nodes were hot-unplugged. */ -return 0 ; -node_get_allowed_targets (pgdat, &allowed_mask);mem_cgroup_node_filter_allowed (memcg, &allowed_mask);if (nodes_empty (allowed_mask))
--
2.52.0.457.g6b5491de43-goog
PATCH 1/1 mm/vmscan: fix uninitialized variable in demote_folio_list()
In reply to Bing JiaoCommit d92027a5f30a6c ("mm/vmscan: select the closest preferred node in demote_folio_list()") refactored the selection logic for demotion target nodes. However, it removed the initial assignment of 'target_nid' while leaving a premature check for 'NUMA_NO_NODE' in place.
Since 'target_nid' is now assigned further down in the function based on preferred and allowed node masks, the initial check at line 1046 is redundant.
Remove the redundant check to resolve the compiler warning.
---
Hi Andrew,
Sorry for bother you again in my patch series.
There is a broken commit in mm-new branch, which is introduced by this patch series, Commit d92027a5f30a6c ("mm/vmscan: select the closest preferred node in demote_folio_list()").
As the broken commit is currently only in *mm-new*, could you please squash the fix into the broken commit before it is pulled into the mainline?
Thanks for your time!
Best regards,
Bing
--- mm/vmscan.c | 4 ---- 1 file changed, 4 deletions(-)
mm/vmscan.c +0 -4
@@ -1044,10 +1044,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (list_empty (demote_folios))return 0 ; -if (target_nid ==NUMA_NO_NODE ) -/* No lower-tier nodes or nodes were hot-unplugged. */ -return 0 ; -node_get_allowed_targets (pgdat, &allowed_mask);mem_cgroup_node_filter_allowed (memcg, &allowed_mask);if (nodes_empty (allowed_mask))
--
2.52.0.457.g6b5491de43-goog
v8 cover mm/vmscan: select the closest preferred node in demote_folio_list()
In reply to Bing Jiao[Patch v7 2/2] tries to traverse the demotion targets hierarchically until a preferred node within mems_allowed is found.
However, node_demotion[node].preferred may not contain all nodes in the next memory tier. For example, in the below typograph:
Top-tier Node 0 ---- Node 1
| |
| |
Far-tier Node 2 Node 3node_demotion[0].preferred: [2]
node_demotion[1].preferred: [3]
node_demotion[2].preferred: []
node_demotion[3].preferred: []
If mems_allowed is [3], demote_folio_list() cannot find a valid demotion target when demoting pages from Node 0. Because next_demotion_node(0) returns Node 2 and [2], which are not in mems_allowed; the second call to next_demotion_node(2) returns NUMA_NO_NODE.
This patch uses find_next_best_node() to find the closest node when all preferred nodes are not in mems_allowed, mimicking the logic of establish_demotion_targets().
Hi Andrew,
Could you please remove Commit fd8cb9a8cedc ("mm/vmscan: select the closest preferred node in demote_folio_list()") and Commit 127714c76c46 ("mm/vmscan: fix uninitialized variable in demote_folio_list()") from the mm-unstable branch, as this patch serves as a full replacement for both.
I suggest allowing some time for this to be thoroughly reviewed in case there are any remaining edge cases or issues I overlooked.
Because no modifications to [Patch v7 1/2], I did not attach a new version of it.
Thanks!
Best regards,
Bing
Bing Jiao (1):
mm/vmscan: select the closest preferred node in demote_folio_list()
include/linux/memory-tiers.h | 6 +++--- mm/memory-tiers.c | 11 +++++++---- mm/vmscan.c | 30 +++++++++++++++++++++++------- 3 files changed, 33 insertions(+), 14 deletions(-)
--
2.52.0.457.g6b5491de43-goog
PATCH v8 2/2 mm/vmscan: select the closest preferred node in demote_folio_list()
The preferred demotion node (migration_target_control.nid) should be the one closest to the source node to minimize migration latency. Currently, a discrepancy exists where demote_folio_list() randomly selects an allowed node if the preferred node from next_demotion_node() is not set in mems_allowed.
To address it, update next_demotion_node() to return preferred nodes, allowing the caller to select the preferred one. Also update demote_folio_list() to get the closest node from allowed if all preferred nodes are not set in mems_allowed.
It ensures that the preferred demotion target is consistently the closest available node to the source node.
--- include/linux/memory-tiers.h | 6 +++--- mm/memory-tiers.c | 11 +++++++---- mm/vmscan.c | 30 +++++++++++++++++++++++------- 3 files changed, 33 insertions(+), 14 deletions(-)
include/linux/memory-tiers.h +3 -3
@@ -53,11 +53,11 @@ struct memory_dev_type *mt_find_alloc_memory_type(int adist, struct list_head *memory_types);void mt_put_memory_types (struct list_head *memory_types);#ifdef CONFIG_MIGRATION -int next_demotion_node (int node); +int next_demotion_node (int node,nodemask_t *preferred_nodes);void node_get_allowed_targets (pg_data_t *pgdat,nodemask_t *targets);bool node_is_toptier (int node);#else -static inline int next_demotion_node (int node) +static inline int next_demotion_node (int node,nodemask_t *preferred_nodes) {return NUMA_NO_NODE ; }
@@ -101,7 +101,7 @@ static inline void clear_node_memory_type(int node, struct memory_dev_type *memt } -static inline int next_demotion_node (int node) +static inline int next_demotion_node (int node,nodemask_t *preferred_nodes) {return NUMA_NO_NODE ; }
mm/memory-tiers.c +7 -4
@@ -320,13 +320,14 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)/** * next_demotion_node() - Get the next node in the demotion path * @node: The starting node to lookup the next node +* @preferred_nodes: The pointer to nodemask of all preferred nodes to return * * Return: node id for next memory node in the demotion path hierarchy -* from @node; NUMA_NO_NODE if @node is terminal. This does not keep -* @node online or guarantee that it *continues* to be the next demotion -* target. +* from @node; NUMA_NO_NODE if @node is terminal. Also returns all preferred +* nodes in @preferred_nodes. This does not keep @node online or guarantee +* that it *continues* to be the next demotion target. */ -int next_demotion_node (int node) +int next_demotion_node (int node,nodemask_t *preferred_nodes) {struct demotion_nodes *nd;int target;
@@ -357,6 +358,8 @@ int next_demotion_node(int node)* target node randomly seems better until now. */ target =node_random (&nd->preferred); +if (preferred_nodes) +nodes_copy (*preferred_nodes, nd->preferred);rcu_read_unlock ();return target;
mm/vmscan.c +23 -7
@@ -1023,9 +1023,10 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,struct pglist_data *pgdat, struct mem_cgroup *memcg) { - int target_nid = next_demotion_node(pgdat->node_id); + int target_nid;unsigned int nr_succeeded;nodemask_t allowed_mask; +nodemask_t preferred; struct migration_target_control mtc = { /*
@@ -1042,17 +1043,32 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (list_empty (demote_folios))return 0 ; -if (target_nid ==NUMA_NO_NODE ) -/* No lower-tier nodes or nodes were hot-unplugged. */ -return 0 ; -node_get_allowed_targets (pgdat, &allowed_mask);mem_cgroup_node_filter_allowed (memcg, &allowed_mask);if (nodes_empty (allowed_mask))return 0 ; -if (!node_isset (target_nid, allowed_mask)) - target_nid =node_random (&allowed_mask); + target_nid =next_demotion_node (pgdat->node_id, &preferred); +if (target_nid ==NUMA_NO_NODE ) +/* No lower-tier nodes (e.g., nodes were hot-unplugged) */ +return 0 ; + +if (!node_isset (target_nid, allowed_mask)) { +/* Filter out preferred nodes that are not in allowed. */ +nodes_and (preferred, preferred, allowed_mask); +if (!nodes_empty (preferred)) { +/* Randomly select one node from preferred. */ + target_nid =node_random (&preferred); + }else { +/* Get the closest node from allowed. */ +nodes_complement (preferred, allowed_mask); + target_nid =find_next_best_node (pgdat->node_id, + &preferred); +if (target_nid ==NUMA_NO_NODE ) +return 0 ; + } + } + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */
--
2.52.0.457.g6b5491de43-goog
v9 cover mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoThis patch series addresses two issues in demote_folio_list(), can_demote(), and next_demotion_node() in reclaim/demotion.
1. demote_folio_list() and can_demote() do not correctly check demotion target against cpuset.mems_effective, which will cause (a) pages are demoted to not-allowed nodes and (b) pages are failed to demote even if the system still have allowed demotion nodes.
Patch 1 fixes this bug by update cpuset_node_allowed() and mem_cgroup_node_allowed() to return effective_mems, allowing directly logic-and operation against demotion targets.
2. next_demotion_node() returns a preferred demotion target, but it does check the node against allowed nodes.
Patch 2 ensures that next_demotion_node() filters against the allowed node mask and selects the closest demotion target to the source node.
===
Hi Andrew,
Sorry for the extra noise in your inbox.
I’m resubmitting the full refreshed patch series together this time. I just realized it is better to include the unmodified patches alongside the modified ones to ensure compatibility with upstream automated tools and to simplify your review process.
The refreshed patch series replaces some commits in mm-untable, including:
- Commit 809cc3db1831 ("mm/vmscan: fix demotion targets checks in reclaim/demotion")
- Commit fd8cb9a8cedc ("mm/vmscan: select the closest preferred node in demote_folio_list()")
- Commit 127714c76c46 ("mm/vmscan: fix uninitialized variable in
demote_folio_list()")
Thanks!
Best regards,
Bing
Bing Jiao (2):
mm/vmscan: fix demotion targets checks in reclaim/demotion mm/vmscan: select the closest perferred node in demote_folio_list()
include/linux/cpuset.h | 6 ++-- include/linux/memcontrol.h | 6 ++-- include/linux/memory-tiers.h | 6 ++-- kernel/cgroup/cpuset.c | 54 ++++++++++++++++++++++++------------ mm/memcontrol.c | 16 +++++++++-- mm/memory-tiers.c | 21 ++++++++++---- mm/vmscan.c | 31 +++++++++++++-------- 7 files changed, 95 insertions(+), 45 deletions(-)
--
2.52.0.457.g6b5491de43-goog
PATCH v9 1/2 mm/vmscan: fix demotion targets checks in reclaim/demotion
Fix two bugs in demote_folio_list() and can_demote() due to incorrect demotion target checks against cpuset.mems_effective in reclaim/demotion.
Commit 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim") introduces the cpuset.mems_effective check and applies it to can_demote(). However:
1. It does not apply this check in demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.
2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set in mems_effective.
These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update cpuset_node_allowed() and mem_cgroup_node_allowed() to return effective_mems, allowing directly logic-and operation against demotion targets. Also update can_demote() and demote_folio_list() accordingly.
Bug 1 reproduction:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Should respect node 0-2 limit.
# Observation: Node 3 shows significant allocation (MemFree drops)
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Bug 2 reproduction:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.
Test script:
echo 1 > /sys/kernel/mm/numa/demotion_enabled
mkdir /sys/fs/cgroup/test
echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control
echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems
echo $$ > /sys/fs/cgroup/test/cgroup.procs
swapoff -a
# Expectation: Pages are demoted to Nodes 4-5
# Observation: No pages are demoted before oom.
stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2---
v7 -> v9:
Minor updates in demote_folio_list() for better code logic.
include/linux/cpuset.h | 6 ++--- include/linux/memcontrol.h | 6 ++--- kernel/cgroup/cpuset.c | 54 +++++++++++++++++++++++++------------- mm/memcontrol.c | 16 +++++++++-- mm/vmscan.c | 34 +++++++++++++++--------- 5 files changed, 78 insertions(+), 38 deletions(-)
include/linux/cpuset.h +3 -3
@@ -174,7 +174,7 @@ static inline void set_mems_allowed(nodemask_t nodemask) task_unlock(current); } -extern bool cpuset_node_allowed (struct cgroup *cgroup,int nid); +extern void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask);#else /* !CONFIG_CPUSETS */ static inline bool cpusets_enabled (void ) {return false; }
@@ -301,9 +301,9 @@ static inline bool read_mems_allowed_retry(unsigned int seq) return false; } -static inline bool cpuset_node_allowed (struct cgroup *cgroup,int nid) +static inline void cpuset_nodes_allowed (struct cgroup *cgroup,nodemask_t *mask) { -return true; +nodes_copy (*mask, node_states[N_MEMORY ]); }#endif /* !CONFIG_CPUSETS */
include/linux/memcontrol.h +3 -3
@@ -1744,7 +1744,7 @@ static inline void count_objcg_events(struct obj_cgroup *objcg,rcu_read_unlock (); } -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid); +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask);void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg);
@@ -1815,9 +1815,9 @@ static inline ino_t page_cgroup_ino(struct page *page)return 0 ; } -static inline bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +static inline void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg, +nodemask_t *mask) { -return true; }static inline void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
kernel/cgroup/cpuset.c +36 -18
76 lines not on this page · raw message
mm/memcontrol.c +14 -2
@@ -5624,9 +5624,21 @@ subsys_initcall(mem_cgroup_swap_init);#endif /* CONFIG_SWAP */ -bool mem_cgroup_node_allowed (struct mem_cgroup *memcg,int nid) +void mem_cgroup_node_filter_allowed (struct mem_cgroup *memcg,nodemask_t *mask) { -return memcg ?cpuset_node_allowed (memcg->css.cgroup, nid) : true; +nodemask_t allowed; + +if (!memcg) +return ; + +/* +* Since this interface is intended for use by migration paths, and +* reclaim and migration are subject to race conditions such as changes +* in effective_mems and hot-unpluging of nodes, inaccurate allowed +* mask is acceptable. +*/ +cpuset_nodes_allowed (memcg->css.cgroup, &allowed); +nodes_and (*mask, *mask, allowed); }void mem_cgroup_show_protected_memory (struct mem_cgroup *memcg)
mm/vmscan.c +22 -12
@@ -344,19 +344,21 @@ static void flush_reclaim_state(struct scan_control *sc)static bool can_demote (int nid,struct scan_control *sc,struct mem_cgroup *memcg) { -int demotion_nid; +struct pglist_data *pgdat =NODE_DATA (nid); +nodemask_t allowed_mask; -if (!numa_demotion_enabled) +if (!pgdat || !numa_demotion_enabled)return false;if (sc && sc->no_demotion)return false; - demotion_nid =next_demotion_node (nid); -if (demotion_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +if (nodes_empty (allowed_mask))return false; -/* If demotion node isn't in the cgroup's mems_allowed, fall back */ -return mem_cgroup_node_allowed (memcg, demotion_nid); +/* Filter out nodes that are not in cgroup's mems_allowed. */ +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +return !nodes_empty (allowed_mask); } static inline bool can_reclaim_anon_pages(struct mem_cgroup *memcg,
@@ -1019,9 +1021,10 @@ static struct folio *alloc_demote_folio(struct folio *src,* Folios which are not demoted are left on @demote_folios. */ static unsigned int demote_folio_list (struct list_head *demote_folios, -struct pglist_data *pgdat) +struct pglist_data *pgdat, +struct mem_cgroup *memcg) { -int target_nid =next_demotion_node (pgdat->node_id); +int target_nid;unsigned int nr_succeeded;nodemask_t allowed_mask;
@@ -1033,7 +1036,6 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,*/ .gfp_mask = (GFP_HIGHUSER_MOVABLE & ~__GFP_RECLAIM) | __GFP_NOMEMALLOC | GFP_NOWAIT, - .nid = target_nid, .nmask = &allowed_mask, .reason = MR_DEMOTION, };
@@ -1041,10 +1043,18 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (list_empty (demote_folios))return 0 ; -if (target_nid ==NUMA_NO_NODE ) +node_get_allowed_targets (pgdat, &allowed_mask); +mem_cgroup_node_filter_allowed (memcg, &allowed_mask); +if (nodes_empty (allowed_mask))return 0 ; -node_get_allowed_targets (pgdat, &allowed_mask); + target_nid =next_demotion_node (pgdat->node_id); +if (target_nid ==NUMA_NO_NODE ) +/* No lower-tier nodes or nodes were hot-unplugged. */ +return 0 ; +if (!node_isset (target_nid, allowed_mask)) + target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */ migrate_pages(demote_folios, alloc_demote_folio, NULL,
@@ -1566,7 +1576,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,/* 'folio_list' is always empty here */ /* Migrate folios selected for demotion */ - nr_demoted =demote_folio_list (&demote_folios, pgdat); + nr_demoted =demote_folio_list (&demote_folios, pgdat, memcg); nr_reclaimed += nr_demoted; stat->nr_demoted += nr_demoted;/* Folios that could not be demoted are still in @demote_folios */
--
2.52.0.457.g6b5491de43-goog
Re: [PATCH v9 1/2] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 57 lines · Bing Jiao
Fix two bugs in
demote_folio_list()andcan_demote()due to incorrect demotion target checks against cpuset.mems_effective in reclaim/demotion.Commit
7d709f49babc("vmscan,cgroup: applymems_effectiveto reclaim") introduces the cpuset.mems_effective check and applies it tocan_demote(). However:1. It does not apply this check in
demote_folio_list(), which leads to situations where pages are demoted to nodes that are explicitly excluded from the task's cpuset.mems.2. It checks only the nodes in the immediate next demotion hierarchy and does not check all allowed demotion targets in
can_demote(). This can cause pages to never be demoted if the nodes in the next demotion hierarchy are not set inmems_effective.These bugs break resource isolation provided by cpuset.mems. This is visible from userspace because pages can either fail to be demoted entirely or are demoted to nodes that are not allowed in multi-tier memory systems.
To address these bugs, update
cpuset_node_allowed()andmem_cgroup_node_allowed()to returneffective_mems, allowing directly logic-and operation against demotion targets. Also updatecan_demote()anddemote_folio_list()accordingly.Bug 1 reproduction:
Assume a system with 4 nodes, where nodes 0-1 are top-tier and nodes 2-3 are far-tier memory. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Should respect node 0-2 limit. # Observation: Node 3 shows significant allocation (MemFree drops) stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1Bug 2 reproduction:
Assume a system with 6 nodes, where nodes 0-2 are top-tier, node 3 is a far-tier node, and nodes 4-5 are the farthest-tier nodes. All nodes have equal capacity.Test script: echo 1 > /sys/kernel/mm/numa/demotion_enabled mkdir /sys/fs/cgroup/test echo +cpuset > /sys/fs/cgroup/cgroup.subtree_control echo "0-2,4-5" > /sys/fs/cgroup/test/cpuset.mems echo $$ > /sys/fs/cgroup/test/cgroup.procs swapoff -a # Expectation: Pages are demoted to Nodes 4-5 # Observation: No pages are demoted before oom. stress-ng --oomable --vm 1 --vm-bytes 150% --mbind 0,1,2
Quoted 2 lines
Fixes 7d709f49babc ("vmscan,cgroup: apply mems_effective to reclaim")Signed-off-by Bing Jiao <[email protected]>
PATCH v9 2/2 mm/vmscan: select the closest perferred node in demote_folio_list()
In reply to Bing JiaoThe preferred demotion node (migration_target_control.nid) should be the one closest to the source node to minimize migration latency. Currently, a discrepancy exists where demote_folio_list() randomly selects an allowed node if the preferred node from next_demotion_node() is not set in mems_effective.
To address it, update next_demotion_node() to select a preferred target against allowed nodes; and to return the closest demotion target if all preferred nodes are not in mems_effective via next_demotion_node().
It ensures that the preferred demotion target is consistently the closest available node to the source node.
---
v7 -> v8:
Fix bugs in v7.
Remove the while loop of getting the preferred node via next_demotion_node().
Use find_next_best_node() to find the closest demotion target.
v8 -> v9:
Move allowed node checks and identification of the closest demotion target into next_demotion_node() for better function splitting.
include/linux/memory-tiers.h | 6 +++--- mm/memory-tiers.c | 21 ++++++++++++++++----- mm/vmscan.c | 5 ++--- 3 files changed, 21 insertions(+), 11 deletions(-)
include/linux/memory-tiers.h +3 -3
@@ -53,11 +53,11 @@ struct memory_dev_type *mt_find_alloc_memory_type(int adist, struct list_head *memory_types);void mt_put_memory_types (struct list_head *memory_types);#ifdef CONFIG_MIGRATION -int next_demotion_node (int node); +int next_demotion_node (int node,const nodemask_t *allowed_mask);void node_get_allowed_targets (pg_data_t *pgdat,nodemask_t *targets);bool node_is_toptier (int node);#else -static inline int next_demotion_node (int node) +static inline int next_demotion_node (int node,const nodemask_t *allowed_mask) {return NUMA_NO_NODE ; }
@@ -101,7 +101,7 @@ static inline void clear_node_memory_type(int node, struct memory_dev_type *memt } -static inline int next_demotion_node (int node) +static inline int next_demotion_node (int node,const nodemask_t *allowed_mask) {return NUMA_NO_NODE ; }
mm/memory-tiers.c +16 -5
@@ -320,16 +320,17 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)/** * next_demotion_node() - Get the next node in the demotion path * @node: The starting node to lookup the next node +* @allowed_mask: The pointer to allowed node mask * * Return: node id for next memory node in the demotion path hierarchy * from @node; NUMA_NO_NODE if @node is terminal. This does not keep * @node online or guarantee that it *continues* to be the next demotion * target. */ -int next_demotion_node (int node) +int next_demotion_node (int node,const nodemask_t *allowed_mask) {struct demotion_nodes *nd; -int target; +nodemask_t mask;if (!node_demotion)return NUMA_NO_NODE ;
@@ -344,6 +345,10 @@ int next_demotion_node(int node)* node_demotion[] reads need to be consistent. */ rcu_read_lock (); +/* Filter out nodes that are not in allowed_mask. */ +nodes_and (mask, nd->preferred, *allowed_mask); +rcu_read_unlock (); +/* * If there are multiple target nodes, just select one * target node randomly.
@@ -356,10 +361,16 @@ int next_demotion_node(int node)* caching issue, which seems more complicated. So selecting * target node randomly seems better until now. */ - target =node_random (&nd->preferred); -rcu_read_unlock (); +if (!nodes_empty (mask)) +return node_random (&mask); -return target; +/* +* Preferred nodes are not in allowed_mask. Filp bits in +* allowed_mask as used node mask. Then, use it to get the +* closest demotion target. +*/ +nodes_complement (mask, *allowed_mask); +return find_next_best_node (node, &mask); }static void disable_all_demotion_targets (void )
mm/vmscan.c +2 -3
@@ -1048,12 +1048,11 @@ static unsigned int demote_folio_list(struct list_head *demote_folios,if (nodes_empty (allowed_mask))return 0 ; - target_nid =next_demotion_node (pgdat->node_id); + target_nid =next_demotion_node (pgdat->node_id, &allowed_mask);if (target_nid ==NUMA_NO_NODE )/* No lower-tier nodes or nodes were hot-unplugged. */ return 0 ; -if (!node_isset (target_nid, allowed_mask)) - target_nid =node_random (&allowed_mask); + mtc.nid = target_nid;/* Demotion ignores all cpuset and mempolicy settings */
--
2.52.0.457.g6b5491de43-goog
Re: [PATCH v9 2/2] mm/vmscan: select the closest perferred node in demote_folio_list()
Quoted 6 lines · Bing Jiao
The preferred demotion node (migration_target_control.nid) should be the one closest to the source node to minimize migration latency. Currently, a discrepancy exists where
demote_folio_list()randomly selects an allowed node if the preferred node fromnext_demotion_node()is not set inmems_effective.
Quoted 8 lines
To address it, update
next_demotion_node()to select a preferred target against allowed nodes; and to return the closest demotion target if all preferred nodes are not inmems_effectivevianext_demotion_node().It ensures that the preferred demotion target is consistently the closest available node to the source node.
Signed-off-by Bing Jiao <[email protected]>
One nit below:
[...]
Quoted 33 lines
mm/memory-tiers.c +7
-2@@ -320,16 +320,17 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)/** * next_demotion_node() - Get the next node in the demotion path * @node: The starting node to lookup the next node +* @allowed_mask: The pointer to allowed node mask * * Return: node id for next memory node in the demotion path hierarchy * from @node; NUMA_NO_NODE if @node is terminal. This does not keep * @node online or guarantee that it *continues* to be the next demotion * target. */ -int next_demotion_node (int node) +int next_demotion_node (int node,const nodemask_t *allowed_mask) {struct demotion_nodes *nd; -int target; +nodemask_t mask;if (!node_demotion)return NUMA_NO_NODE ;@@ -344,6 +345,10 @@ int next_demotion_node(int node)* node_demotion[] reads need to be consistent. */ rcu_read_lock (); +/* Filter out nodes that are not in allowed_mask. */ +nodes_and (mask, nd->preferred, *allowed_mask); +rcu_read_unlock (); +/* * If there are multiple target nodes, just select one * target node randomly. @@ -356,10 +361,16 @@ int next_demotion_node(int node)* caching issue, which seems more complicated. So selecting
mm/memory-tiers.c int next_demotion_node(int node) · line 368
Quoted 10 lines
mm/memory-tiers.c +4
-3* target node randomly seems better until now. */ - target =node_random (&nd->preferred); -rcu_read_unlock (); +if (!nodes_empty (mask)) +return node_random (&mask); -return target; +/* +* Preferred nodes are not in allowed_mask. Filp bits in
Filp -> Flip
Quoted 7 lines
mm/memory-tiers.c +5
-0+* allowed_mask as used node mask. Then, use it to get the +* closest demotion target. +*/ +nodes_complement (mask, *allowed_mask); +return find_next_best_node (node, &mask); }
Andrew Morton
maintainerRe: [PATCH v9 0/2] mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoQuoted 4 lines · On Wed, 14 Jan 2026 20:53:01 +0000 Bing Jiao <[email protected]> wrote:
I’m resubmitting the full refreshed patch series together this time. I just realized it is better to include the unmodified patches alongside the modified ones to ensure compatibility with upstream automated tools and to simplify your review process.
No probs.
[1/2] is cc:stable whereas [2/2] is not. Ordinarily that means I must split the series apart (they take different routes) and often discard the [0/n].
In this case I think I'll leave things as-is, so [1/2]'s entry into the -stable pipeline will occur a few weeks later. I don't think the problem is serious enough to need super-fast-tracking?
Hopefully this approach means we'll get some Reviewed-bys ;)
Re: [PATCH v9 0/2] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 7 lines · Andrew Morton
Quoted 4 lines · On Wed, 14 Jan 2026 20:53:01 +0000 Bing Jiao <[email protected]> wrote:
I’m resubmitting the full refreshed patch series together this time. I just realized it is better to include the unmodified patches alongside the modified ones to ensure compatibility with upstream automated tools and to simplify your review process.
Quoted 5 lines
No probs.
[1/2] is cc:stable whereas [2/2] is not. Ordinarily that means I must split the series apart (they take different routes) and often discard the [0/n].
Hi Andrew,
Thank you for the explanation. I appreciate the insight into the upstream process and the time you have taken to review this series. I wish I had known this earlier so as not to add to your workload.
Quoted 5 lines
In this case I think I'll leave things as-is, so [1/2]'s entry into the -stable pipeline will occur a few weeks later. I don't think the problem is serious enough to need super-fast-tracking?
Hopefully this approach means we'll get some Reviewed-bys ;)
I agree that the issue does not require urgent fast-tracking, so leaving the series as-is for the standard pipeline is appropriate.
Best regards,
Bing
Shakeel Butt
maintainerRe: [PATCH v9 0/2] mm/vmscan: fix demotion targets checks in reclaim/demotion
In reply to Bing JiaoHi Bing,
Please don't reply (i.e. use In-Reply-To) to older revision of your series. Send each revision independently.
Quoted 6 lines · Bing Jiao
This patch series addresses two issues in
demote_folio_list(),can_demote(), andnext_demotion_node()in reclaim/demotion.1.
demote_folio_list()andcan_demote()do not correctly check demotion target against cpuset.mems_effective, which will cause (a) pages are demoted
pages to be demoted
Quoted 1 line
to not-allowed nodes and (b) pages are failed to demote
page fail to demote
Quoted 4 lines
even
if the system still have allowed demotion nodes.Patch 1 fixes this bug by update
updating
Quoted 5 lines
cpuset_node_allowed()and
mem_cgroup_node_allowed()to returneffective_mems, allowing directly logic-and operation against demotion targets.2.
next_demotion_node()returns a preferred demotion target, but it does
does or does not?
Quoted 5 lines
check the node against allowed nodes.
Patch 2 ensures that
next_demotion_node()filters against the allowed node mask and selects the closest demotion target to the source node.
Re: [PATCH v9 0/2] mm/vmscan: fix demotion targets checks in reclaim/demotion
Quoted 26 lines · Shakeel Butt
Hi Bing,
Please don't reply (i.e. use In-Reply-To) to older revision of your series. Send each revision independently.
Quoted 6 lines · Bing Jiao
This patch series addresses two issues in
demote_folio_list(),can_demote(), andnext_demotion_node()in reclaim/demotion.1.
demote_folio_list()andcan_demote()do not correctly check demotion target against cpuset.mems_effective, which will cause (a) pages are demotedpages to be demoted
Quoted 1 line
to not-allowed nodes and (b) pages are failed to demote
page fail to demote
Quoted 4 lines
even
if the system still have allowed demotion nodes.Patch 1 fixes this bug by update
updating
Quoted 12 lines
Quoted 5 lines
cpuset_node_allowed()and
mem_cgroup_node_allowed()to returneffective_mems, allowing directly logic-and operation against demotion targets.2.
next_demotion_node()returns a preferred demotion target, but it doesdoes or does not?
Quoted 4 lines
check the node against allowed nodes.
Patch 2 ensures that
next_demotion_node()filters against the allowed node mask and selects the closest demotion target to the source node.
Hi Shakeel,
Thank you for taking the time to review this patch series and for the helpful corrections. I also appreciate the reminder about the patch replying rule and will make sure to send future revisions independently.
Have a great weekend!
Best,
Bing