Skip to contentSkip to conversation

Private Memory NUMA Nodes

PATCH In review Gregory Price 11 participants 97 messages Jul 20 – Sep 25

Subsystems (30)
lore
Share & tools
lore

Gregory Price

author maintainer
#0 lore raw

v5 cover Private Memory NUMA Nodes

This series introduces the concept of "Private Memory Nodes", which are opted into two basic functionalities by default:
- page allocation (mm/page_alloc.c)
- OOM killing

All other features of mm/ are opted out of managing these NUMA nodes and its memory. Then we add capability bits to allow node owners to opt back into those services (if supported).

NUMA-hosted memory is presently a most-privilege system (any memory on any node, except ZONE_DEVICE, is 100% fungible and accessible). We then slap restrictions on top of it (cpuset, mempolicy, ZONE type, page/folio flags, etc).

The goal here is to flip that dynamic, isolate by default and then opt-in to specific services that the device says is safe.

Isolation at the NUMA/Zonelist layer provides a powerful mechanism for memory hosted on accelerators - re-use of the kernel mm/ code.

- Accelerators (GPUs) can use demotion, numactl, and reclaim.
- Special memory devices (Compressed RAM) with special access controls (promote-on-write) can have generic services written for them.
- Network devices with large memory regions intended for ring buffers can use the buddy and standard networking stack.
- Slow, disaggregated memory pools which aren't suitable as general purpose memory get cleaner interfaces (no need to re-write the buddy in userland, can use migration interface, etc).
- Per-workload dedicated memory nodes (disaggregated VM memory)

And more use cases I have collected over the past few years.

Not included here is a dax-extension [1] that exposes all the internal bits as userland controls for testing - along with a pile of selftests that prove correctness.

Changes Since V4
================
- Massive reduction in complexity.
- no ops struct
- no callback functions
- no __GFP_PRIVATE
- no __GFP_THISNODE requirement
- no task flags (no PF_MEMALLOC_* in the alloc path)
- isolation via a dedicated zonelist:
- private nodes are omitted from FALLBACK/NOFALLBACK
- added ZONELIST_PRIVATE(_NOFALLBACK)
- ALLOC_ZONELIST_PRIVATE alloc_flag
- on top of Brendan Jackman's recent mm/page_alloc.h work [3]
- Zonelist selection rides the allocator's alloc_flags
- rename OPS -> CAPS (capabilities)
- split base functionality (isolation) from opt-ins (CAPS)
- first half of series can be merged without CAPS
- dropped compressed ram example from series
- will submit separately if this moves forward
- Added KVM as first primary in-tree user (mempolicy / CAP_USER_NUMA)
- fully functional dax-kmem extension and huge suite of selftests located at my github, to be discussed separately [1]

Patch Layout
============
The series is broken into two sections:

1) N_MEMORY_PRIVATE Introduction.
Introduce the node state.
Opt those nodes out of mm/ services.

2) NODE_PRIVATE_CAP_* features
A set of mm/ service opt-in flags that augment private nodes to make them more useful (i.e. reclaim = overcommit).

NODE_PRIVATE_CAP_LTPIN for private node folio pinning NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes NODE_PRIVATE_CAP_USER_NUMA for userland numa controls NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim

My hope is to merge at least #1 pulled ahead while #2 is debated.

Allocation Isolation
====================
page_alloc presently controls whether a node's memory can be allocated on a given call by 4 things (in order of authority)

1) ZONELIST membership
If a node is not in the walked zonelist, it's unreachable.

2) __GFP_THISNODE
If this flag is set and the only node in the ZONELIST is the singular preferred node (or the local node, for -1)

3) cpuset.mems membership
cpuset trims any node in its allowed list

4) mempolicy nodemask
the allocator will skip any node not in the nodemask.

Except for #1 (Zonelist membership) there are all kinds of weird corner conditions in which 2-4 can be completely ignored (interrupt context, empty set because cpuset doesn't intersect mempolicy, shared vma, ...)

But ZONELIST membership is *absolute*. If a zone is not in the zonelist being walked, IT CANNOT BE ALLOCATED FROM. PERIOD.

The existing zonelists are constructed like so:
ZONELIST_FALLBACK : All N_MEMORY nodes
ZONELIST_NOFALLBACK : A singleton N_MEMORY node

Private node isolation is implemented via ZONELIST isolation:
ZONELIST_PRIVATE : The private node + N_MEMORY
ZONELIST_PRIVATE_NOFALLBACK : The private node alone.

(mirrors exactly the fallback/nofallback for __GFP_THISNODE)

Private nodes:
1) Never appear in any ZONELIST_FALLBACK
2) Have an empty ZONELIST_NOFALLBACK
3) Only appear in their own ZONELIST_PRIVATE(_NOFALLBACK)

1 & 2 mean all existing in-tree callers to page_alloc can NEVER accidentally allocate from a private node.

An allocation must explicitly ask via a zonelist and a nodemask.

alloc_flags |= ALLOC_ZONELIST_PRIVATE; /* use ZONELIST_PRIVATE */
__alloc_pages(..., nodemask); /* with the private node set */

The page allocator keeps all its original interfaces which only ever touch the default zonelists - avoiding churn.

Making it accessible via Mempolicy: MPOL_F_PRIVATE and page_alloc
=================================================================
The vast majority of the kernel will never need to know about ZONELIST_PRIVATE, because we add MPOL_F_PRIVATE to mempolicy.

When a mempolicy has MPOL_F_PRIVATE, the alloc_mpol() interfaces do the zonelist selection for the source of the allocation.

That really is the whole explanation of the mechanism:

  alloc_flags = mpol_alloc_flags(pol);
  page = __alloc_frozen_pages_noprof(..., alloc_flags);

On mm-new this rides the allocator's existing alloc_flags plumbing:
ALLOC_ZONELIST_PRIVATE is just another alloc_flag, so no new parameter, enum, or alloc_context change is required.

For modules that want to implement their own special handling, they get the _private variants for the page allocator. This lets modules re-use the buddy instead of rewriting it.

- alloc_pages_node_private_noprof()
- folio_alloc_node_private_noprof()

This keeps ALLOC_ flags mm/ internal (these functions add the flags).

Isolating private node folios from kernel services
==================================================
We implement filter points in mm/ to prevent operations on private node memory. Where possible, we even re-use existing filter points from ZONE_DEVICE.

Most filter points are one or two lines of code:

Combining ZONE_DEVICE and N_MEMORY_PRIVATE opt-out spots:
- if (folio_is_zone_device(folio))
+ if (unlikely(folio_is_private_managed(folio)))

Disabling a service:
+ if (!node_is_private(nid)) {
+ kswapd_run(nid);
+ kcompactd_run(nid);
+ }

Disallowing a uapi interaction:
+ if (node_state(nid, N_MEMORY_PRIVATE))
+ return -EINVAL;

In the second half of the series, we replace blanket N_MEMORY_PRIVATE filters with NODE_PRIVATE_CAP_* filters to opt those nodes into that interaction if CAP is set.

We abstract this with a nice clean interface to make it really clear what is happening (nodes have features!)

- if (node_state(pgdat->node_id, N_MEMORY_PRIVATE))
+ if (!node_allows_reclaim(pgdat->node_id))

NODE_PRIVATE_CAP_* features
===========================
This series of commits opts private nodes into various mm/ services.

Capabilities:

  NODE_PRIVATE_CAP_RECLAIM       - direct and kswapd reclaim
  NODE_PRIVATE_CAP_USER_NUMA     - userland numa controls
  NODE_PRIVATE_CAP_DEMOTION      - node is a demotion target
  NODE_PRIVATE_CAP_HOTUNPLUG     - hotunplug may migrate

NODE_PRIVATE_CAP_NUMA_BALANCING - NUMAB may target node folios
NODE_PRIVATE_CAP_LTPIN - Longterm pin operates normally

Some opt-in support is more intensive than others, so these features are broken out in a way that we can defer them as future work streams.

NODE_PRIVATE_CAP_RECLAIM:
Enabling reclaim for these nodes is actually surprisingly trivial.

Without CAP_RECLAIM, when an allocation failure occurs, the system will not attempt to swap the memory - and instead will OOM (typically whatever task is using the most memory on *that* private node).

This capability consists of:
1) enabling kswapd and kcompactd for that node at hotplug time.
2) formalizing opt-out hooks to node_allows_reclaim() opt-in hooks.
3) Sets normal watermarks for these nodes.
4) Allow madvise operations on that node (PAGEOUT).
5) A small tweak to how LRU decides which zones to visit.

NODE_PRIVATE_CAP_USER_NUMA
Enables the following userland interfaces to accept the node:

    mbind()
    set_mempolicy()
    set_mempolicy_home_node()
    move_pages()
    migrate_pages()
  example:
     buf = mmap(..., MAP_ANON);
     mbind(buf, ..., {private_node});
     buf[0] = 0xDEADBEEF; /* Page faults onto the private node */

Later - the KVM example shows how in-kernel mempolicies can also be bound by CAP_USER_NUMA.

Otherwise, that's it - it's just a mempolicy with MPOL_F_PRIVATE.

NODE_PRIVATE_CAP_HOTUNPLUG
This is simple: allow hotunplug to migrate this nodes folios.

Some devices may not be able to tolerate unexpected migrations, so we prevent hotunplug from engaging in migration by default.

Some devices may have an mmu_notifier in their driver that can manage the migration and subsequent refault.

CAP_HOTUNPLUG allows memory_hotplug.c to migrate normally.

NODE_PRIVATE_CAP_DEMOTION
This adds the private node as a valid demotion target, and allows reclaim to demote memory from a private node to a demotion target.

Requires: NODE_PRIVATE_CAP_RECLAIM

NODE_PRIVATE_CAP_NUMA_BALANCING
This enables numa balancing to inject prot_none on private node folio mappings and promote them when faults are taken.

NODE_PRIVATE_CAP_LTPIN
This allows GUP Longterm Pinning to operate normally.

Normally, longterm pinning determines folio eligibility based on its ZONE_* membership (among other things).

ZONE_NORMAL is eligible, while ZONE_MOVABLE folios require migration to ZONE_NORMAL before pinning.

Neither operation is preferable by default on a private node, so the base behavior of FOLL_LONGTERM is to FAIL.

This capability allows longterm pinning to operate normally based on the ZONE membership. Private node memory may be hotplugged as either ZONE_NORMAL or ZONE_MOVABLE.

In-tree User: KVM
=================
Dave Jiang proposed [2] dax-backed guest_memfd() memory as a way of enabling disaggregated memory pools to host dedicated KVM memory.

With private nodes, this is trivial (with a bit of basic plumbing):

static int kvm_gmem_bind_node(struct inode *inode, int node) {
...
/* Bind to a private node - gated on CAP_USER_NUMA */ pol = mpol_bind_node(node);
if (IS_ERR(pol))

    return PTR_ERR(pol);

/* Set the shared policy */
err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, ..., pol);
...
}

KVM doesn't even need to know about private nodes at all, all it does is ask mempolicy whether the requested node is a valid bind.

mm/ component testing with dax driver
=====================================
The dax driver extensions[1] implements a simple interface to create a private node from a dax device created by any source.

I left the dax driver extensions out of this feature set because it locks in the CAP_ bits before anyone has input. It's there primarily for testing at this point.

The simplest way to get a dax device is with the memmap= boot arg.
e.g.: "memmap=0x40000000!0x140000000"

The dax driver extension has the following sysfs entries:

  dax0.0/private         - set the node to private
  dax0.0/dax_file        - make /dev/dax0.0 mmap'able in kmem mode
  dax0.0/adistance       - dictate memory_tierN membership
  dax0.0/reclaim         - CAP_RECLAIM
  dax0.0/demotion        - CAP_DEMOTION
  dax0.0/user_numa       - CAP_USER_NUMA
  dax0.0/hotunplug       - CAP_HOTUNPLUG
  dax0.0/numa_balancing  - CAP_NUMA_BALANCING
  dax0.0/ltpin           - CAP_LTPIN

Now consider the following...

Single node reclaim + mbind support:
echo 1 > dax0.0/private
echo 1 > dax0.0/reclaim
echo 1 > dax0.0/user_numa
echo online_movable > dax0.0/state

Test program:
/* node1: 1GB Private Memory Node, 4GB swap */ buf = mmap(NULL, TWO_GB, PROT_READ | PROT_WRITE,

              MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);

sys_mbind(p, len, MPOL_BIND, mask, MAXNODE, MPOL_MF_STRICT); memset(buf, 0xff, TWO_GB);

We can *guarantee* the ONLY reclaiming tasks are exactly:
- kswapdN (in theory we can even make this optional!)
- the memset task faulting pages in

This also means these are the only tasks capable of becoming locked up and OOMing (as long as it is not under pressure).

The rest of the system remains entirely functional for debugging.

It now becomes possible to micro-benchmark and A/B test reclaim changes with different scenarios (number of tasks, amount of memory, watermark targets, etc), because we have hard controls over exactly which tasks can access that node memory and how.

If we broke CAP_RECLAIM into subflags:
- CAP_RECLAIM_KSWAPD
- CAP_RECLAIM_DIRECT
- CAP_COMPACTION_KCOMPACTD
- CAP_COMPACTION_DIRECT

We can actually test the efficacy of each of these mechanisms in isolation to each other - something that is strictly impossible today.

Bonus Configuration: HBM device memory tiering
==============================================

echo 1 > dax0.0/private       # make the node private
echo 0 > dax0.0/adistance     # highest tier
echo 1 > dax0.0/reclaim       # reclaim active
echo 1 > dax0.0/demotion      # may demote from the node
echo 1 > dax0.0/user_numa     # mbind()

echo online_movable > dax0.0/state
echo 1 > numa/demotion_enabled

This is an HBM device which is treated as the top-tier in the system but for which memory can only enter via explicit mbind().

It can be overcommitted because it can be reclaimed (demotions go to CPU DRAM, and reclaim can swap from it).

If the HBM is managed by an accelerator (GPU), the mmu_notifier allows it to know when reclaim is moving memory out to do device-mmu invalidation prior to migration.

Prereqs, base commit, references
================================
akpm/mm-new - for Brendan Jackman's mm/page_alloc.h work[3]

Prereqs (all already in akpm/mm-new; listed for out-of-tree application):

page_alloc.h split + alloc_flags plumbing this series rides on:
commit e81fae43cd69 ("mm: split out internal page_alloc.h") commit b4ff3b6d0a1d ("mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG")

dax atomic whole-device hotplug (used by the dax extension [1]):
commit d7aa81b9a919 ("mm/memory: add memory_block_aligned_range() helper") commit 3b2f402a1754 ("dax/kmem: add sysfs interface for atomic whole-device hotplug")

[1] https://github.com/gourryinverse/linux/tree/scratch/gourry/managed_nodes/dax_private-mm-new
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

base-commit: c872b70f5d6c742ad34b8e838c92af81c8920b3e

Gregory Price (36):
mm: refactor find_next_best_node to find_next_best_node_in mm/page_alloc: refactor build_node_zonelist() out of build_zonelists() mm/page_alloc: let the bulk and folio allocators carry alloc_flags
numa: introduce N_MEMORY_PRIVATE
mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
cpuset: exclude private nodes from cpuset.mems (default-open) mm/memory_hotplug: disallow migration-driven private node hotunplug mm/mempolicy: skip private node folios when queueing for migration mm/migrate: disallow userland driven migration for private nodes mm/madvise: disallow madvise operations on private node folios mm/compaction: disallow compaction on private nodes mm/page_alloc: clear private node watermarks and system reserves mm/mempolicy: disallow NUMA Balancing prot_none on private nodes mm/damon: skip private node memory in DAMON migration and pageout mm/ksm: skip KSM for managed-memory folios
mm/khugepaged: skip private node folios when trying to collapse. mm/vmscan: disallow reclaim of private node memory
mm/gup: disallow longterm pin of private node folios
proc: include N_MEMORY_PRIVATE nodes in numa_maps output mm/memcontrol: account private-node memory in per-node stats proc/kcore: include private-node RAM in the kcore RAM map mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection mm/mempolicy: apply policy at the kernel zone for private-node binds mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug
mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA
balancing
mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning mm/khugepaged: base private node collapse eligiblity on actor/cap bits Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes mm/mempolicy: add mpol_set_shared_policy_range()
KVM: guest_memfd: bind backing memory to a NUMA node at creation
KVM: selftests: add a guest_memfd FLAG_BIND_NODE test

 Documentation/ABI/stable/sysfs-devices-node   |  10 +
 Documentation/mm/index.rst                    |   1 +
 Documentation/mm/numa_private_nodes.rst       | 160 ++++++++++
 drivers/base/node.c                           | 118 +++++++
 drivers/dax/kmem.c                            |   2 +-
 fs/proc/kcore.c                               |   5 +-
 fs/proc/task_mmu.c                            |  10 +-
 include/linux/gfp.h                           |  22 +-
 include/linux/kvm_host.h                      |   3 +
 include/linux/memory_hotplug.h                |   5 +-
 include/linux/mempolicy.h                     |  16 +
 include/linux/mmzone.h                        |  26 +-
 include/linux/node_private.h                  | 262 ++++++++++++++++
 include/linux/nodemask.h                      |   7 +-
 include/uapi/linux/kvm.h                      |   5 +-
 include/uapi/linux/mempolicy.h                |   1 +
 kernel/cgroup/cpuset.c                        |  26 +-
 mm/compaction.c                               |  13 +
 mm/damon/paddr.c                              |   9 +
 mm/gup.c                                      |  28 +-
 mm/huge_memory.c                              |   5 +
 mm/internal.h                                 |  97 +++++-
 mm/khugepaged.c                               |  18 +-
 mm/ksm.c                                      |   8 +-
 mm/madvise.c                                  |   8 +-
 mm/memcontrol-v1.c                            |   8 +-
 mm/memcontrol.c                               |  13 +-
 mm/memory-tiers.c                             |  40 ++-
 mm/memory_hotplug.c                           | 119 +++++++-
 mm/mempolicy.c                                | 288 +++++++++++++++---
 mm/migrate.c                                  |  19 +-
 mm/mm_init.c                                  |   2 +-
 mm/page_alloc.c                               | 189 +++++++++---
 mm/page_alloc.h                               |  34 +++
 mm/vmscan.c                                   |  57 +++-
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../kvm/guest_memfd_bind_node_test.c          | 213 +++++++++++++
 virt/kvm/guest_memfd.c                        |  39 ++-
 38 files changed, 1728 insertions(+), 160 deletions(-)
 create mode 100644 Documentation/mm/numa_private_nodes.rst
 create mode 100644 include/linux/node_private.h
 create mode 100644 tools/testing/selftests/kvm/guest_memfd_bind_node_test.c
Signature · 1 line
2.53.0-Meta
author maintainer
#1 lore raw

PATCH v5 1/36 mm: refactor find_next_best_node to find_next_best_node_in

find_next_best_node() picks the next-closest node for a fallback list from the full N_MEMORY set. Refactor it into find_next_best_node_in(), which takes an explicit candidates nodemask.

This enables building fallback lists with non-N_MEMORY candidates.

No functional change: every caller still selects from N_MEMORY.

---
 mm/internal.h     |  6 ++++--
 mm/memory-tiers.c |  7 ++++---
 mm/page_alloc.c   | 13 ++++++++-----
 3 files changed, 16 insertions(+), 10 deletions(-)
mm/internal.h +4 -2
index f26423de4ca28..96d78a7778e88 100644
@@ -1103,7 +1103,8 @@ extern int node_reclaim_mode;
 
 extern unsigned long node_reclaim(struct pglist_data *pgdat,
 				  gfp_t gfp_mask, unsigned int order);
-extern int find_next_best_node(int node, nodemask_t *used_node_mask);
+extern int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+				  const nodemask_t *candidates);
 #else
 #define node_reclaim_mode 0
 
@@ -1112,7 +1113,8 @@ static inline unsigned long node_reclaim(struct pglist_data *pgdat,
 {
 	return 0;
 }
-static inline int find_next_best_node(int node, nodemask_t *used_node_mask)
+static inline int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+					 const nodemask_t *candidates)
 {
 	return NUMA_NO_NODE;
 }
mm/memory-tiers.c +4 -3
index 54851d8a195b0..25e121851b586 100644
@@ -370,7 +370,7 @@ int next_demotion_node(int node, const nodemask_t *allowed_mask)
 	 * closest demotion target.
 	 */
 	nodes_complement(mask, *allowed_mask);
-	return find_next_best_node(node, &mask);
+	return find_next_best_node_in(node, &mask, &node_states[N_MEMORY]);
 }
 
 static void disable_all_demotion_targets(void)
@@ -450,7 +450,7 @@ static void establish_demotion_targets(void)
 		memtier = list_next_entry(memtier, list);
 		tier_nodes = get_memtier_nodemask(memtier);
 		/*
-		 * find_next_best_node, use 'used' nodemask as a skip list.
+		 * find_next_best_node_in, use 'used' nodemask as a skip list.
 		 * Add all memory nodes except the selected memory tier
 		 * nodelist to skip list so that we find the best node from the
 		 * memtier nodelist.
@@ -463,7 +463,8 @@ static void establish_demotion_targets(void)
 		 * in the preferred mask when allocating pages during demotion.
 		 */
 		do {
-			target = find_next_best_node(node, &tier_nodes);
+			target = find_next_best_node_in(node, &tier_nodes,
+							&node_states[N_MEMORY]);
 			if (target == NUMA_NO_NODE)
 				break;
 
mm/page_alloc.c +8 -5
index f93a6bb9a872d..acf096f525f49 100644
@@ -5743,9 +5743,10 @@ static int numa_zonelist_order_handler(const struct ctl_table *table, int write,
 static int node_load[MAX_NUMNODES];
 
 /**
- * find_next_best_node - find the next node that should appear in a given node's fallback list
+ * find_next_best_node_in - find the next node that should appear in a given node's fallback list
  * @node: node whose fallback list we're appending
  * @used_node_mask: nodemask_t of already used nodes
+ * @candidates: nodemask_t of nodes eligible for selection
  *
  * We use a number of factors to determine which is the next node that should
  * appear on a given node's fallback list.  The node should not have appeared
@@ -5757,7 +5758,8 @@ static int node_load[MAX_NUMNODES];
  *
  * Return: node id of the found node or %NUMA_NO_NODE if no node is found.
  */
-int find_next_best_node(int node, nodemask_t *used_node_mask)
+int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+			   const nodemask_t *candidates)
 {
 	int n, val;
 	int min_val = INT_MAX;
@@ -5767,12 +5769,12 @@ int find_next_best_node(int node, nodemask_t *used_node_mask)
 	 * Use the local node if we haven't already, but for memoryless local
 	 * node, we should skip it and fall back to other nodes.
 	 */
-	if (!node_isset(node, *used_node_mask) && node_state(node, N_MEMORY)) {
+	if (!node_isset(node, *used_node_mask) && node_isset(node, *candidates)) {
 		node_set(node, *used_node_mask);
 		return node;
 	}
 
-	for_each_node_state(n, N_MEMORY) {
+	for_each_node_mask(n, *candidates) {
 
 		/* Don't want a node to appear more than once */
 		if (node_isset(n, *used_node_mask))
@@ -5857,7 +5859,8 @@ static void build_zonelists(pg_data_t *pgdat)
 	prev_node = local_node;
 
 	memset(node_order, 0, sizeof(node_order));
-	while ((node = find_next_best_node(local_node, &used_mask)) >= 0) {
+	while ((node = find_next_best_node_in(local_node, &used_mask,
+					      &node_states[N_MEMORY])) >= 0) {
 		/*
 		 * We don't want to pressure a particular node.
 		 * So adding penalty to the first node in same
Signature · 1 line
2.53.0-Meta
Acked-by on v5 1/36
#2 lore raw

Re: [PATCH v5 01/36] mm: refactor find_next_best_node to find_next_best_node_in

Quoted 113 lines · Gregory Price

find_next_best_node() picks the next-closest node for a fallback list from the full N_MEMORY set. Refactor it into find_next_best_node_in(), which takes an explicit candidates nodemask.

This enables building fallback lists with non-N_MEMORY candidates.

No functional change: every caller still selects from N_MEMORY.

---
 mm/internal.h     |  6 ++++--
 mm/memory-tiers.c |  7 ++++---
 mm/page_alloc.c   | 13 ++++++++-----
 3 files changed, 16 insertions(+), 10 deletions(-)
mm/internal.h +4 -2
index f26423de4ca28..96d78a7778e88 100644
@@ -1103,7 +1103,8 @@ extern int node_reclaim_mode;
 
 extern unsigned long node_reclaim(struct pglist_data *pgdat,
 				  gfp_t gfp_mask, unsigned int order);
-extern int find_next_best_node(int node, nodemask_t *used_node_mask);
+extern int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+				  const nodemask_t *candidates);
 #else
 #define node_reclaim_mode 0
 
@@ -1112,7 +1113,8 @@ static inline unsigned long node_reclaim(struct pglist_data *pgdat,
 {
 	return 0;
 }
-static inline int find_next_best_node(int node, nodemask_t *used_node_mask)
+static inline int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+					 const nodemask_t *candidates)
 {
 	return NUMA_NO_NODE;
 }
mm/memory-tiers.c +4 -3
index 54851d8a195b0..25e121851b586 100644
@@ -370,7 +370,7 @@ int next_demotion_node(int node, const nodemask_t *allowed_mask)
 	 * closest demotion target.
 	 */
 	nodes_complement(mask, *allowed_mask);
-	return find_next_best_node(node, &mask);
+	return find_next_best_node_in(node, &mask, &node_states[N_MEMORY]);
 }
 
 static void disable_all_demotion_targets(void)
@@ -450,7 +450,7 @@ static void establish_demotion_targets(void)
 		memtier = list_next_entry(memtier, list);
 		tier_nodes = get_memtier_nodemask(memtier);
 		/*
-		 * find_next_best_node, use 'used' nodemask as a skip list.
+		 * find_next_best_node_in, use 'used' nodemask as a skip list.
 		 * Add all memory nodes except the selected memory tier
 		 * nodelist to skip list so that we find the best node from the
 		 * memtier nodelist.
@@ -463,7 +463,8 @@ static void establish_demotion_targets(void)
 		 * in the preferred mask when allocating pages during demotion.
 		 */
 		do {
-			target = find_next_best_node(node, &tier_nodes);
+			target = find_next_best_node_in(node, &tier_nodes,
+							&node_states[N_MEMORY]);
 			if (target == NUMA_NO_NODE)
 				break;
 
mm/page_alloc.c +6 -4
index f93a6bb9a872d..acf096f525f49 100644
@@ -5743,9 +5743,10 @@ static int numa_zonelist_order_handler(const struct ctl_table *table, int write,
 static int node_load[MAX_NUMNODES];
 
 /**
- * find_next_best_node - find the next node that should appear in a given node's fallback list
+ * find_next_best_node_in - find the next node that should appear in a given node's fallback list
  * @node: node whose fallback list we're appending
  * @used_node_mask: nodemask_t of already used nodes
+ * @candidates: nodemask_t of nodes eligible for selection
  *
  * We use a number of factors to determine which is the next node that should
  * appear on a given node's fallback list.  The node should not have appeared
@@ -5757,7 +5758,8 @@ static int node_load[MAX_NUMNODES];
  *
  * Return: node id of the found node or %NUMA_NO_NODE if no node is found.
  */
-int find_next_best_node(int node, nodemask_t *used_node_mask)
+int find_next_best_node_in(int node, nodemask_t *used_node_mask,
+			   const nodemask_t *candidates)
 {
 	int n, val;
 	int min_val = INT_MAX;
@@ -5767,12 +5769,12 @@ int find_next_best_node(int node, nodemask_t *used_node_mask)
 	 * Use the local node if we haven't already, but for memoryless local
 	 * node, we should skip it and fall back to other nodes.
 	 */
-	if (!node_isset(node, *used_node_mask) && node_state(node, N_MEMORY)) {
+	if (!node_isset(node, *used_node_mask) && node_isset(node, *candidates)) {
 		node_set(node, *used_node_mask);
 		return node;
 	}
 
-	for_each_node_state(n, N_MEMORY) {
+	for_each_node_mask(n, *candidates) {
 
 		/* Don't want a node to appear more than once */
 		if (node_isset(n, *used_node_mask))

mm/page_alloc.c static void build_zonelists(pg_data_t *pgdat) · line 5866

Quoted 9 lines
mm/page_alloc.c +2 -1
@@ -5857,7 +5859,8 @@ static void build_zonelists(pg_data_t *pgdat)
 	prev_node = local_node;
 
 	memset(node_order, 0, sizeof(node_order));
-	while ((node = find_next_best_node(local_node, &used_mask)) >= 0) {
+	while ((node = find_next_best_node_in(local_node, &used_mask,
+					      &node_states[N_MEMORY])) >= 0) {
 		/*
 		 * We don't want to pressure a particular node.
 		 * So adding penalty to the first node in same

Looks good!

author maintainer
#3 lore raw

PATCH v5 2/36 mm/page_alloc: refactor build_node_zonelist() out of build_zonelists()

In reply to Gregory Price

Extract per-node fallback-list construction into build_node_zonelist().

This lets use build zonelists from candidate nodemasks instead of just the default N_MEMORY node state list.

No functional change: build_zonelists() builds the same FALLBACK list over N_MEMORY with node_load updates as before.

---
 mm/page_alloc.c | 44 ++++++++++++++++++++++++++++++--------------
 1 file changed, 30 insertions(+), 14 deletions(-)
mm/page_alloc.c +30 -14
index acf096f525f49..4c1ef84c02a3d 100644
@@ -5813,12 +5813,12 @@ int find_next_best_node_in(int node, nodemask_t *used_node_mask,
  * DMA zone, if any--but risks exhausting DMA zone.
  */
 static void build_zonelists_in_node_order(pg_data_t *pgdat, int *node_order,
-		unsigned nr_nodes)
+		unsigned int nr_nodes, int zlidx)
 {
 	struct zoneref *zonerefs;
 	int i;
 
-	zonerefs = pgdat->node_zonelists[ZONELIST_FALLBACK]._zonerefs;
+	zonerefs = pgdat->node_zonelists[zlidx]._zonerefs;
 
 	for (i = 0; i < nr_nodes; i++) {
 		int nr_zones;
@@ -5847,26 +5847,28 @@ static void build_thisnode_zonelists(pg_data_t *pgdat)
 	zonerefs->zone_idx = 0;
 }
 
-static void build_zonelists(pg_data_t *pgdat)
+/*
+ * Build one node-ordered fallback list from a candidate nodemask.
+ * update_load round-robins node_load across equidistant nodes.
+ */
+static void build_node_zonelist(pg_data_t *pgdat, const nodemask_t *candidates,
+				int zlidx, bool update_load,
+				int *node_order, int *nr)
 {
-	static int node_order[MAX_NUMNODES];
-	int node, nr_nodes = 0;
 	nodemask_t used_mask = NODE_MASK_NONE;
-	int local_node, prev_node;
-
-	/* NUMA-aware ordering of nodes */
-	local_node = pgdat->node_id;
-	prev_node = local_node;
+	int local_node = pgdat->node_id;
+	int prev_node = local_node;
+	int node, nr_nodes = 0;
 
-	memset(node_order, 0, sizeof(node_order));
 	while ((node = find_next_best_node_in(local_node, &used_mask,
-					      &node_states[N_MEMORY])) >= 0) {
+					      candidates)) >= 0) {
 		/*
 		 * We don't want to pressure a particular node.
 		 * So adding penalty to the first node in same
 		 * distance group to make it round-robin.
 		 */
-		if (node_distance(local_node, node) !=
+		if (update_load &&
+		    node_distance(local_node, node) !=
 		    node_distance(local_node, prev_node))
 			node_load[node] += 1;
 
@@ -5874,8 +5876,22 @@ static void build_zonelists(pg_data_t *pgdat)
 		prev_node = node;
 	}
 
-	build_zonelists_in_node_order(pgdat, node_order, nr_nodes);
+	build_zonelists_in_node_order(pgdat, node_order, nr_nodes, zlidx);
+	*nr = nr_nodes;
+}
+
+static void build_zonelists(pg_data_t *pgdat)
+{
+	static int node_order[MAX_NUMNODES];
+	int local_node = pgdat->node_id;
+	int node, nr_nodes = 0;
+
+	memset(node_order, 0, sizeof(node_order));
+
+	build_node_zonelist(pgdat, &node_states[N_MEMORY], ZONELIST_FALLBACK,
+			    true, node_order, &nr_nodes);
 	build_thisnode_zonelists(pgdat);
+
 	pr_info("Fallback order for Node %d: ", local_node);
 	for (node = 0; node < nr_nodes; node++)
 		pr_cont("%d ", node_order[node]);
Signature · 1 line
2.53.0-Meta
author maintainer
#4 lore raw

PATCH v5 3/36 mm/page_alloc: let the bulk and folio allocators carry alloc_flags

In reply to Gregory Price

__alloc_pages_noprof() takes an explicit alloc_flags, but the bulk and folio entry points do not, so callers cannot select allocator behaviour (e.g. an alternate zonelist) through them.

Thread alloc_flags through both, matching __alloc_pages_noprof(), and keep the flag-carrying primitives mm-internal (page_alloc.h) so the public gfp.h wrappers stay flag-free:

- add __alloc_pages_bulk_noprof(gfp, alloc_flags, ...) in page_alloc.h alloc_pages_bulk_noprof() becomes a wrapper passing ALLOC_DEFAULT

- give __folio_alloc_noprof() an alloc_flags parameter and moves __folio_alloc_node_noprof() moves into page_alloc.c __folio_alloc_noprof() is no longer exported

No functional change: every caller passes ALLOC_DEFAULT.

---
 include/linux/gfp.h | 13 +------------
 mm/khugepaged.c     |  2 +-
 mm/migrate.c        |  2 +-
 mm/page_alloc.c     | 46 ++++++++++++++++++++++++++++++---------------
 mm/page_alloc.h     |  8 ++++++++
 5 files changed, 42 insertions(+), 29 deletions(-)
include/linux/gfp.h +1 -12
index 872bc53f32ec8..a327e58c313f8 100644
@@ -204,10 +204,6 @@ static inline void arch_free_page(struct page *page, int order) { }
 static inline void arch_alloc_page(struct page *page, int order) { }
 #endif
 
-struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid,
-		nodemask_t *nodemask);
-#define __folio_alloc(...)			alloc_hooks(__folio_alloc_noprof(__VA_ARGS__))
-
 unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
 				nodemask_t *nodemask, int nr_pages,
 				struct page **page_array);
@@ -252,14 +248,7 @@ static inline void warn_if_node_offline(int this_node, gfp_t gfp_mask)
 	dump_stack();
 }
 
-static inline
-struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
-{
-	warn_if_node_offline(nid, gfp);
-
-	return __folio_alloc_noprof(gfp, order, nid, NULL);
-}
-
+struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid);
 #define  __folio_alloc_node(...)		alloc_hooks(__folio_alloc_node_noprof(__VA_ARGS__))
 
 /*
mm/khugepaged.c +1 -1
index 27e8f3077e80f..89ce6bcbc376b 100644
@@ -1243,7 +1243,7 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
 	int node = collapse_find_target_node(cc);
 	struct folio *folio;
 
-	folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask);
+	folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, ALLOC_DEFAULT);
 	if (!folio) {
 		*foliop = NULL;
 		if (is_pmd_order(order))
mm/migrate.c +1 -1
index 60355be9f652c..a8a86141dbb54 100644
@@ -2231,7 +2231,7 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private)
 	if (is_highmem_idx(zidx) || zidx == ZONE_MOVABLE)
 		gfp_mask |= __GFP_HIGHMEM;
 
-	return __folio_alloc(gfp_mask, order, nid, mtc->nmask);
+	return __folio_alloc(gfp_mask, order, nid, mtc->nmask, ALLOC_DEFAULT);
 }
 
 #ifdef CONFIG_NUMA_MIGRATION
mm/page_alloc.c +31 -15

108 lines not on this page · raw message

mm/page_alloc.h +8 -0
index b9259deddb59d..23fc79ce97b60 100644
@@ -258,6 +258,14 @@ struct page *__alloc_pages_noprof(gfp_t gfp, unsigned int order, int preferred_n
 		nodemask_t *nodemask, unsigned int alloc_flags);
 #define __alloc_pages(...)			alloc_hooks(__alloc_pages_noprof(__VA_ARGS__))
 
+struct folio *__folio_alloc_noprof(gfp_t gfp, unsigned int order, int preferred_nid,
+		nodemask_t *nodemask, unsigned int alloc_flags);
+#define __folio_alloc(...)			alloc_hooks(__folio_alloc_noprof(__VA_ARGS__))
+
+unsigned long __alloc_pages_bulk_noprof(gfp_t gfp, unsigned int alloc_flags,
+		int preferred_nid, nodemask_t *nodemask, int nr_pages,
+		struct page **page_array);
+
 extern void zone_pcp_reset(struct zone *zone);
 extern void zone_pcp_disable(struct zone *zone);
 extern void zone_pcp_enable(struct zone *zone);
Signature · 1 line
2.53.0-Meta
#5 lore raw

Re: [PATCH v5 03/36] mm/page_alloc: let the bulk and folio allocators carry alloc_flags

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The newly added `alloc_flags` parameter is ignored during the bulk allocation fastpath when…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The newly added alloc_flags parameter is ignored during the bulk allocation fastpath when calling prep_new_page. --

--- Patch [3]: [PATCH v5 03/36] mm/page_alloc: let the bulk and folio allocators carry alloc_flags --- commit 00718b17ea28c0a7382f9831d41a8a45df59e496
Author: Gregory Price <[email protected]>

mm/page_alloc: let the bulk and folio allocators carry alloc_flags

This commit threads the alloc_flags parameter through the bulk and folio allocators to allow callers to select specific allocator behavior. It updates __alloc_pages_bulk_noprof and __folio_alloc_noprof to accept and pass these flags internally.

Quoted 3 lines
mm/page_alloc.c +0 -0

[ ... ]

mm/page_alloc.c unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid, · line 5246

Quoted 10 lines
mm/page_alloc.c +1 -1
@@ -5238,11 +5239,19 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
 	return nr_populated;
 
 failed:
-	page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask, ALLOC_DEFAULT);
+	page = __alloc_pages_noprof(gfp, 0, preferred_nid, nodemask, alloc_flags);
 	if (page)
 		page_array[nr_populated++] = page;
 	goto out;
 }

[Severity: Medium]
Will the fast path ignore the caller's new alloc_flags parameter?

In the slow path fallback above, the code correctly passes the new alloc_flags parameter to __alloc_pages_noprof(). However, in the fast path batch allocation loop just above this hunk, the pages are still initialized with a hardcoded ALLOC_DEFAULT:

__alloc_pages_bulk_noprof() {
    ...
    while (nr_populated < nr_pages) {
        ...
        page = __rmqueue_pcplist(...);
        ...
        prep_new_page(page, 0, gfp, ALLOC_DEFAULT);
        set_page_refcounted(page);
        page_array[nr_populated++] = page;
    }
}

If a caller passes custom alloc_flags to __alloc_pages_bulk_noprof(), it appears the flags will be dropped by the fast path but honored by the slow path, leading to inconsistent allocator behavior.

Should prep_new_page() in the fast path loop be updated to use the provided alloc_flags instead of ALLOC_DEFAULT?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
author maintainer
#6 lore raw

PATCH v5 4/36 numa: introduce N_MEMORY_PRIVATE

In reply to Gregory Price

Some devices want to hotplug their memory onto a node, but not have it exposed as "general purpose". The intent is to re-use portions of mm/ services for management without exposing it for general consumption.

N_MEMORY nodes are intended to contain general System RAM, so they are unsuited for this purpose.

Create N_MEMORY_PRIVATE for memory nodes whose memory is not intended for general consumption.

N_MEMORY and N_MEMORY_PRIVATE are mutually exclusive states.

This commit adds basic infrastructure for N_MEMORY_PRIVATE nodes:

- struct node_private:
Per-node container stored in NODE_DATA(nid) storing the owner.

- folio/page_is_private_node():
true if it resides on a private node

- folio/page_is_private_managed():
common predicate for checking private node or zone device Used to combine common filtering locations.

- Registration API: node_private_register()/unregister()

- sysfs attribute exposing N_MEMORY_PRIVATE node state.

- fix node_state() and helpers to return false for N_MEMORY_PRIVATE when !CONFIG_NUMA (mutually exclusive w/ N_MEMORY)

---
 Documentation/ABI/stable/sysfs-devices-node |  10 ++
 drivers/base/node.c                         | 113 ++++++++++++++++++++
 include/linux/mmzone.h                      |  15 +++
 include/linux/node_private.h                |  82 ++++++++++++++
 include/linux/nodemask.h                    |   7 +-
 mm/internal.h                               |  12 +++
 6 files changed, 236 insertions(+), 3 deletions(-)
 create mode 100644 include/linux/node_private.h
Documentation/ABI/stable/sysfs-devices-node +10 -0
index 2d0e023f22a71..8a36fbc3d4028 100644
@@ -29,6 +29,16 @@ Description:
 		Nodes that have regular or high memory.
 		Depends on CONFIG_HIGHMEM.
 
+What:		/sys/devices/system/node/has_private_memory
+Contact:	Linux Memory Management list <[email protected]>
+Description:
+		Nodes that have private (N_MEMORY_PRIVATE) memory: memory
+		hotplugged by a driver onto a CPU-less node and isolated from
+		the page allocator's normal and fallback zonelists.  A node is
+		listed here for as long as it holds private memory; it is never
+		listed in has_memory at the same time (the two states are
+		mutually exclusive).  See Documentation/mm/numa_private_nodes.rst.
+
 What:		/sys/devices/system/node/nodeX
 Date:		October 2002
 Contact:	Linux Memory Management list <[email protected]>
drivers/base/node.c +113 -0

137 lines not on this page · raw message

include/linux/mmzone.h +15 -0
index 0507193b3ae34..9815e48c03b97 100644
@@ -26,6 +26,8 @@
 #include <linux/sizes.h>
 #include <asm/page.h>
 
+struct node_private;
+
 /* Free memory management - zoned buddy allocator.  */
 #ifndef CONFIG_ARCH_FORCE_MAX_ORDER
 #define MAX_PAGE_ORDER 10
@@ -1596,12 +1598,25 @@ typedef struct pglist_data {
 	atomic_long_t		vm_stat[NR_VM_NODE_STAT_ITEMS];
 #ifdef CONFIG_NUMA
 	struct memory_tier __rcu *memtier;
+	struct node_private __rcu *node_private;
 #endif
 #ifdef CONFIG_MEMORY_FAILURE
 	struct memory_failure_stats mf_stats;
 #endif
 } pg_data_t;
 
+#ifdef CONFIG_NUMA
+static inline bool pgdat_is_private(pg_data_t *pgdat)
+{
+	return !!pgdat->node_private;
+}
+#else
+static inline bool pgdat_is_private(pg_data_t *pgdat)
+{
+	return false;
+}
+#endif
+
 #define node_present_pages(nid)	(NODE_DATA(nid)->node_present_pages)
 #define node_spanned_pages(nid)	(NODE_DATA(nid)->node_spanned_pages)
 
include/linux/node_private.h new file +82 -0

82 lines not on this page · raw message

include/linux/nodemask.h +4 -3
index b842aa5255464..ba3e6c570a112 100644
@@ -391,6 +391,7 @@ enum node_states {
 	N_HIGH_MEMORY = N_NORMAL_MEMORY,
 #endif
 	N_MEMORY,		/* The node has memory(regular, high, movable) */
+	N_MEMORY_PRIVATE,	/* The node's memory is private */
 	N_CPU,		/* The node has one or more cpus */
 	N_GENERIC_INITIATOR,	/* The node has one or more Generic Initiators */
 	NR_NODE_STATES
@@ -457,7 +458,7 @@ static __always_inline void node_set_offline(int nid)
 
 static __always_inline int node_state(int node, enum node_states state)
 {
-	return node == 0;
+	return node == 0 && state != N_MEMORY_PRIVATE;
 }
 
 static __always_inline void node_set_state(int node, enum node_states state)
@@ -470,11 +471,11 @@ static __always_inline void node_clear_state(int node, enum node_states state)
 
 static __always_inline int num_node_state(enum node_states state)
 {
-	return 1;
+	return state == N_MEMORY_PRIVATE ? 0 : 1;
 }
 
 #define for_each_node_state(node, __state) \
-	for ( (node) = 0; (node) == 0; (node) = 1)
+	for ((node) = 0; (node) == 0 && (__state) != N_MEMORY_PRIVATE; (node) = 1)
 
 #define first_online_node	0
 #define first_memory_node	0
mm/internal.h +12 -0
index 96d78a7778e88..cb9f4a8342e32 100644
@@ -12,6 +12,7 @@
 #include <linux/mm.h>
 #include <linux/mm_inline.h>
 #include <linux/mmu_notifier.h>
+#include <linux/node_private.h>
 #include <linux/pagemap.h>
 #include <linux/pagewalk.h>
 #include <linux/rmap.h>
@@ -98,6 +99,17 @@ extern int sysctl_min_unmapped_ratio;
 extern int sysctl_min_slab_ratio;
 #endif
 
+/* folio_is_private_managed() - folio has special mm management rules */
+static inline bool folio_is_private_managed(struct folio *folio)
+{
+	return folio_is_zone_device(folio) || folio_is_private_node(folio);
+}
+
+static inline bool page_is_private_managed(struct page *page)
+{
+	return folio_is_private_managed(page_folio(page));
+}
+
 /*
  * Maintains state across a page table move. The operation assumes both source
  * and destination VMAs already exist and are specified by the user.
Signature · 1 line
2.53.0-Meta
#7 lore raw

Re: [PATCH v5 04/36] numa: introduce N_MEMORY_PRIVATE

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] Missing bounds validation on `nid` in `node_private_register()` and `node_private_unregister()`…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Missing bounds validation on nid in node_private_register() and node_private_unregister() leads to out-of-bounds reads.
- [Medium] Lockless read of an __rcu pointer in pgdat_is_private() without using rcu_access_pointer().
--

--- Patch [4]: [PATCH v5 04/36] numa: introduce N_MEMORY_PRIVATE --- commit 69b3cbf2395b52c525233c59376e585ae83c3f31
Author: Gregory Price <[email protected]>

numa: introduce N_MEMORY_PRIVATE

Some devices want to hotplug their memory onto a node, but not have it exposed as "general purpose". This commit adds basic infrastructure for N_MEMORY_PRIVATE nodes, intended for memory isolated from the page allocator's normal and fallback zonelists.

Quoted 4 lines
drivers/base/node.c +0 -0
index 3da91929ad4e3..94cd51f51b7e8 100644

[ ... ]

Quoted 10 lines
drivers/base/node.c +6 -0
@@ -868,6 +869,116 @@ void register_memory_blocks_under_node_hotplug(int nid, unsigned long start_pfn,
 			   (void *)&nid, register_mem_block_under_node_hotplug);
 	return;
 }
+
+static DEFINE_MUTEX(node_private_lock);
+
+/**
+ * node_private_register - Register a private node
+ * @nid: Node identifier

[ ... ]

drivers/base/node.c void register_memory_blocks_under_node_hotplug(int nid, unsigned long start_pfn, · line 897

Quoted 9 lines

+ */

drivers/base/node.c +8 -0
+int node_private_register(int nid, struct node_private *np)
+{
+	struct node_private *existing;
+	pg_data_t *pgdat;
+	int ret = 0;
+
+	if (!np || !node_possible(nid))
+		return -EINVAL;

[Severity: Medium]
Could this lead to an out-of-bounds read if a driver passes an invalid nid like NUMA_NO_NODE (-1)?

node_possible() and NODE_DATA() do not perform bounds checking internally. Should we verify nid >= 0 && nid < MAX_NUMNODES before using it?

Quoted 11 lines
drivers/base/node.c +11 -0
+
+	mutex_lock(&node_private_lock);
+	mem_hotplug_begin();
+
+	/* N_MEMORY_PRIVATE and N_MEMORY are mutually exclusive */
+	if (node_state(nid, N_MEMORY)) {
+		ret = -EBUSY;
+		goto out;
+	}
+
+	pgdat = NODE_DATA(nid);

[ ... ]

Quoted 3 lines

+/**
+ * node_private_unregister - Unregister a private node
+ * @nid: Node identifier

[ ... ]

drivers/base/node.c void register_memory_blocks_under_node_hotplug(int nid, unsigned long start_pfn, · line 949

Quoted 8 lines

+ */

drivers/base/node.c +7 -0
+int node_private_unregister(int nid)
+{
+	struct node_private *np;
+	pg_data_t *pgdat;
+
+	if (!node_possible(nid))
+		return 0;

[Severity: Medium]
Is it possible for a driver to pass NUMA_NO_NODE here as well, leading to the same out-of-bounds read issue when node_possible() and NODE_DATA() are evaluated?

Quoted 5 lines
drivers/base/node.c +5 -0
+
+	mutex_lock(&node_private_lock);
+	mem_hotplug_begin();
+
+	pgdat = NODE_DATA(nid);

[ ... ]

Quoted 4 lines
include/linux/mmzone.h +0 -0
index 0507193b3ae34..9815e48c03b97 100644

[ ... ]

Quoted 5 lines
include/linux/mmzone.h +1 -0
@@ -1596,12 +1598,25 @@ typedef struct pglist_data {
 	atomic_long_t		vm_stat[NR_VM_NODE_STAT_ITEMS];
 #ifdef CONFIG_NUMA
 	struct memory_tier __rcu *memtier;
+	struct node_private __rcu *node_private;

include/linux/mmzone.h typedef struct pglist_data { · line 1611

Quoted 10 lines
include/linux/mmzone.h +4 -0
 #endif
 #ifdef CONFIG_MEMORY_FAILURE
 	struct memory_failure_stats mf_stats;
 #endif
 } pg_data_t;
 
+#ifdef CONFIG_NUMA
+static inline bool pgdat_is_private(pg_data_t *pgdat)
+{
+	return !!pgdat->node_private;

[Severity: Medium]
Does this lockless read of the __rcu pointer require rcu_access_pointer()?

Without it, sparse will likely complain about an __rcu address space violation, and we might be missing the READ_ONCE() barrier that prevents compiler load tearing.

Quoted 7 lines
include/linux/mmzone.h +7 -0
+}
+#else
+static inline bool pgdat_is_private(pg_data_t *pgdat)
+{
+	return false;
+}
+#endif
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
author maintainer
#8 lore raw

PATCH v5 5/36 mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.

In reply to Gregory Price

N_MEMORY_PRIVATE nodes must be invisible to ordinary allocation, no allocations may reach one unless explicitly called for.

Make this isolation structural in the zonelists: N_MEMORY_PRIVATE nodes are excluded from FALLBACK and NOFALLBACK entirely, making them effectively invisible to all normal callers of page_alloc.

Add ZONELIST_PRIVATE, which spans (N_MEMORY | N_MEMORY_PRIVATE). When !CONFIG_NUMA, it aliases ZONELIST_FALLBACK and compiles out.

Add ZONELIST_PRIVATE_NOFALLBACK as the __GFP_THISNODE target for private node allocations.

Zonelist selection rides the allocator's alloc_flags. Add ALLOC_ZONELIST_PRIVATE and resolve it in select_zonelist(), which prepare_alloc_pages() applies from ac->alloc_flags; the default (ALLOC_DEFAULT) keeps node_zonelist()'s gfp-based selection.

Add explicit private-node alloc() functions to prevent open-coding (both ride ALLOC_ZONELIST_PRIVATE through the alloc_flags-carrying entry points):
- alloc_pages_node_private_noprof()
- folio_alloc_node_private_noprof()

alloc_contig_range adds an explicit node_is_private() guard because it dervices its target zones from PFNs rather than zonelists. All in-tree callers use N_MEMORY ranges, but the check prevents future callers from violating node isolation.

Important note: By design this zonelist isolates N_MEMORY_PRIVATE from unintentional allocations, but does NOT isolate private-node allocations from falling back to non-private nodes.

---
 include/linux/gfp.h    | 11 ++++++++
 include/linux/mmzone.h | 11 +++++++-
 mm/mm_init.c           |  2 +-
 mm/page_alloc.c        | 64 ++++++++++++++++++++++++++++++++++++++++--
 mm/page_alloc.h        | 26 +++++++++++++++++
 5 files changed, 109 insertions(+), 5 deletions(-)
include/linux/gfp.h +11 -0
index a327e58c313f8..f3e867de744f6 100644
@@ -204,6 +204,17 @@ static inline void arch_free_page(struct page *page, int order) { }
 static inline void arch_alloc_page(struct page *page, int order) { }
 #endif
 
+/* Allocate from an N_MEMORY_PRIVATE node (selects the private zonelist). */
+struct page *alloc_pages_node_private_noprof(gfp_t gfp, unsigned int order,
+		int nid);
+#define alloc_pages_node_private(...)				\
+	alloc_hooks(alloc_pages_node_private_noprof(__VA_ARGS__))
+
+struct folio *folio_alloc_node_private_noprof(gfp_t gfp, unsigned int order,
+		int nid);
+#define folio_alloc_node_private(...)				\
+	alloc_hooks(folio_alloc_node_private_noprof(__VA_ARGS__))
+
 unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
 				nodemask_t *nodemask, int nr_pages,
 				struct page **page_array);
include/linux/mmzone.h +10 -1
index 9815e48c03b97..654c5111f7cec 100644
@@ -1392,13 +1392,22 @@ enum {
 #ifdef CONFIG_NUMA
 	/*
 	 * The NUMA zonelists are doubled because we need zonelists that
-	 * restrict the allocations to a single node for __GFP_THISNODE.
+	 * restrict the allocations to a single node for __GFP_THISNODE
+	 * and N_MEMORY_PRIVATE nodes (isolated from default lists).
 	 */
 	ZONELIST_NOFALLBACK,	/* zonelist without fallback (__GFP_THISNODE) */
+	ZONELIST_PRIVATE,	/* N_MEMORY_PRIVATE access, falls back to DRAM */
+	ZONELIST_PRIVATE_NOFALLBACK, /* N_MEMORY_PRIVATE access, __GFP_THISNODE */
 #endif
 	MAX_ZONELISTS
 };
 
+#ifndef CONFIG_NUMA
+/* Without NUMA only ZONELIST_FALLBACK exists so everything collapses there */
+#define ZONELIST_PRIVATE		ZONELIST_FALLBACK
+#define ZONELIST_PRIVATE_NOFALLBACK	ZONELIST_FALLBACK
+#endif
+
 /*
  * This struct contains information about a zone in a zonelist. It is stored
  * here to avoid dereferences into large structures and lookups of tables
mm/mm_init.c +1 -1
index 711f821f7b3c7..adb50647d29ca 100644
@@ -2701,7 +2701,7 @@ void __init mm_core_init(void)
 	init_zero_page_pfn();
 
 	/* Initializations relying on SMP setup */
-	BUILD_BUG_ON(MAX_ZONELISTS > 2);
+	BUILD_BUG_ON(MAX_ZONELISTS > 4);
 	build_all_zonelists(NULL);
 	page_alloc_init_cpuhp();
 	alloc_tag_sec_init();
mm/page_alloc.c +61 -3

113 lines not on this page · raw message

mm/page_alloc.h +26 -0
index 23fc79ce97b60..741448d83ce77 100644
@@ -57,6 +57,9 @@
  */
 #define ALLOC_NO_CODETAG       0x1000
 
+/* Select the N_MEMORY_PRIVATE zonelist family (see select_zonelist()). */
+#define ALLOC_ZONELIST_PRIVATE 0x2000
+
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
@@ -95,6 +98,29 @@ struct alloc_context {
 	unsigned int alloc_flags;
 };
 
+#ifdef CONFIG_NUMA
+static_assert(ZONELIST_PRIVATE + 1 == ZONELIST_PRIVATE_NOFALLBACK);
+#endif
+
+/*
+ * select_zonelist - resolve the zonelist for an allocation
+ *
+ * The default matches node_zonelist(); ALLOC_ZONELIST_PRIVATE selects the
+ * private-node family so an allocation may reach an N_MEMORY_PRIVATE node.
+ */
+static inline struct zonelist *
+select_zonelist(int nid, gfp_t gfp, unsigned int alloc_flags)
+{
+#ifdef CONFIG_NUMA
+	if (alloc_flags & ALLOC_ZONELIST_PRIVATE) {
+		int idx = ZONELIST_PRIVATE + !!(gfp & __GFP_THISNODE);
+
+		return &NODE_DATA(nid)->node_zonelists[idx];
+	}
+#endif
+	return node_zonelist(nid, gfp);
+}
+
 /*
  * This function returns the order of a free page in the buddy system. In
  * general, page_zone(page)->lock must be held by the caller to prevent the
Signature · 1 line
2.53.0-Meta
#9 lore raw

Re: [PATCH v5 05/36] mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.

Quoted 203 lines · Gregory Price

N_MEMORY_PRIVATE nodes must be invisible to ordinary allocation, no allocations may reach one unless explicitly called for.

Make this isolation structural in the zonelists: N_MEMORY_PRIVATE nodes are excluded from FALLBACK and NOFALLBACK entirely, making them effectively invisible to all normal callers of page_alloc.

Add ZONELIST_PRIVATE, which spans (N_MEMORY | N_MEMORY_PRIVATE). When !CONFIG_NUMA, it aliases ZONELIST_FALLBACK and compiles out.

Add ZONELIST_PRIVATE_NOFALLBACK as the __GFP_THISNODE target for private node allocations.

Zonelist selection rides the allocator's alloc_flags. Add ALLOC_ZONELIST_PRIVATE and resolve it in select_zonelist(), which prepare_alloc_pages() applies from ac->alloc_flags; the default (ALLOC_DEFAULT) keeps node_zonelist()'s gfp-based selection.

Add explicit private-node alloc() functions to prevent open-coding (both ride ALLOC_ZONELIST_PRIVATE through the alloc_flags-carrying entry points):
- alloc_pages_node_private_noprof()
- folio_alloc_node_private_noprof()

alloc_contig_range adds an explicit node_is_private() guard because it dervices its target zones from PFNs rather than zonelists. All in-tree callers use N_MEMORY ranges, but the check prevents future callers from violating node isolation.

Important note: By design this zonelist isolates N_MEMORY_PRIVATE from unintentional allocations, but does NOT isolate private-node allocations from falling back to non-private nodes.

---
 include/linux/gfp.h    | 11 ++++++++
 include/linux/mmzone.h | 11 +++++++-
 mm/mm_init.c           |  2 +-
 mm/page_alloc.c        | 64 ++++++++++++++++++++++++++++++++++++++++--
 mm/page_alloc.h        | 26 +++++++++++++++++
 5 files changed, 109 insertions(+), 5 deletions(-)
include/linux/gfp.h +11 -0
index a327e58c313f8..f3e867de744f6 100644
@@ -204,6 +204,17 @@ static inline void arch_free_page(struct page *page, int order) { }
 static inline void arch_alloc_page(struct page *page, int order) { }
 #endif
 
+/* Allocate from an N_MEMORY_PRIVATE node (selects the private zonelist). */
+struct page *alloc_pages_node_private_noprof(gfp_t gfp, unsigned int order,
+		int nid);
+#define alloc_pages_node_private(...)				\
+	alloc_hooks(alloc_pages_node_private_noprof(__VA_ARGS__))
+
+struct folio *folio_alloc_node_private_noprof(gfp_t gfp, unsigned int order,
+		int nid);
+#define folio_alloc_node_private(...)				\
+	alloc_hooks(folio_alloc_node_private_noprof(__VA_ARGS__))
+
 unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
 				nodemask_t *nodemask, int nr_pages,
 				struct page **page_array);
include/linux/mmzone.h +10 -1
index 9815e48c03b97..654c5111f7cec 100644
@@ -1392,13 +1392,22 @@ enum {
 #ifdef CONFIG_NUMA
 	/*
 	 * The NUMA zonelists are doubled because we need zonelists that
-	 * restrict the allocations to a single node for __GFP_THISNODE.
+	 * restrict the allocations to a single node for __GFP_THISNODE
+	 * and N_MEMORY_PRIVATE nodes (isolated from default lists).
 	 */
 	ZONELIST_NOFALLBACK,	/* zonelist without fallback (__GFP_THISNODE) */
+	ZONELIST_PRIVATE,	/* N_MEMORY_PRIVATE access, falls back to DRAM */
+	ZONELIST_PRIVATE_NOFALLBACK, /* N_MEMORY_PRIVATE access, __GFP_THISNODE */
 #endif
 	MAX_ZONELISTS
 };
 
+#ifndef CONFIG_NUMA
+/* Without NUMA only ZONELIST_FALLBACK exists so everything collapses there */
+#define ZONELIST_PRIVATE		ZONELIST_FALLBACK
+#define ZONELIST_PRIVATE_NOFALLBACK	ZONELIST_FALLBACK
+#endif
+
 /*
  * This struct contains information about a zone in a zonelist. It is stored
  * here to avoid dereferences into large structures and lookups of tables
mm/mm_init.c +1 -1
index 711f821f7b3c7..adb50647d29ca 100644
@@ -2701,7 +2701,7 @@ void __init mm_core_init(void)
 	init_zero_page_pfn();
 
 	/* Initializations relying on SMP setup */
-	BUILD_BUG_ON(MAX_ZONELISTS > 2);
+	BUILD_BUG_ON(MAX_ZONELISTS > 4);
 	build_all_zonelists(NULL);
 	page_alloc_init_cpuhp();
 	alloc_tag_sec_init();
mm/page_alloc.c +48 -3
index aa35bbe5d4c3e..78755a55b1540 100644
@@ -5040,7 +5040,7 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
 		unsigned int *alloc_flags)
 {
 	ac->highest_zoneidx = gfp_zone(gfp_mask);
-	ac->zonelist = node_zonelist(preferred_nid, gfp_mask);
+	ac->zonelist = select_zonelist(preferred_nid, gfp_mask, ac->alloc_flags);
 	ac->nodemask = nodemask;
 	ac->migratetype = gfp_migratetype(gfp_mask);
 
@@ -5346,7 +5346,7 @@ struct page *__alloc_frozen_pages_noprof(gfp_t gfp, unsigned int order,
 	unsigned int fastpath_alloc_flags = alloc_flags;
 
 	/* Other flags could be supported later if needed. */
-	if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG)))
+	if (WARN_ON(alloc_flags & ~(ALLOC_NOLOCK | ALLOC_NO_CODETAG | ALLOC_ZONELIST_PRIVATE)))
 		return NULL;
 
 	if (!alloc_order_allowed(gfp, order, alloc_flags))
@@ -5457,6 +5457,22 @@ struct folio *__folio_alloc_node_noprof(gfp_t gfp, unsigned int order, int nid)
 }
 EXPORT_SYMBOL(__folio_alloc_node_noprof);
 
+struct page *alloc_pages_node_private_noprof(gfp_t gfp, unsigned int order,
+		int nid)
+{
+	return __alloc_pages_noprof(gfp, order, nid, NULL,
+				   ALLOC_ZONELIST_PRIVATE);
+}
+EXPORT_SYMBOL_GPL(alloc_pages_node_private_noprof);
+
+struct folio *folio_alloc_node_private_noprof(gfp_t gfp, unsigned int order,
+		int nid)
+{
+	return __folio_alloc_noprof(gfp, order, nid, NULL,
+				   ALLOC_ZONELIST_PRIVATE);
+}
+EXPORT_SYMBOL_GPL(folio_alloc_node_private_noprof);
+
 /*
  * Common helper functions. Never use with __GFP_HIGHMEM because the returned
  * address cannot represent highmem pages. Use alloc_pages and then kmap if
@@ -5854,9 +5870,21 @@ static void build_zonelists_in_node_order(pg_data_t *pgdat, int *node_order,
 static void build_thisnode_zonelists(pg_data_t *pgdat)
 {
 	struct zoneref *zonerefs;
-	int nr_zones;
+	int nr_zones = 0;
 
+	/*
+	 * NOFALLBACK: the node's own zones. EMPTY for private nodes so a
+	 * stray __GFP_THISNODE allocation can't violate isolation.
+	 */
 	zonerefs = pgdat->node_zonelists[ZONELIST_NOFALLBACK]._zonerefs;
+	if (!node_is_private(pgdat->node_id))
+		nr_zones = build_zonerefs_node(pgdat, zonerefs);
+	zonerefs += nr_zones;
+	zonerefs->zone = NULL;
+	zonerefs->zone_idx = 0;
+
+	/* PRIVATE_NOFALLBACK: the node's own zones, private nodes included. */
+	zonerefs = pgdat->node_zonelists[ZONELIST_PRIVATE_NOFALLBACK]._zonerefs;
 	nr_zones = build_zonerefs_node(pgdat, zonerefs);
 	zonerefs += nr_zones;
 	zonerefs->zone = NULL;
@@ -5899,11 +5927,28 @@ static void build_node_zonelist(pg_data_t *pgdat, const nodemask_t *candidates,
 static void build_zonelists(pg_data_t *pgdat)
 {
 	static int node_order[MAX_NUMNODES];
+	nodemask_t tier_nodes;
 	int local_node = pgdat->node_id;
 	int node, nr_nodes = 0;
 
 	memset(node_order, 0, sizeof(node_order));
 
+	/*
+	 * Zonelists: FALLBACK, NOFALLBACK, PRIVATE
+	 *
+	 * FALLBACK:   Allocation order for all nodes.  Private nodes have lists
+	 *             but never appear as an entry in any list. Allocations
+	 *             targeting a private node w/ FALLBACK land on N_MEMORY.
+	 *
+	 * NOFALLBACK: A list for each node containing only itself.
+	 *             Allocations targeting a private node w/ NOFALLBACK
+	 *             will always fail (their NOFALLBACK is empty)
+	 *
+	 * PRIVATE:    (N_MEMORY | N_MEMORY_PRIVATE) - allows access to
+	 *             private nodes, and falls back to normal memory unless
+	 *             __GFP_THISNODE otherwise constrains it.
+	 */
+
 	build_node_zonelist(pgdat, &node_states[N_MEMORY], ZONELIST_FALLBACK,
 			    true, node_order, &nr_nodes);
 	build_thisnode_zonelists(pgdat);

mm/page_alloc.c static void build_zonelists(pg_data_t *pgdat) · line 5961

Quoted 5 lines
mm/page_alloc.c +2 -0
@@ -5912,6 +5957,10 @@ static void build_zonelists(pg_data_t *pgdat)
 	for (node = 0; node < nr_nodes; node++)
 		pr_cont("%d ", node_order[node]);
 	pr_cont("\n");
+
+	nodes_or(tier_nodes, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);

Hi Gregory,

A question about this part, if my understanding is right, ZONELIST_PRIVATE contains all private nodes, should we use some explicit target nodemask here to make sure only the targeting owner's private node can be touched?

Otherwise it might fall back into a different owner's private node.

--Richard

Quoted 69 lines
mm/page_alloc.c +11 -0
+	build_node_zonelist(pgdat, &tier_nodes, ZONELIST_PRIVATE, false,
+			    node_order, &nr_nodes);
 }
 
 #ifdef CONFIG_HAVE_MEMORYLESS_NODES
@@ -7282,6 +7331,15 @@ int alloc_contig_frozen_range_noprof(unsigned long start, unsigned long end,
 	if (__alloc_contig_verify_gfp_mask(gfp_mask, (gfp_t *)&cc.gfp_mask))
 		return -EINVAL;
 
+	/*
+	 * Private nodes are kept unreachable via the zonelists, but
+	 * alloc_contig works directly on a zone derived from the caller's
+	 * pfn range, bypassing that.  Reject this operation explicitly
+	 * rather than silently carving memory out of an isolated node.
+	 */
+	if (unlikely(node_is_private(zone_to_nid(cc.zone))))
+		return -EBUSY;
+
 	/*
 	 * What we do here is we mark all pageblocks in range as
 	 * MIGRATE_ISOLATE.  Because pageblock and max order pages may
mm/page_alloc.h +26 -0
index 23fc79ce97b60..741448d83ce77 100644
@@ -57,6 +57,9 @@
  */
 #define ALLOC_NO_CODETAG       0x1000
 
+/* Select the N_MEMORY_PRIVATE zonelist family (see select_zonelist()). */
+#define ALLOC_ZONELIST_PRIVATE 0x2000
+
 /* Flags that allow allocations below the min watermark. */
 #define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM)
 
@@ -95,6 +98,29 @@ struct alloc_context {
 	unsigned int alloc_flags;
 };
 
+#ifdef CONFIG_NUMA
+static_assert(ZONELIST_PRIVATE + 1 == ZONELIST_PRIVATE_NOFALLBACK);
+#endif
+
+/*
+ * select_zonelist - resolve the zonelist for an allocation
+ *
+ * The default matches node_zonelist(); ALLOC_ZONELIST_PRIVATE selects the
+ * private-node family so an allocation may reach an N_MEMORY_PRIVATE node.
+ */
+static inline struct zonelist *
+select_zonelist(int nid, gfp_t gfp, unsigned int alloc_flags)
+{
+#ifdef CONFIG_NUMA
+	if (alloc_flags & ALLOC_ZONELIST_PRIVATE) {
+		int idx = ZONELIST_PRIVATE + !!(gfp & __GFP_THISNODE);
+
+		return &NODE_DATA(nid)->node_zonelists[idx];
+	}
+#endif
+	return node_zonelist(nid, gfp);
+}
+
 /*
  * This function returns the order of a free page in the buddy system. In
  * general, page_zone(page)->lock must be held by the caller to prevent the
Signature · 1 line
2.53.0-Meta
author maintainer
#10 lore raw

Re: [PATCH v5 05/36] mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.

Quoted 38 lines · Richard Cheng
Quoted 29 lines · Gregory Price

static void build_zonelists(pg_data_t *pgdat)
{

mm/page_alloc.c +17 -0
 	static int node_order[MAX_NUMNODES];
+	nodemask_t tier_nodes;
 	int local_node = pgdat->node_id;
 	int node, nr_nodes = 0;
 
 	memset(node_order, 0, sizeof(node_order));
 
+	/*
+	 * Zonelists: FALLBACK, NOFALLBACK, PRIVATE
+	 *
+	 * FALLBACK:   Allocation order for all nodes.  Private nodes have lists
+	 *             but never appear as an entry in any list. Allocations
+	 *             targeting a private node w/ FALLBACK land on N_MEMORY.
+	 *
+	 * NOFALLBACK: A list for each node containing only itself.
+	 *             Allocations targeting a private node w/ NOFALLBACK
+	 *             will always fail (their NOFALLBACK is empty)
+	 *
+	 * PRIVATE:    (N_MEMORY | N_MEMORY_PRIVATE) - allows access to
+	 *             private nodes, and falls back to normal memory unless
+	 *             __GFP_THISNODE otherwise constrains it.
+	 */
+
 	build_node_zonelist(pgdat, &node_states[N_MEMORY], ZONELIST_FALLBACK,
 			    true, node_order, &nr_nodes);
 	build_thisnode_zonelists(pgdat);

mm/page_alloc.c static void build_zonelists(pg_data_t *pgdat) · line 5961

Quoted 5 lines
mm/page_alloc.c +2 -0
@@ -5912,6 +5957,10 @@ static void build_zonelists(pg_data_t *pgdat)
 	for (node = 0; node < nr_nodes; node++)
 		pr_cont("%d ", node_order[node]);
 	pr_cont("\n");
+
+	nodes_or(tier_nodes, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);

Hi Gregory,

Quoted 8 lines

A question about this part, if my understanding is right, ZONELIST_PRIVATE contains all private nodes, should we use some explicit target nodemask here to make sure only the targeting owner's private node can be touched?

Otherwise it might fall back into a different owner's private node.

--Richard

Zonelists and the constructors here is a bit difficult to understand, and I see I actually did a poor job of explaining what's going on in the comments here. I will fix that up in v6.

Lets say we have the following nodes:
N_MEMORY : 0 1
N_MEMORY_PRIVATE : 2 3

The following zonelists would exist:

ZONELIST_FALLBACK[0]   = {0,1}
ZONELIST_FALLBACK[1]   = {1,0}
ZONELIST_FALLBACK[2]   = {0,1}
ZONELIST_FALLBACK[3]   = {0,1}

ZONELIST_NOFALLBACK[0] = {0}
ZONELIST_NOFALLBACK[1] = {1}
ZONELIST_NOFALLBACK[2] = {}
ZONELIST_NOFALLBACK[3] = {}

ZONELIST_PRIVATE[0]    = {0,1}
ZONELIST_PRIVATE[1]    = {1,0}
ZONELIST_PRIVATE[2]    = {2,0,1}  <- no fallback to 3
ZONELIST_PRIVATE[3]    = {3,0,1}  <- no fallback to 2

ZONELIST_PRIVATE_NOFALLBACK[0] = {0} ZONELIST_PRIVATE_NOFALLBACK[1] = {1} ZONELIST_PRIVATE_NOFALLBACK[2] = {2} <- __GFP_THISNODE ZONELIST_PRIVATE_NOFALLBACK[3] = {3} <- __GFP_THISNODE

In the current design, we actually can never fallback to another private node, and instead if you wanted to - for example - interleave across said nodes you'd have to do so manually.

I think it's reasonable to later extend zonelist construction to support such a think like:

ZONELIST_PRIVATE[2] = {2,0,1,3}

Later if such a need arises, but I would say wait for the use case.

~Gregory

author maintainer
#11 lore raw

PATCH v5 6/36 cpuset: exclude private nodes from cpuset.mems (default-open)

In reply to Gregory Price

N_MEMORY_PRIVATE node access is gated by per-node capabilities and the zonelist. Isolating by cpuset offers no functionality and creates issues on rebind when a cpuset loses access to that node (in particular: it generates migrations that may not be supported).

Do not partition private nodes via cpuset.mem, instead treat them as globally accessible resources. Do not engage in silent background operations on these nodes due to cpuset.mem membership changing.

The result: cpuset operations checking for node validity always allow N_MEMORY_PRIVATE nodes.

On hot-unplug, we still need to rebind memory policies if the private node has left N_MEMORY_PRIVATE (i.e. no more memory).

---
 kernel/cgroup/cpuset.c | 26 ++++++++++++++++++++++----
 mm/memcontrol.c        |  2 ++
 2 files changed, 24 insertions(+), 4 deletions(-)
kernel/cgroup/cpuset.c +22 -4
index dfd0f827e3b92..05468f95c10bd 100644
@@ -26,6 +26,7 @@
 #include <linux/mempolicy.h>
 #include <linux/mm.h>
 #include <linux/memory.h>
+#include <linux/node_private.h>
 #include <linux/rcupdate.h>
 #include <linux/sched.h>
 #include <linux/sched/deadline.h>
@@ -3867,7 +3868,8 @@ static void cpuset_handle_hotplug(void)
 	static DECLARE_WORK(hk_sd_work, hk_sd_workfn);
 	static cpumask_t new_cpus;
 	static nodemask_t new_mems;
-	bool cpus_updated, mems_updated;
+	static nodemask_t prev_priv_mems;
+	bool cpus_updated, mems_updated, priv_shrank;
 	bool on_dfl = is_in_v2_mode();
 	struct tmpmasks tmp, *ptmp = NULL;
 
@@ -3890,6 +3892,14 @@ static void cpuset_handle_hotplug(void)
 		       !cpumask_empty(subpartitions_cpus);
 	mems_updated = !nodes_equal(top_cpuset.effective_mems, new_mems);
 
+	/*
+	 * Private nodes are not partitioned by cpuset, but if one leaves
+	 * N_MEMORY_PRIVATE we still need to run mpol_rebind_* to clean up
+	 * mempolicies that are binding them.
+	 */
+	priv_shrank = !nodes_subset(prev_priv_mems, node_states[N_MEMORY_PRIVATE]);
+	prev_priv_mems = node_states[N_MEMORY_PRIVATE];
+
 	/* For v1, synchronize cpus_allowed to cpu_active_mask */
 	if (cpus_updated) {
 		cpuset_force_rebuild();
@@ -3922,9 +3932,12 @@ static void cpuset_handle_hotplug(void)
 			top_cpuset.mems_allowed = new_mems;
 		top_cpuset.effective_mems = new_mems;
 		spin_unlock_irq(&callback_lock);
-		cpuset_update_tasks_nodemask(&top_cpuset);
 	}
 
+	/* Rebind task mempolicies if any memory node changed state */
+	if (mems_updated || priv_shrank)
+		cpuset_update_tasks_nodemask(&top_cpuset);
+
 	mutex_unlock(&cpuset_mutex);
 
 	/* if cpus or mems changed, we need to propagate to descendants */
@@ -4155,11 +4168,13 @@ nodemask_t cpuset_mems_allowed(struct task_struct *tsk)
  * cpuset_nodemask_valid_mems_allowed - check nodemask vs. current mems_allowed
  * @nodemask: the nodemask to be checked
  *
- * Are any of the nodes in the nodemask allowed in current->mems_allowed?
+ * Are any of the nodes in the nodemask usable?  N_MEMORY nodes must be in
+ * current->mems_allowed, while N_MEMORY_PRIVATE nodes are always valid.
  */
 int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask)
 {
-	return nodes_intersects(*nodemask, current->mems_allowed);
+	return nodes_intersects(*nodemask, current->mems_allowed) ||
+	       nodes_intersects(*nodemask, node_states[N_MEMORY_PRIVATE]);
 }
 
 /*
@@ -4223,6 +4238,9 @@ bool cpuset_current_node_allowed(int node, gfp_t gfp_mask)
 
 	if (in_interrupt())
 		return true;
+	/* N_MEMORY_PRIVATE nodes are not partitioned by cpusets.mems */
+	if (node_is_private(node))
+		return true;
 	if (node_isset(node, current->mems_allowed))
 		return true;
 	/*
mm/memcontrol.c +2 -0
index 8319ad8c5c23a..f0dde52dc9e0e 100644
@@ -6069,6 +6069,8 @@ void mem_cgroup_node_filter_allowed(struct mem_cgroup *memcg, nodemask_t *mask)
 	 * mask is acceptable.
 	 */
 	cpuset_nodes_allowed(memcg->css.cgroup, &allowed);
+	/* N_MEMORY_PRIVATE nodes are not partitioned by cpuset, include them */
+	nodes_or(allowed, allowed, node_states[N_MEMORY_PRIVATE]);
 	nodes_and(*mask, *mask, allowed);
 }
 
Signature · 1 line
2.53.0-Meta
#12 lore raw

Re: [PATCH v5 06/36] cpuset: exclude private nodes from cpuset.mems (default-open)

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] Tasks in descendant cpusets do not have their mempolicies rebound when a private node is…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Tasks in descendant cpusets do not have their mempolicies rebound when a private node is hot-unplugged.
- [High] Mempolicy creation (mbind, set_mempolicy) and page migration (sys_migrate_pages) unconditionally reject private nodes, breaking UAPI.
- [Medium] Valid private nodes are unconditionally stripped from tasks' memory policies during any unrelated cpuset update. --

--- Patch [6]: [PATCH v5 06/36] cpuset: exclude private nodes from cpuset.mems (default-open) --- commit 9d0d11725cd4a8e139c634034823f38bcc9b8f9f
Author: Gregory Price <[email protected]>

cpuset: exclude private nodes from cpuset.mems (default-open)

This commit modifies cpuset handling to treat N_MEMORY_PRIVATE nodes as globally accessible resources rather than partitioning them via cpuset.mems. It updates node validation checks and hotplug logic to accommodate private nodes.

Quoted 4 lines
kernel/cgroup/cpuset.c +0 -0
index dfd0f827e3b92..05468f95c10bd 100644

[ ... ]

kernel/cgroup/cpuset.c static void cpuset_handle_hotplug(void) · line 3939

Quoted 10 lines
kernel/cgroup/cpuset.c +3 -1
@@ -3922,9 +3932,12 @@ static void cpuset_handle_hotplug(void)
 			top_cpuset.mems_allowed = new_mems;
 		top_cpuset.effective_mems = new_mems;
 		spin_unlock_irq(&callback_lock);
-		cpuset_update_tasks_nodemask(&top_cpuset);
 	}
 
+	/* Rebind task mempolicies if any memory node changed state */
+	if (mems_updated || priv_shrank)
+		cpuset_update_tasks_nodemask(&top_cpuset);

[Severity: Medium]
When cpuset_update_tasks_nodemask() is called here, it passes &cs->mems_allowed to mpol_rebind_mm():

kernel/cgroup/cpuset.c:cpuset_update_tasks_nodemask() {
    ...
    mpol_rebind_mm(mm, &cs->mems_allowed);
}

Because mems_allowed explicitly excludes private nodes, will this inadvertently strip valid private nodes from a task's static memory policy during completely unrelated cpuset operations?

Quoted 5 lines
kernel/cgroup/cpuset.c +1 -0
+
 	mutex_unlock(&cpuset_mutex);
 
 	/* if cpus or mems changed, we need to propagate to descendants */

if (cpus_updated || mems_updated) {

[Severity: High]
Does this skip propagating private node hot-unplug events to descendant cpusets?

If priv_shrank is true but mems_updated is false, this block is skipped. Could tasks in descendant cpusets retain mempolicies pointing to the newly offlined private node?

Additionally, if the loop does execute, cpuset_hotplug_update_tasks() locally checks:

kernel/cgroup/cpuset.c:cpuset_hotplug_update_tasks() {
    ...
    mems_updated = !nodes_equal(new_mems, cs->effective_mems);
    if (!cpus_updated && !mems_updated)
        goto unlock;
}

Since effective_mems ignores private nodes, it appears descendant updates would be skipped here as well.

[ ... ]

Quoted 4 lines
kernel/cgroup/cpuset.c +0 -0
@@ -4155,11 +4168,13 @@ nodemask_t cpuset_mems_allowed(struct task_struct *tsk)
  * cpuset_nodemask_valid_mems_allowed - check nodemask vs. current mems_allowed
  * @nodemask: the nodemask to be checked
  *

kernel/cgroup/cpuset.c nodemask_t cpuset_mems_allowed(struct task_struct *tsk) · line 4178

Quoted 10 lines
kernel/cgroup/cpuset.c +4 -2
- * Are any of the nodes in the nodemask allowed in current->mems_allowed?
+ * Are any of the nodes in the nodemask usable?  N_MEMORY nodes must be in
+ * current->mems_allowed, while N_MEMORY_PRIVATE nodes are always valid.
  */
 int cpuset_nodemask_valid_mems_allowed(const nodemask_t *nodemask)
 {
-	return nodes_intersects(*nodemask, current->mems_allowed);
+	return nodes_intersects(*nodemask, current->mems_allowed) ||
+	       nodes_intersects(*nodemask, node_states[N_MEMORY_PRIVATE]);
 }

[Severity: High]
Are private nodes unconditionally rejected by sys_migrate_pages and mempolicy creation?

While cpuset_nodemask_valid_mems_allowed() was updated here, userspace operations like kernel_migrate_pages() and mpol_set_nodemask() validate against cpuset_mems_allowed(current), which is defined just above this function.

cpuset_mems_allowed() delegates to guarantee_online_mems(), which masks strictly against N_MEMORY.

For example, in mpol_set_nodemask():

mm/mempolicy.c:mpol_set_nodemask() {
    ...
    nodes_and(nsc->mask1, cpuset_current_mems_allowed, node_states[N_MEMORY]);
    ...
}

Does this mean any mbind() or set_mempolicy() targeting an N_MEMORY_PRIVATE node will still be stripped and return -EINVAL?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
author maintainer
#13 lore raw

PATCH v5 7/36 mm/memory_hotplug: disallow migration-driven private node hotunplug

In reply to Gregory Price

By default, device-backed private node memory may not be safe to migrate without coordination.

Do not attempt to hotunplug private node memory, instead fail the hotunplug attempt, and require the owner to drain the folios safely before allowing hotunplug to proceed.

Unlike transient / longterm pins causing failures, this is not a transient condition - so we can break out of the hotunplug attempt entirely instead of spinning forever on a failed migration attempt.

---
 mm/memory_hotplug.c | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)
mm/memory_hotplug.c +17 -4
index 226ab9cb078ad..2b9c0830821e6 100644
@@ -34,6 +34,7 @@
 #include <linux/memblock.h>
 #include <linux/compaction.h>
 #include <linux/rmap.h>
+#include <linux/node_private.h>
 #include <linux/module.h>
 #include <linux/node.h>
 
@@ -1843,11 +1844,12 @@ static int scan_movable_pages(unsigned long start, unsigned long end,
 	return 0;
 }
 
-static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
+static int do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 {
 	struct folio *folio;
 	unsigned long pfn;
 	LIST_HEAD(source);
+	int err = 0;
 	static DEFINE_RATELIMIT_STATE(migrate_rs, DEFAULT_RATELIMIT_INTERVAL,
 				      DEFAULT_RATELIMIT_BURST);
 
@@ -1884,6 +1886,15 @@ static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 			goto put_folio;
 		}
 
+		/* Private node folios cannot migrate, fail outright */
+		if (folio_is_private_node(folio)) {
+			pr_info_ratelimited("memory offline refused: node %d pfn %lx\n",
+					    folio_nid(folio), pfn);
+			folio_put(folio);
+			err = -EBUSY;
+			break;
+		}
+
 		if (!isolate_folio_to_list(folio, &source)) {
 			if (__ratelimit(&migrate_rs)) {
 				pr_warn("failed to isolate pfn %lx\n",
@@ -1931,6 +1942,7 @@ static void do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 			putback_movable_pages(&source);
 		}
 	}
+	return err;
 }
 
 static int __init cmdline_parse_movable_node(char *p)
@@ -2067,10 +2079,11 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 			ret = scan_movable_pages(pfn, end_pfn, &pfn);
 			if (!ret) {
 				/*
-				 * TODO: fatal migration failures should bail
-				 * out
+				 * A fatal migration failure (e.g. a folio on a
+				 * non-migratable private node) bails out; transient
+				 * failures leave ret zero and are retried below.
 				 */
-				do_migrate_range(pfn, end_pfn);
+				ret = do_migrate_range(pfn, end_pfn);
 			}
 		} while (!ret);
 
Signature · 1 line
2.53.0-Meta
author maintainer
#14 lore raw

PATCH v5 8/36 mm/mempolicy: skip private node folios when queueing for migration

In reply to Gregory Price

Private nodes are already kept out of policy nodemasks (only N_MEMORY nodes are allowed), but an mbind(MPOL_MF_MOVE) walk can still encounter a private-node folio in the range. Skip such folios so mempolicy-driven migration never moves private-node memory.

We already do this for ZONE_DEVICE folios - use the same filter points for ZONE_DEVICE (with an additional point for large pages).

---
 mm/mempolicy.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
mm/mempolicy.c +6 -1
index 5720f7f54d942..8e8763f6e5f5b 100644
@@ -111,6 +111,7 @@
 #include <linux/mmu_notifier.h>
 #include <linux/printk.h>
 #include <linux/leafops.h>
+#include <linux/node_private.h>
 #include <linux/gcd.h>
 
 #include <asm/tlbflush.h>
@@ -668,6 +669,8 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
 	}
 	if (!queue_folio_required(folio, qp))
 		return;
+	if (folio_is_private_node(folio))
+		return;
 	if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
 	    !vma_migratable(walk->vma) ||
 	    !migrate_folio_add(folio, qp->pagelist, qp->flags))
@@ -722,7 +725,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
 			continue;
 		}
 		folio = vm_normal_folio(vma, addr, ptent);
-		if (!folio || folio_is_zone_device(folio))
+		if (!folio || folio_is_private_managed(folio))
 			continue;
 		if (folio_test_large(folio) && max_nr != 1)
 			nr = folio_pte_batch(folio, pte, ptent, max_nr);
@@ -797,6 +800,8 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
 	folio = pfn_folio(pte_pfn(ptep));
 	if (!queue_folio_required(folio, qp))
 		goto unlock;
+	if (folio_is_private_node(folio))
+		goto unlock;
 	if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
 	    !vma_migratable(walk->vma)) {
 		qp->nr_failed++;
Signature · 1 line
2.53.0-Meta
#15 lore raw

Re: [PATCH v5 08/36] mm/mempolicy: skip private node folios when queueing for migration

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [Critical] Blind conversion of PMD to folio in queue_folios_pmd() causes out-of-bounds dereference on raw PFN mappings.
- [High] queue_folios_pte_range() sets walk->action = ACTION_AGAIN when pte_offset_map_lock() returns NULL, leading to an infinite retry loop for migration entries. --

--- Patch [8]: [PATCH v5 08/36] mm/mempolicy: skip private node folios when queueing for migration --- commit 0603242bf30dacf881c78a4de8f6d67872a88c83
Author: Gregory Price <[email protected]>
Subject: mm/mempolicy: skip private node folios when queueing for migration

Private nodes are already kept out of policy nodemasks (only N_MEMORY nodes are allowed), but an mbind(MPOL_MF_MOVE) walk can still encounter a private-node folio in the range. Skip such folios so mempolicy-driven migration never moves private-node memory.

Quoted 4 lines
mm/mempolicy.c +0 -0
index 5720f7f54d942..8e8763f6e5f5b 100644

[ ... ]

mm/mempolicy.c static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk) · line 676

Quoted 9 lines
mm/mempolicy.c +2 -0
@@ -668,6 +669,8 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
 	}
 	if (!queue_folio_required(folio, qp))
 		return;
+	if (folio_is_private_node(folio))
+		return;
 	if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
 	    !vma_migratable(walk->vma) ||
 	    !migrate_folio_add(folio, qp->pagelist, qp->flags))

[Severity: Critical]
This is a pre-existing issue, but can this cause an out-of-bounds dereference on raw PFN mappings?

Earlier in mm/mempolicy.c:queue_folios_pmd(), the code unconditionally calls pmd_folio() and dereferences it:

	folio = pmd_folio(pmdval);
	if (is_huge_zero_folio(folio)) {

If an unprivileged user calls mbind(..., MPOL_MF_STRICT) on a memory region mapped by a driver using raw PFNs via huge PMDs (e.g., vmf_insert_pfn_pmd() in a VM_PFNMAP VMA), pmd_trans_huge_lock() will succeed. However, treating a raw PFN as a struct folio yields a bogus pointer, leading to a kernel panic when accessed. Does this need a check like pmd_special() to filter out raw PFN mappings?

mm/mempolicy.c static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, · line 731

Quoted 9 lines
mm/mempolicy.c +1 -1
@@ -722,7 +725,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
 			continue;
 		}
 		folio = vm_normal_folio(vma, addr, ptent);
-		if (!folio || folio_is_zone_device(folio))
+		if (!folio || folio_is_private_managed(folio))
 			continue;
 		if (folio_test_large(folio) && max_nr != 1)
 			nr = folio_pte_batch(folio, pte, ptent, max_nr);

[Severity: High]
This is a pre-existing issue, but can this lead to an infinite retry loop?

Just above this section in mm/mempolicy.c:queue_folios_pte_range(), there is:

	mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl);
	if (!pte) {
		walk->action = ACTION_AGAIN;
		return 0;
	}

Since pte_offset_map_lock() returns NULL non-transiently for migration entries, setting walk->action = ACTION_AGAIN and returning 0 on this failure instructs walk_pte_range() to retry the exact same PMD indefinitely with no limit. Because this early return skips the cond_resched() at the bottom of the function, it creates a tight CPU livelock. Should this instead return 0 without setting ACTION_AGAIN to gracefully skip the entry?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
author maintainer
#16 lore raw

PATCH v5 9/36 mm/migrate: disallow userland driven migration for private nodes

In reply to Gregory Price

Use the same filter locations as zone_device folios to disallow userland driven migration requests for private node memory.

---
 mm/migrate.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/migrate.c +2 -2
index a8a86141dbb54..d20674c07b947 100644
@@ -2268,7 +2268,7 @@ static int __add_folio_for_migration(struct folio *folio, int node,
 	if (is_zero_folio(folio) || is_huge_zero_folio(folio))
 		return -EFAULT;
 
-	if (folio_is_zone_device(folio))
+	if (folio_is_private_managed(folio))
 		return -ENOENT;
 
 	if (folio_nid(folio) == node)
@@ -2477,7 +2477,7 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
 		if (folio) {
 			if (is_zero_folio(folio) || is_huge_zero_folio(folio))
 				err = -EFAULT;
-			else if (folio_is_zone_device(folio))
+			else if (folio_is_private_managed(folio))
 				err = -ENOENT;
 			else
 				err = folio_nid(folio);
Signature · 1 line
2.53.0-Meta
#17 lore raw

Re: [PATCH v5 09/36] mm/migrate: disallow userland driven migration for private nodes

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [High] Incomplete replacement of zone_device checks allows uncoordinated background migrations of private node memory.
- [High] node_private_unregister() permanently returns -EBUSY for private nodes because offline_pages() never clears N_MEMORY_PRIVATE.

Pre-existing issues:
- [High] User-space addresses are not stripped of memory tags (e.g., ARM64 MTE) before VMA lookup in do_pages_stat_array(), causing valid tagged addresses to fail with -EFAULT. --

--- Patch [9]: [PATCH v5 09/36] mm/migrate: disallow userland driven migration for private nodes --- commit 7ed55bccdffe7ca2bf917693b091b54ebe294b02
Author: Gregory Price <[email protected]>

mm/migrate: disallow userland driven migration for private nodes

Use the same filter locations as zone_device folios to disallow userland driven migration requests for private node memory.

mm/migrate.c static int __add_folio_for_migration(struct folio *folio, int node, · line 2271

Quoted 10 lines
mm/migrate.c +1 -1
index a8a86141dbb54..d20674c07b947 100644
@@ -2268,7 +2268,7 @@ static int __add_folio_for_migration(struct folio *folio, int node,
 	if (is_zero_folio(folio) || is_huge_zero_folio(folio))
 		return -EFAULT;
 
-	if (folio_is_zone_device(folio))
+	if (folio_is_private_managed(folio))

[Severity: High]
By updating these filters, do we also need to update similar checks in uncoordinated background migration paths?

If background tasks encounter a private node folio, they will evaluate folio_is_zone_device() or is_zone_device_page() as false. Could this allow NUMA balancing, KSM, or khugepaged to silently migrate private node memory to general-purpose nodes?

For example, in NUMA balancing:

mm/memory.c:do_numa_page() {
    ...
    if (!folio || folio_is_zone_device(folio))
        goto out_map;
    ...
}

In THP collapse:

mm/khugepaged.c:khugepaged_scan_pmd() {
    ...
    if (WARN_ON_ONCE(page && is_zone_device_page(page)))
        page = NULL;
    ...
}

And in KSM:

mm/ksm.c:get_mergeable_page() {
    ...
    if (!folio_is_zone_device(folio) &&
        folio_test_anon(folio)) {
    ...
}

mm/migrate.c static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages, · line 2480

Quoted 9 lines
mm/migrate.c +1 -1
 		return -ENOENT;
 
 	if (folio_nid(folio) == node)
@@ -2477,7 +2477,7 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
 		if (folio) {
 			if (is_zero_folio(folio) || is_huge_zero_folio(folio))
 				err = -EFAULT;
-			else if (folio_is_zone_device(folio))
+			else if (folio_is_private_managed(folio))

[Severity: High]
This isn't a bug introduced by this patch, but does do_pages_stat_array() fail to handle tagged pointers?

When a user-space program passes an array of tagged pointers to move_pages with nodes == NULL, the pointers are cast directly to unsigned long without being sanitized by untagged_addr_remote():

mm/migrate.c:do_pages_stat_array() {
    ...
    unsigned long addr = (unsigned long)(*pages);
    struct vm_area_struct *vma;
    // ...
    vma = vma_lookup(mm, addr);
    ...
}

Could this cause vma_lookup() to treat the tag bits as part of the virtual address, causing valid tagged addresses (like those used with ARM64 MTE) to fail with -EFAULT?

mm/migrate.c static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages, · line 2483

Quoted 3 lines
mm/migrate.c +0 -0
 				err = -ENOENT;
 			else
 				err = folio_nid(folio);

[Severity: High]
On a related note regarding private node teardown, does offline_pages() need to clear N_MEMORY_PRIVATE?

If a driver attempts to teardown a device and offline its memory, offline_pages() clears N_MEMORY but omits N_MEMORY_PRIVATE:

mm/memory_hotplug.c:offline_pages() {
    ...
    if (node_arg.nid >= 0)
        node_clear_state(node, N_MEMORY);
    ...
}

Will this cause node_private_unregister() to evaluate node_is_private(nid) as true and permanently return -EBUSY?

drivers/base/node.c:node_private_unregister() {
    ...
    if (node_is_private(nid)) {
        mem_hotplug_done();
        mutex_unlock(&node_private_lock);
        return -EBUSY;
    }
    ...
}

Could this permanent -EBUSY prevent drivers using the N_MEMORY_PRIVATE feature from cleanly unloading?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
author maintainer
#18 lore raw

PATCH v5 10/36 mm/madvise: disallow madvise operations on private node folios

In reply to Gregory Price

Use the same filter locations as zone_device, plus additional filters for huge pages to avoid madvise operations on private node memory.

---
 mm/huge_memory.c | 5 +++++
 mm/madvise.c     | 8 ++++++--
 2 files changed, 11 insertions(+), 2 deletions(-)
mm/huge_memory.c +5 -0
index 5bd8d4f59a7b8..1df91b4e5c2bc 100644
@@ -37,6 +37,7 @@
 #include <linux/page_owner.h>
 #include <linux/sched/sysctl.h>
 #include <linux/memory-tiers.h>
+#include <linux/node_private.h>
 #include <linux/compat.h>
 #include <linux/pgalloc.h>
 #include <linux/pgalloc_tag.h>
@@ -2338,6 +2339,10 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 	}
 
 	folio = pmd_folio(orig_pmd);
+
+	if (folio_is_private_node(folio))
+		goto out;
+
 	/*
 	 * If other processes are mapping this folio, we couldn't discard
 	 * the folio unless they all do MADV_FREE so let's skip the folio.
mm/madvise.c +6 -2
index 07a21ca31bad4..29f35a23919a0 100644
@@ -32,6 +32,7 @@
 #include <linux/leafops.h>
 #include <linux/shmem_fs.h>
 #include <linux/mmu_notifier.h>
+#include <linux/node_private.h>
 
 #include <asm/tlb.h>
 
@@ -395,6 +396,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 
 		folio = pmd_folio(orig_pmd);
 
+		if (folio_is_private_node(folio))
+			goto huge_unlock;
+
 		/* Do not interfere with other mappings of this folio */
 		if (folio_maybe_mapped_shared(folio))
 			goto huge_unlock;
@@ -474,7 +478,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 			continue;
 
 		folio = vm_normal_folio(vma, addr, ptent);
-		if (!folio || folio_is_zone_device(folio))
+		if (!folio || folio_is_private_managed(folio))
 			continue;
 
 		/*
@@ -703,7 +707,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
 		}
 
 		folio = vm_normal_folio(vma, addr, ptent);
-		if (!folio || folio_is_zone_device(folio))
+		if (!folio || folio_is_private_managed(folio))
 			continue;
 
 		/*
Signature · 1 line
2.53.0-Meta
#19 lore raw

Re: [PATCH v5 10/36] mm/madvise: disallow madvise operations on private node folios

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] madvise_cold_or_pageout_pte_range() calls folio_get() before blocking on folio_lock() when attempting to split a large PMD-mapped folio, violating the required lock-before-refcount ordering for large folio splits. --

--- Patch [10]: [PATCH v5 10/36] mm/madvise: disallow madvise operations on private node folios --- commit 76f9465747c7c18f9f36e91d7a628706fb9905f9
Author: Gregory Price <[email protected]>

mm/madvise: disallow madvise operations on private node folios

Use the same filter locations as zone_device, plus additional filters for huge pages to avoid madvise operations on private node memory.

[ ... ]

Quoted 4 lines
mm/madvise.c +0 -0
index 07a21ca31bad4..29f35a23919a0 100644

[ ... ]

mm/madvise.c static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, · line 404

Quoted 10 lines
mm/madvise.c +3 -0
@@ -395,6 +396,9 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 
 		folio = pmd_folio(orig_pmd);
 
+		if (folio_is_private_node(folio))
+			goto huge_unlock;
+
 		/* Do not interfere with other mappings of this folio */
 		if (folio_maybe_mapped_shared(folio))
 			goto huge_unlock;

[Severity: High]
This is a pre-existing issue, but does the sequence just below this block in madvise_cold_or_pageout_pte_range() cause split_folio() to fail under contention?

A few lines down, the code attempts to split a PMD-mapped large folio:

    folio_get(folio);
    spin_unlock(ptl);
    folio_lock(folio);
    err = split_folio(folio);

When multiple tasks concurrently attempt to split the same folio, taking the reference before blocking on the lock inflates the refcount. This causes split_folio() to spuriously return -EAGAIN because the expected refcount does not match.

Could we trylock the folio before raising the reference count here, similar to how it is handled in madvise_free_huge_pmd() in huge_memory.c?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
author maintainer
#20 lore raw

PATCH v5 11/36 mm/compaction: disallow compaction on private nodes

In reply to Gregory Price

Skip compaction on private nodes. Compaction requires the ability to migrate pages, which is not yet supported on private nodes.

---
 mm/compaction.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
mm/compaction.c +12 -0
index 9f81055a358ed..8c1351cce7bcc 100644
@@ -2462,6 +2462,9 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
 		    !__cpuset_zone_allowed(zone, gfp_mask))
 			continue;
 
+		if (node_is_private(zone_to_nid(zone)))
+			continue;
+
 		/*
 		 * Do not consider all the reclaimable memory because we do not
 		 * want to trash just for a single high order allocation which
@@ -2853,6 +2856,9 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
 			!__cpuset_zone_allowed(zone, gfp_mask))
 				continue;
 
+		if (node_is_private(zone_to_nid(zone)))
+			continue;
+
 		if (prio > MIN_COMPACT_PRIORITY
 					&& compaction_deferred(zone, order)) {
 			rc = max_t(enum compact_result, COMPACT_DEFERRED, rc);
@@ -2922,6 +2928,9 @@ static int compact_node(pg_data_t *pgdat, bool proactive)
 		.proactive_compaction = proactive,
 	};
 
+	if (node_is_private(pgdat->node_id))
+		return 0;
+
 	for (zoneid = 0; zoneid < MAX_NR_ZONES; zoneid++) {
 		zone = &pgdat->node_zones[zoneid];
 		if (!populated_zone(zone))
@@ -3017,6 +3026,9 @@ static ssize_t compact_store(struct device *dev,
 {
 	int nid = dev->id;
 
+	if (node_is_private(nid))
+		return -EINVAL;
+
 	if (nid >= 0 && nid < nr_node_ids && node_online(nid)) {
 		/* Flush pending updates to the LRU lists */
 		lru_add_drain_all();
Signature · 1 line
2.53.0-Meta
#21 lore raw

Re: [PATCH v5 11/36] mm/compaction: disallow compaction on private nodes

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] The kcompactd background compaction daemon is not prevented from compacting private nodes. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The kcompactd background compaction daemon is not prevented from compacting private nodes. --

--- Patch [11]: [PATCH v5 11/36] mm/compaction: disallow compaction on private nodes --- commit 8f0db5de5c5b2025b3451b6bb90c3cd06411f91d
Author: Gregory Price <[email protected]>

mm/compaction: disallow compaction on private nodes

This commit skips compaction on private nodes because they do not yet support page migration. It adds checks to direct compaction and sysfs paths to abort compaction early for private nodes.

Quoted 4 lines
mm/compaction.c +0 -0
index 9f81055a358ed..8c1351cce7bcc 100644

[ ... ]

mm/compaction.c static ssize_t compact_store(struct device *dev, · line 3034

Quoted 10 lines
mm/compaction.c +3 -0
@@ -3017,6 +3026,9 @@ static ssize_t compact_store(struct device *dev,
 {
 	int nid = dev->id;
 
+	if (node_is_private(nid))
+		return -EINVAL;
+
 	if (nid >= 0 && nid < nr_node_ids && node_online(nid)) {
 		/* Flush pending updates to the LRU lists */
 		lru_add_drain_all();

[Severity: High]
The commit message states we should disallow compaction on private nodes because migration is not yet supported. Does this patch miss the background compaction daemon paths?

Under memory pressure, kswapd calls wakeup_kcompactd(), which currently bypasses private node checks:

mm/compaction.c:wakeup_kcompactd() {
    ...
    if (!kcompactd_node_suitable(pgdat))
        return;

    wake_up_interruptible(&pgdat->kcompactd_wait);
}

This wakes up the daemon which will then unconditionally execute compact_zone() for the node without verifying if the node is private:

mm/compaction.c:kcompactd_do_work() {
    ...
    for (zoneid = 0; zoneid <= cc.highest_zoneidx; zoneid++) {
        ...
        status = compact_zone(&cc, NULL);
    ...
}

Could this result in the background daemon uselessly waking up and attempting unsupported page migration on private nodes?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
author maintainer
#22 lore raw

PATCH v5 12/36 mm/page_alloc: clear private node watermarks and system reserves

In reply to Gregory Price

Set private node watermarks to 0 (reclaim isn't supported yet, so they are pointless), and do not include private nodes in system reserves.

The oom killer can't actually resolve general system pressure by releasing private node capacity, so including them in system reserve calculations dilutes DRAM watermarks and causes poor OOM kill behavior.

---
 mm/page_alloc.c | 24 ++++++++++++++++++++++--
 1 file changed, 22 insertions(+), 2 deletions(-)
mm/page_alloc.c +22 -2
index 78755a55b1540..2b08bea2379a9 100644
@@ -6555,6 +6555,10 @@ static void calculate_totalreserve_pages(void)
 
 		pgdat->totalreserve_pages = 0;
 
+		/* private nodes have zero watermarks */
+		if (node_is_private(pgdat->node_id))
+			continue;
+
 		for (i = 0; i < MAX_NR_ZONES; i++) {
 			struct zone *zone = pgdat->node_zones + i;
 			long max = 0;
@@ -6647,9 +6651,15 @@ static void __setup_per_zone_wmarks(void)
 	struct zone *zone;
 	unsigned long flags;
 
-	/* Calculate total number of !ZONE_HIGHMEM and !ZONE_MOVABLE pages */
+	/*
+	 * Calculate total number of !ZONE_HIGHMEM and !ZONE_MOVABLE pages
+	 *
+	 * Private nodes are excluded because they are not in the regular
+	 * allocation fallback path - including them dilutes DRAM watermarks.
+	 */
 	for_each_zone(zone) {
-		if (!is_highmem(zone) && zone_idx(zone) != ZONE_MOVABLE)
+		if (!is_highmem(zone) && zone_idx(zone) != ZONE_MOVABLE &&
+		    !node_is_private(zone_to_nid(zone)))
 			lowmem_pages += zone_managed_pages(zone);
 	}
 
@@ -6657,6 +6667,16 @@ static void __setup_per_zone_wmarks(void)
 		u64 tmp;
 
 		spin_lock_irqsave(&zone->lock, flags);
+		if (node_is_private(zone_to_nid(zone))) {
+			zone->_watermark[WMARK_MIN] = 0;
+			zone->_watermark[WMARK_LOW] = 0;
+			zone->_watermark[WMARK_HIGH] = 0;
+			zone->_watermark[WMARK_PROMO] = 0;
+			zone->watermark_boost = 0;
+			spin_unlock_irqrestore(&zone->lock, flags);
+			continue;
+		}
+
 		tmp = (u64)pages_min * zone_managed_pages(zone);
 		tmp = div64_ul(tmp, lowmem_pages);
 		if (is_highmem(zone) || zone_idx(zone) == ZONE_MOVABLE) {
Signature · 1 line
2.53.0-Meta
author maintainer
#23 lore raw

PATCH v5 13/36 mm/mempolicy: disallow NUMA Balancing prot_none on private nodes

In reply to Gregory Price

Skip private node memory the same way zone device memory is skipped, since private nodes do not presently support migration.

---
 mm/mempolicy.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/mempolicy.c +1 -1
index 8e8763f6e5f5b..2b76c57a460c9 100644
@@ -845,7 +845,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 {
 	int nid;
 
-	if (!folio || folio_is_zone_device(folio) || folio_test_ksm(folio))
+	if (!folio || folio_is_private_managed(folio) || folio_test_ksm(folio))
 		return false;
 
 	/* Also skip shared copy-on-write folios */
Signature · 1 line
2.53.0-Meta
author maintainer
#24 lore raw

PATCH v5 14/36 mm/damon: skip private node memory in DAMON migration and pageout

In reply to Gregory Price

DAMON operates on physical address ranges, which can cover private node memory. Skip private-node folios in both DAMON's migration and reclaim paths.

---
 mm/damon/paddr.c | 9 +++++++++
 1 file changed, 9 insertions(+)
mm/damon/paddr.c +9 -0
index e4f98d67461f5..c741a94319750 100644
@@ -12,6 +12,7 @@
 #include <linux/swap.h>
 #include <linux/memory-tiers.h>
 #include <linux/mm_inline.h>
+#include <linux/node_private.h>
 
 #include "../internal.h"
 #include "ops-common.h"
@@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
 			continue;
 		}
 
+		/* private node memory is not reclaimable by default */
+		if (folio_is_private_node(folio))
+			goto put_folio;
+
 		if (damos_pa_filter_out(s, folio))
 			goto put_folio;
 		else
@@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
 		else
 			*sz_filter_passed += folio_size(folio) / addr_unit;
 
+		/* private nodes do not support migration by default */
+		if (folio_is_private_node(folio))
+			goto put_folio;
+
 		if (!folio_isolate_lru(folio))
 			goto put_folio;
 		list_add(&folio->lru, &folio_list);
Signature · 1 line
2.53.0-Meta
#25 lore raw

Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Inconsistent placement of the `folio_is_private_node()` check causes incorrect…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Inconsistent placement of the folio_is_private_node() check causes incorrect sz_filter_passed accounting and unnecessary filter side-effects in damon_pa_migrate() compared to damon_pa_pageout(). --

--- Patch [14]: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout --- commit eb373ee8a3d83be1e35d66f74d2720ee3b1c26fa
Author: Gregory Price <[email protected]>
mm/damon: skip private node memory in DAMON migration and pageout

This commit updates DAMON's physical address operations to skip private-node folios during both migration and pageout reclaim paths.

Quoted 15 lines
mm/damon/paddr.c +4 -0
index e4f98d67461f5..c741a94319750 100644
@@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
 			continue;
 		}
 
+		/* private node memory is not reclaimable by default */
+		if (folio_is_private_node(folio))
+			goto put_folio;
+
 		if (damos_pa_filter_out(s, folio))
 			goto put_folio;
 		else

[ ... ]

mm/damon/paddr.c static unsigned long damon_pa_migrate(struct damon_region *r, · line 355

Quoted 8 lines
mm/damon/paddr.c +4 -0
@@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
 		else
 			*sz_filter_passed += folio_size(folio) / addr_unit;
 
+		/* private nodes do not support migration by default */
+		if (folio_is_private_node(folio))
+			goto put_folio;
+

[Severity: Medium]
Does placing this check after the filter evaluation cause incorrect DAMON statistics?

Because the check happens here rather than before the filter evaluation, it appears unmigratable private node folios will be erroneously counted in sz_filter_passed. This might also trigger unnecessary filter logic evaluations before we ultimately skip the folio.

Could we move this up before damos_pa_filter_out() in damon_pa_migrate() so it matches the logic flow used earlier in damon_pa_pageout()?

Quoted 3 lines
mm/damon/paddr.c +0 -0
 		if (!folio_isolate_lru(folio))
 			goto put_folio;
 		list_add(&folio->lru, &folio_list);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
maintainer
#26 lore raw

Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout

In reply to Gregory Price
Quoted 22 lines · On Mon, 20 Jul 2026 15:34:08 -0400 Gregory Price <[email protected]> wrote:

DAMON operates on physical address ranges, which can cover private node memory. Skip private-node folios in both DAMON's migration and reclaim paths.

---
 mm/damon/paddr.c | 9 +++++++++
 1 file changed, 9 insertions(+)
mm/damon/paddr.c +1 -0
index e4f98d67461f5..c741a94319750 100644
@@ -12,6 +12,7 @@
 #include <linux/swap.h>
 #include <linux/memory-tiers.h>
 #include <linux/mm_inline.h>
+#include <linux/node_private.h>
 
 #include "../internal.h"
 #include "ops-common.h"

mm/damon/paddr.c static unsigned long damon_pa_pageout(struct damon_region *r, · line 257

Quoted 7 lines
mm/damon/paddr.c +4 -0
@@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
 			continue;
 		}
 
+		/* private node memory is not reclaimable by default */
+		if (folio_is_private_node(folio))
+			goto put_folio;
+

"by default". Does that mean it could be reclaimable in some situations? If so, could we check if it is reclaimable?

Also, what happens if we just try paging out the private node memory? Will it simply fail? Or, make some problems?

mm/damon/paddr.c static unsigned long damon_pa_migrate(struct damon_region *r, · line 355

Quoted 11 lines
mm/damon/paddr.c +4 -0
 		if (damos_pa_filter_out(s, folio))
 			goto put_folio;
 		else
@@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
 		else
 			*sz_filter_passed += folio_size(folio) / addr_unit;
 
+		/* private nodes do not support migration by default */
+		if (folio_is_private_node(folio))
+			goto put_folio;
+

Same questions.

Quoted 5 lines
mm/damon/paddr.c +0 -0
 		if (!folio_isolate_lru(folio))
 			goto put_folio;
 		list_add(&folio->lru, &folio_list);
Signature · 1 line
2.53.0-Meta

Thanks,
SJ

author maintainer
#27 lore raw

Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout

Quoted 27 lines · SJ Park
Quoted 25 lines · On Mon, 20 Jul 2026 15:34:08 -0400 Gregory Price <[email protected]> wrote:

DAMON operates on physical address ranges, which can cover private node memory. Skip private-node folios in both DAMON's migration and reclaim paths.

---
 mm/damon/paddr.c | 9 +++++++++
 1 file changed, 9 insertions(+)
mm/damon/paddr.c +1 -0
index e4f98d67461f5..c741a94319750 100644
@@ -12,6 +12,7 @@
 #include <linux/swap.h>
 #include <linux/memory-tiers.h>
 #include <linux/mm_inline.h>
+#include <linux/node_private.h>
 
 #include "../internal.h"
 #include "ops-common.h"
@@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
 			continue;
 		}
 
Quoted 8 lines

mm/damon/paddr.c static unsigned long damon_pa_pageout(struct damon_region *r, · line 257

Quoted 4 lines
mm/damon/paddr.c +4 -0
+		/* private node memory is not reclaimable by default */
+		if (folio_is_private_node(folio))
+			goto put_folio;
+

"by default". Does that mean it could be reclaimable in some situations? If so, could we check if it is reclaimable?

See the damon changes in:

https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#mfb99303c85dd9d4c3f58832dc28fc55583a7217a

Quoted 3 lines

Also, what happens if we just try paging out the private node memory? Will it simply fail? Or, make some problems?

user controls for MADV_ commands that affect private node folios would skip those folios - exactly same as ZONE_DEVICE folios.

Which I'm not sure damon actually handles correctly presently (zone device). I remember looking around damon and thinking to myself "what if this is a zone device page?" and not being able to find a satisfying answer.

Quoted 7 lines
Quoted 7 lines
mm/damon/paddr.c +0 -0
 		if (damos_pa_filter_out(s, folio))
 			goto put_folio;
 		else
@@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
 		else
 			*sz_filter_passed += folio_size(folio) / addr_unit;
 
Quoted 7 lines

mm/damon/paddr.c static unsigned long damon_pa_migrate(struct damon_region *r, · line 355

Quoted 4 lines
mm/damon/paddr.c +4 -0
+		/* private nodes do not support migration by default */
+		if (folio_is_private_node(folio))
+			goto put_folio;
+

Same questions.

Technically there is nothing in migration core to prevent migration operations, it's done on a service basis - hotunplug, reclaim/demotion, user numa (mbind, migrate/move_pages) etc.

Operations on private nodes/private node folioes are refused if the capability bit is not set.

~Gregory

maintainer
#28 lore raw

Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout

Quoted 37 lines · On Wed, 22 Jul 2026 08:16:56 -0400 Gregory Price <[email protected]> wrote:
Quoted 27 lines · SJ Park
Quoted 25 lines · On Mon, 20 Jul 2026 15:34:08 -0400 Gregory Price <[email protected]> wrote:

DAMON operates on physical address ranges, which can cover private node memory. Skip private-node folios in both DAMON's migration and reclaim paths.

---
 mm/damon/paddr.c | 9 +++++++++
 1 file changed, 9 insertions(+)
mm/damon/paddr.c +1 -0
index e4f98d67461f5..c741a94319750 100644
@@ -12,6 +12,7 @@
 #include <linux/swap.h>
 #include <linux/memory-tiers.h>
 #include <linux/mm_inline.h>
+#include <linux/node_private.h>
 
 #include "../internal.h"
 #include "ops-common.h"
@@ -250,6 +251,10 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
 			continue;
 		}
 
Quoted 8 lines

mm/damon/paddr.c static unsigned long damon_pa_pageout(struct damon_region *r, · line 257

Quoted 4 lines
mm/damon/paddr.c +4 -0
+		/* private node memory is not reclaimable by default */
+		if (folio_is_private_node(folio))
+			goto put_folio;
+

"by default". Does that mean it could be reclaimable in some situations? If so, could we check if it is reclaimable?

Quoted 3 lines

See the damon changes in:

https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#mfb99303c85dd9d4c3f58832dc28fc55583a7217a

Summarizing what you want to say, quoting something from the patch, or at least calling it "26th patch of this series" would have made reviewing much easier. Cc-ing damon@ for only patches that toucing DAMON source files and the cover letter of the series could also be helpful. Please consider doing some of these for future replies.

So, I understand later patches will make it optionally reclaimable and update this restriction by the 26th patch? That sounds fair. But, could we drop "by default" from the above comment for reducing the confusion?

Quoted 28 lines
Quoted 4 lines

Also, what happens if we just try paging out the private node memory? Will it simply fail? Or, make some problems?

user controls for MADV_ commands that affect private node folios would skip those folios - exactly same as ZONE_DEVICE folios.

Which I'm not sure damon actually handles correctly presently (zone device). I remember looking around damon and thinking to myself "what if this is a zone device page?" and not being able to find a satisfying answer.

Quoted 7 lines
Quoted 7 lines
mm/damon/paddr.c +0 -0
 		if (damos_pa_filter_out(s, folio))
 			goto put_folio;
 		else
@@ -344,6 +349,10 @@ static unsigned long damon_pa_migrate(struct damon_region *r,
 		else
 			*sz_filter_passed += folio_size(folio) / addr_unit;
 
Quoted 7 lines

mm/damon/paddr.c static unsigned long damon_pa_migrate(struct damon_region *r, · line 355

Quoted 4 lines
mm/damon/paddr.c +4 -0
+		/* private nodes do not support migration by default */
+		if (folio_is_private_node(folio))
+			goto put_folio;
+

Same questions.

Quoted 6 lines

Technically there is nothing in migration core to prevent migration operations, it's done on a service basis - hotunplug, reclaim/demotion, user numa (mbind, migrate/move_pages) etc.

Operations on private nodes/private node folioes are refused if the capability bit is not set.

I haven't had a chance to read the entire series, sorry about that. So, do you mean the above blocking is not really needed, or that will conditionally be allowed by another later patch?

Thanks,
SJ

[...]

author maintainer
#29 lore raw

Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout

Quoted 8 lines · SJ Park
Quoted 6 lines · On Wed, 22 Jul 2026 08:16:56 -0400 Gregory Price <[email protected]> wrote:
Quoted 3 lines

"by default". Does that mean it could be reclaimable in some situations? If so, could we check if it is reclaimable?

See the damon changes in:

Quoted 8 lines
Quoted 1 line

https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#mfb99303c85dd9d4c3f58832dc28fc55583a7217a

Summarizing what you want to say, quoting something from the patch, or at least calling it "26th patch of this series" would have made reviewing much easier. Cc-ing damon@ for only patches that toucing DAMON source files and the cover letter of the series could also be helpful. Please consider doing some of these for future replies.

I have been very much discouraged from this kind of trimming because it removes the context of the series from the individual patches - making it even more confusing.

But I understand, I will try to remember to do this for you.

Quoted 3 lines

So, I understand later patches will make it optionally reclaimable and update this restriction by the 26th patch? That sounds fair. But, could we drop "by default" from the above comment for reducing the confusion?

Yes, but you bring up a good point - I think this two-step process is actually poor form and that I need to redo it.

Maybe this can be redone as node states (features?) such as:

N_MEMORY_RECLAIMABLE
N_MEMORY_DEMOTABLE
N_MEMORY_TIERABLE

so these damon changes will forego this confusing intermediate step of:

if (folio_is_private_node(folio))

and go straight to

if (node_is_reclaimable(folio_nid(folio)))

In the first patch - without ever having to revisit damon again.

Quoted 5 lines
Quoted 5 lines

Technically there is nothing in migration core to prevent migration operations, it's done on a service basis - hotunplug, reclaim/demotion, user numa (mbind, migrate/move_pages) etc.

Quoted 7 lines
Quoted 2 lines

Operations on private nodes/private node folioes are refused if the capability bit is not set.

I haven't had a chance to read the entire series, sorry about that. So, do you mean the above blocking is not really needed, or that will conditionally be allowed by another later patch?

I will reduce this all to a single patch in v6, I see where things can be improved now based on a few pieces of feedback.

Thanks SJ!
~Gregory

maintainer
#30 lore raw

Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout

Quoted 18 lines · On Wed, 22 Jul 2026 23:25:15 -0400 Gregory Price <[email protected]> wrote:
Quoted 8 lines · SJ Park
Quoted 6 lines · On Wed, 22 Jul 2026 08:16:56 -0400 Gregory Price <[email protected]> wrote:
Quoted 3 lines

"by default". Does that mean it could be reclaimable in some situations? If so, could we check if it is reclaimable?

See the damon changes in:

Quoted 8 lines
Quoted 1 line

https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#mfb99303c85dd9d4c3f58832dc28fc55583a7217a

Summarizing what you want to say, quoting something from the patch, or at least calling it "26th patch of this series" would have made reviewing much easier. Cc-ing damon@ for only patches that toucing DAMON source files and the cover letter of the series could also be helpful. Please consider doing some of these for future replies.

Quoted 3 lines

I have been very much discouraged from this kind of trimming because it removes the context of the series from the individual patches - making it even more confusing.

That's fair and making sense.

Quoted 2 lines

But I understand, I will try to remember to do this for you.

Thank you, Gregory. Nonetheless, I wouldn't insist selective Cc-ing. I agree your point, and I expect doing that would be tedious if you don't have a script or tool (I use a tool). It is just one of my suggestion to help reviewing in this context, insted of only lore link. Just one among the other three suggestions (summarizing your thought, quoting, or simply calling it "26th patch of this series") should also suffice.

Quoted 16 lines
Quoted 4 lines

So, I understand later patches will make it optionally reclaimable and update this restriction by the 26th patch? That sounds fair. But, could we drop "by default" from the above comment for reducing the confusion?

Yes, but you bring up a good point - I think this two-step process is actually poor form and that I need to redo it.

Maybe this can be redone as node states (features?) such as:

N_MEMORY_RECLAIMABLE
N_MEMORY_DEMOTABLE
N_MEMORY_TIERABLE

so these damon changes will forego this confusing intermediate step of:

Quoted 7 lines

if (folio_is_private_node(folio))

and go straight to

if (node_is_reclaimable(folio_nid(folio)))

In the first patch - without ever having to revisit damon again.

Indeed that sounds better than my suggestion.

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

Technically there is nothing in migration core to prevent migration operations, it's done on a service basis - hotunplug, reclaim/demotion, user numa (mbind, migrate/move_pages) etc.

Operations on private nodes/private node folioes are refused if the capability bit is not set.

Quoted 7 lines
Quoted 4 lines

I haven't had a chance to read the entire series, sorry about that. So, do you mean the above blocking is not really needed, or that will conditionally be allowed by another later patch?

I will reduce this all to a single patch in v6, I see where things can be improved now based on a few pieces of feedback.

Sounds good, thank you Gregory. Looking forward to the next version!

Thanks,
SJ

[...]

#31 lore raw

Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout

In reply to Gregory Price
Quoted 15 lines · Gregory Price
Quoted 5 lines · SJ Park
Quoted 3 lines · On Wed, 22 Jul 2026 08:16:56 -0400 Gregory Price <[email protected]> wrote:

See the damon changes in:

Quoted 8 lines
Quoted 1 line

https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#mfb99303c85dd9d4c3f58832dc28fc55583a7217a

Summarizing what you want to say, quoting something from the patch, or at least calling it "26th patch of this series" would have made reviewing much easier. Cc-ing damon@ for only patches that toucing DAMON source files and the cover letter of the series could also be helpful. Please consider doing some of these for future replies.

Quoted 3 lines

I have been very much discouraged from this kind of trimming because it removes the context of the series from the individual patches - making it even more confusing.

Yes, 99% of upstream maintainers will scream at you if you CC them only on individual patches :)

Signature · 3 lines
Cheers,

David
maintainer
#32 lore raw

Re: [PATCH v5 14/36] mm/damon: skip private node memory in DAMON migration and pageout

Quoted 16 lines · On Thu, 10 Sep 2026 10:58:05 +0200 "David Hildenbrand (Arm)"…
Quoted 15 lines · Gregory Price
Quoted 13 lines · SJ Park
Quoted 4 lines · On Wed, 22 Jul 2026 08:16:56 -0400 Gregory Price <[email protected]> wrote:

See the damon changes in:

https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#mfb99303c85dd9d4c3f58832dc28fc55583a7217a

Summarizing what you want to say, quoting something from the patch, or at least calling it "26th patch of this series" would have made reviewing much easier. Cc-ing damon@ for only patches that toucing DAMON source files and the cover letter of the series could also be helpful. Please consider doing some of these for future replies.

Quoted 6 lines
Quoted 3 lines

I have been very much discouraged from this kind of trimming because it removes the context of the series from the individual patches - making it even more confusing.

Yes, 99% of upstream maintainers will scream at you if you CC them only on individual patches :)

I agree that not everyone would like selective Cc-ing, like I also replied. But the number might be changed after we got modern mailing archives and tools like lore (public-inbox)?

At least for me, selective Cc-ing works better as long as the cover letter is Cc-ing all recipients of all patches. I can understand the high level picture from the cover letter and directly jump to the specific patch that require my attention. Of course the sender might failed at figuring out to which mails I really need to pay attention. If the cover letter doesn't tell me some details or I feel the sender might not Cc-ed me on all the patches that require my attention, I can fetch the entire thread from lore (public-inbox).

Meanwhile, my private inbox have limited space and suboptimal UI. Hence I use my private inbox as just an alarm and usually work on only public-inbox with a CLI tool. Hence keeping noise in private inbox lower is more important to my specific workflow. Again, my workflow may be very different from others.

Thanks,
SJ

[...]

author maintainer
#33 lore raw

PATCH v5 15/36 mm/ksm: skip KSM for managed-memory folios

In reply to Gregory Price

Since private memory is intended for explicit placement, allowing KSM to operate on it by default would completely defeat the purpose.

Skip KSM merge and scan operations for private node folios using the same filter location as zone_device folios.

---
 mm/ksm.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
mm/ksm.c +5 -3
index 47006f494fcb1..dcc5c46a29eac 100644
@@ -827,7 +827,7 @@ static struct page *get_mergeable_page(struct ksm_rmap_item *rmap_item)
 
 	folio = folio_walk_start(&fw, vma, addr, 0);
 	if (folio) {
-		if (!folio_is_zone_device(folio) &&
+		if (!folio_is_private_managed(folio) &&
 		    folio_test_anon(folio)) {
 			folio_get(folio);
 			page = fw.page;
@@ -2558,7 +2558,8 @@ static int ksm_next_page_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned lon
 				goto not_found_unlock;
 			folio = page_folio(page);
 
-			if (folio_is_zone_device(folio) || !folio_test_anon(folio))
+			if (unlikely(folio_is_private_managed(folio)) ||
+			    !folio_test_anon(folio))
 				goto not_found_unlock;
 
 			page += ((addr & (PMD_SIZE - 1)) >> PAGE_SHIFT);
@@ -2582,7 +2583,8 @@ static int ksm_next_page_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned lon
 			continue;
 		folio = page_folio(page);
 
-		if (folio_is_zone_device(folio) || !folio_test_anon(folio))
+		if (unlikely(folio_is_private_managed(folio)) ||
+		    !folio_test_anon(folio))
 			continue;
 		goto found_unlock;
 	}
Signature · 1 line
2.53.0-Meta
author maintainer
#34 lore raw

PATCH v5 16/36 mm/khugepaged: skip private node folios when trying to collapse.

In reply to Gregory Price

A collapse operation causes new THP allocation to occur, and may migrate memory from one node to another.

Handle this the same as zone_device for now (disallow collapse).

---
 mm/khugepaged.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
mm/khugepaged.c +3 -3
index 89ce6bcbc376b..fb4378cc17b10 100644
@@ -700,7 +700,7 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 			goto out;
 		}
 		page = vm_normal_page(vma, addr, pteval);
-		if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
+		if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
 			result = SCAN_PAGE_NULL;
 			goto out;
 		}
@@ -1687,7 +1687,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		}
 
 		page = vm_normal_page(vma, addr, pteval);
-		if (unlikely(!page) || unlikely(is_zone_device_page(page))) {
+		if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
 			result = SCAN_PAGE_NULL;
 			goto out_unmap;
 		}
@@ -1944,7 +1944,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign
 		}
 
 		page = vm_normal_page(vma, addr, ptent);
-		if (WARN_ON_ONCE(page && is_zone_device_page(page)))
+		if (WARN_ON_ONCE(page && page_is_private_managed(page)))
 			page = NULL;
 		/*
 		 * Note that uprobe, debugger, or MAP_PRIVATE may change the
Signature · 1 line
2.53.0-Meta
#35 lore raw

Re: [PATCH v5 16/36] mm/khugepaged: skip private node folios when trying to collapse.

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] The replacement of is_zone_device_page() with page_is_private_managed() inside a WARN_ON_ONCE in try_collapse_pte_mapped_thp() triggers spurious kernel warnings and aborts PMD restoration for valid memory on private NUMA nodes.

Pre-existing issues:
- [Critical] Functions structurally modifying page tables under mmap_write_lock() fail to call vma_start_write(), leading to critical Use-After-Free race conditions with concurrent per-VMA lock readers (like MADV_DONTNEED). --

--- Patch [16]: [PATCH v5 16/36] mm/khugepaged: skip private node folios when trying to collapse. --- commit c62053ea9e5edc37182a3d1e8367dfb0ed6fbf08
Author: Gregory Price <[email protected]>

mm/khugepaged: skip private node folios when trying to collapse.

A collapse operation causes new THP allocation to occur, and may migrate memory from one node to another. Handle this the same as zone_device for now (disallow collapse).

Quoted 4 lines
mm/khugepaged.c +0 -0
index 89ce6bcbc376b..fb4378cc17b10 100644

[ ... ]

mm/khugepaged.c static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign · line 1947

Quoted 6 lines
mm/khugepaged.c +1 -1
@@ -1944,7 +1944,7 @@ static enum scan_result try_collapse_pte_mapped_thp(struct mm_struct *mm, unsign
 		}
 
 		page = vm_normal_page(vma, addr, ptent);
-		if (WARN_ON_ONCE(page && is_zone_device_page(page)))
+		if (WARN_ON_ONCE(page && page_is_private_managed(page)))

[Severity: High]
Does replacing is_zone_device_page() with page_is_private_managed() inside this WARN_ON_ONCE trigger spurious warnings and abort PMD restoration?

When a user maps a file-backed THP on a private NUMA node, temporarily splits the PMD into PTEs, and then triggers a PMD collapse, we reach try_collapse_pte_mapped_thp(). Because page_is_private_managed() can be true for valid memory on private nodes, won't this trigger the warning and unintentionally set page to NULL?

If panic_on_warn is enabled, this could be used by an unprivileged user to reliably crash the system.

[Severity: Critical]
This is a pre-existing issue, but does try_collapse_pte_mapped_thp() need to ensure vma_start_write() is called to prevent a use-after-free with concurrent per-VMA lock readers?

When a thread registers a uprobe via register_for_each_vma(), it acquires the mmap_write_lock() and eventually calls into this function:

kernel/events/uprobes.c:register_for_each_vma() {
    ...
    mmap_write_lock(mm);
    if (check_stable_address_space(mm))
        goto unlock;
    vma = find_vma(mm, info->vaddr);
    ...
}

try_collapse_pte_mapped_thp() then clears PMDs and frees PTE pages under the assumption of exclusive access:

mm/khugepaged.c:try_collapse_pte_mapped_thp() {
    ...
    result = find_pmd_or_thp_or_none(mm, haddr, &pmd);
    ...
    start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl);
    ...
}

However, if another thread concurrently executes a VMA-locked operation like MADV_DONTNEED, it only holds the VMA read lock. Because vma_start_write() was never called to drain these concurrent readers, the two threads can race to free or access the same PTE page.

Quoted 3 lines
mm/khugepaged.c +0 -0
 			page = NULL;
 		/*
 		 * Note that uprobe, debugger, or MAP_PRIVATE may change the
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
author maintainer
#36 lore raw

PATCH v5 17/36 mm/vmscan: disallow reclaim of private node memory

In reply to Gregory Price

Private nodes do not support reclaim by default, so prevent all reclaimers from operating on private nodes.

shrink_node() is the common chokepoint for most reclaimers (direct, kswapd, proactive, mglru, etc), so it's the best place to filter.

DAMON's proactive pageout bypasses shrink_node() entirely, so we disable DAMON in a separate commit as its own discrete service.

Other reclaim entry points are already covered - e.g. MGLRU's debugfs interface rejects nids that are not N_MEMORY (N_MEMORY_PRIVATE is mutually exclusive), and madvise pageout/cold skips private nodes.

---
 mm/vmscan.c | 7 +++++++
 1 file changed, 7 insertions(+)
mm/vmscan.c +7 -0
index e26c6931f5fde..86b2334c23b98 100644
@@ -6141,6 +6141,13 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)
 	struct lruvec *target_lruvec;
 	bool reclaimable = false;
 
+	/*
+	 * Private nodes do not support reclaim by default, filtering here
+	 * captures all normal reclaim paths that may attempt eviction.
+	 */
+	if (node_is_private(pgdat->node_id))
+		return;
+
 	if ((lru_gen_enabled() || lru_gen_switching()) && root_reclaim(sc)) {
 		memset(&sc->nr, 0, sizeof(sc->nr));
 		lru_gen_shrink_node(pgdat, sc);
Signature · 1 line
2.53.0-Meta
#37 lore raw

Re: [PATCH v5 17/36] mm/vmscan: disallow reclaim of private node memory

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Soft-limit reclaim bypasses the new private node filter, allowing eviction on private nodes. --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Soft-limit reclaim bypasses the new private node filter, allowing eviction on private nodes. --

--- Patch [17]: [PATCH v5 17/36] mm/vmscan: disallow reclaim of private node memory --- commit 9fc4bff86adde3091fee2924952f9af9700e6308
Author: Gregory Price <[email protected]>

mm/vmscan: disallow reclaim of private node memory

This commit attempts to prevent reclaim on private nodes by adding a filter to shrink_node().

Quoted 5 lines
mm/vmscan.c +0 -0
index e26c6931f5fde..86b2334c23b98 100644

mm/vmscan.c static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) · line 6150

Quoted 10 lines
mm/vmscan.c +7 -0
@@ -6141,6 +6141,13 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)
 	struct lruvec *target_lruvec;
 	bool reclaimable = false;
 
+	/*
+	 * Private nodes do not support reclaim by default, filtering here
+	 * captures all normal reclaim paths that may attempt eviction.
+	 */
+	if (node_is_private(pgdat->node_id))
+		return;
+

[Severity: High]
Does this filter successfully capture all normal reclaim paths?

When the system uses memcg v1 and a cgroup exceeds its soft limit, global direct reclaim in shrink_zones() or kswapd in balance_pgdat() will invoke memcg1_soft_limit_reclaim(). This bypasses shrink_node() entirely:

mm/vmscan.c:shrink_zones() {
    ...
    nr_soft_reclaimed = memcg1_soft_limit_reclaim(zone->zone_pgdat,
                                                  sc->order, sc->gfp_mask,
                                                  &nr_soft_scanned);
    ...
}

This leads to mem_cgroup_shrink_node() directly calling shrink_lruvec() without checking if the node is private:

mm/vmscan.c:mem_cgroup_shrink_node() {
    ...
    shrink_lruvec(lruvec, &sc);
    ...
}

Will this allow unintended eviction on private nodes during soft limit reclaim?

Quoted 3 lines
mm/vmscan.c +0 -0
 	if ((lru_gen_enabled() || lru_gen_switching()) && root_reclaim(sc)) {
 		memset(&sc->nr, 0, sizeof(sc->nr));
 		lru_gen_shrink_node(pgdat, sc);
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
author maintainer
#38 lore raw

PATCH v5 18/36 mm/gup: disallow longterm pin of private node folios

In reply to Gregory Price

FOLL_LONGTERM on a N_MEMORY_PRIVATE node folio is wrong both ways the GUP longterm path resolves it:

a ZONE_NORMAL folio would be pinned in place a ZONE_MOVABLE folio would be migrated off first

A private node folio should do neither - the pin should fail in place.

Add folio_longterm_pin_forbidden() (true for any private node folio) and reject such a pin in check_and_migrate_movable_pages_or_folios() before any isolation, so nothing is migrated.

The gup-fast path defers a private folio to the slow path via folio_allows_longterm_pin().

---
 mm/gup.c      | 27 +++++++++++++++++++++++++--
 mm/internal.h | 25 +++++++++++++++++++++++++
 2 files changed, 50 insertions(+), 2 deletions(-)
mm/gup.c +25 -2
index 1d32e9a3dc79c..a7d4de223785c 100644
@@ -547,10 +547,10 @@ static struct folio *try_grab_folio_fast(struct page *page, int refs,
 	/*
 	 * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a
 	 * right zone, so fail and let the caller fall back to the slow
-	 * path.
+	 * path.  Fail for private-node folios here so slow path rejects.
 	 */
 	if (unlikely((flags & FOLL_LONGTERM) &&
-		     !folio_is_longterm_pinnable(folio))) {
+		     !folio_allows_longterm_pin(folio))) {
 		folio_put_refs(folio, refs);
 		return NULL;
 	}
@@ -2389,12 +2389,35 @@ migrate_longterm_unpinnable_folios(struct list_head *movable_folio_list,
 	return ret;
 }
 
+/*
+ * True if any folio sits on a private node whose folios may not be longterm
+ * pinned.  Such folios can neither be pinned nor migrated, so the whole pin
+ * must fail before migration.  Checked before any isolation occurs.
+ */
+static bool pofs_has_ltpin_forbidden(struct pages_or_folios *pofs)
+{
+	struct folio *folio;
+	long i = 0;
+
+	for (folio = pofs_get_folio(pofs, i); folio;
+	     folio = pofs_next_folio(folio, pofs, &i)) {
+		if (folio_longterm_pin_forbidden(folio))
+			return true;
+	}
+	return false;
+}
+
 static long
 check_and_migrate_movable_pages_or_folios(struct pages_or_folios *pofs)
 {
 	LIST_HEAD(movable_folio_list);
 	unsigned long collected;
 
+	if (pofs_has_ltpin_forbidden(pofs)) {
+		pofs_unpin(pofs);
+		return -EFAULT;
+	}
+
 	collected = collect_longterm_unpinnable_folios(&movable_folio_list,
 						       pofs);
 	if (!collected)
mm/internal.h +25 -0
index cb9f4a8342e32..85c460296cea1 100644
@@ -110,6 +110,31 @@ static inline bool page_is_private_managed(struct page *page)
 	return folio_is_private_managed(page_folio(page));
 }
 
+/*
+ * folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
+ *
+ * checks folio_is_longterm_pinnable() rules plus private node permissions.
+ * private node permission is checked here to resolve circular header
+ * dependencies in folio_is_longterm_pinnable.
+ */
+static inline bool folio_allows_longterm_pin(struct folio *folio)
+{
+	return folio_is_longterm_pinnable(folio) &&
+	       !folio_is_private_node(folio);
+}
+
+/*
+ * folio_longterm_pin_forbidden() - must a longterm pin of this folio fail
+ * outright (neither pinned in place nor migrated off the node)?
+ *
+ * True for any folio on a private node: such memory can be neither pinned
+ * nor migrated, so the pin must be rejected with the folio left in place.
+ */
+static inline bool folio_longterm_pin_forbidden(struct folio *folio)
+{
+	return folio_is_private_node(folio);
+}
+
 /*
  * Maintains state across a page table move. The operation assumes both source
  * and destination VMAs already exist and are specified by the user.
Signature · 1 line
2.53.0-Meta
author maintainer
#39 lore raw

PATCH v5 19/36 proc: include N_MEMORY_PRIVATE nodes in numa_maps output

In reply to Gregory Price

numa_maps collects per-node page counts in the page-table walkers and emits them in show_numa_map. All three filtered by N_MEMORY, so pages on N_MEMORY_PRIVATE nodes were never gathered/printed.

Accept N_MEMORY_PRIVATE pages in both walkers and emit private nodes in show_numa_map, so private-node mappings are visible in numa_maps.

---
 fs/proc/task_mmu.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
fs/proc/task_mmu.c +8 -2
index 817e3e0f91943..e116cd5f157b9 100644
@@ -3389,7 +3389,8 @@ static struct page *can_gather_numa_stats(pte_t pte, struct vm_area_struct *vma,
 		return NULL;
 
 	nid = page_to_nid(page);
-	if (!node_isset(nid, node_states[N_MEMORY]))
+	if (!node_isset(nid, node_states[N_MEMORY]) &&
+	    !node_isset(nid, node_states[N_MEMORY_PRIVATE]))
 		return NULL;
 
 	return page;
@@ -3414,7 +3415,8 @@ static struct page *can_gather_numa_stats_pmd(pmd_t pmd,
 		return NULL;
 
 	nid = page_to_nid(page);
-	if (!node_isset(nid, node_states[N_MEMORY]))
+	if (!node_isset(nid, node_states[N_MEMORY]) &&
+	    !node_isset(nid, node_states[N_MEMORY_PRIVATE]))
 		return NULL;
 
 	return page;
@@ -3602,6 +3604,10 @@ static int show_numa_map(struct seq_file *m, void *v)
 		if (md->node[nid])
 			seq_printf(m, " N%d=%lu", nid, md->node[nid]);
 
+	for_each_node_state(nid, N_MEMORY_PRIVATE)
+		if (md->node[nid])
+			seq_printf(m, " N%d=%lu", nid, md->node[nid]);
+
 	seq_printf(m, " kernelpagesize_kB=%lu", vma_kernel_pagesize(vma) >> 10);
 out:
 	seq_putc(m, '\n');
Signature · 1 line
2.53.0-Meta
#40 lore raw

Re: [PATCH v5 19/36] proc: include N_MEMORY_PRIVATE nodes in numa_maps output

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [High] Using ACTION_AGAIN when pte_offset_map_lock() fails can create an infinite loop. --

--- Patch [19]: [PATCH v5 19/36] proc: include N_MEMORY_PRIVATE nodes in numa_maps output --- commit 744e7cd9225bc94011608ebdc2dbdb49cbdf8476
Author: Gregory Price <[email protected]>

proc: include N_MEMORY_PRIVATE nodes in numa_maps output

numa_maps collects per-node page counts in the page-table walkers and emits them in show_numa_map. All three filtered by N_MEMORY, so pages on N_MEMORY_PRIVATE nodes were never gathered/printed.

Accept N_MEMORY_PRIVATE pages in both walkers and emit private nodes in show_numa_map, so private-node mappings are visible in numa_maps.

Quoted 4 lines
fs/proc/task_mmu.c +0 -0
index 817e3e0f91943..e116cd5f157b9 100644

[ ... ]

fs/proc/task_mmu.c static struct page *can_gather_numa_stats_pmd(pmd_t pmd, · line 3422

Quoted 10 lines
fs/proc/task_mmu.c +2 -1
@@ -3414,7 +3415,8 @@ static struct page *can_gather_numa_stats_pmd(pmd_t pmd,
 		return NULL;
 
 	nid = page_to_nid(page);
-	if (!node_isset(nid, node_states[N_MEMORY]))
+	if (!node_isset(nid, node_states[N_MEMORY]) &&
+	    !node_isset(nid, node_states[N_MEMORY_PRIVATE]))
 		return NULL;
 
 	return page;

[Severity: High]
This is a pre-existing issue, but can the error handling in gather_pte_stats() lead to an infinite loop?

When reading from procfs memory mapping files (like numa_maps), pte_offset_map_lock() can fail non-transiently for a PMD migration entry and return NULL:

fs/proc/task_mmu.c:gather_pte_stats() {
	...
	orig_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl);
	if (!pte) {
		walk->action = ACTION_AGAIN;
		return 0;
	}
	...
}

Because this sets walk->action = ACTION_AGAIN, wouldn't the walk_pmd_range() core unconditionally retry the same PMD entry indefinitely without making forward progress?

The same pattern also appears to exist in smaps_pte_range(), clear_refs_pte_range(), pagemap_pmd_range(), and pagemap_scan_pmd_entry().

[ ... ]

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=19
author maintainer
#41 lore raw

PATCH v5 20/36 mm/memcontrol: account private-node memory in per-node stats

In reply to Gregory Price

Private nodes folios are charged like any other - the node's per-cpu lruvec counters are increment even for N_MEMORY_PRIVATE.

The memcg-level totals in memory.stat therefore include this memory, but the per-node views do not:

- mem_cgroup_css_rstat_flush() folds the per-node percpu deltas into the aggregate only for N_MEMORY nodes, so a private node's lruvec aggregate is never refreshed.

- memory.numa_stat (v2) and the v1 numa_stat skip private nodes, so they never emit an Nx= field for one.

So memory.numa_stat does not sum to memory.stat once a cgroup has memory on a private node.

Include N_MEMORY_PRIVATE in the flush/display process so private-node memory is folded in and reported per node.

This is pure accounting of memory that already exists and is already charged, so it is unconditional - regardless of future capabilities.

---
 mm/memcontrol-v1.c |  8 ++++++--
 mm/memcontrol.c    | 11 +++++++++--
 2 files changed, 15 insertions(+), 4 deletions(-)
mm/memcontrol-v1.c +6 -2
index 2dc599484d006..5e464e52dfb9a 100644
@@ -2132,14 +2132,18 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)
 	const struct numa_stat *stat;
 	int nid;
 	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
+	nodemask_t reportable;
 
 	mem_cgroup_flush_stats(memcg);
 
+	/* Private nodes hold cgroup memory too, report them. */
+	nodes_or(reportable, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+
 	for (stat = stats; stat < ARRAY_END(stats); stat++) {
 		seq_printf(m, "%s=%lu", stat->name,
 			   mem_cgroup_nr_lru_pages(memcg, stat->lru_mask,
 						   false));
-		for_each_node_state(nid, N_MEMORY)
+		for_each_node_mask(nid, reportable)
 			seq_printf(m, " N%d=%lu", nid,
 				   mem_cgroup_node_nr_lru_pages(memcg, nid,
 							stat->lru_mask, false));
@@ -2151,7 +2155,7 @@ static int memcg_numa_stat_show(struct seq_file *m, void *v)
 		seq_printf(m, "hierarchical_%s=%lu", stat->name,
 			   mem_cgroup_nr_lru_pages(memcg, stat->lru_mask,
 						   true));
-		for_each_node_state(nid, N_MEMORY)
+		for_each_node_mask(nid, reportable)
 			seq_printf(m, " N%d=%lu", nid,
 				   mem_cgroup_node_nr_lru_pages(memcg, nid,
 							stat->lru_mask, true));
mm/memcontrol.c +9 -2
index f0dde52dc9e0e..4d79e238bc57d 100644
@@ -4496,6 +4496,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
 	struct mem_cgroup *parent = parent_mem_cgroup(memcg);
 	struct memcg_vmstats_percpu *statc;
 	struct aggregate_control ac;
+	nodemask_t reportable;
 	int nid;
 
 	flush_nmi_stats(memcg, parent);
@@ -4524,7 +4525,9 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
 	};
 	mem_cgroup_stat_aggregate(&ac);
 
-	for_each_node_state(nid, N_MEMORY) {
+	/* Private nodes must also be accounted or numa_stat is misleading */
+	nodes_or(reportable, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+	for_each_node_mask(nid, reportable) {
 		struct mem_cgroup_per_node *pn = memcg->nodeinfo[nid];
 		struct lruvec_stats *lstats = pn->lruvec_stats;
 		struct lruvec_stats *plstats = NULL;
@@ -4947,9 +4950,13 @@ static int memory_numa_stat_show(struct seq_file *m, void *v)
 {
 	int i;
 	struct mem_cgroup *memcg = mem_cgroup_from_seq(m);
+	nodemask_t reportable;
 
 	mem_cgroup_flush_stats(memcg);
 
+	/* Private nodes hold cgroup memory too, report them. */
+	nodes_or(reportable, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+
 	for (i = 0; i < ARRAY_SIZE(memory_stats); i++) {
 		int nid;
 
@@ -4957,7 +4964,7 @@ static int memory_numa_stat_show(struct seq_file *m, void *v)
 			continue;
 
 		seq_printf(m, "%s", memory_stats[i].name);
-		for_each_node_state(nid, N_MEMORY) {
+		for_each_node_mask(nid, reportable) {
 			u64 size;
 			struct lruvec *lruvec;
 
Signature · 1 line
2.53.0-Meta
#42 lore raw

Re: [PATCH v5 20/36] mm/memcontrol: account private-node memory in per-node stats

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] The patch updates `mem_cgroup_css_rstat_flush` to include `N_MEMORY_PRIVATE` nodes, but misses…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] The patch updates mem_cgroup_css_rstat_flush to include N_MEMORY_PRIVATE nodes, but misses flush_nmi_stats() which still strictly iterates over N_MEMORY, leaking NMI slab stats on private nodes.
- [Medium] hugetlb_cgroup_read_numa_stat() in the hugetlb cgroup controller skips N_MEMORY_PRIVATE nodes, leading to hidden node memory and incorrect hierarchical totals.
--

--- Patch [20]: [PATCH v5 20/36] mm/memcontrol: account private-node memory in per-node stats --- commit 923e253e1d279f965b728ad3baafd46c0f7044eb
Author: Gregory Price <[email protected]>

mm/memcontrol: account private-node memory in per-node stats

Private nodes folios are charged like any other - the node's per-cpu lruvec counters are increment even for N_MEMORY_PRIVATE.

[ ... ]

mm/memcontrol.c static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) · line 4502

Quoted 12 lines
mm/memcontrol.c +1 -0
index f0dde52dc9e0e..4d79e238bc57d 100644
@@ -4496,6 +4496,7 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu)
 	struct mem_cgroup *parent = parent_mem_cgroup(memcg);
 	struct memcg_vmstats_percpu *statc;
 	struct aggregate_control ac;
+	nodemask_t reportable;
 	int nid;
 
 	flush_nmi_stats(memcg, parent);

[Severity: Medium]
Will this still leak NMI slab stats on private nodes?

flush_nmi_stats() still strictly iterates over N_MEMORY instead of including N_MEMORY_PRIVATE.

If a slab allocation occurs in NMI context on an N_MEMORY_PRIVATE node, the node-local atomic counter is incremented, but it appears flush_nmi_stats() will fail to drain it because it only iterates over N_MEMORY.

[ ... ]

[Severity: Medium]
Does mm/hugetlb_cgroup.c:hugetlb_cgroup_read_numa_stat() also need to be updated to include N_MEMORY_PRIVATE?

It currently only covers N_MEMORY when calculating and printing per-node hierarchical usage:

	for_each_node_state(nid, N_MEMORY) {
		usage = 0;
		...

If hugetlb memory is allocated on an N_MEMORY_PRIVATE node and charged to a cgroup, reading the cgroup's NUMA stat file would skip accumulating or displaying this node's usage.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=20
author maintainer
#43 lore raw

PATCH v5 21/36 proc/kcore: include private-node RAM in the kcore RAM map

In reply to Gregory Price

The kcore_ram_list bounds its scan at the highest pfn in N_MEMORY.

As a result, private nodes placed above the last N_MEMORY node, which is the common case for device/CXL memory - is truncated out of /proc/kcore.

Include N_MEMORY_PRIVATE nodes in the scan boundary. /proc/kcore is root-only (CAP_SYS_RAWIO), so this exposes nothing new - it just lets a debugger read private-node memory like any other RAM.

---
 fs/proc/kcore.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
fs/proc/kcore.c +4 -1
index 390547b992acd..f67194dc2a9b8 100644
@@ -251,11 +251,14 @@ static int kcore_ram_list(struct list_head *list)
 {
 	int nid, ret;
 	unsigned long end_pfn;
+	nodemask_t mem;
 
 	/* Not initialized....update now */
 	/* find out "max pfn" */
 	end_pfn = 0;
-	for_each_node_state(nid, N_MEMORY) {
+	/* Include private node memory in the scan */
+	nodes_or(mem, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+	for_each_node_mask(nid, mem) {
 		unsigned long node_end;
 		node_end = node_end_pfn(nid);
 		if (end_pfn < node_end)
Signature · 1 line
2.53.0-Meta
maintainer Acked-by on v5 21/36
#44 lore raw

Re: [PATCH v5 21/36] proc/kcore: include private-node RAM in the kcore RAM map

Quoted 37 lines · Gregory Price

The kcore_ram_list bounds its scan at the highest pfn in N_MEMORY.

As a result, private nodes placed above the last N_MEMORY node, which is the common case for device/CXL memory - is truncated out of /proc/kcore.

Include N_MEMORY_PRIVATE nodes in the scan boundary. /proc/kcore is root-only (CAP_SYS_RAWIO), so this exposes nothing new - it just lets a debugger read private-node memory like any other RAM.

---
 fs/proc/kcore.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
fs/proc/kcore.c +4 -1
index 390547b992acd..f67194dc2a9b8 100644
@@ -251,11 +251,14 @@ static int kcore_ram_list(struct list_head *list)
 {
 	int nid, ret;
 	unsigned long end_pfn;
+	nodemask_t mem;
 
 	/* Not initialized....update now */
 	/* find out "max pfn" */
 	end_pfn = 0;
-	for_each_node_state(nid, N_MEMORY) {
+	/* Include private node memory in the scan */
+	nodes_or(mem, node_states[N_MEMORY], node_states[N_MEMORY_PRIVATE]);
+	for_each_node_mask(nid, mem) {
 		unsigned long node_end;
 		node_end = node_end_pfn(nid);
 		if (end_pfn < node_end)
Signature · 1 line
2.53.0-Meta

I'm sure this is the least of your concern with this series, but FWIW:

If everything goes well, kcore.c is quiet enough that this can probably just be merged through whatever tree takes the rest of this work.

author maintainer
#45 lore raw

PATCH v5 22/36 mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection

In reply to Gregory Price

Add MPOL_F_PRIVATE, the internal flag noting that a policy contains a nodemask with a private node, and making private node memory reachable via standard mempolicies.

Plumb zonelist selection into the mpol allocator interfaces via the allocator's alloc_flags, using the alloc_flags-carrying page_alloc interfaces.

mpol_alloc_flags() maps MPOL_F_PRIVATE to the alloc_flags an allocation uses:

    ALLOC_DEFAULT            -  normal allocation
    ALLOC_ZONELIST_PRIVATE   -  private node allocation

With this, a VMA with (mpol->flags & MPOL_F_PRIVATE) can successfully services faults like any other mbind() from the private node:

   buf = mmap(..., MAP_ANON);
   mbind(buf, ..., {private_node});
   buf[0] = 0xdeadbeef;  // Page faulted from private node memory

Like any other bind, the nodemask relaxes if it is unsatisfiable.

For example: apply_policy_zone() allows an unmovable allocation targeting the VMA to fallback to a viable node instead of failing.

cpuset rebinding does not affect private nodes - the nodes in the original mask are preserved, and we enforce two remap rules:

1) never allow N_MEMORY nodes to remap to N_MEMORY_PRIVATE
2) never remap N_MEMORY_PRIVATE nodes at all (they stay in place)

In the case of an empty nodemask as a result of rebind, revert to cpuset - which never includes N_MEMORY_PRIVATE and always guarantees at least one N_MEMORY node.

As of this patch, nothing can actually sets MPOL_F_PRIVATE, so this is a no-op that simply adds the plumbing throughout mempolicy.

---
 include/uapi/linux/mempolicy.h |  1 +
 mm/mempolicy.c                 | 79 +++++++++++++++++++++++-----------
 2 files changed, 55 insertions(+), 25 deletions(-)
include/uapi/linux/mempolicy.h +1 -0
index 7f6fc9599693b..87af18c84b947 100644
@@ -67,6 +67,7 @@ enum mempolicy_mode {
 #define MPOL_F_SHARED  (1 << 0)	/* identify shared policies */
 #define MPOL_F_MOF	(1 << 3) /* this policy wants migrate on fault */
 #define MPOL_F_MORON	(1 << 4) /* Migrate On protnone Reference On Node */
+#define MPOL_F_PRIVATE	(1 << 5) /* nodemask contains private nodes */
 
 /*
  * Enabling zone reclaim means the page allocator will attempt to fulfill
mm/mempolicy.c +54 -25

195 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#46 lore raw

Re: [PATCH v5 22/36] mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 5 potential issue(s) to consider:

New issues:
- [High] Private nodes are permanently lost from MPOL_F_STATIC_NODES policies during memory offlining/rebinding.
- [High] Mempolicies targeting private nodes will silently fail to allocate from them due to cpuset restrictions.
- [Medium] Non-atomic clearing of MPOL_F_PRIVATE in pol->flags causes a data race with lockless page allocators.

Pre-existing issues:
- [High] Lockless and unprotected access to pol->nodes in alloc_pages_bulk_interleave can lead to a divide-by-zero crash.
- [High] Unconditional GFP_KERNEL allocation in alloc_pages_bulk_weighted_interleave violates caller's GFP constraints. --

--- Patch [22]: [PATCH v5 22/36] mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection --- commit 11224b898bbcf008cfe6641a6cd14bd637edb1d1
Author: Gregory Price <[email protected]>

mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection

This patch adds MPOL_F_PRIVATE to mempolicy flags to identify policies containing private nodes, allowing private node memory to be reachable. It also plumbs zonelist selection into the mempolicy allocator interfaces using the alloc_flags mechanism.

Quoted 4 lines
mm/mempolicy.c +0 -0
index 2b76c57a460c9..90110e9761122 100644

[ ... ]

mm/mempolicy.c static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes) · line 515

Quoted 9 lines
mm/mempolicy.c +4 -1
@@ -505,22 +509,33 @@ static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes)
 
 static void mpol_rebind_nodemask(struct mempolicy *pol, const nodemask_t *nodes)
 {
-	nodemask_t tmp;
+	nodemask_t tmp, priv;
+
+	/* preserve online private nodes to re-add later */
+	nodes_and(priv, pol->nodes, node_states[N_MEMORY_PRIVATE]);

[Severity: High]
Does this evaluate private nodes against pol->nodes instead of pol->w.user_nodemask?

If a private node targeted by an MPOL_F_STATIC_NODES policy goes offline, it is removed from pol->nodes. When the node comes back online and mpol_rebind_nodemask() runs, attempting to restore private nodes from pol->nodes will find it empty.

Could this result in permanent loss of private nodes from static policies?

Quoted 2 lines
mm/mempolicy.c +0 -0
 	if (pol->flags & MPOL_F_STATIC_NODES)
 		nodes_and(tmp, pol->w.user_nodemask, *nodes);

[ ... ]

mm/mempolicy.c static void mpol_rebind_default(struct mempolicy *pol, const nodemask_t *nodes) · line 538

Quoted 12 lines
mm/mempolicy.c +8 -2
-	if (nodes_empty(tmp))
+	/* private nodes are identity-mapped during remap, drop them here */
+	nodes_and(tmp, tmp, node_states[N_MEMORY]);
+	if (nodes_empty(tmp) && nodes_empty(priv))
 		tmp = *nodes;
 
-	pol->nodes = tmp;
+	/* If any online private nodes remain, add them back */
+	nodes_or(pol->nodes, tmp, priv);
+	/* If no online private nodes remain, strip the private flag */
+	if (nodes_empty(priv))
+		pol->flags &= ~MPOL_F_PRIVATE;

[Severity: Medium]
Does clearing MPOL_F_PRIVATE non-atomically create a data race with lockless page allocators?

When mpol_alloc_flags() concurrently reads (pol->flags & MPOL_F_PRIVATE) locklessly in the page allocator fastpath, could this result in torn reads and trigger KCSAN warnings?

Quoted 1 line

}

[ ... ]

mm/mempolicy.c static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order, · line 2516

Quoted 6 lines
mm/mempolicy.c +1 -1
@@ -2488,7 +2513,7 @@ static struct page *alloc_pages_mpol(gfp_t gfp, unsigned int order,
 		}
 	}
 
-	page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask, ALLOC_DEFAULT);
+	page = __alloc_frozen_pages_noprof(gfp, order, nid, nodemask, aflags);

[Severity: High]
For MPOL_BIND policies, will cpuset_nodemask_valid_mems_allowed() return false because private nodes are excluded from cpuset.mems, setting nodemask to NULL?

When a NULL nodemask is passed to the allocator, the fastpath defaults to cpuset_current_mems_allowed with ALLOC_CPUSET. Since private nodes are absent from the cpuset, won't __cpuset_zone_allowed() skip the private nodes regardless of ALLOC_ZONELIST_PRIVATE?

Will mempolicies targeting private nodes silently fail to allocate from them and fall back to standard nodes?

Quoted 1 line
mm/mempolicy.c +0 -0
 	if (unlikely(pol->mode == MPOL_INTERLEAVE ||

[ ... ]

mm/mempolicy.c static unsigned long alloc_pages_bulk_interleave(gfp_t gfp, · line 2627

Quoted 7 lines
mm/mempolicy.c +1 -0
@@ -2597,6 +2622,7 @@ static unsigned long alloc_pages_bulk_interleave(gfp_t gfp,
 		struct mempolicy *pol, unsigned long nr_pages,
 		struct page **page_array)
 {
+	unsigned int aflags = mpol_alloc_flags(pol);
 	int nodes;
 	unsigned long nr_pages_per_node;

[Severity: High]
This isn't a bug introduced by this patch, but is it safe to read pol->nodes locklessly in alloc_pages_bulk_interleave()?

alloc_pages_bulk_interleave() reads the nodemask word-by-word via nodes_weight(pol->nodes) further down in this function. Concurrently, mpol_rebind_nodemask() mutates pol->nodes word-by-word via nodes_or(pol->nodes, tmp, priv).

On systems with >64 nodes, could a torn read result in a weight of 0, causing a divide-by-zero panic when calculating nr_pages_per_node = nr_pages / nodes?

[ ... ]

mm/mempolicy.c static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp, · line 2663

Quoted 7 lines
mm/mempolicy.c +1 -0
@@ -2632,6 +2658,7 @@ static unsigned long alloc_pages_bulk_weighted_interleave(gfp_t gfp,
 		struct mempolicy *pol, unsigned long nr_pages,
 		struct page **page_array)
 {
+	unsigned int aflags = mpol_alloc_flags(pol);
 	struct weighted_interleave_state *state;
 	struct task_struct *me = current;

[Severity: High]
This is a pre-existing issue, but does alloc_pages_bulk_weighted_interleave() violate the caller's GFP constraints?

This function can be called via alloc_pages_bulk_mempolicy_noprof() from vmalloc with restrictive flags like GFP_NOFS or GFP_NOIO. However, it allocates its weights array unconditionally using kzalloc(nr_node_ids, GFP_KERNEL) later in the function.

Could this strip the __GFP_FS or __GFP_IO protection and allow the allocation to recurse into the filesystem or block layer, potentially causing a deadlock?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=22
author maintainer
#47 lore raw

PATCH v5 23/36 mm/mempolicy: apply policy at the kernel zone for private-node binds

In reply to Gregory Price

apply_policy_zone() raises dynamic_policy_zone to ZONE_MOVABLE when none of a policy's nodes have non-movable memory, so a bind to movable-only node does not force kernel-zone allocations onto a zone that cannot satisfy them - resulting retries forever (livelock).

The current code uses N_HIGH_MEMORY to denote eligibility, but private nodes are deliberately not in N_HIGH_MEMORY nor N_NORMAL_MEMORY. This means a private node onlined into ZONE_NORMAL is judged as movable-only.

This means a policy set with set_mempolicy() binding a private node with ZONE_NORMAL memory will shunt otherwise eligible allocations (like the task's page tables) back to N_MEMORY nodes.

This also becomes relevant for in-kernel users that want non-movable memory (such as KVM placing pinned VM memory solely on a private node).

Add policy_private_has_kernel_zone() to consult the actual zones of the policy's private nodes and keep the policy applied at lower zones when a kernel-zoned private node can satisfy the allocation.

Only a mempolicy binding a private node pays the zone scan, all other mempolicies preserve fast-path performance.

Additionally: update the comments in apply_policy_zone for clarity, because this particular policy mechanism is very confusing.

---
 mm/mempolicy.c | 48 ++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 42 insertions(+), 6 deletions(-)
mm/mempolicy.c +42 -6
index 90110e9761122..e83c2c7a94c1d 100644
@@ -2091,6 +2091,34 @@ bool vma_policy_mof(struct vm_area_struct *vma)
 	return mof;
 }
 
+/*
+ * true if any policy *private* node can satisfy a !__GFP_MOVABLE allocation.
+ * Policies without private nodes (~MPOL_F_PRIVATE) always return false.
+ *
+ * This provides a way for otherwise-bound kernel allocations to make their
+ * way onto private nodes with ZONE_NORMAL memory without having to audit
+ * every caller.  The cost is a zone scan for private nodes in the mask.
+ */
+static bool policy_private_has_kernel_zone(const struct mempolicy *pol)
+{
+	int nid;
+
+	if (!(pol->flags & MPOL_F_PRIVATE))
+		return false;
+
+	for_each_node_mask(nid, pol->nodes) {
+		pg_data_t *pgdat = NODE_DATA(nid);
+		enum zone_type zt;
+
+		if (!node_is_private(nid))
+			continue;
+		for (zt = ZONE_NORMAL; zt < ZONE_MOVABLE; zt++)
+			if (managed_zone(&pgdat->node_zones[zt]))
+				return true;
+	}
+	return false;
+}
+
 bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
 {
 	enum zone_type dynamic_policy_zone = policy_zone;
@@ -2098,14 +2126,22 @@ bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone)
 	BUG_ON(dynamic_policy_zone == ZONE_MOVABLE);
 
 	/*
-	 * if policy->nodes has movable memory only,
-	 * we apply policy when gfp_zone(gfp) = ZONE_MOVABLE only.
+	 * dynamic_policy_zone is the lowest zone this policy is enforced for,
+	 * allocations below it ignore the nodemask and fall back freely.
+	 *
+	 * If all policy-nodes are movable-only (all ZONE_MOVABLE), raise the
+	 * dynamic_policy_zone to ZONE_MOVABLE so that only movable allocations
+	 * stay bound - otherwise this would force kernel-zone allocations
+	 * (e.g. page tables) onto a zone that cannot satisfy them.  This
+	 * results in allocation failure and retry loop (livelock).
 	 *
-	 * policy->nodes is intersect with node_states[N_MEMORY].
-	 * so if the following test fails, it implies
-	 * policy->nodes has movable memory only.
+	 * N_HIGH_MEMORY tells us "has a non-movable zone" for ordinary nodes.
+	 * Private nodes are deliberately excluded from N_HIGH_MEMORY, so we
+	 * need to check their actual zones - a kernel-zoned private node can
+	 * hold the allocation and must keep the policy applied.
 	 */
-	if (!nodes_intersects(policy->nodes, node_states[N_HIGH_MEMORY]))
+	if (!nodes_intersects(policy->nodes, node_states[N_HIGH_MEMORY]) &&
+	    !policy_private_has_kernel_zone(policy))
 		dynamic_policy_zone = ZONE_MOVABLE;
 
 	return zone >= dynamic_policy_zone;
Signature · 1 line
2.53.0-Meta
#48 lore raw

Re: [PATCH v5 23/36] mm/mempolicy: apply policy at the kernel zone for private-node binds

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] The `policy_private_has_kernel_zone()` function incorrectly skips checking `ZONE_DMA` and…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The policy_private_has_kernel_zone() function incorrectly skips checking ZONE_DMA and ZONE_DMA32, which causes mempolicy bypass for private nodes possessing only lower-zone memory. --

--- Patch [23]: [PATCH v5 23/36] mm/mempolicy: apply policy at the kernel zone for private-node binds --- commit 04782bb72086af5fe29a7dbf4184affd04c79df6
Author: Gregory Price <[email protected]>

mm/mempolicy: apply policy at the kernel zone for private-node binds

This commit introduces policy_private_has_kernel_zone() to prevent mempolicy bypass when private nodes have non-movable memory.

Quoted 4 lines
mm/mempolicy.c +0 -0
index 90110e9761122..e83c2c7a94c1d 100644

[ ... ]

Quoted 1 line
mm/mempolicy.c +0 -0
@@ -2091,6 +2091,34 @@ bool vma_policy_mof(struct vm_area_struct *vma)

[ ... ]

Quoted 4 lines
mm/mempolicy.c +4 -0
+static bool policy_private_has_kernel_zone(const struct mempolicy *pol)
+{
+	int nid;
+

mm/mempolicy.c bool vma_policy_mof(struct vm_area_struct *vma) · line 2115

Quoted 10 lines
mm/mempolicy.c +10 -0
+	if (!(pol->flags & MPOL_F_PRIVATE))
+		return false;
+
+	for_each_node_mask(nid, pol->nodes) {
+		pg_data_t *pgdat = NODE_DATA(nid);
+		enum zone_type zt;
+
+		if (!node_is_private(nid))
+			continue;
+		for (zt = ZONE_NORMAL; zt < ZONE_MOVABLE; zt++)

[Severity: Medium]
Does this loop miss checking ZONE_DMA and ZONE_DMA32 by starting at ZONE_NORMAL?

If a user process sets a mempolicy binding to a private NUMA node that only contains memory in ZONE_DMA or ZONE_DMA32, this loop will return false because it skips those populated lower zones.

This would cause apply_policy_zone() to incorrectly raise dynamic_policy_zone to ZONE_MOVABLE, resulting in kernel allocations falling back to other system nodes and bypassing the policy intent.

Quoted 5 lines
mm/mempolicy.c +5 -0
+			if (managed_zone(&pgdat->node_zones[zt]))
+				return true;
+	}
+	return false;
+}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=23
author maintainer
#49 lore raw

PATCH v5 24/36 mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services

In reply to Gregory Price

Add and export interfaces to enable modules to build a mempolicy with (MPOL_BIND | MPOL_F_PRIVATE) pinned to a private NUMA node, so the drivers can stamp it onto VMAs they control (via vma->vm_policy).

mpol_bind_node() - acts the same as a userland mbind()
mpol_private_bind() - bypasses the CAP_USER_NUMA check.

Export it to modules (private:kmem, bind:kvm).

Widen __mpol_put()'s module export to kmem so the driver can release the policy.

Adjust mpol_set_nodemask to check for a pre-set MPOL_F_PRIVATE flag to allow the mpol_private_bind() check to bypass the N_MEMORY filter.

---
 include/linux/mempolicy.h |  14 ++++++
 mm/mempolicy.c            | 102 +++++++++++++++++++++++++++++++++++++-
 2 files changed, 114 insertions(+), 2 deletions(-)
include/linux/mempolicy.h +14 -0
index 65c732d440d2f..715951a5b03c1 100644
@@ -7,6 +7,7 @@
 #define _LINUX_MEMPOLICY_H 1
 
 #include <linux/sched.h>
+#include <linux/err.h>
 #include <linux/mmzone.h>
 #include <linux/slab.h>
 #include <linux/rbtree.h>
@@ -128,6 +129,9 @@ void mpol_free_shared_policy(struct shared_policy *sp);
 struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,
 					    pgoff_t idx);
 
+struct mempolicy *mpol_private_bind(int nid);
+struct mempolicy *mpol_bind_node(int nid);
+
 struct mempolicy *get_task_policy(struct task_struct *p);
 struct mempolicy *__get_vma_policy(struct vm_area_struct *vma,
 		unsigned long addr, pgoff_t *ilx);
@@ -226,6 +230,16 @@ mpol_shared_policy_lookup(struct shared_policy *sp, pgoff_t idx)
 	return NULL;
 }
 
+static inline struct mempolicy *mpol_private_bind(int nid)
+{
+	return ERR_PTR(-EOPNOTSUPP);
+}
+
+static inline struct mempolicy *mpol_bind_node(int nid)
+{
+	return ERR_PTR(-EOPNOTSUPP);
+}
+
 static inline struct mempolicy *get_vma_policy(struct vm_area_struct *vma,
 				unsigned long addr, int order, pgoff_t *ilx)
 {
mm/mempolicy.c +100 -2

126 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#50 lore raw

Re: [PATCH v5 24/36] mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] `mpol_bind_node()` and `mpol_private_bind()` fail to validate `nid` against bounds, leading to…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mpol_bind_node() and mpol_private_bind() fail to validate nid against bounds, leading to stack buffer overflows and out-of-bounds reads.
- [Medium] __mpol_bind_node() calls mpol_set_nodemask() without holding task_lock(current) or mmap_lock, causing a data race and potential torn read.
- [Medium] mpol_bind_node() unconditionally rejects private node bindings, breaking the documented API contract.
--

--- Patch [24]: [PATCH v5 24/36] mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services --- commit 056427d8f98ec56375b0a715f69798d4dfde2ec3
Author: Gregory Price <[email protected]>

mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services

This patch adds and exports interfaces to enable modules to build MPOL_BIND mempolicies pinned to a private NUMA node. It exports mpol_bind_node() and mpol_private_bind() for module use, widens the __mpol_put() export, and adjusts mpol_set_nodemask() to handle MPOL_F_PRIVATE flags.

Quoted 9 lines
mm/mempolicy.c +1 -0
index e83c2c7a94c1d..a3ffb09897489 100644
@@ -432,6 +432,18 @@ static int mpol_set_nodemask(struct mempolicy *pol,
 	else
 		pol->w.cpuset_mems_allowed = cpuset_current_mems_allowed;
 
+	/*

mm/mempolicy.c static int mpol_set_nodemask(struct mempolicy *pol, · line 445

Quoted 10 lines
mm/mempolicy.c +10 -0
+	 * Private nodes are not in cpuset.mems, so they're always stripped.
+	 * Driver-allocated policies will already have MPOL_F_PRIVATE set,
+	 * if that's the case, add back in the requested set of private nodes.
+	 */
+	for_each_node_mask(nid, *nodes) {
+		if (!node_is_private(nid))
+			continue;
+		if (pol->flags & MPOL_F_PRIVATE)
+			node_set(nid, nsc->mask2);
+	}

[Severity: Medium]
The commit message mentions that N_MEMORY_PRIVATE node bindings are honored if the node has CAP_USER_NUMA. Does this implementation actually perform that capability check?

It looks like this only checks for MPOL_F_PRIVATE. If mpol_bind_node() calls __mpol_bind_node() with flags=0, wouldn't the private node be stripped unconditionally, causing the nodemask to become empty and returning -EINVAL?

[ ... ]

mm/mempolicy.c static long do_set_mempolicy(unsigned short mode, unsigned short flags, · line 1163

Quoted 13 lines
mm/mempolicy.c +13 -0
+static struct mempolicy *__mpol_bind_node(int nid, unsigned short flags)
+{
+	struct mempolicy *pol;
+	nodemask_t nodes;
+	int err;
+
+	NODEMASK_SCRATCH(scratch);
+
+	if (!scratch)
+		return ERR_PTR(-ENOMEM);
+
+	nodes_clear(nodes);
+	node_set(nid, nodes);

[Severity: High]
Should nid be validated against MAX_NUMNODES before being used?

If an invalid or unsanitized node ID like NUMA_NO_NODE (-1) or a value greater than MAX_NUMNODES is passed from a module, could node_set() perform an out-of-bounds write on the stack-allocated nodemask_t?

mm/mempolicy.c static long do_set_mempolicy(unsigned short mode, unsigned short flags, · line 1171

Quoted 8 lines
mm/mempolicy.c +8 -0
+
+	pol = mpol_new(MPOL_BIND, flags, &nodes);
+	if (IS_ERR(pol)) {
+		NODEMASK_SCRATCH_FREE(scratch);
+		return pol;
+	}
+
+	err = mpol_set_nodemask(pol, &nodes, scratch);

[Severity: Medium]
Does this call to mpol_set_nodemask() need to hold task_lock(current) or mmap_lock?

mpol_set_nodemask() reads cpuset_current_mems_allowed (current->mems_allowed). Without locking, couldn't this race with concurrent cpuset updates and result in a torn read or corrupted mempolicy nodemask?

Quoted 19 lines
mm/mempolicy.c +19 -0
+	NODEMASK_SCRATCH_FREE(scratch);
+	if (err) {
+		mpol_put(pol);
+		return ERR_PTR(err);
+	}
+	return pol;
+}
+
+/**
+ * mpol_private_bind - build an MPOL_BIND policy pinned to a private node
+ * @nid: an N_MEMORY_PRIVATE node
+ *
+ * Returns a refcounted mempolicy that binds allocations to @nid with the
+ * private-placement intent (MPOL_F_PRIVATE).  This binds to @nid regardless
+ * of the node's CAP_USER_NUMA, providing a privileged way for node-owners
+ * to bind driver/service owned VMAs to the node.
+ *
+ * Like any MPOL_BIND it is relaxable: an unsatisfiable request falls back
+ * rather than failing.

mm/mempolicy.c static long do_set_mempolicy(unsigned short mode, unsigned short flags, · line 1200

Quoted 10 lines
mm/mempolicy.c +10 -0
+ *
+ * Must be called while @nid is N_MEMORY_PRIVATE.
+ *
+ * The caller owns the reference and frees it with mpol_put().
+ *
+ * Return: the policy, or an ERR_PTR on failure.
+ */
+struct mempolicy *mpol_private_bind(int nid)
+{
+	if (!node_is_private(nid))

[Severity: High]
Similar to the issue in __mpol_bind_node(), if an invalid nid is passed in, could node_is_private() read out of bounds?

Quoted 4 lines
mm/mempolicy.c +4 -0
+		return ERR_PTR(-EINVAL);
+	return __mpol_bind_node(nid, MPOL_F_PRIVATE);
+}
+EXPORT_SYMBOL_FOR_MODULES(mpol_private_bind, "kmem");
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=24
author maintainer
#51 lore raw

PATCH v5 25/36 mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug

In reply to Gregory Price

Add add_private_memory_driver_managed() to let modules hotplug memory onto an N_MEMORY_PRIVATE node they control.

Ownership is denoted via struct node_private and a private node can only have a single owner.

The common __add_memory_driver_managed function is only exported to in-tree modules that allow hotplug of N_MEMORY nodes. The new private wrapper does not allow N_MEMORY hotplug, only N_MEMORY_PRIVATE.

When node_private is NULL hotplug behaviour is unchanged.

When node_private is non-NULL the node owner is registered before any memory is added. Later during hotplug, when memory is actually onlined, hotplug marks the node N_MEMORY_PRIVATE instead of N_MEMORY.

If the add fails the registration is undone.

offline_pages() clears N_MEMORY_PRIVATE the same as N_MEMORY, and offline_and_remove_memory_ranges() drops private node registration once the last node's memory is removed.

Private nodes are opted out of starting kswapd/kcompactd by default.

---
 drivers/dax/kmem.c             |  2 +-
 include/linux/memory_hotplug.h |  5 +-
 mm/memory_hotplug.c            | 99 ++++++++++++++++++++++++++++++----
 3 files changed, 94 insertions(+), 12 deletions(-)
drivers/dax/kmem.c +1 -1
index a48d699cf344c..c3faf18e7f85d 100644
@@ -126,7 +126,7 @@ static int dax_kmem_do_hotplug(struct dev_dax *dev_dax,
 		 */
 		rc = __add_memory_driver_managed(data->mgid, range.start,
 				range_len(&range), kmem_name, mhp_flags,
-				online_type);
+				online_type, NULL);
 
 		if (rc) {
 			dev_warn(dev, "mapping%d: %#llx-%#llx memory add failed\n",
include/linux/memory_hotplug.h +4 -1
index b39605d308963..fcceb0a423579 100644
@@ -305,7 +305,10 @@ extern int add_memory_resource(int nid, struct resource *resource,
 			       mhp_t mhp_flags);
 int __add_memory_driver_managed(int nid, u64 start, u64 size,
 				const char *resource_name, mhp_t mhp_flags,
-				enum mmop online_type);
+				enum mmop online_type, struct node_private *np);
+int add_private_memory_driver_managed(int nid, u64 start, u64 size,
+				      const char *resource_name, mhp_t mhp_flags,
+				      enum mmop online_type, struct node_private *np);
 extern int add_memory_driver_managed(int nid, u64 start, u64 size,
 				     const char *resource_name,
 				     mhp_t mhp_flags);
mm/memory_hotplug.c +89 -10

198 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#52 lore raw

Re: [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider: - [High] Private node registration and unregistration incorrectly rely on N_MEMORY and N_MEMORY_PRIVATE…

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] Private node registration and unregistration incorrectly rely on N_MEMORY and N_MEMORY_PRIVATE states, which only track online memory, causing isolation breakage and hijacking of normal memory nodes.
- [High] A TOCTOU race condition exists between memory unplug and private node registration due to lock ordering.
- [Medium] Direct access of __rcu pointer in pgdat_is_private() without using rcu_access_pointer().
--

--- Patch [25]: [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug --- commit 976d363a06939b47fdc2d106fc8f3ec46359abd3
Author: Gregory Price <[email protected]>
Subject: mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug

This commit introduces add_private_memory_driver_managed() to allow modules to hotplug memory into an N_MEMORY_PRIVATE node. It introduces logic to register node ownership and opts these private nodes out of reclaim/compaction by default.

Quoted 4 lines
mm/memory_hotplug.c +0 -0
index 2b9c0830821e6..be230ac9efe5a 100644

[ ... ]

mm/memory_hotplug.c int online_pages(unsigned long pfn, unsigned long nr_pages, · line 1213

Quoted 12 lines
mm/memory_hotplug.c +6 -1
@@ -1204,13 +1204,20 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 	online_pages_range(pfn, nr_pages);
 	adjust_present_page_count(pfn_to_page(pfn), group, nr_pages);
 
+	/*
+	 * N_MEMORY and N_MEMORY_PRIVATE are mutually exclusive, determine
+	 * which is correct based on whether the pgdat->private is set.
+	 */
 	if (node_arg.nid >= 0)
-		node_set_state(nid, N_MEMORY);
+		node_set_state(nid, pgdat_is_private(NODE_DATA(nid)) ?
+				    N_MEMORY_PRIVATE : N_MEMORY);

[Severity: Medium]
Does pgdat_is_private() directly evaluate the __rcu annotated pgdat->node_private pointer without using rcu_access_pointer()? A direct boolean evaluation of an __rcu pointer can result in compiler load tearing.

[ ... ]

Quoted 8 lines
mm/memory_hotplug.c +4 -0
@@ -1688,6 +1702,19 @@ int __add_memory_driver_managed(int nid, u64 start, u64 size,
 	if (online_type < MMOP_OFFLINE || online_type > MMOP_ONLINE_MOVABLE)
 		return -EINVAL;
 
+	/* Register a private-node owner before adding memory. */
+	if (np) {
+		if (mhp_flags & MHP_NID_IS_MGID) {
+			group = memory_group_find_by_id(nid);

mm/memory_hotplug.c int __add_memory_driver_managed(int nid, u64 start, u64 size, · line 1718

Quoted 10 lines
mm/memory_hotplug.c +9 -0
+			if (!group)
+				return -EINVAL;
+			real_nid = group->nid;
+		}
+		rc = node_private_register(real_nid, np);
+		if (rc)
+			return rc;
+	}
+
 	lock_device_hotplug();

[Severity: High]
Is there a race condition here since node_private_register() is called before acquiring device_hotplug_lock?

If Thread A drops the lock in offline_and_remove_memory_ranges() before calling node_private_unregister(), Thread B could successfully register the node here. Thread B then blocks on device_hotplug_lock, allowing Thread A to clear the registration, which might cause Thread B to online its newly added private memory incorrectly as normal system RAM.

[ ... ]

mm/memory_hotplug.c int __add_memory_driver_managed(int nid, u64 start, u64 size, · line 1735

Quoted 8 lines
mm/memory_hotplug.c +2 -0
@@ -1702,10 +1729,41 @@ int __add_memory_driver_managed(int nid, u64 start, u64 size,
 
 out_unlock:
 	unlock_device_hotplug();
+	if (rc < 0 && np)
+		node_private_unregister(real_nid);
 	return rc;
 }

[Severity: High]
If this error path is hit, does node_private_unregister() unconditionally drop the private node registration based only on the online state, ignoring offline memory blocks on the same node? This could break previously added offline blocks on this node.

[ ... ]

mm/memory_hotplug.c int offline_and_remove_memory_ranges(const struct range *ranges, · line 2619

Quoted 11 lines
mm/memory_hotplug.c +4 -0
@@ -2535,6 +2610,10 @@ int offline_and_remove_memory_ranges(const struct range *ranges,
 out_unlock:
 	unlock_device_hotplug();
 
+	/* Drop private node registration if they are now memoryless */
+	for_each_node_mask(nid, nodes)
+		node_private_unregister(nid);
+
 	kfree(online_types);
 	return rc;
 }

[Severity: High]
Could node_private_unregister() blindly clear pgdat->node_private here if it only checks the online N_MEMORY state instead of present memory? If remaining memblocks on the node are offline, they might become normal system RAM when later onlined.

Additionally, because device_hotplug_lock is dropped before unregistering the private nodes, does this open a race window where a concurrent add_private_memory_driver_managed() could have its state wiped out?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=25
#53 lore raw

Re: [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug

In reply to Gregory Price

Hi Gregory,

Quoted 2 lines · Gregory Price

Add add_private_memory_driver_managed() to let modules hotplug memory onto an N_MEMORY_PRIVATE node they control.

I am trying to understand how the nid passed to add_private_memory_driver_managed() is expected to be provisioned.

From what I can see, both node_private_register() and __add_memory_resource() require the nid to already be present in node_possible_map. My understanding is therefore that this patch supports hotplugging private memory onto an existing possible node, rather than creating a new possible node ID at runtime.

For example, in the use case I am looking at, the device's private node is not described by firmware. Would its nid need to be reserved during early NUMA initialization, or is there another mechanism that I may have missed?

Also, could you please Cc me on future revisions of this series? I would appreciate it and would be happy to follow the work:)

Thanks,
Qiqi

author maintainer
#54 lore raw

Re: [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug

Quoted 16 lines · Qiqi Li

Hi Gregory,

Quoted 2 lines · Gregory Price

Add add_private_memory_driver_managed() to let modules hotplug memory onto an N_MEMORY_PRIVATE node they control.

I am trying to understand how the nid passed to add_private_memory_driver_managed() is expected to be provisioned.

From what I can see, both node_private_register() and __add_memory_resource() require the nid to already be present in node_possible_map. My understanding is therefore that this patch supports hotplugging private memory onto an existing possible node, rather than creating a new possible node ID at runtime.

Quoted 5 lines

For example, in the use case I am looking at, the device's private node is not described by firmware. Would its nid need to be reserved during early NUMA initialization, or is there another mechanism that I may have missed?

It would need to be reserved, and you could use numa emulation but that's clunky. I have an RFC for exactly this issue - though i haven't pushed it further yet.

https://lore.kernel.org/all/[email protected]/

Quoted 5 lines

Also, could you please Cc me on future revisions of this series? I would appreciate it and would be happy to follow the work:)

Thanks,
Qiqi

#55 lore raw

Re: [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug

Quoted 5 lines · Gregory Price

It would need to be reserved, and you could use numa emulation but that's clunky. I have an RFC for exactly this issue - though i haven't pushed it further yet.

https://lore.kernel.org/all/[email protected]/

Hi Gregory,

Sorry for the late reply.

I took a look at the exclusive-node RFC. The approach looks useful for device-backed private memory without a firmware-described NUMA node. Thanks for the pointer.

We are currently looking into the private-node and CRAM design as a whole. I noticed that CRAM was not included with the v5 series, and saw that others were also interested in a v5 version.

If you already have a CRAM branch or patches based on v5, even if they are still work in progress, would you mind sharing them? It would be very helpful for us to continue testing the overall design.

Thanks,
Qiqi

author maintainer
#56 lore raw

Re: [PATCH v5 25/36] mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug

Quoted 15 lines · Qiqi Li
Quoted 5 lines · Gregory Price

It would need to be reserved, and you could use numa emulation but that's clunky. I have an RFC for exactly this issue - though i haven't pushed it further yet.

https://lore.kernel.org/all/[email protected]/

Hi Gregory,

Sorry for the late reply.

I took a look at the exclusive-node RFC. The approach looks useful for device-backed private memory without a firmware-described NUMA node. Thanks for the pointer.

Quoted 8 lines

We are currently looking into the private-node and CRAM design as a whole. I noticed that CRAM was not included with the v5 series, and saw that others were also interested in a v5 version.

If you already have a CRAM branch or patches based on v5, even if they are still work in progress, would you mind sharing them? It would be very helpful for us to continue testing the overall design.

I dropped it off because it was starting to become unwieldly, but i am continuing to maintain a branch on my github:

https://github.com/gourryinverse/linux/commits/scratch/gourry/cramtest/cram_72/

I have not posted a new version in some time and it has been reworked significantly in the past few months - but I plan to talk about this particular branch and the tests I've run with it at the device memory LPC micro-conference.

I validated this branch against most in-tree (relevant) filesystems for the page-cache extension, but I imagine the anon-only version subset is the more immediately interesting for upstream development.

I may include the anon-only version in v6, but I've been a bit busy shoring up the base private memory node core and implementing kernel selftests and LTP tests to ensure there's no userland api regressions.

I hope to get something out later this month.

~Gregory

author maintainer
#57 lore raw

PATCH v5 26/36 mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim

In reply to Gregory Price

Provide a mechanism to opt private nodes into the reclaim process.

Reclaim as a "singular service" is actually made up of:

- kswapd reclaim
- direct reclaim
- kcompactd compaction
- direct compaction
- both mglru / lru paths
- madvise reclaim hints
- damon reclaim operations

CAP_RECLAIM gates whether the kernel may do inter-node placement (compaction) or swap for the folios on that private node.

node_allows_reclaim() encapsulates the policy as a whole. Ordinary nodes are always reclaimable, private nodes only when opted in.

With the exception of madvise and DAMON, reclaim operations are highly integrated with one another, so they are opted in/out of together - otherwise reclaim becomes unpredictable.

To prevent bisect debugging failures, this stays in a single commit.

For example: reclaim without compaction will OOM on high-order allocation failure despite a large amount of free memory. Normally compaction would (potentially) resolve this issue.

If a private node opts into reclaim, we create normal watermarks for that node - otherwise pgdat_balanced() is always true and reclaim thinks there is no work to do.

Cross-node operations (demotion, promotion, khugepaged) are NOT included in CAP_RECLAIM because some workflows may desire different behaviors for this kind of operation:

- prefer direct-to-swap, do not demote
- remain resident and OOM
- __GFP_THISNODE: fail and let the driver augment reclaim

Normal swap-out is allowed because there are clear userland controls (not registering swap, cgroup.swap, etc) to control that per-workload.

---
 include/linux/node_private.h | 33 +++++++++++++++++++++++++
 mm/compaction.c              |  9 ++++---
 mm/damon/paddr.c             |  4 +--
 mm/huge_memory.c             |  2 +-
 mm/internal.h                | 13 ++++++++++
 mm/madvise.c                 |  6 ++---
 mm/memory_hotplug.c          |  3 +--
 mm/page_alloc.c              |  2 +-
 mm/vmscan.c                  | 48 ++++++++++++++++++++++++++++++------
 9 files changed, 100 insertions(+), 20 deletions(-)
include/linux/node_private.h +33 -0
index 475496c84249f..f7cbae1309904 100644
@@ -7,6 +7,13 @@
 
 struct page;
 
+/*
+ * Per-node service opt-ins (node_private.caps).  A private node is isolated
+ * from all general mm services by default; the registering driver sets these
+ * to let specific services operate on its node.
+ */
+#define NODE_PRIVATE_CAP_RECLAIM	(1UL << 0)	/* allow mm reclaim */
+
 /**
  * struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
  *
@@ -41,6 +48,27 @@ static inline bool node_is_private(int nid)
 	return node_state(nid, N_MEMORY_PRIVATE);
 }
 
+/**
+ * node_allows_reclaim - may the mm reclaim from this node?
+ * @nid: the node to test
+ *
+ * Only a private node is ever excluded.  Every other node can safely
+ * be operated on by reclaim.
+ */
+static inline bool node_allows_reclaim(int nid)
+{
+	struct node_private *np;
+	bool ret;
+
+	if (!node_state(nid, N_MEMORY_PRIVATE))
+		return true;
+	rcu_read_lock();
+	np = rcu_dereference(NODE_DATA(nid)->node_private);
+	ret = np && (np->caps & NODE_PRIVATE_CAP_RECLAIM);
+	rcu_read_unlock();
+	return ret;
+}
+
 #else /* !CONFIG_NUMA */
 
 static inline bool folio_is_private_node(struct folio *folio)
@@ -58,6 +86,11 @@ static inline bool node_is_private(int nid)
 	return false;
 }
 
+static inline bool node_allows_reclaim(int nid)
+{
+	return true;
+}
+
 #endif /* CONFIG_NUMA */
 
 #if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
mm/compaction.c +5 -4
index 8c1351cce7bcc..b9c25c599732f 100644
@@ -25,6 +25,7 @@
 #include <linux/psi.h>
 #include <linux/cpuset.h>
 #include "page_alloc.h"
+#include <linux/node_private.h>
 #include "internal.h"
 
 #ifdef CONFIG_COMPACTION
@@ -2462,7 +2463,7 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
 		    !__cpuset_zone_allowed(zone, gfp_mask))
 			continue;
 
-		if (node_is_private(zone_to_nid(zone)))
+		if (!node_allows_reclaim(zone_to_nid(zone)))
 			continue;
 
 		/*
@@ -2856,7 +2857,7 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
 			!__cpuset_zone_allowed(zone, gfp_mask))
 				continue;
 
-		if (node_is_private(zone_to_nid(zone)))
+		if (!node_allows_reclaim(zone_to_nid(zone)))
 			continue;
 
 		if (prio > MIN_COMPACT_PRIORITY
@@ -2928,7 +2929,7 @@ static int compact_node(pg_data_t *pgdat, bool proactive)
 		.proactive_compaction = proactive,
 	};
 
-	if (node_is_private(pgdat->node_id))
+	if (!node_allows_reclaim(pgdat->node_id))
 		return 0;
 
 	for (zoneid = 0; zoneid < MAX_NR_ZONES; zoneid++) {
@@ -3026,7 +3027,7 @@ static ssize_t compact_store(struct device *dev,
 {
 	int nid = dev->id;
 
-	if (node_is_private(nid))
+	if (!node_allows_reclaim(nid))
 		return -EINVAL;
 
 	if (nid >= 0 && nid < nr_node_ids && node_online(nid)) {
mm/damon/paddr.c +2 -2
index c741a94319750..b668cf55d1b67 100644
@@ -251,8 +251,8 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
 			continue;
 		}
 
-		/* private node memory is not reclaimable by default */
-		if (folio_is_private_node(folio))
+		/* DAMOS pageout is reclaim; gate a private node on CAP_RECLAIM */
+		if (!node_allows_reclaim(folio_nid(folio)))
 			goto put_folio;
 
 		if (damos_pa_filter_out(s, folio))
mm/huge_memory.c +1 -1
index 1df91b4e5c2bc..cebc89eb0541f 100644
@@ -2340,7 +2340,7 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 
 	folio = pmd_folio(orig_pmd);
 
-	if (folio_is_private_node(folio))
+	if (!node_allows_reclaim(folio_nid(folio)))
 		goto out;
 
 	/*
mm/internal.h +13 -0
index 85c460296cea1..8329034ae561f 100644
@@ -110,6 +110,19 @@ static inline bool page_is_private_managed(struct page *page)
 	return folio_is_private_managed(page_folio(page));
 }
 
+/*
+ * folio_allows_madvise() - may madvise reclaim hints act on this folio?
+ *
+ * madvise reclaim hints (COLD/PAGEOUT/FREE) are userland-driven reclaim, so
+ * they follow reclaim opt-in: false for ZONE_DEVICE and for N_MEMORY_PRIVATE
+ * nodes without CAP_RECLAIM, true for all other normal folios.
+ */
+static inline bool folio_allows_madvise(struct folio *folio)
+{
+	return !folio_is_zone_device(folio) &&
+	       node_allows_reclaim(folio_nid(folio));
+}
+
 /*
  * folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
  *
mm/madvise.c +3 -3
index 29f35a23919a0..56ca974542707 100644
@@ -396,7 +396,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 
 		folio = pmd_folio(orig_pmd);
 
-		if (folio_is_private_node(folio))
+		if (!node_allows_reclaim(folio_nid(folio)))
 			goto huge_unlock;
 
 		/* Do not interfere with other mappings of this folio */
@@ -478,7 +478,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 			continue;
 
 		folio = vm_normal_folio(vma, addr, ptent);
-		if (!folio || folio_is_private_managed(folio))
+		if (!folio || !folio_allows_madvise(folio))
 			continue;
 
 		/*
@@ -707,7 +707,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
 		}
 
 		folio = vm_normal_folio(vma, addr, ptent);
-		if (!folio || folio_is_private_managed(folio))
+		if (!folio || !folio_allows_madvise(folio))
 			continue;
 
 		/*
mm/memory_hotplug.c +1 -2
index be230ac9efe5a..1f42ed303366c 100644
@@ -1237,8 +1237,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 	/* reinitialise watermarks and update pcp limits */
 	init_per_zone_wmark_min();
 
-	/* Private nodes opt-out of reclaim/compaction by default */
-	if (!node_is_private(nid)) {
+	if (node_allows_reclaim(nid)) {
 		kswapd_run(nid);
 		kcompactd_run(nid);
 	}
mm/page_alloc.c +1 -1
index 2b08bea2379a9..2667a4564b7ac 100644
@@ -6667,7 +6667,7 @@ static void __setup_per_zone_wmarks(void)
 		u64 tmp;
 
 		spin_lock_irqsave(&zone->lock, flags);
-		if (node_is_private(zone_to_nid(zone))) {
+		if (!node_allows_reclaim(zone_to_nid(zone))) {
 			zone->_watermark[WMARK_MIN] = 0;
 			zone->_watermark[WMARK_LOW] = 0;
 			zone->_watermark[WMARK_HIGH] = 0;
mm/vmscan.c +41 -7

109 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#58 lore raw

Re: [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] calculate_totalreserve_pages() unconditionally skips private nodes, failing to account for their…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] calculate_totalreserve_pages() unconditionally skips private nodes, failing to account for their watermarks in totalreserve_pages.
- [Medium] Incorrect zone watermark calculation for private nodes allowing reclaim.
--

--- Patch [26]: [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim --- commit 29ab758bf14800171bcfa7488632ce55ed680b45
Author: Gregory Price <[email protected]>
Subject: mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim

This commit introduces NODE_PRIVATE_CAP_RECLAIM to allow private memory nodes to opt into reclaim processes. It replaces various node_is_private() checks with node_allows_reclaim() across the memory management subsystem. Private nodes that opt in will now have normal watermarks calculated for them.

mm/page_alloc.c static void __setup_per_zone_wmarks(void) · line 6673

Quoted 13 lines
mm/page_alloc.c +1 -1
index 2b08bea2379a9..2667a4564b7ac 100644
@@ -6667,7 +6667,7 @@ static void __setup_per_zone_wmarks(void)
 		u64 tmp;
 
 		spin_lock_irqsave(&zone->lock, flags);
-		if (node_is_private(zone_to_nid(zone))) {
+		if (!node_allows_reclaim(zone_to_nid(zone))) {
 			zone->_watermark[WMARK_MIN] = 0;
 			zone->_watermark[WMARK_LOW] = 0;
 			zone->_watermark[WMARK_HIGH] = 0;

[Severity: Medium]
Will this lead to an incorrect watermark calculation for private nodes that allow reclaim?

Earlier in __setup_per_zone_wmarks(), lowmem_pages is calculated by explicitly excluding private memory:

	for_each_zone(zone) {
		if (!is_highmem(zone) && zone_idx(zone) != ZONE_MOVABLE &&
		    !node_is_private(zone_to_nid(zone)))
			lowmem_pages += zone_managed_pages(zone);
	}

Because lowmem_pages excludes private memory, when we calculate WMARK_MIN for a private node later in this function using:

		tmp = (u64)pages_min * zone_managed_pages(zone);
		tmp = div64_ul(tmp, lowmem_pages);

does this mean the private node will reserve a disproportionately high percentage of its memory if the system has a small amount of normal memory and a large private node?

[Severity: Medium]
Also, does this miss updating the totalreserve_pages counter?

At the end of __setup_per_zone_wmarks(), we call calculate_totalreserve_pages(). However, that function unconditionally skips all private nodes:

	for_each_online_pgdat(pgdat) {
		/* private nodes have zero watermarks */
		if (node_is_private(pgdat->node_id))
			continue;

If a private node has CAP_RECLAIM and non-zero watermarks, does skipping it here cause totalreserve_pages to remain artificially low?

This might cause node_dirtyable_memory() and global_dirtyable_memory() to overestimate available memory, which could inflate dirty limits and writeback thresholds.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=26
#59 lore raw

Re: [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim

In reply to Gregory Price
Quoted 369 lines · Gregory Price

Provide a mechanism to opt private nodes into the reclaim process.

Reclaim as a "singular service" is actually made up of:

- kswapd reclaim
- direct reclaim
- kcompactd compaction
- direct compaction
- both mglru / lru paths
- madvise reclaim hints
- damon reclaim operations

CAP_RECLAIM gates whether the kernel may do inter-node placement (compaction) or swap for the folios on that private node.

node_allows_reclaim() encapsulates the policy as a whole. Ordinary nodes are always reclaimable, private nodes only when opted in.

With the exception of madvise and DAMON, reclaim operations are highly integrated with one another, so they are opted in/out of together - otherwise reclaim becomes unpredictable.

To prevent bisect debugging failures, this stays in a single commit.

For example: reclaim without compaction will OOM on high-order allocation failure despite a large amount of free memory. Normally compaction would (potentially) resolve this issue.

If a private node opts into reclaim, we create normal watermarks for that node - otherwise pgdat_balanced() is always true and reclaim thinks there is no work to do.

Cross-node operations (demotion, promotion, khugepaged) are NOT included in CAP_RECLAIM because some workflows may desire different behaviors for this kind of operation:

- prefer direct-to-swap, do not demote
- remain resident and OOM
- __GFP_THISNODE: fail and let the driver augment reclaim

Normal swap-out is allowed because there are clear userland controls (not registering swap, cgroup.swap, etc) to control that per-workload.

---
 include/linux/node_private.h | 33 +++++++++++++++++++++++++
 mm/compaction.c              |  9 ++++---
 mm/damon/paddr.c             |  4 +--
 mm/huge_memory.c             |  2 +-
 mm/internal.h                | 13 ++++++++++
 mm/madvise.c                 |  6 ++---
 mm/memory_hotplug.c          |  3 +--
 mm/page_alloc.c              |  2 +-
 mm/vmscan.c                  | 48 ++++++++++++++++++++++++++++++------
 9 files changed, 100 insertions(+), 20 deletions(-)
include/linux/node_private.h +33 -0
index 475496c84249f..f7cbae1309904 100644
@@ -7,6 +7,13 @@
 
 struct page;
 
+/*
+ * Per-node service opt-ins (node_private.caps).  A private node is isolated
+ * from all general mm services by default; the registering driver sets these
+ * to let specific services operate on its node.
+ */
+#define NODE_PRIVATE_CAP_RECLAIM	(1UL << 0)	/* allow mm reclaim */
+
 /**
  * struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
  *
@@ -41,6 +48,27 @@ static inline bool node_is_private(int nid)
 	return node_state(nid, N_MEMORY_PRIVATE);
 }
 
+/**
+ * node_allows_reclaim - may the mm reclaim from this node?
+ * @nid: the node to test
+ *
+ * Only a private node is ever excluded.  Every other node can safely
+ * be operated on by reclaim.
+ */
+static inline bool node_allows_reclaim(int nid)
+{
+	struct node_private *np;
+	bool ret;
+
+	if (!node_state(nid, N_MEMORY_PRIVATE))
+		return true;
+	rcu_read_lock();
+	np = rcu_dereference(NODE_DATA(nid)->node_private);
+	ret = np && (np->caps & NODE_PRIVATE_CAP_RECLAIM);
+	rcu_read_unlock();
+	return ret;
+}
+
 #else /* !CONFIG_NUMA */
 
 static inline bool folio_is_private_node(struct folio *folio)
@@ -58,6 +86,11 @@ static inline bool node_is_private(int nid)
 	return false;
 }
 
+static inline bool node_allows_reclaim(int nid)
+{
+	return true;
+}
+
 #endif /* CONFIG_NUMA */
 
 #if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
mm/compaction.c +5 -4
index 8c1351cce7bcc..b9c25c599732f 100644
@@ -25,6 +25,7 @@
 #include <linux/psi.h>
 #include <linux/cpuset.h>
 #include "page_alloc.h"
+#include <linux/node_private.h>
 #include "internal.h"
 
 #ifdef CONFIG_COMPACTION
@@ -2462,7 +2463,7 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
 		    !__cpuset_zone_allowed(zone, gfp_mask))
 			continue;
 
-		if (node_is_private(zone_to_nid(zone)))
+		if (!node_allows_reclaim(zone_to_nid(zone)))
 			continue;
 
 		/*
@@ -2856,7 +2857,7 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
 			!__cpuset_zone_allowed(zone, gfp_mask))
 				continue;
 
-		if (node_is_private(zone_to_nid(zone)))
+		if (!node_allows_reclaim(zone_to_nid(zone)))
 			continue;
 
 		if (prio > MIN_COMPACT_PRIORITY
@@ -2928,7 +2929,7 @@ static int compact_node(pg_data_t *pgdat, bool proactive)
 		.proactive_compaction = proactive,
 	};
 
-	if (node_is_private(pgdat->node_id))
+	if (!node_allows_reclaim(pgdat->node_id))
 		return 0;
 
 	for (zoneid = 0; zoneid < MAX_NR_ZONES; zoneid++) {
@@ -3026,7 +3027,7 @@ static ssize_t compact_store(struct device *dev,
 {
 	int nid = dev->id;
 
-	if (node_is_private(nid))
+	if (!node_allows_reclaim(nid))
 		return -EINVAL;
 
 	if (nid >= 0 && nid < nr_node_ids && node_online(nid)) {
mm/damon/paddr.c +2 -2
index c741a94319750..b668cf55d1b67 100644
@@ -251,8 +251,8 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
 			continue;
 		}
 
-		/* private node memory is not reclaimable by default */
-		if (folio_is_private_node(folio))
+		/* DAMOS pageout is reclaim; gate a private node on CAP_RECLAIM */
+		if (!node_allows_reclaim(folio_nid(folio)))
 			goto put_folio;
 
 		if (damos_pa_filter_out(s, folio))
mm/huge_memory.c +1 -1
index 1df91b4e5c2bc..cebc89eb0541f 100644
@@ -2340,7 +2340,7 @@ bool madvise_free_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
 
 	folio = pmd_folio(orig_pmd);
 
-	if (folio_is_private_node(folio))
+	if (!node_allows_reclaim(folio_nid(folio)))
 		goto out;
 
 	/*
mm/internal.h +13 -0
index 85c460296cea1..8329034ae561f 100644
@@ -110,6 +110,19 @@ static inline bool page_is_private_managed(struct page *page)
 	return folio_is_private_managed(page_folio(page));
 }
 
+/*
+ * folio_allows_madvise() - may madvise reclaim hints act on this folio?
+ *
+ * madvise reclaim hints (COLD/PAGEOUT/FREE) are userland-driven reclaim, so
+ * they follow reclaim opt-in: false for ZONE_DEVICE and for N_MEMORY_PRIVATE
+ * nodes without CAP_RECLAIM, true for all other normal folios.
+ */
+static inline bool folio_allows_madvise(struct folio *folio)
+{
+	return !folio_is_zone_device(folio) &&
+	       node_allows_reclaim(folio_nid(folio));
+}
+
 /*
  * folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
  *
mm/madvise.c +3 -3
index 29f35a23919a0..56ca974542707 100644
@@ -396,7 +396,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 
 		folio = pmd_folio(orig_pmd);
 
-		if (folio_is_private_node(folio))
+		if (!node_allows_reclaim(folio_nid(folio)))
 			goto huge_unlock;
 
 		/* Do not interfere with other mappings of this folio */
@@ -478,7 +478,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
 			continue;
 
 		folio = vm_normal_folio(vma, addr, ptent);
-		if (!folio || folio_is_private_managed(folio))
+		if (!folio || !folio_allows_madvise(folio))
 			continue;
 
 		/*
@@ -707,7 +707,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,
 		}
 
 		folio = vm_normal_folio(vma, addr, ptent);
-		if (!folio || folio_is_private_managed(folio))
+		if (!folio || !folio_allows_madvise(folio))
 			continue;
 
 		/*
mm/memory_hotplug.c +1 -2
index be230ac9efe5a..1f42ed303366c 100644
@@ -1237,8 +1237,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 	/* reinitialise watermarks and update pcp limits */
 	init_per_zone_wmark_min();
 
-	/* Private nodes opt-out of reclaim/compaction by default */
-	if (!node_is_private(nid)) {
+	if (node_allows_reclaim(nid)) {
 		kswapd_run(nid);
 		kcompactd_run(nid);
 	}
mm/page_alloc.c +1 -1
index 2b08bea2379a9..2667a4564b7ac 100644
@@ -6667,7 +6667,7 @@ static void __setup_per_zone_wmarks(void)
 		u64 tmp;
 
 		spin_lock_irqsave(&zone->lock, flags);
-		if (node_is_private(zone_to_nid(zone))) {
+		if (!node_allows_reclaim(zone_to_nid(zone))) {
 			zone->_watermark[WMARK_MIN] = 0;
 			zone->_watermark[WMARK_LOW] = 0;
 			zone->_watermark[WMARK_HIGH] = 0;
mm/vmscan.c +29 -6
index 86b2334c23b98..f1722693ac2db 100644
@@ -5396,6 +5396,21 @@ static const struct attribute_group lru_gen_attr_group = {
  *                          debugfs interface
  ******************************************************************************/
 
+/*
+ * Nodes the lru_gen debugfs interface lists: ordinary memory nodes plus any
+ * N_MEMORY_PRIVATE nodes opted into reclaim.  run_cmd() already accepts the
+ * latter, so keep the listing in sync with what it accepts.
+ */
+static void lru_gen_seq_nodes(nodemask_t *nodes)
+{
+	int nid;
+
+	*nodes = node_states[N_MEMORY];
+	for_each_node_state(nid, N_MEMORY_PRIVATE)
+		if (node_allows_reclaim(nid))
+			node_set(nid, *nodes);
+}
+
 static void *lru_gen_seq_start(struct seq_file *m, loff_t *pos)
 {
 	struct mem_cgroup *memcg;
@@ -5407,9 +5422,11 @@ static void *lru_gen_seq_start(struct seq_file *m, loff_t *pos)
 
 	memcg = mem_cgroup_iter(NULL, NULL, NULL);
 	do {
+		nodemask_t nodes;
 		int nid;
 
-		for_each_node_state(nid, N_MEMORY) {
+		lru_gen_seq_nodes(&nodes);
+		for_each_node_mask(nid, nodes) {
 			if (!nr_to_skip--)
 				return get_lruvec(memcg, nid);
 		}
@@ -5431,16 +5448,18 @@ static void *lru_gen_seq_next(struct seq_file *m, void *v, loff_t *pos)
 {
 	int nid = lruvec_pgdat(v)->node_id;
 	struct mem_cgroup *memcg = lruvec_memcg(v);
+	nodemask_t nodes;
 
 	++*pos;
 
-	nid = next_memory_node(nid);
+	lru_gen_seq_nodes(&nodes);
+	nid = next_node(nid, nodes);
 	if (nid == MAX_NUMNODES) {
 		memcg = mem_cgroup_iter(NULL, memcg, NULL);
 		if (!memcg)
 			return NULL;
 
-		nid = first_memory_node;
+		nid = first_node(nodes);
 	}
 
 	return get_lruvec(memcg, nid);
@@ -5509,10 +5528,12 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)
 	struct lru_gen_folio *lrugen = &lruvec->lrugen;
 	int nid = lruvec_pgdat(lruvec)->node_id;
 	struct mem_cgroup *memcg = lruvec_memcg(lruvec);
+	nodemask_t nodes;
 	DEFINE_MAX_SEQ(lruvec);
 	DEFINE_MIN_SEQ(lruvec);
 
-	if (nid == first_memory_node) {
+	lru_gen_seq_nodes(&nodes);
+	if (nid == first_node(nodes)) {
 		const char *path = memcg ? m->private : "";
 
 #ifdef CONFIG_MEMCG
@@ -5612,7 +5633,9 @@ static int run_cmd(char cmd, u64 memcg_id, int nid, unsigned long seq,
 	int err = -EINVAL;
 	struct mem_cgroup *memcg = NULL;
 
-	if (nid < 0 || nid >= MAX_NUMNODES || !node_state(nid, N_MEMORY))
+	if (nid < 0 || nid >= MAX_NUMNODES ||
+	    !(node_state(nid, N_MEMORY) ||
+	      (node_is_private(nid) && node_allows_reclaim(nid))))
 		return -EINVAL;
 
 	if (!mem_cgroup_disabled()) {
@@ -6145,7 +6168,7 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc)
 	 * Private nodes do not support reclaim by default, filtering here
 	 * captures all normal reclaim paths that may attempt eviction.
 	 */
-	if (node_is_private(pgdat->node_id))
+	if (!node_allows_reclaim(pgdat->node_id))
 		return;
 
 	if ((lru_gen_enabled() || lru_gen_switching()) && root_reclaim(sc)) {
@@ -6758,6 +6781,16 @@ unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg,
 	return sc.nr_reclaimed;
 }

mm/vmscan.c unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg, · line 6792

Quoted 10 lines
mm/vmscan.c +9 -0
 
+static struct zonelist *memcg_reclaim_zonelist(int nid, gfp_t gfp_mask)
+{
+	unsigned int aflags = ALLOC_DEFAULT;
+
+	if (unlikely(!nodes_empty(node_states[N_MEMORY_PRIVATE])))
+		aflags = ALLOC_ZONELIST_PRIVATE;
+
+	return select_zonelist(nid, gfp_mask, aflags);
+}

I applied this code change and notice in mem_cgroup_shrink_node(), it calls shrink_lruvec() directly and never passes through shrink_node().

This patch makes private nodes reachable through memcg reclaim zonelist, but what prevents the direct per-node memcg path from reclaiming a private node that has not set NODE_PRIVATE_CAP_RECLAIM ?

--Richard

Quoted 18 lines
mm/vmscan.c +3 -1
+
 unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
 					   unsigned long nr_pages,
 					   gfp_t gfp_mask,
@@ -6784,7 +6817,8 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
 	 * equal pressure on all the nodes. This is based on the assumption that
 	 * the reclaim does not bail out early.
 	 */
-	struct zonelist *zonelist = node_zonelist(numa_node_id(), sc.gfp_mask);
+	struct zonelist *zonelist = memcg_reclaim_zonelist(numa_node_id(),
+							   sc.gfp_mask);
 
 	set_task_reclaim_state(current, &sc.reclaim_state);
 	trace_mm_vmscan_memcg_reclaim_begin(sc.gfp_mask, 0, memcg);
Signature · 1 line
2.53.0-Meta
author maintainer
#60 lore raw

Re: [PATCH v5 26/36] mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim

Quoted 15 lines · Richard Cheng

mm/vmscan.c unsigned long mem_cgroup_shrink_node(struct mem_cgroup *memcg, · line 6792

Quoted 9 lines · Gregory Price
mm/vmscan.c +9 -0
+static struct zonelist *memcg_reclaim_zonelist(int nid, gfp_t gfp_mask)
+{
+	unsigned int aflags = ALLOC_DEFAULT;
+
+	if (unlikely(!nodes_empty(node_states[N_MEMORY_PRIVATE])))
+		aflags = ALLOC_ZONELIST_PRIVATE;
+
+	return select_zonelist(nid, gfp_mask, aflags);
+}

I applied this code change and notice in mem_cgroup_shrink_node(), it calls shrink_lruvec() directly and never passes through shrink_node().

Quoted 4 lines

This patch makes private nodes reachable through memcg reclaim zonelist, but what prevents the direct per-node memcg path from reclaiming a private node that has not set NODE_PRIVATE_CAP_RECLAIM ?

I legitimately missed this.

Sashiko also picked it up on the filter patch:
https://sashiko.dev/#/message/20260721183528.B4B9C1F00A3A%40smtp.kernel.org

If nothing else - implementing this feature has taught me some sub-components of mm/ have very well centralized logic, and others (like vmscan.c) have 3 or 4 different paths in with a lot of strange quirks carved out.

I think some of the vmscan paths can be improved to simplify this interaction, but i will fix this particular miss in v6 and come back around to see whether we can clean up these different paths as a general improvement.

~Gregory

author maintainer
#61 lore raw

PATCH v5 27/36 mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls

In reply to Gregory Price

Provide a mechanism to opt private nodes into userland numa management.

Add node_allows_user_numa() to encapsulate whether a node supports userland NUMA controls (always true for normal nodes).

Placement - setting mempolicy via:
- mbind()
- set_mempolicy()
- set_mempolicy_home_node()

For mempolicy, enforcement lives in one place: mpol_set_nodemask()

Private nodes are not N_MEMORY, so they are trimmed from a nodemask like a cpuset-trimmed node. All-private nodemasks without a valid node collapse to empty and the mempolicy fails.

home_node is not special-cased - it is only a preferred-nid hint, and placement is governed by the bind nodemask, so a home node pointed at an invalid node simply falls back via the normal fallback zonelists.

(All the same behavior as a node/mask not intersecting cpuset.mems).

Migration - relocation of pages to/from a private node:
- mbind(MPOL_MF_MOVE)
- move_pages()
- migrate_pages()

mbind(MPOL_MF_MOVE) is a migration, so mempolicy and migration share a single opt-in control.

The migration interfaces all check node eligibility and use ALLOC_ZONELIST_PRIVATE to allow eligible migration requests to move a folio to a private node..

alloc_migration_target() carries mtc->zlsel into the allocator via __folio_alloc_zonelist().

---
 include/linux/node_private.h | 33 +++++++++++++++++++++++++++++++++
 mm/internal.h                |  1 +
 mm/mempolicy.c               | 27 ++++++++++++++++++++-------
 mm/migrate.c                 | 19 ++++++++++++++-----
 4 files changed, 68 insertions(+), 12 deletions(-)
include/linux/node_private.h +33 -0
index f7cbae1309904..655fe9ec5cb61 100644
@@ -13,6 +13,7 @@ struct page;
  * to let specific services operate on its node.
  */
 #define NODE_PRIVATE_CAP_RECLAIM	(1UL << 0)	/* allow mm reclaim */
+#define NODE_PRIVATE_CAP_USER_NUMA	(1UL << 1)	/* allow mempolicy */
 
 /**
  * struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
@@ -69,6 +70,33 @@ static inline bool node_allows_reclaim(int nid)
 	return ret;
 }
 
+/**
+ * node_allows_user_numa - may userspace place or migrate memory here?
+ * @nid: the node to test
+ *
+ * Gate all userspace-directed memory operations on a private node.
+ *   - mbind()/set_mempolicy()
+ *   - move_pages()/migrate_pages()
+ *
+ * return: true for N_MEMORY and N_MEMORY_PRIVATE with CAP_USER_NUMA.
+ *         false for memoryless or opted-out private node.
+ */
+static inline bool node_allows_user_numa(int nid)
+{
+	struct node_private *np;
+	bool ret;
+
+	if (node_state(nid, N_MEMORY))
+		return true;
+	if (!node_state(nid, N_MEMORY_PRIVATE))
+		return false;
+	rcu_read_lock();
+	np = rcu_dereference(NODE_DATA(nid)->node_private);
+	ret = np && (np->caps & NODE_PRIVATE_CAP_USER_NUMA);
+	rcu_read_unlock();
+	return ret;
+}
+
 #else /* !CONFIG_NUMA */
 
 static inline bool folio_is_private_node(struct folio *folio)
@@ -91,6 +119,11 @@ static inline bool node_allows_reclaim(int nid)
 	return true;
 }
 
+static inline bool node_allows_user_numa(int nid)
+{
+	return true;
+}
+
 #endif /* CONFIG_NUMA */
 
 #if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
mm/internal.h +1 -0
index 8329034ae561f..62e68acae08a3 100644
@@ -1250,6 +1250,7 @@ struct migration_target_control {
 	nodemask_t *nmask;
 	gfp_t gfp_mask;
 	enum migrate_reason reason;
+	unsigned int alloc_flags;
 };
 
 /*
mm/mempolicy.c +20 -7
index a3ffb09897489..fe42a510590a2 100644
@@ -434,13 +434,14 @@ static int mpol_set_nodemask(struct mempolicy *pol,
 
 	/*
 	 * Private nodes are not in cpuset.mems, so they're always stripped.
-	 * Driver-allocated policies will already have MPOL_F_PRIVATE set,
-	 * if that's the case, add back in the requested set of private nodes.
+	 * Driver-allocated policies (MPOL_F_PRIVATE) and CAP_USER_NUMA private
+	 * nodes should be added back into the nodemask.
 	 */
 	for_each_node_mask(nid, *nodes) {
 		if (!node_is_private(nid))
 			continue;
-		if (pol->flags & MPOL_F_PRIVATE)
+		if ((pol->flags & MPOL_F_PRIVATE) ||
+		    node_allows_user_numa(nid))
 			node_set(nid, nsc->mask2);
 	}
 
@@ -696,7 +697,7 @@ static void queue_folios_pmd(pmd_t *pmd, struct mm_walk *walk)
 	}
 	if (!queue_folio_required(folio, qp))
 		return;
-	if (folio_is_private_node(folio))
+	if (!node_allows_user_numa(folio_nid(folio)))
 		return;
 	if (!(qp->flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
 	    !vma_migratable(walk->vma) ||
@@ -752,7 +753,8 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
 			continue;
 		}
 		folio = vm_normal_folio(vma, addr, ptent);
-		if (!folio || folio_is_private_managed(folio))
+		if (!folio || folio_is_zone_device(folio) ||
+		    !node_allows_user_numa(folio_nid(folio)))
 			continue;
 		if (folio_test_large(folio) && max_nr != 1)
 			nr = folio_pte_batch(folio, pte, ptent, max_nr);
@@ -827,7 +829,7 @@ static int queue_folios_hugetlb(pte_t *pte, unsigned long hmask,
 	folio = pfn_folio(pte_pfn(ptep));
 	if (!queue_folio_required(folio, qp))
 		goto unlock;
-	if (folio_is_private_node(folio))
+	if (!node_allows_user_numa(folio_nid(folio)))
 		goto unlock;
 	if (!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) ||
 	    !vma_migratable(walk->vma)) {
@@ -1412,6 +1414,8 @@ static long migrate_to_node(struct mm_struct *mm, int source, int dest,
 		.nid = dest,
 		.gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_THISNODE,
 		.reason = MR_SYSCALL,
+		.alloc_flags = node_is_private(dest) ?
+			 ALLOC_ZONELIST_PRIVATE : ALLOC_DEFAULT,
 	};
 
 	nodes_clear(nmask);
@@ -1985,9 +1989,10 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,
 	struct mm_struct *mm = NULL;
 	struct task_struct *task;
 	nodemask_t task_nodes;
-	int err;
+	nodemask_t priv_ok;
 	nodemask_t *old;
 	nodemask_t *new;
+	int err, nid;
 	NODEMASK_SCRATCH(scratch);
 
 	if (!scratch)
@@ -2027,7 +2032,14 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,
 	}
 	rcu_read_unlock();
 
+	/* Private nodes are stripped by cpuset checks. Allow eligible ones. */
+	nodes_clear(priv_ok);
+	for_each_node_mask(nid, *new)
+		if (node_is_private(nid) && node_allows_user_numa(nid))
+			node_set(nid, priv_ok);
+
 	task_nodes = cpuset_mems_allowed(task);
+	nodes_or(task_nodes, task_nodes, priv_ok);
 	/* Is the user allowed to access the target nodes? */
 	if (!nodes_subset(*new, task_nodes) && !capable(CAP_SYS_NICE)) {
 		err = -EPERM;
@@ -2035,6 +2047,7 @@ static int kernel_migrate_pages(pid_t pid, unsigned long maxnode,
 	}
 
 	task_nodes = cpuset_mems_allowed(current);
+	nodes_or(task_nodes, task_nodes, priv_ok);
 	if (!nodes_and(*new, *new, task_nodes))
 		goto out_put;
 
mm/migrate.c +14 -5
index d20674c07b947..b548d79352a38 100644
@@ -2231,7 +2231,8 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private)
 	if (is_highmem_idx(zidx) || zidx == ZONE_MOVABLE)
 		gfp_mask |= __GFP_HIGHMEM;
 
-	return __folio_alloc(gfp_mask, order, nid, mtc->nmask, ALLOC_DEFAULT);
+	return __folio_alloc(gfp_mask, order, nid, mtc->nmask,
+			     mtc->alloc_flags);
 }
 
 #ifdef CONFIG_NUMA_MIGRATION
@@ -2253,6 +2254,8 @@ static int do_move_pages_to_node(struct list_head *pagelist, int node)
 		.nid = node,
 		.gfp_mask = GFP_HIGHUSER_MOVABLE | __GFP_THISNODE,
 		.reason = MR_SYSCALL,
+		.alloc_flags = node_is_private(node) ?
+			 ALLOC_ZONELIST_PRIVATE : ALLOC_DEFAULT,
 	};
 
 	err = migrate_pages(pagelist, alloc_migration_target, NULL,
@@ -2268,7 +2271,8 @@ static int __add_folio_for_migration(struct folio *folio, int node,
 	if (is_zero_folio(folio) || is_huge_zero_folio(folio))
 		return -EFAULT;
 
-	if (folio_is_private_managed(folio))
+	if (folio_is_zone_device(folio) ||
+	    !node_allows_user_numa(folio_nid(folio)))
 		return -ENOENT;
 
 	if (folio_nid(folio) == node)
@@ -2392,11 +2396,14 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,
 		err = -ENODEV;
 		if (node < 0 || node >= MAX_NUMNODES)
 			goto out_flush;
-		if (!node_state(node, N_MEMORY))
+
+		if (!node_allows_user_numa(node))
 			goto out_flush;
 
 		err = -EACCES;
-		if (!node_isset(node, task_nodes))
+		/* Private nodes are not partitioned by cpuset.mem */
+		if (!node_is_private(node) &&
+		    !node_isset(node, task_nodes))
 			goto out_flush;
 
 		if (current_node == NUMA_NO_NODE) {
@@ -2477,7 +2484,9 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
 		if (folio) {
 			if (is_zero_folio(folio) || is_huge_zero_folio(folio))
 				err = -EFAULT;
-			else if (folio_is_private_managed(folio))
+			else if (folio_is_zone_device(folio) ||
+				 (folio_is_private_node(folio) &&
+				  !node_allows_user_numa(folio_nid(folio))))
 				err = -ENOENT;
 			else
 				err = folio_nid(folio);
Signature · 1 line
2.53.0-Meta
#62 lore raw

Re: [PATCH v5 27/36] mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:

New issues:
- [Medium] Relative node indices are incorrectly treated as absolute node IDs for private node checking in mpol_set_nodemask.
- [Medium] Hugetlb folio migration to private nodes unconditionally fails because the hugetlb allocator ignores ALLOC_ZONELIST_PRIVATE.

Pre-existing issues:
- [High] Returning ACTION_AGAIN on pte_offset_map_lock() failure causes an infinite loop in the page table walker. --

--- Patch [27]: [PATCH v5 27/36] mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls --- commit 90c010ac54e1befaa9a30ca8c81fdac4b3524857
Author: Gregory Price <[email protected]>

mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls

This commit provides a mechanism to opt private nodes into userland numa management. It adds node_allows_user_numa() to encapsulate whether a node supports userland NUMA controls and modifies the memory policy and migration interfaces to support these nodes.

Quoted 11 lines
mm/mempolicy.c +1 -2
index a3ffb09897489..fe42a510590a2 100644
@@ -434,13 +434,14 @@ static int mpol_set_nodemask(struct mempolicy *pol,
 
 	/*
 	 * Private nodes are not in cpuset.mems, so they're always stripped.
-	 * Driver-allocated policies will already have MPOL_F_PRIVATE set,
-	 * if that's the case, add back in the requested set of private nodes.
+	 * Driver-allocated policies (MPOL_F_PRIVATE) and CAP_USER_NUMA private

mm/mempolicy.c static int mpol_set_nodemask(struct mempolicy *pol, · line 446

Quoted 10 lines
mm/mempolicy.c +3 -1
+	 * nodes should be added back into the nodemask.
 	 */
 	for_each_node_mask(nid, *nodes) {
 		if (!node_is_private(nid))
 			continue;
-		if (pol->flags & MPOL_F_PRIVATE)
+		if ((pol->flags & MPOL_F_PRIVATE) ||
+		    node_allows_user_numa(nid))
 			node_set(nid, nsc->mask2);
 	}

[Severity: Medium]
When MPOL_F_RELATIVE_NODES is set, the *nodes mask contains relative indices instead of absolute node IDs. Does this loop incorrectly evaluate those relative indices as absolute node IDs when calling node_is_private(nid)?

If the relative index happens to match an absolute private node ID, it appears that node could be erroneously added to the final policy mask.

[ ... ]

mm/mempolicy.c static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, · line 760

Quoted 10 lines
mm/mempolicy.c +2 -1
@@ -752,7 +753,8 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,
 			continue;
 		}
 		folio = vm_normal_folio(vma, addr, ptent);
-		if (!folio || folio_is_private_managed(folio))
+		if (!folio || folio_is_zone_device(folio) ||
+		    !node_allows_user_numa(folio_nid(folio)))
 			continue;
 		if (folio_test_large(folio) && max_nr != 1)
 			nr = folio_pte_batch(folio, pte, ptent, max_nr);

[Severity: High]
This is a pre-existing issue, but earlier in queue_folios_pte_range() we return ACTION_AGAIN if pte_offset_map_lock() fails:

queue_folios_pte_range()

    ...
    mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl);
    if (!pte) {
        walk->action = ACTION_AGAIN;
        return 0;
    }

Does setting ACTION_AGAIN cause an infinite retry loop in the pagewalker when pte_offset_map_lock() fails non-transiently, such as when encountering migration entries?

mm/migrate.c struct folio *alloc_migration_target(struct folio *src, unsigned long private) · line 2236

Quoted 12 lines
mm/migrate.c +2 -1
index d20674c07b947..b548d79352a38 100644
@@ -2231,7 +2231,8 @@ struct folio *alloc_migration_target(struct folio *src, unsigned long private)
 	if (is_highmem_idx(zidx) || zidx == ZONE_MOVABLE)
 		gfp_mask |= __GFP_HIGHMEM;
 
-	return __folio_alloc(gfp_mask, order, nid, mtc->nmask, ALLOC_DEFAULT);
+	return __folio_alloc(gfp_mask, order, nid, mtc->nmask,
+			     mtc->alloc_flags);
 }

[Severity: Medium]
This correctly updates the standard folio allocation to use mtc->alloc_flags, but earlier in alloc_migration_target(), the hugetlb allocation path does not pass this flag:

    if (folio_test_hugetlb(src)) {
        ...
        return alloc_hugetlb_folio_nodemask(h, nid, mtc->nmask, gfp_mask,
                                            htlb_allow_alloc_fallback(mtc->reason));
    }

Will hugetlb migration to private nodes fail unconditionally because it misses the ALLOC_ZONELIST_PRIVATE flag needed by the hugetlb allocator?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=27
author maintainer
#63 lore raw

PATCH v5 28/36 mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes

In reply to Gregory Price

Add CAP_HOTUNPLUG, which allows memory_hotplug to migrate a private node's folios for the purpose of hotunplugging memory.

Without this, hotunplug fails if any folio on the node is allocated.

Add node_allows_hotunplug() - which returns true for any normal node and private nodes with CAP_HOTUNPLUG.

---
 include/linux/node_private.h | 26 ++++++++++++++++++++++++++
 mm/memory_hotplug.c          |  4 ++--
 2 files changed, 28 insertions(+), 2 deletions(-)
include/linux/node_private.h +26 -0
index 655fe9ec5cb61..7b617b1fa9c28 100644
@@ -14,6 +14,7 @@ struct page;
  */
 #define NODE_PRIVATE_CAP_RECLAIM	(1UL << 0)	/* allow mm reclaim */
 #define NODE_PRIVATE_CAP_USER_NUMA	(1UL << 1)	/* allow mempolicy */
+#define NODE_PRIVATE_CAP_HOTUNPLUG	(1UL << 2)	/* allow hot-unplug */
 
 /**
  * struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
@@ -97,6 +98,26 @@ static inline bool node_allows_user_numa(int nid)
 	return ret;
 }
 
+/**
+ * node_allows_hotunplug - may hot-unplug migrate this node's folios?
+ * @nid: the node to test
+ *
+ * True for normal nodes and private nodes opted into CAP_HOTUNPLUG.
+ */
+static inline bool node_allows_hotunplug(int nid)
+{
+	struct node_private *np;
+	bool ret;
+
+	if (!node_state(nid, N_MEMORY_PRIVATE))
+		return true;
+	rcu_read_lock();
+	np = rcu_dereference(NODE_DATA(nid)->node_private);
+	ret = np && (np->caps & NODE_PRIVATE_CAP_HOTUNPLUG);
+	rcu_read_unlock();
+	return ret;
+}
+
 #else /* !CONFIG_NUMA */
 
 static inline bool folio_is_private_node(struct folio *folio)
@@ -124,6 +145,11 @@ static inline bool node_allows_user_numa(int nid)
 	return true;
 }
 
+static inline bool node_allows_hotunplug(int nid)
+{
+	return true;
+}
+
 #endif /* CONFIG_NUMA */
 
 #if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
mm/memory_hotplug.c +2 -2
index 1f42ed303366c..e9573f90e0dcd 100644
@@ -1943,8 +1943,8 @@ static int do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
 			goto put_folio;
 		}
 
-		/* Private node folios cannot migrate, fail outright */
-		if (folio_is_private_node(folio)) {
+		/* Fail outright on private nodes w/o hotunplug support */
+		if (!node_allows_hotunplug(folio_nid(folio))) {
 			pr_info_ratelimited("memory offline refused: node %d pfn %lx\n",
 					    folio_nid(folio), pfn);
 			folio_put(folio);
Signature · 1 line
2.53.0-Meta
author maintainer
#64 lore raw

Re: [PATCH v5 28/36] mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes

Quoted 9 lines · Gregory Price

Add CAP_HOTUNPLUG, which allows memory_hotplug to migrate a private node's folios for the purpose of hotunplugging memory.

Without this, hotunplug fails if any folio on the node is allocated.

Add node_allows_hotunplug() - which returns true for any normal node and private nodes with CAP_HOTUNPLUG.

after this question:
https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#m5b8166212a276fece2380f2dd03d6864375a576b

and some off-list feedback about limiting initial cap bits to active use cases, I realized that this CAP bit can be entirely eliminated.

hotplug notifier callbacks already allow a driver to prevent memory block state transitions - so that control already exists.

v6 will remove both the filtering here and the cap bit.

~Gregory

author maintainer
#65 lore raw

PATCH v5 29/36 mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion

In reply to Gregory Price

A private node is invisible to the tiering/demotion hierarchy by default:
memory-tiers.c never includes it, so reclaim never demotes onto it.

Add NODE_PRIVATE_CAP_DEMOTION to opt a private node into demotion.

When set, memory-tiers adds the node to the demotion set and reclaim's demote path may target it (allocating from the private zonelist).

Demotion is driven by reclaim, so CAP_DEMOTION requires CAP_RECLAIM. Otherwise the node can either fill up and drive odd system-wide OOM behavior, or demotion doesn't work (nothing can demote from the node).

---
 drivers/base/node.c          |  5 +++++
 include/linux/node_private.h | 30 +++++++++++++++++++++++++++++
 mm/memory-tiers.c            | 37 +++++++++++++++++++++++++++---------
 mm/vmscan.c                  |  4 ++++
 4 files changed, 67 insertions(+), 9 deletions(-)
drivers/base/node.c +5 -0
index 94cd51f51b7e8..3d61ca1b805dc 100644
@@ -896,6 +896,11 @@ int node_private_register(int nid, struct node_private *np)
 	if (!np || !node_possible(nid))
 		return -EINVAL;
 
+	/* Demotion is driven by reclaim, so it requires reclaim. */
+	if ((np->caps & NODE_PRIVATE_CAP_DEMOTION) &&
+	    !(np->caps & NODE_PRIVATE_CAP_RECLAIM))
+		return -EINVAL;
+
 	mutex_lock(&node_private_lock);
 	mem_hotplug_begin();
 
include/linux/node_private.h +30 -0
index 7b617b1fa9c28..87b03444b2c97 100644
@@ -15,6 +15,7 @@ struct page;
 #define NODE_PRIVATE_CAP_RECLAIM	(1UL << 0)	/* allow mm reclaim */
 #define NODE_PRIVATE_CAP_USER_NUMA	(1UL << 1)	/* allow mempolicy */
 #define NODE_PRIVATE_CAP_HOTUNPLUG	(1UL << 2)	/* allow hot-unplug */
+#define NODE_PRIVATE_CAP_DEMOTION	(1UL << 3)	/* allow tiering demotion */
 
 /**
  * struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
@@ -118,6 +119,30 @@ static inline bool node_allows_hotunplug(int nid)
 	return ret;
 }
 
+/**
+ * node_allows_demotion - may kernel tiering demote to this node?
+ * @nid: the node to test
+ *
+ * Governs whether a private node participates in the demotion hierarchy.
+ * Demotion accumulates pages on the node, so CAP_DEMOTION requires CAP_RECLAIM
+ * (enforced at registration) as a safety valve.
+ *
+ * return: true for normal nodes and private nodes opted into CAP_DEMOTION.
+ */
+static inline bool node_allows_demotion(int nid)
+{
+	struct node_private *np;
+	bool ret;
+
+	if (!node_state(nid, N_MEMORY_PRIVATE))
+		return true;
+	rcu_read_lock();
+	np = rcu_dereference(NODE_DATA(nid)->node_private);
+	ret = np && (np->caps & NODE_PRIVATE_CAP_DEMOTION);
+	rcu_read_unlock();
+	return ret;
+}
+
 #else /* !CONFIG_NUMA */
 
 static inline bool folio_is_private_node(struct folio *folio)
@@ -150,6 +175,11 @@ static inline bool node_allows_hotunplug(int nid)
 	return true;
 }
 
+static inline bool node_allows_demotion(int nid)
+{
+	return true;
+}
+
 #endif /* CONFIG_NUMA */
 
 #if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
mm/memory-tiers.c +28 -9
index 25e121851b586..c673080d153e4 100644
@@ -7,6 +7,7 @@
 #include <linux/memory-tiers.h>
 #include <linux/notifier.h>
 #include <linux/sched/sysctl.h>
+#include <linux/node_private.h>
 
 #include "internal.h"
 
@@ -317,6 +318,21 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
 	rcu_read_unlock();
 }
 
+/* Tiering set: N_MEMORY | (N_MEMORY_PRIVATE w/ CAP_DEMOTION) */
+static nodemask_t tierable_nodes;
+
+static void update_tierable_nodes(void)
+{
+	int node;
+
+	lockdep_assert_held_once(&memory_tier_lock);
+
+	tierable_nodes = node_states[N_MEMORY];
+	for_each_node_state(node, N_MEMORY_PRIVATE)
+		if (node_allows_demotion(node))
+			node_set(node, tierable_nodes);
+}
+
 /**
  * next_demotion_node() - Get the next node in the demotion path
  * @node: The starting node to lookup the next node
@@ -330,7 +346,7 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
 int next_demotion_node(int node, const nodemask_t *allowed_mask)
 {
 	struct demotion_nodes *nd;
-	nodemask_t mask;
+	nodemask_t mask, tierable;
 
 	if (!node_demotion)
 		return NUMA_NO_NODE;
@@ -370,7 +386,8 @@ int next_demotion_node(int node, const nodemask_t *allowed_mask)
 	 * closest demotion target.
 	 */
 	nodes_complement(mask, *allowed_mask);
-	return find_next_best_node_in(node, &mask, &node_states[N_MEMORY]);
+	tierable = tierable_nodes;
+	return find_next_best_node_in(node, &mask, &tierable);
 }
 
 static void disable_all_demotion_targets(void)
@@ -378,7 +395,7 @@ static void disable_all_demotion_targets(void)
 	struct memory_tier *memtier;
 	int node;
 
-	for_each_node_state(node, N_MEMORY) {
+	for_each_node_mask(node, tierable_nodes) {
 		node_demotion[node].preferred = NODE_MASK_NONE;
 		/*
 		 * We are holding memory_tier_lock, it is safe
@@ -401,7 +418,7 @@ static void dump_demotion_targets(void)
 {
 	int node;
 
-	for_each_node_state(node, N_MEMORY) {
+	for_each_node_mask(node, tierable_nodes) {
 		struct memory_tier *memtier = __node_get_memory_tier(node);
 		nodemask_t preferred = node_demotion[node].preferred;
 
@@ -435,9 +452,10 @@ static void establish_demotion_targets(void)
 	if (!node_demotion)
 		return;
 
+	update_tierable_nodes();
 	disable_all_demotion_targets();
 
-	for_each_node_state(node, N_MEMORY) {
+	for_each_node_mask(node, tierable_nodes) {
 		best_distance = -1;
 		nd = &node_demotion[node];
 
@@ -455,7 +473,7 @@ static void establish_demotion_targets(void)
 		 * nodelist to skip list so that we find the best node from the
 		 * memtier nodelist.
 		 */
-		nodes_andnot(tier_nodes, node_states[N_MEMORY], tier_nodes);
+		nodes_andnot(tier_nodes, tierable_nodes, tier_nodes);
 
 		/*
 		 * Find all the nodes in the memory tier node list of same best distance.
@@ -464,7 +482,7 @@ static void establish_demotion_targets(void)
 		 */
 		do {
 			target = find_next_best_node_in(node, &tier_nodes,
-							&node_states[N_MEMORY]);
+							&tierable_nodes);
 			if (target == NUMA_NO_NODE)
 				break;
 
@@ -503,7 +521,7 @@ static void establish_demotion_targets(void)
 	 * allocation to a set of nodes that is closer the above selected
 	 * preferred node.
 	 */
-	lower_tier = node_states[N_MEMORY];
+	lower_tier = tierable_nodes;
 	list_for_each_entry(memtier, &memory_tiers, list) {
 		/*
 		 * Keep removing current tier from lower_tier nodes,
@@ -550,7 +568,8 @@ static struct memory_tier *set_node_memory_tier(int node)
 
 	lockdep_assert_held_once(&memory_tier_lock);
 
-	if (!node_state(node, N_MEMORY))
+	/* Include N_MEMORY and N_MEMORY_PRIVATE with CAP_DEMOTION */
+	if (!node_state(node, N_MEMORY) && !node_allows_demotion(node))
 		return ERR_PTR(-EINVAL);
 
 	mt_calc_adistance(node, &adist);
mm/vmscan.c +4 -0
index f1722693ac2db..b617f7cd1e716 100644
@@ -963,6 +963,10 @@ static struct folio *alloc_demote_folio(struct folio *src,
 
 	mtc = (struct migration_target_control *)private;
 
+	if (mtc->nmask &&
+	    nodes_intersects(*mtc->nmask, node_states[N_MEMORY_PRIVATE]))
+		mtc->alloc_flags = ALLOC_ZONELIST_PRIVATE;
+
 	/*
 	 * make sure we allocate from the target node first also trying to
 	 * demote or reclaim pages from the target node via kswapd if we are
Signature · 1 line
2.53.0-Meta
#66 lore raw

Re: [PATCH v5 29/36] mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Concurrent, unprotected access to the global `tierable_nodes` nodemask can result in a torn…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Concurrent, unprotected access to the global tierable_nodes nodemask can result in a torn read. --

--- Patch [29]: [PATCH v5 29/36] mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion --- commit e4de60011eb318a533982e5d4b896e77cc0ad141
Author: Gregory Price <[email protected]>

mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion

This commit introduces NODE_PRIVATE_CAP_DEMOTION, allowing private memory nodes to be included in the tiering and demotion hierarchy. By opting into this capability, memory-tiers will include the private node in demotion paths, allowing page reclaim to target it.

Quoted 23 lines
mm/memory-tiers.c +15 -0
index 25e121851b586..c673080d153e4 100644
@@ -317,6 +318,21 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
 	rcu_read_unlock();
 }
 
+/* Tiering set: N_MEMORY | (N_MEMORY_PRIVATE w/ CAP_DEMOTION) */
+static nodemask_t tierable_nodes;
+
+static void update_tierable_nodes(void)
+{
+	int node;
+
+	lockdep_assert_held_once(&memory_tier_lock);
+
+	tierable_nodes = node_states[N_MEMORY];
+	for_each_node_state(node, N_MEMORY_PRIVATE)
+		if (node_allows_demotion(node))
+			node_set(node, tierable_nodes);
+}
+

[ ... ]

Quoted 9 lines
mm/memory-tiers.c +1 -1
@@ -330,7 +346,7 @@ void node_get_allowed_targets(pg_data_t *pgdat, nodemask_t *targets)
 int next_demotion_node(int node, const nodemask_t *allowed_mask)
 {
 	struct demotion_nodes *nd;
-	nodemask_t mask;
+	nodemask_t mask, tierable;
 
 	if (!node_demotion)
 		return NUMA_NO_NODE;

[ ... ]

mm/memory-tiers.c int next_demotion_node(int node, const nodemask_t *allowed_mask) · line 389

Quoted 6 lines
mm/memory-tiers.c +1 -1
@@ -370,7 +386,8 @@ int next_demotion_node(int node, const nodemask_t *allowed_mask)
 	 * closest demotion target.
 	 */
 	nodes_complement(mask, *allowed_mask);
-	return find_next_best_node_in(node, &mask, &node_states[N_MEMORY]);
+	tierable = tierable_nodes;

[Severity: Medium]
Could this lockless copy result in a torn read of the tierable_nodes nodemask?

In update_tierable_nodes(), the global tierable_nodes mask is updated via a non-atomic struct assignment and iterative node_set() modifications. Here in next_demotion_node(), the mask is copied locklessly to the stack during the reclaim hotpath.

Since nodemask_t can be up to 128 bytes depending on MAX_NUMNODES, concurrent updates from memory hotplug could cause next_demotion_node() to see a partially updated mask. This could cause page reclaim to transiently skip valid private nodes or behave suboptimally during demotion.

Quoted 4 lines
mm/memory-tiers.c +1 -0
+	return find_next_best_node_in(node, &mask, &tierable);
 }
 
 static void disable_all_demotion_targets(void)
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=29
author maintainer
#67 lore raw

PATCH v5 30/36 mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing

In reply to Gregory Price

By default NUMA balancing does not scan or prot_none private-node folios, so the kernel never migrates them via access sampling.

Add NODE_PRIVATE_CAP_NUMA_BALANCING to opt a private node in.

Add folio_allows_numa_balance() to gate change_prot_numa() scans on node eligibility. Opted-in private node participate like normal.

Unlike demotion, NUMA balancing is not reclaim-driven, so this capability stands alone and does not require CAP_RECLAIM.

---
 include/linux/node_private.h | 29 +++++++++++++++++++++++++++++
 mm/internal.h                | 13 +++++++++++++
 mm/mempolicy.c               |  2 +-
 3 files changed, 43 insertions(+), 1 deletion(-)
include/linux/node_private.h +29 -0
index 87b03444b2c97..5c3e070ed0deb 100644
@@ -16,6 +16,7 @@ struct page;
 #define NODE_PRIVATE_CAP_USER_NUMA	(1UL << 1)	/* allow mempolicy */
 #define NODE_PRIVATE_CAP_HOTUNPLUG	(1UL << 2)	/* allow hot-unplug */
 #define NODE_PRIVATE_CAP_DEMOTION	(1UL << 3)	/* allow tiering demotion */
+#define NODE_PRIVATE_CAP_NUMA_BALANCING	(1UL << 4)	/* allow NUMA balancing */
 
 /**
  * struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
@@ -143,6 +144,29 @@ static inline bool node_allows_demotion(int nid)
 	return ret;
 }
 
+/**
+ * node_allows_numa_balancing - may NUMA balancing scan/migrate this node?
+ * @nid: the node to test
+ *
+ * Access-based promotion/migration.  Unlike demotion this is not reclaim-driven,
+ * so CAP_NUMA_BALANCING stands alone (no CAP_RECLAIM dependency).
+ *
+ * return: true for normal nodes and private nodes opted into CAP_NUMA_BALANCING.
+ */
+static inline bool node_allows_numa_balancing(int nid)
+{
+	struct node_private *np;
+	bool ret;
+
+	if (!node_state(nid, N_MEMORY_PRIVATE))
+		return true;
+	rcu_read_lock();
+	np = rcu_dereference(NODE_DATA(nid)->node_private);
+	ret = np && (np->caps & NODE_PRIVATE_CAP_NUMA_BALANCING);
+	rcu_read_unlock();
+	return ret;
+}
+
 #else /* !CONFIG_NUMA */
 
 static inline bool folio_is_private_node(struct folio *folio)
@@ -180,6 +204,11 @@ static inline bool node_allows_demotion(int nid)
 	return true;
 }
 
+static inline bool node_allows_numa_balancing(int nid)
+{
+	return true;
+}
+
 #endif /* CONFIG_NUMA */
 
 #if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
mm/internal.h +13 -0
index 62e68acae08a3..01ab8b32b0bd8 100644
@@ -123,6 +123,19 @@ static inline bool folio_allows_madvise(struct folio *folio)
 	       node_allows_reclaim(folio_nid(folio));
 }
 
+/*
+ * folio_allows_numa_balance() - may NUMA balancing scan/migrate this folio?
+ *
+ * NUMA balancing is access-aware tiering migration, so it follows the tiering
+ * opt-in: false for ZONE_DEVICE and for N_MEMORY_PRIVATE nodes without
+ * CAP_NUMA_BALANCING, true for all other folios.
+ */
+static inline bool folio_allows_numa_balance(struct folio *folio)
+{
+	return !folio_is_zone_device(folio) &&
+	       node_allows_numa_balancing(folio_nid(folio));
+}
+
 /*
  * folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
  *
mm/mempolicy.c +1 -1
index fe42a510590a2..4daba81fff7c7 100644
@@ -874,7 +874,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma,
 {
 	int nid;
 
-	if (!folio || folio_is_private_managed(folio) || folio_test_ksm(folio))
+	if (!folio || !folio_allows_numa_balance(folio) || folio_test_ksm(folio))
 		return false;
 
 	/* Also skip shared copy-on-write folios */
Signature · 1 line
2.53.0-Meta
author maintainer
#68 lore raw

PATCH v5 31/36 mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning

In reply to Gregory Price

Add NODE_PRIVATE_CAP_LTPIN so a node can opt back into gup pins.

An opted-in node is then pinned exactly like ordinary memory, or migrated and pinned if in ZONE_MOVABLE.

---
 include/linux/node_private.h | 29 +++++++++++++++++++++++++++++
 mm/gup.c                     |  3 ++-
 mm/internal.h                |  9 +++++----
 3 files changed, 36 insertions(+), 5 deletions(-)
include/linux/node_private.h +29 -0
index 5c3e070ed0deb..6f568bc0d46c7 100644
@@ -17,6 +17,7 @@ struct page;
 #define NODE_PRIVATE_CAP_HOTUNPLUG	(1UL << 2)	/* allow hot-unplug */
 #define NODE_PRIVATE_CAP_DEMOTION	(1UL << 3)	/* allow tiering demotion */
 #define NODE_PRIVATE_CAP_NUMA_BALANCING	(1UL << 4)	/* allow NUMA balancing */
+#define NODE_PRIVATE_CAP_LTPIN		(1UL << 5)	/* allow GUP pins */
 
 /**
  * struct node_private - Per-node container for N_MEMORY_PRIVATE nodes
@@ -167,6 +168,29 @@ static inline bool node_allows_numa_balancing(int nid)
 	return ret;
 }
 
+/**
+ * node_allows_ltpin - may a folio on this node be long-term GUP-pinned?
+ * @nid: the node to test
+ *
+ * Opted-out private nodes cause longterm pins to outright fail regardless
+ * of ZONE placement (NORMAL would allow, MOVABLE would migrate first).
+ *
+ * Opted-in private nodes allow longterm pins to operate normally.
+ */
+static inline bool node_allows_ltpin(int nid)
+{
+	struct node_private *np;
+	bool ret;
+
+	if (!node_state(nid, N_MEMORY_PRIVATE))
+		return true;
+	rcu_read_lock();
+	np = rcu_dereference(NODE_DATA(nid)->node_private);
+	ret = np && (np->caps & NODE_PRIVATE_CAP_LTPIN);
+	rcu_read_unlock();
+	return ret;
+}
+
 #else /* !CONFIG_NUMA */
 
 static inline bool folio_is_private_node(struct folio *folio)
@@ -209,6 +233,11 @@ static inline bool node_allows_numa_balancing(int nid)
 	return true;
 }
 
+static inline bool node_allows_ltpin(int nid)
+{
+	return true;
+}
+
 #endif /* CONFIG_NUMA */
 
 #if defined(CONFIG_NUMA) && defined(CONFIG_MEMORY_HOTPLUG)
mm/gup.c +2 -1
index a7d4de223785c..1e313aa590bc1 100644
@@ -547,7 +547,8 @@ static struct folio *try_grab_folio_fast(struct page *page, int refs,
 	/*
 	 * Can't do FOLL_LONGTERM + FOLL_PIN gup fast path if not in a
 	 * right zone, so fail and let the caller fall back to the slow
-	 * path.  Fail for private-node folios here so slow path rejects.
+	 * path.  Fail for non-opted-in private node folios here so the
+	 * slow path can reject the pin entirely.
 	 */
 	if (unlikely((flags & FOLL_LONGTERM) &&
 		     !folio_allows_longterm_pin(folio))) {
mm/internal.h +5 -4
index 01ab8b32b0bd8..9621eb127c28c 100644
@@ -146,19 +146,20 @@ static inline bool folio_allows_numa_balance(struct folio *folio)
 static inline bool folio_allows_longterm_pin(struct folio *folio)
 {
 	return folio_is_longterm_pinnable(folio) &&
-	       !folio_is_private_node(folio);
+	       node_allows_ltpin(folio_nid(folio));
 }
 
 /*
  * folio_longterm_pin_forbidden() - must a longterm pin of this folio fail
  * outright (neither pinned in place nor migrated off the node)?
  *
- * True for any folio on a private node: such memory can be neither pinned
- * nor migrated, so the pin must be rejected with the folio left in place.
+ * True only for a folio on a private node that did not opt into longterm
+ * pinning (NODE_PRIVATE_CAP_LTPIN); node_allows_ltpin() is true for ordinary
+ * nodes and for opted-in private nodes, so this never trips them.
  */
 static inline bool folio_longterm_pin_forbidden(struct folio *folio)
 {
-	return folio_is_private_node(folio);
+	return !node_allows_ltpin(folio_nid(folio));
 }
 
 /*
Signature · 1 line
2.53.0-Meta
author maintainer
#69 lore raw

PATCH v5 32/36 mm/khugepaged: base private node collapse eligiblity on actor/cap bits

In reply to Gregory Price

Collapse can move base pages across nodes. The khugepaged daemon does this transparently, while MADV_COLLAPSE is a userland construct.

Isolate private nodes from khugepaged exactly like ZONE_DEVICE. Never collapse private node folios silently (the owner may not support migration).

MADV_COLLAPSE is user-initiated, so it is allowed on a private node opted into CAP_USER_NUMA, consistent with other userspace interfaces.

folio_allows_collapse() encodes this: false for khugepaged on a private node, CAP_USER_NUMA for MADV_COLLAPSE.

---
 mm/internal.h   | 26 ++++++++++++++++++++++++++
 mm/khugepaged.c | 16 +++++++++++++---
 2 files changed, 39 insertions(+), 3 deletions(-)
mm/internal.h +26 -0
index 9621eb127c28c..9dbc5752cd3fe 100644
@@ -136,6 +136,32 @@ static inline bool folio_allows_numa_balance(struct folio *folio)
 	       node_allows_numa_balancing(folio_nid(folio));
 }
 
+/*
+ * folio_allows_collapse() - may collapse fold this folio into a THP?
+ * @is_khugepaged: true for the khugepaged, false for MADV_COLLAPSE.
+ *
+ * Collapse is a residency operation gated by the actor calling it:
+ *   - khugepaged never operates on private-node folios (like ZONE_DEVICE)
+ *   - MADV_COLLAPSE is gated by CAP_USER_NUMA
+ *
+ * Never true for ZONE_DEVICE.
+ */
+static inline bool folio_allows_collapse(struct folio *folio, bool is_khugepaged)
+{
+	int nid = folio_nid(folio);
+
+	if (folio_is_zone_device(folio))
+		return false;
+	if (is_khugepaged)
+		return !node_is_private(nid);
+	return node_allows_user_numa(nid);
+}
+
+static inline bool page_allows_collapse(struct page *page, bool is_khugepaged)
+{
+	return folio_allows_collapse(page_folio(page), is_khugepaged);
+}
+
 /*
  * folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
  *
mm/khugepaged.c +13 -3
index fb4378cc17b10..5f20839857738 100644
@@ -700,7 +700,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 			goto out;
 		}
 		page = vm_normal_page(vma, addr, pteval);
-		if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
+		if (unlikely(!page) ||
+		    unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {
 			result = SCAN_PAGE_NULL;
 			goto out;
 		}
@@ -1241,9 +1242,17 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
 	gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() :
 		     GFP_TRANSHUGE);
 	int node = collapse_find_target_node(cc);
+	unsigned int aflags;
 	struct folio *folio;
+	bool allow;
 
-	folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, ALLOC_DEFAULT);
+	/* Private node access: khugepaged never, madvise with CAP_USER_NUMA */
+	allow = cc->is_khugepaged ? !node_is_private(node)
+				  : node_allows_user_numa(node);
+	aflags = (allow && node_is_private(node)) ?
+		ALLOC_ZONELIST_PRIVATE : ALLOC_DEFAULT;
+
+	folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, aflags);
 	if (!folio) {
 		*foliop = NULL;
 		if (is_pmd_order(order))
@@ -1687,7 +1696,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		}
 
 		page = vm_normal_page(vma, addr, pteval);
-		if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
+		if (unlikely(!page) ||
+		    unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {
 			result = SCAN_PAGE_NULL;
 			goto out_unmap;
 		}
Signature · 1 line
2.53.0-Meta
#70 lore raw

Re: [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Missing `folio_allows_collapse()` checks in `collapse_scan_file` allows khugepaged to silently…

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Missing folio_allows_collapse() checks in collapse_scan_file allows khugepaged to silently migrate file-backed/shmem folios off private nodes. --

--- Patch [32]: [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits --- commit e336e94fb86a5e84bcd43a7f17af3bf83d2b9126
Author: Gregory Price <[email protected]>

mm/khugepaged: base private node collapse eligiblity on actor/cap bits

Collapse can move base pages across nodes. The khugepaged daemon does this transparently, while MADV_COLLAPSE is a userland construct.

Isolate private nodes from khugepaged exactly like ZONE_DEVICE. Never collapse private node folios silently (the owner may not support migration).

Quoted 4 lines
mm/khugepaged.c +0 -0
index fb4378cc17b10..5f20839857738 100644

[ ... ]

mm/khugepaged.c static enum scan_result collapse_scan_pmd(struct mm_struct *mm, · line 1703

Quoted 10 lines
mm/khugepaged.c +2 -1
@@ -1687,7 +1696,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		}
 
 		page = vm_normal_page(vma, addr, pteval);
-		if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
+		if (unlikely(!page) ||
+		    unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {
 			result = SCAN_PAGE_NULL;
 			goto out_unmap;
 		}

[Severity: High]
Since this patch updates collapse_scan_pmd() to check page_allows_collapse(), does collapse_scan_file() need a similar check?

Looking at collapse_scan_file(), the loop scans folios and tallies their residency without verifying collapse eligibility:

mm/khugepaged.c:collapse_scan_file() {
    ...
		node = folio_nid(folio);
		if (collapse_scan_abort(node, cc)) {
			result = SCAN_SCAN_ABORT;
			folio_put(folio);
			break;
		}
		cc->node_load[node]++;
    ...
}

If an unprivileged process maps file-backed memory to a private node, this allows the scan to proceed. alloc_charge_folio() will then deny allocation on the private node (since is_khugepaged is true) and fall back to a normal node via ALLOC_DEFAULT.

Once the fallback THP is allocated, collapse_file() copies the memory from the private node to the newly allocated THP on the normal node. Could this result in a silent migration that bypasses the intended isolation guarantees?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=32
#71 lore raw

Re: [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits

In reply to Gregory Price
Quoted 90 lines · Gregory Price

Collapse can move base pages across nodes. The khugepaged daemon does this transparently, while MADV_COLLAPSE is a userland construct.

Isolate private nodes from khugepaged exactly like ZONE_DEVICE. Never collapse private node folios silently (the owner may not support migration).

MADV_COLLAPSE is user-initiated, so it is allowed on a private node opted into CAP_USER_NUMA, consistent with other userspace interfaces.

folio_allows_collapse() encodes this: false for khugepaged on a private node, CAP_USER_NUMA for MADV_COLLAPSE.

---
 mm/internal.h   | 26 ++++++++++++++++++++++++++
 mm/khugepaged.c | 16 +++++++++++++---
 2 files changed, 39 insertions(+), 3 deletions(-)
mm/internal.h +26 -0
index 9621eb127c28c..9dbc5752cd3fe 100644
@@ -136,6 +136,32 @@ static inline bool folio_allows_numa_balance(struct folio *folio)
 	       node_allows_numa_balancing(folio_nid(folio));
 }
 
+/*
+ * folio_allows_collapse() - may collapse fold this folio into a THP?
+ * @is_khugepaged: true for the khugepaged, false for MADV_COLLAPSE.
+ *
+ * Collapse is a residency operation gated by the actor calling it:
+ *   - khugepaged never operates on private-node folios (like ZONE_DEVICE)
+ *   - MADV_COLLAPSE is gated by CAP_USER_NUMA
+ *
+ * Never true for ZONE_DEVICE.
+ */
+static inline bool folio_allows_collapse(struct folio *folio, bool is_khugepaged)
+{
+	int nid = folio_nid(folio);
+
+	if (folio_is_zone_device(folio))
+		return false;
+	if (is_khugepaged)
+		return !node_is_private(nid);
+	return node_allows_user_numa(nid);
+}
+
+static inline bool page_allows_collapse(struct page *page, bool is_khugepaged)
+{
+	return folio_allows_collapse(page_folio(page), is_khugepaged);
+}
+
 /*
  * folio_allows_longterm_pin() - may this folio be long-term GUP-pinned?
  *
mm/khugepaged.c +11 -2
index fb4378cc17b10..5f20839857738 100644
@@ -700,7 +700,8 @@ static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma,
 			goto out;
 		}
 		page = vm_normal_page(vma, addr, pteval);
-		if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
+		if (unlikely(!page) ||
+		    unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {
 			result = SCAN_PAGE_NULL;
 			goto out;
 		}
@@ -1241,9 +1242,17 @@ static enum scan_result alloc_charge_folio(struct folio **foliop, struct mm_stru
 	gfp_t gfp = (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() :
 		     GFP_TRANSHUGE);
 	int node = collapse_find_target_node(cc);
+	unsigned int aflags;
 	struct folio *folio;
+	bool allow;
 
-	folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, ALLOC_DEFAULT);
+	/* Private node access: khugepaged never, madvise with CAP_USER_NUMA */
+	allow = cc->is_khugepaged ? !node_is_private(node)
+				  : node_allows_user_numa(node);
+	aflags = (allow && node_is_private(node)) ?
+		ALLOC_ZONELIST_PRIVATE : ALLOC_DEFAULT;
+
+	folio = __folio_alloc(gfp, order, node, &cc->alloc_nmask, aflags);
 	if (!folio) {
 		*foliop = NULL;
 		if (is_pmd_order(order))

mm/khugepaged.c static enum scan_result collapse_scan_pmd(struct mm_struct *mm, · line 1700

Quoted 6 lines
mm/khugepaged.c +2 -1
@@ -1687,7 +1696,8 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm,
 		}
 
 		page = vm_normal_page(vma, addr, pteval);
-		if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
+		if (unlikely(!page) ||
+		    unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {

I think collapse_scan_file() need something like folio_allows_collapse() check as well.

--Richard

Quoted 7 lines
mm/khugepaged.c +0 -0
 			result = SCAN_PAGE_NULL;
 			goto out_unmap;
 		}
Signature · 1 line
2.53.0-Meta
author maintainer
#72 lore raw

Re: [PATCH v5 32/36] mm/khugepaged: base private node collapse eligiblity on actor/cap bits

Quoted 9 lines · Richard Cheng

mm/khugepaged.c static enum scan_result __collapse_huge_page_isolate(struct vm_area_struct *vma, · line 704

Quoted 5 lines · Gregory Price
mm/khugepaged.c +2 -1
 		page = vm_normal_page(vma, addr, pteval);
-		if (unlikely(!page) || unlikely(page_is_private_managed(page))) {
+		if (unlikely(!page) ||
+		    unlikely(!page_allows_collapse(page, cc->is_khugepaged))) {

I think collapse_scan_file() need something like folio_allows_collapse() check as well.

I believe you are correct, sashiko up some stuff here too.

Thanks!
~Gregory

author maintainer
#73 lore raw

PATCH v5 33/36 Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes

In reply to Gregory Price

Add a design overview of private memory nodes:
- the isolation model (structural zonelist exclusion)
- ZONELIST_PRIVATE
- driver provisioning API
- capability opt-in model and its dependency rules
- observability surfaces

---
 Documentation/mm/index.rst              |   1 +
 Documentation/mm/numa_private_nodes.rst | 160 ++++++++++++++++++++++++
 2 files changed, 161 insertions(+)
 create mode 100644 Documentation/mm/numa_private_nodes.rst
Documentation/mm/index.rst +1 -0
index 13a79f5d092c0..f60704df6104c 100644
@@ -65,6 +65,7 @@ documentation, or deleted if it has served its purpose.
    mmu_notifier
    multigen_lru
    numa
+   numa_private_nodes
    overcommit-accounting
    page_migration
    page_frags
Documentation/mm/numa_private_nodes.rst new file +160 -0

160 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
author maintainer
#74 lore raw

PATCH v5 34/36 mm/mempolicy: add mpol_set_shared_policy_range()

In reply to Gregory Price

mpol_set_shared_policy() installs a policy over a VMA's page-offset range, which is the only programmatic way to populate an inode's shared policy after init.

Two limitations make it unusable for binding an entire backing inode from in-kernel code:

- It requires a VMA, so it cannot cover unmapped offsets. (e.g. unmapped file folios faulted by pagecache)

- mpol_shared_policy_init(), the only no-VMA installer, reconstructs the policy from mpol->w.user_nodemask. That field is only populated for static/relative or mount-string policies.

a policy built programmatically (e.g. by mpol_bind_node()) leaves it empty and stores w.cpuset_mems_allowed instead, so _init mangles it.

Add mpol_set_shared_policy_range(), which installs an already-built, fully contextualised policy verbatim over an arbitrary [start, end) page range with no VMA.

Reimplement mpol_set_shared_policy() as a thin wrapper that derives the range from the VMA, so both share a single underlying path.

Assisted-by Claude:claude-opus-4-8
---
 include/linux/mempolicy.h |  2 ++
 mm/mempolicy.c            | 35 +++++++++++++++++++++++++++++------
 2 files changed, 31 insertions(+), 6 deletions(-)
include/linux/mempolicy.h +2 -0
index 715951a5b03c1..1348e9f5f2cf9 100644
@@ -125,6 +125,8 @@ int vma_dup_policy(struct vm_area_struct *src, struct vm_area_struct *dst);
 void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol);
 int mpol_set_shared_policy(struct shared_policy *sp,
 			   struct vm_area_struct *vma, struct mempolicy *mpol);
+int mpol_set_shared_policy_range(struct shared_policy *sp, pgoff_t start,
+				 pgoff_t end, struct mempolicy *mpol);
 void mpol_free_shared_policy(struct shared_policy *sp);
 struct mempolicy *mpol_shared_policy_lookup(struct shared_policy *sp,
 					    pgoff_t idx);
mm/mempolicy.c +29 -6
index 4daba81fff7c7..23c4c25097450 100644
@@ -3424,24 +3424,47 @@ void mpol_shared_policy_init(struct shared_policy *sp, struct mempolicy *mpol)
 }
 EXPORT_SYMBOL_FOR_MODULES(mpol_shared_policy_init, "kvm");
 
-int mpol_set_shared_policy(struct shared_policy *sp,
-			struct vm_area_struct *vma, struct mempolicy *pol)
+/**
+ * mpol_set_shared_policy_range - install @pol over [@start, @end) of @sp
+ * @sp:    the shared policy tree
+ * @start: first page offset (inclusive)
+ * @end:   last page offset (exclusive)
+ * @pol:   a fully-built, validated policy, or NULL to clear the range
+ *
+ * Installs @pol over the given range, replacing any overlapping policy.
+ * @sp takes its own reference, the caller retains its reference on @pol.
+ *
+ * The policy is not reconstructed, so the policy is preserved exactly.
+ *
+ * Unlike mpol_set_shared_policy(), no VMA is required, so a range that
+ * is never mapped into a VMA can be covered, including the whole file.
+ *
+ * Return: 0 on success, -ENOMEM on allocation failure.
+ */
+int mpol_set_shared_policy_range(struct shared_policy *sp, pgoff_t start,
+				 pgoff_t end, struct mempolicy *pol)
 {
-	const pgoff_t pgoff = vma_start_pgoff(vma);
-	const pgoff_t pgoff_end = vma_end_pgoff(vma);
 	struct sp_node *new = NULL;
 	int err;
 
 	if (pol) {
-		new = sp_alloc(pgoff, pgoff_end, pol);
+		new = sp_alloc(start, end, pol);
 		if (!new)
 			return -ENOMEM;
 	}
-	err = shared_policy_replace(sp, pgoff, pgoff_end, new);
+	err = shared_policy_replace(sp, start, end, new);
 	if (err && new)
 		sp_free(new);
 	return err;
 }
+EXPORT_SYMBOL_FOR_MODULES(mpol_set_shared_policy_range, "kvm");
+
+int mpol_set_shared_policy(struct shared_policy *sp,
+			struct vm_area_struct *vma, struct mempolicy *pol)
+{
+	return mpol_set_shared_policy_range(sp, vma->vm_pgoff,
+					   vma->vm_pgoff + vma_pages(vma), pol);
+}
 EXPORT_SYMBOL_FOR_MODULES(mpol_set_shared_policy, "kvm");
 
 /* Free a backing policy store on inode delete. */
Signature · 1 line
2.53.0-Meta
author maintainer
#75 lore raw

PATCH v5 35/36 KVM: guest_memfd: bind backing memory to a NUMA node at creation

In reply to Gregory Price

guest_memfd presently allocates its page-cache folios through a per-inode shared mempolicy (kvm_gmem_get_folio()).

Today that policy can only be set after the fact, via mbind() on a host mmap of the fd. This requires the fd to be mmap-able and cannot reach folios that are only ever guest-faulted (no host VMA).

Neither holds for a non-mappable (confidential) guest_memfd.

Add GUEST_MEMFD_FLAG_BIND_NODE.

When set, KVM builds an MPOL_BIND policy for the requested node and installs it over the whole inode's shared policy, so every folio is allocated on the requested node with no userspace mbind().

The flag is advertised through KVM_CAP_GUEST_MEMFD_FLAGS only when CONFIG_NUMA is enabled.

Assisted-by Claude:claude-opus-4-8
---
 include/linux/kvm_host.h |  3 +++
 include/uapi/linux/kvm.h |  5 ++++-
 virt/kvm/guest_memfd.c   | 39 +++++++++++++++++++++++++++++++++++++--
 3 files changed, 44 insertions(+), 3 deletions(-)
include/linux/kvm_host.h +3 -0
index ab8cfaec82d31..f6448a73d464c 100644
@@ -739,6 +739,9 @@ static inline u64 kvm_gmem_get_supported_flags(struct kvm *kvm)
 	if (!kvm || kvm_arch_supports_gmem_init_shared(kvm))
 		flags |= GUEST_MEMFD_FLAG_INIT_SHARED;
 
+	if (IS_ENABLED(CONFIG_NUMA))
+		flags |= GUEST_MEMFD_FLAG_BIND_NODE;
+
 	return flags;
 }
 #endif
include/uapi/linux/kvm.h +4 -1
index 419011097fa8e..a8ff150e2cbb7 100644
@@ -1654,11 +1654,14 @@ struct kvm_memory_attributes {
 #define KVM_CREATE_GUEST_MEMFD	_IOWR(KVMIO,  0xd4, struct kvm_create_guest_memfd)
 #define GUEST_MEMFD_FLAG_MMAP		(1ULL << 0)
 #define GUEST_MEMFD_FLAG_INIT_SHARED	(1ULL << 1)
+#define GUEST_MEMFD_FLAG_BIND_NODE	(1ULL << 2)
 
 struct kvm_create_guest_memfd {
 	__u64 size;
 	__u64 flags;
-	__u64 reserved[6];
+	__u32 node;
+	__u32 pad;
+	__u64 reserved[5];
 };
 
 #define KVM_PRE_FAULT_MEMORY	_IOWR(KVMIO, 0xd5, struct kvm_pre_fault_memory)
virt/kvm/guest_memfd.c +37 -2
index f0e5da4908660..de320df08e0ff 100644
@@ -456,6 +456,26 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma,
 	 */
 	return mpol_shared_policy_lookup(&GMEM_I(inode)->policy, pgoff);
 }
+
+static int kvm_gmem_bind_node(struct inode *inode, int node)
+{
+	struct mempolicy *pol;
+	int err;
+
+	pol = mpol_bind_node(node);
+	if (IS_ERR(pol))
+		return PTR_ERR(pol);
+
+	err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, 0,
+					  MAX_LFS_FILESIZE >> PAGE_SHIFT, pol);
+	mpol_put(pol);
+	return err;
+}
+#else
+static int kvm_gmem_bind_node(struct inode *inode, int node)
+{
+	return -EINVAL;
+}
 #endif /* CONFIG_NUMA */
 
 static const struct vm_operations_struct kvm_gmem_vm_ops = {
@@ -557,7 +577,7 @@ bool __weak kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return true;
 }
 
-static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
+static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags, int node)
 {
 	static const char *name = "[kvm-gmem]";
 	struct gmem_file *f;
@@ -598,6 +618,12 @@ static int __kvm_gmem_create(struct kvm *kvm, loff_t size, u64 flags)
 
 	GMEM_I(inode)->flags = flags;
 
+	if (flags & GUEST_MEMFD_FLAG_BIND_NODE) {
+		err = kvm_gmem_bind_node(inode, node);
+		if (err)
+			goto err_inode;
+	}
+
 	file = alloc_file_pseudo(inode, kvm_gmem_mnt, name, O_RDWR, &kvm_gmem_fops);
 	if (IS_ERR(file)) {
 		err = PTR_ERR(file);
@@ -630,6 +656,7 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
 {
 	loff_t size = args->size;
 	u64 flags = args->flags;
+	int node = NUMA_NO_NODE;
 
 	if (flags & ~kvm_gmem_get_supported_flags(kvm))
 		return -EINVAL;
@@ -637,7 +664,15 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
 	if (size <= 0 || !PAGE_ALIGNED(size))
 		return -EINVAL;
 
-	return __kvm_gmem_create(kvm, size, flags);
+	if (flags & GUEST_MEMFD_FLAG_BIND_NODE) {
+		if (args->pad || args->node >= MAX_NUMNODES)
+			return -EINVAL;
+		node = args->node;
+	} else if (args->node || args->pad) {
+		return -EINVAL;
+	}
+
+	return __kvm_gmem_create(kvm, size, flags, node);
 }
 
 int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot,
Signature · 1 line
2.53.0-Meta
#76 lore raw

Re: [PATCH v5 35/36] KVM: guest_memfd: bind backing memory to a NUMA node at creation

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Critical] Unconditional validation of flag-gated syscall parameters outside their flag scope breaks ABI…

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] Unconditional validation of flag-gated syscall parameters outside their flag scope breaks ABI compatibility for existing userspace.
- [Medium] Missing lock when calling mpol_set_nodemask leads to data race and torn read of current->mems_allowed.
--

--- Patch [35]: [PATCH v5 35/36] KVM: guest_memfd: bind backing memory to a NUMA node at creation --- commit 45130876f6e87292f19cee2648796441ea4622ce
Author: Gregory Price <[email protected]>

KVM: guest_memfd: bind backing memory to a NUMA node at creation

guest_memfd presently allocates its page-cache folios through a per-inode shared mempolicy. This patch adds GUEST_MEMFD_FLAG_BIND_NODE to set an MPOL_BIND policy for the requested node and install it over the whole inode's shared policy.

Quoted 4 lines
virt/kvm/guest_memfd.c +0 -0
index f0e5da4908660..de320df08e0ff 100644

[ ... ]

virt/kvm/guest_memfd.c static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma, · line 465

Quoted 11 lines
virt/kvm/guest_memfd.c +7 -0
@@ -456,6 +456,26 @@ static struct mempolicy *kvm_gmem_get_policy(struct vm_area_struct *vma,
 	 */
 	return mpol_shared_policy_lookup(&GMEM_I(inode)->policy, pgoff);
 }
+
+static int kvm_gmem_bind_node(struct inode *inode, int node)
+{
+	struct mempolicy *pol;
+	int err;
+
+	pol = mpol_bind_node(node);

[Severity: Medium]
Can calling mpol_bind_node() here without holding task_lock(current) cause a data race?

Looking at mpol_bind_node(), it calls __mpol_bind_node() which then calls mpol_set_nodemask(). This accesses current->mems_allowed locklessly.

If a task's cpuset is concurrently modified, this could lead to a torn read of current->mems_allowed on systems with >64 NUMA nodes, potentially failing or creating a policy with an incorrect or empty nodemask.

Quoted 8 lines
virt/kvm/guest_memfd.c +8 -0
+	if (IS_ERR(pol))
+		return PTR_ERR(pol);
+
+	err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, 0,
+					  MAX_LFS_FILESIZE >> PAGE_SHIFT, pol);
+	mpol_put(pol);
+	return err;
+}

[ ... ]

Quoted 9 lines
virt/kvm/guest_memfd.c +1 -0
@@ -630,6 +656,7 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
 {
 	loff_t size = args->size;
 	u64 flags = args->flags;
+	int node = NUMA_NO_NODE;
 
 	if (flags & ~kvm_gmem_get_supported_flags(kvm))
 		return -EINVAL;

virt/kvm/guest_memfd.c int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args) · line 671

Quoted 9 lines
virt/kvm/guest_memfd.c +5 -1
@@ -637,7 +664,15 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args)
 	if (size <= 0 || !PAGE_ALIGNED(size))
 		return -EINVAL;
 
-	return __kvm_gmem_create(kvm, size, flags);
+	if (flags & GUEST_MEMFD_FLAG_BIND_NODE) {
+		if (args->pad || args->node >= MAX_NUMNODES)
+			return -EINVAL;
+		node = args->node;
+	} else if (args->node || args->pad) {

[Severity: Critical]
Does unconditionally checking args->node and args->pad when the GUEST_MEMFD_FLAG_BIND_NODE flag is absent break ABI compatibility?

Since these fields were previously part of the unused reserved[] array, existing userspace binaries might allocate struct kvm_create_guest_memfd on the stack without zeroing them.

If an older binary calls the ioctl without the new flag, and the stack garbage happens to be non-zero, this will now fail with -EINVAL.

Quoted 5 lines
virt/kvm/guest_memfd.c +4 -0
+		return -EINVAL;
+	}
+
+	return __kvm_gmem_create(kvm, size, flags, node);
 }
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=35
#77 lore raw
In reply to Gregory Price
Quoted 36 lines · Gregory Price

This series introduces the concept of "Private Memory Nodes", which are opted into two basic functionalities by default:
- page allocation (mm/page_alloc.c)
- OOM killing

All other features of mm/ are opted out of managing these NUMA nodes and its memory. Then we add capability bits to allow node owners to opt back into those services (if supported).

NUMA-hosted memory is presently a most-privilege system (any memory on any node, except ZONE_DEVICE, is 100% fungible and accessible). We then slap restrictions on top of it (cpuset, mempolicy, ZONE type, page/folio flags, etc).

The goal here is to flip that dynamic, isolate by default and then opt-in to specific services that the device says is safe.

Isolation at the NUMA/Zonelist layer provides a powerful mechanism for memory hosted on accelerators - re-use of the kernel mm/ code.

- Accelerators (GPUs) can use demotion, numactl, and reclaim.
- Special memory devices (Compressed RAM) with special access controls (promote-on-write) can have generic services written for them.
- Network devices with large memory regions intended for ring buffers can use the buddy and standard networking stack.
- Slow, disaggregated memory pools which aren't suitable as general purpose memory get cleaner interfaces (no need to re-write the buddy in userland, can use migration interface, etc).
- Per-workload dedicated memory nodes (disaggregated VM memory)

And more use cases I have collected over the past few years.

Not included here is a dax-extension [1] that exposes all the internal bits as userland controls for testing - along with a pile of selftests that prove correctness.

Quoted 8 lines

Changes Since V4
================
- Massive reduction in complexity.
- no ops struct
- no callback functions
- no __GFP_PRIVATE
- no __GFP_THISNODE requirement
- no task flags (no PF_MEMALLOC_* in the alloc path)

Very happy to see this

Quoted 27 lines

- isolation via a dedicated zonelist:
- private nodes are omitted from FALLBACK/NOFALLBACK
- added ZONELIST_PRIVATE(_NOFALLBACK)
- ALLOC_ZONELIST_PRIVATE alloc_flag
- on top of Brendan Jackman's recent mm/page_alloc.h work [3]
- Zonelist selection rides the allocator's alloc_flags
- rename OPS -> CAPS (capabilities)
- split base functionality (isolation) from opt-ins (CAPS)
- first half of series can be merged without CAPS
- dropped compressed ram example from series
- will submit separately if this moves forward
- Added KVM as first primary in-tree user (mempolicy / CAP_USER_NUMA)
- fully functional dax-kmem extension and huge suite of selftests located at my github, to be discussed separately [1]

Patch Layout
============
The series is broken into two sections:

1) N_MEMORY_PRIVATE Introduction.
Introduce the node state.
Opt those nodes out of mm/ services.

2) NODE_PRIVATE_CAP_* features
A set of mm/ service opt-in flags that augment private nodes to make them more useful (i.e. reclaim = overcommit).

Quoted 7 lines

NODE_PRIVATE_CAP_LTPIN for private node folio pinning NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes NODE_PRIVATE_CAP_USER_NUMA for userland numa controls NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim

Looks reasonable, I wonder why USER_NUMA/HOTUNPUG is an opt-in?

Quoted 392 lines

My hope is to merge at least #1 pulled ahead while #2 is debated.

Allocation Isolation
====================
page_alloc presently controls whether a node's memory can be allocated on a given call by 4 things (in order of authority)

1) ZONELIST membership
If a node is not in the walked zonelist, it's unreachable.

2) __GFP_THISNODE
If this flag is set and the only node in the ZONELIST is the singular preferred node (or the local node, for -1)

3) cpuset.mems membership
cpuset trims any node in its allowed list

4) mempolicy nodemask
the allocator will skip any node not in the nodemask.

Except for #1 (Zonelist membership) there are all kinds of weird corner conditions in which 2-4 can be completely ignored (interrupt context, empty set because cpuset doesn't intersect mempolicy, shared vma, ...)

But ZONELIST membership is *absolute*. If a zone is not in the zonelist being walked, IT CANNOT BE ALLOCATED FROM. PERIOD.

The existing zonelists are constructed like so:
ZONELIST_FALLBACK : All N_MEMORY nodes
ZONELIST_NOFALLBACK : A singleton N_MEMORY node

Private node isolation is implemented via ZONELIST isolation:
ZONELIST_PRIVATE : The private node + N_MEMORY
ZONELIST_PRIVATE_NOFALLBACK : The private node alone.

(mirrors exactly the fallback/nofallback for __GFP_THISNODE)

Private nodes:
1) Never appear in any ZONELIST_FALLBACK
2) Have an empty ZONELIST_NOFALLBACK
3) Only appear in their own ZONELIST_PRIVATE(_NOFALLBACK)

1 & 2 mean all existing in-tree callers to page_alloc can NEVER accidentally allocate from a private node.

An allocation must explicitly ask via a zonelist and a nodemask.

alloc_flags |= ALLOC_ZONELIST_PRIVATE; /* use ZONELIST_PRIVATE */
__alloc_pages(..., nodemask); /* with the private node set */

The page allocator keeps all its original interfaces which only ever touch the default zonelists - avoiding churn.

Making it accessible via Mempolicy: MPOL_F_PRIVATE and page_alloc
=================================================================
The vast majority of the kernel will never need to know about ZONELIST_PRIVATE, because we add MPOL_F_PRIVATE to mempolicy.

When a mempolicy has MPOL_F_PRIVATE, the alloc_mpol() interfaces do the zonelist selection for the source of the allocation.

That really is the whole explanation of the mechanism:

  alloc_flags = mpol_alloc_flags(pol);
  page = __alloc_frozen_pages_noprof(..., alloc_flags);

On mm-new this rides the allocator's existing alloc_flags plumbing:
ALLOC_ZONELIST_PRIVATE is just another alloc_flag, so no new parameter, enum, or alloc_context change is required.

For modules that want to implement their own special handling, they get the _private variants for the page allocator. This lets modules re-use the buddy instead of rewriting it.

- alloc_pages_node_private_noprof()
- folio_alloc_node_private_noprof()

This keeps ALLOC_ flags mm/ internal (these functions add the flags).

Isolating private node folios from kernel services
==================================================
We implement filter points in mm/ to prevent operations on private node memory. Where possible, we even re-use existing filter points from ZONE_DEVICE.

Most filter points are one or two lines of code:

Combining ZONE_DEVICE and N_MEMORY_PRIVATE opt-out spots:
- if (folio_is_zone_device(folio))
+ if (unlikely(folio_is_private_managed(folio)))

Disabling a service:
+ if (!node_is_private(nid)) {
+ kswapd_run(nid);
+ kcompactd_run(nid);
+ }

Disallowing a uapi interaction:
+ if (node_state(nid, N_MEMORY_PRIVATE))
+ return -EINVAL;

In the second half of the series, we replace blanket N_MEMORY_PRIVATE filters with NODE_PRIVATE_CAP_* filters to opt those nodes into that interaction if CAP is set.

We abstract this with a nice clean interface to make it really clear what is happening (nodes have features!)

- if (node_state(pgdat->node_id, N_MEMORY_PRIVATE))
+ if (!node_allows_reclaim(pgdat->node_id))

NODE_PRIVATE_CAP_* features
===========================
This series of commits opts private nodes into various mm/ services.

Capabilities:

  NODE_PRIVATE_CAP_RECLAIM       - direct and kswapd reclaim
  NODE_PRIVATE_CAP_USER_NUMA     - userland numa controls
  NODE_PRIVATE_CAP_DEMOTION      - node is a demotion target
  NODE_PRIVATE_CAP_HOTUNPLUG     - hotunplug may migrate

NODE_PRIVATE_CAP_NUMA_BALANCING - NUMAB may target node folios
NODE_PRIVATE_CAP_LTPIN - Longterm pin operates normally

Some opt-in support is more intensive than others, so these features are broken out in a way that we can defer them as future work streams.

NODE_PRIVATE_CAP_RECLAIM:
Enabling reclaim for these nodes is actually surprisingly trivial.

Without CAP_RECLAIM, when an allocation failure occurs, the system will not attempt to swap the memory - and instead will OOM (typically whatever task is using the most memory on *that* private node).

This capability consists of:
1) enabling kswapd and kcompactd for that node at hotplug time.
2) formalizing opt-out hooks to node_allows_reclaim() opt-in hooks.
3) Sets normal watermarks for these nodes.
4) Allow madvise operations on that node (PAGEOUT).
5) A small tweak to how LRU decides which zones to visit.

NODE_PRIVATE_CAP_USER_NUMA
Enables the following userland interfaces to accept the node:

    mbind()
    set_mempolicy()
    set_mempolicy_home_node()
    move_pages()
    migrate_pages()
  example:
     buf = mmap(..., MAP_ANON);
     mbind(buf, ..., {private_node});
     buf[0] = 0xDEADBEEF; /* Page faults onto the private node */

Later - the KVM example shows how in-kernel mempolicies can also be bound by CAP_USER_NUMA.

Otherwise, that's it - it's just a mempolicy with MPOL_F_PRIVATE.

NODE_PRIVATE_CAP_HOTUNPLUG
This is simple: allow hotunplug to migrate this nodes folios.

Some devices may not be able to tolerate unexpected migrations, so we prevent hotunplug from engaging in migration by default.

Some devices may have an mmu_notifier in their driver that can manage the migration and subsequent refault.

CAP_HOTUNPLUG allows memory_hotplug.c to migrate normally.

NODE_PRIVATE_CAP_DEMOTION
This adds the private node as a valid demotion target, and allows reclaim to demote memory from a private node to a demotion target.

Requires: NODE_PRIVATE_CAP_RECLAIM

NODE_PRIVATE_CAP_NUMA_BALANCING
This enables numa balancing to inject prot_none on private node folio mappings and promote them when faults are taken.

NODE_PRIVATE_CAP_LTPIN
This allows GUP Longterm Pinning to operate normally.

Normally, longterm pinning determines folio eligibility based on its ZONE_* membership (among other things).

ZONE_NORMAL is eligible, while ZONE_MOVABLE folios require migration to ZONE_NORMAL before pinning.

Neither operation is preferable by default on a private node, so the base behavior of FOLL_LONGTERM is to FAIL.

This capability allows longterm pinning to operate normally based on the ZONE membership. Private node memory may be hotplugged as either ZONE_NORMAL or ZONE_MOVABLE.

In-tree User: KVM
=================
Dave Jiang proposed [2] dax-backed guest_memfd() memory as a way of enabling disaggregated memory pools to host dedicated KVM memory.

With private nodes, this is trivial (with a bit of basic plumbing):

static int kvm_gmem_bind_node(struct inode *inode, int node) {
...
/* Bind to a private node - gated on CAP_USER_NUMA */ pol = mpol_bind_node(node);
if (IS_ERR(pol))

    return PTR_ERR(pol);

/* Set the shared policy */
err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, ..., pol);
...
}

KVM doesn't even need to know about private nodes at all, all it does is ask mempolicy whether the requested node is a valid bind.

mm/ component testing with dax driver
=====================================
The dax driver extensions[1] implements a simple interface to create a private node from a dax device created by any source.

I left the dax driver extensions out of this feature set because it locks in the CAP_ bits before anyone has input. It's there primarily for testing at this point.

The simplest way to get a dax device is with the memmap= boot arg.
e.g.: "memmap=0x40000000!0x140000000"

The dax driver extension has the following sysfs entries:

  dax0.0/private         - set the node to private
  dax0.0/dax_file        - make /dev/dax0.0 mmap'able in kmem mode
  dax0.0/adistance       - dictate memory_tierN membership
  dax0.0/reclaim         - CAP_RECLAIM
  dax0.0/demotion        - CAP_DEMOTION
  dax0.0/user_numa       - CAP_USER_NUMA
  dax0.0/hotunplug       - CAP_HOTUNPLUG
  dax0.0/numa_balancing  - CAP_NUMA_BALANCING
  dax0.0/ltpin           - CAP_LTPIN

Now consider the following...

Single node reclaim + mbind support:
echo 1 > dax0.0/private
echo 1 > dax0.0/reclaim
echo 1 > dax0.0/user_numa
echo online_movable > dax0.0/state

Test program:

   /* node1: 1GB Private Memory Node, 4GB swap */
   buf = mmap(NULL, TWO_GB, PROT_READ | PROT_WRITE,
              MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
   sys_mbind(p, len, MPOL_BIND, mask, MAXNODE, MPOL_MF_STRICT);
   memset(buf, 0xff, TWO_GB);

We can *guarantee* the ONLY reclaiming tasks are exactly:
- kswapdN (in theory we can even make this optional!)
- the memset task faulting pages in

This also means these are the only tasks capable of becoming locked up and OOMing (as long as it is not under pressure).

The rest of the system remains entirely functional for debugging.

It now becomes possible to micro-benchmark and A/B test reclaim changes with different scenarios (number of tasks, amount of memory, watermark targets, etc), because we have hard controls over exactly which tasks can access that node memory and how.

If we broke CAP_RECLAIM into subflags:
- CAP_RECLAIM_KSWAPD
- CAP_RECLAIM_DIRECT
- CAP_COMPACTION_KCOMPACTD
- CAP_COMPACTION_DIRECT

We can actually test the efficacy of each of these mechanisms in isolation to each other - something that is strictly impossible today.

Bonus Configuration: HBM device memory tiering
==============================================

echo 1 > dax0.0/private       # make the node private
echo 0 > dax0.0/adistance     # highest tier
echo 1 > dax0.0/reclaim       # reclaim active
echo 1 > dax0.0/demotion      # may demote from the node
echo 1 > dax0.0/user_numa     # mbind()

echo online_movable > dax0.0/state
echo 1 > numa/demotion_enabled

This is an HBM device which is treated as the top-tier in the system but for which memory can only enter via explicit mbind().

It can be overcommitted because it can be reclaimed (demotions go to CPU DRAM, and reclaim can swap from it).

If the HBM is managed by an accelerator (GPU), the mmu_notifier allows it to know when reclaim is moving memory out to do device-mmu invalidation prior to migration.

Prereqs, base commit, references
================================
akpm/mm-new - for Brendan Jackman's mm/page_alloc.h work[3]

Prereqs (all already in akpm/mm-new; listed for out-of-tree application):

page_alloc.h split + alloc_flags plumbing this series rides on:
commit e81fae43cd69 ("mm: split out internal page_alloc.h") commit b4ff3b6d0a1d ("mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG")

dax atomic whole-device hotplug (used by the dax extension [1]):
commit d7aa81b9a919 ("mm/memory: add memory_block_aligned_range() helper") commit 3b2f402a1754 ("dax/kmem: add sysfs interface for atomic whole-device hotplug")

[1] https://github.com/gourryinverse/linux/tree/scratch/gourry/managed_nodes/dax_private-mm-new
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

base-commit: c872b70f5d6c742ad34b8e838c92af81c8920b3e

Gregory Price (36):
mm: refactor find_next_best_node to find_next_best_node_in mm/page_alloc: refactor build_node_zonelist() out of build_zonelists() mm/page_alloc: let the bulk and folio allocators carry alloc_flags
numa: introduce N_MEMORY_PRIVATE
mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
cpuset: exclude private nodes from cpuset.mems (default-open) mm/memory_hotplug: disallow migration-driven private node hotunplug mm/mempolicy: skip private node folios when queueing for migration mm/migrate: disallow userland driven migration for private nodes mm/madvise: disallow madvise operations on private node folios mm/compaction: disallow compaction on private nodes mm/page_alloc: clear private node watermarks and system reserves mm/mempolicy: disallow NUMA Balancing prot_none on private nodes mm/damon: skip private node memory in DAMON migration and pageout mm/ksm: skip KSM for managed-memory folios
mm/khugepaged: skip private node folios when trying to collapse. mm/vmscan: disallow reclaim of private node memory
mm/gup: disallow longterm pin of private node folios
proc: include N_MEMORY_PRIVATE nodes in numa_maps output mm/memcontrol: account private-node memory in per-node stats proc/kcore: include private-node RAM in the kcore RAM map mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection mm/mempolicy: apply policy at the kernel zone for private-node binds mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug
mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA
balancing
mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning mm/khugepaged: base private node collapse eligiblity on actor/cap bits Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes mm/mempolicy: add mpol_set_shared_policy_range()
KVM: guest_memfd: bind backing memory to a NUMA node at creation
KVM: selftests: add a guest_memfd FLAG_BIND_NODE test

 Documentation/ABI/stable/sysfs-devices-node   |  10 +
 Documentation/mm/index.rst                    |   1 +
 Documentation/mm/numa_private_nodes.rst       | 160 ++++++++++
 drivers/base/node.c                           | 118 +++++++
 drivers/dax/kmem.c                            |   2 +-
 fs/proc/kcore.c                               |   5 +-
 fs/proc/task_mmu.c                            |  10 +-
 include/linux/gfp.h                           |  22 +-
 include/linux/kvm_host.h                      |   3 +
 include/linux/memory_hotplug.h                |   5 +-
 include/linux/mempolicy.h                     |  16 +
 include/linux/mmzone.h                        |  26 +-
 include/linux/node_private.h                  | 262 ++++++++++++++++
 include/linux/nodemask.h                      |   7 +-
 include/uapi/linux/kvm.h                      |   5 +-
 include/uapi/linux/mempolicy.h                |   1 +
 kernel/cgroup/cpuset.c                        |  26 +-
 mm/compaction.c                               |  13 +
 mm/damon/paddr.c                              |   9 +
 mm/gup.c                                      |  28 +-
 mm/huge_memory.c                              |   5 +
 mm/internal.h                                 |  97 +++++-
 mm/khugepaged.c                               |  18 +-
 mm/ksm.c                                      |   8 +-
 mm/madvise.c                                  |   8 +-
 mm/memcontrol-v1.c                            |   8 +-
 mm/memcontrol.c                               |  13 +-
 mm/memory-tiers.c                             |  40 ++-
 mm/memory_hotplug.c                           | 119 +++++++-
 mm/mempolicy.c                                | 288 +++++++++++++++---
 mm/migrate.c                                  |  19 +-
 mm/mm_init.c                                  |   2 +-
 mm/page_alloc.c                               | 189 +++++++++---
 mm/page_alloc.h                               |  34 +++
Quoted 9 lines
 mm/vmscan.c                                   |  57 +++-
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../kvm/guest_memfd_bind_node_test.c          | 213 +++++++++++++
 virt/kvm/guest_memfd.c                        |  39 ++-
 38 files changed, 1728 insertions(+), 160 deletions(-)
 create mode 100644 Documentation/mm/numa_private_nodes.rst
 create mode 100644 include/linux/node_private.h
 create mode 100644 tools/testing/selftests/kvm/guest_memfd_bind_node_test.c

Thanks,
Balbir

author maintainer
#78 lore raw
Quoted 22 lines · Balbir Singh
Quoted 13 lines · Gregory Price

Patch Layout
============
The series is broken into two sections:

1) N_MEMORY_PRIVATE Introduction.
Introduce the node state.
Opt those nodes out of mm/ services.

2) NODE_PRIVATE_CAP_* features
A set of mm/ service opt-in flags that augment private nodes to make them more useful (i.e. reclaim = overcommit).

Quoted 7 lines

NODE_PRIVATE_CAP_LTPIN for private node folio pinning NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes NODE_PRIVATE_CAP_USER_NUMA for userland numa controls NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim

Quoted 2 lines

Looks reasonable, I wonder why USER_NUMA/HOTUNPUG is an opt-in?

Ideologically: Private nodes default to fully isolated, why should any given feature be special?

Concretely:

User-numa

Some devices don't want the user to have control over placement. I have been working on compressed memory, for example, which only ever wants to be used as a reclaim-demotion target.

(The reasoning for this is another thread, i plan on publishing
my research on this this year)

Hot-unplug:

Some devices can't necessarily handle unexpected migration, and hot-unplug is fundamentally a migration. So the HOTUNPLUG cap actually means "hot-unplug can execute migrations".

If the entire device has pre-drained the memory (all memory is free) then unplug works - it's just not very hot (no migrations) :]

Maybe a naming issue?

~Gregory

#79 lore raw
Quoted 31 lines · Gregory Price
Quoted 22 lines · Balbir Singh
Quoted 13 lines · Gregory Price

Patch Layout
============
The series is broken into two sections:

1) N_MEMORY_PRIVATE Introduction.
Introduce the node state.
Opt those nodes out of mm/ services.

2) NODE_PRIVATE_CAP_* features
A set of mm/ service opt-in flags that augment private nodes to make them more useful (i.e. reclaim = overcommit).

Quoted 7 lines

NODE_PRIVATE_CAP_LTPIN for private node folio pinning NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes NODE_PRIVATE_CAP_USER_NUMA for userland numa controls NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim

Quoted 2 lines

Looks reasonable, I wonder why USER_NUMA/HOTUNPUG is an opt-in?

Ideologically: Private nodes default to fully isolated, why should any given feature be special?

Concretely:

Quoted 9 lines

User-numa

Some devices don't want the user to have control over placement. I have been working on compressed memory, for example, which only ever wants to be used as a reclaim-demotion target.

(The reasoning for this is another thread, i plan on publishing
my research on this this year)

I don't fully understand, how do we allocate memory on these devices then? Is the driver expected to allocate and map via vm_insert_page()?

Quoted 11 lines

Hot-unplug:

Some devices can't necessarily handle unexpected migration, and hot-unplug is fundamentally a migration. So the HOTUNPLUG cap actually means "hot-unplug can execute migrations".

If the entire device has pre-drained the memory (all memory is free) then unplug works - it's just not very hot (no migrations) :]

Maybe a naming issue?

Yes, I would prefer MIGRATION in the name, HOTUNPLUG made me wonder how these devices come online and because it is a device, it can go offline while the system is still online

Quoted 2 lines

~Gregory

Balbir

author maintainer
#80 lore raw
Quoted 8 lines · Balbir Singh
Quoted 7 lines · Gregory Price

User-numa

Some devices don't want the user to have control over placement. I have been working on compressed memory, for example, which only ever wants to be used as a reclaim-demotion target.

Quoted 7 lines
Quoted 3 lines

(The reasoning for this is another thread, i plan on publishing
my research on this this year)

I don't fully understand, how do we allocate memory on these devices then? Is the driver expected to allocate and map via vm_insert_page()?

It depends on your use case, but yes that's one possibility.

In another use case, you could enable (CAP_RECLAIM | CAP_DEMOTION) and pages are allocated via alloc_demotion_target() in the reclaim path.

For general "device hosted memory", you can do a variety of things:

Direct alloc
ioctl(...) -> explicit alloc call
mmap(/dev/my_device) + mm_fault handler

Or a kernel-internal mempolicy
ioctl(...) -> set kernel-internal mempolicy on a vma mmap(/dev/my_device) + kernel-internal only mempolicy on the vma

The KVM guest_memfd patch included at the end of the series uses the kernel-internal mempolicy mechanism:
https://lore.kernel.org/linux-mm/al-pkvmgIxGu3LzM@gourry-fedora-PF4VCD3F/T/#me04a7e3556677babd82bfe3947b1a83b7f0017e1

This pattern is needed because the memory is never mapped into userland, so userland actually *can't* set a mempolicy on this memory itself.

and then of course: CAP_USER_NUMA + mbind/set_mempolicy directly

This is kind of the point - the source of the memory has some control over how it can be used.

Quoted 9 lines
Quoted 9 lines

Hot-unplug:

Some devices can't necessarily handle unexpected migration, and hot-unplug is fundamentally a migration. So the HOTUNPLUG cap actually means "hot-unplug can execute migrations".

If the entire device has pre-drained the memory (all memory is free) then unplug works - it's just not very hot (no migrations) :]

Quoted 7 lines
Quoted 2 lines

Maybe a naming issue?

Yes, I would prefer MIGRATION in the name, HOTUNPLUG made me wonder how these devices come online and because it is a device, it can go offline while the system is still online

There's a difference between device hotplug and memory hotplug. The kernel's memory hotunplug system can at best be described as "best effort", and if hotplugged as ZONE_NORMAL (on a non-private node) you're unlikely to ever be able to hotunplug.

It's possible that this CAP bit should just go away and if the device doesn't want its memory to be run migratable then it needs to hold extra references on the folios. That's probably reasonable and looks a lot like a long term pin.

(note: I have to respin for Sashiko fixes, i'm likely going to drop most of the CAP bits except USER_NUMA in v6 and have them come in with specific use cases).

~Gregory

#81 lore raw
In reply to Gregory Price
Quoted 2 lines · Gregory Price

KVM: guest_memfd: bind backing memory to a NUMA node at creation
KVM: selftests: add a guest_memfd FLAG_BIND_NODE test

I can't find the last patch in my inbox or on the lore. Not sure what went wrong.

Thanks,
Zenghui

author maintainer
#82 lore raw
Quoted 9 lines · Zenghui Yu
Quoted 2 lines · Gregory Price

KVM: guest_memfd: bind backing memory to a NUMA node at creation
KVM: selftests: add a guest_memfd FLAG_BIND_NODE test

I can't find the last patch in my inbox or on the lore. Not sure what went wrong.

Thanks,
Zenghui

Apologies, my email account got rate-limited to the number of recipients.

It should be in everyone's box shortly.

~Gregory

author maintainer
#83 lore raw

PATCH v5 36/36 KVM: selftests: add a guest_memfd FLAG_BIND_NODE test

In reply to Gregory Price

Add a standalone test for GUEST_MEMFD_FLAG_BIND_NODE.

It creates a guest_memfd bound to a given node, mmaps and faults it from the host, and checks with move_pages(2) that every folio landed on that node - with no userspace mbind().

It also covers the rejection paths with -EINVAL
- a nonzero pad
- an out-of-range node
- a node supplied without the flag

An unbindable (memoryless/offline) node is rejected at create time.

It uses a default (non-protected) VM so the host can fault the fd to observe placement, and needs CONFIG_NUMA.

Assisted-by Claude:claude-opus-4-8
---
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../kvm/guest_memfd_bind_node_test.c          | 213 ++++++++++++++++++
 2 files changed, 215 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/guest_memfd_bind_node_test.c
tools/testing/selftests/kvm/Makefile.kvm +2 -0
index d28a057fa6c2d..f0ddb5cc39db9 100644
@@ -155,6 +155,7 @@ TEST_GEN_PROGS_x86 += access_tracking_perf_test
 TEST_GEN_PROGS_x86 += coalesced_io_test
 TEST_GEN_PROGS_x86 += dirty_log_perf_test
 TEST_GEN_PROGS_x86 += guest_memfd_test
+TEST_GEN_PROGS_x86 += guest_memfd_bind_node_test
 TEST_GEN_PROGS_x86 += hardware_disable_test
 TEST_GEN_PROGS_x86 += mmu_stress_test
 TEST_GEN_PROGS_x86 += rseq_test
@@ -194,6 +195,7 @@ TEST_GEN_PROGS_arm64 += coalesced_io_test
 TEST_GEN_PROGS_arm64 += dirty_log_perf_test
 TEST_GEN_PROGS_arm64 += get-reg-list
 TEST_GEN_PROGS_arm64 += guest_memfd_test
+TEST_GEN_PROGS_arm64 += guest_memfd_bind_node_test
 TEST_GEN_PROGS_arm64 += mmu_stress_test
 TEST_GEN_PROGS_arm64 += rseq_test
 TEST_GEN_PROGS_arm64 += steal_time
tools/testing/selftests/kvm/guest_memfd_bind_node_test.c new file +213 -0

213 lines not on this page · raw message

Signature · 1 line
2.53.0-Meta
#84 lore raw
In reply to Gregory Price
Quoted 5 lines · Gregory Price

This series introduces the concept of "Private Memory Nodes", which are opted into two basic functionalities by default:
- page allocation (mm/page_alloc.c)
- OOM killing

Quoted 9 lines

All other features of mm/ are opted out of managing these NUMA nodes and its memory. Then we add capability bits to allow node owners to opt back into those services (if supported).

NUMA-hosted memory is presently a most-privilege system (any memory on any node, except ZONE_DEVICE, is 100% fungible and accessible). We then slap restrictions on top of it (cpuset, mempolicy, ZONE type, page/folio flags, etc).

Hi Gregory,

I applied your series on mm-next and give a quick review, not thoroughly, still I have some questions below regarding the design.

Quoted 6 lines

The goal here is to flip that dynamic, isolate by default and then opt-in to specific services that the device says is safe.

Isolation at the NUMA/Zonelist layer provides a powerful mechanism for memory hosted on accelerators - re-use of the kernel mm/ code.

Quoted 10 lines

- Accelerators (GPUs) can use demotion, numactl, and reclaim.
- Special memory devices (Compressed RAM) with special access controls (promote-on-write) can have generic services written for them.
- Network devices with large memory regions intended for ring buffers can use the buddy and standard networking stack.
- Slow, disaggregated memory pools which aren't suitable as general purpose memory get cleaner interfaces (no need to re-write the buddy in userland, can use migration interface, etc).
- Per-workload dedicated memory nodes (disaggregated VM memory)

For accelerator part, take CXL Type-2 as example, it has its own protocol CXL.cache, CXL.mem which is the rule they need to obey during memory transaction, Adding more rules for them since they are NUMA node confused me, I wonder the reason ?

Because NUMA node, at least for me, representing topology/locality rather than something with ownership or capability or rules.

Quoted 6 lines

And more use cases I have collected over the past few years.

Not included here is a dax-extension [1] that exposes all the internal bits as userland controls for testing - along with a pile of selftests that prove correctness.

Quoted 11 lines

Changes Since V4
================
- Massive reduction in complexity.
- no ops struct
- no callback functions
- no __GFP_PRIVATE
- no __GFP_THISNODE requirement
- no task flags (no PF_MEMALLOC_* in the alloc path)
- isolation via a dedicated zonelist:
- private nodes are omitted from FALLBACK/NOFALLBACK
- added ZONELIST_PRIVATE(_NOFALLBACK)

I saw the reply in patch 5, so in fact there's not only one dedicate zonelist, but numerous ?
I raise the question because zonelist was supposed to be a global, unbypasssable thing in MM design, but now what you are trying to do is to seperate the whole global list into several parts ?

Can you explain why in current design you don't consider to support something like ZONELIST_PRIVATE[n]={0, .. ,n-1} ? that's my imagination of what a global zonelist should look like, no matter private or non.

Quoted 33 lines

- ALLOC_ZONELIST_PRIVATE alloc_flag
- on top of Brendan Jackman's recent mm/page_alloc.h work [3]
- Zonelist selection rides the allocator's alloc_flags
- rename OPS -> CAPS (capabilities)
- split base functionality (isolation) from opt-ins (CAPS)
- first half of series can be merged without CAPS
- dropped compressed ram example from series
- will submit separately if this moves forward
- Added KVM as first primary in-tree user (mempolicy / CAP_USER_NUMA)
- fully functional dax-kmem extension and huge suite of selftests located at my github, to be discussed separately [1]

Patch Layout
============
The series is broken into two sections:

1) N_MEMORY_PRIVATE Introduction.
Introduce the node state.
Opt those nodes out of mm/ services.

2) NODE_PRIVATE_CAP_* features
A set of mm/ service opt-in flags that augment private nodes to make them more useful (i.e. reclaim = overcommit).

NODE_PRIVATE_CAP_LTPIN for private node folio pinning NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes NODE_PRIVATE_CAP_USER_NUMA for userland numa controls NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim

My hope is to merge at least #1 pulled ahead while #2 is debated.

Quoted 8 lines

Allocation Isolation
====================
page_alloc presently controls whether a node's memory can be allocated on a given call by 4 things (in order of authority)

1) ZONELIST membership
If a node is not in the walked zonelist, it's unreachable.

So a device gets hotplugged in the system will get a dedicated zonelist here ? And make sure it obeys the device's own protocol if it has one ?

Quoted 7 lines

2) __GFP_THISNODE
If this flag is set and the only node in the ZONELIST is the singular preferred node (or the local node, for -1)

3) cpuset.mems membership
cpuset trims any node in its allowed list

Sounds sane to me.

Quoted 25 lines

4) mempolicy nodemask
the allocator will skip any node not in the nodemask.

Except for #1 (Zonelist membership) there are all kinds of weird corner conditions in which 2-4 can be completely ignored (interrupt context, empty set because cpuset doesn't intersect mempolicy, shared vma, ...)

But ZONELIST membership is *absolute*. If a zone is not in the zonelist being walked, IT CANNOT BE ALLOCATED FROM. PERIOD.

The existing zonelists are constructed like so:
ZONELIST_FALLBACK : All N_MEMORY nodes
ZONELIST_NOFALLBACK : A singleton N_MEMORY node

Private node isolation is implemented via ZONELIST isolation:
ZONELIST_PRIVATE : The private node + N_MEMORY
ZONELIST_PRIVATE_NOFALLBACK : The private node alone.

(mirrors exactly the fallback/nofallback for __GFP_THISNODE)

Private nodes:
1) Never appear in any ZONELIST_FALLBACK
2) Have an empty ZONELIST_NOFALLBACK
3) Only appear in their own ZONELIST_PRIVATE(_NOFALLBACK)

Quoted 8 lines

1 & 2 mean all existing in-tree callers to page_alloc can NEVER accidentally allocate from a private node.

An allocation must explicitly ask via a zonelist and a nodemask.

alloc_flags |= ALLOC_ZONELIST_PRIVATE; /* use ZONELIST_PRIVATE */
__alloc_pages(..., nodemask); /* with the private node set */

As I stated above, NUMA concept was quite naive at first glance for my limited knowledge.

This is quite alot to add for NUMA node concept, I'll want to see more explanation in v6 and learn from it, thanks.

Quoted 47 lines

The page allocator keeps all its original interfaces which only ever touch the default zonelists - avoiding churn.

Making it accessible via Mempolicy: MPOL_F_PRIVATE and page_alloc
=================================================================
The vast majority of the kernel will never need to know about ZONELIST_PRIVATE, because we add MPOL_F_PRIVATE to mempolicy.

When a mempolicy has MPOL_F_PRIVATE, the alloc_mpol() interfaces do the zonelist selection for the source of the allocation.

That really is the whole explanation of the mechanism:

  alloc_flags = mpol_alloc_flags(pol);
  page = __alloc_frozen_pages_noprof(..., alloc_flags);

On mm-new this rides the allocator's existing alloc_flags plumbing:
ALLOC_ZONELIST_PRIVATE is just another alloc_flag, so no new parameter, enum, or alloc_context change is required.

For modules that want to implement their own special handling, they get the _private variants for the page allocator. This lets modules re-use the buddy instead of rewriting it.

- alloc_pages_node_private_noprof()
- folio_alloc_node_private_noprof()

This keeps ALLOC_ flags mm/ internal (these functions add the flags).

Isolating private node folios from kernel services
==================================================
We implement filter points in mm/ to prevent operations on private node memory. Where possible, we even re-use existing filter points from ZONE_DEVICE.

Most filter points are one or two lines of code:

Combining ZONE_DEVICE and N_MEMORY_PRIVATE opt-out spots:
- if (folio_is_zone_device(folio))
+ if (unlikely(folio_is_private_managed(folio)))

Disabling a service:
+ if (!node_is_private(nid)) {
+ kswapd_run(nid);
+ kcompactd_run(nid);
+ }

Quoted 4 lines

Disallowing a uapi interaction:
+ if (node_state(nid, N_MEMORY_PRIVATE))
+ return -EINVAL;

I'm not sure of why do we re-implement more filter and basically doing the same thing ? Any unavoidable scenario ?

re-using the exisintg filter would be nice if that's possible.

Quoted 194 lines

In the second half of the series, we replace blanket N_MEMORY_PRIVATE filters with NODE_PRIVATE_CAP_* filters to opt those nodes into that interaction if CAP is set.

We abstract this with a nice clean interface to make it really clear what is happening (nodes have features!)

- if (node_state(pgdat->node_id, N_MEMORY_PRIVATE))
+ if (!node_allows_reclaim(pgdat->node_id))

NODE_PRIVATE_CAP_* features
===========================
This series of commits opts private nodes into various mm/ services.

Capabilities:

  NODE_PRIVATE_CAP_RECLAIM       - direct and kswapd reclaim
  NODE_PRIVATE_CAP_USER_NUMA     - userland numa controls
  NODE_PRIVATE_CAP_DEMOTION      - node is a demotion target
  NODE_PRIVATE_CAP_HOTUNPLUG     - hotunplug may migrate

NODE_PRIVATE_CAP_NUMA_BALANCING - NUMAB may target node folios
NODE_PRIVATE_CAP_LTPIN - Longterm pin operates normally

Some opt-in support is more intensive than others, so these features are broken out in a way that we can defer them as future work streams.

NODE_PRIVATE_CAP_RECLAIM:
Enabling reclaim for these nodes is actually surprisingly trivial.

Without CAP_RECLAIM, when an allocation failure occurs, the system will not attempt to swap the memory - and instead will OOM (typically whatever task is using the most memory on *that* private node).

This capability consists of:
1) enabling kswapd and kcompactd for that node at hotplug time.
2) formalizing opt-out hooks to node_allows_reclaim() opt-in hooks.
3) Sets normal watermarks for these nodes.
4) Allow madvise operations on that node (PAGEOUT).
5) A small tweak to how LRU decides which zones to visit.

NODE_PRIVATE_CAP_USER_NUMA
Enables the following userland interfaces to accept the node:

    mbind()
    set_mempolicy()
    set_mempolicy_home_node()
    move_pages()
    migrate_pages()
  example:
     buf = mmap(..., MAP_ANON);
     mbind(buf, ..., {private_node});
     buf[0] = 0xDEADBEEF; /* Page faults onto the private node */

Later - the KVM example shows how in-kernel mempolicies can also be bound by CAP_USER_NUMA.

Otherwise, that's it - it's just a mempolicy with MPOL_F_PRIVATE.

NODE_PRIVATE_CAP_HOTUNPLUG
This is simple: allow hotunplug to migrate this nodes folios.

Some devices may not be able to tolerate unexpected migrations, so we prevent hotunplug from engaging in migration by default.

Some devices may have an mmu_notifier in their driver that can manage the migration and subsequent refault.

CAP_HOTUNPLUG allows memory_hotplug.c to migrate normally.

NODE_PRIVATE_CAP_DEMOTION
This adds the private node as a valid demotion target, and allows reclaim to demote memory from a private node to a demotion target.

Requires: NODE_PRIVATE_CAP_RECLAIM

NODE_PRIVATE_CAP_NUMA_BALANCING
This enables numa balancing to inject prot_none on private node folio mappings and promote them when faults are taken.

NODE_PRIVATE_CAP_LTPIN
This allows GUP Longterm Pinning to operate normally.

Normally, longterm pinning determines folio eligibility based on its ZONE_* membership (among other things).

ZONE_NORMAL is eligible, while ZONE_MOVABLE folios require migration to ZONE_NORMAL before pinning.

Neither operation is preferable by default on a private node, so the base behavior of FOLL_LONGTERM is to FAIL.

This capability allows longterm pinning to operate normally based on the ZONE membership. Private node memory may be hotplugged as either ZONE_NORMAL or ZONE_MOVABLE.

In-tree User: KVM
=================
Dave Jiang proposed [2] dax-backed guest_memfd() memory as a way of enabling disaggregated memory pools to host dedicated KVM memory.

With private nodes, this is trivial (with a bit of basic plumbing):

static int kvm_gmem_bind_node(struct inode *inode, int node) {
...
/* Bind to a private node - gated on CAP_USER_NUMA */ pol = mpol_bind_node(node);
if (IS_ERR(pol))

    return PTR_ERR(pol);

/* Set the shared policy */
err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, ..., pol);
...
}

KVM doesn't even need to know about private nodes at all, all it does is ask mempolicy whether the requested node is a valid bind.

mm/ component testing with dax driver
=====================================
The dax driver extensions[1] implements a simple interface to create a private node from a dax device created by any source.

I left the dax driver extensions out of this feature set because it locks in the CAP_ bits before anyone has input. It's there primarily for testing at this point.

The simplest way to get a dax device is with the memmap= boot arg.
e.g.: "memmap=0x40000000!0x140000000"

The dax driver extension has the following sysfs entries:

  dax0.0/private         - set the node to private
  dax0.0/dax_file        - make /dev/dax0.0 mmap'able in kmem mode
  dax0.0/adistance       - dictate memory_tierN membership
  dax0.0/reclaim         - CAP_RECLAIM
  dax0.0/demotion        - CAP_DEMOTION
  dax0.0/user_numa       - CAP_USER_NUMA
  dax0.0/hotunplug       - CAP_HOTUNPLUG
  dax0.0/numa_balancing  - CAP_NUMA_BALANCING
  dax0.0/ltpin           - CAP_LTPIN

Now consider the following...

Single node reclaim + mbind support:
echo 1 > dax0.0/private
echo 1 > dax0.0/reclaim
echo 1 > dax0.0/user_numa
echo online_movable > dax0.0/state

Test program:

   /* node1: 1GB Private Memory Node, 4GB swap */
   buf = mmap(NULL, TWO_GB, PROT_READ | PROT_WRITE,
              MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
   sys_mbind(p, len, MPOL_BIND, mask, MAXNODE, MPOL_MF_STRICT);
   memset(buf, 0xff, TWO_GB);

We can *guarantee* the ONLY reclaiming tasks are exactly:
- kswapdN (in theory we can even make this optional!)
- the memset task faulting pages in

This also means these are the only tasks capable of becoming locked up and OOMing (as long as it is not under pressure).

The rest of the system remains entirely functional for debugging.

It now becomes possible to micro-benchmark and A/B test reclaim changes with different scenarios (number of tasks, amount of memory, watermark targets, etc), because we have hard controls over exactly which tasks can access that node memory and how.

If we broke CAP_RECLAIM into subflags:
- CAP_RECLAIM_KSWAPD
- CAP_RECLAIM_DIRECT
- CAP_COMPACTION_KCOMPACTD
- CAP_COMPACTION_DIRECT

We can actually test the efficacy of each of these mechanisms in isolation to each other - something that is strictly impossible today.

Bonus Configuration: HBM device memory tiering
==============================================

echo 1 > dax0.0/private       # make the node private
echo 0 > dax0.0/adistance     # highest tier
echo 1 > dax0.0/reclaim       # reclaim active
echo 1 > dax0.0/demotion      # may demote from the node
echo 1 > dax0.0/user_numa     # mbind()

echo online_movable > dax0.0/state
echo 1 > numa/demotion_enabled

This is an HBM device which is treated as the top-tier in the system but for which memory can only enter via explicit mbind().

It can be overcommitted because it can be reclaimed (demotions go to CPU DRAM, and reclaim can swap from it).

Quoted 8 lines

If the HBM is managed by an accelerator (GPU), the mmu_notifier allows it to know when reclaim is moving memory out to do device-mmu invalidation prior to migration.

Prereqs, base commit, references
================================
akpm/mm-new - for Brendan Jackman's mm/page_alloc.h work[3]

Still thanks for the work, I learn alot from your work as well, thanks.

Best regards,
Richard Cheng.

Quoted 102 lines

Prereqs (all already in akpm/mm-new; listed for out-of-tree application):

page_alloc.h split + alloc_flags plumbing this series rides on:
commit e81fae43cd69 ("mm: split out internal page_alloc.h") commit b4ff3b6d0a1d ("mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG")

dax atomic whole-device hotplug (used by the dax extension [1]):
commit d7aa81b9a919 ("mm/memory: add memory_block_aligned_range() helper") commit 3b2f402a1754 ("dax/kmem: add sysfs interface for atomic whole-device hotplug")

[1] https://github.com/gourryinverse/linux/tree/scratch/gourry/managed_nodes/dax_private-mm-new
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

base-commit: c872b70f5d6c742ad34b8e838c92af81c8920b3e

Gregory Price (36):
mm: refactor find_next_best_node to find_next_best_node_in mm/page_alloc: refactor build_node_zonelist() out of build_zonelists() mm/page_alloc: let the bulk and folio allocators carry alloc_flags
numa: introduce N_MEMORY_PRIVATE
mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
cpuset: exclude private nodes from cpuset.mems (default-open) mm/memory_hotplug: disallow migration-driven private node hotunplug mm/mempolicy: skip private node folios when queueing for migration mm/migrate: disallow userland driven migration for private nodes mm/madvise: disallow madvise operations on private node folios mm/compaction: disallow compaction on private nodes mm/page_alloc: clear private node watermarks and system reserves mm/mempolicy: disallow NUMA Balancing prot_none on private nodes mm/damon: skip private node memory in DAMON migration and pageout mm/ksm: skip KSM for managed-memory folios
mm/khugepaged: skip private node folios when trying to collapse. mm/vmscan: disallow reclaim of private node memory
mm/gup: disallow longterm pin of private node folios
proc: include N_MEMORY_PRIVATE nodes in numa_maps output mm/memcontrol: account private-node memory in per-node stats proc/kcore: include private-node RAM in the kcore RAM map mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection mm/mempolicy: apply policy at the kernel zone for private-node binds mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug
mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA
balancing
mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning mm/khugepaged: base private node collapse eligiblity on actor/cap bits Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes mm/mempolicy: add mpol_set_shared_policy_range()
KVM: guest_memfd: bind backing memory to a NUMA node at creation
KVM: selftests: add a guest_memfd FLAG_BIND_NODE test

 Documentation/ABI/stable/sysfs-devices-node   |  10 +
 Documentation/mm/index.rst                    |   1 +
 Documentation/mm/numa_private_nodes.rst       | 160 ++++++++++
 drivers/base/node.c                           | 118 +++++++
 drivers/dax/kmem.c                            |   2 +-
 fs/proc/kcore.c                               |   5 +-
 fs/proc/task_mmu.c                            |  10 +-
 include/linux/gfp.h                           |  22 +-
 include/linux/kvm_host.h                      |   3 +
 include/linux/memory_hotplug.h                |   5 +-
 include/linux/mempolicy.h                     |  16 +
 include/linux/mmzone.h                        |  26 +-
 include/linux/node_private.h                  | 262 ++++++++++++++++
 include/linux/nodemask.h                      |   7 +-
 include/uapi/linux/kvm.h                      |   5 +-
 include/uapi/linux/mempolicy.h                |   1 +
 kernel/cgroup/cpuset.c                        |  26 +-
 mm/compaction.c                               |  13 +
 mm/damon/paddr.c                              |   9 +
 mm/gup.c                                      |  28 +-
 mm/huge_memory.c                              |   5 +
 mm/internal.h                                 |  97 +++++-
 mm/khugepaged.c                               |  18 +-
 mm/ksm.c                                      |   8 +-
 mm/madvise.c                                  |   8 +-
 mm/memcontrol-v1.c                            |   8 +-
 mm/memcontrol.c                               |  13 +-
 mm/memory-tiers.c                             |  40 ++-
 mm/memory_hotplug.c                           | 119 +++++++-
 mm/mempolicy.c                                | 288 +++++++++++++++---
 mm/migrate.c                                  |  19 +-
 mm/mm_init.c                                  |   2 +-
 mm/page_alloc.c                               | 189 +++++++++---
 mm/page_alloc.h                               |  34 +++
 mm/vmscan.c                                   |  57 +++-
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../kvm/guest_memfd_bind_node_test.c          | 213 +++++++++++++
 virt/kvm/guest_memfd.c                        |  39 ++-
 38 files changed, 1728 insertions(+), 160 deletions(-)
 create mode 100644 Documentation/mm/numa_private_nodes.rst
 create mode 100644 include/linux/node_private.h
 create mode 100644 tools/testing/selftests/kvm/guest_memfd_bind_node_test.c
Signature · 1 line
2.53.0-Meta
author maintainer
#85 lore raw
Quoted 6 lines · Richard Cheng

On Mon, Jul 20, 2026 at 03:33:54PM +0800, Gregory Price wrote:
Hi Gregory,

I applied your series on mm-next and give a quick review, not thoroughly, still I have some questions below regarding the design.

Hi Richard,

Thank you for the read.

Just a heads up, this was based on mm-new because of some recent work on Brendan's page_alloc.h cleanup, but if you managed to get it applied then maybe that's made its way forward already.

Quoted 17 lines
Quoted 6 lines

The goal here is to flip that dynamic, isolate by default and then opt-in to specific services that the device says is safe.

Isolation at the NUMA/Zonelist layer provides a powerful mechanism for memory hosted on accelerators - re-use of the kernel mm/ code.

Quoted 10 lines

- Accelerators (GPUs) can use demotion, numactl, and reclaim.
- Special memory devices (Compressed RAM) with special access controls (promote-on-write) can have generic services written for them.
- Network devices with large memory regions intended for ring buffers can use the buddy and standard networking stack.
- Slow, disaggregated memory pools which aren't suitable as general purpose memory get cleaner interfaces (no need to re-write the buddy in userland, can use migration interface, etc).
- Per-workload dedicated memory nodes (disaggregated VM memory)

Quoted 4 lines

For accelerator part, take CXL Type-2 as example, it has its own protocol CXL.cache, CXL.mem which is the rule they need to obey during memory transaction, Adding more rules for them since they are NUMA node confused me, I wonder the reason ?

CXL protocols simply state how to do the memory transaction at a physical / transport level. It does not make any statement on how the operating systems are to make sense of these devices or what constructs / abstractions to build to actually manage the devices themselves.

This series is not necessarily attached to CXL in particular, you could just as easily carve out memory from the general pool onto a private node to ensure it only gets used for a particular use.

In fact, that is how I have been testing this with dax/kmem:
https://github.com/gourryinverse/linux/commit/f279e741d9c3643597525bfab916b29b95cb635b

Quoted 3 lines

Because NUMA node, at least for me, representing topology/locality rather than something with ownership or capability or rules.

The NUMA abstraction representing topology/locality is a construct we (the OS developers) have decided on historically - but there's nothing that dictates we can never create new useful abstractions with it.

Consider:
N_NORMAL_MEMORY, /* The node has regular memory */
N_HIGH_MEMORY, /* The node has regular or high memory */

These have nothing to do with topology or locality, they are node states that only have meaning in the context of linux mm/.

To make something clear - there is no *requirement* for any particular device to use this abstraction. It simply enables a cleaner way for devices carrying memory to re-utilize mm/ services while ensuring their memory does not silently get used under system pressure (or some vagrant in userspace doing numactl --interleave all).

Ignoring all the CAP bits entirely, if you just took the base series your driver could re-use the buddy allocator without any special logic AND have confidence that your driver is the only possible user of that memory (barring some truly obscene bug).

Quoted 11 lines
Quoted 3 lines

- isolation via a dedicated zonelist:
- private nodes are omitted from FALLBACK/NOFALLBACK
- added ZONELIST_PRIVATE(_NOFALLBACK)

I saw the reply in patch 5, so in fact there's not only one dedicate zonelist, but numerous ?
I raise the question because zonelist was supposed to be a global, unbypasssable thing in MM design, but now what you are trying to do is to seperate the whole global list into several parts ?

Quoted 4 lines

Can you explain why in current design you don't consider to support something like ZONELIST_PRIVATE[n]={0, .. ,n-1} ? that's my imagination of what a global zonelist should look like, no matter private or non.

First let me say that there's nothing that prevents us from doing this, and we *could* make this the default case - but this decision was intentional by me.

Having them all present in each-other's zonelists by default creates a number of implicit opt-ins that are unclear:

- any direct zonelist iterator now iterates all zones on all private nodes, even those nodes do not opt into the same services

e.g. zonelist iteration in reclaim that targets Private Node A would attempt to reclaim Private Node B as well. That would require and extra explicit filter.

I ask: Why do this? Just isolate in the zonelist, and if there is
a desire for intersections - make it explicit, not implicit.

- fallback allocations can now occur across private nodes, even if those nodes are intended for different purposes.

Obviously you can use nodemask to tighten the allocation target, but I use numactl --interleave --all as an example of a clear case where intersected nodelists may not give you the behavior you want.

So for consistency - everything including zonelists have full isolation. This way all interactions with a private node must be explicit - always.

This is actually one of the problems with ZONE_DEVICE - and you can see it in this patch set. Some of the hooks in mm/ for zone_device only apply to PTE cases, and are absent from PMD cases - only because PMD mappings in ZONE_DEVICE aren't supported.

That kind of implicit behavior is quite bad.

That said, future improvements could include something like:

for_reclaimable_zone(ZONELIST_PRIVATE) {}

where we loosen this isolation, and formalize a filter, but I would like to see the usecase for it first. Loosening the isolation defeats the entire purpose of the series, so there should be a strong reason to do so.

Quoted 10 lines
Quoted 8 lines

Allocation Isolation
====================
page_alloc presently controls whether a node's memory can be allocated on a given call by 4 things (in order of authority)

1) ZONELIST membership
If a node is not in the walked zonelist, it's unreachable.

So a device gets hotplugged in the system will get a dedicated zonelist here ?

Yes.

Quoted 2 lines

And make sure it obeys the device's own protocol if it has one ?

I'm not sure i follow this question, can you help me understand?

If by protocol you mean the CAP bits (opting into reclaim, demotion, etc), then yes. If you mean something else (CXL) then I think that's orthogonal and unrelated.

Quoted 14 lines
Quoted 5 lines

Private nodes:
1) Never appear in any ZONELIST_FALLBACK
2) Have an empty ZONELIST_NOFALLBACK
3) Only appear in their own ZONELIST_PRIVATE(_NOFALLBACK)

Quoted 8 lines

1 & 2 mean all existing in-tree callers to page_alloc can NEVER accidentally allocate from a private node.

An allocation must explicitly ask via a zonelist and a nodemask.

alloc_flags |= ALLOC_ZONELIST_PRIVATE; /* use ZONELIST_PRIVATE */
__alloc_pages(..., nodemask); /* with the private node set */

Quoted 6 lines

As I stated above, NUMA concept was quite naive at first glance for my limited knowledge.

This is quite alot to add for NUMA node concept, I'll want to see more explanation in v6 and learn from it, thanks.

Sure, I can expand on it. I think it's not as much to add as you think though - it's simply adding the concept of isolation to a NUMA node.

Some more explanation below, but if you think there is something i should explicit spell out in v6 cover, please let me know.

---

I agree that NUMA as a concept was best-effort for, comically enough, somewhat *Uniform* memory access - instead of *Non*-Uniform memory access.

The current abstraction quite nicely handles the case where all memory on the system is roughly of the same calibre (DDR4, DDR5, etc) and roughly for the same purpose (general system memory).

But it is quite incapable of handling truly heterogeneous memory systems (precious HBM attached to the CPU, GPU's with HBM over a coherent link, hardware-compressed memory expansion, network devices w/ memory, etc).

If you look at the history of ZONE_DEVICE, what it fundamentally does is slaps an isolation mechanism on top of NUMA nodes because the NUMA abstraction doesn't provide one.

The problem with that approach is now you have to reason about a node having both fungible and non-fungible memory. It creates the need for something like migrate_device.c when a properly isolated NUMA node could just use migrate.c directly (with a coherent link).

One example of what isolation on the node enables:

A Private Node can hotplug memory in ZONE_NORMAL - which means it can be GUP pinned. That means driver support for GPU direct storage is simply alloc_pages_node() + pin(). The driver doesn't have to worry about something like SLAB accidentally using that same memory.

All without having to rewrite a bunch of mm/ in a driver, and with having to do some kind of heroics with ZONE_DEVICE that causes even more special mm/ interactions.

That only comes from adding an isolation primitive to a NUMA node.

Quoted 23 lines
Quoted 18 lines

Isolating private node folios from kernel services
==================================================
We implement filter points in mm/ to prevent operations on private node memory. Where possible, we even re-use existing filter points from ZONE_DEVICE.

Most filter points are one or two lines of code:

Combining ZONE_DEVICE and N_MEMORY_PRIVATE opt-out spots:
- if (folio_is_zone_device(folio))
+ if (unlikely(folio_is_private_managed(folio)))

Disabling a service:
+ if (!node_is_private(nid)) {
+ kswapd_run(nid);
+ kcompactd_run(nid);
+ }

Quoted 4 lines

Disallowing a uapi interaction:
+ if (node_state(nid, N_MEMORY_PRIVATE))
+ return -EINVAL;

Quoted 5 lines

I'm not sure of why do we re-implement more filter and basically doing the same thing ? Any unavoidable scenario ?

re-using the exisintg filter would be nice if that's possible.

We re-use (combine) existing filters where possible.

We add new ones where ZONE_DEVICE did not implement filters due to some *implicit* filter already existing.

Two clear examples:
- reclaim does not target ZONE_DEVICE
- ZONE_DEVICE does not support PMD

Both cases result in private-node filters that otherwise would have to exist for ZONE_DEVICE (and if ZONE_DEVICE ever grows PMD support, those filters will have to be added).

Quoted 20 lines
Quoted 20 lines

Bonus Configuration: HBM device memory tiering
==============================================

echo 1 > dax0.0/private       # make the node private
echo 0 > dax0.0/adistance     # highest tier
echo 1 > dax0.0/reclaim       # reclaim active
echo 1 > dax0.0/demotion      # may demote from the node
echo 1 > dax0.0/user_numa     # mbind()

echo online_movable > dax0.0/state
echo 1 > numa/demotion_enabled

This is an HBM device which is treated as the top-tier in the system but for which memory can only enter via explicit mbind().

It can be overcommitted because it can be reclaimed (demotions go to CPU DRAM, and reclaim can swap from it).

If the HBM is managed by an accelerator (GPU), the mmu_notifier allows it to know when reclaim is moving memory out to do device-mmu invalidation prior to migration.

Quoted 7 lines
Quoted 4 lines

Prereqs, base commit, references
================================
akpm/mm-new - for Brendan Jackman's mm/page_alloc.h work[3]

Still thanks for the work, I learn alot from your work as well, thanks.

Of course, and thank you for reading.

If nothing else I hope this series helps folks understand the page allocator better (it certainly has helped me). If there is anything you think I can improve or better explain, I am happy to discuss.

I am planning a larger publication of all my research sometime in the future, but I think code is more impactful and useful - so I am prioritizing that for now :].

~Gregory

author maintainer
#86 lore raw
Quoted 9 lines · Gregory Price
Quoted 8 lines · Richard Cheng

I saw the reply in patch 5, so in fact there's not only one dedicate zonelist, but numerous ?
I raise the question because zonelist was supposed to be a global, unbypasssable thing in MM design, but now what you are trying to do is to seperate the whole global list into several parts ?

Quoted 9 lines
Quoted 4 lines

Can you explain why in current design you don't consider to support something like ZONELIST_PRIVATE[n]={0, .. ,n-1} ? that's my imagination of what a global zonelist should look like, no matter private or non.

First let me say that there's nothing that prevents us from doing this, and we *could* make this the default case - but this decision was intentional by me.

You caught me on something here - I am wrong about my own code.

I described to you how I originally implemented zonelist isolation, which I walked back when I was testing mempolicy and demotion.

The current code and the comments in the code are correct, we get the following zonelists.

N_MEMORY : 0,1
N_MEMORY_PRIVATE: 2,3

ZONELIST_PRIVATE[0]     = {0,1,2,3}
ZONELIST_PRIVATE[1]     = {1,0,2,3}
ZONELIST_PRIVATE[2]     = {2,0,1,3}
ZONELIST_PRIVATE[3]     = {3,0,1,2}

Meaning everything except for MPOL_F_RELATIVE_NODES works (because of cpuset isolation, but more on this in a moment).

So what I shipped here is what you are describing.

Looking back at my notes - I realized a few things that lead me to walk back that isolation:

1) mempolicy makes more sense this way.
2) reclaim/demotion is easier to implement without isolation.
3) The individual service checks and nodelists handle the rest.
4) It overally just makes more sense. If something breaks isolation it's because there's a bug, not because of a structural issue.

For example in demotion we do this:

demote_folio_list():

    /* Get demotion targets, find the best one, and demote */
    node_get_allowed_targets(pgdat, &allowed_mask);
    target_nid = next_demotion_node(pgdat->node_id, &allowed_mask);
    migrate_pages(demote_folios, alloc_demote_folio, ...);

Where the first step already filters on CAP_DEMOTION. The result is still clean - no special iterators needed.

What I realized from your questions here is that since i walked back the zonelist isolation, I think i can actually include any CAP_USER_NUMA node in cpuset.mems and allow parititioning fairly trivially.

I think this might be the last majority complexity falling out.

Thank you for the questions - this has helped.

~Gregory

#87 lore raw
In reply to Gregory Price
Quoted 38 lines · Gregory Price
Quoted 6 lines · Richard Cheng

On Mon, Jul 20, 2026 at 03:33:54PM +0800, Gregory Price wrote:
Hi Gregory,

I applied your series on mm-next and give a quick review, not thoroughly, still I have some questions below regarding the design.

Hi Richard,

Thank you for the read.

Just a heads up, this was based on mm-new because of some recent work on Brendan's page_alloc.h cleanup, but if you managed to get it applied then maybe that's made its way forward already.

Quoted 17 lines
Quoted 6 lines

The goal here is to flip that dynamic, isolate by default and then opt-in to specific services that the device says is safe.

Isolation at the NUMA/Zonelist layer provides a powerful mechanism for memory hosted on accelerators - re-use of the kernel mm/ code.

Quoted 10 lines

- Accelerators (GPUs) can use demotion, numactl, and reclaim.
- Special memory devices (Compressed RAM) with special access controls (promote-on-write) can have generic services written for them.
- Network devices with large memory regions intended for ring buffers can use the buddy and standard networking stack.
- Slow, disaggregated memory pools which aren't suitable as general purpose memory get cleaner interfaces (no need to re-write the buddy in userland, can use migration interface, etc).
- Per-workload dedicated memory nodes (disaggregated VM memory)

Quoted 4 lines

For accelerator part, take CXL Type-2 as example, it has its own protocol CXL.cache, CXL.mem which is the rule they need to obey during memory transaction, Adding more rules for them since they are NUMA node confused me, I wonder the reason ?

Quoted 5 lines

CXL protocols simply state how to do the memory transaction at a physical / transport level. It does not make any statement on how the operating systems are to make sense of these devices or what constructs / abstractions to build to actually manage the devices themselves.

Ahhh thanks this clears my question.
Then this abstraction makes sense.

Quoted 15 lines

This series is not necessarily attached to CXL in particular, you could just as easily carve out memory from the general pool onto a private node to ensure it only gets used for a particular use.

In fact, that is how I have been testing this with dax/kmem:
https://github.com/gourryinverse/linux/commit/f279e741d9c3643597525bfab916b29b95cb635b

Quoted 3 lines

Because NUMA node, at least for me, representing topology/locality rather than something with ownership or capability or rules.

The NUMA abstraction representing topology/locality is a construct we (the OS developers) have decided on historically - but there's nothing that dictates we can never create new useful abstractions with it.

Quoted 7 lines

Consider:
N_NORMAL_MEMORY, /* The node has regular memory */
N_HIGH_MEMORY, /* The node has regular or high memory */

These have nothing to do with topology or locality, they are node states that only have meaning in the context of linux mm/.

Ok, I see.

Quoted 38 lines

To make something clear - there is no *requirement* for any particular device to use this abstraction. It simply enables a cleaner way for devices carrying memory to re-utilize mm/ services while ensuring their memory does not silently get used under system pressure (or some vagrant in userspace doing numactl --interleave all).

Ignoring all the CAP bits entirely, if you just took the base series your driver could re-use the buddy allocator without any special logic AND have confidence that your driver is the only possible user of that memory (barring some truly obscene bug).

Quoted 11 lines
Quoted 3 lines

- isolation via a dedicated zonelist:
- private nodes are omitted from FALLBACK/NOFALLBACK
- added ZONELIST_PRIVATE(_NOFALLBACK)

I saw the reply in patch 5, so in fact there's not only one dedicate zonelist, but numerous ?
I raise the question because zonelist was supposed to be a global, unbypasssable thing in MM design, but now what you are trying to do is to seperate the whole global list into several parts ?

Quoted 4 lines

Can you explain why in current design you don't consider to support something like ZONELIST_PRIVATE[n]={0, .. ,n-1} ? that's my imagination of what a global zonelist should look like, no matter private or non.

First let me say that there's nothing that prevents us from doing this, and we *could* make this the default case - but this decision was intentional by me.

Having them all present in each-other's zonelists by default creates a number of implicit opt-ins that are unclear:

- any direct zonelist iterator now iterates all zones on all private nodes, even those nodes do not opt into the same services

Quoted 7 lines

e.g. zonelist iteration in reclaim that targets Private Node A would attempt to reclaim Private Node B as well. That would require and extra explicit filter.

I ask: Why do this? Just isolate in the zonelist, and if there is
a desire for intersections - make it explicit, not implicit.

Fair enough, thanks for the headup.

Quoted 11 lines

- fallback allocations can now occur across private nodes, even if those nodes are intended for different purposes.

Obviously you can use nodemask to tighten the allocation target, but I use numactl --interleave --all as an example of a clear case where intersected nodelists may not give you the behavior you want.

So for consistency - everything including zonelists have full isolation. This way all interactions with a private node must be explicit - always.

Good choice I think, avoiding those implicit opt-in assumption will make future developers more aware of things and do not just stumble into them.

Quoted 8 lines

This is actually one of the problems with ZONE_DEVICE - and you can see it in this patch set. Some of the hooks in mm/ for zone_device only apply to PTE cases, and are absent from PMD cases - only because PMD mappings in ZONE_DEVICE aren't supported.

That kind of implicit behavior is quite bad.

Quoted 8 lines

That said, future improvements could include something like:

for_reclaimable_zone(ZONELIST_PRIVATE) {}

where we loosen this isolation, and formalize a filter, but I would like to see the usecase for it first. Loosening the isolation defeats the entire purpose of the series, so there should be a strong reason to do so.

Totally agree.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

Allocation Isolation
====================
page_alloc presently controls whether a node's memory can be allocated on a given call by 4 things (in order of authority)

1) ZONELIST membership
If a node is not in the walked zonelist, it's unreachable.

Quoted 9 lines
Quoted 1 line

So a device gets hotplugged in the system will get a dedicated zonelist here ?

Yes.

Quoted 2 lines

And make sure it obeys the device's own protocol if it has one ?

I'm not sure i follow this question, can you help me understand?

Nevermind, you just answer this part above, thanks alot.

Quoted 144 lines

If by protocol you mean the CAP bits (opting into reclaim, demotion, etc), then yes. If you mean something else (CXL) then I think that's orthogonal and unrelated.

Quoted 14 lines
Quoted 5 lines

Private nodes:
1) Never appear in any ZONELIST_FALLBACK
2) Have an empty ZONELIST_NOFALLBACK
3) Only appear in their own ZONELIST_PRIVATE(_NOFALLBACK)

Quoted 8 lines

1 & 2 mean all existing in-tree callers to page_alloc can NEVER accidentally allocate from a private node.

An allocation must explicitly ask via a zonelist and a nodemask.

alloc_flags |= ALLOC_ZONELIST_PRIVATE; /* use ZONELIST_PRIVATE */
__alloc_pages(..., nodemask); /* with the private node set */

Quoted 6 lines

As I stated above, NUMA concept was quite naive at first glance for my limited knowledge.

This is quite alot to add for NUMA node concept, I'll want to see more explanation in v6 and learn from it, thanks.

Sure, I can expand on it. I think it's not as much to add as you think though - it's simply adding the concept of isolation to a NUMA node.

Some more explanation below, but if you think there is something i should explicit spell out in v6 cover, please let me know.

---

I agree that NUMA as a concept was best-effort for, comically enough, somewhat *Uniform* memory access - instead of *Non*-Uniform memory access.

The current abstraction quite nicely handles the case where all memory on the system is roughly of the same calibre (DDR4, DDR5, etc) and roughly for the same purpose (general system memory).

But it is quite incapable of handling truly heterogeneous memory systems (precious HBM attached to the CPU, GPU's with HBM over a coherent link, hardware-compressed memory expansion, network devices w/ memory, etc).

If you look at the history of ZONE_DEVICE, what it fundamentally does is slaps an isolation mechanism on top of NUMA nodes because the NUMA abstraction doesn't provide one.

The problem with that approach is now you have to reason about a node having both fungible and non-fungible memory. It creates the need for something like migrate_device.c when a properly isolated NUMA node could just use migrate.c directly (with a coherent link).

One example of what isolation on the node enables:

A Private Node can hotplug memory in ZONE_NORMAL - which means it can be GUP pinned. That means driver support for GPU direct storage is simply alloc_pages_node() + pin(). The driver doesn't have to worry about something like SLAB accidentally using that same memory.

All without having to rewrite a bunch of mm/ in a driver, and with having to do some kind of heroics with ZONE_DEVICE that causes even more special mm/ interactions.

That only comes from adding an isolation primitive to a NUMA node.

Quoted 23 lines
Quoted 18 lines

Isolating private node folios from kernel services
==================================================
We implement filter points in mm/ to prevent operations on private node memory. Where possible, we even re-use existing filter points from ZONE_DEVICE.

Most filter points are one or two lines of code:

Combining ZONE_DEVICE and N_MEMORY_PRIVATE opt-out spots:
- if (folio_is_zone_device(folio))
+ if (unlikely(folio_is_private_managed(folio)))

Disabling a service:
+ if (!node_is_private(nid)) {
+ kswapd_run(nid);
+ kcompactd_run(nid);
+ }

Quoted 4 lines

Disallowing a uapi interaction:
+ if (node_state(nid, N_MEMORY_PRIVATE))
+ return -EINVAL;

Quoted 5 lines

I'm not sure of why do we re-implement more filter and basically doing the same thing ? Any unavoidable scenario ?

re-using the exisintg filter would be nice if that's possible.

We re-use (combine) existing filters where possible.

We add new ones where ZONE_DEVICE did not implement filters due to some *implicit* filter already existing.

Two clear examples:
- reclaim does not target ZONE_DEVICE
- ZONE_DEVICE does not support PMD

Both cases result in private-node filters that otherwise would have to exist for ZONE_DEVICE (and if ZONE_DEVICE ever grows PMD support, those filters will have to be added).

Quoted 20 lines
Quoted 20 lines

Bonus Configuration: HBM device memory tiering
==============================================

echo 1 > dax0.0/private       # make the node private
echo 0 > dax0.0/adistance     # highest tier
echo 1 > dax0.0/reclaim       # reclaim active
echo 1 > dax0.0/demotion      # may demote from the node
echo 1 > dax0.0/user_numa     # mbind()

echo online_movable > dax0.0/state
echo 1 > numa/demotion_enabled

This is an HBM device which is treated as the top-tier in the system but for which memory can only enter via explicit mbind().

It can be overcommitted because it can be reclaimed (demotions go to CPU DRAM, and reclaim can swap from it).

If the HBM is managed by an accelerator (GPU), the mmu_notifier allows it to know when reclaim is moving memory out to do device-mmu invalidation prior to migration.

Quoted 7 lines
Quoted 4 lines

Prereqs, base commit, references
================================
akpm/mm-new - for Brendan Jackman's mm/page_alloc.h work[3]

Still thanks for the work, I learn alot from your work as well, thanks.

Of course, and thank you for reading.

If nothing else I hope this series helps folks understand the page allocator better (it certainly has helped me). If there is anything you think I can improve or better explain, I am happy to discuss.

Quoted 5 lines

I am planning a larger publication of all my research sometime in the future, but I think code is more impactful and useful - so I am prioritizing that for now :].

~Gregory

That would be helpful for me, thanks alot for all the explanation again !

--Richard

#88 lore raw
In reply to Gregory Price
Quoted 4 lines · Gregory Price

Changes Since V4
================
- dropped compressed ram example from series
- will submit separately if this moves forward

We intend to test this series on a compression capable CXL expander hardware. Since compressed ram example (cram) is not part of this series, how do you suggest to do that? Do you have a version of cram module compatible with this series?

~Arun

author maintainer
#89 lore raw
Quoted 6 lines · Arun George/Arun George
Quoted 4 lines · Gregory Price

Changes Since V4
================
- dropped compressed ram example from series
- will submit separately if this moves forward

Quoted 6 lines

We intend to test this series on a compression capable CXL expander hardware. Since compressed ram example (cram) is not part of this series, how do you suggest to do that? Do you have a version of cram module compatible with this series?

~Arun

Hi Arun,

I dropped it from the base series because i want to try to move forward with this without RFC. The compressed service itself will be a separate submission after I get decent signal on whether this will be accepted.

I plan to RFC the new setup for compressed ram this a bit later, but I will share my working branch with you for testing.

~Gregory

#90 lore raw

Hi Gregory,

Thanks for sharing the working branch.

Some of us at Samsung are testing the series (both v4 & v5) on a compression capable CXL expander and are observing encouraging results.

Good things first! We feel that the isolation using private nodes and the capability selection using NODE_PRIVATE_CAP_* work functionally well for the compressed memory use cases.

On improvement points, we observed much higher 'page_faults', 'allocation_stalls' etc. which contributed to the higher tail latencies in some tests. I hope these are already part of the optimization plans. The higher latencies might have resulted from the write_protection applied to the private node, I believe.

On the tools side, we used MLC and Taobench for the tests.

Note that our intention for this phase of experiments was to test only the 'private node' layer based isolation, and not the cram and below layers for the compressed memory. Therefore we did not enable/utilize the compression capability in the hardware for these set of experiments (enabling compression would require the cram level ballooning/memory shrinking and cxl level interfacing driver to the compression device). That would be different set of experiments where we would be testing the cram balloon shrinkers and our alternate algorithms (upstream targeted). So cram and below layers were used only for enumeration of private node regions for these tests and not for the run-time memory shrinkers.

======= Test Methodology ==============

We explored these 3 cases:

1) (Baseline – existing CXL infra): DRAM 16GB + CXL 32GB. Here we
used the existing CXL driver infra to enable the device. No private node is involved. And compression is disabled on device.

2) (private node in default write protected path): DRAM 16GB + CXL
Private Node 32GB. Here private node infra is used to enable the device. And compression is disabled on device.

3) (private node with no write protection): DRAM 16GB + CXL Private
Node 32GB. Here private node infra is used to enable the device without the write protection/fencing enabled. Compression is disabled on device. We could not complete these runs as they resulted in kernel panics. Guess the code path is not stable yet for this (We had hoped that case 1 and case 3 results would be similar). We also observed some unmovable page warnings logs in 'dmesg' before crash (might be related to the panic). Adding the log snippets at the end.

========= Results summary ===================

- TaoBench: ‘private node’ showed much higher latencies. We believe this could be due to the migration back into the DRAM for overwrites (due to write-protection) based on the kernel VM stats.

- MLC: For lower inject delay, the private node shows better latency; but when inject delay goes high, the baseline is better.

Overall kernel VM stats suggested much higher ‘page faults, ‘allocation stalls’ etc. in ‘private node’ case which is corroborated with the results.

========== Setup ======================

CPU: 144 Cores (single socket), DRAM: 16GB, CXL: 32GB,

Kernel: 7.2.0-rc2+ (Private Node), 6.18.32 (Baseline)

Private node opt CAPs:

#define CRAM_NP_CAPS    (NODE_PRIVATE_CAP_RECLAIM |
                         NODE_PRIVATE_CAP_HOTUNPLUG | \
                         NODE_PRIVATE_CAP_DEMOTION |
                         NODE_PRIVATE_CAP_USER_NUMA | \
                         NODE_PRIVATE_CAP_NUMA_BALANCING |
                         NODE_PRIVATE_POLICY_WRITE_FENCE)

Enabled 'numa balancing' and 'demotion'.

   $ echo 1 > /proc/sys/kernel/numa_balancing
   $ echo true > /sys/kernel/mm/numa/demotion_enabled

========== TaoBench Experiments ====================

CMD:

$ python3 benchpress_cli.py run tao_bench_standalone -i '{"bind_mem": 0, 
"bind_cpu": 0, "memsize": 32, "test_time": 300, "warmup_time": 0, 
"clients_per_thread": 100, "set_get_ratio": "1:9"}'

1. (baseline) DRAM 16GB + CXL 32GB ==>

$ grep -A8 "^ALL STATS" benchmark_metrics_27cfff0e/client_0.log

===================================================
Type Avg. Latency p50 p95 p99
----------------------------------------------------

Sets   2.75215     2.73500      4.57500     5.79100
Gets   2.84236     2.81500      4.76700     5.95100

2. (private node in default write protected path)

      DRAM 16GB + CXL Private Node(uncompressed) 32GB   ===>
$ grep -A8 "^ALL STATS" benchmark_metrics_5181fda5/client_0.log

===================================================
Type Avg. Latency p50 p95 p99
----------------------------------------------------

Sets   5.58395     4.73500      12.35100     21.75900
Gets   4.83344     4.19100      10.43100     17.91900

Vmstat metrics comparison of both cases for Taobench ($cat /proc/vmstat):

-------------------------------------------------------------------
Metric | Baseline | CXL Private Node | Ratio
-------------------------------------------------------------------
pgfault 37,121,808 171,512,616 4.6x

pgactivate          5,360,928       161,947,594         30.2x
pgreuse             4,346,396           407,397         10.7x less

pgrefill 88,775,854 436,486,518 4.9x
pgdemote_kswapd 5,933,439 129,586,750 21.8x
pgdemote_direct 24,837 35,505,114 1429x
allocstall_normal 469 153,600 327x

kswapd_low_wmark_hit_quickly 671         27,821         41.5x
pageoutrun              1,333            29,217         21.9x
numa_pte_updates   24,508,077                 0          na
numa_hint_faults   21,152,264                 0          na
numa_pages_migrated 3,330,897                 0          na
numa_hit           46,667,202       353,812,703          7.6x

---------- Takeaways -----

Private (CRAM) node performance is lower compared to a normal CXL memory allocation path. VM stats shows more page_faults and allocation_stalls on the private node case. Could it be the allocator waits for migration path to demote pages to private node? Or the actual hot pages in dram got demoted to private node to make space during overwrites? We will try further analysis on this.

========== MLC Experiments ====================

CMD:
$./mlc --loaded_latency -j0 -c0 -b1g -k1-15 -W5 -r

Experiment Results:

1. (Baseline) DRAM 16GB + CXL 32GB

Inject Latency Bandwidth
Delay (ns) MB/sec
================

  00000  686.18   40137.5
  00002  686.69   40124.0
  00008  709.74   40362.7
  00015  711.75   40350.0
  00050  710.72   40357.4
  00100  712.22   40316.0
  00200  695.29   40026.5
  00300  602.91   38845.3
  00400  482.28   36240.4
  00500  410.03   33606.6
  00700  275.13   26352.5
  01000  241.82   18851.7
  01300  230.91   14695.8
  01700  215.27   11403.2
  02500  217.76    7900.9
  03500  194.42    5787.0
  05000  192.17    4166.4
  09000  190.36    2475.0
  20000  188.09    1312.1

2. (private node in default write protected path) DRAM 16GB + CXL Private Node(Uncompressed) 32GB

Inject Latency Bandwidth
Delay (ns) MB/sec
================

  00000  320.09   12186.2
  00002  543.42   22726.1
  00008  482.11   28385.2
  00015  488.26   26304.6
  00050  485.19   26898.4
  00100  477.65   24668.9
  00200  483.68   21273.8
  00300  486.44   17655.9
  00400  480.41   16470.0
  00500  481.30   13589.9
  00700  493.84    9960.8
  01000  488.66    8267.7
  01300  491.31    6433.6
  01700  476.97    6211.2
  02500  472.97    5037.0
  03500  466.05    3533.8
  05000  446.44    2783.1
  09000  417.03    1877.5
  20000  387.22    1035.3

Vmstat metrics comparison of both cases for MLC
($cat /proc/vmstat):

-------------------------------------------------------------------
Metric | Baseline | CXL Private Node | Ratio
-------------------------------------------------------------------
pgfault 10,287,329 46540,782 4.6x

pgactivate           1141468        65,098,272          57x
pgrefill             260,699        47,592,751         182.5x
pgdemote_kswapd    1,085,783        26,063,754          24x
pgdemote_direct    4,562,539        15,809,485           3.5x
pgmigrate_fail        41,929         2,763,725          65.9x
allocstall_normal        161               425           2.6x
kswapd_low_wmark_hit_quickly  47           763          16.2x
pageoutrun                55               936          17x

Takeaway: Our MLC tests show a trade-off between the two configurations. During low inject delay, the private Node is faster. But when inject delay goes high, the baseline shows better latency. Could it be the TLB cache effects?

--------------------------------------------------------------------
Case 3 dmesg log snippet:

[   82.525741] page: refcount:1 mapcount:0 mapping:0000000000000000 
index:0x0 pfn:0x5f5800
[   82.525753] flags: 
0x17ffffc0002000(reserved|node=0|zone=2|lastcpupid=0x1fffff)
[   82.525762] raw: 0017ffffc0002000 ffefc85717d60008 ffefc85717d60008 
0000000000000000
[   82.525764] raw: 0000000000000000 0000000000000000 00000001ffffffff 
0000000000000000
[   82.525766] page dumped because: unmovable page

..............

We will update on the compression enabled experiments further.

~ Arun

Quoted 14 lines · Gregory Price
Quoted 7 lines · Arun George/Arun George

On 21-07-2026 01:03 am, Gregory Price wrote:
We intend to test this series on a compression capable CXL expander hardware. Since compressed ram example (cram) is not part of this series, how do you suggest to do that? Do you have a version of cram module compatible with this series?

~Arun

Hi Arun,

Quoted 1 line

I plan to RFC the new setup for compressed ram this a bit later, but I

will share my working branch with you for testing.

~Gregory

author maintainer
#91 lore raw
Quoted 7 lines · Arun George/Arun George

Hi Gregory,

Thanks for sharing the working branch.

Some of us at Samsung are testing the series (both v4 & v5) on a compression capable CXL expander and are observing encouraging results.

This is really cool, thank you so much for spending the cycles to test.

I'm working on getting a v6 out soon. Glad to see the progress!

Quoted 10 lines

Good things first! We feel that the isolation using private nodes and the capability selection using NODE_PRIVATE_CAP_* work functionally well for the compressed memory use cases.

On improvement points, we observed much higher 'page_faults', 'allocation_stalls' etc. which contributed to the higher tail latencies in some tests. I hope these are already part of the optimization plans. The higher latencies might have resulted from the write_protection applied to the private node, I believe.

Yes, this is expected, and I think there are questions about how it should be optimized both in a provisioning sense and a software sense.

For example, exposing a full memory expander as 100% compressible may not make sense - given that this could many a lot of memory. I have some data that shows it may make more sense to split a memory expander into two regions to better fit the compressible region to meet the actual capacity the cold-tail is capable of consuming.

On the software side there may be some ability to adjust this at runtime with hotplug instead of requiring a reboot, but I think that needs to be experimented with.

Quoted 2 lines

On the tools side, we used MLC and Taobench for the tests.

Great! I also have some TaoBench and fio results i'm looking to share, glad to have some additional peer review using Tao.

Quoted 11 lines

Note that our intention for this phase of experiments was to test only the 'private node' layer based isolation, and not the cram and below layers for the compressed memory. Therefore we did not enable/utilize the compression capability in the hardware for these set of experiments (enabling compression would require the cram level ballooning/memory shrinking and cxl level interfacing driver to the compression device). That would be different set of experiments where we would be testing the cram balloon shrinkers and our alternate algorithms (upstream targeted). So cram and below layers were used only for enumeration of private node regions for these tests and not for the run-time memory shrinkers.

Agreed, i'm looking forward to getting past the isolation bits and get more focused on the compression implementation details.

Quoted 12 lines

======= Test Methodology ==============

We explored these 3 cases:

1) (Baseline – existing CXL infra): DRAM 16GB + CXL 32GB. Here we
used the existing CXL driver infra to enable the device. No private node is involved. And compression is disabled on device.

2) (private node in default write protected path): DRAM 16GB + CXL
Private Node 32GB. Here private node infra is used to enable the device. And compression is disabled on device.

Quoted 9 lines

3) (private node with no write protection): DRAM 16GB + CXL Private
Node 32GB. Here private node infra is used to enable the device without the write protection/fencing enabled. Compression is disabled on device. We could not complete these runs as they resulted in kernel panics. Guess the code path is not stable yet for this (We had hoped that case 1 and case 3 results would be similar). We also observed some unmovable page warnings logs in 'dmesg' before crash (might be related to the panic). Adding the log snippets at the end.

Very similar to my test setup, seems like good signal we are of similar mind on the use case. (For readers: there was no coordination here).

I personally did not test private-node without write protection, although it makes sense to test the throughput of a demotion-only node as a baseline. Smart.

My guess is there was probably a reclaim throughput issue, or there may have been a bug in the v4/v5 code. I will look at adding some pressure tests to my suite that test demotion-only explicitly.

Quoted 9 lines

========= Results summary ===================

Private (CRAM) node performance is lower compared to a normal CXL memory allocation path. VM stats shows more page_faults and allocation_stalls on the private node case. Could it be the allocator waits for migration path to demote pages to private node? Or the actual hot pages in dram got demoted to private node to make space during overwrites? We will try further analysis on this.

- allocator waits for migration?

Yes. And worse, once the node is full, migration will fail and you'll swap directly from the top tier and reduce your reclaim behavior on the lower node. It becomes very important to push proactive reclaim on a demotion-only tier to ensure there is sufficient headroom to receive demotions in the future, since direct-reclaim basically never targets a lower-tier on the first pass.

There's a big discussion about whether tiered systems need to invert their reclaim behavior (reclaim from the lowest node first to find progress there, then do demotion - rather than the other way around).

- Hot pages in dram

Also yes. The comparison to make isn't only against a raw CXL node, but also against Zswap or Zram or Swap. How does the workload fair on a system with 16GB RAM and 32GB Zswap/Swap?

This is important context.

Quoted 15 lines

========== MLC Experiments ====================

CMD:

   $./mlc --loaded_latency -j0 -c0 -b1g -k1-15 -W5 -r

Experiment Results:

1. (Baseline) DRAM 16GB + CXL 32GB

Inject Latency Bandwidth
Delay (ns) MB/sec
================

  00000  686.18   40137.5
  00002  686.69   40124.0
  00008  709.74   40362.7

...

Quoted 9 lines

2. (private node in default write protected path) DRAM 16GB + CXL Private Node(Uncompressed) 32GB

Inject Latency Bandwidth
Delay (ns) MB/sec
================

  00000  320.09   12186.2
  00002  543.42   22726.1
  00008  482.11   28385.2

...

Quoted 5 lines

Takeaway: Our MLC tests show a trade-off between the two configurations. During low inject delay, the private Node is faster. But when inject delay goes high, the baseline shows better latency. Could it be the TLB cache effects?

These latency results suggest to me that the memory being tested was always DRAM, though the initial memory fault would have been CXL.

i.e. even if the memory was faulted directly onto the node via mempolicy, the very first write to it would have caused promotion.

If you dropped the write protection you'd probably see it equals the baseline.

The bandwidth numbers support this. Higher stall, but faster latency means the migration was more beneficial for latency than letting the memory sit on CXL.

Neat. I'm not sure these results are particularly useful in describing how a workload would react.

Quoted 12 lines

--------------------------------------------------------------------
Case 3 dmesg log snippet:

[   82.525741] page: refcount:1 mapcount:0 mapping:0000000000000000 
index:0x0 pfn:0x5f5800
[   82.525753] flags: 
0x17ffffc0002000(reserved|node=0|zone=2|lastcpupid=0x1fffff)
[   82.525762] raw: 0017ffffc0002000 ffefc85717d60008 ffefc85717d60008 
0000000000000000
[   82.525764] raw: 0000000000000000 0000000000000000 00000001ffffffff 
0000000000000000
[   82.525766] page dumped because: unmovable page
Quoted 2 lines

..............

Was this memory hotplugged via cram.c or did you replicate the code and hotplug it another way? If so, did you hotplug it as ZONE_NORMAL or ZONE_MOVABLE?

For a compressed tier, it must always be hotplugged as ZONE_MOVABLE, without exception, because otherwise it can allow a GUP pin or kernel allocation that would eventually become permanently stuck there - and by definition of a compression tier must be 100% movable memory.

~Gregory

#92 lore raw
In reply to Gregory Price

Hi Gregory,

Quoted 360 lines

This series introduces the concept of "Private Memory Nodes", which are opted into two basic functionalities by default:
- page allocation (mm/page_alloc.c)
- OOM killing

All other features of mm/ are opted out of managing these NUMA nodes and its memory. Then we add capability bits to allow node owners to opt back into those services (if supported).

NUMA-hosted memory is presently a most-privilege system (any memory on any node, except ZONE_DEVICE, is 100% fungible and accessible). We then slap restrictions on top of it (cpuset, mempolicy, ZONE type, page/folio flags, etc).

The goal here is to flip that dynamic, isolate by default and then opt-in to specific services that the device says is safe.

Isolation at the NUMA/Zonelist layer provides a powerful mechanism for memory hosted on accelerators - re-use of the kernel mm/ code.

- Accelerators (GPUs) can use demotion, numactl, and reclaim.
- Special memory devices (Compressed RAM) with special access controls (promote-on-write) can have generic services written for them.
- Network devices with large memory regions intended for ring buffers can use the buddy and standard networking stack.
- Slow, disaggregated memory pools which aren't suitable as general purpose memory get cleaner interfaces (no need to re-write the buddy in userland, can use migration interface, etc).
- Per-workload dedicated memory nodes (disaggregated VM memory)

And more use cases I have collected over the past few years.

Not included here is a dax-extension [1] that exposes all the internal bits as userland controls for testing - along with a pile of selftests that prove correctness.

Changes Since V4
================
- Massive reduction in complexity.
- no ops struct
- no callback functions
- no __GFP_PRIVATE
- no __GFP_THISNODE requirement
- no task flags (no PF_MEMALLOC_* in the alloc path)
- isolation via a dedicated zonelist:
- private nodes are omitted from FALLBACK/NOFALLBACK
- added ZONELIST_PRIVATE(_NOFALLBACK)
- ALLOC_ZONELIST_PRIVATE alloc_flag
- on top of Brendan Jackman's recent mm/page_alloc.h work [3]
- Zonelist selection rides the allocator's alloc_flags
- rename OPS -> CAPS (capabilities)
- split base functionality (isolation) from opt-ins (CAPS)
- first half of series can be merged without CAPS
- dropped compressed ram example from series
- will submit separately if this moves forward
- Added KVM as first primary in-tree user (mempolicy / CAP_USER_NUMA)
- fully functional dax-kmem extension and huge suite of selftests located at my github, to be discussed separately [1]

Patch Layout
============
The series is broken into two sections:

1) N_MEMORY_PRIVATE Introduction.
Introduce the node state.
Opt those nodes out of mm/ services.

2) NODE_PRIVATE_CAP_* features
A set of mm/ service opt-in flags that augment private nodes to make them more useful (i.e. reclaim = overcommit).

NODE_PRIVATE_CAP_LTPIN for private node folio pinning NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA balancing NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes NODE_PRIVATE_CAP_USER_NUMA for userland numa controls NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim

My hope is to merge at least #1 pulled ahead while #2 is debated.

Allocation Isolation
====================
page_alloc presently controls whether a node's memory can be allocated on a given call by 4 things (in order of authority)

1) ZONELIST membership
If a node is not in the walked zonelist, it's unreachable.

2) __GFP_THISNODE
If this flag is set and the only node in the ZONELIST is the singular preferred node (or the local node, for -1)

3) cpuset.mems membership
cpuset trims any node in its allowed list

4) mempolicy nodemask
the allocator will skip any node not in the nodemask.

Except for #1 (Zonelist membership) there are all kinds of weird corner conditions in which 2-4 can be completely ignored (interrupt context, empty set because cpuset doesn't intersect mempolicy, shared vma, ...)

But ZONELIST membership is *absolute*. If a zone is not in the zonelist being walked, IT CANNOT BE ALLOCATED FROM. PERIOD.

The existing zonelists are constructed like so:
ZONELIST_FALLBACK : All N_MEMORY nodes
ZONELIST_NOFALLBACK : A singleton N_MEMORY node

Private node isolation is implemented via ZONELIST isolation:
ZONELIST_PRIVATE : The private node + N_MEMORY
ZONELIST_PRIVATE_NOFALLBACK : The private node alone.

(mirrors exactly the fallback/nofallback for __GFP_THISNODE)

Private nodes:
1) Never appear in any ZONELIST_FALLBACK
2) Have an empty ZONELIST_NOFALLBACK
3) Only appear in their own ZONELIST_PRIVATE(_NOFALLBACK)

1 & 2 mean all existing in-tree callers to page_alloc can NEVER accidentally allocate from a private node.

An allocation must explicitly ask via a zonelist and a nodemask.

alloc_flags |= ALLOC_ZONELIST_PRIVATE; /* use ZONELIST_PRIVATE */
__alloc_pages(..., nodemask); /* with the private node set */

The page allocator keeps all its original interfaces which only ever touch the default zonelists - avoiding churn.

Making it accessible via Mempolicy: MPOL_F_PRIVATE and page_alloc
=================================================================
The vast majority of the kernel will never need to know about ZONELIST_PRIVATE, because we add MPOL_F_PRIVATE to mempolicy.

When a mempolicy has MPOL_F_PRIVATE, the alloc_mpol() interfaces do the zonelist selection for the source of the allocation.

That really is the whole explanation of the mechanism:

  alloc_flags = mpol_alloc_flags(pol);
  page = __alloc_frozen_pages_noprof(..., alloc_flags);

On mm-new this rides the allocator's existing alloc_flags plumbing:
ALLOC_ZONELIST_PRIVATE is just another alloc_flag, so no new parameter, enum, or alloc_context change is required.

For modules that want to implement their own special handling, they get the _private variants for the page allocator. This lets modules re-use the buddy instead of rewriting it.

- alloc_pages_node_private_noprof()
- folio_alloc_node_private_noprof()

This keeps ALLOC_ flags mm/ internal (these functions add the flags).

Isolating private node folios from kernel services
==================================================
We implement filter points in mm/ to prevent operations on private node memory. Where possible, we even re-use existing filter points from ZONE_DEVICE.

Most filter points are one or two lines of code:

Combining ZONE_DEVICE and N_MEMORY_PRIVATE opt-out spots:
- if (folio_is_zone_device(folio))
+ if (unlikely(folio_is_private_managed(folio)))

Disabling a service:
+ if (!node_is_private(nid)) {
+ kswapd_run(nid);
+ kcompactd_run(nid);
+ }

Disallowing a uapi interaction:
+ if (node_state(nid, N_MEMORY_PRIVATE))
+ return -EINVAL;

In the second half of the series, we replace blanket N_MEMORY_PRIVATE filters with NODE_PRIVATE_CAP_* filters to opt those nodes into that interaction if CAP is set.

We abstract this with a nice clean interface to make it really clear what is happening (nodes have features!)

- if (node_state(pgdat->node_id, N_MEMORY_PRIVATE))
+ if (!node_allows_reclaim(pgdat->node_id))

NODE_PRIVATE_CAP_* features
===========================
This series of commits opts private nodes into various mm/ services.

Capabilities:

  NODE_PRIVATE_CAP_RECLAIM       - direct and kswapd reclaim
  NODE_PRIVATE_CAP_USER_NUMA     - userland numa controls
  NODE_PRIVATE_CAP_DEMOTION      - node is a demotion target
  NODE_PRIVATE_CAP_HOTUNPLUG     - hotunplug may migrate

NODE_PRIVATE_CAP_NUMA_BALANCING - NUMAB may target node folios
NODE_PRIVATE_CAP_LTPIN - Longterm pin operates normally

Some opt-in support is more intensive than others, so these features are broken out in a way that we can defer them as future work streams.

NODE_PRIVATE_CAP_RECLAIM:
Enabling reclaim for these nodes is actually surprisingly trivial.

Without CAP_RECLAIM, when an allocation failure occurs, the system will not attempt to swap the memory - and instead will OOM (typically whatever task is using the most memory on *that* private node).

This capability consists of:
1) enabling kswapd and kcompactd for that node at hotplug time.
2) formalizing opt-out hooks to node_allows_reclaim() opt-in hooks.
3) Sets normal watermarks for these nodes.
4) Allow madvise operations on that node (PAGEOUT).
5) A small tweak to how LRU decides which zones to visit.

NODE_PRIVATE_CAP_USER_NUMA
Enables the following userland interfaces to accept the node:

    mbind()
    set_mempolicy()
    set_mempolicy_home_node()
    move_pages()
    migrate_pages()
  example:
     buf = mmap(..., MAP_ANON);
     mbind(buf, ..., {private_node});
     buf[0] = 0xDEADBEEF; /* Page faults onto the private node */

Later - the KVM example shows how in-kernel mempolicies can also be bound by CAP_USER_NUMA.

Otherwise, that's it - it's just a mempolicy with MPOL_F_PRIVATE.

NODE_PRIVATE_CAP_HOTUNPLUG
This is simple: allow hotunplug to migrate this nodes folios.

Some devices may not be able to tolerate unexpected migrations, so we prevent hotunplug from engaging in migration by default.

Some devices may have an mmu_notifier in their driver that can manage the migration and subsequent refault.

CAP_HOTUNPLUG allows memory_hotplug.c to migrate normally.

NODE_PRIVATE_CAP_DEMOTION
This adds the private node as a valid demotion target, and allows reclaim to demote memory from a private node to a demotion target.

Requires: NODE_PRIVATE_CAP_RECLAIM

NODE_PRIVATE_CAP_NUMA_BALANCING
This enables numa balancing to inject prot_none on private node folio mappings and promote them when faults are taken.

NODE_PRIVATE_CAP_LTPIN
This allows GUP Longterm Pinning to operate normally.

Normally, longterm pinning determines folio eligibility based on its ZONE_* membership (among other things).

ZONE_NORMAL is eligible, while ZONE_MOVABLE folios require migration to ZONE_NORMAL before pinning.

Neither operation is preferable by default on a private node, so the base behavior of FOLL_LONGTERM is to FAIL.

This capability allows longterm pinning to operate normally based on the ZONE membership. Private node memory may be hotplugged as either ZONE_NORMAL or ZONE_MOVABLE.

In-tree User: KVM
=================
Dave Jiang proposed [2] dax-backed guest_memfd() memory as a way of enabling disaggregated memory pools to host dedicated KVM memory.

With private nodes, this is trivial (with a bit of basic plumbing):

static int kvm_gmem_bind_node(struct inode *inode, int node) {
...
/* Bind to a private node - gated on CAP_USER_NUMA */ pol = mpol_bind_node(node);
if (IS_ERR(pol))

    return PTR_ERR(pol);

/* Set the shared policy */
err = mpol_set_shared_policy_range(&GMEM_I(inode)->policy, ..., pol);
...
}

KVM doesn't even need to know about private nodes at all, all it does is ask mempolicy whether the requested node is a valid bind.

mm/ component testing with dax driver
=====================================
The dax driver extensions[1] implements a simple interface to create a private node from a dax device created by any source.

I left the dax driver extensions out of this feature set because it locks in the CAP_ bits before anyone has input. It's there primarily for testing at this point.

The simplest way to get a dax device is with the memmap= boot arg.
e.g.: "memmap=0x40000000!0x140000000"

The dax driver extension has the following sysfs entries:

  dax0.0/private         - set the node to private
  dax0.0/dax_file        - make /dev/dax0.0 mmap'able in kmem mode
  dax0.0/adistance       - dictate memory_tierN membership
  dax0.0/reclaim         - CAP_RECLAIM
  dax0.0/demotion        - CAP_DEMOTION
  dax0.0/user_numa       - CAP_USER_NUMA
  dax0.0/hotunplug       - CAP_HOTUNPLUG
  dax0.0/numa_balancing  - CAP_NUMA_BALANCING
  dax0.0/ltpin           - CAP_LTPIN

Now consider the following...

Single node reclaim + mbind support:
echo 1 > dax0.0/private
echo 1 > dax0.0/reclaim
echo 1 > dax0.0/user_numa
echo online_movable > dax0.0/state

Test program:

   /* node1: 1GB Private Memory Node, 4GB swap */
   buf = mmap(NULL, TWO_GB, PROT_READ | PROT_WRITE,
              MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
   sys_mbind(p, len, MPOL_BIND, mask, MAXNODE, MPOL_MF_STRICT);
   memset(buf, 0xff, TWO_GB);

We can *guarantee* the ONLY reclaiming tasks are exactly:
- kswapdN (in theory we can even make this optional!)
- the memset task faulting pages in

This also means these are the only tasks capable of becoming locked up and OOMing (as long as it is not under pressure).

The rest of the system remains entirely functional for debugging.

It now becomes possible to micro-benchmark and A/B test reclaim changes with different scenarios (number of tasks, amount of memory, watermark targets, etc), because we have hard controls over exactly which tasks can access that node memory and how.

If we broke CAP_RECLAIM into subflags:
- CAP_RECLAIM_KSWAPD
- CAP_RECLAIM_DIRECT
- CAP_COMPACTION_KCOMPACTD
- CAP_COMPACTION_DIRECT

We can actually test the efficacy of each of these mechanisms in isolation to each other - something that is strictly impossible today.

Bonus Configuration: HBM device memory tiering
==============================================

Quoted 7 lines
echo 1 > dax0.0/private       # make the node private
echo 0 > dax0.0/adistance     # highest tier
echo 1 > dax0.0/reclaim       # reclaim active
echo 1 > dax0.0/demotion      # may demote from the node
echo 1 > dax0.0/user_numa     # mbind()

echo online_movable > dax0.0/state
echo 1 > numa/demotion_enabled

I am bit confused by the HBM device example being discovered as a DAX device. Is it to
only to showcase the use-case? Or we see HBM device is actually expected to be discovered as
DAX device with N_MEMORY_PRIVATE?

Thanks,
Pankaj

Quoted 117 lines

This is an HBM device which is treated as the top-tier in the system but for which memory can only enter via explicit mbind().

It can be overcommitted because it can be reclaimed (demotions go to CPU DRAM, and reclaim can swap from it).

If the HBM is managed by an accelerator (GPU), the mmu_notifier allows it to know when reclaim is moving memory out to do device-mmu invalidation prior to migration.

Prereqs, base commit, references
================================
akpm/mm-new - for Brendan Jackman's mm/page_alloc.h work[3]

Prereqs (all already in akpm/mm-new; listed for out-of-tree application):

page_alloc.h split + alloc_flags plumbing this series rides on:
commit e81fae43cd69 ("mm: split out internal page_alloc.h") commit b4ff3b6d0a1d ("mm: replace __GFP_NO_CODETAG with ALLOC_NO_CODETAG")

dax atomic whole-device hotplug (used by the dax extension [1]):
commit d7aa81b9a919 ("mm/memory: add memory_block_aligned_range() helper") commit 3b2f402a1754 ("dax/kmem: add sysfs interface for atomic whole-device hotplug")

[1] https://github.com/gourryinverse/linux/tree/scratch/gourry/managed_nodes/dax_private-mm-new
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/[email protected]/

base-commit: c872b70f5d6c742ad34b8e838c92af81c8920b3e

Gregory Price (36):
mm: refactor find_next_best_node to find_next_best_node_in mm/page_alloc: refactor build_node_zonelist() out of build_zonelists() mm/page_alloc: let the bulk and folio allocators carry alloc_flags
numa: introduce N_MEMORY_PRIVATE
mm: add ZONELIST_PRIVATE(_NOFALLBACK) for N_MEMORY_PRIVATE nodes.
cpuset: exclude private nodes from cpuset.mems (default-open) mm/memory_hotplug: disallow migration-driven private node hotunplug mm/mempolicy: skip private node folios when queueing for migration mm/migrate: disallow userland driven migration for private nodes mm/madvise: disallow madvise operations on private node folios mm/compaction: disallow compaction on private nodes mm/page_alloc: clear private node watermarks and system reserves mm/mempolicy: disallow NUMA Balancing prot_none on private nodes mm/damon: skip private node memory in DAMON migration and pageout mm/ksm: skip KSM for managed-memory folios
mm/khugepaged: skip private node folios when trying to collapse. mm/vmscan: disallow reclaim of private node memory
mm/gup: disallow longterm pin of private node folios
proc: include N_MEMORY_PRIVATE nodes in numa_maps output mm/memcontrol: account private-node memory in per-node stats proc/kcore: include private-node RAM in the kcore RAM map mm/mempolicy: add MPOL_F_PRIVATE and zonelist selection mm/mempolicy: apply policy at the kernel zone for private-node binds mm/mempolicy: add in-kernel MPOL_BIND interfaces for drivers/services mm/memory_hotplug: support N_MEMORY_PRIVATE node hotplug
mm: add NODE_PRIVATE_CAP_RECLAIM for opted-in private node reclaim
mm: add NODE_PRIVATE_CAP_USER_NUMA for userland numa controls
mm: add NODE_PRIVATE_CAP_HOTUNPLUG for opted-in private nodes
mm: add NODE_PRIVATE_CAP_DEMOTION for private-node tiering demotion
mm: add NODE_PRIVATE_CAP_NUMA_BALANCING for private-node NUMA
balancing
mm: add NODE_PRIVATE_CAP_LTPIN for private node folio pinning mm/khugepaged: base private node collapse eligiblity on actor/cap bits Documentation/mm: describe private (N_MEMORY_PRIVATE) memory nodes mm/mempolicy: add mpol_set_shared_policy_range()
KVM: guest_memfd: bind backing memory to a NUMA node at creation
KVM: selftests: add a guest_memfd FLAG_BIND_NODE test

 Documentation/ABI/stable/sysfs-devices-node   |  10 +
 Documentation/mm/index.rst                    |   1 +
 Documentation/mm/numa_private_nodes.rst       | 160 ++++++++++
 drivers/base/node.c                           | 118 +++++++
 drivers/dax/kmem.c                            |   2 +-
 fs/proc/kcore.c                               |   5 +-
 fs/proc/task_mmu.c                            |  10 +-
 include/linux/gfp.h                           |  22 +-
 include/linux/kvm_host.h                      |   3 +
 include/linux/memory_hotplug.h                |   5 +-
 include/linux/mempolicy.h                     |  16 +
 include/linux/mmzone.h                        |  26 +-
 include/linux/node_private.h                  | 262 ++++++++++++++++
 include/linux/nodemask.h                      |   7 +-
 include/uapi/linux/kvm.h                      |   5 +-
 include/uapi/linux/mempolicy.h                |   1 +
 kernel/cgroup/cpuset.c                        |  26 +-
 mm/compaction.c                               |  13 +
 mm/damon/paddr.c                              |   9 +
 mm/gup.c                                      |  28 +-
 mm/huge_memory.c                              |   5 +
 mm/internal.h                                 |  97 +++++-
 mm/khugepaged.c                               |  18 +-
 mm/ksm.c                                      |   8 +-
 mm/madvise.c                                  |   8 +-
 mm/memcontrol-v1.c                            |   8 +-
 mm/memcontrol.c                               |  13 +-
 mm/memory-tiers.c                             |  40 ++-
 mm/memory_hotplug.c                           | 119 +++++++-
 mm/mempolicy.c                                | 288 +++++++++++++++---
 mm/migrate.c                                  |  19 +-
 mm/mm_init.c                                  |   2 +-
 mm/page_alloc.c                               | 189 +++++++++---
 mm/page_alloc.h                               |  34 +++
 mm/vmscan.c                                   |  57 +++-
 tools/testing/selftests/kvm/Makefile.kvm      |   2 +
 .../kvm/guest_memfd_bind_node_test.c          | 213 +++++++++++++
 virt/kvm/guest_memfd.c                        |  39 ++-
 38 files changed, 1728 insertions(+), 160 deletions(-)
 create mode 100644 Documentation/mm/numa_private_nodes.rst
 create mode 100644 include/linux/node_private.h
 create mode 100644 tools/testing/selftests/kvm/guest_memfd_bind_node_test.c

--
2.53.0-Meta

author maintainer
#93 lore raw
Quoted 11 lines · Pankaj Gupta
Quoted 10 lines

Bonus Configuration: HBM device memory tiering
==============================================

echo 1 > dax0.0/private       # make the node private
echo 0 > dax0.0/adistance     # highest tier
echo 1 > dax0.0/reclaim       # reclaim active
echo 1 > dax0.0/demotion      # may demote from the node
echo 1 > dax0.0/user_numa     # mbind()

echo online_movable > dax0.0/state
echo 1 > numa/demotion_enabled

Quoted 6 lines

I am bit confused by the HBM device example being discovered as a DAX device. Is it to
only to showcase the use-case? Or we see HBM device is actually expected to be discovered as
DAX device with N_MEMORY_PRIVATE?

"actually expected to be" -> no. there's no requirement HBM be discovered and boudn to DAX, DAX is simply a convenient intermediary to memory-hotplug. This is at best a demonstration.

A device driver could very well just call memory-hotplug directly, or its memory could be brought up as a private node via a boot paremeter (which I have sculpted for v7). The former is preferred, but there's a bit of a chicken/egg scenario for bootstrapping.

Most of these UAPI extensions are going to be consolidated into a debugfs interface for testing purposes only in v7. I've been working on aggressive Linux Test Project test expansion to ensure there is no UAPI regressions.

~Gregory

#94 lore raw
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines

Bonus Configuration: HBM device memory tiering
==============================================

echo 1 > dax0.0/private       # make the node private
echo 0 > dax0.0/adistance     # highest tier
echo 1 > dax0.0/reclaim       # reclaim active
echo 1 > dax0.0/demotion      # may demote from the node
echo 1 > dax0.0/user_numa     # mbind()

echo online_movable > dax0.0/state
echo 1 > numa/demotion_enabled

Quoted 6 lines

I am bit confused by the HBM device example being discovered as a DAX device. Is it to
only to showcase the use-case? Or we see HBM device is actually expected to be discovered as
DAX device with N_MEMORY_PRIVATE?

Quoted 8 lines

"actually expected to be" -> no. there's no requirement HBM be discovered and boudn to DAX, DAX is simply a convenient intermediary to memory-hotplug. This is at best a demonstration.

A device driver could very well just call memory-hotplug directly, or its memory could be brought up as a private node via a boot paremeter (which I have sculpted for v7). The former is preferred, but there's a bit of a chicken/egg scenario for bootstrapping.

I find this interesting as this appears to be evolving towards a vendor agnostic framework for hotplugging memory directly to private nodes.

How you see this working? In particular, How would the kernel know which memory ranges correspond to HBM or other device memory?

Please also CC me when you post v7: [email protected]

Thanks,
Pankaj

Quoted 7 lines

Most of these UAPI extensions are going to be consolidated into a debugfs interface for testing purposes only in v7. I've been working on aggressive Linux Test Project test expansion to ensure there is no UAPI regressions.

~Gregory

author maintainer
#95 lore raw
Quoted 9 lines · Pankaj Gupta
Quoted 5 lines

A device driver could very well just call memory-hotplug directly, or its memory could be brought up as a private node via a boot paremeter (which I have sculpted for v7). The former is preferred, but there's a bit of a chicken/egg scenario for bootstrapping.

I find this interesting as this appears to be evolving towards a vendor agnostic framework for hotplugging memory directly to private nodes.

Agnostic would depend on a common standard for such devices. Otherwise, there is still a need for per-device drivers - but hopefully the interaction itself can be generalized.

Quoted 3 lines

How you see this working? In particular, How would the kernel know which memory ranges correspond to HBM or other device memory?

Device marks the region reserved at boot, driver hotplugs.

or

Device marks the region normal, but boot param sets the node private.

Quoted 2 lines

Please also CC me when you post v7: [email protected]

Sure

~Gregory

author maintainer
#96 lore raw
In reply to Gregory Price
Quoted 5 lines · Gregory Price

This series introduces the concept of "Private Memory Nodes", which are opted into two basic functionalities by default:
- page allocation (mm/page_alloc.c)
- OOM killing

All, I wanted to preface v6 with an update, I probably won't be sending it out on-list before LPC (i've been a bit noisey as it is, I'll give it a rest).

But I would like to give you the highlights for those interested.

The latest working branch can be found here:
https://github.com/gourryinverse/linux/tree/scratch/gourry/managed_nodes/rfc6-nonuma

You will find the following:

- There is no depedency on exported alloc_flags (willy, i found a way), but there is a need for folio_alloc_private() to explicitly ask for the ALLOC_ZONELIST_PRIVATE.

- feature bits have been converted to node_state[] bits

- N_MEMORY_COMMON was added to mean "common memory" - i.e. the existing N_MEMORY set. A private node is now defined as !N_MEMORY_COMMON.

- opt-out sites no longer check for !N_MEMORY_PRIVATE, they instead iterate over node_state[N_MEMORY_COMMON]. This is *much* cleaner.

- opt-in sites get to define bits in terms of their own service, e.g.

  /* for each reclaimable node */
  for_each_node_mask_state(..., N_MEMORY_RECLAIM) {
     ...
  }

Or maybe even:

for_each_reclaimable_node() :)

- There are only 2 node features in the base set:

N_MEMORY_RECLAIM - generic reclaim runs on the node
N_MEMORY_COMPACTION - compaction will run on the node

You'll notice there's no userland numa control support, more on this in a moment.

This is all you need to have functional device-private coherent memory that uses the page allocator.

Disabling N_MEMORY_RECLAIM does not necessarily mean you're prevented from using vmscan.c and the LRU - it just means we'd need to expose an interface for a !N_MEMORY_RECLAIM node to explicitly ask for reclaim to operate on it.

Tiering (Demotion) *to* a node is not supported. We can probably debate whether this should mean tiering *from* a node should be supported or not as well. This is easy to change.

- The working branch also includes a minimal compressed-ram service that only supports anonymous memory.

Supporting file-backed memory (and shmem) is a whole different can of worms that deserves its own discussion.

I will post this as a separate RFC from the base set, but it is available to play with on my working branch.

- I ship a basic dax driver for the base series, anondax which allows existing devices to online a private node with both compaction and reclaim support for testing. This enables this use case:

  fd = open(/dev/my_device, ..., O_DIRECT);
  buf = mmap(fd, ..., MAP_SHARED)
  my_file = open(myfile, ...);
  read(buf, myfile); /* fault directly onto device memory */

With swap (and no tiering), this gives you a private over-commitable node for which aspiring drivers can use to implement their own mmu_notifier callback stream to manage device page tables and such.

- The reason mempolicy (and user numa in general) is not supported is a critical relationship between the OOM killer and page allocator.

In trivial scenarios, if a consumer of a private node overshoots and causes an OOM - it will be the largest consumer and be chosen as the victim.

If there are many consumers - what actually happens in practice is the oom killer attempts to select a victim based on *task* policy... which makes every task a candidate. This leads to, among other things, either an OOM storm and/or a full blown panic because a victim cannot be located (depending on the constraints).

I've concluded that as-is, this is not tractable. But also, it is not actually needed if the devices provide their own chardev to provide a basic wrapper around folio_alloc_private().

This unfortunately means potential users like guest_memfd are unlikely to be able to use the nodes without hard-coding the allocator interface, as opposed to a mempolicy.

I think it's possible instead to *maybe* add N_MEMORY_USER_MIGRATION and allow for explicit movement between nodes, but not full blown mempolicies.

The relationship between page_alloc, cpuset, mempolicy, and the oom killer is just too tight to allow allocations without the use of __GFP_THISNODE - which the private allocation interface will enforce.

As it stands, the series has been heavily minimized in both line count and patch number.

The base series is ~22 commits with the scary diffstat below, but many of the core mm/ patches are similar to the patch below - with nearly all of the real complexity happening in reclaim and memory hotplug.

despite the branch name, I intend to drop RFC from here-on, everything else required for the base series is in mm-new.

See you at LPC
~Gregory

-------

mm/migrate.c +3 -3
index 7bdcdb57652f8..cd61566b44fa1 100644
@@ -2266,7 +2266,7 @@ static int __add_folio_for_migration(struct folio *folio, int node,
        if (is_zero_folio(folio) || is_huge_zero_folio(folio))
                return -EFAULT;
 
-       if (folio_is_zone_device(folio))
+       if (!folio_is_common_memory(folio))
                return -ENOENT;
 
        if (folio_nid(folio) == node)
@@ -2390,7 +2390,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes,
                err = -ENODEV;
                if (node < 0 || node >= MAX_NUMNODES)
                        goto out_flush;
-               if (!node_state(node, N_MEMORY))
+               if (!node_state(node, N_MEMORY_COMMON))
                        goto out_flush;
 
                err = -EACCES;
@@ -2475,7 +2475,7 @@ static void do_pages_stat_array(struct mm_struct *mm, unsigned long nr_pages,
                if (folio) {
                        if (is_zero_folio(folio) || is_huge_zero_folio(folio))
                                err = -EFAULT;
-                       else if (folio_is_zone_device(folio))
+                       else if (!folio_is_common_memory(folio))
                                err = -ENOENT;
                        else
                                err = folio_nid(folio);
 Documentation/ABI/stable/sysfs-devices-node |  21 +++++++++++++++++
 Documentation/admin-guide/cgroup-v2.rst     |   7 ++++++
 Documentation/mm/physical_memory.rst        |  15 ++++++++++++-
 drivers/base/node.c                         |  67 +++++++++++++++++++++++++++++++++++++++++++++++++++++--
 drivers/dax/kmem.c                          |   2 +-
 include/linux/gfp.h                         |   6 +++++
 include/linux/memory_hotplug.h              |   2 +-
 include/linux/mmzone.h                      |  45 +++++++++++++++++++++++++++++++++++++
 include/linux/node.h                        |  12 ++++++++++
 include/linux/nodemask.h                    |  53 ++++++++++++++++++++++++++++++++++++++++++-
 kernel/cgroup/cpuset.c                      |  39 ++++++++++++++++++++++++++------
 kernel/sched/fair.c                         |   6 ++---
 mm/compaction.c                             |  21 +++++++++++------
 mm/damon/ops-common.c                       |  15 ++++++++-----
 mm/damon/vaddr.c                            |  20 +++++++++++++----
 mm/huge_memory.c                            |   2 +-
 mm/internal.h                               |  22 ++++++++++++++++++
 mm/khugepaged.c                             |   9 ++++++--
 mm/ksm.c                                    |   8 ++++---
 mm/madvise.c                                |   6 ++---
 mm/memory-tiers.c                           |  25 +++++++++++----------
 mm/memory_hotplug.c                         | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-------------------------------
 mm/mempolicy.c                              |  51 +++++++++++++++++++++++++-----------------
 mm/migrate.c                                |   6 ++---
 mm/mm_init.c                                |  22 +++++++++---------
 mm/page_alloc.c                             | 115 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------
 mm/page_alloc.h                             |   5 +++++
 mm/vmscan.c                                 |  31 +++++++++++++++-----------
 28 files changed, 585 insertions(+), 157 deletions(-)