mm/percpu: allow embed allocator when total size fits in vmalloc

On large NUMA systems (e.g. 4-node 128GB), the physical span between NUMA nodes may exceed 75% of vmalloc space even though the total percpu allocation is tiny (e.g. ~6MB for 64 CPUs).

The 75% check was designed for pcpu_get_vm_areas() used by dynamic percpu chunks, where vmalloc congruency is required. The embed allocator uses the linear map directly for the first chunk -- physical addresses are accessible as virtual addresses without vmalloc mapping. The relevant constraint is total percpu size, not physical span.

Replace the unconditional fallback with a total-size check: if the aggregate percpu allocation fits within 75% of vmalloc space, proceed with embed regardless of physical NUMA span. Systems where total percpu size genuinely exceeds the vmalloc bound retain the page allocator fallback.

Tested on Sophgo SG2042 (64-hart, 4-NUMA, 128GB DDR4, RISC-V Sv39):
percpu: embed: span 0x17def7a000 > vmalloc 75% but total 0x5c0000
fits -- linear map used
percpu: Embedded 23 pages/cpu s54168 r8192 d31848 u94208

---

This fix is inert on systems where the original check never fired; it only activates when physical span exceeds 75% of vmalloc but total percpu size does not.

Architecture analysis:

  Architecture   | vmalloc      | phys span    | total percpu | result
  ---------------|--------------|--------------|--------------|--------
  RISC-V Sv39    | ~88 GB       | ~102 GB      | ~6 MB        | fixed
  RISC-V Sv48    | ~88 TB       | ~102 GB      | ~6 MB        | unaffected
  RISC-V Sv57    | ~44 PB       | ~102 GB      | ~6 MB        | unaffected
  ARM64 large    | ~248 TB      | varies       | tiny         | unaffected
  32-bit NUMA    | ~128 MB      | varies       | may exceed   | correct fallback

Tejun Heo noted in 2017 [Link] that the only constraint is vmalloc size relative to NUMA node distances, and that making vmalloc bigger would be the best fix. On RISC-V Sv39 with 88GB vmalloc and 102GB physical NUMA span that is not an option -- but the embed allocator does not need congruent vmalloc mapping for the first chunk anyway.

 mm/percpu.c | 29 ++++++++++++++++++++++++-----
 1 file changed, 24 insertions(+), 5 deletions(-)

index b0676b8054ed..72695778872c 100644
@@ -3069,13 +3069,32 @@ int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size,
 
 	/* warn if maximum distance is further than 75% of vmalloc space */
 	if (max_distance > VMALLOC_TOTAL * 3 / 4) {
-		pr_warn("max_distance=0x%lx too large for vmalloc space 0x%lx\n",
+		pr_warn("percpu: embed: max_distance=0x%lx too large for vmalloc space 0x%lx\n",
 				max_distance, VMALLOC_TOTAL);
 #ifdef CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK
-		/* and fail if we have fallback */
-		rc = -EINVAL;
-		goto out_free_areas;
-#endif
+		/*
+		 * The embed allocator uses the linear map directly for the
+		 * first chunk -- physical addresses are accessible as virtual
+		 * addresses without vmalloc mapping. The 75% check was designed
+		 * for pcpu_get_vm_areas() (dynamic chunks) where vmalloc
+		 * congruency is required. On large NUMA systems, physical span
+		 * between nodes may exceed vmalloc bounds even though total
+		 * percpu size is tiny. Check total size, not physical span.
+		 */
+		{
+			unsigned long total_size = 0;
+			for (group = 0; group < ai->nr_groups; group++)
+				total_size += (unsigned long)ai->unit_size *
+					ai->groups[group].nr_units;
+			if (total_size > VMALLOC_TOTAL * 3 / 4) {
+				rc = -EINVAL;
+				goto out_free_areas;
+			}
+			pr_info("percpu: embed: span 0x%lx > vmalloc 75%%"
+				" but total 0x%lx fits -- linear map used\n",
+				max_distance, total_size);
+		}
+#endif /* CONFIG_NEED_PER_CPU_PAGE_FIRST_CHUNK */
 	}
 
 	/*

2.53.0

Nicely presented patch, thanks.

I suggest all the above be moved above the "---" separator. It's useful info for the permanent record. (And replace "[Link]" with "[1]" in the usual fashion).

AI review asked a question:

	https://sashiko.dev/#/patchset/[email protected]

Andrew,

Thank you. v2 will incorporate your formatting suggestions.

The Sashiko concern is valid and matches a WARN_ON() we observed in our boot logs during dynamic percpu chunk allocation. It appears related to the sparse physical group offsets reused after embed initialization, and I'll address it in v2.

Thanks,
Paul Sherman

Andrew,

Thank you for the encouraging words on the initial submission.

Tejun,

Further testing on SG2042 (Sv39, 4 NUMA nodes, 128GB) showed that the proposed change allows the embedded first chunk to be created successfully, but later dynamic chunk population consistently fails in pcpu_populate_chunk(), with warnings originating from __vmap_pages_range_noflush(). Reverting the patch eliminates those warnings on the same hardware.

Looking through the code, my current understanding is that dynamic chunks inherit the first chunk's layout through the shared pcpu_unit_offsets[] table, while vmalloc reservations are driven by pcpu_group_offsets[]. That led me to wonder whether I've encountered a fundamental allocator invariant rather than merely an implementation detail.

Is the requirement that all chunks share a single pcpu_unit_offsets[] layout fundamental to the percpu allocator design, or is there an intended mechanism for decoupling the embedded first-chunk layout from dynamic chunk vmalloc mappings?

The answer will help determine whether the correct next step is to withdraw the patch and instead document the existing invariant, or whether there's a larger design direction worth investigating.

Thanks,
Paul

Hello Paul,

Percpu memory is essentially an offset-based virtual address space. We build the unit and group offsets when setting up the first chunk at boot, and those offsets remain fixed for the lifetime of the machine. Consequently, every later chunk must reproduce the same relative virtual layout.

The embedded first chunk works because its sparse addresses already exist in the linear map. However, pcpu_setup_first_chunk() saves that layout in pcpu_group_offsets, which is subsequently passed to pcpu_get_vm_areas() when creating dynamic chunks. On this machine the resulting span is larger than the vmalloc area, so allowing the embedded setup to succeed means that new dynamic chunks will fail once the first chunk is exhausted.

Machines have grown considerably since this design was introduced. Supporting an embedded layout like this would probably require a purely direct-mapped backend, with congruent physical ranges reserved at boot for future chunks. That may be reasonable for a particular machine if percpu growth can be bounded and the reservation is small relative to available memory, but I don´t think it generalizes based solely on the ratio between chunk size and total memory. We would also need to guarantee the required physical placement and decide how much space to reserve for lifetime percpu growth.

As is, this patch isn't correct, but it might inspire new work.

Thanks,
Dennis