Skip to contentSkip to conversation

CMA balancing

RFC PATCH In review Frank van der Linden 4 participants 33 messages Sep 15, 2025 – Sep 26, 2025

Subsystems (13)
lore
Share & tools
lore
#0 lore raw

cover CMA balancing

This is an RFC on a solution to the long standing problem of OOMs occuring when the kernel runs out of space for unmovable allocations in the face of large amounts of CMA.

Introduction
============

When there is a large amount of CMA (e.g. with hugetlb_cma), it is possible for the kernel to run out of space to get unmovable allocations from. This is because it cannot use the CMA area. If the issue is just that there is a large CMA area, and that there isn't enough space left, that can be considered a misconfigured system. However, there is a scenario in which things could have been dealt with better: if the non-CMA area also has movable allocations in it, and there are CMA pageblocks still available.

The current mitigation for this issue is to start using CMA pageblocks for movable allocations first if the amount of free CMA pageblocks is more than 50% of the total amount of free memory in a zone. But that may not always work out, e.g. the system could easily run in to a scenario where long-lasting movable allocations are made first, which do not go to CMA before the 50% mark is reached. When the non-CMA area fills up, these will get in the way of the kernel's unmovable allocations, and OOMs might occur.

Even always directing movable allocations to CMA first does not completely fix the issue. Take a scenario where there is a large amount of CMA through hugetlb_cma. All of that CMA has been taken up by 1G hugetlb pages. So, movable allocations end up in the non-CMA area. Now, the number of hugetlb pages in the pool is lowered, so some CMA becomes available. At the same time, increased system activity leads to more unmovable allocations. Since the movable allocations are still in the non-CMA area, these kernel allocations might still fail.

Additionally, CMA areas are allocated at the bottom of the zone. There has been some discussion on this in the past. Originally, doing allocations from CMA was deemed something that was best avoided. The arguments were twofold:

1) cma_alloc needs to be quick and should not have to migrate a lot of pages.
2) migration might fail, so the fewer pages it has to migrate the better

These arguments are why CMA is avoided (until the 50% limit is hit), and why CMA areas are allocated at the bottom of a zone. But compaction migrates memory from the bottom to the top of a zone. That means that compaction will actually end up migrating movable allocations out of CMA and in to non-CMA, making the issue of OOMing for unmovable allocations worse.

Solution: CMA balancing
=======================

First, this patch set makes the 50% threshold configurable, which is useful in any case. vm.cma_first_limit is the percentage of free CMA, as part of the total amount of free memory in a zone, above which CMA will be used first for movable allocations. 0 is always, 100 is never.

Then, it creates an interface that allows for moving movable allocations from non-CMA to CMA. CMA areas opt in to taking part in this through a flag. Also, if the flag is set for a CMA area, it is allocated at the top of a zone instead of the bottom.

Lastly, the hugetlb_cma code was modified to try to migrate movable allocations from non-CMA to CMA when a hugetlb CMA page is freed. Only hugetlb CMA areas opt in to CMA balancing, behavior for all other CMA areas is unchanged.

Discussion
==========

This approach works when tested with a hugetlb_cma setup where a large number of 1G pages is active, but the number is sometimes reduced in exchange for larger non-hugetlb overhead.

Arguments against this approach:

* It's kind of heavy-handed. Since there is no easy way to track the amount of movable allocations residing in non-CMA pageblocks, it will likely end up scanning too much memory, as it only knows the upper bound.
* It should be more integrated with watermark handling in the allocation slow path. Again, this would likely require tracking the number of movable allocations in non-CMA pageblocks.

Arguments for this approach:

* Yes, it does more, but the work is restricted to the context of a process that decreases the hugetlb pool, and is not more work than allocating (e.g. freeing a hugetlb page from the pool is now as expensive as allocating a new one).
* hugetlb_cma is really the only situation where you have CMA areas large enough to trigger the OOM scenario, so restricting it to hugetlb should be good enough.

Comments, thoughts?

Frank van der Linden (12):
mm/cma: add tunable for CMA fallback limit mm/cma: clean up flag handling a bit
mm/cma: add flags argument to init functions mm/cma: keep a global sorted list of CMA ranges mm/cma: add helper functions for CMA balancing mm/cma: define and act on CMA_BALANCE flag mm/compaction: optionally use a different isolate function mm/compaction: simplify isolation order checks a bit mm/cma: introduce CMA balancing
mm/hugetlb: do explicit CMA balancing
mm/cma: rebalance CMA when changing cma_first_limit mm/cma: add CMA balance VM event counter

 arch/powerpc/kernel/fadump.c         |   2 +-
 arch/powerpc/kvm/book3s_hv_builtin.c |   2 +-
 drivers/s390/char/vmcp.c             |   2 +-
 include/linux/cma.h                  |  64 +++++-
 include/linux/migrate_mode.h         |   1 +
 include/linux/mm.h                   |   4 +
 include/linux/vm_event_item.h        |   3 +
 include/trace/events/migrate.h       |   3 +-
 kernel/dma/contiguous.c              |  10 +-
 mm/cma.c                             | 318 +++++++++++++++++++++++----
 mm/cma.h                             |  13 +-
 mm/compaction.c                      | 199 +++++++++++++++--
 mm/hugetlb.c                         |  14 +-
 mm/hugetlb_cma.c                     |  18 +-
 mm/hugetlb_cma.h                     |   5 +
 mm/internal.h                        |  11 +-
 mm/migrate.c                         |   8 +
 mm/page_alloc.c                      | 104 +++++++--
 mm/vmstat.c                          |   2 +
 19 files changed, 676 insertions(+), 107 deletions(-)
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
#1 lore raw

RFC PATCH 1/12 mm/cma: add tunable for CMA fallback limit

Add a tunable to experiment with the circumstances under which movable allocations should use CMA pageblocks first, to avoid false OOM conditions.

The limit is the percentage free memory which is being taken up by CMA. If the amount of used memory in CMA pageblocks is above this limit, CMA will be used first. So, 0 would mean always using CMA first, and 100 means never use CMA first.

Currently the default is 50, which matches the existing behavior, so there is no functional change.

---
 include/linux/mm.h |  4 +++
 mm/page_alloc.c    | 84 ++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 74 insertions(+), 14 deletions(-)
include/linux/mm.h +4 -0
index 1ae97a0b8ec7..313ab38dc398 100644
@@ -3253,6 +3253,10 @@ void warn_alloc(gfp_t gfp_mask, nodemask_t *nodemask, const char *fmt, ...);
 
 extern void setup_per_cpu_pageset(void);
 
+#ifdef CONFIG_CMA
+extern int cma_first_limit;
+#endif
+
 /* nommu.c */
 extern atomic_long_t mmap_pages_allocated;
 extern int nommu_shrink_inode_mappings(struct inode *, size_t, size_t);
mm/page_alloc.c +70 -14

109 lines not on this page · raw message

Signature · 1 line
2.51.0.384.g4c02a37b29-goog
#2 lore raw

Re: [RFC PATCH 01/12] mm/cma: add tunable for CMA fallback limit

Quoted 12 lines · Frank van der Linden
mm/page_alloc.c +11 -0
+#ifdef CONFIG_CMA
+	{
+		.procname	= "cma_first_limit",
+		.data		= &cma_first_limit,
+		.maxlen		= sizeof(cma_first_limit),
+		.mode		= 0644,
+		.proc_handler	= proc_dointvec_minmax,
+		.extra1		= SYSCTL_ZERO,
+		.extra2		= SYSCTL_ONE_HUNDRED,
+	},
+#endif
Quoted 2 lines

 };

I like the code, but the name is a little confusing.

Maybe something like "prefer_cma_percent"?

Signature · 1 line
All Rights Reversed.
#3 lore raw

RFC PATCH 2/12 mm/cma: clean up flag handling a bit

In reply to Frank van der Linden

Atomic bit operations aren't needed for the cma flags field, so switch their manipulation over to normal AND/OR operations.

Also export the bit values in linux/cma.h, as we will be adding publicly used values later.

No functional change.

---
 include/linux/cma.h | 12 ++++++++++++
 mm/cma.c            | 16 ++++++++--------
 mm/cma.h            |  7 -------
 3 files changed, 20 insertions(+), 15 deletions(-)
include/linux/cma.h +12 -0
index 62d9c1cf6326..5c3fdc5da908 100644
@@ -20,6 +20,18 @@
 #define CMA_MIN_ALIGNMENT_PAGES pageblock_nr_pages
 #define CMA_MIN_ALIGNMENT_BYTES (PAGE_SIZE * CMA_MIN_ALIGNMENT_PAGES)
 
+enum cma_flags {
+	__CMA_RESERVE_PAGES_ON_ERROR,
+	__CMA_ZONES_VALID,
+	__CMA_ZONES_INVALID,
+	__CMA_ACTIVATED,
+};
+
+#define CMA_RESERVE_PAGES_ON_ERROR	BIT(__CMA_RESERVE_PAGES_ON_ERROR)
+#define CMA_ZONES_VALID			BIT(__CMA_ZONES_VALID)
+#define CMA_ZONES_INVALID		BIT(__CMA_ZONES_INVALID)
+#define CMA_ACTIVATED			BIT(__CMA_ACTIVATED)
+
 struct cma;
 
 extern unsigned long totalcma_pages;
mm/cma.c +8 -8
index 2ffa4befb99a..549d85b2e3a3 100644
@@ -111,8 +111,8 @@ bool cma_validate_zones(struct cma *cma)
 	 * check has already been done. If neither is set, the
 	 * check has not been performed yet.
 	 */
-	valid_bit_set = test_bit(CMA_ZONES_VALID, &cma->flags);
-	if (valid_bit_set || test_bit(CMA_ZONES_INVALID, &cma->flags))
+	valid_bit_set = (cma->flags & CMA_ZONES_VALID);
+	if (valid_bit_set || (cma->flags & CMA_ZONES_INVALID))
 		return valid_bit_set;
 
 	for (r = 0; r < cma->nranges; r++) {
@@ -126,12 +126,12 @@ bool cma_validate_zones(struct cma *cma)
 		 */
 		WARN_ON_ONCE(!pfn_valid(base_pfn));
 		if (pfn_range_intersects_zones(cma->nid, base_pfn, cmr->count)) {
-			set_bit(CMA_ZONES_INVALID, &cma->flags);
+			cma->flags |= CMA_ZONES_INVALID;
 			return false;
 		}
 	}
 
-	set_bit(CMA_ZONES_VALID, &cma->flags);
+	cma->flags |= CMA_ZONES_VALID;
 
 	return true;
 }
@@ -176,7 +176,7 @@ static void __init cma_activate_area(struct cma *cma)
 	INIT_HLIST_HEAD(&cma->mem_head);
 	spin_lock_init(&cma->mem_head_lock);
 #endif
-	set_bit(CMA_ACTIVATED, &cma->flags);
+	cma->flags |= CMA_ACTIVATED;
 
 	return;
 
@@ -185,7 +185,7 @@ static void __init cma_activate_area(struct cma *cma)
 		bitmap_free(cma->ranges[r].bitmap);
 
 	/* Expose all pages to the buddy, they are useless for CMA. */
-	if (!test_bit(CMA_RESERVE_PAGES_ON_ERROR, &cma->flags)) {
+	if (!(cma->flags & CMA_RESERVE_PAGES_ON_ERROR)) {
 		for (r = 0; r < allocrange; r++) {
 			cmr = &cma->ranges[r];
 			end_pfn = cmr->base_pfn + cmr->count;
@@ -211,7 +211,7 @@ core_initcall(cma_init_reserved_areas);
 
 void __init cma_reserve_pages_on_error(struct cma *cma)
 {
-	set_bit(CMA_RESERVE_PAGES_ON_ERROR, &cma->flags);
+	cma->flags |= CMA_RESERVE_PAGES_ON_ERROR;
 }
 
 static int __init cma_new_area(const char *name, phys_addr_t size,
@@ -1085,7 +1085,7 @@ void __init *cma_reserve_early(struct cma *cma, unsigned long size)
 	/*
 	 * Can only be called early in init.
 	 */
-	if (test_bit(CMA_ACTIVATED, &cma->flags))
+	if (cma->flags & CMA_ACTIVATED)
 		return NULL;
 
 	if (!IS_ALIGNED(size, CMA_MIN_ALIGNMENT_BYTES))
mm/cma.h +0 -7
index c70180c36559..25b696774c6a 100644
@@ -64,13 +64,6 @@ struct cma {
 	int nid;
 };
 
-enum cma_flags {
-	CMA_RESERVE_PAGES_ON_ERROR,
-	CMA_ZONES_VALID,
-	CMA_ZONES_INVALID,
-	CMA_ACTIVATED,
-};
-
 extern struct cma cma_areas[MAX_CMA_AREAS];
 extern unsigned int cma_area_count;
 
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Reviewed-by on v1 2/12
#4 lore raw

Re: [RFC PATCH 02/12] mm/cma: clean up flag handling a bit

Quoted 11 lines · Frank van der Linden

Atomic bit operations aren't needed for the cma flags field, so switch their manipulation over to normal AND/OR operations.

Also export the bit values in linux/cma.h, as we will be adding publicly used values later.

No functional change.

Signature · 1 line
All Rights Reversed.
#5 lore raw

RFC PATCH 3/12 mm/cma: add flags argument to init functions

In reply to Frank van der Linden

Add a flags argument to the various CMA init functions, as there will be a need to pass in more flags to control init and runtime behavior other than just the current 'fixed' argument.

Replace the fixed argument with a flags argument, and adapt callers.

---
 arch/powerpc/kernel/fadump.c         |  2 +-
 arch/powerpc/kvm/book3s_hv_builtin.c |  2 +-
 drivers/s390/char/vmcp.c             |  2 +-
 include/linux/cma.h                  | 20 ++++++++++------
 kernel/dma/contiguous.c              | 10 ++++----
 mm/cma.c                             | 36 ++++++++++++++++++----------
 mm/hugetlb_cma.c                     |  2 +-
 7 files changed, 46 insertions(+), 28 deletions(-)
arch/powerpc/kernel/fadump.c +1 -1
index 5782e743fd27..a763419bd1bc 100644
@@ -112,7 +112,7 @@ void __init fadump_cma_init(void)
 		return;
 	}
 
-	rc = cma_init_reserved_mem(base, size, 0, "fadump_cma", &fadump_cma);
+	rc = cma_init_reserved_mem(base, size, 0, 0, "fadump_cma", &fadump_cma);
 	if (rc) {
 		pr_err("Failed to init cma area for firmware-assisted dump,%d\n", rc);
 		/*
arch/powerpc/kvm/book3s_hv_builtin.c +1 -1
index fa0e3a22cac0..23dcb67e797a 100644
@@ -95,7 +95,7 @@ void __init kvm_cma_reserve(void)
 			 (unsigned long)selected_size / SZ_1M);
 		align_size = HPT_ALIGN_PAGES << PAGE_SHIFT;
 		cma_declare_contiguous(0, selected_size, 0, align_size,
-			KVM_CMA_CHUNK_ORDER - PAGE_SHIFT, false, "kvm_cma",
+			KVM_CMA_CHUNK_ORDER - PAGE_SHIFT, 0, "kvm_cma",
 			&kvm_cma);
 	}
 }
drivers/s390/char/vmcp.c +1 -1
index 69899bb86b3e..cd0c0edc496b 100644
@@ -54,7 +54,7 @@ void __init vmcp_cma_reserve(void)
 {
 	if (!machine_is_vm())
 		return;
-	cma_declare_contiguous(0, vmcp_cma_size, 0, 0, 0, false, "vmcp", &vmcp_cma);
+	cma_declare_contiguous(0, vmcp_cma_size, 0, 0, 0, 0, "vmcp", &vmcp_cma);
 }
 
 static void vmcp_response_alloc(struct vmcp_session *session)
include/linux/cma.h +13 -7
index 5c3fdc5da908..ec48f2a11f1d 100644
@@ -25,12 +25,16 @@ enum cma_flags {
 	__CMA_ZONES_VALID,
 	__CMA_ZONES_INVALID,
 	__CMA_ACTIVATED,
+	__CMA_FIXED,
 };
 
 #define CMA_RESERVE_PAGES_ON_ERROR	BIT(__CMA_RESERVE_PAGES_ON_ERROR)
 #define CMA_ZONES_VALID			BIT(__CMA_ZONES_VALID)
 #define CMA_ZONES_INVALID		BIT(__CMA_ZONES_INVALID)
 #define CMA_ACTIVATED			BIT(__CMA_ACTIVATED)
+#define CMA_FIXED			BIT(__CMA_FIXED)
+
+#define CMA_INIT_FLAGS (CMA_FIXED|CMA_RESERVE_PAGES_ON_ERROR)
 
 struct cma;
 
@@ -42,23 +46,25 @@ extern const char *cma_get_name(const struct cma *cma);
 extern int __init cma_declare_contiguous_nid(phys_addr_t base,
 			phys_addr_t size, phys_addr_t limit,
 			phys_addr_t alignment, unsigned int order_per_bit,
-			bool fixed, const char *name, struct cma **res_cma,
-			int nid);
+			unsigned long flags, const char *name,
+			struct cma **res_cma, int nid);
 static inline int __init cma_declare_contiguous(phys_addr_t base,
 			phys_addr_t size, phys_addr_t limit,
 			phys_addr_t alignment, unsigned int order_per_bit,
-			bool fixed, const char *name, struct cma **res_cma)
+			unsigned long flags, const char *name,
+			struct cma **res_cma)
 {
 	return cma_declare_contiguous_nid(base, size, limit, alignment,
-			order_per_bit, fixed, name, res_cma, NUMA_NO_NODE);
+			order_per_bit, flags, name, res_cma, NUMA_NO_NODE);
 }
 extern int __init cma_declare_contiguous_multi(phys_addr_t size,
 			phys_addr_t align, unsigned int order_per_bit,
-			const char *name, struct cma **res_cma, int nid);
+			unsigned long flags, const char *name,
+			struct cma **res_cma, int nid);
 extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size,
 					unsigned int order_per_bit,
-					const char *name,
-					struct cma **res_cma);
+					unsigned long flags,
+					const char *name, struct cma **res_cma);
 extern struct page *cma_alloc(struct cma *cma, unsigned long count, unsigned int align,
 			      bool no_warn);
 extern bool cma_pages_valid(struct cma *cma, const struct page *pages, unsigned long count);
kernel/dma/contiguous.c +6 -4
index d9b9dcba6ff7..7f2eed3b7cc5 100644
@@ -175,7 +175,7 @@ static void __init dma_numa_cma_reserve(void)
 			cma = &dma_contiguous_pernuma_area[nid];
 			snprintf(name, sizeof(name), "pernuma%d", nid);
 			ret = cma_declare_contiguous_nid(0, pernuma_size_bytes, 0, 0,
-							 0, false, name, cma, nid);
+							 0, 0, name, cma, nid);
 			if (ret)
 				pr_warn("%s: reservation failed: err %d, node %d", __func__,
 					ret, nid);
@@ -185,7 +185,7 @@ static void __init dma_numa_cma_reserve(void)
 
 			cma = &dma_contiguous_numa_area[nid];
 			snprintf(name, sizeof(name), "numa%d", nid);
-			ret = cma_declare_contiguous_nid(0, numa_cma_size[nid], 0, 0, 0, false,
+			ret = cma_declare_contiguous_nid(0, numa_cma_size[nid], 0, 0, 0, 0,
 							 name, cma, nid);
 			if (ret)
 				pr_warn("%s: reservation failed: err %d, node %d", __func__,
@@ -279,7 +279,8 @@ int __init dma_contiguous_reserve_area(phys_addr_t size, phys_addr_t base,
 {
 	int ret;
 
-	ret = cma_declare_contiguous(base, size, limit, 0, 0, fixed,
+	ret = cma_declare_contiguous(base, size, limit, 0, 0,
+					fixed ? CMA_FIXED : 0,
 					"reserved", res_cma);
 	if (ret)
 		return ret;
@@ -478,7 +479,8 @@ static int __init rmem_cma_setup(struct reserved_mem *rmem)
 		return -EINVAL;
 	}
 
-	err = cma_init_reserved_mem(rmem->base, rmem->size, 0, rmem->name, &cma);
+	err = cma_init_reserved_mem(rmem->base, rmem->size, 0, 0, rmem->name,
+				    &cma);
 	if (err) {
 		pr_err("Reserved memory: unable to setup CMA region\n");
 		return err;
mm/cma.c +23 -13
index 549d85b2e3a3..00d8d365f0b5 100644
@@ -35,6 +35,7 @@
 
 struct cma cma_areas[MAX_CMA_AREAS];
 unsigned int cma_area_count;
+static DEFINE_MUTEX(cma_mutex);
 
 phys_addr_t cma_get_base(const struct cma *cma)
 {
@@ -215,7 +216,7 @@ void __init cma_reserve_pages_on_error(struct cma *cma)
 }
 
 static int __init cma_new_area(const char *name, phys_addr_t size,
-			       unsigned int order_per_bit,
+			       unsigned int order_per_bit, unsigned long flags,
 			       struct cma **res_cma)
 {
 	struct cma *cma;
@@ -239,6 +240,7 @@ static int __init cma_new_area(const char *name, phys_addr_t size,
 
 	cma->available_count = cma->count = size >> PAGE_SHIFT;
 	cma->order_per_bit = order_per_bit;
+	cma->flags = flags;
 	*res_cma = cma;
 	totalcma_pages += cma->count;
 
@@ -265,7 +267,7 @@ static void __init cma_drop_area(struct cma *cma)
  */
 int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size,
 				 unsigned int order_per_bit,
-				 const char *name,
+				 unsigned long flags, const char *name,
 				 struct cma **res_cma)
 {
 	struct cma *cma;
@@ -288,7 +290,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size,
 	if (!IS_ALIGNED(base | size, CMA_MIN_ALIGNMENT_BYTES))
 		return -EINVAL;
 
-	ret = cma_new_area(name, size, order_per_bit, &cma);
+	ret = cma_new_area(name, size, order_per_bit, flags, &cma);
 	if (ret != 0)
 		return ret;
 
@@ -429,12 +431,18 @@ static phys_addr_t __init cma_alloc_mem(phys_addr_t base, phys_addr_t size,
 static int __init __cma_declare_contiguous_nid(phys_addr_t *basep,
 			phys_addr_t size, phys_addr_t limit,
 			phys_addr_t alignment, unsigned int order_per_bit,
-			bool fixed, const char *name, struct cma **res_cma,
-			int nid)
+			unsigned long flags, const char *name,
+			struct cma **res_cma, int nid)
 {
 	phys_addr_t memblock_end = memblock_end_of_DRAM();
 	phys_addr_t base = *basep;
 	int ret;
+	bool fixed;
+
+	if (flags & ~CMA_INIT_FLAGS)
+		return -EINVAL;
+
+	fixed = (flags & CMA_FIXED);
 
 	pr_debug("%s(size %pa, base %pa, limit %pa alignment %pa)\n",
 		__func__, &size, &base, &limit, &alignment);
@@ -503,7 +511,8 @@ static int __init __cma_declare_contiguous_nid(phys_addr_t *basep,
 		kmemleak_ignore_phys(base);
 	}
 
-	ret = cma_init_reserved_mem(base, size, order_per_bit, name, res_cma);
+	ret = cma_init_reserved_mem(base, size, order_per_bit, flags,
+				    name, res_cma);
 	if (ret) {
 		memblock_phys_free(base, size);
 		return ret;
@@ -526,7 +535,8 @@ static int __init __cma_declare_contiguous_nid(phys_addr_t *basep,
  */
 int __init cma_declare_contiguous_multi(phys_addr_t total_size,
 			phys_addr_t align, unsigned int order_per_bit,
-			const char *name, struct cma **res_cma, int nid)
+			unsigned long flags, const char *name,
+			struct cma **res_cma, int nid)
 {
 	phys_addr_t start = 0, end;
 	phys_addr_t size, sizesum, sizeleft;
@@ -543,7 +553,7 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,
 	 * First, try it the normal way, producing just one range.
 	 */
 	ret = __cma_declare_contiguous_nid(&start, total_size, 0, align,
-			order_per_bit, false, name, res_cma, nid);
+			order_per_bit, flags, name, res_cma, nid);
 	if (ret != -ENOMEM)
 		goto out;
 
@@ -567,7 +577,7 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,
 	sizesum = 0;
 	failed = NULL;
 
-	ret = cma_new_area(name, total_size, order_per_bit, &cma);
+	ret = cma_new_area(name, total_size, order_per_bit, flags, &cma);
 	if (ret != 0)
 		goto out;
 
@@ -716,7 +726,7 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,
  * @limit: End address of the reserved memory (optional, 0 for any).
  * @alignment: Alignment for the CMA area, should be power of 2 or zero
  * @order_per_bit: Order of pages represented by one bit on bitmap.
- * @fixed: hint about where to place the reserved area
+ * @flags: flags controlling various aspects of the area
  * @name: The name of the area. See function cma_init_reserved_mem()
  * @res_cma: Pointer to store the created cma region.
  * @nid: nid of the free area to find, %NUMA_NO_NODE for any node
@@ -732,13 +742,13 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,
 int __init cma_declare_contiguous_nid(phys_addr_t base,
 			phys_addr_t size, phys_addr_t limit,
 			phys_addr_t alignment, unsigned int order_per_bit,
-			bool fixed, const char *name, struct cma **res_cma,
-			int nid)
+			unsigned long flags, const char *name,
+			struct cma **res_cma, int nid)
 {
 	int ret;
 
 	ret = __cma_declare_contiguous_nid(&base, size, limit, alignment,
-			order_per_bit, fixed, name, res_cma, nid);
+			order_per_bit, flags, name, res_cma, nid);
 	if (ret != 0)
 		pr_err("Failed to reserve %ld MiB\n",
 				(unsigned long)size / SZ_1M);
mm/hugetlb_cma.c +1 -1
index f58ef4969e7a..71d0e9a048d4 100644
@@ -221,7 +221,7 @@ void __init hugetlb_cma_reserve(int order)
 		 * huge page demotion.
 		 */
 		res = cma_declare_contiguous_multi(size, PAGE_SIZE << order,
-					HUGETLB_PAGE_ORDER, name,
+					HUGETLB_PAGE_ORDER, 0, name,
 					&hugetlb_cma[nid], nid);
 		if (res) {
 			pr_warn("hugetlb_cma: reservation failed: err %d, node %d",
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
#6 lore raw

Re: [RFC PATCH 03/12] mm/cma: add flags argument to init functions

kernel test robot noticed the following build warnings:

Hi Frank,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:

[auto build test WARNING on powerpc/next]
[also build test WARNING on powerpc/fixes powerpc/topic/ppc-kvm s390/features trace/for-next linus/master v6.17-rc6 next-20250916] [cannot apply to akpm-mm/mm-everything]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Frank-van-der-Linden/mm-cma-add-tunable-for-CMA-fallback-limit/20250916-035514
base:   https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next
patch link:    https://lore.kernel.org/r/20250915195153.462039-4-fvdl%40google.com
patch subject: [RFC PATCH 03/12] mm/cma: add flags argument to init functions
config: i386-buildonly-randconfig-001-20250917 (https://download.01.org/0day-ci/archive/20250917/[email protected]/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250917/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> Warning: mm/cma.c:271 function parameter 'flags' not described in 'cma_init_reserved_mem'
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
Acked-by on v1 3/12
#7 lore raw

Re: [RFC PATCH 03/12] mm/cma: add flags argument to init functions

In reply to Frank van der Linden
Quoted 10 lines · Frank van der Linden

Add a flags argument to the various CMA init functions, as there will be a need to pass in more flags to control init and runtime behavior other than just the current 'fixed' argument.

Replace the fixed argument with a flags argument, and adapt callers.

I haven't gotten to the point in the series yet that shows why you need this, but I suppose this looks reasonable?

Signature · 1 line
All Rights Reversed.
#8 lore raw

RFC PATCH 4/12 mm/cma: keep a global sorted list of CMA ranges

In reply to Frank van der Linden

In order to walk through CMA areas efficiently, it is useful to keep a global sorted list of ranges.

Create this list when activating the areas.

Since users of this list may want to reference the CMA area the range came from, there needs to be a link from the range to that area. So, store a pointer to the CMA structure in the cma_memrange structure. This also reduces the number of arguments to a few internal functions.

---
 mm/cma.c | 72 ++++++++++++++++++++++++++++++++++++++++++--------------
 mm/cma.h |  6 ++---
 2 files changed, 57 insertions(+), 21 deletions(-)
mm/cma.c +54 -18
index 00d8d365f0b5..1f5a7bfc9152 100644
@@ -23,6 +23,7 @@
 #include <linux/sizes.h>
 #include <linux/slab.h>
 #include <linux/string_choices.h>
+#include <linux/sort.h>
 #include <linux/log2.h>
 #include <linux/cma.h>
 #include <linux/highmem.h>
@@ -65,12 +66,11 @@ static unsigned long cma_bitmap_aligned_mask(const struct cma *cma,
  * Find the offset of the base PFN from the specified align_order.
  * The value returned is represented in order_per_bits.
  */
-static unsigned long cma_bitmap_aligned_offset(const struct cma *cma,
-					       const struct cma_memrange *cmr,
+static unsigned long cma_bitmap_aligned_offset(const struct cma_memrange *cmr,
 					       unsigned int align_order)
 {
 	return (cmr->base_pfn & ((1UL << align_order) - 1))
-		>> cma->order_per_bit;
+		>> cmr->cma->order_per_bit;
 }
 
 static unsigned long cma_bitmap_pages_to_bits(const struct cma *cma,
@@ -79,11 +79,12 @@ static unsigned long cma_bitmap_pages_to_bits(const struct cma *cma,
 	return ALIGN(pages, 1UL << cma->order_per_bit) >> cma->order_per_bit;
 }
 
-static void cma_clear_bitmap(struct cma *cma, const struct cma_memrange *cmr,
+static void cma_clear_bitmap(const struct cma_memrange *cmr,
 			     unsigned long pfn, unsigned long count)
 {
 	unsigned long bitmap_no, bitmap_count;
 	unsigned long flags;
+	struct cma *cma = cmr->cma;
 
 	bitmap_no = (pfn - cmr->base_pfn) >> cma->order_per_bit;
 	bitmap_count = cma_bitmap_pages_to_bits(cma, count);
@@ -147,8 +148,7 @@ static void __init cma_activate_area(struct cma *cma)
 	for (allocrange = 0; allocrange < cma->nranges; allocrange++) {
 		cmr = &cma->ranges[allocrange];
 		early_pfn[allocrange] = cmr->early_pfn;
-		cmr->bitmap = bitmap_zalloc(cma_bitmap_maxno(cma, cmr),
-					    GFP_KERNEL);
+		cmr->bitmap = bitmap_zalloc(cma_bitmap_maxno(cmr), GFP_KERNEL);
 		if (!cmr->bitmap)
 			goto cleanup;
 	}
@@ -199,12 +199,45 @@ static void __init cma_activate_area(struct cma *cma)
 	pr_err("CMA area %s could not be activated\n", cma->name);
 }
 
+static struct cma_memrange **cma_ranges;
+static int cma_nranges;
+
+static int cmprange(const void *a, const void *b)
+{
+	struct cma_memrange *r1, *r2;
+
+	r1 = *(struct cma_memrange **)a;
+	r2 = *(struct cma_memrange **)b;
+
+	if (r1->base_pfn < r2->base_pfn)
+		return -1;
+	return r1->base_pfn - r2->base_pfn;
+}
+
 static int __init cma_init_reserved_areas(void)
 {
-	int i;
+	int i, r, nranges;
+	struct cma *cma;
+	struct cma_memrange *cmr;
+
+	nranges = 0;
+	for (i = 0; i < cma_area_count; i++) {
+		cma = &cma_areas[i];
+		nranges += cma->nranges;
+		cma_activate_area(cma);
+	}
+
+	cma_ranges = kcalloc(nranges, sizeof(*cma_ranges), GFP_KERNEL);
+	cma_nranges = 0;
+	for (i = 0; i < cma_area_count; i++) {
+		cma = &cma_areas[i];
+		for (r = 0; r < cma->nranges; r++) {
+			cmr = &cma->ranges[r];
+			cma_ranges[cma_nranges++] = cmr;
+		}
+	}
 
-	for (i = 0; i < cma_area_count; i++)
-		cma_activate_area(&cma_areas[i]);
+	sort(cma_ranges, cma_nranges, sizeof(*cma_ranges), cmprange, NULL);
 
 	return 0;
 }
@@ -297,6 +330,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size,
 	cma->ranges[0].base_pfn = PFN_DOWN(base);
 	cma->ranges[0].early_pfn = PFN_DOWN(base);
 	cma->ranges[0].count = cma->count;
+	cma->ranges[0].cma = cma;
 	cma->nranges = 1;
 	cma->nid = NUMA_NO_NODE;
 
@@ -687,6 +721,7 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,
 		cmrp->base_pfn = PHYS_PFN(mlp->base);
 		cmrp->early_pfn = cmrp->base_pfn;
 		cmrp->count = size >> PAGE_SHIFT;
+		cmrp->cma = cma;
 
 		sizeleft -= size;
 		if (sizeleft == 0)
@@ -772,7 +807,7 @@ static void cma_debug_show_areas(struct cma *cma)
 	for (r = 0; r < cma->nranges; r++) {
 		cmr = &cma->ranges[r];
 
-		nbits = cma_bitmap_maxno(cma, cmr);
+		nbits = cma_bitmap_maxno(cmr);
 
 		pr_info("range %d: ", r);
 		for_each_clear_bitrange(start, end, cmr->bitmap, nbits) {
@@ -786,9 +821,9 @@ static void cma_debug_show_areas(struct cma *cma)
 	spin_unlock_irq(&cma->lock);
 }
 
-static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr,
-				unsigned long count, unsigned int align,
-				struct page **pagep, gfp_t gfp)
+static int cma_range_alloc(struct cma_memrange *cmr,
+			   unsigned long count, unsigned int align,
+			   struct page **pagep, gfp_t gfp)
 {
 	unsigned long mask, offset;
 	unsigned long pfn = -1;
@@ -796,10 +831,11 @@ static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr,
 	unsigned long bitmap_maxno, bitmap_no, bitmap_count;
 	int ret = -EBUSY;
 	struct page *page = NULL;
+	struct cma *cma = cmr->cma;
 
 	mask = cma_bitmap_aligned_mask(cma, align);
-	offset = cma_bitmap_aligned_offset(cma, cmr, align);
-	bitmap_maxno = cma_bitmap_maxno(cma, cmr);
+	offset = cma_bitmap_aligned_offset(cmr, align);
+	bitmap_maxno = cma_bitmap_maxno(cmr);
 	bitmap_count = cma_bitmap_pages_to_bits(cma, count);
 
 	if (bitmap_count > bitmap_maxno)
@@ -840,7 +876,7 @@ static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr,
 			break;
 		}
 
-		cma_clear_bitmap(cma, cmr, pfn, count);
+		cma_clear_bitmap(cmr, pfn, count);
 		if (ret != -EBUSY)
 			break;
 
@@ -879,7 +915,7 @@ static struct page *__cma_alloc(struct cma *cma, unsigned long count,
 	for (r = 0; r < cma->nranges; r++) {
 		page = NULL;
 
-		ret = cma_range_alloc(cma, &cma->ranges[r], count, align,
+		ret = cma_range_alloc(&cma->ranges[r], count, align,
 				       &page, gfp);
 		if (ret != -EBUSY || page)
 			break;
@@ -1011,7 +1047,7 @@ bool cma_release(struct cma *cma, const struct page *pages,
 		return false;
 
 	free_contig_range(pfn, count);
-	cma_clear_bitmap(cma, cmr, pfn, count);
+	cma_clear_bitmap(cmr, pfn, count);
 	cma_sysfs_account_release_pages(cma, count);
 	trace_cma_release(cma->name, pfn, pages, count);
 
mm/cma.h +3 -3
index 25b696774c6a..384d1109d438 100644
@@ -30,6 +30,7 @@ struct cma_memrange {
 		unsigned long early_pfn;
 		unsigned long *bitmap;
 	};
+	struct cma *cma;
 #ifdef CONFIG_CMA_DEBUGFS
 	struct debugfs_u32_array dfs_bitmap;
 #endif
@@ -67,10 +68,9 @@ struct cma {
 extern struct cma cma_areas[MAX_CMA_AREAS];
 extern unsigned int cma_area_count;
 
-static inline unsigned long cma_bitmap_maxno(struct cma *cma,
-		struct cma_memrange *cmr)
+static inline unsigned long cma_bitmap_maxno(struct cma_memrange *cmr)
 {
-	return cmr->count >> cma->order_per_bit;
+	return cmr->count >> cmr->cma->order_per_bit;
 }
 
 #ifdef CONFIG_CMA_SYSFS
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
#9 lore raw

Re: [RFC PATCH 04/12] mm/cma: keep a global sorted list of CMA ranges

kernel test robot noticed the following build errors:

Hi Frank,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on powerpc/next]
[also build test ERROR on powerpc/fixes powerpc/topic/ppc-kvm s390/features linus/master v6.17-rc6 next-20250916] [cannot apply to akpm-mm/mm-everything]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Frank-van-der-Linden/mm-cma-add-tunable-for-CMA-fallback-limit/20250916-035514
base:   https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next
patch link:    https://lore.kernel.org/r/20250915195153.462039-5-fvdl%40google.com
patch subject: [RFC PATCH 04/12] mm/cma: keep a global sorted list of CMA ranges
config: i386-buildonly-randconfig-006-20250917 (https://download.01.org/0day-ci/archive/20250917/[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250917/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> mm/cma_debug.c:58:40: error: too many arguments to function call, expected single argument 'cmr', have 2 arguments
      58 |                 bitmap_maxno = cma_bitmap_maxno(cma, cmr);
         |                                ~~~~~~~~~~~~~~~~      ^~~
   mm/cma.h:71:29: note: 'cma_bitmap_maxno' declared here
      71 | static inline unsigned long cma_bitmap_maxno(struct cma_memrange *cmr)
         |                             ^                ~~~~~~~~~~~~~~~~~~~~~~~~
   mm/cma_debug.c:187:39: error: too many arguments to function call, expected single argument 'cmr', have 2 arguments
     187 |                         DIV_ROUND_UP(cma_bitmap_maxno(cma, cmr),
         |                                      ~~~~~~~~~~~~~~~~      ^~~
   include/linux/math.h:49:22: note: expanded from macro 'DIV_ROUND_UP'
      49 | #define DIV_ROUND_UP __KERNEL_DIV_ROUND_UP
         |                      ^
   include/uapi/linux/const.h:51:40: note: expanded from macro '__KERNEL_DIV_ROUND_UP'
      51 | #define __KERNEL_DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d))
         |                                        ^

mm/cma.h:71:29: note: 'cma_bitmap_maxno' declared here
71 | static inline unsigned long cma_bitmap_maxno(struct cma_memrange *cmr)
| ^ ~~~~~~~~~~~~~~~~~~~~~~~~
2 errors generated.

vim +/cmr +58 mm/cma_debug.c

2e32b947606daa Dmitry Safonov       2015-04-15  45  
2e32b947606daa Dmitry Safonov       2015-04-15  46  static int cma_maxchunk_get(void *data, u64 *val)
2e32b947606daa Dmitry Safonov       2015-04-15  47  {
2e32b947606daa Dmitry Safonov       2015-04-15  48  	struct cma *cma = data;
c009da4258f988 Frank van der Linden 2025-02-28  49  	struct cma_memrange *cmr;
2e32b947606daa Dmitry Safonov       2015-04-15  50  	unsigned long maxchunk = 0;
c009da4258f988 Frank van der Linden 2025-02-28  51  	unsigned long start, end;
c009da4258f988 Frank van der Linden 2025-02-28  52  	unsigned long bitmap_maxno;
c009da4258f988 Frank van der Linden 2025-02-28  53  	int r;
2e32b947606daa Dmitry Safonov       2015-04-15  54  
0ef7dcac998fef Mike Kravetz         2021-05-04  55  	spin_lock_irq(&cma->lock);
c009da4258f988 Frank van der Linden 2025-02-28  56  	for (r = 0; r < cma->nranges; r++) {
c009da4258f988 Frank van der Linden 2025-02-28  57  		cmr = &cma->ranges[r];
c009da4258f988 Frank van der Linden 2025-02-28 @58  		bitmap_maxno = cma_bitmap_maxno(cma, cmr);
c79147e4b02fa3 Yury Norov (NVIDIA   2025-07-19  59) 		for_each_clear_bitrange(start, end, cmr->bitmap, bitmap_maxno)
2e32b947606daa Dmitry Safonov       2015-04-15  60  			maxchunk = max(end - start, maxchunk);
2e32b947606daa Dmitry Safonov       2015-04-15  61  	}
0ef7dcac998fef Mike Kravetz         2021-05-04  62  	spin_unlock_irq(&cma->lock);
2e32b947606daa Dmitry Safonov       2015-04-15  63  	*val = (u64)maxchunk << cma->order_per_bit;
2e32b947606daa Dmitry Safonov       2015-04-15  64  
2e32b947606daa Dmitry Safonov       2015-04-15  65  	return 0;
2e32b947606daa Dmitry Safonov       2015-04-15  66  }
a9ea242a063c62 zhong jiang          2019-11-30  67  DEFINE_DEBUGFS_ATTRIBUTE(cma_maxchunk_fops, cma_maxchunk_get, NULL, "%llu\n");
2e32b947606daa Dmitry Safonov       2015-04-15  68  
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#10 lore raw

Re: [RFC PATCH 04/12] mm/cma: keep a global sorted list of CMA ranges

In reply to Frank van der Linden
Quoted 42 lines · Frank van der Linden

In order to walk through CMA areas efficiently, it is useful to keep a global sorted list of ranges.

Create this list when activating the areas.

Since users of this list may want to reference the CMA area the range came from, there needs to be a link from the range to that area. So, store a pointer to the CMA structure in the cma_memrange structure. This also reduces the number of arguments to a few internal functions.

 static int __init cma_init_reserved_areas(void)
 {

mm/cma.c +12 -1
-	int i;
+	int i, r, nranges;
+	struct cma *cma;
+	struct cma_memrange *cmr;
+
+	nranges = 0;
+	for (i = 0; i < cma_area_count; i++) {
+		cma = &cma_areas[i];
+		nranges += cma->nranges;
+		cma_activate_area(cma);
+	}
+
+	cma_ranges = kcalloc(nranges, sizeof(*cma_ranges),

GFP_KERNEL);

mm/cma.c +8 -0
+	cma_nranges = 0;
+	for (i = 0; i < cma_area_count; i++) {
+		cma = &cma_areas[i];
+		for (r = 0; r < cma->nranges; r++) {
+			cmr = &cma->ranges[r];
+			cma_ranges[cma_nranges++] = cmr;
+		}
+	}

mm/cma.c +1 -2
-	for (i = 0; i < cma_area_count; i++)
-		cma_activate_area(&cma_areas[i]);
+	sort(cma_ranges, cma_nranges, sizeof(*cma_ranges), cmprange,
Quoted 2 lines

NULL);

I am guessing that this is safe because cma_init_reserved_areas is an initcall, which is only called once?

Is that correct?

Is it worth documenting why this function creates
a sorted array of CMA ranges?

Quoted 4 lines


  return 0;
 }

Signature · 1 line
All Rights Reversed.
#11 lore raw

RFC PATCH 5/12 mm/cma: add helper functions for CMA balancing

In reply to Frank van der Linden

Add some CMA helper functions to assist CMA balancing. They are:

cma_get_available.
- Returns the number of available pages in a CMA area

cma_numranges
- Returns the total number of CMA ranges.

cma_next_balance_pagerange
- Get the next CMA page range in a zone that has is available as a target for CMA balancing. This means a range that consists of CMA pageblocks that are managed by the buddy allocator (not allocated through cma_alloc). The array of CMA ranges is walked top down.

cma_next_noncma_pagerange
- Get the next non-CMA page range in a zone. The zone is traversed bottom up.

---
 include/linux/cma.h |  30 +++++++++
 mm/cma.c            | 161 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 191 insertions(+)
include/linux/cma.h +30 -0
index ec48f2a11f1d..0504580d61d0 100644
@@ -37,6 +37,7 @@ enum cma_flags {
 #define CMA_INIT_FLAGS (CMA_FIXED|CMA_RESERVE_PAGES_ON_ERROR)
 
 struct cma;
+struct zone;
 
 extern unsigned long totalcma_pages;
 extern phys_addr_t cma_get_base(const struct cma *cma);
@@ -79,6 +80,12 @@ extern void cma_reserve_pages_on_error(struct cma *cma);
 struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp);
 bool cma_free_folio(struct cma *cma, const struct folio *folio);
 bool cma_validate_zones(struct cma *cma);
+int cma_numranges(void);
+unsigned long cma_get_available(const struct cma *cma);
+bool cma_next_balance_pagerange(struct zone *zone, struct cma *cma, int *rindex,
+			    unsigned long *startpfn, unsigned long *endpfn);
+bool cma_next_noncma_pagerange(struct zone *zone, int *rindex,
+			       unsigned long *startpfn, unsigned long *endpfn);
 #else
 static inline struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp)
 {
@@ -93,6 +100,29 @@ static inline bool cma_validate_zones(struct cma *cma)
 {
 	return false;
 }
+
+static inline int cma_numranges(void)
+{
+	return 0;
+}
+
+static inline unsigned long cma_get_available(const struct cma *cma)
+{
+	return 0;
+}
+
+static inline bool cma_next_balance_pagerange(struct zone *zone,
+			struct cma *cma, int *rindex, unsigned long *start_pfn,
+			unsigned long *end_pfn)
+{
+	return false;
+}
+
+static inline bool cma_next_noncma_pagerange(struct zone *zone, int *rindex,
+			     unsigned long *start_pfn, unsigned long *end_pfn)
+{
+	return false;
+}
 #endif
 
 #endif
mm/cma.c +161 -0

179 lines not on this page · raw message

Signature · 1 line
2.51.0.384.g4c02a37b29-goog
#12 lore raw

Re: [RFC PATCH 05/12] mm/cma: add helper functions for CMA balancing

Quoted 11 lines · Frank van der Linden

+/*
+ * Provide the next free range in a cma memory range, as derived
+ * from the bitmap.
+ *
+ * @cmr: memory range to scan
+ * @start_pfn: the beginning of the previous range
+ * @end_pfn: the end of the previous range, zero for the first call
+ *
+ * The caller can adjust *end_pfn end use it as a starting point.
+ */

The function documentation does not really make it clear that start_pfn and end_pfn are also used for the function OUTPUT, and that the return value indicates whether or not a next area was found.

Same for cma_next_noncma_pagerange. Other than that, this looks reasonable so far.

Signature · 1 line
All Rights Reversed.
#13 lore raw

RFC PATCH 6/12 mm/cma: define and act on CMA_BALANCE flag

In reply to Frank van der Linden

When the CMA_BALANCE flag is set for a CMA area, it means that it opts in to CMA balancing. This means two things:

1) It allows movable allocations to be migrated in to it in the case of a CMA inbalance (too much free memory in CMA pageblocks as compared to other pageblocks).
2) It is allocated top-down, so that compaction will end up migrating pages in to it. Doing this will make sure that compaction doesn't aggrevate a CMA imbalance, and that it won't fight with CMA balance migration from non-CMA to CMA.

---
 include/linux/cma.h |  4 +++-
 mm/cma.c            | 33 ++++++++++++++++++++++++++-------
 2 files changed, 29 insertions(+), 8 deletions(-)
include/linux/cma.h +3 -1
index 0504580d61d0..6e98a516b336 100644
@@ -26,6 +26,7 @@ enum cma_flags {
 	__CMA_ZONES_INVALID,
 	__CMA_ACTIVATED,
 	__CMA_FIXED,
+	__CMA_BALANCE,
 };
 
 #define CMA_RESERVE_PAGES_ON_ERROR	BIT(__CMA_RESERVE_PAGES_ON_ERROR)
@@ -33,8 +34,9 @@ enum cma_flags {
 #define CMA_ZONES_INVALID		BIT(__CMA_ZONES_INVALID)
 #define CMA_ACTIVATED			BIT(__CMA_ACTIVATED)
 #define CMA_FIXED			BIT(__CMA_FIXED)
+#define CMA_BALANCE			BIT(__CMA_BALANCE)
 
-#define CMA_INIT_FLAGS (CMA_FIXED|CMA_RESERVE_PAGES_ON_ERROR)
+#define CMA_INIT_FLAGS (CMA_FIXED|CMA_RESERVE_PAGES_ON_ERROR|CMA_BALANCE)
 
 struct cma;
 struct zone;
mm/cma.c +26 -7
index 53cb1833407b..6050d57f3c2e 100644
@@ -272,6 +272,9 @@ static bool cma_next_free_range(struct cma_memrange *cmr,
 static inline bool cma_should_balance_range(struct zone *zone,
 				      struct cma_memrange *cmr)
 {
+	if (!(cmr->cma->flags & CMA_BALANCE))
+		return false;
+
 	if (page_zone(pfn_to_page(cmr->base_pfn)) != zone)
 		return false;
 
@@ -527,6 +530,12 @@ static bool __init basecmp(struct cma_init_memrange *mlp,
 	return mlp->base < mrp->base;
 }
 
+static bool __init revbasecmp(struct cma_init_memrange *mlp,
+			   struct cma_init_memrange *mrp)
+{
+	return mlp->base > mrp->base;
+}
+
 /*
  * Helper function to create sorted lists.
  */
@@ -575,7 +584,8 @@ static int __init cma_fixed_reserve(phys_addr_t base, phys_addr_t size)
 }
 
 static phys_addr_t __init cma_alloc_mem(phys_addr_t base, phys_addr_t size,
-			phys_addr_t align, phys_addr_t limit, int nid)
+			phys_addr_t align, phys_addr_t limit, int nid,
+			unsigned long flags)
 {
 	phys_addr_t addr = 0;
 
@@ -588,7 +598,8 @@ static phys_addr_t __init cma_alloc_mem(phys_addr_t base, phys_addr_t size,
 	 * like DMA/DMA32.
 	 */
 #ifdef CONFIG_PHYS_ADDR_T_64BIT
-	if (!memblock_bottom_up() && limit >= SZ_4G + size) {
+	if (!(flags & CMA_BALANCE) && !memblock_bottom_up()
+			&& limit >= SZ_4G + size) {
 		memblock_set_bottom_up(true);
 		addr = memblock_alloc_range_nid(size, align, SZ_4G, limit,
 						nid, true);
@@ -695,7 +706,7 @@ static int __init __cma_declare_contiguous_nid(phys_addr_t *basep,
 		if (ret)
 			return ret;
 	} else {
-		base = cma_alloc_mem(base, size, alignment, limit, nid);
+		base = cma_alloc_mem(base, size, alignment, limit, nid, flags);
 		if (!base)
 			return -ENOMEM;
 
@@ -851,7 +862,10 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,
 	list_for_each_safe(mp, next, &ranges) {
 		mlp = list_entry(mp, struct cma_init_memrange, list);
 		list_del(mp);
-		list_insert_sorted(&final_ranges, mlp, basecmp);
+		if (flags & CMA_BALANCE)
+			list_insert_sorted(&final_ranges, mlp, revbasecmp);
+		else
+			list_insert_sorted(&final_ranges, mlp, basecmp);
 		sizesum += mlp->size;
 		if (sizesum >= total_size)
 			break;
@@ -866,7 +880,12 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,
 	list_for_each(mp, &final_ranges) {
 		mlp = list_entry(mp, struct cma_init_memrange, list);
 		size = min(sizeleft, mlp->size);
-		if (memblock_reserve(mlp->base, size)) {
+		if (flags & CMA_BALANCE)
+			start = (mlp->base + mlp->size - size);
+		else
+			start = mlp->base;
+
+		if (memblock_reserve(start, size)) {
 			/*
 			 * Unexpected error. Could go on to
 			 * the next one, but just abort to
@@ -877,9 +896,9 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,
 		}
 
 		pr_debug("created region %d: %016llx - %016llx\n",
-		    nr, (u64)mlp->base, (u64)mlp->base + size);
+		    nr, (u64)start, (u64)start + size);
 		cmrp = &cma->ranges[nr++];
-		cmrp->base_pfn = PHYS_PFN(mlp->base);
+		cmrp->base_pfn = PHYS_PFN(start);
 		cmrp->early_pfn = cmrp->base_pfn;
 		cmrp->count = size >> PAGE_SHIFT;
 		cmrp->cma = cma;
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
#14 lore raw

Re: [RFC PATCH 06/12] mm/cma: define and act on CMA_BALANCE flag

Quoted 13 lines · Frank van der Linden
(quoted diff) +4 -1
@@ -851,7 +862,10 @@ int __init
 cma_declare_contiguous_multi(phys_addr_t total_size,
  	list_for_each_safe(mp, next, &ranges) {
  		mlp = list_entry(mp, struct cma_init_memrange,
 list);
  		list_del(mp);
-		list_insert_sorted(&final_ranges, mlp, basecmp);
+		if (flags & CMA_BALANCE)
+			list_insert_sorted(&final_ranges, mlp,
 revbasecmp);
+		else
+			list_insert_sorted(&final_ranges, mlp,
Quoted 4 lines

basecmp);
  sizesum += mlp->size;
  if (sizesum >= total_size)
  break;

Wait, what is this supposed to do when presented with a mix of CMA_BALANCE and !CMA_BALANCE ranges?

I can't convince myself this would always keep the CMA_BALANCE range near the top of the list, and the others near the bottom.

Also, does the regular compaction code pay any attention to the order of this list?

I have the feeling I am overlooking something big in how this patch is supposed to work, but I am not sure what that would be...

Quoted 17 lines
mm/cma.c +6 -1
@@ -866,7 +880,12 @@ int __init
 cma_declare_contiguous_multi(phys_addr_t total_size,
  	list_for_each(mp, &final_ranges) {
  		mlp = list_entry(mp, struct cma_init_memrange,
 list);
  		size = min(sizeleft, mlp->size);
-		if (memblock_reserve(mlp->base, size)) {
+		if (flags & CMA_BALANCE)
+			start = (mlp->base + mlp->size - size);
+		else
+			start = mlp->base;
+
+		if (memblock_reserve(start, size)) {
  			/*

  * Unexpected error. Could go on to
  * the next one, but just abort to

Signature · 1 line
All Rights Reversed.
#15 lore raw

RFC PATCH 7/12 mm/compaction: optionally use a different isolate function

In reply to Frank van der Linden

For CMA balancing, the compaction migration hooks can largely be reused, except a different function to isolate free pages is needed.

So, add a pointer to the isolation function in the compact_control structure. If it's not NULL, use it, else use isolate_freepages as usual.

No functional change.

---
 mm/compaction.c | 5 ++++-
 mm/internal.h   | 5 +++++
 2 files changed, 9 insertions(+), 1 deletion(-)
mm/compaction.c +4 -1
index bf021b31c7ec..6a2c06e356c5 100644
@@ -1813,7 +1813,10 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long da
 	if (start_order == NR_PAGE_ORDERS) {
 		if (has_isolated_pages)
 			return NULL;
-		isolate_freepages(cc);
+		if (cc->isolate_freepages)
+			cc->isolate_freepages(cc);
+		else
+			isolate_freepages(cc);
 		has_isolated_pages = true;
 		goto again;
 	}
mm/internal.h +5 -0
index 45b725c3dc03..7916d8be8922 100644
@@ -873,6 +873,11 @@ struct compact_control {
 	unsigned int nr_freepages;	/* Number of isolated free pages */
 	unsigned int nr_migratepages;	/* Number of pages to migrate */
 	unsigned long free_pfn;		/* isolate_freepages search base */
+	/*
+	 * Function to use to isolate free pages, if applicable. If NULL,
+	 * default to isolate_freepages().
+	 */
+	void (*isolate_freepages)(struct compact_control *cc);
 	/*
 	 * Acts as an in/out parameter to page isolation for migration.
 	 * isolate_migratepages uses it as a search base.
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Acked-by on v1 7/12
#16 lore raw

Re: [RFC PATCH 07/12] mm/compaction: optionally use a different isolate function

Quoted 4 lines · Frank van der Linden

For CMA balancing, the compaction migration hooks can largely be reused, except a different function to isolate free pages is needed.

Quoted 8 lines

So, add a pointer to the isolation function in the compact_control structure. If it's not NULL, use it, else use isolate_freepages as usual.

No functional change.

A custom isolation function can certainly solve some of the issues!

Signature · 1 line
All Rights Reversed.
#17 lore raw

RFC PATCH 8/12 mm/compaction: simplify isolation order checks a bit

In reply to Frank van der Linden

The code to isolate pages for migration always checked both cc->alloc_contig and skip_isolation_on_order to determine whether a page could be isolated for migration.

Simplify this a little bit by moving the cc->alloc_contig check in to skip_isolation_on_order. Also rename alloc_contig to migrate_large, since there will be an additional user (CMA balancing) of this field soon, not just alloc_contig_range.

No functional change.

---
 mm/compaction.c | 26 ++++++++++++++------------
 mm/internal.h   |  2 +-
 mm/page_alloc.c |  2 +-
 3 files changed, 16 insertions(+), 14 deletions(-)
mm/compaction.c +14 -12
index 6a2c06e356c5..2e6c30f50b89 100644
@@ -793,13 +793,15 @@ static bool too_many_isolated(struct compact_control *cc)
 /**
  * skip_isolation_on_order() - determine when to skip folio isolation based on
  *			       folio order and compaction target order
+ * @cc:			compact control structure containing target order
  * @order:		to-be-isolated folio order
- * @target_order:	compaction target order
  *
  * This avoids unnecessary folio isolations during compaction.
  */
-static bool skip_isolation_on_order(int order, int target_order)
+static bool skip_isolation_on_order(struct compact_control *cc, int order)
 {
+	if (cc->migrate_large)
+		return false;
 	/*
 	 * Unless we are performing global compaction (i.e.,
 	 * is_via_compact_memory), skip any folios that are larger than the
@@ -807,7 +809,7 @@ static bool skip_isolation_on_order(int order, int target_order)
 	 * the desired target_order, so migrating this folio would likely fail
 	 * later.
 	 */
-	if (!is_via_compact_memory(target_order) && order >= target_order)
+	if (!is_via_compact_memory(cc->order) && order >= cc->order)
 		return true;
 	/*
 	 * We limit memory compaction to pageblocks and won't try
@@ -850,6 +852,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 	unsigned long next_skip_pfn = 0;
 	bool skip_updated = false;
 	int ret = 0;
+	unsigned int order;
 
 	cc->migrate_pfn = low_pfn;
 
@@ -948,13 +951,13 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 		}
 
 		if (PageHuge(page)) {
-			const unsigned int order = compound_order(page);
 			/*
 			 * skip hugetlbfs if we are not compacting for pages
 			 * bigger than its order. THPs and other compound pages
 			 * are handled below.
 			 */
-			if (!cc->alloc_contig) {
+			if (!cc->migrate_large) {
+				order = compound_order(page);
 
 				if (order <= MAX_PAGE_ORDER) {
 					low_pfn += (1UL << order) - 1;
@@ -962,7 +965,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 				}
 				goto isolate_fail;
 			}
-			/* for alloc_contig case */
+			/* for migrate_large case */
 			if (locked) {
 				unlock_page_lruvec_irqrestore(locked, flags);
 				locked = NULL;
@@ -1030,11 +1033,11 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 		 * skip them at once. The check is racy, but we can consider
 		 * only valid values and the only danger is skipping too much.
 		 */
-		if (PageCompound(page) && !cc->alloc_contig) {
-			const unsigned int order = compound_order(page);
+		if (PageCompound(page)) {
+			order = compound_order(page);
 
 			/* Skip based on page order and compaction target order. */
-			if (skip_isolation_on_order(order, cc->order)) {
+			if (skip_isolation_on_order(cc, order)) {
 				if (order <= MAX_PAGE_ORDER) {
 					low_pfn += (1UL << order) - 1;
 					nr_scanned += (1UL << order) - 1;
@@ -1182,9 +1185,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
 			/*
 			 * Check LRU folio order under the lock
 			 */
-			if (unlikely(skip_isolation_on_order(folio_order(folio),
-							     cc->order) &&
-				     !cc->alloc_contig)) {
+			order = folio_order(folio);
+			if (unlikely(skip_isolation_on_order(cc, order))) {
 				low_pfn += folio_nr_pages(folio) - 1;
 				nr_scanned += folio_nr_pages(folio) - 1;
 				folio_set_lru(folio);
mm/internal.h +1 -1
index 7916d8be8922..ffcb3aec05ed 100644
@@ -909,7 +909,7 @@ struct compact_control {
 					 * isolation or migration failures to
 					 * ensure forward progress.
 					 */
-	bool alloc_contig;		/* alloc_contig_range allocation */
+	bool migrate_large;             /* Always migrate large/huge pages */
 };
 
 /*
mm/page_alloc.c +1 -1
index d3966d31c039..dc59aaa63ae6 100644
@@ -6903,7 +6903,7 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end,
 		.mode = MIGRATE_SYNC,
 		.ignore_skip_hint = true,
 		.no_set_skip_hint = true,
-		.alloc_contig = true,
+		.migrate_large = true,
 	};
 	INIT_LIST_HEAD(&cc.migratepages);
 	enum pb_isolate_mode mode = (alloc_flags & ACR_FLAGS_CMA) ?
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Acked-by on v1 8/12
#18 lore raw

Re: [RFC PATCH 08/12] mm/compaction: simplify isolation order checks a bit

Quoted 4 lines · Frank van der Linden

The code to isolate pages for migration always checked both cc->alloc_contig and skip_isolation_on_order to determine whether a page could be isolated for migration.

Quoted 9 lines

Simplify this a little bit by moving the cc->alloc_contig check in to skip_isolation_on_order. Also rename alloc_contig to migrate_large, since there will be an additional user (CMA balancing) of this field soon, not just alloc_contig_range.

No functional change.

Signature · 1 line
All Rights Reversed.
#19 lore raw

RFC PATCH 9/12 mm/cma: introduce CMA balancing

In reply to Frank van der Linden

A longstanding problem with having a lot of CMA pageblocks in the system (through hugetlb_cma), is that this limits the amount of memory that the kernel can use for its allocations. Kernel allocations are unmovable and can not come from CMA pageblocks.

This can lead to situations where kernel allocations cause OOMs, when in fact there might still enough memory available. There isn't much that can be done if the non-CMA part of memory is already taken up by unmovable allocations. That scenario can be considered a misconfigured system. But if there are movable allocations in the non-CMA areas, they are unnecessarily taking away space from the kernel.

Currently, the page allocator tries to avoid this scenario by allocating from CMA first if more than half of free pages in a zone come from CMA. But that's not a guarantee.

For example, take the case where a lot of memory is being taken up by 1G hugetlb pages, allocated from hugetlb_cma, and that the hugetlb_cma area has been fully used by hugetlbfs. This means that new movable allocations will land in the non-CMA part of memory, and that the kernel may come under memory pressure. If those allocations are long-lasting, freeing up hugetlb pages will not reduce that pressure, since the kernel can't use the new space, and the long-lasting allocations residing in non-CMA memory will stay put.

To counter this issue, introduce interfaces to explicitly move pages in to CMA areas. The number of pages moved depends on cma_first_limit. It will use that percentage to calculate the target number of pages that should be moved.

A later commit will call one of these interfaces to move pages to CMA if needed, after CMA-allocated hugetlb pages have been freed.

---
 include/linux/migrate_mode.h   |   1 +
 include/trace/events/migrate.h |   3 +-
 mm/compaction.c                | 168 +++++++++++++++++++++++++++++++++
 mm/internal.h                  |   4 +
 4 files changed, 175 insertions(+), 1 deletion(-)
include/linux/migrate_mode.h +1 -0
index 265c4328b36a..3e235499cd73 100644
@@ -25,6 +25,7 @@ enum migrate_reason {
 	MR_LONGTERM_PIN,
 	MR_DEMOTION,
 	MR_DAMON,
+	MR_CMA_BALANCE,
 	MR_TYPES
 };
 
include/trace/events/migrate.h +2 -1
index cd01dd7b3640..53d669ee26be 100644
@@ -23,7 +23,8 @@
 	EM( MR_CONTIG_RANGE,	"contig_range")			\
 	EM( MR_LONGTERM_PIN,	"longterm_pin")			\
 	EM( MR_DEMOTION,	"demotion")			\
-	EMe(MR_DAMON,		"damon")
+	EM( MR_DAMON,		"damon")			\
+	EMe(MR_CMA_BALANCE,	"cma_balance")
 
 /*
  * First define the enums in the above macros to be exported to userspace
mm/compaction.c +168 -0

180 lines not on this page · raw message

mm/internal.h +4 -0
index ffcb3aec05ed..7dcaf7214683 100644
@@ -857,6 +857,8 @@ void memmap_init_range(unsigned long, int, unsigned long, unsigned long,
 
 #if defined CONFIG_COMPACTION || defined CONFIG_CMA
 
+struct cma;
+
 /*
  * in mm/compaction.c
  */
@@ -887,6 +889,7 @@ struct compact_control {
 	unsigned long migrate_pfn;
 	unsigned long fast_start_pfn;	/* a pfn to start linear scan from */
 	struct zone *zone;
+	struct cma *cma;		/* if moving to a specific CMA area */
 	unsigned long total_migrate_scanned;
 	unsigned long total_free_scanned;
 	unsigned short fast_search_fail;/* failures to use free list searches */
@@ -938,6 +941,7 @@ struct cma;
 #ifdef CONFIG_CMA
 void *cma_reserve_early(struct cma *cma, unsigned long size);
 void init_cma_pageblock(struct page *page);
+void balance_node_cma(int nid, struct cma *cma);
 #else
 static inline void *cma_reserve_early(struct cma *cma, unsigned long size)
 {
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Reviewed-by on v1 9/12
#20 lore raw

Re: [RFC PATCH 09/12] mm/cma: introduce CMA balancing

Quoted 7 lines · Frank van der Linden

A longstanding problem with having a lot of CMA pageblocks in the system (through hugetlb_cma), is that this limits the amount of memory that the kernel can use for its allocations. Kernel allocations are unmovable and can not come from CMA pageblocks.

This can lead to situations where kernel allocations cause OOMs, when in fact there might still enough memory available.

We are seeing this situation happen even when the CMA region is relatively small.

When a system has multiple cgroups, and has been reclaiming the non-CMA memory for a particular cgroup for long enough, at some point essentially all the remaining reclaimable memory for that cgroup can be in CMA, and you get early OOM kills.

Quoted 7 lines

To counter this issue, introduce interfaces to explicitly move pages in to CMA areas. The number of pages moved depends on cma_first_limit. It will use that percentage to calculate the target number of pages that should be moved.

I've been working on some code for this situation as well, but your code seems to handle a lot more corner cases than the small hack I was working on.

I hope this issue can be fixed upstream soon-ish :)

Quoted 3 lines

A later commit will call one of these interfaces to move pages to CMA if needed, after CMA-allocated hugetlb pages have been freed.

Is that the right place?

The CMA region will also be used by regular 4kB pages and 2MB THPs, and we may also need to do CMA balancing if the cgroup is up against its memory limit, needs to do a kernel allocation, but its remaining freeable memory is in movable allocations in the CMA region.

I have the hook for migrating pages to CMA on the page reclaim side, when the reclaim code notices that it is doing direct reclaim on behalf of a non-movable (kernel) allocation, but it has been reclaiming a bunch of CMA pages.

Hugetlb is not the only case triggering this issue.

Quoted 3 lines
Signature · 1 line
All Rights Reversed.
#21 lore raw

RFC PATCH 10/12 mm/hugetlb: do explicit CMA balancing

In reply to Frank van der Linden

CMA areas are normally not very large, but HugeTLB CMA is an exception. hugetlb_cma, used for 'gigantic' pages (usually 1G), can take up many gigabytes of memory.

As such, it is potentially the largest source of 'false OOM' conditions, situations where the kernel runs out of space for unmovable allocations, because it can't allocate from CMA pageblocks, and non-CMA memory has been tied up by other movable allocations.

The normal use case of hugetlb_cma is a system where 1G hugetlb pages are sometimes, but not always, needed, so they need to be created and freed dynamically. As such, the best time to address CMA memory imbalances is when CMA hugetlb pages are freed, making multiples of 1G available as buddy managed CMA pageblocks. That is a good time to check if movable allocations fron non-CMA pageblocks should be moved to CMA pageblocks to give the kernel more breathing space.

Do this by calling balance_node_cma on either the hugetlb CMA area for the node that just had its number of hugetlb pages reduced, or for all hugetlb CMA areas if the reduction was not node-specific.

To have the CMA balancing code act on the hugetlb CMA areas, set the CMA_BALANCE flag when creating them.

---
 mm/hugetlb.c     | 14 ++++++++------
 mm/hugetlb_cma.c | 16 ++++++++++++++++
 mm/hugetlb_cma.h |  5 +++++
 3 files changed, 29 insertions(+), 6 deletions(-)
mm/hugetlb.c +8 -6
index eed59cfb5d21..611655876f60 100644
@@ -3971,12 +3971,14 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid,
 
 		list_add(&folio->lru, &page_list);
 	}
-	/* free the pages after dropping lock */
-	spin_unlock_irq(&hugetlb_lock);
-	update_and_free_pages_bulk(h, &page_list);
-	flush_free_hpage_work(h);
-	spin_lock_irq(&hugetlb_lock);
-
+	if (!list_empty(&page_list)) {
+		/* free the pages after dropping lock */
+		spin_unlock_irq(&hugetlb_lock);
+		update_and_free_pages_bulk(h, &page_list);
+		flush_free_hpage_work(h);
+		hugetlb_cma_balance(nid);
+		spin_lock_irq(&hugetlb_lock);
+	}
 	while (count < persistent_huge_pages(h)) {
 		if (!adjust_pool_surplus(h, nodes_allowed, 1))
 			break;
mm/hugetlb_cma.c +16 -0
index 71d0e9a048d4..c0396d35b5bf 100644
@@ -276,3 +276,19 @@ bool __init hugetlb_early_cma(struct hstate *h)
 
 	return hstate_is_gigantic(h) && hugetlb_cma_only;
 }
+
+void hugetlb_cma_balance(int nid)
+{
+	int node;
+
+	if (nid != NUMA_NO_NODE) {
+		if (hugetlb_cma[nid])
+			balance_node_cma(nid, hugetlb_cma[nid]);
+	} else {
+		for_each_online_node(node) {
+			if (hugetlb_cma[node])
+				balance_node_cma(node,
+						 hugetlb_cma[node]);
+		}
+	}
+}
mm/hugetlb_cma.h +5 -0
index f7d7fb9880a2..2f2a35b56d8a 100644
@@ -13,6 +13,7 @@ bool hugetlb_cma_exclusive_alloc(void);
 unsigned long hugetlb_cma_total_size(void);
 void hugetlb_cma_validate_params(void);
 bool hugetlb_early_cma(struct hstate *h);
+void hugetlb_cma_balance(int nid);
 #else
 static inline void hugetlb_cma_free_folio(struct folio *folio)
 {
@@ -53,5 +54,9 @@ static inline bool hugetlb_early_cma(struct hstate *h)
 {
 	return false;
 }
+
+static inline void hugetlb_cma_balance(int nid)
+{
+}
 #endif
 #endif
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
#22 lore raw

Re: [RFC PATCH 10/12] mm/hugetlb: do explicit CMA balancing

Quoted 6 lines · Frank van der Linden

CMA areas are normally not very large, but HugeTLB CMA is an exception. hugetlb_cma, used for 'gigantic' pages (usually 1G), can take up many gigabytes of memory.

As such, it is potentially the largest source of 'false OOM' conditions,

The false OOM kills also seem to happen when a system does not use hugetlbfs at all, but a cgroup simply has most/all of its reclaimable memory in a CMA region, and then tries to do a kernel allocation.

Would it make sense to call hugetlb_cma_balance() from the pageout code instead, when the pageout code is trying to free non-CMA memory, but ended up freeing mostly/only CMA memory?

Signature · 1 line
All Rights Reversed.
#23 lore raw

Re: [RFC PATCH 10/12] mm/hugetlb: do explicit CMA balancing

In reply to Frank van der Linden
kernel test robot noticed the following build errors:

Hi Frank,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on powerpc/next]
[also build test ERROR on powerpc/fixes powerpc/topic/ppc-kvm s390/features trace/for-next linus/master v6.17-rc7 next-20250926] [cannot apply to akpm-mm/mm-everything]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Frank-van-der-Linden/mm-cma-add-tunable-for-CMA-fallback-limit/20250926-101411
base:   https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next
patch link:    https://lore.kernel.org/r/20250915195153.462039-11-fvdl%40google.com
patch subject: [RFC PATCH 10/12] mm/hugetlb: do explicit CMA balancing
config: x86_64-buildonly-randconfig-001-20250926 (https://download.01.org/0day-ci/archive/20250927/[email protected]/config)
compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250927/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

>> ld.lld: error: undefined symbol: balance_node_cma
   >>> referenced by usercopy_64.c
   >>>               vmlinux.o:(hugetlb_cma_balance)
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#24 lore raw

RFC PATCH 11/12 mm/cma: rebalance CMA when changing cma_first_limit

In reply to Frank van der Linden

To keep things consistent, rebalance CMA when changing the cma_first_limit sysctl.

---
 mm/page_alloc.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)
mm/page_alloc.c +19 -1
index dc59aaa63ae6..da1cab63995c 100644
@@ -6640,6 +6640,24 @@ static int percpu_pagelist_high_fraction_sysctl_handler(const struct ctl_table *
 	return ret;
 }
 
+#ifdef CONFIG_CMA
+static int cma_first_limit_sysctl_handler(const struct ctl_table *table,
+				int write, void *buffer, size_t *length,
+				loff_t *ppos)
+{
+	int ret, nid;
+
+	ret = proc_dointvec_minmax(table, write, buffer, length, ppos);
+	if (ret || !write)
+		return ret;
+
+	for_each_node_state(nid, N_MEMORY)
+		balance_node_cma(nid, NULL);
+
+	return 0;
+}
+#endif
+
 static const struct ctl_table page_alloc_sysctl_table[] = {
 	{
 		.procname	= "min_free_kbytes",
@@ -6723,7 +6741,7 @@ static const struct ctl_table page_alloc_sysctl_table[] = {
 		.data		= &cma_first_limit,
 		.maxlen		= sizeof(cma_first_limit),
 		.mode		= 0644,
-		.proc_handler	= proc_dointvec_minmax,
+		.proc_handler	= cma_first_limit_sysctl_handler,
 		.extra1		= SYSCTL_ZERO,
 		.extra2		= SYSCTL_ONE_HUNDRED,
 	},
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
#26 lore raw

RFC PATCH 12/12 mm/cma: add CMA balance VM event counter

In reply to Frank van der Linden

Add VM counters that record the number of migration success / failures during CMA rebalancing. This is similar to other migrate counters.

---
 include/linux/vm_event_item.h | 3 +++
 mm/migrate.c                  | 8 ++++++++
 mm/vmstat.c                   | 2 ++
 3 files changed, 13 insertions(+)
include/linux/vm_event_item.h +3 -0
index 9e15a088ba38..1711ff85a02f 100644
@@ -87,6 +87,9 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		CMA_ALLOC_SUCCESS,
 		CMA_ALLOC_FAIL,
 #endif
+		CMA_BALANCE_MIGRATE_SUCCESS,
+		CMA_BALANCE_MIGRATE_FAIL,
+
 		UNEVICTABLE_PGCULLED,	/* culled to noreclaim list */
 		UNEVICTABLE_PGSCANNED,	/* scanned for reclaimability */
 		UNEVICTABLE_PGRESCUED,	/* rescued from noreclaim list */
mm/migrate.c +8 -0
index 9e5ef39ce73a..63d771daa3bc 100644
@@ -2144,6 +2144,14 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,
 	count_vm_events(THP_MIGRATION_SUCCESS, stats.nr_thp_succeeded);
 	count_vm_events(THP_MIGRATION_FAIL, stats.nr_thp_failed);
 	count_vm_events(THP_MIGRATION_SPLIT, stats.nr_thp_split);
+
+	if (reason == MR_CMA_BALANCE) {
+		count_vm_events(CMA_BALANCE_MIGRATE_SUCCESS,
+				stats.nr_succeeded);
+		count_vm_events(CMA_BALANCE_MIGRATE_FAIL,
+				stats.nr_failed_pages);
+	}
+
 	trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages,
 			       stats.nr_thp_succeeded, stats.nr_thp_failed,
 			       stats.nr_thp_split, stats.nr_split, mode,
mm/vmstat.c +2 -0
index 71cd1ceba191..af811328db09 100644
@@ -1392,6 +1392,8 @@ const char * const vmstat_text[] = {
 #ifdef CONFIG_CMA
 	[I(CMA_ALLOC_SUCCESS)]			= "cma_alloc_success",
 	[I(CMA_ALLOC_FAIL)]			= "cma_alloc_fail",
+	[I(CMA_BALANCE_MIGRATE_SUCCESS)]	= "cma_balance_migrate_success",
+	[I(CMA_BALANCE_MIGRATE_FAIL)]		= "cma_balance_migrate_fail",
 #endif
 	[I(UNEVICTABLE_PGCULLED)]		= "unevictable_pgs_culled",
 	[I(UNEVICTABLE_PGSCANNED)]		= "unevictable_pgs_scanned",
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
#28 lore raw
In reply to Frank van der Linden
Quoted 64 lines · Frank van der Linden

This is an RFC on a solution to the long standing problem of OOMs occuring when the kernel runs out of space for unmovable allocations in the face of large amounts of CMA.

Introduction
============

When there is a large amount of CMA (e.g. with hugetlb_cma), it is possible for the kernel to run out of space to get unmovable allocations from. This is because it cannot use the CMA area. If the issue is just that there is a large CMA area, and that there isn't enough space left, that can be considered a misconfigured system. However, there is a scenario in which things could have been dealt with better: if the non-CMA area also has movable allocations in it, and there are CMA pageblocks still available.

The current mitigation for this issue is to start using CMA pageblocks for movable allocations first if the amount of free CMA pageblocks is more than 50% of the total amount of free memory in a zone. But that may not always work out, e.g. the system could easily run in to a scenario where long-lasting movable allocations are made first, which do not go to CMA before the 50% mark is reached. When the non-CMA area fills up, these will get in the way of the kernel's unmovable allocations, and OOMs might occur.

Even always directing movable allocations to CMA first does not completely fix the issue. Take a scenario where there is a large amount of CMA through hugetlb_cma. All of that CMA has been taken up by 1G hugetlb pages. So, movable allocations end up in the non-CMA area. Now, the number of hugetlb pages in the pool is lowered, so some CMA becomes available. At the same time, increased system activity leads to more unmovable allocations. Since the movable allocations are still in the non-CMA area, these kernel allocations might still fail.

Additionally, CMA areas are allocated at the bottom of the zone. There has been some discussion on this in the past. Originally, doing allocations from CMA was deemed something that was best avoided. The arguments were twofold:

1) cma_alloc needs to be quick and should not have to migrate a lot of pages.
2) migration might fail, so the fewer pages it has to migrate the better

These arguments are why CMA is avoided (until the 50% limit is hit), and why CMA areas are allocated at the bottom of a zone. But compaction migrates memory from the bottom to the top of a zone. That means that compaction will actually end up migrating movable allocations out of CMA and in to non-CMA, making the issue of OOMing for unmovable allocations worse.

Solution: CMA balancing
=======================

First, this patch set makes the 50% threshold configurable, which is useful in any case. vm.cma_first_limit is the percentage of free CMA, as part of the total amount of free memory in a zone, above which CMA will be used first for movable allocations. 0 is always, 100 is never.

Quoted 4 lines

Then, it creates an interface that allows for moving movable allocations from non-CMA to CMA. CMA areas opt in to taking part in this through a flag. Also, if the flag is set for a CMA area, it is allocated at the top of a zone instead of the bottom.

Hm, what if we can teach the compaction code to start off the beginning of the zone or end of cma zone(s) depending on the current balance?

The problem with placing the cma area at the end is that it might significantly decrease the success rate of cma allocations when it's racing with the background compaction, which is hard to control. At least it was clearly so in my measurements several years ago.

Quoted 15 lines

Lastly, the hugetlb_cma code was modified to try to migrate movable allocations from non-CMA to CMA when a hugetlb CMA page is freed. Only hugetlb CMA areas opt in to CMA balancing, behavior for all other CMA areas is unchanged.

Discussion
==========

This approach works when tested with a hugetlb_cma setup where a large number of 1G pages is active, but the number is sometimes reduced in exchange for larger non-hugetlb overhead.

Arguments against this approach:

Quoted 8 lines

* It's kind of heavy-handed. Since there is no easy way to track the amount of movable allocations residing in non-CMA pageblocks, it will likely end up scanning too much memory, as it only knows the upper bound.
* It should be more integrated with watermark handling in the allocation slow path. Again, this would likely require tracking the number of movable allocations in non-CMA pageblocks.

I think the problem is very real and the proposed approach looks reasonable. But I also agree that it's heavy-handed. Doesn't feel like "the final" solution :)

I wonder if we can track the amount of free space outside of cma and move pages out on reaching a certain low threshold? And it can in theory be the part of the generic kswapd/reclaim code.

#29 lore raw
Quoted 76 lines · Roman Gushchin
Quoted 64 lines · Frank van der Linden

This is an RFC on a solution to the long standing problem of OOMs occuring when the kernel runs out of space for unmovable allocations in the face of large amounts of CMA.

Introduction
============

When there is a large amount of CMA (e.g. with hugetlb_cma), it is possible for the kernel to run out of space to get unmovable allocations from. This is because it cannot use the CMA area. If the issue is just that there is a large CMA area, and that there isn't enough space left, that can be considered a misconfigured system. However, there is a scenario in which things could have been dealt with better: if the non-CMA area also has movable allocations in it, and there are CMA pageblocks still available.

The current mitigation for this issue is to start using CMA pageblocks for movable allocations first if the amount of free CMA pageblocks is more than 50% of the total amount of free memory in a zone. But that may not always work out, e.g. the system could easily run in to a scenario where long-lasting movable allocations are made first, which do not go to CMA before the 50% mark is reached. When the non-CMA area fills up, these will get in the way of the kernel's unmovable allocations, and OOMs might occur.

Even always directing movable allocations to CMA first does not completely fix the issue. Take a scenario where there is a large amount of CMA through hugetlb_cma. All of that CMA has been taken up by 1G hugetlb pages. So, movable allocations end up in the non-CMA area. Now, the number of hugetlb pages in the pool is lowered, so some CMA becomes available. At the same time, increased system activity leads to more unmovable allocations. Since the movable allocations are still in the non-CMA area, these kernel allocations might still fail.

Additionally, CMA areas are allocated at the bottom of the zone. There has been some discussion on this in the past. Originally, doing allocations from CMA was deemed something that was best avoided. The arguments were twofold:

1) cma_alloc needs to be quick and should not have to migrate a lot of pages.
2) migration might fail, so the fewer pages it has to migrate the better

These arguments are why CMA is avoided (until the 50% limit is hit), and why CMA areas are allocated at the bottom of a zone. But compaction migrates memory from the bottom to the top of a zone. That means that compaction will actually end up migrating movable allocations out of CMA and in to non-CMA, making the issue of OOMing for unmovable allocations worse.

Solution: CMA balancing
=======================

First, this patch set makes the 50% threshold configurable, which is useful in any case. vm.cma_first_limit is the percentage of free CMA, as part of the total amount of free memory in a zone, above which CMA will be used first for movable allocations. 0 is always, 100 is never.

Quoted 4 lines

Then, it creates an interface that allows for moving movable allocations from non-CMA to CMA. CMA areas opt in to taking part in this through a flag. Also, if the flag is set for a CMA area, it is allocated at the top of a zone instead of the bottom.

Hm, what if we can teach the compaction code to start off the beginning of the zone or end of cma zone(s) depending on the current balance?

Quoted 5 lines

The problem with placing the cma area at the end is that it might significantly decrease the success rate of cma allocations when it's racing with the background compaction, which is hard to control. At least it was clearly so in my measurements several years ago.

Indeed, I saw your change that moved the CMA areas to the bottom of the zone for that reason. In my testing, I saw a slight uptick in cma_alloc failures for HugeTLB (due to migration failures), but it wasn't much at all. Also, our current usage scenario can deal with the occasional failure, so it was less of a concern. I can try to re-run some tests to see if I can gather some harder numbers on that - the problem is of course finding a test case that gives reproducible results.

Quoted 26 lines
Quoted 17 lines

Lastly, the hugetlb_cma code was modified to try to migrate movable allocations from non-CMA to CMA when a hugetlb CMA page is freed. Only hugetlb CMA areas opt in to CMA balancing, behavior for all other CMA areas is unchanged.

Discussion
==========

This approach works when tested with a hugetlb_cma setup where a large number of 1G pages is active, but the number is sometimes reduced in exchange for larger non-hugetlb overhead.

Arguments against this approach:

Quoted 8 lines

* It's kind of heavy-handed. Since there is no easy way to track the amount of movable allocations residing in non-CMA pageblocks, it will likely end up scanning too much memory, as it only knows the upper bound.
* It should be more integrated with watermark handling in the allocation slow path. Again, this would likely require tracking the number of movable allocations in non-CMA pageblocks.

Quoted 7 lines

I think the problem is very real and the proposed approach looks reasonable. But I also agree that it's heavy-handed. Doesn't feel like "the final" solution :)

I wonder if we can track the amount of free space outside of cma and move pages out on reaching a certain low threshold? And it can in theory be the part of the generic kswapd/reclaim code.

I considered this, yes. The first problem is that there is no easy way to express the number that is "pages allocated with __GFP_MOVABLE in non-CMA pageblocks". You can approximate pretty well by checking if they are on the LRU, I suppose.

If you succeed in getting that number accurately, the next issue is defining the right threshold and when to apply them. E.g. at one point I had a change to skip CMA pageblocks for compaction if the target pageblock is non-CMA, and the threshold has been hit. I ended up dropping it, since this more special-case approach was better for our use case. But my idea at the time was to add it as a 3rd mechanism to try harder for allocations (compaction, reclaim, CMA balancing).

It was something like:

1) Track movable allocations in non-CMA areas.
2) If the watermark for an unmovable allocation is below high, stop migrating things (through compaction) from CMA to non-CMA, and always start allocating from CMA first.
3) If the watermark is approaching low, don't try compaction if you know that CMA can be balanced, but do CMA balancing instead, in amounts that satisfy your needs

One problem here is ping-ponging of memory. If you put CMA areas at the bottom of the zone, ompaction moves things one way, CMA balancing the other way.

I think an approach like the above could still work, I just abandoned it in favor of this more special-cased (and thus safer) one for our use case.

- Frank

#30 lore raw
Quoted 83 lines · Frank van der Linden
Quoted 76 lines · Roman Gushchin
Quoted 64 lines · Frank van der Linden

This is an RFC on a solution to the long standing problem of OOMs occuring when the kernel runs out of space for unmovable allocations in the face of large amounts of CMA.

Introduction
============

When there is a large amount of CMA (e.g. with hugetlb_cma), it is possible for the kernel to run out of space to get unmovable allocations from. This is because it cannot use the CMA area. If the issue is just that there is a large CMA area, and that there isn't enough space left, that can be considered a misconfigured system. However, there is a scenario in which things could have been dealt with better: if the non-CMA area also has movable allocations in it, and there are CMA pageblocks still available.

The current mitigation for this issue is to start using CMA pageblocks for movable allocations first if the amount of free CMA pageblocks is more than 50% of the total amount of free memory in a zone. But that may not always work out, e.g. the system could easily run in to a scenario where long-lasting movable allocations are made first, which do not go to CMA before the 50% mark is reached. When the non-CMA area fills up, these will get in the way of the kernel's unmovable allocations, and OOMs might occur.

Even always directing movable allocations to CMA first does not completely fix the issue. Take a scenario where there is a large amount of CMA through hugetlb_cma. All of that CMA has been taken up by 1G hugetlb pages. So, movable allocations end up in the non-CMA area. Now, the number of hugetlb pages in the pool is lowered, so some CMA becomes available. At the same time, increased system activity leads to more unmovable allocations. Since the movable allocations are still in the non-CMA area, these kernel allocations might still fail.

Additionally, CMA areas are allocated at the bottom of the zone. There has been some discussion on this in the past. Originally, doing allocations from CMA was deemed something that was best avoided. The arguments were twofold:

1) cma_alloc needs to be quick and should not have to migrate a lot of pages.
2) migration might fail, so the fewer pages it has to migrate the better

These arguments are why CMA is avoided (until the 50% limit is hit), and why CMA areas are allocated at the bottom of a zone. But compaction migrates memory from the bottom to the top of a zone. That means that compaction will actually end up migrating movable allocations out of CMA and in to non-CMA, making the issue of OOMing for unmovable allocations worse.

Solution: CMA balancing
=======================

First, this patch set makes the 50% threshold configurable, which is useful in any case. vm.cma_first_limit is the percentage of free CMA, as part of the total amount of free memory in a zone, above which CMA will be used first for movable allocations. 0 is always, 100 is never.

Quoted 4 lines

Then, it creates an interface that allows for moving movable allocations from non-CMA to CMA. CMA areas opt in to taking part in this through a flag. Also, if the flag is set for a CMA area, it is allocated at the top of a zone instead of the bottom.

Hm, what if we can teach the compaction code to start off the beginning of the zone or end of cma zone(s) depending on the current balance?

Quoted 5 lines

The problem with placing the cma area at the end is that it might significantly decrease the success rate of cma allocations when it's racing with the background compaction, which is hard to control. At least it was clearly so in my measurements several years ago.

Quoted 8 lines

Indeed, I saw your change that moved the CMA areas to the bottom of the zone for that reason. In my testing, I saw a slight uptick in cma_alloc failures for HugeTLB (due to migration failures), but it wasn't much at all. Also, our current usage scenario can deal with the occasional failure, so it was less of a concern. I can try to re-run some tests to see if I can gather some harder numbers on that - the problem is of course finding a test case that gives reproducible results.

It might heavily depend on which file system you're using.

Quoted 30 lines
Quoted 30 lines
Quoted 17 lines

Lastly, the hugetlb_cma code was modified to try to migrate movable allocations from non-CMA to CMA when a hugetlb CMA page is freed. Only hugetlb CMA areas opt in to CMA balancing, behavior for all other CMA areas is unchanged.

Discussion
==========

This approach works when tested with a hugetlb_cma setup where a large number of 1G pages is active, but the number is sometimes reduced in exchange for larger non-hugetlb overhead.

Arguments against this approach:

Quoted 8 lines

* It's kind of heavy-handed. Since there is no easy way to track the amount of movable allocations residing in non-CMA pageblocks, it will likely end up scanning too much memory, as it only knows the upper bound.
* It should be more integrated with watermark handling in the allocation slow path. Again, this would likely require tracking the number of movable allocations in non-CMA pageblocks.

I think the problem is very real and the proposed approach looks reasonable. But I also agree that it's heavy-handed. Doesn't feel like "the final" solution :)

Quoted 8 lines
Quoted 3 lines

I wonder if we can track the amount of free space outside of cma and move pages out on reaching a certain low threshold? And it can in theory be the part of the generic kswapd/reclaim code.

I considered this, yes. The first problem is that there is no easy way to express the number that is "pages allocated with __GFP_MOVABLE in non-CMA pageblocks". You can approximate pretty well by checking if they are on the LRU, I suppose.

Hm, but why do you need it? Even if there are movable pages outside of cma, but there are no or little free pages, we can start (trying) moving movable pages into cma, right?

Quoted 8 lines

If you succeed in getting that number accurately, the next issue is defining the right threshold and when to apply them. E.g. at one point I had a change to skip CMA pageblocks for compaction if the target pageblock is non-CMA, and the threshold has been hit. I ended up dropping it, since this more special-case approach was better for our use case. But my idea at the time was to add it as a 3rd mechanism to try harder for allocations (compaction, reclaim, CMA balancing).

Agree, but I guess we can pick some magic number.

Quoted 11 lines

It was something like:

1) Track movable allocations in non-CMA areas.
2) If the watermark for an unmovable allocation is below high, stop migrating things (through compaction) from CMA to non-CMA, and always start allocating from CMA first.
3) If the watermark is approaching low, don't try compaction if you know that CMA can be balanced, but do CMA balancing instead, in amounts that satisfy your needs

Quoted 3 lines

One problem here is ping-ponging of memory. If you put CMA areas at the bottom of the zone, ompaction moves things one way, CMA balancing the other way.

Agree, it's a valid concern.

#31 lore raw

[RFC PATCH 13/12] mm,cma: add compaction cma balance helper for direct reclaim

In reply to Frank van der Linden
Quoted 3 lines · On Mon, 15 Sep 2025 19:51:41 +0000 Frank van der Linden <[email protected]> wrote:

This is an RFC on a solution to the long standing problem of OOMs occuring when the kernel runs out of space for unmovable allocations in the face of large amounts of CMA.

In order to make the CMA balancing code useful without hugetlb involvement, eg. when simply allocating a !__GFP_MOVABLE allocation, I added two patches to invoke CMA balancing from the page reclaim code when needed.

With these changes, we might no longer need to call the CMA balancing code from the hugetlb free path any more, and could potentially simplify some things in that area.

---8<---
From 99991606760fdf8399255d7fc1f21b58069a4afe Mon Sep 17 00:00:00 2001
From: Rik van Riel <[email protected]>
Date: Tue, 23 Sep 2025 10:01:42 -0700
Subject: [PATCH 2/3] mm,cma: add compaction cma balance helper for direct reclaim

Add a cma balance helper for the direct reclaim code, which does not balance CMA free memory all the way, but only a limited number of pages.

---
 mm/compaction.c | 20 ++++++++++++++++++--
 mm/internal.h   |  7 +++++++
 2 files changed, 25 insertions(+), 2 deletions(-)
mm/compaction.c +18 -2
index 3200119b8baf..90478c29db60 100644
@@ -2541,7 +2541,7 @@ isolate_free_cma_pages(struct compact_control *cc)
 	cc->free_pfn = next_pfn;
 }
 
-static void balance_zone_cma(struct zone *zone, struct cma *cma)
+static void balance_zone_cma(struct zone *zone, struct cma *cma, int target)
 {
 	struct compact_control cc = {
 		.zone = zone,
@@ -2613,6 +2613,13 @@ static void balance_zone_cma(struct zone *zone, struct cma *cma)
 		nr_pages = min(nr_pages, cma_get_available(cma));
 	nr_pages = min(allocated_noncma, nr_pages);
 
+	/*
+	 * When invoked from page reclaim, use the provided target rather
+	 * than the calculated one.
+	 */
+	if (target)
+		nr_pages = target;
+
 	for (order = 0; order < NR_PAGE_ORDERS; order++)
 		INIT_LIST_HEAD(&cc.freepages[order]);
 	INIT_LIST_HEAD(&cc.migratepages);
@@ -2674,10 +2681,19 @@ void balance_node_cma(int nid, struct cma *cma)
 		if (!populated_zone(zone))
 			continue;
 
-		balance_zone_cma(zone, cma);
+		balance_zone_cma(zone, cma, 0);
 	}
 }
 
+void balance_cma_zonelist(struct zonelist *zonelist, int nr_pages)
+{
+	struct zoneref *z;
+	struct zone *zone;
+
+	for_each_zone_zonelist(zone, z, zonelist, MAX_NR_ZONES - 1)
+		balance_zone_cma(zone, NULL, nr_pages);
+}
+
 #endif /* CONFIG_CMA */
 
 static enum compact_result
mm/internal.h +7 -0
index 7dcaf7214683..5340b94683bf 100644
@@ -942,6 +942,7 @@ struct cma;
 void *cma_reserve_early(struct cma *cma, unsigned long size);
 void init_cma_pageblock(struct page *page);
 void balance_node_cma(int nid, struct cma *cma);
+void balance_cma_zonelist(struct zonelist *zonelist, int nr_pages);
 #else
 static inline void *cma_reserve_early(struct cma *cma, unsigned long size)
 {
@@ -950,6 +951,12 @@ static inline void *cma_reserve_early(struct cma *cma, unsigned long size)
 static inline void init_cma_pageblock(struct page *page)
 {
 }
+static inline void balance_node_cma(int nid, struct cma *cma)
+{
+}
+static inline void balance_cma_zonelist(struct zonelist *zonelist, int nr_pages)
+{
+}
 #endif
 
 
Signature · 1 line
2.47.3
#32 lore raw

cover mm,cma: call CMA balancing from page reclaim code

In reply to Frank van der Linden

Call CMA balancing from the page reclaim code, if page reclaim is reclaiming pages that are unsuitable for the allocator.

To keep direct reclaim latencies low, kswapd will do CMA balancing whenever some of the reclaimed pages are unsuitable for the allocator that woke up kswapd, while the direct reclaimers will only do CMA balancing if most of the reclaimed pages are unsuitable.

---
 mm/vmscan.c | 31 ++++++++++++++++++++++++++++++-
 1 file changed, 30 insertions(+), 1 deletion(-)
mm/vmscan.c +30 -1
index a48aec8bfd92..ec6bde5b07d3 100644
@@ -168,6 +168,9 @@ struct scan_control {
 	/* Number of pages freed so far during a call to shrink_zones() */
 	unsigned long nr_reclaimed;
 
+	/* Number of pages reclaimed, but unsuitable to the allocator */
+	unsigned long nr_unsuitable;
+
 	struct {
 		unsigned int dirty;
 		unsigned int unqueued_dirty;
@@ -1092,6 +1095,19 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask)
 	return !data_race(folio_swap_flags(folio) & SWP_FS_OPS);
 }
 
+#ifdef CONFIG_CMA
+static bool unsuitable_folio(struct folio *folio, struct scan_control *sc)
+{
+	return gfp_migratetype(sc->gfp_mask) != MIGRATE_MOVABLE &&
+		folio_migratetype(folio) == MIGRATE_CMA;
+}
+#else
+static bool unsuitable_folio(struct folio *folio, struct scan_control *sc)
+{
+	return false;
+}
+#endif
+
 /*
  * shrink_folio_list() returns the number of reclaimed pages
  */
@@ -1103,7 +1119,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 	struct folio_batch free_folios;
 	LIST_HEAD(ret_folios);
 	LIST_HEAD(demote_folios);
-	unsigned int nr_reclaimed = 0, nr_demoted = 0;
+	unsigned int nr_reclaimed = 0, nr_demoted = 0, nr_unsuitable = 0;
 	unsigned int pgactivate = 0;
 	bool do_demote_pass;
 	struct swap_iocb *plug = NULL;
@@ -1530,6 +1546,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 					 * leave it off the LRU).
 					 */
 					nr_reclaimed += nr_pages;
+					if (unsuitable_folio(folio, sc))
+						nr_unsuitable += nr_pages;
 					continue;
 				}
 			}
@@ -1560,6 +1578,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 		 * all pages in it.
 		 */
 		nr_reclaimed += nr_pages;
+		if (unsuitable_folio(folio, sc))
+			nr_unsuitable += nr_pages;
 
 		folio_unqueue_deferred_split(folio);
 		if (folio_batch_add(&free_folios, folio) == 0) {
@@ -1641,6 +1661,9 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 
 	if (plug)
 		swap_write_unplug(plug);
+
+	sc->nr_unsuitable += nr_unsuitable;
+
 	return nr_reclaimed;
 }
 
@@ -6431,6 +6454,10 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
 
 	delayacct_freepages_end();
 
+	/* Almost all memory reclaimed was unsuitable? Move data into CMA. */
+	if (sc->nr_unsuitable >= sc->nr_reclaimed - 2)
+		balance_cma_zonelist(zonelist, SWAP_CLUSTER_MAX);
+
 	if (sc->nr_reclaimed)
 		return sc->nr_reclaimed;
 
@@ -7169,6 +7196,8 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx)
 	if (!sc.nr_reclaimed)
 		pgdat->kswapd_failures++;
 
+	if (sc.nr_unsuitable)
+		balance_node_cma(pgdat->node_id, NULL);
 out:
 	clear_reclaim_active(pgdat, highest_zoneidx);
 
Signature · 1 line
2.47.3