CMA balancing
Subsystems (13)
cover CMA balancing
This is an RFC on a solution to the long standing problem of OOMs occuring when the kernel runs out of space for unmovable allocations in the face of large amounts of CMA.
Introduction
============
When there is a large amount of CMA (e.g. with hugetlb_cma), it is possible for the kernel to run out of space to get unmovable allocations from. This is because it cannot use the CMA area. If the issue is just that there is a large CMA area, and that there isn't enough space left, that can be considered a misconfigured system. However, there is a scenario in which things could have been dealt with better: if the non-CMA area also has movable allocations in it, and there are CMA pageblocks still available.
The current mitigation for this issue is to start using CMA pageblocks for movable allocations first if the amount of free CMA pageblocks is more than 50% of the total amount of free memory in a zone. But that may not always work out, e.g. the system could easily run in to a scenario where long-lasting movable allocations are made first, which do not go to CMA before the 50% mark is reached. When the non-CMA area fills up, these will get in the way of the kernel's unmovable allocations, and OOMs might occur.
Even always directing movable allocations to CMA first does not completely fix the issue. Take a scenario where there is a large amount of CMA through hugetlb_cma. All of that CMA has been taken up by 1G hugetlb pages. So, movable allocations end up in the non-CMA area. Now, the number of hugetlb pages in the pool is lowered, so some CMA becomes available. At the same time, increased system activity leads to more unmovable allocations. Since the movable allocations are still in the non-CMA area, these kernel allocations might still fail.
Additionally, CMA areas are allocated at the bottom of the zone. There has been some discussion on this in the past. Originally, doing allocations from CMA was deemed something that was best avoided. The arguments were twofold:
1) cma_alloc needs to be quick and should not have to migrate a lot of pages.
2) migration might fail, so the fewer pages it has to migrate the better
These arguments are why CMA is avoided (until the 50% limit is hit), and why CMA areas are allocated at the bottom of a zone. But compaction migrates memory from the bottom to the top of a zone. That means that compaction will actually end up migrating movable allocations out of CMA and in to non-CMA, making the issue of OOMing for unmovable allocations worse.
Solution: CMA balancing
=======================
First, this patch set makes the 50% threshold configurable, which is useful in any case. vm.cma_first_limit is the percentage of free CMA, as part of the total amount of free memory in a zone, above which CMA will be used first for movable allocations. 0 is always, 100 is never.
Then, it creates an interface that allows for moving movable allocations from non-CMA to CMA. CMA areas opt in to taking part in this through a flag. Also, if the flag is set for a CMA area, it is allocated at the top of a zone instead of the bottom.
Lastly, the hugetlb_cma code was modified to try to migrate movable allocations from non-CMA to CMA when a hugetlb CMA page is freed. Only hugetlb CMA areas opt in to CMA balancing, behavior for all other CMA areas is unchanged.
Discussion
==========
This approach works when tested with a hugetlb_cma setup where a large number of 1G pages is active, but the number is sometimes reduced in exchange for larger non-hugetlb overhead.
Arguments against this approach:
* It's kind of heavy-handed. Since there is no easy way to track the amount of movable allocations residing in non-CMA pageblocks, it will likely end up scanning too much memory, as it only knows the upper bound.
* It should be more integrated with watermark handling in the allocation slow path. Again, this would likely require tracking the number of movable allocations in non-CMA pageblocks.
Arguments for this approach:
* Yes, it does more, but the work is restricted to the context of a process that decreases the hugetlb pool, and is not more work than allocating (e.g. freeing a hugetlb page from the pool is now as expensive as allocating a new one).
* hugetlb_cma is really the only situation where you have CMA areas large enough to trigger the OOM scenario, so restricting it to hugetlb should be good enough.
Comments, thoughts?
Frank van der Linden (12):
mm/cma: add tunable for CMA fallback limit mm/cma: clean up flag handling a bit
mm/cma: add flags argument to init functions mm/cma: keep a global sorted list of CMA ranges mm/cma: add helper functions for CMA balancing mm/cma: define and act on CMA_BALANCE flag mm/compaction: optionally use a different isolate function mm/compaction: simplify isolation order checks a bit mm/cma: introduce CMA balancing
mm/hugetlb: do explicit CMA balancing
mm/cma: rebalance CMA when changing cma_first_limit mm/cma: add CMA balance VM event counter
arch/powerpc/kernel/fadump.c | 2 +- arch/powerpc/kvm/book3s_hv_builtin.c | 2 +- drivers/s390/char/vmcp.c | 2 +- include/linux/cma.h | 64 +++++- include/linux/migrate_mode.h | 1 + include/linux/mm.h | 4 + include/linux/vm_event_item.h | 3 + include/trace/events/migrate.h | 3 +- kernel/dma/contiguous.c | 10 +- mm/cma.c | 318 +++++++++++++++++++++++---- mm/cma.h | 13 +- mm/compaction.c | 199 +++++++++++++++-- mm/hugetlb.c | 14 +- mm/hugetlb_cma.c | 18 +- mm/hugetlb_cma.h | 5 + mm/internal.h | 11 +- mm/migrate.c | 8 + mm/page_alloc.c | 104 +++++++-- mm/vmstat.c | 2 + 19 files changed, 676 insertions(+), 107 deletions(-)
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
RFC PATCH 1/12 mm/cma: add tunable for CMA fallback limit
Add a tunable to experiment with the circumstances under which movable allocations should use CMA pageblocks first, to avoid false OOM conditions.
The limit is the percentage free memory which is being taken up by CMA. If the amount of used memory in CMA pageblocks is above this limit, CMA will be used first. So, 0 would mean always using CMA first, and 100 means never use CMA first.
Currently the default is 50, which matches the existing behavior, so there is no functional change.
--- include/linux/mm.h | 4 +++ mm/page_alloc.c | 84 ++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 74 insertions(+), 14 deletions(-)
include/linux/mm.h +4 -0
@@ -3253,6 +3253,10 @@ void warn_alloc(gfp_t gfp_mask, nodemask_t *nodemask, const char *fmt, ...);extern void setup_per_cpu_pageset (void ); +#ifdef CONFIG_CMA +extern int cma_first_limit; +#endif +/* nommu.c */ extern atomic_long_t mmap_pages_allocated;extern int nommu_shrink_inode_mappings (struct inode *,size_t ,size_t );
mm/page_alloc.c +70 -14
109 lines not on this page · raw message
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [RFC PATCH 01/12] mm/cma: add tunable for CMA fallback limit
Quoted 12 lines · Frank van der Linden
mm/page_alloc.c +11
-0+#ifdef CONFIG_CMA + { + .procname = "cma_first_limit", + .data = &cma_first_limit, + .maxlen = sizeof(cma_first_limit), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + .extra2 = SYSCTL_ONE_HUNDRED, + }, +#endif
Quoted 2 lines
};
I like the code, but the name is a little confusing.
Maybe something like "prefer_cma_percent"?
Signature · 1 line
All Rights Reversed.
RFC PATCH 2/12 mm/cma: clean up flag handling a bit
In reply to Frank van der LindenAtomic bit operations aren't needed for the cma flags field, so switch their manipulation over to normal AND/OR operations.
Also export the bit values in linux/cma.h, as we will be adding publicly used values later.
No functional change.
--- include/linux/cma.h | 12 ++++++++++++ mm/cma.c | 16 ++++++++-------- mm/cma.h | 7 ------- 3 files changed, 20 insertions(+), 15 deletions(-)
include/linux/cma.h +12 -0
@@ -20,6 +20,18 @@#define CMA_MIN_ALIGNMENT_PAGES pageblock_nr_pages#define CMA_MIN_ALIGNMENT_BYTES (PAGE_SIZE * CMA_MIN_ALIGNMENT_PAGES) +enum cma_flags { + __CMA_RESERVE_PAGES_ON_ERROR, + __CMA_ZONES_VALID, + __CMA_ZONES_INVALID, + __CMA_ACTIVATED, +}; + +#define CMA_RESERVE_PAGES_ON_ERROR BIT(__CMA_RESERVE_PAGES_ON_ERROR) +#define CMA_ZONES_VALID BIT(__CMA_ZONES_VALID) +#define CMA_ZONES_INVALID BIT(__CMA_ZONES_INVALID) +#define CMA_ACTIVATED BIT(__CMA_ACTIVATED) +struct cma ;extern unsigned long totalcma_pages;
mm/cma.c +8 -8
@@ -111,8 +111,8 @@ bool cma_validate_zones(struct cma *cma)* check has already been done. If neither is set, the * check has not been performed yet. */ - valid_bit_set =test_bit (CMA_ZONES_VALID , &cma->flags); -if (valid_bit_set ||test_bit (CMA_ZONES_INVALID , &cma->flags)) + valid_bit_set = (cma->flags &CMA_ZONES_VALID ); +if (valid_bit_set || (cma->flags &CMA_ZONES_INVALID ))return valid_bit_set;for (r =0 ; r < cma->nranges; r++) {
@@ -126,12 +126,12 @@ bool cma_validate_zones(struct cma *cma)*/ WARN_ON_ONCE (!pfn_valid (base_pfn));if (pfn_range_intersects_zones (cma->nid, base_pfn, cmr->count)) { -set_bit (CMA_ZONES_INVALID , &cma->flags); + cma->flags |=CMA_ZONES_INVALID ;return false; } } -set_bit (CMA_ZONES_VALID , &cma->flags); + cma->flags |=CMA_ZONES_VALID ;return true; }
@@ -176,7 +176,7 @@ static void __init cma_activate_area(struct cma *cma)INIT_HLIST_HEAD (&cma->mem_head);spin_lock_init (&cma->mem_head_lock);#endif -set_bit (CMA_ACTIVATED , &cma->flags); + cma->flags |=CMA_ACTIVATED ;return ;
@@ -185,7 +185,7 @@ static void __init cma_activate_area(struct cma *cma)bitmap_free (cma->ranges[r].bitmap);/* Expose all pages to the buddy, they are useless for CMA. */ -if (!test_bit (CMA_RESERVE_PAGES_ON_ERROR , &cma->flags)) { +if (!(cma->flags &CMA_RESERVE_PAGES_ON_ERROR )) {for (r =0 ; r < allocrange; r++) { cmr = &cma->ranges[r]; end_pfn = cmr->base_pfn + cmr->count;
@@ -211,7 +211,7 @@ core_initcall(cma_init_reserved_areas);void __init cma_reserve_pages_on_error (struct cma *cma) { -set_bit (CMA_RESERVE_PAGES_ON_ERROR , &cma->flags); + cma->flags |=CMA_RESERVE_PAGES_ON_ERROR ; } static int __init cma_new_area(const char *name, phys_addr_t size,
@@ -1085,7 +1085,7 @@ void __init *cma_reserve_early(struct cma *cma, unsigned long size)/* * Can only be called early in init. */ -if (test_bit (CMA_ACTIVATED , &cma->flags)) +if (cma->flags &CMA_ACTIVATED )return NULL ;if (!IS_ALIGNED (size,CMA_MIN_ALIGNMENT_BYTES ))
mm/cma.h +0 -7
@@ -64,13 +64,6 @@ struct cma {int nid; }; -enum cma_flags { -CMA_RESERVE_PAGES_ON_ERROR , -CMA_ZONES_VALID , -CMA_ZONES_INVALID , -CMA_ACTIVATED , -}; -extern struct cma cma_areas[MAX_CMA_AREAS ];extern unsigned int cma_area_count;
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [RFC PATCH 02/12] mm/cma: clean up flag handling a bit
Quoted 11 lines · Frank van der Linden
Atomic bit operations aren't needed for the cma flags field, so switch their manipulation over to normal AND/OR operations.
Also export the bit values in
linux/cma.h, as we will be adding publicly used values later.No functional change.
Signed-off-by Frank van der Linden <[email protected]>
Signature · 1 line
All Rights Reversed.
RFC PATCH 3/12 mm/cma: add flags argument to init functions
In reply to Frank van der LindenAdd a flags argument to the various CMA init functions, as there will be a need to pass in more flags to control init and runtime behavior other than just the current 'fixed' argument.
Replace the fixed argument with a flags argument, and adapt callers.
--- arch/powerpc/kernel/fadump.c | 2 +- arch/powerpc/kvm/book3s_hv_builtin.c | 2 +- drivers/s390/char/vmcp.c | 2 +- include/linux/cma.h | 20 ++++++++++------ kernel/dma/contiguous.c | 10 ++++---- mm/cma.c | 36 ++++++++++++++++++---------- mm/hugetlb_cma.c | 2 +- 7 files changed, 46 insertions(+), 28 deletions(-)
arch/powerpc/kernel/fadump.c +1 -1
@@ -112,7 +112,7 @@ void __init fadump_cma_init(void)return ; } - rc =cma_init_reserved_mem (base, size,0 ,"fadump_cma" , &fadump_cma); + rc =cma_init_reserved_mem (base, size,0 ,0 ,"fadump_cma" , &fadump_cma);if (rc) {pr_err ("Failed to init cma area for firmware-assisted dump,%d\n" , rc);/*
arch/powerpc/kvm/book3s_hv_builtin.c +1 -1
@@ -95,7 +95,7 @@ void __init kvm_cma_reserve(void) (unsigned long)selected_size / SZ_1M); align_size =HPT_ALIGN_PAGES <<PAGE_SHIFT ;cma_declare_contiguous (0 , selected_size,0 , align_size, -KVM_CMA_CHUNK_ORDER -PAGE_SHIFT , false,"kvm_cma" , +KVM_CMA_CHUNK_ORDER -PAGE_SHIFT ,0 ,"kvm_cma" , &kvm_cma); } }
drivers/s390/char/vmcp.c +1 -1
@@ -54,7 +54,7 @@ void __init vmcp_cma_reserve(void) {if (!machine_is_vm ())return ; -cma_declare_contiguous (0 , vmcp_cma_size,0 ,0 ,0 , false,"vmcp" , &vmcp_cma); +cma_declare_contiguous (0 , vmcp_cma_size,0 ,0 ,0 ,0 ,"vmcp" , &vmcp_cma); }static void vmcp_response_alloc (struct vmcp_session *session)
include/linux/cma.h +13 -7
@@ -25,12 +25,16 @@ enum cma_flags { __CMA_ZONES_VALID, __CMA_ZONES_INVALID, __CMA_ACTIVATED, + __CMA_FIXED, };#define CMA_RESERVE_PAGES_ON_ERROR BIT(__CMA_RESERVE_PAGES_ON_ERROR)#define CMA_ZONES_VALID BIT(__CMA_ZONES_VALID)#define CMA_ZONES_INVALID BIT(__CMA_ZONES_INVALID)#define CMA_ACTIVATED BIT(__CMA_ACTIVATED) +#define CMA_FIXED BIT(__CMA_FIXED) + +#define CMA_INIT_FLAGS (CMA_FIXED|CMA_RESERVE_PAGES_ON_ERROR)struct cma ;
@@ -42,23 +46,25 @@ extern const char *cma_get_name(const struct cma *cma);extern int __init cma_declare_contiguous_nid (phys_addr_t base,phys_addr_t size,phys_addr_t limit,phys_addr_t alignment,unsigned int order_per_bit, -bool fixed,const char *name,struct cma **res_cma, -int nid); +unsigned long flags,const char *name, +struct cma **res_cma,int nid);static inline int __init cma_declare_contiguous (phys_addr_t base,phys_addr_t size,phys_addr_t limit,phys_addr_t alignment,unsigned int order_per_bit, -bool fixed,const char *name,struct cma **res_cma) +unsigned long flags,const char *name, +struct cma **res_cma) {return cma_declare_contiguous_nid (base, size, limit, alignment, - order_per_bit, fixed, name, res_cma,NUMA_NO_NODE ); + order_per_bit, flags, name, res_cma,NUMA_NO_NODE ); }extern int __init cma_declare_contiguous_multi (phys_addr_t size,phys_addr_t align,unsigned int order_per_bit, -const char *name,struct cma **res_cma,int nid); +unsigned long flags,const char *name, +struct cma **res_cma,int nid);extern int cma_init_reserved_mem (phys_addr_t base,phys_addr_t size,unsigned int order_per_bit, -const char *name, -struct cma **res_cma); +unsigned long flags, +const char *name,struct cma **res_cma);extern struct page *cma_alloc (struct cma *cma,unsigned long count,unsigned int align,bool no_warn);extern bool cma_pages_valid (struct cma *cma,const struct page *pages,unsigned long count);
kernel/dma/contiguous.c +6 -4
@@ -175,7 +175,7 @@ static void __init dma_numa_cma_reserve(void) cma = &dma_contiguous_pernuma_area[nid];snprintf (name,sizeof (name),"pernuma%d" , nid); ret =cma_declare_contiguous_nid (0 , pernuma_size_bytes,0 ,0 , -0 , false, name, cma, nid); +0 ,0 , name, cma, nid);if (ret)pr_warn ("%s: reservation failed: err %d, node %d" , __func__, ret, nid);
@@ -185,7 +185,7 @@ static void __init dma_numa_cma_reserve(void) cma = &dma_contiguous_numa_area[nid];snprintf (name,sizeof (name),"numa%d" , nid); - ret =cma_declare_contiguous_nid (0 , numa_cma_size[nid],0 ,0 ,0 , false, + ret =cma_declare_contiguous_nid (0 , numa_cma_size[nid],0 ,0 ,0 ,0 , name, cma, nid);if (ret) pr_warn("%s: reservation failed: err %d, node %d", __func__,
@@ -279,7 +279,8 @@ int __init dma_contiguous_reserve_area(phys_addr_t size, phys_addr_t base, {int ret; - ret =cma_declare_contiguous (base, size, limit,0 ,0 , fixed, + ret =cma_declare_contiguous (base, size, limit,0 ,0 , + fixed ?CMA_FIXED :0 ,"reserved" , res_cma);if (ret)return ret;
@@ -478,7 +479,8 @@ static int __init rmem_cma_setup(struct reserved_mem *rmem)return -EINVAL ; } - err =cma_init_reserved_mem (rmem->base, rmem->size,0 , rmem->name, &cma); + err =cma_init_reserved_mem (rmem->base, rmem->size,0 ,0 , rmem->name, + &cma);if (err) {pr_err ("Reserved memory: unable to setup CMA region\n" );return err;
mm/cma.c +23 -13
@@ -35,6 +35,7 @@struct cma cma_areas[MAX_CMA_AREAS ];unsigned int cma_area_count; +static DEFINE_MUTEX (cma_mutex);phys_addr_t cma_get_base (const struct cma *cma) {
@@ -215,7 +216,7 @@ void __init cma_reserve_pages_on_error(struct cma *cma) }static int __init cma_new_area (const char *name,phys_addr_t size, -unsigned int order_per_bit, +unsigned int order_per_bit,unsigned long flags,struct cma **res_cma) {struct cma *cma;
@@ -239,6 +240,7 @@ static int __init cma_new_area(const char *name, phys_addr_t size, cma->available_count = cma->count = size >>PAGE_SHIFT ; cma->order_per_bit = order_per_bit; + cma->flags = flags; *res_cma = cma; totalcma_pages += cma->count;
@@ -265,7 +267,7 @@ static void __init cma_drop_area(struct cma *cma)*/ int __init cma_init_reserved_mem (phys_addr_t base,phys_addr_t size,unsigned int order_per_bit, -const char *name, +unsigned long flags,const char *name,struct cma **res_cma) {struct cma *cma;
@@ -288,7 +290,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size,if (!IS_ALIGNED (base | size,CMA_MIN_ALIGNMENT_BYTES ))return -EINVAL ; - ret =cma_new_area (name, size, order_per_bit, &cma); + ret =cma_new_area (name, size, order_per_bit, flags, &cma);if (ret !=0 )return ret;
@@ -429,12 +431,18 @@ static phys_addr_t __init cma_alloc_mem(phys_addr_t base, phys_addr_t size,static int __init __cma_declare_contiguous_nid (phys_addr_t *basep,phys_addr_t size,phys_addr_t limit,phys_addr_t alignment,unsigned int order_per_bit, -bool fixed,const char *name,struct cma **res_cma, -int nid) +unsigned long flags,const char *name, +struct cma **res_cma,int nid) {phys_addr_t memblock_end =memblock_end_of_DRAM ();phys_addr_t base = *basep;int ret; +bool fixed; + +if (flags & ~CMA_INIT_FLAGS ) +return -EINVAL ; + + fixed = (flags &CMA_FIXED );pr_debug ("%s(size %pa, base %pa, limit %pa alignment %pa)\n" , __func__, &size, &base, &limit, &alignment);
@@ -503,7 +511,8 @@ static int __init __cma_declare_contiguous_nid(phys_addr_t *basep,kmemleak_ignore_phys (base); } - ret =cma_init_reserved_mem (base, size, order_per_bit, name, res_cma); + ret =cma_init_reserved_mem (base, size, order_per_bit, flags, + name, res_cma);if (ret) {memblock_phys_free (base, size);return ret;
@@ -526,7 +535,8 @@ static int __init __cma_declare_contiguous_nid(phys_addr_t *basep,*/ int __init cma_declare_contiguous_multi (phys_addr_t total_size,phys_addr_t align,unsigned int order_per_bit, -const char *name,struct cma **res_cma,int nid) +unsigned long flags,const char *name, +struct cma **res_cma,int nid) {phys_addr_t start =0 , end;phys_addr_t size, sizesum, sizeleft;
@@ -543,7 +553,7 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,* First, try it the normal way, producing just one range. */ ret =__cma_declare_contiguous_nid (&start, total_size,0 , align, - order_per_bit, false, name, res_cma, nid); + order_per_bit, flags, name, res_cma, nid);if (ret != -ENOMEM ) goto out;
@@ -567,7 +577,7 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size, sizesum =0 ; failed =NULL ; - ret =cma_new_area (name, total_size, order_per_bit, &cma); + ret =cma_new_area (name, total_size, order_per_bit, flags, &cma);if (ret !=0 ) goto out;
@@ -716,7 +726,7 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,* @limit: End address of the reserved memory (optional, 0 for any). * @alignment: Alignment for the CMA area, should be power of 2 or zero * @order_per_bit: Order of pages represented by one bit on bitmap. -* @fixed: hint about where to place the reserved area +* @flags: flags controlling various aspects of the area * @name: The name of the area. See function cma_init_reserved_mem() * @res_cma: Pointer to store the created cma region. * @nid: nid of the free area to find, %NUMA_NO_NODE for any node
@@ -732,13 +742,13 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,int __init cma_declare_contiguous_nid (phys_addr_t base,phys_addr_t size,phys_addr_t limit,phys_addr_t alignment,unsigned int order_per_bit, -bool fixed,const char *name,struct cma **res_cma, -int nid) +unsigned long flags,const char *name, +struct cma **res_cma,int nid) {int ret; ret =__cma_declare_contiguous_nid (&base, size, limit, alignment, - order_per_bit, fixed, name, res_cma, nid); + order_per_bit, flags, name, res_cma, nid);if (ret !=0 )pr_err ("Failed to reserve %ld MiB\n" , (unsigned long )size /SZ_1M );
mm/hugetlb_cma.c +1 -1
@@ -221,7 +221,7 @@ void __init hugetlb_cma_reserve(int order)* huge page demotion. */ res =cma_declare_contiguous_multi (size,PAGE_SIZE << order, -HUGETLB_PAGE_ORDER , name, +HUGETLB_PAGE_ORDER ,0 , name, &hugetlb_cma[nid], nid);if (res) { pr_warn("hugetlb_cma: reservation failed: err %d, node %d",
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
kernel test robot
build botRe: [RFC PATCH 03/12] mm/cma: add flags argument to init functions
kernel test robot noticed the following build warnings:
Hi Frank,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on powerpc/next]
[also build test WARNING on powerpc/fixes powerpc/topic/ppc-kvm s390/features trace/for-next linus/master v6.17-rc6 next-20250916] [cannot apply to akpm-mm/mm-everything]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Frank-van-der-Linden/mm-cma-add-tunable-for-CMA-fallback-limit/20250916-035514 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next patch link: https://lore.kernel.org/r/20250915195153.462039-4-fvdl%40google.com patch subject: [RFC PATCH 03/12] mm/cma: add flags argument to init functions config: i386-buildonly-randconfig-001-20250917 (https://download.01.org/0day-ci/archive/20250917/[email protected]/config) compiler: gcc-14 (Debian 14.2.0-19) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250917/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> Warning: mm/cma.c:271 function parameter 'flags' not described in 'cma_init_reserved_mem'
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [RFC PATCH 03/12] mm/cma: add flags argument to init functions
In reply to Frank van der LindenQuoted 10 lines · Frank van der Linden
Add a flags argument to the various CMA init functions, as there will be a need to pass in more flags to control init and runtime behavior other than just the current 'fixed' argument.
Replace the fixed argument with a flags argument, and adapt callers.
Signed-off-by Frank van der Linden <[email protected]>
I haven't gotten to the point in the series yet that shows why you need this, but I suppose this looks reasonable?
Signature · 1 line
All Rights Reversed.
RFC PATCH 4/12 mm/cma: keep a global sorted list of CMA ranges
In reply to Frank van der LindenIn order to walk through CMA areas efficiently, it is useful to keep a global sorted list of ranges.
Create this list when activating the areas.
Since users of this list may want to reference the CMA area the range came from, there needs to be a link from the range to that area. So, store a pointer to the CMA structure in the cma_memrange structure. This also reduces the number of arguments to a few internal functions.
--- mm/cma.c | 72 ++++++++++++++++++++++++++++++++++++++++++-------------- mm/cma.h | 6 ++--- 2 files changed, 57 insertions(+), 21 deletions(-)
mm/cma.c +54 -18
@@ -23,6 +23,7 @@#include <linux/sizes.h> #include <linux/slab.h> #include <linux/string_choices.h> +#include <linux/sort.h> #include <linux/log2.h> #include <linux/cma.h> #include <linux/highmem.h>
@@ -65,12 +66,11 @@ static unsigned long cma_bitmap_aligned_mask(const struct cma *cma,* Find the offset of the base PFN from the specified align_order. * The value returned is represented in order_per_bits. */ -static unsigned long cma_bitmap_aligned_offset (const struct cma *cma, -const struct cma_memrange *cmr, +static unsigned long cma_bitmap_aligned_offset (const struct cma_memrange *cmr,unsigned int align_order) {return (cmr->base_pfn & ((1UL << align_order) -1 )) - >> cma->order_per_bit; + >> cmr->cma->order_per_bit; } static unsigned long cma_bitmap_pages_to_bits(const struct cma *cma,
@@ -79,11 +79,12 @@ static unsigned long cma_bitmap_pages_to_bits(const struct cma *cma,return ALIGN (pages,1UL << cma->order_per_bit) >> cma->order_per_bit; } -static void cma_clear_bitmap (struct cma *cma,const struct cma_memrange *cmr, +static void cma_clear_bitmap (const struct cma_memrange *cmr,unsigned long pfn,unsigned long count) {unsigned long bitmap_no, bitmap_count;unsigned long flags; +struct cma *cma = cmr->cma; bitmap_no = (pfn - cmr->base_pfn) >> cma->order_per_bit; bitmap_count =cma_bitmap_pages_to_bits (cma, count);
@@ -147,8 +148,7 @@ static void __init cma_activate_area(struct cma *cma)for (allocrange =0 ; allocrange < cma->nranges; allocrange++) { cmr = &cma->ranges[allocrange]; early_pfn[allocrange] = cmr->early_pfn; - cmr->bitmap =bitmap_zalloc (cma_bitmap_maxno (cma, cmr), -GFP_KERNEL ); + cmr->bitmap =bitmap_zalloc (cma_bitmap_maxno (cmr),GFP_KERNEL );if (!cmr->bitmap) goto cleanup; }
@@ -199,12 +199,45 @@ static void __init cma_activate_area(struct cma *cma)pr_err ("CMA area %s could not be activated\n" , cma->name); } +static struct cma_memrange **cma_ranges; +static int cma_nranges; + +static int cmprange (const void *a,const void *b) +{ +struct cma_memrange *r1, *r2; + + r1 = *(struct cma_memrange **)a; + r2 = *(struct cma_memrange **)b; + +if (r1->base_pfn < r2->base_pfn) +return -1 ; +return r1->base_pfn - r2->base_pfn; +} +static int __init cma_init_reserved_areas (void ) { -int i; +int i, r, nranges; +struct cma *cma; +struct cma_memrange *cmr; + + nranges =0 ; +for (i =0 ; i < cma_area_count; i++) { + cma = &cma_areas[i]; + nranges += cma->nranges; +cma_activate_area (cma); + } + + cma_ranges =kcalloc (nranges,sizeof (*cma_ranges),GFP_KERNEL ); + cma_nranges =0 ; +for (i =0 ; i < cma_area_count; i++) { + cma = &cma_areas[i]; +for (r =0 ; r < cma->nranges; r++) { + cmr = &cma->ranges[r]; + cma_ranges[cma_nranges++] = cmr; + } + } -for (i =0 ; i < cma_area_count; i++) -cma_activate_area (&cma_areas[i]); +sort (cma_ranges, cma_nranges,sizeof (*cma_ranges), cmprange,NULL );return 0 ; }
@@ -297,6 +330,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, cma->ranges[0 ].base_pfn =PFN_DOWN (base); cma->ranges[0 ].early_pfn =PFN_DOWN (base); cma->ranges[0 ].count = cma->count; + cma->ranges[0 ].cma = cma; cma->nranges =1 ; cma->nid =NUMA_NO_NODE ;
@@ -687,6 +721,7 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size, cmrp->base_pfn =PHYS_PFN (mlp->base); cmrp->early_pfn = cmrp->base_pfn; cmrp->count = size >>PAGE_SHIFT ; + cmrp->cma = cma; sizeleft -= size;if (sizeleft ==0 )
@@ -772,7 +807,7 @@ static void cma_debug_show_areas(struct cma *cma)for (r =0 ; r < cma->nranges; r++) { cmr = &cma->ranges[r]; - nbits =cma_bitmap_maxno (cma, cmr); + nbits =cma_bitmap_maxno (cmr);pr_info ("range %d: " , r);for_each_clear_bitrange (start, end, cmr->bitmap, nbits) {
@@ -786,9 +821,9 @@ static void cma_debug_show_areas(struct cma *cma)spin_unlock_irq (&cma->lock); } -static int cma_range_alloc (struct cma *cma,struct cma_memrange *cmr, -unsigned long count,unsigned int align, -struct page **pagep,gfp_t gfp) +static int cma_range_alloc (struct cma_memrange *cmr, +unsigned long count,unsigned int align, +struct page **pagep,gfp_t gfp) {unsigned long mask, offset;unsigned long pfn =-1 ;
@@ -796,10 +831,11 @@ static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr,unsigned long bitmap_maxno, bitmap_no, bitmap_count;int ret = -EBUSY ;struct page *page =NULL ; +struct cma *cma = cmr->cma; mask =cma_bitmap_aligned_mask (cma, align); - offset =cma_bitmap_aligned_offset (cma, cmr, align); - bitmap_maxno =cma_bitmap_maxno (cma, cmr); + offset =cma_bitmap_aligned_offset (cmr, align); + bitmap_maxno =cma_bitmap_maxno (cmr); bitmap_count =cma_bitmap_pages_to_bits (cma, count);if (bitmap_count > bitmap_maxno)
@@ -840,7 +876,7 @@ static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr,break ; } -cma_clear_bitmap (cma, cmr, pfn, count); +cma_clear_bitmap (cmr, pfn, count);if (ret != -EBUSY )break ;
@@ -879,7 +915,7 @@ static struct page *__cma_alloc(struct cma *cma, unsigned long count,for (r =0 ; r < cma->nranges; r++) { page =NULL ; - ret =cma_range_alloc (cma, &cma->ranges[r], count, align, + ret =cma_range_alloc (&cma->ranges[r], count, align, &page, gfp);if (ret != -EBUSY || page)break ;
@@ -1011,7 +1047,7 @@ bool cma_release(struct cma *cma, const struct page *pages,return false;free_contig_range (pfn, count); -cma_clear_bitmap (cma, cmr, pfn, count); +cma_clear_bitmap (cmr, pfn, count);cma_sysfs_account_release_pages (cma, count);trace_cma_release (cma->name, pfn, pages, count);
mm/cma.h +3 -3
@@ -30,6 +30,7 @@ struct cma_memrange {unsigned long early_pfn;unsigned long *bitmap; }; +struct cma *cma;#ifdef CONFIG_CMA_DEBUGFS struct debugfs_u32_array dfs_bitmap;#endif
@@ -67,10 +68,9 @@ struct cma {extern struct cma cma_areas[MAX_CMA_AREAS ];extern unsigned int cma_area_count; -static inline unsigned long cma_bitmap_maxno (struct cma *cma, -struct cma_memrange *cmr) +static inline unsigned long cma_bitmap_maxno (struct cma_memrange *cmr) { -return cmr->count >> cma->order_per_bit; +return cmr->count >> cmr->cma->order_per_bit; }#ifdef CONFIG_CMA_SYSFS
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
kernel test robot
build botRe: [RFC PATCH 04/12] mm/cma: keep a global sorted list of CMA ranges
kernel test robot noticed the following build errors:
Hi Frank,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on powerpc/next]
[also build test ERROR on powerpc/fixes powerpc/topic/ppc-kvm s390/features linus/master v6.17-rc6 next-20250916] [cannot apply to akpm-mm/mm-everything]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Frank-van-der-Linden/mm-cma-add-tunable-for-CMA-fallback-limit/20250916-035514 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next patch link: https://lore.kernel.org/r/20250915195153.462039-5-fvdl%40google.com patch subject: [RFC PATCH 04/12] mm/cma: keep a global sorted list of CMA ranges config: i386-buildonly-randconfig-006-20250917 (https://download.01.org/0day-ci/archive/20250917/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250917/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> mm/cma_debug.c:58:40: error: too many arguments to function call, expected single argument 'cmr', have 2 arguments
58 | bitmap_maxno = cma_bitmap_maxno(cma, cmr);
| ~~~~~~~~~~~~~~~~ ^~~
mm/cma.h:71:29: note: 'cma_bitmap_maxno' declared here
71 | static inline unsigned long cma_bitmap_maxno(struct cma_memrange *cmr)
| ^ ~~~~~~~~~~~~~~~~~~~~~~~~
mm/cma_debug.c:187:39: error: too many arguments to function call, expected single argument 'cmr', have 2 arguments
187 | DIV_ROUND_UP(cma_bitmap_maxno(cma, cmr),
| ~~~~~~~~~~~~~~~~ ^~~
include/linux/math.h:49:22: note: expanded from macro 'DIV_ROUND_UP'
49 | #define DIV_ROUND_UP __KERNEL_DIV_ROUND_UP
| ^
include/uapi/linux/const.h:51:40: note: expanded from macro '__KERNEL_DIV_ROUND_UP'
51 | #define __KERNEL_DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d))
| ^ mm/cma.h:71:29: note: 'cma_bitmap_maxno' declared here
71 | static inline unsigned long cma_bitmap_maxno(struct cma_memrange *cmr)
| ^ ~~~~~~~~~~~~~~~~~~~~~~~~
2 errors generated.
vim +/cmr +58 mm/cma_debug.c
2e32b947606daa Dmitry Safonov 2015-04-15 45
2e32b947606daa Dmitry Safonov 2015-04-15 46 static int cma_maxchunk_get(void *data, u64 *val)
2e32b947606daa Dmitry Safonov 2015-04-15 47 {
2e32b947606daa Dmitry Safonov 2015-04-15 48 struct cma *cma = data;
c009da4258f988 Frank van der Linden 2025-02-28 49 struct cma_memrange *cmr;
2e32b947606daa Dmitry Safonov 2015-04-15 50 unsigned long maxchunk = 0;
c009da4258f988 Frank van der Linden 2025-02-28 51 unsigned long start, end;
c009da4258f988 Frank van der Linden 2025-02-28 52 unsigned long bitmap_maxno;
c009da4258f988 Frank van der Linden 2025-02-28 53 int r;
2e32b947606daa Dmitry Safonov 2015-04-15 54
0ef7dcac998fef Mike Kravetz 2021-05-04 55 spin_lock_irq(&cma->lock);
c009da4258f988 Frank van der Linden 2025-02-28 56 for (r = 0; r < cma->nranges; r++) {
c009da4258f988 Frank van der Linden 2025-02-28 57 cmr = &cma->ranges[r];
c009da4258f988 Frank van der Linden 2025-02-28 @58 bitmap_maxno = cma_bitmap_maxno(cma, cmr);
c79147e4b02fa3 Yury Norov (NVIDIA 2025-07-19 59) for_each_clear_bitrange(start, end, cmr->bitmap, bitmap_maxno)
2e32b947606daa Dmitry Safonov 2015-04-15 60 maxchunk = max(end - start, maxchunk);
2e32b947606daa Dmitry Safonov 2015-04-15 61 }
0ef7dcac998fef Mike Kravetz 2021-05-04 62 spin_unlock_irq(&cma->lock);
2e32b947606daa Dmitry Safonov 2015-04-15 63 *val = (u64)maxchunk << cma->order_per_bit;
2e32b947606daa Dmitry Safonov 2015-04-15 64
2e32b947606daa Dmitry Safonov 2015-04-15 65 return 0;
2e32b947606daa Dmitry Safonov 2015-04-15 66 }
a9ea242a063c62 zhong jiang 2019-11-30 67 DEFINE_DEBUGFS_ATTRIBUTE(cma_maxchunk_fops, cma_maxchunk_get, NULL, "%llu\n");
2e32b947606daa Dmitry Safonov 2015-04-15 68 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [RFC PATCH 04/12] mm/cma: keep a global sorted list of CMA ranges
In reply to Frank van der LindenQuoted 42 lines · Frank van der Linden
In order to walk through CMA areas efficiently, it is useful to keep a global sorted list of ranges.
Create this list when activating the areas.
Since users of this list may want to reference the CMA area the range came from, there needs to be a link from the range to that area. So, store a pointer to the CMA structure in the
cma_memrangestructure. This also reduces the number of arguments to a few internal functions.Signed-off-by Frank van der Linden <[email protected]>static int
__initcma_init_reserved_areas(void)
{mm/cma.c +12
-1-int i; +int i, r, nranges; +struct cma *cma; +struct cma_memrange *cmr; + + nranges =0 ; +for (i =0 ; i < cma_area_count; i++) { + cma = &cma_areas[i]; + nranges += cma->nranges; +cma_activate_area (cma); + } + + cma_ranges = kcalloc(nranges, sizeof(*cma_ranges),
GFP_KERNEL);mm/cma.c +8
-0+ cma_nranges =0 ; +for (i =0 ; i < cma_area_count; i++) { + cma = &cma_areas[i]; +for (r =0 ; r < cma->nranges; r++) { + cmr = &cma->ranges[r]; + cma_ranges[cma_nranges++] = cmr; + } + }mm/cma.c +1
-2-for (i =0 ; i < cma_area_count; i++) -cma_activate_area (&cma_areas[i]); + sort(cma_ranges, cma_nranges, sizeof(*cma_ranges), cmprange,
Quoted 2 lines
NULL);
I am guessing that this is safe because cma_init_reserved_areas is an initcall, which is only called once?
Is that correct?
Is it worth documenting why this function creates
a sorted array of CMA ranges?
Quoted 4 lines
return 0;
}
Signature · 1 line
All Rights Reversed.
RFC PATCH 5/12 mm/cma: add helper functions for CMA balancing
In reply to Frank van der LindenAdd some CMA helper functions to assist CMA balancing. They are:
cma_get_available.
- Returns the number of available pages in a CMA area
cma_numranges
- Returns the total number of CMA ranges.
cma_next_balance_pagerange
- Get the next CMA page range in a zone that has is available as a target for CMA balancing. This means a range that consists of CMA pageblocks that are managed by the buddy allocator (not allocated through cma_alloc). The array of CMA ranges is walked top down.
cma_next_noncma_pagerange
- Get the next non-CMA page range in a zone. The zone is traversed bottom up.
--- include/linux/cma.h | 30 +++++++++ mm/cma.c | 161 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 191 insertions(+)
include/linux/cma.h +30 -0
@@ -37,6 +37,7 @@ enum cma_flags {#define CMA_INIT_FLAGS (CMA_FIXED|CMA_RESERVE_PAGES_ON_ERROR)struct cma ; +struct zone ;extern unsigned long totalcma_pages;extern phys_addr_t cma_get_base (const struct cma *cma);
@@ -79,6 +80,12 @@ extern void cma_reserve_pages_on_error(struct cma *cma);struct folio *cma_alloc_folio (struct cma *cma,int order,gfp_t gfp);bool cma_free_folio (struct cma *cma,const struct folio *folio);bool cma_validate_zones (struct cma *cma); +int cma_numranges (void ); +unsigned long cma_get_available (const struct cma *cma); +bool cma_next_balance_pagerange (struct zone *zone,struct cma *cma,int *rindex, +unsigned long *startpfn,unsigned long *endpfn); +bool cma_next_noncma_pagerange (struct zone *zone,int *rindex, +unsigned long *startpfn,unsigned long *endpfn);#else static inline struct folio *cma_alloc_folio (struct cma *cma,int order,gfp_t gfp) {
@@ -93,6 +100,29 @@ static inline bool cma_validate_zones(struct cma *cma) {return false; } + +static inline int cma_numranges (void ) +{ +return 0 ; +} + +static inline unsigned long cma_get_available (const struct cma *cma) +{ +return 0 ; +} + +static inline bool cma_next_balance_pagerange (struct zone *zone, +struct cma *cma,int *rindex,unsigned long *start_pfn, +unsigned long *end_pfn) +{ +return false; +} + +static inline bool cma_next_noncma_pagerange (struct zone *zone,int *rindex, +unsigned long *start_pfn,unsigned long *end_pfn) +{ +return false; +}#endif #endif
mm/cma.c +161 -0
179 lines not on this page · raw message
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [RFC PATCH 05/12] mm/cma: add helper functions for CMA balancing
Quoted 11 lines · Frank van der Linden
+/*
+ * Provide the next free range in a cma memory range, as derived
+ * from the bitmap.
+ *
+ * @cmr: memory range to scan
+ * @start_pfn: the beginning of the previous range
+ * @end_pfn: the end of the previous range, zero for the first call
+ *
+ * The caller can adjust*end_pfnend use it as a starting point.
+ */
The function documentation does not really make it clear that start_pfn and end_pfn are also used for the function OUTPUT, and that the return value indicates whether or not a next area was found.
Same for cma_next_noncma_pagerange. Other than that, this looks reasonable so far.
Signature · 1 line
All Rights Reversed.
RFC PATCH 6/12 mm/cma: define and act on CMA_BALANCE flag
In reply to Frank van der LindenWhen the CMA_BALANCE flag is set for a CMA area, it means that it opts in to CMA balancing. This means two things:
1) It allows movable allocations to be migrated in to it in the case of a CMA inbalance (too much free memory in CMA pageblocks as compared to other pageblocks).
2) It is allocated top-down, so that compaction will end up migrating pages in to it. Doing this will make sure that compaction doesn't aggrevate a CMA imbalance, and that it won't fight with CMA balance migration from non-CMA to CMA.
--- include/linux/cma.h | 4 +++- mm/cma.c | 33 ++++++++++++++++++++++++++------- 2 files changed, 29 insertions(+), 8 deletions(-)
include/linux/cma.h +3 -1
@@ -26,6 +26,7 @@ enum cma_flags { __CMA_ZONES_INVALID, __CMA_ACTIVATED, __CMA_FIXED, + __CMA_BALANCE, };#define CMA_RESERVE_PAGES_ON_ERROR BIT(__CMA_RESERVE_PAGES_ON_ERROR)
@@ -33,8 +34,9 @@ enum cma_flags {#define CMA_ZONES_INVALID BIT(__CMA_ZONES_INVALID)#define CMA_ACTIVATED BIT(__CMA_ACTIVATED)#define CMA_FIXED BIT(__CMA_FIXED) +#define CMA_BALANCE BIT(__CMA_BALANCE) -#define CMA_INIT_FLAGS (CMA_FIXED|CMA_RESERVE_PAGES_ON_ERROR) +#define CMA_INIT_FLAGS (CMA_FIXED|CMA_RESERVE_PAGES_ON_ERROR|CMA_BALANCE)struct cma ;struct zone ;
mm/cma.c +26 -7
@@ -272,6 +272,9 @@ static bool cma_next_free_range(struct cma_memrange *cmr,static inline bool cma_should_balance_range (struct zone *zone,struct cma_memrange *cmr) { +if (!(cmr->cma->flags &CMA_BALANCE )) +return false; +if (page_zone (pfn_to_page (cmr->base_pfn)) != zone)return false;
@@ -527,6 +530,12 @@ static bool __init basecmp(struct cma_init_memrange *mlp,return mlp->base < mrp->base; } +static bool __init revbasecmp (struct cma_init_memrange *mlp, +struct cma_init_memrange *mrp) +{ +return mlp->base > mrp->base; +} +/* * Helper function to create sorted lists. */
@@ -575,7 +584,8 @@ static int __init cma_fixed_reserve(phys_addr_t base, phys_addr_t size) }static phys_addr_t __init cma_alloc_mem (phys_addr_t base,phys_addr_t size, -phys_addr_t align,phys_addr_t limit,int nid) +phys_addr_t align,phys_addr_t limit,int nid, +unsigned long flags) {phys_addr_t addr =0 ;
@@ -588,7 +598,8 @@ static phys_addr_t __init cma_alloc_mem(phys_addr_t base, phys_addr_t size,* like DMA/DMA32. */ #ifdef CONFIG_PHYS_ADDR_T_64BIT -if (!memblock_bottom_up () && limit >=SZ_4G + size) { +if (!(flags &CMA_BALANCE ) && !memblock_bottom_up () + && limit >=SZ_4G + size) {memblock_set_bottom_up (true); addr =memblock_alloc_range_nid (size, align,SZ_4G , limit, nid, true);
@@ -695,7 +706,7 @@ static int __init __cma_declare_contiguous_nid(phys_addr_t *basep,if (ret)return ret; }else { - base =cma_alloc_mem (base, size, alignment, limit, nid); + base =cma_alloc_mem (base, size, alignment, limit, nid, flags);if (!base)return -ENOMEM ;
@@ -851,7 +862,10 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,list_for_each_safe (mp, next, &ranges) { mlp = list_entry(mp, struct cma_init_memrange, list);list_del (mp); -list_insert_sorted (&final_ranges, mlp, basecmp); +if (flags &CMA_BALANCE ) +list_insert_sorted (&final_ranges, mlp, revbasecmp); +else +list_insert_sorted (&final_ranges, mlp, basecmp); sizesum += mlp->size;if (sizesum >= total_size)break ;
@@ -866,7 +880,12 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size,list_for_each (mp, &final_ranges) { mlp = list_entry(mp, struct cma_init_memrange, list); size =min (sizeleft, mlp->size); -if (memblock_reserve (mlp->base, size)) { +if (flags &CMA_BALANCE ) + start = (mlp->base + mlp->size - size); +else + start = mlp->base; + +if (memblock_reserve (start, size)) {/* * Unexpected error. Could go on to * the next one, but just abort to
@@ -877,9 +896,9 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size, }pr_debug ("created region %d: %016llx - %016llx\n" , - nr, (u64 )mlp->base, (u64 )mlp->base + size); + nr, (u64 )start, (u64 )start + size); cmrp = &cma->ranges[nr++]; - cmrp->base_pfn =PHYS_PFN (mlp->base); + cmrp->base_pfn =PHYS_PFN (start); cmrp->early_pfn = cmrp->base_pfn; cmrp->count = size >>PAGE_SHIFT ; cmrp->cma = cma;
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [RFC PATCH 06/12] mm/cma: define and act on CMA_BALANCE flag
Quoted 13 lines · Frank van der Linden
(quoted diff) +4
-1@@ -851,7 +862,10 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size, list_for_each_safe(mp, next, &ranges) { mlp = list_entry(mp, struct cma_init_memrange, list); list_del(mp); - list_insert_sorted(&final_ranges, mlp, basecmp); + if (flags & CMA_BALANCE) + list_insert_sorted(&final_ranges, mlp, revbasecmp); + else + list_insert_sorted(&final_ranges, mlp,
Quoted 4 lines
basecmp);
sizesum +=mlp->size;
if (sizesum >=total_size)
break;
Wait, what is this supposed to do when presented with a mix of CMA_BALANCE and !CMA_BALANCE ranges?
I can't convince myself this would always keep the CMA_BALANCE range near the top of the list, and the others near the bottom.
Also, does the regular compaction code pay any attention to the order of this list?
I have the feeling I am overlooking something big in how this patch is supposed to work, but I am not sure what that would be...
Quoted 17 lines
mm/cma.c +6
-1@@ -866,7 +880,12 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size, list_for_each(mp, &final_ranges) { mlp = list_entry(mp, struct cma_init_memrange, list); size = min(sizeleft, mlp->size); -if (memblock_reserve (mlp->base, size)) { +if (flags &CMA_BALANCE ) + start = (mlp->base + mlp->size - size); +else + start = mlp->base; + +if (memblock_reserve (start, size)) { /** Unexpected error. Could go on to
* the next one, but just abort to
Signature · 1 line
All Rights Reversed.
RFC PATCH 7/12 mm/compaction: optionally use a different isolate function
In reply to Frank van der LindenFor CMA balancing, the compaction migration hooks can largely be reused, except a different function to isolate free pages is needed.
So, add a pointer to the isolation function in the compact_control structure. If it's not NULL, use it, else use isolate_freepages as usual.
No functional change.
--- mm/compaction.c | 5 ++++- mm/internal.h | 5 +++++ 2 files changed, 9 insertions(+), 1 deletion(-)
mm/compaction.c +4 -1
@@ -1813,7 +1813,10 @@ static struct folio *compaction_alloc_noprof(struct folio *src, unsigned long daif (start_order ==NR_PAGE_ORDERS ) {if (has_isolated_pages)return NULL ; -isolate_freepages (cc); +if (cc->isolate_freepages) + cc->isolate_freepages (cc); +else +isolate_freepages (cc); has_isolated_pages = true; goto again; }
mm/internal.h +5 -0
@@ -873,6 +873,11 @@ struct compact_control {unsigned int nr_freepages;/* Number of isolated free pages */ unsigned int nr_migratepages;/* Number of pages to migrate */ unsigned long free_pfn;/* isolate_freepages search base */ +/* +* Function to use to isolate free pages, if applicable. If NULL, +* default to isolate_freepages(). +*/ +void (*isolate_freepages)(struct compact_control *cc);/* * Acts as an in/out parameter to page isolation for migration. * isolate_migratepages uses it as a search base.
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [RFC PATCH 07/12] mm/compaction: optionally use a different isolate function
Quoted 4 lines · Frank van der Linden
For CMA balancing, the compaction migration hooks can largely be reused, except a different function to isolate free pages is needed.
Quoted 8 lines
So, add a pointer to the isolation function in the
compact_controlstructure. If it's not NULL, use it, else useisolate_freepagesas usual.No functional change.
Signed-off-by Frank van der Linden <[email protected]>
A custom isolation function can certainly solve some of the issues!
Signature · 1 line
All Rights Reversed.
RFC PATCH 8/12 mm/compaction: simplify isolation order checks a bit
In reply to Frank van der LindenThe code to isolate pages for migration always checked both cc->alloc_contig and skip_isolation_on_order to determine whether a page could be isolated for migration.
Simplify this a little bit by moving the cc->alloc_contig check in to skip_isolation_on_order. Also rename alloc_contig to migrate_large, since there will be an additional user (CMA balancing) of this field soon, not just alloc_contig_range.
No functional change.
--- mm/compaction.c | 26 ++++++++++++++------------ mm/internal.h | 2 +- mm/page_alloc.c | 2 +- 3 files changed, 16 insertions(+), 14 deletions(-)
mm/compaction.c +14 -12
@@ -793,13 +793,15 @@ static bool too_many_isolated(struct compact_control *cc)/** * skip_isolation_on_order() - determine when to skip folio isolation based on * folio order and compaction target order +* @cc: compact control structure containing target order * @order: to-be-isolated folio order -* @target_order: compaction target order * * This avoids unnecessary folio isolations during compaction. */ -static bool skip_isolation_on_order (int order,int target_order) +static bool skip_isolation_on_order (struct compact_control *cc,int order) { +if (cc->migrate_large) +return false;/* * Unless we are performing global compaction (i.e., * is_via_compact_memory), skip any folios that are larger than the
@@ -807,7 +809,7 @@ static bool skip_isolation_on_order(int order, int target_order)* the desired target_order, so migrating this folio would likely fail * later. */ -if (!is_via_compact_memory (target_order) && order >= target_order) +if (!is_via_compact_memory (cc->order) && order >= cc->order)return true;/* * We limit memory compaction to pageblocks and won't try
@@ -850,6 +852,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,unsigned long next_skip_pfn =0 ;bool skip_updated = false;int ret =0 ; +unsigned int order; cc->migrate_pfn = low_pfn;
@@ -948,13 +951,13 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, }if (PageHuge (page)) { -const unsigned int order =compound_order (page);/* * skip hugetlbfs if we are not compacting for pages * bigger than its order. THPs and other compound pages * are handled below. */ -if (!cc->alloc_contig) { +if (!cc->migrate_large) { + order =compound_order (page);if (order <=MAX_PAGE_ORDER ) { low_pfn += (1UL << order) -1 ;
@@ -962,7 +965,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, } goto isolate_fail; } -/* for alloc_contig case */ +/* for migrate_large case */ if (locked) {unlock_page_lruvec_irqrestore (locked, flags); locked =NULL ;
@@ -1030,11 +1033,11 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,* skip them at once. The check is racy, but we can consider * only valid values and the only danger is skipping too much. */ -if (PageCompound (page) && !cc->alloc_contig) { -const unsigned int order =compound_order (page); +if (PageCompound (page)) { + order =compound_order (page);/* Skip based on page order and compaction target order. */ -if (skip_isolation_on_order (order, cc->order)) { +if (skip_isolation_on_order (cc, order)) {if (order <=MAX_PAGE_ORDER ) { low_pfn += (1UL << order) -1 ; nr_scanned += (1UL << order) -1 ;
@@ -1182,9 +1185,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,/* * Check LRU folio order under the lock */ -if (unlikely (skip_isolation_on_order (folio_order (folio), - cc->order) && - !cc->alloc_contig)) { + order =folio_order (folio); +if (unlikely (skip_isolation_on_order (cc, order))) { low_pfn +=folio_nr_pages (folio) -1 ; nr_scanned +=folio_nr_pages (folio) -1 ;folio_set_lru (folio);
mm/internal.h +1 -1
@@ -909,7 +909,7 @@ struct compact_control {* isolation or migration failures to * ensure forward progress. */ -bool alloc_contig;/* alloc_contig_range allocation */ +bool migrate_large;/* Always migrate large/huge pages */ };/*
mm/page_alloc.c +1 -1
@@ -6903,7 +6903,7 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end, .mode = MIGRATE_SYNC, .ignore_skip_hint = true, .no_set_skip_hint = true, - .alloc_contig = true, + .migrate_large = true, };INIT_LIST_HEAD (&cc.migratepages); enum pb_isolate_mode mode = (alloc_flags & ACR_FLAGS_CMA) ?
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [RFC PATCH 08/12] mm/compaction: simplify isolation order checks a bit
Quoted 4 lines · Frank van der Linden
The code to isolate pages for migration always checked both
cc->alloc_contigandskip_isolation_on_orderto determine whether a page could be isolated for migration.
Quoted 9 lines
Simplify this a little bit by moving the
cc->alloc_contigcheck in toskip_isolation_on_order. Also renamealloc_contigtomigrate_large, since there will be an additional user (CMA balancing) of this field soon, not justalloc_contig_range.No functional change.
Signed-off-by Frank van der Linden <[email protected]>
Signature · 1 line
All Rights Reversed.
RFC PATCH 9/12 mm/cma: introduce CMA balancing
In reply to Frank van der LindenA longstanding problem with having a lot of CMA pageblocks in the system (through hugetlb_cma), is that this limits the amount of memory that the kernel can use for its allocations. Kernel allocations are unmovable and can not come from CMA pageblocks.
This can lead to situations where kernel allocations cause OOMs, when in fact there might still enough memory available. There isn't much that can be done if the non-CMA part of memory is already taken up by unmovable allocations. That scenario can be considered a misconfigured system. But if there are movable allocations in the non-CMA areas, they are unnecessarily taking away space from the kernel.
Currently, the page allocator tries to avoid this scenario by allocating from CMA first if more than half of free pages in a zone come from CMA. But that's not a guarantee.
For example, take the case where a lot of memory is being taken up by 1G hugetlb pages, allocated from hugetlb_cma, and that the hugetlb_cma area has been fully used by hugetlbfs. This means that new movable allocations will land in the non-CMA part of memory, and that the kernel may come under memory pressure. If those allocations are long-lasting, freeing up hugetlb pages will not reduce that pressure, since the kernel can't use the new space, and the long-lasting allocations residing in non-CMA memory will stay put.
To counter this issue, introduce interfaces to explicitly move pages in to CMA areas. The number of pages moved depends on cma_first_limit. It will use that percentage to calculate the target number of pages that should be moved.
A later commit will call one of these interfaces to move pages to CMA if needed, after CMA-allocated hugetlb pages have been freed.
--- include/linux/migrate_mode.h | 1 + include/trace/events/migrate.h | 3 +- mm/compaction.c | 168 +++++++++++++++++++++++++++++++++ mm/internal.h | 4 + 4 files changed, 175 insertions(+), 1 deletion(-)
include/linux/migrate_mode.h +1 -0
@@ -25,6 +25,7 @@ enum migrate_reason {MR_LONGTERM_PIN ,MR_DEMOTION ,MR_DAMON , +MR_CMA_BALANCE , MR_TYPES };
include/trace/events/migrate.h +2 -1
@@ -23,7 +23,8 @@ EM( MR_CONTIG_RANGE, "contig_range") \ EM( MR_LONGTERM_PIN, "longterm_pin") \ EM( MR_DEMOTION, "demotion") \ -EMe (MR_DAMON ,"damon" ) + EM( MR_DAMON, "damon") \ +EMe (MR_CMA_BALANCE ,"cma_balance" )/* * First define the enums in the above macros to be exported to userspace
mm/compaction.c +168 -0
180 lines not on this page · raw message
mm/internal.h +4 -0
@@ -857,6 +857,8 @@ void memmap_init_range(unsigned long, int, unsigned long, unsigned long,#if definedCONFIG_COMPACTION || definedCONFIG_CMA +struct cma ; +/* * in mm/compaction.c */
@@ -887,6 +889,7 @@ struct compact_control {unsigned long migrate_pfn;unsigned long fast_start_pfn;/* a pfn to start linear scan from */ struct zone *zone; +struct cma *cma;/* if moving to a specific CMA area */ unsigned long total_migrate_scanned;unsigned long total_free_scanned;unsigned short fast_search_fail;/* failures to use free list searches */
@@ -938,6 +941,7 @@ struct cma;#ifdef CONFIG_CMA void *cma_reserve_early (struct cma *cma,unsigned long size);void init_cma_pageblock (struct page *page); +void balance_node_cma (int nid,struct cma *cma);#else static inline void *cma_reserve_early (struct cma *cma,unsigned long size) {
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [RFC PATCH 09/12] mm/cma: introduce CMA balancing
Quoted 7 lines · Frank van der Linden
A longstanding problem with having a lot of CMA pageblocks in the system (through
hugetlb_cma), is that this limits the amount of memory that the kernel can use for its allocations. Kernel allocations are unmovable and can not come from CMA pageblocks.This can lead to situations where kernel allocations cause OOMs, when in fact there might still enough memory available.
We are seeing this situation happen even when the CMA region is relatively small.
When a system has multiple cgroups, and has been reclaiming the non-CMA memory for a particular cgroup for long enough, at some point essentially all the remaining reclaimable memory for that cgroup can be in CMA, and you get early OOM kills.
Quoted 7 lines
To counter this issue, introduce interfaces to explicitly move pages in to CMA areas. The number of pages moved depends on
cma_first_limit. It will use that percentage to calculate the target number of pages that should be moved.
I've been working on some code for this situation as well, but your code seems to handle a lot more corner cases than the small hack I was working on.
I hope this issue can be fixed upstream soon-ish :)
Quoted 3 lines
A later commit will call one of these interfaces to move pages to CMA if needed, after CMA-allocated hugetlb pages have been freed.
Is that the right place?
The CMA region will also be used by regular 4kB pages and 2MB THPs, and we may also need to do CMA balancing if the cgroup is up against its memory limit, needs to do a kernel allocation, but its remaining freeable memory is in movable allocations in the CMA region.
I have the hook for migrating pages to CMA on the page reclaim side, when the reclaim code notices that it is doing direct reclaim on behalf of a non-movable (kernel) allocation, but it has been reclaiming a bunch of CMA pages.
Hugetlb is not the only case triggering this issue.
Quoted 3 lines
Signed-off-by Frank van der Linden <[email protected]>
Signature · 1 line
All Rights Reversed.
RFC PATCH 10/12 mm/hugetlb: do explicit CMA balancing
In reply to Frank van der LindenCMA areas are normally not very large, but HugeTLB CMA is an exception. hugetlb_cma, used for 'gigantic' pages (usually 1G), can take up many gigabytes of memory.
As such, it is potentially the largest source of 'false OOM' conditions, situations where the kernel runs out of space for unmovable allocations, because it can't allocate from CMA pageblocks, and non-CMA memory has been tied up by other movable allocations.
The normal use case of hugetlb_cma is a system where 1G hugetlb pages are sometimes, but not always, needed, so they need to be created and freed dynamically. As such, the best time to address CMA memory imbalances is when CMA hugetlb pages are freed, making multiples of 1G available as buddy managed CMA pageblocks. That is a good time to check if movable allocations fron non-CMA pageblocks should be moved to CMA pageblocks to give the kernel more breathing space.
Do this by calling balance_node_cma on either the hugetlb CMA area for the node that just had its number of hugetlb pages reduced, or for all hugetlb CMA areas if the reduction was not node-specific.
To have the CMA balancing code act on the hugetlb CMA areas, set the CMA_BALANCE flag when creating them.
--- mm/hugetlb.c | 14 ++++++++------ mm/hugetlb_cma.c | 16 ++++++++++++++++ mm/hugetlb_cma.h | 5 +++++ 3 files changed, 29 insertions(+), 6 deletions(-)
mm/hugetlb.c +8 -6
@@ -3971,12 +3971,14 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid,list_add (&folio->lru, &page_list); } -/* free the pages after dropping lock */ -spin_unlock_irq (&hugetlb_lock); -update_and_free_pages_bulk (h, &page_list); -flush_free_hpage_work (h); -spin_lock_irq (&hugetlb_lock); - +if (!list_empty (&page_list)) { +/* free the pages after dropping lock */ +spin_unlock_irq (&hugetlb_lock); +update_and_free_pages_bulk (h, &page_list); +flush_free_hpage_work (h); +hugetlb_cma_balance (nid); +spin_lock_irq (&hugetlb_lock); + }while (count <persistent_huge_pages (h)) {if (!adjust_pool_surplus (h, nodes_allowed,1 ))break ;
mm/hugetlb_cma.c +16 -0
@@ -276,3 +276,19 @@ bool __init hugetlb_early_cma(struct hstate *h)return hstate_is_gigantic (h) && hugetlb_cma_only; } + +void hugetlb_cma_balance (int nid) +{ +int node; + +if (nid !=NUMA_NO_NODE ) { +if (hugetlb_cma[nid]) +balance_node_cma (nid, hugetlb_cma[nid]); + }else { +for_each_online_node (node) { +if (hugetlb_cma[node]) +balance_node_cma (node, + hugetlb_cma[node]); + } + } +}
mm/hugetlb_cma.h +5 -0
@@ -13,6 +13,7 @@ bool hugetlb_cma_exclusive_alloc(void);unsigned long hugetlb_cma_total_size (void );void hugetlb_cma_validate_params (void );bool hugetlb_early_cma (struct hstate *h); +void hugetlb_cma_balance (int nid);#else static inline void hugetlb_cma_free_folio (struct folio *folio) {
@@ -53,5 +54,9 @@ static inline bool hugetlb_early_cma(struct hstate *h) {return false; } + +static inline void hugetlb_cma_balance (int nid) +{ +}#endif #endif
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [RFC PATCH 10/12] mm/hugetlb: do explicit CMA balancing
Quoted 6 lines · Frank van der Linden
CMA areas are normally not very large, but HugeTLB CMA is an exception.
hugetlb_cma, used for 'gigantic' pages (usually 1G), can take up many gigabytes of memory.As such, it is potentially the largest source of 'false OOM' conditions,
The false OOM kills also seem to happen when a system does not use hugetlbfs at all, but a cgroup simply has most/all of its reclaimable memory in a CMA region, and then tries to do a kernel allocation.
Would it make sense to call hugetlb_cma_balance() from the pageout code instead, when the pageout code is trying to free non-CMA memory, but ended up freeing mostly/only CMA memory?
Signature · 1 line
All Rights Reversed.
kernel test robot
build botRe: [RFC PATCH 10/12] mm/hugetlb: do explicit CMA balancing
In reply to Frank van der Lindenkernel test robot noticed the following build errors:
Hi Frank,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:
[auto build test ERROR on powerpc/next]
[also build test ERROR on powerpc/fixes powerpc/topic/ppc-kvm s390/features trace/for-next linus/master v6.17-rc7 next-20250926] [cannot apply to akpm-mm/mm-everything]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Frank-van-der-Linden/mm-cma-add-tunable-for-CMA-fallback-limit/20250926-101411 base: https://git.kernel.org/pub/scm/linux/kernel/git/powerpc/linux.git next patch link: https://lore.kernel.org/r/20250915195153.462039-11-fvdl%40google.com patch subject: [RFC PATCH 10/12] mm/hugetlb: do explicit CMA balancing config: x86_64-buildonly-randconfig-001-20250926 (https://download.01.org/0day-ci/archive/20250927/[email protected]/config) compiler: clang version 20.1.8 (https://github.com/llvm/llvm-project 87f0227cb60147a26a1eeb4fb06e3b505e9c7261) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250927/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> ld.lld: error: undefined symbol: balance_node_cma >>> referenced by usercopy_64.c >>> vmlinux.o:(hugetlb_cma_balance)
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RFC PATCH 11/12 mm/cma: rebalance CMA when changing cma_first_limit
In reply to Frank van der LindenTo keep things consistent, rebalance CMA when changing the cma_first_limit sysctl.
--- mm/page_alloc.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-)
mm/page_alloc.c +19 -1
@@ -6640,6 +6640,24 @@ static int percpu_pagelist_high_fraction_sysctl_handler(const struct ctl_table *return ret; } +#ifdef CONFIG_CMA +static int cma_first_limit_sysctl_handler (const struct ctl_table *table, +int write,void *buffer,size_t *length, +loff_t *ppos) +{ +int ret, nid; + + ret =proc_dointvec_minmax (table, write, buffer, length, ppos); +if (ret || !write) +return ret; + +for_each_node_state (nid,N_MEMORY ) +balance_node_cma (nid,NULL ); + +return 0 ; +} +#endif +static const struct ctl_table page_alloc_sysctl_table[] = { { .procname ="min_free_kbytes" ,
@@ -6723,7 +6741,7 @@ static const struct ctl_table page_alloc_sysctl_table[] = { .data = &cma_first_limit, .maxlen = sizeof(cma_first_limit), .mode = 0644, - .proc_handler = proc_dointvec_minmax, + .proc_handler = cma_first_limit_sysctl_handler, .extra1 = SYSCTL_ZERO, .extra2 = SYSCTL_ONE_HUNDRED, },
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [RFC PATCH 11/12] mm/cma: rebalance CMA when changing cma_first_limit
Quoted 5 lines · Frank van der Linden
To keep things consistent, rebalance CMA when changing the
cma_first_limitsysctl.Signed-off-by Frank van der Linden <[email protected]>
Signature · 1 line
All Rights Reversed.
RFC PATCH 12/12 mm/cma: add CMA balance VM event counter
In reply to Frank van der LindenAdd VM counters that record the number of migration success / failures during CMA rebalancing. This is similar to other migrate counters.
--- include/linux/vm_event_item.h | 3 +++ mm/migrate.c | 8 ++++++++ mm/vmstat.c | 2 ++ 3 files changed, 13 insertions(+)
include/linux/vm_event_item.h +3 -0
@@ -87,6 +87,9 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,CMA_ALLOC_SUCCESS ,CMA_ALLOC_FAIL , #endif +CMA_BALANCE_MIGRATE_SUCCESS , +CMA_BALANCE_MIGRATE_FAIL , +UNEVICTABLE_PGCULLED ,/* culled to noreclaim list */ UNEVICTABLE_PGSCANNED ,/* scanned for reclaimability */ UNEVICTABLE_PGRESCUED, /* rescued from noreclaim list */
mm/migrate.c +8 -0
@@ -2144,6 +2144,14 @@ int migrate_pages(struct list_head *from, new_folio_t get_new_folio,count_vm_events (THP_MIGRATION_SUCCESS , stats.nr_thp_succeeded);count_vm_events (THP_MIGRATION_FAIL , stats.nr_thp_failed);count_vm_events (THP_MIGRATION_SPLIT , stats.nr_thp_split); + +if (reason ==MR_CMA_BALANCE ) { +count_vm_events (CMA_BALANCE_MIGRATE_SUCCESS , + stats.nr_succeeded); +count_vm_events (CMA_BALANCE_MIGRATE_FAIL , + stats.nr_failed_pages); + } + trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages, stats.nr_thp_succeeded, stats.nr_thp_failed, stats.nr_thp_split, stats.nr_split, mode,
mm/vmstat.c +2 -0
@@ -1392,6 +1392,8 @@ const char * const vmstat_text[] = { #ifdef CONFIG_CMA [I(CMA_ALLOC_SUCCESS)] = "cma_alloc_success", [I(CMA_ALLOC_FAIL)] = "cma_alloc_fail", + [I(CMA_BALANCE_MIGRATE_SUCCESS)] = "cma_balance_migrate_success", + [I(CMA_BALANCE_MIGRATE_FAIL)] = "cma_balance_migrate_fail", #endif [I(UNEVICTABLE_PGCULLED)] = "unevictable_pgs_culled", [I(UNEVICTABLE_PGSCANNED)] = "unevictable_pgs_scanned",
Signature · 1 line
2.51.0.384.g4c02a37b29-goog
Re: [RFC PATCH 12/12] mm/cma: add CMA balance VM event counter
Quoted 5 lines · Frank van der Linden
Add VM counters that record the number of migration success / failures during CMA rebalancing. This is similar to other migrate counters.
Signed-off-by Frank van der Linden <[email protected]>
Signature · 1 line
All Rights Reversed.
Quoted 64 lines · Frank van der Linden
This is an RFC on a solution to the long standing problem of OOMs occuring when the kernel runs out of space for unmovable allocations in the face of large amounts of CMA.
Introduction
============When there is a large amount of CMA (e.g. with
hugetlb_cma), it is possible for the kernel to run out of space to get unmovable allocations from. This is because it cannot use the CMA area. If the issue is just that there is a large CMA area, and that there isn't enough space left, that can be considered a misconfigured system. However, there is a scenario in which things could have been dealt with better: if the non-CMA area also has movable allocations in it, and there are CMA pageblocks still available.The current mitigation for this issue is to start using CMA pageblocks for movable allocations first if the amount of free CMA pageblocks is more than 50% of the total amount of free memory in a zone. But that may not always work out, e.g. the system could easily run in to a scenario where long-lasting movable allocations are made first, which do not go to CMA before the 50% mark is reached. When the non-CMA area fills up, these will get in the way of the kernel's unmovable allocations, and OOMs might occur.
Even always directing movable allocations to CMA first does not completely fix the issue. Take a scenario where there is a large amount of CMA through
hugetlb_cma. All of that CMA has been taken up by 1G hugetlb pages. So, movable allocations end up in the non-CMA area. Now, the number of hugetlb pages in the pool is lowered, so some CMA becomes available. At the same time, increased system activity leads to more unmovable allocations. Since the movable allocations are still in the non-CMA area, these kernel allocations might still fail.Additionally, CMA areas are allocated at the bottom of the zone. There has been some discussion on this in the past. Originally, doing allocations from CMA was deemed something that was best avoided. The arguments were twofold:
1)
cma_allocneeds to be quick and should not have to migrate a lot of pages.
2) migration might fail, so the fewer pages it has to migrate the betterThese arguments are why CMA is avoided (until the 50% limit is hit), and why CMA areas are allocated at the bottom of a zone. But compaction migrates memory from the bottom to the top of a zone. That means that compaction will actually end up migrating movable allocations out of CMA and in to non-CMA, making the issue of OOMing for unmovable allocations worse.
Solution: CMA balancing
=======================First, this patch set makes the 50% threshold configurable, which is useful in any case. vm.cma_first_limit is the percentage of free CMA, as part of the total amount of free memory in a zone, above which CMA will be used first for movable allocations. 0 is always, 100 is never.
Quoted 4 lines
Then, it creates an interface that allows for moving movable allocations from non-CMA to CMA. CMA areas opt in to taking part in this through a flag. Also, if the flag is set for a CMA area, it is allocated at the top of a zone instead of the bottom.
Hm, what if we can teach the compaction code to start off the beginning of the zone or end of cma zone(s) depending on the current balance?
The problem with placing the cma area at the end is that it might significantly decrease the success rate of cma allocations when it's racing with the background compaction, which is hard to control. At least it was clearly so in my measurements several years ago.
Quoted 15 lines
Lastly, the
hugetlb_cmacode was modified to try to migrate movable allocations from non-CMA to CMA when a hugetlb CMA page is freed. Only hugetlb CMA areas opt in to CMA balancing, behavior for all other CMA areas is unchanged.Discussion
==========This approach works when tested with a
hugetlb_cmasetup where a large number of 1G pages is active, but the number is sometimes reduced in exchange for larger non-hugetlb overhead.Arguments against this approach:
Quoted 8 lines
* It's kind of heavy-handed. Since there is no easy way to track the amount of movable allocations residing in non-CMA pageblocks, it will likely end up scanning too much memory, as it only knows the upper bound.
* It should be more integrated with watermark handling in the allocation slow path. Again, this would likely require tracking the number of movable allocations in non-CMA pageblocks.
I think the problem is very real and the proposed approach looks reasonable. But I also agree that it's heavy-handed. Doesn't feel like "the final" solution :)
I wonder if we can track the amount of free space outside of cma and move pages out on reaching a certain low threshold? And it can in theory be the part of the generic kswapd/reclaim code.
Quoted 76 lines · Roman Gushchin
Quoted 64 lines · Frank van der Linden
This is an RFC on a solution to the long standing problem of OOMs occuring when the kernel runs out of space for unmovable allocations in the face of large amounts of CMA.
Introduction
============When there is a large amount of CMA (e.g. with
hugetlb_cma), it is possible for the kernel to run out of space to get unmovable allocations from. This is because it cannot use the CMA area. If the issue is just that there is a large CMA area, and that there isn't enough space left, that can be considered a misconfigured system. However, there is a scenario in which things could have been dealt with better: if the non-CMA area also has movable allocations in it, and there are CMA pageblocks still available.The current mitigation for this issue is to start using CMA pageblocks for movable allocations first if the amount of free CMA pageblocks is more than 50% of the total amount of free memory in a zone. But that may not always work out, e.g. the system could easily run in to a scenario where long-lasting movable allocations are made first, which do not go to CMA before the 50% mark is reached. When the non-CMA area fills up, these will get in the way of the kernel's unmovable allocations, and OOMs might occur.
Even always directing movable allocations to CMA first does not completely fix the issue. Take a scenario where there is a large amount of CMA through
hugetlb_cma. All of that CMA has been taken up by 1G hugetlb pages. So, movable allocations end up in the non-CMA area. Now, the number of hugetlb pages in the pool is lowered, so some CMA becomes available. At the same time, increased system activity leads to more unmovable allocations. Since the movable allocations are still in the non-CMA area, these kernel allocations might still fail.Additionally, CMA areas are allocated at the bottom of the zone. There has been some discussion on this in the past. Originally, doing allocations from CMA was deemed something that was best avoided. The arguments were twofold:
1)
cma_allocneeds to be quick and should not have to migrate a lot of pages.
2) migration might fail, so the fewer pages it has to migrate the betterThese arguments are why CMA is avoided (until the 50% limit is hit), and why CMA areas are allocated at the bottom of a zone. But compaction migrates memory from the bottom to the top of a zone. That means that compaction will actually end up migrating movable allocations out of CMA and in to non-CMA, making the issue of OOMing for unmovable allocations worse.
Solution: CMA balancing
=======================First, this patch set makes the 50% threshold configurable, which is useful in any case. vm.cma_first_limit is the percentage of free CMA, as part of the total amount of free memory in a zone, above which CMA will be used first for movable allocations. 0 is always, 100 is never.
Quoted 4 lines
Then, it creates an interface that allows for moving movable allocations from non-CMA to CMA. CMA areas opt in to taking part in this through a flag. Also, if the flag is set for a CMA area, it is allocated at the top of a zone instead of the bottom.
Hm, what if we can teach the compaction code to start off the beginning of the zone or end of cma zone(s) depending on the current balance?
Quoted 5 lines
The problem with placing the cma area at the end is that it might significantly decrease the success rate of cma allocations when it's racing with the background compaction, which is hard to control. At least it was clearly so in my measurements several years ago.
Indeed, I saw your change that moved the CMA areas to the bottom of the zone for that reason. In my testing, I saw a slight uptick in cma_alloc failures for HugeTLB (due to migration failures), but it wasn't much at all. Also, our current usage scenario can deal with the occasional failure, so it was less of a concern. I can try to re-run some tests to see if I can gather some harder numbers on that - the problem is of course finding a test case that gives reproducible results.
Quoted 26 lines
Quoted 17 lines
Lastly, the
hugetlb_cmacode was modified to try to migrate movable allocations from non-CMA to CMA when a hugetlb CMA page is freed. Only hugetlb CMA areas opt in to CMA balancing, behavior for all other CMA areas is unchanged.Discussion
==========This approach works when tested with a
hugetlb_cmasetup where a large number of 1G pages is active, but the number is sometimes reduced in exchange for larger non-hugetlb overhead.Arguments against this approach:
Quoted 8 lines
* It's kind of heavy-handed. Since there is no easy way to track the amount of movable allocations residing in non-CMA pageblocks, it will likely end up scanning too much memory, as it only knows the upper bound.
* It should be more integrated with watermark handling in the allocation slow path. Again, this would likely require tracking the number of movable allocations in non-CMA pageblocks.
Quoted 7 lines
I think the problem is very real and the proposed approach looks reasonable. But I also agree that it's heavy-handed. Doesn't feel like "the final" solution :)
I wonder if we can track the amount of free space outside of cma and move pages out on reaching a certain low threshold? And it can in theory be the part of the generic kswapd/reclaim code.
I considered this, yes. The first problem is that there is no easy way to express the number that is "pages allocated with __GFP_MOVABLE in non-CMA pageblocks". You can approximate pretty well by checking if they are on the LRU, I suppose.
If you succeed in getting that number accurately, the next issue is defining the right threshold and when to apply them. E.g. at one point I had a change to skip CMA pageblocks for compaction if the target pageblock is non-CMA, and the threshold has been hit. I ended up dropping it, since this more special-case approach was better for our use case. But my idea at the time was to add it as a 3rd mechanism to try harder for allocations (compaction, reclaim, CMA balancing).
It was something like:
1) Track movable allocations in non-CMA areas.
2) If the watermark for an unmovable allocation is below high, stop migrating things (through compaction) from CMA to non-CMA, and always start allocating from CMA first.
3) If the watermark is approaching low, don't try compaction if you know that CMA can be balanced, but do CMA balancing instead, in amounts that satisfy your needs
One problem here is ping-ponging of memory. If you put CMA areas at the bottom of the zone, ompaction moves things one way, CMA balancing the other way.
I think an approach like the above could still work, I just abandoned it in favor of this more special-cased (and thus safer) one for our use case.
- Frank
Quoted 83 lines · Frank van der Linden
Quoted 76 lines · Roman Gushchin
Quoted 64 lines · Frank van der Linden
This is an RFC on a solution to the long standing problem of OOMs occuring when the kernel runs out of space for unmovable allocations in the face of large amounts of CMA.
Introduction
============When there is a large amount of CMA (e.g. with
hugetlb_cma), it is possible for the kernel to run out of space to get unmovable allocations from. This is because it cannot use the CMA area. If the issue is just that there is a large CMA area, and that there isn't enough space left, that can be considered a misconfigured system. However, there is a scenario in which things could have been dealt with better: if the non-CMA area also has movable allocations in it, and there are CMA pageblocks still available.The current mitigation for this issue is to start using CMA pageblocks for movable allocations first if the amount of free CMA pageblocks is more than 50% of the total amount of free memory in a zone. But that may not always work out, e.g. the system could easily run in to a scenario where long-lasting movable allocations are made first, which do not go to CMA before the 50% mark is reached. When the non-CMA area fills up, these will get in the way of the kernel's unmovable allocations, and OOMs might occur.
Even always directing movable allocations to CMA first does not completely fix the issue. Take a scenario where there is a large amount of CMA through
hugetlb_cma. All of that CMA has been taken up by 1G hugetlb pages. So, movable allocations end up in the non-CMA area. Now, the number of hugetlb pages in the pool is lowered, so some CMA becomes available. At the same time, increased system activity leads to more unmovable allocations. Since the movable allocations are still in the non-CMA area, these kernel allocations might still fail.Additionally, CMA areas are allocated at the bottom of the zone. There has been some discussion on this in the past. Originally, doing allocations from CMA was deemed something that was best avoided. The arguments were twofold:
1)
cma_allocneeds to be quick and should not have to migrate a lot of pages.
2) migration might fail, so the fewer pages it has to migrate the betterThese arguments are why CMA is avoided (until the 50% limit is hit), and why CMA areas are allocated at the bottom of a zone. But compaction migrates memory from the bottom to the top of a zone. That means that compaction will actually end up migrating movable allocations out of CMA and in to non-CMA, making the issue of OOMing for unmovable allocations worse.
Solution: CMA balancing
=======================First, this patch set makes the 50% threshold configurable, which is useful in any case. vm.cma_first_limit is the percentage of free CMA, as part of the total amount of free memory in a zone, above which CMA will be used first for movable allocations. 0 is always, 100 is never.
Quoted 4 lines
Then, it creates an interface that allows for moving movable allocations from non-CMA to CMA. CMA areas opt in to taking part in this through a flag. Also, if the flag is set for a CMA area, it is allocated at the top of a zone instead of the bottom.
Hm, what if we can teach the compaction code to start off the beginning of the zone or end of cma zone(s) depending on the current balance?
Quoted 5 lines
The problem with placing the cma area at the end is that it might significantly decrease the success rate of cma allocations when it's racing with the background compaction, which is hard to control. At least it was clearly so in my measurements several years ago.
Quoted 8 lines
Indeed, I saw your change that moved the CMA areas to the bottom of the zone for that reason. In my testing, I saw a slight uptick in
cma_allocfailures for HugeTLB (due to migration failures), but it wasn't much at all. Also, our current usage scenario can deal with the occasional failure, so it was less of a concern. I can try to re-run some tests to see if I can gather some harder numbers on that - the problem is of course finding a test case that gives reproducible results.
It might heavily depend on which file system you're using.
Quoted 30 lines
Quoted 30 lines
Quoted 17 lines
Lastly, the
hugetlb_cmacode was modified to try to migrate movable allocations from non-CMA to CMA when a hugetlb CMA page is freed. Only hugetlb CMA areas opt in to CMA balancing, behavior for all other CMA areas is unchanged.Discussion
==========This approach works when tested with a
hugetlb_cmasetup where a large number of 1G pages is active, but the number is sometimes reduced in exchange for larger non-hugetlb overhead.Arguments against this approach:
Quoted 8 lines
* It's kind of heavy-handed. Since there is no easy way to track the amount of movable allocations residing in non-CMA pageblocks, it will likely end up scanning too much memory, as it only knows the upper bound.
* It should be more integrated with watermark handling in the allocation slow path. Again, this would likely require tracking the number of movable allocations in non-CMA pageblocks.I think the problem is very real and the proposed approach looks reasonable. But I also agree that it's heavy-handed. Doesn't feel like "the final" solution :)
Quoted 8 lines
Quoted 3 lines
I wonder if we can track the amount of free space outside of cma and move pages out on reaching a certain low threshold? And it can in theory be the part of the generic kswapd/reclaim code.
I considered this, yes. The first problem is that there is no easy way to express the number that is "pages allocated with
__GFP_MOVABLEin non-CMA pageblocks". You can approximate pretty well by checking if they are on the LRU, I suppose.
Hm, but why do you need it? Even if there are movable pages outside of cma, but there are no or little free pages, we can start (trying) moving movable pages into cma, right?
Quoted 8 lines
If you succeed in getting that number accurately, the next issue is defining the right threshold and when to apply them. E.g. at one point I had a change to skip CMA pageblocks for compaction if the target pageblock is non-CMA, and the threshold has been hit. I ended up dropping it, since this more special-case approach was better for our use case. But my idea at the time was to add it as a 3rd mechanism to try harder for allocations (compaction, reclaim, CMA balancing).
Agree, but I guess we can pick some magic number.
Quoted 11 lines
It was something like:
1) Track movable allocations in non-CMA areas.
2) If the watermark for an unmovable allocation is below high, stop migrating things (through compaction) from CMA to non-CMA, and always start allocating from CMA first.
3) If the watermark is approaching low, don't try compaction if you know that CMA can be balanced, but do CMA balancing instead, in amounts that satisfy your needs
Quoted 3 lines
One problem here is ping-ponging of memory. If you put CMA areas at the bottom of the zone, ompaction moves things one way, CMA balancing the other way.
Agree, it's a valid concern.
[RFC PATCH 13/12] mm,cma: add compaction cma balance helper for direct reclaim
In reply to Frank van der LindenQuoted 3 lines · On Mon, 15 Sep 2025 19:51:41 +0000 Frank van der Linden <[email protected]> wrote:
This is an RFC on a solution to the long standing problem of OOMs occuring when the kernel runs out of space for unmovable allocations in the face of large amounts of CMA.
In order to make the CMA balancing code useful without hugetlb involvement, eg. when simply allocating a !__GFP_MOVABLE allocation, I added two patches to invoke CMA balancing from the page reclaim code when needed.
With these changes, we might no longer need to call the CMA balancing code from the hugetlb free path any more, and could potentially simplify some things in that area.
---8<---
From 99991606760fdf8399255d7fc1f21b58069a4afe Mon Sep 17 00:00:00 2001
From: Rik van Riel <[email protected]>
Date: Tue, 23 Sep 2025 10:01:42 -0700
Subject: [PATCH 2/3] mm,cma: add compaction cma balance helper for direct reclaim
Add a cma balance helper for the direct reclaim code, which does not balance CMA free memory all the way, but only a limited number of pages.
--- mm/compaction.c | 20 ++++++++++++++++++-- mm/internal.h | 7 +++++++ 2 files changed, 25 insertions(+), 2 deletions(-)
mm/compaction.c +18 -2
@@ -2541,7 +2541,7 @@ isolate_free_cma_pages(struct compact_control *cc) cc->free_pfn = next_pfn; } -static void balance_zone_cma (struct zone *zone,struct cma *cma) +static void balance_zone_cma (struct zone *zone,struct cma *cma,int target) {struct compact_control cc = { .zone = zone,
@@ -2613,6 +2613,13 @@ static void balance_zone_cma(struct zone *zone, struct cma *cma) nr_pages =min (nr_pages,cma_get_available (cma)); nr_pages =min (allocated_noncma, nr_pages); +/* +* When invoked from page reclaim, use the provided target rather +* than the calculated one. +*/ +if (target) + nr_pages = target; +for (order =0 ; order <NR_PAGE_ORDERS ; order++)INIT_LIST_HEAD (&cc.freepages[order]);INIT_LIST_HEAD (&cc.migratepages);
@@ -2674,10 +2681,19 @@ void balance_node_cma(int nid, struct cma *cma)if (!populated_zone (zone))continue ; - balance_zone_cma(zone, cma); + balance_zone_cma(zone, cma, 0); } } +void balance_cma_zonelist (struct zonelist *zonelist,int nr_pages) +{ +struct zoneref *z; +struct zone *zone; + +for_each_zone_zonelist (zone, z, zonelist,MAX_NR_ZONES -1 ) +balance_zone_cma (zone,NULL , nr_pages); +} +#endif /* CONFIG_CMA */ static enum compact_result
mm/internal.h +7 -0
@@ -942,6 +942,7 @@ struct cma;void *cma_reserve_early (struct cma *cma,unsigned long size);void init_cma_pageblock (struct page *page);void balance_node_cma (int nid,struct cma *cma); +void balance_cma_zonelist (struct zonelist *zonelist,int nr_pages);#else static inline void *cma_reserve_early (struct cma *cma,unsigned long size) {
@@ -950,6 +951,12 @@ static inline void *cma_reserve_early(struct cma *cma, unsigned long size)static inline void init_cma_pageblock (struct page *page) { } +static inline void balance_node_cma (int nid,struct cma *cma) +{ +} +static inline void balance_cma_zonelist (struct zonelist *zonelist,int nr_pages) +{ +}#endif
Signature · 1 line
2.47.3
cover mm,cma: call CMA balancing from page reclaim code
In reply to Frank van der LindenCall CMA balancing from the page reclaim code, if page reclaim is reclaiming pages that are unsuitable for the allocator.
To keep direct reclaim latencies low, kswapd will do CMA balancing whenever some of the reclaimed pages are unsuitable for the allocator that woke up kswapd, while the direct reclaimers will only do CMA balancing if most of the reclaimed pages are unsuitable.
--- mm/vmscan.c | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-)
mm/vmscan.c +30 -1
@@ -168,6 +168,9 @@ struct scan_control {/* Number of pages freed so far during a call to shrink_zones() */ unsigned long nr_reclaimed; +/* Number of pages reclaimed, but unsuitable to the allocator */ +unsigned long nr_unsuitable; +struct {unsigned int dirty;unsigned int unqueued_dirty;
@@ -1092,6 +1095,19 @@ static bool may_enter_fs(struct folio *folio, gfp_t gfp_mask)return !data_race (folio_swap_flags (folio) &SWP_FS_OPS ); } +#ifdef CONFIG_CMA +static bool unsuitable_folio (struct folio *folio,struct scan_control *sc) +{ +return gfp_migratetype (sc->gfp_mask) !=MIGRATE_MOVABLE && +folio_migratetype (folio) ==MIGRATE_CMA ; +} +#else +static bool unsuitable_folio (struct folio *folio,struct scan_control *sc) +{ +return false; +} +#endif +/* * shrink_folio_list() returns the number of reclaimed pages */
@@ -1103,7 +1119,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,struct folio_batch free_folios;LIST_HEAD (ret_folios);LIST_HEAD (demote_folios); -unsigned int nr_reclaimed =0 , nr_demoted =0 ; +unsigned int nr_reclaimed =0 , nr_demoted =0 , nr_unsuitable =0 ;unsigned int pgactivate =0 ;bool do_demote_pass;struct swap_iocb *plug =NULL ;
@@ -1530,6 +1546,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,* leave it off the LRU). */ nr_reclaimed += nr_pages; +if (unsuitable_folio (folio, sc)) + nr_unsuitable += nr_pages;continue ; } }
@@ -1560,6 +1578,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,* all pages in it. */ nr_reclaimed += nr_pages; +if (unsuitable_folio (folio, sc)) + nr_unsuitable += nr_pages;folio_unqueue_deferred_split (folio);if (folio_batch_add (&free_folios, folio) ==0 ) {
@@ -1641,6 +1661,9 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (plug)swap_write_unplug (plug); + + sc->nr_unsuitable += nr_unsuitable; +return nr_reclaimed; }
@@ -6431,6 +6454,10 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,delayacct_freepages_end (); +/* Almost all memory reclaimed was unsuitable? Move data into CMA. */ +if (sc->nr_unsuitable >= sc->nr_reclaimed -2 ) +balance_cma_zonelist (zonelist,SWAP_CLUSTER_MAX ); +if (sc->nr_reclaimed)return sc->nr_reclaimed;
@@ -7169,6 +7196,8 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx)if (!sc.nr_reclaimed) pgdat->kswapd_failures++; +if (sc.nr_unsuitable) +balance_node_cma (pgdat->node_id,NULL ); out:clear_reclaim_active (pgdat, highest_zoneidx);
Signature · 1 line
2.47.3