mm: cma: make cma_release() non-blocking
Subsystems (3)
cover mm: cma: make cma_release() non-blocking
This small patchset makes cma_release() non-blocking and simplifies the code in hugetlbfs, where previously we had to temporarily drop hugetlb_lock around the cma_release() call.
It should help Zi Yan on his work on 1 GB THPs: splitting a gigantic THP under a memory pressure requires a cma_release() call. If it's a blocking function, it complicates the already complicated code. Because there are at least two use cases like this (hugetlbfs is another example), I believe it's just better to make cma_release() non-blocking.
It also makes it more consistent with other memory releasing functions in the kernel: most of them are non-blocking.
Roman Gushchin (2):
mm: cma: make cma_release() non-blocking
mm: hugetlb: don't drop hugetlb_lock around cma_release() call
mm/cma.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++-- mm/hugetlb.c | 6 ------ 2 files changed, 49 insertions(+), 8 deletions(-)
Signature · 1 line
2.26.2
RFC PATCH 1/2 mm: cma: make cma_release() non-blocking
cma_release() has to lock the cma_lock mutex to clear the cma bitmap. It makes it a blocking function, which complicates its usage from non-blocking contexts. For instance, hugetlbfs code is temporarily dropping the hugetlb_lock spinlock to call cma_release().
This patch makes cma_release non-blocking by postponing the cma bitmap clearance. It's done later from a work context. The first page in the cma allocation is used to store the work struct.
To make sure that subsequent cma_alloc() call will pass, cma_alloc() flushes the corresponding workqueue.
Because CMA allocations and de-allocations are usually not that frequent, a single global workqueue is used.
--- mm/cma.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 49 insertions(+), 2 deletions(-)
mm/cma.c +49 -2
@@ -36,10 +36,19 @@#include "cma.h" +struct cma_clear_bitmap_work { +struct work_struct work; +struct cma *cma; +unsigned long pfn; +unsigned int count; +}; +struct cma cma_areas[MAX_CMA_AREAS ];unsigned cma_area_count;static DEFINE_MUTEX (cma_mutex); +struct workqueue_struct *cma_release_wq; +phys_addr_t cma_get_base (const struct cma *cma) {return PFN_PHYS (cma->base_pfn);
@@ -148,6 +157,10 @@ static int __init cma_init_reserved_areas(void)for (i =0 ; i < cma_area_count; i++)cma_activate_area (&cma_areas[i]); + cma_release_wq =create_workqueue ("cma_release" ); +if (!cma_release_wq) +return -ENOMEM ; +return 0 ; }core_initcall (cma_init_reserved_areas);
@@ -437,6 +450,13 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align,return NULL ;for (;;) { +/* +* CMA bitmaps are cleared asynchronously from works, +* scheduled by cma_release(). To make sure the allocation +* will success, cma release workqueue is flushed here. +*/ +flush_workqueue (cma_release_wq); +mutex_lock (&cma->lock); bitmap_no = bitmap_find_next_zero_area_off(cma->bitmap, bitmap_maxno, start, bitmap_count, mask,
@@ -495,6 +515,17 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align,return page; } +static void cma_clear_bitmap_fn (struct work_struct *work) +{ +struct cma_clear_bitmap_work *w; + + w = container_of(work, struct cma_clear_bitmap_work, work); + +cma_clear_bitmap (w->cma, w->pfn, w->count); + +__free_page (pfn_to_page (w->pfn)); +} +/** * cma_release() - release allocated pages * @cma: Contiguous memory region for which the allocation is performed.
@@ -507,6 +538,7 @@ struct page *cma_alloc(struct cma *cma, size_t count, unsigned int align,*/ bool cma_release (struct cma *cma,const struct page *pages,unsigned int count) { +struct cma_clear_bitmap_work *work;unsigned long pfn;if (!cma || !pages)
@@ -521,8 +553,23 @@ bool cma_release(struct cma *cma, const struct page *pages, unsigned int count)VM_BUG_ON (pfn + count > cma->base_pfn + cma->count); -free_contig_range (pfn, count); -cma_clear_bitmap (cma, pfn, count); +/* +* To make cma_release() non-blocking, cma bitmap is cleared from +* a work context (see cma_clear_bitmap_fn()). The first page +* in the cma allocation is used to store the work structure, +* so it's released after the cma bitmap clearance. Other pages +* are released immediately as previously. +*/ +if (count >1 ) +free_contig_range (pfn +1 , count -1 ); + + work = (struct cma_clear_bitmap_work *)page_to_virt (pages); +INIT_WORK (&work->work, cma_clear_bitmap_fn); + work->cma = cma; + work->pfn = pfn; + work->count = count; +queue_work (cma_release_wq, &work->work); +trace_cma_release (pfn, pages, count);return true;
Signature · 1 line
2.26.2
RFC PATCH 2/2 mm: hugetlb: don't drop hugetlb_lock around cma_release() call
In reply to Roman Gushchincma_release() now is a non-blocking function, so there is no more need to drop hugetlb_lock to call it.
--- mm/hugetlb.c | 6 ------ 1 file changed, 6 deletions(-)
mm/hugetlb.c +0 -6
@@ -1312,14 +1312,8 @@ static void update_and_free_page(struct hstate *h, struct page *page)set_compound_page_dtor (page,NULL_COMPOUND_DTOR );set_page_refcounted (page);if (hstate_is_gigantic (h)) { -/* -* Temporarily drop the hugetlb_lock, because -* we might block in free_gigantic_page(). -*/ -spin_unlock (&hugetlb_lock);destroy_compound_gigantic_page (page,huge_page_order (h));free_gigantic_page (page,huge_page_order (h)); -spin_lock (&hugetlb_lock); }else {__free_pages (page,huge_page_order (h)); }
Signature · 1 line
2.26.2
Quoted 11 lines · Roman Gushchin
This small patchset makes
cma_release()non-blocking and simplifies the code in hugetlbfs, where previously we had to temporarily drophugetlb_lockaround thecma_release()call.It should help Zi Yan on his work on 1 GB THPs: splitting a gigantic THP under a memory pressure requires a
cma_release()call. If it's a blocking function, it complicates the already complicated code. Because there are at least two use cases like this (hugetlbfs is another example), I believe it's just better to makecma_release()non-blocking.
Quoted 2 lines
It also makes it more consistent with other memory releasing functions in the kernel: most of them are non-blocking.
Thanks for looking into this Roman.
I may be missing something, but why does cma_release have to be blocking today? Certainly, it takes the bitmap in cma_clear_bitmap and could block. However, I do not see why cma->lock has to be a mutex. I may be missing something, but I do not see any code protected by the mutex doing anything that could sleep?
Could we simply change that mutex to a spinlock?
Signature · 1 line
Mike Kravetz
Quoted 12 lines · Mike Kravetz
Quoted 11 lines · Roman Gushchin
This small patchset makes
cma_release()non-blocking and simplifies the code in hugetlbfs, where previously we had to temporarily drophugetlb_lockaround thecma_release()call.It should help Zi Yan on his work on 1 GB THPs: splitting a gigantic THP under a memory pressure requires a
cma_release()call. If it's a blocking function, it complicates the already complicated code. Because there are at least two use cases like this (hugetlbfs is another example), I believe it's just better to makecma_release()non-blocking.
Quoted 4 lines
Quoted 2 lines
It also makes it more consistent with other memory releasing functions in the kernel: most of them are non-blocking.
Thanks for looking into this Roman.
Hi Mike,
Quoted 8 lines
I may be missing something, but why does
cma_releasehave to be blocking today? Certainly, it takes the bitmap incma_clear_bitmapand could block. However, I do not see whycma->lockhas to be a mutex. I may be missing something, but I do not see any code protected by the mutex doing anything that could sleep?Could we simply change that mutex to a spinlock?
I actually have suggested it few months ago, but the idea was opposed by Joonsoo: https://lkml.org/lkml/2020/4/3/12 .
The time of a bitmap operation is definitely not an issue in my context, but I can't speak for something like an embedded/rt case.
Thanks!
Quoted 27 lines · Roman Gushchin
Quoted 12 lines · Mike Kravetz
Quoted 11 lines · Roman Gushchin
This small patchset makes
cma_release()non-blocking and simplifies the code in hugetlbfs, where previously we had to temporarily drophugetlb_lockaround thecma_release()call.It should help Zi Yan on his work on 1 GB THPs: splitting a gigantic THP under a memory pressure requires a
cma_release()call. If it's a blocking function, it complicates the already complicated code. Because there are at least two use cases like this (hugetlbfs is another example), I believe it's just better to makecma_release()non-blocking.Quoted 4 lines
Quoted 2 lines
It also makes it more consistent with other memory releasing functions in the kernel: most of them are non-blocking.
Thanks for looking into this Roman.
Hi Mike,
Quoted 7 lines
I may be missing something, but why does
cma_releasehave to be blocking today? Certainly, it takes the bitmap incma_clear_bitmapand could block. However, I do not see whycma->lockhas to be a mutex. I may be missing something, but I do not see any code protected by the mutex doing anything that could sleep?
Quoted 8 lines
Quoted 1 line
Could we simply change that mutex to a spinlock?
I actually have suggested it few months ago, but the idea was opposed by Joonsoo: https://lkml.org/lkml/2020/4/3/12 .
The time of a bitmap operation is definitely not an issue in my context, but I can't speak for something like an embedded/rt case.
I wonder if it may be time to look into replacing the cma area bitmap with some other data structure? Joonsoo was concerned about the time required to traverse the bitmap for an 8GB area. With new support for allocating 1GB hugetlb pages from cma, I can imagine someone setting up a cma area that is hundreds of GB if not TB in size. It is going take some time to traverse a bitmap describing such a huge area.
Signature · 1 line
Mike Kravetz
Quoted 37 lines · Mike Kravetz
Quoted 27 lines · Roman Gushchin
Quoted 12 lines · Mike Kravetz
Quoted 11 lines · Roman Gushchin
This small patchset makes
cma_release()non-blocking and simplifies the code in hugetlbfs, where previously we had to temporarily drophugetlb_lockaround thecma_release()call.It should help Zi Yan on his work on 1 GB THPs: splitting a gigantic THP under a memory pressure requires a
cma_release()call. If it's a blocking function, it complicates the already complicated code. Because there are at least two use cases like this (hugetlbfs is another example), I believe it's just better to makecma_release()non-blocking.Quoted 4 lines
Quoted 2 lines
It also makes it more consistent with other memory releasing functions in the kernel: most of them are non-blocking.
Thanks for looking into this Roman.
Hi Mike,
Quoted 7 lines
I may be missing something, but why does
cma_releasehave to be blocking today? Certainly, it takes the bitmap incma_clear_bitmapand could block. However, I do not see whycma->lockhas to be a mutex. I may be missing something, but I do not see any code protected by the mutex doing anything that could sleep?Quoted 8 lines
Quoted 1 line
Could we simply change that mutex to a spinlock?
I actually have suggested it few months ago, but the idea was opposed by Joonsoo: https://lkml.org/lkml/2020/4/3/12 .
The time of a bitmap operation is definitely not an issue in my context, but I can't speak for something like an embedded/rt case.
Quoted 6 lines
I wonder if it may be time to look into replacing the cma area bitmap with some other data structure? Joonsoo was concerned about the time required to traverse the bitmap for an 8GB area. With new support for allocating 1GB hugetlb pages from cma, I can imagine someone setting up a cma area that is hundreds of GB if not TB in size. It is going take some time to traverse a bitmap describing such a huge area.
If the cma area is used exclusively for 1 GB allocations, the bitmap can have only 1 bit per GB, so it shouldn't be a big problem.
Long-term I have some hopes to be able to allocate 1 GB pages without a need to reserve a cma area: we can try to group pages based on their mobility on a 1 GB scale, so that all non-movable pages will reside in few 1 GB blocks. I'm looking into that direction, but don't have any results yet.
If this idea fails and we'll end up allocating a large cma area unconditionally and shrink it on demand (I think Rik suggested something like this), replacing the bitmap with something else sounds like a good idea to me.
As now, I want to unblock Zi Yan on his work on 1 GB THPs, so maybe we can go with introducing cma_release_nowait(), as I suggested in the other e-mail in this thread? Do you have any objections?
Thanks!
Quoted 3 lines · Roman Gushchin
As now, I want to unblock Zi Yan on his work on 1 GB THPs, so maybe we can go with introducing
cma_release_nowait(), as I suggested in the other e-mail in this thread? Do you have any objections?
No objections from me. And, I like that it could be used by the hugetlb code to make it simpler.
Signature · 1 line
Mike Kravetz
Quoted 15 lines · Roman Gushchin
This small patchset makes
cma_release()non-blocking and simplifies the code in hugetlbfs, where previously we had to temporarily drophugetlb_lockaround thecma_release()call.It should help Zi Yan on his work on 1 GB THPs: splitting a gigantic THP under a memory pressure requires a
cma_release()call. If it's a blocking function, it complicates the already complicated code. Because there are at least two use cases like this (hugetlbfs is another example), I believe it's just better to makecma_release()non-blocking.It also makes it more consistent with other memory releasing functions in the kernel: most of them are non-blocking.
Quoted 8 lines
Roman Gushchin (2):
mm: cma: makecma_release()non-blocking
mm: hugetlb: don't drophugetlb_lockaroundcma_release()callmm/cma.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++-- mm/hugetlb.c | 6 ------ 2 files changed, 49 insertions(+), 8 deletions(-)
I don't think this patch is a good idea.It transfers part or even all of the time of cma_release to cma_alloc, which is more concerned by performance indicators. On Android phones, CPU resource competition is intense in many scenarios, As a result, kernel threads and workers can be scheduled only after some ticks or more. In this case, the performance of cma_alloc will deteriorate significantly, which is not good news for many services on Android.
Quoted 23 lines · Xiaqing (A)
Quoted 19 lines · Roman Gushchin
This small patchset makes
cma_release()non-blocking and simplifies the code in hugetlbfs, where previously we had to temporarily drophugetlb_lockaround thecma_release()call.It should help Zi Yan on his work on 1 GB THPs: splitting a gigantic THP under a memory pressure requires a
cma_release()call. If it's a blocking function, it complicates the already complicated code. Because there are at least two use cases like this (hugetlbfs is another example), I believe it's just better to makecma_release()non-blocking.It also makes it more consistent with other memory releasing functions in the kernel: most of them are non-blocking.
Roman Gushchin (2):
mm: cma: makecma_release()non-blocking
mm: hugetlb: don't drophugetlb_lockaroundcma_release()call
Quoted 6 lines
Quoted 4 lines
mm/cma.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++-- mm/hugetlb.c | 6 ------ 2 files changed, 49 insertions(+), 8 deletions(-)I don't think this patch is a good idea.It transfers part or even all of the time of
cma_releasetocma_alloc, which is more concerned by performance indicators.
I'm not quite sure: if cma_alloc() is racing with cma_release(), cma_alloc() will wait for the cma_lock mutex anyway. So we don't really transfer anything to cma_alloc().
Quoted 4 lines
On Android phones, CPU resource competition is intense in many scenarios, As a result, kernel threads and workers can be scheduled only after some ticks or more. In this case, the performance of
cma_allocwill deteriorate significantly, which is not good news for many services on Android.
Ok, I agree, if the cpu is heavily loaded, it might affect the total execution time.
If we aren't going into the mutex->spinlock conversion direction (as Mike suggested), we can address the performance concerns by introducing a cma_release_nowait() function, so that the default cma_release() would work in the old way. cma_release_nowait() can set an atomic flag on a cma area, which will cause following cma_alloc()'s to flush the release queue. In this case there will be no performance penalty unless somebody is using cma_release_nowait().
Will it work for you?
Thank you!
Quoted 38 lines · Roman Gushchin
Quoted 22 lines · Xiaqing (A)
Quoted 19 lines · Roman Gushchin
This small patchset makes
cma_release()non-blocking and simplifies the code in hugetlbfs, where previously we had to temporarily drophugetlb_lockaround thecma_release()call.It should help Zi Yan on his work on 1 GB THPs: splitting a gigantic THP under a memory pressure requires a
cma_release()call. If it's a blocking function, it complicates the already complicated code. Because there are at least two use cases like this (hugetlbfs is another example), I believe it's just better to makecma_release()non-blocking.It also makes it more consistent with other memory releasing functions in the kernel: most of them are non-blocking.
Roman Gushchin (2):
mm: cma: makecma_release()non-blocking
mm: hugetlb: don't drophugetlb_lockaroundcma_release()callQuoted 6 lines
Quoted 4 lines
mm/cma.c | 51 +++++++++++++++++++++++++++++++++++++++++++++++++-- mm/hugetlb.c | 6 ------ 2 files changed, 49 insertions(+), 8 deletions(-)I don't think this patch is a good idea.It transfers part or even all of the time of
cma_releasetocma_alloc, which is more concerned by performance indicators.I'm not quite sure: if
cma_alloc()is racing withcma_release(),cma_alloc()will wait for thecma_lockmutex anyway. So we don't really transfer anything tocma_alloc().Quoted 4 lines
On Android phones, CPU resource competition is intense in many scenarios, As a result, kernel threads and workers can be scheduled only after some ticks or more. In this case, the performance of
cma_allocwill deteriorate significantly, which is not good news for many services on Android.Ok, I agree, if the cpu is heavily loaded, it might affect the total execution time.
Quoted 7 lines
If we aren't going into the
mutex->spinlockconversion direction (as Mike suggested), we can address the performance concerns by introducing acma_release_nowait()function, so that the defaultcma_release()would work in the old way.cma_release_nowait()can set an atomic flag on a cma area, which will cause following cma_alloc()'s to flush the release queue. In this case there will be no performance penalty unless somebody is usingcma_release_nowait().
Will it work for you?
That looks good to me.
Thanks!
Quoted 4 lines
Thank you!