mm/hugetlb: alloc/free gigantic folios
Subsystems (5)
v2 cover mm/hugetlb: alloc/free gigantic folios
Use __GFP_COMP for gigantic folios can greatly reduce not only the amount of code but also the allocation and free time.
Approximate LOC to mm/hugetlb.c: +60, -240
Allocate and free 500 1GB hugeTLB memory without HVO by:
time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
Before After Alloc ~13s ~10s Free ~15s <1s
The above magnitude generally holds for multiple x86 and arm64 CPU models.
Perf profile before:
Alloc
- 99.99% alloc_pool_huge_folio
- __alloc_fresh_hugetlb_folio
- 83.23% alloc_contig_pages_noprof
- 47.46% alloc_contig_range_noprof
- 20.96% isolate_freepages_range
16.10% split_page
- 14.10% start_isolate_page_range
- 12.02% undo_isolate_page_range
Free
- update_and_free_pages_bulk
- 87.71% free_contig_range
- 76.02% free_unref_page
- 41.30% free_unref_page_commit
- 32.58% free_pcppages_bulk
- 24.75% __free_one_page
13.96% _raw_spin_trylock
12.27% __update_and_free_hugetlb_folio
Perf profile after:
Alloc
- 99.99% alloc_pool_huge_folio
alloc_gigantic_folio
- alloc_contig_pages_noprof
- 59.15% alloc_contig_range_noprof
- 20.72% start_isolate_page_range
20.64% prep_new_page
- 17.13% undo_isolate_page_range
Free
- update_and_free_pages_bulk
- __folio_put
- __free_pages_ok
7.46% free_tail_page_prepare
- 1.97% free_one_page
1.86% __free_one_page
Yu Zhao (3):
mm/contig_alloc: support __GFP_COMP
mm/cma: add cma_{alloc,free}_folio()
mm/hugetlb: use __GFP_COMP for gigantic folios
include/linux/cma.h | 16 +++ include/linux/gfp.h | 23 ++++ include/linux/hugetlb.h | 9 +- mm/cma.c | 55 ++++++-- mm/compaction.c | 41 +----- mm/hugetlb.c | 293 ++++++++-------------------------------- mm/page_alloc.c | 111 ++++++++++----- 7 files changed, 226 insertions(+), 322 deletions(-)
Signature · 1 line
2.46.0.76.ge559c4bf1a-goog
PATCH v2 1/3 mm/contig_alloc: support __GFP_COMP
Support __GFP_COMP in alloc_contig_range(). When the flag is set, upon success the function returns a large folio prepared by prep_new_page(), rather than a range of order-0 pages prepared by split_free_pages() (which is renamed from split_map_pages()).
alloc_contig_range() can be used to allocate folios larger than MAX_PAGE_ORDER, e.g., gigantic hugeTLB folios. So on the free path, free_one_page() needs to handle that by split_large_buddy().
--- include/linux/gfp.h | 23 +++++++++ mm/compaction.c | 41 ++-------------- mm/page_alloc.c | 111 +++++++++++++++++++++++++++++++------------- 3 files changed, 108 insertions(+), 67 deletions(-)
include/linux/gfp.h +23 -0
@@ -446,4 +446,27 @@ extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_#endif void free_contig_range (unsigned long pfn,unsigned long nr_pages); +#ifdef CONFIG_CONTIG_ALLOC +static inline struct folio *folio_alloc_gigantic_noprof (int order,gfp_t gfp, +int nid,nodemask_t *node) +{ +struct page *page; + +if (WARN_ON (!order || !(gfp | __GFP_COMP))) +return NULL ; + + page =alloc_contig_pages_noprof (1 << order, gfp, nid, node); + +return page ?page_folio (page) :NULL ; +} +#else +static inline struct folio *folio_alloc_gigantic_noprof (int order,gfp_t gfp, +int nid,nodemask_t *node) +{ +return NULL ; +} +#endif +/* This should be paired with folio_put() rather than free_contig_range(). */ +#define folio_alloc_gigantic (...) alloc_hooks(folio_alloc_gigantic_noprof(__VA_ARGS__)) +#endif /* __LINUX_GFP_H */
mm/compaction.c +5 -36
79 lines not on this page · raw message
mm/page_alloc.c +80 -31
161 lines not on this page · raw message
Signature · 1 line
2.46.0.76.ge559c4bf1a-goog
Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 21 lines · Yu Zhao
Support
__GFP_COMPinalloc_contig_range(). When the flag is set, upon success the function returns a large folio prepared byprep_new_page(), rather than a range of order-0 pages prepared bysplit_free_pages()(which is renamed fromsplit_map_pages()).
alloc_contig_range()can be used to allocate folios larger thanMAX_PAGE_ORDER, e.g., gigantic hugeTLB folios. So on the free path,free_one_page()needs to handle that bysplit_large_buddy().Signed-off-by Yu Zhao <[email protected]>--- include/linux/gfp.h | 23 +++++++++ mm/compaction.c | 41 ++-------------- mm/page_alloc.c | 111 +++++++++++++++++++++++++++++++------------- 3 files changed, 108 insertions(+), 67 deletions(-)include/linux/gfp.h +0
-0
include/linux/gfp.h extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_ · line 455
Quoted 10 lines
include/linux/gfp.h +7
-0@@ -446,4 +446,27 @@ extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_#endif void free_contig_range (unsigned long pfn,unsigned long nr_pages); +#ifdef CONFIG_CONTIG_ALLOC +static inline struct folio *folio_alloc_gigantic_noprof (int order,gfp_t gfp, +int nid,nodemask_t *node) +{ +struct page *page; + +if (WARN_ON (!order || !(gfp | __GFP_COMP)))
It doesn't seem right, it should be !(gfp & __GFP_COMP).
Best Regards,
Yu
Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 26 lines · Yu Liao
Quoted 24 lines · Yu Zhao
Support
__GFP_COMPinalloc_contig_range(). When the flag is set, upon success the function returns a large folio prepared byprep_new_page(), rather than a range of order-0 pages prepared bysplit_free_pages()(which is renamed fromsplit_map_pages()).
alloc_contig_range()can be used to allocate folios larger thanMAX_PAGE_ORDER, e.g., gigantic hugeTLB folios. So on the free path,free_one_page()needs to handle that bysplit_large_buddy().Signed-off-by Yu Zhao <[email protected]>--- include/linux/gfp.h | 23 +++++++++ mm/compaction.c | 41 ++-------------- mm/page_alloc.c | 111 +++++++++++++++++++++++++++++++------------- 3 files changed, 108 insertions(+), 67 deletions(-)include/linux/gfp.h +0
-0@@ -446,4 +446,27 @@ extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_#endif void free_contig_range (unsigned long pfn,unsigned long nr_pages);
Quoted 9 lines
include/linux/gfp.h extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_ · line 455
Quoted 7 lines
include/linux/gfp.h +7
-0+#ifdef CONFIG_CONTIG_ALLOC +static inline struct folio *folio_alloc_gigantic_noprof (int order,gfp_t gfp, +int nid,nodemask_t *node) +{ +struct page *page; + +if (WARN_ON (!order || !(gfp | __GFP_COMP)))It doesn't seem right, it should be !(gfp &
__GFP_COMP).
Thanks. I've asked Andrew to patch this up (and another place in mm/cma.c).
Andrew Morton
maintainerRe: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
In reply to Yu LiaoQuoted 9 lines · On Thu, 22 Aug 2024 16:21:27 +0800 Yu Liao <[email protected]> wrote:
Review
Quoted 7 lines
(quoted diff) +7
-0+#ifdef CONFIG_CONTIG_ALLOC +static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, + int nid, nodemask_t *node) +{ + struct page *page; + + if (WARN_ON(!order || !(gfp | __GFP_COMP)))It doesn't seem right, it should be !(gfp &
__GFP_COMP).
Yup. Although I'm wondering why we're checking __GFP_COMP at all? It's a gigantic page - we know it's compound, so why require that the caller tell us something we already know?
include/linux/gfp.h renamed from include/linux/gfp.h~mm-contig_alloc-support-__gfp_comp-fix +1 -1
@@ -452,7 +452,7 @@ static inline struct folio *folio_alloc_ {struct page *page; -if (WARN_ON (!order || !(gfp | __GFP_COMP))) +if (WARN_ON (!order || !(gfp & __GFP_COMP)))return NULL ; page =alloc_contig_pages_noprof (1 << order, gfp, nid, node);
_
Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
In reply to Yu ZhaoQuoted 21 lines · Yu Zhao
Support
__GFP_COMPinalloc_contig_range(). When the flag is set, upon success the function returns a large folio prepared byprep_new_page(), rather than a range of order-0 pages prepared bysplit_free_pages()(which is renamed fromsplit_map_pages()).
alloc_contig_range()can be used to allocate folios larger thanMAX_PAGE_ORDER, e.g., gigantic hugeTLB folios. So on the free path,free_one_page()needs to handle that bysplit_large_buddy().Signed-off-by Yu Zhao <[email protected]>--- include/linux/gfp.h | 23 +++++++++ mm/compaction.c | 41 ++-------------- mm/page_alloc.c | 111 +++++++++++++++++++++++++++++++------------- 3 files changed, 108 insertions(+), 67 deletions(-)include/linux/gfp.h +0
-0
include/linux/gfp.h extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_ · line 455
Quoted 10 lines
include/linux/gfp.h +7
-0@@ -446,4 +446,27 @@ extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_#endif void free_contig_range (unsigned long pfn,unsigned long nr_pages); +#ifdef CONFIG_CONTIG_ALLOC +static inline struct folio *folio_alloc_gigantic_noprof (int order,gfp_t gfp, +int nid,nodemask_t *node) +{ +struct page *page; + +if (WARN_ON (!order || !(gfp | __GFP_COMP)))
Andrew, could you patch up the line above? This is what it's supposed to check:
include/linux/gfp.h +1 -1
@@ -452,7 +452,7 @@ static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, {struct page *page; -if (WARN_ON (!order || !(gfp | __GFP_COMP))) +if (WARN_ON (!order || !(gfp & __GFP_COMP)))return NULL ; page =alloc_contig_pages_noprof (1 << order, gfp, nid, node);
Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 8 lines · Yu Zhao
Andrew, could you patch up the line above? This is what it's supposed to check:
include/linux/gfp.h +0
-0
include/linux/gfp.h static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, · line 456
Quoted 6 lines
include/linux/gfp.h +1
-1@@ -452,7 +452,7 @@ static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, {struct page *page; -if (WARN_ON (!order || !(gfp | __GFP_COMP))) +if (WARN_ON (!order || !(gfp & __GFP_COMP)))return NULL ;
I don't think we should do this at all. Just this should be enough:
gfp |= __GFP_COMP;
same as folio_alloc() (or now folio_alloc_noprof()).
Do we really caree if somebody tries to allocate a gigantic page with an order of 0? It's weird, but would work, so I don't see the need for the warning.
Quoted 2 lines
include/linux/gfp.h +0
-0page =alloc_contig_pages_noprof (1 << order, gfp, nid, node);
Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 18 lines · Matthew Wilcox
Quoted 8 lines · Yu Zhao
Andrew, could you patch up the line above? This is what it's supposed to check:
include/linux/gfp.h +0
-0include/linux/gfp.h static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, · line 456
Quoted 6 lines
include/linux/gfp.h +1
-1@@ -452,7 +452,7 @@ static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, {struct page *page; -if (WARN_ON (!order || !(gfp | __GFP_COMP))) +if (WARN_ON (!order || !(gfp & __GFP_COMP)))return NULL ;I don't think we should do this at all. Just this should be enough:
Quoted 5 lines
gfp |= __GFP_COMP;same as
folio_alloc()(or nowfolio_alloc_noprof()).
Do we really caree if somebody tries to allocate a gigantic page with an order of 0?
If this ever happens, I'd bet it's a bug.
Quoted 2 lines
It's weird, but would work, so I don't see the need for the warning.
So the warning could catch that, but if we think it's verbose, then please fold the following in:
include/linux/gfp.h +1 -4
@@ -452,10 +452,7 @@ static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, {struct page *page; -if (WARN_ON (!order || !(gfp & __GFP_COMP))) -return NULL ; - - page =alloc_contig_pages_noprof (1 << order, gfp, nid, node); + page =alloc_contig_pages_noprof (1 << order, gfp | __GFP_COMP, nid, node);return page ?page_folio (page) :NULL ; }
Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
In reply to Yu ZhaoQuoted 11 lines
mm/page_alloc.c +11
-0+/* Split a multi-block free page into its individual pageblocks. */ +static void split_large_buddy (struct zone *zone,struct page *page, +unsigned long pfn,int order,fpi_t fpi) +{ +unsigned long end = pfn + (1 << order); + +VM_WARN_ON_ONCE (!IS_ALIGNED (pfn,1 << order)); +/* Caller removed page from freelist, buddy info cleared! */ +VM_WARN_ON_ONCE (PageBuddy (page)); + +if (order > pageblock_order)
mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220
Quoted 10 lines
mm/page_alloc.c +10
-0+ order = pageblock_order; + +while (pfn != end) { +int mt =get_pfnblock_migratetype (page, pfn); + +__free_one_page (page, pfn, zone, order, mt, fpi); + pfn +=1 << order; + page =pfn_to_page (pfn); + } +}
Hi,
stumbling over this while digging through the code ....
mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209
Quoted 11 lines
mm/page_alloc.c +1
-3+static void free_one_page (struct zone *zone,struct page *page,unsigned long pfn,unsigned int order,fpi_t fpi_flags) {unsigned long flags; -int migratetype;spin_lock_irqsave (&zone->lock, flags); - migratetype =get_pfnblock_migratetype (page, pfn); -__free_one_page (page, pfn, zone, order, migratetype, fpi_flags);
This change is rather undesired:
via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.
What your new code will do is split this perfectly reasonable MAX_PAGE_ORDER chunk via split_large_buddy() into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.
Is there a way to avoid this and just process the whole MAX_PAGE_ORDER chunk like we used to?
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerRe: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11
-0+/* Split a multi-block free page into its individual pageblocks. */ +static void split_large_buddy (struct zone *zone,struct page *page, +unsigned long pfn,int order,fpi_t fpi) +{ +unsigned long end = pfn + (1 << order); + +VM_WARN_ON_ONCE (!IS_ALIGNED (pfn,1 << order)); +/* Caller removed page from freelist, buddy info cleared! */ +VM_WARN_ON_ONCE (PageBuddy (page)); + +if (order > pageblock_order)mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220
Quoted 10 lines
mm/page_alloc.c +10
-0+ order = pageblock_order; + +while (pfn != end) { +int mt =get_pfnblock_migratetype (page, pfn); + +__free_one_page (page, pfn, zone, order, mt, fpi); + pfn +=1 << order; + page =pfn_to_page (pfn); + } +}Hi,
stumbling over this while digging through the code ....
mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209
Quoted 10 lines
mm/page_alloc.c +1
-3+static void free_one_page (struct zone *zone,struct page *page,unsigned long pfn,unsigned int order,fpi_t fpi_flags) {unsigned long flags; -int migratetype;spin_lock_irqsave (&zone->lock, flags); - migratetype =get_pfnblock_migratetype (page, pfn); -__free_one_page (page, pfn, zone, order, migratetype, fpi_flags);This change is rather undesired:
Quoted 1 line
via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.
Do you have a concrete example? PMD THP on x86_64 is pageblock_order. We do not have PMD level mTHP yet. Any other possible source?
Quoted 4 lines
What your new code will do is split this perfectly reasonable
MAX_PAGE_ORDERchunk viasplit_large_buddy()into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.Is there a way to avoid this and just process the whole
MAX_PAGE_ORDERchunk like we used to?
Probably split_large_buddy() can check the migratetypes of the to-be-freed page, if order > pageblock_order. If all migratetypes are the same, the page can be freed at MAX_PAGE_ORDER, otherwise pageblock_order.
Best Regards,
Yan, Zi
Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 41 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11
-0+/* Split a multi-block free page into its individual pageblocks. */ +static void split_large_buddy (struct zone *zone,struct page *page, +unsigned long pfn,int order,fpi_t fpi) +{ +unsigned long end = pfn + (1 << order); + +VM_WARN_ON_ONCE (!IS_ALIGNED (pfn,1 << order)); +/* Caller removed page from freelist, buddy info cleared! */ +VM_WARN_ON_ONCE (PageBuddy (page)); + +if (order > pageblock_order)mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220
Quoted 10 lines
mm/page_alloc.c +10
-0+ order = pageblock_order; + +while (pfn != end) { +int mt =get_pfnblock_migratetype (page, pfn); + +__free_one_page (page, pfn, zone, order, mt, fpi); + pfn +=1 << order; + page =pfn_to_page (pfn); + } +}Hi,
stumbling over this while digging through the code ....
mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209
Quoted 10 lines
mm/page_alloc.c +1
-3+static void free_one_page (struct zone *zone,struct page *page,unsigned long pfn,unsigned int order,fpi_t fpi_flags) {unsigned long flags; -int migratetype;spin_lock_irqsave (&zone->lock, flags); - migratetype =get_pfnblock_migratetype (page, pfn); -__free_one_page (page, pfn, zone, order, migratetype, fpi_flags);This change is rather undesired:
Quoted 4 lines
Quoted 1 line
via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.
Do you have a concrete example? PMD THP on
x86_64ispageblock_order. We do not have PMD level mTHP yet. Any other possible source?
Memory init during boot. See deferred_free_pages() and __free_pages_memory()->memblock_free_pages().
So this is used for exposing most memory during boot to the buddy in MAX_PAGE_ORDER granularity.
The other is memory hotplug via generic_online_pages().
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
In reply to Zi YanQuoted 47 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11
-0+/* Split a multi-block free page into its individual pageblocks. */ +static void split_large_buddy (struct zone *zone,struct page *page, +unsigned long pfn,int order,fpi_t fpi) +{ +unsigned long end = pfn + (1 << order); + +VM_WARN_ON_ONCE (!IS_ALIGNED (pfn,1 << order)); +/* Caller removed page from freelist, buddy info cleared! */ +VM_WARN_ON_ONCE (PageBuddy (page)); + +if (order > pageblock_order)mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220
Quoted 10 lines
mm/page_alloc.c +10
-0+ order = pageblock_order; + +while (pfn != end) { +int mt =get_pfnblock_migratetype (page, pfn); + +__free_one_page (page, pfn, zone, order, mt, fpi); + pfn +=1 << order; + page =pfn_to_page (pfn); + } +}Hi,
stumbling over this while digging through the code ....
mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209
Quoted 10 lines
mm/page_alloc.c +1
-3+static void free_one_page (struct zone *zone,struct page *page,unsigned long pfn,unsigned int order,fpi_t fpi_flags) {unsigned long flags; -int migratetype;spin_lock_irqsave (&zone->lock, flags); - migratetype =get_pfnblock_migratetype (page, pfn); -__free_one_page (page, pfn, zone, order, migratetype, fpi_flags);This change is rather undesired:
Quoted 1 line
via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.
Do you have a concrete example? PMD THP on
x86_64ispageblock_order. We do not have PMD level mTHP yet. Any other possible source?Quoted 1 line
Quoted 7 lines
Quoted 3 lines
What your new code will do is split this perfectly reasonable
MAX_PAGE_ORDERchunk viasplit_large_buddy()into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.Is there a way to avoid this and just process the whole
MAX_PAGE_ORDERchunk like we used to?Probably
split_large_buddy()can check the migratetypes of the to-be-freed page, if order >pageblock_order. If all migratetypes are the same, the page can be freed atMAX_PAGE_ORDER, otherwisepageblock_order.
Thinking about this: why do we care about the migratetype?
We only have to fallback to pageblocks if any pageblock is "MIGRATE_ISOLATE" (and maybe MIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerRe: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11
-0+/* Split a multi-block free page into its individual pageblocks. */ +static void split_large_buddy (struct zone *zone,struct page *page, +unsigned long pfn,int order,fpi_t fpi) +{ +unsigned long end = pfn + (1 << order); + +VM_WARN_ON_ONCE (!IS_ALIGNED (pfn,1 << order)); +/* Caller removed page from freelist, buddy info cleared! */ +VM_WARN_ON_ONCE (PageBuddy (page)); + +if (order > pageblock_order)mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220
Quoted 10 lines
mm/page_alloc.c +10
-0+ order = pageblock_order; + +while (pfn != end) { +int mt =get_pfnblock_migratetype (page, pfn); + +__free_one_page (page, pfn, zone, order, mt, fpi); + pfn +=1 << order; + page =pfn_to_page (pfn); + } +}Hi,
stumbling over this while digging through the code ....
mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209
Quoted 10 lines
mm/page_alloc.c +1
-3+static void free_one_page (struct zone *zone,struct page *page,unsigned long pfn,unsigned int order,fpi_t fpi_flags) {unsigned long flags; -int migratetype;spin_lock_irqsave (&zone->lock, flags); - migratetype =get_pfnblock_migratetype (page, pfn); -__free_one_page (page, pfn, zone, order, migratetype, fpi_flags);This change is rather undesired:
Quoted 1 line
via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.
Do you have a concrete example? PMD THP on
x86_64ispageblock_order. We do not have PMD level mTHP yet. Any other possible source?Quoted 4 lines
What your new code will do is split this perfectly reasonable
MAX_PAGE_ORDERchunk viasplit_large_buddy()into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.Is there a way to avoid this and just process the whole
MAX_PAGE_ORDERchunk like we used to?
Quoted 7 lines
Quoted 3 lines
Probably
split_large_buddy()can check the migratetypes of the to-be-freed page, if order >pageblock_order. If all migratetypes are the same, the page can be freed atMAX_PAGE_ORDER, otherwisepageblock_order.Thinking about this: why do we care about the migratetype?
We only have to fallback to pageblocks if any pageblock is "
MIGRATE_ISOLATE" (and maybeMIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)
There are VM_WARN_ONCEs around *_free_list() operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.
An implicit reason is that __free_one_page() does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed via free_one_page(), where split_large_buddy() is used to work around the limitation.
For the two memory init cases you mentioned in the other email, maybe a new fpi flag to make free_one_page() use __free_one_page() for them, since migratetypes should be the same across MAX_PAGE_ORDER range there, right?
Best Regards,
Yan, Zi
Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 62 lines · Zi Yan
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11
-0+/* Split a multi-block free page into its individual pageblocks. */ +static void split_large_buddy (struct zone *zone,struct page *page, +unsigned long pfn,int order,fpi_t fpi) +{ +unsigned long end = pfn + (1 << order); + +VM_WARN_ON_ONCE (!IS_ALIGNED (pfn,1 << order)); +/* Caller removed page from freelist, buddy info cleared! */ +VM_WARN_ON_ONCE (PageBuddy (page)); + +if (order > pageblock_order)mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220
Quoted 10 lines
mm/page_alloc.c +10
-0+ order = pageblock_order; + +while (pfn != end) { +int mt =get_pfnblock_migratetype (page, pfn); + +__free_one_page (page, pfn, zone, order, mt, fpi); + pfn +=1 << order; + page =pfn_to_page (pfn); + } +}Hi,
stumbling over this while digging through the code ....
mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209
Quoted 10 lines
mm/page_alloc.c +1
-3+static void free_one_page (struct zone *zone,struct page *page,unsigned long pfn,unsigned int order,fpi_t fpi_flags) {unsigned long flags; -int migratetype;spin_lock_irqsave (&zone->lock, flags); - migratetype =get_pfnblock_migratetype (page, pfn); -__free_one_page (page, pfn, zone, order, migratetype, fpi_flags);This change is rather undesired:
Quoted 1 line
via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.
Do you have a concrete example? PMD THP on
x86_64ispageblock_order. We do not have PMD level mTHP yet. Any other possible source?Quoted 4 lines
What your new code will do is split this perfectly reasonable
MAX_PAGE_ORDERchunk viasplit_large_buddy()into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.Is there a way to avoid this and just process the whole
MAX_PAGE_ORDERchunk like we used to?Quoted 7 lines
Quoted 3 lines
Probably
split_large_buddy()can check the migratetypes of the to-be-freed page, if order >pageblock_order. If all migratetypes are the same, the page can be freed atMAX_PAGE_ORDER, otherwisepageblock_order.Thinking about this: why do we care about the migratetype?
We only have to fallback to pageblocks if any pageblock is "
MIGRATE_ISOLATE" (and maybeMIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)
Quoted 8 lines
There are
VM_WARN_ONCEsaround*_free_list()operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.An implicit reason is that
__free_one_page()does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed viafree_one_page(), wheresplit_large_buddy()is used to work around the limitation.
Yes, I saw that change. But that can be easily identified cased by unlikely(order > MAX_PAGE_ORDER) to handle that rare case in a special way.
Quoted 3 lines
Quoted 1 line
For the two memory init cases you mentioned in the other email, maybe a new
fpi flag to make
free_one_page()use__free_one_page()for them, since migratetypes should be the same acrossMAX_PAGE_ORDERrange there, right?
In the context of alloc_frozen_range()/free_frozen_range() I want to free MAX_PAGE_ORDER chunks when possible, and not have some odd logic in the freeing path undo some of that effort.
In the common case, the pageblocks will all have the same time when freeing a MAX_PAGE_ORDER page, so we should identify the odd case and only special-case that.
Signature · 3 lines
Cheers, David / dhildenb
Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 75 lines · David Hildenbrand
Quoted 62 lines · Zi Yan
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11
-0+/* Split a multi-block free page into its individual pageblocks. */ +static void split_large_buddy (struct zone *zone,struct page *page, +unsigned long pfn,int order,fpi_t fpi) +{ +unsigned long end = pfn + (1 << order); + +VM_WARN_ON_ONCE (!IS_ALIGNED (pfn,1 << order)); +/* Caller removed page from freelist, buddy info cleared! */ +VM_WARN_ON_ONCE (PageBuddy (page)); + +if (order > pageblock_order)mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220
Quoted 10 lines
mm/page_alloc.c +10
-0+ order = pageblock_order; + +while (pfn != end) { +int mt =get_pfnblock_migratetype (page, pfn); + +__free_one_page (page, pfn, zone, order, mt, fpi); + pfn +=1 << order; + page =pfn_to_page (pfn); + } +}Hi,
stumbling over this while digging through the code ....
mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209
Quoted 10 lines
mm/page_alloc.c +1
-3+static void free_one_page (struct zone *zone,struct page *page,unsigned long pfn,unsigned int order,fpi_t fpi_flags) {unsigned long flags; -int migratetype;spin_lock_irqsave (&zone->lock, flags); - migratetype =get_pfnblock_migratetype (page, pfn); -__free_one_page (page, pfn, zone, order, migratetype, fpi_flags);This change is rather undesired:
Quoted 1 line
via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.
Do you have a concrete example? PMD THP on
x86_64ispageblock_order. We do not have PMD level mTHP yet. Any other possible source?Quoted 4 lines
What your new code will do is split this perfectly reasonable
MAX_PAGE_ORDERchunk viasplit_large_buddy()into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.Is there a way to avoid this and just process the whole
MAX_PAGE_ORDERchunk like we used to?Quoted 7 lines
Quoted 3 lines
Probably
split_large_buddy()can check the migratetypes of the to-be-freed page, if order >pageblock_order. If all migratetypes are the same, the page can be freed atMAX_PAGE_ORDER, otherwisepageblock_order.Thinking about this: why do we care about the migratetype?
We only have to fallback to pageblocks if any pageblock is "
MIGRATE_ISOLATE" (and maybeMIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)Quoted 8 lines
There are
VM_WARN_ONCEsaround*_free_list()operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.An implicit reason is that
__free_one_page()does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed viafree_one_page(), wheresplit_large_buddy()is used to work around the limitation.Yes, I saw that change. But that can be easily identified cased by
unlikely(order > MAX_PAGE_ORDER)to handle that rare case in a special way.
Quoted 8 lines
Quoted 1 line
Quoted 1 line
For the two memory init cases you mentioned in the other email,
maybe a new
Quoted 2 lines
fpi flag to make
free_one_page()use__free_one_page()for them, since migratetypes should be the same acrossMAX_PAGE_ORDERrange there, right?In the context of alloc_frozen_range()/free_frozen_range() I want to free
MAX_PAGE_ORDERchunks when possible, and not have some odd logic in the freeing path undo some of that effort.
Adding a pointer to that discussion:
https://lkml.kernel.org/r/[email protected]
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerRe: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 81 lines · David Hildenbrand
Quoted 80 lines · David Hildenbrand
Quoted 62 lines · Zi Yan
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
(quoted diff) +11
-0+/* Split a multi-block free page into its individual pageblocks. */ +static void split_large_buddy(struct zone *zone, struct page *page, + unsigned long pfn, int order, fpi_t fpi) +{ + unsigned long end = pfn + (1 << order); + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order)); + /* Caller removed page from freelist, buddy info cleared! */ + VM_WARN_ON_ONCE(PageBuddy(page)); + + if (order > pageblock_order)Review
Quoted 10 lines
(quoted diff) +10
-0+ order = pageblock_order; + + while (pfn != end) { + int mt = get_pfnblock_migratetype(page, pfn); + + __free_one_page(page, pfn, zone, order, mt, fpi); + pfn += 1 << order; + page = pfn_to_page(pfn); + } +}Hi,
stumbling over this while digging through the code ....
Review
Quoted 10 lines
(quoted diff) +1
-3+ static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { unsigned long flags; - int migratetype; spin_lock_irqsave(&zone->lock, flags); - migratetype = get_pfnblock_migratetype(page, pfn); - __free_one_page(page, pfn, zone, order, migratetype, fpi_flags);This change is rather undesired:
Quoted 1 line
via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.
Do you have a concrete example? PMD THP on
x86_64ispageblock_order. We do not have PMD level mTHP yet. Any other possible source?Quoted 4 lines
What your new code will do is split this perfectly reasonable
MAX_PAGE_ORDERchunk viasplit_large_buddy()into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.Is there a way to avoid this and just process the whole
MAX_PAGE_ORDERchunk like we used to?Quoted 7 lines
Quoted 3 lines
Probably
split_large_buddy()can check the migratetypes of the to-be-freed page, if order >pageblock_order. If all migratetypes are the same, the page can be freed atMAX_PAGE_ORDER, otherwisepageblock_order.Thinking about this: why do we care about the migratetype?
We only have to fallback to pageblocks if any pageblock is "
MIGRATE_ISOLATE" (and maybeMIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)Quoted 8 lines
There are
VM_WARN_ONCEsaround*_free_list()operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.An implicit reason is that
__free_one_page()does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed viafree_one_page(), wheresplit_large_buddy()is used to work around the limitation.Yes, I saw that change. But that can be easily identified cased by
unlikely(order > MAX_PAGE_ORDER)to handle that rare case in a special way.Quoted 1 line
Quoted 1 line
For the two memory init cases you mentioned in the other email,
maybe a new
Quoted 2 lines
fpi flag to make
free_one_page()use__free_one_page()for them, since migratetypes should be the same acrossMAX_PAGE_ORDERrange there, right?
Quoted 7 lines
Quoted 3 lines
In the context of alloc_frozen_range()/free_frozen_range() I want to free
MAX_PAGE_ORDERchunks when possible, and not have some odd logic in the freeing path undo some of that effort.Adding a pointer to that discussion:
Thanks.
So you are thinking about something like this:
mm/page_alloc.c +6 -1
@@ -1254,7 +1254,12 @@ static void free_one_page(struct zone *zone, struct page *page,unsigned long flags;spin_lock_irqsave (&zone->lock, flags); -split_large_buddy (zone, page, pfn, order, fpi_flags); +if (unlikely (order >MAX_PAGE_ORDER )) +split_large_buddy (zone, page, pfn, order, fpi_flags); +else { +int migratetype =get_pfnblock_migratetype (page, pfn); +__free_one_page (page, pfn, zone, order, migratetype, fpi_flags); + }spin_unlock_irqrestore (&zone->lock, flags);__count_vm_events (PGFREE ,1 << order);
Is it possible to have a MAX_PAGE_ORDER hugetlb folio? If no, we are good. If yes, alloc_contig_range() can change the migratetype of one of half that folio and during migration phase, that folio will be freed via __free_one_page() and causing migratetype mismatch.
Best Regards,
Yan, Zi
Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 103 lines · Zi Yan
Quoted 81 lines · David Hildenbrand
Quoted 80 lines · David Hildenbrand
Quoted 62 lines · Zi Yan
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
(quoted diff) +11
-0+/* Split a multi-block free page into its individual pageblocks. */ +static void split_large_buddy(struct zone *zone, struct page *page, + unsigned long pfn, int order, fpi_t fpi) +{ + unsigned long end = pfn + (1 << order); + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order)); + /* Caller removed page from freelist, buddy info cleared! */ + VM_WARN_ON_ONCE(PageBuddy(page)); + + if (order > pageblock_order)Review
Quoted 10 lines
(quoted diff) +10
-0+ order = pageblock_order; + + while (pfn != end) { + int mt = get_pfnblock_migratetype(page, pfn); + + __free_one_page(page, pfn, zone, order, mt, fpi); + pfn += 1 << order; + page = pfn_to_page(pfn); + } +}Hi,
stumbling over this while digging through the code ....
Review
Quoted 10 lines
(quoted diff) +1
-3+ static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { unsigned long flags; - int migratetype; spin_lock_irqsave(&zone->lock, flags); - migratetype = get_pfnblock_migratetype(page, pfn); - __free_one_page(page, pfn, zone, order, migratetype, fpi_flags);This change is rather undesired:
Quoted 1 line
via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.
Do you have a concrete example? PMD THP on
x86_64ispageblock_order. We do not have PMD level mTHP yet. Any other possible source?Quoted 4 lines
What your new code will do is split this perfectly reasonable
MAX_PAGE_ORDERchunk viasplit_large_buddy()into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.Is there a way to avoid this and just process the whole
MAX_PAGE_ORDERchunk like we used to?Quoted 7 lines
Quoted 3 lines
Probably
split_large_buddy()can check the migratetypes of the to-be-freed page, if order >pageblock_order. If all migratetypes are the same, the page can be freed atMAX_PAGE_ORDER, otherwisepageblock_order.Thinking about this: why do we care about the migratetype?
We only have to fallback to pageblocks if any pageblock is "
MIGRATE_ISOLATE" (and maybeMIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)Quoted 8 lines
There are
VM_WARN_ONCEsaround*_free_list()operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.An implicit reason is that
__free_one_page()does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed viafree_one_page(), wheresplit_large_buddy()is used to work around the limitation.Yes, I saw that change. But that can be easily identified cased by
unlikely(order > MAX_PAGE_ORDER)to handle that rare case in a special way.Quoted 1 line
Quoted 1 line
For the two memory init cases you mentioned in the other email,
maybe a new
Quoted 2 lines
fpi flag to make
free_one_page()use__free_one_page()for them, since migratetypes should be the same acrossMAX_PAGE_ORDERrange there, right?Quoted 7 lines
Quoted 3 lines
In the context of alloc_frozen_range()/free_frozen_range() I want to free
MAX_PAGE_ORDERchunks when possible, and not have some odd logic in the freeing path undo some of that effort.Adding a pointer to that discussion:
Thanks.
So you are thinking about something like this:
mm/page_alloc.c +0
-0@@ -1254,7 +1254,12 @@ static void free_one_page(struct zone *zone, struct page *page,unsigned long flags;
mm/page_alloc.c static void free_one_page(struct zone *zone, struct page *page, · line 1264
Quoted 10 lines
mm/page_alloc.c +5
-1spin_lock_irqsave (&zone->lock, flags); -split_large_buddy (zone, page, pfn, order, fpi_flags); +if (unlikely (order >MAX_PAGE_ORDER )) + split_large_buddy(zone, page, pfn, order, fpi_flags);> + else { +int migratetype =get_pfnblock_migratetype (page, pfn); +__free_one_page (page, pfn, zone, order, migratetype, fpi_flags); + }spin_unlock_irqrestore (&zone->lock, flags);__count_vm_events (PGFREE ,1 << order);
Something in that direction, but likely something like:
if (likely(order <= pageblock_order)) {
int migratetype = get_pfnblock_migratetype(page, pfn);
__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);
} else if (order <= MAX_PAGE_ORDER) {
/* We might have to split when some pageblocks differ. */
maybe_split_large_buddy(zone, page, pfn, order, fpi_flags);
} else {
/* The buddy works in max MAX_PAGE_ORDER chunks. */
for /* each MAX_PAGE_ORDER chunk */
maybe_split_large_buddy(zone, page, pfn, MAX_PAGE_ORDER, fpi_flags);
pfn += MAX_ORDER_NR_PAGES;
page = ...
}
}Whereby maybe_split_large_buddy would check if it has to do anything with the pageblocks, or whether it can just call __free_one_page(order). It would only accept order <=
In the future with free_frozen_pages(), the last else case would vanish.
Quoted 2 lines
Is it possible to have a
MAX_PAGE_ORDERhugetlb folio? If no, we are good.
Hm, I wouldn't rule it out. To be future proof, we'd likely should add a way to handle it. Shouldn't be too hard and expensive: simply read all pageblock migratetypes.
I wonder if there are configs (no page isolation?) where we don't have to check anything.
Quoted 3 lines
If yes,
alloc_contig_range()can change the migratetype of one of half that folio and during migration phase, that folio will be freed via__free_one_page()and causing migratetype mismatch.
Can you remind me how that happens?
Likely, we isolate a single pageblock, and shortly after we free a larger page that covers multiple pageblocks? So this could affect other alloc_contig_range() users as well, I assume?
Thanks!
Signature · 3 lines
Cheers, David / dhildenb
Zi Yan
reviewerRe: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP
Quoted 139 lines · David Hildenbrand
Quoted 103 lines · Zi Yan
Quoted 81 lines · David Hildenbrand
Quoted 80 lines · David Hildenbrand
Quoted 62 lines · Zi Yan
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
(quoted diff) +11
-0+/* Split a multi-block free page into its individual pageblocks. */ +static void split_large_buddy(struct zone *zone, struct page *page, + unsigned long pfn, int order, fpi_t fpi) +{ + unsigned long end = pfn + (1 << order); + + VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order)); + /* Caller removed page from freelist, buddy info cleared! */ + VM_WARN_ON_ONCE(PageBuddy(page)); + + if (order > pageblock_order)Review
Quoted 10 lines
(quoted diff) +10
-0+ order = pageblock_order; + + while (pfn != end) { + int mt = get_pfnblock_migratetype(page, pfn); + + __free_one_page(page, pfn, zone, order, mt, fpi); + pfn += 1 << order; + page = pfn_to_page(pfn); + } +}Hi,
stumbling over this while digging through the code ....
Review
Quoted 10 lines
(quoted diff) +1
-3+ static void free_one_page(struct zone *zone, struct page *page, unsigned long pfn, unsigned int order, fpi_t fpi_flags) { unsigned long flags; - int migratetype; spin_lock_irqsave(&zone->lock, flags); - migratetype = get_pfnblock_migratetype(page, pfn); - __free_one_page(page, pfn, zone, order, migratetype, fpi_flags);This change is rather undesired:
Quoted 1 line
via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.
Do you have a concrete example? PMD THP on
x86_64ispageblock_order. We do not have PMD level mTHP yet. Any other possible source?Quoted 4 lines
What your new code will do is split this perfectly reasonable
MAX_PAGE_ORDERchunk viasplit_large_buddy()into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.Is there a way to avoid this and just process the whole
MAX_PAGE_ORDERchunk like we used to?Quoted 7 lines
Quoted 3 lines
Probably
split_large_buddy()can check the migratetypes of the to-be-freed page, if order >pageblock_order. If all migratetypes are the same, the page can be freed atMAX_PAGE_ORDER, otherwisepageblock_order.Thinking about this: why do we care about the migratetype?
We only have to fallback to pageblocks if any pageblock is "
MIGRATE_ISOLATE" (and maybeMIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)Quoted 8 lines
There are
VM_WARN_ONCEsaround*_free_list()operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.An implicit reason is that
__free_one_page()does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed viafree_one_page(), wheresplit_large_buddy()is used to work around the limitation.Yes, I saw that change. But that can be easily identified cased by
unlikely(order > MAX_PAGE_ORDER)to handle that rare case in a special way.Quoted 1 line
Quoted 1 line
For the two memory init cases you mentioned in the other email,
maybe a new
Quoted 2 lines
fpi flag to make
free_one_page()use__free_one_page()for them, since migratetypes should be the same acrossMAX_PAGE_ORDERrange there, right?Quoted 7 lines
Quoted 3 lines
In the context of alloc_frozen_range()/free_frozen_range() I want to free
MAX_PAGE_ORDERchunks when possible, and not have some odd logic in the freeing path undo some of that effort.Adding a pointer to that discussion:
Thanks.
So you are thinking about something like this:
mm/page_alloc.c +0
-0@@ -1254,7 +1254,12 @@ static void free_one_page(struct zone *zone, struct page *page,unsigned long flags;mm/page_alloc.c static void free_one_page(struct zone *zone, struct page *page, · line 1264
Quoted 10 lines
mm/page_alloc.c +5
-1spin_lock_irqsave (&zone->lock, flags); -split_large_buddy (zone, page, pfn, order, fpi_flags); +if (unlikely (order >MAX_PAGE_ORDER )) + split_large_buddy(zone, page, pfn, order, fpi_flags);> + else { +int migratetype =get_pfnblock_migratetype (page, pfn); +__free_one_page (page, pfn, zone, order, migratetype, fpi_flags); + }spin_unlock_irqrestore (&zone->lock, flags);__count_vm_events (PGFREE ,1 << order);Something in that direction, but likely something like:
if (likely(order <= pageblock_order)) { int migratetype = get_pfnblock_migratetype(page, pfn); __free_one_page(page, pfn, zone, order, migratetype, fpi_flags); } else if (order <= MAX_PAGE_ORDER) { /* We might have to split when some pageblocks differ. */ maybe_split_large_buddy(zone, page, pfn, order, fpi_flags); } else { /* The buddy works in max MAX_PAGE_ORDER chunks. */ for /* each MAX_PAGE_ORDER chunk */ maybe_split_large_buddy(zone, page, pfn, MAX_PAGE_ORDER, fpi_flags); pfn += MAX_ORDER_NR_PAGES; page = ... } }Whereby
maybe_split_large_buddywould check if it has to do anything with the pageblocks, or whether it can just call__free_one_page(order). It would only accept order <=In the future with
free_frozen_pages(), the last else case would vanish.Quoted 1 line
Quoted 8 lines
Quoted 1 line
Is it possible to have a
MAX_PAGE_ORDERhugetlb folio? If no, we are good.Hm, I wouldn't rule it out. To be future proof, we'd likely should add a way to handle it. Shouldn't be too hard and expensive: simply read all pageblock migratetypes.
I wonder if there are configs (no page isolation?) where we don't have to check anything.
CONFIG_CONTIG_ALLOC? CONFIG_MEMORY_ISOLATION seems too much.
Quoted 6 lines
Quoted 4 lines
If yes,
alloc_contig_range()can change the migratetype of one of half that folio and during migration phase, that folio will be freed via__free_one_page()and causing migratetype mismatch.Can you remind me how that happens?
There is a MAX_PAGE_ORDER hugetlb folio across two pageblocks and alloc_contig_range() wants to get a range that overlaps with first pageblock of that hugetlb folio. All pageblocks are marked as isolated first during start_isolate_page_range(), then __alloc_contig_migrate_range() isolates and migrates the hugetlb folio. During migration, the original hugetlb folio is freed and since it is MAX_PAGE_ORDER and based on my code above, it is freed via __free_one_page() with two different migratetypes.
But with your maybe_split_large_buddy(), it would work.
Quoted 4 lines
Likely, we isolate a single pageblock, and shortly after we free a larger page that covers multiple pageblocks? So this could affect other
alloc_contig_range()users as well, I assume?
Best Regards,
Yan, Zi
PATCH v2 2/3 mm/cma: add cma_{alloc,free}_folio()
In reply to Yu ZhaoWith alloc_contig_range() and free_contig_range() supporting large folios, CMA can allocate and free large folios too, by cma_alloc_folio() and cma_free_folio().
--- include/linux/cma.h | 16 +++++++++++++ mm/cma.c | 55 ++++++++++++++++++++++++++++++++------------- 2 files changed, 56 insertions(+), 15 deletions(-)
include/linux/cma.h +16 -0
@@ -52,4 +52,20 @@ extern bool cma_release(struct cma *cma, const struct page *pages, unsigned longextern int cma_for_each_area (int (*it)(struct cma *cma,void *data),void *data);extern void cma_reserve_pages_on_error (struct cma *cma); + +#ifdef CONFIG_CMA +struct folio *cma_alloc_folio (struct cma *cma,int order,gfp_t gfp); +bool cma_free_folio (struct cma *cma,const struct folio *folio); +#else +static inline struct folio *cma_alloc_folio (struct cma *cma,int order,gfp_t gfp) +{ +return NULL ; +} + +static inline bool cma_free_folio (struct cma *cma,const struct folio *folio) +{ +return false; +} +#endif +#endif
mm/cma.c +40 -15
85 lines not on this page · raw message
Signature · 1 line
2.46.0.76.ge559c4bf1a-goog
Re: [PATCH mm-unstable v2 2/3] mm/cma: add cma_{alloc,free}_folio()
Quoted 94 lines · Yu Zhao
With
alloc_contig_range()andfree_contig_range()supporting large folios, CMA can allocate and free large folios too, bycma_alloc_folio()andcma_free_folio().Signed-off-by Yu Zhao <[email protected]>--- include/linux/cma.h | 16 +++++++++++++ mm/cma.c | 55 ++++++++++++++++++++++++++++++++------------- 2 files changed, 56 insertions(+), 15 deletions(-)include/linux/cma.h +16
-0@@ -52,4 +52,20 @@ extern bool cma_release(struct cma *cma, const struct page *pages, unsigned longextern int cma_for_each_area (int (*it)(struct cma *cma,void *data),void *data);extern void cma_reserve_pages_on_error (struct cma *cma); + +#ifdef CONFIG_CMA +struct folio *cma_alloc_folio (struct cma *cma,int order,gfp_t gfp); +bool cma_free_folio (struct cma *cma,const struct folio *folio); +#else +static inline struct folio *cma_alloc_folio (struct cma *cma,int order,gfp_t gfp) +{ +return NULL ; +} + +static inline bool cma_free_folio (struct cma *cma,const struct folio *folio) +{ +return false; +} +#endif +#endif mm/cma.c +15
-15@@ -403,18 +403,8 @@ static void cma_debug_show_areas(struct cma *cma)spin_unlock_irq (&cma->lock); } -/** -* cma_alloc() - allocate pages from contiguous area -* @cma: Contiguous memory region for which the allocation is performed. -* @count: Requested number of pages. -* @align: Requested alignment of pages (in PAGE_SIZE order). -* @no_warn: Avoid printing message about failed allocation -* -* This function allocates part of contiguous memory on specific -* contiguous memory area. -*/ -struct page *cma_alloc (struct cma *cma,unsigned long count, -unsigned int align,bool no_warn) +static struct page *__cma_alloc (struct cma *cma,unsigned long count, +unsigned int align,gfp_t gfp) {unsigned long mask, offset;unsigned long pfn =-1 ;@@ -463,8 +453,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, pfn = cma->base_pfn + (bitmap_no << cma->order_per_bit);mutex_lock (&cma_mutex); - ret =alloc_contig_range (pfn, pfn + count,MIGRATE_CMA , -GFP_KERNEL | (no_warn ? __GFP_NOWARN :0 )); + ret =alloc_contig_range (pfn, pfn + count,MIGRATE_CMA , gfp);mutex_unlock (&cma_mutex);if (ret ==0 ) { page =pfn_to_page (pfn);@@ -494,7 +483,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count,page_kasan_tag_reset (nth_page (page, i)); } -if (ret && !no_warn) { +if (ret && !(gfp & __GFP_NOWARN)) {pr_err_ratelimited ("%s: %s: alloc failed, req-size: %lu pages, ret: %d\n" , __func__, cma->name, count, ret);cma_debug_show_areas (cma);@@ -513,6 +502,34 @@ struct page *cma_alloc(struct cma *cma, unsigned long count,return page; } +/** +* cma_alloc() - allocate pages from contiguous area +* @cma: Contiguous memory region for which the allocation is performed. +* @count: Requested number of pages. +* @align: Requested alignment of pages (in PAGE_SIZE order). +* @no_warn: Avoid printing message about failed allocation +* +* This function allocates part of contiguous memory on specific +* contiguous memory area. +*/ +struct page *cma_alloc(struct cma *cma, unsigned long count,
mm/cma.c struct page *cma_alloc(struct cma *cma, unsigned long count, · line 525
Quoted 10 lines
mm/cma.c +10
-0+ unsigned int align, bool no_warn) +{ +return __cma_alloc (cma, count, align,GFP_KERNEL | (no_warn ? __GFP_NOWARN :0 )); +} + +struct folio *cma_alloc_folio (struct cma *cma,int order,gfp_t gfp) +{ +struct page *page; + +if (WARN_ON (!order || !(gfp | __GFP_COMP)))
And here too. Thank you.
mm/cma.c +1 -1
@@ -522,7 +522,7 @@ struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) {struct page *page; -if (WARN_ON (!order || !(gfp | __GFP_COMP))) +if (WARN_ON (!order || !(gfp & __GFP_COMP)))return NULL ; page =__cma_alloc (cma,1 << order, order, gfp);
Re: [PATCH mm-unstable v2 2/3] mm/cma: add cma_{alloc,free}_folio()
Quoted 113 lines · Yu Zhao
Quoted 94 lines · Yu Zhao
With
alloc_contig_range()andfree_contig_range()supporting large folios, CMA can allocate and free large folios too, bycma_alloc_folio()andcma_free_folio().Signed-off-by Yu Zhao <[email protected]>--- include/linux/cma.h | 16 +++++++++++++ mm/cma.c | 55 ++++++++++++++++++++++++++++++++------------- 2 files changed, 56 insertions(+), 15 deletions(-)include/linux/cma.h +16
-0@@ -52,4 +52,20 @@ extern bool cma_release(struct cma *cma, const struct page *pages, unsigned longextern int cma_for_each_area (int (*it)(struct cma *cma,void *data),void *data);extern void cma_reserve_pages_on_error (struct cma *cma); + +#ifdef CONFIG_CMA +struct folio *cma_alloc_folio (struct cma *cma,int order,gfp_t gfp); +bool cma_free_folio (struct cma *cma,const struct folio *folio); +#else +static inline struct folio *cma_alloc_folio (struct cma *cma,int order,gfp_t gfp) +{ +return NULL ; +} + +static inline bool cma_free_folio (struct cma *cma,const struct folio *folio) +{ +return false; +} +#endif +#endif mm/cma.c +15
-15@@ -403,18 +403,8 @@ static void cma_debug_show_areas(struct cma *cma)spin_unlock_irq (&cma->lock); } -/** -* cma_alloc() - allocate pages from contiguous area -* @cma: Contiguous memory region for which the allocation is performed. -* @count: Requested number of pages. -* @align: Requested alignment of pages (in PAGE_SIZE order). -* @no_warn: Avoid printing message about failed allocation -* -* This function allocates part of contiguous memory on specific -* contiguous memory area. -*/ -struct page *cma_alloc (struct cma *cma,unsigned long count, -unsigned int align,bool no_warn) +static struct page *__cma_alloc (struct cma *cma,unsigned long count, +unsigned int align,gfp_t gfp) {unsigned long mask, offset;unsigned long pfn =-1 ;@@ -463,8 +453,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count, pfn = cma->base_pfn + (bitmap_no << cma->order_per_bit);mutex_lock (&cma_mutex); - ret =alloc_contig_range (pfn, pfn + count,MIGRATE_CMA , -GFP_KERNEL | (no_warn ? __GFP_NOWARN :0 )); + ret =alloc_contig_range (pfn, pfn + count,MIGRATE_CMA , gfp);mutex_unlock (&cma_mutex);if (ret ==0 ) { page =pfn_to_page (pfn);@@ -494,7 +483,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count,page_kasan_tag_reset (nth_page (page, i)); } -if (ret && !no_warn) { +if (ret && !(gfp & __GFP_NOWARN)) {pr_err_ratelimited ("%s: %s: alloc failed, req-size: %lu pages, ret: %d\n" , __func__, cma->name, count, ret);cma_debug_show_areas (cma);@@ -513,6 +502,34 @@ struct page *cma_alloc(struct cma *cma, unsigned long count,return page; } +/** +* cma_alloc() - allocate pages from contiguous area +* @cma: Contiguous memory region for which the allocation is performed. +* @count: Requested number of pages. +* @align: Requested alignment of pages (in PAGE_SIZE order). +* @no_warn: Avoid printing message about failed allocation +* +* This function allocates part of contiguous memory on specific +* contiguous memory area. +*/ +struct page *cma_alloc(struct cma *cma, unsigned long count,mm/cma.c struct page *cma_alloc(struct cma *cma, unsigned long count, · line 525
Quoted 10 lines
mm/cma.c +10
-0+ unsigned int align, bool no_warn) +{ +return __cma_alloc (cma, count, align,GFP_KERNEL | (no_warn ? __GFP_NOWARN :0 )); +} + +struct folio *cma_alloc_folio (struct cma *cma,int order,gfp_t gfp) +{ +struct page *page; + +if (WARN_ON (!order || !(gfp | __GFP_COMP)))And here too. Thank you.
mm/cma.c +0
-0
mm/cma.c struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) · line 528
Quoted 8 lines
mm/cma.c +1
-1@@ -522,7 +522,7 @@ struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) {struct page *page; -if (WARN_ON (!order || !(gfp | __GFP_COMP))) +if (WARN_ON (!order || !(gfp & __GFP_COMP)))return NULL ; page =__cma_alloc (cma,1 << order, order, gfp);
And the following instead of the above, if we don't want to warn the potential bug.
mm/cma.c +1 -4
@@ -522,10 +522,7 @@ struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) {struct page *page; -if (WARN_ON (!order || !(gfp & __GFP_COMP))) -return NULL ; - - page =__cma_alloc (cma,1 << order, order, gfp); + page =__cma_alloc (cma,1 << order, order, gfp | __GFP_COMP);return page ?page_folio (page) :NULL ; }
PATCH v2 3/3 mm/hugetlb: use __GFP_COMP for gigantic folios
In reply to Yu ZhaoUse __GFP_COMP for gigantic folios to greatly reduce not only the amount of code but also the allocation and free time.
LOC (approximately): +60, -240
Allocate and free 500 1GB hugeTLB memory without HVO by:
time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
Before After Alloc ~13s ~10s Free ~15s <1s
The above magnitude generally holds for multiple x86 and arm64 CPU models.
--- include/linux/hugetlb.h | 9 +- mm/hugetlb.c | 293 ++++++++-------------------------------- 2 files changed, 62 insertions(+), 240 deletions(-)
include/linux/hugetlb.h +5 -4
@@ -896,10 +896,11 @@ static inline bool hugepage_movable_supported(struct hstate *h)/* Movability of hugepages depends on migration support. */ static inline gfp_t htlb_alloc_mask (struct hstate *h) { -if (hugepage_movable_supported (h)) -return GFP_HIGHUSER_MOVABLE ; -else -return GFP_HIGHUSER ; +gfp_t gfp = __GFP_COMP | __GFP_NOWARN; + + gfp |=hugepage_movable_supported (h) ?GFP_HIGHUSER_MOVABLE :GFP_HIGHUSER ; + +return gfp; }static inline gfp_t htlb_modify_alloc_mask (struct hstate *h,gfp_t gfp_mask)
mm/hugetlb.c +57 -236
397 lines not on this page · raw message
Signature · 1 line
2.46.0.76.ge559c4bf1a-goog
Zi Yan
reviewerRe: [PATCH mm-unstable v2 3/3] mm/hugetlb: use __GFP_COMP for gigantic folios
Quoted 69 lines · Yu Zhao
Use
__GFP_COMPfor gigantic folios to greatly reduce not only the amount of code but also the allocation and free time.LOC (approximately): +60, -240
Allocate and free 500 1GB hugeTLB memory without HVO by:
time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepagesBefore After Alloc ~13s ~10s Free ~15s <1sThe above magnitude generally holds for multiple x86 and arm64 CPU models.
Signed-off-by Yu Zhao <[email protected]>Reported-by Frank van der Linden <[email protected]>--- include/linux/hugetlb.h | 9 +- mm/hugetlb.c | 293 ++++++++-------------------------------- 2 files changed, 62 insertions(+), 240 deletions(-)include/linux/hugetlb.h +5
-4@@ -896,10 +896,11 @@ static inline bool hugepage_movable_supported(struct hstate *h)/* Movability of hugepages depends on migration support. */ static inline gfp_t htlb_alloc_mask (struct hstate *h) { -if (hugepage_movable_supported (h)) -return GFP_HIGHUSER_MOVABLE ; -else -return GFP_HIGHUSER ; +gfp_t gfp = __GFP_COMP | __GFP_NOWARN; + + gfp |=hugepage_movable_supported (h) ?GFP_HIGHUSER_MOVABLE :GFP_HIGHUSER ; + +return gfp; }static inline gfp_t htlb_modify_alloc_mask (struct hstate *h,gfp_t gfp_mask)mm/hugetlb.c +1
-10@@ -56,16 +56,6 @@ struct hstate hstates[HUGE_MAX_HSTATE];#ifdef CONFIG_CMA static struct cma *hugetlb_cma[MAX_NUMNODES ];static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES ]__initdata ; -static bool hugetlb_cma_folio (struct folio *folio,unsigned int order) -{ -return cma_pages_valid (hugetlb_cma[folio_nid (folio)], &folio->page, -1 << order); -} -#else -static bool hugetlb_cma_folio (struct folio *folio,unsigned int order) -{ -return false; -}#endif static unsigned long hugetlb_cma_size__initdata ;@@ -100,6 +90,17 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma, unsigned long start, unsigned long end);static struct resv_map *vma_resv_map (struct vm_area_struct *vma); +static void hugetlb_free_folio (struct folio *folio)
mm/hugetlb.c static void hugetlb_unshare_pmds(struct vm_area_struct *vma, · line 103
Quoted 10 lines
mm/hugetlb.c +10
-0+{ +#ifdef CONFIG_CMA +int nid =folio_nid (folio); + +if (cma_free_folio (hugetlb_cma[nid], folio)) +return ; +#endif +folio_put (folio); +} +
It seems that we no longer use free_contig_range() to free gigantic folios from alloc_contig_range(). Will it work? Or did I miss anything?
Best Regards,
Yan, Zi
Re: [PATCH mm-unstable v2 3/3] mm/hugetlb: use __GFP_COMP for gigantic folios
Quoted 71 lines · Zi Yan
Quoted 68 lines · Yu Zhao
Use
__GFP_COMPfor gigantic folios to greatly reduce not only the amount of code but also the allocation and free time.LOC (approximately): +60, -240
Allocate and free 500 1GB hugeTLB memory without HVO by:
time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepagesBefore After Alloc ~13s ~10s Free ~15s <1sThe above magnitude generally holds for multiple x86 and arm64 CPU models.
Signed-off-by Yu Zhao <[email protected]>Reported-by Frank van der Linden <[email protected]>--- include/linux/hugetlb.h | 9 +- mm/hugetlb.c | 293 ++++++++-------------------------------- 2 files changed, 62 insertions(+), 240 deletions(-)include/linux/hugetlb.h +5
-4@@ -896,10 +896,11 @@ static inline bool hugepage_movable_supported(struct hstate *h)/* Movability of hugepages depends on migration support. */ static inline gfp_t htlb_alloc_mask (struct hstate *h) { -if (hugepage_movable_supported (h)) -return GFP_HIGHUSER_MOVABLE ; -else -return GFP_HIGHUSER ; +gfp_t gfp = __GFP_COMP | __GFP_NOWARN; + + gfp |=hugepage_movable_supported (h) ?GFP_HIGHUSER_MOVABLE :GFP_HIGHUSER ; + +return gfp; }static inline gfp_t htlb_modify_alloc_mask (struct hstate *h,gfp_t gfp_mask)mm/hugetlb.c +0
-10@@ -56,16 +56,6 @@ struct hstate hstates[HUGE_MAX_HSTATE];#ifdef CONFIG_CMA static struct cma *hugetlb_cma[MAX_NUMNODES ];static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES ]__initdata ; -static bool hugetlb_cma_folio (struct folio *folio,unsigned int order) -{ -return cma_pages_valid (hugetlb_cma[folio_nid (folio)], &folio->page, -1 << order); -} -#else -static bool hugetlb_cma_folio (struct folio *folio,unsigned int order) -{ -return false; -}#endif static unsigned long hugetlb_cma_size__initdata ;@@ -100,6 +90,17 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma, unsigned long start, unsigned long end);static struct resv_map *vma_resv_map (struct vm_area_struct *vma);
Quoted 14 lines
mm/hugetlb.c static void hugetlb_unshare_pmds(struct vm_area_struct *vma, · line 103
Quoted 11 lines
mm/hugetlb.c +11
-0+static void hugetlb_free_folio (struct folio *folio) +{ +#ifdef CONFIG_CMA +int nid =folio_nid (folio); + +if (cma_free_folio (hugetlb_cma[nid], folio)) +return ; +#endif +folio_put (folio); +} +It seems that we no longer use
free_contig_range()to free gigantic folios fromalloc_contig_range().
We switched to two pairs of extern (to the allocator) APIs in this patch:
folio_alloc_gigantic()
folio_put()
and
cma_alloc_folio()
cma_free_folio()
Quoted 1 line
Will it work? Or did I miss anything?
alloc_contig_range and free_contig_range() also works with __GFP_COMP / large folios, but this pair is internal (to the allocator) and shouldn't be used directly except to implement external APIs like above.
Re: [PATCH mm-unstable v2 3/3] mm/hugetlb: use __GFP_COMP for gigantic folios
Quoted 94 lines · Yu Zhao
Quoted 71 lines · Zi Yan
Quoted 68 lines · Yu Zhao
Use
__GFP_COMPfor gigantic folios to greatly reduce not only the amount of code but also the allocation and free time.LOC (approximately): +60, -240
Allocate and free 500 1GB hugeTLB memory without HVO by:
time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepagesBefore After Alloc ~13s ~10s Free ~15s <1sThe above magnitude generally holds for multiple x86 and arm64 CPU models.
Signed-off-by Yu Zhao <[email protected]>Reported-by Frank van der Linden <[email protected]>--- include/linux/hugetlb.h | 9 +- mm/hugetlb.c | 293 ++++++++-------------------------------- 2 files changed, 62 insertions(+), 240 deletions(-)include/linux/hugetlb.h +5
-4@@ -896,10 +896,11 @@ static inline bool hugepage_movable_supported(struct hstate *h)/* Movability of hugepages depends on migration support. */ static inline gfp_t htlb_alloc_mask (struct hstate *h) { -if (hugepage_movable_supported (h)) -return GFP_HIGHUSER_MOVABLE ; -else -return GFP_HIGHUSER ; +gfp_t gfp = __GFP_COMP | __GFP_NOWARN; + + gfp |=hugepage_movable_supported (h) ?GFP_HIGHUSER_MOVABLE :GFP_HIGHUSER ; + +return gfp; }static inline gfp_t htlb_modify_alloc_mask (struct hstate *h,gfp_t gfp_mask)mm/hugetlb.c +0
-10@@ -56,16 +56,6 @@ struct hstate hstates[HUGE_MAX_HSTATE];#ifdef CONFIG_CMA static struct cma *hugetlb_cma[MAX_NUMNODES ];static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES ]__initdata ; -static bool hugetlb_cma_folio (struct folio *folio,unsigned int order) -{ -return cma_pages_valid (hugetlb_cma[folio_nid (folio)], &folio->page, -1 << order); -} -#else -static bool hugetlb_cma_folio (struct folio *folio,unsigned int order) -{ -return false; -}#endif static unsigned long hugetlb_cma_size__initdata ;@@ -100,6 +90,17 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma, unsigned long start, unsigned long end);static struct resv_map *vma_resv_map (struct vm_area_struct *vma);Quoted 14 lines
mm/hugetlb.c static void hugetlb_unshare_pmds(struct vm_area_struct *vma, · line 103
Quoted 11 lines
mm/hugetlb.c +11
-0+static void hugetlb_free_folio (struct folio *folio) +{ +#ifdef CONFIG_CMA +int nid =folio_nid (folio); + +if (cma_free_folio (hugetlb_cma[nid], folio)) +return ; +#endif +folio_put (folio); +} +It seems that we no longer use
free_contig_range()to free gigantic folios fromalloc_contig_range().We switched to two pairs of extern (to the allocator) APIs in this patch:
folio_alloc_gigantic()
folio_put()
and
cma_alloc_folio()
cma_free_folio()
Quoted 6 lines
Quoted 1 line
Will it work? Or did I miss anything?
alloc_contig_rangeandfree_contig_range()also works with__GFP_COMP/ large folios, but this pair is internal (to the allocator) and shouldn't be used directly except to implement external APIs like above.
Oh, I missed split_large_buddy() addition to free_one_page() in patch 1. That handles >MAX_ORDER page freeing. It also makes sure the pageblocks of a free cross-pageblock gigantic hugetlb page go back to the right free lists when each pageblock might have different migratetypes, since the page is freed at pageblock granularity.
The whole series looks good to me.
Best Regards,
Yan, Zi
Quoted 6 lines · Yu Zhao
Use
__GFP_COMPfor gigantic folios can greatly reduce not only the amount of code but also the allocation and free time.Approximate LOC to
mm/hugetlb.c: +60, -240Allocate and free 500 1GB hugeTLB memory without HVO by:
Do you also have numbers with HVO enabled ?
Quoted 7 lines
time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
Before After
Alloc ~13s ~10s Free ~15s <1s
Thanks,
-jane