Skip to contentSkip to conversation

mm/hugetlb: alloc/free gigantic folios

PATCH In review Yu Zhao 7 participants 26 messages Aug 14, 2024 – Nov 20, 2024

Subsystems (5)
lore
Share & tools
lore

Yu Zhao

author
#0 lore raw

v2 cover mm/hugetlb: alloc/free gigantic folios

Use __GFP_COMP for gigantic folios can greatly reduce not only the amount of code but also the allocation and free time.

Approximate LOC to mm/hugetlb.c: +60, -240

Allocate and free 500 1GB hugeTLB memory without HVO by:
time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

       Before  After
Alloc  ~13s    ~10s
Free   ~15s    <1s

The above magnitude generally holds for multiple x86 and arm64 CPU models.

Perf profile before:
Alloc
- 99.99% alloc_pool_huge_folio
- __alloc_fresh_hugetlb_folio
- 83.23% alloc_contig_pages_noprof
- 47.46% alloc_contig_range_noprof
- 20.96% isolate_freepages_range
16.10% split_page
- 14.10% start_isolate_page_range
- 12.02% undo_isolate_page_range

Free
- update_and_free_pages_bulk
- 87.71% free_contig_range
- 76.02% free_unref_page
- 41.30% free_unref_page_commit
- 32.58% free_pcppages_bulk
- 24.75% __free_one_page
13.96% _raw_spin_trylock
12.27% __update_and_free_hugetlb_folio

Perf profile after:
Alloc
- 99.99% alloc_pool_huge_folio
alloc_gigantic_folio
- alloc_contig_pages_noprof
- 59.15% alloc_contig_range_noprof
- 20.72% start_isolate_page_range
20.64% prep_new_page
- 17.13% undo_isolate_page_range

Free
- update_and_free_pages_bulk
- __folio_put
- __free_pages_ok
7.46% free_tail_page_prepare
- 1.97% free_one_page
1.86% __free_one_page

Yu Zhao (3):
mm/contig_alloc: support __GFP_COMP
mm/cma: add cma_{alloc,free}_folio()
mm/hugetlb: use __GFP_COMP for gigantic folios

 include/linux/cma.h     |  16 +++
 include/linux/gfp.h     |  23 ++++
 include/linux/hugetlb.h |   9 +-
 mm/cma.c                |  55 ++++++--
 mm/compaction.c         |  41 +-----
 mm/hugetlb.c            | 293 ++++++++--------------------------------
 mm/page_alloc.c         | 111 ++++++++++-----
 7 files changed, 226 insertions(+), 322 deletions(-)
Signature · 1 line
2.46.0.76.ge559c4bf1a-goog
author
#1 lore raw

PATCH v2 1/3 mm/contig_alloc: support __GFP_COMP

Support __GFP_COMP in alloc_contig_range(). When the flag is set, upon success the function returns a large folio prepared by prep_new_page(), rather than a range of order-0 pages prepared by split_free_pages() (which is renamed from split_map_pages()).

alloc_contig_range() can be used to allocate folios larger than MAX_PAGE_ORDER, e.g., gigantic hugeTLB folios. So on the free path, free_one_page() needs to handle that by split_large_buddy().

---
 include/linux/gfp.h |  23 +++++++++
 mm/compaction.c     |  41 ++--------------
 mm/page_alloc.c     | 111 +++++++++++++++++++++++++++++++-------------
 3 files changed, 108 insertions(+), 67 deletions(-)
include/linux/gfp.h +23 -0
index f53f76e0b17e..59266df56aeb 100644
@@ -446,4 +446,27 @@ extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_
 #endif
 void free_contig_range(unsigned long pfn, unsigned long nr_pages);
 
+#ifdef CONFIG_CONTIG_ALLOC
+static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
+							int nid, nodemask_t *node)
+{
+	struct page *page;
+
+	if (WARN_ON(!order || !(gfp | __GFP_COMP)))
+		return NULL;
+
+	page = alloc_contig_pages_noprof(1 << order, gfp, nid, node);
+
+	return page ? page_folio(page) : NULL;
+}
+#else
+static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
+							int nid, nodemask_t *node)
+{
+	return NULL;
+}
+#endif
+/* This should be paired with folio_put() rather than free_contig_range(). */
+#define folio_alloc_gigantic(...) alloc_hooks(folio_alloc_gigantic_noprof(__VA_ARGS__))
+
 #endif /* __LINUX_GFP_H */
mm/compaction.c +5 -36

79 lines not on this page · raw message

mm/page_alloc.c +80 -31

161 lines not on this page · raw message

Signature · 1 line
2.46.0.76.ge559c4bf1a-goog
#2 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 21 lines · Yu Zhao

Support __GFP_COMP in alloc_contig_range(). When the flag is set, upon success the function returns a large folio prepared by prep_new_page(), rather than a range of order-0 pages prepared by split_free_pages() (which is renamed from split_map_pages()).

alloc_contig_range() can be used to allocate folios larger than MAX_PAGE_ORDER, e.g., gigantic hugeTLB folios. So on the free path, free_one_page() needs to handle that by split_large_buddy().

---
 include/linux/gfp.h |  23 +++++++++
 mm/compaction.c     |  41 ++--------------
 mm/page_alloc.c     | 111 +++++++++++++++++++++++++++++++-------------
 3 files changed, 108 insertions(+), 67 deletions(-)
include/linux/gfp.h +0 -0
index f53f76e0b17e..59266df56aeb 100644

include/linux/gfp.h extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_ · line 455

Quoted 10 lines
include/linux/gfp.h +7 -0
@@ -446,4 +446,27 @@ extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_
 #endif
 void free_contig_range(unsigned long pfn, unsigned long nr_pages);
 
+#ifdef CONFIG_CONTIG_ALLOC
+static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
+							int nid, nodemask_t *node)
+{
+	struct page *page;
+
+	if (WARN_ON(!order || !(gfp | __GFP_COMP)))

It doesn't seem right, it should be !(gfp & __GFP_COMP).

Best Regards,
Yu

author
#3 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 26 lines · Yu Liao
Quoted 24 lines · Yu Zhao

Support __GFP_COMP in alloc_contig_range(). When the flag is set, upon success the function returns a large folio prepared by prep_new_page(), rather than a range of order-0 pages prepared by split_free_pages() (which is renamed from split_map_pages()).

alloc_contig_range() can be used to allocate folios larger than MAX_PAGE_ORDER, e.g., gigantic hugeTLB folios. So on the free path, free_one_page() needs to handle that by split_large_buddy().

---
 include/linux/gfp.h |  23 +++++++++
 mm/compaction.c     |  41 ++--------------
 mm/page_alloc.c     | 111 +++++++++++++++++++++++++++++++-------------
 3 files changed, 108 insertions(+), 67 deletions(-)
include/linux/gfp.h +0 -0
index f53f76e0b17e..59266df56aeb 100644
@@ -446,4 +446,27 @@ extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_
 #endif
 void free_contig_range(unsigned long pfn, unsigned long nr_pages);
 
Quoted 9 lines

include/linux/gfp.h extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_ · line 455

Quoted 7 lines
include/linux/gfp.h +7 -0
+#ifdef CONFIG_CONTIG_ALLOC
+static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
+                                                     int nid, nodemask_t *node)
+{
+     struct page *page;
+
+     if (WARN_ON(!order || !(gfp | __GFP_COMP)))

It doesn't seem right, it should be !(gfp & __GFP_COMP).

Thanks. I've asked Andrew to patch this up (and another place in mm/cma.c).

maintainer
#4 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

In reply to Yu Liao
Quoted 9 lines · On Thu, 22 Aug 2024 16:21:27 +0800 Yu Liao <[email protected]> wrote:

Review

Quoted 7 lines
(quoted diff) +7 -0
+#ifdef CONFIG_CONTIG_ALLOC
+static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
+							int nid, nodemask_t *node)
+{
+	struct page *page;
+
+	if (WARN_ON(!order || !(gfp | __GFP_COMP)))

It doesn't seem right, it should be !(gfp & __GFP_COMP).

Yup. Although I'm wondering why we're checking __GFP_COMP at all? It's a gigantic page - we know it's compound, so why require that the caller tell us something we already know?

include/linux/gfp.h renamed from include/linux/gfp.h~mm-contig_alloc-support-__gfp_comp-fix +1 -1
@@ -452,7 +452,7 @@ static inline struct folio *folio_alloc_
 {
 	struct page *page;
 
-	if (WARN_ON(!order || !(gfp | __GFP_COMP)))
+	if (WARN_ON(!order || !(gfp & __GFP_COMP)))
 		return NULL;
 
 	page = alloc_contig_pages_noprof(1 << order, gfp, nid, node);

_

author
#5 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

In reply to Yu Zhao
Quoted 21 lines · Yu Zhao

Support __GFP_COMP in alloc_contig_range(). When the flag is set, upon success the function returns a large folio prepared by prep_new_page(), rather than a range of order-0 pages prepared by split_free_pages() (which is renamed from split_map_pages()).

alloc_contig_range() can be used to allocate folios larger than MAX_PAGE_ORDER, e.g., gigantic hugeTLB folios. So on the free path, free_one_page() needs to handle that by split_large_buddy().

---
 include/linux/gfp.h |  23 +++++++++
 mm/compaction.c     |  41 ++--------------
 mm/page_alloc.c     | 111 +++++++++++++++++++++++++++++++-------------
 3 files changed, 108 insertions(+), 67 deletions(-)
include/linux/gfp.h +0 -0
index f53f76e0b17e..59266df56aeb 100644

include/linux/gfp.h extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_ · line 455

Quoted 10 lines
include/linux/gfp.h +7 -0
@@ -446,4 +446,27 @@ extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_
 #endif
 void free_contig_range(unsigned long pfn, unsigned long nr_pages);
 
+#ifdef CONFIG_CONTIG_ALLOC
+static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
+							int nid, nodemask_t *node)
+{
+	struct page *page;
+
+	if (WARN_ON(!order || !(gfp | __GFP_COMP)))

Andrew, could you patch up the line above? This is what it's supposed to check:

include/linux/gfp.h +1 -1
index 59266df56aeb..03ba9563c6db 100644
@@ -452,7 +452,7 @@ static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
 {
 	struct page *page;
 
-	if (WARN_ON(!order || !(gfp | __GFP_COMP)))
+	if (WARN_ON(!order || !(gfp & __GFP_COMP)))
 		return NULL;
 
 	page = alloc_contig_pages_noprof(1 << order, gfp, nid, node);
#6 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 8 lines · Yu Zhao

Andrew, could you patch up the line above? This is what it's supposed to check:

include/linux/gfp.h +0 -0
index 59266df56aeb..03ba9563c6db 100644

include/linux/gfp.h static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, · line 456

Quoted 6 lines
include/linux/gfp.h +1 -1
@@ -452,7 +452,7 @@ static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
 {
 	struct page *page;
 
-	if (WARN_ON(!order || !(gfp | __GFP_COMP)))
+	if (WARN_ON(!order || !(gfp & __GFP_COMP)))
 		return NULL;

I don't think we should do this at all. Just this should be enough:

	gfp |= __GFP_COMP;

same as folio_alloc() (or now folio_alloc_noprof()).
Do we really caree if somebody tries to allocate a gigantic page with an order of 0? It's weird, but would work, so I don't see the need for the warning.

Quoted 2 lines
include/linux/gfp.h +0 -0
 	page = alloc_contig_pages_noprof(1 << order, gfp, nid, node);
 
author
#7 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 18 lines · Matthew Wilcox
Quoted 8 lines · Yu Zhao

Andrew, could you patch up the line above? This is what it's supposed to check:

include/linux/gfp.h +0 -0
index 59266df56aeb..03ba9563c6db 100644

include/linux/gfp.h static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp, · line 456

Quoted 6 lines
include/linux/gfp.h +1 -1
@@ -452,7 +452,7 @@ static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
 {
 	struct page *page;
 
-	if (WARN_ON(!order || !(gfp | __GFP_COMP)))
+	if (WARN_ON(!order || !(gfp & __GFP_COMP)))
 		return NULL;

I don't think we should do this at all. Just this should be enough:

Quoted 5 lines
	gfp |= __GFP_COMP;

same as folio_alloc() (or now folio_alloc_noprof()).
Do we really caree if somebody tries to allocate a gigantic page with an order of 0?

If this ever happens, I'd bet it's a bug.

Quoted 2 lines

It's weird, but would work, so I don't see the need for the warning.

So the warning could catch that, but if we think it's verbose, then please fold the following in:

include/linux/gfp.h +1 -4
index a951de920e20..b43934d79dd9 100644
@@ -452,10 +452,7 @@ static inline struct folio *folio_alloc_gigantic_noprof(int order, gfp_t gfp,
 {
 	struct page *page;
 
-	if (WARN_ON(!order || !(gfp & __GFP_COMP)))
-		return NULL;
-
-	page = alloc_contig_pages_noprof(1 << order, gfp, nid, node);
+	page = alloc_contig_pages_noprof(1 << order, gfp | __GFP_COMP, nid, node);
 
 	return page ? page_folio(page) : NULL;
 }
#8 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

In reply to Yu Zhao
Quoted 11 lines
mm/page_alloc.c +11 -0
+/* Split a multi-block free page into its individual pageblocks. */
+static void split_large_buddy(struct zone *zone, struct page *page,
+			      unsigned long pfn, int order, fpi_t fpi)
+{
+	unsigned long end = pfn + (1 << order);
+
+	VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
+	/* Caller removed page from freelist, buddy info cleared! */
+	VM_WARN_ON_ONCE(PageBuddy(page));
+
+	if (order > pageblock_order)

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220

Quoted 10 lines
mm/page_alloc.c +10 -0
+		order = pageblock_order;
+
+	while (pfn != end) {
+		int mt = get_pfnblock_migratetype(page, pfn);
+
+		__free_one_page(page, pfn, zone, order, mt, fpi);
+		pfn += 1 << order;
+		page = pfn_to_page(pfn);
+	}
+}

Hi,

stumbling over this while digging through the code ....

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209

Quoted 11 lines
mm/page_alloc.c +1 -3
+
  static void free_one_page(struct zone *zone, struct page *page,
  			  unsigned long pfn, unsigned int order,
  			  fpi_t fpi_flags)
  {
  	unsigned long flags;
-	int migratetype;
 
  	spin_lock_irqsave(&zone->lock, flags);
-	migratetype = get_pfnblock_migratetype(page, pfn);
-	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);

This change is rather undesired:

via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.

What your new code will do is split this perfectly reasonable MAX_PAGE_ORDER chunk via split_large_buddy() into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.

Is there a way to avoid this and just process the whole MAX_PAGE_ORDER chunk like we used to?

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#9 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11 -0
+/* Split a multi-block free page into its individual pageblocks. */
+static void split_large_buddy(struct zone *zone, struct page *page,
+			      unsigned long pfn, int order, fpi_t fpi)
+{
+	unsigned long end = pfn + (1 << order);
+
+	VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
+	/* Caller removed page from freelist, buddy info cleared! */
+	VM_WARN_ON_ONCE(PageBuddy(page));
+
+	if (order > pageblock_order)

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220

Quoted 10 lines
mm/page_alloc.c +10 -0
+		order = pageblock_order;
+
+	while (pfn != end) {
+		int mt = get_pfnblock_migratetype(page, pfn);
+
+		__free_one_page(page, pfn, zone, order, mt, fpi);
+		pfn += 1 << order;
+		page = pfn_to_page(pfn);
+	}
+}

Hi,

stumbling over this while digging through the code ....

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209

Quoted 10 lines
mm/page_alloc.c +1 -3
+
  static void free_one_page(struct zone *zone, struct page *page,
  			  unsigned long pfn, unsigned int order,
  			  fpi_t fpi_flags)
  {
  	unsigned long flags;
-	int migratetype;
   	spin_lock_irqsave(&zone->lock, flags);
-	migratetype = get_pfnblock_migratetype(page, pfn);
-	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);

This change is rather undesired:

Quoted 1 line

via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.

Do you have a concrete example? PMD THP on x86_64 is pageblock_order. We do not have PMD level mTHP yet. Any other possible source?

Quoted 4 lines

What your new code will do is split this perfectly reasonable MAX_PAGE_ORDER chunk via split_large_buddy() into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.

Is there a way to avoid this and just process the whole MAX_PAGE_ORDER chunk like we used to?

Probably split_large_buddy() can check the migratetypes of the to-be-freed page, if order > pageblock_order. If all migratetypes are the same, the page can be freed at MAX_PAGE_ORDER, otherwise pageblock_order.

Best Regards,
Yan, Zi

#10 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 41 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11 -0
+/* Split a multi-block free page into its individual pageblocks. */
+static void split_large_buddy(struct zone *zone, struct page *page,
+			      unsigned long pfn, int order, fpi_t fpi)
+{
+	unsigned long end = pfn + (1 << order);
+
+	VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
+	/* Caller removed page from freelist, buddy info cleared! */
+	VM_WARN_ON_ONCE(PageBuddy(page));
+
+	if (order > pageblock_order)

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220

Quoted 10 lines
mm/page_alloc.c +10 -0
+		order = pageblock_order;
+
+	while (pfn != end) {
+		int mt = get_pfnblock_migratetype(page, pfn);
+
+		__free_one_page(page, pfn, zone, order, mt, fpi);
+		pfn += 1 << order;
+		page = pfn_to_page(pfn);
+	}
+}

Hi,

stumbling over this while digging through the code ....

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209

Quoted 10 lines
mm/page_alloc.c +1 -3
+
   static void free_one_page(struct zone *zone, struct page *page,
   			  unsigned long pfn, unsigned int order,
   			  fpi_t fpi_flags)
   {
   	unsigned long flags;
-	int migratetype;
    	spin_lock_irqsave(&zone->lock, flags);
-	migratetype = get_pfnblock_migratetype(page, pfn);
-	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);

This change is rather undesired:

Quoted 4 lines
Quoted 1 line

via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.

Do you have a concrete example? PMD THP on x86_64 is pageblock_order. We do not have PMD level mTHP yet. Any other possible source?

Memory init during boot. See deferred_free_pages() and __free_pages_memory()->memblock_free_pages().

So this is used for exposing most memory during boot to the buddy in MAX_PAGE_ORDER granularity.

The other is memory hotplug via generic_online_pages().

Signature · 3 lines
Cheers,

David / dhildenb
#11 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

In reply to Zi Yan
Quoted 47 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11 -0
+/* Split a multi-block free page into its individual pageblocks. */
+static void split_large_buddy(struct zone *zone, struct page *page,
+			      unsigned long pfn, int order, fpi_t fpi)
+{
+	unsigned long end = pfn + (1 << order);
+
+	VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
+	/* Caller removed page from freelist, buddy info cleared! */
+	VM_WARN_ON_ONCE(PageBuddy(page));
+
+	if (order > pageblock_order)

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220

Quoted 10 lines
mm/page_alloc.c +10 -0
+		order = pageblock_order;
+
+	while (pfn != end) {
+		int mt = get_pfnblock_migratetype(page, pfn);
+
+		__free_one_page(page, pfn, zone, order, mt, fpi);
+		pfn += 1 << order;
+		page = pfn_to_page(pfn);
+	}
+}

Hi,

stumbling over this while digging through the code ....

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209

Quoted 10 lines
mm/page_alloc.c +1 -3
+
   static void free_one_page(struct zone *zone, struct page *page,
   			  unsigned long pfn, unsigned int order,
   			  fpi_t fpi_flags)
   {
   	unsigned long flags;
-	int migratetype;
    	spin_lock_irqsave(&zone->lock, flags);
-	migratetype = get_pfnblock_migratetype(page, pfn);
-	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);

This change is rather undesired:

Quoted 1 line

via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.

Do you have a concrete example? PMD THP on x86_64 is pageblock_order. We do not have PMD level mTHP yet. Any other possible source?

Quoted 1 line
Quoted 7 lines
Quoted 3 lines

What your new code will do is split this perfectly reasonable MAX_PAGE_ORDER chunk via split_large_buddy() into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.

Is there a way to avoid this and just process the whole MAX_PAGE_ORDER chunk like we used to?

Probably split_large_buddy() can check the migratetypes of the to-be-freed page, if order > pageblock_order. If all migratetypes are the same, the page can be freed at MAX_PAGE_ORDER, otherwise pageblock_order.

Thinking about this: why do we care about the migratetype?

We only have to fallback to pageblocks if any pageblock is "MIGRATE_ISOLATE" (and maybe MIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#12 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11 -0
+/* Split a multi-block free page into its individual pageblocks. */
+static void split_large_buddy(struct zone *zone, struct page *page,
+			      unsigned long pfn, int order, fpi_t fpi)
+{
+	unsigned long end = pfn + (1 << order);
+
+	VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
+	/* Caller removed page from freelist, buddy info cleared! */
+	VM_WARN_ON_ONCE(PageBuddy(page));
+
+	if (order > pageblock_order)

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220

Quoted 10 lines
mm/page_alloc.c +10 -0
+		order = pageblock_order;
+
+	while (pfn != end) {
+		int mt = get_pfnblock_migratetype(page, pfn);
+
+		__free_one_page(page, pfn, zone, order, mt, fpi);
+		pfn += 1 << order;
+		page = pfn_to_page(pfn);
+	}
+}

Hi,

stumbling over this while digging through the code ....

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209

Quoted 10 lines
mm/page_alloc.c +1 -3
+
   static void free_one_page(struct zone *zone, struct page *page,
   			  unsigned long pfn, unsigned int order,
   			  fpi_t fpi_flags)
   {
   	unsigned long flags;
-	int migratetype;
    	spin_lock_irqsave(&zone->lock, flags);
-	migratetype = get_pfnblock_migratetype(page, pfn);
-	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);

This change is rather undesired:

Quoted 1 line

via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.

Do you have a concrete example? PMD THP on x86_64 is pageblock_order. We do not have PMD level mTHP yet. Any other possible source?

Quoted 4 lines

What your new code will do is split this perfectly reasonable MAX_PAGE_ORDER chunk via split_large_buddy() into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.

Is there a way to avoid this and just process the whole MAX_PAGE_ORDER chunk like we used to?

Quoted 7 lines
Quoted 3 lines

Probably split_large_buddy() can check the migratetypes of the to-be-freed page, if order > pageblock_order. If all migratetypes are the same, the page can be freed at MAX_PAGE_ORDER, otherwise pageblock_order.

Thinking about this: why do we care about the migratetype?

We only have to fallback to pageblocks if any pageblock is "MIGRATE_ISOLATE" (and maybe MIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)

There are VM_WARN_ONCEs around *_free_list() operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.

An implicit reason is that __free_one_page() does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed via free_one_page(), where split_large_buddy() is used to work around the limitation.

For the two memory init cases you mentioned in the other email, maybe a new fpi flag to make free_one_page() use __free_one_page() for them, since migratetypes should be the same across MAX_PAGE_ORDER range there, right?

Best Regards,
Yan, Zi

#13 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 62 lines · Zi Yan
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11 -0
+/* Split a multi-block free page into its individual pageblocks. */
+static void split_large_buddy(struct zone *zone, struct page *page,
+			      unsigned long pfn, int order, fpi_t fpi)
+{
+	unsigned long end = pfn + (1 << order);
+
+	VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
+	/* Caller removed page from freelist, buddy info cleared! */
+	VM_WARN_ON_ONCE(PageBuddy(page));
+
+	if (order > pageblock_order)

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220

Quoted 10 lines
mm/page_alloc.c +10 -0
+		order = pageblock_order;
+
+	while (pfn != end) {
+		int mt = get_pfnblock_migratetype(page, pfn);
+
+		__free_one_page(page, pfn, zone, order, mt, fpi);
+		pfn += 1 << order;
+		page = pfn_to_page(pfn);
+	}
+}

Hi,

stumbling over this while digging through the code ....

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209

Quoted 10 lines
mm/page_alloc.c +1 -3
+
    static void free_one_page(struct zone *zone, struct page *page,
    			  unsigned long pfn, unsigned int order,
    			  fpi_t fpi_flags)
    {
    	unsigned long flags;
-	int migratetype;
     	spin_lock_irqsave(&zone->lock, flags);
-	migratetype = get_pfnblock_migratetype(page, pfn);
-	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);

This change is rather undesired:

Quoted 1 line

via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.

Do you have a concrete example? PMD THP on x86_64 is pageblock_order. We do not have PMD level mTHP yet. Any other possible source?

Quoted 4 lines

What your new code will do is split this perfectly reasonable MAX_PAGE_ORDER chunk via split_large_buddy() into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.

Is there a way to avoid this and just process the whole MAX_PAGE_ORDER chunk like we used to?

Quoted 7 lines
Quoted 3 lines

Probably split_large_buddy() can check the migratetypes of the to-be-freed page, if order > pageblock_order. If all migratetypes are the same, the page can be freed at MAX_PAGE_ORDER, otherwise pageblock_order.

Thinking about this: why do we care about the migratetype?

We only have to fallback to pageblocks if any pageblock is "MIGRATE_ISOLATE" (and maybe MIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)

Quoted 8 lines

There are VM_WARN_ONCEs around *_free_list() operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.

An implicit reason is that __free_one_page() does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed via free_one_page(), where split_large_buddy() is used to work around the limitation.

Yes, I saw that change. But that can be easily identified cased by unlikely(order > MAX_PAGE_ORDER) to handle that rare case in a special way.

Quoted 3 lines
Quoted 1 line

For the two memory init cases you mentioned in the other email, maybe a new

fpi flag to make free_one_page() use __free_one_page() for them, since migratetypes should be the same across MAX_PAGE_ORDER range there, right?

In the context of alloc_frozen_range()/free_frozen_range() I want to free MAX_PAGE_ORDER chunks when possible, and not have some odd logic in the freeing path undo some of that effort.

In the common case, the pageblocks will all have the same time when freeing a MAX_PAGE_ORDER page, so we should identify the odd case and only special-case that.

Signature · 3 lines
Cheers,

David / dhildenb
#14 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 75 lines · David Hildenbrand
Quoted 62 lines · Zi Yan
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
mm/page_alloc.c +11 -0
+/* Split a multi-block free page into its individual pageblocks. */
+static void split_large_buddy(struct zone *zone, struct page *page,
+			      unsigned long pfn, int order, fpi_t fpi)
+{
+	unsigned long end = pfn + (1 << order);
+
+	VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
+	/* Caller removed page from freelist, buddy info cleared! */
+	VM_WARN_ON_ONCE(PageBuddy(page));
+
+	if (order > pageblock_order)

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1220

Quoted 10 lines
mm/page_alloc.c +10 -0
+		order = pageblock_order;
+
+	while (pfn != end) {
+		int mt = get_pfnblock_migratetype(page, pfn);
+
+		__free_one_page(page, pfn, zone, order, mt, fpi);
+		pfn += 1 << order;
+		page = pfn_to_page(pfn);
+	}
+}

Hi,

stumbling over this while digging through the code ....

mm/page_alloc.c static void free_pcppages_bulk(struct zone *zone, int count, · line 1209

Quoted 10 lines
mm/page_alloc.c +1 -3
+
     static void free_one_page(struct zone *zone, struct page *page,
     			  unsigned long pfn, unsigned int order,
     			  fpi_t fpi_flags)
     {
     	unsigned long flags;
-	int migratetype;
      	spin_lock_irqsave(&zone->lock, flags);
-	migratetype = get_pfnblock_migratetype(page, pfn);
-	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);

This change is rather undesired:

Quoted 1 line

via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.

Do you have a concrete example? PMD THP on x86_64 is pageblock_order. We do not have PMD level mTHP yet. Any other possible source?

Quoted 4 lines

What your new code will do is split this perfectly reasonable MAX_PAGE_ORDER chunk via split_large_buddy() into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.

Is there a way to avoid this and just process the whole MAX_PAGE_ORDER chunk like we used to?

Quoted 7 lines
Quoted 3 lines

Probably split_large_buddy() can check the migratetypes of the to-be-freed page, if order > pageblock_order. If all migratetypes are the same, the page can be freed at MAX_PAGE_ORDER, otherwise pageblock_order.

Thinking about this: why do we care about the migratetype?

We only have to fallback to pageblocks if any pageblock is "MIGRATE_ISOLATE" (and maybe MIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)

Quoted 8 lines

There are VM_WARN_ONCEs around *_free_list() operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.

An implicit reason is that __free_one_page() does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed via free_one_page(), where split_large_buddy() is used to work around the limitation.

Yes, I saw that change. But that can be easily identified cased by unlikely(order > MAX_PAGE_ORDER) to handle that rare case in a special way.

Quoted 8 lines
Quoted 1 line
Quoted 1 line

For the two memory init cases you mentioned in the other email,

maybe a new

Quoted 2 lines

fpi flag to make free_one_page() use __free_one_page() for them, since migratetypes should be the same across MAX_PAGE_ORDER range there, right?

In the context of alloc_frozen_range()/free_frozen_range() I want to free MAX_PAGE_ORDER chunks when possible, and not have some odd logic in the freeing path undo some of that effort.

Adding a pointer to that discussion:

https://lkml.kernel.org/r/[email protected]

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#15 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 81 lines · David Hildenbrand
Quoted 80 lines · David Hildenbrand
Quoted 62 lines · Zi Yan
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
(quoted diff) +11 -0
+/* Split a multi-block free page into its individual pageblocks. */
+static void split_large_buddy(struct zone *zone, struct page *page,
+			      unsigned long pfn, int order, fpi_t fpi)
+{
+	unsigned long end = pfn + (1 << order);
+
+	VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
+	/* Caller removed page from freelist, buddy info cleared! */
+	VM_WARN_ON_ONCE(PageBuddy(page));
+
+	if (order > pageblock_order)

Review

Quoted 10 lines
(quoted diff) +10 -0
+		order = pageblock_order;
+
+	while (pfn != end) {
+		int mt = get_pfnblock_migratetype(page, pfn);
+
+		__free_one_page(page, pfn, zone, order, mt, fpi);
+		pfn += 1 << order;
+		page = pfn_to_page(pfn);
+	}
+}

Hi,

stumbling over this while digging through the code ....

Review

Quoted 10 lines
(quoted diff) +1 -3
+
     static void free_one_page(struct zone *zone, struct page *page,
     			  unsigned long pfn, unsigned int order,
     			  fpi_t fpi_flags)
     {
     	unsigned long flags;
-	int migratetype;
      	spin_lock_irqsave(&zone->lock, flags);
-	migratetype = get_pfnblock_migratetype(page, pfn);
-	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);

This change is rather undesired:

Quoted 1 line

via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.

Do you have a concrete example? PMD THP on x86_64 is pageblock_order. We do not have PMD level mTHP yet. Any other possible source?

Quoted 4 lines

What your new code will do is split this perfectly reasonable MAX_PAGE_ORDER chunk via split_large_buddy() into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.

Is there a way to avoid this and just process the whole MAX_PAGE_ORDER chunk like we used to?

Quoted 7 lines
Quoted 3 lines

Probably split_large_buddy() can check the migratetypes of the to-be-freed page, if order > pageblock_order. If all migratetypes are the same, the page can be freed at MAX_PAGE_ORDER, otherwise pageblock_order.

Thinking about this: why do we care about the migratetype?

We only have to fallback to pageblocks if any pageblock is "MIGRATE_ISOLATE" (and maybe MIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)

Quoted 8 lines

There are VM_WARN_ONCEs around *_free_list() operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.

An implicit reason is that __free_one_page() does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed via free_one_page(), where split_large_buddy() is used to work around the limitation.

Yes, I saw that change. But that can be easily identified cased by unlikely(order > MAX_PAGE_ORDER) to handle that rare case in a special way.

Quoted 1 line
Quoted 1 line

For the two memory init cases you mentioned in the other email,

maybe a new

Quoted 2 lines

fpi flag to make free_one_page() use __free_one_page() for them, since migratetypes should be the same across MAX_PAGE_ORDER range there, right?

Quoted 7 lines
Quoted 3 lines

In the context of alloc_frozen_range()/free_frozen_range() I want to free MAX_PAGE_ORDER chunks when possible, and not have some odd logic in the freeing path undo some of that effort.

Adding a pointer to that discussion:

https://lkml.kernel.org/r/[email protected]

Thanks.

So you are thinking about something like this:

mm/page_alloc.c +6 -1
index b6958333054d..3d3341dc1ad1 100644
@@ -1254,7 +1254,12 @@ static void free_one_page(struct zone *zone, struct page *page,
 	unsigned long flags;
 
 	spin_lock_irqsave(&zone->lock, flags);
-	split_large_buddy(zone, page, pfn, order, fpi_flags);
+	if (unlikely(order > MAX_PAGE_ORDER))
+		split_large_buddy(zone, page, pfn, order, fpi_flags);
+	else {
+		int migratetype = get_pfnblock_migratetype(page, pfn);
+		__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);
+	}
 	spin_unlock_irqrestore(&zone->lock, flags);
 
 	__count_vm_events(PGFREE, 1 << order);

Is it possible to have a MAX_PAGE_ORDER hugetlb folio? If no, we are good. If yes, alloc_contig_range() can change the migratetype of one of half that folio and during migration phase, that folio will be freed via __free_one_page() and causing migratetype mismatch.

Best Regards,
Yan, Zi

#16 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 103 lines · Zi Yan
Quoted 81 lines · David Hildenbrand
Quoted 80 lines · David Hildenbrand
Quoted 62 lines · Zi Yan
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
(quoted diff) +11 -0
+/* Split a multi-block free page into its individual pageblocks. */
+static void split_large_buddy(struct zone *zone, struct page *page,
+			      unsigned long pfn, int order, fpi_t fpi)
+{
+	unsigned long end = pfn + (1 << order);
+
+	VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
+	/* Caller removed page from freelist, buddy info cleared! */
+	VM_WARN_ON_ONCE(PageBuddy(page));
+
+	if (order > pageblock_order)

Review

Quoted 10 lines
(quoted diff) +10 -0
+		order = pageblock_order;
+
+	while (pfn != end) {
+		int mt = get_pfnblock_migratetype(page, pfn);
+
+		__free_one_page(page, pfn, zone, order, mt, fpi);
+		pfn += 1 << order;
+		page = pfn_to_page(pfn);
+	}
+}

Hi,

stumbling over this while digging through the code ....

Review

Quoted 10 lines
(quoted diff) +1 -3
+
      static void free_one_page(struct zone *zone, struct page *page,
      			  unsigned long pfn, unsigned int order,
      			  fpi_t fpi_flags)
      {
      	unsigned long flags;
-	int migratetype;
       	spin_lock_irqsave(&zone->lock, flags);
-	migratetype = get_pfnblock_migratetype(page, pfn);
-	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);

This change is rather undesired:

Quoted 1 line

via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.

Do you have a concrete example? PMD THP on x86_64 is pageblock_order. We do not have PMD level mTHP yet. Any other possible source?

Quoted 4 lines

What your new code will do is split this perfectly reasonable MAX_PAGE_ORDER chunk via split_large_buddy() into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.

Is there a way to avoid this and just process the whole MAX_PAGE_ORDER chunk like we used to?

Quoted 7 lines
Quoted 3 lines

Probably split_large_buddy() can check the migratetypes of the to-be-freed page, if order > pageblock_order. If all migratetypes are the same, the page can be freed at MAX_PAGE_ORDER, otherwise pageblock_order.

Thinking about this: why do we care about the migratetype?

We only have to fallback to pageblocks if any pageblock is "MIGRATE_ISOLATE" (and maybe MIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)

Quoted 8 lines

There are VM_WARN_ONCEs around *_free_list() operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.

An implicit reason is that __free_one_page() does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed via free_one_page(), where split_large_buddy() is used to work around the limitation.

Yes, I saw that change. But that can be easily identified cased by unlikely(order > MAX_PAGE_ORDER) to handle that rare case in a special way.

Quoted 1 line
Quoted 1 line

For the two memory init cases you mentioned in the other email,

maybe a new

Quoted 2 lines

fpi flag to make free_one_page() use __free_one_page() for them, since migratetypes should be the same across MAX_PAGE_ORDER range there, right?

Quoted 7 lines
Quoted 3 lines

In the context of alloc_frozen_range()/free_frozen_range() I want to free MAX_PAGE_ORDER chunks when possible, and not have some odd logic in the freeing path undo some of that effort.

Adding a pointer to that discussion:

https://lkml.kernel.org/r/[email protected]

Thanks.

So you are thinking about something like this:

mm/page_alloc.c +0 -0
index b6958333054d..3d3341dc1ad1 100644
@@ -1254,7 +1254,12 @@ static void free_one_page(struct zone *zone, struct page *page,
  	unsigned long flags;
 

mm/page_alloc.c static void free_one_page(struct zone *zone, struct page *page, · line 1264

Quoted 10 lines
mm/page_alloc.c +5 -1
  	spin_lock_irqsave(&zone->lock, flags);
-	split_large_buddy(zone, page, pfn, order, fpi_flags);
+	if (unlikely(order > MAX_PAGE_ORDER))
+		split_large_buddy(zone, page, pfn, order, fpi_flags);> +	else {
+		int migratetype = get_pfnblock_migratetype(page, pfn);
+		__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);
+	}
  	spin_unlock_irqrestore(&zone->lock, flags);
 
  	__count_vm_events(PGFREE, 1 << order);

Something in that direction, but likely something like:

if (likely(order <= pageblock_order)) {
	int migratetype = get_pfnblock_migratetype(page, pfn);
	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);
} else if (order <= MAX_PAGE_ORDER) {
	/* We might have to split when some pageblocks differ. */
	maybe_split_large_buddy(zone, page, pfn, order, fpi_flags);
} else {
	/* The buddy works in max MAX_PAGE_ORDER chunks. */
	for /* each MAX_PAGE_ORDER chunk */
		maybe_split_large_buddy(zone, page, pfn, MAX_PAGE_ORDER, fpi_flags);
		pfn += MAX_ORDER_NR_PAGES;
		page = ...
	}	
}

Whereby maybe_split_large_buddy would check if it has to do anything with the pageblocks, or whether it can just call __free_one_page(order). It would only accept order <=

In the future with free_frozen_pages(), the last else case would vanish.

Quoted 2 lines

Is it possible to have a MAX_PAGE_ORDER hugetlb folio? If no, we are good.

Hm, I wouldn't rule it out. To be future proof, we'd likely should add a way to handle it. Shouldn't be too hard and expensive: simply read all pageblock migratetypes.

I wonder if there are configs (no page isolation?) where we don't have to check anything.

Quoted 3 lines

If yes, alloc_contig_range() can change the migratetype of one of half that folio and during migration phase, that folio will be freed via __free_one_page() and causing migratetype mismatch.

Can you remind me how that happens?

Likely, we isolate a single pageblock, and shortly after we free a larger page that covers multiple pageblocks? So this could affect other alloc_contig_range() users as well, I assume?

Thanks!

Signature · 3 lines
Cheers,

David / dhildenb
reviewer
#17 lore raw

Re: [PATCH mm-unstable v2 1/3] mm/contig_alloc: support __GFP_COMP

Quoted 139 lines · David Hildenbrand
Quoted 103 lines · Zi Yan
Quoted 81 lines · David Hildenbrand
Quoted 80 lines · David Hildenbrand
Quoted 62 lines · Zi Yan
Quoted 52 lines · David Hildenbrand
Quoted 51 lines · Zi Yan
Quoted 39 lines · David Hildenbrand
Quoted 11 lines
(quoted diff) +11 -0
+/* Split a multi-block free page into its individual pageblocks. */
+static void split_large_buddy(struct zone *zone, struct page *page,
+			      unsigned long pfn, int order, fpi_t fpi)
+{
+	unsigned long end = pfn + (1 << order);
+
+	VM_WARN_ON_ONCE(!IS_ALIGNED(pfn, 1 << order));
+	/* Caller removed page from freelist, buddy info cleared! */
+	VM_WARN_ON_ONCE(PageBuddy(page));
+
+	if (order > pageblock_order)

Review

Quoted 10 lines
(quoted diff) +10 -0
+		order = pageblock_order;
+
+	while (pfn != end) {
+		int mt = get_pfnblock_migratetype(page, pfn);
+
+		__free_one_page(page, pfn, zone, order, mt, fpi);
+		pfn += 1 << order;
+		page = pfn_to_page(pfn);
+	}
+}

Hi,

stumbling over this while digging through the code ....

Review

Quoted 10 lines
(quoted diff) +1 -3
+
      static void free_one_page(struct zone *zone, struct page *page,
      			  unsigned long pfn, unsigned int order,
      			  fpi_t fpi_flags)
      {
      	unsigned long flags;
-	int migratetype;
       	spin_lock_irqsave(&zone->lock, flags);
-	migratetype = get_pfnblock_migratetype(page, pfn);
-	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);

This change is rather undesired:

Quoted 1 line

via __free_pages_core()->__free_pages_ok() we can easily end up here with order=MAX_PAGE_ORDER.

Do you have a concrete example? PMD THP on x86_64 is pageblock_order. We do not have PMD level mTHP yet. Any other possible source?

Quoted 4 lines

What your new code will do is split this perfectly reasonable MAX_PAGE_ORDER chunk via split_large_buddy() into pageblock-sized chunks, and let the buddy merging logic undo our unnecessary splitting.

Is there a way to avoid this and just process the whole MAX_PAGE_ORDER chunk like we used to?

Quoted 7 lines
Quoted 3 lines

Probably split_large_buddy() can check the migratetypes of the to-be-freed page, if order > pageblock_order. If all migratetypes are the same, the page can be freed at MAX_PAGE_ORDER, otherwise pageblock_order.

Thinking about this: why do we care about the migratetype?

We only have to fallback to pageblocks if any pageblock is "MIGRATE_ISOLATE" (and maybe MIGRATE_CMA), but not all. Otherwise, we can just ignore the migratetype (or rather overwrite it)

Quoted 8 lines

There are VM_WARN_ONCEs around *_free_list() operations to make sure page migratetype matches the migratetype of the list it is on. Ignoring migratetype would trigger these WARNs. Overwriting it would work but free page migratetype accounting needs to be taken care of.

An implicit reason is that __free_one_page() does not support >MAX_PAGE_ORDER and gigantic hugetlb folios are freed via free_one_page(), where split_large_buddy() is used to work around the limitation.

Yes, I saw that change. But that can be easily identified cased by unlikely(order > MAX_PAGE_ORDER) to handle that rare case in a special way.

Quoted 1 line
Quoted 1 line

For the two memory init cases you mentioned in the other email,

maybe a new

Quoted 2 lines

fpi flag to make free_one_page() use __free_one_page() for them, since migratetypes should be the same across MAX_PAGE_ORDER range there, right?

Quoted 7 lines
Quoted 3 lines

In the context of alloc_frozen_range()/free_frozen_range() I want to free MAX_PAGE_ORDER chunks when possible, and not have some odd logic in the freeing path undo some of that effort.

Adding a pointer to that discussion:

https://lkml.kernel.org/r/[email protected]

Thanks.

So you are thinking about something like this:

mm/page_alloc.c +0 -0
index b6958333054d..3d3341dc1ad1 100644
@@ -1254,7 +1254,12 @@ static void free_one_page(struct zone *zone, struct page *page,
  	unsigned long flags;
 

mm/page_alloc.c static void free_one_page(struct zone *zone, struct page *page, · line 1264

Quoted 10 lines
mm/page_alloc.c +5 -1
  	spin_lock_irqsave(&zone->lock, flags);
-	split_large_buddy(zone, page, pfn, order, fpi_flags);
+	if (unlikely(order > MAX_PAGE_ORDER))
+		split_large_buddy(zone, page, pfn, order, fpi_flags);> +	else {
+		int migratetype = get_pfnblock_migratetype(page, pfn);
+		__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);
+	}
  	spin_unlock_irqrestore(&zone->lock, flags);
 
  	__count_vm_events(PGFREE, 1 << order);

Something in that direction, but likely something like:

if (likely(order <= pageblock_order)) {
	int migratetype = get_pfnblock_migratetype(page, pfn);
	__free_one_page(page, pfn, zone, order, migratetype, fpi_flags);
} else if (order <= MAX_PAGE_ORDER) {
	/* We might have to split when some pageblocks differ. */
	maybe_split_large_buddy(zone, page, pfn, order, fpi_flags);
} else {
	/* The buddy works in max MAX_PAGE_ORDER chunks. */
	for /* each MAX_PAGE_ORDER chunk */
		maybe_split_large_buddy(zone, page, pfn, MAX_PAGE_ORDER, fpi_flags);
		pfn += MAX_ORDER_NR_PAGES;
		page = ...
	}	
}

Whereby maybe_split_large_buddy would check if it has to do anything with the pageblocks, or whether it can just call __free_one_page(order). It would only accept order <=

In the future with free_frozen_pages(), the last else case would vanish.

Quoted 1 line
Quoted 8 lines
Quoted 1 line

Is it possible to have a MAX_PAGE_ORDER hugetlb folio? If no, we are good.

Hm, I wouldn't rule it out. To be future proof, we'd likely should add a way to handle it. Shouldn't be too hard and expensive: simply read all pageblock migratetypes.

I wonder if there are configs (no page isolation?) where we don't have to check anything.

CONFIG_CONTIG_ALLOC? CONFIG_MEMORY_ISOLATION seems too much.

Quoted 6 lines
Quoted 4 lines

If yes, alloc_contig_range() can change the migratetype of one of half that folio and during migration phase, that folio will be freed via __free_one_page() and causing migratetype mismatch.

Can you remind me how that happens?

There is a MAX_PAGE_ORDER hugetlb folio across two pageblocks and alloc_contig_range() wants to get a range that overlaps with first pageblock of that hugetlb folio. All pageblocks are marked as isolated first during start_isolate_page_range(), then __alloc_contig_migrate_range() isolates and migrates the hugetlb folio. During migration, the original hugetlb folio is freed and since it is MAX_PAGE_ORDER and based on my code above, it is freed via __free_one_page() with two different migratetypes.

But with your maybe_split_large_buddy(), it would work.

Quoted 4 lines

Likely, we isolate a single pageblock, and shortly after we free a larger page that covers multiple pageblocks? So this could affect other alloc_contig_range() users as well, I assume?

Best Regards,
Yan, Zi

author
#18 lore raw

PATCH v2 2/3 mm/cma: add cma_{alloc,free}_folio()

In reply to Yu Zhao

With alloc_contig_range() and free_contig_range() supporting large folios, CMA can allocate and free large folios too, by cma_alloc_folio() and cma_free_folio().

---
 include/linux/cma.h | 16 +++++++++++++
 mm/cma.c            | 55 ++++++++++++++++++++++++++++++++-------------
 2 files changed, 56 insertions(+), 15 deletions(-)
include/linux/cma.h +16 -0
index 9db877506ea8..d15b64f51336 100644
@@ -52,4 +52,20 @@ extern bool cma_release(struct cma *cma, const struct page *pages, unsigned long
 extern int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data);
 
 extern void cma_reserve_pages_on_error(struct cma *cma);
+
+#ifdef CONFIG_CMA
+struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp);
+bool cma_free_folio(struct cma *cma, const struct folio *folio);
+#else
+static inline struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp)
+{
+	return NULL;
+}
+
+static inline bool cma_free_folio(struct cma *cma, const struct folio *folio)
+{
+	return false;
+}
+#endif
+
 #endif
mm/cma.c +40 -15

85 lines not on this page · raw message

Signature · 1 line
2.46.0.76.ge559c4bf1a-goog
author
#19 lore raw

Re: [PATCH mm-unstable v2 2/3] mm/cma: add cma_{alloc,free}_folio()

Quoted 94 lines · Yu Zhao

With alloc_contig_range() and free_contig_range() supporting large folios, CMA can allocate and free large folios too, by cma_alloc_folio() and cma_free_folio().

---
 include/linux/cma.h | 16 +++++++++++++
 mm/cma.c            | 55 ++++++++++++++++++++++++++++++++-------------
 2 files changed, 56 insertions(+), 15 deletions(-)
include/linux/cma.h +16 -0
index 9db877506ea8..d15b64f51336 100644
@@ -52,4 +52,20 @@ extern bool cma_release(struct cma *cma, const struct page *pages, unsigned long
 extern int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data);
 
 extern void cma_reserve_pages_on_error(struct cma *cma);
+
+#ifdef CONFIG_CMA
+struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp);
+bool cma_free_folio(struct cma *cma, const struct folio *folio);
+#else
+static inline struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp)
+{
+	return NULL;
+}
+
+static inline bool cma_free_folio(struct cma *cma, const struct folio *folio)
+{
+	return false;
+}
+#endif
+
 #endif
mm/cma.c +15 -15
index 95d6950e177b..4354823d28cf 100644
@@ -403,18 +403,8 @@ static void cma_debug_show_areas(struct cma *cma)
 	spin_unlock_irq(&cma->lock);
 }
 
-/**
- * cma_alloc() - allocate pages from contiguous area
- * @cma:   Contiguous memory region for which the allocation is performed.
- * @count: Requested number of pages.
- * @align: Requested alignment of pages (in PAGE_SIZE order).
- * @no_warn: Avoid printing message about failed allocation
- *
- * This function allocates part of contiguous memory on specific
- * contiguous memory area.
- */
-struct page *cma_alloc(struct cma *cma, unsigned long count,
-		       unsigned int align, bool no_warn)
+static struct page *__cma_alloc(struct cma *cma, unsigned long count,
+				unsigned int align, gfp_t gfp)
 {
 	unsigned long mask, offset;
 	unsigned long pfn = -1;
@@ -463,8 +453,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count,
 
 		pfn = cma->base_pfn + (bitmap_no << cma->order_per_bit);
 		mutex_lock(&cma_mutex);
-		ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA,
-				     GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0));
+		ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA, gfp);
 		mutex_unlock(&cma_mutex);
 		if (ret == 0) {
 			page = pfn_to_page(pfn);
@@ -494,7 +483,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count,
 			page_kasan_tag_reset(nth_page(page, i));
 	}
 
-	if (ret && !no_warn) {
+	if (ret && !(gfp & __GFP_NOWARN)) {
 		pr_err_ratelimited("%s: %s: alloc failed, req-size: %lu pages, ret: %d\n",
 				   __func__, cma->name, count, ret);
 		cma_debug_show_areas(cma);
@@ -513,6 +502,34 @@ struct page *cma_alloc(struct cma *cma, unsigned long count,
 	return page;
 }
 
+/**
+ * cma_alloc() - allocate pages from contiguous area
+ * @cma:   Contiguous memory region for which the allocation is performed.
+ * @count: Requested number of pages.
+ * @align: Requested alignment of pages (in PAGE_SIZE order).
+ * @no_warn: Avoid printing message about failed allocation
+ *
+ * This function allocates part of contiguous memory on specific
+ * contiguous memory area.
+ */
+struct page *cma_alloc(struct cma *cma, unsigned long count,

mm/cma.c struct page *cma_alloc(struct cma *cma, unsigned long count, · line 525

Quoted 10 lines
mm/cma.c +10 -0
+		       unsigned int align, bool no_warn)
+{
+	return __cma_alloc(cma, count, align, GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0));
+}
+
+struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp)
+{
+	struct page *page;
+
+	if (WARN_ON(!order || !(gfp | __GFP_COMP)))

And here too. Thank you.

mm/cma.c +1 -1
index 4354823d28cf..2d9fae939283 100644
@@ -522,7 +522,7 @@ struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp)
 {
 	struct page *page;
 
-	if (WARN_ON(!order || !(gfp | __GFP_COMP)))
+	if (WARN_ON(!order || !(gfp & __GFP_COMP)))
 		return NULL;
 
 	page = __cma_alloc(cma, 1 << order, order, gfp);
author
#20 lore raw

Re: [PATCH mm-unstable v2 2/3] mm/cma: add cma_{alloc,free}_folio()

Quoted 113 lines · Yu Zhao
Quoted 94 lines · Yu Zhao

With alloc_contig_range() and free_contig_range() supporting large folios, CMA can allocate and free large folios too, by cma_alloc_folio() and cma_free_folio().

---
 include/linux/cma.h | 16 +++++++++++++
 mm/cma.c            | 55 ++++++++++++++++++++++++++++++++-------------
 2 files changed, 56 insertions(+), 15 deletions(-)
include/linux/cma.h +16 -0
index 9db877506ea8..d15b64f51336 100644
@@ -52,4 +52,20 @@ extern bool cma_release(struct cma *cma, const struct page *pages, unsigned long
 extern int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data);
 
 extern void cma_reserve_pages_on_error(struct cma *cma);
+
+#ifdef CONFIG_CMA
+struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp);
+bool cma_free_folio(struct cma *cma, const struct folio *folio);
+#else
+static inline struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp)
+{
+	return NULL;
+}
+
+static inline bool cma_free_folio(struct cma *cma, const struct folio *folio)
+{
+	return false;
+}
+#endif
+
 #endif
mm/cma.c +15 -15
index 95d6950e177b..4354823d28cf 100644
@@ -403,18 +403,8 @@ static void cma_debug_show_areas(struct cma *cma)
 	spin_unlock_irq(&cma->lock);
 }
 
-/**
- * cma_alloc() - allocate pages from contiguous area
- * @cma:   Contiguous memory region for which the allocation is performed.
- * @count: Requested number of pages.
- * @align: Requested alignment of pages (in PAGE_SIZE order).
- * @no_warn: Avoid printing message about failed allocation
- *
- * This function allocates part of contiguous memory on specific
- * contiguous memory area.
- */
-struct page *cma_alloc(struct cma *cma, unsigned long count,
-		       unsigned int align, bool no_warn)
+static struct page *__cma_alloc(struct cma *cma, unsigned long count,
+				unsigned int align, gfp_t gfp)
 {
 	unsigned long mask, offset;
 	unsigned long pfn = -1;
@@ -463,8 +453,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count,
 
 		pfn = cma->base_pfn + (bitmap_no << cma->order_per_bit);
 		mutex_lock(&cma_mutex);
-		ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA,
-				     GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0));
+		ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA, gfp);
 		mutex_unlock(&cma_mutex);
 		if (ret == 0) {
 			page = pfn_to_page(pfn);
@@ -494,7 +483,7 @@ struct page *cma_alloc(struct cma *cma, unsigned long count,
 			page_kasan_tag_reset(nth_page(page, i));
 	}
 
-	if (ret && !no_warn) {
+	if (ret && !(gfp & __GFP_NOWARN)) {
 		pr_err_ratelimited("%s: %s: alloc failed, req-size: %lu pages, ret: %d\n",
 				   __func__, cma->name, count, ret);
 		cma_debug_show_areas(cma);
@@ -513,6 +502,34 @@ struct page *cma_alloc(struct cma *cma, unsigned long count,
 	return page;
 }
 
+/**
+ * cma_alloc() - allocate pages from contiguous area
+ * @cma:   Contiguous memory region for which the allocation is performed.
+ * @count: Requested number of pages.
+ * @align: Requested alignment of pages (in PAGE_SIZE order).
+ * @no_warn: Avoid printing message about failed allocation
+ *
+ * This function allocates part of contiguous memory on specific
+ * contiguous memory area.
+ */
+struct page *cma_alloc(struct cma *cma, unsigned long count,

mm/cma.c struct page *cma_alloc(struct cma *cma, unsigned long count, · line 525

Quoted 10 lines
mm/cma.c +10 -0
+		       unsigned int align, bool no_warn)
+{
+	return __cma_alloc(cma, count, align, GFP_KERNEL | (no_warn ? __GFP_NOWARN : 0));
+}
+
+struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp)
+{
+	struct page *page;
+
+	if (WARN_ON(!order || !(gfp | __GFP_COMP)))

And here too. Thank you.

mm/cma.c +0 -0
index 4354823d28cf..2d9fae939283 100644

mm/cma.c struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) · line 528

Quoted 8 lines
mm/cma.c +1 -1
@@ -522,7 +522,7 @@ struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp)
 {
 	struct page *page;
 
-	if (WARN_ON(!order || !(gfp | __GFP_COMP)))
+	if (WARN_ON(!order || !(gfp & __GFP_COMP)))
 		return NULL;
 
 	page = __cma_alloc(cma, 1 << order, order, gfp);

And the following instead of the above, if we don't want to warn the potential bug.

mm/cma.c +1 -4
index 2d9fae939283..00c30dcee200 100644
@@ -522,10 +522,7 @@ struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp)
 {
 	struct page *page;
 
-	if (WARN_ON(!order || !(gfp & __GFP_COMP)))
-		return NULL;
-
-	page = __cma_alloc(cma, 1 << order, order, gfp);
+	page = __cma_alloc(cma, 1 << order, order, gfp | __GFP_COMP);
 
 	return page ? page_folio(page) : NULL;
 }
author
#21 lore raw

PATCH v2 3/3 mm/hugetlb: use __GFP_COMP for gigantic folios

In reply to Yu Zhao

Use __GFP_COMP for gigantic folios to greatly reduce not only the amount of code but also the allocation and free time.

LOC (approximately): +60, -240

Allocate and free 500 1GB hugeTLB memory without HVO by:
time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

       Before  After
Alloc  ~13s    ~10s
Free   ~15s    <1s

The above magnitude generally holds for multiple x86 and arm64 CPU models.

---
 include/linux/hugetlb.h |   9 +-
 mm/hugetlb.c            | 293 ++++++++--------------------------------
 2 files changed, 62 insertions(+), 240 deletions(-)
include/linux/hugetlb.h +5 -4
index 3100a52ceb73..98c47c394b89 100644
@@ -896,10 +896,11 @@ static inline bool hugepage_movable_supported(struct hstate *h)
 /* Movability of hugepages depends on migration support. */
 static inline gfp_t htlb_alloc_mask(struct hstate *h)
 {
-	if (hugepage_movable_supported(h))
-		return GFP_HIGHUSER_MOVABLE;
-	else
-		return GFP_HIGHUSER;
+	gfp_t gfp = __GFP_COMP | __GFP_NOWARN;
+
+	gfp |= hugepage_movable_supported(h) ? GFP_HIGHUSER_MOVABLE : GFP_HIGHUSER;
+
+	return gfp;
 }
 
 static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask)
mm/hugetlb.c +57 -236

397 lines not on this page · raw message

Signature · 1 line
2.46.0.76.ge559c4bf1a-goog
reviewer
#22 lore raw

Re: [PATCH mm-unstable v2 3/3] mm/hugetlb: use __GFP_COMP for gigantic folios

Quoted 69 lines · Yu Zhao

Use __GFP_COMP for gigantic folios to greatly reduce not only the amount of code but also the allocation and free time.

LOC (approximately): +60, -240

Allocate and free 500 1GB hugeTLB memory without HVO by:
time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

       Before  After
Alloc  ~13s    ~10s
Free   ~15s    <1s

The above magnitude generally holds for multiple x86 and arm64 CPU models.

---
 include/linux/hugetlb.h |   9 +-
 mm/hugetlb.c            | 293 ++++++++--------------------------------
 2 files changed, 62 insertions(+), 240 deletions(-)
include/linux/hugetlb.h +5 -4
index 3100a52ceb73..98c47c394b89 100644
@@ -896,10 +896,11 @@ static inline bool hugepage_movable_supported(struct hstate *h)
 /* Movability of hugepages depends on migration support. */
 static inline gfp_t htlb_alloc_mask(struct hstate *h)
 {
-	if (hugepage_movable_supported(h))
-		return GFP_HIGHUSER_MOVABLE;
-	else
-		return GFP_HIGHUSER;
+	gfp_t gfp = __GFP_COMP | __GFP_NOWARN;
+
+	gfp |= hugepage_movable_supported(h) ? GFP_HIGHUSER_MOVABLE : GFP_HIGHUSER;
+
+	return gfp;
 }
 
 static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask)
mm/hugetlb.c +1 -10
index 71d469c8e711..efa77ce87dcc 100644
@@ -56,16 +56,6 @@ struct hstate hstates[HUGE_MAX_HSTATE];
 #ifdef CONFIG_CMA
 static struct cma *hugetlb_cma[MAX_NUMNODES];
 static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata;
-static bool hugetlb_cma_folio(struct folio *folio, unsigned int order)
-{
-	return cma_pages_valid(hugetlb_cma[folio_nid(folio)], &folio->page,
-				1 << order);
-}
-#else
-static bool hugetlb_cma_folio(struct folio *folio, unsigned int order)
-{
-	return false;
-}
 #endif
 static unsigned long hugetlb_cma_size __initdata;
 
@@ -100,6 +90,17 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
 		unsigned long start, unsigned long end);
 static struct resv_map *vma_resv_map(struct vm_area_struct *vma);
 
+static void hugetlb_free_folio(struct folio *folio)

mm/hugetlb.c static void hugetlb_unshare_pmds(struct vm_area_struct *vma, · line 103

Quoted 10 lines
mm/hugetlb.c +10 -0
+{
+#ifdef CONFIG_CMA
+	int nid = folio_nid(folio);
+
+	if (cma_free_folio(hugetlb_cma[nid], folio))
+		return;
+#endif
+	folio_put(folio);
+}
+

It seems that we no longer use free_contig_range() to free gigantic folios from alloc_contig_range(). Will it work? Or did I miss anything?

Best Regards,
Yan, Zi

author
#23 lore raw

Re: [PATCH mm-unstable v2 3/3] mm/hugetlb: use __GFP_COMP for gigantic folios

Quoted 71 lines · Zi Yan
Quoted 68 lines · Yu Zhao

Use __GFP_COMP for gigantic folios to greatly reduce not only the amount of code but also the allocation and free time.

LOC (approximately): +60, -240

Allocate and free 500 1GB hugeTLB memory without HVO by:
time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

       Before  After
Alloc  ~13s    ~10s
Free   ~15s    <1s

The above magnitude generally holds for multiple x86 and arm64 CPU models.

---
 include/linux/hugetlb.h |   9 +-
 mm/hugetlb.c            | 293 ++++++++--------------------------------
 2 files changed, 62 insertions(+), 240 deletions(-)
include/linux/hugetlb.h +5 -4
index 3100a52ceb73..98c47c394b89 100644
@@ -896,10 +896,11 @@ static inline bool hugepage_movable_supported(struct hstate *h)
 /* Movability of hugepages depends on migration support. */
 static inline gfp_t htlb_alloc_mask(struct hstate *h)
 {
-     if (hugepage_movable_supported(h))
-             return GFP_HIGHUSER_MOVABLE;
-     else
-             return GFP_HIGHUSER;
+     gfp_t gfp = __GFP_COMP | __GFP_NOWARN;
+
+     gfp |= hugepage_movable_supported(h) ? GFP_HIGHUSER_MOVABLE : GFP_HIGHUSER;
+
+     return gfp;
 }
 
 static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask)
mm/hugetlb.c +0 -10
index 71d469c8e711..efa77ce87dcc 100644
@@ -56,16 +56,6 @@ struct hstate hstates[HUGE_MAX_HSTATE];
 #ifdef CONFIG_CMA
 static struct cma *hugetlb_cma[MAX_NUMNODES];
 static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata;
-static bool hugetlb_cma_folio(struct folio *folio, unsigned int order)
-{
-     return cma_pages_valid(hugetlb_cma[folio_nid(folio)], &folio->page,
-                             1 << order);
-}
-#else
-static bool hugetlb_cma_folio(struct folio *folio, unsigned int order)
-{
-     return false;
-}
 #endif
 static unsigned long hugetlb_cma_size __initdata;
 
@@ -100,6 +90,17 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
              unsigned long start, unsigned long end);
 static struct resv_map *vma_resv_map(struct vm_area_struct *vma);
 
Quoted 14 lines

mm/hugetlb.c static void hugetlb_unshare_pmds(struct vm_area_struct *vma, · line 103

Quoted 11 lines
mm/hugetlb.c +11 -0
+static void hugetlb_free_folio(struct folio *folio)
+{
+#ifdef CONFIG_CMA
+     int nid = folio_nid(folio);
+
+     if (cma_free_folio(hugetlb_cma[nid], folio))
+             return;
+#endif
+     folio_put(folio);
+}
+

It seems that we no longer use free_contig_range() to free gigantic folios from alloc_contig_range().

We switched to two pairs of extern (to the allocator) APIs in this patch:
folio_alloc_gigantic()
folio_put()
and
cma_alloc_folio()
cma_free_folio()

Quoted 1 line

Will it work? Or did I miss anything?

alloc_contig_range and free_contig_range() also works with __GFP_COMP / large folios, but this pair is internal (to the allocator) and shouldn't be used directly except to implement external APIs like above.

reviewer Acked-by
#24 lore raw

Re: [PATCH mm-unstable v2 3/3] mm/hugetlb: use __GFP_COMP for gigantic folios

Quoted 94 lines · Yu Zhao
Quoted 71 lines · Zi Yan
Quoted 68 lines · Yu Zhao

Use __GFP_COMP for gigantic folios to greatly reduce not only the amount of code but also the allocation and free time.

LOC (approximately): +60, -240

Allocate and free 500 1GB hugeTLB memory without HVO by:
time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

       Before  After
Alloc  ~13s    ~10s
Free   ~15s    <1s

The above magnitude generally holds for multiple x86 and arm64 CPU models.

---
 include/linux/hugetlb.h |   9 +-
 mm/hugetlb.c            | 293 ++++++++--------------------------------
 2 files changed, 62 insertions(+), 240 deletions(-)
include/linux/hugetlb.h +5 -4
index 3100a52ceb73..98c47c394b89 100644
@@ -896,10 +896,11 @@ static inline bool hugepage_movable_supported(struct hstate *h)
 /* Movability of hugepages depends on migration support. */
 static inline gfp_t htlb_alloc_mask(struct hstate *h)
 {
-     if (hugepage_movable_supported(h))
-             return GFP_HIGHUSER_MOVABLE;
-     else
-             return GFP_HIGHUSER;
+     gfp_t gfp = __GFP_COMP | __GFP_NOWARN;
+
+     gfp |= hugepage_movable_supported(h) ? GFP_HIGHUSER_MOVABLE : GFP_HIGHUSER;
+
+     return gfp;
 }
 
 static inline gfp_t htlb_modify_alloc_mask(struct hstate *h, gfp_t gfp_mask)
mm/hugetlb.c +0 -10
index 71d469c8e711..efa77ce87dcc 100644
@@ -56,16 +56,6 @@ struct hstate hstates[HUGE_MAX_HSTATE];
 #ifdef CONFIG_CMA
 static struct cma *hugetlb_cma[MAX_NUMNODES];
 static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata;
-static bool hugetlb_cma_folio(struct folio *folio, unsigned int order)
-{
-     return cma_pages_valid(hugetlb_cma[folio_nid(folio)], &folio->page,
-                             1 << order);
-}
-#else
-static bool hugetlb_cma_folio(struct folio *folio, unsigned int order)
-{
-     return false;
-}
 #endif
 static unsigned long hugetlb_cma_size __initdata;
 
@@ -100,6 +90,17 @@ static void hugetlb_unshare_pmds(struct vm_area_struct *vma,
              unsigned long start, unsigned long end);
 static struct resv_map *vma_resv_map(struct vm_area_struct *vma);
 
Quoted 14 lines

mm/hugetlb.c static void hugetlb_unshare_pmds(struct vm_area_struct *vma, · line 103

Quoted 11 lines
mm/hugetlb.c +11 -0
+static void hugetlb_free_folio(struct folio *folio)
+{
+#ifdef CONFIG_CMA
+     int nid = folio_nid(folio);
+
+     if (cma_free_folio(hugetlb_cma[nid], folio))
+             return;
+#endif
+     folio_put(folio);
+}
+

It seems that we no longer use free_contig_range() to free gigantic folios from alloc_contig_range().

We switched to two pairs of extern (to the allocator) APIs in this patch:
folio_alloc_gigantic()
folio_put()
and
cma_alloc_folio()
cma_free_folio()

Quoted 6 lines
Quoted 1 line

Will it work? Or did I miss anything?

alloc_contig_range and free_contig_range() also works with __GFP_COMP / large folios, but this pair is internal (to the allocator) and shouldn't be used directly except to implement external APIs like above.

Oh, I missed split_large_buddy() addition to free_one_page() in patch 1. That handles >MAX_ORDER page freeing. It also makes sure the pageblocks of a free cross-pageblock gigantic hugetlb page go back to the right free lists when each pageblock might have different migratetypes, since the page is freed at pageblock granularity.

The whole series looks good to me.

Best Regards,
Yan, Zi

#25 lore raw
In reply to Yu Zhao
Quoted 6 lines · Yu Zhao

Use __GFP_COMP for gigantic folios can greatly reduce not only the amount of code but also the allocation and free time.

Approximate LOC to mm/hugetlb.c: +60, -240

Allocate and free 500 1GB hugeTLB memory without HVO by:

Do you also have numbers with HVO enabled ?

Quoted 7 lines

time echo 500 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages time echo 0 >/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages

Before After

Alloc  ~13s    ~10s
Free   ~15s    <1s

Thanks,

-jane