Skip to contentSkip to conversation

mm/page_alloc: add one PCP list for THP

PATCH In review [email protected] 2 participants 3 messages Jun 19, 2024

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/page_alloc: add one PCP list for THP

From: yangge <[email protected]>

Since commit 5d0a661d808f ("mm/page_alloc: use only one PCP list for THP-sized allocations") no longer differentiates the migration type of pages in THP-sized PCP list, it's possible that non-movable allocation requests may get a CMA page from the list, in some cases, it's not acceptable.

If a large number of CMA memory are configured in system (for example, the CMA memory accounts for 50% of the system memory), starting a virtual machine with device passthrough will get stuck. During starting the virtual machine, it will call pin_user_pages_remote(..., FOLL_LONGTERM, ...) to pin memory. Normally if a page is present and in CMA area, pin_user_pages_remote() will migrate the page from CMA area to non-CMA area because of FOLL_LONGTERM flag. But if non-movable allocation requests return CMA memory, migrate_longterm_unpinnable_pages() will migrate a CMA page to another CMA page, which will fail to pass the check in check_and_migrate_movable_pages() and cause migration endless.

Call trace:

pin_user_pages_remote
--__gup_longterm_locked // endless loops in this function ----_get_user_pages_locked ----check_and_migrate_movable_pages ------migrate_longterm_unpinnable_pages --------alloc_migration_target

To fix the problem above, we add one PCP list for THP, which will not introduce a new cacheline. THP will have 2 PCP lists, one PCP list is used by MOVABLE allocation, and the other PCP list is used by UNMOVABLE allocation. MOVABLE allocation contains GPF_MOVABLE, and UNMOVABLE allocation contains GFP_UNMOVABLE and GFP_RECLAIMABLE.

Fixes 5d0a661d808f ("mm/page_alloc: use only one PCP list for THP-sized allocations")
---
 include/linux/mmzone.h | 9 ++++-----
 mm/page_alloc.c        | 9 +++++++--
 2 files changed, 11 insertions(+), 7 deletions(-)
include/linux/mmzone.h +4 -5
index b7546dd..cb7f265 100644
@@ -656,13 +656,12 @@ enum zone_watermarks {
 };
 
 /*
- * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. One additional list
- * for THP which will usually be GFP_MOVABLE. Even if it is another type,
- * it should not contribute to serious fragmentation causing THP allocation
- * failures.
+ * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. Two additional lists
+ * are added for THP. One PCP list is used by GPF_MOVABLE, and the other PCP list
+ * is used by GFP_UNMOVABLE and GFP_RECLAIMABLE.
  */
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-#define NR_PCP_THP 1
+#define NR_PCP_THP 2
 #else
 #define NR_PCP_THP 0
 #endif
mm/page_alloc.c +7 -2
index 8f416a0..0ecbde3 100644
@@ -504,10 +504,15 @@ static void bad_page(struct page *page, const char *reason)
 
 static inline unsigned int order_to_pindex(int migratetype, int order)
 {
+	bool __maybe_unused movable;
+
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	if (order > PAGE_ALLOC_COSTLY_ORDER) {
 		VM_BUG_ON(order != HPAGE_PMD_ORDER);
-		return NR_LOWORDER_PCP_LISTS;
+
+		movable = migratetype == MIGRATE_MOVABLE;
+
+		return NR_LOWORDER_PCP_LISTS + movable;
 	}
 #else
 	VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);
@@ -521,7 +526,7 @@ static inline int pindex_to_order(unsigned int pindex)
 	int order = pindex / MIGRATE_PCPTYPES;
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (pindex == NR_LOWORDER_PCP_LISTS)
+	if (order > PAGE_ALLOC_COSTLY_ORDER)
 		order = HPAGE_PMD_ORDER;
 #else
 	VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);
Signature · 1 line
2.7.4
#1 lore raw

+Mel, the original author of commit 5d0a661d808f
('mm/page_alloc: use only one PCP list..."

Quoted 20 lines · On Wed, Jun 19, 2024 at 9:48 PM <[email protected]> wrote:

From: yangge <[email protected]>

Since commit 5d0a661d808f ("mm/page_alloc: use only one PCP list for THP-sized allocations") no longer differentiates the migration type of pages in THP-sized PCP list, it's possible that non-movable allocation requests may get a CMA page from the list, in some cases, it's not acceptable.

If a large number of CMA memory are configured in system (for example, the CMA memory accounts for 50% of the system memory), starting a virtual machine with device passthrough will get stuck. During starting the virtual machine, it will call pin_user_pages_remote(..., FOLL_LONGTERM, ...) to pin memory. Normally if a page is present and in CMA area, pin_user_pages_remote() will migrate the page from CMA area to non-CMA area because of FOLL_LONGTERM flag. But if non-movable allocation requests return CMA memory, migrate_longterm_unpinnable_pages() will migrate a CMA page to another CMA page, which will fail to pass the check in check_and_migrate_movable_pages() and cause migration endless.

Quoted 8 lines
Call trace:

pin_user_pages_remote
--__gup_longterm_locked // endless loops in this function ----_get_user_pages_locked ----check_and_migrate_movable_pages ------migrate_longterm_unpinnable_pages --------alloc_migration_target

Please also describe its potential negative impact to cma_alloc().

Quoted 2 lines

To fix the problem above, we add one PCP list for THP, which will not introduce a new cacheline. THP will have 2 PCP lists, one PCP

not introduce a new cacheline for struct per_cpu_pages.

Quoted 5 lines

list is used by MOVABLE allocation, and the other PCP list is used by UNMOVABLE allocation. MOVABLE allocation contains GPF_MOVABLE, and UNMOVABLE allocation contains GFP_UNMOVABLE and GFP_RECLAIMABLE.

no this tag.

Quoted 1 line
Fixes 5d0a661d808f ("mm/page_alloc: use only one PCP list for THP-sized allocations")
Quoted 50 lines
---
 include/linux/mmzone.h | 9 ++++-----
 mm/page_alloc.c        | 9 +++++++--
 2 files changed, 11 insertions(+), 7 deletions(-)
include/linux/mmzone.h +4 -5
index b7546dd..cb7f265 100644
@@ -656,13 +656,12 @@ enum zone_watermarks {
 };
 
 /*
- * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. One additional list
- * for THP which will usually be GFP_MOVABLE. Even if it is another type,
- * it should not contribute to serious fragmentation causing THP allocation
- * failures.
+ * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. Two additional lists
+ * are added for THP. One PCP list is used by GPF_MOVABLE, and the other PCP list
+ * is used by GFP_UNMOVABLE and GFP_RECLAIMABLE.
  */
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-#define NR_PCP_THP 1
+#define NR_PCP_THP 2
 #else
 #define NR_PCP_THP 0
 #endif
mm/page_alloc.c +6 -1
index 8f416a0..0ecbde3 100644
@@ -504,10 +504,15 @@ static void bad_page(struct page *page, const char *reason)
 
 static inline unsigned int order_to_pindex(int migratetype, int order)
 {
+       bool __maybe_unused movable;
+
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
        if (order > PAGE_ALLOC_COSTLY_ORDER) {
                VM_BUG_ON(order != HPAGE_PMD_ORDER);
-               return NR_LOWORDER_PCP_LISTS;
+
+               movable = migratetype == MIGRATE_MOVABLE;
+
+               return NR_LOWORDER_PCP_LISTS + movable;
        }
 #else
        VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);

mm/page_alloc.c static inline int pindex_to_order(unsigned int pindex) · line 529

Quoted 5 lines
mm/page_alloc.c +1 -1
@@ -521,7 +526,7 @@ static inline int pindex_to_order(unsigned int pindex)
        int order = pindex / MIGRATE_PCPTYPES;
 
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-       if (pindex == NR_LOWORDER_PCP_LISTS)
+       if (order > PAGE_ALLOC_COSTLY_ORDER)

pindex >= NR_LOWORDER_PCP_LISTS

Quoted 6 lines
mm/page_alloc.c +0 -1
                order = HPAGE_PMD_ORDER;
 #else
        VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);
--

2.7.4

Thanks
Barry

author
#2 lore raw
Quoted 45 lines · Barry Song

+Mel, the original author of commit 5d0a661d808f
('mm/page_alloc: use only one PCP list..."

Quoted 20 lines · On Wed, Jun 19, 2024 at 9:48 PM <[email protected]> wrote:

From: yangge <[email protected]>

Since commit 5d0a661d808f ("mm/page_alloc: use only one PCP list for THP-sized allocations") no longer differentiates the migration type of pages in THP-sized PCP list, it's possible that non-movable allocation requests may get a CMA page from the list, in some cases, it's not acceptable.

If a large number of CMA memory are configured in system (for example, the CMA memory accounts for 50% of the system memory), starting a virtual machine with device passthrough will get stuck. During starting the virtual machine, it will call pin_user_pages_remote(..., FOLL_LONGTERM, ...) to pin memory. Normally if a page is present and in CMA area, pin_user_pages_remote() will migrate the page from CMA area to non-CMA area because of FOLL_LONGTERM flag. But if non-movable allocation requests return CMA memory, migrate_longterm_unpinnable_pages() will migrate a CMA page to another CMA page, which will fail to pass the check in check_and_migrate_movable_pages() and cause migration endless.

Quoted 8 lines
Call trace:

pin_user_pages_remote
--__gup_longterm_locked // endless loops in this function ----_get_user_pages_locked ----check_and_migrate_movable_pages ------migrate_longterm_unpinnable_pages --------alloc_migration_target

Please also describe its potential negative impact to cma_alloc().

Quoted 2 lines

To fix the problem above, we add one PCP list for THP, which will not introduce a new cacheline. THP will have 2 PCP lists, one PCP

not introduce a new cacheline for struct per_cpu_pages.

Quoted 4 lines

list is used by MOVABLE allocation, and the other PCP list is used by UNMOVABLE allocation. MOVABLE allocation contains GPF_MOVABLE, and UNMOVABLE allocation contains GFP_UNMOVABLE and GFP_RECLAIMABLE.

Quoted 8 lines
Quoted 1 line

no this tag.

Quoted 1 line
Fixes 5d0a661d808f ("mm/page_alloc: use only one PCP list for THP-sized allocations")

Thanks, I will prepare the next version.

Quoted 67 lines
Quoted 50 lines
---
  include/linux/mmzone.h | 9 ++++-----
  mm/page_alloc.c        | 9 +++++++--
  2 files changed, 11 insertions(+), 7 deletions(-)
include/linux/mmzone.h +4 -5
index b7546dd..cb7f265 100644
@@ -656,13 +656,12 @@ enum zone_watermarks {
  };
 
  /*
- * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. One additional list
- * for THP which will usually be GFP_MOVABLE. Even if it is another type,
- * it should not contribute to serious fragmentation causing THP allocation
- * failures.
+ * One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. Two additional lists
+ * are added for THP. One PCP list is used by GPF_MOVABLE, and the other PCP list
+ * is used by GFP_UNMOVABLE and GFP_RECLAIMABLE.
   */
  #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-#define NR_PCP_THP 1
+#define NR_PCP_THP 2
  #else
  #define NR_PCP_THP 0
  #endif
mm/page_alloc.c +6 -1
index 8f416a0..0ecbde3 100644
@@ -504,10 +504,15 @@ static void bad_page(struct page *page, const char *reason)
 
  static inline unsigned int order_to_pindex(int migratetype, int order)
  {
+       bool __maybe_unused movable;
+
  #ifdef CONFIG_TRANSPARENT_HUGEPAGE
         if (order > PAGE_ALLOC_COSTLY_ORDER) {
                 VM_BUG_ON(order != HPAGE_PMD_ORDER);
-               return NR_LOWORDER_PCP_LISTS;
+
+               movable = migratetype == MIGRATE_MOVABLE;
+
+               return NR_LOWORDER_PCP_LISTS + movable;
         }
  #else
         VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);

mm/page_alloc.c static inline int pindex_to_order(unsigned int pindex) · line 529

Quoted 5 lines
mm/page_alloc.c +1 -1
@@ -521,7 +526,7 @@ static inline int pindex_to_order(unsigned int pindex)
         int order = pindex / MIGRATE_PCPTYPES;
 
  #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-       if (pindex == NR_LOWORDER_PCP_LISTS)
+       if (order > PAGE_ALLOC_COSTLY_ORDER)

pindex >= NR_LOWORDER_PCP_LISTS

Quoted 6 lines
mm/page_alloc.c +0 -1
                 order = HPAGE_PMD_ORDER;
  #else
         VM_BUG_ON(order > PAGE_ALLOC_COSTLY_ORDER);
--

2.7.4

Thanks
Barry