mm/page_alloc: add one PCP list for THP
Subsystems (4)
PATCH 1/1 mm/page_alloc: add one PCP list for THP
From: yangge <[email protected]>
Since commit 5d0a661d808f ("mm/page_alloc: use only one PCP list for THP-sized allocations") no longer differentiates the migration type of pages in THP-sized PCP list, it's possible that non-movable allocation requests may get a CMA page from the list, in some cases, it's not acceptable.
If a large number of CMA memory are configured in system (for example, the CMA memory accounts for 50% of the system memory), starting a virtual machine with device passthrough will get stuck. During starting the virtual machine, it will call pin_user_pages_remote(..., FOLL_LONGTERM, ...) to pin memory. Normally if a page is present and in CMA area, pin_user_pages_remote() will migrate the page from CMA area to non-CMA area because of FOLL_LONGTERM flag. But if non-movable allocation requests return CMA memory, migrate_longterm_unpinnable_pages() will migrate a CMA page to another CMA page, which will fail to pass the check in check_and_migrate_movable_pages() and cause migration endless.
Call trace:
pin_user_pages_remote
--__gup_longterm_locked // endless loops in this function ----_get_user_pages_locked ----check_and_migrate_movable_pages ------migrate_longterm_unpinnable_pages --------alloc_migration_target
To fix the problem above, we add one PCP list for THP, which will not introduce a new cacheline. THP will have 2 PCP lists, one PCP list is used by MOVABLE allocation, and the other PCP list is used by UNMOVABLE allocation. MOVABLE allocation contains GPF_MOVABLE, and UNMOVABLE allocation contains GFP_UNMOVABLE and GFP_RECLAIMABLE.
--- include/linux/mmzone.h | 9 ++++----- mm/page_alloc.c | 9 +++++++-- 2 files changed, 11 insertions(+), 7 deletions(-)
include/linux/mmzone.h +4 -5
@@ -656,13 +656,12 @@ enum zone_watermarks { };/* -* One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. One additional list -* for THP which will usually be GFP_MOVABLE. Even if it is another type, -* it should not contribute to serious fragmentation causing THP allocation -* failures. +* One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. Two additional lists +* are added for THP. One PCP list is used by GPF_MOVABLE, and the other PCP list +* is used by GFP_UNMOVABLE and GFP_RECLAIMABLE. */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_PCP_THP 1 +#define NR_PCP_THP 2#else #define NR_PCP_THP 0#endif
mm/page_alloc.c +7 -2
@@ -504,10 +504,15 @@ static void bad_page(struct page *page, const char *reason)static inline unsigned int order_to_pindex (int migratetype,int order) { +bool __maybe_unused movable; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE if (order >PAGE_ALLOC_COSTLY_ORDER ) {VM_BUG_ON (order !=HPAGE_PMD_ORDER ); -return NR_LOWORDER_PCP_LISTS ; + + movable = migratetype ==MIGRATE_MOVABLE ; + +return NR_LOWORDER_PCP_LISTS + movable; }#else VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER );
@@ -521,7 +526,7 @@ static inline int pindex_to_order(unsigned int pindex)int order = pindex /MIGRATE_PCPTYPES ;#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (pindex ==NR_LOWORDER_PCP_LISTS ) +if (order >PAGE_ALLOC_COSTLY_ORDER ) order =HPAGE_PMD_ORDER ;#else VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER );
Signature · 1 line
2.7.4
+Mel, the original author of commit 5d0a661d808f
('mm/page_alloc: use only one PCP list..."
Quoted 20 lines · On Wed, Jun 19, 2024 at 9:48 PM <[email protected]> wrote:
From: yangge <[email protected]>
Since commit
5d0a661d808f("mm/page_alloc: use only one PCP list for THP-sized allocations") no longer differentiates the migration type of pages in THP-sized PCP list, it's possible that non-movable allocation requests may get a CMA page from the list, in some cases, it's not acceptable.If a large number of CMA memory are configured in system (for example, the CMA memory accounts for 50% of the system memory), starting a virtual machine with device passthrough will get stuck. During starting the virtual machine, it will call
pin_user_pages_remote(..., FOLL_LONGTERM, ...)to pin memory. Normally if a page is present and in CMA area,pin_user_pages_remote()will migrate the page from CMA area to non-CMA area because ofFOLL_LONGTERMflag. But if non-movable allocation requests return CMA memory,migrate_longterm_unpinnable_pages()will migrate a CMA page to another CMA page, which will fail to pass the check incheck_and_migrate_movable_pages()and cause migration endless.
Quoted 8 lines
Call trace:
pin_user_pages_remote
--__gup_longterm_locked // endless loops in this function ----_get_user_pages_locked ----check_and_migrate_movable_pages ------migrate_longterm_unpinnable_pages --------alloc_migration_target
Please also describe its potential negative impact to cma_alloc().
Quoted 2 lines
To fix the problem above, we add one PCP list for THP, which will not introduce a new cacheline. THP will have 2 PCP lists, one PCP
not introduce a new cacheline for struct per_cpu_pages.
Quoted 5 lines
list is used by MOVABLE allocation, and the other PCP list is used by UNMOVABLE allocation. MOVABLE allocation contains
GPF_MOVABLE, and UNMOVABLE allocation containsGFP_UNMOVABLEandGFP_RECLAIMABLE.
no this tag.
Quoted 1 line
Fixes 5d0a661d808f ("mm/page_alloc: use only one PCP list for THP-sized allocations")
Quoted 50 lines
Signed-off-by yangge <[email protected]>--- include/linux/mmzone.h | 9 ++++----- mm/page_alloc.c | 9 +++++++-- 2 files changed, 11 insertions(+), 7 deletions(-)include/linux/mmzone.h +4
-5@@ -656,13 +656,12 @@ enum zone_watermarks { };/* -* One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. One additional list -* for THP which will usually be GFP_MOVABLE. Even if it is another type, -* it should not contribute to serious fragmentation causing THP allocation -* failures. +* One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. Two additional lists +* are added for THP. One PCP list is used by GPF_MOVABLE, and the other PCP list +* is used by GFP_UNMOVABLE and GFP_RECLAIMABLE. */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_PCP_THP 1 +#define NR_PCP_THP 2#else #define NR_PCP_THP 0#endif mm/page_alloc.c +6
-1@@ -504,10 +504,15 @@ static void bad_page(struct page *page, const char *reason)static inline unsigned int order_to_pindex (int migratetype,int order) { +bool __maybe_unused movable; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE if (order >PAGE_ALLOC_COSTLY_ORDER ) {VM_BUG_ON (order !=HPAGE_PMD_ORDER ); -return NR_LOWORDER_PCP_LISTS ; + + movable = migratetype ==MIGRATE_MOVABLE ; + +return NR_LOWORDER_PCP_LISTS + movable; }#else VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER );
mm/page_alloc.c static inline int pindex_to_order(unsigned int pindex) · line 529
Quoted 5 lines
mm/page_alloc.c +1
-1@@ -521,7 +526,7 @@ static inline int pindex_to_order(unsigned int pindex)int order = pindex /MIGRATE_PCPTYPES ;#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (pindex ==NR_LOWORDER_PCP_LISTS ) +if (order >PAGE_ALLOC_COSTLY_ORDER )
pindex >= NR_LOWORDER_PCP_LISTS
Quoted 6 lines
mm/page_alloc.c +0
-1order =HPAGE_PMD_ORDER ;#else VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); --2.7.4
Thanks
Barry
Quoted 45 lines · Barry Song
+Mel, the original author of commit
5d0a661d808f
('mm/page_alloc: use only one PCP list..."Quoted 20 lines · On Wed, Jun 19, 2024 at 9:48 PM <[email protected]> wrote:
From: yangge <[email protected]>
Since commit
5d0a661d808f("mm/page_alloc: use only one PCP list for THP-sized allocations") no longer differentiates the migration type of pages in THP-sized PCP list, it's possible that non-movable allocation requests may get a CMA page from the list, in some cases, it's not acceptable.If a large number of CMA memory are configured in system (for example, the CMA memory accounts for 50% of the system memory), starting a virtual machine with device passthrough will get stuck. During starting the virtual machine, it will call
pin_user_pages_remote(..., FOLL_LONGTERM, ...)to pin memory. Normally if a page is present and in CMA area,pin_user_pages_remote()will migrate the page from CMA area to non-CMA area because ofFOLL_LONGTERMflag. But if non-movable allocation requests return CMA memory,migrate_longterm_unpinnable_pages()will migrate a CMA page to another CMA page, which will fail to pass the check incheck_and_migrate_movable_pages()and cause migration endless.Quoted 8 lines
Call trace:
pin_user_pages_remote
--__gup_longterm_locked // endless loops in this function ----_get_user_pages_locked ----check_and_migrate_movable_pages ------migrate_longterm_unpinnable_pages --------alloc_migration_targetPlease also describe its potential negative impact to
cma_alloc().Quoted 2 lines
To fix the problem above, we add one PCP list for THP, which will not introduce a new cacheline. THP will have 2 PCP lists, one PCP
not introduce a new cacheline for struct
per_cpu_pages.Quoted 4 lines
list is used by MOVABLE allocation, and the other PCP list is used by UNMOVABLE allocation. MOVABLE allocation contains
GPF_MOVABLE, and UNMOVABLE allocation containsGFP_UNMOVABLEandGFP_RECLAIMABLE.
Quoted 8 lines
Quoted 1 line
no this tag.
Quoted 1 line
Fixes 5d0a661d808f ("mm/page_alloc: use only one PCP list for THP-sized allocations")Cc <[email protected]> ?
Thanks, I will prepare the next version.
Quoted 67 lines
Quoted 50 lines
Signed-off-by yangge <[email protected]>--- include/linux/mmzone.h | 9 ++++----- mm/page_alloc.c | 9 +++++++-- 2 files changed, 11 insertions(+), 7 deletions(-)include/linux/mmzone.h +4
-5@@ -656,13 +656,12 @@ enum zone_watermarks { };/* -* One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. One additional list -* for THP which will usually be GFP_MOVABLE. Even if it is another type, -* it should not contribute to serious fragmentation causing THP allocation -* failures. +* One per migratetype for each PAGE_ALLOC_COSTLY_ORDER. Two additional lists +* are added for THP. One PCP list is used by GPF_MOVABLE, and the other PCP list +* is used by GFP_UNMOVABLE and GFP_RECLAIMABLE. */ #ifdef CONFIG_TRANSPARENT_HUGEPAGE -#define NR_PCP_THP 1 +#define NR_PCP_THP 2#else #define NR_PCP_THP 0#endif mm/page_alloc.c +6
-1@@ -504,10 +504,15 @@ static void bad_page(struct page *page, const char *reason)static inline unsigned int order_to_pindex (int migratetype,int order) { +bool __maybe_unused movable; +#ifdef CONFIG_TRANSPARENT_HUGEPAGE if (order >PAGE_ALLOC_COSTLY_ORDER ) {VM_BUG_ON (order !=HPAGE_PMD_ORDER ); -return NR_LOWORDER_PCP_LISTS ; + + movable = migratetype ==MIGRATE_MOVABLE ; + +return NR_LOWORDER_PCP_LISTS + movable; }#else VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER );mm/page_alloc.c static inline int pindex_to_order(unsigned int pindex) · line 529
Quoted 5 lines
mm/page_alloc.c +1
-1@@ -521,7 +526,7 @@ static inline int pindex_to_order(unsigned int pindex)int order = pindex /MIGRATE_PCPTYPES ;#ifdef CONFIG_TRANSPARENT_HUGEPAGE -if (pindex ==NR_LOWORDER_PCP_LISTS ) +if (order >PAGE_ALLOC_COSTLY_ORDER )pindex >=
NR_LOWORDER_PCP_LISTSQuoted 6 lines
mm/page_alloc.c +0
-1order =HPAGE_PMD_ORDER ;#else VM_BUG_ON (order >PAGE_ALLOC_COSTLY_ORDER ); --2.7.4
Thanks
Barry