mm/compaction: Fix bug in hugetlb handling pathway
Subsystems (3)
PATCH 1/1 mm/compaction: Fix bug in hugetlb handling pathway
The compaction code doesn't take references on pages until we're certain we should attempt to handle it.
In the hugetlb case, isolate_or_dissolve_huge_page() may return -EBUSY without taking a reference to the folio associated with our pfn. If our folio's refcount drops to 0, compound_nr() becomes unpredictable, making low_pfn and nr_scanned unreliable.
The user-visible effect is minimal - this should rarely happen (if ever).
Fix this by storing the folio statistics earlier on the stack (just like the THP and Buddy cases).
Also revert commit 66fe1cf7f581 ("mm: compaction: use helper compound_nr in isolate_migratepages_block")
to make backporting easier.
--- mm/compaction.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
mm/compaction.c +3 -3
@@ -981,13 +981,13 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, }if (PageHuge (page)) { +const unsigned int order =compound_order (page);/* * skip hugetlbfs if we are not compacting for pages * bigger than its order. THPs and other compound pages * are handled below. */ if (!cc->alloc_contig) { -const unsigned int order =compound_order (page);if (order <=MAX_PAGE_ORDER ) { low_pfn += (1UL << order) -1 ;
@@ -1011,8 +1011,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,/* Do not report -EBUSY down the chain */ if (ret == -EBUSY ) ret =0 ; - low_pfn +=compound_nr (page) -1 ; - nr_scanned +=compound_nr (page) -1 ; + low_pfn += (1UL << order) -1 ; + nr_scanned += (1UL << order) -1 ; goto isolate_fail; }
Signature · 1 line
2.48.1
PATCH 1/1 mm/compaction: Use folio in hugetlb pathway
Use a folio in the hugetlb pathway during the compaction migrate-able pageblock scan.
This removes a call to compound_head().
--- include/linux/hugetlb.h | 4 ++-- mm/compaction.c | 8 ++++---- mm/hugetlb.c | 3 +-- 3 files changed, 7 insertions(+), 8 deletions(-)
include/linux/hugetlb.h +2 -2
@@ -695,7 +695,7 @@ struct huge_bootmem_page {bool hugetlb_bootmem_page_zones_valid (int nid,struct huge_bootmem_page *m); -int isolate_or_dissolve_huge_page (struct page *page,struct list_head *list); +int isolate_or_dissolve_huge_folio (struct folio *folio,struct list_head *list);int replace_free_hugepage_folios (unsigned long start_pfn,unsigned long end_pfn);void wait_for_freed_hugetlb_folios (void ); struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
@@ -1083,7 +1083,7 @@ static inline struct folio *filemap_lock_hugetlb_folio(struct hstate *h,return NULL ; } -static inline int isolate_or_dissolve_huge_page (struct page *page, +static inline int isolate_or_dissolve_huge_folio (struct folio *folio,struct list_head *list) {return -ENOMEM ;
mm/compaction.c +4 -4
@@ -1001,10 +1001,11 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, locked =NULL ; } - ret =isolate_or_dissolve_huge_page (page, &cc->migratepages); + folio =page_folio (page); + ret =isolate_or_dissolve_huge_folio (folio, &cc->migratepages);/* -* Fail isolation in case isolate_or_dissolve_huge_page() +* Fail isolation in case isolate_or_dissolve_huge_folio() * reports an error. In case of -ENOMEM, abort right away. */ if (ret <0 ) {
@@ -1016,12 +1017,11 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, goto isolate_fail; } -if (PageHuge (page)) { +if (folio_test_hugetlb (folio)) {/* * Hugepage was successfully isolated and placed * on the cc->migratepages list. */ - folio =page_folio (page); low_pfn +=folio_nr_pages (folio) -1 ; goto isolate_success_no_list; }
mm/hugetlb.c +1 -2
@@ -2896,10 +2896,9 @@ static int alloc_and_dissolve_hugetlb_folio(struct hstate *h,return ret; } -int isolate_or_dissolve_huge_page (struct page *page,struct list_head *list) +int isolate_or_dissolve_huge_folio (struct folio *folio,struct list_head *list) {struct hstate *h; -struct folio *folio =page_folio (page);int ret = -EBUSY ;/*
Signature · 1 line
2.48.1
Re: [PATCH] mm/compaction: Use folio in hugetlb pathway
Quoted 6 lines · Vishal Moola (Oracle)
Use a folio in the hugetlb pathway during the compaction migrate-able pageblock scan.
This removes a call to
compound_head().Signed-off-by Vishal Moola (Oracle) <[email protected]>
Signature · 2 lines
Oscar Salvador SUSE Labs
Zi Yan
reviewerRe: [PATCH] mm/compaction: Use folio in hugetlb pathway
In reply to Vishal Moola (Oracle)Quoted 12 lines · Vishal Moola (Oracle)
Use a folio in the hugetlb pathway during the compaction migrate-able pageblock scan.
This removes a call to
compound_head().Signed-off-by Vishal Moola (Oracle) <[email protected]>--- include/linux/hugetlb.h | 4 ++-- mm/compaction.c | 8 ++++---- mm/hugetlb.c | 3 +-- 3 files changed, 7 insertions(+), 8 deletions(-)
LGTM. Reviewed-by: Zi Yan <[email protected]>
Best Regards,
Yan, Zi
Quoted 8 lines · Vishal Moola (Oracle)
The compaction code doesn't take references on pages until we're certain we should attempt to handle it.
In the hugetlb case,
isolate_or_dissolve_huge_page()may return-EBUSYwithout taking a reference to the folio associated with our pfn. If our folio's refcount drops to 0,compound_nr()becomes unpredictable, makinglow_pfnandnr_scannedunreliable.
The user-visible effect is minimal - this should rarely happen (if ever).
So, with compound_order() we either return the real order of the compound page or '0', right?
Quoted 11 lines
Fix this by storing the folio statistics earlier on the stack (just like the THP and Buddy cases).
Also revert commit
66fe1cf7f581("mm: compaction: use helpercompound_nrinisolate_migratepages_block")
to make backporting easier.Fixes 369fa227c219 ("mm: make alloc_contig_range handle free hugetlb pages")Cc Miaohe Lin <[email protected]>Cc Oscar Salvador <[email protected]>Signed-off-by Vishal Moola (Oracle) <[email protected]>
Signature · 2 lines
Oscar Salvador SUSE Labs
Quoted 4 lines · Oscar Salvador
Quoted 3 lines · Vishal Moola (Oracle)
The compaction code doesn't take references on pages until we're certain we should attempt to handle it.
Quoted 8 lines
Quoted 5 lines
In the hugetlb case,
isolate_or_dissolve_huge_page()may return-EBUSYwithout taking a reference to the folio associated with our pfn. If our folio's refcount drops to 0,compound_nr()becomes unpredictable, makinglow_pfnandnr_scannedunreliable.
The user-visible effect is minimal - this should rarely happen (if ever).So, with
compound_order()we either return the real order of the compound page or '0', right?
Yup. There's a world in which that folio could be freed and reallocated as part of another large order page as well (where it would return the order of that folio).
Quoted 18 lines
Quoted 11 lines
Fix this by storing the folio statistics earlier on the stack (just like the THP and Buddy cases).
Also revert commit
66fe1cf7f581("mm: compaction: use helpercompound_nrinisolate_migratepages_block")
to make backporting easier.Fixes 369fa227c219 ("mm: make alloc_contig_range handle free hugetlb pages")Cc Miaohe Lin <[email protected]>Cc Oscar Salvador <[email protected]>Signed-off-by Vishal Moola (Oracle) <[email protected]>Acked-by Oscar Salvador <[email protected]>Signature · 2 lines
Oscar Salvador SUSE Labs
Zi Yan
reviewerQuoted 24 lines · Vishal Moola (Oracle)
The compaction code doesn't take references on pages until we're certain we should attempt to handle it.
In the hugetlb case,
isolate_or_dissolve_huge_page()may return-EBUSYwithout taking a reference to the folio associated with our pfn. If our folio's refcount drops to 0,compound_nr()becomes unpredictable, makinglow_pfnandnr_scannedunreliable.
The user-visible effect is minimal - this should rarely happen (if ever).Fix this by storing the folio statistics earlier on the stack (just like the THP and Buddy cases).
Also revert commit
66fe1cf7f581("mm: compaction: use helpercompound_nrinisolate_migratepages_block")
to make backporting easier.Fixes 369fa227c219 ("mm: make alloc_contig_range handle free hugetlb pages")Cc Miaohe Lin <[email protected]>Cc Oscar Salvador <[email protected]>Signed-off-by Vishal Moola (Oracle) <[email protected]>--- mm/compaction.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
<snip>
mm/compaction.c isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, · line 1018
Quoted 11 lines
mm/compaction.c +2
-2@@ -1011,8 +1011,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,/* Do not report -EBUSY down the chain */ if (ret == -EBUSY ) ret =0 ; - low_pfn +=compound_nr (page) -1 ; - nr_scanned +=compound_nr (page) -1 ; + low_pfn += (1UL << order) -1 ; + nr_scanned += (1UL << order) -1 ; goto isolate_fail; }
Right after this, there is “low_pfn += folio_nr_pages(folio) - 1” for isolated hugetlb. I wonder if that can use order as well. Maybe not, since the order is obtained without taking a reference, but folio_nr_pages() is called with a reference. They might be different.
Anyway, Reviewed-by: Zi Yan <[email protected]>
Best Regards,
Yan, Zi
Quoted 29 lines · Zi Yan
Quoted 24 lines · Vishal Moola (Oracle)
The compaction code doesn't take references on pages until we're certain we should attempt to handle it.
In the hugetlb case,
isolate_or_dissolve_huge_page()may return-EBUSYwithout taking a reference to the folio associated with our pfn. If our folio's refcount drops to 0,compound_nr()becomes unpredictable, makinglow_pfnandnr_scannedunreliable.
The user-visible effect is minimal - this should rarely happen (if ever).Fix this by storing the folio statistics earlier on the stack (just like the THP and Buddy cases).
Also revert commit
66fe1cf7f581("mm: compaction: use helpercompound_nrinisolate_migratepages_block")
to make backporting easier.Fixes 369fa227c219 ("mm: make alloc_contig_range handle free hugetlb pages")Cc Miaohe Lin <[email protected]>Cc Oscar Salvador <[email protected]>Signed-off-by Vishal Moola (Oracle) <[email protected]>--- mm/compaction.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)<snip>
Quoted 16 lines
mm/compaction.c isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn, · line 1018
Quoted 11 lines
mm/compaction.c +2
-2@@ -1011,8 +1011,8 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,/* Do not report -EBUSY down the chain */ if (ret == -EBUSY ) ret =0 ; - low_pfn +=compound_nr (page) -1 ; - nr_scanned +=compound_nr (page) -1 ; + low_pfn += (1UL << order) -1 ; + nr_scanned += (1UL << order) -1 ; goto isolate_fail; }Right after this, there is “low_pfn +=
folio_nr_pages(folio)- 1” for isolated hugetlb. I wonder if that can use order as well. Maybe not, since the order is obtained without taking a reference, butfolio_nr_pages()is called with a reference. They might be different.
I thought about that as well. There's a very unlikely case where they could be different: We had a hugetlb page, free'd it, then allocated it to another hugetlb page. At this point (the success path) we would want the rest of the compaction counters all the sync up to whichever folio we ended up isolating anyways.
Quoted 1 line
Anyway, Reviewed-by: Zi Yan <[email protected]>
Thanks!
Quoted 3 lines
Best Regards,
Yan, Zi