Skip to contentSkip to conversation

mm/gup: fixups for hugetlb gup rework series

PATCH In review [email protected] 5 participants 12 messages Apr 3, 2024 – Apr 4, 2024

Subsystems (4)
lore
Share & tools
lore

[email protected]

author reviewer
#0 lore raw

cover mm/gup: fixups for hugetlb gup rework series

From: Peter Xu <[email protected]>

Hi,

This is a small patchset that will fix two known issues that got reported today on the previous hugetlb unification series on slow gup [1].

The first issue was reported by Ryan Roberts [2] on a test failure over gup_longterm. Patch 1-2 should fix it. Tested with 32MB hugepages on arm64 VM.

The second issue was reported by Nathan Chancellor [3] on a build issue over loongson's defconfig (loongson3_defconfig). It can be easily reproduced with my own build setup [4], while patch 3 fixes it, and should hopefully fix similar archs where pud_pfn() is not defined even if pud_leaf() can return true.

For the 2nd issue, it's debatable whether it's an arch issue, because logically speaking pud_pfn() and pud_leaf() should either be both defined or both not. But since the current fix will be the simplest and still safe (I at least checked loongarch doesn't support either pud dax or pud hugetlb), I think we can consider having this to unbreak the build first, assuming this could also happen to other archs.

One note is that the last 2 patches should be squashed into corresponding patch, while the 1st patch should be kept standalone.

Thanks,

[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]
[3] https://lore.kernel.org/r/ZgyKLLVZ4vN56uZE@x1n
[4] https://gitlab.com/peterx/lkb-harness/-/blob/main/config.json?ref_type=heads#L32

Peter Xu (3):
mm: Allow anon exclusive check over hugetlb tail pages fixup! mm/gup: handle huge pmd for follow_pmd_mask() fixup! mm/arch: provide pud_pfn() fallback

 include/linux/page-flags.h |  8 +++++++-
 include/linux/pgtable.h    |  6 +++++-
 mm/gup.c                   |  3 ---
 mm/internal.h              | 10 ----------
 4 files changed, 12 insertions(+), 15 deletions(-)
Signature · 1 line
2.44.0
author reviewer
#1 lore raw

PATCH 1/3 mm: Allow anon exclusive check over hugetlb tail pages

From: Peter Xu <[email protected]>

PageAnonExclusive() used to forbid tail pages for hugetlbfs, as that used to be called mostly in hugetlb specific paths and the head page was guaranteed.

As we move forward towards merging hugetlb paths into generic mm, we may start to pass in tail hugetlb pages (when with cont-pte/cont-pmd huge pages) for such check. Allow it to properly fetch the head, in which case the anon-exclusiveness of the head will always represents the tail page.

There's already a sign of it when we look at the fast-gup which already contain the hugetlb processing altogether: we used to have a specific commit 5805192c7b72 ("mm/gup: handle cont-PTE hugetlb pages correctly in gup_must_unshare() via GUP-fast") covering that area. Now with this more generic change, that can also go away.

---
 include/linux/page-flags.h |  8 +++++++-
 mm/internal.h              | 10 ----------
 2 files changed, 7 insertions(+), 11 deletions(-)
include/linux/page-flags.h +7 -1
index 888353c209c0..225357f48a79 100644
@@ -1095,7 +1095,13 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
 static __always_inline int PageAnonExclusive(const struct page *page)
 {
 	VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
-	VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
+	/*
+	 * Allow the anon-exclusive check to work on hugetlb tail pages.
+	 * Here hugetlb pages will always guarantee the anon-exclusiveness
+	 * of the head page represents the tail pages.
+	 */
+	if (PageHuge(page) && !PageHead(page))
+		page = compound_head(page);
 	return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
 }
 
mm/internal.h +0 -10
index 9512de7398d5..87f6e4fd56a5 100644
@@ -1259,16 +1259,6 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma,
 	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))
 		smp_rmb();
 
-	/*
-	 * During GUP-fast we might not get called on the head page for a
-	 * hugetlb page that is mapped using cont-PTE, because GUP-fast does
-	 * not work with the abstracted hugetlb PTEs that always point at the
-	 * head page. For hugetlb, PageAnonExclusive only applies on the head
-	 * page (as it cannot be partially COW-shared), so lookup the head page.
-	 */
-	if (unlikely(!PageHead(page) && PageHuge(page)))
-		page = compound_head(page);
-
 	/*
 	 * Note that PageKsm() pages cannot be exclusive, and consequently,
 	 * cannot get pinned.
Signature · 1 line
2.44.0
Acked-by on v1 1/3
#2 lore raw

Re: [PATCH 1/3] mm: Allow anon exclusive check over hugetlb tail pages

Quoted 6 lines · On 03.04.24 03:32, [email protected] wrote:

From: Peter Xu <[email protected]>

PageAnonExclusive() used to forbid tail pages for hugetlbfs, as that used to be called mostly in hugetlb specific paths and the head page was guaranteed.

Quoted 6 lines

As we move forward towards merging hugetlb paths into generic mm, we may start to pass in tail hugetlb pages (when with cont-pte/cont-pmd huge pages) for such check. Allow it to properly fetch the head, in which case the anon-exclusiveness of the head will always represents the tail page.

There's already a sign of it when we look at the fast-gup which already

"GUP-fast" ;)

Quoted 41 lines

contain the hugetlb processing altogether: we used to have a specific commit 5805192c7b72 ("mm/gup: handle cont-PTE hugetlb pages correctly in gup_must_unshare() via GUP-fast") covering that area. Now with this more generic change, that can also go away.

---
  include/linux/page-flags.h |  8 +++++++-
  mm/internal.h              | 10 ----------
  2 files changed, 7 insertions(+), 11 deletions(-)
include/linux/page-flags.h +7 -1
index 888353c209c0..225357f48a79 100644
@@ -1095,7 +1095,13 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
  static __always_inline int PageAnonExclusive(const struct page *page)
  {
  	VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
-	VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
+	/*
+	 * Allow the anon-exclusive check to work on hugetlb tail pages.
+	 * Here hugetlb pages will always guarantee the anon-exclusiveness
+	 * of the head page represents the tail pages.
+	 */
+	if (PageHuge(page) && !PageHead(page))
+		page = compound_head(page);
  	return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
  }
 
mm/internal.h +0 -3
index 9512de7398d5..87f6e4fd56a5 100644
@@ -1259,16 +1259,6 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma,
  	if (IS_ENABLED(CONFIG_HAVE_FAST_GUP))
  		smp_rmb();
 
-	/*
-	 * During GUP-fast we might not get called on the head page for a
-	 * hugetlb page that is mapped using cont-PTE, because GUP-fast does

mm/internal.h static inline bool gup_must_unshare(struct vm_area_struct *vma, · line 1264

Quoted 10 lines
mm/internal.h +0 -7
-	 * not work with the abstracted hugetlb PTEs that always point at the
-	 * head page. For hugetlb, PageAnonExclusive only applies on the head
-	 * page (as it cannot be partially COW-shared), so lookup the head page.
-	 */
-	if (unlikely(!PageHead(page) && PageHuge(page)))
-		page = compound_head(page);
-
  	/*
  	 * Note that PageKsm() pages cannot be exclusive, and consequently,
  	 * cannot get pinned.
Signature · 3 lines
Cheers,

David / dhildenb
#3 lore raw

Re: [PATCH 1/3] mm: Allow anon exclusive check over hugetlb tail pages

In reply to [email protected]

include/linux/page-flags.h PAGEFLAG(Isolated, isolated, PF_ANY); · line 1104

Quoted 13 lines · On Tue, Apr 02, 2024 at 09:32:47PM -0400, [email protected] wrote:

+++ b/include/linux/page-flags.h

include/linux/page-flags.h +7 -1
@@ -1095,7 +1095,13 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
 static __always_inline int PageAnonExclusive(const struct page *page)
 {
 	VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
-	VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
+	/*
+	 * Allow the anon-exclusive check to work on hugetlb tail pages.
+	 * Here hugetlb pages will always guarantee the anon-exclusiveness
+	 * of the head page represents the tail pages.
+	 */
+	if (PageHuge(page) && !PageHead(page))
+		page = compound_head(page);

I think this should be written as:

	/*
	 * HugeTLB stores this information on the head page; THP keeps
	 * it per page
	 */
	if (PageHuge(page))
		page = compound_head(page);
Quoted 2 lines
include/linux/page-flags.h +0 -0
 	return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
 }
author reviewer
#4 lore raw

Re: [PATCH 1/3] mm: Allow anon exclusive check over hugetlb tail pages

Quoted 17 lines · Matthew Wilcox

include/linux/page-flags.h PAGEFLAG(Isolated, isolated, PF_ANY); · line 1104

Quoted 13 lines · On Tue, Apr 02, 2024 at 09:32:47PM -0400, [email protected] wrote:

+++ b/include/linux/page-flags.h

include/linux/page-flags.h +7 -1
@@ -1095,7 +1095,13 @@ PAGEFLAG(Isolated, isolated, PF_ANY);
 static __always_inline int PageAnonExclusive(const struct page *page)
 {
 	VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
-	VM_BUG_ON_PGFLAGS(PageHuge(page) && !PageHead(page), page);
+	/*
+	 * Allow the anon-exclusive check to work on hugetlb tail pages.
+	 * Here hugetlb pages will always guarantee the anon-exclusiveness
+	 * of the head page represents the tail pages.
+	 */
+	if (PageHuge(page) && !PageHead(page))
+		page = compound_head(page);

I think this should be written as:

Quoted 4 lines
	/*
	 * HugeTLB stores this information on the head page; THP keeps
	 * it per page
	 */

This comment does look clean and cleaner indeed. And yes, mention THP can be helpful too.

Quoted 2 lines
	if (PageHuge(page))
		page = compound_head(page);

I would think PageHead() check would help us to avoid compound_head() on heads, which should still be the majority cases iiuc (assuming page->flags is already around in the cache anyway). I've no strong opinion though, as I can hardly tell a difference in reality.

Quoted 4 lines

include/linux/page-flags.h PAGEFLAG(Isolated, isolated, PF_ANY); · line 1106

Quoted 3 lines
include/linux/page-flags.h +0 -0
 	return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
 }

Thanks,

Signature · 1 line
Peter Xu
#5 lore raw

Re: [PATCH 1/3] mm: Allow anon exclusive check over hugetlb tail pages

Quoted 8 lines · Peter Xu
Quoted 2 lines · Matthew Wilcox
	if (PageHuge(page))
		page = compound_head(page);

I would think PageHead() check would help us to avoid compound_head() on heads, which should still be the majority cases iiuc (assuming page->flags is already around in the cache anyway). I've no strong opinion though, as I can hardly tell a difference in reality.

compound_head() includes a check for PageHead(). Adding the check just makes things slower.

author reviewer
#6 lore raw

Re: [PATCH 1/3] mm: Allow anon exclusive check over hugetlb tail pages

Quoted 5 lines · Matthew Wilcox
Quoted 4 lines · Peter Xu
Quoted 2 lines · Matthew Wilcox
	if (PageHuge(page))
		page = compound_head(page);
Quoted 7 lines
Quoted 4 lines

I would think PageHead() check would help us to avoid compound_head() on heads, which should still be the majority cases iiuc (assuming page->flags is already around in the cache anyway). I've no strong opinion though, as I can hardly tell a difference in reality.

compound_head() includes a check for PageHead(). Adding the check just makes things slower.

They check different fields (compound_head, offset 0x8 for the former). Again I'm ok with either way to go and I don't expect measurable difference.. so if you prefer this I'm happy with it too.

Andrew, could you help update with Matthew's fixup? The comment is definitely better than what I wrote in all cases.

Thanks,

Signature · 1 line
Peter Xu
maintainer
#7 lore raw

Re: [PATCH 1/3] mm: Allow anon exclusive check over hugetlb tail pages

Quoted 2 lines · On Thu, 4 Apr 2024 10:21:21 -0400 Peter Xu <[email protected]> wrote:

Andrew, could you help update with Matthew's fixup? The comment is definitely better than what I wrote in all cases.

From: Andrew Morton <[email protected]>
Subject: mm-allow-anon-exclusive-check-over-hugetlb-tail-pages-fix
Date: Thu Apr 4 01:27:47 PM PDT 2024

simplify PageAnonExclusive(), per Matthew

Cc David Hildenbrand <[email protected]>
Cc Huacai Chen <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Nathan Chancellor <[email protected]>
Cc Peter Xu <[email protected]>
Cc Ryan Roberts <[email protected]>
Cc WANG Xuerui <[email protected]>

---

 include/linux/page-flags.h |    7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
include/linux/page-flags.h renamed from include/linux/page-flags.h~mm-allow-anon-exclusive-check-over-hugetlb-tail-pages-fix +3 -4
@@ -1096,11 +1096,10 @@ static __always_inline int PageAnonExclu
 {
 	VM_BUG_ON_PGFLAGS(!PageAnon(page), page);
 	/*
-	 * Allow the anon-exclusive check to work on hugetlb tail pages.
-	 * Here hugetlb pages will always guarantee the anon-exclusiveness
-	 * of the head page represents the tail pages.
+	 * HugeTLB stores this information on the head page; THP keeps it per
+	 * page
 	 */
-	if (PageHuge(page) && !PageHead(page))
+	if (PageHuge(page))
 		page = compound_head(page);
 	return test_bit(PG_anon_exclusive, &PF_ANY(page, 1)->flags);
 }

_

author reviewer
#8 lore raw

Re: [PATCH 1/3] mm: Allow anon exclusive check over hugetlb tail pages

Quoted 10 lines · Andrew Morton
Quoted 2 lines · On Thu, 4 Apr 2024 10:21:21 -0400 Peter Xu <[email protected]> wrote:

Andrew, could you help update with Matthew's fixup? The comment is definitely better than what I wrote in all cases.

From: Andrew Morton <[email protected]>
Subject: mm-allow-anon-exclusive-check-over-hugetlb-tail-pages-fix
Date: Thu Apr 4 01:27:47 PM PDT 2024

simplify PageAnonExclusive(), per Matthew

Thank you!

Signature · 1 line
Peter Xu
author reviewer
#9 lore raw

PATCH 2/3 fixup! mm/gup: handle huge pmd for follow_pmd_mask()

In reply to [email protected]

From: Peter Xu <[email protected]>

Allow follow_pmd_mask() to take hugetlb tail pages. The old warnings do not help now as hugetlb now allows it to happen, so drop them.

---
 mm/gup.c | 3 ---
 1 file changed, 3 deletions(-)
mm/gup.c +0 -3
index 91d70057aea0..d60b63fcfc82 100644
@@ -775,8 +775,6 @@ static struct page *follow_huge_pmd(struct vm_area_struct *vma,
 	assert_spin_locked(pmd_lockptr(mm, pmd));
 
 	page = pmd_page(pmdval);
-	VM_BUG_ON_PAGE(!PageHead(page) && !is_zone_device_page(page), page);
-
 	if ((flags & FOLL_WRITE) &&
 	    !can_follow_write_pmd(pmdval, page, vma, flags))
 		return NULL;
@@ -805,7 +803,6 @@ static struct page *follow_huge_pmd(struct vm_area_struct *vma,
 
 	page += (addr & ~HPAGE_PMD_MASK) >> PAGE_SHIFT;
 	ctx->page_mask = HPAGE_PMD_NR - 1;
-	VM_BUG_ON_PAGE(!PageCompound(page) && !is_zone_device_page(page), page);
 
 	return page;
 }
Signature · 1 line
2.44.0
author reviewer
#10 lore raw

PATCH 3/3 fixup! mm/arch: provide pud_pfn() fallback

In reply to [email protected]

From: Peter Xu <[email protected]>

---
 include/linux/pgtable.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
include/linux/pgtable.h +5 -1
index fa8f92f6e2d7..0f4b2faa1d71 100644
@@ -1882,9 +1882,13 @@ typedef unsigned int pgtbl_mod_mask;
  * code.  Now it happens too for pud_pfn (and can happen for larger
  * mappings too in the future; we're not there yet).  Instead of defining
  * it for all archs (like pmd_pfn), provide a fallback.
+ *
+ * Note that returning 0 here means any arch that didn't define this can
+ * get severely wrong when it hits a real pud leaf.  It's arch's
+ * responsibility to properly define it when a huge pud is possible.
  */
 #ifndef pud_pfn
-#define pud_pfn(x) ({ BUILD_BUG(); 0; })
+#define pud_pfn(x) 0
 #endif
 
 /*
Signature · 1 line
2.44.0
Tested-by on the v1 series
#11 lore raw
In reply to [email protected]
Quoted 39 lines · On 03/04/2024 02:32, [email protected] wrote:

From: Peter Xu <[email protected]>

Hi,

This is a small patchset that will fix two known issues that got reported today on the previous hugetlb unification series on slow gup [1].

The first issue was reported by Ryan Roberts [2] on a test failure over gup_longterm. Patch 1-2 should fix it. Tested with 32MB hugepages on arm64 VM.

The second issue was reported by Nathan Chancellor [3] on a build issue over loongson's defconfig (loongson3_defconfig). It can be easily reproduced with my own build setup [4], while patch 3 fixes it, and should hopefully fix similar archs where pud_pfn() is not defined even if pud_leaf() can return true.

For the 2nd issue, it's debatable whether it's an arch issue, because logically speaking pud_pfn() and pud_leaf() should either be both defined or both not. But since the current fix will be the simplest and still safe (I at least checked loongarch doesn't support either pud dax or pud hugetlb), I think we can consider having this to unbreak the build first, assuming this could also happen to other archs.

One note is that the last 2 patches should be squashed into corresponding patch, while the 1st patch should be kept standalone.

Thanks,

[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]
[3] https://lore.kernel.org/r/ZgyKLLVZ4vN56uZE@x1n
[4] https://gitlab.com/peterx/lkb-harness/-/blob/main/config.json?ref_type=heads#L32

Peter Xu (3):
mm: Allow anon exclusive check over hugetlb tail pages fixup! mm/gup: handle huge pmd for follow_pmd_mask() fixup! mm/arch: provide pud_pfn() fallback

Quoted 6 lines
 include/linux/page-flags.h |  8 +++++++-
 include/linux/pgtable.h    |  6 +++++-
 mm/gup.c                   |  3 ---
 mm/internal.h              | 10 ----------
 4 files changed, 12 insertions(+), 15 deletions(-)

With these applied, gup_longterm is now passing for me on arm64. So for the series: