Skip to contentSkip to conversation

Fix incorrect access of hugetlb pte entries

PATCH Superseded Dev Jain 3 participants 24 messages Jul 2 – Jul 3

Subsystems (7)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Dev Jain

author
#0 lore raw

v2 cover Fix incorrect access of hugetlb pte entries

There are various places which use ptep_get() to get the pte entry corresponding to a hugetlb folio. Some arches (like s390) have special handling to compute the pteval, so they provide huge_ptep_get(). Use this helper consistently.

Additionally, some code paths may provide huge_ptep_get with an unaligned address. This is a problem on arm64 (I checked other arches and it looks fine for them), which is fixed in patch 1. The fix is made to be
backport-friendly: the cleaner fix would be to perhaps pass the hstate to huge_ptep_get() - that is wider churn and we can do that later.

---
v1->v2:
- Add patch 1
- Mention s390 in cover letter

Dev Jain (6):
arm64: make huge_ptep_get handled unaligned addresses mm/rmap: use huge_ptep_get() in try_to_unmap_one() mm/rmap: use huge_ptep_get() in try_to_migrate_one() mm/migrate: use huge_ptep_get() in remove_migration_pte() mm/page_vma_mapped: use huge_ptep_get() for hugetlb mm/mprotect: use huge_ptep_get() for hugetlb

 arch/arm64/mm/hugetlbpage.c |  2 +-
 include/linux/hugetlb.h     |  3 +++
 mm/migrate.c                |  6 +++++-
 mm/mprotect.c               |  8 +++++++-
 mm/page_vma_mapped.c        |  8 +++++++-
 mm/rmap.c                   | 32 ++++++++++++++++++++------------
 6 files changed, 43 insertions(+), 16 deletions(-)
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v2 1/6 arm64: make huge_ptep_get handled unaligned addresses

huge_ptep_get() can be handed a virtual address pointing to the middle of a contpmd/contpte mapped hugetlb folio (examples of callers are pagemap_hugetlb_range, page_mapped_in_vma).

The arm64 helper rewalks the pgtables in find_num_contig to answer whether the huge pte we have maps a contpmd or a contpte hugetlb folio, and returns CONT_PMDS or CONT_PTES, so that it can collect a/d bits over the contiguous ptes. We can falsely return CONT_PTES instead of CONT_PMDS if the addr is not aligned.

Fix this by aligning the pmdp pointer down to a contpmd base before checking equality with the passed huge pte pointer, to correctly answer whether the huge pte is the base of a contpmd block.

Fixes 29cb80519689 ("arm64: hugetlb: Cleanup huge_pte size discovery mechanisms")
---
 arch/arm64/mm/hugetlbpage.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/mm/hugetlbpage.c +1 -1
index 30772a909aea3..8e799c1fe0aa6 100644
@@ -87,7 +87,7 @@ static int find_num_contig(struct mm_struct *mm, unsigned long addr,
 	p4dp = p4d_offset(pgdp, addr);
 	pudp = pud_offset(p4dp, addr);
 	pmdp = pmd_offset(pudp, addr);
-	if ((pte_t *)pmdp == ptep) {
+	if ((pte_t *)PTR_ALIGN_DOWN(pmdp, sizeof(*pmdp) * CONT_PMDS) == ptep) {
 		*pgsize = PMD_SIZE;
 		return CONT_PMDS;
 	}
Signature · 1 line
2.43.0
maintainer Acked-by on v2 1/6
#2 lore raw

Re: [PATCH v2 1/6] arm64: make huge_ptep_get handled unaligned addresses

Quoted 26 lines · Dev Jain

huge_ptep_get() can be handed a virtual address pointing to the middle of a contpmd/contpte mapped hugetlb folio (examples of callers are pagemap_hugetlb_range, page_mapped_in_vma).

The arm64 helper rewalks the pgtables in find_num_contig to answer whether the huge pte we have maps a contpmd or a contpte hugetlb folio, and returns CONT_PMDS or CONT_PTES, so that it can collect a/d bits over the contiguous ptes. We can falsely return CONT_PTES instead of CONT_PMDS if the addr is not aligned.

Fix this by aligning the pmdp pointer down to a contpmd base before checking equality with the passed huge pte pointer, to correctly answer whether the huge pte is the base of a contpmd block.

Fixes 29cb80519689 ("arm64: hugetlb: Cleanup huge_pte size discovery mechanisms")
---
 arch/arm64/mm/hugetlbpage.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/mm/hugetlbpage.c +0 -0
index 30772a909aea3..8e799c1fe0aa6 100644

arch/arm64/mm/hugetlbpage.c static int find_num_contig(struct mm_struct *mm, unsigned long addr, · line 93

Quoted 8 lines
arch/arm64/mm/hugetlbpage.c +1 -1
@@ -87,7 +87,7 @@ static int find_num_contig(struct mm_struct *mm, unsigned long addr,
 	p4dp = p4d_offset(pgdp, addr);
 	pudp = pud_offset(p4dp, addr);
 	pmdp = pmd_offset(pudp, addr);
-	if ((pte_t *)pmdp == ptep) {
+	if ((pte_t *)PTR_ALIGN_DOWN(pmdp, sizeof(*pmdp) * CONT_PMDS) == ptep) {
 		*pgsize = PMD_SIZE;
 		return CONT_PMDS;
 	}
Signature · 3 lines
Cheers,

David
author
#3 lore raw

PATCH v2 2/6 mm/rmap: use huge_ptep_get() in try_to_unmap_one()

In reply to Dev Jain

try_to_unmap_one() handles hugetlb folios when memory failure needs to replace a poisoned hugetlb mapping with a hwpoison entry. In that case page_vma_mapped_walk() returns the pte pointer to the hugetlb folio in pvmw.pte, but the code reads it with ptep_get().

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

It is not clear whether this has a trivially visible effect to userspace.

Just use huge_ptep_get() for dereferencing a huge pte pointer.

Fixes c7ab0d2fdc84 ("mm: convert try_to_unmap_one() to use page_vma_mapped_walk()")
---
 include/linux/hugetlb.h |  3 +++
 mm/rmap.c               | 16 ++++++++++------
 2 files changed, 13 insertions(+), 6 deletions(-)
include/linux/hugetlb.h +3 -0
index 2abaf99321e90..fdb7bdf7645c5 100644
@@ -1261,6 +1261,9 @@ static inline void hugetlb_count_sub(long l, struct mm_struct *mm)
 {
 }
 
+pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr,
+		    pte_t *ptep);
+
 static inline pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,
 					  unsigned long addr, pte_t *ptep)
 {
mm/rmap.c +10 -6
index 1c77d5dc06e9f..aa8a254efaecc 100644
@@ -2095,11 +2095,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		/* Unexpected PMD-mapped THP? */
 		VM_BUG_ON_FOLIO(!pvmw.pte, folio);
 
-		/*
-		 * Handle PFN swap PTEs, such as device-exclusive ones, that
-		 * actually map pages.
-		 */
-		pteval = ptep_get(pvmw.pte);
+		address = pvmw.address;
+		if (folio_test_hugetlb(folio)) {
+			pteval = huge_ptep_get(mm, address, pvmw.pte);
+		} else {
+			/*
+			 * Handle PFN swap PTEs, such as device-exclusive ones,
+			 * that actually map pages.
+			 */
+			pteval = ptep_get(pvmw.pte);
+		}
 		if (likely(pte_present(pteval))) {
 			pfn = pte_pfn(pteval);
 		} else {
@@ -2110,7 +2115,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		}
 
 		subpage = folio_page(folio, pfn - folio_pfn(folio));
-		address = pvmw.address;
 		anon_exclusive = folio_test_anon(folio) &&
 				 PageAnonExclusive(subpage);
 
Signature · 1 line
2.43.0
maintainer
#4 lore raw

Re: [PATCH v2 2/6] mm/rmap: use huge_ptep_get() in try_to_unmap_one()

Quoted 27 lines · Dev Jain

try_to_unmap_one() handles hugetlb folios when memory failure needs to replace a poisoned hugetlb mapping with a hwpoison entry. In that case page_vma_mapped_walk() returns the pte pointer to the hugetlb folio in pvmw.pte, but the code reads it with ptep_get().

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

It is not clear whether this has a trivially visible effect to userspace.

Just use huge_ptep_get() for dereferencing a huge pte pointer.

Fixes c7ab0d2fdc84 ("mm: convert try_to_unmap_one() to use page_vma_mapped_walk()")
---
  include/linux/hugetlb.h |  3 +++
  mm/rmap.c               | 16 ++++++++++------
  2 files changed, 13 insertions(+), 6 deletions(-)
include/linux/hugetlb.h +0 -0
index 2abaf99321e90..fdb7bdf7645c5 100644

include/linux/hugetlb.h static inline void hugetlb_count_sub(long l, struct mm_struct *mm) · line 1266

Quoted 6 lines
include/linux/hugetlb.h +3 -0
@@ -1261,6 +1261,9 @@ static inline void hugetlb_count_sub(long l, struct mm_struct *mm)
  {
  }
 
+pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr,
+		    pte_t *ptep);
+

Maybe I didn't express my thoughts clearly in the first version, let me explain in more detail.

We should define this stub as a no-op for !CONFIG_HUGETLB_PAGE (like set_huge_pte_at, that is why I mentioned 5d4af6195c87c6 for your reference in your previous version). Currently, you've added a declaration, but the function itself doesn't actually exist, which seems quite strange to me.

Muchun,
Thanks.

Quoted 37 lines
include/linux/hugetlb.h +0 -0
  static inline pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,
  					  unsigned long addr, pte_t *ptep)
  {
mm/rmap.c +10 -6
index 1c77d5dc06e9f..aa8a254efaecc 100644
@@ -2095,11 +2095,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
  		/* Unexpected PMD-mapped THP? */
  		VM_BUG_ON_FOLIO(!pvmw.pte, folio);
 
-		/*
-		 * Handle PFN swap PTEs, such as device-exclusive ones, that
-		 * actually map pages.
-		 */
-		pteval = ptep_get(pvmw.pte);
+		address = pvmw.address;
+		if (folio_test_hugetlb(folio)) {
+			pteval = huge_ptep_get(mm, address, pvmw.pte);
+		} else {
+			/*
+			 * Handle PFN swap PTEs, such as device-exclusive ones,
+			 * that actually map pages.
+			 */
+			pteval = ptep_get(pvmw.pte);
+		}
  		if (likely(pte_present(pteval))) {
  			pfn = pte_pfn(pteval);
  		} else {
@@ -2110,7 +2115,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
  		}
 
  		subpage = folio_page(folio, pfn - folio_pfn(folio));
-		address = pvmw.address;
  		anon_exclusive = folio_test_anon(folio) &&
  				 PageAnonExclusive(subpage);
 
author
#5 lore raw

Re: [PATCH v2 2/6] mm/rmap: use huge_ptep_get() in try_to_unmap_one()

Quoted 39 lines · Muchun Song
Quoted 27 lines · Dev Jain

try_to_unmap_one() handles hugetlb folios when memory failure needs to replace a poisoned hugetlb mapping with a hwpoison entry. In that case page_vma_mapped_walk() returns the pte pointer to the hugetlb folio in pvmw.pte, but the code reads it with ptep_get().

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

It is not clear whether this has a trivially visible effect to userspace.

Just use huge_ptep_get() for dereferencing a huge pte pointer.

Fixes c7ab0d2fdc84 ("mm: convert try_to_unmap_one() to use page_vma_mapped_walk()")

---
  include/linux/hugetlb.h |  3 +++
  mm/rmap.c               | 16 ++++++++++------

  2 files changed, 13 insertions(+), 6 deletions(-)
include/linux/hugetlb.h +0 -0
index 2abaf99321e90..fdb7bdf7645c5 100644

include/linux/hugetlb.h static inline void hugetlb_count_sub(long l, struct mm_struct *mm) · line 1265

Quoted 5 lines
include/linux/hugetlb.h +2 -0
@@ -1261,6 +1261,9 @@ static inline void hugetlb_count_sub(long l, struct mm_struct *mm)
   {
   }
   +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr,
+            pte_t *ptep);
+

Maybe I didn't express my thoughts clearly in the first version, let me explain in more detail.

Quoted 4 lines

We should define this stub as a no-op for !CONFIG_HUGETLB_PAGE (like set_huge_pte_at, that is why I mentioned 5d4af6195c87c6 for your reference in your previous version). Currently, you've added a declaration, but the function itself doesn't actually exist, which seems quite strange to me.

https://lore.kernel.org/all/[email protected]/

David suggested this. Honestly I quite like David's suggestion, what do you think?

Quoted 39 lines

Muchun,
Thanks.

Quoted 35 lines
include/linux/hugetlb.h +0 -0
   static inline pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,
                         unsigned long addr, pte_t *ptep)
   {
mm/rmap.c +0 -0
index 1c77d5dc06e9f..aa8a254efaecc 100644
@@ -2095,11 +2095,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
           /* Unexpected PMD-mapped THP? */
           VM_BUG_ON_FOLIO(!pvmw.pte, folio);

  -        /*
-         * Handle PFN swap PTEs, such as device-exclusive ones, that -         * actually map pages.
-         */
-        pteval = ptep_get(pvmw.pte);
+        address = pvmw.address;
+        if (folio_test_hugetlb(folio)) {
+            pteval = huge_ptep_get(mm, address, pvmw.pte); +        } else {
+            /*
+             * Handle PFN swap PTEs, such as device-exclusive ones, +             * that actually map pages.
+             */
+            pteval = ptep_get(pvmw.pte);
+        }
          if (likely(pte_present(pteval))) {
              pfn = pte_pfn(pteval);
          } else {

mm/rmap.c +0 -1
@@ -2110,7 +2115,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
           }
             subpage = folio_page(folio, pfn - folio_pfn(folio));
-        address = pvmw.address;

          anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(subpage);

maintainer
#6 lore raw

Re: [PATCH v2 2/6] mm/rmap: use huge_ptep_get() in try_to_unmap_one()

Quoted 51 lines · Dev Jain
Quoted 39 lines · Muchun Song
Quoted 27 lines · Dev Jain

try_to_unmap_one() handles hugetlb folios when memory failure needs to replace a poisoned hugetlb mapping with a hwpoison entry. In that case page_vma_mapped_walk() returns the pte pointer to the hugetlb folio in pvmw.pte, but the code reads it with ptep_get().

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

It is not clear whether this has a trivially visible effect to userspace.

Just use huge_ptep_get() for dereferencing a huge pte pointer.

Fixes c7ab0d2fdc84 ("mm: convert try_to_unmap_one() to use page_vma_mapped_walk()")
---
  include/linux/hugetlb.h |  3 +++
  mm/rmap.c               | 16 ++++++++++------
  2 files changed, 13 insertions(+), 6 deletions(-)
include/linux/hugetlb.h +0 -0
index 2abaf99321e90..fdb7bdf7645c5 100644

include/linux/hugetlb.h static inline void hugetlb_count_sub(long l, struct mm_struct *mm) · line 1265

Quoted 5 lines
include/linux/hugetlb.h +2 -0
@@ -1261,6 +1261,9 @@ static inline void hugetlb_count_sub(long l, struct mm_struct *mm)
  {
  }
  +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr,
+            pte_t *ptep);
+

Maybe I didn't express my thoughts clearly in the first version, let me explain in more detail.

Quoted 4 lines

We should define this stub as a no-op for !CONFIG_HUGETLB_PAGE (like set_huge_pte_at, that is why I mentioned 5d4af6195c87c6 for your reference in your previous version). Currently, you've added a declaration, but the function itself doesn't actually exist, which seems quite strange to me.

Quoted 4 lines

https://lore.kernel.org/all/[email protected]/

David suggested this. Honestly I quite like David's suggestion, what do you think?

Thanks for pointing that out, I missed it earlier. That said, looking at hugetlb.h, it already contains quite a few no-op stubs. To keep things consistent, I'd personally prefer a stub here. Since David suggested this, I’d love to hear his thoughts on this as well.

Muchun,
Thanks

Quoted 39 lines
Quoted 39 lines

Muchun,
Thanks.

Quoted 34 lines
include/linux/hugetlb.h +0 -0
  static inline pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,
                        unsigned long addr, pte_t *ptep)
  {
mm/rmap.c +10 -5
index 1c77d5dc06e9f..aa8a254efaecc 100644
@@ -2095,11 +2095,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
          /* Unexpected PMD-mapped THP? */
          VM_BUG_ON_FOLIO(!pvmw.pte, folio);
  -        /*
-         * Handle PFN swap PTEs, such as device-exclusive ones, that
-         * actually map pages.
-         */
-        pteval = ptep_get(pvmw.pte);
+        address = pvmw.address;
+        if (folio_test_hugetlb(folio)) {
+            pteval = huge_ptep_get(mm, address, pvmw.pte);
+        } else {
+            /*
+             * Handle PFN swap PTEs, such as device-exclusive ones,
+             * that actually map pages.
+             */
+            pteval = ptep_get(pvmw.pte);
+        }
          if (likely(pte_present(pteval))) {
              pfn = pte_pfn(pteval);
          } else {
@@ -2110,7 +2115,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
          }
            subpage = folio_page(folio, pfn - folio_pfn(folio));
-        address = pvmw.address;
          anon_exclusive = folio_test_anon(folio) &&
                   PageAnonExclusive(subpage);
#7 lore raw

Re: [PATCH v2 2/6] mm/rmap: use huge_ptep_get() in try_to_unmap_one()

Quoted 20 lines · Muchun Song
Quoted 20 lines · Dev Jain
Quoted 10 lines · Muchun Song

Maybe I didn't express my thoughts clearly in the first version, let me explain in more detail.

We should define this stub as a no-op for !CONFIG_HUGETLB_PAGE (like set_huge_pte_at, that is why I mentioned 5d4af6195c87c6 for your reference in your previous version). Currently, you've added a declaration, but the function itself doesn't actually exist, which seems quite strange to me.

https://lore.kernel.org/all/[email protected]/

Quoted 7 lines
Quoted 2 lines

David suggested this. Honestly I quite like David's suggestion, what do you think?

Thanks for pointing that out, I missed it earlier. That said, looking at hugetlb.h, it already contains quite a few no-op stubs. To keep things consistent, I'd personally prefer a stub here. Since David suggested this, I’d love to hear his thoughts on this as well.

It's the shortest possible way to enforce (through the linker!) that you function is not called from wrong context.

The only alternative is using a stub with a BUILD_BUG() in it. Which will 4 5 LOC :)

Signature · 3 lines
Cheers,

David
#8 lore raw

Re: [PATCH v2 2/6] mm/rmap: use huge_ptep_get() in try_to_unmap_one()

In reply to Dev Jain
Quoted 27 lines · Dev Jain

try_to_unmap_one() handles hugetlb folios when memory failure needs to replace a poisoned hugetlb mapping with a hwpoison entry. In that case page_vma_mapped_walk() returns the pte pointer to the hugetlb folio in pvmw.pte, but the code reads it with ptep_get().

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

It is not clear whether this has a trivially visible effect to userspace.

Just use huge_ptep_get() for dereferencing a huge pte pointer.

Fixes c7ab0d2fdc84 ("mm: convert try_to_unmap_one() to use page_vma_mapped_walk()")
---
 include/linux/hugetlb.h |  3 +++
 mm/rmap.c               | 16 ++++++++++------
 2 files changed, 13 insertions(+), 6 deletions(-)
include/linux/hugetlb.h +0 -0
index 2abaf99321e90..fdb7bdf7645c5 100644

include/linux/hugetlb.h static inline void hugetlb_count_sub(long l, struct mm_struct *mm) · line 1265

Quoted 5 lines
include/linux/hugetlb.h +2 -0
@@ -1261,6 +1261,9 @@ static inline void hugetlb_count_sub(long l, struct mm_struct *mm)
 {
 }
 
+pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr,
+		    pte_t *ptep);

Two tabs, or just in a single line.

If others prefer a stub, I don't care. This here is shortest to let the linker bail out.

Quoted 9 lines
include/linux/hugetlb.h +1 -0
+
 static inline pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,
 					  unsigned long addr, pte_t *ptep)
 {
mm/rmap.c +0 -0
index 1c77d5dc06e9f..aa8a254efaecc 100644

mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2101

Quoted 7 lines
mm/rmap.c +0 -4
@@ -2095,11 +2095,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		/* Unexpected PMD-mapped THP? */
 		VM_BUG_ON_FOLIO(!pvmw.pte, folio);
 
-		/*
-		 * Handle PFN swap PTEs, such as device-exclusive ones, that
-		 * actually map pages.
-		 */

That comment now actually belongs above the pte_present() check below.

Signature · 3 lines
Cheers,

David
author
#9 lore raw

Re: [PATCH v2 2/6] mm/rmap: use huge_ptep_get() in try_to_unmap_one()

Quoted 51 lines · David Hildenbrand (Arm)
Quoted 27 lines · Dev Jain

try_to_unmap_one() handles hugetlb folios when memory failure needs to replace a poisoned hugetlb mapping with a hwpoison entry. In that case page_vma_mapped_walk() returns the pte pointer to the hugetlb folio in pvmw.pte, but the code reads it with ptep_get().

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

It is not clear whether this has a trivially visible effect to userspace.

Just use huge_ptep_get() for dereferencing a huge pte pointer.

Fixes c7ab0d2fdc84 ("mm: convert try_to_unmap_one() to use page_vma_mapped_walk()")
---
 include/linux/hugetlb.h |  3 +++
 mm/rmap.c               | 16 ++++++++++------
 2 files changed, 13 insertions(+), 6 deletions(-)
include/linux/hugetlb.h +0 -0
index 2abaf99321e90..fdb7bdf7645c5 100644

include/linux/hugetlb.h static inline void hugetlb_count_sub(long l, struct mm_struct *mm) · line 1265

Quoted 5 lines
include/linux/hugetlb.h +2 -0
@@ -1261,6 +1261,9 @@ static inline void hugetlb_count_sub(long l, struct mm_struct *mm)
 {
 }
 
+pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr,
+		    pte_t *ptep);

Two tabs, or just in a single line.

If others prefer a stub, I don't care. This here is shortest to let the linker bail out.

Quoted 12 lines
include/linux/hugetlb.h +1 -0
+
 static inline pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,
 					  unsigned long addr, pte_t *ptep)
 {
mm/rmap.c +0 -0
index 1c77d5dc06e9f..aa8a254efaecc 100644
@@ -2095,11 +2095,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		/* Unexpected PMD-mapped THP? */
 		VM_BUG_ON_FOLIO(!pvmw.pte, folio);
 
Quoted 6 lines

mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2101

Quoted 4 lines
mm/rmap.c +0 -4
-		/*
-		 * Handle PFN swap PTEs, such as device-exclusive ones, that
-		 * actually map pages.
-		 */

That comment now actually belongs above the pte_present() check below.

Oops I mindlessly just copied the comment.

I will then rather put it in the else block which gets the pfn from the softleaf, that is the most appropriate.

Quoted 2 lines
#10 lore raw

Re: [PATCH v2 2/6] mm/rmap: use huge_ptep_get() in try_to_unmap_one()

Quoted 16 lines
Quoted 9 lines
Quoted 9 lines
mm/rmap.c +0 -0
index 1c77d5dc06e9f..aa8a254efaecc 100644
@@ -2095,11 +2095,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,
 		/* Unexpected PMD-mapped THP? */
 		VM_BUG_ON_FOLIO(!pvmw.pte, folio);
 
Quoted 6 lines

mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 2101

Quoted 4 lines
mm/rmap.c +0 -4
-		/*
-		 * Handle PFN swap PTEs, such as device-exclusive ones, that
-		 * actually map pages.
-		 */

That comment now actually belongs above the pte_present() check below.

Quoted 4 lines

Oops I mindlessly just copied the comment.

I will then rather put it in the else block which gets the pfn from the softleaf, that is the most appropriate.

Works for me, thanks

Signature · 3 lines
Cheers,

David
author
#11 lore raw

PATCH v2 3/6 mm/rmap: use huge_ptep_get() in try_to_migrate_one()

In reply to Dev Jain

try_to_migrate_one() is used by folio migration to replace a present mapping with a migration entry. For hugetlb folios, page_vma_mapped_walk() returns the pte pointer to the hugetlb folio in pvmw.pte, but the code reads the huge pte entry with ptep_get().

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

It is not clear whether this has a trivially visible effect to userspace.

Use huge_ptep_get() to dereference a huge pte pointer.

Commit a98a2f0c8ce1 copied the bug from try_to_unmap_one into try_to_migrate_one.

Fixes a98a2f0c8ce1 ("mm/rmap: split migration into its own function")
---
 mm/rmap.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
mm/rmap.c +10 -6
index aa8a254efaecc..abc3a44baaa3d 100644
@@ -2505,11 +2505,16 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 		/* Unexpected PMD-mapped THP? */
 		VM_BUG_ON_FOLIO(!pvmw.pte, folio);
 
-		/*
-		 * Handle PFN swap PTEs, such as device-exclusive ones, that
-		 * actually map pages.
-		 */
-		pteval = ptep_get(pvmw.pte);
+		address = pvmw.address;
+		if (folio_test_hugetlb(folio)) {
+			pteval = huge_ptep_get(mm, address, pvmw.pte);
+		} else {
+			/*
+			 * Handle PFN swap PTEs, such as device-exclusive ones,
+			 * that actually map pages.
+			 */
+			pteval = ptep_get(pvmw.pte);
+		}
 		if (likely(pte_present(pteval))) {
 			pfn = pte_pfn(pteval);
 		} else {
@@ -2520,7 +2525,6 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 		}
 
 		subpage = folio_page(folio, pfn - folio_pfn(folio));
-		address = pvmw.address;
 		anon_exclusive = folio_test_anon(folio) &&
 				 PageAnonExclusive(subpage);
 
Signature · 1 line
2.43.0
#12 lore raw

Re: [PATCH v2 3/6] mm/rmap: use huge_ptep_get() in try_to_migrate_one()

Quoted 35 lines · Dev Jain

try_to_migrate_one() is used by folio migration to replace a present mapping with a migration entry. For hugetlb folios, page_vma_mapped_walk() returns the pte pointer to the hugetlb folio in pvmw.pte, but the code reads the huge pte entry with ptep_get().

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

It is not clear whether this has a trivially visible effect to userspace.

Use huge_ptep_get() to dereference a huge pte pointer.

Commit a98a2f0c8ce1 copied the bug from try_to_unmap_one into try_to_migrate_one.

Fixes a98a2f0c8ce1 ("mm/rmap: split migration into its own function")
---
 mm/rmap.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
mm/rmap.c +0 -3
index aa8a254efaecc..abc3a44baaa3d 100644
@@ -2505,11 +2505,16 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,
 		/* Unexpected PMD-mapped THP? */
 		VM_BUG_ON_FOLIO(!pvmw.pte, folio);
 
-		/*
-		 * Handle PFN swap PTEs, such as device-exclusive ones, that
-		 * actually map pages.

mm/rmap.c static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, · line 2515

Quoted 10 lines
mm/rmap.c +8 -2
-		 */
-		pteval = ptep_get(pvmw.pte);
+		address = pvmw.address;
+		if (folio_test_hugetlb(folio)) {
+			pteval = huge_ptep_get(mm, address, pvmw.pte);
+		} else {
+			/*
+			 * Handle PFN swap PTEs, such as device-exclusive ones,
+			 * that actually map pages.
+			 */

Again, the comment is related to the pte_present() handling, so best to move it down there.

Apart from that LGTM.

Signature · 3 lines
Cheers,

David
author
#13 lore raw

PATCH v2 4/6 mm/migrate: use huge_ptep_get() in remove_migration_pte()

In reply to Dev Jain

remove_migration_pte() converts migration entries back to present PTEs after folio migration completes. For hugetlb folios, page_vma_mapped_walk() returns the pte pointer to the hugetlb folio in pvmw.pte, but the code reads it with ptep_get().

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

It is not clear whether this has a trivially visible effect to userspace.

Use huge_ptep_get() to dereference a huge pte pointer.

Fixes 290408d4a250 ("hugetlb: hugepage migration core")
---
 mm/migrate.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
mm/migrate.c +5 -1
index d9b23909d716c..c65f0f43df7eb 100644
@@ -371,7 +371,11 @@ static bool remove_migration_pte(struct folio *folio,
 			continue;
 		}
 #endif
-		old_pte = ptep_get(pvmw.pte);
+		if (folio_test_hugetlb(folio))
+			old_pte = huge_ptep_get(vma->vm_mm, pvmw.address,
+						pvmw.pte);
+		else
+			old_pte = ptep_get(pvmw.pte);
 		if (rmap_walk_arg->map_unused_to_zeropage &&
 		    try_to_map_unused_to_zeropage(&pvmw, folio, old_pte, idx))
 			continue;
Signature · 1 line
2.43.0
maintainer Acked-by on v2 4/6
#14 lore raw

Re: [PATCH v2 4/6] mm/migrate: use huge_ptep_get() in remove_migration_pte()

Quoted 10 lines · Dev Jain

remove_migration_pte() converts migration entries back to present PTEs after folio migration completes. For hugetlb folios, page_vma_mapped_walk() returns the pte pointer to the hugetlb folio in pvmw.pte, but the code reads it with ptep_get().

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

It is not clear whether this has a trivially visible effect to userspace.

Right, for non-present entries it's even weirder. Fortunately your patch #1 also handles that.

Signature · 3 lines
Cheers,

David
author
#15 lore raw

PATCH v2 5/6 mm/page_vma_mapped: use huge_ptep_get() for hugetlb

In reply to Dev Jain

check_pte() is the final validation step in page_vma_mapped_walk(). It reads pvmw->pte with ptep_get() to decide whether the entry maps the PFN range being walked. For hugetlb VMAs, that pointer refers to a hugetlb entry.

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

It is not clear whether this has a trivially visible effect to userspace.

Use huge_ptep_get() to dereference a huge pte pointer.

Fixes ace71a19cec5 ("mm: introduce page_vma_mapped_walk()")
---
 mm/page_vma_mapped.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
mm/page_vma_mapped.c +7 -1
index 2ccbabfb2cc17..18e1d341f463c 100644
@@ -107,7 +107,13 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp,
 static bool check_pte(struct page_vma_mapped_walk *pvmw, unsigned long pte_nr)
 {
 	unsigned long pfn;
-	pte_t ptent = ptep_get(pvmw->pte);
+	pte_t ptent;
+
+	if (is_vm_hugetlb_page(pvmw->vma))
+		ptent = huge_ptep_get(pvmw->vma->vm_mm, pvmw->address,
+				      pvmw->pte);
+	else
+		ptent = ptep_get(pvmw->pte);
 
 	if (pvmw->flags & PVMW_MIGRATION) {
 		const softleaf_t entry = softleaf_from_pte(ptent);
Signature · 1 line
2.43.0
maintainer Reviewed-by on v2 5/6
#16 lore raw

Re: [PATCH v2 5/6] mm/page_vma_mapped: use huge_ptep_get() for hugetlb

Quoted 13 lines · Dev Jain

check_pte() is the final validation step in page_vma_mapped_walk(). It reads pvmw->pte with ptep_get() to decide whether the entry maps the PFN range being walked. For hugetlb VMAs, that pointer refers to a hugetlb entry.

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

Quoted 7 lines

It is not clear whether this has a trivially visible effect to userspace.

Use huge_ptep_get() to dereference a huge pte pointer.

Fixes ace71a19cec5 ("mm: introduce page_vma_mapped_walk()")

Thanks.

maintainer Acked-by on v2 5/6
#17 lore raw

Re: [PATCH v2 5/6] mm/page_vma_mapped: use huge_ptep_get() for hugetlb

In reply to Dev Jain
Quoted 9 lines · Dev Jain

check_pte() is the final validation step in page_vma_mapped_walk(). It reads pvmw->pte with ptep_get() to decide whether the entry maps the PFN range being walked. For hugetlb VMAs, that pointer refers to a hugetlb entry.

On arches which provide their own huge_ptep_get() to dereference a huge pte pointer, accessing via ptep_get() would cause pte_pfn(), pte_present() etc to misbehave.

Quoted 8 lines

It is not clear whether this has a trivially visible effect to userspace.

Use huge_ptep_get() to dereference a huge pte pointer.

Fixes ace71a19cec5 ("mm: introduce page_vma_mapped_walk()")

---

Signature · 3 lines
Cheers,

David
author
#18 lore raw

PATCH v2 6/6 mm/mprotect: use huge_ptep_get() for hugetlb

In reply to Dev Jain

prot_none_hugetlb_entry() is the hugetlb callback for the early mprotect(PROT_NONE) PFN permission walk on x86.

The callback passes the decoded PFN to pfn_modify_allowed(). For a hugetlb callback, the pte pointer refers to a hugetlb entry. On architectures where hugetlb entries need huge_ptep_get(), reading that entry with ptep_get() can make the permission check use the wrong PFN.

Use huge_ptep_get() before decoding the hugetlb PFN.

Currently there is no path which can trigger a bug: huge_ptep_get() is a simple ptep_get() for x86, and the prot_none walk occurs only for x86.

So no need to backport - use the correct helper anyways.

Fixes 42e4089c7890 ("x86/speculation/l1tf: Disallow non privileged high MMIO PROT_NONE mappings")
---
 mm/mprotect.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
mm/mprotect.c +7 -1
index 9cbf932b028cf..23779632d18bf 100644
@@ -699,14 +699,20 @@ static int prot_none_pte_entry(pte_t *pte, unsigned long addr,
 		0 : -EACCES;
 }
 
+#ifdef CONFIG_HUGETLB_PAGE
 static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask,
 				   unsigned long addr, unsigned long next,
 				   struct mm_walk *walk)
 {
-	return pfn_modify_allowed(pte_pfn(ptep_get(pte)),
+	pte_t entry = huge_ptep_get(walk->mm, addr, pte);
+
+	return pfn_modify_allowed(pte_pfn(entry),
 				  *(pgprot_t *)(walk->private)) ?
 		0 : -EACCES;
 }
+#else
+#define prot_none_hugetlb_entry	NULL
+#endif
 
 static int prot_none_test(unsigned long addr, unsigned long next,
 			  struct mm_walk *walk)
Signature · 1 line
2.43.0
author
#19 lore raw

Re: [PATCH v2 6/6] mm/mprotect: use huge_ptep_get() for hugetlb

Quoted 10 lines · Dev Jain

prot_none_hugetlb_entry() is the hugetlb callback for the early mprotect(PROT_NONE) PFN permission walk on x86.

The callback passes the decoded PFN to pfn_modify_allowed(). For a hugetlb callback, the pte pointer refers to a hugetlb entry. On architectures where hugetlb entries need huge_ptep_get(), reading that entry with ptep_get() can make the permission check use the wrong PFN.

Use huge_ptep_get() before decoding the hugetlb PFN.

Quoted 8 lines

Currently there is no path which can trigger a bug: huge_ptep_get() is a simple ptep_get() for x86, and the prot_none walk occurs only for x86.

So no need to backport - use the correct helper anyways.

Fixes 42e4089c7890 ("x86/speculation/l1tf: Disallow non privileged high MMIO PROT_NONE mappings")

---

@Muchun, I did not do the change you suggested: if we do #ifdef around the hugetlb callback, then for CONFIG_HUGETLB_PAGE=n, prot_none_hugetlb_entry() is defined but not used, giving compile error.

Quoted 29 lines
 mm/mprotect.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
mm/mprotect.c +7 -1
index 9cbf932b028cf..23779632d18bf 100644
@@ -699,14 +699,20 @@ static int prot_none_pte_entry(pte_t *pte, unsigned long addr,
 		0 : -EACCES;
 }
 
+#ifdef CONFIG_HUGETLB_PAGE
 static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask,
 				   unsigned long addr, unsigned long next,
 				   struct mm_walk *walk)
 {
-	return pfn_modify_allowed(pte_pfn(ptep_get(pte)),
+	pte_t entry = huge_ptep_get(walk->mm, addr, pte);
+
+	return pfn_modify_allowed(pte_pfn(entry),
 				  *(pgprot_t *)(walk->private)) ?
 		0 : -EACCES;
 }
+#else
+#define prot_none_hugetlb_entry	NULL
+#endif
 
 static int prot_none_test(unsigned long addr, unsigned long next,
 			  struct mm_walk *walk)
maintainer
#20 lore raw

Re: [PATCH v2 6/6] mm/mprotect: use huge_ptep_get() for hugetlb

Quoted 22 lines · Dev Jain
Quoted 15 lines · Dev Jain

prot_none_hugetlb_entry() is the hugetlb callback for the early mprotect(PROT_NONE) PFN permission walk on x86.

The callback passes the decoded PFN to pfn_modify_allowed(). For a hugetlb callback, the pte pointer refers to a hugetlb entry. On architectures where hugetlb entries need huge_ptep_get(), reading that entry with ptep_get() can make the permission check use the wrong PFN.

Use huge_ptep_get() before decoding the hugetlb PFN.

Currently there is no path which can trigger a bug: huge_ptep_get() is a simple ptep_get() for x86, and the prot_none walk occurs only for x86.

So no need to backport - use the correct helper anyways.

Quoted 8 lines
Quoted 3 lines
Fixes 42e4089c7890 ("x86/speculation/l1tf: Disallow non privileged high MMIO PROT_NONE mappings")

---

@Muchun, I did not do the change you suggested: if we do #ifdef around the hugetlb callback, then for CONFIG_HUGETLB_PAGE=n, prot_none_hugetlb_entry() is defined but not used, giving compile error.

Got it. Thanks for your explanation.

maintainer Reviewed-by on v2 6/6
#21 lore raw

Re: [PATCH v2 6/6] mm/mprotect: use huge_ptep_get() for hugetlb

In reply to Dev Jain
Quoted 14 lines · Dev Jain

prot_none_hugetlb_entry() is the hugetlb callback for the early mprotect(PROT_NONE) PFN permission walk on x86.

The callback passes the decoded PFN to pfn_modify_allowed(). For a hugetlb callback, the pte pointer refers to a hugetlb entry. On architectures where hugetlb entries need huge_ptep_get(), reading that entry with ptep_get() can make the permission check use the wrong PFN.

Use huge_ptep_get() before decoding the hugetlb PFN.

Quoted 7 lines

Currently there is no path which can trigger a bug: huge_ptep_get() is a simple ptep_get() for x86, and the prot_none walk occurs only for x86.

So no need to backport - use the correct helper anyways.

Fixes 42e4089c7890 ("x86/speculation/l1tf: Disallow non privileged high MMIO PROT_NONE mappings")

Thanks.

#22 lore raw

Re: [PATCH v2 6/6] mm/mprotect: use huge_ptep_get() for hugetlb

In reply to Dev Jain
Quoted 26 lines · Dev Jain

prot_none_hugetlb_entry() is the hugetlb callback for the early mprotect(PROT_NONE) PFN permission walk on x86.

The callback passes the decoded PFN to pfn_modify_allowed(). For a hugetlb callback, the pte pointer refers to a hugetlb entry. On architectures where hugetlb entries need huge_ptep_get(), reading that entry with ptep_get() can make the permission check use the wrong PFN.

Use huge_ptep_get() before decoding the hugetlb PFN.

Currently there is no path which can trigger a bug: huge_ptep_get() is a simple ptep_get() for x86, and the prot_none walk occurs only for x86.

So no need to backport - use the correct helper anyways.

Fixes 42e4089c7890 ("x86/speculation/l1tf: Disallow non privileged high MMIO PROT_NONE mappings")
---
 mm/mprotect.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
mm/mprotect.c +0 -0
index 9cbf932b028cf..23779632d18bf 100644

mm/mprotect.c static int prot_none_pte_entry(pte_t *pte, unsigned long addr, · line 707

Quoted 10 lines
mm/mprotect.c +2 -1
@@ -699,14 +699,20 @@ static int prot_none_pte_entry(pte_t *pte, unsigned long addr,
 		0 : -EACCES;
 }
 
+#ifdef CONFIG_HUGETLB_PAGE
 static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask,
 				   unsigned long addr, unsigned long next,
 				   struct mm_walk *walk)
 {
-	return pfn_modify_allowed(pte_pfn(ptep_get(pte)),
+	pte_t entry = huge_ptep_get(walk->mm, addr, pte);

Nit: can be const :)

mm/mprotect.c static int prot_none_pte_entry(pte_t *pte, unsigned long addr, · line 711

Quoted 4 lines
mm/mprotect.c +2 -0
+
+	return pfn_modify_allowed(pte_pfn(entry),
 				  *(pgprot_t *)(walk->private)) ?
 		0 : -EACCES;

Indentation is odd.

Can we just make this readable?

if (pfn_modify_allowed ...)
return 0
return -EACCESS;

Signature · 3 lines
Cheers,

David
author
#23 lore raw

Re: [PATCH v2 6/6] mm/mprotect: use huge_ptep_get() for hugetlb

Quoted 47 lines · David Hildenbrand (Arm)
Quoted 26 lines · Dev Jain

prot_none_hugetlb_entry() is the hugetlb callback for the early mprotect(PROT_NONE) PFN permission walk on x86.

The callback passes the decoded PFN to pfn_modify_allowed(). For a hugetlb callback, the pte pointer refers to a hugetlb entry. On architectures where hugetlb entries need huge_ptep_get(), reading that entry with ptep_get() can make the permission check use the wrong PFN.

Use huge_ptep_get() before decoding the hugetlb PFN.

Currently there is no path which can trigger a bug: huge_ptep_get() is a simple ptep_get() for x86, and the prot_none walk occurs only for x86.

So no need to backport - use the correct helper anyways.

Fixes 42e4089c7890 ("x86/speculation/l1tf: Disallow non privileged high MMIO PROT_NONE mappings")
---
 mm/mprotect.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
mm/mprotect.c +0 -0
index 9cbf932b028cf..23779632d18bf 100644

mm/mprotect.c static int prot_none_pte_entry(pte_t *pte, unsigned long addr, · line 707

Quoted 10 lines
mm/mprotect.c +2 -1
@@ -699,14 +699,20 @@ static int prot_none_pte_entry(pte_t *pte, unsigned long addr,
 		0 : -EACCES;
 }
 
+#ifdef CONFIG_HUGETLB_PAGE
 static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask,
 				   unsigned long addr, unsigned long next,
 				   struct mm_walk *walk)
 {
-	return pfn_modify_allowed(pte_pfn(ptep_get(pte)),
+	pte_t entry = huge_ptep_get(walk->mm, addr, pte);

Nit: can be const :)

mm/mprotect.c static int prot_none_pte_entry(pte_t *pte, unsigned long addr, · line 711

Quoted 4 lines
mm/mprotect.c +2 -0
+
+	return pfn_modify_allowed(pte_pfn(entry),
 				  *(pgprot_t *)(walk->private)) ?
 		0 : -EACCES;

Indentation is odd.

Quoted 6 lines

Can we just make this readable?

if (pfn_modify_allowed ...)
return 0
return -EACCESS;

Sure I'll do this.

Quoted 1 line