Skip to contentSkip to conversation

Fixes for pte encapsulation bypasses

PATCH Applied Ryan Roberts 3 participants 13 messages Jun 2, 2023 – Jun 3, 2023

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v3 cover Fixes for pte encapsulation bypasses

This is the first half of v3 of a series to improve the encapsulation of pte entries by disallowing non-arch code from directly dereferencing pte_t pointers. Based on earlier feedback, I've split the series in 2; this first part contains fixes for existing bugs that were discovered during the work. The second part, which contains the new changes I'm adding, will be posted separately.

These fixes have had a fair amount of review now so I hope they can be considered for addition to the appropriate mm- branch as is. And I've cc'ed stable since I believe they should be candidates for backport.

See the v1 cover letter at [1] for rationale and explanation of overall objective (requires both parts of the split series to achieve).

The series is split up as follows:

patch 1-2: Fix bugs where code was _setting_ ptes directly, rather than using
set_pte_at() and friends. Data corruption was theoretically possible.

patch 3:    Minor refactoring requested in v2 review.
patch 4:    Fix highmem unmapping issue I spotted while doing the work.

Patches are based on v6.4-rc4 and a branch is available at [3].

Changes since v2 [2]:
- patch 2: minor commit message rewording to fix review nits
- patch 3: Refactored damon code to use {pte|pmd}p_clear_young_notify()
- patch 1-4: applied Ack/Reviewed-by tags; thanks for those!

Changes since v1 [1]:
- patch 1: Refactored pfn to use local variable
- patch 1-2: Minor rewording of commit message: 'verify' -> 'check'
- patch 1-3: applied Ack/Reviewed-by tags; thanks for those!

[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/bugs/pte_encapsulation_bypasses-lkml_v3

Thanks,
Ryan

Ryan Roberts (4):
mm: vmalloc must set pte via arch code mm/damon/ops-common: atomically test and clear young on ptes and pmds mm/damon/ops-common: Refactor to use {pte|pmd}p_clear_young_notify()
mm: Fix failure to unmap pte on highmem systems

 mm/damon/ops-common.c | 30 ++++--------------------------
 mm/damon/ops-common.h |  4 ++--
 mm/damon/paddr.c      |  4 ++--
 mm/damon/vaddr.c      |  4 ++--
 mm/memory.c           |  6 ++----
 mm/vmalloc.c          | 10 ++++++++--
 6 files changed, 20 insertions(+), 38 deletions(-)

--
2.25.1

#1 lore raw

PATCH v3 1/4 mm: vmalloc must set pte via arch code

It is bad practice to directly set pte entries within a pte table. Instead all modifications must go through arch-provided helpers such as set_pte_at() to give the arch code visibility and allow it to check (and potentially modify) the operation.

---
 mm/vmalloc.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
mm/vmalloc.c +8 -2
index 9683573f1225..48202ec5f79a 100644
@@ -2899,10 +2899,16 @@ struct vmap_pfn_data {
 static int vmap_pfn_apply(pte_t *pte, unsigned long addr, void *private)
 {
 	struct vmap_pfn_data *data = private;
+	unsigned long pfn = data->pfns[data->idx];
+	pte_t ptent;
 
-	if (WARN_ON_ONCE(pfn_valid(data->pfns[data->idx])))
+	if (WARN_ON_ONCE(pfn_valid(pfn)))
 		return -EINVAL;
-	*pte = pte_mkspecial(pfn_pte(data->pfns[data->idx++], data->prot));
+
+	ptent = pte_mkspecial(pfn_pte(pfn, data->prot));
+	set_pte_at(&init_mm, addr, pte, ptent);
+
+	data->idx++;
 	return 0;
 }
 
Signature · 1 line
2.25.1
#2 lore raw

PATCH v3 2/4 mm/damon/ops-common: atomically test and clear young on ptes and pmds

In reply to Ryan Roberts

It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper, ptep_test_and_clear_young() to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.

Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").
---
 mm/damon/ops-common.c | 16 ++++++----------
 mm/damon/ops-common.h |  4 ++--
 mm/damon/paddr.c      |  4 ++--
 mm/damon/vaddr.c      |  4 ++--
 4 files changed, 12 insertions(+), 16 deletions(-)
mm/damon/ops-common.c +6 -10
index cc63cf953636..acc264b97903 100644
@@ -37,7 +37,7 @@ struct folio *damon_get_folio(unsigned long pfn)
 	return folio;
 }
 
-void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, unsigned long addr)
+void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr)
 {
 	bool referenced = false;
 	struct folio *folio = damon_get_folio(pte_pfn(*pte));
@@ -45,13 +45,11 @@ void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, unsigned long addr)
 	if (!folio)
 		return;
 
-	if (pte_young(*pte)) {
+	if (ptep_test_and_clear_young(vma, addr, pte))
 		referenced = true;
-		*pte = pte_mkold(*pte);
-	}
 
 #ifdef CONFIG_MMU_NOTIFIER
-	if (mmu_notifier_clear_young(mm, addr, addr + PAGE_SIZE))
+	if (mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE))
 		referenced = true;
 #endif /* CONFIG_MMU_NOTIFIER */
 
@@ -62,7 +60,7 @@ void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, unsigned long addr)
 	folio_put(folio);
 }
 
-void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm, unsigned long addr)
+void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr)
 {
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
 	bool referenced = false;
@@ -71,13 +69,11 @@ void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm, unsigned long addr)
 	if (!folio)
 		return;
 
-	if (pmd_young(*pmd)) {
+	if (pmdp_test_and_clear_young(vma, addr, pmd))
 		referenced = true;
-		*pmd = pmd_mkold(*pmd);
-	}
 
 #ifdef CONFIG_MMU_NOTIFIER
-	if (mmu_notifier_clear_young(mm, addr, addr + HPAGE_PMD_SIZE))
+	if (mmu_notifier_clear_young(vma->vm_mm, addr, addr + HPAGE_PMD_SIZE))
 		referenced = true;
 #endif /* CONFIG_MMU_NOTIFIER */
 
mm/damon/ops-common.h +2 -2
index 14f4bc69f29b..18d837d11bce 100644
@@ -9,8 +9,8 @@
 
 struct folio *damon_get_folio(unsigned long pfn);
 
-void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, unsigned long addr);
-void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm, unsigned long addr);
+void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr);
+void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr);
 
 int damon_cold_score(struct damon_ctx *c, struct damon_region *r,
 			struct damos *s);
mm/damon/paddr.c +2 -2
index 467b99166b43..5b3a3463d078 100644
@@ -24,9 +24,9 @@ static bool __damon_pa_mkold(struct folio *folio, struct vm_area_struct *vma,
 	while (page_vma_mapped_walk(&pvmw)) {
 		addr = pvmw.address;
 		if (pvmw.pte)
-			damon_ptep_mkold(pvmw.pte, vma->vm_mm, addr);
+			damon_ptep_mkold(pvmw.pte, vma, addr);
 		else
-			damon_pmdp_mkold(pvmw.pmd, vma->vm_mm, addr);
+			damon_pmdp_mkold(pvmw.pmd, vma, addr);
 	}
 	return true;
 }
mm/damon/vaddr.c +2 -2
index 1fec16d7263e..37994fb6120c 100644
@@ -311,7 +311,7 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr,
 		}
 
 		if (pmd_trans_huge(*pmd)) {
-			damon_pmdp_mkold(pmd, walk->mm, addr);
+			damon_pmdp_mkold(pmd, walk->vma, addr);
 			spin_unlock(ptl);
 			return 0;
 		}
@@ -323,7 +323,7 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr,
 	pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl);
 	if (!pte_present(*pte))
 		goto out;
-	damon_ptep_mkold(pte, walk->mm, addr);
+	damon_ptep_mkold(pte, walk->vma, addr);
 out:
 	pte_unmap_unlock(pte, ptl);
 	return 0;
Signature · 1 line
2.25.1
#3 lore raw

Re: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds

Quoted 10 lines · Ryan Roberts

It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper, ptep_test_and_clear_young() to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.

Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").

Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?

Quoted 4 lines
#4 lore raw

Re: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds

Quoted 10 lines · Yu Zhao
Quoted 9 lines · Ryan Roberts

It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper, ptep_test_and_clear_young() to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.

Quoted 4 lines
Quoted 1 line
Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").

Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?

It was overlooked - incompetance strikes again! I was intending to cc the whole series. What's the best way to fix this? Can I just add stable in cc on reply to the cover letter or will I have to resend the lot?

Quoted 5 lines
Quoted 5 lines
#5 lore raw

Re: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds

Quoted 12 lines · Ryan Roberts
Quoted 10 lines · Yu Zhao
Quoted 9 lines · Ryan Roberts

It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper, ptep_test_and_clear_young() to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").

Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?

It was overlooked - incompetance strikes again! I was intending to cc the whole series. What's the best way to fix this? Can I just add stable in cc on reply to the cover letter or will I have to resend the lot?

Resending the whole series would be more reliable for the process (and easier for Andrew). You might want to include a few new reviewed/acked-bys anyway (I just acked the next patch).

maintainer
#6 lore raw

Re: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds

In reply to Ryan Roberts

Hi Ryan,

Quoted 11 lines · On Fri, 2 Jun 2023 18:14:25 +0100 Ryan Roberts <[email protected]> wrote:
Quoted 10 lines · Yu Zhao
Quoted 9 lines · Ryan Roberts

It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper, ptep_test_and_clear_young() to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").

Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?

It was overlooked - incompetance strikes again! I was intending to cc the whole series.

Not the whole patches in this series but only this patch is intended to be merged in stable series, right? If I'm not wrong, you could add 'Cc: <[email protected]>' tag here[1] when resending, to let stable kernel maintainers easily understand exactly what patches should be merged in the stable kernels. So, you wouldn't need to touch coverletter or cc whole series but only this one.

[1] https://www.kernel.org/doc/html/v4.10/process/stable-kernel-rules.html

Thanks,
SJ

Quoted 8 lines

What's the best way to fix this? Can I just add stable in cc on reply to the cover letter or will I have to resend the lot?

Quoted 5 lines
Quoted 5 lines
maintainer
#7 lore raw

Re: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds

Quoted 23 lines · On Fri, 2 Jun 2023 19:15:01 +0000 SeongJae Park <[email protected]> wrote:

Hi Ryan,

Quoted 11 lines · On Fri, 2 Jun 2023 18:14:25 +0100 Ryan Roberts <[email protected]> wrote:
Quoted 10 lines · Yu Zhao
Quoted 9 lines · Ryan Roberts

It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper, ptep_test_and_clear_young() to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").

Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?

It was overlooked - incompetance strikes again! I was intending to cc the whole series.

Quoted 8 lines

Not the whole patches in this series but only this patch is intended to be merged in stable series, right? If I'm not wrong, you could add 'Cc: <[email protected]>' tag here[1] when resending, to let stable kernel maintainers easily understand exactly what patches should be merged in the stable kernels. So, you wouldn't need to touch coverletter or cc whole series but only this one.

[1] https://www.kernel.org/doc/html/v4.10/process/stable-kernel-rules.html

And I just found Andrew added the tag while adding this to the -mm queue. Thank you, Andrew!

[1] https://lore.kernel.org/mm-commits/[email protected]/

Thanks,
SJ

Quoted 5 lines

Thanks,
SJ

#8 lore raw

Re: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds

Quoted 32 lines · SeongJae Park
Quoted 30 lines · On Fri, 2 Jun 2023 19:15:01 +0000 SeongJae Park <[email protected]> wrote:

Hi Ryan,

Quoted 11 lines · On Fri, 2 Jun 2023 18:14:25 +0100 Ryan Roberts <[email protected]> wrote:
Quoted 10 lines · Yu Zhao
Quoted 9 lines · Ryan Roberts

It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper, ptep_test_and_clear_young() to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").

Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?

It was overlooked - incompetance strikes again! I was intending to cc the whole series.

Not the whole patches in this series but only this patch is intended to be merged in stable series, right? If I'm not wrong, you could add 'Cc: <[email protected]>' tag here[1] when resending, to let stable kernel maintainers easily understand exactly what patches should be merged in the stable kernels. So, you wouldn't need to touch coverletter or cc whole series but only this one.

Quoted 4 lines
Quoted 1 line

[1] https://www.kernel.org/doc/html/v4.10/process/stable-kernel-rules.html

And I just found Andrew added the tag while adding this to the -mm queue. Thank you, Andrew!

Yes indeed - thanks for fixing that up for me, Andrew!

Quoted 12 lines

[1] https://lore.kernel.org/mm-commits/[email protected]/

Thanks,
SJ

Quoted 5 lines

Thanks,
SJ

#9 lore raw

PATCH v3 3/4 mm/damon/ops-common: Refactor to use {pte|pmd}p_clear_young_notify()

In reply to Ryan Roberts

With the fix in place to atomically test and clear young on ptes and pmds, simplify the code to handle the clearing for both the primary mmu and the mmu notifier with a single API call.

---
 mm/damon/ops-common.c | 22 ++--------------------
 1 file changed, 2 insertions(+), 20 deletions(-)
mm/damon/ops-common.c +2 -20
index acc264b97903..d4ab81229136 100644
@@ -39,21 +39,12 @@ struct folio *damon_get_folio(unsigned long pfn)
 
 void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr)
 {
-	bool referenced = false;
 	struct folio *folio = damon_get_folio(pte_pfn(*pte));
 
 	if (!folio)
 		return;
 
-	if (ptep_test_and_clear_young(vma, addr, pte))
-		referenced = true;
-
-#ifdef CONFIG_MMU_NOTIFIER
-	if (mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE))
-		referenced = true;
-#endif /* CONFIG_MMU_NOTIFIER */
-
-	if (referenced)
+	if (ptep_clear_young_notify(vma, addr, pte))
 		folio_set_young(folio);
 
 	folio_set_idle(folio);
@@ -63,21 +54,12 @@ void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr
 void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr)
 {
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	bool referenced = false;
 	struct folio *folio = damon_get_folio(pmd_pfn(*pmd));
 
 	if (!folio)
 		return;
 
-	if (pmdp_test_and_clear_young(vma, addr, pmd))
-		referenced = true;
-
-#ifdef CONFIG_MMU_NOTIFIER
-	if (mmu_notifier_clear_young(vma->vm_mm, addr, addr + HPAGE_PMD_SIZE))
-		referenced = true;
-#endif /* CONFIG_MMU_NOTIFIER */
-
-	if (referenced)
+	if (pmdp_clear_young_notify(vma, addr, pmd))
 		folio_set_young(folio);
 
 	folio_set_idle(folio);
Signature · 1 line
2.25.1
Acked-by on v3 3/4
#10 lore raw

Re: [PATCH v3 3/4] mm/damon/ops-common: Refactor to use {pte|pmd}p_clear_young_notify()

Quoted 6 lines · Ryan Roberts

With the fix in place to atomically test and clear young on ptes and pmds, simplify the code to handle the clearing for both the primary mmu and the mmu notifier with a single API call.

maintainer Reviewed-by on v3 3/4
#11 lore raw

Re: [PATCH v3 3/4] mm/damon/ops-common: Refactor to use {pte|pmd}p_clear_young_notify()

In reply to Ryan Roberts

Hi Ryan,

Quoted 5 lines · On Fri, 2 Jun 2023 10:29:48 +0100 Ryan Roberts <[email protected]> wrote:

With the fix in place to atomically test and clear young on ptes and pmds, simplify the code to handle the clearing for both the primary mmu and the mmu notifier with a single API call.

Thanks,
SJ

Quoted 58 lines
---
 mm/damon/ops-common.c | 22 ++--------------------
 1 file changed, 2 insertions(+), 20 deletions(-)
mm/damon/ops-common.c +2 -20
index acc264b97903..d4ab81229136 100644
@@ -39,21 +39,12 @@ struct folio *damon_get_folio(unsigned long pfn)
 
 void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr)
 {
-	bool referenced = false;
 	struct folio *folio = damon_get_folio(pte_pfn(*pte));
 
 	if (!folio)
 		return;
 
-	if (ptep_test_and_clear_young(vma, addr, pte))
-		referenced = true;
-
-#ifdef CONFIG_MMU_NOTIFIER
-	if (mmu_notifier_clear_young(vma->vm_mm, addr, addr + PAGE_SIZE))
-		referenced = true;
-#endif /* CONFIG_MMU_NOTIFIER */
-
-	if (referenced)
+	if (ptep_clear_young_notify(vma, addr, pte))
 		folio_set_young(folio);
 
 	folio_set_idle(folio);
@@ -63,21 +54,12 @@ void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addr
 void damon_pmdp_mkold(pmd_t *pmd, struct vm_area_struct *vma, unsigned long addr)
 {
 #ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	bool referenced = false;
 	struct folio *folio = damon_get_folio(pmd_pfn(*pmd));
 
 	if (!folio)
 		return;
 
-	if (pmdp_test_and_clear_young(vma, addr, pmd))
-		referenced = true;
-
-#ifdef CONFIG_MMU_NOTIFIER
-	if (mmu_notifier_clear_young(vma->vm_mm, addr, addr + HPAGE_PMD_SIZE))
-		referenced = true;
-#endif /* CONFIG_MMU_NOTIFIER */
-
-	if (referenced)
+	if (pmdp_clear_young_notify(vma, addr, pmd))
 		folio_set_young(folio);
 
 	folio_set_idle(folio);
Signature · 1 line
2.25.1
#12 lore raw

PATCH v3 4/4 mm: Fix failure to unmap pte on highmem systems

In reply to Ryan Roberts

The loser of a race to service a pte for a device private entry in the swap path previously unlocked the ptl, but failed to unmap the pte. This only affects highmem systems since unmapping a pte is a noop on non-highmem systems.

Fixes 16ce101db85d ("mm/memory.c: fix race when faulting a device private page")
---
 mm/memory.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)
mm/memory.c +2 -4
index f69fbc251198..ed429e20a1bb 100644
@@ -3728,10 +3728,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
 			vmf->page = pfn_swap_entry_to_page(entry);
 			vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd,
 					vmf->address, &vmf->ptl);
-			if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) {
-				spin_unlock(vmf->ptl);
-				goto out;
-			}
+			if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte)))
+				goto unlock;
 
 			/*
 			 * Get a page reference while we know the page can't be
Signature · 1 line
2.25.1