Fixes for pte encapsulation bypasses
Subsystems (4)
v3 cover Fixes for pte encapsulation bypasses
This is the first half of v3 of a series to improve the encapsulation of pte entries by disallowing non-arch code from directly dereferencing pte_t pointers. Based on earlier feedback, I've split the series in 2; this first part contains fixes for existing bugs that were discovered during the work. The second part, which contains the new changes I'm adding, will be posted separately.
These fixes have had a fair amount of review now so I hope they can be considered for addition to the appropriate mm- branch as is. And I've cc'ed stable since I believe they should be candidates for backport.
See the v1 cover letter at [1] for rationale and explanation of overall objective (requires both parts of the split series to achieve).
The series is split up as follows:
patch 1-2: Fix bugs where code was _setting_ ptes directly, rather than using
set_pte_at() and friends. Data corruption was theoretically possible.
patch 3: Minor refactoring requested in v2 review. patch 4: Fix highmem unmapping issue I spotted while doing the work.
Patches are based on v6.4-rc4 and a branch is available at [3].
Changes since v2 [2]:
- patch 2: minor commit message rewording to fix review nits
- patch 3: Refactored damon code to use {pte|pmd}p_clear_young_notify()
- patch 1-4: applied Ack/Reviewed-by tags; thanks for those!
Changes since v1 [1]:
- patch 1: Refactored pfn to use local variable
- patch 1-2: Minor rewording of commit message: 'verify' -> 'check'
- patch 1-3: applied Ack/Reviewed-by tags; thanks for those!
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://gitlab.arm.com/linux-arm/linux-rr/-/tree/bugs/pte_encapsulation_bypasses-lkml_v3
Thanks,
Ryan
Ryan Roberts (4):
mm: vmalloc must set pte via arch code mm/damon/ops-common: atomically test and clear young on ptes and pmds mm/damon/ops-common: Refactor to use {pte|pmd}p_clear_young_notify()
mm: Fix failure to unmap pte on highmem systems
mm/damon/ops-common.c | 30 ++++-------------------------- mm/damon/ops-common.h | 4 ++-- mm/damon/paddr.c | 4 ++-- mm/damon/vaddr.c | 4 ++-- mm/memory.c | 6 ++---- mm/vmalloc.c | 10 ++++++++-- 6 files changed, 20 insertions(+), 38 deletions(-)
--
2.25.1
PATCH v3 1/4 mm: vmalloc must set pte via arch code
It is bad practice to directly set pte entries within a pte table. Instead all modifications must go through arch-provided helpers such as set_pte_at() to give the arch code visibility and allow it to check (and potentially modify) the operation.
--- mm/vmalloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
mm/vmalloc.c +8 -2
@@ -2899,10 +2899,16 @@ struct vmap_pfn_data {static int vmap_pfn_apply (pte_t *pte,unsigned long addr,void *private) {struct vmap_pfn_data *data = private; +unsigned long pfn = data->pfns[data->idx]; +pte_t ptent; -if (WARN_ON_ONCE (pfn_valid (data->pfns[data->idx]))) +if (WARN_ON_ONCE (pfn_valid (pfn)))return -EINVAL ; - *pte =pte_mkspecial (pfn_pte (data->pfns[data->idx++], data->prot)); + + ptent =pte_mkspecial (pfn_pte (pfn, data->prot)); +set_pte_at (&init_mm, addr, pte, ptent); + + data->idx++;return 0 ; }
Signature · 1 line
2.25.1
PATCH v3 2/4 mm/damon/ops-common: atomically test and clear young on ptes and pmds
In reply to Ryan RobertsIt is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper, ptep_test_and_clear_young() to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.
--- mm/damon/ops-common.c | 16 ++++++---------- mm/damon/ops-common.h | 4 ++-- mm/damon/paddr.c | 4 ++-- mm/damon/vaddr.c | 4 ++-- 4 files changed, 12 insertions(+), 16 deletions(-)
mm/damon/ops-common.c +6 -10
@@ -37,7 +37,7 @@ struct folio *damon_get_folio(unsigned long pfn)return folio; } -void damon_ptep_mkold (pte_t *pte,struct mm_struct *mm,unsigned long addr) +void damon_ptep_mkold (pte_t *pte,struct vm_area_struct *vma,unsigned long addr) {bool referenced = false;struct folio *folio =damon_get_folio (pte_pfn (*pte));
@@ -45,13 +45,11 @@ void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, unsigned long addr)if (!folio)return ; -if (pte_young (*pte)) { +if (ptep_test_and_clear_young (vma, addr, pte)) referenced = true; - *pte =pte_mkold (*pte); - }#ifdef CONFIG_MMU_NOTIFIER -if (mmu_notifier_clear_young (mm, addr, addr +PAGE_SIZE )) +if (mmu_notifier_clear_young (vma->vm_mm, addr, addr +PAGE_SIZE )) referenced = true;#endif /* CONFIG_MMU_NOTIFIER */
@@ -62,7 +60,7 @@ void damon_ptep_mkold(pte_t *pte, struct mm_struct *mm, unsigned long addr)folio_put (folio); } -void damon_pmdp_mkold (pmd_t *pmd,struct mm_struct *mm,unsigned long addr) +void damon_pmdp_mkold (pmd_t *pmd,struct vm_area_struct *vma,unsigned long addr) {#ifdef CONFIG_TRANSPARENT_HUGEPAGE bool referenced = false;
@@ -71,13 +69,11 @@ void damon_pmdp_mkold(pmd_t *pmd, struct mm_struct *mm, unsigned long addr)if (!folio)return ; -if (pmd_young (*pmd)) { +if (pmdp_test_and_clear_young (vma, addr, pmd)) referenced = true; - *pmd =pmd_mkold (*pmd); - }#ifdef CONFIG_MMU_NOTIFIER -if (mmu_notifier_clear_young (mm, addr, addr +HPAGE_PMD_SIZE )) +if (mmu_notifier_clear_young (vma->vm_mm, addr, addr +HPAGE_PMD_SIZE )) referenced = true;#endif /* CONFIG_MMU_NOTIFIER */
mm/damon/ops-common.h +2 -2
@@ -9,8 +9,8 @@struct folio *damon_get_folio (unsigned long pfn); -void damon_ptep_mkold (pte_t *pte,struct mm_struct *mm,unsigned long addr); -void damon_pmdp_mkold (pmd_t *pmd,struct mm_struct *mm,unsigned long addr); +void damon_ptep_mkold (pte_t *pte,struct vm_area_struct *vma,unsigned long addr); +void damon_pmdp_mkold (pmd_t *pmd,struct vm_area_struct *vma,unsigned long addr);int damon_cold_score (struct damon_ctx *c,struct damon_region *r,struct damos *s);
mm/damon/paddr.c +2 -2
@@ -24,9 +24,9 @@ static bool __damon_pa_mkold(struct folio *folio, struct vm_area_struct *vma,while (page_vma_mapped_walk (&pvmw)) { addr = pvmw.address;if (pvmw.pte) -damon_ptep_mkold (pvmw.pte, vma->vm_mm, addr); +damon_ptep_mkold (pvmw.pte, vma, addr);else -damon_pmdp_mkold (pvmw.pmd, vma->vm_mm, addr); +damon_pmdp_mkold (pvmw.pmd, vma, addr); }return true; }
mm/damon/vaddr.c +2 -2
@@ -311,7 +311,7 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr, }if (pmd_trans_huge (*pmd)) { -damon_pmdp_mkold (pmd, walk->mm, addr); +damon_pmdp_mkold (pmd, walk->vma, addr);spin_unlock (ptl);return 0 ; }
@@ -323,7 +323,7 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr, pte =pte_offset_map_lock (walk->mm, pmd, addr, &ptl);if (!pte_present (*pte)) goto out; -damon_ptep_mkold (pte, walk->mm, addr); +damon_ptep_mkold (pte, walk->vma, addr); out:pte_unmap_unlock (pte, ptl);return 0 ;
Signature · 1 line
2.25.1
Re: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds
Quoted 10 lines · Ryan Roberts
It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper,
ptep_test_and_clear_young()to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").
Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?
Quoted 4 lines
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Zi Yan <[email protected]>Reviewed-by SeongJae Park <[email protected]>Reviewed-by Mike Rapoport (IBM) <[email protected]>
Re: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds
Quoted 10 lines · Yu Zhao
Quoted 9 lines · Ryan Roberts
It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper,
ptep_test_and_clear_young()to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.
Quoted 4 lines
Quoted 1 line
Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?
It was overlooked - incompetance strikes again! I was intending to cc the whole series. What's the best way to fix this? Can I just add stable in cc on reply to the cover letter or will I have to resend the lot?
Quoted 5 lines
Quoted 5 lines
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Zi Yan <[email protected]>Reviewed-by SeongJae Park <[email protected]>Reviewed-by Mike Rapoport (IBM) <[email protected]>
Re: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds
Quoted 12 lines · Ryan Roberts
Quoted 10 lines · Yu Zhao
Quoted 9 lines · Ryan Roberts
It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper,
ptep_test_and_clear_young()to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?
It was overlooked - incompetance strikes again! I was intending to cc the whole series. What's the best way to fix this? Can I just add stable in cc on reply to the cover letter or will I have to resend the lot?
Resending the whole series would be more reliable for the process (and easier for Andrew). You might want to include a few new reviewed/acked-bys anyway (I just acked the next patch).
SeongJae Park
maintainerRe: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds
In reply to Ryan RobertsHi Ryan,
Quoted 11 lines · On Fri, 2 Jun 2023 18:14:25 +0100 Ryan Roberts <[email protected]> wrote:
Quoted 10 lines · Yu Zhao
Quoted 9 lines · Ryan Roberts
It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper,
ptep_test_and_clear_young()to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?
It was overlooked - incompetance strikes again! I was intending to cc the whole series.
Not the whole patches in this series but only this patch is intended to be merged in stable series, right? If I'm not wrong, you could add 'Cc: <[email protected]>' tag here[1] when resending, to let stable kernel maintainers easily understand exactly what patches should be merged in the stable kernels. So, you wouldn't need to touch coverletter or cc whole series but only this one.
[1] https://www.kernel.org/doc/html/v4.10/process/stable-kernel-rules.html
Thanks,
SJ
Quoted 8 lines
What's the best way to fix this? Can I just add stable in cc on reply to the cover letter or will I have to resend the lot?
Quoted 5 lines
Quoted 5 lines
Signed-off-by Ryan Roberts <[email protected]>Reviewed-by Zi Yan <[email protected]>Reviewed-by SeongJae Park <[email protected]>Reviewed-by Mike Rapoport (IBM) <[email protected]>
SeongJae Park
maintainerRe: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds
Quoted 23 lines · On Fri, 2 Jun 2023 19:15:01 +0000 SeongJae Park <[email protected]> wrote:
Hi Ryan,
Quoted 11 lines · On Fri, 2 Jun 2023 18:14:25 +0100 Ryan Roberts <[email protected]> wrote:
Quoted 10 lines · Yu Zhao
Quoted 9 lines · Ryan Roberts
It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper,
ptep_test_and_clear_young()to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?
It was overlooked - incompetance strikes again! I was intending to cc the whole series.
Quoted 8 lines
Not the whole patches in this series but only this patch is intended to be merged in stable series, right? If I'm not wrong, you could add 'Cc: <[email protected]>' tag here[1] when resending, to let stable kernel maintainers easily understand exactly what patches should be merged in the stable kernels. So, you wouldn't need to touch coverletter or cc whole series but only this one.
[1] https://www.kernel.org/doc/html/v4.10/process/stable-kernel-rules.html
And I just found Andrew added the tag while adding this to the -mm queue. Thank you, Andrew!
[1] https://lore.kernel.org/mm-commits/[email protected]/
Thanks,
SJ
Quoted 5 lines
Thanks,
SJ
Re: [PATCH v3 2/4] mm/damon/ops-common: atomically test and clear young on ptes and pmds
Quoted 32 lines · SeongJae Park
Quoted 30 lines · On Fri, 2 Jun 2023 19:15:01 +0000 SeongJae Park <[email protected]> wrote:
Hi Ryan,
Quoted 11 lines · On Fri, 2 Jun 2023 18:14:25 +0100 Ryan Roberts <[email protected]> wrote:
Quoted 10 lines · Yu Zhao
Quoted 9 lines · Ryan Roberts
It is racy to non-atomically read a pte, then clear the young bit, then write it back as this could discard dirty information. Further, it is bad practice to directly set a pte entry within a table. Instead clearing young must go through the arch-provided helper,
ptep_test_and_clear_young()to ensure it is modified atomically and to give the arch code visibility and allow it to check (and potentially modify) the operation.Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Fixes 3f49584b262c ("mm/damon: implement primitives for the virtual memory address spaces").Just to double check: was "Cc: [email protected]" overlooked or deemed unnecessary?
It was overlooked - incompetance strikes again! I was intending to cc the whole series.
Not the whole patches in this series but only this patch is intended to be merged in stable series, right? If I'm not wrong, you could add 'Cc: <[email protected]>' tag here[1] when resending, to let stable kernel maintainers easily understand exactly what patches should be merged in the stable kernels. So, you wouldn't need to touch coverletter or cc whole series but only this one.
Quoted 4 lines
And I just found Andrew added the tag while adding this to the -mm queue. Thank you, Andrew!
Yes indeed - thanks for fixing that up for me, Andrew!
Quoted 12 lines
[1] https://lore.kernel.org/mm-commits/[email protected]/
Thanks,
SJQuoted 5 lines
Thanks,
SJ
PATCH v3 3/4 mm/damon/ops-common: Refactor to use {pte|pmd}p_clear_young_notify()
In reply to Ryan RobertsWith the fix in place to atomically test and clear young on ptes and pmds, simplify the code to handle the clearing for both the primary mmu and the mmu notifier with a single API call.
--- mm/damon/ops-common.c | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-)
mm/damon/ops-common.c +2 -20
@@ -39,21 +39,12 @@ struct folio *damon_get_folio(unsigned long pfn)void damon_ptep_mkold (pte_t *pte,struct vm_area_struct *vma,unsigned long addr) { -bool referenced = false;struct folio *folio =damon_get_folio (pte_pfn (*pte));if (!folio)return ; -if (ptep_test_and_clear_young (vma, addr, pte)) - referenced = true; - -#ifdef CONFIG_MMU_NOTIFIER -if (mmu_notifier_clear_young (vma->vm_mm, addr, addr +PAGE_SIZE )) - referenced = true; -#endif /* CONFIG_MMU_NOTIFIER */ - -if (referenced) +if (ptep_clear_young_notify (vma, addr, pte))folio_set_young (folio);folio_set_idle (folio);
@@ -63,21 +54,12 @@ void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addrvoid damon_pmdp_mkold (pmd_t *pmd,struct vm_area_struct *vma,unsigned long addr) {#ifdef CONFIG_TRANSPARENT_HUGEPAGE -bool referenced = false;struct folio *folio =damon_get_folio (pmd_pfn (*pmd));if (!folio)return ; -if (pmdp_test_and_clear_young (vma, addr, pmd)) - referenced = true; - -#ifdef CONFIG_MMU_NOTIFIER -if (mmu_notifier_clear_young (vma->vm_mm, addr, addr +HPAGE_PMD_SIZE )) - referenced = true; -#endif /* CONFIG_MMU_NOTIFIER */ - -if (referenced) +if (pmdp_clear_young_notify (vma, addr, pmd))folio_set_young (folio);folio_set_idle (folio);
Signature · 1 line
2.25.1
Re: [PATCH v3 3/4] mm/damon/ops-common: Refactor to use {pte|pmd}p_clear_young_notify()
Quoted 6 lines · Ryan Roberts
With the fix in place to atomically test and clear young on ptes and pmds, simplify the code to handle the clearing for both the primary mmu and the mmu notifier with a single API call.
Signed-off-by Ryan Roberts <[email protected]>
Re: [PATCH v3 3/4] mm/damon/ops-common: Refactor to use {pte|pmd}p_clear_young_notify()
In reply to Ryan RobertsHi Ryan,
Quoted 5 lines · On Fri, 2 Jun 2023 10:29:48 +0100 Ryan Roberts <[email protected]> wrote:
With the fix in place to atomically test and clear young on ptes and pmds, simplify the code to handle the clearing for both the primary mmu and the mmu notifier with a single API call.
Signed-off-by Ryan Roberts <[email protected]>
Thanks,
SJ
Quoted 58 lines
--- mm/damon/ops-common.c | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-)mm/damon/ops-common.c +2
-20@@ -39,21 +39,12 @@ struct folio *damon_get_folio(unsigned long pfn)void damon_ptep_mkold (pte_t *pte,struct vm_area_struct *vma,unsigned long addr) { -bool referenced = false;struct folio *folio =damon_get_folio (pte_pfn (*pte));if (!folio)return ; -if (ptep_test_and_clear_young (vma, addr, pte)) - referenced = true; - -#ifdef CONFIG_MMU_NOTIFIER -if (mmu_notifier_clear_young (vma->vm_mm, addr, addr +PAGE_SIZE )) - referenced = true; -#endif /* CONFIG_MMU_NOTIFIER */ - -if (referenced) +if (ptep_clear_young_notify (vma, addr, pte))folio_set_young (folio);folio_set_idle (folio);@@ -63,21 +54,12 @@ void damon_ptep_mkold(pte_t *pte, struct vm_area_struct *vma, unsigned long addrvoid damon_pmdp_mkold (pmd_t *pmd,struct vm_area_struct *vma,unsigned long addr) {#ifdef CONFIG_TRANSPARENT_HUGEPAGE -bool referenced = false;struct folio *folio =damon_get_folio (pmd_pfn (*pmd));if (!folio)return ; -if (pmdp_test_and_clear_young (vma, addr, pmd)) - referenced = true; - -#ifdef CONFIG_MMU_NOTIFIER -if (mmu_notifier_clear_young (vma->vm_mm, addr, addr +HPAGE_PMD_SIZE )) - referenced = true; -#endif /* CONFIG_MMU_NOTIFIER */ - -if (referenced) +if (pmdp_clear_young_notify (vma, addr, pmd))folio_set_young (folio);folio_set_idle (folio);Signature · 1 line
2.25.1
PATCH v3 4/4 mm: Fix failure to unmap pte on highmem systems
In reply to Ryan RobertsThe loser of a race to service a pte for a device private entry in the swap path previously unlocked the ptl, but failed to unmap the pte. This only affects highmem systems since unmapping a pte is a noop on non-highmem systems.
--- mm/memory.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
mm/memory.c +2 -4
@@ -3728,10 +3728,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->page =pfn_swap_entry_to_page (entry); vmf->pte =pte_offset_map_lock (vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); -if (unlikely (!pte_same (*vmf->pte, vmf->orig_pte))) { -spin_unlock (vmf->ptl); - goto out; - } +if (unlikely (!pte_same (*vmf->pte, vmf->orig_pte))) + goto unlock;/* * Get a page reference while we know the page can't be
Signature · 1 line
2.25.1