Fix cache flush issues considering PMD sharing
Subsystems (4)
cover Fix cache flush issues considering PMD sharing
Hi,
This patch set fixes some cache flushing issues if PMD sharing is possible for hugetlb pages, which were found by code inspection. Please help to review. Thanks.
Baolin Wang (2):
mm: hugetlb: Considering PMD sharing when flushing cache/TLBs
mm: rmap: Move the cache flushing to the correct place for hugetlb PMD
sharing
mm/hugetlb.c | 17 +++++++++++++++-- mm/mremap.c | 2 +- mm/rmap.c | 40 ++++++++++++++++++++++------------------ 3 files changed, 38 insertions(+), 21 deletions(-)
Signature · 1 line
1.8.3.1
PATCH 1/2 mm: hugetlb: Considering PMD sharing when flushing cache/TLBs
When moving hugetlb page tables, the cache flushing is called in move_page_tables() without considering the shared PMDs, which may be cause cache issues on some architectures.
Thus we should move the hugetlb cache flushing into move_hugetlb_page_tables() with considering the shared PMDs ranges, calculated by adjust_range_if_pmd_sharing_possible(). Meanwhile also expanding the TLBs flushing range in case of shared PMDs.
Note this is discovered via code inspection, and did not meet a real problem in practice so far.
--- mm/hugetlb.c | 17 +++++++++++++++-- mm/mremap.c | 2 +- 2 files changed, 16 insertions(+), 3 deletions(-)
mm/hugetlb.c +15 -2
@@ -4937,10 +4937,17 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,unsigned long old_addr_copy;pte_t *src_pte, *dst_pte;struct mmu_notifier_range range; +bool shared_pmd = false;mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , vma, mm, old_addr, old_end);adjust_range_if_pmd_sharing_possible (vma, &range.start, &range.end); +/* +* In case of shared PMDs, we should cover the maximum possible +* range. +*/ +flush_cache_range (vma, range.start, range.end); +mmu_notifier_invalidate_range_start (&range);/* Prevent race with file truncation */ i_mmap_lock_write (mapping);
@@ -4957,8 +4964,10 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,*/ old_addr_copy = old_addr; -if (huge_pmd_unshare (mm, vma, &old_addr_copy, src_pte)) +if (huge_pmd_unshare (mm, vma, &old_addr_copy, src_pte)) { + shared_pmd = true;continue ; + } dst_pte =huge_pte_alloc (mm, new_vma, new_addr, sz);if (!dst_pte)
@@ -4966,7 +4975,11 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,move_huge_pte (vma, old_addr, new_addr, src_pte, dst_pte); } -flush_tlb_range (vma, old_end - len, old_end); + +if (shared_pmd) +flush_tlb_range (vma, range.start, range.end); +else +flush_tlb_range (vma, old_end - len, old_end);mmu_notifier_invalidate_range_end (&range);i_mmap_unlock_write (mapping);
mm/mremap.c +1 -1
@@ -490,12 +490,12 @@ unsigned long move_page_tables(struct vm_area_struct *vma,return 0 ; old_end = old_addr + len; -flush_cache_range (vma, old_addr, old_end);if (is_vm_hugetlb_page (vma))return move_hugetlb_page_tables (vma, new_vma, old_addr, new_addr, len); +flush_cache_range (vma, old_addr, old_end);mmu_notifier_range_init (&range,MMU_NOTIFY_UNMAP ,0 , vma, vma->vm_mm, old_addr, old_end);mmu_notifier_invalidate_range_start (&range);
Signature · 1 line
1.8.3.1
Re: [PATCH 1/2] mm: hugetlb: Considering PMD sharing when flushing cache/TLBs
Quoted 12 lines · Baolin Wang
When moving hugetlb page tables, the cache flushing is called in
move_page_tables()without considering the shared PMDs, which may be cause cache issues on some architectures.Thus we should move the hugetlb cache flushing into
move_hugetlb_page_tables()with considering the shared PMDs ranges, calculated byadjust_range_if_pmd_sharing_possible(). Meanwhile also expanding the TLBs flushing range in case of shared PMDs.Note this is discovered via code inspection, and did not meet a real problem in practice so far.
Quoted 6 lines
Fixes 550a7d60bd5e ("mm, hugepages: add mremap() support for hugepage backed vma")Signed-off-by Baolin Wang <[email protected]>--- mm/hugetlb.c | 17 +++++++++++++++-- mm/mremap.c | 2 +- 2 files changed, 16 insertions(+), 3 deletions(-)
Thanks!
Signature · 1 line
Mike Kravetz
Quoted 66 lines
mm/hugetlb.c +15
-2@@ -4937,10 +4937,17 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,unsigned long old_addr_copy;pte_t *src_pte, *dst_pte;struct mmu_notifier_range range; +bool shared_pmd = false;mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , vma, mm, old_addr, old_end);adjust_range_if_pmd_sharing_possible (vma, &range.start, &range.end); +/* +* In case of shared PMDs, we should cover the maximum possible +* range. +*/ +flush_cache_range (vma, range.start, range.end); +mmu_notifier_invalidate_range_start (&range);/* Prevent race with file truncation */ i_mmap_lock_write (mapping);@@ -4957,8 +4964,10 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,*/ old_addr_copy = old_addr; -if (huge_pmd_unshare (mm, vma, &old_addr_copy, src_pte)) +if (huge_pmd_unshare (mm, vma, &old_addr_copy, src_pte)) { + shared_pmd = true;continue ; + } dst_pte =huge_pte_alloc (mm, new_vma, new_addr, sz);if (!dst_pte)@@ -4966,7 +4975,11 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,move_huge_pte (vma, old_addr, new_addr, src_pte, dst_pte); } -flush_tlb_range (vma, old_end - len, old_end); + +if (shared_pmd) +flush_tlb_range (vma, range.start, range.end); +else +flush_tlb_range (vma, old_end - len, old_end);mmu_notifier_invalidate_range_end (&range);i_mmap_unlock_write (mapping);mm/mremap.c +1
-1@@ -490,12 +490,12 @@ unsigned long move_page_tables(struct vm_area_struct *vma,return 0 ; old_end = old_addr + len; -flush_cache_range (vma, old_addr, old_end);if (is_vm_hugetlb_page (vma))return move_hugetlb_page_tables (vma, new_vma, old_addr, new_addr, len); +flush_cache_range (vma, old_addr, old_end);mmu_notifier_range_init (&range,MMU_NOTIFY_UNMAP ,0 , vma, vma->vm_mm, old_addr, old_end);mmu_notifier_invalidate_range_start (&range);
PATCH 2/2 mm: rmap: Move the cache flushing to the correct place for hugetlb PMD sharing
In reply to Baolin WangThe cache level flush will always be first when changing an existing virtual–>physical mapping to a new value, since this allows us to properly handle systems whose caches are strict and require a virtual–>physical translation to exist for a virtual address. So we should move the cache flushing before huge_pmd_unshare().
As Muchun pointed out[1], now the architectures whose supporting hugetlb PMD sharing have no cache flush issues in practice. But I think we should still follow the cache/TLB flushing rules when changing a valid virtual address mapping in case of potential issues in future.
[1] https://lore.kernel.org/all/YmT%2F%[email protected]/
--- mm/rmap.c | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-)
mm/rmap.c +22 -18
@@ -1535,15 +1535,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* do this outside rmap routines. */ VM_BUG_ON (!(flags &TTU_RMAP_LOCKED )); +/* +* huge_pmd_unshare unmapped an entire PMD page. +* There is no way of knowing exactly which PMDs may +* be cached for this mm, so we must flush them all. +* start/end were already adjusted above to cover this +* range. +*/ +flush_cache_range (vma, range.start, range.end); +if (huge_pmd_unshare (mm, vma, &address, pvmw.pte)) { -/* -* huge_pmd_unshare unmapped an entire PMD -* page. There is no way of knowing exactly -* which PMDs may be cached for this mm, so -* we must flush them all. start/end were -* already adjusted above to cover this range. -*/ -flush_cache_range (vma, range.start, range.end);flush_tlb_range (vma, range.start, range.end);mmu_notifier_invalidate_range (mm, range.start, range.end);
@@ -1560,13 +1561,14 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,page_vma_mapped_walk_done (&pvmw);break ; } + }else { +flush_cache_page (vma, address,pte_pfn (*pvmw.pte)); }/* * Nuke the page table entry. When having to clear * PageAnonExclusive(), we always have to flush. */ -flush_cache_page (vma, address,pte_pfn (*pvmw.pte));if (should_defer_flush (mm, flags) && !anon_exclusive) {/* * We clear the PTE but do not flush so potentially
@@ -1890,15 +1892,16 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,* do this outside rmap routines. */ VM_BUG_ON (!(flags &TTU_RMAP_LOCKED )); +/* +* huge_pmd_unshare unmapped an entire PMD page. +* There is no way of knowing exactly which PMDs may +* be cached for this mm, so we must flush them all. +* start/end were already adjusted above to cover this +* range. +*/ +flush_cache_range (vma, range.start, range.end); +if (huge_pmd_unshare (mm, vma, &address, pvmw.pte)) { -/* -* huge_pmd_unshare unmapped an entire PMD -* page. There is no way of knowing exactly -* which PMDs may be cached for this mm, so -* we must flush them all. start/end were -* already adjusted above to cover this range. -*/ -flush_cache_range (vma, range.start, range.end);flush_tlb_range (vma, range.start, range.end);mmu_notifier_invalidate_range (mm, range.start, range.end);
@@ -1915,10 +1918,11 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,page_vma_mapped_walk_done (&pvmw);break ; } + }else { +flush_cache_page (vma, address,pte_pfn (*pvmw.pte)); }/* Nuke the page table entry. */ -flush_cache_page (vma, address,pte_pfn (*pvmw.pte)); pteval =ptep_clear_flush (vma, address, pvmw.pte);/* Set the dirty flag on the folio now the pte is gone. */
Signature · 1 line
1.8.3.1
Re: [PATCH 2/2] mm: rmap: Move the cache flushing to the correct place for hugetlb PMD sharing
Quoted 22 lines · Baolin Wang
The cache level flush will always be first when changing an existing virtual–>physical mapping to a new value, since this allows us to properly handle systems whose caches are strict and require a virtual–>physical translation to exist for a virtual address. So we should move the cache flushing before
huge_pmd_unshare().As Muchun pointed out[1], now the architectures whose supporting hugetlb PMD sharing have no cache flush issues in practice. But I think we should still follow the cache/TLB flushing rules when changing a valid virtual address mapping in case of potential issues in future.
[1] https://lore.kernel.org/all/YmT%2F%[email protected]/
Signed-off-by Baolin Wang <[email protected]>--- mm/rmap.c | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-)mm/rmap.c +0
-0
mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 1539
Quoted 5 lines
mm/rmap.c +2
-0@@ -1535,15 +1535,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* do this outside rmap routines. */ VM_BUG_ON (!(flags &TTU_RMAP_LOCKED )); +/* +* huge_pmd_unshare unmapped an entire PMD page.
Perhaps update this comment to say that huge_pmd_unshare 'may' unmap an entire PMD page?
Quoted 20 lines
mm/rmap.c +7
-8+* There is no way of knowing exactly which PMDs may +* be cached for this mm, so we must flush them all. +* start/end were already adjusted above to cover this +* range. +*/ +flush_cache_range (vma, range.start, range.end); +if (huge_pmd_unshare (mm, vma, &address, pvmw.pte)) { -/* -* huge_pmd_unshare unmapped an entire PMD -* page. There is no way of knowing exactly -* which PMDs may be cached for this mm, so -* we must flush them all. start/end were -* already adjusted above to cover this range. -*/ -flush_cache_range (vma, range.start, range.end);flush_tlb_range (vma, range.start, range.end);mmu_notifier_invalidate_range (mm, range.start, range.end);
mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 1565
Quoted 5 lines
mm/rmap.c +2
-0@@ -1560,13 +1561,14 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,page_vma_mapped_walk_done (&pvmw);break ; } + }else { +flush_cache_page (vma, address,pte_pfn (*pvmw.pte));
I know this call to flush_cache_page() existed before your change. But, when looking at this now I wonder how hugetlb pages are handled? Are there any versions of flush_cache_page() that take page size into account?
Signature · 1 line
Mike Kravetz
Quoted 48 lines
mm/rmap.c +11
-10}/* * Nuke the page table entry. When having to clear * PageAnonExclusive(), we always have to flush. */ -flush_cache_page (vma, address,pte_pfn (*pvmw.pte));if (should_defer_flush (mm, flags) && !anon_exclusive) {/* * We clear the PTE but do not flush so potentially @@ -1890,15 +1892,16 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,* do this outside rmap routines. */ VM_BUG_ON (!(flags &TTU_RMAP_LOCKED )); +/* +* huge_pmd_unshare unmapped an entire PMD page. +* There is no way of knowing exactly which PMDs may +* be cached for this mm, so we must flush them all. +* start/end were already adjusted above to cover this +* range. +*/ +flush_cache_range (vma, range.start, range.end); +if (huge_pmd_unshare (mm, vma, &address, pvmw.pte)) { -/* -* huge_pmd_unshare unmapped an entire PMD -* page. There is no way of knowing exactly -* which PMDs may be cached for this mm, so -* we must flush them all. start/end were -* already adjusted above to cover this range. -*/ -flush_cache_range (vma, range.start, range.end);flush_tlb_range (vma, range.start, range.end);mmu_notifier_invalidate_range (mm, range.start, range.end);@@ -1915,10 +1918,11 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma,page_vma_mapped_walk_done (&pvmw);break ; } + }else { +flush_cache_page (vma, address,pte_pfn (*pvmw.pte)); }/* Nuke the page table entry. */ -flush_cache_page (vma, address,pte_pfn (*pvmw.pte)); pteval =ptep_clear_flush (vma, address, pvmw.pte);/* Set the dirty flag on the folio now the pte is gone. */
Re: [PATCH 2/2] mm: rmap: Move the cache flushing to the correct place for hugetlb PMD sharing
Quoted 18 lines · Mike Kravetz
Quoted 17 lines · Baolin Wang
The cache level flush will always be first when changing an existing virtual–>physical mapping to a new value, since this allows us to properly handle systems whose caches are strict and require a virtual–>physical translation to exist for a virtual address. So we should move the cache flushing before
huge_pmd_unshare().As Muchun pointed out[1], now the architectures whose supporting hugetlb PMD sharing have no cache flush issues in practice. But I think we should still follow the cache/TLB flushing rules when changing a valid virtual address mapping in case of potential issues in future.
[1] https://lore.kernel.org/all/YmT%2F%[email protected]/
Signed-off-by Baolin Wang <[email protected]>--- mm/rmap.c | 40 ++++++++++++++++++++++------------------ 1 file changed, 22 insertions(+), 18 deletions(-)
Quoted 13 lines
mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 1539
Quoted 10 lines
mm/rmap.c +2
-0@@ -1535,15 +1535,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* do this outside rmap routines. */ VM_BUG_ON (!(flags &TTU_RMAP_LOCKED )); +/* +* huge_pmd_unshare unmapped an entire PMD page. Perhaps update this comment to say that
huge_pmd_unshare'may' unmap an entire PMD page?
Sure, will do.
Quoted 22 lines
Quoted 22 lines
mm/rmap.c +7
-8+* There is no way of knowing exactly which PMDs may +* be cached for this mm, so we must flush them all. +* start/end were already adjusted above to cover this +* range. +*/ +flush_cache_range (vma, range.start, range.end); +if (huge_pmd_unshare (mm, vma, &address, pvmw.pte)) { -/* -* huge_pmd_unshare unmapped an entire PMD -* page. There is no way of knowing exactly -* which PMDs may be cached for this mm, so -* we must flush them all. start/end were -* already adjusted above to cover this range. -*/ -flush_cache_range (vma, range.start, range.end);flush_tlb_range (vma, range.start, range.end);mmu_notifier_invalidate_range (mm, range.start, range.end);@@ -1560,13 +1561,14 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,page_vma_mapped_walk_done (&pvmw);
Quoted 8 lines
mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 1565
Quoted 4 lines
mm/rmap.c +2
-0break ; } + }else { +flush_cache_page (vma, address,pte_pfn (*pvmw.pte));I know this call to
flush_cache_page()existed before your change. But, when looking at this now I wonder how hugetlb pages are handled? Are there any versions offlush_cache_page()that take page size into account?
Thanks for reminding. I checked the flush_cache_page() implementation on some architectures (like arm32), they did not consider the hugetlb pages, so I think we may miss flushing the whole cache for hguetlb pages on some architectures.
With this patch, we can mitigate this issue, since we change to use flush_cache_range() to cover the possible range to flush cache for hugetlb pages. Bur for anon hugetlb pages, we should also convert to use flush_cache_range() instead. I think we can do this conversion in a separate patch set with checking all the places, where using flush_cache_page() to flush cache for hugetlb pages. How do you think?
Re: [PATCH 2/2] mm: rmap: Move the cache flushing to the correct place for hugetlb PMD sharing
Quoted 68 lines · Baolin Wang
Quoted 18 lines · Mike Kravetz
Quoted 17 lines · Baolin Wang
The cache level flush will always be first when changing an existing virtual–>physical mapping to a new value, since this allows us to properly handle systems whose caches are strict and require a virtual–>physical translation to exist for a virtual address. So we should move the cache flushing before
huge_pmd_unshare().As Muchun pointed out[1], now the architectures whose supporting hugetlb PMD sharing have no cache flush issues in practice. But I think we should still follow the cache/TLB flushing rules when changing a valid virtual address mapping in case of potential issues in future.
[1] https://lore.kernel.org/all/YmT%2F%[email protected]/
Signed-off-by Baolin Wang <[email protected]>---
mm/rmap.c| 40 ++++++++++++++++++++++------------------1 file changed, 22 insertions(+), 18 deletions(-)Quoted 13 lines
mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 1539
Quoted 10 lines
mm/rmap.c +2
-0@@ -1535,15 +1535,16 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* do this outside rmap routines. */ VM_BUG_ON(!(flags & TTU_RMAP_LOCKED)); + /* +* huge_pmd_unshare unmapped an entire PMD page. Perhaps update this comment to say that
huge_pmd_unshare'may' unmap an entire PMD page?Sure, will do.
Quoted 22 lines
Quoted 22 lines
mm/rmap.c +7
-8+* There is no way of knowing exactly which PMDs may +* be cached for this mm, so we must flush them all. +* start/end were already adjusted above to cover this +* range. +*/ + flush_cache_range(vma, range.start, range.end); + if (huge_pmd_unshare(mm, vma, &address, pvmw.pte)) { - /* - * huge_pmd_unshare unmapped an entire PMD - * page. There is no way of knowing exactly - * which PMDs may be cached for this mm, so - * we must flush them all. start/end were - * already adjusted above to cover this range. - */ - flush_cache_range(vma, range.start, range.end); flush_tlb_range(vma, range.start, range.end); mmu_notifier_invalidate_range(mm, range.start, range.end);@@ -1560,13 +1561,14 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, page_vma_mapped_walk_done(&pvmw);Quoted 8 lines
mm/rmap.c static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, · line 1565
Quoted 4 lines
mm/rmap.c +2
-0break; } + } else { + flush_cache_page(vma, address, pte_pfn(*pvmw.pte));I know this call to
flush_cache_page()existed before your change. But, when looking at this now I wonder how hugetlb pages are handled? Are there any versions offlush_cache_page()that take page size into account?
Quoted 4 lines
Thanks for reminding. I checked the
flush_cache_page()implementation on some architectures (like arm32), they did not consider the hugetlb pages, so I think we may miss flushing the whole cache for hguetlb pages on some architectures.With this patch, we can mitigate this issue, since we change to use
flush_cache_range()to cover the possible range to flush cache for hugetlb pages. Bur for anon hugetlb pages, we should also convert to useflush_cache_range()instead. I think we can do this conversion in a separate patch set with checking all the places, where usingflush_cache_page()to flush cache for hugetlb pages. How do you think?
Yes, I am OK with that approach.
Signature · 1 line
Mike Kravetz