mm: fixes for stale TLB entries due to delayed/batched TLB flushes
Subsystems (8)
David Hildenbrand (Arm)
maintainercover mm: fixes for stale TLB entries due to delayed/batched TLB flushes
David, what keeps you up at night? Well, the last 3 nights, TLB flushes.
We have two mechanisms that can delay/batch TLB flushes and cause problems:
(A) mmu_gather zap: we can delay TLB flushes until after dropping the PT lock, after having zapped PTEs and adjusted the mapcounts.
(B) TTU_BATCH_FLUSH when used by reclaim: we can defer (batch) TLB flushes, after having unmapped folios and adjusted the mapcounts, and after having discarded lazyfree folios, removed folios from the page-cache/swap-cache or just dropped the folio lock.
Code that installs new folios (and doesn't just refault the old ones) must be aware of that, otherwise we can end up with stale TLB entries while new folios are already mapped into page tables.
So in general, whenever we install a new folio into the page tables, we must make sure that no such stale TLB entries exist.
Even worse, whenever we replace a folio in the page-cache that might differ content-wise from the old folio, or we end up modifying a new folio through write(), we must make sure that no such stale TLB entries exist.
There are some exceptions (e.g., only once MADV_DONTNEED finished is when a new read from the same address gives any guarantees), so the primary focus is on folios that got unmapped transparently by the system, without involving MADV_DONTNEED.
This series handles the following scenarios:
Patch #1: Zapping: flush the TLB before installing a CoW'ed folio.
Handles CoW of a folio that is still in the page-cache.
Patch #2: Reclaim: flush deferred writable TLB entries before unlocking
the folio. Handles writes through stale TLB entries getting lost when reclaim keeps a dirty folio and somebody else starts writeback.
Patch #3: Reclaim: don't defer TLB flushes when unmapping lazyfree
folios. Handles mapping a new anon folio after reclaim discarded a lazyfree folio (do_anonymous_page(), migrate_vma_insert_page(), UFFDIO_COPY, UFFDIO_MOVE).
Patch #4: Reclaim: unlock unmapped folios only after the deferred TLB
flush. Handles refaulting a new page-cache folio, CoW of a folio that is still in the page-cache, and swapping in a new folio (do_swap_page(), unuse_pte(), UFFDIO_CONTINUE). Also handles invalidation replacing a folio that reclaim kept, and write() modifying a new page-cache folio.
Patches #1 and #2 are small fixes that should be merged as soon as possible. Patch #4 makes the flush added by patch #2 unnecessary by keeping folios locked until the deferred TLB flush, and removes it again.
Patches #3 and #4 are RFC, because #4 is rather intrusive and #3 comes with a performance impact (see below). But there isn't really a good alternative to fix these things for good that I see.
In theory, UFFDIO_COPY installing a new anon folio into a MAP_PRIVATE shmem mapping is affected by concurrent zapping just like CoW faults in patch #1. But the required setups (e.g., userfaultfd on a MAP_PRIVATE mapping of a tmpfs file that backs a loop device) are so unusual that we will leave it alone for now. Same applies to the hugetlb counterpart.
Remaining Problem
-----------------
There is one remaining known problem that is not handled by this series:
Zapping (e.g., unmap_mapping_range()) removes the rmap of clean page-cache folios before performing the deferred TLB flush. If the mapcount drops to 0, a concurrent invalidation (e.g., invalidate_inode_pages2_range()) will skip the unmapped folio and can remove it from the page-cache before the pending TLB flush.
A page fault (except a CoW fault, see patch #1) can then map a new folio, or write() can modify it, while other CPUs still read the old folio through stale TLB entries.
I think that this is low priority: it requires two concurrent invalidations of the same file range while another process has the file mapped, and CPUs can only read stale data until the zapper flushes the TLB; writes are never lost.
It could maybe be fixed in the invalidation code? unmap_mapping_*() holds the i_mmap_rwsem in read mode until it flushed the TLB, so folio_unmap_invalidate() could briefly take it in write mode iff the folio has unexpected references. But dropbehind (RWF_DONTCACHE) invalidates folios on writeback completion while holding an additional reference, where we must not sleep on the i_mmap_rwsem, so that would need some care.
Something for the future :)
Testing
-------
I threw a bunch of selftests at it and had AI generate some stress tests, in particular for the benchmarks below. I also ran stress-ng to force reclaim.
I managed to reproduce the issues in patch #3 and #4, as described in the patch descriptions.
Stale reads out of 1024000 checked pages in 2000 iterations:
base series
lazyfree, MGLRU 16242 0 lazyfree, classic LRU 16274 0 swapcache, MGLRU 24076 0 swapcache, classic LRU 23028 0
Performance Impact
------------------
I ran some benchmarks in an x86-64 VM with limited CPUs, RAM and zram enabled. Running inside a VM might make TLB flushes more expensive, so the results below might be a bit exaggerated. AI results below.
Classic LRU, 4 threads randomly touching a working set twice the size of their memcg for 30 seconds. Pages touched per second:
base series
anon-rw 681k 1055k
anon-ro 1013k 1027k
file-ro 417k 419k file-rw 173k 183k lazyfree 1512k 333k
Swapping out dirty anon folios ("anon-rw") is about 55% faster, because patch #4 now also batches TLB flushes of stale writable TLB entries. That is what Zhang Peng's series [1] tried to achieve, and his work also inspired how patch #4 locks folios again, but the used mechanism is different.
Reclaiming clean anon folios ("anon-ro"), clean page-cache folios ("file-ro") and dirty page-cache folios ("file-rw") is not noticeably affected.
Reclaiming lazyfree folios ("lazyfree") is about 4.5x slower, because we stop batching TLB flushes when discarding them entirely, which seems to be the right thing to do for now. We can look into optimizing this separately later.
[1] https://lore.kernel.org/linux-mm/[email protected]/
---
David Hildenbrand (Arm) (4):
mm: flush stale TLB entries before installing a CoW'ed folio mm/vmscan: flush deferred writable TLB entries before unlocking the folio
RFC: mm/rmap: don't defer TLB flushes when unmapping lazyfree folios
RFC: mm/vmscan: unlock unmapped folios only after the deferred TLB flush
mm/internal.h | 5 ++ mm/memory.c | 6 +++ mm/rmap.c | 20 +++++++- mm/shmem.c | 2 +- mm/swap.h | 5 +- mm/swapfile.c | 7 ++- mm/vmscan.c | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---- 7 files changed, 184 insertions(+), 16 deletions(-)
---
base-commit: 0c524518413094607a6178c53a65815762bc9c45
change-id: 20261114-file_folio_flush-7d1749fab2d8
Best regards,
Signature · 1 line
David Hildenbrand (Arm) <[email protected]>
David Hildenbrand (Arm)
maintainerPATCH 1/4 mm: flush stale TLB entries before installing a CoW'ed folio
Zapping PTEs in zap_pte_range() defers the TLB flush until after dropping the PTL for clean page-cache folios.
A concurrent write fault in a MAP_PRIVATE mapping can then CoW the page-cache folio and install a new anon folio without a prior TLB flush:
CPU0 CPU1 LOAD(ADDR) -> #PF installs R/O PTE -> A fuse_open() -> invalidate_inode_pages2() -> zap_pte_range() clears R/O PTE // delayed flush // Other CPUs (incl CPU0) still cache ADDR -> A STORE(ADDR) -> #PF do_cow_fault() copies A to B, installs R/W PTE -> B // no TLB flush: PTE was none
CPUs with stale TLB entries will still read from the old folio while writes go to the new folio, until the TLB is flushed.
We really must process pending TLB flushes if we are not faulting the same folio back in.
Fix it by flushing the TLB before installing an anon (CoW'ed) folio in set_pte_range() if a TLB flush is pending. Zapping itself stays as fast as before.
Other faults are not affected: unmap_mapping_*() users only remove folios from the page-cache after their own TLB flush, so until then, a refault maps the same folio again. The exception is a concurrent invalidation that removes the already-unmapped folio from the page-cache before the zapper flushed the TLB; that is an unlikely corner case that we'll not handle for now.
Note that we don't care about MADV_DONTNEED followed by a new read fault, as MADV_DONTNEED only guarantees its semantics once the operation completed. CoW faults will flush the TLB either way.
For example, imagine that MADV_DONTNEED zaps a page-cache folio and doesn't flush the TLB (but still holds a reference). If the page-cache replaces the folio, a new read fault might map the new folio without flushing the TLB. That's okay, as MADV_DONTNEED did not complete yet. Same applies to MADV_DONTNEED of CoW'ed folios when re-faulting the page-cache folio.
Memory reclaim, which removes folios from the page-cache before performing its own batched TLB flush, will be handled separately.
--- mm/memory.c | 6 ++++++ 1 file changed, 6 insertions(+)
mm/memory.c +6 -0
@@ -5888,6 +5888,12 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio,/* copy-on-write page */ if (write && !(vma->vm_flags &VM_SHARED )) {VM_BUG_ON_FOLIO (nr !=1 , folio); +/* +* Concurrent zapping might have dropped the PTL but not flushed +* the TLB yet. Flush before installing a new folio. +*/ +if (mm_tlb_flush_pending (vma->vm_mm)) +flush_tlb_page (vma, addr);folio_add_new_anon_rmap (folio, vma, addr,RMAP_EXCLUSIVE );folio_add_lru_vma (folio, vma); }else {
Signature · 1 line
2.56.1
Re: [PATCH 1/4] mm: flush stale TLB entries before installing a CoW'ed folio
David Hildenbrand (Arm)
maintainerPATCH 2/4 mm/vmscan: flush deferred writable TLB entries before unlocking the folio
In reply to David Hildenbrand (Arm)try_to_unmap() with TTU_BATCH_FLUSH transfers the PTE dirty bit to the folio and defers the TLB flush.
Reclaim must flush the TLB before unlocking the folio, otherwise stale writable TLB entries might be left behind temporarily: writeback started by other mechanisms (e.g., fsync) will not find any mappings in folio_mkclean(), so it will neither write-protect nor flush anything.
A write through a stale TLB entry after the folio was cleaned gets lost once the folio is reclaimed.
However, reclaim currently does not flush the TLB if the now-dirty folio does not get written back (e.g., gets activated or kept instead). Fix it by flushing right after try_to_unmap() if the folio is dirty, which makes the existing flush before pageout() unnecessary.
For dirty page-cache folios that reclaim activates, we will now flush the TLB immediately after unmapping them instead of batching the flush, which is the right thing to do. Performance-wise, it shouldn't really matter much, because folios with dirty PTEs are usually young and reclaim won't even try to unmap them.
--- mm/vmscan.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
mm/vmscan.c +6 -6
@@ -1455,6 +1455,12 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, flags |=TTU_SYNC ;try_to_unmap (folio, flags); +/* +* Flush deferred writable TLB entries before we could +* drop the folio lock and others could start writeback. +*/ +if (folio_test_dirty (folio)) +try_to_unmap_flush_dirty ();if (folio_mapped (folio)) { stat->nr_unmap_fail += nr_pages; if (!was_swapbacked &&
@@ -1496,12 +1502,6 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,if (!sc->may_writepage) goto keep_locked; -/* -* Folio is dirty. Flush the TLB if a writable entry -* potentially exists to avoid CPU writes after I/O -* starts and then write it out here. -*/ -try_to_unmap_flush_dirty ();switch (pageout (&ctx, mapping, folio, folio_list)) {case PAGE_KEEP : goto keep_locked;
Signature · 1 line
2.56.1
David Hildenbrand (Arm)
maintainerPATCH 3/4 RFC: mm/rmap: don't defer TLB flushes when unmapping lazyfree folios
In reply to David Hildenbrand (Arm)When reclaim discards a lazyfree folio, it clears the PTE but delays the TLB flush. Until flushed, mapping a new folio into the now-none PTE can leave some CPUs still reading from the old folio while others already write to the new folio.
This is hard to trigger in practice and would mostly look like a weird use-after-free bug in an allocator that uses MADV_FREE, but it can be reproduced reliably with three threads pinned to different CPUs:
(1) Thread A populates the pages, MADV_FREE's them and then MADV_PAGEOUT's them.
(2) Thread B reads all pages to keep TLB entries cached and verifies the content of pages published by thread C.
(3) Thread C waits until mincore() indicates that a page is no longer present, writes a new value to it, and publishes it to thread B.
Thread B can then read the old value after the new folio was mapped and the content updated.
One way to handle it would be adding conditional flushes to do_anonymous_page(), migrate_vma_insert_page(), UFFDIO_COPY and UFFDIO_MOVE ... essentially anywhere we install new anonymous folios, which would mean that these places flush quite frequently on concurrent reclaim.
To perform targeted flushes, we'd have to keep some PTE markers behind, which is also rather nasty and not an easy fix.
So let's not defer the TLB flush for lazyfree folios, which includes folios in VM_DROPPABLE mappings.
To avoid flushing if the folio was redirtied or has additional references, perform all non-deferred TLB flushes only once we know that we won't restore the PTEs, right before removing the rmap.
The downside is that we no longer batch TLB flushes when discarding lazyfree folios, which harms performance, but is the right thing to do for now.
--- mm/rmap.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-)
mm/rmap.c +13 -2
@@ -2228,6 +2228,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,struct mmu_notifier_range range;enum ttu_flags flags = (enum ttu_flags )(long )arg;unsigned long nr_pages =1 , end_addr; +bool flush;unsigned long pfn;int ptes =0 ;
@@ -2253,6 +2254,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,while (page_vma_mapped_walk (&pvmw)) { nr_pages =1 ; + flush = false;/* * If the folio is in an mlock()d vma, we must not swap it out.
@@ -2339,11 +2341,18 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* architecture must guarantee that a clear->dirty * transition on a cached TLB entry is written through * and traps if the PTE is unmapped. +* +* If we don't defer the flush, we flush only once we +* know that we won't restore the PTEs below. We never +* defer the flush for lazyfree folios: if we discard +* them, the PTE will be none, and a page fault could +* map a new folio while stale TLB entries still exist. */ -if (should_defer_flush (mm, flags)) +if (!folio_test_lazyfree (folio) && +should_defer_flush (mm, flags))set_tlb_ubc_flush_pending (mm, pteval, address, end_addr);else -flush_tlb_range (vma, address, end_addr); + flush = true;if (pte_dirty (pteval))folio_mark_dirty (folio); }else {
@@ -2402,6 +2411,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,add_mm_counter (mm,mm_counter_file (folio), -nr_pages); } finish_unmap: +if (flush) +flush_tlb_range (vma, address, end_addr);folio_remove_rmap_ptes (folio, page, nr_pages, vma);if (vma_test_any_mask (vma,VMA_LOCKED_MASK ))mlock_drain_local ();
Signature · 1 line
2.56.1
David Hildenbrand (Arm)
maintainerPATCH 4/4 RFC: mm/vmscan: unlock unmapped folios only after the deferred TLB flush
In reply to David Hildenbrand (Arm)try_to_unmap() with TTU_BATCH_FLUSH zaps the PTEs, transfers the PTE dirty bit to the folio and defers the TLB flush. Until the TLB is flushed, other CPUs might still access the folio through stale TLB entries.
However, reclaim unlocks folios it keeps and removes folios it frees from the page-cache / swap-cache before performing the deferred TLB flush.
Stale writable TLB entries are currently handled by flushing them right after try_to_unmap() if the folio is dirty, so writes through stale TLB entries cannot get lost, but they are also not batched.
Stale read-only TLB entries, on the other hand, are problematic:
(1) A page fault can map a new page-cache folio, or swap in a new folio once reclaim removed the old one from the swap-cache (e.g., a clean swap-cache folio, or a dirty folio mapped by clean PTEs after synchronous swap-out to zram or zswap). Modifications of the new folio (e.g., through write() or page table mappings) are then not visible to CPUs that still read the old folio.
(2) A CoW fault in a MAP_PRIVATE mapping can copy a page-cache folio that reclaim kept. Writes go to the new anon folio, while other CPUs still read the old folio.
(3) Invalidation (e.g., invalidate_inode_pages2()) can remove a folio that reclaim kept from the page-cache without flushing the TLB, as the folio is no longer mapped. A page fault can then map a new folio while other CPUs still read the old folio.
This can be reproduced easily by swapping out an anon memory range with MADV_PAGEOUT, reading it back in, and swapping it out again (MADV_PAGEOUT) while one thread keeps reading the range and another thread writes to pages as soon as reclaim removed the folios from the swap-cache. The first thread will occasionally read the old content.
Fix it by reclaiming folios in batches of up to 64 folios:
(a) Lock each folio, check whether we want to reclaim it, and unmap it. Keep the folio locked if any deferred TLB flush is pending after unmapping it, otherwise unlock it right away.
(b) Flush the TLB once, and unlock all folios that are still locked.
(c) Lock the folios again, one at a time, to try reclaiming them. Activate folios that are still or again mapped, and skip folios that are under writeback or poisoned by now, or that somebody else holds locked.
As others might map or unmap folios while they are unlocked, we might now activate or reclaim some folios that we would previously have kept or activated, which is fine.
Keeping folios locked until the TLB flush makes sure that nobody can remove them from the page-cache / swap-cache, copy them or start writeback. Page faults will either wait on the folio lock, or find the old folio and map it again, which is fine.
Unlocking folios before trying to reclaim them makes sure that we never hold more than one folio lock while performing I/O or waiting on the filesystem.
The idea of unlocking folios and locking them again using a trylock before trying to reclaim them was inspired by [1] from Zhang Peng.
As nobody can start writeback before the TLB flush, we no longer have to flush stale writable TLB entries right after try_to_unmap(), so remove the try_to_unmap_flush_dirty() call and batch these flushes as well. This effectively enables batching TLB flushes of stale writable TLB entries, which is what [1] tried to achieve.
We must not wait for a folio lock, a folio flag or I/O while keeping other folios locked, as that could deadlock. So flush the TLB and unlock the batch early before waiting for writeback (legacy memcg), and before allocating swap space would perform synchronous discard I/O (folio_alloc_swap() now returns -EBUSY if allocation failed but retrying with synchronous discard might help).
Allocating swap space can otherwise only sleep when allocating memory for the swap table, which cannot enter direct reclaim or compaction, as reclaim runs with PF_MEMALLOC.
We'll limit batching to 64 folios for now; the limit mostly matters for, e.g., MADV_PAGEOUT and DAMON that can pass more, or when reclaim splits large folios.
Note that khugepaged and batched (async) folio migration already keep up to HPAGE_PMD_NR (e.g., 512 with 4 KiB pages) small folios locked after unmapping them with a deferred TLB flush; migration even allocates new folios, possibly entering direct reclaim, before flushing the TLB and unlocking them. Both also call MMU notifiers while keeping other folios locked, just like we do now.
Keep code churn to a minimum. Some follow-up cleanups and optimizations make sense, in particular:
(1) Cleanly separate the stages of shrink_folio_list() (see the TODO).
(2) Avoid unlocking and locking folios again that do not require a TLB flush. But only if it's worth it and doesn't make the code too ugly.
(3) Remove the remaining try_to_unmap_flush() calls before freeing folios; we only free folios that we flushed already or that we never unmapped.
(4) Remove the now-unused try_to_unmap_flush_dirty() infrastructure.
[1] https://lore.kernel.org/linux-mm/[email protected]/
--- mm/internal.h | 5 ++ mm/rmap.c | 5 ++ mm/shmem.c | 2 +- mm/swap.h | 5 +- mm/swapfile.c | 7 ++- mm/vmscan.c | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++++---- 6 files changed, 165 insertions(+), 14 deletions(-)
mm/internal.h +5 -0
@@ -1253,6 +1253,7 @@ extern struct workqueue_struct *mm_percpu_wq;#ifdef CONFIG_ARCH_WANT_BATCHED_UNMAP_TLB_FLUSH void try_to_unmap_flush (void );void try_to_unmap_flush_dirty (void ); +bool try_to_unmap_flush_pending (void );void flush_tlb_batched_pending (struct mm_struct *mm);#else static inline void try_to_unmap_flush (void )
@@ -1261,6 +1262,10 @@ static inline void try_to_unmap_flush(void)static inline void try_to_unmap_flush_dirty (void ) { } +static inline bool try_to_unmap_flush_pending (void ) +{ +return false; +}static inline void flush_tlb_batched_pending (struct mm_struct *mm) { }
mm/rmap.c +5 -0
@@ -731,6 +731,11 @@ void try_to_unmap_flush_dirty(void)try_to_unmap_flush (); } +bool try_to_unmap_flush_pending (void ) +{ +return current->tlb_ubc.flush_required; +} +/* * Bits 0-14 of mm->tlb_flush_batched record pending generations. * Bits 16-30 of mm->tlb_flush_batched bit record flushed generations.
mm/shmem.c +1 -1
@@ -1936,7 +1936,7 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,folio_mark_uptodate (folio); } - ret =folio_alloc_swap (folio); + ret =folio_alloc_swap (folio,/* may_sync_discard= */ true);if (!ret) {bool first_swapped =shmem_recalc_inode (inode,0 , nr_pages);int error;
mm/swap.h +3 -2
@@ -243,7 +243,7 @@ extern int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp);* * folio_put_swap(): does the opposite thing of folio_dup_swap(). */ -int folio_alloc_swap (struct folio *folio); +int folio_alloc_swap (struct folio *folio,bool may_sync_discard);int folio_dup_swap (struct folio *folio,struct page *page);void folio_put_swap (struct folio *folio,struct page *page);
@@ -363,7 +363,8 @@ static inline struct swap_info_struct *__swap_entry_to_info(swp_entry_t entry)return NULL ; } -static inline int folio_alloc_swap (struct folio *folio) +static inline int folio_alloc_swap (struct folio *folio, +bool may_sync_discard) {return -EINVAL ; }
mm/swapfile.c +6 -1
@@ -1732,16 +1732,19 @@ static int swap_dup_entries_cluster(struct swap_info_struct *si,/** * folio_alloc_swap - allocate swap space for a folio * @folio: folio we want to move to swap +* @may_sync_discard: whether we may wait for synchronous discard I/O * * Allocate swap space for the folio and add the folio to the * swap cache. * * Context: Caller needs to hold the folio lock. * Return: %0 on success, %-E2BIG if splitting the folio might allow swapout, +* %-EBUSY if allocation failed but retrying with @may_sync_discard might +* succeed, * %-ENOSPC if no global swap space is available, or %-ENOMEM if splitting * would not help. */ -int folio_alloc_swap (struct folio *folio) +int folio_alloc_swap (struct folio *folio,bool may_sync_discard) {unsigned int order =folio_order (folio);unsigned int size =1 << order;
@@ -1774,6 +1777,8 @@ int folio_alloc_swap(struct folio *folio)local_unlock (&percpu_swap_cluster.lock);if (!order &&unlikely (!folio_test_swapcache (folio))) { +if (!may_sync_discard) +return -EBUSY ;if (swap_sync_discard ()) goto again; }
mm/vmscan.c +145 -10
213 lines not on this page · raw message
Signature · 1 line
2.56.1