Skip to content

mm/memory: reuse the whole exclusive large folio on a write fault

Yuan-Hao Hsu linux-mm lkml 2 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v2 Latest2 patches 9 repliesIn review
  2. v11 patch 5 repliesSuperseded

Patches in v2 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2mm/memory: reuse 16 PTEs of an exclusive large folio on a write faultmm/memory.c+73 -22 replies
2/2mm/memory: reuse the whole exclusive large folio on a write faultmm/memory.c+7 -143 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v2 yet.

Read the v2 thread

Cover letter [PATCH v2 0/2] mm/memory: reuse the whole exclusive large folio on a write fault

After fork() and the child's exit, the parent's write faults reuse a large anonymous folio one PTE at a time, although wp_can_reuse_anon_folio() has already found the whole folio exclusive.

v2, after the comments on v1 [1]:
- Split in two. 1/2 handles an aligned block of 16 PTEs around the fault, the contpte-sized version David was fine with in the 2024 discussion of Barry's RFC [2]. 2/2 lifts that to the folio.
- The earlier discussion is linked; the changelogs say how its two reservations, the latency of the individual fault and how far to go around it, are answered.
- The bound of the walk is spelled out (Barry): folio, VMA and block or page table, each PTE read once by folio_pte_batch_flags().
- Changelogs cut to what is needed to judge the change.
- Same base as v1; 1/2 + 2/2 is the code of v1.

Two points from the AI review of v1, both done as mprotect() does them:
change_pte_range() does not flush_cache_range() before making PTEs writable, and it makes clean exclusive anonymous PTEs writable without pte_mkdirty() (the dirty rule is for shared file mappings, see can_change_shared_pte_writable()).

Controls, unchanged: order-0 pages, PMD-mapped THPs, the COW copy path and the order-0 and cow/fork/write-fault modes of David's pte-mapped-folio-benchmarks.

What does not get faster on x86: stores to pages this CPU still holds a read-only TLB entry for. The fault makes the PTEs writable but, like mprotect(), does not flush, so such a page takes one spurious fault, about the cost of the reuse fault it replaces. That is the case for pages read since fork(), and for a loop that only stores one byte per page in ascending order (David's reuse-byte mode): the CPU runs the next stores speculatively while the first one faults and caches their read-only translations. Shown with kprobes (135,687 handle_mm_fault() for 8,457 do_wp_page()) and an LFENCE after every store (4,100 faults instead of 65,400); the untouched PMD-mapped case behaves the same. A flush_tlb_local() in the helper would fix it (that loop 28 -> 4 ms, memset() 38 -> 21 ms, +140 ns per fault), but generic code has no way to ask x86 for a flush that stays on this CPU, so that is for later.

Tested with DEBUG_VM, DEBUG_VM_PGTABLE, PROVE_LOCKING and
PAGE_TABLE_CHECK_ENFORCED: the mm selftests, a 12-scenario COW test (child alive, vmsplice, PROT_READ VMA inside the folio, soft-dirty and uffd-wp counts, mremap, holes, pageout, FOLL_FORCE), a fork/pageout/mprotect/vmsplice stress, NUMA balancing on numa=fake=2 (protnone PTEs left alone), and arm64 under QEMU for the counters. Cross-built for arm64 4K/16K/64K, i386 with and without PAE, x86 without THP, arm, arm nommu, riscv64, powerpc64le and s390x.

[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]

Yuan-Hao Hsu (2):
mm/memory: reuse 16 PTEs of an exclusive large folio on a write fault mm/memory: reuse the whole exclusive large folio on a write fault

 mm/memory.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 66 insertions(+), 2 deletions(-)

base-commit: 238650ef6c7c7cca08e032527329424c9fbd70e5

Signature · 1 line
2.43.0