mm-memory-optimize-fork-with-pte-mapped-thp.patch removed from -mm tree
The quilt patch titled
Subject: mm/memory: optimize fork() with PTE-mapped THP
has been removed from the -mm tree. Its filename was
mm-memory-optimize-fork-with-pte-mapped-thp.patch
This patch was dropped because it was merged into the mm-stable branch of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
------------------------------------------------------
From: David Hildenbrand <[email protected]>
Subject: mm/memory: optimize fork() with PTE-mapped THP
Date: Mon, 29 Jan 2024 13:46:47 +0100
Let's implement PTE batching when consecutive (present) PTEs map consecutive pages of the same large folio, and all other PTE bits besides the PFNs are equal.
We will optimize folio_pte_batch() separately, to ignore selected PTE bits. This patch is based on work by Ryan Roberts.
Use __always_inline for __copy_present_ptes() and keep the handling for single PTEs completely separate from the multi-PTE case: we really want the compiler to optimize for the single-PTE case with small folios, to not degrade performance.
Note that PTE batching will never exceed a single page table and will always stay within VMA boundaries.
Further, processing PTE-mapped THP that maybe pinned and have PageAnonExclusive set on at least one subpage should work as expected, but there is room for improvement: We will repeatedly (1) detect a PTE batch
(2) detect that we have to copy a page (3) fall back and allocate a single page to copy a single page. For now we won't care as pinned pages are a corner case, and we should rather look into maintaining only a single PageAnonExclusive bit for large folios.
---
include/linux/pgtable.h | 31 ++++++++++ mm/memory.c | 112 +++++++++++++++++++++++++++++++------- 2 files changed, 124 insertions(+), 19 deletions(-)
include/linux/pgtable.h renamed from include/linux/pgtable.h~mm-memory-optimize-fork-with-pte-mapped-thp +31 -0
@@ -650,6 +650,37 @@ static inline void ptep_set_wrprotect(st }#endif +#ifndef wrprotect_ptes +/** +* wrprotect_ptes - Write-protect PTEs that map consecutive pages of the same +* folio. +* @mm: Address space the pages are mapped into. +* @addr: Address the first page is mapped at. +* @ptep: Page table pointer for the first entry. +* @nr: Number of entries to write-protect. +* +* May be overridden by the architecture; otherwise, implemented as a simple +* loop over ptep_set_wrprotect(). +* +* Note that PTE bits in the PTE range besides the PFN can differ. For example, +* some PTEs might be write-protected. +* +* Context: The caller holds the page table lock. The PTEs map consecutive +* pages that belong to the same folio. The PTEs are all in the same PMD. +*/ +static inline void wrprotect_ptes (struct mm_struct *mm,unsigned long addr, +pte_t *ptep,unsigned int nr) +{ +for (;;) { +ptep_set_wrprotect (mm, addr, ptep); +if (--nr ==0 ) +break ; + ptep++; + addr +=PAGE_SIZE ; + } +} +#endif +/* * On some architectures hardware does not set page access bit when accessing * memory page, it is responsibility of software setting this bit. It brings
mm/memory.c renamed from mm/memory.c~mm-memory-optimize-fork-with-pte-mapped-thp +93 -19
190 lines not on this page · raw message
_
Patches currently in -mm which might be from [email protected] are
mm-memory-factor-out-zapping-of-present-pte-into-zap_present_pte.patch mm-memory-handle-page-case-in-zap_present_pte-separately.patch mm-memory-further-separate-anon-and-pagecache-folio-handling-in-zap_present_pte.patch mm-memory-factor-out-zapping-folio-pte-into-zap_present_folio_pte.patch mm-mmu_gather-pass-delay_rmap-instead-of-encoded-page-to-__tlb_remove_page_size.patch mm-mmu_gather-define-encoded_page_flag_delay_rmap.patch mm-mmu_gather-add-tlb_remove_tlb_entries.patch mm-mmu_gather-add-__tlb_remove_folio_pages.patch mm-mmu_gather-improve-cond_resched-handling-with-large-folios-and-expensive-page-freeing.patch mm-memory-optimize-unmap-zap-with-pte-mapped-thp.patch