Batch unmap of uffd-wp file folios
Subsystems (4)
v2 cover Batch unmap of uffd-wp file folios
Currently, batched unmapping is supported if:
1) folio is a file folio, not belonging to uffd-wp VMA
2) folio is anonymous and not swapbacked (lazyfree), not belonging to uffd-wp VMA
So the cases which are not supported are
1) folio belonging to uffd-wp VMA
2) folio is anonymous and swapbacked
It is easy to see that this adds a lot of cognitive load while reading try_to_unmap_one - we need to remember throughout whether nr_pages == 1 or > 1.
The uffd-wp handling in try_to_unmap_one is regarding preserving the uffd-wp state for file folios via pte_install_uffd_wp_if_needed (for anon folio, we handle that while constructing the swap pte).
Stop special casing on uffd-wp VMAs by simply adding batching support to pte_install_uffd_wp_if_needed.
---v1->v2:
- Move declaration to mm/internal.h
- Use two tabs for parameter indentation, use for(;;) established pattern for batch setting of ptes
This is a split from:
https://lore.kernel.org/all/[email protected]/
mm-selftests pass.
Dev Jain (3):
mm/memory: move pte_install_uffd_wp_if_needed() into memory.c mm/memory: batch set uffd-wp markers during zapping mm/rmap: batch unmap file folios belonging to uffd-wp VMAs
include/linux/mm_inline.h | 53 ------------------------ mm/internal.h | 4 ++ mm/memory.c | 85 ++++++++++++++++++++++++++++++--------- mm/rmap.c | 6 +-- 4 files changed, 73 insertions(+), 75 deletions(-)
Signature · 1 line
2.43.0
PATCH v2 1/3 mm/memory: move pte_install_uffd_wp_if_needed() into memory.c
pte_install_uffd_wp_if_needed() has grown too large for mm_inline.h. Move it to memory.c.
This helper is only used inside mm/, so declare it in mm/internal.h instead of a public header.
While at it, convert the comment to kerneldoc and rename the local arguments from pte/pteval to ptep/pte so the pointer and PTE value are easier to distinguish.
--- include/linux/mm_inline.h | 53 ---------------------------------- mm/internal.h | 3 ++ mm/memory.c | 60 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 63 insertions(+), 53 deletions(-)
include/linux/mm_inline.h +0 -53
@@ -566,59 +566,6 @@ static inline pte_marker copy_pte_marker(return dstm; } -/* -* If this pte is wr-protected by uffd-wp in any form, arm the special pte to -* replace a none pte. NOTE! This should only be called when *pte is already -* cleared so we will never accidentally replace something valuable. Meanwhile -* none pte also means we are not demoting the pte so tlb flushed is not needed. -* E.g., when pte cleared the caller should have taken care of the tlb flush. -* -* Must be called with pgtable lock held so that no thread will see the none -* pte, and if they see it, they'll fault and serialize at the pgtable lock. -* -* Returns true if an uffd-wp pte was installed, false otherwise. -*/ -static inline bool -pte_install_uffd_wp_if_needed (struct vm_area_struct *vma,unsigned long addr, -pte_t *pte,pte_t pteval) -{ -bool arm_uffd_pte = false; - -if (!uffd_supports_wp_marker ()) -return false; - -/* The current status of the pte should be "cleared" before calling */ -WARN_ON_ONCE (!pte_none (ptep_get (pte))); - -/* -* NOTE: userfaultfd_wp_unpopulated() doesn't need this whole -* thing, because when zapping either it means it's dropping the -* page, or in TTU where the present pte will be quickly replaced -* with a swap pte. There's no way of leaking the bit. -*/ -if (vma_is_anonymous (vma) || !userfaultfd_wp (vma)) -return false; - -/* A uffd-wp wr-protected normal pte */ -if (unlikely (pte_present (pteval) &&pte_uffd (pteval))) - arm_uffd_pte = true; - -/* -* A uffd-wp wr-protected swap pte. Note: this should even cover an -* existing pte marker with uffd-wp bit set. -*/ -if (unlikely (pte_swp_uffd_any (pteval))) - arm_uffd_pte = true; - -if (unlikely (arm_uffd_pte)) { -set_pte_at (vma->vm_mm, addr, pte, -make_pte_marker (PTE_MARKER_UFFD_WP )); -return true; - } - -return false; -} -static inline bool vma_has_recency (const struct vm_area_struct *vma) {if (vma->vm_flags & (VM_SEQ_READ |VM_RAND_READ ))
mm/internal.h +3 -0
@@ -276,6 +276,9 @@ void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap);#ifdef CONFIG_MMU +bool pte_install_uffd_wp_if_needed (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,pte_t pte); +static inline void get_anon_vma (struct anon_vma *anon_vma) {atomic_inc (&anon_vma->refcount);
mm/memory.c +60 -0
66 lines not on this page · raw message
Signature · 1 line
2.43.0
Re: [PATCH v2 1/3] mm/memory: move pte_install_uffd_wp_if_needed() into memory.c
Quoted 11 lines · Dev Jain
pte_install_uffd_wp_if_needed()has grown too large formm_inline.h. Move it tomemory.c.This helper is only used inside
mm/, so declare it inmm/internal.hinstead of a public header.While at it, convert the comment to kerneldoc and rename the local arguments from pte/pteval to ptep/pte so the pointer and PTE value are easier to distinguish.
Signed-off-by Dev Jain <[email protected]>
Signature · 3 lines
Cheers, David
PATCH v2 2/3 mm/memory: batch set uffd-wp markers during zapping
In reply to Dev JainEnable batch setting of uffd-wp PTE markers.
The code paths passing nr > 1 to zap_install_uffd_wp_if_needed() produce that nr through either folio_pte_batch() or swap_pte_batch(), therefore batching is correct:
1) All PTEs belong to the same type of VMA: anonymous or non-anonymous, wp-armed or non-wp-armed.
2) All PTEs are either marked with uffd-wp or not marked with uffd-wp; the same applies to the pte_swp_uffd_any() check.
3) uffd_supports_wp_marker() is independent of the function parameters.
Use set_pte_at() in a loop instead of set_ptes(), because set_ptes() cannot handle nonpresent to nonpresent conversion for nr_pages > 1.
Rename the helper to cond_install_uffd_wp_ptes().
--- mm/internal.h | 5 +++-- mm/memory.c | 61 +++++++++++++++++++++------------------------------ mm/rmap.c | 2 +- 3 files changed, 29 insertions(+), 39 deletions(-)
mm/internal.h +3 -2
@@ -276,8 +276,9 @@ void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap);#ifdef CONFIG_MMU -bool pte_install_uffd_wp_if_needed (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep,pte_t pte); +bool cond_install_uffd_wp_ptes (struct vm_area_struct *vma, +unsigned long addr,pte_t *ptep,pte_t pte, +unsigned long nr_ptes);static inline void get_anon_vma (struct anon_vma *anon_vma) {
mm/memory.c +25 -36
94 lines not on this page · raw message
mm/rmap.c +1 -1
@@ -2348,7 +2348,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* we may want to replace a none pte with a marker pte if * it's file-backed, so we don't lose the tracking info. */ -pte_install_uffd_wp_if_needed (vma, address, pvmw.pte, pteval); +cond_install_uffd_wp_ptes (vma, address, pvmw.pte, pteval,1 );/* Update high watermark before we lower rss */ update_hiwater_rss (mm);
Signature · 1 line
2.43.0
Re: [PATCH v2 2/3] mm/memory: batch set uffd-wp markers during zapping
Quoted 12 lines · Dev Jain
Enable batch setting of uffd-wp PTE markers.
The code paths passing nr > 1 to
zap_install_uffd_wp_if_needed()produce that nr through eitherfolio_pte_batch()orswap_pte_batch(), therefore batching is correct:1) All PTEs belong to the same type of VMA: anonymous or non-anonymous, wp-armed or non-wp-armed.
2) All PTEs are either marked with uffd-wp or not marked with uffd-wp; the same applies to the
pte_swp_uffd_any()check.
Quoted 8 lines
3)
uffd_supports_wp_marker()is independent of the function parameters.Use
set_pte_at()in a loop instead ofset_ptes(), becauseset_ptes()cannot handle nonpresent to nonpresent conversion fornr_pages> 1.Rename the helper to
cond_install_uffd_wp_ptes().Signed-off-by Dev Jain <[email protected]>
Signature · 3 lines
Cheers, David
PATCH v2 3/3 mm/rmap: batch unmap file folios belonging to uffd-wp VMAs
In reply to Dev JainCommit a67fe41e214f ("mm: rmap: support batched unmapping for file large folios") extended batched unmapping for file folios. That also required making pte_install_uffd_wp_if_needed() support batching, but that was left out for the time being. Correctness was maintained by stopping batching if the VMA the folio belongs to is marked uffd-wp.
Now that cond_install_uffd_wp_ptes() supports batching, call it with the full batch length and allow folio_unmap_pte_batch() to batch file folios belonging to uffd-wp VMAs.
For file folios, if the uffd-wp bit is set, unmapping converts present PTEs into uffd-wp markers. We must ensure that the same PTE range is not reprocessed by the try_to_unmap_one() loop.
The page_vma_mapped_walk API ensures this: check_pte() only returns true if any PFN in [pvmw->pfn, pvmw->pfn + nr_pages) is mapped by the PTE. There is no PFN underlying a uffd-wp marker PTE, so check_pte() returns false and the walk skips ahead until it reaches a present entry again.
--- mm/rmap.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
mm/rmap.c +2 -4
@@ -1968,9 +1968,6 @@ static inline unsigned int folio_unmap_pte_batch(struct folio *folio,if (pte_unused (pte))return 1 ; -if (userfaultfd_protected (vma)) -return 1 ; -/* * If unmap fails, we need to restore the ptes. To avoid accidentally * upgrading write permissions for ptes that were not originally
@@ -2348,7 +2345,8 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma,* we may want to replace a none pte with a marker pte if * it's file-backed, so we don't lose the tracking info. */ -cond_install_uffd_wp_ptes (vma, address, pvmw.pte, pteval,1 ); +cond_install_uffd_wp_ptes (vma, address, pvmw.pte, pteval, + nr_pages);/* Update high watermark before we lower rss */ update_hiwater_rss (mm);
Signature · 1 line
2.43.0