mm: introduce anon_vma flags, reduce kernel allocs
Subsystems (13)
cover mm: introduce anon_vma flags, reduce kernel allocs
VMA resources are scarce. This is a data structure whose weight we wish to reduce (certainly as slab allocations are unreclaimable and - for now - unmigratable).
So adding additional fields is generally unviable, and VMA flags are equally as contended, and prevent VMA merge, further impacting overhead.
We can however make use of the time-honoured kernel tradition of grabbing bits where we can.
Since we can rely upon anon_vma allocations being at least system word-aligned, we have a handful of bits in the vma->anon_vma available to use as flags.
In this series we establish doing so, and immediately use this to solve a problem encountered as part of the guard region feature (MADV_GUARD_INSTALL, MADV_GUARD_REMOVE).
We absolutely must preserve guard regions over fork, however it turns out the only reasonable means of doing so is to establish an anon_vma even if the VMA is unfaulted.
This creates unnecessary overhead, a problem extenuated by the extension of this functionality to file-backed regions, where such-allocated memory may never be utilised or freed until the end of the VMA's lifetime.
We can avoid this if we have a means of indicating to fork that we wish to copy page tables without having to have this overhead.
Having flags available in vma->anon_vma allows us to do so - we can therefore introduce a flag, ANON_VMA_UNFAULTED, which indicates that this is the case.
We introduce wrapper functions to mask off these bits, and nearly every part of the kernel behaves precisely the same as a result, with only the desired change in behaviour in the forking logic.
On fault, or any operation that actually requires an established anon_vma, the ANON_VMA_UNFAULTED flag is cleared and replaced by an actual anon_vma.
An additional advantage of having this mechanism is that we can also remove this flag, should no 'real' anon_vma be established, and the user is executing MADV_GUARD_REMOVE on the whole VMA, meaning we can prevent future unneeded page table operations.
A benefit of this change, aside from saving kernel memory allocations, is that THP page collapse is no longer impacted if we apply guard regions then remove them in their entirety from a VMA, as otherwise the immediate collapse of aligned page tables in retract_page_tables() cannot proceed.
Lorenzo Stoakes (2):
mm: introduce anon_vma flags and use wrapper functions mm/madvise: utilise anon_vma unfaulted flag on guard region install
fs/coredump.c | 2 +- include/linux/mm_types.h | 67 ++++++++++++++++++++- include/linux/rmap.h | 4 +- kernel/fork.c | 4 +- mm/debug.c | 6 +- mm/huge_memory.c | 4 +- mm/khugepaged.c | 12 ++-- mm/ksm.c | 16 +++--- mm/madvise.c | 49 ++++++++++------ mm/memory.c | 6 +- mm/mmap.c | 2 +- mm/mprotect.c | 2 +- mm/mremap.c | 8 +-- mm/rmap.c | 42 +++++++------- mm/swapfile.c | 2 +- mm/userfaultfd.c | 2 +- mm/vma.c | 99 +++++++++++++++++++++++++------- mm/vma.h | 6 +- security/selinux/hooks.c | 2 +- tools/testing/vma/vma.c | 95 +++++++++++++++--------------- tools/testing/vma/vma_internal.h | 78 ++++++++++++++++++++++--- 21 files changed, 358 insertions(+), 150 deletions(-)
--
2.48.1
RFC PATCH 1/2 mm: introduce anon_vma flags and use wrapper functions
VMAs are a highly contested resource, one which we are proactively trying to reduce in size as much as possible. This leaves us with very little means of encoding additional state.
However, we can reclaim lower bits of aligned pointers for our purposes, an established pattern in the kernel.
The vma->anon_vma filed will always be populated with an at least system word-aligned pointer which grants us a handful of bits.
We can then utilise these bits for a very useful purpose - being able to indicate that an 'unfaulted' anon_vma state exists.
This is useful specifically for the purpose of being able to indicate that, on fork, page tables ought to be copied, which currently otherwise requires that a full anon_vma and all associated state need be initialised to be able to do so.
VMA flags are hugely contended (and we have run out of space for them for 32-bit systems), so this seems a sensible means of locating this information.
We do so via the ANON_VMA_UNFAULTED flag. This flag is cleared on fault or any operation that would otherwise assign or unconditionally clear vma->anon_vma.
There is no impact on any kernel behaviour other than forking.
We only read/write these values at a time and under locking conditions where previously we would have done so.
We convert the vma->anon_vma field to an unsigned long to reduce the instances in which inappropriate use of this field might go unnoticed and to encourage correct wrapper function utilisation.
The wrappers are succinct and the meaning of them is clear. We also take the opportunity to use READ_ONCE() / WRITE_ONCE() to prevent any unanticipated tearing.
With this in place, we can subsequently update the guard region implementation to utilise this new functionality, therefore proactively avoiding unnecessary memory usage upon installation of guard regions.
This also opens the door to future changes to anon_vma usage as we are now able to encode additional state here.
We also introduce the vma_maybe_has_pagetables() function to make it abundantly and explicitly clear what this checks, differentiating existing vma->anon_vma checks between the two meaningful cases - checking whether there is an anon_vma and checking if page tables might exist.
We update the two cases which do the latter - the fork test, and khugepaged. In both cases the same kernel behaviour will manifest, only now spelled out clearly.
Additionally, update the VMA unit tests to reflect these changes.
--- fs/coredump.c | 2 +- include/linux/mm_types.h | 67 ++++++++++++++++++++- include/linux/rmap.h | 4 +- kernel/fork.c | 4 +- mm/debug.c | 6 +- mm/huge_memory.c | 4 +- mm/khugepaged.c | 12 ++-- mm/ksm.c | 16 +++--- mm/memory.c | 6 +- mm/mmap.c | 2 +- mm/mprotect.c | 2 +- mm/mremap.c | 8 +-- mm/rmap.c | 42 +++++++------- mm/swapfile.c | 2 +- mm/userfaultfd.c | 2 +- mm/vma.c | 99 +++++++++++++++++++++++++------- mm/vma.h | 6 +- security/selinux/hooks.c | 2 +- tools/testing/vma/vma.c | 95 +++++++++++++++--------------- tools/testing/vma/vma_internal.h | 78 ++++++++++++++++++++++--- 20 files changed, 327 insertions(+), 132 deletions(-)
fs/coredump.c +1 -1
@@ -1115,7 +1115,7 @@ static unsigned long vma_dump_size(struct vm_area_struct *vma, }/* Dump segments that have been written to. */ -if ((!IS_ENABLED (CONFIG_MMU ) || vma->anon_vma) &&FILTER (ANON_PRIVATE )) +if ((!IS_ENABLED (CONFIG_MMU ) ||vma_anon_vma (vma)) &&FILTER (ANON_PRIVATE )) goto whole;if (vma->vm_file ==NULL )return 0 ;
include/linux/mm_types.h +66 -1
79 lines not on this page · raw message
include/linux/rmap.h +2 -2
@@ -159,7 +159,7 @@ int anon_vma_fork(struct vm_area_struct *, struct vm_area_struct *);static inline int anon_vma_prepare (struct vm_area_struct *vma) { -if (likely (vma->anon_vma)) +if (likely (vma_anon_vma (vma)))return 0 ;return __anon_vma_prepare (vma);
@@ -168,7 +168,7 @@ static inline int anon_vma_prepare(struct vm_area_struct *vma)static inline void anon_vma_merge (struct vm_area_struct *vma,struct vm_area_struct *next) { -VM_BUG_ON_VMA (vma->anon_vma != next->anon_vma, vma); +VM_BUG_ON_VMA (vma_anon_vma (vma) !=vma_anon_vma (next), vma);unlink_anon_vmas (next); }
kernel/fork.c +2 -2
@@ -456,7 +456,7 @@ static void vm_area_init_from(const struct vm_area_struct *src, dest->vm_ops = src->vm_ops; dest->vm_start = src->vm_start; dest->vm_end = src->vm_end; - dest->anon_vma = src->anon_vma; +vma_dup_anon_vma (dest, src); dest->vm_pgoff = src->vm_pgoff; dest->vm_file = src->vm_file; dest->vm_private_data = src->vm_private_data;
@@ -683,7 +683,7 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,* Don't prepare anon_vma until fault since we don't * copy page for current vma. */ - tmp->anon_vma =NULL ; +vma_clear_anon_vma (tmp); }else if (anon_vma_fork (tmp, mpnt)) goto fail_nomem_anon_vma_fork;vm_flags_clear (tmp,VM_LOCKED_MASK );
mm/debug.c +3 -3
@@ -187,7 +187,7 @@ EXPORT_SYMBOL(dump_page);void dump_vma (const struct vm_area_struct *vma) {pr_emerg ("vma %px start %px end %px mm %px\n" -"prot %lx anon_vma %px vm_ops %px\n" +"prot %lx anon_vma %px (flags %u) vm_ops %px\n" "pgoff %lx file %px private_data %px\n" #ifdef CONFIG_PER_VMA_LOCK "refcnt %x\n"
@@ -195,8 +195,8 @@ void dump_vma(const struct vm_area_struct *vma)"flags: %#lx(%pGv)\n" , vma, (void *)vma->vm_start, (void *)vma->vm_end, vma->vm_mm, (unsigned long )pgprot_val (vma->vm_page_prot), - vma->anon_vma, vma->vm_ops, vma->vm_pgoff, - vma->vm_file, vma->vm_private_data, +vma_anon_vma (vma),vma_anon_vma_flags (vma), vma->vm_ops, + vma->vm_pgoff, vma->vm_file, vma->vm_private_data, #ifdef CONFIG_PER_VMA_LOCK refcount_read(&vma->vm_refcnt), #endif
mm/huge_memory.c +2 -2
@@ -202,7 +202,7 @@ unsigned long __thp_vma_allowable_orders(struct vm_area_struct *vma,* Allow page fault since anon_vma may be not initialized until * the first page fault. */ -if (!vma->anon_vma) +if (!vma_anon_vma (vma))return (smaps || in_pf) ? orders :0 ;return orders;
@@ -1930,7 +1930,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf)pmd_t orig_pmd = vmf->orig_pmd; vmf->ptl =pmd_lockptr (vma->vm_mm, vmf->pmd); -VM_BUG_ON_VMA (!vma->anon_vma, vma); +VM_BUG_ON_VMA (!vma_anon_vma (vma), vma);if (is_huge_zero_pmd (orig_pmd)) {vm_fault_t ret =do_huge_zero_wp_pmd (vmf);
mm/khugepaged.c +6 -6
@@ -943,7 +943,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address,* thp_vma_allowable_order may return true for qualified file * vmas. */ -if (expect_anon && (!(*vmap)->anon_vma || !vma_is_anonymous (*vmap))) +if (expect_anon && (!vma_anon_vma (*vmap) || !vma_is_anonymous (*vmap)))return SCAN_PAGE_ANON ;return SCAN_SUCCEED ; }
@@ -1173,7 +1173,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_up_write;vma_start_write (vma); -anon_vma_lock_write (vma->anon_vma); +anon_vma_lock_write (vma_anon_vma (vma));mmu_notifier_range_init (&range,MMU_NOTIFY_CLEAR ,0 , mm, address, address +HPAGE_PMD_SIZE );
@@ -1214,7 +1214,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,*/ pmd_populate (mm, pmd,pmd_pgtable (_pmd));spin_unlock (pmd_ptl); -anon_vma_unlock_write (vma->anon_vma); +anon_vma_unlock_write (vma_anon_vma (vma)); goto out_up_write; }
@@ -1222,7 +1222,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address,* All pages are isolated and locked so anon_vma rmap * can't run anymore. */ -anon_vma_unlock_write (vma->anon_vma); +anon_vma_unlock_write (vma_anon_vma (vma)); result = __collapse_huge_page_copy(pte, folio, pmd, _pmd, vma, address, pte_ptl,
@@ -1730,7 +1730,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)* got written to. These VMAs are likely not worth removing * page tables from, as PMD-mapping is likely to be split later. */ -if (READ_ONCE (vma->anon_vma)) +if (vma_maybe_has_pagetables (vma))continue ; addr = vma->vm_start + ((pgoff - vma->vm_pgoff) <<PAGE_SHIFT );
@@ -1786,7 +1786,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)* repeating the anon_vma check protects from one category, * and repeating the userfaultfd_wp() check from another. */ -if (likely (!vma->anon_vma && !userfaultfd_wp (vma))) { +if (likely (!vma_maybe_has_pagetables (vma) && !userfaultfd_wp (vma))) { pgt_pmd =pmdp_collapse_flush (vma, addr, pmd);pmdp_get_lockless_sync (); success = true;
mm/ksm.c +8 -8
@@ -706,7 +706,7 @@ static struct vm_area_struct *find_mergeable_vma(struct mm_struct *mm,if (ksm_test_exit (mm))return NULL ; vma =vma_lookup (mm, addr); -if (!vma || !(vma->vm_flags &VM_MERGEABLE ) || !vma->anon_vma) +if (!vma || !(vma->vm_flags &VM_MERGEABLE ) || !vma_anon_vma (vma))return NULL ;return vma; }
@@ -1191,7 +1191,7 @@ static int unmerge_and_remove_all_rmap_items(void) goto mm_exiting;for_each_vma (vmi, vma) { -if (!(vma->vm_flags &VM_MERGEABLE ) || !vma->anon_vma) +if (!(vma->vm_flags &VM_MERGEABLE ) || !vma_anon_vma (vma))continue ; err =unmerge_ksm_pages (vma, vma->vm_start, vma->vm_end, false);
@@ -1570,8 +1570,8 @@ static int try_to_merge_with_ksm_page(struct ksm_rmap_item *rmap_item,remove_rmap_item_from_tree (rmap_item);/* Must get reference to anon_vma while still holding mmap_lock */ - rmap_item->anon_vma = vma->anon_vma; -get_anon_vma (vma->anon_vma); + rmap_item->anon_vma =vma_anon_vma (vma); +get_anon_vma (vma_anon_vma (vma)); out:mmap_read_unlock (mm); trace_ksm_merge_with_ksm_page(kpage, page_to_pfn(kpage ? kpage : page),
@@ -2537,7 +2537,7 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)continue ;if (ksm_scan.address < vma->vm_start) ksm_scan.address = vma->vm_start; -if (!vma->anon_vma) +if (!vma_anon_vma (vma)) ksm_scan.address = vma->vm_end;while (ksm_scan.address < vma->vm_end) {
@@ -2714,7 +2714,7 @@ static int __ksm_del_vma(struct vm_area_struct *vma)if (!(vma->vm_flags &VM_MERGEABLE ))return 0 ; -if (vma->anon_vma) { +if (vma_anon_vma (vma)) { err =unmerge_ksm_pages (vma, vma->vm_start, vma->vm_end, true);if (err)return err;
@@ -2852,7 +2852,7 @@ int ksm_madvise(struct vm_area_struct *vma, unsigned long start,if (!(*vm_flags &VM_MERGEABLE ))return 0 ;/* just ignore the advice */ -if (vma->anon_vma) { +if (vma_anon_vma (vma)) { err =unmerge_ksm_pages (vma, start, end, true);if (err)return err;
@@ -2969,7 +2969,7 @@ struct folio *ksm_might_need_to_copy(struct folio *folio, }else if (!anon_vma) {return folio;/* no need to copy it */ }else if (folio->index ==linear_page_index (vma, addr) && - anon_vma->root == vma->anon_vma->root) { + anon_vma->root ==vma_anon_vma (vma)->root) {return folio;/* still no need to copy it */ }if (PageHWPoison (page))
mm/memory.c +3 -3
@@ -536,7 +536,7 @@ static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr,if (page)dump_page (page,"bad pte" );pr_alert ("addr:%px vm_flags:%08lx anon_vma:%px mapping:%px index:%lx\n" , - (void *)addr, vma->vm_flags, vma->anon_vma, mapping, index); + (void *)addr, vma->vm_flags,vma_anon_vma (vma), mapping, index); pr_alert("file:%pD fault:%ps mmap:%ps read_folio:%ps\n", vma->vm_file, vma->vm_ops ? vma->vm_ops->fault : NULL,
@@ -1359,7 +1359,7 @@ vma_needs_copy(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma)if (src_vma->vm_flags & (VM_PFNMAP |VM_MIXEDMAP ))return true; -if (src_vma->anon_vma) +if (vma_maybe_has_pagetables (src_vma))return true;/*
@@ -3445,7 +3445,7 @@ vm_fault_t __vmf_anon_prepare(struct vm_fault *vmf)struct vm_area_struct *vma = vmf->vma;vm_fault_t ret =0 ; -if (likely (vma->anon_vma)) +if (likely (vma_anon_vma (vma)))return 0 ;if (vmf->flags &FAULT_FLAG_VMA_LOCK ) {if (!mmap_read_trylock (vma->vm_mm))
mm/mmap.c +1 -1
@@ -1350,7 +1350,7 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma)* Similarly in do_mmap and in do_brk_flags. */ if (vma_is_anonymous (vma)) { -BUG_ON (vma->anon_vma); +BUG_ON (vma_anon_vma (vma)); vma->vm_pgoff = vma->vm_start >>PAGE_SHIFT ; }
mm/mprotect.c +1 -1
@@ -651,7 +651,7 @@ mprotect_fixup(struct vma_iterator *vmi, struct mmu_gather *tlb, newflags |=VM_ACCOUNT ; } }else if ((oldflags &VM_ACCOUNT ) &&vma_is_anonymous (vma) && - !vma->anon_vma) { + !vma_anon_vma (vma)) { newflags &= ~VM_ACCOUNT ; }
mm/mremap.c +4 -4
@@ -142,14 +142,14 @@ static void take_rmap_locks(struct vm_area_struct *vma) {if (vma->vm_file)i_mmap_lock_write (vma->vm_file->f_mapping); -if (vma->anon_vma) -anon_vma_lock_write (vma->anon_vma); +if (vma_anon_vma (vma)) +anon_vma_lock_write (vma_anon_vma (vma)); }static void drop_rmap_locks (struct vm_area_struct *vma) { -if (vma->anon_vma) -anon_vma_unlock_write (vma->anon_vma); +if (vma_anon_vma (vma)) +anon_vma_unlock_write (vma_anon_vma (vma));if (vma->vm_file)i_mmap_unlock_write (vma->vm_file->f_mapping); }
mm/rmap.c +21 -21
125 lines not on this page · raw message
mm/swapfile.c +1 -1
@@ -2234,7 +2234,7 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type)mmap_read_lock (mm);for_each_vma (vmi, vma) { -if (vma->anon_vma && !is_vm_hugetlb_page (vma)) { +if (vma_anon_vma (vma) && !is_vm_hugetlb_page (vma)) { ret =unuse_vma (vma, type);if (ret)break ;
mm/userfaultfd.c +1 -1
@@ -77,7 +77,7 @@ static struct vm_area_struct *uffd_lock_vma(struct mm_struct *mm,* We know we're going to need to use anon_vma, so check * that early. */ -if (!(vma->vm_flags &VM_SHARED ) &&unlikely (!vma->anon_vma)) +if (!(vma->vm_flags &VM_SHARED ) &&unlikely (!vma_anon_vma (vma)))vma_end_read (vma);else return vma;
mm/vma.c +79 -20
208 lines not on this page · raw message
mm/vma.h +5 -1
@@ -174,7 +174,7 @@ static inline pgoff_t vma_pgoff_offset(struct vm_area_struct *vma, .flags = vma_->vm_flags, \ .pgoff = vma_pgoff_offset(vma_, start_), \ .file = vma_->vm_file, \ - .anon_vma = vma_->anon_vma, \ + .anon_vma = vma_anon_vma(vma_), \ .policy = vma_policy(vma_), \ .uffd_ctx = vma_->vm_userfaultfd_ctx, \ .anon_name = anon_vma_name(vma_), \
@@ -265,6 +265,10 @@ __must_check struct vm_area_structstruct vm_area_struct *vma, unsigned long delta); +/* Functions for setting/clearing vma->anon_vma unfaulted flag. */ +void vma_set_anon_vma_unfaulted (struct vm_area_struct *vma); +void vma_clear_anon_vma_unfaulted (struct vm_area_struct *vma); +void unlink_file_vma_batch_init (struct unlink_vma_file_batch *vb);void unlink_file_vma_batch_final (struct unlink_vma_file_batch *vb);
security/selinux/hooks.c +1 -1
@@ -3869,7 +3869,7 @@ static int selinux_file_mprotect(struct vm_area_struct *vma, vma_is_stack_for_current(vma))) { rc =avc_has_perm (sid, sid,SECCLASS_PROCESS ,PROCESS__EXECSTACK ,NULL ); - } else if (vma->vm_file && vma->anon_vma) { + } else if (vma->vm_file && vma_anon_vma(vma)) {/* * We are making executable a file mapping that has * had some COW done. Since pages might have been
tools/testing/vma/vma.c +49 -46
321 lines not on this page · raw message
tools/testing/vma/vma_internal.h +71 -7
126 lines not on this page · raw message
Signature · 1 line
2.48.1
RFC PATCH 2/2 mm/madvise: utilise anon_vma unfaulted flag on guard region install
In reply to Lorenzo StoakesWe have introduced the ability to indicate that a VMA's anon_vma field is 'unfaulted', that is that we have a desire for there to be propagation of page tables on fork, but no anon_vma is yet initialised.
Utilise that on guard region installation (via MADV_GUARD_INSTALL) to ensure that page table propagation on fork occurs, but without occupying one byte of memory more than is required.
Note that this is a no-op if a 'real' anon_vma is already in place.
This also avoids any issue with THP inferring that it should not immediately attempt huge page collapse.
More importantly, for file-backed mappings, this avoids otherwise unnecessary kernel memory allocation purely for the purposes of indicating on-fork page table propagation requirements.
We adjust when we do this, so we do it only after a successful guard region installation and one which installs a guard region of at least one page in size.
This means we only set this once guard regions are definitely installed. We are safe from a fork racing here, because we hold the mmap read lock, and the fork requires a write lock.
We also adjust MADV_GUARD_REMOVE to remove this flag if the range specified spans the entire VMA (and no 'real' anon_vma has been installed yet), meaning we do not cause unnecessary page table propagation.
This is protected from racing with guard region installation through use of the mm->page_table_lock, which is being used to prevent races between mmap read-locked modifiers of vma->anon_vma.
--- mm/madvise.c | 49 +++++++++++++++++++++++++++++++------------------ 1 file changed, 31 insertions(+), 18 deletions(-)
mm/madvise.c +31 -18
@@ -1119,24 +1119,14 @@ static long madvise_guard_install(struct vm_area_struct *vma,struct vm_area_struct **prev, unsigned long start, unsigned long end) { - long err; + long err = 0; +unsigned long nr_pages;int i; *prev = vma;if (!is_valid_guard_vma (vma,/* allow_locked = */ false))return -EINVAL ; -/* -* If we install guard markers, then the range is no longer -* empty from a page table perspective and therefore it's -* appropriate to have an anon_vma. -* -* This ensures that on fork, we copy page tables correctly. -*/ - err =anon_vma_prepare (vma); -if (err) -return err; -/* * Optimistically try to install the guard marker pages first. If any * non-guard pages are encountered, give up and zap the range before
@@ -1150,19 +1140,20 @@ static long madvise_guard_install(struct vm_area_struct *vma,* with no zap or looping. */ for (i =0 ; i <MAX_MADVISE_GUARD_RETRIES ; i++) { -unsigned long nr_pages =0 ; +/* We count existing guard region pages each retry also. */ + nr_pages =0 ;/* Returns < 0 on error, == 0 if success, > 0 if zap needed. */ err =walk_page_range_mm (vma->vm_mm, start, end, &guard_install_walk_ops, &nr_pages);if (err <0 ) -return err; +break ;if (err ==0 ) {unsigned long nr_expected_pages =PHYS_PFN (end - start);VM_WARN_ON (nr_pages != nr_expected_pages); -return 0 ; +break ; }/*
@@ -1172,12 +1163,19 @@ static long madvise_guard_install(struct vm_area_struct *vma,zap_page_range_single (vma, start, end - start,NULL ); } +/* Ensure that page tables are propagated on fork. */ +if (nr_pages >0 ) +vma_set_anon_vma_unfaulted (vma); +/* * We were unable to install the guard pages due to being raced by page * faults. This should not happen ordinarily. We return to userspace and * immediately retry, relieving lock contention. */ -return restart_syscall (); +if (err >0 ) +return restart_syscall (); + +return err; } static int guard_remove_pud_entry(pud_t *pud, unsigned long addr,
@@ -1229,6 +1227,8 @@ static long madvise_guard_remove(struct vm_area_struct *vma,struct vm_area_struct **prev, unsigned long start, unsigned long end) { + long err; + *prev = vma;/* * We're ok with removing guards in mlock()'d ranges, as this is a
@@ -1237,8 +1237,21 @@ static long madvise_guard_remove(struct vm_area_struct *vma,if (!is_valid_guard_vma (vma,/* allow_locked = */ true))return -EINVAL ; -return walk_page_range (vma->vm_mm, start, end, - &guard_remove_walk_ops,NULL ); + err =walk_page_range (vma->vm_mm, start, end, + &guard_remove_walk_ops,NULL ); + +/* +* If we have successfully cleared the guard flags, and we span the +* whole VMA, clear the unfaulted state so this VMA doesn't +* unnecessarily propagate page tables. +* +* The operation is protected via mm->page_table_lock avoiding races +* with a guard install operation. +*/ +if (!err && start == vma->vm_start && end == vma->vm_end) +vma_clear_anon_vma_unfaulted (vma); + +return err; }/*
Signature · 1 line
2.48.1
Quoted 9 lines · Lorenzo Stoakes
So adding additional fields is generally unviable, and VMA flags are equally as contended, and prevent VMA merge, further impacting overhead.
We can however make use of the time-honoured kernel tradition of grabbing bits where we can.
Since we can rely upon
anon_vmaallocations being at least system word-aligned, we have a handful of bits in thevma->anon_vmaavailable to use as flags.
I'm not a huge fan when there's a much better solution. It's an unsigned long, but we can only use the first 32 bits because of 32-bit compatibility? This is a noose we've made for our own neck.
(there are many more places to fix up; this is illustrative):
include/linux/hugetlb_inline.h +1 -1
@@ -8,7 +8,7 @@static inline bool is_vm_hugetlb_page (struct vm_area_struct *vma) { -return !!(vma->vm_flags &VM_HUGETLB ); +return test_bit (VM_HUGETLB , vma->vm_flags); }#else
include/linux/mm_types.h +2 -1
@@ -571,7 +571,8 @@ static inline void *folio_get_private(struct folio *folio)return folio->private; } -typedef unsigned long vm_flags_t ; +#define VM_FLAGS_COUNT (8 / sizeof(unsigned long)) +typedef unsigned long vm_flags_t [VM_FLAGS_COUNT ];/* * A region containing a mapping of a non-memory backed file under NOMMU
Quoted 7 lines · Matthew Wilcox
Quoted 6 lines · Lorenzo Stoakes
So adding additional fields is generally unviable, and VMA flags are equally as contended, and prevent VMA merge, further impacting overhead.
We can however make use of the time-honoured kernel tradition of grabbing bits where we can.
Quoted 7 lines
Quoted 3 lines
Since we can rely upon
anon_vmaallocations being at least system word-aligned, we have a handful of bits in thevma->anon_vmaavailable to use as flags.I'm not a huge fan when there's a much better solution. It's an unsigned long, but we can only use the first 32 bits because of 32-bit compatibility? This is a noose we've made for our own neck.
Sure, as discussed off-list this is something I'm going to look at, it's not either/or at all :)
I have in the back of my mind _other uses_ for these flags which isn't stated here (perhaps should be), perhaps we can have other kinds of data type we reference here.
But perhaps it's a bit early for that...
Other than that was a morning's work to see if this _could_ work as a _immediate_ 'what can we do to address this issue?' solution.
The vm_flags solution is viable, as we can modify merging behaviour for a 'sticky' VMA flag that doesn't impact merging, i.e. mask out for merge compatibility testing, but ensure propagated on split/merge (same behaviour as what the anon_vma flags achieve).
Quoted 21 lines
(there are many more places to fix up; this is illustrative):
include/linux/hugetlb_inline.h +1
-1@@ -8,7 +8,7 @@static inline bool is_vm_hugetlb_page (struct vm_area_struct *vma) { -return !!(vma->vm_flags &VM_HUGETLB ); +return test_bit (VM_HUGETLB , vma->vm_flags); }#else include/linux/mm_types.h +0
-0
include/linux/mm_types.h static inline void *folio_get_private(struct folio *folio) · line 578
Quoted 9 lines
include/linux/mm_types.h +2
-1@@ -571,7 +571,8 @@ static inline void *folio_get_private(struct folio *folio)return folio->private; } -typedef unsigned long vm_flags_t ; +#define VM_FLAGS_COUNT (8 / sizeof(unsigned long)) +typedef unsigned long vm_flags_t [VM_FLAGS_COUNT ];/* * A region containing a mapping of a non-memory backed file under NOMMU
Ohhhh does test_bit() automagically figure things out for sizeof(addr) including on 32-bit?
Maybe I can quickly slap together something for that quicker than I thought then? I was _very concerned_ about tearing and that being a total PITA hence deferring a bit.
But... if you're saying I can churn this to death and it'll 'just work' then say no more... ;)
Let's put this on hold then until/when I have actual reasons to have different anon_vma types and I'll look at the 'make vm_flags 64-bit everywhere' thing instead.