mm/mremap: introduce more mergeable mremap via MREMAP_RELOCATE_ANON
Subsystems (5)
cover mm/mremap: introduce more mergeable mremap via MREMAP_RELOCATE_ANON
REVIEWERS NOTES:
This relies upon the series sent in
https://lore.kernel.org/all/[email protected]/
and must be rebased against this.
This RFC has been thoroughly tested but could do with a little further stress testing, especially under heavy reclaim load, so consider this a fairly early version, uploaded in advance of my LSF/MM/BPF topic in relation to this work :)
~~
A long standing issue with VMA merging of anonymous VMAs is the requirement to maintain both vma->vm_pgoff and anon_vma compatibility between merge candidates.
For anonymous mappings, vma->vm_pgoff (and consequently, folio->index) refer to virtual page offsets, that is, va >> PAGE_SHIFT.
However upon mremap() of an anonymous mapping that has been faulted (that is, where vma->anon_vma != NULL), we would then need to walk page tables to be able to access let alone manipulate folio->index, mapping fields to permit an update of this virtual page offset.
Therefore in these instances, we do not do so, instead retaining the virtual page offset the VMA was first faulted in at as it's vma->vm_pgoff field, and of course consequently folio->index.
On each occasion we use linear_page_index() to determine the appropriate offset, cleverly offset the vma->vm_pgoff field by the difference between the virtual address and actual VMA start.
Doing so in effect fragments the virtual address space, meaning that we are no longer able to merge these VMAs with adjacent ones that could, at least theoretically, be merged.
This also creates a difference in behaviour, often surprising to users, between mappings which are faulted and those which are not - as for the latter we adjust vma->vm_pgoff upon mremap() to aid mergeability.
This is problematic firstly because this proliferates kernel allocations that are pure memory pressure - unreclaimable and unmovable - i.e. vm_area_struct, anon_vma, anon_vma_chain objects that need not exist.
Secondly, mremap() exhibits an implicit uAPI in that it does not permit remaps which span multiple VMAs (though it does permit remaps that constitute a part of a single VMA).
This means that a user must concern themselves with whether merges succeed or not should they wish to use mremap() in such a way which causes multiple mremap() calls to be performed upon mappings.
This series provides users with an option to accept the overhead of actually updating the VMA and underlying folios via the MREMAP_RELOCATE_ANON flag.
If MREMAP_RELOCATE_ANON is specified, but an ordinary merge would result in the mremap() succeeding, then no attempt is made at relocation of folios as this is not required.
Even if no merge is possible upon moving of the region, vma->vm_pgoff and folio->index fields are appropriately updated in order that subsequent mremap() or mprotect() calls will succeed in merging.
This flag falls back to the ordinary means of mremap() should the operation not be feasible. It also transparently undoes the operation, carefully holding rmap locks such that no racing rmap operation encounters incorrect or missing VMAs.
In addition, the MREMAP_MUST_RELOCATE_ANON flag is supplied in case the user needs to know whether or not the operation succeeded - this flag is identical to MREMAP_RELOCATE_ANON, only if the operation cannot succeed, the mremap() fails with -EFAULT.
Note that no-op mremap() operations (such as an unpopulated range, or a merge that would trivially succeed already) will succeed under MREMAP_MUST_RELOCATE_ANON.
mremap() already walks page tables, so it isn't an order of magntitude increase in workload, but constitutes the need to walk to page table leaf level and manipulate folios.
The operations all succeed under THP and in general are compatible with underlying large folios of any size. In fact, the larger the folio, the more efficient the operation is.
Performance testing indicate that time taken using MREMAP_RELOCATE_ANON is on the same order of magnitude of ordinary mremap() operations, with both exhibiting time to the proportion of the mapping which is populated.
Of course, mremap() operations that are entirely aligned are significantly faster as they need only move a VMA and a smaller number of higher order page tables, but this is unavoidable.
Lorenzo Stoakes (7):
mm/mremap: introduce more mergeable mremap via MREMAP_RELOCATE_ANON mm/mremap: add MREMAP_MUST_RELOCATE_ANON
mm/mremap: add MREMAP[_MUST]_RELOCATE_ANON support for THP folios tools UAPI: Update copy of linux/mman.h from the kernel sources tools/testing/selftests: add mremap() cases that merge normally tools/testing/selftests: add MREMAP_RELOCATE_ANON merge test cases tools/testing/selftests: expand mremap() tests for
MREMAP_RELOCATE_ANON
include/uapi/linux/mman.h | 8 +- mm/internal.h | 1 + mm/mremap.c | 610 +++++++++- mm/vma.c | 29 +- mm/vma.h | 5 +- tools/include/uapi/linux/mman.h | 8 +- tools/testing/selftests/mm/merge.c | 1338 +++++++++++++++++++++- tools/testing/selftests/mm/mremap_test.c | 270 +++-- tools/testing/vma/vma.c | 5 +- 9 files changed, 2128 insertions(+), 146 deletions(-)
--
2.48.1
RFC PATCH 1/7 mm/mremap: introduce more mergeable mremap via MREMAP_RELOCATE_ANON
When mremap() moves a mapping around in memory, it goes to great lengths to avoid having to walk page tables as this is expensive and time-consuming.
Rather, if the VMA was faulted (that is vma->anon_vma != NULL), the virtual page offset stored in the VMA at vma->vm_pgoff will remain the same, as well all the folio indexes pointed at the associated anon_vma object.
This means the VMA and page tables can simply be moved and this affects the change (and if we can move page tables at a higher page table level, this is even faster).
While this is efficient, it does lead to big problems with VMA merging - in essence it causes faulted anonymous VMAs to not be mergeable under many circumstances once moved.
This is limiting and leads to both a proliferation of unreclaimable, unmovable kernel metadata (VMAs, anon_vma's, anon_vma_chain's) and has an impact on further use of mremap(), which has a requirement that the VMA moved (which can also be a partial range within a VMA) may span only a single VMA.
This makes the mergeability or not of VMAs in effect a uAPI concern.
In some use cases, users may wish to accept the overhead of actually going to the trouble of updating VMAs and folios to affect mremap() moves. Let's provide them with the choice.
This patch add a new MREMAP_RELOCATE_ANON flag to do just that, which attempts to perform such an operation. If it is unable to do so, it cleanly falls back to the usual method.
It carefully takes the rmap locks such that at no time will a racing rmap user encounter incorrect or missing VMAs.
It is also designed to interact cleanly with the existing mremap() error fallback mechanism (inverting the remap should the page table move fail).
Also, if we could merge cleanly without such a change, we do so, avoiding the overhead of the operation if it is not required.
In the instance that no merge may occur when the move is performed, we still perform the folio and VMA updates to ensure that future mremap() or mprotect() calls will result in merges.
In this implementation, we simply give up if we encounter large folios. A subsequent commit will extend the functionality to allow for these cases.
We restrict this flag to purely anonymous memory only.
--- include/uapi/linux/mman.h | 1 + mm/internal.h | 1 + mm/mremap.c | 354 +++++++++++++++++++++++++++++++++++--- mm/vma.c | 29 +++- mm/vma.h | 5 +- tools/testing/vma/vma.c | 5 +- 6 files changed, 367 insertions(+), 28 deletions(-)
include/uapi/linux/mman.h +1 -0
@@ -9,6 +9,7 @@#define MREMAP_MAYMOVE 1#define MREMAP_FIXED 2#define MREMAP_DONTUNMAP 4 +#define MREMAP_RELOCATE_ANON 8#define OVERCOMMIT_GUESS 0#define OVERCOMMIT_ALWAYS 1
mm/internal.h +1 -0
@@ -46,6 +46,7 @@ struct folio_batch;struct pagetable_move_control {struct vm_area_struct *old;/* Source VMA. */ struct vm_area_struct *new;/* Destination VMA. */ +struct vm_area_struct *relocate_locked;/* VMA which is rmap locked. */ unsigned long old_addr;/* Address from which the move begins. */ unsigned long old_end;/* Exclusive address at which old range ends. */ unsigned long new_addr;/* Address to move page tables to. */
mm/mremap.c +333 -21
459 lines not on this page · raw message
mm/vma.c +25 -4
@@ -774,8 +774,7 @@ static bool can_merge_remove_vma(struct vm_area_struct *vma)* - The caller must hold a WRITE lock on the mm_struct->mmap_lock. * - vmi must be positioned within [@vmg->middle->vm_start, @vmg->middle->vm_end). */ -static __must_check struct vm_area_struct *vma_merge_existing_range ( -struct vma_merge_struct *vmg) +struct vm_area_struct *vma_merge_existing_range (struct vma_merge_struct *vmg) {struct vm_area_struct *middle = vmg->middle;struct vm_area_struct *prev = vmg->prev;
@@ -1756,7 +1755,7 @@ int vma_link(struct mm_struct *mm, struct vm_area_struct *vma)*/ struct vm_area_struct *copy_vma (struct vm_area_struct **vmap,unsigned long addr,unsigned long len,pgoff_t pgoff, -bool *need_rmap_locks) +bool *need_rmap_locks,bool *relocate_anon) {struct vm_area_struct *vma = *vmap;unsigned long vma_start = vma->vm_start;
@@ -1782,7 +1781,19 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, vmg.middle =NULL ;/* New VMA range. */ vmg.pgoff = pgoff; vmg.next =vma_iter_next_rewind (&vmi,NULL ); + new_vma =vma_merge_new_range (&vmg); +if (*relocate_anon) { +/* +* If merge succeeds, no need to relocate. Otherwise, reset +* pgoff for newly established VMA which we will relocate folios +* to. +*/ +if (new_vma) + *relocate_anon = false; +else + pgoff = addr >>PAGE_SHIFT ; + }if (new_vma) {/*
@@ -1813,7 +1824,9 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,vma_set_range (new_vma, addr, addr + len, pgoff);if (vma_dup_policy (vma, new_vma)) goto out_free_vma; -if (anon_vma_clone (new_vma, vma)) +if (*relocate_anon) + new_vma->anon_vma =NULL ; +else if (anon_vma_clone (new_vma, vma)) goto out_free_mempol;if (new_vma->vm_file)get_file (new_vma->vm_file);
@@ -1821,6 +1834,14 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, new_vma->vm_ops->open (new_vma);if (vma_link (mm, new_vma)) goto out_vma_link; +/* +* If we're attempting to relocate anonymous VMAs, we +* don't want to reuse an anon_vma as set by +* vm_area_dup(), or copy anon_vma_chain or anything +* like this. +*/ +if (*relocate_anon &&__anon_vma_prepare (new_vma)) + goto out_vma_link; *need_rmap_locks = false; }return new_vma;
mm/vma.h +4 -1
@@ -260,6 +260,9 @@ __must_check struct vm_area_struct__must_check struct vm_area_struct *vma_merge_new_range (struct vma_merge_struct *vmg); +__must_check struct vm_area_struct +*vma_merge_existing_range (struct vma_merge_struct *vmg); +__must_check struct vm_area_struct *vma_merge_extend(struct vma_iterator *vmi, struct vm_area_struct *vma,
@@ -280,7 +283,7 @@ int vma_link(struct mm_struct *mm, struct vm_area_struct *vma);struct vm_area_struct *copy_vma (struct vm_area_struct **vmap,unsigned long addr,unsigned long len,pgoff_t pgoff, -bool *need_rmap_locks); +bool *need_rmap_locks,bool *relocate_anon);struct anon_vma *find_mergeable_anon_vma (struct vm_area_struct *vma);
tools/testing/vma/vma.c +3 -2
@@ -1543,13 +1543,14 @@ static bool test_copy_vma(void)unsigned long flags =VM_READ |VM_WRITE |VM_MAYREAD |VM_MAYWRITE ;struct mm_struct mm = {};bool need_locks = false; +bool relocate_anon = false;VMA_ITERATOR (vmi, &mm,0 );struct vm_area_struct *vma, *vma_new, *vma_next;/* Move backwards and do not merge. */ vma =alloc_and_link_vma (&mm,0x3000 ,0x5000 ,3 , flags); - vma_new =copy_vma (&vma,0 ,0x2000 ,0 , &need_locks); + vma_new =copy_vma (&vma,0 ,0x2000 ,0 , &need_locks, &relocate_anon);ASSERT_NE (vma_new, vma);ASSERT_EQ (vma_new->vm_start,0 );ASSERT_EQ (vma_new->vm_end,0x2000 );
@@ -1562,7 +1563,7 @@ static bool test_copy_vma(void) vma =alloc_and_link_vma (&mm,0 ,0x2000 ,0 , flags); vma_next =alloc_and_link_vma (&mm,0x6000 ,0x8000 ,6 , flags); - vma_new =copy_vma (&vma,0x4000 ,0x2000 ,4 , &need_locks); + vma_new =copy_vma (&vma,0x4000 ,0x2000 ,4 , &need_locks, &relocate_anon);vma_assert_attached (vma_new);ASSERT_EQ (vma_new, vma_next);
Signature · 1 line
2.48.1
Jann Horn
reviewerQuoted 4 lines · Lorenzo Stoakes
mm/mremap.c +0
-0
[...]
Quoted 28 lines
mm/mremap.c +28
-0+/* +* If the folio mapped at the specified pte entry can have its index and mapping +* relocated, then do so. +* +* Returns the number of pages we have traversed, or 0 if the operation failed. +*/ +static unsigned long relocate_anon (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr,pte_t pte, +bool undo) +{ +struct page *page; +struct folio *folio; +struct vm_area_struct *old, *new; +pgoff_t new_index; +unsigned long ret =1 ; + + old = pmc->old; + new = pmc->new; + +/* Ensure we have truly got an anon folio. */ + page =vm_normal_page (old, old_addr, pte); +if (!page) +return ret; + folio =page_folio (page); +folio_lock (folio); + +/* no-op. */ +if (!folio_test_anon (folio) ||folio_test_ksm (folio))
mm/mremap.c static unsigned long pmc_progress(struct pagetable_move_control *pmc) · line 828
Quoted 10 lines
mm/mremap.c +10
-0+ goto out; + +/* +* This should not happen as we explicitly disallow this, but check +* anyway. +*/ +if (folio_test_large (folio)) { + ret =0 ; + goto out; + }
Do I understand correctly that you assume here that the page is exclusively mapped? Maybe we could at least WARN_ON(folio_mapcount(folio) != 1) or something like that?
(I was also wondering if the PageAnonExclusive bit is somehow relevant, but we should probably not look at or touch that here, unless we want to think about cases where we _used to_ have a child from which the page may have been GUP'd...)
mm/mremap.c static unsigned long pmc_progress(struct pagetable_move_control *pmc) · line 842
Quoted 13 lines
mm/mremap.c +13
-0+if (!undo) + new_index =linear_page_index (new, new_addr); +else + new_index =linear_page_index (old, old_addr); + +/* +* The PTL should keep us safe from unmapping, and the fact the folio is +* a PTE keeps the folio referenced. +* +* The mmap/VMA locks should keep us safe from fork and other processes. +* +* The rmap locks should keep us safe from anything happening to the +* VMA/anon_vma.
"The rmap locks"? I only see that we're holding the rmap lock on the new anon_vma; are we also holding a lock on the old anon_vma?
Quoted 9 lines
mm/mremap.c +9
-0+* The folio lock should keep us safe from reclaim, migration, etc. +*/ +folio_move_anon_rmap (folio, undo ? old : new); +WRITE_ONCE (folio->index, new_index); + +out: +folio_unlock (folio); +return ret; +}
[...]
Quoted 34 lines
mm/mremap.c +34
-0+static bool relocate_anon_ptes (struct pagetable_move_control *pmc, +unsigned long extent,pmd_t *pmd,bool undo) +{ +struct mm_struct *mm = current->mm; +struct pte_state state = { + .old_addr = pmc->old_addr, + .new_addr = pmc->new_addr, + .old_end = pmc->old_addr + extent, + }; +spinlock_t *ptl; +pte_t *ptep_start; +bool ret; +unsigned long nr_pages; + + ptep_start =pte_offset_map_lock (mm, pmd, pmc->old_addr, &ptl); +/* +* We prevent faults with mmap write lock, hold the rmap lock and should +* not fail to obtain this lock. Just give up if we can't. +*/ +if (!ptep_start) +return false; + + state.ptep = ptep_start; + +for (; !pte_done (&state);pte_next (&state, nr_pages)) { +pte_t pte =ptep_get (state.ptep); + +if (pte_none (pte) || !pte_present (pte)) { + nr_pages =1 ; +continue ; + } + + nr_pages =relocate_anon (pmc, state.old_addr, state.new_addr, pte, undo); +if (!nr_pages) {
mm/mremap.c static unsigned long pmc_progress(struct pagetable_move_control *pmc) · line 909
Quoted 10 lines
mm/mremap.c +10
-0+ ret = false; + goto out; + } + } + + ret = true; +out: +pte_unmap_unlock (ptep_start, ptl); +return ret; +}
Just to make sure I understand correctly:
This function is changing the ->pgoff and ->mapping of pages while they are still mapped in the old VMA, right? And if that fails midway through for whatever reason, we go and change all the already-changed ->pgoff and ->mapping values back?
[...]
Quoted 25 lines
mm/mremap.c +21
-0@@ -1132,6 +1347,42 @@ static void unmap_source_vma(struct vma_remap_struct *vrm) } } +/* +* Should we attempt to relocate anonymous folios to the location that the VMA +* is being moved to by updating index and mapping fields accordingly? +*/ +static bool should_relocate_anon (struct vma_remap_struct *vrm, +struct pagetable_move_control *pmc) +{ +struct vm_area_struct *old = vrm->vma; + +/* Currently we only do this if requested. */ +if (!(vrm->flags &MREMAP_RELOCATE_ANON )) +return false; + +/* We can't deal with special or hugetlb mappings. */ +if (old->vm_flags & (VM_SPECIAL |VM_HUGETLB )) +return false; + +/* We only support anonymous mappings. */ +if (!vma_is_anonymous (old)) +return false; +
mm/mremap.c static void unmap_source_vma(struct vma_remap_struct *vrm) · line 1380
Quoted 10 lines
mm/mremap.c +10
-0+/* If no folios are mapped, then no need to attempt this. */ +if (!old->anon_vma) +return false; + +/* +* If the old VMA is a child (i.e. has been forked), then the index +* references multiple VMAs, we have to bail. +*/ +if (!list_is_singular (&old->anon_vma_chain)) +return false;
I think this is wrong: list_is_singular(&old->anon_vma_chain) tells you whether pages in the VMA might be shared due to this mm_struct being forked from a parent mm_struct; but it won't tell you whether pages in the VMA might be shared due to this mm_struct being the parent of another mm_struct (other way around). I guess checking old->anon_vma->root->num_children could maybe work...
Quoted 32 lines
mm/mremap.c +15
-1+ +/* Otherwise, we're good to go! */ +return true; +} +/* * Copy vrm->vma over to vrm->new_addr possibly adjusting size as part of the * process. Additionally handle an error occurring on moving of page tables, @@ -1151,9 +1402,11 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm,struct vm_area_struct *new_vma;int err =0 ;PAGETABLE_MOVE (pmc,NULL ,NULL , vrm->addr, vrm->new_addr, vrm->old_len); +bool relocate_anon =should_relocate_anon (vrm, &pmc); +again: new_vma =copy_vma (&vma, vrm->new_addr, vrm->new_len, new_pgoff, - &pmc.need_rmap_locks); + &pmc.need_rmap_locks, &relocate_anon);if (!new_vma) {vrm_uncharge (vrm); *new_vma_ptr =NULL ;@@ -1163,12 +1416,66 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm, pmc.old = vma; pmc.new = new_vma; +if (relocate_anon) { +/* +* We have a new VMA to reassign folios to. We take a lock on +* its anon_vma so reclaim doesn't fail to unmap mappings. +* +* We have acquired a VMA write lock by now (in vma_link()), so +* we do not have to worry about racing faults.
mm/mremap.c static int copy_vma_and_data(struct vma_remap_struct *vrm, · line 1435
Quoted 10 lines
mm/mremap.c +10
-0+*/ +anon_vma_lock_write (new_vma->anon_vma); + pmc.relocate_locked = new_vma; + +if (!relocate_anon_folios (&pmc,/* undo= */ false)) { +unsigned long start = new_vma->vm_start; +unsigned long size = new_vma->vm_end - start; + +/* Undo if fails. */ +relocate_anon_folios (&pmc,/* undo= */ true);
This relocate_anon_folios() has to make sure to only operate on the subset of PTEs that we already edited successfully, right?
Quoted 10 lines
mm/mremap.c +10
-0+vrm_stat_account (vrm, vrm->new_len); + +anon_vma_unlock_write (new_vma->anon_vma); + pmc.relocate_locked =NULL ; + +do_munmap (current->mm, start, size,NULL ); + relocate_anon = false; + goto again; + } + }
[...]
Quoted 4 lines
mm/vma.c +0
-0
[...]
mm/vma.c struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, · line 1844
Quoted 12 lines
mm/vma.c +8
-0@@ -1821,6 +1834,14 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, new_vma->vm_ops->open (new_vma);if (vma_link (mm, new_vma)) goto out_vma_link; +/* +* If we're attempting to relocate anonymous VMAs, we +* don't want to reuse an anon_vma as set by +* vm_area_dup(), or copy anon_vma_chain or anything +* like this. +*/ +if (*relocate_anon &&__anon_vma_prepare (new_vma)) + goto out_vma_link;
Is this bailout really okay? We go to the same label as if vma_link() failed, but we're in a very different state: For example, the VMA has already been inserted into the VMA tree with vma_iter_store() as part of vma_link() (unless that changed in one of the prerequisite patches).
Quoted 3 lines
mm/vma.c +0
-0*need_rmap_locks = false; }return new_vma;
Quoted 50 lines · Jann Horn
Quoted 4 lines · Lorenzo Stoakes
mm/mremap.c +0
-0[...]
Quoted 28 lines
mm/mremap.c +28
-0+/* +* If the folio mapped at the specified pte entry can have its index and mapping +* relocated, then do so. +* +* Returns the number of pages we have traversed, or 0 if the operation failed. +*/ +static unsigned long relocate_anon (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr,pte_t pte, +bool undo) +{ +struct page *page; +struct folio *folio; +struct vm_area_struct *old, *new; +pgoff_t new_index; +unsigned long ret =1 ; + + old = pmc->old; + new = pmc->new; + +/* Ensure we have truly got an anon folio. */ + page =vm_normal_page (old, old_addr, pte); +if (!page) +return ret; + folio =page_folio (page); +folio_lock (folio); + +/* no-op. */ +if (!folio_test_anon (folio) ||folio_test_ksm (folio))mm/mremap.c static unsigned long pmc_progress(struct pagetable_move_control *pmc) · line 828
Quoted 10 lines
mm/mremap.c +10
-0+ goto out; + +/* +* This should not happen as we explicitly disallow this, but check +* anyway. +*/ +if (folio_test_large (folio)) { + ret =0 ; + goto out; + }Do I understand correctly that you assume here that the page is exclusively mapped? Maybe we could at least
WARN_ON(folio_mapcount(folio) != 1)or something like that?
Quoted 4 lines
(I was also wondering if the PageAnonExclusive bit is somehow relevant, but we should probably not look at or touch that here, unless we want to think about cases where we
_usedto_have a child from which the page may have been GUP'd...)
UFFDIO_MOVE implements something similar. Right now we keep it simple:
if (folio_test_large(src_folio) ||
folio_maybe_dma_pinned(src_folio) ||
!PageAnonExclusive(&src_folio->page)) {
err = -EBUSY;
goto out;
}Whereby we
a) Make sure we cover all PTEs (-> small folio, single PTE). Large PTE-mapped folios are split.
b) Make sure there are no GUP pins (maybe not required here?)
c) The folio is exclusive to this process
In general, things I can reason about with confidence are:
a) As alternative to PageAnonExclusive(), we can check folio_mapcount()==1 under the folio lock for small folios / PMD-mapped folios. As you (Jann) say, there might be unexpected references on the folio from other processes.
b) For large (pte-mapped) folios, we could try batching multiple PTEs (folio_pte_batch() etc.). We'd be processing all mappings with folio_lock + folio_mapcount() == #PTEs.
c) In -next, there is now be the option to use folio lock + folio_maybe_mapped_shared() == false. But it doesn't tell you into how many VMAs a large folio is mapped into.
In the following case:
[ folio ]
[ VMA#1 ] [ VMA#2 ]
c) would not tell you if you are fine modifying the folio when moving VMA#2.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 65 lines · David Hildenbrand
Quoted 50 lines · Jann Horn
Quoted 4 lines · Lorenzo Stoakes
mm/mremap.c +0
-0[...]
Quoted 28 lines
mm/mremap.c +28
-0+/* +* If the folio mapped at the specified pte entry can have its index and mapping +* relocated, then do so. +* +* Returns the number of pages we have traversed, or 0 if the operation failed. +*/ +static unsigned long relocate_anon (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr,pte_t pte, +bool undo) +{ +struct page *page; +struct folio *folio; +struct vm_area_struct *old, *new; +pgoff_t new_index; +unsigned long ret =1 ; + + old = pmc->old; + new = pmc->new; + +/* Ensure we have truly got an anon folio. */ + page =vm_normal_page (old, old_addr, pte); +if (!page) +return ret; + folio =page_folio (page); +folio_lock (folio); + +/* no-op. */ +if (!folio_test_anon (folio) ||folio_test_ksm (folio))mm/mremap.c static unsigned long pmc_progress(struct pagetable_move_control *pmc) · line 828
Quoted 10 lines
mm/mremap.c +10
-0+ goto out; + +/* +* This should not happen as we explicitly disallow this, but check +* anyway. +*/ +if (folio_test_large (folio)) { + ret =0 ; + goto out; + }Do I understand correctly that you assume here that the page is exclusively mapped? Maybe we could at least
WARN_ON(folio_mapcount(folio) != 1)or something like that?Quoted 4 lines
(I was also wondering if the PageAnonExclusive bit is somehow relevant, but we should probably not look at or touch that here, unless we want to think about cases where we
_usedto_have a child from which the page may have been GUP'd...)
UFFDIO_MOVEimplements something similar. Right now we keep it simple:if (folio_test_large(src_folio) || folio_maybe_dma_pinned(src_folio) || !PageAnonExclusive(&src_folio->page)) { err = -EBUSY; goto out; }
Quoted 8 lines
Whereby we
a) Make sure we cover all PTEs (-> small folio, single PTE). Large PTE-mapped folios are split.
b) Make sure there are no GUP pins (maybe not required here?)
c) The folio is exclusive to this process
On additional note as my memory comes back: if PAE is set, there cannot be other (inactive) mappings from the swapcache. So whenever we use folio lock + mapcount data, the possibility of the swapcache (having inactive mappings from other processes etc.) must be considered.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 71 lines · David Hildenbrand
Quoted 70 lines · David Hildenbrand
Quoted 50 lines · Jann Horn
Quoted 4 lines · Lorenzo Stoakes
mm/mremap.c +0
-0[...]
Quoted 28 lines
mm/mremap.c +28
-0+/* +* If the folio mapped at the specified pte entry can have its index and mapping +* relocated, then do so. +* +* Returns the number of pages we have traversed, or 0 if the operation failed. +*/ +static unsigned long relocate_anon (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr,pte_t pte, +bool undo) +{ +struct page *page; +struct folio *folio; +struct vm_area_struct *old, *new; +pgoff_t new_index; +unsigned long ret =1 ; + + old = pmc->old; + new = pmc->new; + +/* Ensure we have truly got an anon folio. */ + page =vm_normal_page (old, old_addr, pte); +if (!page) +return ret; + folio =page_folio (page); +folio_lock (folio); + +/* no-op. */ +if (!folio_test_anon (folio) ||folio_test_ksm (folio))mm/mremap.c static unsigned long pmc_progress(struct pagetable_move_control *pmc) · line 828
Quoted 10 lines
mm/mremap.c +10
-0+ goto out; + +/* +* This should not happen as we explicitly disallow this, but check +* anyway. +*/ +if (folio_test_large (folio)) { + ret =0 ; + goto out; + }Do I understand correctly that you assume here that the page is exclusively mapped? Maybe we could at least
WARN_ON(folio_mapcount(folio) != 1)or something like that?Quoted 4 lines
(I was also wondering if the PageAnonExclusive bit is somehow relevant, but we should probably not look at or touch that here, unless we want to think about cases where we
_usedto_have a child from which the page may have been GUP'd...)
UFFDIO_MOVEimplements something similar. Right now we keep it simple:if (folio_test_large(src_folio) || folio_maybe_dma_pinned(src_folio) || !PageAnonExclusive(&src_folio->page)) { err = -EBUSY; goto out; }Whereby we
a) Make sure we cover all PTEs (-> small folio, single PTE). Large PTE-mapped folios are split.
Quoted 8 lines
Quoted 3 lines
b) Make sure there are no GUP pins (maybe not required here?)
c) The folio is exclusive to this process
On additional note as my memory comes back: if PAE is set, there cannot be other (inactive) mappings from the swapcache. So whenever we use folio lock
+ mapcount data, the possibility of the swapcache (having inactive mappings from other processes etc.) must be considered.
Ack, do you have a feel for how such a check would work?
Quoted 6 lines
--
Cheers,David / dhildenb
An aside in general:
I realise all of this is very fiddly, this (albeit early) RFC is several revisions deep and fully expect considerably more fiddly cases to come up :)
I plan to do some deeper testing on real iron running very heavy workloads to encourage reclaim and races btw.
One test I've done in past is to just hack in forced-on MREMAP_RELOCATE_ANON so _all_ mremap()'s that move do it, which has been a good way of finding issues, but also in tests I add in series I intentionally trigger reclaim via MADV_PAGEOUT.
So in general - I'm going to proactively try to really eek out all and any weird edge case stuff where possible before more seriously pushing this series forward.
Thanks again for early review, it's much appreciated! :) and see you at the topic I'm giving on this and the pub after... ;)
Quoted 74 lines · Lorenzo Stoakes
Quoted 73 lines · David Hildenbrand
Quoted 72 lines · David Hildenbrand
Quoted 50 lines · Jann Horn
Quoted 4 lines · Lorenzo Stoakes
mm/mremap.c +0
-0[...]
Quoted 28 lines
mm/mremap.c +28
-0+/* +* If the folio mapped at the specified pte entry can have its index and mapping +* relocated, then do so. +* +* Returns the number of pages we have traversed, or 0 if the operation failed. +*/ +static unsigned long relocate_anon (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr,pte_t pte, +bool undo) +{ +struct page *page; +struct folio *folio; +struct vm_area_struct *old, *new; +pgoff_t new_index; +unsigned long ret =1 ; + + old = pmc->old; + new = pmc->new; + +/* Ensure we have truly got an anon folio. */ + page =vm_normal_page (old, old_addr, pte); +if (!page) +return ret; + folio =page_folio (page); +folio_lock (folio); + +/* no-op. */ +if (!folio_test_anon (folio) ||folio_test_ksm (folio))mm/mremap.c static unsigned long pmc_progress(struct pagetable_move_control *pmc) · line 828
Quoted 10 lines
mm/mremap.c +10
-0+ goto out; + +/* +* This should not happen as we explicitly disallow this, but check +* anyway. +*/ +if (folio_test_large (folio)) { + ret =0 ; + goto out; + }Do I understand correctly that you assume here that the page is exclusively mapped? Maybe we could at least
WARN_ON(folio_mapcount(folio) != 1)or something like that?Quoted 4 lines
(I was also wondering if the PageAnonExclusive bit is somehow relevant, but we should probably not look at or touch that here, unless we want to think about cases where we
_usedto_have a child from which the page may have been GUP'd...)
UFFDIO_MOVEimplements something similar. Right now we keep it simple:if (folio_test_large(src_folio) || folio_maybe_dma_pinned(src_folio) || !PageAnonExclusive(&src_folio->page)) { err = -EBUSY; goto out; }Whereby we
a) Make sure we cover all PTEs (-> small folio, single PTE). Large PTE-mapped folios are split.
b) Make sure there are no GUP pins (maybe not required here?)
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
c) The folio is exclusive to this process
On additional note as my memory comes back: if PAE is set, there cannot be other (inactive) mappings from the swapcache. So whenever we use folio lock
+ mapcount data, the possibility of the swapcache (having inactive mappings from other processes etc.) must be considered.Ack, do you have a feel for how such a check would work?
Likely under folio lock
if (folio_test_swapcache(folio) && !folio_free_swap(folio)) {
/* unable to move. */
folio_unlock(folio)
return -ENOTGOINGTOHAPPEN;
}Signature · 3 lines
Cheers, David / dhildenb
Quoted 84 lines · David Hildenbrand
Quoted 74 lines · Lorenzo Stoakes
Quoted 73 lines · David Hildenbrand
Quoted 72 lines · David Hildenbrand
Quoted 50 lines · Jann Horn
Quoted 4 lines · Lorenzo Stoakes
mm/mremap.c +0
-0[...]
Quoted 28 lines
mm/mremap.c +28
-0+/* +* If the folio mapped at the specified pte entry can have its index and mapping +* relocated, then do so. +* +* Returns the number of pages we have traversed, or 0 if the operation failed. +*/ +static unsigned long relocate_anon (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr,pte_t pte, +bool undo) +{ +struct page *page; +struct folio *folio; +struct vm_area_struct *old, *new; +pgoff_t new_index; +unsigned long ret =1 ; + + old = pmc->old; + new = pmc->new; + +/* Ensure we have truly got an anon folio. */ + page =vm_normal_page (old, old_addr, pte); +if (!page) +return ret; + folio =page_folio (page); +folio_lock (folio); + +/* no-op. */ +if (!folio_test_anon (folio) ||folio_test_ksm (folio))mm/mremap.c static unsigned long pmc_progress(struct pagetable_move_control *pmc) · line 828
Quoted 10 lines
mm/mremap.c +10
-0+ goto out; + +/* +* This should not happen as we explicitly disallow this, but check +* anyway. +*/ +if (folio_test_large (folio)) { + ret =0 ; + goto out; + }Do I understand correctly that you assume here that the page is exclusively mapped? Maybe we could at least
WARN_ON(folio_mapcount(folio) != 1)or something like that?Quoted 4 lines
(I was also wondering if the PageAnonExclusive bit is somehow relevant, but we should probably not look at or touch that here, unless we want to think about cases where we
_usedto_have a child from which the page may have been GUP'd...)
UFFDIO_MOVEimplements something similar. Right now we keep it simple:if (folio_test_large(src_folio) || folio_maybe_dma_pinned(src_folio) || !PageAnonExclusive(&src_folio->page)) { err = -EBUSY; goto out; }Whereby we
a) Make sure we cover all PTEs (-> small folio, single PTE). Large PTE-mapped folios are split.
b) Make sure there are no GUP pins (maybe not required here?)
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
c) The folio is exclusive to this process
On additional note as my memory comes back: if PAE is set, there cannot be other (inactive) mappings from the swapcache. So whenever we use folio lock
+ mapcount data, the possibility of the swapcache (having inactive mappings from other processes etc.) must be considered.Ack, do you have a feel for how such a check would work?
Quoted 7 lines
Likely under folio lock
if (folio_test_swapcache(folio) && !folio_free_swap(folio)) { /* unable to move. */ folio_unlock(folio) return -ENOTGOINGTOHAPPEN; }
Thanks! Will adapt.
Quoted 6 lines
--
Cheers,David / dhildenb
Thanks for early review, pints_owed++ :>)
Quoted 65 lines · David Hildenbrand
Quoted 50 lines · Jann Horn
Quoted 4 lines · Lorenzo Stoakes
mm/mremap.c +0
-0[...]
Quoted 28 lines
mm/mremap.c +28
-0+/* +* If the folio mapped at the specified pte entry can have its index and mapping +* relocated, then do so. +* +* Returns the number of pages we have traversed, or 0 if the operation failed. +*/ +static unsigned long relocate_anon (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr,pte_t pte, +bool undo) +{ +struct page *page; +struct folio *folio; +struct vm_area_struct *old, *new; +pgoff_t new_index; +unsigned long ret =1 ; + + old = pmc->old; + new = pmc->new; + +/* Ensure we have truly got an anon folio. */ + page =vm_normal_page (old, old_addr, pte); +if (!page) +return ret; + folio =page_folio (page); +folio_lock (folio); + +/* no-op. */ +if (!folio_test_anon (folio) ||folio_test_ksm (folio))mm/mremap.c static unsigned long pmc_progress(struct pagetable_move_control *pmc) · line 828
Quoted 10 lines
mm/mremap.c +10
-0+ goto out; + +/* +* This should not happen as we explicitly disallow this, but check +* anyway. +*/ +if (folio_test_large (folio)) { + ret =0 ; + goto out; + }Do I understand correctly that you assume here that the page is exclusively mapped? Maybe we could at least
WARN_ON(folio_mapcount(folio) != 1)or something like that?Quoted 4 lines
(I was also wondering if the PageAnonExclusive bit is somehow relevant, but we should probably not look at or touch that here, unless we want to think about cases where we
_usedto_have a child from which the page may have been GUP'd...)
UFFDIO_MOVEimplements something similar. Right now we keep it simple:if (folio_test_large(src_folio) || folio_maybe_dma_pinned(src_folio) || !PageAnonExclusive(&src_folio->page)) { err = -EBUSY; goto out; }
Quoted 8 lines
Whereby we
a) Make sure we cover all PTEs (-> small folio, single PTE). Large PTE-mapped folios are split.
b) Make sure there are no GUP pins (maybe not required here?)
c) The folio is exclusive to this process
Yeah, later I actually add handling for large folios :) but this is something separate.
The maybe dma pinned thing is a thing and probably I need to add this. Will do so.
Quoted 7 lines
In general, things I can reason about with confidence are:
a) As alternative to
PageAnonExclusive(), we can check folio_mapcount()==1 under the folio lock for small folios / PMD-mapped folios. As you (Jann) say, there might be unexpected references on the folio from other processes.
Ack for sure will add.
Quoted 4 lines
b) For large (pte-mapped) folios, we could try batching multiple PTEs (
folio_pte_batch()etc.). We'd be processing all mappings withfolio_lock+folio_mapcount()== #PTEs.
Interesting, hadn't thought about this, maybe we can discuss at LSF?
Quoted 11 lines
c) In -next, there is now be the option to use folio lock +
folio_maybe_mapped_shared()== false. But it doesn't tell you into how many VMAs a large folio is mapped into.In the following case:
[ folio ]
[ VMA#1 ] [ VMA#2 ]c) would not tell you if you are fine modifying the folio when moving VMA#2.
Right, I feel like prior checks made should assert this is not the case, however? But mapcount check should be a last ditch assurance?
(actually at least one of the 'prior checks' for large folios are added in a later commit but still :P)
Quoted 7 lines
--
Cheers,David / dhildenb
Quoted 7 lines
Quoted 7 lines
c) In -next, there is now be the option to use folio lock +
folio_maybe_mapped_shared()== false. But it doesn't tell you into how many VMAs a large folio is mapped into.In the following case:
Quoted 7 lines
Quoted 4 lines
[ folio ]
[ VMA#1 ] [ VMA#2 ]c) would not tell you if you are fine modifying the folio when moving VMA#2.
Right, I feel like prior checks made should assert this is not the case, however? But mapcount check should be a last ditch assurance?
Something nice might be hiding in c) that might be able to handle a single folio being covered by multiple vmas.
I was thinking about the following:
[ folio0 ] [ VMA#0 ]
Then we do a partial (old-school) mremap()
[ folio0 ] [ folio0 ]
[ VMA#1 ] [ VMA#2 ]
To then extend VMA#1 and fault in pages
[ folio0 ][ folio1 ] [ folio0 ]
[ VMA#1 ] [ VMA#2 ]
If that is possible (did not try!, maybe something prevents us from extending VMA#1) mremap(MREMAP_RELOCATE_ANON) of VMA#1 / VMA#2 cannot work.
We'd have to detect that scenario (partial mremap). You might be doing that with the anon-vma magic, something different might be: Assume we flag large folios if they were partially mremapped in any process.
Then (with folio lock only)
1) folio_maybe_mapped_shared() == false: mapped into single process
2) folio_maybe_partially_mremaped() == false: not scattered in virtual address space
It would be sufficient to check if the folio fully falls into the memap() range to decide if we can adjust the folio index etc.
We *might* be able to use that in the COW-reuse path for large folios to perform a folio_move_anon_rmap(), which we currently only perform for small folios / PMD-mapped folios (single mapping). Not sure yet if actually multiple VMAs are involved.
Just throwing it out there ...
Quoted 3 lines
(actually at least one of the 'prior checks' for large folios are added in a later commit but still :P)
Yeah, I'm looking at the bigger picture; small folios are easy :P
Signature · 3 lines
Cheers, David / dhildenb
Quoted 33 lines · David Hildenbrand
Quoted 7 lines
Quoted 7 lines
c) In -next, there is now be the option to use folio lock +
folio_maybe_mapped_shared()== false. But it doesn't tell you into how many VMAs a large folio is mapped into.In the following case:
Quoted 7 lines
Quoted 4 lines
[ folio ]
[ VMA#1 ] [ VMA#2 ]c) would not tell you if you are fine modifying the folio when moving VMA#2.
Right, I feel like prior checks made should assert this is not the case, however? But mapcount check should be a last ditch assurance?
Something nice might be hiding in c) that might be able to handle a single folio being covered by multiple vmas.
I was thinking about the following:
[ folio0 ] [ VMA#0 ]Then we do a partial (old-school)
mremap()[ folio0 ] [ folio0 ]
[ VMA#1 ] [ VMA#2 ]To then extend VMA#1 and fault in pages
[ folio0 ][ folio1 ] [ folio0 ]
[ VMA#1 ] [ VMA#2 ]
Quoted 6 lines
If that is possible (did not try!, maybe something prevents us from extending VMA#1)
mremap(MREMAP_RELOCATE_ANON)of VMA#1 / VMA#2 cannot work.We'd have to detect that scenario (partial mremap). You might be doing that with the anon-vma magic, something different might be: Assume we flag large folios if they were partially mremapped in any process.
Do we have spare folio flags? :)) I always lose track of the situation with this and Matthew's levels of tolerance for it :P
Quoted 7 lines
Then (with folio lock only)
1)
folio_maybe_mapped_shared()== false: mapped into single process
2)folio_maybe_partially_mremaped()== false: not scattered in virtual address space
Quoted 7 lines
It would be sufficient to check if the folio fully falls into the
memap()range to decide if we can adjust the folio index etc.We *might* be able to use that in the COW-reuse path for large folios to perform a
folio_move_anon_rmap(), which we currently only perform for small folios / PMD-mapped folios (single mapping). Not sure yet if actually multiple VMAs are involved.
Interesting... this is the wp_can_reuse_anon_folio() stuff? I'll have a look into that!
I'm concerned about partial cases moreso though, e.g.:
mremap this
<----------->
[ folio0 ]
[ VMA#0 ]I mean, I'm leaning more towards just breaking up the folio, especialy if we consider a case like a biiig range:
mremap this
<--------------------------------------------------->[ folio0 ][ folio1 ][ folio2 ][ folio3 ][ folio4 ][ folio5 ] (say order-9 each)
[ VMA#0 ]
Then at this point, refusing to do the whole thing seems maybe a bad idea, at which point splitting the folios for folio0, 5 might be sensible.
I guess a user is saying 'please, I really care about merging' so might well be willing to tolerate losing some of the huge page benefits, at least at the edges here.
Quoted 10 lines
Just throwing it out there ...
Quoted 3 lines
(actually at least one of the 'prior checks' for large folios are added in a later commit but still :P)
Yeah, I'm looking at the bigger picture; small folios are easy :P
Yeah, back when life was simpler... :P
Quoted 7 lines
--
Cheers,David / dhildenb
Cheers, Lorenzo
Quoted 37 lines · Lorenzo Stoakes
Quoted 36 lines · David Hildenbrand
Quoted 7 lines
Quoted 7 lines
c) In -next, there is now be the option to use folio lock +
folio_maybe_mapped_shared()== false. But it doesn't tell you into how many VMAs a large folio is mapped into.In the following case:
Quoted 7 lines
Quoted 4 lines
[ folio ]
[ VMA#1 ] [ VMA#2 ]c) would not tell you if you are fine modifying the folio when moving VMA#2.
Right, I feel like prior checks made should assert this is not the case, however? But mapcount check should be a last ditch assurance?
Something nice might be hiding in c) that might be able to handle a single folio being covered by multiple vmas.
I was thinking about the following:
[ folio0 ] [ VMA#0 ]Then we do a partial (old-school)
mremap()[ folio0 ] [ folio0 ]
[ VMA#1 ] [ VMA#2 ]To then extend VMA#1 and fault in pages
[ folio0 ][ folio1 ] [ folio0 ]
[ VMA#1 ] [ VMA#2 ]If that is possible (did not try!, maybe something prevents us from extending VMA#1)
mremap(MREMAP_RELOCATE_ANON)of VMA#1 / VMA#2 cannot work.
Quoted 6 lines
Quoted 3 lines
We'd have to detect that scenario (partial mremap). You might be doing that with the anon-vma magic, something different might be: Assume we flag large folios if they were partially mremapped in any process.
Do we have spare folio flags? :)) I always lose track of the situation with this and Matthew's levels of tolerance for it :P
For large folios yes (currently stored in page[1].flags)!
Quoted 11 lines
Quoted 11 lines
Then (with folio lock only)
1)
folio_maybe_mapped_shared()== false: mapped into single process
2)folio_maybe_partially_mremaped()== false: not scattered in virtual address spaceIt would be sufficient to check if the folio fully falls into the
memap()range to decide if we can adjust the folio index etc.
Quoted 7 lines
Quoted 4 lines
We *might* be able to use that in the COW-reuse path for large folios to perform a
folio_move_anon_rmap(), which we currently only perform for small folios / PMD-mapped folios (single mapping). Not sure yet if actually multiple VMAs are involved.Interesting... this is the
wp_can_reuse_anon_folio()stuff? I'll have a look into that!
Yes. For small folios / PMD-mapped THPs it's easy. I readded it in
Author: David Hildenbrand <[email protected]> Date: Mon May 9 18:20:43 2022 -0700
mm/rmap: use page_move_anon_rmap() when reusing a mapped PageAnon() page exclusively
We'd need something along the for PTE-mapped large folios like (todo):
mm/memory.c +8 -0
@@ -3793,6 +3793,15 @@ static bool __wp_can_reuse_large_anon_folio(struct folio *folio, exclusive = true; unlock:folio_unlock_large_mapcount (folio); + +if (exclusive && (folio->mapping == vma->anon_vma |PAGE_MAPPING_ANON ) && +todo_check_folio_in_vma (folio, vma) && +folio_trylock (folio)) { +folio_move_anon_rmap (folio, vma); +folio_unlock (folio); + } +out:return exclusive; }
Whereby a "partially mremapped" indication could come in handy.
Note that I never learned how important that optimization actually is in practice, so I didn't care too much for now for PTE-mapped large folios.
Quoted 7 lines
I'm concerned about partial cases moreso though, e.g.:
mremap this <-----------> [ folio0 ] [ VMA#0 ]
Right, there is no easy way to not split I think. We should just try to split and if it fails, too bad.
Quoted 9 lines
I mean, I'm leaning more towards just breaking up the folio, especialy if we consider a case like a biiig range:
mremap this <--------------------------------------------------->[ folio0 ][ folio1 ][ folio2 ][ folio3 ][ folio4 ][ folio5 ] (say order-9 each)
[ VMA#0 ]
Quoted 6 lines
Then at this point, refusing to do the whole thing seems maybe a bad idea, at which point splitting the folios for folio0, 5 might be sensible.
I guess a user is saying 'please, I really care about merging' so might well be willing to tolerate losing some of the huge page benefits, at least at the edges here.
Yes. In uffdio_move we simply split all PTE-mapped large folios, because this way the folio_move_anon_rmap() is way easier to handle (and we could always try optimizing it later if really required). Once we have khugeapged with mthp support in place, it might just fix it up for us later.
Signature · 3 lines
Cheers, David / dhildenb
Quoted 37 lines · Lorenzo Stoakes
Quoted 36 lines · David Hildenbrand
Quoted 7 lines
Quoted 7 lines
c) In -next, there is now be the option to use folio lock +
folio_maybe_mapped_shared()== false. But it doesn't tell you into how many VMAs a large folio is mapped into.In the following case:
Quoted 7 lines
Quoted 4 lines
[ folio ]
[ VMA#1 ] [ VMA#2 ]c) would not tell you if you are fine modifying the folio when moving VMA#2.
Right, I feel like prior checks made should assert this is not the case, however? But mapcount check should be a last ditch assurance?
Something nice might be hiding in c) that might be able to handle a single folio being covered by multiple vmas.
I was thinking about the following:
[ folio0 ] [ VMA#0 ]Then we do a partial (old-school)
mremap()[ folio0 ] [ folio0 ]
[ VMA#1 ] [ VMA#2 ]To then extend VMA#1 and fault in pages
[ folio0 ][ folio1 ] [ folio0 ]
[ VMA#1 ] [ VMA#2 ]If that is possible (did not try!, maybe something prevents us from extending VMA#1)
mremap(MREMAP_RELOCATE_ANON)of VMA#1 / VMA#2 cannot work.
Quoted 11 lines
Quoted 3 lines
We'd have to detect that scenario (partial mremap). You might be doing that with the anon-vma magic, something different might be: Assume we flag large folios if they were partially mremapped in any process.
Do we have spare folio flags? :)) I always lose track of the situation with this and Matthew's levels of tolerance for it :P
Quoted 4 lines
Then (with folio lock only)
1)
folio_maybe_mapped_shared()== false: mapped into single process
Am looking at this series again :) This function is very handy thanks, will use in the upcoming RFCv2!
Quoted 2 lines
Quoted 2 lines
2)
folio_maybe_partially_mremaped()== false: not scattered in virtual address space
This is intriguing, I think perhaps best to defer this to a later date :) have added a personal todo accordingly. We are able to fairly reasonably detect this case right now so it's not urgent.
Quoted 56 lines
Quoted 8 lines
It would be sufficient to check if the folio fully falls into the
memap()range to decide if we can adjust the folio index etc.We *might* be able to use that in the COW-reuse path for large folios to perform a
folio_move_anon_rmap(), which we currently only perform for small folios / PMD-mapped folios (single mapping). Not sure yet if actually multiple VMAs are involved.Interesting... this is the
wp_can_reuse_anon_folio()stuff? I'll have a look into that!I'm concerned about partial cases moreso though, e.g.:
mremap this <-----------> [ folio0 ] [ VMA#0 ]I mean, I'm leaning more towards just breaking up the folio, especialy if we consider a case like a biiig range:
mremap this <--------------------------------------------------->[ folio0 ][ folio1 ][ folio2 ][ folio3 ][ folio4 ][ folio5 ] (say order-9 each)
[ VMA#0 ]Then at this point, refusing to do the whole thing seems maybe a bad idea, at which point splitting the folios for folio0, 5 might be sensible.
I guess a user is saying 'please, I really care about merging' so might well be willing to tolerate losing some of the huge page benefits, at least at the edges here.
Quoted 10 lines
Just throwing it out there ...
Quoted 3 lines
(actually at least one of the 'prior checks' for large folios are added in a later commit but still :P)
Yeah, I'm looking at the bigger picture; small folios are easy :P
Yeah, back when life was simpler... :P
Quoted 7 lines
--
Cheers,David / dhildenb
Cheers, Lorenzo
Thanks for the early review :) I gather you're not coming to LSF? Which is a huge pity, would have loved to see you there.
But this is much appreciated! :)
Quoted 41 lines · Jann Horn
Quoted 4 lines · Lorenzo Stoakes
mm/mremap.c +0
-0[...]
Quoted 34 lines
mm/mremap.c +34
-0+/* +* If the folio mapped at the specified pte entry can have its index and mapping +* relocated, then do so. +* +* Returns the number of pages we have traversed, or 0 if the operation failed. +*/ +static unsigned long relocate_anon (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr,pte_t pte, +bool undo) +{ +struct page *page; +struct folio *folio; +struct vm_area_struct *old, *new; +pgoff_t new_index; +unsigned long ret =1 ; + + old = pmc->old; + new = pmc->new; + +/* Ensure we have truly got an anon folio. */ + page =vm_normal_page (old, old_addr, pte); +if (!page) +return ret; + folio =page_folio (page); +folio_lock (folio); + +/* no-op. */ +if (!folio_test_anon (folio) ||folio_test_ksm (folio)) + goto out; + +/* +* This should not happen as we explicitly disallow this, but check +* anyway. +*/
Quoted 8 lines
mm/mremap.c static unsigned long pmc_progress(struct pagetable_move_control *pmc) · line 828
Quoted 4 lines
mm/mremap.c +4
-0+if (folio_test_large (folio)) { + ret =0 ; + goto out; + }Do I understand correctly that you assume here that the page is exclusively mapped? Maybe we could at least
WARN_ON(folio_mapcount(folio) != 1)or something like that?
Ack and agreed, will add!
Quoted 5 lines
(I was also wondering if the PageAnonExclusive bit is somehow relevant, but we should probably not look at or touch that here, unless we want to think about cases where we
_usedto_have a child from which the page may have been GUP'd...)
Yeah I was thinking the same re: this flag, but given locks we hold this should not be the case, however you later do note the point that I need to check anon_vma->num_children == 1 (i.e. self-parented) to ensure that we aren't looking at a mapping in a parent of a child which would imply a folio that maybe isn't.
Quoted 15 lines
Quoted 4 lines
mm/mremap.c +3
-0+if (!undo) + new_index =linear_page_index (new, new_addr); +else mm/mremap.c static unsigned long pmc_progress(struct pagetable_move_control *pmc) · line 842
Quoted 10 lines
mm/mremap.c +10
-0+ new_index =linear_page_index (old, old_addr); + +/* +* The PTL should keep us safe from unmapping, and the fact the folio is +* a PTE keeps the folio referenced. +* +* The mmap/VMA locks should keep us safe from fork and other processes. +* +* The rmap locks should keep us safe from anything happening to the +* VMA/anon_vma.
Quoted 2 lines
"The rmap locks"? I only see that we're holding the rmap lock on the new
anon_vma; are we also holding a lock on the oldanon_vma?
I should rephrase (this is partly because I kept changing how I did the locking - hey I did warn in preamble this is early :P)
I don't believe we need to hold the _source_ rmap lock, because we establish a folio lock prior to adjusting the folio so it shouldn't be possible for an rmap walk to go terribly wrong here.
But do correct me if you feel this is an invalid assumption.
Quoted 53 lines
Quoted 10 lines
mm/mremap.c +9
-0+* The folio lock should keep us safe from reclaim, migration, etc. +*/ +folio_move_anon_rmap (folio, undo ? old : new); +WRITE_ONCE (folio->index, new_index); + +out: +folio_unlock (folio); +return ret; +}[...]
Quoted 42 lines
mm/mremap.c +42
-0+static bool relocate_anon_ptes (struct pagetable_move_control *pmc, +unsigned long extent,pmd_t *pmd,bool undo) +{ +struct mm_struct *mm = current->mm; +struct pte_state state = { + .old_addr = pmc->old_addr, + .new_addr = pmc->new_addr, + .old_end = pmc->old_addr + extent, + }; +spinlock_t *ptl; +pte_t *ptep_start; +bool ret; +unsigned long nr_pages; + + ptep_start =pte_offset_map_lock (mm, pmd, pmc->old_addr, &ptl); +/* +* We prevent faults with mmap write lock, hold the rmap lock and should +* not fail to obtain this lock. Just give up if we can't. +*/ +if (!ptep_start) +return false; + + state.ptep = ptep_start; + +for (; !pte_done (&state);pte_next (&state, nr_pages)) { +pte_t pte =ptep_get (state.ptep); + +if (pte_none (pte) || !pte_present (pte)) { + nr_pages =1 ; +continue ; + } + + nr_pages =relocate_anon (pmc, state.old_addr, state.new_addr, pte, undo); +if (!nr_pages) { + ret = false; + goto out; + } + } + + ret = true; +out: +pte_unmap_unlock (ptep_start, ptl);
Quoted 8 lines
mm/mremap.c
Quoted 2 lines
mm/mremap.c +2
-0+return ret; +}Just to make sure I understand correctly:
This function is changing the ->pgoff and ->mapping of pages while they are still mapped in the old VMA, right? And if that fails midway through for whatever reason, we go and change all the already-changed ->pgoff and ->mapping values back?
Yup. Write lock is held on both VMAs, and rmap lock for new VMA.
Quoted 36 lines
[...]
Quoted 34 lines
mm/mremap.c +30
-0@@ -1132,6 +1347,42 @@ static void unmap_source_vma(struct vma_remap_struct *vrm) } } +/* +* Should we attempt to relocate anonymous folios to the location that the VMA +* is being moved to by updating index and mapping fields accordingly? +*/ +static bool should_relocate_anon (struct vma_remap_struct *vrm, +struct pagetable_move_control *pmc) +{ +struct vm_area_struct *old = vrm->vma; + +/* Currently we only do this if requested. */ +if (!(vrm->flags &MREMAP_RELOCATE_ANON )) +return false; + +/* We can't deal with special or hugetlb mappings. */ +if (old->vm_flags & (VM_SPECIAL |VM_HUGETLB )) +return false; + +/* We only support anonymous mappings. */ +if (!vma_is_anonymous (old)) +return false; + +/* If no folios are mapped, then no need to attempt this. */ +if (!old->anon_vma) +return false; + +/* +* If the old VMA is a child (i.e. has been forked), then the index +* references multiple VMAs, we have to bail. +*/ +if (!list_is_singular (&old->anon_vma_chain))
Quoted 8 lines
mm/mremap.c static void unmap_source_vma(struct vma_remap_struct *vrm) · line 1380
Quoted 1 line
mm/mremap.c +1
-0+return false;I think this is wrong:
list_is_singular(&old->anon_vma_chain)tells you whether pages in the VMA might be shared due to thismm_structbeing forked from a parentmm_struct; but it won't tell you whether pages in the VMA might be shared due to thismm_structbeing the parent of anothermm_struct(other way around). I guess checkingold->anon_vma->root->num_childrencould maybe work...
Yeah you're completely right, this is entirely an oversight on my part.
I don't think we need to be looking at old->anon_vma->root though? As for anon_vma->root != anon_vma here we'd need to be a child, and we just asserted we're not right? (could be an additional check though).
But definitely need an old->anon_vma->num_children check.
Holding old vma lock should prevent further forks, so we could just take the rmap lock - lock - rmap unlock here?
Quoted 38 lines
Quoted 38 lines
mm/mremap.c +20
-1+ +/* Otherwise, we're good to go! */ +return true; +} +/* * Copy vrm->vma over to vrm->new_addr possibly adjusting size as part of the * process. Additionally handle an error occurring on moving of page tables, @@ -1151,9 +1402,11 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm,struct vm_area_struct *new_vma;int err =0 ;PAGETABLE_MOVE (pmc,NULL ,NULL , vrm->addr, vrm->new_addr, vrm->old_len); +bool relocate_anon =should_relocate_anon (vrm, &pmc); +again: new_vma =copy_vma (&vma, vrm->new_addr, vrm->new_len, new_pgoff, - &pmc.need_rmap_locks); + &pmc.need_rmap_locks, &relocate_anon);if (!new_vma) {vrm_uncharge (vrm); *new_vma_ptr =NULL ;@@ -1163,12 +1416,66 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm, pmc.old = vma; pmc.new = new_vma; +if (relocate_anon) { +/* +* We have a new VMA to reassign folios to. We take a lock on +* its anon_vma so reclaim doesn't fail to unmap mappings. +* +* We have acquired a VMA write lock by now (in vma_link()), so +* we do not have to worry about racing faults. +*/ +anon_vma_lock_write (new_vma->anon_vma); + pmc.relocate_locked = new_vma; + +if (!relocate_anon_folios (&pmc,/* undo= */ false)) {
Quoted 8 lines
mm/mremap.c static int copy_vma_and_data(struct vma_remap_struct *vrm, · line 1435
Quoted 5 lines
mm/mremap.c +5
-0+unsigned long start = new_vma->vm_start; +unsigned long size = new_vma->vm_end - start; + +/* Undo if fails. */ +relocate_anon_folios (&pmc,/* undo= */ true);This
relocate_anon_folios()has to make sure to only operate on the subset of PTEs that we already edited successfully, right?
Assumption is that we'd hit the failure again at the end of the subset, we always start from the beginning.
This may or may not be a safe one :)
Quoted 27 lines
Quoted 11 lines
mm/mremap.c +10
-0+vrm_stat_account (vrm, vrm->new_len); + +anon_vma_unlock_write (new_vma->anon_vma); + pmc.relocate_locked =NULL ; + +do_munmap (current->mm, start, size,NULL ); + relocate_anon = false; + goto again; + } + }[...]
Quoted 4 lines
mm/vma.c +0
-0[...]
Quoted 10 lines
mm/vma.c +6
-0@@ -1821,6 +1834,14 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, new_vma->vm_ops->open (new_vma);if (vma_link (mm, new_vma)) goto out_vma_link; +/* +* If we're attempting to relocate anonymous VMAs, we +* don't want to reuse an anon_vma as set by +* vm_area_dup(), or copy anon_vma_chain or anything +* like this. +*/
Quoted 8 lines
mm/vma.c struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, · line 1844
Quoted 2 lines
mm/vma.c +2
-0+if (*relocate_anon &&__anon_vma_prepare (new_vma)) + goto out_vma_link;Is this bailout really okay? We go to the same label as if
vma_link()failed, but we're in a very different state: For example, the VMA has already been inserted into the VMA tree withvma_iter_store()as part ofvma_link()(unless that changed in one of the prerequisite patches).
Yeah I think you're right, this is an oversight (hey - error paths - fun right??) - will correct.
Quoted 4 lines
Quoted 4 lines
mm/vma.c +0
-0*need_rmap_locks = false; }return new_vma;
RFC PATCH 2/7 mm/mremap: add MREMAP_MUST_RELOCATE_ANON
In reply to Lorenzo StoakesThis flag is the same as MREMAP_RELOCATE_ANON, however it returns an -EFAULT error should folios not be able to be relocated.
The operation is undone when this occurs so the user can choose to proceed without setting this flag at this stage.
This is useful for cases where a use case absolutely requires mergeability, or moreover a user needs to know whether it succeeded or not for internal bookkeeping purposes.
If the move would be a no-op (could be merged, or folios in range are unmapped), then the operation proceeds normally.
It is only in instances where we would have fallen back to the usual mremap() logic if we were using MREMAP_RELOCATE_ANON that we return -EFAULT for MREMAP_MUST_RELOCATE_ANON.
--- include/uapi/linux/mman.h | 9 +++++---- mm/mremap.c | 35 ++++++++++++++++++++++++++--------- 2 files changed, 31 insertions(+), 13 deletions(-)
include/uapi/linux/mman.h +5 -4
@@ -6,10 +6,11 @@#include <asm-generic/hugetlb_encode.h> #include <linux/types.h> -#define MREMAP_MAYMOVE 1 -#define MREMAP_FIXED 2 -#define MREMAP_DONTUNMAP 4 -#define MREMAP_RELOCATE_ANON 8 +#define MREMAP_MAYMOVE 1 +#define MREMAP_FIXED 2 +#define MREMAP_DONTUNMAP 4 +#define MREMAP_RELOCATE_ANON 8 +#define MREMAP_MUST_RELOCATE_ANON 16#define OVERCOMMIT_GUESS 0#define OVERCOMMIT_ALWAYS 1
mm/mremap.c +26 -9
@@ -1352,14 +1352,18 @@ static void unmap_source_vma(struct vma_remap_struct *vrm)* is being moved to by updating index and mapping fields accordingly? */ static bool should_relocate_anon (struct vma_remap_struct *vrm, -struct pagetable_move_control *pmc) +struct pagetable_move_control *pmc,int *errp) {struct vm_area_struct *old = vrm->vma;/* Currently we only do this if requested. */ -if (!(vrm->flags &MREMAP_RELOCATE_ANON )) +if (!(vrm->flags & (MREMAP_RELOCATE_ANON |MREMAP_MUST_RELOCATE_ANON )))return false; +/* Failures are fatal in the 'must' case. */ +if (vrm->flags &MREMAP_MUST_RELOCATE_ANON ) + *errp = -EFAULT ; +/* We can't deal with special or hugetlb mappings. */ if (old->vm_flags & (VM_SPECIAL |VM_HUGETLB ))return false;
@@ -1368,10 +1372,6 @@ static bool should_relocate_anon(struct vma_remap_struct *vrm,if (!vma_is_anonymous (old))return false; -/* If no folios are mapped, then no need to attempt this. */ -if (!old->anon_vma) -return false; -/* * If the old VMA is a child (i.e. has been forked), then the index * references multiple VMAs, we have to bail.
@@ -1379,6 +1379,13 @@ static bool should_relocate_anon(struct vma_remap_struct *vrm,if (!list_is_singular (&old->anon_vma_chain))return false; +/* Below issues are non-fatal in 'must' case. */ + *errp =0 ; + +/* If no folios are mapped, then no need to attempt this. */ +if (!old->anon_vma) +return false; +/* Otherwise, we're good to go! */ return true; }
@@ -1402,7 +1409,10 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm,struct vm_area_struct *new_vma;int err =0 ;PAGETABLE_MOVE (pmc,NULL ,NULL , vrm->addr, vrm->new_addr, vrm->old_len); -bool relocate_anon =should_relocate_anon (vrm, &pmc); +bool relocate_anon =should_relocate_anon (vrm, &pmc, &err); + +if (err) +return err; again: new_vma = copy_vma(&vma, vrm->new_addr, vrm->new_len, new_pgoff,
@@ -1440,6 +1450,12 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm,do_munmap (current->mm, start, size,NULL ); relocate_anon = false; +if (vrm->flags &MREMAP_MUST_RELOCATE_ANON ) { +vrm_uncharge (vrm); + *new_vma_ptr =NULL ; +return -EFAULT ; + } + goto again; } }
@@ -1794,7 +1810,7 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm)/* Ensure no unexpected flag values. */ if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP | -MREMAP_RELOCATE_ANON )) +MREMAP_RELOCATE_ANON |MREMAP_MUST_RELOCATE_ANON ))return -EINVAL ;/* Start address must be page-aligned. */
@@ -1810,7 +1826,8 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm)return -EINVAL ;/* We can't relocate without allowing a move. */ -if ((flags &MREMAP_RELOCATE_ANON ) && !(flags &MREMAP_MAYMOVE )) +if ((flags & (MREMAP_RELOCATE_ANON |MREMAP_MUST_RELOCATE_ANON )) && + !(flags &MREMAP_MAYMOVE ))return -EINVAL ;/* Remainder of checks are for cases with specific new_addr. */
Signature · 1 line
2.48.1
RFC PATCH 3/7 mm/mremap: add MREMAP[_MUST]_RELOCATE_ANON support for THP folios
In reply to Lorenzo StoakesTransparent Huge Pages (THP) represent a problem for the existing implementation, as we don't know if a large folio is fully spanned or not.
If not, then we have a folio that straddles different anon_vma's and thus which cannot be relocated. We could split the folios, but it seems unlikely that a user would prefer such a heavy handed operation (which eliminates the THP performance improvement) to simply disallowing anonymous relocation.
Introduce the ability to both relocate huge PMD and PUDs if wholly contained by the new VMAs and, if the PMD or PUD has been split, to do so at the PTE level also.
In the case that an ordinary mremap() call would split the THP (that is, at the page table level) then we preempt and do so during the relocate anonymous folio step.
We are able to handle large folios of variable size correctly at PTE level generally.
--- mm/mremap.c | 275 +++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 241 insertions(+), 34 deletions(-)
mm/mremap.c +241 -34
@@ -524,40 +524,67 @@ enum pgt_entry { HPAGE_PUD, }; -/* -* Returns an extent of the corresponding size for the pgt_entry specified if -* valid. Else returns a smaller extent bounded by the end of the source and -* destination pgt_entry. -*/ -static __always_inline unsigned long get_extent (enum pgt_entry entry, -struct pagetable_move_control *pmc) +static void __get_mask_size (enum pgt_entry entry, +unsigned long *mask,unsigned long *size) { -unsigned long next, extent, mask, size; -unsigned long old_addr = pmc->old_addr; -unsigned long old_end = pmc->old_end; -unsigned long new_addr = pmc->new_addr; -switch (entry) {case HPAGE_PMD :case NORMAL_PMD : - mask =PMD_MASK ; - size =PMD_SIZE ; + *mask =PMD_MASK ; + *size =PMD_SIZE ;break ;case HPAGE_PUD :case NORMAL_PUD : - mask =PUD_MASK ; - size =PUD_SIZE ; + *mask =PUD_MASK ; + *size =PUD_SIZE ;break ;default :BUILD_BUG ();break ; } +} + +/* Same as get extent, only ignores new address. */ +static unsigned long __get_old_extent (struct pagetable_move_control *pmc, +unsigned long mask,unsigned long size) +{ +unsigned long next, extent; +unsigned long old_addr = pmc->old_addr; +unsigned long old_end = pmc->old_end; next = (old_addr + size) & mask;/* even if next overflowed, extent below will be ok */ extent = next - old_addr;if (extent > old_end - old_addr) extent = old_end - old_addr; + +return extent; +} + +static unsigned long get_old_extent (enum pgt_entry entry, +struct pagetable_move_control *pmc) +{ +unsigned long mask, size; + +__get_mask_size (entry, &mask, &size); +return __get_old_extent (pmc, mask, size); +} + +/* +* Returns an extent of the corresponding size for the pgt_entry specified if +* valid. Else returns a smaller extent bounded by the end of the source and +* destination pgt_entry. +*/ +static __always_inline unsigned long get_extent (enum pgt_entry entry, +struct pagetable_move_control *pmc) +{ +unsigned long next, extent, mask, size; +unsigned long new_addr = pmc->new_addr; + +__get_mask_size (entry, &mask, &size); + + extent =__get_old_extent (pmc, mask, size); + next = (new_addr + size) & mask;if (extent > next - new_addr) extent = next - new_addr;
@@ -787,15 +814,153 @@ static unsigned long pmc_progress(struct pagetable_move_control *pmc)return old_addr < orig_old_addr ?0 : old_addr - orig_old_addr; } +/* Assumes folio lock is held. */ +static bool relocate_large_folio (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr, +struct folio *folio,bool undo) +{ +pgoff_t new_index; +struct vm_area_struct *old = pmc->old; +struct vm_area_struct *new = pmc->new; + +VM_WARN_ON_ONCE (!folio_test_locked (folio)); + +/* Impermissible. */ +if (!folio_test_large (folio) ||folio_test_ksm (folio) || +folio_test_large_maybe_mapped_shared (folio)) +return false; + +/* no-op. */ +if (!folio_test_anon (folio)) +return true; + +if (!undo) + new_index =linear_page_index (new, new_addr); +else + new_index =linear_page_index (old, old_addr); + +/* See comment in relocate_anon_pte(). */ +folio_move_anon_rmap (folio, undo ? old : new); +WRITE_ONCE (folio->index, new_index); +return true; +} + +static bool relocate_anon_pud (struct pagetable_move_control *pmc, +pud_t *pudp,bool undo) +{ +spinlock_t *ptl; +pud_t pud; +struct folio *folio; +struct page *page; +bool ret; +unsigned long old_addr = pmc->old_addr; +unsigned long new_addr = pmc->new_addr; + +VM_WARN_ON (old_addr & ~HPAGE_PUD_MASK ); +VM_WARN_ON (new_addr & ~HPAGE_PUD_MASK ); + + ptl =pud_trans_huge_lock (pudp, pmc->old); +if (!ptl) +return false; + + pud =pudp_get (pudp); +if (!pud_present (pud)) { + ret = true; + goto out; + } +if (!pud_leaf (pud)) { + ret = false; + goto out; + } + + page =pud_page (pud); +if (!page) { + ret = true; + goto out; + } + + folio =page_folio (page); +folio_lock (folio); + ret =relocate_large_folio (pmc, old_addr, new_addr, folio, undo); +folio_unlock (folio); + +out: +spin_unlock (ptl); +return ret; +} + +static bool relocate_anon_pmd (struct pagetable_move_control *pmc, +pmd_t *pmdp,bool undo) +{ +spinlock_t *ptl; +pmd_t pmd; +struct folio *folio; +bool ret; +unsigned long old_addr = pmc->old_addr; +unsigned long new_addr = pmc->new_addr; + +VM_WARN_ON (old_addr & ~HPAGE_PMD_MASK ); +VM_WARN_ON (new_addr & ~HPAGE_PMD_MASK ); + + ptl =pmd_trans_huge_lock (pmdp, pmc->old); +if (!ptl) +return false; + + pmd =pmdp_get (pmdp); +if (!pmd_present (pmd)) { + ret = true; + goto out; + } +if (is_huge_zero_pmd (pmd)) { + ret = true; + goto out; + } +if (!pmd_leaf (pmd)) { + ret = false; + goto out; + } + + folio =pmd_folio (pmd); +if (!folio) { + ret = true; + goto out; + } + +folio_lock (folio); + ret =relocate_large_folio (pmc, old_addr, new_addr, folio, undo); +folio_unlock (folio); +out: +spin_unlock (ptl); +return ret; +} + +/* +* Is the THP discovered at old_addr fully spanned at both the old and new VMAs? +*/ +static bool is_thp_fully_spanned (struct pagetable_move_control *pmc, +unsigned long old_addr, +size_t thp_size) +{ +unsigned long old_end = pmc->old_end; +unsigned long orig_old_addr = old_end - pmc->len_in; +unsigned long aligned_start = old_addr & ~(thp_size -1 ); +unsigned long aligned_end = aligned_start + thp_size; + +if (aligned_start < orig_old_addr || aligned_end > old_end) +return false; + +return true; +} +/* * If the folio mapped at the specified pte entry can have its index and mapping * relocated, then do so. * * Returns the number of pages we have traversed, or 0 if the operation failed. */ -static unsigned long relocate_anon (struct pagetable_move_control *pmc, -unsigned long old_addr,unsigned long new_addr,pte_t pte, -bool undo) +static unsigned long relocate_anon_pte (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr, +pte_t pte,bool undo) {struct page *page;struct folio *folio;
@@ -818,12 +983,18 @@ static unsigned long relocate_anon(struct pagetable_move_control *pmc,if (!folio_test_anon (folio) ||folio_test_ksm (folio)) goto out; -/* -* This should not happen as we explicitly disallow this, but check -* anyway. -*/ +/* If a split huge PMD, try to relocate all at once. */ if (folio_test_large (folio)) { - ret =0 ; +size_t size =folio_size (folio); + +if (is_thp_fully_spanned (pmc, old_addr, size) && +relocate_large_folio (pmc, old_addr, new_addr, folio, undo)) { +VM_WARN_ON_ONCE (old_addr & (size -1 )); + ret =folio_nr_pages (folio); + }else { + ret =0 ; + } + goto out; }
@@ -886,7 +1057,6 @@ static bool relocate_anon_ptes(struct pagetable_move_control *pmc,return false; state.ptep = ptep_start; -for (; !pte_done (&state);pte_next (&state, nr_pages)) {pte_t pte =ptep_get (state.ptep);
@@ -895,7 +1065,8 @@ static bool relocate_anon_ptes(struct pagetable_move_control *pmc,continue ; } - nr_pages =relocate_anon (pmc, state.old_addr, state.new_addr, pte, undo); + nr_pages =relocate_anon_pte (pmc, state.old_addr, + state.new_addr, pte, undo);if (!nr_pages) { ret = false; goto out;
@@ -922,29 +1093,65 @@ static bool __relocate_anon_folios(struct pagetable_move_control *pmc, bool undopmd_t pmd;pud_t pud; - extent =get_extent (NORMAL_PUD , pmc); + extent =get_old_extent (NORMAL_PUD , pmc); pudp =get_old_pud (mm, pmc->old_addr);if (!pudp)continue ; pud =pudp_get (pudp); +if (pud_trans_huge (pud)) { +unsigned long old_addr = pmc->old_addr; + +if (extent !=HPAGE_PUD_SIZE || + !is_thp_fully_spanned (pmc, old_addr,HPAGE_PUD_SIZE )) +return false; + +VM_WARN_ON_ONCE (old_addr & ~HPAGE_PUD_MASK ); -if (pud_trans_huge (pud) ||pud_devmap (pud)) +/* We may relocate iff the new address is aligned. */ +if (!(pmc->new_addr & ~HPAGE_PUD_MASK )) { +if (!relocate_anon_pud (pmc, pudp, undo)) +return false; +continue ; + } + +/* Otherwise, we split so we can do this with PTEs. */ +split_huge_pud (pmc->old, pudp, old_addr); + } + +if (pud_devmap (pud))return false; - extent =get_extent (NORMAL_PMD , pmc); + extent =get_old_extent (NORMAL_PMD , pmc); pmdp =get_old_pmd (mm, pmc->old_addr);if (!pmdp)continue ; pmd =pmdp_get (pmdp); - -if (is_swap_pmd (pmd) ||pmd_trans_huge (pmd) || -pmd_devmap (pmd)) -return false; -if (pmd_none (pmd))continue ; +if (pmd_trans_huge (pmd)) { +unsigned long old_addr = pmc->old_addr; + +if (extent !=HPAGE_PMD_SIZE || + !is_thp_fully_spanned (pmc, old_addr,HPAGE_PMD_SIZE )) +return false; + +VM_WARN_ON_ONCE (old_addr & ~HPAGE_PMD_MASK ); + +/* We may relocate iff the new address is aligned. */ +if (!(pmc->new_addr & ~HPAGE_PMD_MASK )) { +if (!relocate_anon_pmd (pmc, pmdp, undo)) +return false; +continue ; + } + +/* Otherwise, we split so we can do this with PTEs. */ +split_huge_pmd (pmc->old, pmdp, old_addr); + }else if (is_swap_pmd (pmd) ||pmd_devmap (pmd)) { +return false; + } +if (!relocate_anon_ptes (pmc, extent, pmdp, undo))return false; }
Signature · 1 line
2.48.1
RFC PATCH 4/7 tools UAPI: Update copy of linux/mman.h from the kernel sources
In reply to Lorenzo StoakesImport newly introduced MREMAP_RELOCATE_ANON_* defines.
--- tools/include/uapi/linux/mman.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
tools/include/uapi/linux/mman.h +5 -3
@@ -6,9 +6,11 @@#include <asm-generic/hugetlb_encode.h> #include <linux/types.h> -#define MREMAP_MAYMOVE 1 -#define MREMAP_FIXED 2 -#define MREMAP_DONTUNMAP 4 +#define MREMAP_MAYMOVE 1 +#define MREMAP_FIXED 2 +#define MREMAP_DONTUNMAP 4 +#define MREMAP_RELOCATE_ANON 8 +#define MREMAP_MUST_RELOCATE_ANON 16#define OVERCOMMIT_GUESS 0#define OVERCOMMIT_ALWAYS 1
Signature · 1 line
2.48.1
RFC PATCH 5/7 tools/testing/selftests: add mremap() cases that merge normally
In reply to Lorenzo StoakesUse a direct system call version of mremap() as, when we move to using MREMAP_[MUST_]RELOCATE_ANON, the glibc wrapper will disallow this.
Also import linux/mman.h (which will amount to the local tools cache of mman.h) to enusre these header values are available when later added.
Then, add tests asserting all the mremap() merge cases that function correctly without MREMAP_[MUST_]RELOCATE_ANON.
This constitutes moving around unfaulted VMAs and moving around faulted VMAs back into position immediately adjacent to VMAs also faulted in with that moved VMA.
By doing so we provide a baseline set of expectations on mremap() operations and VMA merge which we can expand upon for MREMAP_[MUST_]RELOCATE_ANON cases in a subsequent commit.
--- tools/testing/selftests/mm/merge.c | 608 ++++++++++++++++++++++++++++- 1 file changed, 606 insertions(+), 2 deletions(-)
tools/testing/selftests/mm/merge.c +606 -2
@@ -8,6 +8,16 @@#include <sys/mman.h> #include <sys/wait.h> #include "vm_util.h" +#include <linux/mman.h> +#include <sys/syscall.h> + +static void *sys_mremap (void *old_address,unsigned long old_size, +unsigned long new_size,int flags,void *new_address) +{ +return (void *)syscall (__NR_mremap, (unsigned long )old_address, + old_size, new_size, flags, + (unsigned long )new_address); +}FIXTURE (merge) {
@@ -20,7 +30,7 @@ FIXTURE_SETUP(merge) { self->page_size =psize ();/* Carve out PROT_NONE region to map over. */ - self->carveout =mmap (NULL ,12 * self->page_size,PROT_NONE , + self->carveout =mmap (NULL ,30 * self->page_size,PROT_NONE ,MAP_ANON |MAP_PRIVATE ,-1 ,0 );ASSERT_NE (self->carveout,MAP_FAILED );/* Setup PROCMAP_QUERY interface. */
@@ -29,7 +39,7 @@ FIXTURE_SETUP(merge)FIXTURE_TEARDOWN (merge) { -ASSERT_EQ (munmap (self->carveout,12 * self->page_size),0 ); +ASSERT_EQ (munmap (self->carveout,30 * self->page_size),0 );ASSERT_EQ (close_procmap (&self->procmap),0 ); }
@@ -451,4 +461,598 @@ TEST_F(merge, forked_source_vma)ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr2 +5 * page_size); } +TEST_F(merge, mremap_unfaulted_to_faulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2; + +/* +* Map two distinct areas: +* +* |-----------| |-----------| +* | unfaulted | | unfaulted | +* |-----------| |-----------| +* ptr ptr2 +*/ + ptr =mmap (&carveout[page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[7 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* Offset ptr2 further away. */ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Fault in ptr: +* \ +* |-----------| / |-----------| +* | faulted | \ | unfaulted | +* |-----------| / |-----------| +* ptr \ ptr2 +*/ + ptr[0 ] ='x' ; + +/* +* Now move ptr2 adjacent to ptr: +* +* |-----------|-----------| +* | faulted | unfaulted | +* |-----------|-----------| +* ptr ptr2 +* +* It should merge: +* +* |----------------------| +* | faulted | +* |----------------------| +* ptr +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +10 * page_size); +} + +TEST_F(merge, mremap_unfaulted_behind_faulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2; + +/* +* Map two distinct areas: +* +* |-----------| |-----------| +* | unfaulted | | unfaulted | +* |-----------| |-----------| +* ptr ptr2 +*/ + ptr =mmap (&carveout[6 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[14 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* Offset ptr2 further away. */ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Fault in ptr: +* \ +* |-----------| / |-----------| +* | faulted | \ | unfaulted | +* |-----------| / |-----------| +* ptr \ ptr2 +*/ + ptr[0 ] ='x' ; + +/* +* Now move ptr2 adjacent, but behind, ptr: +* +* |-----------|-----------| +* | unfaulted | faulted | +* |-----------|-----------| +* ptr2 ptr +* +* It should merge: +* +* |----------------------| +* | faulted | +* |----------------------| +* ptr2 +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &carveout[page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr2)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr2); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr2 +10 * page_size); +} + +TEST_F(merge, mremap_unfaulted_between_faulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2, *ptr3; + +/* +* Map three distinct areas: +* +* |-----------| |-----------| |-----------| +* | unfaulted | | unfaulted | | unfaulted | +* |-----------| |-----------| |-----------| +* ptr ptr2 ptr3 +*/ + ptr =mmap (&carveout[page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[7 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + ptr3 =mmap (&carveout[14 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* Offset ptr3 further away. */ + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr3 + page_size *2000 ); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* Offset ptr2 further away. */ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Fault in ptr, ptr3: +* \ \ +* |-----------| / |-----------| / |-----------| +* | faulted | \ | unfaulted | \ | faulted | +* |-----------| / |-----------| / |-----------| +* ptr \ ptr2 \ ptr3 +*/ + ptr[0 ] ='x' ; + ptr3[0 ] ='x' ; + +/* +* Move ptr3 back into place, leaving a place for ptr2: +* \ +* |-----------| |-----------| / |-----------| +* | faulted | | faulted | \ | unfaulted | +* |-----------| |-----------| / |-----------| +* ptr ptr3 \ ptr2 +*/ + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[10 * page_size]); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* +* Finally, move ptr2 into place: +* +* |-----------|-----------|-----------| +* | faulted | unfaulted | faulted | +* |-----------|-----------|-----------| +* ptr ptr2 ptr3 +* +* It should merge, but only ptr, ptr2: +* +* |-----------------------|-----------| +* | faulted | unfaulted | +* |-----------------------|-----------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +10 * page_size); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr3)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr3); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr3 +5 * page_size); +} + +TEST_F(merge, mremap_unfaulted_between_faulted_unfaulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2, *ptr3; + +/* +* Map three distinct areas: +* +* |-----------| |-----------| |-----------| +* | unfaulted | | unfaulted | | unfaulted | +* |-----------| |-----------| |-----------| +* ptr ptr2 ptr3 +*/ + ptr =mmap (&carveout[page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[7 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + ptr3 =mmap (&carveout[14 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* Offset ptr3 further away. */ + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr3 + page_size *2000 ); +ASSERT_NE (ptr3,MAP_FAILED ); + + +/* Offset ptr2 further away. */ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Fault in ptr: +* \ \ +* |-----------| / |-----------| / |-----------| +* | faulted | \ | unfaulted | \ | unfaulted | +* |-----------| / |-----------| / |-----------| +* ptr \ ptr2 \ ptr3 +*/ + ptr[0 ] ='x' ; + +/* +* Move ptr3 back into place, leaving a place for ptr2: +* \ +* |-----------| |-----------| / |-----------| +* | faulted | | unfaulted | \ | unfaulted | +* |-----------| |-----------| / |-----------| +* ptr ptr3 \ ptr2 +*/ + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[10 * page_size]); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* +* Finally, move ptr2 into place: +* +* |-----------|-----------|-----------| +* | faulted | unfaulted | unfaulted | +* |-----------|-----------|-----------| +* ptr ptr2 ptr3 +* +* It should merge: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +15 * page_size); +} + +TEST_F(merge, mremap_unfaulted_between_correctly_placed_faulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2; + +/* +* Map one larger area: +* +* |-----------------------------------| +* | unfaulted | +* |-----------------------------------| +*/ + ptr =mmap (&carveout[page_size],15 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + +/* +* Fault in ptr: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +*/ + ptr[0 ] ='x' ; + +/* +* Unmap middle: +* +* |-----------| |-----------| +* | faulted | | faulted | +* |-----------| |-----------| +* +* Now the faulted areas are compatible with each other (anon_vma the +* same, vma->vm_pgoff equal to virtual page offset). +*/ +ASSERT_EQ (munmap (&ptr[5 * page_size],5 * page_size),0 ); + +/* +* Map a new area, ptr2: +* \ +* |-----------| |-----------| / |-----------| +* | faulted | | faulted | \ | unfaulted | +* |-----------| |-----------| / |-----------| +* ptr \ ptr2 +*/ + ptr2 =mmap (&carveout[20 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Finally, move ptr2 into place: +* +* |-----------|-----------|-----------| +* | faulted | unfaulted | faulted | +* |-----------|-----------|-----------| +* ptr ptr2 ptr3 +* +* It should merge: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +15 * page_size); +} + +TEST_F(merge, mremap_correct_placed_faulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2, *ptr3; + +/* +* Map one larger area: +* +* |-----------------------------------| +* | unfaulted | +* |-----------------------------------| +*/ + ptr =mmap (&carveout[page_size],15 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + +/* +* Fault in ptr: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +*/ + ptr[0 ] ='x' ; + +/* +* Offset the final and middle 5 pages further away: +* \ \ +* |-----------| / |-----------| / |-----------| +* | faulted | \ | faulted | \ | faulted | +* |-----------| / |-----------| / |-----------| +* ptr \ ptr2 \ ptr3 +*/ + ptr3 = &ptr[10 * page_size]; + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr3 + page_size *2000 ); +ASSERT_NE (ptr3,MAP_FAILED ); + ptr2 = &ptr[5 * page_size]; + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Move ptr2 into its correct place: +* \ +* |-----------|-----------| / |-----------| +* | faulted | faulted | \ | faulted | +* |-----------|-----------| / |-----------| +* ptr ptr2 \ ptr3 +* +* It should merge: +* \ +* |-----------------------| / |-----------| +* | faulted | \ | faulted | +* |-----------------------| / |-----------| +* ptr \ ptr3 +*/ + + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +10 * page_size); + +/* +* Now move ptr out of place: +* \ \ +* |-----------| / |-----------| / |-----------| +* | faulted | \ | faulted | \ | faulted | +* |-----------| / |-----------| / |-----------| +* ptr2 \ ptr \ ptr3 +*/ + ptr =sys_mremap (ptr,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr + page_size *1000 ); +ASSERT_NE (ptr,MAP_FAILED ); + +/* +* Now move ptr back into place: +* \ +* |-----------|-----------| / |-----------| +* | faulted | faulted | \ | faulted | +* |-----------|-----------| / |-----------| +* ptr ptr2 \ ptr3 +* +* It should merge: +* \ +* |-----------------------| / |-----------| +* | faulted | \ | faulted | +* |-----------------------| / |-----------| +* ptr \ ptr3 +*/ + ptr =sys_mremap (ptr,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &carveout[page_size]); +ASSERT_NE (ptr,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +10 * page_size); + +/* +* Now move ptr out of place again: +* \ \ +* |-----------| / |-----------| / |-----------| +* | faulted | \ | faulted | \ | faulted | +* |-----------| / |-----------| / |-----------| +* ptr2 \ ptr \ ptr3 +*/ + ptr =sys_mremap (ptr,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr + page_size *1000 ); +ASSERT_NE (ptr,MAP_FAILED ); + +/* +* Now move ptr3 back into place: +* \ +* |-----------|-----------| / |-----------| +* | faulted | faulted | \ | faulted | +* |-----------|-----------| / |-----------| +* ptr2 ptr3 \ ptr +* +* It should merge: +* \ +* |-----------------------| / |-----------| +* | faulted | \ | faulted | +* |-----------------------| / |-----------| +* ptr2 \ ptr +*/ + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr2[5 * page_size]); +ASSERT_NE (ptr3,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr2)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr2); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr2 +10 * page_size); + +/* +* Now move ptr back into place: +* +* |-----------|-----------------------| +* | faulted | faulted | +* |-----------|-----------------------| +* ptr ptr2 +* +* It should merge: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +* ptr +*/ + ptr =sys_mremap (ptr,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &carveout[page_size]); +ASSERT_NE (ptr,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +15 * page_size); + +/* +* Now move ptr2 out of the way: +* \ +* |-----------| |-----------| / |-----------| +* | faulted | | faulted | \ | faulted | +* |-----------| |-----------| / |-----------| +* ptr ptr3 \ ptr2 +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Now move it back: +* +* |-----------|-----------|-----------| +* | faulted | faulted | faulted | +* |-----------|-----------|-----------| +* ptr ptr2 ptr3 +* +* It should merge: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +* ptr +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +15 * page_size); + +/* +* Move ptr3 out of place: +* \ +* |-----------------------| / |-----------| +* | faulted | \ | faulted | +* |-----------------------| / |-----------| +* ptr \ ptr3 +*/ + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr3 + page_size *1000 ); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* +* Now move it back: +* +* |-----------|-----------|-----------| +* | faulted | faulted | faulted | +* |-----------|-----------|-----------| +* ptr ptr2 ptr3 +* +* It should merge: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +* ptr +*/ + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[10 * page_size]); +ASSERT_NE (ptr3,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +15 * page_size); +} +TEST_HARNESS_MAIN
Signature · 1 line
2.48.1
RFC PATCH 6/7 tools/testing/selftests: add MREMAP_RELOCATE_ANON merge test cases
In reply to Lorenzo StoakesAdd test cases to the mm self test asserting that the merge cases which the newly introduced MREMAP[_MUST]_RELOCATE_ANON results in merges occurring as expected, which otherwise without it would not succeed.
This extends the newly introduced VMA merge self tests for these cases and exhaustively attempts each merge case, asserting expected behaviour.
We use the MREMAP_MUST_RELOCATE_ANON variant to ensure that, should the anon relocate fail, we observe an error, as quietly demoting the move to non-relocate anon would cause unusual test failures.
We carefully document each case to make clear what we are testing.
--- tools/testing/selftests/mm/merge.c | 730 +++++++++++++++++++++++++++++ 1 file changed, 730 insertions(+)
tools/testing/selftests/mm/merge.c +730 -0
@@ -1055,4 +1055,734 @@ TEST_F(merge, mremap_correct_placed_faulted)ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +15 * page_size); } +TEST_F(merge, mremap_relocate_anon_faulted_after_unfaulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2; + +/* +* Map two distinct areas: +* +* |-----------| |-----------| +* | unfaulted | | unfaulted | +* |-----------| |-----------| +* ptr ptr2 +*/ + ptr =mmap (&carveout[page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[7 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Offset ptr2 further away. Note we don't have to use +* MREMAP_RELOCATE_ANON yet. +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Fault ptr2 in: +* \ +* |-----------| / |-----------| +* | unfaulted | \ | faulted | +* |-----------| / |-----------| +* ptr \ ptr2 +*/ + ptr2[0 ] ='x' ; + +/* +* Move ptr2 after ptr, using MREMAP_MUST_RELOCATE_ANON: +* +* |-----------|-----------| +* | unfaulted | faulted | +* |-----------|-----------| +* ptr ptr2 +* +* It should merge: +* +* |-----------------------| +* | faulted | +* |-----------------------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED |MREMAP_MUST_RELOCATE_ANON , + &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +10 * page_size); +} + +TEST_F(merge, mremap_relocate_anon_faulted_before_unfaulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2; + +/* +* Map two distinct areas: +* +* |-----------| |-----------| +* | unfaulted | | unfaulted | +* |-----------| |-----------| +* ptr ptr2 +*/ + ptr =mmap (&carveout[6 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[12 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Offset ptr2 further away. Note we don't have to use +* MREMAP_RELOCATE_ANON yet. +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Fault ptr2 in: +* \ +* |-----------| / |-----------| +* | unfaulted | \ | faulted | +* |-----------| / |-----------| +* ptr \ ptr2 +*/ + ptr2[0 ] ='x' ; + +/* +* Move ptr2 before ptr, using MREMAP_MUST_RELOCATE_ANON: +* +* |-----------|-----------| +* | faulted | unfaulted | +* |-----------|-----------| +* ptr2 ptr +* +* It should merge: +* +* |-----------------------| +* | faulted | +* |-----------------------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED |MREMAP_MUST_RELOCATE_ANON , + &carveout[page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr2)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr2); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr2 +10 * page_size); +} + +TEST_F(merge, mremap_relocate_anon_faulted_between_unfaulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2, *ptr3; + +/* +* Map three distinct areas: +* +* |-----------| |-----------| |-----------| +* | unfaulted | | unfaulted | | unfaulted | +* |-----------| |-----------| |-----------| +* ptr ptr2 ptr3 +*/ + ptr =mmap (&carveout[page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[7 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + ptr3 =mmap (&carveout[14 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* +* Offset ptr2 further away, and move ptr3 into position: +* \ +* |-----------| |-----------| / |-----------| +* | unfaulted | | unfaulted | \ | unfaulted | +* |-----------| |-----------| / |-----------| +* ptr ptr3 \ ptr2 +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr3 + page_size *2000 ); +ASSERT_NE (ptr3,MAP_FAILED ); + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[10 * page_size]); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* +* Fault in ptr2: +* \ +* |-----------| |-----------| / |-----------| +* | unfaulted | | unfaulted | \ | faulted | +* |-----------| |-----------| / |-----------| +* ptr ptr3 \ ptr2 +*/ + ptr2[0 ] ='x' ; + +/* +* Move ptr2 between ptr, ptr3, using MREMAP_MUST_RELOCATE_ANON: +* +* |-----------|-----------|-----------| +* | unfaulted | faulted | unfaulted | +* |-----------|-----------|-----------| +* +* It should merge: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED |MREMAP_MUST_RELOCATE_ANON , + &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +15 * page_size); +} + +TEST_F(merge, mremap_relocate_anon_faulted_after_faulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2; + +/* +* Map two distinct areas: +* +* |-----------| |-----------| +* | unfaulted | | unfaulted | +* |-----------| |-----------| +* ptr ptr2 +*/ + ptr =mmap (&carveout[page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[7 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Offset ptr2 further away. Note we don't have to use +* MREMAP_RELOCATE_ANON yet. +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Fault ptr and ptr2 in: +* \ +* |-----------| / |-----------| +* | faulted | \ | faulted | +* |-----------| / |-----------| +* ptr \ ptr2 +*/ + ptr[0 ] ='x' ; + ptr2[0 ] ='x' ; + +/* +* Move ptr2 after ptr, using MREMAP_MUST_RELOCATE_ANON: +* +* |-----------|-----------| +* | faulted | faulted | +* |-----------|-----------| +* ptr ptr2 +* +* It should merge: +* +* |-----------------------| +* | faulted | +* |-----------------------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED |MREMAP_MUST_RELOCATE_ANON , + &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +10 * page_size); +} + +TEST_F(merge, mremap_relocate_anon_faulted_before_faulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2; + +/* +* Map two distinct areas: +* +* |-----------| |-----------| +* | unfaulted | | unfaulted | +* |-----------| |-----------| +* ptr ptr2 +*/ + ptr =mmap (&carveout[6 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[12 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Offset ptr2 further away. Note we don't have to use +* MREMAP_RELOCATE_ANON yet. +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Fault ptr, ptr2 in: +* \ +* |-----------| / |-----------| +* | faulted | \ | faulted | +* |-----------| / |-----------| +* ptr \ ptr2 +*/ + ptr[0 ] ='x' ; + ptr2[0 ] ='x' ; + +/* +* Move ptr2 before ptr, using MREMAP_MUST_RELOCATE_ANON: +* +* |-----------|-----------| +* | faulted | faulted | +* |-----------|-----------| +* ptr2 ptr +* +* It should merge: +* +* |-----------------------| +* | faulted | +* |-----------------------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED |MREMAP_MUST_RELOCATE_ANON , + &carveout[page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr2)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr2); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr2 +10 * page_size); +} + +TEST_F(merge, mremap_relocate_anon_faulted_between_faulted_unfaulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2, *ptr3; + +/* +* Map three distinct areas: +* +* |-----------| |-----------| |-----------| +* | unfaulted | | unfaulted | | unfaulted | +* |-----------| |-----------| |-----------| +* ptr ptr2 ptr3 +*/ + ptr =mmap (&carveout[page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[7 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + ptr3 =mmap (&carveout[14 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* +* Offset ptr2 further away, and move ptr3 into position: +* \ +* |-----------| |-----------| / |-----------| +* | unfaulted | | unfaulted | \ | unfaulted | +* |-----------| |-----------| / |-----------| +* ptr ptr3 \ ptr2 +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr3 + page_size *2000 ); +ASSERT_NE (ptr3,MAP_FAILED ); + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[10 * page_size]); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* +* Fault in ptr, ptr2: +* \ +* |-----------| |-----------| / |-----------| +* | faulted | | unfaulted | \ | faulted | +* |-----------| |-----------| / |-----------| +* ptr ptr3 \ ptr2 +*/ + ptr[0 ] ='x' ; + ptr2[0 ] ='x' ; + +/* +* Move ptr2 between ptr, ptr3, using MREMAP_MUST_RELOCATE_ANON: +* +* |-----------|-----------|-----------| +* | faulted | faulted | unfaulted | +* |-----------|-----------|-----------| +* +* It should merge: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED |MREMAP_MUST_RELOCATE_ANON , + &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +15 * page_size); +} + +TEST_F(merge, mremap_relocate_anon_faulted_between_unfaulted_faulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2, *ptr3; + +/* +* Map three distinct areas: +* +* |-----------| |-----------| |-----------| +* | unfaulted | | unfaulted | | unfaulted | +* |-----------| |-----------| |-----------| +* ptr ptr2 ptr3 +*/ + ptr =mmap (&carveout[page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[7 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + ptr3 =mmap (&carveout[14 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* +* Offset ptr2 further away, and move ptr3 into position: +* \ +* |-----------| |-----------| / |-----------| +* | unfaulted | | unfaulted | \ | unfaulted | +* |-----------| |-----------| / |-----------| +* ptr ptr3 \ ptr2 +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr3 + page_size *2000 ); +ASSERT_NE (ptr3,MAP_FAILED ); + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[10 * page_size]); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* +* Fault in ptr2, ptr3: +* \ +* |-----------| |-----------| / |-----------| +* | unfaulted | | faulted | \ | faulted | +* |-----------| |-----------| / |-----------| +* ptr ptr3 \ ptr2 +*/ + ptr2[0 ] ='x' ; + ptr3[0 ] ='x' ; + +/* +* Move ptr2 between ptr, ptr3, using MREMAP_MUST_RELOCATE_ANON: +* +* |-----------|-----------|-----------| +* | unfaulted | faulted | faulted | +* |-----------|-----------|-----------| +* +* It should merge: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED |MREMAP_MUST_RELOCATE_ANON , + &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +15 * page_size); +} + +TEST_F(merge, mremap_relocate_anon_faulted_between_faulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2, *ptr3; + +/* +* Map three distinct areas: +* +* |-----------| |-----------| |-----------| +* | unfaulted | | unfaulted | | unfaulted | +* |-----------| |-----------| |-----------| +* ptr ptr2 ptr3 +*/ + ptr =mmap (&carveout[page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[7 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + ptr3 =mmap (&carveout[14 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* +* Offset ptr2 further away, and move ptr3 into position: +* \ +* |-----------| |-----------| / |-----------| +* | unfaulted | | unfaulted | \ | unfaulted | +* |-----------| |-----------| / |-----------| +* ptr ptr3 \ ptr2 +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr2 + page_size *1000 ); +ASSERT_NE (ptr2,MAP_FAILED ); + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , ptr3 + page_size *2000 ); +ASSERT_NE (ptr3,MAP_FAILED ); + ptr3 =sys_mremap (ptr3,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED , &ptr[10 * page_size]); +ASSERT_NE (ptr3,MAP_FAILED ); + +/* +* Fault in ptr, ptr2, ptr3: +* \ +* |-----------| |-----------| / |-----------| +* | faulted | | faulted | \ | faulted | +* |-----------| |-----------| / |-----------| +* ptr ptr3 \ ptr2 +*/ + ptr[0 ] ='x' ; + ptr2[0 ] ='x' ; + ptr3[0 ] ='x' ; + +/* +* Move ptr2 between ptr, ptr3, using MREMAP_MUST_RELOCATE_ANON: +* +* |-----------|-----------|-----------| +* | faulted | faulted | faulted | +* |-----------|-----------|-----------| +* +* It should merge, but only the latter two VMAs: +* +* |-----------|-----------------------| +* | faulted | faulted | +* |-----------|-----------------------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED |MREMAP_MUST_RELOCATE_ANON , + &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr2)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr2); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr2 +10 * page_size); +} + +TEST_F(merge, mremap_relocate_anon_faulted_between_correctly_placed_faulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2; + +/* +* Map one larger area: +* +* |-----------------------------------| +* | unfaulted | +* |-----------------------------------| +*/ + ptr =mmap (&carveout[page_size],15 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + +/* +* Fault in ptr: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +*/ + ptr[0 ] ='x' ; + +/* +* Unmap middle: +* +* |-----------| |-----------| +* | faulted | | faulted | +* |-----------| |-----------| +* +* Now the faulted areas are compatible with each other (anon_vma the +* same, vma->vm_pgoff equal to virtual page offset). +*/ +ASSERT_EQ (munmap (&ptr[5 * page_size],5 * page_size),0 ); + +/* +* Map a new area, ptr2: +* \ +* |-----------| |-----------| / |-----------| +* | faulted | | faulted | \ | unfaulted | +* |-----------| |-----------| / |-----------| +* ptr \ ptr2 +*/ + ptr2 =mmap (&carveout[20 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Fault it in: +* \ +* |-----------| |-----------| / |-----------| +* | faulted | | faulted | \ | faulted | +* |-----------| |-----------| / |-----------| +* ptr \ ptr2 +*/ + ptr2[0 ] ='x' ; + +/* +* Finally, move ptr2 into place, using MREMAP_MUST_RELOCATE_ANON: +* +* |-----------|-----------|-----------| +* | faulted | faulted | faulted | +* |-----------|-----------|-----------| +* ptr ptr2 ptr3 +* +* It should merge: +* +* |-----------------------------------| +* | faulted | +* |-----------------------------------| +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED |MREMAP_MUST_RELOCATE_ANON , + &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +15 * page_size); +} + +TEST_F(merge, mremap_relocate_anon_mprotect_faulted_faulted) +{ +unsigned int page_size = self->page_size; +char *carveout = self->carveout; +struct procmap_fd *procmap = &self->procmap; +char *ptr, *ptr2; + + +/* +* Map two distinct areas: +* +* |-----------| |-----------| +* | unfaulted | | unfaulted | +* |-----------| |-----------| +* ptr ptr2 +*/ + ptr =mmap (&carveout[page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr,MAP_FAILED ); + ptr2 =mmap (&carveout[12 * page_size],5 * page_size,PROT_READ |PROT_WRITE , +MAP_ANON |MAP_PRIVATE |MAP_FIXED ,-1 ,0 ); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* +* Fault in ptr, ptr2, mprotect() ptr2 read-only: +* +* RW RO +* |-----------| |-----------| +* | faulted | | faulted | +* |-----------| |-----------| +* ptr ptr2 +*/ + ptr[0 ] ='x' ; + ptr2[0 ] ='x' ; +ASSERT_EQ (mprotect (ptr2,5 * page_size,PROT_READ ),0 ); + +/* +* Move ptr2 next to ptr: +* +* RW RO +* |-----------|-----------| +* | faulted | faulted | +* |-----------|-----------| +* ptr ptr2 +*/ + ptr2 =sys_mremap (ptr2,5 * page_size,5 * page_size, +MREMAP_MAYMOVE |MREMAP_FIXED |MREMAP_MUST_RELOCATE_ANON , + &ptr[5 * page_size]); +ASSERT_NE (ptr2,MAP_FAILED ); + +/* No merge should happen. */ +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +5 * page_size); + +/* +* Now mremap ptr2 RW: +* +* RW RW +* |-----------|-----------| +* | faulted | faulted | +* |-----------|-----------| +* ptr ptr2 +* +* This should result in a merge: +* +* RW +* |-----------------------| +* | faulted | +* |-----------------------| +* ptr +*/ +ASSERT_EQ (mprotect (ptr2,5 * page_size,PROT_READ |PROT_WRITE ),0 ); + +ASSERT_TRUE (find_vma_procmap (procmap, ptr)); +ASSERT_EQ (procmap->query.vma_start, (unsigned long )ptr); +ASSERT_EQ (procmap->query.vma_end, (unsigned long )ptr +10 * page_size); +} +TEST_HARNESS_MAIN
Signature · 1 line
2.48.1
RFC PATCH 7/7 tools/testing/selftests: expand mremap() tests for MREMAP_RELOCATE_ANON
In reply to Lorenzo StoakesAdjust every relevant test (that is, one that moves memory) to also perform the same test using MREMAP_MUST_RELOCATE_ANON to assert that it is behaving as expected.
In order to avoid glibc not being up-to-date, also move to using the mremap() system call direct, and import the linux/mman.h header, which will use the tool linux header wrappers, to get the latest mremap defines.
Also take careful precaution in the instance where we might unexpectedly fail the 'mremap move within range' test due to large folios mapped outside of the range we are relocating.
In these instances, if we test with MREMAP_MUST_RELOCATE_ANON, we ensure the folios in question are not huge. If testing with MREMAP_RELOCATE_ANON we do not - this asserts that this correctly falls back to non-relocate anon behaviour.
In cases where MREMAP_MUST_RELOCATE_ANON is used, we attempt to immediately trigger reclaim to also assert that the rmap state is uncorrupted.
--- tools/testing/selftests/mm/mremap_test.c | 270 +++++++++++++++-------- 1 file changed, 176 insertions(+), 94 deletions(-)
tools/testing/selftests/mm/mremap_test.c +176 -94
@@ -8,9 +8,11 @@#include <stdlib.h> #include <stdio.h> #include <string.h> +#include <linux/mman.h> #include <sys/mman.h> #include <time.h> #include <stdbool.h> +#include <sys/syscall.h> #include "../kselftest.h"
@@ -29,11 +31,20 @@#define SIZE_MB (m) ((size_t)m * (1024 * 1024))#define SIZE_KB (k) ((size_t)k * 1024) +static void *sys_mremap (void *old_address,unsigned long old_size, +unsigned long new_size,int flags,void *new_address) +{ +return (void *)syscall (__NR_mremap, (unsigned long )old_address, + old_size, new_size, flags, + (unsigned long )new_address); +} +struct config {unsigned long long src_alignment;unsigned long long dest_alignment;unsigned long long region_size;int overlapping; +bool use_relocate_anon;unsigned int dest_preamble_size; };
@@ -60,7 +71,8 @@ enum {#define PTE page_size#define MAKE_TEST (source_align, destination_align, size, \ - overlaps, should_fail, test_name) \ + overlaps, use_relocate_anon, should_fail, \ + test_name) \ (struct test){ \ .name = test_name, \ .config = { \
@@ -68,6 +80,7 @@ enum { .dest_alignment = destination_align, \ .region_size = size, \ .overlapping = overlaps, \ + .use_relocate_anon = use_relocate_anon, \ }, \ .expect_failure = should_fail \ }
@@ -184,6 +197,12 @@ static void *get_source_mapping(struct config c)unsigned long long addr =0ULL ;void *src_addr =NULL ;unsigned long long mmap_min_addr; +int mmap_flags =MAP_FIXED_NOREPLACE |MAP_ANONYMOUS ; + +if (c.use_relocate_anon) + mmap_flags |=MAP_PRIVATE ; +else + mmap_flags |=MAP_SHARED ; mmap_min_addr =get_mmap_min_addr ();/*
@@ -198,8 +217,7 @@ static void *get_source_mapping(struct config c) goto retry; src_addr =mmap ((void *) addr, c.region_size,PROT_READ |PROT_WRITE , -MAP_FIXED_NOREPLACE |MAP_ANONYMOUS |MAP_SHARED , --1 ,0 ); + mmap_flags,-1 ,0 );if (src_addr ==MAP_FAILED ) {if (errno ==EPERM || errno ==EEXIST ) goto retry;
@@ -251,7 +269,7 @@ static void mremap_expand_merge(FILE *maps_fp, unsigned long page_size) }munmap (start + page_size, page_size); - remap =mremap (start, page_size,2 * page_size,0 ); + remap =sys_mremap (start, page_size,2 * page_size,0 ,0 );if (remap ==MAP_FAILED ) {ksft_print_msg ("mremap failed: %s\n" ,strerror (errno));munmap (start, page_size);
@@ -292,7 +310,8 @@ static void mremap_expand_merge_offset(FILE *maps_fp, unsigned long page_size)/* Unmap final page to ensure we have space to expand. */ munmap (start +2 * page_size, page_size); - remap =mremap (start + page_size, page_size,2 * page_size,0 ); + + remap =sys_mremap (start + page_size, page_size,2 * page_size,0 ,0 );if (remap ==MAP_FAILED ) {ksft_print_msg ("mremap failed: %s\n" ,strerror (errno));munmap (start,2 * page_size);
@@ -324,20 +343,35 @@ static void mremap_expand_merge_offset(FILE *maps_fp, unsigned long page_size)* * |DDDDddddSSSSssss| */ -static void mremap_move_within_range (unsigned int pattern_seed,char *rand_addr) +static void mremap_move_within_range (unsigned int pattern_seed,char *rand_addr, +char *test_suffix,int extra_flags) {char *test_name ="mremap mremap move within range" ;void *src, *dest;unsigned int i, success =1 ; -size_t size =SIZE_MB (20 );void *ptr =mmap (NULL , size,PROT_READ |PROT_WRITE ,MAP_PRIVATE |MAP_ANONYMOUS ,-1 ,0 ); +int mremap_flags =MREMAP_MAYMOVE |MREMAP_FIXED ; +if (ptr ==MAP_FAILED ) {perror ("mmap" ); success =0 ; goto out; } + +/* +* If THP is enabled, we may end up spanning a range which has large +* folios not enclosed within the mapping, which will disallow the +* relocate. +* +* In this case, disallow huge pages in the range. +*/ +if (extra_flags &MREMAP_MUST_RELOCATE_ANON ) +madvise (ptr, size,MADV_NOHUGEPAGE ); + + mremap_flags |= extra_flags; +memset (ptr,0 , size); src = ptr +SIZE_MB (6 );
@@ -348,8 +382,8 @@ static void mremap_move_within_range(unsigned int pattern_seed, char *rand_addr) dest = src -SIZE_MB (2 ); -void *new_ptr =mremap (src +SIZE_MB (1 ),SIZE_MB (1 ),SIZE_MB (1 ), -MREMAP_MAYMOVE |MREMAP_FIXED , dest +SIZE_MB (1 )); +void *new_ptr =sys_mremap (src +SIZE_MB (1 ),SIZE_MB (1 ),SIZE_MB (1 ), + mremap_flags, dest +SIZE_MB (1 ));if (new_ptr ==MAP_FAILED ) {perror ("mremap" ); success =0 ;
@@ -375,9 +409,9 @@ static void mremap_move_within_range(unsigned int pattern_seed, char *rand_addr)perror ("munmap" );if (success) -ksft_test_result_pass ("%s\n" , test_name); +ksft_test_result_pass ("%s%s\n" , test_name, test_suffix);else -ksft_test_result_fail ("%s\n" , test_name); +ksft_test_result_fail ("%s%s\n" , test_name, test_suffix); }/* Returns the time taken for the remap on success else returns -1. */
@@ -390,6 +424,10 @@ static long long remap_region(struct config c, unsigned int threshold_mb,long long start_ns, end_ns, align_mask, ret, offset;unsigned long long threshold;unsigned long num_chunks; +int mremap_flags =MREMAP_MAYMOVE |MREMAP_FIXED ; + +if (c.use_relocate_anon) + mremap_flags |=MREMAP_MUST_RELOCATE_ANON ;if (threshold_mb ==VALIDATION_NO_THRESHOLD ) threshold = c.region_size;
@@ -431,10 +469,15 @@ static long long remap_region(struct config c, unsigned int threshold_mb, }if (c.dest_preamble_size) { +int mmap_flags =MAP_FIXED_NOREPLACE |MAP_ANONYMOUS ; + +if (c.use_relocate_anon) + mmap_flags |=MAP_PRIVATE ; +else + mmap_flags |=MAP_SHARED ; + dest_preamble_addr =mmap ((void *) addr - c.dest_preamble_size, c.dest_preamble_size, -PROT_READ |PROT_WRITE , -MAP_FIXED_NOREPLACE |MAP_ANONYMOUS |MAP_SHARED , --1 ,0 ); +PROT_READ |PROT_WRITE , mmap_flags,-1 ,0 );if (dest_preamble_addr ==MAP_FAILED ) {ksft_print_msg ("Failed to map dest preamble region: %s\n" ,strerror (errno));
@@ -447,8 +490,8 @@ static long long remap_region(struct config c, unsigned int threshold_mb, }clock_gettime (CLOCK_MONOTONIC , &t_start); - dest_addr =mremap (src_addr, c.region_size, c.region_size, -MREMAP_MAYMOVE |MREMAP_FIXED , (char *) addr); + dest_addr =sys_mremap (src_addr, c.region_size, c.region_size, + mremap_flags, (char *) addr);clock_gettime (CLOCK_MONOTONIC , &t_end);if (dest_addr ==MAP_FAILED ) {
@@ -549,6 +592,10 @@ static long long remap_region(struct config c, unsigned int threshold_mb,* subsequent tests. So we clean up mappings after each test. */ clean_up_dest: +/* Trigger reclaim to assert that adjusted rmap state is valid. */ +if (c.use_relocate_anon) +madvise (dest_addr, c.region_size,MADV_PAGEOUT ); +munmap (dest_addr, c.region_size); clean_up_dest_preamble:if (c.dest_preamble_size && dest_preamble_addr)
@@ -565,16 +612,19 @@ static long long remap_region(struct config c, unsigned int threshold_mb,* down address landed on a mapping that maybe does not exist. */ static void mremap_move_1mb_from_start (unsigned int pattern_seed, -char *rand_addr) +char *rand_addr,char *test_suffix, +int extra_flags) {char *test_name ="mremap move 1mb from start at 1MB+256KB aligned src" ;void *src =NULL , *dest =NULL ;unsigned int i, success =1 ; - +int mremap_flags =MREMAP_MAYMOVE |MREMAP_FIXED ;/* Config to reuse get_source_mapping() to do an aligned mmap. */ struct config c = { .src_alignment =SIZE_MB (1 ) +SIZE_KB (256 ), - .region_size =SIZE_MB (6 ) + .region_size =SIZE_MB (6 ), + .use_relocate_anon = extra_flags & (MREMAP_RELOCATE_ANON | +MREMAP_MUST_RELOCATE_ANON ), }; src =get_source_mapping (c);
@@ -583,6 +633,12 @@ static void mremap_move_1mb_from_start(unsigned int pattern_seed, goto out; } +/* See comment in mremap_move_within_range(). */ +if (extra_flags &MREMAP_MUST_RELOCATE_ANON ) +madvise (src, c.region_size,MADV_NOHUGEPAGE ); + + mremap_flags |= extra_flags; + c.src_alignment =SIZE_MB (1 ) +SIZE_KB (256 ); dest =get_source_mapping (c);if (!dest) {
@@ -599,8 +655,8 @@ static void mremap_move_1mb_from_start(unsigned int pattern_seed,*/ munmap (dest,SIZE_MB (1 )); -void *new_ptr =mremap (src +SIZE_MB (1 ),SIZE_MB (1 ),SIZE_MB (1 ), -MREMAP_MAYMOVE |MREMAP_FIXED , dest +SIZE_MB (1 )); +void *new_ptr =sys_mremap (src +SIZE_MB (1 ),SIZE_MB (1 ),SIZE_MB (1 ), + mremap_flags, dest +SIZE_MB (1 ));if (new_ptr ==MAP_FAILED ) {perror ("mremap" ); success =0 ;
@@ -629,9 +685,10 @@ static void mremap_move_1mb_from_start(unsigned int pattern_seed,perror ("munmap dest" );if (success) -ksft_test_result_pass ("%s\n" , test_name); +ksft_test_result_pass ("%s%s\n" , test_name, test_suffix); +else -ksft_test_result_fail ("%s\n" , test_name); +ksft_test_result_fail ("%s%s\n" , test_name, test_suffix); } static void run_mremap_test_case(struct test test_case, int *failures,
@@ -640,13 +697,17 @@ static void run_mremap_test_case(struct test test_case, int *failures, {long long remap_time =remap_region (test_case.config, threshold_mb, rand_addr); +char *relocate_anon_suffix =" [MREMAP_MUST_RELOCATE_ANON]" ; +struct config *c = &test_case.config;if (remap_time <0 ) {if (test_case.expect_failure) -ksft_test_result_xfail ("%s\n\tExpected mremap failure\n" , - test_case.name); +ksft_test_result_xfail ("%s%s\n\tExpected mremap failure\n" , + test_case.name, + c->use_relocate_anon ? relocate_anon_suffix :"" );else { -ksft_test_result_fail ("%s\n" , test_case.name); +ksft_test_result_fail ("%s%s\n" , test_case.name, + c->use_relocate_anon ? relocate_anon_suffix :"" ); *failures +=1 ; } }else {
@@ -656,10 +717,13 @@ static void run_mremap_test_case(struct test test_case, int *failures,*/ if (threshold_mb ==VALIDATION_NO_THRESHOLD || test_case.config.region_size <= threshold_mb * _1MB) -ksft_test_result_pass ("%s\n\tmremap time: %12lldns\n" , - test_case.name, remap_time); +ksft_test_result_pass ("%s%s\n\tmremap time: %12lldns\n" , + test_case.name, + c->use_relocate_anon ? relocate_anon_suffix :"" , + remap_time);else -ksft_test_result_pass ("%s\n" , test_case.name); +ksft_test_result_pass ("%s%s\n" , test_case.name, + c->use_relocate_anon ? relocate_anon_suffix :"" ); } }
@@ -703,8 +767,8 @@ static int parse_args(int argc, char **argv, unsigned int *threshold_mb,return 0 ; } -#define MAX_TEST 15 -#define MAX_PERF_TEST 3 +#define MAX_TEST 30 +#define MAX_PERF_TEST 6int main (int argc,char **argv) {int failures =0 ;
@@ -721,12 +785,15 @@ int main(int argc, char **argv)char *rand_addr;size_t rand_size;int num_expand_tests =2 ; -int num_misc_tests =2 ; +int num_misc_tests =6 ;struct test test_cases[MAX_TEST ] = {};struct test perf_test_cases[MAX_PERF_TEST ];int page_size;time_t t;FILE *maps_fp; +bool use_relocate_anon = false; +struct test *test_case = test_cases; +struct test *perf_test_case = perf_test_cases; pattern_seed = (unsigned int )time (&t);
@@ -763,66 +830,71 @@ int main(int argc, char **argv) page_size =sysconf (_SC_PAGESIZE); -/* Expected mremap failures */ - test_cases[0 ] =MAKE_TEST (page_size, page_size, page_size, -OVERLAPPING ,EXPECT_FAILURE , -"mremap - Source and Destination Regions Overlapping" ); - - test_cases[1 ] =MAKE_TEST (page_size, page_size/4 , page_size, -NON_OVERLAPPING ,EXPECT_FAILURE , -"mremap - Destination Address Misaligned (1KB-aligned)" ); - test_cases[2 ] =MAKE_TEST (page_size/4 , page_size, page_size, -NON_OVERLAPPING ,EXPECT_FAILURE , -"mremap - Source Address Misaligned (1KB-aligned)" ); - -/* Src addr PTE aligned */ - test_cases[3 ] =MAKE_TEST (PTE ,PTE ,PTE *2 , -NON_OVERLAPPING ,EXPECT_SUCCESS , -"8KB mremap - Source PTE-aligned, Destination PTE-aligned" ); - -/* Src addr 1MB aligned */ - test_cases[4 ] =MAKE_TEST (_1MB,PTE , _2MB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"2MB mremap - Source 1MB-aligned, Destination PTE-aligned" ); - test_cases[5 ] =MAKE_TEST (_1MB, _1MB, _2MB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"2MB mremap - Source 1MB-aligned, Destination 1MB-aligned" ); - -/* Src addr PMD aligned */ - test_cases[6 ] =MAKE_TEST (PMD ,PTE , _4MB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"4MB mremap - Source PMD-aligned, Destination PTE-aligned" ); - test_cases[7 ] =MAKE_TEST (PMD , _1MB, _4MB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"4MB mremap - Source PMD-aligned, Destination 1MB-aligned" ); - test_cases[8 ] =MAKE_TEST (PMD ,PMD , _4MB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"4MB mremap - Source PMD-aligned, Destination PMD-aligned" ); - -/* Src addr PUD aligned */ - test_cases[9 ] =MAKE_TEST (PUD ,PTE , _2GB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"2GB mremap - Source PUD-aligned, Destination PTE-aligned" ); - test_cases[10 ] =MAKE_TEST (PUD , _1MB, _2GB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"2GB mremap - Source PUD-aligned, Destination 1MB-aligned" ); - test_cases[11 ] =MAKE_TEST (PUD ,PMD , _2GB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"2GB mremap - Source PUD-aligned, Destination PMD-aligned" ); - test_cases[12 ] =MAKE_TEST (PUD ,PUD , _2GB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"2GB mremap - Source PUD-aligned, Destination PUD-aligned" ); - -/* Src and Dest addr 1MB aligned. 5MB mremap. */ - test_cases[13 ] =MAKE_TEST (_1MB, _1MB, _5MB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"5MB mremap - Source 1MB-aligned, Destination 1MB-aligned" ); - -/* Src and Dest addr 1MB aligned. 5MB mremap. */ - test_cases[14 ] =MAKE_TEST (_1MB, _1MB, _5MB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"5MB mremap - Source 1MB-aligned, Dest 1MB-aligned with 40MB Preamble" ); - test_cases[14 ].config.dest_preamble_size =10 * _4MB; - - perf_test_cases[0 ] =MAKE_TEST (page_size, page_size, _1GB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"1GB mremap - Source PTE-aligned, Destination PTE-aligned" ); -/* -* mremap 1GB region - Page table level aligned time -* comparison. -*/ - perf_test_cases[1 ] =MAKE_TEST (PMD ,PMD , _1GB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"1GB mremap - Source PMD-aligned, Destination PMD-aligned" ); - perf_test_cases[2 ] =MAKE_TEST (PUD ,PUD , _1GB,NON_OVERLAPPING ,EXPECT_SUCCESS , -"1GB mremap - Source PUD-aligned, Destination PUD-aligned" ); +do { +/* Expected mremap failures */ + *test_case++ =MAKE_TEST (page_size, page_size, page_size, +OVERLAPPING , use_relocate_anon,EXPECT_FAILURE , +"mremap - Source and Destination Regions Overlapping" ); + + *test_case++ =MAKE_TEST (page_size, page_size/4 , page_size, +NON_OVERLAPPING , use_relocate_anon,EXPECT_FAILURE , +"mremap - Destination Address Misaligned (1KB-aligned)" ); + *test_case++ =MAKE_TEST (page_size/4 , page_size, page_size, +NON_OVERLAPPING , use_relocate_anon,EXPECT_FAILURE , +"mremap - Source Address Misaligned (1KB-aligned)" ); + +/* Src addr PTE aligned */ + *test_case++ =MAKE_TEST (PTE ,PTE ,PTE *2 , +NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"8KB mremap - Source PTE-aligned, Destination PTE-aligned" ); + +/* Src addr 1MB aligned */ + *test_case++ =MAKE_TEST (_1MB,PTE , _2MB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"2MB mremap - Source 1MB-aligned, Destination PTE-aligned" ); + *test_case++ =MAKE_TEST (_1MB, _1MB, _2MB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"2MB mremap - Source 1MB-aligned, Destination 1MB-aligned" ); + +/* Src addr PMD aligned */ + *test_case++ =MAKE_TEST (PMD ,PTE , _4MB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"4MB mremap - Source PMD-aligned, Destination PTE-aligned" ); + *test_case++ =MAKE_TEST (PMD , _1MB, _4MB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"4MB mremap - Source PMD-aligned, Destination 1MB-aligned" ); + *test_case++ =MAKE_TEST (PMD ,PMD , _4MB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"4MB mremap - Source PMD-aligned, Destination PMD-aligned" ); + +/* Src addr PUD aligned */ + *test_case++ =MAKE_TEST (PUD ,PTE , _2GB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"2GB mremap - Source PUD-aligned, Destination PTE-aligned" ); + *test_case++ =MAKE_TEST (PUD , _1MB, _2GB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"2GB mremap - Source PUD-aligned, Destination 1MB-aligned" ); + *test_case++ =MAKE_TEST (PUD ,PMD , _2GB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"2GB mremap - Source PUD-aligned, Destination PMD-aligned" ); + *test_case++ =MAKE_TEST (PUD ,PUD , _2GB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"2GB mremap - Source PUD-aligned, Destination PUD-aligned" ); + +/* Src and Dest addr 1MB aligned. 5MB mremap. */ + *test_case++ =MAKE_TEST (_1MB, _1MB, _5MB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"5MB mremap - Source 1MB-aligned, Destination 1MB-aligned" ); + +/* Src and Dest addr 1MB aligned. 5MB mremap. */ + *test_case =MAKE_TEST (_1MB, _1MB, _5MB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"5MB mremap - Source 1MB-aligned, Dest 1MB-aligned with 40MB Preamble" ); + test_case++->config.dest_preamble_size =10 * _4MB; + + *perf_test_case++ =MAKE_TEST (page_size, page_size, _1GB,NON_OVERLAPPING , + use_relocate_anon,EXPECT_SUCCESS , +"1GB mremap - Source PTE-aligned, Destination PTE-aligned" ); +/* +* mremap 1GB region - Page table level aligned time +* comparison. +*/ + *perf_test_case++ =MAKE_TEST (PMD ,PMD , _1GB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"1GB mremap - Source PMD-aligned, Destination PMD-aligned" ); + *perf_test_case++ =MAKE_TEST (PUD ,PUD , _1GB,NON_OVERLAPPING , use_relocate_anon,EXPECT_SUCCESS , +"1GB mremap - Source PUD-aligned, Destination PUD-aligned" ); + + use_relocate_anon = !use_relocate_anon; + }while (use_relocate_anon); run_perf_tests = (threshold_mb ==VALIDATION_NO_THRESHOLD ) || (threshold_mb * _1MB >= _1GB);
@@ -846,8 +918,18 @@ int main(int argc, char **argv)fclose (maps_fp); -mremap_move_within_range (pattern_seed, rand_addr); -mremap_move_1mb_from_start (pattern_seed, rand_addr); +mremap_move_within_range (pattern_seed, rand_addr, +"" ,0 ); +mremap_move_within_range (pattern_seed, rand_addr, +"[MREMAP_RELOCATE_ANON]" ,MREMAP_RELOCATE_ANON ); +mremap_move_within_range (pattern_seed, rand_addr, +"[MREMAP_MUST_RELOCATE_ANON]" ,MREMAP_MUST_RELOCATE_ANON ); +mremap_move_1mb_from_start (pattern_seed, rand_addr, +"" ,0 ); +mremap_move_1mb_from_start (pattern_seed, rand_addr, +"[MREMAP_RELOCATE_ANON]" ,MREMAP_RELOCATE_ANON ); +mremap_move_1mb_from_start (pattern_seed, rand_addr, +"[MREMAP_MUST_RELOCATE_ANON]" ,MREMAP_MUST_RELOCATE_ANON );if (run_perf_tests) { ksft_print_msg("\n%s\n",
Signature · 1 line
2.48.1