mm/rmap: index MAP_PRIVATE file-backed folios by virt pgoff
Subsystems (12)
Lorenzo Stoakes
maintainercover mm/rmap: index MAP_PRIVATE file-backed folios by virt pgoff
In memory management we've managed to manufacture a great deal of confusion around the concept of anonymous memory. We have:
1. 'Pure anon' memory - anonymous VMAs whose folios are anonymous and swap-backed (thus for reclaim purposes, treated as anonymous). These are simple enough.
2. shmem - file-backed VMAs, file-backed folios (from rmap perspective) so present in the page cache and mapped by an address_space object, but whose folios are also swap-backed (thus treated as anonymous for reclaim purposes).
3. MAP_PRIVATE-mapped /dev/zero - a strange beast whose VMAs have vma->vm_file set, but whose mmap_prepare callback clears vma->vm_ops to satisfy vma_is_anonymous(), which results in VMAs that were mmap()'d referencing a file and whose VMAs reference a file, but are in every other sense anonymous, including the folios.
4. Other MAP_PRIVATE-file backed mappings - These possess file-backed VMAs and have file-backed folios until CoW'd, at which point those CoW'd folios are anonymous.
This series focuses upon issue 4.
In order for us to traverse VMAs using the reverse mapping, we require two fields - folio->mapping and folio->index. The first tells the rmap code where to look for VMAs, and the second tells it at which offset the folio starts within the referenced object.
For anonymous folios, folio->mapping points at an anon_vma object. For file-backed folios, it points at an address_space. And:
* For file-backed folios folio->index is simply the page offset of the start of the folio within the file.
* For anonymous folios belonging to pure anon mappings, folio->index is equal to the virtual page offset of the folio.
* For anonymous folios belonging to file-backed mappings (i.e. CoW'd folios of a MAP_PRIVATE file-backed mapping), folio->index is equal to the file page offset.
This series establishes a new virtual page offset property of VMAs to allow us to map anonymous folios at their virtual page offset, consistent with pure anon.
The purpose of doing so is to lay the foundations for the scalable CoW work. This is necessary because firstly the approach first looks in the maple tree for the VMA located at folio->index << PAGE_SHIFT, before falling back to looking up tracked remaps if necessary.
The MAP_PRIVATE file-backed case means that folio indices will very often conflict with one another and this remap tracking becomes substantially more contended, and of course the fast path can never be used.
This also makes it possible, in future, to unshare anonymously mapped folios with deep fork hierarchies on remap, eliminating the need for remap tracking in the vast majority of cases.
Simlar to page offset of pure anonymous VMAs, we update the virtual page offset of unfaulted file-backed VMAs on remap, but do not once CoW'd (i.e. vma->anon_vma is non-NULL).
Overall, there is little impact on mergeability - for shared file-backed mappings, we treat the anonymous page offset as equal to the file-backed one (via vma_start_anon_pgoff()), so merge behaviour remains the same there.
The only impact is on MAP_PRIVATE-mapped file-backed mappings, which must now match on virtual page offset as well as file page offset to be merged.
To fail to merge like this would require CoW'ing the mapping then remapping in such a way as to eliminate compatibility, but this is very much an edge case that should very little impact (and which scalable CoW may very well address in any case).
REVIEWER/MERGER NOTE: This is dependent on [0], which must be merged before this.
[0]:https://lore.kernel.org/linux-mm/[email protected]/
Lorenzo Stoakes (10):
mm/vma: introduce VMA virtual page offset field and add helpers
mm: introduce linear_virt_page_index()
mm: abstract vma_address() and introduce vma_anon_address()
mm: update print_bad_page_map() to show virtual page index
mm: introduce and use vma_filebacked_address()
mm: propagate VMA virtual page offset on map, remap, split + merge mm/rmap: track whether the page VMA mapped walk is anonymous
mm: introduce and use linear_folio_page_index() mm/rmap: use virt pgoff for MAP_PRIVATE file-backed anon folios tools/testing/vma: expand VMA merge tests to assert virt pgoff
include/linux/mm.h | 61 ++++++++++++- include/linux/mm_types.h | 4 + include/linux/pagemap.h | 66 +++++++++++++++ include/linux/rmap.h | 2 + mm/huge_memory.c | 3 +- mm/internal.h | 69 +++++++++++---- mm/interval_tree.c | 4 +- mm/ksm.c | 2 +- mm/memory-failure.c | 4 +- mm/memory.c | 8 +- mm/migrate.c | 6 +- mm/mremap.c | 6 +- mm/page_vma_mapped.c | 6 +- mm/rmap.c | 23 +++-- mm/userfaultfd.c | 6 +- mm/vma.c | 70 +++++++++++---- mm/vma.h | 141 ++++++++++++++++++++++++------- mm/vma_exec.c | 2 +- mm/vma_init.c | 1 + tools/testing/vma/include/dup.h | 48 +++++++++++ tools/testing/vma/shared.c | 3 +- tools/testing/vma/tests/merge.c | 22 ++++- tools/testing/vma/tests/vma.c | 4 +- tools/testing/vma/vma_internal.h | 1 + 24 files changed, 468 insertions(+), 94 deletions(-)
--
2.54.0
Lorenzo Stoakes
maintainerRFC PATCH 1/10 mm/vma: introduce VMA virtual page offset field and add helpers
This patch establishes fields within the vm_area_struct type to store the virtual page offset of VMAs.
The virtual page offset of a VMA is equal to vma->vm_start >> PAGE_SHIFT if they are unfaulted or were not remapped, otherwise it is equal to this value at the point of first fault.
Currently, anonymous folios belonging to CoW'd MAP_PRIVATE-mapped file-backed VMAs are tracked by their file offset. By adding virtual offset as a property of VMAs, we can now track them by their virtual page offset instead.
By tracking this, we provide the means by which to eliminate this inconsistency, and more importantly lay the foundations for future work for the scalable CoW anonymous rmap rework.
This patch simply adds the fields and some simple helpers. Subsequent patches will update mm code to make use of these fields correctly.
The fields chosen are packed in the VMA such that, for 64-bit kernel builds, no additional space is taken up.
The first field is present on cacheline 0 containing key VMA fields, and the second on cacheline 3, which contains file-backed reverse mapping fields.
Given the relative time spent accessing reverse mapping fields as well as updating them, there shouldn't be any performance impact here from false sharing.
Update the VMA userland tests to account for this change.
No callsites are updated yet, so no functional change intended.
--- include/linux/mm.h | 59 +++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 4 +++ mm/vma.h | 14 ++++++++ mm/vma_init.c | 1 + tools/testing/vma/include/dup.h | 26 +++++++++++++++ 5 files changed, 104 insertions(+)
include/linux/mm.h +59 -0
@@ -4335,6 +4335,65 @@ static inline pgoff_t vma_last_pgoff(const struct vm_area_struct *vma)return vma_end_pgoff (vma) -1 ; } +/** +* vma_start_virt_pgoff() - Get the virtual page offset of the start of @vma +* @vma: The VMA whose virtual page offset is required. +* +* If unfaulted, then this is vma->vm_start >> PAGE_SHIFT, if faulted then the +* virtual page offset at the time of first fault. +* +* If the VMA is anonymous, this returns the same value as vma_start_pgoff(). +* +* This value is used for tracking MAP_PRIVATE file-backed mappings by their +* virtual page offset. +* +* Returns: The virtual page offset of the start of @vma. +*/ +static inline pgoff_t vma_start_virt_pgoff (const struct vm_area_struct *vma) +{ +pgoff_t pgoff =0 ; + +#ifdef CONFIG_64BIT + pgoff += vma->__vm_virt_pgoff_hi; + pgoff <<=32 ; +#endif + pgoff += vma->__vm_virt_pgoff_lo; +return pgoff; +} + +/** +* vma_end_virt_pgoff() - Get the virtual page offset of the exclusive end of +* @vma. +* @vma: The VMA whose end virtual page offset is required. +* +* This returns the virtual exclusive end page offset of @vma, which is useful +* for expressing page offset ranges. +* +* See the description of vma_start_virt_pgoff() for a description of VMA +* virtual page offsets. +* +* Returns: The exclusive end virtual page offset of @vma. +*/ +static inline pgoff_t vma_end_virt_pgoff (const struct vm_area_struct *vma) +{ +return vma_start_virt_pgoff (vma) +vma_pages (vma); +} + +/** +* vma_last_virt_pgoff() - Get the virtual page offset of the last page in +* @vma. +* @vma: The VMA whose last virtual page offset is required. +* +* See the description of vma_start_virt_pgoff() for a description of VMA +* virtual page offsets. +* +* Returns: The last virtual page offset of @vma. +*/ +static inline pgoff_t vma_last_virt_pgoff (const struct vm_area_struct *vma) +{ +return vma_end_virt_pgoff (vma) -1 ; +} +static inline unsigned long vma_desc_size (const struct vm_area_desc *desc) {return desc->end - desc->start;
include/linux/mm_types.h +4 -0
@@ -964,6 +964,7 @@ struct vm_area_struct {*/ unsigned int vm_lock_seq;#endif +unsigned int __vm_virt_pgoff_lo;/* Low 32-bits of virtual pgoff. */ /* * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma * list, after a COW of one of the file pages. A MAP_SHARED vma
@@ -1038,6 +1039,9 @@ struct vm_area_struct {#ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map vmlock_dep_map;#endif +#endif +#ifdef CONFIG_64BIT +unsigned int __vm_virt_pgoff_hi;/* High 32-bits of virtual pgoff. */ #endif /* * For areas with an address space and backing store,
mm/vma.h +14 -0
@@ -263,6 +263,20 @@ static inline void vma_set_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) vma->vm_pgoff = pgoff; } +static inline void __vma_set_virt_pgoff (struct vm_area_struct *vma,pgoff_t pgoff) +{ +#ifdef CONFIG_64BIT + vma->__vm_virt_pgoff_hi = pgoff >>32 ; +#endif + vma->__vm_virt_pgoff_lo = pgoff &GENMASK (31 ,0 ); +} + +static inline void vma_set_virt_pgoff (struct vm_area_struct *vma,pgoff_t pgoff) +{ +vma_assert_can_modify (vma); +__vma_set_virt_pgoff (vma, pgoff); +} +static inline void vma_add_pgoff (struct vm_area_struct *vma,pgoff_t delta) {vma_assert_can_modify (vma);
mm/vma_init.c +1 -0
@@ -51,6 +51,7 @@ static void vm_area_init_from(const struct vm_area_struct *src, dest->vm_end = src->vm_end; dest->anon_vma = src->anon_vma; dest->vm_pgoff =vma_start_pgoff (src); +__vma_set_virt_pgoff (dest,vma_start_virt_pgoff (src)); dest->vm_file = src->vm_file; dest->vm_private_data = src->vm_private_data;vm_flags_init (dest, src->vm_flags);
tools/testing/vma/include/dup.h +26 -0
@@ -573,6 +573,7 @@ struct vm_area_struct {*/ unsigned int vm_lock_seq;#endif +unsigned int __vm_virt_pgoff_lo;/* * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma
@@ -608,6 +609,9 @@ struct vm_area_struct {#ifdef CONFIG_PER_VMA_LOCK /* Unstable RCU readers are allowed to read this. */ refcount_t vm_refcnt; +#endif +#ifdef CONFIG_64BIT +unsigned int __vm_virt_pgoff_hi;#endif /* * For areas with an address space and backing store,
@@ -1322,6 +1326,28 @@ static inline pgoff_t vma_end_pgoff(const struct vm_area_struct *vma)return vma_start_pgoff (vma) +vma_pages (vma); } +static inline pgoff_t vma_start_virt_pgoff (const struct vm_area_struct *vma) +{ +pgoff_t pgoff =0 ; + +#ifdef CONFIG_64BIT + pgoff += vma->__vm_virt_pgoff_hi; + pgoff <<=32 ; +#endif + pgoff += vma->__vm_virt_pgoff_lo; +return pgoff; +} + +static inline pgoff_t vma_end_virt_pgoff (const struct vm_area_struct *vma) +{ +return vma_start_virt_pgoff (vma) +vma_pages (vma); +} + +static inline pgoff_t vma_last_virt_pgoff (const struct vm_area_struct *vma) +{ +return vma_end_virt_pgoff (vma) -1 ; +} +static inline int vfs_mmap_prepare (struct file *file,struct vm_area_desc *desc) {return file->f_op->mmap_prepare (desc);
Signature · 1 line
2.54.0
Lorenzo Stoakes
maintainerRFC PATCH 2/10 mm: introduce linear_virt_page_index()
In reply to Lorenzo StoakesThis function provides the virtual equivalent of linear_page_index(), instead offsetting based on the virtual page offset of the VMA.
It is valid only for anonymous or MAP_PRIVATE file-backed mappings. It must not be called for shared file-backed mappings.
For pure anon VMAs, this will be equal to linear_page_index().
We implement the algorithm in __linear_virt_page_index(), which is provided for internal mm code that might be interacting with shared VMAs.
In linear_virt_page_index() we assert that both of these invariants are true.
Note that MAP_PRIVATE-/dev/zero mappings will satisfy vma_is_anonymous() but not fulfill this invariant, so when asserting this we check vma->vm_file to account for this.
We do not update callsites yet, so no functional change intended.
Also const-ify vma_is_anonymous() to make it compatible with the const-ified linear_virt_page_index().
VMA userland tests are updated accordingly.
--- include/linux/mm.h | 2 +- include/linux/pagemap.h | 41 +++++++++++++++++++++++++++++++++ tools/testing/vma/include/dup.h | 22 ++++++++++++++++++ 3 files changed, 64 insertions(+), 1 deletion(-)
include/linux/mm.h +1 -1
@@ -1533,7 +1533,7 @@ static inline void vma_desc_set_anonymous(struct vm_area_desc *desc) desc->vm_ops =NULL ; } -static inline bool vma_is_anonymous (struct vm_area_struct *vma) +static inline bool vma_is_anonymous (const struct vm_area_struct *vma) {return !vma->vm_ops; }
include/linux/pagemap.h +41 -0
47 lines not on this page · raw message
tools/testing/vma/include/dup.h +22 -0
@@ -1601,3 +1601,25 @@ static inline pgoff_t linear_page_index(const struct vm_area_struct *vma, pgoff +=vma_start_pgoff (vma);return pgoff; } + +static inline pgoff_t __linear_virt_page_index (const struct vm_area_struct *vma, +const unsigned long address) +{ +pgoff_t pgoff; + + pgoff =linear_page_delta (vma, address); + pgoff +=vma_start_virt_pgoff (vma); +return pgoff; +} + +static inline pgoff_t linear_virt_page_index (const struct vm_area_struct *vma, +const unsigned long address) +{ +const pgoff_t pgoff =__linear_virt_page_index (vma, address); + +VM_WARN_ON_ONCE (vma_test (vma,VMA_SHARED_BIT )); +if (!vma->vm_file)/* Is anonymous except MAP_PRIVATE-/dev/zero */ +VM_WARN_ON_ONCE (pgoff !=linear_page_index (vma, address)); + +return pgoff; +}
Signature · 1 line
2.54.0
Lorenzo Stoakes
maintainerRFC PATCH 3/10 mm: abstract vma_address() and introduce vma_anon_address()
In reply to Lorenzo StoakesIntroduce __vma_address() which abstracts the VMA start page offset field as pgoff_start, then update vma_address() to use it.
Then introduce vma_anon_address() which does the equivalent of vma_address(), only using the virtual page offset of the VMA rather than the file-backed one.
Also add an assert to ensure that the function is not called for mappings which are file-backed but not MAP_PRIVATE to ensure it is only used in the correct places.
This will be necessary for determining the address of a folio's index within a VMA when the folio belongs to a MAP_PRIVATE file-backed VMA but has been CoW'd, and thus is anonymous, once the anonymous VMA page offset field is used for the reverse mapping.
No callers are updated, so no functional change intended.
--- mm/internal.h | 49 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 37 insertions(+), 12 deletions(-)
mm/internal.h +37 -12
@@ -1224,19 +1224,9 @@ void mlock_drain_remote(int cpu);extern pmd_t maybe_pmd_mkwrite (pmd_t pmd,struct vm_area_struct *vma); -/** -* vma_address - Find the virtual address a page range is mapped at -* @vma: The vma which maps this object. -* @pgoff: The page offset within its object. -* @nr_pages: The number of pages to consider. -* -* If any page in this range is mapped by this VMA, return the first address -* where any of these pages appear. Otherwise, return -EFAULT. -*/ -static inline unsigned long vma_address (const struct vm_area_struct *vma, -pgoff_t pgoff,unsigned long nr_pages) +static inline unsigned long __vma_address (const struct vm_area_struct *vma, +pgoff_t pgoff,pgoff_t pgoff_start,unsigned long nr_pages) { -const pgoff_t pgoff_start =vma_start_pgoff (vma);unsigned long address;if (pgoff >= pgoff_start) {
@@ -1254,6 +1244,41 @@ static inline unsigned long vma_address(const struct vm_area_struct *vma,return address; } +/** +* vma_address - Find the virtual address a page range is mapped at. +* @vma: The vma which maps this object. +* @pgoff: The page offset within its object. +* @nr_pages: The number of pages to consider. +* +* If any page in this range is mapped by this VMA, return the first address +* where any of these pages appear. Otherwise, return -EFAULT. +*/ +static inline unsigned long vma_address (const struct vm_area_struct *vma, +pgoff_t pgoff,unsigned long nr_pages) +{ +return __vma_address (vma, pgoff,vma_start_pgoff (vma), nr_pages); +} + +/** +* vma_anon_address - Find the address an anonymous folio with index @pgoff_virt +* is mapped at. +* @vma: The vma which maps this object. +* @pgoff_virt: The virtual page index belonging to the folio. +* @nr_pages: The number of pages to consider. +* +* This is only valid for anonymous or MAP_PRIVATE-mapped file-backed VMAs. +* +* Returns: If any page in this range is mapped by this VMA, return the first address +* where any of these pages appear. Otherwise, return -EFAULT. +*/ +static inline unsigned long vma_anon_address (const struct vm_area_struct *vma, +pgoff_t pgoff_virt,unsigned long nr_pages) +{ +VM_WARN_ON_ONCE (!vma_is_anonymous (vma) &&vma_test (vma,VMA_SHARED_BIT )); + +return __vma_address (vma, pgoff_virt,vma_start_virt_pgoff (vma), nr_pages); +} +/* * Then at what user virtual address will none of the range be found in vma? * Assumes that vma_address() already returned a good starting address.
Signature · 1 line
2.54.0
Lorenzo Stoakes
maintainerRFC PATCH 4/10 mm: update print_bad_page_map() to show virtual page index
In reply to Lorenzo StoakesThis is potentially useful debugging information and matches the existing page offset provided.
We only do so if the VMA is not a shared mapping, as shared mappings ignore the virtual page offset.
We use __linear_virt_page_offset() as the VMA may be shared.
--- mm/memory.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
mm/memory.c +5 -3
@@ -588,21 +588,23 @@ static void print_bad_page_map(struct vm_area_struct *vma, enum pgtable_level level) {struct address_space *mapping; -pgoff_t index; +pgoff_t index, virt_index;if (is_bad_page_map_ratelimited ())return ; mapping = vma->vm_file ? vma->vm_file->f_mapping :NULL ; index =linear_page_index (vma, addr); + virt_index =__linear_virt_page_index (vma, addr);pr_alert ("BUG: Bad page map in process %s %s:%08llx" , current->comm,pgtable_level_to_str (level), entry);__print_bad_page_map_pgtable (vma->vm_mm, addr);if (page)dump_page (page,"bad page map" ); -pr_alert ("addr:%px vm_flags:%08lx anon_vma:%px mapping:%px index:%lx\n" , - (void *)addr, vma->vm_flags, vma->anon_vma, mapping, index); +pr_alert ("addr:%px vm_flags:%08lx anon_vma:%px mapping:%px index:%lx virt_index:%lx\n" , + (void *)addr, vma->vm_flags, vma->anon_vma, mapping, index, + virt_index); pr_alert("file:%pD fault:%ps mmap:%ps mmap_prepare: %ps read_folio:%ps\n", vma->vm_file, vma->vm_ops ? vma->vm_ops->fault : NULL,
Signature · 1 line
2.54.0
Lorenzo Stoakes
maintainerRFC PATCH 5/10 mm: introduce and use vma_filebacked_address()
In reply to Lorenzo StoakesIn cases where we know that the VMA is file-backed, use vma_filebacked_address() rather than vma_address().
This lays the foundation for using the virtual page offset for anonymous VMAs via vma_anon_address().
Also add an assert to ensure that the VMA whose address is required is not anonymous.
No functional change intended.
--- mm/internal.h | 18 ++++++++++++++++++ mm/memory-failure.c | 4 ++-- mm/page_vma_mapped.c | 6 +++++- mm/rmap.c | 10 ++++++---- 4 files changed, 31 insertions(+), 7 deletions(-)
mm/internal.h +18 -0
@@ -1244,6 +1244,24 @@ static inline unsigned long __vma_address(const struct vm_area_struct *vma,return address; } +/** +* vma_filebacked_address - Find the virtual address a file-backed page range is +* mapped at. +* @vma: The vma which maps this object. +* @pgoff: The page offset within its object. +* @nr_pages: The number of pages to consider. +* +* Returns: If any page in this range is mapped by this VMA, return the first +* address where any of these pages appear. Otherwise, return -EFAULT. +*/ +static inline unsigned long vma_filebacked_address (const struct vm_area_struct *vma, +pgoff_t pgoff,unsigned long nr_pages) +{ +VM_WARN_ON_ONCE (vma_is_anonymous (vma)); + +return __vma_address (vma, pgoff,vma_start_pgoff (vma), nr_pages); +} +/** * vma_address - Find the virtual address a page range is mapped at. * @vma: The vma which maps this object.
mm/memory-failure.c +2 -2
@@ -608,7 +608,7 @@ static void add_to_kill_fsdax(struct task_struct *tsk, const struct page *p,struct vm_area_struct *vma, struct list_head *to_kill, pgoff_t pgoff) { - unsigned long addr = vma_address(vma, pgoff, 1); + unsigned long addr = vma_filebacked_address(vma, pgoff, 1);__add_to_kill (tsk, p, vma, to_kill, addr); }
@@ -2207,7 +2207,7 @@ static void add_to_kill_pgoff(struct task_struct *tsk, }/* Check for pgoff not backed by struct page */ - tk->addr =vma_address (vma, pgoff,1 ); + tk->addr =vma_filebacked_address (vma, pgoff,1 ); tk->size_shift =PAGE_SHIFT ;if (tk->addr == -EFAULT )
mm/page_vma_mapped.c +5 -1
@@ -347,6 +347,7 @@ unsigned long page_mapped_in_vma(const struct page *page, struct vm_area_struct *vma) {const struct folio *folio =page_folio (page); +const pgoff_t pgoff =page_pgoff (folio, page);struct page_vma_mapped_walk pvmw = { .pfn =page_to_pfn (page), .nr_pages =1 ,
@@ -354,7 +355,10 @@ unsigned long page_mapped_in_vma(const struct page *page, .flags = PVMW_SYNC, }; - pvmw.address =vma_address (vma,page_pgoff (folio, page),1 ); +if (folio_test_anon (folio)) + pvmw.address =vma_address (vma, pgoff,1 ); +else + pvmw.address =vma_filebacked_address (vma, pgoff,1 );if (pvmw.address == -EFAULT ) goto out;if (!page_vma_mapped_walk (&pvmw))
mm/rmap.c +6 -4
@@ -865,14 +865,15 @@ unsigned long page_address_in_vma(const struct folio *folio,if (!vma->anon_vma || !anon_vma || vma->anon_vma->root != anon_vma->root)return -EFAULT ; +/* KSM folios don't reach here because of the !anon_vma check */ +return vma_address (vma,page_pgoff (folio, page),1 ); }else if (!vma->vm_file) {return -EFAULT ; }else if (vma->vm_file->f_mapping != folio->mapping) {return -EFAULT ; } -/* KSM folios don't reach here because of the !anon_vma check */ -return vma_address (vma,page_pgoff (folio, page),1 ); +return vma_filebacked_address (vma,page_pgoff (folio, page),1 ); }/*
@@ -1321,7 +1322,7 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff,if (invalid_mkclean_vma (vma,NULL ))return 0 ; - pvmw.address =vma_address (vma, pgoff, nr_pages); + pvmw.address =vma_filebacked_address (vma, pgoff, nr_pages);VM_BUG_ON_VMA (pvmw.address == -EFAULT , vma);return page_vma_mkclean_one (&pvmw);
@@ -3051,7 +3052,8 @@ static void __rmap_walk_file(struct folio *folio, struct address_space *mapping, } lookup: mapping_interval_tree_foreach(vma, mapping, pgoff_start, pgoff_end) { -unsigned long address =vma_address (vma, pgoff_start, nr_pages); +unsigned long address =vma_filebacked_address (vma, pgoff_start, + nr_pages);VM_BUG_ON_VMA (address == -EFAULT , vma);cond_resched ();
Signature · 1 line
2.54.0
Lorenzo Stoakes
maintainerRFC PATCH 6/10 mm: propagate VMA virtual page offset on map, remap, split + merge
In reply to Lorenzo StoakesWe must correctly update VMA virtual page offset state on all VMA operations that would result in it changing, with special attention given to remapping.
We cover most cases by simply updating vma_set_range() to do so (with a new virtual page offset parameter), but also notably must update the merging and mapping logic to propagate this parameter correctly.
The remap logic remains the same - we may update the virtual page offset if the VMA is unfaulted, but now this applies to MAP_PRIVATE file-backed mappings too, so we update the code to reflect this.
Note that we use __linear_virt_page_index() upon remap as the VMA may be shared, in order that we update the field consistently regardless of VMA type.
Also while we're here, replace a VMA_BUG_ON_VMA() with a VMA_WARN_ON_ONCE_VMA().
We also introduce vma_anon_pgoff_addr(), vma_start_anon_pgoff(), and vma_end_anon_pgoff() which differ from their virtual page offset equivalents in that a shared file-backed mapping returns its file page offset otherwise the virtual page offset is used.
This means we don't predicate merges for shared file-backed mappings on virtual page offset.
We simply ensure state is correctly propagated here, so no functional changes are intended.
Finally, we update insert_vm_struct() to correctly set the virtual page offset on insertion of a VMA.
Also update VMA userland tests to reflect this change.
--- mm/mremap.c | 6 +- mm/vma.c | 56 ++++++++++---- mm/vma.h | 127 ++++++++++++++++++++++++------- mm/vma_exec.c | 2 +- tools/testing/vma/shared.c | 3 +- tools/testing/vma/tests/merge.c | 4 +- tools/testing/vma/tests/vma.c | 4 +- tools/testing/vma/vma_internal.h | 1 + 8 files changed, 152 insertions(+), 51 deletions(-)
mm/mremap.c +4 -2
@@ -1254,7 +1254,9 @@ static void unmap_source_vma(struct vma_remap_struct *vrm)static int copy_vma_and_data (struct vma_remap_struct *vrm,struct vm_area_struct **new_vma_ptr) { -const unsigned long new_pgoff =linear_page_index (vrm->vma, vrm->addr); +const pgoff_t new_pgoff =linear_page_index (vrm->vma, vrm->addr); +const pgoff_t new_virt_pgoff = +__linear_virt_page_index (vrm->vma, vrm->addr);struct vm_area_struct *vma = vrm->vma;struct vm_area_struct *new_vma;unsigned long moved_len;
@@ -1262,7 +1264,7 @@ static int copy_vma_and_data(struct vma_remap_struct *vrm,PAGETABLE_MOVE (pmc,NULL ,NULL , vrm->addr, vrm->new_addr, vrm->old_len); new_vma =copy_vma (&vma, vrm->new_addr, vrm->new_len, new_pgoff, - &pmc.need_rmap_locks); + new_virt_pgoff, &pmc.need_rmap_locks);if (!new_vma) {vrm_uncharge (vrm); *new_vma_ptr =NULL ;
mm/vma.c +41 -15
200 lines not on this page · raw message
mm/vma.h +98 -29
178 lines not on this page · raw message
mm/vma_exec.c +1 -1
@@ -41,7 +41,7 @@ int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift)unsigned long new_end = old_end - shift;VMA_ITERATOR (vmi, mm, new_start);VMG_STATE (vmg, mm, &vmi, new_start, old_end,EMPTY_VMA_FLAGS , -vma_start_pgoff (vma)); +vma_start_pgoff (vma),vma_start_anon_pgoff (vma));struct vm_area_struct *next;struct mmu_gather tlb;PAGETABLE_MOVE (pmc, vma, vma, old_start, new_start, length);
tools/testing/vma/shared.c +2 -1
@@ -23,7 +23,8 @@ struct vm_area_struct *alloc_vma(struct mm_struct *mm, vma->vm_start = start; vma->vm_end = end; - vma->vm_pgoff = pgoff; +vma_set_pgoff (vma, pgoff); +vma_set_virt_pgoff (vma, start >>PAGE_SHIFT ); vma->flags = vma_flags;vma_assert_detached (vma);
tools/testing/vma/tests/merge.c +3 -1
@@ -45,6 +45,7 @@ void vmg_set_range(struct vma_merge_struct *vmg, unsigned long start, vmg->start = start; vmg->end = end; vmg->pgoff = pgoff; + vmg->anon_pgoff = start >>PAGE_SHIFT ; vmg->vma_flags = vma_flags; vmg->just_expand = false;
@@ -108,6 +109,7 @@ static bool test_simple_merge(void) .end = 0x2000, .vma_flags = vma_flags, .pgoff = 1, + .anon_pgoff = 1, };ASSERT_FALSE (attach_vma (&mm, vma_left));
@@ -1431,7 +1433,7 @@ static bool test_expand_only_mode(void)struct mm_struct mm = {};VMA_ITERATOR (vmi, &mm,0 );struct vm_area_struct *vma_prev, *vma; -VMG_STATE (vmg, &mm, &vmi,0x5000 ,0x9000 , vma_flags,5 ); +VMG_STATE (vmg, &mm, &vmi,0x5000 ,0x9000 , vma_flags,5 ,5 );/* * Place a VMA prior to the one we're expanding so we assert that we do
tools/testing/vma/tests/vma.c +2 -2
@@ -38,7 +38,7 @@ static bool test_copy_vma(void)/* Move backwards and do not merge. */ vma =alloc_and_link_vma (&mm,0x3000 ,0x5000 ,3 , vma_flags); - vma_new =copy_vma (&vma,0 ,0x2000 ,0 , &need_locks); + vma_new =copy_vma (&vma,0 ,0x2000 ,0 ,3 , &need_locks);ASSERT_NE (vma_new, vma);ASSERT_EQ (vma_new->vm_start,0 );ASSERT_EQ (vma_new->vm_end,0x2000 );
@@ -51,7 +51,7 @@ static bool test_copy_vma(void) vma =alloc_and_link_vma (&mm,0 ,0x2000 ,0 , vma_flags); vma_next =alloc_and_link_vma (&mm,0x6000 ,0x8000 ,6 , vma_flags); - vma_new =copy_vma (&vma,0x4000 ,0x2000 ,4 , &need_locks); + vma_new =copy_vma (&vma,0x4000 ,0x2000 ,4 ,4 , &need_locks);vma_assert_attached (vma_new);ASSERT_EQ (vma_new, vma_next);
tools/testing/vma/vma_internal.h +1 -0
@@ -53,6 +53,7 @@ typedef __bitwise unsigned int vm_fault_t;#define VM_WARN_ON (_expr) (WARN_ON(_expr))#define VM_WARN_ON_ONCE (_expr) (WARN_ON_ONCE(_expr)) +#define VM_WARN_ON_ONCE_VMA (_expr, _vma) (WARN_ON_ONCE(_expr))#define VM_WARN_ON_VMG (_expr, _vmg) (WARN_ON(_expr))#define VM_BUG_ON (_expr) (BUG_ON(_expr))#define VM_BUG_ON_VMA (_expr, _vma) (BUG_ON(_expr))
Signature · 1 line
2.54.0
Lorenzo Stoakes
maintainerRFC PATCH 7/10 mm/rmap: track whether the page VMA mapped walk is anonymous
In reply to Lorenzo StoakesUpdate the page_vma_mapped_walk structure to track whether the walk is over an anonymous folio or not.
This is necessary in order to determine the correct VMA page offset (virtual or not) in vma_address_end() ready for a subsequent change which adjusts which page offset to use depending on this parameter, and update the comment slightly.
No functional change intended.
--- include/linux/rmap.h | 2 ++ mm/internal.h | 13 ++++++++----- mm/rmap.c | 3 +++ 3 files changed, 13 insertions(+), 5 deletions(-)
include/linux/rmap.h +2 -0
@@ -871,6 +871,7 @@ struct page_vma_mapped_walk {pte_t *pte;spinlock_t *ptl;unsigned int flags; +bool is_anon_walk; };#define DEFINE_FOLIO_VMA_WALK (name, _folio, _vma, _address, _flags) \
@@ -881,6 +882,7 @@ struct page_vma_mapped_walk { .vma = _vma, \ .address = _address, \ .flags = _flags, \ + .is_anon_walk = folio_test_anon(_folio), \ }static inline void page_vma_mapped_walk_done (struct page_vma_mapped_walk *pvmw)
mm/internal.h +8 -5
@@ -1298,22 +1298,25 @@ static inline unsigned long vma_anon_address(const struct vm_area_struct *vma, }/* -* Then at what user virtual address will none of the range be found in vma? +* At what user virtual address will none of the range be found in vma? * Assumes that vma_address() already returned a good starting address. */ static inline unsigned long vma_address_end (struct page_vma_mapped_walk *pvmw) { -struct vm_area_struct *vma = pvmw->vma; -pgoff_t pgoff; +const struct vm_area_struct *vma = pvmw->vma; +const pgoff_t pgoff = pvmw->pgoff; +pgoff_t pgoff_vma_start;unsigned long address; +pgoff_t pgoff_end;/* Common case, plus ->pgoff is invalid for KSM */ if (pvmw->nr_pages ==1 )return pvmw->address +PAGE_SIZE ; - pgoff = pvmw->pgoff + pvmw->nr_pages; + pgoff_vma_start =vma_start_pgoff (vma); + pgoff_end = pgoff + pvmw->nr_pages; address = vma->vm_start + - ((pgoff -vma_start_pgoff (vma)) <<PAGE_SHIFT ); + ((pgoff_end - pgoff_vma_start) <<PAGE_SHIFT );/* Check for address beyond vma (or wrapped through 0?) */ if (address < vma->vm_start || address > vma->vm_end) address = vma->vm_end;
mm/rmap.c +3 -0
@@ -1240,8 +1240,10 @@ static bool mapping_wrprotect_range_one(struct folio *folio, .vma = vma, .address = address, .flags = PVMW_SYNC, + .is_anon_walk = false, }; +VM_WARN_ON_ONCE (folio_test_anon (folio)); state->cleaned +=page_vma_mkclean_one (&pvmw);return true;
@@ -1317,6 +1319,7 @@ int pfn_mkclean_range(unsigned long pfn, unsigned long nr_pages, pgoff_t pgoff, .pgoff = pgoff, .vma = vma, .flags = PVMW_SYNC, + .is_anon_walk = false, };if (invalid_mkclean_vma (vma,NULL ))
Signature · 1 line
2.54.0
Lorenzo Stoakes
maintainerRFC PATCH 8/10 mm: introduce and use linear_folio_page_index()
In reply to Lorenzo StoakesThis function is, for now, a placeholder; it will be used in future to determine whether to use the virtual page index or not, based on whether the folio is anonymous or not.
Currently it simply wraps linear_page_index(), so this does not change behaviour.
We update callers that will, once the change is introduced to track anonymous folios by virtual page offset if MAP_PRIVATE file-backed, need to determine which index to use based on folio type.
No functional change intended.
--- include/linux/pagemap.h | 18 ++++++++++++++++++ mm/huge_memory.c | 3 ++- mm/migrate.c | 6 ++++-- mm/userfaultfd.c | 6 ++++-- 4 files changed, 28 insertions(+), 5 deletions(-)
include/linux/pagemap.h +18 -0
@@ -1146,6 +1146,24 @@ static inline pgoff_t linear_virt_page_index(const struct vm_area_struct *vma,return pgoff; } +/** +* linear_folio_page_index() - Determine the absolute page offset of +* @address within @vma from @folio. +* @folio: The folio whose linear page index is sought. +* @vma: The VMA in which @address resides. +* @address: The address whose absolute page offset is required. +* +* For compatibility, currently identical to linear_page_index(). +* +* Returns: The absolute page offset of @address within @vma. +*/ +static inline pgoff_t linear_folio_page_index (const struct folio *folio, +const struct vm_area_struct *vma, +const unsigned long address) +{ +return linear_page_index (vma, address); +} +struct wait_page_key {struct folio *folio;int bit_nr;
mm/huge_memory.c +2 -1
@@ -2832,7 +2832,8 @@ int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pm }folio_move_anon_rmap (src_folio, dst_vma); - src_folio->index =linear_page_index (dst_vma, dst_addr); + src_folio->index =linear_folio_page_index (src_folio, dst_vma, + dst_addr); _dst_pmd =folio_mk_pmd (src_folio, dst_vma->vm_page_prot);/* Follow mremap() behavior and treat the entry dirty after the move */
mm/migrate.c +4 -2
@@ -358,8 +358,10 @@ static bool remove_migration_pte(struct folio *folio,unsigned long idx =0 ;/* pgoff is invalid for ksm pages, but they are never large */ -if (folio_test_large (folio) && !folio_test_hugetlb (folio)) - idx =linear_page_index (vma, pvmw.address) - pvmw.pgoff; +if (folio_test_large (folio) && !folio_test_hugetlb (folio)) { + idx +=linear_folio_page_index (folio, vma, pvmw.address); + idx -= pvmw.pgoff; + } new =folio_page (folio, idx);#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
mm/userfaultfd.c +4 -2
@@ -1283,7 +1283,8 @@ static long move_present_ptes(struct mm_struct *mm, }folio_move_anon_rmap (src_folio, dst_vma); - src_folio->index =linear_page_index (dst_vma, dst_addr); + src_folio->index =linear_folio_page_index (src_folio, dst_vma, + dst_addr); orig_dst_pte =folio_mk_pte (src_folio, dst_vma->vm_page_prot);/* Set soft dirty bit so userspace can notice the pte was moved */
@@ -1352,7 +1353,8 @@ static int move_swap_pte(struct mm_struct *mm, struct vm_area_struct *dst_vma,*/ if (src_folio) {folio_move_anon_rmap (src_folio, dst_vma); - src_folio->index =linear_page_index (dst_vma, dst_addr); + src_folio->index =linear_folio_page_index (src_folio, dst_vma, + dst_addr); }else {/* * Check if the swap entry is cached after acquiring the src_pte
Signature · 1 line
2.54.0
Lorenzo Stoakes
maintainerRFC PATCH 9/10 mm/rmap: use virt pgoff for MAP_PRIVATE file-backed anon folios
In reply to Lorenzo StoakesCurrently, anonymous folios belonging to CoW'd MAP_PRIVATE file-backed mappings are indexed by their page offset within the file in which their were originally mapped.
This differs from anonymous foios belonging to pure anon mappings which are indexed by their virtual page offset (the address at which they'd belong in the VMA when first faulted).
This change fixes this inconsistency, always indexing anonymous folios by their virtual page offset regardless of the VMA to which they belong.
We have laid the foundations for making this change to the point where we need only 'switch it on', and this patch switches it on by:
* Using linear_virt_page_index() in __folio_set_anon() to assign the folio's index to the anonymous linear index rather than the file-backed one.
* Otherwise using linear_virt_page_index() in all instances where anonymous folios are being referenced or manipulated.
* Replacing vma_address() with vma_filebacked_address() or vma_anon_address() as appropriate.
* Updating the rmap lock logic in copy_vma() to also account for virtual page offsets.
* Updating the merging logic to check that virtual page offsets are aligned as well as filebacked ones for anonymous or MAP_PRIVATE file-backed VMAs.
* Updating linear_folio_page_index() to invoke linear_virt_page_index() if the folio is anonymous.
This will have no impact on merging of anonymous VMAs or shared file-backed VMAs, whose page offset and anonymous page offset will be identical.
However, MAP_PRIVATE file-backed mappings must now be aligned on virtual page offset as well.
In most instances this should have no impact on merging of file-backed mappings, which are usually not merged all that often, let alone MAP_PRIVATE mapped ones, and rarely remapped and faulted before being moved back in place (the case in which a merge may now fail).
This change lays the foundations for future scalable CoW work which needs to track at least some remaps.
This change means that most remap tracking can be avoided, and in nearly all cases the anonymous page offset can be used to quickly find the VMA in an mm.
--- include/linux/pagemap.h | 9 ++++++++- mm/internal.h | 25 +++++++------------------ mm/interval_tree.c | 4 ++-- mm/ksm.c | 2 +- mm/page_vma_mapped.c | 2 +- mm/rmap.c | 12 ++++++------ mm/vma.c | 14 ++++++++++++-- 7 files changed, 37 insertions(+), 31 deletions(-)
include/linux/pagemap.h +8 -1
@@ -1153,7 +1153,11 @@ static inline pgoff_t linear_virt_page_index(const struct vm_area_struct *vma,* @vma: The VMA in which @address resides. * @address: The address whose absolute page offset is required. * -* For compatibility, currently identical to linear_page_index(). +* Determines whether to obtain the virtual linear page index based on whether +* @folio is anonymous or not. +* +* See the descriptions of linear_virt_page_index() and linear_page_index() for +* details of each. * * Returns: The absolute page offset of @address within @vma. */
@@ -1161,6 +1165,9 @@ static inline pgoff_t linear_folio_page_index(const struct folio *folio,const struct vm_area_struct *vma, const unsigned long address) { + if (folio_test_anon(folio)) + return linear_virt_page_index(vma, address); + return linear_page_index(vma, address); }
mm/internal.h +7 -18
@@ -1263,23 +1263,8 @@ static inline unsigned long vma_filebacked_address(const struct vm_area_struct * }/** -* vma_address - Find the virtual address a page range is mapped at. -* @vma: The vma which maps this object. -* @pgoff: The page offset within its object. -* @nr_pages: The number of pages to consider. -* -* If any page in this range is mapped by this VMA, return the first address -* where any of these pages appear. Otherwise, return -EFAULT. -*/ -static inline unsigned long vma_address (const struct vm_area_struct *vma, -pgoff_t pgoff,unsigned long nr_pages) -{ -return __vma_address (vma, pgoff,vma_start_pgoff (vma), nr_pages); -} - -/** -* vma_anon_address - Find the address an anonymous folio with index @pgoff_virt -* is mapped at. +* vma_anon_address - Find the virtual address an anonymous page range is mapped +* at. * @vma: The vma which maps this object. * @pgoff_virt: The virtual page index belonging to the folio. * @nr_pages: The number of pages to consider.
@@ -1313,7 +1298,11 @@ static inline unsigned long vma_address_end(struct page_vma_mapped_walk *pvmw)if (pvmw->nr_pages ==1 )return pvmw->address +PAGE_SIZE ; - pgoff_vma_start =vma_start_pgoff (vma); +if (pvmw->is_anon_walk) + pgoff_vma_start =vma_start_virt_pgoff (vma); +else + pgoff_vma_start =vma_start_pgoff (vma); + pgoff_end = pgoff + pvmw->nr_pages; address = vma->vm_start + ((pgoff_end - pgoff_vma_start) <<PAGE_SHIFT );
mm/interval_tree.c +2 -2
@@ -83,12 +83,12 @@ mapping_interval_tree_iter_next(struct vm_area_struct *vma,static pgoff_t avc_start_pgoff (struct anon_vma_chain *avc) { -return vma_start_pgoff (avc->vma); +return vma_start_virt_pgoff (avc->vma); }static pgoff_t avc_last_pgoff (struct anon_vma_chain *avc) { -return vma_last_pgoff (avc->vma); +return vma_last_virt_pgoff (avc->vma); } INTERVAL_TREE_DEFINE(struct anon_vma_chain, rb, pgoff_t, rb_subtree_last,
mm/ksm.c +1 -1
@@ -3120,7 +3120,7 @@ struct folio *ksm_might_need_to_copy(struct folio *folio,return folio;/* no need to copy it */ }else if (!anon_vma) {return folio;/* no need to copy it */ - }else if (folio->index ==linear_page_index (vma, addr) && + }else if (folio->index ==linear_virt_page_index (vma, addr) && anon_vma->root == vma->anon_vma->root) {return folio;/* still no need to copy it */ }
mm/page_vma_mapped.c +1 -1
@@ -356,7 +356,7 @@ unsigned long page_mapped_in_vma(const struct page *page, };if (folio_test_anon (folio)) - pvmw.address =vma_address (vma, pgoff,1 ); + pvmw.address =vma_anon_address (vma, pgoff,1 );else pvmw.address =vma_filebacked_address (vma, pgoff,1 );if (pvmw.address == -EFAULT )
mm/rmap.c +6 -6
@@ -866,7 +866,7 @@ unsigned long page_address_in_vma(const struct folio *folio, vma->anon_vma->root != anon_vma->root)return -EFAULT ;/* KSM folios don't reach here because of the !anon_vma check */ -return vma_address (vma,page_pgoff (folio, page),1 ); +return vma_anon_address (vma,page_pgoff (folio, page),1 ); }else if (!vma->vm_file) {return -EFAULT ; }else if (vma->vm_file->f_mapping != folio->mapping) {
@@ -1486,7 +1486,7 @@ static void __folio_set_anon(struct folio *folio, struct vm_area_struct *vma,*/ anon_vma = (void *) anon_vma +FOLIO_MAPPING_ANON ;WRITE_ONCE (folio->mapping, (struct address_space *) anon_vma); - folio->index =linear_page_index (vma, address); + folio->index =linear_virt_page_index (vma, address); }/**
@@ -1513,8 +1513,8 @@ static void __page_check_anon_rmap(const struct folio *folio,*/ VM_BUG_ON_FOLIO (folio_anon_vma (folio)->root != vma->anon_vma->root, folio); -VM_BUG_ON_PAGE (page_pgoff (folio, page) !=linear_page_index (vma, address), - page); +VM_BUG_ON_PAGE (page_pgoff (folio, page) != +linear_virt_page_index (vma, address), page); } static __always_inline void __folio_add_anon_rmap(struct folio *folio,
@@ -2992,10 +2992,10 @@ static void rmap_walk_anon(struct folio *folio, pgoff_end = pgoff_start +folio_nr_pages (folio) -1 ; anon_vma_interval_tree_foreach(avc, anon_vma, pgoff_start, pgoff_end) {struct vm_area_struct *vma = avc->vma; -unsigned long address =vma_address (vma, pgoff_start, +const unsigned long address =vma_anon_address (vma, pgoff_start,folio_nr_pages (folio)); -VM_BUG_ON_VMA (address == -EFAULT , vma); +VM_WARN_ON_ONCE_VMA (address == -EFAULT , vma);cond_resched ();if (rwc->invalid_vma && rwc->invalid_vma (vma, rwc->arg))
mm/vma.c +12 -2
@@ -233,6 +233,8 @@ static bool can_vma_merge_before(struct vma_merge_struct *vmg)return false;if (vmg_end_pgoff (vmg) !=vma_start_pgoff (vmg->next))return false; +if (vmg_end_anon_pgoff (vmg) !=vma_start_anon_pgoff (vmg->next)) +return false;return true; }
@@ -253,6 +255,8 @@ static bool can_vma_merge_after(struct vma_merge_struct *vmg)return false;if (vma_end_pgoff (vmg->prev) !=vmg_start_pgoff (vmg))return false; +if (vma_end_anon_pgoff (vmg->prev) !=vmg_start_anon_pgoff (vmg)) +return false;return true; }
@@ -1991,7 +1995,8 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, *vmap = vma = new_vma; } *need_rmap_locks = - (vma_start_pgoff (new_vma) <=vma_start_pgoff (vma)); + (vma_start_pgoff (new_vma) <=vma_start_pgoff (vma)) || + (vma_start_anon_pgoff (new_vma) <=vma_start_anon_pgoff (vma)); }else { new_vma =vm_area_dup (vma);if (!new_vma)
@@ -2062,7 +2067,12 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct *if (!vma_flags_empty (&diff))return false;/* Page offset must align. */ -return vma_end_pgoff (a) ==vma_start_pgoff (b); +if (vma_end_pgoff (a) !=vma_start_pgoff (b)) +return false; +/* Anon page offset must align. */ +if (vma_end_anon_pgoff (a) !=vma_start_anon_pgoff (b)) +return false; +return true; }/*
Signature · 1 line
2.54.0
Lorenzo Stoakes
maintainerRFC PATCH 10/10 tools/testing/vma: expand VMA merge tests to assert virt pgoff
In reply to Lorenzo StoakesNow we have introduced the VMA virtual page offset attribute and update it when VMAs are manipulated, update VMA merge tests to assert that the virtual page offset is as expected.
--- tools/testing/vma/tests/merge.c | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
tools/testing/vma/tests/merge.c +18 -0
@@ -121,6 +121,7 @@ static bool test_simple_merge(void)ASSERT_EQ (vma->vm_start,0 );ASSERT_EQ (vma->vm_end,0x3000 );ASSERT_EQ (vma_start_pgoff (vma),0 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),0 );ASSERT_FLAGS_SAME_MASK (&vma->flags, vma_flags);detach_free_vma (vma);
@@ -153,6 +154,7 @@ static bool test_simple_modify(void)ASSERT_EQ (vma->vm_start,0x1000 );ASSERT_EQ (vma->vm_end,0x2000 );ASSERT_EQ (vma_start_pgoff (vma),1 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),1 );/* * Now walk through the three split VMAs and make sure they are as
@@ -165,6 +167,7 @@ static bool test_simple_modify(void)ASSERT_EQ (vma->vm_start,0 );ASSERT_EQ (vma->vm_end,0x1000 );ASSERT_EQ (vma_start_pgoff (vma),0 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),0 );detach_free_vma (vma);vma_iter_clear (&vmi);
@@ -174,6 +177,7 @@ static bool test_simple_modify(void)ASSERT_EQ (vma->vm_start,0x1000 );ASSERT_EQ (vma->vm_end,0x2000 );ASSERT_EQ (vma_start_pgoff (vma),1 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),1 );detach_free_vma (vma);vma_iter_clear (&vmi);
@@ -183,6 +187,7 @@ static bool test_simple_modify(void)ASSERT_EQ (vma->vm_start,0x2000 );ASSERT_EQ (vma->vm_end,0x3000 );ASSERT_EQ (vma_start_pgoff (vma),2 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),2 );detach_free_vma (vma);mtree_destroy (&mm.mm_mt);
@@ -212,6 +217,7 @@ static bool test_simple_expand(void)ASSERT_EQ (vma->vm_start,0 );ASSERT_EQ (vma->vm_end,0x3000 );ASSERT_EQ (vma_start_pgoff (vma),0 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),0 );detach_free_vma (vma);mtree_destroy (&mm.mm_mt);
@@ -234,6 +240,7 @@ static bool test_simple_shrink(void)ASSERT_EQ (vma->vm_start,0 );ASSERT_EQ (vma->vm_end,0x1000 );ASSERT_EQ (vma_start_pgoff (vma),0 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),0 );detach_free_vma (vma);mtree_destroy (&mm.mm_mt);
@@ -346,6 +353,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky,ASSERT_EQ (vma->vm_start,0 );ASSERT_EQ (vma->vm_end,0x5000 );ASSERT_EQ (vma_start_pgoff (vma),0 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),0 );ASSERT_EQ (vma->anon_vma, &dummy_anon_vma);ASSERT_TRUE (vma_write_started (vma));ASSERT_EQ (mm.map_count,3 );
@@ -367,6 +375,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky,ASSERT_EQ (vma->vm_start,0x6000 );ASSERT_EQ (vma->vm_end,0x9000 );ASSERT_EQ (vma_start_pgoff (vma),6 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),6 );ASSERT_EQ (vma->anon_vma, &dummy_anon_vma);ASSERT_TRUE (vma_write_started (vma));ASSERT_EQ (mm.map_count,3 );
@@ -387,6 +396,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky,ASSERT_EQ (vma->vm_start,0 );ASSERT_EQ (vma->vm_end,0x9000 );ASSERT_EQ (vma_start_pgoff (vma),0 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),0 );ASSERT_EQ (vma->anon_vma, &dummy_anon_vma);ASSERT_TRUE (vma_write_started (vma));ASSERT_EQ (mm.map_count,2 );
@@ -407,6 +417,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky,ASSERT_EQ (vma->vm_start,0xa000 );ASSERT_EQ (vma->vm_end,0xc000 );ASSERT_EQ (vma_start_pgoff (vma),0xa ); +ASSERT_EQ (vma_start_virt_pgoff (vma),0xa );ASSERT_EQ (vma->anon_vma, &dummy_anon_vma);ASSERT_TRUE (vma_write_started (vma));ASSERT_EQ (mm.map_count,2 );
@@ -426,6 +437,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky,ASSERT_EQ (vma->vm_start,0 );ASSERT_EQ (vma->vm_end,0xc000 );ASSERT_EQ (vma_start_pgoff (vma),0 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),0 );ASSERT_EQ (vma->anon_vma, &dummy_anon_vma);ASSERT_TRUE (vma_write_started (vma));ASSERT_EQ (mm.map_count,1 );
@@ -446,6 +458,7 @@ static bool __test_merge_new(bool is_sticky, bool a_is_sticky, bool b_is_sticky,ASSERT_EQ (vma->vm_start,0 );ASSERT_EQ (vma->vm_end,0xc000 );ASSERT_EQ (vma_start_pgoff (vma),0 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),0 );ASSERT_EQ (vma->anon_vma, &dummy_anon_vma);detach_free_vma (vma);
@@ -808,6 +821,7 @@ static bool test_vma_merge_new_with_close(void)ASSERT_EQ (vma->vm_start,0 );ASSERT_EQ (vma->vm_end,0x5000 );ASSERT_EQ (vma_start_pgoff (vma),0 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),0 );ASSERT_EQ (vma->vm_ops, &vm_ops);ASSERT_TRUE (vma_write_started (vma));ASSERT_EQ (mm.map_count,2 );
@@ -868,6 +882,7 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, boASSERT_EQ (vma->vm_start,0x2000 );ASSERT_EQ (vma->vm_end,0x3000 );ASSERT_EQ (vma_start_pgoff (vma),2 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),2 );ASSERT_TRUE (vma_write_started (vma));ASSERT_TRUE (vma_write_started (vma_next));ASSERT_EQ (mm.map_count,2 );
@@ -934,6 +949,7 @@ static bool __test_merge_existing(bool prev_is_sticky, bool middle_is_sticky, boASSERT_EQ (vma->vm_start,0x6000 );ASSERT_EQ (vma->vm_end,0x7000 );ASSERT_EQ (vma_start_pgoff (vma),6 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),6 );ASSERT_TRUE (vma_write_started (vma_prev));ASSERT_TRUE (vma_write_started (vma));ASSERT_EQ (mm.map_count,2 );
@@ -1419,6 +1435,7 @@ static bool test_merge_extend(void)ASSERT_EQ (vma->vm_start,0 );ASSERT_EQ (vma->vm_end,0x4000 );ASSERT_EQ (vma_start_pgoff (vma),0 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),0 );ASSERT_TRUE (vma_write_started (vma));ASSERT_EQ (mm.map_count,1 );
@@ -1459,6 +1476,7 @@ static bool test_expand_only_mode(void)ASSERT_EQ (vma->vm_start,0x3000 );ASSERT_EQ (vma->vm_end,0x9000 );ASSERT_EQ (vma_start_pgoff (vma),3 ); +ASSERT_EQ (vma_start_virt_pgoff (vma),3 );ASSERT_TRUE (vma_write_started (vma));ASSERT_EQ (vma_iter_addr (&vmi),0x3000 );vma_assert_attached (vma);
Signature · 1 line
2.54.0