refactor mremap and fix bug
Subsystems (3)
cover refactor mremap and fix bug
The existing mremap() logic has grown organically over a very long period of time, resulting in code that is in many parts, very difficult to follow and full of subtleties and sources of confusion.
In addition, it is difficult to thread state through the operation correctly, as function arguments have expanded, some parameters are expected to be temporarily altered during the operation, others are intended to remain static and some can be overridden.
This series completely refactors the mremap implementation, sensibly separating functions, adding comments to explain the more subtle aspects of the implementation and making use of small structs to thread state through everything.
The reason for doing so is to lay the groundwork for planned future changes to the mremap logic, changes which require the ability to easily pass around state.
Additionally, it would be unhelpful to add yet more logic to code that is already difficult to follow without first refactoring it like this.
The first patch in this series additionally fixes a bug when a VMA with start address zero is partially remapped.
Tested on real hardware under heavy workload and all self tests are passing.
Lorenzo Stoakes (7):
mm/mremap: correctly handle partial mremap() of VMA starting at 0 mm/mremap: refactor mremap() system call implementation mm/mremap: introduce and use vma_remap_struct threaded state mm/mremap: initial refactor of move_vma()
mm/mremap: complete refactor of move_vma()
mm/mremap: refactor move_page_tables(), abstracting state mm/mremap: thread state through move page table operation
mm/internal.h | 49 +- mm/mmap.c | 5 +- mm/mremap.c | 1440 +++++++++++++++++++++++++++++++++---------------- 3 files changed, 1033 insertions(+), 461 deletions(-)
--
2.48.1
PATCH 1/7 mm/mremap: correctly handle partial mremap() of VMA starting at 0
Consider the case of a a partial mremap() (that results in a VMA split) of an accountable VMA (i.e. which has the VM_ACCOUNT flag set) whose start address is zero, with the MREMAP_MAYMOVE flag specified and a scenario where a move does in fact occur:
addr end
| |
v v
|-------------|
| vma |
|-------------|
0This move is affected by unmapping the range [addr, end). In order to prevent an incorrect decrement of accounted memory which has already been determined, the mremap() code in move_vma() clears VM_ACCOUNT from the VMA prior to doing so, before reestablishing it in each of the VMAs post-split:
addr end
| |
v v
|---| |---|
| A | | B |
|---| |---|Commit 6b73cff239e5 ("mm: change munmap splitting order and move_vma()") changed this logic such as to determine whether there is a need to do so by establishing account_start and account_end and, in the instance where such an operation is required, assigning them to vma->vm_start and vma->vm_end.
Later the code checks if the operation is required for 'A' referenced above thusly:
if (account_start) {
...
}However, if the VMA described above has vma->vm_start == 0, which is now assigned to account_start, this branch will not be executed.
As a result, the VMA 'A' above will remain stripped of its VM_ACCOUNT flag, incorrectly.
The fix is to simply convert these variables to booleans and set them as required.
--- mm/mremap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
mm/mremap.c +5 -5
@@ -705,8 +705,8 @@ static unsigned long move_vma(struct vm_area_struct *vma,unsigned long vm_flags = vma->vm_flags;unsigned long new_pgoff;unsigned long moved_len; -unsigned long account_start =0 ; -unsigned long account_end =0 ; +bool account_start = false; +bool account_end = false;unsigned long hiwater_vm;int err =0 ;bool need_rmap_locks;
@@ -790,9 +790,9 @@ static unsigned long move_vma(struct vm_area_struct *vma,if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP )) {vm_flags_clear (vma,VM_ACCOUNT );if (vma->vm_start < old_addr) - account_start = vma->vm_start; + account_start = true;if (vma->vm_end > old_addr + old_len) - account_end = vma->vm_end; + account_end = true; }/*
@@ -832,7 +832,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,/* OOM: unable to split vma, just get accounts right */ if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP ))vm_acct_memory (old_len >>PAGE_SHIFT ); - account_start = account_end =0 ; + account_start = account_end = false; }if (vm_flags &VM_LOCKED ) {
Signature · 1 line
2.48.1
Re: [PATCH 1/7] mm/mremap: correctly handle partial mremap() of VMA starting at 0
Quoted 43 lines · Lorenzo Stoakes
Consider the case of a a partial
mremap()(that results in a VMA split) of an accountable VMA (i.e. which has theVM_ACCOUNTflag set) whose start address is zero, with theMREMAP_MAYMOVEflag specified and a scenario where a move does in fact occur:addr end | | v v |-------------| | vma | |-------------| 0This move is affected by unmapping the range [addr, end). In order to prevent an incorrect decrement of accounted memory which has already been determined, the
mremap()code inmove_vma()clearsVM_ACCOUNTfrom the VMA prior to doing so, before reestablishing it in each of the VMAs post-split:addr end | | v v |---| |---| | A | | B | |---| |---|Commit
6b73cff239e5("mm: change munmap splitting order andmove_vma()") changed this logic such as to determine whether there is a need to do so by establishingaccount_startandaccount_endand, in the instance where such an operation is required, assigning them tovma->vm_startandvma->vm_end.Later the code checks if the operation is required for 'A' referenced above thusly:
if (account_start) { ... }However, if the VMA described above has
vma->vm_start== 0, which is now assigned toaccount_start, this branch will not be executed.As a result, the VMA 'A' above will remain stripped of its
VM_ACCOUNTflag, incorrectly.
Quoted 7 lines
The fix is to simply convert these variables to booleans and set them as required.
Signed-off-by Lorenzo Stoakes <[email protected]>Fixes 6b73cff239e5 ("mm: change munmap splitting order and move_vma()")---
Looks good to me,
Signature · 2 lines
Cheers, Harry
Quoted 43 lines
mm/mremap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)mm/mremap.c +5
-5@@ -705,8 +705,8 @@ static unsigned long move_vma(struct vm_area_struct *vma,unsigned long vm_flags = vma->vm_flags;unsigned long new_pgoff;unsigned long moved_len; -unsigned long account_start =0 ; -unsigned long account_end =0 ; +bool account_start = false; +bool account_end = false;unsigned long hiwater_vm;int err =0 ;bool need_rmap_locks;@@ -790,9 +790,9 @@ static unsigned long move_vma(struct vm_area_struct *vma,if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP )) {vm_flags_clear (vma,VM_ACCOUNT );if (vma->vm_start < old_addr) - account_start = vma->vm_start; + account_start = true;if (vma->vm_end > old_addr + old_len) - account_end = vma->vm_end; + account_end = true; }/* @@ -832,7 +832,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,/* OOM: unable to split vma, just get accounts right */ if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP ))vm_acct_memory (old_len >>PAGE_SHIFT ); - account_start = account_end =0 ; + account_start = account_end = false; }if (vm_flags &VM_LOCKED ) {Signature · 1 line
2.48.1
Re: [PATCH 1/7] mm/mremap: correctly handle partial mremap() of VMA starting at 0
In reply to Lorenzo StoakesQuoted 43 lines · Lorenzo Stoakes
Consider the case of a a partial
mremap()(that results in a VMA split) of an accountable VMA (i.e. which has theVM_ACCOUNTflag set) whose start address is zero, with theMREMAP_MAYMOVEflag specified and a scenario where a move does in fact occur:addr end | | v v |-------------| | vma | |-------------| 0This move is affected by unmapping the range [addr, end). In order to prevent an incorrect decrement of accounted memory which has already been determined, the
mremap()code inmove_vma()clearsVM_ACCOUNTfrom the VMA prior to doing so, before reestablishing it in each of the VMAs post-split:addr end | | v v |---| |---| | A | | B | |---| |---|Commit
6b73cff239e5("mm: change munmap splitting order andmove_vma()") changed this logic such as to determine whether there is a need to do so by establishingaccount_startandaccount_endand, in the instance where such an operation is required, assigning them tovma->vm_startandvma->vm_end.Later the code checks if the operation is required for 'A' referenced above thusly:
if (account_start) { ... }However, if the VMA described above has
vma->vm_start== 0, which is now assigned toaccount_start, this branch will not be executed.As a result, the VMA 'A' above will remain stripped of its
VM_ACCOUNTflag, incorrectly.
Quoted 6 lines
The fix is to simply convert these variables to booleans and set them as required.
Signed-off-by Lorenzo Stoakes <[email protected]>Fixes 6b73cff239e5 ("mm: change munmap splitting order and move_vma()")
Thanks for taking care of this.
Quoted 43 lines
--- mm/mremap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)mm/mremap.c +5
-5@@ -705,8 +705,8 @@ static unsigned long move_vma(struct vm_area_struct *vma,unsigned long vm_flags = vma->vm_flags;unsigned long new_pgoff;unsigned long moved_len; -unsigned long account_start =0 ; -unsigned long account_end =0 ; +bool account_start = false; +bool account_end = false;unsigned long hiwater_vm;int err =0 ;bool need_rmap_locks;@@ -790,9 +790,9 @@ static unsigned long move_vma(struct vm_area_struct *vma,if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP )) {vm_flags_clear (vma,VM_ACCOUNT );if (vma->vm_start < old_addr) - account_start = vma->vm_start; + account_start = true;if (vma->vm_end > old_addr + old_len) - account_end = vma->vm_end; + account_end = true; }/* @@ -832,7 +832,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,/* OOM: unable to split vma, just get accounts right */ if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP ))vm_acct_memory (old_len >>PAGE_SHIFT ); - account_start = account_end =0 ; + account_start = account_end = false; }if (vm_flags &VM_LOCKED ) {Signature · 1 line
2.48.1
Re: [PATCH 1/7] mm/mremap: correctly handle partial mremap() of VMA starting at 0
In reply to Lorenzo StoakesQuoted 43 lines · Lorenzo Stoakes
Consider the case of a a partial
mremap()(that results in a VMA split) of an accountable VMA (i.e. which has theVM_ACCOUNTflag set) whose start address is zero, with theMREMAP_MAYMOVEflag specified and a scenario where a move does in fact occur:addr end | | v v |-------------| | vma | |-------------| 0This move is affected by unmapping the range [addr, end). In order to prevent an incorrect decrement of accounted memory which has already been determined, the
mremap()code inmove_vma()clearsVM_ACCOUNTfrom the VMA prior to doing so, before reestablishing it in each of the VMAs post-split:addr end | | v v |---| |---| | A | | B | |---| |---|Commit
6b73cff239e5("mm: change munmap splitting order andmove_vma()") changed this logic such as to determine whether there is a need to do so by establishingaccount_startandaccount_endand, in the instance where such an operation is required, assigning them tovma->vm_startandvma->vm_end.Later the code checks if the operation is required for 'A' referenced above thusly:
if (account_start) { ... }However, if the VMA described above has
vma->vm_start== 0, which is now assigned toaccount_start, this branch will not be executed.As a result, the VMA 'A' above will remain stripped of its
VM_ACCOUNTflag, incorrectly.
Quoted 6 lines
The fix is to simply convert these variables to booleans and set them as required.
Signed-off-by Lorenzo Stoakes <[email protected]>Fixes 6b73cff239e5 ("mm: change munmap splitting order and move_vma()")
Looks like the variables would be more accurately named account_prev and account_next? But maybe it's refactored later anyway. Keeps the diff to stable backport smaller.
Re: [PATCH 1/7] mm/mremap: correctly handle partial mremap() of VMA starting at 0
Quoted 47 lines · Vlastimil Babka
Quoted 46 lines · Lorenzo Stoakes
Consider the case of a a partial
mremap()(that results in a VMA split) of an accountable VMA (i.e. which has theVM_ACCOUNTflag set) whose start address is zero, with theMREMAP_MAYMOVEflag specified and a scenario where a move does in fact occur:addr end | | v v |-------------| | vma | |-------------| 0This move is affected by unmapping the range [addr, end). In order to prevent an incorrect decrement of accounted memory which has already been determined, the
mremap()code inmove_vma()clearsVM_ACCOUNTfrom the VMA prior to doing so, before reestablishing it in each of the VMAs post-split:addr end | | v v |---| |---| | A | | B | |---| |---|Commit
6b73cff239e5("mm: change munmap splitting order andmove_vma()") changed this logic such as to determine whether there is a need to do so by establishingaccount_startandaccount_endand, in the instance where such an operation is required, assigning them tovma->vm_startandvma->vm_end.Later the code checks if the operation is required for 'A' referenced above thusly:
if (account_start) { ... }However, if the VMA described above has
vma->vm_start== 0, which is now assigned toaccount_start, this branch will not be executed.As a result, the VMA 'A' above will remain stripped of its
VM_ACCOUNTflag, incorrectly.The fix is to simply convert these variables to booleans and set them as required.
Quoted 7 lines
Quoted 3 lines
Signed-off-by Lorenzo Stoakes <[email protected]>Fixes 6b73cff239e5 ("mm: change munmap splitting order and move_vma()")Looks like the variables would be more accurately named
account_prevandaccount_next? But maybe it's refactored later anyway. Keeps the diff to stable backport smaller.
Yeah indeed, I think it's better to keep it relatively smaller.
Quoted 3 lines
Reviewed-by Vlastimil Babka <[email protected]>
Thanks!
PATCH 2/7 mm/mremap: refactor mremap() system call implementation
In reply to Lorenzo StoakesPlace checks into a separate function so the mremap() system call is less egregiously long, remove unnecessary mremap_to() offset_in_page() check and just check that earlier so we keep all such basic checks together.
Separate out the VMA in-place expansion, hugetlb and expand/move logic into separate, readable functions.
De-duplicate code where possible, add comments and ensure that all error handling explicitly specifies the error at the point of it occurring rather than setting a prefixed error value and implicitly setting (which is bug prone).
This lays the groundwork for subsequent patches further simplifying and extending the mremap() implementation.
--- mm/mremap.c | 405 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 251 insertions(+), 154 deletions(-)
mm/mremap.c +251 -154
@@ -942,33 +942,14 @@ static unsigned long mremap_to(unsigned long addr, unsigned long old_len,unsigned long ret;unsigned long map_flags =0 ; -if (offset_in_page (new_addr)) -return -EINVAL ; - +/* Is the new length or address silly? */ if (new_len >TASK_SIZE || new_addr >TASK_SIZE - new_len)return -EINVAL ; -/* Ensure the old/new locations do not overlap */ +/* Ensure the old/new locations do not overlap. */ if (addr + old_len > new_addr && new_addr + new_len > addr)return -EINVAL ; -/* -* move_vma() need us to stay 4 maps below the threshold, otherwise -* it will bail out at the very beginning. -* That is a problem if we have already unmaped the regions here -* (new_addr, and old_addr), because userspace will not know the -* state of the vma's after it gets -ENOMEM. -* So, to avoid such scenario we can pre-compute if the whole -* operation has high chances to success map-wise. -* Worst-scenario case is when both vma's (new_addr and old_addr) get -* split in 3 before unmapping it. -* That means 2 more maps (1 for each) to the ones we already hold. -* Check whether current map count plus 2 still leads us to 4 maps below -* the threshold, otherwise return -ENOMEM here to be more safe. -*/ -if ((mm->map_count +2 ) >= sysctl_max_map_count -3 ) -return -ENOMEM ; -if (flags &MREMAP_FIXED ) {/* * In mremap_to().
@@ -1035,6 +1016,218 @@ static int vma_expandable(struct vm_area_struct *vma, unsigned long delta)return 1 ; } +/* Do the mremap() flags require that the new_addr parameter be specified? */ +static bool implies_new_addr (unsigned long flags) +{ +return flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +} + +/* +* Are the parameters passed to mremap() valid? If so return 0, otherwise return +* error. +*/ +static unsigned long check_mremap_params (unsigned long addr, +unsigned long flags, +unsigned long old_len, +unsigned long new_len, +unsigned long new_addr) +{ +/* Ensure no unexpected flag values. */ +if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP )) +return -EINVAL ; + +/* Start address must be page-aligned. */ +if (offset_in_page (addr)) +return -EINVAL ; + +/* +* We allow a zero old-len as a special case +* for DOS-emu "duplicate shm area" thing. But +* a zero new-len is nonsensical. +*/ +if (!PAGE_ALIGN (new_len)) +return -EINVAL ; + +/* Remainder of checks are for cases with specific new_addr. */ +if (!implies_new_addr (flags)) +return 0 ; + +/* The new address must be page-aligned. */ +if (offset_in_page (new_addr)) +return -EINVAL ; + +/* A fixed address implies a move. */ +if (!(flags &MREMAP_MAYMOVE )) +return -EINVAL ; + +/* MREMAP_DONTUNMAP does not allow resizing in the process. */ +if (flags &MREMAP_DONTUNMAP && old_len != new_len) +return -EINVAL ; + +/* +* move_vma() need us to stay 4 maps below the threshold, otherwise +* it will bail out at the very beginning. +* That is a problem if we have already unmaped the regions here +* (new_addr, and old_addr), because userspace will not know the +* state of the vma's after it gets -ENOMEM. +* So, to avoid such scenario we can pre-compute if the whole +* operation has high chances to success map-wise. +* Worst-scenario case is when both vma's (new_addr and old_addr) get +* split in 3 before unmapping it. +* That means 2 more maps (1 for each) to the ones we already hold. +* Check whether current map count plus 2 still leads us to 4 maps below +* the threshold, otherwise return -ENOMEM here to be more safe. +*/ +if ((current->mm->map_count +2 ) >= sysctl_max_map_count -3 ) +return -ENOMEM ; + +return 0 ; +} + +/* +* We know we can expand the VMA in-place by delta pages, so do so. +* +* If we discover the VMA is locked, update mm_struct statistics accordingly and +* indicate so to the caller. +*/ +static unsigned long expand_vma_inplace (struct vm_area_struct *vma, +unsigned long delta,bool *locked) +{ +struct mm_struct *mm = current->mm; +long pages = delta >>PAGE_SHIFT ; +VMA_ITERATOR (vmi, mm, vma->vm_end); +long charged =0 ; + +if (vma->vm_flags &VM_ACCOUNT ) { +if (security_vm_enough_memory_mm (mm, pages)) +return -ENOMEM ; + + charged = pages; + } + +/* +* Function vma_merge_extend() is called on the +* extension we are adding to the already existing vma, +* vma_merge_extend() will merge this extension with the +* already existing vma (expand operation itself) and +* possibly also with the next vma if it becomes +* adjacent to the expanded vma and otherwise +* compatible. +*/ + vma =vma_merge_extend (&vmi, vma, delta); +if (!vma) { +vm_unacct_memory (charged); +return -ENOMEM ; + } + +vm_stat_account (mm, vma->vm_flags, pages); +if (vma->vm_flags &VM_LOCKED ) { + mm->locked_vm += pages; + *locked = true; + } + +return 0 ; +} + +static bool align_hugetlb (struct vm_area_struct *vma, +unsigned long addr, +unsigned long new_addr, +unsigned long *old_len_ptr, +unsigned long *new_len_ptr, +unsigned long *delta_ptr) +{ +unsigned long old_len = *old_len_ptr; +unsigned long new_len = *new_len_ptr; +struct hstate *h__maybe_unused =hstate_vma (vma); + + old_len =ALIGN (old_len,huge_page_size (h)); + new_len =ALIGN (new_len,huge_page_size (h)); + +/* addrs must be huge page aligned */ +if (addr & ~huge_page_mask (h)) +return false; +if (new_addr & ~huge_page_mask (h)) +return false; + +/* +* Don't allow remap expansion, because the underlying hugetlb +* reservation is not yet capable to handle split reservation. +*/ +if (new_len > old_len) +return false; + + *old_len_ptr = old_len; + *new_len_ptr = new_len; + *delta_ptr =abs_diff (old_len, new_len); +return true; +} + +/* +* We are mremap()'ing without specifying a fixed address to move to, but are +* requesting that the VMA's size be increased. +* +* Try to do so in-place, if this fails, then move the VMA to a new location to +* action the change. +*/ +static unsigned long expand_vma (struct vm_area_struct *vma, +unsigned long addr,unsigned long old_len, +unsigned long new_len,unsigned long flags, +bool *locked_ptr,unsigned long *new_addr_ptr, +struct vm_userfaultfd_ctx *uf_ptr, +struct list_head *uf_unmap_ptr) +{ +unsigned long err; +unsigned long map_flags; +unsigned long new_addr;/* We ignore any user-supplied one. */ +pgoff_t pgoff; + + err =resize_is_valid (vma, addr, old_len, new_len, flags); +if (err) +return err; + +/* +* [addr, old_len) spans precisely to the end of the VMA, so try to +* expand it in-place. +*/ +if (old_len == vma->vm_end - addr && +vma_expandable (vma, new_len - old_len)) { + err =expand_vma_inplace (vma, new_len - old_len, locked_ptr); +if (IS_ERR_VALUE (err)) +return err; + +/* +* We want to populate the newly expanded portion of the VMA to +* satisfy the expectation that mlock()'ing a VMA maintains all +* of its pages in memory. +*/ +if (*locked_ptr) + *new_addr_ptr = addr; + +/* OK we're done! */ +return addr; + } + +/* +* We weren't able to just expand or shrink the area, +* we need to create a new one and move it. +*/ + +/* We're not allowed to move the VMA, so error out. */ +if (!(flags &MREMAP_MAYMOVE )) +return -ENOMEM ; + +/* Find a new location to move the VMA to. */ + map_flags = (vma->vm_flags &VM_MAYSHARE ) ?MAP_SHARED :0 ; + pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >>PAGE_SHIFT ); + new_addr =get_unmapped_area (vma->vm_file,0 , new_len, pgoff, map_flags); +if (IS_ERR_VALUE (new_addr)) +return new_addr; + *new_addr_ptr = new_addr; + +return move_vma (vma, addr, old_len, new_len, new_addr, + locked_ptr, flags, uf_ptr, uf_unmap_ptr); +} +/* * Expand (or shrink) an existing mapping, potentially moving it at the * same time (controlled by the MREMAP_MAYMOVE flag and available VM space)
@@ -1048,7 +1241,8 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; -unsigned long ret = -EINVAL ; +unsigned long ret; +unsigned long delta;bool locked = false;struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ;LIST_HEAD (uf_unmap_early);
@@ -1067,70 +1261,38 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len,*/ addr =untagged_addr (addr); -if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP )) -return ret; - -if (flags &MREMAP_FIXED && !(flags &MREMAP_MAYMOVE )) -return ret; - -/* -* MREMAP_DONTUNMAP is always a move and it does not allow resizing -* in the process. -*/ -if (flags &MREMAP_DONTUNMAP && - (!(flags &MREMAP_MAYMOVE ) || old_len != new_len)) -return ret; - - -if (offset_in_page (addr)) + ret =check_mremap_params (addr, flags, old_len, new_len, new_addr); +if (ret)return ret; old_len =PAGE_ALIGN (old_len); new_len =PAGE_ALIGN (new_len); + delta =abs_diff (old_len, new_len); -/* -* We allow a zero old-len as a special case -* for DOS-emu "duplicate shm area" thing. But -* a zero new-len is nonsensical. -*/ -if (!new_len) -return ret; - -if (mmap_write_lock_killable (current->mm)) +if (mmap_write_lock_killable (mm))return -EINTR ; + vma =vma_lookup (mm, addr);if (!vma) { ret = -EFAULT ; goto out; } -/* Don't allow remapping vmas when they have already been sealed */ +/* If mseal()'d, mremap() is prohibited. */ if (!can_modify_vma (vma)) { ret = -EPERM ; goto out; } -if (is_vm_hugetlb_page (vma)) { -struct hstate *h__maybe_unused =hstate_vma (vma); - - old_len =ALIGN (old_len,huge_page_size (h)); - new_len =ALIGN (new_len,huge_page_size (h)); - -/* addrs must be huge page aligned */ -if (addr & ~huge_page_mask (h)) - goto out; -if (new_addr & ~huge_page_mask (h)) - goto out; - -/* -* Don't allow remap expansion, because the underlying hugetlb -* reservation is not yet capable to handle split reservation. -*/ -if (new_len > old_len) - goto out; +/* Align to hugetlb page size, if required. */ +if (is_vm_hugetlb_page (vma) && + !align_hugetlb (vma, addr, new_addr, &old_len, &new_len, &delta)) { + ret = -EINVAL ; + goto out; } -if (flags & (MREMAP_FIXED |MREMAP_DONTUNMAP )) { +/* Are we RELOCATING the VMA to a SPECIFIC address? */ +if (implies_new_addr (flags)) { ret =mremap_to (addr, old_len, new_addr, new_len, &locked, flags, &uf, &uf_unmap_early, &uf_unmap);
@@ -1138,109 +1300,44 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, }/* -* Always allow a shrinking remap: that just unmaps -* the unnecessary pages.. -* do_vmi_munmap does all the needed commit accounting, and -* unlocks the mmap_lock if so directed. +* From here on in we are only RESIZING the VMA, attempting to do so +* in-place, moving the VMA if we cannot. */ -if (old_len >= new_len) { -VMA_ITERATOR (vmi, mm, addr + new_len); -if (old_len == new_len) { - ret = addr; - goto out; - } +/* NO-OP CASE - resizing to the same size. */ +if (new_len == old_len) { + ret = addr; + goto out; + } + +/* SHRINK CASE. Can always be done in-place. */ +if (new_len < old_len) { +VMA_ITERATOR (vmi, mm, addr + new_len); - ret =do_vmi_munmap (&vmi, mm, addr + new_len, old_len - new_len, +/* +* Simply unmap the shrunken portion of the VMA. This does all +* the needed commit accounting, unlocking the mmap lock. +*/ + ret =do_vmi_munmap (&vmi, mm, addr + new_len, delta, &uf_unmap, true);if (ret) goto out; +/* We succeeded, mmap lock released for us. */ ret = addr; goto out_unlocked; } -/* -* Ok, we need to grow.. -*/ - ret =resize_is_valid (vma, addr, old_len, new_len, flags); -if (ret) - goto out; - -/* old_len exactly to the end of the area.. -*/ -if (old_len == vma->vm_end - addr) { -unsigned long delta = new_len - old_len; - -/* can we just expand the current mapping? */ -if (vma_expandable (vma, delta)) { -long pages = delta >>PAGE_SHIFT ; -VMA_ITERATOR (vmi, mm, vma->vm_end); -long charged =0 ; - -if (vma->vm_flags &VM_ACCOUNT ) { -if (security_vm_enough_memory_mm (mm, pages)) { - ret = -ENOMEM ; - goto out; - } - charged = pages; - } - -/* -* Function vma_merge_extend() is called on the -* extension we are adding to the already existing vma, -* vma_merge_extend() will merge this extension with the -* already existing vma (expand operation itself) and -* possibly also with the next vma if it becomes -* adjacent to the expanded vma and otherwise -* compatible. -*/ - vma =vma_merge_extend (&vmi, vma, delta); -if (!vma) { -vm_unacct_memory (charged); - ret = -ENOMEM ; - goto out; - } +/* EXPAND case. We try to do in-place, if we can't, then we move it. */ + ret =expand_vma (vma, addr, old_len, new_len, flags, &locked, &new_addr, + &uf, &uf_unmap); -vm_stat_account (mm, vma->vm_flags, pages); -if (vma->vm_flags &VM_LOCKED ) { - mm->locked_vm += pages; - locked = true; - new_addr = addr; - } - ret = addr; - goto out; - } - } - -/* -* We weren't able to just expand or shrink the area, -* we need to create a new one and move it.. -*/ - ret = -ENOMEM ; -if (flags &MREMAP_MAYMOVE ) { -unsigned long map_flags =0 ; -if (vma->vm_flags &VM_MAYSHARE ) - map_flags |=MAP_SHARED ; - - new_addr =get_unmapped_area (vma->vm_file,0 , new_len, - vma->vm_pgoff + - ((addr - vma->vm_start) >>PAGE_SHIFT ), - map_flags); -if (IS_ERR_VALUE (new_addr)) { - ret = new_addr; - goto out; - } - - ret =move_vma (vma, addr, old_len, new_len, new_addr, - &locked, flags, &uf, &uf_unmap); - } out:if (offset_in_page (ret)) locked = false; -mmap_write_unlock (current->mm); +mmap_write_unlock (mm);if (locked && new_len > old_len) -mm_populate (new_addr + old_len, new_len - old_len); +mm_populate (new_addr + old_len, delta); out_unlocked:userfaultfd_unmap_complete (mm, &uf_unmap_early);mremap_userfaultfd_complete (&uf, addr, ret, old_len);
Signature · 1 line
2.48.1
Re: [PATCH 2/7] mm/mremap: refactor mremap() system call implementation
Quoted 69 lines · Lorenzo Stoakes
Place checks into a separate function so the
mremap()system call is less egregiously long, remove unnecessarymremap_to()offset_in_page()check and just check that earlier so we keep all such basic checks together.Separate out the VMA in-place expansion, hugetlb and expand/move logic into separate, readable functions.
De-duplicate code where possible, add comments and ensure that all error handling explicitly specifies the error at the point of it occurring rather than setting a prefixed error value and implicitly setting (which is bug prone).
This lays the groundwork for subsequent patches further simplifying and extending the
mremap()implementation.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/mremap.c | 405 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 251 insertions(+), 154 deletions(-)mm/mremap.c +7
-21@@ -942,33 +942,14 @@ static unsigned long mremap_to(unsigned long addr, unsigned long old_len,unsigned long ret;unsigned long map_flags =0 ; -if (offset_in_page (new_addr)) -return -EINVAL ; - +/* Is the new length or address silly? */ if (new_len >TASK_SIZE || new_addr >TASK_SIZE - new_len)return -EINVAL ; -/* Ensure the old/new locations do not overlap */ +/* Ensure the old/new locations do not overlap. */ if (addr + old_len > new_addr && new_addr + new_len > addr)return -EINVAL ; -/* -* move_vma() need us to stay 4 maps below the threshold, otherwise -* it will bail out at the very beginning. -* That is a problem if we have already unmaped the regions here -* (new_addr, and old_addr), because userspace will not know the -* state of the vma's after it gets -ENOMEM. -* So, to avoid such scenario we can pre-compute if the whole -* operation has high chances to success map-wise. -* Worst-scenario case is when both vma's (new_addr and old_addr) get -* split in 3 before unmapping it. -* That means 2 more maps (1 for each) to the ones we already hold. -* Check whether current map count plus 2 still leads us to 4 maps below -* the threshold, otherwise return -ENOMEM here to be more safe. -*/ -if ((mm->map_count +2 ) >= sysctl_max_map_count -3 ) -return -ENOMEM ; -if (flags &MREMAP_FIXED ) {/* * In mremap_to(). @@ -1035,6 +1016,218 @@ static int vma_expandable(struct vm_area_struct *vma, unsigned long delta)return 1 ; } +/* Do the mremap() flags require that the new_addr parameter be specified? */ +static bool implies_new_addr (unsigned long flags) +{ +return flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +}
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1033
Quoted 10 lines
mm/mremap.c +10
-0+ +/* +* Are the parameters passed to mremap() valid? If so return 0, otherwise return +* error. +*/ +static unsigned long check_mremap_params (unsigned long addr, +unsigned long flags, +unsigned long old_len, +unsigned long new_len, +unsigned long new_addr)
If you use two tabs for the arguments this will be more compact and more readable.
Quoted 151 lines
mm/mremap.c +151
-0+{ +/* Ensure no unexpected flag values. */ +if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP )) +return -EINVAL ; + +/* Start address must be page-aligned. */ +if (offset_in_page (addr)) +return -EINVAL ; + +/* +* We allow a zero old-len as a special case +* for DOS-emu "duplicate shm area" thing. But +* a zero new-len is nonsensical. +*/ +if (!PAGE_ALIGN (new_len)) +return -EINVAL ; + +/* Remainder of checks are for cases with specific new_addr. */ +if (!implies_new_addr (flags)) +return 0 ; + +/* The new address must be page-aligned. */ +if (offset_in_page (new_addr)) +return -EINVAL ; + +/* A fixed address implies a move. */ +if (!(flags &MREMAP_MAYMOVE )) +return -EINVAL ; + +/* MREMAP_DONTUNMAP does not allow resizing in the process. */ +if (flags &MREMAP_DONTUNMAP && old_len != new_len) +return -EINVAL ; + +/* +* move_vma() need us to stay 4 maps below the threshold, otherwise +* it will bail out at the very beginning. +* That is a problem if we have already unmaped the regions here +* (new_addr, and old_addr), because userspace will not know the +* state of the vma's after it gets -ENOMEM. +* So, to avoid such scenario we can pre-compute if the whole +* operation has high chances to success map-wise. +* Worst-scenario case is when both vma's (new_addr and old_addr) get +* split in 3 before unmapping it. +* That means 2 more maps (1 for each) to the ones we already hold. +* Check whether current map count plus 2 still leads us to 4 maps below +* the threshold, otherwise return -ENOMEM here to be more safe. +*/ +if ((current->mm->map_count +2 ) >= sysctl_max_map_count -3 ) +return -ENOMEM ; + +return 0 ; +} + +/* +* We know we can expand the VMA in-place by delta pages, so do so. +* +* If we discover the VMA is locked, update mm_struct statistics accordingly and +* indicate so to the caller. +*/ +static unsigned long expand_vma_inplace (struct vm_area_struct *vma, +unsigned long delta,bool *locked) +{ +struct mm_struct *mm = current->mm; +long pages = delta >>PAGE_SHIFT ; +VMA_ITERATOR (vmi, mm, vma->vm_end); +long charged =0 ; + +if (vma->vm_flags &VM_ACCOUNT ) { +if (security_vm_enough_memory_mm (mm, pages)) +return -ENOMEM ; + + charged = pages; + } + +/* +* Function vma_merge_extend() is called on the +* extension we are adding to the already existing vma, +* vma_merge_extend() will merge this extension with the +* already existing vma (expand operation itself) and +* possibly also with the next vma if it becomes +* adjacent to the expanded vma and otherwise +* compatible. +*/ + vma =vma_merge_extend (&vmi, vma, delta); +if (!vma) { +vm_unacct_memory (charged); +return -ENOMEM ; + } + +vm_stat_account (mm, vma->vm_flags, pages); +if (vma->vm_flags &VM_LOCKED ) { + mm->locked_vm += pages; + *locked = true; + } + +return 0 ; +} + +static bool align_hugetlb (struct vm_area_struct *vma, +unsigned long addr, +unsigned long new_addr, +unsigned long *old_len_ptr, +unsigned long *new_len_ptr, +unsigned long *delta_ptr) +{ +unsigned long old_len = *old_len_ptr; +unsigned long new_len = *new_len_ptr; +struct hstate *h__maybe_unused =hstate_vma (vma); + + old_len =ALIGN (old_len,huge_page_size (h)); + new_len =ALIGN (new_len,huge_page_size (h)); + +/* addrs must be huge page aligned */ +if (addr & ~huge_page_mask (h)) +return false; +if (new_addr & ~huge_page_mask (h)) +return false; + +/* +* Don't allow remap expansion, because the underlying hugetlb +* reservation is not yet capable to handle split reservation. +*/ +if (new_len > old_len) +return false; + + *old_len_ptr = old_len; + *new_len_ptr = new_len; + *delta_ptr =abs_diff (old_len, new_len); +return true; +} + +/* +* We are mremap()'ing without specifying a fixed address to move to, but are +* requesting that the VMA's size be increased. +* +* Try to do so in-place, if this fails, then move the VMA to a new location to +* action the change. +*/ +static unsigned long expand_vma (struct vm_area_struct *vma, +unsigned long addr,unsigned long old_len, +unsigned long new_len,unsigned long flags, +bool *locked_ptr,unsigned long *new_addr_ptr, +struct vm_userfaultfd_ctx *uf_ptr, +struct list_head *uf_unmap_ptr) +{ +unsigned long err; +unsigned long map_flags; +unsigned long new_addr;/* We ignore any user-supplied one. */ +pgoff_t pgoff; + + err =resize_is_valid (vma, addr, old_len, new_len, flags);
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1194
Quoted 10 lines
mm/mremap.c +10
-0+if (err) +return err; + +/* +* [addr, old_len) spans precisely to the end of the VMA, so try to +* expand it in-place. +*/ +if (old_len == vma->vm_end - addr && +vma_expandable (vma, new_len - old_len)) { + err =expand_vma_inplace (vma, new_len - old_len, locked_ptr);
You use delta twice here (new_len - old_len). I assume this is different in the next patches.
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1196
Quoted 2 lines
mm/mremap.c +2
-0+if (IS_ERR_VALUE (err)) +return err;
Doesn't expand_vma_inplace() return 0 on success, error otherwise?
Quoted 8 lines
mm/mremap.c +8
-0+ +/* +* We want to populate the newly expanded portion of the VMA to +* satisfy the expectation that mlock()'ing a VMA maintains all +* of its pages in memory. +*/ +if (*locked_ptr) + *new_addr_ptr = addr;
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1214
Quoted 10 lines
mm/mremap.c +10
-0+ +/* OK we're done! */ +return addr; + } + +/* +* We weren't able to just expand or shrink the area, +* we need to create a new one and move it. +*/ +
So it's more of an expand_or_move_vma()?
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1217
Quoted 3 lines
mm/mremap.c +3
-0+/* We're not allowed to move the VMA, so error out. */ +if (!(flags &MREMAP_MAYMOVE )) +return -ENOMEM ;
and by flags we mean... the flags from the syscall. This gets confusing with map_flags. At least there's only two and not six flags.
Quoted 68 lines
mm/mremap.c +20
-26+ +/* Find a new location to move the VMA to. */ + map_flags = (vma->vm_flags &VM_MAYSHARE ) ?MAP_SHARED :0 ; + pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >>PAGE_SHIFT ); + new_addr =get_unmapped_area (vma->vm_file,0 , new_len, pgoff, map_flags); +if (IS_ERR_VALUE (new_addr)) +return new_addr; + *new_addr_ptr = new_addr; + +return move_vma (vma, addr, old_len, new_len, new_addr, + locked_ptr, flags, uf_ptr, uf_unmap_ptr); +} +/* * Expand (or shrink) an existing mapping, potentially moving it at the * same time (controlled by the MREMAP_MAYMOVE flag and available VM space) @@ -1048,7 +1241,8 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; -unsigned long ret = -EINVAL ; +unsigned long ret; +unsigned long delta;bool locked = false;struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ;LIST_HEAD (uf_unmap_early);@@ -1067,70 +1261,38 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len,*/ addr =untagged_addr (addr); -if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP )) -return ret; - -if (flags &MREMAP_FIXED && !(flags &MREMAP_MAYMOVE )) -return ret; - -/* -* MREMAP_DONTUNMAP is always a move and it does not allow resizing -* in the process. -*/ -if (flags &MREMAP_DONTUNMAP && - (!(flags &MREMAP_MAYMOVE ) || old_len != new_len)) -return ret; - - -if (offset_in_page (addr)) + ret =check_mremap_params (addr, flags, old_len, new_len, new_addr); +if (ret)return ret; old_len =PAGE_ALIGN (old_len); new_len =PAGE_ALIGN (new_len); + delta =abs_diff (old_len, new_len); -/* -* We allow a zero old-len as a special case -* for DOS-emu "duplicate shm area" thing. But -* a zero new-len is nonsensical. -*/ -if (!new_len) -return ret; - -if (mmap_write_lock_killable (current->mm)) +if (mmap_write_lock_killable (mm))return -EINTR ; + vma =vma_lookup (mm, addr);if (!vma) {
mm/mremap.c SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, · line 1285
Quoted 10 lines
mm/mremap.c +1
-1ret = -EFAULT ; goto out; } -/* Don't allow remapping vmas when they have already been sealed */ +/* If mseal()'d, mremap() is prohibited. */ if (!can_modify_vma (vma)) { ret = -EPERM ; goto out; }
This could be delayed to the munmap() call, which will also check to ensure this would fail.
It doesn't really cost anything though so I don't think it's worth it here. Maybe something we can keep in mind for the future...
Quoted 167 lines
mm/mremap.c +29
-106-if (is_vm_hugetlb_page (vma)) { -struct hstate *h__maybe_unused =hstate_vma (vma); - - old_len =ALIGN (old_len,huge_page_size (h)); - new_len =ALIGN (new_len,huge_page_size (h)); - -/* addrs must be huge page aligned */ -if (addr & ~huge_page_mask (h)) - goto out; -if (new_addr & ~huge_page_mask (h)) - goto out; - -/* -* Don't allow remap expansion, because the underlying hugetlb -* reservation is not yet capable to handle split reservation. -*/ -if (new_len > old_len) - goto out; +/* Align to hugetlb page size, if required. */ +if (is_vm_hugetlb_page (vma) && + !align_hugetlb (vma, addr, new_addr, &old_len, &new_len, &delta)) { + ret = -EINVAL ; + goto out; } -if (flags & (MREMAP_FIXED |MREMAP_DONTUNMAP )) { +/* Are we RELOCATING the VMA to a SPECIFIC address? */ +if (implies_new_addr (flags)) { ret =mremap_to (addr, old_len, new_addr, new_len, &locked, flags, &uf, &uf_unmap_early, &uf_unmap);@@ -1138,109 +1300,44 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, }/* -* Always allow a shrinking remap: that just unmaps -* the unnecessary pages.. -* do_vmi_munmap does all the needed commit accounting, and -* unlocks the mmap_lock if so directed. +* From here on in we are only RESIZING the VMA, attempting to do so +* in-place, moving the VMA if we cannot. */ -if (old_len >= new_len) { -VMA_ITERATOR (vmi, mm, addr + new_len); -if (old_len == new_len) { - ret = addr; - goto out; - } +/* NO-OP CASE - resizing to the same size. */ +if (new_len == old_len) { + ret = addr; + goto out; + } + +/* SHRINK CASE. Can always be done in-place. */ +if (new_len < old_len) { +VMA_ITERATOR (vmi, mm, addr + new_len); - ret =do_vmi_munmap (&vmi, mm, addr + new_len, old_len - new_len, +/* +* Simply unmap the shrunken portion of the VMA. This does all +* the needed commit accounting, unlocking the mmap lock. +*/ + ret =do_vmi_munmap (&vmi, mm, addr + new_len, delta, &uf_unmap, true);if (ret) goto out; +/* We succeeded, mmap lock released for us. */ ret = addr; goto out_unlocked; } -/* -* Ok, we need to grow.. -*/ - ret =resize_is_valid (vma, addr, old_len, new_len, flags); -if (ret) - goto out; - -/* old_len exactly to the end of the area.. -*/ -if (old_len == vma->vm_end - addr) { -unsigned long delta = new_len - old_len; - -/* can we just expand the current mapping? */ -if (vma_expandable (vma, delta)) { -long pages = delta >>PAGE_SHIFT ; -VMA_ITERATOR (vmi, mm, vma->vm_end); -long charged =0 ; - -if (vma->vm_flags &VM_ACCOUNT ) { -if (security_vm_enough_memory_mm (mm, pages)) { - ret = -ENOMEM ; - goto out; - } - charged = pages; - } - -/* -* Function vma_merge_extend() is called on the -* extension we are adding to the already existing vma, -* vma_merge_extend() will merge this extension with the -* already existing vma (expand operation itself) and -* possibly also with the next vma if it becomes -* adjacent to the expanded vma and otherwise -* compatible. -*/ - vma =vma_merge_extend (&vmi, vma, delta); -if (!vma) { -vm_unacct_memory (charged); - ret = -ENOMEM ; - goto out; - } +/* EXPAND case. We try to do in-place, if we can't, then we move it. */ + ret =expand_vma (vma, addr, old_len, new_len, flags, &locked, &new_addr, + &uf, &uf_unmap); -vm_stat_account (mm, vma->vm_flags, pages); -if (vma->vm_flags &VM_LOCKED ) { - mm->locked_vm += pages; - locked = true; - new_addr = addr; - } - ret = addr; - goto out; - } - } - -/* -* We weren't able to just expand or shrink the area, -* we need to create a new one and move it.. -*/ - ret = -ENOMEM ; -if (flags &MREMAP_MAYMOVE ) { -unsigned long map_flags =0 ; -if (vma->vm_flags &VM_MAYSHARE ) - map_flags |=MAP_SHARED ; - - new_addr =get_unmapped_area (vma->vm_file,0 , new_len, - vma->vm_pgoff + - ((addr - vma->vm_start) >>PAGE_SHIFT ), - map_flags); -if (IS_ERR_VALUE (new_addr)) { - ret = new_addr; - goto out; - } - - ret =move_vma (vma, addr, old_len, new_len, new_addr, - &locked, flags, &uf, &uf_unmap); - } out:if (offset_in_page (ret)) locked = false; -mmap_write_unlock (current->mm); +mmap_write_unlock (mm);if (locked && new_len > old_len) -mm_populate (new_addr + old_len, new_len - old_len); +mm_populate (new_addr + old_len, delta); out_unlocked:userfaultfd_unmap_complete (mm, &uf_unmap_early);mremap_userfaultfd_complete (&uf, addr, ret, old_len);Signature · 1 line
2.48.1
Re: [PATCH 2/7] mm/mremap: refactor mremap() system call implementation
Thanks for taking a look! :) I know this one is a bit painful...
Quoted 75 lines · Liam R. Howlett
Quoted 74 lines · Lorenzo Stoakes
Place checks into a separate function so the
mremap()system call is less egregiously long, remove unnecessarymremap_to()offset_in_page()check and just check that earlier so we keep all such basic checks together.Separate out the VMA in-place expansion, hugetlb and expand/move logic into separate, readable functions.
De-duplicate code where possible, add comments and ensure that all error handling explicitly specifies the error at the point of it occurring rather than setting a prefixed error value and implicitly setting (which is bug prone).
This lays the groundwork for subsequent patches further simplifying and extending the
mremap()implementation.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/mremap.c | 405 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 251 insertions(+), 154 deletions(-)mm/mremap.c +12
-21@@ -942,33 +942,14 @@ static unsigned long mremap_to(unsigned long addr, unsigned long old_len,unsigned long ret;unsigned long map_flags =0 ; -if (offset_in_page (new_addr)) -return -EINVAL ; - +/* Is the new length or address silly? */ if (new_len >TASK_SIZE || new_addr >TASK_SIZE - new_len)return -EINVAL ; -/* Ensure the old/new locations do not overlap */ +/* Ensure the old/new locations do not overlap. */ if (addr + old_len > new_addr && new_addr + new_len > addr)return -EINVAL ; -/* -* move_vma() need us to stay 4 maps below the threshold, otherwise -* it will bail out at the very beginning. -* That is a problem if we have already unmaped the regions here -* (new_addr, and old_addr), because userspace will not know the -* state of the vma's after it gets -ENOMEM. -* So, to avoid such scenario we can pre-compute if the whole -* operation has high chances to success map-wise. -* Worst-scenario case is when both vma's (new_addr and old_addr) get -* split in 3 before unmapping it. -* That means 2 more maps (1 for each) to the ones we already hold. -* Check whether current map count plus 2 still leads us to 4 maps below -* the threshold, otherwise return -ENOMEM here to be more safe. -*/ -if ((mm->map_count +2 ) >= sysctl_max_map_count -3 ) -return -ENOMEM ; -if (flags &MREMAP_FIXED ) {/* * In mremap_to(). @@ -1035,6 +1016,218 @@ static int vma_expandable(struct vm_area_struct *vma, unsigned long delta)return 1 ; } +/* Do the mremap() flags require that the new_addr parameter be specified? */ +static bool implies_new_addr (unsigned long flags) +{ +return flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +} + +/* +* Are the parameters passed to mremap() valid? If so return 0, otherwise return +* error. +*/
Quoted 8 lines
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1033
Quoted 5 lines
mm/mremap.c +5
-0+static unsigned long check_mremap_params (unsigned long addr, +unsigned long flags, +unsigned long old_len, +unsigned long new_len, +unsigned long new_addr)If you use two tabs for the arguments this will be more compact and more readable.
Sure, but a later commits eliminates this :)
Quoted 166 lines
Quoted 152 lines
mm/mremap.c +151
-0+{ +/* Ensure no unexpected flag values. */ +if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP )) +return -EINVAL ; + +/* Start address must be page-aligned. */ +if (offset_in_page (addr)) +return -EINVAL ; + +/* +* We allow a zero old-len as a special case +* for DOS-emu "duplicate shm area" thing. But +* a zero new-len is nonsensical. +*/ +if (!PAGE_ALIGN (new_len)) +return -EINVAL ; + +/* Remainder of checks are for cases with specific new_addr. */ +if (!implies_new_addr (flags)) +return 0 ; + +/* The new address must be page-aligned. */ +if (offset_in_page (new_addr)) +return -EINVAL ; + +/* A fixed address implies a move. */ +if (!(flags &MREMAP_MAYMOVE )) +return -EINVAL ; + +/* MREMAP_DONTUNMAP does not allow resizing in the process. */ +if (flags &MREMAP_DONTUNMAP && old_len != new_len) +return -EINVAL ; + +/* +* move_vma() need us to stay 4 maps below the threshold, otherwise +* it will bail out at the very beginning. +* That is a problem if we have already unmaped the regions here +* (new_addr, and old_addr), because userspace will not know the +* state of the vma's after it gets -ENOMEM. +* So, to avoid such scenario we can pre-compute if the whole +* operation has high chances to success map-wise. +* Worst-scenario case is when both vma's (new_addr and old_addr) get +* split in 3 before unmapping it. +* That means 2 more maps (1 for each) to the ones we already hold. +* Check whether current map count plus 2 still leads us to 4 maps below +* the threshold, otherwise return -ENOMEM here to be more safe. +*/ +if ((current->mm->map_count +2 ) >= sysctl_max_map_count -3 ) +return -ENOMEM ; + +return 0 ; +} + +/* +* We know we can expand the VMA in-place by delta pages, so do so. +* +* If we discover the VMA is locked, update mm_struct statistics accordingly and +* indicate so to the caller. +*/ +static unsigned long expand_vma_inplace (struct vm_area_struct *vma, +unsigned long delta,bool *locked) +{ +struct mm_struct *mm = current->mm; +long pages = delta >>PAGE_SHIFT ; +VMA_ITERATOR (vmi, mm, vma->vm_end); +long charged =0 ; + +if (vma->vm_flags &VM_ACCOUNT ) { +if (security_vm_enough_memory_mm (mm, pages)) +return -ENOMEM ; + + charged = pages; + } + +/* +* Function vma_merge_extend() is called on the +* extension we are adding to the already existing vma, +* vma_merge_extend() will merge this extension with the +* already existing vma (expand operation itself) and +* possibly also with the next vma if it becomes +* adjacent to the expanded vma and otherwise +* compatible. +*/ + vma =vma_merge_extend (&vmi, vma, delta); +if (!vma) { +vm_unacct_memory (charged); +return -ENOMEM ; + } + +vm_stat_account (mm, vma->vm_flags, pages); +if (vma->vm_flags &VM_LOCKED ) { + mm->locked_vm += pages; + *locked = true; + } + +return 0 ; +} + +static bool align_hugetlb (struct vm_area_struct *vma, +unsigned long addr, +unsigned long new_addr, +unsigned long *old_len_ptr, +unsigned long *new_len_ptr, +unsigned long *delta_ptr) +{ +unsigned long old_len = *old_len_ptr; +unsigned long new_len = *new_len_ptr; +struct hstate *h__maybe_unused =hstate_vma (vma); + + old_len =ALIGN (old_len,huge_page_size (h)); + new_len =ALIGN (new_len,huge_page_size (h)); + +/* addrs must be huge page aligned */ +if (addr & ~huge_page_mask (h)) +return false; +if (new_addr & ~huge_page_mask (h)) +return false; + +/* +* Don't allow remap expansion, because the underlying hugetlb +* reservation is not yet capable to handle split reservation. +*/ +if (new_len > old_len) +return false; + + *old_len_ptr = old_len; + *new_len_ptr = new_len; + *delta_ptr =abs_diff (old_len, new_len); +return true; +} + +/* +* We are mremap()'ing without specifying a fixed address to move to, but are +* requesting that the VMA's size be increased. +* +* Try to do so in-place, if this fails, then move the VMA to a new location to +* action the change. +*/ +static unsigned long expand_vma (struct vm_area_struct *vma, +unsigned long addr,unsigned long old_len, +unsigned long new_len,unsigned long flags, +bool *locked_ptr,unsigned long *new_addr_ptr, +struct vm_userfaultfd_ctx *uf_ptr, +struct list_head *uf_unmap_ptr) +{ +unsigned long err; +unsigned long map_flags; +unsigned long new_addr;/* We ignore any user-supplied one. */ +pgoff_t pgoff; + + err =resize_is_valid (vma, addr, old_len, new_len, flags);mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1194
Quoted 10 lines
mm/mremap.c +10
-0+if (err) +return err; + +/* +* [addr, old_len) spans precisely to the end of the VMA, so try to +* expand it in-place. +*/ +if (old_len == vma->vm_end - addr && +vma_expandable (vma, new_len - old_len)) { + err =expand_vma_inplace (vma, new_len - old_len, locked_ptr);You use delta twice here (
new_len-old_len). I assume this is different in the next patches.
Quoted 4 lines
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1196
Quoted 2 lines
mm/mremap.c +2
-0+if (IS_ERR_VALUE (err)) +return err;Doesn't
expand_vma_inplace()return 0 on success, error otherwise?
Yeah, this is copying some already dubious logic from the original (trying to _somewhat_ minimise the delta here).
At any rate, a later commit addresses this!
Quoted 13 lines
Quoted 13 lines
mm/mremap.c +12
-0+ +/* +* We want to populate the newly expanded portion of the VMA to +* satisfy the expectation that mlock()'ing a VMA maintains all +* of its pages in memory. +*/ +if (*locked_ptr) + *new_addr_ptr = addr; + +/* OK we're done! */ +return addr; + }
Quoted 8 lines
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1214
Quoted 6 lines
mm/mremap.c +6
-0+ +/* +* We weren't able to just expand or shrink the area, +* we need to create a new one and move it. +*/ +So it's more of an
expand_or_move_vma()?
I think that's misleading, because it would be expand_or_move_and_expand_vma() or expand_in_place_or_move_and_expand()...
Unavoidably we have to abbreviate, I tried to differentiate between the two cases with the 'in place' stuff.
So we _try_ to do it in place, if not we have to expand elsewhere.
Quoted 7 lines
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1217
Quoted 4 lines
mm/mremap.c +3
-0+/* We're not allowed to move the VMA, so error out. */ +if (!(flags &MREMAP_MAYMOVE )) +return -ENOMEM ;and by flags we mean... the flags from the syscall. This gets confusing with
map_flags. At least there's only two and not six flags.
3 flags (MREMAP_FIXED, MREMAP_MAYMOVE, MREMAP_DONTUNMAP) :>)
This becomes clearer later, I'm not sure saying mremap_flags really adds anything but noise though.
Quoted 83 lines
Quoted 69 lines
mm/mremap.c +20
-26+ +/* Find a new location to move the VMA to. */ + map_flags = (vma->vm_flags &VM_MAYSHARE ) ?MAP_SHARED :0 ; + pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >>PAGE_SHIFT ); + new_addr =get_unmapped_area (vma->vm_file,0 , new_len, pgoff, map_flags); +if (IS_ERR_VALUE (new_addr)) +return new_addr; + *new_addr_ptr = new_addr; + +return move_vma (vma, addr, old_len, new_len, new_addr, + locked_ptr, flags, uf_ptr, uf_unmap_ptr); +} +/* * Expand (or shrink) an existing mapping, potentially moving it at the * same time (controlled by the MREMAP_MAYMOVE flag and available VM space) @@ -1048,7 +1241,8 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; -unsigned long ret = -EINVAL ; +unsigned long ret; +unsigned long delta;bool locked = false;struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ;LIST_HEAD (uf_unmap_early);@@ -1067,70 +1261,38 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len,*/ addr =untagged_addr (addr); -if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP )) -return ret; - -if (flags &MREMAP_FIXED && !(flags &MREMAP_MAYMOVE )) -return ret; - -/* -* MREMAP_DONTUNMAP is always a move and it does not allow resizing -* in the process. -*/ -if (flags &MREMAP_DONTUNMAP && - (!(flags &MREMAP_MAYMOVE ) || old_len != new_len)) -return ret; - - -if (offset_in_page (addr)) + ret =check_mremap_params (addr, flags, old_len, new_len, new_addr); +if (ret)return ret; old_len =PAGE_ALIGN (old_len); new_len =PAGE_ALIGN (new_len); + delta =abs_diff (old_len, new_len); -/* -* We allow a zero old-len as a special case -* for DOS-emu "duplicate shm area" thing. But -* a zero new-len is nonsensical. -*/ -if (!new_len) -return ret; - -if (mmap_write_lock_killable (current->mm)) +if (mmap_write_lock_killable (mm))return -EINTR ; + vma =vma_lookup (mm, addr);if (!vma) {mm/mremap.c SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, · line 1285
Quoted 10 lines
mm/mremap.c +1
-1ret = -EFAULT ; goto out; } -/* Don't allow remapping vmas when they have already been sealed */ +/* If mseal()'d, mremap() is prohibited. */ if (!can_modify_vma (vma)) { ret = -EPERM ; goto out; }This could be delayed to the
munmap()call, which will also check to ensure this would fail.
Quoted 2 lines
It doesn't really cost anything though so I don't think it's worth it here. Maybe something we can keep in mind for the future...
Happy to address but I think would be better in a later commit, as this one is more like 'keep things the same but restructure', later commits do deeper changes.
Quoted 168 lines
Quoted 168 lines
mm/mremap.c +29
-107-if (is_vm_hugetlb_page (vma)) { -struct hstate *h__maybe_unused =hstate_vma (vma); - - old_len =ALIGN (old_len,huge_page_size (h)); - new_len =ALIGN (new_len,huge_page_size (h)); - -/* addrs must be huge page aligned */ -if (addr & ~huge_page_mask (h)) - goto out; -if (new_addr & ~huge_page_mask (h)) - goto out; - -/* -* Don't allow remap expansion, because the underlying hugetlb -* reservation is not yet capable to handle split reservation. -*/ -if (new_len > old_len) - goto out; +/* Align to hugetlb page size, if required. */ +if (is_vm_hugetlb_page (vma) && + !align_hugetlb (vma, addr, new_addr, &old_len, &new_len, &delta)) { + ret = -EINVAL ; + goto out; } -if (flags & (MREMAP_FIXED |MREMAP_DONTUNMAP )) { +/* Are we RELOCATING the VMA to a SPECIFIC address? */ +if (implies_new_addr (flags)) { ret =mremap_to (addr, old_len, new_addr, new_len, &locked, flags, &uf, &uf_unmap_early, &uf_unmap);@@ -1138,109 +1300,44 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, }/* -* Always allow a shrinking remap: that just unmaps -* the unnecessary pages.. -* do_vmi_munmap does all the needed commit accounting, and -* unlocks the mmap_lock if so directed. +* From here on in we are only RESIZING the VMA, attempting to do so +* in-place, moving the VMA if we cannot. */ -if (old_len >= new_len) { -VMA_ITERATOR (vmi, mm, addr + new_len); -if (old_len == new_len) { - ret = addr; - goto out; - } +/* NO-OP CASE - resizing to the same size. */ +if (new_len == old_len) { + ret = addr; + goto out; + } + +/* SHRINK CASE. Can always be done in-place. */ +if (new_len < old_len) { +VMA_ITERATOR (vmi, mm, addr + new_len); - ret =do_vmi_munmap (&vmi, mm, addr + new_len, old_len - new_len, +/* +* Simply unmap the shrunken portion of the VMA. This does all +* the needed commit accounting, unlocking the mmap lock. +*/ + ret =do_vmi_munmap (&vmi, mm, addr + new_len, delta, &uf_unmap, true);if (ret) goto out; +/* We succeeded, mmap lock released for us. */ ret = addr; goto out_unlocked; } -/* -* Ok, we need to grow.. -*/ - ret =resize_is_valid (vma, addr, old_len, new_len, flags); -if (ret) - goto out; - -/* old_len exactly to the end of the area.. -*/ -if (old_len == vma->vm_end - addr) { -unsigned long delta = new_len - old_len; - -/* can we just expand the current mapping? */ -if (vma_expandable (vma, delta)) { -long pages = delta >>PAGE_SHIFT ; -VMA_ITERATOR (vmi, mm, vma->vm_end); -long charged =0 ; - -if (vma->vm_flags &VM_ACCOUNT ) { -if (security_vm_enough_memory_mm (mm, pages)) { - ret = -ENOMEM ; - goto out; - } - charged = pages; - } - -/* -* Function vma_merge_extend() is called on the -* extension we are adding to the already existing vma, -* vma_merge_extend() will merge this extension with the -* already existing vma (expand operation itself) and -* possibly also with the next vma if it becomes -* adjacent to the expanded vma and otherwise -* compatible. -*/ - vma =vma_merge_extend (&vmi, vma, delta); -if (!vma) { -vm_unacct_memory (charged); - ret = -ENOMEM ; - goto out; - } +/* EXPAND case. We try to do in-place, if we can't, then we move it. */ + ret =expand_vma (vma, addr, old_len, new_len, flags, &locked, &new_addr, + &uf, &uf_unmap); -vm_stat_account (mm, vma->vm_flags, pages); -if (vma->vm_flags &VM_LOCKED ) { - mm->locked_vm += pages; - locked = true; - new_addr = addr; - } - ret = addr; - goto out; - } - } - -/* -* We weren't able to just expand or shrink the area, -* we need to create a new one and move it.. -*/ - ret = -ENOMEM ; -if (flags &MREMAP_MAYMOVE ) { -unsigned long map_flags =0 ; -if (vma->vm_flags &VM_MAYSHARE ) - map_flags |=MAP_SHARED ; - - new_addr =get_unmapped_area (vma->vm_file,0 , new_len, - vma->vm_pgoff + - ((addr - vma->vm_start) >>PAGE_SHIFT ), - map_flags); -if (IS_ERR_VALUE (new_addr)) { - ret = new_addr; - goto out; - } - - ret =move_vma (vma, addr, old_len, new_len, new_addr, - &locked, flags, &uf, &uf_unmap); - } out:if (offset_in_page (ret)) locked = false; -mmap_write_unlock (current->mm); +mmap_write_unlock (mm);if (locked && new_len > old_len) -mm_populate (new_addr + old_len, new_len - old_len); +mm_populate (new_addr + old_len, delta); out_unlocked:userfaultfd_unmap_complete (mm, &uf_unmap_early);mremap_userfaultfd_complete (&uf, addr, ret, old_len); --2.48.1
Re: [PATCH 2/7] mm/mremap: refactor mremap() system call implementation
Quoted 87 lines · Lorenzo Stoakes
Thanks for taking a look! :) I know this one is a bit painful...
Quoted 75 lines · Liam R. Howlett
Quoted 74 lines · Lorenzo Stoakes
Place checks into a separate function so the
mremap()system call is less egregiously long, remove unnecessarymremap_to()offset_in_page()check and just check that earlier so we keep all such basic checks together.Separate out the VMA in-place expansion, hugetlb and expand/move logic into separate, readable functions.
De-duplicate code where possible, add comments and ensure that all error handling explicitly specifies the error at the point of it occurring rather than setting a prefixed error value and implicitly setting (which is bug prone).
This lays the groundwork for subsequent patches further simplifying and extending the
mremap()implementation.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/mremap.c | 405 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 251 insertions(+), 154 deletions(-)mm/mremap.c +12
-21@@ -942,33 +942,14 @@ static unsigned long mremap_to(unsigned long addr, unsigned long old_len,unsigned long ret;unsigned long map_flags =0 ; -if (offset_in_page (new_addr)) -return -EINVAL ; - +/* Is the new length or address silly? */ if (new_len >TASK_SIZE || new_addr >TASK_SIZE - new_len)return -EINVAL ; -/* Ensure the old/new locations do not overlap */ +/* Ensure the old/new locations do not overlap. */ if (addr + old_len > new_addr && new_addr + new_len > addr)return -EINVAL ; -/* -* move_vma() need us to stay 4 maps below the threshold, otherwise -* it will bail out at the very beginning. -* That is a problem if we have already unmaped the regions here -* (new_addr, and old_addr), because userspace will not know the -* state of the vma's after it gets -ENOMEM. -* So, to avoid such scenario we can pre-compute if the whole -* operation has high chances to success map-wise. -* Worst-scenario case is when both vma's (new_addr and old_addr) get -* split in 3 before unmapping it. -* That means 2 more maps (1 for each) to the ones we already hold. -* Check whether current map count plus 2 still leads us to 4 maps below -* the threshold, otherwise return -ENOMEM here to be more safe. -*/ -if ((mm->map_count +2 ) >= sysctl_max_map_count -3 ) -return -ENOMEM ; -if (flags &MREMAP_FIXED ) {/* * In mremap_to(). @@ -1035,6 +1016,218 @@ static int vma_expandable(struct vm_area_struct *vma, unsigned long delta)return 1 ; } +/* Do the mremap() flags require that the new_addr parameter be specified? */ +static bool implies_new_addr (unsigned long flags) +{ +return flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +} + +/* +* Are the parameters passed to mremap() valid? If so return 0, otherwise return +* error. +*/ Quoted 8 lines
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1033
Quoted 5 lines
mm/mremap.c +5
-0+static unsigned long check_mremap_params (unsigned long addr, +unsigned long flags, +unsigned long old_len, +unsigned long new_len, +unsigned long new_addr)If you use two tabs for the arguments this will be more compact and more readable.
Quoted 1 line
Sure, but a later commits eliminates this :)
Perfect.
Quoted 172 lines
Quoted 167 lines
Quoted 153 lines
mm/mremap.c +151
-0+{ +/* Ensure no unexpected flag values. */ +if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP )) +return -EINVAL ; + +/* Start address must be page-aligned. */ +if (offset_in_page (addr)) +return -EINVAL ; + +/* +* We allow a zero old-len as a special case +* for DOS-emu "duplicate shm area" thing. But +* a zero new-len is nonsensical. +*/ +if (!PAGE_ALIGN (new_len)) +return -EINVAL ; + +/* Remainder of checks are for cases with specific new_addr. */ +if (!implies_new_addr (flags)) +return 0 ; + +/* The new address must be page-aligned. */ +if (offset_in_page (new_addr)) +return -EINVAL ; + +/* A fixed address implies a move. */ +if (!(flags &MREMAP_MAYMOVE )) +return -EINVAL ; + +/* MREMAP_DONTUNMAP does not allow resizing in the process. */ +if (flags &MREMAP_DONTUNMAP && old_len != new_len) +return -EINVAL ; + +/* +* move_vma() need us to stay 4 maps below the threshold, otherwise +* it will bail out at the very beginning. +* That is a problem if we have already unmaped the regions here +* (new_addr, and old_addr), because userspace will not know the +* state of the vma's after it gets -ENOMEM. +* So, to avoid such scenario we can pre-compute if the whole +* operation has high chances to success map-wise. +* Worst-scenario case is when both vma's (new_addr and old_addr) get +* split in 3 before unmapping it. +* That means 2 more maps (1 for each) to the ones we already hold. +* Check whether current map count plus 2 still leads us to 4 maps below +* the threshold, otherwise return -ENOMEM here to be more safe. +*/ +if ((current->mm->map_count +2 ) >= sysctl_max_map_count -3 ) +return -ENOMEM ; + +return 0 ; +} + +/* +* We know we can expand the VMA in-place by delta pages, so do so. +* +* If we discover the VMA is locked, update mm_struct statistics accordingly and +* indicate so to the caller. +*/ +static unsigned long expand_vma_inplace (struct vm_area_struct *vma, +unsigned long delta,bool *locked) +{ +struct mm_struct *mm = current->mm; +long pages = delta >>PAGE_SHIFT ; +VMA_ITERATOR (vmi, mm, vma->vm_end); +long charged =0 ; + +if (vma->vm_flags &VM_ACCOUNT ) { +if (security_vm_enough_memory_mm (mm, pages)) +return -ENOMEM ; + + charged = pages; + } + +/* +* Function vma_merge_extend() is called on the +* extension we are adding to the already existing vma, +* vma_merge_extend() will merge this extension with the +* already existing vma (expand operation itself) and +* possibly also with the next vma if it becomes +* adjacent to the expanded vma and otherwise +* compatible. +*/ + vma =vma_merge_extend (&vmi, vma, delta); +if (!vma) { +vm_unacct_memory (charged); +return -ENOMEM ; + } + +vm_stat_account (mm, vma->vm_flags, pages); +if (vma->vm_flags &VM_LOCKED ) { + mm->locked_vm += pages; + *locked = true; + } + +return 0 ; +} + +static bool align_hugetlb (struct vm_area_struct *vma, +unsigned long addr, +unsigned long new_addr, +unsigned long *old_len_ptr, +unsigned long *new_len_ptr, +unsigned long *delta_ptr) +{ +unsigned long old_len = *old_len_ptr; +unsigned long new_len = *new_len_ptr; +struct hstate *h__maybe_unused =hstate_vma (vma); + + old_len =ALIGN (old_len,huge_page_size (h)); + new_len =ALIGN (new_len,huge_page_size (h)); + +/* addrs must be huge page aligned */ +if (addr & ~huge_page_mask (h)) +return false; +if (new_addr & ~huge_page_mask (h)) +return false; + +/* +* Don't allow remap expansion, because the underlying hugetlb +* reservation is not yet capable to handle split reservation. +*/ +if (new_len > old_len) +return false; + + *old_len_ptr = old_len; + *new_len_ptr = new_len; + *delta_ptr =abs_diff (old_len, new_len); +return true; +} + +/* +* We are mremap()'ing without specifying a fixed address to move to, but are +* requesting that the VMA's size be increased. +* +* Try to do so in-place, if this fails, then move the VMA to a new location to +* action the change. +*/ +static unsigned long expand_vma (struct vm_area_struct *vma, +unsigned long addr,unsigned long old_len, +unsigned long new_len,unsigned long flags, +bool *locked_ptr,unsigned long *new_addr_ptr, +struct vm_userfaultfd_ctx *uf_ptr, +struct list_head *uf_unmap_ptr) +{ +unsigned long err; +unsigned long map_flags; +unsigned long new_addr;/* We ignore any user-supplied one. */ +pgoff_t pgoff; + + err =resize_is_valid (vma, addr, old_len, new_len, flags);mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1194
Quoted 10 lines
mm/mremap.c +10
-0+if (err) +return err; + +/* +* [addr, old_len) spans precisely to the end of the VMA, so try to +* expand it in-place. +*/ +if (old_len == vma->vm_end - addr && +vma_expandable (vma, new_len - old_len)) { + err =expand_vma_inplace (vma, new_len - old_len, locked_ptr);You use delta twice here (
new_len-old_len). I assume this is different in the next patches.Quoted 4 lines
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1196
Quoted 2 lines
mm/mremap.c +2
-0+if (IS_ERR_VALUE (err)) +return err;Doesn't
expand_vma_inplace()return 0 on success, error otherwise?
Quoted 4 lines
Yeah, this is copying some already dubious logic from the original (trying to _somewhat_ minimise the delta here).
At any rate, a later commit addresses this!
Thanks.
Quoted 23 lines
Quoted 14 lines
Quoted 14 lines
mm/mremap.c +12
-0+ +/* +* We want to populate the newly expanded portion of the VMA to +* satisfy the expectation that mlock()'ing a VMA maintains all +* of its pages in memory. +*/ +if (*locked_ptr) + *new_addr_ptr = addr; + +/* OK we're done! */ +return addr; + }Quoted 8 lines
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1214
Quoted 6 lines
mm/mremap.c +6
-0+ +/* +* We weren't able to just expand or shrink the area, +* we need to create a new one and move it. +*/ +So it's more of an
expand_or_move_vma()?
Quoted 7 lines
I think that's misleading, because it would be
expand_or_move_and_expand_vma()orexpand_in_place_or_move_and_expand()...Unavoidably we have to abbreviate, I tried to differentiate between the two cases with the 'in place' stuff.
So we _try_ to do it in place, if not we have to expand elsewhere.
Fair enough.
Quoted 9 lines
Quoted 8 lines
mm/mremap.c static int vma_expandable(struct vm_area_struct *vma, unsigned long delta) · line 1217
Quoted 5 lines
mm/mremap.c +3
-0+/* We're not allowed to move the VMA, so error out. */ +if (!(flags &MREMAP_MAYMOVE )) +return -ENOMEM ;and by flags we mean... the flags from the syscall. This gets confusing with
map_flags. At least there's only two and not six flags.
Quoted 4 lines
3 flags (
MREMAP_FIXED,MREMAP_MAYMOVE,MREMAP_DONTUNMAP) :>)This becomes clearer later, I'm not sure saying
mremap_flagsreally adds anything but noise though.
Okay.
Quoted 84 lines
Quoted 84 lines
Quoted 70 lines
mm/mremap.c +20
-26+ +/* Find a new location to move the VMA to. */ + map_flags = (vma->vm_flags &VM_MAYSHARE ) ?MAP_SHARED :0 ; + pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >>PAGE_SHIFT ); + new_addr =get_unmapped_area (vma->vm_file,0 , new_len, pgoff, map_flags); +if (IS_ERR_VALUE (new_addr)) +return new_addr; + *new_addr_ptr = new_addr; + +return move_vma (vma, addr, old_len, new_len, new_addr, + locked_ptr, flags, uf_ptr, uf_unmap_ptr); +} +/* * Expand (or shrink) an existing mapping, potentially moving it at the * same time (controlled by the MREMAP_MAYMOVE flag and available VM space) @@ -1048,7 +1241,8 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, {struct mm_struct *mm = current->mm;struct vm_area_struct *vma; -unsigned long ret = -EINVAL ; +unsigned long ret; +unsigned long delta;bool locked = false;struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ;LIST_HEAD (uf_unmap_early);@@ -1067,70 +1261,38 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len,*/ addr =untagged_addr (addr); -if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP )) -return ret; - -if (flags &MREMAP_FIXED && !(flags &MREMAP_MAYMOVE )) -return ret; - -/* -* MREMAP_DONTUNMAP is always a move and it does not allow resizing -* in the process. -*/ -if (flags &MREMAP_DONTUNMAP && - (!(flags &MREMAP_MAYMOVE ) || old_len != new_len)) -return ret; - - -if (offset_in_page (addr)) + ret =check_mremap_params (addr, flags, old_len, new_len, new_addr); +if (ret)return ret; old_len =PAGE_ALIGN (old_len); new_len =PAGE_ALIGN (new_len); + delta =abs_diff (old_len, new_len); -/* -* We allow a zero old-len as a special case -* for DOS-emu "duplicate shm area" thing. But -* a zero new-len is nonsensical. -*/ -if (!new_len) -return ret; - -if (mmap_write_lock_killable (current->mm)) +if (mmap_write_lock_killable (mm))return -EINTR ; + vma =vma_lookup (mm, addr);if (!vma) {mm/mremap.c SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, · line 1285
Quoted 10 lines
mm/mremap.c +1
-1ret = -EFAULT ; goto out; } -/* Don't allow remapping vmas when they have already been sealed */ +/* If mseal()'d, mremap() is prohibited. */ if (!can_modify_vma (vma)) { ret = -EPERM ; goto out; }This could be delayed to the
munmap()call, which will also check to ensure this would fail.
Quoted 6 lines
Quoted 2 lines
It doesn't really cost anything though so I don't think it's worth it here. Maybe something we can keep in mind for the future...
Happy to address but I think would be better in a later commit, as this one is more like 'keep things the same but restructure', later commits do deeper changes.
Right, yes. Me too.
...
Thanks,
Liam
Re: [PATCH 2/7] mm/mremap: refactor mremap() system call implementation
In reply to Lorenzo StoakesQuoted 12 lines · Lorenzo Stoakes
Place checks into a separate function so the
mremap()system call is less egregiously long, remove unnecessarymremap_to()offset_in_page()check and just check that earlier so we keep all such basic checks together.Separate out the VMA in-place expansion, hugetlb and expand/move logic into separate, readable functions.
De-duplicate code where possible, add comments and ensure that all error handling explicitly specifies the error at the point of it occurring rather than setting a prefixed error value and implicitly setting (which is bug prone).
Quoted 5 lines
This lays the groundwork for subsequent patches further simplifying and extending the
mremap()implementation.Signed-off-by Lorenzo Stoakes <[email protected]>---
Nice refactoring—reviewing it was a nice learning experience.
Looks good to me,
Signature · 2 lines
Cheers, Harry
Quoted 2 lines
mm/mremap.c | 405 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 251 insertions(+), 154 deletions(-)
Re: [PATCH 2/7] mm/mremap: refactor mremap() system call implementation
Quoted 16 lines · Harry Yoo
Quoted 15 lines · Lorenzo Stoakes
Place checks into a separate function so the
mremap()system call is less egregiously long, remove unnecessarymremap_to()offset_in_page()check and just check that earlier so we keep all such basic checks together.Separate out the VMA in-place expansion, hugetlb and expand/move logic into separate, readable functions.
De-duplicate code where possible, add comments and ensure that all error handling explicitly specifies the error at the point of it occurring rather than setting a prefixed error value and implicitly setting (which is bug prone).
This lays the groundwork for subsequent patches further simplifying and extending the
mremap()implementation.
Quoted 7 lines
Quoted 2 lines
Signed-off-by Lorenzo Stoakes <[email protected]>---
Nice refactoring—reviewing it was a nice learning experience.
Looks good to me,
Reviewed-by Harry Yoo <[email protected]>
Thanks!
Quoted 7 lines
--
Cheers,
HarryQuoted 2 lines
mm/mremap.c | 405 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 251 insertions(+), 154 deletions(-)
PATCH 3/7 mm/mremap: introduce and use vma_remap_struct threaded state
In reply to Lorenzo StoakesA number of mremap() calls both pass around and modify a large number of parameters, making the code less readable and often repeatedly having to determine things such as VMA, size delta, and more.
Avoid this by using the common pattern of passing a state object through the operation, updating it as we go. We introduce the vma_remap_struct or 'VRM' for this purpose.
This also gives us the ability to accumulate further state through the operation that would otherwise require awkward and error-prone pointer passing.
We can also now trivially define helper functions that operate on a VRM object.
This pattern has proven itself to be very powerful when implemented for VMA merge, VMA unmapping and memory mapping operations, so it is battle-tested and functional.
We both introduce the data structure and use it, introducing helper functions as needed to make things readable, we move some state such as mmap lock and mlock() status to the VRM, we introduce a means of classifying the type of mremap() operation and de-duplicate the get_unmapped_area() lookup.
We also neatly thread userfaultfd state throughout the operation.
Note that there is further refactoring to be done, chiefly adjust move_vma() to accept a VRM parameter. We defer this as there is pre-requisite work required to be able to do so which we will do in a subsequent patch.
--- mm/mremap.c | 559 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 354 insertions(+), 205 deletions(-)
mm/mremap.c +354 -205
@@ -32,6 +32,43 @@#include "internal.h" +/* Classify the kind of remap operation being performed. */ +enum mremap_operation { +MREMAP_NO_RESIZE ,/* old_len == new_len, if not moved, do nothing. */ +MREMAP_SHRINK ,/* old_len > new_len. */ +MREMAP_EXPAND ,/* old_len < new_len. */ +}; + +/* +* Describes a VMA mremap() operation and is threaded throughout it. +* +* Any of the fields may be mutated by the operation, however these values will +* always accurately reflect the remap (for instance, we may adjust lengths and +* delta to account for hugetlb alignment). +*/ +struct vma_remap_struct { +/* User-provided state. */ +unsigned long addr;/* User-specified address from which we remap. */ +unsigned long old_len;/* Length of range being remapped. */ +unsigned long new_len;/* Desired new length of mapping. */ +unsigned long flags;/* user-specified MREMAP_* flags. */ +unsigned long new_addr;/* Optionally, desired new address. */ + +/* uffd state. */ +struct vm_userfaultfd_ctx *uf; +struct list_head *uf_unmap_early; +struct list_head *uf_unmap; + +/* VMA state, determined in do_mremap(). */ +struct vm_area_struct *vma; + +/* Internal state, determined in do_mremap(). */ +unsigned long delta;/* Absolute delta of old_len, new_len. */ +bool locked;/* Was the VMA mlock()'d (has the VM_LOCKED flag set). */ +enum mremap_operation remap_type;/* expand, shrink, etc. */ +bool mmap_locked;/* Is current->mm currently write-locked? */ +}; +static pud_t *get_old_pud (struct mm_struct *mm,unsigned long addr) {pgd_t *pgd;
@@ -693,6 +730,97 @@ unsigned long move_page_tables(struct vm_area_struct *vma,return len + old_addr - old_end;/* how much done */ } +/* Set vrm->delta to the difference in VMA size specified by user. */ +static void vrm_set_delta (struct vma_remap_struct *vrm) +{ + vrm->delta =abs_diff (vrm->old_len, vrm->new_len); +} + +/* Determine what kind of remap this is - shrink, expand or no resize at all. */ +static enum mremap_operation vrm_remap_type (struct vma_remap_struct *vrm) +{ +if (vrm->delta ==0 ) +return MREMAP_NO_RESIZE ; + +if (vrm->old_len > vrm->new_len) +return MREMAP_SHRINK ; + +return MREMAP_EXPAND ; +} + +/* Set the vrm->remap_type, assumes state is sufficient set up for this. */ +static void vrm_set_remap_type (struct vma_remap_struct *vrm) +{ + vrm->remap_type =vrm_remap_type (vrm); +} + +/* +* When moving a VMA to vrm->new_adr, does this result in the new and old VMAs +* overlapping? +*/ +static bool vrm_overlaps (struct vma_remap_struct *vrm) +{ +unsigned long start_old = vrm->addr; +unsigned long start_new = vrm->new_addr; +unsigned long end_old = vrm->addr + vrm->old_len; +unsigned long end_new = vrm->new_addr + vrm->new_len; + +/* +* start_old end_old +* |-----------| +* | | +* |-----------| +* |-------------| +* | | +* |-------------| +* start_new end_new +*/ +if (end_old > start_new && end_new > start_old) +return true; + +return false; +} + +/* Do the mremap() flags require that the new_addr parameter be specified? */ +static bool vrm_implies_new_addr (struct vma_remap_struct *vrm) +{ +return vrm->flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +} + +/* +* Find an unmapped area for the requested vrm->new_addr. +* +* If MREMAP_FIXED then this is equivalent to a MAP_FIXED mmap() call. If only +* MREMAP_DONTUNMAP is set, then this is equivalent to providing a hint to +* mmap(), otherwise this is equivalent to mmap() specifying a NULL address. +* +* Returns 0 on success (with vrm->new_addr updated), or an error code upon +* failure. +*/ +static unsigned long vrm_set_new_addr (struct vma_remap_struct *vrm) +{ +struct vm_area_struct *vma = vrm->vma; +unsigned long map_flags =0 ; +/* Page Offset _into_ the VMA. */ +pgoff_t internal_pgoff = (vrm->addr - vma->vm_start) >>PAGE_SHIFT ; +pgoff_t pgoff = vma->vm_pgoff + internal_pgoff; +unsigned long new_addr =vrm_implies_new_addr (vrm) ? vrm->new_addr :0 ; +unsigned long res; + +if (vrm->flags &MREMAP_FIXED ) + map_flags |=MAP_FIXED ; +if (vma->vm_flags &VM_MAYSHARE ) + map_flags |=MAP_SHARED ; + + res =get_unmapped_area (vma->vm_file, new_addr, vrm->new_len, pgoff, + map_flags); +if (IS_ERR_VALUE (res)) +return res; + + vrm->new_addr = res; +return 0 ; +} + static unsigned long move_vma(struct vm_area_struct *vma, unsigned long old_addr, unsigned long old_len, unsigned long new_len, unsigned long new_addr,
@@ -860,18 +988,15 @@ static unsigned long move_vma(struct vm_area_struct *vma,* resize_is_valid() - Ensure the vma can be resized to the new length at the give * address. * -* @vma: The vma to resize -* @addr: The old address -* @old_len: The current size -* @new_len: The desired size -* @flags: The vma flags -* * Return 0 on success, error otherwise. */ -static int resize_is_valid (struct vm_area_struct *vma,unsigned long addr, -unsigned long old_len,unsigned long new_len,unsigned long flags) +static int resize_is_valid (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; +struct vm_area_struct *vma = vrm->vma; +unsigned long addr = vrm->addr; +unsigned long old_len = vrm->old_len; +unsigned long new_len = vrm->new_len;unsigned long pgoff;/*
@@ -883,11 +1008,12 @@ static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr,* behavior. As a result, fail such attempts. */ if (!old_len && !(vma->vm_flags & (VM_SHARED |VM_MAYSHARE ))) { -pr_warn_once ("%s (%d): attempted to duplicate a private mapping with mremap. This is not supported.\n" , current->comm, current->pid); +pr_warn_once ("%s (%d): attempted to duplicate a private mapping with mremap. This is not supported.\n" , + current->comm, current->pid);return -EINVAL ; } -if ((flags &MREMAP_DONTUNMAP ) && +if ((vrm->flags &MREMAP_DONTUNMAP ) && (vma->vm_flags & (VM_DONTEXPAND |VM_PFNMAP )))return -EINVAL ;
@@ -907,99 +1033,114 @@ static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr,if (vma->vm_flags & (VM_DONTEXPAND |VM_PFNMAP ))return -EFAULT ; -if (!mlock_future_ok (mm, vma->vm_flags, new_len - old_len)) +if (!mlock_future_ok (mm, vma->vm_flags, vrm->delta))return -EAGAIN ; -if (!may_expand_vm (mm, vma->vm_flags, - (new_len - old_len) >>PAGE_SHIFT )) +if (!may_expand_vm (mm, vma->vm_flags, vrm->delta >>PAGE_SHIFT ))return -ENOMEM ;return 0 ; }/* -* mremap_to() - remap a vma to a new location -* @addr: The old address -* @old_len: The old size -* @new_addr: The target address -* @new_len: The new size -* @locked: If the returned vma is locked (VM_LOCKED) -* @flags: the mremap flags -* @uf: The mremap userfaultfd context -* @uf_unmap_early: The userfaultfd unmap early context -* @uf_unmap: The userfaultfd unmap context +* The user has requested that the VMA be shrunk (i.e., old_len > new_len), so +* execute this, optionally dropping the mmap lock when we do so. * +* In both cases this invalidates the VMA, however if we don't drop the lock, +* then load the correct VMA into vrm->vma afterwards. +*/ +static unsigned long shrink_vma (struct vma_remap_struct *vrm, +bool drop_lock) +{ +struct mm_struct *mm = current->mm; +unsigned long unmap_start = vrm->addr + vrm->new_len; +unsigned long unmap_bytes = vrm->delta; +unsigned long res; +VMA_ITERATOR (vmi, mm, unmap_start); + +VM_BUG_ON (vrm->remap_type !=MREMAP_SHRINK ); + + res =do_vmi_munmap (&vmi, mm, unmap_start, unmap_bytes, + vrm->uf_unmap, drop_lock); + vrm->vma =NULL ;/* Invalidated. */ +if (res) +return res; + +/* +* If we've not dropped the lock, then we should reload the VMA to +* replace the invalidated VMA with the one that may have now been +* split. +*/ +if (drop_lock) + vrm->mmap_locked = false; +else + vrm->vma =vma_lookup (mm, vrm->addr); + +return 0 ; +} + +/* +* mremap_to() - remap a vma to a new location. * Returns: The new address of the vma or an error. */ -static unsigned long mremap_to (unsigned long addr,unsigned long old_len, -unsigned long new_addr,unsigned long new_len,bool *locked, -unsigned long flags,struct vm_userfaultfd_ctx *uf, -struct list_head *uf_unmap_early, -struct list_head *uf_unmap) +static unsigned long mremap_to (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; -struct vm_area_struct *vma; -unsigned long ret; -unsigned long map_flags =0 ; +unsigned long err;/* Is the new length or address silly? */ -if (new_len >TASK_SIZE || new_addr >TASK_SIZE - new_len) +if (vrm->new_len >TASK_SIZE || + vrm->new_addr >TASK_SIZE - vrm->new_len)return -EINVAL ; -/* Ensure the old/new locations do not overlap. */ -if (addr + old_len > new_addr && new_addr + new_len > addr) +if (vrm_overlaps (vrm))return -EINVAL ; -if (flags &MREMAP_FIXED ) { +if (vrm->flags &MREMAP_FIXED ) {/* * In mremap_to(). * VMA is moved to dst address, and munmap dst first. * do_munmap will check if dst is sealed. */ - ret =do_munmap (mm, new_addr, new_len, uf_unmap_early); -if (ret) -return ret; - } + err =do_munmap (mm, vrm->new_addr, vrm->new_len, + vrm->uf_unmap_early); + vrm->vma =NULL ;/* Invalidated. */ +if (err) +return err; -if (old_len > new_len) { - ret =do_munmap (mm, addr+new_len, old_len - new_len, uf_unmap); -if (ret) -return ret; - old_len = new_len; +/* +* If we remap a portion of a VMA elsewhere in the same VMA, +* this can invalidate the old VMA and iterator. Reset. +*/ + vrm->vma =vma_lookup (mm, vrm->addr); } - vma =vma_lookup (mm, addr); -if (!vma) -return -EFAULT ; +if (vrm->remap_type ==MREMAP_SHRINK ) { + err =shrink_vma (vrm,/* drop_lock= */ false); +if (err) +return err; - ret =resize_is_valid (vma, addr, old_len, new_len, flags); -if (ret) -return ret; +/* Set up for the move now shrink has been executed. */ + vrm->old_len = vrm->new_len; + } + + err =resize_is_valid (vrm); +if (err) +return err;/* MREMAP_DONTUNMAP expands by old_len since old_len == new_len */ -if (flags &MREMAP_DONTUNMAP && - !may_expand_vm (mm, vma->vm_flags, old_len >>PAGE_SHIFT )) { +if (vrm->flags &MREMAP_DONTUNMAP && + !may_expand_vm (mm, vrm->vma->vm_flags, vrm->old_len >>PAGE_SHIFT )) {return -ENOMEM ; } -if (flags &MREMAP_FIXED ) - map_flags |=MAP_FIXED ; - -if (vma->vm_flags &VM_MAYSHARE ) - map_flags |=MAP_SHARED ; - - ret =get_unmapped_area (vma->vm_file, new_addr, new_len, vma->vm_pgoff + - ((addr - vma->vm_start) >>PAGE_SHIFT ), - map_flags); -if (IS_ERR_VALUE (ret)) -return ret; - -/* We got a new mapping */ -if (!(flags &MREMAP_FIXED )) - new_addr = ret; + err =vrm_set_new_addr (vrm); +if (err) +return err; -return move_vma (vma, addr, old_len, new_len, new_addr, locked, flags, - uf, uf_unmap); +return move_vma (vrm->vma, vrm->addr, vrm->old_len, vrm->new_len, + vrm->new_addr, &vrm->locked, vrm->flags, + vrm->uf, vrm->uf_unmap); }static int vma_expandable (struct vm_area_struct *vma,unsigned long delta)
@@ -1016,22 +1157,33 @@ static int vma_expandable(struct vm_area_struct *vma, unsigned long delta)return 1 ; } -/* Do the mremap() flags require that the new_addr parameter be specified? */ -static bool implies_new_addr (unsigned long flags) +/* Determine whether we are actually able to execute an in-place expansion. */ +static bool vrm_can_expand_in_place (struct vma_remap_struct *vrm) { -return flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +/* Number of bytes from vrm->addr to end of VMA. */ +unsigned long suffix_bytes = vrm->vma->vm_end - vrm->addr; + +/* If end of range aligns to end of VMA, we can just expand in-place. */ +if (suffix_bytes != vrm->old_len) +return false; + +/* Check whether this is feasible. */ +if (!vma_expandable (vrm->vma, vrm->delta)) +return false; + +return true; }/* * Are the parameters passed to mremap() valid? If so return 0, otherwise return * error. */ -static unsigned long check_mremap_params (unsigned long addr, -unsigned long flags, -unsigned long old_len, -unsigned long new_len, -unsigned long new_addr) +static unsigned long check_mremap_params (struct vma_remap_struct *vrm) + { +unsigned long addr = vrm->addr; +unsigned long flags = vrm->flags; +/* Ensure no unexpected flag values. */ if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP ))return -EINVAL ;
@@ -1045,15 +1197,15 @@ static unsigned long check_mremap_params(unsigned long addr,* for DOS-emu "duplicate shm area" thing. But * a zero new-len is nonsensical. */ -if (!PAGE_ALIGN (new_len)) +if (!PAGE_ALIGN (vrm->new_len))return -EINVAL ;/* Remainder of checks are for cases with specific new_addr. */ -if (!implies_new_addr (flags)) +if (!vrm_implies_new_addr (vrm))return 0 ;/* The new address must be page-aligned. */ -if (offset_in_page (new_addr)) +if (offset_in_page (vrm->new_addr))return -EINVAL ;/* A fixed address implies a move. */
@@ -1061,7 +1213,7 @@ static unsigned long check_mremap_params(unsigned long addr,return -EINVAL ;/* MREMAP_DONTUNMAP does not allow resizing in the process. */ -if (flags &MREMAP_DONTUNMAP && old_len != new_len) +if (flags &MREMAP_DONTUNMAP && vrm->old_len != vrm->new_len)return -EINVAL ;/*
@@ -1090,11 +1242,11 @@ static unsigned long check_mremap_params(unsigned long addr,* If we discover the VMA is locked, update mm_struct statistics accordingly and * indicate so to the caller. */ -static unsigned long expand_vma_inplace (struct vm_area_struct *vma, -unsigned long delta,bool *locked) +static unsigned long expand_vma_in_place (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; -long pages = delta >>PAGE_SHIFT ; +long pages = vrm->delta >>PAGE_SHIFT ; +struct vm_area_struct *vma = vrm->vma;VMA_ITERATOR (vmi, mm, vma->vm_end);long charged =0 ;
@@ -1114,7 +1266,7 @@ static unsigned long expand_vma_inplace(struct vm_area_struct *vma,* adjacent to the expanded vma and otherwise * compatible. */ - vma =vma_merge_extend (&vmi, vma, delta); + vma = vrm->vma =vma_merge_extend (&vmi, vma, vrm->delta);if (!vma) {vm_unacct_memory (charged);return -ENOMEM ;
@@ -1123,42 +1275,34 @@ static unsigned long expand_vma_inplace(struct vm_area_struct *vma,vm_stat_account (mm, vma->vm_flags, pages);if (vma->vm_flags &VM_LOCKED ) { mm->locked_vm += pages; - *locked = true; + vrm->locked = true; }return 0 ; } -static bool align_hugetlb (struct vm_area_struct *vma, -unsigned long addr, -unsigned long new_addr, -unsigned long *old_len_ptr, -unsigned long *new_len_ptr, -unsigned long *delta_ptr) +static bool align_hugetlb (struct vma_remap_struct *vrm) { -unsigned long old_len = *old_len_ptr; -unsigned long new_len = *new_len_ptr; -struct hstate *h__maybe_unused =hstate_vma (vma); +struct hstate *h__maybe_unused =hstate_vma (vrm->vma); - old_len =ALIGN (old_len,huge_page_size (h)); - new_len =ALIGN (new_len,huge_page_size (h)); + vrm->old_len =ALIGN (vrm->old_len,huge_page_size (h)); + vrm->new_len =ALIGN (vrm->new_len,huge_page_size (h));/* addrs must be huge page aligned */ -if (addr & ~huge_page_mask (h)) +if (vrm->addr & ~huge_page_mask (h))return false; -if (new_addr & ~huge_page_mask (h)) +if (vrm->new_addr & ~huge_page_mask (h))return false;/* * Don't allow remap expansion, because the underlying hugetlb * reservation is not yet capable to handle split reservation. */ -if (new_len > old_len) +if (vrm->new_len > vrm->old_len)return false; - *old_len_ptr = old_len; - *new_len_ptr = new_len; - *delta_ptr =abs_diff (old_len, new_len); +vrm_set_delta (vrm); +return true; }
@@ -1169,19 +1313,16 @@ static bool align_hugetlb(struct vm_area_struct *vma,* Try to do so in-place, if this fails, then move the VMA to a new location to * action the change. */ -static unsigned long expand_vma (struct vm_area_struct *vma, -unsigned long addr,unsigned long old_len, -unsigned long new_len,unsigned long flags, -bool *locked_ptr,unsigned long *new_addr_ptr, -struct vm_userfaultfd_ctx *uf_ptr, -struct list_head *uf_unmap_ptr) +static unsigned long expand_vma (struct vma_remap_struct *vrm) {unsigned long err; -unsigned long map_flags; -unsigned long new_addr;/* We ignore any user-supplied one. */ -pgoff_t pgoff; +struct vm_area_struct *vma = vrm->vma; +unsigned long addr = vrm->addr; +unsigned long old_len = vrm->old_len; +unsigned long new_len = vrm->new_len; +unsigned long flags = vrm->flags; - err =resize_is_valid (vma, addr, old_len, new_len, flags); + err =resize_is_valid (vrm);if (err)return err;
@@ -1189,10 +1330,9 @@ static unsigned long expand_vma(struct vm_area_struct *vma,* [addr, old_len) spans precisely to the end of the VMA, so try to * expand it in-place. */ -if (old_len == vma->vm_end - addr && -vma_expandable (vma, new_len - old_len)) { - err =expand_vma_inplace (vma, new_len - old_len, locked_ptr); -if (IS_ERR_VALUE (err)) +if (vrm_can_expand_in_place (vrm)) { + err =expand_vma_in_place (vrm); +if (err)return err;/*
@@ -1200,8 +1340,8 @@ static unsigned long expand_vma(struct vm_area_struct *vma,* satisfy the expectation that mlock()'ing a VMA maintains all * of its pages in memory. */ -if (*locked_ptr) - *new_addr_ptr = addr; +if (vrm->locked) + vrm->new_addr = addr;/* OK we're done! */ return addr;
@@ -1217,62 +1357,65 @@ static unsigned long expand_vma(struct vm_area_struct *vma,return -ENOMEM ;/* Find a new location to move the VMA to. */ - map_flags = (vma->vm_flags &VM_MAYSHARE ) ?MAP_SHARED :0 ; - pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >>PAGE_SHIFT ); - new_addr =get_unmapped_area (vma->vm_file,0 , new_len, pgoff, map_flags); -if (IS_ERR_VALUE (new_addr)) -return new_addr; - *new_addr_ptr = new_addr; + err =vrm_set_new_addr (vrm); +if (err) +return err; -return move_vma (vma, addr, old_len, new_len, new_addr, - locked_ptr, flags, uf_ptr, uf_unmap_ptr); +return move_vma (vma, addr, old_len, new_len, vrm->new_addr, + &vrm->locked, flags, vrm->uf, vrm->uf_unmap); }/* -* Expand (or shrink) an existing mapping, potentially moving it at the -* same time (controlled by the MREMAP_MAYMOVE flag and available VM space) -* -* MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise -* This option implies MREMAP_MAYMOVE. +* Attempt to resize the VMA in-place, if we cannot, then move the VMA to the +* first available address to perform the operation. */ -SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, - unsigned long, new_len, unsigned long, flags, - unsigned long, new_addr) +static unsigned long mremap_at (struct vma_remap_struct *vrm) +{ +unsigned long res; + +switch (vrm->remap_type) { +case MREMAP_NO_RESIZE : +/* NO-OP CASE - resizing to the same size. */ +return vrm->addr; +case MREMAP_SHRINK : +/* +* SHRINK CASE. Can always be done in-place. +* +* Simply unmap the shrunken portion of the VMA. This does all +* the needed commit accounting, and we indicate that the mmap +* lock should be dropped. +*/ + res =shrink_vma (vrm,/* drop_lock= */ true); +if (res) +return res; + +return vrm->addr; +case MREMAP_EXPAND : +return expand_vma (vrm); + } + +BUG (); +} + +static unsigned long do_mremap (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma;unsigned long ret; -unsigned long delta; -bool locked = false; -struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ; -LIST_HEAD (uf_unmap_early); -LIST_HEAD (uf_unmap); -/* -* There is a deliberate asymmetry here: we strip the pointer tag -* from the old address but leave the new address alone. This is -* for consistency with mmap(), where we prevent the creation of -* aliasing mappings in userspace by leaving the tag bits of the -* mapping address intact. A non-zero tag will cause the subsequent -* range checks to reject the address as invalid. -* -* See Documentation/arch/arm64/tagged-address-abi.rst for more -* information. -*/ - addr =untagged_addr (addr); - - ret =check_mremap_params (addr, flags, old_len, new_len, new_addr); + ret =check_mremap_params (vrm);if (ret)return ret; - old_len =PAGE_ALIGN (old_len); - new_len =PAGE_ALIGN (new_len); - delta =abs_diff (old_len, new_len); + vrm->old_len =PAGE_ALIGN (vrm->old_len); + vrm->new_len =PAGE_ALIGN (vrm->new_len); +vrm_set_delta (vrm);if (mmap_write_lock_killable (mm))return -EINTR ; + vrm->mmap_locked = true; - vma =vma_lookup (mm, addr); + vma = vrm->vma =vma_lookup (mm, vrm->addr);if (!vma) { ret = -EFAULT ; goto out;
@@ -1285,62 +1428,68 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, }/* Align to hugetlb page size, if required. */ -if (is_vm_hugetlb_page (vma) && - !align_hugetlb (vma, addr, new_addr, &old_len, &new_len, &delta)) { +if (is_vm_hugetlb_page (vma) && !align_hugetlb (vrm)) { ret = -EINVAL ; goto out; } -/* Are we RELOCATING the VMA to a SPECIFIC address? */ -if (implies_new_addr (flags)) { - ret =mremap_to (addr, old_len, new_addr, new_len, - &locked, flags, &uf, &uf_unmap_early, - &uf_unmap); - goto out; - } +vrm_set_remap_type (vrm); -/* -* From here on in we are only RESIZING the VMA, attempting to do so -* in-place, moving the VMA if we cannot. -*/ +/* Actually execute mremap. */ + ret =vrm_implies_new_addr (vrm) ?mremap_to (vrm) :mremap_at (vrm); -/* NO-OP CASE - resizing to the same size. */ -if (new_len == old_len) { - ret = addr; - goto out; - } - -/* SHRINK CASE. Can always be done in-place. */ -if (new_len < old_len) { -VMA_ITERATOR (vmi, mm, addr + new_len); +out: +if (vrm->mmap_locked) { +mmap_write_unlock (mm); + vrm->mmap_locked = false; -/* -* Simply unmap the shrunken portion of the VMA. This does all -* the needed commit accounting, unlocking the mmap lock. -*/ - ret =do_vmi_munmap (&vmi, mm, addr + new_len, delta, - &uf_unmap, true); -if (ret) - goto out; - -/* We succeeded, mmap lock released for us. */ - ret = addr; - goto out_unlocked; +if (!offset_in_page (ret) && vrm->locked && vrm->new_len > vrm->old_len) +mm_populate (vrm->new_addr + vrm->old_len, vrm->delta); } -/* EXPAND case. We try to do in-place, if we can't, then we move it. */ - ret =expand_vma (vma, addr, old_len, new_len, flags, &locked, &new_addr, - &uf, &uf_unmap); +userfaultfd_unmap_complete (mm, vrm->uf_unmap_early); +mremap_userfaultfd_complete (vrm->uf, vrm->addr, ret, vrm->old_len); +userfaultfd_unmap_complete (mm, vrm->uf_unmap); -out: -if (offset_in_page (ret)) - locked = false; -mmap_write_unlock (mm); -if (locked && new_len > old_len) -mm_populate (new_addr + old_len, delta); -out_unlocked: -userfaultfd_unmap_complete (mm, &uf_unmap_early); -mremap_userfaultfd_complete (&uf, addr, ret, old_len); -userfaultfd_unmap_complete (mm, &uf_unmap);return ret; } + +/* +* Expand (or shrink) an existing mapping, potentially moving it at the +* same time (controlled by the MREMAP_MAYMOVE flag and available VM space) +* +* MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise +* This option implies MREMAP_MAYMOVE. +*/ +SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, + unsigned long, new_len, unsigned long, flags, + unsigned long, new_addr) +{ +struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ; +LIST_HEAD (uf_unmap_early); +LIST_HEAD (uf_unmap); +/* +* There is a deliberate asymmetry here: we strip the pointer tag +* from the old address but leave the new address alone. This is +* for consistency with mmap(), where we prevent the creation of +* aliasing mappings in userspace by leaving the tag bits of the +* mapping address intact. A non-zero tag will cause the subsequent +* range checks to reject the address as invalid. +* +* See Documentation/arch/arm64/tagged-address-abi.rst for more +* information. +*/ +struct vma_remap_struct vrm = { + .addr =untagged_addr (addr), + .old_len = old_len, + .new_len = new_len, + .flags = flags, + .new_addr = new_addr, + + .uf = &uf, + .uf_unmap_early = &uf_unmap_early, + .uf_unmap = &uf_unmap, + }; + +return do_mremap (&vrm); +}
Signature · 1 line
2.48.1
Re: [PATCH 3/7] mm/mremap: introduce and use vma_remap_struct threaded state
Quoted 42 lines · Lorenzo Stoakes
A number of
mremap()calls both pass around and modify a large number of parameters, making the code less readable and often repeatedly having to determine things such as VMA, size delta, and more.Avoid this by using the common pattern of passing a state object through the operation, updating it as we go. We introduce the
vma_remap_structor 'VRM' for this purpose.This also gives us the ability to accumulate further state through the operation that would otherwise require awkward and error-prone pointer passing.
We can also now trivially define helper functions that operate on a VRM object.
This pattern has proven itself to be very powerful when implemented for VMA merge, VMA unmapping and memory mapping operations, so it is battle-tested and functional.
We both introduce the data structure and use it, introducing helper functions as needed to make things readable, we move some state such as mmap lock and
mlock()status to the VRM, we introduce a means of classifying the type ofmremap()operation and de-duplicate theget_unmapped_area()lookup.We also neatly thread userfaultfd state throughout the operation.
Note that there is further refactoring to be done, chiefly adjust
move_vma()to accept a VRM parameter. We defer this as there is pre-requisite work required to be able to do so which we will do in a subsequent patch.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/mremap.c | 559 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 354 insertions(+), 205 deletions(-)mm/mremap.c +0
-0
mm/mremap.c line 39
Quoted 8 lines
mm/mremap.c +5
-0@@ -32,6 +32,43 @@#include "internal.h" +/* Classify the kind of remap operation being performed. */ +enum mremap_operation { +MREMAP_NO_RESIZE ,/* old_len == new_len, if not moved, do nothing. */ +MREMAP_SHRINK ,/* old_len > new_len. */ +MREMAP_EXPAND ,/* old_len < new_len. */
Can we fix the spacing so the comments line up, please?
It might be worth having a MREMAP_INVALID here and init to that, then a VM_BUG_ON(), but maybe I'm just paranoid.
Quoted 6 lines
mm/mremap.c +6
-0+}; + +/* +* Describes a VMA mremap() operation and is threaded throughout it. +* +* Any of the fields may be mutated by the operation, however these values will
mm/mremap.c line 55
Quoted 10 lines
mm/mremap.c +10
-0+* always accurately reflect the remap (for instance, we may adjust lengths and +* delta to account for hugetlb alignment). +*/ +struct vma_remap_struct { +/* User-provided state. */ +unsigned long addr;/* User-specified address from which we remap. */ +unsigned long old_len;/* Length of range being remapped. */ +unsigned long new_len;/* Desired new length of mapping. */ +unsigned long flags;/* user-specified MREMAP_* flags. */ +unsigned long new_addr;/* Optionally, desired new address. */
Same comment about the comment spacing here. Might be better to have user_flags? Since we have the passed in flags, the map flags and the vma flags.
mm/mremap.c line 60
Quoted 5 lines
mm/mremap.c +5
-0+ +/* uffd state. */ +struct vm_userfaultfd_ctx *uf; +struct list_head *uf_unmap_early; +struct list_head *uf_unmap;
... sigh, yeah.
mm/mremap.c line 67
Quoted 7 lines
mm/mremap.c +7
-0+ +/* VMA state, determined in do_mremap(). */ +struct vm_area_struct *vma; + +/* Internal state, determined in do_mremap(). */ +unsigned long delta;/* Absolute delta of old_len, new_len. */ +bool locked;/* Was the VMA mlock()'d (has the VM_LOCKED flag set). */
bool mlocked ?
Quoted 23 lines
mm/mremap.c +16
-0+enum mremap_operation remap_type;/* expand, shrink, etc. */ +bool mmap_locked;/* Is current->mm currently write-locked? */ +}; +static pud_t *get_old_pud (struct mm_struct *mm,unsigned long addr) {pgd_t *pgd;@@ -693,6 +730,97 @@ unsigned long move_page_tables(struct vm_area_struct *vma,return len + old_addr - old_end;/* how much done */ } +/* Set vrm->delta to the difference in VMA size specified by user. */ +static void vrm_set_delta (struct vma_remap_struct *vrm) +{ + vrm->delta =abs_diff (vrm->old_len, vrm->new_len); +} + +/* Determine what kind of remap this is - shrink, expand or no resize at all. */ +static enum mremap_operation vrm_remap_type (struct vma_remap_struct *vrm) +{ +if (vrm->delta ==0 ) +return MREMAP_NO_RESIZE ; +
mm/mremap.c unsigned long move_page_tables(struct vm_area_struct *vma, · line 754
Quoted 10 lines
mm/mremap.c +10
-0+if (vrm->old_len > vrm->new_len) +return MREMAP_SHRINK ; + +return MREMAP_EXPAND ; +} + +/* Set the vrm->remap_type, assumes state is sufficient set up for this. */ +static void vrm_set_remap_type (struct vma_remap_struct *vrm) +{ + vrm->remap_type =vrm_remap_type (vrm);
The vrm_remap_type() function is only used once, maybe we don't need both set and get?
Quoted 24 lines
mm/mremap.c +24
-0+} + +/* +* When moving a VMA to vrm->new_adr, does this result in the new and old VMAs +* overlapping? +*/ +static bool vrm_overlaps (struct vma_remap_struct *vrm) +{ +unsigned long start_old = vrm->addr; +unsigned long start_new = vrm->new_addr; +unsigned long end_old = vrm->addr + vrm->old_len; +unsigned long end_new = vrm->new_addr + vrm->new_len; + +/* +* start_old end_old +* |-----------| +* | | +* |-----------| +* |-------------| +* | | +* |-------------| +* start_new end_new +*/ +if (end_old > start_new && end_new > start_old)
mm/mremap.c unsigned long move_page_tables(struct vm_area_struct *vma, · line 788
Quoted 10 lines
mm/mremap.c +10
-0+return true; + +return false; +} + +/* Do the mremap() flags require that the new_addr parameter be specified? */ +static bool vrm_implies_new_addr (struct vma_remap_struct *vrm) +{ +return vrm->flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +}
These five might benefit from being inlined (although I hope our compiler is good enough for this).
Quoted 185 lines
mm/mremap.c +93
-39+ +/* +* Find an unmapped area for the requested vrm->new_addr. +* +* If MREMAP_FIXED then this is equivalent to a MAP_FIXED mmap() call. If only +* MREMAP_DONTUNMAP is set, then this is equivalent to providing a hint to +* mmap(), otherwise this is equivalent to mmap() specifying a NULL address. +* +* Returns 0 on success (with vrm->new_addr updated), or an error code upon +* failure. +*/ +static unsigned long vrm_set_new_addr (struct vma_remap_struct *vrm) +{ +struct vm_area_struct *vma = vrm->vma; +unsigned long map_flags =0 ; +/* Page Offset _into_ the VMA. */ +pgoff_t internal_pgoff = (vrm->addr - vma->vm_start) >>PAGE_SHIFT ; +pgoff_t pgoff = vma->vm_pgoff + internal_pgoff; +unsigned long new_addr =vrm_implies_new_addr (vrm) ? vrm->new_addr :0 ; +unsigned long res; + +if (vrm->flags &MREMAP_FIXED ) + map_flags |=MAP_FIXED ; +if (vma->vm_flags &VM_MAYSHARE ) + map_flags |=MAP_SHARED ; + + res =get_unmapped_area (vma->vm_file, new_addr, vrm->new_len, pgoff, + map_flags); +if (IS_ERR_VALUE (res)) +return res; + + vrm->new_addr = res; +return 0 ; +} + static unsigned long move_vma(struct vm_area_struct *vma, unsigned long old_addr, unsigned long old_len, unsigned long new_len, unsigned long new_addr,@@ -860,18 +988,15 @@ static unsigned long move_vma(struct vm_area_struct *vma,* resize_is_valid() - Ensure the vma can be resized to the new length at the give * address. * -* @vma: The vma to resize -* @addr: The old address -* @old_len: The current size -* @new_len: The desired size -* @flags: The vma flags -* * Return 0 on success, error otherwise. */ -static int resize_is_valid (struct vm_area_struct *vma,unsigned long addr, -unsigned long old_len,unsigned long new_len,unsigned long flags) +static int resize_is_valid (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; +struct vm_area_struct *vma = vrm->vma; +unsigned long addr = vrm->addr; +unsigned long old_len = vrm->old_len; +unsigned long new_len = vrm->new_len;unsigned long pgoff;/* @@ -883,11 +1008,12 @@ static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr,* behavior. As a result, fail such attempts. */ if (!old_len && !(vma->vm_flags & (VM_SHARED |VM_MAYSHARE ))) { -pr_warn_once ("%s (%d): attempted to duplicate a private mapping with mremap. This is not supported.\n" , current->comm, current->pid); +pr_warn_once ("%s (%d): attempted to duplicate a private mapping with mremap. This is not supported.\n" , + current->comm, current->pid);return -EINVAL ; } -if ((flags &MREMAP_DONTUNMAP ) && +if ((vrm->flags &MREMAP_DONTUNMAP ) && (vma->vm_flags & (VM_DONTEXPAND |VM_PFNMAP )))return -EINVAL ;@@ -907,99 +1033,114 @@ static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr,if (vma->vm_flags & (VM_DONTEXPAND |VM_PFNMAP ))return -EFAULT ; -if (!mlock_future_ok (mm, vma->vm_flags, new_len - old_len)) +if (!mlock_future_ok (mm, vma->vm_flags, vrm->delta))return -EAGAIN ; -if (!may_expand_vm (mm, vma->vm_flags, - (new_len - old_len) >>PAGE_SHIFT )) +if (!may_expand_vm (mm, vma->vm_flags, vrm->delta >>PAGE_SHIFT ))return -ENOMEM ;return 0 ; }/* -* mremap_to() - remap a vma to a new location -* @addr: The old address -* @old_len: The old size -* @new_addr: The target address -* @new_len: The new size -* @locked: If the returned vma is locked (VM_LOCKED) -* @flags: the mremap flags -* @uf: The mremap userfaultfd context -* @uf_unmap_early: The userfaultfd unmap early context -* @uf_unmap: The userfaultfd unmap context +* The user has requested that the VMA be shrunk (i.e., old_len > new_len), so +* execute this, optionally dropping the mmap lock when we do so. * +* In both cases this invalidates the VMA, however if we don't drop the lock, +* then load the correct VMA into vrm->vma afterwards. +*/ +static unsigned long shrink_vma (struct vma_remap_struct *vrm, +bool drop_lock) +{ +struct mm_struct *mm = current->mm; +unsigned long unmap_start = vrm->addr + vrm->new_len; +unsigned long unmap_bytes = vrm->delta; +unsigned long res; +VMA_ITERATOR (vmi, mm, unmap_start); + +VM_BUG_ON (vrm->remap_type !=MREMAP_SHRINK ); + + res =do_vmi_munmap (&vmi, mm, unmap_start, unmap_bytes, + vrm->uf_unmap, drop_lock); + vrm->vma =NULL ;/* Invalidated. */ +if (res) +return res; + +/* +* If we've not dropped the lock, then we should reload the VMA to +* replace the invalidated VMA with the one that may have now been +* split. +*/ +if (drop_lock) + vrm->mmap_locked = false; +else + vrm->vma =vma_lookup (mm, vrm->addr); + +return 0 ; +} + +/* +* mremap_to() - remap a vma to a new location. * Returns: The new address of the vma or an error. */ -static unsigned long mremap_to (unsigned long addr,unsigned long old_len, -unsigned long new_addr,unsigned long new_len,bool *locked, -unsigned long flags,struct vm_userfaultfd_ctx *uf, -struct list_head *uf_unmap_early, -struct list_head *uf_unmap) +static unsigned long mremap_to (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; -struct vm_area_struct *vma; -unsigned long ret; -unsigned long map_flags =0 ; +unsigned long err;/* Is the new length or address silly? */ -if (new_len >TASK_SIZE || new_addr >TASK_SIZE - new_len) +if (vrm->new_len >TASK_SIZE || + vrm->new_addr >TASK_SIZE - vrm->new_len)return -EINVAL ; -/* Ensure the old/new locations do not overlap. */ -if (addr + old_len > new_addr && new_addr + new_len > addr) +if (vrm_overlaps (vrm))return -EINVAL ; -if (flags &MREMAP_FIXED ) { +if (vrm->flags &MREMAP_FIXED ) {/* * In mremap_to(). * VMA is moved to dst address, and munmap dst first. * do_munmap will check if dst is sealed. */ - ret =do_munmap (mm, new_addr, new_len, uf_unmap_early); -if (ret) -return ret; - } + err =do_munmap (mm, vrm->new_addr, vrm->new_len, + vrm->uf_unmap_early); + vrm->vma =NULL ;/* Invalidated. */ +if (err) +return err;
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1115
Quoted 10 lines
mm/mremap.c +5
-5-if (old_len > new_len) { - ret =do_munmap (mm, addr+new_len, old_len - new_len, uf_unmap); -if (ret) -return ret; - old_len = new_len; +/* +* If we remap a portion of a VMA elsewhere in the same VMA, +* this can invalidate the old VMA and iterator. Reset. +*/ + vrm->vma =vma_lookup (mm, vrm->addr);
You say it invalidates the iterator, but this doesn't change an iterator?
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1119
Quoted 7 lines
mm/mremap.c +2
-3} - vma =vma_lookup (mm, addr); -if (!vma) -return -EFAULT ; +if (vrm->remap_type ==MREMAP_SHRINK ) { + err =shrink_vma (vrm,/* drop_lock= */ false);
It is not immediately clear if we could have a MREMAP_FIXED also MREMAP_SHRINK. In that case, we would try to do_munmap() twice. This shouldn't be an issue as do_vmi_munmap() would catch it, but I am not sure if you noticed this.
Quoted 10 lines
mm/mremap.c +6
-3+if (err) +return err; - ret =resize_is_valid (vma, addr, old_len, new_len, flags); -if (ret) -return ret; +/* Set up for the move now shrink has been executed. */ + vrm->old_len = vrm->new_len; + } +
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1134
Quoted 10 lines
mm/mremap.c +5
-2+ err =resize_is_valid (vrm); +if (err) +return err;/* MREMAP_DONTUNMAP expands by old_len since old_len == new_len */ -if (flags &MREMAP_DONTUNMAP && - !may_expand_vm (mm, vma->vm_flags, old_len >>PAGE_SHIFT )) { +if (vrm->flags &MREMAP_DONTUNMAP && + !may_expand_vm (mm, vrm->vma->vm_flags, vrm->old_len >>PAGE_SHIFT )) {return -ENOMEM ;
nit: whitespace here is a bit odd to read.
Quoted 16 lines
mm/mremap.c +0
-14} -if (flags &MREMAP_FIXED ) - map_flags |=MAP_FIXED ; - -if (vma->vm_flags &VM_MAYSHARE ) - map_flags |=MAP_SHARED ; - - ret =get_unmapped_area (vma->vm_file, new_addr, new_len, vma->vm_pgoff + - ((addr - vma->vm_start) >>PAGE_SHIFT ), - map_flags); -if (IS_ERR_VALUE (ret)) -return ret; - -/* We got a new mapping */ -if (!(flags &MREMAP_FIXED ))
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1143
Quoted 10 lines
mm/mremap.c +6
-3- new_addr = ret; + err =vrm_set_new_addr (vrm); +if (err) +return err; -return move_vma (vma, addr, old_len, new_len, new_addr, locked, flags, - uf, uf_unmap); +return move_vma (vrm->vma, vrm->addr, vrm->old_len, vrm->new_len, + vrm->new_addr, &vrm->locked, vrm->flags, + vrm->uf, vrm->uf_unmap);
I see where this is going..
Quoted 225 lines
mm/mremap.c +54
-60}static int vma_expandable (struct vm_area_struct *vma,unsigned long delta)@@ -1016,22 +1157,33 @@ static int vma_expandable(struct vm_area_struct *vma, unsigned long delta)return 1 ; } -/* Do the mremap() flags require that the new_addr parameter be specified? */ -static bool implies_new_addr (unsigned long flags) +/* Determine whether we are actually able to execute an in-place expansion. */ +static bool vrm_can_expand_in_place (struct vma_remap_struct *vrm) { -return flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +/* Number of bytes from vrm->addr to end of VMA. */ +unsigned long suffix_bytes = vrm->vma->vm_end - vrm->addr; + +/* If end of range aligns to end of VMA, we can just expand in-place. */ +if (suffix_bytes != vrm->old_len) +return false; + +/* Check whether this is feasible. */ +if (!vma_expandable (vrm->vma, vrm->delta)) +return false; + +return true; }/* * Are the parameters passed to mremap() valid? If so return 0, otherwise return * error. */ -static unsigned long check_mremap_params (unsigned long addr, -unsigned long flags, -unsigned long old_len, -unsigned long new_len, -unsigned long new_addr) +static unsigned long check_mremap_params (struct vma_remap_struct *vrm) + { +unsigned long addr = vrm->addr; +unsigned long flags = vrm->flags; +/* Ensure no unexpected flag values. */ if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP ))return -EINVAL ;@@ -1045,15 +1197,15 @@ static unsigned long check_mremap_params(unsigned long addr,* for DOS-emu "duplicate shm area" thing. But * a zero new-len is nonsensical. */ -if (!PAGE_ALIGN (new_len)) +if (!PAGE_ALIGN (vrm->new_len))return -EINVAL ;/* Remainder of checks are for cases with specific new_addr. */ -if (!implies_new_addr (flags)) +if (!vrm_implies_new_addr (vrm))return 0 ;/* The new address must be page-aligned. */ -if (offset_in_page (new_addr)) +if (offset_in_page (vrm->new_addr))return -EINVAL ;/* A fixed address implies a move. */ @@ -1061,7 +1213,7 @@ static unsigned long check_mremap_params(unsigned long addr,return -EINVAL ;/* MREMAP_DONTUNMAP does not allow resizing in the process. */ -if (flags &MREMAP_DONTUNMAP && old_len != new_len) +if (flags &MREMAP_DONTUNMAP && vrm->old_len != vrm->new_len)return -EINVAL ;/* @@ -1090,11 +1242,11 @@ static unsigned long check_mremap_params(unsigned long addr,* If we discover the VMA is locked, update mm_struct statistics accordingly and * indicate so to the caller. */ -static unsigned long expand_vma_inplace (struct vm_area_struct *vma, -unsigned long delta,bool *locked) +static unsigned long expand_vma_in_place (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; -long pages = delta >>PAGE_SHIFT ; +long pages = vrm->delta >>PAGE_SHIFT ; +struct vm_area_struct *vma = vrm->vma;VMA_ITERATOR (vmi, mm, vma->vm_end);long charged =0 ;@@ -1114,7 +1266,7 @@ static unsigned long expand_vma_inplace(struct vm_area_struct *vma,* adjacent to the expanded vma and otherwise * compatible. */ - vma =vma_merge_extend (&vmi, vma, delta); + vma = vrm->vma =vma_merge_extend (&vmi, vma, vrm->delta);if (!vma) {vm_unacct_memory (charged);return -ENOMEM ;@@ -1123,42 +1275,34 @@ static unsigned long expand_vma_inplace(struct vm_area_struct *vma,vm_stat_account (mm, vma->vm_flags, pages);if (vma->vm_flags &VM_LOCKED ) { mm->locked_vm += pages; - *locked = true; + vrm->locked = true; }return 0 ; } -static bool align_hugetlb (struct vm_area_struct *vma, -unsigned long addr, -unsigned long new_addr, -unsigned long *old_len_ptr, -unsigned long *new_len_ptr, -unsigned long *delta_ptr) +static bool align_hugetlb (struct vma_remap_struct *vrm) { -unsigned long old_len = *old_len_ptr; -unsigned long new_len = *new_len_ptr; -struct hstate *h__maybe_unused =hstate_vma (vma); +struct hstate *h__maybe_unused =hstate_vma (vrm->vma); - old_len =ALIGN (old_len,huge_page_size (h)); - new_len =ALIGN (new_len,huge_page_size (h)); + vrm->old_len =ALIGN (vrm->old_len,huge_page_size (h)); + vrm->new_len =ALIGN (vrm->new_len,huge_page_size (h));/* addrs must be huge page aligned */ -if (addr & ~huge_page_mask (h)) +if (vrm->addr & ~huge_page_mask (h))return false; -if (new_addr & ~huge_page_mask (h)) +if (vrm->new_addr & ~huge_page_mask (h))return false;/* * Don't allow remap expansion, because the underlying hugetlb * reservation is not yet capable to handle split reservation. */ -if (new_len > old_len) +if (vrm->new_len > vrm->old_len)return false; - *old_len_ptr = old_len; - *new_len_ptr = new_len; - *delta_ptr =abs_diff (old_len, new_len); +vrm_set_delta (vrm); +return true; }@@ -1169,19 +1313,16 @@ static bool align_hugetlb(struct vm_area_struct *vma,* Try to do so in-place, if this fails, then move the VMA to a new location to * action the change. */ -static unsigned long expand_vma (struct vm_area_struct *vma, -unsigned long addr,unsigned long old_len, -unsigned long new_len,unsigned long flags, -bool *locked_ptr,unsigned long *new_addr_ptr, -struct vm_userfaultfd_ctx *uf_ptr, -struct list_head *uf_unmap_ptr) +static unsigned long expand_vma (struct vma_remap_struct *vrm) {unsigned long err; -unsigned long map_flags; -unsigned long new_addr;/* We ignore any user-supplied one. */ -pgoff_t pgoff; +struct vm_area_struct *vma = vrm->vma; +unsigned long addr = vrm->addr; +unsigned long old_len = vrm->old_len; +unsigned long new_len = vrm->new_len; +unsigned long flags = vrm->flags; - err =resize_is_valid (vma, addr, old_len, new_len, flags); + err =resize_is_valid (vrm);if (err)return err;@@ -1189,10 +1330,9 @@ static unsigned long expand_vma(struct vm_area_struct *vma,* [addr, old_len) spans precisely to the end of the VMA, so try to * expand it in-place. */ -if (old_len == vma->vm_end - addr && -vma_expandable (vma, new_len - old_len)) { - err =expand_vma_inplace (vma, new_len - old_len, locked_ptr); -if (IS_ERR_VALUE (err)) +if (vrm_can_expand_in_place (vrm)) { + err =expand_vma_in_place (vrm); +if (err)return err;/* @@ -1200,8 +1340,8 @@ static unsigned long expand_vma(struct vm_area_struct *vma,* satisfy the expectation that mlock()'ing a VMA maintains all * of its pages in memory. */ -if (*locked_ptr) - *new_addr_ptr = addr; +if (vrm->locked) + vrm->new_addr = addr;/* OK we're done! */ return addr;@@ -1217,62 +1357,65 @@ static unsigned long expand_vma(struct vm_area_struct *vma,return -ENOMEM ;/* Find a new location to move the VMA to. */ - map_flags = (vma->vm_flags &VM_MAYSHARE ) ?MAP_SHARED :0 ; - pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >>PAGE_SHIFT ); - new_addr =get_unmapped_area (vma->vm_file,0 , new_len, pgoff, map_flags); -if (IS_ERR_VALUE (new_addr)) -return new_addr; - *new_addr_ptr = new_addr; + err =vrm_set_new_addr (vrm); +if (err) +return err; -return move_vma (vma, addr, old_len, new_len, new_addr, - locked_ptr, flags, uf_ptr, uf_unmap_ptr); +return move_vma (vma, addr, old_len, new_len, vrm->new_addr, + &vrm->locked, flags, vrm->uf, vrm->uf_unmap); }/* -* Expand (or shrink) an existing mapping, potentially moving it at the -* same time (controlled by the MREMAP_MAYMOVE flag and available VM space)
mm/mremap.c static unsigned long expand_vma(struct vm_area_struct *vma, · line 1372
Quoted 10 lines
mm/mremap.c +3
-6-* -* MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise -* This option implies MREMAP_MAYMOVE. +* Attempt to resize the VMA in-place, if we cannot, then move the VMA to the +* first available address to perform the operation. */ -SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, - unsigned long, new_len, unsigned long, flags, - unsigned long, new_addr) +static unsigned long mremap_at (struct vma_remap_struct *vrm)
I hate this and mremap_to() names. I don't have a proposed better name for either and maybe it's just my experience with mremap_to() that has tainted the view of the name itself, but I find them not very descriptive and abruptly ended. I guess move was already taken.
I also have the added baggage of parsing "at" to potentially mean "doing".
mremap_inplace() seems equally annoying. This is the worst bike shed.
Quoted 117 lines
mm/mremap.c +42
-49+{ +unsigned long res; + +switch (vrm->remap_type) { +case MREMAP_NO_RESIZE : +/* NO-OP CASE - resizing to the same size. */ +return vrm->addr; +case MREMAP_SHRINK : +/* +* SHRINK CASE. Can always be done in-place. +* +* Simply unmap the shrunken portion of the VMA. This does all +* the needed commit accounting, and we indicate that the mmap +* lock should be dropped. +*/ + res =shrink_vma (vrm,/* drop_lock= */ true); +if (res) +return res; + +return vrm->addr; +case MREMAP_EXPAND : +return expand_vma (vrm); + } + +BUG (); +} + +static unsigned long do_mremap (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma;unsigned long ret; -unsigned long delta; -bool locked = false; -struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ; -LIST_HEAD (uf_unmap_early); -LIST_HEAD (uf_unmap); -/* -* There is a deliberate asymmetry here: we strip the pointer tag -* from the old address but leave the new address alone. This is -* for consistency with mmap(), where we prevent the creation of -* aliasing mappings in userspace by leaving the tag bits of the -* mapping address intact. A non-zero tag will cause the subsequent -* range checks to reject the address as invalid. -* -* See Documentation/arch/arm64/tagged-address-abi.rst for more -* information. -*/ - addr =untagged_addr (addr); - - ret =check_mremap_params (addr, flags, old_len, new_len, new_addr); + ret =check_mremap_params (vrm);if (ret)return ret; - old_len =PAGE_ALIGN (old_len); - new_len =PAGE_ALIGN (new_len); - delta =abs_diff (old_len, new_len); + vrm->old_len =PAGE_ALIGN (vrm->old_len); + vrm->new_len =PAGE_ALIGN (vrm->new_len); +vrm_set_delta (vrm);if (mmap_write_lock_killable (mm))return -EINTR ; + vrm->mmap_locked = true; - vma =vma_lookup (mm, addr); + vma = vrm->vma =vma_lookup (mm, vrm->addr);if (!vma) { ret = -EFAULT ; goto out;@@ -1285,62 +1428,68 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, }/* Align to hugetlb page size, if required. */ -if (is_vm_hugetlb_page (vma) && - !align_hugetlb (vma, addr, new_addr, &old_len, &new_len, &delta)) { +if (is_vm_hugetlb_page (vma) && !align_hugetlb (vrm)) { ret = -EINVAL ; goto out; } -/* Are we RELOCATING the VMA to a SPECIFIC address? */ -if (implies_new_addr (flags)) { - ret =mremap_to (addr, old_len, new_addr, new_len, - &locked, flags, &uf, &uf_unmap_early, - &uf_unmap); - goto out; - } +vrm_set_remap_type (vrm); -/* -* From here on in we are only RESIZING the VMA, attempting to do so -* in-place, moving the VMA if we cannot. -*/ +/* Actually execute mremap. */ + ret =vrm_implies_new_addr (vrm) ?mremap_to (vrm) :mremap_at (vrm); -/* NO-OP CASE - resizing to the same size. */ -if (new_len == old_len) { - ret = addr; - goto out; - } - -/* SHRINK CASE. Can always be done in-place. */ -if (new_len < old_len) { -VMA_ITERATOR (vmi, mm, addr + new_len); +out: +if (vrm->mmap_locked) { +mmap_write_unlock (mm); + vrm->mmap_locked = false; -/* -* Simply unmap the shrunken portion of the VMA. This does all -* the needed commit accounting, unlocking the mmap lock. -*/
mm/mremap.c SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, · line 1447
Quoted 10 lines
mm/mremap.c +2
-8- ret =do_vmi_munmap (&vmi, mm, addr + new_len, delta, - &uf_unmap, true); -if (ret) - goto out; - -/* We succeeded, mmap lock released for us. */ - ret = addr; - goto out_unlocked; +if (!offset_in_page (ret) && vrm->locked && vrm->new_len > vrm->old_len) +mm_populate (vrm->new_addr + vrm->old_len, vrm->delta);
It isn't clear to me why we only populate if we are locked here. Actually, I'm not sure why we keep holding the lock until here or why it matters to drop it early. The main reason we want to drop the lock is to reduce the mmap lock time held for the populate operation.
So we can either drop the lock before we get here once the vma tree is updated, or we can unconditionally unlock.
I think we can simplify this further if we just keep the lock held until we downgrade here and populate if necessary after it's dropped. That is, shrink just does nothing with the lock and we just unlock it regardless.
I'm pretty sure that we would struggle to measure the performance impact holding the lock for the return path when the populate is removed from the critical section.
Quoted 64 lines
mm/mremap.c +43
-13} -/* EXPAND case. We try to do in-place, if we can't, then we move it. */ - ret =expand_vma (vma, addr, old_len, new_len, flags, &locked, &new_addr, - &uf, &uf_unmap); +userfaultfd_unmap_complete (mm, vrm->uf_unmap_early); +mremap_userfaultfd_complete (vrm->uf, vrm->addr, ret, vrm->old_len); +userfaultfd_unmap_complete (mm, vrm->uf_unmap); -out: -if (offset_in_page (ret)) - locked = false; -mmap_write_unlock (mm); -if (locked && new_len > old_len) -mm_populate (new_addr + old_len, delta); -out_unlocked: -userfaultfd_unmap_complete (mm, &uf_unmap_early); -mremap_userfaultfd_complete (&uf, addr, ret, old_len); -userfaultfd_unmap_complete (mm, &uf_unmap);return ret; } + +/* +* Expand (or shrink) an existing mapping, potentially moving it at the +* same time (controlled by the MREMAP_MAYMOVE flag and available VM space) +* +* MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise +* This option implies MREMAP_MAYMOVE. +*/ +SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, + unsigned long, new_len, unsigned long, flags, + unsigned long, new_addr) +{ +struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ; +LIST_HEAD (uf_unmap_early); +LIST_HEAD (uf_unmap); +/* +* There is a deliberate asymmetry here: we strip the pointer tag +* from the old address but leave the new address alone. This is +* for consistency with mmap(), where we prevent the creation of +* aliasing mappings in userspace by leaving the tag bits of the +* mapping address intact. A non-zero tag will cause the subsequent +* range checks to reject the address as invalid. +* +* See Documentation/arch/arm64/tagged-address-abi.rst for more +* information. +*/ +struct vma_remap_struct vrm = { + .addr =untagged_addr (addr), + .old_len = old_len, + .new_len = new_len, + .flags = flags, + .new_addr = new_addr, + + .uf = &uf, + .uf_unmap_early = &uf_unmap_early, + .uf_unmap = &uf_unmap, + }; + +return do_mremap (&vrm); +}Signature · 1 line
2.48.1
Re: [PATCH 3/7] mm/mremap: introduce and use vma_remap_struct threaded state
Quoted 46 lines · Liam R. Howlett
Quoted 45 lines · Lorenzo Stoakes
A number of
mremap()calls both pass around and modify a large number of parameters, making the code less readable and often repeatedly having to determine things such as VMA, size delta, and more.Avoid this by using the common pattern of passing a state object through the operation, updating it as we go. We introduce the
vma_remap_structor 'VRM' for this purpose.This also gives us the ability to accumulate further state through the operation that would otherwise require awkward and error-prone pointer passing.
We can also now trivially define helper functions that operate on a VRM object.
This pattern has proven itself to be very powerful when implemented for VMA merge, VMA unmapping and memory mapping operations, so it is battle-tested and functional.
We both introduce the data structure and use it, introducing helper functions as needed to make things readable, we move some state such as mmap lock and
mlock()status to the VRM, we introduce a means of classifying the type ofmremap()operation and de-duplicate theget_unmapped_area()lookup.We also neatly thread userfaultfd state throughout the operation.
Note that there is further refactoring to be done, chiefly adjust
move_vma()to accept a VRM parameter. We defer this as there is pre-requisite work required to be able to do so which we will do in a subsequent patch.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/mremap.c | 559 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 354 insertions(+), 205 deletions(-)mm/mremap.c +0
-0@@ -32,6 +32,43 @@#include "internal.h"
Quoted 7 lines
mm/mremap.c line 39
Quoted 5 lines
mm/mremap.c +5
-0+/* Classify the kind of remap operation being performed. */ +enum mremap_operation { +MREMAP_NO_RESIZE ,/* old_len == new_len, if not moved, do nothing. */ +MREMAP_SHRINK ,/* old_len > new_len. */ +MREMAP_EXPAND ,/* old_len < new_len. */ Can we fix the spacing so the comments line up, please?
I've had review the other way before :) But sure, I mean I actually agree. Perhaps I was following stuff I already saw in the kernel on this, but yeah let's align this!
Quoted 3 lines
It might be worth having a
MREMAP_INVALIDhere and init to that, then aVM_BUG_ON(), but maybe I'm just paranoid.
Sure can do.
Quoted 13 lines
Quoted 13 lines
mm/mremap.c +12
-0+}; + +/* +* Describes a VMA mremap() operation and is threaded throughout it. +* +* Any of the fields may be mutated by the operation, however these values will +* always accurately reflect the remap (for instance, we may adjust lengths and +* delta to account for hugetlb alignment). +*/ +struct vma_remap_struct { +/* User-provided state. */ +unsigned long addr;/* User-specified address from which we remap. */
Quoted 8 lines
mm/mremap.c line 55
Quoted 4 lines
mm/mremap.c +4
-0+unsigned long old_len;/* Length of range being remapped. */ +unsigned long new_len;/* Desired new length of mapping. */ +unsigned long flags;/* user-specified MREMAP_* flags. */ +unsigned long new_addr;/* Optionally, desired new address. */ Same comment about the comment spacing here. Might be better to have
user_flags? Since we have the passed in flags, the map flags and the vma flags.
Sure can align.
We don't pass in map flags? We set map flags to pass to get_unmapped_area() in one place. Or in a final version, it's in one place.
We don't set map_flags in vma_remap_struct either. In fact here we only have one flags field, which is commented as such here.
I'm not sure about user_flags, I feel like that's confusing, mremap_flags be ok? But then that feels somewhat redundant in a struct literally all about an mremap?
It's fine this isn't a big deal, mremap_flags?
Quoted 8 lines
mm/mremap.c line 60
Quoted 6 lines
mm/mremap.c +5
-0+ +/* uffd state. */ +struct vm_userfaultfd_ctx *uf; +struct list_head *uf_unmap_early; +struct list_head *uf_unmap;... sigh, yeah.
Well, better put here right?
Quoted 10 lines
mm/mremap.c line 67
Quoted 8 lines
mm/mremap.c +7
-0+ +/* VMA state, determined in do_mremap(). */ +struct vm_area_struct *vma; + +/* Internal state, determined in do_mremap(). */ +unsigned long delta;/* Absolute delta of old_len, new_len. */ +bool locked;/* Was the VMA mlock()'d (has the VM_LOCKED flag set). */ bool mlocked ?
Sure will rename.
Quoted 29 lines
Quoted 29 lines
mm/mremap.c +21
-0+enum mremap_operation remap_type;/* expand, shrink, etc. */ +bool mmap_locked;/* Is current->mm currently write-locked? */ +}; +static pud_t *get_old_pud (struct mm_struct *mm,unsigned long addr) {pgd_t *pgd;@@ -693,6 +730,97 @@ unsigned long move_page_tables(struct vm_area_struct *vma,return len + old_addr - old_end;/* how much done */ } +/* Set vrm->delta to the difference in VMA size specified by user. */ +static void vrm_set_delta (struct vma_remap_struct *vrm) +{ + vrm->delta =abs_diff (vrm->old_len, vrm->new_len); +} + +/* Determine what kind of remap this is - shrink, expand or no resize at all. */ +static enum mremap_operation vrm_remap_type (struct vma_remap_struct *vrm) +{ +if (vrm->delta ==0 ) +return MREMAP_NO_RESIZE ; + +if (vrm->old_len > vrm->new_len) +return MREMAP_SHRINK ; + +return MREMAP_EXPAND ; +}
Quoted 8 lines
mm/mremap.c unsigned long move_page_tables(struct vm_area_struct *vma, · line 754
Quoted 5 lines
mm/mremap.c +5
-0+ +/* Set the vrm->remap_type, assumes state is sufficient set up for this. */ +static void vrm_set_remap_type (struct vma_remap_struct *vrm) +{ + vrm->remap_type =vrm_remap_type (vrm);The
vrm_remap_type()function is only used once, maybe we don't need both set and get?
I sort of felt it was neater to have a separate function but you're right this is a bit silly, will inline!
Quoted 30 lines
Quoted 30 lines
mm/mremap.c +29
-0+} + +/* +* When moving a VMA to vrm->new_adr, does this result in the new and old VMAs +* overlapping? +*/ +static bool vrm_overlaps (struct vma_remap_struct *vrm) +{ +unsigned long start_old = vrm->addr; +unsigned long start_new = vrm->new_addr; +unsigned long end_old = vrm->addr + vrm->old_len; +unsigned long end_new = vrm->new_addr + vrm->new_len; + +/* +* start_old end_old +* |-----------| +* | | +* |-----------| +* |-------------| +* | | +* |-------------| +* start_new end_new +*/ +if (end_old > start_new && end_new > start_old) +return true; + +return false; +} +
Quoted 8 lines
mm/mremap.c unsigned long move_page_tables(struct vm_area_struct *vma, · line 788
Quoted 5 lines
mm/mremap.c +5
-0+/* Do the mremap() flags require that the new_addr parameter be specified? */ +static bool vrm_implies_new_addr (struct vma_remap_struct *vrm) +{ +return vrm->flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +}These five might benefit from being inlined (although I hope our compiler is good enough for this).
Hm, I thought there was no difference when placed in a compilation unit like this?
Quoted 186 lines
Quoted 186 lines
mm/mremap.c +93
-39+ +/* +* Find an unmapped area for the requested vrm->new_addr. +* +* If MREMAP_FIXED then this is equivalent to a MAP_FIXED mmap() call. If only +* MREMAP_DONTUNMAP is set, then this is equivalent to providing a hint to +* mmap(), otherwise this is equivalent to mmap() specifying a NULL address. +* +* Returns 0 on success (with vrm->new_addr updated), or an error code upon +* failure. +*/ +static unsigned long vrm_set_new_addr (struct vma_remap_struct *vrm) +{ +struct vm_area_struct *vma = vrm->vma; +unsigned long map_flags =0 ; +/* Page Offset _into_ the VMA. */ +pgoff_t internal_pgoff = (vrm->addr - vma->vm_start) >>PAGE_SHIFT ; +pgoff_t pgoff = vma->vm_pgoff + internal_pgoff; +unsigned long new_addr =vrm_implies_new_addr (vrm) ? vrm->new_addr :0 ; +unsigned long res; + +if (vrm->flags &MREMAP_FIXED ) + map_flags |=MAP_FIXED ; +if (vma->vm_flags &VM_MAYSHARE ) + map_flags |=MAP_SHARED ; + + res =get_unmapped_area (vma->vm_file, new_addr, vrm->new_len, pgoff, + map_flags); +if (IS_ERR_VALUE (res)) +return res; + + vrm->new_addr = res; +return 0 ; +} + static unsigned long move_vma(struct vm_area_struct *vma, unsigned long old_addr, unsigned long old_len, unsigned long new_len, unsigned long new_addr,@@ -860,18 +988,15 @@ static unsigned long move_vma(struct vm_area_struct *vma,* resize_is_valid() - Ensure the vma can be resized to the new length at the give * address. * -* @vma: The vma to resize -* @addr: The old address -* @old_len: The current size -* @new_len: The desired size -* @flags: The vma flags -* * Return 0 on success, error otherwise. */ -static int resize_is_valid (struct vm_area_struct *vma,unsigned long addr, -unsigned long old_len,unsigned long new_len,unsigned long flags) +static int resize_is_valid (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; +struct vm_area_struct *vma = vrm->vma; +unsigned long addr = vrm->addr; +unsigned long old_len = vrm->old_len; +unsigned long new_len = vrm->new_len;unsigned long pgoff;/* @@ -883,11 +1008,12 @@ static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr,* behavior. As a result, fail such attempts. */ if (!old_len && !(vma->vm_flags & (VM_SHARED |VM_MAYSHARE ))) { -pr_warn_once ("%s (%d): attempted to duplicate a private mapping with mremap. This is not supported.\n" , current->comm, current->pid); +pr_warn_once ("%s (%d): attempted to duplicate a private mapping with mremap. This is not supported.\n" , + current->comm, current->pid);return -EINVAL ; } -if ((flags &MREMAP_DONTUNMAP ) && +if ((vrm->flags &MREMAP_DONTUNMAP ) && (vma->vm_flags & (VM_DONTEXPAND |VM_PFNMAP )))return -EINVAL ;@@ -907,99 +1033,114 @@ static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr,if (vma->vm_flags & (VM_DONTEXPAND |VM_PFNMAP ))return -EFAULT ; -if (!mlock_future_ok (mm, vma->vm_flags, new_len - old_len)) +if (!mlock_future_ok (mm, vma->vm_flags, vrm->delta))return -EAGAIN ; -if (!may_expand_vm (mm, vma->vm_flags, - (new_len - old_len) >>PAGE_SHIFT )) +if (!may_expand_vm (mm, vma->vm_flags, vrm->delta >>PAGE_SHIFT ))return -ENOMEM ;return 0 ; }/* -* mremap_to() - remap a vma to a new location -* @addr: The old address -* @old_len: The old size -* @new_addr: The target address -* @new_len: The new size -* @locked: If the returned vma is locked (VM_LOCKED) -* @flags: the mremap flags -* @uf: The mremap userfaultfd context -* @uf_unmap_early: The userfaultfd unmap early context -* @uf_unmap: The userfaultfd unmap context +* The user has requested that the VMA be shrunk (i.e., old_len > new_len), so +* execute this, optionally dropping the mmap lock when we do so. * +* In both cases this invalidates the VMA, however if we don't drop the lock, +* then load the correct VMA into vrm->vma afterwards. +*/ +static unsigned long shrink_vma (struct vma_remap_struct *vrm, +bool drop_lock) +{ +struct mm_struct *mm = current->mm; +unsigned long unmap_start = vrm->addr + vrm->new_len; +unsigned long unmap_bytes = vrm->delta; +unsigned long res; +VMA_ITERATOR (vmi, mm, unmap_start); + +VM_BUG_ON (vrm->remap_type !=MREMAP_SHRINK ); + + res =do_vmi_munmap (&vmi, mm, unmap_start, unmap_bytes, + vrm->uf_unmap, drop_lock); + vrm->vma =NULL ;/* Invalidated. */ +if (res) +return res; + +/* +* If we've not dropped the lock, then we should reload the VMA to +* replace the invalidated VMA with the one that may have now been +* split. +*/ +if (drop_lock) + vrm->mmap_locked = false; +else + vrm->vma =vma_lookup (mm, vrm->addr); + +return 0 ; +} + +/* +* mremap_to() - remap a vma to a new location. * Returns: The new address of the vma or an error. */ -static unsigned long mremap_to (unsigned long addr,unsigned long old_len, -unsigned long new_addr,unsigned long new_len,bool *locked, -unsigned long flags,struct vm_userfaultfd_ctx *uf, -struct list_head *uf_unmap_early, -struct list_head *uf_unmap) +static unsigned long mremap_to (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; -struct vm_area_struct *vma; -unsigned long ret; -unsigned long map_flags =0 ; +unsigned long err;/* Is the new length or address silly? */ -if (new_len >TASK_SIZE || new_addr >TASK_SIZE - new_len) +if (vrm->new_len >TASK_SIZE || + vrm->new_addr >TASK_SIZE - vrm->new_len)return -EINVAL ; -/* Ensure the old/new locations do not overlap. */ -if (addr + old_len > new_addr && new_addr + new_len > addr) +if (vrm_overlaps (vrm))return -EINVAL ; -if (flags &MREMAP_FIXED ) { +if (vrm->flags &MREMAP_FIXED ) {/* * In mremap_to(). * VMA is moved to dst address, and munmap dst first. * do_munmap will check if dst is sealed. */ - ret =do_munmap (mm, new_addr, new_len, uf_unmap_early); -if (ret) -return ret; - } + err =do_munmap (mm, vrm->new_addr, vrm->new_len, + vrm->uf_unmap_early); + vrm->vma =NULL ;/* Invalidated. */ +if (err) +return err;
Quoted 13 lines
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1115
Quoted 10 lines
mm/mremap.c +5
-5-if (old_len > new_len) { - ret =do_munmap (mm, addr+new_len, old_len - new_len, uf_unmap); -if (ret) -return ret; - old_len = new_len; +/* +* If we remap a portion of a VMA elsewhere in the same VMA, +* this can invalidate the old VMA and iterator. Reset. +*/ + vrm->vma =vma_lookup (mm, vrm->addr);You say it invalidates the iterator, but this doesn't change an iterator?
Yeah there used to be one, didn't update comment, will fix.
Quoted 13 lines
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1119
Quoted 8 lines
mm/mremap.c +2
-3} - vma =vma_lookup (mm, addr); -if (!vma) -return -EFAULT ; +if (vrm->remap_type ==MREMAP_SHRINK ) { + err =shrink_vma (vrm,/* drop_lock= */ false);It is not immediately clear if we could have a
MREMAP_FIXEDalsoMREMAP_SHRINK. In that case, we would try todo_munmap()twice. This shouldn't be an issue asdo_vmi_munmap()would catch it, but I am not sure if you noticed this.
I think this case is pretty subtle actually. The logic already looked like this, only more hidden.
So we:
1. Unmap the target area at the new, shorter, length if MREMAP_FIXED.
2. Then remove the 'extra' portion in the source that we are shrinking 'off'.
E.g.:
|--.------.---| |--------.---.---| | . . | -> | . . | |--.------.---| |--------.---.---|
Implies:
1. Remove the delta from the original:
|--.---| |---| |--------.---.---| | . | | | -> | . . | |--.---| |---| |--------.---.---|
2. Unmap target area:
|--.---| |---| |--------| |---| | . | | | -> | | | | |--.---| |---| |--------| |---|
The reason we do this is because MREMAP_DONTUNMAP might be set, and we start by doing a _copy_ of the VMA, then move page tables, then finally, if MREMAP_DONTUNMAP is _not_ set we finally unmap the original, shrunk VMA.
I mean this is all very very insane and maybe we can just like... not do this haha.
But a future thing, perhaps.
Quoted 15 lines
Quoted 15 lines
mm/mremap.c +9
-3+if (err) +return err; - ret =resize_is_valid (vma, addr, old_len, new_len, flags); -if (ret) -return ret; +/* Set up for the move now shrink has been executed. */ + vrm->old_len = vrm->new_len; + } + + err =resize_is_valid (vrm); +if (err) +return err;
Quoted 8 lines
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1134
Quoted 6 lines
mm/mremap.c +2
-2/* MREMAP_DONTUNMAP expands by old_len since old_len == new_len */ -if (flags &MREMAP_DONTUNMAP && - !may_expand_vm (mm, vma->vm_flags, old_len >>PAGE_SHIFT )) { +if (vrm->flags &MREMAP_DONTUNMAP && + !may_expand_vm (mm, vrm->vma->vm_flags, vrm->old_len >>PAGE_SHIFT )) {return -ENOMEM ;nit: whitespace here is a bit odd to read.
Will find a way to fix, I agree this is a bit horrid.
Quoted 22 lines
Quoted 22 lines
mm/mremap.c +3
-15} -if (flags &MREMAP_FIXED ) - map_flags |=MAP_FIXED ; - -if (vma->vm_flags &VM_MAYSHARE ) - map_flags |=MAP_SHARED ; - - ret =get_unmapped_area (vma->vm_file, new_addr, new_len, vma->vm_pgoff + - ((addr - vma->vm_start) >>PAGE_SHIFT ), - map_flags); -if (IS_ERR_VALUE (ret)) -return ret; - -/* We got a new mapping */ -if (!(flags &MREMAP_FIXED )) - new_addr = ret; + err =vrm_set_new_addr (vrm); +if (err) +return err;
Quoted 7 lines
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1143
Quoted 5 lines
mm/mremap.c +3
-2-return move_vma (vma, addr, old_len, new_len, new_addr, locked, flags, - uf, uf_unmap); +return move_vma (vrm->vma, vrm->addr, vrm->old_len, vrm->new_len, + vrm->new_addr, &vrm->locked, vrm->flags, + vrm->uf, vrm->uf_unmap);I see where this is going..
;)
Quoted 242 lines
Quoted 226 lines
mm/mremap.c +54
-60}static int vma_expandable (struct vm_area_struct *vma,unsigned long delta)@@ -1016,22 +1157,33 @@ static int vma_expandable(struct vm_area_struct *vma, unsigned long delta)return 1 ; } -/* Do the mremap() flags require that the new_addr parameter be specified? */ -static bool implies_new_addr (unsigned long flags) +/* Determine whether we are actually able to execute an in-place expansion. */ +static bool vrm_can_expand_in_place (struct vma_remap_struct *vrm) { -return flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +/* Number of bytes from vrm->addr to end of VMA. */ +unsigned long suffix_bytes = vrm->vma->vm_end - vrm->addr; + +/* If end of range aligns to end of VMA, we can just expand in-place. */ +if (suffix_bytes != vrm->old_len) +return false; + +/* Check whether this is feasible. */ +if (!vma_expandable (vrm->vma, vrm->delta)) +return false; + +return true; }/* * Are the parameters passed to mremap() valid? If so return 0, otherwise return * error. */ -static unsigned long check_mremap_params (unsigned long addr, -unsigned long flags, -unsigned long old_len, -unsigned long new_len, -unsigned long new_addr) +static unsigned long check_mremap_params (struct vma_remap_struct *vrm) + { +unsigned long addr = vrm->addr; +unsigned long flags = vrm->flags; +/* Ensure no unexpected flag values. */ if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP ))return -EINVAL ;@@ -1045,15 +1197,15 @@ static unsigned long check_mremap_params(unsigned long addr,* for DOS-emu "duplicate shm area" thing. But * a zero new-len is nonsensical. */ -if (!PAGE_ALIGN (new_len)) +if (!PAGE_ALIGN (vrm->new_len))return -EINVAL ;/* Remainder of checks are for cases with specific new_addr. */ -if (!implies_new_addr (flags)) +if (!vrm_implies_new_addr (vrm))return 0 ;/* The new address must be page-aligned. */ -if (offset_in_page (new_addr)) +if (offset_in_page (vrm->new_addr))return -EINVAL ;/* A fixed address implies a move. */ @@ -1061,7 +1213,7 @@ static unsigned long check_mremap_params(unsigned long addr,return -EINVAL ;/* MREMAP_DONTUNMAP does not allow resizing in the process. */ -if (flags &MREMAP_DONTUNMAP && old_len != new_len) +if (flags &MREMAP_DONTUNMAP && vrm->old_len != vrm->new_len)return -EINVAL ;/* @@ -1090,11 +1242,11 @@ static unsigned long check_mremap_params(unsigned long addr,* If we discover the VMA is locked, update mm_struct statistics accordingly and * indicate so to the caller. */ -static unsigned long expand_vma_inplace (struct vm_area_struct *vma, -unsigned long delta,bool *locked) +static unsigned long expand_vma_in_place (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; -long pages = delta >>PAGE_SHIFT ; +long pages = vrm->delta >>PAGE_SHIFT ; +struct vm_area_struct *vma = vrm->vma;VMA_ITERATOR (vmi, mm, vma->vm_end);long charged =0 ;@@ -1114,7 +1266,7 @@ static unsigned long expand_vma_inplace(struct vm_area_struct *vma,* adjacent to the expanded vma and otherwise * compatible. */ - vma =vma_merge_extend (&vmi, vma, delta); + vma = vrm->vma =vma_merge_extend (&vmi, vma, vrm->delta);if (!vma) {vm_unacct_memory (charged);return -ENOMEM ;@@ -1123,42 +1275,34 @@ static unsigned long expand_vma_inplace(struct vm_area_struct *vma,vm_stat_account (mm, vma->vm_flags, pages);if (vma->vm_flags &VM_LOCKED ) { mm->locked_vm += pages; - *locked = true; + vrm->locked = true; }return 0 ; } -static bool align_hugetlb (struct vm_area_struct *vma, -unsigned long addr, -unsigned long new_addr, -unsigned long *old_len_ptr, -unsigned long *new_len_ptr, -unsigned long *delta_ptr) +static bool align_hugetlb (struct vma_remap_struct *vrm) { -unsigned long old_len = *old_len_ptr; -unsigned long new_len = *new_len_ptr; -struct hstate *h__maybe_unused =hstate_vma (vma); +struct hstate *h__maybe_unused =hstate_vma (vrm->vma); - old_len =ALIGN (old_len,huge_page_size (h)); - new_len =ALIGN (new_len,huge_page_size (h)); + vrm->old_len =ALIGN (vrm->old_len,huge_page_size (h)); + vrm->new_len =ALIGN (vrm->new_len,huge_page_size (h));/* addrs must be huge page aligned */ -if (addr & ~huge_page_mask (h)) +if (vrm->addr & ~huge_page_mask (h))return false; -if (new_addr & ~huge_page_mask (h)) +if (vrm->new_addr & ~huge_page_mask (h))return false;/* * Don't allow remap expansion, because the underlying hugetlb * reservation is not yet capable to handle split reservation. */ -if (new_len > old_len) +if (vrm->new_len > vrm->old_len)return false; - *old_len_ptr = old_len; - *new_len_ptr = new_len; - *delta_ptr =abs_diff (old_len, new_len); +vrm_set_delta (vrm); +return true; }@@ -1169,19 +1313,16 @@ static bool align_hugetlb(struct vm_area_struct *vma,* Try to do so in-place, if this fails, then move the VMA to a new location to * action the change. */ -static unsigned long expand_vma (struct vm_area_struct *vma, -unsigned long addr,unsigned long old_len, -unsigned long new_len,unsigned long flags, -bool *locked_ptr,unsigned long *new_addr_ptr, -struct vm_userfaultfd_ctx *uf_ptr, -struct list_head *uf_unmap_ptr) +static unsigned long expand_vma (struct vma_remap_struct *vrm) {unsigned long err; -unsigned long map_flags; -unsigned long new_addr;/* We ignore any user-supplied one. */ -pgoff_t pgoff; +struct vm_area_struct *vma = vrm->vma; +unsigned long addr = vrm->addr; +unsigned long old_len = vrm->old_len; +unsigned long new_len = vrm->new_len; +unsigned long flags = vrm->flags; - err =resize_is_valid (vma, addr, old_len, new_len, flags); + err =resize_is_valid (vrm);if (err)return err;@@ -1189,10 +1330,9 @@ static unsigned long expand_vma(struct vm_area_struct *vma,* [addr, old_len) spans precisely to the end of the VMA, so try to * expand it in-place. */ -if (old_len == vma->vm_end - addr && -vma_expandable (vma, new_len - old_len)) { - err =expand_vma_inplace (vma, new_len - old_len, locked_ptr); -if (IS_ERR_VALUE (err)) +if (vrm_can_expand_in_place (vrm)) { + err =expand_vma_in_place (vrm); +if (err)return err;/* @@ -1200,8 +1340,8 @@ static unsigned long expand_vma(struct vm_area_struct *vma,* satisfy the expectation that mlock()'ing a VMA maintains all * of its pages in memory. */ -if (*locked_ptr) - *new_addr_ptr = addr; +if (vrm->locked) + vrm->new_addr = addr;/* OK we're done! */ return addr;@@ -1217,62 +1357,65 @@ static unsigned long expand_vma(struct vm_area_struct *vma,return -ENOMEM ;/* Find a new location to move the VMA to. */ - map_flags = (vma->vm_flags &VM_MAYSHARE ) ?MAP_SHARED :0 ; - pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >>PAGE_SHIFT ); - new_addr =get_unmapped_area (vma->vm_file,0 , new_len, pgoff, map_flags); -if (IS_ERR_VALUE (new_addr)) -return new_addr; - *new_addr_ptr = new_addr; + err =vrm_set_new_addr (vrm); +if (err) +return err; -return move_vma (vma, addr, old_len, new_len, new_addr, - locked_ptr, flags, uf_ptr, uf_unmap_ptr); +return move_vma (vma, addr, old_len, new_len, vrm->new_addr, + &vrm->locked, flags, vrm->uf, vrm->uf_unmap); }/* -* Expand (or shrink) an existing mapping, potentially moving it at the -* same time (controlled by the MREMAP_MAYMOVE flag and available VM space) mm/mremap.c static unsigned long expand_vma(struct vm_area_struct *vma, · line 1372
Quoted 10 lines
mm/mremap.c +3
-6-* -* MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise -* This option implies MREMAP_MAYMOVE. +* Attempt to resize the VMA in-place, if we cannot, then move the VMA to the +* first available address to perform the operation. */ -SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, - unsigned long, new_len, unsigned long, flags, - unsigned long, new_addr) +static unsigned long mremap_at (struct vma_remap_struct *vrm)I hate this and
mremap_to()names. I don't have a proposed better name for either and maybe it's just my experience withmremap_to()that has tainted the view of the name itself, but I find them not very descriptive and abruptly ended. I guess move was already taken.
Quoted 4 lines
I also have the added baggage of parsing "at" to potentially mean "doing".
mremap_inplace()seems equally annoying. This is the worst bike shed.
Haha well yeah it's a bikeshed but yeah I agree, it's horrid I agree.
I was using the _at() only in line with the _to().
The 'in place' can kind of conflict with other references to in place which is used to differentiate between the case of us having to get_unmapped_area() and move vs. not having to.
So I feel like we're stuck with this...
Quoted 137 lines
Quoted 118 lines
mm/mremap.c +42
-49+{ +unsigned long res; + +switch (vrm->remap_type) { +case MREMAP_NO_RESIZE : +/* NO-OP CASE - resizing to the same size. */ +return vrm->addr; +case MREMAP_SHRINK : +/* +* SHRINK CASE. Can always be done in-place. +* +* Simply unmap the shrunken portion of the VMA. This does all +* the needed commit accounting, and we indicate that the mmap +* lock should be dropped. +*/ + res =shrink_vma (vrm,/* drop_lock= */ true); +if (res) +return res; + +return vrm->addr; +case MREMAP_EXPAND : +return expand_vma (vrm); + } + +BUG (); +} + +static unsigned long do_mremap (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma;unsigned long ret; -unsigned long delta; -bool locked = false; -struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ; -LIST_HEAD (uf_unmap_early); -LIST_HEAD (uf_unmap); -/* -* There is a deliberate asymmetry here: we strip the pointer tag -* from the old address but leave the new address alone. This is -* for consistency with mmap(), where we prevent the creation of -* aliasing mappings in userspace by leaving the tag bits of the -* mapping address intact. A non-zero tag will cause the subsequent -* range checks to reject the address as invalid. -* -* See Documentation/arch/arm64/tagged-address-abi.rst for more -* information. -*/ - addr =untagged_addr (addr); - - ret =check_mremap_params (addr, flags, old_len, new_len, new_addr); + ret =check_mremap_params (vrm);if (ret)return ret; - old_len =PAGE_ALIGN (old_len); - new_len =PAGE_ALIGN (new_len); - delta =abs_diff (old_len, new_len); + vrm->old_len =PAGE_ALIGN (vrm->old_len); + vrm->new_len =PAGE_ALIGN (vrm->new_len); +vrm_set_delta (vrm);if (mmap_write_lock_killable (mm))return -EINTR ; + vrm->mmap_locked = true; - vma =vma_lookup (mm, addr); + vma = vrm->vma =vma_lookup (mm, vrm->addr);if (!vma) { ret = -EFAULT ; goto out;@@ -1285,62 +1428,68 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, }/* Align to hugetlb page size, if required. */ -if (is_vm_hugetlb_page (vma) && - !align_hugetlb (vma, addr, new_addr, &old_len, &new_len, &delta)) { +if (is_vm_hugetlb_page (vma) && !align_hugetlb (vrm)) { ret = -EINVAL ; goto out; } -/* Are we RELOCATING the VMA to a SPECIFIC address? */ -if (implies_new_addr (flags)) { - ret =mremap_to (addr, old_len, new_addr, new_len, - &locked, flags, &uf, &uf_unmap_early, - &uf_unmap); - goto out; - } +vrm_set_remap_type (vrm); -/* -* From here on in we are only RESIZING the VMA, attempting to do so -* in-place, moving the VMA if we cannot. -*/ +/* Actually execute mremap. */ + ret =vrm_implies_new_addr (vrm) ?mremap_to (vrm) :mremap_at (vrm); -/* NO-OP CASE - resizing to the same size. */ -if (new_len == old_len) { - ret = addr; - goto out; - } - -/* SHRINK CASE. Can always be done in-place. */ -if (new_len < old_len) { -VMA_ITERATOR (vmi, mm, addr + new_len); +out: +if (vrm->mmap_locked) { +mmap_write_unlock (mm); + vrm->mmap_locked = false; -/* -* Simply unmap the shrunken portion of the VMA. This does all -* the needed commit accounting, unlocking the mmap lock. -*/ mm/mremap.c SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, · line 1447
Quoted 10 lines
mm/mremap.c +2
-8- ret =do_vmi_munmap (&vmi, mm, addr + new_len, delta, - &uf_unmap, true); -if (ret) - goto out; - -/* We succeeded, mmap lock released for us. */ - ret = addr; - goto out_unlocked; +if (!offset_in_page (ret) && vrm->locked && vrm->new_len > vrm->old_len) +mm_populate (vrm->new_addr + vrm->old_len, vrm->delta);It isn't clear to me why we only populate if we are locked here. Actually, I'm not sure why we keep holding the lock until here or why it matters to drop it early. The main reason we want to drop the lock is to reduce the mmap lock time held for the populate operation.
So we can either drop the lock before we get here once the vma tree is updated, or we can unconditionally unlock.
Quoted 8 lines
I think we can simplify this further if we just keep the lock held until we downgrade here and populate if necessary after it's dropped. That is, shrink just does nothing with the lock and we just unlock it regardless.
I'm pretty sure that we would struggle to measure the performance impact holding the lock for the return path when the populate is removed from the critical section.
I'm in agreement that this is a horrid and weird mess, and I stared at this a long time myself.
But I feel we should defer changing this until after the refactor series as it was an existing eldrich horror changing of which would sort of sit outside of 'obvious small fixups/safe moving around/etc.' that I limited myself to here.
Let's definitely come back and address this.
Quoted 65 lines
Quoted 65 lines
mm/mremap.c +43
-14} -/* EXPAND case. We try to do in-place, if we can't, then we move it. */ - ret =expand_vma (vma, addr, old_len, new_len, flags, &locked, &new_addr, - &uf, &uf_unmap); +userfaultfd_unmap_complete (mm, vrm->uf_unmap_early); +mremap_userfaultfd_complete (vrm->uf, vrm->addr, ret, vrm->old_len); +userfaultfd_unmap_complete (mm, vrm->uf_unmap); -out: -if (offset_in_page (ret)) - locked = false; -mmap_write_unlock (mm); -if (locked && new_len > old_len) -mm_populate (new_addr + old_len, delta); -out_unlocked: -userfaultfd_unmap_complete (mm, &uf_unmap_early); -mremap_userfaultfd_complete (&uf, addr, ret, old_len); -userfaultfd_unmap_complete (mm, &uf_unmap);return ret; } + +/* +* Expand (or shrink) an existing mapping, potentially moving it at the +* same time (controlled by the MREMAP_MAYMOVE flag and available VM space) +* +* MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise +* This option implies MREMAP_MAYMOVE. +*/ +SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, + unsigned long, new_len, unsigned long, flags, + unsigned long, new_addr) +{ +struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ; +LIST_HEAD (uf_unmap_early); +LIST_HEAD (uf_unmap); +/* +* There is a deliberate asymmetry here: we strip the pointer tag +* from the old address but leave the new address alone. This is +* for consistency with mmap(), where we prevent the creation of +* aliasing mappings in userspace by leaving the tag bits of the +* mapping address intact. A non-zero tag will cause the subsequent +* range checks to reject the address as invalid. +* +* See Documentation/arch/arm64/tagged-address-abi.rst for more +* information. +*/ +struct vma_remap_struct vrm = { + .addr =untagged_addr (addr), + .old_len = old_len, + .new_len = new_len, + .flags = flags, + .new_addr = new_addr, + + .uf = &uf, + .uf_unmap_early = &uf_unmap_early, + .uf_unmap = &uf_unmap, + }; + +return do_mremap (&vrm); +} --2.48.1
Re: [PATCH 3/7] mm/mremap: introduce and use vma_remap_struct threaded state
Quoted 92 lines · Lorenzo Stoakes
Quoted 46 lines · Liam R. Howlett
Quoted 45 lines · Lorenzo Stoakes
A number of
mremap()calls both pass around and modify a large number of parameters, making the code less readable and often repeatedly having to determine things such as VMA, size delta, and more.Avoid this by using the common pattern of passing a state object through the operation, updating it as we go. We introduce the
vma_remap_structor 'VRM' for this purpose.This also gives us the ability to accumulate further state through the operation that would otherwise require awkward and error-prone pointer passing.
We can also now trivially define helper functions that operate on a VRM object.
This pattern has proven itself to be very powerful when implemented for VMA merge, VMA unmapping and memory mapping operations, so it is battle-tested and functional.
We both introduce the data structure and use it, introducing helper functions as needed to make things readable, we move some state such as mmap lock and
mlock()status to the VRM, we introduce a means of classifying the type ofmremap()operation and de-duplicate theget_unmapped_area()lookup.We also neatly thread userfaultfd state throughout the operation.
Note that there is further refactoring to be done, chiefly adjust
move_vma()to accept a VRM parameter. We defer this as there is pre-requisite work required to be able to do so which we will do in a subsequent patch.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/mremap.c | 559 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 354 insertions(+), 205 deletions(-)mm/mremap.c +0
-0@@ -32,6 +32,43 @@#include "internal.h" Quoted 7 lines
mm/mremap.c line 39
Quoted 5 lines
mm/mremap.c +5
-0+/* Classify the kind of remap operation being performed. */ +enum mremap_operation { +MREMAP_NO_RESIZE ,/* old_len == new_len, if not moved, do nothing. */ +MREMAP_SHRINK ,/* old_len > new_len. */ +MREMAP_EXPAND ,/* old_len < new_len. */ Can we fix the spacing so the comments line up, please?
I've had review the other way before :) But sure, I mean I actually agree. Perhaps I was following stuff I already saw in the kernel on this, but yeah let's align this!
Quoted 3 lines
It might be worth having a
MREMAP_INVALIDhere and init to that, then aVM_BUG_ON(), but maybe I'm just paranoid.Sure can do.
Quoted 13 lines
Quoted 13 lines
mm/mremap.c +12
-0+}; + +/* +* Describes a VMA mremap() operation and is threaded throughout it. +* +* Any of the fields may be mutated by the operation, however these values will +* always accurately reflect the remap (for instance, we may adjust lengths and +* delta to account for hugetlb alignment). +*/ +struct vma_remap_struct { +/* User-provided state. */ +unsigned long addr;/* User-specified address from which we remap. */ Quoted 8 lines
mm/mremap.c line 55
Quoted 4 lines
mm/mremap.c +4
-0+unsigned long old_len;/* Length of range being remapped. */ +unsigned long new_len;/* Desired new length of mapping. */ +unsigned long flags;/* user-specified MREMAP_* flags. */ +unsigned long new_addr;/* Optionally, desired new address. */ Same comment about the comment spacing here. Might be better to have
user_flags? Since we have the passed in flags, the map flags and the vma flags.Sure can align.
We don't pass in map flags? We set map flags to pass to
get_unmapped_area()in one place. Or in a final version, it's in one place.
Quoted 8 lines
We don't set
map_flagsinvma_remap_structeither. In fact here we only have one flags field, which is commented as such here.I'm not sure about
user_flags, I feel like that's confusing,mremap_flagsbe ok? But then that feels somewhat redundant in a struct literally all about an mremap?It's fine this isn't a big deal,
mremap_flags?
Good point. I have no strong opinion on this either, feel free to keep it as flags.
Quoted 23 lines
Quoted 9 lines
mm/mremap.c line 60
Quoted 7 lines
mm/mremap.c +5
-0+ +/* uffd state. */ +struct vm_userfaultfd_ctx *uf; +struct list_head *uf_unmap_early; +struct list_head *uf_unmap;... sigh, yeah.
Well, better put here right?
Quoted 10 lines
mm/mremap.c line 67
Quoted 8 lines
mm/mremap.c +7
-0+ +/* VMA state, determined in do_mremap(). */ +struct vm_area_struct *vma; + +/* Internal state, determined in do_mremap(). */ +unsigned long delta;/* Absolute delta of old_len, new_len. */ +bool locked;/* Was the VMA mlock()'d (has the VM_LOCKED flag set). */ bool mlocked ?
Quoted 1 line
Sure will rename.
Thanks.
Quoted 39 lines
Quoted 30 lines
Quoted 30 lines
mm/mremap.c +21
-0+enum mremap_operation remap_type;/* expand, shrink, etc. */ +bool mmap_locked;/* Is current->mm currently write-locked? */ +}; +static pud_t *get_old_pud (struct mm_struct *mm,unsigned long addr) {pgd_t *pgd;@@ -693,6 +730,97 @@ unsigned long move_page_tables(struct vm_area_struct *vma,return len + old_addr - old_end;/* how much done */ } +/* Set vrm->delta to the difference in VMA size specified by user. */ +static void vrm_set_delta (struct vma_remap_struct *vrm) +{ + vrm->delta =abs_diff (vrm->old_len, vrm->new_len); +} + +/* Determine what kind of remap this is - shrink, expand or no resize at all. */ +static enum mremap_operation vrm_remap_type (struct vma_remap_struct *vrm) +{ +if (vrm->delta ==0 ) +return MREMAP_NO_RESIZE ; + +if (vrm->old_len > vrm->new_len) +return MREMAP_SHRINK ; + +return MREMAP_EXPAND ; +}Quoted 8 lines
mm/mremap.c unsigned long move_page_tables(struct vm_area_struct *vma, · line 754
Quoted 5 lines
mm/mremap.c +5
-0+ +/* Set the vrm->remap_type, assumes state is sufficient set up for this. */ +static void vrm_set_remap_type (struct vma_remap_struct *vrm) +{ + vrm->remap_type =vrm_remap_type (vrm);The
vrm_remap_type()function is only used once, maybe we don't need both set and get?
Quoted 2 lines
I sort of felt it was neater to have a separate function but you're right this is a bit silly, will inline!
Yeah, we can move it externally if we need it. Thanks.
Quoted 40 lines
Quoted 31 lines
Quoted 31 lines
mm/mremap.c +29
-0+} + +/* +* When moving a VMA to vrm->new_adr, does this result in the new and old VMAs +* overlapping? +*/ +static bool vrm_overlaps (struct vma_remap_struct *vrm) +{ +unsigned long start_old = vrm->addr; +unsigned long start_new = vrm->new_addr; +unsigned long end_old = vrm->addr + vrm->old_len; +unsigned long end_new = vrm->new_addr + vrm->new_len; + +/* +* start_old end_old +* |-----------| +* | | +* |-----------| +* |-------------| +* | | +* |-------------| +* start_new end_new +*/ +if (end_old > start_new && end_new > start_old) +return true; + +return false; +} +Quoted 8 lines
mm/mremap.c unsigned long move_page_tables(struct vm_area_struct *vma, · line 788
Quoted 5 lines
mm/mremap.c +5
-0+/* Do the mremap() flags require that the new_addr parameter be specified? */ +static bool vrm_implies_new_addr (struct vma_remap_struct *vrm) +{ +return vrm->flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +}These five might benefit from being inlined (although I hope our compiler is good enough for this).
Quoted 2 lines
Hm, I thought there was no difference when placed in a compilation unit like this?
I think it's up to the compiler, and I'm not sure it changes unless we force it anyways. Feel free to leave it.
Quoted 201 lines
Quoted 187 lines
Quoted 187 lines
mm/mremap.c +93
-39+ +/* +* Find an unmapped area for the requested vrm->new_addr. +* +* If MREMAP_FIXED then this is equivalent to a MAP_FIXED mmap() call. If only +* MREMAP_DONTUNMAP is set, then this is equivalent to providing a hint to +* mmap(), otherwise this is equivalent to mmap() specifying a NULL address. +* +* Returns 0 on success (with vrm->new_addr updated), or an error code upon +* failure. +*/ +static unsigned long vrm_set_new_addr (struct vma_remap_struct *vrm) +{ +struct vm_area_struct *vma = vrm->vma; +unsigned long map_flags =0 ; +/* Page Offset _into_ the VMA. */ +pgoff_t internal_pgoff = (vrm->addr - vma->vm_start) >>PAGE_SHIFT ; +pgoff_t pgoff = vma->vm_pgoff + internal_pgoff; +unsigned long new_addr =vrm_implies_new_addr (vrm) ? vrm->new_addr :0 ; +unsigned long res; + +if (vrm->flags &MREMAP_FIXED ) + map_flags |=MAP_FIXED ; +if (vma->vm_flags &VM_MAYSHARE ) + map_flags |=MAP_SHARED ; + + res =get_unmapped_area (vma->vm_file, new_addr, vrm->new_len, pgoff, + map_flags); +if (IS_ERR_VALUE (res)) +return res; + + vrm->new_addr = res; +return 0 ; +} + static unsigned long move_vma(struct vm_area_struct *vma, unsigned long old_addr, unsigned long old_len, unsigned long new_len, unsigned long new_addr,@@ -860,18 +988,15 @@ static unsigned long move_vma(struct vm_area_struct *vma,* resize_is_valid() - Ensure the vma can be resized to the new length at the give * address. * -* @vma: The vma to resize -* @addr: The old address -* @old_len: The current size -* @new_len: The desired size -* @flags: The vma flags -* * Return 0 on success, error otherwise. */ -static int resize_is_valid (struct vm_area_struct *vma,unsigned long addr, -unsigned long old_len,unsigned long new_len,unsigned long flags) +static int resize_is_valid (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; +struct vm_area_struct *vma = vrm->vma; +unsigned long addr = vrm->addr; +unsigned long old_len = vrm->old_len; +unsigned long new_len = vrm->new_len;unsigned long pgoff;/* @@ -883,11 +1008,12 @@ static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr,* behavior. As a result, fail such attempts. */ if (!old_len && !(vma->vm_flags & (VM_SHARED |VM_MAYSHARE ))) { -pr_warn_once ("%s (%d): attempted to duplicate a private mapping with mremap. This is not supported.\n" , current->comm, current->pid); +pr_warn_once ("%s (%d): attempted to duplicate a private mapping with mremap. This is not supported.\n" , + current->comm, current->pid);return -EINVAL ; } -if ((flags &MREMAP_DONTUNMAP ) && +if ((vrm->flags &MREMAP_DONTUNMAP ) && (vma->vm_flags & (VM_DONTEXPAND |VM_PFNMAP )))return -EINVAL ;@@ -907,99 +1033,114 @@ static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr,if (vma->vm_flags & (VM_DONTEXPAND |VM_PFNMAP ))return -EFAULT ; -if (!mlock_future_ok (mm, vma->vm_flags, new_len - old_len)) +if (!mlock_future_ok (mm, vma->vm_flags, vrm->delta))return -EAGAIN ; -if (!may_expand_vm (mm, vma->vm_flags, - (new_len - old_len) >>PAGE_SHIFT )) +if (!may_expand_vm (mm, vma->vm_flags, vrm->delta >>PAGE_SHIFT ))return -ENOMEM ;return 0 ; }/* -* mremap_to() - remap a vma to a new location -* @addr: The old address -* @old_len: The old size -* @new_addr: The target address -* @new_len: The new size -* @locked: If the returned vma is locked (VM_LOCKED) -* @flags: the mremap flags -* @uf: The mremap userfaultfd context -* @uf_unmap_early: The userfaultfd unmap early context -* @uf_unmap: The userfaultfd unmap context +* The user has requested that the VMA be shrunk (i.e., old_len > new_len), so +* execute this, optionally dropping the mmap lock when we do so. * +* In both cases this invalidates the VMA, however if we don't drop the lock, +* then load the correct VMA into vrm->vma afterwards. +*/ +static unsigned long shrink_vma (struct vma_remap_struct *vrm, +bool drop_lock) +{ +struct mm_struct *mm = current->mm; +unsigned long unmap_start = vrm->addr + vrm->new_len; +unsigned long unmap_bytes = vrm->delta; +unsigned long res; +VMA_ITERATOR (vmi, mm, unmap_start); + +VM_BUG_ON (vrm->remap_type !=MREMAP_SHRINK ); + + res =do_vmi_munmap (&vmi, mm, unmap_start, unmap_bytes, + vrm->uf_unmap, drop_lock); + vrm->vma =NULL ;/* Invalidated. */ +if (res) +return res; + +/* +* If we've not dropped the lock, then we should reload the VMA to +* replace the invalidated VMA with the one that may have now been +* split. +*/ +if (drop_lock) + vrm->mmap_locked = false; +else + vrm->vma =vma_lookup (mm, vrm->addr); + +return 0 ; +} + +/* +* mremap_to() - remap a vma to a new location. * Returns: The new address of the vma or an error. */ -static unsigned long mremap_to (unsigned long addr,unsigned long old_len, -unsigned long new_addr,unsigned long new_len,bool *locked, -unsigned long flags,struct vm_userfaultfd_ctx *uf, -struct list_head *uf_unmap_early, -struct list_head *uf_unmap) +static unsigned long mremap_to (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; -struct vm_area_struct *vma; -unsigned long ret; -unsigned long map_flags =0 ; +unsigned long err;/* Is the new length or address silly? */ -if (new_len >TASK_SIZE || new_addr >TASK_SIZE - new_len) +if (vrm->new_len >TASK_SIZE || + vrm->new_addr >TASK_SIZE - vrm->new_len)return -EINVAL ; -/* Ensure the old/new locations do not overlap. */ -if (addr + old_len > new_addr && new_addr + new_len > addr) +if (vrm_overlaps (vrm))return -EINVAL ; -if (flags &MREMAP_FIXED ) { +if (vrm->flags &MREMAP_FIXED ) {/* * In mremap_to(). * VMA is moved to dst address, and munmap dst first. * do_munmap will check if dst is sealed. */ - ret =do_munmap (mm, new_addr, new_len, uf_unmap_early); -if (ret) -return ret; - } + err =do_munmap (mm, vrm->new_addr, vrm->new_len, + vrm->uf_unmap_early); + vrm->vma =NULL ;/* Invalidated. */ +if (err) +return err;Quoted 13 lines
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1115
Quoted 10 lines
mm/mremap.c +5
-5-if (old_len > new_len) { - ret =do_munmap (mm, addr+new_len, old_len - new_len, uf_unmap); -if (ret) -return ret; - old_len = new_len; +/* +* If we remap a portion of a VMA elsewhere in the same VMA, +* this can invalidate the old VMA and iterator. Reset. +*/ + vrm->vma =vma_lookup (mm, vrm->addr);You say it invalidates the iterator, but this doesn't change an iterator?
Quoted 1 line
Yeah there used to be one, didn't update comment, will fix.
Thanks.
Quoted 38 lines
Quoted 4 lines
Quoted 4 lines
mm/mremap.c +0
-0}Quoted 10 lines
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1119
Quoted 5 lines
mm/mremap.c +2
-3- vma =vma_lookup (mm, addr); -if (!vma) -return -EFAULT ; +if (vrm->remap_type ==MREMAP_SHRINK ) { + err =shrink_vma (vrm,/* drop_lock= */ false);It is not immediately clear if we could have a
MREMAP_FIXEDalsoMREMAP_SHRINK. In that case, we would try todo_munmap()twice. This shouldn't be an issue asdo_vmi_munmap()would catch it, but I am not sure if you noticed this.I think this case is pretty subtle actually. The logic already looked like this, only more hidden.
So we:
1. Unmap the target area at the new, shorter, length if
MREMAP_FIXED.
2. Then remove the 'extra' portion in the source that we are shrinking 'off'.E.g.:
|--.------.---| |--------.---.---| | . . | -> | . . | |--.------.---| |--------.---.---|Implies:
1. Remove the delta from the original:
|--.---| |---| |--------.---.---| | . | | | -> | . . | |--.---| |---| |--------.---.---|
Quoted 5 lines
2. Unmap target area:
|--.---| |---| |--------| |---| | . | | | -> | | | | |--.---| |---| |--------| |---|
Nice pictures.
Quoted 9 lines
The reason we do this is because
MREMAP_DONTUNMAPmight be set, and we start by doing a _copy_ of the VMA, then move page tables, then finally, ifMREMAP_DONTUNMAPis _not_ set we finally unmap the original, shrunk VMA.I mean this is all very very insane and maybe we can just like... not do this haha.
But a future thing, perhaps.
Right, thanks.
Quoted 309 lines
Quoted 17 lines
Quoted 17 lines
mm/mremap.c +9
-3+if (err) +return err; - ret =resize_is_valid (vma, addr, old_len, new_len, flags); -if (ret) -return ret; +/* Set up for the move now shrink has been executed. */ + vrm->old_len = vrm->new_len; + } + + err =resize_is_valid (vrm); +if (err) +return err;Quoted 8 lines
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1134
Quoted 6 lines
mm/mremap.c +2
-2/* MREMAP_DONTUNMAP expands by old_len since old_len == new_len */ -if (flags &MREMAP_DONTUNMAP && - !may_expand_vm (mm, vma->vm_flags, old_len >>PAGE_SHIFT )) { +if (vrm->flags &MREMAP_DONTUNMAP && + !may_expand_vm (mm, vrm->vma->vm_flags, vrm->old_len >>PAGE_SHIFT )) {return -ENOMEM ;nit: whitespace here is a bit odd to read.
Will find a way to fix, I agree this is a bit horrid.
Quoted 22 lines
Quoted 22 lines
mm/mremap.c +3
-15} -if (flags &MREMAP_FIXED ) - map_flags |=MAP_FIXED ; - -if (vma->vm_flags &VM_MAYSHARE ) - map_flags |=MAP_SHARED ; - - ret =get_unmapped_area (vma->vm_file, new_addr, new_len, vma->vm_pgoff + - ((addr - vma->vm_start) >>PAGE_SHIFT ), - map_flags); -if (IS_ERR_VALUE (ret)) -return ret; - -/* We got a new mapping */ -if (!(flags &MREMAP_FIXED )) - new_addr = ret; + err =vrm_set_new_addr (vrm); +if (err) +return err;Quoted 7 lines
mm/mremap.c static int resize_is_valid(struct vm_area_struct *vma, unsigned long addr, · line 1143
Quoted 5 lines
mm/mremap.c +3
-2-return move_vma (vma, addr, old_len, new_len, new_addr, locked, flags, - uf, uf_unmap); +return move_vma (vrm->vma, vrm->addr, vrm->old_len, vrm->new_len, + vrm->new_addr, &vrm->locked, vrm->flags, + vrm->uf, vrm->uf_unmap);I see where this is going..
;)
Quoted 242 lines
Quoted 226 lines
mm/mremap.c +54
-60}static int vma_expandable (struct vm_area_struct *vma,unsigned long delta)@@ -1016,22 +1157,33 @@ static int vma_expandable(struct vm_area_struct *vma, unsigned long delta)return 1 ; } -/* Do the mremap() flags require that the new_addr parameter be specified? */ -static bool implies_new_addr (unsigned long flags) +/* Determine whether we are actually able to execute an in-place expansion. */ +static bool vrm_can_expand_in_place (struct vma_remap_struct *vrm) { -return flags & (MREMAP_FIXED |MREMAP_DONTUNMAP ); +/* Number of bytes from vrm->addr to end of VMA. */ +unsigned long suffix_bytes = vrm->vma->vm_end - vrm->addr; + +/* If end of range aligns to end of VMA, we can just expand in-place. */ +if (suffix_bytes != vrm->old_len) +return false; + +/* Check whether this is feasible. */ +if (!vma_expandable (vrm->vma, vrm->delta)) +return false; + +return true; }/* * Are the parameters passed to mremap() valid? If so return 0, otherwise return * error. */ -static unsigned long check_mremap_params (unsigned long addr, -unsigned long flags, -unsigned long old_len, -unsigned long new_len, -unsigned long new_addr) +static unsigned long check_mremap_params (struct vma_remap_struct *vrm) + { +unsigned long addr = vrm->addr; +unsigned long flags = vrm->flags; +/* Ensure no unexpected flag values. */ if (flags & ~(MREMAP_FIXED |MREMAP_MAYMOVE |MREMAP_DONTUNMAP ))return -EINVAL ;@@ -1045,15 +1197,15 @@ static unsigned long check_mremap_params(unsigned long addr,* for DOS-emu "duplicate shm area" thing. But * a zero new-len is nonsensical. */ -if (!PAGE_ALIGN (new_len)) +if (!PAGE_ALIGN (vrm->new_len))return -EINVAL ;/* Remainder of checks are for cases with specific new_addr. */ -if (!implies_new_addr (flags)) +if (!vrm_implies_new_addr (vrm))return 0 ;/* The new address must be page-aligned. */ -if (offset_in_page (new_addr)) +if (offset_in_page (vrm->new_addr))return -EINVAL ;/* A fixed address implies a move. */ @@ -1061,7 +1213,7 @@ static unsigned long check_mremap_params(unsigned long addr,return -EINVAL ;/* MREMAP_DONTUNMAP does not allow resizing in the process. */ -if (flags &MREMAP_DONTUNMAP && old_len != new_len) +if (flags &MREMAP_DONTUNMAP && vrm->old_len != vrm->new_len)return -EINVAL ;/* @@ -1090,11 +1242,11 @@ static unsigned long check_mremap_params(unsigned long addr,* If we discover the VMA is locked, update mm_struct statistics accordingly and * indicate so to the caller. */ -static unsigned long expand_vma_inplace (struct vm_area_struct *vma, -unsigned long delta,bool *locked) +static unsigned long expand_vma_in_place (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; -long pages = delta >>PAGE_SHIFT ; +long pages = vrm->delta >>PAGE_SHIFT ; +struct vm_area_struct *vma = vrm->vma;VMA_ITERATOR (vmi, mm, vma->vm_end);long charged =0 ;@@ -1114,7 +1266,7 @@ static unsigned long expand_vma_inplace(struct vm_area_struct *vma,* adjacent to the expanded vma and otherwise * compatible. */ - vma =vma_merge_extend (&vmi, vma, delta); + vma = vrm->vma =vma_merge_extend (&vmi, vma, vrm->delta);if (!vma) {vm_unacct_memory (charged);return -ENOMEM ;@@ -1123,42 +1275,34 @@ static unsigned long expand_vma_inplace(struct vm_area_struct *vma,vm_stat_account (mm, vma->vm_flags, pages);if (vma->vm_flags &VM_LOCKED ) { mm->locked_vm += pages; - *locked = true; + vrm->locked = true; }return 0 ; } -static bool align_hugetlb (struct vm_area_struct *vma, -unsigned long addr, -unsigned long new_addr, -unsigned long *old_len_ptr, -unsigned long *new_len_ptr, -unsigned long *delta_ptr) +static bool align_hugetlb (struct vma_remap_struct *vrm) { -unsigned long old_len = *old_len_ptr; -unsigned long new_len = *new_len_ptr; -struct hstate *h__maybe_unused =hstate_vma (vma); +struct hstate *h__maybe_unused =hstate_vma (vrm->vma); - old_len =ALIGN (old_len,huge_page_size (h)); - new_len =ALIGN (new_len,huge_page_size (h)); + vrm->old_len =ALIGN (vrm->old_len,huge_page_size (h)); + vrm->new_len =ALIGN (vrm->new_len,huge_page_size (h));/* addrs must be huge page aligned */ -if (addr & ~huge_page_mask (h)) +if (vrm->addr & ~huge_page_mask (h))return false; -if (new_addr & ~huge_page_mask (h)) +if (vrm->new_addr & ~huge_page_mask (h))return false;/* * Don't allow remap expansion, because the underlying hugetlb * reservation is not yet capable to handle split reservation. */ -if (new_len > old_len) +if (vrm->new_len > vrm->old_len)return false; - *old_len_ptr = old_len; - *new_len_ptr = new_len; - *delta_ptr =abs_diff (old_len, new_len); +vrm_set_delta (vrm); +return true; }@@ -1169,19 +1313,16 @@ static bool align_hugetlb(struct vm_area_struct *vma,* Try to do so in-place, if this fails, then move the VMA to a new location to * action the change. */ -static unsigned long expand_vma (struct vm_area_struct *vma, -unsigned long addr,unsigned long old_len, -unsigned long new_len,unsigned long flags, -bool *locked_ptr,unsigned long *new_addr_ptr, -struct vm_userfaultfd_ctx *uf_ptr, -struct list_head *uf_unmap_ptr) +static unsigned long expand_vma (struct vma_remap_struct *vrm) {unsigned long err; -unsigned long map_flags; -unsigned long new_addr;/* We ignore any user-supplied one. */ -pgoff_t pgoff; +struct vm_area_struct *vma = vrm->vma; +unsigned long addr = vrm->addr; +unsigned long old_len = vrm->old_len; +unsigned long new_len = vrm->new_len; +unsigned long flags = vrm->flags; - err =resize_is_valid (vma, addr, old_len, new_len, flags); + err =resize_is_valid (vrm);if (err)return err;@@ -1189,10 +1330,9 @@ static unsigned long expand_vma(struct vm_area_struct *vma,* [addr, old_len) spans precisely to the end of the VMA, so try to * expand it in-place. */ -if (old_len == vma->vm_end - addr && -vma_expandable (vma, new_len - old_len)) { - err =expand_vma_inplace (vma, new_len - old_len, locked_ptr); -if (IS_ERR_VALUE (err)) +if (vrm_can_expand_in_place (vrm)) { + err =expand_vma_in_place (vrm); +if (err)return err;/* @@ -1200,8 +1340,8 @@ static unsigned long expand_vma(struct vm_area_struct *vma,* satisfy the expectation that mlock()'ing a VMA maintains all * of its pages in memory. */ -if (*locked_ptr) - *new_addr_ptr = addr; +if (vrm->locked) + vrm->new_addr = addr;/* OK we're done! */ return addr;@@ -1217,62 +1357,65 @@ static unsigned long expand_vma(struct vm_area_struct *vma,return -ENOMEM ;/* Find a new location to move the VMA to. */ - map_flags = (vma->vm_flags &VM_MAYSHARE ) ?MAP_SHARED :0 ; - pgoff = vma->vm_pgoff + ((addr - vma->vm_start) >>PAGE_SHIFT ); - new_addr =get_unmapped_area (vma->vm_file,0 , new_len, pgoff, map_flags); -if (IS_ERR_VALUE (new_addr)) -return new_addr; - *new_addr_ptr = new_addr; + err =vrm_set_new_addr (vrm); +if (err) +return err; -return move_vma (vma, addr, old_len, new_len, new_addr, - locked_ptr, flags, uf_ptr, uf_unmap_ptr); +return move_vma (vma, addr, old_len, new_len, vrm->new_addr, + &vrm->locked, flags, vrm->uf, vrm->uf_unmap); }/* -* Expand (or shrink) an existing mapping, potentially moving it at the -* same time (controlled by the MREMAP_MAYMOVE flag and available VM space) mm/mremap.c static unsigned long expand_vma(struct vm_area_struct *vma, · line 1372
Quoted 10 lines
mm/mremap.c +3
-6-* -* MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise -* This option implies MREMAP_MAYMOVE. +* Attempt to resize the VMA in-place, if we cannot, then move the VMA to the +* first available address to perform the operation. */ -SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, - unsigned long, new_len, unsigned long, flags, - unsigned long, new_addr) +static unsigned long mremap_at (struct vma_remap_struct *vrm)I hate this and
mremap_to()names. I don't have a proposed better name for either and maybe it's just my experience withmremap_to()that has tainted the view of the name itself, but I find them not very descriptive and abruptly ended. I guess move was already taken.Quoted 4 lines
I also have the added baggage of parsing "at" to potentially mean "doing".
mremap_inplace()seems equally annoying. This is the worst bike shed.Haha well yeah it's a bikeshed but yeah I agree, it's horrid I agree.
Quoted 7 lines
I was using the
_at()only in line with the_to().The 'in place' can kind of conflict with other references to in place which is used to differentiate between the case of us having to
get_unmapped_area()and move vs. not having to.So I feel like we're stuck with this...
Yes, I agree.
Quoted 150 lines
Quoted 138 lines
Quoted 119 lines
mm/mremap.c +42
-49+{ +unsigned long res; + +switch (vrm->remap_type) { +case MREMAP_NO_RESIZE : +/* NO-OP CASE - resizing to the same size. */ +return vrm->addr; +case MREMAP_SHRINK : +/* +* SHRINK CASE. Can always be done in-place. +* +* Simply unmap the shrunken portion of the VMA. This does all +* the needed commit accounting, and we indicate that the mmap +* lock should be dropped. +*/ + res =shrink_vma (vrm,/* drop_lock= */ true); +if (res) +return res; + +return vrm->addr; +case MREMAP_EXPAND : +return expand_vma (vrm); + } + +BUG (); +} + +static unsigned long do_mremap (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm;struct vm_area_struct *vma;unsigned long ret; -unsigned long delta; -bool locked = false; -struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ; -LIST_HEAD (uf_unmap_early); -LIST_HEAD (uf_unmap); -/* -* There is a deliberate asymmetry here: we strip the pointer tag -* from the old address but leave the new address alone. This is -* for consistency with mmap(), where we prevent the creation of -* aliasing mappings in userspace by leaving the tag bits of the -* mapping address intact. A non-zero tag will cause the subsequent -* range checks to reject the address as invalid. -* -* See Documentation/arch/arm64/tagged-address-abi.rst for more -* information. -*/ - addr =untagged_addr (addr); - - ret =check_mremap_params (addr, flags, old_len, new_len, new_addr); + ret =check_mremap_params (vrm);if (ret)return ret; - old_len =PAGE_ALIGN (old_len); - new_len =PAGE_ALIGN (new_len); - delta =abs_diff (old_len, new_len); + vrm->old_len =PAGE_ALIGN (vrm->old_len); + vrm->new_len =PAGE_ALIGN (vrm->new_len); +vrm_set_delta (vrm);if (mmap_write_lock_killable (mm))return -EINTR ; + vrm->mmap_locked = true; - vma =vma_lookup (mm, addr); + vma = vrm->vma =vma_lookup (mm, vrm->addr);if (!vma) { ret = -EFAULT ; goto out;@@ -1285,62 +1428,68 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, }/* Align to hugetlb page size, if required. */ -if (is_vm_hugetlb_page (vma) && - !align_hugetlb (vma, addr, new_addr, &old_len, &new_len, &delta)) { +if (is_vm_hugetlb_page (vma) && !align_hugetlb (vrm)) { ret = -EINVAL ; goto out; } -/* Are we RELOCATING the VMA to a SPECIFIC address? */ -if (implies_new_addr (flags)) { - ret =mremap_to (addr, old_len, new_addr, new_len, - &locked, flags, &uf, &uf_unmap_early, - &uf_unmap); - goto out; - } +vrm_set_remap_type (vrm); -/* -* From here on in we are only RESIZING the VMA, attempting to do so -* in-place, moving the VMA if we cannot. -*/ +/* Actually execute mremap. */ + ret =vrm_implies_new_addr (vrm) ?mremap_to (vrm) :mremap_at (vrm); -/* NO-OP CASE - resizing to the same size. */ -if (new_len == old_len) { - ret = addr; - goto out; - } - -/* SHRINK CASE. Can always be done in-place. */ -if (new_len < old_len) { -VMA_ITERATOR (vmi, mm, addr + new_len); +out: +if (vrm->mmap_locked) { +mmap_write_unlock (mm); + vrm->mmap_locked = false; -/* -* Simply unmap the shrunken portion of the VMA. This does all -* the needed commit accounting, unlocking the mmap lock. -*/ mm/mremap.c SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, · line 1447
Quoted 10 lines
mm/mremap.c +2
-8- ret =do_vmi_munmap (&vmi, mm, addr + new_len, delta, - &uf_unmap, true); -if (ret) - goto out; - -/* We succeeded, mmap lock released for us. */ - ret = addr; - goto out_unlocked; +if (!offset_in_page (ret) && vrm->locked && vrm->new_len > vrm->old_len) +mm_populate (vrm->new_addr + vrm->old_len, vrm->delta);It isn't clear to me why we only populate if we are locked here. Actually, I'm not sure why we keep holding the lock until here or why it matters to drop it early. The main reason we want to drop the lock is to reduce the mmap lock time held for the populate operation.
So we can either drop the lock before we get here once the vma tree is updated, or we can unconditionally unlock.
Quoted 8 lines
I think we can simplify this further if we just keep the lock held until we downgrade here and populate if necessary after it's dropped. That is, shrink just does nothing with the lock and we just unlock it regardless.
I'm pretty sure that we would struggle to measure the performance impact holding the lock for the return path when the populate is removed from the critical section.
I'm in agreement that this is a horrid and weird mess, and I stared at this a long time myself.
Quoted 6 lines
But I feel we should defer changing this until after the refactor series as it was an existing eldrich horror changing of which would sort of sit outside of 'obvious small fixups/safe moving around/etc.' that I limited myself to here.
Let's definitely come back and address this.
Ack, yeah. Shouldn't change locking from a known mess to an unknown but-maybe-cleaner mess in a refactor.
Quoted 66 lines
Quoted 66 lines
Quoted 66 lines
mm/mremap.c +43
-14} -/* EXPAND case. We try to do in-place, if we can't, then we move it. */ - ret =expand_vma (vma, addr, old_len, new_len, flags, &locked, &new_addr, - &uf, &uf_unmap); +userfaultfd_unmap_complete (mm, vrm->uf_unmap_early); +mremap_userfaultfd_complete (vrm->uf, vrm->addr, ret, vrm->old_len); +userfaultfd_unmap_complete (mm, vrm->uf_unmap); -out: -if (offset_in_page (ret)) - locked = false; -mmap_write_unlock (mm); -if (locked && new_len > old_len) -mm_populate (new_addr + old_len, delta); -out_unlocked: -userfaultfd_unmap_complete (mm, &uf_unmap_early); -mremap_userfaultfd_complete (&uf, addr, ret, old_len); -userfaultfd_unmap_complete (mm, &uf_unmap);return ret; } + +/* +* Expand (or shrink) an existing mapping, potentially moving it at the +* same time (controlled by the MREMAP_MAYMOVE flag and available VM space) +* +* MREMAP_FIXED option added 5-Dec-1999 by Benjamin LaHaise +* This option implies MREMAP_MAYMOVE. +*/ +SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, + unsigned long, new_len, unsigned long, flags, + unsigned long, new_addr) +{ +struct vm_userfaultfd_ctx uf =NULL_VM_UFFD_CTX ; +LIST_HEAD (uf_unmap_early); +LIST_HEAD (uf_unmap); +/* +* There is a deliberate asymmetry here: we strip the pointer tag +* from the old address but leave the new address alone. This is +* for consistency with mmap(), where we prevent the creation of +* aliasing mappings in userspace by leaving the tag bits of the +* mapping address intact. A non-zero tag will cause the subsequent +* range checks to reject the address as invalid. +* +* See Documentation/arch/arm64/tagged-address-abi.rst for more +* information. +*/ +struct vma_remap_struct vrm = { + .addr =untagged_addr (addr), + .old_len = old_len, + .new_len = new_len, + .flags = flags, + .new_addr = new_addr, + + .uf = &uf, + .uf_unmap_early = &uf_unmap_early, + .uf_unmap = &uf_unmap, + }; + +return do_mremap (&vrm); +} --2.48.1
PATCH 4/7 mm/mremap: initial refactor of move_vma()
In reply to Lorenzo StoakesUpdate move_vma() to use the threaded VRM object, de-duplicate code and separate into smaller functions to aid readability and debug-ability.
This in turn allows further simplification of expand_vma() as we can simply thread VRM through the function.
We also take the opportunity to abstract the account charging page count into the VRM in order that we can correctly thread this through the operation.
We additionally do the same for tracking mm statistics - exec_vm, stack_vm, data_vm, and locked_vm.
As part of this change, we slightly modify when locked pages statistics are counted for in mm_struct statistics. However this should cause no issues, as there is no chance of underflow, nor will any rlimit failures occur as a result.
This is an intermediate step before a further refactoring of move_vma() in order to aid review.
--- mm/mremap.c | 186 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 122 insertions(+), 64 deletions(-)
mm/mremap.c +122 -64
@@ -67,6 +67,7 @@ struct vma_remap_struct {bool locked;/* Was the VMA mlock()'d (has the VM_LOCKED flag set). */ enum mremap_operation remap_type;/* expand, shrink, etc. */ bool mmap_locked;/* Is current->mm currently write-locked? */ +unsigned long charged;/* If VM_ACCOUNT, # pages to account. */ };static pud_t *get_old_pud (struct mm_struct *mm,unsigned long addr)
@@ -821,35 +822,88 @@ static unsigned long vrm_set_new_addr(struct vma_remap_struct *vrm)return 0 ; } -static unsigned long move_vma (struct vm_area_struct *vma, -unsigned long old_addr,unsigned long old_len, -unsigned long new_len,unsigned long new_addr, -bool *locked,unsigned long flags, -struct vm_userfaultfd_ctx *uf,struct list_head *uf_unmap) +/* +* Keep track of pages which have been added to the memory mapping. If the VMA +* is accounted, also check to see if there is sufficient memory. +* +* Returns true on success, false if insufficient memory to charge. +*/ +static bool vrm_charge (struct vma_remap_struct *vrm) { -long to_account = new_len - old_len; -struct mm_struct *mm = vma->vm_mm; -struct vm_area_struct *new_vma; -unsigned long vm_flags = vma->vm_flags; -unsigned long new_pgoff; -unsigned long moved_len; -bool account_start = false; -bool account_end = false; -unsigned long hiwater_vm; -int err =0 ; -bool need_rmap_locks; -struct vma_iterator vmi; +unsigned long charged; + +if (!(vrm->vma->vm_flags &VM_ACCOUNT )) +return true; + +/* +* If we don't unmap the old mapping, then we account the entirety of +* the length of the new one. Otherwise it's just the delta in size. +*/ +if (vrm->flags &MREMAP_DONTUNMAP ) + charged = vrm->new_len >>PAGE_SHIFT ; +else + charged = vrm->delta >>PAGE_SHIFT ; + + +/* This accounts 'charged' pages of memory. */ +if (security_vm_enough_memory_mm (current->mm, charged)) +return false; + + vrm->charged = charged; +return true; +} + +/* +* an error has occurred so we will not be using vrm->charged memory. Unaccount +* this memory if the VMA is accounted. +*/ +static void vrm_uncharge (struct vma_remap_struct *vrm) +{ +if (!(vrm->vma->vm_flags &VM_ACCOUNT )) +return ; + +vm_unacct_memory (vrm->charged); + vrm->charged =0 ; +} + +/* +* Update mm exec_vm, stack_vm, data_vm, and locked_vm fields as needed to +* account for 'bytes' memory used, and if locked, indicate this in the VRM so +* we can handle this correctly later. +*/ +static void vrm_stat_account (struct vma_remap_struct *vrm, +unsigned long bytes) +{ +unsigned long pages = bytes >>PAGE_SHIFT ; +struct mm_struct *mm = current->mm; +struct vm_area_struct *vma = vrm->vma; + +vm_stat_account (mm, vma->vm_flags, pages); +if (vma->vm_flags &VM_LOCKED ) { + mm->locked_vm += pages; + vrm->locked = true; + } +} + +/* +* Perform checks before attempting to write a VMA prior to it being +* moved. +*/ +static unsigned long prep_move_vma (struct vma_remap_struct *vrm, +unsigned long *vm_flags_ptr) +{ +unsigned long err; +struct vm_area_struct *vma = vrm->vma; +unsigned long old_addr = vrm->addr; +unsigned long old_len = vrm->old_len;/* * We'd prefer to avoid failure later on in do_munmap: * which may split one vma into three before unmapping. */ -if (mm->map_count >= sysctl_max_map_count -3 ) +if (current->mm->map_count >= sysctl_max_map_count -3 )return -ENOMEM ; -if (unlikely (flags &MREMAP_DONTUNMAP )) - to_account = new_len; -if (vma->vm_ops && vma->vm_ops->may_split) {if (vma->vm_start != old_addr) err = vma->vm_ops->may_split (vma, old_addr);
@@ -867,22 +921,46 @@ static unsigned long move_vma(struct vm_area_struct *vma,* so KSM can come around to merge on vma and new_vma afterwards. */ err =ksm_madvise (vma, old_addr, old_addr + old_len, -MADV_UNMERGEABLE , &vm_flags); +MADV_UNMERGEABLE , vm_flags_ptr);if (err)return err; -if (vm_flags &VM_ACCOUNT ) { -if (security_vm_enough_memory_mm (mm, to_account >>PAGE_SHIFT )) -return -ENOMEM ; - } +return 0 ; +} + +static unsigned long move_vma (struct vma_remap_struct *vrm) +{ +struct mm_struct *mm = current->mm; +struct vm_area_struct *vma = vrm->vma; +struct vm_area_struct *new_vma; +unsigned long vm_flags = vma->vm_flags; +unsigned long old_addr = vrm->addr, new_addr = vrm->new_addr; +unsigned long old_len = vrm->old_len, new_len = vrm->new_len; +unsigned long new_pgoff; +unsigned long moved_len; +unsigned long account_start = false; +unsigned long account_end = false; +unsigned long hiwater_vm; +int err; +bool need_rmap_locks; +struct vma_iterator vmi; + + err =prep_move_vma (vrm, &vm_flags); +if (err) +return err; + +/* If accounted, charge the number of bytes the operation will use. */ +if (!vrm_charge (vrm)) +return -ENOMEM ;vma_start_write (vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >>PAGE_SHIFT ); - new_vma =copy_vma (&vma, new_addr, new_len, new_pgoff, + new_vma =copy_vma (&vrm->vma, new_addr, new_len, new_pgoff, &need_rmap_locks); +/* This may have been updated. */ + vma = vrm->vma;if (!new_vma) { -if (vm_flags &VM_ACCOUNT ) -vm_unacct_memory (to_account >>PAGE_SHIFT ); +vrm_uncharge (vrm);return -ENOMEM ; }
@@ -907,7 +985,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, old_addr = new_addr; new_addr = err; }else { -mremap_userfaultfd_prep (new_vma, uf); +mremap_userfaultfd_prep (new_vma, vrm->uf); }if (is_vm_hugetlb_page (vma)) {
@@ -915,7 +993,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, }/* Conceal VM_ACCOUNT so old reservation is not undone */ -if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP )) { +if (vm_flags &VM_ACCOUNT && !(vrm->flags &MREMAP_DONTUNMAP )) {vm_flags_clear (vma,VM_ACCOUNT );if (vma->vm_start < old_addr) account_start = true;
@@ -933,13 +1011,12 @@ static unsigned long move_vma(struct vm_area_struct *vma,* If this were a serious issue, we'd add a flag to do_munmap(). */ hiwater_vm = mm->hiwater_vm; -vm_stat_account (mm, vma->vm_flags, new_len >>PAGE_SHIFT );/* Tell pfnmap has moved from this vma */ if (unlikely (vma->vm_flags &VM_PFNMAP ))untrack_pfn_clear (vma); -if (unlikely (!err && (flags &MREMAP_DONTUNMAP ))) { +if (unlikely (!err && (vrm->flags &MREMAP_DONTUNMAP ))) {/* We always clear VM_LOCKED[ONFAULT] on the old vma */ vm_flags_clear (vma,VM_LOCKED_MASK );
@@ -952,22 +1029,20 @@ static unsigned long move_vma(struct vm_area_struct *vma,unlink_anon_vmas (vma);/* Because we won't unmap we don't need to touch locked_vm */ +vrm_stat_account (vrm, new_len);return new_addr; } +vrm_stat_account (vrm, new_len); +vma_iter_init (&vmi, mm, old_addr); -if (do_vmi_munmap (&vmi, mm, old_addr, old_len, uf_unmap, false) <0 ) { +if (do_vmi_munmap (&vmi, mm, old_addr, old_len, vrm->uf_unmap, false) <0 ) {/* OOM: unable to split vma, just get accounts right */ -if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP )) +if (vm_flags &VM_ACCOUNT && !(vrm->flags &MREMAP_DONTUNMAP ))vm_acct_memory (old_len >>PAGE_SHIFT ); account_start = account_end = false; } -if (vm_flags &VM_LOCKED ) { - mm->locked_vm += new_len >>PAGE_SHIFT ; - *locked = true; - } - mm->hiwater_vm = hiwater_vm;/* Restore VM_ACCOUNT if one or two pieces of vma left */
@@ -1138,9 +1213,7 @@ static unsigned long mremap_to(struct vma_remap_struct *vrm)if (err)return err; -return move_vma (vrm->vma, vrm->addr, vrm->old_len, vrm->new_len, - vrm->new_addr, &vrm->locked, vrm->flags, - vrm->uf, vrm->uf_unmap); +return move_vma (vrm); }static int vma_expandable (struct vm_area_struct *vma,unsigned long delta)
@@ -1245,17 +1318,11 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm)static unsigned long expand_vma_in_place (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; -long pages = vrm->delta >>PAGE_SHIFT ;struct vm_area_struct *vma = vrm->vma;VMA_ITERATOR (vmi, mm, vma->vm_end); -long charged =0 ; - -if (vma->vm_flags &VM_ACCOUNT ) { -if (security_vm_enough_memory_mm (mm, pages)) -return -ENOMEM ; - charged = pages; - } +if (!vrm_charge (vrm)) +return -ENOMEM ;/* * Function vma_merge_extend() is called on the
@@ -1268,15 +1335,11 @@ static unsigned long expand_vma_in_place(struct vma_remap_struct *vrm)*/ vma = vrm->vma =vma_merge_extend (&vmi, vma, vrm->delta);if (!vma) { -vm_unacct_memory (charged); +vrm_uncharge (vrm);return -ENOMEM ; } -vm_stat_account (mm, vma->vm_flags, pages); -if (vma->vm_flags &VM_LOCKED ) { - mm->locked_vm += pages; - vrm->locked = true; - } +vrm_stat_account (vrm, vrm->delta);return 0 ; }
@@ -1316,11 +1379,7 @@ static bool align_hugetlb(struct vma_remap_struct *vrm)static unsigned long expand_vma (struct vma_remap_struct *vrm) {unsigned long err; -struct vm_area_struct *vma = vrm->vma;unsigned long addr = vrm->addr; -unsigned long old_len = vrm->old_len; -unsigned long new_len = vrm->new_len; -unsigned long flags = vrm->flags; err =resize_is_valid (vrm);if (err)
@@ -1353,7 +1412,7 @@ static unsigned long expand_vma(struct vma_remap_struct *vrm)*/ /* We're not allowed to move the VMA, so error out. */ -if (!(flags &MREMAP_MAYMOVE )) +if (!(vrm->flags &MREMAP_MAYMOVE ))return -ENOMEM ;/* Find a new location to move the VMA to. */
@@ -1361,8 +1420,7 @@ static unsigned long expand_vma(struct vma_remap_struct *vrm)if (err)return err; -return move_vma (vma, addr, old_len, new_len, vrm->new_addr, - &vrm->locked, flags, vrm->uf, vrm->uf_unmap); +return move_vma (vrm); }/*
Signature · 1 line
2.48.1
Re: [PATCH 4/7] mm/mremap: initial refactor of move_vma()
Quoted 23 lines · Lorenzo Stoakes
Update
move_vma()to use the threaded VRM object, de-duplicate code and separate into smaller functions to aid readability and debug-ability.This in turn allows further simplification of
expand_vma()as we can simply thread VRM through the function.We also take the opportunity to abstract the account charging page count into the VRM in order that we can correctly thread this through the operation.
We additionally do the same for tracking mm statistics -
exec_vm,stack_vm,data_vm, andlocked_vm.As part of this change, we slightly modify when locked pages statistics are counted for in
mm_structstatistics. However this should cause no issues, as there is no chance of underflow, nor will any rlimit failures occur as a result.This is an intermediate step before a further refactoring of
move_vma()in order to aid review.Signed-off-by Lorenzo Stoakes <[email protected]>---
[..]
mm/mremap.c static unsigned long vrm_set_new_addr(struct vma_remap_struct *vrm) · line 895
Quoted 8 lines
mm/mremap.c +8
-0+/* +* Perform checks before attempting to write a VMA prior to it being +* moved. +*/ +static unsigned long prep_move_vma (struct vma_remap_struct *vrm, +unsigned long *vm_flags_ptr) +{ +unsigned long err;
I am getting a warning on mm-unstable because 'err' is sometimes used uninitialized, I think here:
if (vma->vm_ops && vma->vm_ops->may_split) {
if (vma->vm_start != old_addr)
err = vma->vm_ops->may_split(vma, old_addr);
if (!err && vma->vm_end != old_addr + old_len)
err = vma->vm_ops->may_split(vma, old_addr + old_len);
if (err)
return err;
}Quoted 18 lines
mm/mremap.c +4
-4+struct vm_area_struct *vma = vrm->vma; +unsigned long old_addr = vrm->addr; +unsigned long old_len = vrm->old_len;/* * We'd prefer to avoid failure later on in do_munmap: * which may split one vma into three before unmapping. */ -if (mm->map_count >= sysctl_max_map_count -3 ) +if (current->mm->map_count >= sysctl_max_map_count -3 )return -ENOMEM ; -if (unlikely (flags &MREMAP_DONTUNMAP )) - to_account = new_len; -if (vma->vm_ops && vma->vm_ops->may_split) {if (vma->vm_start != old_addr) err = vma->vm_ops->may_split (vma, old_addr);
[..]
Andrew Morton
maintainerRe: [PATCH 4/7] mm/mremap: initial refactor of move_vma()
Quoted 37 lines · On Tue, 4 Mar 2025 21:45:27 +0000 Yosry Ahmed <[email protected]> wrote:
Quoted 23 lines · Lorenzo Stoakes
Update
move_vma()to use the threaded VRM object, de-duplicate code and separate into smaller functions to aid readability and debug-ability.This in turn allows further simplification of
expand_vma()as we can simply thread VRM through the function.We also take the opportunity to abstract the account charging page count into the VRM in order that we can correctly thread this through the operation.
We additionally do the same for tracking mm statistics -
exec_vm,stack_vm,data_vm, andlocked_vm.As part of this change, we slightly modify when locked pages statistics are counted for in
mm_structstatistics. However this should cause no issues, as there is no chance of underflow, nor will any rlimit failures occur as a result.This is an intermediate step before a further refactoring of
move_vma()in order to aid review.Signed-off-by Lorenzo Stoakes <[email protected]>---
[..]
Review
Quoted 8 lines
(quoted diff) +8
-0+/* + * Perform checks before attempting to write a VMA prior to it being + * moved. + */ +static unsigned long prep_move_vma(struct vma_remap_struct *vrm, + unsigned long *vm_flags_ptr) +{ + unsigned long err;I am getting a warning on mm-unstable because 'err' is sometimes used uninitialized, I think here:
Quoted 8 lines
if (vma->vm_ops && vma->vm_ops->may_split) {
if (vma->vm_start != old_addr)
err = vma->vm_ops->may_split(vma, old_addr);
if (!err && vma->vm_end != old_addr + old_len)
err = vma->vm_ops->may_split(vma, old_addr + old_len);
if (err)
return err;
}yep, thanks. I added this:
mm/mremap.c renamed from mm/mremap.c~mm-mremap-initial-refactor-of-move_vma-fix +1 -1
@@ -892,7 +892,7 @@ static void vrm_stat_account(struct vma_static unsigned long prep_move_vma (struct vma_remap_struct *vrm,unsigned long *vm_flags_ptr) { -unsigned long err; +unsigned long err =0 ;struct vm_area_struct *vma = vrm->vma;unsigned long old_addr = vrm->addr;unsigned long old_len = vrm->old_len;
_
Re: [PATCH 4/7] mm/mremap: initial refactor of move_vma()
Quoted 61 lines · Andrew Morton
Quoted 37 lines · On Tue, 4 Mar 2025 21:45:27 +0000 Yosry Ahmed <[email protected]> wrote:
Quoted 23 lines · Lorenzo Stoakes
Update
move_vma()to use the threaded VRM object, de-duplicate code and separate into smaller functions to aid readability and debug-ability.This in turn allows further simplification of
expand_vma()as we can simply thread VRM through the function.We also take the opportunity to abstract the account charging page count into the VRM in order that we can correctly thread this through the operation.
We additionally do the same for tracking mm statistics -
exec_vm,stack_vm,data_vm, andlocked_vm.As part of this change, we slightly modify when locked pages statistics are counted for in
mm_structstatistics. However this should cause no issues, as there is no chance of underflow, nor will any rlimit failures occur as a result.This is an intermediate step before a further refactoring of
move_vma()in order to aid review.Signed-off-by Lorenzo Stoakes <[email protected]>---
[..]
Review
Quoted 8 lines
(quoted diff) +8
-0+/* + * Perform checks before attempting to write a VMA prior to it being + * moved. + */ +static unsigned long prep_move_vma(struct vma_remap_struct *vrm, + unsigned long *vm_flags_ptr) +{ + unsigned long err;I am getting a warning on mm-unstable because 'err' is sometimes used uninitialized, I think here:
Quoted 8 lines
if (vma->vm_ops && vma->vm_ops->may_split) { if (vma->vm_start != old_addr) err = vma->vm_ops->may_split(vma, old_addr); if (!err && vma->vm_end != old_addr + old_len) err = vma->vm_ops->may_split(vma, old_addr + old_len); if (err) return err; }yep, thanks. I added this:
mm/mremap.c renamed from mm/mremap.c~mm-mremap-initial-refactor-of-move_vma-fix +1
-1@@ -892,7 +892,7 @@ static void vrm_stat_account(struct vma_static unsigned long prep_move_vma (struct vma_remap_struct *vrm,unsigned long *vm_flags_ptr) { -unsigned long err; +unsigned long err =0 ;struct vm_area_struct *vma = vrm->vma;unsigned long old_addr = vrm->addr;unsigned long old_len = vrm->old_len;
Quoted 2 lines
_
Thanks Andrew! Apologies guys, my bad. I don't know why this wasn't flagged locally... :/
This fix looks correct!
Re: [PATCH 4/7] mm/mremap: initial refactor of move_vma()
In reply to Lorenzo StoakesQuoted 18 lines · Lorenzo Stoakes
Update
move_vma()to use the threaded VRM object, de-duplicate code and separate into smaller functions to aid readability and debug-ability.This in turn allows further simplification of
expand_vma()as we can simply thread VRM through the function.We also take the opportunity to abstract the account charging page count into the VRM in order that we can correctly thread this through the operation.
We additionally do the same for tracking mm statistics -
exec_vm,stack_vm,data_vm, andlocked_vm.As part of this change, we slightly modify when locked pages statistics are counted for in
mm_structstatistics. However this should cause no issues, as there is no chance of underflow, nor will any rlimit failures occur as a result.
Quoted 4 lines
This is an intermediate step before a further refactoring of
move_vma()in order to aid review.Signed-off-by Lorenzo Stoakes <[email protected]>
Quoted 109 lines
--- mm/mremap.c | 186 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 122 insertions(+), 64 deletions(-)mm/mremap.c +72
-17@@ -67,6 +67,7 @@ struct vma_remap_struct {bool locked;/* Was the VMA mlock()'d (has the VM_LOCKED flag set). */ enum mremap_operation remap_type;/* expand, shrink, etc. */ bool mmap_locked;/* Is current->mm currently write-locked? */ +unsigned long charged;/* If VM_ACCOUNT, # pages to account. */ };static pud_t *get_old_pud (struct mm_struct *mm,unsigned long addr)@@ -821,35 +822,88 @@ static unsigned long vrm_set_new_addr(struct vma_remap_struct *vrm)return 0 ; } -static unsigned long move_vma (struct vm_area_struct *vma, -unsigned long old_addr,unsigned long old_len, -unsigned long new_len,unsigned long new_addr, -bool *locked,unsigned long flags, -struct vm_userfaultfd_ctx *uf,struct list_head *uf_unmap) +/* +* Keep track of pages which have been added to the memory mapping. If the VMA +* is accounted, also check to see if there is sufficient memory. +* +* Returns true on success, false if insufficient memory to charge. +*/ +static bool vrm_charge (struct vma_remap_struct *vrm) { -long to_account = new_len - old_len; -struct mm_struct *mm = vma->vm_mm; -struct vm_area_struct *new_vma; -unsigned long vm_flags = vma->vm_flags; -unsigned long new_pgoff; -unsigned long moved_len; -bool account_start = false; -bool account_end = false; -unsigned long hiwater_vm; -int err =0 ; -bool need_rmap_locks; -struct vma_iterator vmi; +unsigned long charged; + +if (!(vrm->vma->vm_flags &VM_ACCOUNT )) +return true; + +/* +* If we don't unmap the old mapping, then we account the entirety of +* the length of the new one. Otherwise it's just the delta in size. +*/ +if (vrm->flags &MREMAP_DONTUNMAP ) + charged = vrm->new_len >>PAGE_SHIFT ; +else + charged = vrm->delta >>PAGE_SHIFT ; + + +/* This accounts 'charged' pages of memory. */ +if (security_vm_enough_memory_mm (current->mm, charged)) +return false; + + vrm->charged = charged; +return true; +} + +/* +* an error has occurred so we will not be using vrm->charged memory. Unaccount +* this memory if the VMA is accounted. +*/ +static void vrm_uncharge (struct vma_remap_struct *vrm) +{ +if (!(vrm->vma->vm_flags &VM_ACCOUNT )) +return ; + +vm_unacct_memory (vrm->charged); + vrm->charged =0 ; +} + +/* +* Update mm exec_vm, stack_vm, data_vm, and locked_vm fields as needed to +* account for 'bytes' memory used, and if locked, indicate this in the VRM so +* we can handle this correctly later. +*/ +static void vrm_stat_account (struct vma_remap_struct *vrm, +unsigned long bytes) +{ +unsigned long pages = bytes >>PAGE_SHIFT ; +struct mm_struct *mm = current->mm; +struct vm_area_struct *vma = vrm->vma; + +vm_stat_account (mm, vma->vm_flags, pages); +if (vma->vm_flags &VM_LOCKED ) { + mm->locked_vm += pages; + vrm->locked = true; + } +} + +/* +* Perform checks before attempting to write a VMA prior to it being +* moved. +*/ +static unsigned long prep_move_vma (struct vma_remap_struct *vrm, +unsigned long *vm_flags_ptr) +{ +unsigned long err; +struct vm_area_struct *vma = vrm->vma;
mm/mremap.c static unsigned long vrm_set_new_addr(struct vma_remap_struct *vrm) · line 905
Quoted 10 lines
mm/mremap.c +3
-1+unsigned long old_addr = vrm->addr; +unsigned long old_len = vrm->old_len;/* * We'd prefer to avoid failure later on in do_munmap: * which may split one vma into three before unmapping. */ -if (mm->map_count >= sysctl_max_map_count -3 ) +if (current->mm->map_count >= sysctl_max_map_count -3 )return -ENOMEM ;
With the refactoring, it is pointing out some things that need to be reworked at a later date.
Quoted 206 lines
mm/mremap.c +47
-46-if (unlikely (flags &MREMAP_DONTUNMAP )) - to_account = new_len; -if (vma->vm_ops && vma->vm_ops->may_split) {if (vma->vm_start != old_addr) err = vma->vm_ops->may_split (vma, old_addr);@@ -867,22 +921,46 @@ static unsigned long move_vma(struct vm_area_struct *vma,* so KSM can come around to merge on vma and new_vma afterwards. */ err =ksm_madvise (vma, old_addr, old_addr + old_len, -MADV_UNMERGEABLE , &vm_flags); +MADV_UNMERGEABLE , vm_flags_ptr);if (err)return err; -if (vm_flags &VM_ACCOUNT ) { -if (security_vm_enough_memory_mm (mm, to_account >>PAGE_SHIFT )) -return -ENOMEM ; - } +return 0 ; +} + +static unsigned long move_vma (struct vma_remap_struct *vrm) +{ +struct mm_struct *mm = current->mm; +struct vm_area_struct *vma = vrm->vma; +struct vm_area_struct *new_vma; +unsigned long vm_flags = vma->vm_flags; +unsigned long old_addr = vrm->addr, new_addr = vrm->new_addr; +unsigned long old_len = vrm->old_len, new_len = vrm->new_len; +unsigned long new_pgoff; +unsigned long moved_len; +unsigned long account_start = false; +unsigned long account_end = false; +unsigned long hiwater_vm; +int err; +bool need_rmap_locks; +struct vma_iterator vmi; + + err =prep_move_vma (vrm, &vm_flags); +if (err) +return err; + +/* If accounted, charge the number of bytes the operation will use. */ +if (!vrm_charge (vrm)) +return -ENOMEM ;vma_start_write (vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >>PAGE_SHIFT ); - new_vma =copy_vma (&vma, new_addr, new_len, new_pgoff, + new_vma =copy_vma (&vrm->vma, new_addr, new_len, new_pgoff, &need_rmap_locks); +/* This may have been updated. */ + vma = vrm->vma;if (!new_vma) { -if (vm_flags &VM_ACCOUNT ) -vm_unacct_memory (to_account >>PAGE_SHIFT ); +vrm_uncharge (vrm);return -ENOMEM ; }@@ -907,7 +985,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, old_addr = new_addr; new_addr = err; }else { -mremap_userfaultfd_prep (new_vma, uf); +mremap_userfaultfd_prep (new_vma, vrm->uf); }if (is_vm_hugetlb_page (vma)) {@@ -915,7 +993,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, }/* Conceal VM_ACCOUNT so old reservation is not undone */ -if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP )) { +if (vm_flags &VM_ACCOUNT && !(vrm->flags &MREMAP_DONTUNMAP )) {vm_flags_clear (vma,VM_ACCOUNT );if (vma->vm_start < old_addr) account_start = true;@@ -933,13 +1011,12 @@ static unsigned long move_vma(struct vm_area_struct *vma,* If this were a serious issue, we'd add a flag to do_munmap(). */ hiwater_vm = mm->hiwater_vm; -vm_stat_account (mm, vma->vm_flags, new_len >>PAGE_SHIFT );/* Tell pfnmap has moved from this vma */ if (unlikely (vma->vm_flags &VM_PFNMAP ))untrack_pfn_clear (vma); -if (unlikely (!err && (flags &MREMAP_DONTUNMAP ))) { +if (unlikely (!err && (vrm->flags &MREMAP_DONTUNMAP ))) {/* We always clear VM_LOCKED[ONFAULT] on the old vma */ vm_flags_clear (vma,VM_LOCKED_MASK );@@ -952,22 +1029,20 @@ static unsigned long move_vma(struct vm_area_struct *vma,unlink_anon_vmas (vma);/* Because we won't unmap we don't need to touch locked_vm */ +vrm_stat_account (vrm, new_len);return new_addr; } +vrm_stat_account (vrm, new_len); +vma_iter_init (&vmi, mm, old_addr); -if (do_vmi_munmap (&vmi, mm, old_addr, old_len, uf_unmap, false) <0 ) { +if (do_vmi_munmap (&vmi, mm, old_addr, old_len, vrm->uf_unmap, false) <0 ) {/* OOM: unable to split vma, just get accounts right */ -if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP )) +if (vm_flags &VM_ACCOUNT && !(vrm->flags &MREMAP_DONTUNMAP ))vm_acct_memory (old_len >>PAGE_SHIFT ); account_start = account_end = false; } -if (vm_flags &VM_LOCKED ) { - mm->locked_vm += new_len >>PAGE_SHIFT ; - *locked = true; - } - mm->hiwater_vm = hiwater_vm;/* Restore VM_ACCOUNT if one or two pieces of vma left */ @@ -1138,9 +1213,7 @@ static unsigned long mremap_to(struct vma_remap_struct *vrm)if (err)return err; -return move_vma (vrm->vma, vrm->addr, vrm->old_len, vrm->new_len, - vrm->new_addr, &vrm->locked, vrm->flags, - vrm->uf, vrm->uf_unmap); +return move_vma (vrm); }static int vma_expandable (struct vm_area_struct *vma,unsigned long delta)@@ -1245,17 +1318,11 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm)static unsigned long expand_vma_in_place (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; -long pages = vrm->delta >>PAGE_SHIFT ;struct vm_area_struct *vma = vrm->vma;VMA_ITERATOR (vmi, mm, vma->vm_end); -long charged =0 ; - -if (vma->vm_flags &VM_ACCOUNT ) { -if (security_vm_enough_memory_mm (mm, pages)) -return -ENOMEM ; - charged = pages; - } +if (!vrm_charge (vrm)) +return -ENOMEM ;/* * Function vma_merge_extend() is called on the @@ -1268,15 +1335,11 @@ static unsigned long expand_vma_in_place(struct vma_remap_struct *vrm)*/ vma = vrm->vma =vma_merge_extend (&vmi, vma, vrm->delta);if (!vma) { -vm_unacct_memory (charged); +vrm_uncharge (vrm);return -ENOMEM ; } -vm_stat_account (mm, vma->vm_flags, pages); -if (vma->vm_flags &VM_LOCKED ) { - mm->locked_vm += pages; - vrm->locked = true; - } +vrm_stat_account (vrm, vrm->delta);return 0 ; }@@ -1316,11 +1379,7 @@ static bool align_hugetlb(struct vma_remap_struct *vrm)static unsigned long expand_vma (struct vma_remap_struct *vrm) {unsigned long err; -struct vm_area_struct *vma = vrm->vma;unsigned long addr = vrm->addr; -unsigned long old_len = vrm->old_len; -unsigned long new_len = vrm->new_len; -unsigned long flags = vrm->flags; err =resize_is_valid (vrm);if (err)@@ -1353,7 +1412,7 @@ static unsigned long expand_vma(struct vma_remap_struct *vrm)*/ /* We're not allowed to move the VMA, so error out. */ -if (!(flags &MREMAP_MAYMOVE )) +if (!(vrm->flags &MREMAP_MAYMOVE ))return -ENOMEM ;/* Find a new location to move the VMA to. */ @@ -1361,8 +1420,7 @@ static unsigned long expand_vma(struct vma_remap_struct *vrm)if (err)return err; -return move_vma (vma, addr, old_len, new_len, vrm->new_addr, - &vrm->locked, flags, vrm->uf, vrm->uf_unmap); +return move_vma (vrm); }/* Signature · 1 line
2.48.1
Re: [PATCH 4/7] mm/mremap: initial refactor of move_vma()
Quoted 19 lines · Liam R. Howlett
Quoted 18 lines · Lorenzo Stoakes
Update
move_vma()to use the threaded VRM object, de-duplicate code and separate into smaller functions to aid readability and debug-ability.This in turn allows further simplification of
expand_vma()as we can simply thread VRM through the function.We also take the opportunity to abstract the account charging page count into the VRM in order that we can correctly thread this through the operation.
We additionally do the same for tracking mm statistics -
exec_vm,stack_vm,data_vm, andlocked_vm.As part of this change, we slightly modify when locked pages statistics are counted for in
mm_structstatistics. However this should cause no issues, as there is no chance of underflow, nor will any rlimit failures occur as a result.
Quoted 6 lines
Quoted 4 lines
This is an intermediate step before a further refactoring of
move_vma()in order to aid review.Signed-off-by Lorenzo Stoakes <[email protected]>Reviewed-by Liam R. Howlett <[email protected]>
Thanks!
Quoted 113 lines
Quoted 113 lines
--- mm/mremap.c | 186 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 122 insertions(+), 64 deletions(-)mm/mremap.c +74
-17@@ -67,6 +67,7 @@ struct vma_remap_struct {bool locked;/* Was the VMA mlock()'d (has the VM_LOCKED flag set). */ enum mremap_operation remap_type;/* expand, shrink, etc. */ bool mmap_locked;/* Is current->mm currently write-locked? */ +unsigned long charged;/* If VM_ACCOUNT, # pages to account. */ };static pud_t *get_old_pud (struct mm_struct *mm,unsigned long addr)@@ -821,35 +822,88 @@ static unsigned long vrm_set_new_addr(struct vma_remap_struct *vrm)return 0 ; } -static unsigned long move_vma (struct vm_area_struct *vma, -unsigned long old_addr,unsigned long old_len, -unsigned long new_len,unsigned long new_addr, -bool *locked,unsigned long flags, -struct vm_userfaultfd_ctx *uf,struct list_head *uf_unmap) +/* +* Keep track of pages which have been added to the memory mapping. If the VMA +* is accounted, also check to see if there is sufficient memory. +* +* Returns true on success, false if insufficient memory to charge. +*/ +static bool vrm_charge (struct vma_remap_struct *vrm) { -long to_account = new_len - old_len; -struct mm_struct *mm = vma->vm_mm; -struct vm_area_struct *new_vma; -unsigned long vm_flags = vma->vm_flags; -unsigned long new_pgoff; -unsigned long moved_len; -bool account_start = false; -bool account_end = false; -unsigned long hiwater_vm; -int err =0 ; -bool need_rmap_locks; -struct vma_iterator vmi; +unsigned long charged; + +if (!(vrm->vma->vm_flags &VM_ACCOUNT )) +return true; + +/* +* If we don't unmap the old mapping, then we account the entirety of +* the length of the new one. Otherwise it's just the delta in size. +*/ +if (vrm->flags &MREMAP_DONTUNMAP ) + charged = vrm->new_len >>PAGE_SHIFT ; +else + charged = vrm->delta >>PAGE_SHIFT ; + + +/* This accounts 'charged' pages of memory. */ +if (security_vm_enough_memory_mm (current->mm, charged)) +return false; + + vrm->charged = charged; +return true; +} + +/* +* an error has occurred so we will not be using vrm->charged memory. Unaccount +* this memory if the VMA is accounted. +*/ +static void vrm_uncharge (struct vma_remap_struct *vrm) +{ +if (!(vrm->vma->vm_flags &VM_ACCOUNT )) +return ; + +vm_unacct_memory (vrm->charged); + vrm->charged =0 ; +} + +/* +* Update mm exec_vm, stack_vm, data_vm, and locked_vm fields as needed to +* account for 'bytes' memory used, and if locked, indicate this in the VRM so +* we can handle this correctly later. +*/ +static void vrm_stat_account (struct vma_remap_struct *vrm, +unsigned long bytes) +{ +unsigned long pages = bytes >>PAGE_SHIFT ; +struct mm_struct *mm = current->mm; +struct vm_area_struct *vma = vrm->vma; + +vm_stat_account (mm, vma->vm_flags, pages); +if (vma->vm_flags &VM_LOCKED ) { + mm->locked_vm += pages; + vrm->locked = true; + } +} + +/* +* Perform checks before attempting to write a VMA prior to it being +* moved. +*/ +static unsigned long prep_move_vma (struct vma_remap_struct *vrm, +unsigned long *vm_flags_ptr) +{ +unsigned long err; +struct vm_area_struct *vma = vrm->vma; +unsigned long old_addr = vrm->addr; +unsigned long old_len = vrm->old_len;
Quoted 10 lines
mm/mremap.c static unsigned long vrm_set_new_addr(struct vma_remap_struct *vrm) · line 905
Quoted 7 lines
mm/mremap.c +1
-1/* * We'd prefer to avoid failure later on in do_munmap: * which may split one vma into three before unmapping. */ -if (mm->map_count >= sysctl_max_map_count -3 ) +if (current->mm->map_count >= sysctl_max_map_count -3 )return -ENOMEM ;With the refactoring, it is pointing out some things that need to be reworked at a later date.
Yeah agreed, let's revisit this!
Quoted 207 lines
Quoted 207 lines
mm/mremap.c +47
-47-if (unlikely (flags &MREMAP_DONTUNMAP )) - to_account = new_len; -if (vma->vm_ops && vma->vm_ops->may_split) {if (vma->vm_start != old_addr) err = vma->vm_ops->may_split (vma, old_addr);@@ -867,22 +921,46 @@ static unsigned long move_vma(struct vm_area_struct *vma,* so KSM can come around to merge on vma and new_vma afterwards. */ err =ksm_madvise (vma, old_addr, old_addr + old_len, -MADV_UNMERGEABLE , &vm_flags); +MADV_UNMERGEABLE , vm_flags_ptr);if (err)return err; -if (vm_flags &VM_ACCOUNT ) { -if (security_vm_enough_memory_mm (mm, to_account >>PAGE_SHIFT )) -return -ENOMEM ; - } +return 0 ; +} + +static unsigned long move_vma (struct vma_remap_struct *vrm) +{ +struct mm_struct *mm = current->mm; +struct vm_area_struct *vma = vrm->vma; +struct vm_area_struct *new_vma; +unsigned long vm_flags = vma->vm_flags; +unsigned long old_addr = vrm->addr, new_addr = vrm->new_addr; +unsigned long old_len = vrm->old_len, new_len = vrm->new_len; +unsigned long new_pgoff; +unsigned long moved_len; +unsigned long account_start = false; +unsigned long account_end = false; +unsigned long hiwater_vm; +int err; +bool need_rmap_locks; +struct vma_iterator vmi; + + err =prep_move_vma (vrm, &vm_flags); +if (err) +return err; + +/* If accounted, charge the number of bytes the operation will use. */ +if (!vrm_charge (vrm)) +return -ENOMEM ;vma_start_write (vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >>PAGE_SHIFT ); - new_vma =copy_vma (&vma, new_addr, new_len, new_pgoff, + new_vma =copy_vma (&vrm->vma, new_addr, new_len, new_pgoff, &need_rmap_locks); +/* This may have been updated. */ + vma = vrm->vma;if (!new_vma) { -if (vm_flags &VM_ACCOUNT ) -vm_unacct_memory (to_account >>PAGE_SHIFT ); +vrm_uncharge (vrm);return -ENOMEM ; }@@ -907,7 +985,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, old_addr = new_addr; new_addr = err; }else { -mremap_userfaultfd_prep (new_vma, uf); +mremap_userfaultfd_prep (new_vma, vrm->uf); }if (is_vm_hugetlb_page (vma)) {@@ -915,7 +993,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, }/* Conceal VM_ACCOUNT so old reservation is not undone */ -if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP )) { +if (vm_flags &VM_ACCOUNT && !(vrm->flags &MREMAP_DONTUNMAP )) {vm_flags_clear (vma,VM_ACCOUNT );if (vma->vm_start < old_addr) account_start = true;@@ -933,13 +1011,12 @@ static unsigned long move_vma(struct vm_area_struct *vma,* If this were a serious issue, we'd add a flag to do_munmap(). */ hiwater_vm = mm->hiwater_vm; -vm_stat_account (mm, vma->vm_flags, new_len >>PAGE_SHIFT );/* Tell pfnmap has moved from this vma */ if (unlikely (vma->vm_flags &VM_PFNMAP ))untrack_pfn_clear (vma); -if (unlikely (!err && (flags &MREMAP_DONTUNMAP ))) { +if (unlikely (!err && (vrm->flags &MREMAP_DONTUNMAP ))) {/* We always clear VM_LOCKED[ONFAULT] on the old vma */ vm_flags_clear (vma,VM_LOCKED_MASK );@@ -952,22 +1029,20 @@ static unsigned long move_vma(struct vm_area_struct *vma,unlink_anon_vmas (vma);/* Because we won't unmap we don't need to touch locked_vm */ +vrm_stat_account (vrm, new_len);return new_addr; } +vrm_stat_account (vrm, new_len); +vma_iter_init (&vmi, mm, old_addr); -if (do_vmi_munmap (&vmi, mm, old_addr, old_len, uf_unmap, false) <0 ) { +if (do_vmi_munmap (&vmi, mm, old_addr, old_len, vrm->uf_unmap, false) <0 ) {/* OOM: unable to split vma, just get accounts right */ -if (vm_flags &VM_ACCOUNT && !(flags &MREMAP_DONTUNMAP )) +if (vm_flags &VM_ACCOUNT && !(vrm->flags &MREMAP_DONTUNMAP ))vm_acct_memory (old_len >>PAGE_SHIFT ); account_start = account_end = false; } -if (vm_flags &VM_LOCKED ) { - mm->locked_vm += new_len >>PAGE_SHIFT ; - *locked = true; - } - mm->hiwater_vm = hiwater_vm;/* Restore VM_ACCOUNT if one or two pieces of vma left */ @@ -1138,9 +1213,7 @@ static unsigned long mremap_to(struct vma_remap_struct *vrm)if (err)return err; -return move_vma (vrm->vma, vrm->addr, vrm->old_len, vrm->new_len, - vrm->new_addr, &vrm->locked, vrm->flags, - vrm->uf, vrm->uf_unmap); +return move_vma (vrm); }static int vma_expandable (struct vm_area_struct *vma,unsigned long delta)@@ -1245,17 +1318,11 @@ static unsigned long check_mremap_params(struct vma_remap_struct *vrm)static unsigned long expand_vma_in_place (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; -long pages = vrm->delta >>PAGE_SHIFT ;struct vm_area_struct *vma = vrm->vma;VMA_ITERATOR (vmi, mm, vma->vm_end); -long charged =0 ; - -if (vma->vm_flags &VM_ACCOUNT ) { -if (security_vm_enough_memory_mm (mm, pages)) -return -ENOMEM ; - charged = pages; - } +if (!vrm_charge (vrm)) +return -ENOMEM ;/* * Function vma_merge_extend() is called on the @@ -1268,15 +1335,11 @@ static unsigned long expand_vma_in_place(struct vma_remap_struct *vrm)*/ vma = vrm->vma =vma_merge_extend (&vmi, vma, vrm->delta);if (!vma) { -vm_unacct_memory (charged); +vrm_uncharge (vrm);return -ENOMEM ; } -vm_stat_account (mm, vma->vm_flags, pages); -if (vma->vm_flags &VM_LOCKED ) { - mm->locked_vm += pages; - vrm->locked = true; - } +vrm_stat_account (vrm, vrm->delta);return 0 ; }@@ -1316,11 +1379,7 @@ static bool align_hugetlb(struct vma_remap_struct *vrm)static unsigned long expand_vma (struct vma_remap_struct *vrm) {unsigned long err; -struct vm_area_struct *vma = vrm->vma;unsigned long addr = vrm->addr; -unsigned long old_len = vrm->old_len; -unsigned long new_len = vrm->new_len; -unsigned long flags = vrm->flags; err =resize_is_valid (vrm);if (err)@@ -1353,7 +1412,7 @@ static unsigned long expand_vma(struct vma_remap_struct *vrm)*/ /* We're not allowed to move the VMA, so error out. */ -if (!(flags &MREMAP_MAYMOVE )) +if (!(vrm->flags &MREMAP_MAYMOVE ))return -ENOMEM ;/* Find a new location to move the VMA to. */ @@ -1361,8 +1420,7 @@ static unsigned long expand_vma(struct vma_remap_struct *vrm)if (err)return err; -return move_vma (vma, addr, old_len, new_len, vrm->new_addr, - &vrm->locked, flags, vrm->uf, vrm->uf_unmap); +return move_vma (vrm); }/* -- 2.48.1
PATCH 5/7 mm/mremap: complete refactor of move_vma()
In reply to Lorenzo StoakesWe invoke ksm_madvise() with an intentionally dummy flags field, so no need to pass around.
Additionally, the code tries to be 'clever' with account_start, account_end, using these to both check that vma->vm_start != 0 and that we ought to account the newly split portion of VMA post-move, either before or after it.
We need to do this because we intentionally removed VM_ACCOUNT on the VMA prior to unmapping, so we don't erroneously unaccount memory (we have already calculated the correct amount to account and accounted it, any subsequent subtraction will be incorrect).
This patch significantly expands the comment (from 2002!) about 'concealing' the flag to make it abundantly clear what's going on, as well as adding and expanding a number of other comments also.
We can remove account_start, account_end by instead tracking when we account (i.e. vma->vm_flags has the VM_ACCOUNT flag set, and this is not an MREMAP_DONTUNMAP operation), and figuring out when to reinstate the VM_ACCOUNT flag on prior/subsequent VMAs separately.
We additionally break the function into logical pieces and attack the very confusing error handling logic (where, for instance, new_addr is set to err).
After this change the code is considerably more readable and easy to manipulate.
--- mm/mremap.c | 292 ++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 203 insertions(+), 89 deletions(-)
mm/mremap.c +203 -89
@@ -889,13 +889,13 @@ static void vrm_stat_account(struct vma_remap_struct *vrm,* Perform checks before attempting to write a VMA prior to it being * moved. */ -static unsigned long prep_move_vma (struct vma_remap_struct *vrm, -unsigned long *vm_flags_ptr) +static unsigned long prep_move_vma (struct vma_remap_struct *vrm) {unsigned long err;struct vm_area_struct *vma = vrm->vma;unsigned long old_addr = vrm->addr;unsigned long old_len = vrm->old_len; +unsigned long dummy = vma->vm_flags;/* * We'd prefer to avoid failure later on in do_munmap:
@@ -921,56 +921,150 @@ static unsigned long prep_move_vma(struct vma_remap_struct *vrm,* so KSM can come around to merge on vma and new_vma afterwards. */ err =ksm_madvise (vma, old_addr, old_addr + old_len, -MADV_UNMERGEABLE , vm_flags_ptr); +MADV_UNMERGEABLE , &dummy);if (err)return err;return 0 ; } -static unsigned long move_vma (struct vma_remap_struct *vrm) +/* +* Unmap source VMA for VMA move, turning it from a copy to a move, being +* careful to ensure we do not underflow memory account while doing so if an +* accountable move. +* +* This is best effort, if we fail to unmap then we simply try +*/ +static void unmap_source_vma (struct vma_remap_struct *vrm) {struct mm_struct *mm = current->mm; +unsigned long addr = vrm->addr; +unsigned long len = vrm->old_len;struct vm_area_struct *vma = vrm->vma; -struct vm_area_struct *new_vma; -unsigned long vm_flags = vma->vm_flags; -unsigned long old_addr = vrm->addr, new_addr = vrm->new_addr; -unsigned long old_len = vrm->old_len, new_len = vrm->new_len; -unsigned long new_pgoff; -unsigned long moved_len; -unsigned long account_start = false; -unsigned long account_end = false; -unsigned long hiwater_vm; +VMA_ITERATOR (vmi, mm, addr);int err; -bool need_rmap_locks; -struct vma_iterator vmi; +unsigned long vm_start; +unsigned long vm_end; +/* +* It might seem odd that we check for MREMAP_DONTUNMAP here, given this +* function implies that we unmap the original VMA, which seems +* contradictory. +* +* However, this occurs when this operation was attempted and an error +* arose, in which case we _do_ wish to unmap the _new_ VMA, which means +* we actually _do_ want it be unaccounted. +*/ +bool accountable_move = (vma->vm_flags &VM_ACCOUNT ) && + !(vrm->flags &MREMAP_DONTUNMAP ); - err =prep_move_vma (vrm, &vm_flags); -if (err) -return err; +/* +* So we perform a trick here to prevent incorrect accounting. Any merge +* or new VMA allocation performed in copy_vma() does not adjust +* accounting, it is expected that callers handle this. +* +* And indeed we already have, accounting appropriately in the case of +* both in vrm_charge(). +* +* However, when we unmap the existing VMA (to effect the move), this +* code will, if the VMA has VM_ACCOUNT set, attempt to unaccount +* removed pages. +* +* To avoid this we temporarily clear this flag, reinstating on any +* portions of the original VMA that remain. +*/ +if (accountable_move) { +vm_flags_clear (vma,VM_ACCOUNT ); +/* We are about to split vma, so store the start/end. */ + vm_start = vma->vm_start; + vm_end = vma->vm_end; + } -/* If accounted, charge the number of bytes the operation will use. */ -if (!vrm_charge (vrm)) -return -ENOMEM ; + err =do_vmi_munmap (&vmi, mm, addr, len, vrm->uf_unmap,/* unlock= */ false); + vrm->vma =NULL ;/* Invalidated. */ +if (err) { +/* OOM: unable to split vma, just get accounts right */ +vm_acct_memory (len >>PAGE_SHIFT ); +return ; + } -vma_start_write (vma); - new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >>PAGE_SHIFT ); - new_vma =copy_vma (&vrm->vma, new_addr, new_len, new_pgoff, +/* +* If we mremap() from a VMA like this: +* +* addr end +* | | +* v v +* |-------------| +* | | +* |-------------| +* +* Having cleared VM_ACCOUNT from the whole VMA, after we unmap above +* we'll end up with: +* +* addr end +* | | +* v v +* |---| |---| +* | A | | B | +* |---| |---| +* +* The VMI is still pointing at addr, so vma_prev() will give us A, and +* a subsequent or lone vma_next() will give as B. +* +* do_vmi_munmap() will have restored the VMI back to addr. +*/ +if (accountable_move) { +unsigned long end = addr + len; + +if (vm_start < addr) { +struct vm_area_struct *prev =vma_prev (&vmi); + +vm_flags_set (prev,VM_ACCOUNT );/* Acquires VMA lock. */ + } + +if (vm_end > end) { +struct vm_area_struct *next =vma_next (&vmi); + +vm_flags_set (next,VM_ACCOUNT );/* Acquires VMA lock. */ + } + } +} + +/* +* Copy vrm->vma over to vrm->new_addr possibly adjusting size as part of the +* process. Additionally handle an error occurring on moving of page tables, +* where we reset vrm state to cause unmapping of the new VMA. +* +* Outputs the newly installed VMA to new_vma_ptr. Returns 0 on success or an +* error code. +*/ +static int copy_vma_and_data (struct vma_remap_struct *vrm, +struct vm_area_struct **new_vma_ptr) +{ +unsigned long internal_offset = vrm->addr - vrm->vma->vm_start; +unsigned long internal_pgoff = internal_offset >>PAGE_SHIFT ; +unsigned long new_pgoff = vrm->vma->vm_pgoff + internal_pgoff; +unsigned long moved_len; +bool need_rmap_locks; +struct vm_area_struct *vma; +struct vm_area_struct *new_vma; +int err =0 ; + + new_vma =copy_vma (&vrm->vma, vrm->new_addr, vrm->new_len, new_pgoff, &need_rmap_locks); -/* This may have been updated. */ - vma = vrm->vma;if (!new_vma) {vrm_uncharge (vrm); + *new_vma_ptr =NULL ;return -ENOMEM ; } + vma = vrm->vma; - moved_len =move_page_tables (vma, old_addr, new_vma, new_addr, old_len, - need_rmap_locks, false); -if (moved_len < old_len) { + moved_len =move_page_tables (vma, vrm->addr, new_vma, + vrm->new_addr, vrm->old_len, + need_rmap_locks,/* for_stack= */ false); +if (moved_len < vrm->old_len) err = -ENOMEM ; - }else if (vma->vm_ops && vma->vm_ops->mremap) { +else if (vma->vm_ops && vma->vm_ops->mremap) err = vma->vm_ops->mremap (new_vma); - }if (unlikely (err)) {/*
@@ -978,28 +1072,84 @@ static unsigned long move_vma(struct vma_remap_struct *vrm)* which will succeed since page tables still there, * and then proceed to unmap new area instead of old. */ -move_page_tables (new_vma, new_addr, vma, old_addr, moved_len, - true, false); - vma = new_vma; - old_len = new_len; - old_addr = new_addr; - new_addr = err; +move_page_tables (new_vma, vrm->new_addr, vma, vrm->addr, + moved_len,/* need_rmap_locks = */ true, +/* for_stack= */ false); + vrm->vma = new_vma; + vrm->old_len = vrm->new_len; + vrm->addr = vrm->new_addr; }else {mremap_userfaultfd_prep (new_vma, vrm->uf); } -if (is_vm_hugetlb_page (vma)) { +if (is_vm_hugetlb_page (vma))clear_vma_resv_huge_pages (vma); - } -/* Conceal VM_ACCOUNT so old reservation is not undone */ -if (vm_flags &VM_ACCOUNT && !(vrm->flags &MREMAP_DONTUNMAP )) { -vm_flags_clear (vma,VM_ACCOUNT ); -if (vma->vm_start < old_addr) - account_start = true; -if (vma->vm_end > old_addr + old_len) - account_end = true; - } +/* Tell pfnmap has moved from this vma */ +if (unlikely (vma->vm_flags &VM_PFNMAP )) +untrack_pfn_clear (vma); + + *new_vma_ptr = new_vma; +return err; +} + +/* +* Perform final tasks for MADV_DONTUNMAP operation, clearing mlock() and +* account flags on remaining VMA by convention (it cannot be mlock()'d any +* longer, as pages in range are no longer mapped), and removing anon_vma_chain +* links from it (if the entire VMA was copied over). +*/ +static void dontunmap_complete (struct vma_remap_struct *vrm, +struct vm_area_struct *new_vma) +{ +unsigned long start = vrm->addr; +unsigned long end = vrm->addr + vrm->old_len; +unsigned long old_start = vrm->vma->vm_start; +unsigned long old_end = vrm->vma->vm_end; + +/* +* We always clear VM_LOCKED[ONFAULT] | VM_ACCOUNT on the old +* vma. +*/ +vm_flags_clear (vrm->vma,VM_LOCKED_MASK |VM_ACCOUNT ); + +/* +* anon_vma links of the old vma is no longer needed after its page +* table has been moved. +*/ +if (new_vma != vrm->vma && start == old_start && end == old_end) +unlink_anon_vmas (vrm->vma); + +/* Because we won't unmap we don't need to touch locked_vm. */ +} + +static unsigned long move_vma (struct vma_remap_struct *vrm) +{ +struct mm_struct *mm = current->mm; +struct vm_area_struct *new_vma; +unsigned long hiwater_vm; +int err; + + err =prep_move_vma (vrm); +if (err) +return err; + +/* If accounted, charge the number of bytes the operation will use. */ +if (!vrm_charge (vrm)) +return -ENOMEM ; + +/* We don't want racing faults. */ +vma_start_write (vrm->vma); + +/* Perform copy step. */ + err =copy_vma_and_data (vrm, &new_vma); +/* +* If we established the copied-to VMA, we attempt to recover from the +* error by setting the destination VMA to the source VMA and unmapping +* it below. +*/ +if (err && !new_vma) +return err;/* * If we failed to move page tables we still do total_vm increment
@@ -1012,51 +1162,15 @@ static unsigned long move_vma(struct vma_remap_struct *vrm)*/ hiwater_vm = mm->hiwater_vm; -/* Tell pfnmap has moved from this vma */ -if (unlikely (vma->vm_flags &VM_PFNMAP )) -untrack_pfn_clear (vma); - -if (unlikely (!err && (vrm->flags &MREMAP_DONTUNMAP ))) { -/* We always clear VM_LOCKED[ONFAULT] on the old vma */ -vm_flags_clear (vma,VM_LOCKED_MASK ); - -/* -* anon_vma links of the old vma is no longer needed after its page -* table has been moved. -*/ -if (new_vma != vma && vma->vm_start == old_addr && - vma->vm_end == (old_addr + old_len)) -unlink_anon_vmas (vma); - -/* Because we won't unmap we don't need to touch locked_vm */ -vrm_stat_account (vrm, new_len); -return new_addr; - } - -vrm_stat_account (vrm, new_len); - -vma_iter_init (&vmi, mm, old_addr); -if (do_vmi_munmap (&vmi, mm, old_addr, old_len, vrm->uf_unmap, false) <0 ) { -/* OOM: unable to split vma, just get accounts right */ -if (vm_flags &VM_ACCOUNT && !(vrm->flags &MREMAP_DONTUNMAP )) -vm_acct_memory (old_len >>PAGE_SHIFT ); - account_start = account_end = false; - } +vrm_stat_account (vrm, vrm->new_len); +if (unlikely (!err && (vrm->flags &MREMAP_DONTUNMAP ))) +dontunmap_complete (vrm, new_vma); +else +unmap_source_vma (vrm); mm->hiwater_vm = hiwater_vm; -/* Restore VM_ACCOUNT if one or two pieces of vma left */ -if (account_start) { - vma =vma_prev (&vmi); -vm_flags_set (vma,VM_ACCOUNT ); - } - -if (account_end) { - vma =vma_next (&vmi); -vm_flags_set (vma,VM_ACCOUNT ); - } - -return new_addr; +return err ? (unsigned long )err : vrm->new_addr; }/*
Signature · 1 line
2.48.1
PATCH 6/7 mm/mremap: refactor move_page_tables(), abstracting state
In reply to Lorenzo StoakesA lot of state is threaded throughout the page table moving logic within the mremap code, including boolean values which control behaviour specifically in regard to whether rmap locks need be held over the operation and whether the VMA belongs to a temporary stack being moved by move_arg_pages() (and consequently, relocate_vma_down()).
As we already transmit state throughout this operation, it is neater and more readable to maintain a small state object. We do so in the form of pagetable_move_control.
In addition, this allows us to update parameters within the state as we manipulate things, for instance with regard to the page table realignment logic.
In future I want to add additional functionality to the page table logic, so this is an additional motivation for making it easier to do so.
This patch changes move_page_tables() to accept a pointer to a pagetable_move_control struct, and performs changes at this level only. Further page table logic will be updated in a subsequent patch.
We additionally also take the opportunity to add significant comments describing the address realignment logic to make it abundantly clear what is going on in this code.
--- mm/internal.h | 46 ++++++++++++-- mm/mmap.c | 5 +- mm/mremap.c | 172 ++++++++++++++++++++++++++++++++++++-------------- 3 files changed, 171 insertions(+), 52 deletions(-)
mm/internal.h +42 -4
@@ -24,6 +24,47 @@struct folio_batch ; +/* +* Maintains state across a page table move. The operation assumes both source +* and destination VMAs already exist and are specified by the user. +* +* Partial moves are permitted, but: +* from_vma->vm_start <= from_addr < from_vma->vm_end - len +* to_vma->vm_start <= to_addr < to_vma->vm_end - len +* +* Must be maintained. +* +* mmap lock must be held in write and VMA write locks must be held on any VMA +* that is visible. +* +* Use the PAGETABLE_MOVE() macro to initialise this struct. +* +* NOTE: The page table move is affected by reading from [old_addr, old_end), +* and old_addr may be updated for better page table alignment, so len_in +* represents the length of the range being copied as specified by the user. +*/ +struct pagetable_move_control { +struct vm_area_struct *old;/* Source VMA. */ +struct vm_area_struct *new;/* Destination VMA. */ +unsigned long old_addr;/* Address from which the move begins. */ +unsigned long old_end;/* Exclusive address at which old range ends. */ +unsigned long new_addr;/* Address to move page tables to. */ +unsigned long len_in;/* Bytes to remap specified by user. */ + +bool need_rmap_locks;/* Do rmap locks need to be taken? */ +bool for_stack;/* Is this an early temp stack being moved? */ +}; + +#define PAGETABLE_MOVE (name, old_, new_, old_addr_, new_addr_, len_) \ + struct pagetable_move_control name = { \ + .old = old_, \ + .new = new_, \ + .old_addr = old_addr_, \ + .old_end = (old_addr_) + (len_), \ + .new_addr = new_addr_, \ + .len_in = len_, \ + } +/* * The set of flags that only affect watermark checking and reclaim * behaviour. This is used by the MM to obey the caller constraints
@@ -1527,10 +1568,7 @@ extern struct list_lru shadow_nodes; }while (0 )/* mremap.c */ -unsigned long move_page_tables(struct vm_area_struct *vma, -unsigned long old_addr,struct vm_area_struct *new_vma, -unsigned long new_addr,unsigned long len, -bool need_rmap_locks,bool for_stack); +unsigned long move_page_tables(struct pagetable_move_control *pmc);#ifdef CONFIG_UNACCEPTED_MEMORY void accept_page (struct page *page);
mm/mmap.c +3 -2
@@ -1694,6 +1694,7 @@ int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift)VMG_STATE (vmg, mm, &vmi, new_start, old_end,0 , vma->vm_pgoff);struct vm_area_struct *next;struct mmu_gather tlb; +PAGETABLE_MOVE (pmc, vma, vma, old_start, new_start, length);BUG_ON (new_start > new_end);
@@ -1716,8 +1717,8 @@ int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift)* move the page tables downwards, on failure we rely on * process cleanup to remove whatever mess we made. */ -if (length !=move_page_tables (vma, old_start, - vma, new_start, length, false, true)) + pmc.for_stack = true; +if (length !=move_page_tables (&pmc))return -ENOMEM ;tlb_gather_mmu (&tlb, mm);
mm/mremap.c +126 -46
275 lines not on this page · raw message
Signature · 1 line
2.48.1
PATCH 7/7 mm/mremap: thread state through move page table operation
In reply to Lorenzo StoakesFinish refactoring the page table logic by threading the PMC state throughout the operation, allowing us to control the operation as we go.
Additionally, update the old_addr, new_addr fields in move_page_tables() as we progress through the process making use of the fact we have this state object now to track this.
With these changes made, not only is the code far more readable, but we can finally transmit state throughout the entire operation, which lays the groundwork for sensibly making changes in future to how the mremap() operation is performed.
Additionally take the opportunity to refactor the means of determining the progress of the operation, abstracting this to pmc_progress() and simplifying the logic to make it clearer what's going on.
--- mm/internal.h | 3 + mm/mremap.c | 196 +++++++++++++++++++++++++++++--------------------- 2 files changed, 117 insertions(+), 82 deletions(-)
mm/internal.h +3 -0
@@ -39,6 +39,9 @@ struct folio_batch;* * Use the PAGETABLE_MOVE() macro to initialise this struct. * +* The old_addr and new_addr fields are updated as the page table move is +* executed. +* * NOTE: The page table move is affected by reading from [old_addr, old_end), * and old_addr may be updated for better page table alignment, so len_in * represents the length of the range being copied as specified by the user.
mm/mremap.c +114 -82
418 lines not on this page · raw message
Signature · 1 line
2.48.1
kernel test robot
build botRe: [PATCH 7/7] mm/mremap: thread state through move page table operation
kernel test robot noticed the following build errors:
Hi Lorenzo,
kernel test robot noticed the following build errors:
[auto build test ERROR on akpm-mm/mm-everything]
[also build test ERROR on linus/master v6.14-rc5 next-20250303] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Lorenzo-Stoakes/mm-mremap-correctly-handle-partial-mremap-of-VMA-starting-at-0/20250303-192101 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/4cc18ebfc0cb32e9e42d86503cef3265108343e4.1740911247.git.lorenzo.stoakes%40oracle.com patch subject: [PATCH 7/7] mm/mremap: thread state through move page table operation config: x86_64-buildonly-randconfig-001-20250304 (https://download.01.org/0day-ci/archive/20250304/[email protected]/config) compiler: clang version 19.1.7 (https://github.com/llvm/llvm-project cd708029e0b2869e80abe31ddb175f7c35361f90) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250304/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from mm/mremap.c:11:
In file included from include/linux/mm.h:2302:
include/linux/vmstat.h:518:36: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
518 | return node_stat_name(NR_LRU_BASE + lru) + 3; // skip "nr_"
| ~~~~~~~~~~~ ^ ~~~
In file included from mm/mremap.c:12:
include/linux/mm_inline.h:47:41: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
47 | __mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
| ~~~~~~~~~~~ ^ ~~~
include/linux/mm_inline.h:49:22: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
49 | NR_ZONE_LRU_BASE + lru, nr_pages);
| ~~~~~~~~~~~~~~~~ ^ ~~~
>> mm/mremap.c:581:43: error: too few arguments to function call, expected 5, have 3
581 | move_huge_pud(pmc, old_entry, new_entry);
| ~~~~~~~~~~~~~ ^
mm/mremap.c:478:13: note: 'move_huge_pud' declared here
478 | static bool move_huge_pud(struct vm_area_struct *vma, unsigned long old_addr,
| ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
479 | unsigned long new_addr, pud_t *old_pud, pud_t *new_pud)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~3 warnings and 1 error generated.
vim +581 mm/mremap.c
552
553 /*
554 * Attempts to speedup the move by moving entry at the level corresponding to
555 * pgt_entry. Returns true if the move was successful, else false.
556 */
557 static bool move_pgt_entry(struct pagetable_move_control *pmc,
558 enum pgt_entry entry, void *old_entry, void *new_entry)
559 {
560 bool moved = false;
561 bool need_rmap_locks = should_take_rmap_locks(pmc, entry);
562
563 /* See comment in move_ptes() */
564 if (need_rmap_locks)
565 take_rmap_locks(pmc->old);
566
567 switch (entry) {
568 case NORMAL_PMD:
569 moved = move_normal_pmd(pmc, old_entry, new_entry);
570 break;
571 case NORMAL_PUD:
572 moved = move_normal_pud(pmc, old_entry, new_entry);
573 break;
574 case HPAGE_PMD:
575 moved = IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
576 move_huge_pmd(pmc->old, pmc->old_addr, pmc->new_addr, old_entry,
577 new_entry);
578 break;
579 case HPAGE_PUD:
580 moved = IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
> 581 move_huge_pud(pmc, old_entry, new_entry);
582 break;
583
584 default:
585 WARN_ON_ONCE(1);
586 break;
587 }
588
589 if (need_rmap_locks)
590 drop_rmap_locks(pmc->old);
591
592 return moved;
593 }
594
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH 7/7] mm/mremap: thread state through move page table operation
In reply to Lorenzo Stoakeskernel test robot noticed the following build warnings:
Hi Lorenzo,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on linus/master v6.14-rc5 next-20250303] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Lorenzo-Stoakes/mm-mremap-correctly-handle-partial-mremap-of-VMA-starting-at-0/20250303-192101 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/4cc18ebfc0cb32e9e42d86503cef3265108343e4.1740911247.git.lorenzo.stoakes%40oracle.com patch subject: [PATCH 7/7] mm/mremap: thread state through move page table operation config: x86_64-buildonly-randconfig-005-20250304 (https://download.01.org/0day-ci/archive/20250304/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250304/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
mm/mremap.c: In function 'move_pgt_entry':
mm/mremap.c:581:39: error: passing argument 1 of 'move_huge_pud' from incompatible pointer type [-Werror=incompatible-pointer-types]
581 | move_huge_pud(pmc, old_entry, new_entry);
| ^~~
| |
| struct pagetable_move_control *
mm/mremap.c:478:50: note: expected 'struct vm_area_struct *' but argument is of type 'struct pagetable_move_control *'
478 | static bool move_huge_pud(struct vm_area_struct *vma, unsigned long old_addr,
| ~~~~~~~~~~~~~~~~~~~~~~~^~~
>> mm/mremap.c:581:44: warning: passing argument 2 of 'move_huge_pud' makes integer from pointer without a cast [-Wint-conversion]
581 | move_huge_pud(pmc, old_entry, new_entry);
| ^~~~~~~~~
| |
| void *
mm/mremap.c:478:69: note: expected 'long unsigned int' but argument is of type 'void *'
478 | static bool move_huge_pud(struct vm_area_struct *vma, unsigned long old_addr,
| ~~~~~~~~~~~~~~^~~~~~~~
mm/mremap.c:581:55: warning: passing argument 3 of 'move_huge_pud' makes integer from pointer without a cast [-Wint-conversion]
581 | move_huge_pud(pmc, old_entry, new_entry);
| ^~~~~~~~~
| |
| void *
mm/mremap.c:479:41: note: expected 'long unsigned int' but argument is of type 'void *'
479 | unsigned long new_addr, pud_t *old_pud, pud_t *new_pud)
| ~~~~~~~~~~~~~~^~~~~~~~
mm/mremap.c:581:25: error: too few arguments to function 'move_huge_pud'
581 | move_huge_pud(pmc, old_entry, new_entry);
| ^~~~~~~~~~~~~ mm/mremap.c:478:13: note: declared here
478 | static bool move_huge_pud(struct vm_area_struct *vma, unsigned long old_addr,
| ^~~~~~~~~~~~~
cc1: some warnings being treated as errors
vim +/move_huge_pud +581 mm/mremap.c
552
553 /*
554 * Attempts to speedup the move by moving entry at the level corresponding to
555 * pgt_entry. Returns true if the move was successful, else false.
556 */
557 static bool move_pgt_entry(struct pagetable_move_control *pmc,
558 enum pgt_entry entry, void *old_entry, void *new_entry)
559 {
560 bool moved = false;
561 bool need_rmap_locks = should_take_rmap_locks(pmc, entry);
562
563 /* See comment in move_ptes() */
564 if (need_rmap_locks)
565 take_rmap_locks(pmc->old);
566
567 switch (entry) {
568 case NORMAL_PMD:
569 moved = move_normal_pmd(pmc, old_entry, new_entry);
570 break;
571 case NORMAL_PUD:
572 moved = move_normal_pud(pmc, old_entry, new_entry);
573 break;
574 case HPAGE_PMD:
575 moved = IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
576 move_huge_pmd(pmc->old, pmc->old_addr, pmc->new_addr, old_entry,
577 new_entry);
578 break;
579 case HPAGE_PUD:
580 moved = IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
> 581 move_huge_pud(pmc, old_entry, new_entry);
582 break;
583
584 default:
585 WARN_ON_ONCE(1);
586 break;
587 }
588
589 if (need_rmap_locks)
590 drop_rmap_locks(pmc->old);
591
592 return moved;
593 }
594
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Andrew Morton
maintainerRe: [PATCH 7/7] mm/mremap: thread state through move page table operation
Quoted 23 lines · On Tue, 4 Mar 2025 07:30:16 +0800 kernel test robot <[email protected]> wrote:
Hi Lorenzo,
kernel test robot noticed the following build warnings:
[auto build test WARNING on akpm-mm/mm-everything]
[also build test WARNING on linus/master v6.14-rc5 next-20250303] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]url: https://github.com/intel-lab-lkp/linux/commits/Lorenzo-Stoakes/mm-mremap-correctly-handle-partial-mremap-of-VMA-starting-at-0/20250303-192101 base: https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything patch link: https://lore.kernel.org/r/4cc18ebfc0cb32e9e42d86503cef3265108343e4.1740911247.git.lorenzo.stoakes%40oracle.com patch subject: [PATCH 7/7] mm/mremap: thread state through move page table operation config: x86_64-buildonly-randconfig-005-20250304 (https://download.01.org/0day-ci/archive/20250304/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20250304/[email protected]/reproduce)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
Quoted 4 lines
All warnings (new ones prefixed by >>):
mm/mremap.c: In function 'move_pgt_entry':
mm/mremap.c:581:39: error: passing argument 1 of 'move_huge_pud' from incompatible pointer type [-Werror=incompatible-pointer-types]
Thanks.
mm/mremap.c renamed from mm/mremap.c~mm-mremap-thread-state-through-move-page-table-operation-fix +3 -2
@@ -475,8 +475,9 @@ static bool move_huge_pud(struct pagetab return true; }#else -static bool move_huge_pud (struct vm_area_struct *vma,unsigned long old_addr, -unsigned long new_addr,pud_t *old_pud,pud_t *new_pud) +static bool move_huge_pud (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr, +pud_t *old_pud,pud_t *new_pud) {WARN_ON_ONCE (1 );return false;
_
Andrew Morton
maintainerRe: [PATCH 7/7] mm/mremap: thread state through move page table operation
Quoted 4 lines · On Mon, 3 Mar 2025 16:05:55 -0800 Andrew Morton <[email protected]>…
mm/mremap.c renamed from mm/mremap.c~mm-mremap-thread-state-through-move-page-table-operation-fix +0
-0@@ -475,8 +475,9 @@ static bool move_huge_pud(struct pagetabreturn true;
mm/mremap.c static bool move_huge_pud(struct pagetab · line 483
Quoted 10 lines
mm/mremap.c renamed from mm/mremap.c~mm-mremap-thread-state-through-move-page-table-operation-fix +3
-2}#else -static bool move_huge_pud (struct vm_area_struct *vma,unsigned long old_addr, -unsigned long new_addr,pud_t *old_pud,pud_t *new_pud) +static bool move_huge_pud (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr, +pud_t *old_pud,pud_t *new_pud) {WARN_ON_ONCE (1 );return false;
err,
mm/mremap.c renamed from mm/mremap.c~mm-mremap-thread-state-through-move-page-table-operation-fix +2 -2
@@ -475,8 +475,8 @@ static bool move_huge_pud(struct pagetab return true; }#else -static bool move_huge_pud (struct vm_area_struct *vma,unsigned long old_addr, -unsigned long new_addr,pud_t *old_pud,pud_t *new_pud) +static bool move_huge_pud (struct pagetable_move_control *pmc, +pud_t *old_pud,pud_t *new_pud) {WARN_ON_ONCE (1 );return false;
_
Re: [PATCH 7/7] mm/mremap: thread state through move page table operation
Quoted 32 lines · Andrew Morton
Quoted 4 lines · On Mon, 3 Mar 2025 16:05:55 -0800 Andrew Morton <[email protected]>…
mm/mremap.c renamed from mm/mremap.c~mm-mremap-thread-state-through-move-page-table-operation-fix +0
-0@@ -475,8 +475,9 @@ static bool move_huge_pud(struct pagetabreturn true;mm/mremap.c static bool move_huge_pud(struct pagetab · line 483
Quoted 10 lines
mm/mremap.c renamed from mm/mremap.c~mm-mremap-thread-state-through-move-page-table-operation-fix +3
-2}#else -static bool move_huge_pud (struct vm_area_struct *vma,unsigned long old_addr, -unsigned long new_addr,pud_t *old_pud,pud_t *new_pud) +static bool move_huge_pud (struct pagetable_move_control *pmc, +unsigned long old_addr,unsigned long new_addr, +pud_t *old_pud,pud_t *new_pud) {WARN_ON_ONCE (1 );return false;err,
mm/mremap.c renamed from mm/mremap.c~mm-mremap-thread-state-through-move-page-table-operation-fix +2
-2@@ -475,8 +475,8 @@ static bool move_huge_pud(struct pagetab return true; }#else -static bool move_huge_pud (struct vm_area_struct *vma,unsigned long old_addr, -unsigned long new_addr,pud_t *old_pud,pud_t *new_pud) +static bool move_huge_pud (struct pagetable_move_control *pmc, +pud_t *old_pud,pud_t *new_pud) {WARN_ON_ONCE (1 );return false;
Quoted 2 lines
_
Thanks! Apologies for that, thought I'd caught all these... :)