mm: further simplify VMA merge operation
Subsystems (4)
v2 cover mm: further simplify VMA merge operation
While significant efforts have been made to improve the VMA merge operation, there remains remnants of the bad (or rather confusing) old days, which make the code difficult to understand, more bug prone and thus harder to modify.
This series attempts to significantly improve matters in a number of respects - with a focus on simplifying the commit_merge() function which actually actions the merge operation - and importantly, adjusting the two most confusing merge cases - those in which we 'adjust' the VMA immediately adjacent to the one being merged.
One source of confusion are the VMAs being threaded through the operation themselves - vmg->prev, vmg->vma and vmg->next.
At the start of the operation, vmg->vma is either NULL if a new VMA is propose to be added, or if not then a pointer to an existing VMA being modified, and prev/next are (perhaps not present) VMAs sat immediately before and after the range specified in vmg->start, end, respectively.
However, during the VMA merge operation, we change vmg->start, end and pgoff to span the newly merged range and vmg->vma to either be:
a. The ultimately returned VMA (in most cases) or
b. A VMA which we will manipulate, but ultimately instead return vmg->next.
Case b. especially here is confusing for somebody reading this code, but the fact we update this state, along with vmg->start, end, pgoff only makes matters worse.
We simplify things by replacing vmg->vma with vmg->middle and never changing it - this is always either NULL (for a new VMA) or the VMA being modified between vmg->prev and vmg->next.
We further simplify by placing the merged VMA in a new vmg->target field - whether case b. above is the case or not. The reader of the code can now simply rely on vmg->middle being the middle VMA and vmg->target being the ultimately merged VMA.
We additionally tackle the confusing cases where we 'adjust' VMAs other than the one we ultimately return as the merged VMA (this includes case b. above). These are:
(1)
merge <-----------> |------||--------| |------------|---| | prev || middle | -> | target | m | |------||--------| |------------|---|
In which case middle must be adjusted so middle->vm_start is increased as well as performing the merge.
(2) (equivalent to case b. above)
<-------------> |---------||------| |---|-------------| | middle || next | -> | m | target | |---------||------| |---|-------------|
In which case next must be adjusted so next->vm_start is decreased as well as performing the merge.
This cases have previously been performed by calculating and passing around a dubious and confusing 'adj_start' parameter along side a pointer to an 'adjust' VMA indicating which VMA requires additional adjustment (middle in case 1 and next in case 2).
With the VMG structure in place we are able to avoid this by simply setting a merge flag to describe each case:
(1) Sets the vmg->__adjust_middle_start flag
(2) Sets the vmg->__adjust_next_start flag
By doing so it turns out we can vastly simplify the logic and calculate what is required to perform the operation.
Taken together the refactorings make it far easier to understand what is being done even in these more confusing cases, make the code far more maintainable, debuggable, and testable, providing more internal state indicating what is happening in the merge operation.
The changes have no functional net impact on the merge operation and everything should still behave as it did before.
v2:
* Fixed buildbot warnings due to missing a change to a static inline stubbed out wrapper for vma_adjust_trans_huge().
* Converted vmg merge_flags to bitfield flags as per Vlastimil.
* Update comments which were rendered inaccurate in 5/5 as per Vlastimil.
v1:
https://lore.kernel.org/all/[email protected]/
Lorenzo Stoakes (5):
mm: simplify vma merge structure and expand comments
mm: further refactor commit_merge()
mm: eliminate adj_start parameter from commit_merge()
mm: make vmg->target consistent and further simplify commit_merge()
mm: completely abstract unnecessary adj_start calculation
include/linux/huge_mm.h | 4 +- mm/debug.c | 28 +-- mm/huge_memory.c | 19 +- mm/mmap.c | 2 +- mm/vma.c | 327 ++++++++++++++++--------------- mm/vma.h | 86 ++++++-- tools/testing/vma/vma.c | 63 +++--- tools/testing/vma/vma_internal.h | 4 +- 8 files changed, 302 insertions(+), 231 deletions(-)
--
2.48.1
PATCH v2 1/5 mm: simplify vma merge structure and expand comments
The merge code, while much improved, still has a number of points of confusion. As part of a broader series cleaning this up to make this more maintainable, we start by addressing some confusion around vma_merge_struct fields.
So far, the caller either provides no vmg->vma (a new VMA) or supplies the existing VMA which is being altered, setting vmg->start,end,pgoff to the proposed VMA dimensions.
vmg->vma is then updated, as are vmg->start,end,pgoff as the merge process proceeds and the appropriate merge strategy is determined.
This is rather confusing, as vmg->vma starts off as the 'middle' VMA between vmg->prev,next, but becomes the 'target' VMA, except in one specific edge case (merge next, shrink middle).
Int his patch we introduce vmg->middle to describe the VMA that is between vmg->prev and vmg->next, and does NOT change during the merge operation.
We replace vmg->vma with vmg->target, and use this only during the merge operation itself.
Aside from the merge right, shrink middle case, this becomes the VMA that forms the basis of the VMA that is returned. This edge case can be addressed in a future commit.
We also add a number of comments to explain what is going on.
Finally, we adjust the ASCII diagrams showing each merge case in vma_merge_existing_range() to be clearer - the arrow range previously showed the vmg->start, end spanned area, but it is clearer to change this to show the final merged VMA.
This patch has no change in functional behaviour.
--- mm/debug.c | 18 ++--- mm/mmap.c | 2 +- mm/vma.c | 166 +++++++++++++++++++++------------------- mm/vma.h | 42 ++++++++-- tools/testing/vma/vma.c | 52 ++++++------- 5 files changed, 159 insertions(+), 121 deletions(-)
mm/debug.c +9 -9
@@ -261,7 +261,7 @@ void dump_vmg(const struct vma_merge_struct *vmg, const char *reason)pr_warn ("vmg %px state: mm %px pgoff %lx\n" "vmi %px [%lx,%lx)\n" -"prev %px next %px vma %px\n" +"prev %px middle %px next %px target %px\n" "start %lx end %lx flags %lx\n" "file %px anon_vma %px policy %px\n" "uffd_ctx %px\n"
@@ -270,7 +270,7 @@ void dump_vmg(const struct vma_merge_struct *vmg, const char *reason) vmg, vmg->mm, vmg->pgoff, vmg->vmi, vmg->vmi ?vma_iter_addr (vmg->vmi) :0 , vmg->vmi ?vma_iter_end (vmg->vmi) :0 , - vmg->prev, vmg->next, vmg->vma, + vmg->prev, vmg->middle, vmg->next, vmg->target, vmg->start, vmg->end, vmg->flags, vmg->file, vmg->anon_vma, vmg->policy, #ifdef CONFIG_USERFAULTFD
@@ -288,13 +288,6 @@ void dump_vmg(const struct vma_merge_struct *vmg, const char *reason)pr_warn ("vmg %px mm: (NULL)\n" , vmg); } -if (vmg->vma) { -pr_warn ("vmg %px vma:\n" , vmg); -dump_vma (vmg->vma); - }else { -pr_warn ("vmg %px vma: (NULL)\n" , vmg); - } -if (vmg->prev) {pr_warn ("vmg %px prev:\n" , vmg);dump_vma (vmg->prev);
@@ -302,6 +295,13 @@ void dump_vmg(const struct vma_merge_struct *vmg, const char *reason)pr_warn ("vmg %px prev: (NULL)\n" , vmg); } +if (vmg->middle) { +pr_warn ("vmg %px middle:\n" , vmg); +dump_vma (vmg->middle); + }else { +pr_warn ("vmg %px middle: (NULL)\n" , vmg); + } +if (vmg->next) {pr_warn ("vmg %px next:\n" , vmg);dump_vma (vmg->next);
mm/mmap.c +1 -1
@@ -1707,7 +1707,7 @@ int relocate_vma_down(struct vm_area_struct *vma, unsigned long shift)/* * cover the whole range: [new_start, old_end) */ - vmg.vma = vma; + vmg.middle = vma;if (vma_expand (&vmg))return -ENOMEM ;
mm/vma.c +86 -80
389 lines not on this page · raw message
mm/vma.h +37 -5
61 lines not on this page · raw message
tools/testing/vma/vma.c +26 -26
204 lines not on this page · raw message
Signature · 1 line
2.48.1
PATCH v2 2/5 mm: further refactor commit_merge()
In reply to Lorenzo StoakesThe current VMA merge mechanism contains a number of confusing mechanisms around removal of VMAs on merge and the shrinking of the VMA adjacent to vma->target in the case of merges which result in a partial merge with that adjacent VMA.
Since we now have a STABLE set of VMAs - prev, middle, next - we are now able to have the caller of commit_merge() explicitly tell us which VMAs need deleting, using newly introduced internal VMA merge flags.
Doing so allows us to embed this state within the VMG and remove the confusing remove, remove2 parameters from commit_merge().
We additionally are able to eliminate the highly confusing and misleading 'expanded' parameter - a parameter that in reality refers to whether or not the return VMA is the target one or the one immediately adjacent.
We can infer which is the case from whether or not the adj_start parameter is negative. This also allows us to simplify further logic around iterator configuration and VMA iterator stores.
Doing so means we can also eliminate the adjust parameter, as we are able to infer which VMA ought to be adjusted from adj_start - a positive value implies we adjust the start of 'middle', a negative one implies we adjust the start of 'next'.
We are then able to have commit_merge() explicitly return the target VMA, or NULL on inability to pre-allocate memory. Errors were previously filtered so behaviour does not change.
We additionally move from the slightly odd use of a bitwise-flag enum vmg->merge_flags field to vmg bitfields.
This patch has no change in functional behaviour.
--- mm/debug.c | 6 ++- mm/vma.c | 101 +++++++++++++++++++++------------------- mm/vma.h | 38 +++++++++------ tools/testing/vma/vma.c | 9 +++- 4 files changed, 87 insertions(+), 67 deletions(-)
mm/debug.c +4 -2
@@ -266,7 +266,8 @@ void dump_vmg(const struct vma_merge_struct *vmg, const char *reason)"file %px anon_vma %px policy %px\n" "uffd_ctx %px\n" "anon_name %px\n" -"merge_flags %x state %x\n" , +"state %x\n" +"just_expand %b __remove_middle %b __remove_next %b\n" , vmg, vmg->mm, vmg->pgoff, vmg->vmi, vmg->vmi ?vma_iter_addr (vmg->vmi) :0 , vmg->vmi ? vma_iter_end(vmg->vmi) : 0,
@@ -279,7 +280,8 @@ void dump_vmg(const struct vma_merge_struct *vmg, const char *reason) (void *)0 , #endif vmg->anon_name, - (int)vmg->merge_flags, (int)vmg->state); + (int )vmg->state, + vmg->just_expand, vmg->__remove_middle, vmg->__remove_next);if (vmg->mm) {pr_warn ("vmg %px mm:\n" , vmg);
mm/vma.c +52 -49
251 lines not on this page · raw message
mm/vma.h +24 -14
@@ -58,17 +58,6 @@ enum vma_merge_state { VMA_MERGE_SUCCESS, }; -enum vma_merge_flags { -VMG_FLAG_DEFAULT =0 , -/* -* If we can expand, simply do so. We know there is nothing to merge to -* the right. Does not reset state upon failure to merge. The VMA -* iterator is assumed to be positioned at the previous VMA, rather than -* at the gap. -*/ -VMG_FLAG_JUST_EXPAND =1 <<0 , -}; -/* * Describes a VMA merge operation and is threaded throughout it. *
@@ -117,8 +106,31 @@ struct vma_merge_struct {struct mempolicy *policy;struct vm_userfaultfd_ctx uffd_ctx;struct anon_vma_name *anon_name; -enum vma_merge_flags merge_flags;enum vma_merge_state state; + +/* Flags which callers can use to modify merge behaviour: */ + +/* +* If we can expand, simply do so. We know there is nothing to merge to +* the right. Does not reset state upon failure to merge. The VMA +* iterator is assumed to be positioned at the previous VMA, rather than +* at the gap. +*/ + bool just_expand :1; + +/* Internal flags set during merge process: */ + +/* +* Internal flag used during the merge operation to indicate we will +* remove vmg->middle. +*/ + bool __remove_middle :1; +/* +* Internal flag used during the merge operationr to indicate we will +* remove vmg->next. +*/ + bool __remove_next :1; + };static inline bool vmg_nomem (struct vma_merge_struct *vmg)
@@ -142,7 +154,6 @@ static inline pgoff_t vma_pgoff_offset(struct vm_area_struct *vma, .flags = flags_, \ .pgoff = pgoff_, \ .state = VMA_MERGE_START, \ - .merge_flags = VMG_FLAG_DEFAULT, \ }#define VMG_VMA_STATE (name, vmi_, prev_, vma_, start_, end_) \
@@ -162,7 +173,6 @@ static inline pgoff_t vma_pgoff_offset(struct vm_area_struct *vma, .uffd_ctx = vma_->vm_userfaultfd_ctx, \ .anon_name = anon_vma_name(vma_), \ .state = VMA_MERGE_START, \ - .merge_flags = VMG_FLAG_DEFAULT, \ }#ifdef CONFIG_DEBUG_VM_MAPLE_TREE
tools/testing/vma/vma.c +7 -2
@@ -149,11 +149,16 @@ static void vmg_set_range(struct vma_merge_struct *vmg, unsigned long start, vmg->prev =NULL ; vmg->middle =NULL ; vmg->next =NULL ; + vmg->target =NULL ; vmg->start = start; vmg->end = end; vmg->pgoff = pgoff; vmg->flags = flags; + + vmg->just_expand = false; + vmg->__remove_middle = false; + vmg->__remove_next = false; }/*
@@ -1546,7 +1551,7 @@ static bool test_expand_only_mode(void)/* * Place a VMA prior to the one we're expanding so we assert that we do * not erroneously try to traverse to the previous VMA even though we -* have, through the use of VMG_FLAG_JUST_EXPAND, indicated we do not +* have, through the use of the just_expand flag, indicated we do not * need to do so. */ alloc_and_link_vma (&mm,0 ,0x2000 ,0 , flags);
@@ -1558,7 +1563,7 @@ static bool test_expand_only_mode(void)vma_iter_set (&vmi,0x3000 ); vma_prev =alloc_and_link_vma (&mm,0x3000 ,0x5000 ,3 , flags); vmg.prev = vma_prev; - vmg.merge_flags =VMG_FLAG_JUST_EXPAND ; + vmg.just_expand = true; vma =vma_merge_new_range (&vmg);ASSERT_NE (vma,NULL );
Signature · 1 line
2.48.1
PATCH v2 3/5 mm: eliminate adj_start parameter from commit_merge()
In reply to Lorenzo StoakesIntroduce internal vmg->__adjust_middle_start and vmg->__adjust_next_start merge flags, enabling us to indicate to commit_merge() that we are performing a merge which either spans only part of vmg->middle, or part of vmg->next respectively.
In the former instance, we change the start of vmg->middle to match the attributes of vmg->prev, without spanning all of vmg->middle.
This implies that vmg->prev->vm_end and vmg->middle->vm_start are both increased to form the new merged VMA (vmg->prev) and the new subsequent VMA (vmg->middle).
In the latter case, we change the end of vmg->middle to match the attributes of vmg->next, without spanning all of vmg->next.
This implies that vmg->middle->vm_end and vmg->next->vm_start are both decreased to form the new merged VMA (vmg->next) and the new prior VMA (vmg->middle).
Since we now have a stable set of prev, middle, next VMAs threaded through vmg and with these flags set know what is happening, we can perform the calculation in commit_merge() instead.
This allows us to drop the confusing adj_start parameter and instead pass semantic information to commit_merge().
In the latter case the -(middle->vm_end - start) calculation becomes -(middle->vm-end - vmg->end), however this is correct as vmg->end is set to the start parameter.
This is because in this case (rather confusingly), we manipulate vmg->middle, but ultimately return vmg->next, whose range will be correctly specified. At this point vmg->start, end is the new range for the prior VMA rather than the merged one.
This patch has no change in functional behaviour.
--- mm/debug.c | 8 +++++-- mm/vma.c | 50 ++++++++++++++++++++++++----------------- mm/vma.h | 10 +++++++++ tools/testing/vma/vma.c | 2 ++ 4 files changed, 48 insertions(+), 22 deletions(-)
mm/debug.c +6 -2
@@ -267,7 +267,9 @@ void dump_vmg(const struct vma_merge_struct *vmg, const char *reason)"uffd_ctx %px\n" "anon_name %px\n" "state %x\n" -"just_expand %b __remove_middle %b __remove_next %b\n" , +"just_expand %b\n" +"__adjust_middle_start %b __adjust_next_start %b\n" +"__remove_middle %b __remove_next %b\n" , vmg, vmg->mm, vmg->pgoff, vmg->vmi, vmg->vmi ?vma_iter_addr (vmg->vmi) :0 , vmg->vmi ? vma_iter_end(vmg->vmi) : 0,
@@ -281,7 +283,9 @@ void dump_vmg(const struct vma_merge_struct *vmg, const char *reason)#endif vmg->anon_name, (int )vmg->state, - vmg->just_expand, vmg->__remove_middle, vmg->__remove_next); + vmg->just_expand, + vmg->__adjust_middle_start, vmg->__adjust_next_start, + vmg->__remove_middle, vmg->__remove_next);if (vmg->mm) {pr_warn ("vmg %px mm:\n" , vmg);
mm/vma.c +30 -20
103 lines not on this page · raw message
mm/vma.h +10 -0
@@ -120,6 +120,16 @@ struct vma_merge_struct {/* Internal flags set during merge process: */ +/* +* Internal flag indicating the merge increases vmg->middle->vm_start +* (and thereby, vmg->prev->vm_end). +*/ + bool __adjust_middle_start :1; +/* +* Internal flag indicating the merge decreases vmg->next->vm_start +* (and thereby, vmg->middle->vm_end). +*/ + bool __adjust_next_start :1;/* * Internal flag used during the merge operation to indicate we will * remove vmg->middle.
tools/testing/vma/vma.c +2 -0
@@ -159,6 +159,8 @@ static void vmg_set_range(struct vma_merge_struct *vmg, unsigned long start, vmg->just_expand = false; vmg->__remove_middle = false; vmg->__remove_next = false; + vmg->__adjust_middle_start = false; + vmg->__adjust_next_start = false; }/*
--
2.48.1
PATCH v2 4/5 mm: make vmg->target consistent and further simplify commit_merge()
In reply to Lorenzo StoakesIt is confusing for vmg->target to sometimes be the target merged VMA and in one case not.
Fix this by having commit_merge() use its awareness of the vmg->_adjust_next_start case to know that it is manipulating a separate vma, abstracted in the 'vma' local variable.
Place removal and adjust VMA determination logic into init_multi_vma_prep(), as the flags give us enough information to do so, and since this is the function that sets up the vma_prepare struct it makes sense to do so here.
Doing this significantly simplifies commit_merge(), allowing us to eliminate the 'merge_target' handling, initialise the VMA iterator in a more sensible place and simply return vmg->target consistently.
This also allows us to simplify setting vmg->target in vma_merge_existing_range() since we are then left only with two cases - merge left (or both) where the target is vmg->prev or merge right in which the target is vmg->next.
This makes it easy for somebody reading the code to know what VMA will actually be the one returned and merged into and removes a great deal of the confusing 'adjust' nonsense.
This patch has no change in functional behaviour.
--- mm/vma.c | 119 ++++++++++++++++++++++++++++--------------------------- mm/vma.h | 6 +-- 2 files changed, 62 insertions(+), 63 deletions(-)
mm/vma.c +61 -58
@@ -106,24 +106,40 @@ static inline bool are_anon_vmas_compatible(struct vm_area_struct *vma1,* init_multi_vma_prep() - Initializer for struct vma_prepare * @vp: The vma_prepare struct * @vma: The vma that will be altered once locked -* @next: The next vma if it is to be adjusted -* @remove: The first vma to be removed -* @remove2: The second vma to be removed +* @vmg: The merge state that will be used to determine adjustment and VMA +* removal. */ static void init_multi_vma_prep (struct vma_prepare *vp,struct vm_area_struct *vma, -struct vm_area_struct *next, -struct vm_area_struct *remove, -struct vm_area_struct *remove2) +struct vma_merge_struct *vmg) { +struct vm_area_struct *adjust; +struct vm_area_struct **remove = &vp->remove; +memset (vp,0 ,sizeof (struct vma_prepare )); vp->vma = vma; vp->anon_vma = vma->anon_vma; - vp->remove = remove ? remove : remove2; - vp->remove2 = remove ? remove2 :NULL ; - vp->adj_next = next; -if (!vp->anon_vma && next) - vp->anon_vma = next->anon_vma; + +if (vmg && vmg->__remove_middle) { + *remove = vmg->middle; + remove = &vp->remove2; + } +if (vmg && vmg->__remove_next) + *remove = vmg->next; + +if (vmg && vmg->__adjust_middle_start) + adjust = vmg->middle; +else if (vmg && vmg->__adjust_next_start) + adjust = vmg->next; +else + adjust =NULL ; + + vp->adj_next = adjust; +if (!vp->anon_vma && adjust) + vp->anon_vma = adjust->anon_vma; + +VM_WARN_ON (vp->anon_vma && adjust && adjust->anon_vma && + vp->anon_vma != adjust->anon_vma); vp->file = vma->vm_file;if (vp->file)
@@ -360,7 +376,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi,*/ static void init_vma_prep (struct vma_prepare *vp,struct vm_area_struct *vma) { -init_multi_vma_prep (vp, vma,NULL ,NULL ,NULL ); +init_multi_vma_prep (vp, vma,NULL ); }/*
@@ -634,76 +650,63 @@ void validate_mm(struct mm_struct *mm)*/ static struct vm_area_struct *commit_merge (struct vma_merge_struct *vmg) { -struct vm_area_struct *remove =NULL ; -struct vm_area_struct *remove2 =NULL ; +struct vm_area_struct *vma;struct vma_prepare vp; -struct vm_area_struct *adjust =NULL ; +struct vm_area_struct *adjust;long adj_start; -bool merge_target;/* * If modifying an existing VMA and we don't remove vmg->middle, then we * shrink the adjacent VMA. */ if (vmg->__adjust_middle_start) { + vma = vmg->target; adjust = vmg->middle;/* The POSITIVE value by which we offset vmg->middle->vm_start. */ adj_start = vmg->end - vmg->middle->vm_start; - merge_target = true; + +/* Note: vma iterator must be pointing to 'start'. */ +vma_iter_config (vmg->vmi, vmg->start, vmg->end); }else if (vmg->__adjust_next_start) { +/* +* In this case alone, the VMA we manipulate is vmg->middle, but +* we ultimately return vmg->next. +*/ + vma = vmg->middle; adjust = vmg->next;/* The NEGATIVE value by which we offset vmg->next->vm_start. */ adj_start = -(vmg->middle->vm_end - vmg->end); -/* -* In all cases but this - merge right, shrink next - we write -* vmg->target to the maple tree and return this as the merged VMA. -*/ - merge_target = false; + +vma_iter_config (vmg->vmi, vmg->next->vm_start + adj_start, + vmg->next->vm_end); }else { + vma = vmg->target; adjust =NULL ; adj_start =0 ; - merge_target = true; - } - -if (vmg->__remove_middle) - remove = vmg->middle; -if (vmg->__remove_next) - remove2 = vmg->next; - -init_multi_vma_prep (&vp, vmg->target, adjust, remove, remove2); -VM_WARN_ON (vp.anon_vma && adjust && adjust->anon_vma && - vp.anon_vma != adjust->anon_vma); - -if (merge_target) { -/* Note: vma iterator must be pointing to 'start'. */ +/* Note: vma iterator must be pointing to 'start'. */ vma_iter_config (vmg->vmi, vmg->start, vmg->end); - }else { -vma_iter_config (vmg->vmi, adjust->vm_start + adj_start, - adjust->vm_end); } -if (vma_iter_prealloc (vmg->vmi, vmg->target)) +init_multi_vma_prep (&vp, vma, vmg); + +if (vma_iter_prealloc (vmg->vmi, vma))return NULL ;vma_prepare (&vp); -vma_adjust_trans_huge (vmg->target, vmg->start, vmg->end, adj_start); -vma_set_range (vmg->target, vmg->start, vmg->end, vmg->pgoff); - -if (merge_target) -vma_iter_store (vmg->vmi, vmg->target); +vma_adjust_trans_huge (vma, vmg->start, vmg->end, adj_start); +vma_set_range (vma, vmg->start, vmg->end, vmg->pgoff);if (adj_start) { adjust->vm_start += adj_start; adjust->vm_pgoff +=PHYS_PFN (adj_start); - -if (!merge_target) -vma_iter_store (vmg->vmi, adjust); } -vma_complete (&vp, vmg->vmi, vmg->target->vm_mm); +vma_iter_store (vmg->vmi, vmg->target); + +vma_complete (&vp, vmg->vmi, vma->vm_mm); -return merge_target ? vmg->target : vmg->next; +return vmg->target; }/* We can only remove VMAs when merging if they do not have a close hook. */
@@ -833,11 +836,15 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(/* No matter what happens, we will be adjusting middle. */ vma_start_write (middle); -if (merge_left) -vma_start_write (prev); - -if (merge_right) +if (merge_right) {vma_start_write (next); + vmg->target = next; + } + +if (merge_left) { +vma_start_write (prev); + vmg->target = prev; + }if (merge_both) {/*
@@ -847,7 +854,6 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(* extend delete delete */ - vmg->target = prev; vmg->start = prev->vm_start; vmg->end = next->vm_end; vmg->pgoff = prev->vm_pgoff;
@@ -868,7 +874,6 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(* extend shrink/delete */ - vmg->target = prev; vmg->start = prev->vm_start; vmg->pgoff = prev->vm_pgoff;
@@ -892,7 +897,6 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(VM_WARN_ON_VMG (vmg->start > middle->vm_start && prev && middle != prev, vmg);if (vmg->__remove_middle) { - vmg->target = next; vmg->end = next->vm_end; vmg->pgoff = next->vm_pgoff - pglen; }else {
@@ -903,7 +907,6 @@ static __must_check struct vm_area_struct *vma_merge_existing_range(* merged VMA is NOT vmg->target, but rather vmg->next. */ vmg->__adjust_next_start = true; - vmg->target = middle; vmg->start = middle->vm_start; vmg->end = start; vmg->pgoff = middle->vm_pgoff;
mm/vma.h +1 -5
@@ -82,11 +82,7 @@ struct vma_merge_struct {struct vm_area_struct *prev;struct vm_area_struct *middle;struct vm_area_struct *next; -/* -* This is the VMA we ultimately target to become the merged VMA, except -* for the one exception of merge right, shrink next (for details of -* this scenario see vma_merge_existing_range()). -*/ +/* This is the VMA we ultimately target to become the merged VMA. */ struct vm_area_struct *target;/* * Initially, the start, end, pgoff fields are provided by the caller
--
2.48.1
PATCH v2 5/5 mm: completely abstract unnecessary adj_start calculation
In reply to Lorenzo StoakesThe adj_start calculation has been a constant source of confusion in the VMA merge code.
There are two cases to consider, one where we adjust the start of the vmg->middle VMA (i.e. the vmg->__adjust_middle_start merge flag is set), in which case adj_start is calculated as:
(1) adj_start = vmg->end - vmg->middle->vm_start
And the case where we adjust the start of the vmg->next VMA (i.e. the vmg->__adjust_next_start merge flag is set), in which case adj_start is calculated as:
(2) adj_start = -(vmg->middle->vm_end - vmg->end)
We apply (1) thusly:
vmg->middle->vm_start =
vmg->middle->vm_start + vmg->end - vmg->middle->vm_start
Which simplifies to:
vmg->middle->vm_start = vmg->end
Similarly, we apply (2) as:
vmg->next->vm_start =
vmg->next->vm_start + -(vmg->middle->vm_end - vmg->end)
Noting that for these VMAs to be mergeable vmg->middle->vm_end == vmg->next->vm_start and so this simplifies to:
vmg->next->vm_start =
vmg->next->vm_start + -(vmg->next->vm_start - vmg->end)
Which simplifies to:
vmg->next->vm_start = vmg->end
Therefore in each case, we simply need to adjust the start of the VMA to vmg->end (!) and can do away with this adj_start calculation. The only caveat is that we must ensure we update the vm_pgoff field correctly.
We therefore abstract this entire calculation to a new function vmg_adjust_set_range() which performs this calculation and sets the adjusted VMA's new range using the general vma_set_range() function.
We also must update vma_adjust_trans_huge() which expects the now-abstracted adj_start parameter. It turns out this is wholly unnecessary.
In vma_adjust_trans_huge() the relevant code is:
if (adjust_next > 0) {
struct vm_area_struct *next = find_vma(vma->vm_mm, vma->vm_end);
unsigned long nstart = next->vm_start;
nstart += adjust_next;
split_huge_pmd_if_needed(next, nstart);
}The only case where this is relevant is when vmg->__adjust_middle_start is specified (in which case adj_next would have been positive), i.e. the one in which the vma specified is vmg->prev and this the sought 'next' VMA would be vmg->middle.
We can therefore eliminate the find_vma() invocation altogether and simply provide the vmg->middle VMA in this instance, or NULL otherwise.
Again we have an adj_next offset calculation:
next->vm_start + vmg->end - vmg->middle->vm_start
Where next == vmg->middle this simplifies to vmg->end as previously demonstrated.
Therefore nstart is equal to vmg->end, which is already passed to vma_adjust_trans_huge() via the 'end' parameter and so this code (rather delightfully) simplifies to:
if (next) split_huge_pmd_if_needed(next, end);
With these changes in place, it becomes silly for commit_merge() to return vmg->target, as it is always the same and threaded through vmg, so we finally change commit_merge() to return an error value once again.
This patch has no change in functional behaviour.
--- include/linux/huge_mm.h | 4 +- mm/huge_memory.c | 19 ++---- mm/vma.c | 99 +++++++++++++++----------------- tools/testing/vma/vma_internal.h | 4 +- 4 files changed, 57 insertions(+), 69 deletions(-)
include/linux/huge_mm.h +2 -2
@@ -404,7 +404,7 @@ int madvise_collapse(struct vm_area_struct *vma,struct vm_area_struct **prev, unsigned long start, unsigned long end);void vma_adjust_trans_huge (struct vm_area_struct *vma,unsigned long start, -unsigned long end,long adjust_next); +unsigned long end,struct vm_area_struct *next);spinlock_t *__pmd_trans_huge_lock (pmd_t *pmd,struct vm_area_struct *vma);spinlock_t *__pud_trans_huge_lock (pud_t *pud,struct vm_area_struct *vma);
@@ -571,7 +571,7 @@ static inline int madvise_collapse(struct vm_area_struct *vma,static inline void vma_adjust_trans_huge (struct vm_area_struct *vma,unsigned long start,unsigned long end, -long adjust_next) +struct vm_area_struct *next) { }static inline int is_swap_pmd (pmd_t pmd)
mm/huge_memory.c +6 -13
@@ -3017,9 +3017,9 @@ static inline void split_huge_pmd_if_needed(struct vm_area_struct *vma, unsigned }void vma_adjust_trans_huge (struct vm_area_struct *vma, -unsigned long start, -unsigned long end, -long adjust_next) +unsigned long start, +unsigned long end, +struct vm_area_struct *next) {/* Check if we need to split start first. */ split_huge_pmd_if_needed (vma, start);
@@ -3027,16 +3027,9 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma,/* Check if we need to split end next. */ split_huge_pmd_if_needed (vma, end); -/* -* If we're also updating the next vma vm_start, -* check if we need to split it. -*/ -if (adjust_next >0 ) { -struct vm_area_struct *next =find_vma (vma->vm_mm, vma->vm_end); -unsigned long nstart = next->vm_start; - nstart += adjust_next; -split_huge_pmd_if_needed (next, nstart); - } +/* If we're incrementing next->vm_start, we might need to split it. */ +if (next) +split_huge_pmd_if_needed (next, end); }static void unmap_folio (struct folio *folio)
mm/vma.c +47 -52
172 lines not on this page · raw message
tools/testing/vma/vma_internal.h +2 -2
@@ -796,12 +796,12 @@ static inline void vma_start_write(struct vm_area_struct *vma)static inline void vma_adjust_trans_huge (struct vm_area_struct *vma,unsigned long start,unsigned long end, -long adjust_next) +struct vm_area_struct *next) { (void )vma; (void )start; (void )end; - (void )adjust_next; + (void )next; }static inline void vma_iter_free (struct vma_iterator *vmi)
--
2.48.1