mm: clean up anon_vma implementation
Subsystems (5)
cover mm: clean up anon_vma implementation
The anon_vma logic is hugely confusing and, much like a bundle of wires entangled with one another, pulling on one thread seems only to lead to more entanglement elsewhere.
There is a mish-mash of the core implementation, how that implementation is invoked, how helper functions are invoked and concepts such as adjacent anon_vma merge and anon_vma object reuse.
This series tries to improve the situation somewhat.
It starts by establishing some invariants in the core anon_vma_clone() and unlink_anon_vmas() functions, largely expressed via VM_WARN_ON_ONCE() asserts.
These act as some form of self-documentation as to the conditions we find ourselves in when invoking these functions.
We also add kdoc comments for anon_vma_clone() and unlink_anon_vmas().
We then makes use of these known conditions to directly skip unfaulted VMAs (rather than implicitly via an empty vma->anon_vma_chain list).
We remove the confusing anon_vma_merge() function (we already have a concept of anon_vma merge in that we merge anon_vma's that would otherwise be compatible except for attributes that mprotect() could change - which anon_vma_merge() has nothing to do with).
We make the anon_vma functions internal to mm as they do not need to be used by any other part of the kernel, which allows for future abstraction without concern about this.
We then reduce the time over which we hold the anon rmap lock in anon_vma_clone(), as it turns out we can allocate anon_vma_chain objects without holding this lock, since the objects are not yet accessible from the rmap.
This should reduce anon_vma lock contention.
This additionally allows us to remove a confusing GFP_NOWAIT, GFP_KERNEL allocation fallback strategy.
Finally, we explicitly indicate which operation is being performed upon anon_vma_clone(), and separate out fork-only logic to make it very clear that anon_vma reuse only occurs on fork.
Lorenzo Stoakes (8):
mm/rmap: improve anon_vma_clone(), unlink_anon_vmas() comments, add
asserts
mm/rmap: skip unfaulted VMAs on anon_vma clone, unlink mm/rmap: remove unnecessary root lock dance in anon_vma clone, unmap mm/rmap: remove anon_vma_merge() function
mm/rmap: make anon_vma functions internal
mm/mmap_lock: add vma_is_detached() helper
mm/rmap: allocate anon_vma_chain objects unlocked when possible mm/rmap: separate out fork-only logic on anon_vma_clone()
include/linux/mmap_lock.h | 9 +- include/linux/rmap.h | 67 --------- mm/internal.h | 67 +++++++++ mm/rmap.c | 232 +++++++++++++++++++------------ mm/vma.c | 8 +- tools/testing/vma/vma_internal.h | 16 ++- 6 files changed, 233 insertions(+), 166 deletions(-)
--
2.52.0
PATCH 1/8 mm/rmap: improve anon_vma_clone(), unlink_anon_vmas() comments, add asserts
Add kdoc comments, describe exactly what these functinos are used for in detail, pointing out importantly that the anon_vma_clone() !dst->anon_vma && src->anon_vma dance is ONLY for fork.
Both are confusing functions that will be refactored in a subsequent patch but the first stage is establishing documentation and some invariatns.
Add some basic CONFIG_DEBUG_VM asserts that help document expected state, specifically:
anon_vma_clone()
- mmap write lock held.
- We do nothing if src VMA is not faulted.
- The destination VMA has no anon_vma_chain yet.
- We are always operating on the same active VMA (i.e. vma->anon-vma).
- If not forking, must operate on the same mm_struct.
unlink_anon_vmas()
- mmap lock held (read on unmap downgraded).
- That unfaulted VMAs are no-ops.
--- mm/rmap.c | 82 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 64 insertions(+), 18 deletions(-)
mm/rmap.c +64 -18
@@ -257,30 +257,60 @@ static inline void unlock_anon_vma_root(struct anon_vma *root)up_write (&root->rwsem); } -/* -* Attach the anon_vmas from src to dst. -* Returns 0 on success, -ENOMEM on failure. -* -* anon_vma_clone() is called by vma_expand(), vma_merge(), __split_vma(), -* copy_vma() and anon_vma_fork(). The first four want an exact copy of src, -* while the last one, anon_vma_fork(), may try to reuse an existing anon_vma to -* prevent endless growth of anon_vma. Since dst->anon_vma is set to NULL before -* call, we can identify this case by checking (!dst->anon_vma && -* src->anon_vma). -* -* If (!dst->anon_vma && src->anon_vma) is true, this function tries to find -* and reuse existing anon_vma which has no vmas and only one child anon_vma. -* This prevents degradation of anon_vma hierarchy to endless linear chain in -* case of constantly forking task. On the other hand, an anon_vma with more -* than one child isn't reused even if there was no alive vma, thus rmap -* walker has a good chance of avoiding scanning the whole hierarchy when it -* searches where page is mapped. +static void check_anon_vma_clone (struct vm_area_struct *dst, +struct vm_area_struct *src) +{ +/* The write lock must be held. */ +mmap_assert_write_locked (src->vm_mm); +/* If not a fork (implied by dst->anon_vma) then must be on same mm. */ +VM_WARN_ON_ONCE (dst->anon_vma && dst->vm_mm != src->vm_mm); + +/* No source anon_vma is a no-op. */ +VM_WARN_ON_ONCE (!src->anon_vma && !list_empty (&src->anon_vma_chain)); +VM_WARN_ON_ONCE (!src->anon_vma && dst->anon_vma); +/* We are establishing a new anon_vma_chain. */ +VM_WARN_ON_ONCE (!list_empty (&dst->anon_vma_chain)); +/* +* On fork, dst->anon_vma is set NULL (temporarily). Otherwise, anon_vma +* must be the same across dst and src. +*/ +VM_WARN_ON_ONCE (dst->anon_vma && dst->anon_vma != src->anon_vma); +} + +/** +* anon_vma_clone - Establishes new anon_vma_chain objects in @dst linking to +* all of the anon_vma objects contained within @src anon_vma_chain's. +* @dst: The destination VMA with an empty anon_vma_chain. +* @src: The source VMA we wish to duplicate. +* +* This is the heart of the VMA side of the anon_vma implementation - we invoke +* this function whenever we need to set up a new VMA's anon_vma state. +* +* This is invoked for: +* +* - VMA Merge, but only when @dst is unfaulted and @src is faulted - meaning we +* clone @src into @dst. +* - VMA split. +* - VMA (m)remap. +* - Fork of faulted VMA. +* +* In all cases other than fork this is simply a duplication. Fork additionally +* adds a new active anon_vma. +* +* ONLY in the case of fork do we try to 'reuse' existing anon_vma's in an +* anon_vma hierarchy, reusing anon_vma's which have no VMA associated with them +* but do have a single child. This is to avoid waste of memory when repeatedly +* forking. +* +* Returns: 0 on success, -ENOMEM on failure. */ int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ; +check_anon_vma_clone (dst, src); +list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {struct anon_vma *anon_vma;
@@ -392,11 +422,27 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)return -ENOMEM ; } +/** +* unlink_anon_vmas() - remove all links between a VMA and anon_vma's, freeing +* anon_vma_chain objects. +* @vma: The VMA whose links to anon_vma objects is to be severed. +* +* As part of the process anon_vma_chain's are freed, +* anon_vma->num_children,num_active_vmas is updated as required and, if the +* relevant anon_vma references no further VMAs, its reference count is +* decremented. +*/ void unlink_anon_vmas (struct vm_area_struct *vma) {struct anon_vma_chain *avc, *next;struct anon_vma *root =NULL ; +/* Always hold mmap lock, read-lock on unmap possibly. */ +mmap_assert_locked (vma->vm_mm); + +/* Unfaulted is a no-op. */ +VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain)); +/* * Unlink each anon_vma chained to the VMA. This list is ordered * from newest to oldest, ensuring the root anon_vma gets freed last.
Signature · 1 line
2.52.0
Re: [PATCH 1/8] mm/rmap: improve anon_vma_clone(), unlink_anon_vmas() comments, add asserts
Quoted 17 lines · Lorenzo Stoakes
Add kdoc comments, describe exactly what these functinos are used for in detail, pointing out importantly that the
anon_vma_clone()!dst->anon_vma &&src->anon_vmadance is ONLY for fork.Both are confusing functions that will be refactored in a subsequent patch but the first stage is establishing documentation and some invariatns.
Add some basic
CONFIG_DEBUG_VMasserts that help document expected state, specifically:
anon_vma_clone()
- mmap write lock held.
- We do nothing if src VMA is not faulted.
- The destination VMA has noanon_vma_chainyet.
- We are always operating on the same active VMA (i.e. vma->anon-vma).
- If not forking, must operate on the samemm_struct.
Quoted 5 lines
unlink_anon_vmas()
- mmap lock held (read on unmap downgraded).
- That unfaulted VMAs are no-ops.Signed-off-by Lorenzo Stoakes <[email protected]>
Quoted 118 lines
--- mm/rmap.c | 82 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 64 insertions(+), 18 deletions(-)mm/rmap.c +64
-18@@ -257,30 +257,60 @@ static inline void unlock_anon_vma_root(struct anon_vma *root)up_write (&root->rwsem); } -/* -* Attach the anon_vmas from src to dst. -* Returns 0 on success, -ENOMEM on failure. -* -* anon_vma_clone() is called by vma_expand(), vma_merge(), __split_vma(), -* copy_vma() and anon_vma_fork(). The first four want an exact copy of src, -* while the last one, anon_vma_fork(), may try to reuse an existing anon_vma to -* prevent endless growth of anon_vma. Since dst->anon_vma is set to NULL before -* call, we can identify this case by checking (!dst->anon_vma && -* src->anon_vma). -* -* If (!dst->anon_vma && src->anon_vma) is true, this function tries to find -* and reuse existing anon_vma which has no vmas and only one child anon_vma. -* This prevents degradation of anon_vma hierarchy to endless linear chain in -* case of constantly forking task. On the other hand, an anon_vma with more -* than one child isn't reused even if there was no alive vma, thus rmap -* walker has a good chance of avoiding scanning the whole hierarchy when it -* searches where page is mapped. +static void check_anon_vma_clone (struct vm_area_struct *dst, +struct vm_area_struct *src) +{ +/* The write lock must be held. */ +mmap_assert_write_locked (src->vm_mm); +/* If not a fork (implied by dst->anon_vma) then must be on same mm. */ +VM_WARN_ON_ONCE (dst->anon_vma && dst->vm_mm != src->vm_mm); + +/* No source anon_vma is a no-op. */ +VM_WARN_ON_ONCE (!src->anon_vma && !list_empty (&src->anon_vma_chain)); +VM_WARN_ON_ONCE (!src->anon_vma && dst->anon_vma); +/* We are establishing a new anon_vma_chain. */ +VM_WARN_ON_ONCE (!list_empty (&dst->anon_vma_chain)); +/* +* On fork, dst->anon_vma is set NULL (temporarily). Otherwise, anon_vma +* must be the same across dst and src. +*/ +VM_WARN_ON_ONCE (dst->anon_vma && dst->anon_vma != src->anon_vma); +} + +/** +* anon_vma_clone - Establishes new anon_vma_chain objects in @dst linking to +* all of the anon_vma objects contained within @src anon_vma_chain's. +* @dst: The destination VMA with an empty anon_vma_chain. +* @src: The source VMA we wish to duplicate. +* +* This is the heart of the VMA side of the anon_vma implementation - we invoke +* this function whenever we need to set up a new VMA's anon_vma state. +* +* This is invoked for: +* +* - VMA Merge, but only when @dst is unfaulted and @src is faulted - meaning we +* clone @src into @dst. +* - VMA split. +* - VMA (m)remap. +* - Fork of faulted VMA. +* +* In all cases other than fork this is simply a duplication. Fork additionally +* adds a new active anon_vma. +* +* ONLY in the case of fork do we try to 'reuse' existing anon_vma's in an +* anon_vma hierarchy, reusing anon_vma's which have no VMA associated with them +* but do have a single child. This is to avoid waste of memory when repeatedly +* forking. +* +* Returns: 0 on success, -ENOMEM on failure. */ int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ; +check_anon_vma_clone (dst, src); +list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {struct anon_vma *anon_vma;@@ -392,11 +422,27 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)return -ENOMEM ; } +/** +* unlink_anon_vmas() - remove all links between a VMA and anon_vma's, freeing +* anon_vma_chain objects. +* @vma: The VMA whose links to anon_vma objects is to be severed. +* +* As part of the process anon_vma_chain's are freed, +* anon_vma->num_children,num_active_vmas is updated as required and, if the +* relevant anon_vma references no further VMAs, its reference count is +* decremented. +*/ void unlink_anon_vmas (struct vm_area_struct *vma) {struct anon_vma_chain *avc, *next;struct anon_vma *root =NULL ; +/* Always hold mmap lock, read-lock on unmap possibly. */ +mmap_assert_locked (vma->vm_mm); + +/* Unfaulted is a no-op. */ +VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain)); +/* * Unlink each anon_vma chained to the VMA. This list is ordered * from newest to oldest, ensuring the root anon_vma gets freed last. Signature · 1 line
2.52.0
Suren Baghdasaryan
maintainerRe: [PATCH 1/8] mm/rmap: improve anon_vma_clone(), unlink_anon_vmas() comments, add asserts
Quoted 9 lines · Liam R. Howlett
Quoted 7 lines · Lorenzo Stoakes
Add kdoc comments, describe exactly what these functinos are used for in detail, pointing out importantly that the
anon_vma_clone()!dst->anon_vma &&src->anon_vmadance is ONLY for fork.Both are confusing functions that will be refactored in a subsequent patch but the first stage is establishing documentation and some invariatns.
Quoted 8 lines
Quoted 8 lines
Add some basic
CONFIG_DEBUG_VMasserts that help document expected state, specifically:
anon_vma_clone()
- mmap write lock held.
- We do nothing if src VMA is not faulted.
- The destination VMA has noanon_vma_chainyet.
- We are always operating on the same active VMA (i.e. vma->anon-vma).
nit: s/vma->anon-vma/vma->anon_vma
Quoted 4 lines
Quoted 4 lines
- If not forking, must operate on the same
mm_struct.
unlink_anon_vmas()
- mmap lock held (read on unmap downgraded).
Out of curiosity I looked for the place where unlink_anon_vmas() is called with mmap_lock downgraded to read but could not find it. Could you please point me to it?
Quoted 37 lines
Quoted 3 lines
- That unfaulted VMAs are no-ops.
Signed-off-by Lorenzo Stoakes <[email protected]>Reviewed-by Liam R. Howlett <[email protected]>Quoted 31 lines
--- mm/rmap.c | 82 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 64 insertions(+), 18 deletions(-)mm/rmap.c +1
-18@@ -257,30 +257,60 @@ static inline void unlock_anon_vma_root(struct anon_vma *root)up_write (&root->rwsem); } -/* -* Attach the anon_vmas from src to dst. -* Returns 0 on success, -ENOMEM on failure. -* -* anon_vma_clone() is called by vma_expand(), vma_merge(), __split_vma(), -* copy_vma() and anon_vma_fork(). The first four want an exact copy of src, -* while the last one, anon_vma_fork(), may try to reuse an existing anon_vma to -* prevent endless growth of anon_vma. Since dst->anon_vma is set to NULL before -* call, we can identify this case by checking (!dst->anon_vma && -* src->anon_vma). -* -* If (!dst->anon_vma && src->anon_vma) is true, this function tries to find -* and reuse existing anon_vma which has no vmas and only one child anon_vma. -* This prevents degradation of anon_vma hierarchy to endless linear chain in -* case of constantly forking task. On the other hand, an anon_vma with more -* than one child isn't reused even if there was no alive vma, thus rmap -* walker has a good chance of avoiding scanning the whole hierarchy when it -* searches where page is mapped. +static void check_anon_vma_clone(struct vm_area_struct *dst,
Quoted 8 lines
mm/rmap.c static inline void unlock_anon_vma_root(struct anon_vma *root) · line 268
Quoted 8 lines
mm/rmap.c +8
-0+ struct vm_area_struct *src) +{ +/* The write lock must be held. */ +mmap_assert_write_locked (src->vm_mm); +/* If not a fork (implied by dst->anon_vma) then must be on same mm. */ +VM_WARN_ON_ONCE (dst->anon_vma && dst->vm_mm != src->vm_mm); + +/* No source anon_vma is a no-op. */
I'm confused about the above comment. Do you mean that if !src->anon_vma then it's a no-op and therefore this function shouldn't be called? If so, we could simply have VM_WARN_ON_ONCE(!src->anon_vma) but checks below have more conditions. Can this comment be perhaps expanded please so that the reader clearly understands what is allowed and what is not. For example, combination (!src->anon_vma && !dst->anon_vma) is allowed and we correctly not triggering a warning here, however that's still a no-op IIUC.
Quoted 7 lines
mm/rmap.c static inline void unlock_anon_vma_root(struct anon_vma *root) · line 275
Quoted 7 lines
mm/rmap.c +7
-0+VM_WARN_ON_ONCE (!src->anon_vma && !list_empty (&src->anon_vma_chain)); +VM_WARN_ON_ONCE (!src->anon_vma && dst->anon_vma); +/* We are establishing a new anon_vma_chain. */ +VM_WARN_ON_ONCE (!list_empty (&dst->anon_vma_chain)); +/* +* On fork, dst->anon_vma is set NULL (temporarily). Otherwise, anon_vma +* must be the same across dst and src.
This is the second time in this small function where we have to remind that dst->anon_vma==NULL means that we are forking. Maybe it's better to introduce a bool forking = dst->anon_vma==NULL; variable at the beginning and use it in all these checks?
I know, I'm nitpicking but as you said, anon_vma code is very compicated, so the more clarity we can bring to it the better.
Quoted 60 lines
Quoted 60 lines
mm/rmap.c +42
-0+*/ +VM_WARN_ON_ONCE (dst->anon_vma && dst->anon_vma != src->anon_vma); +} + +/** +* anon_vma_clone - Establishes new anon_vma_chain objects in @dst linking to +* all of the anon_vma objects contained within @src anon_vma_chain's. +* @dst: The destination VMA with an empty anon_vma_chain. +* @src: The source VMA we wish to duplicate. +* +* This is the heart of the VMA side of the anon_vma implementation - we invoke +* this function whenever we need to set up a new VMA's anon_vma state. +* +* This is invoked for: +* +* - VMA Merge, but only when @dst is unfaulted and @src is faulted - meaning we +* clone @src into @dst. +* - VMA split. +* - VMA (m)remap. +* - Fork of faulted VMA. +* +* In all cases other than fork this is simply a duplication. Fork additionally +* adds a new active anon_vma. +* +* ONLY in the case of fork do we try to 'reuse' existing anon_vma's in an +* anon_vma hierarchy, reusing anon_vma's which have no VMA associated with them +* but do have a single child. This is to avoid waste of memory when repeatedly +* forking. +* +* Returns: 0 on success, -ENOMEM on failure. */ int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ; +check_anon_vma_clone (dst, src); +list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {struct anon_vma *anon_vma;@@ -392,11 +422,27 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)return -ENOMEM ; } +/** +* unlink_anon_vmas() - remove all links between a VMA and anon_vma's, freeing +* anon_vma_chain objects. +* @vma: The VMA whose links to anon_vma objects is to be severed. +* +* As part of the process anon_vma_chain's are freed, +* anon_vma->num_children,num_active_vmas is updated as required and, if the +* relevant anon_vma references no further VMAs, its reference count is +* decremented. +*/ void unlink_anon_vmas (struct vm_area_struct *vma) {struct anon_vma_chain *avc, *next;struct anon_vma *root =NULL ;
Quoted 5 lines
mm/rmap.c int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · line 444
Quoted 5 lines
mm/rmap.c +5
-0+/* Always hold mmap lock, read-lock on unmap possibly. */ +mmap_assert_locked (vma->vm_mm); + +/* Unfaulted is a no-op. */ +VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain));
Hmm. anon_vma_clone() calls unlink_anon_vmas() after setting dst->anon_vma=NULL in the enomem_failure path. This warning would imply that in such case dst->anon_vma_chain is always non-empty. But I don't think we can always expect that... What if the very first call to anon_vma_chain_alloc() in anon_vma_clone()'s loop failed, I think this would result in dst->anon_vma_chain being empty, no?
Quoted 7 lines
Quoted 7 lines
mm/rmap.c +1
-1+/* * Unlink each anon_vma chained to the VMA. This list is ordered * from newest to oldest, ensuring the root anon_vma gets freed last. -- 2.52.0
Suren Baghdasaryan
maintainerRe: [PATCH 1/8] mm/rmap: improve anon_vma_clone(), unlink_anon_vmas() comments, add asserts
Quoted 65 lines · Suren Baghdasaryan
Quoted 9 lines · Liam R. Howlett
Quoted 7 lines · Lorenzo Stoakes
Add kdoc comments, describe exactly what these functinos are used for in detail, pointing out importantly that the
anon_vma_clone()!dst->anon_vma &&src->anon_vmadance is ONLY for fork.Both are confusing functions that will be refactored in a subsequent patch but the first stage is establishing documentation and some invariatns.
Quoted 8 lines
Quoted 8 lines
Add some basic
CONFIG_DEBUG_VMasserts that help document expected state, specifically:
anon_vma_clone()
- mmap write lock held.
- We do nothing if src VMA is not faulted.
- The destination VMA has noanon_vma_chainyet.
- We are always operating on the same active VMA (i.e. vma->anon-vma).nit: s/vma->anon-vma/vma->anon_vma
Quoted 4 lines
Quoted 4 lines
- If not forking, must operate on the same
mm_struct.
unlink_anon_vmas()
- mmap lock held (read on unmap downgraded).Out of curiosity I looked for the place where
unlink_anon_vmas()is called withmmap_lockdowngraded to read but could not find it. Could you please point me to it?Quoted 33 lines
Quoted 3 lines
- That unfaulted VMAs are no-ops.
Signed-off-by Lorenzo Stoakes <[email protected]>Reviewed-by Liam R. Howlett <[email protected]>Quoted 27 lines
--- mm/rmap.c | 82 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 64 insertions(+), 18 deletions(-)mm/rmap.c +0
-15@@ -257,30 +257,60 @@ static inline void unlock_anon_vma_root(struct anon_vma *root)up_write (&root->rwsem); } -/* -* Attach the anon_vmas from src to dst. -* Returns 0 on success, -ENOMEM on failure. -* -* anon_vma_clone() is called by vma_expand(), vma_merge(), __split_vma(), -* copy_vma() and anon_vma_fork(). The first four want an exact copy of src, -* while the last one, anon_vma_fork(), may try to reuse an existing anon_vma to -* prevent endless growth of anon_vma. Since dst->anon_vma is set to NULL before -* call, we can identify this case by checking (!dst->anon_vma && -* src->anon_vma). -* -* If (!dst->anon_vma && src->anon_vma) is true, this function tries to find -* and reuse existing anon_vma which has no vmas and only one child anon_vma. -* This prevents degradation of anon_vma hierarchy to endless linear chain in -* case of constantly forking task. On the other hand, an anon_vma with more
Quoted 8 lines
Quoted 8 lines
mm/rmap.c static inline void unlock_anon_vma_root(struct anon_vma *root) · line 264
Quoted 8 lines
mm/rmap.c +5
-3-* than one child isn't reused even if there was no alive vma, thus rmap -* walker has a good chance of avoiding scanning the whole hierarchy when it -* searches where page is mapped. +static void check_anon_vma_clone (struct vm_area_struct *dst, +struct vm_area_struct *src) +{ +/* The write lock must be held. */ +mmap_assert_write_locked (src->vm_mm);
As a side-note, I think we might be able to claim vma_assert_locked(src) here if we move vma_start_write(vma) at
https://elixir.bootlin.com/linux/v6.19-rc3/source/mm/vma.c#L538 to line 527. I looked over other places where we call anon_vma_clone() and they seem to write-lock src VMA before cloning anon_vma. mmap_assert_write_locked() is sufficient here because we do not change anon_vma under VMA lock (see __vmf_anon_prepare() upgrading to mmap lock if a new anon_vma has to be established during page fault) but I think we can be more strict here.
Quoted 110 lines
Quoted 4 lines
mm/rmap.c static inline void unlock_anon_vma_root(struct anon_vma *root) · line 268
Quoted 4 lines
mm/rmap.c +4
-0+/* If not a fork (implied by dst->anon_vma) then must be on same mm. */ +VM_WARN_ON_ONCE (dst->anon_vma && dst->vm_mm != src->vm_mm); + +/* No source anon_vma is a no-op. */ I'm confused about the above comment. Do you mean that if !src->anon_vma then it's a no-op and therefore this function shouldn't be called? If so, we could simply have
VM_WARN_ON_ONCE(!src->anon_vma)but checks below have more conditions. Can this comment be perhaps expanded please so that the reader clearly understands what is allowed and what is not. For example, combination (!src->anon_vma && !dst->anon_vma) is allowed and we correctly not triggering a warning here, however that's still a no-op IIUC.Quoted 7 lines
mm/rmap.c static inline void unlock_anon_vma_root(struct anon_vma *root) · line 275
Quoted 7 lines
mm/rmap.c +7
-0+VM_WARN_ON_ONCE (!src->anon_vma && !list_empty (&src->anon_vma_chain)); +VM_WARN_ON_ONCE (!src->anon_vma && dst->anon_vma); +/* We are establishing a new anon_vma_chain. */ +VM_WARN_ON_ONCE (!list_empty (&dst->anon_vma_chain)); +/* +* On fork, dst->anon_vma is set NULL (temporarily). Otherwise, anon_vma +* must be the same across dst and src. This is the second time in this small function where we have to remind that dst->anon_vma==NULL means that we are forking. Maybe it's better to introduce a
bool forking = dst->anon_vma==NULL;variable at the beginning and use it in all these checks?I know, I'm nitpicking but as you said,
anon_vmacode is very compicated, so the more clarity we can bring to it the better.Quoted 60 lines
Quoted 60 lines
mm/rmap.c +42
-0+*/ +VM_WARN_ON_ONCE (dst->anon_vma && dst->anon_vma != src->anon_vma); +} + +/** +* anon_vma_clone - Establishes new anon_vma_chain objects in @dst linking to +* all of the anon_vma objects contained within @src anon_vma_chain's. +* @dst: The destination VMA with an empty anon_vma_chain. +* @src: The source VMA we wish to duplicate. +* +* This is the heart of the VMA side of the anon_vma implementation - we invoke +* this function whenever we need to set up a new VMA's anon_vma state. +* +* This is invoked for: +* +* - VMA Merge, but only when @dst is unfaulted and @src is faulted - meaning we +* clone @src into @dst. +* - VMA split. +* - VMA (m)remap. +* - Fork of faulted VMA. +* +* In all cases other than fork this is simply a duplication. Fork additionally +* adds a new active anon_vma. +* +* ONLY in the case of fork do we try to 'reuse' existing anon_vma's in an +* anon_vma hierarchy, reusing anon_vma's which have no VMA associated with them +* but do have a single child. This is to avoid waste of memory when repeatedly +* forking. +* +* Returns: 0 on success, -ENOMEM on failure. */ int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ; +check_anon_vma_clone (dst, src); +list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {struct anon_vma *anon_vma;@@ -392,11 +422,27 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)return -ENOMEM ; } +/** +* unlink_anon_vmas() - remove all links between a VMA and anon_vma's, freeing +* anon_vma_chain objects. +* @vma: The VMA whose links to anon_vma objects is to be severed. +* +* As part of the process anon_vma_chain's are freed, +* anon_vma->num_children,num_active_vmas is updated as required and, if the +* relevant anon_vma references no further VMAs, its reference count is +* decremented. +*/ void unlink_anon_vmas (struct vm_area_struct *vma) {struct anon_vma_chain *avc, *next;struct anon_vma *root =NULL ;Quoted 5 lines
mm/rmap.c int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · line 444
Quoted 5 lines
mm/rmap.c +5
-0+/* Always hold mmap lock, read-lock on unmap possibly. */ +mmap_assert_locked (vma->vm_mm); + +/* Unfaulted is a no-op. */ +VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain));Hmm.
anon_vma_clone()callsunlink_anon_vmas()after setting dst->anon_vma=NULL in theenomem_failurepath. This warning would imply that in such casedst->anon_vma_chainis always non-empty. But I don't think we can always expect that... What if the very first call toanon_vma_chain_alloc()in anon_vma_clone()'s loop failed, I think this would result indst->anon_vma_chainbeing empty, no?Quoted 7 lines
Quoted 7 lines
mm/rmap.c +1
-1+/* * Unlink each anon_vma chained to the VMA. This list is ordered * from newest to oldest, ensuring the root anon_vma gets freed last. -- 2.52.0
Re: [PATCH 1/8] mm/rmap: improve anon_vma_clone(), unlink_anon_vmas() comments, add asserts
In reply to Suren BaghdasaryanQuoted 14 lines · Suren Baghdasaryan
Quoted 12 lines · Liam R. Howlett
Quoted 10 lines · Lorenzo Stoakes
Add kdoc comments, describe exactly what these functinos are used for in detail, pointing out importantly that the
anon_vma_clone()!dst->anon_vma &&src->anon_vmadance is ONLY for fork.Both are confusing functions that will be refactored in a subsequent patch but the first stage is establishing documentation and some invariatns.
Add some basic
CONFIG_DEBUG_VMasserts that help document expected state, specifically:
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
anon_vma_clone()
- mmap write lock held.
- We do nothing if src VMA is not faulted.
- The destination VMA has noanon_vma_chainyet.
- We are always operating on the same active VMA (i.e. vma->anon-vma).nit: s/vma->anon-vma/vma->anon_vma
Thanks will correct.
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
- If not forking, must operate on the same
mm_struct.
unlink_anon_vmas()
- mmap lock held (read on unmap downgraded).Out of curiosity I looked for the place where
unlink_anon_vmas()is called withmmap_lockdowngraded to read but could not find it. Could you please point me to it?
In brk() we call:
-> do_vmi_align_munmap()
-> ... (below)
On munmap() we call:
-> __vm_munmap()
-> do_vmi_munmap()
-> do_vmi_align_munmap()
-> ... (below)
On mremap() when shrinking a VMA in place we call:
-> mremap_at()
-> shrink_vma()
-> do_vmi_munmap()
-> do_vmi_align_munmap()
-> ... (below)
And the ... is:
-> vms_complete_munmap_vmas() [ does downgrade since vms->unlock ] -> vms_clear_ptes()
-> free_pgtables()
I've improved the comment anyway to make it a little clearer.
Quoted 42 lines
Quoted 42 lines
Quoted 4 lines
- That unfaulted VMAs are no-ops.
Signed-off-by Lorenzo Stoakes <[email protected]>Reviewed-by Liam R. Howlett <[email protected]>Quoted 35 lines
--- mm/rmap.c | 82 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 64 insertions(+), 18 deletions(-)mm/rmap.c +5
-18@@ -257,30 +257,60 @@ static inline void unlock_anon_vma_root(struct anon_vma *root)up_write (&root->rwsem); } -/* -* Attach the anon_vmas from src to dst. -* Returns 0 on success, -ENOMEM on failure. -* -* anon_vma_clone() is called by vma_expand(), vma_merge(), __split_vma(), -* copy_vma() and anon_vma_fork(). The first four want an exact copy of src, -* while the last one, anon_vma_fork(), may try to reuse an existing anon_vma to -* prevent endless growth of anon_vma. Since dst->anon_vma is set to NULL before -* call, we can identify this case by checking (!dst->anon_vma && -* src->anon_vma). -* -* If (!dst->anon_vma && src->anon_vma) is true, this function tries to find -* and reuse existing anon_vma which has no vmas and only one child anon_vma. -* This prevents degradation of anon_vma hierarchy to endless linear chain in -* case of constantly forking task. On the other hand, an anon_vma with more -* than one child isn't reused even if there was no alive vma, thus rmap -* walker has a good chance of avoiding scanning the whole hierarchy when it -* searches where page is mapped. +static void check_anon_vma_clone (struct vm_area_struct *dst, +struct vm_area_struct *src) +{ +/* The write lock must be held. */ +mmap_assert_write_locked (src->vm_mm);
Quoted 8 lines
Quoted 4 lines
mm/rmap.c static inline void unlock_anon_vma_root(struct anon_vma *root) · line 268
Quoted 4 lines
mm/rmap.c +4
-0+/* If not a fork (implied by dst->anon_vma) then must be on same mm. */ +VM_WARN_ON_ONCE (dst->anon_vma && dst->vm_mm != src->vm_mm); + +/* No source anon_vma is a no-op. */ I'm confused about the above comment. Do you mean that if !src->anon_vma then it's a no-op and therefore this function shouldn't be called? If so, we could simply have
VM_WARN_ON_ONCE(!src->anon_vma)
It's a no-op :) so it makes no sense to specify other fields. In a later commit we literally bail out of anon_vma_clone() if it's not specified. In fact the very next patch...
Quoted 5 lines
but checks below have more conditions. Can this comment be perhaps expanded please so that the reader clearly understands what is allowed and what is not. For example, combination (!src->anon_vma && !dst->anon_vma) is allowed and we correctly not triggering a warning here, however that's still a no-op IIUC.
Yup it's correct and fine but it's a no-op, hence we have nothing to do, as you say.
I thought it was self-documenting, given I literally spell out the expected conditions in the asserts but obviously this isn't entirely clear. I'm trying _not_ to write paragraphs here as that can actually make things _more_ confusing.
Will update the comment to say more:
/* If we have anything to do src->anon_vma must be provided. */
Quoted 9 lines
Quoted 8 lines
mm/rmap.c static inline void unlock_anon_vma_root(struct anon_vma *root) · line 275
Quoted 8 lines
mm/rmap.c +7
-0+VM_WARN_ON_ONCE (!src->anon_vma && !list_empty (&src->anon_vma_chain)); +VM_WARN_ON_ONCE (!src->anon_vma && dst->anon_vma); +/* We are establishing a new anon_vma_chain. */ +VM_WARN_ON_ONCE (!list_empty (&dst->anon_vma_chain)); +/* +* On fork, dst->anon_vma is set NULL (temporarily). Otherwise, anon_vma +* must be the same across dst and src.
Quoted 4 lines
This is the second time in this small function where we have to remind that dst->anon_vma==NULL means that we are forking. Maybe it's better to introduce a
bool forking = dst->anon_vma==NULL;variable at the beginning and use it in all these checks?
Later we make changes along these lines, so for the purposes of keeping things broken up I'd rather not.
And yes, anon_vma is a complicated mess, this is why I'm trying to do things one step at a time, so we document the things you'd have to go research to understand, later we change the code.
Quoted 3 lines
I know, I'm nitpicking but as you said,
anon_vmacode is very compicated, so the more clarity we can bring to it the better.
Right, sure, but it has to be one thing at a time.
Quoted 61 lines
Quoted 61 lines
Quoted 61 lines
mm/rmap.c +42
-0+*/ +VM_WARN_ON_ONCE (dst->anon_vma && dst->anon_vma != src->anon_vma); +} + +/** +* anon_vma_clone - Establishes new anon_vma_chain objects in @dst linking to +* all of the anon_vma objects contained within @src anon_vma_chain's. +* @dst: The destination VMA with an empty anon_vma_chain. +* @src: The source VMA we wish to duplicate. +* +* This is the heart of the VMA side of the anon_vma implementation - we invoke +* this function whenever we need to set up a new VMA's anon_vma state. +* +* This is invoked for: +* +* - VMA Merge, but only when @dst is unfaulted and @src is faulted - meaning we +* clone @src into @dst. +* - VMA split. +* - VMA (m)remap. +* - Fork of faulted VMA. +* +* In all cases other than fork this is simply a duplication. Fork additionally +* adds a new active anon_vma. +* +* ONLY in the case of fork do we try to 'reuse' existing anon_vma's in an +* anon_vma hierarchy, reusing anon_vma's which have no VMA associated with them +* but do have a single child. This is to avoid waste of memory when repeatedly +* forking. +* +* Returns: 0 on success, -ENOMEM on failure. */ int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ; +check_anon_vma_clone (dst, src); +list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {struct anon_vma *anon_vma;@@ -392,11 +422,27 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)return -ENOMEM ; } +/** +* unlink_anon_vmas() - remove all links between a VMA and anon_vma's, freeing +* anon_vma_chain objects. +* @vma: The VMA whose links to anon_vma objects is to be severed. +* +* As part of the process anon_vma_chain's are freed, +* anon_vma->num_children,num_active_vmas is updated as required and, if the +* relevant anon_vma references no further VMAs, its reference count is +* decremented. +*/ void unlink_anon_vmas (struct vm_area_struct *vma) {struct anon_vma_chain *avc, *next;struct anon_vma *root =NULL ;
Quoted 12 lines
Quoted 5 lines
mm/rmap.c int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · line 444
Quoted 5 lines
mm/rmap.c +5
-0+/* Always hold mmap lock, read-lock on unmap possibly. */ +mmap_assert_locked (vma->vm_mm); + +/* Unfaulted is a no-op. */ +VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain));Hmm.
anon_vma_clone()callsunlink_anon_vmas()after setting dst->anon_vma=NULL in theenomem_failurepath. This warning would imply that in such casedst->anon_vma_chainis always non-empty. But I don't think we can always expect that... What if the very first call toanon_vma_chain_alloc()in anon_vma_clone()'s loop failed, I think this would result indst->anon_vma_chainbeing empty, no?
OK well that's a good spot, though this is never going to actually happen in reality as an allocation failure here would really be 'too small to fail'.
It's a pity we have to give up a completely sensible invariant because of terribly written code for an event that will never happen.
But sure will drop this then, that's awful to have to do though :/
Hey maybe we'd have bot reports on this (would require fault injection) if this had been taken to any tree at any point. Ah well.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
mm/rmap.c +1
-1+/* * Unlink each anon_vma chained to the VMA. This list is ordered * from newest to oldest, ensuring the root anon_vma gets freed last. -- 2.52.0
Thanks, Lorenzo
Re: [PATCH 1/8] mm/rmap: improve anon_vma_clone(), unlink_anon_vmas() comments, add asserts
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines
Quoted 2 lines
mm/rmap.c int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · line 444
Quoted 2 lines
mm/rmap.c +2
-0+/* Unfaulted is a no-op. */ +VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain));Hmm.
anon_vma_clone()callsunlink_anon_vmas()after setting dst->anon_vma=NULL in theenomem_failurepath. This warning would imply that in such casedst->anon_vma_chainis always non-empty. But I don't think we can always expect that... What if the very first call toanon_vma_chain_alloc()in anon_vma_clone()'s loop failed, I think this would result indst->anon_vma_chainbeing empty, no?
Quoted 7 lines
OK well that's a good spot, though this is never going to actually happen in reality as an allocation failure here would really be 'too small to fail'.
It's a pity we have to give up a completely sensible invariant because of terribly written code for an event that will never happen.
But sure will drop this then, that's awful to have to do though :/
Actually let me just update the stupid hack exit path code to increment anon_vma->num_active_vmas in this case, and do it that way (so unlink_anon_vmas() drops it again).
That actually makes the whole thing _less_ of a hack as it really makes zero sense for the anon_vma not to be specified but to be working through vma->anon_vma_chain, and that's a very important invariant.
Re: [PATCH 1/8] mm/rmap: improve anon_vma_clone(), unlink_anon_vmas() comments, add asserts
Quoted 19 lines · Lorenzo Stoakes
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines
Quoted 2 lines
mm/rmap.c int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · line 444
Quoted 2 lines
mm/rmap.c +2
-0+/* Unfaulted is a no-op. */ +VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain));Hmm.
anon_vma_clone()callsunlink_anon_vmas()after setting dst->anon_vma=NULL in theenomem_failurepath. This warning would imply that in such casedst->anon_vma_chainis always non-empty. But I don't think we can always expect that... What if the very first call toanon_vma_chain_alloc()in anon_vma_clone()'s loop failed, I think this would result indst->anon_vma_chainbeing empty, no?Quoted 7 lines
OK well that's a good spot, though this is never going to actually happen in reality as an allocation failure here would really be 'too small to fail'.
It's a pity we have to give up a completely sensible invariant because of terribly written code for an event that will never happen.
But sure will drop this then, that's awful to have to do though :/
Quoted 7 lines
Actually let me just update the stupid hack exit path code to increment
anon_vma->num_active_vmasin this case, and do it that way (sounlink_anon_vmas()drops it again).That actually makes the whole thing _less_ of a hack as it really makes zero sense for the
anon_vmanot to be specified but to be working throughvma->anon_vma_chain, and that's a very important invariant.
Scratch that, this error path ruins the whole thing (we are assigning anon_vma later, because of course we are). How I hate this code.
Will rethink...
Re: [PATCH 1/8] mm/rmap: improve anon_vma_clone(), unlink_anon_vmas() comments, add asserts
Quoted 24 lines · Lorenzo Stoakes
Quoted 23 lines · Lorenzo Stoakes
Quoted 10 lines · Lorenzo Stoakes
Quoted 9 lines
Quoted 2 lines
mm/rmap.c int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · line 444
Quoted 2 lines
mm/rmap.c +2
-0+/* Unfaulted is a no-op. */ +VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain));Hmm.
anon_vma_clone()callsunlink_anon_vmas()after setting dst->anon_vma=NULL in theenomem_failurepath. This warning would imply that in such casedst->anon_vma_chainis always non-empty. But I don't think we can always expect that... What if the very first call toanon_vma_chain_alloc()in anon_vma_clone()'s loop failed, I think this would result indst->anon_vma_chainbeing empty, no?Quoted 7 lines
OK well that's a good spot, though this is never going to actually happen in reality as an allocation failure here would really be 'too small to fail'.
It's a pity we have to give up a completely sensible invariant because of terribly written code for an event that will never happen.
But sure will drop this then, that's awful to have to do though :/
Actually let me just update the stupid hack exit path code to increment
anon_vma->num_active_vmasin this case, and do it that way (sounlink_anon_vmas()drops it again).
Quoted 8 lines
Quoted 3 lines
That actually makes the whole thing _less_ of a hack as it really makes zero sense for the
anon_vmanot to be specified but to be working throughvma->anon_vma_chain, and that's a very important invariant.Scratch that, this error path ruins the whole thing (we are assigning
anon_vmalater, because of course we are). How I hate this code.Will rethink...
OK I think will add a specific partial cleanup path.
We know we are not going to hit any empty anon_vma's because we hold the exclusive mmap write lock, so this can be radically simplified. That way we don't have VMAs with partially established anon_vma state (both vma->anon_vma and vma->anon_vma_chain) being sent to the core unlink function which is also an improvement.
Suren Baghdasaryan
maintainerRe: [PATCH 1/8] mm/rmap: improve anon_vma_clone(), unlink_anon_vmas() comments, add asserts
In reply to Lorenzo StoakesQuoted 56 lines · Lorenzo Stoakes
Quoted 14 lines · Suren Baghdasaryan
Quoted 12 lines · Liam R. Howlett
Quoted 10 lines · Lorenzo Stoakes
Add kdoc comments, describe exactly what these functinos are used for in detail, pointing out importantly that the
anon_vma_clone()!dst->anon_vma &&src->anon_vmadance is ONLY for fork.Both are confusing functions that will be refactored in a subsequent patch but the first stage is establishing documentation and some invariatns.
Add some basic
CONFIG_DEBUG_VMasserts that help document expected state, specifically:Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
anon_vma_clone()
- mmap write lock held.
- We do nothing if src VMA is not faulted.
- The destination VMA has noanon_vma_chainyet.
- We are always operating on the same active VMA (i.e. vma->anon-vma).nit: s/vma->anon-vma/vma->anon_vma
Thanks will correct.
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
- If not forking, must operate on the same
mm_struct.
unlink_anon_vmas()
- mmap lock held (read on unmap downgraded).Out of curiosity I looked for the place where
unlink_anon_vmas()is called withmmap_lockdowngraded to read but could not find it. Could you please point me to it?In
brk()we call:->
do_vmi_align_munmap()
-> ... (below)On
munmap()we call:->
__vm_munmap()
->do_vmi_munmap()
->do_vmi_align_munmap()
-> ... (below)On
mremap()when shrinking a VMA in place we call:->
mremap_at()
->shrink_vma()
->do_vmi_munmap()
->do_vmi_align_munmap()
-> ... (below)
Quoted 7 lines
And the ... is:
->
vms_complete_munmap_vmas()[ does downgrade sincevms->unlock] ->vms_clear_ptes()
->free_pgtables()I've improved the comment anyway to make it a little clearer.
Ah, now I see. Thanks!
Quoted 62 lines
Quoted 43 lines
Quoted 43 lines
Quoted 5 lines
- That unfaulted VMAs are no-ops.
Signed-off-by Lorenzo Stoakes <[email protected]>Reviewed-by Liam R. Howlett <[email protected]>Quoted 35 lines
--- mm/rmap.c | 82 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 64 insertions(+), 18 deletions(-)mm/rmap.c +5
-18@@ -257,30 +257,60 @@ static inline void unlock_anon_vma_root(struct anon_vma *root)up_write (&root->rwsem); } -/* -* Attach the anon_vmas from src to dst. -* Returns 0 on success, -ENOMEM on failure. -* -* anon_vma_clone() is called by vma_expand(), vma_merge(), __split_vma(), -* copy_vma() and anon_vma_fork(). The first four want an exact copy of src, -* while the last one, anon_vma_fork(), may try to reuse an existing anon_vma to -* prevent endless growth of anon_vma. Since dst->anon_vma is set to NULL before -* call, we can identify this case by checking (!dst->anon_vma && -* src->anon_vma). -* -* If (!dst->anon_vma && src->anon_vma) is true, this function tries to find -* and reuse existing anon_vma which has no vmas and only one child anon_vma. -* This prevents degradation of anon_vma hierarchy to endless linear chain in -* case of constantly forking task. On the other hand, an anon_vma with more -* than one child isn't reused even if there was no alive vma, thus rmap -* walker has a good chance of avoiding scanning the whole hierarchy when it -* searches where page is mapped. +static void check_anon_vma_clone (struct vm_area_struct *dst, +struct vm_area_struct *src) +{ +/* The write lock must be held. */ +mmap_assert_write_locked (src->vm_mm);Quoted 8 lines
Quoted 4 lines
mm/rmap.c static inline void unlock_anon_vma_root(struct anon_vma *root) · line 268
Quoted 4 lines
mm/rmap.c +4
-0+/* If not a fork (implied by dst->anon_vma) then must be on same mm. */ +VM_WARN_ON_ONCE (dst->anon_vma && dst->vm_mm != src->vm_mm); + +/* No source anon_vma is a no-op. */ I'm confused about the above comment. Do you mean that if !src->anon_vma then it's a no-op and therefore this function shouldn't be called? If so, we could simply have
VM_WARN_ON_ONCE(!src->anon_vma)It's a no-op :) so it makes no sense to specify other fields. In a later commit we literally bail out of
anon_vma_clone()if it's not specified. In fact the very next patch...Quoted 5 lines
but checks below have more conditions. Can this comment be perhaps expanded please so that the reader clearly understands what is allowed and what is not. For example, combination (!src->anon_vma && !dst->anon_vma) is allowed and we correctly not triggering a warning here, however that's still a no-op IIUC.
Quoted 7 lines
Yup it's correct and fine but it's a no-op, hence we have nothing to do, as you say.
I thought it was self-documenting, given I literally spell out the expected conditions in the asserts but obviously this isn't entirely clear. I'm trying _not_ to write paragraphs here as that can actually make things _more_ confusing.
Yeah, that comment just confused me a bit. If it's no-op then other conditions should not matter, yet we are asserting them. Anyway, I undersdand the intention and new new comment or no comment at all are fine with me.
Quoted 27 lines
Will update the comment to say more:
/* If we have anything to do src->anon_vma must be provided. */Quoted 9 lines
Quoted 8 lines
mm/rmap.c static inline void unlock_anon_vma_root(struct anon_vma *root) · line 275
Quoted 8 lines
mm/rmap.c +7
-0+VM_WARN_ON_ONCE (!src->anon_vma && !list_empty (&src->anon_vma_chain)); +VM_WARN_ON_ONCE (!src->anon_vma && dst->anon_vma); +/* We are establishing a new anon_vma_chain. */ +VM_WARN_ON_ONCE (!list_empty (&dst->anon_vma_chain)); +/* +* On fork, dst->anon_vma is set NULL (temporarily). Otherwise, anon_vma +* must be the same across dst and src. Quoted 4 lines
This is the second time in this small function where we have to remind that dst->anon_vma==NULL means that we are forking. Maybe it's better to introduce a
bool forking = dst->anon_vma==NULL;variable at the beginning and use it in all these checks?Later we make changes along these lines, so for the purposes of keeping things broken up I'd rather not.
And yes,
anon_vmais a complicated mess, this is why I'm trying to do things one step at a time, so we document the things you'd have to go research to understand, later we change the code.Quoted 1 line
Quoted 4 lines
Quoted 2 lines
I know, I'm nitpicking but as you said,
anon_vmacode is very compicated, so the more clarity we can bring to it the better.Right, sure, but it has to be one thing at a time.
Ack.
Quoted 78 lines
Quoted 62 lines
Quoted 62 lines
Quoted 62 lines
mm/rmap.c +42
-0+*/ +VM_WARN_ON_ONCE (dst->anon_vma && dst->anon_vma != src->anon_vma); +} + +/** +* anon_vma_clone - Establishes new anon_vma_chain objects in @dst linking to +* all of the anon_vma objects contained within @src anon_vma_chain's. +* @dst: The destination VMA with an empty anon_vma_chain. +* @src: The source VMA we wish to duplicate. +* +* This is the heart of the VMA side of the anon_vma implementation - we invoke +* this function whenever we need to set up a new VMA's anon_vma state. +* +* This is invoked for: +* +* - VMA Merge, but only when @dst is unfaulted and @src is faulted - meaning we +* clone @src into @dst. +* - VMA split. +* - VMA (m)remap. +* - Fork of faulted VMA. +* +* In all cases other than fork this is simply a duplication. Fork additionally +* adds a new active anon_vma. +* +* ONLY in the case of fork do we try to 'reuse' existing anon_vma's in an +* anon_vma hierarchy, reusing anon_vma's which have no VMA associated with them +* but do have a single child. This is to avoid waste of memory when repeatedly +* forking. +* +* Returns: 0 on success, -ENOMEM on failure. */ int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ; +check_anon_vma_clone (dst, src); +list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {struct anon_vma *anon_vma;@@ -392,11 +422,27 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)return -ENOMEM ; } +/** +* unlink_anon_vmas() - remove all links between a VMA and anon_vma's, freeing +* anon_vma_chain objects. +* @vma: The VMA whose links to anon_vma objects is to be severed. +* +* As part of the process anon_vma_chain's are freed, +* anon_vma->num_children,num_active_vmas is updated as required and, if the +* relevant anon_vma references no further VMAs, its reference count is +* decremented. +*/ void unlink_anon_vmas (struct vm_area_struct *vma) {struct anon_vma_chain *avc, *next;struct anon_vma *root =NULL ;Quoted 12 lines
Quoted 5 lines
mm/rmap.c int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma) · line 444
Quoted 5 lines
mm/rmap.c +5
-0+/* Always hold mmap lock, read-lock on unmap possibly. */ +mmap_assert_locked (vma->vm_mm); + +/* Unfaulted is a no-op. */ +VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain));Hmm.
anon_vma_clone()callsunlink_anon_vmas()after setting dst->anon_vma=NULL in theenomem_failurepath. This warning would imply that in such casedst->anon_vma_chainis always non-empty. But I don't think we can always expect that... What if the very first call toanon_vma_chain_alloc()in anon_vma_clone()'s loop failed, I think this would result indst->anon_vma_chainbeing empty, no?OK well that's a good spot, though this is never going to actually happen in reality as an allocation failure here would really be 'too small to fail'.
Quoted 7 lines
It's a pity we have to give up a completely sensible invariant because of terribly written code for an event that will never happen.
But sure will drop this then, that's awful to have to do though :/
Hey maybe we'd have bot reports on this (would require fault injection) if this had been taken to any tree at any point. Ah well.
I'll look into the new version to see the final result. Thanks!
Quoted 11 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
mm/rmap.c +1
-1+/* * Unlink each anon_vma chained to the VMA. This list is ordered * from newest to oldest, ensuring the root anon_vma gets freed last. -- 2.52.0
Thanks, Lorenzo
Re: [PATCH 1/8] mm/rmap: improve anon_vma_clone(), unlink_anon_vmas() comments, add asserts
In reply to Liam R. HowlettQuoted 18 lines · Liam R. Howlett
Quoted 17 lines · Lorenzo Stoakes
Add kdoc comments, describe exactly what these functinos are used for in detail, pointing out importantly that the
anon_vma_clone()!dst->anon_vma &&src->anon_vmadance is ONLY for fork.Both are confusing functions that will be refactored in a subsequent patch but the first stage is establishing documentation and some invariatns.
Add some basic
CONFIG_DEBUG_VMasserts that help document expected state, specifically:
anon_vma_clone()
- mmap write lock held.
- We do nothing if src VMA is not faulted.
- The destination VMA has noanon_vma_chainyet.
- We are always operating on the same active VMA (i.e. vma->anon-vma).
- If not forking, must operate on the samemm_struct.
Quoted 7 lines
Quoted 5 lines
unlink_anon_vmas()
- mmap lock held (read on unmap downgraded).
- That unfaulted VMAs are no-ops.Signed-off-by Lorenzo Stoakes <[email protected]>Reviewed-by Liam R. Howlett <[email protected]>
Thanks!
PATCH 2/8 mm/rmap: skip unfaulted VMAs on anon_vma clone, unlink
In reply to Lorenzo StoakesFor both anon_vma_clone() and unlink_anon_vmas(), if the source VMA or the VMA to be linked are unfaulted (e.g. !vma->anon_vma), then the functions do nothing. Simply exit early in these cases.
In the unlink_anon_vmas() case we can also remove a conditional that checks whether vma->anon_vma is set.
--- mm/rmap.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)
mm/rmap.c +11 -9
@@ -309,6 +309,9 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ; +if (!src->anon_vma) +return 0 ; +check_anon_vma_clone (dst, src);list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {
@@ -441,7 +444,8 @@ void unlink_anon_vmas(struct vm_area_struct *vma)mmap_assert_locked (vma->vm_mm);/* Unfaulted is a no-op. */ -VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain)); +if (!vma->anon_vma) +return ;/* * Unlink each anon_vma chained to the VMA. This list is ordered
@@ -465,15 +469,13 @@ void unlink_anon_vmas(struct vm_area_struct *vma)list_del (&avc->same_vma);anon_vma_chain_free (avc); } -if (vma->anon_vma) { - vma->anon_vma->num_active_vmas--; -/* -* vma would still be needed after unlink, and anon_vma will be prepared -* when handle fault. -*/ - vma->anon_vma =NULL ; - } + vma->anon_vma->num_active_vmas--; +/* +* vma would still be needed after unlink, and anon_vma will be prepared +* when handle fault. +*/ + vma->anon_vma =NULL ;unlock_anon_vma_root (root);/*
Signature · 1 line
2.52.0
Re: [PATCH 2/8] mm/rmap: skip unfaulted VMAs on anon_vma clone, unlink
Quoted 27 lines · Lorenzo Stoakes
For both
anon_vma_clone()andunlink_anon_vmas(), if the source VMA or the VMA to be linked are unfaulted (e.g. !vma->anon_vma), then the functions do nothing. Simply exit early in these cases.In the
unlink_anon_vmas()case we can also remove a conditional that checks whethervma->anon_vmais set.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/rmap.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)mm/rmap.c +3
-0@@ -309,6 +309,9 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ; +if (!src->anon_vma) +return 0 ; +check_anon_vma_clone (dst, src);list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {
mm/rmap.c void unlink_anon_vmas(struct vm_area_struct *vma) · line 448
Quoted 6 lines
mm/rmap.c +2
-1@@ -441,7 +444,8 @@ void unlink_anon_vmas(struct vm_area_struct *vma)mmap_assert_locked (vma->vm_mm);/* Unfaulted is a no-op. */ -VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain)); +if (!vma->anon_vma) +return ;
I guess it doesn't matter because you just added the !list_empty() check, but did you mean to drop that part?
Quoted 28 lines
mm/rmap.c +6
-8/* * Unlink each anon_vma chained to the VMA. This list is ordered @@ -465,15 +469,13 @@ void unlink_anon_vmas(struct vm_area_struct *vma)list_del (&avc->same_vma);anon_vma_chain_free (avc); } -if (vma->anon_vma) { - vma->anon_vma->num_active_vmas--; -/* -* vma would still be needed after unlink, and anon_vma will be prepared -* when handle fault. -*/ - vma->anon_vma =NULL ; - } + vma->anon_vma->num_active_vmas--; +/* +* vma would still be needed after unlink, and anon_vma will be prepared +* when handle fault. +*/ + vma->anon_vma =NULL ;unlock_anon_vma_root (root);/* Signature · 1 line
2.52.0
Suren Baghdasaryan
maintainerRe: [PATCH 2/8] mm/rmap: skip unfaulted VMAs on anon_vma clone, unlink
Quoted 22 lines · Liam R. Howlett
Quoted 20 lines · Lorenzo Stoakes
For both
anon_vma_clone()andunlink_anon_vmas(), if the source VMA or the VMA to be linked are unfaulted (e.g. !vma->anon_vma), then the functions do nothing. Simply exit early in these cases.In the
unlink_anon_vmas()case we can also remove a conditional that checks whethervma->anon_vmais set.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/rmap.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)mm/rmap.c +0
-0@@ -309,6 +309,9 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ;
Quoted 4 lines
mm/rmap.c int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) · line 315
Quoted 4 lines
mm/rmap.c +3
-0+if (!src->anon_vma) +return 0 ; +check_anon_vma_clone (dst, src);
check_anon_vma_clone() is used only here and contains a couple of warnings with "!src->anon_vma && ..." conditions, which now will never be triggered even if the second part of those conditions was true. This seems like a regression (you are not checking conditions you were checking before this change). To avoid that, you can place this early exit after the call to check_anon_vma_clone(dst, src).
Quoted 41 lines
mm/rmap.c void unlink_anon_vmas(struct vm_area_struct *vma) · line 448
Quoted 9 lines
mm/rmap.c +2
-1list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {@@ -441,7 +444,8 @@ void unlink_anon_vmas(struct vm_area_struct *vma)mmap_assert_locked (vma->vm_mm);/* Unfaulted is a no-op. */ -VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain)); +if (!vma->anon_vma) +return ;I guess it doesn't matter because you just added the !list_empty() check, but did you mean to drop that part?
Quoted 28 lines
mm/rmap.c +6
-9/* * Unlink each anon_vma chained to the VMA. This list is ordered @@ -465,15 +469,13 @@ void unlink_anon_vmas(struct vm_area_struct *vma)list_del (&avc->same_vma);anon_vma_chain_free (avc); } -if (vma->anon_vma) { - vma->anon_vma->num_active_vmas--; -/* -* vma would still be needed after unlink, and anon_vma will be prepared -* when handle fault. -*/ - vma->anon_vma =NULL ; - } + vma->anon_vma->num_active_vmas--; +/* +* vma would still be needed after unlink, and anon_vma will be prepared +* when handle fault. +*/ + vma->anon_vma =NULL ;unlock_anon_vma_root (root);/* -- 2.52.0
Re: [PATCH 2/8] mm/rmap: skip unfaulted VMAs on anon_vma clone, unlink
Quoted 24 lines · Suren Baghdasaryan
Quoted 22 lines · Liam R. Howlett
Quoted 20 lines · Lorenzo Stoakes
For both
anon_vma_clone()andunlink_anon_vmas(), if the source VMA or the VMA to be linked are unfaulted (e.g. !vma->anon_vma), then the functions do nothing. Simply exit early in these cases.In the
unlink_anon_vmas()case we can also remove a conditional that checks whethervma->anon_vmais set.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/rmap.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)mm/rmap.c +0
-0@@ -309,6 +309,9 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ;
Quoted 11 lines
Quoted 4 lines
mm/rmap.c int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) · line 315
Quoted 4 lines
mm/rmap.c +3
-0+if (!src->anon_vma) +return 0 ; +check_anon_vma_clone (dst, src);
check_anon_vma_clone()is used only here and contains a couple of warnings with "!src->anon_vma && ..." conditions, which now will never be triggered even if the second part of those conditions was true. This seems like a regression (you are not checking conditions you were checking before this change). To avoid that, you can place this early exit after the call tocheck_anon_vma_clone(dst, src).
Yup am aware. Was because later on we do stuff that assumes src.
I guess I can move it and then change the later commit.
Quoted 42 lines
Quoted 42 lines
mm/rmap.c void unlink_anon_vmas(struct vm_area_struct *vma) · line 448
Quoted 10 lines
mm/rmap.c +2
-1list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {@@ -441,7 +444,8 @@ void unlink_anon_vmas(struct vm_area_struct *vma)mmap_assert_locked (vma->vm_mm);/* Unfaulted is a no-op. */ -VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain)); +if (!vma->anon_vma) +return ;I guess it doesn't matter because you just added the !list_empty() check, but did you mean to drop that part?
Quoted 28 lines
mm/rmap.c +6
-9/* * Unlink each anon_vma chained to the VMA. This list is ordered @@ -465,15 +469,13 @@ void unlink_anon_vmas(struct vm_area_struct *vma)list_del (&avc->same_vma);anon_vma_chain_free (avc); } -if (vma->anon_vma) { - vma->anon_vma->num_active_vmas--; -/* -* vma would still be needed after unlink, and anon_vma will be prepared -* when handle fault. -*/ - vma->anon_vma =NULL ; - } + vma->anon_vma->num_active_vmas--; +/* +* vma would still be needed after unlink, and anon_vma will be prepared +* when handle fault. +*/ + vma->anon_vma =NULL ;unlock_anon_vma_root (root);/* -- 2.52.0
Re: [PATCH 2/8] mm/rmap: skip unfaulted VMAs on anon_vma clone, unlink
In reply to Liam R. HowlettQuoted 30 lines · Liam R. Howlett
Quoted 29 lines · Lorenzo Stoakes
For both
anon_vma_clone()andunlink_anon_vmas(), if the source VMA or the VMA to be linked are unfaulted (e.g. !vma->anon_vma), then the functions do nothing. Simply exit early in these cases.In the
unlink_anon_vmas()case we can also remove a conditional that checks whethervma->anon_vmais set.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/rmap.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)mm/rmap.c +3
-0@@ -309,6 +309,9 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ; +if (!src->anon_vma) +return 0 ; +check_anon_vma_clone (dst, src);list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {@@ -441,7 +444,8 @@ void unlink_anon_vmas(struct vm_area_struct *vma)mmap_assert_locked (vma->vm_mm);
Quoted 7 lines
mm/rmap.c void unlink_anon_vmas(struct vm_area_struct *vma) · line 448
Quoted 4 lines
mm/rmap.c +2
-1/* Unfaulted is a no-op. */ -VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain)); +if (!vma->anon_vma) +return ;I guess it doesn't matter because you just added the !list_empty() check, but did you mean to drop that part?
I did mean to. Really this doesn't happen in reality, the assert was more of a place holder I suppose.
I don't think we should be falling over ourselves to assert impossible things, really the debug-only asserts are intended to essentially document what's going on.
Anyway it's moot, as I've had to drop both the assert and the condition here sadly, because of the fact we (of course) use unlink_anon_vmas() to clean up incompletely set up anon_vma's on a destination VMA.
When has doing things on incompletely setup up VMAs ever gone wrong :)
As ever with anon_vma, there are always deeper depths of horror to find.
Quoted 29 lines
Quoted 29 lines
mm/rmap.c +6
-9/* * Unlink each anon_vma chained to the VMA. This list is ordered @@ -465,15 +469,13 @@ void unlink_anon_vmas(struct vm_area_struct *vma)list_del (&avc->same_vma);anon_vma_chain_free (avc); } -if (vma->anon_vma) { - vma->anon_vma->num_active_vmas--; -/* -* vma would still be needed after unlink, and anon_vma will be prepared -* when handle fault. -*/ - vma->anon_vma =NULL ; - } + vma->anon_vma->num_active_vmas--; +/* +* vma would still be needed after unlink, and anon_vma will be prepared +* when handle fault. +*/ + vma->anon_vma =NULL ;unlock_anon_vma_root (root);/* -- 2.52.0
Re: [PATCH 2/8] mm/rmap: skip unfaulted VMAs on anon_vma clone, unlink
Quoted 46 lines · Lorenzo Stoakes
Quoted 30 lines · Liam R. Howlett
Quoted 29 lines · Lorenzo Stoakes
For both
anon_vma_clone()andunlink_anon_vmas(), if the source VMA or the VMA to be linked are unfaulted (e.g. !vma->anon_vma), then the functions do nothing. Simply exit early in these cases.In the
unlink_anon_vmas()case we can also remove a conditional that checks whethervma->anon_vmais set.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/rmap.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-)mm/rmap.c +3
-0@@ -309,6 +309,9 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)struct anon_vma_chain *avc, *pavc;struct anon_vma *root =NULL ; +if (!src->anon_vma) +return 0 ; +check_anon_vma_clone (dst, src);list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {@@ -441,7 +444,8 @@ void unlink_anon_vmas(struct vm_area_struct *vma)mmap_assert_locked (vma->vm_mm);Quoted 7 lines
mm/rmap.c void unlink_anon_vmas(struct vm_area_struct *vma) · line 448
Quoted 4 lines
mm/rmap.c +2
-1/* Unfaulted is a no-op. */ -VM_WARN_ON_ONCE (!vma->anon_vma && !list_empty (&vma->anon_vma_chain)); +if (!vma->anon_vma) +return ;I guess it doesn't matter because you just added the !list_empty() check, but did you mean to drop that part?
I did mean to. Really this doesn't happen in reality, the assert was more of a place holder I suppose.
I don't think we should be falling over ourselves to assert impossible things, really the debug-only asserts are intended to essentially document what's going on.
Quoted 8 lines
Anyway it's moot, as I've had to drop both the assert and the condition here sadly, because of the fact we (of course) use
unlink_anon_vmas()to clean up incompletely set up anon_vma's on a destination VMA.When has doing things on incompletely setup up VMAs ever gone wrong :)
As ever with
anon_vma, there are always deeper depths of horror to find.
OK scratch that see 1/8 thread... I will add a specific partial cleanup path to deal with the anon_vma_clone() horror show and then we get to keep all this.
In which case I guess I'll reinstate the check just to be super careful :)
PATCH 3/8 mm/rmap: remove unnecessary root lock dance in anon_vma clone, unmap
In reply to Lorenzo StoakesThe root anon_vma of all anon_vma's linked to a VMA must by definition be the same - a VMA and all of its descendants/ancestors must exist in the same CoW chain.
Commit bb4aa39676f7 ("mm: avoid repeated anon_vma lock/unlock sequences in anon_vma_clone()") introduced paranoid checking of the root anon_vma remaining the same throughout all AVC's in 2011.
I think 15 years later we can safely assume that this is always the case.
Additionally, since unfaulted VMAs being cloned from or unlinked are no-op's, we can simply lock the anon_vma's associated with this rather than doing any specific dance around this.
This removes unnecessary checks and makes it clear that the root anon_vma is shared between all anon_vma's in a given VMA's anon_vma_chain.
--- mm/rmap.c | 48 ++++++++++++------------------------------------ 1 file changed, 12 insertions(+), 36 deletions(-)
mm/rmap.c +12 -36
@@ -231,32 +231,6 @@ int __anon_vma_prepare(struct vm_area_struct *vma)return -ENOMEM ; } -/* -* This is a useful helper function for locking the anon_vma root as -* we traverse the vma->anon_vma_chain, looping over anon_vma's that -* have the same vma. -* -* Such anon_vma's should have the same root, so you'd expect to see -* just a single mutex_lock for the whole traversal. -*/ -static inline struct anon_vma *lock_anon_vma_root (struct anon_vma *root,struct anon_vma *anon_vma) -{ -struct anon_vma *new_root = anon_vma->root; -if (new_root != root) { -if (WARN_ON_ONCE (root)) -up_write (&root->rwsem); - root = new_root; -down_write (&root->rwsem); - } -return root; -} - -static inline void unlock_anon_vma_root (struct anon_vma *root) -{ -if (root) -up_write (&root->rwsem); -} -static void check_anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {
@@ -307,26 +281,25 @@ static void check_anon_vma_clone(struct vm_area_struct *dst,int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {struct anon_vma_chain *avc, *pavc; -struct anon_vma *root =NULL ;if (!src->anon_vma)return 0 ;check_anon_vma_clone (dst, src); +anon_vma_lock_write (src->anon_vma);list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {struct anon_vma *anon_vma; avc =anon_vma_chain_alloc (GFP_NOWAIT );if (unlikely (!avc)) { -unlock_anon_vma_root (root); - root =NULL ; +anon_vma_unlock_write (src->anon_vma); avc =anon_vma_chain_alloc (GFP_KERNEL );if (!avc) goto enomem_failure; +anon_vma_lock_write (src->anon_vma); } anon_vma = pavc->anon_vma; - root =lock_anon_vma_root (root, anon_vma);anon_vma_chain_link (dst, avc, anon_vma);/*
@@ -343,7 +316,8 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) }if (dst->anon_vma) dst->anon_vma->num_active_vmas++; -unlock_anon_vma_root (root); + +anon_vma_unlock_write (src->anon_vma);return 0 ; enomem_failure:
@@ -438,15 +412,17 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)void unlink_anon_vmas (struct vm_area_struct *vma) {struct anon_vma_chain *avc, *next; -struct anon_vma *root =NULL ; +struct anon_vma *active_anon_vma = vma->anon_vma;/* Always hold mmap lock, read-lock on unmap possibly. */ mmap_assert_locked (vma->vm_mm);/* Unfaulted is a no-op. */ -if (!vma->anon_vma) +if (!active_anon_vma)return ; +anon_vma_lock_write (active_anon_vma); +/* * Unlink each anon_vma chained to the VMA. This list is ordered * from newest to oldest, ensuring the root anon_vma gets freed last.
@@ -454,7 +430,6 @@ void unlink_anon_vmas(struct vm_area_struct *vma)list_for_each_entry_safe (avc, next, &vma->anon_vma_chain, same_vma) {struct anon_vma *anon_vma = avc->anon_vma; - root =lock_anon_vma_root (root, anon_vma);anon_vma_interval_tree_remove (avc, &anon_vma->rb_root);/*
@@ -470,13 +445,14 @@ void unlink_anon_vmas(struct vm_area_struct *vma)anon_vma_chain_free (avc); } - vma->anon_vma->num_active_vmas--; + active_anon_vma->num_active_vmas--;/* * vma would still be needed after unlink, and anon_vma will be prepared * when handle fault. */ vma->anon_vma =NULL ; -unlock_anon_vma_root (root); +anon_vma_unlock_write (active_anon_vma); +/* * Iterate the list once more, it now only contains empty and unlinked
Signature · 1 line
2.52.0
Suren Baghdasaryan
maintainerRe: [PATCH 3/8] mm/rmap: remove unnecessary root lock dance in anon_vma clone, unmap
Quoted 74 lines · Lorenzo Stoakes
The root
anon_vmaof all anon_vma's linked to a VMA must by definition be the same - a VMA and all of its descendants/ancestors must exist in the same CoW chain.Commit
bb4aa39676f7("mm: avoid repeatedanon_vmalock/unlock sequences inanon_vma_clone()") introduced paranoid checking of the rootanon_vmaremaining the same throughout all AVC's in 2011.I think 15 years later we can safely assume that this is always the case.
Additionally, since unfaulted VMAs being cloned from or unlinked are no-op's, we can simply lock the anon_vma's associated with this rather than doing any specific dance around this.
This removes unnecessary checks and makes it clear that the root
anon_vmais shared between all anon_vma's in a given VMA'sanon_vma_chain.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/rmap.c | 48 ++++++++++++------------------------------------ 1 file changed, 12 insertions(+), 36 deletions(-)mm/rmap.c +1
-27@@ -231,32 +231,6 @@ int __anon_vma_prepare(struct vm_area_struct *vma)return -ENOMEM ; } -/* -* This is a useful helper function for locking the anon_vma root as -* we traverse the vma->anon_vma_chain, looping over anon_vma's that -* have the same vma. -* -* Such anon_vma's should have the same root, so you'd expect to see -* just a single mutex_lock for the whole traversal. -*/ -static inline struct anon_vma *lock_anon_vma_root (struct anon_vma *root,struct anon_vma *anon_vma) -{ -struct anon_vma *new_root = anon_vma->root; -if (new_root != root) { -if (WARN_ON_ONCE (root)) -up_write (&root->rwsem); - root = new_root; -down_write (&root->rwsem); - } -return root; -} - -static inline void unlock_anon_vma_root (struct anon_vma *root) -{ -if (root) -up_write (&root->rwsem); -} -static void check_anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {@@ -307,26 +281,25 @@ static void check_anon_vma_clone(struct vm_area_struct *dst,int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {struct anon_vma_chain *avc, *pavc; -struct anon_vma *root =NULL ;if (!src->anon_vma)return 0 ;check_anon_vma_clone (dst, src); +anon_vma_lock_write (src->anon_vma);list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {struct anon_vma *anon_vma;
mm/rmap.c static void check_anon_vma_clone(struct vm_area_struct *dst, · line 300
Quoted 10 lines
mm/rmap.c +2
-2avc =anon_vma_chain_alloc (GFP_NOWAIT );if (unlikely (!avc)) { -unlock_anon_vma_root (root); - root =NULL ; +anon_vma_unlock_write (src->anon_vma); avc =anon_vma_chain_alloc (GFP_KERNEL );if (!avc) goto enomem_failure; +anon_vma_lock_write (src->anon_vma);
So, we drop and then reacquire src->anon_vma->root->rwsem, expecting src->anon_vma and src->anon_vma->root to be the same. And IIUC src->vm_mm's mmap lock is what guarantees all this. If so, could you please add a clarifying comment here?
Quoted 64 lines
mm/rmap.c +9
-8} anon_vma = pavc->anon_vma; - root =lock_anon_vma_root (root, anon_vma);anon_vma_chain_link (dst, avc, anon_vma);/* @@ -343,7 +316,8 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) }if (dst->anon_vma) dst->anon_vma->num_active_vmas++; -unlock_anon_vma_root (root); + +anon_vma_unlock_write (src->anon_vma);return 0 ; enomem_failure:@@ -438,15 +412,17 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)void unlink_anon_vmas (struct vm_area_struct *vma) {struct anon_vma_chain *avc, *next; -struct anon_vma *root =NULL ; +struct anon_vma *active_anon_vma = vma->anon_vma;/* Always hold mmap lock, read-lock on unmap possibly. */ mmap_assert_locked (vma->vm_mm);/* Unfaulted is a no-op. */ -if (!vma->anon_vma) +if (!active_anon_vma)return ; +anon_vma_lock_write (active_anon_vma); +/* * Unlink each anon_vma chained to the VMA. This list is ordered * from newest to oldest, ensuring the root anon_vma gets freed last. @@ -454,7 +430,6 @@ void unlink_anon_vmas(struct vm_area_struct *vma)list_for_each_entry_safe (avc, next, &vma->anon_vma_chain, same_vma) {struct anon_vma *anon_vma = avc->anon_vma; - root =lock_anon_vma_root (root, anon_vma);anon_vma_interval_tree_remove (avc, &anon_vma->rb_root);/* @@ -470,13 +445,14 @@ void unlink_anon_vmas(struct vm_area_struct *vma)anon_vma_chain_free (avc); } - vma->anon_vma->num_active_vmas--; + active_anon_vma->num_active_vmas--;/* * vma would still be needed after unlink, and anon_vma will be prepared * when handle fault. */ vma->anon_vma =NULL ; -unlock_anon_vma_root (root); +anon_vma_unlock_write (active_anon_vma); +/* * Iterate the list once more, it now only contains empty and unlinked -- 2.52.0
Re: [PATCH 3/8] mm/rmap: remove unnecessary root lock dance in anon_vma clone, unmap
Quoted 77 lines · Suren Baghdasaryan
Quoted 75 lines · Lorenzo Stoakes
The root
anon_vmaof all anon_vma's linked to a VMA must by definition be the same - a VMA and all of its descendants/ancestors must exist in the same CoW chain.Commit
bb4aa39676f7("mm: avoid repeatedanon_vmalock/unlock sequences inanon_vma_clone()") introduced paranoid checking of the rootanon_vmaremaining the same throughout all AVC's in 2011.I think 15 years later we can safely assume that this is always the case.
Additionally, since unfaulted VMAs being cloned from or unlinked are no-op's, we can simply lock the anon_vma's associated with this rather than doing any specific dance around this.
This removes unnecessary checks and makes it clear that the root
anon_vmais shared between all anon_vma's in a given VMA'sanon_vma_chain.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/rmap.c | 48 ++++++++++++------------------------------------ 1 file changed, 12 insertions(+), 36 deletions(-)mm/rmap.c +1
-27@@ -231,32 +231,6 @@ int __anon_vma_prepare(struct vm_area_struct *vma)return -ENOMEM ; } -/* -* This is a useful helper function for locking the anon_vma root as -* we traverse the vma->anon_vma_chain, looping over anon_vma's that -* have the same vma. -* -* Such anon_vma's should have the same root, so you'd expect to see -* just a single mutex_lock for the whole traversal. -*/ -static inline struct anon_vma *lock_anon_vma_root (struct anon_vma *root,struct anon_vma *anon_vma) -{ -struct anon_vma *new_root = anon_vma->root; -if (new_root != root) { -if (WARN_ON_ONCE (root)) -up_write (&root->rwsem); - root = new_root; -down_write (&root->rwsem); - } -return root; -} - -static inline void unlock_anon_vma_root (struct anon_vma *root) -{ -if (root) -up_write (&root->rwsem); -} -static void check_anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {@@ -307,26 +281,25 @@ static void check_anon_vma_clone(struct vm_area_struct *dst,int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {struct anon_vma_chain *avc, *pavc; -struct anon_vma *root =NULL ;if (!src->anon_vma)return 0 ;check_anon_vma_clone (dst, src); +anon_vma_lock_write (src->anon_vma);list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {struct anon_vma *anon_vma;
Quoted 12 lines
mm/rmap.c static void check_anon_vma_clone(struct vm_area_struct *dst, · line 300
Quoted 9 lines
mm/rmap.c +2
-2avc =anon_vma_chain_alloc (GFP_NOWAIT );if (unlikely (!avc)) { -unlock_anon_vma_root (root); - root =NULL ; +anon_vma_unlock_write (src->anon_vma); avc =anon_vma_chain_alloc (GFP_KERNEL );if (!avc) goto enomem_failure; +anon_vma_lock_write (src->anon_vma);So, we drop and then reacquire
src->anon_vma->root->rwsem, expectingsrc->anon_vmaandsrc->anon_vma->rootto be the same. And IIUC
I mean did you read the commit message? :)
We're not expecting that, they _have_ to be the same. It simply makes no sense for them _not_ to be the same.
This is kind of the entire point of the patch.
Quoted 2 lines
src->vm_mm's mmap lock is what guarantees all this. If so, could you please add a clarifying comment here?
No that's not what guarantees it? I don't understand what you mean?
I mean in a sense, if you had a totally broken situation where you didn't take exclusive locks and could do some horribly broken racing here, then sure you might end up with something broken, but I think it's super confusing to say 'oh this lock guarantees it', well no it guarantees that you aren't completely broken, what guarantees the shared root is how anon_vma_fork() works, which is to:
- Clone.
- If not reused an anon_vma (which by recursion would also have same root) allocate new anon_vma.
- If allocated new, set root to source VMA's anon_vma, which by definition also has to be in its anon_vma_chain and have the same root (itself, if we're cloning from the ultimate parent).
But I don't think it'd be helpful to document all this, or we get into _adding_ confusion by putting _too much_ in a comment.
So I guess I'll just say,a s I do in the newly introduced clenaup_partial_anon_vmas():
/* All anon_vma's share the same root. */
Quoted 65 lines
Quoted 65 lines
mm/rmap.c +9
-8} anon_vma = pavc->anon_vma; - root =lock_anon_vma_root (root, anon_vma);anon_vma_chain_link (dst, avc, anon_vma);/* @@ -343,7 +316,8 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) }if (dst->anon_vma) dst->anon_vma->num_active_vmas++; -unlock_anon_vma_root (root); + +anon_vma_unlock_write (src->anon_vma);return 0 ; enomem_failure:@@ -438,15 +412,17 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)void unlink_anon_vmas (struct vm_area_struct *vma) {struct anon_vma_chain *avc, *next; -struct anon_vma *root =NULL ; +struct anon_vma *active_anon_vma = vma->anon_vma;/* Always hold mmap lock, read-lock on unmap possibly. */ mmap_assert_locked (vma->vm_mm);/* Unfaulted is a no-op. */ -if (!vma->anon_vma) +if (!active_anon_vma)return ; +anon_vma_lock_write (active_anon_vma); +/* * Unlink each anon_vma chained to the VMA. This list is ordered * from newest to oldest, ensuring the root anon_vma gets freed last. @@ -454,7 +430,6 @@ void unlink_anon_vmas(struct vm_area_struct *vma)list_for_each_entry_safe (avc, next, &vma->anon_vma_chain, same_vma) {struct anon_vma *anon_vma = avc->anon_vma; - root =lock_anon_vma_root (root, anon_vma);anon_vma_interval_tree_remove (avc, &anon_vma->rb_root);/* @@ -470,13 +445,14 @@ void unlink_anon_vmas(struct vm_area_struct *vma)anon_vma_chain_free (avc); } - vma->anon_vma->num_active_vmas--; + active_anon_vma->num_active_vmas--;/* * vma would still be needed after unlink, and anon_vma will be prepared * when handle fault. */ vma->anon_vma =NULL ; -unlock_anon_vma_root (root); +anon_vma_unlock_write (active_anon_vma); +/* * Iterate the list once more, it now only contains empty and unlinked -- 2.52.0
Suren Baghdasaryan
maintainerRe: [PATCH 3/8] mm/rmap: remove unnecessary root lock dance in anon_vma clone, unmap
Quoted 92 lines · Lorenzo Stoakes
Quoted 77 lines · Suren Baghdasaryan
Quoted 75 lines · Lorenzo Stoakes
The root
anon_vmaof all anon_vma's linked to a VMA must by definition be the same - a VMA and all of its descendants/ancestors must exist in the same CoW chain.Commit
bb4aa39676f7("mm: avoid repeatedanon_vmalock/unlock sequences inanon_vma_clone()") introduced paranoid checking of the rootanon_vmaremaining the same throughout all AVC's in 2011.I think 15 years later we can safely assume that this is always the case.
Additionally, since unfaulted VMAs being cloned from or unlinked are no-op's, we can simply lock the anon_vma's associated with this rather than doing any specific dance around this.
This removes unnecessary checks and makes it clear that the root
anon_vmais shared between all anon_vma's in a given VMA'sanon_vma_chain.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/rmap.c | 48 ++++++++++++------------------------------------ 1 file changed, 12 insertions(+), 36 deletions(-)mm/rmap.c +1
-27@@ -231,32 +231,6 @@ int __anon_vma_prepare(struct vm_area_struct *vma)return -ENOMEM ; } -/* -* This is a useful helper function for locking the anon_vma root as -* we traverse the vma->anon_vma_chain, looping over anon_vma's that -* have the same vma. -* -* Such anon_vma's should have the same root, so you'd expect to see -* just a single mutex_lock for the whole traversal. -*/ -static inline struct anon_vma *lock_anon_vma_root (struct anon_vma *root,struct anon_vma *anon_vma) -{ -struct anon_vma *new_root = anon_vma->root; -if (new_root != root) { -if (WARN_ON_ONCE (root)) -up_write (&root->rwsem); - root = new_root; -down_write (&root->rwsem); - } -return root; -} - -static inline void unlock_anon_vma_root (struct anon_vma *root) -{ -if (root) -up_write (&root->rwsem); -} -static void check_anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {@@ -307,26 +281,25 @@ static void check_anon_vma_clone(struct vm_area_struct *dst,int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {struct anon_vma_chain *avc, *pavc; -struct anon_vma *root =NULL ;if (!src->anon_vma)return 0 ;check_anon_vma_clone (dst, src); +anon_vma_lock_write (src->anon_vma);list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {struct anon_vma *anon_vma;Quoted 12 lines
mm/rmap.c static void check_anon_vma_clone(struct vm_area_struct *dst, · line 300
Quoted 9 lines
mm/rmap.c +2
-2avc =anon_vma_chain_alloc (GFP_NOWAIT );if (unlikely (!avc)) { -unlock_anon_vma_root (root); - root =NULL ; +anon_vma_unlock_write (src->anon_vma); avc =anon_vma_chain_alloc (GFP_KERNEL );if (!avc) goto enomem_failure; +anon_vma_lock_write (src->anon_vma);So, we drop and then reacquire
src->anon_vma->root->rwsem, expectingsrc->anon_vmaandsrc->anon_vma->rootto be the same. And IIUC
Quoted 4 lines
I mean did you read the commit message? :)
We're not expecting that, they _have_ to be the same. It simply makes no sense for them _not_ to be the same.
Sorry, maybe I chose my words badly to explain my concern. I meant that we expect those fields to still be valid between the time when we drop and re-ackquire the lock. The comment next to anon_vma.rwsem definition says "W: modification, R: walking the list". Here we are walking the list with the lock but are dropping the lock in the process. I think there needs to be an explanation why this is safe.
Quoted 22 lines
This is kind of the entire point of the patch.
Quoted 2 lines
src->vm_mm's mmap lock is what guarantees all this. If so, could you please add a clarifying comment here?
No that's not what guarantees it? I don't understand what you mean?
I mean in a sense, if you had a totally broken situation where you didn't take exclusive locks and could do some horribly broken racing here, then sure you might end up with something broken, but I think it's super confusing to say 'oh this lock guarantees it', well no it guarantees that you aren't completely broken, what guarantees the shared root is how
anon_vma_fork()works, which is to:- Clone.
- If not reused ananon_vma(which by recursion would also have same root) allocate newanon_vma.
- If allocated new, set root to source VMA'sanon_vma, which by definition also has to be in itsanon_vma_chainand have the same root (itself, if we're cloning from the ultimate parent).
Quoted 7 lines
But I don't think it'd be helpful to document all this, or we get into _adding_ confusion by putting
_toomuch_in a comment.So I guess I'll just say,a s I do in the newly introduced
clenaup_partial_anon_vmas():/* All anon_vma's share the same root. */
Yeah, my concern was not the root being different but that the list itself is stable after we drop the lock.
Quoted 66 lines
Quoted 66 lines
Quoted 66 lines
mm/rmap.c +9
-8} anon_vma = pavc->anon_vma; - root =lock_anon_vma_root (root, anon_vma);anon_vma_chain_link (dst, avc, anon_vma);/* @@ -343,7 +316,8 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) }if (dst->anon_vma) dst->anon_vma->num_active_vmas++; -unlock_anon_vma_root (root); + +anon_vma_unlock_write (src->anon_vma);return 0 ; enomem_failure:@@ -438,15 +412,17 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)void unlink_anon_vmas (struct vm_area_struct *vma) {struct anon_vma_chain *avc, *next; -struct anon_vma *root =NULL ; +struct anon_vma *active_anon_vma = vma->anon_vma;/* Always hold mmap lock, read-lock on unmap possibly. */ mmap_assert_locked (vma->vm_mm);/* Unfaulted is a no-op. */ -if (!vma->anon_vma) +if (!active_anon_vma)return ; +anon_vma_lock_write (active_anon_vma); +/* * Unlink each anon_vma chained to the VMA. This list is ordered * from newest to oldest, ensuring the root anon_vma gets freed last. @@ -454,7 +430,6 @@ void unlink_anon_vmas(struct vm_area_struct *vma)list_for_each_entry_safe (avc, next, &vma->anon_vma_chain, same_vma) {struct anon_vma *anon_vma = avc->anon_vma; - root =lock_anon_vma_root (root, anon_vma);anon_vma_interval_tree_remove (avc, &anon_vma->rb_root);/* @@ -470,13 +445,14 @@ void unlink_anon_vmas(struct vm_area_struct *vma)anon_vma_chain_free (avc); } - vma->anon_vma->num_active_vmas--; + active_anon_vma->num_active_vmas--;/* * vma would still be needed after unlink, and anon_vma will be prepared * when handle fault. */ vma->anon_vma =NULL ; -unlock_anon_vma_root (root); +anon_vma_unlock_write (active_anon_vma); +/* * Iterate the list once more, it now only contains empty and unlinked -- 2.52.0
Re: [PATCH 3/8] mm/rmap: remove unnecessary root lock dance in anon_vma clone, unmap
Quoted 99 lines · Suren Baghdasaryan
Quoted 92 lines · Lorenzo Stoakes
Quoted 77 lines · Suren Baghdasaryan
Quoted 75 lines · Lorenzo Stoakes
The root
anon_vmaof all anon_vma's linked to a VMA must by definition be the same - a VMA and all of its descendants/ancestors must exist in the same CoW chain.Commit
bb4aa39676f7("mm: avoid repeatedanon_vmalock/unlock sequences inanon_vma_clone()") introduced paranoid checking of the rootanon_vmaremaining the same throughout all AVC's in 2011.I think 15 years later we can safely assume that this is always the case.
Additionally, since unfaulted VMAs being cloned from or unlinked are no-op's, we can simply lock the anon_vma's associated with this rather than doing any specific dance around this.
This removes unnecessary checks and makes it clear that the root
anon_vmais shared between all anon_vma's in a given VMA'sanon_vma_chain.Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/rmap.c | 48 ++++++++++++------------------------------------ 1 file changed, 12 insertions(+), 36 deletions(-)mm/rmap.c +1
-27@@ -231,32 +231,6 @@ int __anon_vma_prepare(struct vm_area_struct *vma)return -ENOMEM ; } -/* -* This is a useful helper function for locking the anon_vma root as -* we traverse the vma->anon_vma_chain, looping over anon_vma's that -* have the same vma. -* -* Such anon_vma's should have the same root, so you'd expect to see -* just a single mutex_lock for the whole traversal. -*/ -static inline struct anon_vma *lock_anon_vma_root (struct anon_vma *root,struct anon_vma *anon_vma) -{ -struct anon_vma *new_root = anon_vma->root; -if (new_root != root) { -if (WARN_ON_ONCE (root)) -up_write (&root->rwsem); - root = new_root; -down_write (&root->rwsem); - } -return root; -} - -static inline void unlock_anon_vma_root (struct anon_vma *root) -{ -if (root) -up_write (&root->rwsem); -} -static void check_anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {@@ -307,26 +281,25 @@ static void check_anon_vma_clone(struct vm_area_struct *dst,int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) {struct anon_vma_chain *avc, *pavc; -struct anon_vma *root =NULL ;if (!src->anon_vma)return 0 ;check_anon_vma_clone (dst, src); +anon_vma_lock_write (src->anon_vma);list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) {struct anon_vma *anon_vma;Quoted 12 lines
mm/rmap.c static void check_anon_vma_clone(struct vm_area_struct *dst, · line 300
Quoted 9 lines
mm/rmap.c +2
-2avc =anon_vma_chain_alloc (GFP_NOWAIT );if (unlikely (!avc)) { -unlock_anon_vma_root (root); - root =NULL ; +anon_vma_unlock_write (src->anon_vma); avc =anon_vma_chain_alloc (GFP_KERNEL );if (!avc) goto enomem_failure; +anon_vma_lock_write (src->anon_vma);So, we drop and then reacquire
src->anon_vma->root->rwsem, expectingsrc->anon_vmaandsrc->anon_vma->rootto be the same. And IIUCQuoted 4 lines
I mean did you read the commit message? :)
We're not expecting that, they _have_ to be the same. It simply makes no sense for them _not_ to be the same.
Quoted 6 lines
Sorry, maybe I chose my words badly to explain my concern. I meant that we expect those fields to still be valid between the time when we drop and re-ackquire the lock. The comment next to anon_vma.rwsem definition says "W: modification, R: walking the list". Here we are walking the list with the lock but are dropping the lock in the process. I think there needs to be an explanation why this is safe.
This already happened though? And yes it's sketchy.
I don't think this is necessary as later I change this anyway, so we'd just be adding an explanation I'd have to delete later.
I already provide explanation as to the locking when I go ahead and change the scope of the anon_vma rmap lock elsewhere so this general 'explaining lock scope' pattern is happening in the final result of the series.
Quoted 27 lines
Quoted 27 lines
This is kind of the entire point of the patch.
Quoted 2 lines
src->vm_mm's mmap lock is what guarantees all this. If so, could you please add a clarifying comment here?
No that's not what guarantees it? I don't understand what you mean?
I mean in a sense, if you had a totally broken situation where you didn't take exclusive locks and could do some horribly broken racing here, then sure you might end up with something broken, but I think it's super confusing to say 'oh this lock guarantees it', well no it guarantees that you aren't completely broken, what guarantees the shared root is how
anon_vma_fork()works, which is to:- Clone.
- If not reused ananon_vma(which by recursion would also have same root) allocate newanon_vma.
- If allocated new, set root to source VMA'sanon_vma, which by definition also has to be in itsanon_vma_chainand have the same root (itself, if we're cloning from the ultimate parent).But I don't think it'd be helpful to document all this, or we get into _adding_ confusion by putting
_toomuch_in a comment.
Quoted 7 lines
Quoted 4 lines
So I guess I'll just say,a s I do in the newly introduced
clenaup_partial_anon_vmas():/* All anon_vma's share the same root. */Yeah, my concern was not the root being different but that the list itself is stable after we drop the lock.
Again, I'm going to end up deleting any explanation that I add in a later patch where I extensively change this, which seems like it'd not be a useful thing to do in the series.
So I think we should leave it as-is.
Thanks, Lorenzo
PATCH 4/8 mm/rmap: remove anon_vma_merge() function
In reply to Lorenzo StoakesThis function is confusing, we already have the concept of anon_vma merge to adjacent VMA's anon_vma's to increase probability of anon_vma compatibility and therefore VMA merge (see is_mergeable_anon_vma() etc.), as well as anon_vma reuse, along side the usual VMA merge logic.
We can remove the anon_vma check as it is redundant - a merge would not have been permitted with removal if the anon_vma's were not the same (and in the case of an unfaulted/faulted merge, we would have already set the unfaulted VMA's anon_vma to vp->remove->anon_vma in dup_anon_vma()).
Avoid overloading this term when we're very simply unlinking anon_vma state from a removed VMA upon merge.
--- include/linux/rmap.h | 7 ------- mm/vma.c | 2 +- tools/testing/vma/vma_internal.h | 5 ----- 3 files changed, 1 insertion(+), 13 deletions(-)
include/linux/rmap.h +0 -7
@@ -165,13 +165,6 @@ static inline int anon_vma_prepare(struct vm_area_struct *vma)return __anon_vma_prepare (vma); } -static inline void anon_vma_merge (struct vm_area_struct *vma, -struct vm_area_struct *next) -{ -VM_BUG_ON_VMA (vma->anon_vma != next->anon_vma, vma); -unlink_anon_vmas (next); -} -struct anon_vma *folio_get_anon_vma (const struct folio *folio);#ifdef CONFIG_MM_ID
mm/vma.c +1 -1
@@ -376,7 +376,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi,fput (vp->file); }if (vp->remove->anon_vma) -anon_vma_merge (vp->vma, vp->remove); +unlink_anon_vmas (vp->remove); mm->map_count--;mpol_put (vma_policy (vp->remove));if (!vp->remove2)
tools/testing/vma/vma_internal.h +0 -5
@@ -1265,11 +1265,6 @@ static inline void i_mmap_unlock_write(struct address_space *mapping) { } -static inline void anon_vma_merge (struct vm_area_struct *vma, -struct vm_area_struct *next) -{ -} - static inline int userfaultfd_unmap_prep(struct vm_area_struct *vma, unsigned long start, unsigned long end,
--
2.52.0
Re: [PATCH 4/8] mm/rmap: remove anon_vma_merge() function
Quoted 11 lines · Lorenzo Stoakes
This function is confusing, we already have the concept of
anon_vmamerge to adjacent VMA's anon_vma's to increase probability ofanon_vmacompatibility and therefore VMA merge (seeis_mergeable_anon_vma()etc.), as well asanon_vmareuse, along side the usual VMA merge logic.We can remove the
anon_vmacheck as it is redundant - a merge would not have been permitted with removal if the anon_vma's were not the same (and in the case of an unfaulted/faulted merge, we would have already set the unfaulted VMA'sanon_vmatovp->remove->anon_vmaindup_anon_vma()).
Quoted 4 lines
Avoid overloading this term when we're very simply unlinking
anon_vmastate from a removed VMA upon merge.Signed-off-by Lorenzo Stoakes <[email protected]>
Quoted 55 lines
--- include/linux/rmap.h | 7 ------- mm/vma.c | 2 +- tools/testing/vma/vma_internal.h | 5 ----- 3 files changed, 1 insertion(+), 13 deletions(-)include/linux/rmap.h +0
-7@@ -165,13 +165,6 @@ static inline int anon_vma_prepare(struct vm_area_struct *vma)return __anon_vma_prepare (vma); } -static inline void anon_vma_merge (struct vm_area_struct *vma, -struct vm_area_struct *next) -{ -VM_BUG_ON_VMA (vma->anon_vma != next->anon_vma, vma); -unlink_anon_vmas (next); -} -struct anon_vma *folio_get_anon_vma (const struct folio *folio);#ifdef CONFIG_MM_ID mm/vma.c +1
-1@@ -376,7 +376,7 @@ static void vma_complete(struct vma_prepare *vp, struct vma_iterator *vmi,fput (vp->file); }if (vp->remove->anon_vma) -anon_vma_merge (vp->vma, vp->remove); +unlink_anon_vmas (vp->remove); mm->map_count--;mpol_put (vma_policy (vp->remove));if (!vp->remove2)tools/testing/vma/vma_internal.h +0
-6@@ -1265,11 +1265,6 @@ static inline void i_mmap_unlock_write(struct address_space *mapping) { } -static inline void anon_vma_merge (struct vm_area_struct *vma, -struct vm_area_struct *next) -{ -} - static inline int userfaultfd_unmap_prep(struct vm_area_struct *vma, unsigned long start, unsigned long end, --2.52.0
Re: [PATCH 4/8] mm/rmap: remove anon_vma_merge() function
Quoted 13 lines · Suren Baghdasaryan
Quoted 11 lines · Lorenzo Stoakes
This function is confusing, we already have the concept of
anon_vmamerge to adjacent VMA's anon_vma's to increase probability ofanon_vmacompatibility and therefore VMA merge (seeis_mergeable_anon_vma()etc.), as well asanon_vmareuse, along side the usual VMA merge logic.We can remove the
anon_vmacheck as it is redundant - a merge would not have been permitted with removal if the anon_vma's were not the same (and in the case of an unfaulted/faulted merge, we would have already set the unfaulted VMA'sanon_vmatovp->remove->anon_vmaindup_anon_vma()).
Quoted 6 lines
Quoted 4 lines
Avoid overloading this term when we're very simply unlinking
anon_vmastate from a removed VMA upon merge.Signed-off-by Lorenzo Stoakes <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>
Thanks!
PATCH 5/8 mm/rmap: make anon_vma functions internal
In reply to Lorenzo StoakesThe bulk of the anon_vma operations are only used by mm, so formalise this by putting the function prototypes and inlines in mm/internal.h. This allows us to make changes without having to worry about the rest of the kernel.
--- include/linux/rmap.h | 60 -------------------------------------------- mm/internal.h | 58 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 60 deletions(-)
include/linux/rmap.h +0 -60
@@ -104,68 +104,8 @@ enum ttu_flags { };#ifdef CONFIG_MMU -static inline void get_anon_vma (struct anon_vma *anon_vma) -{ -atomic_inc (&anon_vma->refcount); -} - -void __put_anon_vma (struct anon_vma *anon_vma); - -static inline void put_anon_vma (struct anon_vma *anon_vma) -{ -if (atomic_dec_and_test (&anon_vma->refcount)) -__put_anon_vma (anon_vma); -} - -static inline void anon_vma_lock_write (struct anon_vma *anon_vma) -{ -down_write (&anon_vma->root->rwsem); -} -static inline int anon_vma_trylock_write (struct anon_vma *anon_vma) -{ -return down_write_trylock (&anon_vma->root->rwsem); -} - -static inline void anon_vma_unlock_write (struct anon_vma *anon_vma) -{ -up_write (&anon_vma->root->rwsem); -} - -static inline void anon_vma_lock_read (struct anon_vma *anon_vma) -{ -down_read (&anon_vma->root->rwsem); -} - -static inline int anon_vma_trylock_read (struct anon_vma *anon_vma) -{ -return down_read_trylock (&anon_vma->root->rwsem); -} - -static inline void anon_vma_unlock_read (struct anon_vma *anon_vma) -{ -up_read (&anon_vma->root->rwsem); -} - - -/* -* anon_vma helper functions. -*/ void anon_vma_init (void );/* create anon_vma_cachep */ -int __anon_vma_prepare (struct vm_area_struct *); -void unlink_anon_vmas (struct vm_area_struct *); -int anon_vma_clone (struct vm_area_struct *,struct vm_area_struct *); -int anon_vma_fork (struct vm_area_struct *,struct vm_area_struct *); - -static inline int anon_vma_prepare (struct vm_area_struct *vma) -{ -if (likely (vma->anon_vma)) -return 0 ; - -return __anon_vma_prepare (vma); -} - -struct anon_vma *folio_get_anon_vma (const struct folio *folio);#ifdef CONFIG_MM_ID static __always_inline void folio_lock_large_mapcount(struct folio *folio)
mm/internal.h +58 -0
64 lines not on this page · raw message
Signature · 1 line
2.52.0
Re: [PATCH 5/8] mm/rmap: make anon_vma functions internal
Quoted 7 lines · Lorenzo Stoakes
The bulk of the
anon_vmaoperations are only used by mm, so formalise this by putting the function prototypes and inlines inmm/internal.h. This allows us to make changes without having to worry about the rest of the kernel.Signed-off-by Lorenzo Stoakes <[email protected]>
Quoted 150 lines
--- include/linux/rmap.h | 60 -------------------------------------------- mm/internal.h | 58 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+), 60 deletions(-)include/linux/rmap.h +0
-60@@ -104,68 +104,8 @@ enum ttu_flags { };#ifdef CONFIG_MMU -static inline void get_anon_vma (struct anon_vma *anon_vma) -{ -atomic_inc (&anon_vma->refcount); -} - -void __put_anon_vma (struct anon_vma *anon_vma); - -static inline void put_anon_vma (struct anon_vma *anon_vma) -{ -if (atomic_dec_and_test (&anon_vma->refcount)) -__put_anon_vma (anon_vma); -} - -static inline void anon_vma_lock_write (struct anon_vma *anon_vma) -{ -down_write (&anon_vma->root->rwsem); -} -static inline int anon_vma_trylock_write (struct anon_vma *anon_vma) -{ -return down_write_trylock (&anon_vma->root->rwsem); -} - -static inline void anon_vma_unlock_write (struct anon_vma *anon_vma) -{ -up_write (&anon_vma->root->rwsem); -} - -static inline void anon_vma_lock_read (struct anon_vma *anon_vma) -{ -down_read (&anon_vma->root->rwsem); -} - -static inline int anon_vma_trylock_read (struct anon_vma *anon_vma) -{ -return down_read_trylock (&anon_vma->root->rwsem); -} - -static inline void anon_vma_unlock_read (struct anon_vma *anon_vma) -{ -up_read (&anon_vma->root->rwsem); -} - - -/* -* anon_vma helper functions. -*/ void anon_vma_init (void );/* create anon_vma_cachep */ -int __anon_vma_prepare (struct vm_area_struct *); -void unlink_anon_vmas (struct vm_area_struct *); -int anon_vma_clone (struct vm_area_struct *,struct vm_area_struct *); -int anon_vma_fork (struct vm_area_struct *,struct vm_area_struct *); - -static inline int anon_vma_prepare (struct vm_area_struct *vma) -{ -if (likely (vma->anon_vma)) -return 0 ; - -return __anon_vma_prepare (vma); -} - -struct anon_vma *folio_get_anon_vma (const struct folio *folio);#ifdef CONFIG_MM_ID static __always_inline void folio_lock_large_mapcount(struct folio *folio)mm/internal.h +58
-1@@ -199,6 +199,64 @@ static inline void vma_close(struct vm_area_struct *vma)#ifdef CONFIG_MMU +static inline void get_anon_vma (struct anon_vma *anon_vma) +{ +atomic_inc (&anon_vma->refcount); +} + +void __put_anon_vma (struct anon_vma *anon_vma); + +static inline void put_anon_vma (struct anon_vma *anon_vma) +{ +if (atomic_dec_and_test (&anon_vma->refcount)) +__put_anon_vma (anon_vma); +} + +static inline void anon_vma_lock_write (struct anon_vma *anon_vma) +{ +down_write (&anon_vma->root->rwsem); +} + +static inline int anon_vma_trylock_write (struct anon_vma *anon_vma) +{ +return down_write_trylock (&anon_vma->root->rwsem); +} + +static inline void anon_vma_unlock_write (struct anon_vma *anon_vma) +{ +up_write (&anon_vma->root->rwsem); +} + +static inline void anon_vma_lock_read (struct anon_vma *anon_vma) +{ +down_read (&anon_vma->root->rwsem); +} + +static inline int anon_vma_trylock_read (struct anon_vma *anon_vma) +{ +return down_read_trylock (&anon_vma->root->rwsem); +} + +static inline void anon_vma_unlock_read (struct anon_vma *anon_vma) +{ +up_read (&anon_vma->root->rwsem); +} + +struct anon_vma *folio_get_anon_vma (const struct folio *folio); + +int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src); +int anon_vma_fork (struct vm_area_struct *vma,struct vm_area_struct *pvma); +int __anon_vma_prepare (struct vm_area_struct *vma); +void unlink_anon_vmas (struct vm_area_struct *vma); + +static inline int anon_vma_prepare (struct vm_area_struct *vma) +{ +if (likely (vma->anon_vma)) +return 0 ; + +return __anon_vma_prepare (vma); +} +/* Flags for folio_pte_batch(). */ typedef int __bitwise fpb_t ; --2.52.0
Re: [PATCH 5/8] mm/rmap: make anon_vma functions internal
Quoted 11 lines · Suren Baghdasaryan
Quoted 7 lines · Lorenzo Stoakes
The bulk of the
anon_vmaoperations are only used by mm, so formalise this by putting the function prototypes and inlines inmm/internal.h. This allows us to make changes without having to worry about the rest of the kernel.Signed-off-by Lorenzo Stoakes <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>
Thanks!
PATCH 6/8 mm/mmap_lock: add vma_is_attached() helper
In reply to Lorenzo StoakesThis makes it easy to explicitly check for VMA detachment, which is useful for things like asserts.
Note that we intentionally do not allow this function to be available should CONFIG_PER_VMA_LOCK be set - this is because vma_assert_attached() and vma_assert_detached() are no-ops if !CONFIG_PER_VMA_LOCK, so there is no correct state for vma_is_attached() to be in if this configuration option is not specified.
Therefore users elsewhere must invoke this function only after checking for CONFIG_PER_VMA_LOCK.
We rework the assert functions to utilise this.
--- include/linux/mmap_lock.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
include/linux/mmap_lock.h +7 -2
@@ -251,6 +251,11 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) !__is_vma_write_locked(vma, &mm_lock_seq), vma); } +static inline bool vma_is_attached (struct vm_area_struct *vma) +{ +return refcount_read (&vma->vm_refcnt); +} +/* * WARNING: to avoid racing with vma_mark_attached()/vma_mark_detached(), these * assertions should be made either under mmap_write_lock or when the object
@@ -258,12 +263,12 @@ static inline void vma_assert_locked(struct vm_area_struct *vma)*/ static inline void vma_assert_attached (struct vm_area_struct *vma) { -WARN_ON_ONCE (!refcount_read (&vma->vm_refcnt)); +WARN_ON_ONCE (!vma_is_attached (vma)); }static inline void vma_assert_detached (struct vm_area_struct *vma) { -WARN_ON_ONCE (refcount_read (&vma->vm_refcnt)); +WARN_ON_ONCE (vma_is_attached (vma)); }static inline void vma_mark_attached (struct vm_area_struct *vma)
--
2.52.0
Re: [PATCH 6/8] mm/mmap_lock: add vma_is_attached() helper
Quoted 10 lines · Lorenzo Stoakes
This makes it easy to explicitly check for VMA detachment, which is useful for things like asserts.
Note that we intentionally do not allow this function to be available should
CONFIG_PER_VMA_LOCKbe set - this is becausevma_assert_attached()andvma_assert_detached()are no-ops if !CONFIG_PER_VMA_LOCK, so there is no correct state forvma_is_attached()to be in if this configuration option is not specified.
Quoted 4 lines
Therefore users elsewhere must invoke this function only after checking for
CONFIG_PER_VMA_LOCK.We rework the assert functions to utilise this.
Thank you! This nicely documents vm_refcnt attached state. Another step in this direction is adding:
static inline bool vma_is_read_locked(struct vm_area_struct *vma)
{
return refcount_read(&vma->vm_refcnt) > 1;
}and changing vma_assert_locked() to use it.
But I can do that in a separate patch, so LGTM.
Quoted 2 lines
Signed-off-by Lorenzo Stoakes <[email protected]>
Quoted 37 lines
--- include/linux/mmap_lock.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)include/linux/mmap_lock.h +7
-3@@ -251,6 +251,11 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) !__is_vma_write_locked(vma, &mm_lock_seq), vma); } +static inline bool vma_is_attached (struct vm_area_struct *vma) +{ +return refcount_read (&vma->vm_refcnt); +} +/* * WARNING: to avoid racing with vma_mark_attached()/vma_mark_detached(), these * assertions should be made either under mmap_write_lock or when the object @@ -258,12 +263,12 @@ static inline void vma_assert_locked(struct vm_area_struct *vma)*/ static inline void vma_assert_attached (struct vm_area_struct *vma) { -WARN_ON_ONCE (!refcount_read (&vma->vm_refcnt)); +WARN_ON_ONCE (!vma_is_attached (vma)); }static inline void vma_assert_detached (struct vm_area_struct *vma) { -WARN_ON_ONCE (refcount_read (&vma->vm_refcnt)); +WARN_ON_ONCE (vma_is_attached (vma)); }static inline void vma_mark_attached (struct vm_area_struct *vma) --2.52.0
Re: [PATCH 6/8] mm/mmap_lock: add vma_is_attached() helper
Quoted 12 lines · Suren Baghdasaryan
Quoted 10 lines · Lorenzo Stoakes
This makes it easy to explicitly check for VMA detachment, which is useful for things like asserts.
Note that we intentionally do not allow this function to be available should
CONFIG_PER_VMA_LOCKbe set - this is becausevma_assert_attached()andvma_assert_detached()are no-ops if !CONFIG_PER_VMA_LOCK, so there is no correct state forvma_is_attached()to be in if this configuration option is not specified.
Quoted 6 lines
Quoted 4 lines
Therefore users elsewhere must invoke this function only after checking for
CONFIG_PER_VMA_LOCK.We rework the assert functions to utilise this.
Thank you! This nicely documents
vm_refcntattached state. Another
You're welcome! :)
Quoted 9 lines
step in this direction is adding:
static inline bool
vma_is_read_locked(struct vm_area_struct *vma){ return refcount_read(&vma->vm_refcnt) > 1; }and changing
vma_assert_locked()to use it.
But I can do that in a separate patch, so LGTM.
Right, yeah makes sense separately I think as this change was to allow us to use this for an assert :)
Quoted 5 lines
Quoted 3 lines
Signed-off-by Lorenzo Stoakes <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>
Thanks!
Quoted 38 lines
Quoted 38 lines
--- include/linux/mmap_lock.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)include/linux/mmap_lock.h +7
-3@@ -251,6 +251,11 @@ static inline void vma_assert_locked(struct vm_area_struct *vma) !__is_vma_write_locked(vma, &mm_lock_seq), vma); } +static inline bool vma_is_attached (struct vm_area_struct *vma) +{ +return refcount_read (&vma->vm_refcnt); +} +/* * WARNING: to avoid racing with vma_mark_attached()/vma_mark_detached(), these * assertions should be made either under mmap_write_lock or when the object @@ -258,12 +263,12 @@ static inline void vma_assert_locked(struct vm_area_struct *vma)*/ static inline void vma_assert_attached (struct vm_area_struct *vma) { -WARN_ON_ONCE (!refcount_read (&vma->vm_refcnt)); +WARN_ON_ONCE (!vma_is_attached (vma)); }static inline void vma_assert_detached (struct vm_area_struct *vma) { -WARN_ON_ONCE (refcount_read (&vma->vm_refcnt)); +WARN_ON_ONCE (vma_is_attached (vma)); }static inline void vma_mark_attached (struct vm_area_struct *vma) --2.52.0
PATCH 7/8 mm/rmap: allocate anon_vma_chain objects unlocked when possible
In reply to Lorenzo StoakesThere is no reason to allocate the anon_vma_chain under the anon_vma write lock when cloning - we can in fact assign these to the destination VMA safely as we hold the exclusive mmap lock and therefore preclude anybody else accessing these fields.
We only need take the anon_vma write lock when we link rbtree edges from the anon_vma to the newly established AVCs.
This also allows us to eliminate the weird GFP_NOWAIT, GFP_KERNEL dance introduced in commit dd34739c03f2 ("mm: avoid anon_vma_chain allocation under anon_vma lock"), further simplifying this logic.
This should reduce lock anon_vma contention, and clarifies exactly where the anon_vma lock is required.
We cannot adjust __anon_vma_prepare() in the same way as this is only protected by VMA read lock, so we have to perform the allocation here under the anon_vma write lock and page_table_lock (to protect against racing threads), and we wish to retain the lock ordering.
--- mm/rmap.c | 49 +++++++++++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 20 deletions(-)
mm/rmap.c +29 -20
@@ -146,14 +146,13 @@ static void anon_vma_chain_free(struct anon_vma_chain *anon_vma_chain)kmem_cache_free (anon_vma_chain_cachep, anon_vma_chain); } -static void anon_vma_chain_link (struct vm_area_struct *vma, -struct anon_vma_chain *avc, -struct anon_vma *anon_vma) +static void anon_vma_chain_assign (struct vm_area_struct *vma, +struct anon_vma_chain *avc, +struct anon_vma *anon_vma) { avc->vma = vma; avc->anon_vma = anon_vma;list_add (&avc->same_vma, &vma->anon_vma_chain); -anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); }/**
@@ -210,7 +209,8 @@ int __anon_vma_prepare(struct vm_area_struct *vma)spin_lock (&mm->page_table_lock);if (likely (!vma->anon_vma)) { vma->anon_vma = anon_vma; -anon_vma_chain_link (vma, avc, anon_vma); +anon_vma_chain_assign (vma, avc, anon_vma); +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); anon_vma->num_active_vmas++; allocated =NULL ; avc =NULL ;
@@ -287,20 +287,28 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)check_anon_vma_clone (dst, src); +/* +* Allocate AVCs. We don't need an anon_vma lock for this as we +* are not updating the anon_vma rbtree nor are we changing +* anon_vma statistics. +* +* We hold the mmap write lock so there's no possibliity of +* the unlinked AVC's being observed yet. +*/ +list_for_each_entry (pavc, &src->anon_vma_chain, same_vma) { + avc =anon_vma_chain_alloc (GFP_KERNEL ); +if (!avc) + goto enomem_failure; + +anon_vma_chain_assign (dst, avc, pavc->anon_vma); + } + +/* Now link the anon_vma's back to the newly inserted AVCs. */ anon_vma_lock_write (src->anon_vma); -list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) { -struct anon_vma *anon_vma; - - avc =anon_vma_chain_alloc (GFP_NOWAIT ); -if (unlikely (!avc)) { -anon_vma_unlock_write (src->anon_vma); - avc =anon_vma_chain_alloc (GFP_KERNEL ); -if (!avc) - goto enomem_failure; -anon_vma_lock_write (src->anon_vma); - } - anon_vma = pavc->anon_vma; -anon_vma_chain_link (dst, avc, anon_vma); +list_for_each_entry_reverse (avc, &dst->anon_vma_chain, same_vma) { +struct anon_vma *anon_vma = avc->anon_vma; + +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root);/* * Reuse existing anon_vma if it has no vma and only one
@@ -316,7 +324,6 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) }if (dst->anon_vma) dst->anon_vma->num_active_vmas++; -anon_vma_unlock_write (src->anon_vma);return 0 ;
@@ -385,8 +392,10 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)get_anon_vma (anon_vma->root);/* Mark this anon_vma as the one where our new (COWed) pages go. */ vma->anon_vma = anon_vma; +anon_vma_chain_assign (vma, avc, anon_vma); +/* Now let rmap see it. */ anon_vma_lock_write (anon_vma); -anon_vma_chain_link (vma, avc, anon_vma); +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); anon_vma->parent->num_children++;anon_vma_unlock_write (anon_vma);
Signature · 1 line
2.52.0
Re: [PATCH 7/8] mm/rmap: allocate anon_vma_chain objects unlocked when possible
Quoted 16 lines · Lorenzo Stoakes
There is no reason to allocate the
anon_vma_chainunder theanon_vmawrite lock when cloning - we can in fact assign these to the destination VMA safely as we hold the exclusive mmap lock and therefore preclude anybody else accessing these fields.We only need take the
anon_vmawrite lock when we link rbtree edges from theanon_vmato the newly established AVCs.This also allows us to eliminate the weird
GFP_NOWAIT,GFP_KERNELdance introduced in commitdd34739c03f2("mm: avoidanon_vma_chainallocation underanon_vmalock"), further simplifying this logic.This should reduce lock
anon_vmacontention, and clarifies exactly where theanon_vmalock is required.
Quoted 6 lines
We cannot adjust
__anon_vma_prepare()in the same way as this is only protected by VMA read lock, so we have to perform the allocation here under theanon_vmawrite lock andpage_table_lock(to protect against racing threads), and we wish to retain the lock ordering.Signed-off-by Lorenzo Stoakes <[email protected]>
One nit but otherwise nice cleanup.
Quoted 37 lines
--- mm/rmap.c | 49 +++++++++++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 20 deletions(-)mm/rmap.c +5
-5@@ -146,14 +146,13 @@ static void anon_vma_chain_free(struct anon_vma_chain *anon_vma_chain)kmem_cache_free (anon_vma_chain_cachep, anon_vma_chain); } -static void anon_vma_chain_link (struct vm_area_struct *vma, -struct anon_vma_chain *avc, -struct anon_vma *anon_vma) +static void anon_vma_chain_assign (struct vm_area_struct *vma, +struct anon_vma_chain *avc, +struct anon_vma *anon_vma) { avc->vma = vma; avc->anon_vma = anon_vma;list_add (&avc->same_vma, &vma->anon_vma_chain); -anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); }/** @@ -210,7 +209,8 @@ int __anon_vma_prepare(struct vm_area_struct *vma)spin_lock (&mm->page_table_lock);if (likely (!vma->anon_vma)) { vma->anon_vma = anon_vma; -anon_vma_chain_link (vma, avc, anon_vma); +anon_vma_chain_assign (vma, avc, anon_vma); +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); anon_vma->num_active_vmas++; allocated =NULL ; avc =NULL ;
mm/rmap.c int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) · line 295
Quoted 9 lines
mm/rmap.c +6
-0@@ -287,20 +287,28 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)check_anon_vma_clone (dst, src); +/* +* Allocate AVCs. We don't need an anon_vma lock for this as we +* are not updating the anon_vma rbtree nor are we changing +* anon_vma statistics. +* +* We hold the mmap write lock so there's no possibliity of
To be more specific, we are holding src's mmap write lock. I think clarifying that will avoid any confusion.
Quoted 55 lines
mm/rmap.c +18
-16+* the unlinked AVC's being observed yet. +*/ +list_for_each_entry (pavc, &src->anon_vma_chain, same_vma) { + avc =anon_vma_chain_alloc (GFP_KERNEL ); +if (!avc) + goto enomem_failure; + +anon_vma_chain_assign (dst, avc, pavc->anon_vma); + } + +/* Now link the anon_vma's back to the newly inserted AVCs. */ anon_vma_lock_write (src->anon_vma); -list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) { -struct anon_vma *anon_vma; - - avc =anon_vma_chain_alloc (GFP_NOWAIT ); -if (unlikely (!avc)) { -anon_vma_unlock_write (src->anon_vma); - avc =anon_vma_chain_alloc (GFP_KERNEL ); -if (!avc) - goto enomem_failure; -anon_vma_lock_write (src->anon_vma); - } - anon_vma = pavc->anon_vma; -anon_vma_chain_link (dst, avc, anon_vma); +list_for_each_entry_reverse (avc, &dst->anon_vma_chain, same_vma) { +struct anon_vma *anon_vma = avc->anon_vma; + +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root);/* * Reuse existing anon_vma if it has no vma and only one @@ -316,7 +324,6 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) }if (dst->anon_vma) dst->anon_vma->num_active_vmas++; -anon_vma_unlock_write (src->anon_vma);return 0 ;@@ -385,8 +392,10 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)get_anon_vma (anon_vma->root);/* Mark this anon_vma as the one where our new (COWed) pages go. */ vma->anon_vma = anon_vma; +anon_vma_chain_assign (vma, avc, anon_vma); +/* Now let rmap see it. */ anon_vma_lock_write (anon_vma); -anon_vma_chain_link (vma, avc, anon_vma); +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); anon_vma->parent->num_children++;anon_vma_unlock_write (anon_vma); --2.52.0
Re: [PATCH 7/8] mm/rmap: allocate anon_vma_chain objects unlocked when possible
Quoted 23 lines · Suren Baghdasaryan
Quoted 21 lines · Lorenzo Stoakes
There is no reason to allocate the
anon_vma_chainunder theanon_vmawrite lock when cloning - we can in fact assign these to the destination VMA safely as we hold the exclusive mmap lock and therefore preclude anybody else accessing these fields.We only need take the
anon_vmawrite lock when we link rbtree edges from theanon_vmato the newly established AVCs.This also allows us to eliminate the weird
GFP_NOWAIT,GFP_KERNELdance introduced in commitdd34739c03f2("mm: avoidanon_vma_chainallocation underanon_vmalock"), further simplifying this logic.This should reduce lock
anon_vmacontention, and clarifies exactly where theanon_vmalock is required.We cannot adjust
__anon_vma_prepare()in the same way as this is only protected by VMA read lock, so we have to perform the allocation here under theanon_vmawrite lock andpage_table_lock(to protect against racing threads), and we wish to retain the lock ordering.
Quoted 5 lines
Quoted 1 line
Signed-off-by Lorenzo Stoakes <[email protected]>One nit but otherwise nice cleanup.
Reviewed-by Suren Baghdasaryan <[email protected]>
Thanks!
One nice thing with the separate cleanup_partial_anon_vmas()'s function introduced as part of this review (thanks for the good spot!) is we can now simplify this _even further_ since we don't even insert anything into the interval tree at the point of allocation, and so freeing is just a case of freeing up AVC's.
Quoted 41 lines
Quoted 41 lines
--- mm/rmap.c | 49 +++++++++++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 20 deletions(-)mm/rmap.c +5
-5@@ -146,14 +146,13 @@ static void anon_vma_chain_free(struct anon_vma_chain *anon_vma_chain)kmem_cache_free (anon_vma_chain_cachep, anon_vma_chain); } -static void anon_vma_chain_link (struct vm_area_struct *vma, -struct anon_vma_chain *avc, -struct anon_vma *anon_vma) +static void anon_vma_chain_assign (struct vm_area_struct *vma, +struct anon_vma_chain *avc, +struct anon_vma *anon_vma) { avc->vma = vma; avc->anon_vma = anon_vma;list_add (&avc->same_vma, &vma->anon_vma_chain); -anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); }/** @@ -210,7 +209,8 @@ int __anon_vma_prepare(struct vm_area_struct *vma)spin_lock (&mm->page_table_lock);if (likely (!vma->anon_vma)) { vma->anon_vma = anon_vma; -anon_vma_chain_link (vma, avc, anon_vma); +anon_vma_chain_assign (vma, avc, anon_vma); +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); anon_vma->num_active_vmas++; allocated =NULL ; avc =NULL ;@@ -287,20 +287,28 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)check_anon_vma_clone (dst, src);
Quoted 9 lines
mm/rmap.c int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) · line 295
Quoted 6 lines
mm/rmap.c +6
-0+/* +* Allocate AVCs. We don't need an anon_vma lock for this as we +* are not updating the anon_vma rbtree nor are we changing +* anon_vma statistics. +* +* We hold the mmap write lock so there's no possibliity of To be more specific, we are holding src's mmap write lock. I think clarifying that will avoid any confusion.
Well, it's the same mm for both right? :) and actually the observations would be made around dst no? As that's where the unlinked AVC's are being established.
I think more clear is 'We hold the exclusive mmap write lock' just to highlight that it excludes anybody else from accessing these fields in the VMA.
Quoted 56 lines
Quoted 56 lines
mm/rmap.c +18
-16+* the unlinked AVC's being observed yet. +*/ +list_for_each_entry (pavc, &src->anon_vma_chain, same_vma) { + avc =anon_vma_chain_alloc (GFP_KERNEL ); +if (!avc) + goto enomem_failure; + +anon_vma_chain_assign (dst, avc, pavc->anon_vma); + } + +/* Now link the anon_vma's back to the newly inserted AVCs. */ anon_vma_lock_write (src->anon_vma); -list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) { -struct anon_vma *anon_vma; - - avc =anon_vma_chain_alloc (GFP_NOWAIT ); -if (unlikely (!avc)) { -anon_vma_unlock_write (src->anon_vma); - avc =anon_vma_chain_alloc (GFP_KERNEL ); -if (!avc) - goto enomem_failure; -anon_vma_lock_write (src->anon_vma); - } - anon_vma = pavc->anon_vma; -anon_vma_chain_link (dst, avc, anon_vma); +list_for_each_entry_reverse (avc, &dst->anon_vma_chain, same_vma) { +struct anon_vma *anon_vma = avc->anon_vma; + +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root);/* * Reuse existing anon_vma if it has no vma and only one @@ -316,7 +324,6 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) }if (dst->anon_vma) dst->anon_vma->num_active_vmas++; -anon_vma_unlock_write (src->anon_vma);return 0 ;@@ -385,8 +392,10 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)get_anon_vma (anon_vma->root);/* Mark this anon_vma as the one where our new (COWed) pages go. */ vma->anon_vma = anon_vma; +anon_vma_chain_assign (vma, avc, anon_vma); +/* Now let rmap see it. */ anon_vma_lock_write (anon_vma); -anon_vma_chain_link (vma, avc, anon_vma); +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); anon_vma->parent->num_children++;anon_vma_unlock_write (anon_vma); --2.52.0
Suren Baghdasaryan
maintainerRe: [PATCH 7/8] mm/rmap: allocate anon_vma_chain objects unlocked when possible
Quoted 90 lines · Lorenzo Stoakes
Quoted 23 lines · Suren Baghdasaryan
Quoted 21 lines · Lorenzo Stoakes
There is no reason to allocate the
anon_vma_chainunder theanon_vmawrite lock when cloning - we can in fact assign these to the destination VMA safely as we hold the exclusive mmap lock and therefore preclude anybody else accessing these fields.We only need take the
anon_vmawrite lock when we link rbtree edges from theanon_vmato the newly established AVCs.This also allows us to eliminate the weird
GFP_NOWAIT,GFP_KERNELdance introduced in commitdd34739c03f2("mm: avoidanon_vma_chainallocation underanon_vmalock"), further simplifying this logic.This should reduce lock
anon_vmacontention, and clarifies exactly where theanon_vmalock is required.We cannot adjust
__anon_vma_prepare()in the same way as this is only protected by VMA read lock, so we have to perform the allocation here under theanon_vmawrite lock andpage_table_lock(to protect against racing threads), and we wish to retain the lock ordering.Quoted 5 lines
Quoted 1 line
Signed-off-by Lorenzo Stoakes <[email protected]>One nit but otherwise nice cleanup.
Reviewed-by Suren Baghdasaryan <[email protected]>Thanks!
One nice thing with the separate cleanup_partial_anon_vmas()'s function introduced as part of this review (thanks for the good spot!) is we can now simplify this
_evenfurther_since we don't even insert anything into the interval tree at the point of allocation, and so freeing is just a case of freeing up AVC's.Quoted 41 lines
Quoted 41 lines
--- mm/rmap.c | 49 +++++++++++++++++++++++++++++-------------------- 1 file changed, 29 insertions(+), 20 deletions(-)mm/rmap.c +5
-5@@ -146,14 +146,13 @@ static void anon_vma_chain_free(struct anon_vma_chain *anon_vma_chain)kmem_cache_free (anon_vma_chain_cachep, anon_vma_chain); } -static void anon_vma_chain_link (struct vm_area_struct *vma, -struct anon_vma_chain *avc, -struct anon_vma *anon_vma) +static void anon_vma_chain_assign (struct vm_area_struct *vma, +struct anon_vma_chain *avc, +struct anon_vma *anon_vma) { avc->vma = vma; avc->anon_vma = anon_vma;list_add (&avc->same_vma, &vma->anon_vma_chain); -anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); }/** @@ -210,7 +209,8 @@ int __anon_vma_prepare(struct vm_area_struct *vma)spin_lock (&mm->page_table_lock);if (likely (!vma->anon_vma)) { vma->anon_vma = anon_vma; -anon_vma_chain_link (vma, avc, anon_vma); +anon_vma_chain_assign (vma, avc, anon_vma); +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); anon_vma->num_active_vmas++; allocated =NULL ; avc =NULL ;@@ -287,20 +287,28 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)check_anon_vma_clone (dst, src);Quoted 9 lines
mm/rmap.c int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) · line 295
Quoted 6 lines
mm/rmap.c +6
-0+/* +* Allocate AVCs. We don't need an anon_vma lock for this as we +* are not updating the anon_vma rbtree nor are we changing +* anon_vma statistics. +* +* We hold the mmap write lock so there's no possibliity of To be more specific, we are holding src's mmap write lock. I think clarifying that will avoid any confusion.
Quoted 1 line
Well, it's the same mm for both right? :)
Hmm. I think in dup_mmap()->anon_vma_fork()->anon_vma_clone() call chain the dst->vm_mm and src->vm_mm are different, no? After assignment at https://elixir.bootlin.com/linux/v6.19-rc4/source/mm/mmap.c#L1779src->vm_mm is pointing to oldmm while dst->vm_mm is pointing to mm. Am I reading this wrong?
Quoted 64 lines
and actually the observations
would be made around dst no? As that's where the unlinked AVC's are being established.I think more clear is 'We hold the exclusive mmap write lock' just to highlight that it excludes anybody else from accessing these fields in the VMA.
Quoted 56 lines
Quoted 56 lines
mm/rmap.c +18
-16+* the unlinked AVC's being observed yet. +*/ +list_for_each_entry (pavc, &src->anon_vma_chain, same_vma) { + avc =anon_vma_chain_alloc (GFP_KERNEL ); +if (!avc) + goto enomem_failure; + +anon_vma_chain_assign (dst, avc, pavc->anon_vma); + } + +/* Now link the anon_vma's back to the newly inserted AVCs. */ anon_vma_lock_write (src->anon_vma); -list_for_each_entry_reverse (pavc, &src->anon_vma_chain, same_vma) { -struct anon_vma *anon_vma; - - avc =anon_vma_chain_alloc (GFP_NOWAIT ); -if (unlikely (!avc)) { -anon_vma_unlock_write (src->anon_vma); - avc =anon_vma_chain_alloc (GFP_KERNEL ); -if (!avc) - goto enomem_failure; -anon_vma_lock_write (src->anon_vma); - } - anon_vma = pavc->anon_vma; -anon_vma_chain_link (dst, avc, anon_vma); +list_for_each_entry_reverse (avc, &dst->anon_vma_chain, same_vma) { +struct anon_vma *anon_vma = avc->anon_vma; + +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root);/* * Reuse existing anon_vma if it has no vma and only one @@ -316,7 +324,6 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) }if (dst->anon_vma) dst->anon_vma->num_active_vmas++; -anon_vma_unlock_write (src->anon_vma);return 0 ;@@ -385,8 +392,10 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)get_anon_vma (anon_vma->root);/* Mark this anon_vma as the one where our new (COWed) pages go. */ vma->anon_vma = anon_vma; +anon_vma_chain_assign (vma, avc, anon_vma); +/* Now let rmap see it. */ anon_vma_lock_write (anon_vma); -anon_vma_chain_link (vma, avc, anon_vma); +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); anon_vma->parent->num_children++;anon_vma_unlock_write (anon_vma); --2.52.0
Re: [PATCH 7/8] mm/rmap: allocate anon_vma_chain objects unlocked when possible
Quoted 25 lines · Suren Baghdasaryan
Quoted 25 lines
Quoted 24 lines
Quoted 12 lines
/**
mm/rmap.c +2
-1@@ -210,7 +209,8 @@ int __anon_vma_prepare(struct vm_area_struct *vma)spin_lock (&mm->page_table_lock);if (likely (!vma->anon_vma)) { vma->anon_vma = anon_vma; -anon_vma_chain_link (vma, avc, anon_vma); +anon_vma_chain_assign (vma, avc, anon_vma); +anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); anon_vma->num_active_vmas++; allocated =NULL ; avc =NULL ;mm/rmap.c int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) · line 295
Quoted 9 lines
mm/rmap.c +6
-0@@ -287,20 +287,28 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)check_anon_vma_clone (dst, src); +/* +* Allocate AVCs. We don't need an anon_vma lock for this as we +* are not updating the anon_vma rbtree nor are we changing +* anon_vma statistics. +* +* We hold the mmap write lock so there's no possibliity of To be more specific, we are holding src's mmap write lock. I think clarifying that will avoid any confusion.
Quoted 7 lines
Quoted 1 line
Well, it's the same mm for both right? :)
Hmm. I think in dup_mmap()->anon_vma_fork()->anon_vma_clone() call chain the
dst->vm_mmandsrc->vm_mmare different, no? After assignment at https://elixir.bootlin.com/linux/v6.19-rc4/source/mm/mmap.c#L1779src->vm_mmis pointing to oldmm whiledst->vm_mmis pointing to mm. Am I reading this wrong?
Yup that's right sorry, and I even make that clear elsewhere, I'll send a fix-patch or something on the v2.
Cheers, Lorenzo
PATCH 8/8 mm/rmap: separate out fork-only logic on anon_vma_clone()
In reply to Lorenzo StoakesSpecify which operation is being performed to anon_vma_clone(), which allows us to do checks specific to each operation type, as well as to separate out and make clear that the anon_vma reuse logic is absolutely specific to fork only.
This opens the door to further refactorings and refinements later as we have more information to work with.
--- mm/internal.h | 11 +++++- mm/rmap.c | 67 ++++++++++++++++++++++---------- mm/vma.c | 6 +-- tools/testing/vma/vma_internal.h | 11 +++++- 4 files changed, 69 insertions(+), 26 deletions(-)
mm/internal.h +10 -1
@@ -244,7 +244,16 @@ static inline void anon_vma_unlock_read(struct anon_vma *anon_vma)struct anon_vma *folio_get_anon_vma (const struct folio *folio); -int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src); +/* Operations which modify VMAs. */ +enum vma_operation { +VMA_OP_SPLIT , +VMA_OP_MERGE_UNFAULTED , +VMA_OP_REMAP , +VMA_OP_FORK , +}; + +int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src, +enum vma_operation operation);int anon_vma_fork (struct vm_area_struct *vma,struct vm_area_struct *pvma);int __anon_vma_prepare (struct vm_area_struct *vma);void unlink_anon_vmas (struct vm_area_struct *vma);
mm/rmap.c +46 -21
113 lines not on this page · raw message
mm/vma.c +3 -3
@@ -525,7 +525,7 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,if (err) goto out_free_vmi; - err =anon_vma_clone (new, vma); + err =anon_vma_clone (new, vma,VMA_OP_SPLIT );if (err) goto out_free_mpol;
@@ -623,7 +623,7 @@ static int dup_anon_vma(struct vm_area_struct *dst,vma_assert_write_locked (dst); dst->anon_vma = src->anon_vma; - ret =anon_vma_clone (dst, src); + ret =anon_vma_clone (dst, src,VMA_OP_MERGE_UNFAULTED );if (ret)return ret;
@@ -1862,7 +1862,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,vma_set_range (new_vma, addr, addr + len, pgoff);if (vma_dup_policy (vma, new_vma)) goto out_free_vma; -if (anon_vma_clone (new_vma, vma)) +if (anon_vma_clone (new_vma, vma,VMA_OP_REMAP )) goto out_free_mempol;if (new_vma->vm_file)get_file (new_vma->vm_file);
tools/testing/vma/vma_internal.h +10 -1
@@ -600,6 +600,14 @@ struct mmap_action { bool hide_from_rmap_until_complete :1; }; +/* Operations which modify VMAs. */ +enum vma_operation { +VMA_OP_SPLIT , +VMA_OP_MERGE_UNFAULTED , +VMA_OP_REMAP , +VMA_OP_FORK , +}; +/* * Describes a VMA that is about to be mmap()'ed. Drivers may choose to * manipulate mutable fields which will cause those fields to be updated in the
@@ -1157,7 +1165,8 @@ static inline int vma_dup_policy(struct vm_area_struct *src, struct vm_area_strureturn 0 ; } -static inline int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) +static inline int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src, +enum vma_operation operation) {/* For testing purposes. We indicate that an anon_vma has been cloned. */ if (src->anon_vma !=NULL ) {
Signature · 1 line
2.52.0
Suren Baghdasaryan
maintainerRe: [PATCH 8/8] mm/rmap: separate out fork-only logic on anon_vma_clone()
Quoted 61 lines · Lorenzo Stoakes
Specify which operation is being performed to
anon_vma_clone(), which allows us to do checks specific to each operation type, as well as to separate out and make clear that theanon_vmareuse logic is absolutely specific to fork only.This opens the door to further refactorings and refinements later as we have more information to work with.
Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/internal.h | 11 +++++- mm/rmap.c | 67 ++++++++++++++++++++++---------- mm/vma.c | 6 +-- tools/testing/vma/vma_internal.h | 11 +++++- 4 files changed, 69 insertions(+), 26 deletions(-)mm/internal.h +10
-1@@ -244,7 +244,16 @@ static inline void anon_vma_unlock_read(struct anon_vma *anon_vma)struct anon_vma *folio_get_anon_vma (const struct folio *folio); -int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src); +/* Operations which modify VMAs. */ +enum vma_operation { +VMA_OP_SPLIT , +VMA_OP_MERGE_UNFAULTED , +VMA_OP_REMAP , +VMA_OP_FORK , +}; + +int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src, +enum vma_operation operation);int anon_vma_fork (struct vm_area_struct *vma,struct vm_area_struct *pvma);int __anon_vma_prepare (struct vm_area_struct *vma);void unlink_anon_vmas (struct vm_area_struct *vma);mm/rmap.c +4
-3@@ -232,12 +232,13 @@ int __anon_vma_prepare(struct vm_area_struct *vma) }static void check_anon_vma_clone (struct vm_area_struct *dst, -struct vm_area_struct *src) +struct vm_area_struct *src, +enum vma_operation operation) {/* The write lock must be held. */ mmap_assert_write_locked (src->vm_mm); -/* If not a fork (implied by dst->anon_vma) then must be on same mm. */ -VM_WARN_ON_ONCE (dst->anon_vma && dst->vm_mm != src->vm_mm); +/* If not a fork then must be on same mm. */ +VM_WARN_ON_ONCE (operation !=VMA_OP_FORK && dst->vm_mm != src->vm_mm);/* No source anon_vma is a no-op. */ VM_WARN_ON_ONCE (!src->anon_vma && !list_empty (&src->anon_vma_chain));
mm/rmap.c static void check_anon_vma_clone(struct vm_area_struct *dst, · line 258
Quoted 9 lines
mm/rmap.c +6
-0@@ -249,6 +250,35 @@ static void check_anon_vma_clone(struct vm_area_struct *dst,* must be the same across dst and src. */ VM_WARN_ON_ONCE (dst->anon_vma && dst->anon_vma != src->anon_vma); + +/* For the anon_vma to be compatible, it can only be singular. */ +VM_WARN_ON_ONCE (operation ==VMA_OP_MERGE_UNFAULTED && + !list_is_singular (&src->anon_vma_chain)); +#ifdef CONFIG_PER_VMA_LOCK +/* Only merging an unfaulted VMA leaves the destination attached. */
Do you mean to say that for all other operations dst should not be yet attached? If so, "leaves the destination attached" is a bit confusing choice of words to indicate that attaching the VMA in other cases has not yet happened. Maybe reword as: "For cases other than merging an unfaulted VMA, the VMA should not be attached yet."
mm/rmap.c static void check_anon_vma_clone(struct vm_area_struct *dst, · line 265
Quoted 7 lines
mm/rmap.c +7
-0+VM_WARN_ON_ONCE (operation !=VMA_OP_MERGE_UNFAULTED && +vma_is_attached (dst)); +#endif +} + +static void find_reusable_anon_vma (struct vm_area_struct *vma, +struct anon_vma *anon_vma)
Naming is hard but a function that assigns vma->anon_vma and increments num_active_vmas should not be called "find_reusable_anon_vma". I would suggest keeping the name but making it return the anon_vma (or NULL if it's not reusable), letting the caller do the assignment and the increment.
Quoted 6 lines
mm/rmap.c +6
-0+{ +/* If already populated, nothing to do.*/ +if (vma->anon_vma) +return ; + +/*
mm/rmap.c static void check_anon_vma_clone(struct vm_area_struct *dst, · line 281
Quoted 10 lines
mm/rmap.c +10
-0+* We reuse an anon_vma if any linking VMAs were unmapped and it has +* only a single child at most. +*/ +if (anon_vma->num_active_vmas >0 ) +return ; +if (anon_vma->num_children >1 ) +return ; + + vma->anon_vma = anon_vma; + anon_vma->num_active_vmas++;
You moved num_active_vmas++ here to fix the accounting later... Sneaky but correct AFAICT :)
Quoted 132 lines
mm/rmap.c +13
-18}/** @@ -256,6 +286,7 @@ static void check_anon_vma_clone(struct vm_area_struct *dst,* all of the anon_vma objects contained within @src anon_vma_chain's. * @dst: The destination VMA with an empty anon_vma_chain. * @src: The source VMA we wish to duplicate. +* @operation: The type of operation which resulted in the clone. * * This is the heart of the VMA side of the anon_vma implementation - we invoke * this function whenever we need to set up a new VMA's anon_vma state. @@ -278,14 +309,16 @@ static void check_anon_vma_clone(struct vm_area_struct *dst,* * Returns: 0 on success, -ENOMEM on failure. */ -int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) +int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src, +enum vma_operation operation) {struct anon_vma_chain *avc, *pavc; +struct anon_vma *active_anon_vma = src->anon_vma; -if (!src->anon_vma) +if (!active_anon_vma)return 0 ; -check_anon_vma_clone (dst, src); +check_anon_vma_clone (dst, src, operation);/* * Allocate AVCs. We don't need an anon_vma lock for this as we @@ -309,22 +342,14 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)struct anon_vma *anon_vma = avc->anon_vma;anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); - -/* -* Reuse existing anon_vma if it has no vma and only one -* anon_vma child. -* -* Root anon_vma is never reused: -* it has self-parent reference and at least one child. -*/ -if (!dst->anon_vma && src->anon_vma && - anon_vma->num_children <2 && - anon_vma->num_active_vmas ==0 ) - dst->anon_vma = anon_vma; +if (operation ==VMA_OP_FORK ) +find_reusable_anon_vma (dst, anon_vma); } -if (dst->anon_vma) + +if (operation !=VMA_OP_FORK ) dst->anon_vma->num_active_vmas++; -anon_vma_unlock_write (src->anon_vma); + +anon_vma_unlock_write (active_anon_vma);return 0 ; enomem_failure:@@ -361,7 +386,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)* First, attach the new VMA to the parent VMA's anon_vmas, * so rmap can find non-COWed pages in child processes. */ - error =anon_vma_clone (vma, pvma); + error =anon_vma_clone (vma, pvma,VMA_OP_FORK );if (error)return error;mm/vma.c +3
-3@@ -525,7 +525,7 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,if (err) goto out_free_vmi; - err =anon_vma_clone (new, vma); + err =anon_vma_clone (new, vma,VMA_OP_SPLIT );if (err) goto out_free_mpol;@@ -623,7 +623,7 @@ static int dup_anon_vma(struct vm_area_struct *dst,vma_assert_write_locked (dst); dst->anon_vma = src->anon_vma; - ret =anon_vma_clone (dst, src); + ret =anon_vma_clone (dst, src,VMA_OP_MERGE_UNFAULTED );if (ret)return ret;@@ -1862,7 +1862,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,vma_set_range (new_vma, addr, addr + len, pgoff);if (vma_dup_policy (vma, new_vma)) goto out_free_vma; -if (anon_vma_clone (new_vma, vma)) +if (anon_vma_clone (new_vma, vma,VMA_OP_REMAP )) goto out_free_mempol;if (new_vma->vm_file)get_file (new_vma->vm_file);tools/testing/vma/vma_internal.h +10
-2@@ -600,6 +600,14 @@ struct mmap_action { bool hide_from_rmap_until_complete :1; }; +/* Operations which modify VMAs. */ +enum vma_operation { +VMA_OP_SPLIT , +VMA_OP_MERGE_UNFAULTED , +VMA_OP_REMAP , +VMA_OP_FORK , +}; +/* * Describes a VMA that is about to be mmap()'ed. Drivers may choose to * manipulate mutable fields which will cause those fields to be updated in the @@ -1157,7 +1165,8 @@ static inline int vma_dup_policy(struct vm_area_struct *src, struct vm_area_strureturn 0 ; } -static inline int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) +static inline int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src, +enum vma_operation operation) {/* For testing purposes. We indicate that an anon_vma has been cloned. */ if (src->anon_vma !=NULL ) { --2.52.0
Re: [PATCH 8/8] mm/rmap: separate out fork-only logic on anon_vma_clone()
Quoted 79 lines · Suren Baghdasaryan
Quoted 61 lines · Lorenzo Stoakes
Specify which operation is being performed to
anon_vma_clone(), which allows us to do checks specific to each operation type, as well as to separate out and make clear that theanon_vmareuse logic is absolutely specific to fork only.This opens the door to further refactorings and refinements later as we have more information to work with.
Signed-off-by Lorenzo Stoakes <[email protected]>--- mm/internal.h | 11 +++++- mm/rmap.c | 67 ++++++++++++++++++++++---------- mm/vma.c | 6 +-- tools/testing/vma/vma_internal.h | 11 +++++- 4 files changed, 69 insertions(+), 26 deletions(-)mm/internal.h +10
-1@@ -244,7 +244,16 @@ static inline void anon_vma_unlock_read(struct anon_vma *anon_vma)struct anon_vma *folio_get_anon_vma (const struct folio *folio); -int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src); +/* Operations which modify VMAs. */ +enum vma_operation { +VMA_OP_SPLIT , +VMA_OP_MERGE_UNFAULTED , +VMA_OP_REMAP , +VMA_OP_FORK , +}; + +int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src, +enum vma_operation operation);int anon_vma_fork (struct vm_area_struct *vma,struct vm_area_struct *pvma);int __anon_vma_prepare (struct vm_area_struct *vma);void unlink_anon_vmas (struct vm_area_struct *vma);mm/rmap.c +4
-3@@ -232,12 +232,13 @@ int __anon_vma_prepare(struct vm_area_struct *vma) }static void check_anon_vma_clone (struct vm_area_struct *dst, -struct vm_area_struct *src) +struct vm_area_struct *src, +enum vma_operation operation) {/* The write lock must be held. */ mmap_assert_write_locked (src->vm_mm); -/* If not a fork (implied by dst->anon_vma) then must be on same mm. */ -VM_WARN_ON_ONCE (dst->anon_vma && dst->vm_mm != src->vm_mm); +/* If not a fork then must be on same mm. */ +VM_WARN_ON_ONCE (operation !=VMA_OP_FORK && dst->vm_mm != src->vm_mm);/* No source anon_vma is a no-op. */ VM_WARN_ON_ONCE (!src->anon_vma && !list_empty (&src->anon_vma_chain));mm/rmap.c static void check_anon_vma_clone(struct vm_area_struct *dst, · line 258
Quoted 9 lines
mm/rmap.c +6
-0@@ -249,6 +250,35 @@ static void check_anon_vma_clone(struct vm_area_struct *dst,* must be the same across dst and src. */ VM_WARN_ON_ONCE (dst->anon_vma && dst->anon_vma != src->anon_vma); + +/* For the anon_vma to be compatible, it can only be singular. */ +VM_WARN_ON_ONCE (operation ==VMA_OP_MERGE_UNFAULTED && + !list_is_singular (&src->anon_vma_chain)); +#ifdef CONFIG_PER_VMA_LOCK +/* Only merging an unfaulted VMA leaves the destination attached. */ Do you mean to say that for all other operations dst should not be yet attached? If so, "leaves the destination attached" is a bit confusing choice of words to indicate that attaching the VMA in other cases has not yet happened. Maybe reword as: "For cases other than merging an unfaulted VMA, the VMA should not be attached yet."
Quoted 13 lines
mm/rmap.c static void check_anon_vma_clone(struct vm_area_struct *dst, · line 265
Quoted 7 lines
mm/rmap.c +7
-0+VM_WARN_ON_ONCE (operation !=VMA_OP_MERGE_UNFAULTED && +vma_is_attached (dst)); +#endif +} + +static void find_reusable_anon_vma (struct vm_area_struct *vma, +struct anon_vma *anon_vma)Naming is hard but a function that assigns
vma->anon_vmaand incrementsnum_active_vmasshould not be called "find_reusable_anon_vma". I would suggest keeping the name but making it return theanon_vma(or NULL if it's not reusable), letting the caller do the assignment and the increment.
I mean totally agree this name is bad actually :) good spot.
Since you might already have populated so we can't return NULL, so I think a rename is better here actually -> try_to_reuse_anon_vma() which more accurately describes what's going on and takes into account the fact it's doing a stats update also!
Quoted 12 lines
Quoted 12 lines
mm/rmap.c +11
-0+{ +/* If already populated, nothing to do.*/ +if (vma->anon_vma) +return ; + +/* +* We reuse an anon_vma if any linking VMAs were unmapped and it has +* only a single child at most. +*/ +if (anon_vma->num_active_vmas >0 ) +return ;
Quoted 8 lines
mm/rmap.c static void check_anon_vma_clone(struct vm_area_struct *dst, · line 281
Quoted 5 lines
mm/rmap.c +5
-0+if (anon_vma->num_children >1 ) +return ; + + vma->anon_vma = anon_vma; + anon_vma->num_active_vmas++;You moved num_active_vmas++ here to fix the accounting later... Sneaky but correct AFAICT :)
;)
I would say in my defence (perhaps unnecessarily, as being sneaky is a key kernel dev skill imo ;) that it's no more sneaky than the code already was. Probably :)
Really this is about making the situation clearer in anon_vma_clone(). We could instead have:
if (dst->anon_vma) dst->anon_vma->num_active_vmas++;
And with the assignment here, we cover off the fork + reuse case.
However this doesn't make it clear that in fact for all non-fork cases we are _always_ incrementing this value, which is what I wanted to make super obvious.
One thing here that I think would help is to rename vma to dst just to carry that across from anon_vma_clone() and to make it clear we're checking/manipulating the destination VMA here.
Quoted 133 lines
Quoted 133 lines
mm/rmap.c +13
-18}/** @@ -256,6 +286,7 @@ static void check_anon_vma_clone(struct vm_area_struct *dst,* all of the anon_vma objects contained within @src anon_vma_chain's. * @dst: The destination VMA with an empty anon_vma_chain. * @src: The source VMA we wish to duplicate. +* @operation: The type of operation which resulted in the clone. * * This is the heart of the VMA side of the anon_vma implementation - we invoke * this function whenever we need to set up a new VMA's anon_vma state. @@ -278,14 +309,16 @@ static void check_anon_vma_clone(struct vm_area_struct *dst,* * Returns: 0 on success, -ENOMEM on failure. */ -int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) +int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src, +enum vma_operation operation) {struct anon_vma_chain *avc, *pavc; +struct anon_vma *active_anon_vma = src->anon_vma; -if (!src->anon_vma) +if (!active_anon_vma)return 0 ; -check_anon_vma_clone (dst, src); +check_anon_vma_clone (dst, src, operation);/* * Allocate AVCs. We don't need an anon_vma lock for this as we @@ -309,22 +342,14 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)struct anon_vma *anon_vma = avc->anon_vma;anon_vma_interval_tree_insert (avc, &anon_vma->rb_root); - -/* -* Reuse existing anon_vma if it has no vma and only one -* anon_vma child. -* -* Root anon_vma is never reused: -* it has self-parent reference and at least one child. -*/ -if (!dst->anon_vma && src->anon_vma && - anon_vma->num_children <2 && - anon_vma->num_active_vmas ==0 ) - dst->anon_vma = anon_vma; +if (operation ==VMA_OP_FORK ) +find_reusable_anon_vma (dst, anon_vma); } -if (dst->anon_vma) + +if (operation !=VMA_OP_FORK ) dst->anon_vma->num_active_vmas++; -anon_vma_unlock_write (src->anon_vma); + +anon_vma_unlock_write (active_anon_vma);return 0 ; enomem_failure:@@ -361,7 +386,7 @@ int anon_vma_fork(struct vm_area_struct *vma, struct vm_area_struct *pvma)* First, attach the new VMA to the parent VMA's anon_vmas, * so rmap can find non-COWed pages in child processes. */ - error =anon_vma_clone (vma, pvma); + error =anon_vma_clone (vma, pvma,VMA_OP_FORK );if (error)return error;mm/vma.c +3
-3@@ -525,7 +525,7 @@ __split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma,if (err) goto out_free_vmi; - err =anon_vma_clone (new, vma); + err =anon_vma_clone (new, vma,VMA_OP_SPLIT );if (err) goto out_free_mpol;@@ -623,7 +623,7 @@ static int dup_anon_vma(struct vm_area_struct *dst,vma_assert_write_locked (dst); dst->anon_vma = src->anon_vma; - ret =anon_vma_clone (dst, src); + ret =anon_vma_clone (dst, src,VMA_OP_MERGE_UNFAULTED );if (ret)return ret;@@ -1862,7 +1862,7 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,vma_set_range (new_vma, addr, addr + len, pgoff);if (vma_dup_policy (vma, new_vma)) goto out_free_vma; -if (anon_vma_clone (new_vma, vma)) +if (anon_vma_clone (new_vma, vma,VMA_OP_REMAP )) goto out_free_mempol;if (new_vma->vm_file)get_file (new_vma->vm_file);tools/testing/vma/vma_internal.h +10
-2@@ -600,6 +600,14 @@ struct mmap_action { bool hide_from_rmap_until_complete :1; }; +/* Operations which modify VMAs. */ +enum vma_operation { +VMA_OP_SPLIT , +VMA_OP_MERGE_UNFAULTED , +VMA_OP_REMAP , +VMA_OP_FORK , +}; +/* * Describes a VMA that is about to be mmap()'ed. Drivers may choose to * manipulate mutable fields which will cause those fields to be updated in the @@ -1157,7 +1165,8 @@ static inline int vma_dup_policy(struct vm_area_struct *src, struct vm_area_strureturn 0 ; } -static inline int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src) +static inline int anon_vma_clone (struct vm_area_struct *dst,struct vm_area_struct *src, +enum vma_operation operation) {/* For testing purposes. We indicate that an anon_vma has been cloned. */ if (src->anon_vma !=NULL ) { --2.52.0