Skip to contentSkip to conversation

mm/rmap.c: don't reuse anon_vma if we just want a copy

PATCH In review Wei Yang 2 participants 5 messages Oct 11, 2019

Subsystems (2)
lore
Share & tools
lore

Wei Yang

author
#0 lore raw

PATCH v4 1/2 mm/rmap.c: don't reuse anon_vma if we just want a copy

Before commit 7a3ef208e662 ("mm: prevent endless growth of anon_vma hierarchy"), anon_vma_clone() doesn't change dst->anon_vma. While after this commit, anon_vma_clone() will try to reuse an exist one on forking.

But this commit go a little bit further for the case not forking. anon_vma_clone() is called from __vma_split(), __split_vma(), copy_vma() and anon_vma_fork(). For the first three places, the purpose here is get a copy of src and we don't expect to touch dst->anon_vma even it is NULL. While after that commit, it is possible to reuse an anon_vma when dst->anon_vma is NULL. This is not we intend to have.

This patch stop reuse anon_vma for non-fork cases.

Fix commit 7a3ef208e662 ("mm: prevent endless growth of anon_vma hierarchy")

---
v4:
* check dst->anon_vma in each iteration
v3:
* use dst->anon_vma and src->anon_vma to get reuse state pointed by Konstantin Khlebnikov

---
 mm/rmap.c | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)
mm/rmap.c +15 -9
index d9a23bb773bf..c34414567474 100644
@@ -250,13 +250,19 @@ static inline void unlock_anon_vma_root(struct anon_vma *root)
  * Attach the anon_vmas from src to dst.
  * Returns 0 on success, -ENOMEM on failure.
  *
- * If dst->anon_vma is NULL this function tries to find and reuse existing
- * anon_vma which has no vmas and only one child anon_vma. This prevents
- * degradation of anon_vma hierarchy to endless linear chain in case of
- * constantly forking task. On the other hand, an anon_vma with more than one
- * child isn't reused even if there was no alive vma, thus rmap walker has a
- * good chance of avoiding scanning the whole hierarchy when it searches where
- * page is mapped.
+ * anon_vma_clone() is called by __vma_split(), __split_vma(), copy_vma() and
+ * anon_vma_fork(). The first three want an exact copy of src, while the last
+ * one, anon_vma_fork(), may try to reuse an existing anon_vma to prevent
+ * endless growth of anon_vma. Since dst->anon_vma is set to NULL before call,
+ * we can identify this case by checking (!dst->anon_vma && src->anon_vma).
+ *
+ * If (!dst->anon_vma && src->anon_vma) is true, this function tries to find
+ * and reuse existing anon_vma which has no vmas and only one child anon_vma.
+ * This prevents degradation of anon_vma hierarchy to endless linear chain in
+ * case of constantly forking task. On the other hand, an anon_vma with more
+ * than one child isn't reused even if there was no alive vma, thus rmap
+ * walker has a good chance of avoiding scanning the whole hierarchy when it
+ * searches where page is mapped.
  */
 int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)
 {
@@ -286,8 +292,8 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)
 		 * will always reuse it. Root anon_vma is never reused:
 		 * it has self-parent reference and at least one child.
 		 */
-		if (!dst->anon_vma && anon_vma != src->anon_vma &&
-				anon_vma->degree < 2)
+		if (!dst->anon_vma && src->anon_vma &&
+		    anon_vma != src->anon_vma && anon_vma->degree < 2)
 			dst->anon_vma = anon_vma;
 	}
 	if (dst->anon_vma)
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH v4 2/2 mm/rmap.c: reuse mergeable anon_vma as parent when fork

In function __anon_vma_prepare(), we will try to find anon_vma if it is possible to reuse it. While on fork, the logic is different.

Since commit 5beb49305251 ("mm: change anon_vma linking to fix multi-process server scalability issue"), function anon_vma_clone() tries to allocate new anon_vma for child process. But the logic here will allocate a new anon_vma for each vma, even in parent this vma is mergeable and share the same anon_vma with its sibling. This may do better for scalability issue, while it is not necessary to do so especially after interval tree is used.

Commit 7a3ef208e662 ("mm: prevent endless growth of anon_vma hierarchy") tries to reuse some anon_vma by counting child anon_vma and attached vmas. While for those mergeable anon_vmas, we can just reuse it and not necessary to go through the logic.

After this change, kernel build test reduces 20% anon_vma allocation.

Do the same kernel build test, it shows run time in sys reduced 11.6%.

Origin:

real    2m50.467s
user    17m52.002s
sys     1m51.953s
real    2m48.662s
user    17m55.464s
sys     1m50.553s
real    2m51.143s
user    17m59.687s
sys     1m53.600s

Patched:

real 2m39.933s
user 17m1.835s
sys 1m38.802s

real 2m39.321s
user 17m1.634s
sys 1m39.206s

real 2m39.575s
user 17m1.420s
sys 1m38.845s

---
 mm/rmap.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
mm/rmap.c +13 -0
index c34414567474..2c13e2bfd393 100644
@@ -268,6 +268,19 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)
 {
 	struct anon_vma_chain *avc, *pavc;
 	struct anon_vma *root = NULL;
+	struct vm_area_struct *prev = dst->vm_prev, *pprev = src->vm_prev;
+
+	/*
+	 * If parent share anon_vma with its vm_prev, keep this sharing in in
+	 * child.
+	 *
+	 * 1. Parent has vm_prev, which implies we have vm_prev.
+	 * 2. Parent and its vm_prev have the same anon_vma.
+	 */
+	if (!dst->anon_vma && src->anon_vma &&
+	    pprev && pprev->anon_vma == src->anon_vma)
+		dst->anon_vma = prev->anon_vma;
+
 
 	list_for_each_entry_reverse(pavc, &src->anon_vma_chain, same_vma) {
 		struct anon_vma *anon_vma;
Signature · 1 line
2.17.1
Acked-by on v4 2/2
#2 lore raw

Re: [Patch v4 2/2] mm/rmap.c: reuse mergeable anon_vma as parent when fork

Quoted 44 lines · Wei Yang

In function __anon_vma_prepare(), we will try to find anon_vma if it is possible to reuse it. While on fork, the logic is different.

Since commit 5beb49305251 ("mm: change anon_vma linking to fix multi-process server scalability issue"), function anon_vma_clone() tries to allocate new anon_vma for child process. But the logic here will allocate a new anon_vma for each vma, even in parent this vma is mergeable and share the same anon_vma with its sibling. This may do better for scalability issue, while it is not necessary to do so especially after interval tree is used.

Commit 7a3ef208e662 ("mm: prevent endless growth of anon_vma hierarchy") tries to reuse some anon_vma by counting child anon_vma and attached vmas. While for those mergeable anon_vmas, we can just reuse it and not necessary to go through the logic.

After this change, kernel build test reduces 20% anon_vma allocation.

Do the same kernel build test, it shows run time in sys reduced 11.6%.

Origin:

real    2m50.467s
user    17m52.002s
sys     1m51.953s
real    2m48.662s
user    17m55.464s
sys     1m50.553s
real    2m51.143s
user    17m59.687s
sys     1m53.600s

Patched:

real 2m39.933s
user 17m1.835s
sys 1m38.802s

real 2m39.321s
user 17m1.634s
sys 1m39.206s

Quoted 5 lines

real 2m39.575s
user 17m1.420s
sys 1m38.845s

Quoted 15 lines
---
  mm/rmap.c | 13 +++++++++++++
  1 file changed, 13 insertions(+)
mm/rmap.c +3 -0
index c34414567474..2c13e2bfd393 100644
@@ -268,6 +268,19 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)
  {
  	struct anon_vma_chain *avc, *pavc;
  	struct anon_vma *root = NULL;
+	struct vm_area_struct *prev = dst->vm_prev, *pprev = src->vm_prev;
+
+	/*

mm/rmap.c int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src) · line 283

Quoted 10 lines
mm/rmap.c +10 -0
+	 * If parent share anon_vma with its vm_prev, keep this sharing in in
+	 * child.
+	 *
+	 * 1. Parent has vm_prev, which implies we have vm_prev.
+	 * 2. Parent and its vm_prev have the same anon_vma.
+	 */
+	if (!dst->anon_vma && src->anon_vma &&
+	    pprev && pprev->anon_vma == src->anon_vma)
+		dst->anon_vma = prev->anon_vma;
+

I believe that in present code "prev" cannot be NULL if !dst->anon_vma && src->anon_vma is true. It would be safer to check this explicitly.

Quoted 4 lines
mm/rmap.c +0 -0
  	list_for_each_entry_reverse(pavc, &src->anon_vma_chain, same_vma) {
  		struct anon_vma *anon_vma;
 
Acked-by on v4 1/2
#3 lore raw
In reply to Wei Yang
Quoted 11 lines · Wei Yang

Before commit 7a3ef208e662 ("mm: prevent endless growth of anon_vma hierarchy"), anon_vma_clone() doesn't change dst->anon_vma. While after this commit, anon_vma_clone() will try to reuse an exist one on forking.

But this commit go a little bit further for the case not forking. anon_vma_clone() is called from __vma_split(), __split_vma(), copy_vma() and anon_vma_fork(). For the first three places, the purpose here is get a copy of src and we don't expect to touch dst->anon_vma even it is NULL. While after that commit, it is possible to reuse an anon_vma when dst->anon_vma is NULL. This is not we intend to have.

Quoted 6 lines

This patch stop reuse anon_vma for non-fork cases.

Fix commit 7a3ef208e662 ("mm: prevent endless growth of anon_vma hierarchy")

Yes, reusing heuristic was designed for fork.
But this isn't strictly necessary - any vmas could share anon_vma. For example all vmas in system could be linked with single anon_vma.

Quoted 54 lines

---
v4:
* check dst->anon_vma in each iteration
v3:
* use dst->anon_vma and src->anon_vma to get reuse state pointed by Konstantin Khlebnikov

---
  mm/rmap.c | 24 +++++++++++++++---------
  1 file changed, 15 insertions(+), 9 deletions(-)
mm/rmap.c +15 -9
index d9a23bb773bf..c34414567474 100644
@@ -250,13 +250,19 @@ static inline void unlock_anon_vma_root(struct anon_vma *root)
   * Attach the anon_vmas from src to dst.
   * Returns 0 on success, -ENOMEM on failure.
   *
- * If dst->anon_vma is NULL this function tries to find and reuse existing
- * anon_vma which has no vmas and only one child anon_vma. This prevents
- * degradation of anon_vma hierarchy to endless linear chain in case of
- * constantly forking task. On the other hand, an anon_vma with more than one
- * child isn't reused even if there was no alive vma, thus rmap walker has a
- * good chance of avoiding scanning the whole hierarchy when it searches where
- * page is mapped.
+ * anon_vma_clone() is called by __vma_split(), __split_vma(), copy_vma() and
+ * anon_vma_fork(). The first three want an exact copy of src, while the last
+ * one, anon_vma_fork(), may try to reuse an existing anon_vma to prevent
+ * endless growth of anon_vma. Since dst->anon_vma is set to NULL before call,
+ * we can identify this case by checking (!dst->anon_vma && src->anon_vma).
+ *
+ * If (!dst->anon_vma && src->anon_vma) is true, this function tries to find
+ * and reuse existing anon_vma which has no vmas and only one child anon_vma.
+ * This prevents degradation of anon_vma hierarchy to endless linear chain in
+ * case of constantly forking task. On the other hand, an anon_vma with more
+ * than one child isn't reused even if there was no alive vma, thus rmap
+ * walker has a good chance of avoiding scanning the whole hierarchy when it
+ * searches where page is mapped.
   */
  int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)
  {
@@ -286,8 +292,8 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)
  		 * will always reuse it. Root anon_vma is never reused:
  		 * it has self-parent reference and at least one child.
  		 */
-		if (!dst->anon_vma && anon_vma != src->anon_vma &&
-				anon_vma->degree < 2)
+		if (!dst->anon_vma && src->anon_vma &&
+		    anon_vma != src->anon_vma && anon_vma->degree < 2)
  			dst->anon_vma = anon_vma;
  	}
  	if (dst->anon_vma)
author
#4 lore raw
Quoted 14 lines · Konstantin Khlebnikov
Quoted 13 lines · Wei Yang

Before commit 7a3ef208e662 ("mm: prevent endless growth of anon_vma hierarchy"), anon_vma_clone() doesn't change dst->anon_vma. While after this commit, anon_vma_clone() will try to reuse an exist one on forking.

But this commit go a little bit further for the case not forking. anon_vma_clone() is called from __vma_split(), __split_vma(), copy_vma() and anon_vma_fork(). For the first three places, the purpose here is get a copy of src and we don't expect to touch dst->anon_vma even it is NULL. While after that commit, it is possible to reuse an anon_vma when dst->anon_vma is NULL. This is not we intend to have.

This patch stop reuse anon_vma for non-fork cases.

Quoted 8 lines
Quoted 4 lines

Fix commit 7a3ef208e662 ("mm: prevent endless growth of anon_vma hierarchy")

Yes, reusing heuristic was designed for fork.
But this isn't strictly necessary - any vmas could share anon_vma. For example all vmas in system could be linked with single anon_vma.

Yes, agree with you.

Thanks for your comment :-)

Quoted 57 lines
Quoted 54 lines

---
v4:
* check dst->anon_vma in each iteration
v3:
* use dst->anon_vma and src->anon_vma to get reuse state pointed by Konstantin Khlebnikov

---
  mm/rmap.c | 24 +++++++++++++++---------
  1 file changed, 15 insertions(+), 9 deletions(-)
mm/rmap.c +15 -9
index d9a23bb773bf..c34414567474 100644
@@ -250,13 +250,19 @@ static inline void unlock_anon_vma_root(struct anon_vma *root)
   * Attach the anon_vmas from src to dst.
   * Returns 0 on success, -ENOMEM on failure.
   *
- * If dst->anon_vma is NULL this function tries to find and reuse existing
- * anon_vma which has no vmas and only one child anon_vma. This prevents
- * degradation of anon_vma hierarchy to endless linear chain in case of
- * constantly forking task. On the other hand, an anon_vma with more than one
- * child isn't reused even if there was no alive vma, thus rmap walker has a
- * good chance of avoiding scanning the whole hierarchy when it searches where
- * page is mapped.
+ * anon_vma_clone() is called by __vma_split(), __split_vma(), copy_vma() and
+ * anon_vma_fork(). The first three want an exact copy of src, while the last
+ * one, anon_vma_fork(), may try to reuse an existing anon_vma to prevent
+ * endless growth of anon_vma. Since dst->anon_vma is set to NULL before call,
+ * we can identify this case by checking (!dst->anon_vma && src->anon_vma).
+ *
+ * If (!dst->anon_vma && src->anon_vma) is true, this function tries to find
+ * and reuse existing anon_vma which has no vmas and only one child anon_vma.
+ * This prevents degradation of anon_vma hierarchy to endless linear chain in
+ * case of constantly forking task. On the other hand, an anon_vma with more
+ * than one child isn't reused even if there was no alive vma, thus rmap
+ * walker has a good chance of avoiding scanning the whole hierarchy when it
+ * searches where page is mapped.
   */
  int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)
  {
@@ -286,8 +292,8 @@ int anon_vma_clone(struct vm_area_struct *dst, struct vm_area_struct *src)
  		 * will always reuse it. Root anon_vma is never reused:
  		 * it has self-parent reference and at least one child.
  		 */
-		if (!dst->anon_vma && anon_vma != src->anon_vma &&
-				anon_vma->degree < 2)
+		if (!dst->anon_vma && src->anon_vma &&
+		    anon_vma != src->anon_vma && anon_vma->degree < 2)
  			dst->anon_vma = anon_vma;
  	}
  	if (dst->anon_vma)
Signature · 2 lines
Wei Yang
Help you, Help me