aim7 scalability issue on 4 socket machine
Subsystems (2)
Aim7 result is bad on my new Nehalem machines (4*8*2 logical cpu). Perf counter shows spinlock consumes 70% cpu time on the machine. Lock_stat shows anon_vma->lock causes most of the spinlock contention. Function tracer shows below call chain creates the spinlock.
do_brk => vma_merge =>vma_adjust
Aim7 consists of lots of subtests. One test is to fork lots of processes and every process calls sbrk for 1000 times to grow/shrink the heap. All the vma of the heap of all sub-processes point to the same anon_vma and use the same anon_vma->lock. When sbrk is called, kernel calls do_brk => vma_merge =>vma_adjust and lock anon_vma->lock to create spinlock contentions.
There is a comment section in front of spin_lock(&anon_vma->lock. It says anon_vma lock can be optimized when just changing vma->vm_end. As a matter of fact, anon_vma->lock is used to protect anon_vma->list when an entry is deleted/inserted or the list is accessed. There is no such deletion/insertion if only vma->end is changed in function vma_adjust.
Below patch fixes it.
Test results with kernel 2.6.31-rc8. The improvement on the machine is about 150%.
Signed-off-by: Zhang Yanmin <[email protected]>
---
mm/mmap.c +23 -12
@@ -512,6 +512,7 @@ void vma_adjust(struct vm_area_struct *vstruct anon_vma *anon_vma =NULL ;long adjust_next =0 ;int remove_next =0 ; +int anon_vma_use_lock;if (next && !insert) {if (end >= next->vm_end) {
@@ -568,22 +569,32 @@ again: remove_next = 1 + (end > next-> } } -/* -* When changing only vma->vm_end, we don't really need -* anon_vma lock: but is that case worth optimizing out? -*/ if (vma->anon_vma) anon_vma = vma->anon_vma; + anon_vma_use_lock =0 ;if (anon_vma) { -spin_lock (&anon_vma->lock);/* -* Easily overlooked: when mprotect shifts the boundary, -* make sure the expanding vma has anon_vma set if the -* shrinking vma had, to cover any anon pages imported. +* When changing only vma->vm_end, we don't really need +* anon_vma lock. +* ana_vma->lock is to protect the access to the list +* started from anon_vma->head. If we don't remove or +* insert a vma to the list, and also don't access +* the list, we don't need ana_vma->lock. */ -if (importer && !importer->anon_vma) { - importer->anon_vma = anon_vma; -__anon_vma_link (importer); +if (remove_next || + insert || + (importer && !importer->anon_vma)) { + anon_vma_use_lock =1 ; +spin_lock (&anon_vma->lock); +/* +* Easily overlooked: when mprotect shifts the boundary, +* make sure the expanding vma has anon_vma set if the +* shrinking vma had, to cover any anon pages imported. +*/ +if (importer && !importer->anon_vma) { + importer->anon_vma = anon_vma; +__anon_vma_link (importer); + } } }
@@ -628,7 +639,7 @@ again: remove_next = 1 + (end > next->__insert_vm_struct (mm, insert); } -if (anon_vma) +if (anon_vma_use_lock)spin_unlock (&anon_vma->lock);if (mapping)spin_unlock (&mapping->i_mmap_lock);
Quoted 19 lines · Yanmin
Aim7 result is bad on my new Nehalem machines (4*8*2 logical cpu). Perf counter shows spinlock consumes 70% cpu time on the machine.
Lock_statshowsanon_vma->lockcauses most of the spinlock contention. Function tracer shows below call chain creates the spinlock.
do_brk=>vma_merge=>vma_adjustAim7 consists of lots of subtests. One test is to fork lots of processes and every process calls sbrk for 1000 times to grow/shrink the heap. All the vma of the heap of all sub-processes point to the same
anon_vmaand use the sameanon_vma->lock. When sbrk is called, kernel callsdo_brk=>vma_merge=>vma_adjust and lockanon_vma->lockto create spinlock contentions.There is a comment section in front of spin_lock(&anon_vma->lock. It says
anon_vmalock can be optimized when just changingvma->vm_end. As a matter of fact,anon_vma->lockis used to protectanon_vma->listwhen an entry is deleted/inserted or the list is accessed. There is no such deletion/insertion if onlyvma->endis changed in functionvma_adjust.
Quoted 3 lines
Below patch fixes it.
Test results with kernel 2.6.31-rc8. The improvement on the machine is about 150%.
Did you see Lee's patch?:
http://lkml.org/lkml/2009/9/9/290
Added Lee and Hugh to CC, retained the below patch for them.
Quoted 69 lines
Signed-off-by: Zhang Yanmin <[email protected]>
---
mm/mmap.c +23
-12@@ -512,6 +512,7 @@ void vma_adjust(struct vm_area_struct *vstruct anon_vma *anon_vma =NULL ;long adjust_next =0 ;int remove_next =0 ; +int anon_vma_use_lock;if (next && !insert) {if (end >= next->vm_end) {@@ -568,22 +569,32 @@ again: remove_next = 1 + (end > next-> } } -/* -* When changing only vma->vm_end, we don't really need -* anon_vma lock: but is that case worth optimizing out? -*/ if (vma->anon_vma) anon_vma = vma->anon_vma; + anon_vma_use_lock =0 ;if (anon_vma) { -spin_lock (&anon_vma->lock);/* -* Easily overlooked: when mprotect shifts the boundary, -* make sure the expanding vma has anon_vma set if the -* shrinking vma had, to cover any anon pages imported. +* When changing only vma->vm_end, we don't really need +* anon_vma lock. +* ana_vma->lock is to protect the access to the list +* started from anon_vma->head. If we don't remove or +* insert a vma to the list, and also don't access +* the list, we don't need ana_vma->lock. */ -if (importer && !importer->anon_vma) { - importer->anon_vma = anon_vma; -__anon_vma_link (importer); +if (remove_next || + insert || + (importer && !importer->anon_vma)) { + anon_vma_use_lock =1 ; +spin_lock (&anon_vma->lock); +/* +* Easily overlooked: when mprotect shifts the boundary, +* make sure the expanding vma has anon_vma set if the +* shrinking vma had, to cover any anon pages imported. +*/ +if (importer && !importer->anon_vma) { + importer->anon_vma = anon_vma; +__anon_vma_link (importer); + } } }@@ -628,7 +639,7 @@ again: remove_next = 1 + (end > next->__insert_vm_struct (mm, insert); } -if (anon_vma) +if (anon_vma_use_lock)spin_unlock (&anon_vma->lock);if (mapping)spin_unlock (&mapping->i_mmap_lock);
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Yanmin
Aim7 result is bad on my new Nehalem machines (4*8*2 logical cpu). Perf counter shows spinlock consumes 70% cpu time on the machine.
Lock_statshowsanon_vma->lockcauses most of the spinlock contention. Function tracer shows below call chain creates the spinlock.
do_brk=>vma_merge=>vma_adjustAim7 consists of lots of subtests. One test is to fork lots of processes and every process calls sbrk for 1000 times to grow/shrink the heap. All the vma of the heap of all sub-processes point to the same
anon_vmaand use the sameanon_vma->lock. When sbrk is called, kernel callsdo_brk=>vma_merge=>vma_adjust and lockanon_vma->lockto create spinlock contentions.There is a comment section in front of spin_lock(&anon_vma->lock. It says
anon_vmalock can be optimized when just changingvma->vm_end. As a matter of fact,anon_vma->lockis used to protectanon_vma->listwhen an entry is deleted/inserted or the list is accessed. There is no such deletion/insertion if onlyvma->endis changed in functionvma_adjust.Below patch fixes it.
Quoted 7 lines
Quoted 1 line
Test results with kernel 2.6.31-rc8. The improvement on the machine is about 150%.
Did you see Lee's patch?:
http://lkml.org/lkml/2009/9/9/290
Added Lee and Hugh to CC, retained the below patch for them.
Thanks a lot for the CC, Peter.
See my reply to that mail for the slightly corrected version.
Yes, Yanmin and Lee appear to be fixing exactly the same issue. I haven't thought through Yanmin's version for correctness, but it lacks the vm_start check I added to Lee's, and I do prefer Lee's style - hey, nothing personal!
So, Yanmin, please retest with http://lkml.org/lkml/2009/9/13/25
and let us know if that works as well for you - thanks.
Hugh
Quoted 34 lines · Hugh Dickins
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Yanmin
Aim7 result is bad on my new Nehalem machines (4*8*2 logical cpu). Perf counter shows spinlock consumes 70% cpu time on the machine.
Lock_statshowsanon_vma->lockcauses most of the spinlock contention. Function tracer shows below call chain creates the spinlock.
do_brk=>vma_merge=>vma_adjustAim7 consists of lots of subtests. One test is to fork lots of processes and every process calls sbrk for 1000 times to grow/shrink the heap. All the vma of the heap of all sub-processes point to the same
anon_vmaand use the sameanon_vma->lock. When sbrk is called, kernel callsdo_brk=>vma_merge=>vma_adjust and lockanon_vma->lockto create spinlock contentions.There is a comment section in front of spin_lock(&anon_vma->lock. It says
anon_vmalock can be optimized when just changingvma->vm_end. As a matter of fact,anon_vma->lockis used to protectanon_vma->listwhen an entry is deleted/inserted or the list is accessed. There is no such deletion/insertion if onlyvma->endis changed in functionvma_adjust.Below patch fixes it.
Quoted 7 lines
Quoted 1 line
Test results with kernel 2.6.31-rc8. The improvement on the machine is about 150%.
Did you see Lee's patch?:
http://lkml.org/lkml/2009/9/9/290
Added Lee and Hugh to CC, retained the below patch for them.
Thanks a lot for the CC, Peter.
See my reply to that mail for the slightly corrected version.
Quoted 7 lines
Yes, Yanmin and Lee appear to be fixing exactly the same issue. I haven't thought through Yanmin's version for correctness, but it lacks the
vm_startcheck I added to Lee's, and I do prefer Lee's style - hey, nothing personal!So, Yanmin, please retest with http://lkml.org/lkml/2009/9/13/25
and let us know if that works as well for you - thanks.
I tested Lee's patch and it does fix the issue.
Yanmin
Andrew Morton
maintainerQuoted 9 lines · Yanmin
Quoted 9 lines
Quoted 5 lines
Did you see Lee's patch?:
http://lkml.org/lkml/2009/9/9/290
Added Lee and Hugh to CC, retained the below patch for them.
Thanks a lot for the CC, Peter.
See my reply to that mail for the slightly corrected version.
Quoted 8 lines
Quoted 7 lines
Yes, Yanmin and Lee appear to be fixing exactly the same issue. I haven't thought through Yanmin's version for correctness, but it lacks the
vm_startcheck I added to Lee's, and I do prefer Lee's style - hey, nothing personal!So, Yanmin, please retest with http://lkml.org/lkml/2009/9/13/25
and let us know if that works as well for you - thanks.I tested Lee's patch and it does fix the issue.
Do we think we should cook up something for -stable?
Either this is a regression or the workload is particularly obscure.
aim7 is sufficiently non-obscure to make me wonder what's happened here?
Quoted 16 lines · Andrew Morton
Quoted 14 lines · Yanmin
Quoted 14 lines
Quoted 5 lines
Did you see Lee's patch?:
http://lkml.org/lkml/2009/9/9/290
Added Lee and Hugh to CC, retained the below patch for them.
Thanks a lot for the CC, Peter.
See my reply to that mail for the slightly corrected version.Yes, Yanmin and Lee appear to be fixing exactly the same issue. I haven't thought through Yanmin's version for correctness, but it lacks the
vm_startcheck I added to Lee's, and I do prefer Lee's style - hey, nothing personal!
Quoted 5 lines
Quoted 3 lines
Quoted 2 lines
So, Yanmin, please retest with http://lkml.org/lkml/2009/9/13/25
and let us know if that works as well for you - thanks.I tested Lee's patch and it does fix the issue.
Do we think we should cook up something for -stable?
It's better to have a patch for -stable.
Quoted 2 lines
Either this is a regression or the workload is particularly obscure.
This issue is not clear on dual socket Nehalem machine (2*4*2 cpu), but is severe on large machine (4*8*2 cpu).
Quoted 2 lines
aim7 is sufficiently non-obscure to make me wonder what's happened here?
I copy previous content below:
Aim7 consists of lots of subtests. One test is to fork lots of processes and every process calls sbrk for 1000 times to grow/shrink the heap. All the vma of the heap of all sub-processes point to the same anon_vma and use the same anon_vma->lock. When sbrk is called, kernel calls do_brk => vma_merge =>vma_adjust and lock anon_vma->lock to create spinlock contentions.
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Yanmin
Quoted 3 lines
So, Yanmin, please retest with http://lkml.org/lkml/2009/9/13/25
and let us know if that works as well for you - thanks.I tested Lee's patch and it does fix the issue.
Thanks for checking and reporting back, Yanmin.
Quoted 2 lines
Do we think we should cook up something for -stable?
Gosh, I laughed at Lee (sorry!) for suggesting it for -stable:
is stable really for getting a better number out of a benchmark?
I'd have thought the next release is the right place for that; but I've no problem if you guys and the stable guys agree it's appropriate.
Quoted 2 lines
Either this is a regression or the workload is particularly obscure.
I've not cross-checked descriptions, but assume Lee was actually testing on exactly the same kind of upcoming Nehalem as Yanmin, and that machine happens to have characteristics which show up badly here.
Quoted 2 lines
aim7 is sufficiently non-obscure to make me wonder what's happened here?
Not a regression, just the onward march of new hardware, I think. Could easily be other such things in other places with other tests.
Hugh
Andrew Morton
maintainerQuoted 26 lines · On Fri, 18 Sep 2009 07:53:58 +0100 (BST) Hugh Dickins…
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Yanmin
Quoted 3 lines
So, Yanmin, please retest with http://lkml.org/lkml/2009/9/13/25
and let us know if that works as well for you - thanks.I tested Lee's patch and it does fix the issue.
Thanks for checking and reporting back, Yanmin.
Quoted 2 lines
Do we think we should cook up something for -stable?
Gosh, I laughed at Lee (sorry!) for suggesting it for -stable:
is stable really for getting a better number out of a benchmark?I'd have thought the next release is the right place for that; but I've no problem if you guys and the stable guys agree it's appropriate.
Quoted 2 lines
Either this is a regression or the workload is particularly obscure.
I've not cross-checked descriptions, but assume Lee was actually testing on exactly the same kind of upcoming Nehalem as Yanmin, and that machine happens to have characteristics which show up badly here.
Quoted 1 line
Quoted 5 lines
Quoted 1 line
aim7 is sufficiently non-obscure to make me wonder what's happened here?
Not a regression, just the onward march of new hardware, I think. Could easily be other such things in other places with other tests.
Well, it comes down to the question "what is -stable for".
If you take it as "bugfixed version of the 2.6.x kernel" then no, speedups aren't appropriate.
If you consider -stable to be "something distros, etc will use" then yes, perhaps we serve those consumers better by including fairly major efficiency improvements.
I suspect most consumers of -stable would prefer the latter approach, as long as we don't go nuts.
Quoted 34 lines · Andrew Morton
Quoted 26 lines · On Fri, 18 Sep 2009 07:53:58 +0100 (BST) Hugh Dickins…
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Yanmin
Quoted 3 lines
So, Yanmin, please retest with http://lkml.org/lkml/2009/9/13/25
and let us know if that works as well for you - thanks.I tested Lee's patch and it does fix the issue.
Thanks for checking and reporting back, Yanmin.
Quoted 2 lines
Do we think we should cook up something for -stable?
Gosh, I laughed at Lee (sorry!) for suggesting it for -stable:
is stable really for getting a better number out of a benchmark?I'd have thought the next release is the right place for that; but I've no problem if you guys and the stable guys agree it's appropriate.
Quoted 2 lines
Either this is a regression or the workload is particularly obscure.
I've not cross-checked descriptions, but assume Lee was actually testing on exactly the same kind of upcoming Nehalem as Yanmin, and that machine happens to have characteristics which show up badly here.
Quoted 1 line
Quoted 5 lines
Quoted 1 line
aim7 is sufficiently non-obscure to make me wonder what's happened here?
Not a regression, just the onward march of new hardware, I think. Could easily be other such things in other places with other tests.
Quoted 8 lines
Well, it comes down to the question "what is -stable for".
If you take it as "bugfixed version of the 2.6.x kernel" then no, speedups aren't appropriate.
If you consider -stable to be "something distros, etc will use" then yes, perhaps we serve those consumers better by including fairly major efficiency improvements.
Well, if speedups are ok, then someone should update stable_rules file...?
...because I do not think it should be accepted based on that.
Pavel
It currently says:
-------------------
Rules on what kind of patches are accepted, and which ones are not, into the
"-stable" tree:
- It must be obviously correct and tested.
- It cannot be bigger than 100 lines, with context.
- It must fix only one thing.
- It must fix a real bug that bothers people (not a, "This could be a problem..." type thing).
- It must fix a problem that causes a build error (but not for things marked CONFIG_BROKEN), an oops, a hang, data corruption, a real security issue, or some "oh, that's not good" issue. In short,
something
critical.
- New device IDs and quirks are also accepted.
- No "theoretical race condition" issues, unless an explanation of
how the
race can be exploited is also provided.
- It cannot contain any "trivial" fixes in it (spelling changes, whitespace cleanups, etc).
- It must follow the Documentation/SubmittingPatches rules.
- It or an equivalent fix must already exist in Linus' tree. Quote
the
respective commit ID in Linus' tree in your patch submission to
-stable.
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 5 lines
- It must be obviously correct and tested.
- It cannot be bigger than 100 lines, with context.
- It must fix only one thing.
- It must fix a real bug that bothers people (not a, "This could be a problem..." type thing).
A significant slow down in a common situation is a "significant bug that bothers people"
-Andi
Quoted 8 lines · Andi Kleen
Quoted 5 lines
- It must be obviously correct and tested.
- It cannot be bigger than 100 lines, with context.
- It must fix only one thing.
- It must fix a real bug that bothers people (not a, "This could be a problem..." type thing).A significant slow down in a common situation is a "significant bug that bothers people"
Well, IIRC it was benchmark that was slowed down, on huge system, so it was exactly "this could be a problem".
Anyway, I don't care about -stable series too much (and sorry for replying to such an old mail), but perhaps the docs should be updated?
Examples cited there are such as "data corruption" or "oops", which is clearly different ballpark then "slowdown".
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 10 lines · Pavel Machek
Quoted 8 lines · Andi Kleen
Quoted 5 lines
- It must be obviously correct and tested.
- It cannot be bigger than 100 lines, with context.
- It must fix only one thing.
- It must fix a real bug that bothers people (not a, "This could be a problem..." type thing).A significant slow down in a common situation is a "significant bug that bothers people"
Quoted 8 lines
Well, IIRC it was benchmark that was slowed down, on huge system, so it was exactly "this could be a problem".
Anyway, I don't care about -stable series too much (and sorry for replying to such an old mail), but perhaps the docs should be updated?
Examples cited there are such as "data corruption" or "oops", which is clearly different ballpark then "slowdown".
Why does it really matter if the intent is the thing that matters here.
Deal with specifics on a case-by-case basis, and if you don't really care about this, then why dig up a many-month old thread?
strange,
greg k-h
Quoted 3 lines
Deal with specifics on a case-by-case basis, and if you don't really care about this, then why dig up a many-month old thread?
Digging it was a mistake, I was working on wrong machine. Sorry.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Quoted 7 lines · Hugh Dickins
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Yanmin
Quoted 3 lines
So, Yanmin, please retest with http://lkml.org/lkml/2009/9/13/25
and let us know if that works as well for you - thanks.I tested Lee's patch and it does fix the issue.
Quoted 7 lines
Thanks for checking and reporting back, Yanmin.
Quoted 2 lines
Do we think we should cook up something for -stable?
Gosh, I laughed at Lee (sorry!) for suggesting it for -stable:
is stable really for getting a better number out of a benchmark?
When your system is large enough scalability problems (e.g. lock contention) can be a serious bug. i.e. when your workload is 150% slower than expected that can well be a show stopper.
Admittedly the workload in this case was a benchmark, but it's not that far fetched to expect the same problem in a real application.
We had a similar problem with the accounting lock some time ago, I think that patch also went in.
So yes I think simple non intrusive fixes for serious scalability problems should be stable candidates.
Quoted 5 lines
Quoted 1 line
Either this is a regression or the workload is particularly obscure.
I've not cross-checked descriptions, but assume Lee was actually testing on exactly the same kind of upcoming Nehalem as Yanmin, and that machine happens to have characteristics which show up badly here.
AFAIK Lee usually tests on large IA64 boxes.
Quoted 4 lines
Quoted 1 line
aim7 is sufficiently non-obscure to make me wonder what's happened here?
Not a regression, just the onward march of new hardware, I think. Could easily be other such things in other places with other tests.
Yes, it's just a much larger machine, so old hidden scalability sins now appear.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 3 lines · Andi Kleen
So yes I think simple non intrusive fixes for serious scalability problems should be stable candidates.
This one is certainly non-intrusive, and you're all in agreement that it better go in, so I won't dispute that.
(But I'm a little worried that this easily fixed issue will turn out to be just the first in a string of not so easily fixed issues.)
Hugh
Quoted 29 lines · Andi Kleen
Quoted 7 lines · Hugh Dickins
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Yanmin
Quoted 3 lines
So, Yanmin, please retest with http://lkml.org/lkml/2009/9/13/25
and let us know if that works as well for you - thanks.I tested Lee's patch and it does fix the issue.
Quoted 7 lines
Thanks for checking and reporting back, Yanmin.
Quoted 2 lines
Do we think we should cook up something for -stable?
Gosh, I laughed at Lee (sorry!) for suggesting it for -stable:
is stable really for getting a better number out of a benchmark?When your system is large enough scalability problems (e.g. lock contention) can be a serious bug. i.e. when your workload is 150% slower than expected that can well be a show stopper.
Admittedly the workload in this case was a benchmark, but it's not that far fetched to expect the same problem in a real application.
We had a similar problem with the accounting lock some time ago, I think that patch also went in.
So yes I think simple non intrusive fixes for serious scalability problems should be stable candidates.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Either this is a regression or the workload is particularly obscure.
I've not cross-checked descriptions, but assume Lee was actually testing on exactly the same kind of upcoming Nehalem as Yanmin, and that machine happens to have characteristics which show up badly here.
AFAIK Lee usually tests on large IA64 boxes.
In this case, it's an x86_64 [DL785] platform--an 8 socket, 4 core Shanghai in a glueless, "twisted ladder" config.
Lee
Quoted 5 lines
Quoted 2 lines
AFAIK Lee usually tests on large IA64 boxes.
In this case, it's an
x86_64[DL785] platform--an 8 socket, 4 core Shanghai in a glueless, "twisted ladder" config.
Thanks for the correction. However it's still a quite different system, so I think it's clear it happens on multiple different systems.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 21 lines · Zhang, Yanmin
???Aim7 result is bad on my new Nehalem machines (4*8*2 logical cpu). Perf counter shows spinlock consumes 70% cpu time on the machine.
Lock_statshowsanon_vma->lockcauses most of the spinlock contention. Function tracer shows below call chain creates the spinlock.
do_brk=>vma_merge=>vma_adjustAim7 consists of lots of subtests. One test is to fork lots of processes and every process calls sbrk for 1000 times to grow/shrink the heap. All the vma of the heap of all sub-processes point to the same
anon_vmaand use the sameanon_vma->lock. When sbrk is called, kernel callsdo_brk=>vma_merge=>vma_adjust and lockanon_vma->lockto create spinlock contentions.There is a comment section in front of spin_lock(&anon_vma->lock. It says
anon_vmalock can be optimized when just changingvma->vm_end. As a matter of fact,anon_vma->lockis used to protectanon_vma->listwhen an entry is deleted/inserted or the list is accessed. There is no such deletion/insertion if onlyvma->endis changed in functionvma_adjust.
Quoted 4 lines
Below patch fixes it.
Test results with kernel 2.6.31-rc8. The improvement on the machine is about 150%.
Impressive speedup!
[ Also, the array of tools you used to debug this is impressive as well
;-) ]
Ingo