mm: vmalloc: Use the vmap_area_lock to protect ne_fit_preload_node
Subsystems (2)
PATCH 1/1 mm: vmalloc: Use the vmap_area_lock to protect ne_fit_preload_node
Replace preempt_enable() and preempt_disable() with the vmap_area_lock spin_lock instead. Calling spin_lock() with preempt disabled is illegal for -rt. Furthermore, enabling preemption inside the spin_lock() doesn't really make sense.
split purpose")
--- mm/vmalloc.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)
mm/vmalloc.c +3 -6
@@ -1091,11 +1091,11 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,* Even if it fails we do not really care about that. Just proceed * as it is. "overflow" path will refill the cache we allocate from. */ -preempt_disable (); +spin_lock (&vmap_area_lock);if (!__this_cpu_read (ne_fit_preload_node)) { -preempt_enable (); +spin_unlock (&vmap_area_lock); pva =kmem_cache_alloc_node (vmap_area_cachep,GFP_KERNEL , node); -preempt_disable (); +spin_lock (&vmap_area_lock);if (__this_cpu_cmpxchg (ne_fit_preload_node,NULL , pva)) {if (pva)
@@ -1103,9 +1103,6 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, } } -spin_lock (&vmap_area_lock); -preempt_enable (); -/* * If an allocation fails, the "vend" address is * returned. Therefore trigger the overflow path.
Signature · 1 line
2.16.4
Quoted 45 lines · Daniel Wagner
Replace
preempt_enable()andpreempt_disable()with thevmap_area_lockspin_lockinstead. Callingspin_lock()with preempt disabled is illegal for -rt. Furthermore, enabling preemption inside thespin_lock()doesn't really make sense.Fixes 82dd23e84be3 ("mm/vmalloc.c: preload a CPU with one object forsplit purpose")
Cc Uladzislau Rezki (Sony) <[email protected]>Signed-off-by Daniel Wagner <[email protected]>--- mm/vmalloc.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)mm/vmalloc.c +3
-6@@ -1091,11 +1091,11 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,* Even if it fails we do not really care about that. Just proceed * as it is. "overflow" path will refill the cache we allocate from. */ -preempt_disable (); +spin_lock (&vmap_area_lock);if (!__this_cpu_read (ne_fit_preload_node)) { -preempt_enable (); +spin_unlock (&vmap_area_lock); pva =kmem_cache_alloc_node (vmap_area_cachep,GFP_KERNEL , node); -preempt_disable (); +spin_lock (&vmap_area_lock);if (__this_cpu_cmpxchg (ne_fit_preload_node,NULL , pva)) {if (pva)@@ -1103,9 +1103,6 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, } } -spin_lock (&vmap_area_lock); -preempt_enable (); -/* * If an allocation fails, the "vend" address is * returned. Therefore trigger the overflow path. Signature · 1 line
2.16.4
Some background. The idea was to avoid of touching several times vmap_area_lock, therefore there are preempt_disable()/preempt_enable() instead, in order to stay on the same CPU.
When it comes to PREEMPT_RT it is a problem, so
--
Vlad Rezki
If you post something that is related to PREEMPT_RT please keep tglx and me in Cc.
Quoted 4 lines · Daniel Wagner
Replace
preempt_enable()andpreempt_disable()with thevmap_area_lockspin_lockinstead. Callingspin_lock()with preempt disabled is illegal for -rt. Furthermore, enabling preemption inside thespin_lock()doesn't really make sense.
This spin_lock will cause all CPUs to block on it while the preempt_disable() does not have this limitation.
I added a migrate_disable() in my -next tree. Looking at it again, I have reasonable doubt that this preempt_disable() is needed.
Quoted 29 lines
Fixes 82dd23e84be3 ("mm/vmalloc.c: preload a CPU with one object forsplit purpose")
Cc Uladzislau Rezki (Sony) <[email protected]>Signed-off-by Daniel Wagner <[email protected]>--- mm/vmalloc.c | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-)mm/vmalloc.c +3
-3@@ -1091,11 +1091,11 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,* Even if it fails we do not really care about that. Just proceed * as it is. "overflow" path will refill the cache we allocate from. */ -preempt_disable (); +spin_lock (&vmap_area_lock);if (!__this_cpu_read (ne_fit_preload_node)) { -preempt_enable (); +spin_unlock (&vmap_area_lock); pva =kmem_cache_alloc_node (vmap_area_cachep,GFP_KERNEL , node); -preempt_disable (); +spin_lock (&vmap_area_lock);if (__this_cpu_cmpxchg (ne_fit_preload_node,NULL , pva)) {if (pva)
mm/vmalloc.c static struct vmap_area *alloc_vmap_area(unsigned long size, · line 1108
Quoted 9 lines
mm/vmalloc.c +0
-3@@ -1103,9 +1103,6 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, } } -spin_lock (&vmap_area_lock); -preempt_enable (); -/* * If an allocation fails, the "vend" address is * returned. Therefore trigger the overflow path.
Sebastian
Uladzislau Rezki
maintainerQuoted 11 lines · Sebastian Andrzej Siewior
If you post something that is related to
PREEMPT_RTplease keep tglx and me in Cc.Quoted 3 lines · Daniel Wagner
Replace
preempt_enable()andpreempt_disable()with thevmap_area_lockspin_lockinstead. Callingspin_lock()with preempt disabled is illegal for -rt. Furthermore, enabling preemption inside theLooking at it again, I have reasonable doubt that this
preempt_disable()is needed.
The intention was to preload a current CPU with one extra object in non-atomic context, thus to use GFP_KERNEL permissive parameters. I.e. that allows us to avoid any allocation(if we stay on the same CPU) when we are in atomic context do splitting.
If we have migrate_disable/enable, then, i think preempt_enable/disable should be replaced by it and not the way how it has been proposed in the patch.
--
Vlad Rezki
Quoted 9 lines · Uladzislau Rezki
Quoted 8 lines · Sebastian Andrzej Siewior
If you post something that is related to
PREEMPT_RTplease keep tglx and me in Cc.Quoted 3 lines · Daniel Wagner
Replace
preempt_enable()andpreempt_disable()with thevmap_area_lockspin_lockinstead. Callingspin_lock()with preempt disabled is illegal for -rt. Furthermore, enabling preemption inside the
Quoted 7 lines
Quoted 3 lines
Looking at it again, I have reasonable doubt that this
preempt_disable()is needed.The intention was to preload a current CPU with one extra object in non-atomic context, thus to use
GFP_KERNELpermissive parameters. I.e. that allows us to avoid any allocation(if we stay on the same CPU) when we are in atomic context do splitting.
You could have been migrated to another CPU before the first preempt_disable(). You could have been migrated to another CPU while memory has been allocated.
I don't really see the point of that preempt_disable() besides keeping debug code quiet.
Quoted 3 lines
If we have migrate_disable/enable, then, i think preempt_enable/disable should be replaced by it and not the way how it has been proposed in the patch.
I don't think this patch is appropriate for upstream.
Sebastian
Uladzislau Rezki
maintainerQuoted 4 lines
You could have been migrated to another CPU while
memory has been allocated.
That is true that we can migrate since we allow preemption when allocate. But it does not really matter on which CPU an allocation occurs and whether we migrate or not.
If we land on another CPU or still stay on the same, we will check anyway one more time if it(another/same CPU) is preloaded or not:
<snip>
preempt_disable();
if (__this_cpu_cmpxchg(ne_fit_preload_node, NULL, pva)
<snip>
if another, we can free the object allocated on previous step if it already has it. If another CPU does not have it, save it in ne_fit_preload_node for another current CPU to reuse later. Further we can not migrate because of:
<snip>
spin_lock(&vmap_area_lock);
preempt_enable();<snip>
--
Vlad Rezki
Quoted 9 lines · Uladzislau Rezki
if another, we can free the object allocated on previous step if it already has it. If another CPU does not have it, save it in
ne_fit_preload_nodefor another current CPU to reuse later. Further we can not migrate because of:<snip>
spin_lock(&vmap_area_lock); preempt_enable();<snip>
ah right. So you keep the lock later on, I somehow missed that part. That way it actually makes sense.
Sebastian
Hi,
Quoted 6 lines · Sebastian Andrzej Siewior
Quoted 3 lines · Uladzislau Rezki
If we have migrate_disable/enable, then, i think preempt_enable/disable should be replaced by it and not the way how it has been proposed in the patch.
I don't think this patch is appropriate for upstream.
Yes, I agree. The discussion made this clear, this is only for -rt trees. Initially I though this should be in mainline too.
Thanks,
Daniel
Quoted 1 line · Daniel Wagner
Hi,
Hi Daniel,
Quoted 10 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Uladzislau Rezki
Quoted 3 lines
If we have migrate_disable/enable, then, i think preempt_enable/disable should be replaced by it and not the way how it has been proposed in the patch.
I don't think this patch is appropriate for upstream.
Yes, I agree. The discussion made this clear, this is only for -rt trees. Initially I though this should be in mainline too.
Sorry, this was _before_ Uladzislau pointed out that you *just* moved the lock that was there from the beginning. I missed that while looking over the patch. Based on that I don't think that this patch is not appropriate for upstream.
Quoted 2 lines
Thanks,
Daniel
Sebastian
Uladzislau Rezki
maintainerHello, Daniel, Sebastian.
Quoted 8 lines
Quoted 8 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Uladzislau Rezki
Quoted 3 lines
If we have migrate_disable/enable, then, i think preempt_enable/disable should be replaced by it and not the way how it has been proposed in the patch.
I don't think this patch is appropriate for upstream.
Quoted 8 lines
Quoted 2 lines
Yes, I agree. The discussion made this clear, this is only for -rt trees. Initially I though this should be in mainline too.
Sorry, this was _before_ Uladzislau pointed out that you *just* moved the lock that was there from the beginning. I missed that while looking over the patch. Based on that I don't think that this patch is not appropriate for upstream.
Yes that is a bit messy :) Then i do not see what that patch fixes in mainline? Instead it will just add an extra blocking, i did not want that therefore used preempt_enable/disable. But, when i saw this patch i got it as a preparation of PREEMPT_RT merging work.
--
Vlad Rezki
Quoted 18 lines · Uladzislau Rezki
Hello, Daniel, Sebastian.
Quoted 8 lines
Quoted 8 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Uladzislau Rezki
Quoted 3 lines
If we have migrate_disable/enable, then, i think preempt_enable/disable should be replaced by it and not the way how it has been proposed in the patch.
I don't think this patch is appropriate for upstream.
Quoted 8 lines
Quoted 2 lines
Yes, I agree. The discussion made this clear, this is only for -rt trees. Initially I though this should be in mainline too.
Sorry, this was _before_ Uladzislau pointed out that you *just* moved the lock that was there from the beginning. I missed that while looking over the patch. Based on that I don't think that this patch is not appropriate for upstream.
Quoted 4 lines
Yes that is a bit messy :) Then i do not see what that patch fixes in mainline? Instead it will just add an extra blocking, i did not want that therefore used preempt_enable/disable. But, when i saw this patch i got it as a preparation of
PREEMPT_RTmerging work.
Maybe I should add some background info here as well. Currently, I am creating an -rt tree on v5.3 for which I need this patch (or a migrate_disable() version of it). So this is slightly independent of the work Sebiastian is doing. Though the mainline effort of PREEMPT_RT will hit this problem as well.
I understood Sebiastian wrong above. I thought he suggest to use the migrate_disable() approach even for mainline.
I supppose, one thing which would help in this discussion, is what do you gain by using preempt_disable() instead of moving the lock up? Do you have performance numbers which could justify the code?
Uladzislau Rezki
maintainerQuoted 30 lines · Daniel Wagner
Quoted 18 lines · Uladzislau Rezki
Hello, Daniel, Sebastian.
Quoted 8 lines
Quoted 8 lines · Sebastian Andrzej Siewior
Quoted 6 lines · Uladzislau Rezki
Quoted 3 lines
If we have migrate_disable/enable, then, i think preempt_enable/disable should be replaced by it and not the way how it has been proposed in the patch.
I don't think this patch is appropriate for upstream.
Quoted 8 lines
Quoted 2 lines
Yes, I agree. The discussion made this clear, this is only for -rt trees. Initially I though this should be in mainline too.
Sorry, this was _before_ Uladzislau pointed out that you *just* moved the lock that was there from the beginning. I missed that while looking over the patch. Based on that I don't think that this patch is not appropriate for upstream.
Quoted 4 lines
Yes that is a bit messy :) Then i do not see what that patch fixes in mainline? Instead it will just add an extra blocking, i did not want that therefore used preempt_enable/disable. But, when i saw this patch i got it as a preparation of
PREEMPT_RTmerging work.Maybe I should add some background info here as well. Currently, I am creating an -rt tree on v5.3 for which I need this patch (or a
migrate_disable()version of it). So this is slightly independent of the work Sebiastian is doing. Though the mainline effort ofPREEMPT_RTwill hit this problem as well.
Quoted 7 lines
I understood Sebiastian wrong above. I thought he suggest to use the
migrate_disable()approach even for mainline.I supppose, one thing which would help in this discussion, is what do you gain by using
preempt_disable()instead of moving the lock up? Do you have performance numbers which could justify the code?
Actually there is a high lock contention on vmap_area_lock, because it is still global. You can have a look at last slide:
so this change will make it a bit higher. From the other hand i agree that for rt it should be fixed, probably it could be done like:
ifdef PREEMPT_RT
migrate_disable()
#else
preempt_disable()
...
but i am not sure it is good either.
--
Vlad Rezki
Quoted 11 lines · Uladzislau Rezki
Quoted 4 lines · Daniel Wagner
I supppose, one thing which would help in this discussion, is what do you gain by using
preempt_disable()instead of moving the lock up? Do you have performance numbers which could justify the code?Actually there is a high lock contention on
vmap_area_lock, because it is still global. You can have a look at last slide:so this change will make it a bit higher.
Thanks! I suspected something like this :(
On the todo-list page you stating that vmap_area_lock could be splitted and therefore reduce the contention. If you could avoid those preempt_disable() tricks and just use plain spin_locks() to protect it would be really helpful.
Quoted 10 lines
From the other hand i agree
that for rt it should be fixed, probably it could be done like:ifdef
PREEMPT_RT
migrate_disable()
#else
preempt_disable()
...but i am not sure it is good either.
I don't think this way to go. I guess Sebastian and Thomas have a better idea how to address this for PREEMPT_RT.
Quoted 8 lines · Uladzislau Rezki
Actually there is a high lock contention on
vmap_area_lock, because it is still global. You can have a look at last slide:so this change will make it a bit higher. From the other hand i agree that for rt it should be fixed, probably it could be done like:
Quoted 7 lines
ifdef
PREEMPT_RT
migrate_disable()
#else
preempt_disable()
...but i am not sure it is good either.
What is to be expected on average? Is the lock acquired and then released again because the slot is empty and memory needs to be allocated or can it be assumed that this hardly happens?
Sebastian
Uladzislau Rezki
maintainerQuoted 15 lines · Sebastian Andrzej Siewior
Quoted 14 lines · Uladzislau Rezki
Actually there is a high lock contention on
vmap_area_lock, because it is still global. You can have a look at last slide:so this change will make it a bit higher. From the other hand i agree that for rt it should be fixed, probably it could be done like:
ifdef
PREEMPT_RT
migrate_disable()
#else
preempt_disable()
...
Quoted 6 lines
Quoted 1 line
but i am not sure it is good either.
What is to be expected on average? Is the lock acquired and then released again because the slot is empty and memory needs to be allocated or can it be assumed that this hardly happens?
The lock is not released(we are not allowed), instead we just try to allocate with GFP_NOWAIT flag. It can happen if preallocation has been failed with GFP_KERNEL flag earlier:
<snip>
...
} else if (type == NE_FIT_TYPE) {
/*
* Split no edge of fit VA.
*
* | |
* L V NVA V R
* |---|-------|---|
*/
lva = __this_cpu_xchg(ne_fit_preload_node, NULL);
if (unlikely(!lva)) {
...
lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT);
... }
...
<snip>
How often we need an extra object for split purpose, the answer is it depends on. For example fork() path falls to that pattern.
I think we can assume that migration can hardly ever happen and that should be considered as rare case. Thus we can do a prealoading without worrying much if a it occurs:
<snip>
urezki@pc636:~/data/ssd/coding/linux-stable$ git diff
mm/vmalloc.c +2 -6
@@ -1089,20 +1089,16 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,* Even if it fails we do not really care about that. Just proceed * as it is. "overflow" path will refill the cache we allocate from. */ -preempt_disable (); -if (!__this_cpu_read (ne_fit_preload_node)) { -preempt_enable (); +if (!this_cpu_read (ne_fit_preload_node)) { pva =kmem_cache_alloc_node (vmap_area_cachep,GFP_KERNEL , node); -preempt_disable (); -if (__this_cpu_cmpxchg (ne_fit_preload_node,NULL , pva)) { +if (this_cpu_cmpxchg (ne_fit_preload_node,NULL , pva)) {if (pva)kmem_cache_free (vmap_area_cachep, pva); } }spin_lock (&vmap_area_lock); -preempt_enable ();/* * If an allocation fails, the "vend" address is
urezki@pc636:~/data/ssd/coding/linux-stable$
<snip>
so, we do not guarantee, instead we minimize number of allocations with GFP_NOWAIT flag. For example on my 4xCPUs i am not able to even trigger the case when CPU is not preloaded.
I can test it tomorrow on my 12xCPUs to see its behavior there.
--
Vlad Rezki
Uladzislau Rezki
maintainerQuoted 84 lines · Uladzislau Rezki
Quoted 15 lines · Sebastian Andrzej Siewior
Quoted 14 lines · Uladzislau Rezki
Actually there is a high lock contention on
vmap_area_lock, because it is still global. You can have a look at last slide:so this change will make it a bit higher. From the other hand i agree that for rt it should be fixed, probably it could be done like:
ifdef
PREEMPT_RT
migrate_disable()
#else
preempt_disable()
...Quoted 6 lines
Quoted 1 line
but i am not sure it is good either.
What is to be expected on average? Is the lock acquired and then released again because the slot is empty and memory needs to be allocated or can it be assumed that this hardly happens?
The lock is not released(we are not allowed), instead we just try to allocate with
GFP_NOWAITflag. It can happen if preallocation has been failed withGFP_KERNELflag earlier:<snip>
...
} else if (type ==NE_FIT_TYPE) {
/*
* Split no edge of fit VA.
** | | * L V NVA V R * |---|-------|---| */ lva = __this_cpu_xchg(ne_fit_preload_node, NULL); if (unlikely(!lva)) { ... lva = kmem_cache_alloc(vmap_area_cachep, GFP_NOWAIT); ...}
...
<snip>How often we need an extra object for split purpose, the answer is it depends on. For example
fork()path falls to that pattern.I think we can assume that migration can hardly ever happen and that should be considered as rare case. Thus we can do a prealoading without worrying much if a it occurs:
<snip>
urezki@pc636:~/data/ssd/coding/linux-stable$ git diffmm/vmalloc.c +2
-6@@ -1089,20 +1089,16 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,* Even if it fails we do not really care about that. Just proceed * as it is. "overflow" path will refill the cache we allocate from. */ -preempt_disable (); -if (!__this_cpu_read (ne_fit_preload_node)) { -preempt_enable (); +if (!this_cpu_read (ne_fit_preload_node)) { pva =kmem_cache_alloc_node (vmap_area_cachep,GFP_KERNEL , node); -preempt_disable (); -if (__this_cpu_cmpxchg (ne_fit_preload_node,NULL , pva)) { +if (this_cpu_cmpxchg (ne_fit_preload_node,NULL , pva)) {if (pva)kmem_cache_free (vmap_area_cachep, pva); } }spin_lock (&vmap_area_lock); -preempt_enable ();/* * If an allocation fails, the "vend" address is urezki@pc636:~/data/ssd/coding/linux-stable$
<snip>
Quoted 6 lines
so, we do not guarantee, instead we minimize number of allocations with
GFP_NOWAITflag. For example on my 4xCPUs i am not able to even trigger the case when CPU is not preloaded.I can test it tomorrow on my 12xCPUs to see its behavior there.
Tested it on different systems. For example on my 8xCPUs system that runs PREEMPT kernel i see only few GFP_NOWAIT allocations, i.e. it happens when we land to another CPU that was not preloaded.
I run the special test case that follows the preload pattern and path. So 20 "unbind" threads run it and each does 1000000 allocations. As a result only 3.5 times among 1000000, during splitting, CPU was not preloaded thus, GFP_NOWAIT was used to obtain an extra object.
It is obvious that slightly modified approach still minimizes allocations in atomic context, so it can happen but the number is negligible and can be ignored, i think.
--
Vlad Rezki
Quoted 10 lines · Uladzislau Rezki
Quoted 6 lines
so, we do not guarantee, instead we minimize number of allocations with
GFP_NOWAITflag. For example on my 4xCPUs i am not able to even trigger the case when CPU is not preloaded.I can test it tomorrow on my 12xCPUs to see its behavior there.
Tested it on different systems. For example on my 8xCPUs system that runs PREEMPT kernel i see only few
GFP_NOWAITallocations, i.e. it happens when we land to another CPU that was not preloaded.
Quoted 8 lines
I run the special test case that follows the preload pattern and path. So 20 "unbind" threads run it and each does 1000000 allocations. As a result only 3.5 times among 1000000, during splitting, CPU was not preloaded thus,
GFP_NOWAITwas used to obtain an extra object.It is obvious that slightly modified approach still minimizes allocations in atomic context, so it can happen but the number is negligible and can be ignored, i think.
Thanks for doing the tests. In this case I would suggest to get rid of the preempt_disable() micro optimization, since there is almost no gain in doing so. Do you send a patch? :)
Thanks,
Daniel
Uladzislau Rezki
maintainerHello, Daniel.
Quoted 16 lines · Daniel Wagner
Quoted 15 lines · Uladzislau Rezki
Quoted 6 lines
so, we do not guarantee, instead we minimize number of allocations with
GFP_NOWAITflag. For example on my 4xCPUs i am not able to even trigger the case when CPU is not preloaded.I can test it tomorrow on my 12xCPUs to see its behavior there.
Tested it on different systems. For example on my 8xCPUs system that runs PREEMPT kernel i see only few
GFP_NOWAITallocations, i.e. it happens when we land to another CPU that was not preloaded.I run the special test case that follows the preload pattern and path. So 20 "unbind" threads run it and each does 1000000 allocations. As a result only 3.5 times among 1000000, during splitting, CPU was not preloaded thus,
GFP_NOWAITwas used to obtain an extra object.
Quoted 8 lines
Quoted 3 lines
It is obvious that slightly modified approach still minimizes allocations in atomic context, so it can happen but the number is negligible and can be ignored, i think.
Thanks for doing the tests. In this case I would suggest to get rid of the
preempt_disable()micro optimization, since there is almost no gain in doing so. Do you send a patch? :)
I can do it, for sure, in case you do not mind, since it was your initiative. Otherwise you can upload v2.
--
Vlad Rezki
Quoted 2 lines · Sebastian Andrzej Siewior
If you post something that is related to
PREEMPT_RTplease keep tglx and me in Cc.
Sure, just forgot to add it this time. My email setup needs a bit more love. Sorry.