allow preemption in slab_alloc_node and slab_free
Subsystems (2)
PATCH 5/5 allow preemption in slab_alloc_node and slab_free
drop preempt_disable/enable in slab_alloc_node and slab_free
__slab_alloc is only called from slub.c:slab_alloc_node it runs with local irqs disabled so it can't be pushed off this CPU asynchronously, the preempt_disable/enable is thus not needed. Aside from that the later this_cpu_cmpxchg_double would catch such a migration event anyay.
slab_free:
slowpath: (if the allocation was on a different CPU) detected by
(page == c->page) c pointing to the per cpu slab, this does not need a consistent ref to tid so the slow path is safe without the preempt_disable/enable
fastpath: if allocation was on the same cpu but we got migrated between
fetching the cpu_slab and the actual push onto the free list then this_cpu_cmpxchg_double would catch this case and loop in redo. So the fast path is also safe without the preempt_disable/enable
Testing:
while : ; do ./hackbench 120 thread 10000 ; done
Time: 296.631
Time: 298.723
Time: 301.468
Time: 303.880
Time: 301.988
Time: 300.038
Time: 299.634
Time: 301.488
which seems to be a good way to stress-test slub
Impact on performance:
The change could negatively impact performance if the removal of the preempt_disable/enable would result in a significant increase of the slow path being taken or looping via goto redo - this was checked by:
static instrumentation:
an instrumentation was added to check how often the redo loop is taken the results showed that the redo loop is very rarely taken (< 1 in 10000) and is below the value with the preempt_disable/enable present. Further the slowpath to fastpath ration improves slightly (not sure if this is statistically significant though)
running slab_test.c:
the slub-benchmark from Christoph Lameter and Mathieu Desnoyers was used the only change being that asm/system.h was droped from the list of includes. The results indicate that the removal of preempt_disable/enable reduces the cycles needed slightly (though quite a few testsystems would need to be checked before this can be confirmed).
--- mm/slub.c | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-)
mm/slub.c +0 -4
@@ -2424,7 +2424,6 @@ redo:* on a different processor between the determination of the pointer * and the retrieval of the tid. */ -preempt_disable (); c =__this_cpu_ptr (s->cpu_slab);/*
@@ -2434,7 +2433,6 @@ redo:* linked list in between. */ tid = c->tid; -preempt_enable (); object = c->freelist; page = c->page;
@@ -2683,11 +2681,9 @@ redo:* data is retrieved via this pointer. If we are on the same cpu * during the cmpxchg then the free will succedd. */ -preempt_disable (); c =__this_cpu_ptr (s->cpu_slab); tid = c->tid; -preempt_enable ();if (likely (page == c->page)) {set_freepointer (s, object, c->freelist);
Signature · 1 line
1.7.2.5
Quoted 5 lines · * Nicholas Mc Guire | 2014-02-10 16:40:16 [+0100]:
__slab_allocis only called from slub.c:slab_alloc_node it runs with local irqs disabled so it can't be pushed off this CPU asynchronously, the preempt_disable/enable is thus not needed. Aside from that the laterthis_cpu_cmpxchg_doublewould catch such a migration event anyay.
Not sure what to do with this one. You do write a longer chapter why it is okay to drop the preemption disable section and that this_cpu_cmpxchg_double() would catch it. And I didn't figure out so far why need to keep preemption disabled while looking at c->tid but not at c->page.
However, it seems that Christoph Lameter found it important to add a note in the comment that this preemption disable here is important. Looking at commit 7cccd80 ("slub: tid must be retrieved from the percpu area of the current processor") it seems that Steven Rostedt run into trouble and now we have that preemption_disable() here.
So if you really get better performance and you haven't seen anything bad happen then you might want to check with Lameter & Rostedt about your patch and getting it merged upstream.
The commit I mentioned is upstream since v3.11-rc1 and I can see it in v3.8-RT tree so it looks serious.
I fail to see it in v3.2-RT, Steven, isn't this something we want there, too?
Sebastian