mm/slub: embed __slab_alloc to its caller
Subsystems (2)
PATCH 1/1 mm/slub: embed __slab_alloc to its caller
Since slab_alloc_node() is the only caller of __slab_alloc(), embed __slab_alloc() to its caller to save function call overhead. This will also expand the caller's code block size a bit, but hackbench tests on both host and guest didn't show a difference w/ or w/o this patch.
Also rename ___slab_alloc() to __slab_alloc().
--- mm/slub.c | 46 ++++++++++++++++------------------------------ 1 file changed, 16 insertions(+), 30 deletions(-)
mm/slub.c +16 -30
@@ -2654,10 +2654,9 @@ static inline void *get_freelist(struct kmem_cache *s, struct page *page)* we need to allocate a new slab. This is the slowest path since it involves * a call to the page allocator and the setup of a new slab. * -* Version of __slab_alloc to use when we know that interrupts are -* already disabled (which is the case for bulk allocation). +* Must be called with interrupts disabled. */ -static void *___slab_alloc (struct kmem_cache *s,gfp_t gfpflags,int node, +static void *__slab_alloc (struct kmem_cache *s,gfp_t gfpflags,int node,unsigned long addr,struct kmem_cache_cpu *c) {void *freelist;
@@ -2758,31 +2757,6 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node,return freelist; } -/* -* Another one that disabled interrupt and compensates for possible -* cpu changes by refetching the per cpu area pointer. -*/ -static void *__slab_alloc (struct kmem_cache *s,gfp_t gfpflags,int node, -unsigned long addr,struct kmem_cache_cpu *c) -{ -void *p; -unsigned long flags; - -local_irq_save (flags); -#ifdef CONFIG_PREEMPTION -/* -* We may have been preempted and rescheduled on a different -* cpu before disabling interrupts. Need to reload cpu area -* pointer. -*/ - c =this_cpu_ptr (s->cpu_slab); -#endif - - p =___slab_alloc (s, gfpflags, node, addr, c); -local_irq_restore (flags); -return p; -} -/* * If the object has been wiped upon free, make sure it's fully initialized by * zeroing out freelist pointer.
@@ -2854,7 +2828,19 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, object = c->freelist; page = c->page;if (unlikely (!object || !page || !node_match (page, node))) { +unsigned long flags; + +local_irq_save (flags); +#ifdef CONFIG_PREEMPTION +/* +* We may have been preempted and rescheduled on a different +* cpu before disabling interrupts. Need to reload cpu area +* pointer. +*/ + c =this_cpu_ptr (s->cpu_slab); +#endif object =__slab_alloc (s, gfpflags, node, addr, c); +local_irq_restore (flags); }else {void *next_object =get_freepointer_safe (s, object);
@@ -3299,7 +3285,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,* We may have removed an object from c->freelist using * the fastpath in the previous iteration; in that case, * c->tid has not been bumped yet. -* Since ___slab_alloc() may reenable interrupts while +* Since __slab_alloc() may reenable interrupts while * allocating memory, we should bump c->tid now. */ c->tid =next_tid (c->tid);
@@ -3308,7 +3294,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,* Invoking slow path likely have side-effect * of re-populating per CPU c->freelist */ - p[i] =___slab_alloc (s, flags,NUMA_NO_NODE , + p[i] =__slab_alloc (s, flags,NUMA_NO_NODE , _RET_IP_, c);if (unlikely (!p[i])) goto error;
Signature · 1 line
2.27.0
Quoted 5 lines · Abel Wu
Since
slab_alloc_node()is the only caller of__slab_alloc(), embed__slab_alloc()to its caller to save function call overhead. This will also expand the caller's code block size a bit, but hackbench tests on both host and guest didn't show a difference w/ or w/o this patch.
slab_alloc_node is an always_inline function. It is intentional that only the fast path was inlined and not the slow path.
Quoted 12 lines · Christoph Lameter
Quoted 5 lines · Abel Wu
Since
slab_alloc_node()is the only caller of__slab_alloc(), embed__slab_alloc()to its caller to save function call overhead. This will also expand the caller's code block size a bit, but hackbench tests on both host and guest didn't show a difference w/ or w/o this patch.
slab_alloc_nodeis analways_inlinefunction. It is intentional that only the fast path was inlined and not the slow path.
Oh I got it. Thanks for your excellent explanation.
Best Regards,
Abel
Quoted 10 lines · Abel Wu
Quoted 10 lines · Christoph Lameter
Quoted 5 lines · Abel Wu
Since
slab_alloc_node()is the only caller of__slab_alloc(), embed__slab_alloc()to its caller to save function call overhead. This will also expand the caller's code block size a bit, but hackbench tests on both host and guest didn't show a difference w/ or w/o this patch.
Quoted 4 lines
Quoted 2 lines
slab_alloc_nodeis analways_inlinefunction. It is intentional that only the fast path was inlined and not the slow path.Oh I got it. Thanks for your excellent explanation.
BTW, there's a script in the Linux source to nicely see the effect of such changes:
./scripts/bloat-o-meter slub.o.before mm/slub.o add/remove: 0/1 grow/shrink: 9/0 up/down: 1660/-1130 (530)
Function old new delta __slab_alloc 127 1130 +1003 __kmalloc_track_caller 877 965 +88 __kmalloc 878 966 +88 kmem_cache_alloc 778 862 +84 __kmalloc_node_track_caller 996 1080 +84 kmem_cache_alloc_node_trace 813 896 +83 kmem_cache_alloc_node 800 881 +81 kmem_cache_alloc_trace 786 862 +76 __kmalloc_node 998 1071 +73
___slab_alloc 1130 - -1130
Total: Before=57782, After=58312, chg +0.92%
And yeah, bloating all the entry points wouldn't be nice. Thanks,
Vlastimil