MM: More bulk API work
Subsystems (3)
cover MM: More bulk API work
This series contain three aspects:
1. cleanup and code sharing between SLUB and SLAB
2. implementing accelerated bulk API for SLAB allocator
3. new API kfree_bulk()
Reviewers please review the changed order of debug calls in the SLAB allocator, as they are changed to do the same as the SLUB allocator.
Patchset based on top Linus tree at commit ee9a7d2cb0cf1.
---
Jesper Dangaard Brouer (10):
slub: cleanup code for kmem cgroup support to kmem_cache_free_bulk mm/slab: move SLUB alloc hooks to common mm/slab.h
mm: fault-inject take over bootstrap kmem_cache check
slab: use slab_pre_alloc_hook in SLAB allocator shared with SLUB
mm: kmemcheck skip object if slab allocation failed
slab: use slab_post_alloc_hook in SLAB allocator shared with SLUB
slab: implement bulk alloc in SLAB allocator
slab: avoid running debug SLAB code with IRQs disabled for alloc_bulk
slab: implement bulk free in SLAB allocator
mm: new API kfree_bulk() for SLAB+SLUB allocators
include/linux/fault-inject.h | 5 +- include/linux/slab.h | 8 +++ mm/failslab.c | 11 +++- mm/kmemcheck.c | 3 + mm/slab.c | 121 +++++++++++++++++++++++++++--------------- mm/slab.h | 62 ++++++++++++++++++++++ mm/slab_common.c | 8 ++- mm/slub.c | 92 +++++++++----------------------- 8 files changed, 194 insertions(+), 116 deletions(-)
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/10 slub: cleanup code for kmem cgroup support to kmem_cache_free_bulk
This change is primarily an attempt to make it easier to realize the optimizations the compiler performs in-case CONFIG_MEMCG_KMEM is not enabled.
Performance wise, even when CONFIG_MEMCG_KMEM is compiled in, the overhead is zero. This is because, as long as no process have enabled kmem cgroups accounting, the assignment is replaced by asm-NOP operations. This is possible because memcg_kmem_enabled() uses a static_key_false() construct.
It also helps readability as it avoid accessing the p[] array like:
p[size - 1] which "expose" that the array is processed backwards inside helper function build_detached_freelist().
Lastly this also makes the code more robust, in error case like passing NULL pointers in the array. Which were previously handled before commit 033745189b1b ("slub: add missing kmem cgroup support to kmem_cache_free_bulk").
--- mm/slub.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
mm/slub.c +10 -10
@@ -2833,8 +2833,9 @@ struct detached_freelist {* synchronization primitive. Look ahead in the array is limited due * to performance reasons. */ -static int build_detached_freelist (struct kmem_cache *s,size_t size, -void **p,struct detached_freelist *df) +static inline +int build_detached_freelist (struct kmem_cache **s,size_t size, +void **p,struct detached_freelist *df) {size_t first_skipped_index =0 ;int lookahead =3 ;
@@ -2850,8 +2851,11 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,if (!object)return 0 ; +/* Support for memcg, compiler can optimize this out */ + *s =cache_from_obj (*s, object); +/* Start new detached freelist */ -set_freepointer (s, object,NULL ); +set_freepointer (*s, object,NULL ); df->page =virt_to_head_page (object); df->tail = object; df->freelist = object;
@@ -2866,7 +2870,7 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,/* df->page is always set at this point */ if (df->page ==virt_to_head_page (object)) {/* Opportunity build freelist */ -set_freepointer (s, object, df->freelist); +set_freepointer (*s, object, df->freelist); df->freelist = object; df->cnt++; p[size] =NULL ;/* mark object processed */
@@ -2885,7 +2889,6 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,return first_skipped_index; } -/* Note that interrupts must be enabled when calling this function. */ void kmem_cache_free_bulk (struct kmem_cache *orig_s,size_t size,void **p) {
@@ -2894,12 +2897,9 @@ void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p) do { struct detached_freelist df; - struct kmem_cache *s; - - /* Support for memcg */ - s = cache_from_obj(orig_s, p[size - 1]); + struct kmem_cache *s = orig_s; - size = build_detached_freelist(s, size, p, &df); + size = build_detached_freelist(&s, size, p, &df); if (unlikely(!df.page)) continue;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 01/10] slub: cleanup code for kmem cgroup support to kmem_cache_free_bulk
mm/slub.c static int build_detached_freelist(struct kmem_cache *s, size_t size, · line 2856
Quoted 3 lines · Jesper Dangaard Brouer
mm/slub.c +3
-0+/* Support for memcg, compiler can optimize this out */ + *s =cache_from_obj (*s, object); +
Well the indirection on *s presumably cannot be optimized out. And the indirection is not needed when cgroups are not active.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 01/10] slub: cleanup code for kmem cgroup support to kmem_cache_free_bulk
Quoted 8 lines · On Thu, 7 Jan 2016 09:54:24 -0600 (CST) Christoph Lameter <[email protected]> wrote:
mm/slub.c static int build_detached_freelist(struct kmem_cache *s, size_t size, · line 2856
Quoted 3 lines · Jesper Dangaard Brouer
mm/slub.c +3
-0+/* Support for memcg, compiler can optimize this out */ + *s =cache_from_obj (*s, object); +Well the indirection on *s presumably cannot be optimized out. And the indirection is not needed when cgroups are not active.
The indirection is optimized out, because build_detached_freelist is inlined (and I marked it so for readability (even-though it was getting inlined before by GCC)).
Signature · 11 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 01/10] slub: cleanup code for kmem cgroup support to kmem_cache_free_bulk
In reply to Jesper Dangaard BrouerQuoted 10 lines · Jesper Dangaard Brouer
This change is primarily an attempt to make it easier to realize the optimizations the compiler performs in-case
CONFIG_MEMCG_KMEMis not enabled.Performance wise, even when
CONFIG_MEMCG_KMEMis compiled in, the overhead is zero. This is because, as long as no process have enabled kmem cgroups accounting, the assignment is replaced by asm-NOP operations. This is possible becausememcg_kmem_enabled()uses astatic_key_false()construct.
Quoted 3 lines
It also helps readability as it avoid accessing the p[] array like:
p[size - 1] which "expose" that the array is processed backwards inside helper functionbuild_detached_freelist().
That part is cleande up but overall code doesn't looks readable to me. How about below change?
Thanks.
---------------------->8------------------
struct detached_freelist {
+ struct kmem_cache *s;
struct page *page;
void *tail;
void *freelist;
@@ -2852,8 +2853,11 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,
if (!object)
return 0;+ /* Support for memcg */
+ df->s = cache_from_obj(s, object);
+
/* Start new detached freelist */
- set_freepointer(s, object, NULL);
+ set_freepointer(df.s, object, NULL);
df->page = virt_to_head_page(object);
df->tail = object;
df->freelist = object;
@@ -2868,7 +2872,7 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,
/* df->page is always set at this point */
if (df->page == virt_to_head_page(object)) {
/* Opportunity build freelist */
- set_freepointer(s, object, df->freelist);
+ set_freepointer(df.s, object, df->freelist);
df->freelist = object;
df->cnt++;
p[size] = NULL; /* mark object processed */@@ -2889,23 +2893,19 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,
/* Note that interrupts must be enabled when calling this function. */
-void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p) +void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p)
{
if (WARN_ON(!size))
return;
do {
struct detached_freelist df;- struct kmem_cache *s;
-
- /* Support for memcg */
- s = cache_from_obj(orig_s, p[size - 1]);
size = build_detached_freelist(s, size, p, &df);
if (unlikely(!df.page))
continue;- slab_free(s, df.page, df.freelist, df.tail, df.cnt, _RET_IP_);
+ slab_free(df.s, df.page, df.freelist, df.tail, df.cnt, _RET_IP_);
} while (likely(size)); }
EXPORT_SYMBOL(kmem_cache_free_bulk);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 01/10] slub: cleanup code for kmem cgroup support to kmem_cache_free_bulk
Quoted 11 lines · On Fri, 8 Jan 2016 11:58:39 +0900 Joonsoo Kim <[email protected]> wrote:
Quoted 10 lines · Jesper Dangaard Brouer
This change is primarily an attempt to make it easier to realize the optimizations the compiler performs in-case
CONFIG_MEMCG_KMEMis not enabled.Performance wise, even when
CONFIG_MEMCG_KMEMis compiled in, the overhead is zero. This is because, as long as no process have enabled kmem cgroups accounting, the assignment is replaced by asm-NOP operations. This is possible becausememcg_kmem_enabled()uses astatic_key_false()construct.
Quoted 5 lines
Quoted 3 lines
It also helps readability as it avoid accessing the p[] array like:
p[size - 1] which "expose" that the array is processed backwards inside helper functionbuild_detached_freelist().That part is cleande up but overall code doesn't looks readable to me.
True, I also don't like my "*s" indirection, even-though it gets removed in the compiled code.
Quoted 1 line
How about below change?
Looks more C readable, but I have to verify that the compiler can still realize the optimization, of just using "s" directly, when CONFIG_MEMCG_KMEM is not activated.
Quoted 8 lines
---------------------->8------------------
structdetached_freelist{+ struct kmem_cache *s; struct page *page; void *tail; void *freelist; int cnt;}
I'll likely place "s" at another point, as 16 bytes alignment of this struct can influence performance on Intel CPUs. (e.g. freelist+cnt gets updated at almost the same time, and were 16 bytes aligned before)
mm/slub.c static int build_detached_freelist(struct kmem_cache *s, size_t size, · line 2860
Quoted 10 lines
mm/slub.c +4
-1@@ -2852,8 +2853,11 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,if (!object)return 0 ; +/* Support for memcg */ + df->s =cache_from_obj (s, object); +/* Start new detached freelist */ -set_freepointer (s, object,NULL ); +set_freepointer (df.s, object,NULL );
Not compile tested ;-) ... df->s
Quoted 25 lines
mm/slub.c +2
-3df->page =virt_to_head_page (object); df->tail = object; df->freelist = object;@@ -2868,7 +2872,7 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,/* df->page is always set at this point */ if (df->page ==virt_to_head_page (object)) {/* Opportunity build freelist */ -set_freepointer (s, object, df->freelist); +set_freepointer (df.s, object, df->freelist); df->freelist = object; df->cnt++; p[size] =NULL ;/* mark object processed */ @@ -2889,23 +2893,19 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,/* Note that interrupts must be enabled when calling this function. */ -void kmem_cache_free_bulk (struct kmem_cache *orig_s,size_t size,void **p) +void kmem_cache_free_bulk (struct kmem_cache *s,size_t size,void **p) {if (WARN_ON (!size))return ; do { struct detached_freelist df; - struct kmem_cache *s;
mm/slub.c static int build_detached_freelist(struct kmem_cache *s, size_t size, · line 2908
Quoted 10 lines
mm/slub.c +1
-4- -/* Support for memcg */ - s =cache_from_obj (orig_s, p[size -1 ]); size =build_detached_freelist (s, size, p, &df);if (unlikely (!df.page))continue ; -slab_free (s, df.page, df.freelist, df.tail, df.cnt, _RET_IP_); +slab_free (df.s, df.page, df.freelist, df.tail, df.cnt, _RET_IP_);
Argh... line will be 81 chars wide...
Review static int build_detached_freelist(struct kmem_cache *s, size_t size, · line 2911
Quoted 3 lines
(quoted diff) +0
-0} while (likely(size)); } EXPORT_SYMBOL(kmem_cache_free_bulk);
I'll try it out. Thanks for your suggestion.
Signature · 11 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/10 mm/slab: move SLUB alloc hooks to common mm/slab.h
In reply to Jesper Dangaard BrouerFirst step towards sharing alloc_hook's between SLUB and SLAB allocators. Move the SLUB allocators *_alloc_hook to the common mm/slab.h for internal slab definitions.
--- mm/slab.h | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/slub.c | 54 ----------------------------------------------------- 2 files changed, 62 insertions(+), 54 deletions(-)
mm/slab.h +62 -0
@@ -38,6 +38,10 @@ struct kmem_cache {#endif #include <linux/memcontrol.h> +#include <linux/fault-inject.h> +#include <linux/kmemcheck.h> +#include <linux/kasan.h> +#include <linux/kmemleak.h> /* * State of the slab allocator.
@@ -319,6 +323,64 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)return s; } +static inline size_t slab_ksize (const struct kmem_cache *s) +{ +#ifndef CONFIG_SLUB +return s->object_size; + +#else /* CONFIG_SLUB */ +# ifdef CONFIG_SLUB_DEBUG +/* +* Debugging requires use of the padding between object +* and whatever may come after it. +*/ +if (s->flags & (SLAB_RED_ZONE |SLAB_POISON )) +return s->object_size; +# endif +/* +* If we have the need to store the freelist pointer +* back there or track user information then we can +* only use the space before that information. +*/ +if (s->flags & (SLAB_DESTROY_BY_RCU |SLAB_STORE_USER )) +return s->inuse; +/* +* Else we can use all the padding etc for the allocation +*/ +return s->size; +#endif +} + +static inline struct kmem_cache *slab_pre_alloc_hook (struct kmem_cache *s, +gfp_t flags) +{ + flags &= gfp_allowed_mask; +lockdep_trace_alloc (flags); +might_sleep_if (gfpflags_allow_blocking (flags)); + +if (should_failslab (s->object_size, flags, s->flags)) +return NULL ; + +return memcg_kmem_get_cache (s, flags); +} + +static inline void slab_post_alloc_hook (struct kmem_cache *s,gfp_t flags, +size_t size,void **p) +{ +size_t i; + + flags &= gfp_allowed_mask; +for (i =0 ; i < size; i++) { +void *object = p[i]; + +kmemcheck_slab_alloc (s, flags, object,slab_ksize (s)); +kmemleak_alloc_recursive (object, s->object_size,1 , + s->flags, flags); +kasan_slab_alloc (s, object); + } +memcg_kmem_put_cache (s); +} +#ifndef CONFIG_SLOB /* * The slab lists for all objects.
mm/slub.c +0 -54
66 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/10 mm: fault-inject take over bootstrap kmem_cache check
In reply to Jesper Dangaard BrouerRemove the SLAB specific function slab_should_failslab(), by moving the check against fault-injection for the bootstrap slab, into the shared function should_failslab() (used by both SLAB and SLUB).
This is a step towards sharing alloc_hook's between SLUB and SLAB.
This bootstrap slab "kmem_cache" is used for allocating struct kmem_cache objects to the allocator itself.
--- include/linux/fault-inject.h | 5 ++--- mm/failslab.c | 11 ++++++++--- mm/slab.c | 12 ++---------- mm/slab.h | 2 +- 4 files changed, 13 insertions(+), 17 deletions(-)
include/linux/fault-inject.h +2 -3
@@ -62,10 +62,9 @@ static inline struct dentry *fault_create_debugfs_attr(const char *name,#endif /* CONFIG_FAULT_INJECTION */ #ifdef CONFIG_FAILSLAB -extern bool should_failslab (size_t size,gfp_t gfpflags,unsigned long flags); +extern bool should_failslab (struct kmem_cache *s,gfp_t gfpflags);#else -static inline bool should_failslab (size_t size,gfp_t gfpflags, -unsigned long flags) +static inline bool should_failslab (struct kmem_cache *s,gfp_t gfpflags) {return false; }
mm/failslab.c +8 -3
@@ -1,5 +1,6 @@#include <linux/fault-inject.h> #include <linux/slab.h> +#include "slab.h" static struct { struct fault_attr attr;
@@ -11,18 +12,22 @@ static struct { .cache_filter = false, }; -bool should_failslab (size_t size,gfp_t gfpflags,unsigned long cache_flags) +bool should_failslab (struct kmem_cache *s,gfp_t gfpflags) { +/* No fault-injection for bootstrap cache */ +if (unlikely (s == kmem_cache)) +return false; +if (gfpflags & __GFP_NOFAIL)return false;if (failslab.ignore_gfp_reclaim && (gfpflags & __GFP_RECLAIM))return false; -if (failslab.cache_filter && !(cache_flags &SLAB_FAILSLAB )) +if (failslab.cache_filter && !(s->flags &SLAB_FAILSLAB ))return false; -return should_fail (&failslab.attr, size); +return should_fail (&failslab.attr, s->object_size); }static int __init setup_failslab (char *str)
mm/slab.c +2 -10
@@ -2917,14 +2917,6 @@ static void *cache_alloc_debugcheck_after(struct kmem_cache *cachep,#define cache_alloc_debugcheck_after (a,b,objp,d) (objp)#endif -static bool slab_should_failslab (struct kmem_cache *cachep,gfp_t flags) -{ -if (unlikely (cachep == kmem_cache)) -return false; - -return should_failslab (cachep->object_size, flags, cachep->flags); -} -static inline void *____cache_alloc (struct kmem_cache *cachep,gfp_t flags) {void *objp;
@@ -3152,7 +3144,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,lockdep_trace_alloc (flags); -if (slab_should_failslab (cachep, flags)) +if (should_failslab (cachep, flags))return NULL ; cachep =memcg_kmem_get_cache (cachep, flags);
@@ -3240,7 +3232,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)lockdep_trace_alloc (flags); -if (slab_should_failslab (cachep, flags)) +if (should_failslab (cachep, flags))return NULL ; cachep =memcg_kmem_get_cache (cachep, flags);
mm/slab.h +1 -1
@@ -358,7 +358,7 @@ static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s,lockdep_trace_alloc (flags);might_sleep_if (gfpflags_allow_blocking (flags)); -if (should_failslab (s->object_size, flags, s->flags)) +if (should_failslab (s, flags))return NULL ;return memcg_kmem_get_cache (s, flags);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/10 slab: use slab_pre_alloc_hook in SLAB allocator shared with SLUB
In reply to Jesper Dangaard BrouerDedublicate code in SLAB allocator functions slab_alloc() and slab_alloc_node() by using the slab_pre_alloc_hook() call, which is now shared between SLUB and SLAB.
--- mm/slab.c | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-)
mm/slab.c +4 -14
@@ -3140,15 +3140,10 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,void *ptr;int slab_node =numa_mem_id (); - flags &= gfp_allowed_mask; - -lockdep_trace_alloc (flags); - -if (should_failslab (cachep, flags)) + cachep =slab_pre_alloc_hook (cachep, flags); +if (!cachep)return NULL ; - cachep =memcg_kmem_get_cache (cachep, flags); -cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags);
@@ -3228,15 +3223,10 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)unsigned long save_flags;void *objp; - flags &= gfp_allowed_mask; - -lockdep_trace_alloc (flags); - -if (should_failslab (cachep, flags)) + cachep =slab_pre_alloc_hook (cachep, flags); +if (!cachep)return NULL ; - cachep =memcg_kmem_get_cache (cachep, flags); -cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags); objp =__do_cache_alloc (cachep, flags);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 04/10] slab: use slab_pre_alloc_hook in SLAB allocator shared with SLUB
Quoted 15 lines · Jesper Dangaard Brouer
Dedublicate code in SLAB allocator functions
slab_alloc()andslab_alloc_node()by using theslab_pre_alloc_hook()call, which is now shared between SLUB and SLAB.Signed-off-by Jesper Dangaard Brouer <[email protected]>--- mm/slab.c | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-)mm/slab.c +0
-0@@ -3140,15 +3140,10 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,void *ptr;
mm/slab.c slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, · line 3145
Quoted 10 lines
mm/slab.c +2
-5int slab_node =numa_mem_id (); - flags &= gfp_allowed_mask; - -lockdep_trace_alloc (flags); - -if (should_failslab (cachep, flags)) + cachep =slab_pre_alloc_hook (cachep, flags); +if (!cachep)return NULL ;
How about adding unlikely here?
Quoted 8 lines
mm/slab.c +0
-2- cachep =memcg_kmem_get_cache (cachep, flags); -cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags);@@ -3228,15 +3223,10 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)unsigned long save_flags;
mm/slab.c slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) · line 3228
Quoted 10 lines
mm/slab.c +2
-5void *objp; - flags &= gfp_allowed_mask; - -lockdep_trace_alloc (flags); - -if (should_failslab (cachep, flags)) + cachep =slab_pre_alloc_hook (cachep, flags); +if (!cachep)return NULL ;
Dito.
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/10 mm: kmemcheck skip object if slab allocation failed
In reply to Jesper Dangaard BrouerIn the SLAB allocator kmemcheck_slab_alloc() is guarded against being called in case the object is NULL. In SLUB allocator this NULL pointer invocation can happen, which seems like an oversight.
Move the NULL pointer check into kmemcheck code (kmemcheck_slab_alloc) so the check gets moved out of the fastpath, when not compiled with CONFIG_KMEMCHECK.
This is a step towards sharing post_alloc_hook between SLUB and SLAB, because slab_post_alloc_hook() does not perform this check before calling kmemcheck_slab_alloc().
--- mm/kmemcheck.c | 3 +++ 1 file changed, 3 insertions(+)
mm/kmemcheck.c +3 -0
@@ -60,6 +60,9 @@ void kmemcheck_free_shadow(struct page *page, int order)void kmemcheck_slab_alloc (struct kmem_cache *s,gfp_t gfpflags,void *object,size_t size) { +if (unlikely (!object))/* Skip object if allocation failed */ +return ; +/* * Has already been memset(), which initializes the shadow for us * as well.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/10 slab: use slab_post_alloc_hook in SLAB allocator shared with SLUB
In reply to Jesper Dangaard BrouerReviewers notice that the order in slab_post_alloc_hook() of kmemcheck_slab_alloc() and kmemleak_alloc_recursive() gets swapped compared to slab.c / SLAB allocator.
Also notice memset now occurs before calling kmemcheck_slab_alloc() and kmemleak_alloc_recursive().
I assume this reordering of kmemcheck, kmemleak and memset is okay because this is the order they are used by the SLUB allocator.
This patch completes the sharing of alloc_hook's between SLUB and SLAB.
--- mm/slab.c | 22 ++++++---------------- 1 file changed, 6 insertions(+), 16 deletions(-)
mm/slab.c +6 -16
@@ -3172,16 +3172,11 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, out:local_irq_restore (save_flags); ptr =cache_alloc_debugcheck_after (cachep, flags, ptr, caller); -kmemleak_alloc_recursive (ptr, cachep->object_size,1 , cachep->flags, - flags); -if (likely (ptr)) { -kmemcheck_slab_alloc (cachep, flags, ptr, cachep->object_size); -if (unlikely (flags & __GFP_ZERO)) -memset (ptr,0 , cachep->object_size); - } +if (unlikely (flags & __GFP_ZERO) && ptr) +memset (ptr,0 , cachep->object_size); -memcg_kmem_put_cache (cachep); +slab_post_alloc_hook (cachep, flags,1 , &ptr);return ptr; }
@@ -3232,17 +3227,12 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp =__do_cache_alloc (cachep, flags);local_irq_restore (save_flags); objp =cache_alloc_debugcheck_after (cachep, flags, objp, caller); -kmemleak_alloc_recursive (objp, cachep->object_size,1 , cachep->flags, - flags);prefetchw (objp); -if (likely (objp)) { -kmemcheck_slab_alloc (cachep, flags, objp, cachep->object_size); -if (unlikely (flags & __GFP_ZERO)) -memset (objp,0 , cachep->object_size); - } +if (unlikely (flags & __GFP_ZERO) && objp) +memset (objp,0 , cachep->object_size); -memcg_kmem_put_cache (cachep); +slab_post_alloc_hook (cachep, flags,1 , &objp);return objp; }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 7/10 slab: implement bulk alloc in SLAB allocator
In reply to Jesper Dangaard BrouerThis patch implements the alloc side of bulk API for the SLAB allocator.
Further optimization are still possible by changing the call to __do_cache_alloc() into something that can return multiple objects. This optimization is left for later, given end results already show in the area of 80% speedup.
--- mm/slab.c | 37 +++++++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-)
mm/slab.c +35 -2
@@ -3392,9 +3392,42 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p)EXPORT_SYMBOL (kmem_cache_free_bulk);int kmem_cache_alloc_bulk (struct kmem_cache *s,gfp_t flags,size_t size, -void **p) +void **p) { -return __kmem_cache_alloc_bulk (s, flags, size, p); +size_t i; + + s =slab_pre_alloc_hook (s, flags); +if (!s) +return 0 ; + +cache_alloc_debugcheck_before (s, flags); + +local_irq_disable (); +for (i =0 ; i < size; i++) { +void *objp =__do_cache_alloc (s, flags); + +/* this call could be done outside IRQ disabled section */ + objp =cache_alloc_debugcheck_after (s, flags, objp, _RET_IP_); + +if (unlikely (!objp)) + goto error; + p[i] = objp; + } +local_irq_enable (); + +/* Clear memory outside IRQ disabled section */ +if (unlikely (flags & __GFP_ZERO)) +for (i =0 ; i < size; i++) +memset (p[i],0 , s->object_size); + +slab_post_alloc_hook (s, flags, size, p); +/* FIXME: Trace call missing. Christoph would like a bulk variant */ +return size; +error: +local_irq_enable (); +slab_post_alloc_hook (s, flags, i, p); +__kmem_cache_free_bulk (s, i, p); +return 0 ; }EXPORT_SYMBOL (kmem_cache_alloc_bulk);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 8/10 slab: avoid running debug SLAB code with IRQs disabled for alloc_bulk
In reply to Jesper Dangaard BrouerMove the call to cache_alloc_debugcheck_after() outside the IRQ disabled section in kmem_cache_alloc_bulk().
When CONFIG_DEBUG_SLAB is disabled the compiler should remove this code.
--- mm/slab.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
mm/slab.c +13 -3
@@ -3391,6 +3391,16 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p) }EXPORT_SYMBOL (kmem_cache_free_bulk); +static __always_inline void +cache_alloc_debugcheck_after_bulk (struct kmem_cache *s,gfp_t flags, +size_t size,void **p,unsigned long caller) +{ +size_t i; + +for (i =0 ; i < size; i++) + p[i] =cache_alloc_debugcheck_after (s, flags, p[i], caller); +} +int kmem_cache_alloc_bulk (struct kmem_cache *s,gfp_t flags,size_t size,void **p) {
@@ -3406,15 +3416,14 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,for (i =0 ; i < size; i++) {void *objp =__do_cache_alloc (s, flags); -/* this call could be done outside IRQ disabled section */ - objp =cache_alloc_debugcheck_after (s, flags, objp, _RET_IP_); -if (unlikely (!objp)) goto error; p[i] = objp; }local_irq_enable (); +cache_alloc_debugcheck_after_bulk (s, flags, size, p, _RET_IP_); +/* Clear memory outside IRQ disabled section */ if (unlikely (flags & __GFP_ZERO))for (i =0 ; i < size; i++)
@@ -3425,6 +3434,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,return size; error:local_irq_enable (); +cache_alloc_debugcheck_after_bulk (s, flags, i, p, _RET_IP_);slab_post_alloc_hook (s, flags, i, p);__kmem_cache_free_bulk (s, i, p);return 0 ;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 9/10 slab: implement bulk free in SLAB allocator
In reply to Jesper Dangaard BrouerThis patch implements the free side of bulk API for the SLAB allocator kmem_cache_free_bulk(), and concludes the implementation of optimized bulk API for SLAB allocator.
Benchmarked[1] cost of alloc+free (obj size 256 bytes) on CPU i7-4790K @ 4.00GHz, with no debug options, no PREEMPT and CONFIG_MEMCG_KMEM=y but no active user of kmemcg.
SLAB single alloc+free cost: 87 cycles(tsc) 21.814 ns with this optimized config.
bulk- Current fallback - optimized SLAB bulk
1 - 102 cycles(tsc) 25.747 ns - 41 cycles(tsc) 10.490 ns - improved 59.8%
2 - 94 cycles(tsc) 23.546 ns - 26 cycles(tsc) 6.567 ns - improved 72.3% 3 - 92 cycles(tsc) 23.127 ns - 20 cycles(tsc) 5.244 ns - improved 78.3% 4 - 90 cycles(tsc) 22.663 ns - 18 cycles(tsc) 4.588 ns - improved 80.0% 8 - 88 cycles(tsc) 22.242 ns - 14 cycles(tsc) 3.656 ns - improved 84.1% 16 - 88 cycles(tsc) 22.010 ns - 13 cycles(tsc) 3.480 ns - improved 85.2% 30 - 89 cycles(tsc) 22.305 ns - 13 cycles(tsc) 3.303 ns - improved 85.4% 32 - 89 cycles(tsc) 22.277 ns - 13 cycles(tsc) 3.309 ns - improved 85.4% 34 - 88 cycles(tsc) 22.246 ns - 13 cycles(tsc) 3.294 ns - improved 85.2% 48 - 88 cycles(tsc) 22.121 ns - 13 cycles(tsc) 3.492 ns - improved 85.2% 64 - 88 cycles(tsc) 22.052 ns - 13 cycles(tsc) 3.411 ns - improved 85.2% 128 - 89 cycles(tsc) 22.452 ns - 15 cycles(tsc) 3.841 ns - improved 83.1% 158 - 89 cycles(tsc) 22.403 ns - 14 cycles(tsc) 3.746 ns - improved 84.3% 250 - 91 cycles(tsc) 22.775 ns - 16 cycles(tsc) 4.111 ns - improved 82.4%
Notice it is not recommended to do very large bulk operation with this bulk API, because local IRQs are disabled in this period.
[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test01.c
--- mm/slab.c | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-)
mm/slab.c +23 -6
@@ -3385,12 +3385,6 @@ void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) }EXPORT_SYMBOL (kmem_cache_alloc); -void kmem_cache_free_bulk (struct kmem_cache *s,size_t size,void **p) -{ -__kmem_cache_free_bulk (s, size, p); -} -EXPORT_SYMBOL (kmem_cache_free_bulk); - static __always_inline voidcache_alloc_debugcheck_after_bulk (struct kmem_cache *s,gfp_t flags,size_t size,void **p,unsigned long caller)
@@ -3584,6 +3578,29 @@ void kmem_cache_free(struct kmem_cache *cachep, void *objp) }EXPORT_SYMBOL (kmem_cache_free); +void kmem_cache_free_bulk (struct kmem_cache *orig_s,size_t size,void **p) +{ +struct kmem_cache *s; +size_t i; + +local_irq_disable (); +for (i =0 ; i < size; i++) { +void *objp = p[i]; + + s =cache_from_obj (orig_s, objp); + +debug_check_no_locks_freed (objp, s->object_size); +if (!(s->flags &SLAB_DEBUG_OBJECTS )) +debug_check_no_obj_freed (objp, s->object_size); + +__cache_free (s, objp, _RET_IP_); + } +local_irq_enable (); + +/* FIXME: add tracing */ +} +EXPORT_SYMBOL (kmem_cache_free_bulk); +/** * kfree - free previously allocated memory * @objp: pointer returned by kmalloc.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 10/10 mm: new API kfree_bulk() for SLAB+SLUB allocators
In reply to Jesper Dangaard BrouerThis patch introduce a new API call kfree_bulk() for bulk freeing memory objects not bound to a single kmem_cache.
Christoph pointed out that it is possible to implement freeing of objects, without knowing the kmem_cache pointer as that information is available from the object's page->slab_cache. Proposing to remove the kmem_cache argument from the bulk free API.
Jesper demonstrated that these extra steps per object comes at a performance cost. It is only in the case CONFIG_MEMCG_KMEM is compiled in and activated runtime that these steps are done anyhow. The extra cost is most visible for SLAB allocator, because the SLUB allocator does the page lookup (virt_to_head_page()) anyhow.
Thus, the conclusion was to keep the kmem_cache free bulk API with a kmem_cache pointer, but we can still implement a kfree_bulk() API fairly easily. Simply by handling if kmem_cache_free_bulk() gets called with a kmem_cache NULL pointer.
This does increase the code size a bit, but implementing a separate kfree_bulk() call would likely increase code size even more.
Below benchmarks cost of alloc+free (obj size 256 bytes) on CPU i7-4790K @ 4.00GHz, no PREEMPT and CONFIG_MEMCG_KMEM=y.
Code size increase for SLAB:
add/remove: 0/0 grow/shrink: 1/0 up/down: 74/0 (74)
function old new delta kmem_cache_free_bulk 660 734 +74
SLAB fastpath: 85 cycles(tsc) 21.468 ns (step:0)
sz - fallback - kmem_cache_free_bulk - kfree_bulk
1 - 101 cycles 25.291 ns - 41 cycles 10.499 ns - 130 cycles 32.522 ns 2 - 95 cycles 23.964 ns - 26 cycles 6.558 ns - 56 cycles 14.134 ns 3 - 93 cycles 23.281 ns - 20 cycles 5.244 ns - 41 cycles 10.393 ns 4 - 92 cycles 23.123 ns - 18 cycles 4.589 ns - 26 cycles 6.612 ns 8 - 90 cycles 22.696 ns - 24 cycles 6.211 ns - 32 cycles 8.175 ns 16 - 108 cycles 27.175 ns - 13 cycles 3.418 ns - 21 cycles 5.480 ns 30 - 90 cycles 22.708 ns - 14 cycles 3.667 ns - 20 cycles 5.222 ns 32 - 90 cycles 22.687 ns - 13 cycles 3.337 ns - 20 cycles 5.170 ns 34 - 90 cycles 22.699 ns - 14 cycles 3.622 ns - 21 cycles 5.269 ns 48 - 90 cycles 22.585 ns - 14 cycles 3.525 ns - 21 cycles 5.261 ns 64 - 90 cycles 22.523 ns - 13 cycles 3.440 ns - 20 cycles 5.190 ns 128 - 91 cycles 22.962 ns - 15 cycles 3.883 ns - 22 cycles 5.622 ns 158 - 91 cycles 22.877 ns - 15 cycles 3.770 ns - 22 cycles 5.582 ns 250 - 93 cycles 23.282 ns - 16 cycles 4.133 ns - 24 cycles 6.047 ns
SLAB when enabling MEMCG_KMEM runtime:
- kmemcg fastpath: 134 cycles(tsc) 33.514 ns (step:0)
1 - 146 cycles 36.634 ns - 66 cycles 16.705 ns - 67 cycles 16.793 ns 2 - 137 cycles 34.452 ns - 41 cycles 10.326 ns - 42 cycles 10.736 ns 3 - 135 cycles 33.856 ns - 34 cycles 8.545 ns - 36 cycles 9.049 ns 4 - 132 cycles 33.232 ns - 33 cycles 8.306 ns - 29 cycles 7.368 ns 8 - 134 cycles 33.612 ns - 24 cycles 6.250 ns - 24 cycles 6.130 ns 16 - 131 cycles 33.003 ns - 23 cycles 5.981 ns - 22 cycles 5.552 ns 30 - 131 cycles 32.920 ns - 21 cycles 5.499 ns - 21 cycles 5.397 ns 32 - 131 cycles 32.861 ns - 21 cycles 5.482 ns - 21 cycles 5.301 ns 34 - 131 cycles 32.837 ns - 21 cycles 5.461 ns - 20 cycles 5.236 ns 48 - 130 cycles 32.725 ns - 23 cycles 5.878 ns - 21 cycles 5.367 ns 64 - 130 cycles 32.625 ns - 21 cycles 5.374 ns - 21 cycles 5.251 ns 128 - 132 cycles 33.048 ns - 22 cycles 5.725 ns - 22 cycles 5.662 ns 158 - 132 cycles 33.235 ns - 22 cycles 5.641 ns - 22 cycles 5.579 ns 250 - 134 cycles 33.557 ns - 24 cycles 6.035 ns - 23 cycles 5.934 ns
Code size increase for SLUB:
function old new delta kmem_cache_free_bulk 717 799 +82
SLUB benchmark:
SLUB fastpath: 46 cycles(tsc) 11.691 ns (step:0)
sz - fallback - kmem_cache_free_bulk - kfree_bulk
1 - 61 cycles 15.486 ns - 53 cycles 13.364 ns - 57 cycles 14.464 ns 2 - 54 cycles 13.703 ns - 32 cycles 8.110 ns - 33 cycles 8.482 ns 3 - 53 cycles 13.272 ns - 25 cycles 6.362 ns - 27 cycles 6.947 ns 4 - 51 cycles 12.994 ns - 24 cycles 6.087 ns - 24 cycles 6.078 ns 8 - 50 cycles 12.576 ns - 21 cycles 5.354 ns - 22 cycles 5.513 ns 16 - 49 cycles 12.368 ns - 20 cycles 5.054 ns - 20 cycles 5.042 ns 30 - 49 cycles 12.273 ns - 18 cycles 4.748 ns - 19 cycles 4.758 ns 32 - 49 cycles 12.401 ns - 19 cycles 4.821 ns - 19 cycles 4.810 ns 34 - 98 cycles 24.519 ns - 24 cycles 6.154 ns - 24 cycles 6.157 ns 48 - 83 cycles 20.833 ns - 21 cycles 5.446 ns - 21 cycles 5.429 ns 64 - 75 cycles 18.891 ns - 20 cycles 5.247 ns - 20 cycles 5.238 ns 128 - 93 cycles 23.271 ns - 27 cycles 6.856 ns - 27 cycles 6.823 ns 158 - 102 cycles 25.581 ns - 30 cycles 7.714 ns - 30 cycles 7.695 ns 250 - 107 cycles 26.917 ns - 38 cycles 9.514 ns - 38 cycles 9.506 ns
SLUB when enabling MEMCG_KMEM runtime:
- kmemcg fastpath: 71 cycles(tsc) 17.897 ns (step:0)
1 - 85 cycles 21.484 ns - 78 cycles 19.569 ns - 75 cycles 18.938 ns 2 - 81 cycles 20.363 ns - 45 cycles 11.258 ns - 44 cycles 11.076 ns 3 - 78 cycles 19.709 ns - 33 cycles 8.354 ns - 32 cycles 8.044 ns 4 - 77 cycles 19.430 ns - 28 cycles 7.216 ns - 28 cycles 7.003 ns 8 - 101 cycles 25.288 ns - 23 cycles 5.849 ns - 23 cycles 5.787 ns 16 - 76 cycles 19.148 ns - 20 cycles 5.162 ns - 20 cycles 5.081 ns 30 - 76 cycles 19.067 ns - 19 cycles 4.868 ns - 19 cycles 4.821 ns 32 - 76 cycles 19.052 ns - 19 cycles 4.857 ns - 19 cycles 4.815 ns 34 - 121 cycles 30.291 ns - 25 cycles 6.333 ns - 25 cycles 6.268 ns 48 - 108 cycles 27.111 ns - 21 cycles 5.498 ns - 21 cycles 5.458 ns 64 - 100 cycles 25.164 ns - 20 cycles 5.242 ns - 20 cycles 5.229 ns 128 - 155 cycles 38.976 ns - 27 cycles 6.886 ns - 27 cycles 6.892 ns 158 - 132 cycles 33.034 ns - 30 cycles 7.711 ns - 30 cycles 7.728 ns 250 - 130 cycles 32.612 ns - 38 cycles 9.560 ns - 38 cycles 9.549 ns
--- include/linux/slab.h | 8 ++++++++ mm/slab.c | 5 ++++- mm/slab_common.c | 8 ++++++-- mm/slub.c | 22 +++++++++++++++++++--- 4 files changed, 37 insertions(+), 6 deletions(-)
include/linux/slab.h +8 -0
@@ -318,6 +318,14 @@ void kmem_cache_free(struct kmem_cache *, void *);void kmem_cache_free_bulk (struct kmem_cache *,size_t ,void **);int kmem_cache_alloc_bulk (struct kmem_cache *,gfp_t ,size_t ,void **); +static __always_inline void kfree_bulk(size_t size, void **p) +{ +/* Reusing call to kmem_cache_free_bulk() allow kfree_bulk to +* use same code icache +*/ +kmem_cache_free_bulk (NULL , size, p); +} +#ifdef CONFIG_NUMA void *__kmalloc_node (size_t size,gfp_t flags,int node) __assume_kmalloc_alignment;void *kmem_cache_alloc_node (struct kmem_cache *,gfp_t flags,int node) __assume_slab_alignment;
mm/slab.c +4 -1
@@ -3587,7 +3587,10 @@ void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p)for (i =0 ; i < size; i++) {void *objp = p[i]; - s =cache_from_obj (orig_s, objp); +if (!orig_s)/* called via kfree_bulk */ + s =virt_to_cache (objp); +else + s =cache_from_obj (orig_s, objp);debug_check_no_locks_freed (objp, s->object_size);if (!(s->flags &SLAB_DEBUG_OBJECTS ))
mm/slab_common.c +6 -2
@@ -108,8 +108,12 @@ void __kmem_cache_free_bulk(struct kmem_cache *s, size_t nr, void **p) {size_t i; -for (i =0 ; i < nr; i++) -kmem_cache_free (s, p[i]); +for (i =0 ; i < nr; i++) { +if (s) +kmem_cache_free (s, p[i]); +else +kfree (p[i]); + } } int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr,
mm/slub.c +19 -3
@@ -2786,23 +2786,39 @@ int build_detached_freelist(struct kmem_cache **s, size_t size,size_t first_skipped_index =0 ;int lookahead =3 ;void *object; +struct page *page;/* Always re-init detached_freelist */ df->page =NULL ;do { object = p[--size]; +/* Do we need !ZERO_OR_NULL_PTR(object) here? (for kfree) */ }while (!object && size);if (!object)return 0 ; -/* Support for memcg, compiler can optimize this out */ - *s =cache_from_obj (*s, object); + page =virt_to_head_page (object); +if (!*s) { +/* Handle kalloc'ed objects */ +if (unlikely (!PageSlab (page))) { +BUG_ON (!PageCompound (page)); +kfree_hook (object); +__free_kmem_pages (page,compound_order (page)); + p[size] =NULL ;/* mark object processed */ +return size; + } +/* Derive kmem_cache from object */ + *s = page->slab_cache; + }else { +/* Support for memcg, compiler can optimize this out */ + *s =cache_from_obj (*s, object); + }/* Start new detached freelist */ + df->page = page;set_freepointer (*s, object,NULL ); - df->page =virt_to_head_page (object); df->tail = object; df->freelist = object; p[size] =NULL ;/* mark object processed */
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 10/10] mm: new API kfree_bulk() for SLAB+SLUB allocators
Quoted 27 lines · Jesper Dangaard Brouer
This patch introduce a new API call
kfree_bulk()for bulk freeing memory objects not bound to a singlekmem_cache.Christoph pointed out that it is possible to implement freeing of objects, without knowing the
kmem_cachepointer as that information is available from the object'spage->slab_cache. Proposing to remove thekmem_cacheargument from the bulk free API.Jesper demonstrated that these extra steps per object comes at a performance cost. It is only in the case
CONFIG_MEMCG_KMEMis compiled in and activated runtime that these steps are done anyhow. The extra cost is most visible for SLAB allocator, because the SLUB allocator does the page lookup (virt_to_head_page()) anyhow.Thus, the conclusion was to keep the
kmem_cachefree bulk API with akmem_cachepointer, but we can still implement akfree_bulk()API fairly easily. Simply by handling ifkmem_cache_free_bulk()gets called with akmem_cacheNULL pointer.This does increase the code size a bit, but implementing a separate
kfree_bulk()call would likely increase code size even more.Below benchmarks cost of alloc+free (obj size 256 bytes) on CPU i7-4790K @ 4.00GHz, no PREEMPT and
CONFIG_MEMCG_KMEM=y.Code size increase for SLAB:
Quoted 7 lines
add/remove: 0/0 grow/shrink: 1/0 up/down: 74/0 (74)
function old new delta kmem_cache_free_bulk 660 734 +74SLAB fastpath: 85 cycles(tsc) 21.468 ns (step:0)
sz - fallback -kmem_cache_free_bulk-kfree_bulk
1 - 101 cycles 25.291 ns - 41 cycles 10.499 ns - 130 cycles 32.522 ns
This looks experimental error. Why does kfree_bulk() takes more time than fallback?
Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 10/10] mm: new API kfree_bulk() for SLAB+SLUB allocators
Quoted 32 lines · On Fri, 8 Jan 2016 12:03:48 +0900 Joonsoo Kim <[email protected]> wrote:
Quoted 31 lines · Jesper Dangaard Brouer
This patch introduce a new API call
kfree_bulk()for bulk freeing memory objects not bound to a singlekmem_cache.Christoph pointed out that it is possible to implement freeing of objects, without knowing the
kmem_cachepointer as that information is available from the object'spage->slab_cache. Proposing to remove thekmem_cacheargument from the bulk free API.Jesper demonstrated that these extra steps per object comes at a performance cost. It is only in the case
CONFIG_MEMCG_KMEMis compiled in and activated runtime that these steps are done anyhow. The extra cost is most visible for SLAB allocator, because the SLUB allocator does the page lookup (virt_to_head_page()) anyhow.Thus, the conclusion was to keep the
kmem_cachefree bulk API with akmem_cachepointer, but we can still implement akfree_bulk()API fairly easily. Simply by handling ifkmem_cache_free_bulk()gets called with akmem_cacheNULL pointer.This does increase the code size a bit, but implementing a separate
kfree_bulk()call would likely increase code size even more.Below benchmarks cost of alloc+free (obj size 256 bytes) on CPU i7-4790K @ 4.00GHz, no PREEMPT and
CONFIG_MEMCG_KMEM=y.Code size increase for SLAB:
add/remove: 0/0 grow/shrink: 1/0 up/down: 74/0 (74)
function old new delta kmem_cache_free_bulk 660 734 +74
Quoted 6 lines
Quoted 3 lines
SLAB fastpath: 85 cycles(tsc) 21.468 ns (step:0)
sz - fallback -kmem_cache_free_bulk-kfree_bulk
1 - 101 cycles 25.291 ns - 41 cycles 10.499 ns - 130 cycles 32.522 nsThis looks experimental error. Why does
kfree_bulk()takes more time than fallback?
This does look like an experimental error. Sometimes instabilities occurs, when slab_caches gets merged, but I tried to counter that by using boot param slab_nomerge.
In the case for SLAB kfree_bulk() single object, then it can be slower than the fallback, because it will likely always hit a branch mispredict for the kfree case (which is okay, as that is not the case we optimize for, single obj free).
Signature · 11 lines
Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat Author of http://www.iptv-analyzer.org LinkedIn: http://www.linkedin.com/in/brouer -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · Jesper Dangaard Brouer
This series contain three aspects:
1. cleanup and code sharing between SLUB and SLAB
2. implementing accelerated bulk API for SLAB allocator
3. new APIkfree_bulk()
FWIW, looks ok to me from a quick patch read-through. Nothing raises my hackles like happened with the old slab work.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
v2 cover MM: More bulk API work
In reply to Jesper Dangaard BrouerThis series contain three aspects:
1. cleanup and code sharing between SLUB and SLAB
2. implementing accelerated bulk API for SLAB allocator
3. new API kfree_bulk()
Reviewers please review the changed order of debug calls in the SLAB allocator, as they are changed to do the same as the SLUB allocator.
Patchset based on top Linus tree at commit afd2ff9b7e1b ("Linux 4.4").
Test module for exercising the new kfree_bulk() API is avail here:
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test05_kfree_bulk.c
---
Jesper Dangaard Brouer (11):
slub: cleanup code for kmem cgroup support to kmem_cache_free_bulk mm/slab: move SLUB alloc hooks to common mm/slab.h
mm: fault-inject take over bootstrap kmem_cache check
slab: use slab_pre_alloc_hook in SLAB allocator shared with SLUB
mm: kmemcheck skip object if slab allocation failed
slab: use slab_post_alloc_hook in SLAB allocator shared with SLUB
slab: implement bulk alloc in SLAB allocator
slab: avoid running debug SLAB code with IRQs disabled for alloc_bulk
slab: implement bulk free in SLAB allocator
mm: new API kfree_bulk() for SLAB+SLUB allocators
mm: fix some spelling
include/linux/fault-inject.h | 5 +- include/linux/memcontrol.h | 2 - include/linux/slab.h | 11 +++- mm/failslab.c | 11 +++- mm/kmemcheck.c | 3 + mm/slab.c | 121 +++++++++++++++++++++++++++--------------- mm/slab.h | 64 ++++++++++++++++++++++ mm/slab_common.c | 8 ++- mm/slub.c | 93 +++++++++----------------------- 9 files changed, 198 insertions(+), 120 deletions(-)
--
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 1/11 slub: cleanup code for kmem cgroup support to kmem_cache_free_bulk
This change is primarily an attempt to make it easier to realize the optimizations the compiler performs in-case CONFIG_MEMCG_KMEM is not enabled.
Performance wise, even when CONFIG_MEMCG_KMEM is compiled in, the overhead is zero. This is because, as long as no process have enabled kmem cgroups accounting, the assignment is replaced by asm-NOP operations. This is possible because memcg_kmem_enabled() uses a static_key_false() construct.
It also helps readability as it avoid accessing the p[] array like:
p[size - 1] which "expose" that the array is processed backwards inside helper function build_detached_freelist().
Lastly this also makes the code more robust, in error case like passing NULL pointers in the array. Which were previously handled before commit 033745189b1b ("slub: add missing kmem cgroup support to kmem_cache_free_bulk").
---
V2: used Joonsoo Kim's suggestion to store "s" in struct detached_freelist.
- Suggested-by: Joonsoo Kim <[email protected]>
- Verified ASM code generated is still optimal, with different ifdef config's
mm/slub.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
mm/slub.c +11 -11
@@ -2819,6 +2819,7 @@ struct detached_freelist {void *tail;void *freelist;int cnt; +struct kmem_cache *s; };/*
@@ -2833,8 +2834,9 @@ struct detached_freelist {* synchronization primitive. Look ahead in the array is limited due * to performance reasons. */ -static int build_detached_freelist (struct kmem_cache *s,size_t size, -void **p,struct detached_freelist *df) +static inline +int build_detached_freelist (struct kmem_cache *s,size_t size, +void **p,struct detached_freelist *df) {size_t first_skipped_index =0 ;int lookahead =3 ;
@@ -2850,8 +2852,11 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,if (!object)return 0 ; +/* Support for memcg, compiler can optimize this out */ + df->s =cache_from_obj (s, object); +/* Start new detached freelist */ -set_freepointer (s, object,NULL ); +set_freepointer (df->s, object,NULL ); df->page =virt_to_head_page (object); df->tail = object; df->freelist = object;
@@ -2866,7 +2871,7 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,/* df->page is always set at this point */ if (df->page ==virt_to_head_page (object)) {/* Opportunity build freelist */ -set_freepointer (s, object, df->freelist); +set_freepointer (df->s, object, df->freelist); df->freelist = object; df->cnt++; p[size] =NULL ;/* mark object processed */
@@ -2885,25 +2890,20 @@ static int build_detached_freelist(struct kmem_cache *s, size_t size,return first_skipped_index; } -/* Note that interrupts must be enabled when calling this function. */ -void kmem_cache_free_bulk (struct kmem_cache *orig_s,size_t size,void **p) +void kmem_cache_free_bulk (struct kmem_cache *s,size_t size,void **p) {if (WARN_ON (!size))return ;do {struct detached_freelist df; -struct kmem_cache *s; - -/* Support for memcg */ - s =cache_from_obj (orig_s, p[size -1 ]); size =build_detached_freelist (s, size, p, &df);if (unlikely (!df.page))continue ; -slab_free (s, df.page, df.freelist, df.tail, df.cnt, _RET_IP_); +slab_free (df.s, df.page, df.freelist, df.tail, df.cnt,_RET_IP_); }while (likely (size)); }EXPORT_SYMBOL (kmem_cache_free_bulk);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 2/11 mm/slab: move SLUB alloc hooks to common mm/slab.h
In reply to Jesper Dangaard BrouerFirst step towards sharing alloc_hook's between SLUB and SLAB allocators. Move the SLUB allocators *_alloc_hook to the common mm/slab.h for internal slab definitions.
--- mm/slab.h | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ mm/slub.c | 54 ----------------------------------------------------- 2 files changed, 62 insertions(+), 54 deletions(-)
mm/slab.h +62 -0
@@ -38,6 +38,10 @@ struct kmem_cache {#endif #include <linux/memcontrol.h> +#include <linux/fault-inject.h> +#include <linux/kmemcheck.h> +#include <linux/kasan.h> +#include <linux/kmemleak.h> /* * State of the slab allocator.
@@ -319,6 +323,64 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)return s; } +static inline size_t slab_ksize (const struct kmem_cache *s) +{ +#ifndef CONFIG_SLUB +return s->object_size; + +#else /* CONFIG_SLUB */ +# ifdef CONFIG_SLUB_DEBUG +/* +* Debugging requires use of the padding between object +* and whatever may come after it. +*/ +if (s->flags & (SLAB_RED_ZONE |SLAB_POISON )) +return s->object_size; +# endif +/* +* If we have the need to store the freelist pointer +* back there or track user information then we can +* only use the space before that information. +*/ +if (s->flags & (SLAB_DESTROY_BY_RCU |SLAB_STORE_USER )) +return s->inuse; +/* +* Else we can use all the padding etc for the allocation +*/ +return s->size; +#endif +} + +static inline struct kmem_cache *slab_pre_alloc_hook (struct kmem_cache *s, +gfp_t flags) +{ + flags &= gfp_allowed_mask; +lockdep_trace_alloc (flags); +might_sleep_if (gfpflags_allow_blocking (flags)); + +if (should_failslab (s->object_size, flags, s->flags)) +return NULL ; + +return memcg_kmem_get_cache (s, flags); +} + +static inline void slab_post_alloc_hook (struct kmem_cache *s,gfp_t flags, +size_t size,void **p) +{ +size_t i; + + flags &= gfp_allowed_mask; +for (i =0 ; i < size; i++) { +void *object = p[i]; + +kmemcheck_slab_alloc (s, flags, object,slab_ksize (s)); +kmemleak_alloc_recursive (object, s->object_size,1 , + s->flags, flags); +kasan_slab_alloc (s, object); + } +memcg_kmem_put_cache (s); +} +#ifndef CONFIG_SLOB /* * The slab lists for all objects.
mm/slub.c +0 -54
66 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 3/11 mm: fault-inject take over bootstrap kmem_cache check
In reply to Jesper Dangaard BrouerRemove the SLAB specific function slab_should_failslab(), by moving the check against fault-injection for the bootstrap slab, into the shared function should_failslab() (used by both SLAB and SLUB).
This is a step towards sharing alloc_hook's between SLUB and SLAB.
This bootstrap slab "kmem_cache" is used for allocating struct kmem_cache objects to the allocator itself.
--- include/linux/fault-inject.h | 5 ++--- mm/failslab.c | 11 ++++++++--- mm/slab.c | 12 ++---------- mm/slab.h | 2 +- 4 files changed, 13 insertions(+), 17 deletions(-)
include/linux/fault-inject.h +2 -3
@@ -62,10 +62,9 @@ static inline struct dentry *fault_create_debugfs_attr(const char *name,#endif /* CONFIG_FAULT_INJECTION */ #ifdef CONFIG_FAILSLAB -extern bool should_failslab (size_t size,gfp_t gfpflags,unsigned long flags); +extern bool should_failslab (struct kmem_cache *s,gfp_t gfpflags);#else -static inline bool should_failslab (size_t size,gfp_t gfpflags, -unsigned long flags) +static inline bool should_failslab (struct kmem_cache *s,gfp_t gfpflags) {return false; }
mm/failslab.c +8 -3
@@ -1,5 +1,6 @@#include <linux/fault-inject.h> #include <linux/slab.h> +#include "slab.h" static struct { struct fault_attr attr;
@@ -11,18 +12,22 @@ static struct { .cache_filter = false, }; -bool should_failslab (size_t size,gfp_t gfpflags,unsigned long cache_flags) +bool should_failslab (struct kmem_cache *s,gfp_t gfpflags) { +/* No fault-injection for bootstrap cache */ +if (unlikely (s == kmem_cache)) +return false; +if (gfpflags & __GFP_NOFAIL)return false;if (failslab.ignore_gfp_reclaim && (gfpflags & __GFP_RECLAIM))return false; -if (failslab.cache_filter && !(cache_flags &SLAB_FAILSLAB )) +if (failslab.cache_filter && !(s->flags &SLAB_FAILSLAB ))return false; -return should_fail (&failslab.attr, size); +return should_fail (&failslab.attr, s->object_size); }static int __init setup_failslab (char *str)
mm/slab.c +2 -10
@@ -2917,14 +2917,6 @@ static void *cache_alloc_debugcheck_after(struct kmem_cache *cachep,#define cache_alloc_debugcheck_after (a,b,objp,d) (objp)#endif -static bool slab_should_failslab (struct kmem_cache *cachep,gfp_t flags) -{ -if (unlikely (cachep == kmem_cache)) -return false; - -return should_failslab (cachep->object_size, flags, cachep->flags); -} -static inline void *____cache_alloc (struct kmem_cache *cachep,gfp_t flags) {void *objp;
@@ -3152,7 +3144,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,lockdep_trace_alloc (flags); -if (slab_should_failslab (cachep, flags)) +if (should_failslab (cachep, flags))return NULL ; cachep =memcg_kmem_get_cache (cachep, flags);
@@ -3240,7 +3232,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)lockdep_trace_alloc (flags); -if (slab_should_failslab (cachep, flags)) +if (should_failslab (cachep, flags))return NULL ; cachep =memcg_kmem_get_cache (cachep, flags);
mm/slab.h +1 -1
@@ -358,7 +358,7 @@ static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s,lockdep_trace_alloc (flags);might_sleep_if (gfpflags_allow_blocking (flags)); -if (should_failslab (s->object_size, flags, s->flags)) +if (should_failslab (s, flags))return NULL ;return memcg_kmem_get_cache (s, flags);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 4/11 slab: use slab_pre_alloc_hook in SLAB allocator shared with SLUB
In reply to Jesper Dangaard BrouerDedublicate code in SLAB allocator functions slab_alloc() and slab_alloc_node() by using the slab_pre_alloc_hook() call, which is now shared between SLUB and SLAB.
---
V2: added unlikely() per request of Kim
mm/slab.c | 18 ++++-------------- 1 file changed, 4 insertions(+), 14 deletions(-)
mm/slab.c +4 -14
@@ -3140,15 +3140,10 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,void *ptr;int slab_node =numa_mem_id (); - flags &= gfp_allowed_mask; - -lockdep_trace_alloc (flags); - -if (should_failslab (cachep, flags)) + cachep =slab_pre_alloc_hook (cachep, flags); +if (unlikely (!cachep))return NULL ; - cachep =memcg_kmem_get_cache (cachep, flags); -cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags);
@@ -3228,15 +3223,10 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)unsigned long save_flags;void *objp; - flags &= gfp_allowed_mask; - -lockdep_trace_alloc (flags); - -if (should_failslab (cachep, flags)) + cachep =slab_pre_alloc_hook (cachep, flags); +if (unlikely (!cachep))return NULL ; - cachep =memcg_kmem_get_cache (cachep, flags); -cache_alloc_debugcheck_before (cachep, flags);local_irq_save (save_flags); objp =__do_cache_alloc (cachep, flags);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 5/11 mm: kmemcheck skip object if slab allocation failed
In reply to Jesper Dangaard BrouerIn the SLAB allocator kmemcheck_slab_alloc() is guarded against being called in case the object is NULL. In SLUB allocator this NULL pointer invocation can happen, which seems like an oversight.
Move the NULL pointer check into kmemcheck code (kmemcheck_slab_alloc) so the check gets moved out of the fastpath, when not compiled with CONFIG_KMEMCHECK.
This is a step towards sharing post_alloc_hook between SLUB and SLAB, because slab_post_alloc_hook() does not perform this check before calling kmemcheck_slab_alloc().
--- mm/kmemcheck.c | 3 +++ 1 file changed, 3 insertions(+)
mm/kmemcheck.c +3 -0
@@ -60,6 +60,9 @@ void kmemcheck_free_shadow(struct page *page, int order)void kmemcheck_slab_alloc (struct kmem_cache *s,gfp_t gfpflags,void *object,size_t size) { +if (unlikely (!object))/* Skip object if allocation failed */ +return ; +/* * Has already been memset(), which initializes the shadow for us * as well.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 6/11 slab: use slab_post_alloc_hook in SLAB allocator shared with SLUB
In reply to Jesper Dangaard BrouerReviewers notice that the order in slab_post_alloc_hook() of kmemcheck_slab_alloc() and kmemleak_alloc_recursive() gets swapped compared to slab.c / SLAB allocator.
Also notice memset now occurs before calling kmemcheck_slab_alloc() and kmemleak_alloc_recursive().
I assume this reordering of kmemcheck, kmemleak and memset is okay because this is the order they are used by the SLUB allocator.
This patch completes the sharing of alloc_hook's between SLUB and SLAB.
--- mm/slab.c | 22 ++++++---------------- 1 file changed, 6 insertions(+), 16 deletions(-)
mm/slab.c +6 -16
@@ -3172,16 +3172,11 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, out:local_irq_restore (save_flags); ptr =cache_alloc_debugcheck_after (cachep, flags, ptr, caller); -kmemleak_alloc_recursive (ptr, cachep->object_size,1 , cachep->flags, - flags); -if (likely (ptr)) { -kmemcheck_slab_alloc (cachep, flags, ptr, cachep->object_size); -if (unlikely (flags & __GFP_ZERO)) -memset (ptr,0 , cachep->object_size); - } +if (unlikely (flags & __GFP_ZERO) && ptr) +memset (ptr,0 , cachep->object_size); -memcg_kmem_put_cache (cachep); +slab_post_alloc_hook (cachep, flags,1 , &ptr);return ptr; }
@@ -3232,17 +3227,12 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp =__do_cache_alloc (cachep, flags);local_irq_restore (save_flags); objp =cache_alloc_debugcheck_after (cachep, flags, objp, caller); -kmemleak_alloc_recursive (objp, cachep->object_size,1 , cachep->flags, - flags);prefetchw (objp); -if (likely (objp)) { -kmemcheck_slab_alloc (cachep, flags, objp, cachep->object_size); -if (unlikely (flags & __GFP_ZERO)) -memset (objp,0 , cachep->object_size); - } +if (unlikely (flags & __GFP_ZERO) && objp) +memset (objp,0 , cachep->object_size); -memcg_kmem_put_cache (cachep); +slab_post_alloc_hook (cachep, flags,1 , &objp);return objp; }
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 7/11 slab: implement bulk alloc in SLAB allocator
In reply to Jesper Dangaard BrouerThis patch implements the alloc side of bulk API for the SLAB allocator.
Further optimization are still possible by changing the call to __do_cache_alloc() into something that can return multiple objects. This optimization is left for later, given end results already show in the area of 80% speedup.
--- mm/slab.c | 37 +++++++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-)
mm/slab.c +35 -2
@@ -3392,9 +3392,42 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p)EXPORT_SYMBOL (kmem_cache_free_bulk);int kmem_cache_alloc_bulk (struct kmem_cache *s,gfp_t flags,size_t size, -void **p) +void **p) { -return __kmem_cache_alloc_bulk (s, flags, size, p); +size_t i; + + s =slab_pre_alloc_hook (s, flags); +if (!s) +return 0 ; + +cache_alloc_debugcheck_before (s, flags); + +local_irq_disable (); +for (i =0 ; i < size; i++) { +void *objp =__do_cache_alloc (s, flags); + +/* this call could be done outside IRQ disabled section */ + objp =cache_alloc_debugcheck_after (s, flags, objp, _RET_IP_); + +if (unlikely (!objp)) + goto error; + p[i] = objp; + } +local_irq_enable (); + +/* Clear memory outside IRQ disabled section */ +if (unlikely (flags & __GFP_ZERO)) +for (i =0 ; i < size; i++) +memset (p[i],0 , s->object_size); + +slab_post_alloc_hook (s, flags, size, p); +/* FIXME: Trace call missing. Christoph would like a bulk variant */ +return size; +error: +local_irq_enable (); +slab_post_alloc_hook (s, flags, i, p); +__kmem_cache_free_bulk (s, i, p); +return 0 ; }EXPORT_SYMBOL (kmem_cache_alloc_bulk);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 8/11 slab: avoid running debug SLAB code with IRQs disabled for alloc_bulk
In reply to Jesper Dangaard BrouerMove the call to cache_alloc_debugcheck_after() outside the IRQ disabled section in kmem_cache_alloc_bulk().
When CONFIG_DEBUG_SLAB is disabled the compiler should remove this code.
--- mm/slab.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-)
mm/slab.c +13 -3
@@ -3391,6 +3391,16 @@ void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p) }EXPORT_SYMBOL (kmem_cache_free_bulk); +static __always_inline void +cache_alloc_debugcheck_after_bulk (struct kmem_cache *s,gfp_t flags, +size_t size,void **p,unsigned long caller) +{ +size_t i; + +for (i =0 ; i < size; i++) + p[i] =cache_alloc_debugcheck_after (s, flags, p[i], caller); +} +int kmem_cache_alloc_bulk (struct kmem_cache *s,gfp_t flags,size_t size,void **p) {
@@ -3406,15 +3416,14 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,for (i =0 ; i < size; i++) {void *objp =__do_cache_alloc (s, flags); -/* this call could be done outside IRQ disabled section */ - objp =cache_alloc_debugcheck_after (s, flags, objp, _RET_IP_); -if (unlikely (!objp)) goto error; p[i] = objp; }local_irq_enable (); +cache_alloc_debugcheck_after_bulk (s, flags, size, p, _RET_IP_); +/* Clear memory outside IRQ disabled section */ if (unlikely (flags & __GFP_ZERO))for (i =0 ; i < size; i++)
@@ -3425,6 +3434,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,return size; error:local_irq_enable (); +cache_alloc_debugcheck_after_bulk (s, flags, i, p, _RET_IP_);slab_post_alloc_hook (s, flags, i, p);__kmem_cache_free_bulk (s, i, p);return 0 ;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 9/11 slab: implement bulk free in SLAB allocator
In reply to Jesper Dangaard BrouerThis patch implements the free side of bulk API for the SLAB allocator kmem_cache_free_bulk(), and concludes the implementation of optimized bulk API for SLAB allocator.
Benchmarked[1] cost of alloc+free (obj size 256 bytes) on CPU i7-4790K @ 4.00GHz, with no debug options, no PREEMPT and CONFIG_MEMCG_KMEM=y but no active user of kmemcg.
SLAB single alloc+free cost: 87 cycles(tsc) 21.814 ns with this optimized config.
bulk- Current fallback - optimized SLAB bulk
1 - 102 cycles(tsc) 25.747 ns - 41 cycles(tsc) 10.490 ns - improved 59.8%
2 - 94 cycles(tsc) 23.546 ns - 26 cycles(tsc) 6.567 ns - improved 72.3% 3 - 92 cycles(tsc) 23.127 ns - 20 cycles(tsc) 5.244 ns - improved 78.3% 4 - 90 cycles(tsc) 22.663 ns - 18 cycles(tsc) 4.588 ns - improved 80.0% 8 - 88 cycles(tsc) 22.242 ns - 14 cycles(tsc) 3.656 ns - improved 84.1% 16 - 88 cycles(tsc) 22.010 ns - 13 cycles(tsc) 3.480 ns - improved 85.2% 30 - 89 cycles(tsc) 22.305 ns - 13 cycles(tsc) 3.303 ns - improved 85.4% 32 - 89 cycles(tsc) 22.277 ns - 13 cycles(tsc) 3.309 ns - improved 85.4% 34 - 88 cycles(tsc) 22.246 ns - 13 cycles(tsc) 3.294 ns - improved 85.2% 48 - 88 cycles(tsc) 22.121 ns - 13 cycles(tsc) 3.492 ns - improved 85.2% 64 - 88 cycles(tsc) 22.052 ns - 13 cycles(tsc) 3.411 ns - improved 85.2% 128 - 89 cycles(tsc) 22.452 ns - 15 cycles(tsc) 3.841 ns - improved 83.1% 158 - 89 cycles(tsc) 22.403 ns - 14 cycles(tsc) 3.746 ns - improved 84.3% 250 - 91 cycles(tsc) 22.775 ns - 16 cycles(tsc) 4.111 ns - improved 82.4%
Notice it is not recommended to do very large bulk operation with this bulk API, because local IRQs are disabled in this period.
[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test01.c
--- mm/slab.c | 29 +++++++++++++++++++++++------ 1 file changed, 23 insertions(+), 6 deletions(-)
mm/slab.c +23 -6
@@ -3385,12 +3385,6 @@ void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) }EXPORT_SYMBOL (kmem_cache_alloc); -void kmem_cache_free_bulk (struct kmem_cache *s,size_t size,void **p) -{ -__kmem_cache_free_bulk (s, size, p); -} -EXPORT_SYMBOL (kmem_cache_free_bulk); - static __always_inline voidcache_alloc_debugcheck_after_bulk (struct kmem_cache *s,gfp_t flags,size_t size,void **p,unsigned long caller)
@@ -3584,6 +3578,29 @@ void kmem_cache_free(struct kmem_cache *cachep, void *objp) }EXPORT_SYMBOL (kmem_cache_free); +void kmem_cache_free_bulk (struct kmem_cache *orig_s,size_t size,void **p) +{ +struct kmem_cache *s; +size_t i; + +local_irq_disable (); +for (i =0 ; i < size; i++) { +void *objp = p[i]; + + s =cache_from_obj (orig_s, objp); + +debug_check_no_locks_freed (objp, s->object_size); +if (!(s->flags &SLAB_DEBUG_OBJECTS )) +debug_check_no_obj_freed (objp, s->object_size); + +__cache_free (s, objp, _RET_IP_); + } +local_irq_enable (); + +/* FIXME: add tracing */ +} +EXPORT_SYMBOL (kmem_cache_free_bulk); +/** * kfree - free previously allocated memory * @objp: pointer returned by kmalloc.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 10/11 mm: new API kfree_bulk() for SLAB+SLUB allocators
In reply to Jesper Dangaard BrouerThis patch introduce a new API call kfree_bulk() for bulk freeing memory objects not bound to a single kmem_cache.
Christoph pointed out that it is possible to implement freeing of objects, without knowing the kmem_cache pointer as that information is available from the object's page->slab_cache. Proposing to remove the kmem_cache argument from the bulk free API.
Jesper demonstrated that these extra steps per object comes at a performance cost. It is only in the case CONFIG_MEMCG_KMEM is compiled in and activated runtime that these steps are done anyhow. The extra cost is most visible for SLAB allocator, because the SLUB allocator does the page lookup (virt_to_head_page()) anyhow.
Thus, the conclusion was to keep the kmem_cache free bulk API with a kmem_cache pointer, but we can still implement a kfree_bulk() API fairly easily. Simply by handling if kmem_cache_free_bulk() gets called with a kmem_cache NULL pointer.
This does increase the code size a bit, but implementing a separate kfree_bulk() call would likely increase code size even more.
Below benchmarks cost of alloc+free (obj size 256 bytes) on CPU i7-4790K @ 4.00GHz, no PREEMPT and CONFIG_MEMCG_KMEM=y.
Code size increase for SLAB:
add/remove: 0/0 grow/shrink: 1/0 up/down: 74/0 (74)
function old new delta kmem_cache_free_bulk 660 734 +74
SLAB fastpath: 87 cycles(tsc) 21.814
sz - fallback - kmem_cache_free_bulk - kfree_bulk
1 - 103 cycles 25.878 ns - 41 cycles 10.498 ns - 81 cycles 20.312 ns 2 - 94 cycles 23.673 ns - 26 cycles 6.682 ns - 42 cycles 10.649 ns 3 - 92 cycles 23.181 ns - 21 cycles 5.325 ns - 39 cycles 9.950 ns 4 - 90 cycles 22.727 ns - 18 cycles 4.673 ns - 26 cycles 6.693 ns 8 - 89 cycles 22.270 ns - 14 cycles 3.664 ns - 23 cycles 5.835 ns 16 - 88 cycles 22.038 ns - 14 cycles 3.503 ns - 22 cycles 5.543 ns 30 - 89 cycles 22.284 ns - 13 cycles 3.310 ns - 20 cycles 5.197 ns 32 - 88 cycles 22.249 ns - 13 cycles 3.420 ns - 20 cycles 5.166 ns 34 - 88 cycles 22.224 ns - 14 cycles 3.643 ns - 20 cycles 5.170 ns 48 - 88 cycles 22.088 ns - 14 cycles 3.507 ns - 20 cycles 5.203 ns 64 - 88 cycles 22.063 ns - 13 cycles 3.428 ns - 20 cycles 5.152 ns 128 - 89 cycles 22.483 ns - 15 cycles 3.891 ns - 23 cycles 5.885 ns 158 - 89 cycles 22.381 ns - 15 cycles 3.779 ns - 22 cycles 5.548 ns 250 - 91 cycles 22.798 ns - 16 cycles 4.152 ns - 23 cycles 5.967 ns
SLAB when enabling MEMCG_KMEM runtime:
- kmemcg fastpath: 130 cycles(tsc) 32.684 ns (step:0)
1 - 148 cycles 37.220 ns - 66 cycles 16.622 ns - 66 cycles 16.583 ns 2 - 141 cycles 35.510 ns - 51 cycles 12.820 ns - 58 cycles 14.625 ns 3 - 140 cycles 35.017 ns - 37 cycles 9.326 ns - 33 cycles 8.474 ns 4 - 137 cycles 34.507 ns - 31 cycles 7.888 ns - 33 cycles 8.300 ns 8 - 140 cycles 35.069 ns - 25 cycles 6.461 ns - 25 cycles 6.436 ns 16 - 138 cycles 34.542 ns - 23 cycles 5.945 ns - 22 cycles 5.670 ns 30 - 136 cycles 34.227 ns - 22 cycles 5.502 ns - 22 cycles 5.587 ns 32 - 136 cycles 34.253 ns - 21 cycles 5.475 ns - 21 cycles 5.324 ns 34 - 136 cycles 34.254 ns - 21 cycles 5.448 ns - 20 cycles 5.194 ns 48 - 136 cycles 34.075 ns - 21 cycles 5.458 ns - 21 cycles 5.367 ns 64 - 135 cycles 33.994 ns - 21 cycles 5.350 ns - 21 cycles 5.259 ns 128 - 137 cycles 34.446 ns - 23 cycles 5.816 ns - 22 cycles 5.688 ns 158 - 137 cycles 34.379 ns - 22 cycles 5.727 ns - 22 cycles 5.602 ns 250 - 138 cycles 34.755 ns - 24 cycles 6.093 ns - 23 cycles 5.986 ns
Code size increase for SLUB:
function old new delta kmem_cache_free_bulk 717 799 +82
SLUB benchmark:
SLUB fastpath: 46 cycles(tsc) 11.691 ns (step:0)
sz - fallback - kmem_cache_free_bulk - kfree_bulk
1 - 61 cycles 15.486 ns - 53 cycles 13.364 ns - 57 cycles 14.464 ns 2 - 54 cycles 13.703 ns - 32 cycles 8.110 ns - 33 cycles 8.482 ns 3 - 53 cycles 13.272 ns - 25 cycles 6.362 ns - 27 cycles 6.947 ns 4 - 51 cycles 12.994 ns - 24 cycles 6.087 ns - 24 cycles 6.078 ns 8 - 50 cycles 12.576 ns - 21 cycles 5.354 ns - 22 cycles 5.513 ns 16 - 49 cycles 12.368 ns - 20 cycles 5.054 ns - 20 cycles 5.042 ns 30 - 49 cycles 12.273 ns - 18 cycles 4.748 ns - 19 cycles 4.758 ns 32 - 49 cycles 12.401 ns - 19 cycles 4.821 ns - 19 cycles 4.810 ns 34 - 98 cycles 24.519 ns - 24 cycles 6.154 ns - 24 cycles 6.157 ns 48 - 83 cycles 20.833 ns - 21 cycles 5.446 ns - 21 cycles 5.429 ns 64 - 75 cycles 18.891 ns - 20 cycles 5.247 ns - 20 cycles 5.238 ns 128 - 93 cycles 23.271 ns - 27 cycles 6.856 ns - 27 cycles 6.823 ns 158 - 102 cycles 25.581 ns - 30 cycles 7.714 ns - 30 cycles 7.695 ns 250 - 107 cycles 26.917 ns - 38 cycles 9.514 ns - 38 cycles 9.506 ns
SLUB when enabling MEMCG_KMEM runtime:
- kmemcg fastpath: 71 cycles(tsc) 17.897 ns (step:0)
1 - 85 cycles 21.484 ns - 78 cycles 19.569 ns - 75 cycles 18.938 ns 2 - 81 cycles 20.363 ns - 45 cycles 11.258 ns - 44 cycles 11.076 ns 3 - 78 cycles 19.709 ns - 33 cycles 8.354 ns - 32 cycles 8.044 ns 4 - 77 cycles 19.430 ns - 28 cycles 7.216 ns - 28 cycles 7.003 ns 8 - 101 cycles 25.288 ns - 23 cycles 5.849 ns - 23 cycles 5.787 ns 16 - 76 cycles 19.148 ns - 20 cycles 5.162 ns - 20 cycles 5.081 ns 30 - 76 cycles 19.067 ns - 19 cycles 4.868 ns - 19 cycles 4.821 ns 32 - 76 cycles 19.052 ns - 19 cycles 4.857 ns - 19 cycles 4.815 ns 34 - 121 cycles 30.291 ns - 25 cycles 6.333 ns - 25 cycles 6.268 ns 48 - 108 cycles 27.111 ns - 21 cycles 5.498 ns - 21 cycles 5.458 ns 64 - 100 cycles 25.164 ns - 20 cycles 5.242 ns - 20 cycles 5.229 ns 128 - 155 cycles 38.976 ns - 27 cycles 6.886 ns - 27 cycles 6.892 ns 158 - 132 cycles 33.034 ns - 30 cycles 7.711 ns - 30 cycles 7.728 ns 250 - 130 cycles 32.612 ns - 38 cycles 9.560 ns - 38 cycles 9.549 ns
----
V2: Re-ran SLAB benchmark, Kim noticed measurement instability
- Also tested with SLOB allocator
include/linux/slab.h | 9 +++++++++ mm/slab.c | 5 ++++- mm/slab_common.c | 8 ++++++-- mm/slub.c | 21 ++++++++++++++++++--- 4 files changed, 37 insertions(+), 6 deletions(-)
include/linux/slab.h +9 -0
@@ -318,6 +318,15 @@ void kmem_cache_free(struct kmem_cache *, void *);void kmem_cache_free_bulk (struct kmem_cache *,size_t ,void **);int kmem_cache_alloc_bulk (struct kmem_cache *,gfp_t ,size_t ,void **); +/* +* Caller must not use kfree_bulk() on memory not originally allocated +* by kmalloc(), because the SLOB allocator cannot handle this. +*/ +static __always_inline void kfree_bulk(size_t size, void **p) +{ +kmem_cache_free_bulk (NULL , size, p); +} +#ifdef CONFIG_NUMA void *__kmalloc_node (size_t size,gfp_t flags,int node) __assume_kmalloc_alignment;void *kmem_cache_alloc_node (struct kmem_cache *,gfp_t flags,int node) __assume_slab_alignment;
mm/slab.c +4 -1
@@ -3587,7 +3587,10 @@ void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p)for (i =0 ; i < size; i++) {void *objp = p[i]; - s =cache_from_obj (orig_s, objp); +if (!orig_s)/* called via kfree_bulk */ + s =virt_to_cache (objp); +else + s =cache_from_obj (orig_s, objp);debug_check_no_locks_freed (objp, s->object_size);if (!(s->flags &SLAB_DEBUG_OBJECTS ))
mm/slab_common.c +6 -2
@@ -108,8 +108,12 @@ void __kmem_cache_free_bulk(struct kmem_cache *s, size_t nr, void **p) {size_t i; -for (i =0 ; i < nr; i++) -kmem_cache_free (s, p[i]); +for (i =0 ; i < nr; i++) { +if (s) +kmem_cache_free (s, p[i]); +else +kfree (p[i]); + } } int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t nr,
mm/slub.c +18 -3
@@ -2787,23 +2787,38 @@ int build_detached_freelist(struct kmem_cache *s, size_t size,size_t first_skipped_index =0 ;int lookahead =3 ;void *object; +struct page *page;/* Always re-init detached_freelist */ df->page =NULL ;do { object = p[--size]; +/* Do we need !ZERO_OR_NULL_PTR(object) here? (for kfree) */ }while (!object && size);if (!object)return 0 ; -/* Support for memcg, compiler can optimize this out */ - df->s =cache_from_obj (s, object); + page =virt_to_head_page (object); +if (!s) { +/* Handle kalloc'ed objects */ +if (unlikely (!PageSlab (page))) { +BUG_ON (!PageCompound (page)); +kfree_hook (object); +__free_kmem_pages (page,compound_order (page)); + p[size] =NULL ;/* mark object processed */ +return size; + } +/* Derive kmem_cache from object */ + df->s = page->slab_cache; + }else { + df->s =cache_from_obj (s, object);/* Support for memcg */ + }/* Start new detached freelist */ + df->page = page;set_freepointer (df->s, object,NULL ); - df->page =virt_to_head_page (object); df->tail = object; df->freelist = object; p[size] =NULL ;/* mark object processed */
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 11/11 mm: fix some spelling
In reply to Jesper Dangaard BrouerFixup trivial spelling errors, noticed while reading the code.
--- include/linux/memcontrol.h | 2 +- include/linux/slab.h | 2 +- mm/slab.h | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
include/linux/memcontrol.h +1 -1
@@ -757,7 +757,7 @@ int __memcg_kmem_charge(struct page *page, gfp_t gfp, int order);void __memcg_kmem_uncharge (struct page *page,int order);/* -* helper for acessing a memcg's index. It will be used as an index in the +* helper for accessing a memcg's index. It will be used as an index in the * child cache array in kmem_cache, and also to derive its name. This function * will return -1 when this is not a kmem-limited memcg. */
include/linux/slab.h +1 -1
@@ -309,7 +309,7 @@ void *kmem_cache_alloc(struct kmem_cache *, gfp_t flags) __assume_slab_alignmentvoid kmem_cache_free (struct kmem_cache *,void *);/* -* Bulk allocation and freeing operations. These are accellerated in an +* Bulk allocation and freeing operations. These are accelerated in an * allocator specific way to avoid taking locks repeatedly or building * metadata structures unnecessarily. *
mm/slab.h +1 -1
@@ -170,7 +170,7 @@ ssize_t slabinfo_write(struct file *file, const char __user *buffer,/* * Generic implementation of bulk operations * These are useful for situations in which the allocator cannot -* perform optimizations. In that case segments of the objecct listed +* perform optimizations. In that case segments of the object listed * may be allocated or freed using these operations. */ void __kmem_cache_free_bulk (struct kmem_cache *,size_t ,void **);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>