slab: optimize constant-size kzalloc calls
Subsystems (3)
PATCH 1/1 slab: optimize constant-size kzalloc calls
From: Pekka Enberg <[email protected]>
As suggested by Eric Dumazet, this patch optimizes kzalloc() calls that pass a compile-time constant size. Please note that the patch increases kernel text slightly (~200 bytes for defconfig on x86).
This patch requires the kmem_cache_zalloc() patches I sent earlier.
---
include/linux/slab.h | 30 +++++++++++++++++++++++++++--- mm/util.c | 6 +++--- 2 files changed, 30 insertions(+), 6 deletions(-)
include/linux/slab.h +27 -3
@@ -109,7 +109,30 @@ found:return __kmalloc (size, flags); } -extern void *kzalloc (size_t ,gfp_t ); +extern void *__kzalloc (size_t ,gfp_t ); + +static inline void *kzalloc (size_t size,gfp_t flags) +{ +if (__builtin_constant_p (size)) { +int i =0 ; +#define CACHE (x) \ + if (size <= x) \ + goto found; \ + else \ + i++; +#include "kmalloc_sizes.h" +#undef CACHE + { +extern void __you_cannot_kzalloc_that_much (void ); +__you_cannot_kzalloc_that_much (); + } +found: +return kmem_cache_zalloc ((flags &GFP_DMA ) ? + malloc_sizes[i].cs_dmacachep : + malloc_sizes[i].cs_cachep, flags); + } +return __kzalloc (size, flags); +}/** * kcalloc - allocate memory for an array. The memory is set to zero.
@@ -160,14 +183,14 @@ void *kmem_cache_zalloc(struct kmem_cachvoid kmem_cache_free (struct kmem_cache *c,void *b);const char *kmem_cache_name (struct kmem_cache *);void *kmalloc (size_t size,gfp_t flags); -void *kzalloc (size_t size,gfp_t flags); +void *__kzalloc (size_t size,gfp_t flags);void kfree (const void *m);unsigned int ksize (const void *m);unsigned int kmem_cache_size (struct kmem_cache *c);static inline void *kcalloc (size_t n,size_t size,gfp_t flags) { -return kzalloc (n * size, flags); +return __kzalloc (n * size, flags); }#define kmem_cache_shrink (d) (0)
@@ -175,6 +198,7 @@ static inline void *kcalloc(size_t n, si#define kmem_ptr_validate (a, b) (0)#define kmem_cache_alloc_node (c, f, n) kmem_cache_alloc(c, f)#define kmalloc_node (s, f, n) kmalloc(s, f) +#define kzalloc (s, f) __kzalloc(s, f)#endif /* CONFIG_SLOB */
mm/util.c +3 -3
@@ -3,18 +3,18 @@#include <linux/module.h> /** -* kzalloc - allocate memory. The memory is set to zero. +* __kzalloc - allocate memory. The memory is set to zero. * @size: how many bytes of memory are required. * @flags: the type of memory to allocate. */ -void *kzalloc (size_t size,gfp_t flags) +void *__kzalloc (size_t size,gfp_t flags) {void *ret =kmalloc (size, flags);if (ret)memset (ret,0 , size);return ret; } -EXPORT_SYMBOL (kzalloc); +EXPORT_SYMBOL (__kzalloc);/* * kstrdup - allocate space for and copy an existing string
Quoted 3 lines · Pekka Enberg
As suggested by Eric Dumazet, this patch optimizes
kzalloc()calls
that pass a compile-time constant size.
Logical.
Quoted 2 lines
Please note that the patch
increases kernel text slightly (~200 bytes for defconfig on x86)
Why?
Quoted 2 lines · Pekka Enberg
Quoted 2 lines
Please note that the patch
increases kernel text slightly (~200 bytes for defconfig on x86)
Quoted 1 line · Andrew Morton
Why?
Because of the malloc_sizes lookup. As kindly explained by Eric, we're now doing:
kzalloc(100, GFP_KERNEL);
and with the patch, we will do:
kmem_cache_zalloc(malloc_sizes[4].cs_dmacachep, GFP_KERNEL);
which explains the the difference. The end result should be faster kzalloc(), though.
Pekka
Hi,
I feel like I mist be dreaming, but this patch, which was inlcuded in Linus' tree as 40c07ae8daa659b8feb149c84731629386873c16 calls __you_cannot_kzalloc_that_much(), but that does not seem to exist.
On i386 at least that causes a build failure
# make mrproper # make allyesconfig # make [snip] drivers/built-in.o: In function `kzalloc':include/linux/slab.h:128: undefined reference to `__you_cannot_kzalloc_that_much' :include/linux/slab.h:128: undefined reference to `__you_cannot_kzalloc_that_much' make: *** [.tmp_vmlinux1] Error 1
Signature · 1 line
Horms
From: Horms <[email protected]>
Date: Wed, 29 Mar 2006 10:57:46 +0900
Quoted 5 lines
I feel like I mist be dreaming, but this patch, which was inlcuded in Linus' tree as
40c07ae8daa659b8feb149c84731629386873c16calls__you_cannot_kzalloc_that_much(), but that does not seem to exist.On i386 at least that causes a build failure
It's a purposeful build time error introduced so that invalid calls that specify too large kzalloc() length arguments are caught at build time.
Quoted 7 lines · David S. Miller
From: Horms <[email protected]>
Date: Wed, 29 Mar 2006 10:57:46 +0900Quoted 4 lines
I feel like I mist be dreaming, but this patch, which was inlcuded in Linus' tree as
40c07ae8daa659b8feb149c84731629386873c16calls__you_cannot_kzalloc_that_much(), but that does not seem to exist.
Quoted 5 lines
Quoted 1 line
On i386 at least that causes a build failure
It's a purposeful build time error introduced so that invalid calls that specify too large
kzalloc()length arguments are caught at build time.
Thanks, I knew that I had to be missing the point somewhere. I'll focus my attention on the caller.
Signature · 1 line
Horms