Cpu alloc slub support: Replace percpu allocator in slub.c
Subsystems (3)
cover Cpu alloc slub support: Replace percpu allocator in slub.c
Slub also has its own per cpu allocator. Get rid of it and use cpu_alloc().
Signature · 5 lines
-- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/3 cpu alloc: Use in slub
Using cpu alloc removes the needs for the per cpu arrays in the kmem_cache struct. These could get quite big if we have to support system of up to thousands of cpus. The use of cpu_alloc means that:
1. The size of kmem_cache for SMP configuration shrinks since we will only need 1 pointer instead of NR_CPUS. The same pointer can be used by all processors. Reduces cache footprint of the allocator.
2. We can dynamically size kmem_cache according to the actual nodes in the system meaning less memory overhead for configurations that may potentially support up to 1k NUMA nodes / 4k cpus.
3. We can remove the diddle widdle with allocating and releasing of kmem_cache_cpu structures when bringing up and shutting down cpus. The cpu alloc logic will do it all for us. Removes some portions of the cpu hotplug functionality.
4. Fastpath performance increases.
include/linux/slub_def.h +2 -6
@@ -68,6 +68,7 @@* Slab cache management. */ struct kmem_cache { +struct kmem_cache_cpu *cpu_slab;/* Used for retriving partial slabs etc */ unsigned long flags;int size;/* The size of an object including meta data */
@@ -100,12 +101,7 @@* Defragmentation by allocating from a remote node. */ int remote_node_defrag_ratio; -struct kmem_cache_node *node[MAX_NUMNODES ]; -#endif -#ifdef CONFIG_SMP -struct kmem_cache_cpu *cpu_slab[NR_CPUS ]; -#else -struct kmem_cache_cpu cpu_slab; +struct kmem_cache_node *node[];#endif };
mm/slub.c +25 -155
316 lines not on this page · raw message
Signature · 5 lines
-- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/3 cpu alloc: Remove slub fields
In reply to Christoph LameterRemove the fields in kmem_cache_cpu that were used to cache data from kmem_cache when they were in different cachelines. The cacheline that holds the per cpu array pointer now also holds these values. We can cut down the struct kmem_cache_cpu size to almost half.
The get_freepointer() and set_freepointer() functions that used to be only intended for the slow path now are also useful for the hot path since access to the field does not require accessing an additional cacheline anymore. This results in consistent use of setting the freepointer for objects throughout SLUB.
Also we initialize all possible kmem_cache_cpu structures when a slab is created. No need to initialize them when a processor or node comes online. And all fields are set to zero. So just use __GFP_ZERO on cpu alloc.
include/linux/slub_def.h +0 -2
@@ -36,8 +36,6 @@void **freelist;/* Pointer to first free per cpu object */ struct page *page;/* The slab from which we are allocating */ int node;/* The node of the page (or -1 for debug) */ -unsigned int offset;/* Freepointer offset (in word units) */ -unsigned int objsize;/* Size of an object (from kmem_cache) */ #ifdef CONFIG_SLUB_STATS unsigned stat[NR_SLUB_STAT_ITEMS ];#endif
mm/slub.c +15 -62
170 lines not on this page · raw message
Re: [patch 2/3] cpu alloc: Remove slub fields
Hi Christoph,
mm/slub.c line 2169
Quoted 9 lines · Christoph Lameter
mm/slub.c +3
-1@@ -2196,8 +2163,11 @@if (!init_kmem_cache_nodes (s, gfpflags & ~SLUB_DMA )) goto error; -if (alloc_kmem_cache_cpus (s, gfpflags & ~SLUB_DMA )) + s->cpu_slab = CPU_ALLOC(struct kmem_cache_cpu, + (flags & ~SLUB_DMA ) | __GFP_ZERO); +if (!s->cpu_slab)return 1 ;
This should be s->cpu_slab, no?
mm/slub.c line 2173
Quoted 4 lines
mm/slub.c +1
-0+free_kmem_cache_nodes (s); error:if (flags &SLAB_PANIC )
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [patch 2/3] cpu alloc: Remove slub fields
Quoted 7 lines · Pekka Enberg
Hi Christoph,
Quoted 4 lines · Christoph Lameter
mm/slub.c +0
-0@@ -2196,8 +2163,11 @@if (!init_kmem_cache_nodes (s, gfpflags & ~SLUB_DMA )) goto error;
Quoted 7 lines
mm/slub.c line 2169
Quoted 5 lines
mm/slub.c +3
-1-if (alloc_kmem_cache_cpus (s, gfpflags & ~SLUB_DMA )) + s->cpu_slab = CPU_ALLOC(struct kmem_cache_cpu, + (flags & ~SLUB_DMA ) | __GFP_ZERO); +if (!s->cpu_slab)return 1 ;This should be
s->cpu_slab, no?
Correct. Newer rev that will be based on cpu_alloc V6 will have that.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/3 Increase default reserve percpu area
In reply to Christoph LameterSLUB now requires a portion of the per cpu reserve. There are on average about 70 real slabs on a system (aliases do not count) and each needs 12 bytes of per cpu space. Thats 840 bytes. In debug mode all slabs will be real slabs which will make us end up with 150 -> 1800. Give it some slack and add 2000 bytes to the default size.
Things work fine without this patch but then slub will reduce the percpu reserve for modules.
Also define a reserve if CONFIG_MODULES is off.
include/linux/percpu.h +2 -2
@@ -38,9 +38,9 @@/* Enough to cover all DEFINE_PER_CPUs in kernel, including modules. */ #ifndef PERCPU_AREA_SIZE #ifdef CONFIG_MODULES -#define PERCPU_RESERVE_SIZE 8192 +#define PERCPU_RESERVE_SIZE 10000#else -#define PERCPU_RESERVE_SIZE 0 +#define PERCPU_RESERVE_SIZE 2000#endif #define PERCPU_AREA_SIZE \
Re: [patch 3/3] Increase default reserve percpu area
Quoted 8 lines · Christoph Lameter
SLUB now requires a portion of the per cpu reserve. There are on average about 70 real slabs on a system (aliases do not count) and each needs 12 bytes of per cpu space. Thats 840 bytes. In debug mode all slabs will be real slabs which will make us end up with 150 -> 1800. Give it some slack and add 2000 bytes to the default size.
Things work fine without this patch but then slub will reduce the percpu reserve for modules.
Hmm, shouldn't this be dynamically configured at runtime by multiplying the number of possible CPUs with some constant?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [patch 3/3] Increase default reserve percpu area
Quoted 8 lines · Pekka Enberg
Quoted 6 lines · Christoph Lameter
SLUB now requires a portion of the per cpu reserve. There are on average about 70 real slabs on a system (aliases do not count) and each needs 12 bytes of per cpu space. Thats 840 bytes. In debug mode all slabs will be real slabs which will make us end up with 150 -> 1800. Give it some slack and add 2000 bytes to the default size.
Quoted 5 lines
Quoted 2 lines
Things work fine without this patch but then slub will reduce the percpu reserve for modules.
Hmm, shouldn't this be dynamically configured at runtime by multiplying the number of possible CPUs with some constant?
Oh, never mind, setup_per_cpu_areas() already does that, obviously.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · Christoph Lameter
Slub also has its own per cpu allocator. Get rid of it and use
cpu_alloc().
So, do you want to stick these into the slab tree? I would then need to pick up the cpualloc patches as well, I suppose. Andrew, pls help.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>