Introduce per cpu atomic operations and avoid per cpu address arithmetic
Subsystems (5)
PATCH v6 5/7 this_cpu: Remove slub kmem_cache fields
In reply to [email protected]Remove the fields in struct kmem_cache_cpu that were used to cache data from struct kmem_cache when they were in different cachelines. The cacheline that holds the per cpu array pointer now also holds these values. We can cut down the struct kmem_cache_cpu size to almost half.
The get_freepointer() and set_freepointer() functions that used to be only intended for the slow path now are also useful for the hot path since access to the size field does not require accessing an additional cacheline anymore. This results in consistent use of functions for setting the freepointer of objects throughout SLUB.
Also we initialize all possible kmem_cache_cpu structures when a slab is created. No need to initialize them when a processor or node comes online.
--- include/linux/slub_def.h | 2 - mm/slub.c | 76 +++++++++++------------------------------------ 2 files changed, 19 insertions(+), 59 deletions(-)
include/linux/slub_def.h +0 -2
@@ -38,8 +38,6 @@ struct kmem_cache_cpu {void **freelist;/* Pointer to first free per cpu object */ struct page *page;/* The slab from which we are allocating */ int node;/* The node of the page (or -1 for debug) */ -unsigned int offset;/* Freepointer offset (in word units) */ -unsigned int objsize;/* Size of an object (from kmem_cache) */ #ifdef CONFIG_SLUB_STATS unsigned stat[NR_SLUB_STAT_ITEMS ];#endif
mm/slub.c +19 -57
180 lines not on this page · raw message
v6 cover Introduce per cpu atomic operations and avoid per cpu address arithmetic
V5->V6:
- Drop patches merged by Tejun.
- Drop irqless slub fastpath for now.
- Patches against Tejun percpu for-next branch.
V4->V5:
- Avoid setup_per_cpu_area() modifications and fold the remainder of the patch into the page allocator patch.
- Irq disable / per cpu ptr fixes for page allocator patch.
V3->V4:
- Fix various macro definitions.
- Provide experimental percpu based fastpath that does not disable interrupts for SLUB.
V2->V3:
- Available via git tree against latest upstream from
git://git.kernel.org/pub/scm/linux/kernel/git/christoph/percpu.git linus
- Rework SLUB per cpu operations. Get rid of dynamic DMA slab creation for CONFIG_ZONE_DMA
- Create fallback framework so that 64 bit ops on 32 bit platforms can fallback to the use of preempt or interrupt disable. 64 bit platforms can use 64 bit atomic per cpu ops.
V1->V2:
- Various minor fixes
- Add SLUB conversion
- Add Page allocator conversion
- Patch against the git tree of today
The patchset introduces various operations to allow efficient access to per cpu variables for the current processor. Currently there is no way in the core to calculate the address of the instance of a per cpu variable without a table lookup. So we see a lot of
per_cpu_ptr(x, smp_processor_id())
The patchset introduces a way to calculate the address using the offset that is available in arch specific ways (register or special memory locations) using
this_cpu_ptr(x)
In addition macros are provided that can operate on per cpu variables in a per cpu atomic way. With that scalars in structures allocated with the new percpu allocator can be modified without disabling preempt or interrupts. This works by generating a single instruction that does both the relocation of the address to the proper percpu area and the RMW action.
F.e.
this_cpu_add(x->var, 20)
can be used to generate an instruction that uses a segment register for the relocation of the per cpu address into the per cpu area of the current processor and then increments the variable by 20. The instruction cannot be interrupted and therefore the modification is atomic vs the cpu (it either happens or not). Rescheduling or interrupt can only happen before or after the instruction.
Per cpu atomicness does not provide protection from concurrent modifications from other processors. In general per cpu data is modified only from the processor that the per cpu area is associated with. So per cpu atomicness provides a fast and effective means of dealing with concurrency. It may allow development of better fastpaths for allocators and other important subsystems.
The per cpu atomic RMW operations can be used to avoid having to dimension pointer arrays in the allocators (patches for page allocator and slub are provided) and avoid pointer lookups in the hot paths of the allocators thereby decreasing latency of critical OS paths. The macros could be used to revise the critical paths in the allocators to no longer need to disable interrupts (not included).
Per cpu atomic RMW operations are useful to decrease the overhead of counter maintenance in the kernel. A this_cpu_inc() f.e. can generate a single instruction that has no needs for registers on x86. preempt on / off can be avoided in many places.
Patchset will reduce the code size and increase speed of operations for dynamically allocated per cpu based statistics. A set of patches modifies the fastpaths of the SLUB allocator reducing code size and cache footprint through the per cpu atomic operations.
---
PATCH v6 2/7 this_cpu ops: Remove pageset_notifier
Remove the pageset notifier since it only marks that a processor exists on a specific node. Move that code into the vmstat notifier.
--- mm/page_alloc.c | 28 ---------------------------- mm/vmstat.c | 1 + 2 files changed, 1 insertion(+), 28 deletions(-)
mm/vmstat.c +1 -0
@@ -906,6 +906,7 @@ static int __cpuinit vmstat_cpuup_callbacase CPU_ONLINE :case CPU_ONLINE_FROZEN :start_cpu_timer (cpu); +node_set_state (cpu_to_node (cpu),N_CPU );break ;case CPU_DOWN_PREPARE :case CPU_DOWN_PREPARE_FROZEN :
mm/page_alloc.c +0 -28
@@ -3108,27 +3108,6 @@ static void setup_pagelist_highmark(stru pcp->batch =PAGE_SHIFT *8 ; } - -static int __cpuinit pageset_cpuup_callback (struct notifier_block *nfb, -unsigned long action, -void *hcpu) -{ -int cpu = (long )hcpu; - -switch (action) { -case CPU_UP_PREPARE : -case CPU_UP_PREPARE_FROZEN : -node_set_state (cpu_to_node (cpu),N_CPU ); -break ; -default : -break ; - } -return NOTIFY_OK ; -} - -static struct notifier_block __cpuinitdata pageset_notifier = - { &pageset_cpuup_callback,NULL ,0 }; -/* * Allocate per cpu pagesets and initialize them. * Before this call only boot pagesets were available.
@@ -3154,13 +3133,6 @@ void __init setup_per_cpu_pageset(void) percpu_pagelist_fraction)); } } - -/* -* The boot cpu is always the first active. -* The boot node has a processor -*/ -node_set_state (cpu_to_node (smp_processor_id ()),N_CPU ); -register_cpu_notifier (&pageset_notifier); }static noinline __init_refok
PATCH v6 1/7 this_cpu_ops: page allocator conversion
In reply to [email protected]Use the per cpu allocator functionality to avoid per cpu arrays in struct zone.
This drastically reduces the size of struct zone for systems with large amounts of processors and allows placement of critical variables of struct zone in one cacheline even on very large systems.
Another effect is that the pagesets of one processor are placed near one another. If multiple pagesets from different zones fit into one cacheline then additional cacheline fetches can be avoided on the hot paths when allocating memory from multiple zones.
Bootstrap becomes simpler if we use the same scheme for UP, SMP, NUMA. #ifdefs are reduced and we can drop the zone_pcp macro.
Hotplug handling is also simplified since cpu alloc can bring up and shut down cpu areas for a specific cpu as a whole. So there is no need to allocate or free individual pagesets.
V4-V5:
- Fix up cases where per_cpu_ptr is called before irq disable
- Integrate the bootstrap logic that was separate before.
--- include/linux/mm.h | 4 - include/linux/mmzone.h | 12 --- mm/page_alloc.c | 187 ++++++++++++++++++------------------------------- mm/vmstat.c | 14 ++- 4 files changed, 81 insertions(+), 136 deletions(-)
include/linux/mm.h +0 -4
@@ -1061,11 +1061,7 @@ extern void si_meminfo(struct sysinfo *extern void si_meminfo_node (struct sysinfo *val,int nid);extern int after_bootmem; -#ifdef CONFIG_NUMA extern void setup_per_cpu_pageset (void ); -#else -static inline void setup_per_cpu_pageset (void ) {} -#endif extern void zone_pcp_update (struct zone *zone);
include/linux/mmzone.h +2 -10
@@ -184,13 +184,7 @@ struct per_cpu_pageset {s8 stat_threshold;s8 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS ]; #endif -}____cacheline_aligned_in_smp ; - -#ifdef CONFIG_NUMA -#define zone_pcp (__z, __cpu) ((__z)->pageset[(__cpu)]) -#else -#define zone_pcp (__z, __cpu) (&(__z)->pageset[(__cpu)]) -#endif +};#endif /* !__GENERATING_BOUNDS.H */
@@ -306,10 +300,8 @@ struct zone {*/ unsigned long min_unmapped_pages;unsigned long min_slab_pages; -struct per_cpu_pageset *pageset[NR_CPUS ]; -#else -struct per_cpu_pageset pageset[NR_CPUS ];#endif +struct per_cpu_pageset *pageset;/* * free areas of different sizes */
mm/page_alloc.c +71 -116
313 lines not on this page · raw message
mm/vmstat.c +8 -6
@@ -139,7 +139,8 @@ static void refresh_zone_stat_thresholds threshold =calculate_threshold (zone);for_each_online_cpu (cpu) -zone_pcp (zone, cpu)->stat_threshold = threshold; +per_cpu_ptr (zone->pageset, cpu)->stat_threshold + = threshold; } }
@@ -149,7 +150,8 @@ static void refresh_zone_stat_thresholdsvoid __mod_zone_page_state (struct zone *zone,enum zone_stat_item item,int delta) { -struct per_cpu_pageset *pcp =zone_pcp (zone,smp_processor_id ()); +struct per_cpu_pageset *pcp =this_cpu_ptr (zone->pageset); +s8 *p = pcp->vm_stat_diff + item;long x;
@@ -202,7 +204,7 @@ EXPORT_SYMBOL(mod_zone_page_state);*/ void __inc_zone_state (struct zone *zone,enum zone_stat_item item) { -struct per_cpu_pageset *pcp =zone_pcp (zone,smp_processor_id ()); +struct per_cpu_pageset *pcp =this_cpu_ptr (zone->pageset);s8 *p = pcp->vm_stat_diff + item; (*p)++;
@@ -223,7 +225,7 @@ EXPORT_SYMBOL(__inc_zone_page_state);void __dec_zone_state (struct zone *zone,enum zone_stat_item item) { -struct per_cpu_pageset *pcp =zone_pcp (zone,smp_processor_id ()); +struct per_cpu_pageset *pcp =this_cpu_ptr (zone->pageset);s8 *p = pcp->vm_stat_diff + item; (*p)--;
@@ -300,7 +302,7 @@ void refresh_cpu_vm_stats(int cpu)for_each_populated_zone (zone) {struct per_cpu_pageset *p; - p =zone_pcp (zone, cpu); + p =per_cpu_ptr (zone->pageset, cpu);for (i =0 ; i <NR_VM_ZONE_STAT_ITEMS ; i++)if (p->vm_stat_diff[i]) {
@@ -738,7 +740,7 @@ static void zoneinfo_show_print(struct sfor_each_online_cpu (i) {struct per_cpu_pageset *pageset; - pageset =zone_pcp (zone, i); + pageset =per_cpu_ptr (zone->pageset, i);seq_printf (m,"\n cpu: %i" "\n count: %i"
PATCH v6 3/7 Use this_cpu operations in slub
In reply to [email protected]Using per cpu allocations removes the needs for the per cpu arrays in the kmem_cache struct. These could get quite big if we have to support systems with thousands of cpus. The use of this_cpu_xx operations results in:
1. The size of kmem_cache for SMP configuration shrinks since we will only need 1 pointer instead of NR_CPUS. The same pointer can be used by all processors. Reduces cache footprint of the allocator.
2. We can dynamically size kmem_cache according to the actual nodes in the system meaning less memory overhead for configurations that may potentially support up to 1k NUMA nodes / 4k cpus.
3. We can remove the diddle widdle with allocating and releasing of kmem_cache_cpu structures when bringing up and shutting down cpus. The cpu alloc logic will do it all for us. Removes some portions of the cpu hotplug functionality.
4. Fastpath performance increases since per cpu pointer lookups and address calculations are avoided.
--- include/linux/slub_def.h | 6 - mm/slub.c | 207 ++++++++++------------------------------------- 2 files changed, 49 insertions(+), 164 deletions(-)
include/linux/slub_def.h +1 -5
@@ -69,6 +69,7 @@ struct kmem_cache_order_objects {* Slab cache management. */ struct kmem_cache { +struct kmem_cache_cpu *cpu_slab;/* Used for retriving partial slabs etc */ unsigned long flags;int size;/* The size of an object including meta data */
@@ -104,11 +105,6 @@ struct kmem_cache {int remote_node_defrag_ratio;struct kmem_cache_node *node[MAX_NUMNODES ];#endif -#ifdef CONFIG_SMP -struct kmem_cache_cpu *cpu_slab[NR_CPUS ]; -#else -struct kmem_cache_cpu cpu_slab; -#endif };/*
mm/slub.c +48 -156
350 lines not on this page · raw message
PATCH v6 7/7 this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to [email protected]Use this_cpu_* operations in the hotpath to avoid calculations of kmem_cache_cpu pointer addresses.
On x86 there is a trade off: Multiple uses segment prefixes against an address calculation and more register pressure. Code size is reduced also therefore it is an advantage icache wise.
The use of prefixes is necessary if we want to use a scheme for fastpaths that do not require disabling interrupts.
--- mm/slub.c | 80 ++++++++++++++++++++++++++++++-------------------------------- 1 file changed, 39 insertions(+), 41 deletions(-)
mm/slub.c +39 -41
@@ -1512,10 +1512,10 @@ static void flush_all(struct kmem_cache* Check if the objects in a per cpu structure fit numa * locality expectations. */ -static inline int node_match (struct kmem_cache_cpu *c,int node) +static inline int node_match (struct kmem_cache *s,int node) {#ifdef CONFIG_NUMA -if (node !=-1 && c->node != node) +if (node !=-1 &&__this_cpu_read (s->cpu_slab->node) != node)return 0 ;#endif return 1 ;
@@ -1603,46 +1603,46 @@ slab_out_of_memory(struct kmem_cache *s,* a call to the page allocator and the setup of a new slab. */ static void *__slab_alloc (struct kmem_cache *s,gfp_t gfpflags,int node, -unsigned long addr,struct kmem_cache_cpu *c) +unsigned long addr) {void **object; -struct page *new; +struct page *page =__this_cpu_read (s->cpu_slab->page);/* We handle __GFP_ZERO in the caller */ gfpflags &= ~__GFP_ZERO; -if (!c->page) +if (!page) goto new_slab; -slab_lock (c->page); -if (unlikely (!node_match (c, node))) +slab_lock (page); +if (unlikely (!node_match (s, node))) goto another_slab;stat (s,ALLOC_REFILL ); load_freelist: - object = c->page->freelist; + object = page->freelist;if (unlikely (!object)) goto another_slab; -if (unlikely (SLABDEBUG &&PageSlubDebug (c->page))) +if (unlikely (SLABDEBUG &&PageSlubDebug (page))) goto debug; - c->freelist =get_freepointer (s, object); - c->page->inuse = c->page->objects; - c->page->freelist =NULL ; - c->node =page_to_nid (c->page); +__this_cpu_write (s->cpu_slab->node,page_to_nid (page)); +__this_cpu_write (s->cpu_slab->freelist,get_freepointer (s, object)); + page->inuse = page->objects; + page->freelist =NULL ; unlock_out: -slab_unlock (c->page); +slab_unlock (page);stat (s,ALLOC_SLOWPATH );return object; another_slab: -deactivate_slab (s, c); +deactivate_slab (s,__this_cpu_ptr (s->cpu_slab)); new_slab: - new =get_partial (s, gfpflags, node); -if (new) { - c->page = new; + page =get_partial (s, gfpflags, node); +if (page) { +__this_cpu_write (s->cpu_slab->page, page);stat (s,ALLOC_FROM_PARTIAL ); goto load_freelist; }
@@ -1650,31 +1650,30 @@ new_slab:if (gfpflags & __GFP_WAIT)local_irq_enable (); - new =new_slab (s, gfpflags, node); + page =new_slab (s, gfpflags, node);if (gfpflags & __GFP_WAIT)local_irq_disable (); -if (new) { - c =__this_cpu_ptr (s->cpu_slab); +if (page) {stat (s,ALLOC_SLAB ); -if (c->page) -flush_slab (s, c); -slab_lock (new); -__SetPageSlubFrozen (new); - c->page = new; +if (__this_cpu_read (s->cpu_slab->page)) +flush_slab (s,__this_cpu_ptr (s->cpu_slab)); +slab_lock (page); +__SetPageSlubFrozen (page); +__this_cpu_write (s->cpu_slab->page, page); goto load_freelist; }if (!(gfpflags & __GFP_NOWARN) &&printk_ratelimit ())slab_out_of_memory (s, gfpflags, node);return NULL ; debug: -if (!alloc_debug_processing (s, c->page, object, addr)) +if (!alloc_debug_processing (s, page, object, addr)) goto another_slab; - c->page->inuse++; - c->page->freelist =get_freepointer (s, object); - c->node =-1 ; + page->inuse++; + page->freelist =get_freepointer (s, object); +__this_cpu_write (s->cpu_slab->node,-1 ); goto unlock_out; }
@@ -1692,7 +1691,6 @@ static __always_inline void *slab_alloc( gfp_t gfpflags, int node, unsigned long addr) { void **object; -struct kmem_cache_cpu *c;unsigned long flags; gfpflags &= gfp_allowed_mask;
@@ -1704,14 +1702,14 @@ static __always_inline void *slab_alloc(return NULL ;local_irq_save (flags); - c =__this_cpu_ptr (s->cpu_slab); - object = c->freelist; -if (unlikely (!object || !node_match (c, node))) + object =__this_cpu_read (s->cpu_slab->freelist); +if (unlikely (!object || !node_match (s, node))) - object =__slab_alloc (s, gfpflags, node, addr, c); + object =__slab_alloc (s, gfpflags, node, addr);else { - c->freelist =get_freepointer (s, object); +__this_cpu_write (s->cpu_slab->freelist, +get_freepointer (s, object));stat (s,ALLOC_FASTPATH ); }local_irq_restore (flags);
@@ -1847,19 +1845,19 @@ static __always_inline void slab_free(st struct page *page, void *x, unsigned long addr) { void **object = (void *)x; -struct kmem_cache_cpu *c;unsigned long flags;kmemleak_free_recursive (x, s->flags);local_irq_save (flags); - c =__this_cpu_ptr (s->cpu_slab);kmemcheck_slab_free (s, object, s->objsize);debug_check_no_locks_freed (object, s->objsize);if (!(s->flags &SLAB_DEBUG_OBJECTS ))debug_check_no_obj_freed (object, s->objsize); -if (likely (page == c->page && c->node >=0 )) { -set_freepointer (s, object, c->freelist); - c->freelist = object; + +if (likely (page ==__this_cpu_read (s->cpu_slab->page) && +__this_cpu_read (s->cpu_slab->node) >=0 )) { +set_freepointer (s, object,__this_cpu_read (s->cpu_slab->freelist)); +__this_cpu_write (s->cpu_slab->freelist, object);stat (s,FREE_FASTPATH ); }else __slab_free (s, page, x, addr);
PATCH v6 4/7 SLUB: Get rid of dynamic DMA kmalloc cache allocation
In reply to [email protected]Dynamic DMA kmalloc cache allocation is troublesome since the new percpu allocator does not support allocations in atomic contexts. Reserve some statically allocated kmalloc_cpu structures instead.
--- include/linux/slub_def.h | 19 +++++++++++-------- mm/slub.c | 24 ++++++++++-------------- 2 files changed, 21 insertions(+), 22 deletions(-)
include/linux/slub_def.h +11 -8
@@ -131,11 +131,21 @@ struct kmem_cache {#define SLUB_PAGE_SHIFT (PAGE_SHIFT + 2) +#ifdef CONFIG_ZONE_DMA +#define SLUB_DMA __GFP_DMA +/* Reserve extra caches for potential DMA use */ +#define KMALLOC_CACHES (2 * SLUB_PAGE_SHIFT - 6) +#else +/* Disable DMA functionality */ +#define SLUB_DMA (__force gfp_t)0 +#define KMALLOC_CACHES SLUB_PAGE_SHIFT +#endif +/* * We keep the general caches in an array of slab caches that are used for * 2^x bytes of allocations. */ -extern struct kmem_cache kmalloc_caches[SLUB_PAGE_SHIFT ]; +extern struct kmem_cache kmalloc_caches[KMALLOC_CACHES ];/* * Sorry that the following has to be that ugly but some versions of GCC
@@ -203,13 +213,6 @@ static __always_inline struct kmem_cachereturn &kmalloc_caches[index]; } -#ifdef CONFIG_ZONE_DMA -#define SLUB_DMA __GFP_DMA -#else -/* Disable DMA functionality */ -#define SLUB_DMA (__force gfp_t)0 -#endif -void *kmem_cache_alloc (struct kmem_cache *,gfp_t );void *__kmalloc (size_t size,gfp_t flags);
mm/slub.c +10 -14
@@ -2090,7 +2090,7 @@ static inline int alloc_kmem_cache_cpus( {int cpu; -if (s < kmalloc_caches +SLUB_PAGE_SHIFT && s >= kmalloc_caches) +if (s < kmalloc_caches +KMALLOC_CACHES && s >= kmalloc_caches)/* * Boot time creation of the kmalloc array. Use static per cpu data * since the per cpu allocator is not available yet.
@@ -2537,7 +2537,7 @@ EXPORT_SYMBOL(kmem_cache_destroy);* Kmalloc subsystem *******************************************************************/ -struct kmem_cache kmalloc_caches[SLUB_PAGE_SHIFT] __cacheline_aligned; +struct kmem_cache kmalloc_caches[KMALLOC_CACHES] __cacheline_aligned;EXPORT_SYMBOL (kmalloc_caches);static int __init setup_slub_min_order (char *str)
@@ -2627,6 +2627,7 @@ static noinline struct kmem_cache *dma_kchar *text;size_t realsize;unsigned long slabflags; +int i; s = kmalloc_caches_dma[index];if (s)
@@ -2647,18 +2648,13 @@ static noinline struct kmem_cache *dma_k text =kasprintf (flags & ~SLUB_DMA ,"kmalloc_dma-%d" , (unsigned int )realsize); -if (flags & __GFP_WAIT) - s =kmalloc (kmem_size, flags & ~SLUB_DMA ); -else { -int i; + s =NULL ; +for (i =0 ; i <KMALLOC_CACHES ; i++) +if (kmalloc_caches[i].size) +break ; - s =NULL ; -for (i =0 ; i <SLUB_PAGE_SHIFT ; i++) -if (kmalloc_caches[i].size) { - s = kmalloc_caches + i; -break ; - } - } +BUG_ON (i >=KMALLOC_CACHES ); + s = kmalloc_caches + i;/* * Must defer sysfs creation to a workqueue because we don't know
@@ -2672,7 +2668,7 @@ static noinline struct kmem_cache *dma_kif (!s || !text || !kmem_cache_open (s, flags, text, realsize,ARCH_KMALLOC_MINALIGN , slabflags,NULL )) { -kfree (s); + s->size =0 ;kfree (text); goto unlock_out; }
PATCH v6 6/7 Make slub statistics use this_cpu_inc
In reply to [email protected]this_cpu_inc() translates into a single instruction on x86 and does not need any register. So use it in stat(). We also want to avoid the calculation of the per cpu kmem_cache_cpu structure pointer. So pass a kmem_cache pointer instead of a kmem_cache_cpu pointer.
--- mm/slub.c | 43 ++++++++++++++++++++----------------------- 1 file changed, 20 insertions(+), 23 deletions(-)
mm/slub.c +20 -23
@@ -217,10 +217,10 @@ static inline void sysfs_slab_remove(str#endif -static inline void stat (struct kmem_cache_cpu *c,enum stat_item si) +static inline void stat (struct kmem_cache *s,enum stat_item si) {#ifdef CONFIG_SLUB_STATS - c->stat[si]++; +__this_cpu_inc (s->cpu_slab->stat[si]);#endif }
@@ -1108,7 +1108,7 @@ static struct page *allocate_slab(structif (!page)return NULL ; -stat (this_cpu_ptr (s->cpu_slab),ORDER_FALLBACK ); +stat (s,ORDER_FALLBACK ); } if (kmemcheck_enabled
@@ -1406,23 +1406,22 @@ static struct page *get_partial(struct kstatic void unfreeze_slab (struct kmem_cache *s,struct page *page,int tail) {struct kmem_cache_node *n =get_node (s,page_to_nid (page)); -struct kmem_cache_cpu *c =this_cpu_ptr (s->cpu_slab);__ClearPageSlubFrozen (page);if (page->inuse) {if (page->freelist) {add_partial (n, page, tail); -stat (c, tail ?DEACTIVATE_TO_TAIL :DEACTIVATE_TO_HEAD ); +stat (s, tail ?DEACTIVATE_TO_TAIL :DEACTIVATE_TO_HEAD ); }else { -stat (c,DEACTIVATE_FULL ); +stat (s,DEACTIVATE_FULL );if (SLABDEBUG &&PageSlubDebug (page) && (s->flags &SLAB_STORE_USER ))add_full (n, page); }slab_unlock (page); }else { -stat (c,DEACTIVATE_EMPTY ); +stat (s,DEACTIVATE_EMPTY );if (n->nr_partial < s->min_partial) {/* * Adding an empty slab to the partial slabs in order
@@ -1438,7 +1437,7 @@ static void unfreeze_slab(struct kmem_caslab_unlock (page); }else {slab_unlock (page); -stat (__this_cpu_ptr (s->cpu_slab),FREE_SLAB ); +stat (s,FREE_SLAB );discard_slab (s, page); } }
@@ -1453,7 +1452,7 @@ static void deactivate_slab(struct kmem_int tail =1 ;if (page->freelist) -stat (c,DEACTIVATE_REMOTE_FREES ); +stat (s,DEACTIVATE_REMOTE_FREES );/* * Merge cpu freelist into slab freelist. Typically we get here * because both freelists are empty. So this is unlikely
@@ -1479,7 +1478,7 @@ static void deactivate_slab(struct kmem_static inline void flush_slab (struct kmem_cache *s,struct kmem_cache_cpu *c) { -stat (c,CPUSLAB_FLUSH ); +stat (s,CPUSLAB_FLUSH );slab_lock (c->page);deactivate_slab (s, c); }
@@ -1619,7 +1618,7 @@ static void *__slab_alloc(struct kmem_caif (unlikely (!node_match (c, node))) goto another_slab; -stat (c,ALLOC_REFILL ); +stat (s,ALLOC_REFILL ); load_freelist: object = c->page->freelist;
@@ -1634,7 +1633,7 @@ load_freelist: c->node =page_to_nid (c->page); unlock_out:slab_unlock (c->page); -stat (c,ALLOC_SLOWPATH ); +stat (s,ALLOC_SLOWPATH );return object; another_slab:
@@ -1644,7 +1643,7 @@ new_slab: new =get_partial (s, gfpflags, node);if (new) { c->page = new; -stat (c,ALLOC_FROM_PARTIAL ); +stat (s,ALLOC_FROM_PARTIAL ); goto load_freelist; }
@@ -1658,7 +1657,7 @@ new_slab:if (new) { c =__this_cpu_ptr (s->cpu_slab); -stat (c,ALLOC_SLAB ); +stat (s,ALLOC_SLAB );if (c->page)flush_slab (s, c);slab_lock (new);
@@ -1713,7 +1712,7 @@ static __always_inline void *slab_alloc(else { c->freelist =get_freepointer (s, object); -stat (c,ALLOC_FASTPATH ); +stat (s,ALLOC_FASTPATH ); }local_irq_restore (flags);
@@ -1780,10 +1779,8 @@ static void __slab_free(struct kmem_cach {void *prior;void **object = (void *)x; -struct kmem_cache_cpu *c; - c =__this_cpu_ptr (s->cpu_slab); -stat (c,FREE_SLOWPATH ); +stat (s,FREE_SLOWPATH );slab_lock (page);if (unlikely (SLABDEBUG &&PageSlubDebug (page)))
@@ -1796,7 +1793,7 @@ checks_ok: page->inuse--;if (unlikely (PageSlubFrozen (page))) { -stat (c,FREE_FROZEN ); +stat (s,FREE_FROZEN ); goto out_unlock; }
@@ -1809,7 +1806,7 @@ checks_ok:*/ if (unlikely (!prior)) {add_partial (get_node (s,page_to_nid (page)), page,1 ); -stat (c,FREE_ADD_PARTIAL ); +stat (s,FREE_ADD_PARTIAL ); } out_unlock:
@@ -1822,10 +1819,10 @@ slab_empty:* Slab still on the partial list. */ remove_partial (s, page); -stat (c,FREE_REMOVE_PARTIAL ); +stat (s,FREE_REMOVE_PARTIAL ); }slab_unlock (page); -stat (c,FREE_SLAB ); +stat (s,FREE_SLAB );discard_slab (s, page);return ;
@@ -1863,7 +1860,7 @@ static __always_inline void slab_free(stif (likely (page == c->page && c->node >=0 )) {set_freepointer (s, object, c->freelist); c->freelist = object; -stat (c,FREE_FASTPATH ); +stat (s,FREE_FASTPATH ); }else __slab_free (s, page, x, addr);
Re: [this_cpu_xx V6 5/7] this_cpu: Remove slub kmem_cache fields
In reply to [email protected]Review static int kmem_cache_open(struct kmem_c · line 2372
Quoted 11 lines · On Wed, 7 Oct 2009, [email protected] wrote:
(quoted diff) +4
-1@@ -2387,8 +2364,11 @@ static int kmem_cache_open(struct kmem_c if (!init_kmem_cache_nodes(s, gfpflags & ~SLUB_DMA)) goto error; - if (alloc_kmem_cache_cpus(s, gfpflags & ~SLUB_DMA)) + if (!alloc_kmem_cache_cpus(s, gfpflags & ~SLUB_DMA)) + + if (s->cpu_slab) return 1; + free_kmem_cache_nodes(s);
Argh. I goofed while fixing a diff problem shortly before release.
The following patch fixes the patch:
--- mm/slub.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
mm/slub.c +1 -3
@@ -2364,9 +2364,7 @@ static int kmem_cache_open(struct kmem_cif (!init_kmem_cache_nodes (s, gfpflags & ~SLUB_DMA )) goto error; -if (!alloc_kmem_cache_cpus (s, gfpflags & ~SLUB_DMA )) - -if (s->cpu_slab) +if (alloc_kmem_cache_cpus (s, gfpflags & ~SLUB_DMA ))return 1 ;free_kmem_cache_nodes (s);
Re: [this_cpu_xx V6 1/7] this_cpu_ops: page allocator conversion
In reply to [email protected]Hello, Christoph.
Quoted 7 lines · [email protected] wrote:
mm/page_alloc.c +7
-0+/* +* Boot pageset table. One per cpu which is going to be used for all +* zones and all nodes. The parameters will be set in such a way +* that an item put on a list will immediately be handed over to +* the buddy list. This is safe since pageset manipulation is done +* with interrupts disabled. +*
mm/page_alloc.c static void build_zonelist_cache(pg_data · line 2749
Quoted 10 lines
mm/page_alloc.c +10
-0+* The boot_pagesets must be kept even after bootup is complete for +* unused processors and/or zones. They do play a role for bootstrapping +* hotplugged processors. +* +* zoneinfo_show() and maybe other functions do +* not check if the processor is online before following the pageset pointer. +* Other parts of the kernel may not check if the zone is available. +*/ +static void setup_pageset (struct per_cpu_pageset *p,unsigned long batch); +static DEFINE_PER_CPU(struct per_cpu_pageset, boot_pageset);
This looks much better but I'm not sure whether it's safe. percpu offsets have not been set up before setup_per_cpu_areas() is complete on most archs but if all that's necessary is getting the page allocator up and running as soon as static per cpu areas and offsets are set up (which basically means as soon as cpu init is complete on ia64 and setup_per_cpu_areas() is complete on all other archs). This should be correct. Is this what you're expecting?
Thanks.
Signature · 1 line
tejun
Re: [this_cpu_xx V6 1/7] this_cpu_ops: page allocator conversion
Quoted 7 lines · Tejun Heo
This looks much better but I'm not sure whether it's safe. percpu offsets have not been set up before
setup_per_cpu_areas()is complete on most archs but if all that's necessary is getting the page allocator up and running as soon as static per cpu areas and offsets are set up (which basically means as soon as cpu init is complete on ia64 andsetup_per_cpu_areas()is complete on all other archs). This should be correct. Is this what you're expecting?
Also, as I'm not very familiar with the code, I'd really appreciate Mel Gorman's acked or reviewed-by.
Thanks.
Signature · 1 line
tejun
Re: [this_cpu_xx V6 1/7] this_cpu_ops: page allocator conversion
In reply to [email protected]Quoted 18 lines · On Wed, Oct 07, 2009 at 05:10:25PM -0400, [email protected] wrote:
Use the per cpu allocator functionality to avoid per cpu arrays in struct zone.
This drastically reduces the size of struct zone for systems with large amounts of processors and allows placement of critical variables of struct zone in one cacheline even on very large systems.
Another effect is that the pagesets of one processor are placed near one another. If multiple pagesets from different zones fit into one cacheline then additional cacheline fetches can be avoided on the hot paths when allocating memory from multiple zones.
Bootstrap becomes simpler if we use the same scheme for UP, SMP, NUMA. #ifdefs are reduced and we can drop the
zone_pcpmacro.Hotplug handling is also simplified since cpu alloc can bring up and shut down cpu areas for a specific cpu as a whole. So there is no need to allocate or free individual pagesets.
Quoted 7 lines
V4-V5:
- Fix up cases whereper_cpu_ptris called before irq disable
- Integrate the bootstrap logic that was separate before.Cc Mel Gorman <[email protected]>Signed-off-by Christoph Lameter <[email protected]>
I haven't tested the patch series but it now looks good to my eyes at least. Thanks
Quoted 448 lines
--- include/linux/mm.h | 4 - include/linux/mmzone.h | 12 --- mm/page_alloc.c | 187 ++++++++++++++++++------------------------------- mm/vmstat.c | 14 ++- 4 files changed, 81 insertions(+), 136 deletions(-)Index:
linux-2.6/include/linux/mm.h
===================================================================include/linux/mm.h +0
-4@@ -1061,11 +1061,7 @@ extern void si_meminfo(struct sysinfo *extern void si_meminfo_node (struct sysinfo *val,int nid);extern int after_bootmem; -#ifdef CONFIG_NUMA extern void setup_per_cpu_pageset (void ); -#else -static inline void setup_per_cpu_pageset (void ) {} -#endif extern void zone_pcp_update (struct zone *zone);Index:
linux-2.6/include/linux/mmzone.h
===================================================================include/linux/mmzone.h +2
-10@@ -184,13 +184,7 @@ struct per_cpu_pageset {s8 stat_threshold;s8 vm_stat_diff[NR_VM_ZONE_STAT_ITEMS ]; #endif -}____cacheline_aligned_in_smp ; - -#ifdef CONFIG_NUMA -#define zone_pcp (__z, __cpu) ((__z)->pageset[(__cpu)]) -#else -#define zone_pcp (__z, __cpu) (&(__z)->pageset[(__cpu)]) -#endif +};#endif /* !__GENERATING_BOUNDS.H */ @@ -306,10 +300,8 @@ struct zone {*/ unsigned long min_unmapped_pages;unsigned long min_slab_pages; -struct per_cpu_pageset *pageset[NR_CPUS ]; -#else -struct per_cpu_pageset pageset[NR_CPUS ];#endif +struct per_cpu_pageset *pageset;/* * free areas of different sizes */ Index:
linux-2.6/mm/page_alloc.c
===================================================================mm/page_alloc.c +71
-116@@ -1011,10 +1011,10 @@ static void drain_pages(unsigned int cpustruct per_cpu_pageset *pset;struct per_cpu_pages *pcp; - pset =zone_pcp (zone, cpu); +local_irq_save (flags); + pset =per_cpu_ptr (zone->pageset, cpu); pcp = &pset->pcp; -local_irq_save (flags);free_pcppages_bulk (zone, pcp->count, pcp); pcp->count =0 ;local_irq_restore (flags);@@ -1098,7 +1098,6 @@ static void free_hot_cold_page(struct paarch_free_page (page,0 );kernel_map_pages (page,1 ,0 ); - pcp = &zone_pcp (zone,get_cpu ())->pcp; migratetype =get_pageblock_migratetype (page);set_page_private (page, migratetype);local_irq_save (flags);@@ -1121,6 +1120,7 @@ static void free_hot_cold_page(struct pa migratetype =MIGRATE_MOVABLE ; } + pcp = &this_cpu_ptr (zone->pageset)->pcp;if (cold)list_add_tail (&page->lru, &pcp->lists[migratetype]);else @@ -1133,7 +1133,6 @@ static void free_hot_cold_page(struct pa out:local_irq_restore (flags); -put_cpu (); }void free_hot_page (struct page *page)@@ -1183,17 +1182,15 @@ struct page *buffered_rmqueue(struct zonunsigned long flags;struct page *page;int cold = !!(gfp_flags & __GFP_COLD); -int cpu; again: - cpu =get_cpu ();if (likely (order ==0 )) {struct per_cpu_pages *pcp;struct list_head *list; - pcp = &zone_pcp (zone, cpu)->pcp; - list = &pcp->lists[migratetype];local_irq_save (flags); + pcp = &this_cpu_ptr (zone->pageset)->pcp; + list = &pcp->lists[migratetype];if (list_empty (list)) { pcp->count += rmqueue_bulk(zone, 0, pcp->batch, list,@@ -1234,7 +1231,6 @@ again:__count_zone_vm_events (PGALLOC , zone,1 << order);zone_statistics (preferred_zone, zone);local_irq_restore (flags); -put_cpu ();VM_BUG_ON (bad_range (zone, page));if (prep_new_page (page, order, gfp_flags))@@ -1243,7 +1239,6 @@ again: failed:local_irq_restore (flags); -put_cpu ();return NULL ; }@@ -2172,7 +2167,7 @@ void show_free_areas(void)for_each_online_cpu (cpu) {struct per_cpu_pageset *pageset; - pageset =zone_pcp (zone, cpu); + pageset =per_cpu_ptr (zone->pageset, cpu); printk("CPU %4d: hi:%5d, btch:%4d usd:%4d\n", cpu, pageset->pcp.high,@@ -2735,10 +2730,29 @@ static void build_zonelist_cache(pg_data#endif /* CONFIG_NUMA */ +/* +* Boot pageset table. One per cpu which is going to be used for all +* zones and all nodes. The parameters will be set in such a way +* that an item put on a list will immediately be handed over to +* the buddy list. This is safe since pageset manipulation is done +* with interrupts disabled. +* +* The boot_pagesets must be kept even after bootup is complete for +* unused processors and/or zones. They do play a role for bootstrapping +* hotplugged processors. +* +* zoneinfo_show() and maybe other functions do +* not check if the processor is online before following the pageset pointer. +* Other parts of the kernel may not check if the zone is available. +*/ +static void setup_pageset (struct per_cpu_pageset *p,unsigned long batch); +static DEFINE_PER_CPU(struct per_cpu_pageset, boot_pageset); +/* return values int ....just for stop_machine() */ static int __build_all_zonelists (void *dummy) {int nid; +int cpu;#ifdef CONFIG_NUMA memset (node_load,0 ,sizeof (node_load));@@ -2749,6 +2763,14 @@ static int __build_all_zonelists(void *dbuild_zonelists (pgdat);build_zonelist_cache (pgdat); } + +/* +* Initialize the boot_pagesets that are going to be used +* for bootstrapping processors. +*/ +for_each_possible_cpu (cpu) +setup_pageset (&per_cpu (boot_pageset, cpu),0 ); +return 0 ; }@@ -3087,120 +3109,60 @@ static void setup_pagelist_highmark(stru } -#ifdef CONFIG_NUMA -/* -* Boot pageset table. One per cpu which is going to be used for all -* zones and all nodes. The parameters will be set in such a way -* that an item put on a list will immediately be handed over to -* the buddy list. This is safe since pageset manipulation is done -* with interrupts disabled. -* -* Some NUMA counter updates may also be caught by the boot pagesets. -* -* The boot_pagesets must be kept even after bootup is complete for -* unused processors and/or zones. They do play a role for bootstrapping -* hotplugged processors. -* -* zoneinfo_show() and maybe other functions do -* not check if the processor is online before following the pageset pointer. -* Other parts of the kernel may not check if the zone is available. -*/ -static struct per_cpu_pageset boot_pageset[NR_CPUS ]; - -/* -* Dynamically allocate memory for the -* per cpu pageset array in struct zone. -*/ -static int __cpuinit process_zones (int cpu) -{ -struct zone *zone, *dzone; -int node =cpu_to_node (cpu); - -node_set_state (node,N_CPU );/* this node has a cpu */ - -for_each_populated_zone (zone) { -zone_pcp (zone, cpu) =kmalloc_node (sizeof (struct per_cpu_pageset ), -GFP_KERNEL , node); -if (!zone_pcp (zone, cpu)) - goto bad; - -setup_pageset (zone_pcp (zone, cpu),zone_batchsize (zone)); - -if (percpu_pagelist_fraction) -setup_pagelist_highmark (zone_pcp (zone, cpu), - (zone->present_pages / percpu_pagelist_fraction)); - } - -return 0 ; -bad: -for_each_zone (dzone) { -if (!populated_zone (dzone)) -continue ; -if (dzone == zone) -break ; -kfree (zone_pcp (dzone, cpu)); -zone_pcp (dzone, cpu) = &boot_pageset[cpu]; - } -return -ENOMEM ; -} - -static inline void free_zone_pagesets (int cpu) -{ -struct zone *zone; - -for_each_zone (zone) { -struct per_cpu_pageset *pset =zone_pcp (zone, cpu); - -/* Free per_cpu_pageset if it is slab allocated */ -if (pset != &boot_pageset[cpu]) -kfree (pset); -zone_pcp (zone, cpu) = &boot_pageset[cpu]; - } -} -static int __cpuinit pageset_cpuup_callback (struct notifier_block *nfb,unsigned long action,void *hcpu) {int cpu = (long )hcpu; -int ret =NOTIFY_OK ;switch (action) {case CPU_UP_PREPARE :case CPU_UP_PREPARE_FROZEN : -if (process_zones (cpu)) - ret =NOTIFY_BAD ; -break ; -case CPU_UP_CANCELED : -case CPU_UP_CANCELED_FROZEN : -case CPU_DEAD : -case CPU_DEAD_FROZEN : -free_zone_pagesets (cpu); +node_set_state (cpu_to_node (cpu),N_CPU );break ;default :break ; } -return ret; +return NOTIFY_OK ; }static struct notifier_block __cpuinitdata pageset_notifier = { &pageset_cpuup_callback,NULL ,0 }; +/* +* Allocate per cpu pagesets and initialize them. +* Before this call only boot pagesets were available. +* Boot pagesets will no longer be used by this processorr +* after setup_per_cpu_pageset(). +*/ void __init setup_per_cpu_pageset (void ) { -int err; +struct zone *zone; +int cpu; + +for_each_populated_zone (zone) { + zone->pageset = alloc_percpu(struct per_cpu_pageset); + +for_each_possible_cpu (cpu) { +struct per_cpu_pageset *pcp =per_cpu_ptr (zone->pageset, cpu); + +setup_pageset (pcp,zone_batchsize (zone)); + +if (percpu_pagelist_fraction) +setup_pagelist_highmark (pcp, + (zone->present_pages / + percpu_pagelist_fraction)); + } + } -/* Initialize per_cpu_pageset for cpu 0. -* A cpuup callback will do this for every cpu -* as it comes online +/* +* The boot cpu is always the first active. +* The boot node has a processor */ - err =process_zones (smp_processor_id ()); -BUG_ON (err); +node_set_state (cpu_to_node (smp_processor_id ()),N_CPU );register_cpu_notifier (&pageset_notifier); } -#endif - static noinline __init_refokint zone_wait_table_init (struct zone *zone,unsigned long zone_size_pages) {@@ -3254,7 +3216,7 @@ static int __zone_pcp_update(void *data)struct per_cpu_pageset *pset;struct per_cpu_pages *pcp; - pset =zone_pcp (zone, cpu); + pset =per_cpu_ptr (zone->pageset, cpu); pcp = &pset->pcp;local_irq_save (flags);@@ -3272,21 +3234,13 @@ void zone_pcp_update(struct zone *zone)static __meminit void zone_pcp_init (struct zone *zone) { -int cpu; -unsigned long batch =zone_batchsize (zone); +/* Use boot pagesets until we have the per cpu allocator up */ + zone->pageset = &per_cpu_var (boot_pageset); -for (cpu =0 ; cpu <NR_CPUS ; cpu++) { -#ifdef CONFIG_NUMA -/* Early boot. Slab allocator not functional yet */ -zone_pcp (zone, cpu) = &boot_pageset[cpu]; -setup_pageset (&boot_pageset[cpu],0 ); -#else -setup_pageset (zone_pcp (zone,cpu), batch); -#endif - }if (zone->present_pages) -printk (KERN_DEBUG " %s zone: %lu pages, LIFO batch:%lu\n" , - zone->name, zone->present_pages, batch); +printk (KERN_DEBUG " %s zone: %lu pages, LIFO batch:%u\n" , + zone->name, zone->present_pages, +zone_batchsize (zone)); } __meminit int init_currently_empty_zone(struct zone *zone,@@ -4800,10 +4754,11 @@ int percpu_pagelist_fraction_sysctl_handif (!write || (ret == -EINVAL ))return ret;for_each_populated_zone (zone) { -for_each_online_cpu (cpu) { +for_each_possible_cpu (cpu) {unsigned long high; high = zone->present_pages / percpu_pagelist_fraction; -setup_pagelist_highmark (zone_pcp (zone, cpu), high); +setup_pagelist_highmark ( +per_cpu_ptr (zone->pageset, cpu), high); } }return 0 ;Index:
linux-2.6/mm/vmstat.c
===================================================================mm/vmstat.c +8
-6@@ -139,7 +139,8 @@ static void refresh_zone_stat_thresholds threshold =calculate_threshold (zone);for_each_online_cpu (cpu) -zone_pcp (zone, cpu)->stat_threshold = threshold; +per_cpu_ptr (zone->pageset, cpu)->stat_threshold + = threshold; } }@@ -149,7 +150,8 @@ static void refresh_zone_stat_thresholdsvoid __mod_zone_page_state (struct zone *zone,enum zone_stat_item item,int delta) { -struct per_cpu_pageset *pcp =zone_pcp (zone,smp_processor_id ()); +struct per_cpu_pageset *pcp =this_cpu_ptr (zone->pageset); +s8 *p = pcp->vm_stat_diff + item;long x;@@ -202,7 +204,7 @@ EXPORT_SYMBOL(mod_zone_page_state);*/ void __inc_zone_state (struct zone *zone,enum zone_stat_item item) { -struct per_cpu_pageset *pcp =zone_pcp (zone,smp_processor_id ()); +struct per_cpu_pageset *pcp =this_cpu_ptr (zone->pageset);s8 *p = pcp->vm_stat_diff + item; (*p)++;@@ -223,7 +225,7 @@ EXPORT_SYMBOL(__inc_zone_page_state);void __dec_zone_state (struct zone *zone,enum zone_stat_item item) { -struct per_cpu_pageset *pcp =zone_pcp (zone,smp_processor_id ()); +struct per_cpu_pageset *pcp =this_cpu_ptr (zone->pageset);s8 *p = pcp->vm_stat_diff + item; (*p)--;@@ -300,7 +302,7 @@ void refresh_cpu_vm_stats(int cpu)for_each_populated_zone (zone) {struct per_cpu_pageset *p; - p =zone_pcp (zone, cpu); + p =per_cpu_ptr (zone->pageset, cpu);for (i =0 ; i <NR_VM_ZONE_STAT_ITEMS ; i++)if (p->vm_stat_diff[i]) {@@ -738,7 +740,7 @@ static void zoneinfo_show_print(struct sfor_each_online_cpu (i) {struct per_cpu_pageset *pageset; - pageset =zone_pcp (zone, i); + pageset =per_cpu_ptr (zone->pageset, i);seq_printf (m,"\n cpu: %i" "\n count: %i"Signature · 1 line
--
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Re: [this_cpu_xx V6 1/7] this_cpu_ops: page allocator conversion
In reply to Tejun HeoQuoted 10 lines · Tejun Heo
mm/page_alloc.c static void build_zonelist_cache(pg_data · line 2749
Quoted 2 lines
mm/page_alloc.c +2
-0+static void setup_pageset (struct per_cpu_pageset *p,unsigned long batch); +static DEFINE_PER_CPU(struct per_cpu_pageset, boot_pageset);This looks much better but I'm not sure whether it's safe. percpu offsets have not been set up before
setup_per_cpu_areas()is complete on most archs but if all that's necessary is getting the page allocator up and running as soon as static per cpu areas and offsets are set up (which basically means as soon as cpu init is complete on ia64 andsetup_per_cpu_areas()is complete on all other archs). This should be correct. Is this what you're expecting?
paging_init() is called after the per cpu areas have been initialized. So I thought this would be safe. Tested it on x86.
zone_pcp_init() only sets up the per cpu pointers to the pagesets. That works regardless of the boot stage. Then then build_all_zonelists() initializes the actual contents of the per cpu variables.
Finally the per cpu pagesets are allocated from the percpu allocator when all allocators are up and the pagesets are sized.
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
In reply to [email protected]Hello,
Quoted 10 lines · [email protected] wrote:
mm/slub.c +1
-1@@ -1507,7 +1498,7 @@ static inline void flush_slab(struct kme*/ static inline void __flush_cpu_slab (struct kmem_cache *s,int cpu) { -struct kmem_cache_cpu *c =get_cpu_slab (s, cpu); +struct kmem_cache_cpu *c =per_cpu_ptr (s->cpu_slab, cpu);if (likely (c && c->page))flush_slab (s, c);
mm/slub.c new_slab: · line 1664
Quoted 5 lines
mm/slub.c +1
-1@@ -1673,7 +1661,7 @@ new_slab:local_irq_disable ();if (new) { - c =get_cpu_slab (s,smp_processor_id ()); + c =__this_cpu_ptr (s->cpu_slab);
Shouldn't this be this_cpu_ptr() without the double underscore?
Thanks.
Signature · 1 line
tejun
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
Quoted 15 lines · Tejun Heo
Hello,
Quoted 12 lines · [email protected] wrote:
mm/slub.c +1
-1@@ -1507,7 +1498,7 @@ static inline void flush_slab(struct kme*/ static inline void __flush_cpu_slab (struct kmem_cache *s,int cpu) { -struct kmem_cache_cpu *c =get_cpu_slab (s, cpu); +struct kmem_cache_cpu *c =per_cpu_ptr (s->cpu_slab, cpu);if (likely (c && c->page))flush_slab (s, c);@@ -1673,7 +1661,7 @@ new_slab:local_irq_disable ();
Quoted 5 lines
mm/slub.c new_slab: · line 1664
Quoted 3 lines
mm/slub.c +1
-1if (new) { - c =get_cpu_slab (s,smp_processor_id ()); + c =__this_cpu_ptr (s->cpu_slab);Shouldn't this be
this_cpu_ptr()without the double underscore?
Oh... another similar conversions in slab_alloc() and slab_free() too.
Thanks.
Signature · 1 line
tejun
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to [email protected]Quoted 7 lines · [email protected] wrote:
Use this_cpu_* operations in the hotpath to avoid calculations of
kmem_cache_cpupointer addresses.On x86 there is a trade off: Multiple uses segment prefixes against an address calculation and more register pressure. Code size is reduced also therefore it is an advantage icache wise.
Quoted 6 lines
The use of prefixes is necessary if we want to use a scheme for fastpaths that do not require disabling interrupts.
Cc Mathieu Desnoyers <[email protected]>Cc Pekka Enberg <[email protected]>Signed-off-by Christoph Lameter <[email protected]>
The rest of the patches look good to me but I'm no expert in this area of code. But you're the maintainer of the allocator and the changes definitely are percpu related, so if you're comfortable with it, I can happily carry the patches through percpu tree.
Thanks.
Signature · 1 line
tejun
Pekka Enberg
Acked-byRe: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
Quoted 11 lines · Tejun Heo
Quoted 10 lines · [email protected] wrote:
Use this_cpu_* operations in the hotpath to avoid calculations of
kmem_cache_cpupointer addresses.On x86 there is a trade off: Multiple uses segment prefixes against an address calculation and more register pressure. Code size is reduced also therefore it is an advantage icache wise.
The use of prefixes is necessary if we want to use a scheme for fastpaths that do not require disabling interrupts.
Quoted 8 lines
Quoted 3 lines
Cc Mathieu Desnoyers <[email protected]>Cc Pekka Enberg <[email protected]>Signed-off-by Christoph Lameter <[email protected]>The rest of the patches look good to me but I'm no expert in this area of code. But you're the maintainer of the allocator and the changes definitely are percpu related, so if you're comfortable with it, I can happily carry the patches through percpu tree.
The patch looks sane to me but the changelog contains no relevant numbers on performance. I am fine with the patch going in -percpu but the patch probably needs some more beating performance-wise before it can go into .33. I'm CC'ing some more people who are known to do SLAB performance testing just in case they're interested in looking at the patch. In any case,
Pekka
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
In reply to Tejun HeoQuoted 4 lines · Tejun Heo
Review
Quoted 2 lines
(quoted diff) +1
-1- c = get_cpu_slab(s, smp_processor_id()); + c = __this_cpu_ptr(s->cpu_slab);Shouldn't this be
this_cpu_ptr()without the double underscore?
Interrupts are disabled so no concurrent fast path can occur.
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to Pekka EnbergQuoted 6 lines · Pekka Enberg
The patch looks sane to me but the changelog contains no relevant numbers on performance. I am fine with the patch going in -percpu but the patch probably needs some more beating performance-wise before it can go into .33. I'm CC'ing some more people who are known to do SLAB performance testing just in case they're interested in looking at the patch. In any case,
I am warming up my synthetic in kernel tests right now. Hope I have something by tomorrow.
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
In reply to Christoph LameterQuoted 8 lines · Christoph Lameter
Quoted 3 lines · Tejun Heo
Review
Quoted 2 lines
(quoted diff) +1
-1- c = get_cpu_slab(s, smp_processor_id()); + c = __this_cpu_ptr(s->cpu_slab);Shouldn't this be
this_cpu_ptr()without the double underscore?Interrupts are disabled so no concurrent fast path can occur.
The only difference between this_cpu_ptr() and __this_cpu_ptr() is the usage of my_cpu_offset and __my_cpu_offset which in turn are only different in whether they check preemption status to make sure the cpu is pinned down when called.
The only places where the underbar prefixed versions should be used are places where cpu locality is nice but not critical and preemption debug check wouldn't work properly for whatever reason. The above is none of the two and the conversion is buried in a patch which is supposed to do something else. Am I missing something?
Signature · 1 line
tejun
David Rientjes
reviewerRe: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to Pekka EnbergQuoted 7 lines · Pekka Enberg
The patch looks sane to me but the changelog contains no relevant numbers on performance. I am fine with the patch going in -percpu but the patch probably needs some more beating performance-wise before it can go into .33. I'm CC'ing some more people who are known to do SLAB performance testing just in case they're interested in looking at the patch. In any case,
I ran 60-second netperf TCP_RR benchmarks with various thread counts over two machines, both four quad-core Opterons. I ran the trials ten times each with both vanilla per-cpu#for-next at 9288f99 and with v6 of this patchset. The transfer rates were virtually identical showing no improvement or regression with this patchset in this benchmark.
[ As I reported in http://marc.info/?l=linux-kernel&m=123839191416472,
this benchmark continues to be the most significant regression slub has compared to slab. ]
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
In reply to Tejun HeoQuoted 4 lines · Tejun Heo
The only difference between
this_cpu_ptr()and__this_cpu_ptr()is the usage ofmy_cpu_offsetand__my_cpu_offsetwhich in turn are only different in whether they check preemption status to make sure the cpu is pinned down when called.
Correct.
Quoted 5 lines
The only places where the underbar prefixed versions should be used are places where cpu locality is nice but not critical and preemption debug check wouldn't work properly for whatever reason. The above is none of the two and the conversion is buried in a patch which is supposed to do something else. Am I missing something?
I used __this_cpu_* whenever the context is already providing enough safety that preempt disable or irq disable would not matter. The use of __this_cpu_ptr was entirely for consistent usage here. this_cpu_ptr would be safer because it has additional checks that preemption really is disabled. So if someone gets confused about logic flow later it can be dtected.
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to David RientjesQuoted 9 lines · David Rientjes
I ran 60-second netperf
TCP_RRbenchmarks with various thread counts over two machines, both four quad-core Opterons. I ran the trials ten times each with both vanilla per-cpu#for-next at 9288f99 and with v6 of this patchset. The transfer rates were virtually identical showing no improvement or regression with this patchset in this benchmark.[ As I reported in http://marc.info/?l=linux-kernel&m=123839191416472,
this benchmark continues to be the most significant regression slub has compared to slab. ]
Hmmm... Last time I ran the in kernel benchmarks this showed a reduction in cycle counts. Did not get to get my tests yet.
Can you also try the irqless hotpath?
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
In reply to Christoph LameterQuoted 15 lines · Christoph Lameter
Quoted 4 lines · Tejun Heo
The only difference between
this_cpu_ptr()and__this_cpu_ptr()is the usage ofmy_cpu_offsetand__my_cpu_offsetwhich in turn are only different in whether they check preemption status to make sure the cpu is pinned down when called.Correct.
Quoted 5 lines
The only places where the underbar prefixed versions should be used are places where cpu locality is nice but not critical and preemption debug check wouldn't work properly for whatever reason. The above is none of the two and the conversion is buried in a patch which is supposed to do something else. Am I missing something?
Quoted 6 lines
I used __this_cpu_* whenever the context is already providing enough safety that preempt disable or irq disable would not matter. The use of
__this_cpu_ptrwas entirely for consistent usage here.this_cpu_ptrwould be safer because it has additional checks that preemption really is disabled. So if someone gets confused about logic flow later it can be dtected.
Yeah, widespread use of underscored versions isn't very desirable. The underscored versions should notify certain specific exceptional conditions instead of being used as general optimization (which doesn't make much sense after all as the optimization is only meaningful with debug option turned on). Are you interested in doing a sweeping patch to drop underscores from __this_cpu_*() conversions?
Thanks.
Signature · 1 line
tejun
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
Quoted 6 lines · Tejun Heo
Yeah, widespread use of underscored versions isn't very desirable. The underscored versions should notify certain specific exceptional conditions instead of being used as general optimization (which doesn't make much sense after all as the optimization is only meaningful with debug option turned on). Are you interested in doing a sweeping patch to drop underscores from __this_cpu_*() conversions?
Nope. __this_cpu_add/dec cannot be converted.
__this_cpu_ptr could be converted to this_cpu_ptr but I think the __ are useful there too to show that we are in a preempt section.
The calls to raw_smp_processor_id and smp_processor_id() are only useful in the fallback case. There is no need for those if the arch has a way to provide the current percpu offset. So we in effect have two meanings of __ right now.
1. We do not care about the preempt state (thus we call raw_smp_processor_id so that the preempt state does not trigger)
2. We do not need to disable preempt before the operation.
__this_cpu_ptr only implies 1. __this_cpu_add uses 1 and 2.
Quoted 5 lines · On Wed, Oct 07, 2009 at 05:10:24PM -0400, [email protected] wrote:
V5->V6:
- Drop patches merged by Tejun.
- Drop irqless slub fastpath for now.
- Patches against Tejun percpu for-next branch.
FWIW, this fails to boot on latest mmotm on x86-64 even though the patches apply. It fails to create basic slab cackes like kmalloc-64.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 2 lines · Mel Gorman
FWIW, this fails to boot on latest mmotm on x86-64 even though the patches apply. It fails to create basic slab cackes like kmalloc-64.
There was a fixup patch for one of the slub patches. Was that merged?
Quoted 7 lines · Christoph Lameter
Quoted 2 lines · Mel Gorman
FWIW, this fails to boot on latest mmotm on x86-64 even though the patches apply. It fails to create basic slab cackes like kmalloc-64.
There was a fixup patch for one of the slub patches. Was that merged?
No. I missed it without the change in subject line and had just exported the thread series itself. Sorry.
I might have something useful on this in the morning assuming no other PEBKAC-related messes.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
I am stuck too. Sysfs is screwed up somehow and triggers the hangcheck timer.
[FIX] patch "SLUB: Get rid of dynamic DMA kmalloc cache allocation"
In reply to [email protected]Slight bug when creating kmalloc dma caches on the fly. When searching for an unused statically allocated kmem_cache structure we need to check for size == 0 not the other way around.
--- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/slub.c +1 -1
@@ -2650,7 +2650,7 @@ static noinline struct kmem_cache *dma_k s =NULL ;for (i =0 ; i <KMALLOC_CACHES ; i++) -if (kmalloc_caches[i].size) +if (!kmalloc_caches[i].size)break ;BUG_ON (i >=KMALLOC_CACHES );
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to Christoph LameterHere are some cycle numbers w/o the slub patches and with. I will post the full test results and the patches to do these in kernel tests in a new thread. The regression may be due to caching behavior of SLUB that will not change with these patches.
Alloc fastpath wins ~ 50%. kfree also has a 50% win if the fastpath is being used. First test does 10000 kmallocs and then frees them all. Second test alloc one and free one and does that 10000 times.
no this_cpu ops
1. Kmalloc: Repeatedly allocate then free test 10000 times kmalloc(8) -> 239 cycles kfree -> 261 cycles 10000 times kmalloc(16) -> 249 cycles kfree -> 208 cycles 10000 times kmalloc(32) -> 215 cycles kfree -> 232 cycles 10000 times kmalloc(64) -> 164 cycles kfree -> 216 cycles 10000 times kmalloc(128) -> 266 cycles kfree -> 275 cycles 10000 times kmalloc(256) -> 478 cycles kfree -> 199 cycles 10000 times kmalloc(512) -> 449 cycles kfree -> 201 cycles 10000 times kmalloc(1024) -> 484 cycles kfree -> 398 cycles 10000 times kmalloc(2048) -> 475 cycles kfree -> 559 cycles 10000 times kmalloc(4096) -> 792 cycles kfree -> 506 cycles 10000 times kmalloc(8192) -> 753 cycles kfree -> 679 cycles 10000 times kmalloc(16384) -> 968 cycles kfree -> 712 cycles
2. Kmalloc: alloc/free test
10000 times kmalloc(8)/kfree -> 292 cycles
10000 times kmalloc(16)/kfree -> 308 cycles
10000 times kmalloc(32)/kfree -> 326 cycles
10000 times kmalloc(64)/kfree -> 303 cycles
10000 times kmalloc(128)/kfree -> 257 cycles
10000 times kmalloc(256)/kfree -> 262 cycles
10000 times kmalloc(512)/kfree -> 293 cycles
10000 times kmalloc(1024)/kfree -> 262 cycles 10000 times kmalloc(2048)/kfree -> 289 cycles 10000 times kmalloc(4096)/kfree -> 274 cycles 10000 times kmalloc(8192)/kfree -> 265 cycles 10000 times kmalloc(16384)/kfree -> 1041 cycles
with this_cpu_xx
1. Kmalloc: Repeatedly allocate then free test 10000 times kmalloc(8) -> 134 cycles kfree -> 212 cycles 10000 times kmalloc(16) -> 109 cycles kfree -> 116 cycles 10000 times kmalloc(32) -> 157 cycles kfree -> 231 cycles 10000 times kmalloc(64) -> 168 cycles kfree -> 169 cycles 10000 times kmalloc(128) -> 263 cycles kfree -> 260 cycles 10000 times kmalloc(256) -> 430 cycles kfree -> 251 cycles 10000 times kmalloc(512) -> 415 cycles kfree -> 258 cycles 10000 times kmalloc(1024) -> 406 cycles kfree -> 432 cycles 10000 times kmalloc(2048) -> 457 cycles kfree -> 579 cycles 10000 times kmalloc(4096) -> 624 cycles kfree -> 553 cycles 10000 times kmalloc(8192) -> 851 cycles kfree -> 851 cycles 10000 times kmalloc(16384) -> 907 cycles kfree -> 722 cycles
2. Kmalloc: alloc/free test
10000 times kmalloc(8)/kfree -> 232 cycles
10000 times kmalloc(16)/kfree -> 150 cycles
10000 times kmalloc(32)/kfree -> 278 cycles
10000 times kmalloc(64)/kfree -> 263 cycles
10000 times kmalloc(128)/kfree -> 280 cycles
10000 times kmalloc(256)/kfree -> 279 cycles
10000 times kmalloc(512)/kfree -> 299 cycles
10000 times kmalloc(1024)/kfree -> 289 cycles 10000 times kmalloc(2048)/kfree -> 288 cycles 10000 times kmalloc(4096)/kfree -> 321 cycles 10000 times kmalloc(8192)/kfree -> 285 cycles 10000 times kmalloc(16384)/kfree -> 1002 cycles
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
Hi Christoph,
Quoted 8 lines · Christoph Lameter
Here are some cycle numbers w/o the slub patches and with. I will post the full test results and the patches to do these in kernel tests in a new thread. The regression may be due to caching behavior of SLUB that will not change with these patches.
Alloc fastpath wins ~ 50%. kfree also has a 50% win if the fastpath is being used. First test does 10000 kmallocs and then frees them all. Second test alloc one and free one and does that 10000 times.
I wonder how reliable these numbers are. We did similar testing a while back because we thought kmalloc-96 caches had weird cache behavior but finally figured out the anomaly was explained by the order of the tests run, not cache size.
AFAICT, we have similar artifact in these tests as well:
Quoted 7 lines
no
this_cpuops1. Kmalloc: Repeatedly allocate then free test 10000 times kmalloc(8) -> 239 cycles kfree -> 261 cycles 10000 times kmalloc(16) -> 249 cycles kfree -> 208 cycles 10000 times kmalloc(32) -> 215 cycles kfree -> 232 cycles 10000 times kmalloc(64) -> 164 cycles kfree -> 216 cycles
Notice the jump from 32 to 64 and then back to 64. One would expect we see linear increase as object size grows as we hit the page allocator more often, no?
Quoted 23 lines
10000 times kmalloc(128) -> 266 cycles kfree -> 275 cycles 10000 times kmalloc(256) -> 478 cycles kfree -> 199 cycles 10000 times kmalloc(512) -> 449 cycles kfree -> 201 cycles 10000 times kmalloc(1024) -> 484 cycles kfree -> 398 cycles 10000 times kmalloc(2048) -> 475 cycles kfree -> 559 cycles 10000 times kmalloc(4096) -> 792 cycles kfree -> 506 cycles 10000 times kmalloc(8192) -> 753 cycles kfree -> 679 cycles 10000 times kmalloc(16384) -> 968 cycles kfree -> 712 cycles
2. Kmalloc: alloc/free test
10000 times kmalloc(8)/kfree -> 292 cycles
10000 times kmalloc(16)/kfree -> 308 cycles
10000 times kmalloc(32)/kfree -> 326 cycles
10000 times kmalloc(64)/kfree -> 303 cycles
10000 times kmalloc(128)/kfree -> 257 cycles
10000 times kmalloc(256)/kfree -> 262 cycles
10000 times kmalloc(512)/kfree -> 293 cycles
10000 times kmalloc(1024)/kfree -> 262 cycles 10000 times kmalloc(2048)/kfree -> 289 cycles 10000 times kmalloc(4096)/kfree -> 274 cycles 10000 times kmalloc(8192)/kfree -> 265 cycles 10000 times kmalloc(16384)/kfree -> 1041 cycles
Quoted 5 lines
with
this_cpu_xx1. Kmalloc: Repeatedly allocate then free test 10000 times kmalloc(8) -> 134 cycles kfree -> 212 cycles 10000 times kmalloc(16) -> 109 cycles kfree -> 116 cycles
Same artifact here.
Quoted 10 lines
10000 times kmalloc(32) -> 157 cycles kfree -> 231 cycles 10000 times kmalloc(64) -> 168 cycles kfree -> 169 cycles 10000 times kmalloc(128) -> 263 cycles kfree -> 260 cycles 10000 times kmalloc(256) -> 430 cycles kfree -> 251 cycles 10000 times kmalloc(512) -> 415 cycles kfree -> 258 cycles 10000 times kmalloc(1024) -> 406 cycles kfree -> 432 cycles 10000 times kmalloc(2048) -> 457 cycles kfree -> 579 cycles 10000 times kmalloc(4096) -> 624 cycles kfree -> 553 cycles 10000 times kmalloc(8192) -> 851 cycles kfree -> 851 cycles 10000 times kmalloc(16384) -> 907 cycles kfree -> 722 cycles
And looking at these numbers:
Quoted 5 lines
2. Kmalloc: alloc/free test
10000 times kmalloc(8)/kfree -> 232 cycles
10000 times kmalloc(16)/kfree -> 150 cycles
10000 times kmalloc(32)/kfree -> 278 cycles
10000 times kmalloc(64)/kfree -> 263 cycles
Quoted 8 lines
10000 times kmalloc(128)/kfree -> 280 cycles
10000 times kmalloc(256)/kfree -> 279 cycles
10000 times kmalloc(512)/kfree -> 299 cycles
10000 times kmalloc(1024)/kfree -> 289 cycles
10000 times kmalloc(2048)/kfree -> 288 cycles
10000 times kmalloc(4096)/kfree -> 321 cycles
10000 times kmalloc(8192)/kfree -> 285 cycles
10000 times kmalloc(16384)/kfree -> 1002 cycles
If there's 50% improvement in the kmalloc() path, why does the this_cpu() version seem to be roughly as fast as the mainline version?
Pekka
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
Quoted 4 lines · Pekka Enberg
I wonder how reliable these numbers are. We did similar testing a while back because we thought kmalloc-96 caches had weird cache behavior but finally figured out the anomaly was explained by the order of the tests run, not cache size.
Well you need to look behind these numbers to see when the allocator uses the fastpath or slow path. Only the fast path is optimized here.
Quoted 10 lines
AFAICT, we have similar artifact in these tests as well:
Quoted 7 lines
no
this_cpuops1. Kmalloc: Repeatedly allocate then free test 10000 times kmalloc(8) -> 239 cycles kfree -> 261 cycles 10000 times kmalloc(16) -> 249 cycles kfree -> 208 cycles 10000 times kmalloc(32) -> 215 cycles kfree -> 232 cycles 10000 times kmalloc(64) -> 164 cycles kfree -> 216 cycles
Quoted 3 lines
Notice the jump from 32 to 64 and then back to 64. One would expect we see linear increase as object size grows as we hit the page allocator more often, no?
64 is the cacheline size for the machine. At that point you have the advantage of no overlapping data between different allocations and the prefetcher may do a particularly good job.
Quoted 4 lines
Quoted 1 line
10000 times kmalloc(16384)/kfree -> 1002 cycles
If there's 50% improvement in the
kmalloc()path, why does thethis_cpu()version seem to be roughly as fast as the mainline version?
Its not that the kmalloc() is faster. The instructions used for the fastpath generate less cycles. Other components figure into the total latency as well.
16k allocations for example are not handled by slub anymore. Fastpath has no effect. The wins there is just the improved percpu handling in the page allocator.
I have some numbers here for irqless which drops another half of the fastpath latency (and it adds some code to the slow path, sigh):
1. Kmalloc: Repeatedly allocate then free test 10000 times kmalloc(8) -> 55 cycles kfree -> 251 cycles 10000 times kmalloc(16) -> 201 cycles kfree -> 261 cycles 10000 times kmalloc(32) -> 220 cycles kfree -> 261 cycles 10000 times kmalloc(64) -> 186 cycles kfree -> 224 cycles 10000 times kmalloc(128) -> 205 cycles kfree -> 125 cycles 10000 times kmalloc(256) -> 351 cycles kfree -> 267 cycles 10000 times kmalloc(512) -> 330 cycles kfree -> 310 cycles 10000 times kmalloc(1024) -> 416 cycles kfree -> 419 cycles 10000 times kmalloc(2048) -> 537 cycles kfree -> 439 cycles 10000 times kmalloc(4096) -> 458 cycles kfree -> 594 cycles 10000 times kmalloc(8192) -> 810 cycles kfree -> 678 cycles 10000 times kmalloc(16384) -> 879 cycles kfree -> 746 cycles
2. Kmalloc: alloc/free test
10000 times kmalloc(8)/kfree -> 66 cycles
10000 times kmalloc(16)/kfree -> 187 cycles
10000 times kmalloc(32)/kfree -> 116 cycles
10000 times kmalloc(64)/kfree -> 107 cycles
10000 times kmalloc(128)/kfree -> 115 cycles
10000 times kmalloc(256)/kfree -> 65 cycles
10000 times kmalloc(512)/kfree -> 66 cycles
10000 times kmalloc(1024)/kfree -> 206 cycles 10000 times kmalloc(2048)/kfree -> 65 cycles
10000 times kmalloc(4096)/kfree -> 193 cycles 10000 times kmalloc(8192)/kfree -> 65 cycles
10000 times kmalloc(16384)/kfree -> 976 cycles
David Rientjes
reviewerRe: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
Quoted 8 lines · Christoph Lameter
Quoted 4 lines
I wonder how reliable these numbers are. We did similar testing a while back because we thought kmalloc-96 caches had weird cache behavior but finally figured out the anomaly was explained by the order of the tests run, not cache size.
Well you need to look behind these numbers to see when the allocator uses the fastpath or slow path. Only the fast path is optimized here.
With the netperf -t TCP_RR -l 60 benchmark I ran, CONFIG_SLUB_STATS shows the allocation fastpath is utilized quite a bit for a couple of key caches:
cache ALLOC_FASTPATH ALLOC_SLOWPATH kmalloc-256 98125871 31585955 kmalloc-2048 77243698 52347453
For an optimized fastpath, I'd expect such a workload would result in at least a slightly higher transfer rate.
I'll try the irqless patch, but this particular benchmark may not appropriately demonstrate any performance gain because of the added code in the also significantly-used slowpath.
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to Pekka EnbergQuoted 4 lines · Pekka Enberg
I wonder how reliable these numbers are. We did similar testing a while back because we thought kmalloc-96 caches had weird cache behavior but finally figured out the anomaly was explained by the order of the tests run, not cache size.
The tests were all run directly after booting the respective kernel.
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to David RientjesQuoted 2 lines · David Rientjes
For an optimized fastpath, I'd expect such a workload would result in at least a slightly higher transfer rate.
There will be no improvements if the load is dominated by the instructions in the network layer or caching issues. None of that is changed by the path. It only reduces the cycle count in the fastpath.
David Rientjes
reviewerRe: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
Quoted 7 lines · Christoph Lameter
Quoted 2 lines
For an optimized fastpath, I'd expect such a workload would result in at least a slightly higher transfer rate.
There will be no improvements if the load is dominated by the instructions in the network layer or caching issues. None of that is changed by the path. It only reduces the cycle count in the fastpath.
Right, but CONFIG_SLAB shows a 5-6% improvement over CONFIG_SLUB in the same workload so it shows that the slab allocator does have an impact in transfer rate. I understand that the performance gain with this patchset, however, may not be representative with the benchmark since it also frequently uses the slowpath for kmalloc-256 about 25% of the time and the added code of the irqless patch may mask the fastpath gain.
David Rientjes
reviewerRe: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to Christoph LameterQuoted 6 lines · Christoph Lameter
Quoted 6 lines
I ran 60-second netperf
TCP_RRbenchmarks with various thread counts over two machines, both four quad-core Opterons. I ran the trials ten times each with both vanilla per-cpu#for-next at 9288f99 and with v6 of this patchset. The transfer rates were virtually identical showing no improvement or regression with this patchset in this benchmark.
Quoted 9 lines
Quoted 3 lines
[ As I reported in http://marc.info/?l=linux-kernel&m=123839191416472,
this benchmark continues to be the most significant regression slub has compared to slab. ]Hmmm... Last time I ran the in kernel benchmarks this showed a reduction in cycle counts. Did not get to get my tests yet.
Can you also try the irqless hotpath?
v6 of your patchset applied to percpu#for-next now at dec54bf "this_cpu:
Use this_cpu_xx in trace_functions_graph.c" works fine, but when I apply the irqless patch from http://marc.info/?l=linux-kernel&m=125503037213262
it hangs my netserver machine within the first 60 seconds when running this benchmark. These kernels both include the fixes to kmem_cache_open() and dma_kmalloc_cache() you posted earlier. I'll have to debug why that's happening before collecting results.
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
In reply to Christoph LameterHello, Christoph.
Quoted 8 lines · Christoph Lameter
Quoted 6 lines
Yeah, widespread use of underscored versions isn't very desirable. The underscored versions should notify certain specific exceptional conditions instead of being used as general optimization (which doesn't make much sense after all as the optimization is only meaningful with debug option turned on). Are you interested in doing a sweeping patch to drop underscores from __this_cpu_*() conversions?
Nope. __this_cpu_add/dec cannot be converted.
Right.
Quoted 2 lines
__this_cpu_ptrcould be converted tothis_cpu_ptrbut I think the __ are useful there too to show that we are in a preempt section.
That doesn't make much sense. __ for this_cpu_ptr() means "bypass sanity check, we're knowingly violating the required conditions" not "we know sanity checks will pass here".
Quoted 11 lines
The calls to
raw_smp_processor_idandsmp_processor_id()are only useful in the fallback case. There is no need for those if the arch has a way to provide the current percpu offset. So we in effect have two meanings of __ right now.1. We do not care about the preempt state (thus we call
raw_smp_processor_idso that the preempt state does not trigger)2. We do not need to disable preempt before the operation.
__this_cpu_ptronly implies 1.__this_cpu_adduses 1 and 2.
Yeah, we need to clean it up. The naming is too confusing.
Thanks.
Signature · 1 line
tejun
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to David RientjesQuoted 10 lines · David Rientjes
Quoted 7 lines · Christoph Lameter
Quoted 2 lines
For an optimized fastpath, I'd expect such a workload would result in at least a slightly higher transfer rate.
There will be no improvements if the load is dominated by the instructions in the network layer or caching issues. None of that is changed by the path. It only reduces the cycle count in the fastpath.
Quoted 7 lines
Right, but
CONFIG_SLABshows a 5-6% improvement overCONFIG_SLUBin the same workload so it shows that the slab allocator does have an impact in transfer rate. I understand that the performance gain with this patchset, however, may not be representative with the benchmark since it also frequently uses the slowpath for kmalloc-256 about 25% of the time and the added code of the irqless patch may mask the fastpath gain.
I have a bit more detailed results based on the following machine
CPU type: AMD Phenom 9950
CPU counts: 1 CPU (4 cores)
CPU Speed: 1.3GHz
Motherboard: Gigabyte GA-MA78GM-S2H
Memory: 8GB
The reference kernel used is mmotm-2009-10-09-01-07. The patches applied are the patches in this thread. The headings are a bit munged but it's
SLUB-vanilla where vanilla is mmotm-2009-10-09-01-07
SLUB-this-cpu mmotm-2009-10-09-01-07 + patches in this thread
SLAB-* same as above but SLAB configured instead of SLUB.
I know it wasn't necessary to run SLAB-this-cpu but it gives an idea to what degree results can vary between reboots even if results are stable once the machine is running.
The benchmarks run were kernbench, netperf UDP_STREAM and TCP_STREAM and sysbench with postgres.
Kernbench is 5 kernel compiles and an average taken. One kernel compile is done at the start to warm the benchmark up and this result is discarded.
Netperf is the _STREAM tests as opposed to the _RR tests reported elsewhere. No special effort is done to bind processes to any particular CPU. The results reported tried to be 99% confidence that the estimated mean was within 1% of the true mean. Results where netperf failed to achieve the necessary confidence are marked with a * and the line after such a result states what percentage the estimated mean is to the true mean. The test is run with different packet sizes.
Sysbench is a read-only test (to avoid IO) and is the "complex" workload. The test is run with varying numbers of threads.
In all the results, SLUB-vanilla is the reference baseline. This allows a comparison between SLUB-vanilla and SLAB-vanilla as well with the patches applied.
kernbench-SLUB-vanilla-kernbench kernbench-SLUBkernbench-SLAB-vanilla-kernbench kernbench-SLAB
SLUB-vanilla this-cpu SLAB-vanilla this-cpu
Elapsed min 92.95 ( 0.00%) 92.62 ( 0.36%) 92.93 ( 0.02%) 92.62 ( 0.36%) Elapsed mean 93.11 ( 0.00%) 92.74 ( 0.40%) 93.00 ( 0.13%) 92.82 ( 0.32%) Elapsed stddev 0.10 ( 0.00%) 0.14 (-40.55%) 0.04 (55.47%) 0.18 (-84.33%) Elapsed max 93.20 ( 0.00%) 92.95 ( 0.27%) 93.05 ( 0.16%) 93.09 ( 0.12%) User min 323.21 ( 0.00%) 322.60 ( 0.19%) 322.50 ( 0.22%) 323.26 (-0.02%) User mean 323.81 ( 0.00%) 323.20 ( 0.19%) 323.16 ( 0.20%) 323.54 ( 0.08%) User stddev 0.40 ( 0.00%) 0.46 (-15.30%) 0.48 (-20.92%) 0.29 (26.07%) User max 324.32 ( 0.00%) 323.72 ( 0.19%) 323.86 ( 0.14%) 323.98 ( 0.10%) System min 35.95 ( 0.00%) 35.50 ( 1.25%) 35.35 ( 1.67%) 36.01 (-0.17%) System mean 36.30 ( 0.00%) 35.96 ( 0.96%) 36.17 ( 0.36%) 36.23 ( 0.21%) System stddev 0.25 ( 0.00%) 0.45 (-75.60%) 0.56 (-121.14%) 0.14 (46.14%) System max 36.65 ( 0.00%) 36.67 (-0.05%) 36.94 (-0.79%) 36.39 ( 0.71%) CPU min 386.00 ( 0.00%) 386.00 ( 0.00%) 386.00 ( 0.00%) 386.00 ( 0.00%) CPU mean 386.25 ( 0.00%) 386.75 (-0.13%) 386.00 ( 0.06%) 387.25 (-0.26%) CPU stddev 0.43 ( 0.00%) 0.83 (-91.49%) 0.00 (100.00%) 0.83 (-91.49%) CPU max 387.00 ( 0.00%) 388.00 (-0.26%) 386.00 ( 0.26%) 388.00 (-0.26%)
Small gains in the User, System and Elapsed times with this-cpu patches applied. It is interest to note for the mean times that the patches more than close the gap between SLUB and SLAB for the most part - the exception being User which has marginally better performance. This might indicate that SLAB is still slightly better at giving back cache-hot memory but this is speculation.
NETPERF UDP_STREAM
Packet netperf-udp udp-SLUB netperf-udp udp-SLAB
Size SLUB-vanilla this-cpu SLAB-vanilla this-cpu
64 148.48 ( 0.00%) 152.03 ( 2.34%) 147.45 (-0.70%) 150.07 ( 1.06%)
128 294.65 ( 0.00%) 299.92 ( 1.76%) 289.20 (-1.88%) 290.15 (-1.55%)
256 583.63 ( 0.00%) 609.14 ( 4.19%) 590.78 ( 1.21%) 586.42 ( 0.48%)
1024 2217.90 ( 0.00%) 2261.99 ( 1.95%) 2219.64 ( 0.08%) 2207.93 (-0.45%)
2048 4164.27 ( 0.00%) 4161.47 (-0.07%) 4216.46 ( 1.24%) 4155.11 (-0.22%)
3312 6284.17 ( 0.00%) 6383.24 ( 1.55%) 6231.88 (-0.84%) 6243.82 (-0.65%)
4096 7399.42 ( 0.00%) 7686.38 ( 3.73%) 7394.89 (-0.06%) 7487.91 ( 1.18%)
6144 10014.35 ( 0.00%) 10199.48 ( 1.82%) 9927.92 (-0.87%)* 10067.40 ( 0.53%)
1.00% 1.00% 1.08% 1.00%
8192 11232.50 ( 0.00%)* 11368.13 ( 1.19%)* 12280.88 ( 8.54%)* 12244.23 ( 8.26%)
1.65% 1.64% 1.32% 1.00%
10240 12961.87 ( 0.00%) 13099.82 ( 1.05%)* 13816.33 ( 6.18%)* 13927.18 ( 6.93%)
1.00% 1.03% 1.21% 1.00%
12288 14403.74 ( 0.00%)* 14276.89 (-0.89%)* 15173.09 ( 5.07%)* 15464.05 ( 6.86%)*
1.31% 1.63% 1.93% 1.55%
14336 15229.98 ( 0.00%)* 15218.52 (-0.08%)* 16412.94 ( 7.21%) 16252.98 ( 6.29%)
1.37% 2.76% 1.00% 1.00%
16384 15367.60 ( 0.00%)* 16038.71 ( 4.18%) 16635.91 ( 7.62%) 17128.87 (10.28%)*
1.29% 1.00% 1.00% 6.36% The patches mostly improve the performance of netperf UDP_STREAM by a good whack so the patches are a plus here. However, it should also be noted that SLAB was mostly faster than SLUB, particularly for large packet sizes. Refresh my memory, how do SLUB and SLAB differ in regards to off-loading large allocations to the page allocator these days?
NETPERF TCP_STREAM
Packet netperf-tcp tcp-SLUB netperf-tcp tcp-SLAB
Size SLUB-vanilla this-cpu SLAB-vanilla this-cpu
64 1773.00 ( 0.00%) 1731.63 (-2.39%)* 1794.48 ( 1.20%) 2029.46 (12.64%)
1.00% 2.43% 1.00% 1.00%
128 3181.12 ( 0.00%) 3471.22 ( 8.36%) 3296.37 ( 3.50%) 3251.33 ( 2.16%)
256 4794.35 ( 0.00%) 4797.38 ( 0.06%) 4912.99 ( 2.41%) 4846.86 ( 1.08%)
1024 9438.10 ( 0.00%) 8681.05 (-8.72%)* 8270.58 (-14.12%) 8268.85 (-14.14%)
1.00% 7.31% 1.00% 1.00%
2048 9196.06 ( 0.00%) 9375.72 ( 1.92%) 11474.59 (19.86%) 9420.01 ( 2.38%)
3312 10338.49 ( 0.00%)* 10021.82 (-3.16%)* 12018.72 (13.98%)* 12069.28 (14.34%)*
9.49% 6.36% 1.21% 2.12%
4096 9931.20 ( 0.00%)* 10285.38 ( 3.44%)* 12265.59 (19.03%)* 10175.33 ( 2.40%)*
1.31% 1.38% 9.97% 8.33%
6144 12775.08 ( 0.00%)* 10559.63 (-20.98%) 13139.34 ( 2.77%) 13210.79 ( 3.30%)*
1.45% 1.00% 1.00% 2.99%
8192 10933.93 ( 0.00%)* 10534.41 (-3.79%)* 10876.42 (-0.53%)* 10738.25 (-1.82%)*
14.29% 2.10% 12.50% 9.55%
10240 12868.58 ( 0.00%) 12991.65 ( 0.95%) 10892.20 (-18.14%) 13106.01 ( 1.81%)
12288 11854.97 ( 0.00%) 12122.34 ( 2.21%)* 12129.79 ( 2.27%)* 12411.84 ( 4.49%)*
1.00% 6.61% 5.78% 8.95%
14336 12552.48 ( 0.00%)* 12501.71 (-0.41%)* 12274.54 (-2.26%) 12322.63 (-1.87%)*
6.05% 2.58% 1.00% 2.23%
16384 11733.09 ( 0.00%)* 12735.05 ( 7.87%)* 13195.68 (11.08%)* 14401.62 (18.53%)
1.14% 9.79% 10.30% 1.00% The results for the patches are a bit all over the place for TCP_STREAM with big gains and losses depending on the packet size, particularly 6144 for some reason. SLUB vs SLAB shows SLAB often has really massive advantages and this is not always for the larger packet sizes where the page allocator might be a suspect.
SYSBENCH
sysbench-SLUB-vanilla-sysbench sysbench-SLUBsysbench-SLAB-vanilla-sysbench sysbench-SLAB
SLUB-vanilla this-cpu SLAB-vanilla this-cpu
1 26950.79 ( 0.00%) 26822.05 (-0.48%) 26919.89 (-0.11%) 26746.18 (-0.77%)
2 51555.51 ( 0.00%) 51928.02 ( 0.72%) 51370.02 (-0.36%) 51129.82 (-0.83%)
3 76204.23 ( 0.00%) 76333.58 ( 0.17%) 76483.99 ( 0.37%) 75954.52 (-0.33%)
4 100599.12 ( 0.00%) 101757.98 ( 1.14%) 100499.65 (-0.10%) 101605.61 ( 0.99%)
5 100211.45 ( 0.00%) 100435.33 ( 0.22%) 100150.98 (-0.06%) 99398.11 (-0.82%)
6 99390.81 ( 0.00%) 99840.85 ( 0.45%) 99234.38 (-0.16%) 99244.42 (-0.15%)
7 98740.56 ( 0.00%) 98727.61 (-0.01%) 98305.88 (-0.44%) 98123.56 (-0.63%)
8 98075.89 ( 0.00%) 98048.62 (-0.03%) 98183.99 ( 0.11%) 97587.82 (-0.50%)
9 96502.22 ( 0.00%) 97276.80 ( 0.80%) 96819.88 ( 0.33%) 97320.51 ( 0.84%)
10 96598.70 ( 0.00%) 96545.37 (-0.06%) 96222.51 (-0.39%) 96221.69 (-0.39%)
11 95500.66 ( 0.00%) 95671.11 ( 0.18%) 95003.21 (-0.52%) 95246.81 (-0.27%)
12 94572.87 ( 0.00%) 95266.70 ( 0.73%) 93807.60 (-0.82%) 94859.82 ( 0.30%)
13 93811.85 ( 0.00%) 94309.18 ( 0.53%) 93219.81 (-0.64%) 93051.63 (-0.82%)
14 92972.16 ( 0.00%) 93849.87 ( 0.94%) 92641.50 (-0.36%) 92916.70 (-0.06%)
15 92276.06 ( 0.00%) 92454.94 ( 0.19%) 91094.04 (-1.30%) 91972.79 (-0.33%)
16 90265.35 ( 0.00%) 90416.26 ( 0.17%) 89309.26 (-1.07%) 90103.89 (-0.18%)The patches mostly gain for sysbench although the gains are very marginal and SLUB has a minor advantage over SLAB. I haven't actually checked how slab-intensive this workload is. The differences are no marginal, I would guess the answer is "not very".
Overall based on these results, I would say that the patches are a "Good Thing" for this machine at least. With the patches applied, SLUB has a marginal advantage over SLAB for kernbench. However, netperf TCP_STREAM and UDP_STREAM both show significant disadvantages for SLUB and this cannot be always explained by differing behaviour with respect to page-allocator offloading.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
The test did not include the irqless patch I hope?
Quoted 6 lines · Mel Gorman
Small gains in the User, System and Elapsed times with this-cpu patches applied. It is interest to note for the mean times that the patches more than close the gap between SLUB and SLAB for the most part - the exception being User which has marginally better performance. This might indicate that SLAB is still slightly better at giving back cache-hot memory but this is speculation.
The queuing in SLAB allows a better cache hot behavior. Without a queue SLUB has a difficult time improvising cache hot behavior based on objects restricted to a slab page. Therefore the size of the slab page will affect how much "queueing" SLUB can do.
Quoted 5 lines
The patches mostly improve the performance of netperf
UDP_STREAMby a good whack so the patches are a plus here. However, it should also be noted that SLAB was mostly faster than SLUB, particularly for large packet sizes. Refresh my memory, how do SLUB and SLAB differ in regards to off-loading large allocations to the page allocator these days?
SLUB offloads allocations > 8k to the page allocator. SLAB does create large slabs.
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
In reply to Tejun HeoQuoted 6 lines · Tejun Heo
Quoted 2 lines
__this_cpu_ptrcould be converted tothis_cpu_ptrbut I think the __ are useful there too to show that we are in a preempt section.That doesn't make much sense. __ for
this_cpu_ptr()means "bypass sanity check, we're knowingly violating the required conditions" not "we know sanity checks will pass here".
Are you defining what __ means for this_cpu_ptr?
Quoted 5 lines
Quoted 5 lines
The calls to
raw_smp_processor_idandsmp_processor_id()are only useful in the fallback case. There is no need for those if the arch has a way to provide the current percpu offset. So we in effect have two meanings of __ right now.
Quoted 8 lines
Quoted 6 lines
1. We do not care about the preempt state (thus we call
raw_smp_processor_idso that the preempt state does not trigger)2. We do not need to disable preempt before the operation.
__this_cpu_ptronly implies 1.__this_cpu_adduses 1 and 2.Yeah, we need to clean it up. The naming is too confusing.
Its consistent if __ means both 1 and 2. If we want to distinguish it then we may want to create raw_this_cpu_xx which means that we do not call smp_processor_id() on fallback but raw_smp_processor_id(). Does not matter if the arch provides a per cpu offset.
This would mean duplicating all the macros. The use of raw_this_cpu_xx should be rare so maybe the best approach is to say that __ means only that the macro does not need to disable preempt but it still checks for preemption being off. Then audit the __this_cpu_xx uses and see if there are any that require a raw_ variant.
The vm event counters require both no check and no preempt since they can be implemented in a racy way.
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to Christoph LameterQuoted 2 lines · Christoph Lameter
The test did not include the irqless patch I hope?
Correct. Only the patches in this thread were tested.
Quoted 9 lines · Mel Gorman
Quoted 7 lines
Small gains in the User, System and Elapsed times with this-cpu patches applied. It is interest to note for the mean times that the patches more than close the gap between SLUB and SLAB for the most part - the exception being User which has marginally better performance. This might indicate that SLAB is still slightly better at giving back cache-hot memory but this is speculation.
Quoted 5 lines
The queuing in SLAB allows a better cache hot behavior. Without a queue SLUB has a difficult time improvising cache hot behavior based on objects restricted to a slab page. Therefore the size of the slab page will affect how much "queueing" SLUB can do.
Ok, so the speculation is a plausible explanation.
Quoted 9 lines
Quoted 5 lines
The patches mostly improve the performance of netperf
UDP_STREAMby a good whack so the patches are a plus here. However, it should also be noted that SLAB was mostly faster than SLUB, particularly for large packet sizes. Refresh my memory, how do SLUB and SLAB differ in regards to off-loading large allocations to the page allocator these days?SLUB offloads allocations > 8k to the page allocator. SLAB does create large slabs.
Allocations >8k might explain then why 8K and 16K packets for UDP_STREAM performance suffers. That can be marked as future possible work to sort out within the allocator.
However, does it explain why TCP_STREAM suffers so badly even for packet sizes like 2K? It's also important to note in some cases, SLAB was far slower even when the packet sizes were greater than 8k so I don't think the page allocator is an adequate explanation for TCP_STREAM.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
Hi Mel,
Quoted 9 lines · Mel Gorman
Quoted 8 lines
Quoted 5 lines
The patches mostly improve the performance of netperf
UDP_STREAMby a good whack so the patches are a plus here. However, it should also be noted that SLAB was mostly faster than SLUB, particularly for large packet sizes. Refresh my memory, how do SLUB and SLAB differ in regards to off-loading large allocations to the page allocator these days?SLUB offloads allocations > 8k to the page allocator. SLAB does create large slabs.
Quoted 8 lines
Allocations >8k might explain then why 8K and 16K packets for
UDP_STREAMperformance suffers. That can be marked as future possible work to sort out within the allocator.However, does it explain why
TCP_STREAMsuffers so badly even for packet sizes like 2K? It's also important to note in some cases, SLAB was far slower even when the packet sizes were greater than 8k so I don't think the page allocator is an adequate explanation forTCP_STREAM.
SLAB is able to queue lots of large objects but SLUB can't do that because it has no queues. In SLUB, each CPU gets a page assigned to it that serves as a "queue" but the size of the queue gets smaller as object size approaches page size.
We try to offset that with higher order allocations but IIRC we don't increase the order linearly with object size and cap it to some reasonable maximum.
Pekka
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
Quoted 7 lines · Pekka Enberg
SLAB is able to queue lots of large objects but SLUB can't do that because it has no queues. In SLUB, each CPU gets a page assigned to it that serves as a "queue" but the size of the queue gets smaller as object size approaches page size.
We try to offset that with higher order allocations but IIRC we don't increase the order linearly with object size and cap it to some reasonable maximum.
You can test to see if larger pages have an influence by passing
slub_max_order=6
or so on the kernel command line.
You can force a large page use in slub by setting
slub_min_order=3
f.e.
Or you can force a mininum number of objecxcts in slub through f.e.
slub_min_objects=50
slub_max_order=6 slub_min_objects=50
should result in pretty large slabs with lots of in page objects that allow slub to queue better.
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
Hi Christoph,
Quoted 7 lines · Pekka Enberg
Quoted 7 lines
SLAB is able to queue lots of large objects but SLUB can't do that because it has no queues. In SLUB, each CPU gets a page assigned to it that serves as a "queue" but the size of the queue gets smaller as object size approaches page size.
We try to offset that with higher order allocations but IIRC we don't increase the order linearly with object size and cap it to some reasonable maximum.
Quoted 12 lines · Christoph Lameter
You can test to see if larger pages have an influence by passing
slub_max_order=6
or so on the kernel command line.
You can force a large page use in slub by setting
slub_min_order=3
f.e.
Quoted 8 lines
Or you can force a mininum number of objecxcts in slub through f.e.
slub_min_objects=50
slub_max_order=6 slub_min_objects=50
should result in pretty large slabs with lots of in page objects that allow slub to queue better.
Yeah, that should help but it's probably not something we can do for mainline. I'm not sure how we can fix SLUB to support large objects out-of-the-box as efficiently as SLAB does.
Pekka
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
Quoted 3 lines · Pekka Enberg
Yeah, that should help but it's probably not something we can do for mainline. I'm not sure how we can fix SLUB to support large objects out-of-the-box as efficiently as SLAB does.
We could add a per cpu "queue" through a pointer array in kmem_cache_cpu. Which is more SLQB than SLUB.
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
In reply to Christoph LameterQuoted 9 lines · Christoph Lameter
Quoted 5 lines · Tejun Heo
Quoted 2 lines
__this_cpu_ptrcould be converted tothis_cpu_ptrbut I think the __ are useful there too to show that we are in a preempt section.That doesn't make much sense. __ for
this_cpu_ptr()means "bypass sanity check, we're knowingly violating the required conditions" not "we know sanity checks will pass here".Are you defining what __ means for
this_cpu_ptr?
I was basically stating the different between raw_smp_processor_id() and smp_processor_id() which I thought applied the same to __this_cpu_ptr() and this_cpu_ptr().
Quoted 19 lines
Quoted 5 lines
Quoted 5 lines
The calls to
raw_smp_processor_idandsmp_processor_id()are only useful in the fallback case. There is no need for those if the arch has a way to provide the current percpu offset. So we in effect have two meanings of __ right now.Quoted 8 lines
Quoted 6 lines
1. We do not care about the preempt state (thus we call
raw_smp_processor_idso that the preempt state does not trigger)2. We do not need to disable preempt before the operation.
__this_cpu_ptronly implies 1.__this_cpu_adduses 1 and 2.Yeah, we need to clean it up. The naming is too confusing.
Its consistent if __ means both 1 and 2. If we want to distinguish it then we may want to create
raw_this_cpu_xxwhich means that we do not callsmp_processor_id()on fallback butraw_smp_processor_id(). Does not matter if the arch provides a per cpu offset.
Quoted 8 lines
This would mean duplicating all the macros. The use of
raw_this_cpu_xxshould be rare so maybe the best approach is to say that __ means only that the macro does not need to disable preempt but it still checks for preemption being off. Then audit the__this_cpu_xxuses and see if there are any that require araw_variant.The vm event counters require both no check and no preempt since they can be implemented in a racy way.
The biggest grief I have is that the meaning of __ is different among different accessors. If that can be cleared up, we would be in much better shape without adding any extra macros. Can we just remove all __'s and use meaningful pre or suffixes like raw or irq or whatever?
Thanks.
Signature · 1 line
tejun
David Rientjes
reviewerRe: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to Mel GormanQuoted 26 lines · Mel Gorman
NETPERF
TCP_STREAMPacket netperf-tcp tcp-SLUB netperf-tcp tcp-SLAB Size SLUB-vanilla this-cpu SLAB-vanilla this-cpu 64 1773.00 ( 0.00%) 1731.63 (-2.39%)* 1794.48 ( 1.20%) 2029.46 (12.64%) 1.00% 2.43% 1.00% 1.00% 128 3181.12 ( 0.00%) 3471.22 ( 8.36%) 3296.37 ( 3.50%) 3251.33 ( 2.16%) 256 4794.35 ( 0.00%) 4797.38 ( 0.06%) 4912.99 ( 2.41%) 4846.86 ( 1.08%) 1024 9438.10 ( 0.00%) 8681.05 (-8.72%)* 8270.58 (-14.12%) 8268.85 (-14.14%) 1.00% 7.31% 1.00% 1.00% 2048 9196.06 ( 0.00%) 9375.72 ( 1.92%) 11474.59 (19.86%) 9420.01 ( 2.38%) 3312 10338.49 ( 0.00%)* 10021.82 (-3.16%)* 12018.72 (13.98%)* 12069.28 (14.34%)* 9.49% 6.36% 1.21% 2.12% 4096 9931.20 ( 0.00%)* 10285.38 ( 3.44%)* 12265.59 (19.03%)* 10175.33 ( 2.40%)* 1.31% 1.38% 9.97% 8.33% 6144 12775.08 ( 0.00%)* 10559.63 (-20.98%) 13139.34 ( 2.77%) 13210.79 ( 3.30%)* 1.45% 1.00% 1.00% 2.99% 8192 10933.93 ( 0.00%)* 10534.41 (-3.79%)* 10876.42 (-0.53%)* 10738.25 (-1.82%)* 14.29% 2.10% 12.50% 9.55% 10240 12868.58 ( 0.00%) 12991.65 ( 0.95%) 10892.20 (-18.14%) 13106.01 ( 1.81%) 12288 11854.97 ( 0.00%) 12122.34 ( 2.21%)* 12129.79 ( 2.27%)* 12411.84 ( 4.49%)* 1.00% 6.61% 5.78% 8.95% 14336 12552.48 ( 0.00%)* 12501.71 (-0.41%)* 12274.54 (-2.26%) 12322.63 (-1.87%)* 6.05% 2.58% 1.00% 2.23% 16384 11733.09 ( 0.00%)* 12735.05 ( 7.87%)* 13195.68 (11.08%)* 14401.62 (18.53%) 1.14% 9.79% 10.30% 1.00%
Quoted 6 lines
The results for the patches are a bit all over the place for
TCP_STREAMwith big gains and losses depending on the packet size, particularly 6144 for some reason. SLUB vs SLAB shows SLAB often has really massive advantages and this is not always for the larger packet sizes where the page allocator might be a suspect.
TCP_STREAM stresses a few specific caches:
ALLOC_FASTPATH ALLOC_SLOWPATH FREE_FASTPATH FREE_SLOWPATH
kmalloc-256 3868530 3450592 95628 7223491
kmalloc-1024 2440434 429 2430825 10034
kmalloc-4096 3860625 1036723 85571 4811779
This demonstrates that freeing to full (or partial) slabs causes a lot of pain since the fastpath normally can't be utilized and that's probably beyond the scope of this patchset.
It's also different from the cpu slab thrashing issue I identified with the TCP_RR benchmark and had a patchset to somewhat improve. The criticism was the addition of an increment to a fastpath counter in struct kmem_cache_cpu which could probably now be much cheaper with these optimizations.
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to Christoph LameterQuoted 24 lines · Christoph Lameter
Quoted 7 lines · Pekka Enberg
SLAB is able to queue lots of large objects but SLUB can't do that because it has no queues. In SLUB, each CPU gets a page assigned to it that serves as a "queue" but the size of the queue gets smaller as object size approaches page size.
We try to offset that with higher order allocations but IIRC we don't increase the order linearly with object size and cap it to some reasonable maximum.
You can test to see if larger pages have an influence by passing
slub_max_order=6
or so on the kernel command line.
You can force a large page use in slub by setting
slub_min_order=3
f.e.
Or you can force a mininum number of objecxcts in slub through f.e.
Quoted 9 lines
slub_min_objects=50
slub_max_order=6 slub_min_objects=50
should result in pretty large slabs with lots of in page objects that allow slub to queue better.
Here are the results of that suggestion. They are side-by-side with the other results so the columns are
SLUB-vanilla No other patches applied, SLUB configured
vanilla-highorder No other patches + slub_max_order=6 slub_min_objects=50
SLUB-this-cpu The patches in this set applied
this-cpu-higher These patches + slub_max_order=6 slub_min_objects=50
SLAB-vanilla No other patches, SLAB configured
SLAB-this-cpu Thes patches, SLAB configured
SLUB-vanilla vanilla-highorder SLUB-this-cpu this-cpu-highorder SLAB-vanilla SLAB-this-cpu
Elapsed min 92.95 ( 0.00%) 92.64 ( 0.33%) 92.62 ( 0.36%) 92.77 ( 0.19%) 92.93 ( 0.02%) 92.62 ( 0.36%) Elapsed mean 93.11 ( 0.00%) 92.89 ( 0.24%) 92.74 ( 0.40%) 92.82 ( 0.31%) 93.00 ( 0.13%) 92.82 ( 0.32%) Elapsed stddev 0.10 ( 0.00%) 0.15 (-58.74%) 0.14 (-40.55%) 0.09 ( 7.73%) 0.04 (55.47%) 0.18 (-84.33%) Elapsed max 93.20 ( 0.00%) 93.04 ( 0.17%) 92.95 ( 0.27%) 92.98 ( 0.24%) 93.05 ( 0.16%) 93.09 ( 0.12%) User min 323.21 ( 0.00%) 323.38 (-0.05%) 322.60 ( 0.19%) 323.26 (-0.02%) 322.50 ( 0.22%) 323.26 (-0.02%) User mean 323.81 ( 0.00%) 323.64 ( 0.05%) 323.20 ( 0.19%) 323.56 ( 0.08%) 323.16 ( 0.20%) 323.54 ( 0.08%) User stddev 0.40 ( 0.00%) 0.38 ( 4.24%) 0.46 (-15.30%) 0.27 (33.20%) 0.48 (-20.92%) 0.29 (26.07%) User max 324.32 ( 0.00%) 324.30 ( 0.01%) 323.72 ( 0.19%) 323.96 ( 0.11%) 323.86 ( 0.14%) 323.98 ( 0.10%) System min 35.95 ( 0.00%) 35.33 ( 1.72%) 35.50 ( 1.25%) 35.95 ( 0.00%) 35.35 ( 1.67%) 36.01 (-0.17%) System mean 36.30 ( 0.00%) 35.99 ( 0.87%) 35.96 ( 0.96%) 36.20 ( 0.28%) 36.17 ( 0.36%) 36.23 ( 0.21%) System stddev 0.25 ( 0.00%) 0.41 (-59.25%) 0.45 (-75.60%) 0.15 (41.61%) 0.56 (-121.14%) 0.14 (46.14%) System max 36.65 ( 0.00%) 36.44 ( 0.57%) 36.67 (-0.05%) 36.32 ( 0.90%) 36.94 (-0.79%) 36.39 ( 0.71%) CPU min 386.00 ( 0.00%) 386.00 ( 0.00%) 386.00 ( 0.00%) 386.00 ( 0.00%) 386.00 ( 0.00%) 386.00 ( 0.00%) CPU mean 386.25 ( 0.00%) 386.75 (-0.13%) 386.75 (-0.13%) 386.75 (-0.13%) 386.00 ( 0.06%) 387.25 (-0.26%) CPU stddev 0.43 ( 0.00%) 0.83 (-91.49%) 0.83 (-91.49%) 0.43 ( 0.00%) 0.00 (100.00%) 0.83 (-91.49%) CPU max 387.00 ( 0.00%) 388.00 (-0.26%) 388.00 (-0.26%) 387.00 ( 0.00%) 386.00 ( 0.26%) 388.00 (-0.26%)
The high-order allocations help here, but not by a massive amount. In some cases it made things slightly worse. However, the standard deviations are generally high enough to file most of the results under "noise"
NETPERF UDP
SLUB-vanilla vanilla-highorder SLUB-this-cpu this-cpu-highorder SLAB-vanilla SLAB-this-cpu
64 148.48 ( 0.00%) 146.28 (-1.50%) 152.03 ( 2.34%) 152.20 ( 2.44%) 147.45 (-0.70%) 150.07 ( 1.06%)
128 294.65 ( 0.00%) 286.80 (-2.74%) 299.92 ( 1.76%) 302.55 ( 2.61%) 289.20 (-1.88%) 290.15 (-1.55%)
256 583.63 ( 0.00%) 564.84 (-3.33%) 609.14 ( 4.19%) 587.53 ( 0.66%) 590.78 ( 1.21%) 586.42 ( 0.48%)
1024 2217.90 ( 0.00%) 2176.12 (-1.92%) 2261.99 ( 1.95%) 2312.12 ( 4.08%) 2219.64 ( 0.08%) 2207.93 (-0.45%)
2048 4164.27 ( 0.00%) 4154.96 (-0.22%) 4161.47 (-0.07%) 4244.60 ( 1.89%) 4216.46 ( 1.24%) 4155.11 (-0.22%)
3312 6284.17 ( 0.00%) 6121.32 (-2.66%) 6383.24 ( 1.55%) 6356.61 ( 1.14%) 6231.88 (-0.84%) 6243.82 (-0.65%)
4096 7399.42 ( 0.00%) 7327.40 (-0.98%)* 7686.38 ( 3.73%) 7633.64 ( 3.07%) 7394.89 (-0.06%) 7487.91 ( 1.18%)
1.00% 1.07% 1.00% 1.00% 1.00% 1.00%
6144 10014.35 ( 0.00%) 10061.59 ( 0.47%) 10199.48 ( 1.82%) 10223.16 ( 2.04%) 9927.92 (-0.87%)* 10067.40 ( 0.53%)
1.00% 1.00% 1.00% 1.00% 1.08% 1.00%
8192 11232.50 ( 0.00%)* 11222.92 (-0.09%)* 11368.13 ( 1.19%)* 11403.82 ( 1.50%)* 12280.88 ( 8.54%)* 12244.23 ( 8.26%)
1.65% 1.37% 1.64% 1.16% 1.32% 1.00%
10240 12961.87 ( 0.00%) 12746.40 (-1.69%)* 13099.82 ( 1.05%)* 12767.02 (-1.53%)* 13816.33 ( 6.18%)* 13927.18 ( 6.93%)
1.00% 2.34% 1.03% 1.26% 1.21% 1.00%
12288 14403.74 ( 0.00%)* 14136.36 (-1.89%)* 14276.89 (-0.89%)* 14246.18 (-1.11%)* 15173.09 ( 5.07%)* 15464.05 ( 6.86%)*
1.31% 1.60% 1.63% 1.60% 1.93% 1.55%
14336 15229.98 ( 0.00%)* 14962.61 (-1.79%)* 15218.52 (-0.08%)* 15243.51 ( 0.09%) 16412.94 ( 7.21%) 16252.98 ( 6.29%)
1.37% 1.66% 2.76% 1.00% 1.00% 1.00%
16384 15367.60 ( 0.00%)* 15543.13 ( 1.13%)* 16038.71 ( 4.18%) 15870.54 ( 3.17%)* 16635.91 ( 7.62%) 17128.87 (10.28%)*
1.29% 1.34% 1.00% 2.18% 1.00% 6.36% Configuring use of high-order pages actually hurt SLUB mostly on the unpatched kernel. The results are mixed with the patches applied. Hard to draw anything very conclusive to be honest. Based on these results, I wouldn't push the high-order allocations aggressively.
NETPERF TCP
SLUB-vanilla vanilla-highorder SLUB-this-cpu this-cpu-highorder SLAB-vanilla SLAB-this-cpu
64 1773.00 ( 0.00%) 1812.07 ( 2.16%)* 1731.63 (-2.39%)* 1717.99 (-3.20%)* 1794.48 ( 1.20%) 2029.46 (12.64%)
1.00% 5.88% 2.43% 2.83% 1.00% 1.00%
128 3181.12 ( 0.00%) 3193.06 ( 0.37%)* 3471.22 ( 8.36%) 3154.79 (-0.83%) 3296.37 ( 3.50%) 3251.33 ( 2.16%)
1.00% 1.70% 1.00% 1.00% 1.00% 1.00%
256 4794.35 ( 0.00%) 4813.37 ( 0.40%) 4797.38 ( 0.06%) 4819.16 ( 0.51%) 4912.99 ( 2.41%) 4846.86 ( 1.08%)
1024 9438.10 ( 0.00%) 8144.02 (-15.89%) 8681.05 (-8.72%)* 8204.11 (-15.04%) 8270.58 (-14.12%) 8268.85 (-14.14%)
1.00% 1.00% 7.31% 1.00% 1.00% 1.00%
2048 9196.06 ( 0.00%) 11233.72 (18.14%) 9375.72 ( 1.92%) 10487.89 (12.32%)* 11474.59 (19.86%) 9420.01 ( 2.38%)
1.00% 1.00% 1.00% 9.43% 1.00% 1.00%
3312 10338.49 ( 0.00%)* 9730.79 (-6.25%)* 10021.82 (-3.16%)* 10089.90 (-2.46%)* 12018.72 (13.98%)* 12069.28 (14.34%)*
9.49% 2.51% 6.36% 5.96% 1.21% 2.12%
4096 9931.20 ( 0.00%)* 12447.88 (20.22%) 10285.38 ( 3.44%)* 10548.56 ( 5.85%)* 12265.59 (19.03%)* 10175.33 ( 2.40%)*
1.31% 1.00% 1.38% 8.22% 9.97% 8.33%
6144 12775.08 ( 0.00%)* 10489.24 (-21.79%)* 10559.63 (-20.98%) 11033.15 (-15.79%)* 13139.34 ( 2.77%) 13210.79 ( 3.30%)*
1.45% 8.46% 1.00% 12.65% 1.00% 2.99%
8192 10933.93 ( 0.00%)* 10340.42 (-5.74%)* 10534.41 (-3.79%)* 10845.36 (-0.82%)* 10876.42 (-0.53%)* 10738.25 (-1.82%)*
14.29% 2.38% 2.10% 1.83% 12.50% 9.55%
10240 12868.58 ( 0.00%) 11211.60 (-14.78%)* 12991.65 ( 0.95%) 11330.97 (-13.57%)* 10892.20 (-18.14%) 13106.01 ( 1.81%)
1.00% 11.36% 1.00% 6.64% 1.00% 1.00%
12288 11854.97 ( 0.00%) 11854.51 (-0.00%) 12122.34 ( 2.21%)* 12258.61 ( 3.29%)* 12129.79 ( 2.27%)* 12411.84 ( 4.49%)*
1.00% 1.00% 6.61% 5.69% 5.78% 8.95%
14336 12552.48 ( 0.00%)* 12309.15 (-1.98%) 12501.71 (-0.41%)* 13683.57 ( 8.27%)* 12274.54 (-2.26%) 12322.63 (-1.87%)*
6.05% 1.00% 2.58% 2.46% 1.00% 2.23%
16384 11733.09 ( 0.00%)* 11856.66 ( 1.04%)* 12735.05 ( 7.87%)* 13482.61 (12.98%)* 13195.68 (11.08%)* 14401.62 (18.53%)
1.14% 1.05% 9.79% 11.52% 10.30% 1.00% Configuring high-rder helper in a few cases here and in one or two cases close the gap with SLAB, particularly for large packet sizes. However, it still suffered for the small packet sizes.
SYSBENCH
SLUB-vanilla vanilla-highorder SLUB-this-cpu this-cpu-highorder SLAB-vanilla SLAB-this-cpu
1 26950.79 ( 0.00%) 26723.98 (-0.85%) 26822.05 (-0.48%) 26877.71 (-0.27%) 26919.89 (-0.11%) 26746.18 (-0.77%)
2 51555.51 ( 0.00%) 51231.41 (-0.63%) 51928.02 ( 0.72%) 51794.47 ( 0.46%) 51370.02 (-0.36%) 51129.82 (-0.83%)
3 76204.23 ( 0.00%) 76060.77 (-0.19%) 76333.58 ( 0.17%) 76270.53 ( 0.09%) 76483.99 ( 0.37%) 75954.52 (-0.33%)
4 100599.12 ( 0.00%) 100825.16 ( 0.22%) 101757.98 ( 1.14%) 100273.02 (-0.33%) 100499.65 (-0.10%) 101605.61 ( 0.99%)
5 100211.45 ( 0.00%) 100096.77 (-0.11%) 100435.33 ( 0.22%) 101132.16 ( 0.91%) 100150.98 (-0.06%) 99398.11 (-0.82%)
6 99390.81 ( 0.00%) 99305.36 (-0.09%) 99840.85 ( 0.45%) 99200.53 (-0.19%) 99234.38 (-0.16%) 99244.42 (-0.15%)
7 98740.56 ( 0.00%) 98625.23 (-0.12%) 98727.61 (-0.01%) 98470.75 (-0.27%) 98305.88 (-0.44%) 98123.56 (-0.63%)
8 98075.89 ( 0.00%) 97609.30 (-0.48%) 98048.62 (-0.03%) 97092.44 (-1.01%) 98183.99 ( 0.11%) 97587.82 (-0.50%)
9 96502.22 ( 0.00%) 96685.39 ( 0.19%) 97276.80 ( 0.80%) 96800.23 ( 0.31%) 96819.88 ( 0.33%) 97320.51 ( 0.84%)
10 96598.70 ( 0.00%) 96272.05 (-0.34%) 96545.37 (-0.06%) 95936.97 (-0.69%) 96222.51 (-0.39%) 96221.69 (-0.39%)
11 95500.66 ( 0.00%) 95141.00 (-0.38%) 95671.11 ( 0.18%) 96057.84 ( 0.58%) 95003.21 (-0.52%) 95246.81 (-0.27%)
12 94572.87 ( 0.00%) 94811.46 ( 0.25%) 95266.70 ( 0.73%) 93767.06 (-0.86%) 93807.60 (-0.82%) 94859.82 ( 0.30%)
13 93811.85 ( 0.00%) 93597.39 (-0.23%) 94309.18 ( 0.53%) 93323.96 (-0.52%) 93219.81 (-0.64%) 93051.63 (-0.82%)
14 92972.16 ( 0.00%) 92936.53 (-0.04%) 93849.87 ( 0.94%) 92545.83 (-0.46%) 92641.50 (-0.36%) 92916.70 (-0.06%)
15 92276.06 ( 0.00%) 91559.63 (-0.78%) 92454.94 ( 0.19%) 91748.29 (-0.58%) 91094.04 (-1.30%) 91972.79 (-0.33%)
16 90265.35 ( 0.00%) 89707.32 (-0.62%) 90416.26 ( 0.17%) 89253.93 (-1.13%) 89309.26 (-1.07%) 90103.89 (-0.18%)High-order didn't really help here either.
Overall, it would appear that high-order allocations occasionally help but the margins are pretty small.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
In reply to Tejun HeoQuoted 5 lines · Tejun Heo
Quoted 1 line
Are you defining what __ means for
this_cpu_ptr?I was basically stating the different between
raw_smp_processor_id()andsmp_processor_id()which I thought applied the same to__this_cpu_ptr()andthis_cpu_ptr().
Ii does apply. __this_cpu_ptr does not use smp_processor_id() but raw_smp_processor_id(). this_cpu_ptr does not need to disable preempt so we dont do anything on that level.
Quoted 7 lines
Quoted 2 lines
The vm event counters require both no check and no preempt since they can be implemented in a racy way.
The biggest grief I have is that the meaning of __ is different among different accessors. If that can be cleared up, we would be in much better shape without adding any extra macros. Can we just remove all __'s and use meaningful pre or suffixes like raw or irq or whatever?
It currently means that we do not deal with preempt and do not check for preemption. That is consistent.
Sure we could change the API to have even more macros than the large amount it already has so that we can check for proper preempt disablement.
I guess that would mean adding
raw_nopreempt_this_cpu_xx and nopreempt_this_cpu_xx variants? The thing gets huge. I think we could just leave it. __ suggests that serialization and checking is not performed like in the full versions and that is true.
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to David RientjesQuoted 11 lines · David Rientjes
TCP_STREAMstresses a few specific caches:
ALLOC_FASTPATHALLOC_SLOWPATHFREE_FASTPATHFREE_SLOWPATH
kmalloc-256 3868530 3450592 95628 7223491
kmalloc-1024 2440434 429 2430825 10034
kmalloc-4096 3860625 1036723 85571 4811779This demonstrates that freeing to full (or partial) slabs causes a lot of pain since the fastpath normally can't be utilized and that's probably beyond the scope of this patchset.
Quoted 5 lines
It's also different from the cpu slab thrashing issue I identified with the
TCP_RRbenchmark and had a patchset to somewhat improve. The criticism was the addition of an increment to a fastpath counter in structkmem_cache_cpuwhich could probably now be much cheaper with these optimizations.
Can you redo the patch?
David Rientjes
reviewerRe: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to Mel GormanQuoted 28 lines · Mel Gorman
NETPERF TCP
SLUB-vanilla vanilla-highorder SLUB-this-cpu this-cpu-highorder SLAB-vanilla SLAB-this-cpu 64 1773.00 ( 0.00%) 1812.07 ( 2.16%)* 1731.63 (-2.39%)* 1717.99 (-3.20%)* 1794.48 ( 1.20%) 2029.46 (12.64%) 1.00% 5.88% 2.43% 2.83% 1.00% 1.00% 128 3181.12 ( 0.00%) 3193.06 ( 0.37%)* 3471.22 ( 8.36%) 3154.79 (-0.83%) 3296.37 ( 3.50%) 3251.33 ( 2.16%) 1.00% 1.70% 1.00% 1.00% 1.00% 1.00% 256 4794.35 ( 0.00%) 4813.37 ( 0.40%) 4797.38 ( 0.06%) 4819.16 ( 0.51%) 4912.99 ( 2.41%) 4846.86 ( 1.08%) 1024 9438.10 ( 0.00%) 8144.02 (-15.89%) 8681.05 (-8.72%)* 8204.11 (-15.04%) 8270.58 (-14.12%) 8268.85 (-14.14%) 1.00% 1.00% 7.31% 1.00% 1.00% 1.00% 2048 9196.06 ( 0.00%) 11233.72 (18.14%) 9375.72 ( 1.92%) 10487.89 (12.32%)* 11474.59 (19.86%) 9420.01 ( 2.38%) 1.00% 1.00% 1.00% 9.43% 1.00% 1.00% 3312 10338.49 ( 0.00%)* 9730.79 (-6.25%)* 10021.82 (-3.16%)* 10089.90 (-2.46%)* 12018.72 (13.98%)* 12069.28 (14.34%)* 9.49% 2.51% 6.36% 5.96% 1.21% 2.12% 4096 9931.20 ( 0.00%)* 12447.88 (20.22%) 10285.38 ( 3.44%)* 10548.56 ( 5.85%)* 12265.59 (19.03%)* 10175.33 ( 2.40%)* 1.31% 1.00% 1.38% 8.22% 9.97% 8.33% 6144 12775.08 ( 0.00%)* 10489.24 (-21.79%)* 10559.63 (-20.98%) 11033.15 (-15.79%)* 13139.34 ( 2.77%) 13210.79 ( 3.30%)* 1.45% 8.46% 1.00% 12.65% 1.00% 2.99% 8192 10933.93 ( 0.00%)* 10340.42 (-5.74%)* 10534.41 (-3.79%)* 10845.36 (-0.82%)* 10876.42 (-0.53%)* 10738.25 (-1.82%)* 14.29% 2.38% 2.10% 1.83% 12.50% 9.55% 10240 12868.58 ( 0.00%) 11211.60 (-14.78%)* 12991.65 ( 0.95%) 11330.97 (-13.57%)* 10892.20 (-18.14%) 13106.01 ( 1.81%) 1.00% 11.36% 1.00% 6.64% 1.00% 1.00% 12288 11854.97 ( 0.00%) 11854.51 (-0.00%) 12122.34 ( 2.21%)* 12258.61 ( 3.29%)* 12129.79 ( 2.27%)* 12411.84 ( 4.49%)* 1.00% 1.00% 6.61% 5.69% 5.78% 8.95% 14336 12552.48 ( 0.00%)* 12309.15 (-1.98%) 12501.71 (-0.41%)* 13683.57 ( 8.27%)* 12274.54 (-2.26%) 12322.63 (-1.87%)* 6.05% 1.00% 2.58% 2.46% 1.00% 2.23% 16384 11733.09 ( 0.00%)* 11856.66 ( 1.04%)* 12735.05 ( 7.87%)* 13482.61 (12.98%)* 13195.68 (11.08%)* 14401.62 (18.53%) 1.14% 1.05% 9.79% 11.52% 10.30% 1.00%
Quoted 4 lines
Configuring high-rder helper in a few cases here and in one or two cases close the gap with SLAB, particularly for large packet sizes. However, it still suffered for the small packet sizes.
This is understandable considering the statistics that I posted for this workload on my machine, higher order cpu slabs will naturally get freed to more often from the fastpath, which also causes it to utilize the allocation fastpath more often (and we can see the optimization of this patchset), in addition to avoiding partial list handling.
The pain with the smaller packet sizes is probably the overhead from the page allocator more than slub, a characteristic that also caused the TCP_RR benchmark to suffer. It can be mitigated somewhat with slab preallocation or a higher min_partial setting, but that's probably not an optimal solution.
David Rientjes
reviewerRe: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
In reply to Christoph LameterQuoted 11 lines · Christoph Lameter
Quoted 11 lines
TCP_STREAMstresses a few specific caches:
ALLOC_FASTPATHALLOC_SLOWPATHFREE_FASTPATHFREE_SLOWPATH
kmalloc-256 3868530 3450592 95628 7223491
kmalloc-1024 2440434 429 2430825 10034
kmalloc-4096 3860625 1036723 85571 4811779This demonstrates that freeing to full (or partial) slabs causes a lot of pain since the fastpath normally can't be utilized and that's probably beyond the scope of this patchset.
Quoted 8 lines
Quoted 5 lines
It's also different from the cpu slab thrashing issue I identified with the
TCP_RRbenchmark and had a patchset to somewhat improve. The criticism was the addition of an increment to a fastpath counter in structkmem_cache_cpuwhich could probably now be much cheaper with these optimizations.Can you redo the patch?
Sure, but it would be even more inexpensive if we can figure out why the irqless patch is hanging my netserver machine within the first 60 seconds on the TCP_RR benchmark. I guess nobody else has reproduced that yet.
Re: [this_cpu_xx V6 7/7] this_cpu: slub aggressive use of this_cpu operations in the hotpaths
Quoted 3 lines · David Rientjes
Sure, but it would be even more inexpensive if we can figure out why the irqless patch is hanging my netserver machine within the first 60 seconds on the
TCP_RRbenchmark. I guess nobody else has reproduced that yet.
Nope. Sorry. I have tried running some tests but so far nothing.
Re: [this_cpu_xx V6 3/7] Use this_cpu operations in slub
In reply to Christoph LameterHello, Christoph.
Quoted 7 lines · Christoph Lameter
Quoted 4 lines
The biggest grief I have is that the meaning of __ is different among different accessors. If that can be cleared up, we would be in much better shape without adding any extra macros. Can we just remove all __'s and use meaningful pre or suffixes like raw or irq or whatever?
It currently means that we do not deal with preempt and do not check for preemption. That is consistent.
If you define it inclusively, it can be consistent.
Quoted 8 lines
Sure we could change the API to have even more macros than the large amount it already has so that we can check for proper preempt disablement.
I guess that would mean adding
raw_nopreempt_this_cpu_xxandnopreempt_this_cpu_xxvariants? The thing gets huge. I think we could just leave it. __ suggests that serialization and checking is not performed like in the full versions and that is true.
I don't think we'll need to add new variants. Just renaming existing ones so that they have more specific pre/suffix should make things clearer. I'll give a shot at that once the sparse annotation patchset is merged.
Thanks.
Signature · 1 line
tejun