mm/slub: Fix count_partial() problem
v3 cover mm/slub: Fix count_partial() problem
count_partial() can hold n->list_lock spinlock for quite long, which makes much trouble to the system. This series eliminate this problem.
v1->v2:
- Improved changelog and variable naming for PATCH 1~2.
- PATCH3 adds per-cpu counter to avoid performance regression in concurrent __slab_free().
v2->v3:
- Changed "page->inuse" to the safe "new.inuse", etc.
- Used CONFIG_SLUB_DEBUG and CONFIG_SYSFS condition for new counters.
- atomic_long_t -> unsigned long
[Testing]
There seems might be a little performance impact under extreme __slab_free() concurrent calls according to my tests.
On my 32-cpu 2-socket physical machine:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
1) perf stat --null --repeat 10 -- hackbench 20 thread 20000
== original, no patched
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.536050899 seconds time elapsed ( +- 0.24% )
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.588049142 seconds time elapsed ( +- 0.35% )
== patched with patch1~4
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.670892273 seconds time elapsed ( +- 0.29% )
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.746755689 seconds time elapsed ( +- 0.21% )
2) perf stat --null --repeat 10 -- hackbench 32 thread 20000
== original, no patched
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.784911855 seconds time elapsed ( +- 0.14% )
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.868687608 seconds time elapsed ( +- 0.19% )
== patched with patch1~4
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.681273015 seconds time elapsed ( +- 0.21% )
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.681238459 seconds time elapsed ( +- 0.09% )
Xunlei Pang (4):
mm/slub: Introduce two counters for partial objects mm/slub: Get rid of count_partial()
percpu: Export per_cpu_sum()
mm/slub: Use percpu partial free counter
include/linux/percpu-defs.h | 10 ++++ kernel/locking/percpu-rwsem.c | 10 ---- mm/slab.h | 4 ++ mm/slub.c | 120 +++++++++++++++++++++++++++++------------- 4 files changed, 97 insertions(+), 47 deletions(-)
Signature · 1 line
1.8.3.1
PATCH v3 1/4 mm/slub: Introduce two counters for partial objects
The node list_lock in count_partial() spends long time iterating in case of large amount of partial page lists, which can cause thunder herd effect to the list_lock contention.
We have HSF RT(High-speed Service Framework Response-Time) monitors, the RT figures fluctuated randomly, then we deployed a tool detecting "irq off" and "preempt off" to dump the culprit's calltrace, capturing the list_lock cost nearly 100ms with irq off issued by "ss", this also caused network timeouts.
This patch introduces two counters to maintain the actual number of partial objects dynamically instead of iterating the partial page lists with list_lock held.
New counters of kmem_cache_node: partial_free_objs, partial_total_objs. The main operations are under list_lock in slow path, its performance impact should be minimal except the __slab_free() path which will be addressed later.
--- mm/slab.h | 4 ++++ mm/slub.c | 46 +++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 49 insertions(+), 1 deletion(-)
mm/slab.h +4 -0
@@ -547,6 +547,10 @@ struct kmem_cache_node {#ifdef CONFIG_SLUB unsigned long nr_partial;struct list_head partial; +#if defined(CONFIG_SLUB_DEBUG ) || defined(CONFIG_SYSFS ) +atomic_long_t partial_free_objs; +unsigned long partial_total_objs; +#endif #ifdef CONFIG_SLUB_DEBUG atomic_long_t nr_slabs;atomic_long_t total_objects;
mm/slub.c +45 -1
123 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v3 2/4 mm/slub: Get rid of count_partial()
In reply to Xunlei PangNow the partial counters are ready, let's use them directly and get rid of count_partial().
--- mm/slub.c | 54 ++++++++++++++++++++++-------------------------------- 1 file changed, 22 insertions(+), 32 deletions(-)
mm/slub.c +22 -32
@@ -2533,11 +2533,6 @@ static inline int node_match(struct page *page, int node) }#ifdef CONFIG_SLUB_DEBUG -static int count_free (struct page *page) -{ -return page->objects - page->inuse; -} -static inline unsigned long node_nr_objs (struct kmem_cache_node *n) {return atomic_long_read (&n->total_objects);
@@ -2545,19 +2540,26 @@ static inline unsigned long node_nr_objs(struct kmem_cache_node *n)#endif /* CONFIG_SLUB_DEBUG */ #if defined(CONFIG_SLUB_DEBUG ) || defined(CONFIG_SYSFS ) -static unsigned long count_partial (struct kmem_cache_node *n, -int (*get_count)(struct page *)) +enum partial_item {PARTIAL_FREE ,PARTIAL_INUSE ,PARTIAL_TOTAL }; + +static unsigned long partial_counter (struct kmem_cache_node *n, +enum partial_item item) { -unsigned long flags; -unsigned long x =0 ; -struct page *page; +unsigned long ret =0 ; -spin_lock_irqsave (&n->list_lock, flags); -list_for_each_entry (page, &n->partial, slab_list) - x +=get_count (page); -spin_unlock_irqrestore (&n->list_lock, flags); -return x; +if (item ==PARTIAL_FREE ) { + ret =atomic_long_read (&n->partial_free_objs); + }else if (item ==PARTIAL_TOTAL ) { + ret = n->partial_total_objs; + }else if (item ==PARTIAL_INUSE ) { + ret = n->partial_total_objs -atomic_long_read (&n->partial_free_objs); +if ((long )ret <0 ) + ret =0 ; + } + +return ret; } +#endif /* CONFIG_SLUB_DEBUG || CONFIG_SYSFS */ static noinline void
@@ -2587,7 +2589,7 @@ static unsigned long count_partial(struct kmem_cache_node *n,unsigned long nr_objs;unsigned long nr_free; - nr_free =count_partial (n, count_free); + nr_free =partial_counter (n,PARTIAL_FREE ); nr_slabs =node_nr_slabs (n); nr_objs =node_nr_objs (n);
@@ -4643,18 +4645,6 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,EXPORT_SYMBOL (__kmalloc_node_track_caller);#endif -#ifdef CONFIG_SYSFS -static int count_inuse (struct page *page) -{ -return page->inuse; -} - -static int count_total (struct page *page) -{ -return page->objects; -} -#endif -#ifdef CONFIG_SLUB_DEBUG static void validate_slab (struct kmem_cache *s,struct page *page) {
@@ -5091,7 +5081,7 @@ static ssize_t show_slab_objects(struct kmem_cache *s, x =atomic_long_read (&n->total_objects); else if (flags & SO_OBJECTS) x =atomic_long_read (&n->total_objects) - -count_partial (n, count_free); +partial_counter (n,PARTIAL_FREE ); else x =atomic_long_read (&n->nr_slabs); total += x;
@@ -5105,9 +5095,9 @@ static ssize_t show_slab_objects(struct kmem_cache *s,for_each_kmem_cache_node (s, node, n) {if (flags &SO_TOTAL ) - x =count_partial (n, count_total); + x =partial_counter (n,PARTIAL_TOTAL );else if (flags &SO_OBJECTS ) - x =count_partial (n, count_inuse); + x =partial_counter (n,PARTIAL_INUSE );else x = n->nr_partial; total += x;
@@ -5873,7 +5863,7 @@ void get_slabinfo(struct kmem_cache *s, struct slabinfo *sinfo)for_each_kmem_cache_node (s, node, n) { nr_slabs +=node_nr_slabs (n); nr_objs +=node_nr_objs (n); - nr_free +=count_partial (n, count_free); + nr_free +=partial_counter (n,PARTIAL_FREE ); } sinfo->active_objs = nr_objs - nr_free;
Signature · 1 line
1.8.3.1
PATCH v3 3/4 percpu: Export per_cpu_sum()
In reply to Xunlei Pangper_cpu_sum() is useful, and deserves to be exported.
--- include/linux/percpu-defs.h | 10 ++++++++++ kernel/locking/percpu-rwsem.c | 10 ---------- 2 files changed, 10 insertions(+), 10 deletions(-)
include/linux/percpu-defs.h +10 -0
@@ -220,6 +220,16 @@ (void )__vpp_verify; \ } while (0) +#define per_cpu_sum (var) \ +({ \ + typeof(var) __sum = 0; \ + int cpu; \ + compiletime_assert_atomic_type(__sum); \ + for_each_possible_cpu(cpu) \ + __sum += per_cpu(var, cpu); \ + __sum; \ +}) +#ifdef CONFIG_SMP /*
kernel/locking/percpu-rwsem.c +0 -10
@@ -178,16 +178,6 @@ bool __percpu_down_read(struct percpu_rw_semaphore *sem, bool try) }EXPORT_SYMBOL_GPL (__percpu_down_read); -#define per_cpu_sum (var) \ -({ \ - typeof(var) __sum = 0; \ - int cpu; \ - compiletime_assert_atomic_type(__sum); \ - for_each_possible_cpu(cpu) \ - __sum += per_cpu(var, cpu); \ - __sum; \ -}) -/* * Return true if the modular sum of the sem->read_count per-CPU variable is * zero. If this sum is zero, then it is stable due to the fact that if any
Signature · 1 line
1.8.3.1
PATCH v3 4/4 mm/slub: Use percpu partial free counter
In reply to Xunlei PangThe only concern of introducing partial counter is that, partial_free_objs may cause cache and atomic operation contention in case of same SLUB concurrent __slab_free().
This patch changes it to be a percpu counter, also replace the counter variables to avoid cacheline issues.
--- mm/slab.h | 6 ++++-- mm/slub.c | 30 +++++++++++++++++++++++------- 2 files changed, 27 insertions(+), 9 deletions(-)
mm/slab.h +4 -2
@@ -546,16 +546,18 @@ struct kmem_cache_node {#ifdef CONFIG_SLUB unsigned long nr_partial; -struct list_head partial;#if defined(CONFIG_SLUB_DEBUG ) || defined(CONFIG_SYSFS ) -atomic_long_t partial_free_objs;unsigned long partial_total_objs;#endif +struct list_head partial;#ifdef CONFIG_SLUB_DEBUG atomic_long_t nr_slabs;atomic_long_t total_objects;struct list_head full;#endif +#if defined(CONFIG_SLUB_DEBUG ) || defined(CONFIG_SYSFS ) +unsigned long __percpu *partial_free_objs; +#endif #endif };
mm/slub.c +23 -7
@@ -1894,7 +1894,7 @@ static void discard_slab(struct kmem_cache *s, struct page *page)static inline void __update_partial_free (struct kmem_cache_node *n,long delta) { -atomic_long_add (delta, &n->partial_free_objs); +this_cpu_add (*n->partial_free_objs, delta); } static inline void
@@ -2548,11 +2548,16 @@ static unsigned long partial_counter(struct kmem_cache_node *n,unsigned long ret =0 ;if (item ==PARTIAL_FREE ) { - ret =atomic_long_read (&n->partial_free_objs); + ret =per_cpu_sum (*n->partial_free_objs); +if ((long )ret <0 ) + ret =0 ; }else if (item ==PARTIAL_TOTAL ) { ret = n->partial_total_objs; }else if (item ==PARTIAL_INUSE ) { - ret = n->partial_total_objs -atomic_long_read (&n->partial_free_objs); + ret =per_cpu_sum (*n->partial_free_objs); +if ((long )ret <0 ) + ret =0 ; + ret = n->partial_total_objs - ret;if ((long )ret <0 ) ret =0 ; }
@@ -3552,14 +3557,16 @@ static inline int calculate_order(unsigned int size)return -ENOSYS ; } -static void +static int init_kmem_cache_node (struct kmem_cache_node *n) { n->nr_partial =0 ;spin_lock_init (&n->list_lock);INIT_LIST_HEAD (&n->partial);#if defined(CONFIG_SLUB_DEBUG ) || defined(CONFIG_SYSFS ) -atomic_long_set (&n->partial_free_objs,0 ); + n->partial_free_objs = alloc_percpu(unsigned long); +if (!n->partial_free_objs) +return -ENOMEM ; n->partial_total_objs =0 ;#endif #ifdef CONFIG_SLUB_DEBUG
@@ -3567,6 +3574,8 @@ static inline int calculate_order(unsigned int size)atomic_long_set (&n->total_objects,0 );INIT_LIST_HEAD (&n->full);#endif + +return 0 ; }static inline int alloc_kmem_cache_cpus (struct kmem_cache *s)
@@ -3626,7 +3635,7 @@ static void early_kmem_cache_node_alloc(int node) page->inuse =1 ; page->frozen =0 ; kmem_cache_node->node[node] = n; -init_kmem_cache_node (n); +BUG_ON (init_kmem_cache_node (n) <0 );inc_slabs_node (kmem_cache_node, node, page->objects);/*
@@ -3644,6 +3653,9 @@ static void free_kmem_cache_nodes(struct kmem_cache *s)for_each_kmem_cache_node (s, node, n) { s->node[node] =NULL ; +#if defined(CONFIG_SLUB_DEBUG ) || defined(CONFIG_SYSFS ) +free_percpu (n->partial_free_objs); +#endif kmem_cache_free (kmem_cache_node, n); } }
@@ -3674,7 +3686,11 @@ static int init_kmem_cache_nodes(struct kmem_cache *s)return 0 ; } -init_kmem_cache_node (n); +if (init_kmem_cache_node (n) <0 ) { +free_kmem_cache_nodes (s); +return 0 ; + } + s->node[node] = n; }return 1 ;
Signature · 1 line
1.8.3.1
Quoted 2 lines · Xunlei Pang
count_partial()can holdn->list_lockspinlock for quite long, which makes much trouble to the system. This series eliminate this problem.
Before I check the details, I have two high-level comments:
- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?
- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <active_objs> in /proc/slabinfo because there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of /proc/slabinfo won't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just report active_objs == num_objs and it won't actually break any ABI.
At the same time somebody might actually want accurate object statistics at the expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it to CONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just report active_objs == num_objs.
Vlastimil
Quoted 74 lines
v1->v2:
- Improved changelog and variable naming for PATCH 1~2.
- PATCH3 adds per-cpu counter to avoid performance regression in concurrent__slab_free().
v2->v3:
- Changed "page->inuse" to the safe "new.inuse", etc.
- UsedCONFIG_SLUB_DEBUGandCONFIG_SYSFScondition for new counters.
-atomic_long_t-> unsigned long[Testing]
There seems might be a little performance impact under extreme__slab_free()concurrent calls according to my tests.On my 32-cpu 2-socket physical machine:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz1) perf stat --null --repeat 10 -- hackbench 20 thread 20000
== original, no patched
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):24.536050899 seconds time elapsed ( +- 0.24% )
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.588049142 seconds time elapsed ( +- 0.35% )
== patched with patch1~4
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):24.670892273 seconds time elapsed ( +- 0.29% )
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.746755689 seconds time elapsed ( +- 0.21% )
2) perf stat --null --repeat 10 -- hackbench 32 thread 20000
== original, no patched
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):39.784911855 seconds time elapsed ( +- 0.14% )
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.868687608 seconds time elapsed ( +- 0.19% )
== patched with patch1~4
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):39.681273015 seconds time elapsed ( +- 0.21% )
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.681238459 seconds time elapsed ( +- 0.09% )
Xunlei Pang (4):
mm/slub: Introduce two counters for partial objects mm/slub: Get rid ofcount_partial()
percpu: Exportper_cpu_sum()
mm/slub: Use percpu partial free counterinclude/linux/percpu-defs.h | 10 ++++ kernel/locking/percpu-rwsem.c | 10 ---- mm/slab.h | 4 ++ mm/slub.c | 120 +++++++++++++++++++++++++++++------------- 4 files changed, 97 insertions(+), 47 deletions(-)
Quoted 18 lines · Vlastimil Babka
Quoted 2 lines · Xunlei Pang
count_partial()can holdn->list_lockspinlock for quite long, which makes much trouble to the system. This series eliminate this problem.Before I check the details, I have two high-level comments:
- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?
- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <
active_objs> in/proc/slabinfobecause there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of/proc/slabinfowon't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just reportactive_objs==num_objsand it won't actually break any ABI.
At the same time somebody might actually want accurate object statistics at the
Quoted 8 lines
expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with
CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it toCONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just reportactive_objs==num_objs.
It sounds really good to me! The only thing, I'd avoid introducing a new option and use CONFIG_SLUB_STATS instead.
It seems like CONFIG_SLUB_DEBUG is a more popular option than CONFIG_SLUB_STATS. CONFIG_SLUB_DEBUG is enabled on my Fedora workstation, CONFIG_SLUB_STATS is off. I doubt an average user needs this data, so I'd go with CONFIG_SLUB_STATS.
Thanks!
Quoted 78 lines
Vlastimil
Quoted 74 lines
v1->v2:
- Improved changelog and variable naming for PATCH 1~2.
- PATCH3 adds per-cpu counter to avoid performance regression in concurrent__slab_free().
v2->v3:
- Changed "page->inuse" to the safe "new.inuse", etc.
- UsedCONFIG_SLUB_DEBUGandCONFIG_SYSFScondition for new counters.
-atomic_long_t-> unsigned long[Testing]
There seems might be a little performance impact under extreme__slab_free()concurrent calls according to my tests.On my 32-cpu 2-socket physical machine:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz1) perf stat --null --repeat 10 -- hackbench 20 thread 20000
== original, no patched
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):24.536050899 seconds time elapsed ( +- 0.24% )
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.588049142 seconds time elapsed ( +- 0.35% )
== patched with patch1~4
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):24.670892273 seconds time elapsed ( +- 0.29% )
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.746755689 seconds time elapsed ( +- 0.21% )
2) perf stat --null --repeat 10 -- hackbench 32 thread 20000
== original, no patched
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):39.784911855 seconds time elapsed ( +- 0.14% )
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.868687608 seconds time elapsed ( +- 0.19% )
== patched with patch1~4
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):39.681273015 seconds time elapsed ( +- 0.21% )
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.681238459 seconds time elapsed ( +- 0.09% )
Xunlei Pang (4):
mm/slub: Introduce two counters for partial objects mm/slub: Get rid ofcount_partial()
percpu: Exportper_cpu_sum()
mm/slub: Use percpu partial free counterinclude/linux/percpu-defs.h | 10 ++++ kernel/locking/percpu-rwsem.c | 10 ---- mm/slab.h | 4 ++ mm/slub.c | 120 +++++++++++++++++++++++++++++------------- 4 files changed, 97 insertions(+), 47 deletions(-)
Quoted 30 lines · Roman Gushchin
Quoted 18 lines · Vlastimil Babka
Quoted 2 lines · Xunlei Pang
count_partial()can holdn->list_lockspinlock for quite long, which makes much trouble to the system. This series eliminate this problem.Before I check the details, I have two high-level comments:
- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?
- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <
active_objs> in/proc/slabinfobecause there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of/proc/slabinfowon't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just reportactive_objs==num_objsand it won't actually break any ABI.
At the same time somebody might actually want accurate object statistics at theQuoted 8 lines
expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with
CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it toCONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just reportactive_objs==num_objs.
Quoted 6 lines
It sounds really good to me! The only thing, I'd avoid introducing a new option and use
CONFIG_SLUB_STATSinstead.It seems like
CONFIG_SLUB_DEBUGis a more popular option thanCONFIG_SLUB_STATS.CONFIG_SLUB_DEBUGis enabled on my Fedora workstation,CONFIG_SLUB_STATSis off. I doubt an average user needs this data, so I'd go withCONFIG_SLUB_STATS.
I think CONFIG_SLUB_DEBUG is enabled by default on most distros since it is supposed not incur too much overhead unless specific debug (i.e. red_zone) is turned on on demand.
Quoted 82 lines
Thanks!
Quoted 78 lines
Vlastimil
Quoted 74 lines
v1->v2:
- Improved changelog and variable naming for PATCH 1~2.
- PATCH3 adds per-cpu counter to avoid performance regression in concurrent__slab_free().
v2->v3:
- Changed "page->inuse" to the safe "new.inuse", etc.
- UsedCONFIG_SLUB_DEBUGandCONFIG_SYSFScondition for new counters.
-atomic_long_t-> unsigned long[Testing]
There seems might be a little performance impact under extreme__slab_free()concurrent calls according to my tests.On my 32-cpu 2-socket physical machine:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz1) perf stat --null --repeat 10 -- hackbench 20 thread 20000
== original, no patched
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):24.536050899 seconds time elapsed ( +- 0.24% )
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.588049142 seconds time elapsed ( +- 0.35% )
== patched with patch1~4
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):24.670892273 seconds time elapsed ( +- 0.29% )
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.746755689 seconds time elapsed ( +- 0.21% )
2) perf stat --null --repeat 10 -- hackbench 32 thread 20000
== original, no patched
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):39.784911855 seconds time elapsed ( +- 0.14% )
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.868687608 seconds time elapsed ( +- 0.19% )
== patched with patch1~4
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):39.681273015 seconds time elapsed ( +- 0.21% )
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.681238459 seconds time elapsed ( +- 0.09% )
Xunlei Pang (4):
mm/slub: Introduce two counters for partial objects mm/slub: Get rid ofcount_partial()
percpu: Exportper_cpu_sum()
mm/slub: Use percpu partial free counterinclude/linux/percpu-defs.h | 10 ++++ kernel/locking/percpu-rwsem.c | 10 ---- mm/slab.h | 4 ++ mm/slub.c | 120 +++++++++++++++++++++++++++++------------- 4 files changed, 97 insertions(+), 47 deletions(-)
Quoted 7 lines · Yang Shi
Quoted 3 lines
It seems like
CONFIG_SLUB_DEBUGis a more popular option thanCONFIG_SLUB_STATS.CONFIG_SLUB_DEBUGis enabled on my Fedora workstation,CONFIG_SLUB_STATSis off. I doubt an average user needs this data, so I'd go withCONFIG_SLUB_STATS.I think
CONFIG_SLUB_DEBUGis enabled by default on most distros since it is supposed not incur too much overhead unless specific debug (i.e.red_zone) is turned on on demand.
Correct. CONFIG_SLUB_DEBUG includes the code so the debugging can be enabled on Distro kernels with a kernel command line option. So you dont have to recompile the kernel to find weird memory corruption issues from strange device drivers.
Somehow my email address dropped off this thread.
Quoted 5 lines · Christoph Lameter
Quoted 3 lines · Yang Shi
Quoted 3 lines
It seems like
CONFIG_SLUB_DEBUGis a more popular option thanCONFIG_SLUB_STATS.CONFIG_SLUB_DEBUGis enabled on my Fedora workstation,CONFIG_SLUB_STATSis off. I doubt an average user needs this data, so I'd go withCONFIG_SLUB_STATS.
Hm I can imagine that (after due performance testing) we would consider having accurate slabinfo in our distro kernel, just as we have CONFIG_SLUB_DEBUG but not the full stats.
Quoted 10 lines
Quoted 3 lines
I think
CONFIG_SLUB_DEBUGis enabled by default on most distros since it is supposed not incur too much overhead unless specific debug (i.e.red_zone) is turned on on demand.Correct.
CONFIG_SLUB_DEBUGincludes the code so the debugging can be enabled on Distro kernels with a kernel command line option. So you dont have to recompile the kernel to find weird memory corruption issues from strange device drivers.Somehow my email address dropped off this thread.
Hm I see [email protected] in all e-mails of the thread, but now you replaced it with [email protected] ?
Quoted 9 lines · Vlastimil Babka
Quoted 2 lines · Xunlei Pang
count_partial()can holdn->list_lockspinlock for quite long, which makes much trouble to the system. This series eliminate this problem.Before I check the details, I have two high-level comments:
- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?
Quoted 8 lines
- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <
active_objs> in/proc/slabinfobecause there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of/proc/slabinfowon't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just reportactive_objs==num_objsand it won't actually break any ABI.
If maintainers don't mind this inaccuracy which I also doubt its importance, then it becomes easy. For fear that some people who really cares, introducing an extra config(default-off) for it would be a good option.
Quoted 9 lines
At the same time somebody might actually want accurate object statistics at the expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with
CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it toCONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just reportactive_objs==num_objs.
For percpu slabs, the numbers can be retrieved from the existing slub_percpu_partial()->pobjects, looks no need extra work.
Quoted 77 lines
Vlastimil
Quoted 74 lines
v1->v2:
- Improved changelog and variable naming for PATCH 1~2.
- PATCH3 adds per-cpu counter to avoid performance regression in concurrent__slab_free().
v2->v3:
- Changed "page->inuse" to the safe "new.inuse", etc.
- UsedCONFIG_SLUB_DEBUGandCONFIG_SYSFScondition for new counters.
-atomic_long_t-> unsigned long[Testing]
There seems might be a little performance impact under extreme__slab_free()concurrent calls according to my tests.On my 32-cpu 2-socket physical machine:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz1) perf stat --null --repeat 10 -- hackbench 20 thread 20000
== original, no patched
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):24.536050899 seconds time elapsed ( +- 0.24% )
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.588049142 seconds time elapsed ( +- 0.35% )
== patched with patch1~4
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):24.670892273 seconds time elapsed ( +- 0.29% )
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):
24.746755689 seconds time elapsed ( +- 0.21% )
2) perf stat --null --repeat 10 -- hackbench 32 thread 20000
== original, no patched
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):39.784911855 seconds time elapsed ( +- 0.14% )
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.868687608 seconds time elapsed ( +- 0.19% )
== patched with patch1~4
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):39.681273015 seconds time elapsed ( +- 0.21% )
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):
39.681238459 seconds time elapsed ( +- 0.09% )
Xunlei Pang (4):
mm/slub: Introduce two counters for partial objects mm/slub: Get rid ofcount_partial()
percpu: Exportper_cpu_sum()
mm/slub: Use percpu partial free counterinclude/linux/percpu-defs.h | 10 ++++ kernel/locking/percpu-rwsem.c | 10 ---- mm/slab.h | 4 ++ mm/slub.c | 120 +++++++++++++++++++++++++++++------------- 4 files changed, 97 insertions(+), 47 deletions(-)
Quoted 19 lines · Xunlei Pang
Quoted 9 lines · Vlastimil Babka
Quoted 2 lines · Xunlei Pang
count_partial()can holdn->list_lockspinlock for quite long, which makes much trouble to the system. This series eliminate this problem.Before I check the details, I have two high-level comments:
- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?
Quoted 8 lines
- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <
active_objs> in/proc/slabinfobecause there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of/proc/slabinfowon't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just reportactive_objs==num_objsand it won't actually break any ABI.
Quoted 4 lines
If maintainers don't mind this inaccuracy which I also doubt its importance, then it becomes easy. For fear that some people who really cares, introducing an extra config(default-off) for it would be a good option.
Great.
Quoted 11 lines
Quoted 9 lines
At the same time somebody might actually want accurate object statistics at the expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with
CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it toCONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just reportactive_objs==num_objs.For percpu slabs, the numbers can be retrieved from the existing slub_percpu_partial()->pobjects, looks no need extra work.
Hm, unfortunately it's not that simple, the number there is a snapshot that can become wildly inacurate afterwards.
Quoted 36 lines · Vlastimil Babka
Quoted 19 lines · Xunlei Pang
Quoted 9 lines · Vlastimil Babka
Quoted 2 lines · Xunlei Pang
count_partial()can holdn->list_lockspinlock for quite long, which makes much trouble to the system. This series eliminate this problem.Before I check the details, I have two high-level comments:
- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?
Quoted 8 lines
- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <
active_objs> in/proc/slabinfobecause there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of/proc/slabinfowon't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just reportactive_objs==num_objsand it won't actually break any ABI.Quoted 4 lines
If maintainers don't mind this inaccuracy which I also doubt its importance, then it becomes easy. For fear that some people who really cares, introducing an extra config(default-off) for it would be a good option.
Great.
Quoted 9 lines
Quoted 9 lines
At the same time somebody might actually want accurate object statistics at the expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with
CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it toCONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just reportactive_objs==num_objs.
Quoted 6 lines
Quoted 2 lines
For percpu slabs, the numbers can be retrieved from the existing slub_percpu_partial()->pobjects, looks no need extra work.
Hm, unfortunately it's not that simple, the number there is a snapshot that can become wildly inacurate afterwards.
It's hard to make it absoultely accurate using percpu, the data can change during you iterating all the cpus and total_objects, I can't imagine its real-world usage, not to mention the percpu freelist cache. I think sysfs slabs_cpu_partial should work enough for common debug purpose.