Skip to contentSkip to conversation

mm/slub: Fix count_partial() problem

PATCH Superseded Xunlei Pang 5 participants 13 messages Mar 9, 2021 – Mar 16, 2021

Subsystems (4)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover mm/slub: Fix count_partial() problem

count_partial() can hold n->list_lock spinlock for quite long, which makes much trouble to the system. This series eliminate this problem.

v1->v2:
- Improved changelog and variable naming for PATCH 1~2.
- PATCH3 adds per-cpu counter to avoid performance regression in concurrent __slab_free().

v2->v3:
- Changed "page->inuse" to the safe "new.inuse", etc.
- Used CONFIG_SLUB_DEBUG and CONFIG_SYSFS condition for new counters.
- atomic_long_t -> unsigned long

[Testing]
There seems might be a little performance impact under extreme __slab_free() concurrent calls according to my tests.

On my 32-cpu 2-socket physical machine:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz

1) perf stat --null --repeat 10 -- hackbench 20 thread 20000

== original, no patched
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.536050899 seconds time elapsed ( +- 0.24% )

Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.588049142 seconds time elapsed ( +- 0.35% )

== patched with patch1~4
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.670892273 seconds time elapsed ( +- 0.29% )

Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.746755689 seconds time elapsed ( +- 0.21% )

2) perf stat --null --repeat 10 -- hackbench 32 thread 20000

== original, no patched
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.784911855 seconds time elapsed ( +- 0.14% )

Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.868687608 seconds time elapsed ( +- 0.19% )

== patched with patch1~4
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.681273015 seconds time elapsed ( +- 0.21% )

Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.681238459 seconds time elapsed ( +- 0.09% )

Xunlei Pang (4):
mm/slub: Introduce two counters for partial objects mm/slub: Get rid of count_partial()
percpu: Export per_cpu_sum()
mm/slub: Use percpu partial free counter

 include/linux/percpu-defs.h   |  10 ++++
 kernel/locking/percpu-rwsem.c |  10 ----
 mm/slab.h                     |   4 ++
 mm/slub.c                     | 120 +++++++++++++++++++++++++++++-------------
 4 files changed, 97 insertions(+), 47 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v3 1/4 mm/slub: Introduce two counters for partial objects

The node list_lock in count_partial() spends long time iterating in case of large amount of partial page lists, which can cause thunder herd effect to the list_lock contention.

We have HSF RT(High-speed Service Framework Response-Time) monitors, the RT figures fluctuated randomly, then we deployed a tool detecting "irq off" and "preempt off" to dump the culprit's calltrace, capturing the list_lock cost nearly 100ms with irq off issued by "ss", this also caused network timeouts.

This patch introduces two counters to maintain the actual number of partial objects dynamically instead of iterating the partial page lists with list_lock held.

New counters of kmem_cache_node: partial_free_objs, partial_total_objs. The main operations are under list_lock in slow path, its performance impact should be minimal except the __slab_free() path which will be addressed later.

---
 mm/slab.h |  4 ++++
 mm/slub.c | 46 +++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 49 insertions(+), 1 deletion(-)
mm/slab.h +4 -0
index 076582f..817bfa0 100644
@@ -547,6 +547,10 @@ struct kmem_cache_node {
 #ifdef CONFIG_SLUB
 	unsigned long nr_partial;
 	struct list_head partial;
+#if defined(CONFIG_SLUB_DEBUG) || defined(CONFIG_SYSFS)
+	atomic_long_t partial_free_objs;
+	unsigned long partial_total_objs;
+#endif
 #ifdef CONFIG_SLUB_DEBUG
 	atomic_long_t nr_slabs;
 	atomic_long_t total_objects;
mm/slub.c +45 -1

123 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#2 lore raw

PATCH v3 2/4 mm/slub: Get rid of count_partial()

In reply to Xunlei Pang

Now the partial counters are ready, let's use them directly and get rid of count_partial().

---
 mm/slub.c | 54 ++++++++++++++++++++++--------------------------------
 1 file changed, 22 insertions(+), 32 deletions(-)
mm/slub.c +22 -32
index 4d02831..3f76b57 100644
@@ -2533,11 +2533,6 @@ static inline int node_match(struct page *page, int node)
 }
 
 #ifdef CONFIG_SLUB_DEBUG
-static int count_free(struct page *page)
-{
-	return page->objects - page->inuse;
-}
-
 static inline unsigned long node_nr_objs(struct kmem_cache_node *n)
 {
 	return atomic_long_read(&n->total_objects);
@@ -2545,19 +2540,26 @@ static inline unsigned long node_nr_objs(struct kmem_cache_node *n)
 #endif /* CONFIG_SLUB_DEBUG */
 
 #if defined(CONFIG_SLUB_DEBUG) || defined(CONFIG_SYSFS)
-static unsigned long count_partial(struct kmem_cache_node *n,
-					int (*get_count)(struct page *))
+enum partial_item { PARTIAL_FREE, PARTIAL_INUSE, PARTIAL_TOTAL };
+
+static unsigned long partial_counter(struct kmem_cache_node *n,
+		enum partial_item item)
 {
-	unsigned long flags;
-	unsigned long x = 0;
-	struct page *page;
+	unsigned long ret = 0;
 
-	spin_lock_irqsave(&n->list_lock, flags);
-	list_for_each_entry(page, &n->partial, slab_list)
-		x += get_count(page);
-	spin_unlock_irqrestore(&n->list_lock, flags);
-	return x;
+	if (item == PARTIAL_FREE) {
+		ret = atomic_long_read(&n->partial_free_objs);
+	} else if (item == PARTIAL_TOTAL) {
+		ret = n->partial_total_objs;
+	} else if (item == PARTIAL_INUSE) {
+		ret = n->partial_total_objs - atomic_long_read(&n->partial_free_objs);
+		if ((long)ret < 0)
+			ret = 0;
+	}
+
+	return ret;
 }
+
 #endif /* CONFIG_SLUB_DEBUG || CONFIG_SYSFS */
 
 static noinline void
@@ -2587,7 +2589,7 @@ static unsigned long count_partial(struct kmem_cache_node *n,
 		unsigned long nr_objs;
 		unsigned long nr_free;
 
-		nr_free  = count_partial(n, count_free);
+		nr_free  = partial_counter(n, PARTIAL_FREE);
 		nr_slabs = node_nr_slabs(n);
 		nr_objs  = node_nr_objs(n);
 
@@ -4643,18 +4645,6 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 EXPORT_SYMBOL(__kmalloc_node_track_caller);
 #endif
 
-#ifdef CONFIG_SYSFS
-static int count_inuse(struct page *page)
-{
-	return page->inuse;
-}
-
-static int count_total(struct page *page)
-{
-	return page->objects;
-}
-#endif
-
 #ifdef CONFIG_SLUB_DEBUG
 static void validate_slab(struct kmem_cache *s, struct page *page)
 {
@@ -5091,7 +5081,7 @@ static ssize_t show_slab_objects(struct kmem_cache *s,
 				x = atomic_long_read(&n->total_objects);
 			else if (flags & SO_OBJECTS)
 				x = atomic_long_read(&n->total_objects) -
-					count_partial(n, count_free);
+					partial_counter(n, PARTIAL_FREE);
 			else
 				x = atomic_long_read(&n->nr_slabs);
 			total += x;
@@ -5105,9 +5095,9 @@ static ssize_t show_slab_objects(struct kmem_cache *s,
 
 		for_each_kmem_cache_node(s, node, n) {
 			if (flags & SO_TOTAL)
-				x = count_partial(n, count_total);
+				x = partial_counter(n, PARTIAL_TOTAL);
 			else if (flags & SO_OBJECTS)
-				x = count_partial(n, count_inuse);
+				x = partial_counter(n, PARTIAL_INUSE);
 			else
 				x = n->nr_partial;
 			total += x;
@@ -5873,7 +5863,7 @@ void get_slabinfo(struct kmem_cache *s, struct slabinfo *sinfo)
 	for_each_kmem_cache_node(s, node, n) {
 		nr_slabs += node_nr_slabs(n);
 		nr_objs += node_nr_objs(n);
-		nr_free += count_partial(n, count_free);
+		nr_free += partial_counter(n, PARTIAL_FREE);
 	}
 
 	sinfo->active_objs = nr_objs - nr_free;
Signature · 1 line
1.8.3.1
#3 lore raw

PATCH v3 3/4 percpu: Export per_cpu_sum()

In reply to Xunlei Pang

per_cpu_sum() is useful, and deserves to be exported.

---
 include/linux/percpu-defs.h   | 10 ++++++++++
 kernel/locking/percpu-rwsem.c | 10 ----------
 2 files changed, 10 insertions(+), 10 deletions(-)
include/linux/percpu-defs.h +10 -0
index dff7040..0e71b68 100644
@@ -220,6 +220,16 @@
 	(void)__vpp_verify;						\
 } while (0)
 
+#define per_cpu_sum(var)						\
+({									\
+	typeof(var) __sum = 0;						\
+	int cpu;							\
+	compiletime_assert_atomic_type(__sum);				\
+	for_each_possible_cpu(cpu)					\
+		__sum += per_cpu(var, cpu);				\
+	__sum;								\
+})
+
 #ifdef CONFIG_SMP
 
 /*
kernel/locking/percpu-rwsem.c +0 -10
index 70a32a5..0980e51 100644
@@ -178,16 +178,6 @@ bool __percpu_down_read(struct percpu_rw_semaphore *sem, bool try)
 }
 EXPORT_SYMBOL_GPL(__percpu_down_read);
 
-#define per_cpu_sum(var)						\
-({									\
-	typeof(var) __sum = 0;						\
-	int cpu;							\
-	compiletime_assert_atomic_type(__sum);				\
-	for_each_possible_cpu(cpu)					\
-		__sum += per_cpu(var, cpu);				\
-	__sum;								\
-})
-
 /*
  * Return true if the modular sum of the sem->read_count per-CPU variable is
  * zero.  If this sum is zero, then it is stable due to the fact that if any
Signature · 1 line
1.8.3.1
#4 lore raw

PATCH v3 4/4 mm/slub: Use percpu partial free counter

In reply to Xunlei Pang

The only concern of introducing partial counter is that, partial_free_objs may cause cache and atomic operation contention in case of same SLUB concurrent __slab_free().

This patch changes it to be a percpu counter, also replace the counter variables to avoid cacheline issues.

---
 mm/slab.h |  6 ++++--
 mm/slub.c | 30 +++++++++++++++++++++++-------
 2 files changed, 27 insertions(+), 9 deletions(-)
mm/slab.h +4 -2
index 817bfa0..c819597 100644
@@ -546,16 +546,18 @@ struct kmem_cache_node {
 
 #ifdef CONFIG_SLUB
 	unsigned long nr_partial;
-	struct list_head partial;
 #if defined(CONFIG_SLUB_DEBUG) || defined(CONFIG_SYSFS)
-	atomic_long_t partial_free_objs;
 	unsigned long partial_total_objs;
 #endif
+	struct list_head partial;
 #ifdef CONFIG_SLUB_DEBUG
 	atomic_long_t nr_slabs;
 	atomic_long_t total_objects;
 	struct list_head full;
 #endif
+#if defined(CONFIG_SLUB_DEBUG) || defined(CONFIG_SYSFS)
+	unsigned long __percpu *partial_free_objs;
+#endif
 #endif
 
 };
mm/slub.c +23 -7
index 3f76b57..b6ec065 100644
@@ -1894,7 +1894,7 @@ static void discard_slab(struct kmem_cache *s, struct page *page)
 static inline void
 __update_partial_free(struct kmem_cache_node *n, long delta)
 {
-	atomic_long_add(delta, &n->partial_free_objs);
+	this_cpu_add(*n->partial_free_objs, delta);
 }
 
 static inline void
@@ -2548,11 +2548,16 @@ static unsigned long partial_counter(struct kmem_cache_node *n,
 	unsigned long ret = 0;
 
 	if (item == PARTIAL_FREE) {
-		ret = atomic_long_read(&n->partial_free_objs);
+		ret = per_cpu_sum(*n->partial_free_objs);
+		if ((long)ret < 0)
+			ret = 0;
 	} else if (item == PARTIAL_TOTAL) {
 		ret = n->partial_total_objs;
 	} else if (item == PARTIAL_INUSE) {
-		ret = n->partial_total_objs - atomic_long_read(&n->partial_free_objs);
+		ret = per_cpu_sum(*n->partial_free_objs);
+		if ((long)ret < 0)
+			ret = 0;
+		ret = n->partial_total_objs - ret;
 		if ((long)ret < 0)
 			ret = 0;
 	}
@@ -3552,14 +3557,16 @@ static inline int calculate_order(unsigned int size)
 	return -ENOSYS;
 }
 
-static void
+static int
 init_kmem_cache_node(struct kmem_cache_node *n)
 {
 	n->nr_partial = 0;
 	spin_lock_init(&n->list_lock);
 	INIT_LIST_HEAD(&n->partial);
 #if defined(CONFIG_SLUB_DEBUG) || defined(CONFIG_SYSFS)
-	atomic_long_set(&n->partial_free_objs, 0);
+	n->partial_free_objs = alloc_percpu(unsigned long);
+	if (!n->partial_free_objs)
+		return -ENOMEM;
 	n->partial_total_objs = 0;
 #endif
 #ifdef CONFIG_SLUB_DEBUG
@@ -3567,6 +3574,8 @@ static inline int calculate_order(unsigned int size)
 	atomic_long_set(&n->total_objects, 0);
 	INIT_LIST_HEAD(&n->full);
 #endif
+
+	return 0;
 }
 
 static inline int alloc_kmem_cache_cpus(struct kmem_cache *s)
@@ -3626,7 +3635,7 @@ static void early_kmem_cache_node_alloc(int node)
 	page->inuse = 1;
 	page->frozen = 0;
 	kmem_cache_node->node[node] = n;
-	init_kmem_cache_node(n);
+	BUG_ON(init_kmem_cache_node(n) < 0);
 	inc_slabs_node(kmem_cache_node, node, page->objects);
 
 	/*
@@ -3644,6 +3653,9 @@ static void free_kmem_cache_nodes(struct kmem_cache *s)
 
 	for_each_kmem_cache_node(s, node, n) {
 		s->node[node] = NULL;
+#if defined(CONFIG_SLUB_DEBUG) || defined(CONFIG_SYSFS)
+		free_percpu(n->partial_free_objs);
+#endif
 		kmem_cache_free(kmem_cache_node, n);
 	}
 }
@@ -3674,7 +3686,11 @@ static int init_kmem_cache_nodes(struct kmem_cache *s)
 			return 0;
 		}
 
-		init_kmem_cache_node(n);
+		if (init_kmem_cache_node(n) < 0) {
+			free_kmem_cache_nodes(s);
+			return 0;
+		}
+
 		s->node[node] = n;
 	}
 	return 1;
Signature · 1 line
1.8.3.1
#5 lore raw
In reply to Xunlei Pang
Quoted 2 lines · Xunlei Pang

count_partial() can hold n->list_lock spinlock for quite long, which makes much trouble to the system. This series eliminate this problem.

Before I check the details, I have two high-level comments:

- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?

- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <active_objs> in /proc/slabinfo because there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of /proc/slabinfo won't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just report active_objs == num_objs and it won't actually break any ABI.
At the same time somebody might actually want accurate object statistics at the expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it to CONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just report active_objs == num_objs.

Vlastimil

Quoted 74 lines

v1->v2:
- Improved changelog and variable naming for PATCH 1~2.
- PATCH3 adds per-cpu counter to avoid performance regression in concurrent __slab_free().

v2->v3:
- Changed "page->inuse" to the safe "new.inuse", etc.
- Used CONFIG_SLUB_DEBUG and CONFIG_SYSFS condition for new counters.
- atomic_long_t -> unsigned long

[Testing]
There seems might be a little performance impact under extreme __slab_free() concurrent calls according to my tests.

On my 32-cpu 2-socket physical machine:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz

1) perf stat --null --repeat 10 -- hackbench 20 thread 20000

== original, no patched
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.536050899 seconds time elapsed ( +- 0.24% )

Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.588049142 seconds time elapsed ( +- 0.35% )

== patched with patch1~4
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.670892273 seconds time elapsed ( +- 0.29% )

Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.746755689 seconds time elapsed ( +- 0.21% )

2) perf stat --null --repeat 10 -- hackbench 32 thread 20000

== original, no patched
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.784911855 seconds time elapsed ( +- 0.14% )

Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.868687608 seconds time elapsed ( +- 0.19% )

== patched with patch1~4
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.681273015 seconds time elapsed ( +- 0.21% )

Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.681238459 seconds time elapsed ( +- 0.09% )

Xunlei Pang (4):
mm/slub: Introduce two counters for partial objects mm/slub: Get rid of count_partial()
percpu: Export per_cpu_sum()
mm/slub: Use percpu partial free counter

 include/linux/percpu-defs.h   |  10 ++++
 kernel/locking/percpu-rwsem.c |  10 ----
 mm/slab.h                     |   4 ++
 mm/slub.c                     | 120 +++++++++++++++++++++++++++++-------------
 4 files changed, 97 insertions(+), 47 deletions(-)
#6 lore raw
Quoted 18 lines · Vlastimil Babka
Quoted 2 lines · Xunlei Pang

count_partial() can hold n->list_lock spinlock for quite long, which makes much trouble to the system. This series eliminate this problem.

Before I check the details, I have two high-level comments:

- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?

- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <active_objs> in /proc/slabinfo because there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of /proc/slabinfo won't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just report active_objs == num_objs and it won't actually break any ABI.
At the same time somebody might actually want accurate object statistics at the

Quoted 8 lines

expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it to CONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just report active_objs == num_objs.

It sounds really good to me! The only thing, I'd avoid introducing a new option and use CONFIG_SLUB_STATS instead.

It seems like CONFIG_SLUB_DEBUG is a more popular option than CONFIG_SLUB_STATS. CONFIG_SLUB_DEBUG is enabled on my Fedora workstation, CONFIG_SLUB_STATS is off. I doubt an average user needs this data, so I'd go with CONFIG_SLUB_STATS.

Thanks!

Quoted 78 lines

Vlastimil

Quoted 74 lines

v1->v2:
- Improved changelog and variable naming for PATCH 1~2.
- PATCH3 adds per-cpu counter to avoid performance regression in concurrent __slab_free().

v2->v3:
- Changed "page->inuse" to the safe "new.inuse", etc.
- Used CONFIG_SLUB_DEBUG and CONFIG_SYSFS condition for new counters.
- atomic_long_t -> unsigned long

[Testing]
There seems might be a little performance impact under extreme __slab_free() concurrent calls according to my tests.

On my 32-cpu 2-socket physical machine:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz

1) perf stat --null --repeat 10 -- hackbench 20 thread 20000

== original, no patched
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.536050899 seconds time elapsed ( +- 0.24% )

Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.588049142 seconds time elapsed ( +- 0.35% )

== patched with patch1~4
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.670892273 seconds time elapsed ( +- 0.29% )

Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.746755689 seconds time elapsed ( +- 0.21% )

2) perf stat --null --repeat 10 -- hackbench 32 thread 20000

== original, no patched
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.784911855 seconds time elapsed ( +- 0.14% )

Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.868687608 seconds time elapsed ( +- 0.19% )

== patched with patch1~4
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.681273015 seconds time elapsed ( +- 0.21% )

Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.681238459 seconds time elapsed ( +- 0.09% )

Xunlei Pang (4):
mm/slub: Introduce two counters for partial objects mm/slub: Get rid of count_partial()
percpu: Export per_cpu_sum()
mm/slub: Use percpu partial free counter

 include/linux/percpu-defs.h   |  10 ++++
 kernel/locking/percpu-rwsem.c |  10 ----
 mm/slab.h                     |   4 ++
 mm/slub.c                     | 120 +++++++++++++++++++++++++++++-------------
 4 files changed, 97 insertions(+), 47 deletions(-)
#7 lore raw
Quoted 30 lines · Roman Gushchin
Quoted 18 lines · Vlastimil Babka
Quoted 2 lines · Xunlei Pang

count_partial() can hold n->list_lock spinlock for quite long, which makes much trouble to the system. This series eliminate this problem.

Before I check the details, I have two high-level comments:

- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?

- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <active_objs> in /proc/slabinfo because there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of /proc/slabinfo won't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just report active_objs == num_objs and it won't actually break any ABI.
At the same time somebody might actually want accurate object statistics at the

Quoted 8 lines

expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it to CONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just report active_objs == num_objs.

Quoted 6 lines

It sounds really good to me! The only thing, I'd avoid introducing a new option and use CONFIG_SLUB_STATS instead.

It seems like CONFIG_SLUB_DEBUG is a more popular option than CONFIG_SLUB_STATS. CONFIG_SLUB_DEBUG is enabled on my Fedora workstation, CONFIG_SLUB_STATS is off. I doubt an average user needs this data, so I'd go with CONFIG_SLUB_STATS.

I think CONFIG_SLUB_DEBUG is enabled by default on most distros since it is supposed not incur too much overhead unless specific debug (i.e. red_zone) is turned on on demand.

Quoted 82 lines

Thanks!

Quoted 78 lines

Vlastimil

Quoted 74 lines

v1->v2:
- Improved changelog and variable naming for PATCH 1~2.
- PATCH3 adds per-cpu counter to avoid performance regression in concurrent __slab_free().

v2->v3:
- Changed "page->inuse" to the safe "new.inuse", etc.
- Used CONFIG_SLUB_DEBUG and CONFIG_SYSFS condition for new counters.
- atomic_long_t -> unsigned long

[Testing]
There seems might be a little performance impact under extreme __slab_free() concurrent calls according to my tests.

On my 32-cpu 2-socket physical machine:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz

1) perf stat --null --repeat 10 -- hackbench 20 thread 20000

== original, no patched
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.536050899 seconds time elapsed ( +- 0.24% )

Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.588049142 seconds time elapsed ( +- 0.35% )

== patched with patch1~4
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.670892273 seconds time elapsed ( +- 0.29% )

Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.746755689 seconds time elapsed ( +- 0.21% )

2) perf stat --null --repeat 10 -- hackbench 32 thread 20000

== original, no patched
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.784911855 seconds time elapsed ( +- 0.14% )

Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.868687608 seconds time elapsed ( +- 0.19% )

== patched with patch1~4
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.681273015 seconds time elapsed ( +- 0.21% )

Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.681238459 seconds time elapsed ( +- 0.09% )

Xunlei Pang (4):
mm/slub: Introduce two counters for partial objects mm/slub: Get rid of count_partial()
percpu: Export per_cpu_sum()
mm/slub: Use percpu partial free counter

 include/linux/percpu-defs.h   |  10 ++++
 kernel/locking/percpu-rwsem.c |  10 ----
 mm/slab.h                     |   4 ++
 mm/slub.c                     | 120 +++++++++++++++++++++++++++++-------------
 4 files changed, 97 insertions(+), 47 deletions(-)
#8 lore raw
Quoted 7 lines · Yang Shi
Quoted 3 lines

It seems like CONFIG_SLUB_DEBUG is a more popular option than CONFIG_SLUB_STATS. CONFIG_SLUB_DEBUG is enabled on my Fedora workstation, CONFIG_SLUB_STATS is off. I doubt an average user needs this data, so I'd go with CONFIG_SLUB_STATS.

I think CONFIG_SLUB_DEBUG is enabled by default on most distros since it is supposed not incur too much overhead unless specific debug (i.e. red_zone) is turned on on demand.

Correct. CONFIG_SLUB_DEBUG includes the code so the debugging can be enabled on Distro kernels with a kernel command line option. So you dont have to recompile the kernel to find weird memory corruption issues from strange device drivers.

Somehow my email address dropped off this thread.

#9 lore raw
Quoted 5 lines · Christoph Lameter
Quoted 3 lines · Yang Shi
Quoted 3 lines

It seems like CONFIG_SLUB_DEBUG is a more popular option than CONFIG_SLUB_STATS. CONFIG_SLUB_DEBUG is enabled on my Fedora workstation, CONFIG_SLUB_STATS is off. I doubt an average user needs this data, so I'd go with CONFIG_SLUB_STATS.

Hm I can imagine that (after due performance testing) we would consider having accurate slabinfo in our distro kernel, just as we have CONFIG_SLUB_DEBUG but not the full stats.

Quoted 10 lines
Quoted 3 lines

I think CONFIG_SLUB_DEBUG is enabled by default on most distros since it is supposed not incur too much overhead unless specific debug (i.e. red_zone) is turned on on demand.

Correct. CONFIG_SLUB_DEBUG includes the code so the debugging can be enabled on Distro kernels with a kernel command line option. So you dont have to recompile the kernel to find weird memory corruption issues from strange device drivers.

Somehow my email address dropped off this thread.

Hm I see [email protected] in all e-mails of the thread, but now you replaced it with [email protected] ?

#10 lore raw
In reply to Vlastimil Babka
Quoted 9 lines · Vlastimil Babka
Quoted 2 lines · Xunlei Pang

count_partial() can hold n->list_lock spinlock for quite long, which makes much trouble to the system. This series eliminate this problem.

Before I check the details, I have two high-level comments:

- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?

Quoted 8 lines

- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <active_objs> in /proc/slabinfo because there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of /proc/slabinfo won't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just report active_objs == num_objs and it won't actually break any ABI.

If maintainers don't mind this inaccuracy which I also doubt its importance, then it becomes easy. For fear that some people who really cares, introducing an extra config(default-off) for it would be a good option.

Quoted 9 lines

At the same time somebody might actually want accurate object statistics at the expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it to CONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just report active_objs == num_objs.

For percpu slabs, the numbers can be retrieved from the existing slub_percpu_partial()->pobjects, looks no need extra work.

Quoted 77 lines

Vlastimil

Quoted 74 lines

v1->v2:
- Improved changelog and variable naming for PATCH 1~2.
- PATCH3 adds per-cpu counter to avoid performance regression in concurrent __slab_free().

v2->v3:
- Changed "page->inuse" to the safe "new.inuse", etc.
- Used CONFIG_SLUB_DEBUG and CONFIG_SYSFS condition for new counters.
- atomic_long_t -> unsigned long

[Testing]
There seems might be a little performance impact under extreme __slab_free() concurrent calls according to my tests.

On my 32-cpu 2-socket physical machine:
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz

1) perf stat --null --repeat 10 -- hackbench 20 thread 20000

== original, no patched
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.536050899 seconds time elapsed ( +- 0.24% )

Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.588049142 seconds time elapsed ( +- 0.35% )

== patched with patch1~4
Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.670892273 seconds time elapsed ( +- 0.29% )

Performance counter stats for 'hackbench 20 thread 20000' (10 runs):

24.746755689 seconds time elapsed ( +- 0.21% )

2) perf stat --null --repeat 10 -- hackbench 32 thread 20000

== original, no patched
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.784911855 seconds time elapsed ( +- 0.14% )

Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.868687608 seconds time elapsed ( +- 0.19% )

== patched with patch1~4
Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.681273015 seconds time elapsed ( +- 0.21% )

Performance counter stats for 'hackbench 32 thread 20000' (10 runs):

39.681238459 seconds time elapsed ( +- 0.09% )

Xunlei Pang (4):
mm/slub: Introduce two counters for partial objects mm/slub: Get rid of count_partial()
percpu: Export per_cpu_sum()
mm/slub: Use percpu partial free counter

 include/linux/percpu-defs.h   |  10 ++++
 kernel/locking/percpu-rwsem.c |  10 ----
 mm/slab.h                     |   4 ++
 mm/slub.c                     | 120 +++++++++++++++++++++++++++++-------------
 4 files changed, 97 insertions(+), 47 deletions(-)
#11 lore raw
Quoted 19 lines · Xunlei Pang
Quoted 9 lines · Vlastimil Babka
Quoted 2 lines · Xunlei Pang

count_partial() can hold n->list_lock spinlock for quite long, which makes much trouble to the system. This series eliminate this problem.

Before I check the details, I have two high-level comments:

- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?

Quoted 8 lines

- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <active_objs> in /proc/slabinfo because there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of /proc/slabinfo won't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just report active_objs == num_objs and it won't actually break any ABI.

Quoted 4 lines

If maintainers don't mind this inaccuracy which I also doubt its importance, then it becomes easy. For fear that some people who really cares, introducing an extra config(default-off) for it would be a good option.

Great.

Quoted 11 lines
Quoted 9 lines

At the same time somebody might actually want accurate object statistics at the expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it to CONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just report active_objs == num_objs.

For percpu slabs, the numbers can be retrieved from the existing slub_percpu_partial()->pobjects, looks no need extra work.

Hm, unfortunately it's not that simple, the number there is a snapshot that can become wildly inacurate afterwards.

#12 lore raw
Quoted 36 lines · Vlastimil Babka
Quoted 19 lines · Xunlei Pang
Quoted 9 lines · Vlastimil Babka
Quoted 2 lines · Xunlei Pang

count_partial() can hold n->list_lock spinlock for quite long, which makes much trouble to the system. This series eliminate this problem.

Before I check the details, I have two high-level comments:

- patch 1 introduces some counting scheme that patch 4 then changes, could we do this in one step to avoid the churn?

Quoted 8 lines

- the series addresses the concern that spinlock is being held, but doesn't address the fact that counting partial per-node slabs is not nearly enough if we want accurate <active_objs> in /proc/slabinfo because there are also percpu slabs and per-cpu partial slabs, where we don't track the free objects at all. So after this series while the readers of /proc/slabinfo won't block the spinlock, they will get the same garbage data as before. So Christoph is not wrong to say that we can just report active_objs == num_objs and it won't actually break any ABI.

Quoted 4 lines

If maintainers don't mind this inaccuracy which I also doubt its importance, then it becomes easy. For fear that some people who really cares, introducing an extra config(default-off) for it would be a good option.

Great.

Quoted 9 lines
Quoted 9 lines

At the same time somebody might actually want accurate object statistics at the expense of peak performance, and it would be nice to give them such option in SLUB. Right now we don't provide this accuracy even with CONFIG_SLUB_STATS, although that option provides many additional tuning stats, with additional overhead.
So my proposal would be a new config for "accurate active objects" (or just tie it to CONFIG_SLUB_DEBUG?) that would extend the approach of percpu counters in patch 4 to all alloc/free, so that it includes percpu slabs. Without this config enabled, let's just report active_objs == num_objs.

Quoted 6 lines
Quoted 2 lines

For percpu slabs, the numbers can be retrieved from the existing slub_percpu_partial()->pobjects, looks no need extra work.

Hm, unfortunately it's not that simple, the number there is a snapshot that can become wildly inacurate afterwards.

It's hard to make it absoultely accurate using percpu, the data can change during you iterating all the cpus and total_objects, I can't imagine its real-world usage, not to mention the percpu freelist cache. I think sysfs slabs_cpu_partial should work enough for common debug purpose.