Skip to contentSkip to conversation

mm/slub: batch partial slab list operations

PATCH Applied Hao Li 3 participants 8 messages May 29 – Jun 1

Subsystems (2)
lore
Share & tools
lore

Hao Li

author reviewer
#0 lore raw

v3 cover mm/slub: batch partial slab list operations

The loops in get_partial_node_bulk() and __refill_objects_node() perform individual list operations for each slab. This patchset optimize this.

Patch 1 is a cleanup patch that wraps duplicated code into helper functions.

Patch 2 reduces the number of list operations by processing slabs in batches, thereby minimizing the time spent inside the critical section.

Changes in v3:
- Reorder the two patches for better logical flow. (Thanks Vlastimil)
- Introduce set_node_partial_state for symmetry. (Thanks Vlastimil)

Changes in v2:
- Patch 1 applies the same optimization to __refill_objects_node. (Thanks Harry and Vlastimil)
- Patch 2 introduces a helper to wraps duplicated code. (Thanks Harry)

Hao Li (2):
mm/slub: introduce helpers for node partial slab state mm/slub: detach and reattach partial slabs in batch

 mm/slub.c | 51 +++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 37 insertions(+), 14 deletions(-)

base-commit: 6b41dc6749232c14f3e069f8115084a7bbeee402

Signature · 1 line
2.54.0
author reviewer
#1 lore raw

PATCH v3 1/2 mm/slub: introduce helpers for node partial slab state

Wrap partial slab count inc/dec and flag set/clear into helper functions to reduce code duplication.

Note that __add_partial() is called locklessly in early_kmem_cache_node_alloc(), but since there is no such use case for removal, __remove_partial() does not exist.

---
 mm/slub.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
mm/slub.c +17 -6
index b9f10b869914..45aa0a834f97 100644
@@ -3521,15 +3521,21 @@ static inline void slab_clear_node_partial(struct slab *slab)
 /*
  * Management of partially allocated slabs.
  */
+static inline void set_node_partial_state(struct kmem_cache_node *n,
+					struct slab *slab)
+{
+	slab_set_node_partial(slab);
+	n->nr_partial++;
+}
+
 static inline void
 __add_partial(struct kmem_cache_node *n, struct slab *slab, enum add_mode mode)
 {
-	n->nr_partial++;
 	if (mode == ADD_TO_TAIL)
 		list_add_tail(&slab->slab_list, &n->partial);
 	else
 		list_add(&slab->slab_list, &n->partial);
-	slab_set_node_partial(slab);
+	set_node_partial_state(n, slab);
 }
 
 static inline void add_partial(struct kmem_cache_node *n,
@@ -3539,13 +3545,19 @@ static inline void add_partial(struct kmem_cache_node *n,
 	__add_partial(n, slab, mode);
 }
 
+static inline void clear_node_partial_state(struct kmem_cache_node *n,
+					struct slab *slab)
+{
+	slab_clear_node_partial(slab);
+	n->nr_partial--;
+}
+
 static inline void remove_partial(struct kmem_cache_node *n,
 					struct slab *slab)
 {
 	lockdep_assert_held(&n->list_lock);
 	list_del(&slab->slab_list);
-	slab_clear_node_partial(slab);
-	n->nr_partial--;
+	clear_node_partial_state(n, slab);
 }
 
 /*
@@ -8271,8 +8283,7 @@ static int __kmem_cache_do_shrink(struct kmem_cache *s)
 
 			if (free == slab->objects) {
 				list_move(&slab->slab_list, &discard);
-				slab_clear_node_partial(slab);
-				n->nr_partial--;
+				clear_node_partial_state(n, slab);
 				dec_slabs_node(s, node, slab->objects);
 			} else if (free <= SHRINK_PROMOTE_MAX)
 				list_move(&slab->slab_list, promote + free - 1);
Signature · 1 line
2.54.0
author reviewer
#5 lore raw

PATCH v3 2/2 mm/slub: detach and reattach partial slabs in batch

In reply to Hao Li

get_partial_node_bulk() moves each selected slab from the node's partial list to the local pc->slabs list using a remove_partial() and list_add() pair. In practice, the loop often detaches several adjacent slabs. Doing this individually repeatedly manipulates list pointers while holding n->list_lock, which causes unnecessary churn.

To demonstrate this, the counts below show how often single vs. multiple consecutive slabs are retrieved during a will-it-scale mmap stress test:

consecutive_slabs_count        frequency
= 1                            277345324
= 2                            335238023
= 3                            175717884
>= 4                           88862337

The data confirms that retrieving multiple contiguous slabs is highly frequent.

To optimize this, track contiguous runs of matching slabs and move each run in a single operation using list_bulk_move_tail(). This reduces list pointer churn inside the lock critical section.

Apply the same optimization to __refill_objects_node() when reattaching leftover partial slabs back to the node's partial list.

The will-it-scale mmap benchmark shows a 2% ~ 5% performance improvement after applying this patch.

---
 mm/slub.c | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)
mm/slub.c +20 -8
index 45aa0a834f97..4fdf5c36a69b 100644
@@ -3751,6 +3751,7 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
 				  bool allow_spin)
 {
 	struct slab *slab, *slab2;
+	struct slab *first = NULL, *last = NULL;
 	unsigned int total_free = 0;
 	unsigned long flags;
 
@@ -3769,8 +3770,15 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
 		struct freelist_counters flc;
 		unsigned int slab_free;
 
-		if (!pfmemalloc_match(slab, pc->flags))
+		if (!pfmemalloc_match(slab, pc->flags)) {
+			if (first) {
+				list_bulk_move_tail(&pc->slabs,
+						    &first->slab_list,
+						    &last->slab_list);
+				first = NULL;
+			}
 			continue;
+		}
 
 		/*
 		 * determine the number of free objects in the slab racily
@@ -3787,15 +3795,20 @@ static bool get_partial_node_bulk(struct kmem_cache *s,
 		    && total_free + slab_free > pc->max_objects)
 			break;
 
-		remove_partial(n, slab);
-
-		list_add(&slab->slab_list, &pc->slabs);
+		if (!first)
+			first = slab;
+		last = slab;
+		clear_node_partial_state(n, slab);
 
 		total_free += slab_free;
 		if (total_free >= pc->max_objects)
 			break;
 	}
 
+	if (first)
+		list_bulk_move_tail(&pc->slabs, &first->slab_list,
+				    &last->slab_list);
+
 	spin_unlock_irqrestore(&n->list_lock, flags);
 	return total_free > 0;
 }
@@ -7206,11 +7219,10 @@ __refill_objects_node(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int mi
 	if (!list_empty(&pc.slabs)) {
 		spin_lock_irqsave(&n->list_lock, flags);
 
-		list_for_each_entry_safe(slab, slab2, &pc.slabs, slab_list) {
+		list_for_each_entry(slab, &pc.slabs, slab_list)
+			set_node_partial_state(n, slab);
 
-			list_del(&slab->slab_list);
-			add_partial(n, slab, ADD_TO_TAIL);
-		}
+		list_splice_tail(&pc.slabs, &n->partial);
 
 		spin_unlock_irqrestore(&n->list_lock, flags);
 	}
Signature · 1 line
2.54.0
maintainer Applied
#7 lore raw
In reply to Hao Li
Quoted 18 lines · Hao Li

The loops in get_partial_node_bulk() and __refill_objects_node() perform individual list operations for each slab. This patchset optimize this.

Patch 1 is a cleanup patch that wraps duplicated code into helper functions.

Patch 2 reduces the number of list operations by processing slabs in batches, thereby minimizing the time spent inside the critical section.

Changes in v3:
- Reorder the two patches for better logical flow. (Thanks Vlastimil)
- Introduce set_node_partial_state for symmetry. (Thanks Vlastimil)

Changes in v2:
- Patch 1 applies the same optimization to __refill_objects_node. (Thanks Harry and Vlastimil)
- Patch 2 introduces a helper to wraps duplicated code. (Thanks Harry)

Quoted 6 lines

Hao Li (2):
mm/slub: introduce helpers for node partial slab state mm/slub: detach and reattach partial slabs in batch

 mm/slub.c | 51 +++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 37 insertions(+), 14 deletions(-)

Applied to slab/for-next, thanks!

Quoted 1 line

base-commit: 6b41dc6749232c14f3e069f8115084a7bbeee402

maintainer Reviewed-by on v3 1/2
#2 lore raw

Re: [PATCH v3 1/2] mm/slub: introduce helpers for node partial slab state

In reply to Hao Li
Quoted 6 lines · Hao Li

Wrap partial slab count inc/dec and flag set/clear into helper functions to reduce code duplication.

Note that __add_partial() is called locklessly in early_kmem_cache_node_alloc(), but since there is no such use case for removal, __remove_partial() does not exist.

nit: __remove_partial() -> clear_partial_node_state() oh wait, it does exist. Let's drop this paragraph?

Otherwise LGTM!

Quoted 2 lines
Signature · 2 lines
Cheers,
Harry / Hyeonggon
maintainer Reviewed-by on v3 2/2
#6 lore raw

Re: [PATCH v3 2/2] mm/slub: detach and reattach partial slabs in batch

In reply to Hao Li
Quoted 22 lines · Hao Li

get_partial_node_bulk() moves each selected slab from the node's partial list to the local pc->slabs list using a remove_partial() and list_add() pair. In practice, the loop often detaches several adjacent slabs. Doing this individually repeatedly manipulates list pointers while holding n->list_lock, which causes unnecessary churn.

To demonstrate this, the counts below show how often single vs. multiple consecutive slabs are retrieved during a will-it-scale mmap stress test:

consecutive_slabs_count        frequency
= 1                            277345324
= 2                            335238023
= 3                            175717884
Quoted 1 line

= 4 88862337

The data confirms that retrieving multiple contiguous slabs is highly frequent.

To optimize this, track contiguous runs of matching slabs and move each run in a single operation using list_bulk_move_tail(). This reduces list pointer churn inside the lock critical section.

Quoted 8 lines

Apply the same optimization to __refill_objects_node() when reattaching leftover partial slabs back to the node's partial list.

The will-it-scale mmap benchmark shows a 2% ~ 5% performance improvement after applying this patch.

---

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author reviewer
#3 lore raw

Re: [PATCH v3 1/2] mm/slub: introduce helpers for node partial slab state

In reply to Harry Yoo
Quoted 11 lines · Harry Yoo
Quoted 6 lines · Hao Li

Wrap partial slab count inc/dec and flag set/clear into helper functions to reduce code duplication.

Note that __add_partial() is called locklessly in early_kmem_cache_node_alloc(), but since there is no such use case for removal, __remove_partial() does not exist.

nit: __remove_partial() -> clear_partial_node_state()

Ah, in this sentence, I just wanted to clarify that __remove_partial doesn't exist. it's not a typo :)

Quoted 1 line

oh wait, it does exist. Let's drop this paragraph?

This paragraph is just a quick side note to explain why __remove_partial() doesn't exist. I noticed that __add_partial lacks a symmetric counterpart, which felt a bit abrupt/inconsistent, so I wanted to share some context on why that is.

Historically, the __remove_partial function did exist. Back then, we had a use case where free_partial called __remove_partial without holding the lock. Later, free_partial was changed to call remove_partial under the lock, which is why __remove_partial was eventually removed in commit 52b4b950b50740bff507a62907e86710743c22e7.

Quoted 3 lines

Otherwise LGTM!

Thanks!

Quoted 4 lines
Quoted 3 lines
Signature · 2 lines
Thanks,
Hao
maintainer
#4 lore raw

Re: [PATCH v3 1/2] mm/slub: introduce helpers for node partial slab state

Quoted 16 lines · Hao Li
Quoted 11 lines · Harry Yoo
Quoted 6 lines · Hao Li

Wrap partial slab count inc/dec and flag set/clear into helper functions to reduce code duplication.

Note that __add_partial() is called locklessly in early_kmem_cache_node_alloc(), but since there is no such use case for removal, __remove_partial() does not exist.

nit: __remove_partial() -> clear_partial_node_state()

Ah, in this sentence, I just wanted to clarify that __remove_partial doesn't exist. it's not a typo :)

Quoted 6 lines
Quoted 1 line

oh wait, it does exist. Let's drop this paragraph?

This paragraph is just a quick side note to explain why __remove_partial() doesn't exist. I noticed that __add_partial lacks a symmetric counterpart, which felt a bit abrupt/inconsistent, so I wanted to share some context on why that is.

Ah, I see. I was confused because it didn't touch those functions :) Thanks!

Signature · 2 lines
Cheers,
Harry / Hyeonggon