mm/slub: batch partial slab list operations
Subsystems (2)
Hao Li
reviewerv3 cover mm/slub: batch partial slab list operations
The loops in get_partial_node_bulk() and __refill_objects_node() perform individual list operations for each slab. This patchset optimize this.
Patch 1 is a cleanup patch that wraps duplicated code into helper functions.
Patch 2 reduces the number of list operations by processing slabs in batches, thereby minimizing the time spent inside the critical section.
Changes in v3:
- Reorder the two patches for better logical flow. (Thanks Vlastimil)
- Introduce set_node_partial_state for symmetry. (Thanks Vlastimil)
Changes in v2:
- Patch 1 applies the same optimization to __refill_objects_node. (Thanks Harry and Vlastimil)
- Patch 2 introduces a helper to wraps duplicated code. (Thanks Harry)
Hao Li (2):
mm/slub: introduce helpers for node partial slab state mm/slub: detach and reattach partial slabs in batch
mm/slub.c | 51 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 37 insertions(+), 14 deletions(-)
base-commit: 6b41dc6749232c14f3e069f8115084a7bbeee402
Signature · 1 line
2.54.0
Hao Li
reviewerPATCH v3 1/2 mm/slub: introduce helpers for node partial slab state
Wrap partial slab count inc/dec and flag set/clear into helper functions to reduce code duplication.
Note that __add_partial() is called locklessly in early_kmem_cache_node_alloc(), but since there is no such use case for removal, __remove_partial() does not exist.
--- mm/slub.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
mm/slub.c +17 -6
@@ -3521,15 +3521,21 @@ static inline void slab_clear_node_partial(struct slab *slab)/* * Management of partially allocated slabs. */ +static inline void set_node_partial_state (struct kmem_cache_node *n, +struct slab *slab) +{ +slab_set_node_partial (slab); + n->nr_partial++; +} +static inline void __add_partial (struct kmem_cache_node *n,struct slab *slab,enum add_mode mode) { - n->nr_partial++;if (mode ==ADD_TO_TAIL )list_add_tail (&slab->slab_list, &n->partial);else list_add (&slab->slab_list, &n->partial); -slab_set_node_partial (slab); +set_node_partial_state (n, slab); } static inline void add_partial(struct kmem_cache_node *n,
@@ -3539,13 +3545,19 @@ static inline void add_partial(struct kmem_cache_node *n,__add_partial (n, slab, mode); } +static inline void clear_node_partial_state (struct kmem_cache_node *n, +struct slab *slab) +{ +slab_clear_node_partial (slab); + n->nr_partial--; +} +static inline void remove_partial (struct kmem_cache_node *n,struct slab *slab) {lockdep_assert_held (&n->list_lock);list_del (&slab->slab_list); -slab_clear_node_partial (slab); - n->nr_partial--; +clear_node_partial_state (n, slab); }/*
@@ -8271,8 +8283,7 @@ static int __kmem_cache_do_shrink(struct kmem_cache *s)if (free == slab->objects) {list_move (&slab->slab_list, &discard); -slab_clear_node_partial (slab); - n->nr_partial--; +clear_node_partial_state (n, slab);dec_slabs_node (s, node, slab->objects); }else if (free <=SHRINK_PROMOTE_MAX )list_move (&slab->slab_list, promote + free -1 );
Signature · 1 line
2.54.0
Re: [PATCH v3 1/2] mm/slub: introduce helpers for node partial slab state
Quoted 6 lines · Hao Li
Wrap partial slab count inc/dec and flag set/clear into helper functions to reduce code duplication.
Note that
__add_partial()is called locklessly inearly_kmem_cache_node_alloc(), but since there is no such use case for removal,__remove_partial()does not exist.
nit: __remove_partial() -> clear_partial_node_state() oh wait, it does exist. Let's drop this paragraph?
Otherwise LGTM!
Quoted 2 lines
Suggested-by Harry Yoo <[email protected]>Signed-off-by Hao Li <[email protected]>
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerRe: [PATCH v3 1/2] mm/slub: introduce helpers for node partial slab state
Quoted 11 lines · Harry Yoo
Quoted 6 lines · Hao Li
Wrap partial slab count inc/dec and flag set/clear into helper functions to reduce code duplication.
Note that
__add_partial()is called locklessly inearly_kmem_cache_node_alloc(), but since there is no such use case for removal,__remove_partial()does not exist.nit:
__remove_partial()->clear_partial_node_state()
Ah, in this sentence, I just wanted to clarify that __remove_partial doesn't exist. it's not a typo :)
Quoted 1 line
oh wait, it does exist. Let's drop this paragraph?
This paragraph is just a quick side note to explain why __remove_partial() doesn't exist. I noticed that __add_partial lacks a symmetric counterpart, which felt a bit abrupt/inconsistent, so I wanted to share some context on why that is.
Historically, the __remove_partial function did exist. Back then, we had a use case where free_partial called __remove_partial without holding the lock. Later, free_partial was changed to call remove_partial under the lock, which is why __remove_partial was eventually removed in commit 52b4b950b50740bff507a62907e86710743c22e7.
Quoted 3 lines
Otherwise LGTM!
Reviewed-by Harry Yoo (Oracle) <[email protected]>
Thanks!
Quoted 4 lines
Quoted 3 lines
Suggested-by Harry Yoo <[email protected]>Signed-off-by Hao Li <[email protected]>
Signature · 2 lines
Thanks, Hao
Harry Yoo
maintainerRe: [PATCH v3 1/2] mm/slub: introduce helpers for node partial slab state
Quoted 16 lines · Hao Li
Quoted 11 lines · Harry Yoo
Quoted 6 lines · Hao Li
Wrap partial slab count inc/dec and flag set/clear into helper functions to reduce code duplication.
Note that
__add_partial()is called locklessly inearly_kmem_cache_node_alloc(), but since there is no such use case for removal,__remove_partial()does not exist.nit:
__remove_partial()->clear_partial_node_state()Ah, in this sentence, I just wanted to clarify that
__remove_partialdoesn't exist. it's not a typo :)
Quoted 6 lines
Quoted 1 line
oh wait, it does exist. Let's drop this paragraph?
This paragraph is just a quick side note to explain why
__remove_partial()doesn't exist. I noticed that__add_partiallacks a symmetric counterpart, which felt a bit abrupt/inconsistent, so I wanted to share some context on why that is.
Ah, I see. I was confused because it didn't touch those functions :) Thanks!
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerPATCH v3 2/2 mm/slub: detach and reattach partial slabs in batch
In reply to Hao Liget_partial_node_bulk() moves each selected slab from the node's partial list to the local pc->slabs list using a remove_partial() and list_add() pair. In practice, the loop often detaches several adjacent slabs. Doing this individually repeatedly manipulates list pointers while holding n->list_lock, which causes unnecessary churn.
To demonstrate this, the counts below show how often single vs. multiple consecutive slabs are retrieved during a will-it-scale mmap stress test:
consecutive_slabs_count frequency = 1 277345324 = 2 335238023 = 3 175717884 >= 4 88862337
The data confirms that retrieving multiple contiguous slabs is highly frequent.
To optimize this, track contiguous runs of matching slabs and move each run in a single operation using list_bulk_move_tail(). This reduces list pointer churn inside the lock critical section.
Apply the same optimization to __refill_objects_node() when reattaching leftover partial slabs back to the node's partial list.
The will-it-scale mmap benchmark shows a 2% ~ 5% performance improvement after applying this patch.
--- mm/slub.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-)
mm/slub.c +20 -8
@@ -3751,6 +3751,7 @@ static bool get_partial_node_bulk(struct kmem_cache *s, bool allow_spin) {struct slab *slab, *slab2; +struct slab *first =NULL , *last =NULL ;unsigned int total_free =0 ;unsigned long flags;
@@ -3769,8 +3770,15 @@ static bool get_partial_node_bulk(struct kmem_cache *s,struct freelist_counters flc;unsigned int slab_free; -if (!pfmemalloc_match (slab, pc->flags)) +if (!pfmemalloc_match (slab, pc->flags)) { +if (first) { +list_bulk_move_tail (&pc->slabs, + &first->slab_list, + &last->slab_list); + first =NULL ; + }continue ; + }/* * determine the number of free objects in the slab racily
@@ -3787,15 +3795,20 @@ static bool get_partial_node_bulk(struct kmem_cache *s, && total_free + slab_free > pc->max_objects) break; -remove_partial (n, slab); - -list_add (&slab->slab_list, &pc->slabs); +if (!first) + first = slab; + last = slab; +clear_node_partial_state (n, slab); total_free += slab_free;if (total_free >= pc->max_objects)break ; } +if (first) +list_bulk_move_tail (&pc->slabs, &first->slab_list, + &last->slab_list); +spin_unlock_irqrestore (&n->list_lock, flags);return total_free >0 ; }
@@ -7206,11 +7219,10 @@ __refill_objects_node(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int miif (!list_empty (&pc.slabs)) {spin_lock_irqsave (&n->list_lock, flags); -list_for_each_entry_safe (slab, slab2, &pc.slabs, slab_list) { +list_for_each_entry (slab, &pc.slabs, slab_list) +set_node_partial_state (n, slab); -list_del (&slab->slab_list); -add_partial (n, slab,ADD_TO_TAIL ); - } +list_splice_tail (&pc.slabs, &n->partial);spin_unlock_irqrestore (&n->list_lock, flags); }
Signature · 1 line
2.54.0
Re: [PATCH v3 2/2] mm/slub: detach and reattach partial slabs in batch
Quoted 22 lines · Hao Li
get_partial_node_bulk()moves each selected slab from the node's partial list to the localpc->slabslist using aremove_partial()andlist_add()pair. In practice, the loop often detaches several adjacent slabs. Doing this individually repeatedly manipulates list pointers while holdingn->list_lock, which causes unnecessary churn.To demonstrate this, the counts below show how often single vs. multiple consecutive slabs are retrieved during a will-it-scale mmap stress test:
consecutive_slabs_count frequency = 1 277345324 = 2 335238023 = 3 175717884Quoted 1 line
= 4 88862337
The data confirms that retrieving multiple contiguous slabs is highly frequent.
To optimize this, track contiguous runs of matching slabs and move each run in a single operation using
list_bulk_move_tail(). This reduces list pointer churn inside the lock critical section.
Quoted 8 lines
Apply the same optimization to
__refill_objects_node()when reattaching leftover partial slabs back to the node's partial list.The will-it-scale mmap benchmark shows a 2% ~ 5% performance improvement after applying this patch.
Signed-off-by Hao Li <[email protected]>---
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 18 lines · Hao Li
The loops in
get_partial_node_bulk()and__refill_objects_node()perform individual list operations for each slab. This patchset optimize this.Patch 1 is a cleanup patch that wraps duplicated code into helper functions.
Patch 2 reduces the number of list operations by processing slabs in batches, thereby minimizing the time spent inside the critical section.
Changes in v3:
- Reorder the two patches for better logical flow. (Thanks Vlastimil)
- Introduceset_node_partial_statefor symmetry. (Thanks Vlastimil)Changes in v2:
- Patch 1 applies the same optimization to__refill_objects_node. (Thanks Harry and Vlastimil)
- Patch 2 introduces a helper to wraps duplicated code. (Thanks Harry)
Quoted 6 lines
Hao Li (2):
mm/slub: introduce helpers for node partial slab state mm/slub: detach and reattach partial slabs in batchmm/slub.c | 51 +++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 37 insertions(+), 14 deletions(-)
Applied to slab/for-next, thanks!
Quoted 1 line
base-commit:
6b41dc6749232c14f3e069f8115084a7bbeee402