mm/slub: batch-detach node partial slabs
Subsystems (2)
Hao Li
reviewerPATCH 1/1 mm/slub: batch-detach node partial slabs
get_partial_node_bulk() used to move each selected slab from the node partial list to the local pc->slabs list using a remove_partial() and list_add() pair. In practice, the loop often detaches several adjacent slabs, so this repeatedly manipulates list pointers while holding n->list_lock, which causes unnecessary churn.
Instead, track contiguous runs of matching slabs and move each run with list_bulk_move_tail() in one operation. This reduces list pointer churn inside the lock critical section.
The mmap2 testcase shows a 5% improvement after applying this patch.
--- mm/slub.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-)
mm/slub.c +18 -4
@@ -3739,6 +3739,7 @@ static bool get_partial_node_bulk(struct kmem_cache *s, bool allow_spin) {struct slab *slab, *slab2; +struct slab *first =NULL , *last =NULL ;unsigned int total_free =0 ;unsigned long flags;
@@ -3757,8 +3758,15 @@ static bool get_partial_node_bulk(struct kmem_cache *s,struct freelist_counters flc;unsigned int slab_free; -if (!pfmemalloc_match (slab, pc->flags)) +if (!pfmemalloc_match (slab, pc->flags)) { +if (first) { +list_bulk_move_tail (&pc->slabs, + &first->slab_list, + &last->slab_list); + first =NULL ; + }continue ; + }/* * determine the number of free objects in the slab racily
@@ -3775,15 +3783,21 @@ static bool get_partial_node_bulk(struct kmem_cache *s, && total_free + slab_free > pc->max_objects) break; -remove_partial (n, slab); - -list_add (&slab->slab_list, &pc->slabs); +if (!first) + first = slab; + last = slab; +slab_clear_node_partial (slab); + n->nr_partial--; total_free += slab_free;if (total_free >= pc->max_objects)break ; } +if (first) +list_bulk_move_tail (&pc->slabs, &first->slab_list, + &last->slab_list); +spin_unlock_irqrestore (&n->list_lock, flags);return total_free >0 ; }
Signature · 1 line
2.54.0
Harry Yoo
maintainerQuoted 8 lines · Hao Li
get_partial_node_bulk()used to move each selected slab from the node partial list to the localpc->slabslist using aremove_partial()andlist_add()pair. In practice, the loop often detaches several adjacent slabs, so this repeatedly manipulates list pointers while holdingn->list_lock, which causes unnecessary churn.Instead, track contiguous runs of matching slabs and move each run with
list_bulk_move_tail()in one operation.
TIL list_bulk_move_tail() :D
Quoted 1 line
This reduces list pointer churn> inside the lock critical section.
Similar to this, can we return all slabs in pc->slabs at once when returning those slabs to the list? ... I see Vlastimil removed 'nr of empty slabs' check in the other series already.
Now that it inserts slabs to the tail with Vlastimil's patchset, let's do a list_splice_tail() instead?
Quoted 14 lines
The mmap2 testcase shows a 5% improvement after applying this patch.
Signed-off-by Hao Li <[email protected]>--- mm/slub.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-)mm/slub.c +0
-0@@ -3775,15 +3783,21 @@ static bool get_partial_node_bulk(struct kmem_cache *s, && total_free + slab_free > pc->max_objects)break ;
mm/slub.c static bool get_partial_node_bulk(struct kmem_cache *s, · line 3791
Quoted 10 lines
mm/slub.c +5
-3-remove_partial (n, slab); - -list_add (&slab->slab_list, &pc->slabs); +if (!first) + first = slab; + last = slab; +slab_clear_node_partial (slab); + n->nr_partial--;
Perhaps factor out those two statements into to a common function and call it in get_partial_node_bulk() and remove_partial()?
Quoted 12 lines
mm/slub.c +4
-0total_free += slab_free;if (total_free >= pc->max_objects)break ; } +if (first) +list_bulk_move_tail (&pc->slabs, &first->slab_list, + &last->slab_list); +spin_unlock_irqrestore (&n->list_lock, flags);return total_free >0 ; }
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerQuoted 9 lines · Harry Yoo
Quoted 6 lines · Hao Li
get_partial_node_bulk()used to move each selected slab from the node partial list to the localpc->slabslist using aremove_partial()andlist_add()pair. In practice, the loop often detaches several adjacent slabs, so this repeatedly manipulates list pointers while holdingn->list_lock, which causes unnecessary churn.
Quoted 4 lines
Quoted 2 lines
Instead, track contiguous runs of matching slabs and move each run with
list_bulk_move_tail()in one operation.TIL
list_bulk_move_tail():D
I had to dig through list.h for ages just to find it :P
Quoted 9 lines
Quoted 2 lines
This reduces list pointer churn> inside the lock critical section.
Similar to this, can we return all slabs in
pc->slabsat once when returning those slabs to the list? ... I see Vlastimil removed 'nr of empty slabs' check in the other series already.Now that it inserts slabs to the tail with Vlastimil's patchset, let's do a
list_splice_tail()instead?
Great idea! then both get and put operations will be highly efficient.
Quoted 22 lines
Quoted 21 lines
The mmap2 testcase shows a 5% improvement after applying this patch.
Signed-off-by Hao Li <[email protected]>--- mm/slub.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-)mm/slub.c +3
-3@@ -3775,15 +3783,21 @@ static bool get_partial_node_bulk(struct kmem_cache *s, && total_free + slab_free > pc->max_objects)break ; -remove_partial (n, slab); - -list_add (&slab->slab_list, &pc->slabs); +if (!first) + first = slab; + last = slab;
Quoted 5 lines
mm/slub.c static bool get_partial_node_bulk(struct kmem_cache *s, · line 3790
Quoted 2 lines
mm/slub.c +2
-0+slab_clear_node_partial (slab); + n->nr_partial--;Perhaps factor out those two statements into to a common function and call it in
get_partial_node_bulk()andremove_partial()?
Agreed, this is very reasonable.
Quoted 12 lines
mm/slub.c static bool get_partial_node_bulk(struct kmem_cache *s, · line 3803
Quoted 11 lines
mm/slub.c +4
-0total_free += slab_free;if (total_free >= pc->max_objects)break ; } +if (first) +list_bulk_move_tail (&pc->slabs, &first->slab_list, + &last->slab_list); +spin_unlock_irqrestore (&n->list_lock, flags);return total_free >0 ; }
Thanks for the review!
Signature · 2 lines
Thanks, Hao
Vlastimil Babka (SUSE)
maintainerQuoted 9 lines · Harry Yoo
Quoted 6 lines · Hao Li
get_partial_node_bulk()used to move each selected slab from the node partial list to the localpc->slabslist using aremove_partial()andlist_add()pair. In practice, the loop often detaches several adjacent slabs, so this repeatedly manipulates list pointers while holdingn->list_lock, which causes unnecessary churn.
Quoted 6 lines
Quoted 2 lines
Instead, track contiguous runs of matching slabs and move each run with
list_bulk_move_tail()in one operation.TIL
list_bulk_move_tail():DQuoted 1 line
This reduces list pointer churn> inside the lock critical section.
Nice!
Quoted 6 lines
Similar to this, can we return all slabs in
pc->slabsat once when returning those slabs to the list? ... I see Vlastimil removed 'nr of empty slabs' check in the other series already.Now that it inserts slabs to the tail with Vlastimil's patchset, let's do a
list_splice_tail()instead?
Why not, although it should be rare to return more than one slab in that path. So in case this change gets tricky for some reason, we can leave it.
BTW, if you also get the same idea as I had and try to replace the current "remove from pc.slabs initially, reinsert to pc.slabs if it was a partial refill" with a "keep in pc.slabs and only remove if the refill was full" to reduce pointer churn, don't try that, it crashes rather quickly :D
Quoted 39 lines
Quoted 14 lines
The mmap2 testcase shows a 5% improvement after applying this patch.
Signed-off-by Hao Li <[email protected]>---
mm/slub.c| 22 ++++++++++++++++++----1 file changed, 18 insertions(+), 4 deletions(-)mm/slub.c +0
-0@@ -3775,15 +3783,21 @@ static bool get_partial_node_bulk(structkmem_cache *s, && total_free + slab_free > pc->max_objects)mm/slub.c static bool get_partial_node_bulk(struct · line 3792
Quoted 10 lines
mm/slub.c +5
-2break; - remove_partial(n, slab); - - list_add(&slab->slab_list, &pc->slabs); + if (!first) + first = slab; + last = slab; + slab_clear_node_partial(slab); + n->nr_partial--;Perhaps factor out those two statements into to a common function and call it in
get_partial_node_bulk()andremove_partial()?Quoted 11 lines
mm/slub.c +3
-0total_free += slab_free; if (total_free >= pc->max_objects) break; } + if (first) + list_bulk_move_tail(&pc->slabs, &first->slab_list, + &last->slab_list); + spin_unlock_irqrestore(&n->list_lock, flags); return total_free > 0; }
Hao Li
reviewerQuoted 23 lines · Vlastimil Babka (SUSE)
Quoted 9 lines · Harry Yoo
Quoted 6 lines · Hao Li
get_partial_node_bulk()used to move each selected slab from the node partial list to the localpc->slabslist using aremove_partial()andlist_add()pair. In practice, the loop often detaches several adjacent slabs, so this repeatedly manipulates list pointers while holdingn->list_lock, which causes unnecessary churn.Quoted 6 lines
Quoted 2 lines
Instead, track contiguous runs of matching slabs and move each run with
list_bulk_move_tail()in one operation.TIL
list_bulk_move_tail():DQuoted 1 line
This reduces list pointer churn> inside the lock critical section.
Nice!
Quoted 4 lines
Similar to this, can we return all slabs in
pc->slabsat once when returning those slabs to the list? ... I see Vlastimil removed 'nr of empty slabs' check in the other series already.
Quoted 5 lines
Quoted 2 lines
Now that it inserts slabs to the tail with Vlastimil's patchset, let's do a
list_splice_tail()instead?Why not, although it should be rare to return more than one slab in that path.
Yeah, it's definitely still worth doing.
Quoted 1 line
So in case this change gets tricky for some reason, we can leave it.
sure, I will try and test.
Quoted 6 lines
BTW, if you also get the same idea as I had and try to replace the current "remove from pc.slabs initially, reinsert to pc.slabs if it was a partial refill" with a "keep in pc.slabs and only remove if the refill was full" to reduce pointer churn, don't try that, it crashes rather quickly :D
Ah, fair enough. But honestly, it did seem like a good idea initially.
Quoted 41 lines
Quoted 40 lines
Quoted 15 lines
The mmap2 testcase shows a 5% improvement after applying this patch.
Signed-off-by Hao Li <[email protected]>---
mm/slub.c| 22 ++++++++++++++++++----1 file changed, 18 insertions(+), 4 deletions(-)mm/slub.c +0
-0@@ -3775,15 +3783,21 @@ static bool get_partial_node_bulk(structkmem_cache *s, && total_free + slab_free > pc->max_objects)mm/slub.c static bool get_partial_node_bulk(struct · line 3792
Quoted 10 lines
mm/slub.c +5
-2break; - remove_partial(n, slab); - - list_add(&slab->slab_list, &pc->slabs); + if (!first) + first = slab; + last = slab; + slab_clear_node_partial(slab); + n->nr_partial--;Perhaps factor out those two statements into to a common function and call it in
get_partial_node_bulk()andremove_partial()?Quoted 11 lines
mm/slub.c +3
-0total_free += slab_free; if (total_free >= pc->max_objects) break; } + if (first) + list_bulk_move_tail(&pc->slabs, &first->slab_list, + &last->slab_list); + spin_unlock_irqrestore(&n->list_lock, flags); return total_free > 0; }
Signature · 2 lines
Thanks, Hao