slub: spill refill leftover objects into percpu sheaves
Subsystems (2)
Hao Li
reviewerRFC PATCH 1/1 slub: spill refill leftover objects into percpu sheaves
When performing objects refill, we tend to optimistically assume that there will be more allocation requests coming next; this is the fundamental assumption behind this optimization.
When __refill_objects_node() isolates a partial slab and satisfies a bulk allocation from its freelist, the slab can still have a small tail of free objects left over. Today those objects are freed back to the slab immediately.
If the leftover tail is local and small enough to fit, keep it in the current CPU's sheaves instead. This avoids pushing those objects back through the __slab_free slowpath.
Add a helper to obtain both the freelist and its free-object count, and then spill the remaining objects into a percpu sheaf when:
- the tail fits in a sheaf
- the slab is local to the current CPU
- the slab is not pfmemalloc
- the target sheaf has enough free space
Otherwise keep the existing fallback and free the tail back to the slab.
Also add a SHEAF_SPILL stat so the new path can be observed in SLUB stats.
On the mmap2 case in the will-it-scale benchmark suite, this patch can improve performance by about 2~5%.
---
This patch is an exploratory attempt to address the leftover objects and partial slab issues in the refill path, and it is marked as RFC to warmly welcome any feedback, suggestions, and discussion!
--- mm/slub.c | 107 ++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 88 insertions(+), 19 deletions(-)
mm/slub.c +88 -19
@@ -353,6 +353,7 @@ enum stat_item {SHEAF_REFILL ,/* Objects refilled to a sheaf */ SHEAF_ALLOC ,/* Allocation of an empty sheaf */ SHEAF_FREE ,/* Freeing of an empty sheaf */ +SHEAF_SPILL ,BARN_GET ,/* Got full sheaf from barn */ BARN_GET_FAIL ,/* Failed to get full sheaf from barn */ BARN_PUT, /* Put full sheaf to barn */
@@ -4279,7 +4280,9 @@ static inline bool pfmemalloc_match(struct slab *slab, gfp_t gfpflags)* Assumes this is performed only for caches without debugging so we * don't need to worry about adding the slab to the full list. */ -static inline void *get_freelist_nofreeze (struct kmem_cache *s,struct slab *slab) +static inline void *__get_freelist_nofreeze (struct kmem_cache *s, +struct slab *slab,int *freecount, +const char *n) {struct freelist_counters old, new;
@@ -4293,11 +4296,26 @@ static inline void *get_freelist_nofreeze(struct kmem_cache *s, struct slab *sla new.inuse = old.objects; - }while (!slab_update_freelist (s, slab, &old, &new,"get_freelist_nofreeze" )); + }while (!slab_update_freelist (s, slab, &old, &new, n)); + +if (freecount) + *freecount = old.objects - old.inuse;return old.freelist; } +static inline void *get_freelist_nofreeze (struct kmem_cache *s,struct slab *slab) +{ +return __get_freelist_nofreeze (s, slab,NULL ,"get_freelist_nofreeze" ); +} + +static inline void *get_freelist_and_freecount_nofreeze (struct kmem_cache *s, +struct slab *slab, +int *freecount) +{ +return __get_freelist_nofreeze (s, slab, freecount,"get_freelist_and_freecount_nofreeze" ); +} +/* * If the object has been wiped upon free, make sure it's fully initialized by * zeroing out freelist pointer.
@@ -7028,10 +7046,15 @@ __refill_objects_node(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int mireturn 0 ;list_for_each_entry_safe (slab, slab2, &pc.slabs, slab_list) { +void *head; +void *tail; +struct slub_percpu_sheaves *pcs; +int freecount, local_node, i, cnt =0 ; +struct slab_sheaf *spill;list_del (&slab->slab_list); - object =get_freelist_nofreeze (s, slab); + object =get_freelist_and_freecount_nofreeze (s, slab, &freecount);while (object && refilled < max) { p[refilled] = object;
@@ -7039,28 +7062,72 @@ __refill_objects_node(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int mimaybe_wipe_obj_freeptr (s, p[refilled]); refilled++; + freecount--; } +if (!freecount) { +if (refilled >= max) +break ; +continue ; + }/* -* Freelist had more objects than we can accommodate, we need to -* free them back. We can treat it like a detached freelist, just -* need to find the tail object. +* Freelist had more objects than we can accommodate, we first +* try to spill them into percpu sheaf. */ -if (unlikely (object)) { -void *head = object; -void *tail; -int cnt =0 ; - -do { - tail = object; - cnt++; - object =get_freepointer (s, object); - }while (object); -__slab_free (s, slab, head, tail, cnt, _RET_IP_); +if (freecount > s->sheaf_capacity) + goto skip_spill; +if (slab_test_pfmemalloc (slab)) + goto skip_spill; + +if (!local_trylock (&s->cpu_sheaves->lock)) + goto skip_spill; + + local_node =numa_mem_id (); +if (slab_nid (slab) != local_node) { +local_unlock (&s->cpu_sheaves->lock); + goto skip_spill; } -if (refilled >= max) -break ; + pcs =this_cpu_ptr (s->cpu_sheaves); +if (pcs->spare && + (freecount <= (s->sheaf_capacity - pcs->spare->size))) + spill = pcs->spare; +else if (freecount <= (s->sheaf_capacity - pcs->main->size)) + spill = pcs->main; +else { +local_unlock (&s->cpu_sheaves->lock); + goto skip_spill; + } + +if (freecount > (s->sheaf_capacity - spill->size)) { +local_unlock (&s->cpu_sheaves->lock); + goto skip_spill; + } + +for (i =0 ; i < freecount; i++) { + spill->objects[spill->size] = object; + object =get_freepointer (s, object); +maybe_wipe_obj_freeptr (s, spill->objects[spill->size]); + spill->size++; + } + +local_unlock (&s->cpu_sheaves->lock); +stat (s,SHEAF_SPILL ); +break ; +skip_spill: +/* +* Freelist had more objects than we can accommodate or spill, +* we need to free them back. We can treat it like a detached freelist, +* just need to find the tail object. +*/ + head = object; +do { + tail = object; + cnt++; + object =get_freepointer (s, object); + }while (object); +__slab_free (s, slab, head, tail, cnt, _RET_IP_); +break ; }if (unlikely (!list_empty (&pc.slabs))) {
@@ -9247,6 +9314,7 @@ STAT_ATTR(SHEAF_FLUSH, sheaf_flush);STAT_ATTR (SHEAF_REFILL , sheaf_refill);STAT_ATTR (SHEAF_ALLOC , sheaf_alloc);STAT_ATTR (SHEAF_FREE , sheaf_free); +STAT_ATTR (SHEAF_SPILL , sheaf_spill);STAT_ATTR (BARN_GET , barn_get);STAT_ATTR (BARN_GET_FAIL , barn_get_fail);STAT_ATTR (BARN_PUT , barn_put);
@@ -9335,6 +9403,7 @@ static struct attribute *slab_attrs[] = { &sheaf_refill_attr.attr, &sheaf_alloc_attr.attr, &sheaf_free_attr.attr, + &sheaf_spill_attr.attr, &barn_get_attr.attr, &barn_get_fail_attr.attr, &barn_put_attr.attr,
Signature · 1 line
2.50.1
Harry Yoo (Oracle)
maintainerQuoted 3 lines · Hao Li
When performing objects refill, we tend to optimistically assume that there will be more allocation requests coming next; this is the fundamental assumption behind this optimization.
I think the reason why currently we have two sheaves per CPU instead of one bigger sheaf is to avoid unfairly pessimizing when the alloc/free pattern frequently changes.
By refilling more objects, frees are more likely to hit the slowpath. How can it be argued that this optimization is beneficial to have in general, not just for caches with specific alloc/free patterns?
Quoted 8 lines
When
__refill_objects_node()isolates a partial slab and satisfies a bulk allocation from its freelist, the slab can still have a small tail of free objects left over. Today those objects are freed back to the slab immediately.If the leftover tail is local and small enough to fit, keep it in the current CPU's sheaves instead. This avoids pushing those objects back through the
__slab_freeslowpath.
So there are two different paths:
1. When refilling prefilled sheaves, spill objects into ->main and ->spare.
2. When refilling ->main sheaf, spill objects into ->spare.
Quoted 7 lines
Add a helper to obtain both the freelist and its free-object count, and then spill the remaining objects into a percpu sheaf when:
- the tail fits in a sheaf
- the slab is local to the current CPU
- the slab is not pfmemalloc
- the target sheaf has enough free space
Quoted 8 lines
Otherwise keep the existing fallback and free the tail back to the slab.
Also add a
SHEAF_SPILLstat so the new path can be observed in SLUB stats.On the mmap2 case in the will-it-scale benchmark suite,
this patch can improve performance by about 2~5%.
Where do you think the improvement comes from? (hopefully w/ some data)
e.g.:
1. the benefit is from largely or partly from
reduced contention on n->list_lock.
2. this change reduces # of alloc slowpath at the cost of increased of free slowpath hits, but that's better because the slowpath frees are mostly lockless.
3. the alloc/free pattern of the workload is benefiting from spilling objects to the CPU's sheaves.
or something else?
Quoted 6 lines
Signed-off-by Hao Li <[email protected]>---
This patch is an exploratory attempt to address the leftover objects and partial slab issues in the refill path, and it is marked as RFC to warmly welcome any feedback, suggestions, and discussion!
Yeah, let's discuss!
By the way, have you also been considering having min-max capacity for sheaves? (that I think Vlastimil suggested somewhere)
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerQuoted 8 lines · Harry Yoo (Oracle)
Quoted 3 lines · Hao Li
When performing objects refill, we tend to optimistically assume that there will be more allocation requests coming next; this is the fundamental assumption behind this optimization.
I think the reason why currently we have two sheaves per CPU instead of one bigger sheaf is to avoid unfairly pessimizing when the alloc/free pattern frequently changes.
Yes.
Quoted 4 lines
By refilling more objects, frees are more likely to hit the slowpath. How can it be argued that this optimization is beneficial to have in general, not just for caches with specific alloc/free patterns?
Yes, that's a very valid concern. My thinking here is that the leftover objects have to be kept somewhere after all, so in this current experimental implementation I'm trading off future free-path performance for better allocation performance. It's a pretty tough trade-off either way :/
Quoted 10 lines
Quoted 9 lines
When
__refill_objects_node()isolates a partial slab and satisfies a bulk allocation from its freelist, the slab can still have a small tail of free objects left over. Today those objects are freed back to the slab immediately.If the leftover tail is local and small enough to fit, keep it in the current CPU's sheaves instead. This avoids pushing those objects back through the
__slab_freeslowpath.
Quoted 5 lines
So there are two different paths:
1. When refilling prefilled sheaves, spill objects into ->main and ->spare.
2. When refilling ->main sheaf, spill objects into ->spare.
the current experimental code is biased toward spilling into the spare sheaf when possible.
for kernels without kernel preemption enabled or !RT, the spare sheaf is generally NULL at that point, so the main sheaf may still end up being the primary place to absorb the spill...
Quoted 10 lines
Quoted 10 lines
Add a helper to obtain both the freelist and its free-object count, and then spill the remaining objects into a percpu sheaf when:
- the tail fits in a sheaf
- the slab is local to the current CPU
- the slab is not pfmemalloc
- the target sheaf has enough free spaceOtherwise keep the existing fallback and free the tail back to the slab.
Quoted 8 lines
Quoted 6 lines
Also add a
SHEAF_SPILLstat so the new path can be observed in SLUB stats.On the mmap2 case in the will-it-scale benchmark suite,
this patch can improve performance by about 2~5%.
Where do you think the improvement comes from? (hopefully w/ some data)
Yes, this is necessary.
Quoted 4 lines
e.g.:
1. the benefit is from largely or partly fromreduced contention on n->list_lock.
Before this patch is applied, the mmap benchmark shows the following hot path:
- 7.85% native_queued_spin_lock_slowpath
-7.85% _raw_spin_lock_irqsave
- 3.69% __slab_free
+ 1.84% __refill_objects_node
+ 1.77% __kmem_cache_free_bulk
+ 3.27% __refill_objects_node
With the patch applied, the __refill_objects_node -> __slab_free hotspot goes away, and the native_queued_spin_lock_slowpath drops to roughly 3.5%. The remaining lock contention is mostly between __refill_objects_node -> add_partial and __kmem_cache_free_bulk -> __slab_free.
Quoted 4 lines
2. this change reduces # of alloc slowpath at the cost of increased of free slowpath hits, but that's better because the slowpath frees are mostly lockless.
The alloc slowpath remains at 0 both w/ or w/o the patch, whereas the free slowpath increases by 2x after applying the patch.
Quoted 5 lines
3. the alloc/free pattern of the workload is benefiting from spilling objects to the CPU's sheaves.
or something else?
The 2-5% throughput improvement does seem to come with some trade-offs. The main one is that leftover objects get hidden in the percpu sheaves now, which reduces the objects on the node partial list and thus indirectly increases slab alloc/free frequency to about 4x of the baseline.
This is a drawback of the current approach. :/
I experimented with several alternative ideas, and the pattern seems fairly
consistent: as soon as leftover objects are hidden at the percpu level, slab alloc/free churn tends to go up.
Quoted 9 lines
Quoted 7 lines
Signed-off-by Hao Li <[email protected]>---
This patch is an exploratory attempt to address the leftover objects and partial slab issues in the refill path, and it is marked as RFC to warmly welcome any feedback, suggestions, and discussion!
Yeah, let's discuss!
Sure! Thanks for the discussion!
Quoted 3 lines
By the way, have you also been considering having min-max capacity for sheaves? (that I think Vlastimil suggested somewhere)
Yes, I also tried it.
I experimented with using a manually chosen threshold to allow refill to leave the sheaf in a partially filled state. However, since concurrent frees are inherently unpredictable, this seems can only reduce the probability of generating leftover objects, while at the same time affecting alloc-side throughput. In my testing, the results were not very encouraging: it seems hard to observe improvement, and in most cases it ended up causing a performance regression.
my impression is that it could be difficult to prevent leftovers proactively. It may be easier to deal with them after they appear.
Besides, I also tried another idea: maintaining a dedicated spill sheaf in the barn, protected by the barn lock, and placing leftover objects there. Then, during refill, barn_replace_empty_sheaf() would first try the spill sheaf, and if it contained objects, it would swap spill and main, avoiding consumption from barn->full_list.
With this approach, I still couldn't observe meaningful performance change. Slab alloc/free churn still present, although the increase was relatively small, at around 1.x
My guess is that while this approach pulls leftovers up to the barn level and avoids the cost of pushing them back down to the node partial list level, the serialized nature of the barn lock means leftovers cannot be deposited into the spill sheaf with high concurrency. As a result, the placement is not fast enough, and the performance gain remains limited.
Signature · 2 lines
Thanks, Hao
Harry Yoo (Oracle)
maintainerQuoted 15 lines · Hao Li
Quoted 14 lines · Harry Yoo (Oracle)
Quoted 12 lines · Hao Li
When performing objects refill, we tend to optimistically assume that there will be more allocation requests coming next; this is the fundamental assumption behind this optimization.
When
__refill_objects_node()isolates a partial slab and satisfies a bulk allocation from its freelist, the slab can still have a small tail of free objects left over. Today those objects are freed back to the slab immediately.If the leftover tail is local and small enough to fit, keep it in the current CPU's sheaves instead. This avoids pushing those objects back through the
__slab_freeslowpath.
Quoted 8 lines
Quoted 5 lines
So there are two different paths:
1. When refilling prefilled sheaves, spill objects into ->main and ->spare.
2. When refilling ->main sheaf, spill objects into ->spare.the current experimental code is biased toward spilling into the spare sheaf when possible.
Oh ok.
Quoted 2 lines
for kernels without kernel preemption enabled or !RT, the spare sheaf is generally NULL at that point,
Right. We're either refilling the previously-spare-sheaf (->spare = NULL now) or an empty sheaf because ->spare was NULL. (in both cases 1 and 2)
Quoted 36 lines
so the main sheaf may still end up being the
primary place to absorb the spill...Quoted 9 lines
Quoted 9 lines
Add a helper to obtain both the freelist and its free-object count, and then spill the remaining objects into a percpu sheaf when:
- the tail fits in a sheaf
- the slab is local to the current CPU
- the slab is not pfmemalloc
- the target sheaf has enough free spaceOtherwise keep the existing fallback and free the tail back to the slab.
Quoted 8 lines
Quoted 6 lines
Also add a
SHEAF_SPILLstat so the new path can be observed in SLUB stats.On the mmap2 case in the will-it-scale benchmark suite,
this patch can improve performance by about 2~5%.
Where do you think the improvement comes from? (hopefully w/ some data)
Yes, this is necessary.
Quoted 3 lines
e.g.:
1. the benefit is from largely or partly fromreduced contention on n->list_lock.Before this patch is applied, the mmap benchmark shows the following hot path:
- 7.85%
native_queued_spin_lock_slowpath
-7.85%_raw_spin_lock_irqsave
- 3.69%__slab_free
+ 1.84%__refill_objects_node
+ 1.77%__kmem_cache_free_bulk
+ 3.27%__refill_objects_node
Quoted 2 lines
With the patch applied, the
__refill_objects_node->__slab_freehotspot goes away, and thenative_queued_spin_lock_slowpathdrops to roughly 3.5%.
Sounds like returning slabs back indeed increases contention on slowpath.
Quoted 10 lines
The
remaining lock contention is mostly between__refill_objects_node->add_partialand__kmem_cache_free_bulk->__slab_free.Quoted 4 lines
2. this change reduces # of alloc slowpath at the cost of increased of free slowpath hits, but that's better because the slowpath frees are mostly lockless.
The alloc slowpath remains at 0 both w/ or w/o the patch, whereas the
(assuming you used SLUB_STATS for this)
That's weird, I think we should check SHEAF_REFILL instead of ALLOC_SLOWPATH.
Quoted 1 line
free slowpath increases by 2x after applying the patch.
from which cache was this stat collected?
Quoted 4 lines
Quoted 4 lines
3. the alloc/free pattern of the workload is benefiting from spilling objects to the CPU's sheaves.
Quoted 8 lines
Quoted 1 line
or something else?
The 2-5% throughput improvement does seem to come with some trade-offs. The main one is that leftover objects get hidden in the percpu sheaves now, which reduces the objects on the node partial list and thus indirectly increases slab alloc/free frequency to about 4x of the baseline.
This is a drawback of the current approach. :/
Sounds like s->min_partial is too small now that we cache more objects per CPU.
/me wonders if increasing sheaf capacity would make more sense rather than optimizing slowpath (if it comes with increased memory usage anyway), but then stares at his (yet) unfinished patch series...
Quoted 19 lines
I experimented with several alternative ideas, and the pattern seems fairly
consistent: as soon as leftover objects are hidden at the percpu level, slab alloc/free churn tends to go up.Quoted 8 lines
Quoted 6 lines
Signed-off-by Hao Li <[email protected]>---
This patch is an exploratory attempt to address the leftover objects and partial slab issues in the refill path, and it is marked as RFC to warmly welcome any feedback, suggestions, and discussion!
Yeah, let's discuss!
Sure! Thanks for the discussion!
Quoted 3 lines
By the way, have you also been considering having min-max capacity for sheaves? (that I think Vlastimil suggested somewhere)
Quoted 6 lines
Yes, I also tried it.
I experimented with using a manually chosen threshold to allow refill to leave the sheaf in a partially filled state. However, since concurrent frees are inherently unpredictable, this seems can only reduce the probability of generating leftover objects,
If concurrent frees are a problem we could probably grab slab->freelist under n->list_lock (e.g. keep them at the end of the sheaf) and fill the sheaf outside the lock to avoid grabbing too many objects.
Quoted 1 line
while at the same time affecting alloc-side throughput.
Shouldn't we set sheaf's min capacity as the same as s->sheaf_capacity and allow higher max capcity to avoid this?
Quoted 6 lines
In my testing, the results were not very encouraging: it seems hard to observe improvement, and in most cases it ended up causing a performance regression.
my impression is that it could be difficult to prevent leftovers proactively. It may be easier to deal with them after they appear.
Either way doesn't work if the slab order is too high...
IIRC using higher slab order used to have some benefit but now that we have sheaves, it probably doesn't make sense anymore to have oo_objects(s->oo) > s->sheaf_capacity?
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerQuoted 25 lines · Harry Yoo (Oracle)
Quoted 15 lines · Hao Li
Quoted 14 lines · Harry Yoo (Oracle)
Quoted 12 lines · Hao Li
When performing objects refill, we tend to optimistically assume that there will be more allocation requests coming next; this is the fundamental assumption behind this optimization.
When
__refill_objects_node()isolates a partial slab and satisfies a bulk allocation from its freelist, the slab can still have a small tail of free objects left over. Today those objects are freed back to the slab immediately.If the leftover tail is local and small enough to fit, keep it in the current CPU's sheaves instead. This avoids pushing those objects back through the
__slab_freeslowpath.Quoted 8 lines
Quoted 5 lines
So there are two different paths:
1. When refilling prefilled sheaves, spill objects into ->main and ->spare.
2. When refilling ->main sheaf, spill objects into ->spare.the current experimental code is biased toward spilling into the spare sheaf when possible.
Quoted 7 lines
Oh ok.
Quoted 2 lines
for kernels without kernel preemption enabled or !RT, the spare sheaf is generally NULL at that point,
Right. We're either refilling the previously-spare-sheaf (->spare = NULL now) or an empty sheaf because ->spare was NULL. (in both cases 1 and 2)
Yes.
Quoted 37 lines
Quoted 37 lines
so the main sheaf may still end up being the
primary place to absorb the spill...Quoted 9 lines
Quoted 9 lines
Add a helper to obtain both the freelist and its free-object count, and then spill the remaining objects into a percpu sheaf when:
- the tail fits in a sheaf
- the slab is local to the current CPU
- the slab is not pfmemalloc
- the target sheaf has enough free spaceOtherwise keep the existing fallback and free the tail back to the slab.
Quoted 8 lines
Quoted 6 lines
Also add a
SHEAF_SPILLstat so the new path can be observed in SLUB stats.On the mmap2 case in the will-it-scale benchmark suite,
this patch can improve performance by about 2~5%.
Where do you think the improvement comes from? (hopefully w/ some data)
Yes, this is necessary.
Quoted 3 lines
e.g.:
1. the benefit is from largely or partly fromreduced contention on n->list_lock.Before this patch is applied, the mmap benchmark shows the following hot path:
- 7.85%
native_queued_spin_lock_slowpath
-7.85%_raw_spin_lock_irqsave
- 3.69%__slab_free
+ 1.84%__refill_objects_node
+ 1.77%__kmem_cache_free_bulk
+ 3.27%__refill_objects_node
Quoted 4 lines
Quoted 2 lines
With the patch applied, the
__refill_objects_node->__slab_freehotspot goes away, and thenative_queued_spin_lock_slowpathdrops to roughly 3.5%.Sounds like returning slabs back indeed increases contention on slowpath.
Indeed!
Quoted 6 lines
Quoted 6 lines
The
remaining lock contention is mostly between__refill_objects_node->add_partialand__kmem_cache_free_bulk->__slab_free.Quoted 1 line
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
2. this change reduces # of alloc slowpath at the cost of increased of free slowpath hits, but that's better because the slowpath frees are mostly lockless.
The alloc slowpath remains at 0 both w/ or w/o the patch, whereas the
(assuming you used
SLUB_STATSfor this)
Yes, I enable it.
Quoted 2 lines
That's weird, I think we should check
SHEAF_REFILLinstead ofALLOC_SLOWPATH.
Yes, I will compare each metrics for later testing. Maybe we can see more clues.
Quoted 4 lines
Quoted 2 lines
free slowpath increases by 2x after applying the patch.
from which cache was this stat collected?
It's for /sys/kernel/slab/maple_node/
Quoted 12 lines
Quoted 12 lines
Quoted 6 lines
3. the alloc/free pattern of the workload is benefiting from spilling objects to the CPU's sheaves.
or something else?
The 2-5% throughput improvement does seem to come with some trade-offs. The main one is that leftover objects get hidden in the percpu sheaves now, which reduces the objects on the node partial list and thus indirectly increases slab alloc/free frequency to about 4x of the baseline.
Quoted 4 lines
Quoted 1 line
This is a drawback of the current approach. :/
Sounds like
s->min_partialis too small now that we cache more objects per CPU.
Exactly. for the mmap test case, the slab partial list keeps thrashing. It makes me wonder whether SLUB might handle transient pressure better if empty slabs could be regulated with a "dynamic burst threshold"
Quoted 4 lines
/me wonders if increasing sheaf capacity would make more sense rather than optimizing slowpath (if it comes with increased memory usage anyway),
Yes, finding ways to avoid falling onto the slowpath is also very worthwhile.
Quoted 23 lines
but then stares at his (yet) unfinished patch series...
Quoted 21 lines
I experimented with several alternative ideas, and the pattern seems fairly
consistent: as soon as leftover objects are hidden at the percpu level, slab alloc/free churn tends to go up.Quoted 8 lines
Quoted 6 lines
Signed-off-by Hao Li <[email protected]>---
This patch is an exploratory attempt to address the leftover objects and partial slab issues in the refill path, and it is marked as RFC to warmly welcome any feedback, suggestions, and discussion!
Yeah, let's discuss!
Sure! Thanks for the discussion!
Quoted 3 lines
By the way, have you also been considering having min-max capacity for sheaves? (that I think Vlastimil suggested somewhere)
Yes, I also tried it.
Quoted 8 lines
Quoted 4 lines
I experimented with using a manually chosen threshold to allow refill to leave the sheaf in a partially filled state. However, since concurrent frees are inherently unpredictable, this seems can only reduce the probability of generating leftover objects,
If concurrent frees are a problem we could probably grab
slab->freelistundern->list_lock(e.g. keep them at the end of the sheaf) and fill the sheaf outside the lock to avoid grabbing too many objects.
Do you mean doing an on-list bulk allocation?
Quoted 5 lines
Quoted 2 lines
while at the same time affecting alloc-side throughput.
Shouldn't we set sheaf's min capacity as the same as
s->sheaf_capacityand allow higher max capcity to avoid this?
I'm not sure I fully understand this. since the array size is fixed, how would we allow more entries to be filled?
Quoted 5 lines
Quoted 5 lines
In my testing, the results were not very encouraging: it seems hard to observe improvement, and in most cases it ended up causing a performance regression.
Quoted 8 lines
Quoted 2 lines
my impression is that it could be difficult to prevent leftovers proactively. It may be easier to deal with them after they appear.
Either way doesn't work if the slab order is too high...
IIRC using higher slab order used to have some benefit but now that we have sheaves, it probably doesn't make sense anymore to have
oo_objects(s->oo)>s->sheaf_capacity?
Do you mean considering making the capacity of each sheaf larger than oo_objects?
That could reduce the probability of leftovers, though I think that would be more of a separate optimization of sheaf capacity.
Signature · 2 lines
Thanks, Hao
Harry Yoo (Oracle)
maintainerQuoted 46 lines · Hao Li
Quoted 20 lines · Harry Yoo (Oracle)
Quoted 19 lines · Hao Li
Quoted 2 lines · Harry Yoo (Oracle)
On Fri, Apr 10, 2026 at 07:16:57PM +0800, Hao Li wrote:
Where do you think the improvement comes from? (hopefully w/ some data)Yes, this is necessary.
Quoted 3 lines
e.g.:
1. the benefit is from largely or partly fromreduced contention on n->list_lock.Before this patch is applied, the mmap benchmark shows the following hot path:
- 7.85%
native_queued_spin_lock_slowpath
-7.85%_raw_spin_lock_irqsave
- 3.69%__slab_free
+ 1.84%__refill_objects_node
+ 1.77%__kmem_cache_free_bulk
+ 3.27%__refill_objects_nodeQuoted 4 lines
Quoted 2 lines
With the patch applied, the
__refill_objects_node->__slab_freehotspot goes away, and thenative_queued_spin_lock_slowpathdrops to roughly 3.5%.Sounds like returning slabs back indeed increases contention on slowpath.
Indeed!
Quoted 5 lines
Quoted 5 lines
The
remaining lock contention is mostly between__refill_objects_node->add_partialand__kmem_cache_free_bulk->__slab_free.Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
2. this change reduces # of alloc slowpath at the cost of increased of free slowpath hits, but that's better because the slowpath frees are mostly lockless.
The alloc slowpath remains at 0 both w/ or w/o the patch, whereas the
(assuming you used
SLUB_STATSfor this)Yes, I enable it.
Quoted 2 lines
That's weird, I think we should check
SHEAF_REFILLinstead ofALLOC_SLOWPATH.
Quoted 8 lines
Yes, I will compare each metrics for later testing. Maybe we can see more clues.
Quoted 3 lines
Quoted 1 line
free slowpath increases by 2x after applying the patch.
from which cache was this stat collected?
It's for
/sys/kernel/slab/maple_node/
Ack. And you also mentioned (off-list) that kmem_cache_refill_sheaf() is not on the profile. That's good to know.
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
3. the alloc/free pattern of the workload is benefiting from spilling objects to the CPU's sheaves.
or something else?
The 2-5% throughput improvement does seem to come with some trade-offs. The main one is that leftover objects get hidden in the percpu sheaves now, which reduces the objects on the node partial list and thus indirectly increases slab alloc/free frequency to about 4x of the baseline.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
This is a drawback of the current approach. :/
Sounds like
s->min_partialis too small now that we cache more objects per CPU.Exactly. for the mmap test case, the slab partial list keeps thrashing. It makes me wonder whether SLUB might handle transient pressure better if empty slabs could be regulated with a "dynamic burst threshold"
Haha, we'll be constantly challenged to find balance between "sacrifice memory to make every benchmark happy" vs. "provide reasonable scalability in general but let users tune it themselves".
If we could implement a reasonably simple yet effective automatic tuning method, having one in the kernel would be nice (though of course having it userspace would be the best).
Quoted 5 lines
Quoted 3 lines
/me wonders if increasing sheaf capacity would make more sense rather than optimizing slowpath (if it comes with increased memory usage anyway),
Yes, finding ways to avoid falling onto the slowpath is also very worthwhile.
Could you please take a look at how much changing 1) sheaf capacity and
2) nr of full/empty sheaves at the barn affects the performance of mmap / ublk performance?
I've been trying to reproduce the regression on my machine but haven't had much success so far :(
(I'll try to post the RFC patchset to allow changing those parameters at runtime in few weeks but if you're eager you could try experimenting by changing the code :D)
Quoted 28 lines
Quoted 28 lines
but then stares at his (yet) unfinished patch series...
Quoted 25 lines
I experimented with several alternative ideas, and the pattern seems fairly
consistent: as soon as leftover objects are hidden at the percpu level, slab alloc/free churn tends to go up.Quoted 8 lines
Quoted 6 lines
Signed-off-by Hao Li <[email protected]>---
This patch is an exploratory attempt to address the leftover objects and partial slab issues in the refill path, and it is marked as RFC to warmly welcome any feedback, suggestions, and discussion!
Yeah, let's discuss!
Sure! Thanks for the discussion!
Quoted 3 lines
By the way, have you also been considering having min-max capacity for sheaves? (that I think Vlastimil suggested somewhere)
Yes, I also tried it.
I experimented with using a manually chosen threshold to allow refill to leave the sheaf in a partially filled state. However, since concurrent frees are inherently unpredictable, this seems can only reduce the probability of generating leftover objects,
Quoted 5 lines
Quoted 3 lines
If concurrent frees are a problem we could probably grab
slab->freelistundern->list_lock(e.g. keep them at the end of the sheaf) and fill the sheaf outside the lock to avoid grabbing too many objects.Do you mean doing an on-list bulk allocation?
Just brainstorming... it's quite messy :)
something like
__refill_objects_node(s, p, gfp, min, max, n, allow_spin) {
// in practice we don't know how many slabs we'll grab.
// so probably keep them somewhere e.g.) the end of `p` array?
void *freelists[min];
nr_freelists = 0;
nr_objs = 0;
spin_lock_irqsave();
for each slab in n->partial {
freelist = slab->freelist;
do {
[...]
old.freelist = slab->freelist;
[...]
} while (!__slab_update_freelist(...));
freelists[nr_freelists++] = old.freelist;
nr_objs += (old.objects - old.inuse);
if (!new.inuse)
remove_partial();
if (nr_objs >= min)
break;
}
spin_unlock_irqrestore();
i = 0;
j = 0;
while (i < nr_freelists) {
freelist = freelists[i++];
while (freelist != NULL) {
if (j == max) {
// free remaining objects
}
next = get_freepointer(s, freelist);
p[j++] = freelist;
freelist = next;
}
}
}This way, we know how many objects we grabbed but yeah it's tricky.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
while at the same time affecting alloc-side throughput.
Shouldn't we set sheaf's min capacity as the same as
s->sheaf_capacityand allow higher max capcity to avoid this?I'm not sure I fully understand this. since the array size is fixed, how would we allow more entries to be filled?
I don't really want to speak on behalf of Vlastimil but I was imagining something like:
before: sheaf->capacity (32, min = max);
after: sheaf->capacity (48 or 64, max), sheaf->threshold (32, min)
so that sheaf refill will succeed if at least ->threshold objects are filled, but the threshold better not be smaller than 32 (the previous sheaf->capacity)?
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
In my testing, the results were not very encouraging: it seems hard to observe improvement, and in most cases it ended up causing a performance regression.
my impression is that it could be difficult to prevent leftovers proactively. > > > It may be easier to deal with them after they appear.
Quoted 8 lines
Quoted 5 lines
Either way doesn't work if the slab order is too high...
IIRC using higher slab order used to have some benefit but now that we have sheaves, it probably doesn't make sense anymore to have
oo_objects(s->oo)>s->sheaf_capacity?Do you mean considering making the capacity of each sheaf larger than
oo_objects?
I mean the other way around. calculate_order() tends to increase slab order with higher number of CPUs (by setting higher min_objects), but is it still worth having oo_objects higher than the sheaf capacity?
Quoted 2 lines
That could reduce the probability of leftovers, though I think that would be more of a separate optimization of sheaf capacity.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerQuoted 78 lines · Harry Yoo (Oracle)
Quoted 46 lines · Hao Li
Quoted 20 lines · Harry Yoo (Oracle)
Quoted 19 lines · Hao Li
Quoted 2 lines · Harry Yoo (Oracle)
On Fri, Apr 10, 2026 at 07:16:57PM +0800, Hao Li wrote:
Where do you think the improvement comes from? (hopefully w/ some data)Yes, this is necessary.
Quoted 3 lines
e.g.:
1. the benefit is from largely or partly fromreduced contention on n->list_lock.Before this patch is applied, the mmap benchmark shows the following hot path:
- 7.85%
native_queued_spin_lock_slowpath
-7.85%_raw_spin_lock_irqsave
- 3.69%__slab_free
+ 1.84%__refill_objects_node
+ 1.77%__kmem_cache_free_bulk
+ 3.27%__refill_objects_nodeQuoted 4 lines
Quoted 2 lines
With the patch applied, the
__refill_objects_node->__slab_freehotspot goes away, and thenative_queued_spin_lock_slowpathdrops to roughly 3.5%.Sounds like returning slabs back indeed increases contention on slowpath.
Indeed!
Quoted 5 lines
Quoted 5 lines
The
remaining lock contention is mostly between__refill_objects_node->add_partialand__kmem_cache_free_bulk->__slab_free.Quoted 1 line
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
2. this change reduces # of alloc slowpath at the cost of increased of free slowpath hits, but that's better because the slowpath frees are mostly lockless.
The alloc slowpath remains at 0 both w/ or w/o the patch, whereas the
(assuming you used
SLUB_STATSfor this)Yes, I enable it.
Quoted 2 lines
That's weird, I think we should check
SHEAF_REFILLinstead ofALLOC_SLOWPATH.Quoted 8 lines
Yes, I will compare each metrics for later testing. Maybe we can see more clues.
Quoted 3 lines
Quoted 1 line
free slowpath increases by 2x after applying the patch.
from which cache was this stat collected?
It's for
/sys/kernel/slab/maple_node/Ack. And you also mentioned (off-list) that
kmem_cache_refill_sheaf()is not on the profile. That's good to know.Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
Quoted 4 lines
3. the alloc/free pattern of the workload is benefiting from spilling objects to the CPU's sheaves.
or something else?
The 2-5% throughput improvement does seem to come with some trade-offs. The main one is that leftover objects get hidden in the percpu sheaves now, which reduces the objects on the node partial list and thus indirectly increases slab alloc/free frequency to about 4x of the baseline.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
This is a drawback of the current approach. :/
Sounds like
s->min_partialis too small now that we cache more objects per CPU.Exactly. for the mmap test case, the slab partial list keeps thrashing. It makes me wonder whether SLUB might handle transient pressure better if empty slabs could be regulated with a "dynamic burst threshold"
Quoted 7 lines
Haha, we'll be constantly challenged to find balance between "sacrifice memory to make every benchmark happy" vs. "provide reasonable scalability in general but let users tune it themselves".
If we could implement a reasonably simple yet effective automatic tuning method, having one in the kernel would be nice (though of course having it userspace would be the best).
Yes, I'm gradually feeling that SLUB's flow is so tight and simple that doing a one-size-fits-all optimization is super hard.
It might be better to just export some parameters to userspace and let users tune them.
After all, introducing an auto-tuning mechanism into a core allocator like SLUB might make it as unpredictable and hard to control as memory reclaim. :P
Quoted 11 lines
Quoted 6 lines
Quoted 4 lines
/me wonders if increasing sheaf capacity would make more sense rather than optimizing slowpath (if it comes with increased memory usage anyway),
Yes, finding ways to avoid falling onto the slowpath is also very worthwhile.
Could you please take a look at how much changing 1) sheaf capacity and
2) nr of full/empty sheaves at the barn affects the performance of mmap / ublk performance?
Quoted 6 lines
I've been trying to reproduce the regression on my machine but haven't had much success so far :(
(I'll try to post the RFC patchset to allow changing those parameters at runtime in few weeks but if you're eager you could try experimenting by changing the code :D)
Sure thing. I'll run some tests and organize the data.
Quoted 78 lines
Quoted 29 lines
Quoted 29 lines
but then stares at his (yet) unfinished patch series...
Quoted 25 lines
I experimented with several alternative ideas, and the pattern seems fairly
consistent: as soon as leftover objects are hidden at the percpu level, slab alloc/free churn tends to go up.Quoted 8 lines
Quoted 6 lines
Signed-off-by Hao Li <[email protected]>---
This patch is an exploratory attempt to address the leftover objects and partial slab issues in the refill path, and it is marked as RFC to warmly welcome any feedback, suggestions, and discussion!
Yeah, let's discuss!
Sure! Thanks for the discussion!
Quoted 3 lines
By the way, have you also been considering having min-max capacity for sheaves? (that I think Vlastimil suggested somewhere)
Yes, I also tried it.
I experimented with using a manually chosen threshold to allow refill to leave the sheaf in a partially filled state. However, since concurrent frees are inherently unpredictable, this seems can only reduce the probability of generating leftover objects,
Quoted 5 lines
Quoted 3 lines
If concurrent frees are a problem we could probably grab
slab->freelistundern->list_lock(e.g. keep them at the end of the sheaf) and fill the sheaf outside the lock to avoid grabbing too many objects.Do you mean doing an on-list bulk allocation?
Just brainstorming... it's quite messy :)
something like
__refill_objects_node(s, p, gfp, min, max, n, allow_spin){// in practice we don't know how many slabs we'll grab. // so probably keep them somewhere e.g.) the end of `p` array? void *freelists[min]; nr_freelists = 0; nr_objs = 0; spin_lock_irqsave(); for each slab in n->partial { freelist = slab->freelist; do { [...] old.freelist = slab->freelist; [...] } while (!__slab_update_freelist(...)); freelists[nr_freelists++] = old.freelist; nr_objs += (old.objects - old.inuse); if (!new.inuse) remove_partial(); if (nr_objs >= min) break; } spin_unlock_irqrestore(); i = 0; j = 0; while (i < nr_freelists) { freelist = freelists[i++]; while (freelist != NULL) { if (j == max) { // free remaining objects } next = get_freepointer(s, freelist); p[j++] = freelist; freelist = next; } } }
Quoted 1 line
This way, we know how many objects we grabbed but yeah it's tricky.
Thanks for this brainstorming.
If we do an atomic operation like __slab_update_freelist under the lock, I'm worried it might prolong the critical section. But testing is the best way to know for sure. I'm quite curious, so it's definitely worth a try.
Quoted 12 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
while at the same time affecting alloc-side throughput.
Shouldn't we set sheaf's min capacity as the same as
s->sheaf_capacityand allow higher max capcity to avoid this?I'm not sure I fully understand this. since the array size is fixed, how would we allow more entries to be filled?
I don't really want to speak on behalf of Vlastimil but I was imagining something like:
Quoted 6 lines
before:
sheaf->capacity(32, min = max);
after:sheaf->capacity(48 or 64, max),sheaf->threshold(32, min)so that sheaf refill will succeed if at least ->threshold objects are filled, but the threshold better not be smaller than 32 (the previous
sheaf->capacity)?
I feel that simply increasing the sheaf capacity should generally improve overall performance, although we'd likely see more slab alloc/free churn compared to the baseline.
One thing I'm wondering about is how we determine if an optimization is truly worth doing.
Micro-optimizations like this rarely have purely positive effects; they often come with fluctuations or regressions in other metrics-such as more frequent slab allocations and frees, which adds pressure to the buddy system. So this kind of ties our hands a bit :/
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 6 lines
In my testing, the results were not very encouraging: it seems hard to observe improvement, and in most cases it ended up causing a performance regression.
my impression is that it could be difficult to prevent leftovers proactively.
It may be easier to deal with them after they appear.
Either way doesn't work if the slab order is too high...
IIRC using higher slab order used to have some benefit but now that we have sheaves, it probably doesn't make sense anymore to have
oo_objects(s->oo)>s->sheaf_capacity?
Quoted 6 lines
Quoted 2 lines
Do you mean considering making the capacity of each sheaf larger than
oo_objects?I mean the other way around.
calculate_order()tends to increase slab order with higher number of CPUs (by setting highermin_objects), but is it still worth havingoo_objectshigher than the sheaf capacity?
Sorry, I just got confused for a second. Why is it a good thing for oo_objects to be larger than the sheaf capacity? I didn't quite catch the logic behind that...
Signature · 2 lines
Thanks, Hao
Harry Yoo (Oracle)
maintainerQuoted 22 lines · Hao Li
Quoted 21 lines · Harry Yoo (Oracle)
Quoted 7 lines · Hao Li
Quoted 6 lines · Harry Yoo (Oracle)
Quoted 5 lines · Hao Li
The 2-5% throughput improvement does seem to come with some trade-offs. The main one is that leftover objects get hidden in the percpu sheaves now, which reduces the objects on the node partial list and thus indirectly increases slab alloc/free frequency to about 4x of the baseline.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
This is a drawback of the current approach. :/
Sounds like
s->min_partialis too small now that we cache more objects per CPU.Exactly. for the mmap test case, the slab partial list keeps thrashing. It makes me wonder whether SLUB might handle transient pressure better if empty slabs could be regulated with a "dynamic burst threshold"
Haha, we'll be constantly challenged to find balance between "sacrifice memory to make every benchmark happy" vs. "provide reasonable scalability in general but let users tune it themselves".
Quoted 6 lines
Quoted 3 lines
If we could implement a reasonably simple yet effective automatic tuning method, having one in the kernel would be nice (though of course having it userspace would be the best).
Yes, I'm gradually feeling that SLUB's flow is so tight and simple that doing a one-size-fits-all optimization is super hard.
Agreed.
Quoted 2 lines
It might be better to just export some parameters to userspace and let users tune them.
Agreed.
However, people often don't try tuning and report
"regressions"... *ahem* ;)
Quoted 2 lines
After all, introducing an auto-tuning mechanism into a core allocator like SLUB might make it as unpredictable and hard to control as memory reclaim. :P
:-)
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
/me wonders if increasing sheaf capacity would make more sense rather than optimizing slowpath (if it comes with increased memory usage anyway),
Yes, finding ways to avoid falling onto the slowpath is also very worthwhile.
Could you please take a look at how much changing 1) sheaf capacity and
2) nr of full/empty sheaves at the barn affects the performance of mmap / ublk performance?
FWIW, we have two more parameters in SLUB
(yes, we have too many parameters):
3) s->remote_node_defrag_ratio and 4) s->min_partial.
The default s->remote_node_defrag_ratio value is configured to favor smaller memory usage over NUMA locality (which, in some cases, is not ideal).
Higher s->min_partial makes SLUB cache more slabs in n->partial.
Quoted 8 lines
Quoted 6 lines
I've been trying to reproduce the regression on my machine but haven't had much success so far :(
(I'll try to post the RFC patchset to allow changing those parameters at runtime in few weeks but if you're eager you could try experimenting by changing the code :D)
Sure thing. I'll run some tests and organize the data.
Thanks a lot for taking a look ;-)
Quoted 75 lines
Quoted 75 lines
Quoted 28 lines
Quoted 28 lines
but then stares at his (yet) unfinished patch series...
Quoted 25 lines
I experimented with several alternative ideas, and the pattern seems fairly
consistent: as soon as leftover objects are hidden at the percpu level, slab alloc/free churn tends to go up.Quoted 8 lines
Quoted 6 lines
Signed-off-by Hao Li <[email protected]>---
This patch is an exploratory attempt to address the leftover objects and partial slab issues in the refill path, and it is marked as RFC to warmly welcome any feedback, suggestions, and discussion!
Yeah, let's discuss!
Sure! Thanks for the discussion!
Quoted 3 lines
By the way, have you also been considering having min-max capacity for sheaves? (that I think Vlastimil suggested somewhere)
Yes, I also tried it.
I experimented with using a manually chosen threshold to allow refill to leave the sheaf in a partially filled state. However, since concurrent frees are inherently unpredictable, this seems can only reduce the probability of generating leftover objects,
Quoted 5 lines
Quoted 3 lines
If concurrent frees are a problem we could probably grab
slab->freelistundern->list_lock(e.g. keep them at the end of the sheaf) and fill the sheaf outside the lock to avoid grabbing too many objects.Do you mean doing an on-list bulk allocation?
Just brainstorming... it's quite messy :)
something like
__refill_objects_node(s, p, gfp, min, max, n, allow_spin){// in practice we don't know how many slabs we'll grab. // so probably keep them somewhere e.g.) the end of `p` array? void *freelists[min]; nr_freelists = 0; nr_objs = 0;Quoted 1 line
Quoted 1 line
spin_lock_irqsave();for each slab in n->partial { freelist = slab->freelist; do { [...] old.freelist = slab->freelist; [...] > > } while (!__slab_update_freelist(...)); freelists[nr_freelists++] = old.freelist; nr_objs += (old.objects - old.inuse); if (!new.inuse) remove_partial(); if (nr_objs >= min) break; } spin_unlock_irqrestore(); i = 0; j = 0; while (i < nr_freelists) { freelist = freelists[i++]; while (freelist != NULL) { if (j == max) { // free remaining objects } next = get_freepointer(s, freelist); p[j++] = freelist; freelist = next; } } }
Quoted 5 lines
Quoted 1 line
This way, we know how many objects we grabbed but yeah it's tricky.
Thanks for this brainstorming.
If we do an atomic operation like__slab_update_freelistunder the lock, I'm worried it might prolong the critical section.
Fun fact: SLUB had been doing __slab_update_freelist() under the lock for a long time until very recently. (If interested, see commit 8cd3fa428b5 ("slub: Delay freezing of partial slabs"))
Quoted 1 line
But testing is the best way to know for sure.
Yeah, that really depends on which is more problematic:
doing cmpxchg128 under n->list_lock vs. grabbing too many objects (due to concurrent frees) so that we return slab back to the list.
But without allowing partial refills, we'll end up returning slab back to the list anyway.
Quoted 1 line
I'm quite curious, so it's definitely worth a try.
Again, appreciate spending time on investigating the performance.
Quoted 14 lines
Quoted 14 lines
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
while at the same time affecting alloc-side throughput.
Shouldn't we set sheaf's min capacity as the same as
s->sheaf_capacityand allow higher max capcity to avoid this?I'm not sure I fully understand this. since the array size is fixed, how would we allow more entries to be filled?
I don't really want to speak on behalf of Vlastimil but I was imagining something like:
before:
sheaf->capacity(32, min = max);
after:sheaf->capacity(48 or 64, max),sheaf->threshold(32, min)
Quoted 6 lines
Quoted 3 lines
so that sheaf refill will succeed if at least ->threshold objects are filled, but the threshold better not be smaller than 32 (the previous
sheaf->capacity)?I feel that simply increasing the sheaf capacity should generally improve overall performance,
Well, if simply increasing the sheaf capacity entirely solves the problem, this isn't really worth it.
To reiterate the question: To allow partial refill to work while avoiding regressions, we need to make sheaves larger than before. That inevitably consumes (although we don't always fully refill them) more memory. Is this trade-off worth it?
If we can avoid the contention by tuning parameters without such a micro-optimization, I think it's worth it only if tuning requires much more memory to reach the same performance.
Or if there are corner cases (e.g., perhaps cross-CPU frees or remote frees? dunno) where tuning does not resolve this issue, I'd say it's worth it.
Quoted 2 lines
although we'd likely see more slab alloc/free churn compared to the baseline.
I think it's okay to allocate/free slabs more frequently, as long as that doesn't hurt performance (luckily, today, buddy's PCP caches high order pages, unlike a decade ago).
But if higher alloc/free churn means higher slab memory usage, that's a concern.
Quoted 2 lines
One thing I'm wondering about is how we determine if an optimization is truly worth doing.
We do an optimization when we can convince people that it's worth it (or at least a reasonable tradeoff).
Quoted 22 lines
Micro-optimizations like this rarely have purely positive effects; they often come with fluctuations or regressions in other metrics-such as more frequent slab allocations and frees, which adds pressure to the buddy system.
So this kind of ties our hands a bit :/
Quoted 16 lines
Quoted 15 lines
Quoted 4 lines
Quoted 4 lines
In my testing, the results were not very encouraging: it seems hard to observe improvement, and in most cases it ended up causing a performance regression.
Quoted 8 lines
Quoted 1 line
my impression is that it could be difficult to prevent leftovers proactively.
It may be easier to deal with them after they appear.
Either way doesn't work if the slab order is too high...
IIRC using higher slab order used to have some benefit but now that we have sheaves, it probably doesn't make sense anymore to have
oo_objects(s->oo)>s->sheaf_capacity?Do you mean considering making the capacity of each sheaf larger than
oo_objects?
Quoted 7 lines
Quoted 3 lines
I mean the other way around.
calculate_order()tends to increase slab order with higher number of CPUs (by setting highermin_objects), but is it still worth havingoo_objectshigher than the sheaf capacity?Sorry, I just got confused for a second. Why is it a good thing for
oo_objectsto be larger than the sheaf capacity? I didn't quite catch the logic behind that...
I think nobody intended that "It's a good thing for oo_objects() to be larger than the sheaf capacity": when the slab order calculation logic was written, sheaves didn't exist.
Having high order slabs used to have some benefits (before sheaves):
1. CPU slab could serve more objects (in fastpath).
2. Percpu partial slab list needed fewer slabs to cache the same number of objects, which reduced the contention on n->list_lock.
3. (please add if there's more)
I chatted about this off-list, and a few people mentioned that having high order slabs may still help reduce avoiding external fragmentation (at the cost of internal fragmentation).
[A bit irrelevant but fun SLUB history]
In the traditional SLUB, we didn't precisely track the number of objects cached per CPU. In extreme cases, the traditional SLUB could somehow tranlsate "cache 30 objects per CPU" to "cache 30 _slabs_ per CPU".
See how extreme it could be, compared to what we've been discussing here? ;)
At some point Vlastimil fixed that imprecise logic and instead assumed that all slabs are half full (which was still imprecise but better than before) (If interested, see b47291ef02b0b ("mm, slub: change percpu partial accounting from objects to pages"))
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerQuoted 4 lines · Harry Yoo (Oracle)
/me wonders if increasing sheaf capacity would make more sense rather than optimizing slowpath (if it comes with increased memory usage anyway), but then stares at his (yet) unfinished patch series...
Oh, take it easy, Harry. this is just an experimental patch discussion. Looking forward to your next patch! :P
Quoted 28 lines · Hao Li
When performing objects refill, we tend to optimistically assume that there will be more allocation requests coming next; this is the fundamental assumption behind this optimization.
When
__refill_objects_node()isolates a partial slab and satisfies a bulk allocation from its freelist, the slab can still have a small tail of free objects left over. Today those objects are freed back to the slab immediately.If the leftover tail is local and small enough to fit, keep it in the current CPU's sheaves instead. This avoids pushing those objects back through the
__slab_freeslowpath.Add a helper to obtain both the freelist and its free-object count, and then spill the remaining objects into a percpu sheaf when:
- the tail fits in a sheaf
- the slab is local to the current CPU
- the slab is not pfmemalloc
- the target sheaf has enough free spaceOtherwise keep the existing fallback and free the tail back to the slab.
Also add a
SHEAF_SPILLstat so the new path can be observed in SLUB stats.On the mmap2 case in the will-it-scale benchmark suite, this patch can improve performance by about 2~5%.
Quoted 7 lines
Signed-off-by Hao Li <[email protected]>---
This patch is an exploratory attempt to address the leftover objects and partial slab issues in the refill path, and it is marked as RFC to warmly welcome any feedback, suggestions, and discussion!
I was also looking at these regressions, but I went from a different direction, and ended up with 3 patches:
1. the regressions showed a lot of increase in the cache misses, which gave me the idea that a cache would help (and it seemed to help)
2. Allowing smaller refills (but potentially more frequent);
3. A cute (but with small impact) use of prefetch();
The numbers are here (the commentary from the bot are very hit or miss, so don't pay too much attention to them):
https://github.com/vcgomes/linux/commit/c898c39ee8def5252942281353eda6acdd83d4ea
I am re-running the tests against a more recent tree, but if you want to take a look:
https://github.com/vcgomes/linux/tree/mm-sheaves-regression-timerfd
Also, if you feel it's useful, I can send a RFC.
Quoted 181 lines
--- mm/slub.c | 107 ++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 88 insertions(+), 19 deletions(-)mm/slub.c +88
-19@@ -353,6 +353,7 @@ enum stat_item {SHEAF_REFILL ,/* Objects refilled to a sheaf */ SHEAF_ALLOC ,/* Allocation of an empty sheaf */ SHEAF_FREE ,/* Freeing of an empty sheaf */ +SHEAF_SPILL ,BARN_GET ,/* Got full sheaf from barn */ BARN_GET_FAIL ,/* Failed to get full sheaf from barn */ BARN_PUT, /* Put full sheaf to barn */@@ -4279,7 +4280,9 @@ static inline bool pfmemalloc_match(struct slab *slab, gfp_t gfpflags)* Assumes this is performed only for caches without debugging so we * don't need to worry about adding the slab to the full list. */ -static inline void *get_freelist_nofreeze (struct kmem_cache *s,struct slab *slab) +static inline void *__get_freelist_nofreeze (struct kmem_cache *s, +struct slab *slab,int *freecount, +const char *n) {struct freelist_counters old, new;@@ -4293,11 +4296,26 @@ static inline void *get_freelist_nofreeze(struct kmem_cache *s, struct slab *sla new.inuse = old.objects; - }while (!slab_update_freelist (s, slab, &old, &new,"get_freelist_nofreeze" )); + }while (!slab_update_freelist (s, slab, &old, &new, n)); + +if (freecount) + *freecount = old.objects - old.inuse;return old.freelist; } +static inline void *get_freelist_nofreeze (struct kmem_cache *s,struct slab *slab) +{ +return __get_freelist_nofreeze (s, slab,NULL ,"get_freelist_nofreeze" ); +} + +static inline void *get_freelist_and_freecount_nofreeze (struct kmem_cache *s, +struct slab *slab, +int *freecount) +{ +return __get_freelist_nofreeze (s, slab, freecount,"get_freelist_and_freecount_nofreeze" ); +} +/* * If the object has been wiped upon free, make sure it's fully initialized by * zeroing out freelist pointer. @@ -7028,10 +7046,15 @@ __refill_objects_node(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int mireturn 0 ;list_for_each_entry_safe (slab, slab2, &pc.slabs, slab_list) { +void *head; +void *tail; +struct slub_percpu_sheaves *pcs; +int freecount, local_node, i, cnt =0 ; +struct slab_sheaf *spill;list_del (&slab->slab_list); - object =get_freelist_nofreeze (s, slab); + object =get_freelist_and_freecount_nofreeze (s, slab, &freecount);while (object && refilled < max) { p[refilled] = object;@@ -7039,28 +7062,72 @@ __refill_objects_node(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int mimaybe_wipe_obj_freeptr (s, p[refilled]); refilled++; + freecount--; } +if (!freecount) { +if (refilled >= max) +break ; +continue ; + }/* -* Freelist had more objects than we can accommodate, we need to -* free them back. We can treat it like a detached freelist, just -* need to find the tail object. +* Freelist had more objects than we can accommodate, we first +* try to spill them into percpu sheaf. */ -if (unlikely (object)) { -void *head = object; -void *tail; -int cnt =0 ; - -do { - tail = object; - cnt++; - object =get_freepointer (s, object); - }while (object); -__slab_free (s, slab, head, tail, cnt, _RET_IP_); +if (freecount > s->sheaf_capacity) + goto skip_spill; +if (slab_test_pfmemalloc (slab)) + goto skip_spill; + +if (!local_trylock (&s->cpu_sheaves->lock)) + goto skip_spill; + + local_node =numa_mem_id (); +if (slab_nid (slab) != local_node) { +local_unlock (&s->cpu_sheaves->lock); + goto skip_spill; } -if (refilled >= max) -break ; + pcs =this_cpu_ptr (s->cpu_sheaves); +if (pcs->spare && + (freecount <= (s->sheaf_capacity - pcs->spare->size))) + spill = pcs->spare; +else if (freecount <= (s->sheaf_capacity - pcs->main->size)) + spill = pcs->main; +else { +local_unlock (&s->cpu_sheaves->lock); + goto skip_spill; + } + +if (freecount > (s->sheaf_capacity - spill->size)) { +local_unlock (&s->cpu_sheaves->lock); + goto skip_spill; + } + +for (i =0 ; i < freecount; i++) { + spill->objects[spill->size] = object; + object =get_freepointer (s, object); +maybe_wipe_obj_freeptr (s, spill->objects[spill->size]); + spill->size++; + } + +local_unlock (&s->cpu_sheaves->lock); +stat (s,SHEAF_SPILL ); +break ; +skip_spill: +/* +* Freelist had more objects than we can accommodate or spill, +* we need to free them back. We can treat it like a detached freelist, +* just need to find the tail object. +*/ + head = object; +do { + tail = object; + cnt++; + object =get_freepointer (s, object); + }while (object); +__slab_free (s, slab, head, tail, cnt, _RET_IP_); +break ; }if (unlikely (!list_empty (&pc.slabs))) {@@ -9247,6 +9314,7 @@ STAT_ATTR(SHEAF_FLUSH, sheaf_flush);STAT_ATTR (SHEAF_REFILL , sheaf_refill);STAT_ATTR (SHEAF_ALLOC , sheaf_alloc);STAT_ATTR (SHEAF_FREE , sheaf_free); +STAT_ATTR (SHEAF_SPILL , sheaf_spill);STAT_ATTR (BARN_GET , barn_get);STAT_ATTR (BARN_GET_FAIL , barn_get_fail);STAT_ATTR (BARN_PUT , barn_put);@@ -9335,6 +9403,7 @@ static struct attribute *slab_attrs[] = { &sheaf_refill_attr.attr, &sheaf_alloc_attr.attr, &sheaf_free_attr.attr, + &sheaf_spill_attr.attr, &barn_get_attr.attr, &barn_get_fail_attr.attr, &barn_put_attr.attr,Signature · 1 line
2.50.1
Signature · 1 line
Vinicius
Hao Li
reviewerQuoted 41 lines · Vinicius Costa Gomes
Quoted 28 lines · Hao Li
When performing objects refill, we tend to optimistically assume that there will be more allocation requests coming next; this is the fundamental assumption behind this optimization.
When
__refill_objects_node()isolates a partial slab and satisfies a bulk allocation from its freelist, the slab can still have a small tail of free objects left over. Today those objects are freed back to the slab immediately.If the leftover tail is local and small enough to fit, keep it in the current CPU's sheaves instead. This avoids pushing those objects back through the
__slab_freeslowpath.Add a helper to obtain both the freelist and its free-object count, and then spill the remaining objects into a percpu sheaf when:
- the tail fits in a sheaf
- the slab is local to the current CPU
- the slab is not pfmemalloc
- the target sheaf has enough free spaceOtherwise keep the existing fallback and free the tail back to the slab.
Also add a
SHEAF_SPILLstat so the new path can be observed in SLUB stats.On the mmap2 case in the will-it-scale benchmark suite, this patch can improve performance by about 2~5%.
Quoted 7 lines
Signed-off-by Hao Li <[email protected]>---
This patch is an exploratory attempt to address the leftover objects and partial slab issues in the refill path, and it is marked as RFC to warmly welcome any feedback, suggestions, and discussion!
I was also looking at these regressions, but I went from a different direction, and ended up with 3 patches:
Quoted 7 lines
1. the regressions showed a lot of increase in the cache misses, which gave me the idea that a cache would help (and it seemed to help)
2. Allowing smaller refills (but potentially more frequent);
3. A cute (but with small impact) use of
prefetch();
Great!
Thanks for sharing those infos!
Quoted 11 lines
The numbers are here (the commentary from the bot are very hit or miss, so don't pay too much attention to them):
https://github.com/vcgomes/linux/commit/c898c39ee8def5252942281353eda6acdd83d4ea
I am re-running the tests against a more recent tree, but if you want to take a look:
https://github.com/vcgomes/linux/tree/mm-sheaves-regression-timerfd
Also, if you feel it's useful, I can send a RFC.
I also tried stashing leftover objects into the PCS before, but at the time I observed that this could quickly drain the node partial list, which then led to slab alloc/free churn, and the end result was a performance regression. So I gave up this direction :/
I took a quick look at the code and performance report in your GitHub repo, and the performance gains you showed there are really interesting to me! I'm going to try testing it on my own machine as well.
Also, the idea of a warm_slab looks very interesting. I had previously thought about something along similar lines, except that I was considering stashing leftover untouched slabs at the per-CPU level. But that would effectively bring back the old stale CPU partial list in another form, so I ended up dropping that direction. But you direction is really interesting!
I'd be very happy to see a new approach come out of this. If it's convenient for you, I'd love to see an RFC. Thanks!
Signature · 2 lines
Thanks, Hao
Vlastimil Babka (SUSE)
maintainerQuoted 27 lines · Hao Li
On Wed, Apr 15, 2026 at 01:55:54PM -0700, Vinicius Costa Gomes wrote:
Great!
Thanks for sharing those infos!Quoted 11 lines
The numbers are here (the commentary from the bot are very hit or miss, so don't pay too much attention to them):
https://github.com/vcgomes/linux/commit/c898c39ee8def5252942281353eda6acdd83d4ea
I am re-running the tests against a more recent tree, but if you want to take a look:
https://github.com/vcgomes/linux/tree/mm-sheaves-regression-timerfd
Also, if you feel it's useful, I can send a RFC.
I also tried stashing leftover objects into the PCS before, but at the time I observed that this could quickly drain the node partial list, which then led to slab alloc/free churn, and the end result was a performance regression. So I gave up this direction :/
I took a quick look at the code and performance report in your GitHub repo, and the performance gains you showed there are really interesting to me! I'm going to try testing it on my own machine as well.
Quoted 5 lines
Also, the idea of a
warm_slablooks very interesting. I had previously thought about something along similar lines, except that I was considering stashing leftover untouched slabs at the per-CPU level. But that would effectively bring back the old stale CPU partial list in another form, so I ended up dropping that direction. But you direction is really interesting!
I wonder about the conditions when the warm_slab freelist is actually warm. If we have leftover objects on the slab's freelist after a refill, we didn't walk that part of the freelist to make it cache-hot. So why is draining that stashed slab beneficial? Is the benefit rather because of caching more objects percpu again, and avoiding the partial list and list_lock?
The freelist could be warm in the case the slab was just allocated, but that should be a minority cases compared to taken from partial list?
Quoted 3 lines
I'd be very happy to see a new approach come out of this. If it's convenient for you, I'd love to see an RFC. Thanks!
Harry Yoo (Oracle)
maintainerQuoted 39 lines · Hao Li
Quoted 38 lines · Vinicius Costa Gomes
Quoted 35 lines · Hao Li
When performing objects refill, we tend to optimistically assume that there will be more allocation requests coming next; this is the fundamental assumption behind this optimization.
When
__refill_objects_node()isolates a partial slab and satisfies a bulk allocation from its freelist, the slab can still have a small tail of free objects left over. Today those objects are freed back to the slab immediately.If the leftover tail is local and small enough to fit, keep it in the current CPU's sheaves instead. This avoids pushing those objects back through the
__slab_freeslowpath.Add a helper to obtain both the freelist and its free-object count, and then spill the remaining objects into a percpu sheaf when:
- the tail fits in a sheaf
- the slab is local to the current CPU
- the slab is not pfmemalloc
- the target sheaf has enough free spaceOtherwise keep the existing fallback and free the tail back to the slab.
Also add a
SHEAF_SPILLstat so the new path can be observed in SLUB stats.On the mmap2 case in the will-it-scale benchmark suite, this patch can improve performance by about 2~5%.
Signed-off-by Hao Li <[email protected]>---
This patch is an exploratory attempt to address the leftover objects and partial slab issues in the refill path, and it is marked as RFC to warmly welcome any feedback, suggestions, and discussion!
Quoted 5 lines
Quoted 5 lines
I was also looking at these regressions, but I went from a different direction, and ended up with 3 patches:
1. the regressions showed a lot of increase in the cache misses, which gave me the idea that a cache would help (and it seemed to help)
I really appreciate looking into the performance change but think we should first try fixing existing corner cases and/or try tuning existing parameters (s->sheaf_capacity, MAX_{FULL,EMPTY}_SHEAVES, s->min_partial, and s->remote_node_defrag_ratio) before making such design changes.
Exploring a design change too soon without fully exploring the limitation of current design isn't worth the effort.
in the first patch description:
| When the sheaf allocator needs to refill from the node partial list, it | calls __refill_objects_node() which walks the freelist of a cold slab | page — one that has not been in any CPU's cache since it was last freed. | On NUMA systems with many concurrent threads, the majority of these walks | hit remote DRAM, causing a significant increase in LLC misses.
IIUC you're arguing that iterating over slab->freelist just to return the slab back to the list unnecessarily results in higher cache footprint, right? (and even worse those slabs are from remote nodes) (unlike Hao who argued it's more of a n->list_lock contention thing)
in __refill_objects_node():
| __refill_objects_node(struct kmem_cache *s, void **p, gfp_t gfp, unsigned int min,
| unsigned int max, struct kmem_cache_node *n,
| bool allow_spin)
| {
| struct partial_bulk_context pc;
| struct slab *slab, *slab2;
| unsigned int refilled = 0;
| unsigned long flags;
| void *object;
|
| pc.flags = gfp;
| pc.min_objects = min;
| pc.max_objects = max;
|
| if (!get_partial_node_bulk(s, n, &pc, allow_spin))
| return 0;
|
| list_for_each_entry_safe(slab, slab2, &pc.slabs, slab_list) {
|
| list_del(&slab->slab_list);
|
| object = get_freelist_nofreeze(s, slab);
|
| while (object && refilled < max) {
| p[refilled] = object;
| object = get_freepointer(s, object);
| maybe_wipe_obj_freeptr(s, p[refilled]);
|
| refilled++;
| }
|
| /*
| * Freelist had more objects than we can accommodate, we need to
| * free them back. We can treat it like a detached freelist, just
| * need to find the tail object.
| */
| if (unlikely(object)) {
| void *head = object;
| void *tail;
| int cnt = 0;
|
| do {
| tail = object;
| cnt++;
| object = get_freepointer(s, object);
| } while (object);So here we make the slab "warm" although we're not going to use it, just to get the tail object.
As Vlastimil suggested off-list, we could probably assume that nobody is has freed objects to the slab and try __slab_update_freelist() and avoid iterating over the freelist? (a kind of blind Compare-And-Swap)
and then fall back if that fails.
| __slab_free(s, slab, head, tail, cnt, _RET_IP_);
| }
Back to the description:
| Add a per-CPU warm slab stash: a single (slab, freelist-head) pair stored | in struct slub_percpu_sheaves.
Oh, calling it "warm slab" is very misleading. Warming the slab in __refill_objects_node() when it has more objects than (sheaf->capacity - sheaf->size) is current behavior
| When __refill_objects_node() drains a slab | from the partial list but has excess objects (more than it needs for the | current refill), it stashes the remainder instead of returning them to | the partial list. On the next refill, drain_warm_slab() serves the | stashed objects first, skipping the cold partial-list walk entirely.
and your patch changes that. It's not warm anymore.
Quoted 19 lines
Quoted 3 lines
2. Allowing smaller refills (but potentially more frequent);
3. A cute (but with small impact) use of
prefetch();Great!
Thanks for sharing those infos!Quoted 11 lines
The numbers are here (the commentary from the bot are very hit or miss, so don't pay too much attention to them):
https://github.com/vcgomes/linux/commit/c898c39ee8def5252942281353eda6acdd83d4ea
I am re-running the tests against a more recent tree, but if you want to take a look:
https://github.com/vcgomes/linux/tree/mm-sheaves-regression-timerfd
Also, if you feel it's useful, I can send a RFC.
Quoted 8 lines
I also tried stashing leftover objects into the PCS before, but at the time I observed that this could quickly drain the node partial list, which then led to slab alloc/free churn, and the end result was a performance regression. So I gave up this direction :/
I took a quick look at the code and performance report in your GitHub repo, and the performance gains you showed there are really interesting to me! I'm going to try testing it on my own machine as well.
Comparing the patch 1 with Hao's patch (spilling objects)... it doesn't touch those leftover objects (reduced cache footprint) and also does not spill them into sheaves (hit the free slowpath less frequently).
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerQuoted 13 lines · Vinicius Costa Gomes
I was also looking at these regressions, but I went from a different direction, and ended up with 3 patches:
1. the regressions showed a lot of increase in the cache misses, which gave me the idea that a cache would help (and it seemed to help)
2. Allowing smaller refills (but potentially more frequent);
3. A cute (but with small impact) use of
prefetch();The numbers are here (the commentary from the bot are very hit or miss, so don't pay too much attention to them):
Quoted 9 lines
https://github.com/vcgomes/linux/commit/c898c39ee8def5252942281353eda6acdd83d4ea
I am re-running the tests against a more recent tree, but if you want to take a look:
https://github.com/vcgomes/linux/tree/mm-sheaves-regression-timerfd
Also, if you feel it's useful, I can send a RFC.
Hi Vinicius,
I tested the three patches in your GitHub repository. Under a 96-process stress workload, mmap2 achieved about a 3% performance improvement.
For the slub stats, I observed some differences. Here are the results:
(baseline vs 3 patches)alloc_fastpath +4.6%alloc_slowpath +0% (and no slowpath in both test)free_fastpath +0%free_slowpath +189%alloc_slab +247%free_slab +247%barn_get +8%barn_put +8%barn_get_fail +16%barn_put_fail +0%free_add_partial -37%free_remove_partial +247%sheaf_refill +3.88%
I notice one thing that seems consistent with my approach is the churn in alloc_slab and free_slab. My impression is that this may be a common issue with solutions designed at the per-CPU level...
Signature · 2 lines
Thanks, Hao