mm/slab: allow sheaf refill if blocking is not allowed
Subsystems (2)
Vlastimil Babka (SUSE)
maintainerPATCH 1/1 mm/slab: allow sheaf refill if blocking is not allowed
Ming Lei reported [1] a regression in the ublk null target benchmark due to sheaves. The profile shows that the alloc_from_pcs() fastpath fails and allocations fall back to ___slab_alloc(). It also shows the allocations happen through mempool_alloc().
The strategy of mempool_alloc() is to call the underlying allocator (here slab) without __GFP_DIRECT_RECLAIM first. This does not play well with __pcs_replace_empty_main() checking for gfpflags_allow_blocking() to decide if it should refill an empty sheaf or fallback to the slowpath, so we end up falling back.
We could change the mempool strategy but there might be other paths doing the same ting. So instead allow sheaf refill when blocking is not allowed, changing the condition to gfpflags_allow_spinning(). The original condition was unnecessarily restrictive.
Note this doesn't fully resolve the regression [1] as another component of that are memoryless nodes, which is to be addressed separately.
--- mm/slub.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)
mm/slub.c +9 -12
@@ -4567,7 +4567,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,struct slab_sheaf *empty =NULL ;struct slab_sheaf *full;struct node_barn *barn; -bool can_alloc; +bool allow_spin;lockdep_assert_held (this_cpu_ptr (&s->cpu_sheaves->lock));
@@ -4588,8 +4588,9 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,return NULL ; } - full =barn_replace_empty_sheaf (barn, pcs->main, -gfpflags_allow_spinning (gfp)); + allow_spin =gfpflags_allow_spinning (gfp); + + full =barn_replace_empty_sheaf (barn, pcs->main, allow_spin);if (full) {stat (s,BARN_GET );
@@ -4599,9 +4600,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,stat (s,BARN_GET_FAIL ); - can_alloc =gfpflags_allow_blocking (gfp); - -if (can_alloc) { +if (allow_spin) {if (pcs->spare) { empty = pcs->spare; pcs->spare =NULL ;
@@ -4612,7 +4611,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,local_unlock (&s->cpu_sheaves->lock); -if (!can_alloc) +if (!allow_spin)return NULL ;if (empty) {
@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,if (!full)return NULL ; -/* -* we can reach here only when gfpflags_allow_blocking -* so this must not be an irq -*/ -local_lock (&s->cpu_sheaves->lock); +if (!local_trylock (&s->cpu_sheaves->lock)) + goto barn_put; pcs =this_cpu_ptr (s->cpu_sheaves);/*
@@ -4667,6 +4663,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,return pcs; } +barn_put:barn_put_full_sheaf (barn, full);stat (s,BARN_PUT );
Signature · 1 line
2.53.0
Quoted 32 lines · Vlastimil Babka (SUSE)
Ming Lei reported [1] a regression in the ublk null target benchmark due to sheaves. The profile shows that the
alloc_from_pcs()fastpath fails and allocations fall back to___slab_alloc(). It also shows the allocations happen throughmempool_alloc().The strategy of
mempool_alloc()is to call the underlying allocator (here slab) without__GFP_DIRECT_RECLAIMfirst. This does not play well with__pcs_replace_empty_main()checking forgfpflags_allow_blocking()to decide if it should refill an empty sheaf or fallback to the slowpath, so we end up falling back.We could change the mempool strategy but there might be other paths doing the same ting. So instead allow sheaf refill when blocking is not allowed, changing the condition to
gfpflags_allow_spinning(). The original condition was unnecessarily restrictive.Note this doesn't fully resolve the regression [1] as another component of that are memoryless nodes, which is to be addressed separately.
Reported-by Ming Lei <[email protected]>Fixes e47c897a2949 ("slab: add sheaves to most caches")Signed-off-by Vlastimil Babka (SUSE) <[email protected]>--- mm/slub.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)mm/slub.c +0
-0
mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635
Quoted 10 lines
mm/slub.c +2
-5@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,if (!full)return NULL ; -/* -* we can reach here only when gfpflags_allow_blocking -* so this must not be an irq -*/ -local_lock (&s->cpu_sheaves->lock); +if (!local_trylock (&s->cpu_sheaves->lock)) + goto barn_put;
My AI buddy says (don't worry, I filtered it):
| When local_trylock() fails above, the function jumps to barn_put and returns
| pcs without holding the lock. This appears to violate the function's contract
| documented in the comment at the beginning of __pcs_replace_empty_main():
|
| "If not successful, returns NULL and the local lock unlocked."
|
| The caller in alloc_from_pcs() checks for NULL to detect failure:
|
| if (unlikely(pcs->main->size == 0)) {
| pcs = __pcs_replace_empty_main(s, pcs, gfp);
| if (unlikely(!pcs))
| return NULL;
| }
|
| If the trylock fails and pcs (non-NULL) is returned, the caller proceeds
| without realizing the lock was never re-acquired. This leads to accessing
| pcs->main without the lock and later trying to unlock a lock that isn't held.And the analysis sounds correct to me.
perhaps it should be:
if (!local_trylock(&s->cpu_sheaves->lock)) {
pcs = NULL;
goto barn_put;
}Quoted 10 lines
mm/slub.c +1
-0pcs =this_cpu_ptr (s->cpu_sheaves);/* @@ -4667,6 +4663,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,return pcs; } +barn_put:barn_put_full_sheaf (barn, full);stat (s,BARN_PUT );
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 63 lines · Harry Yoo
Quoted 32 lines · Vlastimil Babka (SUSE)
Ming Lei reported [1] a regression in the ublk null target benchmark due to sheaves. The profile shows that the
alloc_from_pcs()fastpath fails and allocations fall back to___slab_alloc(). It also shows the allocations happen throughmempool_alloc().The strategy of
mempool_alloc()is to call the underlying allocator (here slab) without__GFP_DIRECT_RECLAIMfirst. This does not play well with__pcs_replace_empty_main()checking forgfpflags_allow_blocking()to decide if it should refill an empty sheaf or fallback to the slowpath, so we end up falling back.We could change the mempool strategy but there might be other paths doing the same ting. So instead allow sheaf refill when blocking is not allowed, changing the condition to
gfpflags_allow_spinning(). The original condition was unnecessarily restrictive.Note this doesn't fully resolve the regression [1] as another component of that are memoryless nodes, which is to be addressed separately.
Reported-by Ming Lei <[email protected]>Fixes e47c897a2949 ("slab: add sheaves to most caches")Signed-off-by Vlastimil Babka (SUSE) <[email protected]>--- mm/slub.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)mm/slub.c +0
-0mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635
Quoted 10 lines
mm/slub.c +2
-5@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,if (!full)return NULL ; -/* -* we can reach here only when gfpflags_allow_blocking -* so this must not be an irq -*/ -local_lock (&s->cpu_sheaves->lock); +if (!local_trylock (&s->cpu_sheaves->lock)) + goto barn_put;My AI buddy says (don't worry, I filtered it):
| When local_trylock() fails above, the function jumps to barn_put and returns | pcs without holding the lock. This appears to violate the function's contract | documented in the comment at the beginning of __pcs_replace_empty_main(): | | "If not successful, returns NULL and the local lock unlocked." | | The caller in alloc_from_pcs() checks for NULL to detect failure: | | if (unlikely(pcs->main->size == 0)) { | pcs = __pcs_replace_empty_main(s, pcs, gfp); | if (unlikely(!pcs)) | return NULL; | } | | If the trylock fails and pcs (non-NULL) is returned, the caller proceeds | without realizing the lock was never re-acquired. This leads to accessing | pcs->main without the lock and later trying to unlock a lock that isn't held.
Quoted 8 lines
And the analysis sounds correct to me.
perhaps it should be:
if (!local_trylock(&s->cpu_sheaves->lock)) { pcs = NULL; goto barn_put; }
Thanks a lot Harry. In fact I realized this mistake after initially sending the patch to Ming in a reply, and fixed it locally (same as you suggest).
Or so I thought, because the fix got apparently lost.
So I'll do that now in slab/for-next-fixes
Or actually I think a more robust way is to set pcs = NULL after the unlock, unconditionally, so I'll do that.
Quoted 11 lines
Quoted 10 lines
mm/slub.c +1
-0pcs =this_cpu_ptr (s->cpu_sheaves);/* @@ -4667,6 +4663,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,return pcs; } +barn_put:barn_put_full_sheaf (barn, full);stat (s,BARN_PUT );
Quoted 73 lines · Vlastimil Babka
Quoted 63 lines · Harry Yoo
Quoted 32 lines · Vlastimil Babka (SUSE)
Ming Lei reported [1] a regression in the ublk null target benchmark due to sheaves. The profile shows that the
alloc_from_pcs()fastpath fails and allocations fall back to___slab_alloc(). It also shows the allocations happen throughmempool_alloc().The strategy of
mempool_alloc()is to call the underlying allocator (here slab) without__GFP_DIRECT_RECLAIMfirst. This does not play well with__pcs_replace_empty_main()checking forgfpflags_allow_blocking()to decide if it should refill an empty sheaf or fallback to the slowpath, so we end up falling back.We could change the mempool strategy but there might be other paths doing the same ting. So instead allow sheaf refill when blocking is not allowed, changing the condition to
gfpflags_allow_spinning(). The original condition was unnecessarily restrictive.Note this doesn't fully resolve the regression [1] as another component of that are memoryless nodes, which is to be addressed separately.
Reported-by Ming Lei <[email protected]>Fixes e47c897a2949 ("slab: add sheaves to most caches")Signed-off-by Vlastimil Babka (SUSE) <[email protected]>--- mm/slub.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)mm/slub.c +0
-0mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635
Quoted 10 lines
mm/slub.c +2
-5@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,if (!full)return NULL ; -/* -* we can reach here only when gfpflags_allow_blocking -* so this must not be an irq -*/ -local_lock (&s->cpu_sheaves->lock); +if (!local_trylock (&s->cpu_sheaves->lock)) + goto barn_put;My AI buddy says (don't worry, I filtered it):
| When local_trylock() fails above, the function jumps to barn_put and returns | pcs without holding the lock. This appears to violate the function's contract | documented in the comment at the beginning of __pcs_replace_empty_main(): | | "If not successful, returns NULL and the local lock unlocked." | | The caller in alloc_from_pcs() checks for NULL to detect failure: | | if (unlikely(pcs->main->size == 0)) { | pcs = __pcs_replace_empty_main(s, pcs, gfp); | if (unlikely(!pcs)) | return NULL; | } | | If the trylock fails and pcs (non-NULL) is returned, the caller proceeds | without realizing the lock was never re-acquired. This leads to accessing | pcs->main without the lock and later trying to unlock a lock that isn't held.Quoted 8 lines
And the analysis sounds correct to me.
perhaps it should be:
if (!local_trylock(&s->cpu_sheaves->lock)) { pcs = NULL; goto barn_put; }
Quoted 4 lines
Thanks a lot Harry. In fact I realized this mistake after initially sending the patch to Ming in a reply, and fixed it locally (same as you suggest).
Or so I thought, because the fix got apparently lost.
That happens sometimes, yeah :)
Quoted 1 line
So I'll do that now in slab/for-next-fixes
Thanks.
Quoted 2 lines
Or actually I think a more robust way is to set pcs = NULL after the unlock, unconditionally, so I'll do that.
Oh, that sounds better!
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
mm/slub.c +1
-0pcs =this_cpu_ptr (s->cpu_sheaves);/* @@ -4667,6 +4663,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,return pcs; } +barn_put:barn_put_full_sheaf (barn, full);stat (s,BARN_PUT );
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerQuoted 73 lines · Vlastimil Babka (SUSE)
Ming Lei reported [1] a regression in the ublk null target benchmark due to sheaves. The profile shows that the
alloc_from_pcs()fastpath fails and allocations fall back to___slab_alloc(). It also shows the allocations happen throughmempool_alloc().The strategy of
mempool_alloc()is to call the underlying allocator (here slab) without__GFP_DIRECT_RECLAIMfirst. This does not play well with__pcs_replace_empty_main()checking forgfpflags_allow_blocking()to decide if it should refill an empty sheaf or fallback to the slowpath, so we end up falling back.We could change the mempool strategy but there might be other paths doing the same ting. So instead allow sheaf refill when blocking is not allowed, changing the condition to
gfpflags_allow_spinning(). The original condition was unnecessarily restrictive.Note this doesn't fully resolve the regression [1] as another component of that are memoryless nodes, which is to be addressed separately.
Reported-by Ming Lei <[email protected]>Fixes e47c897a2949 ("slab: add sheaves to most caches")Signed-off-by Vlastimil Babka (SUSE) <[email protected]>--- mm/slub.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-)mm/slub.c +6
-7@@ -4567,7 +4567,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,struct slab_sheaf *empty =NULL ;struct slab_sheaf *full;struct node_barn *barn; -bool can_alloc; +bool allow_spin;lockdep_assert_held (this_cpu_ptr (&s->cpu_sheaves->lock));@@ -4588,8 +4588,9 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,return NULL ; } - full =barn_replace_empty_sheaf (barn, pcs->main, -gfpflags_allow_spinning (gfp)); + allow_spin =gfpflags_allow_spinning (gfp); + + full =barn_replace_empty_sheaf (barn, pcs->main, allow_spin);if (full) {stat (s,BARN_GET );@@ -4599,9 +4600,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,stat (s,BARN_GET_FAIL ); - can_alloc =gfpflags_allow_blocking (gfp); - -if (can_alloc) { +if (allow_spin) {if (pcs->spare) { empty = pcs->spare; pcs->spare =NULL ;@@ -4612,7 +4611,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,local_unlock (&s->cpu_sheaves->lock); -if (!can_alloc) +if (!allow_spin)return NULL ;if (empty) {
mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635
Quoted 10 lines
mm/slub.c +2
-5@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,if (!full)return NULL ; -/* -* we can reach here only when gfpflags_allow_blocking -* so this must not be an irq -*/ -local_lock (&s->cpu_sheaves->lock); +if (!local_trylock (&s->cpu_sheaves->lock)) + goto barn_put;
A quick question to make sure I understand this correctly.
My understanding is that after this patch, there is now a new case where allocations with __GFP_KSWAPD_RECLAIM set (e.g GFP_ATOMIC) can also reach this lock-reacquire path.
If we were to keep using local_lock here:
1. On non-RT kernels it seems fine, since alloc_from_pcs() already does a local_trylock(&s->cpu_sheaves->lock) check.
2. But on PREEMPT_RT, local_lock could potentially schedule away, which may add latency. So the idea of using local_trylock here is to fail fast and return without incurring that latency - is that the intent behind this change?
Signature · 2 lines
Thanks, Hao
Vlastimil Babka (SUSE)
maintainerQuoted 19 lines · Hao Li
mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635
Quoted 11 lines · Vlastimil Babka (SUSE)
mm/slub.c +2
-5@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,if (!full)return NULL ; -/* -* we can reach here only when gfpflags_allow_blocking -* so this must not be an irq -*/ -local_lock (&s->cpu_sheaves->lock); +if (!local_trylock (&s->cpu_sheaves->lock)) + goto barn_put;A quick question to make sure I understand this correctly.
My understanding is that after this patch, there is now a new case where allocations with
__GFP_KSWAPD_RECLAIMset (e.gGFP_ATOMIC) can also reach this lock-reacquire path.
Quoted 8 lines
If we were to keep using
local_lockhere:1. On non-RT kernels it seems fine, since
alloc_from_pcs()already does alocal_trylock(&s->cpu_sheaves->lock)check.2. But on
PREEMPT_RT,local_lockcould potentially schedule away, which may add latency. So the idea of usinglocal_trylockhere is to fail fast and return without incurring that latency - is that the intent behind this change?
Great question, thanks!
So the main intent is that lockdep would complain if it saw this local_lock() happening in e.g. an irq handler. It doesn't know that it's safe from deadlocks because we already succeeded a trylock before and thus the irq handler didn't interrupt anyone holding the lock.
Trying to teach lockdep such things leads to the complicated initial design of kmalloc_nolock() before it could be simplified by sheaves.
On !RT it makes no difference as the trylock will succeed always. On RT it may not, but indeed they may prefer avoiding the latency as you say.
Hao Li
reviewerQuoted 31 lines · Vlastimil Babka (SUSE)
Quoted 19 lines · Hao Li
mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635
Quoted 11 lines · Vlastimil Babka (SUSE)
mm/slub.c +2
-5@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,if (!full)return NULL ; -/* -* we can reach here only when gfpflags_allow_blocking -* so this must not be an irq -*/ -local_lock (&s->cpu_sheaves->lock); +if (!local_trylock (&s->cpu_sheaves->lock)) + goto barn_put;A quick question to make sure I understand this correctly.
My understanding is that after this patch, there is now a new case where allocations with
__GFP_KSWAPD_RECLAIMset (e.gGFP_ATOMIC) can also reach this lock-reacquire path.Quoted 8 lines
If we were to keep using
local_lockhere:1. On non-RT kernels it seems fine, since
alloc_from_pcs()already does alocal_trylock(&s->cpu_sheaves->lock)check.2. But on
PREEMPT_RT,local_lockcould potentially schedule away, which may add latency. So the idea of usinglocal_trylockhere is to fail fast and return without incurring that latency - is that the intent behind this change?Great question, thanks!
Quoted 7 lines
So the main intent is that lockdep would complain if it saw this
local_lock()happening in e.g. an irq handler. It doesn't know that it's safe from deadlocks because we already succeeded a trylock before and thus the irq handler didn't interrupt anyone holding the lock.Trying to teach lockdep such things leads to the complicated initial design of
kmalloc_nolock()before it could be simplified by sheaves.
Oh, yes - I hadn't considered the impact of lockdep. This is a good point!
Quoted 3 lines
On !RT it makes no difference as the trylock will succeed always. On RT it may not, but indeed they may prefer avoiding the latency as you say.
Yes, make sense, thanks!
Signature · 2 lines
Thanks, Hao