Skip to contentSkip to conversation

mm/slab: allow sheaf refill if blocking is not allowed

PATCH In review Vlastimil Babka (SUSE) 4 participants 7 messages Mar 2 – Mar 5

Subsystems (2)
lore
Share & tools
lore

Vlastimil Babka (SUSE)

author maintainer
#0 lore raw

PATCH 1/1 mm/slab: allow sheaf refill if blocking is not allowed

Ming Lei reported [1] a regression in the ublk null target benchmark due to sheaves. The profile shows that the alloc_from_pcs() fastpath fails and allocations fall back to ___slab_alloc(). It also shows the allocations happen through mempool_alloc().

The strategy of mempool_alloc() is to call the underlying allocator (here slab) without __GFP_DIRECT_RECLAIM first. This does not play well with __pcs_replace_empty_main() checking for gfpflags_allow_blocking() to decide if it should refill an empty sheaf or fallback to the slowpath, so we end up falling back.

We could change the mempool strategy but there might be other paths doing the same ting. So instead allow sheaf refill when blocking is not allowed, changing the condition to gfpflags_allow_spinning(). The original condition was unnecessarily restrictive.

Note this doesn't fully resolve the regression [1] as another component of that are memoryless nodes, which is to be addressed separately.

Fixes e47c897a2949 ("slab: add sheaves to most caches")
---
 mm/slub.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
mm/slub.c +9 -12
index b1e9f16ba435..17b200695e9b 100644
@@ -4567,7 +4567,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	struct slab_sheaf *empty = NULL;
 	struct slab_sheaf *full;
 	struct node_barn *barn;
-	bool can_alloc;
+	bool allow_spin;
 
 	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
 
@@ -4588,8 +4588,9 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 		return NULL;
 	}
 
-	full = barn_replace_empty_sheaf(barn, pcs->main,
-					gfpflags_allow_spinning(gfp));
+	allow_spin = gfpflags_allow_spinning(gfp);
+
+	full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
 
 	if (full) {
 		stat(s, BARN_GET);
@@ -4599,9 +4600,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 
 	stat(s, BARN_GET_FAIL);
 
-	can_alloc = gfpflags_allow_blocking(gfp);
-
-	if (can_alloc) {
+	if (allow_spin) {
 		if (pcs->spare) {
 			empty = pcs->spare;
 			pcs->spare = NULL;
@@ -4612,7 +4611,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 
 	local_unlock(&s->cpu_sheaves->lock);
 
-	if (!can_alloc)
+	if (!allow_spin)
 		return NULL;
 
 	if (empty) {
@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	if (!full)
 		return NULL;
 
-	/*
-	 * we can reach here only when gfpflags_allow_blocking
-	 * so this must not be an irq
-	 */
-	local_lock(&s->cpu_sheaves->lock);
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto barn_put;
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 
 	/*
@@ -4667,6 +4663,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 		return pcs;
 	}
 
+barn_put:
 	barn_put_full_sheaf(barn, full);
 	stat(s, BARN_PUT);
 
Signature · 1 line
2.53.0
#1 lore raw
Quoted 32 lines · Vlastimil Babka (SUSE)

Ming Lei reported [1] a regression in the ublk null target benchmark due to sheaves. The profile shows that the alloc_from_pcs() fastpath fails and allocations fall back to ___slab_alloc(). It also shows the allocations happen through mempool_alloc().

The strategy of mempool_alloc() is to call the underlying allocator (here slab) without __GFP_DIRECT_RECLAIM first. This does not play well with __pcs_replace_empty_main() checking for gfpflags_allow_blocking() to decide if it should refill an empty sheaf or fallback to the slowpath, so we end up falling back.

We could change the mempool strategy but there might be other paths doing the same ting. So instead allow sheaf refill when blocking is not allowed, changing the condition to gfpflags_allow_spinning(). The original condition was unnecessarily restrictive.

Note this doesn't fully resolve the regression [1] as another component of that are memoryless nodes, which is to be addressed separately.

---
 mm/slub.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
mm/slub.c +0 -0
index b1e9f16ba435..17b200695e9b 100644

mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635

Quoted 10 lines
mm/slub.c +2 -5
@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	if (!full)
 		return NULL;
 
-	/*
-	 * we can reach here only when gfpflags_allow_blocking
-	 * so this must not be an irq
-	 */
-	local_lock(&s->cpu_sheaves->lock);
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto barn_put;

My AI buddy says (don't worry, I filtered it):

| When local_trylock() fails above, the function jumps to barn_put and returns
| pcs without holding the lock. This appears to violate the function's contract
| documented in the comment at the beginning of __pcs_replace_empty_main():
| 
|     "If not successful, returns NULL and the local lock unlocked."
| 
| The caller in alloc_from_pcs() checks for NULL to detect failure:
| 
|     if (unlikely(pcs->main->size == 0)) {
|         pcs = __pcs_replace_empty_main(s, pcs, gfp);
|         if (unlikely(!pcs))
|             return NULL;
|     }
| 
| If the trylock fails and pcs (non-NULL) is returned, the caller proceeds
| without realizing the lock was never re-acquired. This leads to accessing
| pcs->main without the lock and later trying to unlock a lock that isn't held.

And the analysis sounds correct to me.

perhaps it should be:

if (!local_trylock(&s->cpu_sheaves->lock)) {
	pcs = NULL;
	goto barn_put;
}
Quoted 10 lines
mm/slub.c +1 -0
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 
 	/*
@@ -4667,6 +4663,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 		return pcs;
 	}
 
+barn_put:
 	barn_put_full_sheaf(barn, full);
 	stat(s, BARN_PUT);
Signature · 2 lines
Cheers,
Harry / Hyeonggon
#2 lore raw
Quoted 63 lines · Harry Yoo
Quoted 32 lines · Vlastimil Babka (SUSE)

Ming Lei reported [1] a regression in the ublk null target benchmark due to sheaves. The profile shows that the alloc_from_pcs() fastpath fails and allocations fall back to ___slab_alloc(). It also shows the allocations happen through mempool_alloc().

The strategy of mempool_alloc() is to call the underlying allocator (here slab) without __GFP_DIRECT_RECLAIM first. This does not play well with __pcs_replace_empty_main() checking for gfpflags_allow_blocking() to decide if it should refill an empty sheaf or fallback to the slowpath, so we end up falling back.

We could change the mempool strategy but there might be other paths doing the same ting. So instead allow sheaf refill when blocking is not allowed, changing the condition to gfpflags_allow_spinning(). The original condition was unnecessarily restrictive.

Note this doesn't fully resolve the regression [1] as another component of that are memoryless nodes, which is to be addressed separately.

---
 mm/slub.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
mm/slub.c +0 -0
index b1e9f16ba435..17b200695e9b 100644

mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635

Quoted 10 lines
mm/slub.c +2 -5
@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	if (!full)
 		return NULL;
 
-	/*
-	 * we can reach here only when gfpflags_allow_blocking
-	 * so this must not be an irq
-	 */
-	local_lock(&s->cpu_sheaves->lock);
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto barn_put;

My AI buddy says (don't worry, I filtered it):

| When local_trylock() fails above, the function jumps to barn_put and returns
| pcs without holding the lock. This appears to violate the function's contract
| documented in the comment at the beginning of __pcs_replace_empty_main():
| 
|     "If not successful, returns NULL and the local lock unlocked."
| 
| The caller in alloc_from_pcs() checks for NULL to detect failure:
| 
|     if (unlikely(pcs->main->size == 0)) {
|         pcs = __pcs_replace_empty_main(s, pcs, gfp);
|         if (unlikely(!pcs))
|             return NULL;
|     }
| 
| If the trylock fails and pcs (non-NULL) is returned, the caller proceeds
| without realizing the lock was never re-acquired. This leads to accessing
| pcs->main without the lock and later trying to unlock a lock that isn't held.
Quoted 8 lines

And the analysis sounds correct to me.

perhaps it should be:

if (!local_trylock(&s->cpu_sheaves->lock)) {
	pcs = NULL;
	goto barn_put;
}

Thanks a lot Harry. In fact I realized this mistake after initially sending the patch to Ming in a reply, and fixed it locally (same as you suggest).
Or so I thought, because the fix got apparently lost.
So I'll do that now in slab/for-next-fixes

Or actually I think a more robust way is to set pcs = NULL after the unlock, unconditionally, so I'll do that.

Quoted 11 lines
Quoted 10 lines
mm/slub.c +1 -0
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 
 	/*
@@ -4667,6 +4663,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 		return pcs;
 	}
 
+barn_put:
 	barn_put_full_sheaf(barn, full);
 	stat(s, BARN_PUT);
#3 lore raw
Quoted 73 lines · Vlastimil Babka
Quoted 63 lines · Harry Yoo
Quoted 32 lines · Vlastimil Babka (SUSE)

Ming Lei reported [1] a regression in the ublk null target benchmark due to sheaves. The profile shows that the alloc_from_pcs() fastpath fails and allocations fall back to ___slab_alloc(). It also shows the allocations happen through mempool_alloc().

The strategy of mempool_alloc() is to call the underlying allocator (here slab) without __GFP_DIRECT_RECLAIM first. This does not play well with __pcs_replace_empty_main() checking for gfpflags_allow_blocking() to decide if it should refill an empty sheaf or fallback to the slowpath, so we end up falling back.

We could change the mempool strategy but there might be other paths doing the same ting. So instead allow sheaf refill when blocking is not allowed, changing the condition to gfpflags_allow_spinning(). The original condition was unnecessarily restrictive.

Note this doesn't fully resolve the regression [1] as another component of that are memoryless nodes, which is to be addressed separately.

---
 mm/slub.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
mm/slub.c +0 -0
index b1e9f16ba435..17b200695e9b 100644

mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635

Quoted 10 lines
mm/slub.c +2 -5
@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	if (!full)
 		return NULL;
 
-	/*
-	 * we can reach here only when gfpflags_allow_blocking
-	 * so this must not be an irq
-	 */
-	local_lock(&s->cpu_sheaves->lock);
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto barn_put;

My AI buddy says (don't worry, I filtered it):

| When local_trylock() fails above, the function jumps to barn_put and returns
| pcs without holding the lock. This appears to violate the function's contract
| documented in the comment at the beginning of __pcs_replace_empty_main():
| 
|     "If not successful, returns NULL and the local lock unlocked."
| 
| The caller in alloc_from_pcs() checks for NULL to detect failure:
| 
|     if (unlikely(pcs->main->size == 0)) {
|         pcs = __pcs_replace_empty_main(s, pcs, gfp);
|         if (unlikely(!pcs))
|             return NULL;
|     }
| 
| If the trylock fails and pcs (non-NULL) is returned, the caller proceeds
| without realizing the lock was never re-acquired. This leads to accessing
| pcs->main without the lock and later trying to unlock a lock that isn't held.
Quoted 8 lines

And the analysis sounds correct to me.

perhaps it should be:

if (!local_trylock(&s->cpu_sheaves->lock)) {
	pcs = NULL;
	goto barn_put;
}
Quoted 4 lines

Thanks a lot Harry. In fact I realized this mistake after initially sending the patch to Ming in a reply, and fixed it locally (same as you suggest).
Or so I thought, because the fix got apparently lost.

That happens sometimes, yeah :)

Quoted 1 line

So I'll do that now in slab/for-next-fixes

Thanks.

Quoted 2 lines

Or actually I think a more robust way is to set pcs = NULL after the unlock, unconditionally, so I'll do that.

Oh, that sounds better!

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
mm/slub.c +1 -0
 	pcs = this_cpu_ptr(s->cpu_sheaves);
 
 	/*
@@ -4667,6 +4663,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 		return pcs;
 	}
 
+barn_put:
 	barn_put_full_sheaf(barn, full);
 	stat(s, BARN_PUT);
Signature · 2 lines
Cheers,
Harry / Hyeonggon
reviewer
#4 lore raw
In reply to Vlastimil Babka (SUSE)
Quoted 73 lines · Vlastimil Babka (SUSE)

Ming Lei reported [1] a regression in the ublk null target benchmark due to sheaves. The profile shows that the alloc_from_pcs() fastpath fails and allocations fall back to ___slab_alloc(). It also shows the allocations happen through mempool_alloc().

The strategy of mempool_alloc() is to call the underlying allocator (here slab) without __GFP_DIRECT_RECLAIM first. This does not play well with __pcs_replace_empty_main() checking for gfpflags_allow_blocking() to decide if it should refill an empty sheaf or fallback to the slowpath, so we end up falling back.

We could change the mempool strategy but there might be other paths doing the same ting. So instead allow sheaf refill when blocking is not allowed, changing the condition to gfpflags_allow_spinning(). The original condition was unnecessarily restrictive.

Note this doesn't fully resolve the regression [1] as another component of that are memoryless nodes, which is to be addressed separately.

Fixes e47c897a2949 ("slab: add sheaves to most caches")
---
 mm/slub.c | 21 +++++++++------------
 1 file changed, 9 insertions(+), 12 deletions(-)
mm/slub.c +6 -7
index b1e9f16ba435..17b200695e9b 100644
@@ -4567,7 +4567,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	struct slab_sheaf *empty = NULL;
 	struct slab_sheaf *full;
 	struct node_barn *barn;
-	bool can_alloc;
+	bool allow_spin;
 
 	lockdep_assert_held(this_cpu_ptr(&s->cpu_sheaves->lock));
 
@@ -4588,8 +4588,9 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 		return NULL;
 	}
 
-	full = barn_replace_empty_sheaf(barn, pcs->main,
-					gfpflags_allow_spinning(gfp));
+	allow_spin = gfpflags_allow_spinning(gfp);
+
+	full = barn_replace_empty_sheaf(barn, pcs->main, allow_spin);
 
 	if (full) {
 		stat(s, BARN_GET);
@@ -4599,9 +4600,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 
 	stat(s, BARN_GET_FAIL);
 
-	can_alloc = gfpflags_allow_blocking(gfp);
-
-	if (can_alloc) {
+	if (allow_spin) {
 		if (pcs->spare) {
 			empty = pcs->spare;
 			pcs->spare = NULL;
@@ -4612,7 +4611,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 
 	local_unlock(&s->cpu_sheaves->lock);
 
-	if (!can_alloc)
+	if (!allow_spin)
 		return NULL;
 
 	if (empty) {

mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635

Quoted 10 lines
mm/slub.c +2 -5
@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	if (!full)
 		return NULL;
 
-	/*
-	 * we can reach here only when gfpflags_allow_blocking
-	 * so this must not be an irq
-	 */
-	local_lock(&s->cpu_sheaves->lock);
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto barn_put;

A quick question to make sure I understand this correctly.

My understanding is that after this patch, there is now a new case where allocations with __GFP_KSWAPD_RECLAIM set (e.g GFP_ATOMIC) can also reach this lock-reacquire path.

If we were to keep using local_lock here:

1. On non-RT kernels it seems fine, since alloc_from_pcs() already does a local_trylock(&s->cpu_sheaves->lock) check.

2. But on PREEMPT_RT, local_lock could potentially schedule away, which may add latency. So the idea of using local_trylock here is to fail fast and return without incurring that latency - is that the intent behind this change?

Signature · 2 lines
Thanks,
Hao
author maintainer
#5 lore raw
Quoted 19 lines · Hao Li

mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635

Quoted 11 lines · Vlastimil Babka (SUSE)
mm/slub.c +2 -5
@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	if (!full)
 		return NULL;
 
-	/*
-	 * we can reach here only when gfpflags_allow_blocking
-	 * so this must not be an irq
-	 */
-	local_lock(&s->cpu_sheaves->lock);
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto barn_put;

A quick question to make sure I understand this correctly.

My understanding is that after this patch, there is now a new case where allocations with __GFP_KSWAPD_RECLAIM set (e.g GFP_ATOMIC) can also reach this lock-reacquire path.

Quoted 8 lines

If we were to keep using local_lock here:

1. On non-RT kernels it seems fine, since alloc_from_pcs() already does a local_trylock(&s->cpu_sheaves->lock) check.

2. But on PREEMPT_RT, local_lock could potentially schedule away, which may add latency. So the idea of using local_trylock here is to fail fast and return without incurring that latency - is that the intent behind this change?

Great question, thanks!

So the main intent is that lockdep would complain if it saw this local_lock() happening in e.g. an irq handler. It doesn't know that it's safe from deadlocks because we already succeeded a trylock before and thus the irq handler didn't interrupt anyone holding the lock.

Trying to teach lockdep such things leads to the complicated initial design of kmalloc_nolock() before it could be simplified by sheaves.

On !RT it makes no difference as the trylock will succeed always. On RT it may not, but indeed they may prefer avoiding the latency as you say.

reviewer
#6 lore raw
Quoted 31 lines · Vlastimil Babka (SUSE)
Quoted 19 lines · Hao Li

mm/slub.c __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs, · line 4635

Quoted 11 lines · Vlastimil Babka (SUSE)
mm/slub.c +2 -5
@@ -4632,11 +4631,8 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 	if (!full)
 		return NULL;
 
-	/*
-	 * we can reach here only when gfpflags_allow_blocking
-	 * so this must not be an irq
-	 */
-	local_lock(&s->cpu_sheaves->lock);
+	if (!local_trylock(&s->cpu_sheaves->lock))
+		goto barn_put;

A quick question to make sure I understand this correctly.

My understanding is that after this patch, there is now a new case where allocations with __GFP_KSWAPD_RECLAIM set (e.g GFP_ATOMIC) can also reach this lock-reacquire path.

Quoted 8 lines

If we were to keep using local_lock here:

1. On non-RT kernels it seems fine, since alloc_from_pcs() already does a local_trylock(&s->cpu_sheaves->lock) check.

2. But on PREEMPT_RT, local_lock could potentially schedule away, which may add latency. So the idea of using local_trylock here is to fail fast and return without incurring that latency - is that the intent behind this change?

Great question, thanks!

Quoted 7 lines

So the main intent is that lockdep would complain if it saw this local_lock() happening in e.g. an irq handler. It doesn't know that it's safe from deadlocks because we already succeeded a trylock before and thus the irq handler didn't interrupt anyone holding the lock.

Trying to teach lockdep such things leads to the complicated initial design of kmalloc_nolock() before it could be simplified by sheaves.

Oh, yes - I hadn't considered the impact of lockdep. This is a good point!

Quoted 3 lines

On !RT it makes no difference as the trylock will succeed always. On RT it may not, but indeed they may prefer avoiding the latency as you say.

Yes, make sense, thanks!

Signature · 2 lines
Thanks,
Hao