Skip to contentSkip to conversation

slab: fix memory leak when refill_sheaf() fails

PATCH In review Qing Wang 4 participants 12 messages Mar 11 – Mar 12

Subsystems (2)
lore
Share & tools
lore

Qing Wang

author
#0 lore raw

PATCH 1/1 slab: fix memory leak when refill_sheaf() fails

When refill_sheaf() partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will update sheaf->size and return -ENOMEM. However, the callers (alloc_full_sheaf() and __pcs_replace_empty_main()) directly call free_empty_sheaf() on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.

Fix this by calling sheaf_flush_unused() before free_empty_sheaf() to free objects of sheaf->objects[]. And also add a WARN_ON() in free_empty_sheaf() to catch any future cases where a non-empty sheaf is being freed.

Fixes 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves")
---
 mm/slub.c | 4 ++++
 1 file changed, 4 insertions(+)
mm/slub.c +4 -0
index 20cb4f3b636d..73b2cfd0e123 100644
@@ -2797,6 +2797,7 @@ static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
 	if (s->flags & SLAB_KMALLOC)
 		mark_obj_codetag_empty(sheaf);
 
+	WARN_ON(sheaf->size > 0);
 	kfree(sheaf);
 
 	stat(s, SHEAF_FREE);
@@ -2828,6 +2829,7 @@ static int refill_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf,
 	return 0;
 }
 
+static void sheaf_flush_unused(struct kmem_cache *s, struct slab_sheaf *sheaf);
 
 static struct slab_sheaf *alloc_full_sheaf(struct kmem_cache *s, gfp_t gfp)
 {
@@ -2837,6 +2839,7 @@ static struct slab_sheaf *alloc_full_sheaf(struct kmem_cache *s, gfp_t gfp)
 		return NULL;
 
 	if (refill_sheaf(s, sheaf, gfp | __GFP_NOMEMALLOC | __GFP_NOWARN)) {
+		sheaf_flush_unused(s, sheaf);
 		free_empty_sheaf(s, sheaf);
 		return NULL;
 	}
@@ -4623,6 +4626,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,
 			 * we must be very low on memory so don't bother
 			 * with the barn
 			 */
+			sheaf_flush_unused(s, empty);
 			free_empty_sheaf(s, empty);
 		}
 	} else {
Signature · 1 line
2.34.1
Reviewed-by on v1 1/1
#1 lore raw
Quoted 5 lines · Qing Wang

When refill_sheaf() partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will update sheaf->size and return -ENOMEM. However, the callers (alloc_full_sheaf() and __pcs_replace_empty_main()) directly call free_empty_sheaf() on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.

Nice catch, thanks!
Probably the need to fail new_slab() made it quite hard to trigger and notice.

Quoted 6 lines

Fix this by calling sheaf_flush_unused() before free_empty_sheaf() to free objects of sheaf->objects[]. And also add a WARN_ON() in free_empty_sheaf() to catch any future cases where a non-empty sheaf is being freed.

Fixes 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves")

I think we need to add Cc: [email protected]

Quoted 10 lines
---
 mm/slub.c | 4 ++++
 1 file changed, 4 insertions(+)
mm/slub.c +0 -0
index 20cb4f3b636d..73b2cfd0e123 100644

mm/slub.c static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf) · line 2800

Quoted 4 lines
mm/slub.c +1 -0
@@ -2797,6 +2797,7 @@ static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
 	if (s->flags & SLAB_KMALLOC)
 		mark_obj_codetag_empty(sheaf);
 
+	WARN_ON(sheaf->size > 0);

nit: perhaps VM_WARN_ON_ONCE(); will be enough?

Otherwise looks good to me, so:

Signature · 2 lines
Cheers,
Harry / Hyeonggon
#2 lore raw
Quoted 9 lines · Harry Yoo
Quoted 5 lines · Qing Wang

When refill_sheaf() partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will update sheaf->size and return -ENOMEM. However, the callers (alloc_full_sheaf() and __pcs_replace_empty_main()) directly call free_empty_sheaf() on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.

Nice catch, thanks!
Probably the need to fail new_slab() made it quite hard to trigger and notice.

Just out of curiosity, could you please tell us how you discovered this and confirmed that it's fixed, given that kmemleak won't detect it?

Quoted 4 lines
Quoted 4 lines

Fix this by calling sheaf_flush_unused() before free_empty_sheaf() to free objects of sheaf->objects[]. And also add a WARN_ON() in free_empty_sheaf() to catch any future cases where a non-empty sheaf is being freed.

Signature · 2 lines
Cheers,
Harry / Hyeonggon
author
#3 lore raw
Quoted 2 lines · Harry Yoo

Just out of curiosity, could you please tell us how you discovered this and confirmed that it's fixed, given that kmemleak won't detect it?

Just discovered it by code reading (while learing this featch):). And there is no good way to reproduce it, so confirmed it by code logic.

--
cheers,
Qing

#4 lore raw
Quoted 6 lines · Qing Wang
Quoted 2 lines · Harry Yoo

Just out of curiosity, could you please tell us how you discovered this and confirmed that it's fixed, given that kmemleak won't detect it?

Just discovered it by code reading (while learing this featch):). And there is no good way to reproduce it, so confirmed it by code logic.

Ah, that makes sense.
Thanks!

Signature · 2 lines
Cheers,
Harry / Hyeonggon
maintainer
#5 lore raw
In reply to Harry Yoo
Quoted 8 lines · Harry Yoo
Quoted 5 lines · Qing Wang

When refill_sheaf() partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will update sheaf->size and return -ENOMEM. However, the callers (alloc_full_sheaf() and __pcs_replace_empty_main()) directly call free_empty_sheaf() on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.

Nice catch, thanks!

Indeed, thanks!

Quoted 1 line

Probably the need to fail new_slab() made it quite hard to trigger and notice.

Agreed.

Quoted 6 lines
Quoted 6 lines

Fix this by calling sheaf_flush_unused() before free_empty_sheaf() to free objects of sheaf->objects[]. And also add a WARN_ON() in free_empty_sheaf() to catch any future cases where a non-empty sheaf is being freed.

Fixes 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves")

Actually I think that commit was fine as it was using bulk alloc to refill and that was undoing any partial successes. I think this one is correct and replaced it so:

Fixes ed30c4adfc2b ("slab: add optimized sheaf refill from partial list")
Quoted 1 line

I think we need to add Cc: [email protected]

And therefore we don't, unless I'm mistaken.

Quoted 13 lines
Quoted 13 lines
---
 mm/slub.c | 4 ++++
 1 file changed, 4 insertions(+)
mm/slub.c +0 -0
index 20cb4f3b636d..73b2cfd0e123 100644
@@ -2797,6 +2797,7 @@ static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
 	if (s->flags & SLAB_KMALLOC)
 		mark_obj_codetag_empty(sheaf);
 
Quoted 3 lines

mm/slub.c static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf) · line 2800

Quoted 1 line
mm/slub.c +1 -0
+	WARN_ON(sheaf->size > 0);

nit: perhaps VM_WARN_ON_ONCE(); will be enough?

Yep replaced it too.

Added to slab/for-next-fixes, thanks!

Quoted 3 lines

Otherwise looks good to me, so:

#6 lore raw
Quoted 21 lines · Vlastimil Babka
Quoted 8 lines · Harry Yoo
Quoted 5 lines · Qing Wang

When refill_sheaf() partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will update sheaf->size and return -ENOMEM. However, the callers (alloc_full_sheaf() and __pcs_replace_empty_main()) directly call free_empty_sheaf() on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.

Nice catch, thanks!

Indeed, thanks!

Quoted 1 line

Probably the need to fail new_slab() made it quite hard to trigger and notice.

Agreed.

Quoted 5 lines
Quoted 5 lines

Fix this by calling sheaf_flush_unused() before free_empty_sheaf() to free objects of sheaf->objects[]. And also add a WARN_ON() in free_empty_sheaf() to catch any future cases where a non-empty sheaf is being freed.

Quoted 5 lines
Quoted 1 line
Quoted 1 line
Fixes 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves")

Actually I think that commit was fine as it was using bulk alloc to refill and that was undoing any partial successes. I think this one is correct and replaced it so:

Oops, I missed that.

Yeah, at first look I wondered "Well, refill_sheaf()" doesn't partially fill sheaves!" Then I looked at the code right now and it did :/

In 2d517aa09bbc did not partially refill sheaves as it used __kmem_cache_alloc_bulk() to refill sheaves,

Quoted 1 line
Fixes ed30c4adfc2b ("slab: add optimized sheaf refill from partial list")

and this changed the behavior by replacing it with __refill_objects(). So this is the right Fixes: tag.

Quoted 3 lines
Quoted 1 line

I think we need to add Cc: [email protected]

And therefore we don't, unless I'm mistaken.

You're right.

Quoted 20 lines
Quoted 13 lines
Quoted 13 lines
---
 mm/slub.c | 4 ++++
 1 file changed, 4 insertions(+)
mm/slub.c +0 -0
index 20cb4f3b636d..73b2cfd0e123 100644
@@ -2797,6 +2797,7 @@ static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)
 	if (s->flags & SLAB_KMALLOC)
 		mark_obj_codetag_empty(sheaf);
 
Quoted 3 lines

mm/slub.c static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf) · line 2800

Quoted 1 line
mm/slub.c +1 -0
+	WARN_ON(sheaf->size > 0);

nit: perhaps VM_WARN_ON_ONCE(); will be enough?

Yep replaced it too.

Added to slab/for-next-fixes, thanks!

Signature · 2 lines
Cheers,
Harry / Hyeonggon
reviewer Reviewed-by on v1 1/1
#7 lore raw
In reply to Qing Wang
Quoted 6 lines · Qing Wang

When refill_sheaf() partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will update sheaf->size and return -ENOMEM. However, the callers (alloc_full_sheaf() and __pcs_replace_empty_main()) directly call free_empty_sheaf() on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.

Quoted 8 lines

Fix this by calling sheaf_flush_unused() before free_empty_sheaf() to free objects of sheaf->objects[]. And also add a WARN_ON() in free_empty_sheaf() to catch any future cases where a non-empty sheaf is being freed.

Fixes 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves")

---

This fix looks good to me.
So feel free to add:

I also want to bring up another point here, although it may be outside the scope of the current fix.

When I looked into the refill_sheaf() path, I found a refill failure does not guarantee that the sheaf remains intact: refill_sheaf() can partially fill the sheaf before failing. This non-intact behavior propagates to its caller, __prefill_sheaf_pfmemalloc(), which therefore also cannot assume that the sheaf is still intact after a refill failure.

However, the comment for kmem_cache_refill_sheaf() says that "if the refill fails (returning -ENOMEM), the existing sheaf is left intact." That means the behavior of __prefill_sheaf_pfmemalloc() - where the sheaf may be left partially filled on refill failure - contradicts the API contract of kmem_cache_refill_sheaf().

Maybe we can add rollback logic to __prefill_sheaf_pfmemalloc() so that it provides intact semantics, preventing the non-intact behavior of refill_sheaf() from propagating up to kmem_cache_refill_sheaf().

Signature · 2 lines
Thanks,
Hao
reviewer
#8 lore raw
Quoted 36 lines · Hao Li
Quoted 6 lines · Qing Wang

When refill_sheaf() partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will update sheaf->size and return -ENOMEM. However, the callers (alloc_full_sheaf() and __pcs_replace_empty_main()) directly call free_empty_sheaf() on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.

Quoted 8 lines

Fix this by calling sheaf_flush_unused() before free_empty_sheaf() to free objects of sheaf->objects[]. And also add a WARN_ON() in free_empty_sheaf() to catch any future cases where a non-empty sheaf is being freed.

Fixes 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves")

---

This fix looks good to me.
So feel free to add:

I also want to bring up another point here, although it may be outside the scope of the current fix.

When I looked into the refill_sheaf() path, I found a refill failure does not guarantee that the sheaf remains intact: refill_sheaf() can partially fill the sheaf before failing. This non-intact behavior propagates to its caller, __prefill_sheaf_pfmemalloc(), which therefore also cannot assume that the sheaf is still intact after a refill failure.

However, the comment for kmem_cache_refill_sheaf() says that "if the refill fails (returning -ENOMEM), the existing sheaf is left intact." That means the behavior of __prefill_sheaf_pfmemalloc() - where the sheaf may be left partially filled on refill failure - contradicts the API contract of kmem_cache_refill_sheaf().

Quoted 3 lines

Maybe we can add rollback logic to __prefill_sheaf_pfmemalloc() so that it provides intact semantics, preventing the non-intact behavior of refill_sheaf() from propagating up to kmem_cache_refill_sheaf().

Looking at this a bit more, after checking the current callers, it seems that the existing callers of kmem_cache_refill_sheaf() are not relying on the sheaf remaining intact on refill failure.

If so, then another possible option might be to update the comment for kmem_cache_refill_sheaf() to match the current behavior, rather than adding rollback logic.

So it may just come down to whether we want to preserve the documented semantics in the implementation, or adjust the comment to reflect what the code already does.

I may be missing some intended dependency here, though.

Signature · 2 lines
Thanks,
Hao
maintainer
#9 lore raw
Quoted 20 lines · Hao Li
Quoted 16 lines

I also want to bring up another point here, although it may be outside the scope of the current fix.

When I looked into the refill_sheaf() path, I found a refill failure does not guarantee that the sheaf remains intact: refill_sheaf() can partially fill the sheaf before failing. This non-intact behavior propagates to its caller, __prefill_sheaf_pfmemalloc(), which therefore also cannot assume that the sheaf is still intact after a refill failure.

However, the comment for kmem_cache_refill_sheaf() says that "if the refill fails (returning -ENOMEM), the existing sheaf is left intact." That means the behavior of __prefill_sheaf_pfmemalloc() - where the sheaf may be left partially filled on refill failure - contradicts the API contract of kmem_cache_refill_sheaf().

Quoted 3 lines

Maybe we can add rollback logic to __prefill_sheaf_pfmemalloc() so that it provides intact semantics, preventing the non-intact behavior of refill_sheaf() from propagating up to kmem_cache_refill_sheaf().

Quoted 7 lines

Looking at this a bit more, after checking the current callers, it seems that the existing callers of kmem_cache_refill_sheaf() are not relying on the sheaf remaining intact on refill failure.

If so, then another possible option might be to update the comment for kmem_cache_refill_sheaf() to match the current behavior, rather than adding rollback logic.

I agree with this option. Having possibly more objects than before the call shouldn't be an issue for the callers.

Quoted 6 lines

So it may just come down to whether we want to preserve the documented semantics in the implementation, or adjust the comment to reflect what the code already does.

I may be missing some intended dependency here, though.

#10 lore raw
Quoted 25 lines · Vlastimil Babka
Quoted 24 lines · Hao Li
Quoted 16 lines

I also want to bring up another point here, although it may be outside the scope of the current fix.

When I looked into the refill_sheaf() path, I found a refill failure does not guarantee that the sheaf remains intact: refill_sheaf() can partially fill the sheaf before failing. This non-intact behavior propagates to its caller, __prefill_sheaf_pfmemalloc(), which therefore also cannot assume that the sheaf is still intact after a refill failure.

However, the comment for kmem_cache_refill_sheaf() says that "if the refill fails (returning -ENOMEM), the existing sheaf is left intact." That means the behavior of __prefill_sheaf_pfmemalloc() - where the sheaf may be left partially filled on refill failure - contradicts the API contract of kmem_cache_refill_sheaf().

Quoted 3 lines

Maybe we can add rollback logic to __prefill_sheaf_pfmemalloc() so that it provides intact semantics, preventing the non-intact behavior of refill_sheaf() from propagating up to kmem_cache_refill_sheaf().

Looking at this a bit more, after checking the current callers, it seems that the existing callers of kmem_cache_refill_sheaf() are not relying on the sheaf remaining intact on refill failure.

Quoted 6 lines
Quoted 3 lines

If so, then another possible option might be to update the comment for kmem_cache_refill_sheaf() to match the current behavior, rather than adding rollback logic.

I agree with this option. Having possibly more objects than before the call shouldn't be an issue for the callers.

+1 for this!

Signature · 2 lines
Cheers,
Harry / Hyeonggon
reviewer
#11 lore raw
Quoted 26 lines · Harry Yoo
Quoted 25 lines · Vlastimil Babka
Quoted 24 lines · Hao Li
Quoted 16 lines

I also want to bring up another point here, although it may be outside the scope of the current fix.

When I looked into the refill_sheaf() path, I found a refill failure does not guarantee that the sheaf remains intact: refill_sheaf() can partially fill the sheaf before failing. This non-intact behavior propagates to its caller, __prefill_sheaf_pfmemalloc(), which therefore also cannot assume that the sheaf is still intact after a refill failure.

However, the comment for kmem_cache_refill_sheaf() says that "if the refill fails (returning -ENOMEM), the existing sheaf is left intact." That means the behavior of __prefill_sheaf_pfmemalloc() - where the sheaf may be left partially filled on refill failure - contradicts the API contract of kmem_cache_refill_sheaf().

Quoted 3 lines

Maybe we can add rollback logic to __prefill_sheaf_pfmemalloc() so that it provides intact semantics, preventing the non-intact behavior of refill_sheaf() from propagating up to kmem_cache_refill_sheaf().

Looking at this a bit more, after checking the current callers, it seems that the existing callers of kmem_cache_refill_sheaf() are not relying on the sheaf remaining intact on refill failure.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

If so, then another possible option might be to update the comment for kmem_cache_refill_sheaf() to match the current behavior, rather than adding rollback logic.

I agree with this option. Having possibly more objects than before the call shouldn't be an issue for the callers.

+1 for this!

Great! Thanks to both of you for confirming!

Signature · 2 lines
Thanks,
Hao