slab: fix memory leak when refill_sheaf() fails
Subsystems (2)
PATCH 1/1 slab: fix memory leak when refill_sheaf() fails
When refill_sheaf() partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will update sheaf->size and return -ENOMEM. However, the callers (alloc_full_sheaf() and __pcs_replace_empty_main()) directly call free_empty_sheaf() on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.
Fix this by calling sheaf_flush_unused() before free_empty_sheaf() to free objects of sheaf->objects[]. And also add a WARN_ON() in free_empty_sheaf() to catch any future cases where a non-empty sheaf is being freed.
--- mm/slub.c | 4 ++++ 1 file changed, 4 insertions(+)
mm/slub.c +4 -0
@@ -2797,6 +2797,7 @@ static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)if (s->flags &SLAB_KMALLOC )mark_obj_codetag_empty (sheaf); +WARN_ON (sheaf->size >0 );kfree (sheaf);stat (s,SHEAF_FREE );
@@ -2828,6 +2829,7 @@ static int refill_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf,return 0 ; } +static void sheaf_flush_unused (struct kmem_cache *s,struct slab_sheaf *sheaf);static struct slab_sheaf *alloc_full_sheaf (struct kmem_cache *s,gfp_t gfp) {
@@ -2837,6 +2839,7 @@ static struct slab_sheaf *alloc_full_sheaf(struct kmem_cache *s, gfp_t gfp)return NULL ;if (refill_sheaf (s, sheaf, gfp | __GFP_NOMEMALLOC | __GFP_NOWARN)) { +sheaf_flush_unused (s, sheaf);free_empty_sheaf (s, sheaf);return NULL ; }
@@ -4623,6 +4626,7 @@ __pcs_replace_empty_main(struct kmem_cache *s, struct slub_percpu_sheaves *pcs,* we must be very low on memory so don't bother * with the barn */ +sheaf_flush_unused (s, empty);free_empty_sheaf (s, empty); } }else {
Signature · 1 line
2.34.1
Quoted 5 lines · Qing Wang
When
refill_sheaf()partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will updatesheaf->sizeand return-ENOMEM. However, the callers (alloc_full_sheaf()and__pcs_replace_empty_main()) directly callfree_empty_sheaf()on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.
Nice catch, thanks!
Probably the need to fail new_slab() made it quite hard to trigger and notice.
Quoted 6 lines
Fix this by calling
sheaf_flush_unused()beforefree_empty_sheaf()to free objects of sheaf->objects[]. And also add aWARN_ON()infree_empty_sheaf()to catch any future cases where a non-empty sheaf is being freed.Fixes 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves")
I think we need to add Cc: [email protected]
Quoted 10 lines
Signed-off-by Qing Wang <[email protected]>--- mm/slub.c | 4 ++++ 1 file changed, 4 insertions(+)mm/slub.c +0
-0
mm/slub.c static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf) · line 2800
Quoted 4 lines
mm/slub.c +1
-0@@ -2797,6 +2797,7 @@ static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)if (s->flags &SLAB_KMALLOC )mark_obj_codetag_empty (sheaf); +WARN_ON (sheaf->size >0 );
nit: perhaps VM_WARN_ON_ONCE(); will be enough?
Otherwise looks good to me, so:
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 9 lines · Harry Yoo
Quoted 5 lines · Qing Wang
When
refill_sheaf()partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will updatesheaf->sizeand return-ENOMEM. However, the callers (alloc_full_sheaf()and__pcs_replace_empty_main()) directly callfree_empty_sheaf()on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.Nice catch, thanks!
Probably the need to failnew_slab()made it quite hard to trigger and notice.
Just out of curiosity, could you please tell us how you discovered this and confirmed that it's fixed, given that kmemleak won't detect it?
Quoted 4 lines
Quoted 4 lines
Fix this by calling
sheaf_flush_unused()beforefree_empty_sheaf()to free objects of sheaf->objects[]. And also add aWARN_ON()infree_empty_sheaf()to catch any future cases where a non-empty sheaf is being freed.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 2 lines · Harry Yoo
Just out of curiosity, could you please tell us how you discovered this and confirmed that it's fixed, given that kmemleak won't detect it?
Just discovered it by code reading (while learing this featch):). And there is no good way to reproduce it, so confirmed it by code logic.
--
cheers,
Qing
Quoted 6 lines · Qing Wang
Quoted 2 lines · Harry Yoo
Just out of curiosity, could you please tell us how you discovered this and confirmed that it's fixed, given that kmemleak won't detect it?
Just discovered it by code reading (while learing this featch):). And there is no good way to reproduce it, so confirmed it by code logic.
Ah, that makes sense.
Thanks!
Signature · 2 lines
Cheers, Harry / Hyeonggon
Vlastimil Babka
maintainerQuoted 8 lines · Harry Yoo
Quoted 5 lines · Qing Wang
When
refill_sheaf()partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will updatesheaf->sizeand return-ENOMEM. However, the callers (alloc_full_sheaf()and__pcs_replace_empty_main()) directly callfree_empty_sheaf()on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.Nice catch, thanks!
Indeed, thanks!
Quoted 1 line
Probably the need to fail
new_slab()made it quite hard to trigger and notice.
Agreed.
Quoted 6 lines
Quoted 6 lines
Fix this by calling
sheaf_flush_unused()beforefree_empty_sheaf()to free objects of sheaf->objects[]. And also add aWARN_ON()infree_empty_sheaf()to catch any future cases where a non-empty sheaf is being freed.Fixes 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves")
Actually I think that commit was fine as it was using bulk alloc to refill and that was undoing any partial successes. I think this one is correct and replaced it so:
Quoted 1 line
I think we need to add Cc: [email protected]
And therefore we don't, unless I'm mistaken.
Quoted 13 lines
Quoted 13 lines
Signed-off-by Qing Wang <[email protected]>--- mm/slub.c | 4 ++++ 1 file changed, 4 insertions(+)mm/slub.c +0
-0@@ -2797,6 +2797,7 @@ static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)if (s->flags &SLAB_KMALLOC )mark_obj_codetag_empty (sheaf);
Quoted 3 lines
mm/slub.c static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf) · line 2800
Quoted 1 line
mm/slub.c +1
-0+WARN_ON (sheaf->size >0 );nit: perhaps
VM_WARN_ON_ONCE(); will be enough?
Yep replaced it too.
Added to slab/for-next-fixes, thanks!
Quoted 3 lines
Otherwise looks good to me, so:
Reviewed-by Harry Yoo <[email protected]>
Quoted 21 lines · Vlastimil Babka
Quoted 8 lines · Harry Yoo
Quoted 5 lines · Qing Wang
When
refill_sheaf()partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will updatesheaf->sizeand return-ENOMEM. However, the callers (alloc_full_sheaf()and__pcs_replace_empty_main()) directly callfree_empty_sheaf()on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.Nice catch, thanks!
Indeed, thanks!
Quoted 1 line
Probably the need to fail
new_slab()made it quite hard to trigger and notice.Agreed.
Quoted 5 lines
Quoted 5 lines
Fix this by calling
sheaf_flush_unused()beforefree_empty_sheaf()to free objects of sheaf->objects[]. And also add aWARN_ON()infree_empty_sheaf()to catch any future cases where a non-empty sheaf is being freed.
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Fixes 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves")Actually I think that commit was fine as it was using bulk alloc to refill and that was undoing any partial successes. I think this one is correct and replaced it so:
Oops, I missed that.
Yeah, at first look I wondered "Well, refill_sheaf()" doesn't partially fill sheaves!" Then I looked at the code right now and it did :/
In 2d517aa09bbc did not partially refill sheaves as it used __kmem_cache_alloc_bulk() to refill sheaves,
Quoted 1 line
Fixes ed30c4adfc2b ("slab: add optimized sheaf refill from partial list")
and this changed the behavior by replacing it with __refill_objects(). So this is the right Fixes: tag.
Quoted 3 lines
Quoted 1 line
I think we need to add Cc: [email protected]
And therefore we don't, unless I'm mistaken.
You're right.
Quoted 20 lines
Quoted 13 lines
Quoted 13 lines
Signed-off-by Qing Wang <[email protected]>--- mm/slub.c | 4 ++++ 1 file changed, 4 insertions(+)mm/slub.c +0
-0@@ -2797,6 +2797,7 @@ static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf)if (s->flags &SLAB_KMALLOC )mark_obj_codetag_empty (sheaf);Quoted 3 lines
mm/slub.c static void free_empty_sheaf(struct kmem_cache *s, struct slab_sheaf *sheaf) · line 2800
Quoted 1 line
mm/slub.c +1
-0+WARN_ON (sheaf->size >0 );nit: perhaps
VM_WARN_ON_ONCE(); will be enough?Yep replaced it too.
Added to slab/for-next-fixes, thanks!
Signature · 2 lines
Cheers, Harry / Hyeonggon
Quoted 6 lines · Qing Wang
When
refill_sheaf()partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will updatesheaf->sizeand return-ENOMEM. However, the callers (alloc_full_sheaf()and__pcs_replace_empty_main()) directly callfree_empty_sheaf()on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.
Quoted 8 lines
Fix this by calling
sheaf_flush_unused()beforefree_empty_sheaf()to free objects of sheaf->objects[]. And also add aWARN_ON()infree_empty_sheaf()to catch any future cases where a non-empty sheaf is being freed.Fixes 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves")Signed-off-by Qing Wang <[email protected]>---
This fix looks good to me.
So feel free to add:
I also want to bring up another point here, although it may be outside the scope of the current fix.
When I looked into the refill_sheaf() path, I found a refill failure does not guarantee that the sheaf remains intact: refill_sheaf() can partially fill the sheaf before failing. This non-intact behavior propagates to its caller, __prefill_sheaf_pfmemalloc(), which therefore also cannot assume that the sheaf is still intact after a refill failure.
However, the comment for kmem_cache_refill_sheaf() says that "if the refill fails (returning -ENOMEM), the existing sheaf is left intact." That means the behavior of __prefill_sheaf_pfmemalloc() - where the sheaf may be left partially filled on refill failure - contradicts the API contract of kmem_cache_refill_sheaf().
Maybe we can add rollback logic to __prefill_sheaf_pfmemalloc() so that it provides intact semantics, preventing the non-intact behavior of refill_sheaf() from propagating up to kmem_cache_refill_sheaf().
Signature · 2 lines
Thanks, Hao
Hao Li
reviewerQuoted 36 lines · Hao Li
Quoted 6 lines · Qing Wang
When
refill_sheaf()partially fills one sheaf (e.g., fills 5 objects but need to fill 10), it will updatesheaf->sizeand return-ENOMEM. However, the callers (alloc_full_sheaf()and__pcs_replace_empty_main()) directly callfree_empty_sheaf()on failure, which only does kfree(sheaf), causing the partially allocated objects memory in sheaf->objects[] leaked.Quoted 8 lines
Fix this by calling
sheaf_flush_unused()beforefree_empty_sheaf()to free objects of sheaf->objects[]. And also add aWARN_ON()infree_empty_sheaf()to catch any future cases where a non-empty sheaf is being freed.Fixes 2d517aa09bbc ("slab: add opt-in caching layer of percpu sheaves")Signed-off-by Qing Wang <[email protected]>---
This fix looks good to me.
So feel free to add:Reviewed-by Hao Li <[email protected]>I also want to bring up another point here, although it may be outside the scope of the current fix.
When I looked into the
refill_sheaf()path, I found a refill failure does not guarantee that the sheaf remains intact:refill_sheaf()can partially fill the sheaf before failing. This non-intact behavior propagates to its caller,__prefill_sheaf_pfmemalloc(), which therefore also cannot assume that the sheaf is still intact after a refill failure.However, the comment for
kmem_cache_refill_sheaf()says that "if the refill fails (returning-ENOMEM), the existing sheaf is left intact." That means the behavior of__prefill_sheaf_pfmemalloc()- where the sheaf may be left partially filled on refill failure - contradicts the API contract ofkmem_cache_refill_sheaf().
Quoted 3 lines
Maybe we can add rollback logic to
__prefill_sheaf_pfmemalloc()so that it provides intact semantics, preventing the non-intact behavior ofrefill_sheaf()from propagating up tokmem_cache_refill_sheaf().
Looking at this a bit more, after checking the current callers, it seems that the existing callers of kmem_cache_refill_sheaf() are not relying on the sheaf remaining intact on refill failure.
If so, then another possible option might be to update the comment for kmem_cache_refill_sheaf() to match the current behavior, rather than adding rollback logic.
So it may just come down to whether we want to preserve the documented semantics in the implementation, or adjust the comment to reflect what the code already does.
I may be missing some intended dependency here, though.
Signature · 2 lines
Thanks, Hao
Vlastimil Babka
maintainerQuoted 20 lines · Hao Li
Quoted 16 lines
I also want to bring up another point here, although it may be outside the scope of the current fix.
When I looked into the
refill_sheaf()path, I found a refill failure does not guarantee that the sheaf remains intact:refill_sheaf()can partially fill the sheaf before failing. This non-intact behavior propagates to its caller,__prefill_sheaf_pfmemalloc(), which therefore also cannot assume that the sheaf is still intact after a refill failure.However, the comment for
kmem_cache_refill_sheaf()says that "if the refill fails (returning-ENOMEM), the existing sheaf is left intact." That means the behavior of__prefill_sheaf_pfmemalloc()- where the sheaf may be left partially filled on refill failure - contradicts the API contract ofkmem_cache_refill_sheaf().Quoted 3 lines
Maybe we can add rollback logic to
__prefill_sheaf_pfmemalloc()so that it provides intact semantics, preventing the non-intact behavior ofrefill_sheaf()from propagating up tokmem_cache_refill_sheaf().
Quoted 7 lines
Looking at this a bit more, after checking the current callers, it seems that the existing callers of
kmem_cache_refill_sheaf()are not relying on the sheaf remaining intact on refill failure.If so, then another possible option might be to update the comment for
kmem_cache_refill_sheaf()to match the current behavior, rather than adding rollback logic.
I agree with this option. Having possibly more objects than before the call shouldn't be an issue for the callers.
Quoted 6 lines
So it may just come down to whether we want to preserve the documented semantics in the implementation, or adjust the comment to reflect what the code already does.
I may be missing some intended dependency here, though.
Quoted 25 lines · Vlastimil Babka
Quoted 24 lines · Hao Li
Quoted 16 lines
I also want to bring up another point here, although it may be outside the scope of the current fix.
When I looked into the
refill_sheaf()path, I found a refill failure does not guarantee that the sheaf remains intact:refill_sheaf()can partially fill the sheaf before failing. This non-intact behavior propagates to its caller,__prefill_sheaf_pfmemalloc(), which therefore also cannot assume that the sheaf is still intact after a refill failure.However, the comment for
kmem_cache_refill_sheaf()says that "if the refill fails (returning-ENOMEM), the existing sheaf is left intact." That means the behavior of__prefill_sheaf_pfmemalloc()- where the sheaf may be left partially filled on refill failure - contradicts the API contract ofkmem_cache_refill_sheaf().Quoted 3 lines
Maybe we can add rollback logic to
__prefill_sheaf_pfmemalloc()so that it provides intact semantics, preventing the non-intact behavior ofrefill_sheaf()from propagating up tokmem_cache_refill_sheaf().Looking at this a bit more, after checking the current callers, it seems that the existing callers of
kmem_cache_refill_sheaf()are not relying on the sheaf remaining intact on refill failure.
Quoted 6 lines
Quoted 3 lines
If so, then another possible option might be to update the comment for
kmem_cache_refill_sheaf()to match the current behavior, rather than adding rollback logic.I agree with this option. Having possibly more objects than before the call shouldn't be an issue for the callers.
+1 for this!
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hao Li
reviewerQuoted 26 lines · Harry Yoo
Quoted 25 lines · Vlastimil Babka
Quoted 24 lines · Hao Li
Quoted 16 lines
I also want to bring up another point here, although it may be outside the scope of the current fix.
When I looked into the
refill_sheaf()path, I found a refill failure does not guarantee that the sheaf remains intact:refill_sheaf()can partially fill the sheaf before failing. This non-intact behavior propagates to its caller,__prefill_sheaf_pfmemalloc(), which therefore also cannot assume that the sheaf is still intact after a refill failure.However, the comment for
kmem_cache_refill_sheaf()says that "if the refill fails (returning-ENOMEM), the existing sheaf is left intact." That means the behavior of__prefill_sheaf_pfmemalloc()- where the sheaf may be left partially filled on refill failure - contradicts the API contract ofkmem_cache_refill_sheaf().Quoted 3 lines
Maybe we can add rollback logic to
__prefill_sheaf_pfmemalloc()so that it provides intact semantics, preventing the non-intact behavior ofrefill_sheaf()from propagating up tokmem_cache_refill_sheaf().Looking at this a bit more, after checking the current callers, it seems that the existing callers of
kmem_cache_refill_sheaf()are not relying on the sheaf remaining intact on refill failure.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
If so, then another possible option might be to update the comment for
kmem_cache_refill_sheaf()to match the current behavior, rather than adding rollback logic.I agree with this option. Having possibly more objects than before the call shouldn't be an issue for the callers.
+1 for this!
Great! Thanks to both of you for confirming!
Signature · 2 lines
Thanks, Hao