slub: fix unreclaimable slab stat for bulk free
Subsystems (2)
PATCH 1/1 slub: fix unreclaimable slab stat for bulk free
SLUB uses page allocator for higher order allocations and update unreclaimable slab stat for such allocations. At the moment, the bulk free for SLUB does not share code with normal free code path for these type of allocations and have missed the stat update. So, fix the stat update by common code. The user visible impact of the bug is the potential of inconsistent unreclaimable slab stat visible through meminfo and vmstat.
--- mm/slub.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
mm/slub.c +12 -10
@@ -3238,6 +3238,16 @@ struct detached_freelist {struct kmem_cache *s; }; +static inline void free_nonslab_page (struct page *page) +{ +unsigned int order =compound_order (page); + +VM_BUG_ON_PAGE (!PageCompound (page), page); +kfree_hook (page_address (page)); +mod_lruvec_page_state (page,NR_SLAB_UNRECLAIMABLE_B , -(PAGE_SIZE << order)); +__free_pages (page, order); +} +/* * This function progressively scans the array with free objects (with * a limited look ahead) and extract objects belonging to the same
@@ -3274,9 +3284,7 @@ int build_detached_freelist(struct kmem_cache *s, size_t size,if (!s) {/* Handle kalloc'ed objects */ if (unlikely (!PageSlab (page))) { -BUG_ON (!PageCompound (page)); -kfree_hook (object); -__free_pages (page,compound_order (page)); +free_nonslab_page (page); p[size] =NULL ;/* mark object processed */ return size; }
@@ -4252,13 +4260,7 @@ void kfree(const void *x) page =virt_to_head_page (x);if (unlikely (!PageSlab (page))) { -unsigned int order =compound_order (page); - -BUG_ON (!PageCompound (page)); -kfree_hook (object); -mod_lruvec_page_state (page,NR_SLAB_UNRECLAIMABLE_B , - -(PAGE_SIZE << order)); -__free_pages (page, order); +free_nonslab_page (page);return ; }slab_free (page->slab_cache, page, object,NULL ,1 , _RET_IP_);
Signature · 1 line
2.32.0.432.gabb21c7263-goog
Quoted 10 lines · Shakeel Butt
SLUB uses page allocator for higher order allocations and update unreclaimable slab stat for such allocations. At the moment, the bulk free for SLUB does not share code with normal free code path for these type of allocations and have missed the stat update. So, fix the stat update by common code. The user visible impact of the bug is the potential of inconsistent unreclaimable slab stat visible through meminfo and vmstat.
Fixes 6a486c0ad4dc ("mm, sl[ou]b: improve memory accounting")Signed-off-by Shakeel Butt <[email protected]>
LGTM
Quoted 53 lines
--- mm/slub.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)mm/slub.c +12
-10@@ -3238,6 +3238,16 @@ struct detached_freelist {struct kmem_cache *s; }; +static inline void free_nonslab_page (struct page *page) +{ +unsigned int order =compound_order (page); + +VM_BUG_ON_PAGE (!PageCompound (page), page); +kfree_hook (page_address (page)); +mod_lruvec_page_state (page,NR_SLAB_UNRECLAIMABLE_B , -(PAGE_SIZE << order)); +__free_pages (page, order); +} +/* * This function progressively scans the array with free objects (with * a limited look ahead) and extract objects belonging to the same @@ -3274,9 +3284,7 @@ int build_detached_freelist(struct kmem_cache *s, size_t size,if (!s) {/* Handle kalloc'ed objects */ if (unlikely (!PageSlab (page))) { -BUG_ON (!PageCompound (page)); -kfree_hook (object); -__free_pages (page,compound_order (page)); +free_nonslab_page (page); p[size] =NULL ;/* mark object processed */ return size; }@@ -4252,13 +4260,7 @@ void kfree(const void *x) page =virt_to_head_page (x);if (unlikely (!PageSlab (page))) { -unsigned int order =compound_order (page); - -BUG_ON (!PageCompound (page)); -kfree_hook (object); -mod_lruvec_page_state (page,NR_SLAB_UNRECLAIMABLE_B , - -(PAGE_SIZE << order)); -__free_pages (page, order); +free_nonslab_page (page);return ; }slab_free (page->slab_cache, page, object,NULL ,1 , _RET_IP_);Signature · 1 line
2.32.0.432.gabb21c7263-goog
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 10 lines · Shakeel Butt
SLUB uses page allocator for higher order allocations and update unreclaimable slab stat for such allocations. At the moment, the bulk free for SLUB does not share code with normal free code path for these type of allocations and have missed the stat update. So, fix the stat update by common code. The user visible impact of the bug is the potential of inconsistent unreclaimable slab stat visible through meminfo and vmstat.
Fixes 6a486c0ad4dc ("mm, sl[ou]b: improve memory accounting")Signed-off-by Shakeel Butt <[email protected]>
Thanks!
Quoted 11 lines · Shakeel Butt
SLUB uses page allocator for higher order allocations and update unreclaimable slab stat for such allocations. At the moment, the bulk free for SLUB does not share code with normal free code path for these type of allocations and have missed the stat update. So, fix the stat update by common code. The user visible impact of the bug is the potential of inconsistent unreclaimable slab stat visible through meminfo and vmstat.
Fixes 6a486c0ad4dc ("mm, sl[ou]b: improve memory accounting")Signed-off-by Shakeel Butt <[email protected]>
Quoted 19 lines · Shakeel Butt
SLUB uses page allocator for higher order allocations and update unreclaimable slab stat for such allocations. At the moment, the bulk free for SLUB does not share code with normal free code path for these type of allocations and have missed the stat update. So, fix the stat update by common code. The user visible impact of the bug is the potential of inconsistent unreclaimable slab stat visible through meminfo and vmstat.
Fixes 6a486c0ad4dc ("mm, sl[ou]b: improve memory accounting")Signed-off-by Shakeel Butt <[email protected]>--- mm/slub.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)mm/slub.c +0
-0
mm/slub.c struct detached_freelist { · line 3245
Quoted 8 lines
mm/slub.c +5
-0@@ -3238,6 +3238,16 @@ struct detached_freelist {struct kmem_cache *s; }; +static inline void free_nonslab_page (struct page *page) +{ +unsigned int order =compound_order (page); + +VM_BUG_ON_PAGE (!PageCompound (page), page);
Could we add WARN_ON here, or we got nothing when CONFIG_DEBUG_VM is disabled.
Quoted 34 lines
mm/slub.c +7
-10+kfree_hook (page_address (page)); +mod_lruvec_page_state (page,NR_SLAB_UNRECLAIMABLE_B , -(PAGE_SIZE << order)); +__free_pages (page, order); +} +/* * This function progressively scans the array with free objects (with * a limited look ahead) and extract objects belonging to the same @@ -3274,9 +3284,7 @@ int build_detached_freelist(struct kmem_cache *s, size_t size,if (!s) {/* Handle kalloc'ed objects */ if (unlikely (!PageSlab (page))) { -BUG_ON (!PageCompound (page)); -kfree_hook (object); -__free_pages (page,compound_order (page)); +free_nonslab_page (page); p[size] =NULL ;/* mark object processed */ return size; }@@ -4252,13 +4260,7 @@ void kfree(const void *x) page =virt_to_head_page (x);if (unlikely (!PageSlab (page))) { -unsigned int order =compound_order (page); - -BUG_ON (!PageCompound (page)); -kfree_hook (object); -mod_lruvec_page_state (page,NR_SLAB_UNRECLAIMABLE_B , - -(PAGE_SIZE << order)); -__free_pages (page, order); +free_nonslab_page (page);return ; }slab_free (page->slab_cache, page, object,NULL ,1 , _RET_IP_);
Quoted 25 lines · Kefeng Wang
Quoted 22 lines · Shakeel Butt
SLUB uses page allocator for higher order allocations and update unreclaimable slab stat for such allocations. At the moment, the bulk free for SLUB does not share code with normal free code path for these type of allocations and have missed the stat update. So, fix the stat update by common code. The user visible impact of the bug is the potential of inconsistent unreclaimable slab stat visible through meminfo and vmstat.
Fixes 6a486c0ad4dc ("mm, sl[ou]b: improve memory accounting")Signed-off-by Shakeel Butt <[email protected]>--- mm/slub.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)mm/slub.c +0
-0@@ -3238,6 +3238,16 @@ struct detached_freelist {struct kmem_cache *s; };
Quoted 8 lines
mm/slub.c struct detached_freelist { · line 3245
Quoted 5 lines
mm/slub.c +5
-0+static inline void free_nonslab_page (struct page *page) +{ +unsigned int order =compound_order (page); + +VM_BUG_ON_PAGE (!PageCompound (page), page);Could we add
WARN_ONhere, or we got nothing whenCONFIG_DEBUG_VMis disabled.
I don't have a strong opinion on this. Please send a patch with reasoning if you want WARN_ON_ONCE here.
Quoted 32 lines · Shakeel Butt
Quoted 24 lines · Kefeng Wang
Quoted 22 lines · Shakeel Butt
SLUB uses page allocator for higher order allocations and update unreclaimable slab stat for such allocations. At the moment, the bulk free for SLUB does not share code with normal free code path for these type of allocations and have missed the stat update. So, fix the stat update by common code. The user visible impact of the bug is the potential of inconsistent unreclaimable slab stat visible through meminfo and vmstat.
Fixes 6a486c0ad4dc ("mm, sl[ou]b: improve memory accounting")Signed-off-by Shakeel Butt <[email protected]>--- mm/slub.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)mm/slub.c +0
-0@@ -3238,6 +3238,16 @@ struct detached_freelist {struct kmem_cache *s; };Quoted 7 lines
mm/slub.c struct detached_freelist { · line 3245
Quoted 5 lines
mm/slub.c +5
-0+static inline void free_nonslab_page (struct page *page) +{ +unsigned int order =compound_order (page); + +VM_BUG_ON_PAGE (!PageCompound (page), page);Could we add
WARN_ONhere, or we got nothing whenCONFIG_DEBUG_VMis disabled.
Quoted 2 lines
I don't have a strong opinion on this. Please send a patch with reasoning if you want
WARN_ON_ONCEhere.
Ok, we met a BUG_ON(!PageCompound(page)) in kfree() twice in lts4.4, we are still debugging it.
It's different to analyses due to no vmcore, and can't be reproduced.
WARN_ON() here could help us to notice the issue.
Also is there any experience or known fix/way to debug this kinds of issue? memory corruption?
Any suggestion will be appreciated, thanks.
Quoted 2 lines
.
Quoted 11 lines · Kefeng Wang
Quoted 5 lines · Shakeel Butt
Quoted 2 lines · Kefeng Wang
On 2021/7/28 23:53, Shakeel Butt wrote:
I don't have a strong opinion on this. Please send a patch with reasoning if you want
WARN_ON_ONCEhere.Ok, we met a BUG_ON(!PageCompound(page)) in
kfree()twice in lts4.4, we are still debugging it.
Quoted 6 lines
It's different to analyses due to no vmcore, and can't be reproduced.
WARN_ON()here could help us to notice the issue.Also is there any experience or known fix/way to debug this kinds of issue? memory corruption?
This would typically be a use-after-free/double-free - a problem of the slab user, not slab itself.
Quoted 1 line
Any suggestion will be appreciated, thanks.
debug_pagealloc could help to catch a use-after-free earlier
Quoted 3 lines
Quoted 3 lines
.
Quoted 9 lines · Vlastimil Babka
Quoted 8 lines · Kefeng Wang
Quoted 4 lines · Shakeel Butt
Quoted 1 line · Kefeng Wang
On 2021/7/28 23:53, Shakeel Butt wrote:
I don't have a strong opinion on this. Please send a patch with reasoning if you want
WARN_ON_ONCEhere.Ok, we met a BUG_ON(!PageCompound(page)) in
kfree()twice in lts4.4, we are still debugging it.
Quoted 8 lines
Quoted 6 lines
It's different to analyses due to no vmcore, and can't be reproduced.
WARN_ON()here could help us to notice the issue.Also is there any experience or known fix/way to debug this kinds of issue? memory corruption?
This would typically be a use-after-free/double-free - a problem of the slab user, not slab itself.
We enable KASAN to find whether or not there are some UAF/double free, no related
log for now.
Quoted 3 lines
Quoted 2 lines
Any suggestion will be appreciated, thanks.
debug_pagealloccould help to catch a use-after-free earlier
OK, will try this, thanks.
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
.
.