mm/slub: Improve data handling of krealloc() when orig_size is enabled
Subsystems (4)
v2 cover mm/slub: Improve data handling of krealloc() when orig_size is enabled
Danilo Krummrich's patch [1] raised one problem about krealloc() that its caller doesn't pass the old request size, say the object is 64 bytes kmalloc one, but caller originally only requested 48 bytes. Then when krealloc() shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when __GFP_ZERO is set).
Thus with slub debug redzone and object tracking enabled, parts of the object after krealloc() might contain redzone data instead of zeroes, which is violating the __GFP_ZERO guarantees. Good thing is in this case, kmalloc caches do have this 'orig_size' feature, which could be used to improve the situation here.
To make the 'orig_size' accurate, we adjust some kasan/slub meta data handling. Also add a slub kunit test case for krealloc().
This patchset has dependency over patches in both -mm tree and -slab trees, so it is written based on linux-next tree '20240910' version.
[1]. https://lore.kernel.org/lkml/[email protected]/
Thanks,
Feng
Changelog:
Since v1:
* Drop the patch changing generic kunit code from this patchset, and will send it separately.
* Separate the krealloc moving form slab_common.c to slub.c to a new patch for better review (Danilo/Vlastimil)
* Improve commit log and comments (Vlastimil/Danilo)
* Rework the kunit test case to remove its dependency over slub_debug (which is incomplete in v1) (Vlastimil)
* Add ack and review tag from developers.
Feng Tang (5):
mm/kasan: Don't store metadata inside kmalloc object when
slub_debug_orig_size is on
mm/slub: Consider kfence case for get_orig_size() mm/slub: Move krealloc() and related code to slub.c mm/slub: Improve redzone check and zeroing for krealloc() mm/slub, kunit: Add testcase for krealloc redzone and zeroing
lib/slub_kunit.c | 42 +++++++++++++++ mm/kasan/generic.c | 7 ++- mm/slab.h | 6 +++ mm/slab_common.c | 84 ------------------------------ mm/slub.c | 125 ++++++++++++++++++++++++++++++++++++++------- 5 files changed, 160 insertions(+), 104 deletions(-)
Signature · 1 line
2.34.1
PATCH v2 1/5 mm/kasan: Don't store metadata inside kmalloc object when slub_debug_orig_size is on
For a kmalloc object, when both kasan and slub redzone sanity check are enabled, they could both manipulate its data space like storing kasan free meta data and setting up kmalloc redzone, and may affect accuracy of that object's 'orig_size'.
As an accurate 'orig_size' will be needed by some function like krealloc() soon, save kasan's free meta data in slub's metadata area instead of inside object when 'orig_size' is enabled.
This will make it easier to maintain/understand the code. Size wise, when these two options are both enabled, the slub meta data space is already huge, and this just slightly increase the overall size.
--- mm/kasan/generic.c | 7 +++++-- mm/slab.h | 6 ++++++ mm/slub.c | 17 ----------------- 3 files changed, 11 insertions(+), 19 deletions(-)
mm/kasan/generic.c +5 -2
@@ -392,9 +392,12 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,* 1. Object is SLAB_TYPESAFE_BY_RCU, which means that it can * be touched after it was freed, or * 2. Object has a constructor, which means it's expected to -* retain its content until the next allocation. +* retain its content until the next allocation, or +* 3. It is from a kmalloc cache which enables the debug option +* to store original size. */ -if ((cache->flags &SLAB_TYPESAFE_BY_RCU ) || cache->ctor) { +if ((cache->flags &SLAB_TYPESAFE_BY_RCU ) || cache->ctor || +slub_debug_orig_size (cache)) { cache->kasan_info.free_meta_offset = *size; *size +=sizeof (struct kasan_free_meta ); goto free_meta_added;
mm/slab.h +6 -0
@@ -689,6 +689,12 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)void __check_heap_object (const void *ptr,unsigned long n,const struct slab *slab,bool to_user); +static inline bool slub_debug_orig_size (struct kmem_cache *s) +{ +return (kmem_cache_debug_flags (s,SLAB_STORE_USER ) && + (s->flags &SLAB_KMALLOC )); +} +#ifdef CONFIG_SLUB_DEBUG void skip_orig_size_check (struct kmem_cache *s,const void *object);#endif
mm/slub.c +0 -17
@@ -230,12 +230,6 @@ static inline bool kmem_cache_debug(struct kmem_cache *s)return kmem_cache_debug_flags (s,SLAB_DEBUG_FLAGS ); } -static inline bool slub_debug_orig_size (struct kmem_cache *s) -{ -return (kmem_cache_debug_flags (s,SLAB_STORE_USER ) && - (s->flags &SLAB_KMALLOC )); -} -void *fixup_red_left (struct kmem_cache *s,void *p) {if (kmem_cache_debug_flags (s,SLAB_RED_ZONE ))
@@ -760,21 +754,10 @@ static inline void set_orig_size(struct kmem_cache *s, void *object, unsigned int orig_size) { void *p = kasan_reset_tag(object); -unsigned int kasan_meta_size;if (!slub_debug_orig_size (s))return ; -/* -* KASAN can save its free meta data inside of the object at offset 0. -* If this meta data size is larger than 'orig_size', it will overlap -* the data redzone in [orig_size+1, object_size]. Thus, we adjust -* 'orig_size' to be as at least as big as KASAN's meta data. -*/ - kasan_meta_size =kasan_metadata_size (s, true); -if (kasan_meta_size > orig_size) - orig_size = kasan_meta_size; - p +=get_info_end (s); p +=sizeof (struct track ) *2 ;
Signature · 1 line
2.34.1
PATCH v2 2/5 mm/slub: Consider kfence case for get_orig_size()
In reply to Feng TangWhen 'orig_size' of kmalloc object is enabled by debug option, it should either contains the actual requested size or the cache's 'object_size'.
But it's not true if that object is a kfence-allocated one, and its 'orig_size' in metadata could be zero or other values. This is not a big issue for current 'orig_size' usage, as init_object() and check_object() during alloc/free process will be skipped for kfence addresses.
As 'orig_size' will be used by some function block like krealloc(), handle it by returning the 'object_size' in get_orig_size() for kfence addresses.
--- mm/slub.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/slub.c +1 -1
@@ -768,7 +768,7 @@ static inline unsigned int get_orig_size(struct kmem_cache *s, void *object) {void *p =kasan_reset_tag (object); -if (!slub_debug_orig_size (s)) +if (!slub_debug_orig_size (s) ||is_kfence_address (object))return s->object_size; p +=get_info_end (s);
Signature · 1 line
2.34.1
PATCH v2 3/5 mm/slub: Move krealloc() and related code to slub.c
In reply to Feng TangThis is a preparation for the following refactoring of krealloc(), for more efficient function calling as it will call some internal functions defined in slub.c.
--- mm/slab_common.c | 84 ------------------------------------------------ mm/slub.c | 84 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 84 insertions(+), 84 deletions(-)
mm/slab_common.c +0 -84
@@ -1185,90 +1185,6 @@ module_init(slab_proc_init);#endif /* CONFIG_SLUB_DEBUG */ -static __always_inline __realloc_size(2) void * -__do_krealloc(const void *p, size_t new_size, gfp_t flags) -{ - void *ret; - size_t ks; - - /* Check for double-free before calling ksize. */ - if (likely(!ZERO_OR_NULL_PTR(p))) { -if (!kasan_check_byte (p)) -return NULL ; - ks =ksize (p); - } else - ks =0 ; - -/* If the object still fits, repoison it precisely. */ -if (ks >= new_size) { -/* Zero out spare memory. */ -if (want_init_on_alloc (flags)) { -kasan_disable_current (); -memset ((void *)p + new_size,0 , ks - new_size); -kasan_enable_current (); - } - - p =kasan_krealloc ((void *)p, new_size, flags); -return (void *)p; - } - - ret =kmalloc_node_track_caller_noprof (new_size, flags,NUMA_NO_NODE , _RET_IP_); -if (ret && p) { -/* Disable KASAN checks as the object's redzone is accessed. */ -kasan_disable_current (); -memcpy (ret,kasan_reset_tag (p), ks); -kasan_enable_current (); - } - -return ret; -} - -/** -* krealloc - reallocate memory. The contents will remain unchanged. -* @p: object to reallocate memory for. -* @new_size: how many bytes of memory are required. -* @flags: the type of memory to allocate. -* -* If @p is %NULL, krealloc() behaves exactly like kmalloc(). If @new_size -* is 0 and @p is not a %NULL pointer, the object pointed to is freed. -* -* If __GFP_ZERO logic is requested, callers must ensure that, starting with the -* initial memory allocation, every subsequent call to this API for the same -* memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that -* __GFP_ZERO is not fully honored by this API. -* -* This is the case, since krealloc() only knows about the bucket size of an -* allocation (but not the exact size it was allocated with) and hence -* implements the following semantics for shrinking and growing buffers with -* __GFP_ZERO. -* -* new bucket -* 0 size size -* |--------|----------------| -* | keep | zero | -* -* In any case, the contents of the object pointed to are preserved up to the -* lesser of the new and old sizes. -* -* Return: pointer to the allocated memory or %NULL in case of error -*/ -void *krealloc_noprof (const void *p,size_t new_size,gfp_t flags) -{ -void *ret; - -if (unlikely (!new_size)) { -kfree (p); -return ZERO_SIZE_PTR ; - } - - ret =__do_krealloc (p, new_size, flags); -if (ret &&kasan_reset_tag (p) !=kasan_reset_tag (ret)) -kfree (p); - -return ret; -} -EXPORT_SYMBOL (krealloc_noprof); -/** * kfree_sensitive - Clear sensitive information in memory before freeing * @p: object to free memory of
mm/slub.c +84 -0
90 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v2 4/5 mm/slub: Improve redzone check and zeroing for krealloc()
In reply to Feng TangFor current krealloc(), one problem is its caller doesn't pass the old request size, say the object is 64 bytes kmalloc one, but caller may only requested 48 bytes. Then when krealloc() shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when __GFP_ZERO is set).
Thus with slub debug redzone and object tracking enabled, parts of the object after krealloc() might contain redzone data instead of zeroes, which is violating the __GFP_ZERO guarantees. Good thing is in this case, kmalloc caches do have this 'orig_size' feature. So solve the problem by utilize 'org_size' to do accurate data zeroing and preserving.
--- mm/slub.c | 54 ++++++++++++++++++++++++++++++++++++++---------------- 1 file changed, 38 insertions(+), 16 deletions(-)
mm/slub.c +38 -16
@@ -4717,33 +4717,51 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) {void *ret;size_t ks; +int orig_size =0 ; +struct kmem_cache *s; -/* Check for double-free before calling ksize. */ +/* Check for double-free. */ if (likely (!ZERO_OR_NULL_PTR (p))) {if (!kasan_check_byte (p))return NULL ; - ks =ksize (p); + + s =virt_to_cache (p); + orig_size =get_orig_size (s, (void *)p); + ks = s->object_size; }else ks =0 ; -/* If the object still fits, repoison it precisely. */ -if (ks >= new_size) { -/* Zero out spare memory. */ -if (want_init_on_alloc (flags)) { -kasan_disable_current (); +/* If the object doesn't fit, allocate a bigger one */ +if (new_size > ks) + goto alloc_new; + +/* Zero out spare memory. */ +if (want_init_on_alloc (flags)) { +kasan_disable_current (); +if (orig_size < new_size) +memset ((void *)p + orig_size,0 , new_size - orig_size); +else memset ((void *)p + new_size,0 , ks - new_size); -kasan_enable_current (); - } +kasan_enable_current (); + } - p =kasan_krealloc ((void *)p, new_size, flags); -return (void *)p; +if (slub_debug_orig_size (s) && !is_kfence_address (p)) { +set_orig_size (s, (void *)p, new_size); +if (s->flags &SLAB_RED_ZONE && new_size < ks) +memset_no_sanitize_memory ((void *)p + new_size, +SLUB_RED_ACTIVE , ks - new_size); } + p =kasan_krealloc ((void *)p, new_size, flags); +return (void *)p; + +alloc_new: ret =kmalloc_node_track_caller_noprof (new_size, flags,NUMA_NO_NODE , _RET_IP_);if (ret && p) {/* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current (); -memcpy (ret,kasan_reset_tag (p), ks); +if (orig_size) +memcpy (ret,kasan_reset_tag (p), orig_size);kasan_enable_current (); }
@@ -4764,16 +4782,20 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)* memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that * __GFP_ZERO is not fully honored by this API. * -* This is the case, since krealloc() only knows about the bucket size of an -* allocation (but not the exact size it was allocated with) and hence -* implements the following semantics for shrinking and growing buffers with -* __GFP_ZERO. +* When slub_debug_orig_size() is off, krealloc() only knows about the bucket +* size of an allocation (but not the exact size it was allocated with) and +* hence implements the following semantics for shrinking and growing buffers +* with __GFP_ZERO. * * new bucket * 0 size size * |--------|----------------| * | keep | zero | * +* Otherwise, the original allocation size 'orig_size' could be used to +* precisely clear the requested size, and the new size will also be stored +* as the new 'orig_size'. +* * In any case, the contents of the object pointed to are preserved up to the * lesser of the new and old sizes. *
Signature · 1 line
2.34.1
Re: [PATCH v2 4/5] mm/slub: Improve redzone check and zeroing for krealloc()
Hi Feng,
kernel test robot noticed the following build warnings:
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Feng-Tang/mm-kasan-Don-t-store-metadata-inside-kmalloc-object-when-slub_debug_orig_size-is-on/20240911-144724 base: next-20240910 patch link: https://lore.kernel.org/r/20240911064535.557650-5-feng.tang%40intel.com patch subject: [PATCH v2 4/5] mm/slub: Improve redzone check and zeroing for krealloc() config: i386-randconfig-141-20240912 (https://download.01.org/0day-ci/archive/20240912/[email protected]/config) compiler: clang version 18.1.8 (https://github.com/llvm/llvm-project 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]> | Reported-by: Dan Carpenter <[email protected]> | Closes: https://lore.kernel.org/r/[email protected]/
smatch warnings:mm/slub.c:4748 __do_krealloc() error: uninitialized symbol 's'.
vim +/s +4748 mm/slub.c
474fff8e68012d Feng Tang 2024-09-11 4715 static __always_inline __realloc_size(2) void *
474fff8e68012d Feng Tang 2024-09-11 4716 __do_krealloc(const void *p, size_t new_size, gfp_t flags)
474fff8e68012d Feng Tang 2024-09-11 4717 {
474fff8e68012d Feng Tang 2024-09-11 4718 void *ret;
474fff8e68012d Feng Tang 2024-09-11 4719 size_t ks;
89a7731cc91033 Feng Tang 2024-09-11 4720 int orig_size = 0;
89a7731cc91033 Feng Tang 2024-09-11 4721 struct kmem_cache *s;
474fff8e68012d Feng Tang 2024-09-11 4722
89a7731cc91033 Feng Tang 2024-09-11 4723 /* Check for double-free. */
474fff8e68012d Feng Tang 2024-09-11 4724 if (likely(!ZERO_OR_NULL_PTR(p))) {
474fff8e68012d Feng Tang 2024-09-11 4725 if (!kasan_check_byte(p))
474fff8e68012d Feng Tang 2024-09-11 4726 return NULL;
89a7731cc91033 Feng Tang 2024-09-11 4727
89a7731cc91033 Feng Tang 2024-09-11 4728 s = virt_to_cache(p);
89a7731cc91033 Feng Tang 2024-09-11 4729 orig_size = get_orig_size(s, (void *)p);
89a7731cc91033 Feng Tang 2024-09-11 4730 ks = s->object_size;
474fff8e68012d Feng Tang 2024-09-11 4731 } else
474fff8e68012d Feng Tang 2024-09-11 4732 ks = 0;ks is zero
474fff8e68012d Feng Tang 2024-09-11 4733 89a7731cc91033 Feng Tang 2024-09-11 4734 /* If the object doesn't fit, allocate a bigger one */ 89a7731cc91033 Feng Tang 2024-09-11 4735 if (new_size > ks)
Assume new_size is also zero so this is false
89a7731cc91033 Feng Tang 2024-09-11 4736 goto alloc_new;
89a7731cc91033 Feng Tang 2024-09-11 4737
474fff8e68012d Feng Tang 2024-09-11 4738 /* Zero out spare memory. */
474fff8e68012d Feng Tang 2024-09-11 4739 if (want_init_on_alloc(flags)) {
474fff8e68012d Feng Tang 2024-09-11 4740 kasan_disable_current();
89a7731cc91033 Feng Tang 2024-09-11 4741 if (orig_size < new_size)
89a7731cc91033 Feng Tang 2024-09-11 4742 memset((void *)p + orig_size, 0, new_size - orig_size);
89a7731cc91033 Feng Tang 2024-09-11 4743 else
474fff8e68012d Feng Tang 2024-09-11 4744 memset((void *)p + new_size, 0, ks - new_size);
474fff8e68012d Feng Tang 2024-09-11 4745 kasan_enable_current();
474fff8e68012d Feng Tang 2024-09-11 4746 }
474fff8e68012d Feng Tang 2024-09-11 4747
89a7731cc91033 Feng Tang 2024-09-11 @4748 if (slub_debug_orig_size(s) && !is_kfence_address(p)) {
^Uninitialized in that case
89a7731cc91033 Feng Tang 2024-09-11 4749 set_orig_size(s, (void *)p, new_size); 89a7731cc91033 Feng Tang 2024-09-11 4750 if (s->flags & SLAB_RED_ZONE && new_size < ks)
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH v2 4/5] mm/slub: Improve redzone check and zeroing for krealloc()
Hi Dan,
Thanks for the report!
Quoted 43 lines · Dan Carpenter
Hi Feng,
kernel test robot noticed the following build warnings:
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Feng-Tang/mm-kasan-Don-t-store-metadata-inside-kmalloc-object-when-slub_debug_orig_size-is-on/20240911-144724 base: next-20240910 patch link: https://lore.kernel.org/r/20240911064535.557650-5-feng.tang%40intel.com patch subject: [PATCH v2 4/5] mm/slub: Improve redzone check and zeroing for krealloc() config: i386-randconfig-141-20240912 (https://download.01.org/0day-ci/archive/20240912/[email protected]/config) compiler: clang version 18.1.8 (https://github.com/llvm/llvm-project 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]> | Reported-by: Dan Carpenter <[email protected]> | Closes: https://lore.kernel.org/r/[email protected]/smatch warnings:
mm/slub.c:4748__do_krealloc()error: uninitialized symbol 's'.vim +/s +4748
mm/slub.c474fff8e68012d Feng Tang 2024-09-11 4715 static __always_inline __realloc_size(2) void * 474fff8e68012d Feng Tang 2024-09-11 4716 __do_krealloc(const void *p, size_t new_size, gfp_t flags) 474fff8e68012d Feng Tang 2024-09-11 4717 { 474fff8e68012d Feng Tang 2024-09-11 4718 void *ret; 474fff8e68012d Feng Tang 2024-09-11 4719 size_t ks; 89a7731cc91033 Feng Tang 2024-09-11 4720 int orig_size = 0; 89a7731cc91033 Feng Tang 2024-09-11 4721 struct kmem_cache *s; 474fff8e68012d Feng Tang 2024-09-11 4722 89a7731cc91033 Feng Tang 2024-09-11 4723 /* Check for double-free. */ 474fff8e68012d Feng Tang 2024-09-11 4724 if (likely(!ZERO_OR_NULL_PTR(p))) { 474fff8e68012d Feng Tang 2024-09-11 4725 if (!kasan_check_byte(p)) 474fff8e68012d Feng Tang 2024-09-11 4726 return NULL; 89a7731cc91033 Feng Tang 2024-09-11 4727 89a7731cc91033 Feng Tang 2024-09-11 4728 s = virt_to_cache(p); 89a7731cc91033 Feng Tang 2024-09-11 4729 orig_size = get_orig_size(s, (void *)p); 89a7731cc91033 Feng Tang 2024-09-11 4730 ks = s->object_size; 474fff8e68012d Feng Tang 2024-09-11 4731 } else 474fff8e68012d Feng Tang 2024-09-11 4732 ks = 0;
Quoted 7 lines
ks is zero
474fff8e68012d Feng Tang 2024-09-11 4733 89a7731cc91033 Feng Tang 2024-09-11 4734 /* If the object doesn't fit, allocate a bigger one */ 89a7731cc91033 Feng Tang 2024-09-11 4735 if (new_size > ks)Assume
new_sizeis also zero so this is false
Yes, this is possible, though unlikely :)
Below patch should fix it:
---
mm/slub.c +3 -2
@@ -4728,8 +4728,9 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags) s =virt_to_cache (p); orig_size =get_orig_size (s, (void *)p); ks = s->object_size; - }else - ks =0 ; + }else { + goto alloc_new; + }/* If the object doesn't fit, allocate a bigger one */ if (new_size > ks)
Thanks,
Feng
Quoted 24 lines
89a7731cc91033 Feng Tang 2024-09-11 4736 goto alloc_new; 89a7731cc91033 Feng Tang 2024-09-11 4737 474fff8e68012d Feng Tang 2024-09-11 4738 /* Zero out spare memory. */ 474fff8e68012d Feng Tang 2024-09-11 4739 if (want_init_on_alloc(flags)) { 474fff8e68012d Feng Tang 2024-09-11 4740 kasan_disable_current(); 89a7731cc91033 Feng Tang 2024-09-11 4741 if (orig_size < new_size) 89a7731cc91033 Feng Tang 2024-09-11 4742 memset((void *)p + orig_size, 0, new_size - orig_size); 89a7731cc91033 Feng Tang 2024-09-11 4743 else 474fff8e68012d Feng Tang 2024-09-11 4744 memset((void *)p + new_size, 0, ks - new_size); 474fff8e68012d Feng Tang 2024-09-11 4745 kasan_enable_current(); 474fff8e68012d Feng Tang 2024-09-11 4746 } 474fff8e68012d Feng Tang 2024-09-11 4747 89a7731cc91033 Feng Tang 2024-09-11 @4748 if (slub_debug_orig_size(s) && !is_kfence_address(p)) { ^Uninitialized in that case
89a7731cc91033 Feng Tang 2024-09-11 4749 set_orig_size(s, (void *)p, new_size); 89a7731cc91033 Feng Tang 2024-09-11 4750 if (s->flags & SLAB_RED_ZONE && new_size < ks)Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH v2 5/5 mm/slub, kunit: Add testcase for krealloc redzone and zeroing
In reply to Feng TangDanilo Krummrich raised issue about krealloc+GFP_ZERO [1], and Vlastimil suggested to add some test case which can sanity test the kmalloc-redzone and zeroing by utilizing the kmalloc's 'orig_size' debug feature.
It covers the grow and shrink case of krealloc() re-using current kmalloc object, and the case of re-allocating a new bigger object.
[1]. https://lore.kernel.org/lkml/[email protected]/
---
Hi Danilo,
I keep your Reviewed-By tag, as I think this v2 mostly changes what kmalloc slab to be used. Let me know if you want it dropped, thanks.
lib/slub_kunit.c | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+)
lib/slub_kunit.c +42 -0
@@ -186,6 +186,47 @@ static void test_leak_destroy(struct kunit *test)KUNIT_EXPECT_EQ (test,1 , slab_errors); } +static void test_krealloc_redzone_zeroing (struct kunit *test) +{ +u8 *p; +int i; +struct kmem_cache *s =test_kmem_cache_create ("TestSlub_krealloc" ,64 , +SLAB_KMALLOC |SLAB_STORE_USER |SLAB_RED_ZONE ); + + p =__kmalloc_cache_noprof (s,GFP_KERNEL ,48 ); +memset (p,0xff ,48 ); + +kasan_disable_current (); +OPTIMIZER_HIDE_VAR (p); + +/* Test shrink */ + p =krealloc (p,40 ,GFP_KERNEL | __GFP_ZERO); +for (i =40 ; i <64 ; i++) +KUNIT_EXPECT_EQ (test, p[i],SLUB_RED_ACTIVE ); + +/* Test grow within the same 64B kmalloc object */ + p =krealloc (p,56 ,GFP_KERNEL | __GFP_ZERO); +for (i =40 ; i <56 ; i++) +KUNIT_EXPECT_EQ (test, p[i],0 ); +for (i =56 ; i <64 ; i++) +KUNIT_EXPECT_EQ (test, p[i],SLUB_RED_ACTIVE ); + +validate_slab_cache (s); +KUNIT_EXPECT_EQ (test,0 , slab_errors); + +memset (p,0xff ,56 ); +/* Test grow with allocating a bigger 128B object */ + p =krealloc (p,112 ,GFP_KERNEL | __GFP_ZERO); +for (i =0 ; i <56 ; i++) +KUNIT_EXPECT_EQ (test, p[i],0xff ); +for (i =56 ; i <112 ; i++) +KUNIT_EXPECT_EQ (test, p[i],0 ); + +kfree (p); +kasan_enable_current (); +kmem_cache_destroy (s); +} +static int test_init (struct kunit *test) { slab_errors =0 ;
@@ -208,6 +249,7 @@ static struct kunit_case test_cases[] = {KUNIT_CASE (test_kmalloc_redzone_access),KUNIT_CASE (test_kfree_rcu),KUNIT_CASE (test_leak_destroy), + KUNIT_CASE(test_krealloc_redzone_zeroing), {} };
Signature · 1 line
2.34.1
Quoted 13 lines · Feng Tang
Danilo Krummrich's patch [1] raised one problem about
krealloc()that its caller doesn't pass the old request size, say the object is 64 bytes kmalloc one, but caller originally only requested 48 bytes. Then whenkrealloc()shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when__GFP_ZEROis set).Thus with slub debug redzone and object tracking enabled, parts of the object after
krealloc()might contain redzone data instead of zeroes, which is violating the__GFP_ZEROguarantees. Good thing is in this case, kmalloc caches do have this 'orig_size' feature, which could be used to improve the situation here.
Quoted 7 lines
To make the '
orig_size' accurate, we adjust some kasan/slub meta data handling. Also add a slub kunit test case forkrealloc().This patchset has dependency over patches in both -mm tree and -slab trees, so it is written based on linux-next tree '20240910' version.
Thanks, added to slab/for-next
Quoted 31 lines
Thanks,
FengChangelog:
Since v1:
* Drop the patch changing generic kunit code from this patchset, and will send it separately.
* Separate the krealloc moving formslab_common.ctoslub.cto a new patch for better review (Danilo/Vlastimil)
* Improve commit log and comments (Vlastimil/Danilo)
* Rework the kunit test case to remove its dependency overslub_debug(which is incomplete in v1) (Vlastimil)
* Add ack and review tag from developers.Feng Tang (5):
mm/kasan: Don't store metadata inside kmalloc object when
slub_debug_orig_sizeis on
mm/slub: Consider kfence case forget_orig_size()mm/slub: Movekrealloc()and related code toslub.cmm/slub: Improve redzone check and zeroing forkrealloc()mm/slub, kunit: Add testcase for krealloc redzone and zeroinglib/slub_kunit.c | 42 +++++++++++++++ mm/kasan/generic.c | 7 ++- mm/slab.h | 6 +++ mm/slab_common.c | 84 ------------------------------ mm/slub.c | 125 ++++++++++++++++++++++++++++++++++++++------- 5 files changed, 160 insertions(+), 104 deletions(-)
Quoted 18 lines · Vlastimil Babka
Quoted 16 lines · Feng Tang
Danilo Krummrich's patch [1] raised one problem about
krealloc()that its caller doesn't pass the old request size, say the object is 64 bytes kmalloc one, but caller originally only requested 48 bytes. Then whenkrealloc()shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when__GFP_ZEROis set).Thus with slub debug redzone and object tracking enabled, parts of the object after
krealloc()might contain redzone data instead of zeroes, which is violating the__GFP_ZEROguarantees. Good thing is in this case, kmalloc caches do have this 'orig_size' feature, which could be used to improve the situation here.To make the '
orig_size' accurate, we adjust some kasan/slub meta data handling. Also add a slub kunit test case forkrealloc().
Quoted 6 lines
Quoted 4 lines
This patchset has dependency over patches in both -mm tree and -slab trees, so it is written based on linux-next tree '20240910' version.
Thanks, added to slab/for-next
This series just hit -next, and we're seeing several "KFENCE: memory corruption ...". Here's one:
https://lore.kernel.org/all/[email protected]/
One more (no link):
Quoted 8 lines
==================================================================
BUG: KFENCE: memory corruption in xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051Corrupted memory at
0xffff88823bf5a0d0[0x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x00] (in kfence-#172):xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051 xfs_iext_destroy+0x66/0x100 fs/xfs/libxfs/xfs_iext_tree.c:1062 xfs_inode_free_callback+0x91/0x1d0 fs/xfs/xfs_icache.c:145 rcu_do_batch kernel/rcu/tree.c:2567 [inline]
[...]
Quoted 7 lines
kfence-#172: 0xffff88823bf5a000-0xffff88823bf5a0cf, size=208, cache=kmalloc-256
allocated by task 5494 on cpu 0 at 101.266046s (0.409225s ago):
__do_krealloc mm/slub.c:4784 [inline] krealloc_noprof+0xd6/0x2e0 mm/slub.c:4838 xfs_iext_realloc_root fs/xfs/libxfs/xfs_iext_tree.c:613 [inline]
[...]
Quoted 5 lines
freed by task 16 on cpu 0 at 101.573936s (0.186416s ago):
xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051 xfs_iext_destroy+0x66/0x100 fs/xfs/libxfs/xfs_iext_tree.c:1062 xfs_inode_free_callback+0x91/0x1d0 fs/xfs/xfs_icache.c:145
[...]
Quoted 4 lines
CPU: 0 UID: 0 PID: 16 Comm: ksoftirqd/0 Not tainted 6.12.0-rc1-next-20241003-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024 ==================================================================
Unfortunately there's no reproducer yet it seems. Unless it's immediately obvious to say what's wrong, is it possible to take this series out of -next to confirm this series is causing the memory corruptions? Syzbot should then stop finding these crashes.
Thanks,
-- Marco
Quoted 56 lines · Marco Elver
Quoted 18 lines · Vlastimil Babka
Quoted 16 lines · Feng Tang
Danilo Krummrich's patch [1] raised one problem about
krealloc()that its caller doesn't pass the old request size, say the object is 64 bytes kmalloc one, but caller originally only requested 48 bytes. Then whenkrealloc()shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when__GFP_ZEROis set).Thus with slub debug redzone and object tracking enabled, parts of the object after
krealloc()might contain redzone data instead of zeroes, which is violating the__GFP_ZEROguarantees. Good thing is in this case, kmalloc caches do have this 'orig_size' feature, which could be used to improve the situation here.To make the '
orig_size' accurate, we adjust some kasan/slub meta data handling. Also add a slub kunit test case forkrealloc().Quoted 6 lines
Quoted 4 lines
This patchset has dependency over patches in both -mm tree and -slab trees, so it is written based on linux-next tree '20240910' version.
Thanks, added to slab/for-next
This series just hit -next, and we're seeing several "KFENCE: memory corruption ...". Here's one:
https://lore.kernel.org/all/[email protected]/One more (no link):
Quoted 8 lines
==================================================================
BUG: KFENCE: memory corruption in xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051Corrupted memory at
0xffff88823bf5a0d0[0x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x00] (in kfence-#172):xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051 xfs_iext_destroy+0x66/0x100 fs/xfs/libxfs/xfs_iext_tree.c:1062 xfs_inode_free_callback+0x91/0x1d0 fs/xfs/xfs_icache.c:145 rcu_do_batch kernel/rcu/tree.c:2567 [inline][...]
Quoted 7 lines
kfence-#172: 0xffff88823bf5a000-0xffff88823bf5a0cf, size=208, cache=kmalloc-256
allocated by task 5494 on cpu 0 at 101.266046s (0.409225s ago):
__do_krealloc mm/slub.c:4784 [inline] krealloc_noprof+0xd6/0x2e0 mm/slub.c:4838 xfs_iext_realloc_root fs/xfs/libxfs/xfs_iext_tree.c:613 [inline][...]
Quoted 5 lines
freed by task 16 on cpu 0 at 101.573936s (0.186416s ago):
xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051 xfs_iext_destroy+0x66/0x100 fs/xfs/libxfs/xfs_iext_tree.c:1062 xfs_inode_free_callback+0x91/0x1d0 fs/xfs/xfs_icache.c:145[...]
Quoted 1 line
Quoted 8 lines
Quoted 3 lines
CPU: 0 UID: 0 PID: 16 Comm: ksoftirqd/0 Not tainted 6.12.0-rc1-next-20241003-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024 ==================================================================Unfortunately there's no reproducer yet it seems. Unless it's immediately obvious to say what's wrong, is it possible to take this series out of -next to confirm this series is causing the memory corruptions? Syzbot should then stop finding these crashes.
I think it's commit d0a38fad51cc7 doing in __do_krealloc()
- ks = ksize(p);
+
+ s = virt_to_cache(p); + orig_size = get_orig_size(s, (void *)p); + ks = s->object_size;
so for kfence objects we don't get their actual allocation size but the potentially larger bucket size?
I guess we could do:
ks = kfence_ksize(p) ?: s->object_size;
?
Quoted 2 lines
Thanks,
-- Marco
Quoted 10 lines · Vlastimil Babka
On 10/4/24 08:44, Marco Elver wrote:
I think it's commit
d0a38fad51cc7doing in__do_krealloc()(quoted diff) +4
-1- ks = ksize(p); + + s = virt_to_cache(p); + orig_size = get_orig_size(s, (void *)p); + ks = s->object_size;
Quoted 8 lines
so for kfence objects we don't get their actual allocation size but the potentially larger bucket size?
I guess we could do:
ks =
kfence_ksize(p)?:s->object_size;?
Hmm this probably is not the whole story, we also have:
- memcpy(ret, kasan_reset_tag(p), ks);
+ if (orig_size)
+ memcpy(ret, kasan_reset_tag(p), orig_size);
orig_size for kfence will be again s->object_size so the memcpy might be a (read) buffer overflow from a kfence allocation.
I think get_orig_size() should perhaps return kfence_ksize(p) for kfence allocations, in addition to the change above.
Or alternatively we don't change get_orig_size() (in a different commit) at all, but __do_krealloc() will have an "if is_kfence_address()" that sets both orig_size and ks to kfence_ksize(p) appropriately. That might be easier to follow.
But either way means rewriting 2 commits. I think it's indeed better to drop the series now from -next and submit a v3.
Vlastimil
Quoted 3 lines
Quoted 2 lines
Thanks,
-- Marco
Quoted 32 lines · Vlastimil Babka
Quoted 10 lines · Vlastimil Babka
On 10/4/24 08:44, Marco Elver wrote:
I think it's commit
d0a38fad51cc7doing in__do_krealloc()(quoted diff) +4
-1- ks = ksize(p); + + s = virt_to_cache(p); + orig_size = get_orig_size(s, (void *)p); + ks = s->object_size;Quoted 8 lines
so for kfence objects we don't get their actual allocation size but the potentially larger bucket size?
I guess we could do:
ks =
kfence_ksize(p)?:s->object_size;?
Hmm this probably is not the whole story, we also have:
(quoted diff) +2
-1- memcpy(ret, kasan_reset_tag(p), ks); + if (orig_size) + memcpy(ret, kasan_reset_tag(p), orig_size);
orig_sizefor kfence will be agains->object_sizeso the memcpy might be a (read) buffer overflow from a kfence allocation.I think
get_orig_size()should perhaps returnkfence_ksize(p)for kfence allocations, in addition to the change above.
Quoted 7 lines
Or alternatively we don't change
get_orig_size()(in a different commit) at all, but__do_krealloc()will have an "ifis_kfence_address()" that sets bothorig_sizeand ks tokfence_ksize(p)appropriately. That might be easier to follow.But either way means rewriting 2 commits. I think it's indeed better to drop the series now from -next and submit a v3.
Yes, we can revert now. Sorry for the inconvenience.
Thanks,
Feng
Quoted 29 lines · Vlastimil Babka
Quoted 10 lines · Vlastimil Babka
On 10/4/24 08:44, Marco Elver wrote:
I think it's commit
d0a38fad51cc7doing in__do_krealloc()(quoted diff) +4
-1- ks = ksize(p); + + s = virt_to_cache(p); + orig_size = get_orig_size(s, (void *)p); + ks = s->object_size;Quoted 8 lines
so for kfence objects we don't get their actual allocation size but the potentially larger bucket size?
I guess we could do:
ks =
kfence_ksize(p)?:s->object_size;?
Hmm this probably is not the whole story, we also have:
(quoted diff) +2
-1- memcpy(ret, kasan_reset_tag(p), ks); + if (orig_size) + memcpy(ret, kasan_reset_tag(p), orig_size);
orig_sizefor kfence will be agains->object_sizeso the memcpy might be a (read) buffer overflow from a kfence allocation.
Quoted 7 lines
I think
get_orig_size()should perhaps returnkfence_ksize(p)for kfence allocations, in addition to the change above.Or alternatively we don't change
get_orig_size()(in a different commit) at all, but__do_krealloc()will have an "ifis_kfence_address()" that sets bothorig_sizeand ks tokfence_ksize(p)appropriately. That might be easier to follow.
Thanks for the suggestion!
As there were error report about the NULL slab for big kmalloc object, how about the following code for
__do_krealloc(const void *p, size_t new_size, gfp_t flags)
{
void *ret;
size_t ks = 0;
int orig_size = 0;
struct kmem_cache *s = NULL;
/* Check for double-free. */
if (likely(!ZERO_OR_NULL_PTR(p))) {
if (!kasan_check_byte(p))
return NULL;
ks = ksize(p);
/* Some objects have no orig_size, like big kmalloc case */
if (is_kfence_address(p)) {
orig_size = kfence_ksize(p);
} else if (virt_to_slab(p)) {
s = virt_to_cache(p);
orig_size = get_orig_size(s, (void *)p);
}
} else {
goto alloc_new;
}
/* If the object doesn't fit, allocate a bigger one */
if (new_size > ks)
goto alloc_new;
/* Zero out spare memory. */
if (want_init_on_alloc(flags)) {
kasan_disable_current();
if (orig_size && orig_size < new_size)
memset((void *)p + orig_size, 0, new_size - orig_size);
else
memset((void *)p + new_size, 0, ks - new_size);
kasan_enable_current();
}
/* Setup kmalloc redzone when needed */
if (s && slub_debug_orig_size(s) && !is_kfence_address(p)) {
set_orig_size(s, (void *)p, new_size);
if (s->flags & SLAB_RED_ZONE && new_size < ks)
memset_no_sanitize_memory((void *)p + new_size,
SLUB_RED_ACTIVE, ks - new_size);
}
p = kasan_krealloc((void *)p, new_size, flags);
return (void *)p;alloc_new:
ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_);
if (ret && p) {
/* Disable KASAN checks as the object's redzone is accessed. */
kasan_disable_current();
memcpy(ret, kasan_reset_tag(p), orig_size ?: ks);
kasan_enable_current();
}
return ret;
}I've run it with the reproducer of syzbot, so far the issue hasn't been reproduced on my local machine.
Thanks,
Feng
Quoted 10 lines
But either way means rewriting 2 commits. I think it's indeed better to drop the series now from -next and submit a v3.
Vlastimil
Quoted 3 lines
Quoted 2 lines
Thanks,
-- Marco
Quoted 7 lines · Feng Tang
On Fri, Oct 04, 2024 at 05:52:10PM +0800, Vlastimil Babka wrote:
Thanks for the suggestion!As there were error report about the NULL slab for big kmalloc object, how about the following code for
__do_krealloc(const void *p, size_t new_size, gfp_t flags)
Quoted 12 lines
{
void *ret;
size_t ks = 0;
int orig_size = 0;
struct kmem_cache *s = NULL;
/* Check for double-free. */
if (likely(!ZERO_OR_NULL_PTR(p))) {
if (!kasan_check_byte(p))
return NULL;
ks = ksize(p);I think this will result in __ksize() doing
skip_orig_size_check(folio_slab(folio)->slab_cache, object);
and we don't want that?
Also the checks below repeat some of the checks of ksize().
So I think in __do_krealloc() we should do things manually to determine ks and not call ksize(). Just not break any of the cases ksize() handles (kfence, large kmalloc).
Quoted 65 lines
/* Some objects have no orig_size, like big kmalloc case */ if (is_kfence_address(p)) { orig_size = kfence_ksize(p); } else if (virt_to_slab(p)) { s = virt_to_cache(p); orig_size = get_orig_size(s, (void *)p); } } else { goto alloc_new; } /* If the object doesn't fit, allocate a bigger one */ if (new_size > ks) goto alloc_new; /* Zero out spare memory. */ if (want_init_on_alloc(flags)) { kasan_disable_current(); if (orig_size && orig_size < new_size) memset((void *)p + orig_size, 0, new_size - orig_size); else memset((void *)p + new_size, 0, ks - new_size); kasan_enable_current(); } /* Setup kmalloc redzone when needed */ if (s && slub_debug_orig_size(s) && !is_kfence_address(p)) { set_orig_size(s, (void *)p, new_size); if (s->flags & SLAB_RED_ZONE && new_size < ks) memset_no_sanitize_memory((void *)p + new_size, SLUB_RED_ACTIVE, ks - new_size); } p = kasan_krealloc((void *)p, new_size, flags); return (void *)p;
alloc_new:ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); memcpy(ret, kasan_reset_tag(p), orig_size ?: ks); kasan_enable_current(); } return ret; }I've run it with the reproducer of syzbot, so far the issue hasn't been reproduced on my local machine.
Thanks,
FengQuoted 10 lines
But either way means rewriting 2 commits. I think it's indeed better to drop the series now from -next and submit a v3.
Vlastimil
Quoted 3 lines
Quoted 2 lines
Thanks,
-- Marco
Quoted 19 lines · Vlastimil Babka
Quoted 18 lines · Feng Tang
On Fri, Oct 04, 2024 at 05:52:10PM +0800, Vlastimil Babka wrote:
Thanks for the suggestion!As there were error report about the NULL slab for big kmalloc object, how about the following code for
__do_krealloc(const void *p, size_t new_size, gfp_t flags){ void *ret; size_t ks = 0; int orig_size = 0; struct kmem_cache *s = NULL; /* Check for double-free. */ if (likely(!ZERO_OR_NULL_PTR(p))) { if (!kasan_check_byte(p)) return NULL;
Quoted 5 lines
Quoted 1 line
ks = ksize(p);I think this will result in
__ksize()doing
skip_orig_size_check(folio_slab(folio)->slab_cache, object);
and we don't want that?
I think that's fine. As later code will re-set the orig_size anyway.
Quoted 1 line
Also the checks below repeat some of the checks of
ksize().
Yes, there is some redundancy, mostly the virt_to_slab()
Quoted 3 lines
So I think in
__do_krealloc()we should do things manually to determine ks and not callksize(). Just not break any of the casesksize()handles (kfence, large kmalloc).
OK, originally I tried not to expose internals of __ksize(). Let me try this way.
Thanks,
Feng
Quoted 49 lines
Quoted 49 lines
/* Some objects have no orig_size, like big kmalloc case */ if (is_kfence_address(p)) { orig_size = kfence_ksize(p); } else if (virt_to_slab(p)) { s = virt_to_cache(p); orig_size = get_orig_size(s, (void *)p); } } else { goto alloc_new; } /* If the object doesn't fit, allocate a bigger one */ if (new_size > ks) goto alloc_new; /* Zero out spare memory. */ if (want_init_on_alloc(flags)) { kasan_disable_current(); if (orig_size && orig_size < new_size) memset((void *)p + orig_size, 0, new_size - orig_size); else memset((void *)p + new_size, 0, ks - new_size); kasan_enable_current(); } /* Setup kmalloc redzone when needed */ if (s && slub_debug_orig_size(s) && !is_kfence_address(p)) { set_orig_size(s, (void *)p, new_size); if (s->flags & SLAB_RED_ZONE && new_size < ks) memset_no_sanitize_memory((void *)p + new_size, SLUB_RED_ACTIVE, ks - new_size); } p = kasan_krealloc((void *)p, new_size, flags); return (void *)p;
alloc_new:ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); memcpy(ret, kasan_reset_tag(p), orig_size ?: ks); kasan_enable_current(); } return ret; }
Quoted 20 lines · Feng Tang
Quoted 19 lines · Vlastimil Babka
Quoted 18 lines · Feng Tang
On Fri, Oct 04, 2024 at 05:52:10PM +0800, Vlastimil Babka wrote:
Thanks for the suggestion!As there were error report about the NULL slab for big kmalloc object, how about the following code for
__do_krealloc(const void *p, size_t new_size, gfp_t flags){ void *ret; size_t ks = 0; int orig_size = 0; struct kmem_cache *s = NULL; /* Check for double-free. */ if (likely(!ZERO_OR_NULL_PTR(p))) { if (!kasan_check_byte(p)) return NULL;
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
ks = ksize(p);I think this will result in
__ksize()doing
skip_orig_size_check(folio_slab(folio)->slab_cache, object);
and we don't want that?I think that's fine. As later code will re-set the
orig_sizeanyway.
But you also read it first.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
/* Some objects have no orig_size, like big kmalloc case */ if (is_kfence_address(p)) { orig_size = kfence_ksize(p); } else if (virt_to_slab(p)) { s = virt_to_cache(p); orig_size = get_orig_size(s, (void *)p);
here.
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
}Also the checks below repeat some of the checks of
ksize().
Quoted 8 lines
Yes, there is some redundancy, mostly the
virt_to_slab()Quoted 3 lines
So I think in
__do_krealloc()we should do things manually to determine ks and not callksize(). Just not break any of the casesksize()handles (kfence, large kmalloc).OK, originally I tried not to expose internals of
__ksize(). Let me try this way.
ksize() makes assumptions that a user outside of slab itself is calling it.
But we (well mostly Kees) also introduced kmalloc_size_roundup() to avoid querying ksize() for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if the skip_orig_size_check() mechanism can be removed now?
Still I think __do_krealloc() should rather do its own thing and not call ksize().
Quoted 45 lines
Thanks,
FengQuoted 42 lines
Quoted 42 lines
} else { goto alloc_new; } /* If the object doesn't fit, allocate a bigger one */ if (new_size > ks) goto alloc_new; /* Zero out spare memory. */ if (want_init_on_alloc(flags)) { kasan_disable_current(); if (orig_size && orig_size < new_size) memset((void *)p + orig_size, 0, new_size - orig_size); else memset((void *)p + new_size, 0, ks - new_size); kasan_enable_current(); } /* Setup kmalloc redzone when needed */ if (s && slub_debug_orig_size(s) && !is_kfence_address(p)) { set_orig_size(s, (void *)p, new_size); if (s->flags & SLAB_RED_ZONE && new_size < ks) memset_no_sanitize_memory((void *)p + new_size, SLUB_RED_ACTIVE, ks - new_size); } p = kasan_krealloc((void *)p, new_size, flags); return (void *)p;
alloc_new:ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_); if (ret && p) { /* Disable KASAN checks as the object's redzone is accessed. */ kasan_disable_current(); memcpy(ret, kasan_reset_tag(p), orig_size ?: ks); kasan_enable_current(); } return ret; }
Quoted 31 lines · Vlastimil Babka
Quoted 20 lines · Feng Tang
Quoted 19 lines · Vlastimil Babka
Quoted 18 lines · Feng Tang
On Fri, Oct 04, 2024 at 05:52:10PM +0800, Vlastimil Babka wrote:
Thanks for the suggestion!As there were error report about the NULL slab for big kmalloc object, how about the following code for
__do_krealloc(const void *p, size_t new_size, gfp_t flags){ void *ret; size_t ks = 0; int orig_size = 0; struct kmem_cache *s = NULL; /* Check for double-free. */ if (likely(!ZERO_OR_NULL_PTR(p))) { if (!kasan_check_byte(p)) return NULL;Quoted 7 lines
Quoted 5 lines
Quoted 1 line
ks = ksize(p);I think this will result in
__ksize()doing
skip_orig_size_check(folio_slab(folio)->slab_cache, object);
and we don't want that?I think that's fine. As later code will re-set the
orig_sizeanyway.But you also read it first.
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
/* Some objects have no orig_size, like big kmalloc case */ if (is_kfence_address(p)) { orig_size = kfence_ksize(p); } else if (virt_to_slab(p)) { s = virt_to_cache(p); orig_size = get_orig_size(s, (void *)p);here.
Aha, you are right!
Quoted 14 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
}Also the checks below repeat some of the checks of
ksize().Quoted 8 lines
Yes, there is some redundancy, mostly the
virt_to_slab()Quoted 3 lines
So I think in
__do_krealloc()we should do things manually to determine ks and not callksize(). Just not break any of the casesksize()handles (kfence, large kmalloc).OK, originally I tried not to expose internals of
__ksize(). Let me try this way.
Quoted 6 lines
ksize()makes assumptions that a user outside of slab itself is calling it.But we (well mostly Kees) also introduced
kmalloc_size_roundup()to avoid queryingksize()for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if theskip_orig_size_check()mechanism can be removed now?
I did a quick grep, and fortunately it seems that the ksize() user are much less than before. We used to see some trouble in network code, which is now very clean without the need to skip orig_size check. Will check other call site later.
Quoted 2 lines
Still I think
__do_krealloc()should rather do its own thing and not callksize().
Yes. I made some changes:
static __always_inline __realloc_size(2) void * __do_krealloc(const void *p, size_t new_size, gfp_t flags)
{
void *ret;
size_t ks = 0;
int orig_size = 0;
struct kmem_cache *s = NULL;
/* Check for double-free. */
if (unlikely(ZERO_OR_NULL_PTR(p)))
goto alloc_new;
if (!kasan_check_byte(p))
return NULL;
if (is_kfence_address(p)) {
ks = orig_size = kfence_ksize(p);
} else {
struct folio *folio;
folio = virt_to_folio(p);
if (unlikely(!folio_test_slab(folio))) {
/* Big kmalloc object */
WARN_ON(folio_size(folio) <= KMALLOC_MAX_CACHE_SIZE);
WARN_ON(p != folio_address(folio));
ks = folio_size(folio);
} else {
s = folio_slab(folio)->slab_cache;
orig_size = get_orig_size(s, (void *)p);
ks = s->object_size;
}
}
/* If the old object doesn't fit, allocate a bigger one */
if (new_size > ks)
goto alloc_new;
/* Zero out spare memory. */
if (want_init_on_alloc(flags)) {
kasan_disable_current();
if (orig_size && orig_size < new_size)
memset((void *)p + orig_size, 0, new_size - orig_size);
else
memset((void *)p + new_size, 0, ks - new_size);
kasan_enable_current();
}
/* Setup kmalloc redzone when needed */
if (s && slub_debug_orig_size(s)) {
set_orig_size(s, (void *)p, new_size);
if (s->flags & SLAB_RED_ZONE && new_size < ks)
memset_no_sanitize_memory((void *)p + new_size,
SLUB_RED_ACTIVE, ks - new_size);
}
p = kasan_krealloc((void *)p, new_size, flags);
return (void *)p;alloc_new:
ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_);
if (ret && p) {
/* Disable KASAN checks as the object's redzone is accessed. */
kasan_disable_current();
memcpy(ret, kasan_reset_tag(p), orig_size ?: ks);
kasan_enable_current();
}
return ret;
}Thanks,
Feng
Quoted 14 lines · Feng Tang
Quoted 6 lines · Vlastimil Babka
Quoted 4 lines · Feng Tang
Quoted 1 line · Vlastimil Babka
On 10/14/24 09:52, Feng Tang wrote:
OK, originally I tried not to expose internals of
__ksize(). Let me try this way.Quoted 6 lines
ksize()makes assumptions that a user outside of slab itself is calling it.But we (well mostly Kees) also introduced
kmalloc_size_roundup()to avoid queryingksize()for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if theskip_orig_size_check()mechanism can be removed now?
Quoted 4 lines
I did a quick grep, and fortunately it seems that the
ksize()user are much less than before. We used to see some trouble in network code, which is now very clean without the need to skiporig_sizecheck. Will check other call site later.
Right -- only things that are performing a reallocation should be using ksize(). e.g. see __slab_build_skb()
Signature · 1 line
Kees Cook
Quoted 16 lines · Feng
Quoted 8 lines · Vlastimil Babka
Quoted 7 lines
Quoted 4 lines
So I think in
__do_krealloc()we should do things manually to determine ks and not callksize(). Just not break any of the casesksize()handles (kfence, large kmalloc).OK, originally I tried not to expose internals of
__ksize(). Let me try this way.Quoted 6 lines
ksize()makes assumptions that a user outside of slab itself is calling it.But we (well mostly Kees) also introduced
kmalloc_size_roundup()to avoid queryingksize()for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if theskip_orig_size_check()mechanism can be removed now?
Quoted 4 lines
I did a quick grep, and fortunately it seems that the
ksize()user are much less than before. We used to see some trouble in network code, which is now very clean without the need to skiporig_sizecheck. Will check other call site later.
I did more further check about ksize() usage, and there are still some places to be handled. The thing stands out is kfree_sensitive(), and another potential one is sound/soc/codecs/cs-amp-lib-test.c
Some details:
* Thanks to Kees Cook, who has cured many cases of ksize() as below:
drivers/base/devres.c: total_old_size = ksize(container_of(ptr, struct devres, data));
drivers/net/ethernet/intel/igb/igb_main.c: } else if (size > ksize(q_vector)) {
net/core/skbuff.c: *size = ksize(data);
net/openvswitch/flow_netlink.c: new_acts_size = max(next_offset + req_size, ksize(*sfa) * 2);
kernel/bpf/verifier.c: alloc_bytes = max(ksize(orig), kmalloc_size_roundup(bytes));* Some callers use ksize() mostly for calculation or sanity check, and not for accessing those extra space, which are fine:
drivers/gpu/drm/drm_managed.c: WARN_ON(dev + 1 > (struct drm_device *) (container + ksize(container))); lib/kunit/string-stream-test.c: actual_bytes_used = ksize(stream); lib/kunit/string-stream-test.c: actual_bytes_used += ksize(frag_container); lib/kunit/string-stream-test.c: actual_bytes_used += ksize(frag_container->fragment); mm/nommu.c: return ksize(objp); mm/util.c: memcpy(n, kasan_reset_tag(p), ksize(p)); security/tomoyo/gc.c: tomoyo_memory_used[TOMOYO_MEMORY_POLICY] -= ksize(ptr); security/tomoyo/memory.c: const size_t s = ksize(ptr); drivers/md/dm-vdo/memory-alloc.c: add_kmalloc_block(ksize(p)); drivers/md/dm-vdo/memory-alloc.c: add_kmalloc_block(ksize(p)); drivers/md/dm-vdo/memory-alloc.c: remove_kmalloc_block(ksize(ptr));
* One usage may need to be handled
sound/soc/codecs/cs-amp-lib-test.c: KUNIT_ASSERT_GE_MSG(test, ksize(buf), priv->cal_blob->size, "Buffer to small");
* bigger problem is the kfree_sensitive(), which will use ksize() to get the total size and then zero all of them.
One solution for this could be get the kmem_cache first, and do the skip_orig_size_check()
Thanks,
Feng
Quoted 54 lines · Feng Tang
Quoted 16 lines · Feng
Quoted 8 lines · Vlastimil Babka
Quoted 7 lines
Quoted 4 lines
So I think in
__do_krealloc()we should do things manually to determine ks and not callksize(). Just not break any of the casesksize()handles (kfence, large kmalloc).OK, originally I tried not to expose internals of
__ksize(). Let me try this way.Quoted 6 lines
ksize()makes assumptions that a user outside of slab itself is calling it.But we (well mostly Kees) also introduced
kmalloc_size_roundup()to avoid queryingksize()for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if theskip_orig_size_check()mechanism can be removed now?Quoted 4 lines
I did a quick grep, and fortunately it seems that the
ksize()user are much less than before. We used to see some trouble in network code, which is now very clean without the need to skiporig_sizecheck. Will check other call site later.I did more further check about
ksize()usage, and there are still some places to be handled. The thing stands out iskfree_sensitive(), and another potential one issound/soc/codecs/cs-amp-lib-test.cSome details:
* Thanks to Kees Cook, who has cured many cases of
ksize()as below:drivers/base/devres.c: total_old_size = ksize(container_of(ptr, struct devres, data)); drivers/net/ethernet/intel/igb/igb_main.c: } else if (size > ksize(q_vector)) { net/core/skbuff.c: *size = ksize(data); net/openvswitch/flow_netlink.c: new_acts_size = max(next_offset + req_size, ksize(*sfa) * 2); kernel/bpf/verifier.c: alloc_bytes = max(ksize(orig), kmalloc_size_roundup(bytes));* Some callers use
ksize()mostly for calculation or sanity check, and not for accessing those extra space, which are fine:drivers/gpu/drm/drm_managed.c: WARN_ON(dev + 1 > (struct drm_device *) (container + ksize(container))); lib/kunit/string-stream-test.c: actual_bytes_used = ksize(stream); lib/kunit/string-stream-test.c: actual_bytes_used += ksize(frag_container); lib/kunit/string-stream-test.c: actual_bytes_used += ksize(frag_container->fragment); mm/nommu.c: return ksize(objp); mm/util.c: memcpy(n, kasan_reset_tag(p), ksize(p)); security/tomoyo/gc.c: tomoyo_memory_used[TOMOYO_MEMORY_POLICY] -= ksize(ptr); security/tomoyo/memory.c: const size_t s = ksize(ptr); drivers/md/dm-vdo/memory-alloc.c: add_kmalloc_block(ksize(p)); drivers/md/dm-vdo/memory-alloc.c: add_kmalloc_block(ksize(p)); drivers/md/dm-vdo/memory-alloc.c: remove_kmalloc_block(ksize(ptr));* One usage may need to be handled
Quoted 7 lines
sound/soc/codecs/cs-amp-lib-test.c: KUNIT_ASSERT_GE_MSG(test, ksize(buf), priv->cal_blob->size, "Buffer to small");* bigger problem is the
kfree_sensitive(), which will useksize()to get the total size and then zero all of them.One solution for this could be get the
kmem_cachefirst, and do theskip_orig_size_check()
Maybe add a parameter for __ksize() that controls if we do skip_orig_size_check(), current ksize() will pass "false" to it (once remaining wrong users are handled), then another ksize_internal() variant will pass "true" and be used from kfree_sensitive()?
Quoted 2 lines
Thanks,
Feng
Quoted 60 lines · Vlastimil Babka
Quoted 59 lines · Feng Tang
Quoted 16 lines · Feng
Quoted 8 lines · Vlastimil Babka
Quoted 7 lines
Quoted 4 lines
So I think in
__do_krealloc()we should do things manually to determine ks and not callksize(). Just not break any of the casesksize()handles (kfence, large kmalloc).OK, originally I tried not to expose internals of
__ksize(). Let me try this way.Quoted 6 lines
ksize()makes assumptions that a user outside of slab itself is calling it.But we (well mostly Kees) also introduced
kmalloc_size_roundup()to avoid queryingksize()for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if theskip_orig_size_check()mechanism can be removed now?Quoted 4 lines
I did a quick grep, and fortunately it seems that the
ksize()user are much less than before. We used to see some trouble in network code, which is now very clean without the need to skiporig_sizecheck. Will check other call site later.I did more further check about
ksize()usage, and there are still some places to be handled. The thing stands out iskfree_sensitive(), and another potential one issound/soc/codecs/cs-amp-lib-test.cSome details:
* Thanks to Kees Cook, who has cured many cases of
ksize()as below:drivers/base/devres.c: total_old_size = ksize(container_of(ptr, struct devres, data)); drivers/net/ethernet/intel/igb/igb_main.c: } else if (size > ksize(q_vector)) { net/core/skbuff.c: *size = ksize(data); net/openvswitch/flow_netlink.c: new_acts_size = max(next_offset + req_size, ksize(*sfa) * 2); kernel/bpf/verifier.c: alloc_bytes = max(ksize(orig), kmalloc_size_roundup(bytes));* Some callers use
ksize()mostly for calculation or sanity check, and not for accessing those extra space, which are fine:drivers/gpu/drm/drm_managed.c: WARN_ON(dev + 1 > (struct drm_device *) (container + ksize(container))); lib/kunit/string-stream-test.c: actual_bytes_used = ksize(stream); lib/kunit/string-stream-test.c: actual_bytes_used += ksize(frag_container); lib/kunit/string-stream-test.c: actual_bytes_used += ksize(frag_container->fragment); mm/nommu.c: return ksize(objp); mm/util.c: memcpy(n, kasan_reset_tag(p), ksize(p)); security/tomoyo/gc.c: tomoyo_memory_used[TOMOYO_MEMORY_POLICY] -= ksize(ptr); security/tomoyo/memory.c: const size_t s = ksize(ptr); drivers/md/dm-vdo/memory-alloc.c: add_kmalloc_block(ksize(p)); drivers/md/dm-vdo/memory-alloc.c: add_kmalloc_block(ksize(p)); drivers/md/dm-vdo/memory-alloc.c: remove_kmalloc_block(ksize(ptr));* One usage may need to be handled
sound/soc/codecs/cs-amp-lib-test.c: KUNIT_ASSERT_GE_MSG(test, ksize(buf), priv->cal_blob->size, "Buffer to small");* bigger problem is the
kfree_sensitive(), which will useksize()to get the total size and then zero all of them.
Quoted 7 lines
Quoted 2 lines
One solution for this could be get the
kmem_cachefirst, and do theskip_orig_size_check()Maybe add a parameter for
__ksize()that controls if we doskip_orig_size_check(), currentksize()will pass "false" to it (once remaining wrong users are handled), then anotherksize_internal()variant will pass "true" and be used fromkfree_sensitive()?
Sounds good to me! And for future wrong usages of ksize(), we can fix them case by case when they are deteced.
Thanks,
Feng
Quoted 55 lines · Vlastimil Babka
Quoted 20 lines · Feng Tang
Quoted 19 lines · Vlastimil Babka
Quoted 18 lines · Feng Tang
On Fri, Oct 04, 2024 at 05:52:10PM +0800, Vlastimil Babka wrote:
Thanks for the suggestion!As there were error report about the NULL slab for big kmalloc object, how about the following code for
__do_krealloc(const void *p, size_t new_size, gfp_t flags){ void *ret; size_t ks = 0; int orig_size = 0; struct kmem_cache *s = NULL; /* Check for double-free. */ if (likely(!ZERO_OR_NULL_PTR(p))) { if (!kasan_check_byte(p)) return NULL;Quoted 7 lines
Quoted 5 lines
Quoted 1 line
ks = ksize(p);I think this will result in
__ksize()doing
skip_orig_size_check(folio_slab(folio)->slab_cache, object);
and we don't want that?I think that's fine. As later code will re-set the
orig_sizeanyway.But you also read it first.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
/* Some objects have no orig_size, like big kmalloc case */ if (is_kfence_address(p)) { orig_size = kfence_ksize(p); } else if (virt_to_slab(p)) { s = virt_to_cache(p); orig_size = get_orig_size(s, (void *)p);here.
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
}Also the checks below repeat some of the checks of
ksize().Quoted 8 lines
Yes, there is some redundancy, mostly the
virt_to_slab()Quoted 3 lines
So I think in
__do_krealloc()we should do things manually to determine ks and not callksize(). Just not break any of the casesksize()handles (kfence, large kmalloc).OK, originally I tried not to expose internals of
__ksize(). Let me try this way.
ksize()makes assumptions that a user outside of slab itself is calling it.
Quoted 7 lines
But we (well mostly Kees) also introduced
kmalloc_size_roundup()to avoid queryingksize()for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if theskip_orig_size_check()mechanism can be removed now?Still I think
__do_krealloc()should rather do its own thing and not callksize().
The goal was to avoid having users of the allocation APIs change the sizes of allocations without calling into realloc. This is because otherwise the "alloc_size" attribute used by compilers inform __builtin_dynamic_object_size() can get confused:
ptr = alloc(less_than_bucket_size);
...
size = ksize(ptr); /* larger size! */memcpy(ptr, src, size); /* compiler instrumentation doesn't see that ptr "grows" */
So the callers use kmalloc_size_roundup() to just allocate the rounded up size immediately. Internally, the allocator can do what it wants.
Signature · 1 line
Kees Cook