Skip to contentSkip to conversation

mm/slub: Improve data handling of krealloc() when orig_size is enabled

PATCH Superseded Feng Tang 5 participants 23 messages Sep 11, 2024 – Nov 4, 2024

Subsystems (4)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Feng Tang

author
#0 lore raw

v2 cover mm/slub: Improve data handling of krealloc() when orig_size is enabled

Danilo Krummrich's patch [1] raised one problem about krealloc() that its caller doesn't pass the old request size, say the object is 64 bytes kmalloc one, but caller originally only requested 48 bytes. Then when krealloc() shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when __GFP_ZERO is set).

Thus with slub debug redzone and object tracking enabled, parts of the object after krealloc() might contain redzone data instead of zeroes, which is violating the __GFP_ZERO guarantees. Good thing is in this case, kmalloc caches do have this 'orig_size' feature, which could be used to improve the situation here.

To make the 'orig_size' accurate, we adjust some kasan/slub meta data handling. Also add a slub kunit test case for krealloc().

This patchset has dependency over patches in both -mm tree and -slab trees, so it is written based on linux-next tree '20240910' version.

[1]. https://lore.kernel.org/lkml/[email protected]/

Thanks,
Feng

Changelog:

Since v1:
* Drop the patch changing generic kunit code from this patchset, and will send it separately.
* Separate the krealloc moving form slab_common.c to slub.c to a new patch for better review (Danilo/Vlastimil)
* Improve commit log and comments (Vlastimil/Danilo)
* Rework the kunit test case to remove its dependency over slub_debug (which is incomplete in v1) (Vlastimil)
* Add ack and review tag from developers.

Feng Tang (5):
mm/kasan: Don't store metadata inside kmalloc object when
slub_debug_orig_size is on
mm/slub: Consider kfence case for get_orig_size() mm/slub: Move krealloc() and related code to slub.c mm/slub: Improve redzone check and zeroing for krealloc() mm/slub, kunit: Add testcase for krealloc redzone and zeroing

 lib/slub_kunit.c   |  42 +++++++++++++++
 mm/kasan/generic.c |   7 ++-
 mm/slab.h          |   6 +++
 mm/slab_common.c   |  84 ------------------------------
 mm/slub.c          | 125 ++++++++++++++++++++++++++++++++++++++-------
 5 files changed, 160 insertions(+), 104 deletions(-)
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH v2 1/5 mm/kasan: Don't store metadata inside kmalloc object when slub_debug_orig_size is on

For a kmalloc object, when both kasan and slub redzone sanity check are enabled, they could both manipulate its data space like storing kasan free meta data and setting up kmalloc redzone, and may affect accuracy of that object's 'orig_size'.

As an accurate 'orig_size' will be needed by some function like krealloc() soon, save kasan's free meta data in slub's metadata area instead of inside object when 'orig_size' is enabled.

This will make it easier to maintain/understand the code. Size wise, when these two options are both enabled, the slub meta data space is already huge, and this just slightly increase the overall size.

---
 mm/kasan/generic.c |  7 +++++--
 mm/slab.h          |  6 ++++++
 mm/slub.c          | 17 -----------------
 3 files changed, 11 insertions(+), 19 deletions(-)
mm/kasan/generic.c +5 -2
index 6310a180278b..8b9e348113b1 100644
@@ -392,9 +392,12 @@ void kasan_cache_create(struct kmem_cache *cache, unsigned int *size,
 	 * 1. Object is SLAB_TYPESAFE_BY_RCU, which means that it can
 	 *    be touched after it was freed, or
 	 * 2. Object has a constructor, which means it's expected to
-	 *    retain its content until the next allocation.
+	 *    retain its content until the next allocation, or
+	 * 3. It is from a kmalloc cache which enables the debug option
+	 *    to store original size.
 	 */
-	if ((cache->flags & SLAB_TYPESAFE_BY_RCU) || cache->ctor) {
+	if ((cache->flags & SLAB_TYPESAFE_BY_RCU) || cache->ctor ||
+	     slub_debug_orig_size(cache)) {
 		cache->kasan_info.free_meta_offset = *size;
 		*size += sizeof(struct kasan_free_meta);
 		goto free_meta_added;
mm/slab.h +6 -0
index f22fb760b286..f72a8849b988 100644
@@ -689,6 +689,12 @@ void __kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
 void __check_heap_object(const void *ptr, unsigned long n,
 			 const struct slab *slab, bool to_user);
 
+static inline bool slub_debug_orig_size(struct kmem_cache *s)
+{
+	return (kmem_cache_debug_flags(s, SLAB_STORE_USER) &&
+			(s->flags & SLAB_KMALLOC));
+}
+
 #ifdef CONFIG_SLUB_DEBUG
 void skip_orig_size_check(struct kmem_cache *s, const void *object);
 #endif
mm/slub.c +0 -17
index 21f71cb6cc06..87c95f170f13 100644
@@ -230,12 +230,6 @@ static inline bool kmem_cache_debug(struct kmem_cache *s)
 	return kmem_cache_debug_flags(s, SLAB_DEBUG_FLAGS);
 }
 
-static inline bool slub_debug_orig_size(struct kmem_cache *s)
-{
-	return (kmem_cache_debug_flags(s, SLAB_STORE_USER) &&
-			(s->flags & SLAB_KMALLOC));
-}
-
 void *fixup_red_left(struct kmem_cache *s, void *p)
 {
 	if (kmem_cache_debug_flags(s, SLAB_RED_ZONE))
@@ -760,21 +754,10 @@ static inline void set_orig_size(struct kmem_cache *s,
 				void *object, unsigned int orig_size)
 {
 	void *p = kasan_reset_tag(object);
-	unsigned int kasan_meta_size;
 
 	if (!slub_debug_orig_size(s))
 		return;
 
-	/*
-	 * KASAN can save its free meta data inside of the object at offset 0.
-	 * If this meta data size is larger than 'orig_size', it will overlap
-	 * the data redzone in [orig_size+1, object_size]. Thus, we adjust
-	 * 'orig_size' to be as at least as big as KASAN's meta data.
-	 */
-	kasan_meta_size = kasan_metadata_size(s, true);
-	if (kasan_meta_size > orig_size)
-		orig_size = kasan_meta_size;
-
 	p += get_info_end(s);
 	p += sizeof(struct track) * 2;
 
Signature · 1 line
2.34.1
author
#2 lore raw

PATCH v2 2/5 mm/slub: Consider kfence case for get_orig_size()

In reply to Feng Tang

When 'orig_size' of kmalloc object is enabled by debug option, it should either contains the actual requested size or the cache's 'object_size'.

But it's not true if that object is a kfence-allocated one, and its 'orig_size' in metadata could be zero or other values. This is not a big issue for current 'orig_size' usage, as init_object() and check_object() during alloc/free process will be skipped for kfence addresses.

As 'orig_size' will be used by some function block like krealloc(), handle it by returning the 'object_size' in get_orig_size() for kfence addresses.

---
 mm/slub.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/slub.c +1 -1
index 87c95f170f13..021991e17287 100644
@@ -768,7 +768,7 @@ static inline unsigned int get_orig_size(struct kmem_cache *s, void *object)
 {
 	void *p = kasan_reset_tag(object);
 
-	if (!slub_debug_orig_size(s))
+	if (!slub_debug_orig_size(s) || is_kfence_address(object))
 		return s->object_size;
 
 	p += get_info_end(s);
Signature · 1 line
2.34.1
author
#3 lore raw

PATCH v2 3/5 mm/slub: Move krealloc() and related code to slub.c

In reply to Feng Tang

This is a preparation for the following refactoring of krealloc(), for more efficient function calling as it will call some internal functions defined in slub.c.

---
 mm/slab_common.c | 84 ------------------------------------------------
 mm/slub.c        | 84 ++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+), 84 deletions(-)
mm/slab_common.c +0 -84
index af6b14769fbd..5734b61a106f 100644
@@ -1185,90 +1185,6 @@ module_init(slab_proc_init);
 
 #endif /* CONFIG_SLUB_DEBUG */
 
-static __always_inline __realloc_size(2) void *
-__do_krealloc(const void *p, size_t new_size, gfp_t flags)
-{
-	void *ret;
-	size_t ks;
-
-	/* Check for double-free before calling ksize. */
-	if (likely(!ZERO_OR_NULL_PTR(p))) {
-		if (!kasan_check_byte(p))
-			return NULL;
-		ks = ksize(p);
-	} else
-		ks = 0;
-
-	/* If the object still fits, repoison it precisely. */
-	if (ks >= new_size) {
-		/* Zero out spare memory. */
-		if (want_init_on_alloc(flags)) {
-			kasan_disable_current();
-			memset((void *)p + new_size, 0, ks - new_size);
-			kasan_enable_current();
-		}
-
-		p = kasan_krealloc((void *)p, new_size, flags);
-		return (void *)p;
-	}
-
-	ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_);
-	if (ret && p) {
-		/* Disable KASAN checks as the object's redzone is accessed. */
-		kasan_disable_current();
-		memcpy(ret, kasan_reset_tag(p), ks);
-		kasan_enable_current();
-	}
-
-	return ret;
-}
-
-/**
- * krealloc - reallocate memory. The contents will remain unchanged.
- * @p: object to reallocate memory for.
- * @new_size: how many bytes of memory are required.
- * @flags: the type of memory to allocate.
- *
- * If @p is %NULL, krealloc() behaves exactly like kmalloc().  If @new_size
- * is 0 and @p is not a %NULL pointer, the object pointed to is freed.
- *
- * If __GFP_ZERO logic is requested, callers must ensure that, starting with the
- * initial memory allocation, every subsequent call to this API for the same
- * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
- * __GFP_ZERO is not fully honored by this API.
- *
- * This is the case, since krealloc() only knows about the bucket size of an
- * allocation (but not the exact size it was allocated with) and hence
- * implements the following semantics for shrinking and growing buffers with
- * __GFP_ZERO.
- *
- *         new             bucket
- * 0       size             size
- * |--------|----------------|
- * |  keep  |      zero      |
- *
- * In any case, the contents of the object pointed to are preserved up to the
- * lesser of the new and old sizes.
- *
- * Return: pointer to the allocated memory or %NULL in case of error
- */
-void *krealloc_noprof(const void *p, size_t new_size, gfp_t flags)
-{
-	void *ret;
-
-	if (unlikely(!new_size)) {
-		kfree(p);
-		return ZERO_SIZE_PTR;
-	}
-
-	ret = __do_krealloc(p, new_size, flags);
-	if (ret && kasan_reset_tag(p) != kasan_reset_tag(ret))
-		kfree(p);
-
-	return ret;
-}
-EXPORT_SYMBOL(krealloc_noprof);
-
 /**
  * kfree_sensitive - Clear sensitive information in memory before freeing
  * @p: object to free memory of
mm/slub.c +84 -0

90 lines not on this page · raw message

Signature · 1 line
2.34.1
author
#4 lore raw

PATCH v2 4/5 mm/slub: Improve redzone check and zeroing for krealloc()

In reply to Feng Tang

For current krealloc(), one problem is its caller doesn't pass the old request size, say the object is 64 bytes kmalloc one, but caller may only requested 48 bytes. Then when krealloc() shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when __GFP_ZERO is set).

Thus with slub debug redzone and object tracking enabled, parts of the object after krealloc() might contain redzone data instead of zeroes, which is violating the __GFP_ZERO guarantees. Good thing is in this case, kmalloc caches do have this 'orig_size' feature. So solve the problem by utilize 'org_size' to do accurate data zeroing and preserving.

---
 mm/slub.c | 54 ++++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 38 insertions(+), 16 deletions(-)
mm/slub.c +38 -16
index c1796f9dd30f..e0fb0a26c796 100644
@@ -4717,33 +4717,51 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)
 {
 	void *ret;
 	size_t ks;
+	int orig_size = 0;
+	struct kmem_cache *s;
 
-	/* Check for double-free before calling ksize. */
+	/* Check for double-free. */
 	if (likely(!ZERO_OR_NULL_PTR(p))) {
 		if (!kasan_check_byte(p))
 			return NULL;
-		ks = ksize(p);
+
+		s = virt_to_cache(p);
+		orig_size = get_orig_size(s, (void *)p);
+		ks = s->object_size;
 	} else
 		ks = 0;
 
-	/* If the object still fits, repoison it precisely. */
-	if (ks >= new_size) {
-		/* Zero out spare memory. */
-		if (want_init_on_alloc(flags)) {
-			kasan_disable_current();
+	/* If the object doesn't fit, allocate a bigger one */
+	if (new_size > ks)
+		goto alloc_new;
+
+	/* Zero out spare memory. */
+	if (want_init_on_alloc(flags)) {
+		kasan_disable_current();
+		if (orig_size < new_size)
+			memset((void *)p + orig_size, 0, new_size - orig_size);
+		else
 			memset((void *)p + new_size, 0, ks - new_size);
-			kasan_enable_current();
-		}
+		kasan_enable_current();
+	}
 
-		p = kasan_krealloc((void *)p, new_size, flags);
-		return (void *)p;
+	if (slub_debug_orig_size(s) && !is_kfence_address(p)) {
+		set_orig_size(s, (void *)p, new_size);
+		if (s->flags & SLAB_RED_ZONE && new_size < ks)
+			memset_no_sanitize_memory((void *)p + new_size,
+						SLUB_RED_ACTIVE, ks - new_size);
 	}
 
+	p = kasan_krealloc((void *)p, new_size, flags);
+	return (void *)p;
+
+alloc_new:
 	ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_);
 	if (ret && p) {
 		/* Disable KASAN checks as the object's redzone is accessed. */
 		kasan_disable_current();
-		memcpy(ret, kasan_reset_tag(p), ks);
+		if (orig_size)
+			memcpy(ret, kasan_reset_tag(p), orig_size);
 		kasan_enable_current();
 	}
 
@@ -4764,16 +4782,20 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)
  * memory allocation is flagged with __GFP_ZERO. Otherwise, it is possible that
  * __GFP_ZERO is not fully honored by this API.
  *
- * This is the case, since krealloc() only knows about the bucket size of an
- * allocation (but not the exact size it was allocated with) and hence
- * implements the following semantics for shrinking and growing buffers with
- * __GFP_ZERO.
+ * When slub_debug_orig_size() is off, krealloc() only knows about the bucket
+ * size of an allocation (but not the exact size it was allocated with) and
+ * hence implements the following semantics for shrinking and growing buffers
+ * with __GFP_ZERO.
  *
  *         new             bucket
  * 0       size             size
  * |--------|----------------|
  * |  keep  |      zero      |
  *
+ * Otherwise, the original allocation size 'orig_size' could be used to
+ * precisely clear the requested size, and the new size will also be stored
+ * as the new 'orig_size'.
+ *
  * In any case, the contents of the object pointed to are preserved up to the
  * lesser of the new and old sizes.
  *
Signature · 1 line
2.34.1
#5 lore raw

Re: [PATCH v2 4/5] mm/slub: Improve redzone check and zeroing for krealloc()

Hi Feng,

kernel test robot noticed the following build warnings:

https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Feng-Tang/mm-kasan-Don-t-store-metadata-inside-kmalloc-object-when-slub_debug_orig_size-is-on/20240911-144724
base:   next-20240910
patch link:    https://lore.kernel.org/r/20240911064535.557650-5-feng.tang%40intel.com
patch subject: [PATCH v2 4/5] mm/slub: Improve redzone check and zeroing for krealloc()
config: i386-randconfig-141-20240912 (https://download.01.org/0day-ci/archive/20240912/[email protected]/config)
compiler: clang version 18.1.8 (https://github.com/llvm/llvm-project 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags

| Reported-by: kernel test robot <[email protected]>
| Reported-by: Dan Carpenter <[email protected]>
| Closes: https://lore.kernel.org/r/[email protected]/

smatch warnings:
mm/slub.c:4748 __do_krealloc() error: uninitialized symbol 's'.

vim +/s +4748 mm/slub.c

474fff8e68012d Feng Tang       2024-09-11  4715  static __always_inline __realloc_size(2) void *
474fff8e68012d Feng Tang       2024-09-11  4716  __do_krealloc(const void *p, size_t new_size, gfp_t flags)
474fff8e68012d Feng Tang       2024-09-11  4717  {
474fff8e68012d Feng Tang       2024-09-11  4718  	void *ret;
474fff8e68012d Feng Tang       2024-09-11  4719  	size_t ks;
89a7731cc91033 Feng Tang       2024-09-11  4720  	int orig_size = 0;
89a7731cc91033 Feng Tang       2024-09-11  4721  	struct kmem_cache *s;
474fff8e68012d Feng Tang       2024-09-11  4722  
89a7731cc91033 Feng Tang       2024-09-11  4723  	/* Check for double-free. */
474fff8e68012d Feng Tang       2024-09-11  4724  	if (likely(!ZERO_OR_NULL_PTR(p))) {
474fff8e68012d Feng Tang       2024-09-11  4725  		if (!kasan_check_byte(p))
474fff8e68012d Feng Tang       2024-09-11  4726  			return NULL;
89a7731cc91033 Feng Tang       2024-09-11  4727  
89a7731cc91033 Feng Tang       2024-09-11  4728  		s = virt_to_cache(p);
89a7731cc91033 Feng Tang       2024-09-11  4729  		orig_size = get_orig_size(s, (void *)p);
89a7731cc91033 Feng Tang       2024-09-11  4730  		ks = s->object_size;
474fff8e68012d Feng Tang       2024-09-11  4731  	} else
474fff8e68012d Feng Tang       2024-09-11  4732  		ks = 0;

ks is zero

474fff8e68012d Feng Tang       2024-09-11  4733  
89a7731cc91033 Feng Tang       2024-09-11  4734  	/* If the object doesn't fit, allocate a bigger one */
89a7731cc91033 Feng Tang       2024-09-11  4735  	if (new_size > ks)

Assume new_size is also zero so this is false

89a7731cc91033 Feng Tang       2024-09-11  4736  		goto alloc_new;
89a7731cc91033 Feng Tang       2024-09-11  4737  
474fff8e68012d Feng Tang       2024-09-11  4738  	/* Zero out spare memory. */
474fff8e68012d Feng Tang       2024-09-11  4739  	if (want_init_on_alloc(flags)) {
474fff8e68012d Feng Tang       2024-09-11  4740  		kasan_disable_current();
89a7731cc91033 Feng Tang       2024-09-11  4741  		if (orig_size < new_size)
89a7731cc91033 Feng Tang       2024-09-11  4742  			memset((void *)p + orig_size, 0, new_size - orig_size);
89a7731cc91033 Feng Tang       2024-09-11  4743  		else
474fff8e68012d Feng Tang       2024-09-11  4744  			memset((void *)p + new_size, 0, ks - new_size);
474fff8e68012d Feng Tang       2024-09-11  4745  		kasan_enable_current();
474fff8e68012d Feng Tang       2024-09-11  4746  	}
474fff8e68012d Feng Tang       2024-09-11  4747  
89a7731cc91033 Feng Tang       2024-09-11 @4748  	if (slub_debug_orig_size(s) && !is_kfence_address(p)) {
                                                                                 ^

Uninitialized in that case

89a7731cc91033 Feng Tang       2024-09-11  4749  		set_orig_size(s, (void *)p, new_size);
89a7731cc91033 Feng Tang       2024-09-11  4750  		if (s->flags & SLAB_RED_ZONE && new_size < ks)
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author
#6 lore raw

Re: [PATCH v2 4/5] mm/slub: Improve redzone check and zeroing for krealloc()

Hi Dan,

Thanks for the report!

Quoted 43 lines · Dan Carpenter

Hi Feng,

kernel test robot noticed the following build warnings:

https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Feng-Tang/mm-kasan-Don-t-store-metadata-inside-kmalloc-object-when-slub_debug_orig_size-is-on/20240911-144724
base:   next-20240910
patch link:    https://lore.kernel.org/r/20240911064535.557650-5-feng.tang%40intel.com
patch subject: [PATCH v2 4/5] mm/slub: Improve redzone check and zeroing for krealloc()
config: i386-randconfig-141-20240912 (https://download.01.org/0day-ci/archive/20240912/[email protected]/config)
compiler: clang version 18.1.8 (https://github.com/llvm/llvm-project 3b5b5c1ec4a3095ab096dd780e84d7ab81f3d7ff)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags

| Reported-by: kernel test robot <[email protected]>
| Reported-by: Dan Carpenter <[email protected]>
| Closes: https://lore.kernel.org/r/[email protected]/

smatch warnings:
mm/slub.c:4748 __do_krealloc() error: uninitialized symbol 's'.

vim +/s +4748 mm/slub.c

474fff8e68012d Feng Tang       2024-09-11  4715  static __always_inline __realloc_size(2) void *
474fff8e68012d Feng Tang       2024-09-11  4716  __do_krealloc(const void *p, size_t new_size, gfp_t flags)
474fff8e68012d Feng Tang       2024-09-11  4717  {
474fff8e68012d Feng Tang       2024-09-11  4718  	void *ret;
474fff8e68012d Feng Tang       2024-09-11  4719  	size_t ks;
89a7731cc91033 Feng Tang       2024-09-11  4720  	int orig_size = 0;
89a7731cc91033 Feng Tang       2024-09-11  4721  	struct kmem_cache *s;
474fff8e68012d Feng Tang       2024-09-11  4722  
89a7731cc91033 Feng Tang       2024-09-11  4723  	/* Check for double-free. */
474fff8e68012d Feng Tang       2024-09-11  4724  	if (likely(!ZERO_OR_NULL_PTR(p))) {
474fff8e68012d Feng Tang       2024-09-11  4725  		if (!kasan_check_byte(p))
474fff8e68012d Feng Tang       2024-09-11  4726  			return NULL;
89a7731cc91033 Feng Tang       2024-09-11  4727  
89a7731cc91033 Feng Tang       2024-09-11  4728  		s = virt_to_cache(p);
89a7731cc91033 Feng Tang       2024-09-11  4729  		orig_size = get_orig_size(s, (void *)p);
89a7731cc91033 Feng Tang       2024-09-11  4730  		ks = s->object_size;
474fff8e68012d Feng Tang       2024-09-11  4731  	} else
474fff8e68012d Feng Tang       2024-09-11  4732  		ks = 0;
Quoted 7 lines

ks is zero

474fff8e68012d Feng Tang       2024-09-11  4733  
89a7731cc91033 Feng Tang       2024-09-11  4734  	/* If the object doesn't fit, allocate a bigger one */
89a7731cc91033 Feng Tang       2024-09-11  4735  	if (new_size > ks)

Assume new_size is also zero so this is false

Yes, this is possible, though unlikely :)

Below patch should fix it:
---

mm/slub.c +3 -2
index e0fb0a26c796..d5219634e96a 100644
@@ -4728,8 +4728,9 @@ __do_krealloc(const void *p, size_t new_size, gfp_t flags)
 		s = virt_to_cache(p);
 		orig_size = get_orig_size(s, (void *)p);
 		ks = s->object_size;
-	} else
-		ks = 0;
+	} else {
+		goto alloc_new;
+	}
 
 	/* If the object doesn't fit, allocate a bigger one */
 	if (new_size > ks)

Thanks,
Feng

Quoted 24 lines
89a7731cc91033 Feng Tang       2024-09-11  4736  		goto alloc_new;
89a7731cc91033 Feng Tang       2024-09-11  4737  
474fff8e68012d Feng Tang       2024-09-11  4738  	/* Zero out spare memory. */
474fff8e68012d Feng Tang       2024-09-11  4739  	if (want_init_on_alloc(flags)) {
474fff8e68012d Feng Tang       2024-09-11  4740  		kasan_disable_current();
89a7731cc91033 Feng Tang       2024-09-11  4741  		if (orig_size < new_size)
89a7731cc91033 Feng Tang       2024-09-11  4742  			memset((void *)p + orig_size, 0, new_size - orig_size);
89a7731cc91033 Feng Tang       2024-09-11  4743  		else
474fff8e68012d Feng Tang       2024-09-11  4744  			memset((void *)p + new_size, 0, ks - new_size);
474fff8e68012d Feng Tang       2024-09-11  4745  		kasan_enable_current();
474fff8e68012d Feng Tang       2024-09-11  4746  	}
474fff8e68012d Feng Tang       2024-09-11  4747  
89a7731cc91033 Feng Tang       2024-09-11 @4748  	if (slub_debug_orig_size(s) && !is_kfence_address(p)) {
                                                                                 ^

Uninitialized in that case

89a7731cc91033 Feng Tang       2024-09-11  4749  		set_orig_size(s, (void *)p, new_size);
89a7731cc91033 Feng Tang       2024-09-11  4750  		if (s->flags & SLAB_RED_ZONE && new_size < ks)
Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
author
#7 lore raw

PATCH v2 5/5 mm/slub, kunit: Add testcase for krealloc redzone and zeroing

In reply to Feng Tang

Danilo Krummrich raised issue about krealloc+GFP_ZERO [1], and Vlastimil suggested to add some test case which can sanity test the kmalloc-redzone and zeroing by utilizing the kmalloc's 'orig_size' debug feature.

It covers the grow and shrink case of krealloc() re-using current kmalloc object, and the case of re-allocating a new bigger object.

[1]. https://lore.kernel.org/lkml/[email protected]/

---
Hi Danilo,

I keep your Reviewed-By tag, as I think this v2 mostly changes what kmalloc slab to be used. Let me know if you want it dropped, thanks.

 lib/slub_kunit.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)
lib/slub_kunit.c +42 -0
index 6e3a1e5a7142..b3d158f38b98 100644
@@ -186,6 +186,47 @@ static void test_leak_destroy(struct kunit *test)
 	KUNIT_EXPECT_EQ(test, 1, slab_errors);
 }
 
+static void test_krealloc_redzone_zeroing(struct kunit *test)
+{
+	u8 *p;
+	int i;
+	struct kmem_cache *s = test_kmem_cache_create("TestSlub_krealloc", 64,
+				SLAB_KMALLOC|SLAB_STORE_USER|SLAB_RED_ZONE);
+
+	p = __kmalloc_cache_noprof(s, GFP_KERNEL, 48);
+	memset(p, 0xff, 48);
+
+	kasan_disable_current();
+	OPTIMIZER_HIDE_VAR(p);
+
+	/* Test shrink */
+	p = krealloc(p, 40, GFP_KERNEL | __GFP_ZERO);
+	for (i = 40; i < 64; i++)
+		KUNIT_EXPECT_EQ(test, p[i], SLUB_RED_ACTIVE);
+
+	/* Test grow within the same 64B kmalloc object */
+	p = krealloc(p, 56, GFP_KERNEL | __GFP_ZERO);
+	for (i = 40; i < 56; i++)
+		KUNIT_EXPECT_EQ(test, p[i], 0);
+	for (i = 56; i < 64; i++)
+		KUNIT_EXPECT_EQ(test, p[i], SLUB_RED_ACTIVE);
+
+	validate_slab_cache(s);
+	KUNIT_EXPECT_EQ(test, 0, slab_errors);
+
+	memset(p, 0xff, 56);
+	/* Test grow with allocating a bigger 128B object */
+	p = krealloc(p, 112, GFP_KERNEL | __GFP_ZERO);
+	for (i = 0; i < 56; i++)
+		KUNIT_EXPECT_EQ(test, p[i], 0xff);
+	for (i = 56; i < 112; i++)
+		KUNIT_EXPECT_EQ(test, p[i], 0);
+
+	kfree(p);
+	kasan_enable_current();
+	kmem_cache_destroy(s);
+}
+
 static int test_init(struct kunit *test)
 {
 	slab_errors = 0;
@@ -208,6 +249,7 @@ static struct kunit_case test_cases[] = {
 	KUNIT_CASE(test_kmalloc_redzone_access),
 	KUNIT_CASE(test_kfree_rcu),
 	KUNIT_CASE(test_leak_destroy),
+	KUNIT_CASE(test_krealloc_redzone_zeroing),
 	{}
 };
 
Signature · 1 line
2.34.1
#8 lore raw
In reply to Feng Tang
Quoted 13 lines · Feng Tang

Danilo Krummrich's patch [1] raised one problem about krealloc() that its caller doesn't pass the old request size, say the object is 64 bytes kmalloc one, but caller originally only requested 48 bytes. Then when krealloc() shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when __GFP_ZERO is set).

Thus with slub debug redzone and object tracking enabled, parts of the object after krealloc() might contain redzone data instead of zeroes, which is violating the __GFP_ZERO guarantees. Good thing is in this case, kmalloc caches do have this 'orig_size' feature, which could be used to improve the situation here.

Quoted 7 lines

To make the 'orig_size' accurate, we adjust some kasan/slub meta data handling. Also add a slub kunit test case for krealloc().

This patchset has dependency over patches in both -mm tree and -slab trees, so it is written based on linux-next tree '20240910' version.

[1]. https://lore.kernel.org/lkml/[email protected]/

Thanks, added to slab/for-next

Quoted 31 lines

Thanks,
Feng

Changelog:

Since v1:
* Drop the patch changing generic kunit code from this patchset, and will send it separately.
* Separate the krealloc moving form slab_common.c to slub.c to a new patch for better review (Danilo/Vlastimil)
* Improve commit log and comments (Vlastimil/Danilo)
* Rework the kunit test case to remove its dependency over slub_debug (which is incomplete in v1) (Vlastimil)
* Add ack and review tag from developers.

Feng Tang (5):
mm/kasan: Don't store metadata inside kmalloc object when
slub_debug_orig_size is on
mm/slub: Consider kfence case for get_orig_size() mm/slub: Move krealloc() and related code to slub.c mm/slub: Improve redzone check and zeroing for krealloc() mm/slub, kunit: Add testcase for krealloc redzone and zeroing

 lib/slub_kunit.c   |  42 +++++++++++++++
 mm/kasan/generic.c |   7 ++-
 mm/slab.h          |   6 +++
 mm/slab_common.c   |  84 ------------------------------
 mm/slub.c          | 125 ++++++++++++++++++++++++++++++++++++++-------
 5 files changed, 160 insertions(+), 104 deletions(-)
#9 lore raw
Quoted 18 lines · Vlastimil Babka
Quoted 16 lines · Feng Tang

Danilo Krummrich's patch [1] raised one problem about krealloc() that its caller doesn't pass the old request size, say the object is 64 bytes kmalloc one, but caller originally only requested 48 bytes. Then when krealloc() shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when __GFP_ZERO is set).

Thus with slub debug redzone and object tracking enabled, parts of the object after krealloc() might contain redzone data instead of zeroes, which is violating the __GFP_ZERO guarantees. Good thing is in this case, kmalloc caches do have this 'orig_size' feature, which could be used to improve the situation here.

To make the 'orig_size' accurate, we adjust some kasan/slub meta data handling. Also add a slub kunit test case for krealloc().

Quoted 6 lines
Quoted 4 lines

This patchset has dependency over patches in both -mm tree and -slab trees, so it is written based on linux-next tree '20240910' version.

[1]. https://lore.kernel.org/lkml/[email protected]/

Thanks, added to slab/for-next

This series just hit -next, and we're seeing several "KFENCE: memory corruption ...". Here's one:
https://lore.kernel.org/all/[email protected]/

One more (no link):

Quoted 8 lines

==================================================================

BUG: KFENCE: memory corruption in xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051

Corrupted memory at 0xffff88823bf5a0d0 [ 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ] (in kfence-#172):

xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051
xfs_iext_destroy+0x66/0x100 fs/xfs/libxfs/xfs_iext_tree.c:1062
xfs_inode_free_callback+0x91/0x1d0 fs/xfs/xfs_icache.c:145
rcu_do_batch kernel/rcu/tree.c:2567 [inline]

[...]

Quoted 7 lines

kfence-#172: 0xffff88823bf5a000-0xffff88823bf5a0cf, size=208, cache=kmalloc-256

allocated by task 5494 on cpu 0 at 101.266046s (0.409225s ago):

__do_krealloc mm/slub.c:4784 [inline]
krealloc_noprof+0xd6/0x2e0 mm/slub.c:4838
xfs_iext_realloc_root fs/xfs/libxfs/xfs_iext_tree.c:613 [inline]

[...]

Quoted 5 lines

freed by task 16 on cpu 0 at 101.573936s (0.186416s ago):

xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051
xfs_iext_destroy+0x66/0x100 fs/xfs/libxfs/xfs_iext_tree.c:1062
xfs_inode_free_callback+0x91/0x1d0 fs/xfs/xfs_icache.c:145

[...]

Quoted 4 lines
CPU: 0 UID: 0 PID: 16 Comm: ksoftirqd/0 Not tainted 6.12.0-rc1-next-20241003-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
==================================================================

Unfortunately there's no reproducer yet it seems. Unless it's immediately obvious to say what's wrong, is it possible to take this series out of -next to confirm this series is causing the memory corruptions? Syzbot should then stop finding these crashes.

Thanks,
-- Marco

#10 lore raw
Quoted 56 lines · Marco Elver
Quoted 18 lines · Vlastimil Babka
Quoted 16 lines · Feng Tang

Danilo Krummrich's patch [1] raised one problem about krealloc() that its caller doesn't pass the old request size, say the object is 64 bytes kmalloc one, but caller originally only requested 48 bytes. Then when krealloc() shrinks or grows in the same object, or allocate a new bigger object, it lacks this 'original size' information to do accurate data preserving or zeroing (when __GFP_ZERO is set).

Thus with slub debug redzone and object tracking enabled, parts of the object after krealloc() might contain redzone data instead of zeroes, which is violating the __GFP_ZERO guarantees. Good thing is in this case, kmalloc caches do have this 'orig_size' feature, which could be used to improve the situation here.

To make the 'orig_size' accurate, we adjust some kasan/slub meta data handling. Also add a slub kunit test case for krealloc().

Quoted 6 lines
Quoted 4 lines

This patchset has dependency over patches in both -mm tree and -slab trees, so it is written based on linux-next tree '20240910' version.

[1]. https://lore.kernel.org/lkml/[email protected]/

Thanks, added to slab/for-next

This series just hit -next, and we're seeing several "KFENCE: memory corruption ...". Here's one:
https://lore.kernel.org/all/[email protected]/

One more (no link):

Quoted 8 lines

==================================================================

BUG: KFENCE: memory corruption in xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051

Corrupted memory at 0xffff88823bf5a0d0 [ 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ] (in kfence-#172):

xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051
xfs_iext_destroy+0x66/0x100 fs/xfs/libxfs/xfs_iext_tree.c:1062
xfs_inode_free_callback+0x91/0x1d0 fs/xfs/xfs_icache.c:145
rcu_do_batch kernel/rcu/tree.c:2567 [inline]

[...]

Quoted 7 lines

kfence-#172: 0xffff88823bf5a000-0xffff88823bf5a0cf, size=208, cache=kmalloc-256

allocated by task 5494 on cpu 0 at 101.266046s (0.409225s ago):

__do_krealloc mm/slub.c:4784 [inline]
krealloc_noprof+0xd6/0x2e0 mm/slub.c:4838
xfs_iext_realloc_root fs/xfs/libxfs/xfs_iext_tree.c:613 [inline]

[...]

Quoted 5 lines

freed by task 16 on cpu 0 at 101.573936s (0.186416s ago):

xfs_iext_destroy_node+0xab/0x670 fs/xfs/libxfs/xfs_iext_tree.c:1051
xfs_iext_destroy+0x66/0x100 fs/xfs/libxfs/xfs_iext_tree.c:1062
xfs_inode_free_callback+0x91/0x1d0 fs/xfs/xfs_icache.c:145

[...]

Quoted 1 line
Quoted 8 lines
Quoted 3 lines
CPU: 0 UID: 0 PID: 16 Comm: ksoftirqd/0 Not tainted 6.12.0-rc1-next-20241003-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/13/2024
==================================================================

Unfortunately there's no reproducer yet it seems. Unless it's immediately obvious to say what's wrong, is it possible to take this series out of -next to confirm this series is causing the memory corruptions? Syzbot should then stop finding these crashes.

I think it's commit d0a38fad51cc7 doing in __do_krealloc()

- ks = ksize(p);
+

+               s = virt_to_cache(p);
+               orig_size = get_orig_size(s, (void *)p);
+               ks = s->object_size;

so for kfence objects we don't get their actual allocation size but the potentially larger bucket size?

I guess we could do:

ks = kfence_ksize(p) ?: s->object_size;

?

Quoted 2 lines

Thanks,
-- Marco

#11 lore raw
Quoted 10 lines · Vlastimil Babka

On 10/4/24 08:44, Marco Elver wrote:

I think it's commit d0a38fad51cc7 doing in __do_krealloc()

(quoted diff) +4 -1
-               ks = ksize(p);
+
+               s = virt_to_cache(p);
+               orig_size = get_orig_size(s, (void *)p);
+               ks = s->object_size;
Quoted 8 lines

so for kfence objects we don't get their actual allocation size but the potentially larger bucket size?

I guess we could do:

ks = kfence_ksize(p) ?: s->object_size;

?

Hmm this probably is not the whole story, we also have:

- memcpy(ret, kasan_reset_tag(p), ks);
+ if (orig_size)
+ memcpy(ret, kasan_reset_tag(p), orig_size);

orig_size for kfence will be again s->object_size so the memcpy might be a (read) buffer overflow from a kfence allocation.

I think get_orig_size() should perhaps return kfence_ksize(p) for kfence allocations, in addition to the change above.

Or alternatively we don't change get_orig_size() (in a different commit) at all, but __do_krealloc() will have an "if is_kfence_address()" that sets both orig_size and ks to kfence_ksize(p) appropriately. That might be easier to follow.

But either way means rewriting 2 commits. I think it's indeed better to drop the series now from -next and submit a v3.

Vlastimil

Quoted 3 lines
Quoted 2 lines

Thanks,
-- Marco

author
#12 lore raw
Quoted 32 lines · Vlastimil Babka
Quoted 10 lines · Vlastimil Babka

On 10/4/24 08:44, Marco Elver wrote:

I think it's commit d0a38fad51cc7 doing in __do_krealloc()

(quoted diff) +4 -1
-               ks = ksize(p);
+
+               s = virt_to_cache(p);
+               orig_size = get_orig_size(s, (void *)p);
+               ks = s->object_size;
Quoted 8 lines

so for kfence objects we don't get their actual allocation size but the potentially larger bucket size?

I guess we could do:

ks = kfence_ksize(p) ?: s->object_size;

?

Hmm this probably is not the whole story, we also have:

(quoted diff) +2 -1
-               memcpy(ret, kasan_reset_tag(p), ks);
+               if (orig_size)
+                       memcpy(ret, kasan_reset_tag(p), orig_size);

orig_size for kfence will be again s->object_size so the memcpy might be a (read) buffer overflow from a kfence allocation.

I think get_orig_size() should perhaps return kfence_ksize(p) for kfence allocations, in addition to the change above.

Quoted 7 lines

Or alternatively we don't change get_orig_size() (in a different commit) at all, but __do_krealloc() will have an "if is_kfence_address()" that sets both orig_size and ks to kfence_ksize(p) appropriately. That might be easier to follow.

But either way means rewriting 2 commits. I think it's indeed better to drop the series now from -next and submit a v3.

Yes, we can revert now. Sorry for the inconvenience.

Thanks,
Feng

author
#13 lore raw
In reply to Vlastimil Babka
Quoted 29 lines · Vlastimil Babka
Quoted 10 lines · Vlastimil Babka

On 10/4/24 08:44, Marco Elver wrote:

I think it's commit d0a38fad51cc7 doing in __do_krealloc()

(quoted diff) +4 -1
-               ks = ksize(p);
+
+               s = virt_to_cache(p);
+               orig_size = get_orig_size(s, (void *)p);
+               ks = s->object_size;
Quoted 8 lines

so for kfence objects we don't get their actual allocation size but the potentially larger bucket size?

I guess we could do:

ks = kfence_ksize(p) ?: s->object_size;

?

Hmm this probably is not the whole story, we also have:

(quoted diff) +2 -1
-               memcpy(ret, kasan_reset_tag(p), ks);
+               if (orig_size)
+                       memcpy(ret, kasan_reset_tag(p), orig_size);

orig_size for kfence will be again s->object_size so the memcpy might be a (read) buffer overflow from a kfence allocation.

Quoted 7 lines

I think get_orig_size() should perhaps return kfence_ksize(p) for kfence allocations, in addition to the change above.

Or alternatively we don't change get_orig_size() (in a different commit) at all, but __do_krealloc() will have an "if is_kfence_address()" that sets both orig_size and ks to kfence_ksize(p) appropriately. That might be easier to follow.

Thanks for the suggestion!

As there were error report about the NULL slab for big kmalloc object, how about the following code for

__do_krealloc(const void *p, size_t new_size, gfp_t flags)

{
	void *ret;
	size_t ks = 0;
	int orig_size = 0;
	struct kmem_cache *s = NULL;

	/* Check for double-free. */
	if (likely(!ZERO_OR_NULL_PTR(p))) {
		if (!kasan_check_byte(p))
			return NULL;

		ks = ksize(p);

		/* Some objects have no orig_size, like big kmalloc case */
		if (is_kfence_address(p)) {
			orig_size = kfence_ksize(p);
		} else if (virt_to_slab(p)) {
			s = virt_to_cache(p);
			orig_size = get_orig_size(s, (void *)p);
		}
	} else {
		goto alloc_new;
	}

	/* If the object doesn't fit, allocate a bigger one */
	if (new_size > ks)
		goto alloc_new;

	/* Zero out spare memory. */
	if (want_init_on_alloc(flags)) {
		kasan_disable_current();
		if (orig_size && orig_size < new_size)
			memset((void *)p + orig_size, 0, new_size - orig_size);
		else
			memset((void *)p + new_size, 0, ks - new_size);
		kasan_enable_current();
	}

	/* Setup kmalloc redzone when needed */
	if (s && slub_debug_orig_size(s) && !is_kfence_address(p)) {
		set_orig_size(s, (void *)p, new_size);
		if (s->flags & SLAB_RED_ZONE && new_size < ks)
			memset_no_sanitize_memory((void *)p + new_size,
						SLUB_RED_ACTIVE, ks - new_size);
	}

	p = kasan_krealloc((void *)p, new_size, flags);
	return (void *)p;

alloc_new:

	ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_);
	if (ret && p) {
		/* Disable KASAN checks as the object's redzone is accessed. */
		kasan_disable_current();
		memcpy(ret, kasan_reset_tag(p), orig_size ?: ks);
		kasan_enable_current();
	}

	return ret;
}

I've run it with the reproducer of syzbot, so far the issue hasn't been reproduced on my local machine.

Thanks,
Feng

Quoted 10 lines

But either way means rewriting 2 commits. I think it's indeed better to drop the series now from -next and submit a v3.

Vlastimil

Quoted 3 lines
Quoted 2 lines

Thanks,
-- Marco

#14 lore raw
Quoted 7 lines · Feng Tang

On Fri, Oct 04, 2024 at 05:52:10PM +0800, Vlastimil Babka wrote:
Thanks for the suggestion!

As there were error report about the NULL slab for big kmalloc object, how about the following code for

__do_krealloc(const void *p, size_t new_size, gfp_t flags)

Quoted 12 lines
{
	void *ret;
	size_t ks = 0;
	int orig_size = 0;
	struct kmem_cache *s = NULL;

	/* Check for double-free. */
	if (likely(!ZERO_OR_NULL_PTR(p))) {
		if (!kasan_check_byte(p))
			return NULL;

		ks = ksize(p);

I think this will result in __ksize() doing
skip_orig_size_check(folio_slab(folio)->slab_cache, object);
and we don't want that?

Also the checks below repeat some of the checks of ksize().

So I think in __do_krealloc() we should do things manually to determine ks and not call ksize(). Just not break any of the cases ksize() handles (kfence, large kmalloc).

Quoted 65 lines
		/* Some objects have no orig_size, like big kmalloc case */
		if (is_kfence_address(p)) {
			orig_size = kfence_ksize(p);
		} else if (virt_to_slab(p)) {
			s = virt_to_cache(p);
			orig_size = get_orig_size(s, (void *)p);
		}
	} else {
		goto alloc_new;
	}

	/* If the object doesn't fit, allocate a bigger one */
	if (new_size > ks)
		goto alloc_new;

	/* Zero out spare memory. */
	if (want_init_on_alloc(flags)) {
		kasan_disable_current();
		if (orig_size && orig_size < new_size)
			memset((void *)p + orig_size, 0, new_size - orig_size);
		else
			memset((void *)p + new_size, 0, ks - new_size);
		kasan_enable_current();
	}

	/* Setup kmalloc redzone when needed */
	if (s && slub_debug_orig_size(s) && !is_kfence_address(p)) {
		set_orig_size(s, (void *)p, new_size);
		if (s->flags & SLAB_RED_ZONE && new_size < ks)
			memset_no_sanitize_memory((void *)p + new_size,
						SLUB_RED_ACTIVE, ks - new_size);
	}

	p = kasan_krealloc((void *)p, new_size, flags);
	return (void *)p;

alloc_new:

	ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_);
	if (ret && p) {
		/* Disable KASAN checks as the object's redzone is accessed. */
		kasan_disable_current();
		memcpy(ret, kasan_reset_tag(p), orig_size ?: ks);
		kasan_enable_current();
	}

	return ret;
}

I've run it with the reproducer of syzbot, so far the issue hasn't been reproduced on my local machine.

Thanks,
Feng

Quoted 10 lines

But either way means rewriting 2 commits. I think it's indeed better to drop the series now from -next and submit a v3.

Vlastimil

Quoted 3 lines
Quoted 2 lines

Thanks,
-- Marco

author
#15 lore raw
Quoted 19 lines · Vlastimil Babka
Quoted 18 lines · Feng Tang

On Fri, Oct 04, 2024 at 05:52:10PM +0800, Vlastimil Babka wrote:
Thanks for the suggestion!

As there were error report about the NULL slab for big kmalloc object, how about the following code for

__do_krealloc(const void *p, size_t new_size, gfp_t flags)

{
	void *ret;
	size_t ks = 0;
	int orig_size = 0;
	struct kmem_cache *s = NULL;

	/* Check for double-free. */
	if (likely(!ZERO_OR_NULL_PTR(p))) {
		if (!kasan_check_byte(p))
			return NULL;
Quoted 5 lines
Quoted 1 line
		ks = ksize(p);

I think this will result in __ksize() doing
skip_orig_size_check(folio_slab(folio)->slab_cache, object);
and we don't want that?

I think that's fine. As later code will re-set the orig_size anyway.

Quoted 1 line

Also the checks below repeat some of the checks of ksize().

Yes, there is some redundancy, mostly the virt_to_slab()

Quoted 3 lines

So I think in __do_krealloc() we should do things manually to determine ks and not call ksize(). Just not break any of the cases ksize() handles (kfence, large kmalloc).

OK, originally I tried not to expose internals of __ksize(). Let me try this way.

Thanks,
Feng

Quoted 49 lines
Quoted 49 lines

		/* Some objects have no orig_size, like big kmalloc case */
		if (is_kfence_address(p)) {
			orig_size = kfence_ksize(p);
		} else if (virt_to_slab(p)) {
			s = virt_to_cache(p);
			orig_size = get_orig_size(s, (void *)p);
		}
	} else {
		goto alloc_new;
	}

	/* If the object doesn't fit, allocate a bigger one */
	if (new_size > ks)
		goto alloc_new;

	/* Zero out spare memory. */
	if (want_init_on_alloc(flags)) {
		kasan_disable_current();
		if (orig_size && orig_size < new_size)
			memset((void *)p + orig_size, 0, new_size - orig_size);
		else
			memset((void *)p + new_size, 0, ks - new_size);
		kasan_enable_current();
	}

	/* Setup kmalloc redzone when needed */
	if (s && slub_debug_orig_size(s) && !is_kfence_address(p)) {
		set_orig_size(s, (void *)p, new_size);
		if (s->flags & SLAB_RED_ZONE && new_size < ks)
			memset_no_sanitize_memory((void *)p + new_size,
						SLUB_RED_ACTIVE, ks - new_size);
	}

	p = kasan_krealloc((void *)p, new_size, flags);
	return (void *)p;

alloc_new:

	ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_);
	if (ret && p) {
		/* Disable KASAN checks as the object's redzone is accessed. */
		kasan_disable_current();
		memcpy(ret, kasan_reset_tag(p), orig_size ?: ks);
		kasan_enable_current();
	}

	return ret;
}
#16 lore raw
Quoted 20 lines · Feng Tang
Quoted 19 lines · Vlastimil Babka
Quoted 18 lines · Feng Tang

On Fri, Oct 04, 2024 at 05:52:10PM +0800, Vlastimil Babka wrote:
Thanks for the suggestion!

As there were error report about the NULL slab for big kmalloc object, how about the following code for

__do_krealloc(const void *p, size_t new_size, gfp_t flags)

{
	void *ret;
	size_t ks = 0;
	int orig_size = 0;
	struct kmem_cache *s = NULL;

	/* Check for double-free. */
	if (likely(!ZERO_OR_NULL_PTR(p))) {
		if (!kasan_check_byte(p))
			return NULL;
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
		ks = ksize(p);

I think this will result in __ksize() doing
skip_orig_size_check(folio_slab(folio)->slab_cache, object);
and we don't want that?

I think that's fine. As later code will re-set the orig_size anyway.

But you also read it first.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
		/* Some objects have no orig_size, like big kmalloc case */
		if (is_kfence_address(p)) {
			orig_size = kfence_ksize(p);
		} else if (virt_to_slab(p)) {
			s = virt_to_cache(p);
			orig_size = get_orig_size(s, (void *)p);

here.

Quoted 4 lines
Quoted 3 lines
Quoted 1 line
		}

Also the checks below repeat some of the checks of ksize().

Quoted 8 lines

Yes, there is some redundancy, mostly the virt_to_slab()

Quoted 3 lines

So I think in __do_krealloc() we should do things manually to determine ks and not call ksize(). Just not break any of the cases ksize() handles (kfence, large kmalloc).

OK, originally I tried not to expose internals of __ksize(). Let me try this way.

ksize() makes assumptions that a user outside of slab itself is calling it.

But we (well mostly Kees) also introduced kmalloc_size_roundup() to avoid querying ksize() for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if the skip_orig_size_check() mechanism can be removed now?

Still I think __do_krealloc() should rather do its own thing and not call ksize().

Quoted 45 lines

Thanks,
Feng

Quoted 42 lines
Quoted 42 lines

	} else {
		goto alloc_new;
	}

	/* If the object doesn't fit, allocate a bigger one */
	if (new_size > ks)
		goto alloc_new;

	/* Zero out spare memory. */
	if (want_init_on_alloc(flags)) {
		kasan_disable_current();
		if (orig_size && orig_size < new_size)
			memset((void *)p + orig_size, 0, new_size - orig_size);
		else
			memset((void *)p + new_size, 0, ks - new_size);
		kasan_enable_current();
	}

	/* Setup kmalloc redzone when needed */
	if (s && slub_debug_orig_size(s) && !is_kfence_address(p)) {
		set_orig_size(s, (void *)p, new_size);
		if (s->flags & SLAB_RED_ZONE && new_size < ks)
			memset_no_sanitize_memory((void *)p + new_size,
						SLUB_RED_ACTIVE, ks - new_size);
	}

	p = kasan_krealloc((void *)p, new_size, flags);
	return (void *)p;

alloc_new:

	ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_);
	if (ret && p) {
		/* Disable KASAN checks as the object's redzone is accessed. */
		kasan_disable_current();
		memcpy(ret, kasan_reset_tag(p), orig_size ?: ks);
		kasan_enable_current();
	}

	return ret;
}
author
#17 lore raw
Quoted 31 lines · Vlastimil Babka
Quoted 20 lines · Feng Tang
Quoted 19 lines · Vlastimil Babka
Quoted 18 lines · Feng Tang

On Fri, Oct 04, 2024 at 05:52:10PM +0800, Vlastimil Babka wrote:
Thanks for the suggestion!

As there were error report about the NULL slab for big kmalloc object, how about the following code for

__do_krealloc(const void *p, size_t new_size, gfp_t flags)

{
	void *ret;
	size_t ks = 0;
	int orig_size = 0;
	struct kmem_cache *s = NULL;

	/* Check for double-free. */
	if (likely(!ZERO_OR_NULL_PTR(p))) {
		if (!kasan_check_byte(p))
			return NULL;
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
		ks = ksize(p);

I think this will result in __ksize() doing
skip_orig_size_check(folio_slab(folio)->slab_cache, object);
and we don't want that?

I think that's fine. As later code will re-set the orig_size anyway.

But you also read it first.

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
		/* Some objects have no orig_size, like big kmalloc case */
		if (is_kfence_address(p)) {
			orig_size = kfence_ksize(p);
		} else if (virt_to_slab(p)) {
			s = virt_to_cache(p);
			orig_size = get_orig_size(s, (void *)p);

here.

Aha, you are right!

Quoted 14 lines
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
		}

Also the checks below repeat some of the checks of ksize().

Quoted 8 lines

Yes, there is some redundancy, mostly the virt_to_slab()

Quoted 3 lines

So I think in __do_krealloc() we should do things manually to determine ks and not call ksize(). Just not break any of the cases ksize() handles (kfence, large kmalloc).

OK, originally I tried not to expose internals of __ksize(). Let me try this way.

Quoted 6 lines

ksize() makes assumptions that a user outside of slab itself is calling it.

But we (well mostly Kees) also introduced kmalloc_size_roundup() to avoid querying ksize() for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if the skip_orig_size_check() mechanism can be removed now?

I did a quick grep, and fortunately it seems that the ksize() user are much less than before. We used to see some trouble in network code, which is now very clean without the need to skip orig_size check. Will check other call site later.

Quoted 2 lines

Still I think __do_krealloc() should rather do its own thing and not call ksize().

Yes. I made some changes:

static __always_inline __realloc_size(2) void * __do_krealloc(const void *p, size_t new_size, gfp_t flags)

{
	void *ret;
	size_t ks = 0;
	int orig_size = 0;
	struct kmem_cache *s = NULL;

	/* Check for double-free. */
	if (unlikely(ZERO_OR_NULL_PTR(p)))
		goto alloc_new;

	if (!kasan_check_byte(p))
		return NULL;

	if (is_kfence_address(p)) {
		ks = orig_size = kfence_ksize(p);
	} else {
		struct folio *folio;

		folio = virt_to_folio(p);
		if (unlikely(!folio_test_slab(folio))) {
			/* Big kmalloc object */
			WARN_ON(folio_size(folio) <= KMALLOC_MAX_CACHE_SIZE);
			WARN_ON(p != folio_address(folio));
			ks = folio_size(folio);
		} else {
			s = folio_slab(folio)->slab_cache;
			orig_size = get_orig_size(s, (void *)p);
			ks = s->object_size;
		}
	}

	/* If the old object doesn't fit, allocate a bigger one */
	if (new_size > ks)
		goto alloc_new;

	/* Zero out spare memory. */
	if (want_init_on_alloc(flags)) {
		kasan_disable_current();
		if (orig_size && orig_size < new_size)
			memset((void *)p + orig_size, 0, new_size - orig_size);
		else
			memset((void *)p + new_size, 0, ks - new_size);
		kasan_enable_current();
	}

	/* Setup kmalloc redzone when needed */
	if (s && slub_debug_orig_size(s)) {
		set_orig_size(s, (void *)p, new_size);
		if (s->flags & SLAB_RED_ZONE && new_size < ks)
			memset_no_sanitize_memory((void *)p + new_size,
						SLUB_RED_ACTIVE, ks - new_size);
	}

	p = kasan_krealloc((void *)p, new_size, flags);
	return (void *)p;

alloc_new:

	ret = kmalloc_node_track_caller_noprof(new_size, flags, NUMA_NO_NODE, _RET_IP_);
	if (ret && p) {
		/* Disable KASAN checks as the object's redzone is accessed. */
		kasan_disable_current();
		memcpy(ret, kasan_reset_tag(p), orig_size ?: ks);
		kasan_enable_current();
	}

	return ret;
}

Thanks,
Feng

#18 lore raw
Quoted 14 lines · Feng Tang
Quoted 6 lines · Vlastimil Babka
Quoted 4 lines · Feng Tang
Quoted 1 line · Vlastimil Babka

On 10/14/24 09:52, Feng Tang wrote:

OK, originally I tried not to expose internals of __ksize(). Let me try this way.

Quoted 6 lines

ksize() makes assumptions that a user outside of slab itself is calling it.

But we (well mostly Kees) also introduced kmalloc_size_roundup() to avoid querying ksize() for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if the skip_orig_size_check() mechanism can be removed now?

Quoted 4 lines

I did a quick grep, and fortunately it seems that the ksize() user are much less than before. We used to see some trouble in network code, which is now very clean without the need to skip orig_size check. Will check other call site later.

Right -- only things that are performing a reallocation should be using ksize(). e.g. see __slab_build_skb()

Signature · 1 line
Kees Cook
author
#19 lore raw
In reply to Feng Tang
Quoted 16 lines · Feng
Quoted 8 lines · Vlastimil Babka
Quoted 7 lines
Quoted 4 lines

So I think in __do_krealloc() we should do things manually to determine ks and not call ksize(). Just not break any of the cases ksize() handles (kfence, large kmalloc).

OK, originally I tried not to expose internals of __ksize(). Let me try this way.

Quoted 6 lines

ksize() makes assumptions that a user outside of slab itself is calling it.

But we (well mostly Kees) also introduced kmalloc_size_roundup() to avoid querying ksize() for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if the skip_orig_size_check() mechanism can be removed now?

Quoted 4 lines

I did a quick grep, and fortunately it seems that the ksize() user are much less than before. We used to see some trouble in network code, which is now very clean without the need to skip orig_size check. Will check other call site later.

I did more further check about ksize() usage, and there are still some places to be handled. The thing stands out is kfree_sensitive(), and another potential one is sound/soc/codecs/cs-amp-lib-test.c

Some details:

* Thanks to Kees Cook, who has cured many cases of ksize() as below:

  drivers/base/devres.c:        total_old_size = ksize(container_of(ptr, struct devres, data));
  drivers/net/ethernet/intel/igb/igb_main.c:        } else if (size > ksize(q_vector)) {   
  net/core/skbuff.c:        *size = ksize(data);
  net/openvswitch/flow_netlink.c:        new_acts_size = max(next_offset + req_size, ksize(*sfa) * 2);
  kernel/bpf/verifier.c:        alloc_bytes = max(ksize(orig), kmalloc_size_roundup(bytes));

* Some callers use ksize() mostly for calculation or sanity check, and not for accessing those extra space, which are fine:

  drivers/gpu/drm/drm_managed.c:        WARN_ON(dev + 1 > (struct drm_device *) (container + ksize(container)));
  lib/kunit/string-stream-test.c:        actual_bytes_used = ksize(stream);
  lib/kunit/string-stream-test.c:                actual_bytes_used += ksize(frag_container);
  lib/kunit/string-stream-test.c:                actual_bytes_used += ksize(frag_container->fragment);
  mm/nommu.c:                return ksize(objp);
  mm/util.c:                        memcpy(n, kasan_reset_tag(p), ksize(p));
  security/tomoyo/gc.c:        tomoyo_memory_used[TOMOYO_MEMORY_POLICY] -= ksize(ptr);
  security/tomoyo/memory.c:                const size_t s = ksize(ptr);
  drivers/md/dm-vdo/memory-alloc.c:                        add_kmalloc_block(ksize(p));
  drivers/md/dm-vdo/memory-alloc.c:                add_kmalloc_block(ksize(p));
  drivers/md/dm-vdo/memory-alloc.c:                        remove_kmalloc_block(ksize(ptr));

* One usage may need to be handled

  sound/soc/codecs/cs-amp-lib-test.c:        KUNIT_ASSERT_GE_MSG(test, ksize(buf), priv->cal_blob->size, "Buffer to small");

* bigger problem is the kfree_sensitive(), which will use ksize() to get the total size and then zero all of them.

One solution for this could be get the kmem_cache first, and do the skip_orig_size_check()

Thanks,
Feng

#20 lore raw
Quoted 54 lines · Feng Tang
Quoted 16 lines · Feng
Quoted 8 lines · Vlastimil Babka
Quoted 7 lines
Quoted 4 lines

So I think in __do_krealloc() we should do things manually to determine ks and not call ksize(). Just not break any of the cases ksize() handles (kfence, large kmalloc).

OK, originally I tried not to expose internals of __ksize(). Let me try this way.

Quoted 6 lines

ksize() makes assumptions that a user outside of slab itself is calling it.

But we (well mostly Kees) also introduced kmalloc_size_roundup() to avoid querying ksize() for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if the skip_orig_size_check() mechanism can be removed now?

Quoted 4 lines

I did a quick grep, and fortunately it seems that the ksize() user are much less than before. We used to see some trouble in network code, which is now very clean without the need to skip orig_size check. Will check other call site later.

I did more further check about ksize() usage, and there are still some places to be handled. The thing stands out is kfree_sensitive(), and another potential one is sound/soc/codecs/cs-amp-lib-test.c

Some details:

* Thanks to Kees Cook, who has cured many cases of ksize() as below:

  drivers/base/devres.c:        total_old_size = ksize(container_of(ptr, struct devres, data));
  drivers/net/ethernet/intel/igb/igb_main.c:        } else if (size > ksize(q_vector)) {   
  net/core/skbuff.c:        *size = ksize(data);
  net/openvswitch/flow_netlink.c:        new_acts_size = max(next_offset + req_size, ksize(*sfa) * 2);
  kernel/bpf/verifier.c:        alloc_bytes = max(ksize(orig), kmalloc_size_roundup(bytes));

* Some callers use ksize() mostly for calculation or sanity check, and not for accessing those extra space, which are fine:

  drivers/gpu/drm/drm_managed.c:        WARN_ON(dev + 1 > (struct drm_device *) (container + ksize(container)));
  lib/kunit/string-stream-test.c:        actual_bytes_used = ksize(stream);
  lib/kunit/string-stream-test.c:                actual_bytes_used += ksize(frag_container);
  lib/kunit/string-stream-test.c:                actual_bytes_used += ksize(frag_container->fragment);
  mm/nommu.c:                return ksize(objp);
  mm/util.c:                        memcpy(n, kasan_reset_tag(p), ksize(p));
  security/tomoyo/gc.c:        tomoyo_memory_used[TOMOYO_MEMORY_POLICY] -= ksize(ptr);
  security/tomoyo/memory.c:                const size_t s = ksize(ptr);
  drivers/md/dm-vdo/memory-alloc.c:                        add_kmalloc_block(ksize(p));
  drivers/md/dm-vdo/memory-alloc.c:                add_kmalloc_block(ksize(p));
  drivers/md/dm-vdo/memory-alloc.c:                        remove_kmalloc_block(ksize(ptr));

* One usage may need to be handled

Quoted 7 lines
  sound/soc/codecs/cs-amp-lib-test.c:        KUNIT_ASSERT_GE_MSG(test, ksize(buf), priv->cal_blob->size, "Buffer to small");

* bigger problem is the kfree_sensitive(), which will use ksize() to get the total size and then zero all of them.

One solution for this could be get the kmem_cache first, and do the skip_orig_size_check()

Maybe add a parameter for __ksize() that controls if we do skip_orig_size_check(), current ksize() will pass "false" to it (once remaining wrong users are handled), then another ksize_internal() variant will pass "true" and be used from kfree_sensitive()?

Quoted 2 lines

Thanks,
Feng

author
#21 lore raw
Quoted 60 lines · Vlastimil Babka
Quoted 59 lines · Feng Tang
Quoted 16 lines · Feng
Quoted 8 lines · Vlastimil Babka
Quoted 7 lines
Quoted 4 lines

So I think in __do_krealloc() we should do things manually to determine ks and not call ksize(). Just not break any of the cases ksize() handles (kfence, large kmalloc).

OK, originally I tried not to expose internals of __ksize(). Let me try this way.

Quoted 6 lines

ksize() makes assumptions that a user outside of slab itself is calling it.

But we (well mostly Kees) also introduced kmalloc_size_roundup() to avoid querying ksize() for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if the skip_orig_size_check() mechanism can be removed now?

Quoted 4 lines

I did a quick grep, and fortunately it seems that the ksize() user are much less than before. We used to see some trouble in network code, which is now very clean without the need to skip orig_size check. Will check other call site later.

I did more further check about ksize() usage, and there are still some places to be handled. The thing stands out is kfree_sensitive(), and another potential one is sound/soc/codecs/cs-amp-lib-test.c

Some details:

* Thanks to Kees Cook, who has cured many cases of ksize() as below:

  drivers/base/devres.c:        total_old_size = ksize(container_of(ptr, struct devres, data));
  drivers/net/ethernet/intel/igb/igb_main.c:        } else if (size > ksize(q_vector)) {   
  net/core/skbuff.c:        *size = ksize(data);
  net/openvswitch/flow_netlink.c:        new_acts_size = max(next_offset + req_size, ksize(*sfa) * 2);
  kernel/bpf/verifier.c:        alloc_bytes = max(ksize(orig), kmalloc_size_roundup(bytes));

* Some callers use ksize() mostly for calculation or sanity check, and not for accessing those extra space, which are fine:

  drivers/gpu/drm/drm_managed.c:        WARN_ON(dev + 1 > (struct drm_device *) (container + ksize(container)));
  lib/kunit/string-stream-test.c:        actual_bytes_used = ksize(stream);
  lib/kunit/string-stream-test.c:                actual_bytes_used += ksize(frag_container);
  lib/kunit/string-stream-test.c:                actual_bytes_used += ksize(frag_container->fragment);
  mm/nommu.c:                return ksize(objp);
  mm/util.c:                        memcpy(n, kasan_reset_tag(p), ksize(p));
  security/tomoyo/gc.c:        tomoyo_memory_used[TOMOYO_MEMORY_POLICY] -= ksize(ptr);
  security/tomoyo/memory.c:                const size_t s = ksize(ptr);
  drivers/md/dm-vdo/memory-alloc.c:                        add_kmalloc_block(ksize(p));
  drivers/md/dm-vdo/memory-alloc.c:                add_kmalloc_block(ksize(p));
  drivers/md/dm-vdo/memory-alloc.c:                        remove_kmalloc_block(ksize(ptr));

* One usage may need to be handled

  sound/soc/codecs/cs-amp-lib-test.c:        KUNIT_ASSERT_GE_MSG(test, ksize(buf), priv->cal_blob->size, "Buffer to small");

* bigger problem is the kfree_sensitive(), which will use ksize() to get the total size and then zero all of them.

Quoted 7 lines
Quoted 2 lines

One solution for this could be get the kmem_cache first, and do the skip_orig_size_check()

Maybe add a parameter for __ksize() that controls if we do skip_orig_size_check(), current ksize() will pass "false" to it (once remaining wrong users are handled), then another ksize_internal() variant will pass "true" and be used from kfree_sensitive()?

Sounds good to me! And for future wrong usages of ksize(), we can fix them case by case when they are deteced.

Thanks,
Feng

#22 lore raw
In reply to Vlastimil Babka
Quoted 55 lines · Vlastimil Babka
Quoted 20 lines · Feng Tang
Quoted 19 lines · Vlastimil Babka
Quoted 18 lines · Feng Tang

On Fri, Oct 04, 2024 at 05:52:10PM +0800, Vlastimil Babka wrote:
Thanks for the suggestion!

As there were error report about the NULL slab for big kmalloc object, how about the following code for

__do_krealloc(const void *p, size_t new_size, gfp_t flags)

{
	void *ret;
	size_t ks = 0;
	int orig_size = 0;
	struct kmem_cache *s = NULL;

	/* Check for double-free. */
	if (likely(!ZERO_OR_NULL_PTR(p))) {
		if (!kasan_check_byte(p))
			return NULL;
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
		ks = ksize(p);

I think this will result in __ksize() doing
skip_orig_size_check(folio_slab(folio)->slab_cache, object);
and we don't want that?

I think that's fine. As later code will re-set the orig_size anyway.

But you also read it first.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
		/* Some objects have no orig_size, like big kmalloc case */
		if (is_kfence_address(p)) {
			orig_size = kfence_ksize(p);
		} else if (virt_to_slab(p)) {
			s = virt_to_cache(p);
			orig_size = get_orig_size(s, (void *)p);

here.

Quoted 4 lines
Quoted 3 lines
Quoted 1 line
		}

Also the checks below repeat some of the checks of ksize().

Quoted 8 lines

Yes, there is some redundancy, mostly the virt_to_slab()

Quoted 3 lines

So I think in __do_krealloc() we should do things manually to determine ks and not call ksize(). Just not break any of the cases ksize() handles (kfence, large kmalloc).

OK, originally I tried not to expose internals of __ksize(). Let me try this way.

ksize() makes assumptions that a user outside of slab itself is calling it.

Quoted 7 lines

But we (well mostly Kees) also introduced kmalloc_size_roundup() to avoid querying ksize() for the purposes of writing beyond the original kmalloc(size) up to the bucket size. So maybe we can also investigate if the skip_orig_size_check() mechanism can be removed now?

Still I think __do_krealloc() should rather do its own thing and not call ksize().

The goal was to avoid having users of the allocation APIs change the sizes of allocations without calling into realloc. This is because otherwise the "alloc_size" attribute used by compilers inform __builtin_dynamic_object_size() can get confused:

ptr = alloc(less_than_bucket_size);
...
size = ksize(ptr); /* larger size! */
memcpy(ptr, src, size); /* compiler instrumentation doesn't see that ptr "grows" */

So the callers use kmalloc_size_roundup() to just allocate the rounded up size immediately. Internally, the allocator can do what it wants.

Signature · 1 line
Kees Cook