Skip to contentSkip to conversation

mm/slab: reduce slab accounting memory overhead by allocating slabobj_ext metadata within unsed slab space

PATCH Applied Harry Yoo 2 participants 18 messages Jan 13

Subsystems (6)
lore
Share & tools
lore

Harry Yoo

author
#0 lore raw

v6 cover mm/slab: reduce slab accounting memory overhead by allocating slabobj_ext metadata within unsed slab space

V5: https://lore.kernel.org/linux-mm/[email protected]
V5 -> V6:

- Patch 1: Added Closes: tag for related discussion (Vlastimil)
https://lore.kernel.org/linux-mm/[email protected]

- Patch 3: Addressed Vlastimil's comments
https://lore.kernel.org/linux-mm/[email protected]

- Patch 4: Fixed incorrect function prototype of slab_obj_ext() on !CONFIG_SLAB_OBJ_EXT builds and kept pointer type in free_slab_obj_exts() (Hao, Vlastimil)
https://lore.kernel.org/linux-mm/n6kyluk3nahdxytwek4ijzy4en6mc6ps7fjjgftww4ith7llom@cijm4who24w2
https://lore.kernel.org/linux-mm/[email protected]

- Patch 7, 9: Rewrote obj_exts_in_slab() to check if the pointer is within the slab's range, and distinguish by stride (Vlastimil)
https://lore.kernel.org/linux-mm/[email protected]

- Patch 9: Fixed potentioal memory leak due to incorrect impl. of obj_exts_in_object() (Vlastimil)
https://lore.kernel.org/linux-mm/[email protected]/

- Patch 9: Fixed incorrect ksize() implementation (Hao)
https://lore.kernel.org/linux-mm/fgx3lapibabra4x7tewx55nuvxz235ruvm3agpprjbdcmt3rc6@h54ln5tfdssz

When CONFIG_MEMCG and CONFIG_MEM_ALLOC_PROFILING are enabled, the kernel allocates two pointers per object: one for the memory cgroup (actually, obj_cgroup) to which it belongs, and another for the code location that requested the allocation.

In two special cases, this overhead can be eliminated by allocating slabobj_ext metadata from unused space within a slab:

Case 1. The "leftover" space after the last slab object is larger than
the size of an array of slabobj_ext.

Case 2. The per-object alignment padding is larger than
sizeof(struct slabobj_ext).

For these two cases, one or two pointers can be saved per slab object.
Examples: ext4 inode cache (case 1) and xfs inode cache (case 2). That's approximately 0.7-0.8% (memcg) or 1.5-1.6% (memcg + mem profiling) of the total inode cache size.

Implementing case 2 is not straightforward, because the existing code assumes that slab->obj_exts is an array of slabobj_ext, while case 2 breaks the assumption.

As suggested by Vlastimil, abstract access to individual slabobj_ext metadata via a new helper named slab_obj_ext():

static inline struct slabobj_ext *slab_obj_ext(struct slab *slab,
unsigned long obj_exts, unsigned int index)

{
        return (struct slabobj_ext *)(obj_exts + slab_get_stride(slab) * index);
} 

In the normal case (including case 1), slab->obj_exts points to an array of slabobj_ext, and the stride is sizeof(struct slabobj_ext).

In case 2, the stride is s->size and
slab->obj_exts = slab_address(slab) + s->red_left_pad + (offset of slabobj_ext)

With this approach, the memcg charging fastpath doesn't need to care the storage method of slabobj_ext.

Harry Yoo (9):
mm/slab: use unsigned long for orig_size to ensure proper metadata
align
mm/slab: allow specifying free pointer offset when using constructor
ext4: specify the free pointer offset for ext4_inode_cache mm/slab: abstract slabobj_ext access via new slab_obj_ext() helper mm/slab: use stride to access slabobj_ext mm/memcontrol,alloc_tag: handle slabobj_ext access under KASAN poison mm/slab: save memory by allocating slabobj_ext array from leftover mm/slab: move [__]ksize and slab_ksize() to mm/slub.c mm/slab: place slabobj_ext metadata in unused space within s->size

 fs/ext4/super.c      |  20 +-
 include/linux/slab.h |  39 ++--
 mm/memcontrol.c      |  31 +++-
 mm/slab.h            | 145 +++++++++++----
 mm/slab_common.c     |  69 +------
 mm/slub.c            | 429 +++++++++++++++++++++++++++++++++++++------
 6 files changed, 552 insertions(+), 181 deletions(-)
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH v6 1/9 mm/slab: use unsigned long for orig_size to ensure proper metadata align

When both KASAN and SLAB_STORE_USER are enabled, accesses to struct kasan_alloc_meta fields can be misaligned on 64-bit architectures. This occurs because orig_size is currently defined as unsigned int, which only guarantees 4-byte alignment. When struct kasan_alloc_meta is placed after orig_size, it may end up at a 4-byte boundary rather than the required 8-byte boundary on 64-bit systems.

Note that 64-bit architectures without HAVE_EFFICIENT_UNALIGNED_ACCESS are assumed to require 64-bit accesses to be 64-bit aligned. See HAVE_64BIT_ALIGNED_ACCESS and commit adab66b71abf ("Revert:
"ring-buffer: Remove HAVE_64BIT_ALIGNED_ACCESS"") for more details.

Change orig_size from unsigned int to unsigned long to ensure proper alignment for any subsequent metadata. This should not waste additional memory because kmalloc objects are already aligned to at least ARCH_KMALLOC_MINALIGN.

Fixes 6edf2576a6cc ("mm/slub: enable debugging memory wasting of kmalloc")
---
 mm/slub.c | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)
mm/slub.c +7 -7
index 64d71a728d3d..2494ca8080f5 100644
@@ -857,7 +857,7 @@ static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab,
  * request size in the meta data area, for better debug and sanity check.
  */
 static inline void set_orig_size(struct kmem_cache *s,
-				void *object, unsigned int orig_size)
+				void *object, unsigned long orig_size)
 {
 	void *p = kasan_reset_tag(object);
 
@@ -867,10 +867,10 @@ static inline void set_orig_size(struct kmem_cache *s,
 	p += get_info_end(s);
 	p += sizeof(struct track) * 2;
 
-	*(unsigned int *)p = orig_size;
+	*(unsigned long *)p = orig_size;
 }
 
-static inline unsigned int get_orig_size(struct kmem_cache *s, void *object)
+static inline unsigned long get_orig_size(struct kmem_cache *s, void *object)
 {
 	void *p = kasan_reset_tag(object);
 
@@ -883,7 +883,7 @@ static inline unsigned int get_orig_size(struct kmem_cache *s, void *object)
 	p += get_info_end(s);
 	p += sizeof(struct track) * 2;
 
-	return *(unsigned int *)p;
+	return *(unsigned long *)p;
 }
 
 #ifdef CONFIG_SLUB_DEBUG
@@ -1198,7 +1198,7 @@ static void print_trailer(struct kmem_cache *s, struct slab *slab, u8 *p)
 		off += 2 * sizeof(struct track);
 
 	if (slub_debug_orig_size(s))
-		off += sizeof(unsigned int);
+		off += sizeof(unsigned long);
 
 	off += kasan_metadata_size(s, false);
 
@@ -1410,7 +1410,7 @@ static int check_pad_bytes(struct kmem_cache *s, struct slab *slab, u8 *p)
 		off += 2 * sizeof(struct track);
 
 		if (s->flags & SLAB_KMALLOC)
-			off += sizeof(unsigned int);
+			off += sizeof(unsigned long);
 	}
 
 	off += kasan_metadata_size(s, false);
@@ -7961,7 +7961,7 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s)
 
 		/* Save the original kmalloc request size */
 		if (flags & SLAB_KMALLOC)
-			size += sizeof(unsigned int);
+			size += sizeof(unsigned long);
 	}
 #endif
 
Signature · 1 line
2.43.0
author
#2 lore raw

PATCH v6 2/9 mm/slab: allow specifying free pointer offset when using constructor

In reply to Harry Yoo

When a slab cache has a constructor, the free pointer is placed after the object because certain fields must not be overwritten even after the object is freed.

However, some fields that the constructor does not initialize can safely be overwritten after free. Allow specifying the free pointer offset within the object, reducing the overall object size when some fields can be reused for the free pointer.

Adjust the document accordingly.

---
 include/linux/slab.h | 30 ++++++++++++++++--------------
 mm/slab_common.c     |  2 +-
 mm/slub.c            |  6 ++++--
 3 files changed, 21 insertions(+), 17 deletions(-)
include/linux/slab.h +16 -14
index 2482992248dc..4554c04a9bd7 100644
@@ -299,24 +299,26 @@ struct kmem_cache_args {
 	unsigned int usersize;
 	/**
 	 * @freeptr_offset: Custom offset for the free pointer
-	 * in &SLAB_TYPESAFE_BY_RCU caches
+	 * in caches with &SLAB_TYPESAFE_BY_RCU or @ctor
 	 *
-	 * By default &SLAB_TYPESAFE_BY_RCU caches place the free pointer
-	 * outside of the object. This might cause the object to grow in size.
-	 * Cache creators that have a reason to avoid this can specify a custom
-	 * free pointer offset in their struct where the free pointer will be
-	 * placed.
+	 * By default, &SLAB_TYPESAFE_BY_RCU and @ctor caches place the free
+	 * pointer outside of the object. This might cause the object to grow
+	 * in size. Cache creators that have a reason to avoid this can specify
+	 * a custom free pointer offset in their data structure where the free
+	 * pointer will be placed.
 	 *
-	 * Note that placing the free pointer inside the object requires the
-	 * caller to ensure that no fields are invalidated that are required to
-	 * guard against object recycling (See &SLAB_TYPESAFE_BY_RCU for
-	 * details).
+	 * For caches with &SLAB_TYPESAFE_BY_RCU, the caller must ensure that
+	 * the free pointer does not overlay fields required to guard against
+	 * object recycling (See &SLAB_TYPESAFE_BY_RCU for details).
 	 *
-	 * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
-	 * is specified, %use_freeptr_offset must be set %true.
+	 * For caches with @ctor, the caller must ensure that the free pointer
+	 * does not overlay fields initialized by the constructor.
+	 *
+	 * Currently, only caches with &SLAB_TYPESAFE_BY_RCU or @ctor
+	 * may specify @freeptr_offset.
 	 *
-	 * Note that @ctor currently isn't supported with custom free pointers
-	 * as a @ctor requires an external free pointer.
+	 * Using %0 as a value for @freeptr_offset is valid. If @freeptr_offset
+	 * is specified, @use_freeptr_offset must be set %true.
 	 */
 	unsigned int freeptr_offset;
 	/**
mm/slab_common.c +1 -1
index eed7ea556cb1..c4cf9ed2ec92 100644
@@ -231,7 +231,7 @@ static struct kmem_cache *create_cache(const char *name,
 	err = -EINVAL;
 	if (args->use_freeptr_offset &&
 	    (args->freeptr_offset >= object_size ||
-	     !(flags & SLAB_TYPESAFE_BY_RCU) ||
+	     (!(flags & SLAB_TYPESAFE_BY_RCU) && !args->ctor) ||
 	     !IS_ALIGNED(args->freeptr_offset, __alignof__(freeptr_t))))
 		goto out;
 
mm/slub.c +4 -2
index 2494ca8080f5..5f75e0d5cf16 100644
@@ -7919,7 +7919,8 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s)
 	s->inuse = size;
 
 	if (((flags & SLAB_TYPESAFE_BY_RCU) && !args->use_freeptr_offset) ||
-	    (flags & SLAB_POISON) || s->ctor ||
+	    (flags & SLAB_POISON) ||
+	    (s->ctor && !args->use_freeptr_offset) ||
 	    ((flags & SLAB_RED_ZONE) &&
 	     (s->object_size < sizeof(void *) || slub_debug_orig_size(s)))) {
 		/*
@@ -7940,7 +7941,8 @@ static int calculate_sizes(struct kmem_cache_args *args, struct kmem_cache *s)
 		 */
 		s->offset = size;
 		size += sizeof(void *);
-	} else if ((flags & SLAB_TYPESAFE_BY_RCU) && args->use_freeptr_offset) {
+	} else if (((flags & SLAB_TYPESAFE_BY_RCU) || s->ctor) &&
+			args->use_freeptr_offset) {
 		s->offset = args->freeptr_offset;
 	} else {
 		/*
Signature · 1 line
2.43.0
author
#3 lore raw

PATCH v6 3/9 ext4: specify the free pointer offset for ext4_inode_cache

In reply to Harry Yoo

Convert ext4_inode_cache to use the kmem_cache_args interface and specify a free pointer offset.

Since ext4_inode_cache uses a constructor, the free pointer would be placed after the object to prevent overwriting fields used by the constructor.

However, some fields such as ->i_flags are not used by the constructor and can safely be repurposed for the free pointer.

Specify the free pointer offset at i_flags to reduce the object size.

---
 fs/ext4/super.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)
fs/ext4/super.c +13 -6
index 87205660c5d0..6f1c2c497871 100644
@@ -1491,12 +1491,19 @@ static void init_once(void *foo)
 
 static int __init init_inodecache(void)
 {
-	ext4_inode_cachep = kmem_cache_create_usercopy("ext4_inode_cache",
-				sizeof(struct ext4_inode_info), 0,
-				SLAB_RECLAIM_ACCOUNT | SLAB_ACCOUNT,
-				offsetof(struct ext4_inode_info, i_data),
-				sizeof_field(struct ext4_inode_info, i_data),
-				init_once);
+	struct kmem_cache_args args = {
+		.useroffset = offsetof(struct ext4_inode_info, i_data),
+		.usersize = sizeof_field(struct ext4_inode_info, i_data),
+		.use_freeptr_offset = true,
+		.freeptr_offset = offsetof(struct ext4_inode_info, i_flags),
+		.ctor = init_once,
+	};
+
+	ext4_inode_cachep = kmem_cache_create("ext4_inode_cache",
+				sizeof(struct ext4_inode_info),
+				&args,
+				SLAB_RECLAIM_ACCOUNT | SLAB_ACCOUNT);
+
 	if (ext4_inode_cachep == NULL)
 		return -ENOMEM;
 	return 0;
Signature · 1 line
2.43.0
author
#4 lore raw

PATCH v6 4/9 mm/slab: abstract slabobj_ext access via new slab_obj_ext() helper

In reply to Harry Yoo

Currently, the slab allocator assumes that slab->obj_exts is a pointer to an array of struct slabobj_ext objects. However, to support storage methods where struct slabobj_ext is embedded within objects, the slab allocator should not make this assumption. Instead of directly dereferencing the slabobj_exts array, abstract access to struct slabobj_ext via helper functions.

Introduce a new API slabobj_ext metadata access:

slab_obj_ext(slab, obj_exts, index) - returns the pointer to struct slabobj_ext element at the given index.

Directly dereferencing the return value of slab_obj_exts() is no longer allowed. Instead, slab_obj_ext() must always be used to access individual struct slabobj_ext objects.

Convert all users to use these APIs.
No functional changes intended.

---
 mm/memcontrol.c | 23 ++++++++++++++++-------
 mm/slab.h       | 43 +++++++++++++++++++++++++++++++++++++------
 mm/slub.c       | 45 ++++++++++++++++++++++++++-------------------
 3 files changed, 79 insertions(+), 32 deletions(-)
mm/memcontrol.c +16 -7
index be810c1fbfc3..fd9105a953b0 100644
@@ -2596,7 +2596,8 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
 	 * Memcg membership data for each individual object is saved in
 	 * slab->obj_exts.
 	 */
-	struct slabobj_ext *obj_exts;
+	unsigned long obj_exts;
+	struct slabobj_ext *obj_ext;
 	unsigned int off;
 
 	obj_exts = slab_obj_exts(slab);
@@ -2604,8 +2605,9 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
 		return NULL;
 
 	off = obj_to_index(slab->slab_cache, slab, p);
-	if (obj_exts[off].objcg)
-		return obj_cgroup_memcg(obj_exts[off].objcg);
+	obj_ext = slab_obj_ext(slab, obj_exts, off);
+	if (obj_ext->objcg)
+		return obj_cgroup_memcg(obj_ext->objcg);
 
 	return NULL;
 }
@@ -3191,6 +3193,9 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
 	}
 
 	for (i = 0; i < size; i++) {
+		unsigned long obj_exts;
+		struct slabobj_ext *obj_ext;
+
 		slab = virt_to_slab(p[i]);
 
 		if (!slab_obj_exts(slab) &&
@@ -3213,29 +3218,33 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
 					slab_pgdat(slab), cache_vmstat_idx(s)))
 			return false;
 
+		obj_exts = slab_obj_exts(slab);
 		off = obj_to_index(s, slab, p[i]);
+		obj_ext = slab_obj_ext(slab, obj_exts, off);
 		obj_cgroup_get(objcg);
-		slab_obj_exts(slab)[off].objcg = objcg;
+		obj_ext->objcg = objcg;
 	}
 
 	return true;
 }
 
 void __memcg_slab_free_hook(struct kmem_cache *s, struct slab *slab,
-			    void **p, int objects, struct slabobj_ext *obj_exts)
+			    void **p, int objects, unsigned long obj_exts)
 {
 	size_t obj_size = obj_full_size(s);
 
 	for (int i = 0; i < objects; i++) {
 		struct obj_cgroup *objcg;
+		struct slabobj_ext *obj_ext;
 		unsigned int off;
 
 		off = obj_to_index(s, slab, p[i]);
-		objcg = obj_exts[off].objcg;
+		obj_ext = slab_obj_ext(slab, obj_exts, off);
+		objcg = obj_ext->objcg;
 		if (!objcg)
 			continue;
 
-		obj_exts[off].objcg = NULL;
+		obj_ext->objcg = NULL;
 		refill_obj_stock(objcg, obj_size, true, -obj_size,
 				 slab_pgdat(slab), cache_vmstat_idx(s));
 		obj_cgroup_put(objcg);
mm/slab.h +37 -6

68 lines not on this page · raw message

mm/slub.c +26 -19

132 lines not on this page · raw message

Signature · 1 line
2.43.0
author
#5 lore raw

PATCH v6 5/9 mm/slab: use stride to access slabobj_ext

In reply to Harry Yoo

Use a configurable stride value when accessing slab object extension metadata instead of assuming a fixed sizeof(struct slabobj_ext).

Store stride value in free bits of slab->counters field. This allows for flexibility in cases where the extension is embedded within slab objects.

Since these free bits exist only on 64-bit, any future optimizations that need to change stride value cannot be enabled on 32-bit architectures.

---
 mm/slab.h | 37 +++++++++++++++++++++++++++++++++----
 mm/slub.c |  2 ++
 2 files changed, 35 insertions(+), 4 deletions(-)
mm/slab.h +33 -4
index dd23d861a8d1..ab7b3e386fbb 100644
@@ -55,6 +55,14 @@ struct freelist_counters {
 					 * that the slab was corrupted
 					 */
 					unsigned frozen:1;
+#ifdef CONFIG_64BIT
+					/*
+					 * Some optimizations use free bits in 'counters' field
+					 * to save memory. In case ->stride field is not available,
+					 * such optimizations are disabled.
+					 */
+					unsigned short stride;
+#endif
 				};
 			};
 		};
@@ -531,6 +539,26 @@ static inline unsigned long slab_obj_exts(struct slab *slab)
 	return obj_exts & ~OBJEXTS_FLAGS_MASK;
 }
 
+#ifdef CONFIG_64BIT
+static inline void slab_set_stride(struct slab *slab, unsigned short stride)
+{
+	slab->stride = stride;
+}
+static inline unsigned short slab_get_stride(struct slab *slab)
+{
+	return slab->stride;
+}
+#else
+static inline void slab_set_stride(struct slab *slab, unsigned short stride)
+{
+	VM_WARN_ON_ONCE(stride != sizeof(struct slabobj_ext));
+}
+static inline unsigned short slab_get_stride(struct slab *slab)
+{
+	return sizeof(struct slabobj_ext);
+}
+#endif
+
 /*
  * slab_obj_ext - get the pointer to the slab object extension metadata
  * associated with an object in a slab.
@@ -544,12 +572,9 @@ static inline struct slabobj_ext *slab_obj_ext(struct slab *slab,
 					       unsigned long obj_exts,
 					       unsigned int index)
 {
-	struct slabobj_ext *obj_ext;
-
 	VM_WARN_ON_ONCE(obj_exts != slab_obj_exts(slab));
 
-	obj_ext = (struct slabobj_ext *)obj_exts;
-	return &obj_ext[index];
+	return (struct slabobj_ext *)(obj_exts + slab_get_stride(slab) * index);
 }
 
 int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
@@ -569,6 +594,10 @@ static inline struct slabobj_ext *slab_obj_ext(struct slab *slab,
 	return NULL;
 }
 
+static inline void slab_set_stride(struct slab *slab, unsigned int stride) { }
+static inline unsigned int slab_get_stride(struct slab *slab) { return 0; }
+
+
 #endif /* CONFIG_SLAB_OBJ_EXT */
 
 static inline enum node_stat_item cache_vmstat_idx(struct kmem_cache *s)
mm/slub.c +2 -0
index 7c855b84e9e2..41c541381627 100644
@@ -2163,6 +2163,8 @@ int alloc_slab_obj_exts(struct slab *slab, struct kmem_cache *s,
 retry:
 	old_exts = READ_ONCE(slab->obj_exts);
 	handle_failed_objexts_alloc(old_exts, vec, objects);
+	slab_set_stride(slab, sizeof(struct slabobj_ext));
+
 	if (new_slab) {
 		/*
 		 * If the slab is brand new and nobody can yet access its
Signature · 1 line
2.43.0
author
#6 lore raw

PATCH v6 6/9 mm/memcontrol,alloc_tag: handle slabobj_ext access under KASAN poison

In reply to Harry Yoo

In the near future, slabobj_ext may reside outside the allocated slab object range within a slab, which could be reported as an out-of-bounds access by KASAN.

As suggested by Andrey Konovalov [1], explicitly disable KASAN and KMSAN checks when accessing slabobj_ext within slab allocator, memory profiling, and memory cgroup code. While an alternative approach could be to unpoison slabobj_ext, out-of-bounds accesses outside the slab allocator are generally more common.

Move metadata_access_enable()/disable() helpers to mm/slab.h so that it can be used outside mm/slub.c. However, as suggested by Suren Baghdasaryan [2], instead of calling them directly from mm code (which is more prone to errors), change users to access slabobj_ext via get/put APIs:

- Users should call get_slab_obj_exts() to access slabobj_metadata and call put_slab_obj_exts() when it's done.

- From now on, accessing it outside the section covered by get_slab_obj_exts() ~ put_slab_obj_exts() is illegal. This ensures that accesses to slabobj_ext metadata won't be reported as access violations.

Call kasan_reset_tag() in slab_obj_ext() before returning the address to prevent SW or HW tag-based KASAN from reporting false positives.

---
 mm/memcontrol.c | 12 +++++++--
 mm/slab.h       | 54 +++++++++++++++++++++++++++++++++++---
 mm/slub.c       | 69 ++++++++++++++++++++++++-------------------------
 3 files changed, 95 insertions(+), 40 deletions(-)
mm/memcontrol.c +10 -2
index fd9105a953b0..50ca00122571 100644
@@ -2604,10 +2604,16 @@ struct mem_cgroup *mem_cgroup_from_obj_slab(struct slab *slab, void *p)
 	if (!obj_exts)
 		return NULL;
 
+	get_slab_obj_exts(obj_exts);
 	off = obj_to_index(slab->slab_cache, slab, p);
 	obj_ext = slab_obj_ext(slab, obj_exts, off);
-	if (obj_ext->objcg)
-		return obj_cgroup_memcg(obj_ext->objcg);
+	if (obj_ext->objcg) {
+		struct obj_cgroup *objcg = obj_ext->objcg;
+
+		put_slab_obj_exts(obj_exts);
+		return obj_cgroup_memcg(objcg);
+	}
+	put_slab_obj_exts(obj_exts);
 
 	return NULL;
 }
@@ -3219,10 +3225,12 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *lru,
 			return false;
 
 		obj_exts = slab_obj_exts(slab);
+		get_slab_obj_exts(obj_exts);
 		off = obj_to_index(s, slab, p[i]);
 		obj_ext = slab_obj_ext(slab, obj_exts, off);
 		obj_cgroup_get(objcg);
 		obj_ext->objcg = objcg;
+		put_slab_obj_exts(obj_exts);
 	}
 
 	return true;
mm/slab.h +51 -3

85 lines not on this page · raw message

mm/slub.c +34 -35

155 lines not on this page · raw message

Signature · 1 line
2.43.0
author
#7 lore raw

PATCH v6 7/9 mm/slab: save memory by allocating slabobj_ext array from leftover

In reply to Harry Yoo

The leftover space in a slab is always smaller than s->size, and kmem caches for large objects that are not power-of-two sizes tend to have a greater amount of leftover space per slab. In some cases, the leftover space is larger than the size of the slabobj_ext array for the slab.

An excellent example of such a cache is ext4_inode_cache. On my system, the object size is 1136, with a preferred order of 3, 28 objects per slab, and 960 bytes of leftover space per slab.

Since the size of the slabobj_ext array is only 224 bytes (w/o mem profiling) or 448 bytes (w/ mem profiling) per slab, the entire array fits within the leftover space.

Allocate the slabobj_exts array from this unused space instead of using kcalloc() when it is large enough. The array is allocated from unused space only when creating new slabs, and it doesn't try to utilize unused space if alloc_slab_obj_exts() is called after slab creation because implementing lazy allocation involves more expensive synchronization.

The implementation and evaluation of lazy allocation from unused space is left as future-work. As pointed by Vlastimil Babka [1], it could be beneficial when a slab cache without SLAB_ACCOUNT can be created, and some of the allocations from the cache use __GFP_ACCOUNT. For example, xarray does that.

To avoid unnecessary overhead when MEMCG (with SLAB_ACCOUNT) and MEM_ALLOC_PROFILING are not used for the cache, allocate the slabobj_ext array only when either of them is enabled on slab allocation.

[ MEMCG=y, MEM_ALLOC_PROFILING=n ]

Before patch (creating ~2.64M directories on ext4):
Slab: 4747880 kB
SReclaimable: 4169652 kB
SUnreclaim: 578228 kB

After patch (creating ~2.64M directories on ext4):
Slab: 4724020 kB
SReclaimable: 4169188 kB
SUnreclaim: 554832 kB (-22.84 MiB)

Enjoy the memory savings!

---
 mm/slub.c | 154 ++++++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 149 insertions(+), 5 deletions(-)
mm/slub.c +149 -5
index 14c58038a37e..e4a4e01de42f 100644
@@ -886,6 +886,97 @@ static inline unsigned long get_orig_size(struct kmem_cache *s, void *object)
 	return *(unsigned long *)p;
 }
 
+#ifdef CONFIG_SLAB_OBJ_EXT
+
+/*
+ * Check if memory cgroup or memory allocation profiling is enabled.
+ * If enabled, SLUB tries to reduce memory overhead of accounting
+ * slab objects. If neither is enabled when this function is called,
+ * the optimization is simply skipped to avoid affecting caches that do not
+ * need slabobj_ext metadata.
+ *
+ * However, this may disable optimization when memory cgroup or memory
+ * allocation profiling is used, but slabs are created too early
+ * even before those subsystems are initialized.
+ */
+static inline bool need_slab_obj_exts(struct kmem_cache *s)
+{
+	if (memcg_kmem_online() && (s->flags & SLAB_ACCOUNT))
+		return true;
+
+	if (mem_alloc_profiling_enabled())
+		return true;
+
+	return false;
+}
+
+static inline unsigned int obj_exts_size_in_slab(struct slab *slab)
+{
+	return sizeof(struct slabobj_ext) * slab->objects;
+}
+
+static inline unsigned long obj_exts_offset_in_slab(struct kmem_cache *s,
+						    struct slab *slab)
+{
+	unsigned long objext_offset;
+
+	objext_offset = s->size * slab->objects;
+	objext_offset = ALIGN(objext_offset, sizeof(struct slabobj_ext));
+	return objext_offset;
+}
+
+static inline bool obj_exts_fit_within_slab_leftover(struct kmem_cache *s,
+						     struct slab *slab)
+{
+	unsigned long objext_offset = obj_exts_offset_in_slab(s, slab);
+	unsigned long objext_size = obj_exts_size_in_slab(slab);
+
+	return objext_offset + objext_size <= slab_size(slab);
+}
+
+static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
+{
+	unsigned long obj_exts;
+	unsigned long start;
+	unsigned long end;
+
+	obj_exts = slab_obj_exts(slab);
+	if (!obj_exts)
+		return false;
+
+	start = (unsigned long)slab_address(slab);
+	end = start + slab_size(slab);
+	return (obj_exts >= start) && (obj_exts < end);
+}
+#else
+static inline bool need_slab_obj_exts(struct kmem_cache *s)
+{
+	return false;
+}
+
+static inline unsigned int obj_exts_size_in_slab(struct slab *slab)
+{
+	return 0;
+}
+
+static inline unsigned long obj_exts_offset_in_slab(struct kmem_cache *s,
+						    struct slab *slab)
+{
+	return 0;
+}
+
+static inline bool obj_exts_fit_within_slab_leftover(struct kmem_cache *s,
+						     struct slab *slab)
+{
+	return false;
+}
+
+static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
+{
+	return false;
+}
+#endif
+
 #ifdef CONFIG_SLUB_DEBUG
 
 /*
@@ -1421,7 +1512,15 @@ slab_pad_check(struct kmem_cache *s, struct slab *slab)
 	start = slab_address(slab);
 	length = slab_size(slab);
 	end = start + length;
-	remainder = length % s->size;
+
+	if (obj_exts_in_slab(s, slab)) {
+		remainder = length;
+		remainder -= obj_exts_offset_in_slab(s, slab);
+		remainder -= obj_exts_size_in_slab(slab);
+	} else {
+		remainder = length % s->size;
+	}
+
 	if (!remainder)
 		return;
 
@@ -2195,6 +2294,11 @@ static inline void free_slab_obj_exts(struct slab *slab)
 		return;
 	}
 
+	if (obj_exts_in_slab(slab->slab_cache, slab)) {
+		slab->obj_exts = 0;
+		return;
+	}
+
 	/*
 	 * obj_exts was created with __GFP_NO_OBJ_EXT flag, therefore its
 	 * corresponding extension will be NULL. alloc_tag_sub() will throw a
@@ -2210,6 +2314,35 @@ static inline void free_slab_obj_exts(struct slab *slab)
 	slab->obj_exts = 0;
 }
 
+/*
+ * Try to allocate slabobj_ext array from unused space.
+ * This function must be called on a freshly allocated slab to prevent
+ * concurrency problems.
+ */
+static void alloc_slab_obj_exts_early(struct kmem_cache *s, struct slab *slab)
+{
+	void *addr;
+	unsigned long obj_exts;
+
+	if (!need_slab_obj_exts(s))
+		return;
+
+	if (obj_exts_fit_within_slab_leftover(s, slab)) {
+		addr = slab_address(slab) + obj_exts_offset_in_slab(s, slab);
+		addr = kasan_reset_tag(addr);
+		obj_exts = (unsigned long)addr;
+
+		get_slab_obj_exts(obj_exts);
+		memset(addr, 0, obj_exts_size_in_slab(slab));
+		put_slab_obj_exts(obj_exts);
+
+		if (IS_ENABLED(CONFIG_MEMCG))
+			obj_exts |= MEMCG_DATA_OBJEXTS;
+		slab->obj_exts = obj_exts;
+		slab_set_stride(slab, sizeof(struct slabobj_ext));
+	}
+}
+
 #else /* CONFIG_SLAB_OBJ_EXT */
 
 static inline void init_slab_obj_exts(struct slab *slab)
@@ -2226,6 +2359,11 @@ static inline void free_slab_obj_exts(struct slab *slab)
 {
 }
 
+static inline void alloc_slab_obj_exts_early(struct kmem_cache *s,
+						       struct slab *slab)
+{
+}
+
 #endif /* CONFIG_SLAB_OBJ_EXT */
 
 #ifdef CONFIG_MEM_ALLOC_PROFILING
@@ -3222,7 +3360,9 @@ static inline bool shuffle_freelist(struct kmem_cache *s, struct slab *slab)
 static __always_inline void account_slab(struct slab *slab, int order,
 					 struct kmem_cache *s, gfp_t gfp)
 {
-	if (memcg_kmem_online() && (s->flags & SLAB_ACCOUNT))
+	if (memcg_kmem_online() &&
+			(s->flags & SLAB_ACCOUNT) &&
+			!slab_obj_exts(slab))
 		alloc_slab_obj_exts(slab, s, gfp, true);
 
 	mod_node_page_state(slab_pgdat(slab), cache_vmstat_idx(s),
@@ -3286,9 +3426,6 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 	slab->objects = oo_objects(oo);
 	slab->inuse = 0;
 	slab->frozen = 0;
-	init_slab_obj_exts(slab);
-
-	account_slab(slab, oo_order(oo), s, flags);
 
 	slab->slab_cache = s;
 
@@ -3297,6 +3434,13 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node)
 	start = slab_address(slab);
 
 	setup_slab_debug(s, slab, start);
+	init_slab_obj_exts(slab);
+	/*
+	 * Poison the slab before initializing the slabobj_ext array
+	 * to prevent the array from being overwritten.
+	 */
+	alloc_slab_obj_exts_early(s, slab);
+	account_slab(slab, oo_order(oo), s, flags);
 
 	shuffle = shuffle_freelist(s, slab);
 
Signature · 1 line
2.43.0
author
#8 lore raw

PATCH v6 8/9 mm/slab: move [__]ksize and slab_ksize() to mm/slub.c

In reply to Harry Yoo

To access SLUB's internal implementation details beyond cache flags in ksize(), move __ksize(), ksize(), and slab_ksize() to mm/slub.c.

---
 mm/slab.h        | 25 --------------
 mm/slab_common.c | 61 ----------------------------------
 mm/slub.c        | 86 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 86 insertions(+), 86 deletions(-)
mm/slab.h +0 -25
index 5176c762ec7c..957586d68b3c 100644
@@ -665,31 +665,6 @@ void kvfree_rcu_cb(struct rcu_head *head);
 
 size_t __ksize(const void *objp);
 
-static inline size_t slab_ksize(const struct kmem_cache *s)
-{
-#ifdef CONFIG_SLUB_DEBUG
-	/*
-	 * Debugging requires use of the padding between object
-	 * and whatever may come after it.
-	 */
-	if (s->flags & (SLAB_RED_ZONE | SLAB_POISON))
-		return s->object_size;
-#endif
-	if (s->flags & SLAB_KASAN)
-		return s->object_size;
-	/*
-	 * If we have the need to store the freelist pointer
-	 * back there or track user information then we can
-	 * only use the space before that information.
-	 */
-	if (s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_STORE_USER))
-		return s->inuse;
-	/*
-	 * Else we can use all the padding etc for the allocation
-	 */
-	return s->size;
-}
-
 static inline unsigned int large_kmalloc_order(const struct page *page)
 {
 	return page[1].flags.f & 0xff;
mm/slab_common.c +0 -61

73 lines not on this page · raw message

mm/slub.c +86 -0

92 lines not on this page · raw message

Signature · 1 line
2.43.0
#9 lore raw

Re: [PATCH V6 8/9] mm/slab: move [__]ksize and slab_ksize() to mm/slub.c

Quoted 57 lines · Harry Yoo

To access SLUB's internal implementation details beyond cache flags in ksize(), move __ksize(), ksize(), and slab_ksize() to mm/slub.c.

---
 mm/slab.h        | 25 --------------
 mm/slab_common.c | 61 ----------------------------------
 mm/slub.c        | 86 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 86 insertions(+), 86 deletions(-)
mm/slab.h +0 -25
index 5176c762ec7c..957586d68b3c 100644
@@ -665,31 +665,6 @@ void kvfree_rcu_cb(struct rcu_head *head);
 
 size_t __ksize(const void *objp);
 
-static inline size_t slab_ksize(const struct kmem_cache *s)
-{
-#ifdef CONFIG_SLUB_DEBUG
-	/*
-	 * Debugging requires use of the padding between object
-	 * and whatever may come after it.
-	 */
-	if (s->flags & (SLAB_RED_ZONE | SLAB_POISON))
-		return s->object_size;
-#endif
-	if (s->flags & SLAB_KASAN)
-		return s->object_size;
-	/*
-	 * If we have the need to store the freelist pointer
-	 * back there or track user information then we can
-	 * only use the space before that information.
-	 */
-	if (s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_STORE_USER))
-		return s->inuse;
-	/*
-	 * Else we can use all the padding etc for the allocation
-	 */
-	return s->size;
-}
-
 static inline unsigned int large_kmalloc_order(const struct page *page)
 {
 	return page[1].flags.f & 0xff;
mm/slab_common.c +0 -3
index c4cf9ed2ec92..aed91fd6fd10 100644
@@ -983,43 +983,6 @@ void __init create_kmalloc_caches(void)
 						       0, SLAB_NO_MERGE, NULL);
 }
 
-/**
- * __ksize -- Report full size of underlying allocation
- * @object: pointer to the object

mm/slab_common.c void __init create_kmalloc_caches(void) · line 998

Quoted 10 lines
mm/slab_common.c +0 -10
- *
- * This should only be used internally to query the true size of allocations.
- * It is not meant to be a way to discover the usable size of an allocation
- * after the fact. Instead, use kmalloc_size_roundup(). Using memory beyond
- * the originally requested allocation size may trigger KASAN, UBSAN_BOUNDS,
- * and/or FORTIFY_SOURCE.
- *
- * Return: size of the actual memory used by @object in bytes
- */
-size_t __ksize(const void *object)

Think it could be also static and not in slab.h? I'll make that change locally.

author
#10 lore raw

Re: [PATCH V6 8/9] mm/slab: move [__]ksize and slab_ksize() to mm/slub.c

Quoted 55 lines · Vlastimil Babka
Quoted 54 lines · Harry Yoo

To access SLUB's internal implementation details beyond cache flags in ksize(), move __ksize(), ksize(), and slab_ksize() to mm/slub.c.

---
 mm/slab.h        | 25 --------------
 mm/slab_common.c | 61 ----------------------------------
 mm/slub.c        | 86 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 86 insertions(+), 86 deletions(-)
mm/slab.h +0 -25
index 5176c762ec7c..957586d68b3c 100644
@@ -665,31 +665,6 @@ void kvfree_rcu_cb(struct rcu_head *head);
 
 size_t __ksize(const void *objp);
 
-static inline size_t slab_ksize(const struct kmem_cache *s)
-{
-#ifdef CONFIG_SLUB_DEBUG
-	/*
-	 * Debugging requires use of the padding between object
-	 * and whatever may come after it.
-	 */
-	if (s->flags & (SLAB_RED_ZONE | SLAB_POISON))
-		return s->object_size;
-#endif
-	if (s->flags & SLAB_KASAN)
-		return s->object_size;
-	/*
-	 * If we have the need to store the freelist pointer
-	 * back there or track user information then we can
-	 * only use the space before that information.
-	 */
-	if (s->flags & (SLAB_TYPESAFE_BY_RCU | SLAB_STORE_USER))
-		return s->inuse;
-	/*
-	 * Else we can use all the padding etc for the allocation
-	 */
-	return s->size;
-}
-
 static inline unsigned int large_kmalloc_order(const struct page *page)
 {
 	return page[1].flags.f & 0xff;
mm/slab_common.c +0 -0
index c4cf9ed2ec92..aed91fd6fd10 100644
@@ -983,43 +983,6 @@ void __init create_kmalloc_caches(void)
 						       0, SLAB_NO_MERGE, NULL);
 }
 
Quoted 16 lines

mm/slab_common.c void __init create_kmalloc_caches(void) · line 998

Quoted 13 lines
mm/slab_common.c +0 -13
-/**
- * __ksize -- Report full size of underlying allocation
- * @object: pointer to the object
- *
- * This should only be used internally to query the true size of allocations.
- * It is not meant to be a way to discover the usable size of an allocation
- * after the fact. Instead, use kmalloc_size_roundup(). Using memory beyond
- * the originally requested allocation size may trigger KASAN, UBSAN_BOUNDS,
- * and/or FORTIFY_SOURCE.
- *
- * Return: size of the actual memory used by @object in bytes
- */
-size_t __ksize(const void *object)

Think it could be also static and not in slab.h? I'll make that change locally.

Uh, great. Thanks!

By the way size_t __ksize(const void *objp); is in both include/linux/slab.h and mm/slab.h.

Signature · 2 lines
Cheers,
Harry / Hyeonggon
#11 lore raw

Re: [PATCH V6 8/9] mm/slab: move [__]ksize and slab_ksize() to mm/slub.c

Quoted 19 lines · Harry Yoo
Quoted 17 lines · Vlastimil Babka
Quoted 4 lines
mm/slab_common.c +0 -3
-/**
- * __ksize -- Report full size of underlying allocation
- * @object: pointer to the object

mm/slab_common.c void __init create_kmalloc_caches(void) · line 998

Quoted 10 lines
mm/slab_common.c +0 -10
- *
- * This should only be used internally to query the true size of allocations.
- * It is not meant to be a way to discover the usable size of an allocation
- * after the fact. Instead, use kmalloc_size_roundup(). Using memory beyond
- * the originally requested allocation size may trigger KASAN, UBSAN_BOUNDS,
- * and/or FORTIFY_SOURCE.
- *
- * Return: size of the actual memory used by @object in bytes
- */
-size_t __ksize(const void *object)

Think it could be also static and not in slab.h? I'll make that change locally.

Quoted 4 lines

Uh, great. Thanks!

By the way size_t __ksize(const void *objp); is in both include/linux/slab.h and mm/slab.h.

Ack. Also moving the kerneldoc to ksize() as it should have been there anyway.

author
#12 lore raw

PATCH v6 9/9 mm/slab: place slabobj_ext metadata in unused space within s->size

In reply to Harry Yoo

When a cache has high s->align value and s->object_size is not aligned to it, each object ends up with some unused space because of alignment. If this wasted space is big enough, we can use it to store the slabobj_ext metadata instead of wasting it.

On my system, this happens with caches like kmem_cache, mm_struct, pid, task_struct, sighand_cache, xfs_inode, and others.

To place the slabobj_ext metadata within each object, the existing slab_obj_ext() logic can still be used by setting:

- slab->obj_exts = slab_address(slab) + (slabobj_ext offset)
- stride = s->size

slab_obj_ext() doesn't need know where the metadata is stored, so this method works without adding extra overhead to slab_obj_ext().

A good example benefiting from this optimization is xfs_inode (object_size: 992, align: 64). To measure memory savings, 2 millions of files were created on XFS.

[ MEMCG=y, MEM_ALLOC_PROFILING=n ]

Before patch (creating ~2.64M directories on xfs):

  Slab:            5175976 kB
  SReclaimable:    3837524 kB
  SUnreclaim:      1338452 kB

After patch (creating ~2.64M directories on xfs):

  Slab:            5152912 kB
  SReclaimable:    3838568 kB
  SUnreclaim:      1314344 kB (-23.54 MiB)

Enjoy the memory savings!

---
 include/linux/slab.h |  9 +++++
 mm/slab_common.c     |  6 ++-
 mm/slub.c            | 89 +++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 93 insertions(+), 11 deletions(-)
include/linux/slab.h +9 -0
index 4554c04a9bd7..da512d9ab1a0 100644
@@ -59,6 +59,9 @@ enum _slab_flag_bits {
 	_SLAB_CMPXCHG_DOUBLE,
 #ifdef CONFIG_SLAB_OBJ_EXT
 	_SLAB_NO_OBJ_EXT,
+#endif
+#if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
+	_SLAB_OBJ_EXT_IN_OBJ,
 #endif
 	_SLAB_FLAGS_LAST_BIT
 };
@@ -244,6 +247,12 @@ enum _slab_flag_bits {
 #define SLAB_NO_OBJ_EXT		__SLAB_FLAG_UNUSED
 #endif
 
+#if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
+#define SLAB_OBJ_EXT_IN_OBJ	__SLAB_FLAG_BIT(_SLAB_OBJ_EXT_IN_OBJ)
+#else
+#define SLAB_OBJ_EXT_IN_OBJ	__SLAB_FLAG_UNUSED
+#endif
+
 /*
  * ZERO_SIZE_PTR will be returned for zero sized kmalloc requests.
  *
mm/slab_common.c +4 -2
index aed91fd6fd10..7418719b5ebf 100644
@@ -43,11 +43,13 @@ DEFINE_MUTEX(slab_mutex);
 struct kmem_cache *kmem_cache;
 
 /*
- * Set of flags that will prevent slab merging
+ * Set of flags that will prevent slab merging.
+ * Any flag that adds per-object metadata should be included,
+ * since slab merging can update s->inuse that affects the metadata layout.
  */
 #define SLAB_NEVER_MERGE (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
 		SLAB_TRACE | SLAB_TYPESAFE_BY_RCU | SLAB_NOLEAKTRACE | \
-		SLAB_FAILSLAB | SLAB_NO_MERGE)
+		SLAB_FAILSLAB | SLAB_NO_MERGE | SLAB_OBJ_EXT_IN_OBJ)
 
 #define SLAB_MERGE_SAME (SLAB_RECLAIM_ACCOUNT | SLAB_CACHE_DMA | \
 			 SLAB_CACHE_DMA32 | SLAB_ACCOUNT)
mm/slub.c +80 -9

166 lines not on this page · raw message

Signature · 1 line
2.43.0
#13 lore raw

Re: [PATCH V6 9/9] mm/slab: place slabobj_ext metadata in unused space within s->size

Quoted 29 lines · Harry Yoo

When a cache has high s->align value and s->object_size is not aligned to it, each object ends up with some unused space because of alignment. If this wasted space is big enough, we can use it to store the slabobj_ext metadata instead of wasting it.

On my system, this happens with caches like kmem_cache, mm_struct, pid, task_struct, sighand_cache, xfs_inode, and others.

To place the slabobj_ext metadata within each object, the existing slab_obj_ext() logic can still be used by setting:

- slab->obj_exts = slab_address(slab) + (slabobj_ext offset)
- stride = s->size

slab_obj_ext() doesn't need know where the metadata is stored, so this method works without adding extra overhead to slab_obj_ext().

A good example benefiting from this optimization is xfs_inode (object_size: 992, align: 64). To measure memory savings, 2 millions of files were created on XFS.

[ MEMCG=y, MEM_ALLOC_PROFILING=n ]

Before patch (creating ~2.64M directories on xfs):

  Slab:            5175976 kB
  SReclaimable:    3837524 kB
  SUnreclaim:      1338452 kB

After patch (creating ~2.64M directories on xfs):

Quoted 8 lines
  Slab:            5152912 kB
  SReclaimable:    3838568 kB
  SUnreclaim:      1314344 kB (-23.54 MiB)

Enjoy the memory savings!

Does this look OK to you or was there a reason you didn't do it? :)

mm/slub.c +1 -2
index ba15df4ca417..deb69bd9646a 100644
@@ -981,8 +981,7 @@ static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
 #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
 static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
 {
-       return obj_exts_in_slab(s, slab) &&
-              (slab_get_stride(slab) == s->size);
+       return obj_exts_in_slab(s, slab) && (s->flags & SLAB_OBJ_EXT_IN_OBJ);
 }
 
 static unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
author
#14 lore raw

Re: [PATCH V6 9/9] mm/slab: place slabobj_ext metadata in unused space within s->size

Quoted 46 lines · Vlastimil Babka
Quoted 29 lines · Harry Yoo

When a cache has high s->align value and s->object_size is not aligned to it, each object ends up with some unused space because of alignment. If this wasted space is big enough, we can use it to store the slabobj_ext metadata instead of wasting it.

On my system, this happens with caches like kmem_cache, mm_struct, pid, task_struct, sighand_cache, xfs_inode, and others.

To place the slabobj_ext metadata within each object, the existing slab_obj_ext() logic can still be used by setting:

- slab->obj_exts = slab_address(slab) + (slabobj_ext offset)
- stride = s->size

slab_obj_ext() doesn't need know where the metadata is stored, so this method works without adding extra overhead to slab_obj_ext().

A good example benefiting from this optimization is xfs_inode (object_size: 992, align: 64). To measure memory savings, 2 millions of files were created on XFS.

[ MEMCG=y, MEM_ALLOC_PROFILING=n ]

Before patch (creating ~2.64M directories on xfs):

  Slab:            5175976 kB
  SReclaimable:    3837524 kB
  SUnreclaim:      1338452 kB

After patch (creating ~2.64M directories on xfs):

Quoted 8 lines
  Slab:            5152912 kB
  SReclaimable:    3838568 kB
  SUnreclaim:      1314344 kB (-23.54 MiB)

Enjoy the memory savings!

Does this look OK to you or was there a reason you didn't do it? :)

mm/slub.c +0 -0
index ba15df4ca417..deb69bd9646a 100644

mm/slub.c static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab) · line 984

Quoted 6 lines
mm/slub.c +1 -2
@@ -981,8 +981,7 @@ static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
 #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
 static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
 {
-       return obj_exts_in_slab(s, slab) &&
-              (slab_get_stride(slab) == s->size);
+       return obj_exts_in_slab(s, slab) && (s->flags & SLAB_OBJ_EXT_IN_OBJ);

There was a reason why I didn't do it :)

In alloc_slab_obj_exts_early(), when both obj_exts_fit_within_slab_leftover() and (s->flags & SLAB_OBJ_EXT_IN_OBJ) returns true, it allocates the metadata from the slab's leftover space.

I noticed it as I saw a slab error in slab_pad_check() complaining that the padding area was overwritten, but turned out the problem was because obj_exts_in_object() returning true when it shouldn't.

Quoted 3 lines
mm/slub.c +0 -0
 }
 
 static unsigned int obj_exts_offset_in_object(struct kmem_cache *s)
Signature · 2 lines
Cheers,
Harry / Hyeonggon
author
#15 lore raw

Re: [PATCH V6 9/9] mm/slab: place slabobj_ext metadata in unused space within s->size

Quoted 56 lines · Harry Yoo
Quoted 46 lines · Vlastimil Babka
Quoted 29 lines · Harry Yoo

When a cache has high s->align value and s->object_size is not aligned to it, each object ends up with some unused space because of alignment. If this wasted space is big enough, we can use it to store the slabobj_ext metadata instead of wasting it.

On my system, this happens with caches like kmem_cache, mm_struct, pid, task_struct, sighand_cache, xfs_inode, and others.

To place the slabobj_ext metadata within each object, the existing slab_obj_ext() logic can still be used by setting:

- slab->obj_exts = slab_address(slab) + (slabobj_ext offset)
- stride = s->size

slab_obj_ext() doesn't need know where the metadata is stored, so this method works without adding extra overhead to slab_obj_ext().

A good example benefiting from this optimization is xfs_inode (object_size: 992, align: 64). To measure memory savings, 2 millions of files were created on XFS.

[ MEMCG=y, MEM_ALLOC_PROFILING=n ]

Before patch (creating ~2.64M directories on xfs):

  Slab:            5175976 kB
  SReclaimable:    3837524 kB
  SUnreclaim:      1338452 kB

After patch (creating ~2.64M directories on xfs):

Quoted 8 lines
  Slab:            5152912 kB
  SReclaimable:    3838568 kB
  SUnreclaim:      1314344 kB (-23.54 MiB)

Enjoy the memory savings!

Does this look OK to you or was there a reason you didn't do it? :)

mm/slub.c +0 -0
index ba15df4ca417..deb69bd9646a 100644

mm/slub.c static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab) · line 984

Quoted 6 lines
mm/slub.c +1 -2
@@ -981,8 +981,7 @@ static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
 #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
 static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
 {
-       return obj_exts_in_slab(s, slab) &&
-              (slab_get_stride(slab) == s->size);
+       return obj_exts_in_slab(s, slab) && (s->flags & SLAB_OBJ_EXT_IN_OBJ);

There was a reason why I didn't do it :)

Quoted 7 lines

In alloc_slab_obj_exts_early(), when both obj_exts_fit_within_slab_leftover() and (s->flags & SLAB_OBJ_EXT_IN_OBJ) returns true, it allocates the metadata from the slab's leftover space.

I noticed it as I saw a slab error in slab_pad_check() complaining that the padding area was overwritten, but turned out the problem was because obj_exts_in_object() returning true when it shouldn't.

Perhaps a comment like this?

mm/slub.c +9 -0
index ba15df4ca417..c40c3559039e 100644
@@ -981,6 +981,15 @@ static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
 #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
 static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
 {
+	/*
+	 * When SLAB_OBJ_EXT_IN_OBJ is set, slabobj_ext metadata can be stored
+	 * in one of two ways:
+	 * 1. As an array in the slab's leftover space (after the last object)
+	 * 2. Inline with each object (within s->size)
+	 *
+	 * The actual placement is determined by the stride size rather than
+	 * the SLAB_OBJ_EXT_IN_OBJ flag itself.
+	 */
 	return obj_exts_in_slab(s, slab) &&
 	       (slab_get_stride(slab) == s->size);
 }
#16 lore raw

Re: [PATCH V6 9/9] mm/slab: place slabobj_ext metadata in unused space within s->size

Quoted 40 lines · Harry Yoo
Quoted 19 lines · Harry Yoo
Quoted 9 lines · Vlastimil Babka

On 1/13/26 7:18 AM, Harry Yoo wrote:

Does this look OK to you or was there a reason you didn't do it? :)

mm/slub.c +0 -0
index ba15df4ca417..deb69bd9646a 100644

mm/slub.c static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab) · line 984

Quoted 6 lines
mm/slub.c +1 -2
@@ -981,8 +981,7 @@ static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
 #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
 static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
 {
-       return obj_exts_in_slab(s, slab) &&
-              (slab_get_stride(slab) == s->size);
+       return obj_exts_in_slab(s, slab) && (s->flags & SLAB_OBJ_EXT_IN_OBJ);

There was a reason why I didn't do it :)

Quoted 7 lines

In alloc_slab_obj_exts_early(), when both obj_exts_fit_within_slab_leftover() and (s->flags & SLAB_OBJ_EXT_IN_OBJ) returns true, it allocates the metadata from the slab's leftover space.

I noticed it as I saw a slab error in slab_pad_check() complaining that the padding area was overwritten, but turned out the problem was because obj_exts_in_object() returning true when it shouldn't.

Perhaps a comment like this?

mm/slub.c +2 -0
index ba15df4ca417..c40c3559039e 100644
@@ -981,6 +981,15 @@ static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
 #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
 static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
 {
+	/*
+	 * When SLAB_OBJ_EXT_IN_OBJ is set, slabobj_ext metadata can be stored

mm/slub.c static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab) · line 995

Quoted 10 lines
mm/slub.c +7 -0
+	 * in one of two ways:
+	 * 1. As an array in the slab's leftover space (after the last object)
+	 * 2. Inline with each object (within s->size)
+	 *
+	 * The actual placement is determined by the stride size rather than
+	 * the SLAB_OBJ_EXT_IN_OBJ flag itself.
+	 */
 	return obj_exts_in_slab(s, slab) &&
 	       (slab_get_stride(slab) == s->size);
 }

I meanwhile wrote this one. I think the part about depending on slab's size is important so one doesn't wonder why we don't simply clear SLAB_OBJ_EXT_IN_OBJ if it fits within_slab_leftover. As discussed off-list, will use it. Thanks!

mm/slub.c +6 -0
@@ -981,6 +981,12 @@ static inline bool obj_exts_in_slab(struct kmem_cache *s, struct slab *slab)
 #if defined(CONFIG_SLAB_OBJ_EXT) && defined(CONFIG_64BIT)
 static bool obj_exts_in_object(struct kmem_cache *s, struct slab *slab)
 {
+       /*
+        * Note we cannot rely on the SLAB_OBJ_EXT_IN_OBJ flag here and need to
+        * check the stride. A cache can have SLAB_OBJ_EXT_IN_OBJ set, but
+        * allocations within_slab_leftover are preferred. And those may be
+        * possible or not depending on the particular slab's size.
+        */
        return obj_exts_in_slab(s, slab) &&
               (slab_get_stride(slab) == s->size);
 }
#17 lore raw
In reply to Harry Yoo
Quoted 19 lines · Harry Yoo

V5: https://lore.kernel.org/linux-mm/[email protected]
V5 -> V6:

- Patch 1: Added Closes: tag for related discussion (Vlastimil)
https://lore.kernel.org/linux-mm/[email protected]

- Patch 3: Addressed Vlastimil's comments
https://lore.kernel.org/linux-mm/[email protected]

- Patch 4: Fixed incorrect function prototype of slab_obj_ext() on !CONFIG_SLAB_OBJ_EXT builds and kept pointer type in free_slab_obj_exts() (Hao, Vlastimil)
https://lore.kernel.org/linux-mm/n6kyluk3nahdxytwek4ijzy4en6mc6ps7fjjgftww4ith7llom@cijm4who24w2
https://lore.kernel.org/linux-mm/[email protected]

- Patch 7, 9: Rewrote obj_exts_in_slab() to check if the pointer is within the slab's range, and distinguish by stride (Vlastimil)
https://lore.kernel.org/linux-mm/[email protected]

Quoted 6 lines

- Patch 9: Fixed potentioal memory leak due to incorrect impl. of obj_exts_in_object() (Vlastimil)
https://lore.kernel.org/linux-mm/[email protected]/

- Patch 9: Fixed incorrect ksize() implementation (Hao)
https://lore.kernel.org/linux-mm/fgx3lapibabra4x7tewx55nuvxz235ruvm3agpprjbdcmt3rc6@h54ln5tfdssz

Merged to slab/for-7.0/obj_metadata and slab/for-next, thanks!