net: skb: isolate skb data area allocations into a separate bucket

kmem_buckets_create() had the same sentence about calling context as kmem_cache_create(), but lacked the "Context:" prefix, so kernel-doc rendered in the body instead of as a "context" section.

Give it the missing prefix. Additionally fix the "a interrupt" typo __kmem_cache_create_args() had.

Fixes b32801d1255be ("mm/slab: Introduce kmem_buckets_create() and family")

Assisted-by LLM

Signed-off-by Kees Cook <[email protected]>

---

kmem_buckets_create() had the same sentence about calling context as kmem_cache_create(), but lacked the "Context:" prefix, so kernel-doc rendered in the body instead of as a "context" section.

Give it the missing prefix. Additionally fix the "a interrupt" typo __kmem_cache_create_args() had.

Fixes b32801d1255be ("mm/slab: Introduce kmem_buckets_create() and family")

Assisted-by LLM

Signed-off-by Kees Cook <[email protected]>

kmem_buckets_create() had the same sentence about calling context as kmem_cache_create(), but lacked the "Context:" prefix, so kernel-doc rendered in the body instead of as a "context" section.

Give it the missing prefix. Additionally fix the "a interrupt" typo __kmem_cache_create_args() had.

Fixes b32801d1255be ("mm/slab: Introduce kmem_buckets_create() and family")

Assisted-by LLM

Signed-off-by Kees Cook <[email protected]>

---

A bucket set is created with kmem_cache_create_usercopy(..., align = 0), so calculate_alignment() falls back to arch_slab_minalign(), typically 8 bytes. The general kmalloc caches it stands in for are created through create_boot_cache(), which starts from ARCH_KMALLOC_MINALIGN and raises it to the largest power-of-two divisor of the size:

	if (flags & SLAB_KMALLOC)
		align = max(align, 1U << (ffs(size) - 1));

This is only a problem when slab metadata is enabled with CONFIG_KASAN=y, CONFIG_SLUB_DEBUG_ON=y, or "slab_debug=...", because metadata changes the stride size off a power of two, for example:

	size 128:  bucket align=8 size=224  | kmalloc align=128  size=384
	size 512:  bucket align=8 size=608  | kmalloc align=512  size=1536
	size 2048: bucket align=8 size=2144 | kmalloc align=2048 size=6144

So bucket allocations will fail the IS_ALIGNED(p, ARCH_DMA_MINALIGN) check, potentially creating problems for non-coherent DMA situation.

Take the alignment from the cache being mirrored, which is where the size and the name suffix already come from. Nothing changes where the alignment was already implied by the size.

Fixes b32801d1255be ("mm/slab: Introduce kmem_buckets_create() and family")

Assisted-by LLM

Signed-off-by Kees Cook <[email protected]>

---
Cc: Vlastimil Babka <[email protected]>
Cc: Harry Yoo <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Hao Li <[email protected]>
Cc: Christoph Lameter <[email protected]>
Cc: David Rientjes <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: <[email protected]>
Cc: Pedro Falcato <[email protected]>
Cc: Kuniyuki Iwashima <[email protected]>
Cc: <[email protected]>

---
 mm/slab_common.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

mm/slab_common.c +2 -1

index 270408ce5a9d..cc58f192f349 100644
@@ -487,7 +487,8 @@ kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags,
 			if (WARN_ON(!cache_name))
 				goto fail;
 			(*b)[aligned_idx] = kmem_cache_create_usercopy(cache_name, size,
-					0, flags, cache_useroffset,
+					kmalloc_caches[KMALLOC_NORMAL][idx]->align,
+					flags, cache_useroffset,
 					cache_usersize, ctor);
 			kfree(cache_name);
 			if (WARN_ON(!(*b)[aligned_idx]))

Signature · 1 line

2.34.1

Quoted 13 lines · Kees Cook

A bucket set is created with kmem_cache_create_usercopy(..., align = 0), so calculate_alignment() falls back to arch_slab_minalign(), typically 8 bytes. The general kmalloc caches it stands in for are created through create_boot_cache(), which starts from ARCH_KMALLOC_MINALIGN and raises it to the largest power-of-two divisor of the size:

	if (flags & SLAB_KMALLOC)
		align = max(align, 1U << (ffs(size) - 1));

This is only a problem when slab metadata is enabled with CONFIG_KASAN=y, CONFIG_SLUB_DEBUG_ON=y, or "slab_debug=...", because metadata changes the stride size off a power of two, for example:

Quoted 3 lines

	size 128:  bucket align=8 size=224  | kmalloc align=128  size=384
	size 512:  bucket align=8 size=608  | kmalloc align=512  size=1536
	size 2048: bucket align=8 size=2144 | kmalloc align=2048 size=6144

Hmm... I think what adds confusion here is that in new_kmalloc_cache() we adjust the size based on alignment, but in create_boot_cache() we don't do that. Perhaps let's make it consistent and move it to new_kmalloc_cache()?

Quoted 2 lines

So bucket allocations will fail the IS_ALIGNED(p, ARCH_DMA_MINALIGN) check, potentially creating problems for non-coherent DMA situation.

I was wondering "Why should they respect kmalloc alignment..." but yeah, It makes sense if the users were using kmalloc and depended on its alignment.

Well, but that's already done in new_kmalloc_cache() and kmem_buckets_create() should already honor ARCH_KMALLOC_MINALIGN?

The largest-power-of-two-divisor-alignment guarantee was introduced by commit ad59baa31695 ("slab, rust: extend kmalloc() alignment guarantees to remove Rust padding")

...which makes me wonder what you're trying to fix here?

mm/slab_common.c kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, · line 494

Quoted 9 lines

mm/slab_common.c +2 -1

 			if (WARN_ON(!cache_name))
 				goto fail;
 			(*b)[aligned_idx] = kmem_cache_create_usercopy(cache_name, size,
-					0, flags, cache_useroffset,
+					kmalloc_caches[KMALLOC_NORMAL][idx]->align,
+					flags, cache_useroffset,
 					cache_usersize, ctor);
 			kfree(cache_name);
 			if (WARN_ON(!(*b)[aligned_idx]))

Quoted 10 lines · Harry Yoo

mm/slab_common.c kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, · line 494

Quoted 9 lines · Kees Cook

mm/slab_common.c +2 -1

 			if (WARN_ON(!cache_name))
 				goto fail;
 			(*b)[aligned_idx] = kmem_cache_create_usercopy(cache_name, size,
-					0, flags, cache_useroffset,
+					kmalloc_caches[KMALLOC_NORMAL][idx]->align,
+					flags, cache_useroffset,
 					cache_usersize, ctor);
 			kfree(cache_name);
 			if (WARN_ON(!(*b)[aligned_idx]))

It looks "obviously correct", but I probably failed to correctly describe it. I'm happy to do whatever here.

Quoted 14 lines · Kees Cook

Quoted 10 lines · Harry Yoo

mm/slab_common.c kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, · line 494

Quoted 9 lines · Kees Cook

mm/slab_common.c +2 -1

 			if (WARN_ON(!cache_name))
 				goto fail;
 			(*b)[aligned_idx] = kmem_cache_create_usercopy(cache_name, size,
-					0, flags, cache_useroffset,
+					kmalloc_caches[KMALLOC_NORMAL][idx]->align,
+					flags, cache_useroffset,
 					cache_usersize, ctor);
 			kfree(cache_name);
 			if (WARN_ON(!(*b)[aligned_idx]))

It looks "obviously correct", but I probably failed to correctly describe it. I'm happy to do whatever here.

That is correct, but is there any reason why kmem_buckets users should not specify their own alignment requirement and inherit the alignment from kmalloc?

For DMA yeah it might be confusing for every user to figure out that the alignment should be __kmalloc_minalign(), but otherwise there isn't really a good enough reason to inherit the alignment from kmalloc, no?

kmem_buckets_create() clones kmalloc_caches[KMALLOC_NORMAL]. kmalloc_slab() figures out the kmalloc type the caller asks for, but then ignored it whenever a bucket set was in use, returning a normal cache regardless. This would be a problem if a caller asked for GFP_DMA, __GFP_ACCOUNT, etc. None of the current users do this, so there is problem, but it makes adding new users fragile. For example, skb data[1] needs to handle GFP_DMA (rarely) and __GFP_ACCOUNT (often).

Send those allocations to the general caches instead so nothing breaks and regular allocations remain isolated with the bucket. The kmem_bucket_type enum contains only a single item here, but will be expanded in the next patch.

Built and tests pass with ARCH=x86_64 defconfig with GCC 16.2.0, with CONFIG_SLAB_BUCKETS as y and n.

Assisted-by LLM

Link https://lore.kernel.org/all/[email protected]/ [1] open here

Signed-off-by Kees Cook <[email protected]>

---
Cc: Vlastimil Babka <[email protected]>
Cc: Harry Yoo <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Hao Li <[email protected]>
Cc: Christoph Lameter <[email protected]>
Cc: David Rientjes <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: <[email protected]>
Cc: Pedro Falcato <[email protected]>
Cc: Kuniyuki Iwashima <[email protected]>
Cc: <[email protected]>

---
 include/linux/slab.h   | 13 +++++++++++
 mm/slab.h              | 23 ++++++++++++++++--
 lib/tests/slub_kunit.c | 53 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 87 insertions(+), 2 deletions(-)

include/linux/slab.h +0 -0

index 18a2351f9084..ab9ab3d34847 100644
@@ -742,6 +742,19 @@ typedef struct kmem_cache * kmem_buckets[KMALLOC_SHIFT_HIGH + 1];
 

include/linux/slab.h +5 -0

+enum kmem_bucket_type {
+	KMEM_BUCKET_NORMAL = 0,
+	NR_KMEM_BUCKET_TYPES
+};
+
 /*
  * Define gfp bits that should not be set for KMALLOC_NORMAL.
  */

mm/slab.h +6 -0

index 8fd6835e4235..7f1bfee83b92 100644
@@ -421,6 +421,26 @@ static inline unsigned int size_index_elem(unsigned int bytes)
 	return (bytes - 1) / 8;
 }
 
+/*
+ * Which set of buckets to use for the given kmalloc_cache_type. If not
+ * handled by the kmem_buckets, fall back to general caches.
+ */
+static inline kmem_buckets *
+kmalloc_choose_bucket(kmem_buckets *bucket, enum kmalloc_cache_type type)

Big thanks for continuing this effort :))

[...]

Quoted 8 lines

+/*
+ * The kmalloc types a bucket set can hold a copy of. This is deliberately not
+ * enum kmalloc_cache_type: the KMALLOC_PARTITION copies are all "normal" to a
+ * bucket set, which already separates what they were there to separate, so
+ * indexing by those would mean up to KMALLOC_PARTITION_CACHES_NR unusable
+ * rows per set. Allocations of any type not listed here are served by the
+ * general caches.
+ */

This sounds odd. Is there a good reason why KMALLOC_PARTITIONs are kmalloc_cache_types? Perhaps that bit should be reworked instead?

[...]

mm/slab.h static inline unsigned int size_index_elem(unsigned int bytes) · line 439

Quoted 4 lines

mm/slab.h +4 -0

+	if (type <= KMALLOC_PARTITION_END)
+		btype = KMEM_BUCKET_NORMAL;
+	else
+		return &kmalloc_caches[type];	/* No set holds a row for it. */

Hitting this case sounds like a bug in the kernel. WARN_ON_ONCE()?

Otherwise LGTM.

A bucket set holds one row of caches, cloned from KMALLOC_NORMAL, and an allocation of any other kmalloc type falls back to the general caches. Extend this to handle __GFP_ACCOUNT, so that a single bucket user can isolate either GFP_KERNEL or GFP_KERNEL_ACCOUNT allocations, as is needed for skb data, where AF_UNIX uses:

	sk->sk_allocation = GFP_KERNEL_ACCOUNT;

The coverage is selected at bucket creation time:

	b = kmem_buckets_create_types(name, flags, 0, INT_MAX, NULL,
				      BIT(KMEM_BUCKET_NORMAL) |
				      BIT(KMEM_BUCKET_CGROUP));

The prior kmem_buckets_create() function keeps its name and defaults to only KMEM_BUCKET_NORMAL, leaving existing users as-is.

Only the accounted type is offered. Nothing wants a reclaimable or no-obj-ext row, and of the twelve places passing GFP_DMA to an skb allocator, all rare hardware: b44, b43legacy, prestera and s390 ctcm.

The choice is made at creation rather than every set getting every type because the rows, when populated, are not free. Each holds 13 caches, and a cache is a 1208 byte struct plus an unconditional per-cpu allocation, a node struct, and an entry in /proc/slabinfo and under /sys/kernel/slab.

KMEM_BUCKET_CGROUP collapses to KMEM_BUCKET_NORMAL without CONFIG_MEMCG, exactly as KMALLOC_CGROUP does, so NR_KMEM_BUCKET_TYPES is 1 there and a bucket set is the same single row it is today. Where the type is asked for but the system is not creating caches of it (under "cgroup.memory=nokmem") the row is aliased to the normal one, as new_kmalloc_cache() does for the general caches, so those allocations stay isolated rather than falling back to the general caches.

Built and tests pass (and skip as expected) on ARCH=x86_64 defconfig with GCC 16.2.0 in all combinations of CONFIG_SLAB_BUCKETS=y/n and CONFIG_MEMCG=y/n/y+"cgroup.memory=nokmem".

Assisted-by LLM

Signed-off-by Kees Cook <[email protected]>

---
Cc: Vlastimil Babka <[email protected]>
Cc: Harry Yoo <[email protected]>
Cc: Andrew Morton <[email protected]>
Cc: Hao Li <[email protected]>
Cc: Christoph Lameter <[email protected]>
Cc: David Rientjes <[email protected]>
Cc: Roman Gushchin <[email protected]>
Cc: <[email protected]>
Cc: Pedro Falcato <[email protected]>
Cc: Kuniyuki Iwashima <[email protected]>
Cc: <[email protected]>

---
 include/linux/slab.h   |  45 ++++++++++++-
 mm/slab.h              |  23 ++++++-
 lib/tests/slub_kunit.c |  65 +++++++++++++++---
 mm/slab_common.c       | 145 ++++++++++++++++++++++++++++++++---------
 4 files changed, 235 insertions(+), 43 deletions(-)

lib/tests/slub_kunit.c +0 -0

index 823607e06248..58f800582170 100644

On Mon, Sep 21, 2026 at 12:58:17AM -0700, Kees Cook wrote:
Didn't take a deeper look at this yet but this is causing an error:

MODPOST Module.symvers
ERROR: modpost: lib/tests/slub_kunit.ko: symbol 'mem_cgroup_kmem_disabled' undefined!

A bucket set holds one row of caches, cloned from KMALLOC_NORMAL, and an allocation of any other kmalloc type falls back to the general caches. Extend this to handle __GFP_ACCOUNT, so that a single bucket user can isolate either GFP_KERNEL or GFP_KERNEL_ACCOUNT allocations, as is needed for skb data, where AF_UNIX uses:

	sk->sk_allocation = GFP_KERNEL_ACCOUNT;

The coverage is selected at bucket creation time:

	b = kmem_buckets_create_types(name, flags, 0, INT_MAX, NULL,
				      BIT(KMEM_BUCKET_NORMAL) |
				      BIT(KMEM_BUCKET_CGROUP));

The prior kmem_buckets_create() function keeps its name and defaults to only KMEM_BUCKET_NORMAL, leaving existing users as-is.

Only the accounted type is offered. Nothing wants a reclaimable or no-obj-ext row, and of the twelve places passing GFP_DMA to an skb allocator, all rare hardware: b44, b43legacy, prestera and s390 ctcm.

Quoted 17 lines · Kees Cook

A bucket set holds one row of caches, cloned from KMALLOC_NORMAL, and an allocation of any other kmalloc type falls back to the general caches. Extend this to handle __GFP_ACCOUNT, so that a single bucket user can isolate either GFP_KERNEL or GFP_KERNEL_ACCOUNT allocations, as is needed for skb data, where AF_UNIX uses:

	sk->sk_allocation = GFP_KERNEL_ACCOUNT;

The coverage is selected at bucket creation time:

	b = kmem_buckets_create_types(name, flags, 0, INT_MAX, NULL,
				      BIT(KMEM_BUCKET_NORMAL) |
				      BIT(KMEM_BUCKET_CGROUP));

The prior kmem_buckets_create() function keeps its name and defaults to only KMEM_BUCKET_NORMAL, leaving existing users as-is.

Quoted 3 lines

Only the accounted type is offered. Nothing wants a reclaimable or no-obj-ext row, and of the twelve places passing GFP_DMA to an skb allocator, all rare hardware: b44, b43legacy, prestera and s390 ctcm.

Well, yes, BUT I don't think silently falling back is a great option. I would rather things be done Properly(tm) and kmem_buckets allowing instantiation of most/all kmalloc cache types. Given it's security stuff, I would prefer being quite explicit over silent.

From: Pedro Falcato <[email protected]>

SKB data area allocations (as done from alloc_skb()) use kmalloc(). These allocations can be variably sized and their contents can be more or less controlled from userspace, which makes them useful for attackers that want to overwrite a use-after-free'd object from the same kmalloc slab (which often just requires the sizes to roughly match into the same kmalloc bucket). [0] is an easy example of an exploit that uses netlink skb allocation to target another similarly-sized accidentally freed object.

While other mitigations like CONFIG_RANDOM_KMALLOC_CACHES exist, these are probabilistic. Use the existing kmem buckets API to further isolate these allocations in a guaranteed fashion, when CONFIG_SLAB_BUCKETS=y.

Ask for the accounted kmalloc type as well as the normal one. AF_UNIX sets sk_allocation to GFP_KERNEL_ACCOUNT, so without it every AF_UNIX skb data area would fall back to the general caches, and those are the ones most worth isolating. GFP_DMA is left to fall back, being passed to an skb allocator only by rare devices.

Link https://github.com/google/security-research/blob/master/pocs/linux/kernelctf/CVE-2023-4207_lts_cos_mitigation_2/docs/exploit.md [0]

Reviewed-by Kees Cook <[email protected]>

Signed-off-by Pedro Falcato <[email protected]>

Quoted 20 lines · Kees Cook

From: Pedro Falcato <[email protected]>

SKB data area allocations (as done from alloc_skb()) use kmalloc(). These allocations can be variably sized and their contents can be more or less controlled from userspace, which makes them useful for attackers that want to overwrite a use-after-free'd object from the same kmalloc slab (which often just requires the sizes to roughly match into the same kmalloc bucket). [0] is an easy example of an exploit that uses netlink skb allocation to target another similarly-sized accidentally freed object.

While other mitigations like CONFIG_RANDOM_KMALLOC_CACHES exist, these are probabilistic. Use the existing kmem buckets API to further isolate these allocations in a guaranteed fashion, when CONFIG_SLAB_BUCKETS=y.

Ask for the accounted kmalloc type as well as the normal one. AF_UNIX sets sk_allocation to GFP_KERNEL_ACCOUNT, so without it every AF_UNIX skb data area would fall back to the general caches, and those are the ones most worth isolating. GFP_DMA is left to fall back, being passed to an skb allocator only by rare devices.

Quoted 3 lines

Link https://github.com/google/security-research/blob/master/pocs/linux/kernelctf/CVE-2023-4207_lts_cos_mitigation_2/docs/exploit.md [0]

Reviewed-by Kees Cook <[email protected]>

Signed-off-by Pedro Falcato <[email protected]>

I would be curious to learn how about the memory usage delta. However I see buckets are protected by their own kconfig, small systems can unselect them.

For the networking bits:

Acked-by Paolo Abeni <[email protected]>