Skip to contentSkip to conversation

mm/slab: Support selecting slab hardening features at boot

PATCH In review Chris Down 6 participants 14 messages Nov 13 – Nov 16

Subsystems (4)
lore
Share & tools
lore

Chris Down

author
#0 lore raw

v3 cover mm/slab: Support selecting slab hardening features at boot

Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. At Meta, for example, some hosts are built with these options and the rest of the fleet isn't, and that's one of the things that keeps those kernels apart.

This series does two things:

- SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM are cheap enough with sheaves that it makes more sense to just enable them by default than to make them switchable. With both on, system level benchmarks didn't change measurably, only slowpath heavy microbenchmarks did [1].

- SLAB_BUCKETS and KMALLOC_PARTITION_CACHES cost memory (Harry measured about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so they get slab_buckets= and kmalloc_partition= boot parameters, which default to on when the option is built in.

Whether those two should be folded into a single slab_hardening= is still open [3].

v3:

- Rebase onto slab/for-next.
- Always build the slab_buckets and kmalloc_partition tests and skip them at runtime when the option isn't built in, instead of leaving them out with #ifdef (Kees).
- Declare slab_buckets_enabled and kmalloc_partition_enabled in mm/slab.h without any #if (Kees).
- Drop the remaining #ifdefs around slab_buckets= and kmalloc_partition=. Both are now always registered and ignored when the option isn't built in (Kees).
- Keep both flags as plain __ro_after_init bools rather than static keys, since they're only read when caches are created and not on the allocation path.

v2:

- Make SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM default y instead of adding boot parameters for them (Vlastimil).
- Skip the memdup_user() part of the slab_buckets test on !MMU (Sashiko).
- Keep slab_buckets= and kmalloc_partition= separate for now. We can discuss whether to merge them into slab_hardening= or similar, as Harry suggested.

v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]

[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]
[3] https://lore.kernel.org/r/aud4gPOUzvHtLYnJ@thinkstation

Chris Down (6):
mm/slab: Enable SLAB_FREELIST_HARDENED by default mm/slab: Enable SLAB_FREELIST_RANDOM by default mm/slab: Support disabling SLAB_BUCKETS at boot
slub_kunit: Add a test for the slab_buckets boot parameter mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
slub_kunit: Add a test for the kmalloc_partition boot parameter

 .../admin-guide/kernel-parameters.txt         | 15 ++++
 lib/tests/slub_kunit.c                        | 85 +++++++++++++++++++
 mm/Kconfig                                    | 11 +++
 mm/slab.h                                     |  3 +
 mm/slab_common.c                              | 52 ++++++++++--
 5 files changed, 157 insertions(+), 9 deletions(-)

base-commit: ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75

Signature · 1 line
2.56.1
author
#1 lore raw

PATCH v3 1/6 mm/slab: Enable SLAB_FREELIST_HARDENED by default

SLAB_FREELIST_HARDENED obfuscates the free pointers stored in free objects, and catches naive double frees and frees to the wrong cache. Most distros already enable it, and with sheaves its cost is small, since the per-cpu fast paths don't touch freelists. Measurements also show this quite clearly [1][2].

So let's just enable it by default. SLAB_BUCKETS defaults to SLAB_FREELIST_HARDENED, so it's now enabled by default too.

---
 mm/Kconfig | 1 +
 1 file changed, 1 insertion(+)
mm/Kconfig +1 -0
index afff1b25b543..f330c2c29fd2 100644
@@ -208,6 +208,7 @@ config SLAB_FREELIST_RANDOM
 config SLAB_FREELIST_HARDENED
 	bool "Harden slab freelist metadata"
 	depends on !SLUB_TINY
+	default y
 	help
 	  Many kernel heap attacks try to target slab cache metadata and
 	  other infrastructure. This options makes minor performance
Signature · 1 line
2.56.1
author
#2 lore raw

PATCH v3 2/6 mm/slab: Enable SLAB_FREELIST_RANDOM by default

In reply to Chris Down

SLAB_FREELIST_RANDOM randomises the order in which the objects in a new slab are handed out. Like SLAB_FREELIST_HARDENED, most distros already enable it, and its cost is only paid when a new slab is set up.

With it on, the same system level benchmarks didn't change measurably. Allocating from new slabs does get slower, by up to about 30% per object in a microbenchmark which does little else.

So let's enable it by default too. SHUFFLE_PAGE_ALLOCATOR defaults to SLAB_FREELIST_RANDOM && ACPI_NUMA, so it's now built in by default on those systems, but page shuffling itself stays off unless it's enabled with page_alloc.shuffle.

---
 mm/Kconfig | 1 +
 1 file changed, 1 insertion(+)
mm/Kconfig +1 -0
index f330c2c29fd2..d174537d941e 100644
@@ -200,6 +200,7 @@ config SLAB_MERGE_DEFAULT
 config SLAB_FREELIST_RANDOM
 	bool "Randomize slab freelist"
 	depends on !SLUB_TINY
+	default y
 	help
 	  Randomizes the freelist order used on creating new pages. This
 	  security feature reduces the predictability of the kernel slab
Signature · 1 line
2.56.1
author
#3 lore raw

PATCH v3 3/6 mm/slab: Support disabling SLAB_BUCKETS at boot

In reply to Chris Down

SLAB_BUCKETS gives allocations with user-controlled size and contents, like memdup_user() and msg_msg, their own set of kmalloc caches, at the cost of the memory for those caches. It can only be chosen at build time, so hosts which want it and hosts which would rather not pay for it need separate kernels.

Add a slab_buckets= early parameter, enabled by default on kernels built with CONFIG_SLAB_BUCKETS=y, so that it can be turned off at boot.

---
 .../admin-guide/kernel-parameters.txt         |  8 ++++++
 mm/Kconfig                                    |  4 +++
 mm/slab_common.c                              | 26 +++++++++++++++----
 3 files changed, 33 insertions(+), 5 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +8 -0
index 0b3c140babd9..c27bbe205051 100644
@@ -6990,6 +6990,14 @@ Kernel parameters
 			apic=verbose is specified.
 			Example: apic=debug show_lapic=all
 
+	slab_buckets= [MM,EARLY]
+			Format: <bool>
+			Enable or disable separate kmalloc buckets on kernels
+			built with CONFIG_SLAB_BUCKETS=y. When disabled,
+			allocations use regular kmalloc caches. Ignored
+			without that option.
+			Default: on.
+
 	slab_debug[=options[,slabs][;[options[,slabs]]...]]	[MM]
 			Enabling slab_debug allows one to determine the
 			culprit if slab objects become corrupted. Enabling
mm/Kconfig +4 -0
index d174537d941e..e602ebabd6e4 100644
@@ -231,6 +231,10 @@ config SLAB_BUCKETS
 	  of extra pages since the bulk of user-controlled allocations
 	  are relatively long-lived.
 
+	  The separate buckets can be disabled at boot with the
+	  "slab_buckets=0" kernel parameter, in which case these
+	  allocations use the regular kmalloc caches.
+
 	  If unsure, say Y.
 
 config SLUB_STATS
mm/slab_common.c +21 -5
index 01c64ef661e7..272d22f411c2 100644
@@ -410,6 +410,16 @@ EXPORT_SYMBOL(__kmem_cache_create_args);
 
 static struct kmem_cache *kmem_buckets_cache __ro_after_init;
 
+static bool slab_buckets_enabled __ro_after_init = IS_ENABLED(CONFIG_SLAB_BUCKETS);
+
+static int __init setup_slab_buckets(char *str)
+{
+	if (!IS_ENABLED(CONFIG_SLAB_BUCKETS))
+		return 0;
+	return kstrtobool(str, &slab_buckets_enabled);
+}
+early_param("slab_buckets", setup_slab_buckets);
+
 /**
  * kmem_buckets_create - Create a set of caches that handle dynamic sized
  *			 allocations via kmem_buckets_alloc()
@@ -424,10 +434,12 @@ static struct kmem_cache *kmem_buckets_cache __ro_after_init;
  *
  * Context: Cannot be called within an interrupt, but can be interrupted.
  *
- * Return: a pointer to the cache on success, NULL on failure. When
- * CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned, and
- * subsequent calls to kmem_buckets_alloc() will fall back to kmalloc().
- * (i.e. callers only need to check for NULL on failure.)
+ * Return: a pointer to the cache on success, NULL on failure or when the
+ * separate buckets were disabled with the "slab_buckets=0" boot parameter.
+ * When CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned. In all
+ * of these cases subsequent calls to kmem_buckets_alloc() with the returned
+ * value fall back to kmalloc(), so callers that can live without separate
+ * buckets do not need to check the return value.
  */
 kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags,
 				  unsigned int useroffset,
@@ -448,6 +460,10 @@ kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags,
 	if (!IS_ENABLED(CONFIG_SLAB_BUCKETS))
 		return ZERO_SIZE_PTR;
 
+	/* NULL makes kmalloc_slab() use the regular kmalloc caches. */
+	if (!slab_buckets_enabled)
+		return NULL;
+
 	if (WARN_ON(!kmem_buckets_cache))
 		return NULL;
 
@@ -1059,7 +1075,7 @@ void __init create_kmalloc_caches(void)
 	/* Kmalloc array is now usable */
 	slab_state = UP;
 
-	if (IS_ENABLED(CONFIG_SLAB_BUCKETS))
+	if (IS_ENABLED(CONFIG_SLAB_BUCKETS) && slab_buckets_enabled)
 		kmem_buckets_cache = kmem_cache_create("kmalloc_buckets",
 						       sizeof(kmem_buckets),
 						       0, SLAB_NO_MERGE, NULL);
Signature · 1 line
2.56.1
author
#5 lore raw

PATCH v3 4/6 slub_kunit: Add a test for the slab_buckets boot parameter

In reply to Chris Down

Check that memdup_user() allocates from the memdup_user buckets by default, and from the regular kmalloc caches when they're turned off with slab_buckets=0. The test is skipped on kernels built without CONFIG_SLAB_BUCKETS.

Make slab_buckets_enabled global and export it to KUnit so that the test can see it.

---
 lib/tests/slub_kunit.c | 40 ++++++++++++++++++++++++++++++++++++++++
 mm/slab.h              |  2 ++
 mm/slab_common.c       |  4 +++-
 3 files changed, 45 insertions(+), 1 deletion(-)
lib/tests/slub_kunit.c +40 -0
index e3b63f0338d5..1af3f087134f 100644
@@ -9,6 +9,8 @@
 #include <linux/delay.h>
 #include <linux/perf_event.h>
 #include <linux/kprobes.h>
+#include <linux/mman.h>
+#include <linux/string.h>
 #include "../mm/slab.h"
 
 static struct kunit_resource resource;
@@ -296,6 +298,42 @@ static void test_krealloc_redzone_zeroing(struct kunit *test)
 	kmem_cache_destroy(s);
 }
 
+static void test_slab_buckets(struct kunit *test)
+{
+	unsigned long user_addr;
+	const char *name;
+	void *p;
+
+	if (!IS_ENABLED(CONFIG_SLAB_BUCKETS))
+		kunit_skip(test, "requires CONFIG_SLAB_BUCKETS");
+
+	if (!slab_buckets_enabled)
+		KUNIT_EXPECT_NULL(test, kmem_buckets_create("TestSlub_buckets",
+							    0, 0, INT_MAX, NULL));
+
+	if (!IS_ENABLED(CONFIG_MMU))
+		kunit_skip(test, "userspace allocation needs an MMU");
+
+	user_addr = kunit_vm_mmap(test, NULL, 0, PAGE_SIZE,
+				  PROT_READ | PROT_WRITE,
+				  MAP_ANONYMOUS | MAP_PRIVATE, 0);
+	KUNIT_ASSERT_NE_MSG(test, user_addr, 0, "Could not create userspace mm");
+	KUNIT_ASSERT_LT_MSG(test, user_addr, (unsigned long)TASK_SIZE,
+			    "Failed to allocate user memory");
+
+	p = memdup_user((void __user *)user_addr, 64);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, p);
+
+	name = virt_to_slab(p)->slab_cache->name;
+	if (slab_buckets_enabled)
+		KUNIT_EXPECT_TRUE_MSG(test, strstarts(name, "memdup_user-"),
+				      "allocated from %s", name);
+	else
+		KUNIT_EXPECT_TRUE_MSG(test, strstarts(name, "kmalloc-"),
+				      "allocated from %s", name);
+	kfree(p);
+}
+
 #if defined(CONFIG_PERF_EVENTS) || (defined(CONFIG_KPROBES) && defined(CONFIG_SMP))
 #define NR_ITERATIONS 1000
 #define NR_OBJECTS 1000
@@ -489,6 +527,7 @@ static struct kunit_case test_cases[] = {
 	KUNIT_CASE(test_kfree_rcu_wq_destroy),
 	KUNIT_CASE(test_leak_destroy),
 	KUNIT_CASE(test_krealloc_redzone_zeroing),
+	KUNIT_CASE(test_slab_buckets),
 #ifdef CONFIG_PERF_EVENTS
 	KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_perf),
 #endif
@@ -505,5 +544,6 @@ static struct kunit_suite test_suite = {
 };
 kunit_test_suite(test_suite);
 
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
 MODULE_DESCRIPTION("Kunit tests for slub allocator");
 MODULE_LICENSE("GPL");
mm/slab.h +2 -0
index e94ac63d10dc..6a28951f6d5d 100644
@@ -390,6 +390,8 @@ extern struct list_head slab_caches;
 /* The slab cache that manages slab cache information */
 extern struct kmem_cache *kmem_cache;
 
+extern bool slab_buckets_enabled;
+
 /* A table of kmalloc cache names and sizes */
 extern const struct kmalloc_info_struct {
 	const char *name[NR_KMALLOC_TYPES];
mm/slab_common.c +3 -1
index 272d22f411c2..5575706b8938 100644
@@ -29,6 +29,7 @@
 #include <linux/memcontrol.h>
 #include <linux/stackdepot.h>
 #include <trace/events/rcu.h>
+#include <kunit/visibility.h>
 
 #include "../kernel/rcu/rcu.h"
 #include "internal.h"
@@ -410,7 +411,8 @@ EXPORT_SYMBOL(__kmem_cache_create_args);
 
 static struct kmem_cache *kmem_buckets_cache __ro_after_init;
 
-static bool slab_buckets_enabled __ro_after_init = IS_ENABLED(CONFIG_SLAB_BUCKETS);
+bool slab_buckets_enabled __ro_after_init = IS_ENABLED(CONFIG_SLAB_BUCKETS);
+EXPORT_SYMBOL_IF_KUNIT(slab_buckets_enabled);
 
 static int __init setup_slab_buckets(char *str)
 {
Signature · 1 line
2.56.1
author
#6 lore raw

PATCH v3 5/6 mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot

In reply to Chris Down

KMALLOC_PARTITION_CACHES creates several copies of each normal kmalloc cache and spreads allocations across them. That costs a fair bit of
memory: Harry measured about 18 MiB more slab memory after boot on a 64 CPU, 64 GiB VM, mostly from under-used slabs [1]. Like SLAB_BUCKETS, it can only be chosen at build time.

Add a kmalloc_partition= early parameter, enabled by default on kernels built with CONFIG_KMALLOC_PARTITION_CACHES=y. When it's off, the copies alias the normal caches rather than being created.

Let's leave kmalloc_type() unchanged since each index simply resolves to the same cache when partitioning is off.

---
 .../admin-guide/kernel-parameters.txt         |  7 ++++++
 mm/Kconfig                                    |  5 ++++
 mm/slab_common.c                              | 23 +++++++++++++++----
 3 files changed, 31 insertions(+), 4 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +7 -0
index c27bbe205051..91caeef33c3a 100644
@@ -3045,6 +3045,13 @@ Kernel parameters
 			Configure the RouterBoard 532 series on-chip
 			Ethernet adapter MAC address.
 
+	kmalloc_partition= [MM,EARLY]
+			Format: <bool>
+			Enable or disable kmalloc cache partitioning on
+			kernels built with CONFIG_KMALLOC_PARTITION_CACHES=y.
+			Ignored without that option.
+			Default: on.
+
 	kmemleak=	[KNL,EARLY] Boot-time kmemleak enable/disable
 			Valid arguments: on, off
 			Default: on
mm/Kconfig +5 -0
index e602ebabd6e4..5bb6e33fab49 100644
@@ -266,6 +266,11 @@ config KMALLOC_PARTITION_CACHES
 	  limited degree of memory and CPU overhead that relates to hardware
 	  and system workload.
 
+	  The partitioning can be disabled at boot with the
+	  "kmalloc_partition=0" kernel parameter, in which case the copies
+	  are not created and all normal kmalloc allocations use the
+	  regular kmalloc caches.
+
 choice
 	prompt "Partitioned slab cache mode"
 	depends on KMALLOC_PARTITION_CACHES
mm/slab_common.c +19 -4
index 5575706b8938..20b066d61703 100644
@@ -781,6 +781,17 @@ unsigned long random_kmalloc_seed __ro_after_init;
 EXPORT_SYMBOL(random_kmalloc_seed);
 #endif
 
+static bool kmalloc_partition_enabled __ro_after_init =
+	IS_ENABLED(CONFIG_KMALLOC_PARTITION_CACHES);
+
+static int __init setup_kmalloc_partition(char *str)
+{
+	if (!IS_ENABLED(CONFIG_KMALLOC_PARTITION_CACHES))
+		return 0;
+	return kstrtobool(str, &kmalloc_partition_enabled);
+}
+early_param("kmalloc_partition", setup_kmalloc_partition);
+
 /*
  * Conversion table for small slabs sizes / 8 to the index in the
  * kmalloc array. This is necessary for slabs < 192 since we have non power
@@ -1012,10 +1023,14 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
 		flags |= SLAB_CACHE_DMA;
 	}
 
-#ifdef CONFIG_KMALLOC_PARTITION_CACHES
-	if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
-		flags |= SLAB_NO_MERGE;
-#endif
+	if (kmalloc_partition_enabled) {
+		if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
+			flags |= SLAB_NO_MERGE;
+	} else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {
+		/* Use the normal caches, which have already been created. */
+		kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
+		return;
+	}
 
 	/*
 	 * If memcg_kmem is enabled and this is a KMALLOC_NORMAL cache and not
Signature · 1 line
2.56.1
author
#9 lore raw

PATCH v3 6/6 slub_kunit: Add a test for the kmalloc_partition boot parameter

In reply to Chris Down

Check that the kmalloc partitions are separate caches by default, and alias the normal kmalloc caches when they're turned off with kmalloc_partition=0. The test is skipped on kernels built without CONFIG_KMALLOC_PARTITION_CACHES.

Make kmalloc_partition_enabled global and export it to KUnit so that the test can see it.

---
 lib/tests/slub_kunit.c | 45 ++++++++++++++++++++++++++++++++++++++++++
 mm/slab.h              |  1 +
 mm/slab_common.c       |  3 ++-
 3 files changed, 48 insertions(+), 1 deletion(-)
lib/tests/slub_kunit.c +45 -0
index 1af3f087134f..61227c242ae2 100644
@@ -334,6 +334,50 @@ static void test_slab_buckets(struct kunit *test)
 	kfree(p);
 }
 
+/*
+ * Each partition copy is a separate cache, unless the copies were disabled
+ * with kmalloc_partition=0: then they alias the KMALLOC_NORMAL caches.
+ */
+static void test_kmalloc_partition(struct kunit *test)
+{
+	enum kmalloc_cache_type type;
+	struct kmem_cache *s;
+	unsigned int idx;
+	bool found;
+	void *p;
+
+	if (!IS_ENABLED(CONFIG_KMALLOC_PARTITION_CACHES))
+		kunit_skip(test, "requires CONFIG_KMALLOC_PARTITION_CACHES");
+
+	for (type = KMALLOC_PARTITION_START + 1; type <= KMALLOC_PARTITION_END; type++) {
+		for (idx = 0; idx < ARRAY_SIZE(kmalloc_caches[type]); idx++) {
+			struct kmem_cache *normal = kmalloc_caches[KMALLOC_NORMAL][idx];
+			struct kmem_cache *part = kmalloc_caches[type][idx];
+
+			if (!kmalloc_partition_enabled) {
+				KUNIT_EXPECT_PTR_EQ(test, part, normal);
+			} else if (normal) {
+				KUNIT_EXPECT_NOT_NULL(test, part);
+				KUNIT_EXPECT_PTR_NE(test, part, normal);
+			}
+		}
+	}
+
+	p = kmalloc(64, GFP_KERNEL);
+	KUNIT_ASSERT_NOT_NULL(test, p);
+	s = virt_to_slab(p)->slab_cache;
+	idx = kmalloc_index(64);
+	if (!kmalloc_partition_enabled) {
+		KUNIT_EXPECT_PTR_EQ(test, s, kmalloc_caches[KMALLOC_NORMAL][idx]);
+	} else {
+		found = false;
+		for (type = KMALLOC_PARTITION_START; type <= KMALLOC_PARTITION_END; type++)
+			found |= s == kmalloc_caches[type][idx];
+		KUNIT_EXPECT_TRUE_MSG(test, found, "allocated from %s", s->name);
+	}
+	kfree(p);
+}
+
 #if defined(CONFIG_PERF_EVENTS) || (defined(CONFIG_KPROBES) && defined(CONFIG_SMP))
 #define NR_ITERATIONS 1000
 #define NR_OBJECTS 1000
@@ -528,6 +572,7 @@ static struct kunit_case test_cases[] = {
 	KUNIT_CASE(test_leak_destroy),
 	KUNIT_CASE(test_krealloc_redzone_zeroing),
 	KUNIT_CASE(test_slab_buckets),
+	KUNIT_CASE(test_kmalloc_partition),
 #ifdef CONFIG_PERF_EVENTS
 	KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_perf),
 #endif
mm/slab.h +1 -0
index 6a28951f6d5d..2a9d6a9a9948 100644
@@ -391,6 +391,7 @@ extern struct list_head slab_caches;
 extern struct kmem_cache *kmem_cache;
 
 extern bool slab_buckets_enabled;
+extern bool kmalloc_partition_enabled;
 
 /* A table of kmalloc cache names and sizes */
 extern const struct kmalloc_info_struct {
mm/slab_common.c +2 -1
index 20b066d61703..d0695e4d2ab6 100644
@@ -781,8 +781,9 @@ unsigned long random_kmalloc_seed __ro_after_init;
 EXPORT_SYMBOL(random_kmalloc_seed);
 #endif
 
-static bool kmalloc_partition_enabled __ro_after_init =
+bool kmalloc_partition_enabled __ro_after_init =
 	IS_ENABLED(CONFIG_KMALLOC_PARTITION_CACHES);
+EXPORT_SYMBOL_IF_KUNIT(kmalloc_partition_enabled);
 
 static int __init setup_kmalloc_partition(char *str)
 {
Signature · 1 line
2.56.1
#10 lore raw
In reply to Chris Down
Quoted 14 lines · Chris Down

Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. At Meta, for example, some hosts are built with these options and the rest of the fleet isn't, and that's one of the things that keeps those kernels apart.

This series does two things:

- SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM are cheap enough with sheaves that it makes more sense to just enable them by default than to make them switchable. With both on, system level benchmarks didn't change measurably, only slowpath heavy microbenchmarks did [1].

Quoted 7 lines

- SLAB_BUCKETS and KMALLOC_PARTITION_CACHES cost memory (Harry measured about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so they get slab_buckets= and kmalloc_partition= boot parameters, which default to on when the option is built in.

Whether those two should be folded into a single slab_hardening= is still open [3].

A coarse slab_hardening= option can easily be replaced by documentation. While less knobs is better, an intransparent coarse variant of slab_hardening= just makes the admin wonder "what does that actually to, and do I want it?". They still have to go and look up what it actually does, and at that point it doesn't add much value (alternative suggestion below).

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

light: "I'm willying to sacrifice _some_ memory and CPU for security"
full: "I'm willying to sacrifice more overhead for security

Who will decide what is light and full? I think there just won't be any consensus reached.

One thing that users may want is an "all off" option, similar to mitigations=off, should they wish to get max performance back with a single option. So you could consider slab_hardening=off. But at that point it may be more consistent to drop slab_buckets= and kmalloc_partition= in favor of arguments to slab_hardening=, such as slab_hardening=nobuckets,partition. This would mirror mitigations=, and should cater to the usecase of slab_hardening=off to get max performance with a single option.

But I would not add options such as 'light', 'full'.

Thanks,
-- Marco

maintainer
#11 lore raw
Quoted 30 lines · Marco Elver
Quoted 14 lines · Chris Down

Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. At Meta, for example, some hosts are built with these options and the rest of the fleet isn't, and that's one of the things that keeps those kernels apart.

This series does two things:

- SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM are cheap enough with sheaves that it makes more sense to just enable them by default than to make them switchable. With both on, system level benchmarks didn't change measurably, only slowpath heavy microbenchmarks did [1].

Quoted 7 lines

- SLAB_BUCKETS and KMALLOC_PARTITION_CACHES cost memory (Harry measured about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so they get slab_buckets= and kmalloc_partition= boot parameters, which default to on when the option is built in.

Whether those two should be folded into a single slab_hardening= is still open [3].

A coarse slab_hardening= option can easily be replaced by documentation. While less knobs is better, an intransparent coarse variant of slab_hardening= just makes the admin wonder "what does that actually to, and do I want it?". They still have to go and look up what it actually does, and at that point it doesn't add much value (alternative suggestion below).

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

light: "I'm willying to sacrifice _some_ memory and CPU for security"
full: "I'm willying to sacrifice more overhead for security

Who will decide what is light and full? I think there just won't be any consensus reached.

You're probably right.

Quoted 3 lines

One thing that users may want is an "all off" option, similar to mitigations=off, should they wish to get max performance back with a single option. So you could consider slab_hardening=off.

slab_hardening=off sounds good to me.

However this assumes "all on" is the right thing to do by default, but not enabling slab hardening is not as nearly critical as not enabling mitigations to CPU vulnerabilities, no?

I'd rather mirror hardened_usercopy= instead of mitigations=off.

slab_hardening=on (turn on all slab hardening features) slab_hardening=off (turn off all slab hardening features) (Otherwise it is determined by CONFIG_*.)

Quoted 5 lines

But at that point it may be more consistent to drop slab_buckets= and kmalloc_partition= in favor of arguments to slab_hardening=, such as slab_hardening=nobuckets,partition. This would mirror mitigations=, and should cater to the usecase of slab_hardening=off to get max performance with a single option.

I hope we don't reach this point, but more fine-grained control should be achivable if it is indeed required.

Quoted 1 line

But I would not add options such as 'light', 'full'.

Fair enough.

Quoted 2 lines

Thanks,
-- Marco

Signature · 2 lines
Cheers,
Harry / Hyeonggon
maintainer
#12 lore raw
Quoted 49 lines · Harry Yoo
Quoted 30 lines · Marco Elver
Quoted 14 lines · Chris Down

Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. At Meta, for example, some hosts are built with these options and the rest of the fleet isn't, and that's one of the things that keeps those kernels apart.

This series does two things:

- SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM are cheap enough with sheaves that it makes more sense to just enable them by default than to make them switchable. With both on, system level benchmarks didn't change measurably, only slowpath heavy microbenchmarks did [1].

Quoted 7 lines

- SLAB_BUCKETS and KMALLOC_PARTITION_CACHES cost memory (Harry measured about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so they get slab_buckets= and kmalloc_partition= boot parameters, which default to on when the option is built in.

Whether those two should be folded into a single slab_hardening= is still open [3].

A coarse slab_hardening= option can easily be replaced by documentation. While less knobs is better, an intransparent coarse variant of slab_hardening= just makes the admin wonder "what does that actually to, and do I want it?". They still have to go and look up what it actually does, and at that point it doesn't add much value (alternative suggestion below).

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

light: "I'm willying to sacrifice _some_ memory and CPU for security"
full: "I'm willying to sacrifice more overhead for security

Who will decide what is light and full? I think there just won't be any consensus reached.

You're probably right.

Quoted 3 lines

One thing that users may want is an "all off" option, similar to mitigations=off, should they wish to get max performance back with a single option. So you could consider slab_hardening=off.

slab_hardening=off sounds good to me.

However this assumes "all on" is the right thing to do by default, but not enabling slab hardening is not as nearly critical as not enabling mitigations to CPU vulnerabilities, no?

Quoted 5 lines

I'd rather mirror hardened_usercopy= instead of mitigations=off.

slab_hardening=on (turn on all slab hardening features) slab_hardening=off (turn off all slab hardening features) (Otherwise it is determined by CONFIG_*.)

Should have been CONFIG_*_DEFAULT_ON.

Quoted 19 lines
Quoted 5 lines

But at that point it may be more consistent to drop slab_buckets= and kmalloc_partition= in favor of arguments to slab_hardening=, such as slab_hardening=nobuckets,partition. This would mirror mitigations=, and should cater to the usecase of slab_hardening=off to get max performance with a single option.

I hope we don't reach this point, but more fine-grained control should be achivable if it is indeed required.

Quoted 1 line

But I would not add options such as 'light', 'full'.

Fair enough.

Quoted 2 lines

Thanks,
-- Marco

Signature · 2 lines
Cheers,
Harry / Hyeonggon
Signature · 2 lines
Cheers,
Harry / Hyeonggon
#7 lore raw

Re: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot

In reply to Chris Down
kernel test robot noticed the following build warnings:

Hi Chris,

kernel test robot noticed the following build warnings:

[auto build test WARNING on ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75]

url:    https://github.com/intel-lab-lkp/linux/commits/Chris-Down/mm-slab-Enable-SLAB_FREELIST_HARDENED-by-default/20261113-005125
base:   ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75
patch link:    https://lore.kernel.org/r/9a4caff0336c78a7adc746786c81c9c01f89c40b.1794506262.git.chris%40chrisdown.name
patch subject: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
config: hexagon-allnoconfig (https://download.01.org/0day-ci/archive/20261114/[email protected]/config)
compiler: clang version 24.0.0git (https://github.com/llvm/llvm-project 1b1fed44d3dcca649a9877915d57c3357a98124e)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261114/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> mm/slab_common.c:1029:44: warning: non-overlapping comparisons always evaluate to false [-Wtautological-overlap-compare]
    1029 |         } else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {
         |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1 warning generated.

vim +1029 mm/slab_common.c

999
1000 static void __init
1001 new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
1002 {
1003 slab_flags_t flags = 0;
1004 unsigned int minalign = __kmalloc_minalign();
1005 unsigned int aligned_size = kmalloc_info[idx].size;
1006 int aligned_idx = idx;
1007
1008 if ((KMALLOC_RECLAIM != KMALLOC_NORMAL) && (type == KMALLOC_RECLAIM)) {
1009 flags |= SLAB_RECLAIM_ACCOUNT;
1010 } else if (IS_ENABLED(CONFIG_MEMCG) && (type == KMALLOC_CGROUP)) {
1011 if (mem_cgroup_kmem_disabled()) {
1012 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1013 return;
1014 }
1015 flags |= SLAB_ACCOUNT;
1016 } else if (IS_ENABLED(CONFIG_SLAB_OBJ_EXT) && type == KMALLOC_NO_OBJ_EXT) {
1017 if (!need_kmalloc_no_objext()) {
1018 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1019 return;
1020 }
1021 flags |= SLAB_NO_OBJ_EXT | SLAB_NO_MERGE;
1022 } else if (IS_ENABLED(CONFIG_ZONE_DMA) && (type == KMALLOC_DMA)) {
1023 flags |= SLAB_CACHE_DMA;
1024 }
1025
1026 if (kmalloc_partition_enabled) {
1027 if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
1028 flags |= SLAB_NO_MERGE;

> 1029		} else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {

1030 /* Use the normal caches, which have already been created. */
1031 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1032 return;
1033 }
1034
1035 /*
1036 * If memcg_kmem is enabled and this is a KMALLOC_NORMAL cache and not
1037 * aliased with any other type, make sure it's never merged with any other
1038 * cache.
1039 *
1040 * In other cases the kmalloc cache may end up being used for a
1041 * __GFP_ACCOUNT allocation so mark it as such. The exception is a
1042 * KMALLOC_NO_OBJ_EXT cache.
1043 */
1044 if (!mem_cgroup_kmem_disabled()) {
1045 if (type == KMALLOC_NORMAL && KMALLOC_RECLAIM != KMALLOC_NORMAL)
1046 flags |= SLAB_NO_MERGE;
1047 else if (!(flags & SLAB_NO_OBJ_EXT))
1048 flags |= SLAB_MAY_ACCOUNT;
1049 }
1050
1051 if (minalign > ARCH_KMALLOC_MINALIGN) {
1052 aligned_size = ALIGN(aligned_size, minalign);
1053 aligned_idx = __kmalloc_index(aligned_size, false);
1054 }
1055
1056 if (!kmalloc_caches[type][aligned_idx])
1057 kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
1058 kmalloc_info[aligned_idx].name[type],
1059 aligned_size, flags);
1060 if (idx != aligned_idx)
1061 kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
1062 }
1063

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

#8 lore raw

Re: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot

In reply to Chris Down
kernel test robot noticed the following build warnings:

Hi Chris,

kernel test robot noticed the following build warnings:

[auto build test WARNING on ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75]

url:    https://github.com/intel-lab-lkp/linux/commits/Chris-Down/mm-slab-Enable-SLAB_FREELIST_HARDENED-by-default/20261113-005125
base:   ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75
patch link:    https://lore.kernel.org/r/9a4caff0336c78a7adc746786c81c9c01f89c40b.1794506262.git.chris%40chrisdown.name
patch subject: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
config: arm-allnoconfig (https://download.01.org/0day-ci/archive/20261114/[email protected]/config)
compiler: clang version 17.0.6 (https://github.com/llvm/llvm-project 6009708b4367171ccdbf4b5905cb6a803753fe18)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261114/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

>> mm/slab_common.c:1029:44: warning: overlapping comparisons always evaluate to false [-Wtautological-overlap-compare]
    1029 |         } else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {
         |                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1 warning generated.

vim +1029 mm/slab_common.c

999
1000 static void __init
1001 new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
1002 {
1003 slab_flags_t flags = 0;
1004 unsigned int minalign = __kmalloc_minalign();
1005 unsigned int aligned_size = kmalloc_info[idx].size;
1006 int aligned_idx = idx;
1007
1008 if ((KMALLOC_RECLAIM != KMALLOC_NORMAL) && (type == KMALLOC_RECLAIM)) {
1009 flags |= SLAB_RECLAIM_ACCOUNT;
1010 } else if (IS_ENABLED(CONFIG_MEMCG) && (type == KMALLOC_CGROUP)) {
1011 if (mem_cgroup_kmem_disabled()) {
1012 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1013 return;
1014 }
1015 flags |= SLAB_ACCOUNT;
1016 } else if (IS_ENABLED(CONFIG_SLAB_OBJ_EXT) && type == KMALLOC_NO_OBJ_EXT) {
1017 if (!need_kmalloc_no_objext()) {
1018 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1019 return;
1020 }
1021 flags |= SLAB_NO_OBJ_EXT | SLAB_NO_MERGE;
1022 } else if (IS_ENABLED(CONFIG_ZONE_DMA) && (type == KMALLOC_DMA)) {
1023 flags |= SLAB_CACHE_DMA;
1024 }
1025
1026 if (kmalloc_partition_enabled) {
1027 if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
1028 flags |= SLAB_NO_MERGE;

> 1029		} else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {

1030 /* Use the normal caches, which have already been created. */
1031 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1032 return;
1033 }
1034
1035 /*
1036 * If memcg_kmem is enabled and this is a KMALLOC_NORMAL cache and not
1037 * aliased with any other type, make sure it's never merged with any other
1038 * cache.
1039 *
1040 * In other cases the kmalloc cache may end up being used for a
1041 * __GFP_ACCOUNT allocation so mark it as such. The exception is a
1042 * KMALLOC_NO_OBJ_EXT cache.
1043 */
1044 if (!mem_cgroup_kmem_disabled()) {
1045 if (type == KMALLOC_NORMAL && KMALLOC_RECLAIM != KMALLOC_NORMAL)
1046 flags |= SLAB_NO_MERGE;
1047 else if (!(flags & SLAB_NO_OBJ_EXT))
1048 flags |= SLAB_MAY_ACCOUNT;
1049 }
1050
1051 if (minalign > ARCH_KMALLOC_MINALIGN) {
1052 aligned_size = ALIGN(aligned_size, minalign);
1053 aligned_idx = __kmalloc_index(aligned_size, false);
1054 }
1055
1056 if (!kmalloc_caches[type][aligned_idx])
1057 kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
1058 kmalloc_info[aligned_idx].name[type],
1059 aligned_size, flags);
1060 if (idx != aligned_idx)
1061 kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
1062 }
1063

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

#4 lore raw

Re: [PATCH v3 3/6] mm/slab: Support disabling SLAB_BUCKETS at boot

In reply to Chris Down
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/slab: setup_slab_buckets() adds per-feature knob instead of slab_hardening --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/slab: setup_slab_buckets() adds per-feature knob instead of slab_hardening
--

commit d1769f1af5f0bdc819b4474cbb49e0bbed23c8ff
Author: Chris Down <[email protected]>

mm/slab: Support disabling SLAB_BUCKETS at boot

This commit adds a slab_buckets= early boot parameter to allow disabling separate kmalloc buckets on kernels built with CONFIG_SLAB_BUCKETS=y.

Quoted 7 lines
mm/slab_common.c +0 -0
index 01c64ef661e7f..272d22f411c2c 100644
@@ -410,6 +410,16 @@ EXPORT_SYMBOL(__kmem_cache_create_args);
 
 static struct kmem_cache *kmem_buckets_cache __ro_after_init;

mm/slab_common.c EXPORT_SYMBOL(__kmem_cache_create_args); · line 421

Quoted 10 lines
mm/slab_common.c +9 -0
 
+static bool slab_buckets_enabled __ro_after_init = IS_ENABLED(CONFIG_SLAB_BUCKETS);
+
+static int __init setup_slab_buckets(char *str)
+{
+	if (!IS_ENABLED(CONFIG_SLAB_BUCKETS))
+		return 0;
+	return kstrtobool(str, &slab_buckets_enabled);
+}
+early_param("slab_buckets", setup_slab_buckets);

[Severity: Low]
As reported by Harry Yoo on v1 [1], should boot-time slab hardening controls use a single coarse-grained parameter such as slab_hardening= rather than introducing separate per-feature knobs like slab_buckets= here and kmalloc_partition= in commit "mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot"?

[1] https://lore.kernel.org/r/auIN44CnAyzvR_Ll@thinkstation

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#13 lore raw
In reply to Chris Down
Quoted 13 lines · Chris Down

Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. At Meta, for example, some hosts are built with these options and the rest of the fleet isn't, and that's one of the things that keeps those kernels apart.

This series does two things:

- SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM are cheap enough with sheaves that it makes more sense to just enable them by default than to make them switchable. With both on, system level benchmarks didn't change measurably, only slowpath heavy microbenchmarks did [1].

Quoted 4 lines

- SLAB_BUCKETS and KMALLOC_PARTITION_CACHES cost memory (Harry measured about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so they get slab_buckets= and kmalloc_partition= boot parameters, which default to on when the option is built in.

I'm not thrilled about the "default to on". We usually do these features opt-in. Now our distro kernel would likely need to have these "disable this" options in the default command line, with documentation to remove it for a hardened setup.

But yeah doing it opt-in, the hardened setups would now have to start passing the extra params to enable this, even if until now enabling the config option was enough. Also not great, but we can argue they are still the minority?

Of course there's always the possibility of adding another _DEFAUT_ON configs. Not great either. Sigh.

Quoted 52 lines

Whether those two should be folded into a single slab_hardening= is still open [3].

v3:

- Rebase onto slab/for-next.
- Always build the slab_buckets and kmalloc_partition tests and skip them at runtime when the option isn't built in, instead of leaving them out with #ifdef (Kees).
- Declare slab_buckets_enabled and kmalloc_partition_enabled in mm/slab.h without any #if (Kees).
- Drop the remaining #ifdefs around slab_buckets= and kmalloc_partition=. Both are now always registered and ignored when the option isn't built in (Kees).
- Keep both flags as plain __ro_after_init bools rather than static keys, since they're only read when caches are created and not on the allocation path.

v2:

- Make SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM default y instead of adding boot parameters for them (Vlastimil).
- Skip the memdup_user() part of the slab_buckets test on !MMU (Sashiko).
- Keep slab_buckets= and kmalloc_partition= separate for now. We can discuss whether to merge them into slab_hardening= or similar, as Harry suggested.

v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]

[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]
[3] https://lore.kernel.org/r/aud4gPOUzvHtLYnJ@thinkstation

Chris Down (6):
mm/slab: Enable SLAB_FREELIST_HARDENED by default mm/slab: Enable SLAB_FREELIST_RANDOM by default mm/slab: Support disabling SLAB_BUCKETS at boot
slub_kunit: Add a test for the slab_buckets boot parameter mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
slub_kunit: Add a test for the kmalloc_partition boot parameter

 .../admin-guide/kernel-parameters.txt         | 15 ++++
 lib/tests/slub_kunit.c                        | 85 +++++++++++++++++++
 mm/Kconfig                                    | 11 +++
 mm/slab.h                                     |  3 +
 mm/slab_common.c                              | 52 ++++++++++--
 5 files changed, 157 insertions(+), 9 deletions(-)

base-commit: ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75