mm/slab: Support selecting slab hardening features at boot
Subsystems (4)
v3 cover mm/slab: Support selecting slab hardening features at boot
Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. At Meta, for example, some hosts are built with these options and the rest of the fleet isn't, and that's one of the things that keeps those kernels apart.
This series does two things:
- SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM are cheap enough with sheaves that it makes more sense to just enable them by default than to make them switchable. With both on, system level benchmarks didn't change measurably, only slowpath heavy microbenchmarks did [1].
- SLAB_BUCKETS and KMALLOC_PARTITION_CACHES cost memory (Harry measured about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so they get slab_buckets= and kmalloc_partition= boot parameters, which default to on when the option is built in.
Whether those two should be folded into a single slab_hardening= is still open [3].
v3:
- Rebase onto slab/for-next.
- Always build the slab_buckets and kmalloc_partition tests and skip them at runtime when the option isn't built in, instead of leaving them out with #ifdef (Kees).
- Declare slab_buckets_enabled and kmalloc_partition_enabled in mm/slab.h without any #if (Kees).
- Drop the remaining #ifdefs around slab_buckets= and kmalloc_partition=. Both are now always registered and ignored when the option isn't built in (Kees).
- Keep both flags as plain __ro_after_init bools rather than static keys, since they're only read when caches are created and not on the allocation path.
v2:
- Make SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM default y instead of adding boot parameters for them (Vlastimil).
- Skip the memdup_user() part of the slab_buckets test on !MMU (Sashiko).
- Keep slab_buckets= and kmalloc_partition= separate for now. We can discuss whether to merge them into slab_hardening= or similar, as Harry suggested.
v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected]
[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]
[3] https://lore.kernel.org/r/aud4gPOUzvHtLYnJ@thinkstation
Chris Down (6):
mm/slab: Enable SLAB_FREELIST_HARDENED by default mm/slab: Enable SLAB_FREELIST_RANDOM by default mm/slab: Support disabling SLAB_BUCKETS at boot
slub_kunit: Add a test for the slab_buckets boot parameter mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
slub_kunit: Add a test for the kmalloc_partition boot parameter
.../admin-guide/kernel-parameters.txt | 15 ++++ lib/tests/slub_kunit.c | 85 +++++++++++++++++++ mm/Kconfig | 11 +++ mm/slab.h | 3 + mm/slab_common.c | 52 ++++++++++-- 5 files changed, 157 insertions(+), 9 deletions(-)
base-commit: ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75
Signature · 1 line
2.56.1
PATCH v3 1/6 mm/slab: Enable SLAB_FREELIST_HARDENED by default
SLAB_FREELIST_HARDENED obfuscates the free pointers stored in free objects, and catches naive double frees and frees to the wrong cache. Most distros already enable it, and with sheaves its cost is small, since the per-cpu fast paths don't touch freelists. Measurements also show this quite clearly [1][2].
So let's just enable it by default. SLAB_BUCKETS defaults to SLAB_FREELIST_HARDENED, so it's now enabled by default too.
--- mm/Kconfig | 1 + 1 file changed, 1 insertion(+)
mm/Kconfig +1 -0
@@ -208,6 +208,7 @@ config SLAB_FREELIST_RANDOM config SLAB_FREELIST_HARDENED bool "Harden slab freelist metadata" depends on !SLUB_TINY + default y help Many kernel heap attacks try to target slab cache metadata and other infrastructure. This options makes minor performance
Signature · 1 line
2.56.1
PATCH v3 2/6 mm/slab: Enable SLAB_FREELIST_RANDOM by default
In reply to Chris DownSLAB_FREELIST_RANDOM randomises the order in which the objects in a new slab are handed out. Like SLAB_FREELIST_HARDENED, most distros already enable it, and its cost is only paid when a new slab is set up.
With it on, the same system level benchmarks didn't change measurably. Allocating from new slabs does get slower, by up to about 30% per object in a microbenchmark which does little else.
So let's enable it by default too. SHUFFLE_PAGE_ALLOCATOR defaults to SLAB_FREELIST_RANDOM && ACPI_NUMA, so it's now built in by default on those systems, but page shuffling itself stays off unless it's enabled with page_alloc.shuffle.
--- mm/Kconfig | 1 + 1 file changed, 1 insertion(+)
mm/Kconfig +1 -0
@@ -200,6 +200,7 @@ config SLAB_MERGE_DEFAULT config SLAB_FREELIST_RANDOM bool "Randomize slab freelist" depends on !SLUB_TINY + default y help Randomizes the freelist order used on creating new pages. This security feature reduces the predictability of the kernel slab
Signature · 1 line
2.56.1
PATCH v3 3/6 mm/slab: Support disabling SLAB_BUCKETS at boot
In reply to Chris DownSLAB_BUCKETS gives allocations with user-controlled size and contents, like memdup_user() and msg_msg, their own set of kmalloc caches, at the cost of the memory for those caches. It can only be chosen at build time, so hosts which want it and hosts which would rather not pay for it need separate kernels.
Add a slab_buckets= early parameter, enabled by default on kernels built with CONFIG_SLAB_BUCKETS=y, so that it can be turned off at boot.
--- .../admin-guide/kernel-parameters.txt | 8 ++++++ mm/Kconfig | 4 +++ mm/slab_common.c | 26 +++++++++++++++---- 3 files changed, 33 insertions(+), 5 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +8 -0
@@ -6990,6 +6990,14 @@ Kernel parameters apic=verbose is specified. Example: apic=debug show_lapic=all + slab_buckets= [MM,EARLY] + Format: <bool> + Enable or disable separate kmalloc buckets on kernels + built with CONFIG_SLAB_BUCKETS=y. When disabled, + allocations use regular kmalloc caches. Ignored + without that option. + Default: on. + slab_debug[=options[,slabs][;[options[,slabs]]...]] [MM] Enabling slab_debug allows one to determine the culprit if slab objects become corrupted. Enabling
mm/Kconfig +4 -0
@@ -231,6 +231,10 @@ config SLAB_BUCKETS of extra pages since the bulk of user-controlled allocations are relatively long-lived. + The separate buckets can be disabled at boot with the + "slab_buckets=0" kernel parameter, in which case these + allocations use the regular kmalloc caches. + If unsure, say Y. config SLUB_STATS
mm/slab_common.c +21 -5
@@ -410,6 +410,16 @@ EXPORT_SYMBOL(__kmem_cache_create_args);static struct kmem_cache *kmem_buckets_cache__ro_after_init ; +static bool slab_buckets_enabled__ro_after_init =IS_ENABLED (CONFIG_SLAB_BUCKETS ); + +static int __init setup_slab_buckets (char *str) +{ +if (!IS_ENABLED (CONFIG_SLAB_BUCKETS )) +return 0 ; +return kstrtobool (str, &slab_buckets_enabled); +} +early_param ("slab_buckets" , setup_slab_buckets); +/** * kmem_buckets_create - Create a set of caches that handle dynamic sized * allocations via kmem_buckets_alloc()
@@ -424,10 +434,12 @@ static struct kmem_cache *kmem_buckets_cache __ro_after_init;* * Context: Cannot be called within an interrupt, but can be interrupted. * -* Return: a pointer to the cache on success, NULL on failure. When -* CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned, and -* subsequent calls to kmem_buckets_alloc() will fall back to kmalloc(). -* (i.e. callers only need to check for NULL on failure.) +* Return: a pointer to the cache on success, NULL on failure or when the +* separate buckets were disabled with the "slab_buckets=0" boot parameter. +* When CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned. In all +* of these cases subsequent calls to kmem_buckets_alloc() with the returned +* value fall back to kmalloc(), so callers that can live without separate +* buckets do not need to check the return value. */ kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, unsigned int useroffset,
@@ -448,6 +460,10 @@ kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags,if (!IS_ENABLED (CONFIG_SLAB_BUCKETS ))return ZERO_SIZE_PTR ; +/* NULL makes kmalloc_slab() use the regular kmalloc caches. */ +if (!slab_buckets_enabled) +return NULL ; +if (WARN_ON (!kmem_buckets_cache))return NULL ;
@@ -1059,7 +1075,7 @@ void __init create_kmalloc_caches(void)/* Kmalloc array is now usable */ slab_state =UP ; -if (IS_ENABLED (CONFIG_SLAB_BUCKETS )) +if (IS_ENABLED (CONFIG_SLAB_BUCKETS ) && slab_buckets_enabled) kmem_buckets_cache =kmem_cache_create ("kmalloc_buckets" ,sizeof (kmem_buckets),0 ,SLAB_NO_MERGE ,NULL );
Signature · 1 line
2.56.1
[email protected]
AI reviewRe: [PATCH v3 3/6] mm/slab: Support disabling SLAB_BUCKETS at boot
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] mm/slab: setup_slab_buckets() adds per-feature knob instead of slab_hardening --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] mm/slab: setup_slab_buckets() adds per-feature knob instead of slab_hardening
--
commit d1769f1af5f0bdc819b4474cbb49e0bbed23c8ff
Author: Chris Down <[email protected]>
mm/slab: Support disabling SLAB_BUCKETS at boot
This commit adds a slab_buckets= early boot parameter to allow disabling separate kmalloc buckets on kernels built with CONFIG_SLAB_BUCKETS=y.
Quoted 7 lines
mm/slab_common.c +0
-0@@ -410,6 +410,16 @@ EXPORT_SYMBOL(__kmem_cache_create_args);static struct kmem_cache *kmem_buckets_cache__ro_after_init ;
mm/slab_common.c EXPORT_SYMBOL(__kmem_cache_create_args); · line 421
Quoted 10 lines
mm/slab_common.c +9
-0+static bool slab_buckets_enabled__ro_after_init =IS_ENABLED (CONFIG_SLAB_BUCKETS ); + +static int __init setup_slab_buckets (char *str) +{ +if (!IS_ENABLED (CONFIG_SLAB_BUCKETS )) +return 0 ; +return kstrtobool (str, &slab_buckets_enabled); +} +early_param ("slab_buckets" , setup_slab_buckets);
[Severity: Low]
As reported by Harry Yoo on v1 [1], should boot-time slab hardening controls use a single coarse-grained parameter such as slab_hardening= rather than introducing separate per-feature knobs like slab_buckets= here and kmalloc_partition= in commit "mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot"?
[1] https://lore.kernel.org/r/auIN44CnAyzvR_Ll@thinkstation
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
PATCH v3 4/6 slub_kunit: Add a test for the slab_buckets boot parameter
In reply to Chris DownCheck that memdup_user() allocates from the memdup_user buckets by default, and from the regular kmalloc caches when they're turned off with slab_buckets=0. The test is skipped on kernels built without CONFIG_SLAB_BUCKETS.
Make slab_buckets_enabled global and export it to KUnit so that the test can see it.
--- lib/tests/slub_kunit.c | 40 ++++++++++++++++++++++++++++++++++++++++ mm/slab.h | 2 ++ mm/slab_common.c | 4 +++- 3 files changed, 45 insertions(+), 1 deletion(-)
lib/tests/slub_kunit.c +40 -0
@@ -9,6 +9,8 @@#include <linux/delay.h> #include <linux/perf_event.h> #include <linux/kprobes.h> +#include <linux/mman.h> +#include <linux/string.h> #include "../mm/slab.h" static struct kunit_resource resource;
@@ -296,6 +298,42 @@ static void test_krealloc_redzone_zeroing(struct kunit *test)kmem_cache_destroy (s); } +static void test_slab_buckets (struct kunit *test) +{ +unsigned long user_addr; +const char *name; +void *p; + +if (!IS_ENABLED (CONFIG_SLAB_BUCKETS )) +kunit_skip (test,"requires CONFIG_SLAB_BUCKETS" ); + +if (!slab_buckets_enabled) +KUNIT_EXPECT_NULL (test,kmem_buckets_create ("TestSlub_buckets" , +0 ,0 ,INT_MAX ,NULL )); + +if (!IS_ENABLED (CONFIG_MMU )) +kunit_skip (test,"userspace allocation needs an MMU" ); + + user_addr =kunit_vm_mmap (test,NULL ,0 ,PAGE_SIZE , +PROT_READ |PROT_WRITE , +MAP_ANONYMOUS |MAP_PRIVATE ,0 ); +KUNIT_ASSERT_NE_MSG (test, user_addr,0 ,"Could not create userspace mm" ); +KUNIT_ASSERT_LT_MSG (test, user_addr, (unsigned long )TASK_SIZE , +"Failed to allocate user memory" ); + + p =memdup_user ((void __user *)user_addr,64 ); +KUNIT_ASSERT_NOT_ERR_OR_NULL (test, p); + + name =virt_to_slab (p)->slab_cache->name; +if (slab_buckets_enabled) +KUNIT_EXPECT_TRUE_MSG (test,strstarts (name,"memdup_user-" ), +"allocated from %s" , name); +else +KUNIT_EXPECT_TRUE_MSG (test,strstarts (name,"kmalloc-" ), +"allocated from %s" , name); +kfree (p); +} +#if defined(CONFIG_PERF_EVENTS ) || (defined(CONFIG_KPROBES ) && defined(CONFIG_SMP ))#define NR_ITERATIONS 1000#define NR_OBJECTS 1000
@@ -489,6 +527,7 @@ static struct kunit_case test_cases[] = {KUNIT_CASE (test_kfree_rcu_wq_destroy),KUNIT_CASE (test_leak_destroy),KUNIT_CASE (test_krealloc_redzone_zeroing), +KUNIT_CASE (test_slab_buckets), #ifdef CONFIG_PERF_EVENTS KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_perf), #endif
@@ -505,5 +544,6 @@ static struct kunit_suite test_suite = { };kunit_test_suite (test_suite); +MODULE_IMPORT_NS ("EXPORTED_FOR_KUNIT_TESTING" );MODULE_DESCRIPTION ("Kunit tests for slub allocator" );MODULE_LICENSE ("GPL" );
mm/slab.h +2 -0
@@ -390,6 +390,8 @@ extern struct list_head slab_caches;/* The slab cache that manages slab cache information */ extern struct kmem_cache *kmem_cache; +extern bool slab_buckets_enabled; +/* A table of kmalloc cache names and sizes */ extern const struct kmalloc_info_struct { const char *name[NR_KMALLOC_TYPES];
mm/slab_common.c +3 -1
@@ -29,6 +29,7 @@#include <linux/memcontrol.h> #include <linux/stackdepot.h> #include <trace/events/rcu.h> +#include <kunit/visibility.h> #include "../kernel/rcu/rcu.h" #include "internal.h"
@@ -410,7 +411,8 @@ EXPORT_SYMBOL(__kmem_cache_create_args);static struct kmem_cache *kmem_buckets_cache__ro_after_init ; -static bool slab_buckets_enabled__ro_after_init =IS_ENABLED (CONFIG_SLAB_BUCKETS ); +bool slab_buckets_enabled__ro_after_init =IS_ENABLED (CONFIG_SLAB_BUCKETS ); +EXPORT_SYMBOL_IF_KUNIT (slab_buckets_enabled);static int __init setup_slab_buckets (char *str) {
Signature · 1 line
2.56.1
PATCH v3 5/6 mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
In reply to Chris DownKMALLOC_PARTITION_CACHES creates several copies of each normal kmalloc cache and spreads allocations across them. That costs a fair bit of
memory: Harry measured about 18 MiB more slab memory after boot on a 64 CPU, 64 GiB VM, mostly from under-used slabs [1]. Like SLAB_BUCKETS, it can only be chosen at build time.
Add a kmalloc_partition= early parameter, enabled by default on kernels built with CONFIG_KMALLOC_PARTITION_CACHES=y. When it's off, the copies alias the normal caches rather than being created.
Let's leave kmalloc_type() unchanged since each index simply resolves to the same cache when partitioning is off.
--- .../admin-guide/kernel-parameters.txt | 7 ++++++ mm/Kconfig | 5 ++++ mm/slab_common.c | 23 +++++++++++++++---- 3 files changed, 31 insertions(+), 4 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +7 -0
@@ -3045,6 +3045,13 @@ Kernel parameters Configure the RouterBoard 532 series on-chip Ethernet adapter MAC address. + kmalloc_partition= [MM,EARLY] + Format: <bool> + Enable or disable kmalloc cache partitioning on + kernels built with CONFIG_KMALLOC_PARTITION_CACHES=y. + Ignored without that option. + Default: on. + kmemleak= [KNL,EARLY] Boot-time kmemleak enable/disable Valid arguments: on, off Default: on
mm/Kconfig +5 -0
@@ -266,6 +266,11 @@ config KMALLOC_PARTITION_CACHES limited degree of memory and CPU overhead that relates to hardware and system workload. + The partitioning can be disabled at boot with the + "kmalloc_partition=0" kernel parameter, in which case the copies + are not created and all normal kmalloc allocations use the + regular kmalloc caches. + choice prompt "Partitioned slab cache mode" depends on KMALLOC_PARTITION_CACHES
mm/slab_common.c +19 -4
@@ -781,6 +781,17 @@ unsigned long random_kmalloc_seed __ro_after_init;EXPORT_SYMBOL (random_kmalloc_seed);#endif +static bool kmalloc_partition_enabled__ro_after_init = +IS_ENABLED (CONFIG_KMALLOC_PARTITION_CACHES ); + +static int __init setup_kmalloc_partition (char *str) +{ +if (!IS_ENABLED (CONFIG_KMALLOC_PARTITION_CACHES )) +return 0 ; +return kstrtobool (str, &kmalloc_partition_enabled); +} +early_param ("kmalloc_partition" , setup_kmalloc_partition); +/* * Conversion table for small slabs sizes / 8 to the index in the * kmalloc array. This is necessary for slabs < 192 since we have non power
@@ -1012,10 +1023,14 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type) flags |=SLAB_CACHE_DMA ; } -#ifdef CONFIG_KMALLOC_PARTITION_CACHES -if (type >=KMALLOC_PARTITION_START && type <=KMALLOC_PARTITION_END ) - flags |=SLAB_NO_MERGE ; -#endif +if (kmalloc_partition_enabled) { +if (type >=KMALLOC_PARTITION_START && type <=KMALLOC_PARTITION_END ) + flags |=SLAB_NO_MERGE ; + }else if (type >KMALLOC_PARTITION_START && type <=KMALLOC_PARTITION_END ) { +/* Use the normal caches, which have already been created. */ + kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL ][idx]; +return ; + }/* * If memcg_kmem is enabled and this is a KMALLOC_NORMAL cache and not
Signature · 1 line
2.56.1
kernel test robot
build botRe: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
kernel test robot noticed the following build warnings:
Hi Chris,
kernel test robot noticed the following build warnings:
[auto build test WARNING on ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75]
url: https://github.com/intel-lab-lkp/linux/commits/Chris-Down/mm-slab-Enable-SLAB_FREELIST_HARDENED-by-default/20261113-005125 base: ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75 patch link: https://lore.kernel.org/r/9a4caff0336c78a7adc746786c81c9c01f89c40b.1794506262.git.chris%40chrisdown.name patch subject: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot config: hexagon-allnoconfig (https://download.01.org/0day-ci/archive/20261114/[email protected]/config) compiler: clang version 24.0.0git (https://github.com/llvm/llvm-project 1b1fed44d3dcca649a9877915d57c3357a98124e) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261114/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> mm/slab_common.c:1029:44: warning: non-overlapping comparisons always evaluate to false [-Wtautological-overlap-compare]
1029 | } else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1 warning generated.
vim +1029 mm/slab_common.c
999
1000 static void __init
1001 new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
1002 {
1003 slab_flags_t flags = 0;
1004 unsigned int minalign = __kmalloc_minalign();
1005 unsigned int aligned_size = kmalloc_info[idx].size;
1006 int aligned_idx = idx;
1007
1008 if ((KMALLOC_RECLAIM != KMALLOC_NORMAL) && (type == KMALLOC_RECLAIM)) {
1009 flags |= SLAB_RECLAIM_ACCOUNT;
1010 } else if (IS_ENABLED(CONFIG_MEMCG) && (type == KMALLOC_CGROUP)) {
1011 if (mem_cgroup_kmem_disabled()) {
1012 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1013 return;
1014 }
1015 flags |= SLAB_ACCOUNT;
1016 } else if (IS_ENABLED(CONFIG_SLAB_OBJ_EXT) && type == KMALLOC_NO_OBJ_EXT) {
1017 if (!need_kmalloc_no_objext()) {
1018 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1019 return;
1020 }
1021 flags |= SLAB_NO_OBJ_EXT | SLAB_NO_MERGE;
1022 } else if (IS_ENABLED(CONFIG_ZONE_DMA) && (type == KMALLOC_DMA)) {
1023 flags |= SLAB_CACHE_DMA;
1024 }
1025
1026 if (kmalloc_partition_enabled) {
1027 if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
1028 flags |= SLAB_NO_MERGE;
> 1029 } else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) { 1030 /* Use the normal caches, which have already been created. */
1031 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1032 return;
1033 }
1034
1035 /*
1036 * If memcg_kmem is enabled and this is a KMALLOC_NORMAL cache and not
1037 * aliased with any other type, make sure it's never merged with any other
1038 * cache.
1039 *
1040 * In other cases the kmalloc cache may end up being used for a
1041 * __GFP_ACCOUNT allocation so mark it as such. The exception is a
1042 * KMALLOC_NO_OBJ_EXT cache.
1043 */
1044 if (!mem_cgroup_kmem_disabled()) {
1045 if (type == KMALLOC_NORMAL && KMALLOC_RECLAIM != KMALLOC_NORMAL)
1046 flags |= SLAB_NO_MERGE;
1047 else if (!(flags & SLAB_NO_OBJ_EXT))
1048 flags |= SLAB_MAY_ACCOUNT;
1049 }
1050
1051 if (minalign > ARCH_KMALLOC_MINALIGN) {
1052 aligned_size = ALIGN(aligned_size, minalign);
1053 aligned_idx = __kmalloc_index(aligned_size, false);
1054 }
1055
1056 if (!kmalloc_caches[type][aligned_idx])
1057 kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
1058 kmalloc_info[aligned_idx].name[type],
1059 aligned_size, flags);
1060 if (idx != aligned_idx)
1061 kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
1062 }
1063
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
In reply to Chris Downkernel test robot noticed the following build warnings:
Hi Chris,
kernel test robot noticed the following build warnings:
[auto build test WARNING on ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75]
url: https://github.com/intel-lab-lkp/linux/commits/Chris-Down/mm-slab-Enable-SLAB_FREELIST_HARDENED-by-default/20261113-005125 base: ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75 patch link: https://lore.kernel.org/r/9a4caff0336c78a7adc746786c81c9c01f89c40b.1794506262.git.chris%40chrisdown.name patch subject: [PATCH v3 5/6] mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot config: arm-allnoconfig (https://download.01.org/0day-ci/archive/20261114/[email protected]/config) compiler: clang version 17.0.6 (https://github.com/llvm/llvm-project 6009708b4367171ccdbf4b5905cb6a803753fe18) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261114/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> mm/slab_common.c:1029:44: warning: overlapping comparisons always evaluate to false [-Wtautological-overlap-compare]
1029 | } else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) {
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1 warning generated.
vim +1029 mm/slab_common.c
999
1000 static void __init
1001 new_kmalloc_cache(int idx, enum kmalloc_cache_type type)
1002 {
1003 slab_flags_t flags = 0;
1004 unsigned int minalign = __kmalloc_minalign();
1005 unsigned int aligned_size = kmalloc_info[idx].size;
1006 int aligned_idx = idx;
1007
1008 if ((KMALLOC_RECLAIM != KMALLOC_NORMAL) && (type == KMALLOC_RECLAIM)) {
1009 flags |= SLAB_RECLAIM_ACCOUNT;
1010 } else if (IS_ENABLED(CONFIG_MEMCG) && (type == KMALLOC_CGROUP)) {
1011 if (mem_cgroup_kmem_disabled()) {
1012 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1013 return;
1014 }
1015 flags |= SLAB_ACCOUNT;
1016 } else if (IS_ENABLED(CONFIG_SLAB_OBJ_EXT) && type == KMALLOC_NO_OBJ_EXT) {
1017 if (!need_kmalloc_no_objext()) {
1018 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1019 return;
1020 }
1021 flags |= SLAB_NO_OBJ_EXT | SLAB_NO_MERGE;
1022 } else if (IS_ENABLED(CONFIG_ZONE_DMA) && (type == KMALLOC_DMA)) {
1023 flags |= SLAB_CACHE_DMA;
1024 }
1025
1026 if (kmalloc_partition_enabled) {
1027 if (type >= KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END)
1028 flags |= SLAB_NO_MERGE;
> 1029 } else if (type > KMALLOC_PARTITION_START && type <= KMALLOC_PARTITION_END) { 1030 /* Use the normal caches, which have already been created. */
1031 kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL][idx];
1032 return;
1033 }
1034
1035 /*
1036 * If memcg_kmem is enabled and this is a KMALLOC_NORMAL cache and not
1037 * aliased with any other type, make sure it's never merged with any other
1038 * cache.
1039 *
1040 * In other cases the kmalloc cache may end up being used for a
1041 * __GFP_ACCOUNT allocation so mark it as such. The exception is a
1042 * KMALLOC_NO_OBJ_EXT cache.
1043 */
1044 if (!mem_cgroup_kmem_disabled()) {
1045 if (type == KMALLOC_NORMAL && KMALLOC_RECLAIM != KMALLOC_NORMAL)
1046 flags |= SLAB_NO_MERGE;
1047 else if (!(flags & SLAB_NO_OBJ_EXT))
1048 flags |= SLAB_MAY_ACCOUNT;
1049 }
1050
1051 if (minalign > ARCH_KMALLOC_MINALIGN) {
1052 aligned_size = ALIGN(aligned_size, minalign);
1053 aligned_idx = __kmalloc_index(aligned_size, false);
1054 }
1055
1056 if (!kmalloc_caches[type][aligned_idx])
1057 kmalloc_caches[type][aligned_idx] = create_kmalloc_cache(
1058 kmalloc_info[aligned_idx].name[type],
1059 aligned_size, flags);
1060 if (idx != aligned_idx)
1061 kmalloc_caches[type][idx] = kmalloc_caches[type][aligned_idx];
1062 }
1063
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
PATCH v3 6/6 slub_kunit: Add a test for the kmalloc_partition boot parameter
In reply to Chris DownCheck that the kmalloc partitions are separate caches by default, and alias the normal kmalloc caches when they're turned off with kmalloc_partition=0. The test is skipped on kernels built without CONFIG_KMALLOC_PARTITION_CACHES.
Make kmalloc_partition_enabled global and export it to KUnit so that the test can see it.
--- lib/tests/slub_kunit.c | 45 ++++++++++++++++++++++++++++++++++++++++++ mm/slab.h | 1 + mm/slab_common.c | 3 ++- 3 files changed, 48 insertions(+), 1 deletion(-)
lib/tests/slub_kunit.c +45 -0
@@ -334,6 +334,50 @@ static void test_slab_buckets(struct kunit *test)kfree (p); } +/* +* Each partition copy is a separate cache, unless the copies were disabled +* with kmalloc_partition=0: then they alias the KMALLOC_NORMAL caches. +*/ +static void test_kmalloc_partition (struct kunit *test) +{ +enum kmalloc_cache_type type; +struct kmem_cache *s; +unsigned int idx; +bool found; +void *p; + +if (!IS_ENABLED (CONFIG_KMALLOC_PARTITION_CACHES )) +kunit_skip (test,"requires CONFIG_KMALLOC_PARTITION_CACHES" ); + +for (type =KMALLOC_PARTITION_START +1 ; type <=KMALLOC_PARTITION_END ; type++) { +for (idx =0 ; idx <ARRAY_SIZE (kmalloc_caches[type]); idx++) { +struct kmem_cache *normal = kmalloc_caches[KMALLOC_NORMAL ][idx]; +struct kmem_cache *part = kmalloc_caches[type][idx]; + +if (!kmalloc_partition_enabled) { +KUNIT_EXPECT_PTR_EQ (test, part, normal); + }else if (normal) { +KUNIT_EXPECT_NOT_NULL (test, part); +KUNIT_EXPECT_PTR_NE (test, part, normal); + } + } + } + + p =kmalloc (64 ,GFP_KERNEL ); +KUNIT_ASSERT_NOT_NULL (test, p); + s =virt_to_slab (p)->slab_cache; + idx =kmalloc_index (64 ); +if (!kmalloc_partition_enabled) { +KUNIT_EXPECT_PTR_EQ (test, s, kmalloc_caches[KMALLOC_NORMAL ][idx]); + }else { + found = false; +for (type =KMALLOC_PARTITION_START ; type <=KMALLOC_PARTITION_END ; type++) + found |= s == kmalloc_caches[type][idx]; +KUNIT_EXPECT_TRUE_MSG (test, found,"allocated from %s" , s->name); + } +kfree (p); +} +#if defined(CONFIG_PERF_EVENTS ) || (defined(CONFIG_KPROBES ) && defined(CONFIG_SMP ))#define NR_ITERATIONS 1000#define NR_OBJECTS 1000
@@ -528,6 +572,7 @@ static struct kunit_case test_cases[] = {KUNIT_CASE (test_leak_destroy),KUNIT_CASE (test_krealloc_redzone_zeroing),KUNIT_CASE (test_slab_buckets), +KUNIT_CASE (test_kmalloc_partition), #ifdef CONFIG_PERF_EVENTS KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_perf), #endif
mm/slab.h +1 -0
@@ -391,6 +391,7 @@ extern struct list_head slab_caches;extern struct kmem_cache *kmem_cache;extern bool slab_buckets_enabled; +extern bool kmalloc_partition_enabled;/* A table of kmalloc cache names and sizes */ extern const struct kmalloc_info_struct {
mm/slab_common.c +2 -1
@@ -781,8 +781,9 @@ unsigned long random_kmalloc_seed __ro_after_init;EXPORT_SYMBOL (random_kmalloc_seed);#endif -static bool kmalloc_partition_enabled__ro_after_init = +bool kmalloc_partition_enabled__ro_after_init =IS_ENABLED (CONFIG_KMALLOC_PARTITION_CACHES ); +EXPORT_SYMBOL_IF_KUNIT (kmalloc_partition_enabled);static int __init setup_kmalloc_partition (char *str) {
Signature · 1 line
2.56.1
Quoted 14 lines · Chris Down
Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. At Meta, for example, some hosts are built with these options and the rest of the fleet isn't, and that's one of the things that keeps those kernels apart.
This series does two things:
-
SLAB_FREELIST_HARDENEDandSLAB_FREELIST_RANDOMare cheap enough with sheaves that it makes more sense to just enable them by default than to make them switchable. With both on, system level benchmarks didn't change measurably, only slowpath heavy microbenchmarks did [1].
Quoted 7 lines
-
SLAB_BUCKETSandKMALLOC_PARTITION_CACHEScost memory (Harry measured about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so they get slab_buckets= and kmalloc_partition= boot parameters, which default to on when the option is built in.Whether those two should be folded into a single slab_hardening= is still open [3].
A coarse slab_hardening= option can easily be replaced by documentation. While less knobs is better, an intransparent coarse variant of slab_hardening= just makes the admin wonder "what does that actually to, and do I want it?". They still have to go and look up what it actually does, and at that point it doesn't add much value (alternative suggestion below).
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
light: "I'm willying to sacrifice _some_ memory and CPU for security"
full: "I'm willying to sacrifice more overhead for security
Who will decide what is light and full? I think there just won't be any consensus reached.
One thing that users may want is an "all off" option, similar to mitigations=off, should they wish to get max performance back with a single option. So you could consider slab_hardening=off. But at that point it may be more consistent to drop slab_buckets= and kmalloc_partition= in favor of arguments to slab_hardening=, such as slab_hardening=nobuckets,partition. This would mirror mitigations=, and should cater to the usecase of slab_hardening=off to get max performance with a single option.
But I would not add options such as 'light', 'full'.
Thanks,
-- Marco
Harry Yoo
maintainerQuoted 30 lines · Marco Elver
Quoted 14 lines · Chris Down
Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. At Meta, for example, some hosts are built with these options and the rest of the fleet isn't, and that's one of the things that keeps those kernels apart.
This series does two things:
-
SLAB_FREELIST_HARDENEDandSLAB_FREELIST_RANDOMare cheap enough with sheaves that it makes more sense to just enable them by default than to make them switchable. With both on, system level benchmarks didn't change measurably, only slowpath heavy microbenchmarks did [1].Quoted 7 lines
-
SLAB_BUCKETSandKMALLOC_PARTITION_CACHEScost memory (Harry measured about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so they get slab_buckets= and kmalloc_partition= boot parameters, which default to on when the option is built in.Whether those two should be folded into a single slab_hardening= is still open [3].
A coarse slab_hardening= option can easily be replaced by documentation. While less knobs is better, an intransparent coarse variant of slab_hardening= just makes the admin wonder "what does that actually to, and do I want it?". They still have to go and look up what it actually does, and at that point it doesn't add much value (alternative suggestion below).
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
light: "I'm willying to sacrifice _some_ memory and CPU for security"
full: "I'm willying to sacrifice more overhead for securityWho will decide what is light and full? I think there just won't be any consensus reached.
You're probably right.
Quoted 3 lines
One thing that users may want is an "all off" option, similar to mitigations=off, should they wish to get max performance back with a single option. So you could consider slab_hardening=off.
slab_hardening=off sounds good to me.
However this assumes "all on" is the right thing to do by default, but not enabling slab hardening is not as nearly critical as not enabling mitigations to CPU vulnerabilities, no?
I'd rather mirror hardened_usercopy= instead of mitigations=off.
slab_hardening=on (turn on all slab hardening features) slab_hardening=off (turn off all slab hardening features) (Otherwise it is determined by CONFIG_*.)
Quoted 5 lines
But at that point it may be more consistent to drop slab_buckets= and kmalloc_partition= in favor of arguments to slab_hardening=, such as slab_hardening=nobuckets,partition. This would mirror mitigations=, and should cater to the usecase of slab_hardening=off to get max performance with a single option.
I hope we don't reach this point, but more fine-grained control should be achivable if it is indeed required.
Quoted 1 line
But I would not add options such as 'light', 'full'.
Fair enough.
Quoted 2 lines
Thanks,
-- Marco
Signature · 2 lines
Cheers, Harry / Hyeonggon
Harry Yoo
maintainerQuoted 49 lines · Harry Yoo
Quoted 30 lines · Marco Elver
Quoted 14 lines · Chris Down
Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. At Meta, for example, some hosts are built with these options and the rest of the fleet isn't, and that's one of the things that keeps those kernels apart.
This series does two things:
-
SLAB_FREELIST_HARDENEDandSLAB_FREELIST_RANDOMare cheap enough with sheaves that it makes more sense to just enable them by default than to make them switchable. With both on, system level benchmarks didn't change measurably, only slowpath heavy microbenchmarks did [1].Quoted 7 lines
-
SLAB_BUCKETSandKMALLOC_PARTITION_CACHEScost memory (Harry measured about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so they get slab_buckets= and kmalloc_partition= boot parameters, which default to on when the option is built in.Whether those two should be folded into a single slab_hardening= is still open [3].
A coarse slab_hardening= option can easily be replaced by documentation. While less knobs is better, an intransparent coarse variant of slab_hardening= just makes the admin wonder "what does that actually to, and do I want it?". They still have to go and look up what it actually does, and at that point it doesn't add much value (alternative suggestion below).
Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
light: "I'm willying to sacrifice _some_ memory and CPU for security"
full: "I'm willying to sacrifice more overhead for securityWho will decide what is light and full? I think there just won't be any consensus reached.
You're probably right.
Quoted 3 lines
One thing that users may want is an "all off" option, similar to mitigations=off, should they wish to get max performance back with a single option. So you could consider slab_hardening=off.
slab_hardening=off sounds good to me.
However this assumes "all on" is the right thing to do by default, but not enabling slab hardening is not as nearly critical as not enabling mitigations to CPU vulnerabilities, no?
Quoted 5 lines
I'd rather mirror hardened_usercopy= instead of mitigations=off.
slab_hardening=on (turn on all slab hardening features) slab_hardening=off (turn off all slab hardening features) (Otherwise it is determined by CONFIG_*.)
Should have been CONFIG_*_DEFAULT_ON.
Quoted 19 lines
Quoted 5 lines
But at that point it may be more consistent to drop slab_buckets= and kmalloc_partition= in favor of arguments to slab_hardening=, such as slab_hardening=nobuckets,partition. This would mirror mitigations=, and should cater to the usecase of slab_hardening=off to get max performance with a single option.
I hope we don't reach this point, but more fine-grained control should be achivable if it is indeed required.
Quoted 1 line
But I would not add options such as 'light', 'full'.
Fair enough.
Quoted 2 lines
Thanks,
-- MarcoSignature · 2 lines
Cheers, Harry / Hyeonggon
Signature · 2 lines
Cheers, Harry / Hyeonggon
Vlastimil Babka (SUSE)
maintainerQuoted 13 lines · Chris Down
Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. At Meta, for example, some hosts are built with these options and the rest of the fleet isn't, and that's one of the things that keeps those kernels apart.
This series does two things:
-
SLAB_FREELIST_HARDENEDandSLAB_FREELIST_RANDOMare cheap enough with sheaves that it makes more sense to just enable them by default than to make them switchable. With both on, system level benchmarks didn't change measurably, only slowpath heavy microbenchmarks did [1].
Quoted 4 lines
-
SLAB_BUCKETSandKMALLOC_PARTITION_CACHEScost memory (Harry measured about 1 MiB and 18 MiB respectively after boot on a 64 CPU VM [2]), so they get slab_buckets= and kmalloc_partition= boot parameters, which default to on when the option is built in.
I'm not thrilled about the "default to on". We usually do these features opt-in. Now our distro kernel would likely need to have these "disable this" options in the default command line, with documentation to remove it for a hardened setup.
But yeah doing it opt-in, the hardened setups would now have to start passing the extra params to enable this, even if until now enabling the config option was enough. Also not great, but we can argue they are still the minority?
Of course there's always the possibility of adding another _DEFAUT_ON configs. Not great either. Sigh.
Quoted 52 lines
Whether those two should be folded into a single slab_hardening= is still open [3].
v3:
- Rebase onto slab/for-next.
- Always build theslab_bucketsandkmalloc_partitiontests and skip them at runtime when the option isn't built in, instead of leaving them out with #ifdef (Kees).
- Declareslab_buckets_enabledandkmalloc_partition_enabledinmm/slab.hwithout any #if (Kees).
- Drop the remaining #ifdefs around slab_buckets= and kmalloc_partition=. Both are now always registered and ignored when the option isn't built in (Kees).
- Keep both flags as plain__ro_after_initbools rather than static keys, since they're only read when caches are created and not on the allocation path.v2:
- Make
SLAB_FREELIST_HARDENEDandSLAB_FREELIST_RANDOMdefault y instead of adding boot parameters for them (Vlastimil).
- Skip thememdup_user()part of theslab_bucketstest on !MMU (Sashiko).
- Keep slab_buckets= and kmalloc_partition= separate for now. We can discuss whether to merge them into slab_hardening= or similar, as Harry suggested.v2: https://lore.kernel.org/r/[email protected]
v1: https://lore.kernel.org/r/[email protected][1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]
[3] https://lore.kernel.org/r/aud4gPOUzvHtLYnJ@thinkstationChris Down (6):
mm/slab: EnableSLAB_FREELIST_HARDENEDby default mm/slab: EnableSLAB_FREELIST_RANDOMby default mm/slab: Support disablingSLAB_BUCKETSat boot
slub_kunit: Add a test for theslab_bucketsboot parameter mm/slab: Support disablingKMALLOC_PARTITION_CACHESat boot
slub_kunit: Add a test for thekmalloc_partitionboot parameter.../admin-guide/kernel-parameters.txt | 15 ++++ lib/tests/slub_kunit.c | 85 +++++++++++++++++++ mm/Kconfig | 11 +++ mm/slab.h | 3 + mm/slab_common.c | 52 ++++++++++-- 5 files changed, 157 insertions(+), 9 deletions(-)base-commit:
ce6518e61e3b4bf4f97f7ac6204049b9eba7eb75