mm/slab: Support selecting slab hardening features at boot
Subsystems (5)
cover mm/slab: Support selecting slab hardening features at boot
Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. Let's allow those hosts to use one image and select each policy at boot instead.
This series adds four new boot parameters that default to their values from CONFIG_*:
slab_freelist_hardened=
slab_freelist_random=
slab_buckets=
kmalloc_partition=
These mostly act at cache initialisation time. For free list hardening we use a static key settled before the first free pointer is written.
Chris Down (8):
mm/slub: Support disabling freelist hardening at boot
slub_kunit: Add a test for slab_freelist_hardened= mm/slub: Support disabling freelist randomisation at boot
slub_kunit: Add a test for slab_freelist_random= mm/slab: Support disabling SLAB_BUCKETS at boot
slub_kunit: Add a test for the slab_buckets boot parameter mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
slub_kunit: Add a test for the kmalloc_partition boot parameter
.../admin-guide/kernel-parameters.txt | 29 +++ lib/tests/slub_kunit.c | 186 ++++++++++++++++++ mm/Kconfig | 15 ++ mm/slab.h | 24 +++ mm/slab_common.c | 48 ++++- mm/slub.c | 68 ++++++- scripts/gdb/linux/slab.py | 8 +- 7 files changed, 361 insertions(+), 17 deletions(-)
base-commit: d15d7703166daf7bd62c844a3cb1b3a6279adb4c
Signature · 1 line
2.54.0
PATCH 1/8 mm/slub: Support disabling freelist hardening at boot
Add the slab_freelist_hardened= early parameter, enabled by default on kernels built with CONFIG_SLAB_FREELIST_HARDENED=y.
--- .../admin-guide/kernel-parameters.txt | 7 +++ mm/Kconfig | 3 ++ mm/slab.h | 13 +++++ mm/slub.c | 49 +++++++++++++++---- scripts/gdb/linux/slab.py | 8 ++- 5 files changed, 69 insertions(+), 11 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +7 -0
@@ -7014,6 +7014,13 @@ Kernel parameters option which can be undone by adding the "hash_pointers=always" option. + slab_freelist_hardened= [MM,EARLY] + Format: <bool> + Enable or disable slab freelist hardening on kernels + built with CONFIG_SLAB_FREELIST_HARDENED=y. Ignored + without that option. + Default: on. + slab_max_order= [MM] Determines the maximum allowed order for slabs. A high setting may cause OOMs due to memory
mm/Kconfig +3 -0
@@ -214,6 +214,9 @@ config SLAB_FREELIST_HARDENED sacrifices to harden the kernel slab allocator against common freelist exploit methods. + The hardening is enabled by default and can be disabled at boot + with "slab_freelist_hardened=0". + config SLAB_BUCKETS bool "Support allocation from separate kmalloc buckets" depends on !SLUB_TINY
mm/slab.h +13 -0
@@ -540,6 +540,19 @@ static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t flareturn false; } +#ifdef CONFIG_SLAB_FREELIST_HARDENED +DECLARE_STATIC_KEY_TRUE (slab_freelist_hardened); +#endif + +static inline bool slab_freelist_hardened_enabled (void ) +{ +#ifdef CONFIG_SLAB_FREELIST_HARDENED +return static_branch_likely (&slab_freelist_hardened); +#else +return false; +#endif +} +#if IS_ENABLED (CONFIG_SLUB_DEBUG ) &&IS_ENABLED (CONFIG_KUNIT )bool slab_in_kunit_test (void );#else
mm/slub.c +39 -10
87 lines not on this page · raw message
scripts/gdb/linux/slab.py +7 -1
@@ -96,8 +96,14 @@ def swab64(x): ((x & 0xff00000000000000) >> 56)) return result +def freelist_hardened(): + if not constants.LX_CONFIG_SLAB_FREELIST_HARDENED: + return False + key = gdb.parse_and_eval("slab_freelist_hardened") + return int(key['key']['enabled']['counter']) > 0 + def freelist_ptr_decode(cache, ptr, ptr_addr): - if constants.LX_CONFIG_SLAB_FREELIST_HARDENED: + if freelist_hardened(): return ptr['v'] ^ cache['random'] ^ swab64(int(ptr_addr)) else: return ptr['v']
Signature · 1 line
2.54.0
PATCH 2/8 slub_kunit: Add a test for slab_freelist_hardened=
In reply to Chris Down--- lib/tests/slub_kunit.c | 52 ++++++++++++++++++++++++++++++++++++++++++ mm/slub.c | 2 ++ 2 files changed, 54 insertions(+)
lib/tests/slub_kunit.c +52 -0
@@ -9,6 +9,8 @@#include <linux/delay.h> #include <linux/perf_event.h> #include <linux/kprobes.h> +#include <linux/string_choices.h> +#include <linux/swab.h> #include "../mm/slab.h" static struct kunit_resource resource;
@@ -107,6 +109,54 @@ static void test_first_word(struct kunit *test)kmem_cache_destroy (s); } +static void test_freelist_hardened (struct kunit *test) +{ +struct kmem_cache *s =test_kmem_cache_create ("TestSlub_freelist_hardened" , +64 ,SLAB_CONSISTENCY_CHECKS ); +bool hardened =slab_freelist_hardened_enabled (); +unsigned long ptr_addr, raw; +void *a, *b, *c; + + a =kmem_cache_alloc (s,GFP_KERNEL ); + b =kmem_cache_alloc (s,GFP_KERNEL ); + c =kmem_cache_alloc (s,GFP_KERNEL ); +KUNIT_ASSERT_NOT_NULL (test, a); +KUNIT_ASSERT_NOT_NULL (test, b); +KUNIT_ASSERT_NOT_NULL (test, c); + +if (virt_to_slab (a) !=virt_to_slab (b) || +virt_to_slab (b) !=virt_to_slab (c)) { +kmem_cache_free (s, a); +kmem_cache_free (s, b); +kmem_cache_free (s, c); +kmem_cache_destroy (s); +kunit_skip (test,"objects were allocated from different slabs" ); + } + +/* +* Caches with debug flags have no sheaves, so a and b are pushed onto +* the slab freelist in that order. +*/ +kmem_cache_free (s, a); +kmem_cache_free (s, b); + + ptr_addr = (unsigned long )b + s->offset; + raw = *(unsigned long *)ptr_addr; + +kunit_info (test,"freelist hardening is %s\n" ,str_on_off (hardened)); +KUNIT_EXPECT_EQ (test, raw == (unsigned long )a, !hardened); +#ifdef CONFIG_SLAB_FREELIST_HARDENED +if (hardened) +KUNIT_EXPECT_EQ (test, raw, + (unsigned long )a ^ s->random ^swab (ptr_addr)); +#endif + +kmem_cache_free (s, c); +validate_slab_cache (s); +KUNIT_EXPECT_EQ (test,0 , slab_errors); +kmem_cache_destroy (s); +} +static void test_clobber_50th_byte (struct kunit *test) { struct kmem_cache *s = test_kmem_cache_create("TestSlub_50th_word_free",
@@ -480,6 +530,7 @@ static struct kunit_case test_cases[] = {#ifndef CONFIG_KASAN KUNIT_CASE (test_next_pointer),KUNIT_CASE (test_first_word), +KUNIT_CASE (test_freelist_hardened), KUNIT_CASE(test_clobber_50th_byte), #endif
@@ -505,5 +556,6 @@ static struct kunit_suite test_suite = { };kunit_test_suite (test_suite); +MODULE_IMPORT_NS ("EXPORTED_FOR_KUNIT_TESTING" );MODULE_DESCRIPTION ("Kunit tests for slub allocator" );MODULE_LICENSE ("GPL" );
mm/slub.c +2 -0
@@ -46,6 +46,7 @@#include <linux/prandom.h> #include <kunit/test.h> #include <kunit/test-bug.h> +#include <kunit/visibility.h> #include <linux/sort.h> #include <linux/irq_work.h> #include <linux/kprobes.h>
@@ -220,6 +221,7 @@ static DEFINE_STATIC_KEY_FALSE(strict_numa);* so it must not change once the first slab has been populated. */ DEFINE_STATIC_KEY_TRUE_RO (slab_freelist_hardened); +EXPORT_SYMBOL_IF_KUNIT (slab_freelist_hardened);static bool slab_freelist_hardened_early__initdata = true;static int __init setup_slab_freelist_hardened (char *str)
Signature · 1 line
2.54.0
PATCH 3/8 mm/slub: Support disabling freelist randomisation at boot
In reply to Chris DownAdd a slab_freelist_random= early parameter, enabled by default on kernels built with CONFIG_SLAB_FREELIST_RANDOM=y.
--- Documentation/admin-guide/kernel-parameters.txt | 7 +++++++ mm/Kconfig | 3 +++ mm/slub.c | 16 ++++++++++++++++ 3 files changed, 26 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +7 -0
@@ -7021,6 +7021,13 @@ Kernel parameters without that option. Default: on. + slab_freelist_random= [MM,EARLY] + Format: <bool> + Enable or disable slab freelist randomisation on + kernels built with CONFIG_SLAB_FREELIST_RANDOM=y. + Ignored without that option. + Default: on. + slab_max_order= [MM] Determines the maximum allowed order for slabs. A high setting may cause OOMs due to memory
mm/Kconfig +3 -0
@@ -205,6 +205,9 @@ config SLAB_FREELIST_RANDOM security feature reduces the predictability of the kernel slab allocator against heap overflows. + The randomisation is enabled by default and can be disabled at + boot with "slab_freelist_random=0". + config SLAB_FREELIST_HARDENED bool "Harden slab freelist metadata" depends on !SLUB_TINY
mm/slub.c +16 -0
@@ -3424,12 +3424,23 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node, }#ifdef CONFIG_SLAB_FREELIST_RANDOM +static bool slab_freelist_random__ro_after_init = true; + +static int __init setup_slab_freelist_random (char *str) +{ +return kstrtobool (str, &slab_freelist_random); +} +early_param ("slab_freelist_random" , setup_slab_freelist_random); +/* Pre-initialize the random sequence cache */ static int init_cache_random_seq (struct kmem_cache *s) {unsigned int count =oo_objects (s->oo);int err; +if (!slab_freelist_random) +return 0 ; +/* Bailout if already initialised */ if (s->random_seq)return 0 ;
@@ -3456,6 +3467,11 @@ static void __init init_freelist_randomization(void) {struct kmem_cache *s; +if (!slab_freelist_random) { +pr_info ("SLUB: freelist randomisation disabled\n" ); +return ; + } +mutex_lock (&slab_mutex);list_for_each_entry (s, &slab_caches, list)
Signature · 1 line
2.54.0
PATCH 4/8 slub_kunit: Add a test for slab_freelist_random=
In reply to Chris Down--- lib/tests/slub_kunit.c | 51 ++++++++++++++++++++++++++++++++++++++++++ mm/slab.h | 3 +++ mm/slub.c | 3 ++- 3 files changed, 56 insertions(+), 1 deletion(-)
lib/tests/slub_kunit.c +51 -0
@@ -190,6 +190,56 @@ static void test_clobber_redzone_free(struct kunit *test)kmem_cache_destroy (s); } +static void test_freelist_random (struct kunit *test) +{ +struct kmem_cache *s =test_kmem_cache_create ("TestSlub_freelist_random" , +64 ,SLAB_CONSISTENCY_CHECKS ); +bool random = false; +unsigned int i, nr, nr_linear =0 ; +void **objs, *first; + +#ifdef CONFIG_SLAB_FREELIST_RANDOM + random = slab_freelist_random; +KUNIT_EXPECT_EQ (test, !!s->random_seq, random); +#endif +kunit_info (test,"freelist randomisation is %s\n" ,str_on_off (random)); + +/* +* Caches with debug flags have no sheaves, so objects come straight +* off the slab freelist. +*/ + first =kmem_cache_alloc (s,GFP_KERNEL ); +KUNIT_ASSERT_NOT_NULL (test, first); + nr =virt_to_slab (first)->objects; + objs =kunit_kcalloc (test, nr,sizeof (*objs),GFP_KERNEL ); +KUNIT_ASSERT_NOT_NULL (test, objs); + objs[0 ] = first; + +for (i =1 ; i < nr; i++) { + objs[i] =kmem_cache_alloc (s,GFP_KERNEL ); +KUNIT_ASSERT_NOT_NULL (test, objs[i]); + } + +for (i =1 ; i < nr; i++) { +if (virt_to_slab (objs[i]) !=virt_to_slab (objs[0 ])) +break ; +if (objs[i] == objs[i -1 ] + s->size) + nr_linear++; + } + +if (i == nr) { +/* Without randomisation, objects are handed out in address order. */ +KUNIT_EXPECT_EQ (test, nr_linear == nr -1 , !random); + }else { +kunit_info (test,"objects were allocated from different slabs\n" ); + } + +for (i =0 ; i < nr; i++) +kmem_cache_free (s, objs[i]); +kmem_cache_destroy (s); +KUNIT_EXPECT_EQ (test,0 , slab_errors); +} +static void test_kmalloc_redzone_access (struct kunit *test) { struct kmem_cache *s = test_kmem_cache_create("TestSlub_RZ_kmalloc", 32,
@@ -535,6 +585,7 @@ static struct kunit_case test_cases[] = {#endif KUNIT_CASE (test_clobber_redzone_free), +KUNIT_CASE (test_freelist_random),KUNIT_CASE (test_kmalloc_redzone_access),KUNIT_CASE (test_kfree_rcu), KUNIT_CASE(test_kfree_rcu_wq_destroy),
mm/slab.h +3 -0
@@ -887,6 +887,9 @@ static inline void dump_unreclaimable_slab(void)void ___cache_free (struct kmem_cache *cache,void *x,unsigned long addr);#ifdef CONFIG_SLAB_FREELIST_RANDOM +#if IS_ENABLED (CONFIG_KUNIT ) +extern bool slab_freelist_random; +#endif int cache_random_seq_create (struct kmem_cache *cachep,unsigned int count,gfp_t gfp);void cache_random_seq_destroy (struct kmem_cache *cachep);
mm/slub.c +2 -1
@@ -3424,7 +3424,8 @@ static inline struct slab *alloc_slab_page(gfp_t flags, int node, }#ifdef CONFIG_SLAB_FREELIST_RANDOM -static bool slab_freelist_random__ro_after_init = true; +VISIBLE_IF_KUNIT bool slab_freelist_random __ro_after_init = true; +EXPORT_SYMBOL_IF_KUNIT (slab_freelist_random);static int __init setup_slab_freelist_random (char *str) {
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH 4/8] slub_kunit: Add a test for slab_freelist_random=
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] slub_kunit: missing kasan_reset_tag() in test_freelist_random() --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] slub_kunit: missing kasan_reset_tag() in test_freelist_random() --
commit a6a4a9e9d2827c11abe21a80276d4c98393a0acc
Author: Chris Down <[email protected]>
slub_kunit: Add a test for slab_freelist_random=
This commit exposes slab_freelist_random to KUnit and adds a test case in slub_kunit to verify whether slab objects are allocated in linear address order or randomized order based on slab_freelist_random.
Quoted 39 lines
lib/tests/slub_kunit.c +31
-0@@ -190,6 +190,56 @@ static void test_clobber_redzone_free(struct kunit *test)kmem_cache_destroy (s); } +static void test_freelist_random (struct kunit *test) +{ +struct kmem_cache *s =test_kmem_cache_create ("TestSlub_freelist_random" , +64 ,SLAB_CONSISTENCY_CHECKS ); +bool random = false; +unsigned int i, nr, nr_linear =0 ; +void **objs, *first; + +#ifdef CONFIG_SLAB_FREELIST_RANDOM + random = slab_freelist_random; +KUNIT_EXPECT_EQ (test, !!s->random_seq, random); +#endif +kunit_info (test,"freelist randomisation is %s\n" ,str_on_off (random)); + +/* +* Caches with debug flags have no sheaves, so objects come straight +* off the slab freelist. +*/ + first =kmem_cache_alloc (s,GFP_KERNEL ); +KUNIT_ASSERT_NOT_NULL (test, first); + nr =virt_to_slab (first)->objects; + objs =kunit_kcalloc (test, nr,sizeof (*objs),GFP_KERNEL ); +KUNIT_ASSERT_NOT_NULL (test, objs); + objs[0 ] = first; + +for (i =1 ; i < nr; i++) { + objs[i] =kmem_cache_alloc (s,GFP_KERNEL ); +KUNIT_ASSERT_NOT_NULL (test, objs[i]); + } + +for (i =1 ; i < nr; i++) {
lib/tests/slub_kunit.c static void test_clobber_redzone_free(struct kunit *test) · line 233
Quoted 10 lines
lib/tests/slub_kunit.c +10
-0+if (virt_to_slab (objs[i]) !=virt_to_slab (objs[0 ])) +break ; +if (objs[i] == objs[i -1 ] + s->size) + nr_linear++; + } + +if (i == nr) { +/* Without randomisation, objects are handed out in address order. */ +KUNIT_EXPECT_EQ (test, nr_linear == nr -1 , !random); + }else {
[Severity: Medium]
Can this pointer comparison fail under tag-based KASAN modes (CONFIG_KASAN_SW_TAGS or CONFIG_KASAN_HW_TAGS) when freelist randomization is disabled?
Because test_freelist_random is registered in test_cases[] outside the #ifndef CONFIG_KASAN block, it runs under tag-based KASAN configurations. The cache s is created without a constructor or SLAB_TYPESAFE_BY_RCU, so each kmem_cache_alloc() call passes through:
slab_post_alloc_hook()
-> kasan_slab_alloc()
-> __kasan_slab_alloc()
-> assign_tag()
-> set_tag()which embeds a random tag in the top byte of each returned pointer in objs[].
While virt_to_slab() strips the tag via virt_to_page() so the loop reaches i == nr, the direct comparison:
if (objs[i] == objs[i - 1] + s->size)
compares the tagged pointers without calling kasan_reset_tag(). When consecutive allocations receive different tags, nr_linear will not reach nr - 1 even though the untagged object addresses are contiguous, causing KUNIT_EXPECT_EQ(test, nr_linear == nr - 1, !random) to fail when !random.
Should objs[i] and objs[i - 1] be untagged with kasan_reset_tag() before comparing their addresses?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
PATCH 5/8 mm/slab: Support disabling SLAB_BUCKETS at boot
In reply to Chris DownAdd a slab_buckets= early parameter, enabled by default on kernels built with CONFIG_SLAB_BUCKETS=y.
--- .../admin-guide/kernel-parameters.txt | 8 ++++++ mm/Kconfig | 4 +++ mm/slab_common.c | 26 +++++++++++++++---- 3 files changed, 33 insertions(+), 5 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +8 -0
@@ -7001,6 +7001,14 @@ Kernel parameters apic=verbose is specified. Example: apic=debug show_lapic=all + slab_buckets= [MM,EARLY] + Format: <bool> + Enable or disable separate kmalloc buckets on kernels + built with CONFIG_SLAB_BUCKETS=y. When disabled, + allocations use regular kmalloc caches. Ignored + without that option. + Default: on. + slab_debug[=options[,slabs][;[options[,slabs]]...]] [MM] Enabling slab_debug allows one to determine the culprit if slab objects become corrupted. Enabling
mm/Kconfig +4 -0
@@ -235,6 +235,10 @@ config SLAB_BUCKETS of extra pages since the bulk of user-controlled allocations are relatively long-lived. + The separate buckets can be disabled at boot with the + "slab_buckets=0" kernel parameter, in which case these + allocations use the regular kmalloc caches. + If unsure, say Y. config SLUB_STATS
mm/slab_common.c +21 -5
@@ -410,6 +410,16 @@ EXPORT_SYMBOL(__kmem_cache_create_args);static struct kmem_cache *kmem_buckets_cache__ro_after_init ; +static bool slab_buckets_enabled__ro_after_init =IS_ENABLED (CONFIG_SLAB_BUCKETS ); + +#ifdef CONFIG_SLAB_BUCKETS +static int __init setup_slab_buckets (char *str) +{ +return kstrtobool (str, &slab_buckets_enabled); +} +early_param ("slab_buckets" , setup_slab_buckets); +#endif +/** * kmem_buckets_create - Create a set of caches that handle dynamic sized * allocations via kmem_buckets_alloc()
@@ -424,10 +434,12 @@ static struct kmem_cache *kmem_buckets_cache __ro_after_init;* * Context: Cannot be called within an interrupt, but can be interrupted. * -* Return: a pointer to the cache on success, NULL on failure. When -* CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned, and -* subsequent calls to kmem_buckets_alloc() will fall back to kmalloc(). -* (i.e. callers only need to check for NULL on failure.) +* Return: a pointer to the cache on success, NULL on failure or when the +* separate buckets were disabled with the "slab_buckets=0" boot parameter. +* When CONFIG_SLAB_BUCKETS is not enabled, ZERO_SIZE_PTR is returned. In all +* of these cases subsequent calls to kmem_buckets_alloc() with the returned +* value fall back to kmalloc(), so callers that can live without separate +* buckets do not need to check the return value. */ kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags, unsigned int useroffset,
@@ -448,6 +460,10 @@ kmem_buckets *kmem_buckets_create(const char *name, slab_flags_t flags,if (!IS_ENABLED (CONFIG_SLAB_BUCKETS ))return ZERO_SIZE_PTR ; +/* NULL makes kmalloc_slab() use the regular kmalloc caches. */ +if (!slab_buckets_enabled) +return NULL ; +if (WARN_ON (!kmem_buckets_cache))return NULL ;
@@ -1059,7 +1075,7 @@ void __init create_kmalloc_caches(void)/* Kmalloc array is now usable */ slab_state =UP ; -if (IS_ENABLED (CONFIG_SLAB_BUCKETS )) +if (IS_ENABLED (CONFIG_SLAB_BUCKETS ) && slab_buckets_enabled) kmem_buckets_cache =kmem_cache_create ("kmalloc_buckets" ,sizeof (kmem_buckets),0 ,SLAB_NO_MERGE ,NULL );
Signature · 1 line
2.54.0
PATCH 6/8 slub_kunit: Add a test for the slab_buckets boot parameter
In reply to Chris Down--- lib/tests/slub_kunit.c | 37 +++++++++++++++++++++++++++++++++++++ mm/slab.h | 4 ++++ mm/slab_common.c | 4 +++- 3 files changed, 44 insertions(+), 1 deletion(-)
lib/tests/slub_kunit.c +37 -0
@@ -11,6 +11,8 @@#include <linux/kprobes.h> #include <linux/string_choices.h> #include <linux/swab.h> +#include <linux/mman.h> +#include <linux/string.h> #include "../mm/slab.h" static struct kunit_resource resource;
@@ -396,6 +398,38 @@ static void test_krealloc_redzone_zeroing(struct kunit *test)kmem_cache_destroy (s); } +#ifdef CONFIG_SLAB_BUCKETS +static void test_slab_buckets (struct kunit *test) +{ +unsigned long user_addr; +const char *name; +void *p; + +if (!slab_buckets_enabled) +KUNIT_EXPECT_NULL (test,kmem_buckets_create ("TestSlub_buckets" , +0 ,0 ,INT_MAX ,NULL )); + + user_addr =kunit_vm_mmap (test,NULL ,0 ,PAGE_SIZE , +PROT_READ |PROT_WRITE , +MAP_ANONYMOUS |MAP_PRIVATE ,0 ); +KUNIT_ASSERT_NE_MSG (test, user_addr,0 ,"Could not create userspace mm" ); +KUNIT_ASSERT_LT_MSG (test, user_addr, (unsigned long )TASK_SIZE , +"Failed to allocate user memory" ); + + p =memdup_user ((void __user *)user_addr,64 ); +KUNIT_ASSERT_NOT_ERR_OR_NULL (test, p); + + name =virt_to_slab (p)->slab_cache->name; +if (slab_buckets_enabled) +KUNIT_EXPECT_TRUE_MSG (test,strstarts (name,"memdup_user-" ), +"allocated from %s" , name); +else +KUNIT_EXPECT_TRUE_MSG (test,strstarts (name,"kmalloc-" ), +"allocated from %s" , name); +kfree (p); +} +#endif +#if defined(CONFIG_PERF_EVENTS ) || (defined(CONFIG_KPROBES ) && defined(CONFIG_SMP ))#define NR_ITERATIONS 1000#define NR_OBJECTS 1000
@@ -591,6 +625,9 @@ static struct kunit_case test_cases[] = {KUNIT_CASE (test_kfree_rcu_wq_destroy),KUNIT_CASE (test_leak_destroy),KUNIT_CASE (test_krealloc_redzone_zeroing), +#ifdef CONFIG_SLAB_BUCKETS +KUNIT_CASE (test_slab_buckets), +#endif #ifdef CONFIG_PERF_EVENTS KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_perf), #endif
mm/slab.h +4 -0
@@ -390,6 +390,10 @@ extern struct list_head slab_caches;/* The slab cache that manages slab cache information */ extern struct kmem_cache *kmem_cache; +#if IS_ENABLED (CONFIG_KUNIT ) +extern bool slab_buckets_enabled; +#endif +/* A table of kmalloc cache names and sizes */ extern const struct kmalloc_info_struct { const char *name[NR_KMALLOC_TYPES];
mm/slab_common.c +3 -1
@@ -29,6 +29,7 @@#include <linux/memcontrol.h> #include <linux/stackdepot.h> #include <trace/events/rcu.h> +#include <kunit/visibility.h> #include "../kernel/rcu/rcu.h" #include "internal.h"
@@ -410,7 +411,8 @@ EXPORT_SYMBOL(__kmem_cache_create_args);static struct kmem_cache *kmem_buckets_cache__ro_after_init ; -static bool slab_buckets_enabled__ro_after_init =IS_ENABLED (CONFIG_SLAB_BUCKETS ); +VISIBLE_IF_KUNIT bool slab_buckets_enabled __ro_after_init = IS_ENABLED(CONFIG_SLAB_BUCKETS); +EXPORT_SYMBOL_IF_KUNIT (slab_buckets_enabled);#ifdef CONFIG_SLAB_BUCKETS static int __init setup_slab_buckets (char *str)
Signature · 1 line
2.54.0
[email protected]
AI reviewRe: [PATCH 6/8] slub_kunit: Add a test for the slab_buckets boot parameter
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] mm: slub_kunit: test_slab_buckets() fails on !CONFIG_MMU systems --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] mm: slub_kunit: test_slab_buckets() fails on !CONFIG_MMU systems --
commit 4bcb44da46a191f12f48d8ca5b203a12aabf1dc9
Author: Chris Down <[email protected]>
slub_kunit: Add a test for the slab_buckets boot parameter
This commit exposes slab_buckets_enabled to KUnit tests and adds a test_slab_buckets() test case in lib/tests/slub_kunit.c to verify that kmem_buckets_create() and memdup_user() behave as expected based on whether slab_buckets is enabled.
Quoted 4 lines
lib/tests/slub_kunit.c +0
-0
[ ... ]
Quoted 9 lines
lib/tests/slub_kunit.c +5
-0@@ -396,6 +398,38 @@ static void test_krealloc_redzone_zeroing(struct kunit *test)kmem_cache_destroy (s); } +#ifdef CONFIG_SLAB_BUCKETS +static void test_slab_buckets (struct kunit *test) +{ +unsigned long user_addr; +const char *name;
lib/tests/slub_kunit.c static void test_krealloc_redzone_zeroing(struct kunit *test) · line 415
Quoted 10 lines
lib/tests/slub_kunit.c +10
-0+void *p; + +if (!slab_buckets_enabled) +KUNIT_EXPECT_NULL (test,kmem_buckets_create ("TestSlub_buckets" , +0 ,0 ,INT_MAX ,NULL )); + + user_addr =kunit_vm_mmap (test,NULL ,0 ,PAGE_SIZE , +PROT_READ |PROT_WRITE , +MAP_ANONYMOUS |MAP_PRIVATE ,0 ); +KUNIT_ASSERT_NE_MSG (test, user_addr,0 ,"Could not create userspace mm" );
[Severity: Medium]
Will test_slab_buckets() unconditionally fail on !CONFIG_MMU systems?
Neither CONFIG_SLUB_KUNIT_TEST nor CONFIG_SLAB_BUCKETS depends on CONFIG_MMU. When test_slab_buckets() runs in a KUnit kthread, kunit_vm_mmap() calls kunit_vm_mmap_init() -> kunit_attach_mm() in lib/kunit/user_alloc.c:
/* arch_pick_mmap_layout() is only sane with MMU systems. */ if (!IS_ENABLED(CONFIG_MMU)) return -EINVAL;
On !CONFIG_MMU builds, kunit_vm_mmap() returns 0 and triggers KUNIT_ASSERT_NE_MSG(test, user_addr, 0, "Could not create userspace mm").
Should test_slab_buckets() check !IS_ENABLED(CONFIG_MMU) and skip the userspace allocation part of the test with kunit_skip() (similar to usercopy_test_init() in lib/tests/usercopy_kunit.c)?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
PATCH 7/8 mm/slab: Support disabling KMALLOC_PARTITION_CACHES at boot
In reply to Chris DownAdd a kmalloc_partition= early parameter, enabled by default on kernels built with CONFIG_KMALLOC_PARTITION_CACHES=y.
Let's leave kmalloc_type() unchanged since each index simply resolves to the same cache when partitioning is off.
--- .../admin-guide/kernel-parameters.txt | 7 +++++++ mm/Kconfig | 5 +++++ mm/slab_common.c | 19 ++++++++++++++++++- 3 files changed, 30 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +7 -0
@@ -3054,6 +3054,13 @@ Kernel parameters Configure the RouterBoard 532 series on-chip Ethernet adapter MAC address. + kmalloc_partition= [MM,EARLY] + Format: <bool> + Enable or disable kmalloc cache partitioning on + kernels built with CONFIG_KMALLOC_PARTITION_CACHES=y. + Ignored without that option. + Default: on. + kmemleak= [KNL,EARLY] Boot-time kmemleak enable/disable Valid arguments: on, off Default: on
mm/Kconfig +5 -0
@@ -270,6 +270,11 @@ config KMALLOC_PARTITION_CACHES limited degree of memory and CPU overhead that relates to hardware and system workload. + The partitioning can be disabled at boot with the + "kmalloc_partition=0" kernel parameter, in which case the copies + are not created and all normal kmalloc allocations use the + regular kmalloc caches. + choice prompt "Partitioned slab cache mode" depends on KMALLOC_PARTITION_CACHES
mm/slab_common.c +18 -1
@@ -781,6 +781,16 @@ unsigned long random_kmalloc_seed __ro_after_init;EXPORT_SYMBOL (random_kmalloc_seed);#endif +#ifdef CONFIG_KMALLOC_PARTITION_CACHES +static bool kmalloc_partition_enabled__ro_after_init = true; + +static int __init setup_kmalloc_partition (char *str) +{ +return kstrtobool (str, &kmalloc_partition_enabled); +} +early_param ("kmalloc_partition" , setup_kmalloc_partition); +#endif +/* * Conversion table for small slabs sizes / 8 to the index in the * kmalloc array. This is necessary for slabs < 192 since we have non power
@@ -1013,8 +1023,15 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type type) }#ifdef CONFIG_KMALLOC_PARTITION_CACHES -if (type >=KMALLOC_PARTITION_START && type <=KMALLOC_PARTITION_END ) +if (!kmalloc_partition_enabled) { +/* Use the normal caches, which have already been created. */ +if (type >KMALLOC_PARTITION_START && type <=KMALLOC_PARTITION_END ) { + kmalloc_caches[type][idx] = kmalloc_caches[KMALLOC_NORMAL ][idx]; +return ; + } + }else if (type >=KMALLOC_PARTITION_START && type <=KMALLOC_PARTITION_END ) { flags |=SLAB_NO_MERGE ; + }#endif /*
Signature · 1 line
2.54.0
PATCH 8/8 slub_kunit: Add a test for the kmalloc_partition boot parameter
In reply to Chris Down--- lib/tests/slub_kunit.c | 46 ++++++++++++++++++++++++++++++++++++++++++ mm/slab.h | 4 ++++ mm/slab_common.c | 3 ++- 3 files changed, 52 insertions(+), 1 deletion(-)
lib/tests/slub_kunit.c +46 -0
@@ -430,6 +430,49 @@ static void test_slab_buckets(struct kunit *test) }#endif +#ifdef CONFIG_KMALLOC_PARTITION_CACHES +/* +* Each partition copy is a separate cache, unless the copies were disabled +* with kmalloc_partition=0: then they alias the KMALLOC_NORMAL caches. +*/ +static void test_kmalloc_partition (struct kunit *test) +{ +enum kmalloc_cache_type type; +struct kmem_cache *s; +unsigned int idx; +bool found; +void *p; + +for (type =KMALLOC_PARTITION_START +1 ; type <=KMALLOC_PARTITION_END ; type++) { +for (idx =0 ; idx <ARRAY_SIZE (kmalloc_caches[type]); idx++) { +struct kmem_cache *normal = kmalloc_caches[KMALLOC_NORMAL ][idx]; +struct kmem_cache *part = kmalloc_caches[type][idx]; + +if (!kmalloc_partition_enabled) { +KUNIT_EXPECT_PTR_EQ (test, part, normal); + }else if (normal) { +KUNIT_EXPECT_NOT_NULL (test, part); +KUNIT_EXPECT_PTR_NE (test, part, normal); + } + } + } + + p =kmalloc (64 ,GFP_KERNEL ); +KUNIT_ASSERT_NOT_NULL (test, p); + s =virt_to_slab (p)->slab_cache; + idx =kmalloc_index (64 ); +if (!kmalloc_partition_enabled) { +KUNIT_EXPECT_PTR_EQ (test, s, kmalloc_caches[KMALLOC_NORMAL ][idx]); + }else { + found = false; +for (type =KMALLOC_PARTITION_START ; type <=KMALLOC_PARTITION_END ; type++) + found |= s == kmalloc_caches[type][idx]; +KUNIT_EXPECT_TRUE_MSG (test, found,"allocated from %s" , s->name); + } +kfree (p); +} +#endif +#if defined(CONFIG_PERF_EVENTS ) || (defined(CONFIG_KPROBES ) && defined(CONFIG_SMP ))#define NR_ITERATIONS 1000#define NR_OBJECTS 1000
@@ -628,6 +671,9 @@ static struct kunit_case test_cases[] = {#ifdef CONFIG_SLAB_BUCKETS KUNIT_CASE(test_slab_buckets),#endif +#ifdef CONFIG_KMALLOC_PARTITION_CACHES + KUNIT_CASE(test_kmalloc_partition), +#endif #ifdef CONFIG_PERF_EVENTS KUNIT_CASE_SLOW(test_kmalloc_nolock_and_friends_perf),#endif
mm/slab.h +4 -0
@@ -394,6 +394,10 @@ extern struct kmem_cache *kmem_cache;extern bool slab_buckets_enabled;#endif +#if IS_ENABLED (CONFIG_KUNIT ) && defined(CONFIG_KMALLOC_PARTITION_CACHES ) +extern bool kmalloc_partition_enabled; +#endif +/* A table of kmalloc cache names and sizes */ extern const struct kmalloc_info_struct { const char *name[NR_KMALLOC_TYPES];
mm/slab_common.c +2 -1
@@ -782,7 +782,8 @@ EXPORT_SYMBOL(random_kmalloc_seed);#endif #ifdef CONFIG_KMALLOC_PARTITION_CACHES -static bool kmalloc_partition_enabled__ro_after_init = true; +VISIBLE_IF_KUNIT bool kmalloc_partition_enabled __ro_after_init = true; +EXPORT_SYMBOL_IF_KUNIT (kmalloc_partition_enabled);static int __init setup_kmalloc_partition (char *str) {
Signature · 1 line
2.54.0
Vlastimil Babka (SUSE)
maintainerQuoted 10 lines · Chris Down
Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. Let's allow those hosts to use one image and select each policy at boot instead.
This series adds four new boot parameters that default to their values from CONFIG_*:
slab_freelist_hardened=
slab_freelist_random=
These two should be only a small cpu overhead, which is why distros generally enable them [1]. Also the code basically only runs in slowpaths because percpu caching doesn't use freelists. It's possible the static key conversion doesn't help anything. You could try to measure the impact, I'd be surprised perhaps :)
Maybe we should just make them default=y.
Quoted 2 lines
slab_buckets=
kmalloc_partition=
These are more valid as there's a memory overhead tradeoff.
Quoted 24 lines
These mostly act at cache initialisation time. For free list hardening we use a static key settled before the first free pointer is written.
Chris Down (8):
mm/slub: Support disabling freelist hardening at boot
slub_kunit: Add a test for slab_freelist_hardened= mm/slub: Support disabling freelist randomisation at boot
slub_kunit: Add a test for slab_freelist_random= mm/slab: Support disablingSLAB_BUCKETSat boot
slub_kunit: Add a test for theslab_bucketsboot parameter mm/slab: Support disablingKMALLOC_PARTITION_CACHESat boot
slub_kunit: Add a test for thekmalloc_partitionboot parameter.../admin-guide/kernel-parameters.txt | 29 +++ lib/tests/slub_kunit.c | 186 ++++++++++++++++++ mm/Kconfig | 15 ++ mm/slab.h | 24 +++ mm/slab_common.c | 48 ++++- mm/slub.c | 68 ++++++- scripts/gdb/linux/slab.py | 8 +- 7 files changed, 361 insertions(+), 17 deletions(-)base-commit:
d15d7703166daf7bd62c844a3cb1b3a6279adb4c
Harry Yoo
maintainerQuoted 2 lines · Vlastimil Babka (SUSE)
These two should be only a small cpu overhead, which is why distros generally enable them [1].
Yeah I'd expect the impact on them to be small enough.
Quoted 4 lines
Quoted 2 lines
slab_buckets=
kmalloc_partition=These are more valid as there's a memory overhead tradeoff.
quickly measuring it on a VM with 64 CPUs and 64 GiB RAM:
w/ partitioned kmalloc: 94.0 MiB (235 kmalloc caches)
w/o partitioned kmalloc: 76.1 MiB (~17.9 MiB diff) (40 kmalloc caches)
Mostly due to a large number number of under-utilized slabs after boot. (per memalloc profiling)
slab_buckets had relately smaller impact, about ~1MiB after boot.
(Of course this is just after boot, and actual bloat depends on memory
usage and fragmentation on production)
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hi Vlastimil,
Quoted 5 lines · Vlastimil Babka (SUSE)
These two should be only a small cpu overhead, which is why distros generally enable them [1]. Also the code basically only runs in slowpaths because percpu caching doesn't use freelists. It's possible the static key conversion doesn't help anything. You could try to measure the impact, I'd be surprised perhaps :)
Well I measured it, and you're right :-)
I built kernels from d15d7703166d ("mm/slub: initialize the free pointer of the bootstrap kmem_cache_node") toggling SLAB_FREELIST_HARDENED and SLAB_FREELIST_RANDOM, with SLAB_BUCKETS, KMALLOC_PARTITION_CACHES and SHUFFLE_PAGE_ALLOCATOR pinned off on an AMD Genoa.
With both on vs both off, sched messaging, perf bench syscall fork, a tinyconfig kernel build and netperf over lo all moved by less than 0.8%, which is within the noise here. So the slowpaths do pay, as you'd expect. Here's the change in ns/object vs both off, over 64, 256 and 1024 byte objects:
hard rand both
batch alloc +5.5..+7.8% +7.6..+12.5% +13..+15% bulk alloc -2.0..+3.9% +28..+32% +33..+38% batch/bulk free +0.5..+4.6% -1.7..+0.6% +0.8..+4.5%
These are from a small module which just does ktime_get_ns() on one pinned CPU. batch allocates 4096 objects with kmalloc() and then frees them 1000 times. bulk does the same with kmem_cache_alloc_bulk() and kmem_cache_free_bulk() on a private cache, so it keeps creating new slabs, which is where the random seq gets applied.
So for us the right answer is to likely just turn these two on rather than make them switchable, too.
Quoted 1 line
Maybe we should just make them default=y.
Agreed. In v2 I'll replace the boot parameters for these two with patches which make them default y, and keep slab_buckets= and kmalloc_partition=.
Thank you for spotting this!
Harry Yoo
maintainerHi Chris!
Quoted 5 lines · Chris Down
Slab hardening settings are currently all build time choices. This means that, unfortunately, hosts with different requirements end up needing separate kernel images. Let's allow those hosts to use one image and select each policy at boot instead.
Quoted 7 lines
This series adds four new boot parameters that default to their values from CONFIG_*:
slab_freelist_hardened=
slab_freelist_random=
slab_buckets=
kmalloc_partition=
If we end up supporting boot-time parameters for hardening, I think it should be all enabled (or disabled) by:
slab_hardening=none|light|full
# please feel free to suggest bertter names
none: "Turn of all of them to maximize performance"
light: "I'm willying to sacrifice _some_ memory and CPU for security"
full: "I'm willying to sacrifice more overhead for security
IMHO more fine-grained knob for each feature doesn't make much sense.
Signature · 2 lines
Cheers, Harry / Hyeonggon
Hi Harry,
Quoted 6 lines · Harry Yoo
If we end up supporting boot-time parameters for hardening, I think it should be all enabled (or disabled) by:
slab_hardening=none|light|full
# please feel free to suggest bertter names
Quoted 7 lines
none: "Turn of all of them to maximize performance"
light: "I'm willying to sacrifice _some_ memory and CPU for security"
full: "I'm willying to sacrifice more overhead for security
IMHO more fine-grained knob for each feature doesn't make much sense.
I think if we did the 4 knobs that is a good idea. Given now it's two, I think it seems reasonable to keep it as two separate knobs unless you object? Otherwise I think given it will only cover the latter two, it should be something like kmalloc_isolation= instead. Let me know what you think.
Thanks!
Harry Yoo
maintainerQuoted 15 lines · Chris Down
Hi Harry,
Quoted 12 lines · Harry Yoo
If we end up supporting boot-time parameters for hardening, I think it should be all enabled (or disabled) by:
slab_hardening=none|light|full
# please feel free to suggest bertter namesnone: "Turn of all of them to maximize performance"
light: "I'm willying to sacrifice _some_ memory and CPU for security"
full: "I'm willying to sacrifice more overhead for security
Quoted 7 lines
Quoted 1 line
IMHO more fine-grained knob for each feature doesn't make much sense.
I think if we did the 4 knobs that is a good idea. Given now it's two, I think it seems reasonable to keep it as two separate knobs unless you object? Otherwise I think given it will only cover the latter two, it should be something like kmalloc_isolation= instead. Let me know what you think.
There has been constant, steady efforts to add more and more hardening features over the last decade. So I'd say slab_hardening= is more future proof :-)
Kees or Marco might have more insights on this.
Signature · 2 lines
Cheers, Harry / Hyeonggon