mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Subsystems (13)
A message that is not in the archive (sent off-list or not yet received): [email protected]
PATCH v6 1/3 mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed.
init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The two exceptions are slab caches with constructors and SLAB_TYPESAFE_BY_RCU flag. Those are never zero-initialized to preserve their semantics.
Both init_on_alloc and init_on_free default to zero, but those defaults can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_INIT_ON_FREE_DEFAULT_ON.
Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.
The new features are also going to pave the way for hardware memory tagging (e.g. arm64's MTE), which will require both on_alloc and on_free hooks to set the tags for heap objects. With MTE, tagging will have the same cost as memory initialization.
Although init_on_free is rather costly, there are paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructre for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series.
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
To: Kees Cook <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: James Morris <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Marco Elver <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v2:
- unconditionally initialize pages in kernel_init_free_pages()
- comment from Randy Dunlap: drop 'default false' lines from Kconfig.hardening
v3:
- don't call kernel_init_free_pages() from memblock_free_pages()
- adopted some Kees' comments for the patch description
v4:
- use NULL instead of 0 in slab_alloc_node() (found by kbuild test robot)
- don't write to NULL object in slab_alloc_node() (found by Android testing)
v5:
- adjusted documentation wording as suggested by Kees
- disable SLAB_POISON if auto-initialization is on
- don't wipe RCU cache allocations made without __GFP_ZERO
- dropped SLOB support
--- .../admin-guide/kernel-parameters.txt | 9 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 +++++++ kernel/kexec_core.c | 2 +- mm/dmapool.c | 2 +- mm/page_alloc.c | 63 ++++++++++++++++--- mm/slab.c | 16 ++++- mm/slab.h | 19 ++++++ mm/slub.c | 33 ++++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 29 +++++++++ 11 files changed, 180 insertions(+), 19 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +9 -0
@@ -1673,6 +1673,15 @@ initrd= [BOOT] Specify the location of the initial ramdisk + init_on_alloc= [MM] Fill newly allocated pages and heap objects with + zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. + + init_on_free= [MM] Fill freed pages and heap objects with zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. + init_pkru= [x86] Specify the default memory protection keys rights register contents for all processes. 0x55555554 by default (disallow access to all but pkey 0). Can
drivers/infiniband/core/uverbs_ioctl.c +1 -1
@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used =ALIGN (new_used,sizeof (*pbundle->internal_buffer)); -if (flags & __GFP_ZERO) +if (want_init_on_alloc (flags))memset (res,0 , size);return res; }
include/linux/mm.h +22 -0
@@ -2685,6 +2685,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { }#endif +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_alloc); +#else +DECLARE_STATIC_KEY_FALSE (init_on_alloc); +#endif +static inline bool want_init_on_alloc (gfp_t flags) +{ +if (static_branch_unlikely (&init_on_alloc)) +return true; +return flags & __GFP_ZERO; +} + +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_free); +#else +DECLARE_STATIC_KEY_FALSE (init_on_free); +#endif +static inline bool want_init_on_free (void ) +{ +return static_branch_unlikely (&init_on_free); +} +extern bool _debug_pagealloc_enabled;static inline bool debug_pagealloc_enabled (void )
kernel/kexec_core.c +1 -1
@@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)arch_kexec_post_alloc_pages (page_address (pages), count, gfp_mask); -if (gfp_mask & __GFP_ZERO) +if (want_init_on_alloc (gfp_mask))for (i =0 ; i < count; i++)clear_highpage (pages + i); }
mm/dmapool.c +1 -1
@@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,#endif spin_unlock_irqrestore (&pool->lock, flags); -if (mem_flags & __GFP_ZERO) +if (want_init_on_alloc (mem_flags))memset (retval,0 , pool->size);return retval;
mm/page_alloc.c +56 -7
95 lines not on this page · raw message
mm/slab.c +13 -3
@@ -1830,6 +1830,14 @@ static bool set_objfreelist_slab_cache(struct kmem_cache *cachep, cachep->num =0 ; +/* +* If slab auto-initialization on free is enabled, store the freelist +* off-slab, so that its contents don't end up in one of the allocated +* objects. +*/ +if (unlikely (slab_want_init_on_free (cachep))) +return false; +if (cachep->ctor || flags &SLAB_TYPESAFE_BY_RCU )return false;
@@ -3263,7 +3271,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,local_irq_restore (save_flags); ptr =cache_alloc_debugcheck_after (cachep, flags, ptr, caller); -if (unlikely (flags & __GFP_ZERO) && ptr) +if (unlikely (slab_want_init_on_alloc (flags, cachep)) && ptr)memset (ptr,0 , cachep->object_size);slab_post_alloc_hook (cachep, flags,1 , &ptr);
@@ -3320,7 +3328,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp =cache_alloc_debugcheck_after (cachep, flags, objp, caller);prefetchw (objp); -if (unlikely (flags & __GFP_ZERO) && objp) +if (unlikely (slab_want_init_on_alloc (flags, cachep)) && objp)memset (objp,0 , cachep->object_size);slab_post_alloc_hook (cachep, flags,1 , &objp);
@@ -3441,6 +3449,8 @@ void ___cache_free(struct kmem_cache *cachep, void *objp,struct array_cache *ac =cpu_cache_get (cachep);check_irq_off (); +if (unlikely (slab_want_init_on_free (cachep))) +memset (objp,0 , cachep->object_size);kmemleak_free_recursive (objp, cachep->flags); objp =cache_free_debugcheck (cachep, objp, caller);
@@ -3528,7 +3538,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,cache_alloc_debugcheck_after_bulk (s, flags, size, p, _RET_IP_);/* Clear memory outside IRQ disabled section */ -if (unlikely (flags & __GFP_ZERO)) +if (unlikely (slab_want_init_on_alloc (flags, s)))for (i =0 ; i < size; i++)memset (p[i],0 , s->object_size);
mm/slab.h +19 -0
@@ -524,4 +524,23 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep,static inline void cache_random_seq_destroy (struct kmem_cache *cachep) { }#endif /* CONFIG_SLAB_FREELIST_RANDOM */ +static inline bool slab_want_init_on_alloc (gfp_t flags,struct kmem_cache *c) +{ +if (static_branch_unlikely (&init_on_alloc)) { +if (c->ctor) +return false; +if (c->flags &SLAB_TYPESAFE_BY_RCU ) +return flags & __GFP_ZERO; +return true; + } +return flags & __GFP_ZERO; +} + +static inline bool slab_want_init_on_free (struct kmem_cache *c) +{ +if (static_branch_unlikely (&init_on_free)) +return !(c->ctor || (c->flags &SLAB_TYPESAFE_BY_RCU )); +return false; +} +#endif /* MM_SLAB_H */
mm/slub.c +28 -5
@@ -1279,6 +1279,12 @@ static int __init setup_slub_debug(char *str)if (*str ==',' ) slub_debug_slabs = str +1 ; out: +if ((static_branch_unlikely (&init_on_alloc) || +static_branch_unlikely (&init_on_free)) && + (slub_debug &SLAB_POISON )) { +pr_warn ("disabling SLAB_POISON: can't be used together with memory auto-initialization\n" ); + slub_debug &= ~SLAB_POISON ; + }return 1 ; }
@@ -1424,6 +1430,19 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x)static inline bool slab_free_freelist_hook (struct kmem_cache *s,void **head,void **tail) { + +void *object; +void *next = *head; +void *old_tail = *tail ? *tail : *head; + +if (slab_want_init_on_free (s)) +do { + object = next; + next =get_freepointer (s, object); +memset (object,0 , s->size); +set_freepointer (s, object, next); + }while (object != old_tail); +/* * Compiler cannot detect this function can be removed if slab_free_hook() * evaluates to nothing. Thus, catch all relevant config debug options here.
@@ -1433,9 +1452,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s,defined (CONFIG_DEBUG_OBJECTS_FREE ) || \defined (CONFIG_KASAN ) - void *object; -void *next = *head; -void *old_tail = *tail ? *tail : *head; + next = *head;/* Head and tail of the reconstructed freelist */ *head =NULL ;
@@ -2741,8 +2758,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,prefetch_freepointer (s, next_object);stat (s,ALLOC_FASTPATH ); } +/* +* If the object has been wiped upon free, make sure it's fully +* initialized by zeroing out freelist pointer. +*/ +if (unlikely (slab_want_init_on_free (s)) && object) + *(void **)object =NULL ; -if (unlikely (gfpflags & __GFP_ZERO) && object) +if (unlikely (slab_want_init_on_alloc (gfpflags, s)) && object)memset (object,0 , s->object_size);slab_post_alloc_hook (s, gfpflags,1 , &object);
@@ -3163,7 +3186,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,local_irq_enable ();/* Clear memory outside IRQ disabled fastpath loop */ -if (unlikely (flags & __GFP_ZERO)) { +if (unlikely (slab_want_init_on_alloc (flags, s))) {int j;for (j =0 ; j < i; j++)
net/core/sock.c +1 -1
@@ -1602,7 +1602,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk =kmem_cache_alloc (slab, priority & ~__GFP_ZERO);if (!sk)return sk; -if (priority & __GFP_ZERO) +if (want_init_on_alloc (priority))sk_prot_clear_nulls (sk, prot->obj_size); }else sk =kmalloc (prot->obj_size, priority);
security/Kconfig.hardening +29 -0
@@ -160,6 +160,35 @@ config STACKLEAK_RUNTIME_DISABLE runtime to control kernel stack erasing for kernels built with CONFIG_GCC_PLUGIN_STACKLEAK. +config INIT_ON_ALLOC_DEFAULT_ON + bool "Enable heap memory zeroing on allocation by default" + help + This has the effect of setting "init_on_alloc=1" on the kernel + command line. This can be disabled with "init_on_alloc=0". + When "init_on_alloc" is enabled, all page allocator and slab + allocator memory will be zeroed when allocated, eliminating + many kinds of "uninitialized heap memory" flaws, especially + heap content exposures. The performance impact varies by + workload, but most cases see <1% impact. Some synthetic + workloads have measured as high as 7%. + +config INIT_ON_FREE_DEFAULT_ON + bool "Enable heap memory zeroing on free by default" + help + This has the effect of setting "init_on_free=1" on the kernel + command line. This can be disabled with "init_on_free=0". + Similar to "init_on_alloc", when "init_on_free" is enabled, + all page allocator and slab allocator memory will be zeroed + when freed, eliminating many kinds of "uninitialized heap memory" + flaws, especially heap content exposures. The primary difference + with "init_on_free" is that data lifetime in memory is reduced, + as anything freed is wiped immediately, making live forensics or + cold boot memory attacks unable to recover freed memory contents. + The performance impact varies by workload, but is more expensive + than "init_on_alloc" due to the negative cache effects of + touching "cold" memory areas. Most cases see 3-5% impact. Some + synthetic workloads have measured as high as 8%. + endmenu endmenu
Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
Quoted 40 lines · Alexander Potapenko
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed.init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The two exceptions are slab caches with constructors and
SLAB_TYPESAFE_BY_RCUflag. Those are never zero-initialized to preserve their semantics.Both
init_on_allocandinit_on_freedefault to zero, but those defaults can be overridden withCONFIG_INIT_ON_ALLOC_DEFAULT_ONandCONFIG_INIT_ON_FREE_DEFAULT_ON.Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.
The new features are also going to pave the way for hardware memory tagging (e.g. arm64's MTE), which will require both
on_allocandon_freehooks to set the tags for heap objects. With MTE, tagging will have the same cost as memory initialization.
Quoted 8 lines
Although
init_on_freeis rather costly, there are paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructre for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series.Signed-off-by Alexander Potapenko <[email protected]>
-Kees
Quoted 457 lines
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
To: Kees Cook <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: James Morris <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Marco Elver <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v2:
- unconditionally initialize pages inkernel_init_free_pages()
- comment from Randy Dunlap: drop 'default false' lines fromKconfig.hardening
v3:
- don't callkernel_init_free_pages()frommemblock_free_pages()
- adopted some Kees' comments for the patch description
v4:
- use NULL instead of 0 inslab_alloc_node()(found by kbuild test robot)
- don't write to NULL object inslab_alloc_node()(found by Android testing)
v5:
- adjusted documentation wording as suggested by Kees
- disableSLAB_POISONif auto-initialization is on
- don't wipe RCU cache allocations made without__GFP_ZERO
- dropped SLOB support--- .../admin-guide/kernel-parameters.txt | 9 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 +++++++ kernel/kexec_core.c | 2 +- mm/dmapool.c | 2 +- mm/page_alloc.c | 63 ++++++++++++++++--- mm/slab.c | 16 ++++- mm/slab.h | 19 ++++++ mm/slub.c | 33 ++++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 29 +++++++++ 11 files changed, 180 insertions(+), 19 deletions(-)Documentation/admin-guide/kernel-parameters.txt +9
-0@@ -1673,6 +1673,15 @@ initrd= [BOOT] Specify the location of the initial ramdisk + init_on_alloc= [MM] Fill newly allocated pages and heap objects with + zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. + + init_on_free= [MM] Fill freed pages and heap objects with zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. + init_pkru= [x86] Specify the default memory protection keys rights register contents for all processes. 0x55555554 by default (disallow access to all but pkey 0). Candrivers/infiniband/core/uverbs_ioctl.c +1
-1@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used =ALIGN (new_used,sizeof (*pbundle->internal_buffer)); -if (flags & __GFP_ZERO) +if (want_init_on_alloc (flags))memset (res,0 , size);return res; }include/linux/mm.h +22
-0@@ -2685,6 +2685,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { }#endif +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_alloc); +#else +DECLARE_STATIC_KEY_FALSE (init_on_alloc); +#endif +static inline bool want_init_on_alloc (gfp_t flags) +{ +if (static_branch_unlikely (&init_on_alloc)) +return true; +return flags & __GFP_ZERO; +} + +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_free); +#else +DECLARE_STATIC_KEY_FALSE (init_on_free); +#endif +static inline bool want_init_on_free (void ) +{ +return static_branch_unlikely (&init_on_free); +} +extern bool _debug_pagealloc_enabled;static inline bool debug_pagealloc_enabled (void )kernel/kexec_core.c +1
-1@@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)arch_kexec_post_alloc_pages (page_address (pages), count, gfp_mask); -if (gfp_mask & __GFP_ZERO) +if (want_init_on_alloc (gfp_mask))for (i =0 ; i < count; i++)clear_highpage (pages + i); }mm/dmapool.c +1
-1@@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,#endif spin_unlock_irqrestore (&pool->lock, flags); -if (mem_flags & __GFP_ZERO) +if (want_init_on_alloc (mem_flags))memset (retval,0 , pool->size);return retval;mm/page_alloc.c +56
-7@@ -136,6 +136,48 @@ unsigned long totalcma_pages __read_mostly;int percpu_pagelist_fraction;gfp_t gfp_allowed_mask__read_mostly =GFP_BOOT_MASK ; +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE (init_on_alloc); +#else +DEFINE_STATIC_KEY_FALSE (init_on_alloc); +#endif +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DEFINE_STATIC_KEY_TRUE (init_on_free); +#else +DEFINE_STATIC_KEY_FALSE (init_on_free); +#endif + +static int __init early_init_on_alloc (char *buf) +{ +int ret; +bool bool_result; + +if (!buf) +return -EINVAL ; + ret =kstrtobool (buf, &bool_result); +if (bool_result) +static_branch_enable (&init_on_alloc); +else +static_branch_disable (&init_on_alloc); +return ret; +} +early_param ("init_on_alloc" , early_init_on_alloc); + +static int __init early_init_on_free (char *buf) +{ +int ret; +bool bool_result; + +if (!buf) +return -EINVAL ; + ret =kstrtobool (buf, &bool_result); +if (bool_result) +static_branch_enable (&init_on_free); +else +static_branch_disable (&init_on_free); +return ret; +} +early_param ("init_on_free" , early_init_on_free);/* * A cached value of the page's pageblock's migratetype, used when the page is @@ -1090,6 +1132,14 @@ static int free_tail_pages_check(struct page *head_page, struct page *page)return ret; } +static void kernel_init_free_pages (struct page *page,int numpages) +{ +int i; + +for (i =0 ; i < numpages; i++) +clear_highpage (page + i); +} + static __always_inline bool free_pages_prepare(struct page *page,unsigned int order,bool check_free) {@@ -1142,6 +1192,8 @@ static __always_inline bool free_pages_prepare(struct page *page, }arch_free_page (page, order);kernel_poison_pages (page,1 << order,0 ); +if (want_init_on_free ()) +kernel_init_free_pages (page,1 << order);if (debug_pagealloc_enabled ())kernel_map_pages (page,1 << order,0 );@@ -2020,8 +2072,8 @@ static inline int check_new_page(struct page *page)static inline bool free_pages_prezeroed (void ) { -return IS_ENABLED (CONFIG_PAGE_POISONING_ZERO ) && -page_poisoning_enabled (); +return (IS_ENABLED (CONFIG_PAGE_POISONING_ZERO ) && +page_poisoning_enabled ()) ||want_init_on_free (); }#ifdef CONFIG_DEBUG_VM @@ -2075,13 +2127,10 @@ inline void post_alloc_hook(struct page *page, unsigned int order,static void prep_new_page (struct page *page,unsigned int order,gfp_t gfp_flags,unsigned int alloc_flags) { -int i; -post_alloc_hook (page, order, gfp_flags); -if (!free_pages_prezeroed () && (gfp_flags & __GFP_ZERO)) -for (i =0 ; i < (1 << order); i++) -clear_highpage (page + i); +if (!free_pages_prezeroed () &&want_init_on_alloc (gfp_flags)) +kernel_init_free_pages (page,1 << order);if (order && (gfp_flags & __GFP_COMP))prep_compound_page (page, order);mm/slab.c +13
-3@@ -1830,6 +1830,14 @@ static bool set_objfreelist_slab_cache(struct kmem_cache *cachep, cachep->num =0 ; +/* +* If slab auto-initialization on free is enabled, store the freelist +* off-slab, so that its contents don't end up in one of the allocated +* objects. +*/ +if (unlikely (slab_want_init_on_free (cachep))) +return false; +if (cachep->ctor || flags &SLAB_TYPESAFE_BY_RCU )return false;@@ -3263,7 +3271,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,local_irq_restore (save_flags); ptr =cache_alloc_debugcheck_after (cachep, flags, ptr, caller); -if (unlikely (flags & __GFP_ZERO) && ptr) +if (unlikely (slab_want_init_on_alloc (flags, cachep)) && ptr)memset (ptr,0 , cachep->object_size);slab_post_alloc_hook (cachep, flags,1 , &ptr);@@ -3320,7 +3328,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp =cache_alloc_debugcheck_after (cachep, flags, objp, caller);prefetchw (objp); -if (unlikely (flags & __GFP_ZERO) && objp) +if (unlikely (slab_want_init_on_alloc (flags, cachep)) && objp)memset (objp,0 , cachep->object_size);slab_post_alloc_hook (cachep, flags,1 , &objp);@@ -3441,6 +3449,8 @@ void ___cache_free(struct kmem_cache *cachep, void *objp,struct array_cache *ac =cpu_cache_get (cachep);check_irq_off (); +if (unlikely (slab_want_init_on_free (cachep))) +memset (objp,0 , cachep->object_size);kmemleak_free_recursive (objp, cachep->flags); objp =cache_free_debugcheck (cachep, objp, caller);@@ -3528,7 +3538,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,cache_alloc_debugcheck_after_bulk (s, flags, size, p, _RET_IP_);/* Clear memory outside IRQ disabled section */ -if (unlikely (flags & __GFP_ZERO)) +if (unlikely (slab_want_init_on_alloc (flags, s)))for (i =0 ; i < size; i++)memset (p[i],0 , s->object_size);mm/slab.h +19
-0@@ -524,4 +524,23 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep,static inline void cache_random_seq_destroy (struct kmem_cache *cachep) { }#endif /* CONFIG_SLAB_FREELIST_RANDOM */ +static inline bool slab_want_init_on_alloc (gfp_t flags,struct kmem_cache *c) +{ +if (static_branch_unlikely (&init_on_alloc)) { +if (c->ctor) +return false; +if (c->flags &SLAB_TYPESAFE_BY_RCU ) +return flags & __GFP_ZERO; +return true; + } +return flags & __GFP_ZERO; +} + +static inline bool slab_want_init_on_free (struct kmem_cache *c) +{ +if (static_branch_unlikely (&init_on_free)) +return !(c->ctor || (c->flags &SLAB_TYPESAFE_BY_RCU )); +return false; +} +#endif /* MM_SLAB_H */ mm/slub.c +28
-5@@ -1279,6 +1279,12 @@ static int __init setup_slub_debug(char *str)if (*str ==',' ) slub_debug_slabs = str +1 ; out: +if ((static_branch_unlikely (&init_on_alloc) || +static_branch_unlikely (&init_on_free)) && + (slub_debug &SLAB_POISON )) { +pr_warn ("disabling SLAB_POISON: can't be used together with memory auto-initialization\n" ); + slub_debug &= ~SLAB_POISON ; + }return 1 ; }@@ -1424,6 +1430,19 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x)static inline bool slab_free_freelist_hook (struct kmem_cache *s,void **head,void **tail) { + +void *object; +void *next = *head; +void *old_tail = *tail ? *tail : *head; + +if (slab_want_init_on_free (s)) +do { + object = next; + next =get_freepointer (s, object); +memset (object,0 , s->size); +set_freepointer (s, object, next); + }while (object != old_tail); +/* * Compiler cannot detect this function can be removed if slab_free_hook() * evaluates to nothing. Thus, catch all relevant config debug options here. @@ -1433,9 +1452,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s,defined (CONFIG_DEBUG_OBJECTS_FREE ) || \defined (CONFIG_KASAN ) - void *object; -void *next = *head; -void *old_tail = *tail ? *tail : *head; + next = *head;/* Head and tail of the reconstructed freelist */ *head =NULL ;@@ -2741,8 +2758,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,prefetch_freepointer (s, next_object);stat (s,ALLOC_FASTPATH ); } +/* +* If the object has been wiped upon free, make sure it's fully +* initialized by zeroing out freelist pointer. +*/ +if (unlikely (slab_want_init_on_free (s)) && object) + *(void **)object =NULL ; -if (unlikely (gfpflags & __GFP_ZERO) && object) +if (unlikely (slab_want_init_on_alloc (gfpflags, s)) && object)memset (object,0 , s->object_size);slab_post_alloc_hook (s, gfpflags,1 , &object);@@ -3163,7 +3186,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,local_irq_enable ();/* Clear memory outside IRQ disabled fastpath loop */ -if (unlikely (flags & __GFP_ZERO)) { +if (unlikely (slab_want_init_on_alloc (flags, s))) {int j;for (j =0 ; j < i; j++)net/core/sock.c +1
-1@@ -1602,7 +1602,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk =kmem_cache_alloc (slab, priority & ~__GFP_ZERO);if (!sk)return sk; -if (priority & __GFP_ZERO) +if (want_init_on_alloc (priority))sk_prot_clear_nulls (sk, prot->obj_size); }else sk =kmalloc (prot->obj_size, priority);security/Kconfig.hardening +29
-0@@ -160,6 +160,35 @@ config STACKLEAK_RUNTIME_DISABLE runtime to control kernel stack erasing for kernels built with CONFIG_GCC_PLUGIN_STACKLEAK. +config INIT_ON_ALLOC_DEFAULT_ON + bool "Enable heap memory zeroing on allocation by default" + help + This has the effect of setting "init_on_alloc=1" on the kernel + command line. This can be disabled with "init_on_alloc=0". + When "init_on_alloc" is enabled, all page allocator and slab + allocator memory will be zeroed when allocated, eliminating + many kinds of "uninitialized heap memory" flaws, especially + heap content exposures. The performance impact varies by + workload, but most cases see <1% impact. Some synthetic + workloads have measured as high as 7%. + +config INIT_ON_FREE_DEFAULT_ON + bool "Enable heap memory zeroing on free by default" + help + This has the effect of setting "init_on_free=1" on the kernel + command line. This can be disabled with "init_on_free=0". + Similar to "init_on_alloc", when "init_on_free" is enabled, + all page allocator and slab allocator memory will be zeroed + when freed, eliminating many kinds of "uninitialized heap memory" + flaws, especially heap content exposures. The primary difference + with "init_on_free" is that data lifetime in memory is reduced, + as anything freed is wiped immediately, making live forensics or + cold boot memory attacks unable to recover freed memory contents. + The performance impact varies by workload, but is more expensive + than "init_on_alloc" due to the negative cache effects of + touching "cold" memory areas. Most cases see 3-5% impact. Some + synthetic workloads have measured as high as 8%. + endmenu endmenuSignature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
Signature · 1 line
Kees Cook
Quoted 17 lines · Kees Cook
Quoted 7 lines · Alexander Potapenko
[...]
mm/slub.c +0
-0mm/slub.c static __always_inline void *slab_alloc_node(struct kmem_cache *s, · line 2766
Quoted 9 lines
mm/slub.c +6
-0@@ -2741,8 +2758,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,prefetch_freepointer (s, next_object);stat (s,ALLOC_FASTPATH ); } +/* +* If the object has been wiped upon free, make sure it's fully +* initialized by zeroing out freelist pointer. +*/ +if (unlikely (slab_want_init_on_free (s)) && object) + *(void **)object =NULL ;
In looking at metadata again, I noticed that I don't think this is correct, as it needs to be using s->offset to find the location of the freelist pointer:
memset(object + s->offset, 0, sizeof(void *));
Quoted 4 lines
mm/slub.c static __always_inline void *slab_alloc_node(struct kmem_cache *s, · line 2769
Quoted 4 lines
mm/slub.c +1
-1-if (unlikely (gfpflags & __GFP_ZERO) && object) +if (unlikely (slab_want_init_on_alloc (gfpflags, s)) && object)memset (object,0 , s->object_size);
init_on_alloc is using "object_size" but init_on_free is using "size". I assume the "alloc" wipe is smaller because metadata was just written for the allocation?
Signature · 1 line
Kees Cook
Quoted 24 lines · Kees Cook
Quoted 17 lines · Kees Cook
Quoted 7 lines · Alexander Potapenko
[...]
mm/slub.c +0
-0mm/slub.c static __always_inline void *slab_alloc_node(struct kmem_cache *s, · line 2766
Quoted 9 lines
mm/slub.c +6
-0@@ -2741,8 +2758,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,prefetch_freepointer (s, next_object);stat (s,ALLOC_FASTPATH ); } +/* +* If the object has been wiped upon free, make sure it's fully +* initialized by zeroing out freelist pointer. +*/ +if (unlikely (slab_want_init_on_free (s)) && object) + *(void **)object =NULL ;In looking at metadata again, I noticed that I don't think this is correct, as it needs to be using
s->offsetto find the location of the freelist pointer:
Quoted 1 line
memset(object + s->offset, 0, sizeof(void *));
In the cases we support s->offset is always zero (we don't initialize slabs with ctors or RCU), but using its value is a sane generalization.
Quoted 8 lines
Quoted 4 lines
mm/slub.c static __always_inline void *slab_alloc_node(struct kmem_cache *s, · line 2769
Quoted 4 lines
mm/slub.c +1
-1-if (unlikely (gfpflags & __GFP_ZERO) && object) +if (unlikely (slab_want_init_on_alloc (gfpflags, s)) && object)memset (object,0 , s->object_size);
init_on_allocis using "object_size" butinit_on_freeis using "size". I assume the "alloc" wipe is smaller because metadata was just written for the allocation?
As noted in another thread, using "size" is incorrect, because it may overwrite the redzone after the object.
I'll send a patch to fix that.
Overwriting the metadata indeed shouldn't make sense in the allocation case.
Quoted 2 lines
--
Kees Cook
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
PATCH v6 2/3 mm: init: report memory auto-initialization features at boot time
In reply to a message not in the archivePrint the currently enabled stack and heap initialization modes.
Stack initialization is enabled by a config flag, while heap initialization is configured at boot time with defaults being set in the config. It's more convenient for the user to have all information about these hardening measures in one place.
The possible options for stack are:
- "all" for CONFIG_INIT_STACK_ALL;
- "byref_all" for CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL;
- "byref" for CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF;
- "__user" for CONFIG_GCC_PLUGIN_STRUCTLEAK_USER;
- "off" otherwise.
Depending on the values of init_on_alloc and init_on_free boottime options we also report "heap alloc" and "heap free" as "on"/"off".
In the init_on_free mode initializing pages at boot time may take some time, so print a notice about that as well.
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: James Morris <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: Matthew Wilcox <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Souptick Joarder <[email protected]>
Cc: Marco Elver <[email protected]>
Cc: Kaiwan N Billimoria <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v6:
- update patch description, fixed message about clearing memory
--- init/main.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
init/main.c +24 -0
@@ -520,6 +520,29 @@ static inline void initcall_debug_enable(void) }#endif +/* Report memory auto-initialization states for this boot. */ +void __init report_meminit (void ) +{ +const char *stack; + +if (IS_ENABLED (CONFIG_INIT_STACK_ALL )) + stack ="all" ; +else if (IS_ENABLED (CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL )) + stack ="byref_all" ; +else if (IS_ENABLED (CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF )) + stack ="byref" ; +else if (IS_ENABLED (CONFIG_GCC_PLUGIN_STRUCTLEAK_USER )) + stack ="__user" ; +else + stack ="off" ; + +pr_info ("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n" , + stack,want_init_on_alloc (GFP_KERNEL ) ?"on" :"off" , +want_init_on_free () ?"on" :"off" ); +if (want_init_on_free ()) +pr_info ("mem auto-init: clearing system memory may take some time...\n" ); +} +/* * Set up kernel memory allocators */
@@ -530,6 +553,7 @@ static void __init mm_init(void)* bigger than MAX_ORDER unless SPARSEMEM. */ page_ext_init_flatmem (); +report_meminit ();mem_init ();kmem_cache_init ();pgtable_init ();
Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
Re: [PATCH v6 2/3] mm: init: report memory auto-initialization features at boot time
Quoted 6 lines · Alexander Potapenko
Print the currently enabled stack and heap initialization modes.
Stack initialization is enabled by a config flag, while heap initialization is configured at boot time with defaults being set in the config. It's more convenient for the user to have all information about these hardening measures in one place.
Perhaps for clarity, add this to the end of the sentence:
"... at boot, so the user can reason about the expected behavior of the running system."
Quoted 7 lines
The possible options for stack are:
- "all" forCONFIG_INIT_STACK_ALL;
- "byref_all" forCONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL;
- "byref" forCONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF;
- "__user" forCONFIG_GCC_PLUGIN_STRUCTLEAK_USER;
- "off" otherwise.
Quoted 5 lines
Depending on the values of
init_on_allocandinit_on_freeboottime options we also report "heap alloc" and "heap free" as "on"/"off".In the
init_on_freemode initializing pages at boot time may take some time, so print a notice about that as well.
Perhaps give an example too:
This depends on how much memory is installed, the memory bandwidth, etc. On a relatively modern x86 system, it takes about 0.75s/GB to wipe all memory:
[ 0.418722] mem auto-init: stack:byref_all, heap alloc:off, heap free:on [ 0.419765] mem auto-init: clearing system memory may take some time... [ 12.376605] Memory: 16408564K/16776672K available (14339K kernel code, 1397K rwdata, 3756K rodata, 1636K init, 11460K bss, 368108K reserved, 0K cma-reserved)
More notes below...
Quoted 35 lines
Signed-off-by Alexander Potapenko <[email protected]>Suggested-by Kees Cook <[email protected]>To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: James Morris <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: Matthew Wilcox <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Souptick Joarder <[email protected]>
Cc: Marco Elver <[email protected]>
Cc: Kaiwan N Billimoria <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v6:
- update patch description, fixed message about clearing memory--- init/main.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)init/main.c +0
-0
init/main.c static inline void initcall_debug_enable(void) · line 524
Quoted 5 lines
init/main.c +2
-0@@ -520,6 +520,29 @@ static inline void initcall_debug_enable(void) }#endif +/* Report memory auto-initialization states for this boot. */ +void __init report_meminit (void )
Sorry I missed this before: it should be a static function.
Quoted 35 lines
init/main.c +22
-0+{ +const char *stack; + +if (IS_ENABLED (CONFIG_INIT_STACK_ALL )) + stack ="all" ; +else if (IS_ENABLED (CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL )) + stack ="byref_all" ; +else if (IS_ENABLED (CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF )) + stack ="byref" ; +else if (IS_ENABLED (CONFIG_GCC_PLUGIN_STRUCTLEAK_USER )) + stack ="__user" ; +else + stack ="off" ; + +pr_info ("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n" , + stack,want_init_on_alloc (GFP_KERNEL ) ?"on" :"off" , +want_init_on_free () ?"on" :"off" ); +if (want_init_on_free ()) +pr_info ("mem auto-init: clearing system memory may take some time...\n" ); +} +/* * Set up kernel memory allocators */ @@ -530,6 +553,7 @@ static void __init mm_init(void)* bigger than MAX_ORDER unless SPARSEMEM. */ page_ext_init_flatmem (); +report_meminit ();mem_init ();kmem_cache_init ();pgtable_init ();Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
But other than that:
Signature · 1 line
Kees Cook
PATCH v6 3/3 lib: introduce test_meminit module
In reply to a message not in the archiveAdd tests for heap and pagealloc initialization.
These can be used to check init_on_alloc and init_on_free implementations as well as other approaches to initialization.
Expected test output in the case the kernel provides heap initialization (e.g. when running with either init_on_alloc=1 or init_on_free=1):
test_meminit: all 10 tests in test_pages passed
test_meminit: all 40 tests in test_kvmalloc passed
test_meminit: all 60 tests in test_kmemcache passed
test_meminit: all 10 tests in test_rcu_persistent passed
test_meminit: all 120 tests passed!
To: Kees Cook <[email protected]>
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Marco Elver <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v3:
- added example test output to the description
- fixed a missing include spotted by kbuild test robot <[email protected]>
- added a missing MODULE_LICENSE
- call do_kmem_cache_size() with size >= sizeof(void*) to unbreak debug builds
v5:
- added tests for RCU slabs and __GFP_ZERO
--- lib/Kconfig.debug | 8 + lib/Makefile | 1 + lib/test_meminit.c | 362 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 371 insertions(+) create mode 100644 lib/test_meminit.c
lib/Kconfig.debug +8 -0
@@ -2040,6 +2040,14 @@ config TEST_STACKINIT If unsure, say N. +config TEST_MEMINIT + tristate "Test heap/page initialization" + help + Test if the kernel is zero-initializing heap and page allocations. + This can be useful to test init_on_alloc and init_on_free features. + + If unsure, say N. + endif # RUNTIME_TESTING_MENU config MEMTEST
lib/Makefile +1 -0
@@ -91,6 +91,7 @@ obj-$(CONFIG_TEST_DEBUG_VIRTUAL) += test_debug_virtual.o obj-$(CONFIG_TEST_MEMCAT_P) += test_memcat_p.o obj-$(CONFIG_TEST_OBJAGG) += test_objagg.o obj-$(CONFIG_TEST_STACKINIT) += test_stackinit.o +obj-$(CONFIG_TEST_MEMINIT) += test_meminit.o obj-$(CONFIG_TEST_LIVEPATCH) += livepatch/
lib/test_meminit.c new file +362 -0
362 lines not on this page · raw message
Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
Re: [PATCH v6 3/3] lib: introduce test_meminit module
Quoted 7 lines · Alexander Potapenko
Add tests for heap and pagealloc initialization.
These can be used to checkinit_on_allocandinit_on_freeimplementations as well as other approaches to initialization.Expected test output in the case the kernel provides heap initialization (e.g. when running with either init_on_alloc=1 or init_on_free=1):
Quoted 7 lines
test_meminit: all 10 tests intest_pagespassed
test_meminit: all 40 tests intest_kvmallocpassed
test_meminit: all 60 tests intest_kmemcachepassed
test_meminit: all 10 tests intest_rcu_persistentpassed
test_meminit: all 120 tests passed!Signed-off-by Alexander Potapenko <[email protected]>
-Kees
Quoted 431 lines
To: Kees Cook <[email protected]>
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Marco Elver <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v3:
- added example test output to the description
- fixed a missing include spotted by kbuild test robot <[email protected]>
- added a missingMODULE_LICENSE
- calldo_kmem_cache_size()with size >=sizeof(void*)to unbreak debug builds
v5:
- added tests for RCU slabs and__GFP_ZERO--- lib/Kconfig.debug | 8 + lib/Makefile | 1 + lib/test_meminit.c | 362 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 371 insertions(+) create mode 100644 lib/test_meminit.clib/Kconfig.debug +8
-0@@ -2040,6 +2040,14 @@ config TEST_STACKINIT If unsure, say N. +config TEST_MEMINIT + tristate "Test heap/page initialization" + help + Test if the kernel is zero-initializing heap and page allocations. + This can be useful to test init_on_alloc and init_on_free features. + + If unsure, say N. + endif # RUNTIME_TESTING_MENU config MEMTESTlib/Makefile +1
-0@@ -91,6 +91,7 @@ obj-$(CONFIG_TEST_DEBUG_VIRTUAL) += test_debug_virtual.o obj-$(CONFIG_TEST_MEMCAT_P) += test_memcat_p.o obj-$(CONFIG_TEST_OBJAGG) += test_objagg.o obj-$(CONFIG_TEST_STACKINIT) += test_stackinit.o +obj-$(CONFIG_TEST_MEMINIT) += test_meminit.o obj-$(CONFIG_TEST_LIVEPATCH) += livepatch/lib/test_meminit.c new file +362
-0@@ -0,0 +1,362 @@ +// SPDX-License-Identifier: GPL-2.0 +/* +* Test cases for SL[AOU]B/page initialization at alloc/free time. +*/ +#define pr_fmt (fmt) KBUILD_MODNAME ": " fmt + +#include <linux/init.h> +#include <linux/kernel.h> +#include <linux/mm.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <linux/string.h> +#include <linux/vmalloc.h> + +#define GARBAGE_INT (0x09A7BA9E) +#define GARBAGE_BYTE (0x9E) + +#define REPORT_FAILURES_IN_FN () \ + do { \ + if (failures) \ + pr_info("%s failed %d out of %d times\n", \ + __func__, failures, num_tests); \ + else \ + pr_info("all %d tests in %s passed\n", \ + num_tests, __func__); \ + } while (0) + +/* Calculate the number of uninitialized bytes in the buffer. */ +static int __init count_nonzero_bytes (void *ptr,size_t size) +{ +int i, ret =0 ; +unsigned char *p = (unsigned char *)ptr; + +for (i =0 ; i < size; i++) +if (p[i]) + ret++; +return ret; +} + +/* Fill a buffer with garbage, skipping |skip| first bytes. */ +static void __init fill_with_garbage_skip (void *ptr,size_t size,size_t skip) +{ +unsigned int *p = (unsigned int *)ptr; +int i =0 ; + +if (skip) { +WARN_ON (skip > size); + p += skip; + } +while (size >=sizeof (*p)) { + p[i] =GARBAGE_INT ; + i++; + size -=sizeof (*p); + } +if (size) +memset (&p[i],GARBAGE_BYTE , size); +} + +static void __init fill_with_garbage (void *ptr,size_t size) +{ +fill_with_garbage_skip (ptr, size,0 ); +} + +static int __init do_alloc_pages_order (int order,int *total_failures) +{ +struct page *page; +void *buf; +size_t size =PAGE_SIZE << order; + + page =alloc_pages (GFP_KERNEL , order); + buf =page_address (page); +fill_with_garbage (buf, size); +__free_pages (page, order); + + page =alloc_pages (GFP_KERNEL , order); + buf =page_address (page); +if (count_nonzero_bytes (buf, size)) + (*total_failures)++; +fill_with_garbage (buf, size); +__free_pages (page, order); +return 1 ; +} + +/* Test the page allocator by calling alloc_pages with different orders. */ +static int __init test_pages (int *total_failures) +{ +int failures =0 , num_tests =0 ; +int i; + +for (i =0 ; i <10 ; i++) + num_tests +=do_alloc_pages_order (i, &failures); + +REPORT_FAILURES_IN_FN (); + *total_failures += failures; +return num_tests; +} + +/* Test kmalloc() with given parameters. */ +static int __init do_kmalloc_size (size_t size,int *total_failures) +{ +void *buf; + + buf =kmalloc (size,GFP_KERNEL ); +fill_with_garbage (buf, size); +kfree (buf); + + buf =kmalloc (size,GFP_KERNEL ); +if (count_nonzero_bytes (buf, size)) + (*total_failures)++; +fill_with_garbage (buf, size); +kfree (buf); +return 1 ; +} + +/* Test vmalloc() with given parameters. */ +static int __init do_vmalloc_size (size_t size,int *total_failures) +{ +void *buf; + + buf =vmalloc (size); +fill_with_garbage (buf, size); +vfree (buf); + + buf =vmalloc (size); +if (count_nonzero_bytes (buf, size)) + (*total_failures)++; +fill_with_garbage (buf, size); +vfree (buf); +return 1 ; +} + +/* Test kmalloc()/vmalloc() by allocating objects of different sizes. */ +static int __init test_kvmalloc (int *total_failures) +{ +int failures =0 , num_tests =0 ; +int i, size; + +for (i =0 ; i <20 ; i++) { + size =1 << i; + num_tests +=do_kmalloc_size (size, &failures); + num_tests +=do_vmalloc_size (size, &failures); + } + +REPORT_FAILURES_IN_FN (); + *total_failures += failures; +return num_tests; +} + +#define CTOR_BYTES (sizeof(unsigned int)) +#define CTOR_PATTERN (0x41414141) +/* Initialize the first 4 bytes of the object. */ +static void test_ctor (void *obj) +{ + *(unsigned int *)obj =CTOR_PATTERN ; +} + +/* +* Check the invariants for the buffer allocated from a slab cache. +* If the cache has a test constructor, the first 4 bytes of the object must +* always remain equal to CTOR_PATTERN. +* If the cache isn't an RCU-typesafe one, or if the allocation is done with +* __GFP_ZERO, then the object contents must be zeroed after allocation. +* If the cache is an RCU-typesafe one, the object contents must never be +* zeroed after the first use. This is checked by memcmp() in +* do_kmem_cache_size(). +*/ +static bool __init check_buf (void *buf,int size,bool want_ctor, +bool want_rcu,bool want_zero) +{ +int bytes; +bool fail = false; + + bytes =count_nonzero_bytes (buf, size); +WARN_ON (want_ctor && want_zero); +if (want_zero) +return bytes; +if (want_ctor) { +if (*(unsigned int *)buf !=CTOR_PATTERN ) + fail =1 ; + }else { +if (bytes) + fail = !want_rcu; + } +return fail; +} + +/* +* Test kmem_cache with given parameters: +* want_ctor - use a constructor; +* want_rcu - use SLAB_TYPESAFE_BY_RCU; +* want_zero - use __GFP_ZERO. +*/ +static int __init do_kmem_cache_size (size_t size,bool want_ctor, +bool want_rcu,bool want_zero, +int *total_failures) +{ +struct kmem_cache *c; +int iter; +bool fail = false; +gfp_t alloc_mask =GFP_KERNEL | (want_zero ? __GFP_ZERO :0 ); +void *buf, *buf_copy; + + c =kmem_cache_create ("test_cache" , size,1 , + want_rcu ?SLAB_TYPESAFE_BY_RCU :0 , + want_ctor ? test_ctor :NULL ); +for (iter =0 ; iter <10 ; iter++) { + buf =kmem_cache_alloc (c, alloc_mask); +/* Check that buf is zeroed, if it must be. */ + fail =check_buf (buf, size, want_ctor, want_rcu, want_zero); +fill_with_garbage_skip (buf, size, want_ctor ?CTOR_BYTES :0 ); +/* +* If this is an RCU cache, use a critical section to ensure we +* can touch objects after they're freed. +*/ +if (want_rcu) { +rcu_read_lock (); +/* +* Copy the buffer to check that it's not wiped on +* free(). +*/ + buf_copy =kmalloc (size,GFP_KERNEL ); +if (buf_copy) +memcpy (buf_copy, buf, size); + } +kmem_cache_free (c, buf); +if (want_rcu) { +/* +* Check that |buf| is intact after kmem_cache_free(). +* |want_zero| is false, because we wrote garbage to +* the buffer already. +*/ + fail |=check_buf (buf, size, want_ctor, want_rcu, + false); +if (buf_copy) { + fail |= (bool )memcmp (buf, buf_copy, size); +kfree (buf_copy); + } +rcu_read_unlock (); + } + } +kmem_cache_destroy (c); + + *total_failures += fail; +return 1 ; +} + +/* +* Check that the data written to an RCU-allocated object survives +* reallocation. +*/ +static int __init do_kmem_cache_rcu_persistent (int size,int *total_failures) +{ +struct kmem_cache *c; +void *buf, *buf_contents, *saved_ptr; +void **used_objects; +int i, iter, maxiter =1024 ; +bool fail = false; + + c =kmem_cache_create ("test_cache" , size, size,SLAB_TYPESAFE_BY_RCU , +NULL ); + buf =kmem_cache_alloc (c,GFP_KERNEL ); + saved_ptr = buf; +fill_with_garbage (buf, size); + buf_contents =kmalloc (size,GFP_KERNEL ); +if (!buf_contents) + goto out; + used_objects =kmalloc_array (maxiter,sizeof (void *),GFP_KERNEL ); +if (!used_objects) { +kfree (buf_contents); + goto out; + } +memcpy (buf_contents, buf, size); +kmem_cache_free (c, buf); +/* +* Run for a fixed number of iterations. If we never hit saved_ptr, +* assume the test passes. +*/ +for (iter =0 ; iter < maxiter; iter++) { + buf =kmem_cache_alloc (c,GFP_KERNEL ); + used_objects[iter] = buf; +if (buf == saved_ptr) { + fail =memcmp (buf_contents, buf, size); +for (i =0 ; i <= iter; i++) +kmem_cache_free (c, used_objects[i]); + goto free_out; + } + } + +free_out: +kmem_cache_destroy (c); +kfree (buf_contents); +kfree (used_objects); +out: + *total_failures += fail; +return 1 ; +} + +/* +* Test kmem_cache allocation by creating caches of different sizes, with and +* without constructors, with and without SLAB_TYPESAFE_BY_RCU. +*/ +static int __init test_kmemcache (int *total_failures) +{ +int failures =0 , num_tests =0 ; +int i, flags, size; +bool ctor, rcu, zero; + +for (i =0 ; i <10 ; i++) { + size =8 << i; +for (flags =0 ; flags <8 ; flags++) { + ctor = flags &1 ; + rcu = flags &2 ; + zero = flags &4 ; +if (ctor & zero) +continue ; + num_tests +=do_kmem_cache_size (size, ctor, rcu, zero, + &failures); + } + } +REPORT_FAILURES_IN_FN (); + *total_failures += failures; +return num_tests; +} + +/* Test the behavior of SLAB_TYPESAFE_BY_RCU caches of different sizes. */ +static int __init test_rcu_persistent (int *total_failures) +{ +int failures =0 , num_tests =0 ; +int i, size; + +for (i =0 ; i <10 ; i++) { + size =8 << i; + num_tests +=do_kmem_cache_rcu_persistent (size, &failures); + } +REPORT_FAILURES_IN_FN (); + *total_failures += failures; +return num_tests; +} + +/* +* Run the tests. Each test function returns the number of executed tests and +* updates |failures| with the number of failed tests. +*/ +static int __init test_meminit_init (void ) +{ +int failures =0 , num_tests =0 ; + + num_tests +=test_pages (&failures); + num_tests +=test_kvmalloc (&failures); + num_tests +=test_kmemcache (&failures); + num_tests +=test_rcu_persistent (&failures); + +if (failures ==0 ) +pr_info ("all %d tests passed!\n" , num_tests); +else +pr_info ("failures: %d out of %d\n" , failures, num_tests); + +return failures ? -EINVAL :0 ; +} +module_init (test_meminit_init); + +MODULE_LICENSE ("GPL" );Signature · 1 line
2.22.0.rc1.311.g5d7573a151-goog
Signature · 1 line
Kees Cook