RFC: add init_on_alloc/init_on_free boot options
Subsystems (18)
v2 cover RFC: add init_on_alloc/init_on_free boot options
Provide init_on_alloc and init_on_free boot options.
These are aimed at preventing possible information leaks and making the control-flow bugs that depend on uninitialized values more deterministic.
Enabling either of the options guarantees that the memory returned by the page allocator and SL[AOU]B is initialized with zeroes.
Enabling init_on_free also guarantees that pages and heap objects are initialized right after they're freed, so it won't be possible to access stale data by using a dangling pointer.
Alexander Potapenko (4):
mm: security: introduce init_on_alloc=1 and init_on_free=1 boot
options
lib: introduce test_meminit module
gfp: mm: introduce __GFP_NO_AUTOINIT
net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations
.../admin-guide/kernel-parameters.txt | 8 + drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/gfp.h | 13 +- include/linux/mm.h | 22 ++ include/net/sock.h | 5 + kernel/kexec_core.c | 5 +- lib/Kconfig.debug | 8 + lib/Makefile | 1 + lib/test_meminit.c | 205 ++++++++++++++++++ mm/dmapool.c | 2 +- mm/page_alloc.c | 68 +++++- mm/slab.c | 18 +- mm/slab.h | 16 ++ mm/slob.c | 23 +- mm/slub.c | 28 ++- net/core/sock.c | 31 ++- net/unix/af_unix.c | 13 +- security/Kconfig.hardening | 14 ++ 18 files changed, 443 insertions(+), 39 deletions(-) create mode 100644 lib/test_meminit.c
Signature · 1 line
2.21.0.1020.gf2820cf01a-goog
PATCH v2 1/4 mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed.
init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The only exception is slab caches with constructors. Those are never zero-initialized to preserve their semantics.
For SLOB allocator init_on_free=1 also implies init_on_alloc=1 behavior, i.e. objects are zeroed at both allocation and deallocation time. This is done because SLOB may otherwise return multiple freelist pointers in the allocated object. For SLAB and SLUB enabling either init_on_alloc or init_on_free leads to one-time initialization of the object.
Both init_on_alloc and init_on_free default to zero, but those defaults can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_INIT_ON_FREE_DEFAULT_ON.
Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
To: Kees Cook <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: James Morris <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v2:
- unconditionally initialize pages in kernel_init_free_pages()
- comment from Randy Dunlap: drop 'default false' lines from Kconfig.hardening
--- .../admin-guide/kernel-parameters.txt | 8 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 ++++++ kernel/kexec_core.c | 2 +- mm/dmapool.c | 2 +- mm/page_alloc.c | 68 ++++++++++++++++--- mm/slab.c | 16 ++++- mm/slab.h | 16 +++++ mm/slob.c | 22 +++++- mm/slub.c | 27 ++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 14 ++++ 12 files changed, 178 insertions(+), 23 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +8 -0
@@ -1673,6 +1673,14 @@ initrd= [BOOT] Specify the location of the initial ramdisk + init_on_alloc= [MM] Fill newly allocated pages and heap objects with + zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. + init_on_free= [MM] Fill freed pages and heap objects with zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. + init_pkru= [x86] Specify the default memory protection keys rights register contents for all processes. 0x55555554 by default (disallow access to all but pkey 0). Can
drivers/infiniband/core/uverbs_ioctl.c +1 -1
@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used =ALIGN (new_used,sizeof (*pbundle->internal_buffer)); -if (flags & __GFP_ZERO) +if (want_init_on_alloc (flags))memset (res,0 , size);return res; }
include/linux/mm.h +22 -0
@@ -2610,6 +2610,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { }#endif +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_alloc); +#else +DECLARE_STATIC_KEY_FALSE (init_on_alloc); +#endif +static inline bool want_init_on_alloc (gfp_t flags) +{ +if (static_branch_unlikely (&init_on_alloc)) +return true; +return flags & __GFP_ZERO; +} + +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_free); +#else +DECLARE_STATIC_KEY_FALSE (init_on_free); +#endif +static inline bool want_init_on_free (void ) +{ +return static_branch_unlikely (&init_on_free); +} +extern bool _debug_pagealloc_enabled;static inline bool debug_pagealloc_enabled (void )
kernel/kexec_core.c +1 -1
@@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)arch_kexec_post_alloc_pages (page_address (pages), count, gfp_mask); -if (gfp_mask & __GFP_ZERO) +if (want_init_on_alloc (gfp_mask))for (i =0 ; i < count; i++)clear_highpage (pages + i); }
mm/dmapool.c +1 -1
@@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,#endif spin_unlock_irqrestore (&pool->lock, flags); -if (mem_flags & __GFP_ZERO) +if (want_init_on_alloc (mem_flags))memset (retval,0 , pool->size);return retval;
mm/page_alloc.c +59 -9
109 lines not on this page · raw message
mm/slab.c +13 -3
@@ -1855,6 +1855,14 @@ static bool set_objfreelist_slab_cache(struct kmem_cache *cachep, cachep->num =0 ; +/* +* If slab auto-initialization on free is enabled, store the freelist +* off-slab, so that its contents don't end up in one of the allocated +* objects. +*/ +if (unlikely (slab_want_init_on_free (cachep))) +return false; +if (cachep->ctor || flags &SLAB_TYPESAFE_BY_RCU )return false;
@@ -3294,7 +3302,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,local_irq_restore (save_flags); ptr =cache_alloc_debugcheck_after (cachep, flags, ptr, caller); -if (unlikely (flags & __GFP_ZERO) && ptr) +if (unlikely (slab_want_init_on_alloc (flags, cachep)) && ptr)memset (ptr,0 , cachep->object_size);slab_post_alloc_hook (cachep, flags,1 , &ptr);
@@ -3351,7 +3359,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp =cache_alloc_debugcheck_after (cachep, flags, objp, caller);prefetchw (objp); -if (unlikely (flags & __GFP_ZERO) && objp) +if (unlikely (slab_want_init_on_alloc (flags, cachep)) && objp)memset (objp,0 , cachep->object_size);slab_post_alloc_hook (cachep, flags,1 , &objp);
@@ -3472,6 +3480,8 @@ void ___cache_free(struct kmem_cache *cachep, void *objp,struct array_cache *ac =cpu_cache_get (cachep);check_irq_off (); +if (unlikely (slab_want_init_on_free (cachep))) +memset (objp,0 , cachep->object_size);kmemleak_free_recursive (objp, cachep->flags); objp =cache_free_debugcheck (cachep, objp, caller);
@@ -3559,7 +3569,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,cache_alloc_debugcheck_after_bulk (s, flags, size, p, _RET_IP_);/* Clear memory outside IRQ disabled section */ -if (unlikely (flags & __GFP_ZERO)) +if (unlikely (slab_want_init_on_alloc (flags, s)))for (i =0 ; i < size; i++)memset (p[i],0 , s->object_size);
mm/slab.h +16 -0
@@ -524,4 +524,20 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep,static inline void cache_random_seq_destroy (struct kmem_cache *cachep) { }#endif /* CONFIG_SLAB_FREELIST_RANDOM */ +static inline bool slab_want_init_on_alloc (gfp_t flags,struct kmem_cache *c) +{ +if (static_branch_unlikely (&init_on_alloc)) +return !(c->ctor); +else +return flags & __GFP_ZERO; +} + +static inline bool slab_want_init_on_free (struct kmem_cache *c) +{ +if (static_branch_unlikely (&init_on_free)) +return !(c->ctor); +else +return false; +} +#endif /* MM_SLAB_H */
mm/slob.c +20 -2
@@ -212,6 +212,19 @@ static void slob_free_pages(void *b, int order)free_pages ((unsigned long )b, order); } +/* +* init_on_free=1 also implies initialization at allocation time. +* This is because newly allocated objects may contain freelist pointers +* somewhere in the middle. +*/ +static inline bool slob_want_init_on_alloc (gfp_t flags,struct kmem_cache *c) +{ +if (static_branch_unlikely (&init_on_alloc) || +static_branch_unlikely (&init_on_free)) +return c ? (!c->ctor) : true; +return flags & __GFP_ZERO; +} +/* * Allocate a slob block within a given slob_page sp. */
@@ -330,8 +343,6 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node)BUG_ON (!b);spin_unlock_irqrestore (&slob_lock, flags); } -if (unlikely (gfp & __GFP_ZERO)) -memset (b,0 , size);return b; }
@@ -366,6 +377,9 @@ static void slob_free(void *block, int size)return ; } +if (unlikely (want_init_on_free ())) +memset (block,0 , size); +if (!slob_page_free (sp)) {/* This slob page is about to become partially free. Easy! */ sp->units = units;
@@ -461,6 +475,8 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller) }kmemleak_alloc (ret, size,1 , gfp); +if (unlikely (slob_want_init_on_alloc (gfp,0 ))) +memset (ret,0 , size);return ret; }
@@ -559,6 +575,8 @@ static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node)WARN_ON_ONCE (flags & __GFP_ZERO); c->ctor (b); } +if (unlikely (slob_want_init_on_alloc (flags, c))) +memset (b,0 , c->size);kmemleak_alloc_recursive (b, c->size,1 , c->flags, flags);return b;
mm/slub.c +22 -5
@@ -1423,6 +1423,19 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x)static inline bool slab_free_freelist_hook (struct kmem_cache *s,void **head,void **tail) { + +void *object; +void *next = *head; +void *old_tail = *tail ? *tail : *head; + +if (slab_want_init_on_free (s)) +do { + object = next; + next =get_freepointer (s, object); +memset (object,0 , s->size); +set_freepointer (s, object, next); + }while (object != old_tail); +/* * Compiler cannot detect this function can be removed if slab_free_hook() * evaluates to nothing. Thus, catch all relevant config debug options here.
@@ -1432,9 +1445,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s,defined (CONFIG_DEBUG_OBJECTS_FREE ) || \defined (CONFIG_KASAN ) - void *object; -void *next = *head; -void *old_tail = *tail ? *tail : *head; + next = *head;/* Head and tail of the reconstructed freelist */ *head =NULL ;
@@ -2740,8 +2751,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,prefetch_freepointer (s, next_object);stat (s,ALLOC_FASTPATH ); } +/* +* If the object has been wiped upon free, make sure it's fully +* initialized by zeroing out freelist pointer. +*/ +if (slab_want_init_on_free (s)) + *(void **)object =0 ; -if (unlikely (gfpflags & __GFP_ZERO) && object) +if (unlikely (slab_want_init_on_alloc (gfpflags, s)) && object)memset (object,0 , s->object_size);slab_post_alloc_hook (s, gfpflags,1 , &object);
@@ -3163,7 +3180,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,local_irq_enable ();/* Clear memory outside IRQ disabled fastpath loop */ -if (unlikely (flags & __GFP_ZERO)) { +if (unlikely (slab_want_init_on_alloc (flags, s))) {int j;for (j =0 ; j < i; j++)
net/core/sock.c +1 -1
@@ -1602,7 +1602,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk =kmem_cache_alloc (slab, priority & ~__GFP_ZERO);if (!sk)return sk; -if (priority & __GFP_ZERO) +if (want_init_on_alloc (priority))sk_prot_clear_nulls (sk, prot->obj_size); }else sk =kmalloc (prot->obj_size, priority);
security/Kconfig.hardening +14 -0
@@ -159,6 +159,20 @@ config STACKLEAK_RUNTIME_DISABLE runtime to control kernel stack erasing for kernels built with CONFIG_GCC_PLUGIN_STACKLEAK. +config INIT_ON_ALLOC_DEFAULT_ON + bool "Set init_on_alloc=1 by default" + help + Enable init_on_alloc=1 by default, making the kernel initialize every + page and heap allocation with zeroes. + init_on_alloc can be overridden via command line. + +config INIT_ON_FREE_DEFAULT_ON + bool "Set init_on_free=1 by default" + help + Enable init_on_free=1 by default, making the kernel initialize freed + pages and slab memory with zeroes. + init_on_free can be overridden via command line. + endmenu endmenu
Signature · 1 line
2.21.0.1020.gf2820cf01a-goog
Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 5 lines · Alexander Potapenko
Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
I wonder if the patch series should be reorganized to introduce __GFP_NO_AUTOINIT first, so that when the commit with benchmarks appears, we get the "final" numbers...
Quoted 4 lines
Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
I'm working on reproducing these benchmarks. I'd really like to narrow down the +24% number here. But it does
Quoted 2 lines
The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.
I think the use of static keys here is really great: this is available by default for anyone that wants to turn it on.
I'm thinking, given the configuable nature of this, it'd be worth adding a little more detail at boot time. I think maybe a separate patch could be added to describe the kernel's memory auto-initialization features, and add something like this to mm_init():
+void __init report_meminit(void)
+{
+ const char *stack;
+
+ if (IS_ENABLED(CONFIG_INIT_STACK_ALL))
+ stack = "all";
+ else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL))
+ stack = "byref_all";
+ else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF))
+ stack = "byref";
+ else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_USER))
+ stack = "__user";
+ else
+ stack = "off";
+
+ /* Report memory auto-initialization states for this boot. */
+ pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n",
+ stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off",
+ want_init_on_free() ? "on" : "off");
+}
To get a boot line like:
mem auto-init: stack:off, heap alloc:off, heap free:on
And one other thought I had was that in the init_on_free=1 case, there is a large pause at boot while memory is being cleared. I think it'd be handy to include a comment about that, just to keep people from being surprised:
init/main.c +2 -0
@@ -529,6 +529,8 @@ static void __init mm_init(void)* bigger than MAX_ORDER unless SPARSEMEM. */ page_ext_init_flatmem (); +if (want_init_on_free ()) +pr_info ("Clearing system memory ...\n" );mem_init ();kmem_cache_init ();pgtable_init ();
Beyond these thoughts, I think this series is in good shape.
Andrew (or anyone else) do you have any concerns about this?
Signature · 1 line
Kees Cook
Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
From: Kees Cook <[email protected]>
Date: Thu, May 16, 2019 at 6:20 PM
To: Alexander Potapenko
Cc: <[email protected]>, <[email protected]>, <[email protected]>, Masahiro Yamada, James Morris, Serge E. Hallyn, Nick Desaulniers, Kostya Serebryany, Dmitry Vyukov, Sandeep Patil, Laura Abbott, Randy Dunlap, Jann Horn, Mark Rutland, <[email protected]>, <[email protected]>
Quoted 11 lines · Alexander Potapenko
Quoted 5 lines
Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
I wonder if the patch series should be reorganized to introduce
__GFP_NO_AUTOINITfirst, so that when the commit with benchmarks appears, we get the "final" numbers...
Quoted 7 lines
Quoted 4 lines
Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
I'm working on reproducing these benchmarks. I'd really like to narrow down the +24% number here. But it does
I suspect the slowdown of init_on_free is bigger than that of PAX_SANITIZE_MEMORY, as we've set the goal to have fully zeroed memory at alloc time.
If we want a mode that only wipes the user data upon free() but doesn't eliminate all uninit memory, then we can make it faster.
Quoted 32 lines
Quoted 2 lines
The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.
I think the use of static keys here is really great: this is available by default for anyone that wants to turn it on.
I'm thinking, given the configuable nature of this, it'd be worth adding a little more detail at boot time. I think maybe a separate patch could be added to describe the kernel's memory auto-initialization features, and add something like this to
mm_init():(quoted diff) +20
-0+void __init report_meminit(void) +{ + const char *stack; + + if (IS_ENABLED(CONFIG_INIT_STACK_ALL)) + stack = "all"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)) + stack = "byref_all"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF)) + stack = "byref"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_USER)) + stack = "__user"; + else + stack = "off"; + + /* Report memory auto-initialization states for this boot. */ + pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n", + stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off", + want_init_on_free() ? "on" : "off"); +}
Quoted 3 lines
To get a boot line like:
mem auto-init: stack:off, heap alloc:off, heap free:on
For stack there's no binary on/off, as you can potentially build half of the kernel with stack instrumentation and another half without it. We could make the instrumentation insert a static global flag into each translation unit, but this won't give us any interesting info.
Quoted 24 lines
And one other thought I had was that in the init_on_free=1 case, there is a large pause at boot while memory is being cleared. I think it'd be handy to include a comment about that, just to keep people from being surprised:
init/main.c +2
-0@@ -529,6 +529,8 @@ static void __init mm_init(void)* bigger than MAX_ORDER unless SPARSEMEM. */ page_ext_init_flatmem (); +if (want_init_on_free ()) +pr_info ("Clearing system memory ...\n" );mem_init ();kmem_cache_init ();pgtable_init ();Beyond these thoughts, I think this series is in good shape.
Andrew (or anyone else) do you have any concerns about this?
--
Kees Cook
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 5 lines · Alexander Potapenko
I suspect the slowdown of
init_on_freeis bigger than that ofPAX_SANITIZE_MEMORY, as we've set the goal to have fully zeroed memory at alloc time.
If we want a mode that only wipes the user data uponfree()but doesn't eliminate all uninit memory, then we can make it faster.
Yeah, I sent a separate email that discusses this a bit more.
I think the goals of init_on_alloc and init_on_free are likely a bit different. Given init_on_alloc's much more cache-friendly performance, I think that it's likely the right way forward for getting to fully zeroed memory at alloc time. (Though I note that it already includes exclusions:
such tradeoffs won't be unique to init_on_free.)
init_on_free appears to give us similar coverage (but also reduces the lifetime of unused data), but isn't cache-friendly so it looks to need a lot more tuning/trade-offs. (Not that we shouldn't include it! It'll just need a bit more care to be reasonable.)
Quoted 21 lines
Quoted 21 lines
(quoted diff) +20
-0+void __init report_meminit(void) +{ + const char *stack; + + if (IS_ENABLED(CONFIG_INIT_STACK_ALL)) + stack = "all"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL)) + stack = "byref_all"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF)) + stack = "byref"; + else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_USER)) + stack = "__user"; + else + stack = "off"; + + /* Report memory auto-initialization states for this boot. */ + pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n", + stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off", + want_init_on_free() ? "on" : "off"); +}
Quoted 7 lines
Quoted 3 lines
To get a boot line like:
mem auto-init: stack:off, heap alloc:off, heap free:on
For stack there's no binary on/off, as you can potentially build half of the kernel with stack instrumentation and another half without it. We could make the instrumentation insert a static global flag into each translation unit, but this won't give us any interesting info.
Well, yes, that's technically true, but I think reporting the kernel config here would make sense. If someone intentionally bypasses the stack auto-init for portions of the kernel, we can't meaningfully report it here. There will be exceptions for stack auto-init and heap auto-init.
Signature · 1 line
Kees Cook
Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
In reply to Alexander Potapenkomm/slab.h static inline int cache_random_seq_create(struct kmem_cache *cachep, · line 538
Quoted 11 lines · Alexander Potapenko
[...]
mm/slab.h +4
-0@@ -524,4 +524,20 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep, +static inline bool slab_want_init_on_free (struct kmem_cache *c) +{ +if (static_branch_unlikely (&init_on_free)) +return !(c->ctor);
BTW, why is this checking for c->ctor here? Shouldn't it not matter for the free case?
Quoted 3 lines
mm/slab.h +3
-0+else +return false; +}
Signature · 1 line
Kees Cook
Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 16 lines · Kees Cook
mm/slab.h static inline int cache_random_seq_create(struct kmem_cache *cachep, · line 538
Quoted 11 lines · Alexander Potapenko
[...]
mm/slab.h +4
-0@@ -524,4 +524,20 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep, +static inline bool slab_want_init_on_free (struct kmem_cache *c) +{ +if (static_branch_unlikely (&init_on_free)) +return !(c->ctor);BTW, why is this checking for
c->ctorhere? Shouldn't it not matter for the free case?
It does matter, see e.g. the handling of __OBJECT_POISON in slub.c If we just return true here, the kernel crashes.
Quoted 6 lines
mm/slab.h static inline int cache_random_seq_create(struct kmem_cache *cachep, · line 541
Quoted 3 lines
mm/slab.h +3
-0+else +return false; +}--
Kees Cook
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
In reply to Alexander PotapenkoQuoted 11 lines · Alexander Potapenko
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed.init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Why do we need both? The later is more robust because even free memory cannot be sniffed and the overhead might be shifted from the allocation context (e.g. to RCU) but why cannot we stick to a single model?
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 10 lines · Michal Hocko
Quoted 8 lines · Alexander Potapenko
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed.
Quoted 7 lines
Quoted 3 lines
init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Why do we need both? The later is more robust because even free memory cannot be sniffed and the overhead might be shifted from the allocation context (e.g. to RCU) but why cannot we stick to a single model?
init_on_free appears to be slower because of cache effects. It's several % in the best case vs. <1% for init_on_alloc.
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 15 lines · Alexander Potapenko
Quoted 14 lines · Michal Hocko
Quoted 11 lines · Alexander Potapenko
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed.init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Quoted 5 lines
Quoted 3 lines
Why do we need both? The later is more robust because even free memory cannot be sniffed and the overhead might be shifted from the allocation context (e.g. to RCU) but why cannot we stick to a single model?
init_on_freeappears to be slower because of cache effects. It's several % in the best case vs. <1% forinit_on_alloc.
This doesn't really explain why we need both.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 16 lines · Michal Hocko
Quoted 15 lines · Alexander Potapenko
Quoted 14 lines · Michal Hocko
Quoted 11 lines · Alexander Potapenko
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed.init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Why do we need both? The later is more robust because even free memory cannot be sniffed and the overhead might be shifted from the allocation context (e.g. to RCU) but why cannot we stick to a single model?
init_on_freeappears to be slower because of cache effects. It's several % in the best case vs. <1% forinit_on_alloc.This doesn't really explain why we need both.
There are a couple reasons. The first is that once we have hardware with memory tagging (e.g. arm64's MTE) we'll need both on_alloc and on_free hooks to do change the tags. With MTE, zeroing comes for "free" with tagging (though tagging is as slow as zeroing, so it's really the tagging that is free...), so we'll need to re-use the init_on_free infrastructure.
The second reason is for very paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructre for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series.
All that said, init_on_alloc looks desirable enough that distros will likely build with it enabled by default (I hope), and the very paranoid users will switch to (or additionally enable) init_on_free for their systems.
Signature · 1 line
Kees Cook
Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 23 lines · Kees Cook
Quoted 22 lines · Michal Hocko
Quoted 20 lines · Alexander Potapenko
Quoted 10 lines · Michal Hocko
Quoted 8 lines · Alexander Potapenko
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed.Quoted 7 lines
Quoted 3 lines
init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Why do we need both? The later is more robust because even free memory cannot be sniffed and the overhead might be shifted from the allocation context (e.g. to RCU) but why cannot we stick to a single model?
init_on_freeappears to be slower because of cache effects. It's several % in the best case vs. <1% forinit_on_alloc.
Quoted 7 lines
Quoted 1 line
This doesn't really explain why we need both.
There are a couple reasons. The first is that once we have hardware with memory tagging (e.g. arm64's MTE) we'll need both
on_allocandon_freehooks to do change the tags. With MTE, zeroing comes for "free" with tagging (though tagging is as slow as zeroing, so it's really the tagging that is free...), so we'll need to re-use theinit_on_freeinfrastructure.
I am not sure I follow, but ...
Quoted 8 lines
The second reason is for very paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructre for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series.
Quoted 4 lines
All that said,
init_on_alloclooks desirable enough that distros will likely build with it enabled by default (I hope), and the very paranoid users will switch to (or additionally enable)init_on_freefor their systems.
... this should all be part of the changelog.
Signature · 2 lines
Michal Hocko SUSE Labs
PATCH v2 2/4 lib: introduce test_meminit module
In reply to Alexander PotapenkoAdd tests for heap and pagealloc initialization.
These can be used to check init_on_alloc and init_on_free implementations as well as other approaches to initialization.
To: Kees Cook <[email protected]>
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
--- lib/Kconfig.debug | 8 ++ lib/Makefile | 1 + lib/test_meminit.c | 205 +++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 214 insertions(+) create mode 100644 lib/test_meminit.c
lib/Kconfig.debug +8 -0
@@ -2020,6 +2020,14 @@ config TEST_STACKINIT If unsure, say N. +config TEST_MEMINIT + tristate "Test level of heap/page initialization" + help + Test if the kernel is zero-initializing heap and page allocations. + This can be useful to test init_on_alloc and init_on_free features. + + If unsure, say N. + endif # RUNTIME_TESTING_MENU config MEMTEST
lib/Makefile +1 -0
@@ -91,6 +91,7 @@ obj-$(CONFIG_TEST_DEBUG_VIRTUAL) += test_debug_virtual.o obj-$(CONFIG_TEST_MEMCAT_P) += test_memcat_p.o obj-$(CONFIG_TEST_OBJAGG) += test_objagg.o obj-$(CONFIG_TEST_STACKINIT) += test_stackinit.o +obj-$(CONFIG_TEST_MEMINIT) += test_meminit.o obj-$(CONFIG_TEST_LIVEPATCH) += livepatch/
lib/test_meminit.c new file +205 -0
205 lines not on this page · raw message
Signature · 1 line
2.21.0.1020.gf2820cf01a-goog
Re: [PATCH v2 2/4] lib: introduce test_meminit module
Quoted 3 lines · Alexander Potapenko
Add tests for heap and pagealloc initialization.
These can be used to checkinit_on_allocandinit_on_freeimplementations as well as other approaches to initialization.
This is nice! Easy way to test the results. It might be helpful to show here what to expect when loading this module:
with either init_on_alloc=1 or init_on_free=1, I happily see:
test_meminit: all 10 tests in test_pages passed
test_meminit: all 40 tests in test_kvmalloc passed
test_meminit: all 20 tests in test_kmemcache passed
test_meminit: all 70 tests passed!
and without:
test_meminit: test_pages failed 10 out of 10 times
test_meminit: test_kvmalloc failed 40 out of 40 times
test_meminit: test_kmemcache failed 10 out of 20 times
test_meminit: failures: 60 out of 70
Quoted 2 lines
Signed-off-by Alexander Potapenko <[email protected]>
note below...
lib/test_meminit.c
Quoted 10 lines
[...]
lib/test_meminit.c new file +2
-0@@ -0,0 +1,205 @@ +// SPDX-License-Identifier: GPL-2.0 +module_init (test_meminit_init);
I get a warning at build about missing the license:
WARNING: modpost: missing MODULE_LICENSE() in lib/test_meminit.o
So, following the SPDX line, just add:
MODULE_LICENSE("GPL");
Signature · 1 line
Kees Cook
Re: [PATCH v2 2/4] lib: introduce test_meminit module
Quoted 8 lines · Kees Cook
Quoted 3 lines · Alexander Potapenko
Add tests for heap and pagealloc initialization.
These can be used to checkinit_on_allocandinit_on_freeimplementations as well as other approaches to initialization.This is nice! Easy way to test the results. It might be helpful to show here what to expect when loading this module:
Do you want me to add the expected output to the patch description?
Quoted 36 lines
with either init_on_alloc=1 or init_on_free=1, I happily see:
test_meminit: all 10 tests intest_pagespassed
test_meminit: all 40 tests intest_kvmallocpassed
test_meminit: all 20 tests intest_kmemcachepassed
test_meminit: all 70 tests passed!and without:
test_meminit:test_pagesfailed 10 out of 10 times
test_meminit:test_kvmallocfailed 40 out of 40 times
test_meminit:test_kmemcachefailed 10 out of 20 times
test_meminit: failures: 60 out of 70Quoted 2 lines
Signed-off-by Alexander Potapenko <[email protected]>Reviewed-by Kees Cook <[email protected]>Tested-by Kees Cook <[email protected]>note below...
lib/test_meminit.c
Quoted 10 lines
[...]
lib/test_meminit.c new file +2
-0@@ -0,0 +1,205 @@ +// SPDX-License-Identifier: GPL-2.0 +module_init (test_meminit_init);I get a warning at build about missing the license:
Quoted 5 lines
WARNING: modpost: missing
MODULE_LICENSE()inlib/test_meminit.oSo, following the SPDX line, just add:
MODULE_LICENSE("GPL");
Will do, thanks!
Quoted 2 lines
--
Kees Cook
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v2 2/4] lib: introduce test_meminit module
Quoted 11 lines · Alexander Potapenko
Quoted 8 lines · Kees Cook
Quoted 3 lines · Alexander Potapenko
Add tests for heap and pagealloc initialization.
These can be used to checkinit_on_allocandinit_on_freeimplementations as well as other approaches to initialization.This is nice! Easy way to test the results. It might be helpful to show here what to expect when loading this module:
Do you want me to add the expected output to the patch description?
Yes, I think it's worth having, as a way to show people what to expect when running the test, without having to actually enable, build, and run it themselves.
Signature · 1 line
Kees Cook
PATCH v2 3/4 gfp: mm: introduce __GFP_NO_AUTOINIT
In reply to Alexander PotapenkoWhen passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NO_AUTOINIT tells it to not initialize the requested memory if the init_on_alloc boot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.
__GFP_NO_AUTOINIT doesn't affect init_on_free behavior, except for SLOB, where init_on_free implies init_on_alloc.
__GFP_NO_AUTOINIT basically defeats the hardening against information leaks provided by init_on_alloc, so one should use it with caution.
This patch also adds __GFP_NO_AUTOINIT to alloc_pages() calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.
Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +6.84% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.25% sys time (st.err 0.72%)
Linux build with -j12, init_on_free=1: +8.52% wall time (st.err 0.42%) Linux build with -j12, init_on_free=1: +24.31% sys time (st.err 0.47%) Linux build with -j12, init_on_alloc=1: -0.16% wall time (st.err 0.40%) Linux build with -j12, init_on_alloc=1: +1.24% sys time (st.err 0.39%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.
To: Andrew Morton <[email protected]>
To: Kees Cook <[email protected]>
To: Christoph Lameter <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: James Morris <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Souptick Joarder <[email protected]>
Cc: Matthew Wilcox <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v2:
- renamed __GFP_NOINIT to __GFP_NO_AUTOINIT, updated patch name/description
--- include/linux/gfp.h | 13 +++++++++---- include/linux/mm.h | 2 +- kernel/kexec_core.c | 3 ++- mm/slab.c | 2 +- mm/slob.c | 3 ++- mm/slub.c | 1 + 6 files changed, 16 insertions(+), 8 deletions(-)
include/linux/gfp.h +9 -4
@@ -44,6 +44,7 @@ struct vm_area_struct;#else #define ___GFP_NOLOCKDEP 0#endif +#define ___GFP_NO_AUTOINIT 0x1000000u/* If the above are modified, __GFP_BITS_SHIFT may need updating */ /*
@@ -208,16 +209,20 @@ struct vm_area_struct;* %__GFP_COMP address compound page metadata. * * %__GFP_ZERO returns a zeroed page on success. +* +* %__GFP_NO_AUTOINIT requests non-initialized memory from the underlying +* allocator. */ -#define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) -#define __GFP_COMP ((__force gfp_t)___GFP_COMP) -#define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) +#define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) +#define __GFP_COMP ((__force gfp_t)___GFP_COMP) +#define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) +#define __GFP_NO_AUTOINIT ((__force gfp_t)___GFP_NO_AUTOINIT)/* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)/* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (23 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (25)#define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))/**
include/linux/mm.h +1 -1
@@ -2618,7 +2618,7 @@ DECLARE_STATIC_KEY_FALSE(init_on_alloc);static inline bool want_init_on_alloc (gfp_t flags) {if (static_branch_unlikely (&init_on_alloc)) -return true; +return !(flags & __GFP_NO_AUTOINIT);return flags & __GFP_ZERO; }
kernel/kexec_core.c +2 -1
@@ -302,7 +302,8 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) {struct page *pages; - pages =alloc_pages (gfp_mask & ~__GFP_ZERO, order); + pages =alloc_pages ((gfp_mask & ~__GFP_ZERO) | __GFP_NO_AUTOINIT, + order);if (pages) {unsigned int count, i;
mm/slab.c +1 -1
@@ -1393,7 +1393,7 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,struct page *page;int nr_pages; - flags |= cachep->allocflags; + flags |= (cachep->allocflags | __GFP_NO_AUTOINIT); page =__alloc_pages_node (nodeid, flags, cachep->gfporder);if (!page) {
mm/slob.c +2 -1
@@ -192,6 +192,7 @@ static void *slob_new_pages(gfp_t gfp, int order, int node) {void *page; + gfp |= __GFP_NO_AUTOINIT;#ifdef CONFIG_NUMA if (node !=NUMA_NO_NODE ) page =__alloc_pages_node (node, gfp, order);
@@ -221,7 +222,7 @@ static inline bool slob_want_init_on_alloc(gfp_t flags, struct kmem_cache *c) {if (static_branch_unlikely (&init_on_alloc) ||static_branch_unlikely (&init_on_free)) -return c ? (!c->ctor) : true; +return c ? (!c->ctor) : !(flags & __GFP_NO_AUTOINIT);return flags & __GFP_ZERO; }
mm/slub.c +1 -0
@@ -1495,6 +1495,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s,struct page *page;unsigned int order =oo_order (oo); + flags |= __GFP_NO_AUTOINIT;if (node ==NUMA_NO_NODE ) page =alloc_pages (flags, order);else
Signature · 1 line
2.21.0.1020.gf2820cf01a-goog
Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT
[It would be great to keep people involved in the previous version in the CC list]
Quoted 9 lines · Alexander Potapenko
When passed to an allocator (either pagealloc or SL[AOU]B),
__GFP_NO_AUTOINITtells it to not initialize the requested memory if theinit_on_allocboot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.
__GFP_NO_AUTOINITdoesn't affectinit_on_freebehavior, except for SLOB, whereinit_on_freeimpliesinit_on_alloc.
Quoted 6 lines
__GFP_NO_AUTOINITbasically defeats the hardening against information leaks provided byinit_on_alloc, so one should use it with caution.This patch also adds
__GFP_NO_AUTOINITtoalloc_pages()calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.
I still do not like the idea of a new gfp flag as explained in the previous email. People will simply use it incorectly or arbitrarily. We have that juicy experience from the past.
Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or GFP_ZERO) do not need it.
So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT
Quoted 3 lines · Michal this flag Hocko
[It would be great to keep people involved in the previous version in the CC list]
Yes, I've been trying to keep everyone in the loop, but your email fell through the cracks.
Sorry about that.
Quoted 13 lines · Alexander Potapenko
Quoted 12 lines
When passed to an allocator (either pagealloc or SL[AOU]B),
__GFP_NO_AUTOINITtells it to not initialize the requested memory if theinit_on_allocboot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.
__GFP_NO_AUTOINITdoesn't affectinit_on_freebehavior, except for SLOB, whereinit_on_freeimpliesinit_on_alloc.
__GFP_NO_AUTOINITbasically defeats the hardening against information leaks provided byinit_on_alloc, so one should use it with caution.
Quoted 7 lines
Quoted 3 lines
This patch also adds
__GFP_NO_AUTOINITtoalloc_pages()calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.I still do not like the idea of a new gfp flag as explained in the previous email. People will simply use it incorectly or arbitrarily. We have that juicy experience from the past.
Just to preserve some context, here's the previous email:
https://patchwork.kernel.org/patch/10907595/
(plus the patch removing GFP_TEMPORARY for the curious ones:
https://lwn.net/Articles/729145/)
Quoted 2 lines
Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or
GFP_ZERO) do not need it.
Sorry, I didn't understand this piece. Could you please elaborate?
Quoted 4 lines
So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?
There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB. However the benchmark results provided in this and the previous patch don't show any noticeable difference - most certainly because the cost of initializing the page is amortized.
The second one was to fine-tune hackbench, for which the slowdown drops by a factor of 2.
But optimizing a mitigation for certain benchmarks is a questionable measure, so maybe we could really go without it.
Kees, what do you think?
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT
Quoted 8 lines · Alexander Potapenko
Quoted 3 lines · Michal this flag Hocko
[It would be great to keep people involved in the previous version in the CC list]
Yes, I've been trying to keep everyone in the loop, but your email fell through the cracks.
Sorry about that.
No problem
Quoted 17 lines
Quoted 17 lines · Alexander Potapenko
Quoted 15 lines
When passed to an allocator (either pagealloc or SL[AOU]B),
__GFP_NO_AUTOINITtells it to not initialize the requested memory if theinit_on_allocboot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.
__GFP_NO_AUTOINITdoesn't affectinit_on_freebehavior, except for SLOB, whereinit_on_freeimpliesinit_on_alloc.
__GFP_NO_AUTOINITbasically defeats the hardening against information leaks provided byinit_on_alloc, so one should use it with caution.This patch also adds
__GFP_NO_AUTOINITtoalloc_pages()calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.
Quoted 8 lines
Quoted 3 lines
I still do not like the idea of a new gfp flag as explained in the previous email. People will simply use it incorectly or arbitrarily. We have that juicy experience from the past.
Just to preserve some context, here's the previous email:
https://patchwork.kernel.org/patch/10907595/
(plus the patch removingGFP_TEMPORARYfor the curious ones:
https://lwn.net/Articles/729145/)
Not only. GFP_REPEAT being another one and probably others I cannot remember from the top of my head.
Quoted 3 lines
Quoted 2 lines
Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or
GFP_ZERO) do not need it.Sorry, I didn't understand this piece. Could you please elaborate?
The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed. GFP_ZERO should be self explanatory.
Quoted 7 lines
Quoted 4 lines
So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?
There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.
Could you elaborate please?
Quoted 8 lines
However the benchmark results provided in this and the previous patch don't show any noticeable difference - most certainly because the cost of initializing the page is amortized.
The second one was to fine-tune hackbench, for which the slowdown drops by a factor of 2.
But optimizing a mitigation for certain benchmarks is a questionable measure, so maybe we could really go without it.
Agreed. Over optimization based on an artificial workloads tend to be dubious IMHO.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT
Quoted 42 lines · Michal Hocko
Quoted 8 lines · Alexander Potapenko
Quoted 3 lines · Michal this flag Hocko
[It would be great to keep people involved in the previous version in the CC list]
Yes, I've been trying to keep everyone in the loop, but your email fell through the cracks.
Sorry about that.No problem
Quoted 17 lines
Quoted 17 lines · Alexander Potapenko
Quoted 15 lines
When passed to an allocator (either pagealloc or SL[AOU]B),
__GFP_NO_AUTOINITtells it to not initialize the requested memory if theinit_on_allocboot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.
__GFP_NO_AUTOINITdoesn't affectinit_on_freebehavior, except for SLOB, whereinit_on_freeimpliesinit_on_alloc.
__GFP_NO_AUTOINITbasically defeats the hardening against information leaks provided byinit_on_alloc, so one should use it with caution.This patch also adds
__GFP_NO_AUTOINITtoalloc_pages()calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.Quoted 8 lines
Quoted 3 lines
I still do not like the idea of a new gfp flag as explained in the previous email. People will simply use it incorectly or arbitrarily. We have that juicy experience from the past.
Just to preserve some context, here's the previous email:
https://patchwork.kernel.org/patch/10907595/
(plus the patch removingGFP_TEMPORARYfor the curious ones:
https://lwn.net/Articles/729145/)Not only.
GFP_REPEATbeing another one and probably others I cannot remember from the top of my head.
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or
GFP_ZERO) do not need it.Sorry, I didn't understand this piece. Could you please elaborate?
The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed.
GFP_ZEROshould be self explanatory.
Ah, I see. We already do that, see the want_init_on_alloc() implementation here: https://patchwork.kernel.org/patch/10943087/
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?
There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.
Could you elaborate please?
When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.
Quoted 15 lines
Quoted 8 lines
However the benchmark results provided in this and the previous patch don't show any noticeable difference - most certainly because the cost of initializing the page is amortized.
The second one was to fine-tune hackbench, for which the slowdown drops by a factor of 2.
But optimizing a mitigation for certain benchmarks is a questionable measure, so maybe we could really go without it.Agreed. Over optimization based on an artificial workloads tend to be dubious IMHO.
--
Michal Hocko
SUSE Labs
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT
Quoted 57 lines · Alexander Potapenko
Quoted 42 lines · Michal Hocko
Quoted 8 lines · Alexander Potapenko
Quoted 3 lines · Michal this flag Hocko
[It would be great to keep people involved in the previous version in the CC list]
Yes, I've been trying to keep everyone in the loop, but your email fell through the cracks.
Sorry about that.No problem
Quoted 17 lines
Quoted 17 lines · Alexander Potapenko
Quoted 15 lines
When passed to an allocator (either pagealloc or SL[AOU]B),
__GFP_NO_AUTOINITtells it to not initialize the requested memory if theinit_on_allocboot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.
__GFP_NO_AUTOINITdoesn't affectinit_on_freebehavior, except for SLOB, whereinit_on_freeimpliesinit_on_alloc.
__GFP_NO_AUTOINITbasically defeats the hardening against information leaks provided byinit_on_alloc, so one should use it with caution.This patch also adds
__GFP_NO_AUTOINITtoalloc_pages()calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.Quoted 8 lines
Quoted 3 lines
I still do not like the idea of a new gfp flag as explained in the previous email. People will simply use it incorectly or arbitrarily. We have that juicy experience from the past.
Just to preserve some context, here's the previous email:
https://patchwork.kernel.org/patch/10907595/
(plus the patch removingGFP_TEMPORARYfor the curious ones:
https://lwn.net/Articles/729145/)Not only.
GFP_REPEATbeing another one and probably others I cannot remember from the top of my head.Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or
GFP_ZERO) do not need it.Sorry, I didn't understand this piece. Could you please elaborate?
The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed.
GFP_ZEROshould be self explanatory.Ah, I see. We already do that, see the
want_init_on_alloc()implementation here: https://patchwork.kernel.org/patch/10943087/Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.
Could you elaborate please?
When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator
... when the feature is enabled ...
Quoted 5 lines
and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.
OK, I see what you mean now. Is there any way to special case the page allocation for this feature? E.g. your implementation tries to make this zeroying special but why cannot you simply do this
struct page *
____alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)
{
//current implementation
}struct page *
__alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)
{
if (your_feature_enabled)
gfp_mask |= __GFP_ZERO;
return ____alloc_pages_nodemask(gfp_mask, order, preferred_nid,
nodemask);
}and use ____alloc_pages_nodemask from the slab or other internal allocators?
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT
Quoted 53 lines · Michal Hocko
Quoted 14 lines · Alexander Potapenko
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or
GFP_ZERO) do not need it.Sorry, I didn't understand this piece. Could you please elaborate?
The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed.
GFP_ZEROshould be self explanatory.Ah, I see. We already do that, see the
want_init_on_alloc()implementation here: https://patchwork.kernel.org/patch/10943087/Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.
Could you elaborate please?
When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator
... when the feature is enabled ...
Quoted 5 lines
and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.OK, I see what you mean now. Is there any way to special case the page allocation for this feature? E.g. your implementation tries to make this zeroying special but why cannot you simply do this
struct page *
____alloc_pages_nodemask(gfp_tgfp_mask, unsigned int order, intpreferred_nid,
nodemask_t*nodemask){ //current implementation }struct page *
__alloc_pages_nodemask(gfp_tgfp_mask, unsigned int order, intpreferred_nid,
nodemask_t*nodemask){ if (your_feature_enabled) gfp_mask |= __GFP_ZERO; return ____alloc_pages_nodemask(gfp_mask, order, preferred_nid, nodemask); }
Quoted 2 lines
and use
____alloc_pages_nodemaskfrom the slab or other internal allocators?
If an additional allocator function is preferred over a new GFP flag, then I don't see any reason not to do this. (Though adding more "__"s seems a bit unfriendly to code-documentation.) What might be better naming?
This would mean that the skb changes later in the series would use the "no auto init" version of the allocator too, then.
Signature · 1 line
Kees Cook
Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT
Quoted 54 lines · Kees Cook
Quoted 53 lines · Michal Hocko
Quoted 14 lines · Alexander Potapenko
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or
GFP_ZERO) do not need it.Sorry, I didn't understand this piece. Could you please elaborate?
The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed.
GFP_ZEROshould be self explanatory.Ah, I see. We already do that, see the
want_init_on_alloc()implementation here: https://patchwork.kernel.org/patch/10943087/Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.
Could you elaborate please?
When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator
... when the feature is enabled ...
Quoted 5 lines
and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.OK, I see what you mean now. Is there any way to special case the page allocation for this feature? E.g. your implementation tries to make this zeroying special but why cannot you simply do this
struct page *
____alloc_pages_nodemask(gfp_tgfp_mask, unsigned int order, intpreferred_nid,
nodemask_t*nodemask){ //current implementation }struct page *
__alloc_pages_nodemask(gfp_tgfp_mask, unsigned int order, intpreferred_nid,
nodemask_t*nodemask){ if (your_feature_enabled) gfp_mask |= __GFP_ZERO; return ____alloc_pages_nodemask(gfp_mask, order, preferred_nid, nodemask); }
Quoted 6 lines
Quoted 2 lines
and use
____alloc_pages_nodemaskfrom the slab or other internal allocators?If an additional allocator function is preferred over a new GFP flag, then I don't see any reason not to do this. (Though adding more "__"s seems a bit unfriendly to code-documentation.) What might be better naming?
The naminig is the last thing I would be worried about. Let's focus on the most simplistic implementation first. And means, can we really make it as simple as above? At least on the page allocator level.
Quoted 2 lines
This would mean that the skb changes later in the series would use the "no auto init" version of the allocator too, then.
No, this would be an internal function to MM. I would really like to optimize once there are numbers from _real_ workloads to base those optimizations.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT
Quoted 56 lines · Michal Hocko
Quoted 54 lines · Kees Cook
Quoted 53 lines · Michal Hocko
Quoted 14 lines · Alexander Potapenko
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or
GFP_ZERO) do not need it.Sorry, I didn't understand this piece. Could you please elaborate?
The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed.
GFP_ZEROshould be self explanatory.Ah, I see. We already do that, see the
want_init_on_alloc()implementation here: https://patchwork.kernel.org/patch/10943087/Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.
Could you elaborate please?
When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator
... when the feature is enabled ...
Quoted 5 lines
and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.OK, I see what you mean now. Is there any way to special case the page allocation for this feature? E.g. your implementation tries to make this zeroying special but why cannot you simply do this
struct page *
____alloc_pages_nodemask(gfp_tgfp_mask, unsigned int order, intpreferred_nid,
nodemask_t*nodemask){ //current implementation }struct page *
__alloc_pages_nodemask(gfp_tgfp_mask, unsigned int order, intpreferred_nid,
nodemask_t*nodemask){ if (your_feature_enabled) gfp_mask |= __GFP_ZERO; return ____alloc_pages_nodemask(gfp_mask, order, preferred_nid, nodemask); }
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
and use
____alloc_pages_nodemaskfrom the slab or other internal allocators?
Given that calling alloc_pages() with __GFP_NO_AUTOINIT doesn't visibly improve the chosen benchmarks,
and the next patch in the series ("net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations") only improves hackbench, shall we maybe drop both patches altogether?
Quoted 17 lines
Quoted 3 lines
If an additional allocator function is preferred over a new GFP flag, then I don't see any reason not to do this. (Though adding more "__"s seems a bit unfriendly to code-documentation.) What might be better naming?
The naminig is the last thing I would be worried about. Let's focus on the most simplistic implementation first. And means, can we really make it as simple as above? At least on the page allocator level.
Quoted 2 lines
This would mean that the skb changes later in the series would use the "no auto init" version of the allocator too, then.
No, this would be an internal function to MM. I would really like to optimize once there are numbers from _real_ workloads to base those optimizations.
--
Michal Hocko
SUSE Labs
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT
Quoted 57 lines · Alexander Potapenko
Quoted 56 lines · Michal Hocko
Quoted 54 lines · Kees Cook
Quoted 53 lines · Michal Hocko
Quoted 14 lines · Alexander Potapenko
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or
GFP_ZERO) do not need it.Sorry, I didn't understand this piece. Could you please elaborate?
The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed.
GFP_ZEROshould be self explanatory.Ah, I see. We already do that, see the
want_init_on_alloc()implementation here: https://patchwork.kernel.org/patch/10943087/Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.
Could you elaborate please?
When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator
... when the feature is enabled ...
Quoted 5 lines
and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.OK, I see what you mean now. Is there any way to special case the page allocation for this feature? E.g. your implementation tries to make this zeroying special but why cannot you simply do this
struct page *
____alloc_pages_nodemask(gfp_tgfp_mask, unsigned int order, intpreferred_nid,
nodemask_t*nodemask){ //current implementation }struct page *
__alloc_pages_nodemask(gfp_tgfp_mask, unsigned int order, intpreferred_nid,
nodemask_t*nodemask){ if (your_feature_enabled) gfp_mask |= __GFP_ZERO; return ____alloc_pages_nodemask(gfp_mask, order, preferred_nid, nodemask); }
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
and use
____alloc_pages_nodemaskfrom the slab or other internal allocators?Given that calling
alloc_pages()with__GFP_NO_AUTOINITdoesn't visibly improve the chosen benchmarks,
and the next patch in the series ("net: apply__GFP_NO_AUTOINITtoAF_UNIXsk_buffallocations") only improves hackbench, shall we maybe drop both patches altogether?
Ohh, by all means. I was suggesting the same few emails ago. The above is just a hint on how to implement the feature on the page allocator level rather than hooking into the prep_new_page and add another branch to zero memory.
Signature · 2 lines
Michal Hocko SUSE Labs
PATCH v2 4/4 net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations
In reply to Alexander PotapenkoAdd sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in the init_on_alloc mode from 6.84% to 3.45%.
Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)
Linux build with -j12, init_on_free=1: +8.34% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.13% sys time (st.err 0.47%) Linux build with -j12, init_on_alloc=1: -0.04% wall time (st.err 0.46%) Linux build with -j12, init_on_alloc=1: +0.50% sys time (st.err 0.45%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
To: Kees Cook <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: James Morris <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v2:
- changed __GFP_NOINIT to __GFP_NO_AUTOINIT
--- include/net/sock.h | 5 +++++ net/core/sock.c | 29 +++++++++++++++++++++++++---- net/unix/af_unix.c | 13 +++++++------ 3 files changed, 37 insertions(+), 10 deletions(-)
include/net/sock.h +5 -0
@@ -1626,6 +1626,11 @@ struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size,struct sk_buff *sock_alloc_send_pskb (struct sock *sk,unsigned long header_len,unsigned long data_len,int noblock,int *errcode,int max_page_order); +struct sk_buff *sock_alloc_send_pskb_noinit (struct sock *sk, +unsigned long header_len, +unsigned long data_len, +int noblock,int *errcode, +int max_page_order);void *sock_kmalloc (struct sock *sk,int size,gfp_t priority);void sock_kfree_s (struct sock *sk,void *mem,int size);void sock_kzfree_s (struct sock *sk,void *mem,int size);
net/core/sock.c +25 -4
@@ -2192,9 +2192,11 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo)* Generic send/receive buffer handlers */ -struct sk_buff *sock_alloc_send_pskb (struct sock *sk,unsigned long header_len, -unsigned long data_len,int noblock, -int *errcode,int max_page_order) +struct sk_buff *sock_alloc_send_pskb_internal (struct sock *sk, +unsigned long header_len, +unsigned long data_len, +int noblock,int *errcode, +int max_page_order,gfp_t gfp) {struct sk_buff *skb;long timeo;
@@ -2223,7 +2225,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, timeo =sock_wait_for_wmem (sk, timeo); } skb =alloc_skb_with_frags (header_len, data_len, max_page_order, - errcode, sk->sk_allocation); + errcode, sk->sk_allocation | gfp);if (skb)skb_set_owner_w (skb, sk);return skb;
@@ -2234,8 +2236,27 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, *errcode = err;return NULL ; } + +struct sk_buff *sock_alloc_send_pskb (struct sock *sk,unsigned long header_len, +unsigned long data_len,int noblock, +int *errcode,int max_page_order) +{ +return sock_alloc_send_pskb_internal (sk, header_len, data_len, + noblock, errcode, max_page_order,/*gfp*/ 0 ); +}EXPORT_SYMBOL (sock_alloc_send_pskb); +struct sk_buff *sock_alloc_send_pskb_noinit (struct sock *sk, +unsigned long header_len, +unsigned long data_len, +int noblock,int *errcode, +int max_page_order) +{ +return sock_alloc_send_pskb_internal (sk, header_len, data_len, + noblock, errcode, max_page_order,/*gfp*/ __GFP_NO_AUTOINIT); +} +EXPORT_SYMBOL (sock_alloc_send_pskb_noinit); +struct sk_buff *sock_alloc_send_skb (struct sock *sk,unsigned long size,int noblock,int *errcode) {
net/unix/af_unix.c +7 -6
@@ -1627,9 +1627,9 @@ static int unix_dgram_sendmsg(struct socket *sock, struct msghdr *msg,BUILD_BUG_ON (SKB_MAX_ALLOC <PAGE_SIZE ); } - skb =sock_alloc_send_pskb (sk, len - data_len, data_len, - msg->msg_flags &MSG_DONTWAIT , &err, -PAGE_ALLOC_COSTLY_ORDER ); + skb =sock_alloc_send_pskb_noinit (sk, len - data_len, data_len, + msg->msg_flags &MSG_DONTWAIT , &err, +PAGE_ALLOC_COSTLY_ORDER );if (skb ==NULL ) goto out;
@@ -1824,9 +1824,10 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, data_len =min_t (size_t, size,PAGE_ALIGN (data_len)); - skb =sock_alloc_send_pskb (sk, size - data_len, data_len, - msg->msg_flags &MSG_DONTWAIT , &err, -get_order (UNIX_SKB_FRAGS_SZ )); + skb =sock_alloc_send_pskb_noinit (sk, size - data_len, data_len, + msg->msg_flags &MSG_DONTWAIT , + &err, +get_order (UNIX_SKB_FRAGS_SZ ));if (!skb) goto out_err;
Signature · 1 line
2.21.0.1020.gf2820cf01a-goog
Re: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations
Quoted 4 lines · Alexander Potapenko
Add
sock_alloc_send_pskb_noinit(), which is similar tosock_alloc_send_pskb(), but allocates with__GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in theinit_on_allocmode from 6.84% to 3.45%.
Out of curiosity, why the creation of the new function over adding a gfp flag argument to sock_alloc_send_pskb() and updating callers? (There are only 6 callers, and this change already updates 2 of those.)
Quoted 5 lines
Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)
In the commit log it might be worth mentioning that this is only
changing the init_on_alloc case (in case it's not already obvious to
folks). Perhaps there needs to be a split of __GFP_NO_AUTOINIT into__GFP_NO_AUTO_ALLOC_INIT and __GFP_NO_AUTO_FREE_INIT? Right now __GFP_NO_AUTOINIT is only checked for init_on_alloc:
static inline bool want_init_on_alloc(gfp_t flags)
{
if (static_branch_unlikely(&init_on_alloc))
return !(flags & __GFP_NO_AUTOINIT);
return flags & __GFP_ZERO;
}
...
static inline bool want_init_on_free(void)
{
return static_branch_unlikely(&init_on_free);
}On a related note, it might be nice to add an exclusion list to the kmem_cache_create() cases, since it seems likely that further tuning will be needed there. For example, with the init_on_free-similar PAX_MEMORY_SANITIZE changes in the last public release of PaX/grsecurity, the following were excluded from wipe-on-free:
buffer_head
names_cache
mm_struct
vm_area_struct
anon_vma
anon_vma_chain
skbuff_head_cache
skbuff_fclone_cache
Adding these and others (with details on why they were selected), might improve init_on_free performance further without trading too much coverage.
Having a kernel param with a comma-separated list of cache names and the logic to add __GFP_NO_AUTOINIT at creation time would be a nice (and cheap!) debug feature to let folks tune things for their specific workloads, if they choose to. (And it could maybe also know what "none" meant, to actually remove the built-in exclusions, similar to what PaX's "pax_sanitize_slab=full" does.)
Signature · 1 line
Kees Cook
Re: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations
Quoted 15 lines · Kees Cook
Quoted 4 lines · Alexander Potapenko
Add
sock_alloc_send_pskb_noinit(), which is similar tosock_alloc_send_pskb(), but allocates with__GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in theinit_on_allocmode from 6.84% to 3.45%.Out of curiosity, why the creation of the new function over adding a gfp flag argument to
sock_alloc_send_pskb()and updating callers? (There are only 6 callers, and this change already updates 2 of those.)Quoted 5 lines
Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)
So I've run some of my own wall-clock timings of kernel builds (which should be an pretty big "worst case" situation, and I see much smaller performance changes:
everything off
Run times: 289.18 288.61 289.66 287.71 287.67 Min: 287.67 Max: 289.66 Mean: 288.57 Std Dev: 0.79 baseline
init_on_alloc=1
Run times: 289.72 286.95 287.87 287.34 287.35 Min: 286.95 Max: 289.72 Mean: 287.85 Std Dev: 0.98 0.25% faster (within the std dev noise)
init_on_free=1
Run times: 303.26 301.44 301.19 301.55 301.39 Min: 301.19 Max: 303.26 Mean: 301.77 Std Dev: 0.75 4.57% slower
init_on_free=1 with the PAX_MEMORY_SANITIZE slabs excluded:
Run times: 299.19 299.85 298.95 298.23 298.64 Min: 298.23 Max: 299.85 Mean: 298.97 Std Dev: 0.55 3.60% slower
So the tuning certainly improved things by 1%. My perf numbers don't show the 24% hit you were seeing at all, though.
Quoted 5 lines
In the commit log it might be worth mentioning that this is only changing the
init_on_alloccase (in case it's not already obvious to folks). Perhaps there needs to be a split of__GFP_NO_AUTOINITinto__GFP_NO_AUTO_ALLOC_INITand__GFP_NO_AUTO_FREE_INIT? Right now__GFP_NO_AUTOINITis only checked forinit_on_alloc:
I was obviously crazy here. :) GFP isn't present for free(), but a SLAB flag works (as was done in PAX_MEMORY_SANITIZE). I'll send the patch I used for the above timing test.
Signature · 1 line
Kees Cook
Re: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations
Quoted 15 lines · Kees Cook
Quoted 13 lines · Kees Cook
Quoted 4 lines · Alexander Potapenko
Add
sock_alloc_send_pskb_noinit(), which is similar tosock_alloc_send_pskb(), but allocates with__GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in theinit_on_allocmode from 6.84% to 3.45%.Out of curiosity, why the creation of the new function over adding a gfp flag argument to
sock_alloc_send_pskb()and updating callers? (There are only 6 callers, and this change already updates 2 of those.)Quoted 3 lines
Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)
So I've run some of my own wall-clock timings of kernel builds (which should be an pretty big "worst case" situation, and I see much smaller performance changes:
How many cores were you using? I suspect the numbers may vary a bit depending on that.
Quoted 15 lines
everything off
Run times: 289.18 288.61 289.66 287.71 287.67 Min: 287.67 Max: 289.66 Mean: 288.57 Std Dev: 0.79 baselineinit_on_alloc=1
Run times: 289.72 286.95 287.87 287.34 287.35 Min: 286.95 Max: 289.72 Mean: 287.85 Std Dev: 0.98 0.25% faster (within the std dev noise)init_on_free=1
Run times: 303.26 301.44 301.19 301.55 301.39 Min: 301.19 Max: 303.26 Mean: 301.77 Std Dev: 0.75 4.57% slower
Quoted 7 lines
init_on_free=1 with the
PAX_MEMORY_SANITIZEslabs excluded:Run times: 299.19 299.85 298.95 298.23 298.64 Min: 298.23 Max: 299.85 Mean: 298.97 Std Dev: 0.55 3.60% slowerSo the tuning certainly improved things by 1%. My perf numbers don't show the 24% hit you were seeing at all, though.
Note that 24% is the _sys_ time slowdown. The wall time slowdown seen in this case was 8.34%
Quoted 12 lines
Quoted 5 lines
In the commit log it might be worth mentioning that this is only changing the
init_on_alloccase (in case it's not already obvious to folks). Perhaps there needs to be a split of__GFP_NO_AUTOINITinto__GFP_NO_AUTO_ALLOC_INITand__GFP_NO_AUTO_FREE_INIT? Right now__GFP_NO_AUTOINITis only checked forinit_on_alloc:I was obviously crazy here. :) GFP isn't present for
free(), but a SLAB flag works (as was done inPAX_MEMORY_SANITIZE). I'll send the patch I used for the above timing test.--
Kees Cook
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations
Quoted 45 lines · Alexander Potapenko
Quoted 15 lines · Kees Cook
Quoted 13 lines · Kees Cook
Quoted 4 lines · Alexander Potapenko
Add
sock_alloc_send_pskb_noinit(), which is similar tosock_alloc_send_pskb(), but allocates with__GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in theinit_on_allocmode from 6.84% to 3.45%.Out of curiosity, why the creation of the new function over adding a gfp flag argument to
sock_alloc_send_pskb()and updating callers? (There are only 6 callers, and this change already updates 2 of those.)Quoted 3 lines
Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)
So I've run some of my own wall-clock timings of kernel builds (which should be an pretty big "worst case" situation, and I see much smaller performance changes:
How many cores were you using? I suspect the numbers may vary a bit depending on that.
Quoted 20 lines
everything off
Run times: 289.18 288.61 289.66 287.71 287.67 Min: 287.67 Max: 289.66 Mean: 288.57 Std Dev: 0.79 baselineinit_on_alloc=1
Run times: 289.72 286.95 287.87 287.34 287.35 Min: 286.95 Max: 289.72 Mean: 287.85 Std Dev: 0.98 0.25% faster (within the std dev noise)init_on_free=1
Run times: 303.26 301.44 301.19 301.55 301.39 Min: 301.19 Max: 303.26 Mean: 301.77 Std Dev: 0.75 4.57% slowerinit_on_free=1 with the
PAX_MEMORY_SANITIZEslabs excluded:Run times: 299.19 299.85 298.95 298.23 298.64 Min: 298.23 Max: 299.85 Mean: 298.97 Std Dev: 0.55 3.60% slower
Quoted 4 lines
Quoted 2 lines
So the tuning certainly improved things by 1%. My perf numbers don't show the 24% hit you were seeing at all, though.
Note that 24% is the _sys_ time slowdown. The wall time slowdown seen in this case was 8.34%
I've collected more stats running QEMU with different numbers of cores. The slowdown values of init_on_free compared to baseline are:
2 CPUs - 5.94% for wall time (20.08% for sys time)
6 CPUs - 7.43% for wall time (23.55% for sys time)
12 CPUs - 8.41% for wall time (24.25% for sys time)
24 CPUs - 9.49% for wall time (17.98% for sys time)
I'm building a defconfig of some fixed KMSAN tree with Clang, but that shouldn't matter much.
Quoted 26 lines
Quoted 6 lines
Quoted 5 lines
In the commit log it might be worth mentioning that this is only changing the
init_on_alloccase (in case it's not already obvious to folks). Perhaps there needs to be a split of__GFP_NO_AUTOINITinto__GFP_NO_AUTO_ALLOC_INITand__GFP_NO_AUTO_FREE_INIT? Right now__GFP_NO_AUTOINITis only checked forinit_on_alloc:Quoted 6 lines
I was obviously crazy here. :) GFP isn't present for
free(), but a SLAB flag works (as was done inPAX_MEMORY_SANITIZE). I'll send the patch I used for the above timing test.--
Kees Cook--
Alexander Potapenko
Software EngineerGoogle Germany GmbH
Erika-Mann-Straße, 33
80636 MünchenGeschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations
In reply to Alexander PotapenkoQuoted 15 lines · Alexander Potapenko
Quoted 14 lines · Kees Cook
Quoted 13 lines · Kees Cook
Quoted 4 lines · Alexander Potapenko
Add
sock_alloc_send_pskb_noinit(), which is similar tosock_alloc_send_pskb(), but allocates with__GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in theinit_on_allocmode from 6.84% to 3.45%.Out of curiosity, why the creation of the new function over adding a gfp flag argument to
sock_alloc_send_pskb()and updating callers? (There are only 6 callers, and this change already updates 2 of those.)Quoted 3 lines
Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)
So I've run some of my own wall-clock timings of kernel builds (which should be an pretty big "worst case" situation, and I see much smaller performance changes:
How many cores were you using? I suspect the numbers may vary a bit depending on that.
I was using 4.
Quoted 15 lines
Quoted 15 lines
init_on_alloc=1
Run times: 289.72 286.95 287.87 287.34 287.35 Min: 286.95 Max: 289.72 Mean: 287.85 Std Dev: 0.98 0.25% faster (within the std dev noise)init_on_free=1
Run times: 303.26 301.44 301.19 301.55 301.39 Min: 301.19 Max: 303.26 Mean: 301.77 Std Dev: 0.75 4.57% slowerinit_on_free=1 with the
PAX_MEMORY_SANITIZEslabs excluded:Run times: 299.19 299.85 298.95 298.23 298.64 Min: 298.23 Max: 299.85 Mean: 298.97 Std Dev: 0.55 3.60% slower
Quoted 4 lines
Quoted 2 lines
So the tuning certainly improved things by 1%. My perf numbers don't show the 24% hit you were seeing at all, though.
Note that 24% is the _sys_ time slowdown. The wall time slowdown seen in this case was 8.34%
Ah! Gotcha. Yeah, seems the impact for init_on_free is pretty variable. The init_on_alloc appears close to free, though.
Signature · 1 line
Kees Cook
[PATCH 5/4] mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches
In reply to Alexander PotapenkoIn order to improve the init_on_free performance, some frequently freed caches with less sensitive contents can be excluded from the init_on_free behavior.
This patch is modified from Brad Spengler/PaX Team's code in the last public patch of grsecurity/PaX based on my understanding of the code. Changes or omissions from the original code are mine and don't reflect the original grsecurity/PaX code.
--- fs/buffer.c | 2 +- fs/dcache.c | 3 ++- include/linux/slab.h | 3 +++ kernel/fork.c | 6 ++++-- mm/rmap.c | 5 +++-- mm/slab.h | 5 +++-- net/core/skbuff.c | 6 ++++-- 7 files changed, 20 insertions(+), 10 deletions(-)
fs/buffer.c +1 -1
@@ -3457,7 +3457,7 @@ void __init buffer_init(void) bh_cachep =kmem_cache_create ("buffer_head" ,sizeof (struct buffer_head ),0 , (SLAB_RECLAIM_ACCOUNT |SLAB_PANIC | -SLAB_MEM_SPREAD ), +SLAB_MEM_SPREAD |SLAB_NO_FREE_INIT ),NULL );/*
fs/dcache.c +2 -1
@@ -3139,7 +3139,8 @@ void __init vfs_caches_init_early(void)void __init vfs_caches_init (void ) { names_cachep =kmem_cache_create_usercopy ("names_cache" ,PATH_MAX ,0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC ,0 ,PATH_MAX ,NULL ); +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_NO_FREE_INIT ,0 , +PATH_MAX ,NULL );dcache_init ();inode_init ();
include/linux/slab.h +3 -0
@@ -92,6 +92,9 @@/* Avoid kmemleak tracing */ #define SLAB_NOLEAKTRACE ((slab_flags_t __force)0x00800000U) +/* Exclude slab from zero-on-free when init_on_free is enabled */ +#define SLAB_NO_FREE_INIT ((slab_flags_t __force)0x01000000U) +/* Fault injection mark */ #ifdef CONFIG_FAILSLAB # define SLAB_FAILSLAB ((slab_flags_t __force)0x02000000U)
kernel/fork.c +4 -2
@@ -2550,11 +2550,13 @@ void __init proc_caches_init(void) mm_cachep =kmem_cache_create_usercopy ("mm_struct" , mm_size,ARCH_MIN_MMSTRUCT_ALIGN , -SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT , +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT , offsetof(struct mm_struct , saved_auxv), sizeof_field(struct mm_struct, saved_auxv),NULL ); - vm_area_cachep =KMEM_CACHE (vm_area_struct,SLAB_PANIC |SLAB_ACCOUNT ); + vm_area_cachep =KMEM_CACHE (vm_area_struct,SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT );mmap_init ();nsproxy_cache_init (); }
mm/rmap.c +3 -2
@@ -432,10 +432,11 @@ static void anon_vma_ctor(void *data)void __init anon_vma_init (void ) { anon_vma_cachep =kmem_cache_create ("anon_vma" ,sizeof (struct anon_vma ), -0 ,SLAB_TYPESAFE_BY_RCU |SLAB_PANIC |SLAB_ACCOUNT , +0 ,SLAB_TYPESAFE_BY_RCU |SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT , anon_vma_ctor); anon_vma_chain_cachep =KMEM_CACHE (anon_vma_chain, -SLAB_PANIC |SLAB_ACCOUNT ); +SLAB_PANIC |SLAB_ACCOUNT |SLAB_NO_FREE_INIT ); }/*
mm/slab.h +3 -2
@@ -129,7 +129,8 @@ static inline slab_flags_t kmem_cache_flags(unsigned int object_size,/* Legal flag mask for kmem_cache_create(), for various configurations */ #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \ SLAB_CACHE_DMA32 | SLAB_PANIC | \ - SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS ) + SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \ + SLAB_NO_FREE_INIT)#if defined(CONFIG_DEBUG_SLAB )#define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER)
@@ -535,7 +536,7 @@ static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c)static inline bool slab_want_init_on_free (struct kmem_cache *c) {if (static_branch_unlikely (&init_on_free)) -return !(c->ctor); +return !(c->ctor) && ((c->flags &SLAB_NO_FREE_INIT ) ==0 );else return false; }
net/core/skbuff.c +4 -2
@@ -3981,14 +3981,16 @@ void __init skb_init(void) skbuff_head_cache =kmem_cache_create_usercopy ("skbuff_head_cache" ,sizeof (struct sk_buff ),0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC , +SLAB_HWCACHE_ALIGN |SLAB_PANIC | +SLAB_NO_FREE_INIT , offsetof(struct sk_buff , cb), sizeof_field(struct sk_buff, cb),NULL ); skbuff_fclone_cache =kmem_cache_create ("skbuff_fclone_cache" ,sizeof (struct sk_buff_fclones ),0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC , +SLAB_HWCACHE_ALIGN |SLAB_PANIC | +SLAB_NO_FREE_INIT ,NULL );skb_extensions_init (); }
Signature · 5 lines
2.17.1 -- Kees Cook
Re: [PATCH 5/4] mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches
Quoted 4 lines · Kees Cook
In order to improve the
init_on_freeperformance, some frequently freed caches with less sensitive contents can be excluded from theinit_on_freebehavior.
Did you see any notable performance improvement with this patch? A similar one gave me only 1-2% on the parallel Linux build.
Quoted 147 lines
This patch is modified from Brad Spengler/PaX Team's code in the last public patch of grsecurity/PaX based on my understanding of the code. Changes or omissions from the original code are mine and don't reflect the original grsecurity/PaX code.
Signed-off-by Kees Cook <[email protected]>--- fs/buffer.c | 2 +- fs/dcache.c | 3 ++- include/linux/slab.h | 3 +++ kernel/fork.c | 6 ++++-- mm/rmap.c | 5 +++-- mm/slab.h | 5 +++-- net/core/skbuff.c | 6 ++++-- 7 files changed, 20 insertions(+), 10 deletions(-)fs/buffer.c +1
-1@@ -3457,7 +3457,7 @@ void __init buffer_init(void) bh_cachep =kmem_cache_create ("buffer_head" ,sizeof (struct buffer_head ),0 , (SLAB_RECLAIM_ACCOUNT |SLAB_PANIC | -SLAB_MEM_SPREAD ), +SLAB_MEM_SPREAD |SLAB_NO_FREE_INIT ),NULL );/* fs/dcache.c +2
-1@@ -3139,7 +3139,8 @@ void __init vfs_caches_init_early(void)void __init vfs_caches_init (void ) { names_cachep =kmem_cache_create_usercopy ("names_cache" ,PATH_MAX ,0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC ,0 ,PATH_MAX ,NULL ); +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_NO_FREE_INIT ,0 , +PATH_MAX ,NULL );dcache_init ();inode_init ();include/linux/slab.h +3
-0@@ -92,6 +92,9 @@/* Avoid kmemleak tracing */ #define SLAB_NOLEAKTRACE ((slab_flags_t __force)0x00800000U) +/* Exclude slab from zero-on-free when init_on_free is enabled */ +#define SLAB_NO_FREE_INIT ((slab_flags_t __force)0x01000000U) +/* Fault injection mark */ #ifdef CONFIG_FAILSLAB # define SLAB_FAILSLAB ((slab_flags_t __force)0x02000000U)kernel/fork.c +4
-2@@ -2550,11 +2550,13 @@ void __init proc_caches_init(void) mm_cachep =kmem_cache_create_usercopy ("mm_struct" , mm_size,ARCH_MIN_MMSTRUCT_ALIGN , -SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT , +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT , offsetof(struct mm_struct , saved_auxv), sizeof_field(struct mm_struct, saved_auxv),NULL ); - vm_area_cachep =KMEM_CACHE (vm_area_struct,SLAB_PANIC |SLAB_ACCOUNT ); + vm_area_cachep =KMEM_CACHE (vm_area_struct,SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT );mmap_init ();nsproxy_cache_init (); }mm/rmap.c +3
-2@@ -432,10 +432,11 @@ static void anon_vma_ctor(void *data)void __init anon_vma_init (void ) { anon_vma_cachep =kmem_cache_create ("anon_vma" ,sizeof (struct anon_vma ), -0 ,SLAB_TYPESAFE_BY_RCU |SLAB_PANIC |SLAB_ACCOUNT , +0 ,SLAB_TYPESAFE_BY_RCU |SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT , anon_vma_ctor); anon_vma_chain_cachep =KMEM_CACHE (anon_vma_chain, -SLAB_PANIC |SLAB_ACCOUNT ); +SLAB_PANIC |SLAB_ACCOUNT |SLAB_NO_FREE_INIT ); }/* mm/slab.h +3
-2@@ -129,7 +129,8 @@ static inline slab_flags_t kmem_cache_flags(unsigned int object_size,/* Legal flag mask for kmem_cache_create(), for various configurations */ #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \ SLAB_CACHE_DMA32 | SLAB_PANIC | \ - SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS ) + SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \ + SLAB_NO_FREE_INIT)#if defined(CONFIG_DEBUG_SLAB )#define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER)@@ -535,7 +536,7 @@ static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c)static inline bool slab_want_init_on_free (struct kmem_cache *c) {if (static_branch_unlikely (&init_on_free)) -return !(c->ctor); +return !(c->ctor) && ((c->flags &SLAB_NO_FREE_INIT ) ==0 );else return false; }net/core/skbuff.c +4
-3@@ -3981,14 +3981,16 @@ void __init skb_init(void) skbuff_head_cache =kmem_cache_create_usercopy ("skbuff_head_cache" ,sizeof (struct sk_buff ),0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC , +SLAB_HWCACHE_ALIGN |SLAB_PANIC | +SLAB_NO_FREE_INIT , offsetof(struct sk_buff , cb), sizeof_field(struct sk_buff, cb),NULL ); skbuff_fclone_cache =kmem_cache_create ("skbuff_fclone_cache" ,sizeof (struct sk_buff_fclones ),0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC , +SLAB_HWCACHE_ALIGN |SLAB_PANIC | +SLAB_NO_FREE_INIT ,NULL );skb_extensions_init (); } --2.17.1
--
Kees Cook
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH 5/4] mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches
Quoted 7 lines · Alexander Potapenko
Quoted 4 lines · Kees Cook
In order to improve the
init_on_freeperformance, some frequently freed caches with less sensitive contents can be excluded from theinit_on_freebehavior.Did you see any notable performance improvement with this patch? A similar one gave me only 1-2% on the parallel Linux build.
Yup, that's in the other thread. I saw similar. But 1-2% on a 5% hit is a lot. ;)
Signature · 1 line
Kees Cook
Re: [PATCH 5/4] mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches
In reply to Kees CookHi Kees,
Quoted 8 lines · Kees Cook
In order to improve the
init_on_freeperformance, some frequently freed caches with less sensitive contents can be excluded from theinit_on_freebehavior.This patch is modified from Brad Spengler/PaX Team's code in the last public patch of grsecurity/PaX based on my understanding of the code. Changes or omissions from the original code are mine and don't reflect the original grsecurity/PaX code.
you might want to give secunet credit for this one, as can be seen here:
https://github.com/minipli/linux-grsec/commit/309d494e7a3f6533ca68fc8b3bd89fa76fd2c2df
However, please keep the "Changes or omissions from the original code..." part as your version slightly differs.
Thanks,
Mathias
Quoted 143 lines
Signed-off-by Kees Cook <[email protected]>--- fs/buffer.c | 2 +- fs/dcache.c | 3 ++- include/linux/slab.h | 3 +++ kernel/fork.c | 6 ++++-- mm/rmap.c | 5 +++-- mm/slab.h | 5 +++-- net/core/skbuff.c | 6 ++++-- 7 files changed, 20 insertions(+), 10 deletions(-)fs/buffer.c +1
-1@@ -3457,7 +3457,7 @@ void __init buffer_init(void) bh_cachep =kmem_cache_create ("buffer_head" ,sizeof (struct buffer_head ),0 , (SLAB_RECLAIM_ACCOUNT |SLAB_PANIC | -SLAB_MEM_SPREAD ), +SLAB_MEM_SPREAD |SLAB_NO_FREE_INIT ),NULL );/* fs/dcache.c +2
-1@@ -3139,7 +3139,8 @@ void __init vfs_caches_init_early(void)void __init vfs_caches_init (void ) { names_cachep =kmem_cache_create_usercopy ("names_cache" ,PATH_MAX ,0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC ,0 ,PATH_MAX ,NULL ); +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_NO_FREE_INIT ,0 , +PATH_MAX ,NULL );dcache_init ();inode_init ();include/linux/slab.h +3
-0@@ -92,6 +92,9 @@/* Avoid kmemleak tracing */ #define SLAB_NOLEAKTRACE ((slab_flags_t __force)0x00800000U) +/* Exclude slab from zero-on-free when init_on_free is enabled */ +#define SLAB_NO_FREE_INIT ((slab_flags_t __force)0x01000000U) +/* Fault injection mark */ #ifdef CONFIG_FAILSLAB # define SLAB_FAILSLAB ((slab_flags_t __force)0x02000000U)kernel/fork.c +4
-2@@ -2550,11 +2550,13 @@ void __init proc_caches_init(void) mm_cachep =kmem_cache_create_usercopy ("mm_struct" , mm_size,ARCH_MIN_MMSTRUCT_ALIGN , -SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT , +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT , offsetof(struct mm_struct , saved_auxv), sizeof_field(struct mm_struct, saved_auxv),NULL ); - vm_area_cachep =KMEM_CACHE (vm_area_struct,SLAB_PANIC |SLAB_ACCOUNT ); + vm_area_cachep =KMEM_CACHE (vm_area_struct,SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT );mmap_init ();nsproxy_cache_init (); }mm/rmap.c +3
-2@@ -432,10 +432,11 @@ static void anon_vma_ctor(void *data)void __init anon_vma_init (void ) { anon_vma_cachep =kmem_cache_create ("anon_vma" ,sizeof (struct anon_vma ), -0 ,SLAB_TYPESAFE_BY_RCU |SLAB_PANIC |SLAB_ACCOUNT , +0 ,SLAB_TYPESAFE_BY_RCU |SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT , anon_vma_ctor); anon_vma_chain_cachep =KMEM_CACHE (anon_vma_chain, -SLAB_PANIC |SLAB_ACCOUNT ); +SLAB_PANIC |SLAB_ACCOUNT |SLAB_NO_FREE_INIT ); }/* mm/slab.h +3
-2@@ -129,7 +129,8 @@ static inline slab_flags_t kmem_cache_flags(unsigned int object_size,/* Legal flag mask for kmem_cache_create(), for various configurations */ #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \ SLAB_CACHE_DMA32 | SLAB_PANIC | \ - SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS ) + SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \ + SLAB_NO_FREE_INIT)#if defined(CONFIG_DEBUG_SLAB )#define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER)@@ -535,7 +536,7 @@ static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c)static inline bool slab_want_init_on_free (struct kmem_cache *c) {if (static_branch_unlikely (&init_on_free)) -return !(c->ctor); +return !(c->ctor) && ((c->flags &SLAB_NO_FREE_INIT ) ==0 );else return false; }net/core/skbuff.c +4
-3@@ -3981,14 +3981,16 @@ void __init skb_init(void) skbuff_head_cache =kmem_cache_create_usercopy ("skbuff_head_cache" ,sizeof (struct sk_buff ),0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC , +SLAB_HWCACHE_ALIGN |SLAB_PANIC | +SLAB_NO_FREE_INIT , offsetof(struct sk_buff , cb), sizeof_field(struct sk_buff, cb),NULL ); skbuff_fclone_cache =kmem_cache_create ("skbuff_fclone_cache" ,sizeof (struct sk_buff_fclones ),0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC , +SLAB_HWCACHE_ALIGN |SLAB_PANIC | +SLAB_NO_FREE_INIT ,NULL );skb_extensions_init (); } --2.17.1
--
Kees Cook
Re: [PATCH 5/4] mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches
Quoted 12 lines · Mathias Krause
Hi Kees,
Quoted 8 lines · Kees Cook
In order to improve the
init_on_freeperformance, some frequently freed caches with less sensitive contents can be excluded from theinit_on_freebehavior.This patch is modified from Brad Spengler/PaX Team's code in the last public patch of grsecurity/PaX based on my understanding of the code. Changes or omissions from the original code are mine and don't reflect the original grsecurity/PaX code.
Quoted 6 lines
you might want to give secunet credit for this one, as can be seen here:
https://github.com/minipli/linux-grsec/commit/309d494e7a3f6533ca68fc8b3bd89fa76fd2c2df
However, please keep the "Changes or omissions from the original code..." part as your version slightly differs.
Ah-ha! Thanks for finding the specific commit; I'll adjust attribution. Are you able to describe how you chose the various excluded kmem caches? (I assume it wasn't just "highest numbers in the stats reporting".) And why run the ctor after wipe? Doesn't that means you're just running the ctor again at the next allocation time?
Thanks!
-Kees
Quoted 147 lines
Thanks,
MathiasQuoted 143 lines
Signed-off-by Kees Cook <[email protected]>--- fs/buffer.c | 2 +- fs/dcache.c | 3 ++- include/linux/slab.h | 3 +++ kernel/fork.c | 6 ++++-- mm/rmap.c | 5 +++-- mm/slab.h | 5 +++-- net/core/skbuff.c | 6 ++++-- 7 files changed, 20 insertions(+), 10 deletions(-)fs/buffer.c +1
-1@@ -3457,7 +3457,7 @@ void __init buffer_init(void) bh_cachep =kmem_cache_create ("buffer_head" ,sizeof (struct buffer_head ),0 , (SLAB_RECLAIM_ACCOUNT |SLAB_PANIC | -SLAB_MEM_SPREAD ), +SLAB_MEM_SPREAD |SLAB_NO_FREE_INIT ),NULL );/* fs/dcache.c +2
-1@@ -3139,7 +3139,8 @@ void __init vfs_caches_init_early(void)void __init vfs_caches_init (void ) { names_cachep =kmem_cache_create_usercopy ("names_cache" ,PATH_MAX ,0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC ,0 ,PATH_MAX ,NULL ); +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_NO_FREE_INIT ,0 , +PATH_MAX ,NULL );dcache_init ();inode_init ();include/linux/slab.h +3
-0@@ -92,6 +92,9 @@/* Avoid kmemleak tracing */ #define SLAB_NOLEAKTRACE ((slab_flags_t __force)0x00800000U) +/* Exclude slab from zero-on-free when init_on_free is enabled */ +#define SLAB_NO_FREE_INIT ((slab_flags_t __force)0x01000000U) +/* Fault injection mark */ #ifdef CONFIG_FAILSLAB # define SLAB_FAILSLAB ((slab_flags_t __force)0x02000000U)kernel/fork.c +4
-2@@ -2550,11 +2550,13 @@ void __init proc_caches_init(void) mm_cachep =kmem_cache_create_usercopy ("mm_struct" , mm_size,ARCH_MIN_MMSTRUCT_ALIGN , -SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT , +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT , offsetof(struct mm_struct , saved_auxv), sizeof_field(struct mm_struct, saved_auxv),NULL ); - vm_area_cachep =KMEM_CACHE (vm_area_struct,SLAB_PANIC |SLAB_ACCOUNT ); + vm_area_cachep =KMEM_CACHE (vm_area_struct,SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT );mmap_init ();nsproxy_cache_init (); }mm/rmap.c +3
-2@@ -432,10 +432,11 @@ static void anon_vma_ctor(void *data)void __init anon_vma_init (void ) { anon_vma_cachep =kmem_cache_create ("anon_vma" ,sizeof (struct anon_vma ), -0 ,SLAB_TYPESAFE_BY_RCU |SLAB_PANIC |SLAB_ACCOUNT , +0 ,SLAB_TYPESAFE_BY_RCU |SLAB_PANIC |SLAB_ACCOUNT | +SLAB_NO_FREE_INIT , anon_vma_ctor); anon_vma_chain_cachep =KMEM_CACHE (anon_vma_chain, -SLAB_PANIC |SLAB_ACCOUNT ); +SLAB_PANIC |SLAB_ACCOUNT |SLAB_NO_FREE_INIT ); }/* mm/slab.h +3
-2@@ -129,7 +129,8 @@ static inline slab_flags_t kmem_cache_flags(unsigned int object_size,/* Legal flag mask for kmem_cache_create(), for various configurations */ #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \ SLAB_CACHE_DMA32 | SLAB_PANIC | \ - SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS ) + SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \ + SLAB_NO_FREE_INIT)#if defined(CONFIG_DEBUG_SLAB )#define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER)@@ -535,7 +536,7 @@ static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c)static inline bool slab_want_init_on_free (struct kmem_cache *c) {if (static_branch_unlikely (&init_on_free)) -return !(c->ctor); +return !(c->ctor) && ((c->flags &SLAB_NO_FREE_INIT ) ==0 );else return false; }net/core/skbuff.c +4
-3@@ -3981,14 +3981,16 @@ void __init skb_init(void) skbuff_head_cache =kmem_cache_create_usercopy ("skbuff_head_cache" ,sizeof (struct sk_buff ),0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC , +SLAB_HWCACHE_ALIGN |SLAB_PANIC | +SLAB_NO_FREE_INIT , offsetof(struct sk_buff , cb), sizeof_field(struct sk_buff, cb),NULL ); skbuff_fclone_cache =kmem_cache_create ("skbuff_fclone_cache" ,sizeof (struct sk_buff_fclones ),0 , -SLAB_HWCACHE_ALIGN |SLAB_PANIC , +SLAB_HWCACHE_ALIGN |SLAB_PANIC | +SLAB_NO_FREE_INIT ,NULL );skb_extensions_init (); } --2.17.1
--
Kees Cook
Signature · 1 line
Kees Cook