add init_on_alloc/init_on_free boot options
Subsystems (11)
v8 cover add init_on_alloc/init_on_free boot options
Provide init_on_alloc and init_on_free boot options.
These are aimed at preventing possible information leaks and making the control-flow bugs that depend on uninitialized values more deterministic.
Enabling either of the options guarantees that the memory returned by the page allocator and SL[AU]B is initialized with zeroes.
SLOB allocator isn't supported at the moment, as its emulation of kmem caches complicates handling of SLAB_TYPESAFE_BY_RCU caches correctly.
Enabling init_on_free also guarantees that pages and heap objects are initialized right after they're freed, so it won't be possible to access stale data by using a dangling pointer.
As suggested by Michal Hocko, right now we don't let the heap users to disable initialization for certain allocations. There's not enough evidence that doing so can speed up real-life cases, and introducing ways to opt-out may result in things going out of control.
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
To: Kees Cook <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: James Morris <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Marco Elver <[email protected]>
Cc: Qian Cai <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
Alexander Potapenko (2):
mm: security: introduce init_on_alloc=1 and init_on_free=1 boot
options
mm: init: report memory auto-initialization features at boot time
.../admin-guide/kernel-parameters.txt | 9 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 ++++++ init/main.c | 24 +++++++ mm/dmapool.c | 4 +- mm/page_alloc.c | 71 +++++++++++++++++-- mm/slab.c | 16 ++++- mm/slab.h | 19 +++++ mm/slub.c | 43 +++++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 29 +++++++++ 12 files changed, 204 insertions(+), 19 deletions(-)
---
v3: dropped __GFP_NO_AUTOINIT patches
v5: dropped support for SLOB allocator, handle SLAB_TYPESAFE_BY_RCU
v6: changed wording in boot-time message
v7: dropped the test_meminit.c patch (picked by Andrew Morton already),
minor wording changes
v8: fixes for interoperability with other heap debugging features
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
PATCH v8 1/2 mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
This is expected to be on-by-default on Android and Chrome OS. And it gives the opportunity for anyone else to use it under distros too via the boot args. (The init_on_free feature is regularly requested by folks where memory forensics is included in their threat models.)
init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed.
init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The two exceptions are slab caches with constructors and SLAB_TYPESAFE_BY_RCU flag. Those are never zero-initialized to preserve their semantics.
Both init_on_alloc and init_on_free default to zero, but those defaults can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_INIT_ON_FREE_DEFAULT_ON.
If either SLUB poisoning or page poisoning is enabled, we disable init_on_alloc and init_on_free so that initialization doesn't interfere with debugging.
Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.
The new features are also going to pave the way for hardware memory tagging (e.g. arm64's MTE), which will require both on_alloc and on_free hooks to set the tags for heap objects. With MTE, tagging will have the same cost as memory initialization.
Although init_on_free is rather costly, there are paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructure for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series.
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
To: Kees Cook <[email protected]>
---
v2:
- unconditionally initialize pages in kernel_init_free_pages()
- comment from Randy Dunlap: drop 'default false' lines from Kconfig.hardening
v3:
- don't call kernel_init_free_pages() from memblock_free_pages()
- adopted some Kees' comments for the patch description
v4:
- use NULL instead of 0 in slab_alloc_node() (found by kbuild test robot)
- don't write to NULL object in slab_alloc_node() (found by Android testing)
v5:
- adjusted documentation wording as suggested by Kees
- disable SLAB_POISON if auto-initialization is on
- don't wipe RCU cache allocations made without __GFP_ZERO
- dropped SLOB support
v7:
- rebase the patch, added the Acked-by: tag
v8:
- addressed comments by Michal Hocko: revert kernel/kexec_core.c and apply initialization in dma_pool_free()
- disable init_on_alloc/init_on_free if slab poisoning or page poisoning are enabled, as requested by Qian Cai
- skip the redzone when initializing a freed heap object, as requested by Qian Cai and Kees Cook
- use s->offset to address the freeptr (suggested by Kees Cook)
- updated the patch description, added Signed-off-by: tag
--- .../admin-guide/kernel-parameters.txt | 9 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 ++++++ mm/dmapool.c | 4 +- mm/page_alloc.c | 71 +++++++++++++++++-- mm/slab.c | 16 ++++- mm/slab.h | 19 +++++ mm/slub.c | 43 +++++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 29 ++++++++ 10 files changed, 199 insertions(+), 18 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +9 -0
@@ -1673,6 +1673,15 @@ initrd= [BOOT] Specify the location of the initial ramdisk + init_on_alloc= [MM] Fill newly allocated pages and heap objects with + zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. + + init_on_free= [MM] Fill freed pages and heap objects with zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. + init_pkru= [x86] Specify the default memory protection keys rights register contents for all processes. 0x55555554 by default (disallow access to all but pkey 0). Can
drivers/infiniband/core/uverbs_ioctl.c +1 -1
@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used =ALIGN (new_used,sizeof (*pbundle->internal_buffer)); -if (flags & __GFP_ZERO) +if (want_init_on_alloc (flags))memset (res,0 , size);return res; }
include/linux/mm.h +22 -0
@@ -2696,6 +2696,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { }#endif +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_alloc); +#else +DECLARE_STATIC_KEY_FALSE (init_on_alloc); +#endif +static inline bool want_init_on_alloc (gfp_t flags) +{ +if (static_branch_unlikely (&init_on_alloc)) +return true; +return flags & __GFP_ZERO; +} + +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_free); +#else +DECLARE_STATIC_KEY_FALSE (init_on_free); +#endif +static inline bool want_init_on_free (void ) +{ +return static_branch_unlikely (&init_on_free); +} +extern bool _debug_pagealloc_enabled;static inline bool debug_pagealloc_enabled (void )
mm/dmapool.c +3 -1
@@ -378,7 +378,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,#endif spin_unlock_irqrestore (&pool->lock, flags); -if (mem_flags & __GFP_ZERO) +if (want_init_on_alloc (mem_flags))memset (retval,0 , pool->size);return retval;
@@ -428,6 +428,8 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) } offset = vaddr - page->vaddr; +if (want_init_on_free ()) +memset (vaddr,0 , pool->size);#ifdef DMAPOOL_DEBUG if ((dma - page->dma) != offset) {spin_unlock_irqrestore (&pool->lock, flags);
mm/page_alloc.c +64 -7
103 lines not on this page · raw message
mm/slab.c +13 -3
@@ -1830,6 +1830,14 @@ static bool set_objfreelist_slab_cache(struct kmem_cache *cachep, cachep->num =0 ; +/* +* If slab auto-initialization on free is enabled, store the freelist +* off-slab, so that its contents don't end up in one of the allocated +* objects. +*/ +if (unlikely (slab_want_init_on_free (cachep))) +return false; +if (cachep->ctor || flags &SLAB_TYPESAFE_BY_RCU )return false;
@@ -3263,7 +3271,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,local_irq_restore (save_flags); ptr =cache_alloc_debugcheck_after (cachep, flags, ptr, caller); -if (unlikely (flags & __GFP_ZERO) && ptr) +if (unlikely (slab_want_init_on_alloc (flags, cachep)) && ptr)memset (ptr,0 , cachep->object_size);slab_post_alloc_hook (cachep, flags,1 , &ptr);
@@ -3320,7 +3328,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp =cache_alloc_debugcheck_after (cachep, flags, objp, caller);prefetchw (objp); -if (unlikely (flags & __GFP_ZERO) && objp) +if (unlikely (slab_want_init_on_alloc (flags, cachep)) && objp)memset (objp,0 , cachep->object_size);slab_post_alloc_hook (cachep, flags,1 , &objp);
@@ -3441,6 +3449,8 @@ void ___cache_free(struct kmem_cache *cachep, void *objp,struct array_cache *ac =cpu_cache_get (cachep);check_irq_off (); +if (unlikely (slab_want_init_on_free (cachep))) +memset (objp,0 , cachep->object_size);kmemleak_free_recursive (objp, cachep->flags); objp =cache_free_debugcheck (cachep, objp, caller);
@@ -3528,7 +3538,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,cache_alloc_debugcheck_after_bulk (s, flags, size, p, _RET_IP_);/* Clear memory outside IRQ disabled section */ -if (unlikely (flags & __GFP_ZERO)) +if (unlikely (slab_want_init_on_alloc (flags, s)))for (i =0 ; i < size; i++)memset (p[i],0 , s->object_size);
mm/slab.h +19 -0
@@ -524,4 +524,23 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep,static inline void cache_random_seq_destroy (struct kmem_cache *cachep) { }#endif /* CONFIG_SLAB_FREELIST_RANDOM */ +static inline bool slab_want_init_on_alloc (gfp_t flags,struct kmem_cache *c) +{ +if (static_branch_unlikely (&init_on_alloc)) { +if (c->ctor) +return false; +if (c->flags &SLAB_TYPESAFE_BY_RCU ) +return flags & __GFP_ZERO; +return true; + } +return flags & __GFP_ZERO; +} + +static inline bool slab_want_init_on_free (struct kmem_cache *c) +{ +if (static_branch_unlikely (&init_on_free)) +return !(c->ctor || (c->flags &SLAB_TYPESAFE_BY_RCU )); +return false; +} +#endif /* MM_SLAB_H */
mm/slub.c +38 -5
74 lines not on this page · raw message
net/core/sock.c +1 -1
@@ -1596,7 +1596,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk =kmem_cache_alloc (slab, priority & ~__GFP_ZERO);if (!sk)return sk; -if (priority & __GFP_ZERO) +if (want_init_on_alloc (priority))sk_prot_clear_nulls (sk, prot->obj_size); }else sk =kmalloc (prot->obj_size, priority);
security/Kconfig.hardening +29 -0
@@ -160,6 +160,35 @@ config STACKLEAK_RUNTIME_DISABLE runtime to control kernel stack erasing for kernels built with CONFIG_GCC_PLUGIN_STACKLEAK. +config INIT_ON_ALLOC_DEFAULT_ON + bool "Enable heap memory zeroing on allocation by default" + help + This has the effect of setting "init_on_alloc=1" on the kernel + command line. This can be disabled with "init_on_alloc=0". + When "init_on_alloc" is enabled, all page allocator and slab + allocator memory will be zeroed when allocated, eliminating + many kinds of "uninitialized heap memory" flaws, especially + heap content exposures. The performance impact varies by + workload, but most cases see <1% impact. Some synthetic + workloads have measured as high as 7%. + +config INIT_ON_FREE_DEFAULT_ON + bool "Enable heap memory zeroing on free by default" + help + This has the effect of setting "init_on_free=1" on the kernel + command line. This can be disabled with "init_on_free=0". + Similar to "init_on_alloc", when "init_on_free" is enabled, + all page allocator and slab allocator memory will be zeroed + when freed, eliminating many kinds of "uninitialized heap memory" + flaws, especially heap content exposures. The primary difference + with "init_on_free" is that data lifetime in memory is reduced, + as anything freed is wiped immediately, making live forensics or + cold boot memory attacks unable to recover freed memory contents. + The performance impact varies by workload, but is more expensive + than "init_on_alloc" due to the negative cache effects of + touching "cold" memory areas. Most cases see 3-5% impact. Some + synthetic workloads have measured as high as 8%. + endmenu endmenu
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
Re: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
On Wed 26-06-19 14:19:42, Alexander Potapenko wrote:
[...]
Quoted 4 lines
mm/dmapool.c +0
-0
[...]
mm/dmapool.c void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) · line 432
Quoted 6 lines
mm/dmapool.c +2
-0@@ -428,6 +428,8 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) } offset = vaddr - page->vaddr; +if (want_init_on_free ()) +memset (vaddr,0 , pool->size);
any reason why this is not in DMAPOOL_DEBUG else branch? Why would you want to both zero on free and poison on free?
Quoted 3 lines
mm/dmapool.c +0
-0#ifdef DMAPOOL_DEBUG if ((dma - page->dma) != offset) {spin_unlock_irqrestore (&pool->lock, flags);
[...]
mm/dmapool.c static __always_inline bool free_pages_prepare(struct page *page, · line 1204
Quoted 6 lines
mm/dmapool.c +2
-0@@ -1142,6 +1200,8 @@ static __always_inline bool free_pages_prepare(struct page *page, }arch_free_page (page, order);kernel_poison_pages (page,1 << order,0 ); +if (want_init_on_free ()) +kernel_init_free_pages (page,1 << order);
same here. If you don't want to make this exclusive then you have to zero before poisoning otherwise you are going to blow up on the poison check, right?
Quoted 3 lines
mm/dmapool.c +0
-0if (debug_pagealloc_enabled ())kernel_map_pages (page,1 << order,0 );
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 11 lines · Michal Hocko
On Wed 26-06-19 14:19:42, Alexander Potapenko wrote:
[...]Quoted 4 lines
mm/dmapool.c +0
-0[...]
Quoted 3 lines
mm/dmapool.c +0
-0@@ -428,6 +428,8 @@ void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) }
Quoted 6 lines
mm/dmapool.c void dma_pool_free(struct dma_pool *pool, void *vaddr, dma_addr_t dma) · line 432
Quoted 3 lines
mm/dmapool.c +2
-0offset = vaddr - page->vaddr; +if (want_init_on_free ()) +memset (vaddr,0 , pool->size);any reason why this is not in
DMAPOOL_DEBUGelse branch? Why would you want to both zero on free and poison on free?
This makes sense, thanks.
Quoted 6 lines
Quoted 3 lines
mm/dmapool.c +0
-0#ifdef DMAPOOL_DEBUG if ((dma - page->dma) != offset) {spin_unlock_irqrestore (&pool->lock, flags);[...]
Quoted 10 lines
mm/dmapool.c static __always_inline bool free_pages_prepare(struct page *page, · line 1204
Quoted 6 lines
mm/dmapool.c +2
-0@@ -1142,6 +1200,8 @@ static __always_inline bool free_pages_prepare(struct page *page, }arch_free_page (page, order);kernel_poison_pages (page,1 << order,0 ); +if (want_init_on_free ()) +kernel_init_free_pages (page,1 << order);same here. If you don't want to make this exclusive then you have to zero before poisoning otherwise you are going to blow up on the poison check, right?
Note that we disable initialization if page poisoning is on. As I mentioned on another thread we can eventually merge this code with page poisoning, but right now it's better to make the user decide which of the features they want instead of letting them guess how the combination of the two is going to work.
Quoted 6 lines
mm/dmapool.c static __always_inline bool free_pages_prepare(struct page *page, · line 1207
Quoted 3 lines
mm/dmapool.c +0
-0if (debug_pagealloc_enabled ())kernel_map_pages (page,1 << order,0 );--
Michal Hocko
SUSE Labs
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 1 line · Alexander Potapenko
On Wed, Jun 26, 2019 at 4:49 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 11 lines
Quoted 10 lines
mm/page_alloc.c static __always_inline bool free_pages_prepare(struct page *page, · line 1204
Quoted 6 lines
mm/page_alloc.c +2
-0@@ -1142,6 +1200,8 @@ static __always_inline bool free_pages_prepare(struct page *page, }arch_free_page (page, order);kernel_poison_pages (page,1 << order,0 ); +if (want_init_on_free ()) +kernel_init_free_pages (page,1 << order);same here. If you don't want to make this exclusive then you have to zero before poisoning otherwise you are going to blow up on the poison check, right?
Note that we disable initialization if page poisoning is on.
Ohh, right. Missed that in the init code.
Quoted 4 lines
As I mentioned on another thread we can eventually merge this code with page poisoning, but right now it's better to make the user decide which of the features they want instead of letting them guess how the combination of the two is going to work.
Strictly speaking zeroying is a subset of poisoning. If somebody asks for both the poisoning surely satisfies any data leak guarantees zeroying would give. So I am not sure we have to really make them exclusive wrt. to the configuraion. I will leave that to you but it would be better if the code didn't break subtly once the early init restriction is removed for one way or another. So either always make sure that zeroying is done _before_ poisoning or that you do not zero when poisoning. The later sounds the best wrt. the code quality from my POV.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 23 lines · Michal Hocko
Quoted 1 line · Alexander Potapenko
On Wed, Jun 26, 2019 at 4:49 PM Michal Hocko <[email protected]> wrote:
[...]
Quoted 11 lines
Quoted 10 lines
mm/page_alloc.c static __always_inline bool free_pages_prepare(struct page *page, · line 1204
Quoted 6 lines
mm/page_alloc.c +2
-0@@ -1142,6 +1200,8 @@ static __always_inline bool free_pages_prepare(struct page *page, }arch_free_page (page, order);kernel_poison_pages (page,1 << order,0 ); +if (want_init_on_free ()) +kernel_init_free_pages (page,1 << order);same here. If you don't want to make this exclusive then you have to zero before poisoning otherwise you are going to blow up on the poison check, right?
Note that we disable initialization if page poisoning is on.
Ohh, right. Missed that in the init code.
Quoted 4 lines
As I mentioned on another thread we can eventually merge this code with page poisoning, but right now it's better to make the user decide which of the features they want instead of letting them guess how the combination of the two is going to work.
Quoted 9 lines
Strictly speaking zeroying is a subset of poisoning. If somebody asks for both the poisoning surely satisfies any data leak guarantees zeroying would give. So I am not sure we have to really make them exclusive wrt. to the configuraion. I will leave that to you but it would be better if the code didn't break subtly once the early init restriction is removed for one way or another. So either always make sure that zeroying is done _before_ poisoning or that you do not zero when poisoning. The later sounds the best wrt. the code quality from my POV.
I somewhat liked the idea of always having zero-initialized page/heap memory if init_on_{alloc,free} is on.
But in production mode we won't have page or slab poisoning anyway, and for debugging this doesn't really matter much.
I've sent v9 with poisoning support added.
Quoted 3 lines
--
Michal Hocko
SUSE Labs
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
In reply to Alexander PotapenkoQuoted 191 lines · Alexander Potapenko
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
This is expected to be on-by-default on Android and Chrome OS. And it gives the opportunity for anyone else to use it under distros too via the boot args. (The
init_on_freefeature is regularly requested by folks where memory forensics is included in their threat models.)init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed.init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The two exceptions are slab caches with constructors and
SLAB_TYPESAFE_BY_RCUflag. Those are never zero-initialized to preserve their semantics.Both
init_on_allocandinit_on_freedefault to zero, but those defaults can be overridden withCONFIG_INIT_ON_ALLOC_DEFAULT_ONandCONFIG_INIT_ON_FREE_DEFAULT_ON.If either SLUB poisoning or page poisoning is enabled, we disable
init_on_allocandinit_on_freeso that initialization doesn't interfere with debugging.Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.
The new features are also going to pave the way for hardware memory tagging (e.g. arm64's MTE), which will require both
on_allocandon_freehooks to set the tags for heap objects. With MTE, tagging will have the same cost as memory initialization.Although
init_on_freeis rather costly, there are paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructure for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series.Signed-off-by Alexander Potapenko <[email protected]>Acked-by Kees Cook <[email protected]>To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
To: Kees Cook <[email protected]>Cc Masahiro Yamada <[email protected]>Cc Michal Hocko <[email protected]>Cc James Morris <[email protected]>Cc "Serge E. Hallyn" <[email protected]>Cc Nick Desaulniers <[email protected]>Cc Kostya Serebryany <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc Sandeep Patil <[email protected]>Cc Laura Abbott <[email protected]>Cc Randy Dunlap <[email protected]>Cc Jann Horn <[email protected]>Cc Mark Rutland <[email protected]>Cc Marco Elver <[email protected]>Cc Qian Cai <[email protected]>Signed-off-by Andrew Morton <[email protected]>---
v2:
- unconditionally initialize pages inkernel_init_free_pages()
- comment from Randy Dunlap: drop 'default false' lines fromKconfig.hardening
v3:
- don't callkernel_init_free_pages()frommemblock_free_pages()
- adopted some Kees' comments for the patch description
v4:
- use NULL instead of 0 inslab_alloc_node()(found by kbuild test robot)
- don't write to NULL object inslab_alloc_node()(found by Android testing)
v5:
- adjusted documentation wording as suggested by Kees
- disableSLAB_POISONif auto-initialization is on
- don't wipe RCU cache allocations made without__GFP_ZERO
- dropped SLOB support
v7:
- rebase the patch, added the Acked-by: tag
v8:
- addressed comments by Michal Hocko: revertkernel/kexec_core.cand apply initialization indma_pool_free()
- disable init_on_alloc/init_on_free if slab poisoning or page poisoning are enabled, as requested by Qian Cai
- skip the redzone when initializing a freed heap object, as requested by Qian Cai and Kees Cook
- uses->offsetto address the freeptr (suggested by Kees Cook)
- updated the patch description, added Signed-off-by: tag
---
.../admin-guide/kernel-parameters.txt | 9 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 ++++++ mm/dmapool.c | 4 +- mm/page_alloc.c | 71 +++++++++++++++++--mm/slab.c | 16 ++++- mm/slab.h | 19 +++++ mm/slub.c | 43 +++++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 29 ++++++++ 10 files changed, 199 insertions(+), 18 deletions(-)Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -1673,6 +1673,15 @@
initrd= [BOOT] Specify the location of the initial
ramdiskDocumentation/admin-guide/kernel-parameters.txt +1
-0+ init_on_alloc= [MM] Fill newly allocated pages and heapobjects with
Documentation/admin-guide/kernel-parameters.txt +5
-0+ zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. + + init_on_free= [MM] Fill freed pages and heap objects withzeroes.
Documentation/admin-guide/kernel-parameters.txt +3
-0+ Format: 0 | 1 + Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. +init_pkru= [x86] Specify the default memory protection keys
rights
register contents for all processes. 0x55555554 by default (disallow access to all but pkey 0). Candrivers/infiniband/core/uverbs_ioctl.c +0
-0@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used;
pbundle->internal_used=
ALIGN(new_used, sizeof(*pbundle->internal_buffer));drivers/infiniband/core/uverbs_ioctl.c +1
-1-if (flags & __GFP_ZERO) +if (want_init_on_alloc (flags))
memset(res, 0, size);
return res;
}include/linux/mm.h +0
-0@@ -2696,6 +2696,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { } #endifinclude/linux/mm.h +22
-0+#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_alloc); +#else +DECLARE_STATIC_KEY_FALSE (init_on_alloc); +#endif +static inline bool want_init_on_alloc (gfp_t flags) +{ +if (static_branch_unlikely (&init_on_alloc)) +return true; +return flags & __GFP_ZERO; +} + +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_free); +#else +DECLARE_STATIC_KEY_FALSE (init_on_free); +#endif +static inline bool want_init_on_free (void ) +{ +return static_branch_unlikely (&init_on_free); +} +
Quoted 3 lines
extern bool
_debug_pagealloc_enabled;
static inline booldebug_pagealloc_enabled(void)
Do those really necessary need to be static keys?
Adding either init_on_free=0 or init_on_alloc=0 to the kernel cmdline will generate a warning with kernels built with clang.
[ 0.000000] static_key_disable(): static key 'init_on_free+0x0/0x4' used before call to jump_label_init()
[ 0.000000] WARNING: CPU: 0 PID: 0 at ./include/linux/jump_label.h:317 early_init_on_free+0x1c0/0x200
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.2.0-rc6-next-20190626+ #9
[ 0.000000] pstate: 60000089 (nZCv daIf -PAN -UAO)
[ 0.000000] pc : early_init_on_free+0x1c0/0x200
[ 0.000000] lr : early_init_on_free+0x1c0/0x200
[ 0.000000] sp : ffff100012c07df0
[ 0.000000] x29: ffff100012c07e20 x28: ffff1000110a01ec
[ 0.000000] x27: 000000000000005f x26: ffff100011716cd0
[ 0.000000] x25: ffff100010d36166 x24: ffff100010d3615d
[ 0.000000] x23: ffff100010d364b5 x22: ffff1000117164a0
[ 0.000000] x21: 0000000000000000 x20: 0000000000000000
[ 0.000000] x19: 0000000000000000 x18: 000000000000002e
[ 0.000000] x17: 000000000000000f x16: 0000000000000040
[ 0.000000] x15: 0000000000000000 x14: 6c61632065726f66
[ 0.000000] x13: 6562206465737520 x12: 273478302f307830
[ 0.000000] x11: 0000000000000000 x10: 0000000000000000
[ 0.000000] x9 : 0000000000000000 x8 : 0000000000000000
[ 0.000000] x7 : 6d756a206f74206c x6 : ffff100014426625
[ 0.000000] x5 : ffff100012c07b28 x4 : 0000000000000007
[ 0.000000] x3 : ffff1000101aadf4 x2 : 0000000000000001
[ 0.000000] x1 : 0000000000000001 x0 : 000000000000005d
[ 0.000000] Call trace:
[ 0.000000] early_init_on_free+0x1c0/0x200
[ 0.000000] do_early_param+0xd0/0x104
[ 0.000000] parse_args+0x1f0/0x524
[ 0.000000] parse_early_param+0x70/0x8c
[ 0.000000] setup_arch+0xa8/0x268
[ 0.000000] start_kernel+0x80/0x560
Re: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 26 lines · Qian Cai
Quoted 9 lines · Alexander Potapenko
Both
init_on_allocandinit_on_freedefault to zero, but those defaults can be overridden withCONFIG_INIT_ON_ALLOC_DEFAULT_ONandCONFIG_INIT_ON_FREE_DEFAULT_ON.
[...]mm/page_alloc.c +5
-0+static int __init early_init_on_alloc (char *buf) +{ +int ret; +bool bool_result; +mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 168
Quoted 10 lines
mm/page_alloc.c +10
-0+if (!buf) +return -EINVAL ; + ret =kstrtobool (buf, &bool_result); +if (bool_result) +static_branch_enable (&init_on_alloc); +else +static_branch_disable (&init_on_alloc); +return ret; +} +early_param ("init_on_alloc" , early_init_on_alloc);Do those really necessary need to be static keys?
Adding either init_on_free=0 or init_on_alloc=0 to the kernel cmdline will generate a warning with kernels built with clang.
Quoted 8 lines
[ 0.000000]
static_key_disable(): static key 'init_on_free+0x0/0x4' used before call tojump_label_init()
[ 0.000000] WARNING: CPU: 0 PID: 0 at./include/linux/jump_label.h:317early_init_on_free+0x1c0/0x200
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.2.0-rc6-next-20190626+ #9
[ 0.000000] pstate: 60000089 (nZCv daIf -PAN -UAO)
I think the issue here is that arm64 doesn't initialize static keys early enough.
init/main.c has the general case:
asmlinkage __visible void __init start_kernel(void)
{
...
setup_arch(&command_line);
...
smp_prepare_boot_cpu();
...
/* parameters may set static keys */
jump_label_init();
parse_early_param();
...
}however, x86 does even earlier early params in setup_arch():
void __init setup_arch(char **cmdline_p)
{
...
jump_label_init();
...
parse_early_param();
...
}arm64 does similar very early early params in setup_arch()[1] too, but not jump_label_init() which is too late in smp_prepare_boot_cpu():
void __init setup_arch(char **cmdline_p)
{
...
parse_early_param();
...
}void __init smp_prepare_boot_cpu(void)
{
...
jump_label_init();
...
}I can send a patch to fix this...
-Kees
[1] since efd9e03facd07 ("arm64: Use static keys for CPU features")
Quoted 26 lines
[ 0.000000] pc :
early_init_on_free+0x1c0/0x200[ 0.000000] lr :early_init_on_free+0x1c0/0x200[ 0.000000] sp :ffff100012c07df0
[ 0.000000] x29:ffff100012c07e20x28:ffff1000110a01ec[ 0.000000] x27:000000000000005fx26:ffff100011716cd0[ 0.000000] x25:ffff100010d36166x24:ffff100010d3615d[ 0.000000] x23:ffff100010d364b5x22:ffff1000117164a0[ 0.000000] x21: 0000000000000000 x20: 0000000000000000 [ 0.000000] x19: 0000000000000000 x18:000000000000002e[ 0.000000] x17:000000000000000fx16: 0000000000000040 [ 0.000000] x15: 0000000000000000 x14:6c61632065726f66[ 0.000000] x13: 6562206465737520 x12:273478302f307830[ 0.000000] x11: 0000000000000000 x10: 0000000000000000 [ 0.000000] x9 : 0000000000000000 x8 : 0000000000000000 [ 0.000000] x7 :6d756a206f74206cx6 :ffff100014426625[ 0.000000] x5 :ffff100012c07b28x4 : 0000000000000007 [ 0.000000] x3 :ffff1000101aadf4x2 : 0000000000000001 [ 0.000000] x1 : 0000000000000001 x0 :000000000000005d[ 0.000000] Call trace:
[ 0.000000] early_init_on_free+0x1c0/0x200
[ 0.000000] do_early_param+0xd0/0x104
[ 0.000000] parse_args+0x1f0/0x524
[ 0.000000] parse_early_param+0x70/0x8c
[ 0.000000] setup_arch+0xa8/0x268
[ 0.000000] start_kernel+0x80/0x560
Signature · 1 line
Kees Cook
Re: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 36 lines · Kees Cook
Quoted 26 lines · Qian Cai
Quoted 9 lines · Alexander Potapenko
Both
init_on_allocandinit_on_freedefault to zero, but those defaults can be overridden withCONFIG_INIT_ON_ALLOC_DEFAULT_ONandCONFIG_INIT_ON_FREE_DEFAULT_ON.
[...]mm/page_alloc.c +5
-0+static int __init early_init_on_alloc (char *buf) +{ +int ret; +bool bool_result; +mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 168
Quoted 10 lines
mm/page_alloc.c +10
-0+if (!buf) +return -EINVAL ; + ret =kstrtobool (buf, &bool_result); +if (bool_result) +static_branch_enable (&init_on_alloc); +else +static_branch_disable (&init_on_alloc); +return ret; +} +early_param ("init_on_alloc" , early_init_on_alloc);Do those really necessary need to be static keys?
Adding either init_on_free=0 or init_on_alloc=0 to the kernel cmdline will generate a warning with kernels built with clang.
Quoted 8 lines
[ 0.000000]
static_key_disable(): static key 'init_on_free+0x0/0x4' used before call tojump_label_init()
[ 0.000000] WARNING: CPU: 0 PID: 0 at./include/linux/jump_label.h:317early_init_on_free+0x1c0/0x200
[ 0.000000] Modules linked in:
[ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.2.0-rc6-next-20190626+ #9
[ 0.000000] pstate: 60000089 (nZCv daIf -PAN -UAO)
Quoted 2 lines
I think the issue here is that arm64 doesn't initialize static keys early enough.
This sounds familiar: http://lkml.kernel.org/r/CABXOdTd-cqHM_feAO1tvwn4Z=kM6WHKYAbDJ7LGfMvRPRPG7GA@mail.gmail.com
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 1 line · Michal Hocko
This sounds familiar: http://lkml.kernel.org/r/CABXOdTd-cqHM_feAO1tvwn4Z=kM6WHKYAbDJ7LGfMvRPRPG7GA@mail.gmail.com
Your memory is better than mine! I entirely forgot about this and it was only 2 months ago. Whoops. :P
Signature · 1 line
Kees Cook
Re: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
In reply to Qian CaiQuoted 195 lines · Qian Cai
Quoted 193 lines · Alexander Potapenko
The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
This is expected to be on-by-default on Android and Chrome OS. And it gives the opportunity for anyone else to use it under distros too via the boot args. (The
init_on_freefeature is regularly requested by folks where memory forensics is included in their threat models.)init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed.init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.
Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The two exceptions are slab caches with constructors and
SLAB_TYPESAFE_BY_RCUflag. Those are never zero-initialized to preserve their semantics.Both
init_on_allocandinit_on_freedefault to zero, but those defaults can be overridden withCONFIG_INIT_ON_ALLOC_DEFAULT_ONandCONFIG_INIT_ON_FREE_DEFAULT_ON.If either SLUB poisoning or page poisoning is enabled, we disable
init_on_allocandinit_on_freeso that initialization doesn't interfere with debugging.Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:
hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)
Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)
The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.
The new features are also going to pave the way for hardware memory tagging (e.g. arm64's MTE), which will require both
on_allocandon_freehooks to set the tags for heap objects. With MTE, tagging will have the same cost as memory initialization.Although
init_on_freeis rather costly, there are paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructure for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series.Signed-off-by Alexander Potapenko <[email protected]>Acked-by Kees Cook <[email protected]>To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
To: Kees Cook <[email protected]>Cc Masahiro Yamada <[email protected]>Cc Michal Hocko <[email protected]>Cc James Morris <[email protected]>Cc "Serge E. Hallyn" <[email protected]>Cc Nick Desaulniers <[email protected]>Cc Kostya Serebryany <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc Sandeep Patil <[email protected]>Cc Laura Abbott <[email protected]>Cc Randy Dunlap <[email protected]>Cc Jann Horn <[email protected]>Cc Mark Rutland <[email protected]>Cc Marco Elver <[email protected]>Cc Qian Cai <[email protected]>Signed-off-by Andrew Morton <[email protected]>---
v2:
- unconditionally initialize pages inkernel_init_free_pages()
- comment from Randy Dunlap: drop 'default false' lines fromKconfig.hardening
v3:
- don't callkernel_init_free_pages()frommemblock_free_pages()
- adopted some Kees' comments for the patch description
v4:
- use NULL instead of 0 inslab_alloc_node()(found by kbuild test robot)
- don't write to NULL object inslab_alloc_node()(found by Android testing)
v5:
- adjusted documentation wording as suggested by Kees
- disableSLAB_POISONif auto-initialization is on
- don't wipe RCU cache allocations made without__GFP_ZERO
- dropped SLOB support
v7:
- rebase the patch, added the Acked-by: tag
v8:
- addressed comments by Michal Hocko: revertkernel/kexec_core.cand apply initialization indma_pool_free()
- disable init_on_alloc/init_on_free if slab poisoning or page poisoning are enabled, as requested by Qian Cai
- skip the redzone when initializing a freed heap object, as requested by Qian Cai and Kees Cook
- uses->offsetto address the freeptr (suggested by Kees Cook)
- updated the patch description, added Signed-off-by: tag--- .../admin-guide/kernel-parameters.txt | 9 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 ++++++ mm/dmapool.c | 4 +- mm/page_alloc.c | 71 +++++++++++++++++-- mm/slab.c | 16 ++++- mm/slab.h | 19 +++++ mm/slub.c | 43 +++++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 29 ++++++++ 10 files changed, 199 insertions(+), 18 deletions(-)Documentation/admin-guide/kernel-parameters.txt +0
-0@@ -1673,6 +1673,15 @@ initrd= [BOOT] Specify the location of the initialramdisk
+ init_on_alloc= [MM] Fill newly allocated pages and heap
objects withDocumentation/admin-guide/kernel-parameters.txt +5
-0+ zeroes. + Format: 0 | 1 + Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON. + + init_on_free= [MM] Fill freed pages and heap objects withzeroes.
Documentation/admin-guide/kernel-parameters.txt +3
-0+ Format: 0 | 1 + Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON. + init_pkru= [x86] Specify the default memory protection keysrights
register contents for all processes.0x55555554by default (disallow access to all but pkey 0). Candrivers/infiniband/core/uverbs_ioctl.c +1
-1@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used =ALIGN (new_used,sizeof (*pbundle->internal_buffer)); -if (flags & __GFP_ZERO) +if (want_init_on_alloc (flags))memset (res,0 , size);return res;}
include/linux/mm.h +22
-0@@ -2696,6 +2696,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { }#endif +#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_alloc); +#else +DECLARE_STATIC_KEY_FALSE (init_on_alloc); +#endif +static inline bool want_init_on_alloc (gfp_t flags) +{ +if (static_branch_unlikely (&init_on_alloc)) +return true; +return flags & __GFP_ZERO; +} + +#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON +DECLARE_STATIC_KEY_TRUE (init_on_free); +#else +DECLARE_STATIC_KEY_FALSE (init_on_free); +#endif +static inline bool want_init_on_free (void ) +{ +return static_branch_unlikely (&init_on_free); +} +extern bool _debug_pagealloc_enabled;
Quoted 3 lines
Quoted 1 line
static inline bool
debug_pagealloc_enabled(void)Do those really necessary need to be static keys?
Yes. Initially they weren't, but using static branches saved us a few %%.
Quoted 2 lines
Adding either init_on_free=0 or init_on_alloc=0 to the kernel cmdline will generate a warning with kernels built with clang.
Looks like Kees has taken care of this issue in his "arm64: Move jump_label_init() before parse_early_param()"
Thanks Kees!
Quoted 34 lines
[ 0.000000] static_key_disable(): static key 'init_on_free+0x0/0x4' used before call to jump_label_init() [ 0.000000] WARNING: CPU: 0 PID: 0 at ./include/linux/jump_label.h:317 early_init_on_free+0x1c0/0x200 [ 0.000000] Modules linked in: [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 5.2.0-rc6-next-20190626+ #9 [ 0.000000] pstate: 60000089 (nZCv daIf -PAN -UAO) [ 0.000000] pc : early_init_on_free+0x1c0/0x200 [ 0.000000] lr : early_init_on_free+0x1c0/0x200 [ 0.000000] sp : ffff100012c07df0 [ 0.000000] x29: ffff100012c07e20 x28: ffff1000110a01ec [ 0.000000] x27: 000000000000005f x26: ffff100011716cd0 [ 0.000000] x25: ffff100010d36166 x24: ffff100010d3615d [ 0.000000] x23: ffff100010d364b5 x22: ffff1000117164a0 [ 0.000000] x21: 0000000000000000 x20: 0000000000000000 [ 0.000000] x19: 0000000000000000 x18: 000000000000002e [ 0.000000] x17: 000000000000000f x16: 0000000000000040 [ 0.000000] x15: 0000000000000000 x14: 6c61632065726f66 [ 0.000000] x13: 6562206465737520 x12: 273478302f307830 [ 0.000000] x11: 0000000000000000 x10: 0000000000000000 [ 0.000000] x9 : 0000000000000000 x8 : 0000000000000000 [ 0.000000] x7 : 6d756a206f74206c x6 : ffff100014426625 [ 0.000000] x5 : ffff100012c07b28 x4 : 0000000000000007 [ 0.000000] x3 : ffff1000101aadf4 x2 : 0000000000000001 [ 0.000000] x1 : 0000000000000001 x0 : 000000000000005d [ 0.000000] Call trace: [ 0.000000] early_init_on_free+0x1c0/0x200 [ 0.000000] do_early_param+0xd0/0x104 [ 0.000000] parse_args+0x1f0/0x524 [ 0.000000] parse_early_param+0x70/0x8c [ 0.000000] setup_arch+0xa8/0x268 [ 0.000000] start_kernel+0x80/0x560
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Andrew Morton
maintainerRe: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
In reply to Alexander PotapenkoQuoted 9 lines · On Wed, 26 Jun 2019 14:19:42 +0200 Alexander Potapenko <[email protected]>…
v8:
- addressed comments by Michal Hocko: revertkernel/kexec_core.cand apply initialization indma_pool_free()
- disable init_on_alloc/init_on_free if slab poisoning or page poisoning are enabled, as requested by Qian Cai
- skip the redzone when initializing a freed heap object, as requested by Qian Cai and Kees Cook
- uses->offsetto address the freeptr (suggested by Kees Cook)
- updated the patch description, added Signed-off-by: tag
v8 failed to incorporate
https://ozlabs.org/~akpm/mmots/broken-out/mm-security-introduce-init_on_alloc=1-and-init_on_free=1-boot-options-fix.patch
and
https://ozlabs.org/~akpm/mmots/broken-out/mm-security-introduce-init_on_alloc=1-and-init_on_free=1-boot-options-fix-2.patch
it's conventional to incorporate such fixes when preparing a new version of a patch.
Re: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
Quoted 13 lines · Andrew Morton
Quoted 9 lines · On Wed, 26 Jun 2019 14:19:42 +0200 Alexander Potapenko <[email protected]>…
v8:
- addressed comments by Michal Hocko: revertkernel/kexec_core.cand apply initialization indma_pool_free()
- disable init_on_alloc/init_on_free if slab poisoning or page poisoning are enabled, as requested by Qian Cai
- skip the redzone when initializing a freed heap object, as requested by Qian Cai and Kees Cook
- uses->offsetto address the freeptr (suggested by Kees Cook)
- updated the patch description, added Signed-off-by: tag
Quoted 9 lines
v8 failed to incorporate
https://ozlabs.org/~akpm/mmots/broken-out/mm-security-introduce-init_on_alloc=1-and-init_on_free=1-boot-options-fix.patch
and
https://ozlabs.org/~akpm/mmots/broken-out/mm-security-introduce-init_on_alloc=1-and-init_on_free=1-boot-options-fix-2.patchit's conventional to incorporate such fixes when preparing a new version of a patch.
Ah, sorry about that.
I'll probably send out v9 with proper poison handling and will pick those two patches as well.
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH v8 1/2] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options
In reply to Andrew MortonQuoted 13 lines · Andrew Morton
Quoted 9 lines · On Wed, 26 Jun 2019 14:19:42 +0200 Alexander Potapenko <[email protected]>…
v8:
- addressed comments by Michal Hocko: revertkernel/kexec_core.cand apply initialization indma_pool_free()
- disable init_on_alloc/init_on_free if slab poisoning or page poisoning are enabled, as requested by Qian Cai
- skip the redzone when initializing a freed heap object, as requested by Qian Cai and Kees Cook
- uses->offsetto address the freeptr (suggested by Kees Cook)
- updated the patch description, added Signed-off-by: tag
Quoted 9 lines
v8 failed to incorporate
https://ozlabs.org/~akpm/mmots/broken-out/mm-security-introduce-init_on_alloc=1-and-init_on_free=1-boot-options-fix.patch
and
https://ozlabs.org/~akpm/mmots/broken-out/mm-security-introduce-init_on_alloc=1-and-init_on_free=1-boot-options-fix-2.patchit's conventional to incorporate such fixes when preparing a new version of a patch.
v9 contains these patches (I've also exported init_on_free), so should now be fine to drop them.
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
PATCH v8 2/2 mm: init: report memory auto-initialization features at boot time
In reply to Alexander PotapenkoPrint the currently enabled stack and heap initialization modes.
Stack initialization is enabled by a config flag, while heap initialization is configured at boot time with defaults being set in the config. It's more convenient for the user to have all information about these hardening measures in one place at boot, so the user can reason about the expected behavior of the running system.
The possible options for stack are:
- "all" for CONFIG_INIT_STACK_ALL;
- "byref_all" for CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL;
- "byref" for CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF;
- "__user" for CONFIG_GCC_PLUGIN_STRUCTLEAK_USER;
- "off" otherwise.
Depending on the values of init_on_alloc and init_on_free boottime options we also report "heap alloc" and "heap free" as "on"/"off".
In the init_on_free mode initializing pages at boot time may take a while, so print a notice about that as well. This depends on how much memory is installed, the memory bandwidth, etc.
On a relatively modern x86 system, it takes about 0.75s/GB to wipe all memory:
[ 0.418722] mem auto-init: stack:byref_all, heap alloc:off, heap free:on [ 0.419765] mem auto-init: clearing system memory may take some time... [ 12.376605] Memory: 16408564K/16776672K available (14339K kernel code, 1397K rwdata, 3756K rodata, 1636K init, 11460K bss, 368108K reserved, 0K cma-reserved)
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
---
v6:
- update patch description, fixed message about clearing memory v7:
- rebase the patch, add the Acked-by: tag;
- more description updates as suggested by Kees;
- make report_meminit() static.
v8:
- added the Signed-off-by: tag
--- init/main.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
init/main.c +24 -0
@@ -520,6 +520,29 @@ static inline void initcall_debug_enable(void) }#endif +/* Report memory auto-initialization states for this boot. */ +static void __init report_meminit (void ) +{ +const char *stack; + +if (IS_ENABLED (CONFIG_INIT_STACK_ALL )) + stack ="all" ; +else if (IS_ENABLED (CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL )) + stack ="byref_all" ; +else if (IS_ENABLED (CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF )) + stack ="byref" ; +else if (IS_ENABLED (CONFIG_GCC_PLUGIN_STRUCTLEAK_USER )) + stack ="__user" ; +else + stack ="off" ; + +pr_info ("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n" , + stack,want_init_on_alloc (GFP_KERNEL ) ?"on" :"off" , +want_init_on_free () ?"on" :"off" ); +if (want_init_on_free ()) +pr_info ("mem auto-init: clearing system memory may take some time...\n" ); +} +/* * Set up kernel memory allocators */
@@ -530,6 +553,7 @@ static void __init mm_init(void)* bigger than MAX_ORDER unless SPARSEMEM. */ page_ext_init_flatmem (); +report_meminit ();mem_init ();kmem_cache_init ();pgtable_init ();
Signature · 1 line
2.22.0.410.gd8fdbe21b5-goog
Quoted 2 lines · Alexander Potapenko
Provide
init_on_allocandinit_on_freeboot options.
akpm: May I kindly ask you to replace the two patches from this series in the -mm tree with their newer versions?
Quoted 65 lines
These are aimed at preventing possible information leaks and making the control-flow bugs that depend on uninitialized values more deterministic.
Enabling either of the options guarantees that the memory returned by the page allocator and SL[AU]B is initialized with zeroes.
SLOB allocator isn't supported at the moment, as its emulation of kmem caches complicates handling ofSLAB_TYPESAFE_BY_RCUcaches correctly.Enabling
init_on_freealso guarantees that pages and heap objects are initialized right after they're freed, so it won't be possible to access stale data by using a dangling pointer.As suggested by Michal Hocko, right now we don't let the heap users to disable initialization for certain allocations. There's not enough evidence that doing so can speed up real-life cases, and introducing ways to opt-out may result in things going out of control.
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
To: Kees Cook <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: James Morris <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Marco Elver <[email protected]>
Cc: Qian Cai <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]Alexander Potapenko (2):
mm: security: introduce init_on_alloc=1 and init_on_free=1 boot
options
mm: init: report memory auto-initialization features at boot time.../admin-guide/kernel-parameters.txt | 9 +++ drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 22 ++++++ init/main.c | 24 +++++++ mm/dmapool.c | 4 +- mm/page_alloc.c | 71 +++++++++++++++++-- mm/slab.c | 16 ++++- mm/slab.h | 19 +++++ mm/slub.c | 43 +++++++++-- net/core/sock.c | 2 +- security/Kconfig.hardening | 29 +++++++++ 12 files changed, 204 insertions(+), 19 deletions(-)---
v3: dropped__GFP_NO_AUTOINITpatches
v5: dropped support for SLOB allocator, handleSLAB_TYPESAFE_BY_RCU
v6: changed wording in boot-time message
v7: dropped thetest_meminit.cpatch (picked by Andrew Morton already),
minor wording changes
v8: fixes for interoperability with other heap debugging features
--
2.22.0.410.gd8fdbe21b5-goog
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg