RFC: add init_allocations=1 boot option
Subsystems (11)
cover RFC: add init_allocations=1 boot option
Following the recent discussions here's another take at initializing pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
The patchset introduces a new boot option, init_allocations, which makes page allocator and SL[AOU]B initialize newly allocated memory. init_allocations=0 doesn't (hopefully) add any overhead to the allocation fast path (no noticeable slowdown on hackbench).
With only the the first of the proposed patches the slowdown numbers are:
- 1.1% (stdev 0.2%) sys time slowdown building Linux kernel
- 3.1% (stdev 0.3%) sys time slowdown on af_inet_loopback benchmark
- 9.4% (stdev 0.5%) sys time slowdown on hackbench
The second patch introduces a GFP flag that allows to disable initialization for certain allocations. The third page is an example of applying it to af_unix.c, which helps hackbench greatly.
Slowdown numbers for the whole patchset are:
- 1.8% (stdev 0.8%) on kernel build
- 6.5% (stdev 0.2%) on af_inet_loopback
- 0.12% (stdev 0.6%) on hackbench
Alexander Potapenko (3):
mm: security: introduce the init_allocations=1 boot option
gfp: mm: introduce __GFP_NOINIT
net: apply __GFP_NOINIT to AF_UNIX sk_buff allocations
drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/gfp.h | 6 ++++- include/linux/mm.h | 8 +++++++ include/linux/slab_def.h | 1 + include/linux/slub_def.h | 1 + include/net/sock.h | 5 +++++ kernel/kexec_core.c | 4 ++-- mm/dmapool.c | 2 +- mm/page_alloc.c | 18 ++++++++++++++- mm/slab.c | 14 ++++++------ mm/slab.h | 1 + mm/slab_common.c | 15 +++++++++++++ mm/slob.c | 3 ++- mm/slub.c | 9 ++++---- net/core/sock.c | 31 +++++++++++++++++++++----- net/unix/af_unix.c | 13 ++++++----- 16 files changed, 104 insertions(+), 29 deletions(-)
Signature · 1 line
2.21.0.392.gf8f6787159e-goog
PATCH 1/3 mm: security: introduce the init_allocations=1 boot option
This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed. We don't initialize slab caches with constructors to preserve their semantics. To reduce runtime costs of checking cachep->ctor we replace a call to memset with a call to cachep->poison_fn, which is only executed if the memory block needs to be initialized.
For kernel testing purposes filling allocations with a nonzero pattern would be more suitable, but may require platform-specific code. To have a simple baseline we've decided to start with zero-initialization.
No performance optimizations are done at the moment to reduce double initialization of memory regions.
--- drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 8 ++++++++ include/linux/slab_def.h | 1 + include/linux/slub_def.h | 1 + kernel/kexec_core.c | 2 +- mm/dmapool.c | 2 +- mm/page_alloc.c | 18 +++++++++++++++++- mm/slab.c | 12 ++++++------ mm/slab.h | 1 + mm/slab_common.c | 15 +++++++++++++++ mm/slob.c | 2 +- mm/slub.c | 8 ++++---- net/core/sock.c | 2 +- 13 files changed, 58 insertions(+), 16 deletions(-)
drivers/infiniband/core/uverbs_ioctl.c +1 -1
@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used =ALIGN (new_used,sizeof (*pbundle->internal_buffer)); -if (flags & __GFP_ZERO) +if (want_init_memory (flags))memset (res,0 , size);return res; }
include/linux/mm.h +8 -0
@@ -2597,6 +2597,14 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { }#endif +DECLARE_STATIC_KEY_FALSE (init_allocations); +static inline bool want_init_memory (gfp_t flags) +{ +if (static_branch_unlikely (&init_allocations)) +return true; +return flags & __GFP_ZERO; +} +#ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled;extern void __kernel_map_pages (struct page *page,int numpages,int enable);
include/linux/slab_def.h +1 -0
@@ -37,6 +37,7 @@ struct kmem_cache {/* constructor func */ void (*ctor)(void *obj); +void (*poison_fn)(struct kmem_cache *c,void *object);/* 4) cache creation/removal */ const char *name;
include/linux/slub_def.h +1 -0
@@ -99,6 +99,7 @@ struct kmem_cache {gfp_t allocflags;/* gfp flags to use on each alloc */ int refcount;/* Refcount for slab cache destroy */ void (*ctor)(void *); +void (*poison_fn)(struct kmem_cache *c,void *object);unsigned int inuse;/* Offset to metadata */ unsigned int align;/* Alignment */ unsigned int red_left_pad;/* Left redzone padding size */
kernel/kexec_core.c +1 -1
@@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)arch_kexec_post_alloc_pages (page_address (pages), count, gfp_mask); -if (gfp_mask & __GFP_ZERO) +if (want_init_memory (gfp_mask))for (i =0 ; i < count; i++)clear_highpage (pages + i); }
mm/dmapool.c +1 -1
@@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,#endif spin_unlock_irqrestore (&pool->lock, flags); -if (mem_flags & __GFP_ZERO) +if (want_init_memory (mem_flags))memset (retval,0 , pool->size);return retval;
mm/page_alloc.c +17 -1
@@ -133,6 +133,22 @@ unsigned long totalcma_pages __read_mostly;int percpu_pagelist_fraction;gfp_t gfp_allowed_mask__read_mostly =GFP_BOOT_MASK ; +bool want_init_allocations__read_mostly ; +EXPORT_SYMBOL (want_init_allocations); +DEFINE_STATIC_KEY_FALSE (init_allocations); + +static int __init early_init_allocations (char *buf) +{ +int ret; + +if (!buf) +return -EINVAL ; + ret =kstrtobool (buf, &want_init_allocations); +if (want_init_allocations) +static_branch_enable (&init_allocations); +return ret; +} +early_param ("init_allocations" , early_init_allocations);/* * A cached value of the page's pageblock's migratetype, used when the page is
@@ -2014,7 +2030,7 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flagspost_alloc_hook (page, order, gfp_flags); -if (!free_pages_prezeroed () && (gfp_flags & __GFP_ZERO)) +if (!free_pages_prezeroed () &&want_init_memory (gfp_flags))for (i =0 ; i < (1 << order); i++)clear_highpage (page + i);
mm/slab.c +6 -6
@@ -3331,8 +3331,8 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,local_irq_restore (save_flags); ptr =cache_alloc_debugcheck_after (cachep, flags, ptr, caller); -if (unlikely (flags & __GFP_ZERO) && ptr) -memset (ptr,0 , cachep->object_size); +if (unlikely (want_init_memory (flags)) && ptr) + cachep->poison_fn (cachep, ptr);slab_post_alloc_hook (cachep, flags,1 , &ptr);return ptr;
@@ -3388,8 +3388,8 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp =cache_alloc_debugcheck_after (cachep, flags, objp, caller);prefetchw (objp); -if (unlikely (flags & __GFP_ZERO) && objp) -memset (objp,0 , cachep->object_size); +if (unlikely (want_init_memory (flags)) && objp) + cachep->poison_fn (cachep, objp);slab_post_alloc_hook (cachep, flags,1 , &objp);return objp;
@@ -3596,9 +3596,9 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,cache_alloc_debugcheck_after_bulk (s, flags, size, p, _RET_IP_);/* Clear memory outside IRQ disabled section */ -if (unlikely (flags & __GFP_ZERO)) +if (unlikely (want_init_memory (flags)))for (i =0 ; i < size; i++) -memset (p[i],0 , s->object_size); + s->poison_fn (s, p[i]);slab_post_alloc_hook (s, flags, size, p);/* FIXME: Trace call missing. Christoph would like a bulk variant */
mm/slab.h +1 -0
@@ -27,6 +27,7 @@ struct kmem_cache {const char *name;/* Slab name for sysfs */ int refcount;/* Use counter */ void (*ctor)(void *);/* Called on object slot creation */ +void (*poison_fn)(struct kmem_cache *c,void *object);struct list_head list;/* List of all slab caches on the system */ };
mm/slab_common.c +15 -0
@@ -360,6 +360,16 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align,return NULL ; } +static void poison_zero (struct kmem_cache *c,void *object) +{ +memset (object,0 , c->object_size); +} + +static void poison_dont (struct kmem_cache *c,void *object) +{ +/* Do nothing. Use for caches with constructors. */ +} + static struct kmem_cache *create_cache(const char *name, unsigned int object_size, unsigned int align, slab_flags_t flags, unsigned int useroffset,
@@ -381,6 +391,10 @@ static struct kmem_cache *create_cache(const char *name, s->size = s->object_size = object_size; s->align = align; s->ctor = ctor; +if (ctor) + s->poison_fn = poison_dont; +else + s->poison_fn = poison_zero; s->useroffset = useroffset; s->usersize = usersize;
@@ -974,6 +988,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, s->align =calculate_alignment (flags,ARCH_KMALLOC_MINALIGN , size); s->useroffset = useroffset; s->usersize = usersize; + s->poison_fn = poison_zero;slab_init_memcg_params (s);
mm/slob.c +1 -1
@@ -330,7 +330,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node)BUG_ON (!b);spin_unlock_irqrestore (&slob_lock, flags); } -if (unlikely (gfp & __GFP_ZERO)) +if (unlikely (want_init_memory (gfp)))memset (b,0 , size);return b; }
mm/slub.c +4 -4
@@ -2750,8 +2750,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,stat (s,ALLOC_FASTPATH ); } -if (unlikely (gfpflags & __GFP_ZERO) && object) -memset (object,0 , s->object_size); +if (unlikely (want_init_memory (gfpflags)) && object) + s->poison_fn (s, object);slab_post_alloc_hook (s, gfpflags,1 , &object);
@@ -3172,11 +3172,11 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,local_irq_enable ();/* Clear memory outside IRQ disabled fastpath loop */ -if (unlikely (flags & __GFP_ZERO)) { +if (unlikely (want_init_memory (flags))) {int j;for (j =0 ; j < i; j++) -memset (p[j],0 , s->object_size); + s->poison_fn (s, p[j]); }/* memcg and kmem_cache debug support */
net/core/sock.c +1 -1
@@ -1601,7 +1601,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk =kmem_cache_alloc (slab, priority & ~__GFP_ZERO);if (!sk)return sk; -if (priority & __GFP_ZERO) +if (want_init_memory (priority))sk_prot_clear_nulls (sk, prot->obj_size); }else sk =kmalloc (prot->obj_size, priority);
Signature · 1 line
2.21.0.392.gf8f6787159e-goog
Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option
Quoted 4 lines · Alexander Potapenko
This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
Isn't it better to do this at free time rather than allocation time? If doing it at free, you can't even have information leaks for pages that are in the allocator.
Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option
Quoted 10 lines · Dave Hansen
Quoted 4 lines · Alexander Potapenko
This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
Isn't it better to do this at free time rather than allocation time? If doing it at free, you can't even have information leaks for pages that are in the allocator.
I should have mentioned this in the patch description, as this question is being asked every time I send a patch :)
If we want to avoid double initialization and take advantage of __GFP_NOINIT (see the second and third patches in the series) we need to do initialize the memory at allocation time, because free() and free_pages() don't accept GFP flags.
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option
Quoted 12 lines · Alexander Potapenko
Quoted 10 lines · Dave Hansen
Quoted 4 lines · Alexander Potapenko
This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
Isn't it better to do this at free time rather than allocation time? If doing it at free, you can't even have information leaks for pages that are in the allocator.
Quoted 6 lines
I should have mentioned this in the patch description, as this question is being asked every time I send a patch :)
If we want to avoid double initialization and take advantage of__GFP_NOINIT(see the second and third patches in the series) we need to do initialize the memory at allocation time, becausefree()andfree_pages()don't accept GFP flags.
On a second thought, double zeroing on memory reclaim should be quite rare. Most of the speedup we gain with __GFP_NOINIT is because we assume it's safe to not initialize memory that'll be overwritten anyway. I'll need to check how e.g. hackbench behaves if we choose to zero memory on free() (my guess would be it'll be slower than with __GFP_NOINIT hack, albeit a little safer)
Quoted 13 lines
--
Alexander Potapenko
Software EngineerGoogle Germany GmbH
Erika-Mann-Straße, 33
80636 MünchenGeschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option
In reply to Dave HansenQuoted 9 lines · Dave Hansen
Quoted 4 lines · Alexander Potapenko
This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
Isn't it better to do this at free time rather than allocation time? If doing it at free, you can't even have information leaks for pages that are in the allocator.
I would tend to agree here. Free path is usually less performance sensitive than the allocation. Those really hot paths tend to defer the work.
I am also worried that an opt-out gfp flag would tend to be used incorrectly as the history has shown for others - e.g. __GFP_TEMPORARY. So I would rather see this robust without a fine tuning unless there is real use case that would suffer from this and we can think of a background scrubbing or something similar.
Signature · 2 lines
Michal Hocko SUSE Labs
Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option
In reply to Alexander PotapenkoQuoted 37 lines · Alexander Potapenko
This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed. We don't initialize slab caches with constructors to preserve their semantics. To reduce runtime costs of checkingcachep->ctorwe replace a call to memset with a call tocachep->poison_fn, which is only executed if the memory block needs to be initialized.For kernel testing purposes filling allocations with a nonzero pattern would be more suitable, but may require platform-specific code. To have a simple baseline we've decided to start with zero-initialization.
No performance optimizations are done at the moment to reduce double initialization of memory regions.
Signed-off-by Alexander Potapenko <[email protected]>Cc Andrew Morton <[email protected]>Cc James Morris <[email protected]>Cc "Serge E. Hallyn" <[email protected]>Cc Nick Desaulniers <[email protected]>Cc Kostya Serebryany <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc Kees Cook <[email protected]>Cc Sandeep Patil <[email protected]>Cc Laura Abbott <[email protected]>Cc Randy Dunlap <[email protected]>Cc Jann Horn <[email protected]>Cc Mark Rutland <[email protected]>Cc Qian Cai <[email protected]>Cc Vlastimil Babka <[email protected]>
Quoted 16 lines
--- drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 8 ++++++++ include/linux/slab_def.h | 1 + include/linux/slub_def.h | 1 + kernel/kexec_core.c | 2 +- mm/dmapool.c | 2 +- mm/page_alloc.c | 18 +++++++++++++++++- mm/slab.c | 12 ++++++------ mm/slab.h | 1 + mm/slab_common.c | 15 +++++++++++++++ mm/slob.c | 2 +- mm/slub.c | 8 ++++---- net/core/sock.c | 2 +- 13 files changed, 58 insertions(+), 16 deletions(-)
Hi,
Please document init_allocations=N in Documentation/admin-guide/kernel-parameters.txt.
thanks.
Signature · 1 line
~Randy
Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option
In reply to Alexander PotapenkoQuoted 12 lines · Alexander Potapenko
This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed. We don't initialize slab caches with constructors to preserve their semantics. To reduce runtime costs of checkingcachep->ctorwe replace a call to memset with a call tocachep->poison_fn, which is only executed if the memory block needs to be initialized.
Quoted 3 lines
For kernel testing purposes filling allocations with a nonzero pattern would be more suitable, but may require platform-specific code. To have a simple baseline we've decided to start with zero-initialization.
The memory tagging future may be worth mentioning here too. We'll always need an allocation-time hook. What we do in that hook (tag, zero, poison) can be manage from there.
Quoted 2 lines
No performance optimizations are done at the moment to reduce double initialization of memory regions.
Isn't this addressed in later patches?
Quoted 53 lines
Signed-off-by Alexander Potapenko <[email protected]>Cc Andrew Morton <[email protected]>Cc James Morris <[email protected]>Cc "Serge E. Hallyn" <[email protected]>Cc Nick Desaulniers <[email protected]>Cc Kostya Serebryany <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc Kees Cook <[email protected]>Cc Sandeep Patil <[email protected]>Cc Laura Abbott <[email protected]>Cc Randy Dunlap <[email protected]>Cc Jann Horn <[email protected]>Cc Mark Rutland <[email protected]>Cc Qian Cai <[email protected]>Cc Vlastimil Babka <[email protected]>--- drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 8 ++++++++ include/linux/slab_def.h | 1 + include/linux/slub_def.h | 1 + kernel/kexec_core.c | 2 +- mm/dmapool.c | 2 +- mm/page_alloc.c | 18 +++++++++++++++++- mm/slab.c | 12 ++++++------ mm/slab.h | 1 + mm/slab_common.c | 15 +++++++++++++++ mm/slob.c | 2 +- mm/slub.c | 8 ++++---- net/core/sock.c | 2 +- 13 files changed, 58 insertions(+), 16 deletions(-)drivers/infiniband/core/uverbs_ioctl.c +1
-1@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used =ALIGN (new_used,sizeof (*pbundle->internal_buffer)); -if (flags & __GFP_ZERO) +if (want_init_memory (flags))memset (res,0 , size);return res; }include/linux/mm.h +0
-0
include/linux/mm.h static inline void kernel_poison_pages(struct page *page, int numpages, · line 2600
Quoted 4 lines
include/linux/mm.h +1
-0@@ -2597,6 +2597,14 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { }#endif +DECLARE_STATIC_KEY_FALSE (init_allocations);
I'd like a CONFIG to control this default. We can keep the boot-time option to change it, but I think a CONFIG is warranted here.
Quoted 65 lines
include/linux/mm.h +7
-0+static inline bool want_init_memory (gfp_t flags) +{ +if (static_branch_unlikely (&init_allocations)) +return true; +return flags & __GFP_ZERO; +} +#ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled;extern void __kernel_map_pages (struct page *page,int numpages,int enable);include/linux/slab_def.h +1
-0@@ -37,6 +37,7 @@ struct kmem_cache {/* constructor func */ void (*ctor)(void *obj); +void (*poison_fn)(struct kmem_cache *c,void *object);/* 4) cache creation/removal */ const char *name;include/linux/slub_def.h +1
-0@@ -99,6 +99,7 @@ struct kmem_cache {gfp_t allocflags;/* gfp flags to use on each alloc */ int refcount;/* Refcount for slab cache destroy */ void (*ctor)(void *); +void (*poison_fn)(struct kmem_cache *c,void *object);unsigned int inuse;/* Offset to metadata */ unsigned int align;/* Alignment */ unsigned int red_left_pad;/* Left redzone padding size */ kernel/kexec_core.c +1
-1@@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)arch_kexec_post_alloc_pages (page_address (pages), count, gfp_mask); -if (gfp_mask & __GFP_ZERO) +if (want_init_memory (gfp_mask))for (i =0 ; i < count; i++)clear_highpage (pages + i); }mm/dmapool.c +1
-1@@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,#endif spin_unlock_irqrestore (&pool->lock, flags); -if (mem_flags & __GFP_ZERO) +if (want_init_memory (mem_flags))memset (retval,0 , pool->size);return retval;mm/page_alloc.c +0
-0
mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 136
Quoted 4 lines
mm/page_alloc.c +1
-0@@ -133,6 +133,22 @@ unsigned long totalcma_pages __read_mostly;int percpu_pagelist_fraction;gfp_t gfp_allowed_mask__read_mostly =GFP_BOOT_MASK ; +bool want_init_allocations__read_mostly ;
This can be a stack variable in early_init_allocations() -- it's never used again.
mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 148
Quoted 12 lines
mm/page_alloc.c +12
-0+EXPORT_SYMBOL (want_init_allocations); +DEFINE_STATIC_KEY_FALSE (init_allocations); + +static int __init early_init_allocations (char *buf) +{ +int ret; + +if (!buf) +return -EINVAL ; + ret =kstrtobool (buf, &want_init_allocations); +if (want_init_allocations) +static_branch_enable (&init_allocations);
With the CONFIG, this should have a _disable on an "else" here...
mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 151
Quoted 3 lines
mm/page_alloc.c +3
-0+return ret; +} +early_param ("init_allocations" , early_init_allocations);
Does early_init_allocations() get called before things like prep_new_page() are up and running to call want_init_memory()?
Quoted 17 lines
mm/page_alloc.c +1
-1/* * A cached value of the page's pageblock's migratetype, used when the page is @@ -2014,7 +2030,7 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flagspost_alloc_hook (page, order, gfp_flags); -if (!free_pages_prezeroed () && (gfp_flags & __GFP_ZERO)) +if (!free_pages_prezeroed () &&want_init_memory (gfp_flags))for (i =0 ; i < (1 << order); i++)clear_highpage (page + i);mm/slab.c +0
-0
mm/slab.c slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, · line 3335
Quoted 7 lines
mm/slab.c +2
-2@@ -3331,8 +3331,8 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,local_irq_restore (save_flags); ptr =cache_alloc_debugcheck_after (cachep, flags, ptr, caller); -if (unlikely (flags & __GFP_ZERO) && ptr) -memset (ptr,0 , cachep->object_size); +if (unlikely (want_init_memory (flags)) && ptr) + cachep->poison_fn (cachep, ptr);
So... this _must_ zero when __GFP_ZERO is present, so I'm not sure "poison_fn" is the right name, and it likely needs to take the "flags" argument.
mm/slab.c slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) · line 3392
Quoted 11 lines
mm/slab.c +2
-2slab_post_alloc_hook (cachep, flags,1 , &ptr);return ptr;@@ -3388,8 +3388,8 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp =cache_alloc_debugcheck_after (cachep, flags, objp, caller);prefetchw (objp); -if (unlikely (flags & __GFP_ZERO) && objp) -memset (objp,0 , cachep->object_size); +if (unlikely (want_init_memory (flags)) && objp) + cachep->poison_fn (cachep, objp);
Same.
mm/slab.c int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, · line 3601
Quoted 12 lines
mm/slab.c +2
-2slab_post_alloc_hook (cachep, flags,1 , &objp);return objp;@@ -3596,9 +3596,9 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,cache_alloc_debugcheck_after_bulk (s, flags, size, p, _RET_IP_);/* Clear memory outside IRQ disabled section */ -if (unlikely (flags & __GFP_ZERO)) +if (unlikely (want_init_memory (flags)))for (i =0 ; i < size; i++) -memset (p[i],0 , s->object_size); + s->poison_fn (s, p[i]);
Same.
mm/slab.h struct kmem_cache { · line 30
Quoted 12 lines
mm/slab.c +0
-0slab_post_alloc_hook (s, flags, size, p);/* FIXME: Trace call missing. Christoph would like a bulk variant */ mm/slab.h +1
-0@@ -27,6 +27,7 @@ struct kmem_cache {const char *name;/* Slab name for sysfs */ int refcount;/* Use counter */ void (*ctor)(void *);/* Called on object slot creation */ +void (*poison_fn)(struct kmem_cache *c,void *object);
How about naming it just "initialize"?
Quoted 25 lines
mm/slab.h +0
-0struct list_head list;/* List of all slab caches on the system */ };mm/slab_common.c +10
-0@@ -360,6 +360,16 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align,return NULL ; } +static void poison_zero (struct kmem_cache *c,void *object) +{ +memset (object,0 , c->object_size); +} + +static void poison_dont (struct kmem_cache *c,void *object) +{ +/* Do nothing. Use for caches with constructors. */ +} + static struct kmem_cache *create_cache(const char *name, unsigned int object_size, unsigned int align, slab_flags_t flags, unsigned int useroffset,
mm/slab_common.c static struct kmem_cache *create_cache(const char *name, · line 397
Quoted 7 lines
mm/slab_common.c +4
-0@@ -381,6 +391,10 @@ static struct kmem_cache *create_cache(const char *name, s->size = s->object_size = object_size; s->align = align; s->ctor = ctor; +if (ctor) + s->poison_fn = poison_dont; +else + s->poison_fn = poison_zero;
As mentioned, we must still always zero when __GFP_ZERO is present.
Quoted 67 lines
mm/slab_common.c +1
-0s->useroffset = useroffset; s->usersize = usersize;@@ -974,6 +988,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, s->align =calculate_alignment (flags,ARCH_KMALLOC_MINALIGN , size); s->useroffset = useroffset; s->usersize = usersize; + s->poison_fn = poison_zero;slab_init_memcg_params (s);mm/slob.c +1
-1@@ -330,7 +330,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node)BUG_ON (!b);spin_unlock_irqrestore (&slob_lock, flags); } -if (unlikely (gfp & __GFP_ZERO)) +if (unlikely (want_init_memory (gfp)))memset (b,0 , size);return b; }mm/slub.c +4
-4@@ -2750,8 +2750,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,stat (s,ALLOC_FASTPATH ); } -if (unlikely (gfpflags & __GFP_ZERO) && object) -memset (object,0 , s->object_size); +if (unlikely (want_init_memory (gfpflags)) && object) + s->poison_fn (s, object);slab_post_alloc_hook (s, gfpflags,1 , &object);@@ -3172,11 +3172,11 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,local_irq_enable ();/* Clear memory outside IRQ disabled fastpath loop */ -if (unlikely (flags & __GFP_ZERO)) { +if (unlikely (want_init_memory (flags))) {int j;for (j =0 ; j < i; j++) -memset (p[j],0 , s->object_size); + s->poison_fn (s, p[j]); }/* memcg and kmem_cache debug support */ net/core/sock.c +1
-2@@ -1601,7 +1601,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk =kmem_cache_alloc (slab, priority & ~__GFP_ZERO);if (!sk)return sk; -if (priority & __GFP_ZERO) +if (want_init_memory (priority))sk_prot_clear_nulls (sk, prot->obj_size); }else sk =kmalloc (prot->obj_size, priority); --
Quoted 2 lines
2.21.0.392.gf8f6787159e-goog
Looking good. :) Thanks for working on this!
Signature · 1 line
Kees Cook
Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option
Quoted 14 lines · Kees Cook
Quoted 12 lines · Alexander Potapenko
This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed. We don't initialize slab caches with constructors to preserve their semantics. To reduce runtime costs of checkingcachep->ctorwe replace a call to memset with a call tocachep->poison_fn, which is only executed if the memory block needs to be initialized.
Quoted 7 lines
Quoted 3 lines
For kernel testing purposes filling allocations with a nonzero pattern would be more suitable, but may require platform-specific code. To have a simple baseline we've decided to start with zero-initialization.
The memory tagging future may be worth mentioning here too. We'll always need an allocation-time hook. What we do in that hook (tag, zero, poison) can be manage from there.
Shall we factor out the allocation hook in this patch? Note that I'll be probably dropping this poison_fn() stuff, as it should be enough to just call memset() under a static branch.
Quoted 4 lines
Quoted 2 lines
No performance optimizations are done at the moment to reduce double initialization of memory regions.
Isn't this addressed in later patches?
Agreed.
Quoted 56 lines
Quoted 56 lines
Signed-off-by Alexander Potapenko <[email protected]>Cc Andrew Morton <[email protected]>Cc James Morris <[email protected]>Cc "Serge E. Hallyn" <[email protected]>Cc Nick Desaulniers <[email protected]>Cc Kostya Serebryany <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc Kees Cook <[email protected]>Cc Sandeep Patil <[email protected]>Cc Laura Abbott <[email protected]>Cc Randy Dunlap <[email protected]>Cc Jann Horn <[email protected]>Cc Mark Rutland <[email protected]>Cc Qian Cai <[email protected]>Cc Vlastimil Babka <[email protected]>--- drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/mm.h | 8 ++++++++ include/linux/slab_def.h | 1 + include/linux/slub_def.h | 1 + kernel/kexec_core.c | 2 +- mm/dmapool.c | 2 +- mm/page_alloc.c | 18 +++++++++++++++++- mm/slab.c | 12 ++++++------ mm/slab.h | 1 + mm/slab_common.c | 15 +++++++++++++++ mm/slob.c | 2 +- mm/slub.c | 8 ++++---- net/core/sock.c | 2 +- 13 files changed, 58 insertions(+), 16 deletions(-)drivers/infiniband/core/uverbs_ioctl.c +1
-1@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size, res = (void *)pbundle->internal_buffer + pbundle->internal_used; pbundle->internal_used =ALIGN (new_used,sizeof (*pbundle->internal_buffer)); -if (flags & __GFP_ZERO) +if (want_init_memory (flags))memset (res,0 , size);return res; }include/linux/mm.h +0
-0@@ -2597,6 +2597,14 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { }#endif
Quoted 4 lines
include/linux/mm.h static inline void kernel_poison_pages(struct page *page, int numpages, · line 2600
Quoted 1 line
include/linux/mm.h +1
-0+DECLARE_STATIC_KEY_FALSE (init_allocations);I'd like a CONFIG to control this default. We can keep the boot-time option to change it, but I think a CONFIG is warranted here.
Ok, will do.
Quoted 66 lines
Quoted 66 lines
include/linux/mm.h +7
-0+static inline bool want_init_memory (gfp_t flags) +{ +if (static_branch_unlikely (&init_allocations)) +return true; +return flags & __GFP_ZERO; +} +#ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled;extern void __kernel_map_pages (struct page *page,int numpages,int enable);include/linux/slab_def.h +1
-0@@ -37,6 +37,7 @@ struct kmem_cache {/* constructor func */ void (*ctor)(void *obj); +void (*poison_fn)(struct kmem_cache *c,void *object);/* 4) cache creation/removal */ const char *name;include/linux/slub_def.h +1
-0@@ -99,6 +99,7 @@ struct kmem_cache {gfp_t allocflags;/* gfp flags to use on each alloc */ int refcount;/* Refcount for slab cache destroy */ void (*ctor)(void *); +void (*poison_fn)(struct kmem_cache *c,void *object);unsigned int inuse;/* Offset to metadata */ unsigned int align;/* Alignment */ unsigned int red_left_pad;/* Left redzone padding size */ kernel/kexec_core.c +1
-1@@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)arch_kexec_post_alloc_pages (page_address (pages), count, gfp_mask); -if (gfp_mask & __GFP_ZERO) +if (want_init_memory (gfp_mask))for (i =0 ; i < count; i++)clear_highpage (pages + i); }mm/dmapool.c +1
-1@@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,#endif spin_unlock_irqrestore (&pool->lock, flags); -if (mem_flags & __GFP_ZERO) +if (want_init_memory (mem_flags))memset (retval,0 , pool->size);return retval;mm/page_alloc.c +0
-0@@ -133,6 +133,22 @@ unsigned long totalcma_pages __read_mostly;
Quoted 6 lines
mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 136
Quoted 3 lines
mm/page_alloc.c +1
-0int percpu_pagelist_fraction;gfp_t gfp_allowed_mask__read_mostly =GFP_BOOT_MASK ; +bool want_init_allocations__read_mostly ;This can be a stack variable in
early_init_allocations()-- it's never used again.
Ack.
Quoted 6 lines
Quoted 6 lines
mm/page_alloc.c +6
-0+EXPORT_SYMBOL (want_init_allocations); +DEFINE_STATIC_KEY_FALSE (init_allocations); + +static int __init early_init_allocations (char *buf) +{ +int ret;
Quoted 8 lines
mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 148
Quoted 6 lines
mm/page_alloc.c +6
-0+ +if (!buf) +return -EINVAL ; + ret =kstrtobool (buf, &want_init_allocations); +if (want_init_allocations) +static_branch_enable (&init_allocations);With the CONFIG, this should have a
_disableon an "else" here...
Ack.
Quoted 6 lines
mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 151
Quoted 3 lines
mm/page_alloc.c +3
-0+return ret; +} +early_param ("init_allocations" , early_init_allocations);Does
early_init_allocations()get called before things likeprep_new_page()are up and running to callwant_init_memory()?
Yes, IIUC early params are initialized before the memory subsystem is initialized.
Quoted 20 lines
Quoted 20 lines
mm/page_alloc.c +1
-1/* * A cached value of the page's pageblock's migratetype, used when the page is @@ -2014,7 +2030,7 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flagspost_alloc_hook (page, order, gfp_flags); -if (!free_pages_prezeroed () && (gfp_flags & __GFP_ZERO)) +if (!free_pages_prezeroed () &&want_init_memory (gfp_flags))for (i =0 ; i < (1 << order); i++)clear_highpage (page + i);mm/slab.c +0
-0@@ -3331,8 +3331,8 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,local_irq_restore (save_flags); ptr =cache_alloc_debugcheck_after (cachep, flags, ptr, caller);
Quoted 8 lines
mm/slab.c slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, · line 3335
Quoted 4 lines
mm/slab.c +2
-2-if (unlikely (flags & __GFP_ZERO) && ptr) -memset (ptr,0 , cachep->object_size); +if (unlikely (want_init_memory (flags)) && ptr) + cachep->poison_fn (cachep, ptr);So... this _must_ zero when
__GFP_ZEROis present, so I'm not sure "poison_fn" is the right name, and it likely needs to take the "flags" argument.
I'll rework this part, as it had been pointed out that an empty indirect call still costs something.
We're basically choosing between two options:
- flags & __GFP_ZERO when initialization is disabled;
- and cachep->ctr when it's enabled
A static branch should be enough to switch between those without introducing extra checks or indirections.
Quoted 31 lines
mm/slab.c slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) · line 3392
Quoted 12 lines
mm/slab.c +2
-2slab_post_alloc_hook (cachep, flags,1 , &ptr);return ptr;@@ -3388,8 +3388,8 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) objp =cache_alloc_debugcheck_after (cachep, flags, objp, caller);prefetchw (objp); -if (unlikely (flags & __GFP_ZERO) && objp) -memset (objp,0 , cachep->object_size); +if (unlikely (want_init_memory (flags)) && objp) + cachep->poison_fn (cachep, objp);Same.
mm/slab.c int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, · line 3601
Quoted 12 lines
mm/slab.c +2
-2slab_post_alloc_hook (cachep, flags,1 , &objp);return objp;@@ -3596,9 +3596,9 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,cache_alloc_debugcheck_after_bulk (s, flags, size, p, _RET_IP_);/* Clear memory outside IRQ disabled section */ -if (unlikely (flags & __GFP_ZERO)) +if (unlikely (want_init_memory (flags)))for (i =0 ; i < size; i++) -memset (p[i],0 , s->object_size); + s->poison_fn (s, p[i]);Same.
Quoted 1 line
Quoted 13 lines
mm/slab.h struct kmem_cache { · line 30
Quoted 11 lines
slab_post_alloc_hook(s, flags, size, p); /* FIXME: Trace call missing. Christoph would like a bulk variant */mm/slab.h +1
-0@@ -27,6 +27,7 @@ struct kmem_cache {const char *name;/* Slab name for sysfs */ int refcount;/* Use counter */ void (*ctor)(void *);/* Called on object slot creation */ +void (*poison_fn)(struct kmem_cache *c,void *object);How about naming it just "initialize"?
DItto.
Quoted 26 lines
Quoted 26 lines
mm/slab.h +0
-0struct list_head list;/* List of all slab caches on the system */ };mm/slab_common.c +10
-0@@ -360,6 +360,16 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align,return NULL ; } +static void poison_zero (struct kmem_cache *c,void *object) +{ +memset (object,0 , c->object_size); +} + +static void poison_dont (struct kmem_cache *c,void *object) +{ +/* Do nothing. Use for caches with constructors. */ +} + static struct kmem_cache *create_cache(const char *name, unsigned int object_size, unsigned int align, slab_flags_t flags, unsigned int useroffset,@@ -381,6 +391,10 @@ static struct kmem_cache *create_cache(const char *name, s->size = s->object_size = object_size;
Quoted 8 lines
mm/slab_common.c static struct kmem_cache *create_cache(const char *name, · line 397
Quoted 6 lines
mm/slab_common.c +4
-0s->align = align; s->ctor = ctor; +if (ctor) + s->poison_fn = poison_dont; +else + s->poison_fn = poison_zero;As mentioned, we must still always zero when
__GFP_ZEROis present.
This part will be gone, but in general __GFP_ZERO is incompatible with constructors, see a check in new_slab_objects().
I don't think we must support erroneous behaviour.
Quoted 74 lines
Quoted 67 lines
mm/slab_common.c +1
-0s->useroffset = useroffset; s->usersize = usersize;@@ -974,6 +988,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, s->align =calculate_alignment (flags,ARCH_KMALLOC_MINALIGN , size); s->useroffset = useroffset; s->usersize = usersize; + s->poison_fn = poison_zero;slab_init_memcg_params (s);mm/slob.c +1
-1@@ -330,7 +330,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node)BUG_ON (!b);spin_unlock_irqrestore (&slob_lock, flags); } -if (unlikely (gfp & __GFP_ZERO)) +if (unlikely (want_init_memory (gfp)))memset (b,0 , size);return b; }mm/slub.c +4
-4@@ -2750,8 +2750,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,stat (s,ALLOC_FASTPATH ); } -if (unlikely (gfpflags & __GFP_ZERO) && object) -memset (object,0 , s->object_size); +if (unlikely (want_init_memory (gfpflags)) && object) + s->poison_fn (s, object);slab_post_alloc_hook (s, gfpflags,1 , &object);@@ -3172,11 +3172,11 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,local_irq_enable ();/* Clear memory outside IRQ disabled fastpath loop */ -if (unlikely (flags & __GFP_ZERO)) { +if (unlikely (want_init_memory (flags))) {int j;for (j =0 ; j < i; j++) -memset (p[j],0 , s->object_size); + s->poison_fn (s, p[j]); }/* memcg and kmem_cache debug support */ net/core/sock.c +1
-2@@ -1601,7 +1601,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority, sk =kmem_cache_alloc (slab, priority & ~__GFP_ZERO);if (!sk)return sk; -if (priority & __GFP_ZERO) +if (want_init_memory (priority))sk_prot_clear_nulls (sk, prot->obj_size); }else sk =kmalloc (prot->obj_size, priority); --Quoted 2 lines
2.21.0.392.gf8f6787159e-goog
Looking good. :) Thanks for working on this!
--
Kees Cook
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option
In reply to Alexander PotapenkoQuoted 20 lines · Alexander Potapenko
mm/slab_common.c +9
-0+static void poison_dont (struct kmem_cache *c,void *object) +{ +/* Do nothing. Use for caches with constructors. */ +} + static struct kmem_cache *create_cache(const char *name, unsigned int object_size, unsigned int align, slab_flags_t flags, unsigned int useroffset,@@ -381,6 +391,10 @@ static struct kmem_cache *create_cache(const char *name, s->size = s->object_size = object_size; s->align = align; s->ctor = ctor; +if (ctor) + s->poison_fn = poison_dont; +else + s->poison_fn = poison_zero; s->useroffset = useroffset; s->usersize = usersize;
mm/slab_common.c void __init create_boot_cache(struct kmem_cache *s, const char *name, · line 991
Quoted 4 lines
mm/slab_common.c +1
-0@@ -974,6 +988,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, s->align =calculate_alignment (flags,ARCH_KMALLOC_MINALIGN , size); s->useroffset = useroffset; s->usersize = usersize; + s->poison_fn = poison_zero;
An empty indirect call is probably a pretty bad idea on systems with retpoline. Isn't this just a bool anyway for either calling poison_dont or poison_zero? Can it call anything else?
Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option
In reply to Alexander PotapenkoQuoted 11 lines · Alexander Potapenko
This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
Initialization is done at allocation time at the places where checks for
__GFP_ZEROare performed. We don't initialize slab caches with constructors to preserve their semantics. To reduce runtime costs of checkingcachep->ctorwe replace a call to memset with a call tocachep->poison_fn, which is only executed if the memory block needs to be initialized.
Just check for a ctor and then zero or use whatever pattern ? Why add a new function?
A message that is not in the archive (sent off-list or not yet received): alpine.DEB.2.21.1904260911570.8340@nuc-kabylake
Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option
Hmmmm.... Maybe its better to zero on free? That way you dont need to initialize the allocations. You could even check if someone mucked with the object during allocation. This is a replication of some of the inherent debugging facilities in the allocator though.
Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option
Quoted 4 lines · Christopher Lameter
Hmmmm.... Maybe its better to zero on free? That way you dont need to initialize the allocations. You could even check if someone mucked with the object during allocation.
As mentioned somewhere in the neighboring threads, Kees requested the options to initialize on both allocation and free, because we'll need this functionality for memory tagging someday.
Quoted 2 lines
This is a replication of some of the
inherent debugging facilities in the allocator though.
I'm curious, how often do people use SL[AU]B poisoning and redzones these days? Guess those should be superseded by KASAN already?
I agree it makes sense to reuse the existing debugging code, but on the other hand we probably want to keep the initialization fast enough to be used in production.
Quoted 1 line
Re: poison_fn, it wasn't a good idea, so I'll be dropping it.
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
PATCH 2/3 gfp: mm: introduce __GFP_NOINIT
In reply to Alexander PotapenkoWhen passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NOINIT tells it to not initialize the requested memory if the init_allocations boot option is enabled. This can be useful in the cases the newly allocated memory is going to be initialized by the caller right away.
__GFP_NOINIT basically defeats the hardening against information leaks provided by the init_allocations feature, so one should use it with caution.
This patch also adds __GFP_NOINIT to alloc_pages() calls in SL[AOU]B.
--- include/linux/gfp.h | 6 +++++- include/linux/mm.h | 2 +- kernel/kexec_core.c | 2 +- mm/slab.c | 2 +- mm/slob.c | 1 + mm/slub.c | 1 + 6 files changed, 10 insertions(+), 4 deletions(-)
include/linux/gfp.h +5 -1
@@ -44,6 +44,7 @@ struct vm_area_struct;#else #define ___GFP_NOLOCKDEP 0#endif +#define ___GFP_NOINIT 0x1000000u/* If the above are modified, __GFP_BITS_SHIFT may need updating */ /*
@@ -208,16 +209,19 @@ struct vm_area_struct;* %__GFP_COMP address compound page metadata. * * %__GFP_ZERO returns a zeroed page on success. +* +* %__GFP_NOINIT requests non-initialized memory from the underlying allocator. */ #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)#define __GFP_COMP ((__force gfp_t)___GFP_COMP)#define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) +#define __GFP_NOINIT ((__force gfp_t)___GFP_NOINIT)/* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)/* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (23 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (25)#define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))/**
include/linux/mm.h +1 -1
@@ -2601,7 +2601,7 @@ DECLARE_STATIC_KEY_FALSE(init_allocations);static inline bool want_init_memory (gfp_t flags) {if (static_branch_unlikely (&init_allocations)) -return true; +return !(flags & __GFP_NOINIT);return flags & __GFP_ZERO; }
kernel/kexec_core.c +1 -1
@@ -302,7 +302,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) {struct page *pages; - pages =alloc_pages (gfp_mask & ~__GFP_ZERO, order); + pages =alloc_pages ((gfp_mask & ~__GFP_ZERO) | __GFP_NOINIT, order);if (pages) {unsigned int count, i;
mm/slab.c +1 -1
@@ -1393,7 +1393,7 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,struct page *page;int nr_pages; - flags |= cachep->allocflags; + flags |= (cachep->allocflags | __GFP_NOINIT); page =__alloc_pages_node (nodeid, flags, cachep->gfporder);if (!page) {
mm/slob.c +1 -0
@@ -192,6 +192,7 @@ static void *slob_new_pages(gfp_t gfp, int order, int node) {void *page; + gfp |= __GFP_NOINIT;#ifdef CONFIG_NUMA if (node !=NUMA_NO_NODE ) page =__alloc_pages_node (node, gfp, order);
mm/slub.c +1 -0
@@ -1493,6 +1493,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s,struct page *page;unsigned int order =oo_order (oo); + flags |= __GFP_NOINIT;if (node ==NUMA_NO_NODE ) page =alloc_pages (flags, order);else
Signature · 1 line
2.21.0.392.gf8f6787159e-goog
Re: [PATCH 2/3] gfp: mm: introduce __GFP_NOINIT
Quoted 3 lines · Alexander Potapenko
__GFP_NOINITbasically defeats the hardening against information leaks provided by theinit_allocationsfeature, so one should use it with caution.
Even more than that, shouldn't we try to use it only in places where there is a demonstrated benefit, like performance data?
kernel/kexec_core.c static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) · line 307
Quoted 12 lines
kernel/kexec_core.c +1
-1@@ -302,7 +302,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) {struct page *pages; - pages =alloc_pages (gfp_mask & ~__GFP_ZERO, order); + pages =alloc_pages ((gfp_mask & ~__GFP_ZERO) | __GFP_NOINIT, order);if (pages) {unsigned int count, i;
While this is probably not super security-sensitive, it's also not performance sensitive.
Quoted 31 lines
mm/slab.c +1
-1@@ -1393,7 +1393,7 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,struct page *page;int nr_pages; - flags |= cachep->allocflags; + flags |= (cachep->allocflags | __GFP_NOINIT); page =__alloc_pages_node (nodeid, flags, cachep->gfporder);if (!page) {mm/slob.c +1
-0@@ -192,6 +192,7 @@ static void *slob_new_pages(gfp_t gfp, int order, int node) {void *page; + gfp |= __GFP_NOINIT;#ifdef CONFIG_NUMA if (node !=NUMA_NO_NODE ) page = __alloc_pages_node(node, gfp, ordermm/slub.c +0
-0
mm/slub.c static inline struct page *alloc_slab_page(struct kmem_cache *s, · line 1499
Quoted 7 lines
mm/slub.c +1
-0@@ -1493,6 +1493,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s,struct page *page;unsigned int order =oo_order (oo); + flags |= __GFP_NOINIT;if (node ==NUMA_NO_NODE ) page =alloc_pages (flags, order);else
These sl*b ones seem like a bad idea. We already have rules that sl*b allocations must be initialized by callers, and we have reasonably frequent bugs where the rules are broken.
Setting __GFP_NOINIT might be reasonable to do, though, for slabs that have a constructor. We have much higher confidence that *those* are going to get initialized properly.
Re: [PATCH 2/3] gfp: mm: introduce __GFP_NOINIT
Quoted 10 lines · Dave Hansen
Quoted 8 lines · Alexander Potapenko
kernel/kexec_core.c +0
-0@@ -302,7 +302,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) {struct page *pages;
Quoted 7 lines
kernel/kexec_core.c static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) · line 307
Quoted 4 lines
kernel/kexec_core.c +1
-1- pages =alloc_pages (gfp_mask & ~__GFP_ZERO, order); + pages =alloc_pages ((gfp_mask & ~__GFP_ZERO) | __GFP_NOINIT, order);if (pages) {unsigned int count, i;While this is probably not super security-sensitive, it's also not performance sensitive.
It is, however, a pretty clear case of "and then we immediately zero it".
Quoted 3 lines
These sl*b ones seem like a bad idea. We already have rules that sl*b allocations must be initialized by callers, and we have reasonably frequent bugs where the rules are broken.
Hm? No, this is saying that the page allocator can skip the auto-init because the slab internals will be doing it later.
Quoted 3 lines
Setting
__GFP_NOINITmight be reasonable to do, though, for slabs that have a constructor. We have much higher confidence that *those* are going to get initialized properly.
That's already handled in patch 1.
Signature · 1 line
Kees Cook
Re: [PATCH 2/3] gfp: mm: introduce __GFP_NOINIT
Quoted 6 lines · Kees Cook
Quoted 3 lines
These sl*b ones seem like a bad idea. We already have rules that sl*b allocations must be initialized by callers, and we have reasonably frequent bugs where the rules are broken.
Hm? No, this is saying that the page allocator can skip the auto-init because the slab internals will be doing it later.
Ahhh, got it. I missed all the fun in patch 1.
Re: [PATCH 2/3] gfp: mm: introduce __GFP_NOINIT
In reply to Alexander PotapenkoQuoted 5 lines · Alexander Potapenko
When passed to an allocator (either pagealloc or SL[AOU]B),
__GFP_NOINITtells it to not initialize the requested memory if theinit_allocationsboot option is enabled. This can be useful in the cases the newly allocated memory is going to be initialized by the caller right away.
Maybe add "... as seen when the slab allocator needs to allocate new pages from the page allocator." just to help clarify it here (instead of from the end of the commit log where you mention it offhand).
Quoted 40 lines
__GFP_NOINITbasically defeats the hardening against information leaks provided by theinit_allocationsfeature, so one should use it with caution.This patch also adds
__GFP_NOINITtoalloc_pages()calls in SL[AOU]B.Signed-off-by Alexander Potapenko <[email protected]>Cc Andrew Morton <[email protected]>Cc Masahiro Yamada <[email protected]>Cc James Morris <[email protected]>Cc "Serge E. Hallyn" <[email protected]>Cc Nick Desaulniers <[email protected]>Cc Kostya Serebryany <[email protected]>Cc Dmitry Vyukov <[email protected]>Cc Kees Cook <[email protected]>Cc Sandeep Patil <[email protected]>Cc Laura Abbott <[email protected]>Cc Randy Dunlap <[email protected]>Cc Jann Horn <[email protected]>Cc Mark Rutland <[email protected]>Cc Qian Cai <[email protected]>Cc Vlastimil Babka <[email protected]>--- include/linux/gfp.h | 6 +++++- include/linux/mm.h | 2 +- kernel/kexec_core.c | 2 +- mm/slab.c | 2 +- mm/slob.c | 1 + mm/slub.c | 1 + 6 files changed, 10 insertions(+), 4 deletions(-)include/linux/gfp.h +0
-0
include/linux/gfp.h struct vm_area_struct; · line 48
Quoted 5 lines
include/linux/gfp.h +1
-0@@ -44,6 +44,7 @@ struct vm_area_struct;#else #define ___GFP_NOLOCKDEP 0#endif +#define ___GFP_NOINIT 0x1000000u/* If the above are modified, __GFP_BITS_SHIFT may need updating */
I think you want to add NOINIT below GFP_ACCOUNT, update NOLOCKDEP and then adjust GFP_BITS_SHIFT differently, noted below.
Quoted 10 lines
include/linux/gfp.h +2
-0/* @@ -208,16 +209,19 @@ struct vm_area_struct;* %__GFP_COMP address compound page metadata. * * %__GFP_ZERO returns a zeroed page on success. +* +* %__GFP_NOINIT requests non-initialized memory from the underlying allocator. */ #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)
include/linux/gfp.h struct vm_area_struct; · line 224
Quoted 10 lines
include/linux/gfp.h +2
-1#define __GFP_COMP ((__force gfp_t)___GFP_COMP)#define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) +#define __GFP_NOINIT ((__force gfp_t)___GFP_NOINIT)/* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)/* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (23 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (25)
This should just be 24 + ... with the bit field added above NOLOCKDEP.
Quoted 8 lines
include/linux/gfp.h +0
-0#define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))/** include/linux/mm.h +0
-0
include/linux/mm.h DECLARE_STATIC_KEY_FALSE(init_allocations); · line 2606
Quoted 7 lines
include/linux/mm.h +1
-1@@ -2601,7 +2601,7 @@ DECLARE_STATIC_KEY_FALSE(init_allocations);static inline bool want_init_memory (gfp_t flags) {if (static_branch_unlikely (&init_allocations)) -return true; +return !(flags & __GFP_NOINIT);return flags & __GFP_ZERO; }
You need to test for GFP_ZERO here too: return ((flags & __GFP_NOINIT | __GFP_ZERO) == 0)
Also, I wonder, for the sake of readability, if this should be named __GFP_NO_AUTOINIT ?
I'd also like to see each use of __GFP_NOINIT include a comment above its use where the logic is explained for _why_ it's safe (or reasonable) to use __GFP_NOINIT in each place.
mm/slub.c static inline struct page *alloc_slab_page(struct kmem_cache *s,
Quoted 44 lines
kernel/kexec_core.c +1
-1@@ -302,7 +302,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) {struct page *pages; - pages =alloc_pages (gfp_mask & ~__GFP_ZERO, order); + pages =alloc_pages ((gfp_mask & ~__GFP_ZERO) | __GFP_NOINIT, order);if (pages) {unsigned int count, i;mm/slab.c +1
-1@@ -1393,7 +1393,7 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,struct page *page;int nr_pages; - flags |= cachep->allocflags; + flags |= (cachep->allocflags | __GFP_NOINIT); page =__alloc_pages_node (nodeid, flags, cachep->gfporder);if (!page) {mm/slob.c +1
-0@@ -192,6 +192,7 @@ static void *slob_new_pages(gfp_t gfp, int order, int node) {void *page; + gfp |= __GFP_NOINIT;#ifdef CONFIG_NUMA if (node !=NUMA_NO_NODE ) page =__alloc_pages_node (node, gfp, order);mm/slub.c +0
-0@@ -1493,6 +1493,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s,
What about kmalloc_large_node()?
mm/slub.c static inline struct page *alloc_slab_page(struct kmem_cache *s, · line 1499
Quoted 7 lines
mm/slub.c +1
-0struct page *page;unsigned int order =oo_order (oo); + flags |= __GFP_NOINIT;if (node ==NUMA_NO_NODE ) page =alloc_pages (flags, order);else
And just so I make sure I'm understanding this correctly: __GFP_NOINIT is passed to the page allocator because we know each allocation from the slab will get initialized at "sub allocation" time.
Looks good!
Signature · 1 line
Kees Cook
PATCH 3/3 RFC: net: apply __GFP_NOINIT to AF_UNIX sk_buff allocations
In reply to Alexander PotapenkoAdd sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NOINIT. This helps reduce the slowdown on hackbench from 9% to 0.1%.
--- include/net/sock.h | 5 +++++ net/core/sock.c | 29 +++++++++++++++++++++++++---- net/unix/af_unix.c | 13 +++++++------ 3 files changed, 37 insertions(+), 10 deletions(-)
include/net/sock.h +5 -0
@@ -1612,6 +1612,11 @@ struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size,struct sk_buff *sock_alloc_send_pskb (struct sock *sk,unsigned long header_len,unsigned long data_len,int noblock,int *errcode,int max_page_order); +struct sk_buff *sock_alloc_send_pskb_noinit (struct sock *sk, +unsigned long header_len, +unsigned long data_len, +int noblock,int *errcode, +int max_page_order);void *sock_kmalloc (struct sock *sk,int size,gfp_t priority);void sock_kfree_s (struct sock *sk,void *mem,int size);void sock_kzfree_s (struct sock *sk,void *mem,int size);
net/core/sock.c +25 -4
@@ -2187,9 +2187,11 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo)* Generic send/receive buffer handlers */ -struct sk_buff *sock_alloc_send_pskb (struct sock *sk,unsigned long header_len, -unsigned long data_len,int noblock, -int *errcode,int max_page_order) +struct sk_buff *sock_alloc_send_pskb_internal (struct sock *sk, +unsigned long header_len, +unsigned long data_len, +int noblock,int *errcode, +int max_page_order,gfp_t gfp) {struct sk_buff *skb;long timeo;
@@ -2218,7 +2220,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, timeo =sock_wait_for_wmem (sk, timeo); } skb =alloc_skb_with_frags (header_len, data_len, max_page_order, - errcode, sk->sk_allocation); + errcode, sk->sk_allocation | gfp);if (skb)skb_set_owner_w (skb, sk);return skb;
@@ -2229,8 +2231,27 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len, *errcode = err;return NULL ; } + +struct sk_buff *sock_alloc_send_pskb (struct sock *sk,unsigned long header_len, +unsigned long data_len,int noblock, +int *errcode,int max_page_order) +{ +return sock_alloc_send_pskb_internal (sk, header_len, data_len, + noblock, errcode, max_page_order,/*gfp*/ 0 ); +}EXPORT_SYMBOL (sock_alloc_send_pskb); +struct sk_buff *sock_alloc_send_pskb_noinit (struct sock *sk, +unsigned long header_len, +unsigned long data_len, +int noblock,int *errcode, +int max_page_order) +{ +return sock_alloc_send_pskb_internal (sk, header_len, data_len, + noblock, errcode, max_page_order,/*gfp*/ __GFP_NOINIT); +} +EXPORT_SYMBOL (sock_alloc_send_pskb_noinit); +struct sk_buff *sock_alloc_send_skb (struct sock *sk,unsigned long size,int noblock,int *errcode) {
net/unix/af_unix.c +7 -6
@@ -1627,9 +1627,9 @@ static int unix_dgram_sendmsg(struct socket *sock, struct msghdr *msg,BUILD_BUG_ON (SKB_MAX_ALLOC <PAGE_SIZE ); } - skb =sock_alloc_send_pskb (sk, len - data_len, data_len, - msg->msg_flags &MSG_DONTWAIT , &err, -PAGE_ALLOC_COSTLY_ORDER ); + skb =sock_alloc_send_pskb_noinit (sk, len - data_len, data_len, + msg->msg_flags &MSG_DONTWAIT , &err, +PAGE_ALLOC_COSTLY_ORDER );if (skb ==NULL ) goto out;
@@ -1824,9 +1824,10 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg, data_len =min_t (size_t, size,PAGE_ALIGN (data_len)); - skb =sock_alloc_send_pskb (sk, size - data_len, data_len, - msg->msg_flags &MSG_DONTWAIT , &err, -get_order (UNIX_SKB_FRAGS_SZ )); + skb =sock_alloc_send_pskb_noinit (sk, size - data_len, data_len, + msg->msg_flags &MSG_DONTWAIT , + &err, +get_order (UNIX_SKB_FRAGS_SZ ));if (!skb) goto out_err;
Signature · 1 line
2.21.0.392.gf8f6787159e-goog
Re: [PATCH 3/3] RFC: net: apply __GFP_NOINIT to AF_UNIX sk_buff allocations
Quoted 4 lines · Alexander Potapenko
Add
sock_alloc_send_pskb_noinit(), which is similar tosock_alloc_send_pskb(), but allocates with__GFP_NOINIT. This helps reduce the slowdown on hackbench from 9% to 0.1%.
I would include a detailed justification about why this is safe to do. I imagine (but haven't looked) that the skb is immediately written to after allocation, so this is basically avoiding a "double init". Is that correct?
Signature · 1 line
Kees Cook
Quoted 26 lines · Alexander Potapenko
Following the recent discussions here's another take at initializing pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
The patchset introduces a new boot option,
init_allocations, which makes page allocator and SL[AOU]B initialize newly allocated memory. init_allocations=0 doesn't (hopefully) add any overhead to the allocation fast path (no noticeable slowdown on hackbench).With only the the first of the proposed patches the slowdown numbers are:
- 1.1% (stdev 0.2%) sys time slowdown building Linux kernel
- 3.1% (stdev 0.3%) sys time slowdown onaf_inet_loopbackbenchmark
- 9.4% (stdev 0.5%) sys time slowdown on hackbenchThe second patch introduces a GFP flag that allows to disable initialization for certain allocations. The third page is an example of applying it to
af_unix.c, which helps hackbench greatly.Slowdown numbers for the whole patchset are:
- 1.8% (stdev 0.8%) on kernel build
- 6.5% (stdev 0.2%) onaf_inet_loopback
- 0.12% (stdev 0.6%) on hackbench
Quoted 4 lines
Alexander Potapenko (3):
mm: security: introduce the init_allocations=1 boot option
gfp: mm: introduce__GFP_NOINIT
net: apply__GFP_NOINITtoAF_UNIXsk_buffallocations
Oops, I was hoping git send-email would pull all the Cc: tags from the patches and actually use them.
Quoted 21 lines
drivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/gfp.h | 6 ++++- include/linux/mm.h | 8 +++++++ include/linux/slab_def.h | 1 + include/linux/slub_def.h | 1 + include/net/sock.h | 5 +++++ kernel/kexec_core.c | 4 ++-- mm/dmapool.c | 2 +- mm/page_alloc.c | 18 ++++++++++++++- mm/slab.c | 14 ++++++------ mm/slab.h | 1 + mm/slab_common.c | 15 +++++++++++++ mm/slob.c | 3 ++- mm/slub.c | 9 ++++---- net/core/sock.c | 31 +++++++++++++++++++++----- net/unix/af_unix.c | 13 ++++++----- 16 files changed, 104 insertions(+), 29 deletions(-)--
2.21.0.392.gf8f6787159e-goog
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Quoted 10 lines · Alexander Potapenko
Following the recent discussions here's another take at initializing pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
The patchset introduces a new boot option,
init_allocations, which makes page allocator and SL[AOU]B initialize newly allocated memory. init_allocations=0 doesn't (hopefully) add any overhead to the allocation fast path (no noticeable slowdown on hackbench).
Quoted 7 lines
With only the the first of the proposed patches the slowdown numbers are:
- 1.1% (stdev 0.2%) sys time slowdown building Linux kernel
- 3.1% (stdev 0.3%) sys time slowdown onaf_inet_loopbackbenchmark
- 9.4% (stdev 0.5%) sys time slowdown on hackbenchThe second patch introduces a GFP flag that allows to disable initialization for certain allocations. The third page is an example of
third patch
Quoted 31 lines
applying it to
af_unix.c, which helps hackbench greatly.Slowdown numbers for the whole patchset are:
- 1.8% (stdev 0.8%) on kernel build
- 6.5% (stdev 0.2%) onaf_inet_loopback
- 0.12% (stdev 0.6%) on hackbenchAlexander Potapenko (3):
mm: security: introduce the init_allocations=1 boot option
gfp: mm: introduce__GFP_NOINIT
net: apply__GFP_NOINITtoAF_UNIXsk_buffallocationsdrivers/infiniband/core/uverbs_ioctl.c | 2 +- include/linux/gfp.h | 6 ++++- include/linux/mm.h | 8 +++++++ include/linux/slab_def.h | 1 + include/linux/slub_def.h | 1 + include/net/sock.h | 5 +++++ kernel/kexec_core.c | 4 ++-- mm/dmapool.c | 2 +- mm/page_alloc.c | 18 ++++++++++++++- mm/slab.c | 14 ++++++------ mm/slab.h | 1 + mm/slab_common.c | 15 +++++++++++++ mm/slob.c | 3 ++- mm/slub.c | 9 ++++---- net/core/sock.c | 31 +++++++++++++++++++++----- net/unix/af_unix.c | 13 ++++++----- 16 files changed, 104 insertions(+), 29 deletions(-)
Signature · 1 line
~Randy
Quoted 10 lines · Alexander Potapenko
Following the recent discussions here's another take at initializing pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
The patchset introduces a new boot option,
init_allocations, which makes page allocator and SL[AOU]B initialize newly allocated memory. init_allocations=0 doesn't (hopefully) add any overhead to the allocation fast path (no noticeable slowdown on hackbench).
I continue to prefer to have a way to both at-allocation initialization _and_ poison-on-free, so let's not redirect this to doing it only at free time. We're going to need both hooks when doing Memory Tagging, so let's just get it in place now. The security benefits on tagging, IMO, easily justify a 1-2% performance hit. And likely we'll see this improve with new hardware.
Quoted 5 lines
With only the the first of the proposed patches the slowdown numbers are:
- 1.1% (stdev 0.2%) sys time slowdown building Linux kernel
- 3.1% (stdev 0.3%) sys time slowdown onaf_inet_loopbackbenchmark
- 9.4% (stdev 0.5%) sys time slowdown on hackbench
Quoted 7 lines
The second patch introduces a GFP flag that allows to disable initialization for certain allocations. The third page is an example of applying it to
af_unix.c, which helps hackbench greatly.Slowdown numbers for the whole patchset are:
- 1.8% (stdev 0.8%) on kernel build
- 6.5% (stdev 0.2%) onaf_inet_loopback
Any idea why thes two went _up_?
Quoted 1 line
- 0.12% (stdev 0.6%) on hackbench
Well that's quite an improvement. :)
Signature · 1 line
Kees Cook
Quoted 8 lines · Kees Cook
Quoted 6 lines · Alexander Potapenko
Following the recent discussions here's another take at initializing pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.
Quoted 8 lines
Quoted 4 lines
The patchset introduces a new boot option,
init_allocations, which makes page allocator and SL[AOU]B initialize newly allocated memory. init_allocations=0 doesn't (hopefully) add any overhead to the allocation fast path (no noticeable slowdown on hackbench).I continue to prefer to have a way to both at-allocation initialization _and_ poison-on-free, so let's not redirect this to doing it only at free time.
There's a problem with poison-on-free.
By default SLUB stores the freelist pointer (not sure if it's the only piece of data) in the memory chunk itself, so newly allocated memory is dirty despite it has been zeroed out previously.
We could probably zero out the bits used by the allocator when allocating the memory chunk, but it sounds hacky (yet saves us 8 bytes on every allocation)
A cleaner solution would be to unconditionally relocate the free pointer by short-circuiting
https://elixir.bootlin.com/linux/latest/source/mm/slub.c#L3531
Surprisingly, this doesn't work, because now the sizes of slub caches are a bit off, and create_unique_id() in slub.c returns clashing sysfs names.
Quoted 26 lines
We're going to need both hooks when doing
Memory Tagging, so let's just get it in place now. The security benefits on tagging, IMO, easily justify a 1-2% performance hit. And likely we'll see this improve with new hardware.Quoted 5 lines
With only the the first of the proposed patches the slowdown numbers are:
- 1.1% (stdev 0.2%) sys time slowdown building Linux kernel
- 3.1% (stdev 0.3%) sys time slowdown onaf_inet_loopbackbenchmark
- 9.4% (stdev 0.5%) sys time slowdown on hackbenchQuoted 7 lines
The second patch introduces a GFP flag that allows to disable initialization for certain allocations. The third page is an example of applying it to
af_unix.c, which helps hackbench greatly.Slowdown numbers for the whole patchset are:
- 1.8% (stdev 0.8%) on kernel build
- 6.5% (stdev 0.2%) onaf_inet_loopbackAny idea why thes two went _up_?
Quoted 1 line
- 0.12% (stdev 0.6%) on hackbench
Well that's quite an improvement. :)
--
Kees Cook
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg