Skip to contentSkip to conversation

RFC: add init_allocations=1 boot option

PATCH In review Alexander Potapenko 6 participants 24 messages Apr 18, 2019 – Apr 26, 2019 1 message not in the archive

Subsystems (11)
lore
Share & tools
lore
#0 lore raw

cover RFC: add init_allocations=1 boot option

Following the recent discussions here's another take at initializing pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

The patchset introduces a new boot option, init_allocations, which makes page allocator and SL[AOU]B initialize newly allocated memory. init_allocations=0 doesn't (hopefully) add any overhead to the allocation fast path (no noticeable slowdown on hackbench).

With only the the first of the proposed patches the slowdown numbers are:
- 1.1% (stdev 0.2%) sys time slowdown building Linux kernel
- 3.1% (stdev 0.3%) sys time slowdown on af_inet_loopback benchmark
- 9.4% (stdev 0.5%) sys time slowdown on hackbench

The second patch introduces a GFP flag that allows to disable initialization for certain allocations. The third page is an example of applying it to af_unix.c, which helps hackbench greatly.

Slowdown numbers for the whole patchset are:
- 1.8% (stdev 0.8%) on kernel build
- 6.5% (stdev 0.2%) on af_inet_loopback
- 0.12% (stdev 0.6%) on hackbench

Alexander Potapenko (3):
mm: security: introduce the init_allocations=1 boot option
gfp: mm: introduce __GFP_NOINIT
net: apply __GFP_NOINIT to AF_UNIX sk_buff allocations

 drivers/infiniband/core/uverbs_ioctl.c |  2 +-
 include/linux/gfp.h                    |  6 ++++-
 include/linux/mm.h                     |  8 +++++++
 include/linux/slab_def.h               |  1 +
 include/linux/slub_def.h               |  1 +
 include/net/sock.h                     |  5 +++++
 kernel/kexec_core.c                    |  4 ++--
 mm/dmapool.c                           |  2 +-
 mm/page_alloc.c                        | 18 ++++++++++++++-
 mm/slab.c                              | 14 ++++++------
 mm/slab.h                              |  1 +
 mm/slab_common.c                       | 15 +++++++++++++
 mm/slob.c                              |  3 ++-
 mm/slub.c                              |  9 ++++----
 net/core/sock.c                        | 31 +++++++++++++++++++++-----
 net/unix/af_unix.c                     | 13 ++++++-----
 16 files changed, 104 insertions(+), 29 deletions(-)
Signature · 1 line
2.21.0.392.gf8f6787159e-goog
#1 lore raw

PATCH 1/3 mm: security: introduce the init_allocations=1 boot option

This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed. We don't initialize slab caches with constructors to preserve their semantics. To reduce runtime costs of checking cachep->ctor we replace a call to memset with a call to cachep->poison_fn, which is only executed if the memory block needs to be initialized.

For kernel testing purposes filling allocations with a nonzero pattern would be more suitable, but may require platform-specific code. To have a simple baseline we've decided to start with zero-initialization.

No performance optimizations are done at the moment to reduce double initialization of memory regions.

Cc Andrew Morton <[email protected]>
Cc James Morris <[email protected]>
Cc "Serge E. Hallyn" <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Kostya Serebryany <[email protected]>
Cc Dmitry Vyukov <[email protected]>
Cc Kees Cook <[email protected]>
Cc Sandeep Patil <[email protected]>
Cc Laura Abbott <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Jann Horn <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Qian Cai <[email protected]>
Cc Vlastimil Babka <[email protected]>
---
 drivers/infiniband/core/uverbs_ioctl.c |  2 +-
 include/linux/mm.h                     |  8 ++++++++
 include/linux/slab_def.h               |  1 +
 include/linux/slub_def.h               |  1 +
 kernel/kexec_core.c                    |  2 +-
 mm/dmapool.c                           |  2 +-
 mm/page_alloc.c                        | 18 +++++++++++++++++-
 mm/slab.c                              | 12 ++++++------
 mm/slab.h                              |  1 +
 mm/slab_common.c                       | 15 +++++++++++++++
 mm/slob.c                              |  2 +-
 mm/slub.c                              |  8 ++++----
 net/core/sock.c                        |  2 +-
 13 files changed, 58 insertions(+), 16 deletions(-)
drivers/infiniband/core/uverbs_ioctl.c +1 -1
index e1379949e663..f31234906be2 100644
@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size,
 	res = (void *)pbundle->internal_buffer + pbundle->internal_used;
 	pbundle->internal_used =
 		ALIGN(new_used, sizeof(*pbundle->internal_buffer));
-	if (flags & __GFP_ZERO)
+	if (want_init_memory(flags))
 		memset(res, 0, size);
 	return res;
 }
include/linux/mm.h +8 -0
index 76769749b5a5..b38b71a5efaa 100644
@@ -2597,6 +2597,14 @@ static inline void kernel_poison_pages(struct page *page, int numpages,
 					int enable) { }
 #endif
 
+DECLARE_STATIC_KEY_FALSE(init_allocations);
+static inline bool want_init_memory(gfp_t flags)
+{
+	if (static_branch_unlikely(&init_allocations))
+		return true;
+	return flags & __GFP_ZERO;
+}
+
 #ifdef CONFIG_DEBUG_PAGEALLOC
 extern bool _debug_pagealloc_enabled;
 extern void __kernel_map_pages(struct page *page, int numpages, int enable);
include/linux/slab_def.h +1 -0
index 9a5eafb7145b..9dfe9eb639d7 100644
@@ -37,6 +37,7 @@ struct kmem_cache {
 
 	/* constructor func */
 	void (*ctor)(void *obj);
+	void (*poison_fn)(struct kmem_cache *c, void *object);
 
 /* 4) cache creation/removal */
 	const char *name;
include/linux/slub_def.h +1 -0
index d2153789bd9f..afb928cb7c20 100644
@@ -99,6 +99,7 @@ struct kmem_cache {
 	gfp_t allocflags;	/* gfp flags to use on each alloc */
 	int refcount;		/* Refcount for slab cache destroy */
 	void (*ctor)(void *);
+	void (*poison_fn)(struct kmem_cache *c, void *object);
 	unsigned int inuse;		/* Offset to metadata */
 	unsigned int align;		/* Alignment */
 	unsigned int red_left_pad;	/* Left redzone padding size */
kernel/kexec_core.c +1 -1
index d7140447be75..be84f5f95c97 100644
@@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)
 		arch_kexec_post_alloc_pages(page_address(pages), count,
 					    gfp_mask);
 
-		if (gfp_mask & __GFP_ZERO)
+		if (want_init_memory(gfp_mask))
 			for (i = 0; i < count; i++)
 				clear_highpage(pages + i);
 	}
mm/dmapool.c +1 -1
index 76a160083506..796e38160d39 100644
@@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,
 #endif
 	spin_unlock_irqrestore(&pool->lock, flags);
 
-	if (mem_flags & __GFP_ZERO)
+	if (want_init_memory(mem_flags))
 		memset(retval, 0, pool->size);
 
 	return retval;
mm/page_alloc.c +17 -1
index d96ca5bc555b..e2a21d866ac9 100644
@@ -133,6 +133,22 @@ unsigned long totalcma_pages __read_mostly;
 
 int percpu_pagelist_fraction;
 gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK;
+bool want_init_allocations __read_mostly;
+EXPORT_SYMBOL(want_init_allocations);
+DEFINE_STATIC_KEY_FALSE(init_allocations);
+
+static int __init early_init_allocations(char *buf)
+{
+	int ret;
+
+	if (!buf)
+		return -EINVAL;
+	ret = kstrtobool(buf, &want_init_allocations);
+	if (want_init_allocations)
+		static_branch_enable(&init_allocations);
+	return ret;
+}
+early_param("init_allocations", early_init_allocations);
 
 /*
  * A cached value of the page's pageblock's migratetype, used when the page is
@@ -2014,7 +2030,7 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags
 
 	post_alloc_hook(page, order, gfp_flags);
 
-	if (!free_pages_prezeroed() && (gfp_flags & __GFP_ZERO))
+	if (!free_pages_prezeroed() && want_init_memory(gfp_flags))
 		for (i = 0; i < (1 << order); i++)
 			clear_highpage(page + i);
 
mm/slab.c +6 -6
index 47a380a486ee..dcc5b73cf767 100644
@@ -3331,8 +3331,8 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,
 	local_irq_restore(save_flags);
 	ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller);
 
-	if (unlikely(flags & __GFP_ZERO) && ptr)
-		memset(ptr, 0, cachep->object_size);
+	if (unlikely(want_init_memory(flags)) && ptr)
+		cachep->poison_fn(cachep, ptr);
 
 	slab_post_alloc_hook(cachep, flags, 1, &ptr);
 	return ptr;
@@ -3388,8 +3388,8 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)
 	objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller);
 	prefetchw(objp);
 
-	if (unlikely(flags & __GFP_ZERO) && objp)
-		memset(objp, 0, cachep->object_size);
+	if (unlikely(want_init_memory(flags)) && objp)
+		cachep->poison_fn(cachep, objp);
 
 	slab_post_alloc_hook(cachep, flags, 1, &objp);
 	return objp;
@@ -3596,9 +3596,9 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
 	cache_alloc_debugcheck_after_bulk(s, flags, size, p, _RET_IP_);
 
 	/* Clear memory outside IRQ disabled section */
-	if (unlikely(flags & __GFP_ZERO))
+	if (unlikely(want_init_memory(flags)))
 		for (i = 0; i < size; i++)
-			memset(p[i], 0, s->object_size);
+			s->poison_fn(s, p[i]);
 
 	slab_post_alloc_hook(s, flags, size, p);
 	/* FIXME: Trace call missing. Christoph would like a bulk variant */
mm/slab.h +1 -0
index 43ac818b8592..3b541e8970ee 100644
@@ -27,6 +27,7 @@ struct kmem_cache {
 	const char *name;	/* Slab name for sysfs */
 	int refcount;		/* Use counter */
 	void (*ctor)(void *);	/* Called on object slot creation */
+	void (*poison_fn)(struct kmem_cache *c, void *object);
 	struct list_head list;	/* List of all slab caches on the system */
 };
 
mm/slab_common.c +15 -0
index 58251ba63e4a..37810114b2ea 100644
@@ -360,6 +360,16 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align,
 	return NULL;
 }
 
+static void poison_zero(struct kmem_cache *c, void *object)
+{
+	memset(object, 0, c->object_size);
+}
+
+static void poison_dont(struct kmem_cache *c, void *object)
+{
+	/* Do nothing. Use for caches with constructors. */
+}
+
 static struct kmem_cache *create_cache(const char *name,
 		unsigned int object_size, unsigned int align,
 		slab_flags_t flags, unsigned int useroffset,
@@ -381,6 +391,10 @@ static struct kmem_cache *create_cache(const char *name,
 	s->size = s->object_size = object_size;
 	s->align = align;
 	s->ctor = ctor;
+	if (ctor)
+		s->poison_fn = poison_dont;
+	else
+		s->poison_fn = poison_zero;
 	s->useroffset = useroffset;
 	s->usersize = usersize;
 
@@ -974,6 +988,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name,
 	s->align = calculate_alignment(flags, ARCH_KMALLOC_MINALIGN, size);
 	s->useroffset = useroffset;
 	s->usersize = usersize;
+	s->poison_fn = poison_zero;
 
 	slab_init_memcg_params(s);
 
mm/slob.c +1 -1
index 307c2c9feb44..18981a71e962 100644
@@ -330,7 +330,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node)
 		BUG_ON(!b);
 		spin_unlock_irqrestore(&slob_lock, flags);
 	}
-	if (unlikely(gfp & __GFP_ZERO))
+	if (unlikely(want_init_memory(gfp)))
 		memset(b, 0, size);
 	return b;
 }
mm/slub.c +4 -4
index d30ede89f4a6..e4efb6575510 100644
@@ -2750,8 +2750,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,
 		stat(s, ALLOC_FASTPATH);
 	}
 
-	if (unlikely(gfpflags & __GFP_ZERO) && object)
-		memset(object, 0, s->object_size);
+	if (unlikely(want_init_memory(gfpflags)) && object)
+		s->poison_fn(s, object);
 
 	slab_post_alloc_hook(s, gfpflags, 1, &object);
 
@@ -3172,11 +3172,11 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
 	local_irq_enable();
 
 	/* Clear memory outside IRQ disabled fastpath loop */
-	if (unlikely(flags & __GFP_ZERO)) {
+	if (unlikely(want_init_memory(flags))) {
 		int j;
 
 		for (j = 0; j < i; j++)
-			memset(p[j], 0, s->object_size);
+			s->poison_fn(s, p[j]);
 	}
 
 	/* memcg and kmem_cache debug support */
net/core/sock.c +1 -1
index 782343bb925b..99b288a19b39 100644
@@ -1601,7 +1601,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority,
 		sk = kmem_cache_alloc(slab, priority & ~__GFP_ZERO);
 		if (!sk)
 			return sk;
-		if (priority & __GFP_ZERO)
+		if (want_init_memory(priority))
 			sk_prot_clear_nulls(sk, prot->obj_size);
 	} else
 		sk = kmalloc(prot->obj_size, priority);
Signature · 1 line
2.21.0.392.gf8f6787159e-goog
#2 lore raw

Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option

Quoted 4 lines · Alexander Potapenko

This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

Isn't it better to do this at free time rather than allocation time? If doing it at free, you can't even have information leaks for pages that are in the allocator.

#3 lore raw

Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option

Quoted 10 lines · Dave Hansen
Quoted 4 lines · Alexander Potapenko

This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

Isn't it better to do this at free time rather than allocation time? If doing it at free, you can't even have information leaks for pages that are in the allocator.

I should have mentioned this in the patch description, as this question is being asked every time I send a patch :)
If we want to avoid double initialization and take advantage of __GFP_NOINIT (see the second and third patches in the series) we need to do initialize the memory at allocation time, because free() and free_pages() don't accept GFP flags.

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#4 lore raw

Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option

Quoted 12 lines · Alexander Potapenko
Quoted 10 lines · Dave Hansen
Quoted 4 lines · Alexander Potapenko

This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

Isn't it better to do this at free time rather than allocation time? If doing it at free, you can't even have information leaks for pages that are in the allocator.

Quoted 6 lines

I should have mentioned this in the patch description, as this question is being asked every time I send a patch :)
If we want to avoid double initialization and take advantage of __GFP_NOINIT (see the second and third patches in the series) we need to do initialize the memory at allocation time, because free() and free_pages() don't accept GFP flags.

On a second thought, double zeroing on memory reclaim should be quite rare. Most of the speedup we gain with __GFP_NOINIT is because we assume it's safe to not initialize memory that'll be overwritten anyway. I'll need to check how e.g. hackbench behaves if we choose to zero memory on free() (my guess would be it'll be slower than with __GFP_NOINIT hack, albeit a little safer)

Quoted 13 lines

--
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#5 lore raw

Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option

In reply to Dave Hansen
Quoted 9 lines · Dave Hansen
Quoted 4 lines · Alexander Potapenko

This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

Isn't it better to do this at free time rather than allocation time? If doing it at free, you can't even have information leaks for pages that are in the allocator.

I would tend to agree here. Free path is usually less performance sensitive than the allocation. Those really hot paths tend to defer the work.

I am also worried that an opt-out gfp flag would tend to be used incorrectly as the history has shown for others - e.g. __GFP_TEMPORARY. So I would rather see this robust without a fine tuning unless there is real use case that would suffer from this and we can think of a background scrubbing or something similar.

Signature · 2 lines
Michal Hocko
SUSE Labs
#6 lore raw

Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option

In reply to Alexander Potapenko
Quoted 37 lines · Alexander Potapenko

This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed. We don't initialize slab caches with constructors to preserve their semantics. To reduce runtime costs of checking cachep->ctor we replace a call to memset with a call to cachep->poison_fn, which is only executed if the memory block needs to be initialized.

For kernel testing purposes filling allocations with a nonzero pattern would be more suitable, but may require platform-specific code. To have a simple baseline we've decided to start with zero-initialization.

No performance optimizations are done at the moment to reduce double initialization of memory regions.

Cc Andrew Morton <[email protected]>
Cc James Morris <[email protected]>
Cc "Serge E. Hallyn" <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Kostya Serebryany <[email protected]>
Cc Dmitry Vyukov <[email protected]>
Cc Kees Cook <[email protected]>
Cc Sandeep Patil <[email protected]>
Cc Laura Abbott <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Jann Horn <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Qian Cai <[email protected]>
Cc Vlastimil Babka <[email protected]>
Quoted 16 lines
---
 drivers/infiniband/core/uverbs_ioctl.c |  2 +-
 include/linux/mm.h                     |  8 ++++++++
 include/linux/slab_def.h               |  1 +
 include/linux/slub_def.h               |  1 +
 kernel/kexec_core.c                    |  2 +-
 mm/dmapool.c                           |  2 +-
 mm/page_alloc.c                        | 18 +++++++++++++++++-
 mm/slab.c                              | 12 ++++++------
 mm/slab.h                              |  1 +
 mm/slab_common.c                       | 15 +++++++++++++++
 mm/slob.c                              |  2 +-
 mm/slub.c                              |  8 ++++----
 net/core/sock.c                        |  2 +-
 13 files changed, 58 insertions(+), 16 deletions(-)

Hi,
Please document init_allocations=N in Documentation/admin-guide/kernel-parameters.txt.

thanks.

Signature · 1 line
~Randy
#7 lore raw

Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option

In reply to Alexander Potapenko
Quoted 12 lines · Alexander Potapenko

This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed. We don't initialize slab caches with constructors to preserve their semantics. To reduce runtime costs of checking cachep->ctor we replace a call to memset with a call to cachep->poison_fn, which is only executed if the memory block needs to be initialized.

Quoted 3 lines

For kernel testing purposes filling allocations with a nonzero pattern would be more suitable, but may require platform-specific code. To have a simple baseline we've decided to start with zero-initialization.

The memory tagging future may be worth mentioning here too. We'll always need an allocation-time hook. What we do in that hook (tag, zero, poison) can be manage from there.

Quoted 2 lines

No performance optimizations are done at the moment to reduce double initialization of memory regions.

Isn't this addressed in later patches?

Quoted 53 lines
Cc Andrew Morton <[email protected]>
Cc James Morris <[email protected]>
Cc "Serge E. Hallyn" <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Kostya Serebryany <[email protected]>
Cc Dmitry Vyukov <[email protected]>
Cc Kees Cook <[email protected]>
Cc Sandeep Patil <[email protected]>
Cc Laura Abbott <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Jann Horn <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Qian Cai <[email protected]>
Cc Vlastimil Babka <[email protected]>
---
 drivers/infiniband/core/uverbs_ioctl.c |  2 +-
 include/linux/mm.h                     |  8 ++++++++
 include/linux/slab_def.h               |  1 +
 include/linux/slub_def.h               |  1 +
 kernel/kexec_core.c                    |  2 +-
 mm/dmapool.c                           |  2 +-
 mm/page_alloc.c                        | 18 +++++++++++++++++-
 mm/slab.c                              | 12 ++++++------
 mm/slab.h                              |  1 +
 mm/slab_common.c                       | 15 +++++++++++++++
 mm/slob.c                              |  2 +-
 mm/slub.c                              |  8 ++++----
 net/core/sock.c                        |  2 +-
 13 files changed, 58 insertions(+), 16 deletions(-)
drivers/infiniband/core/uverbs_ioctl.c +1 -1
index e1379949e663..f31234906be2 100644
@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size,
        res = (void *)pbundle->internal_buffer + pbundle->internal_used;
        pbundle->internal_used =
                ALIGN(new_used, sizeof(*pbundle->internal_buffer));
-       if (flags & __GFP_ZERO)
+       if (want_init_memory(flags))
                memset(res, 0, size);
        return res;
 }
include/linux/mm.h +0 -0
index 76769749b5a5..b38b71a5efaa 100644

include/linux/mm.h static inline void kernel_poison_pages(struct page *page, int numpages, · line 2600

Quoted 4 lines
include/linux/mm.h +1 -0
@@ -2597,6 +2597,14 @@ static inline void kernel_poison_pages(struct page *page, int numpages,
                                        int enable) { }
 #endif
 
+DECLARE_STATIC_KEY_FALSE(init_allocations);

I'd like a CONFIG to control this default. We can keep the boot-time option to change it, but I think a CONFIG is warranted here.

Quoted 65 lines
include/linux/mm.h +7 -0
+static inline bool want_init_memory(gfp_t flags)
+{
+       if (static_branch_unlikely(&init_allocations))
+               return true;
+       return flags & __GFP_ZERO;
+}
+
 #ifdef CONFIG_DEBUG_PAGEALLOC
 extern bool _debug_pagealloc_enabled;
 extern void __kernel_map_pages(struct page *page, int numpages, int enable);
include/linux/slab_def.h +1 -0
index 9a5eafb7145b..9dfe9eb639d7 100644
@@ -37,6 +37,7 @@ struct kmem_cache {
 
        /* constructor func */
        void (*ctor)(void *obj);
+       void (*poison_fn)(struct kmem_cache *c, void *object);
 
 /* 4) cache creation/removal */
        const char *name;
include/linux/slub_def.h +1 -0
index d2153789bd9f..afb928cb7c20 100644
@@ -99,6 +99,7 @@ struct kmem_cache {
        gfp_t allocflags;       /* gfp flags to use on each alloc */
        int refcount;           /* Refcount for slab cache destroy */
        void (*ctor)(void *);
+       void (*poison_fn)(struct kmem_cache *c, void *object);
        unsigned int inuse;             /* Offset to metadata */
        unsigned int align;             /* Alignment */
        unsigned int red_left_pad;      /* Left redzone padding size */
kernel/kexec_core.c +1 -1
index d7140447be75..be84f5f95c97 100644
@@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)
                arch_kexec_post_alloc_pages(page_address(pages), count,
                                            gfp_mask);
 
-               if (gfp_mask & __GFP_ZERO)
+               if (want_init_memory(gfp_mask))
                        for (i = 0; i < count; i++)
                                clear_highpage(pages + i);
        }
mm/dmapool.c +1 -1
index 76a160083506..796e38160d39 100644
@@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,
 #endif
        spin_unlock_irqrestore(&pool->lock, flags);
 
-       if (mem_flags & __GFP_ZERO)
+       if (want_init_memory(mem_flags))
                memset(retval, 0, pool->size);
 
        return retval;
mm/page_alloc.c +0 -0
index d96ca5bc555b..e2a21d866ac9 100644

mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 136

Quoted 4 lines
mm/page_alloc.c +1 -0
@@ -133,6 +133,22 @@ unsigned long totalcma_pages __read_mostly;
 
 int percpu_pagelist_fraction;
 gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK;
+bool want_init_allocations __read_mostly;

This can be a stack variable in early_init_allocations() -- it's never used again.

mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 148

Quoted 12 lines
mm/page_alloc.c +12 -0
+EXPORT_SYMBOL(want_init_allocations);
+DEFINE_STATIC_KEY_FALSE(init_allocations);
+
+static int __init early_init_allocations(char *buf)
+{
+       int ret;
+
+       if (!buf)
+               return -EINVAL;
+       ret = kstrtobool(buf, &want_init_allocations);
+       if (want_init_allocations)
+               static_branch_enable(&init_allocations);

With the CONFIG, this should have a _disable on an "else" here...

mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 151

Quoted 3 lines
mm/page_alloc.c +3 -0
+       return ret;
+}
+early_param("init_allocations", early_init_allocations);

Does early_init_allocations() get called before things like prep_new_page() are up and running to call want_init_memory()?

Quoted 17 lines
mm/page_alloc.c +1 -1
 /*
  * A cached value of the page's pageblock's migratetype, used when the page is
@@ -2014,7 +2030,7 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags
 
        post_alloc_hook(page, order, gfp_flags);
 
-       if (!free_pages_prezeroed() && (gfp_flags & __GFP_ZERO))
+       if (!free_pages_prezeroed() && want_init_memory(gfp_flags))
                for (i = 0; i < (1 << order); i++)
                        clear_highpage(page + i);
 
mm/slab.c +0 -0
index 47a380a486ee..dcc5b73cf767 100644

mm/slab.c slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, · line 3335

Quoted 7 lines
mm/slab.c +2 -2
@@ -3331,8 +3331,8 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,
        local_irq_restore(save_flags);
        ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller);
 
-       if (unlikely(flags & __GFP_ZERO) && ptr)
-               memset(ptr, 0, cachep->object_size);
+       if (unlikely(want_init_memory(flags)) && ptr)
+               cachep->poison_fn(cachep, ptr);

So... this _must_ zero when __GFP_ZERO is present, so I'm not sure "poison_fn" is the right name, and it likely needs to take the "flags" argument.

mm/slab.c slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) · line 3392

Quoted 11 lines
mm/slab.c +2 -2
        slab_post_alloc_hook(cachep, flags, 1, &ptr);
        return ptr;
@@ -3388,8 +3388,8 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)
        objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller);
        prefetchw(objp);
 
-       if (unlikely(flags & __GFP_ZERO) && objp)
-               memset(objp, 0, cachep->object_size);
+       if (unlikely(want_init_memory(flags)) && objp)
+               cachep->poison_fn(cachep, objp);

Same.

mm/slab.c int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, · line 3601

Quoted 12 lines
mm/slab.c +2 -2
        slab_post_alloc_hook(cachep, flags, 1, &objp);
        return objp;
@@ -3596,9 +3596,9 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
        cache_alloc_debugcheck_after_bulk(s, flags, size, p, _RET_IP_);
 
        /* Clear memory outside IRQ disabled section */
-       if (unlikely(flags & __GFP_ZERO))
+       if (unlikely(want_init_memory(flags)))
                for (i = 0; i < size; i++)
-                       memset(p[i], 0, s->object_size);
+                       s->poison_fn(s, p[i]);

Same.

mm/slab.h struct kmem_cache { · line 30

Quoted 12 lines
mm/slab.c +0 -0
        slab_post_alloc_hook(s, flags, size, p);
        /* FIXME: Trace call missing. Christoph would like a bulk variant */
mm/slab.h +1 -0
index 43ac818b8592..3b541e8970ee 100644
@@ -27,6 +27,7 @@ struct kmem_cache {
        const char *name;       /* Slab name for sysfs */
        int refcount;           /* Use counter */
        void (*ctor)(void *);   /* Called on object slot creation */
+       void (*poison_fn)(struct kmem_cache *c, void *object);

How about naming it just "initialize"?

Quoted 25 lines
mm/slab.h +0 -0
        struct list_head list;  /* List of all slab caches on the system */
 };
 
mm/slab_common.c +10 -0
index 58251ba63e4a..37810114b2ea 100644
@@ -360,6 +360,16 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align,
        return NULL;
 }
 
+static void poison_zero(struct kmem_cache *c, void *object)
+{
+       memset(object, 0, c->object_size);
+}
+
+static void poison_dont(struct kmem_cache *c, void *object)
+{
+       /* Do nothing. Use for caches with constructors. */
+}
+
 static struct kmem_cache *create_cache(const char *name,
                unsigned int object_size, unsigned int align,
                slab_flags_t flags, unsigned int useroffset,

mm/slab_common.c static struct kmem_cache *create_cache(const char *name, · line 397

Quoted 7 lines
mm/slab_common.c +4 -0
@@ -381,6 +391,10 @@ static struct kmem_cache *create_cache(const char *name,
        s->size = s->object_size = object_size;
        s->align = align;
        s->ctor = ctor;
+       if (ctor)
+               s->poison_fn = poison_dont;
+       else
+               s->poison_fn = poison_zero;

As mentioned, we must still always zero when __GFP_ZERO is present.

Quoted 67 lines
mm/slab_common.c +1 -0
        s->useroffset = useroffset;
        s->usersize = usersize;
 
@@ -974,6 +988,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name,
        s->align = calculate_alignment(flags, ARCH_KMALLOC_MINALIGN, size);
        s->useroffset = useroffset;
        s->usersize = usersize;
+       s->poison_fn = poison_zero;
 
        slab_init_memcg_params(s);
 
mm/slob.c +1 -1
index 307c2c9feb44..18981a71e962 100644
@@ -330,7 +330,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node)
                BUG_ON(!b);
                spin_unlock_irqrestore(&slob_lock, flags);
        }
-       if (unlikely(gfp & __GFP_ZERO))
+       if (unlikely(want_init_memory(gfp)))
                memset(b, 0, size);
        return b;
 }
mm/slub.c +4 -4
index d30ede89f4a6..e4efb6575510 100644
@@ -2750,8 +2750,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,
                stat(s, ALLOC_FASTPATH);
        }
 
-       if (unlikely(gfpflags & __GFP_ZERO) && object)
-               memset(object, 0, s->object_size);
+       if (unlikely(want_init_memory(gfpflags)) && object)
+               s->poison_fn(s, object);
 
        slab_post_alloc_hook(s, gfpflags, 1, &object);
 
@@ -3172,11 +3172,11 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
        local_irq_enable();
 
        /* Clear memory outside IRQ disabled fastpath loop */
-       if (unlikely(flags & __GFP_ZERO)) {
+       if (unlikely(want_init_memory(flags))) {
                int j;
 
                for (j = 0; j < i; j++)
-                       memset(p[j], 0, s->object_size);
+                       s->poison_fn(s, p[j]);
        }
 
        /* memcg and kmem_cache debug support */
net/core/sock.c +1 -2
index 782343bb925b..99b288a19b39 100644
@@ -1601,7 +1601,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority,
                sk = kmem_cache_alloc(slab, priority & ~__GFP_ZERO);
                if (!sk)
                        return sk;
-               if (priority & __GFP_ZERO)
+               if (want_init_memory(priority))
                        sk_prot_clear_nulls(sk, prot->obj_size);
        } else
                sk = kmalloc(prot->obj_size, priority);
--
Quoted 2 lines

2.21.0.392.gf8f6787159e-goog

Looking good. :) Thanks for working on this!

Signature · 1 line
Kees Cook
#8 lore raw

Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option

Quoted 14 lines · Kees Cook
Quoted 12 lines · Alexander Potapenko

This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed. We don't initialize slab caches with constructors to preserve their semantics. To reduce runtime costs of checking cachep->ctor we replace a call to memset with a call to cachep->poison_fn, which is only executed if the memory block needs to be initialized.

Quoted 7 lines
Quoted 3 lines

For kernel testing purposes filling allocations with a nonzero pattern would be more suitable, but may require platform-specific code. To have a simple baseline we've decided to start with zero-initialization.

The memory tagging future may be worth mentioning here too. We'll always need an allocation-time hook. What we do in that hook (tag, zero, poison) can be manage from there.

Shall we factor out the allocation hook in this patch? Note that I'll be probably dropping this poison_fn() stuff, as it should be enough to just call memset() under a static branch.

Quoted 4 lines
Quoted 2 lines

No performance optimizations are done at the moment to reduce double initialization of memory regions.

Isn't this addressed in later patches?

Agreed.

Quoted 56 lines
Quoted 56 lines
Cc Andrew Morton <[email protected]>
Cc James Morris <[email protected]>
Cc "Serge E. Hallyn" <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Kostya Serebryany <[email protected]>
Cc Dmitry Vyukov <[email protected]>
Cc Kees Cook <[email protected]>
Cc Sandeep Patil <[email protected]>
Cc Laura Abbott <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Jann Horn <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Qian Cai <[email protected]>
Cc Vlastimil Babka <[email protected]>
---
 drivers/infiniband/core/uverbs_ioctl.c |  2 +-
 include/linux/mm.h                     |  8 ++++++++
 include/linux/slab_def.h               |  1 +
 include/linux/slub_def.h               |  1 +
 kernel/kexec_core.c                    |  2 +-
 mm/dmapool.c                           |  2 +-
 mm/page_alloc.c                        | 18 +++++++++++++++++-
 mm/slab.c                              | 12 ++++++------
 mm/slab.h                              |  1 +
 mm/slab_common.c                       | 15 +++++++++++++++
 mm/slob.c                              |  2 +-
 mm/slub.c                              |  8 ++++----
 net/core/sock.c                        |  2 +-
 13 files changed, 58 insertions(+), 16 deletions(-)
drivers/infiniband/core/uverbs_ioctl.c +1 -1
index e1379949e663..f31234906be2 100644
@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size,
        res = (void *)pbundle->internal_buffer + pbundle->internal_used;
        pbundle->internal_used =
                ALIGN(new_used, sizeof(*pbundle->internal_buffer));
-       if (flags & __GFP_ZERO)
+       if (want_init_memory(flags))
                memset(res, 0, size);
        return res;
 }
include/linux/mm.h +0 -0
index 76769749b5a5..b38b71a5efaa 100644
@@ -2597,6 +2597,14 @@ static inline void kernel_poison_pages(struct page *page, int numpages,
                                        int enable) { }
 #endif
 
Quoted 4 lines

include/linux/mm.h static inline void kernel_poison_pages(struct page *page, int numpages, · line 2600

Quoted 1 line
include/linux/mm.h +1 -0
+DECLARE_STATIC_KEY_FALSE(init_allocations);

I'd like a CONFIG to control this default. We can keep the boot-time option to change it, but I think a CONFIG is warranted here.

Ok, will do.

Quoted 66 lines
Quoted 66 lines
include/linux/mm.h +7 -0
+static inline bool want_init_memory(gfp_t flags)
+{
+       if (static_branch_unlikely(&init_allocations))
+               return true;
+       return flags & __GFP_ZERO;
+}
+
 #ifdef CONFIG_DEBUG_PAGEALLOC
 extern bool _debug_pagealloc_enabled;
 extern void __kernel_map_pages(struct page *page, int numpages, int enable);
include/linux/slab_def.h +1 -0
index 9a5eafb7145b..9dfe9eb639d7 100644
@@ -37,6 +37,7 @@ struct kmem_cache {
 
        /* constructor func */
        void (*ctor)(void *obj);
+       void (*poison_fn)(struct kmem_cache *c, void *object);
 
 /* 4) cache creation/removal */
        const char *name;
include/linux/slub_def.h +1 -0
index d2153789bd9f..afb928cb7c20 100644
@@ -99,6 +99,7 @@ struct kmem_cache {
        gfp_t allocflags;       /* gfp flags to use on each alloc */
        int refcount;           /* Refcount for slab cache destroy */
        void (*ctor)(void *);
+       void (*poison_fn)(struct kmem_cache *c, void *object);
        unsigned int inuse;             /* Offset to metadata */
        unsigned int align;             /* Alignment */
        unsigned int red_left_pad;      /* Left redzone padding size */
kernel/kexec_core.c +1 -1
index d7140447be75..be84f5f95c97 100644
@@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)
                arch_kexec_post_alloc_pages(page_address(pages), count,
                                            gfp_mask);
 
-               if (gfp_mask & __GFP_ZERO)
+               if (want_init_memory(gfp_mask))
                        for (i = 0; i < count; i++)
                                clear_highpage(pages + i);
        }
mm/dmapool.c +1 -1
index 76a160083506..796e38160d39 100644
@@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,
 #endif
        spin_unlock_irqrestore(&pool->lock, flags);
 
-       if (mem_flags & __GFP_ZERO)
+       if (want_init_memory(mem_flags))
                memset(retval, 0, pool->size);
 
        return retval;
mm/page_alloc.c +0 -0
index d96ca5bc555b..e2a21d866ac9 100644
@@ -133,6 +133,22 @@ unsigned long totalcma_pages __read_mostly;
 
Quoted 6 lines

mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 136

Quoted 3 lines
mm/page_alloc.c +1 -0
 int percpu_pagelist_fraction;
 gfp_t gfp_allowed_mask __read_mostly = GFP_BOOT_MASK;
+bool want_init_allocations __read_mostly;

This can be a stack variable in early_init_allocations() -- it's never used again.

Ack.

Quoted 6 lines
Quoted 6 lines
mm/page_alloc.c +6 -0
+EXPORT_SYMBOL(want_init_allocations);
+DEFINE_STATIC_KEY_FALSE(init_allocations);
+
+static int __init early_init_allocations(char *buf)
+{
+       int ret;
Quoted 8 lines

mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 148

Quoted 6 lines
mm/page_alloc.c +6 -0
+
+       if (!buf)
+               return -EINVAL;
+       ret = kstrtobool(buf, &want_init_allocations);
+       if (want_init_allocations)
+               static_branch_enable(&init_allocations);

With the CONFIG, this should have a _disable on an "else" here...

Ack.

Quoted 6 lines

mm/page_alloc.c unsigned long totalcma_pages __read_mostly; · line 151

Quoted 3 lines
mm/page_alloc.c +3 -0
+       return ret;
+}
+early_param("init_allocations", early_init_allocations);

Does early_init_allocations() get called before things like prep_new_page() are up and running to call want_init_memory()?

Yes, IIUC early params are initialized before the memory subsystem is initialized.

Quoted 20 lines
Quoted 20 lines
mm/page_alloc.c +1 -1
 /*
  * A cached value of the page's pageblock's migratetype, used when the page is
@@ -2014,7 +2030,7 @@ static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags
 
        post_alloc_hook(page, order, gfp_flags);
 
-       if (!free_pages_prezeroed() && (gfp_flags & __GFP_ZERO))
+       if (!free_pages_prezeroed() && want_init_memory(gfp_flags))
                for (i = 0; i < (1 << order); i++)
                        clear_highpage(page + i);
 
mm/slab.c +0 -0
index 47a380a486ee..dcc5b73cf767 100644
@@ -3331,8 +3331,8 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,
        local_irq_restore(save_flags);
        ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller);
 
Quoted 8 lines

mm/slab.c slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, · line 3335

Quoted 4 lines
mm/slab.c +2 -2
-       if (unlikely(flags & __GFP_ZERO) && ptr)
-               memset(ptr, 0, cachep->object_size);
+       if (unlikely(want_init_memory(flags)) && ptr)
+               cachep->poison_fn(cachep, ptr);

So... this _must_ zero when __GFP_ZERO is present, so I'm not sure "poison_fn" is the right name, and it likely needs to take the "flags" argument.

I'll rework this part, as it had been pointed out that an empty indirect call still costs something.
We're basically choosing between two options:
- flags & __GFP_ZERO when initialization is disabled;
- and cachep->ctr when it's enabled
A static branch should be enough to switch between those without introducing extra checks or indirections.

Quoted 31 lines

mm/slab.c slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) · line 3392

Quoted 12 lines
mm/slab.c +2 -2
        slab_post_alloc_hook(cachep, flags, 1, &ptr);
        return ptr;
@@ -3388,8 +3388,8 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)
        objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller);
        prefetchw(objp);
 
-       if (unlikely(flags & __GFP_ZERO) && objp)
-               memset(objp, 0, cachep->object_size);
+       if (unlikely(want_init_memory(flags)) && objp)
+               cachep->poison_fn(cachep, objp);

Same.

mm/slab.c int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, · line 3601

Quoted 12 lines
mm/slab.c +2 -2
        slab_post_alloc_hook(cachep, flags, 1, &objp);
        return objp;
@@ -3596,9 +3596,9 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
        cache_alloc_debugcheck_after_bulk(s, flags, size, p, _RET_IP_);
 
        /* Clear memory outside IRQ disabled section */
-       if (unlikely(flags & __GFP_ZERO))
+       if (unlikely(want_init_memory(flags)))
                for (i = 0; i < size; i++)
-                       memset(p[i], 0, s->object_size);
+                       s->poison_fn(s, p[i]);

Same.

Quoted 1 line
Quoted 13 lines

mm/slab.h struct kmem_cache { · line 30

Quoted 11 lines
        slab_post_alloc_hook(s, flags, size, p);
        /* FIXME: Trace call missing. Christoph would like a bulk variant */
mm/slab.h +1 -0
index 43ac818b8592..3b541e8970ee 100644
@@ -27,6 +27,7 @@ struct kmem_cache {
        const char *name;       /* Slab name for sysfs */
        int refcount;           /* Use counter */
        void (*ctor)(void *);   /* Called on object slot creation */
+       void (*poison_fn)(struct kmem_cache *c, void *object);

How about naming it just "initialize"?

DItto.

Quoted 26 lines
Quoted 26 lines
mm/slab.h +0 -0
        struct list_head list;  /* List of all slab caches on the system */
 };
 
mm/slab_common.c +10 -0
index 58251ba63e4a..37810114b2ea 100644
@@ -360,6 +360,16 @@ struct kmem_cache *find_mergeable(unsigned int size, unsigned int align,
        return NULL;
 }
 
+static void poison_zero(struct kmem_cache *c, void *object)
+{
+       memset(object, 0, c->object_size);
+}
+
+static void poison_dont(struct kmem_cache *c, void *object)
+{
+       /* Do nothing. Use for caches with constructors. */
+}
+
 static struct kmem_cache *create_cache(const char *name,
                unsigned int object_size, unsigned int align,
                slab_flags_t flags, unsigned int useroffset,
@@ -381,6 +391,10 @@ static struct kmem_cache *create_cache(const char *name,
        s->size = s->object_size = object_size;
Quoted 8 lines

mm/slab_common.c static struct kmem_cache *create_cache(const char *name, · line 397

Quoted 6 lines
mm/slab_common.c +4 -0
        s->align = align;
        s->ctor = ctor;
+       if (ctor)
+               s->poison_fn = poison_dont;
+       else
+               s->poison_fn = poison_zero;

As mentioned, we must still always zero when __GFP_ZERO is present.

This part will be gone, but in general __GFP_ZERO is incompatible with constructors, see a check in new_slab_objects().
I don't think we must support erroneous behaviour.

Quoted 74 lines
Quoted 67 lines
mm/slab_common.c +1 -0
        s->useroffset = useroffset;
        s->usersize = usersize;
 
@@ -974,6 +988,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name,
        s->align = calculate_alignment(flags, ARCH_KMALLOC_MINALIGN, size);
        s->useroffset = useroffset;
        s->usersize = usersize;
+       s->poison_fn = poison_zero;
 
        slab_init_memcg_params(s);
 
mm/slob.c +1 -1
index 307c2c9feb44..18981a71e962 100644
@@ -330,7 +330,7 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node)
                BUG_ON(!b);
                spin_unlock_irqrestore(&slob_lock, flags);
        }
-       if (unlikely(gfp & __GFP_ZERO))
+       if (unlikely(want_init_memory(gfp)))
                memset(b, 0, size);
        return b;
 }
mm/slub.c +4 -4
index d30ede89f4a6..e4efb6575510 100644
@@ -2750,8 +2750,8 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,
                stat(s, ALLOC_FASTPATH);
        }
 
-       if (unlikely(gfpflags & __GFP_ZERO) && object)
-               memset(object, 0, s->object_size);
+       if (unlikely(want_init_memory(gfpflags)) && object)
+               s->poison_fn(s, object);
 
        slab_post_alloc_hook(s, gfpflags, 1, &object);
 
@@ -3172,11 +3172,11 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
        local_irq_enable();
 
        /* Clear memory outside IRQ disabled fastpath loop */
-       if (unlikely(flags & __GFP_ZERO)) {
+       if (unlikely(want_init_memory(flags))) {
                int j;
 
                for (j = 0; j < i; j++)
-                       memset(p[j], 0, s->object_size);
+                       s->poison_fn(s, p[j]);
        }
 
        /* memcg and kmem_cache debug support */
net/core/sock.c +1 -2
index 782343bb925b..99b288a19b39 100644
@@ -1601,7 +1601,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority,
                sk = kmem_cache_alloc(slab, priority & ~__GFP_ZERO);
                if (!sk)
                        return sk;
-               if (priority & __GFP_ZERO)
+               if (want_init_memory(priority))
                        sk_prot_clear_nulls(sk, prot->obj_size);
        } else
                sk = kmalloc(prot->obj_size, priority);
--
Quoted 2 lines

2.21.0.392.gf8f6787159e-goog

Looking good. :) Thanks for working on this!

--
Kees Cook

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#9 lore raw

Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option

In reply to Alexander Potapenko
Quoted 20 lines · Alexander Potapenko
mm/slab_common.c +9 -0
+static void poison_dont(struct kmem_cache *c, void *object)
+{
+	/* Do nothing. Use for caches with constructors. */
+}
+
 static struct kmem_cache *create_cache(const char *name,
 		unsigned int object_size, unsigned int align,
 		slab_flags_t flags, unsigned int useroffset,
@@ -381,6 +391,10 @@ static struct kmem_cache *create_cache(const char *name,
 	s->size = s->object_size = object_size;
 	s->align = align;
 	s->ctor = ctor;
+	if (ctor)
+		s->poison_fn = poison_dont;
+	else
+		s->poison_fn = poison_zero;
 	s->useroffset = useroffset;
 	s->usersize = usersize;
 

mm/slab_common.c void __init create_boot_cache(struct kmem_cache *s, const char *name, · line 991

Quoted 4 lines
mm/slab_common.c +1 -0
@@ -974,6 +988,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name,
 	s->align = calculate_alignment(flags, ARCH_KMALLOC_MINALIGN, size);
 	s->useroffset = useroffset;
 	s->usersize = usersize;
+	s->poison_fn = poison_zero;

An empty indirect call is probably a pretty bad idea on systems with retpoline. Isn't this just a bool anyway for either calling poison_dont or poison_zero? Can it call anything else?

#10 lore raw

Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option

In reply to Alexander Potapenko
Quoted 11 lines · Alexander Potapenko

This option adds the possibility to initialize newly allocated pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed. We don't initialize slab caches with constructors to preserve their semantics. To reduce runtime costs of checking cachep->ctor we replace a call to memset with a call to cachep->poison_fn, which is only executed if the memory block needs to be initialized.

Just check for a ctor and then zero or use whatever pattern ? Why add a new function?

A message that is not in the archive (sent off-list or not yet received): alpine.DEB.2.21.1904260911570.8340@nuc-kabylake

#12 lore raw

Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option

Hmmmm.... Maybe its better to zero on free? That way you dont need to initialize the allocations. You could even check if someone mucked with the object during allocation. This is a replication of some of the inherent debugging facilities in the allocator though.

#13 lore raw

Re: [PATCH 1/3] mm: security: introduce the init_allocations=1 boot option

Quoted 4 lines · Christopher Lameter

Hmmmm.... Maybe its better to zero on free? That way you dont need to initialize the allocations. You could even check if someone mucked with the object during allocation.

As mentioned somewhere in the neighboring threads, Kees requested the options to initialize on both allocation and free, because we'll need this functionality for memory tagging someday.

Quoted 2 lines

This is a replication of some of the
inherent debugging facilities in the allocator though.

I'm curious, how often do people use SL[AU]B poisoning and redzones these days? Guess those should be superseded by KASAN already?
I agree it makes sense to reuse the existing debugging code, but on the other hand we probably want to keep the initialization fast enough to be used in production.

Quoted 1 line

Re: poison_fn, it wasn't a good idea, so I'll be dropping it.

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#14 lore raw

PATCH 2/3 gfp: mm: introduce __GFP_NOINIT

In reply to Alexander Potapenko

When passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NOINIT tells it to not initialize the requested memory if the init_allocations boot option is enabled. This can be useful in the cases the newly allocated memory is going to be initialized by the caller right away.

__GFP_NOINIT basically defeats the hardening against information leaks provided by the init_allocations feature, so one should use it with caution.

This patch also adds __GFP_NOINIT to alloc_pages() calls in SL[AOU]B.

Cc Andrew Morton <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc James Morris <[email protected]>
Cc "Serge E. Hallyn" <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Kostya Serebryany <[email protected]>
Cc Dmitry Vyukov <[email protected]>
Cc Kees Cook <[email protected]>
Cc Sandeep Patil <[email protected]>
Cc Laura Abbott <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Jann Horn <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Qian Cai <[email protected]>
Cc Vlastimil Babka <[email protected]>
---
 include/linux/gfp.h | 6 +++++-
 include/linux/mm.h  | 2 +-
 kernel/kexec_core.c | 2 +-
 mm/slab.c           | 2 +-
 mm/slob.c           | 1 +
 mm/slub.c           | 1 +
 6 files changed, 10 insertions(+), 4 deletions(-)
include/linux/gfp.h +5 -1
index fdab7de7490d..66d7f5604fe2 100644
@@ -44,6 +44,7 @@ struct vm_area_struct;
 #else
 #define ___GFP_NOLOCKDEP	0
 #endif
+#define ___GFP_NOINIT		0x1000000u
 /* If the above are modified, __GFP_BITS_SHIFT may need updating */
 
 /*
@@ -208,16 +209,19 @@ struct vm_area_struct;
  * %__GFP_COMP address compound page metadata.
  *
  * %__GFP_ZERO returns a zeroed page on success.
+ *
+ * %__GFP_NOINIT requests non-initialized memory from the underlying allocator.
  */
 #define __GFP_NOWARN	((__force gfp_t)___GFP_NOWARN)
 #define __GFP_COMP	((__force gfp_t)___GFP_COMP)
 #define __GFP_ZERO	((__force gfp_t)___GFP_ZERO)
+#define __GFP_NOINIT	((__force gfp_t)___GFP_NOINIT)
 
 /* Disable lockdep for GFP context tracking */
 #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)
 
 /* Room for N __GFP_FOO bits */
-#define __GFP_BITS_SHIFT (23 + IS_ENABLED(CONFIG_LOCKDEP))
+#define __GFP_BITS_SHIFT (25)
 #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
 
 /**
include/linux/mm.h +1 -1
index b38b71a5efaa..8f03334a9033 100644
@@ -2601,7 +2601,7 @@ DECLARE_STATIC_KEY_FALSE(init_allocations);
 static inline bool want_init_memory(gfp_t flags)
 {
 	if (static_branch_unlikely(&init_allocations))
-		return true;
+		return !(flags & __GFP_NOINIT);
 	return flags & __GFP_ZERO;
 }
 
kernel/kexec_core.c +1 -1
index be84f5f95c97..f9d1f1236cd0 100644
@@ -302,7 +302,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)
 {
 	struct page *pages;
 
-	pages = alloc_pages(gfp_mask & ~__GFP_ZERO, order);
+	pages = alloc_pages((gfp_mask & ~__GFP_ZERO) | __GFP_NOINIT, order);
 	if (pages) {
 		unsigned int count, i;
 
mm/slab.c +1 -1
index dcc5b73cf767..762cb0e7bcc1 100644
@@ -1393,7 +1393,7 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,
 	struct page *page;
 	int nr_pages;
 
-	flags |= cachep->allocflags;
+	flags |= (cachep->allocflags | __GFP_NOINIT);
 
 	page = __alloc_pages_node(nodeid, flags, cachep->gfporder);
 	if (!page) {
mm/slob.c +1 -0
index 18981a71e962..867d2d68a693 100644
@@ -192,6 +192,7 @@ static void *slob_new_pages(gfp_t gfp, int order, int node)
 {
 	void *page;
 
+	gfp |= __GFP_NOINIT;
 #ifdef CONFIG_NUMA
 	if (node != NUMA_NO_NODE)
 		page = __alloc_pages_node(node, gfp, order);
mm/slub.c +1 -0
index e4efb6575510..a79b4cb768a2 100644
@@ -1493,6 +1493,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s,
 	struct page *page;
 	unsigned int order = oo_order(oo);
 
+	flags |= __GFP_NOINIT;
 	if (node == NUMA_NO_NODE)
 		page = alloc_pages(flags, order);
 	else
Signature · 1 line
2.21.0.392.gf8f6787159e-goog
#15 lore raw

Re: [PATCH 2/3] gfp: mm: introduce __GFP_NOINIT

Quoted 3 lines · Alexander Potapenko

__GFP_NOINIT basically defeats the hardening against information leaks provided by the init_allocations feature, so one should use it with caution.

Even more than that, shouldn't we try to use it only in places where there is a demonstrated benefit, like performance data?

kernel/kexec_core.c static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) · line 307

Quoted 12 lines
kernel/kexec_core.c +1 -1
index be84f5f95c97..f9d1f1236cd0 100644
@@ -302,7 +302,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)
 {
 	struct page *pages;
 
-	pages = alloc_pages(gfp_mask & ~__GFP_ZERO, order);
+	pages = alloc_pages((gfp_mask & ~__GFP_ZERO) | __GFP_NOINIT, order);
 	if (pages) {
 		unsigned int count, i;

While this is probably not super security-sensitive, it's also not performance sensitive.

Quoted 31 lines
mm/slab.c +1 -1
index dcc5b73cf767..762cb0e7bcc1 100644
@@ -1393,7 +1393,7 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,
 	struct page *page;
 	int nr_pages;
 
-	flags |= cachep->allocflags;
+	flags |= (cachep->allocflags | __GFP_NOINIT);
 
 	page = __alloc_pages_node(nodeid, flags, cachep->gfporder);
 	if (!page) {
mm/slob.c +1 -0
index 18981a71e962..867d2d68a693 100644
@@ -192,6 +192,7 @@ static void *slob_new_pages(gfp_t gfp, int order, int node)
 {
 	void *page;
 
+	gfp |= __GFP_NOINIT;
 #ifdef CONFIG_NUMA
 	if (node != NUMA_NO_NODE)
 		page = __alloc_pages_node(node, gfp, order
mm/slub.c +0 -0
index e4efb6575510..a79b4cb768a2 100644

mm/slub.c static inline struct page *alloc_slab_page(struct kmem_cache *s, · line 1499

Quoted 7 lines
mm/slub.c +1 -0
@@ -1493,6 +1493,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s,
 	struct page *page;
 	unsigned int order = oo_order(oo);
 
+	flags |= __GFP_NOINIT;
 	if (node == NUMA_NO_NODE)
 		page = alloc_pages(flags, order);
 	else

These sl*b ones seem like a bad idea. We already have rules that sl*b allocations must be initialized by callers, and we have reasonably frequent bugs where the rules are broken.

Setting __GFP_NOINIT might be reasonable to do, though, for slabs that have a constructor. We have much higher confidence that *those* are going to get initialized properly.

#16 lore raw

Re: [PATCH 2/3] gfp: mm: introduce __GFP_NOINIT

Quoted 10 lines · Dave Hansen
Quoted 8 lines · Alexander Potapenko
kernel/kexec_core.c +0 -0
index be84f5f95c97..f9d1f1236cd0 100644
@@ -302,7 +302,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)
 {
      struct page *pages;
 
Quoted 7 lines

kernel/kexec_core.c static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order) · line 307

Quoted 4 lines
kernel/kexec_core.c +1 -1
-     pages = alloc_pages(gfp_mask & ~__GFP_ZERO, order);
+     pages = alloc_pages((gfp_mask & ~__GFP_ZERO) | __GFP_NOINIT, order);
      if (pages) {
              unsigned int count, i;

While this is probably not super security-sensitive, it's also not performance sensitive.

It is, however, a pretty clear case of "and then we immediately zero it".

Quoted 3 lines

These sl*b ones seem like a bad idea. We already have rules that sl*b allocations must be initialized by callers, and we have reasonably frequent bugs where the rules are broken.

Hm? No, this is saying that the page allocator can skip the auto-init because the slab internals will be doing it later.

Quoted 3 lines

Setting __GFP_NOINIT might be reasonable to do, though, for slabs that have a constructor. We have much higher confidence that *those* are going to get initialized properly.

That's already handled in patch 1.

Signature · 1 line
Kees Cook
#17 lore raw

Re: [PATCH 2/3] gfp: mm: introduce __GFP_NOINIT

Quoted 6 lines · Kees Cook
Quoted 3 lines

These sl*b ones seem like a bad idea. We already have rules that sl*b allocations must be initialized by callers, and we have reasonably frequent bugs where the rules are broken.

Hm? No, this is saying that the page allocator can skip the auto-init because the slab internals will be doing it later.

Ahhh, got it. I missed all the fun in patch 1.

#18 lore raw

Re: [PATCH 2/3] gfp: mm: introduce __GFP_NOINIT

In reply to Alexander Potapenko
Quoted 5 lines · Alexander Potapenko

When passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NOINIT tells it to not initialize the requested memory if the init_allocations boot option is enabled. This can be useful in the cases the newly allocated memory is going to be initialized by the caller right away.

Maybe add "... as seen when the slab allocator needs to allocate new pages from the page allocator." just to help clarify it here (instead of from the end of the commit log where you mention it offhand).

Quoted 40 lines

__GFP_NOINIT basically defeats the hardening against information leaks provided by the init_allocations feature, so one should use it with caution.

This patch also adds __GFP_NOINIT to alloc_pages() calls in SL[AOU]B.

Cc Andrew Morton <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc James Morris <[email protected]>
Cc "Serge E. Hallyn" <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Kostya Serebryany <[email protected]>
Cc Dmitry Vyukov <[email protected]>
Cc Kees Cook <[email protected]>
Cc Sandeep Patil <[email protected]>
Cc Laura Abbott <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Jann Horn <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Qian Cai <[email protected]>
Cc Vlastimil Babka <[email protected]>
---
 include/linux/gfp.h | 6 +++++-
 include/linux/mm.h  | 2 +-
 kernel/kexec_core.c | 2 +-
 mm/slab.c           | 2 +-
 mm/slob.c           | 1 +
 mm/slub.c           | 1 +
 6 files changed, 10 insertions(+), 4 deletions(-)
include/linux/gfp.h +0 -0
index fdab7de7490d..66d7f5604fe2 100644

include/linux/gfp.h struct vm_area_struct; · line 48

Quoted 5 lines
include/linux/gfp.h +1 -0
@@ -44,6 +44,7 @@ struct vm_area_struct;
 #else
 #define ___GFP_NOLOCKDEP       0
 #endif
+#define ___GFP_NOINIT          0x1000000u
 /* If the above are modified, __GFP_BITS_SHIFT may need updating */

I think you want to add NOINIT below GFP_ACCOUNT, update NOLOCKDEP and then adjust GFP_BITS_SHIFT differently, noted below.

Quoted 10 lines
include/linux/gfp.h +2 -0
 /*
@@ -208,16 +209,19 @@ struct vm_area_struct;
  * %__GFP_COMP address compound page metadata.
  *
  * %__GFP_ZERO returns a zeroed page on success.
+ *
+ * %__GFP_NOINIT requests non-initialized memory from the underlying allocator.
  */
 #define __GFP_NOWARN   ((__force gfp_t)___GFP_NOWARN)

include/linux/gfp.h struct vm_area_struct; · line 224

Quoted 10 lines
include/linux/gfp.h +2 -1
 #define __GFP_COMP     ((__force gfp_t)___GFP_COMP)
 #define __GFP_ZERO     ((__force gfp_t)___GFP_ZERO)
+#define __GFP_NOINIT   ((__force gfp_t)___GFP_NOINIT)
 
 /* Disable lockdep for GFP context tracking */
 #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)
 
 /* Room for N __GFP_FOO bits */
-#define __GFP_BITS_SHIFT (23 + IS_ENABLED(CONFIG_LOCKDEP))
+#define __GFP_BITS_SHIFT (25)

This should just be 24 + ... with the bit field added above NOLOCKDEP.

Quoted 8 lines
include/linux/gfp.h +0 -0
 #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
 
 /**
include/linux/mm.h +0 -0
index b38b71a5efaa..8f03334a9033 100644

include/linux/mm.h DECLARE_STATIC_KEY_FALSE(init_allocations); · line 2606

Quoted 7 lines
include/linux/mm.h +1 -1
@@ -2601,7 +2601,7 @@ DECLARE_STATIC_KEY_FALSE(init_allocations);
 static inline bool want_init_memory(gfp_t flags)
 {
        if (static_branch_unlikely(&init_allocations))
-               return true;
+               return !(flags & __GFP_NOINIT);
        return flags & __GFP_ZERO;
 }

You need to test for GFP_ZERO here too: return ((flags & __GFP_NOINIT | __GFP_ZERO) == 0)

Also, I wonder, for the sake of readability, if this should be named __GFP_NO_AUTOINIT ?

I'd also like to see each use of __GFP_NOINIT include a comment above its use where the logic is explained for _why_ it's safe (or reasonable) to use __GFP_NOINIT in each place.

mm/slub.c static inline struct page *alloc_slab_page(struct kmem_cache *s,

Quoted 44 lines
kernel/kexec_core.c +1 -1
index be84f5f95c97..f9d1f1236cd0 100644
@@ -302,7 +302,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)
 {
        struct page *pages;
 
-       pages = alloc_pages(gfp_mask & ~__GFP_ZERO, order);
+       pages = alloc_pages((gfp_mask & ~__GFP_ZERO) | __GFP_NOINIT, order);
        if (pages) {
                unsigned int count, i;
 
mm/slab.c +1 -1
index dcc5b73cf767..762cb0e7bcc1 100644
@@ -1393,7 +1393,7 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,
        struct page *page;
        int nr_pages;
 
-       flags |= cachep->allocflags;
+       flags |= (cachep->allocflags | __GFP_NOINIT);
 
        page = __alloc_pages_node(nodeid, flags, cachep->gfporder);
        if (!page) {
mm/slob.c +1 -0
index 18981a71e962..867d2d68a693 100644
@@ -192,6 +192,7 @@ static void *slob_new_pages(gfp_t gfp, int order, int node)
 {
        void *page;
 
+       gfp |= __GFP_NOINIT;
 #ifdef CONFIG_NUMA
        if (node != NUMA_NO_NODE)
                page = __alloc_pages_node(node, gfp, order);
mm/slub.c +0 -0
index e4efb6575510..a79b4cb768a2 100644
@@ -1493,6 +1493,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s,

What about kmalloc_large_node()?

mm/slub.c static inline struct page *alloc_slab_page(struct kmem_cache *s, · line 1499

Quoted 7 lines
mm/slub.c +1 -0
        struct page *page;
        unsigned int order = oo_order(oo);
 
+       flags |= __GFP_NOINIT;
        if (node == NUMA_NO_NODE)
                page = alloc_pages(flags, order);
        else

And just so I make sure I'm understanding this correctly: __GFP_NOINIT is passed to the page allocator because we know each allocation from the slab will get initialized at "sub allocation" time.

Looks good!

Signature · 1 line
Kees Cook
#19 lore raw

PATCH 3/3 RFC: net: apply __GFP_NOINIT to AF_UNIX sk_buff allocations

In reply to Alexander Potapenko

Add sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NOINIT. This helps reduce the slowdown on hackbench from 9% to 0.1%.

Cc Andrew Morton <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc James Morris <[email protected]>
Cc "Serge E. Hallyn" <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Kostya Serebryany <[email protected]>
Cc Dmitry Vyukov <[email protected]>
Cc Kees Cook <[email protected]>
Cc Sandeep Patil <[email protected]>
Cc Laura Abbott <[email protected]>
Cc Randy Dunlap <[email protected]>
Cc Jann Horn <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Qian Cai <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Eric Dumazet <[email protected]>
Cc David S. Miller <[email protected]>
---
 include/net/sock.h |  5 +++++
 net/core/sock.c    | 29 +++++++++++++++++++++++++----
 net/unix/af_unix.c | 13 +++++++------
 3 files changed, 37 insertions(+), 10 deletions(-)
include/net/sock.h +5 -0
index 8de5ee258b93..37fcdda23884 100644
@@ -1612,6 +1612,11 @@ struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size,
 struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
 				     unsigned long data_len, int noblock,
 				     int *errcode, int max_page_order);
+struct sk_buff *sock_alloc_send_pskb_noinit(struct sock *sk,
+					    unsigned long header_len,
+					    unsigned long data_len,
+					    int noblock, int *errcode,
+					    int max_page_order);
 void *sock_kmalloc(struct sock *sk, int size, gfp_t priority);
 void sock_kfree_s(struct sock *sk, void *mem, int size);
 void sock_kzfree_s(struct sock *sk, void *mem, int size);
net/core/sock.c +25 -4
index 99b288a19b39..0a2af1e1fa1c 100644
@@ -2187,9 +2187,11 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo)
  *	Generic send/receive buffer handlers
  */
 
-struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
-				     unsigned long data_len, int noblock,
-				     int *errcode, int max_page_order)
+struct sk_buff *sock_alloc_send_pskb_internal(struct sock *sk,
+					      unsigned long header_len,
+					      unsigned long data_len,
+					      int noblock, int *errcode,
+					      int max_page_order, gfp_t gfp)
 {
 	struct sk_buff *skb;
 	long timeo;
@@ -2218,7 +2220,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
 		timeo = sock_wait_for_wmem(sk, timeo);
 	}
 	skb = alloc_skb_with_frags(header_len, data_len, max_page_order,
-				   errcode, sk->sk_allocation);
+				   errcode, sk->sk_allocation | gfp);
 	if (skb)
 		skb_set_owner_w(skb, sk);
 	return skb;
@@ -2229,8 +2231,27 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
 	*errcode = err;
 	return NULL;
 }
+
+struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+				     unsigned long data_len, int noblock,
+				     int *errcode, int max_page_order)
+{
+	return sock_alloc_send_pskb_internal(sk, header_len, data_len,
+		noblock, errcode, max_page_order, /*gfp*/0);
+}
 EXPORT_SYMBOL(sock_alloc_send_pskb);
 
+struct sk_buff *sock_alloc_send_pskb_noinit(struct sock *sk,
+					    unsigned long header_len,
+					    unsigned long data_len,
+					    int noblock, int *errcode,
+					    int max_page_order)
+{
+	return sock_alloc_send_pskb_internal(sk, header_len, data_len,
+		noblock, errcode, max_page_order, /*gfp*/__GFP_NOINIT);
+}
+EXPORT_SYMBOL(sock_alloc_send_pskb_noinit);
+
 struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size,
 				    int noblock, int *errcode)
 {
net/unix/af_unix.c +7 -6
index ddb838a1b74c..9a45824c3c48 100644
@@ -1627,9 +1627,9 @@ static int unix_dgram_sendmsg(struct socket *sock, struct msghdr *msg,
 		BUILD_BUG_ON(SKB_MAX_ALLOC < PAGE_SIZE);
 	}
 
-	skb = sock_alloc_send_pskb(sk, len - data_len, data_len,
-				   msg->msg_flags & MSG_DONTWAIT, &err,
-				   PAGE_ALLOC_COSTLY_ORDER);
+	skb = sock_alloc_send_pskb_noinit(sk, len - data_len, data_len,
+					  msg->msg_flags & MSG_DONTWAIT, &err,
+					  PAGE_ALLOC_COSTLY_ORDER);
 	if (skb == NULL)
 		goto out;
 
@@ -1824,9 +1824,10 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg,
 
 		data_len = min_t(size_t, size, PAGE_ALIGN(data_len));
 
-		skb = sock_alloc_send_pskb(sk, size - data_len, data_len,
-					   msg->msg_flags & MSG_DONTWAIT, &err,
-					   get_order(UNIX_SKB_FRAGS_SZ));
+		skb = sock_alloc_send_pskb_noinit(sk, size - data_len, data_len,
+						  msg->msg_flags & MSG_DONTWAIT,
+						  &err,
+						  get_order(UNIX_SKB_FRAGS_SZ));
 		if (!skb)
 			goto out_err;
 
Signature · 1 line
2.21.0.392.gf8f6787159e-goog
#20 lore raw

Re: [PATCH 3/3] RFC: net: apply __GFP_NOINIT to AF_UNIX sk_buff allocations

Quoted 4 lines · Alexander Potapenko

Add sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NOINIT. This helps reduce the slowdown on hackbench from 9% to 0.1%.

I would include a detailed justification about why this is safe to do. I imagine (but haven't looked) that the skb is immediately written to after allocation, so this is basically avoiding a "double init". Is that correct?

Signature · 1 line
Kees Cook
#21 lore raw
In reply to Alexander Potapenko
Quoted 26 lines · Alexander Potapenko

Following the recent discussions here's another take at initializing pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

The patchset introduces a new boot option, init_allocations, which makes page allocator and SL[AOU]B initialize newly allocated memory. init_allocations=0 doesn't (hopefully) add any overhead to the allocation fast path (no noticeable slowdown on hackbench).

With only the the first of the proposed patches the slowdown numbers are:
- 1.1% (stdev 0.2%) sys time slowdown building Linux kernel
- 3.1% (stdev 0.3%) sys time slowdown on af_inet_loopback benchmark
- 9.4% (stdev 0.5%) sys time slowdown on hackbench

The second patch introduces a GFP flag that allows to disable initialization for certain allocations. The third page is an example of applying it to af_unix.c, which helps hackbench greatly.

Slowdown numbers for the whole patchset are:
- 1.8% (stdev 0.8%) on kernel build
- 6.5% (stdev 0.2%) on af_inet_loopback
- 0.12% (stdev 0.6%) on hackbench

Quoted 4 lines

Alexander Potapenko (3):
mm: security: introduce the init_allocations=1 boot option
gfp: mm: introduce __GFP_NOINIT
net: apply __GFP_NOINIT to AF_UNIX sk_buff allocations

Oops, I was hoping git send-email would pull all the Cc: tags from the patches and actually use them.

Quoted 21 lines
 drivers/infiniband/core/uverbs_ioctl.c |  2 +-
 include/linux/gfp.h                    |  6 ++++-
 include/linux/mm.h                     |  8 +++++++
 include/linux/slab_def.h               |  1 +
 include/linux/slub_def.h               |  1 +
 include/net/sock.h                     |  5 +++++
 kernel/kexec_core.c                    |  4 ++--
 mm/dmapool.c                           |  2 +-
 mm/page_alloc.c                        | 18 ++++++++++++++-
 mm/slab.c                              | 14 ++++++------
 mm/slab.h                              |  1 +
 mm/slab_common.c                       | 15 +++++++++++++
 mm/slob.c                              |  3 ++-
 mm/slub.c                              |  9 ++++----
 net/core/sock.c                        | 31 +++++++++++++++++++++-----
 net/unix/af_unix.c                     | 13 ++++++-----
 16 files changed, 104 insertions(+), 29 deletions(-)

--
2.21.0.392.gf8f6787159e-goog

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#22 lore raw
In reply to Alexander Potapenko
Quoted 10 lines · Alexander Potapenko

Following the recent discussions here's another take at initializing pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

The patchset introduces a new boot option, init_allocations, which makes page allocator and SL[AOU]B initialize newly allocated memory. init_allocations=0 doesn't (hopefully) add any overhead to the allocation fast path (no noticeable slowdown on hackbench).

Quoted 7 lines

With only the the first of the proposed patches the slowdown numbers are:
- 1.1% (stdev 0.2%) sys time slowdown building Linux kernel
- 3.1% (stdev 0.3%) sys time slowdown on af_inet_loopback benchmark
- 9.4% (stdev 0.5%) sys time slowdown on hackbench

The second patch introduces a GFP flag that allows to disable initialization for certain allocations. The third page is an example of

third patch

Quoted 31 lines

applying it to af_unix.c, which helps hackbench greatly.

Slowdown numbers for the whole patchset are:
- 1.8% (stdev 0.8%) on kernel build
- 6.5% (stdev 0.2%) on af_inet_loopback
- 0.12% (stdev 0.6%) on hackbench

Alexander Potapenko (3):
mm: security: introduce the init_allocations=1 boot option
gfp: mm: introduce __GFP_NOINIT
net: apply __GFP_NOINIT to AF_UNIX sk_buff allocations

 drivers/infiniband/core/uverbs_ioctl.c |  2 +-
 include/linux/gfp.h                    |  6 ++++-
 include/linux/mm.h                     |  8 +++++++
 include/linux/slab_def.h               |  1 +
 include/linux/slub_def.h               |  1 +
 include/net/sock.h                     |  5 +++++
 kernel/kexec_core.c                    |  4 ++--
 mm/dmapool.c                           |  2 +-
 mm/page_alloc.c                        | 18 ++++++++++++++-
 mm/slab.c                              | 14 ++++++------
 mm/slab.h                              |  1 +
 mm/slab_common.c                       | 15 +++++++++++++
 mm/slob.c                              |  3 ++-
 mm/slub.c                              |  9 ++++----
 net/core/sock.c                        | 31 +++++++++++++++++++++-----
 net/unix/af_unix.c                     | 13 ++++++-----
 16 files changed, 104 insertions(+), 29 deletions(-)
Signature · 1 line
~Randy
#23 lore raw
In reply to Alexander Potapenko
Quoted 10 lines · Alexander Potapenko

Following the recent discussions here's another take at initializing pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

The patchset introduces a new boot option, init_allocations, which makes page allocator and SL[AOU]B initialize newly allocated memory. init_allocations=0 doesn't (hopefully) add any overhead to the allocation fast path (no noticeable slowdown on hackbench).

I continue to prefer to have a way to both at-allocation initialization _and_ poison-on-free, so let's not redirect this to doing it only at free time. We're going to need both hooks when doing Memory Tagging, so let's just get it in place now. The security benefits on tagging, IMO, easily justify a 1-2% performance hit. And likely we'll see this improve with new hardware.

Quoted 5 lines

With only the the first of the proposed patches the slowdown numbers are:
- 1.1% (stdev 0.2%) sys time slowdown building Linux kernel
- 3.1% (stdev 0.3%) sys time slowdown on af_inet_loopback benchmark
- 9.4% (stdev 0.5%) sys time slowdown on hackbench

Quoted 7 lines

The second patch introduces a GFP flag that allows to disable initialization for certain allocations. The third page is an example of applying it to af_unix.c, which helps hackbench greatly.

Slowdown numbers for the whole patchset are:
- 1.8% (stdev 0.8%) on kernel build
- 6.5% (stdev 0.2%) on af_inet_loopback

Any idea why thes two went _up_?

Quoted 1 line

- 0.12% (stdev 0.6%) on hackbench

Well that's quite an improvement. :)

Signature · 1 line
Kees Cook
#24 lore raw
Quoted 8 lines · Kees Cook
Quoted 6 lines · Alexander Potapenko

Following the recent discussions here's another take at initializing pages and heap objects with zeroes. This is needed to prevent possible information leaks and make the control-flow bugs that depend on uninitialized values more deterministic.

Quoted 8 lines
Quoted 4 lines

The patchset introduces a new boot option, init_allocations, which makes page allocator and SL[AOU]B initialize newly allocated memory. init_allocations=0 doesn't (hopefully) add any overhead to the allocation fast path (no noticeable slowdown on hackbench).

I continue to prefer to have a way to both at-allocation initialization _and_ poison-on-free, so let's not redirect this to doing it only at free time.

There's a problem with poison-on-free.
By default SLUB stores the freelist pointer (not sure if it's the only piece of data) in the memory chunk itself, so newly allocated memory is dirty despite it has been zeroed out previously.
We could probably zero out the bits used by the allocator when allocating the memory chunk, but it sounds hacky (yet saves us 8 bytes on every allocation)
A cleaner solution would be to unconditionally relocate the free pointer by short-circuiting
https://elixir.bootlin.com/linux/latest/source/mm/slub.c#L3531
Surprisingly, this doesn't work, because now the sizes of slub caches are a bit off, and create_unique_id() in slub.c returns clashing sysfs names.

Quoted 26 lines

We're going to need both hooks when doing
Memory Tagging, so let's just get it in place now. The security benefits on tagging, IMO, easily justify a 1-2% performance hit. And likely we'll see this improve with new hardware.

Quoted 5 lines

With only the the first of the proposed patches the slowdown numbers are:
- 1.1% (stdev 0.2%) sys time slowdown building Linux kernel
- 3.1% (stdev 0.3%) sys time slowdown on af_inet_loopback benchmark
- 9.4% (stdev 0.5%) sys time slowdown on hackbench

Quoted 7 lines

The second patch introduces a GFP flag that allows to disable initialization for certain allocations. The third page is an example of applying it to af_unix.c, which helps hackbench greatly.

Slowdown numbers for the whole patchset are:
- 1.8% (stdev 0.8%) on kernel build
- 6.5% (stdev 0.2%) on af_inet_loopback

Any idea why thes two went _up_?

Quoted 1 line

- 0.12% (stdev 0.6%) on hackbench

Well that's quite an improvement. :)

--
Kees Cook

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg