Skip to contentSkip to conversation

RFC: add init_on_alloc/init_on_free boot options

PATCH Superseded Alexander Potapenko 4 participants 37 messages May 14, 2019 – May 21, 2019

Subsystems (18)

v2 of 10 Superseded Latest is v10 →

lore
Share & tools
lore
#0 lore raw

v2 cover RFC: add init_on_alloc/init_on_free boot options

Provide init_on_alloc and init_on_free boot options.

These are aimed at preventing possible information leaks and making the control-flow bugs that depend on uninitialized values more deterministic.

Enabling either of the options guarantees that the memory returned by the page allocator and SL[AOU]B is initialized with zeroes.

Enabling init_on_free also guarantees that pages and heap objects are initialized right after they're freed, so it won't be possible to access stale data by using a dangling pointer.

Alexander Potapenko (4):
mm: security: introduce init_on_alloc=1 and init_on_free=1 boot
options
lib: introduce test_meminit module
gfp: mm: introduce __GFP_NO_AUTOINIT
net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations

 .../admin-guide/kernel-parameters.txt         |   8 +
 drivers/infiniband/core/uverbs_ioctl.c        |   2 +-
 include/linux/gfp.h                           |  13 +-
 include/linux/mm.h                            |  22 ++
 include/net/sock.h                            |   5 +
 kernel/kexec_core.c                           |   5 +-
 lib/Kconfig.debug                             |   8 +
 lib/Makefile                                  |   1 +
 lib/test_meminit.c                            | 205 ++++++++++++++++++
 mm/dmapool.c                                  |   2 +-
 mm/page_alloc.c                               |  68 +++++-
 mm/slab.c                                     |  18 +-
 mm/slab.h                                     |  16 ++
 mm/slob.c                                     |  23 +-
 mm/slub.c                                     |  28 ++-
 net/core/sock.c                               |  31 ++-
 net/unix/af_unix.c                            |  13 +-
 security/Kconfig.hardening                    |  14 ++
 18 files changed, 443 insertions(+), 39 deletions(-)
 create mode 100644 lib/test_meminit.c
Signature · 1 line
2.21.0.1020.gf2820cf01a-goog
#1 lore raw

PATCH v2 1/4 mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options

The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.

init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed.

init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.

Both init_on_alloc=1 and init_on_free=1 guarantee that the allocator returns zeroed memory. The only exception is slab caches with constructors. Those are never zero-initialized to preserve their semantics.

For SLOB allocator init_on_free=1 also implies init_on_alloc=1 behavior, i.e. objects are zeroed at both allocation and deallocation time. This is done because SLOB may otherwise return multiple freelist pointers in the allocated object. For SLAB and SLUB enabling either init_on_alloc or init_on_free leads to one-time initialization of the object.

Both init_on_alloc and init_on_free default to zero, but those defaults can be overridden with CONFIG_INIT_ON_ALLOC_DEFAULT_ON and CONFIG_INIT_ON_FREE_DEFAULT_ON.

Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:

hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)

Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)

The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.

To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
To: Kees Cook <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: James Morris <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v2:
- unconditionally initialize pages in kernel_init_free_pages()
- comment from Randy Dunlap: drop 'default false' lines from Kconfig.hardening

---
 .../admin-guide/kernel-parameters.txt         |  8 +++
 drivers/infiniband/core/uverbs_ioctl.c        |  2 +-
 include/linux/mm.h                            | 22 ++++++
 kernel/kexec_core.c                           |  2 +-
 mm/dmapool.c                                  |  2 +-
 mm/page_alloc.c                               | 68 ++++++++++++++++---
 mm/slab.c                                     | 16 ++++-
 mm/slab.h                                     | 16 +++++
 mm/slob.c                                     | 22 +++++-
 mm/slub.c                                     | 27 ++++++--
 net/core/sock.c                               |  2 +-
 security/Kconfig.hardening                    | 14 ++++
 12 files changed, 178 insertions(+), 23 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +8 -0
index 08df58805703..cece9a56ddb1 100644
@@ -1673,6 +1673,14 @@
 
 	initrd=		[BOOT] Specify the location of the initial ramdisk
 
+	init_on_alloc=	[MM] Fill newly allocated pages and heap objects with
+			zeroes.
+			Format: 0 | 1
+			Default set by CONFIG_INIT_ON_ALLOC_DEFAULT_ON.
+	init_on_free=	[MM] Fill freed pages and heap objects with zeroes.
+			Format: 0 | 1
+			Default set by CONFIG_INIT_ON_FREE_DEFAULT_ON.
+
 	init_pkru=	[x86] Specify the default memory protection keys rights
 			register contents for all processes.  0x55555554 by
 			default (disallow access to all but pkey 0).  Can
drivers/infiniband/core/uverbs_ioctl.c +1 -1
index 829b0c6944d8..61758201d9b2 100644
@@ -127,7 +127,7 @@ __malloc void *_uverbs_alloc(struct uverbs_attr_bundle *bundle, size_t size,
 	res = (void *)pbundle->internal_buffer + pbundle->internal_used;
 	pbundle->internal_used =
 		ALIGN(new_used, sizeof(*pbundle->internal_buffer));
-	if (flags & __GFP_ZERO)
+	if (want_init_on_alloc(flags))
 		memset(res, 0, size);
 	return res;
 }
include/linux/mm.h +22 -0
index 083d7b4863ed..18d96f1d07c5 100644
@@ -2610,6 +2610,28 @@ static inline void kernel_poison_pages(struct page *page, int numpages,
 					int enable) { }
 #endif
 
+#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON
+DECLARE_STATIC_KEY_TRUE(init_on_alloc);
+#else
+DECLARE_STATIC_KEY_FALSE(init_on_alloc);
+#endif
+static inline bool want_init_on_alloc(gfp_t flags)
+{
+	if (static_branch_unlikely(&init_on_alloc))
+		return true;
+	return flags & __GFP_ZERO;
+}
+
+#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON
+DECLARE_STATIC_KEY_TRUE(init_on_free);
+#else
+DECLARE_STATIC_KEY_FALSE(init_on_free);
+#endif
+static inline bool want_init_on_free(void)
+{
+	return static_branch_unlikely(&init_on_free);
+}
+
 extern bool _debug_pagealloc_enabled;
 
 static inline bool debug_pagealloc_enabled(void)
kernel/kexec_core.c +1 -1
index fd5c95ff9251..2f75dd0d0d81 100644
@@ -315,7 +315,7 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)
 		arch_kexec_post_alloc_pages(page_address(pages), count,
 					    gfp_mask);
 
-		if (gfp_mask & __GFP_ZERO)
+		if (want_init_on_alloc(gfp_mask))
 			for (i = 0; i < count; i++)
 				clear_highpage(pages + i);
 	}
mm/dmapool.c +1 -1
index 76a160083506..493d151067cb 100644
@@ -381,7 +381,7 @@ void *dma_pool_alloc(struct dma_pool *pool, gfp_t mem_flags,
 #endif
 	spin_unlock_irqrestore(&pool->lock, flags);
 
-	if (mem_flags & __GFP_ZERO)
+	if (want_init_on_alloc(mem_flags))
 		memset(retval, 0, pool->size);
 
 	return retval;
mm/page_alloc.c +59 -9

109 lines not on this page · raw message

mm/slab.c +13 -3
index 284ab737faee..d00e9de26a45 100644
@@ -1855,6 +1855,14 @@ static bool set_objfreelist_slab_cache(struct kmem_cache *cachep,
 
 	cachep->num = 0;
 
+	/*
+	 * If slab auto-initialization on free is enabled, store the freelist
+	 * off-slab, so that its contents don't end up in one of the allocated
+	 * objects.
+	 */
+	if (unlikely(slab_want_init_on_free(cachep)))
+		return false;
+
 	if (cachep->ctor || flags & SLAB_TYPESAFE_BY_RCU)
 		return false;
 
@@ -3294,7 +3302,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid,
 	local_irq_restore(save_flags);
 	ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller);
 
-	if (unlikely(flags & __GFP_ZERO) && ptr)
+	if (unlikely(slab_want_init_on_alloc(flags, cachep)) && ptr)
 		memset(ptr, 0, cachep->object_size);
 
 	slab_post_alloc_hook(cachep, flags, 1, &ptr);
@@ -3351,7 +3359,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller)
 	objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller);
 	prefetchw(objp);
 
-	if (unlikely(flags & __GFP_ZERO) && objp)
+	if (unlikely(slab_want_init_on_alloc(flags, cachep)) && objp)
 		memset(objp, 0, cachep->object_size);
 
 	slab_post_alloc_hook(cachep, flags, 1, &objp);
@@ -3472,6 +3480,8 @@ void ___cache_free(struct kmem_cache *cachep, void *objp,
 	struct array_cache *ac = cpu_cache_get(cachep);
 
 	check_irq_off();
+	if (unlikely(slab_want_init_on_free(cachep)))
+		memset(objp, 0, cachep->object_size);
 	kmemleak_free_recursive(objp, cachep->flags);
 	objp = cache_free_debugcheck(cachep, objp, caller);
 
@@ -3559,7 +3569,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
 	cache_alloc_debugcheck_after_bulk(s, flags, size, p, _RET_IP_);
 
 	/* Clear memory outside IRQ disabled section */
-	if (unlikely(flags & __GFP_ZERO))
+	if (unlikely(slab_want_init_on_alloc(flags, s)))
 		for (i = 0; i < size; i++)
 			memset(p[i], 0, s->object_size);
 
mm/slab.h +16 -0
index 43ac818b8592..24ae887359b8 100644
@@ -524,4 +524,20 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep,
 static inline void cache_random_seq_destroy(struct kmem_cache *cachep) { }
 #endif /* CONFIG_SLAB_FREELIST_RANDOM */
 
+static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c)
+{
+	if (static_branch_unlikely(&init_on_alloc))
+		return !(c->ctor);
+	else
+		return flags & __GFP_ZERO;
+}
+
+static inline bool slab_want_init_on_free(struct kmem_cache *c)
+{
+	if (static_branch_unlikely(&init_on_free))
+		return !(c->ctor);
+	else
+		return false;
+}
+
 #endif /* MM_SLAB_H */
mm/slob.c +20 -2
index 307c2c9feb44..351d3dfee000 100644
@@ -212,6 +212,19 @@ static void slob_free_pages(void *b, int order)
 	free_pages((unsigned long)b, order);
 }
 
+/*
+ * init_on_free=1 also implies initialization at allocation time.
+ * This is because newly allocated objects may contain freelist pointers
+ * somewhere in the middle.
+ */
+static inline bool slob_want_init_on_alloc(gfp_t flags, struct kmem_cache *c)
+{
+	if (static_branch_unlikely(&init_on_alloc) ||
+	    static_branch_unlikely(&init_on_free))
+		return c ? (!c->ctor) : true;
+	return flags & __GFP_ZERO;
+}
+
 /*
  * Allocate a slob block within a given slob_page sp.
  */
@@ -330,8 +343,6 @@ static void *slob_alloc(size_t size, gfp_t gfp, int align, int node)
 		BUG_ON(!b);
 		spin_unlock_irqrestore(&slob_lock, flags);
 	}
-	if (unlikely(gfp & __GFP_ZERO))
-		memset(b, 0, size);
 	return b;
 }
 
@@ -366,6 +377,9 @@ static void slob_free(void *block, int size)
 		return;
 	}
 
+	if (unlikely(want_init_on_free()))
+		memset(block, 0, size);
+
 	if (!slob_page_free(sp)) {
 		/* This slob page is about to become partially free. Easy! */
 		sp->units = units;
@@ -461,6 +475,8 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller)
 	}
 
 	kmemleak_alloc(ret, size, 1, gfp);
+	if (unlikely(slob_want_init_on_alloc(gfp, 0)))
+		memset(ret, 0, size);
 	return ret;
 }
 
@@ -559,6 +575,8 @@ static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node)
 		WARN_ON_ONCE(flags & __GFP_ZERO);
 		c->ctor(b);
 	}
+	if (unlikely(slob_want_init_on_alloc(flags, c)))
+		memset(b, 0, c->size);
 
 	kmemleak_alloc_recursive(b, c->size, 1, c->flags, flags);
 	return b;
mm/slub.c +22 -5
index 6b28cd2b5a58..01424e910800 100644
@@ -1423,6 +1423,19 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, void *x)
 static inline bool slab_free_freelist_hook(struct kmem_cache *s,
 					   void **head, void **tail)
 {
+
+	void *object;
+	void *next = *head;
+	void *old_tail = *tail ? *tail : *head;
+
+	if (slab_want_init_on_free(s))
+		do {
+			object = next;
+			next = get_freepointer(s, object);
+			memset(object, 0, s->size);
+			set_freepointer(s, object, next);
+		} while (object != old_tail);
+
 /*
  * Compiler cannot detect this function can be removed if slab_free_hook()
  * evaluates to nothing.  Thus, catch all relevant config debug options here.
@@ -1432,9 +1445,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s,
 	defined(CONFIG_DEBUG_OBJECTS_FREE) ||	\
 	defined(CONFIG_KASAN)
 
-	void *object;
-	void *next = *head;
-	void *old_tail = *tail ? *tail : *head;
+	next = *head;
 
 	/* Head and tail of the reconstructed freelist */
 	*head = NULL;
@@ -2740,8 +2751,14 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s,
 		prefetch_freepointer(s, next_object);
 		stat(s, ALLOC_FASTPATH);
 	}
+	/*
+	 * If the object has been wiped upon free, make sure it's fully
+	 * initialized by zeroing out freelist pointer.
+	 */
+	if (slab_want_init_on_free(s))
+		*(void **)object = 0;
 
-	if (unlikely(gfpflags & __GFP_ZERO) && object)
+	if (unlikely(slab_want_init_on_alloc(gfpflags, s)) && object)
 		memset(object, 0, s->object_size);
 
 	slab_post_alloc_hook(s, gfpflags, 1, &object);
@@ -3163,7 +3180,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
 	local_irq_enable();
 
 	/* Clear memory outside IRQ disabled fastpath loop */
-	if (unlikely(flags & __GFP_ZERO)) {
+	if (unlikely(slab_want_init_on_alloc(flags, s))) {
 		int j;
 
 		for (j = 0; j < i; j++)
net/core/sock.c +1 -1
index 75b1c950b49f..9ceb90c875bc 100644
@@ -1602,7 +1602,7 @@ static struct sock *sk_prot_alloc(struct proto *prot, gfp_t priority,
 		sk = kmem_cache_alloc(slab, priority & ~__GFP_ZERO);
 		if (!sk)
 			return sk;
-		if (priority & __GFP_ZERO)
+		if (want_init_on_alloc(priority))
 			sk_prot_clear_nulls(sk, prot->obj_size);
 	} else
 		sk = kmalloc(prot->obj_size, priority);
security/Kconfig.hardening +14 -0
index 0a1d4ca314f4..87883e3e3c2a 100644
@@ -159,6 +159,20 @@ config STACKLEAK_RUNTIME_DISABLE
 	  runtime to control kernel stack erasing for kernels built with
 	  CONFIG_GCC_PLUGIN_STACKLEAK.
 
+config INIT_ON_ALLOC_DEFAULT_ON
+	bool "Set init_on_alloc=1 by default"
+	help
+	  Enable init_on_alloc=1 by default, making the kernel initialize every
+	  page and heap allocation with zeroes.
+	  init_on_alloc can be overridden via command line.
+
+config INIT_ON_FREE_DEFAULT_ON
+	bool "Set init_on_free=1 by default"
+	help
+	  Enable init_on_free=1 by default, making the kernel initialize freed
+	  pages and slab memory with zeroes.
+	  init_on_free can be overridden via command line.
+
 endmenu
 
 endmenu
Signature · 1 line
2.21.0.1020.gf2820cf01a-goog
#2 lore raw

Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options

Quoted 5 lines · Alexander Potapenko

Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:

hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)

I wonder if the patch series should be reorganized to introduce __GFP_NO_AUTOINIT first, so that when the commit with benchmarks appears, we get the "final" numbers...

Quoted 4 lines

Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)

I'm working on reproducing these benchmarks. I'd really like to narrow down the +24% number here. But it does

Quoted 2 lines

The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.

I think the use of static keys here is really great: this is available by default for anyone that wants to turn it on.

I'm thinking, given the configuable nature of this, it'd be worth adding a little more detail at boot time. I think maybe a separate patch could be added to describe the kernel's memory auto-initialization features, and add something like this to mm_init():

+void __init report_meminit(void)
+{
+ const char *stack;
+
+ if (IS_ENABLED(CONFIG_INIT_STACK_ALL))
+ stack = "all";
+ else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL))
+ stack = "byref_all";
+ else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF))
+ stack = "byref";
+ else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_USER))
+ stack = "__user";
+ else
+ stack = "off";
+
+ /* Report memory auto-initialization states for this boot. */
+ pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n",
+ stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off",
+ want_init_on_free() ? "on" : "off");
+}

To get a boot line like:

mem auto-init: stack:off, heap alloc:off, heap free:on

And one other thought I had was that in the init_on_free=1 case, there is a large pause at boot while memory is being cleared. I think it'd be handy to include a comment about that, just to keep people from being surprised:

init/main.c +2 -0
index cf0c3948ce0e..aea278392338 100644
@@ -529,6 +529,8 @@ static void __init mm_init(void)
 	 * bigger than MAX_ORDER unless SPARSEMEM.
 	 */
 	page_ext_init_flatmem();
+	if (want_init_on_free())
+		pr_info("Clearing system memory ...\n");
 	mem_init();
 	kmem_cache_init();
 	pgtable_init();

Beyond these thoughts, I think this series is in good shape.

Andrew (or anyone else) do you have any concerns about this?

Signature · 1 line
Kees Cook
#3 lore raw

Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options

From: Kees Cook <[email protected]>
Date: Thu, May 16, 2019 at 6:20 PM
To: Alexander Potapenko
Cc: <[email protected]>, <[email protected]>, <[email protected]>, Masahiro Yamada, James Morris, Serge E. Hallyn, Nick Desaulniers, Kostya Serebryany, Dmitry Vyukov, Sandeep Patil, Laura Abbott, Randy Dunlap, Jann Horn, Mark Rutland, <[email protected]>, <[email protected]>

Quoted 11 lines · Alexander Potapenko
Quoted 5 lines

Slowdown for the new features compared to init_on_free=0, init_on_alloc=0:

hackbench, init_on_free=1: +7.62% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.75% sys time (st.err 2.14%)

I wonder if the patch series should be reorganized to introduce __GFP_NO_AUTOINIT first, so that when the commit with benchmarks appears, we get the "final" numbers...

Quoted 7 lines
Quoted 4 lines

Linux build with -j12, init_on_free=1: +8.38% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.42% sys time (st.err 0.52%) Linux build with -j12, init_on_alloc=1: -0.13% wall time (st.err 0.42%) Linux build with -j12, init_on_alloc=1: +0.57% sys time (st.err 0.40%)

I'm working on reproducing these benchmarks. I'd really like to narrow down the +24% number here. But it does

I suspect the slowdown of init_on_free is bigger than that of PAX_SANITIZE_MEMORY, as we've set the goal to have fully zeroed memory at alloc time.
If we want a mode that only wipes the user data upon free() but doesn't eliminate all uninit memory, then we can make it faster.

Quoted 32 lines
Quoted 2 lines

The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.

I think the use of static keys here is really great: this is available by default for anyone that wants to turn it on.

I'm thinking, given the configuable nature of this, it'd be worth adding a little more detail at boot time. I think maybe a separate patch could be added to describe the kernel's memory auto-initialization features, and add something like this to mm_init():

(quoted diff) +20 -0
+void __init report_meminit(void)
+{
+       const char *stack;
+
+       if (IS_ENABLED(CONFIG_INIT_STACK_ALL))
+               stack = "all";
+       else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL))
+               stack = "byref_all";
+       else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF))
+               stack = "byref";
+       else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_USER))
+               stack = "__user";
+       else
+               stack = "off";
+
+       /* Report memory auto-initialization states for this boot. */
+       pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n",
+               stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off",
+               want_init_on_free() ? "on" : "off");
+}
Quoted 3 lines

To get a boot line like:

mem auto-init: stack:off, heap alloc:off, heap free:on

For stack there's no binary on/off, as you can potentially build half of the kernel with stack instrumentation and another half without it. We could make the instrumentation insert a static global flag into each translation unit, but this won't give us any interesting info.

Quoted 24 lines

And one other thought I had was that in the init_on_free=1 case, there is a large pause at boot while memory is being cleared. I think it'd be handy to include a comment about that, just to keep people from being surprised:

init/main.c +2 -0
index cf0c3948ce0e..aea278392338 100644
@@ -529,6 +529,8 @@ static void __init mm_init(void)
         * bigger than MAX_ORDER unless SPARSEMEM.
         */
        page_ext_init_flatmem();
+       if (want_init_on_free())
+               pr_info("Clearing system memory ...\n");
        mem_init();
        kmem_cache_init();
        pgtable_init();

Beyond these thoughts, I think this series is in good shape.

Andrew (or anyone else) do you have any concerns about this?

--
Kees Cook

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#4 lore raw

Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options

Quoted 5 lines · Alexander Potapenko

I suspect the slowdown of init_on_free is bigger than that of PAX_SANITIZE_MEMORY, as we've set the goal to have fully zeroed memory at alloc time.
If we want a mode that only wipes the user data upon free() but doesn't eliminate all uninit memory, then we can make it faster.

Yeah, I sent a separate email that discusses this a bit more.

I think the goals of init_on_alloc and init_on_free are likely a bit different. Given init_on_alloc's much more cache-friendly performance, I think that it's likely the right way forward for getting to fully zeroed memory at alloc time. (Though I note that it already includes exclusions:
such tradeoffs won't be unique to init_on_free.)

init_on_free appears to give us similar coverage (but also reduces the lifetime of unused data), but isn't cache-friendly so it looks to need a lot more tuning/trade-offs. (Not that we shouldn't include it! It'll just need a bit more care to be reasonable.)

Quoted 21 lines
Quoted 21 lines
(quoted diff) +20 -0
+void __init report_meminit(void)
+{
+       const char *stack;
+
+       if (IS_ENABLED(CONFIG_INIT_STACK_ALL))
+               stack = "all";
+       else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF_ALL))
+               stack = "byref_all";
+       else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_BYREF))
+               stack = "byref";
+       else if (IS_ENABLED(CONFIG_GCC_PLUGIN_STRUCTLEAK_USER))
+               stack = "__user";
+       else
+               stack = "off";
+
+       /* Report memory auto-initialization states for this boot. */
+       pr_info("mem auto-init: stack:%s, heap alloc:%s, heap free:%s\n",
+               stack, want_init_on_alloc(GFP_KERNEL) ? "on" : "off",
+               want_init_on_free() ? "on" : "off");
+}
Quoted 7 lines
Quoted 3 lines

To get a boot line like:

mem auto-init: stack:off, heap alloc:off, heap free:on

For stack there's no binary on/off, as you can potentially build half of the kernel with stack instrumentation and another half without it. We could make the instrumentation insert a static global flag into each translation unit, but this won't give us any interesting info.

Well, yes, that's technically true, but I think reporting the kernel config here would make sense. If someone intentionally bypasses the stack auto-init for portions of the kernel, we can't meaningfully report it here. There will be exceptions for stack auto-init and heap auto-init.

Signature · 1 line
Kees Cook
#5 lore raw

Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options

In reply to Alexander Potapenko

mm/slab.h static inline int cache_random_seq_create(struct kmem_cache *cachep, · line 538

Quoted 11 lines · Alexander Potapenko

[...]

mm/slab.h +4 -0
index 43ac818b8592..24ae887359b8 100644
@@ -524,4 +524,20 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep,
[...]
+static inline bool slab_want_init_on_free(struct kmem_cache *c)
+{
+	if (static_branch_unlikely(&init_on_free))
+		return !(c->ctor);

BTW, why is this checking for c->ctor here? Shouldn't it not matter for the free case?

Quoted 3 lines
mm/slab.h +3 -0
+	else
+		return false;
+}
Signature · 1 line
Kees Cook
#6 lore raw

Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options

Quoted 16 lines · Kees Cook

mm/slab.h static inline int cache_random_seq_create(struct kmem_cache *cachep, · line 538

Quoted 11 lines · Alexander Potapenko

[...]

mm/slab.h +4 -0
index 43ac818b8592..24ae887359b8 100644
@@ -524,4 +524,20 @@ static inline int cache_random_seq_create(struct kmem_cache *cachep,
[...]
+static inline bool slab_want_init_on_free(struct kmem_cache *c)
+{
+     if (static_branch_unlikely(&init_on_free))
+             return !(c->ctor);

BTW, why is this checking for c->ctor here? Shouldn't it not matter for the free case?

It does matter, see e.g. the handling of __OBJECT_POISON in slub.c If we just return true here, the kernel crashes.

Quoted 6 lines

mm/slab.h static inline int cache_random_seq_create(struct kmem_cache *cachep, · line 541

Quoted 3 lines
mm/slab.h +3 -0
+     else
+             return false;
+}

--
Kees Cook

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#7 lore raw

Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options

In reply to Alexander Potapenko
Quoted 11 lines · Alexander Potapenko

The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.

init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed.

init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.

Why do we need both? The later is more robust because even free memory cannot be sniffed and the overhead might be shifted from the allocation context (e.g. to RCU) but why cannot we stick to a single model?

Signature · 2 lines
Michal Hocko
SUSE Labs
#8 lore raw

Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options

Quoted 10 lines · Michal Hocko
Quoted 8 lines · Alexander Potapenko

The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.

init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed.

Quoted 7 lines
Quoted 3 lines

init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.

Why do we need both? The later is more robust because even free memory cannot be sniffed and the overhead might be shifted from the allocation context (e.g. to RCU) but why cannot we stick to a single model?

init_on_free appears to be slower because of cache effects. It's several % in the best case vs. <1% for init_on_alloc.

Quoted 3 lines

--
Michal Hocko
SUSE Labs

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#9 lore raw

Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options

Quoted 15 lines · Alexander Potapenko
Quoted 14 lines · Michal Hocko
Quoted 11 lines · Alexander Potapenko

The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.

init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed.

init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.

Quoted 5 lines
Quoted 3 lines

Why do we need both? The later is more robust because even free memory cannot be sniffed and the overhead might be shifted from the allocation context (e.g. to RCU) but why cannot we stick to a single model?

init_on_free appears to be slower because of cache effects. It's several % in the best case vs. <1% for init_on_alloc.

This doesn't really explain why we need both.

Signature · 2 lines
Michal Hocko
SUSE Labs
#10 lore raw

Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options

Quoted 16 lines · Michal Hocko
Quoted 15 lines · Alexander Potapenko
Quoted 14 lines · Michal Hocko
Quoted 11 lines · Alexander Potapenko

The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.

init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed.

init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Why do we need both? The later is more robust because even free memory cannot be sniffed and the overhead might be shifted from the allocation context (e.g. to RCU) but why cannot we stick to a single model?

init_on_free appears to be slower because of cache effects. It's several % in the best case vs. <1% for init_on_alloc.

This doesn't really explain why we need both.

There are a couple reasons. The first is that once we have hardware with memory tagging (e.g. arm64's MTE) we'll need both on_alloc and on_free hooks to do change the tags. With MTE, zeroing comes for "free" with tagging (though tagging is as slow as zeroing, so it's really the tagging that is free...), so we'll need to re-use the init_on_free infrastructure.

The second reason is for very paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructre for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series.

All that said, init_on_alloc looks desirable enough that distros will likely build with it enabled by default (I hope), and the very paranoid users will switch to (or additionally enable) init_on_free for their systems.

Signature · 1 line
Kees Cook
#11 lore raw

Re: [PATCH v2 1/4] mm: security: introduce init_on_alloc=1 and init_on_free=1 boot options

Quoted 23 lines · Kees Cook
Quoted 22 lines · Michal Hocko
Quoted 20 lines · Alexander Potapenko
Quoted 10 lines · Michal Hocko
Quoted 8 lines · Alexander Potapenko

The new options are needed to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.

init_on_alloc=1 makes the kernel initialize newly allocated pages and heap objects with zeroes. Initialization is done at allocation time at the places where checks for __GFP_ZERO are performed.

Quoted 7 lines
Quoted 3 lines

init_on_free=1 makes the kernel initialize freed pages and heap objects with zeroes upon their deletion. This helps to ensure sensitive data doesn't leak via use-after-free accesses.

Why do we need both? The later is more robust because even free memory cannot be sniffed and the overhead might be shifted from the allocation context (e.g. to RCU) but why cannot we stick to a single model?

init_on_free appears to be slower because of cache effects. It's several % in the best case vs. <1% for init_on_alloc.

Quoted 7 lines
Quoted 1 line

This doesn't really explain why we need both.

There are a couple reasons. The first is that once we have hardware with memory tagging (e.g. arm64's MTE) we'll need both on_alloc and on_free hooks to do change the tags. With MTE, zeroing comes for "free" with tagging (though tagging is as slow as zeroing, so it's really the tagging that is free...), so we'll need to re-use the init_on_free infrastructure.

I am not sure I follow, but ...

Quoted 8 lines

The second reason is for very paranoid use-cases where in-memory data lifetime is desired to be minimized. There are various arguments for/against the realism of the associated threat models, but given that we'll need the infrastructre for MTE anyway, and there are people who want wipe-on-free behavior no matter what the performance cost, it seems reasonable to include it in this series.

Quoted 4 lines

All that said, init_on_alloc looks desirable enough that distros will likely build with it enabled by default (I hope), and the very paranoid users will switch to (or additionally enable) init_on_free for their systems.

... this should all be part of the changelog.

Signature · 2 lines
Michal Hocko
SUSE Labs
#12 lore raw

PATCH v2 2/4 lib: introduce test_meminit module

In reply to Alexander Potapenko

Add tests for heap and pagealloc initialization.
These can be used to check init_on_alloc and init_on_free implementations as well as other approaches to initialization.

To: Kees Cook <[email protected]>
To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]

---
 lib/Kconfig.debug  |   8 ++
 lib/Makefile       |   1 +
 lib/test_meminit.c | 205 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 214 insertions(+)
 create mode 100644 lib/test_meminit.c
lib/Kconfig.debug +8 -0
index d695ec1477f3..6c3fc68a4a77 100644
@@ -2020,6 +2020,14 @@ config TEST_STACKINIT
 
 	  If unsure, say N.
 
+config TEST_MEMINIT
+	tristate "Test level of heap/page initialization"
+	help
+	  Test if the kernel is zero-initializing heap and page allocations.
+	  This can be useful to test init_on_alloc and init_on_free features.
+
+	  If unsure, say N.
+
 endif # RUNTIME_TESTING_MENU
 
 config MEMTEST
lib/Makefile +1 -0
index 83d7df2661ff..29c5afbe9882 100644
@@ -91,6 +91,7 @@ obj-$(CONFIG_TEST_DEBUG_VIRTUAL) += test_debug_virtual.o
 obj-$(CONFIG_TEST_MEMCAT_P) += test_memcat_p.o
 obj-$(CONFIG_TEST_OBJAGG) += test_objagg.o
 obj-$(CONFIG_TEST_STACKINIT) += test_stackinit.o
+obj-$(CONFIG_TEST_MEMINIT) += test_meminit.o
 
 obj-$(CONFIG_TEST_LIVEPATCH) += livepatch/
 
lib/test_meminit.c new file +205 -0

205 lines not on this page · raw message

Signature · 1 line
2.21.0.1020.gf2820cf01a-goog
Reviewed-by Tested-by on v2 2/4
#13 lore raw

Re: [PATCH v2 2/4] lib: introduce test_meminit module

Quoted 3 lines · Alexander Potapenko

Add tests for heap and pagealloc initialization.
These can be used to check init_on_alloc and init_on_free implementations as well as other approaches to initialization.

This is nice! Easy way to test the results. It might be helpful to show here what to expect when loading this module:

with either init_on_alloc=1 or init_on_free=1, I happily see:

test_meminit: all 10 tests in test_pages passed
test_meminit: all 40 tests in test_kvmalloc passed
test_meminit: all 20 tests in test_kmemcache passed
test_meminit: all 70 tests passed!

and without:

test_meminit: test_pages failed 10 out of 10 times
test_meminit: test_kvmalloc failed 40 out of 40 times
test_meminit: test_kmemcache failed 10 out of 20 times
test_meminit: failures: 60 out of 70

Quoted 2 lines

note below...

lib/test_meminit.c

Quoted 10 lines

[...]

lib/test_meminit.c new file +2 -0
new file mode 100644
index 000000000000..67d759498030
@@ -0,0 +1,205 @@
+// SPDX-License-Identifier: GPL-2.0
[...]
+module_init(test_meminit_init);

I get a warning at build about missing the license:

WARNING: modpost: missing MODULE_LICENSE() in lib/test_meminit.o

So, following the SPDX line, just add:

MODULE_LICENSE("GPL");

Signature · 1 line
Kees Cook
#14 lore raw

Re: [PATCH v2 2/4] lib: introduce test_meminit module

Quoted 8 lines · Kees Cook
Quoted 3 lines · Alexander Potapenko

Add tests for heap and pagealloc initialization.
These can be used to check init_on_alloc and init_on_free implementations as well as other approaches to initialization.

This is nice! Easy way to test the results. It might be helpful to show here what to expect when loading this module:

Do you want me to add the expected output to the patch description?

Quoted 36 lines

with either init_on_alloc=1 or init_on_free=1, I happily see:

test_meminit: all 10 tests in test_pages passed
test_meminit: all 40 tests in test_kvmalloc passed
test_meminit: all 20 tests in test_kmemcache passed
test_meminit: all 70 tests passed!

and without:

test_meminit: test_pages failed 10 out of 10 times
test_meminit: test_kvmalloc failed 40 out of 40 times
test_meminit: test_kmemcache failed 10 out of 20 times
test_meminit: failures: 60 out of 70

Quoted 2 lines

note below...

lib/test_meminit.c

Quoted 10 lines

[...]

lib/test_meminit.c new file +2 -0
new file mode 100644
index 000000000000..67d759498030
@@ -0,0 +1,205 @@
+// SPDX-License-Identifier: GPL-2.0
[...]
+module_init(test_meminit_init);

I get a warning at build about missing the license:

Quoted 5 lines

WARNING: modpost: missing MODULE_LICENSE() in lib/test_meminit.o

So, following the SPDX line, just add:

MODULE_LICENSE("GPL");

Will do, thanks!

Quoted 2 lines

--
Kees Cook

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#15 lore raw

Re: [PATCH v2 2/4] lib: introduce test_meminit module

Quoted 11 lines · Alexander Potapenko
Quoted 8 lines · Kees Cook
Quoted 3 lines · Alexander Potapenko

Add tests for heap and pagealloc initialization.
These can be used to check init_on_alloc and init_on_free implementations as well as other approaches to initialization.

This is nice! Easy way to test the results. It might be helpful to show here what to expect when loading this module:

Do you want me to add the expected output to the patch description?

Yes, I think it's worth having, as a way to show people what to expect when running the test, without having to actually enable, build, and run it themselves.

Signature · 1 line
Kees Cook
#16 lore raw

PATCH v2 3/4 gfp: mm: introduce __GFP_NO_AUTOINIT

In reply to Alexander Potapenko

When passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NO_AUTOINIT tells it to not initialize the requested memory if the init_on_alloc boot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.

__GFP_NO_AUTOINIT doesn't affect init_on_free behavior, except for SLOB, where init_on_free implies init_on_alloc.

__GFP_NO_AUTOINIT basically defeats the hardening against information leaks provided by init_on_alloc, so one should use it with caution.

This patch also adds __GFP_NO_AUTOINIT to alloc_pages() calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.

Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:

hackbench, init_on_free=1: +6.84% sys time (st.err 0.74%) hackbench, init_on_alloc=1: +7.25% sys time (st.err 0.72%)

Linux build with -j12, init_on_free=1: +8.52% wall time (st.err 0.42%) Linux build with -j12, init_on_free=1: +24.31% sys time (st.err 0.47%) Linux build with -j12, init_on_alloc=1: -0.16% wall time (st.err 0.40%) Linux build with -j12, init_on_alloc=1: +1.24% sys time (st.err 0.39%)

The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.

To: Andrew Morton <[email protected]>
To: Kees Cook <[email protected]>
To: Christoph Lameter <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: James Morris <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Souptick Joarder <[email protected]>
Cc: Matthew Wilcox <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v2:
- renamed __GFP_NOINIT to __GFP_NO_AUTOINIT, updated patch name/description

---
 include/linux/gfp.h | 13 +++++++++----
 include/linux/mm.h  |  2 +-
 kernel/kexec_core.c |  3 ++-
 mm/slab.c           |  2 +-
 mm/slob.c           |  3 ++-
 mm/slub.c           |  1 +
 6 files changed, 16 insertions(+), 8 deletions(-)
include/linux/gfp.h +9 -4
index fdab7de7490d..e1a83bd0ca67 100644
@@ -44,6 +44,7 @@ struct vm_area_struct;
 #else
 #define ___GFP_NOLOCKDEP	0
 #endif
+#define ___GFP_NO_AUTOINIT	0x1000000u
 /* If the above are modified, __GFP_BITS_SHIFT may need updating */
 
 /*
@@ -208,16 +209,20 @@ struct vm_area_struct;
  * %__GFP_COMP address compound page metadata.
  *
  * %__GFP_ZERO returns a zeroed page on success.
+ *
+ * %__GFP_NO_AUTOINIT requests non-initialized memory from the underlying
+ * allocator.
  */
-#define __GFP_NOWARN	((__force gfp_t)___GFP_NOWARN)
-#define __GFP_COMP	((__force gfp_t)___GFP_COMP)
-#define __GFP_ZERO	((__force gfp_t)___GFP_ZERO)
+#define __GFP_NOWARN		((__force gfp_t)___GFP_NOWARN)
+#define __GFP_COMP		((__force gfp_t)___GFP_COMP)
+#define __GFP_ZERO		((__force gfp_t)___GFP_ZERO)
+#define __GFP_NO_AUTOINIT	((__force gfp_t)___GFP_NO_AUTOINIT)
 
 /* Disable lockdep for GFP context tracking */
 #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP)
 
 /* Room for N __GFP_FOO bits */
-#define __GFP_BITS_SHIFT (23 + IS_ENABLED(CONFIG_LOCKDEP))
+#define __GFP_BITS_SHIFT (25)
 #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
 
 /**
include/linux/mm.h +1 -1
index 18d96f1d07c5..ce6c63396002 100644
@@ -2618,7 +2618,7 @@ DECLARE_STATIC_KEY_FALSE(init_on_alloc);
 static inline bool want_init_on_alloc(gfp_t flags)
 {
 	if (static_branch_unlikely(&init_on_alloc))
-		return true;
+		return !(flags & __GFP_NO_AUTOINIT);
 	return flags & __GFP_ZERO;
 }
 
kernel/kexec_core.c +2 -1
index 2f75dd0d0d81..7fc37bacac79 100644
@@ -302,7 +302,8 @@ static struct page *kimage_alloc_pages(gfp_t gfp_mask, unsigned int order)
 {
 	struct page *pages;
 
-	pages = alloc_pages(gfp_mask & ~__GFP_ZERO, order);
+	pages = alloc_pages((gfp_mask & ~__GFP_ZERO) | __GFP_NO_AUTOINIT,
+			    order);
 	if (pages) {
 		unsigned int count, i;
 
mm/slab.c +1 -1
index d00e9de26a45..1089461fc22b 100644
@@ -1393,7 +1393,7 @@ static struct page *kmem_getpages(struct kmem_cache *cachep, gfp_t flags,
 	struct page *page;
 	int nr_pages;
 
-	flags |= cachep->allocflags;
+	flags |= (cachep->allocflags | __GFP_NO_AUTOINIT);
 
 	page = __alloc_pages_node(nodeid, flags, cachep->gfporder);
 	if (!page) {
mm/slob.c +2 -1
index 351d3dfee000..d505f36aa398 100644
@@ -192,6 +192,7 @@ static void *slob_new_pages(gfp_t gfp, int order, int node)
 {
 	void *page;
 
+	gfp |= __GFP_NO_AUTOINIT;
 #ifdef CONFIG_NUMA
 	if (node != NUMA_NO_NODE)
 		page = __alloc_pages_node(node, gfp, order);
@@ -221,7 +222,7 @@ static inline bool slob_want_init_on_alloc(gfp_t flags, struct kmem_cache *c)
 {
 	if (static_branch_unlikely(&init_on_alloc) ||
 	    static_branch_unlikely(&init_on_free))
-		return c ? (!c->ctor) : true;
+		return c ? (!c->ctor) : !(flags & __GFP_NO_AUTOINIT);
 	return flags & __GFP_ZERO;
 }
 
mm/slub.c +1 -0
index 01424e910800..0aa306f5769a 100644
@@ -1495,6 +1495,7 @@ static inline struct page *alloc_slab_page(struct kmem_cache *s,
 	struct page *page;
 	unsigned int order = oo_order(oo);
 
+	flags |= __GFP_NO_AUTOINIT;
 	if (node == NUMA_NO_NODE)
 		page = alloc_pages(flags, order);
 	else
Signature · 1 line
2.21.0.1020.gf2820cf01a-goog
#17 lore raw

Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT

[It would be great to keep people involved in the previous version in the CC list]

Quoted 9 lines · Alexander Potapenko

When passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NO_AUTOINIT tells it to not initialize the requested memory if the init_on_alloc boot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.

__GFP_NO_AUTOINIT doesn't affect init_on_free behavior, except for SLOB, where init_on_free implies init_on_alloc.

Quoted 6 lines

__GFP_NO_AUTOINIT basically defeats the hardening against information leaks provided by init_on_alloc, so one should use it with caution.

This patch also adds __GFP_NO_AUTOINIT to alloc_pages() calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.

I still do not like the idea of a new gfp flag as explained in the previous email. People will simply use it incorectly or arbitrarily. We have that juicy experience from the past.

Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or GFP_ZERO) do not need it.

So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?

Signature · 2 lines
Michal Hocko
SUSE Labs
#18 lore raw

Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT

Quoted 3 lines · Michal this flag Hocko

[It would be great to keep people involved in the previous version in the CC list]

Yes, I've been trying to keep everyone in the loop, but your email fell through the cracks.
Sorry about that.

Quoted 13 lines · Alexander Potapenko
Quoted 12 lines

When passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NO_AUTOINIT tells it to not initialize the requested memory if the init_on_alloc boot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.

__GFP_NO_AUTOINIT doesn't affect init_on_free behavior, except for SLOB, where init_on_free implies init_on_alloc.

__GFP_NO_AUTOINIT basically defeats the hardening against information leaks provided by init_on_alloc, so one should use it with caution.

Quoted 7 lines
Quoted 3 lines

This patch also adds __GFP_NO_AUTOINIT to alloc_pages() calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.

I still do not like the idea of a new gfp flag as explained in the previous email. People will simply use it incorectly or arbitrarily. We have that juicy experience from the past.

Just to preserve some context, here's the previous email:
https://patchwork.kernel.org/patch/10907595/
(plus the patch removing GFP_TEMPORARY for the curious ones:
https://lwn.net/Articles/729145/)

Quoted 2 lines

Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or GFP_ZERO) do not need it.

Sorry, I didn't understand this piece. Could you please elaborate?

Quoted 4 lines

So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?

There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB. However the benchmark results provided in this and the previous patch don't show any noticeable difference - most certainly because the cost of initializing the page is amortized.
The second one was to fine-tune hackbench, for which the slowdown drops by a factor of 2.
But optimizing a mitigation for certain benchmarks is a questionable measure, so maybe we could really go without it.

Kees, what do you think?

Quoted 3 lines

--
Michal Hocko
SUSE Labs

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#19 lore raw

Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT

Quoted 8 lines · Alexander Potapenko
Quoted 3 lines · Michal this flag Hocko

[It would be great to keep people involved in the previous version in the CC list]

Yes, I've been trying to keep everyone in the loop, but your email fell through the cracks.
Sorry about that.

No problem

Quoted 17 lines
Quoted 17 lines · Alexander Potapenko
Quoted 15 lines

When passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NO_AUTOINIT tells it to not initialize the requested memory if the init_on_alloc boot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.

__GFP_NO_AUTOINIT doesn't affect init_on_free behavior, except for SLOB, where init_on_free implies init_on_alloc.

__GFP_NO_AUTOINIT basically defeats the hardening against information leaks provided by init_on_alloc, so one should use it with caution.

This patch also adds __GFP_NO_AUTOINIT to alloc_pages() calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.

Quoted 8 lines
Quoted 3 lines

I still do not like the idea of a new gfp flag as explained in the previous email. People will simply use it incorectly or arbitrarily. We have that juicy experience from the past.

Just to preserve some context, here's the previous email:
https://patchwork.kernel.org/patch/10907595/
(plus the patch removing GFP_TEMPORARY for the curious ones:
https://lwn.net/Articles/729145/)

Not only. GFP_REPEAT being another one and probably others I cannot remember from the top of my head.

Quoted 3 lines
Quoted 2 lines

Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or GFP_ZERO) do not need it.

Sorry, I didn't understand this piece. Could you please elaborate?

The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed. GFP_ZERO should be self explanatory.

Quoted 7 lines
Quoted 4 lines

So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?

There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.

Could you elaborate please?

Quoted 8 lines

However the benchmark results provided in this and the previous patch don't show any noticeable difference - most certainly because the cost of initializing the page is amortized.

The second one was to fine-tune hackbench, for which the slowdown drops by a factor of 2.
But optimizing a mitigation for certain benchmarks is a questionable measure, so maybe we could really go without it.

Agreed. Over optimization based on an artificial workloads tend to be dubious IMHO.

Signature · 2 lines
Michal Hocko
SUSE Labs
#20 lore raw

Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT

Quoted 42 lines · Michal Hocko
Quoted 8 lines · Alexander Potapenko
Quoted 3 lines · Michal this flag Hocko

[It would be great to keep people involved in the previous version in the CC list]

Yes, I've been trying to keep everyone in the loop, but your email fell through the cracks.
Sorry about that.

No problem

Quoted 17 lines
Quoted 17 lines · Alexander Potapenko
Quoted 15 lines

When passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NO_AUTOINIT tells it to not initialize the requested memory if the init_on_alloc boot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.

__GFP_NO_AUTOINIT doesn't affect init_on_free behavior, except for SLOB, where init_on_free implies init_on_alloc.

__GFP_NO_AUTOINIT basically defeats the hardening against information leaks provided by init_on_alloc, so one should use it with caution.

This patch also adds __GFP_NO_AUTOINIT to alloc_pages() calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.

Quoted 8 lines
Quoted 3 lines

I still do not like the idea of a new gfp flag as explained in the previous email. People will simply use it incorectly or arbitrarily. We have that juicy experience from the past.

Just to preserve some context, here's the previous email:
https://patchwork.kernel.org/patch/10907595/
(plus the patch removing GFP_TEMPORARY for the curious ones:
https://lwn.net/Articles/729145/)

Not only. GFP_REPEAT being another one and probably others I cannot remember from the top of my head.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or GFP_ZERO) do not need it.

Sorry, I didn't understand this piece. Could you please elaborate?

The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed. GFP_ZERO should be self explanatory.

Ah, I see. We already do that, see the want_init_on_alloc() implementation here: https://patchwork.kernel.org/patch/10943087/

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?

There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.

Could you elaborate please?

When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.

Quoted 15 lines
Quoted 8 lines

However the benchmark results provided in this and the previous patch don't show any noticeable difference - most certainly because the cost of initializing the page is amortized.

The second one was to fine-tune hackbench, for which the slowdown drops by a factor of 2.
But optimizing a mitigation for certain benchmarks is a questionable measure, so maybe we could really go without it.

Agreed. Over optimization based on an artificial workloads tend to be dubious IMHO.

--
Michal Hocko
SUSE Labs

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#21 lore raw

Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT

Quoted 57 lines · Alexander Potapenko
Quoted 42 lines · Michal Hocko
Quoted 8 lines · Alexander Potapenko
Quoted 3 lines · Michal this flag Hocko

[It would be great to keep people involved in the previous version in the CC list]

Yes, I've been trying to keep everyone in the loop, but your email fell through the cracks.
Sorry about that.

No problem

Quoted 17 lines
Quoted 17 lines · Alexander Potapenko
Quoted 15 lines

When passed to an allocator (either pagealloc or SL[AOU]B), __GFP_NO_AUTOINIT tells it to not initialize the requested memory if the init_on_alloc boot option is enabled. This can be useful in the cases newly allocated memory is going to be initialized by the caller right away.

__GFP_NO_AUTOINIT doesn't affect init_on_free behavior, except for SLOB, where init_on_free implies init_on_alloc.

__GFP_NO_AUTOINIT basically defeats the hardening against information leaks provided by init_on_alloc, so one should use it with caution.

This patch also adds __GFP_NO_AUTOINIT to alloc_pages() calls in SL[AOU]B. Doing so is safe, because the heap allocators initialize the pages they receive before passing memory to the callers.

Quoted 8 lines
Quoted 3 lines

I still do not like the idea of a new gfp flag as explained in the previous email. People will simply use it incorectly or arbitrarily. We have that juicy experience from the past.

Just to preserve some context, here's the previous email:
https://patchwork.kernel.org/patch/10907595/
(plus the patch removing GFP_TEMPORARY for the curious ones:
https://lwn.net/Articles/729145/)

Not only. GFP_REPEAT being another one and probably others I cannot remember from the top of my head.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or GFP_ZERO) do not need it.

Sorry, I didn't understand this piece. Could you please elaborate?

The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed. GFP_ZERO should be self explanatory.

Ah, I see. We already do that, see the want_init_on_alloc() implementation here: https://patchwork.kernel.org/patch/10943087/

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.

Could you elaborate please?

When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator

... when the feature is enabled ...

Quoted 5 lines

and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.

OK, I see what you mean now. Is there any way to special case the page allocation for this feature? E.g. your implementation tries to make this zeroying special but why cannot you simply do this

struct page *
____alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)

{
	//current implementation
}

struct page *
__alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)

{
	if (your_feature_enabled)
		gfp_mask |= __GFP_ZERO;
	return ____alloc_pages_nodemask(gfp_mask, order, preferred_nid,
					nodemask);
}

and use ____alloc_pages_nodemask from the slab or other internal allocators?

Signature · 2 lines
Michal Hocko
SUSE Labs
#22 lore raw

Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT

Quoted 53 lines · Michal Hocko
Quoted 14 lines · Alexander Potapenko
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or GFP_ZERO) do not need it.

Sorry, I didn't understand this piece. Could you please elaborate?

The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed. GFP_ZERO should be self explanatory.

Ah, I see. We already do that, see the want_init_on_alloc() implementation here: https://patchwork.kernel.org/patch/10943087/

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.

Could you elaborate please?

When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator

... when the feature is enabled ...

Quoted 5 lines

and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.

OK, I see what you mean now. Is there any way to special case the page allocation for this feature? E.g. your implementation tries to make this zeroying special but why cannot you simply do this

struct page *
____alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)

{
	//current implementation
}

struct page *
__alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)

{
	if (your_feature_enabled)
		gfp_mask |= __GFP_ZERO;
	return ____alloc_pages_nodemask(gfp_mask, order, preferred_nid,
					nodemask);
}
Quoted 2 lines

and use ____alloc_pages_nodemask from the slab or other internal allocators?

If an additional allocator function is preferred over a new GFP flag, then I don't see any reason not to do this. (Though adding more "__"s seems a bit unfriendly to code-documentation.) What might be better naming?

This would mean that the skb changes later in the series would use the "no auto init" version of the allocator too, then.

Signature · 1 line
Kees Cook
#23 lore raw

Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT

Quoted 54 lines · Kees Cook
Quoted 53 lines · Michal Hocko
Quoted 14 lines · Alexander Potapenko
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or GFP_ZERO) do not need it.

Sorry, I didn't understand this piece. Could you please elaborate?

The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed. GFP_ZERO should be self explanatory.

Ah, I see. We already do that, see the want_init_on_alloc() implementation here: https://patchwork.kernel.org/patch/10943087/

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.

Could you elaborate please?

When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator

... when the feature is enabled ...

Quoted 5 lines

and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.

OK, I see what you mean now. Is there any way to special case the page allocation for this feature? E.g. your implementation tries to make this zeroying special but why cannot you simply do this

struct page *
____alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)

{
	//current implementation
}

struct page *
__alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)

{
	if (your_feature_enabled)
		gfp_mask |= __GFP_ZERO;
	return ____alloc_pages_nodemask(gfp_mask, order, preferred_nid,
					nodemask);
}
Quoted 6 lines
Quoted 2 lines

and use ____alloc_pages_nodemask from the slab or other internal allocators?

If an additional allocator function is preferred over a new GFP flag, then I don't see any reason not to do this. (Though adding more "__"s seems a bit unfriendly to code-documentation.) What might be better naming?

The naminig is the last thing I would be worried about. Let's focus on the most simplistic implementation first. And means, can we really make it as simple as above? At least on the page allocator level.

Quoted 2 lines

This would mean that the skb changes later in the series would use the "no auto init" version of the allocator too, then.

No, this would be an internal function to MM. I would really like to optimize once there are numbers from _real_ workloads to base those optimizations.

Signature · 2 lines
Michal Hocko
SUSE Labs
#24 lore raw

Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT

Quoted 56 lines · Michal Hocko
Quoted 54 lines · Kees Cook
Quoted 53 lines · Michal Hocko
Quoted 14 lines · Alexander Potapenko
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or GFP_ZERO) do not need it.

Sorry, I didn't understand this piece. Could you please elaborate?

The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed. GFP_ZERO should be self explanatory.

Ah, I see. We already do that, see the want_init_on_alloc() implementation here: https://patchwork.kernel.org/patch/10943087/

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.

Could you elaborate please?

When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator

... when the feature is enabled ...

Quoted 5 lines

and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.

OK, I see what you mean now. Is there any way to special case the page allocation for this feature? E.g. your implementation tries to make this zeroying special but why cannot you simply do this

struct page *
____alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)

{
    //current implementation
}

struct page *
__alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)

{
    if (your_feature_enabled)
            gfp_mask |= __GFP_ZERO;
    return ____alloc_pages_nodemask(gfp_mask, order, preferred_nid,
                                    nodemask);
}
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

and use ____alloc_pages_nodemask from the slab or other internal allocators?

Given that calling alloc_pages() with __GFP_NO_AUTOINIT doesn't visibly improve the chosen benchmarks,
and the next patch in the series ("net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations") only improves hackbench, shall we maybe drop both patches altogether?

Quoted 17 lines
Quoted 3 lines

If an additional allocator function is preferred over a new GFP flag, then I don't see any reason not to do this. (Though adding more "__"s seems a bit unfriendly to code-documentation.) What might be better naming?

The naminig is the last thing I would be worried about. Let's focus on the most simplistic implementation first. And means, can we really make it as simple as above? At least on the page allocator level.

Quoted 2 lines

This would mean that the skb changes later in the series would use the "no auto init" version of the allocator too, then.

No, this would be an internal function to MM. I would really like to optimize once there are numbers from _real_ workloads to base those optimizations.
--
Michal Hocko
SUSE Labs

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#25 lore raw

Re: [PATCH v2 3/4] gfp: mm: introduce __GFP_NO_AUTOINIT

Quoted 57 lines · Alexander Potapenko
Quoted 56 lines · Michal Hocko
Quoted 54 lines · Kees Cook
Quoted 53 lines · Michal Hocko
Quoted 14 lines · Alexander Potapenko
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Freeing a memory is an opt-in feature and the slab allocator can already tell many (with constructor or GFP_ZERO) do not need it.

Sorry, I didn't understand this piece. Could you please elaborate?

The allocator can assume that caches with a constructor will initialize the object so additional zeroying is not needed. GFP_ZERO should be self explanatory.

Ah, I see. We already do that, see the want_init_on_alloc() implementation here: https://patchwork.kernel.org/patch/10943087/

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

So can we go without this gfp thing and see whether somebody actually finds a performance problem with the feature enabled and think about what can we do about it rather than add this maint. nightmare from the very beginning?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

There were two reasons to introduce this flag initially. The first was double initialization of pages allocated for SLUB.

Could you elaborate please?

When the kernel allocates an object from SLUB, and SLUB happens to be short on free pages, it requests some from the page allocator. Those pages are initialized by the page allocator

... when the feature is enabled ...

Quoted 5 lines

and split into objects. Finally SLUB initializes one of the available objects and returns it back to the kernel.
Therefore the object is initialized twice for the first time (when it comes directly from the page allocator).
This cost is however amortized by SLUB reusing the object after it's been freed.

OK, I see what you mean now. Is there any way to special case the page allocation for this feature? E.g. your implementation tries to make this zeroying special but why cannot you simply do this

struct page *
____alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)

{
    //current implementation
}

struct page *
__alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
nodemask_t *nodemask)

{
    if (your_feature_enabled)
            gfp_mask |= __GFP_ZERO;
    return ____alloc_pages_nodemask(gfp_mask, order, preferred_nid,
                                    nodemask);
}
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

and use ____alloc_pages_nodemask from the slab or other internal allocators?

Given that calling alloc_pages() with __GFP_NO_AUTOINIT doesn't visibly improve the chosen benchmarks,
and the next patch in the series ("net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations") only improves hackbench, shall we maybe drop both patches altogether?

Ohh, by all means. I was suggesting the same few emails ago. The above is just a hint on how to implement the feature on the page allocator level rather than hooking into the prep_new_page and add another branch to zero memory.

Signature · 2 lines
Michal Hocko
SUSE Labs
#26 lore raw

PATCH v2 4/4 net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations

In reply to Alexander Potapenko

Add sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in the init_on_alloc mode from 6.84% to 3.45%.

Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:

hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)

Linux build with -j12, init_on_free=1: +8.34% wall time (st.err 0.39%) Linux build with -j12, init_on_free=1: +24.13% sys time (st.err 0.47%) Linux build with -j12, init_on_alloc=1: -0.04% wall time (st.err 0.46%) Linux build with -j12, init_on_alloc=1: +0.50% sys time (st.err 0.45%)

The slowdown for init_on_free=0, init_on_alloc=0 compared to the baseline is within the standard error.

To: Andrew Morton <[email protected]>
To: Christoph Lameter <[email protected]>
To: Kees Cook <[email protected]>
Cc: Masahiro Yamada <[email protected]>
Cc: James Morris <[email protected]>
Cc: "Serge E. Hallyn" <[email protected]>
Cc: Nick Desaulniers <[email protected]>
Cc: Kostya Serebryany <[email protected]>
Cc: Dmitry Vyukov <[email protected]>
Cc: Sandeep Patil <[email protected]>
Cc: Laura Abbott <[email protected]>
Cc: Randy Dunlap <[email protected]>
Cc: Jann Horn <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: [email protected]
Cc: [email protected]
Cc: [email protected]
---
v2:
- changed __GFP_NOINIT to __GFP_NO_AUTOINIT

---
 include/net/sock.h |  5 +++++
 net/core/sock.c    | 29 +++++++++++++++++++++++++----
 net/unix/af_unix.c | 13 +++++++------
 3 files changed, 37 insertions(+), 10 deletions(-)
include/net/sock.h +5 -0
index 4d208c0f9c14..0dcb90a0c14d 100644
@@ -1626,6 +1626,11 @@ struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size,
 struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
 				     unsigned long data_len, int noblock,
 				     int *errcode, int max_page_order);
+struct sk_buff *sock_alloc_send_pskb_noinit(struct sock *sk,
+					    unsigned long header_len,
+					    unsigned long data_len,
+					    int noblock, int *errcode,
+					    int max_page_order);
 void *sock_kmalloc(struct sock *sk, int size, gfp_t priority);
 void sock_kfree_s(struct sock *sk, void *mem, int size);
 void sock_kzfree_s(struct sock *sk, void *mem, int size);
net/core/sock.c +25 -4
index 9ceb90c875bc..7c24b70b7069 100644
@@ -2192,9 +2192,11 @@ static long sock_wait_for_wmem(struct sock *sk, long timeo)
  *	Generic send/receive buffer handlers
  */
 
-struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
-				     unsigned long data_len, int noblock,
-				     int *errcode, int max_page_order)
+struct sk_buff *sock_alloc_send_pskb_internal(struct sock *sk,
+					      unsigned long header_len,
+					      unsigned long data_len,
+					      int noblock, int *errcode,
+					      int max_page_order, gfp_t gfp)
 {
 	struct sk_buff *skb;
 	long timeo;
@@ -2223,7 +2225,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
 		timeo = sock_wait_for_wmem(sk, timeo);
 	}
 	skb = alloc_skb_with_frags(header_len, data_len, max_page_order,
-				   errcode, sk->sk_allocation);
+				   errcode, sk->sk_allocation | gfp);
 	if (skb)
 		skb_set_owner_w(skb, sk);
 	return skb;
@@ -2234,8 +2236,27 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
 	*errcode = err;
 	return NULL;
 }
+
+struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
+				     unsigned long data_len, int noblock,
+				     int *errcode, int max_page_order)
+{
+	return sock_alloc_send_pskb_internal(sk, header_len, data_len,
+		noblock, errcode, max_page_order, /*gfp*/0);
+}
 EXPORT_SYMBOL(sock_alloc_send_pskb);
 
+struct sk_buff *sock_alloc_send_pskb_noinit(struct sock *sk,
+					    unsigned long header_len,
+					    unsigned long data_len,
+					    int noblock, int *errcode,
+					    int max_page_order)
+{
+	return sock_alloc_send_pskb_internal(sk, header_len, data_len,
+		noblock, errcode, max_page_order, /*gfp*/__GFP_NO_AUTOINIT);
+}
+EXPORT_SYMBOL(sock_alloc_send_pskb_noinit);
+
 struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size,
 				    int noblock, int *errcode)
 {
net/unix/af_unix.c +7 -6
index e68d7454f2e3..a4c15620b66d 100644
@@ -1627,9 +1627,9 @@ static int unix_dgram_sendmsg(struct socket *sock, struct msghdr *msg,
 		BUILD_BUG_ON(SKB_MAX_ALLOC < PAGE_SIZE);
 	}
 
-	skb = sock_alloc_send_pskb(sk, len - data_len, data_len,
-				   msg->msg_flags & MSG_DONTWAIT, &err,
-				   PAGE_ALLOC_COSTLY_ORDER);
+	skb = sock_alloc_send_pskb_noinit(sk, len - data_len, data_len,
+					  msg->msg_flags & MSG_DONTWAIT, &err,
+					  PAGE_ALLOC_COSTLY_ORDER);
 	if (skb == NULL)
 		goto out;
 
@@ -1824,9 +1824,10 @@ static int unix_stream_sendmsg(struct socket *sock, struct msghdr *msg,
 
 		data_len = min_t(size_t, size, PAGE_ALIGN(data_len));
 
-		skb = sock_alloc_send_pskb(sk, size - data_len, data_len,
-					   msg->msg_flags & MSG_DONTWAIT, &err,
-					   get_order(UNIX_SKB_FRAGS_SZ));
+		skb = sock_alloc_send_pskb_noinit(sk, size - data_len, data_len,
+						  msg->msg_flags & MSG_DONTWAIT,
+						  &err,
+						  get_order(UNIX_SKB_FRAGS_SZ));
 		if (!skb)
 			goto out_err;
 
Signature · 1 line
2.21.0.1020.gf2820cf01a-goog
#27 lore raw

Re: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations

Quoted 4 lines · Alexander Potapenko

Add sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in the init_on_alloc mode from 6.84% to 3.45%.

Out of curiosity, why the creation of the new function over adding a gfp flag argument to sock_alloc_send_pskb() and updating callers? (There are only 6 callers, and this change already updates 2 of those.)

Quoted 5 lines

Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:

hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)

In the commit log it might be worth mentioning that this is only
changing the init_on_alloc case (in case it's not already obvious to
folks). Perhaps there needs to be a split of __GFP_NO_AUTOINIT into
__GFP_NO_AUTO_ALLOC_INIT and __GFP_NO_AUTO_FREE_INIT? Right now __GFP_NO_AUTOINIT is only checked for init_on_alloc:

static inline bool want_init_on_alloc(gfp_t flags)

{
        if (static_branch_unlikely(&init_on_alloc))
                return !(flags & __GFP_NO_AUTOINIT);
        return flags & __GFP_ZERO;
}
...
static inline bool want_init_on_free(void)
{
        return static_branch_unlikely(&init_on_free);
}

On a related note, it might be nice to add an exclusion list to the kmem_cache_create() cases, since it seems likely that further tuning will be needed there. For example, with the init_on_free-similar PAX_MEMORY_SANITIZE changes in the last public release of PaX/grsecurity, the following were excluded from wipe-on-free:

buffer_head
names_cache
mm_struct
vm_area_struct
anon_vma
anon_vma_chain
skbuff_head_cache
skbuff_fclone_cache

Adding these and others (with details on why they were selected), might improve init_on_free performance further without trading too much coverage.

Having a kernel param with a comma-separated list of cache names and the logic to add __GFP_NO_AUTOINIT at creation time would be a nice (and cheap!) debug feature to let folks tune things for their specific workloads, if they choose to. (And it could maybe also know what "none" meant, to actually remove the built-in exclusions, similar to what PaX's "pax_sanitize_slab=full" does.)

Signature · 1 line
Kees Cook
#28 lore raw

Re: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations

Quoted 15 lines · Kees Cook
Quoted 4 lines · Alexander Potapenko

Add sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in the init_on_alloc mode from 6.84% to 3.45%.

Out of curiosity, why the creation of the new function over adding a gfp flag argument to sock_alloc_send_pskb() and updating callers? (There are only 6 callers, and this change already updates 2 of those.)

Quoted 5 lines

Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:

hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)

So I've run some of my own wall-clock timings of kernel builds (which should be an pretty big "worst case" situation, and I see much smaller performance changes:

everything off

	Run times: 289.18 288.61 289.66 287.71 287.67
	Min: 287.67 Max: 289.66 Mean: 288.57 Std Dev: 0.79
		baseline

init_on_alloc=1

	Run times: 289.72 286.95 287.87 287.34 287.35
	Min: 286.95 Max: 289.72 Mean: 287.85 Std Dev: 0.98
		0.25% faster (within the std dev noise)

init_on_free=1

	Run times: 303.26 301.44 301.19 301.55 301.39
	Min: 301.19 Max: 303.26 Mean: 301.77 Std Dev: 0.75
		4.57% slower

init_on_free=1 with the PAX_MEMORY_SANITIZE slabs excluded:

	Run times: 299.19 299.85 298.95 298.23 298.64
	Min: 298.23 Max: 299.85 Mean: 298.97 Std Dev: 0.55
		3.60% slower

So the tuning certainly improved things by 1%. My perf numbers don't show the 24% hit you were seeing at all, though.

Quoted 5 lines

In the commit log it might be worth mentioning that this is only changing the init_on_alloc case (in case it's not already obvious to folks). Perhaps there needs to be a split of __GFP_NO_AUTOINIT into __GFP_NO_AUTO_ALLOC_INIT and __GFP_NO_AUTO_FREE_INIT? Right now __GFP_NO_AUTOINIT is only checked for init_on_alloc:

I was obviously crazy here. :) GFP isn't present for free(), but a SLAB flag works (as was done in PAX_MEMORY_SANITIZE). I'll send the patch I used for the above timing test.

Signature · 1 line
Kees Cook
#29 lore raw

Re: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations

Quoted 15 lines · Kees Cook
Quoted 13 lines · Kees Cook
Quoted 4 lines · Alexander Potapenko

Add sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in the init_on_alloc mode from 6.84% to 3.45%.

Out of curiosity, why the creation of the new function over adding a gfp flag argument to sock_alloc_send_pskb() and updating callers? (There are only 6 callers, and this change already updates 2 of those.)

Quoted 3 lines

Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)

So I've run some of my own wall-clock timings of kernel builds (which should be an pretty big "worst case" situation, and I see much smaller performance changes:

How many cores were you using? I suspect the numbers may vary a bit depending on that.

Quoted 15 lines

everything off

        Run times: 289.18 288.61 289.66 287.71 287.67
        Min: 287.67 Max: 289.66 Mean: 288.57 Std Dev: 0.79
                baseline

init_on_alloc=1

        Run times: 289.72 286.95 287.87 287.34 287.35
        Min: 286.95 Max: 289.72 Mean: 287.85 Std Dev: 0.98
                0.25% faster (within the std dev noise)

init_on_free=1

        Run times: 303.26 301.44 301.19 301.55 301.39
        Min: 301.19 Max: 303.26 Mean: 301.77 Std Dev: 0.75
                4.57% slower
Quoted 7 lines

init_on_free=1 with the PAX_MEMORY_SANITIZE slabs excluded:

        Run times: 299.19 299.85 298.95 298.23 298.64
        Min: 298.23 Max: 299.85 Mean: 298.97 Std Dev: 0.55
                3.60% slower

So the tuning certainly improved things by 1%. My perf numbers don't show the 24% hit you were seeing at all, though.

Note that 24% is the _sys_ time slowdown. The wall time slowdown seen in this case was 8.34%

Quoted 12 lines
Quoted 5 lines

In the commit log it might be worth mentioning that this is only changing the init_on_alloc case (in case it's not already obvious to folks). Perhaps there needs to be a split of __GFP_NO_AUTOINIT into __GFP_NO_AUTO_ALLOC_INIT and __GFP_NO_AUTO_FREE_INIT? Right now __GFP_NO_AUTOINIT is only checked for init_on_alloc:

I was obviously crazy here. :) GFP isn't present for free(), but a SLAB flag works (as was done in PAX_MEMORY_SANITIZE). I'll send the patch I used for the above timing test.

--
Kees Cook

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#30 lore raw

Re: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations

Quoted 45 lines · Alexander Potapenko
Quoted 15 lines · Kees Cook
Quoted 13 lines · Kees Cook
Quoted 4 lines · Alexander Potapenko

Add sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in the init_on_alloc mode from 6.84% to 3.45%.

Out of curiosity, why the creation of the new function over adding a gfp flag argument to sock_alloc_send_pskb() and updating callers? (There are only 6 callers, and this change already updates 2 of those.)

Quoted 3 lines

Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)

So I've run some of my own wall-clock timings of kernel builds (which should be an pretty big "worst case" situation, and I see much smaller performance changes:

How many cores were you using? I suspect the numbers may vary a bit depending on that.

Quoted 20 lines

everything off

        Run times: 289.18 288.61 289.66 287.71 287.67
        Min: 287.67 Max: 289.66 Mean: 288.57 Std Dev: 0.79
                baseline

init_on_alloc=1

        Run times: 289.72 286.95 287.87 287.34 287.35
        Min: 286.95 Max: 289.72 Mean: 287.85 Std Dev: 0.98
                0.25% faster (within the std dev noise)

init_on_free=1

        Run times: 303.26 301.44 301.19 301.55 301.39
        Min: 301.19 Max: 303.26 Mean: 301.77 Std Dev: 0.75
                4.57% slower

init_on_free=1 with the PAX_MEMORY_SANITIZE slabs excluded:

        Run times: 299.19 299.85 298.95 298.23 298.64
        Min: 298.23 Max: 299.85 Mean: 298.97 Std Dev: 0.55
                3.60% slower
Quoted 4 lines
Quoted 2 lines

So the tuning certainly improved things by 1%. My perf numbers don't show the 24% hit you were seeing at all, though.

Note that 24% is the _sys_ time slowdown. The wall time slowdown seen in this case was 8.34%

I've collected more stats running QEMU with different numbers of cores. The slowdown values of init_on_free compared to baseline are:
2 CPUs - 5.94% for wall time (20.08% for sys time)
6 CPUs - 7.43% for wall time (23.55% for sys time)
12 CPUs - 8.41% for wall time (24.25% for sys time)
24 CPUs - 9.49% for wall time (17.98% for sys time)

I'm building a defconfig of some fixed KMSAN tree with Clang, but that shouldn't matter much.

Quoted 26 lines
Quoted 6 lines
Quoted 5 lines

In the commit log it might be worth mentioning that this is only changing the init_on_alloc case (in case it's not already obvious to folks). Perhaps there needs to be a split of __GFP_NO_AUTOINIT into __GFP_NO_AUTO_ALLOC_INIT and __GFP_NO_AUTO_FREE_INIT? Right now __GFP_NO_AUTOINIT is only checked for init_on_alloc:

Quoted 6 lines

I was obviously crazy here. :) GFP isn't present for free(), but a SLAB flag works (as was done in PAX_MEMORY_SANITIZE). I'll send the patch I used for the above timing test.

--
Kees Cook

--
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#31 lore raw

Re: [PATCH v2 4/4] net: apply __GFP_NO_AUTOINIT to AF_UNIX sk_buff allocations

In reply to Alexander Potapenko
Quoted 15 lines · Alexander Potapenko
Quoted 14 lines · Kees Cook
Quoted 13 lines · Kees Cook
Quoted 4 lines · Alexander Potapenko

Add sock_alloc_send_pskb_noinit(), which is similar to sock_alloc_send_pskb(), but allocates with __GFP_NO_AUTOINIT. This helps reduce the slowdown on hackbench in the init_on_alloc mode from 6.84% to 3.45%.

Out of curiosity, why the creation of the new function over adding a gfp flag argument to sock_alloc_send_pskb() and updating callers? (There are only 6 callers, and this change already updates 2 of those.)

Quoted 3 lines

Slowdown for the initialization features compared to init_on_free=0, init_on_alloc=0:

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

hackbench, init_on_free=1: +7.71% sys time (st.err 0.45%) hackbench, init_on_alloc=1: +3.45% sys time (st.err 0.86%)

So I've run some of my own wall-clock timings of kernel builds (which should be an pretty big "worst case" situation, and I see much smaller performance changes:

How many cores were you using? I suspect the numbers may vary a bit depending on that.

I was using 4.

Quoted 15 lines
Quoted 15 lines

init_on_alloc=1

        Run times: 289.72 286.95 287.87 287.34 287.35
        Min: 286.95 Max: 289.72 Mean: 287.85 Std Dev: 0.98
                0.25% faster (within the std dev noise)

init_on_free=1

        Run times: 303.26 301.44 301.19 301.55 301.39
        Min: 301.19 Max: 303.26 Mean: 301.77 Std Dev: 0.75
                4.57% slower

init_on_free=1 with the PAX_MEMORY_SANITIZE slabs excluded:

        Run times: 299.19 299.85 298.95 298.23 298.64
        Min: 298.23 Max: 299.85 Mean: 298.97 Std Dev: 0.55
                3.60% slower
Quoted 4 lines
Quoted 2 lines

So the tuning certainly improved things by 1%. My perf numbers don't show the 24% hit you were seeing at all, though.

Note that 24% is the _sys_ time slowdown. The wall time slowdown seen in this case was 8.34%

Ah! Gotcha. Yeah, seems the impact for init_on_free is pretty variable. The init_on_alloc appears close to free, though.

Signature · 1 line
Kees Cook
#32 lore raw

[PATCH 5/4] mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches

In reply to Alexander Potapenko

In order to improve the init_on_free performance, some frequently freed caches with less sensitive contents can be excluded from the init_on_free behavior.

This patch is modified from Brad Spengler/PaX Team's code in the last public patch of grsecurity/PaX based on my understanding of the code. Changes or omissions from the original code are mine and don't reflect the original grsecurity/PaX code.

---
 fs/buffer.c          | 2 +-
 fs/dcache.c          | 3 ++-
 include/linux/slab.h | 3 +++
 kernel/fork.c        | 6 ++++--
 mm/rmap.c            | 5 +++--
 mm/slab.h            | 5 +++--
 net/core/skbuff.c    | 6 ++++--
 7 files changed, 20 insertions(+), 10 deletions(-)
fs/buffer.c +1 -1
index 0faa41fb4c88..04a85bd4cf2e 100644
@@ -3457,7 +3457,7 @@ void __init buffer_init(void)
 	bh_cachep = kmem_cache_create("buffer_head",
 			sizeof(struct buffer_head), 0,
 				(SLAB_RECLAIM_ACCOUNT|SLAB_PANIC|
-				SLAB_MEM_SPREAD),
+				SLAB_MEM_SPREAD|SLAB_NO_FREE_INIT),
 				NULL);
 
 	/*
fs/dcache.c +2 -1
index 8136bda27a1f..323b039accba 100644
@@ -3139,7 +3139,8 @@ void __init vfs_caches_init_early(void)
 void __init vfs_caches_init(void)
 {
 	names_cachep = kmem_cache_create_usercopy("names_cache", PATH_MAX, 0,
-			SLAB_HWCACHE_ALIGN|SLAB_PANIC, 0, PATH_MAX, NULL);
+			SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_NO_FREE_INIT, 0,
+			PATH_MAX, NULL);
 
 	dcache_init();
 	inode_init();
include/linux/slab.h +3 -0
index 9449b19c5f10..7eba9ad8830d 100644
@@ -92,6 +92,9 @@
 /* Avoid kmemleak tracing */
 #define SLAB_NOLEAKTRACE	((slab_flags_t __force)0x00800000U)
 
+/* Exclude slab from zero-on-free when init_on_free is enabled */
+#define SLAB_NO_FREE_INIT	((slab_flags_t __force)0x01000000U)
+
 /* Fault injection mark */
 #ifdef CONFIG_FAILSLAB
 # define SLAB_FAILSLAB		((slab_flags_t __force)0x02000000U)
kernel/fork.c +4 -2
index b4cba953040a..9868585f5520 100644
@@ -2550,11 +2550,13 @@ void __init proc_caches_init(void)
 
 	mm_cachep = kmem_cache_create_usercopy("mm_struct",
 			mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
-			SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
+			SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT|
+			SLAB_NO_FREE_INIT,
 			offsetof(struct mm_struct, saved_auxv),
 			sizeof_field(struct mm_struct, saved_auxv),
 			NULL);
-	vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT);
+	vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT|
+						    SLAB_NO_FREE_INIT);
 	mmap_init();
 	nsproxy_cache_init();
 }
mm/rmap.c +3 -2
index e5dfe2ae6b0d..b7b8013eeb0a 100644
@@ -432,10 +432,11 @@ static void anon_vma_ctor(void *data)
 void __init anon_vma_init(void)
 {
 	anon_vma_cachep = kmem_cache_create("anon_vma", sizeof(struct anon_vma),
-			0, SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT,
+			0, SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT|
+			SLAB_NO_FREE_INIT,
 			anon_vma_ctor);
 	anon_vma_chain_cachep = KMEM_CACHE(anon_vma_chain,
-			SLAB_PANIC|SLAB_ACCOUNT);
+			SLAB_PANIC|SLAB_ACCOUNT|SLAB_NO_FREE_INIT);
 }
 
 /*
mm/slab.h +3 -2
index 24ae887359b8..f95b4f03c57b 100644
@@ -129,7 +129,8 @@ static inline slab_flags_t kmem_cache_flags(unsigned int object_size,
 /* Legal flag mask for kmem_cache_create(), for various configurations */
 #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
 			 SLAB_CACHE_DMA32 | SLAB_PANIC | \
-			 SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS )
+			 SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
+			 SLAB_NO_FREE_INIT)
 
 #if defined(CONFIG_DEBUG_SLAB)
 #define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER)
@@ -535,7 +536,7 @@ static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c)
 static inline bool slab_want_init_on_free(struct kmem_cache *c)
 {
 	if (static_branch_unlikely(&init_on_free))
-		return !(c->ctor);
+		return !(c->ctor) && ((c->flags & SLAB_NO_FREE_INIT) == 0);
 	else
 		return false;
 }
net/core/skbuff.c +4 -2
index e89be6282693..b65902d2c042 100644
@@ -3981,14 +3981,16 @@ void __init skb_init(void)
 	skbuff_head_cache = kmem_cache_create_usercopy("skbuff_head_cache",
 					      sizeof(struct sk_buff),
 					      0,
-					      SLAB_HWCACHE_ALIGN|SLAB_PANIC,
+					      SLAB_HWCACHE_ALIGN|SLAB_PANIC|
+					      SLAB_NO_FREE_INIT,
 					      offsetof(struct sk_buff, cb),
 					      sizeof_field(struct sk_buff, cb),
 					      NULL);
 	skbuff_fclone_cache = kmem_cache_create("skbuff_fclone_cache",
 						sizeof(struct sk_buff_fclones),
 						0,
-						SLAB_HWCACHE_ALIGN|SLAB_PANIC,
+						SLAB_HWCACHE_ALIGN|SLAB_PANIC|
+						SLAB_NO_FREE_INIT,
 						NULL);
 	skb_extensions_init();
 }
Signature · 5 lines
2.17.1


-- 
Kees Cook
#33 lore raw

Re: [PATCH 5/4] mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches

Quoted 4 lines · Kees Cook

In order to improve the init_on_free performance, some frequently freed caches with less sensitive contents can be excluded from the init_on_free behavior.

Did you see any notable performance improvement with this patch? A similar one gave me only 1-2% on the parallel Linux build.

Quoted 147 lines

This patch is modified from Brad Spengler/PaX Team's code in the last public patch of grsecurity/PaX based on my understanding of the code. Changes or omissions from the original code are mine and don't reflect the original grsecurity/PaX code.

---
 fs/buffer.c          | 2 +-
 fs/dcache.c          | 3 ++-
 include/linux/slab.h | 3 +++
 kernel/fork.c        | 6 ++++--
 mm/rmap.c            | 5 +++--
 mm/slab.h            | 5 +++--
 net/core/skbuff.c    | 6 ++++--
 7 files changed, 20 insertions(+), 10 deletions(-)
fs/buffer.c +1 -1
index 0faa41fb4c88..04a85bd4cf2e 100644
@@ -3457,7 +3457,7 @@ void __init buffer_init(void)
        bh_cachep = kmem_cache_create("buffer_head",
                        sizeof(struct buffer_head), 0,
                                (SLAB_RECLAIM_ACCOUNT|SLAB_PANIC|
-                               SLAB_MEM_SPREAD),
+                               SLAB_MEM_SPREAD|SLAB_NO_FREE_INIT),
                                NULL);
 
        /*
fs/dcache.c +2 -1
index 8136bda27a1f..323b039accba 100644
@@ -3139,7 +3139,8 @@ void __init vfs_caches_init_early(void)
 void __init vfs_caches_init(void)
 {
        names_cachep = kmem_cache_create_usercopy("names_cache", PATH_MAX, 0,
-                       SLAB_HWCACHE_ALIGN|SLAB_PANIC, 0, PATH_MAX, NULL);
+                       SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_NO_FREE_INIT, 0,
+                       PATH_MAX, NULL);
 
        dcache_init();
        inode_init();
include/linux/slab.h +3 -0
index 9449b19c5f10..7eba9ad8830d 100644
@@ -92,6 +92,9 @@
 /* Avoid kmemleak tracing */
 #define SLAB_NOLEAKTRACE       ((slab_flags_t __force)0x00800000U)
 
+/* Exclude slab from zero-on-free when init_on_free is enabled */
+#define SLAB_NO_FREE_INIT      ((slab_flags_t __force)0x01000000U)
+
 /* Fault injection mark */
 #ifdef CONFIG_FAILSLAB
 # define SLAB_FAILSLAB         ((slab_flags_t __force)0x02000000U)
kernel/fork.c +4 -2
index b4cba953040a..9868585f5520 100644
@@ -2550,11 +2550,13 @@ void __init proc_caches_init(void)
 
        mm_cachep = kmem_cache_create_usercopy("mm_struct",
                        mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
-                       SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
+                       SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT|
+                       SLAB_NO_FREE_INIT,
                        offsetof(struct mm_struct, saved_auxv),
                        sizeof_field(struct mm_struct, saved_auxv),
                        NULL);
-       vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT);
+       vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT|
+                                                   SLAB_NO_FREE_INIT);
        mmap_init();
        nsproxy_cache_init();
 }
mm/rmap.c +3 -2
index e5dfe2ae6b0d..b7b8013eeb0a 100644
@@ -432,10 +432,11 @@ static void anon_vma_ctor(void *data)
 void __init anon_vma_init(void)
 {
        anon_vma_cachep = kmem_cache_create("anon_vma", sizeof(struct anon_vma),
-                       0, SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT,
+                       0, SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT|
+                       SLAB_NO_FREE_INIT,
                        anon_vma_ctor);
        anon_vma_chain_cachep = KMEM_CACHE(anon_vma_chain,
-                       SLAB_PANIC|SLAB_ACCOUNT);
+                       SLAB_PANIC|SLAB_ACCOUNT|SLAB_NO_FREE_INIT);
 }
 
 /*
mm/slab.h +3 -2
index 24ae887359b8..f95b4f03c57b 100644
@@ -129,7 +129,8 @@ static inline slab_flags_t kmem_cache_flags(unsigned int object_size,
 /* Legal flag mask for kmem_cache_create(), for various configurations */
 #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
                         SLAB_CACHE_DMA32 | SLAB_PANIC | \
-                        SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS )
+                        SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
+                        SLAB_NO_FREE_INIT)
 
 #if defined(CONFIG_DEBUG_SLAB)
 #define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER)
@@ -535,7 +536,7 @@ static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c)
 static inline bool slab_want_init_on_free(struct kmem_cache *c)
 {
        if (static_branch_unlikely(&init_on_free))
-               return !(c->ctor);
+               return !(c->ctor) && ((c->flags & SLAB_NO_FREE_INIT) == 0);
        else
                return false;
 }
net/core/skbuff.c +4 -3
index e89be6282693..b65902d2c042 100644
@@ -3981,14 +3981,16 @@ void __init skb_init(void)
        skbuff_head_cache = kmem_cache_create_usercopy("skbuff_head_cache",
                                              sizeof(struct sk_buff),
                                              0,
-                                             SLAB_HWCACHE_ALIGN|SLAB_PANIC,
+                                             SLAB_HWCACHE_ALIGN|SLAB_PANIC|
+                                             SLAB_NO_FREE_INIT,
                                              offsetof(struct sk_buff, cb),
                                              sizeof_field(struct sk_buff, cb),
                                              NULL);
        skbuff_fclone_cache = kmem_cache_create("skbuff_fclone_cache",
                                                sizeof(struct sk_buff_fclones),
                                                0,
-                                               SLAB_HWCACHE_ALIGN|SLAB_PANIC,
+                                               SLAB_HWCACHE_ALIGN|SLAB_PANIC|
+                                               SLAB_NO_FREE_INIT,
                                                NULL);
        skb_extensions_init();
 }
--

2.17.1

--
Kees Cook

Signature · 10 lines
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
#34 lore raw

Re: [PATCH 5/4] mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches

Quoted 7 lines · Alexander Potapenko
Quoted 4 lines · Kees Cook

In order to improve the init_on_free performance, some frequently freed caches with less sensitive contents can be excluded from the init_on_free behavior.

Did you see any notable performance improvement with this patch? A similar one gave me only 1-2% on the parallel Linux build.

Yup, that's in the other thread. I saw similar. But 1-2% on a 5% hit is a lot. ;)

Signature · 1 line
Kees Cook
#35 lore raw

Re: [PATCH 5/4] mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches

In reply to Kees Cook

Hi Kees,

Quoted 8 lines · Kees Cook

In order to improve the init_on_free performance, some frequently freed caches with less sensitive contents can be excluded from the init_on_free behavior.

This patch is modified from Brad Spengler/PaX Team's code in the last public patch of grsecurity/PaX based on my understanding of the code. Changes or omissions from the original code are mine and don't reflect the original grsecurity/PaX code.

you might want to give secunet credit for this one, as can be seen here:

https://github.com/minipli/linux-grsec/commit/309d494e7a3f6533ca68fc8b3bd89fa76fd2c2df

However, please keep the "Changes or omissions from the original code..." part as your version slightly differs.

Thanks,
Mathias

Quoted 143 lines
---
 fs/buffer.c          | 2 +-
 fs/dcache.c          | 3 ++-
 include/linux/slab.h | 3 +++
 kernel/fork.c        | 6 ++++--
 mm/rmap.c            | 5 +++--
 mm/slab.h            | 5 +++--
 net/core/skbuff.c    | 6 ++++--
 7 files changed, 20 insertions(+), 10 deletions(-)
fs/buffer.c +1 -1
index 0faa41fb4c88..04a85bd4cf2e 100644
@@ -3457,7 +3457,7 @@ void __init buffer_init(void)
        bh_cachep = kmem_cache_create("buffer_head",
                        sizeof(struct buffer_head), 0,
                                (SLAB_RECLAIM_ACCOUNT|SLAB_PANIC|
-                               SLAB_MEM_SPREAD),
+                               SLAB_MEM_SPREAD|SLAB_NO_FREE_INIT),
                                NULL);
 
        /*
fs/dcache.c +2 -1
index 8136bda27a1f..323b039accba 100644
@@ -3139,7 +3139,8 @@ void __init vfs_caches_init_early(void)
 void __init vfs_caches_init(void)
 {
        names_cachep = kmem_cache_create_usercopy("names_cache", PATH_MAX, 0,
-                       SLAB_HWCACHE_ALIGN|SLAB_PANIC, 0, PATH_MAX, NULL);
+                       SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_NO_FREE_INIT, 0,
+                       PATH_MAX, NULL);
 
        dcache_init();
        inode_init();
include/linux/slab.h +3 -0
index 9449b19c5f10..7eba9ad8830d 100644
@@ -92,6 +92,9 @@
 /* Avoid kmemleak tracing */
 #define SLAB_NOLEAKTRACE       ((slab_flags_t __force)0x00800000U)
 
+/* Exclude slab from zero-on-free when init_on_free is enabled */
+#define SLAB_NO_FREE_INIT      ((slab_flags_t __force)0x01000000U)
+
 /* Fault injection mark */
 #ifdef CONFIG_FAILSLAB
 # define SLAB_FAILSLAB         ((slab_flags_t __force)0x02000000U)
kernel/fork.c +4 -2
index b4cba953040a..9868585f5520 100644
@@ -2550,11 +2550,13 @@ void __init proc_caches_init(void)
 
        mm_cachep = kmem_cache_create_usercopy("mm_struct",
                        mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
-                       SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
+                       SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT|
+                       SLAB_NO_FREE_INIT,
                        offsetof(struct mm_struct, saved_auxv),
                        sizeof_field(struct mm_struct, saved_auxv),
                        NULL);
-       vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT);
+       vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT|
+                                                   SLAB_NO_FREE_INIT);
        mmap_init();
        nsproxy_cache_init();
 }
mm/rmap.c +3 -2
index e5dfe2ae6b0d..b7b8013eeb0a 100644
@@ -432,10 +432,11 @@ static void anon_vma_ctor(void *data)
 void __init anon_vma_init(void)
 {
        anon_vma_cachep = kmem_cache_create("anon_vma", sizeof(struct anon_vma),
-                       0, SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT,
+                       0, SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT|
+                       SLAB_NO_FREE_INIT,
                        anon_vma_ctor);
        anon_vma_chain_cachep = KMEM_CACHE(anon_vma_chain,
-                       SLAB_PANIC|SLAB_ACCOUNT);
+                       SLAB_PANIC|SLAB_ACCOUNT|SLAB_NO_FREE_INIT);
 }
 
 /*
mm/slab.h +3 -2
index 24ae887359b8..f95b4f03c57b 100644
@@ -129,7 +129,8 @@ static inline slab_flags_t kmem_cache_flags(unsigned int object_size,
 /* Legal flag mask for kmem_cache_create(), for various configurations */
 #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
                         SLAB_CACHE_DMA32 | SLAB_PANIC | \
-                        SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS )
+                        SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
+                        SLAB_NO_FREE_INIT)
 
 #if defined(CONFIG_DEBUG_SLAB)
 #define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER)
@@ -535,7 +536,7 @@ static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c)
 static inline bool slab_want_init_on_free(struct kmem_cache *c)
 {
        if (static_branch_unlikely(&init_on_free))
-               return !(c->ctor);
+               return !(c->ctor) && ((c->flags & SLAB_NO_FREE_INIT) == 0);
        else
                return false;
 }
net/core/skbuff.c +4 -3
index e89be6282693..b65902d2c042 100644
@@ -3981,14 +3981,16 @@ void __init skb_init(void)
        skbuff_head_cache = kmem_cache_create_usercopy("skbuff_head_cache",
                                              sizeof(struct sk_buff),
                                              0,
-                                             SLAB_HWCACHE_ALIGN|SLAB_PANIC,
+                                             SLAB_HWCACHE_ALIGN|SLAB_PANIC|
+                                             SLAB_NO_FREE_INIT,
                                              offsetof(struct sk_buff, cb),
                                              sizeof_field(struct sk_buff, cb),
                                              NULL);
        skbuff_fclone_cache = kmem_cache_create("skbuff_fclone_cache",
                                                sizeof(struct sk_buff_fclones),
                                                0,
-                                               SLAB_HWCACHE_ALIGN|SLAB_PANIC,
+                                               SLAB_HWCACHE_ALIGN|SLAB_PANIC|
+                                               SLAB_NO_FREE_INIT,
                                                NULL);
        skb_extensions_init();
 }
--

2.17.1

--
Kees Cook

#36 lore raw

Re: [PATCH 5/4] mm: Introduce SLAB_NO_FREE_INIT and mark excluded caches

Quoted 12 lines · Mathias Krause

Hi Kees,

Quoted 8 lines · Kees Cook

In order to improve the init_on_free performance, some frequently freed caches with less sensitive contents can be excluded from the init_on_free behavior.

This patch is modified from Brad Spengler/PaX Team's code in the last public patch of grsecurity/PaX based on my understanding of the code. Changes or omissions from the original code are mine and don't reflect the original grsecurity/PaX code.

Quoted 6 lines

you might want to give secunet credit for this one, as can be seen here:

https://github.com/minipli/linux-grsec/commit/309d494e7a3f6533ca68fc8b3bd89fa76fd2c2df

However, please keep the "Changes or omissions from the original code..." part as your version slightly differs.

Ah-ha! Thanks for finding the specific commit; I'll adjust attribution. Are you able to describe how you chose the various excluded kmem caches? (I assume it wasn't just "highest numbers in the stats reporting".) And why run the ctor after wipe? Doesn't that means you're just running the ctor again at the next allocation time?

Thanks!

-Kees

Quoted 147 lines

Thanks,
Mathias

Quoted 143 lines
---
 fs/buffer.c          | 2 +-
 fs/dcache.c          | 3 ++-
 include/linux/slab.h | 3 +++
 kernel/fork.c        | 6 ++++--
 mm/rmap.c            | 5 +++--
 mm/slab.h            | 5 +++--
 net/core/skbuff.c    | 6 ++++--
 7 files changed, 20 insertions(+), 10 deletions(-)
fs/buffer.c +1 -1
index 0faa41fb4c88..04a85bd4cf2e 100644
@@ -3457,7 +3457,7 @@ void __init buffer_init(void)
        bh_cachep = kmem_cache_create("buffer_head",
                        sizeof(struct buffer_head), 0,
                                (SLAB_RECLAIM_ACCOUNT|SLAB_PANIC|
-                               SLAB_MEM_SPREAD),
+                               SLAB_MEM_SPREAD|SLAB_NO_FREE_INIT),
                                NULL);
 
        /*
fs/dcache.c +2 -1
index 8136bda27a1f..323b039accba 100644
@@ -3139,7 +3139,8 @@ void __init vfs_caches_init_early(void)
 void __init vfs_caches_init(void)
 {
        names_cachep = kmem_cache_create_usercopy("names_cache", PATH_MAX, 0,
-                       SLAB_HWCACHE_ALIGN|SLAB_PANIC, 0, PATH_MAX, NULL);
+                       SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_NO_FREE_INIT, 0,
+                       PATH_MAX, NULL);
 
        dcache_init();
        inode_init();
include/linux/slab.h +3 -0
index 9449b19c5f10..7eba9ad8830d 100644
@@ -92,6 +92,9 @@
 /* Avoid kmemleak tracing */
 #define SLAB_NOLEAKTRACE       ((slab_flags_t __force)0x00800000U)
 
+/* Exclude slab from zero-on-free when init_on_free is enabled */
+#define SLAB_NO_FREE_INIT      ((slab_flags_t __force)0x01000000U)
+
 /* Fault injection mark */
 #ifdef CONFIG_FAILSLAB
 # define SLAB_FAILSLAB         ((slab_flags_t __force)0x02000000U)
kernel/fork.c +4 -2
index b4cba953040a..9868585f5520 100644
@@ -2550,11 +2550,13 @@ void __init proc_caches_init(void)
 
        mm_cachep = kmem_cache_create_usercopy("mm_struct",
                        mm_size, ARCH_MIN_MMSTRUCT_ALIGN,
-                       SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT,
+                       SLAB_HWCACHE_ALIGN|SLAB_PANIC|SLAB_ACCOUNT|
+                       SLAB_NO_FREE_INIT,
                        offsetof(struct mm_struct, saved_auxv),
                        sizeof_field(struct mm_struct, saved_auxv),
                        NULL);
-       vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT);
+       vm_area_cachep = KMEM_CACHE(vm_area_struct, SLAB_PANIC|SLAB_ACCOUNT|
+                                                   SLAB_NO_FREE_INIT);
        mmap_init();
        nsproxy_cache_init();
 }
mm/rmap.c +3 -2
index e5dfe2ae6b0d..b7b8013eeb0a 100644
@@ -432,10 +432,11 @@ static void anon_vma_ctor(void *data)
 void __init anon_vma_init(void)
 {
        anon_vma_cachep = kmem_cache_create("anon_vma", sizeof(struct anon_vma),
-                       0, SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT,
+                       0, SLAB_TYPESAFE_BY_RCU|SLAB_PANIC|SLAB_ACCOUNT|
+                       SLAB_NO_FREE_INIT,
                        anon_vma_ctor);
        anon_vma_chain_cachep = KMEM_CACHE(anon_vma_chain,
-                       SLAB_PANIC|SLAB_ACCOUNT);
+                       SLAB_PANIC|SLAB_ACCOUNT|SLAB_NO_FREE_INIT);
 }
 
 /*
mm/slab.h +3 -2
index 24ae887359b8..f95b4f03c57b 100644
@@ -129,7 +129,8 @@ static inline slab_flags_t kmem_cache_flags(unsigned int object_size,
 /* Legal flag mask for kmem_cache_create(), for various configurations */
 #define SLAB_CORE_FLAGS (SLAB_HWCACHE_ALIGN | SLAB_CACHE_DMA | \
                         SLAB_CACHE_DMA32 | SLAB_PANIC | \
-                        SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS )
+                        SLAB_TYPESAFE_BY_RCU | SLAB_DEBUG_OBJECTS | \
+                        SLAB_NO_FREE_INIT)
 
 #if defined(CONFIG_DEBUG_SLAB)
 #define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER)
@@ -535,7 +536,7 @@ static inline bool slab_want_init_on_alloc(gfp_t flags, struct kmem_cache *c)
 static inline bool slab_want_init_on_free(struct kmem_cache *c)
 {
        if (static_branch_unlikely(&init_on_free))
-               return !(c->ctor);
+               return !(c->ctor) && ((c->flags & SLAB_NO_FREE_INIT) == 0);
        else
                return false;
 }
net/core/skbuff.c +4 -3
index e89be6282693..b65902d2c042 100644
@@ -3981,14 +3981,16 @@ void __init skb_init(void)
        skbuff_head_cache = kmem_cache_create_usercopy("skbuff_head_cache",
                                              sizeof(struct sk_buff),
                                              0,
-                                             SLAB_HWCACHE_ALIGN|SLAB_PANIC,
+                                             SLAB_HWCACHE_ALIGN|SLAB_PANIC|
+                                             SLAB_NO_FREE_INIT,
                                              offsetof(struct sk_buff, cb),
                                              sizeof_field(struct sk_buff, cb),
                                              NULL);
        skbuff_fclone_cache = kmem_cache_create("skbuff_fclone_cache",
                                                sizeof(struct sk_buff_fclones),
                                                0,
-                                               SLAB_HWCACHE_ALIGN|SLAB_PANIC,
+                                               SLAB_HWCACHE_ALIGN|SLAB_PANIC|
+                                               SLAB_NO_FREE_INIT,
                                                NULL);
        skb_extensions_init();
 }
--

2.17.1

--
Kees Cook

Signature · 1 line
Kees Cook