[3.11 2/4] slob: Rework #ifdeffery in slab.h

cpu partial support can introduce level of indeterminism that is not wanted in certain context (like a realtime kernel). Make it configurable.

Signed-off-by Christoph Lameter <[email protected]>

The changelog is way too vague. Numbers? Anyone who would want to use this in real world scenarios, please speak up!

Quoted 2 lines · Pekka Enberg

The changelog is way too vague. Numbers? Anyone who would want to use this in real world scenarios, please speak up!

Steve?

I'm sure it would be better to actually do cpu_partial processing in small chunks to avoid latency spikes in latency sensitive applications

Quoted 2 lines · Clark Williams

I'm sure it would be better to actually do cpu_partial processing in small chunks to avoid latency spikes in latency sensitive applications

Sounds like a patch I'd be much more interested in applying...

Quoted 5 lines · Pekka Enberg

Quoted 2 lines · Clark Williams

I'm sure it would be better to actually do cpu_partial processing in small chunks to avoid latency spikes in latency sensitive applications

Sounds like a patch I'd be much more interested in applying...

Is this going to happen, otherwise we would really like a fix for RT.

Quoted 5 lines · Steven Rostedt

Quoted 4 lines · Pekka Enberg

Quoted 2 lines · Clark Williams

I'm sure it would be better to actually do cpu_partial processing in small chunks to avoid latency spikes in latency sensitive applications

Quoted 3 lines

Quoted 1 line

Sounds like a patch I'd be much more interested in applying...

Is this going to happen, otherwise we would really like a fix for RT.

Forget it. Just switch cpu_partial processing off. It will be in small chunks then.

What's your recommended method for switching cpu_partial processing off?

I'm not all that keen on repeatedly traversing /sys/kernel/slab looking for 'cpu_partial' entries, mainly because if you do it at boot time (i.e. from a startup script) you miss some of the entries.

Quoted 3 lines · Clark Williams

What's your recommended method for switching cpu_partial processing off?

Quoted 3 lines

I'm not all that keen on repeatedly traversing /sys/kernel/slab looking for 'cpu_partial' entries, mainly because if you do it at boot time (i.e. from a startup script) you miss some of the entries.

Merge the patch that makes a config option and compile it out of the kernel?

Quoted 6 lines · Clark Williams

What's your recommended method for switching cpu_partial processing off?

I'm not all that keen on repeatedly traversing /sys/kernel/slab looking for 'cpu_partial' entries, mainly because if you do it at boot time (i.e. from a startup script) you miss some of the entries.

Merge the patch that makes a config option and compile it out of the kernel?

When I run a stress test of the box (kernel compile along with hackbench), with that patch applied, the system hangs for long periods of time. I have no idea why, but the oom killer would trigger constantly.

Hi Steven,

Quoted 4 lines · Steven Rostedt

When I run a stress test of the box (kernel compile along with hackbench), with that patch applied, the system hangs for long periods of time. I have no idea why, but the oom killer would trigger constantly.

Are you using this patch or the modified version by Joonsoo that landed in Linus' tree?

I was using the patch that Christoph posted. It was also attached to the -rt kernel for 3.6. I could have backported the patch poorly too.

Quoted 2 lines · Steven Rostedt

I was using the patch that Christoph posted. It was also attached to the -rt kernel for 3.6. I could have backported the patch poorly too.

Could you try upstream?

Quoted 6 lines · Christoph Lameter

Quoted 2 lines · Steven Rostedt

I was using the patch that Christoph posted. It was also attached to the -rt kernel for 3.6. I could have backported the patch poorly too.

Could you try upstream?

Yeah, I'm running it now.

cpu partial support can introduce level of indeterminism that is not wanted in certain context (like a realtime kernel). Make it configurable.

Signed-off-by Christoph Lameter <[email protected]>

Index: linux/include/linux/slub_def.h
===================================================================

include/linux/slub_def.h +2 -0

@@ -73,7 +73,9 @@ struct kmem_cache {
 	int size;		/* The size of an object including meta data */
 	int object_size;	/* The size of an object without meta data */
 	int offset;		/* Free pointer offset. */
+#ifdef CONFIG_SLUB_CPU_PARTIAL
 	int cpu_partial;	/* Number of per cpu partial objects to keep around */
+#endif
 	struct kmem_cache_order_objects oo;
 
 	/* Allocation and freeing of slabs */

include/linux/slub_def.h +9 -0

+static inline int kmem_cache_cpu_partial(struct kmem_cache *s)
+{
+#ifdef CONFIG_SLUB_CPU_PARTIAL
+	return s->cpu_partial;
+#else
+	return 0;
+#endif
+}
+
 void *kmem_cache_alloc(struct kmem_cache *, gfp_t);
 void *__kmalloc(size_t size, gfp_t flags);
 

Index: linux/mm/slub.c
===================================================================

mm/slub.c +14 -3

@@ -1573,7 +1573,8 @@ static void *get_partial_node(struct kme
 			put_cpu_partial(s, page, 0);
 			stat(s, CPU_PARTIAL_NODE);
 		}
-		if (kmem_cache_debug(s) || available > s->cpu_partial / 2)
+		if (kmem_cache_debug(s) ||
+			       available > kmem_cache_cpu_partial(s) / 2)
 			break;
 
 	}

@@ -1884,6 +1885,7 @@ redo:
 static void unfreeze_partials(struct kmem_cache *s,
 		struct kmem_cache_cpu *c)
 {
+#ifdef CONFIG_SLUB_CPU_PARTIAL
 	struct kmem_cache_node *n = NULL, *n2 = NULL;
 	struct page *page, *discard_page = NULL;
 

@@ -1938,6 +1940,7 @@ static void unfreeze_partials(struct kme
 		discard_slab(s, page);
 		stat(s, FREE_SLAB);
 	}
+#endif
 }
 
 /*

@@ -1951,6 +1954,7 @@ static void unfreeze_partials(struct kme
  */
 static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain)
 {
+#ifdef CONFIG_SLUB_CPU_PARTIAL
 	struct page *oldpage;
 	int pages;
 	int pobjects;

@@ -1987,6 +1991,7 @@ static void put_cpu_partial(struct kmem_
 		page->next = oldpage;
 
 	} while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page) != oldpage);
+#endif
 }
 
 static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c)

@@ -2495,6 +2500,7 @@ static void __slab_free(struct kmem_cach
 		new.inuse--;
 		if ((!new.inuse || !prior) && !was_frozen) {
 
+#ifdef CONFIG_SLUB_CPU_PARTIAL
 			if (!kmem_cache_debug(s) && !prior)
 
 				/*

@@ -2503,7 +2509,9 @@ static void __slab_free(struct kmem_cach
 				 */
 				new.frozen = 1;
 
-			else { /* Needs to be taken off a list */
+			else
+#endif
+		       		{ /* Needs to be taken off a list */
 
 	                        n = get_node(s, page_to_nid(page));
 				/*

@@ -2525,6 +2533,7 @@ static void __slab_free(struct kmem_cach
 		"__slab_free"));
 
 	if (likely(!n)) {
+#ifdef CONFIG_SLUB_CPU_PARTIAL
 
 		/*
 		 * If we just froze the page then put it onto the

@@ -2534,6 +2543,7 @@ static void __slab_free(struct kmem_cach
 			put_cpu_partial(s, page, 1);
 			stat(s, CPU_PARTIAL_FREE);
 		}
+#endif
 		/*
 		 * The list lock was not taken therefore no list
 		 * activity can be necessary.

@@ -3041,7 +3051,7 @@ static int kmem_cache_open(struct kmem_c
 	 * list to avoid pounding the page allocator excessively.
 	 */
 	set_min_partial(s, ilog2(s->size) / 2);
-
+#ifdef CONFIG_SLUB_CPU_PARTIAL
 	/*
 	 * cpu_partial determined the maximum number of objects kept in the
 	 * per cpu partial lists of a processor.

@@ -3069,6 +3079,7 @@ static int kmem_cache_open(struct kmem_c
 		s->cpu_partial = 13;
 	else
 		s->cpu_partial = 30;
+#endif
 
 #ifdef CONFIG_NUMA
 	s->remote_node_defrag_ratio = 1000;

mm/slub.c +5 -2

 static ssize_t min_partial_store(struct kmem_cache *s, const char *buf,

@@ -4444,7 +4455,7 @@ SLAB_ATTR(min_partial);
 
 static ssize_t cpu_partial_show(struct kmem_cache *s, char *buf)
 {
-	return sprintf(buf, "%u\n", s->cpu_partial);
+	return sprintf(buf, "%u\n", kmem_cache_cpu_partial(s));
 }
 
 static ssize_t cpu_partial_store(struct kmem_cache *s, const char *buf,

@@ -4458,10 +4469,13 @@ static ssize_t cpu_partial_store(struct
 		return err;
 	if (objects && kmem_cache_debug(s))
 		return -EINVAL;
-
+#ifdef CONFIG_SLUB_CPU_PARTIAL
 	s->cpu_partial = objects;
 	flush_all(s);
 	return length;
+#else
+	return -ENOSYS;
+#endif
 }
 SLAB_ATTR(cpu_partial);
 

Index: linux/init/Kconfig
===================================================================

init/Kconfig +11 -1

@@ -1559,6 +1559,17 @@ config SLOB
 
 endchoice
 
+config SLUB_CPU_PARTIAL
+	default y
+	depends on SLUB
+	bool "SLUB per cpu partial cache"
+	help
+	  Per cpu partial caches accellerate objects allocation and freeing
+	  that is local to a processor at the price of more indeterminism
+	  in the latency of the free. On overflow these caches will be cleared
+	  which requires the taking of locks that may cause latency spikes.
+	  Typically one would choose no for a realtime system.
+
 config MMAP_ALLOW_UNINITIALIZED
 	bool "Allow mmapped anonymous memory to be uninitialized"
 	depends on EXPERT && !MMU
 
--

To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

How about maintaining cpu_partial when !CONFIG_SLUB_CPU_PARTIAL?
It makes code less churn and doesn't have much overhead.
At bottom, my implementation with cpu_partial is attached. It uses less '#ifdef'.

Quoted 3 lines · Joonsoo Kim

How about maintaining cpu_partial when !CONFIG_SLUB_CPU_PARTIAL?
It makes code less churn and doesn't have much overhead.
At bottom, my implementation with cpu_partial is attached. It uses less '#ifdef'.

Looks good. I am fine with it.

Acked-by Christoph Lameter <[email protected]>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 9 lines · Christoph Lameter

Quoted 3 lines · Joonsoo Kim

How about maintaining cpu_partial when !CONFIG_SLUB_CPU_PARTIAL?
It makes code less churn and doesn't have much overhead.
At bottom, my implementation with cpu_partial is attached. It uses less '#ifdef'.

Looks good. I am fine with it.

Acked-by Christoph Lameter <[email protected]>

Thanks!

Hello, Pekka.
I attach a right formatted patch with acked by Christoph and signed off by me.

It is based on v3.10-rc6 and top of a patch
"slub: do not put a slab to cpu partial list when cpu_partial is 0".

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

-----------------8<-----------------------------------------------

Quoted 1 line

From a3257adcff89fd89a7ecb26c1247eec511302807 Mon Sep 17 00:00:00 2001

From: Joonsoo Kim <[email protected]>
Date: Wed, 19 Jun 2013 14:05:52 +0900
Subject: [PATCH] slub: Make cpu partial slab support configurable

cpu partial support can introduce level of indeterminism that is not wanted in certain context (like a realtime kernel). Make it configurable.

This patch is based on Christoph Lameter's "slub: Make cpu partial slab support configurable V2".

Acked-by Christoph Lameter <[email protected]>

Signed-off-by Joonsoo Kim <[email protected]>

init/Kconfig +0 -0

index 2d9b831..a7ec1ec 100644
@@ -1559,6 +1559,17 @@ config SLOB
 

endchoice

init/Kconfig +11 -0

+config SLUB_CPU_PARTIAL
+	default y
+	depends on SLUB
+	bool "SLUB per cpu partial cache"
+	help
+	  Per cpu partial caches accellerate objects allocation and freeing
+	  that is local to a processor at the price of more indeterminism
+	  in the latency of the free. On overflow these caches will be cleared
+	  which requires the taking of locks that may cause latency spikes.
+	  Typically one would choose no for a realtime system.
+

config MMAP_ALLOW_UNINITIALIZED

	bool "Allow mmapped anonymous memory to be uninitialized"
	depends on EXPERT && !MMU

mm/slub.c +18 -4

index 7033b4f..a670d22 100644
@@ -123,6 +123,15 @@ static inline int kmem_cache_debug(struct kmem_cache *s)
 #endif
 }
 
+static inline bool kmem_cache_has_cpu_partial(struct kmem_cache *s)
+{
+#ifdef CONFIG_SLUB_CPU_PARTIAL
+	return !kmem_cache_debug(s);
+#else
+	return false;
+#endif
+}
+
 /*
 * Issues still to be resolved:
 *

@@ -1573,7 +1582,8 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n,
 			put_cpu_partial(s, page, 0);
 			stat(s, CPU_PARTIAL_NODE);
 		}
-		if (kmem_cache_debug(s) || available > s->cpu_partial / 2)
+		if (!kmem_cache_has_cpu_partial(s)
+			|| available > s->cpu_partial / 2)
 			break;
 
 	}

@@ -1884,6 +1894,7 @@ redo:
 static void unfreeze_partials(struct kmem_cache *s,
 		struct kmem_cache_cpu *c)
 {
+#ifdef CONFIG_SLUB_CPU_PARTIAL
 	struct kmem_cache_node *n = NULL, *n2 = NULL;
 	struct page *page, *discard_page = NULL;
 

@@ -1938,6 +1949,7 @@ static void unfreeze_partials(struct kmem_cache *s,
 		discard_slab(s, page);
 		stat(s, FREE_SLAB);
 	}
+#endif
 }
 
 /*

@@ -1951,6 +1963,7 @@ static void unfreeze_partials(struct kmem_cache *s,
 */
 static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain)
 {
+#ifdef CONFIG_SLUB_CPU_PARTIAL
 	struct page *oldpage;
 	int pages;
 	int pobjects;

@@ -1990,6 +2003,7 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain)
 		page->next = oldpage;
 
 	} while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page) != oldpage);
+#endif
 }
 
 static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c)

@@ -2498,7 +2512,7 @@ static void __slab_free(struct kmem_cache *s, struct page *page,
 		new.inuse--;
 		if ((!new.inuse || !prior) && !was_frozen) {
 
-			if (!kmem_cache_debug(s) && !prior)
+			if (kmem_cache_has_cpu_partial(s) && !prior)
 
 				/*
 				 * Slab was on no list before and will be partially empty

@@ -3062,7 +3076,7 @@ static int kmem_cache_open(struct kmem_cache *s, unsigned long flags)
 	 *    per node list when we run out of per cpu objects. We only fetch 50%
 	 *    to keep some capacity around for frees.
 	 */
-	if (kmem_cache_debug(s))
+	if (!kmem_cache_has_cpu_partial(s))
 		s->cpu_partial = 0;
 	else if (s->size >= PAGE_SIZE)
 		s->cpu_partial = 2;

@@ -4459,7 +4473,7 @@ static ssize_t cpu_partial_store(struct kmem_cache *s, const char *buf,
 	err = strict_strtoul(buf, 10, &objects);
 	if (err)
 		return err;
-	if (objects && kmem_cache_debug(s))
+	if (objects && !kmem_cache_has_cpu_partial(s))
 		return -EINVAL;
 
 	s->cpu_partial = objects;

Signature · 7 lines

1.7.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 9 lines · Christoph Lameter

Quoted 3 lines · Joonsoo Kim

How about maintaining cpu_partial when !CONFIG_SLUB_CPU_PARTIAL?
It makes code less churn and doesn't have much overhead.
At bottom, my implementation with cpu_partial is attached. It uses less '#ifdef'.

Looks good. I am fine with it.

Acked-by Christoph Lameter <[email protected]>

Thanks!

Hello, Pekka.
I attach a right formatted patch with acked by Christoph and signed off by me.

It is based on v3.10-rc6 and top of a patch
"slub: do not put a slab to cpu partial list when cpu_partial is 0".

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

-----------------8<-----------------------------------------------

Quoted 1 line

From a3257adcff89fd89a7ecb26c1247eec511302807 Mon Sep 17 00:00:00 2001

From: Joonsoo Kim <[email protected]>
Date: Wed, 19 Jun 2013 14:05:52 +0900
Subject: [PATCH] slub: Make cpu partial slab support configurable

cpu partial support can introduce level of indeterminism that is not wanted in certain context (like a realtime kernel). Make it configurable.

This patch is based on Christoph Lameter's "slub: Make cpu partial slab support configurable V2".

Acked-by Christoph Lameter <[email protected]>

Signed-off-by Joonsoo Kim <[email protected]>

init/Kconfig +0 -0

index 2d9b831..a7ec1ec 100644
@@ -1559,6 +1559,17 @@ config SLOB
 

endchoice

init/Kconfig +11 -0

+config SLUB_CPU_PARTIAL
+	default y
+	depends on SLUB
+	bool "SLUB per cpu partial cache"
+	help
+	  Per cpu partial caches accellerate objects allocation and freeing
+	  that is local to a processor at the price of more indeterminism
+	  in the latency of the free. On overflow these caches will be cleared
+	  which requires the taking of locks that may cause latency spikes.
+	  Typically one would choose no for a realtime system.
+

config MMAP_ALLOW_UNINITIALIZED

	bool "Allow mmapped anonymous memory to be uninitialized"
	depends on EXPERT && !MMU

mm/slub.c +18 -4

index 7033b4f..a670d22 100644
@@ -123,6 +123,15 @@ static inline int kmem_cache_debug(struct kmem_cache *s)
 #endif
 }
 
+static inline bool kmem_cache_has_cpu_partial(struct kmem_cache *s)
+{
+#ifdef CONFIG_SLUB_CPU_PARTIAL
+	return !kmem_cache_debug(s);
+#else
+	return false;
+#endif
+}
+
 /*
 * Issues still to be resolved:
 *

@@ -1573,7 +1582,8 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n,
 			put_cpu_partial(s, page, 0);
 			stat(s, CPU_PARTIAL_NODE);
 		}
-		if (kmem_cache_debug(s) || available > s->cpu_partial / 2)
+		if (!kmem_cache_has_cpu_partial(s)
+			|| available > s->cpu_partial / 2)
 			break;
 
 	}

@@ -1884,6 +1894,7 @@ redo:
 static void unfreeze_partials(struct kmem_cache *s,
 		struct kmem_cache_cpu *c)
 {
+#ifdef CONFIG_SLUB_CPU_PARTIAL
 	struct kmem_cache_node *n = NULL, *n2 = NULL;
 	struct page *page, *discard_page = NULL;
 

@@ -1938,6 +1949,7 @@ static void unfreeze_partials(struct kmem_cache *s,
 		discard_slab(s, page);
 		stat(s, FREE_SLAB);
 	}
+#endif
 }
 
 /*

@@ -1951,6 +1963,7 @@ static void unfreeze_partials(struct kmem_cache *s,
 */
 static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain)
 {
+#ifdef CONFIG_SLUB_CPU_PARTIAL
 	struct page *oldpage;
 	int pages;
 	int pobjects;

@@ -1990,6 +2003,7 @@ static void put_cpu_partial(struct kmem_cache *s, struct page *page, int drain)
 		page->next = oldpage;
 
 	} while (this_cpu_cmpxchg(s->cpu_slab->partial, oldpage, page) != oldpage);
+#endif
 }
 
 static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c)

@@ -2498,7 +2512,7 @@ static void __slab_free(struct kmem_cache *s, struct page *page,
 		new.inuse--;
 		if ((!new.inuse || !prior) && !was_frozen) {
 
-			if (!kmem_cache_debug(s) && !prior)
+			if (kmem_cache_has_cpu_partial(s) && !prior)
 
 				/*
 				 * Slab was on no list before and will be partially empty

@@ -3062,7 +3076,7 @@ static int kmem_cache_open(struct kmem_cache *s, unsigned long flags)
 	 *    per node list when we run out of per cpu objects. We only fetch 50%
 	 *    to keep some capacity around for frees.
 	 */
-	if (kmem_cache_debug(s))
+	if (!kmem_cache_has_cpu_partial(s))
 		s->cpu_partial = 0;
 	else if (s->size >= PAGE_SIZE)
 		s->cpu_partial = 2;

@@ -4459,7 +4473,7 @@ static ssize_t cpu_partial_store(struct kmem_cache *s, const char *buf,
 	err = strict_strtoul(buf, 10, &objects);
 	if (err)
 		return err;
-	if (objects && kmem_cache_debug(s))
+	if (objects && !kmem_cache_has_cpu_partial(s))
 		return -EINVAL;
 
 	s->cpu_partial = objects;

Signature · 7 lines

1.7.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 9 lines · Joonsoo Kim

On Wed, Jun 19, 2013 at 02:29:29PM +0000, Christoph Lameter wrote:

-----------------8<-----------------------------------------------

Quoted 1 line

From a3257adcff89fd89a7ecb26c1247eec511302807 Mon Sep 17 00:00:00 2001

From: Joonsoo Kim <[email protected]>
Date: Wed, 19 Jun 2013 14:05:52 +0900
Subject: [PATCH] slub: Make cpu partial slab support configurable

Quoted 6 lines

cpu partial support can introduce level of indeterminism that is not wanted in certain context (like a realtime kernel). Make it configurable.

This patch is based on Christoph Lameter's "slub: Make cpu partial slab support configurable V2".

As you know, actually cpu_partial is the maximum number of objects kept in the per cpu slab and cpu partial lists of a processor instead of just the maximum number of objects kept in cpu partial lists of a processor. The allocation will always fallback to slow path if not config SLUB_CPU_PARTIAL, whether it will lead to more latency?

Regards,
Wanpeng Li

Hello, Pekka.
I attach a right formatted patch with acked by Christoph and signed off by me.

It is based on v3.10-rc6 and top of a patch
"slub: do not put a slab to cpu partial list when cpu_partial is 0".

Quoted 6 lines · Joonsoo Kim

Hello, Pekka.
I attach a right formatted patch with acked by Christoph and signed off by me.

It is based on v3.10-rc6 and top of a patch
"slub: do not put a slab to cpu partial list when cpu_partial is 0".

One more change is needed in __slab_free(), so I attach v2.

------------------8<------------------------------------- From 22fef26a9775745a041ca8820971d475714ee351 Mon Sep 17 00:00:00 2001
From: Joonsoo Kim <[email protected]>
Date: Wed, 19 Jun 2013 14:05:52 +0900
Subject: [PATCH v2] slub: Make cpu partial slab support configurable

cpu partial support can introduce level of indeterminism that is not wanted in certain context (like a realtime kernel). Make it configurable.

This patch is based on Christoph Lameter's "slub: Make cpu partial slab support configurable V2".

Acked-by Christoph Lameter <[email protected]>

Signed-off-by Joonsoo Kim <[email protected]>

The kmalloc_node functions of all slab allcoators are similar now so lets move them into slab.h. This requires some function naming changes in slob.

Signed-off-by Christoph Lameter <[email protected]>

I'm seeing this after "make defconfig" on x86-64:

CC mm/slub.o
mm/slub.c:2445:7: error: conflicting types for i? 1/2 kmem_cache_alloc_node_tracei? 1/2 include/linux/slab.h:311:14: note: previous declaration of
i? 1/2 kmem_cache_alloc_node_tracei? 1/2 was here
mm/slub.c:2455:1: error: conflicting types for i? 1/2 kmem_cache_alloc_node_tracei? 1/2 include/linux/slab.h:311:14: note: previous declaration of
i? 1/2 kmem_cache_alloc_node_tracei? 1/2 was here
make[1]: *** [mm/slub.o] Error 1
make: *** [mm/slub.o] Error 2

Quoted 11 lines · Pekka Enberg

I'm seeing this after "make defconfig" on x86-64:

CC mm/slub.o
mm/slub.c:2445:7: error: conflicting types for �kmem_cache_alloc_node_trace� include/linux/slab.h:311:14: note: previous declaration of �kmem_cache_alloc_node_trace� was here
mm/slub.c:2455:1: error: conflicting types for �kmem_cache_alloc_node_trace� include/linux/slab.h:311:14: note: previous declaration of �kmem_cache_alloc_node_trace� was here
make[1]: *** [mm/slub.o] Error 1
make: *** [mm/slub.o] Error 2

Gosh I dropped the size_t parameter from these functions. CONFIG_TRACING needs these.

Subject: Fix kmem_cache_alloc*_trace parameters

The size parameter is needed.

Signed-off-by Christoph Lameter <[email protected]>

Can you please resend as a single patch against slab/next that compiles?

The kmalloc_node functions of all slab allcoators are similar now so lets move them into slab.h. This requires some function naming changes in slob.

Signed-off-by Christoph Lameter <[email protected]>

Index: linux/include/linux/slab.h
===================================================================

include/linux/slab.h +32 -0

@@ -289,6 +289,38 @@ static __always_inline int kmalloc_index
 }
 #endif /* !CONFIG_SLOB */
 
+void *__kmalloc(size_t size, gfp_t flags);
+void *kmem_cache_alloc(struct kmem_cache *, gfp_t flags);
+
+#ifdef CONFIG_NUMA
+void *__kmalloc_node(size_t size, gfp_t flags, int node);
+void *kmem_cache_alloc_node(struct kmem_cache *, gfp_t flags, int node);
+#else
+static __always_inline void *__kmalloc_node(size_t size, gfp_t flags, int node)
+{
+	return __kmalloc(size, flags);
+}
+
+static __always_inline void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node)
+{
+	return kmem_cache_alloc(s, flags);
+}
+#endif
+
+#ifdef CONFIG_TRACING
+extern void *kmem_cache_alloc_node_trace(struct kmem_cache *s,
+					   gfp_t gfpflags,
+					   int node);
+#else
+static __always_inline void *
+kmem_cache_alloc_node_trace(struct kmem_cache *s,
+			      gfp_t gfpflags,
+			      int node)
+{
+	return kmem_cache_alloc_node(s, gfpflags, node);
+}
+#endif
+
 #ifdef CONFIG_SLAB
 #include <linux/slab_def.h>
 #endif

include/linux/slab.h +12 -30

+		int i = kmalloc_index(size);
+
+		if (!i)
+			return ZERO_SIZE_PTR;
+
+		return kmem_cache_alloc_node_trace(kmalloc_caches[i],
+			       			flags, node);
+	}
+#endif
+	return __kmalloc_node(size, flags, node);
+}
+
 /*
  * Setting ARCH_SLAB_MINALIGN in arch headers allows a different alignment.
  * Intended for arches that get misalignment faults even for 64 bit integer

@@ -441,36 +490,6 @@ static inline void *kcalloc(size_t n, si
 	return kmalloc_array(n, size, flags | __GFP_ZERO);
 }
 
-#if !defined(CONFIG_NUMA) && !defined(CONFIG_SLOB)
-/**
- * kmalloc_node - allocate memory from a specific node
- * @size: how many bytes of memory are required.
- * @flags: the type of memory to allocate (see kcalloc).
- * @node: node to allocate from.
- *
- * kmalloc() for non-local nodes, used to allocate from a specific node
- * if available. Equivalent to kmalloc() in the non-NUMA single-node
- * case.
- */
-static inline void *kmalloc_node(size_t size, gfp_t flags, int node)
-{
-	return kmalloc(size, flags);
-}
-
-static inline void *__kmalloc_node(size_t size, gfp_t flags, int node)
-{
-	return __kmalloc(size, flags);
-}
-
-void *kmem_cache_alloc(struct kmem_cache *, gfp_t);
-
-static inline void *kmem_cache_alloc_node(struct kmem_cache *cachep,
-					gfp_t flags, int node)
-{
-	return kmem_cache_alloc(cachep, flags);
-}
-#endif /* !CONFIG_NUMA && !CONFIG_SLOB */
-
 /*
  * kmalloc_track_caller is a special version of kmalloc that records the
  * calling function of the routine calling it for slab leak tracking instead

Index: linux/include/linux/slub_def.h
===================================================================

include/linux/slub_def.h +0 -37

@@ -115,9 +115,6 @@ static inline int kmem_cache_cpu_partial
 #endif
 }
 
-void *kmem_cache_alloc(struct kmem_cache *, gfp_t);
-void *__kmalloc(size_t size, gfp_t flags);
-
 static __always_inline void *
 kmalloc_order(size_t size, gfp_t flags, unsigned int order)
 {

@@ -185,38 +182,4 @@ static __always_inline void *kmalloc(siz
 	return __kmalloc(size, flags);
 }
 
-#ifdef CONFIG_NUMA
-void *__kmalloc_node(size_t size, gfp_t flags, int node);
-void *kmem_cache_alloc_node(struct kmem_cache *, gfp_t flags, int node);
-
-#ifdef CONFIG_TRACING
-extern void *kmem_cache_alloc_node_trace(struct kmem_cache *s,
-					   gfp_t gfpflags,
-					   int node, size_t size);
-#else
-static __always_inline void *
-kmem_cache_alloc_node_trace(struct kmem_cache *s,
-			      gfp_t gfpflags,
-			      int node, size_t size)
-{
-	return kmem_cache_alloc_node(s, gfpflags, node);
-}
-#endif
-
-static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node)
-{
-	if (__builtin_constant_p(size) &&
-		size <= KMALLOC_MAX_CACHE_SIZE && !(flags & GFP_DMA)) {
-		int index = kmalloc_index(size);
-
-		if (!index)
-			return ZERO_SIZE_PTR;
-
-		return kmem_cache_alloc_node_trace(kmalloc_caches[index],
-			       flags, node, size);
-	}
-	return __kmalloc_node(size, flags, node);
-}
-#endif
-
 #endif /* _LINUX_SLUB_DEF_H */

Index: linux/include/linux/slab_def.h
===================================================================

include/linux/slab_def.h +0 -52

@@ -102,9 +102,6 @@ struct kmem_cache {
 	 */
 };
 
-void *kmem_cache_alloc(struct kmem_cache *, gfp_t);
-void *__kmalloc(size_t size, gfp_t flags);
-
 #ifdef CONFIG_TRACING
 extern void *kmem_cache_alloc_trace(struct kmem_cache *, gfp_t, size_t);
 #else

@@ -145,53 +142,4 @@ static __always_inline void *kmalloc(siz
 	return __kmalloc(size, flags);
 }
 
-#ifdef CONFIG_NUMA
-extern void *__kmalloc_node(size_t size, gfp_t flags, int node);
-extern void *kmem_cache_alloc_node(struct kmem_cache *, gfp_t flags, int node);
-
-#ifdef CONFIG_TRACING
-extern void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
-					 gfp_t flags,
-					 int nodeid,
-					 size_t size);
-#else
-static __always_inline void *
-kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
-			    gfp_t flags,
-			    int nodeid,
-			    size_t size)
-{
-	return kmem_cache_alloc_node(cachep, flags, nodeid);
-}
-#endif
-
-static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node)
-{
-	struct kmem_cache *cachep;
-
-	if (__builtin_constant_p(size)) {
-		int i;
-
-		if (!size)
-			return ZERO_SIZE_PTR;
-
-		if (WARN_ON_ONCE(size > KMALLOC_MAX_SIZE))
-			return NULL;
-
-		i = kmalloc_index(size);
-
-#ifdef CONFIG_ZONE_DMA
-		if (flags & GFP_DMA)
-			cachep = kmalloc_dma_caches[i];
-		else
-#endif
-			cachep = kmalloc_caches[i];
-
-		return kmem_cache_alloc_node_trace(cachep, flags, node, size);
-	}
-	return __kmalloc_node(size, flags, node);
-}
-
-#endif	/* CONFIG_NUMA */
-
 #endif	/* _LINUX_SLAB_DEF_H */

Index: linux/include/linux/slob_def.h
===================================================================

include/linux/slob_def.h +1 -23

@@ -1,24 +1,7 @@
 #ifndef __LINUX_SLOB_DEF_H
 #define __LINUX_SLOB_DEF_H
 
-#include <linux/numa.h>
-
-void *kmem_cache_alloc_node(struct kmem_cache *, gfp_t flags, int node);
-
-static __always_inline void *kmem_cache_alloc(struct kmem_cache *cachep,
-					      gfp_t flags)
-{
-	return kmem_cache_alloc_node(cachep, flags, NUMA_NO_NODE);
-}
-
-void *__kmalloc_node(size_t size, gfp_t flags, int node);
-
-static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node)
-{
-	return __kmalloc_node(size, flags, node);
-}
-
-/**
+/*
  * kmalloc - allocate memory
  * @size: how many bytes of memory are required.
  * @flags: the type of memory to allocate (see kcalloc).

@@ -31,9 +14,4 @@ static __always_inline void *kmalloc(siz
 	return __kmalloc_node(size, flags, NUMA_NO_NODE);
 }
 
-static __always_inline void *__kmalloc(size_t size, gfp_t flags)
-{
-	return kmalloc(size, flags);
-}
-
 #endif /* __LINUX_SLOB_DEF_H */

Index: linux/mm/slab.c
===================================================================

mm/slab.c +1 -1

@@ -3681,7 +3681,7 @@ __do_kmalloc_node(size_t size, gfp_t fla
 	cachep = kmalloc_slab(size, flags);
 	if (unlikely(ZERO_OR_NULL_PTR(cachep)))
 		return cachep;
-	return kmem_cache_alloc_node_trace(cachep, flags, node, size);
+	return kmem_cache_alloc_node_trace(cachep, flags, node);
 }
 
 #if defined(CONFIG_DEBUG_SLAB) || defined(CONFIG_TRACING)

Index: linux/mm/slob.c
===================================================================

mm/slob.c +24 -5

@@ -462,11 +462,11 @@ __do_kmalloc_node(size_t size, gfp_t gfp
 	return ret;
 }
 
-void *__kmalloc_node(size_t size, gfp_t gfp, int node)
+void *__kmalloc(size_t size, gfp_t gfp)
 {
-	return __do_kmalloc_node(size, gfp, node, _RET_IP_);
+	return __do_kmalloc_node(size, gfp, NUMA_NO_NODE, _RET_IP_);
 }
-EXPORT_SYMBOL(__kmalloc_node);
+EXPORT_SYMBOL(__kmalloc);
 
 #ifdef CONFIG_TRACING
 void *__kmalloc_track_caller(size_t size, gfp_t gfp, unsigned long caller)

@@ -534,7 +534,7 @@ int __kmem_cache_create(struct kmem_cach
 	return 0;
 }
 
-void *kmem_cache_alloc_node(struct kmem_cache *c, gfp_t flags, int node)
+void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node)
 {
 	void *b;
 

@@ -560,7 +560,27 @@ void *kmem_cache_alloc_node(struct kmem_
 	kmemleak_alloc_recursive(b, c->size, 1, c->flags, flags);
 	return b;
 }
+EXPORT_SYMBOL(slob_alloc_node);
+
+void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags)
+{
+	return slob_alloc_node(cachep, flags, NUMA_NO_NODE);
+}
+EXPORT_SYMBOL(kmem_cache_alloc);
+
+#ifdef CONFIG_NUMA
+void *__kmalloc_node(size_t size, gfp_t gfp, int node)
+{
+	return __do_kmalloc_node(size, gfp, node, _RET_IP_);
+}
+EXPORT_SYMBOL(__kmalloc_node);
+
+void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t gfp, int node)
+{
+	return slob_alloc_node(cachep, gfp, node);
+}
 EXPORT_SYMBOL(kmem_cache_alloc_node);
+#endif
 
 static void __kmem_cache_free(void *b, int size)
 {
 
--

To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Review

Quoted 3 lines

(quoted diff) +3 -0

+#ifndef CONFIG_SLOB
+	if (__builtin_constant_p(size) &&
+		size <= KMALLOC_MAX_CACHE_SIZE && !(flags & SLAB_CACHE_DMA)) {

s/SLAB_CACHE_DMA/GFP_DMA

Quoted 5 lines · Joonsoo Kim

Review

Quoted 3 lines

(quoted diff) +3 -0

+#ifndef CONFIG_SLOB
+	if (__builtin_constant_p(size) &&
+		size <= KMALLOC_MAX_CACHE_SIZE && !(flags & SLAB_CACHE_DMA)) {

s/SLAB_CACHE_DMA/GFP_DMA

Ok. Could you remove the rest of the email in the future? Its difficult to find your comment in the long diff.