SLUB with queueing (V2) beats SLAB netperf TCP_RR

PATCH Applied Christoph Lameter 6 participants 44 messages Jul 9, 2010 – Aug 3, 2010

Subsystems (4)

Memory management MM core Slab allocator Per-CPU memory allocator

lore

Share & tools

lore

Christoph Lameter

author

2010-07-09 19:12 UTC #0 lore raw

cover SLUB with queueing (V2) beats SLAB netperf TCP_RR

The following patchset cleans some pieces up and then equips SLUB with per cpu queues that work similar to SLABs queues. With that approach SLUB wins significantly in hackbench and improves also on tcp_rr.

Hackbench test script:

#!/bin/bash
uname -a
echo "./hackbench 100 process 200000"
./hackbench 100 process 200000
echo "./hackbench 100 process 20000"
./hackbench 100 process 20000
echo "./hackbench 100 process 20000"
./hackbench 100 process 20000
echo "./hackbench 100 process 20000"
./hackbench 100 process 20000
echo "./hackbench 10 process 20000"
./hackbench 10 process 20000
echo "./hackbench 10 process 20000"
./hackbench 10 process 20000
echo "./hackbench 10 process 20000"
./hackbench 10 process 20000
echo "./hackbench 1 process 20000"
./hackbench 1 process 20000
echo "./hackbench 1 process 20000"
./hackbench 1 process 20000
echo "./hackbench 1 process 20000"
./hackbench 1 process 20000

Dell Dual Quad Penryn on Linux 2.6.35-rc3
Time measurements: Smaller is better:

Procs NR SLAB SLUB SLUB+Queuing %
-------------------------------------------------------------
100 200000 2741.3 2764.7 2231.9 -18
100 20000 279.3 270.3 219.0 -27
100 20000 278.0 273.1 219.2 -26
100 20000 279.0 271.7 218.8 -27
10 20000 34.0 35.6 28.8 -18
10 20000 30.3 35.2 28.4 -6
10 20000 32.9 34.6 28.4 -15
1 20000 6.4 6.7 6.5 +1
1 20000 6.3 6.8 6.5 +3
1 20000 6.4 6.9 6.4 0

SLUB+Q also wins against SLAB in netperf:

Script:

#!/bin/bash

TIME=60 # seconds
HOSTNAME=localhost # netserver

NR_CPUS=$(grep ^processor /proc/cpuinfo | wc -l)
echo NR_CPUS=$NR_CPUS

run_netperf() {
for i in $(seq 1 $1); do
netperf -H $HOSTNAME -t TCP_RR -l $TIME &
done
}

ITERATIONS=0
while [ $ITERATIONS -lt 12 ]; do
RATE=0
ITERATIONS=$[$ITERATIONS + 1]
THREADS=$[$NR_CPUS * $ITERATIONS]
RESULTS=$(run_netperf $THREADS | grep -v '[a-zA-Z]' | awk '{ print $6 }')

for j in $RESULTS; do
RATE=$[$RATE + ${j/.*}]
done
echo threads=$THREADS rate=$RATE
done

Dell Dual Quad Penryn on Linux 2.6.35-rc4

Loop counts: Larger is better.

Threads SLAB SLUB+Q %
8 690869 714788 + 3.4
16 680295 711771 + 4.6
24 672677 703014 + 4.5
32 676780 703914 + 4.0
40 668458 699806 + 4.6
48 667017 698908 + 4.7
56 671227 696034 + 3.6
64 667956 696913 + 4.3
72 668332 694931 + 3.9
80 667073 695658 + 4.2
88 682866 697077 + 2.0
96 668089 694719 + 3.9

SLUB+Q is a merging of SLUB with some queuing concepts from SLAB and a new way of managing objects in the slabs using bitmaps. It uses a percpu queue so that free operations can be properly buffered and a bitmap for managing the free/allocated state in the slabs. It is slightly more inefficient than SLUB (due to the need to place large bitmaps --sized a few words--in some slab pages if there are more than BITS_PER_LONG objects in a slab) but in general does not increase space use too much.

The SLAB scheme of not touching the object during management is adopted. SLUB+Q can efficiently free and allocate cache cold objects without causing cache misses.

The queueing patches are likely still be a bit rough around corner cases and special features and need to see some more widespread testing.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Christoph Lameter

author

2010-07-09 19:12 UTC #1 lore raw

PATCH 1/19 Bugfix for semop() not reporting successful operation

[Necessary to make 2.6.35-rc3 not deadlock. Not sure if this is the "right"(tm) fix]

The last change to improve the scalability moved the actual wake-up out of the section that is protected by spin_lock(sma->sem_perm.lock).

This means that IN_WAKEUP can be in queue.status even when the spinlock is acquired by the current task. Thus the same loop that is performed when queue.status is read without the spinlock acquired must be performed when the spinlock is acquired.

Signed-off-by Manfred Spraul <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

---
 ipc/sem.c |   36 ++++++++++++++++++++++++++++++------
 1 files changed, 30 insertions(+), 6 deletions(-)

ipc/sem.c +30 -6

index 506c849..523665f 100644
@@ -1256,6 +1256,32 @@ out:
 	return un;
 }
 
+
+/** get_queue_result - Retrieve the result code from sem_queue
+ * @q: Pointer to queue structure
+ *
+ * The function retrieve the return code from the pending queue. If 
+ * IN_WAKEUP is found in q->status, then we must loop until the value
+ * is replaced with the final value: This may happen if a task is
+ * woken up by an unrelated event (e.g. signal) and in parallel the task
+ * is woken up by another task because it got the requested semaphores.
+ *
+ * The function can be called with or without holding the semaphore spinlock.
+ */
+static int get_queue_result(struct sem_queue *q)
+{
+	int error;
+
+	error = q->status;
+	while(unlikely(error == IN_WAKEUP)) {
+		cpu_relax();
+		error = q->status;
+	}
+
+	return error;
+}
+
+
 SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops,
 		unsigned, nsops, const struct timespec __user *, timeout)
 {

@@ -1409,11 +1435,7 @@ SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops,
 	else
 		schedule();
 
-	error = queue.status;
-	while(unlikely(error == IN_WAKEUP)) {
-		cpu_relax();
-		error = queue.status;
-	}
+	error = get_queue_result(&queue);
 
 	if (error != -EINTR) {
 		/* fast path: update_queue already obtained all requested

@@ -1427,10 +1449,12 @@ SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops,
 		goto out_free;
 	}
 
+	error = get_queue_result(&queue);
+
 	/*
 	 * If queue.status != -EINTR we are woken up by another process
 	 */
-	error = queue.status;
+
 	if (error != -EINTR) {
 		goto out_unlock_free;
 	}

Signature · 8 lines

1.7.0.1


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Christoph Lameter

author

2010-07-09 19:15 UTC #2 lore raw

PATCH 2/19 percpu: make @dyn_size always mean min dyn_size in first chunk init functions

In reply to Christoph Lameter

In pcpu_build_alloc_info() and pcpu_embed_first_chunk(), @dyn_size was ssize_t, -1 meant auto-size, 0 forced 0 and positive meant minimum size. There's no use case for forcing 0 and the upcoming early alloc support always requires non-zero dynamic size. Make @dyn_size always mean minimum dyn_size.

While at it, make pcpu_build_alloc_info() static which doesn't have any external caller as suggested by David Rientjes.

Signed-off-by Tejun Heo <[email protected]>

Acked-by David Rientjes <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

include/linux/percpu.h +1 -6

Index: work/include/linux/percpu.h
===================================================================
@@ -104,16 +104,11 @@ extern struct pcpu_alloc_info * __init p
 							     int nr_units);
 extern void __init pcpu_free_alloc_info(struct pcpu_alloc_info *ai);
 
-extern struct pcpu_alloc_info * __init pcpu_build_alloc_info(
-				size_t reserved_size, ssize_t dyn_size,
-				size_t atom_size,
-				pcpu_fc_cpu_distance_fn_t cpu_distance_fn);
-
 extern int __init pcpu_setup_first_chunk(const struct pcpu_alloc_info *ai,
 					 void *base_addr);
 
 #ifdef CONFIG_NEED_PER_CPU_EMBED_FIRST_CHUNK
-extern int __init pcpu_embed_first_chunk(size_t reserved_size, ssize_t dyn_size,
+extern int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size,
 				size_t atom_size,
 				pcpu_fc_cpu_distance_fn_t cpu_distance_fn,
 				pcpu_fc_alloc_fn_t alloc_fn,

mm/percpu.c +10 ~~-25~~

Index: work/mm/percpu.c
===================================================================
@@ -1013,20 +1013,6 @@ phys_addr_t per_cpu_ptr_to_phys(void *ad
 		return page_to_phys(pcpu_addr_to_page(addr));
 }
 
-static inline size_t pcpu_calc_fc_sizes(size_t static_size,
-					size_t reserved_size,
-					ssize_t *dyn_sizep)
-{
-	size_t size_sum;
-
-	size_sum = PFN_ALIGN(static_size + reserved_size +
-			     (*dyn_sizep >= 0 ? *dyn_sizep : 0));
-	if (*dyn_sizep != 0)
-		*dyn_sizep = size_sum - static_size - reserved_size;
-
-	return size_sum;
-}
-
 /**
  * pcpu_alloc_alloc_info - allocate percpu allocation info
  * @nr_groups: the number of groups

@@ -1085,7 +1071,7 @@ void __init pcpu_free_alloc_info(struct
 /**
  * pcpu_build_alloc_info - build alloc_info considering distances between CPUs
  * @reserved_size: the size of reserved percpu area in bytes
- * @dyn_size: free size for dynamic allocation in bytes, -1 for auto
+ * @dyn_size: minimum free size for dynamic allocation in bytes
  * @atom_size: allocation atom size
  * @cpu_distance_fn: callback to determine distance between cpus, optional
  *

@@ -1103,8 +1089,8 @@ void __init pcpu_free_alloc_info(struct
  * On success, pointer to the new allocation_info is returned.  On
  * failure, ERR_PTR value is returned.
  */
-struct pcpu_alloc_info * __init pcpu_build_alloc_info(
-				size_t reserved_size, ssize_t dyn_size,
+static struct pcpu_alloc_info * __init pcpu_build_alloc_info(
+				size_t reserved_size, size_t dyn_size,
 				size_t atom_size,
 				pcpu_fc_cpu_distance_fn_t cpu_distance_fn)
 {

@@ -1123,13 +1109,15 @@ struct pcpu_alloc_info * __init pcpu_bui
 	memset(group_map, 0, sizeof(group_map));
 	memset(group_cnt, 0, sizeof(group_cnt));
 
+	size_sum = PFN_ALIGN(static_size + reserved_size + dyn_size);
+	dyn_size = size_sum - static_size - reserved_size;
+
 	/*
 	 * Determine min_unit_size, alloc_size and max_upa such that
 	 * alloc_size is multiple of atom_size and is the smallest
 	 * which can accomodate 4k aligned segments which are equal to
 	 * or larger than min_unit_size.
 	 */
-	size_sum = pcpu_calc_fc_sizes(static_size, reserved_size, &dyn_size);
 	min_unit_size = max_t(size_t, size_sum, PCPU_MIN_UNIT_SIZE);
 
 	alloc_size = roundup(min_unit_size, atom_size);

@@ -1532,7 +1520,7 @@ early_param("percpu_alloc", percpu_alloc
 /**
  * pcpu_embed_first_chunk - embed the first percpu chunk into bootmem
  * @reserved_size: the size of reserved percpu area in bytes
- * @dyn_size: free size for dynamic allocation in bytes, -1 for auto
+ * @dyn_size: minimum free size for dynamic allocation in bytes
  * @atom_size: allocation atom size
  * @cpu_distance_fn: callback to determine distance between cpus, optional
  * @alloc_fn: function to allocate percpu page

@@ -1553,10 +1541,7 @@ early_param("percpu_alloc", percpu_alloc
  * vmalloc space is not orders of magnitude larger than distances
  * between node memory addresses (ie. 32bit NUMA machines).
  *
- * When @dyn_size is positive, dynamic area might be larger than
- * specified to fill page alignment.  When @dyn_size is auto,
- * @dyn_size is just big enough to fill page alignment after static
- * and reserved areas.
+ * @dyn_size specifies the minimum dynamic area size.
  *
  * If the needed size is smaller than the minimum or specified unit
  * size, the leftover is returned using @free_fn.

@@ -1564,7 +1549,7 @@ early_param("percpu_alloc", percpu_alloc
  * RETURNS:
  * 0 on success, -errno on failure.
  */
-int __init pcpu_embed_first_chunk(size_t reserved_size, ssize_t dyn_size,
+int __init pcpu_embed_first_chunk(size_t reserved_size, size_t dyn_size,
 				  size_t atom_size,
 				  pcpu_fc_cpu_distance_fn_t cpu_distance_fn,
 				  pcpu_fc_alloc_fn_t alloc_fn,

@@ -1695,7 +1680,7 @@ int __init pcpu_page_first_chunk(size_t
 
 	snprintf(psize_str, sizeof(psize_str), "%luK", PAGE_SIZE >> 10);
 
-	ai = pcpu_build_alloc_info(reserved_size, -1, PAGE_SIZE, NULL);
+	ai = pcpu_build_alloc_info(reserved_size, 0, PAGE_SIZE, NULL);
 	if (IS_ERR(ai))
 		return PTR_ERR(ai);
 	BUG_ON(ai->nr_groups != 1);

Christoph Lameter

author

2010-07-09 19:44 UTC #3 lore raw

PATCH 3/19 percpu: allow limited allocation before slab is online

In reply to Christoph Lameter

This patch updates percpu allocator such that it can serve limited amount of allocation before slab comes online. This is primarily to allow slab to depend on working percpu allocator.

Two parameters, PERCPU_DYNAMIC_EARLY_SIZE and SLOTS, determine how much memory space and allocation map slots are reserved. If this reserved area is exhausted, WARN_ON_ONCE() will trigger and allocation will fail till slab comes online.

The following changes are made to implement early alloc.

* pcpu_mem_alloc() now checks slab_is_available()

* Chunks are allocated using pcpu_mem_alloc()

* Init paths make sure ai->dyn_size is at least as large as PERCPU_DYNAMIC_EARLY_SIZE.

* Initial alloc maps are allocated in __initdata and copied to kmalloc'd areas once slab is online.

Signed-off-by Tejun Heo <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

---
 include/linux/percpu.h |   13 ++++++++++++
 init/main.c            |    1
 include/linux/percpu.h |   13 ++++++++++++
 init/main.c            |    1 
 mm/percpu.c            |   52 +++++++++++++++++++++++++++++++++++++------------
 3 files changed, 54 insertions(+), 12 deletions(-)

mm/percpu.c +40 ~~-12~~

Index: linux-2.6/mm/percpu.c
===================================================================
@@ -282,6 +282,9 @@ static void __maybe_unused pcpu_next_pop
  */
 static void *pcpu_mem_alloc(size_t size)
 {
+	if (WARN_ON_ONCE(!slab_is_available()))
+		return NULL;
+
 	if (size <= PAGE_SIZE)
 		return kzalloc(size, GFP_KERNEL);
 	else {

@@ -392,13 +395,6 @@ static int pcpu_extend_area_map(struct p
 	old_size = chunk->map_alloc * sizeof(chunk->map[0]);
 	memcpy(new, chunk->map, old_size);
 
-	/*
-	 * map_alloc < PCPU_DFL_MAP_ALLOC indicates that the chunk is
-	 * one of the first chunks and still using static map.
-	 */
-	if (chunk->map_alloc >= PCPU_DFL_MAP_ALLOC)
-		old = chunk->map;
-
 	chunk->map_alloc = new_alloc;
 	chunk->map = new;
 	new = NULL;

@@ -604,7 +600,7 @@ static struct pcpu_chunk *pcpu_alloc_chu
 {
 	struct pcpu_chunk *chunk;
 
-	chunk = kzalloc(pcpu_chunk_struct_size, GFP_KERNEL);
+	chunk = pcpu_mem_alloc(pcpu_chunk_struct_size);
 	if (!chunk)
 		return NULL;

@@ -1109,7 +1105,9 @@ static struct pcpu_alloc_info * __init p
 	memset(group_map, 0, sizeof(group_map));
 	memset(group_cnt, 0, sizeof(group_cnt));
 
-	size_sum = PFN_ALIGN(static_size + reserved_size + dyn_size);
+	/* calculate size_sum and ensure dyn_size is enough for early alloc */
+	size_sum = PFN_ALIGN(static_size + reserved_size +
+			    max_t(size_t, dyn_size, PERCPU_DYNAMIC_EARLY_SIZE));
 	dyn_size = size_sum - static_size - reserved_size;
 
 	/*

@@ -1338,7 +1336,8 @@ int __init pcpu_setup_first_chunk(const 
 				  void *base_addr)
 {
 	static char cpus_buf[4096] __initdata;
-	static int smap[2], dmap[2];
+	static int smap[PERCPU_DYNAMIC_EARLY_SLOTS] __initdata;
+	static int dmap[PERCPU_DYNAMIC_EARLY_SLOTS] __initdata;
 	size_t dyn_size = ai->dyn_size;
 	size_t size_sum = ai->static_size + ai->reserved_size + dyn_size;
 	struct pcpu_chunk *schunk, *dchunk = NULL;

@@ -1361,14 +1360,13 @@ int __init pcpu_setup_first_chunk(const 
 } while (0)
 
 	/* sanity checks */
-	BUILD_BUG_ON(ARRAY_SIZE(smap) >= PCPU_DFL_MAP_ALLOC ||
-		     ARRAY_SIZE(dmap) >= PCPU_DFL_MAP_ALLOC);
 	PCPU_SETUP_BUG_ON(ai->nr_groups <= 0);
 	PCPU_SETUP_BUG_ON(!ai->static_size);
 	PCPU_SETUP_BUG_ON(!base_addr);
 	PCPU_SETUP_BUG_ON(ai->unit_size < size_sum);
 	PCPU_SETUP_BUG_ON(ai->unit_size & ~PAGE_MASK);
 	PCPU_SETUP_BUG_ON(ai->unit_size < PCPU_MIN_UNIT_SIZE);
+	PCPU_SETUP_BUG_ON(ai->dyn_size < PERCPU_DYNAMIC_EARLY_SIZE);
 	PCPU_SETUP_BUG_ON(pcpu_verify_alloc_info(ai) < 0);
 
 	/* process group information and build config tables accordingly */

@@ -1806,3 +1804,33 @@ void __init setup_per_cpu_areas(void)
 		__per_cpu_offset[cpu] = delta + pcpu_unit_offsets[cpu];
 }
 #endif /* CONFIG_HAVE_SETUP_PER_CPU_AREA */
+
+/*
+ * First and reserved chunks are initialized with temporary allocation
+ * map in initdata so that they can be used before slab is online.
+ * This function is called after slab is brought up and replaces those
+ * with properly allocated maps.
+ */
+void __init percpu_init_late(void)
+{
+	struct pcpu_chunk *target_chunks[] =
+		{ pcpu_first_chunk, pcpu_reserved_chunk, NULL };
+	struct pcpu_chunk *chunk;
+	unsigned long flags;
+	int i;
+
+	for (i = 0; (chunk = target_chunks[i]); i++) {
+		int *map;
+		const size_t size = PERCPU_DYNAMIC_EARLY_SLOTS * sizeof(map[0]);
+
+		BUILD_BUG_ON(size > PAGE_SIZE);
+
+		map = pcpu_mem_alloc(size);
+		BUG_ON(!map);
+
+		spin_lock_irqsave(&pcpu_lock, flags);
+		memcpy(map, chunk->map, size);
+		chunk->map = map;
+		spin_unlock_irqrestore(&pcpu_lock, flags);
+	}
+}

init/main.c +1 -0

Index: linux-2.6/init/main.c
===================================================================
@@ -532,6 +532,7 @@ static void __init mm_init(void)
 	page_cgroup_init_flatmem();
 	mem_init();
 	kmem_cache_init();
+	percpu_init_late();
 	pgtable_cache_init();
 	vmalloc_init();
 }

include/linux/percpu.h +13 -0

Index: linux-2.6/include/linux/percpu.h
===================================================================
@@ -45,6 +45,16 @@
 #define PCPU_MIN_UNIT_SIZE		PFN_ALIGN(64 << 10)
 
 /*
+ * Percpu allocator can serve percpu allocations before slab is
+ * initialized which allows slab to depend on the percpu allocator.
+ * The following two parameters decide how much resource to
+ * preallocate for this.  Keep PERCPU_DYNAMIC_RESERVE equal to or
+ * larger than PERCPU_DYNAMIC_EARLY_SIZE.
+ */
+#define PERCPU_DYNAMIC_EARLY_SLOTS	128
+#define PERCPU_DYNAMIC_EARLY_SIZE	(12 << 10)
+
+/*
  * PERCPU_DYNAMIC_RESERVE indicates the amount of free area to piggy
  * back on the first chunk for dynamic percpu allocation if arch is
  * manually allocating and mapping it for faster access (as a part of

@@ -135,6 +145,7 @@ extern bool is_kernel_percpu_address(uns
 #ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA
 extern void __init setup_per_cpu_areas(void);
 #endif
+extern void __init percpu_init_late(void);
 
 #else /* CONFIG_SMP */

@@ -148,6 +159,8 @@ static inline bool is_kernel_percpu_addr
 
 static inline void __init setup_per_cpu_areas(void) { }
 
+static inline void __init percpu_init_late(void) { }
+
 static inline void *pcpu_lpage_remapped(void *kaddr)
 {
 	return NULL;

Christoph Lameter

author

2010-07-09 19:12 UTC #4 lore raw

PATCH 4/19 slub: Use a constant for a unspecified node.

In reply to Christoph Lameter

kmalloc_node() and friends can be passed a constant -1 to indicate that no choice was made for the node from which the object needs to come.

Use NUMA_NO_NODE instead of -1.

CC KAMEZAWA Hiroyuki <[email protected]>

Signed-off-by David Rientjes <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |   14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

mm/slub.c +7 -7

Index: linux-2.6/mm/slub.c
===================================================================
@@ -1073,7 +1073,7 @@ static inline struct page *alloc_slab_pa
 
 	flags |= __GFP_NOTRACK;
 
-	if (node == -1)
+	if (node == NUMA_NO_NODE)
 		return alloc_pages(flags, order);
 	else
 		return alloc_pages_exact_node(node, flags, order);

@@ -1387,7 +1387,7 @@ static struct page *get_any_partial(stru
 static struct page *get_partial(struct kmem_cache *s, gfp_t flags, int node)
 {
 	struct page *page;
-	int searchnode = (node == -1) ? numa_node_id() : node;
+	int searchnode = (node == NUMA_NO_NODE) ? numa_node_id() : node;
 
 	page = get_partial_node(get_node(s, searchnode));
 	if (page || (flags & __GFP_THISNODE))

@@ -1515,7 +1515,7 @@ static void flush_all(struct kmem_cache 
 static inline int node_match(struct kmem_cache_cpu *c, int node)
 {
 #ifdef CONFIG_NUMA
-	if (node != -1 && c->node != node)
+	if (node != NUMA_NO_NODE && c->node != node)
 		return 0;
 #endif
 	return 1;

@@ -1727,7 +1727,7 @@ static __always_inline void *slab_alloc(
 
 void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
 {
-	void *ret = slab_alloc(s, gfpflags, -1, _RET_IP_);
+	void *ret = slab_alloc(s, gfpflags, NUMA_NO_NODE, _RET_IP_);
 
 	trace_kmem_cache_alloc(_RET_IP_, ret, s->objsize, s->size, gfpflags);

@@ -1738,7 +1738,7 @@ EXPORT_SYMBOL(kmem_cache_alloc);
 #ifdef CONFIG_TRACING
 void *kmem_cache_alloc_notrace(struct kmem_cache *s, gfp_t gfpflags)
 {
-	return slab_alloc(s, gfpflags, -1, _RET_IP_);
+	return slab_alloc(s, gfpflags, NUMA_NO_NODE, _RET_IP_);
 }
 EXPORT_SYMBOL(kmem_cache_alloc_notrace);
 #endif

@@ -2728,7 +2728,7 @@ void *__kmalloc(size_t size, gfp_t flags
 	if (unlikely(ZERO_OR_NULL_PTR(s)))
 		return s;
 
-	ret = slab_alloc(s, flags, -1, _RET_IP_);
+	ret = slab_alloc(s, flags, NUMA_NO_NODE, _RET_IP_);
 
 	trace_kmalloc(_RET_IP_, ret, size, s->size, flags);

@@ -3312,7 +3312,7 @@ void *__kmalloc_track_caller(size_t size
 	if (unlikely(ZERO_OR_NULL_PTR(s)))
 		return s;
 
-	ret = slab_alloc(s, gfpflags, -1, caller);
+	ret = slab_alloc(s, gfpflags, NUMA_NO_NODE, caller);
 
 	/* Honor the call site pointer we recieved. */
 	trace_kmalloc(caller, ret, size, s->size, gfpflags);

Christoph Lameter

author

2010-07-09 19:12 UTC #5 lore raw

PATCH 5/19 SLUB: Constants need UL

In reply to Christoph Lameter

UL suffix is missing in some constants. Conform to how slab.h uses constants.

Acked-by David Rientjes <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |   20 +++++++++++++++-----
 1 file changed, 15 insertions(+), 5 deletions(-)

mm/slub.c +2 -2

Index: linux-2.6/mm/slub.c
===================================================================
@@ -162,8 +162,8 @@
 #define MAX_OBJS_PER_PAGE	65535 /* since page.objects is u16 */
 
 /* Internal SLUB flags */
-#define __OBJECT_POISON		0x80000000 /* Poison object */
-#define __SYSFS_ADD_DEFERRED	0x40000000 /* Not yet visible via sysfs */
+#define __OBJECT_POISON		0x80000000UL /* Poison object */
+#define __SYSFS_ADD_DEFERRED	0x40000000UL /* Not yet visible via sysfs */
 
 static int kmem_size = sizeof(struct kmem_cache);

Christoph Lameter

author

2010-07-09 19:12 UTC #6 lore raw

PATCH 6/19 slub: Check kasprintf results in kmem_cache_init()

In reply to Christoph Lameter

Small allocations may fail during slab bringup which is fatal. Add a BUG_ON() so that we fail immediately rather than failing later during sysfs processing.

CC David Rientjes <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

mm/slub.c +6 -3

Index: linux-2.6/mm/slub.c
===================================================================
@@ -3118,9 +3118,12 @@ void __init kmem_cache_init(void)
 	slab_state = UP;
 
 	/* Provide the correct kmalloc names now that the caches are up */
-	for (i = KMALLOC_SHIFT_LOW; i < SLUB_PAGE_SHIFT; i++)
-		kmalloc_caches[i]. name =
-			kasprintf(GFP_NOWAIT, "kmalloc-%d", 1 << i);
+	for (i = KMALLOC_SHIFT_LOW; i < SLUB_PAGE_SHIFT; i++) {
+		char *s = kasprintf(GFP_NOWAIT, "kmalloc-%d", 1 << i);
+
+		BUG_ON(!s);
+		kmalloc_caches[i].name = s;
+	}
 
 #ifdef CONFIG_SMP
 	register_cpu_notifier(&slab_notifier);

David Rientjes

reviewer Acked-by on v1 6/19

2010-07-14 22:17 UTC #7 lore raw

Re: [S+Q2 06/19] slub: Check kasprintf results in kmem_cache_init()

Quoted 6 lines · Christoph Lameter

Small allocations may fail during slab bringup which is fatal. Add a BUG_ON() so that we fail immediately rather than failing later during sysfs processing.
CC David Rientjes <[email protected]>
Signed-off-by Christoph Lameter <[email protected]>

Acked-by David Rientjes <[email protected]>

Christoph Lameter

author

2010-07-09 19:12 UTC #8 lore raw

PATCH 7/19 slub: Allow removal of slab caches during boot

In reply to Christoph Lameter

If a slab cache is removed before we have setup sysfs then simply skip over the sysfs handling.

Cc Benjamin Herrenschmidt <[email protected]>

Cc Roland Dreier <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |    7 +++++++
 1 file changed, 7 insertions(+)

mm/slub.c +7 -0

Index: linux-2.6/mm/slub.c
===================================================================
@@ -4507,6 +4507,13 @@ static int sysfs_slab_add(struct kmem_ca
 
 static void sysfs_slab_remove(struct kmem_cache *s)
 {
+	if (slab_state < SYSFS)
+		/*
+		 * Sysfs has not been setup yet so no need to remove the
+		 * cache from sysfs.
+		 */
+		return;
+
 	kobject_uevent(&s->kobj, KOBJ_REMOVE);
 	kobject_del(&s->kobj);
 	kobject_put(&s->kobj);

David Rientjes

reviewer Acked-by on v1 7/19

2010-07-14 23:49 UTC #9 lore raw

Re: [S+Q2 07/19] slub: Allow removal of slab caches during boot

Quoted 6 lines · Christoph Lameter

If a slab cache is removed before we have setup sysfs then simply skip over the sysfs handling.
Cc Benjamin Herrenschmidt <[email protected]>
Cc Roland Dreier <[email protected]>
Signed-off-by Christoph Lameter <[email protected]>

Acked-by David Rientjes <[email protected]>

I missed this case earlier because I didn't consider slab caches being created and destroyed prior to slab_state == SYSFS, sorry!

Benjamin Herrenschmidt

2010-07-19 00:09 UTC #10 lore raw

Re: [S+Q2 07/19] slub: Allow removal of slab caches during boot

Quoted 5 lines · David Rientjes

Quoted 3 lines · Christoph Lameter
If a slab cache is removed before we have setup sysfs then simply skip over the sysfs handling.

Quoted 8 lines

Quoted 3 lines
Cc Benjamin Herrenschmidt <[email protected]>
Cc Roland Dreier <[email protected]>
Signed-off-by Christoph Lameter <[email protected]>
Acked-by David Rientjes <[email protected]>
I missed this case earlier because I didn't consider slab caches being created and destroyed prior to slab_state == SYSFS, sorry!

Ok so I may be a bit sleepy or something but I still fail to see how this whole thing isn't totally racy...

AFAIK. By the time we switch the slab state, we -do- have all CPUs up and can race happily between creating slab caches and creating the sysfs files...

Ben.

Christoph Lameter

author

2010-07-19 16:42 UTC #11 lore raw

Re: [S+Q2 07/19] slub: Allow removal of slab caches during boot

Quoted 15 lines · Benjamin Herrenschmidt

Quoted 5 lines · David Rientjes
Quoted 3 lines · Christoph Lameter
If a slab cache is removed before we have setup sysfs then simply skip over the sysfs handling.
Quoted 8 lines
Quoted 3 lines
Cc Benjamin Herrenschmidt <[email protected]>
Cc Roland Dreier <[email protected]>
Signed-off-by Christoph Lameter <[email protected]>
Acked-by David Rientjes <[email protected]>
I missed this case earlier because I didn't consider slab caches being created and destroyed prior to slab_state == SYSFS, sorry!

Quoted 6 lines

Ok so I may be a bit sleepy or something but I still fail to see how this whole thing isn't totally racy...
AFAIK. By the time we switch the slab state, we -do- have all CPUs up and can race happily between creating slab caches and creating the sysfs files...

If kmem_cache_init_late() is called after all other processors are up then we need to serialize the activities. But we cannot do that since the slub_lock is taken during kmalloc() for dynamic dma creation (lockdep will complain although we never use dma caches for sysfs....).

After removal of dynamic dma creation we can take the lock for all of slab creation and removal.

Like in the following patch:

Subject: slub: Allow removal of slab caches during boot

Serialize kmem_cache_create and kmem_cache_destroy using the slub_lock. Only possible after the use of the slub_lock during dynamic dma creation has been removed.

Then make sure that the setup of the slab sysfs entries does not race with kmem_cache_create and kmem_cache destroy.

If a slab cache is removed before we have setup sysfs then simply skip over the sysfs handling.

Cc Benjamin Herrenschmidt <[email protected]>

Cc Roland Dreier <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |   24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)

mm/slub.c +15 -9

Index: linux-2.6/mm/slub.c
===================================================================
@@ -2490,7 +2490,6 @@ void kmem_cache_destroy(struct kmem_cach
 	s->refcount--;
 	if (!s->refcount) {
 		list_del(&s->list);
-		up_write(&slub_lock);
 		if (kmem_cache_close(s)) {
 			printk(KERN_ERR "SLUB %s: %s called for cache that "
 				"still has objects.\n", s->name, __func__);

@@ -2499,8 +2498,8 @@ void kmem_cache_destroy(struct kmem_cach
 		if (s->flags & SLAB_DESTROY_BY_RCU)
 			rcu_barrier();
 		sysfs_slab_remove(s);
-	} else
-		up_write(&slub_lock);
+	}
+	up_write(&slub_lock);
 }
 EXPORT_SYMBOL(kmem_cache_destroy);

@@ -3226,14 +3225,12 @@ struct kmem_cache *kmem_cache_create(con
 		 */
 		s->objsize = max(s->objsize, (int)size);
 		s->inuse = max_t(int, s->inuse, ALIGN(size, sizeof(void *)));
-		up_write(&slub_lock);
 
 		if (sysfs_slab_alias(s, name)) {
-			down_write(&slub_lock);
 			s->refcount--;
-			up_write(&slub_lock);
 			goto err;
 		}
+		up_write(&slub_lock);
 		return s;
 	}

@@ -3242,14 +3239,12 @@ struct kmem_cache *kmem_cache_create(con
 		if (kmem_cache_open(s, GFP_KERNEL, name,
 				size, align, flags, ctor)) {
 			list_add(&s->list, &slab_caches);
-			up_write(&slub_lock);
 			if (sysfs_slab_add(s)) {
-				down_write(&slub_lock);
 				list_del(&s->list);
-				up_write(&slub_lock);
 				kfree(s);
 				goto err;
 			}
+			up_write(&slub_lock);
 			return s;
 		}
 		kfree(s);

@@ -4507,6 +4502,13 @@ static int sysfs_slab_add(struct kmem_ca
 
 static void sysfs_slab_remove(struct kmem_cache *s)
 {
+	if (slab_state < SYSFS)
+		/*
+		 * Sysfs has not been setup yet so no need to remove the
+		 * cache from sysfs.
+		 */
+		return;
+
 	kobject_uevent(&s->kobj, KOBJ_REMOVE);
 	kobject_del(&s->kobj);
 	kobject_put(&s->kobj);

@@ -4552,8 +4554,11 @@ static int __init slab_sysfs_init(void)
 	struct kmem_cache *s;
 	int err;
 
+	down_write(&slub_lock);
+
 	slab_kset = kset_create_and_add("slab", &slab_uevent_ops, kernel_kobj);
 	if (!slab_kset) {
+		up_write(&slub_lock);
 		printk(KERN_ERR "Cannot register slab subsystem.\n");
 		return -ENOSYS;
 	}

@@ -4578,6 +4583,7 @@ static int __init slab_sysfs_init(void)
 		kfree(al);
 	}
 
+	up_write(&slub_lock);
 	resiliency_test();
 	return 0;
 }

Pekka Enberg

2010-07-31 09:41 UTC #12 lore raw

Re: [S+Q2 07/19] slub: Allow removal of slab caches during boot

Quoted 26 lines · Christoph Lameter

Quoted 6 lines
Ok so I may be a bit sleepy or something but I still fail to see how this whole thing isn't totally racy...
AFAIK. By the time we switch the slab state, we -do- have all CPUs up and can race happily between creating slab caches and creating the sysfs files...
If kmem_cache_init_late() is called after all other processors are up then we need to serialize the activities. But we cannot do that since the slub_lock is taken during kmalloc() for dynamic dma creation (lockdep will complain although we never use dma caches for sysfs....).
After removal of dynamic dma creation we can take the lock for all of slab creation and removal.
Like in the following patch:
Subject: slub: Allow removal of slab caches during boot
Serialize kmem_cache_create and kmem_cache_destroy using the slub_lock. Only possible after the use of the slub_lock during dynamic dma creation has been removed.
Then make sure that the setup of the slab sysfs entries does not race with kmem_cache_create and kmem_cache destroy.

Quoted 6 lines

If a slab cache is removed before we have setup sysfs then simply skip over the sysfs handling.
Cc Benjamin Herrenschmidt <[email protected]>
Cc Roland Dreier <[email protected]>
Signed-off-by Christoph Lameter <[email protected]>

Christoph, Ben, should I queue this up for 2.6.36?

Quoted 89 lines

---
 mm/slub.c |   24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)

Index: linux-2.6/mm/slub.c
===================================================================

mm/slub.c +14 -9

@@ -2490,7 +2490,6 @@ void kmem_cache_destroy(struct kmem_cach
 	s->refcount--;
 	if (!s->refcount) {
 		list_del(&s->list);
-		up_write(&slub_lock);
 		if (kmem_cache_close(s)) {
 			printk(KERN_ERR "SLUB %s: %s called for cache that "
 				"still has objects.\n", s->name, __func__);

@@ -2499,8 +2498,8 @@ void kmem_cache_destroy(struct kmem_cach
 		if (s->flags & SLAB_DESTROY_BY_RCU)
 			rcu_barrier();
 		sysfs_slab_remove(s);
-	} else
-		up_write(&slub_lock);
+	}
+	up_write(&slub_lock);
 }
 EXPORT_SYMBOL(kmem_cache_destroy);

@@ -3226,14 +3225,12 @@ struct kmem_cache *kmem_cache_create(con
 		 */
 		s->objsize = max(s->objsize, (int)size);
 		s->inuse = max_t(int, s->inuse, ALIGN(size, sizeof(void *)));
-		up_write(&slub_lock);
 
 		if (sysfs_slab_alias(s, name)) {
-			down_write(&slub_lock);
 			s->refcount--;
-			up_write(&slub_lock);
 			goto err;
 		}
+		up_write(&slub_lock);
 		return s;
 	}

@@ -3242,14 +3239,12 @@ struct kmem_cache *kmem_cache_create(con
 		if (kmem_cache_open(s, GFP_KERNEL, name,
 				size, align, flags, ctor)) {
 			list_add(&s->list, &slab_caches);
-			up_write(&slub_lock);
 			if (sysfs_slab_add(s)) {
-				down_write(&slub_lock);
 				list_del(&s->list);
-				up_write(&slub_lock);
 				kfree(s);
 				goto err;
 			}
+			up_write(&slub_lock);
 			return s;
 		}
 		kfree(s);

@@ -4507,6 +4502,13 @@ static int sysfs_slab_add(struct kmem_ca
 
 static void sysfs_slab_remove(struct kmem_cache *s)
 {
+	if (slab_state < SYSFS)
+		/*
+		 * Sysfs has not been setup yet so no need to remove the
+		 * cache from sysfs.
+		 */
+		return;
+
 	kobject_uevent(&s->kobj, KOBJ_REMOVE);
 	kobject_del(&s->kobj);
 	kobject_put(&s->kobj);

@@ -4552,8 +4554,11 @@ static int __init slab_sysfs_init(void)
 	struct kmem_cache *s;
 	int err;
 
+	down_write(&slub_lock);
+
 	slab_kset = kset_create_and_add("slab", &slab_uevent_ops, kernel_kobj);
 	if (!slab_kset) {
+		up_write(&slub_lock);
 		printk(KERN_ERR "Cannot register slab subsystem.\n");
 		return -ENOSYS;
 	}

mm/slub.c static int __init slab_sysfs_init(void) · line 4589

Quoted 7 lines

mm/slub.c +1 -0

@@ -4578,6 +4583,7 @@ static int __init slab_sysfs_init(void)
 		kfree(al);
 	}
 
+	up_write(&slub_lock);
 	resiliency_test();
 	return 0;
 }

Christoph Lameter

author

2010-08-02 15:36 UTC #13 lore raw

Re: [S+Q2 07/19] slub: Allow removal of slab caches during boot

Quoted 1 line · Pekka Enberg

Christoph, Ben, should I queue this up for 2.6.36?

Yes.

Pekka Enberg

Applied

2010-08-03 04:28 UTC #14 lore raw

Re: [S+Q2 07/19] slub: Allow removal of slab caches during boot

Quoted 5 lines · Christoph Lameter

Quoted 1 line · Pekka Enberg
Christoph, Ben, should I queue this up for 2.6.36?
Yes.

Applied, thanks!

Christoph Lameter

author

2010-07-09 19:12 UTC #15 lore raw

PATCH 8/19 slub: Use kmem_cache flags to detect if slab is in debugging mode.

In reply to Christoph Lameter

The cacheline with the flags is reachable from the hot paths after the percpu allocator changes went in. So there is no need anymore to put a flag into each slab page. Get rid of the SlubDebug flag and use the flags in kmem_cache instead.

Acked-by David Rientjes <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

---
 include/linux/page-flags.h |    2 --
 mm/slub.c                  |   33 ++++++++++++---------------------
 2 files changed, 12 insertions(+), 23 deletions(-)

include/linux/page-flags.h +0 -2

Index: linux-2.6/include/linux/page-flags.h
===================================================================
@@ -128,7 +128,6 @@ enum pageflags {
 
 	/* SLUB */
 	PG_slub_frozen = PG_active,
-	PG_slub_debug = PG_error,
 };
 
 #ifndef __GENERATING_BOUNDS_H

@@ -215,7 +214,6 @@ PAGEFLAG(SwapBacked, swapbacked) __CLEAR
 __PAGEFLAG(SlobFree, slob_free)
 
 __PAGEFLAG(SlubFrozen, slub_frozen)
-__PAGEFLAG(SlubDebug, slub_debug)
 
 /*
  * Private page markings that may be used by the filesystem that owns the page

mm/slub.c +12 ~~-21~~

Index: linux-2.6/mm/slub.c
===================================================================
@@ -107,11 +107,17 @@
  * 			the fast path and disables lockless freelists.
  */
 
+#define SLAB_DEBUG_FLAGS (SLAB_RED_ZONE | SLAB_POISON | SLAB_STORE_USER | \
+		SLAB_TRACE | SLAB_DEBUG_FREE)
+
+static inline int kmem_cache_debug(struct kmem_cache *s)
+{
 #ifdef CONFIG_SLUB_DEBUG
-#define SLABDEBUG 1
+	return unlikely(s->flags & SLAB_DEBUG_FLAGS);
 #else
-#define SLABDEBUG 0
+	return 0;
 #endif
+}
 
 /*
  * Issues still to be resolved:

@@ -1157,9 +1163,6 @@ static struct page *new_slab(struct kmem
 	inc_slabs_node(s, page_to_nid(page), page->objects);
 	page->slab = s;
 	page->flags |= 1 << PG_slab;
-	if (s->flags & (SLAB_DEBUG_FREE | SLAB_RED_ZONE | SLAB_POISON |
-			SLAB_STORE_USER | SLAB_TRACE))
-		__SetPageSlubDebug(page);
 
 	start = page_address(page);

@@ -1186,14 +1189,13 @@ static void __free_slab(struct kmem_cach
 	int order = compound_order(page);
 	int pages = 1 << order;
 
-	if (unlikely(SLABDEBUG && PageSlubDebug(page))) {
+	if (kmem_cache_debug(s)) {
 		void *p;
 
 		slab_pad_check(s, page);
 		for_each_object(p, s, page_address(page),
 						page->objects)
 			check_object(s, page, p, 0);
-		__ClearPageSlubDebug(page);
 	}
 
 	kmemcheck_free_shadow(page, compound_order(page));

@@ -1415,8 +1417,7 @@ static void unfreeze_slab(struct kmem_ca
 			stat(s, tail ? DEACTIVATE_TO_TAIL : DEACTIVATE_TO_HEAD);
 		} else {
 			stat(s, DEACTIVATE_FULL);
-			if (SLABDEBUG && PageSlubDebug(page) &&
-						(s->flags & SLAB_STORE_USER))
+			if (kmem_cache_debug(s) && (s->flags & SLAB_STORE_USER))
 				add_full(n, page);
 		}
 		slab_unlock(page);

@@ -1624,7 +1625,7 @@ load_freelist:
 	object = c->page->freelist;
 	if (unlikely(!object))
 		goto another_slab;
-	if (unlikely(SLABDEBUG && PageSlubDebug(c->page)))
+	if (kmem_cache_debug(s))
 		goto debug;
 
 	c->freelist = get_freepointer(s, object);

@@ -1783,7 +1784,7 @@ static void __slab_free(struct kmem_cach
 	stat(s, FREE_SLOWPATH);
 	slab_lock(page);
 
-	if (unlikely(SLABDEBUG && PageSlubDebug(page)))
+	if (kmem_cache_debug(s))
 		goto debug;
 
 checks_ok:

@@ -3398,16 +3399,6 @@ static void validate_slab_slab(struct km
 	} else
 		printk(KERN_INFO "SLUB %s: Skipped busy slab 0x%p\n",
 			s->name, page);
-
-	if (s->flags & DEBUG_DEFAULT_FLAGS) {
-		if (!PageSlubDebug(page))
-			printk(KERN_ERR "SLUB %s: SlubDebug not set "
-				"on slab 0x%p\n", s->name, page);
-	} else {
-		if (PageSlubDebug(page))
-			printk(KERN_ERR "SLUB %s: SlubDebug set on "
-				"slab 0x%p\n", s->name, page);
-	}
 }
 
 static int validate_slab_node(struct kmem_cache *s,

Christoph Lameter

author

2010-07-09 19:12 UTC #16 lore raw

PATCH 9/19 slub: discard_slab_unlock

In reply to Christoph Lameter

The sequence of unlocking a slab and freeing occurs multiple times. Put the common into a single function. There is no intend here to use this to enforce a sequencing of events leading up to the discarding of a slab page.

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |   16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

mm/slub.c +10 -6

Index: linux-2.6/mm/slub.c
===================================================================
@@ -1260,6 +1260,13 @@ static __always_inline int slab_trylock(
 	return rc;
 }
 
+static void discard_slab_unlock(struct kmem_cache *s,
+	struct page *page)
+{
+	slab_unlock(page);
+	discard_slab(s, page);
+}
+
 /*
  * Management of partially allocated slabs
  */

@@ -1437,9 +1444,8 @@ static void unfreeze_slab(struct kmem_ca
 			add_partial(n, page, 1);
 			slab_unlock(page);
 		} else {
-			slab_unlock(page);
 			stat(s, FREE_SLAB);
-			discard_slab(s, page);
+			discard_slab_unlock(s, page);
 		}
 	}
 }

@@ -1822,9 +1828,8 @@ slab_empty:
 		remove_partial(s, page);
 		stat(s, FREE_REMOVE_PARTIAL);
 	}
-	slab_unlock(page);
 	stat(s, FREE_SLAB);
-	discard_slab(s, page);
+	discard_slab_unlock(s, page);
 	return;
 
 debug:

@@ -2893,8 +2898,7 @@ int kmem_cache_shrink(struct kmem_cache 
 				 */
 				list_del(&page->lru);
 				n->nr_partial--;
-				slab_unlock(page);
-				discard_slab(s, page);
+				discard_slab_unlock(s, page);
 			} else {
 				list_move(&page->lru,
 				slabs_by_inuse + page->inuse);

Christoph Lameter

author

2010-07-09 19:12 UTC #17 lore raw

PATCH 10/19 slub: remove dynamic dma slab allocation

In reply to Christoph Lameter

Remove the dynamic dma slab allocation since this causes too many issues with nested locks etc etc. The change avoids passing gfpflags into many functions.

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |  151 ++++++++++++++++----------------------------------------------
 1 file changed, 40 insertions(+), 111 deletions(-)

mm/slub.c +40 ~~-111~~

Index: linux-2.6/mm/slub.c
===================================================================
@@ -2070,7 +2070,7 @@ init_kmem_cache_node(struct kmem_cache_n
 
 static DEFINE_PER_CPU(struct kmem_cache_cpu, kmalloc_percpu[KMALLOC_CACHES]);
 
-static inline int alloc_kmem_cache_cpus(struct kmem_cache *s, gfp_t flags)
+static inline int alloc_kmem_cache_cpus(struct kmem_cache *s)
 {
 	if (s < kmalloc_caches + KMALLOC_CACHES && s >= kmalloc_caches)
 		/*

@@ -2097,7 +2097,7 @@ static inline int alloc_kmem_cache_cpus(
  * when allocating for the kmalloc_node_cache. This is used for bootstrapping
  * memory on a fresh node that has no slab structures yet.
  */
-static void early_kmem_cache_node_alloc(gfp_t gfpflags, int node)
+static void early_kmem_cache_node_alloc(int node)
 {
 	struct page *page;
 	struct kmem_cache_node *n;

@@ -2105,7 +2105,7 @@ static void early_kmem_cache_node_alloc(
 
 	BUG_ON(kmalloc_caches->size < sizeof(struct kmem_cache_node));
 
-	page = new_slab(kmalloc_caches, gfpflags, node);
+	page = new_slab(kmalloc_caches, GFP_KERNEL, node);
 
 	BUG_ON(!page);
 	if (page_to_nid(page) != node) {

@@ -2149,7 +2149,7 @@ static void free_kmem_cache_nodes(struct
 	}
 }
 
-static int init_kmem_cache_nodes(struct kmem_cache *s, gfp_t gfpflags)
+static int init_kmem_cache_nodes(struct kmem_cache *s)
 {
 	int node;

@@ -2157,11 +2157,11 @@ static int init_kmem_cache_nodes(struct 
 		struct kmem_cache_node *n;
 
 		if (slab_state == DOWN) {
-			early_kmem_cache_node_alloc(gfpflags, node);
+			early_kmem_cache_node_alloc(node);
 			continue;
 		}
 		n = kmem_cache_alloc_node(kmalloc_caches,
-						gfpflags, node);
+						GFP_KERNEL, node);
 
 		if (!n) {
 			free_kmem_cache_nodes(s);

@@ -2178,7 +2178,7 @@ static void free_kmem_cache_nodes(struct
 {
 }
 
-static int init_kmem_cache_nodes(struct kmem_cache *s, gfp_t gfpflags)
+static int init_kmem_cache_nodes(struct kmem_cache *s)
 {
 	init_kmem_cache_node(&s->local_node, s);
 	return 1;

@@ -2318,7 +2318,7 @@ static int calculate_sizes(struct kmem_c
 
 }
 
-static int kmem_cache_open(struct kmem_cache *s, gfp_t gfpflags,
+static int kmem_cache_open(struct kmem_cache *s,
 		const char *name, size_t size,
 		size_t align, unsigned long flags,
 		void (*ctor)(void *))

@@ -2354,10 +2354,10 @@ static int kmem_cache_open(struct kmem_c
 #ifdef CONFIG_NUMA
 	s->remote_node_defrag_ratio = 1000;
 #endif
-	if (!init_kmem_cache_nodes(s, gfpflags & ~SLUB_DMA))
+	if (!init_kmem_cache_nodes(s))
 		goto error;
 
-	if (alloc_kmem_cache_cpus(s, gfpflags & ~SLUB_DMA))
+	if (alloc_kmem_cache_cpus(s))
 		return 1;
 
 	free_kmem_cache_nodes(s);

@@ -2517,6 +2517,10 @@ EXPORT_SYMBOL(kmem_cache_destroy);
 struct kmem_cache kmalloc_caches[KMALLOC_CACHES] __cacheline_aligned;
 EXPORT_SYMBOL(kmalloc_caches);
 
+#ifdef CONFIG_ZONE_DMA
+static struct kmem_cache kmalloc_dma_caches[SLUB_PAGE_SHIFT];
+#endif
+
 static int __init setup_slub_min_order(char *str)
 {
 	get_option(&str, &slub_min_order);

@@ -2553,116 +2557,26 @@ static int __init setup_slub_nomerge(cha
 
 __setup("slub_nomerge", setup_slub_nomerge);
 
-static struct kmem_cache *create_kmalloc_cache(struct kmem_cache *s,
-		const char *name, int size, gfp_t gfp_flags)
+static void create_kmalloc_cache(struct kmem_cache *s,
+		const char *name, int size, unsigned int flags)
 {
-	unsigned int flags = 0;
-
-	if (gfp_flags & SLUB_DMA)
-		flags = SLAB_CACHE_DMA;
-
 	/*
 	 * This function is called with IRQs disabled during early-boot on
 	 * single CPU so there's no need to take slub_lock here.
 	 */
-	if (!kmem_cache_open(s, gfp_flags, name, size, ARCH_KMALLOC_MINALIGN,
+	if (!kmem_cache_open(s, name, size, ARCH_KMALLOC_MINALIGN,
 								flags, NULL))
 		goto panic;
 
 	list_add(&s->list, &slab_caches);
 
-	if (sysfs_slab_add(s))
-		goto panic;
-	return s;
+	if (!sysfs_slab_add(s))
+		return;
 
 panic:
 	panic("Creation of kmalloc slab %s size=%d failed.\n", name, size);
 }
 
-#ifdef CONFIG_ZONE_DMA
-static struct kmem_cache *kmalloc_caches_dma[SLUB_PAGE_SHIFT];
-
-static void sysfs_add_func(struct work_struct *w)
-{
-	struct kmem_cache *s;
-
-	down_write(&slub_lock);
-	list_for_each_entry(s, &slab_caches, list) {
-		if (s->flags & __SYSFS_ADD_DEFERRED) {
-			s->flags &= ~__SYSFS_ADD_DEFERRED;
-			sysfs_slab_add(s);
-		}
-	}
-	up_write(&slub_lock);
-}
-
-static DECLARE_WORK(sysfs_add_work, sysfs_add_func);
-
-static noinline struct kmem_cache *dma_kmalloc_cache(int index, gfp_t flags)
-{
-	struct kmem_cache *s;
-	char *text;
-	size_t realsize;
-	unsigned long slabflags;
-	int i;
-
-	s = kmalloc_caches_dma[index];
-	if (s)
-		return s;
-
-	/* Dynamically create dma cache */
-	if (flags & __GFP_WAIT)
-		down_write(&slub_lock);
-	else {
-		if (!down_write_trylock(&slub_lock))
-			goto out;
-	}
-
-	if (kmalloc_caches_dma[index])
-		goto unlock_out;
-
-	realsize = kmalloc_caches[index].objsize;
-	text = kasprintf(flags & ~SLUB_DMA, "kmalloc_dma-%d",
-			 (unsigned int)realsize);
-
-	s = NULL;
-	for (i = 0; i < KMALLOC_CACHES; i++)
-		if (!kmalloc_caches[i].size)
-			break;
-
-	BUG_ON(i >= KMALLOC_CACHES);
-	s = kmalloc_caches + i;
-
-	/*
-	 * Must defer sysfs creation to a workqueue because we don't know
-	 * what context we are called from. Before sysfs comes up, we don't
-	 * need to do anything because our sysfs initcall will start by
-	 * adding all existing slabs to sysfs.
-	 */
-	slabflags = SLAB_CACHE_DMA|SLAB_NOTRACK;
-	if (slab_state >= SYSFS)
-		slabflags |= __SYSFS_ADD_DEFERRED;
-
-	if (!text || !kmem_cache_open(s, flags, text,
-			realsize, ARCH_KMALLOC_MINALIGN, slabflags, NULL)) {
-		s->size = 0;
-		kfree(text);
-		goto unlock_out;
-	}
-
-	list_add(&s->list, &slab_caches);
-	kmalloc_caches_dma[index] = s;
-
-	if (slab_state >= SYSFS)
-		schedule_work(&sysfs_add_work);
-
-unlock_out:
-	up_write(&slub_lock);
-out:
-	return kmalloc_caches_dma[index];
-}
-#endif
-
 /*
  * Conversion table for small slabs sizes / 8 to the index in the
  * kmalloc array. This is necessary for slabs < 192 since we have non power

@@ -2715,7 +2629,7 @@ static struct kmem_cache *get_slab(size_
 
 #ifdef CONFIG_ZONE_DMA
 	if (unlikely((flags & SLUB_DMA)))
-		return dma_kmalloc_cache(index, flags);
+		return &kmalloc_dma_caches[index];
 
 #endif
 	return &kmalloc_caches[index];

@@ -3053,7 +2967,7 @@ void __init kmem_cache_init(void)
 	 * kmem_cache_open for slab_state == DOWN.
 	 */
 	create_kmalloc_cache(&kmalloc_caches[0], "kmem_cache_node",
-		sizeof(struct kmem_cache_node), GFP_NOWAIT);
+		sizeof(struct kmem_cache_node), 0);
 	kmalloc_caches[0].refcount = -1;
 	caches++;

@@ -3066,18 +2980,18 @@ void __init kmem_cache_init(void)
 	/* Caches that are not of the two-to-the-power-of size */
 	if (KMALLOC_MIN_SIZE <= 32) {
 		create_kmalloc_cache(&kmalloc_caches[1],
-				"kmalloc-96", 96, GFP_NOWAIT);
+				"kmalloc-96", 96, 0);
 		caches++;
 	}
 	if (KMALLOC_MIN_SIZE <= 64) {
 		create_kmalloc_cache(&kmalloc_caches[2],
-				"kmalloc-192", 192, GFP_NOWAIT);
+				"kmalloc-192", 192, 0);
 		caches++;
 	}
 
 	for (i = KMALLOC_SHIFT_LOW; i < SLUB_PAGE_SHIFT; i++) {
 		create_kmalloc_cache(&kmalloc_caches[i],
-			"kmalloc", 1 << i, GFP_NOWAIT);
+			"kmalloc", 1 << i, 0);
 		caches++;
 	}

@@ -3150,6 +3064,21 @@ void __init kmem_cache_init(void)
 
 void __init kmem_cache_init_late(void)
 {
+#ifdef CONFIG_ZONE_DMA
+	int i;
+
+	for (i = 0; i < SLUB_PAGE_SHIFT; i++) {
+		struct kmem_cache *s = &kmalloc_caches[i];
+
+		if (s && s->size) {
+			char *name = kasprintf(GFP_KERNEL,
+				 "dma-kmalloc-%d", s->objsize);
+
+			create_kmalloc_cache(&kmalloc_dma_caches[i],
+				name, s->objsize, SLAB_CACHE_DMA);
+		}
+	}
+#endif
 }
 
 /*

@@ -3244,7 +3173,7 @@ struct kmem_cache *kmem_cache_create(con
 
 	s = kmalloc(kmem_size, GFP_KERNEL);
 	if (s) {
-		if (kmem_cache_open(s, GFP_KERNEL, name,
+		if (kmem_cache_open(s, name,
 				size, align, flags, ctor)) {
 			list_add(&s->list, &slab_caches);
 			up_write(&slub_lock);

Christoph Lameter

author

2010-07-09 19:15 UTC #18 lore raw

PATCH 11/19 slub: Remove static kmem_cache_cpu array for boot

In reply to Christoph Lameter

The percpu allocator can now handle allocations in early boot. So drop the static kmem_cache_cpu array.

Cc Tejun Heo <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |   21 ++++++---------------
 1 file changed, 6 insertions(+), 15 deletions(-)

mm/slub.c +4 ~~-13~~

Index: linux-2.6/mm/slub.c
===================================================================
@@ -2068,23 +2068,14 @@ init_kmem_cache_node(struct kmem_cache_n
 #endif
 }
 
-static DEFINE_PER_CPU(struct kmem_cache_cpu, kmalloc_percpu[KMALLOC_CACHES]);
-
 static inline int alloc_kmem_cache_cpus(struct kmem_cache *s)
 {
-	if (s < kmalloc_caches + KMALLOC_CACHES && s >= kmalloc_caches)
-		/*
-		 * Boot time creation of the kmalloc array. Use static per cpu data
-		 * since the per cpu allocator is not available yet.
-		 */
-		s->cpu_slab = kmalloc_percpu + (s - kmalloc_caches);
-	else
-		s->cpu_slab =  alloc_percpu(struct kmem_cache_cpu);
+	BUILD_BUG_ON(PERCPU_DYNAMIC_EARLY_SIZE <
+			SLUB_PAGE_SHIFT * sizeof(struct kmem_cache));
 
-	if (!s->cpu_slab)
-		return 0;
+	s->cpu_slab = alloc_percpu(struct kmem_cache_cpu);
 
-	return 1;
+	return s->cpu_slab != NULL;
 }
 
 #ifdef CONFIG_NUMA

Christoph Lameter

author

2010-07-09 19:12 UTC #19 lore raw

PATCH 12/19 slub: Dynamically size kmalloc cache allocations

In reply to Christoph Lameter

kmalloc caches are statically defined and may take up a lot of space just because the sizes of the node array has to be dimensioned for the largest node count supported.

This patch makes the size of the kmem_cache structure dynamic throughout by creating a kmem_cache slab cache for the kmem_cache objects. The bootstrap occurs by allocating the initial one or two kmem_cache objects from the page allocator.

Signed-off-by Christoph Lameter <[email protected]>

---
 include/linux/slub_def.h |    7 -
 mm/slub.c                |  181 +++++++++++++++++++++++++++++++++++------------
 2 files changed, 139 insertions(+), 49 deletions(-)

include/linux/slub_def.h +2 -5

Index: linux-2.6/include/linux/slub_def.h
===================================================================
@@ -136,19 +136,16 @@ struct kmem_cache {
 
 #ifdef CONFIG_ZONE_DMA
 #define SLUB_DMA __GFP_DMA
-/* Reserve extra caches for potential DMA use */
-#define KMALLOC_CACHES (2 * SLUB_PAGE_SHIFT)
 #else
 /* Disable DMA functionality */
 #define SLUB_DMA (__force gfp_t)0
-#define KMALLOC_CACHES SLUB_PAGE_SHIFT
 #endif
 
 /*
  * We keep the general caches in an array of slab caches that are used for
  * 2^x bytes of allocations.
  */
-extern struct kmem_cache kmalloc_caches[KMALLOC_CACHES];
+extern struct kmem_cache *kmalloc_caches[SLUB_PAGE_SHIFT];
 
 /*
  * Sorry that the following has to be that ugly but some versions of GCC

@@ -213,7 +210,7 @@ static __always_inline struct kmem_cache
 	if (index == 0)
 		return NULL;
 
-	return &kmalloc_caches[index];
+	return kmalloc_caches[index];
 }
 
 void *kmem_cache_alloc(struct kmem_cache *, gfp_t);

mm/slub.c +137 ~~-44~~

302 lines not on this page · raw message

Christoph Lameter

author

2010-07-09 19:12 UTC #20 lore raw

PATCH 13/19 slub: Extract hooks for memory checkers from hotpaths

In reply to Christoph Lameter

Extract the code that memory checkers and other verification tools use from the hotpaths. Makes it easier to add new ones and reduces the disturbances of the hotpaths.

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |   49 ++++++++++++++++++++++++++++++++++++++-----------
 1 file changed, 38 insertions(+), 11 deletions(-)

mm/slub.c +38 ~~-11~~

Index: linux-2.6/mm/slub.c
===================================================================
@@ -793,6 +793,37 @@ static void trace(struct kmem_cache *s, 
 }
 
 /*
+ * Hooks for other subsystems that check memory allocations. In a typical
+ * production configuration these hooks all should produce no code at all.
+ */
+static inline int slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags)
+{
+	lockdep_trace_alloc(flags);
+	might_sleep_if(flags & __GFP_WAIT);
+
+	return should_failslab(s->objsize, flags, s->flags);
+}
+
+static inline void slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags, void *object)
+{
+	kmemcheck_slab_alloc(s, flags, object, s->objsize);
+	kmemleak_alloc_recursive(object, s->objsize, 1, s->flags, flags);
+}
+
+static inline void slab_free_hook(struct kmem_cache *s, void *x)
+{
+	kmemleak_free_recursive(x, s->flags);
+}
+
+static inline void slab_free_hook_irq(struct kmem_cache *s, void *object)
+{
+	kmemcheck_slab_free(s, object, s->objsize);
+	debug_check_no_locks_freed(object, s->objsize);
+	if (!(s->flags & SLAB_DEBUG_OBJECTS))
+		debug_check_no_obj_freed(object, s->objsize);
+}
+
+/*
  * Tracking of fully allocated slabs for debugging purposes.
  */
 static void add_full(struct kmem_cache_node *n, struct page *page)

@@ -1704,10 +1735,7 @@ static __always_inline void *slab_alloc(
 
 	gfpflags &= gfp_allowed_mask;
 
-	lockdep_trace_alloc(gfpflags);
-	might_sleep_if(gfpflags & __GFP_WAIT);
-
-	if (should_failslab(s->objsize, gfpflags, s->flags))
+	if (!slab_pre_alloc_hook(s, gfpflags))
 		return NULL;
 
 	local_irq_save(flags);

@@ -1726,8 +1754,7 @@ static __always_inline void *slab_alloc(
 	if (unlikely(gfpflags & __GFP_ZERO) && object)
 		memset(object, 0, s->objsize);
 
-	kmemcheck_slab_alloc(s, gfpflags, object, s->objsize);
-	kmemleak_alloc_recursive(object, s->objsize, 1, s->flags, gfpflags);
+	slab_post_alloc_hook(s, gfpflags, object);
 
 	return object;
 }

@@ -1856,13 +1883,13 @@ static __always_inline void slab_free(st
 	struct kmem_cache_cpu *c;
 	unsigned long flags;
 
-	kmemleak_free_recursive(x, s->flags);
+	slab_free_hook(s, x);
+
 	local_irq_save(flags);
 	c = __this_cpu_ptr(s->cpu_slab);
-	kmemcheck_slab_free(s, object, s->objsize);
-	debug_check_no_locks_freed(object, s->objsize);
-	if (!(s->flags & SLAB_DEBUG_OBJECTS))
-		debug_check_no_obj_freed(object, s->objsize);
+
+	slab_free_hook_irq(s, x);
+
 	if (likely(page == c->page && c->node >= 0)) {
 		set_freepointer(s, object, c->freelist);
 		c->freelist = object;

Christoph Lameter

author

2010-07-09 19:12 UTC #21 lore raw

PATCH 14/19 slub: Move gfpflag masking out of the hotpath

In reply to Christoph Lameter

Move the gfpflags masking into the hooks for checkers and into the slowpaths. gfpflag masking requires access to a global variable and thus adds an additional cacheline reference to the hotpaths.

If no hooks are active then the gfpflag masking will result in code that the compiler can toss out.

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |    5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

mm/slub.c +3 -2

Index: linux-2.6/mm/slub.c
===================================================================
@@ -798,6 +798,7 @@ static void trace(struct kmem_cache *s, 
  */
 static inline int slab_pre_alloc_hook(struct kmem_cache *s, gfp_t flags)
 {
+	flags &= gfp_allowed_mask;
 	lockdep_trace_alloc(flags);
 	might_sleep_if(flags & __GFP_WAIT);

@@ -806,6 +807,7 @@ static inline int slab_pre_alloc_hook(st
 
 static inline void slab_post_alloc_hook(struct kmem_cache *s, gfp_t flags, void *object)
 {
+	flags &= gfp_allowed_mask;
 	kmemcheck_slab_alloc(s, flags, object, s->objsize);
 	kmemleak_alloc_recursive(object, s->objsize, 1, s->flags, flags);
 }

@@ -1648,6 +1650,7 @@ static void *__slab_alloc(struct kmem_ca
 
 	/* We handle __GFP_ZERO in the caller */
 	gfpflags &= ~__GFP_ZERO;
+	gfpflags &= gfp_allowed_mask;
 
 	if (!c->page)
 		goto new_slab;

@@ -1733,8 +1736,6 @@ static __always_inline void *slab_alloc(
 	struct kmem_cache_cpu *c;
 	unsigned long flags;
 
-	gfpflags &= gfp_allowed_mask;
-
 	if (!slab_pre_alloc_hook(s, gfpflags))
 		return NULL;

Christoph Lameter

author

2010-07-09 19:12 UTC #22 lore raw

PATCH 15/19 SLUB: Add SLAB style per cpu queueing

In reply to Christoph Lameter

This patch adds SLAB style cpu queueing and uses a new way for
managing objects in the slabs using bitmaps. It uses a percpu queue so that
free operations can be properly buffered and a bitmap for managing the free/allocated state in the slabs. It uses slightly more memory (due to the need to place large bitmaps --sized a few words--in some slab pages) but in general does compete well in terms of space use. The storage format using bitmaps avoids the SLAB management structure that SLAB needs for each slab page and therefore metadata is more compact and easily fits into a cacheline.

The SLAB scheme of not touching the object during management is adopted. SLUB can now efficiently free and allocate cache cold objects.

The queueing scheme addresses also the issue that the free slowpath was taken too frequently.

This patch only implements staticallly sized per cpu queues.

Signed-off-by Christoph Lameter <[email protected]>

---
 include/linux/page-flags.h |    5 
 include/linux/slub_def.h   |   36 -
 init/Kconfig               |   13 
 mm/slub.c                  |  944 +++++++++++++++++++--------------------------
 4 files changed, 436 insertions(+), 562 deletions(-)

include/linux/slub_def.h +16 ~~-20~~

Index: linux-2.6/include/linux/slub_def.h
===================================================================
@@ -5,6 +5,7 @@
  * SLUB : A Slab allocator without object queues.
  *
  * (C) 2007 SGI, Christoph Lameter
+ * (C) 2010 Linux Foundation, Christoph Lameter
  */
 #include <linux/types.h>
 #include <linux/gfp.h>

@@ -14,33 +15,29 @@
 #include <linux/kmemleak.h>
 
 enum stat_item {
-	ALLOC_FASTPATH,		/* Allocation from cpu slab */
-	ALLOC_SLOWPATH,		/* Allocation by getting a new cpu slab */
-	FREE_FASTPATH,		/* Free to cpu slub */
-	FREE_SLOWPATH,		/* Freeing not to cpu slab */
-	FREE_FROZEN,		/* Freeing to frozen slab */
-	FREE_ADD_PARTIAL,	/* Freeing moves slab to partial list */
-	FREE_REMOVE_PARTIAL,	/* Freeing removes last object */
-	ALLOC_FROM_PARTIAL,	/* Cpu slab acquired from partial list */
-	ALLOC_SLAB,		/* Cpu slab acquired from page allocator */
-	ALLOC_REFILL,		/* Refill cpu slab from slab freelist */
+	ALLOC_FASTPATH,		/* Allocation from cpu queue */
+	ALLOC_SLOWPATH,		/* Allocation required refilling of queue */
+	FREE_FASTPATH,		/* Free to cpu queue */
+	FREE_SLOWPATH,		/* Required pushing objects out of the queue */
+	FREE_ADD_PARTIAL,	/* Freeing moved slab to partial list */
+	FREE_REMOVE_PARTIAL,	/* Freeing removed from partial list */
+	ALLOC_FROM_PARTIAL,	/* slab with objects acquired from partial */
+	ALLOC_SLAB,		/* New slab acquired from page allocator */
 	FREE_SLAB,		/* Slab freed to the page allocator */
-	CPUSLAB_FLUSH,		/* Abandoning of the cpu slab */
-	DEACTIVATE_FULL,	/* Cpu slab was full when deactivated */
-	DEACTIVATE_EMPTY,	/* Cpu slab was empty when deactivated */
-	DEACTIVATE_TO_HEAD,	/* Cpu slab was moved to the head of partials */
-	DEACTIVATE_TO_TAIL,	/* Cpu slab was moved to the tail of partials */
-	DEACTIVATE_REMOTE_FREES,/* Slab contained remotely freed objects */
+	QUEUE_FLUSH,		/* Flushing of the per cpu queue */
 	ORDER_FALLBACK,		/* Number of times fallback was necessary */
 	NR_SLUB_STAT_ITEMS };
 
+#define QUEUE_SIZE 50
+#define BATCH_SIZE 25
+
 struct kmem_cache_cpu {
-	void **freelist;	/* Pointer to first free per cpu object */
-	struct page *page;	/* The slab from which we are allocating */
-	int node;		/* The node of the page (or -1 for debug) */
 #ifdef CONFIG_SLUB_STATS
 	unsigned stat[NR_SLUB_STAT_ITEMS];
 #endif
+	int objects;		/* Number of objects available */
+	int node;		/* The node of the page (or -1 for debug) */
+	void *object[QUEUE_SIZE];		/* List of objects */
 };
 
 struct kmem_cache_node {

@@ -72,7 +69,6 @@ struct kmem_cache {
 	unsigned long flags;
 	int size;		/* The size of an object including meta data */
 	int objsize;		/* The size of an object without meta data */
-	int offset;		/* Free pointer offset. */
 	struct kmem_cache_order_objects oo;
 
 	/* Allocation and freeing of slabs */

mm/slub.c +414 ~~-530~~

1,478 lines not on this page · raw message

include/linux/page-flags.h +0 -5

Index: linux-2.6/include/linux/page-flags.h
===================================================================
@@ -125,9 +125,6 @@ enum pageflags {
 
 	/* SLOB */
 	PG_slob_free = PG_private,
-
-	/* SLUB */
-	PG_slub_frozen = PG_active,
 };
 
 #ifndef __GENERATING_BOUNDS_H

@@ -213,8 +210,6 @@ PAGEFLAG(SwapBacked, swapbacked) __CLEAR
 
 __PAGEFLAG(SlobFree, slob_free)
 
-__PAGEFLAG(SlubFrozen, slub_frozen)
-
 /*
  * Private page markings that may be used by the filesystem that owns the page
  * for its own purposes.

init/Kconfig +6 -7

Index: linux-2.6/init/Kconfig
===================================================================
@@ -1087,14 +1087,13 @@ config SLAB
 	  per cpu and per node queues.
 
 config SLUB
-	bool "SLUB (Unqueued Allocator)"
+	bool "SLUB"
 	help
-	   SLUB is a slab allocator that minimizes cache line usage
-	   instead of managing queues of cached objects (SLAB approach).
-	   Per cpu caching is realized using slabs of objects instead
-	   of queues of objects. SLUB can use memory efficiently
-	   and has enhanced diagnostics. SLUB is the default choice for
-	   a slab allocator.
+	   SLUB is a slab allocator that minimizes metadata and provides
+	   a clean implementation that is faster than SLAB. SLUB has a
+	   single per cpu queue and uses a bit map to manage objects in
+	   slabs. SLUB can use memory more efficiently and has enhanced
+	   diagnostic and resiliency features compared with SLAB.
 
 config SLOB
 	depends on EMBEDDED

Christoph Lameter

author

2010-07-09 19:12 UTC #23 lore raw

PATCH 16/19 slub: Resize the new cpu queues

In reply to Christoph Lameter

Allow resizing of cpu queue and batch size. This is done in the basic steps that are also followed by SLAB.

The statically allocated per cpu areas are removed since the per cpu allocator is already available when kmem_cache_init is called. We can dynamically size the per cpu data during bootstrap.

Careful: This means that the ->cpu pointer is becoming volatile. References to the ->cpu pointer either

A. Occur with interrupts disabled. This guarantees that nothing on the
processor itself interferes. This only serializes access to a single processor specific area.

B. Occur with slub_lock taken for operations on all per cpu areas.
Taking the slub_lock guarantees that no resizing operation will occur while accessing the percpu areas. The data in the percpu areas is volatile even with slub_lock since the alloc and free functions do not take slub_lock and will operate on fields of kmem_cache_cpu.

C. Are racy: This is true for the statistics. The ->cpu pointer must always
point to a valid kmem_cache_cpu area.

Signed-off-by Christoph Lameter <[email protected]>

---
 include/linux/slub_def.h |    9 --
 mm/slub.c                |  197 +++++++++++++++++++++++++++++++++++++++++------
 2 files changed, 177 insertions(+), 29 deletions(-)

include/linux/slub_def.h +4 -5

Index: linux-2.6/include/linux/slub_def.h
===================================================================
@@ -28,16 +28,13 @@ enum stat_item {
 	ORDER_FALLBACK,		/* Number of times fallback was necessary */
 	NR_SLUB_STAT_ITEMS };
 
-#define QUEUE_SIZE 50
-#define BATCH_SIZE 25
-
 struct kmem_cache_cpu {
 #ifdef CONFIG_SLUB_STATS
 	unsigned stat[NR_SLUB_STAT_ITEMS];
 #endif
 	int objects;		/* Number of objects available */
 	int node;		/* The node of the page (or -1 for debug) */
-	void *object[QUEUE_SIZE];		/* List of objects */
+	void *object[];		/* Dynamic alloc will allow larger sizes  */
 };
 
 struct kmem_cache_node {

@@ -64,12 +61,14 @@ struct kmem_cache_order_objects {
  * Slab cache management.
  */
 struct kmem_cache {
-	struct kmem_cache_cpu *cpu_slab;
+	struct kmem_cache_cpu *cpu;
 	/* Used for retriving partial slabs etc */
 	unsigned long flags;
 	int size;		/* The size of an object including meta data */
 	int objsize;		/* The size of an object without meta data */
 	struct kmem_cache_order_objects oo;
+	int queue;		/* per cpu queue size */
+	int batch;		/* batch size */
 
 	/* Allocation and freeing of slabs */
 	struct kmem_cache_order_objects max;

mm/slub.c +173 ~~-24~~

388 lines not on this page · raw message

Christoph Lameter

author

2010-07-09 19:12 UTC #24 lore raw

PATCH 17/19 SLUB: Get rid of useless function count_free()

In reply to Christoph Lameter

count_free() == available()

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |   11 +++--------
 1 file changed, 3 insertions(+), 8 deletions(-)

mm/slub.c +3 -8

Index: linux-2.6/mm/slub.c
===================================================================
@@ -1652,11 +1652,6 @@ static inline int node_match(struct kmem
 	return 1;
 }
 
-static int count_free(struct page *page)
-{
-	return available(page);
-}
-
 static unsigned long count_partial(struct kmem_cache_node *n,
 					int (*get_count)(struct page *))
 {

@@ -1705,7 +1700,7 @@ slab_out_of_memory(struct kmem_cache *s,
 		if (!n)
 			continue;
 
-		nr_free  = count_partial(n, count_free);
+		nr_free  = count_partial(n, available);
 		nr_slabs = node_nr_slabs(n);
 		nr_objs  = node_nr_objs(n);

@@ -3835,7 +3830,7 @@ static ssize_t show_slab_objects(struct 
 			x = atomic_long_read(&n->total_objects);
 		else if (flags & SO_OBJECTS)
 			x = atomic_long_read(&n->total_objects) -
-				count_partial(n, count_free);
+				count_partial(n, available);
 
 			else
 				x = atomic_long_read(&n->nr_slabs);

@@ -4717,7 +4712,7 @@ static int s_show(struct seq_file *m, vo
 		nr_partials += n->nr_partial;
 		nr_slabs += atomic_long_read(&n->nr_slabs);
 		nr_objs += atomic_long_read(&n->total_objects);
-		nr_free += count_partial(n, count_free);
+		nr_free += count_partial(n, available);
 	}
 
 	nr_inuse = nr_objs - nr_free;

Christoph Lameter

author

2010-07-09 19:12 UTC #25 lore raw

PATCH 18/19 SLUB: Remove MAX_OBJS limitation

In reply to Christoph Lameter

There is no need anymore for the "inuse" field in the page struct. Extend the objects field to 32 bit allowing a practically unlimited number of objects.

Signed-off-by Christoph Lameter <[email protected]>

---
 include/linux/mm_types.h |    5 +----
 mm/slub.c                |    7 -------
 2 files changed, 1 insertion(+), 11 deletions(-)

include/linux/mm_types.h +1 -4

Index: linux-2.6/include/linux/mm_types.h
===================================================================
@@ -40,10 +40,7 @@ struct page {
 					 * to show when page is mapped
 					 * & limit reverse map searches.
 					 */
-		struct {		/* SLUB */
-			u16 inuse;
-			u16 objects;
-		};
+		u32 objects;		/* SLUB */
 	};
 	union {
 	    struct {

mm/slub.c +0 -7

Index: linux-2.6/mm/slub.c
===================================================================
@@ -145,7 +145,6 @@ static inline int kmem_cache_debug(struc
 
 #define OO_SHIFT	16
 #define OO_MASK		((1 << OO_SHIFT) - 1)
-#define MAX_OBJS_PER_PAGE	65535 /* since page.objects is u16 */
 
 /* Internal SLUB flags */
 #define __OBJECT_POISON		0x80000000UL /* Poison object */

@@ -768,9 +767,6 @@ static int verify_slab(struct kmem_cache
 			max_objects = ((void *)page->freelist - start) / s->size;
 	}
 
-	if (max_objects > MAX_OBJS_PER_PAGE)
-		max_objects = MAX_OBJS_PER_PAGE;
-
 	if (page->objects != max_objects) {
 		slab_err(s, page, "Wrong number of objects. Found %d but "
 			"should be %d", page->objects, max_objects);

@@ -1984,9 +1980,6 @@ static inline int slab_order(int size, i
 	int rem;
 	int min_order = slub_min_order;
 
-	if ((PAGE_SIZE << min_order) / size > MAX_OBJS_PER_PAGE)
-		return get_order(size * MAX_OBJS_PER_PAGE) - 1;
-
 	for (order = max(min_order,
 				fls(min_objects * size - 1) - PAGE_SHIFT);
 			order <= max_order; order++) {

Christoph Lameter

author

2010-07-09 19:12 UTC #26 lore raw

PATCH 19/19 slub: Drop allocator announcement

In reply to Christoph Lameter

People get confused because of the output and some of the items listed no longer have the same relevance in the queued form of SLUB.

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |    8 --------
 1 file changed, 8 deletions(-)

mm/slub.c +0 -8

Index: linux-2.6/mm/slub.c
===================================================================
@@ -3153,14 +3153,6 @@ void __init kmem_cache_init(void)
 #ifdef CONFIG_SMP
 	register_cpu_notifier(&slab_notifier);
 #endif
-
-	printk(KERN_INFO
-		"SLUB: Genslabs=%d, HWalign=%d, Order=%d-%d, MinObjects=%d,"
-		" CPUs=%d, Nodes=%d\n",
-		caches, cache_line_size(),
-		slub_min_order, slub_max_order, slub_min_objects,
-		nr_cpu_ids, nr_node_ids);
-
 }
 
 void __init kmem_cache_init_late(void)

Heinz Diehl

2010-07-10 19:57 UTC #27 lore raw

In reply to Christoph Lameter

Quoted 3 lines · Christoph Lameter

The following patchset cleans some pieces up and then equips SLUB with per cpu queues that work similar to SLABs queues. With that approach SLUB wins significantly in hackbench and improves also on tcp_rr.

The patchset applies cleanly, however compilation fails with

[....]
mm/slub.c: In function ‘alloc_kmem_cache_cpus’:
mm/slub.c:2093: error: negative width in bit-field ‘<anonymous>’ make[1]: *** [mm/slub.o] Error 1
make: *** [mm] Error 2
make: *** Waiting for unfinished jobs....
[....]

Christoph Lameter

author

2010-07-12 15:15 UTC #28 lore raw

Quoted 8 lines · Heinz Diehl

Quoted 3 lines · Christoph Lameter
The following patchset cleans some pieces up and then equips SLUB with per cpu queues that work similar to SLABs queues. With that approach SLUB wins significantly in hackbench and improves also on tcp_rr.
The patchset applies cleanly, however compilation fails with

Quoted 7 lines

[....]
mm/slub.c: In function ‘alloc_kmem_cache_cpus’:
mm/slub.c:2093: error: negative width in bit-field ‘<anonymous>’ make[1]: *** [mm/slub.o] Error 1
make: *** [mm] Error 2
make: *** Waiting for unfinished jobs....
[....]

You need a sufficient PERCPU_DYNAMIC_EARLY_SIZE to be configured. What platform is this? Tejon: You suggested the BUILD_BUG_ON(). How can he increase the early size?

Heinz Diehl

2010-07-12 16:39 UTC #29 lore raw

Quoted 2 lines · Christoph Lameter

You need a sufficient PERCPU_DYNAMIC_EARLY_SIZE to be configured. What platform is this?

This is an AMD Phenom II X4-905e with 8GB RAM and an (heavily modified) opensuse 11.1 64-bit with kernel 2.6.35-rc4-git4 (vanilla from kernel.org, no distribution kernel). Dmesg is attached.

Thanks,
Heinz.

Christoph Lameter

author

2010-07-12 17:04 UTC #30 lore raw

Quoted 8 lines · Heinz Diehl

Quoted 2 lines · Christoph Lameter
You need a sufficient PERCPU_DYNAMIC_EARLY_SIZE to be configured. What platform is this?
This is an AMD Phenom II X4-905e with 8GB RAM and an (heavily modified) opensuse 11.1 64-bit with kernel 2.6.35-rc4-git4 (vanilla from kernel.org, no distribution kernel). Dmesg is attached.

Can you get us the config file. What is the value of PERCPU_DYMAMIC_EARLY_SIZE?

I have run this on x86 for a long time. Why does the percpu subsystem have a lower PERCPU_DYNAMIC_EARLY_SIZE on Heinzes system?

Heinz Diehl

2010-07-13 13:56 UTC #31 lore raw

Quoted 2 lines · Christoph Lameter

Can you get us the config file. What is the value of PERCPU_DYMAMIC_EARLY_SIZE?

My .config file is attached. I don't know how to find out what value PERCPU_DYNAMIC_EARLY_SIZE is actually on, how could I do that? There's no such thing in my .config.

Christoph Lameter

author

2010-07-14 02:04 UTC #32 lore raw

Quoted 8 lines · Heinz Diehl

Quoted 2 lines · Christoph Lameter
Can you get us the config file. What is the value of PERCPU_DYMAMIC_EARLY_SIZE?
My .config file is attached. I don't know how to find out what value PERCPU_DYNAMIC_EARLY_SIZE is actually on, how could I do that? There's no such thing in my .config.

I dont see anything in there at first glance that would cause slub to increase its percpu usage. This is straight upstream?

Try to just comment out the BUILD_BUG_ON. I had it misfire before and fixed the formulae to no longer give false positives. Maybe that is another case. Tejun wanted that but never was able to give me an exact formular to check for.

At the Ottawa Linux Symposium right now so responses may be delayed. Hotels Internet connection keeps getting clogged for some reason.

Tejun Heo

maintainer

2010-07-14 11:51 UTC #33 lore raw

Hello,

Quoted 2 lines · Christoph Lameter

I dont see anything in there at first glance that would cause slub to increase its percpu usage. This is straight upstream?

It's basically checking constant expressions there and PERCPU_DYNAMIC_EARLY_SIZE is defined as 12k, so slub is thinking that it's gonna use more memory on that build.

Quoted 4 lines

Try to just comment out the BUILD_BUG_ON. I had it misfire before and fixed the formulae to no longer give false positives. Maybe that is another case. Tejun wanted that but never was able to give me an exact formular to check for.

Yeah, unfortunately, due to alignment requirements, it can't be determined with accuracy. We'll just have to size it sufficiently.

Quoted 2 lines

At the Ottawa Linux Symposium right now so responses may be delayed. Hotels Internet connection keeps getting clogged for some reason.

I'm in suse labs conf until next week so I don't think I'll be doing much till then either.

Thanks.

Signature · 7 lines

tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Heinz Diehl

2010-07-14 14:25 UTC #34 lore raw

In reply to Christoph Lameter

Quoted 2 lines · Christoph Lameter

I dont see anything in there at first glance that would cause slub to increase its percpu usage. This is straight upstream?

Yes ,it's plain vanilla 2.6.35-rc4/-rc5 from kernel.org.

Quoted 1 line

Try to just comment out the BUILD_BUG_ON.

I first bumped it up to 24k, but that was obviously not enough, so I commented out the BUILD_BUG_ON which triggers the build error. Now It builds fine, and I'll do some testing.

Thanks,
Heinz.

David Rientjes

reviewer

2010-07-14 20:22 UTC #35 lore raw

In reply to Christoph Lameter

Quoted 10 lines · Christoph Lameter

Quoted 6 lines
Quoted 2 lines
Can you get us the config file. What is the value of PERCPU_DYMAMIC_EARLY_SIZE?
My .config file is attached. I don't know how to find out what value PERCPU_DYNAMIC_EARLY_SIZE is actually on, how could I do that? There's no such thing in my .config.
I dont see anything in there at first glance that would cause slub to increase its percpu usage. This is straight upstream?

The problem is that he has CONFIG_NODES_SHIFT=10 and struct kmem_cache has an array of struct kmem_cache_node pointers with MAX_NUMNODES entries which blows its size up to over 8K. That's probably overkill for his quad-core 8GB AMD, so I'd recommend lowering CONFIG_NODES_SHIFT to 6.

Tejun Heo

maintainer

2010-07-14 11:46 UTC #36 lore raw

In reply to Christoph Lameter

Hello,

Quoted 3 lines · Christoph Lameter

You need a sufficient PERCPU_DYNAMIC_EARLY_SIZE to be configured. What platform is this? Tejon: You suggested the BUILD_BUG_ON(). How can he increase the early size?

The size is determined by PERCPU_DYNAMIC_EARLY_SIZE, so bumping it up should do it but it would probably be wiser to bump PERCPU_DYNAMIC_RESERVE too. PERCPU_DYNAMIC_EARLY_SIZE is currently 12k. How high should it be?

Thanks.

Signature · 7 lines

tejun

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

David Rientjes

reviewer

2010-07-14 22:26 UTC #37 lore raw

In reply to Christoph Lameter

Quoted 36 lines · Christoph Lameter

SLUB+Q also wins against SLAB in netperf:
Script:
#!/bin/bash
TIME=60 # seconds
HOSTNAME=localhost # netserver
NR_CPUS=$(grep ^processor /proc/cpuinfo | wc -l)
echo NR_CPUS=$NR_CPUS
run_netperf() {
for i in $(seq 1 $1); do
netperf -H $HOSTNAME -t TCP_RR -l $TIME &
done
}
ITERATIONS=0
while [ $ITERATIONS -lt 12 ]; do
RATE=0
ITERATIONS=$[$ITERATIONS + 1]
THREADS=$[$NR_CPUS * $ITERATIONS]
RESULTS=$(run_netperf $THREADS | grep -v '[a-zA-Z]' | awk '{ print $6 }')
for j in $RESULTS; do
RATE=$[$RATE + ${j/.*}]
done
echo threads=$THREADS rate=$RATE
done
Dell Dual Quad Penryn on Linux 2.6.35-rc4
Loop counts: Larger is better.

Quoted 14 lines

Threads SLAB SLUB+Q %
8 690869 714788 + 3.4
16 680295 711771 + 4.6
24 672677 703014 + 4.5
32 676780 703914 + 4.0
40 668458 699806 + 4.6
48 667017 698908 + 4.7
56 671227 696034 + 3.6
64 667956 696913 + 4.3
72 668332 694931 + 3.9
80 667073 695658 + 4.2
88 682866 697077 + 2.0
96 668089 694719 + 3.9

I see you're using my script for collecting netperf TCP_RR benchmark data, thanks very much for looking into this workload for slab allocator performance!

There are a couple differences between how you're using it compared to how I showed the initial regression between slab and slub, however: you're using localhost for your netserver which isn't representative of a real networking round-robin workload and you're using a smaller system with eight cores. We never measured a _significant_ performance problem with slub compared to slab with four or eight cores, the problem only emerges on larger systems.

When running this patchset on two (client and server running netperf-2.4.5) four 2.2GHz quad-core AMD processors with 64GB of memory, here's the results:

		threads	SLAB	SLUB+Q	diff
		16	205580	179109	-12.9%
		32	264024	215613	-18.3%
		48	286175	237036	-17.2%
		64	305309	253222	-17.1%
		80	308248	243848	-20.9%
		96	299845	243848	-18.7%
		112	305560	259427	-15.1%
		128	312668	263803	-15.6%
		144	329671	271335	-17.7%
		160	318737	280290	-12.1%
		176	325295	287918	-11.5%
		192	333356	287995	-13.6%

If you'd like to add statistics to your patchset that are enabled with CONFIG_SLUB_STATS, I'd be happy to run it on this setup and collect more data for you.

Christoph Lameter

author

2010-07-15 20:20 UTC #38 lore raw

Quoted 7 lines · David Rientjes

There are a couple differences between how you're using it compared to how I showed the initial regression between slab and slub, however: you're using localhost for your netserver which isn't representative of a real networking round-robin workload and you're using a smaller system with eight cores. We never measured a _significant_ performance problem with slub compared to slab with four or eight cores, the problem only emerges on larger systems.

Larger systems would more NUMA support than is present in the current patches.

Quoted 3 lines

When running this patchset on two (client and server running netperf-2.4.5) four 2.2GHz quad-core AMD processors with 64GB of memory, here's the results:

What is their NUMA topology? I dont have anything beyond two nodes here.

David Rientjes

reviewer

2010-07-15 20:31 UTC #39 lore raw

Quoted 6 lines · Christoph Lameter

Quoted 3 lines
When running this patchset on two (client and server running netperf-2.4.5) four 2.2GHz quad-core AMD processors with 64GB of memory, here's the results:
What is their NUMA topology? I dont have anything beyond two nodes here.

These two machines happen to have four 16GB nodes with asymmetrical distances:

# cat /sys/devices/system/node/node*/distance
10 20 20 30
20 10 20 20
20 20 10 20
30 20 20 10

David Rientjes

reviewer

2010-07-14 23:52 UTC #40 lore raw

In reply to Christoph Lameter

Quoted 2 lines · Christoph Lameter

The following patchset cleans some pieces up and then equips SLUB with per cpu queues that work similar to SLABs queues.

Pekka, I think patches 4-8 could be applied to your tree now, they're relatively unchanged from what's been posted before. (I didn't ack patch 9 because I think it makes slab_lock() -> slab_unlock() matching more difficult with little win, but I don't feel strongly about it.)

I'd also consider patch 7 for 2.6.35-rc6 (and -stable).

Pekka Enberg

2010-07-16 08:23 UTC #41 lore raw

Quoted 6 lines · David Rientjes

Quoted 2 lines · Christoph Lameter
The following patchset cleans some pieces up and then equips SLUB with per cpu queues that work similar to SLABs queues.
Pekka, I think patches 4-8 could be applied to your tree now, they're relatively unchanged from what's been posted before. (I didn't ack patch 9 because I think it makes slab_lock() -> slab_unlock() matching more difficult with little win, but I don't feel strongly about it.)

Yup, I applied 4-8. Thanks guys!

Quoted 1 line

I'd also consider patch 7 for 2.6.35-rc6 (and -stable).

It's an obvious bug fix but is it triggered in practice? Is there a bugzilla report for that?

David Rientjes

reviewer

2010-07-16 09:02 UTC #42 lore raw

Quoted 5 lines · Pekka Enberg

Quoted 1 line
I'd also consider patch 7 for 2.6.35-rc6 (and -stable).
It's an obvious bug fix but is it triggered in practice? Is there a bugzilla report for that?

Let's ask Benjamin who initially reported the problem with arch_initcall whether or not this is rc (and stable) material.

For reference, we're talking about the sysfs_slab_remove() check on slab_state to prevent the WARN in the kobject code you hit with its fix below:

From: Christoph Lameter <[email protected]>

slub: Allow removal of slab caches during boot

If a slab cache is removed before we have setup sysfs then simply skip over the sysfs handling.

Cc Benjamin Herrenschmidt <[email protected]>

Cc Roland Dreier <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

---
 mm/slub.c |    7 +++++++
 1 file changed, 7 insertions(+)

mm/slub.c +7 -0

Index: linux-2.6/mm/slub.c
===================================================================
@@ -4507,6 +4507,13 @@ static int sysfs_slab_add(struct kmem_ca
 
 static void sysfs_slab_remove(struct kmem_cache *s)
 {
+	if (slab_state < SYSFS)
+		/*
+		 * Sysfs has not been setup yet so no need to remove the
+		 * cache from sysfs.
+		 */
+		return;
+
 	kobject_uevent(&s->kobj, KOBJ_REMOVE);
 	kobject_del(&s->kobj);
 	kobject_put(&s->kobj);

Benjamin Herrenschmidt

2010-07-19 00:18 UTC #43 lore raw

Quoted 8 lines · David Rientjes

Quoted 5 lines · Pekka Enberg
Quoted 1 line
I'd also consider patch 7 for 2.6.35-rc6 (and -stable).
It's an obvious bug fix but is it triggered in practice? Is there a bugzilla report for that?

Quoted 6 lines

Let's ask Benjamin who initially reported the problem with arch_initcall whether or not this is rc (and stable) material.
For reference, we're talking about the sysfs_slab_remove() check on slab_state to prevent the WARN in the kobject code you hit with its fix below:

The only case where I reproduce that is an in-house kernel port that we haven't published yet so it doesn't have to be -stable material as far as I'm concerned.

Cheers,
Ben.

Quoted 36 lines

From: Christoph Lameter <[email protected]>
slub: Allow removal of slab caches during boot
If a slab cache is removed before we have setup sysfs then simply skip over the sysfs handling.
Cc Benjamin Herrenschmidt <[email protected]>
Cc Roland Dreier <[email protected]>
Signed-off-by Christoph Lameter <[email protected]>
---
 mm/slub.c |    7 +++++++
 1 file changed, 7 insertions(+)
Index: linux-2.6/mm/slub.c
===================================================================
mm/slub.c +7 -1
@@ -4507,6 +4507,13 @@ static int sysfs_slab_add(struct kmem_ca
 
 static void sysfs_slab_remove(struct kmem_cache *s)
 {
+	if (slab_state < SYSFS)
+		/*
+		 * Sysfs has not been setup yet so no need to remove the
+		 * cache from sysfs.
+		 */
+		return;
+
 	kobject_uevent(&s->kobj, KOBJ_REMOVE);
 	kobject_del(&s->kobj);
 	kobject_put(&s->kobj);
 
--

Quoted 4 lines

To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>