Skip to contentSkip to conversation

SLUB: Improve allocpercpu to reduce per cpu access overhead

RFC PATCH In review Christoph Lameter 6 participants 62 messages Nov 1, 2007 – Nov 12, 2007

Subsystems (7)
Networking Memory management MM core Generic include/asm header files +3 more subsystems
lore
Share & tools
lore

Christoph Lameter

author
#0 lore raw

cover SLUB: Improve allocpercpu to reduce per cpu access overhead

This patch increases the speed of the SLUB fastpath by improving the per cpu allocator and makes it usable for SLUB.

Currently allocpercpu manages arrays of pointer to per cpu objects. This means that is has to allocate the arrays and then populate them as needed with objects. Although these objects are called per cpu objects they cannot be handled in the same way as per cpu objects by adding the per cpu offset of the respective cpu.

The patch here changes that. We create a small memory pool in the percpu area and allocate from there if alloc per cpu is called. As a result we do not need the per cpu pointer arrays for each object. This reduces memory usage and also the cache foot print of allocpercpu users. Also the per cpu objects for a single processor are tightly packed next to each other decreasing cache footprint even further and making it possible to access multiple objects in the same cacheline.

SLUB has the same mechanism implemented. After fixing up the alloccpu stuff we throw the SLUB method out and use the new allocpercpu handling. Then we optimize allocpercpu addressing by adding a new function

	this_cpu_ptr()

that allows the determination of the per cpu pointer for the current processor in an more efficient way on many platforms.

This increases the speed of SLUB (and likely other kernel subsystems that benefit from the allocpercpu enhancements):

SLAB SLUB SLUB+ SLUB-o SLUB-a

   8    96      86      45      44      38	3 *
  16    84      92      49      48      43	2 *
  32    84      106     61      59      53	+++
  64    102     129     82      88      75	++
 128    147     226     188     181     176	-
 256    200     248     207     285     204	=
 512    300     301     260     209     250	+
1024    416     440     398     264     391	++
2048    720     542     530     390     511	+++
4096    1254    342     342     336     376	3 *

alloc/free test

      SLAB    SLUB    SLUB+   SLUB-o	SLUB-a
      137-146 151     68-72   68-74	56-58	3 *

Note: The per cpu optimization are only half way there because of the screwed up way that x86_64 handles its cpu area that causes addditional cycles to be spend by retrieving a pointer from memory and adding it to the address. The i386 code is much less cycle intensive being able to get to per cpu data using a segment prefix and if we can get that to work on x86_64 then we may be able to get the cycle count for the fastpath down to 20-30 cycles.

author
#1 lore raw

PATCH 1/7 allocpercpu: Make it a true per cpu allocator by allocating from a per cpu array

Currently each call to alloc_percpu allocates an array of pointer to objects. For each operation on a percpu structure we need to follow a pointer from that map. Usually a processor used only the entry for its own processor id in that array. The rest of the bytes in the cacheline are not needed. This repeats itself for each and every per cpu array in use.

Moreover the result of alloc_percpu is not a variable that can be handled like a regular per cpu variable.

The approach here changes the way allocpercpu is done. Objects are placed in preallocated per cpu areas that are indexed via the existing per cpu array of pointers. So we have a single array of pointer to per cpu areas that is used by all per cpu operations. The data is placed tightly next to each other for each processor so that the likelyhood of a single cache line covering data for multiple needs is increased. The cache footprint of the allocpercpu operations sinks dramatically. Some processors have the ability to map the per cpu area of the current processor in a special way so that variables in that area can be reached very efficiently. It is rather typical that a processor only uses its own per processor area. On many architectures the indexing via the per cpu array can then be completely bypassed.

The size of the per cpu alloc area is defined to be 32k per processor for now.

Another advantage of this approach is that the onlining and offlining of the per cpu items is handled in a global way. On onlining a cpu all objects become present without callbacks. Similarly on offlining a cpu all per cpu objects vanish without the need of callbacks. Callbacks may still be needed to do preparation and cleanup of the data areas but the freeing and allocation of the per cpu areas no longer needs to be done by the subsystems.

Signed-off-by Christoph Lameter <[email protected]>
---
 include/linux/percpu.h |   14 +---
 include/linux/vmstat.h |    2 
 mm/allocpercpu.c       |  163 +++++++++++++++++++++++++++++++++++++++++++------
 mm/vmstat.c            |    1 
 4 files changed, 152 insertions(+), 28 deletions(-)
mm/allocpercpu.c +144 -19
Index: linux-2.6/mm/allocpercpu.c
===================================================================
@@ -2,10 +2,140 @@
  * linux/mm/allocpercpu.c
  *
  * Separated from slab.c August 11, 2006 Christoph Lameter <[email protected]>
+ *
+ * (C) 2007 SGI, Christoph Lameter <[email protected]>
+ * 	Basic implementation with allocation and free from a dedicated per
+ * 	cpu area.
+ *
+ * The per cpu allocator allows allocation of memory from a statically
+ * allocated per cpu array and consists of cells of UNIT_SIZE. A byte array
+ * is used to describe the state of each of the available units that can be
+ * allocated via cpu_alloc() and freed via cpu_free(). The possible states are:
+ *
+ * FREE	= The per cpu unit is not allocated
+ * USED	= The per cpu unit is allocated and more units follow.
+ * END	= The last per cpu unit used for an allocation (needed to
+ * 	  establish the size of the allocation on free)
+ *
+ * The per cpu allocator is typically used to allocate small sized object from 8 to 32
+ * bytes and it is rarely used. Allocation is looking for the first available object
+ * in the cpu_alloc_map. If the allocator would be used frequently with varying sizes
+ * of objects then we may end up with fragmentation.
  */
 #include <linux/mm.h>
 #include <linux/module.h>
 
+/*
+ * Maximum allowed per cpu data per cpu
+ */
+#define PER_CPU_ALLOC_SIZE 32768
+
+#define UNIT_SIZE sizeof(unsigned long long)
+#define UNITS_PER_CPU (PER_CPU_ALLOC_SIZE / UNIT_SIZE)
+
+enum unit_type { FREE, END, USED };
+
+static u8 cpu_alloc_map[UNITS_PER_CPU] = { 1, };
+static DEFINE_SPINLOCK(cpu_alloc_map_lock);
+static DEFINE_PER_CPU(int, cpu_area)[UNITS_PER_CPU];
+
+#define CPU_DATA_OFFSET ((unsigned long)&per_cpu__cpu_area)
+
+/*
+ * How many units are needed for an object of a given size
+ */
+static int size_to_units(unsigned long size)
+{
+	return DIV_ROUND_UP(size, UNIT_SIZE);
+}
+
+/*
+ * Mark an object as used in the cpu_alloc_map
+ *
+ * Must hold cpu_alloc_map_lock
+ */
+static void set_map(int start, int length)
+{
+	cpu_alloc_map[start + length - 1] = END;
+	if (length > 1)
+		memset(cpu_alloc_map + start, USED, length - 1);
+}
+
+/*
+ * Mark an area as freed.
+ *
+ * Must hold cpu_alloc_map_lock
+ *
+ * Return the number of units taken up by the object freed.
+ */
+static int clear_map(int start)
+{
+	int units = 0;
+
+	while (cpu_alloc_map[start + units] == USED) {
+		cpu_alloc_map[start + units] = FREE;
+		units++;
+	}
+	BUG_ON(cpu_alloc_map[start] != END);
+	cpu_alloc_map[start] = FREE;
+	return units + 1;
+}
+
+/*
+ * Allocate an object of a certain size
+ *
+ * Returns a per cpu pointer that must not be directly used.
+ */
+static void *cpu_alloc(unsigned long size)
+{
+	unsigned long start = 0;
+	int units = size_to_units(size);
+	unsigned end;
+
+	spin_lock(&cpu_alloc_map_lock);
+	do {
+		while (start < UNITS_PER_CPU &&
+				cpu_alloc_map[start] != FREE)
+			start++;
+		if (start == UNITS_PER_CPU)
+			return NULL;
+
+		end = start + 1;
+		while (end < UNITS_PER_CPU && end - start < units &&
+				cpu_alloc_map[end] == FREE)
+			end++;
+		if (end - start == units)
+			break;
+		start = end;
+	} while (1);
+
+	set_map(start, units);
+	__count_vm_events(ALLOC_PERCPU, units * UNIT_SIZE);
+	spin_unlock(&cpu_alloc_map_lock);
+	return (void *)(start * UNIT_SIZE + CPU_DATA_OFFSET);
+}
+
+/*
+ * Free an object. The pointer must be a per cpu pointer allocated
+ * via cpu_alloc.
+ */
+static inline void cpu_free(void *pcpu)
+{
+	unsigned long start = (unsigned long)pcpu;
+	int index;
+	int units;
+
+	BUG_ON(start < CPU_DATA_OFFSET);
+	index = (start - CPU_DATA_OFFSET) / UNIT_SIZE;
+	BUG_ON(cpu_alloc_map[index] == FREE ||
+			index >= UNITS_PER_CPU);
+
+	spin_lock(&cpu_alloc_map_lock);
+	units = clear_map(index);
+	__count_vm_events(ALLOC_PERCPU, -units * UNIT_SIZE);
+	spin_unlock(&cpu_alloc_map_lock);
+}
+
 /**
  * percpu_depopulate - depopulate per-cpu data for given cpu
  * @__pdata: per-cpu data to depopulate

@@ -16,10 +146,10 @@
  */
 void percpu_depopulate(void *__pdata, int cpu)
 {
-	struct percpu_data *pdata = __percpu_disguise(__pdata);
-
-	kfree(pdata->ptrs[cpu]);
-	pdata->ptrs[cpu] = NULL;
+	/*
+	 * Nothing to do here. Removal can only be effected for all
+	 * per cpu areas of a cpu at once.
+	 */
 }
 EXPORT_SYMBOL_GPL(percpu_depopulate);
 

@@ -30,9 +160,9 @@ EXPORT_SYMBOL_GPL(percpu_depopulate);
  */
 void __percpu_depopulate_mask(void *__pdata, cpumask_t *mask)
 {
-	int cpu;
-	for_each_cpu_mask(cpu, *mask)
-		percpu_depopulate(__pdata, cpu);
+	/*
+	 * Nothing to do
+	 */
 }
 EXPORT_SYMBOL_GPL(__percpu_depopulate_mask);
 

@@ -49,15 +179,11 @@ EXPORT_SYMBOL_GPL(__percpu_depopulate_ma
  */
 void *percpu_populate(void *__pdata, size_t size, gfp_t gfp, int cpu)
 {
-	struct percpu_data *pdata = __percpu_disguise(__pdata);
-	int node = cpu_to_node(cpu);
+	int pdata = (unsigned long)__percpu_disguise(__pdata);
+	void *p = (void *)per_cpu_offset(cpu) + pdata;
 
-	BUG_ON(pdata->ptrs[cpu]);
-	if (node_online(node))
-		pdata->ptrs[cpu] = kmalloc_node(size, gfp|__GFP_ZERO, node);
-	else
-		pdata->ptrs[cpu] = kzalloc(size, gfp);
-	return pdata->ptrs[cpu];
+	memset(p, 0, size);
+	return p;
 }
 EXPORT_SYMBOL_GPL(percpu_populate);
 

@@ -98,14 +224,13 @@ EXPORT_SYMBOL_GPL(__percpu_populate_mask
  */
 void *__percpu_alloc_mask(size_t size, gfp_t gfp, cpumask_t *mask)
 {
-	void *pdata = kzalloc(sizeof(struct percpu_data), gfp);
+	void *pdata = cpu_alloc(size);
 	void *__pdata = __percpu_disguise(pdata);
 
 	if (unlikely(!pdata))
 		return NULL;
 	if (likely(!__percpu_populate_mask(__pdata, size, gfp, mask)))
 		return __pdata;
-	kfree(pdata);
 	return NULL;
 }
 EXPORT_SYMBOL_GPL(__percpu_alloc_mask);

@@ -121,7 +246,7 @@ void percpu_free(void *__pdata)
 {
 	if (unlikely(!__pdata))
 		return;
-	__percpu_depopulate_mask(__pdata, &cpu_possible_map);
-	kfree(__percpu_disguise(__pdata));
+	cpu_free(__percpu_disguise(__pdata));
 }
 EXPORT_SYMBOL_GPL(percpu_free);
+
include/linux/percpu.h +6 -8
Index: linux-2.6/include/linux/percpu.h
===================================================================
@@ -33,20 +33,18 @@
 
 #ifdef CONFIG_SMP
 
-struct percpu_data {
-	void *ptrs[NR_CPUS];
-};
+#define __percpu_disguise(pdata) ((void *)~(unsigned long)(pdata))
 
-#define __percpu_disguise(pdata) (struct percpu_data *)~(unsigned long)(pdata)
 /* 
  * Use this to get to a cpu's version of the per-cpu object dynamically
  * allocated. Non-atomic access to the current CPU's version should
  * probably be combined with get_cpu()/put_cpu().
  */ 
-#define percpu_ptr(ptr, cpu)                              \
-({                                                        \
-        struct percpu_data *__p = __percpu_disguise(ptr); \
-        (__typeof__(ptr))__p->ptrs[(cpu)];	          \
+#define percpu_ptr(ptr, cpu)           			\
+({							\
+	void *p = __percpu_disguise(ptr);		\
+	unsigned long q = per_cpu_offset(cpu);		\
+    	(__typeof__(ptr))(p + q);			\
 })
 
 extern void *percpu_populate(void *__pdata, size_t size, gfp_t gfp, int cpu);
include/linux/vmstat.h +1 -1
Index: linux-2.6/include/linux/vmstat.h
===================================================================
@@ -36,7 +36,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PS
 		FOR_ALL_ZONES(PGSCAN_KSWAPD),
 		FOR_ALL_ZONES(PGSCAN_DIRECT),
 		PGINODESTEAL, SLABS_SCANNED, KSWAPD_STEAL, KSWAPD_INODESTEAL,
-		PAGEOUTRUN, ALLOCSTALL, PGROTATED,
+		PAGEOUTRUN, ALLOCSTALL, PGROTATED, ALLOC_PERCPU,
 		NR_VM_EVENT_ITEMS
 };
mm/vmstat.c +1 -0
Index: linux-2.6/mm/vmstat.c
===================================================================
@@ -642,6 +642,7 @@ static const char * const vmstat_text[] 
 	"allocstall",
 
 	"pgrotated",
+	"alloc_percpu_bytes",
 #endif
 };
#2 lore raw

Re: [patch 1/7] allocpercpu: Make it a true per cpu allocator by allocating from a per cpu array

mm/allocpercpu.c line 38

Quoted 4 lines · Christoph Lameter
mm/allocpercpu.c +4 -0
+
+enum unit_type { FREE, END, USED };
+
+static u8 cpu_alloc_map[UNITS_PER_CPU] = { 1, };

You mean END here instead of 1 :)

mm/allocpercpu.c line 90

Quoted 7 lines
mm/allocpercpu.c +7 -0
+/*
+ * Allocate an object of a certain size
+ *
+ * Returns a per cpu pointer that must not be directly used.
+ */
+static void *cpu_alloc(unsigned long size)
+{

We might need to give an alignment constraint here. Some per_cpu users would like to get a 64 bytes zone, siting in one cache line and not two :)

author
#3 lore raw

Re: [patch 1/7] allocpercpu: Make it a true per cpu allocator by allocating from a per cpu array

Quoted 7 lines · Eric Dumazet

mm/allocpercpu.c line 38

Quoted 4 lines · Christoph Lameter
mm/allocpercpu.c +4 -0
+
+enum unit_type { FREE, END, USED };
+
+static u8 cpu_alloc_map[UNITS_PER_CPU] = { 1, };

You mean END here instead of 1 :)

Sigh. A leftover. This can be removed.

Quoted 10 lines

mm/allocpercpu.c line 90

Quoted 7 lines
mm/allocpercpu.c +7 -0
+/*
+ * Allocate an object of a certain size
+ *
+ * Returns a per cpu pointer that must not be directly used.
+ */
+static void *cpu_alloc(unsigned long size)
+{

We might need to give an alignment constraint here. Some per_cpu users would like to get a 64 bytes zone, siting in one cache line and not two :)

Well not sure about that. Alignment is mostly useful on SMP with cacheline contention. This is a per cpu area that should not be contended.

author
#4 lore raw

PATCH 2/7 allocpercpu: Remove functions that are rarely used.

In reply to Christoph Lameter

Population and depopulation is no longer needed since newly created per cpu areas will have all the fields needed. Teardown of per cpu areas will remove objects no longer needed.

This basically reverts the API to the way it was before the population and depopulation went in. There is only a single user in the kernel that uses these functions in net/iucv/iucv.c which is S/390 specific.

Remove the useless population and depopulation functions there. In that driver we have the single occurrence of a per cpu allocations that uses GFP flags. The allocation from the DMA zone is required in order to have memory below 2G. But it seems that the per cpu areas are also under 2G so we are fine there.

Signed-off-by Christoph Lameter <[email protected]>
---
 include/linux/percpu.h |   42 +------------------
 mm/allocpercpu.c       |  104 +++++--------------------------------------------
 net/iucv/iucv.c        |   31 +++-----------
 3 files changed, 21 insertions(+), 156 deletions(-)
include/linux/percpu.h +3 -39
Index: linux-2.6/include/linux/percpu.h
===================================================================
@@ -47,41 +47,16 @@
     	(__typeof__(ptr))(p + q);			\
 })
 
-extern void *percpu_populate(void *__pdata, size_t size, gfp_t gfp, int cpu);
-extern void percpu_depopulate(void *__pdata, int cpu);
-extern int __percpu_populate_mask(void *__pdata, size_t size, gfp_t gfp,
-				  cpumask_t *mask);
-extern void __percpu_depopulate_mask(void *__pdata, cpumask_t *mask);
-extern void *__percpu_alloc_mask(size_t size, gfp_t gfp, cpumask_t *mask);
+extern void *__alloc_percpu(size_t size);
 extern void percpu_free(void *__pdata);
 
 #else /* CONFIG_SMP */
 
 #define percpu_ptr(ptr, cpu) ({ (void)(cpu); (ptr); })
 
-static inline void percpu_depopulate(void *__pdata, int cpu)
+static __always_inline void *__alloc_percpu(size_t size)
 {
-}
-
-static inline void __percpu_depopulate_mask(void *__pdata, cpumask_t *mask)
-{
-}
-
-static inline void *percpu_populate(void *__pdata, size_t size, gfp_t gfp,
-				    int cpu)
-{
-	return percpu_ptr(__pdata, cpu);
-}
-
-static inline int __percpu_populate_mask(void *__pdata, size_t size, gfp_t gfp,
-					 cpumask_t *mask)
-{
-	return 0;
-}
-
-static __always_inline void *__percpu_alloc_mask(size_t size, gfp_t gfp, cpumask_t *mask)
-{
-	return kzalloc(size, gfp);
+	return kzalloc(size, GFP_KERNEL);
 }
 
 static inline void percpu_free(void *__pdata)

@@ -91,19 +66,8 @@ static inline void percpu_free(void *__p
 
 #endif /* CONFIG_SMP */
 
-#define percpu_populate_mask(__pdata, size, gfp, mask) \
-	__percpu_populate_mask((__pdata), (size), (gfp), &(mask))
-#define percpu_depopulate_mask(__pdata, mask) \
-	__percpu_depopulate_mask((__pdata), &(mask))
-#define percpu_alloc_mask(size, gfp, mask) \
-	__percpu_alloc_mask((size), (gfp), &(mask))
-
-#define percpu_alloc(size, gfp) percpu_alloc_mask((size), (gfp), cpu_online_map)
-
 /* (legacy) interface for use without CPU hotplug handling */
 
-#define __alloc_percpu(size)	percpu_alloc_mask((size), GFP_KERNEL, \
-						  cpu_possible_map)
 #define alloc_percpu(type)	(type *)__alloc_percpu(sizeof(type))
 #define free_percpu(ptr)	percpu_free((ptr))
 #define per_cpu_ptr(ptr, cpu)	percpu_ptr((ptr), (cpu))
mm/allocpercpu.c +11 -93

122 lines not on this page · raw message

net/iucv/iucv.c +7 -24
Index: linux-2.6/net/iucv/iucv.c
===================================================================
@@ -556,25 +556,6 @@ static int __cpuinit iucv_cpu_notify(str
 	long cpu = (long) hcpu;
 
 	switch (action) {
-	case CPU_UP_PREPARE:
-	case CPU_UP_PREPARE_FROZEN:
-		if (!percpu_populate(iucv_irq_data,
-				     sizeof(struct iucv_irq_data),
-				     GFP_KERNEL|GFP_DMA, cpu))
-			return NOTIFY_BAD;
-		if (!percpu_populate(iucv_param, sizeof(union iucv_param),
-				     GFP_KERNEL|GFP_DMA, cpu)) {
-			percpu_depopulate(iucv_irq_data, cpu);
-			return NOTIFY_BAD;
-		}
-		break;
-	case CPU_UP_CANCELED:
-	case CPU_UP_CANCELED_FROZEN:
-	case CPU_DEAD:
-	case CPU_DEAD_FROZEN:
-		percpu_depopulate(iucv_param, cpu);
-		percpu_depopulate(iucv_irq_data, cpu);
-		break;
 	case CPU_ONLINE:
 	case CPU_ONLINE_FROZEN:
 	case CPU_DOWN_FAILED:

@@ -1617,16 +1598,18 @@ static int __init iucv_init(void)
 		rc = PTR_ERR(iucv_root);
 		goto out_bus;
 	}
-	/* Note: GFP_DMA used to get memory below 2G */
-	iucv_irq_data = percpu_alloc(sizeof(struct iucv_irq_data),
-				     GFP_KERNEL|GFP_DMA);
+	/*
+	 * Note: GFP_DMA used to get memory below 2G.
+	 *
+	 * The percpu data is below 2G right ? So this should work too -cl?
+	 */
+	iucv_irq_data = percpu_alloc(struct iucv_irq_data);
 	if (!iucv_irq_data) {
 		rc = -ENOMEM;
 		goto out_root;
 	}
 	/* Allocate parameter blocks. */
-	iucv_param = percpu_alloc(sizeof(union iucv_param),
-				  GFP_KERNEL|GFP_DMA);
+	iucv_param = percpu_alloc(union iucv_param);
 	if (!iucv_param) {
 		rc = -ENOMEM;
 		goto out_extint;
author
#5 lore raw

PATCH 3/7 Allocpercpu: Do __percpu_disguise() only if CONFIG_DEBUG_VM is set

In reply to Christoph Lameter

Disguising costs a few cycles in the hot paths. So switch it off if we are not debuggin.

Signed-off-by Christoph Lameter <[email protected]>
---
 include/linux/percpu.h |    4 ++++
 1 file changed, 4 insertions(+)
include/linux/percpu.h +4 -0
Index: linux-2.6/include/linux/percpu.h
===================================================================
@@ -33,7 +33,11 @@
 
 #ifdef CONFIG_SMP
 
+#ifdef CONFIG_DEBUG_VM
 #define __percpu_disguise(pdata) ((void *)~(unsigned long)(pdata))
+#else
+#define __percpu_disguide(pdata) ((void *)(pdata))
+#endif
 
 /* 
  * Use this to get to a cpu's version of the per-cpu object dynamically
#6 lore raw

Re: [patch 3/7] Allocpercpu: Do __percpu_disguise() only if CONFIG_DEBUG_VM is set

Quoted 14 lines · Christoph Lameter

Disguising costs a few cycles in the hot paths. So switch it off if we are not debuggin.

Signed-off-by Christoph Lameter <[email protected]>
---
 include/linux/percpu.h |    4 ++++
 1 file changed, 4 insertions(+)

Index: linux-2.6/include/linux/percpu.h
===================================================================

include/linux/percpu.h line 40

Quoted 8 lines
include/linux/percpu.h +4 -0
@@ -33,7 +33,11 @@
 
 #ifdef CONFIG_SMP
 
+#ifdef CONFIG_DEBUG_VM
 #define __percpu_disguise(pdata) ((void *)~(unsigned long)(pdata))
+#else
+#define __percpu_disguide(pdata) ((void *)(pdata))
+#endif

Yes good idea, but a litle typo here :)

author
#7 lore raw

PATCH 4/7 Percpu: Add support for this_cpu_offset() to be able to create this_cpu_ptr()

In reply to Christoph Lameter

Support for this_cpu_ptr() is important for those arches that allow a faster way to get to the per cpu area of the local processor.

Signed-off-by Christoph Lameter <[email protected]>
---
 include/asm-generic/percpu.h |    4 ++++
 include/asm-ia64/percpu.h    |    3 +++
 include/asm-powerpc/percpu.h |    3 +++
 include/asm-s390/percpu.h    |    4 ++++
 include/asm-sparc64/percpu.h |    2 ++
 include/asm-x86/percpu_32.h  |    2 ++
 include/asm-x86/percpu_64.h  |    4 ++++
 include/linux/percpu.h       |    7 +++++++
 8 files changed, 29 insertions(+)
include/linux/percpu.h +7 -0
Index: linux-2.6/include/linux/percpu.h
===================================================================
@@ -51,6 +51,13 @@
     	(__typeof__(ptr))(p + q);			\
 })
 
+#define this_cpu_ptr(ptr)           			\
+({							\
+	void *p = ptr;					\
+    	(__typeof__(ptr))(p + this_cpu_offset());	\
+})
+
+
 extern void *__alloc_percpu(size_t size);
 extern void percpu_free(void *__pdata);
include/asm-generic/percpu.h +4 -0
Index: linux-2.6/include/asm-generic/percpu.h
===================================================================
@@ -26,6 +26,8 @@ extern unsigned long __per_cpu_offset[NR
 #define __get_cpu_var(var) per_cpu(var, smp_processor_id())
 #define __raw_get_cpu_var(var) per_cpu(var, raw_smp_processor_id())
 
+#define this_cpu_offset() __per_cpu_offset(raw_smp_processor_id())
+
 /* A macro to avoid #include hell... */
 #define percpu_modcopy(pcpudst, src, size)			\
 do {								\

@@ -53,4 +55,6 @@ do {								\
 #define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
 #define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
 
+#define this_cpu_offset() 0
+
 #endif /* _ASM_GENERIC_PERCPU_H_ */
include/asm-ia64/percpu.h +3 -0
Index: linux-2.6/include/asm-ia64/percpu.h
===================================================================
@@ -51,6 +51,8 @@ extern unsigned long __per_cpu_offset[NR
 /* Equal to __per_cpu_offset[smp_processor_id()], but faster to access: */
 DECLARE_PER_CPU(unsigned long, local_per_cpu_offset);
 
+#define this_cpu_offset() __ia64_per_cpu_var(local_per_cpu_offset)
+
 #define per_cpu(var, cpu)  (*RELOC_HIDE(&per_cpu__##var, __per_cpu_offset[cpu]))
 #define __get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __ia64_per_cpu_var(local_per_cpu_offset)))
 #define __raw_get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __ia64_per_cpu_var(local_per_cpu_offset)))

@@ -65,6 +67,7 @@ extern void *per_cpu_init(void);
 #define __get_cpu_var(var)			per_cpu__##var
 #define __raw_get_cpu_var(var)			per_cpu__##var
 #define per_cpu_init()				(__phys_per_cpu_start)
+#define this_cpu_offset()			0
 
 #endif	/* SMP */
include/asm-powerpc/percpu.h +3 -0
Index: linux-2.6/include/asm-powerpc/percpu.h
===================================================================
@@ -16,6 +16,8 @@
 #define __my_cpu_offset() get_paca()->data_offset
 #define per_cpu_offset(x) (__per_cpu_offset(x))
 
+#define this_cpu_offset() __my_cpu_offset()
+
 /* Separate out the type, so (int[3], foo) works. */
 #define DEFINE_PER_CPU(type, name) \
     __attribute__((__section__(".data.percpu"))) __typeof__(type) per_cpu__##name

@@ -51,6 +53,7 @@ extern void setup_per_cpu_areas(void);
 #define per_cpu(var, cpu)			(*((void)(cpu), &per_cpu__##var))
 #define __get_cpu_var(var)			per_cpu__##var
 #define __raw_get_cpu_var(var)			per_cpu__##var
+#define this_cpu_offset()			0
 
 #endif	/* SMP */
include/asm-s390/percpu.h +4 -0
Index: linux-2.6/include/asm-s390/percpu.h
===================================================================
@@ -51,6 +51,8 @@ extern unsigned long __per_cpu_offset[NR
 #define per_cpu(var,cpu) __reloc_hide(var,__per_cpu_offset[cpu])
 #define per_cpu_offset(x) (__per_cpu_offset[x])
 
+#define this_cpu_offset() S390_lowcore.percpu_offset
+
 /* A macro to avoid #include hell... */
 #define percpu_modcopy(pcpudst, src, size)			\
 do {								\

@@ -71,6 +73,8 @@ do {								\
 #define __raw_get_cpu_var(var) __reloc_hide(var,0)
 #define per_cpu(var,cpu) __reloc_hide(var,0)
 
+#define this_cpu_offset() 0
+
 #endif /* SMP */
 
 #define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
include/asm-sparc64/percpu.h +2 -0
Index: linux-2.6/include/asm-sparc64/percpu.h
===================================================================
@@ -5,6 +5,8 @@
 
 register unsigned long __local_per_cpu_offset asm("g5");
 
+#define this_cpu_offset() __local_per_cpu_offset
+
 #ifdef CONFIG_SMP
 
 #define setup_per_cpu_areas()			do { } while (0)
include/asm-x86/percpu_32.h +2 -0
Index: linux-2.6/include/asm-x86/percpu_32.h
===================================================================
@@ -72,6 +72,8 @@ DECLARE_PER_CPU(unsigned long, this_cpu_
 	RELOC_HIDE(&per_cpu__##var, x86_read_percpu(this_cpu_off));	\
 }))
 
+#define this_cpu_offset() x86_read_percpu(this_cpu_off)
+
 #define __get_cpu_var(var) __raw_get_cpu_var(var)
 
 /* A macro to avoid #include hell... */
include/asm-x86/percpu_64.h +4 -0
Index: linux-2.6/include/asm-x86/percpu_64.h
===================================================================
@@ -14,6 +14,8 @@
 #define __per_cpu_offset(cpu) (cpu_pda(cpu)->data_offset)
 #define __my_cpu_offset() read_pda(data_offset)
 
+#define this_cpu_offset() read_pda(data_offset)
+
 #define per_cpu_offset(x) (__per_cpu_offset(x))
 
 /* Separate out the type, so (int[3], foo) works. */

@@ -58,6 +60,8 @@ extern void setup_per_cpu_areas(void);
 #define __get_cpu_var(var)			per_cpu__##var
 #define __raw_get_cpu_var(var)			per_cpu__##var
 
+#define this_cpu_offset() 0
+
 #endif	/* SMP */
 
 #define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
author
#8 lore raw

PATCH 5/7 SLUB: Use allocpercpu to allocate per cpu data instead of running our own per cpu allocator

In reply to Christoph Lameter

Using allocpercpu removes the needs for the per cpu arrays in the kmem_cache struct. These could get quite big if we have to support system of up to thousands of cpus. The use of alloc_percpu means that:

1. The size of kmem_cache for SMP configuration shrinks since we will only need 1 pointer instead of NR_CPUS. The same pointer can be used by all processors. Reduces cache footprint of the allocator.

2. We can dynamically size kmem_cache according to the actual nodes in the system meaning less memory overhead for configurations that may potentially support up to 1k NUMA nodes.

3. We can remove the diddle widdle with allocating and releasing kmem_cache_cpu structures when bringing up and shuttting down cpus. The allocpercpu logic will do it all for us.

4. Fastpath performance increases by another 20% vs. the earlier improvements. Instead of having fastpath with 40-50 cycles we are now in the 30-40 range.

Signed-off-by Christoph Lameter <[email protected]>
---
 include/linux/slub_def.h |   11 ++--
 mm/slub.c                |  125 ++++-------------------------------------------
 2 files changed, 18 insertions(+), 118 deletions(-)
include/linux/slub_def.h +6 -5
Index: linux-2.6/include/linux/slub_def.h
===================================================================
@@ -34,6 +34,12 @@ struct kmem_cache_node {
  * Slab cache management.
  */
 struct kmem_cache {
+#ifdef CONFIG_SMP
+	/* Per cpu pointer usable for any cpu */
+	struct kmem_cache_cpu *cpu_slab;
+#else
+	struct kmem_cache_cpu cpu_slab;
+#endif
 	/* Used for retriving partial slabs etc */
 	unsigned long flags;
 	int size;		/* The size of an object including meta data */

@@ -63,11 +69,6 @@ struct kmem_cache {
 	int defrag_ratio;
 	struct kmem_cache_node *node[MAX_NUMNODES];
 #endif
-#ifdef CONFIG_SMP
-	struct kmem_cache_cpu *cpu_slab[NR_CPUS];
-#else
-	struct kmem_cache_cpu cpu_slab;
-#endif
 };
 
 /*
mm/slub.c +12 -113

181 lines not on this page · raw message

author
#9 lore raw

PATCH 6/7 SLUB: No need to cache kmem_cache data in kmem_cache_cpu anymore

In reply to Christoph Lameter

Remove the fields in kmem_cache_cpu that were used to cache data from kmem_cache when they were in different cachelines. The cacheline that holds the per cpu array pointer now also holds these values. We can cut down the kmem_cache_cpu size to almost half.

The get_freepointer() and set_freepointer() functions that used to be only intended for the slow path now are also useful for the hot path since access to the field does not require an additional cacheline anymore. This results in consistent use of setting the freepointer for objects throughout SLUB.

Signed-off-by Christoph Lameter <[email protected]>
---
 include/linux/slub_def.h |    3 --
 mm/slub.c                |   50 +++++++++++++++--------------------------------
 2 files changed, 16 insertions(+), 37 deletions(-)
include/linux/slub_def.h +0 -3
Index: linux-2.6/include/linux/slub_def.h
===================================================================
@@ -15,9 +15,6 @@ struct kmem_cache_cpu {
 	void **freelist;
 	struct page *page;
 	int node;
-	unsigned int offset;
-	unsigned int objsize;
-	unsigned int objects;
 };
 
 struct kmem_cache_node {
mm/slub.c +16 -34

139 lines not on this page · raw message

author
#10 lore raw

PATCH 7/7 SLUB: Optimize per cpu access on the local cpu using this_cpu_ptr()

In reply to Christoph Lameter

Use this_cpu_ptr to optimize access to the per cpu area in the fastpaths.

Signed-off-by Christoph Lameter <[email protected]>
---
 mm/slub.c |   27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)
mm/slub.c +19 -8
Index: linux-2.6/mm/slub.c
===================================================================
@@ -248,6 +248,15 @@ static inline struct kmem_cache_cpu *get
 #endif
 }
 
+static inline struct kmem_cache_cpu *this_cpu_slab(struct kmem_cache *s)
+{
+#ifdef CONFIG_SMP
+	return this_cpu_ptr(s->cpu_slab);
+#else
+	return &s->cpu_slab;
+#endif
+}
+
 /*
  * The end pointer in a slab is special. It points to the first object in the
  * slab but has bit 0 set to mark it.

@@ -1521,7 +1530,7 @@ static noinline unsigned long get_new_sl
 	if (!page)
 		return 0;
 
-	*pc = c = get_cpu_slab(s, smp_processor_id());
+	*pc = c = this_cpu_slab(s);
 	if (c->page) {
 		/*
 		 * Someone else populated the cpu_slab while we

@@ -1650,25 +1659,26 @@ static void __always_inline *slab_alloc(
 	struct kmem_cache_cpu *c;
 
 #ifdef CONFIG_FAST_CMPXCHG_LOCAL
-	c = get_cpu_slab(s, get_cpu());
+	preempt_disable();
+	c = this_cpu_slab(s);
 	do {
 		object = c->freelist;
 		if (unlikely(is_end(object) || !node_match(c, node))) {
 			object = __slab_alloc(s, gfpflags, node, addr, c);
 			if (unlikely(!object)) {
-				put_cpu();
+				preempt_enable();
 				goto out;
 			}
 			break;
 		}
 	} while (cmpxchg_local(&c->freelist, object,
 			get_freepointer(s, object)) != object);
-	put_cpu();
+	preempt_enable();
 #else
 	unsigned long flags;
 
 	local_irq_save(flags);
-	c = get_cpu_slab(s, smp_processor_id());
+	c = this_cpu_slab(s);
 	if (unlikely((is_end(c->freelist)) || !node_match(c, node))) {
 
 		object = __slab_alloc(s, gfpflags, node, addr, c);

@@ -1794,7 +1804,8 @@ static void __always_inline slab_free(st
 #ifdef CONFIG_FAST_CMPXCHG_LOCAL
 	void **freelist;
 
-	c = get_cpu_slab(s, get_cpu());
+	preempt_disable();
+	c = this_cpu_slab(s);
 	debug_check_no_locks_freed(object, s->objsize);
 	do {
 		freelist = c->freelist;

@@ -1816,13 +1827,13 @@ static void __always_inline slab_free(st
 		}
 		set_freepointer(s, object, freelist);
 	} while (cmpxchg_local(&c->freelist, freelist, object) != freelist);
-	put_cpu();
+	preempt_enable();
 #else
 	unsigned long flags;
 
 	local_irq_save(flags);
 	debug_check_no_locks_freed(object, s->objsize);
-	c = get_cpu_slab(s, smp_processor_id());
+	c = this_cpu_slab(s);
 	if (likely(page == c->page && c->node >= 0)) {
 		set_freepointer(s, object, c->freelist);
 		c->freelist = object;
author
#12 lore raw
Quoted 5 lines · David Miller

Are these patches against -mm or mainline?

I get a lot of rejects starting with patch 6 against mainline and I really wanted to test them out on sparc64.

Hmmm... They are against the current slab performance head (which is in mm but it has not been released yet ;-).

Do

git pull git://git.kernel.org/pub/scm/linux/kernel/git/christoph/slab.git
performance

and then you should be able to apply these patches.

maintainer
#13 lore raw

From: Christoph Lameter <[email protected]>
Date: Wed, 31 Oct 2007 17:26:16 -0700 (PDT)

Quoted 6 lines

Do

git pull git://git.kernel.org/pub/scm/linux/kernel/git/christoph/slab.git
performance

and then you should be able to apply these patches.

Thanks a lot Chrisoph.

author
#14 lore raw
Quoted 6 lines · David Miller
Quoted 4 lines

git pull git://git.kernel.org/pub/scm/linux/kernel/git/christoph/slab.git
performance

and then you should be able to apply these patches.

Thanks a lot Chrisoph.

Others may have the same issue.

git pull git://git.kernel.org/pub/scm/linux/kernel/git/christoph/slab.git allocpercpu

should get you the whole thing.

maintainer
#15 lore raw

From: Christoph Lameter <[email protected]>
Date: Wed, 31 Oct 2007 17:31:12 -0700 (PDT)

Quoted 5 lines

Others may have the same issue.

git pull git://git.kernel.org/pub/scm/linux/kernel/git/christoph/slab.git allocpercpu

should get you the whole thing.

This patch fixes build failures with DEBUG_VM disabled.

include/linux/percpu.h +1 -1
index 4b167c0..d414703 100644
@@ -36,7 +36,7 @@
 #ifdef CONFIG_DEBUG_VM
 #define __percpu_disguise(pdata) ((void *)~(unsigned long)(pdata))
 #else
-#define __percpu_disguide(pdata) ((void *)(pdata))
+#define __percpu_disguise(pdata) ((void *)(pdata))
 #endif
 
 /*
author
#16 lore raw
Quoted 1 line

This patch fixes build failures with DEBUG_VM disabled.

Well there is more there. Last minute mods sigh. With DEBUG_VM you likely need this patch.

---
 include/linux/percpu.h |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/percpu.h +2 -2
Index: linux-2.6/include/linux/percpu.h
===================================================================
@@ -36,7 +36,7 @@
 #ifdef CONFIG_DEBUG_VM
 #define __percpu_disguise(pdata) ((void *)~(unsigned long)(pdata))
 #else
-#define __percpu_disguide(pdata) ((void *)(pdata))
+#define __percpu_disguise(pdata) ((void *)(pdata))
 #endif
 
 /* 

@@ -53,7 +53,7 @@
 
 #define this_cpu_ptr(ptr)           			\
 ({							\
-	void *p = ptr;					\
+	void *p = __percpu_disguise(ptr);		\
     	(__typeof__(ptr))(p + this_cpu_offset());	\
 })
maintainer
#17 lore raw

From: Christoph Lameter <[email protected]>
Date: Wed, 31 Oct 2007 17:53:23 -0700 (PDT)

Quoted 4 lines
Quoted 1 line

This patch fixes build failures with DEBUG_VM disabled.

Well there is more there. Last minute mods sigh. With DEBUG_VM you likely need this patch.

Without DEBUG_VM I get a loop of crashes shortly after SSHD is started, I'll try to track it down.

author
#18 lore raw
Quoted 2 lines · David Miller

Without DEBUG_VM I get a loop of crashes shortly after SSHD is started, I'll try to track it down.

Check how much per cpu memory is in use by

cat /proc/vmstat

currently we have a 32k limit there.

maintainer
#19 lore raw

From: Christoph Lameter <[email protected]>
Date: Wed, 31 Oct 2007 18:01:34 -0700 (PDT)

Quoted 10 lines · David Miller
Quoted 3 lines

Without DEBUG_VM I get a loop of crashes shortly after SSHD is started, I'll try to track it down.

Check how much per cpu memory is in use by

cat /proc/vmstat

currently we have a 32k limit there.

It crashes when SSHD starts, the serial console GETTY hasn't started up yet so I can't even log in to run those commands Christoph.

All I can do now is bisect and then try to figure out what about the guilty change might cause the problem.

This is on a 64-cpu sparc64 box, and fast cmpxchg local is not set, so maybe it's one of the locking changes.

author
#20 lore raw
Quoted 3 lines · David Miller

It crashes when SSHD starts, the serial console GETTY hasn't started up yet so I can't even log in to run those commands Christoph.

Hmmm... Bad.

Quoted 2 lines

All I can do now is bisect and then try to figure out what about the guilty change might cause the problem.

Reverting the 7th patch should avoid using the sparc register that caches the per cpu area offset? (I though so, does it?)

maintainer
#21 lore raw

From: Christoph Lameter <[email protected]>
Date: Wed, 31 Oct 2007 18:12:11 -0700 (PDT)

Quoted 7 lines · David Miller
Quoted 3 lines

All I can do now is bisect and then try to figure out what about the guilty change might cause the problem.

Reverting the 7th patch should avoid using the sparc register that caches the per cpu area offset? (I though so, does it?)

Yes, that's right, %g5 holds the local cpu's per-cpu offset.

author
#22 lore raw
Quoted 5 lines · David Miller

From: Christoph Lameter <[email protected]>
Date: Wed, 31 Oct 2007 18:12:11 -0700 (PDT)

Quoted 2 lines · David Miller
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

All I can do now is bisect and then try to figure out what about the guilty change might cause the problem.

Reverting the 7th patch should avoid using the sparc register that caches the per cpu area offset? (I though so, does it?)

Yes, that's right, %g5 holds the local cpu's per-cpu offset.

And if I add the address of a percpu variable then I get to the variable for this cpu right?

maintainer
#23 lore raw

From: Christoph Lameter <[email protected]>
Date: Wed, 31 Oct 2007 18:21:02 -0700 (PDT)

Quoted 10 lines · David Miller
Quoted 9 lines

From: Christoph Lameter <[email protected]>
Date: Wed, 31 Oct 2007 18:12:11 -0700 (PDT)

Quoted 5 lines · David Miller
Quoted 3 lines

All I can do now is bisect and then try to figure out what about the guilty change might cause the problem.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

Reverting the 7th patch should avoid using the sparc register that caches the per cpu area offset? (I though so, does it?)

Yes, that's right, %g5 holds the local cpu's per-cpu offset.

And if I add the address of a percpu variable then I get to the variable for this cpu right?

Right.

I bisected the crash down to:

[PATCH] newallocpercpu

author
#24 lore raw
In reply to David Miller

Hmmmm... Got this to run on an ia64 big iron. One problem is the sizing of the pool. Somehow this needs to be dynamic.

Apply this fix on top of the others.

---
 include/asm-ia64/page.h   |    2 +-
 include/asm-ia64/percpu.h |    9 ++++++---
 mm/allocpercpu.c          |   12 ++++++++++--
 3 files changed, 17 insertions(+), 6 deletions(-)
mm/allocpercpu.c +10 -2
Index: linux-2.6/mm/allocpercpu.c
===================================================================
@@ -28,7 +28,12 @@
 /*
  * Maximum allowed per cpu data per cpu
  */
+#ifdef CONFIG_NUMA
+#define PER_CPU_ALLOC_SIZE (32768 + MAX_NUMNODES * 512)
+#else
 #define PER_CPU_ALLOC_SIZE 32768
+#endif
+
 
 #define UNIT_SIZE sizeof(unsigned long long)
 #define UNITS_PER_CPU (PER_CPU_ALLOC_SIZE / UNIT_SIZE)

@@ -37,7 +42,7 @@ enum unit_type { FREE, END, USED };
 
 static u8 cpu_alloc_map[UNITS_PER_CPU] = { 1, };
 static DEFINE_SPINLOCK(cpu_alloc_map_lock);
-static DEFINE_PER_CPU(int, cpu_area)[UNITS_PER_CPU];
+static DEFINE_PER_CPU(unsigned long long, cpu_area)[UNITS_PER_CPU];
 
 #define CPU_DATA_OFFSET ((unsigned long)&per_cpu__cpu_area)
 

@@ -97,8 +102,11 @@ static void *cpu_alloc(unsigned long siz
 		while (start < UNITS_PER_CPU &&
 				cpu_alloc_map[start] != FREE)
 			start++;
-		if (start == UNITS_PER_CPU)
+		if (start == UNITS_PER_CPU) {
+			spin_unlock(&cpu_alloc_map_lock);
+			printk(KERN_CRIT "Dynamic per cpu memory exhausted\n");
 			return NULL;
+		}
 
 		end = start + 1;
 		while (end < UNITS_PER_CPU && end - start < units &&
include/asm-ia64/page.h +1 -1
Index: linux-2.6/include/asm-ia64/page.h
===================================================================
@@ -44,7 +44,7 @@
 #define PAGE_MASK		(~(PAGE_SIZE - 1))
 #define PAGE_ALIGN(addr)	(((addr) + PAGE_SIZE - 1) & PAGE_MASK)
 
-#define PERCPU_PAGE_SHIFT	16	/* log2() of max. size of per-CPU area */
+#define PERCPU_PAGE_SHIFT	20	/* log2() of max. size of per-CPU area */
 #define PERCPU_PAGE_SIZE	(__IA64_UL_CONST(1) << PERCPU_PAGE_SHIFT)
include/asm-ia64/percpu.h +6 -3
Index: linux-2.6/include/asm-ia64/percpu.h
===================================================================
@@ -40,6 +40,12 @@
 #endif
 
 /*
+ * This will make per cpu access to the local area use the virtually mapped
+ * areas.
+ */
+#define this_cpu_offset()			0
+
+/*
  * Pretty much a literal copy of asm-generic/percpu.h, except that percpu_modcopy() is an
  * external routine, to avoid include-hell.
  */

@@ -51,8 +57,6 @@ extern unsigned long __per_cpu_offset[NR
 /* Equal to __per_cpu_offset[smp_processor_id()], but faster to access: */
 DECLARE_PER_CPU(unsigned long, local_per_cpu_offset);
 
-#define this_cpu_offset() __ia64_per_cpu_var(local_per_cpu_offset)
-
 #define per_cpu(var, cpu)  (*RELOC_HIDE(&per_cpu__##var, __per_cpu_offset[cpu]))
 #define __get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __ia64_per_cpu_var(local_per_cpu_offset)))
 #define __raw_get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __ia64_per_cpu_var(local_per_cpu_offset)))

@@ -67,7 +71,6 @@ extern void *per_cpu_init(void);
 #define __get_cpu_var(var)			per_cpu__##var
 #define __raw_get_cpu_var(var)			per_cpu__##var
 #define per_cpu_init()				(__phys_per_cpu_start)
-#define this_cpu_offset()			0
 
 #endif	/* SMP */
maintainer
#25 lore raw

From: Christoph Lameter <[email protected]>
Date: Wed, 31 Oct 2007 21:16:59 -0700 (PDT)

mm/allocpercpu.c line 36

Quoted 9 lines

/*
* Maximum allowed per cpu data per cpu
*/

mm/allocpercpu.c +5 -0
+#ifdef CONFIG_NUMA
+#define PER_CPU_ALLOC_SIZE (32768 + MAX_NUMNODES * 512)
+#else
 #define PER_CPU_ALLOC_SIZE 32768
+#endif
+

Christoph, as Rusty found out years ago when he first wrote this code, you cannot put hard limits on the alloc_percpu() allocations.

They can be done by anyone, any module, and since there was no limit before you cannot reasonably add one now.

As just one of many examples, several networking devices use alloc_percpu() for each instance they bring up. This alone can request arbitrary amounts of per-cpu data.

Therefore, you'll need to do your optimization without imposing any size limits.

maintainer
#26 lore raw
In reply to Christoph Lameter

From: Christoph Lameter <[email protected]>
Date: Wed, 31 Oct 2007 21:16:59 -0700 (PDT)

Quoted 4 lines

Index: linux-2.6/mm/allocpercpu.c
===================================================================

mm/allocpercpu.c +0 -0

...

mm/allocpercpu.c enum unit_type { FREE, END, USED }; · line 48

Quoted 9 lines
mm/allocpercpu.c +1 -1
@@ -37,7 +42,7 @@ enum unit_type { FREE, END, USED };
 
 static u8 cpu_alloc_map[UNITS_PER_CPU] = { 1, };
 static DEFINE_SPINLOCK(cpu_alloc_map_lock);
-static DEFINE_PER_CPU(int, cpu_area)[UNITS_PER_CPU];
+static DEFINE_PER_CPU(unsigned long long, cpu_area)[UNITS_PER_CPU];
 
 #define CPU_DATA_OFFSET ((unsigned long)&per_cpu__cpu_area)

This hunk helped the sparc64 looping OOPS I was getting, but cpus hang in some other fashion soon afterwards.

I'll try to debug this some more later, I've dumped enough time into this already :-)

maintainer
#27 lore raw

From: David Miller <[email protected]>
Date: Thu, 01 Nov 2007 00:01:18 -0700 (PDT)

Quoted 15 lines

From: Christoph Lameter <[email protected]>
Date: Wed, 31 Oct 2007 21:16:59 -0700 (PDT)

Quoted 4 lines

Index: linux-2.6/mm/allocpercpu.c
===================================================================

mm/allocpercpu.c +0 -0

...

Quoted 7 lines
mm/allocpercpu.c +1 -1
@@ -37,7 +42,7 @@ enum unit_type { FREE, END, USED };
 
 static u8 cpu_alloc_map[UNITS_PER_CPU] = { 1, };
 static DEFINE_SPINLOCK(cpu_alloc_map_lock);
-static DEFINE_PER_CPU(int, cpu_area)[UNITS_PER_CPU];
+static DEFINE_PER_CPU(unsigned long long, cpu_area)[UNITS_PER_CPU];
Quoted 5 lines

mm/allocpercpu.c enum unit_type { FREE, END, USED }; · line 48

Quoted 2 lines
mm/allocpercpu.c +0 -0
 #define CPU_DATA_OFFSET ((unsigned long)&per_cpu__cpu_area)

This hunk helped the sparc64 looping OOPS I was getting, but cpus hang in some other fashion soon afterwards.

And if I bump PER_CPU_ALLOC_SIZE up to 128K it seems to mostly work.

You'll definitely need to make this work dynamically somehow.

author
#28 lore raw
Quoted 4 lines · David Miller
Quoted 2 lines

This hunk helped the sparc64 looping OOPS I was getting, but cpus hang in some other fashion soon afterwards.

And if I bump PER_CPU_ALLOC_SIZE up to 128K it seems to mostly work.

Good....

Quoted 1 line

You'll definitely need to make this work dynamically somehow.

Obviously. Any ideas how?

I can probably calculate the size based on the number of online nodes when the per cpu areas are setup. But the setup is done before we even parse command line arguments. That would still mean a fixed size after bootup.

In order to make it truly dynamic we would have to virtually map the area. vmap? But that reduces performance.

maintainer
#29 lore raw

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 06:03:44 -0700 (PDT)

Quoted 2 lines

In order to make it truly dynamic we would have to virtually map the area. vmap? But that reduces performance.

But it would still be faster than the double-indirection we do now, right?

author
#30 lore raw
Quoted 8 lines · David Miller

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 06:03:44 -0700 (PDT)

Quoted 2 lines

In order to make it truly dynamic we would have to virtually map the area. vmap? But that reduces performance.

But it would still be faster than the double-indirection we do now, right?

I think I have an idea how to do this. Its a bit x86_64 specific but here it goes.

We define a virtual area of NR_CPUS * 2M areas that are each mapped by a PMD. That means we have a fixed virtual address for each cpus per cpu area.

First cpu is at PER_CPU_START
Second cpu is at PER_CPU_START + 2M

So the per cpu area for cpu n is easily calculated using

PER_CPU_START + cpu << 19

without any lookups.

On bootup we allocate the 2M pages.

After boot is complete we allow the reduction of the size of the per cpu areas . Lets say we only need 128k per cpu. Then the remaining pages will be returned to the page allocator.

We create some sysfs thingy were one can see the current reserves of per cpu storage. If one wants to reduce memory then one can write something to that to return the remainder of the memory.

maintainer
#31 lore raw

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 15:15:39 -0700 (PDT)

Quoted 3 lines

After boot is complete we allow the reduction of the size of the per cpu areas . Lets say we only need 128k per cpu. Then the remaining pages will be returned to the page allocator.

You don't know how much you will need. I exhausted the limit on sparc64 very late in the boot process when the last few userland services were starting up.

And if I subsequently bring up 100,000 IP tunnels, it will exhaust the per-cpu allocation area.

You have to make it fully dynamic, there is no way around it.

author
#32 lore raw
Quoted 10 lines · David Miller

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 15:15:39 -0700 (PDT)

Quoted 3 lines

After boot is complete we allow the reduction of the size of the per cpu areas . Lets say we only need 128k per cpu. Then the remaining pages will be returned to the page allocator.

You don't know how much you will need. I exhausted the limit on sparc64 very late in the boot process when the last few userland services were starting up.

Well you would be able to specify how much will remain. If not it will just keep the 2M reserve around.

Quoted 2 lines

And if I subsequently bring up 100,000 IP tunnels, it will exhaust the per-cpu allocation area.

Each tunnel needs 4 bytes per cpu?

Quoted 1 line

You have to make it fully dynamic, there is no way around it.

Na. Some reasonable upper limit needs to be set. If we set that to say 32Megabytes and do the virtual mapping then we can just populate the first 2M and only allocate the remainder if we need it. Then we need to rely on Mel's defrag stuff though defrag memory if we need it.

maintainer
#33 lore raw

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 15:48:00 -0700 (PDT)

Quoted 13 lines · David Miller
Quoted 11 lines

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 15:15:39 -0700 (PDT)

Quoted 3 lines

After boot is complete we allow the reduction of the size of the per cpu areas . Lets say we only need 128k per cpu. Then the remaining pages will be returned to the page allocator.

You don't know how much you will need. I exhausted the limit on sparc64 very late in the boot process when the last few userland services were starting up.

Quoted 7 lines

Well you would be able to specify how much will remain. If not it will just keep the 2M reserve around.

Quoted 2 lines

And if I subsequently bring up 100,000 IP tunnels, it will exhaust the per-cpu allocation area.

Each tunnel needs 4 bytes per cpu?

Each IP compression tunnel instance does an alloc_percpu().

Since you're the one who wants to change the semantics and guarentees of this interface, perhaps it might help if you did some greps around the tree to see how alloc_percpu() is actually used. That's what I did when I started running into trouble with your patches.

You cannot put limits of the amount of alloc_percpu() memory available to clients, please let's proceed with that basic understanding in mind. We're wasting a ton of time discussing this fundamental issue.

author
#34 lore raw
Quoted 3 lines · David Miller

You cannot put limits of the amount of alloc_percpu() memory available to clients, please let's proceed with that basic understanding in mind. We're wasting a ton of time discussing this fundamental issue.

There is no point in making absolute demands like "no limits". There are always limits to everything.

A new implementation avoids the need to allocate per cpu arrays and also avoids the 32 bytes per object times cpus that are mostly wasted for small allocations today. So its going to potentially allow more per cpu objects that available today.

A reasonable implementation for 64 bit is likely going to depend on reserving some virtual memory space for the per cpu mappings so that they can be dynamically grown up to what the reserved virtual space allows.

F.e. If we reserve 256G of virtual space and support a maximum of 16k cpus then there is a limit on the per cpu space available of 16MB.

maintainer
#35 lore raw

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 18:06:17 -0700 (PDT)

Quoted 6 lines

A reasonable implementation for 64 bit is likely going to depend on reserving some virtual memory space for the per cpu mappings so that they can be dynamically grown up to what the reserved virtual space allows.

F.e. If we reserve 256G of virtual space and support a maximum of 16k cpus then there is a limit on the per cpu space available of 16MB.

Now that I understand your implementation better, yes this sounds just fine.

#36 lore raw
In reply to David Miller
Quoted 4 lines · David Miller

Since you're the one who wants to change the semantics and guarentees of this interface, perhaps it might help if you did some greps around the tree to see how alloc_percpu() is actually used. That's what I did when I started running into trouble with your patches.

This fancy new BDI stuff also lives off percpu_counter/alloc_percpu().

That means that for example each NFS mount also consumes a number of words - not quite sure from the top of my head how many, might be in the order of 24 bytes or something.

I once before started looking at this, because the current alloc_percpu() can have some false sharing - not that I have machines that are overly bothered by that. I like the idea of a strict percpu region, however do be aware of the users.

author
#37 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · David Miller

Since you're the one who wants to change the semantics and guarentees of this interface, perhaps it might help if you did some greps around the tree to see how alloc_percpu() is actually used. That's what I did when I started running into trouble with your patches.

This fancy new BDI stuff also lives off percpu_counter/alloc_percpu().

Yes there are numerous uses. I even can increase page allocator performance and reduce its memory footprint by using it here.

Quoted 8 lines

That means that for example each NFS mount also consumes a number of words - not quite sure from the top of my head how many, might be in the order of 24 bytes or something.

I once before started looking at this, because the current alloc_percpu() can have some false sharing - not that I have machines that are overly bothered by that. I like the idea of a strict percpu region, however do be aware of the users.

Well I wonder if I should introduce it not as a replacement but as an alternative to allocpercpu? We can then gradually switch over. The existing API does not allow the specification of gfp_masks or alignements.

#38 lore raw
Quoted 3 lines · Christoph Lameter

Well I wonder if I should introduce it not as a replacement but as an alternative to allocpercpu? We can then gradually switch over. The existing API does not allow the specification of gfp_masks or alignements.

I've thought about suggesting that very thing. However, I think we need to have a clear view of where we're going with that so that we don't end up with two per cpu allocators because some users could not be converted over or some such.

author
#39 lore raw
Quoted 10 lines · Peter Zijlstra
Quoted 3 lines · Christoph Lameter

Well I wonder if I should introduce it not as a replacement but as an alternative to allocpercpu? We can then gradually switch over. The existing API does not allow the specification of gfp_masks or alignements.

I've thought about suggesting that very thing. However, I think we need to have a clear view of where we're going with that so that we don't end up with two per cpu allocators because some users could not be converted over or some such.

At least in my tests so far show that it can be a full replacement but then I have only tested on x86_64 and Ia64. Its likely much easier to go for the full replacement rather than in steps.

If we want dynamically sized virtually mapped per cpu areas then we may have issues on 32 bit platforms and with !MMU. So I would think that a fallback to a statically sized version may be needed. On the other hand !MMU and 32 bit do not support a large number of processors. So we may be able to get away on 32 bit with a small virtual memory area.

#40 lore raw
In reply to David Miller
Quoted 2 lines · David Miller

Each IP compression tunnel instance does an alloc_percpu().

Actually all IPComp tunnels share one set of objects which are allocated per-cpu. So only the first tunnel would do that.

In fact that was precisely the reason why per-cpu is used in IPComp as otherwise we can just allocate normal memory.

Cheers,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
author
#41 lore raw
Quoted 6 lines · Herbert Xu
Quoted 2 lines · David Miller

Each IP compression tunnel instance does an alloc_percpu().

Actually all IPComp tunnels share one set of objects which are allocated per-cpu. So only the first tunnel would do that.

Ahh so the need to be able to expand per cpu memory storage on demand is not as critical as we thought.

#42 lore raw
Quoted 9 lines · Christoph Lameter
Quoted 4 lines · Herbert Xu
Quoted 1 line · David Miller

Each IP compression tunnel instance does an alloc_percpu().

Actually all IPComp tunnels share one set of objects which are allocated per-cpu. So only the first tunnel would do that.

Ahh so the need to be able to expand per cpu memory storage on demand is not as critical as we thought.

Yes, but still desirable for future optimizations.

For example, I do think using a per cpu memory storage on net_device refcnt & last_rx could give us some speedups.

author
#43 lore raw
Quoted 8 lines · Eric Dumazet
Quoted 7 lines · Christoph Lameter
Quoted 4 lines · Herbert Xu
Quoted 1 line · David Miller

Each IP compression tunnel instance does an alloc_percpu().

Actually all IPComp tunnels share one set of objects which are allocated per-cpu. So only the first tunnel would do that.

Quoted 8 lines
Quoted 3 lines

Ahh so the need to be able to expand per cpu memory storage on demand is not as critical as we thought.

Yes, but still desirable for future optimizations.

For example, I do think using a per cpu memory storage on net_device refcnt & last_rx could give us some speedups.

Note that there was a new patchset posted (titled cpu alloc v1) that provides on demand extension of the cpu areas.

See http://marc.info/?l=linux-kernel&m=119438261304093&w=2

#44 lore raw
Quoted 7 lines · Christoph Lameter
Quoted 2 lines · Eric Dumazet

For example, I do think using a per cpu memory storage on net_device refcnt & last_rx could give us some speedups.

Note that there was a new patchset posted (titled cpu alloc v1) that provides on demand extension of the cpu areas.

See http://marc.info/?l=linux-kernel&m=119438261304093&w=2

Thank you Christoph. I was traveling last week so I missed that.

This new patchset looks very interesting, you did a fantastic job !

maintainer
#45 lore raw

From: Eric Dumazet <[email protected]>
Date: Mon, 12 Nov 2007 21:18:17 +0100

Quoted 5 lines
Quoted 4 lines · Christoph Lameter
Quoted 2 lines · Eric Dumazet

For example, I do think using a per cpu memory storage on net_device refcnt & last_rx could give us some speedups.

Quoted 8 lines
Quoted 4 lines

Note that there was a new patchset posted (titled cpu alloc v1) that provides on demand extension of the cpu areas.

See http://marc.info/?l=linux-kernel&m=119438261304093&w=2

Thank you Christoph. I was traveling last week so I missed that.

This new patchset looks very interesting, you did a fantastic job !

Yes I like it too. It's in my backlog of things to test on sparc64.

#46 lore raw
In reply to Eric Dumazet
Quoted 8 lines
Quoted 3 lines

Ahh so the need to be able to expand per cpu memory storage on demand is not as critical as we thought.

Yes, but still desirable for future optimizations.

For example, I do think using a per cpu memory storage on net_device refcnt & last_rx could give us some speedups.

We do want to keep a very tight handle on bloat in per-cpu allocations. By definition the total allocation is multiplied by the number of cpus. Only ia64 has outrageous numbers of cpus in a single system image today ... but the trend in multi-core chips looks to have a Moore's law arc to it, so everyone is going to be looking at lots of cpus before long.

-Tony

#47 lore raw
Quoted 4 lines · Tony
Quoted 4 lines
Quoted 2 lines

Ahh so the need to be able to expand per cpu memory storage on demand is not as critical as we thought.

Yes, but still desirable for future optimizations.

Quoted 9 lines
Quoted 1 line

For example, I do think using a per cpu memory storage on net_device refcnt & last_rx could give us some speedups.

We do want to keep a very tight handle on bloat in per-cpu allocations. By definition the total allocation is multiplied by the number of cpus. Only ia64 has outrageous numbers of cpus in a single system image today ... but the trend in multi-core chips looks to have a Moore's law arc to it, so everyone is going to be looking at lots of cpus before long.

I dont think this is a problem. Cpus numbers and ram size are related, even if Moore didnt predicted it;

Nobody wants to ship/build a 4096 cpus machine with 256 MB of ram inside. Or call it a GPU and dont expect it to run linux :)

99,9% of linux machines running on earth have less than 8 cpus and less than 1000 ethernet/network devices.

In case we increase the number of cpus on a machine, the limiting factor is the fact that cpus have to continually exchange on memory bus those highly touched cache lines that contain refcounters or stats.

maintainer
#48 lore raw

From: Eric Dumazet <[email protected]>
Date: Mon, 12 Nov 2007 21:14:47 +0100

Quoted 6 lines

I dont think this is a problem. Cpus numbers and ram size are related, even if Moore didnt predicted it;

Nobody wants to ship/build a 4096 cpus machine with 256 MB of ram inside. Or call it a GPU and dont expect it to run linux :)

Quoted 6 lines

99,9% of linux machines running on earth have less than 8 cpus and less than 1000 ethernet/network devices.

In case we increase the number of cpus on a machine, the limiting factor is the fact that cpus have to continually exchange on memory bus those highly touched cache lines that contain refcounters or stats.

I totally agree with everything Eric is saying here.

maintainer
#49 lore raw
In reply to Herbert Xu

From: Herbert Xu <[email protected]>
Date: Mon, 12 Nov 2007 18:52:35 +0800

Quoted 9 lines
Quoted 2 lines · David Miller

Each IP compression tunnel instance does an alloc_percpu().

Actually all IPComp tunnels share one set of objects which are allocated per-cpu. So only the first tunnel would do that.

In fact that was precisely the reason why per-cpu is used in IPComp as otherwise we can just allocate normal memory.

Hmmm... indeed. Thanks for clearing this up.

#50 lore raw
In reply to Christoph Lameter
Quoted 11 lines · Christoph Lameter
Quoted 8 lines · David Miller

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 15:15:39 -0700 (PDT)

Quoted 2 lines

After boot is complete we allow the reduction of the size of the per cpu areas . Lets say we only need 128k per cpu. Then the remaining pages will be returned to the page allocator.

You don't know how much you will need. I exhausted the limit on sparc64 very late in the boot process when the last few userland services were starting up.

Quoted 6 lines

Well you would be able to specify how much will remain. If not it will just keep the 2M reserve around.

Quoted 2 lines

And if I subsequently bring up 100,000 IP tunnels, it will exhaust the per-cpu allocation area.

Each tunnel needs 4 bytes per cpu?

well, if we move last_rx to a percpu var, we need 8 bytes of percpu space per net_device :)

Quoted 4 lines
Quoted 2 lines

You have to make it fully dynamic, there is no way around it.

Na. Some reasonable upper limit needs to be set. If we set that to say 32Megabytes and do the virtual mapping then we can just populate the first 2M and only allocate the remainder if we need it. Then we need to rely on Mel's defrag stuff though defrag memory if we need it.

If a 2MB page is not available, could we revert using 4KB pages ? (like vmalloc stuff), paying an extra runtime overhead of course.

author
#51 lore raw
Quoted 7 lines · Eric Dumazet
Quoted 4 lines

Na. Some reasonable upper limit needs to be set. If we set that to say 32Megabytes and do the virtual mapping then we can just populate the first 2M and only allocate the remainder if we need it. Then we need to rely on Mel's defrag stuff though defrag memory if we need it.

If a 2MB page is not available, could we revert using 4KB pages ? (like vmalloc stuff), paying an extra runtime overhead of course.

Sure. Its going to be like vmemmap. There will be limited imposed though by the amount of virtual space available. Basically the dynamic per cpu area can be at maximum

available_virtual_space / NR_CPUS

author
#52 lore raw
In reply to Eric Dumazet

Hmmm... On x86_64 we could take 8 terabyte virtual space (bit order 43)

With the worst case scenario of 16k of cpus (bit order 16) we are looking at 43-16 = 27 ~ 128MB per cpu. Each percpu can at max be mapped by 64 pmd entries. 4k support is actually max for projected hw. So we'd get to 512M.

On IA64 we could take half of the vmemmap area which is 45 bits. So we could get up to 512MB (with 16k pages, 64k pages can get us even further) assuming we can at some point run 16 processors per node (4k is the current max which would put the limit on the per cpu area >1GB).

Lets say you have a system with 64 cpus and an area of 128M of per cpu storage. Then we are using 8GB of total memory for per cpu storage. The 128M allows us to store f.e. 16 M of word size counters.

With SLAB and the current allocpercpu you would need the following for 16M counters:

16M*32*64 (minimum alloc size of SLAB is 32 byte and we alloc via
kmalloc) for the data.

16M*64*8 for the pointer arrays. 16M allocpercpu areas for 64 processors
and a pointer size of 8 bytes.

So you would need to use 40G in current systems. The new scheme would only need 8GB for the same amount of counters.

So I think its unreasonable to assume that currently systems exist that can use more than 128m of allocpercpu space (assuming 64 cpus).

---
 include/asm-x86/pgtable_64.h |    4 ++++
 1 file changed, 4 insertions(+)
include/asm-x86/pgtable_64.h +5 -0
Index: linux-2.6/include/asm-x86/pgtable_64.h
===================================================================
@@ -138,10 +138,14 @@ static inline pte_t ptep_get_and_clear_f
 #define VMALLOC_START    _AC(0xffffc20000000000, UL)
 #define VMALLOC_END      _AC(0xffffe1ffffffffff, UL)
 #define VMEMMAP_START	 _AC(0xffffe20000000000, UL)
+#define PERCPU_START	 _AC(0xfffff20000000000, UL)
+#define PERCPU_END	 _AC(0xfffffa0000000000, UL)
 #define MODULES_VADDR    _AC(0xffffffff88000000, UL)
 #define MODULES_END      _AC(0xfffffffffff00000, UL)
 #define MODULES_LEN   (MODULES_END - MODULES_VADDR)
 
+#define PERCPU_MIN_SHIFT	PMD_SHIFT
+#define PERCPU_BITS		43
+
 #define _PAGE_BIT_PRESENT	0
 #define _PAGE_BIT_RW		1

#define _PAGE_BIT_USER 2

#53 lore raw
In reply to Christoph Lameter
Quoted 48 lines · Christoph Lameter

This patch increases the speed of the SLUB fastpath by improving the per cpu allocator and makes it usable for SLUB.

Currently allocpercpu manages arrays of pointer to per cpu objects. This means that is has to allocate the arrays and then populate them as needed with objects. Although these objects are called per cpu objects they cannot be handled in the same way as per cpu objects by adding the per cpu offset of the respective cpu.

The patch here changes that. We create a small memory pool in the percpu area and allocate from there if alloc per cpu is called. As a result we do not need the per cpu pointer arrays for each object. This reduces memory usage and also the cache foot print of allocpercpu users. Also the per cpu objects for a single processor are tightly packed next to each other decreasing cache footprint even further and making it possible to access multiple objects in the same cacheline.

SLUB has the same mechanism implemented. After fixing up the alloccpu stuff we throw the SLUB method out and use the new allocpercpu handling. Then we optimize allocpercpu addressing by adding a new function

	this_cpu_ptr()

that allows the determination of the per cpu pointer for the current processor in an more efficient way on many platforms.

This increases the speed of SLUB (and likely other kernel subsystems that benefit from the allocpercpu enhancements):

SLAB SLUB SLUB+ SLUB-o SLUB-a

   8    96      86      45      44      38	3 *
  16    84      92      49      48      43	2 *
  32    84      106     61      59      53	+++
  64    102     129     82      88      75	++
 128    147     226     188     181     176	-
 256    200     248     207     285     204	=
 512    300     301     260     209     250	+
1024    416     440     398     264     391	++
2048    720     542     530     390     511	+++
4096    1254    342     342     336     376	3 *

alloc/free test

      SLAB    SLUB    SLUB+   SLUB-o	SLUB-a
      137-146 151     68-72   68-74	56-58	3 *
Quoted 8 lines

Note: The per cpu optimization are only half way there because of the screwed up way that x86_64 handles its cpu area that causes addditional cycles to be spend by retrieving a pointer from memory and adding it to the address. The i386 code is much less cycle intensive being able to get to per cpu data using a segment prefix and if we can get that to work on x86_64 then we may be able to get the cycle count for the fastpath down to 20-30 cycles.

Really sounds good Christoph, not only for SLUB, so I guess the 32k limit is not enough because many things will use per_cpu if only per_cpu was reasonably fast (ie not so many dereferences)

I think this question already came in the past and Linus already answered it, but I again ask it. What about VM games with modern cpus (64 bits arches)

Say we reserve on x86_64 a really huge (2^32 bytes) area, and change VM layout so that each cpu maps its own per_cpu area on this area, so that the local per_cpu data sits in the same virtual address on each cpu. Then we dont need a segment prefix nor adding a 'per_cpu offset'. No need to write special asm functions to read/write/increment a per_cpu data and gcc could use normal rules for optimizations.

We only would need adding "per_cpu offset" to get data for a given cpu.

maintainer
#54 lore raw

From: Eric Dumazet <[email protected]>
Date: Thu, 01 Nov 2007 08:17:58 +0100

Quoted 4 lines

Say we reserve on x86_64 a really huge (2^32 bytes) area, and change VM layout so that each cpu maps its own per_cpu area on this area, so that the local per_cpu data sits in the same virtual address on each cpu.

This is a mechanism used partially on IA64 already.

I think you have to be very careful, and you can only use this per-cpu fixed virtual address area in extremely limited cases.

The reason is, I think the address matters, consider list heads, for example.

So you couldn't do:

	list_add(&obj->list, &per_cpu_ptr(list_head));

and use that per-cpu fixed virtual address.

IA64 seems to use it universally for every __get_cpu_var() access, so maybe it works out somehow :-)))

I guess if list modifications by remote cpus are disallowed, it would work (list traversal works because using the fixed virtual address as the list head sentinal is OK), but that is an extremely fragile assumption to base the entire mechanism upon.

author
#55 lore raw
Quoted 2 lines · David Miller

IA64 seems to use it universally for every __get_cpu_var() access, so maybe it works out somehow :-)))

IA64 does not do that. It addds the local cpu offset

#define __get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __ia64_per_cpu_var(local_per_cpu_offset)))
#define __raw_get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __ia64_per_cpu_var(local_per_cpu_offset)))

maintainer
#56 lore raw

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 06:01:14 -0700 (PDT)

Quoted 11 lines · David Miller
Quoted 3 lines

IA64 seems to use it universally for every __get_cpu_var() access, so maybe it works out somehow :-)))

IA64 does not do that. It addds the local cpu offset

#define __get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __ia64_per_cpu_var(local_per_cpu_offset)))
#define __raw_get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __ia64_per_cpu_var(local_per_cpu_offset)))

Oh I see, it's the offset itself which is accessed at the fixed virtual address slot.

author
#57 lore raw
In reply to Eric Dumazet
Quoted 11 lines · Eric Dumazet

I think this question already came in the past and Linus already answered it, but I again ask it. What about VM games with modern cpus (64 bits arches)

Say we reserve on x86_64 a really huge (2^32 bytes) area, and change VM layout so that each cpu maps its own per_cpu area on this area, so that the local per_cpu data sits in the same virtual address on each cpu. Then we dont need a segment prefix nor adding a 'per_cpu offset'. No need to write special asm functions to read/write/increment a per_cpu data and gcc could use normal rules for optimizations.

We only would need adding "per_cpu offset" to get data for a given cpu.

That is basically what IA64 is doing but it not usable because you would have addresses that mean different things on different cpus. List head for example require back pointers. If you put a listhead into such a per cpu area then you may corrupt another cpus per cpu area.

maintainer
#58 lore raw

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 05:57:12 -0700 (PDT)

Quoted 4 lines

That is basically what IA64 is doing but it not usable because you would have addresses that mean different things on different cpus. List head for example require back pointers. If you put a listhead into such a per cpu area then you may corrupt another cpus per cpu area.

Indeed, but as I pointed out in another mail it actually works if you set some rules:

1) List insert and delete is only allowed on local CPU lists.

2) List traversal is allowed on remote CPU lists.

I bet we could get all of the per-cpu users to abide by this rule if we wanted to.

The remaining issue with accessing per-cpu areas at multiple virtual addresses is D-cache aliasing.

author
#59 lore raw
Quoted 2 lines · David Miller

The remaining issue with accessing per-cpu areas at multiple virtual addresses is D-cache aliasing.

But that is not an issue for physicallly mapped caches.

maintainer
#60 lore raw

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 15:11:41 -0700 (PDT)

Quoted 6 lines · David Miller
Quoted 3 lines

The remaining issue with accessing per-cpu areas at multiple virtual addresses is D-cache aliasing.

But that is not an issue for physicallly mapped caches.

Right but I'd like to use this on sparc64 which has L1 D-cache aliasing on some chips :-)

author
#61 lore raw
Quoted 5 lines · David Miller

From: Christoph Lameter <[email protected]>
Date: Thu, 1 Nov 2007 15:11:41 -0700 (PDT)

Quoted 2 lines · David Miller
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

The remaining issue with accessing per-cpu areas at multiple virtual addresses is D-cache aliasing.

But that is not an issue for physicallly mapped caches.

Right but I'd like to use this on sparc64 which has L1 D-cache aliasing on some chips :-)

Hmmm... re my message I just send. Then we have to return the memory with the virtual address not with the physical address on sparc. May result in zones with holes though.