Skip to contentSkip to conversation

Cpu alloc V5: Replace percpu allocator in modules.c

PATCH In review Christoph Lameter 4 participants 14 messages Sep 19, 2008 – Sep 20, 2008

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

cover Cpu alloc V5: Replace percpu allocator in modules.c

Just do the bare mininum to establish a per cpu allocator. Later patchsets will gradually build out the functionality.

The most critical issue that came up on the last round is how to configure the size of the percpu area. Here we simply use a kernel parameter and use the static size of the existing percpu allocator for modules as a default.

The effect of this patchset is to make the size of percpu data for modules configurable. Its no longer fixed at 8000 bytes.

Changes:
V4->V5:
- Fix various things pointed out by Pekka.
- Remove page alignment check from module.c and put it into cpu_alloc.c

V3->V4:
- Gut patches to the bare essentials: Only replace modules.c percpu alloocator
- Make percpu reserve area configurable via a kernel parameter

#1 lore raw

PATCH 1/4 Make the per cpu reserve configurable

The per cpu reserve from which loadable modules allocate their percpu sections is currently fixed at 8000 bytes.

Add a new kernel parameter

percpu=<dynamically allocatable percpu bytes>

The per cpu reserve area will be used in following patches by the per cpu allocator.

---
 arch/ia64/include/asm/percpu.h |    1 +
 include/linux/percpu.h         |    7 ++++++-
 init/main.c                    |   13 +++++++++++++
 3 files changed, 20 insertions(+), 1 deletion(-)
include/linux/percpu.h +2 -1
Index: linux-2.6/include/linux/percpu.h
===================================================================
@@ -34,6 +34,7 @@
 #define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
 #define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
 
+extern unsigned int percpu_reserve;
 /* Enough to cover all DEFINE_PER_CPUs in kernel, including modules. */
 #ifndef PERCPU_ENOUGH_ROOM
 #ifdef CONFIG_MODULES
@@ -43,7 +44,7 @@
 #endif
 
 #define PERCPU_ENOUGH_ROOM						\
-	(__per_cpu_end - __per_cpu_start + PERCPU_MODULE_RESERVE)
+	(__per_cpu_end - __per_cpu_start + percpu_reserve)
 #endif	/* PERCPU_ENOUGH_ROOM */
 
 /*
init/main.c +13 -0
Index: linux-2.6/init/main.c
===================================================================
@@ -253,6 +253,16 @@ static int __init loglevel(char *str)
 
 early_param("loglevel", loglevel);
 
+unsigned int percpu_reserve = PERCPU_MODULE_RESERVE;
+
+static int __init init_percpu_reserve(char *str)
+{
+	get_option(&str, &percpu_reserve);
+	return 0;
+}
+
+early_param("percpu=", init_percpu_reserve);
+
 /*
  * Unknown boot options get handed to init, unless they look like
  * failed parameters
@@ -397,6 +407,9 @@ static void __init setup_per_cpu_areas(v
 
 	/* Copy section for each CPU (we discard the original) */
 	size = ALIGN(PERCPU_ENOUGH_ROOM, PAGE_SIZE);
+	printk(KERN_INFO "percpu area: %d bytes total, %d available.\n",
+			size, size - (__per_cpu_end - __per_cpu_start));
+
 	ptr = alloc_bootmem_pages(size * nr_possible_cpus);
 
 	for_each_possible_cpu(i) {
Documentation/kernel-parameters.txt +7 -0
Index: linux-2.6/Documentation/kernel-parameters.txt
===================================================================
@@ -1643,6 +1643,13 @@ and is between 256 and 4096 characters. 
 			Format: { 0 | 1 }
 			See arch/parisc/kernel/pdc_chassis.c
 
+	percpu=		Configure the number of percpu bytes that can be
+			dynamically allocated. This is used for per cpu
+			variables of modules and other dynamic per cpu data
+			structures. Creation of per cpu structures after boot
+			may fail if this is set too low.
+			Default is 8000 bytes.
+
 	pf.		[PARIDE]
 			See Documentation/paride.txt.
 
Signature · 5 lines
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#2 lore raw

Re: [patch 1/4] Make the per cpu reserve configurable

Review

Quoted 2 lines · On Fri, 19 Sep 2008 07:59:00 -0700 Christoph Lameter <[email protected]>…
(quoted diff) +2 -0
+unsigned int percpu_reserve = PERCPU_MODULE_RESERVE;
+

Is this PERCPU_MODULE_RESERVE default size is fixex to 8192 bytes both on 32bit-arch and 64bit-arch ?
How about enlarging this to twice on 64bit arch now ?

sorry for noise.

Thanks,
-Kame

Quoted 48 lines
init/main.c +11 -0
+static int __init init_percpu_reserve(char *str)
+{
+	get_option(&str, &percpu_reserve);
+	return 0;
+}
+
+early_param("percpu=", init_percpu_reserve);
+
 /*
  * Unknown boot options get handed to init, unless they look like
  * failed parameters
@@ -397,6 +407,9 @@ static void __init setup_per_cpu_areas(v
 
 	/* Copy section for each CPU (we discard the original) */
 	size = ALIGN(PERCPU_ENOUGH_ROOM, PAGE_SIZE);
+	printk(KERN_INFO "percpu area: %d bytes total, %d available.\n",
+			size, size - (__per_cpu_end - __per_cpu_start));
+
 	ptr = alloc_bootmem_pages(size * nr_possible_cpus);
 
 	for_each_possible_cpu(i) {

Index: linux-2.6/Documentation/kernel-parameters.txt
===================================================================

Documentation/kernel-parameters.txt +7 -0
@@ -1643,6 +1643,13 @@ and is between 256 and 4096 characters. 
 			Format: { 0 | 1 }
 			See arch/parisc/kernel/pdc_chassis.c
 
+	percpu=		Configure the number of percpu bytes that can be
+			dynamically allocated. This is used for per cpu
+			variables of modules and other dynamic per cpu data
+			structures. Creation of per cpu structures after boot
+			may fail if this is set too low.
+			Default is 8000 bytes.
+
 	pf.		[PARIDE]
 			See Documentation/paride.txt.
 
Signature · 7 lines
-- 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#3 lore raw

Re: [patch 1/4] Make the per cpu reserve configurable

Quoted 3 lines · KAMEZAWA Hiroyuki

Is this PERCPU_MODULE_RESERVE default size is fixex to 8192 bytes both on 32bit-arch and 64bit-arch ?

Yes.

Quoted 3 lines

How about enlarging this to twice on 64bit arch now ?

sorry for noise.

No actually a good idea to discuss the limit here. Maybe use 10000 for 32 bit and 15000 for 64 bit? Many percpu variables are counters that may be integers.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw

PATCH 2/4 percpu: Rename variables PERCPU_ENOUGH_ROOM -> PERCPU_AREA_SIZE

In reply to Christoph Lameter

Rename PERCPU_ENOUGH_ROOM to PERCPU_AREA_SIZE since its really specifying the size of the percpu areas.

Rename PERCPU_MODULE_RESERVE to PERCPU_RESERVE_SIZE in anticipation of more general use of that reserve.

---
 arch/ia64/include/asm/percpu.h |    2 +-
 arch/powerpc/kernel/setup_64.c |    4 ++--
 arch/sparc64/kernel/smp.c      |    2 +-
 arch/x86/kernel/setup_percpu.c |    3 +--
 include/linux/percpu.h         |   10 +++++-----
 init/main.c                    |    4 ++--
 kernel/lockdep.c               |    2 +-
 kernel/module.c                |    2 +-
 8 files changed, 14 insertions(+), 15 deletions(-)
arch/ia64/include/asm/percpu.h +1 -1
Index: linux-2.6/arch/ia64/include/asm/percpu.h
===================================================================
@@ -6,7 +6,7 @@
  *	David Mosberger-Tang <[email protected]>
  */
 
-#define PERCPU_ENOUGH_ROOM PERCPU_PAGE_SIZE
+#define PERCPU_AREA_SIZE PERCPU_PAGE_SIZE
 
 #ifdef __ASSEMBLY__
 # define THIS_CPU(var)	(per_cpu__##var)  /* use this to mark accesses to per-CPU variables... */
include/linux/percpu.h +5 -5
Index: linux-2.6/include/linux/percpu.h
===================================================================
@@ -36,16 +36,16 @@
 
 extern unsigned int percpu_reserve;
 /* Enough to cover all DEFINE_PER_CPUs in kernel, including modules. */
-#ifndef PERCPU_ENOUGH_ROOM
+#ifndef PERCPU_AREA_SIZE
 #ifdef CONFIG_MODULES
-#define PERCPU_MODULE_RESERVE	8192
+#define PERCPU_RESERVE_SIZE	8192
 #else
-#define PERCPU_MODULE_RESERVE	0
+#define PERCPU_RESERVE_SIZE	0
 #endif
 
-#define PERCPU_ENOUGH_ROOM						\
+#define PERCPU_AREA_SIZE						\
 	(__per_cpu_end - __per_cpu_start + percpu_reserve)
-#endif	/* PERCPU_ENOUGH_ROOM */
+#endif	/* PERCPU_AREA_SIZE */
 
 /*
  * Must be an lvalue. Since @var must be a simple identifier,
arch/powerpc/kernel/setup_64.c +2 -2
Index: linux-2.6/arch/powerpc/kernel/setup_64.c
===================================================================
@@ -599,8 +599,8 @@ void __init setup_per_cpu_areas(void)
 	/* Copy section for each CPU (we discard the original) */
 	size = ALIGN(__per_cpu_end - __per_cpu_start, PAGE_SIZE);
 #ifdef CONFIG_MODULES
-	if (size < PERCPU_ENOUGH_ROOM)
-		size = PERCPU_ENOUGH_ROOM;
+	if (size < PERCPU_AREA_SIZE)
+		size = PERCPU_AREA_SIZE;
 #endif
 
 	for_each_possible_cpu(i) {
arch/sparc64/kernel/smp.c +1 -1
Index: linux-2.6/arch/sparc64/kernel/smp.c
===================================================================
@@ -1386,7 +1386,7 @@ void __init real_setup_per_cpu_areas(voi
 	char *ptr;
 
 	/* Copy section for each CPU (we discard the original) */
-	goal = PERCPU_ENOUGH_ROOM;
+	goal = PERCPU_AREA_SIZE;
 
 	__per_cpu_shift = PAGE_SHIFT;
 	for (size = PAGE_SIZE; size < goal; size <<= 1UL)
arch/x86/kernel/setup_percpu.c +1 -2
Index: linux-2.6/arch/x86/kernel/setup_percpu.c
===================================================================
@@ -140,7 +140,7 @@ static void __init setup_cpu_pda_map(voi
  */
 void __init setup_per_cpu_areas(void)
 {
-	ssize_t size = PERCPU_ENOUGH_ROOM;
+	ssize_t size = PERCPU_AREA_SIZE;
 	char *ptr;
 	int cpu;
 
@@ -148,7 +148,6 @@ void __init setup_per_cpu_areas(void)
 	setup_cpu_pda_map();
 
 	/* Copy section for each CPU (we discard the original) */
-	size = PERCPU_ENOUGH_ROOM;
 	printk(KERN_INFO "PERCPU: Allocating %zd bytes of per cpu data\n",
 			  size);
 
init/main.c +2 -2
Index: linux-2.6/init/main.c
===================================================================
@@ -253,7 +253,7 @@ static int __init loglevel(char *str)
 
 early_param("loglevel", loglevel);
 
-unsigned int percpu_reserve = PERCPU_MODULE_RESERVE;
+unsigned int percpu_reserve = PERCPU_RESERVE_SIZE;
 
 static int __init init_percpu_reserve(char *str)
 {
@@ -406,7 +406,7 @@ static void __init setup_per_cpu_areas(v
 	unsigned long nr_possible_cpus = num_possible_cpus();
 
 	/* Copy section for each CPU (we discard the original) */
-	size = ALIGN(PERCPU_ENOUGH_ROOM, PAGE_SIZE);
+	size = ALIGN(PERCPU_AREA_SIZE, PAGE_SIZE);
 	printk(KERN_INFO "percpu area: %d bytes total, %d available.\n",
 			size, size - (__per_cpu_end - __per_cpu_start));
 
kernel/lockdep.c +1 -1
Index: linux-2.6/kernel/lockdep.c
===================================================================
@@ -639,7 +639,7 @@ static int static_obj(void *obj)
 	 */
 	for_each_possible_cpu(i) {
 		start = (unsigned long) &__per_cpu_start + per_cpu_offset(i);
-		end   = (unsigned long) &__per_cpu_start + PERCPU_ENOUGH_ROOM
+		end   = (unsigned long) &__per_cpu_start + PERCPU_AREA_SIZE
 					+ per_cpu_offset(i);
 
 		if ((addr >= start) && (addr < end))
kernel/module.c +1 -1
Index: linux-2.6/kernel/module.c
===================================================================
@@ -476,7 +476,7 @@ static int percpu_modinit(void)
 	/* Static in-kernel percpu data (used). */
 	pcpu_size[0] = -(__per_cpu_end-__per_cpu_start);
 	/* Free room. */
-	pcpu_size[1] = PERCPU_ENOUGH_ROOM + pcpu_size[0];
+	pcpu_size[1] = PERCPU_AREA_SIZE + pcpu_size[0];
 	if (pcpu_size[1] < 0) {
 		printk(KERN_ERR "No per-cpu room for modules.\n");
 		pcpu_num_used = 1;
#5 lore raw

PATCH 3/4 cpu alloc: The allocator

In reply to Christoph Lameter

The per cpu allocator allows dynamic allocation of memory on all processors simultaneously. A bitmap is used to track used areas. The allocator implements tight packing to reduce the cache footprint and increase speed since cacheline contention is typically not a concern for memory mainly used by a single cpu. Small objects will fill up gaps left by larger allocations that required alignments.

The size of the cpu_alloc area can be changed via the percpu=xxx kernel parameter.

---
 include/linux/percpu.h |   46 ++++++++++++
 include/linux/vmstat.h |    2 
 mm/Makefile            |    2 
 mm/cpu_alloc.c         |  181 +++++++++++++++++++++++++++++++++++++++++++++++++
 mm/vmstat.c            |    1 
 5 files changed, 230 insertions(+), 2 deletions(-)
 create mode 100644 include/linux/cpu_alloc.h
 create mode 100644 mm/cpu_alloc.c
include/linux/vmstat.h +1 -1
Index: linux-2.6/include/linux/vmstat.h
===================================================================
@@ -37,7 +37,7 @@
 		FOR_ALL_ZONES(PGSCAN_KSWAPD),
 		FOR_ALL_ZONES(PGSCAN_DIRECT),
 		PGINODESTEAL, SLABS_SCANNED, KSWAPD_STEAL, KSWAPD_INODESTEAL,
-		PAGEOUTRUN, ALLOCSTALL, PGROTATED,
+		PAGEOUTRUN, ALLOCSTALL, PGROTATED, CPU_BYTES,
 #ifdef CONFIG_HUGETLB_PAGE
 		HTLB_BUDDY_PGALLOC, HTLB_BUDDY_PGALLOC_FAIL,
 #endif
mm/Makefile +1 -1
Index: linux-2.6/mm/Makefile
===================================================================
@@ -11,7 +11,7 @@
 			   maccess.o page_alloc.o page-writeback.o pdflush.o \
 			   readahead.o swap.o truncate.o vmscan.o \
 			   prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \
-			   page_isolation.o mm_init.o $(mmu-y)
+			   page_isolation.o mm_init.o cpu_alloc.o $(mmu-y)
 
 obj-$(CONFIG_PROC_PAGE_MONITOR) += pagewalk.o
 obj-$(CONFIG_BOUNCE)	+= bounce.o
mm/cpu_alloc.c new file +182 -0

182 lines not on this page · raw message

mm/vmstat.c +1 -0
Index: linux-2.6/mm/vmstat.c
===================================================================
@@ -671,6 +671,7 @@
 	"allocstall",
 
 	"pgrotated",
+	"cpu_bytes",
 #ifdef CONFIG_HUGETLB_PAGE
 	"htlb_buddy_alloc_success",
 	"htlb_buddy_alloc_fail",
include/linux/percpu.h +48 -0

52 lines not on this page · raw message

init/main.c +7 -2
Index: linux-2.6/init/main.c
===================================================================
@@ -261,7 +261,7 @@
 	return 0;
 }
 
-early_param("percpu=", init_percpu_reserve);
+early_param("percpu", init_percpu_reserve);
 
 /*
  * Unknown boot options get handed to init, unless they look like
@@ -368,7 +368,11 @@
 #define smp_init()	do { } while (0)
 #endif
 
-static inline void setup_per_cpu_areas(void) { }
+static inline void setup_per_cpu_areas(void)
+{
+	cpu_alloc_init();
+}
+
 static inline void setup_nr_cpu_ids(void) { }
 static inline void smp_prepare_cpus(unsigned int maxcpus) { }
 
@@ -405,6 +409,7 @@
 	char *ptr;
 	unsigned long nr_possible_cpus = num_possible_cpus();
 
+	cpu_alloc_init();
 	/* Copy section for each CPU (we discard the original) */
 	size = ALIGN(PERCPU_AREA_SIZE, PAGE_SIZE);
 	printk(KERN_INFO "percpu area: %d bytes total, %d available.\n",
arch/x86/kernel/setup_percpu.c +1 -0
Index: linux-2.6/arch/x86/kernel/setup_percpu.c
===================================================================
@@ -144,6 +144,7 @@
 	char *ptr;
 	int cpu;
 
+	cpu_alloc_init();
 	/* Setup cpu_pda map */
 	setup_cpu_pda_map();
 
arch/ia64/kernel/setup.c +1 -0
Index: linux-2.6/arch/ia64/kernel/setup.c
===================================================================
@@ -842,6 +842,7 @@
 #ifdef CONFIG_ACPI_HOTPLUG_CPU
 	prefill_possible_map();
 #endif
+	cpu_alloc_init();
 }
 
 /*
arch/powerpc/kernel/setup_64.c +1 -0
Index: linux-2.6/arch/powerpc/kernel/setup_64.c
===================================================================
@@ -611,6 +611,7 @@
 		paca[i].data_offset = ptr - __per_cpu_start;
 		memcpy(ptr, __per_cpu_start, __per_cpu_end - __per_cpu_start);
 	}
+	cpu_alloc_init();
 }
 #endif
 
arch/sparc64/mm/init.c +1 -0
Index: linux-2.6/arch/sparc64/mm/init.c
===================================================================
@@ -1644,6 +1644,7 @@
 /* Dummy function */
 void __init setup_per_cpu_areas(void)
 {
+	cpu_alloc_init();
 }
 
 void __init paging_init(void)
#6 lore raw

Re: [patch 3/4] cpu alloc: The allocator

mm/cpu_alloc.c line 76

Quoted 7 lines
mm/cpu_alloc.c new file +7 -0
+/*
+ * Allocate an object of a certain size
+ *
+ * Returns a special pointer that can be used with CPU_PTR to find the
+ * address of the object for a certain cpu.
+ */
+void *cpu_alloc(unsigned long size, gfp_t gfpflags, unsigned long align)

cpu_alloc is good name?
I think some person suspect cpu-hotplug related function.

per_cpu_alloc() or cpu_mem_alloc() are wrong?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw

Re: [patch 3/4] cpu alloc: The allocator

In reply to Christoph Lameter

Christoph Lameter a ecrit :

Quoted 130 lines

The per cpu allocator allows dynamic allocation of memory on all processors simultaneously. A bitmap is used to track used areas. The allocator implements tight packing to reduce the cache footprint and increase speed since cacheline contention is typically not a concern for memory mainly used by a single cpu. Small objects will fill up gaps left by larger allocations that required alignments.

The size of the cpu_alloc area can be changed via the percpu=xxx kernel parameter.

---
 include/linux/percpu.h |   46 ++++++++++++
 include/linux/vmstat.h |    2 
 mm/Makefile            |    2 
 mm/cpu_alloc.c         |  181 +++++++++++++++++++++++++++++++++++++++++++++++++
 mm/vmstat.c            |    1 
 5 files changed, 230 insertions(+), 2 deletions(-)
 create mode 100644 include/linux/cpu_alloc.h
 create mode 100644 mm/cpu_alloc.c

Index: linux-2.6/include/linux/vmstat.h
===================================================================

include/linux/vmstat.h +1 -1
@@ -37,7 +37,7 @@
 		FOR_ALL_ZONES(PGSCAN_KSWAPD),
 		FOR_ALL_ZONES(PGSCAN_DIRECT),
 		PGINODESTEAL, SLABS_SCANNED, KSWAPD_STEAL, KSWAPD_INODESTEAL,
-		PAGEOUTRUN, ALLOCSTALL, PGROTATED,
+		PAGEOUTRUN, ALLOCSTALL, PGROTATED, CPU_BYTES,
 #ifdef CONFIG_HUGETLB_PAGE
 		HTLB_BUDDY_PGALLOC, HTLB_BUDDY_PGALLOC_FAIL,
 #endif

Index: linux-2.6/mm/Makefile
===================================================================

mm/Makefile +1 -1
@@ -11,7 +11,7 @@
 			   maccess.o page_alloc.o page-writeback.o pdflush.o \
 			   readahead.o swap.o truncate.o vmscan.o \
 			   prio_tree.o util.o mmzone.o vmstat.o backing-dev.o \
-			   page_isolation.o mm_init.o $(mmu-y)
+			   page_isolation.o mm_init.o cpu_alloc.o $(mmu-y)
 
 obj-$(CONFIG_PROC_PAGE_MONITOR) += pagewalk.o
 obj-$(CONFIG_BOUNCE)	+= bounce.o

Index: linux-2.6/mm/cpu_alloc.c
===================================================================

mm/cpu_alloc.c new file +77 -0
@@ -0,0 +1,182 @@
+/*
+ * Cpu allocator - Manage objects allocated for each processor
+ *
+ * (C) 2008 SGI, Christoph Lameter <[email protected]>
+ * 	Basic implementation with allocation and free from a dedicated per
+ * 	cpu area.
+ *
+ * The per cpu allocator allows a dynamic allocation of a piece of memory on
+ * every processor. A bitmap is used to track used areas.
+ * The allocator implements tight packing to reduce the cache footprint
+ * and increase speed since cacheline contention is typically not a concern
+ * for memory mainly used by a single cpu. Small objects will fill up gaps
+ * left by larger allocations that required alignments.
+ */
+#include <linux/mm.h>
+#include <linux/mmzone.h>
+#include <linux/module.h>
+#include <linux/percpu.h>
+#include <linux/bitmap.h>
+#include <asm/sections.h>
+#include <linux/bootmem.h>
+
+/*
+ * Basic allocation unit. A bit map is created to track the use of each
+ * UNIT_SIZE element in the cpu area.
+ */
+#define UNIT_TYPE int
+#define UNIT_SIZE sizeof(UNIT_TYPE)
+
+int units;	/* Actual available units */
+
+/*
+ * How many units are needed for an object of a given size
+ */
+static int size_to_units(unsigned long size)
+{
+	return DIV_ROUND_UP(size, UNIT_SIZE);
+}
+
+/*
+ * Lock to protect the bitmap and the meta data for the cpu allocator.
+ */
+static DEFINE_SPINLOCK(cpu_alloc_map_lock);
+static unsigned long *cpu_alloc_map;
+static int nr_units;		/* Number of available units */
+static int first_free;		/* First known free unit */
+
+/*
+ * Mark an object as used in the cpu_alloc_map
+ *
+ * Must hold cpu_alloc_map_lock
+ */
+static void set_map(int start, int length)
+{
+	while (length-- > 0)
+		__set_bit(start++, cpu_alloc_map);
+}
+
+/*
+ * Mark an area as freed.
+ *
+ * Must hold cpu_alloc_map_lock
+ */
+static void clear_map(int start, int length)
+{
+	while (length-- > 0)
+		__clear_bit(start++, cpu_alloc_map);
+}
+
+/*
+ * Allocate an object of a certain size
+ *
+ * Returns a special pointer that can be used with CPU_PTR to find the
+ * address of the object for a certain cpu.
+ */
+void *cpu_alloc(unsigned long size, gfp_t gfpflags, unsigned long align)
+{

mm/cpu_alloc.c line 87

Quoted 10 lines
mm/cpu_alloc.c new file +10 -0
+	unsigned long start;
+	int units = size_to_units(size);
+	void *ptr;
+	int first;
+	unsigned long flags;
+
+	if (!size)
+		return ZERO_SIZE_PTR;
+
+	WARN_ON(align > PAGE_SIZE);

if (align < UNIT_SIZE)

	align = UNIT_SIZE;
Quoted 10 lines
mm/cpu_alloc.c new file +10 -0
+
+	spin_lock_irqsave(&cpu_alloc_map_lock, flags);
+
+	first = 1;
+	start = first_free;
+
+	for ( ; ; ) {
+
+		start = find_next_zero_bit(cpu_alloc_map, nr_units, start);
+		if (start >= nr_units)

mm/cpu_alloc.c line 107

Quoted 10 lines
mm/cpu_alloc.c new file +10 -0
+			goto out_of_memory;
+
+		if (first)
+			first_free = start;
+
+		/*
+		 * Check alignment and that there is enough space after
+		 * the starting unit.
+		 */
+		if (start % (align / UNIT_SIZE) == 0 &&

or else... divide per 0 ?

mm/cpu_alloc.c line 113

Quoted 6 lines
mm/cpu_alloc.c new file +6 -0
+			find_next_bit(cpu_alloc_map, nr_units, start + 1)
+					>= start + units)
+				break;
+		start++;
+		first = 0;
+	}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#8 lore raw

Re: [patch 3/4] cpu alloc: The allocator

Quoted 12 lines · Eric Dumazet

mm/cpu_alloc.c line 87

Quoted 10 lines
mm/cpu_alloc.c new file +10 -0
+    unsigned long start;
+    int units = size_to_units(size);
+    void *ptr;
+    int first;
+    unsigned long flags;
+
+    if (!size)
+        return ZERO_SIZE_PTR;
+
+    WARN_ON(align > PAGE_SIZE);

if (align < UNIT_SIZE)

Quoted 1 line
    align = UNIT_SIZE;

size_to_units() does round up:

/*
* How many units are needed for an object of a given size */
static int size_to_units(unsigned long size)

{
        return DIV_ROUND_UP(size, UNIT_SIZE);
}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw

Re: [patch 3/4] cpu alloc: The allocator

Completely wrong. We need this patch that Eric suggested:

Subject: cpu_alloc: Allow alignment < UNIT_SIZE

Limit the minimum alignment to UNIT_SIZE.

mm/cpu_alloc.c +3 -0
Index: linux-2.6/mm/cpu_alloc.c
===================================================================
@@ -86,6 +86,9 @@
 
 	WARN_ON(align > PAGE_SIZE);
 
+	if (align < UNIT_SIZE)
+		align = UNIT_SIZE;
+
 	spin_lock_irqsave(&cpu_alloc_map_lock, flags);
 
 	first = 1

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#10 lore raw

Re: [patch 3/4] cpu alloc: The allocator

In reply to Christoph Lameter

Duh. A cast went missing which results in a pointer calculation going haywire.

Signed-off-by <[email protected]>
mm/cpu_alloc.c +1 -1
Index: linux-2.6/mm/cpu_alloc.c
===================================================================
@@ -126,7 +126,7 @@
 
 	spin_unlock_irqrestore(&cpu_alloc_map_lock, flags);
 
-	ptr = __per_cpu_end + start;
+	ptr = (int *)__per_cpu_end + start;
 
 	printk(KERN_INFO "%d per cpu units allocated at offset %lx address %p\n",
 		units, start, ptr)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#11 lore raw

PATCH 4/4 cpu alloc: Use cpu allocator instead of the builtin modules per cpu allocator

In reply to Christoph Lameter

Remove the builtin per cpu allocator from modules.c and use cpu_alloc instead.

---
 include/linux/module.h |    1 
 kernel/module.c        |  178 ++++---------------------------------------------
 2 files changed, 17 insertions(+), 162 deletions(-)
kernel/module.c +11 -162
Index: linux-2.6/kernel/module.c
===================================================================
@@ -337,121 +337,6 @@
 	return NULL;
 }
 
-#ifdef CONFIG_SMP
-/* Number of blocks used and allocated. */
-static unsigned int pcpu_num_used, pcpu_num_allocated;
-/* Size of each block.  -ve means used. */
-static int *pcpu_size;
-
-static int split_block(unsigned int i, unsigned short size)
-{
-	/* Reallocation required? */
-	if (pcpu_num_used + 1 > pcpu_num_allocated) {
-		int *new;
-
-		new = krealloc(pcpu_size, sizeof(new[0])*pcpu_num_allocated*2,
-			       GFP_KERNEL);
-		if (!new)
-			return 0;
-
-		pcpu_num_allocated *= 2;
-		pcpu_size = new;
-	}
-
-	/* Insert a new subblock */
-	memmove(&pcpu_size[i+1], &pcpu_size[i],
-		sizeof(pcpu_size[0]) * (pcpu_num_used - i));
-	pcpu_num_used++;
-
-	pcpu_size[i+1] -= size;
-	pcpu_size[i] = size;
-	return 1;
-}
-
-static inline unsigned int block_size(int val)
-{
-	if (val < 0)
-		return -val;
-	return val;
-}
-
-static void *percpu_modalloc(unsigned long size, unsigned long align,
-			     const char *name)
-{
-	unsigned long extra;
-	unsigned int i;
-	void *ptr;
-
-	if (align > PAGE_SIZE) {
-		printk(KERN_WARNING "%s: per-cpu alignment %li > %li\n",
-		       name, align, PAGE_SIZE);
-		align = PAGE_SIZE;
-	}
-
-	ptr = __per_cpu_start;
-	for (i = 0; i < pcpu_num_used; ptr += block_size(pcpu_size[i]), i++) {
-		/* Extra for alignment requirement. */
-		extra = ALIGN((unsigned long)ptr, align) - (unsigned long)ptr;
-		BUG_ON(i == 0 && extra != 0);
-
-		if (pcpu_size[i] < 0 || pcpu_size[i] < extra + size)
-			continue;
-
-		/* Transfer extra to previous block. */
-		if (pcpu_size[i-1] < 0)
-			pcpu_size[i-1] -= extra;
-		else
-			pcpu_size[i-1] += extra;
-		pcpu_size[i] -= extra;
-		ptr += extra;
-
-		/* Split block if warranted */
-		if (pcpu_size[i] - size > sizeof(unsigned long))
-			if (!split_block(i, size))
-				return NULL;
-
-		/* Mark allocated */
-		pcpu_size[i] = -pcpu_size[i];
-		return ptr;
-	}
-
-	printk(KERN_WARNING "Could not allocate %lu bytes percpu data\n",
-	       size);
-	return NULL;
-}
-
-static void percpu_modfree(void *freeme)
-{
-	unsigned int i;
-	void *ptr = __per_cpu_start + block_size(pcpu_size[0]);
-
-	/* First entry is core kernel percpu data. */
-	for (i = 1; i < pcpu_num_used; ptr += block_size(pcpu_size[i]), i++) {
-		if (ptr == freeme) {
-			pcpu_size[i] = -pcpu_size[i];
-			goto free;
-		}
-	}
-	BUG();
-
- free:
-	/* Merge with previous? */
-	if (pcpu_size[i-1] >= 0) {
-		pcpu_size[i-1] += pcpu_size[i];
-		pcpu_num_used--;
-		memmove(&pcpu_size[i], &pcpu_size[i+1],
-			(pcpu_num_used - i) * sizeof(pcpu_size[0]));
-		i--;
-	}
-	/* Merge with next? */
-	if (i+1 < pcpu_num_used && pcpu_size[i+1] >= 0) {
-		pcpu_size[i] += pcpu_size[i+1];
-		pcpu_num_used--;
-		memmove(&pcpu_size[i+1], &pcpu_size[i+2],
-			(pcpu_num_used - (i+1)) * sizeof(pcpu_size[0]));
-	}
-}
-
 static unsigned int find_pcpusec(Elf_Ehdr *hdr,
 				 Elf_Shdr *sechdrs,
 				 const char *secstrings)
@@ -467,48 +352,6 @@
 		memcpy(pcpudest + per_cpu_offset(cpu), from, size);
 }
 
-static int percpu_modinit(void)
-{
-	pcpu_num_used = 2;
-	pcpu_num_allocated = 2;
-	pcpu_size = kmalloc(sizeof(pcpu_size[0]) * pcpu_num_allocated,
-			    GFP_KERNEL);
-	/* Static in-kernel percpu data (used). */
-	pcpu_size[0] = -(__per_cpu_end-__per_cpu_start);
-	/* Free room. */
-	pcpu_size[1] = PERCPU_AREA_SIZE + pcpu_size[0];
-	if (pcpu_size[1] < 0) {
-		printk(KERN_ERR "No per-cpu room for modules.\n");
-		pcpu_num_used = 1;
-	}
-
-	return 0;
-}
-__initcall(percpu_modinit);
-#else /* ... !CONFIG_SMP */
-static inline void *percpu_modalloc(unsigned long size, unsigned long align,
-				    const char *name)
-{
-	return NULL;
-}
-static inline void percpu_modfree(void *pcpuptr)
-{
-	BUG();
-}
-static inline unsigned int find_pcpusec(Elf_Ehdr *hdr,
-					Elf_Shdr *sechdrs,
-					const char *secstrings)
-{
-	return 0;
-}
-static inline void percpu_modcopy(void *pcpudst, const void *src,
-				  unsigned long size)
-{
-	/* pcpusec should be 0, and size of that section should be 0. */
-	BUG_ON(size != 0);
-}
-#endif /* CONFIG_SMP */
-
 #define MODINFO_ATTR(field)	\
 static void setup_modinfo_##field(struct module *mod, const char *s)  \
 {                                                                     \
@@ -1433,7 +1276,7 @@
 	module_free(mod, mod->module_init);
 	kfree(mod->args);
 	if (mod->percpu)
-		percpu_modfree(mod->percpu);
+		cpu_free(mod->percpu, mod->percpu_size);
 
 	/* Free lock-classes: */
 	lockdep_free_key_range(mod->module_core, mod->core_size);
@@ -1833,6 +1676,7 @@
 	unsigned int markersstringsindex;
 	struct module *mod;
 	long err = 0;
+	unsigned long percpu_size = 0;
 	void *percpu = NULL, *ptr = NULL; /* Stops spurious gcc warning */
 	struct exception_table_entry *extable;
 	mm_segment_t old_fs;
@@ -1981,15 +1825,20 @@
 
 	if (pcpuindex) {
 		/* We have a special allocation for this section. */
-		percpu = percpu_modalloc(sechdrs[pcpuindex].sh_size,
-					 sechdrs[pcpuindex].sh_addralign,
-					 mod->name);
+		unsigned long align = sechdrs[pcpuindex].sh_addralign;
+
+		percpu_size = sechdrs[pcpuindex].sh_size;
+		percpu = cpu_alloc(percpu_size, GFP_KERNEL|__GFP_ZERO, align);
+		if (!percpu)
+			printk(KERN_WARNING "Could not allocate %lu bytes percpu data\n",
+	      							percpu_size);
 		if (!percpu) {
 			err = -ENOMEM;
 			goto free_mod;
 		}
 		sechdrs[pcpuindex].sh_flags &= ~(unsigned long)SHF_ALLOC;
 		mod->percpu = percpu;
+		mod->percpu_size = percpu_size;
 	}
 
 	/* Determine total sizes, and put offsets in sh_entsize.  For now
@@ -2243,7 +2092,7 @@
 	module_free(mod, mod->module_core);
  free_percpu:
 	if (percpu)
-		percpu_modfree(percpu);
+		cpu_free(percpu, percpu_size);
  free_mod:
 	kfree(args);
  free_hdr:
include/linux/module.h +1 -0
Index: linux-2.6/include/linux/module.h
===================================================================
@@ -323,6 +323,7 @@
 
 	/* Per-cpu data. */
 	void *percpu;
+	int percpu_size;
 
 	/* The command line arguments (may be mangled).  People like
 	   keeping pointers to this stuff */
#12 lore raw
In reply to Christoph Lameter

Hi Cristoph,

Quoted 9 lines

Just do the bare mininum to establish a per cpu allocator. Later patchsets will gradually build out the functionality.

The most critical issue that came up on the last round is how to configure the size of the percpu area. Here we simply use a kernel parameter and use the static size of the existing percpu allocator for modules as a default.

The effect of this patchset is to make the size of percpu data for modules configurable. Its no longer fixed at 8000 bytes.

I don't know so much this area.
Could you please what are the problem that you think about?

performance?
fixed-size cause per-cpu starvation by huge user?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#13 lore raw
Quoted 3 lines · KOSAKI Motohiro

I don't know so much this area.
Could you please what are the problem that you think about?

F.e. Someone is loading lots of modules with lots of percpu data. This will currently fail if there are more than 8000 bytes allocated. With the percpu option the size can be configured on bootup. A minor thing but the allocator can later be used for other things. See the full cpu alloc patchsets that have been posted before (last one in May).

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>