Skip to contentSkip to conversation

x86, percpu: implement and use reserved percpu alloc

GIT PULL In review Tejun Heo 4 participants 13 messages Mar 6, 2009 – Mar 8, 2009

Subsystems (3)
lore
Share & tools
lore

Tejun Heo

author maintainer
#0 lore raw

Hello,

Please pull from the following git vector.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git tj-percpu

This patchset implements reserved percpu alloc from the first chunk and uses it for static perpcu variables in modules on x86_64. This is necessary because x86_64 expects symbols to be relocatable using 32bit relocations but dynamic percpu allocation does not guarantee it leading to relocation overflow during module load.

This can happen to any of the x86_64 percpu first chunk allocators but the embedding allocator is most likely to trigger it as the first chunk is located in the normal kernel memory allocation area while all the other chunks will be allocated in the vmalloc area which is very far from the first chunk.

This type of reservation will be necessary for other architectures too where addressing mode used for percpu acess uses offsets with limited range to access symbols (e.g. ia64 w/ the magic 64k percpu page).

This problem was first reported and bisected by Mike Galbraith[2].

This patchset is against the current x86/core/percpu[1] and contains the following patches.

0001-percpu-clean-up-percpu-constants.patch 0002-percpu-cosmetic-renames-in-pcpu_setup_first_chunk.patch 0003-percpu-improve-first-chunk-initial-area-map-handlin.patch 0004-percpu-use-negative-for-auto-for-pcpu_setup_first_c.patch 0005-x86-make-embedding-percpu-allocator-return-excessiv.patch 0006-percpu-add-an-indirection-ptr-for-chunk-page-map-ac.patch 0007-percpu-module-implement-reserved-allocation-and-us.patch 0008-x86-percpu-setup-reserved-percpu-area-for-x86_64.patch

0001-0004 makes misc updates to percpu in preparation of later changes. 0005 fixes a case where excessive memory can be set aside in the first chunk when using the embedding allocator. 0006-0008 implement reserved allocation and use it for x86_64.

Bryan Wu, can you please review #0001? It contains small change in blackfin. It's mostly trivial but just in case.

Diffstat follows.

 arch/blackfin/include/asm/percpu.h |   10 -
 arch/x86/kernel/setup_percpu.c     |   73 +++++++---
 include/linux/percpu.h             |   60 +++-----
 kernel/module.c                    |    2 
 mm/percpu.c                        |  267 ++++++++++++++++++++++++++-----------
 5 files changed, 274 insertions(+), 138 deletions(-)

Thanks.

--
tejun

[1] f254f3909efaf59ca2d0f408de2d044dace60706
[2] http://thread.gmane.org/gmane.linux.kernel/801799

author maintainer
#1 lore raw

PATCH 1/8 percpu: clean up percpu constants

Impact: cleaup

Make the following cleanups.

* There isn't much arch-specific about PERCPU_MODULE_RESERVE. Always define it whether arch overrides PERCPU_ENOUGH_ROOM or not.

* blackfin overrides PERCPU_ENOUGH_ROOM to align static area size. Do it by default.

* percpu allocation sizes doesn't have much to do with the page size. Don't use PAGE_SHIFT in their definition.

---
 arch/blackfin/include/asm/percpu.h |   10 ----------
 include/linux/percpu.h             |   24 +++++++++++++-----------
 2 files changed, 13 insertions(+), 21 deletions(-)
arch/blackfin/include/asm/percpu.h +0 -10
index 797c0c1..c94c7bc 100644
@@ -3,14 +3,4 @@
 
 #include <asm-generic/percpu.h>
 
-#ifdef CONFIG_MODULES
-#define PERCPU_MODULE_RESERVE 8192
-#else
-#define PERCPU_MODULE_RESERVE 0
-#endif
-
-#define PERCPU_ENOUGH_ROOM \
-	(ALIGN(__per_cpu_end - __per_cpu_start, SMP_CACHE_BYTES) + \
-	 PERCPU_MODULE_RESERVE)
-
 #endif	/* __ARCH_BLACKFIN_PERCPU__ */
include/linux/percpu.h +13 -11
index 545b068..2d34b03 100644
@@ -5,6 +5,7 @@
 #include <linux/slab.h> /* For kmalloc() */
 #include <linux/smp.h>
 #include <linux/cpumask.h>
+#include <linux/pfn.h>
 
 #include <asm/percpu.h>
 
@@ -52,17 +53,18 @@
 #define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
 #define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
 
-/* Enough to cover all DEFINE_PER_CPUs in kernel, including modules. */
-#ifndef PERCPU_ENOUGH_ROOM
+/* enough to cover all DEFINE_PER_CPUs in modules */
 #ifdef CONFIG_MODULES
-#define PERCPU_MODULE_RESERVE	8192
+#define PERCPU_MODULE_RESERVE		(8 << 10)
 #else
-#define PERCPU_MODULE_RESERVE	0
+#define PERCPU_MODULE_RESERVE		0
 #endif
 
+#ifndef PERCPU_ENOUGH_ROOM
 #define PERCPU_ENOUGH_ROOM						\
-	(__per_cpu_end - __per_cpu_start + PERCPU_MODULE_RESERVE)
-#endif	/* PERCPU_ENOUGH_ROOM */
+	(ALIGN(__per_cpu_end - __per_cpu_start, SMP_CACHE_BYTES) +	\
+	 PERCPU_MODULE_RESERVE)
+#endif
 
 /*
  * Must be an lvalue. Since @var must be a simple identifier,
@@ -79,7 +81,7 @@
 #ifdef CONFIG_HAVE_DYNAMIC_PER_CPU_AREA
 
 /* minimum unit size, also is the maximum supported allocation size */
-#define PCPU_MIN_UNIT_SIZE		(16UL << PAGE_SHIFT)
+#define PCPU_MIN_UNIT_SIZE		PFN_ALIGN(64 << 10)
 
 /*
  * PERCPU_DYNAMIC_RESERVE indicates the amount of free area to piggy
@@ -96,15 +98,15 @@
 #ifndef PERCPU_DYNAMIC_RESERVE
 #  if BITS_PER_LONG > 32
 #    ifdef CONFIG_MODULES
-#      define PERCPU_DYNAMIC_RESERVE	(6 << PAGE_SHIFT)
+#      define PERCPU_DYNAMIC_RESERVE	(24 << 10)
 #    else
-#      define PERCPU_DYNAMIC_RESERVE	(4 << PAGE_SHIFT)
+#      define PERCPU_DYNAMIC_RESERVE	(16 << 10)
 #    endif
 #  else
 #    ifdef CONFIG_MODULES
-#      define PERCPU_DYNAMIC_RESERVE	(4 << PAGE_SHIFT)
+#      define PERCPU_DYNAMIC_RESERVE	(16 << 10)
 #    else
-#      define PERCPU_DYNAMIC_RESERVE	(2 << PAGE_SHIFT)
+#      define PERCPU_DYNAMIC_RESERVE	(8 << 10)
 #    endif
 #  endif
 #endif	/* PERCPU_DYNAMIC_RESERVE */
Signature · 1 line
1.6.0.2
Acked-by on v1 1/8
#2 lore raw

Re: [PATCH 1/8] percpu: clean up percpu constants

Quoted 10 lines · Tejun Heo

Impact: cleaup

Make the following cleanups.

* There isn't much arch-specific about PERCPU_MODULE_RESERVE.  Always define it whether arch overrides PERCPU_ENOUGH_ROOM or not.

* blackfin overrides PERCPU_ENOUGH_ROOM to align static area size.  Do it by default.

Thanks
-Bryan

Quoted 99 lines

* percpu allocation sizes doesn't have much to do with the page size. Don't use PAGE_SHIFT in their definition.

---
 arch/blackfin/include/asm/percpu.h |   10 ---------- include/linux/percpu.h             |   24 +++++++++++++-----------

 2 files changed, 13 insertions(+), 21 deletions(-)
arch/blackfin/include/asm/percpu.h +0 -10
index 797c0c1..c94c7bc 100644
@@ -3,14 +3,4 @@
 
  #include <asm-generic/percpu.h>
 
-#ifdef CONFIG_MODULES
-#define PERCPU_MODULE_RESERVE 8192
-#else
-#define PERCPU_MODULE_RESERVE 0
-#endif
-
-#define PERCPU_ENOUGH_ROOM \
-       (ALIGN(__per_cpu_end - __per_cpu_start, SMP_CACHE_BYTES) + \
-        PERCPU_MODULE_RESERVE)
-
  #endif /* __ARCH_BLACKFIN_PERCPU__ */
include/linux/percpu.h +13 -12
index 545b068..2d34b03 100644
@@ -5,6 +5,7 @@
  #include <linux/slab.h> /* For kmalloc() */
  #include <linux/smp.h>
  #include <linux/cpumask.h>
+#include <linux/pfn.h>
 
  #include <asm/percpu.h>
 
@@ -52,17 +53,18 @@
  #define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
  #define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
 
-/* Enough to cover all DEFINE_PER_CPUs in kernel, including modules. */
-#ifndef PERCPU_ENOUGH_ROOM
+/* enough to cover all DEFINE_PER_CPUs in modules */
  #ifdef CONFIG_MODULES
-#define PERCPU_MODULE_RESERVE  8192
+#define PERCPU_MODULE_RESERVE          (8 << 10)
  #else
-#define PERCPU_MODULE_RESERVE  0
+#define PERCPU_MODULE_RESERVE          0
  #endif
 
+#ifndef PERCPU_ENOUGH_ROOM
  #define PERCPU_ENOUGH_ROOM                                             \
-       (__per_cpu_end - __per_cpu_start + PERCPU_MODULE_RESERVE)
-#endif /* PERCPU_ENOUGH_ROOM */
+       (ALIGN(__per_cpu_end - __per_cpu_start, SMP_CACHE_BYTES) +      \
+        PERCPU_MODULE_RESERVE)
+#endif
 
  /*
  * Must be an lvalue. Since @var must be a simple identifier,
@@ -79,7 +81,7 @@
  #ifdef CONFIG_HAVE_DYNAMIC_PER_CPU_AREA
 
  /* minimum unit size, also is the maximum supported allocation size */
-#define PCPU_MIN_UNIT_SIZE             (16UL << PAGE_SHIFT)
+#define PCPU_MIN_UNIT_SIZE             PFN_ALIGN(64 << 10)
 
  /*
  * PERCPU_DYNAMIC_RESERVE indicates the amount of free area to piggy
@@ -96,15 +98,15 @@
  #ifndef PERCPU_DYNAMIC_RESERVE
  #  if BITS_PER_LONG > 32
  #    ifdef CONFIG_MODULES
-#      define PERCPU_DYNAMIC_RESERVE   (6 << PAGE_SHIFT)
+#      define PERCPU_DYNAMIC_RESERVE   (24 << 10)
  #    else
-#      define PERCPU_DYNAMIC_RESERVE   (4 << PAGE_SHIFT)
+#      define PERCPU_DYNAMIC_RESERVE   (16 << 10)
  #    endif
  #  else
  #    ifdef CONFIG_MODULES
-#      define PERCPU_DYNAMIC_RESERVE   (4 << PAGE_SHIFT)
+#      define PERCPU_DYNAMIC_RESERVE   (16 << 10)
  #    else
-#      define PERCPU_DYNAMIC_RESERVE   (2 << PAGE_SHIFT)
+#      define PERCPU_DYNAMIC_RESERVE   (8 << 10)
  #    endif
  #  endif
  #endif /* PERCPU_DYNAMIC_RESERVE */
--

1.6.0.2

author maintainer
#3 lore raw

PATCH 2/8 percpu: cosmetic renames in pcpu_setup_first_chunk()

In reply to Tejun Heo

Impact: cosmetic, preparation for future changes

Make the following renames in pcpur_setup_first_chunk() in preparation for future changes.

* s/free_size/dyn_size/
* s/static_vm/first_vm/
* s/static_chunk/schunk/

---
 include/linux/percpu.h |    2 +-
 mm/percpu.c            |   58 ++++++++++++++++++++++++------------------------
 2 files changed, 30 insertions(+), 30 deletions(-)
include/linux/percpu.h +1 -1
index 2d34b03..a0b4ea2 100644
@@ -118,7 +118,7 @@ typedef void (*pcpu_populate_pte_fn_t)(unsigned long addr);
 
 extern size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
 					size_t static_size, size_t unit_size,
-					size_t free_size, void *base_addr,
+					size_t dyn_size, void *base_addr,
 					pcpu_populate_pte_fn_t populate_pte_fn);
 
 /*
mm/percpu.c +29 -29

126 lines not on this page · raw message

Signature · 1 line
1.6.0.2
author maintainer
#4 lore raw

PATCH 3/8 percpu: improve first chunk initial area map handling

In reply to Tejun Heo

Impact: no functional change

When the first chunk is created, its initial area map is not allocated because kmalloc isn't online yet. The map is allocated and initialized on the first allocation request on the chunk. This works fine but the scattering of initialization logic between the init function and allocation path is a bit confusing.

This patch makes the first chunk initialize and use minimal statically allocated map from pcpu_setpu_first_chunk(). The map resizing path still needs to handle this specially but it's more straight-forward and gives more latitude to the init path. This will ease future changes.

---
 mm/percpu.c |   53 +++++++++++++++++++++++++++--------------------------
 1 files changed, 27 insertions(+), 26 deletions(-)
mm/percpu.c +27 -26
index 9531590..503ccad 100644
@@ -93,9 +93,6 @@ static size_t pcpu_chunk_struct_size __read_mostly;
 void *pcpu_base_addr __read_mostly;
 EXPORT_SYMBOL_GPL(pcpu_base_addr);
 
-/* the size of kernel static area */
-static int pcpu_static_size __read_mostly;
-
 /*
  * One mutex to rule them all.
  *
@@ -316,15 +313,28 @@ static int pcpu_split_block(struct pcpu_chunk *chunk, int i, int head, int tail)
 
 	/* reallocation required? */
 	if (chunk->map_alloc < target) {
-		int new_alloc = chunk->map_alloc;
+		int new_alloc;
 		int *new;
 
+		new_alloc = PCPU_DFL_MAP_ALLOC;
 		while (new_alloc < target)
 			new_alloc *= 2;
 
-		new = pcpu_realloc(chunk->map,
-				   chunk->map_alloc * sizeof(new[0]),
-				   new_alloc * sizeof(new[0]));
+		if (chunk->map_alloc < PCPU_DFL_MAP_ALLOC) {
+			/*
+			 * map_alloc smaller than the default size
+			 * indicates that the chunk is one of the
+			 * first chunks and still using static map.
+			 * Allocate a dynamic one and copy.
+			 */
+			new = pcpu_realloc(NULL, 0, new_alloc * sizeof(new[0]));
+			if (new)
+				memcpy(new, chunk->map,
+				       chunk->map_alloc * sizeof(new[0]));
+		} else
+			new = pcpu_realloc(chunk->map,
+					   chunk->map_alloc * sizeof(new[0]),
+					   new_alloc * sizeof(new[0]));
 		if (!new)
 			return -ENOMEM;
 
@@ -367,22 +377,6 @@ static int pcpu_alloc_area(struct pcpu_chunk *chunk, int size, int align)
 	int max_contig = 0;
 	int i, off;
 
-	/*
-	 * The static chunk initially doesn't have map attached
-	 * because kmalloc wasn't available during init.  Give it one.
-	 */
-	if (unlikely(!chunk->map)) {
-		chunk->map = pcpu_realloc(NULL, 0,
-				PCPU_DFL_MAP_ALLOC * sizeof(chunk->map[0]));
-		if (!chunk->map)
-			return -ENOMEM;
-
-		chunk->map_alloc = PCPU_DFL_MAP_ALLOC;
-		chunk->map[chunk->map_used++] = -pcpu_static_size;
-		if (chunk->free_size)
-			chunk->map[chunk->map_used++] = chunk->free_size;
-	}
-
 	for (i = 0, off = 0; i < chunk->map_used; off += abs(chunk->map[i++])) {
 		bool is_last = i + 1 == chunk->map_used;
 		int head, tail;
@@ -874,12 +868,14 @@ size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
 				     pcpu_populate_pte_fn_t populate_pte_fn)
 {
 	static struct vm_struct first_vm;
+	static int smap[2];
 	struct pcpu_chunk *schunk;
 	unsigned int cpu;
 	int nr_pages;
 	int err, i;
 
 	/* santiy checks */
+	BUILD_BUG_ON(ARRAY_SIZE(smap) >= PCPU_DFL_MAP_ALLOC);
 	BUG_ON(!static_size);
 	BUG_ON(!unit_size && dyn_size);
 	BUG_ON(unit_size && unit_size < static_size + dyn_size);
@@ -893,7 +889,6 @@ size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
 		pcpu_unit_pages = max_t(int, PCPU_MIN_UNIT_SIZE >> PAGE_SHIFT,
 					PFN_UP(static_size));
 
-	pcpu_static_size = static_size;
 	pcpu_unit_size = pcpu_unit_pages << PAGE_SHIFT;
 	pcpu_chunk_size = num_possible_cpus() * pcpu_unit_size;
 	pcpu_chunk_struct_size = sizeof(struct pcpu_chunk)
@@ -912,14 +907,20 @@ size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
 	schunk = alloc_bootmem(pcpu_chunk_struct_size);
 	INIT_LIST_HEAD(&schunk->list);
 	schunk->vm = &first_vm;
+	schunk->map = smap;
+	schunk->map_alloc = ARRAY_SIZE(smap);
 
 	if (dyn_size)
 		schunk->free_size = dyn_size;
 	else
-		schunk->free_size = pcpu_unit_size - pcpu_static_size;
+		schunk->free_size = pcpu_unit_size - static_size;
 
 	schunk->contig_hint = schunk->free_size;
 
+	schunk->map[schunk->map_used++] = -static_size;
+	if (schunk->free_size)
+		schunk->map[schunk->map_used++] = schunk->free_size;
+
 	/* allocate vm address */
 	first_vm.flags = VM_ALLOC;
 	first_vm.size = pcpu_chunk_size;
@@ -948,7 +949,7 @@ size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
 			*pcpu_chunk_pagep(schunk, cpu, i) = page;
 		}
 
-		BUG_ON(i < PFN_UP(pcpu_static_size));
+		BUG_ON(i < PFN_UP(static_size));
 
 		if (nr_pages < 0)
 			nr_pages = i;
Signature · 1 line
1.6.0.2
author maintainer
#5 lore raw

PATCH 4/8 percpu: use negative for auto for pcpu_setup_first_chunk() arguments

In reply to Tejun Heo

Impact: argument semantic cleanup

In pcpu_setup_first_chunk(), zero @unit_size and @dyn_size meant auto-sizing. It's okay for @unit_size as 0 doesn't make sense but 0 dynamic reserve size is valid. Alos, if arch @dyn_size is calculated from other parameters, it might end up passing in 0 @dyn_size and malfunction when the size is automatically adjusted.

This patch makes both @unit_size and @dyn_size ssize_t and use -1 for auto sizing.

---
 arch/x86/kernel/setup_percpu.c |    2 +-
 include/linux/percpu.h         |    5 ++-
 mm/percpu.c                    |   46 +++++++++++++++++++++------------------
 3 files changed, 29 insertions(+), 24 deletions(-)
arch/x86/kernel/setup_percpu.c +1 -1
index c29f301..ef3a2cd 100644
@@ -344,7 +344,7 @@ static ssize_t __init setup_pcpu_4k(size_t static_size)
 	pr_info("PERCPU: Allocated %d 4k pages, static data %zu bytes\n",
 		pcpu4k_nr_static_pages, static_size);
 
-	ret = pcpu_setup_first_chunk(pcpu4k_get_page, static_size, 0, 0, NULL,
+	ret = pcpu_setup_first_chunk(pcpu4k_get_page, static_size, -1, -1, NULL,
 				     pcpu4k_populate_pte);
 	goto out_free_ar;
 
include/linux/percpu.h +3 -2
index a0b4ea2..a96fc53 100644
@@ -117,8 +117,9 @@ typedef struct page * (*pcpu_get_page_fn_t)(unsigned int cpu, int pageno);
 typedef void (*pcpu_populate_pte_fn_t)(unsigned long addr);
 
 extern size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
-					size_t static_size, size_t unit_size,
-					size_t dyn_size, void *base_addr,
+					size_t static_size,
+					ssize_t unit_size, ssize_t dyn_size,
+					void *base_addr,
 					pcpu_populate_pte_fn_t populate_pte_fn);
 
 /*
mm/percpu.c +25 -21

85 lines not on this page · raw message

Signature · 1 line
1.6.0.2
author maintainer
#6 lore raw

PATCH 5/8 x86: make embedding percpu allocator return excessive free space

In reply to Tejun Heo

Impact: reduce unnecessary memory usage on certain configurations

Embedding percpu allocator allocates unit_size * smp_num_possible_cpus() bytes consecutively and use it for the first chunk. However, if the static area is small, this can result in excessive prellocated free space in the first chunk due to PCPU_MIN_UNIT_SIZE restriction.

This patch makes embedding percpu allocator preallocate only what's necessary as described by PERPCU_DYNAMIC_RESERVE and return the leftover to the bootmem allocator.

---
 arch/x86/kernel/setup_percpu.c |   44 +++++++++++++++++++++++++--------------
 1 files changed, 28 insertions(+), 16 deletions(-)
arch/x86/kernel/setup_percpu.c +28 -16
index ef3a2cd..38e2b2a 100644
@@ -241,24 +241,31 @@ static ssize_t __init setup_pcpu_remap(size_t static_size)
  * Embedding allocator
  *
  * The first chunk is sized to just contain the static area plus
- * PERCPU_DYNAMIC_RESERVE and allocated as a contiguous area using
- * bootmem allocator and used as-is without being mapped into vmalloc
- * area.  This enables the first chunk to piggy back on the linear
- * physical PMD mapping and doesn't add any additional pressure to
- * TLB.
+ * module and dynamic reserves, and allocated as a contiguous area
+ * using bootmem allocator and used as-is without being mapped into
+ * vmalloc area.  This enables the first chunk to piggy back on the
+ * linear physical PMD mapping and doesn't add any additional pressure
+ * to TLB.  Note that if the needed size is smaller than the minimum
+ * unit size, the leftover is returned to the bootmem allocator.
  */
 static void *pcpue_ptr __initdata;
+static size_t pcpue_size __initdata;
 static size_t pcpue_unit_size __initdata;
 
 static struct page * __init pcpue_get_page(unsigned int cpu, int pageno)
 {
-	return virt_to_page(pcpue_ptr + cpu * pcpue_unit_size
-			    + ((size_t)pageno << PAGE_SHIFT));
+	size_t off = (size_t)pageno << PAGE_SHIFT;
+
+	if (off >= pcpue_size)
+		return NULL;
+
+	return virt_to_page(pcpue_ptr + cpu * pcpue_unit_size + off);
 }
 
 static ssize_t __init setup_pcpu_embed(size_t static_size)
 {
 	unsigned int cpu;
+	size_t dyn_size;
 
 	/*
 	 * If large page isn't supported, there's no benefit in doing
@@ -269,25 +276,30 @@ static ssize_t __init setup_pcpu_embed(size_t static_size)
 		return -EINVAL;
 
 	/* allocate and copy */
-	pcpue_unit_size = PFN_ALIGN(static_size + PERCPU_DYNAMIC_RESERVE);
-	pcpue_unit_size = max_t(size_t, pcpue_unit_size, PCPU_MIN_UNIT_SIZE);
+	pcpue_size = PFN_ALIGN(static_size + PERCPU_DYNAMIC_RESERVE);
+	pcpue_unit_size = max_t(size_t, pcpue_size, PCPU_MIN_UNIT_SIZE);
+	dyn_size = pcpue_size - static_size;
+
 	pcpue_ptr = pcpu_alloc_bootmem(0, num_possible_cpus() * pcpue_unit_size,
 				       PAGE_SIZE);
 	if (!pcpue_ptr)
 		return -ENOMEM;
 
-	for_each_possible_cpu(cpu)
-		memcpy(pcpue_ptr + cpu * pcpue_unit_size, __per_cpu_load,
-		       static_size);
+	for_each_possible_cpu(cpu) {
+		void *ptr = pcpue_ptr + cpu * pcpue_unit_size;
+
+		free_bootmem(__pa(ptr + pcpue_size),
+			     pcpue_unit_size - pcpue_size);
+		memcpy(ptr, __per_cpu_load, static_size);
+	}
 
 	/* we're ready, commit */
 	pr_info("PERCPU: Embedded %zu pages at %p, static data %zu bytes\n",
-		pcpue_unit_size >> PAGE_SHIFT, pcpue_ptr, static_size);
+		pcpue_size >> PAGE_SHIFT, pcpue_ptr, static_size);
 
 	return pcpu_setup_first_chunk(pcpue_get_page, static_size,
-				      pcpue_unit_size,
-				      pcpue_unit_size - static_size, pcpue_ptr,
-				      NULL);
+				      pcpue_unit_size, dyn_size,
+				      pcpue_ptr, NULL);
 }
 
 /*
Signature · 1 line
1.6.0.2
author maintainer
#7 lore raw

PATCH 6/8 percpu: add an indirection ptr for chunk page map access

In reply to Tejun Heo

Impact: allow sharing page map, no functional difference yet

Make chunk->page access indirect by adding a pointer and renaming the actual array to page_ar. This will be used by future changes.

---
 mm/percpu.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)
mm/percpu.c +4 -1
index a84cf99..5b47d9f 100644
@@ -80,7 +80,8 @@ struct pcpu_chunk {
 	int			map_alloc;	/* # of map entries allocated */
 	int			*map;		/* allocation map */
 	bool			immutable;	/* no [de]population allowed */
-	struct page		*page[];	/* #cpus * UNIT_PAGES */
+	struct page		**page;		/* points to page array */
+	struct page		*page_ar[];	/* #cpus * UNIT_PAGES */
 };
 
 static int pcpu_unit_pages __read_mostly;
@@ -696,6 +697,7 @@ static struct pcpu_chunk *alloc_pcpu_chunk(void)
 				  PCPU_DFL_MAP_ALLOC * sizeof(chunk->map[0]));
 	chunk->map_alloc = PCPU_DFL_MAP_ALLOC;
 	chunk->map[chunk->map_used++] = pcpu_unit_size;
+	chunk->page = chunk->page_ar;
 
 	chunk->vm = get_vm_area(pcpu_chunk_size, GFP_KERNEL);
 	if (!chunk->vm) {
@@ -918,6 +920,7 @@ size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
 	schunk->vm = &first_vm;
 	schunk->map = smap;
 	schunk->map_alloc = ARRAY_SIZE(smap);
+	schunk->page = schunk->page_ar;
 	schunk->free_size = dyn_size;
 	schunk->contig_hint = schunk->free_size;
 
Signature · 1 line
1.6.0.2
author maintainer
#8 lore raw

PATCH 7/8 percpu, module: implement reserved allocation and use it for module percpu variables

In reply to Tejun Heo

Impact: add reserved allocation functionality and use it for module
percpu variables

This patch implements reserved allocation from the first chunk. When setting up the first chunk, arch can ask to set aside certain number of bytes right after the core static area which is available only through a separate reserved allocator. This will be used primarily for module static percpu variables on architectures with limited relocation range to ensure that the module perpcu symbols are inside the relocatable range.

If reserved area is requested, the first chunk becomes reserved and isn't available for regular allocation. If the first chunk also includes piggy-back dynamic allocation area, a separate chunk mapping the same region is created to serve dynamic allocation. The first one is called static first chunk and the second dynamic first chunk. Although they share the page map, their different area map initializations guarantee they serve disjoint areas according to their purposes.

If arch doesn't setup reserved area, reserved allocation is handled like any other allocation.

---
 arch/x86/kernel/setup_percpu.c |    8 +-
 include/linux/percpu.h         |   10 ++-
 kernel/module.c                |    2 +-
 mm/percpu.c                    |  153 ++++++++++++++++++++++++++++++++++-----
 4 files changed, 144 insertions(+), 29 deletions(-)
arch/x86/kernel/setup_percpu.c +4 -4
index 38e2b2a..dd4eabc 100644
@@ -217,7 +217,7 @@ proceed:
 	pr_info("PERCPU: Remapped at %p with large pages, static data "
 		"%zu bytes\n", vm.addr, static_size);
 
-	ret = pcpu_setup_first_chunk(pcpur_get_page, static_size, PMD_SIZE,
+	ret = pcpu_setup_first_chunk(pcpur_get_page, static_size, 0, PMD_SIZE,
 				     pcpur_size - static_size, vm.addr, NULL);
 	goto out_free_ar;
 
@@ -297,7 +297,7 @@ static ssize_t __init setup_pcpu_embed(size_t static_size)
 	pr_info("PERCPU: Embedded %zu pages at %p, static data %zu bytes\n",
 		pcpue_size >> PAGE_SHIFT, pcpue_ptr, static_size);
 
-	return pcpu_setup_first_chunk(pcpue_get_page, static_size,
+	return pcpu_setup_first_chunk(pcpue_get_page, static_size, 0,
 				      pcpue_unit_size, dyn_size,
 				      pcpue_ptr, NULL);
 }
@@ -356,8 +356,8 @@ static ssize_t __init setup_pcpu_4k(size_t static_size)
 	pr_info("PERCPU: Allocated %d 4k pages, static data %zu bytes\n",
 		pcpu4k_nr_static_pages, static_size);
 
-	ret = pcpu_setup_first_chunk(pcpu4k_get_page, static_size, -1, -1, NULL,
-				     pcpu4k_populate_pte);
+	ret = pcpu_setup_first_chunk(pcpu4k_get_page, static_size, 0, -1, -1,
+				     NULL, pcpu4k_populate_pte);
 	goto out_free_ar;
 
 enomem:
include/linux/percpu.h +6 -4
index a96fc53..8ff1515 100644
@@ -117,10 +117,10 @@ typedef struct page * (*pcpu_get_page_fn_t)(unsigned int cpu, int pageno);
 typedef void (*pcpu_populate_pte_fn_t)(unsigned long addr);
 
 extern size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
-					size_t static_size,
-					ssize_t unit_size, ssize_t dyn_size,
-					void *base_addr,
-					pcpu_populate_pte_fn_t populate_pte_fn);
+				size_t static_size, size_t reserved_size,
+				ssize_t unit_size, ssize_t dyn_size,
+				void *base_addr,
+				pcpu_populate_pte_fn_t populate_pte_fn);
 
 /*
  * Use this to get to a cpu's version of the per-cpu object
@@ -129,6 +129,8 @@ extern size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
  */
 #define per_cpu_ptr(ptr, cpu)	SHIFT_PERCPU_PTR((ptr), per_cpu_offset((cpu)))
 
+extern void *__alloc_reserved_percpu(size_t size, size_t align);
+
 #else /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */
 
 struct percpu_data {
kernel/module.c +1 -1
index 1f0657a..f0e04d6 100644
@@ -381,7 +381,7 @@ static void *percpu_modalloc(unsigned long size, unsigned long align,
 		align = PAGE_SIZE;
 	}
 
-	ptr = __alloc_percpu(size, align);
+	ptr = __alloc_reserved_percpu(size, align);
 	if (!ptr)
 		printk(KERN_WARNING
 		       "Could not allocate %lu bytes percpu data\n", size);
mm/percpu.c +133 -20

291 lines not on this page · raw message

Signature · 1 line
1.6.0.2
author maintainer
#9 lore raw

PATCH 8/8 x86, percpu: setup reserved percpu area for x86_64

In reply to Tejun Heo

Impact: fix relocation overflow during module load

x86_64 uses 32bit relocations for symbol access and static percpu symbols whether in core or modules must be inside 2GB of the percpu segement base which the dynamic percpu allocator doesn't guarantee. This patch makes x86_64 reserve PERCPU_MODULE_RESERVE bytes in the first chunk so that module percpu areas are always allocated from the first chunk which is always inside the relocatable range.

This problem exists for any percpu allocator but is easily triggered when using the embedding allocator because the second chunk is located beyond 2GB on it.

This patch also changes the meaning of PERCPU_DYNAMIC_RESERVE such that it only indicates the size of the area to reserve for dynamic allocation as static and dynamic areas can be separate. New PERCPU_DYNAMIC_RESERVED is increased by 4k for both 32 and 64bits as the reserved area separation eats away some allocatable space and having slightly more headroom (currently between 4 and 8k after minimal boot sans module area) makes sense for common case performance.

x86_32 can address anywhere from anywhere and doesn't need reserving.

Mike Galbraith first reported the problem first and bisected it to the embedding percpu allocator commit.

---
 arch/x86/kernel/setup_percpu.c |   37 ++++++++++++++++++++++++++++---------
 include/linux/percpu.h         |   35 ++++++++++++-----------------------
 2 files changed, 40 insertions(+), 32 deletions(-)
arch/x86/kernel/setup_percpu.c +28 -9
index dd4eabc..efa615f 100644
@@ -42,6 +42,19 @@ unsigned long __per_cpu_offset[NR_CPUS] __read_mostly = {
 };
 EXPORT_SYMBOL(__per_cpu_offset);
 
+/*
+ * On x86_64 symbols referenced from code should be reachable using
+ * 32bit relocations.  Reserve space for static percpu variables in
+ * modules so that they are always served from the first chunk which
+ * is located at the percpu segment base.  On x86_32, anything can
+ * address anywhere.  No need to reserve space in the first chunk.
+ */
+#ifdef CONFIG_X86_64
+#define PERCPU_FIRST_CHUNK_RESERVE	PERCPU_MODULE_RESERVE
+#else
+#define PERCPU_FIRST_CHUNK_RESERVE	0
+#endif
+
 /**
  * pcpu_need_numa - determine percpu allocation needs to consider NUMA
  *
@@ -141,7 +154,7 @@ static ssize_t __init setup_pcpu_remap(size_t static_size)
 {
 	static struct vm_struct vm;
 	pg_data_t *last;
-	size_t ptrs_size;
+	size_t ptrs_size, dyn_size;
 	unsigned int cpu;
 	ssize_t ret;
 
@@ -169,12 +182,14 @@ proceed:
 	 * Currently supports only single page.  Supporting multiple
 	 * pages won't be too difficult if it ever becomes necessary.
 	 */
-	pcpur_size = PFN_ALIGN(static_size + PERCPU_DYNAMIC_RESERVE);
+	pcpur_size = PFN_ALIGN(static_size + PERCPU_MODULE_RESERVE +
+			       PERCPU_DYNAMIC_RESERVE);
 	if (pcpur_size > PMD_SIZE) {
 		pr_warning("PERCPU: static data is larger than large page, "
 			   "can't use large page\n");
 		return -EINVAL;
 	}
+	dyn_size = pcpur_size - static_size - PERCPU_FIRST_CHUNK_RESERVE;
 
 	/* allocate pointer array and alloc large pages */
 	ptrs_size = PFN_ALIGN(num_possible_cpus() * sizeof(pcpur_ptrs[0]));
@@ -217,8 +232,9 @@ proceed:
 	pr_info("PERCPU: Remapped at %p with large pages, static data "
 		"%zu bytes\n", vm.addr, static_size);
 
-	ret = pcpu_setup_first_chunk(pcpur_get_page, static_size, 0, PMD_SIZE,
-				     pcpur_size - static_size, vm.addr, NULL);
+	ret = pcpu_setup_first_chunk(pcpur_get_page, static_size,
+				     PERCPU_FIRST_CHUNK_RESERVE,
+				     PMD_SIZE, dyn_size, vm.addr, NULL);
 	goto out_free_ar;
 
 enomem:
@@ -276,9 +292,10 @@ static ssize_t __init setup_pcpu_embed(size_t static_size)
 		return -EINVAL;
 
 	/* allocate and copy */
-	pcpue_size = PFN_ALIGN(static_size + PERCPU_DYNAMIC_RESERVE);
+	pcpue_size = PFN_ALIGN(static_size + PERCPU_MODULE_RESERVE +
+			       PERCPU_DYNAMIC_RESERVE);
 	pcpue_unit_size = max_t(size_t, pcpue_size, PCPU_MIN_UNIT_SIZE);
-	dyn_size = pcpue_size - static_size;
+	dyn_size = pcpue_size - static_size - PERCPU_FIRST_CHUNK_RESERVE;
 
 	pcpue_ptr = pcpu_alloc_bootmem(0, num_possible_cpus() * pcpue_unit_size,
 				       PAGE_SIZE);
@@ -297,7 +314,8 @@ static ssize_t __init setup_pcpu_embed(size_t static_size)
 	pr_info("PERCPU: Embedded %zu pages at %p, static data %zu bytes\n",
 		pcpue_size >> PAGE_SHIFT, pcpue_ptr, static_size);
 
-	return pcpu_setup_first_chunk(pcpue_get_page, static_size, 0,
+	return pcpu_setup_first_chunk(pcpue_get_page, static_size,
+				      PERCPU_FIRST_CHUNK_RESERVE,
 				      pcpue_unit_size, dyn_size,
 				      pcpue_ptr, NULL);
 }
@@ -356,8 +374,9 @@ static ssize_t __init setup_pcpu_4k(size_t static_size)
 	pr_info("PERCPU: Allocated %d 4k pages, static data %zu bytes\n",
 		pcpu4k_nr_static_pages, static_size);
 
-	ret = pcpu_setup_first_chunk(pcpu4k_get_page, static_size, 0, -1, -1,
-				     NULL, pcpu4k_populate_pte);
+	ret = pcpu_setup_first_chunk(pcpu4k_get_page, static_size,
+				     PERCPU_FIRST_CHUNK_RESERVE, -1, -1, NULL,
+				     pcpu4k_populate_pte);
 	goto out_free_ar;
 
 enomem:
include/linux/percpu.h +12 -23
index 8ff1515..54a968b 100644
@@ -85,31 +85,20 @@
 
 /*
  * PERCPU_DYNAMIC_RESERVE indicates the amount of free area to piggy
- * back on the first chunk if arch is manually allocating and mapping
- * it for faster access (as a part of large page mapping for example).
- * Note that dynamic percpu allocator covers both static and dynamic
- * areas, so these values are bigger than PERCPU_MODULE_RESERVE.
+ * back on the first chunk for dynamic percpu allocation if arch is
+ * manually allocating and mapping it for faster access (as a part of
+ * large page mapping for example).
  *
- * On typical configuration with modules, the following values leave
- * about 8k of free space on the first chunk after boot on both x86_32
- * and 64 when module support is enabled.  When module support is
- * disabled, it's much tighter.
+ * The following values give between one and two pages of free space
+ * after typical minimal boot (2-way SMP, single disk and NIC) with
+ * both defconfig and a distro config on x86_64 and 32.  More
+ * intelligent way to determine this would be nice.
  */
-#ifndef PERCPU_DYNAMIC_RESERVE
-#  if BITS_PER_LONG > 32
-#    ifdef CONFIG_MODULES
-#      define PERCPU_DYNAMIC_RESERVE	(24 << 10)
-#    else
-#      define PERCPU_DYNAMIC_RESERVE	(16 << 10)
-#    endif
-#  else
-#    ifdef CONFIG_MODULES
-#      define PERCPU_DYNAMIC_RESERVE	(16 << 10)
-#    else
-#      define PERCPU_DYNAMIC_RESERVE	(8 << 10)
-#    endif
-#  endif
-#endif	/* PERCPU_DYNAMIC_RESERVE */
+#if BITS_PER_LONG > 32
+#define PERCPU_DYNAMIC_RESERVE		(20 << 10)
+#else
+#define PERCPU_DYNAMIC_RESERVE		(12 << 10)
+#endif
 
 extern void *pcpu_base_addr;
 
Signature · 1 line
1.6.0.2
#10 lore raw
In reply to Tejun Heo
Quoted 11 lines · Tejun Heo

Hello,

Please pull from the following git vector.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git tj-percpu

This patchset implements reserved percpu alloc from the first chunk and uses it for static perpcu variables in modules on x86_64. This is necessary because x86_64 expects symbols to be relocatable using 32bit relocations but dynamic percpu allocation does not guarantee it leading to relocation overflow during module load.

Tested, my module loading woes are cured.

-Mike

#11 lore raw
In reply to Tejun Heo
Quoted 48 lines · Tejun Heo

Hello,

Please pull from the following git vector.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git tj-percpu

This patchset implements reserved percpu alloc from the first chunk and uses it for static perpcu variables in modules on x86_64. This is necessary because x86_64 expects symbols to be relocatable using 32bit relocations but dynamic percpu allocation does not guarantee it leading to relocation overflow during module load.

This can happen to any of the x86_64 percpu first chunk allocators but the embedding allocator is most likely to trigger it as the first chunk is located in the normal kernel memory allocation area while all the other chunks will be allocated in the vmalloc area which is very far from the first chunk.

This type of reservation will be necessary for other architectures too where addressing mode used for percpu acess uses offsets with limited range to access symbols (e.g. ia64 w/ the magic 64k percpu page).

This problem was first reported and bisected by Mike Galbraith[2].

This patchset is against the current x86/core/percpu[1] and contains the following patches.

0001-percpu-clean-up-percpu-constants.patch 0002-percpu-cosmetic-renames-in-pcpu_setup_first_chunk.patch 0003-percpu-improve-first-chunk-initial-area-map-handlin.patch 0004-percpu-use-negative-for-auto-for-pcpu_setup_first_c.patch 0005-x86-make-embedding-percpu-allocator-return-excessiv.patch 0006-percpu-add-an-indirection-ptr-for-chunk-page-map-ac.patch 0007-percpu-module-implement-reserved-allocation-and-us.patch 0008-x86-percpu-setup-reserved-percpu-area-for-x86_64.patch

0001-0004 makes misc updates to percpu in preparation of later changes. 0005 fixes a case where excessive memory can be set aside in the first chunk when using the embedding allocator. 0006-0008 implement reserved allocation and use it for x86_64.

Bryan Wu, can you please review #0001? It contains small change in blackfin. It's mostly trivial but just in case.

Quoted 8 lines

Diffstat follows.

 arch/blackfin/include/asm/percpu.h |   10 -
 arch/x86/kernel/setup_percpu.c     |   73 +++++++---
 include/linux/percpu.h             |   60 +++-----
 kernel/module.c                    |    2 
 mm/percpu.c                        |  267 ++++++++++++++++++++++++++-----------
 5 files changed, 274 insertions(+), 138 deletions(-)

Pulled, thanks Tejun!

Ingo

#12 lore raw
In reply to Tejun Heo
Quoted 37 lines · Tejun Heo

Hello,

Please pull from the following git vector.

 git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git tj-percpu

This patchset implements reserved percpu alloc from the first chunk and uses it for static perpcu variables in modules on x86_64.  This is necessary because x86_64 expects symbols to be relocatable using 32bit relocations but dynamic percpu allocation does not guarantee it leading to relocation overflow during module load.

This can happen to any of the x86_64 percpu first chunk allocators but the embedding allocator is most likely to trigger it as the first chunk is located in the normal kernel memory allocation area while all the other chunks will be allocated in the vmalloc area which is very far from the first chunk.

This type of reservation will be necessary for other architectures too where addressing mode used for percpu acess uses offsets with limited range to access symbols (e.g. ia64 w/ the magic 64k percpu page).

This problem was first reported and bisected by Mike Galbraith[2].

This patchset is against the current x86/core/percpu[1] and contains the following patches.

 0001-percpu-clean-up-percpu-constants.patch 0002-percpu-cosmetic-renames-in-pcpu_setup_first_chunk.patch 0003-percpu-improve-first-chunk-initial-area-map-handlin.patch 0004-percpu-use-negative-for-auto-for-pcpu_setup_first_c.patch 0005-x86-make-embedding-percpu-allocator-return-excessiv.patch 0006-percpu-add-an-indirection-ptr-for-chunk-page-map-ac.patch 0007-percpu-module-implement-reserved-allocation-and-us.patch 0008-x86-percpu-setup-reserved-percpu-area-for-x86_64.patch

Quoted 8 lines

0001-0004 makes misc updates to percpu in preparation of later changes.  0005 fixes a case where excessive memory can be set aside in the first chunk when using the embedding allocator.  0006-0008 implement reserved allocation and use it for x86_64.

Bryan Wu, can you please review #0001?  It contains small change in blackfin.  It's mostly trivial but just in case.

Hi Tejun,

Sorry for the delay, it looks fine for me.

-Bryan

Quoted 17 lines

Diffstat follows.

 arch/blackfin/include/asm/percpu.h |   10 - arch/x86/kernel/setup_percpu.c     |   73 +++++++--- include/linux/percpu.h             |   60 +++-----
 kernel/module.c                    |    2
 mm/percpu.c                        |  267 ++++++++++++++++++++++++++-----------

 5 files changed, 274 insertions(+), 138 deletions(-)

Thanks.

--
tejun

[1] f254f3909efaf59ca2d0f408de2d044dace60706
[2] http://thread.gmane.org/gmane.linux.kernel/801799