Skip to contentSkip to conversation

percpu: generalize first chunk allocators and improve lpage NUMA support

PATCH In review Tejun Heo 8 participants 53 messages Jun 24, 2009 – Jul 15, 2009

Subsystems (6)
lore
Share & tools
lore

Tejun Heo

author maintainer
#0 lore raw

PATCH 1/1 percpu: generalize first chunk allocators and improve lpage NUMA support

Hello,

This patchset is combination of the following two patchsets.

[1] x86,percpu: generalize 4k and lpage allocator
[2] percpu: teach lpage allocator about NUMA

Changes from the last postings are

* updated to be on top of the current percpu#for-next(bf4bb2b1)

* sparc64 was converted to dynamic percpu allocator and using pcpu_setup_first_chunk() which is changed by this patchset. sparc64 updated accordingly.

This patchset contains the following patches.

0001-x86-make-pcpu_chunk_addr_search-matching-stricter.patch 0002-percpu-drop-unit_size-from-embed-first-chunk-alloc.patch 0003-x86-percpu-generalize-4k-first-chunk-allocator.patch 0004-percpu-make-4k-first-chunk-allocator-map-memory.patch 0005-x86-percpu-generalize-lpage-first-chunk-allocator.patch 0006-percpu-simplify-pcpu_setup_first_chunk.patch 0007-percpu-reorder-a-few-functions-in-mm-percpu.c.patch 0008-percpu-drop-pcpu_chunk-page.patch 0009-percpu-allow-non-linear-sparse-cpu-unit-mappin.patch 0010-percpu-teach-large-page-allocator-about-NUMA.patch

0001-0006 generalizes first chunk allocators. 0007-0010 improves lpage allocator such that NUMA is handled more intelligently.

This patchset first generalizes first chunk allocators, makes the percpu allocator to be able to use non-linear and/or sparse cpu -> unit mapping and then make lpage allocator consider CPU topology and group CPUs in LOCAL_DISTANCE into the same large pages. For example, on an 4/4 NUMA machine, the original code used up 16MB for each chunk but the new code uses only 4MB - one large page for each NUMA node. The grouping code is quite robust and will try to minimize space wastage even when the CPU topology is asymmetric.

David, sparc64 should be able to use lpage (renamed from remap) allocator the same way x86_64 does. Well, at least that was my intention, if something doesn't work or needs improvements for sparc64, please let me know.

This patchset is available in the following git tree and will be published in for-next if there's no major objection. It might get rebased before going into for-next.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git lpage-numa

diffstat follows.

 arch/sparc/kernel/smp_64.c     |   42 -
 arch/x86/include/asm/percpu.h  |    9 
 arch/x86/kernel/setup_percpu.c |  297 ++-------
 arch/x86/mm/pageattr.c         |    1 
 include/linux/percpu.h         |   68 +-
 mm/percpu.c                    | 1276 +++++++++++++++++++++++++++++++----------
 6 files changed, 1139 insertions(+), 554 deletions(-)

Thanks.

--
tejun

[1] http://thread.gmane.org/gmane.linux.kernel/853114
[2] http://lkml.org/lkml/2009/6/17/14

author maintainer
#1 lore raw

PATCH 1/10 x86: make pcpu_chunk_addr_search() matching stricter

The @addr passed into pcpu_chunk_addr_search() is unit0 based address and thus should be matched inside unit0 area. Currently, when it uses chunk size when determining whether the address falls in the first chunk. Addresses in unitN where N>0 shouldn't be passed in anyway, so this doesn't cause any malfunction but fix it for consistency.

[ Impact: mostly cleanup ]

---
 mm/percpu.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
mm/percpu.c +1 -1
index b149845..19dd83b 100644
@@ -290,7 +290,7 @@ static struct pcpu_chunk *pcpu_chunk_addr_search(void *addr)
 	void *first_start = pcpu_first_chunk->vm->addr;
 
 	/* is it in the first chunk? */
-	if (addr >= first_start && addr < first_start + pcpu_chunk_size) {
+	if (addr >= first_start && addr < first_start + pcpu_unit_size) {
 		/* is it in the reserved area? */
 		if (addr < first_start + pcpu_reserved_chunk_limit)
 			return pcpu_reserved_chunk;
Signature · 1 line
1.6.0.2
author maintainer
#2 lore raw

PATCH 2/10 percpu: drop @unit_size from embed first chunk allocator

In reply to Tejun Heo

The only extra feature @unit_size provides is making dead space at the end of the first chunk which doesn't have any valid usecase. Drop the parameter. This will increase consistency with generalized 4k allocator.

[ Impact: drop unused code path ]

---
 arch/x86/kernel/setup_percpu.c |    2 +-
 include/linux/percpu.h         |    2 +-
 mm/percpu.c                    |   16 +++++-----------
 3 files changed, 7 insertions(+), 13 deletions(-)
arch/x86/kernel/setup_percpu.c +1 -1
index 29a3eef..1472820 100644
@@ -342,7 +342,7 @@ static ssize_t __init setup_pcpu_embed(size_t static_size, bool chosen)
 		return -EINVAL;
 
 	return pcpu_embed_first_chunk(static_size, PERCPU_FIRST_CHUNK_RESERVE,
-				      reserve - PERCPU_FIRST_CHUNK_RESERVE, -1);
+				      reserve - PERCPU_FIRST_CHUNK_RESERVE);
 }
 
 /*
include/linux/percpu.h +1 -1
index e500034..83bff05 100644
@@ -69,7 +69,7 @@ extern size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
 
 extern ssize_t __init pcpu_embed_first_chunk(
 				size_t static_size, size_t reserved_size,
-				ssize_t dyn_size, ssize_t unit_size);
+				ssize_t dyn_size);
 
 /*
  * Use this to get to a cpu's version of the per-cpu object
mm/percpu.c +5 -11
index 19dd83b..fe34b6b 100644
@@ -1207,7 +1207,6 @@ static struct page * __init pcpue_get_page(unsigned int cpu, int pageno)
  * @static_size: the size of static percpu area in bytes
  * @reserved_size: the size of reserved percpu area in bytes
  * @dyn_size: free size for dynamic allocation in bytes, -1 for auto
- * @unit_size: unit size in bytes, must be multiple of PAGE_SIZE, -1 for auto
  *
  * This is a helper to ease setting up embedded first percpu chunk and
  * can be called where pcpu_setup_first_chunk() is expected.
@@ -1219,9 +1218,9 @@ static struct page * __init pcpue_get_page(unsigned int cpu, int pageno)
  * page size.
  *
  * When @dyn_size is positive, dynamic area might be larger than
- * specified to fill page alignment.  Also, when @dyn_size is auto,
- * @dyn_size does not fill the whole first chunk but only what's
- * necessary for page alignment after static and reserved areas.
+ * specified to fill page alignment.  When @dyn_size is auto,
+ * @dyn_size is just big enough to fill page alignment after static
+ * and reserved areas.
  *
  * If the needed size is smaller than the minimum or specified unit
  * size, the leftover is returned to the bootmem allocator.
@@ -1231,7 +1230,7 @@ static struct page * __init pcpue_get_page(unsigned int cpu, int pageno)
  * percpu access on success, -errno on failure.
  */
 ssize_t __init pcpu_embed_first_chunk(size_t static_size, size_t reserved_size,
-				      ssize_t dyn_size, ssize_t unit_size)
+				      ssize_t dyn_size)
 {
 	size_t chunk_size;
 	unsigned int cpu;
@@ -1242,12 +1241,7 @@ ssize_t __init pcpu_embed_first_chunk(size_t static_size, size_t reserved_size,
 	if (dyn_size != 0)
 		dyn_size = pcpue_size - static_size - reserved_size;
 
-	if (unit_size >= 0) {
-		BUG_ON(unit_size < pcpue_size);
-		pcpue_unit_size = unit_size;
-	} else
-		pcpue_unit_size = max_t(size_t, pcpue_size, PCPU_MIN_UNIT_SIZE);
-
+	pcpue_unit_size = max_t(size_t, pcpue_size, PCPU_MIN_UNIT_SIZE);
 	chunk_size = pcpue_unit_size * num_possible_cpus();
 
 	pcpue_ptr = __alloc_bootmem_nopanic(chunk_size, PAGE_SIZE,
Signature · 1 line
1.6.0.2
author maintainer
#3 lore raw

PATCH 3/10 x86,percpu: generalize 4k first chunk allocator

In reply to Tejun Heo

Generalize and move x86 setup_pcpu_4k() into pcpu_4k_first_chunk(). setup_pcpu_4k() now is a simple wrapper around the generalized version. Other than taking size parameters and using arch supplied callbacks to allocate/free memory, pcpu_4k_first_chunk() is identical to the original implementation.

This simplifies arch code and will help converting more archs to dynamic percpu allocator.

While at it, s/pcpu_populate_pte_fn_t/pcpu_fc_populate_pte_fn_t/ for consistency.

[ Impact: code reorganization and generalization ]

---
 arch/x86/kernel/setup_percpu.c |   78 +++++++++---------------------------
 include/linux/percpu.h         |   12 +++++-
 mm/percpu.c                    |   85 +++++++++++++++++++++++++++++++++++++++-
 3 files changed, 113 insertions(+), 62 deletions(-)
arch/x86/kernel/setup_percpu.c +19 -59
index 1472820..ab896b3 100644
@@ -124,6 +124,19 @@ static void * __init pcpu_alloc_bootmem(unsigned int cpu, unsigned long size,
 }
 
 /*
+ * Helpers for first chunk memory allocation
+ */
+static void * __init pcpu_fc_alloc(unsigned int cpu, size_t size)
+{
+	return pcpu_alloc_bootmem(cpu, size, size);
+}
+
+static void __init pcpu_fc_free(void *ptr, size_t size)
+{
+	free_bootmem(__pa(ptr), size);
+}
+
+/*
  * Large page remap allocator
  *
  * This allocator uses PMD page as unit.  A PMD page is allocated for
@@ -346,22 +359,11 @@ static ssize_t __init setup_pcpu_embed(size_t static_size, bool chosen)
 }
 
 /*
- * 4k page allocator
+ * 4k allocator
  *
- * This is the basic allocator.  Static percpu area is allocated
- * page-by-page and most of initialization is done by the generic
- * setup function.
+ * Boring fallback 4k allocator.  This allocator puts more pressure on
+ * PTE TLBs but other than that behaves nicely on both UMA and NUMA.
  */
-static struct page **pcpu4k_pages __initdata;
-static int pcpu4k_nr_static_pages __initdata;
-
-static struct page * __init pcpu4k_get_page(unsigned int cpu, int pageno)
-{
-	if (pageno < pcpu4k_nr_static_pages)
-		return pcpu4k_pages[cpu * pcpu4k_nr_static_pages + pageno];
-	return NULL;
-}
-
 static void __init pcpu4k_populate_pte(unsigned long addr)
 {
 	populate_extra_pte(addr);
@@ -369,51 +371,9 @@ static void __init pcpu4k_populate_pte(unsigned long addr)
 
 static ssize_t __init setup_pcpu_4k(size_t static_size)
 {
-	size_t pages_size;
-	unsigned int cpu;
-	int i, j;
-	ssize_t ret;
-
-	pcpu4k_nr_static_pages = PFN_UP(static_size);
-
-	/* unaligned allocations can't be freed, round up to page size */
-	pages_size = PFN_ALIGN(pcpu4k_nr_static_pages * num_possible_cpus()
-			       * sizeof(pcpu4k_pages[0]));
-	pcpu4k_pages = alloc_bootmem(pages_size);
-
-	/* allocate and copy */
-	j = 0;
-	for_each_possible_cpu(cpu)
-		for (i = 0; i < pcpu4k_nr_static_pages; i++) {
-			void *ptr;
-
-			ptr = pcpu_alloc_bootmem(cpu, PAGE_SIZE, PAGE_SIZE);
-			if (!ptr) {
-				pr_warning("PERCPU: failed to allocate "
-					   "4k page for cpu%u\n", cpu);
-				goto enomem;
-			}
-
-			memcpy(ptr, __per_cpu_load + i * PAGE_SIZE, PAGE_SIZE);
-			pcpu4k_pages[j++] = virt_to_page(ptr);
-		}
-
-	/* we're ready, commit */
-	pr_info("PERCPU: Allocated %d 4k pages, static data %zu bytes\n",
-		pcpu4k_nr_static_pages, static_size);
-
-	ret = pcpu_setup_first_chunk(pcpu4k_get_page, static_size,
-				     PERCPU_FIRST_CHUNK_RESERVE, -1,
-				     -1, NULL, pcpu4k_populate_pte);
-	goto out_free_ar;
-
-enomem:
-	while (--j >= 0)
-		free_bootmem(__pa(page_address(pcpu4k_pages[j])), PAGE_SIZE);
-	ret = -ENOMEM;
-out_free_ar:
-	free_bootmem(__pa(pcpu4k_pages), pages_size);
-	return ret;
+	return pcpu_4k_first_chunk(static_size, PERCPU_FIRST_CHUNK_RESERVE,
+				   pcpu_fc_alloc, pcpu_fc_free,
+				   pcpu4k_populate_pte);
 }
 
 /* for explicit first chunk allocator selection */
include/linux/percpu.h +10 -2
index 83bff05..41b5bfa 100644
@@ -59,18 +59,26 @@
 extern void *pcpu_base_addr;
 
 typedef struct page * (*pcpu_get_page_fn_t)(unsigned int cpu, int pageno);
-typedef void (*pcpu_populate_pte_fn_t)(unsigned long addr);
+typedef void * (*pcpu_fc_alloc_fn_t)(unsigned int cpu, size_t size);
+typedef void (*pcpu_fc_free_fn_t)(void *ptr, size_t size);
+typedef void (*pcpu_fc_populate_pte_fn_t)(unsigned long addr);
 
 extern size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
 				size_t static_size, size_t reserved_size,
 				ssize_t dyn_size, ssize_t unit_size,
 				void *base_addr,
-				pcpu_populate_pte_fn_t populate_pte_fn);
+				pcpu_fc_populate_pte_fn_t populate_pte_fn);
 
 extern ssize_t __init pcpu_embed_first_chunk(
 				size_t static_size, size_t reserved_size,
 				ssize_t dyn_size);
 
+extern ssize_t __init pcpu_4k_first_chunk(
+				size_t static_size, size_t reserved_size,
+				pcpu_fc_alloc_fn_t alloc_fn,
+				pcpu_fc_free_fn_t free_fn,
+				pcpu_fc_populate_pte_fn_t populate_pte_fn);
+
 /*
  * Use this to get to a cpu's version of the per-cpu object
  * dynamically allocated. Non-atomic access to the current CPU's
mm/percpu.c +84 -1

97 lines not on this page · raw message

Signature · 1 line
1.6.0.2
author maintainer
#4 lore raw

PATCH 4/10 percpu: make 4k first chunk allocator map memory

In reply to Tejun Heo

At first, percpu first chunk was always setup page-by-page by the generic code. To add other allocators, different parts of the generic initialization was made optional. Now we have three allocators - embed, remap and 4k. embed and remap fully handle allocation and mapping of the first chunk while 4k still depends on generic code for those. This makes the generic alloc/map paths specifci to 4k and makes the code unnecessary complicated with optional generic behaviors.

This patch makes the 4k allocator to allocate and map memory directly instead of depending on the generic code. The only outside visible change is that now dynamic area in the first chunk is allocated up-front instead of on-demand. This doesn't make any meaningful difference as the area is minimal (usually less than a page, just enough to fill the alignment) on 4k allocator. Plus, dynamic area in the first chunk usually gets fully used anyway.

This will allow simplification of pcpu_setpu_first_chunk() and removal of chunk->page array.

[ Impact: no outside visible change other than up-front allocation of dyn area ]

---
 mm/percpu.c |   71 ++++++++++++++++++++++++++++++++++++++++++++--------------
 1 files changed, 54 insertions(+), 17 deletions(-)
mm/percpu.c +54 -17
index 39f4022..c173763 100644
@@ -632,6 +632,13 @@ static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk, int off, int size,
 		pcpu_unmap(chunk, unmap_start, unmap_end, flush);
 }
 
+static int __pcpu_map_pages(unsigned long addr, struct page **pages,
+			    int nr_pages)
+{
+	return map_kernel_range_noflush(addr, nr_pages << PAGE_SHIFT,
+					PAGE_KERNEL, pages);
+}
+
 /**
  * pcpu_map - map pages into a pcpu_chunk
  * @chunk: chunk of interest
@@ -651,11 +658,9 @@ static int pcpu_map(struct pcpu_chunk *chunk, int page_start, int page_end)
 	WARN_ON(chunk->immutable);
 
 	for_each_possible_cpu(cpu) {
-		err = map_kernel_range_noflush(
-				pcpu_chunk_addr(chunk, cpu, page_start),
-				(page_end - page_start) << PAGE_SHIFT,
-				PAGE_KERNEL,
-				pcpu_chunk_pagep(chunk, cpu, page_start));
+		err = __pcpu_map_pages(pcpu_chunk_addr(chunk, cpu, page_start),
+				       pcpu_chunk_pagep(chunk, cpu, page_start),
+				       page_end - page_start);
 		if (err < 0)
 			return err;
 	}
@@ -1274,12 +1279,12 @@ ssize_t __init pcpu_embed_first_chunk(size_t static_size, size_t reserved_size,
  * 4k page first chunk setup helper.
  */
 static struct page **pcpu4k_pages __initdata;
-static int pcpu4k_nr_static_pages __initdata;
+static int pcpu4k_unit_pages __initdata;
 
 static struct page * __init pcpu4k_get_page(unsigned int cpu, int pageno)
 {
-	if (pageno < pcpu4k_nr_static_pages)
-		return pcpu4k_pages[cpu * pcpu4k_nr_static_pages + pageno];
+	if (pageno < pcpu4k_unit_pages)
+		return pcpu4k_pages[cpu * pcpu4k_unit_pages + pageno];
 	return NULL;
 }
 
@@ -1306,22 +1311,24 @@ ssize_t __init pcpu_4k_first_chunk(size_t static_size, size_t reserved_size,
 				   pcpu_fc_free_fn_t free_fn,
 				   pcpu_fc_populate_pte_fn_t populate_pte_fn)
 {
+	static struct vm_struct vm;
 	size_t pages_size;
 	unsigned int cpu;
 	int i, j;
 	ssize_t ret;
 
-	pcpu4k_nr_static_pages = PFN_UP(static_size);
+	pcpu4k_unit_pages = PFN_UP(max_t(size_t, static_size + reserved_size,
+					 PCPU_MIN_UNIT_SIZE));
 
 	/* unaligned allocations can't be freed, round up to page size */
-	pages_size = PFN_ALIGN(pcpu4k_nr_static_pages * num_possible_cpus() *
+	pages_size = PFN_ALIGN(pcpu4k_unit_pages * num_possible_cpus() *
 			       sizeof(pcpu4k_pages[0]));
 	pcpu4k_pages = alloc_bootmem(pages_size);
 
-	/* allocate and copy */
+	/* allocate pages */
 	j = 0;
 	for_each_possible_cpu(cpu)
-		for (i = 0; i < pcpu4k_nr_static_pages; i++) {
+		for (i = 0; i < pcpu4k_unit_pages; i++) {
 			void *ptr;
 
 			ptr = alloc_fn(cpu, PAGE_SIZE);
@@ -1330,18 +1337,48 @@ ssize_t __init pcpu_4k_first_chunk(size_t static_size, size_t reserved_size,
 					   "4k page for cpu%u\n", cpu);
 				goto enomem;
 			}
-
-			memcpy(ptr, __per_cpu_load + i * PAGE_SIZE, PAGE_SIZE);
 			pcpu4k_pages[j++] = virt_to_page(ptr);
 		}
 
+	/* allocate vm area, map the pages and copy static data */
+	vm.flags = VM_ALLOC;
+	vm.size = num_possible_cpus() * pcpu4k_unit_pages << PAGE_SHIFT;
+	vm_area_register_early(&vm, PAGE_SIZE);
+
+	for_each_possible_cpu(cpu) {
+		unsigned long unit_addr = (unsigned long)vm.addr +
+			(cpu * pcpu4k_unit_pages << PAGE_SHIFT);
+
+		for (i = 0; i < pcpu4k_unit_pages; i++)
+			populate_pte_fn(unit_addr + (i << PAGE_SHIFT));
+
+		/* pte already populated, the following shouldn't fail */
+		ret = __pcpu_map_pages(unit_addr,
+				       &pcpu4k_pages[cpu * pcpu4k_unit_pages],
+				       pcpu4k_unit_pages);
+		if (ret < 0)
+			panic("failed to map percpu area, err=%zd\n", ret);
+
+		/*
+		 * FIXME: Archs with virtual cache should flush local
+		 * cache for the linear mapping here - something
+		 * equivalent to flush_cache_vmap() on the local cpu.
+		 * flush_cache_vmap() can't be used as most supporting
+		 * data structures are not set up yet.
+		 */
+
+		/* copy static data */
+		memcpy((void *)unit_addr, __per_cpu_load, static_size);
+	}
+
 	/* we're ready, commit */
-	pr_info("PERCPU: Allocated %d 4k pages, static data %zu bytes\n",
-		pcpu4k_nr_static_pages, static_size);
+	pr_info("PERCPU: %d 4k pages per cpu, static data %zu bytes\n",
+		pcpu4k_unit_pages, static_size);
 
 	ret = pcpu_setup_first_chunk(pcpu4k_get_page, static_size,
 				     reserved_size, -1,
-				     -1, NULL, populate_pte_fn);
+				     pcpu4k_unit_pages << PAGE_SHIFT, vm.addr,
+				     NULL);
 	goto out_free_ar;
 
 enomem:
Signature · 1 line
1.6.0.2
#5 lore raw

Re: [PATCH 04/10] percpu: make 4k first chunk allocator map memory

mm/percpu.c ssize_t __init pcpu_4k_first_chunk(size_t static_size, size_t reserved_size, · line 1376

Quoted 2 lines · Tejun Heo
mm/percpu.c +2 -0
+	pr_info("PERCPU: %d 4k pages per cpu, static data %zu bytes\n",
+		pcpu4k_unit_pages, static_size);

It occurs to me that this is may include a bad assumption. Page size need not be 4K. I don't know that it matters, but looking at this:

mm/percpu.c ssize_t __init pcpu_4k_first_chunk(size_t static_size, size_t reserved_size, · line 1346

Quoted 2 lines
mm/percpu.c +2 -0
+	vm.size = num_possible_cpus() * pcpu4k_unit_pages << PAGE_SHIFT;
+	vm_area_register_early(&vm, PAGE_SIZE);

It might.

David

author maintainer
#6 lore raw

Re: [PATCH 04/10] percpu: make 4k first chunk allocator map memory

Hello,

Quoted 8 lines · David Howells

mm/percpu.c ssize_t __init pcpu_4k_first_chunk(size_t static_size, size_t reserved_size, · line 1376

Quoted 2 lines · Tejun Heo
mm/percpu.c +2 -0
+	pr_info("PERCPU: %d 4k pages per cpu, static data %zu bytes\n",
+		pcpu4k_unit_pages, static_size);

It occurs to me that this is may include a bad assumption. Page size need not be 4K. I don't know that it matters, but looking at this:

Quoted 4 lines

mm/percpu.c ssize_t __init pcpu_4k_first_chunk(size_t static_size, size_t reserved_size, · line 1346

Quoted 2 lines
mm/percpu.c +2 -0
+	vm.size = num_possible_cpus() * pcpu4k_unit_pages << PAGE_SHIFT;
+	vm_area_register_early(&vm, PAGE_SIZE);

It might.

Yeah, I realized that after trying to convert powerpc64 and ia64 and renamed it to page. I'll soon post the patches.

Thanks.

Signature · 1 line
tejun
author maintainer
#7 lore raw

PATCH 5/10 x86,percpu: generalize lpage first chunk allocator

In reply to Tejun Heo

Generalize and move x86 setup_pcpu_lpage() into pcpu_lpage_first_chunk(). setup_pcpu_lpage() now is a simple wrapper around the generalized version. Other than taking size parameters and using arch supplied callbacks to allocate/free/map memory, pcpu_lpage_first_chunk() is identical to the original implementation.

This simplifies arch code and will help converting more archs to dynamic percpu allocator.

While at it, factor out pcpu_calc_fc_sizes() which is common to pcpu_embed_first_chunk() and pcpu_lpage_first_chunk().

[ Impact: code reorganization and generalization ]

---
 arch/x86/include/asm/percpu.h  |    9 --
 arch/x86/kernel/setup_percpu.c |  169 ++------------------------------
 arch/x86/mm/pageattr.c         |    1 +
 include/linux/percpu.h         |   27 +++++
 mm/percpu.c                    |  209 +++++++++++++++++++++++++++++++++++++++-
 5 files changed, 244 insertions(+), 171 deletions(-)
arch/x86/include/asm/percpu.h +0 -9
index 103f1dd..a18c038 100644
@@ -156,15 +156,6 @@ do {							\
 /* We can use this directly for local CPU (faster). */
 DECLARE_PER_CPU(unsigned long, this_cpu_off);
 
-#ifdef CONFIG_NEED_MULTIPLE_NODES
-void *pcpu_lpage_remapped(void *kaddr);
-#else
-static inline void *pcpu_lpage_remapped(void *kaddr)
-{
-	return NULL;
-}
-#endif
-
 #endif /* !__ASSEMBLY__ */
 
 #ifdef CONFIG_SMP
arch/x86/kernel/setup_percpu.c +11 -158

189 lines not on this page · raw message

arch/x86/mm/pageattr.c +1 -0
index 1b734d7..c106f78 100644
@@ -12,6 +12,7 @@
 #include <linux/seq_file.h>
 #include <linux/debugfs.h>
 #include <linux/pfn.h>
+#include <linux/percpu.h>
 
 #include <asm/e820.h>
 #include <asm/processor.h>
include/linux/percpu.h +27 -0
index 41b5bfa..9f6bfd7 100644
@@ -62,6 +62,7 @@ typedef struct page * (*pcpu_get_page_fn_t)(unsigned int cpu, int pageno);
 typedef void * (*pcpu_fc_alloc_fn_t)(unsigned int cpu, size_t size);
 typedef void (*pcpu_fc_free_fn_t)(void *ptr, size_t size);
 typedef void (*pcpu_fc_populate_pte_fn_t)(unsigned long addr);
+typedef void (*pcpu_fc_map_fn_t)(void *ptr, size_t size, void *addr);
 
 extern size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
 				size_t static_size, size_t reserved_size,
@@ -79,6 +80,32 @@ extern ssize_t __init pcpu_4k_first_chunk(
 				pcpu_fc_free_fn_t free_fn,
 				pcpu_fc_populate_pte_fn_t populate_pte_fn);
 
+#ifdef CONFIG_NEED_MULTIPLE_NODES
+extern ssize_t __init pcpu_lpage_first_chunk(
+				size_t static_size, size_t reserved_size,
+				ssize_t dyn_size, size_t lpage_size,
+				pcpu_fc_alloc_fn_t alloc_fn,
+				pcpu_fc_free_fn_t free_fn,
+				pcpu_fc_map_fn_t map_fn);
+
+extern void *pcpu_lpage_remapped(void *kaddr);
+#else
+static inline ssize_t __init pcpu_lpage_first_chunk(
+				size_t static_size, size_t reserved_size,
+				ssize_t dyn_size, size_t lpage_size,
+				pcpu_fc_alloc_fn_t alloc_fn,
+				pcpu_fc_free_fn_t free_fn,
+				pcpu_fc_map_fn_t map_fn)
+{
+	return -EINVAL;
+}
+
+static inline void *pcpu_lpage_remapped(void *kaddr)
+{
+	return NULL;
+}
+#endif
+
 /*
  * Use this to get to a cpu's version of the per-cpu object
  * dynamically allocated. Non-atomic access to the current CPU's
mm/percpu.c +205 -4

227 lines not on this page · raw message

Signature · 1 line
1.6.0.2
author maintainer
#8 lore raw

PATCH 6/10 percpu: simplify pcpu_setup_first_chunk()

In reply to Tejun Heo

Now that all first chunk allocator helpers allocate and map the first chunk themselves, there's no need to have optional default alloc/map in pcpu_setup_first_chunk(). Drop @populate_pte_fn and only leave @dyn_size optional and make all other params mandatory.

This makes it much easier to follow what pcpu_setup_first_chunk() is doing and what actual differences tweaking each parameter results in.

[ Impact: drop unused code path ]

---
 arch/sparc/kernel/smp_64.c |    2 +-
 include/linux/percpu.h     |    5 +-
 mm/percpu.c                |  104 +++++++++++++-------------------------------
 3 files changed, 33 insertions(+), 78 deletions(-)
arch/sparc/kernel/smp_64.c +1 -1
index fa44eaf..ccad7b2 100644
@@ -1528,7 +1528,7 @@ void __init setup_per_cpu_areas(void)
 
 	pcpu_unit_size = pcpu_setup_first_chunk(pcpur_get_page, static_size,
 						PERCPU_MODULE_RESERVE, dyn_size,
-						PCPU_CHUNK_SIZE, vm.addr, NULL);
+						PCPU_CHUNK_SIZE, vm.addr);
 
 	free_bootmem(__pa(pcpur_ptrs), ptrs_size);
 
include/linux/percpu.h +2 -3
index 9f6bfd7..ec64357 100644
@@ -66,9 +66,8 @@ typedef void (*pcpu_fc_map_fn_t)(void *ptr, size_t size, void *addr);
 
 extern size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
 				size_t static_size, size_t reserved_size,
-				ssize_t dyn_size, ssize_t unit_size,
-				void *base_addr,
-				pcpu_fc_populate_pte_fn_t populate_pte_fn);
+				ssize_t dyn_size, size_t unit_size,
+				void *base_addr);
 
 extern ssize_t __init pcpu_embed_first_chunk(
 				size_t static_size, size_t reserved_size,
mm/percpu.c +30 -74

191 lines not on this page · raw message

Signature · 1 line
1.6.0.2
author maintainer
#9 lore raw

PATCH 7/10 percpu: reorder a few functions in mm/percpu.c

In reply to Tejun Heo

(de)populate functions are about to be reimplemented to drop pcpu_chunk->page array. Move a few functions so that the rewrite patch doesn't have code movement making it more difficult to read.

[ Impact: code movement ]

---
 mm/percpu.c |   90 +++++++++++++++++++++++++++++-----------------------------
 1 files changed, 45 insertions(+), 45 deletions(-)
mm/percpu.c +45 -45
index 452d3f3..770db98 100644
@@ -181,12 +181,6 @@ static int pcpu_page_idx(unsigned int cpu, int page_idx)
 	return cpu * pcpu_unit_pages + page_idx;
 }
 
-static struct page **pcpu_chunk_pagep(struct pcpu_chunk *chunk,
-				      unsigned int cpu, int page_idx)
-{
-	return &chunk->page[pcpu_page_idx(cpu, page_idx)];
-}
-
 static unsigned long pcpu_chunk_addr(struct pcpu_chunk *chunk,
 				     unsigned int cpu, int page_idx)
 {
@@ -194,6 +188,12 @@ static unsigned long pcpu_chunk_addr(struct pcpu_chunk *chunk,
 		(pcpu_page_idx(cpu, page_idx) << PAGE_SHIFT);
 }
 
+static struct page **pcpu_chunk_pagep(struct pcpu_chunk *chunk,
+				      unsigned int cpu, int page_idx)
+{
+	return &chunk->page[pcpu_page_idx(cpu, page_idx)];
+}
+
 static bool pcpu_chunk_page_occupied(struct pcpu_chunk *chunk,
 				     int page_idx)
 {
@@ -583,6 +583,45 @@ static void pcpu_unmap(struct pcpu_chunk *chunk, int page_start, int page_end,
 				       pcpu_chunk_addr(chunk, last, page_end));
 }
 
+static int __pcpu_map_pages(unsigned long addr, struct page **pages,
+			    int nr_pages)
+{
+	return map_kernel_range_noflush(addr, nr_pages << PAGE_SHIFT,
+					PAGE_KERNEL, pages);
+}
+
+/**
+ * pcpu_map - map pages into a pcpu_chunk
+ * @chunk: chunk of interest
+ * @page_start: page index of the first page to map
+ * @page_end: page index of the last page to map + 1
+ *
+ * For each cpu, map pages [@page_start,@page_end) into @chunk.
+ * vcache is flushed afterwards.
+ */
+static int pcpu_map(struct pcpu_chunk *chunk, int page_start, int page_end)
+{
+	unsigned int last = num_possible_cpus() - 1;
+	unsigned int cpu;
+	int err;
+
+	/* map must not be done on immutable chunk */
+	WARN_ON(chunk->immutable);
+
+	for_each_possible_cpu(cpu) {
+		err = __pcpu_map_pages(pcpu_chunk_addr(chunk, cpu, page_start),
+				       pcpu_chunk_pagep(chunk, cpu, page_start),
+				       page_end - page_start);
+		if (err < 0)
+			return err;
+	}
+
+	/* flush at once, please read comments in pcpu_unmap() */
+	flush_cache_vmap(pcpu_chunk_addr(chunk, 0, page_start),
+			 pcpu_chunk_addr(chunk, last, page_end));
+	return 0;
+}
+
 /**
  * pcpu_depopulate_chunk - depopulate and unmap an area of a pcpu_chunk
  * @chunk: chunk to depopulate
@@ -632,45 +671,6 @@ static void pcpu_depopulate_chunk(struct pcpu_chunk *chunk, int off, int size,
 		pcpu_unmap(chunk, unmap_start, unmap_end, flush);
 }
 
-static int __pcpu_map_pages(unsigned long addr, struct page **pages,
-			    int nr_pages)
-{
-	return map_kernel_range_noflush(addr, nr_pages << PAGE_SHIFT,
-					PAGE_KERNEL, pages);
-}
-
-/**
- * pcpu_map - map pages into a pcpu_chunk
- * @chunk: chunk of interest
- * @page_start: page index of the first page to map
- * @page_end: page index of the last page to map + 1
- *
- * For each cpu, map pages [@page_start,@page_end) into @chunk.
- * vcache is flushed afterwards.
- */
-static int pcpu_map(struct pcpu_chunk *chunk, int page_start, int page_end)
-{
-	unsigned int last = num_possible_cpus() - 1;
-	unsigned int cpu;
-	int err;
-
-	/* map must not be done on immutable chunk */
-	WARN_ON(chunk->immutable);
-
-	for_each_possible_cpu(cpu) {
-		err = __pcpu_map_pages(pcpu_chunk_addr(chunk, cpu, page_start),
-				       pcpu_chunk_pagep(chunk, cpu, page_start),
-				       page_end - page_start);
-		if (err < 0)
-			return err;
-	}
-
-	/* flush at once, please read comments in pcpu_unmap() */
-	flush_cache_vmap(pcpu_chunk_addr(chunk, 0, page_start),
-			 pcpu_chunk_addr(chunk, last, page_end));
-	return 0;
-}
-
 /**
  * pcpu_populate_chunk - populate and map an area of a pcpu_chunk
  * @chunk: chunk of interest
Signature · 1 line
1.6.0.2
author maintainer
#10 lore raw

PATCH 8/10 percpu: drop pcpu_chunk->page[]

In reply to Tejun Heo

percpu core doesn't need to tack all the allocated pages. It needs to know whether certain pages are populated and a way to reverse map address to page when freeing. This patch drops pcpu_chunk->page[] and use populated bitmap and vmalloc_to_page() lookup instead. Using vmalloc_to_page() exclusively is also possible but complicates first chunk handling, inflates cache footprint and prevents non-standard memory allocation for percpu memory.

pcpu_chunk->page[] was used to track each page's allocation and allowed asymmetric population which happens during failure path; however, with single bitmap for all units, this is no longer possible. Bite the bullet and rewrite (de)populate functions so that things are done in clearly separated steps such that asymmetric population doesn't happen. This makes the (de)population process much more modular and will also ease implementing non-standard memory usage in the future (e.g. large pages).

This makes @get_page_fn parameter to pcpu_setup_first_chunk() unnecessary. The parameter is dropped and all first chunk helpers are updated accordingly. Please note that despite the volume most changes to first chunk helpers are symbol renames for variables which don't need to be referenced outside of the helper anymore.

This change reduces memory usage and cache footprint of pcpu_chunk. Now only #unit_pages bits are necessary per chunk.

[ Impact: reduced memory usage and cache footprint for bookkeeping ]

Cc Ingo Molnar <[email protected]>
Cc David Miller <[email protected]>
---
 arch/sparc/kernel/smp_64.c |   42 +--
 include/linux/percpu.h     |    3 +-
 mm/percpu.c                |  604 ++++++++++++++++++++++++++++----------------
 3 files changed, 400 insertions(+), 249 deletions(-)
arch/sparc/kernel/smp_64.c +15 -27
index ccad7b2..f2f22ee 100644
@@ -1415,19 +1415,6 @@ static void * __init pcpu_alloc_bootmem(unsigned int cpu, unsigned long size,
 #endif
 }
 
-static size_t pcpur_size __initdata;
-static void **pcpur_ptrs __initdata;
-
-static struct page * __init pcpur_get_page(unsigned int cpu, int pageno)
-{
-	size_t off = (size_t)pageno << PAGE_SHIFT;
-
-	if (off >= pcpur_size)
-		return NULL;
-
-	return virt_to_page(pcpur_ptrs[cpu] + off);
-}
-
 #define PCPU_CHUNK_SIZE (4UL * 1024UL * 1024UL)
 
 static void __init pcpu_map_range(unsigned long start, unsigned long end,
@@ -1491,25 +1478,26 @@ void __init setup_per_cpu_areas(void)
 	size_t dyn_size, static_size = __per_cpu_end - __per_cpu_start;
 	static struct vm_struct vm;
 	unsigned long delta, cpu;
-	size_t pcpu_unit_size;
+	size_t size_sum, pcpu_unit_size;
 	size_t ptrs_size;
+	void **ptrs;
 
-	pcpur_size = PFN_ALIGN(static_size + PERCPU_MODULE_RESERVE +
-			       PERCPU_DYNAMIC_RESERVE);
-	dyn_size = pcpur_size - static_size - PERCPU_MODULE_RESERVE;
+	size_sum = PFN_ALIGN(static_size + PERCPU_MODULE_RESERVE +
+			     PERCPU_DYNAMIC_RESERVE);
+	dyn_size = size_sum - static_size - PERCPU_MODULE_RESERVE;
 
 
-	ptrs_size = PFN_ALIGN(num_possible_cpus() * sizeof(pcpur_ptrs[0]));
-	pcpur_ptrs = alloc_bootmem(ptrs_size);
+	ptrs_size = PFN_ALIGN(num_possible_cpus() * sizeof(ptrs[0]));
+	ptrs = alloc_bootmem(ptrs_size);
 
 	for_each_possible_cpu(cpu) {
-		pcpur_ptrs[cpu] = pcpu_alloc_bootmem(cpu, PCPU_CHUNK_SIZE,
-						     PCPU_CHUNK_SIZE);
+		ptrs[cpu] = pcpu_alloc_bootmem(cpu, PCPU_CHUNK_SIZE,
+					       PCPU_CHUNK_SIZE);
 
-		free_bootmem(__pa(pcpur_ptrs[cpu] + pcpur_size),
-			     PCPU_CHUNK_SIZE - pcpur_size);
+		free_bootmem(__pa(ptrs[cpu] + size_sum),
+			     PCPU_CHUNK_SIZE - size_sum);
 
-		memcpy(pcpur_ptrs[cpu], __per_cpu_load, static_size);
+		memcpy(ptrs[cpu], __per_cpu_load, static_size);
 	}
 
 	/* allocate address and map */
@@ -1523,14 +1511,14 @@ void __init setup_per_cpu_areas(void)
 
 		start += cpu * PCPU_CHUNK_SIZE;
 		end = start + PCPU_CHUNK_SIZE;
-		pcpu_map_range(start, end, virt_to_page(pcpur_ptrs[cpu]));
+		pcpu_map_range(start, end, virt_to_page(ptrs[cpu]));
 	}
 
-	pcpu_unit_size = pcpu_setup_first_chunk(pcpur_get_page, static_size,
+	pcpu_unit_size = pcpu_setup_first_chunk(static_size,
 						PERCPU_MODULE_RESERVE, dyn_size,
 						PCPU_CHUNK_SIZE, vm.addr);
 
-	free_bootmem(__pa(pcpur_ptrs), ptrs_size);
+	free_bootmem(__pa(ptrs), ptrs_size);
 
 	delta = (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start;
 	for_each_possible_cpu(cpu) {
include/linux/percpu.h +1 -2
index ec64357..63c8b7a 100644
@@ -58,13 +58,12 @@
 
 extern void *pcpu_base_addr;
 
-typedef struct page * (*pcpu_get_page_fn_t)(unsigned int cpu, int pageno);
 typedef void * (*pcpu_fc_alloc_fn_t)(unsigned int cpu, size_t size);
 typedef void (*pcpu_fc_free_fn_t)(void *ptr, size_t size);
 typedef void (*pcpu_fc_populate_pte_fn_t)(unsigned long addr);
 typedef void (*pcpu_fc_map_fn_t)(void *ptr, size_t size, void *addr);
 
-extern size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,
+extern size_t __init pcpu_setup_first_chunk(
 				size_t static_size, size_t reserved_size,
 				ssize_t dyn_size, size_t unit_size,
 				void *base_addr);
mm/percpu.c +384 -220

860 lines not on this page · raw message

Signature · 1 line
1.6.0.2
author maintainer
#11 lore raw

PATCH 9/10 percpu: allow non-linear / sparse cpu -> unit mapping

In reply to Tejun Heo

Currently cpu and unit are always identity mapped. To allow more efficient large page support on NUMA and lazy allocation for possible but offline cpus, cpu -> unit mapping needs to be non-linear and/or sparse. This can be easily implemented by adding a cpu -> unit mapping array and using it whenever looking up the matching unit for a cpu.

The only unusal conversion is in pcpu_chunk_addr_search(). The passed in address is unit0 based and unit0 might not be in use so it needs to be converted to address of an in-use unit. This is easily done by adding the unit offset for the current processor.

[ Impact: allows non-linear/sparse cpu -> unit mapping, no visible change yet ]

Cc Ingo Molnar <[email protected]>
Cc David Miller <[email protected]>
---
 arch/sparc/kernel/smp_64.c |    2 +-
 include/linux/percpu.h     |    3 +-
 mm/percpu.c                |  129 ++++++++++++++++++++++++++++++++------------
 3 files changed, 97 insertions(+), 37 deletions(-)
arch/sparc/kernel/smp_64.c +1 -1
index f2f22ee..6970333 100644
@@ -1516,7 +1516,7 @@ void __init setup_per_cpu_areas(void)
 
 	pcpu_unit_size = pcpu_setup_first_chunk(static_size,
 						PERCPU_MODULE_RESERVE, dyn_size,
-						PCPU_CHUNK_SIZE, vm.addr);
+						PCPU_CHUNK_SIZE, vm.addr, NULL);
 
 	free_bootmem(__pa(ptrs), ptrs_size);
 
include/linux/percpu.h +2 -1
index 63c8b7a..1e0e887 100644
@@ -57,6 +57,7 @@
 #endif
 
 extern void *pcpu_base_addr;
+extern const int *pcpu_unit_map;
 
 typedef void * (*pcpu_fc_alloc_fn_t)(unsigned int cpu, size_t size);
 typedef void (*pcpu_fc_free_fn_t)(void *ptr, size_t size);
@@ -66,7 +67,7 @@ typedef void (*pcpu_fc_map_fn_t)(void *ptr, size_t size, void *addr);
 extern size_t __init pcpu_setup_first_chunk(
 				size_t static_size, size_t reserved_size,
 				ssize_t dyn_size, size_t unit_size,
-				void *base_addr);
+				void *base_addr, const int *unit_map);
 
 extern ssize_t __init pcpu_embed_first_chunk(
 				size_t static_size, size_t reserved_size,
mm/percpu.c +94 -35

256 lines not on this page · raw message

Signature · 1 line
1.6.0.2
author maintainer
#12 lore raw

PATCH 10/10 percpu: teach large page allocator about NUMA

In reply to Tejun Heo

Large page first chunk allocator is primarily used for NUMA machines; however, its NUMA handling is extremely simplistic. Regardless of their proximity, each cpu is put into separate large page just to return most of the allocated space back wasting large amount of vmalloc space and increasing cache footprint.

This patch teachs NUMA details to large page allocator. Given processor proximity information, pcpu_lpage_build_unit_map() will find fitting cpu -> unit mapping in which cpus in LOCAL_DISTANCE share the same large page and not too much virtual address space is wasted.

This greatly reduces the unit and thus chunk size and wastes much less address space for the first chunk. For example, on 4/4 NUMA machine, the original code occupied 16MB of virtual space for the first chunk while the new code only uses 4MB - one 2MB page for each node.

[ Impact: much better space efficiency on NUMA machines ]

Cc Ingo Molnar <[email protected]>
Cc Jan Beulich <[email protected]>
Cc Andi Kleen <[email protected]>
Cc David Miller <[email protected]>
---
 arch/x86/kernel/setup_percpu.c |   72 ++++++--
 include/linux/percpu.h         |   24 +++-
 mm/percpu.c                    |  358 +++++++++++++++++++++++++++++++---------
 3 files changed, 359 insertions(+), 95 deletions(-)
arch/x86/kernel/setup_percpu.c +55 -17
index 4f2e0ac..7501bb1 100644
@@ -149,36 +149,73 @@ static void __init pcpul_map(void *ptr, size_t size, void *addr)
 	set_pmd(pmd, pmd_v);
 }
 
+static int pcpu_lpage_cpu_distance(unsigned int from, unsigned int to)
+{
+	if (early_cpu_to_node(from) == early_cpu_to_node(to))
+		return LOCAL_DISTANCE;
+	else
+		return REMOTE_DISTANCE;
+}
+
 static ssize_t __init setup_pcpu_lpage(size_t static_size, bool chosen)
 {
 	size_t reserve = PERCPU_MODULE_RESERVE + PERCPU_DYNAMIC_RESERVE;
+	size_t dyn_size = reserve - PERCPU_FIRST_CHUNK_RESERVE;
+	size_t unit_map_size, unit_size;
+	int *unit_map;
+	int nr_units;
+	ssize_t ret;
+
+	/* on non-NUMA, embedding is better */
+	if (!chosen && !pcpu_need_numa())
+		return -EINVAL;
+
+	/* need PSE */
+	if (!cpu_has_pse) {
+		pr_warning("PERCPU: lpage allocator requires PSE\n");
+		return -EINVAL;
+	}
 
+	/* allocate and build unit_map */
+	unit_map_size = num_possible_cpus() * sizeof(int);
+	unit_map = alloc_bootmem_nopanic(unit_map_size);
+	if (!unit_map) {
+		pr_warning("PERCPU: failed to allocate unit_map\n");
+		return -ENOMEM;
+	}
+
+	ret = pcpu_lpage_build_unit_map(static_size,
+					PERCPU_FIRST_CHUNK_RESERVE,
+					&dyn_size, &unit_size, PMD_SIZE,
+					unit_map, pcpu_lpage_cpu_distance);
+	if (ret < 0) {
+		pr_warning("PERCPU: failed to build unit_map\n");
+		goto out_free;
+	}
+	nr_units = ret;
+
+	/* do the parameters look okay? */
 	if (!chosen) {
 		size_t vm_size = VMALLOC_END - VMALLOC_START;
-		size_t tot_size = num_possible_cpus() * PMD_SIZE;
-
-		/* on non-NUMA, embedding is better */
-		if (!pcpu_need_numa())
-			return -EINVAL;
+		size_t tot_size = nr_units * unit_size;
 
 		/* don't consume more than 20% of vmalloc area */
 		if (tot_size > vm_size / 5) {
 			pr_info("PERCPU: too large chunk size %zuMB for "
 				"large page remap\n", tot_size >> 20);
-			return -EINVAL;
+			ret = -EINVAL;
+			goto out_free;
 		}
 	}
 
-	/* need PSE */
-	if (!cpu_has_pse) {
-		pr_warning("PERCPU: lpage allocator requires PSE\n");
-		return -EINVAL;
-	}
-
-	return pcpu_lpage_first_chunk(static_size, PERCPU_FIRST_CHUNK_RESERVE,
-				      reserve - PERCPU_FIRST_CHUNK_RESERVE,
-				      PMD_SIZE,
-				      pcpu_fc_alloc, pcpu_fc_free, pcpul_map);
+	ret = pcpu_lpage_first_chunk(static_size, PERCPU_FIRST_CHUNK_RESERVE,
+				     dyn_size, unit_size, PMD_SIZE,
+				     unit_map, nr_units,
+				     pcpu_fc_alloc, pcpu_fc_free, pcpul_map);
+out_free:
+	if (ret < 0)
+		free_bootmem(__pa(unit_map), unit_map_size);
+	return ret;
 }
 #else
 static ssize_t __init setup_pcpu_lpage(size_t static_size, bool chosen)
@@ -299,7 +336,8 @@ void __init setup_per_cpu_areas(void)
 	/* alrighty, percpu areas up and running */
 	delta = (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start;
 	for_each_possible_cpu(cpu) {
-		per_cpu_offset(cpu) = delta + cpu * pcpu_unit_size;
+		per_cpu_offset(cpu) =
+			delta + pcpu_unit_map[cpu] * pcpu_unit_size;
 		per_cpu(this_cpu_off, cpu) = per_cpu_offset(cpu);
 		per_cpu(cpu_number, cpu) = cpu;
 		setup_percpu_segment(cpu);
include/linux/percpu.h +22 -2
index 1e0e887..8ce91af 100644
@@ -62,6 +62,7 @@ extern const int *pcpu_unit_map;
 typedef void * (*pcpu_fc_alloc_fn_t)(unsigned int cpu, size_t size);
 typedef void (*pcpu_fc_free_fn_t)(void *ptr, size_t size);
 typedef void (*pcpu_fc_populate_pte_fn_t)(unsigned long addr);
+typedef int (pcpu_fc_cpu_distance_fn_t)(unsigned int from, unsigned int to);
 typedef void (*pcpu_fc_map_fn_t)(void *ptr, size_t size, void *addr);
 
 extern size_t __init pcpu_setup_first_chunk(
@@ -80,18 +81,37 @@ extern ssize_t __init pcpu_4k_first_chunk(
 				pcpu_fc_populate_pte_fn_t populate_pte_fn);
 
 #ifdef CONFIG_NEED_MULTIPLE_NODES
+extern int __init pcpu_lpage_build_unit_map(
+				size_t static_size, size_t reserved_size,
+				ssize_t *dyn_sizep, size_t *unit_sizep,
+				size_t lpage_size, int *unit_map,
+				pcpu_fc_cpu_distance_fn_t cpu_distance_fn);
+
 extern ssize_t __init pcpu_lpage_first_chunk(
 				size_t static_size, size_t reserved_size,
-				ssize_t dyn_size, size_t lpage_size,
+				size_t dyn_size, size_t unit_size,
+				size_t lpage_size, const int *unit_map,
+				int nr_units,
 				pcpu_fc_alloc_fn_t alloc_fn,
 				pcpu_fc_free_fn_t free_fn,
 				pcpu_fc_map_fn_t map_fn);
 
 extern void *pcpu_lpage_remapped(void *kaddr);
 #else
+static inline int pcpu_lpage_build_unit_map(
+				size_t static_size, size_t reserved_size,
+				ssize_t *dyn_sizep, size_t *unit_sizep,
+				size_t lpage_size, int *unit_map,
+				pcpu_fc_cpu_distance_fn_t cpu_distance_fn)
+{
+	return -EINVAL;
+}
+
 static inline ssize_t __init pcpu_lpage_first_chunk(
 				size_t static_size, size_t reserved_size,
-				ssize_t dyn_size, size_t lpage_size,
+				size_t dyn_size, size_t unit_size,
+				size_t lpage_size, const int *unit_map,
+				int nr_units,
 				pcpu_fc_alloc_fn_t alloc_fn,
 				pcpu_fc_free_fn_t free_fn,
 				pcpu_fc_map_fn_t map_fn)
mm/percpu.c +282 -76

439 lines not on this page · raw message

Signature · 1 line
1.6.0.2
maintainer
#13 lore raw
In reply to Tejun Heo
Quoted 5 lines · On Wed, 24 Jun 2009 22:30:06 +0900 Tejun Heo <[email protected]> wrote:

This patchset is available in the following git tree and will be published in for-next if there's no major objection. It might get rebased before going into for-next.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git lpage-numa

<tries to read the patches>

Boy, this stuff is complicated. Does it all work?

The Impact: lines were useful :)

I assume from the tremendous number of for_each_possible_cpu()s that CPU hotplug awareness won't be happening.

Do we have a feeling for the amount of wastage here? If

	num_possible_cpus() - num_online_cpus() == N

and N is large, what did it cost?

And what are reasonable values of N?

#14 lore raw
Quoted 6 lines

I assume from the tremendous number of for_each_possible_cpu()s that CPU hotplug awareness won't be happening.

Do we have a feeling for the amount of wastage here? If

	num_possible_cpus() - num_online_cpus() == N

Haven't read the new patches, but per cpu data always was sized for all possible CPUs.

Quoted 3 lines

and N is large, what did it cost?

And what are reasonable values of N?

N should normally not be large anymore, since num_possible_cpus() is sized based on firmware information now.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#15 lore raw
Quoted 11 lines · Andi Kleen

Haven't read the new patches, but per cpu data always was sized for all possible CPUs.

Quoted 3 lines

and N is large, what did it cost?

And what are reasonable values of N?

N should normally not be large anymore, since num_possible_cpus() is sized based on firmware information now.

*Ahem* virtual machines *ahem*...

I have discussed this with Tejun, and the plan is to allocate the percpu information when a processor is first brought online (but not removed when it is offlined again.) It's a real problem for 32-bit VMs, so it's more important than you'd think.

-hpa

#16 lore raw
Quoted 7 lines · H. Peter Anvin
Quoted 6 lines · Andi Kleen

Haven't read the new patches, but per cpu data always was sized for all possible CPUs.

Quoted 1 line

and N is large, what did it cost?

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

And what are reasonable values of N?

N should normally not be large anymore, since num_possible_cpus() is sized based on firmware information now.

*Ahem* virtual machines *ahem*...

And? Even there's not that big typically.

The traditional problem was just for 128 NR_CPUs kernel were nothing was sized based on machine capacity.

Also on large systems the VMs shouldn't be sized for full capacity.

Quoted 5 lines

I have discussed this with Tejun, and the plan is to allocate the percpu information when a processor is first brought online (but not removed when it is offlined again.) It's a real problem for 32-bit VMs, so it's more important than you'd think.

You have to rewrite all code that does for_each_possible_cpu (x) in initialization then to use callbacks. It would be a gigantic change all over the tree.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#17 lore raw
Quoted 10 lines · Andi Kleen
Quoted 2 lines
Quoted 1 line

*Ahem* virtual machines *ahem*...

And? Even there's not that big typically.

The traditional problem was just for 128 NR_CPUs kernel were nothing was sized based on machine capacity.

Also on large systems the VMs shouldn't be sized for full capacity.

We have already have cases where the "possible" CPUs have eaten up the entire vmalloc area on 32 bits. In real use. It's a real problem.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#18 lore raw
Quoted 6 lines · H. Peter Anvin
Quoted 5 lines · Andi Kleen
Quoted 2 lines
Quoted 1 line

*Ahem* virtual machines *ahem*...

And? Even there's not that big typically.

Quoted 8 lines
Quoted 5 lines

The traditional problem was just for 128 NR_CPUs kernel were nothing was sized based on machine capacity.

Also on large systems the VMs shouldn't be sized for full capacity.

We have already have cases where the "possible" CPUs have eaten up the entire vmalloc area on 32 bits. In real use. It's a real problem.

That's hard to believe or a serious bug/misconfiguration somewhere.

Each per CPU data should be <100k (let's say 200k with some slack for modules), so to fill vmalloc you would need hundreds of CPUs, which a 32bit kernel doesn't really support anyways because it doesn't support enough memory for that many CPUs.

Perhaps you had firmware/hypervisor who passed a gigantic impossible value here? If yes thy

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#19 lore raw
Quoted 11 lines · Andi Kleen

That's hard to believe or a serious bug/misconfiguration somewhere.

Each per CPU data should be <100k (let's say 200k with some slack for modules), so to fill vmalloc you would need hundreds of CPUs, which a 32bit kernel doesn't really support anyways because it doesn't support enough memory for that many CPUs.

Perhaps you had firmware/hypervisor who passed a gigantic impossible value here? If yes thy

I don't know why, but in the configuration I saw each percpu area was something like 1.8 MB.

-hpa

#20 lore raw
Quoted 2 lines

I don't know why, but in the configuration I saw each percpu area was something like 1.8 MB.

That sounds like something that needs to be investigated and fixed then. In a sense per cpu space is precious like stack space and shouldn't be wasted.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
author maintainer
#21 lore raw
Quoted 6 lines · Andi Kleen
Quoted 2 lines

I don't know why, but in the configuration I saw each percpu area was something like 1.8 MB.

That sounds like something that needs to be investigated and fixed then. In a sense per cpu space is precious like stack space and shouldn't be wasted.

The large percpu space is most likely from lockdep.

Thanks.

Signature · 1 line
tejun
#22 lore raw
Quoted 9 lines · Tejun Heo
Quoted 5 lines · Andi Kleen
Quoted 2 lines

I don't know why, but in the configuration I saw each percpu area was something like 1.8 MB.

That sounds like something that needs to be investigated and fixed then. In a sense per cpu space is precious like stack space and shouldn't be wasted.

The large percpu space is most likely from lockdep.

That makes sense.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#23 lore raw
In reply to Tejun Heo
Quoted 9 lines · Tejun Heo
Quoted 6 lines · Andi Kleen
Quoted 2 lines

I don't know why, but in the configuration I saw each percpu area was something like 1.8 MB.

That sounds like something that needs to be investigated and fixed then. In a sense per cpu space is precious like stack space and shouldn't be wasted.

The large percpu space is most likely from lockdep.

It would be better then to fix lockdep then to add weird hacks for that elsewhere.

e.g. it could just allocate buffers in a cpu up/down callback and only put a pointer to these buffers into percpu.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
author maintainer
#24 lore raw
In reply to Andrew Morton

Hello,

Quoted 7 lines · Andrew Morton
Quoted 4 lines · On Wed, 24 Jun 2009 22:30:06 +0900 Tejun Heo <[email protected]> wrote:

This patchset is available in the following git tree and will be published in for-next if there's no major objection. It might get rebased before going into for-next.

Quoted 5 lines
Quoted 1 line

git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git lpage-numa

<tries to read the patches>

Boy, this stuff is complicated. Does it all work?

I sure hope so.

Quoted 1 line

The Impact: lines were useful :)

Eh.. well, it looks like it's going the way of dodo tho.

Quoted 10 lines

I assume from the tremendous number of for_each_possible_cpu()s that CPU hotplug awareness won't be happening.

Do we have a feeling for the amount of wastage here? If

	num_possible_cpus() - num_online_cpus() == N

and N is large, what did it cost?

And what are reasonable values of N?

The goal is to eventually implement has_ever_been_online_cpus (any better naming?) mask and allocate only for those cpus. I think I mentioned it in one of the patch descriptions but anyways the unit_map and lpage improvments implemented in this patchset will be used for that purpose. The plan is to treat possible but offline cpus as if they belong to separate group such that they don't end up sharing the same PMD page and later when those cpus come up the generic 4k mapping can kick in and map them.

Thanks.

Signature · 1 line
tejun
#25 lore raw
Quoted 5 lines · Tejun Heo
Quoted 1 line · Andrew Morton

The Impact: lines were useful :)

Eh.. well, it looks like it's going the way of dodo tho.

Yes. I found them useful too but some people (including Andrew ;) were too argumentative about it so i asked you to remove them from this patch-set. So yes, it's all going the way of the dodo.

And yes, i attribute the excellent stability of the x86 code in this cycle partly to the effect of impact lines. They really helped us be very conscious with dangerous changes.

Ingo

#26 lore raw
In reply to Andrew Morton
Quoted 2 lines · Andrew Morton

I assume from the tremendous number of for_each_possible_cpu()s that CPU hotplug awareness won't be happening.

Per cpu areas are allocated for all possible processors. No need to handle offlining and onlining them.

maintainer
#27 lore raw
Quoted 7 lines · On Mon, 29 Jun 2009 19:20:53 -0400 (EDT) Christoph Lameter…
Quoted 2 lines · Andrew Morton

I assume from the tremendous number of for_each_possible_cpu()s that CPU hotplug awareness won't be happening.

Per cpu areas are allocated for all possible processors. No need to handle offlining and onlining them.

Well yes. My point is that this is a bug-not-a-feature ;)

#28 lore raw
Quoted 5 lines · Andrew Morton
Quoted 2 lines · On Mon, 29 Jun 2009 19:20:53 -0400 (EDT) Christoph Lameter…

On Wed, 24 Jun 2009, Andrew Morton wrote:

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I assume from the tremendous number of for_each_possible_cpu()s that CPU hotplug awareness won't be happening.

Per cpu areas are allocated for all possible processors. No need to handle offlining and onlining them.

Well yes. My point is that this is a bug-not-a-feature ;)

Its a feature that I would like to exploit in the future to get rid of some of the hotplug callbacks in the allocator.

Some state also needs to be kept for offlined processors. This is done in per cpu data.

#29 lore raw
In reply to Andrew Morton
Quoted 5 lines · Andrew Morton
Quoted 2 lines · On Mon, 29 Jun 2009 19:20:53 -0400 (EDT) Christoph Lameter…

On Wed, 24 Jun 2009, Andrew Morton wrote:

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I assume from the tremendous number of for_each_possible_cpu()s that CPU hotplug awareness won't be happening.

Per cpu areas are allocated for all possible processors. No need to handle offlining and onlining them.

Well yes. My point is that this is a bug-not-a-feature ;)

Yeah, it's a bug for something like a virtual environment which boots generic kernels that might have 64 possible CPUs (on a true 64-way system), but which will have fewer in practice.

It's pretty basic stuff: the on-demand allocation of percpu resources.

Ingo

#30 lore raw
Quoted 3 lines · Ingo Molnar

Yeah, it's a bug for something like a virtual environment which boots generic kernels that might have 64 possible CPUs (on a true 64-way system), but which will have fewer in practice.

A machine (and a virtual environment) can indicate via the BIOS tables or ACPI that there are less "possible" cpus. That is actually very common.

The difference between actual and possible cpus only has to be the number of processors that could be brought up later. In a regular system that is pretty much zero. In a fancy system with actual hotpluggable cpus there would be a difference but usually the number of hotpluggable cpus is minimal.

#31 lore raw
Quoted 6 lines · Christoph Lameter
Quoted 3 lines · Ingo Molnar

Yeah, it's a bug for something like a virtual environment which boots generic kernels that might have 64 possible CPUs (on a true 64-way system), but which will have fewer in practice.

Quoted 8 lines

A machine (and a virtual environment) can indicate via the BIOS tables or ACPI that there are less "possible" cpus. That is actually very common.

The difference between actual and possible cpus only has to be the number of processors that could be brought up later. In a regular system that is pretty much zero. In a fancy system with actual hotpluggable cpus there would be a difference but usually the number of hotpluggable cpus is minimal.

A hypervisor running on a 4-socket Beckton platform will have 32 (without hyperthreading) or 64 (with hyperthreading) CPUs to
allocate 1Q2010. A hypervisor supporting ACPI hot-plug would be able
then to hot-plug from one to 64 cores into a single linux guest if it wanted to (and didn't overcommit, so gang-scheduling wasn't required).

Such a hypervisor might typically pass the maximum number of CPUs in via the guest ACPI tables when the guest boots, to provide the maximum flexibility in being able to grow and shrink the guest, with only a subset marked as initially present.

I've been working recently with a 128 (shanghai) / 192 (istanbul) core shared memory system which supports hot plugging CPUs into Linux, and the feature would be less useful if a hot-plug event could fail because the per-cpu area couldn't be allocated at the time of the event (which argues for pre-allocation, if a large physically contiguous region is required).

That said, I don't see much use for 32-bit kernels in environments with such CPU counts.

scott

#32 lore raw
In reply to Christoph Lameter
Quoted 5 lines · Christoph Lameter
Quoted 3 lines · Ingo Molnar

Yeah, it's a bug for something like a virtual environment which boots generic kernels that might have 64 possible CPUs (on a true 64-way system), but which will have fewer in practice.

i think this bit should be quoted too, because it is the crux of the issue:

Quoted 7 lines
Quoted 2 lines

It's pretty basic stuff: the on-demand allocation of percpu resources.

A machine (and a virtual environment) can indicate via the BIOS tables or ACPI that there are less "possible" cpus. That is actually very common.

Quoted 5 lines

The difference between actual and possible cpus only has to be the number of processors that could be brought up later. In a regular system that is pretty much zero. In a fancy system with actual hotpluggable cpus there would be a difference but usually the number of hotpluggable cpus is minimal.

You are arguing against the concept of the demand-allocation of resources, and i dont think that technical argument can be won.

Sure you dont have to demand-allocate if you know the demand beforehand and can preallocate and size accordingly.

But what if not? What if the kernel can run on up to 4096 CPUs and runs on a big box. Why should a virtual machine have the illogical choice between either wasting a lot of RAM preallocating stuff, or limiting its own extendability.

In other words: your proposed change in essence reduces the utility of CPU hotplug. It's a bad idea.

Ingo

#33 lore raw
Quoted 8 lines · Ingo Molnar
Quoted 5 lines

The difference between actual and possible cpus only has to be the number of processors that could be brought up later. In a regular system that is pretty much zero. In a fancy system with actual hotpluggable cpus there would be a difference but usually the number of hotpluggable cpus is minimal.

You are arguing against the concept of the demand-allocation of resources, and i dont think that technical argument can be won.

I looked at allocating for online cpus only a couple of years back but at that per cpu state was kept for offlined cpus in per cpu areas. There are numerous assumptions in per cpu handling all over the kernel that a percpu area is always available. We successfully restricted it to only possible cpus. ACPI may be the worst offender there. If you can get all of that addressed then we can move to pure on demand allocation. Which also would complicate a per cpu memory allocator allocator.

Quoted 2 lines

Sure you dont have to demand-allocate if you know the demand beforehand and can preallocate and size accordingly.

Well you know the demand from the BIOS information.

Quoted 4 lines

But what if not? What if the kernel can run on up to 4096 CPUs and runs on a big box. Why should a virtual machine have the illogical choice between either wasting a lot of RAM preallocating stuff, or limiting its own extendability.

The kernel may be able to run on 4096 but the machines config information that is available via ACPI knows how many processors the machine we are booting on is able to add.

Quoted 2 lines

In other words: your proposed change in essence reduces the utility of CPU hotplug. It's a bad idea.

Change? I am talking about what is the case.

#34 lore raw
Quoted 10 lines · Christoph Lameter
Quoted 4 lines · Ingo Molnar

But what if not? What if the kernel can run on up to 4096 CPUs and runs on a big box. Why should a virtual machine have the illogical choice between either wasting a lot of RAM preallocating stuff, or limiting its own extendability.

The kernel may be able to run on 4096 but the machines config information that is available via ACPI knows how many processors the machine we are booting on is able to add.

I think we might be talking past each other.

The usecase i'm talking about is to boot a generic, many-CPUs-capable kernel in a guest image.

How would you allow that guest to stay on 2 virtual CPUs but still be able to hot-plug many other CPUs if the guest context rises above its original CPU utilization?

Ingo

#35 lore raw
Quoted 3 lines

How would you allow that guest to stay on 2 virtual CPUs but still be able to hot-plug many other CPUs if the guest context rises above its original CPU utilization?

(unless you're planning to rewrite lots of possible cpu users all over the tree) -- the only way is to keep the percpu area small and preallocate.

As long as the per cpu data size stays reasonable (not more than a 100-200k) that's very doable. It probably won't work with 4096 guest CPUs without wasting too much memory, but then I don't think we have any Hypervisor that scales to that many CPUs anyways, so it's not the biggest concern. For the 128CPU case it works (although i might need
to enlarge vmalloc area a bit on 32bit)

Unfortunately a few debugging subsystems seem to currently eat much more, but those just need to be fixed to only allocate state for actually running CPUs, not just possible ones.

I suspect we need a scripts/percpubloat.pl

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
author maintainer
#36 lore raw

Hello, Andi.

Quoted 7 lines · Andi Kleen
Quoted 3 lines

How would you allow that guest to stay on 2 virtual CPUs but still be able to hot-plug many other CPUs if the guest context rises above its original CPU utilization?

(unless you're planning to rewrite lots of possible cpu users all over the tree) -- the only way is to keep the percpu area small and preallocate.

Quoted 6 lines

As long as the per cpu data size stays reasonable (not more than a 100-200k) that's very doable. It probably won't work with 4096 guest CPUs without wasting too much memory, but then I don't think we have any Hypervisor that scales to that many CPUs anyways, so it's not the biggest concern. For the 128CPU case it works (although i might need
to enlarge vmalloc area a bit on 32bit)

I don't see much reason why we should put artificial limit on how much percpu memory could be used. For lockdep, that much of percpu memory is actually necessary. Another layer of indirection surely can lessen the pressure on the generic percpu implementation but the problem can be solved by the generic code without too much difficulty.

Thanks.

Signature · 1 line
tejun
#37 lore raw
In reply to Ingo Molnar
Quoted 2 lines · Ingo Molnar

The usecase i'm talking about is to boot a generic, many-CPUs-capable kernel in a guest image.

The many-cpu-capable kernel is not the issue. The number of potential cpus is config information provided by the hardware. In your case the host system provides the number of possible processors.

Quoted 3 lines

How would you allow that guest to stay on 2 virtual CPUs but still be able to hot-plug many other CPUs if the guest context rises above its original CPU utilization?

The use case is a guest that needs to add a couple of thousand processors?

That will require lots of memory and special configuration of the guest environment simulated. Yes that is an extreme case that will require lots of per cpu memory.

#38 lore raw
Quoted 9 lines · Christoph Lameter
Quoted 2 lines · Ingo Molnar

The usecase i'm talking about is to boot a generic, many-CPUs-capable kernel in a guest image.

The many-cpu-capable kernel is not the issue. The number of potential cpus is config information provided by the hardware. In your case the host system provides the number of possible processors.

Say it says "256 CPUs".

But i only run a 2 CPU guest system. But i want to run it with the _on demand capability_ to scale up to 256 virtual CPUs if needed, without having to reboot the guest and without having to allocate all the percpu space for 256 CPUs beforehand. Ok?

It is a simple, basic OS feature and concept.

Ingo

#39 lore raw
Quoted 1 line · Ingo Molnar

Say it says "256 CPUs".

Ok then you found a bug in the virtualization software ;-).

#40 lore raw
Quoted 5 lines · Christoph Lameter
Quoted 1 line · Ingo Molnar

Say it says "256 CPUs".

Ok then you found a bug in the virtualization software ;-).

No, it runs on a 256 CPUs host system, why shouldnt it be possible to expose that _possible_ expansion space to a guest instance?

Ingo

#41 lore raw
In reply to Ingo Molnar
Quoted 6 lines

Say it says "256 CPUs".

But i only run a 2 CPU guest system. But i want to run it with the _on demand capability_ to scale up to 256 virtual CPUs if needed, without having to reboot the guest and without having to allocate all the percpu space for 256 CPUs beforehand. Ok?

Then you would need to fix lots of subsystems. Everyone doing for_each_possible_cpu() today and not having a cpu notifier callback.

git/linux-2.6% gid for_each_possible_cpu | wc -l
247
git/linux-2.6% gid CPU_UP_PREPARE | wc -l
48

Essentially you would discard the concept of possible CPUs and replace it only with online CPUs, but possible CPUs are widely used.

It's unclear to me it's worth all that work (and how to test it properly), assuming we fix the few per cpu pigs today that make it very expensive to preallocate percpu in the big debug configurations.

Quoted 1 line

It is a simple, basic OS feature and concept.

.. that Linux doesn't support without some heavy lifting.

Not answering emails will not change that fact.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
author maintainer
#42 lore raw
In reply to Christoph Lameter

Hello, Christoph.

Quoted 4 lines · Christoph Lameter

I looked at allocating for online cpus only a couple of years back but at that per cpu state was kept for offlined cpus in per cpu areas. There are numerous assumptions in per cpu handling all over the kernel that a percpu area is always available.

The plan is to allocate and keep percpu areas for cpus which have ever been up. There'll be no taking down of percpu areas. Conversion from possible to has_ever_been_up should be much easier than possible -> online. State keeping will work fine too.

Quoted 4 lines

We successfully restricted it to only possible
cpus. ACPI may be the worst offender there. If you can get all of that addressed then we can move to pure on demand allocation. Which also would complicate a per cpu memory allocator allocator.

I don't think it will be too complex. The necessary bits are already there and they are necessary for other stuff too, so...

Thanks.

Signature · 1 line
tejun
#43 lore raw
Quoted 9 lines · Tejun Heo

Hello, Christoph.

Quoted 4 lines · Christoph Lameter

I looked at allocating for online cpus only a couple of years back but at that per cpu state was kept for offlined cpus in per cpu areas. There are numerous assumptions in per cpu handling all over the kernel that a percpu area is always available.

Quoted 4 lines

The plan is to allocate and keep percpu areas for cpus which have ever been up. There'll be no taking down of percpu areas. Conversion from possible to has_ever_been_up should be much easier than possible -> online. State keeping will work fine too.

That sounds like a very sane plan.

Ingo

#44 lore raw
Quoted 10 lines · Ingo Molnar
Quoted 8 lines · Tejun Heo

Hello, Christoph.

Quoted 4 lines · Christoph Lameter

I looked at allocating for online cpus only a couple of years back but at that per cpu state was kept for offlined cpus in per cpu areas. There are numerous assumptions in per cpu handling all over the kernel that a percpu area is always available.

Quoted 8 lines
Quoted 4 lines

The plan is to allocate and keep percpu areas for cpus which have ever been up. There'll be no taking down of percpu areas. Conversion from possible to has_ever_been_up should be much easier than possible -> online. State keeping will work fine too.

That sounds like a very sane plan.

Ingo

Yes, percpu area dynamic *de*-allocation would almost certainly be a nightmare.

-hpa

#45 lore raw
In reply to Tejun Heo
Quoted 2 lines

I don't think it will be too complex. The necessary bits are already there and they are necessary for other stuff too, so...

Are we looking at a different source base? Here's a random example using possible per cpu data I picked in current git: icmp.c

static int __net_init icmp_sk_init(struct net *net)

{
        int i, err;

        net->ipv4.icmp_sk =
                kzalloc(nr_cpu_ids * sizeof(struct sock *), GFP_KERNEL);
        if (net->ipv4.icmp_sk == NULL)
                return -ENOMEM;

        for_each_possible_cpu(i) {
		... allocate per cpu socket and some other setup ...
	}
} 

static void __net_exit icmp_sk_exit(struct net *net)

{
        int i;

        for_each_possible_cpu(i)
                inet_ctl_sock_destroy(net->ipv4.icmp_sk[i]);
        kfree(net->ipv4.icmp_sk);
        net->ipv4.icmp_sk = NULL;
}

You would need to convert that to use a CPU notifier and callbacks setting up the sockets. Then make sure there are no races in all of this. And get it somehow tested (where is the user base who tests cpu hotplug?)

And there is lots of similar code all over the tree

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
author maintainer
#46 lore raw

Hello, Andi.

Quoted 5 lines · Andi Kleen
Quoted 2 lines

I don't think it will be too complex. The necessary bits are already there and they are necessary for other stuff too, so...

Are we looking at a different source base? Here's a random example using possible per cpu data I picked in current git: icmp.c

I was talking about percpu allocator proper. Yeap, the major work would be in auditing and converting for_each_possible_cpu() users.

Quoted 24 lines

static int __net_init icmp_sk_init(struct net *net)

{
        int i, err;

        net->ipv4.icmp_sk =
                kzalloc(nr_cpu_ids * sizeof(struct sock *), GFP_KERNEL);
        if (net->ipv4.icmp_sk == NULL)
                return -ENOMEM;

        for_each_possible_cpu(i) {
		... allocate per cpu socket and some other setup ...
	}
} 

static void __net_exit icmp_sk_exit(struct net *net)

{
        int i;

        for_each_possible_cpu(i)
                inet_ctl_sock_destroy(net->ipv4.icmp_sk[i]);
        kfree(net->ipv4.icmp_sk);
        net->ipv4.icmp_sk = NULL;
}
Quoted 4 lines

You would need to convert that to use a CPU notifier and callbacks setting up the sockets. Then make sure there are no races in all of this. And get it somehow tested (where is the user base who tests cpu hotplug?)

Maybe it would be better to allocate percpu sockets as proper percpu variables. Initialization would still need callback mechanism tho. I was thinking about adding @init callback to percpu_alloc(), which would be much simpler than doing full cpu hotplug callback.

Quoted 1 line

And there is lots of similar code all over the tree

For static percpu variables, it'll be mostly about converting for_each_possible_cpu() to for_each_used_cpu() as both allocation and initialization can be handled by percpu proper. For dynamic areas, allocation can be handled by percpu proper but cpus coming online would need more work to convert. It'll take some effort but there aren't too many alloc_percpu() users yet and I don't think it will be too difficult. I wouldn't know for sure before I actually try tho.

Thanks.

Signature · 1 line
tejun
#47 lore raw
Quoted 4 lines · Tejun Heo
Quoted 1 line

using possible per cpu data I picked in current git: icmp.c

I was talking about percpu allocator proper. Yeap, the major work would be in auditing and converting for_each_possible_cpu() users.

and testing. that's the hard part. cpu hotplug is normally not well tested. Any code change that requires lots of new code for it will be a problem because that code will then likely bitrot.

Quoted 11 lines
Quoted 8 lines
        kfree(net->ipv4.icmp_sk);
        net->ipv4.icmp_sk = NULL;
}

You would need to convert that to use a CPU notifier and callbacks setting up the sockets. Then make sure there are no races in all of this. And get it somehow tested (where is the user base who tests cpu hotplug?)

Maybe it would be better to allocate percpu sockets as proper percpu variables.

That would be like percpu dentries.

A socket is a very complex structure. Trying to pull it out of its normal allocators would be a bad idea.

I actually had a patch some time ago to move this one over to callbacks, but it was already difficult, and I dropped it.

Quoted 3 lines

Initialization would still need callback mechanism tho. I was thinking about adding @init callback to percpu_alloc(), which would be much simpler than doing full cpu hotplug callback.

I'm not sure it will be actually simpler. It's not just initializing the structure, but all the other set up to make it known to the world.

Quoted 6 lines
Quoted 2 lines

And there is lots of similar code all over the tree

For static percpu variables, it'll be mostly about converting for_each_possible_cpu() to for_each_used_cpu() as both allocation and initialization can be handled by percpu proper. For dynamic areas,

That's the easy part, but you would still need all the callbacks for the extension case.

Quoted 3 lines

allocation can be handled by percpu proper but cpus coming online would need more work to convert. It'll take some effort but there aren't too many alloc_percpu() users yet and I don't think it will be

alloc_percpu()? It affects all DEFINE_PER_CPU users. My current tree has hundreds.

Quoted 1 line

too difficult. I wouldn't know for sure before I actually try tho.

I think it's clear that you haven't tried yet :)

I wrote quite a few per cpu callback handlers over the years and in my experience they are all nasty code with subtle races. The problem is that instead of having a single subset init function which is just single threaded and doesn't need to worry about races you now have multi threaded init, which tends to be a can of worms.

I think a far saner strategy than rewriting every user of DEFINE_PER_CPU, ending up with lots of badly tested code is to:

- Fix the few large size percpu pigs that are problematic today to allocate in a callback.
- Then once the per cpu data in all configurations is <200k (better <100 in the non debug builds) again just keep pre-allocating like we always did
- Possibly adjust the vmalloc area on 32bit based on the possible CPU map at the cost of the direct mapping, to make sure there's always enough mapping space.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
author maintainer
#48 lore raw

Hello, Andi.

Quoted 8 lines · Andi Kleen
Quoted 3 lines · Tejun Heo
Quoted 1 line

using possible per cpu data I picked in current git: icmp.c

I was talking about percpu allocator proper. Yeap, the major work would be in auditing and converting for_each_possible_cpu() users.

and testing. that's the hard part. cpu hotplug is normally not well tested. Any code change that requires lots of new code for it will be a problem because that code will then likely bitrot.

It would be nice to have something to test cpu on/offlining automatically. Something which keeps bringing cpus up and down as the system goes through stress testing.

[--snip--]

Quoted 3 lines
Quoted 1 line

too difficult. I wouldn't know for sure before I actually try tho.

I think it's clear that you haven't tried yet :)

No I haven't yet. I had a pretty good idea about how to implement it in percpu allocator but haven't really looked at the users. So, yeap, it's quite possible that I'm under estimating the problem. Oh well, let's see. Thanks for the warnings.

Quoted 5 lines

I wrote quite a few per cpu callback handlers over the years and in my experience they are all nasty code with subtle races. The problem is that instead of having a single subset init function which is just single threaded and doesn't need to worry about races you now have multi threaded init, which tends to be a can of worms.

I tried a couple (didn't end up sending them out) and yeah they could be quite painful. Bringing up part usually isn't as painful as the other way tho.

Quoted 2 lines

I think a far saner strategy than rewriting every user of DEFINE_PER_CPU, ending up with lots of badly tested code is to:

But I don't think it would be that drastic. Most users are quite simple.

Quoted 8 lines

- Fix the few large size percpu pigs that are problematic today to allocate in a callback.
- Then once the per cpu data in all configurations is <200k (better <100 in the non debug builds) again just keep pre-allocating like we always did
- Possibly adjust the vmalloc area on 32bit based on the possible CPU map at the cost of the direct mapping, to make sure there's always enough mapping space.

I think it's something we eventually need to do. There already are cases where lack of scalable and performant percpu allocation leads to design restrictions and between many core cpus and virtualization the requirements are becoming more varied.

Thanks.

Signature · 1 line
tejun
#49 lore raw
Quoted 3 lines · Tejun Heo

It would be nice to have something to test cpu on/offlining automatically. Something which keeps bringing cpus up and down as the system goes through stress testing.

That's an trivial shell script using echo into sysfs files. It doesn't seem to be widely done (the last time I tried it I promptly ran into some RCU bug). You need a large enough machine for it.

But most stress testing does not actually have good code coverage in my experience. It just runs the same small set of core code all over (you can check now, kernel gcov is finally in)

The tricky part is to actually test the code you want to test.

Quoted 4 lines
Quoted 1 line

ending up with lots of badly tested code is to:

But I don't think it would be that drastic. Most users are quite simple.

But how do you test them properly? And educate
the driver writers? Also it would likely increase code sizes drastically.

Philosophically I think code like that should be a simple operation and turning all the per cpu init code into callbacks is not simple. That makes everything more error prone.

And it's imho unclear if that is all worth it just to avoid wasting some memory in the "256 possible CPUs" case (which I doubt is particularly realistic anyways, at least I don't know of any Hypervisor today that scales to 256 CPUs)

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#50 lore raw
Quoted 4 lines · Andi Kleen

And it's imho unclear if that is all worth it just to avoid wasting some memory in the "256 possible CPUs" case (which I doubt is particularly realistic anyways, at least I don't know of any Hypervisor today that scales to 256 CPUs)

I basically agree. Its not worth it given the rare cases where this matters. It will be a lot of code with callbacks in each subsystem.

One of the motivations of working on revising the percpu handling for me was that we could get rid of these screwy callbacks that are rarely tested and cause all sorts of other issues with locking and serialization.

author maintainer
#51 lore raw
Quoted 7 lines · Christoph Lameter
Quoted 4 lines · Andi Kleen

And it's imho unclear if that is all worth it just to avoid wasting some memory in the "256 possible CPUs" case (which I doubt is particularly realistic anyways, at least I don't know of any Hypervisor today that scales to 256 CPUs)

Quoted 6 lines

I basically agree. Its not worth it given the rare cases where this matters. It will be a lot of code with callbacks in each subsystem.

One of the motivations of working on revising the percpu handling for me was that we could get rid of these screwy callbacks that are rarely tested and cause all sorts of other issues with locking and serialization.

Hmmm.... yeah. I have to agree that callbacks are nasty and requiring all users to use callbacks wouldn't be very nice. Once the current dust settles down, I'll look around and see whether this can be solved in some reasonable way.

Thanks.

Signature · 1 line
tejun
author maintainer
#52 lore raw
In reply to Tejun Heo
Quoted 6 lines · Tejun Heo

Hello,

This patchset is combination of the following two patchsets.

[1] x86,percpu: generalize 4k and lpage allocator
[2] percpu: teach lpage allocator about NUMA

Alpha changes from linus#master pulled into #for-next and this patchset has been published in #for-next on top of that.

Thanks.

Signature · 1 line
tejun