Skip to contentSkip to conversation

percpu: convert ia64 to dynamic percpu and drop the old one, take#2

PATCH In review Tejun Heo 6 participants 15 messages Sep 23, 2009 – Oct 2, 2009

Subsystems (2)
lore
Share & tools
lore

Tejun Heo

author maintainer
#0 lore raw

PATCH 1/1 percpu: convert ia64 to dynamic percpu and drop the old one, take#2

Hello, all.

This is the second take of convert-ia64-to-dynamic-percpu patchset. Changes from the last take[L] are

* 0001 now updates ia64 to not define VMALLOC_END as a macro to vmalloc_end instead of disallowing vmalloc_end as a variable name as suggested by Christoph.

* 0002 added to initialize cpu maps early. This is necessary to get contig memory model working.

* 0004 updated so that dyn_size is calculated correctly for contig model.

This patchset contains the following five patches.

0001-ia64-don-t-alias-VMALLOC_END-to-vmalloc_end.patch 0002-ia64-initialize-cpu-maps-early.patch 0003-ia64-allocate-percpu-area-for-cpu0-like-percpu-areas.patch 0004-ia64-convert-to-dynamic-percpu-allocator.patch 0005-percpu-kill-legacy-percpu-allocator.patch

0001 is misc prep to avoid macro / local variable collision. 0002 makes ia64 arch code initialize cpu possible and present maps before memory initialization. 0003 makes ia64 allocate percpu area for cpu0 in the same way it does for other cpus. 0004 converts ia64 to dynamic percpu allocator and 0005 drops now unused legacy allocator.

Contig memory model was tested on a 16p Tiger4 machine. Discontig and sparse tested on 4-way SGI altix. ski seems to be happy with contig up/smp.

This patchset is available in the following git tree.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git convert-ia64

The new commit ID is dcc91f19c6662b24f1f4e5878d773244f1079724 and it's on top of today's Linus 7fa07729e439a6184bd824746d06a49cca553f15.

Thanks.

 arch/ia64/Kconfig              |    3 
 arch/ia64/kernel/setup.c       |   12 --
 arch/ia64/kernel/vmlinux.lds.S |   11 +-
 arch/ia64/mm/contig.c          |   87 ++++++++++++++++----
 arch/ia64/mm/discontig.c       |  120 +++++++++++++++++++++++++--
 include/linux/percpu.h         |   24 -----
 kernel/module.c                |  150 ----------------------------------
 mm/Makefile                    |    4 
 mm/allocpercpu.c               |  177 -----------------------------------------
 mm/percpu.c                    |    2 
 mm/vmalloc.c                   |   16 +--
 11 files changed, 193 insertions(+), 413 deletions(-)

--
tejun

[L] http://thread.gmane.org/gmane.linux.ports.ia64/20812

author maintainer
#1 lore raw

PATCH 1/5 ia64: don't alias VMALLOC_END to vmalloc_end

If CONFIG_VIRTUAL_MEM_MAP is enabled, ia64 defines macro VMALLOC_END as unsigned long variable vmalloc_end which is adjusted to prepare room for vmemmap. This becomes probnlematic if a local variables vmalloc_end is defined in some function (not very unlikely) and VMALLOC_END is used in the function - the function thinks its referencing the global VMALLOC_END value but would be referencing its own local vmalloc_end variable.

There's no reason VMALLOC_END should be a macro. Just define it as an unsigned long variable if CONFIG_VIRTUAL_MEM_MAP is set to avoid nasty surprises.

Cc Tony Luck <[email protected]>
Cc Fenghua Yu <[email protected]>
Cc linux-ia64 <[email protected]>
Cc Christoph Lameter <[email protected]>
---
 arch/ia64/include/asm/meminit.h |    2 +-
 arch/ia64/include/asm/pgtable.h |    3 +--
 arch/ia64/mm/contig.c           |    4 ++--
 arch/ia64/mm/discontig.c        |    4 ++--
 arch/ia64/mm/init.c             |    4 ++--
 5 files changed, 8 insertions(+), 9 deletions(-)
arch/ia64/include/asm/meminit.h +1 -1
index 688a812..61c7b17 100644
@@ -61,7 +61,7 @@ extern int register_active_ranges(u64 start, u64 len, int nid);
 
 #ifdef CONFIG_VIRTUAL_MEM_MAP
 # define LARGE_GAP	0x40000000 /* Use virtual mem map if hole is > than this */
-  extern unsigned long vmalloc_end;
+  extern unsigned long VMALLOC_END;
   extern struct page *vmem_map;
   extern int find_largest_hole(u64 start, u64 end, void *arg);
   extern int create_mem_map_page_table(u64 start, u64 end, void *arg);
arch/ia64/include/asm/pgtable.h +1 -2
index 8840a69..69bf138 100644
@@ -228,8 +228,7 @@ ia64_phys_addr_valid (unsigned long addr)
 #define VMALLOC_START		(RGN_BASE(RGN_GATE) + 0x200000000UL)
 #ifdef CONFIG_VIRTUAL_MEM_MAP
 # define VMALLOC_END_INIT	(RGN_BASE(RGN_GATE) + (1UL << (4*PAGE_SHIFT - 9)))
-# define VMALLOC_END		vmalloc_end
-  extern unsigned long vmalloc_end;
+extern unsigned long VMALLOC_END;
 #else
 #if defined(CONFIG_SPARSEMEM) && defined(CONFIG_SPARSEMEM_VMEMMAP)
 /* SPARSEMEM_VMEMMAP uses half of vmalloc... */
arch/ia64/mm/contig.c +2 -2
index 2f724d2..1341437 100644
@@ -270,8 +270,8 @@ paging_init (void)
 
 		map_size = PAGE_ALIGN(ALIGN(max_low_pfn, MAX_ORDER_NR_PAGES) *
 			sizeof(struct page));
-		vmalloc_end -= map_size;
-		vmem_map = (struct page *) vmalloc_end;
+		VMALLOC_END -= map_size;
+		vmem_map = (struct page *) VMALLOC_END;
 		efi_memmap_walk(create_mem_map_page_table, NULL);
 
 		/*
arch/ia64/mm/discontig.c +2 -2
index d85ba98..9f24b3c 100644
@@ -666,9 +666,9 @@ void __init paging_init(void)
 	sparse_init();
 
 #ifdef CONFIG_VIRTUAL_MEM_MAP
-	vmalloc_end -= PAGE_ALIGN(ALIGN(max_low_pfn, MAX_ORDER_NR_PAGES) *
+	VMALLOC_END -= PAGE_ALIGN(ALIGN(max_low_pfn, MAX_ORDER_NR_PAGES) *
 		sizeof(struct page));
-	vmem_map = (struct page *) vmalloc_end;
+	vmem_map = (struct page *) VMALLOC_END;
 	efi_memmap_walk(create_mem_map_page_table, NULL);
 	printk("Virtual mem_map starts at 0x%p\n", vmem_map);
 #endif
arch/ia64/mm/init.c +2 -2
index 1d28624..f301071 100644
@@ -44,8 +44,8 @@ extern void ia64_tlb_init (void);
 unsigned long MAX_DMA_ADDRESS = PAGE_OFFSET + 0x100000000UL;
 
 #ifdef CONFIG_VIRTUAL_MEM_MAP
-unsigned long vmalloc_end = VMALLOC_END_INIT;
-EXPORT_SYMBOL(vmalloc_end);
+unsigned long VMALLOC_END = VMALLOC_END_INIT;
+EXPORT_SYMBOL(VMALLOC_END);
 struct page *vmem_map;
 EXPORT_SYMBOL(vmem_map);
 #endif
Signature · 1 line
1.6.4.2
author maintainer
#2 lore raw

PATCH 2/5 ia64: initialize cpu maps early

In reply to Tejun Heo

All information necessary to initialize cpu possible and present maps are available once early_acpi_boot_init() is complete. Reorganize setup_arch() and acpi init functions such that,

* CPU information is printed after LAPIC entries are parsed in early_acpi_boot_init().

* smp_build_cpu_map() is called by setup_arch() instead of acpi functions.

* smp_build_cpu_map() is called once all CPU related information is available before memory is initialized.

This is primarily to allow find_memory() to use cpu maps but is also a general cleanup. Please note that with this change, the somewhat ad-hoc early_cpu_possible_map defined and used for NUMA configurations is probably unnecessary. Something to clean up another day.

Cc Tony Luck <[email protected]>
Cc Fenghua Yu <[email protected]>
Cc linux-ia64 <[email protected]>
---
 arch/ia64/kernel/acpi.c  |   33 +++++++++++++++------------------
 arch/ia64/kernel/setup.c |   11 +++++------
 2 files changed, 20 insertions(+), 24 deletions(-)
arch/ia64/kernel/acpi.c +14 -17
index baec6f0..40574ae 100644
@@ -702,11 +702,23 @@ int __init early_acpi_boot_init(void)
 		printk(KERN_ERR PREFIX
 		       "Error parsing MADT - no LAPIC entries\n");
 
+#ifdef CONFIG_SMP
+	if (available_cpus = 0) {
+		printk(KERN_INFO "ACPI: Found 0 CPUS; assuming 1\n");
+		printk(KERN_INFO "CPU 0 (0x%04x)", hard_smp_processor_id());
+		smp_boot_data.cpu_phys_id[available_cpus] +		    hard_smp_processor_id();
+		available_cpus = 1;	/* We've got at least one of these, no? */
+	}
+	smp_boot_data.cpu_count = available_cpus;
+#endif
+	/* Make boot-up look pretty */
+	printk(KERN_INFO "%d CPUs available, %d CPUs total\n", available_cpus,
+	       total_cpus);
+
 	return 0;
 }
 
-
-
 int __init acpi_boot_init(void)
 {
 
@@ -769,18 +781,8 @@ int __init acpi_boot_init(void)
 	if (acpi_table_parse(ACPI_SIG_FADT, acpi_parse_fadt))
 		printk(KERN_ERR PREFIX "Can't find FADT\n");
 
+#ifdef CONFIG_ACPI_NUMA
 #ifdef CONFIG_SMP
-	if (available_cpus = 0) {
-		printk(KERN_INFO "ACPI: Found 0 CPUS; assuming 1\n");
-		printk(KERN_INFO "CPU 0 (0x%04x)", hard_smp_processor_id());
-		smp_boot_data.cpu_phys_id[available_cpus] -		    hard_smp_processor_id();
-		available_cpus = 1;	/* We've got at least one of these, no? */
-	}
-	smp_boot_data.cpu_count = available_cpus;
-
-	smp_build_cpu_map();
-# ifdef CONFIG_ACPI_NUMA
 	if (srat_num_cpus = 0) {
 		int cpu, i = 1;
 		for (cpu = 0; cpu < smp_boot_data.cpu_count; cpu++)
@@ -789,14 +791,9 @@ int __init acpi_boot_init(void)
 				node_cpuid[i++].phys_id  				    smp_boot_data.cpu_phys_id[cpu];
 	}
-# endif
 #endif
-#ifdef CONFIG_ACPI_NUMA
 	build_cpu_to_node_map();
 #endif
-	/* Make boot-up look pretty */
-	printk(KERN_INFO "%d CPUs available, %d CPUs total\n", available_cpus,
-	       total_cpus);
 	return 0;
 }
 
arch/ia64/kernel/setup.c +5 -6
index 1de86c9..5d77c1e 100644
@@ -566,19 +566,18 @@ setup_arch (char **cmdline_p)
 	early_acpi_boot_init();
 # ifdef CONFIG_ACPI_NUMA
 	acpi_numa_init();
-#ifdef CONFIG_ACPI_HOTPLUG_CPU
+#  ifdef CONFIG_ACPI_HOTPLUG_CPU
 	prefill_possible_map();
-#endif
+#  endif
 	per_cpu_scan_finalize((cpus_weight(early_cpu_possible_map) = 0 ?
 		32 : cpus_weight(early_cpu_possible_map)),
 		additional_cpus > 0 ? additional_cpus : 0);
 # endif
-#else
-# ifdef CONFIG_SMP
-	smp_build_cpu_map();	/* happens, e.g., with the Ski simulator */
-# endif
 #endif /* CONFIG_APCI_BOOT */
 
+#ifdef CONFIG_SMP
+	smp_build_cpu_map();
+#endif
 	find_memory();
 
 	/* process SAL system table: */
Signature · 1 line
1.6.4.2
author maintainer
#3 lore raw

PATCH 3/5 ia64: allocate percpu area for cpu0 like percpu areas for other cpus

In reply to Tejun Heo

cpu0 used special percpu area reserved by the linker, __cpu0_per_cpu, which is set up early in boot by head.S. However, this doesn't guarantee that the area will be on the same node as cpu0 and the percpu area for cpu0 ends up very far away from percpu areas for other cpus which cause problems for congruent percpu allocator.

This patch makes percpu area initialization allocate percpu area for cpu0 like any other cpus and copy it from __cpu0_per_cpu which now resides in the __init area. This means that for cpu0, percpu area is first setup at __cpu0_per_cpu early by head.S and then moved to an area in the linear mapping during memory initialization and it's not allowed to take a pointer to percpu variables between head.S and memory initialization.

Cc Tony Luck <[email protected]>
Cc Fenghua Yu <[email protected]>
Cc linux-ia64 <[email protected]>
---
 arch/ia64/kernel/vmlinux.lds.S |   11 +++++----
 arch/ia64/mm/contig.c          |   47 +++++++++++++++++++++++++--------------
 arch/ia64/mm/discontig.c       |   35 ++++++++++++++++++++---------
 3 files changed, 60 insertions(+), 33 deletions(-)
arch/ia64/kernel/vmlinux.lds.S +6 -5
index 0a0c77b..1295ba3 100644
@@ -166,6 +166,12 @@ SECTIONS
 	}
 #endif
 
+#ifdef	CONFIG_SMP
+  . = ALIGN(PERCPU_PAGE_SIZE);
+  __cpu0_per_cpu = .;
+  . = . + PERCPU_PAGE_SIZE;	/* cpu0 per-cpu space */
+#endif
+
   . = ALIGN(PAGE_SIZE);
   __init_end = .;
 
@@ -198,11 +204,6 @@ SECTIONS
   data : { } :data
   .data : AT(ADDR(.data) - LOAD_OFFSET)
 	{
-#ifdef	CONFIG_SMP
-  . = ALIGN(PERCPU_PAGE_SIZE);
-		__cpu0_per_cpu = .;
-  . = . + PERCPU_PAGE_SIZE;	/* cpu0 per-cpu space */
-#endif
 		INIT_TASK_DATA(PAGE_SIZE)
 		CACHELINE_ALIGNED_DATA(SMP_CACHE_BYTES)
 		READ_MOSTLY_DATA(SMP_CACHE_BYTES)
arch/ia64/mm/contig.c +27 -14

63 lines not on this page · raw message

arch/ia64/mm/discontig.c +24 -11
index 9f24b3c..200282b 100644
@@ -143,17 +143,30 @@ static void *per_cpu_node_setup(void *cpu_data, int node)
 	int cpu;
 
 	for_each_possible_early_cpu(cpu) {
-		if (cpu = 0) {
-			void *cpu0_data = __cpu0_per_cpu;
-			__per_cpu_offset[cpu] = (char*)cpu0_data -
-				__per_cpu_start;
-		} else if (node = node_cpuid[cpu].nid) {
-			memcpy(__va(cpu_data), __phys_per_cpu_start,
-			       __per_cpu_end - __per_cpu_start);
-			__per_cpu_offset[cpu] = (char*)__va(cpu_data) -
-				__per_cpu_start;
-			cpu_data += PERCPU_PAGE_SIZE;
-		}
+		void *src = cpu = 0 ? __cpu0_per_cpu : __phys_per_cpu_start;
+
+		if (node != node_cpuid[cpu].nid)
+			continue;
+
+		memcpy(__va(cpu_data), src, __per_cpu_end - __per_cpu_start);
+		__per_cpu_offset[cpu] = (char *)__va(cpu_data) -
+			__per_cpu_start;
+
+		/*
+		 * percpu area for cpu0 is moved from the __init area
+		 * which is setup by head.S and used till this point.
+		 * Update ar.k3.  This move is ensures that percpu
+		 * area for cpu0 is on the correct node and its
+		 * virtual address isn't insanely far from other
+		 * percpu areas which is important for congruent
+		 * percpu allocator.
+		 */
+		if (cpu = 0)
+			ia64_set_kr(IA64_KR_PER_CPU_DATA,
+				    (unsigned long)cpu_data -
+				    (unsigned long)__per_cpu_start);
+
+		cpu_data += PERCPU_PAGE_SIZE;
 	}
 #endif
 	return cpu_data;
Signature · 1 line
1.6.4.2
author maintainer
#4 lore raw

PATCH 4/5 ia64: convert to dynamic percpu allocator

In reply to Tejun Heo

Unlike other archs, ia64 reserves space for percpu areas during early memory initialization. These areas occupy a contiguous region indexed by cpu number on contiguous memory model or are grouped by node on discontiguous memory model.

As allocation and initialization are done by the arch code, all that setup_per_cpu_areas() needs to do is communicating the determined layout to the percpu allocator. This patch implements setup_per_cpu_areas() for both contig and discontig memory models and drops HAVE_LEGACY_PER_CPU_AREA.

Please note that for contig model, the allocation itself is modified only to allocate for possible cpus instead of NR_CPUS. As dynamic percpu allocator can handle non-direct mapping, there's no reason to allocate memory for cpus which aren't possible.

Cc Tony Luck <[email protected]>
Cc Fenghua Yu <[email protected]>
Cc linux-ia64 <[email protected]>
---
 arch/ia64/Kconfig        |    3 --
 arch/ia64/kernel/setup.c |   12 ------
 arch/ia64/mm/contig.c    |   58 ++++++++++++++++++++++++++++---
 arch/ia64/mm/discontig.c |   85 ++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 138 insertions(+), 20 deletions(-)
arch/ia64/Kconfig +0 -3
index 011a1cd..e624611 100644
@@ -89,9 +89,6 @@ config GENERIC_TIME_VSYSCALL
 	bool
 	default y
 
-config HAVE_LEGACY_PER_CPU_AREA
-	def_bool y
-
 config HAVE_SETUP_PER_CPU_AREA
 	def_bool y
 
arch/ia64/kernel/setup.c +0 -12
index 5d77c1e..bc1ef4a 100644
@@ -855,18 +855,6 @@ identify_cpu (struct cpuinfo_ia64 *c)
 }
 
 /*
- * In UP configuration, setup_per_cpu_areas() is defined in
- * include/linux/percpu.h
- */
-#ifdef CONFIG_SMP
-void __init
-setup_per_cpu_areas (void)
-{
-	/* start_kernel() requires this... */
-}
-#endif
-
-/*
  * Do the following calculations:
  *
  * 1. the max. cache line size.
arch/ia64/mm/contig.c +53 -5

73 lines not on this page · raw message

arch/ia64/mm/discontig.c +85 -0

91 lines not on this page · raw message

Signature · 1 line
1.6.4.2
author maintainer
#5 lore raw

PATCH 5/5 percpu: kill legacy percpu allocator

In reply to Tejun Heo

With ia64 converted, there's no arch left which still uses legacy percpu allocator. Kill it.

Cc Ingo Molnar <[email protected]>
Cc Rusty Russell <[email protected]>
Cc Christoph Lameter <[email protected]>
---
 include/linux/percpu.h |   24 -------
 kernel/module.c        |  150 ----------------------------------------
 mm/Makefile            |    4 -
 mm/allocpercpu.c       |  177 ------------------------------------------------
 mm/percpu.c            |    2 -
 5 files changed, 0 insertions(+), 357 deletions(-)
 delete mode 100644 mm/allocpercpu.c
include/linux/percpu.h +0 -24
index 878836c..5baf5b8 100644
@@ -34,8 +34,6 @@
 
 #ifdef CONFIG_SMP
 
-#ifndef CONFIG_HAVE_LEGACY_PER_CPU_AREA
-
 /* minimum unit size, also is the maximum supported allocation size */
 #define PCPU_MIN_UNIT_SIZE		PFN_ALIGN(64 << 10)
 
@@ -130,28 +128,6 @@ extern int __init pcpu_page_first_chunk(size_t reserved_size,
 #define per_cpu_ptr(ptr, cpu)	SHIFT_PERCPU_PTR((ptr), per_cpu_offset((cpu)))
 
 extern void *__alloc_reserved_percpu(size_t size, size_t align);
-
-#else /* CONFIG_HAVE_LEGACY_PER_CPU_AREA */
-
-struct percpu_data {
-	void *ptrs[1];
-};
-
-/* pointer disguising messes up the kmemleak objects tracking */
-#ifndef CONFIG_DEBUG_KMEMLEAK
-#define __percpu_disguise(pdata) (struct percpu_data *)~(unsigned long)(pdata)
-#else
-#define __percpu_disguise(pdata) (struct percpu_data *)(pdata)
-#endif
-
-#define per_cpu_ptr(ptr, cpu)						\
-({									\
-        struct percpu_data *__p = __percpu_disguise(ptr);		\
-        (__typeof__(ptr))__p->ptrs[(cpu)];				\
-})
-
-#endif /* CONFIG_HAVE_LEGACY_PER_CPU_AREA */
-
 extern void *__alloc_percpu(size_t size, size_t align);
 extern void free_percpu(void *__pdata);
 
kernel/module.c +0 -150

162 lines not on this page · raw message

mm/Makefile +0 -4
index 728a9fd..3230eb5 100644
@@ -34,11 +34,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o
 obj-$(CONFIG_MEMORY_HOTPLUG) += memory_hotplug.o
 obj-$(CONFIG_FS_XIP) += filemap_xip.o
 obj-$(CONFIG_MIGRATION) += migrate.o
-ifndef CONFIG_HAVE_LEGACY_PER_CPU_AREA
 obj-$(CONFIG_SMP) += percpu.o
-else
-obj-$(CONFIG_SMP) += allocpercpu.o
-endif
 obj-$(CONFIG_QUICKLIST) += quicklist.o
 obj-$(CONFIG_CGROUP_MEM_RES_CTLR) += memcontrol.o page_cgroup.o
 obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o
mm/allocpercpu.c deleted +0 -177

177 lines not on this page · raw message

mm/percpu.c +0 -2
index 43d8cac..adbc5a4 100644
@@ -46,8 +46,6 @@
  *
  * To use this allocator, arch code should do the followings.
  *
- * - drop CONFIG_HAVE_LEGACY_PER_CPU_AREA
- *
  * - define __addr_to_pcpu_ptr() and __pcpu_ptr_to_addr() to translate
  *   regular address to percpu pointer and back if they need to be
  *   different from the default
Signature · 1 line
1.6.4.2
#7 lore raw
In reply to Tejun Heo
Quoted 33 lines · Tejun Heo

Hello, all.

This is the second take of convert-ia64-to-dynamic-percpu patchset. Changes from the last take[L] are

* 0001 now updates ia64 to not define VMALLOC_END as a macro to vmalloc_end instead of disallowing vmalloc_end as a variable name as suggested by Christoph.

* 0002 added to initialize cpu maps early. This is necessary to get contig memory model working.

* 0004 updated so that dyn_size is calculated correctly for contig model.

This patchset contains the following five patches.

0001-ia64-don-t-alias-VMALLOC_END-to-vmalloc_end.patch 0002-ia64-initialize-cpu-maps-early.patch 0003-ia64-allocate-percpu-area-for-cpu0-like-percpu-areas.patch 0004-ia64-convert-to-dynamic-percpu-allocator.patch 0005-percpu-kill-legacy-percpu-allocator.patch

0001 is misc prep to avoid macro / local variable collision. 0002 makes ia64 arch code initialize cpu possible and present maps before memory initialization. 0003 makes ia64 allocate percpu area for cpu0 in the same way it does for other cpus. 0004 converts ia64 to dynamic percpu allocator and 0005 drops now unused legacy allocator.

Contig memory model was tested on a 16p Tiger4 machine. Discontig and sparse tested on 4-way SGI altix. ski seems to be happy with contig up/smp.

Quoted 6 lines

This patchset is available in the following git tree.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git convert-ia64

The new commit ID is dcc91f19c6662b24f1f4e5878d773244f1079724 and it's on top of today's Linus 7fa07729e439a6184bd824746d06a49cca553f15.

Tony, can you please ack ia64 part?

Thanks.

Signature · 1 line
tejun
#13 lore raw

PATCH 3/5 ia64: allocate percpu area for cpu0 like percpu areas for other cpus

In reply to Tejun Heo

cpu0 used special percpu area reserved by the linker, __cpu0_per_cpu, which is set up early in boot by head.S. However, this doesn't guarantee that the area will be on the same node as cpu0 and the percpu area for cpu0 ends up very far away from percpu areas for other cpus which cause problems for congruent percpu allocator.

This patch makes percpu area initialization allocate percpu area for cpu0 like any other cpus and copy it from __cpu0_per_cpu which now resides in the __init area. This means that for cpu0, percpu area is first setup at __cpu0_per_cpu early by head.S and then moved to an area in the linear mapping during memory initialization and it's not allowed to take a pointer to percpu variables between head.S and memory initialization.

Cc Tony Luck <[email protected]>
Cc Fenghua Yu <[email protected]>
Cc linux-ia64 <[email protected]>

---
Tony Luck didn't receive this one and couldn't find it in archives either. Repost. Unchanged from the original posting.

 arch/ia64/kernel/vmlinux.lds.S |   11 +++++----
 arch/ia64/mm/contig.c          |   47 +++++++++++++++++++++++++--------------
 arch/ia64/mm/discontig.c       |   35 ++++++++++++++++++++---------
 3 files changed, 60 insertions(+), 33 deletions(-)
arch/ia64/kernel/vmlinux.lds.S +6 -5
index 0a0c77b..1295ba3 100644
@@ -166,6 +166,12 @@ SECTIONS
 	}
 #endif
 
+#ifdef	CONFIG_SMP
+  . = ALIGN(PERCPU_PAGE_SIZE);
+  __cpu0_per_cpu = .;
+  . = . + PERCPU_PAGE_SIZE;	/* cpu0 per-cpu space */
+#endif
+
   . = ALIGN(PAGE_SIZE);
   __init_end = .;
 
@@ -198,11 +204,6 @@ SECTIONS
   data : { } :data
   .data : AT(ADDR(.data) - LOAD_OFFSET)
 	{
-#ifdef	CONFIG_SMP
-  . = ALIGN(PERCPU_PAGE_SIZE);
-		__cpu0_per_cpu = .;
-  . = . + PERCPU_PAGE_SIZE;	/* cpu0 per-cpu space */
-#endif
 		INIT_TASK_DATA(PAGE_SIZE)
 		CACHELINE_ALIGNED_DATA(SMP_CACHE_BYTES)
 		READ_MOSTLY_DATA(SMP_CACHE_BYTES)
arch/ia64/mm/contig.c +27 -14

63 lines not on this page · raw message

arch/ia64/mm/discontig.c +24 -11
index 9f24b3c..200282b 100644
@@ -143,17 +143,30 @@ static void *per_cpu_node_setup(void *cpu_data, int node)
 	int cpu;
 
 	for_each_possible_early_cpu(cpu) {
-		if (cpu = 0) {
-			void *cpu0_data = __cpu0_per_cpu;
-			__per_cpu_offset[cpu] = (char*)cpu0_data -
-				__per_cpu_start;
-		} else if (node = node_cpuid[cpu].nid) {
-			memcpy(__va(cpu_data), __phys_per_cpu_start,
-			       __per_cpu_end - __per_cpu_start);
-			__per_cpu_offset[cpu] = (char*)__va(cpu_data) -
-				__per_cpu_start;
-			cpu_data += PERCPU_PAGE_SIZE;
-		}
+		void *src = cpu = 0 ? __cpu0_per_cpu : __phys_per_cpu_start;
+
+		if (node != node_cpuid[cpu].nid)
+			continue;
+
+		memcpy(__va(cpu_data), src, __per_cpu_end - __per_cpu_start);
+		__per_cpu_offset[cpu] = (char *)__va(cpu_data) -
+			__per_cpu_start;
+
+		/*
+		 * percpu area for cpu0 is moved from the __init area
+		 * which is setup by head.S and used till this point.
+		 * Update ar.k3.  This move is ensures that percpu
+		 * area for cpu0 is on the correct node and its
+		 * virtual address isn't insanely far from other
+		 * percpu areas which is important for congruent
+		 * percpu allocator.
+		 */
+		if (cpu = 0)
+			ia64_set_kr(IA64_KR_PER_CPU_DATA,
+				    (unsigned long)cpu_data -
+				    (unsigned long)__per_cpu_start);
+
+		cpu_data += PERCPU_PAGE_SIZE;
 	}
 #endif
 	return cpu_data;
Signature · 1 line
1.6.4.2
#9 lore raw

Great. Ill post a new rev of the per cpu ops patchset.

Tony: Could we use a global register for the per cpu address? That would make IA64 work similar to sparc.

#10 lore raw
Quoted 2 lines

Tony: Could we use a global register for the per cpu address? That would make IA64 work similar to sparc.

Would that be a useful trade of resources for convenience? We've already hard-wired r13 for "current". Grabbing another one would require fixing (since user code will have clobbered it). Possibly re-working any existing code that already uses whatever register you choose.

How would that compare with using [r13]?

We might have a krN free ... but kregs are a lot slower than real registers.

-Tony

#11 lore raw
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Tony" = Tony Luck <Luck> writes:

Tony: Could we use a global register for the per cpu address? That would make IA64 work similar to sparc.

Tony> Would that be a useful trade of resources for convenience? Tony> We've already hard-wired r13 for "current". Grabbing another one Tony> would require fixing (since user code will have clobbered it). Tony> Possibly re-working any existing code that already uses whatever Tony> register you choose.

r3, r4 and r5 are currently unused by the kernel, and unused
by GCC and ICC. Only hand-written assembler and weird compilers use
those registers(and my virtual-machine monitor :-(). If you wanted to experiment, that'd be a starting place.

I'm not sure of the advantage though -- TLB mapping is relatively cheap, and we're no longer hard-wiring the translation register.

You';d have to do somne careful benchmarking on a wide variety of workloads and machines to get a definitive answer.

---
Dr Peter Chubb www.nicta.com.au peter DOT chubb AT nicta.com.au
http://www.ertos.nicta.com.au ERTOS within National ICT Australia
From Imagination to Impact Imagining the (ICT) Future

#12 lore raw
Quoted 7 lines · Peter Chubb

r3, r4 and r5 are currently unused by the kernel, and unused
by GCC and ICC. Only hand-written assembler and weird compilers use
those registers(and my virtual-machine monitor :-(). If you wanted to experiment, that'd be a starting place.

I'm not sure of the advantage though -- TLB mapping is relatively cheap, and we're no longer hard-wiring the translation register.

Dynamic and static per cpu variables could use relative access to that register. This would reduce code size, avoid the use of a TLB entry.

Quoted 2 lines

You';d have to do somne careful benchmarking on a wide variety of workloads and machines to get a definitive answer.

I have some patches here that make heavy use of dynamic percpu allocations in the allocators to optimize the alloc / free paths.

#14 lore raw
In reply to Tejun Heo
Quoted 4 lines · Tejun Heo

Hello, all.

This is the second take of convert-ia64-to-dynamic-percpu patchset. Changes from the last take[L] are

Patchset pushed out for linux-next with Rusty and Tony's acks added.

Thanks.

Signature · 1 line
tejun