Skip to contentSkip to conversation

x86,percpu: fix pageattr handling with remap allocator

PATCH In review Tejun Heo 7 participants 38 messages May 14, 2009 – May 22, 2009

Subsystems (5)
lore
Share & tools
lore

Tejun Heo

author maintainer
#0 lore raw

PATCH 1/1 x86,percpu: fix pageattr handling with remap allocator

Hello,

Upon ack, please pull from the following git tree.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git x86-percpu-pageattr

This patchset fixes subtile bug in pageattr handling when remap percpu first chunk allocator is in use and implements percpu_alloc kernel parameter so that allocator can be selected from boot prompt.

This problem was spotted by Jan Beulich.

The remap allocator allocates a PMD page per cpu, returns whatever is unnecessary to the page allocator and remaps the PMD page into vmalloc area to construct the first percpu chunk. This is to take advantage of large page mapping. However this creates active aliases for the recycled pages. When some user allocates the recycled pages and tries to change pageattr on it, remapped PMD alias might end up having different attributes with the regular page mapped addresses which can lead to subtle data corruption according to Andi.

Similar problem exists for the high mapped area of x86_64 kernel which is handled by detecting the alias and splitting up the high map PMD and applying the same attributes there. This patch implements the same workaround for remapped first chunk.

Also, it's still unclear whether the remap allocator is worth the trouble. Andi thinks it would be better to simply use 4k mapping on NUMA machines as PMD TLB is much more precious resource than PTE TLB. To ease testing, this patch implements percpu_alloc kernel parameter which allows explicitly selecting which allocator to use.

This patchset contains the following four patches.

0001-x86-prepare-setup_pcpu_remap-for-pageattr-fix.patch 0002-x86-simplify-cpa_process_alias.patch 0003-x86-fix-pageattr-handling-for-remap-percpu-allocato.patch 0004-x86-implement-percpu_alloc-kernel-parameter.patch

0001-0002 preps for pageattr fix. 0003 fixes the pageattr bug. 0004 implements percpu_alloc kernel parameter.

This patchset is on top of the current linux-2.6#master (210af919c949a7d6bd330916ef376cec2907d81e) and contains the following changes.

 Documentation/kernel-parameters.txt |    6 +
 arch/x86/include/asm/percpu.h       |    9 +
 arch/x86/kernel/setup_percpu.c      |  186 ++++++++++++++++++++++++++----------
 arch/x86/mm/pageattr.c              |   59 +++++++----
 mm/percpu.c                         |   13 +-
 5 files changed, 197 insertions(+), 76 deletions(-)

Thanks.

--
tejun

author maintainer
#1 lore raw

PATCH 1/4 x86: prepare setup_pcpu_remap() for pageattr fix

Make the following changes in preparation of coming pageattr updates.

* Define and use array of struct pcpur_ent instead of array of pointers. The only difference is ->cpu field which is set but unused yet.

* Rename variables according to the above change.

* Rename local variable vm to pcpur_vm and move it out of the function.

[ Impact: no functional difference ]

Cc Jan Beulich <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 arch/x86/kernel/setup_percpu.c |   58 ++++++++++++++++++++++-----------------
 1 files changed, 33 insertions(+), 25 deletions(-)
arch/x86/kernel/setup_percpu.c +33 -25
index 3a97a4c..c17059c 100644
@@ -137,8 +137,14 @@ static void * __init pcpu_alloc_bootmem(unsigned int cpu, unsigned long size,
  * better than only using 4k mappings while still being NUMA friendly.
  */
 #ifdef CONFIG_NEED_MULTIPLE_NODES
+struct pcpur_ent {
+	unsigned int	cpu;
+	void		*ptr;
+};
+
 static size_t pcpur_size __initdata;
-static void **pcpur_ptrs __initdata;
+static struct pcpur_ent *pcpur_map __initdata;
+static struct vm_struct pcpur_vm;
 
 static struct page * __init pcpur_get_page(unsigned int cpu, int pageno)
 {
@@ -147,13 +153,12 @@ static struct page * __init pcpur_get_page(unsigned int cpu, int pageno)
 	if (off >= pcpur_size)
 		return NULL;
 
-	return virt_to_page(pcpur_ptrs[cpu] + off);
+	return virt_to_page(pcpur_map[cpu].ptr + off);
 }
 
 static ssize_t __init setup_pcpu_remap(size_t static_size)
 {
-	static struct vm_struct vm;
-	size_t ptrs_size, dyn_size;
+	size_t map_size, dyn_size;
 	unsigned int cpu;
 	ssize_t ret;
 
@@ -178,12 +183,14 @@ static ssize_t __init setup_pcpu_remap(size_t static_size)
 	dyn_size = pcpur_size - static_size - PERCPU_FIRST_CHUNK_RESERVE;
 
 	/* allocate pointer array and alloc large pages */
-	ptrs_size = PFN_ALIGN(num_possible_cpus() * sizeof(pcpur_ptrs[0]));
-	pcpur_ptrs = alloc_bootmem(ptrs_size);
+	map_size = PFN_ALIGN(num_possible_cpus() * sizeof(pcpur_map[0]));
+	pcpur_map = alloc_bootmem(map_size);
 
 	for_each_possible_cpu(cpu) {
-		pcpur_ptrs[cpu] = pcpu_alloc_bootmem(cpu, PMD_SIZE, PMD_SIZE);
-		if (!pcpur_ptrs[cpu])
+		pcpur_map[cpu].cpu = cpu;
+		pcpur_map[cpu].ptr = pcpu_alloc_bootmem(cpu, PMD_SIZE,
+							PMD_SIZE);
+		if (!pcpur_map[cpu].ptr)
 			goto enomem;
 
 		/*
@@ -194,42 +201,43 @@ static ssize_t __init setup_pcpu_remap(size_t static_size)
 		 * not well-specified to have a PAT-incompatible area
 		 * (unmapped RAM, device memory, etc.) in that hole.
 		 */
-		free_bootmem(__pa(pcpur_ptrs[cpu] + pcpur_size),
+		free_bootmem(__pa(pcpur_map[cpu].ptr + pcpur_size),
 			     PMD_SIZE - pcpur_size);
 
-		memcpy(pcpur_ptrs[cpu], __per_cpu_load, static_size);
+		memcpy(pcpur_map[cpu].ptr, __per_cpu_load, static_size);
 	}
 
 	/* allocate address and map */
-	vm.flags = VM_ALLOC;
-	vm.size = num_possible_cpus() * PMD_SIZE;
-	vm_area_register_early(&vm, PMD_SIZE);
+	pcpur_vm.flags = VM_ALLOC;
+	pcpur_vm.size = num_possible_cpus() * PMD_SIZE;
+	vm_area_register_early(&pcpur_vm, PMD_SIZE);
 
 	for_each_possible_cpu(cpu) {
-		pmd_t *pmd;
+		pmd_t *pmd, pmd_v;
 
-		pmd = populate_extra_pmd((unsigned long)vm.addr
-					 + cpu * PMD_SIZE);
-		set_pmd(pmd, pfn_pmd(page_to_pfn(virt_to_page(pcpur_ptrs[cpu])),
-				     PAGE_KERNEL_LARGE));
+		pmd = populate_extra_pmd((unsigned long)pcpur_vm.addr +
+					 cpu * PMD_SIZE);
+		pmd_v = pfn_pmd(page_to_pfn(virt_to_page(pcpur_map[cpu].ptr)),
+				PAGE_KERNEL_LARGE);
+		set_pmd(pmd, pmd_v);
 	}
 
 	/* we're ready, commit */
 	pr_info("PERCPU: Remapped at %p with large pages, static data "
-		"%zu bytes\n", vm.addr, static_size);
+		"%zu bytes\n", pcpur_vm.addr, static_size);
 
 	ret = pcpu_setup_first_chunk(pcpur_get_page, static_size,
 				     PERCPU_FIRST_CHUNK_RESERVE, dyn_size,
-				     PMD_SIZE, vm.addr, NULL);
-	goto out_free_ar;
+				     PMD_SIZE, pcpur_vm.addr, NULL);
+	goto out_free_map;
 
 enomem:
 	for_each_possible_cpu(cpu)
-		if (pcpur_ptrs[cpu])
-			free_bootmem(__pa(pcpur_ptrs[cpu]), PMD_SIZE);
+		if (pcpur_map[cpu].ptr)
+			free_bootmem(__pa(pcpur_map[cpu].ptr), PMD_SIZE);
 	ret = -ENOMEM;
-out_free_ar:
-	free_bootmem(__pa(pcpur_ptrs), ptrs_size);
+out_free_map:
+	free_bootmem(__pa(pcpur_map), map_size);
 	return ret;
 }
 #else
Signature · 1 line
1.6.0.2
author maintainer
#2 lore raw

PATCH 2/4 x86: simplify cpa_process_alias()

In reply to Tejun Heo

The two existing alias conditions in cpa_process_alias() are mutually exclusive and future ones are likely to be exclusive too. Simplify control flow to ease adding other alias cases.

The within(vaddr, (unsigned long)_text, _brk_end) test is removed as it's guaranteed to be false if vaddr is in the page mapped area.

[ Impact: cleanup, no functional difference ]

Cc Jan Beulich <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 arch/x86/mm/pageattr.c |   42 ++++++++++++++++++------------------------
 1 files changed, 18 insertions(+), 24 deletions(-)
arch/x86/mm/pageattr.c +18 -24
index 797f9f1..1097b61 100644
@@ -686,7 +686,6 @@ static int __change_page_attr_set_clr(struct cpa_data *cpa, int checkalias);
 static int cpa_process_alias(struct cpa_data *cpa)
 {
 	struct cpa_data alias_cpa;
-	int ret = 0;
 	unsigned long temp_cpa_vaddr, vaddr;
 
 	if (cpa->pfn >= max_pfn_mapped)
@@ -715,38 +714,33 @@ static int cpa_process_alias(struct cpa_data *cpa)
 		alias_cpa.vaddr = &temp_cpa_vaddr;
 		alias_cpa.flags &= ~(CPA_PAGES_ARRAY | CPA_ARRAY);
 
-
-		ret = __change_page_attr_set_clr(&alias_cpa, 0);
+		return __change_page_attr_set_clr(&alias_cpa, 0);
 	}
 
-#ifdef CONFIG_X86_64
-	if (ret)
-		return ret;
-	/*
-	 * No need to redo, when the primary call touched the high
-	 * mapping already:
-	 */
-	if (within(vaddr, (unsigned long) _text, _brk_end))
-		return 0;
+	/* vaddr is in page mapped area */
 
+#ifdef CONFIG_X86_64
 	/*
 	 * If the physical address is inside the kernel map, we need
 	 * to touch the high mapped kernel as well:
 	 */
-	if (!within(cpa->pfn, highmap_start_pfn(), highmap_end_pfn()))
-		return 0;
-
-	alias_cpa = *cpa;
-	temp_cpa_vaddr = (cpa->pfn << PAGE_SHIFT) + __START_KERNEL_map - phys_base;
-	alias_cpa.vaddr = &temp_cpa_vaddr;
-	alias_cpa.flags &= ~(CPA_PAGES_ARRAY | CPA_ARRAY);
+	if (within(cpa->pfn, highmap_start_pfn(), highmap_end_pfn())) {
+		alias_cpa = *cpa;
+		temp_cpa_vaddr = (cpa->pfn << PAGE_SHIFT) +
+				 __START_KERNEL_map - phys_base;
+		alias_cpa.vaddr = &temp_cpa_vaddr;
+		alias_cpa.flags &= ~(CPA_PAGES_ARRAY | CPA_ARRAY);
 
-	/*
-	 * The high mapping range is imprecise, so ignore the return value.
-	 */
-	__change_page_attr_set_clr(&alias_cpa, 0);
+		/*
+		 * The high mapping range is imprecise, so ignore the
+		 * return value.
+		 */
+		__change_page_attr_set_clr(&alias_cpa, 0);
+		return 0;
+	}
 #endif
-	return ret;
+
+	return 0;
 }
 
 static int __change_page_attr_set_clr(struct cpa_data *cpa, int checkalias)
Signature · 1 line
1.6.0.2
#3 lore raw

Re: [PATCH 2/4] x86: simplify cpa_process_alias()

Quoted 7 lines
Quoted 1 line
Quoted 1 line

Tejun Heo <[email protected]> 14.05.09 14:49 >>>

The two existing alias conditions in cpa_process_alias() are mutually exclusive and future ones are likely to be exclusive too. Simplify control flow to ease adding other alias cases.

The within(vaddr, (unsigned long)_text, _brk_end) test is removed as it's guaranteed to be false if vaddr is in the page mapped area.

I don't think that's correct - just consider the case where the originally passed in virtual address is from the vmalloc area: In that case, both the 1:1 mapping *and* the kernel mapping need to be checked.

Jan

author maintainer
#4 lore raw

Re: [PATCH 2/4] x86: simplify cpa_process_alias()

Hello,

Quoted 5 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line

Tejun Heo <[email protected]> 14.05.09 14:49 >>>

The two existing alias conditions in cpa_process_alias() are mutually exclusive and future ones are likely to be exclusive too. Simplify control flow to ease adding other alias cases.

Quoted 7 lines
Quoted 2 lines

The within(vaddr, (unsigned long)_text, _brk_end) test is removed as it's guaranteed to be false if vaddr is in the page mapped area.

I don't think that's correct - just consider the case where the originally passed in virtual address is from the vmalloc area: In that case, both the 1:1 mapping *and* the kernel mapping need to be checked.

Yeap, you're right. I was reading the second condition in reversal. Will update and repost. Thanks for spotting it.

Signature · 1 line
tejun
#5 lore raw

PATCH 2/4 x86: reorganize cpa_process_alias()

Reorganize cpa_process_alias() so that new alias condition can be added easily.

Jan Beulich spotted problem in the original cleanup thread which incorrectly assumed the two existing conditions were mutially exclusive.

[ Impact: code reorganization ]

Cc Jan Beulich <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Ingo Molnar <[email protected]>
---
 arch/x86/mm/pageattr.c |   44 ++++++++++++++++++++------------------------
 1 files changed, 20 insertions(+), 24 deletions(-)
arch/x86/mm/pageattr.c +20 -24
index 797f9f1..8bc64b0 100644
@@ -686,8 +686,8 @@ static int __change_page_attr_set_clr(struct cpa_data *cpa, int checkalias);
 static int cpa_process_alias(struct cpa_data *cpa)
 {
 	struct cpa_data alias_cpa;
-	int ret = 0;
 	unsigned long temp_cpa_vaddr, vaddr;
+	int ret;
 
 	if (cpa->pfn >= max_pfn_mapped)
 		return 0;
@@ -715,38 +715,34 @@ static int cpa_process_alias(struct cpa_data *cpa)
 		alias_cpa.vaddr = &temp_cpa_vaddr;
 		alias_cpa.flags &= ~(CPA_PAGES_ARRAY | CPA_ARRAY);
 
-
 		ret = __change_page_attr_set_clr(&alias_cpa, 0);
+		if (ret)
+			return ret;
 	}
 
 #ifdef CONFIG_X86_64
-	if (ret)
-		return ret;
 	/*
-	 * No need to redo, when the primary call touched the high
-	 * mapping already:
-	 */
-	if (within(vaddr, (unsigned long) _text, _brk_end))
-		return 0;
-
-	/*
-	 * If the physical address is inside the kernel map, we need
+	 * If the primary call didn't touch the high mapping already
+	 * and the physical address is inside the kernel map, we need
 	 * to touch the high mapped kernel as well:
 	 */
-	if (!within(cpa->pfn, highmap_start_pfn(), highmap_end_pfn()))
-		return 0;
-
-	alias_cpa = *cpa;
-	temp_cpa_vaddr = (cpa->pfn << PAGE_SHIFT) + __START_KERNEL_map - phys_base;
-	alias_cpa.vaddr = &temp_cpa_vaddr;
-	alias_cpa.flags &= ~(CPA_PAGES_ARRAY | CPA_ARRAY);
+	if (!within(vaddr, (unsigned long)_text, _brk_end) &&
+	    within(cpa->pfn, highmap_start_pfn(), highmap_end_pfn())) {
+		alias_cpa = *cpa;
+		temp_cpa_vaddr = (cpa->pfn << PAGE_SHIFT) +
+				 __START_KERNEL_map - phys_base;
+		alias_cpa.vaddr = &temp_cpa_vaddr;
+		alias_cpa.flags &= ~(CPA_PAGES_ARRAY | CPA_ARRAY);
 
-	/*
-	 * The high mapping range is imprecise, so ignore the return value.
-	 */
-	__change_page_attr_set_clr(&alias_cpa, 0);
+		/*
+		 * The high mapping range is imprecise, so ignore the
+		 * return value.
+		 */
+		__change_page_attr_set_clr(&alias_cpa, 0);
+	}
 #endif
-	return ret;
+
+	return 0;
 }
 
 static int __change_page_attr_set_clr(struct cpa_data *cpa, int checkalias)
Signature · 1 line
1.6.0.2
author maintainer
#6 lore raw

PATCH 3/4 x86: fix pageattr handling for remap percpu allocator

In reply to Tejun Heo

Remap allocator aliases a PMD page for each cpu and returns whatever is unused to the page allocator. When the pageattr of the recycled pages are changed, this makes the two aliases point to the overlapping regions with different attributes which isn't allowed and known to cause subtle data corruption in certain cases.

This can be handled in simliar manner to the x86_64 highmap alias. pageattr code should detect if the target pages have PMD alias and split the PMD alias and synchronize the attributes.

pcpur allocator is updated to keep the allocated PMD pages map sorted in ascending address order and provide pcpu_pmd_remapped() function which binary searches the array to determine whether the given address is aliased and if so to which address. pageattr is updated to use pcpu_pmd_remapped() to detect the PMD alias and split it up as necessary from cpa_process_alias().

This problem has been spotted by Jan Beulich.

[ Impact: fix subtle pageattr bug ]

---
 arch/x86/include/asm/percpu.h  |    9 +++++
 arch/x86/kernel/setup_percpu.c |   68 ++++++++++++++++++++++++++++++++++++---
 arch/x86/mm/pageattr.c         |   21 ++++++++++++
 3 files changed, 92 insertions(+), 6 deletions(-)
arch/x86/include/asm/percpu.h +9 -0
index aee103b..cad3531 100644
@@ -155,6 +155,15 @@ do {							\
 /* We can use this directly for local CPU (faster). */
 DECLARE_PER_CPU(unsigned long, this_cpu_off);
 
+#ifdef CONFIG_NEED_MULTIPLE_NODES
+void *pcpu_pmd_remapped(void *kaddr);
+#else
+static inline void *pcpu_pmd_remapped(void *kaddr)
+{
+	return NULL;
+}
+#endif
+
 #endif /* !__ASSEMBLY__ */
 
 #ifdef CONFIG_SMP
arch/x86/kernel/setup_percpu.c +62 -6

92 lines not on this page · raw message

arch/x86/mm/pageattr.c +21 -0
index 1097b61..a3d860b 100644
@@ -11,6 +11,7 @@
 #include <linux/interrupt.h>
 #include <linux/seq_file.h>
 #include <linux/debugfs.h>
+#include <linux/pfn.h>
 
 #include <asm/e820.h>
 #include <asm/processor.h>
@@ -687,6 +688,7 @@ static int cpa_process_alias(struct cpa_data *cpa)
 {
 	struct cpa_data alias_cpa;
 	unsigned long temp_cpa_vaddr, vaddr;
+	void *remapped;
 
 	if (cpa->pfn >= max_pfn_mapped)
 		return 0;
@@ -740,6 +742,25 @@ static int cpa_process_alias(struct cpa_data *cpa)
 	}
 #endif
 
+	/*
+	 * If the PMD page was partially used for per-cpu remapping,
+	 * the remapped area needs to be split and modified.  Note
+	 * that the partial recycling only happens at the tail of a
+	 * partially used PMD page, so touching single PMD page is
+	 * always enough.
+	 */
+	remapped = pcpu_pmd_remapped((void *)vaddr);
+	if (remapped) {
+		int max_pages = PFN_DOWN(PMD_SIZE - (vaddr & ~PMD_MASK));
+
+		alias_cpa = *cpa;
+		temp_cpa_vaddr = (unsigned long)remapped;
+		alias_cpa.vaddr = &temp_cpa_vaddr;
+		alias_cpa.numpages = min(alias_cpa.numpages, max_pages);
+		alias_cpa.flags &= ~(CPA_PAGES_ARRAY | CPA_ARRAY);
+		return __change_page_attr_set_clr(&alias_cpa, 0);
+	}
+
 	return 0;
 }
 
Signature · 1 line
1.6.0.2
author maintainer
#7 lore raw

Re: [PATCH UPDATED 3/4] x86: fix pageattr handling for remap percpu allocator

Remap allocator aliases a PMD page for each cpu and returns whatever is unused to the page allocator. When the pageattr of the recycled pages are changed, this makes the two aliases point to the overlapping regions with different attributes which isn't allowed and known to cause subtle data corruption in certain cases.

This can be handled in simliar manner to the x86_64 highmap alias. pageattr code should detect if the target pages have PMD alias and split the PMD alias and synchronize the attributes.

pcpur allocator is updated to keep the allocated PMD pages map sorted in ascending address order and provide pcpu_pmd_remapped() function which binary searches the array to determine whether the given address is aliased and if so to which address. pageattr is updated to use pcpu_pmd_remapped() to detect the PMD alias and split it up as necessary from cpa_process_alias().

This problem has been spotted by Jan Beulich.

[ Impact: fix subtle pageattr bug ]

---
 arch/x86/include/asm/percpu.h  |    9 +++++
 arch/x86/kernel/setup_percpu.c |   68 ++++++++++++++++++++++++++++++++++++---
 arch/x86/mm/pageattr.c         |   23 +++++++++++++
 3 files changed, 94 insertions(+), 6 deletions(-)
arch/x86/include/asm/percpu.h +9 -0
index aee103b..cad3531 100644
@@ -155,6 +155,15 @@ do {							\
 /* We can use this directly for local CPU (faster). */
 DECLARE_PER_CPU(unsigned long, this_cpu_off);
 
+#ifdef CONFIG_NEED_MULTIPLE_NODES
+void *pcpu_pmd_remapped(void *kaddr);
+#else
+static inline void *pcpu_pmd_remapped(void *kaddr)
+{
+	return NULL;
+}
+#endif
+
 #endif /* !__ASSEMBLY__ */
 
 #ifdef CONFIG_SMP
arch/x86/kernel/setup_percpu.c +62 -6

92 lines not on this page · raw message

arch/x86/mm/pageattr.c +23 -0
index 8bc64b0..a1bcead 100644
@@ -11,6 +11,7 @@
 #include <linux/interrupt.h>
 #include <linux/seq_file.h>
 #include <linux/debugfs.h>
+#include <linux/pfn.h>
 
 #include <asm/e820.h>
 #include <asm/processor.h>
@@ -687,6 +688,7 @@ static int cpa_process_alias(struct cpa_data *cpa)
 {
 	struct cpa_data alias_cpa;
 	unsigned long temp_cpa_vaddr, vaddr;
+	void *remapped;
 	int ret;
 
 	if (cpa->pfn >= max_pfn_mapped)
@@ -742,6 +744,27 @@ static int cpa_process_alias(struct cpa_data *cpa)
 	}
 #endif
 
+	/*
+	 * If the PMD page was partially used for per-cpu remapping,
+	 * the remapped area needs to be split and modified.  Note
+	 * that the partial recycling only happens at the tail of a
+	 * partially used PMD page, so touching single PMD page is
+	 * always enough.
+	 */
+	remapped = pcpu_pmd_remapped((void *)vaddr);
+	if (remapped) {
+		int max_pages = PFN_DOWN(PMD_SIZE - (vaddr & ~PMD_MASK));
+
+		alias_cpa = *cpa;
+		temp_cpa_vaddr = (unsigned long)remapped;
+		alias_cpa.vaddr = &temp_cpa_vaddr;
+		alias_cpa.numpages = min(alias_cpa.numpages, max_pages);
+		alias_cpa.flags &= ~(CPA_PAGES_ARRAY | CPA_ARRAY);
+		ret = __change_page_attr_set_clr(&alias_cpa, 0);
+		if (ret)
+			return ret;
+	}
+
 	return 0;
 }
 
Signature · 1 line
1.6.0.2
author maintainer
#8 lore raw

PATCH 4/4 x86: implement percpu_alloc kernel parameter

In reply to Tejun Heo

According to Andi, it isn't clear whether remap allocator is worth the trouble as there are many processors where PMD TLB is far scarcer than PTE TLB. The advantage or disadvantage probably depends on the actual size of percpu area and specific processor. As performance degradation due to TLB pressure tends to be highly workload specific and subtle, it is difficult to decide which way to go without more data.

This patch implements percpu_alloc kernel parameter to allow selecting which first chunk allocator to use to ease debugging and testing.

While at it, make sure all the failure paths report why something failed to help determining why certain allocator isn't working. Also, kill the "Great future plan" comment which had already been realized quite some time ago.

[ Impact: allow explicit percpu first chunk allocator selection ]

---
 Documentation/kernel-parameters.txt |    6 +++
 arch/x86/kernel/setup_percpu.c      |   68 +++++++++++++++++++++++------------
 mm/percpu.c                         |   13 +++++--
 3 files changed, 60 insertions(+), 27 deletions(-)
Documentation/kernel-parameters.txt +6 -0
index e87bdbf..929bb3a 100644
@@ -1837,6 +1837,12 @@ and is between 256 and 4096 characters. It is defined in the file
 			Format: { 0 | 1 }
 			See arch/parisc/kernel/pdc_chassis.c
 
+	percpu_alloc=	[X86] Select which percpu first chunk allocator to use.
+			Allowed values are one of "remap", "embed" and "4k".
+			See comments in arch/x86/kernel/setup_percpu.c for
+			details on each allocator.  This parameter is primarily
+			for debugging and performance comparison.
+
 	pf.		[PARIDE]
 			See Documentation/blockdev/paride.txt.
 
arch/x86/kernel/setup_percpu.c +45 -23

113 lines not on this page · raw message

mm/percpu.c +9 -4
index 1aa5d8f..d42f2ce 100644
@@ -1287,6 +1287,7 @@ static struct page * __init pcpue_get_page(unsigned int cpu, int pageno)
 ssize_t __init pcpu_embed_first_chunk(size_t static_size, size_t reserved_size,
 				      ssize_t dyn_size, ssize_t unit_size)
 {
+	size_t chunk_size;
 	unsigned int cpu;
 
 	/* determine parameters and allocate */
@@ -1301,11 +1302,15 @@ ssize_t __init pcpu_embed_first_chunk(size_t static_size, size_t reserved_size,
 	} else
 		pcpue_unit_size = max_t(size_t, pcpue_size, PCPU_MIN_UNIT_SIZE);
 
-	pcpue_ptr = __alloc_bootmem_nopanic(
-					num_possible_cpus() * pcpue_unit_size,
-					PAGE_SIZE, __pa(MAX_DMA_ADDRESS));
-	if (!pcpue_ptr)
+	chunk_size = pcpue_unit_size * num_possible_cpus();
+
+	pcpue_ptr = __alloc_bootmem_nopanic(chunk_size, PAGE_SIZE,
+					    __pa(MAX_DMA_ADDRESS));
+	if (!pcpue_ptr) {
+		pr_warning("PERCPU: failed to allocate %zu bytes for "
+			   "embedding\n", chunk_size);
 		return -ENOMEM;
+	}
 
 	/* return the leftover and copy */
 	for_each_possible_cpu(cpu) {
Signature · 1 line
1.6.0.2
#9 lore raw
In reply to Tejun Heo
Quoted 9 lines
Quoted 1 line
Quoted 1 line

Tejun Heo <[email protected]> 14.05.09 14:49 >>>

The remap allocator allocates a PMD page per cpu, returns whatever is unnecessary to the page allocator and remaps the PMD page into vmalloc area to construct the first percpu chunk. This is to take advantage of large page mapping. However this creates active aliases for the recycled pages. When some user allocates the recycled pages and tries to change pageattr on it, remapped PMD alias might end up having different attributes with the regular page mapped addresses which can lead to subtle data corruption according to Andi.

In order to reduce the amount of work to do during lookup as well as the chance of having a collision at all, wouldn't it be reasonable to use as much of an allocated 2/4M page as possible rather than returning whatever is left after a single CPU got its per-CPU memory chunk from it? I.e. you'd return only those (few) pages that either don't fit another CPU's chunk anymore or that are left after running through all CPUs.

Or is there some hidden requirement that each CPU's per-CPU area must start on a PMD boundary?

This would additionally address a potential problem on 32-bits - currently, for a 32-CPU system you consume half of the vmalloc space with PAE (on non-PAE you'd even exhaust it, but I think it's unreasonable to expect a system having 32 CPUs to not need PAE).

Jan

author maintainer
#10 lore raw

Hello, Jan.

Quoted 10 lines · Jan Beulich

In order to reduce the amount of work to do during lookup as well as the chance of having a collision at all, wouldn't it be reasonable to use as much of an allocated 2/4M page as possible rather than returning whatever is left after a single CPU got its per-CPU memory chunk from it? I.e. you'd return only those (few) pages that either don't fit another CPU's chunk anymore or that are left after running through all CPUs.

Or is there some hidden requirement that each CPU's per-CPU area must start on a PMD boundary?

The whole point of doing the remapping is giving each CPU its own PMD mapping for perpcu area, so, yeah, that's the requirement. I don't think the requirement is hidden tho.

How hot is the cpa path? On my test systems, there were only few calls during init and then nothing. Does it become very hot if, for example, GEM is used? But I really don't think the log2 binary search overhead would be anything noticeable compared to TLB shootdown and all other stuff going on there.

Quoted 4 lines

This would additionally address a potential problem on 32-bits - currently, for a 32-CPU system you consume half of the vmalloc space with PAE (on non-PAE you'd even exhaust it, but I think it's unreasonable to expect a system having 32 CPUs to not need PAE).

I recall having about the same conversation before. Looking up...

-- QUOTE --
Actually, I've been looking at the numbers and I'm not sure if the concern is valid. On x86_32, the practical number of maximum processors would be around 16 so it will end up 32M, which isn't nice and it would probably a good idea to introduce a parameter to select which allocator to use but still it's far from consuming all the VM area. On x86_64, the vmalloc area is obscenely large at 245, ie 32 terabytes. Even with 4096 processors, single chunk is measly 0.02%.

If it's a problem for other archs or extreme x86_32 configurations, we can add some safety measures but in general I don't think it is a problem.
-- END OF QUOTE --

So, yeah, if there are 32bit 32-way NUMA machines out there, it would be wise to skip remap allocator on such machines. Maybe we can implement a heuristic - something like "if vm area consumption goes over 25%, don't use remap".

Thanks.

Signature · 1 line
tejun
#11 lore raw
Quoted 9 lines
Quoted 9 lines
Quoted 1 line

Tejun Heo <[email protected]> 14.05.09 17:55 >>>

In order to reduce the amount of work to do during lookup as well as the chance of having a collision at all, wouldn't it be reasonable to use as much of an allocated 2/4M page as possible rather than returning whatever is left after a single CPU got its per-CPU memory chunk from it? I.e. you'd return only those (few) pages that either don't fit another CPU's chunk anymore or that are left after running through all CPUs.

Quoted 6 lines
Quoted 2 lines

Or is there some hidden requirement that each CPU's per-CPU area must start on a PMD boundary?

The whole point of doing the remapping is giving each CPU its own PMD mapping for perpcu area, so, yeah, that's the requirement. I don't think the requirement is hidden tho.

No, from looking at the code the requirement seems to only be that you get memory allocated from the correct node and mapped by a large page. There's nothing said why the final virtual address would need to be large page aligned. I.e., with a slight modification to take the NUMA requirement into account (I noticed I ignored that aspect after I had already sent that mail), the previous suggestion would still appear usable to me.

Quoted 5 lines

How hot is the cpa path? On my test systems, there were only few calls during init and then nothing. Does it become very hot if, for example, GEM is used? But I really don't think the log2 binary search overhead would be anything noticeable compared to TLB shootdown and all other stuff going on there.

I would view cutting down on that only as a nice side effect, not a primary reason to do the change. The primary reason is this:

Quoted 7 lines
Quoted 4 lines

This would additionally address a potential problem on 32-bits - currently, for a 32-CPU system you consume half of the vmalloc space with PAE (on non-PAE you'd even exhaust it, but I think it's unreasonable to expect a system having 32 CPUs to not need PAE).

I recall having about the same conversation before. Looking up...

Quoted 9 lines

-- QUOTE --
Actually, I've been looking at the numbers and I'm not sure if the concern is valid. On x86_32, the practical number of maximum processors would be around 16 so it will end up 32M, which isn't nice and it would probably a good idea to introduce a parameter to select which allocator to use but still it's far from consuming all the VM area. On x86_64, the vmalloc area is obscenely large at 245, ie 32 terabytes. Even with 4096 processors, single chunk is measly 0.02%.

Just to note - there must be a reason we (SuSE/Novell) build our default 32-bit kernel with support for 128 CPUs, which now is simply broken.

Quoted 9 lines

If it's a problem for other archs or extreme x86_32 configurations, we can add some safety measures but in general I don't think it is a problem.
-- END OF QUOTE --

So, yeah, if there are 32bit 32-way NUMA machines out there, it would be wise to skip remap allocator on such machines. Maybe we can implement a heuristic - something like "if vm area consumption goes over 25%, don't use remap".

Possibly, as a secondary consideration on top of the suggested reduction of virtual address space consumption.

Jan

author maintainer
#12 lore raw

Hello,

Quoted 10 lines · Jan Beulich
Quoted 3 lines

The whole point of doing the remapping is giving each CPU its own PMD mapping for perpcu area, so, yeah, that's the requirement. I don't think the requirement is hidden tho.

No, from looking at the code the requirement seems to only be that you get memory allocated from the correct node and mapped by a large page. There's nothing said why the final virtual address would need to be large page aligned. I.e., with a slight modification to take the NUMA requirement into account (I noticed I ignored that aspect after I had already sent that mail), the previous suggestion would still appear usable to me.

The requirement is having separate PMD mapping per NUMA node. What has been implemented is the simplest form of that - one mapping per CPU. Sure it can be further improved with more knowledge of the topology. If you're interested, please go ahead.

Quoted 16 lines
Quoted 6 lines
Quoted 4 lines

This would additionally address a potential problem on 32-bits - currently, for a 32-CPU system you consume half of the vmalloc space with PAE (on non-PAE you'd even exhaust it, but I think it's unreasonable to expect a system having 32 CPUs to not need PAE).

I recall having about the same conversation before. Looking up...

Quoted 9 lines

-- QUOTE --
Actually, I've been looking at the numbers and I'm not sure if the concern is valid. On x86_32, the practical number of maximum processors would be around 16 so it will end up 32M, which isn't nice and it would probably a good idea to introduce a parameter to select which allocator to use but still it's far from consuming all the VM area. On x86_64, the vmalloc area is obscenely large at 245, ie 32 terabytes. Even with 4096 processors, single chunk is measly 0.02%.

Quoted 2 lines

Just to note - there must be a reason we (SuSE/Novell) build our default 32-bit kernel with support for 128 CPUs, which now is simply broken.

It's not broken, it will just fall back to 4k allocator. Also, please take a look at the refreshed patchset, remap allocator is not used anymore if it's gonna occupy more than 20% (random number from the top of my head) of vmalloc area.

Quoted 7 lines
Quoted 4 lines

So, yeah, if there are 32bit 32-way NUMA machines out there, it would be wise to skip remap allocator on such machines. Maybe we can implement a heuristic - something like "if vm area consumption goes over 25%, don't use remap".

Possibly, as a secondary consideration on top of the suggested reduction of virtual address space consumption.

Yeah, further improvements welcome. No objection whatsoever there.

Thanks.

Signature · 1 line
tejun
#13 lore raw
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines

Tejun Heo <[email protected]> 15.05.09 10:11 >>>

Quoted 4 lines

This would additionally address a potential problem on 32-bits - currently, for a 32-CPU system you consume half of the vmalloc space with PAE (on non-PAE you'd even exhaust it, but I think it's unreasonable to expect a system having 32 CPUs to not need PAE).

I recall having about the same conversation before. Looking up...

-- QUOTE --
Actually, I've been looking at the numbers and I'm not sure if the concern is valid. On x86_32, the practical number of maximum processors would be around 16 so it will end up 32M, which isn't nice and it would probably a good idea to introduce a parameter to select which allocator to use but still it's far from consuming all the VM area. On x86_64, the vmalloc area is obscenely large at 245, ie 32 terabytes. Even with 4096 processors, single chunk is measly 0.02%.

Quoted 4 lines
Quoted 2 lines

Just to note - there must be a reason we (SuSE/Novell) build our default 32-bit kernel with support for 128 CPUs, which now is simply broken.

It's not broken, it will just fall back to 4k allocator. Also, please

I'm afraid I have to disagree: There's no check (not even in vm_area_register_early()) whether the vmalloc area is actually large enough to fulfill the request.

Quoted 3 lines

take a look at the refreshed patchset, remap allocator is not used anymore if it's gonna occupy more than 20% (random number from the top of my head) of vmalloc area.

Yeah, I saw this only after following this thread.

Jan

author maintainer
#14 lore raw
Quoted 16 lines · Jan Beulich
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines

Tejun Heo <[email protected]> 15.05.09 10:11 >>>

Quoted 4 lines

This would additionally address a potential problem on 32-bits - currently, for a 32-CPU system you consume half of the vmalloc space with PAE (on non-PAE you'd even exhaust it, but I think it's unreasonable to expect a system having 32 CPUs to not need PAE).

I recall having about the same conversation before. Looking up...

-- QUOTE --
Actually, I've been looking at the numbers and I'm not sure if the concern is valid. On x86_32, the practical number of maximum processors would be around 16 so it will end up 32M, which isn't nice and it would probably a good idea to introduce a parameter to select which allocator to use but still it's far from consuming all the VM area. On x86_64, the vmalloc area is obscenely large at 245, ie 32 terabytes. Even with 4096 processors, single chunk is measly 0.02%.

Quoted 7 lines
Quoted 3 lines
Quoted 2 lines

Just to note - there must be a reason we (SuSE/Novell) build our default 32-bit kernel with support for 128 CPUs, which now is simply broken.

It's not broken, it will just fall back to 4k allocator. Also, please

I'm afraid I have to disagree: There's no check (not even in vm_area_register_early()) whether the vmalloc area is actually large enough to fulfill the request.

Hah... indeed. Well, it's solved now.

Thanks.

Signature · 1 line
tejun
#15 lore raw
In reply to Tejun Heo
Quoted 5 lines · Tejun Heo

Hello,

Upon ack, please pull from the following git tree.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git x86-percpu-pageattr

git tree updated to contain two updated patches. The new commit ID is 42fec16c045bc5259002faacdcd23e4530b6482e.

Thanks.

Signature · 1 line
tejun
#16 lore raw
Quoted 9 lines · Tejun Heo
Quoted 5 lines · Tejun Heo

Hello,

Upon ack, please pull from the following git tree.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git x86-percpu-pageattr

git tree updated to contain two updated patches. The new commit ID is 42fec16c045bc5259002faacdcd23e4530b6482e.

Will soon post an updated patchset. Please wait a bit.

Thanks.

Signature · 1 line
tejun
#17 lore raw

You might want to drop linux-kernel-owner from the CC:, I already get enough copies :-)

#18 lore raw
Quoted 2 lines · David Miller

You might want to drop linux-kernel-owner from the CC:, I already get enough copies :-)

Hehe.. Where did that come from? Sorry about that. :-)

Signature · 1 line
tejun
#19 lore raw
In reply to Tejun Heo
Quoted 10 lines · Tejun Heo

Hello,

Upon ack, please pull from the following git tree.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git x86-percpu-pageattr

This patchset fixes subtile bug in pageattr handling when remap percpu first chunk allocator is in use and implements percpu_alloc kernel parameter so that allocator can be selected from boot prompt.

Quoted 6 lines

This problem was spotted by Jan Beulich.

The remap allocator allocates a PMD page per cpu, returns whatever is unnecessary to the page allocator and remaps the PMD page into vmalloc area to construct the first percpu chunk. This is to take advantage of large page mapping.

Tejun, Can you please educate me why we need to map this first percpu chunk (which is pre-allocated during boot and is physically contiguous) into vmalloc area? Perhaps even for the other dynamically allocated secondary chunks? (as far as I can see, all the chunk allocations seems to be physically contiguous and later mapped into vmalloc area)..

That should simplify these things quite a bit(atleast for first percpu chunk). I am missing something obvious I guess.

thanks,
suresh

author maintainer
#20 lore raw

Hello, Suresh.

Quoted 3 lines · Suresh Siddha

Tejun, Can you please educate me why we need to map this first percpu chunk (which is pre-allocated during boot and is physically contiguous) into vmalloc area?

To make areas for each cpu congruent such that the address offset of a percpu symbol for CPU N is always the same from the address for CPU 0.

Quoted 6 lines

Perhaps even for the other dynamically allocated secondary chunks? (as far as I can see, all the chunk allocations seems to be physically contiguous and later mapped into vmalloc area)..

That should simplify these things quite a bit(atleast for first percpu chunk). I am missing something obvious I guess.

Hmm... Sorry I don't really follow. Can you please elaborate the question?

thanks.

Signature · 1 line
tejun
#21 lore raw
Quoted 9 lines · Tejun Heo

Hello, Suresh.

Quoted 3 lines · Suresh Siddha

Tejun, Can you please educate me why we need to map this first percpu chunk (which is pre-allocated during boot and is physically contiguous) into vmalloc area?

To make areas for each cpu congruent such that the address offset of a percpu symbol for CPU N is always the same from the address for CPU 0.

But for the first percpu chunk, isn't it the case that the physical address allocations for a particular cpu is contiguous (as you are using one bootmem allocation for whole PMD_SIZE for any given cpu)? So both the kernel direct mapping aswell as the vmalloc mappings are contiguous for the first chunk, on any given cpu. Right?

Quoted 9 lines
Quoted 6 lines

Perhaps even for the other dynamically allocated secondary chunks? (as far as I can see, all the chunk allocations seems to be physically contiguous and later mapped into vmalloc area)..

That should simplify these things quite a bit(atleast for first percpu chunk). I am missing something obvious I guess.

Hmm... Sorry I don't really follow. Can you please elaborate the question?

For the first percpu chunk, we can use the kernel direct mapping and avoid the vmalloc mapping of PMD_SIZE. And avoid the vmap address aliasing problem (wrt to free pages that we have given back to -mm) that we are trying to avoid with this patchset (as the existing cpa code already takes care of the kernel direct mappings).

thanks,
suresh

author maintainer
#22 lore raw

Hello, Suresh.

Quoted 7 lines · Suresh Siddha
Quoted 6 lines · Tejun Heo

Hello, Suresh.

Quoted 3 lines · Suresh Siddha

Tejun, Can you please educate me why we need to map this first percpu chunk (which is pre-allocated during boot and is physically contiguous) into vmalloc area?

Quoted 8 lines
Quoted 2 lines

To make areas for each cpu congruent such that the address offset of a percpu symbol for CPU N is always the same from the address for CPU 0.

But for the first percpu chunk, isn't it the case that the physical address allocations for a particular cpu is contiguous (as you are using one bootmem allocation for whole PMD_SIZE for any given cpu)? So both the kernel direct mapping aswell as the vmalloc mappings are contiguous for the first chunk, on any given cpu. Right?

Hmmm... okay. Percpu areas are composed of multiple chunks. A single chunk is composed of multiple units, one unit for each CPU. Units in a single chunk should be contiguous and of the same size such that unit_addr_for_cpu_N == chunk_addr + N * unit_size whereas chunks don't need to have any special address relation to other chunks. Combined, this results in percpu addresses for CPU N are always offset by N * unit_size from the percpu adresses for CPU 0 which can be efficiently determined using some extra resource in the processor (segment register on x86 for example).

For remap first chunk allocator, each unit for each CPU is allocated separately using the bootmem allocator. Each unit is continuous but they still need to be assembed into a single contiguous area to be used as the first chunk, which is where the remapping comes in. So, the extra requirement is that units in the same chunk need to be contiguous and NUMA allocation means units will be spread according to NUMA configuration, so they need to be put together by remapping them.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Perhaps even for the other dynamically allocated secondary chunks? (as far as I can see, all the chunk allocations seems to be physically contiguous and later mapped into vmalloc area)..

That should simplify these things quite a bit(atleast for first percpu chunk). I am missing something obvious I guess.

Quoted 8 lines
Quoted 2 lines

Hmm... Sorry I don't really follow. Can you please elaborate the question?

For the first percpu chunk, we can use the kernel direct mapping and avoid the vmalloc mapping of PMD_SIZE. And avoid the vmap address aliasing problem (wrt to free pages that we have given back to -mm) that we are trying to avoid with this patchset (as the existing cpa code already takes care of the kernel direct mappings).

Hmmm.... If you can show me how to use the linear mapping directly, I'll be happy as a clam.

Thanks.

Signature · 1 line
tejun
#23 lore raw
Quoted 4 lines · Tejun Heo

Units in
a single chunk should be contiguous and of the same size such that unit_addr_for_cpu_N == chunk_addr + N * unit_size whereas chunks don't need to have any special address relation to other chunks.

And the size of the unit is same across chunks.

This is what I was missing :)

Can we don't use PERCPU_DYNAMIC_RESERVE in the first chunk and for dynamic per_cpu_ptr's we can use some other offset such as per_cpu_ptr_offset() or some such thing?

Then we can separate the static and dynamic chunks. And use large page kernel-direct mappings for fast access for critical common things and use small page accesses for dynamic and not so common accesses.

Just checking to see if we can reduce the complexity of setting up the percpu areas (different versions for embed, non-embed etc) and handling all these aliases with simple code, rather than making it complex.

thanks,
suresh

#24 lore raw
Quoted 5 lines · Suresh Siddha

Can we don't use PERCPU_DYNAMIC_RESERVE in the first chunk and for dynamic per_cpu_ptr's we can use some other offset such as per_cpu_ptr_offset() or some such thing?

Quoted 8 lines

Then we can separate the static and dynamic chunks. And use large page kernel-direct mappings for fast access for critical common things and use small page accesses for dynamic and not so common accesses.

Just checking to see if we can reduce the complexity of setting up the percpu areas (different versions for embed, non-embed etc) and handling all these aliases with simple code, rather than making it complex.

I'm confused what you're suggesting here. The whole point of the percpu unification work is that we can use %gs:absolute type references to hit a variable right away. Although in theory we could use both %fs and %gs for pointers, setting up both would greatly increase the cost of entering the kernel, especially on 64 bits.

This means all percpu data has to have a constant (virtual) offset from the beginning of the static percpu area.

-hpa

#25 lore raw
Quoted 15 lines · H. Peter Anvin
Quoted 5 lines · Suresh Siddha

Can we don't use PERCPU_DYNAMIC_RESERVE in the first chunk and for dynamic per_cpu_ptr's we can use some other offset such as per_cpu_ptr_offset() or some such thing?

Quoted 8 lines

Then we can separate the static and dynamic chunks. And use large page kernel-direct mappings for fast access for critical common things and use small page accesses for dynamic and not so common accesses.

Just checking to see if we can reduce the complexity of setting up the percpu areas (different versions for embed, non-embed etc) and handling all these aliases with simple code, rather than making it complex.

Quoted 8 lines

I'm confused what you're suggesting here. The whole point of the percpu unification work is that we can use %gs:absolute type references to hit a variable right away. Although in theory we could use both %fs and %gs for pointers, setting up both would greatly increase the cost of entering the kernel, especially on 64 bits.

This means all percpu data has to have a constant (virtual) offset from the beginning of the static percpu area.

This %gs:absolute type accesses are for static percpu data.

But what I was referring to is the dynamic percpu data(accessed through per_cpu_ptr()). Instead of combining some part of the dynamic percpu data into the static percpu data(first percpu chunk), we can use different chunks for dynamic percpu data and governed by a different per_cpu_dynamic_offset[NR_CPUS] array.

Then we can use large-page kernel direct mappings for static percpu data (or %gs:offset) and small-page vmalloc mappings for dynamic percpu data.

thanks,
suresh

author maintainer
#26 lore raw

Hello, Suresh.

Quoted 10 lines · Suresh Siddha

This %gs:absolute type accesses are for static percpu data.

But what I was referring to is the dynamic percpu data(accessed through per_cpu_ptr()). Instead of combining some part of the dynamic percpu data into the static percpu data(first percpu chunk), we can use different chunks for dynamic percpu data and governed by a different per_cpu_dynamic_offset[NR_CPUS] array.

Then we can use large-page kernel direct mappings for static percpu data (or %gs:offset) and small-page vmalloc mappings for dynamic percpu data.

Hmmm... I can't really follow what you're suggesting. Can you please explain it in more detailed way?

Thanks.

Signature · 1 line
tejun
#27 lore raw
Quoted 11 lines · Tejun Heo

Hello, Suresh.

Quoted 8 lines · Suresh Siddha

This %gs:absolute type accesses are for static percpu data.

But what I was referring to is the dynamic percpu data(accessed through per_cpu_ptr()). Instead of combining some part of the dynamic percpu data into the static percpu data(first percpu chunk), we can use different chunks for dynamic percpu data and governed by a different per_cpu_dynamic_offset[NR_CPUS] array.

Quoted 5 lines
Quoted 2 lines

Then we can use large-page kernel direct mappings for static percpu data (or %gs:offset) and small-page vmalloc mappings for dynamic percpu data.

Hmmm... I can't really follow what you're suggesting. Can you please explain it in more detailed way?

Ok. Before I make another attempt walking that hill :)

I was talking to Peter and it seems there are some requests to change the first percpu unit allocation for each possible cpu using bootmem allocator, to allocating the corresponding unit at the cpu online time.

Do you have plans to change this? If we do this allocation during the corresponding cpu online time and don't end up using big pages, then also we avoid all these aliasing issues...

thanks,
suresh

author maintainer
#28 lore raw

Hello,

Quoted 4 lines · Suresh Siddha
Quoted 2 lines

Hmmm... I can't really follow what you're suggesting. Can you please explain it in more detailed way?

Ok. Before I make another attempt walking that hill :)

Heh... the problem is that unless I understand what you're trying to achieve (and vice-versa), our discussion is likely to be riddled with confusions, and currently either you're misunderstanding the whole thing or I'm being slow (not too unusual :-). It would be nice to determine which way it is.

Quoted 5 lines

I was talking to Peter and it seems there are some requests to change the first percpu unit allocation for each possible cpu using bootmem allocator, to allocating the corresponding unit at the cpu online time.

Do you have plans to change this?

Yeap, once the remaining archs are converted, that's the next stop. With recently posted patchset, only three remain - sparc64, powerpc64 and sparc64. Davem is working on sparc64. I'm planning on doing powerpc64. ia64 is a bit tricky as it already remaps percpu areas but I'm sure we'll find a way around it. After that, yeap, the dynamic online thing.

Quoted 3 lines

If we do this allocation during the corresponding cpu online time and don't end up using big pages, then also we avoid all these aliasing issues...

The dynamic onlining will probably use 4k pages so, yeah, it won't have the alias issues but that's not the issue here, right? You can already avoid aliasing that way by simply using 4k allocator from the get-go.

Thanks.

Signature · 1 line
tejun
#29 lore raw
Quoted 5 lines · Tejun Heo

Heh... the problem is that unless I understand what you're trying to achieve (and vice-versa), our discussion is likely to be riddled with confusions, and currently either you're misunderstanding the whole thing or I'm being slow (not too unusual :-). It would be nice to determine which way it is.

At the start of this, I was trying to see a way where we can use large page mappings for percpu areas and at the same time, avoid the complexity of cpa() that we are adding in this patchset. So, I was going down the path of exploiting your bootmem allocation for the first percpu chunk (as kernel identity mappings are already taken care by cpa()).

Quoted 4 lines

The dynamic onlining will probably use 4k pages so, yeah, it won't have the alias issues but that's not the issue here, right? You can already avoid aliasing that way by simply using 4k allocator from the get-go.

But now that I learnt about dynamic online allocation, can we avoid the complexity brought by this patchset, by simply using 4k allocator from get-go.

i.e., can we drop this remap pageattr handling patchset and simply use 4k mapping for now? And move to dynamic allocation at a later point.

This will simplify quite a bit of code.

thanks,
suresh

author maintainer
#30 lore raw

Hello,

Quoted 8 lines · Suresh Siddha
Quoted 4 lines

The dynamic onlining will probably use 4k pages so, yeah, it won't have the alias issues but that's not the issue here, right? You can already avoid aliasing that way by simply using 4k allocator from the get-go.

But now that I learnt about dynamic online allocation, can we avoid the complexity brought by this patchset, by simply using 4k allocator from get-go.

Sure, we can.

Quoted 2 lines

i.e., can we drop this remap pageattr handling patchset and simply use 4k mapping for now? And move to dynamic allocation at a later point.

4k or not, x86 is already on dynamic allocation. The only difference is how the first chunk is allocated.

Quoted 1 line

This will simplify quite a bit of code.

Yes it will. The question is which way would be better. Till now, there hasn't been any actual data on how remap compares to 4k. The only thing we know is that, on UMA, embed should behave exactly the same for static percpu variables as before the whole dynamic allocator.

On NUMA, both remap and 4k add some level of TLB pressure. remap will waste one more PMD TLB entry (dup) while 4k adds a bunch of 4k ones (non-dup but what used to be accessed by PMD TLB is now accessed with PTE TLB). Some say using one more PMD TLB is better while others disagree. So, the best course of action here seems to offer both and easy way to select between them so that data can be gathered, which is what this patchset does.

I don't think the added complexity for cpa() justifies dropping remap without further testing. The added complexity isn't that big. Most of the confusion in this patchset came from my ignorance on the subject. cpa() is a fragile thing but we need it anyway, so...

Thanks.

Signature · 1 line
tejun
#31 lore raw
Quoted 4 lines · Tejun Heo

On NUMA, both remap and 4k add some level of TLB pressure. remap will waste one more PMD TLB entry (dup) while 4k adds a bunch of 4k ones (non-dup but what used to be accessed by PMD TLB is now accessed with PTE TLB).

Correction: 4k TLBs are dups too.

Thanks.

Signature · 1 line
tejun
#32 lore raw
In reply to Tejun Heo
Quoted 2 lines · Tejun Heo

Yes it will. The question is which way would be better. Till now, there hasn't been any actual data on how remap compares to 4k.

I am not sure if we see any measurable difference. Even if we use 4k entries, it will be few entries that kernel will be referring to frequently.

Quoted 7 lines

On NUMA, both remap and 4k add some level of TLB pressure. remap will waste one more PMD TLB entry (dup) while 4k adds a bunch of 4k ones (non-dup but what used to be accessed by PMD TLB is now accessed with PTE TLB). Some say using one more PMD TLB is better while others disagree. So, the best course of action here seems to offer both and easy way to select between them so that data can be gathered, which is what this patchset does.

So with the planned future change of percpu unit allocation during cpu online, you are planning to try large page allocation first and then fallback to 4k pages, if that doesn't succeed. And then populate new percpu ptr accordingly and then sort wrt to other cpu ptr's, so that we can keep aliases in sync for future(and in parallel) cpa()'s that might be happening.

There is nothing wrong with all this. Just the code complexity (and maintenance) for what we are trying to gain ;)

Quoted 8 lines

I don't think the added complexity for cpa() justifies dropping remap without further testing. The added complexity isn't that big. Most of the confusion in this patchset came from my ignorance on the subject. cpa() is a fragile thing but we need it anyway, so...

Thanks.

author maintainer
#33 lore raw

Hello, Suresh.

Quoted 7 lines · Suresh Siddha
Quoted 2 lines · Tejun Heo

Yes it will. The question is which way would be better. Till now, there hasn't been any actual data on how remap compares to 4k.

I am not sure if we see any measurable difference. Even if we use 4k entries, it will be few entries that kernel will be referring to frequently.

Yeah, I hope so too but I *really* want to see some numbers before taking further actions. Remap is chosen as the default because it deviates less from the original behavior but I'll be excited to drop it if 4k works just fine.

Quoted 15 lines
Quoted 7 lines

On NUMA, both remap and 4k add some level of TLB pressure. remap will waste one more PMD TLB entry (dup) while 4k adds a bunch of 4k ones (non-dup but what used to be accessed by PMD TLB is now accessed with PTE TLB). Some say using one more PMD TLB is better while others disagree. So, the best course of action here seems to offer both and easy way to select between them so that data can be gathered, which is what this patchset does.

So with the planned future change of percpu unit allocation during cpu online, you are planning to try large page allocation first and then fallback to 4k pages, if that doesn't succeed. And then populate new percpu ptr accordingly and then sort wrt to other cpu ptr's, so that we can keep aliases in sync for future(and in parallel) cpa()'s that might be happening.

Quoted 2 lines

There is nothing wrong with all this. Just the code complexity (and maintenance) for what we are trying to gain ;)

No, I'll let the first chunk allocation happen the same way for cpus available on boot and then just do 4k allocations for whatever necessary afterward. The needed code change in percpu proper isn't that big. What would take more effort is auditing all percpu users.

Thanks.

Signature · 1 line
tejun
#34 lore raw
Quoted 4 lines · Tejun Heo

No, I'll let the first chunk allocation happen the same way for cpus available on boot and then just do 4k allocations for whatever necessary afterward. The needed code change in percpu proper isn't that big.

Ok. It will be more cleaner this way.

thanks,
suresh

#36 lore raw
Quoted 1 line · Tejun Heo

Ingo, Ping.

hpa is handling this - Peter will reply if he has any updates.

Ingo

#37 lore raw
Quoted 5 lines · Ingo Molnar
Quoted 1 line · Tejun Heo

Ingo, Ping.

hpa is handling this - Peter will reply if he has any updates.

Okay, but please ignore this thread. I thought I was replying to take#2 thread. At any rate, the git vector is the same and contains the final version.

Thanks.

Signature · 1 line
tejun