x86,percpu: fix pageattr handling with remap allocator, take#2
Subsystems (5)
Tejun Heo
maintainerPATCH 5/5 x86: ensure percpu remap doesn't consume too much vmalloc space
In reply to Tejun HeoOn extreme configuration (e.g. 32bit 32-way NUMA machine), remap percpu first chunk allocator can consume too much of vmalloc space. Make it fall back to 4k allocator if the consumption goes over 20%.
[ Impact: add sanity check for remap percpu first chunk allocator ]
--- arch/x86/kernel/setup_percpu.c | 18 +++++++++++++++--- 1 files changed, 15 insertions(+), 3 deletions(-)
arch/x86/kernel/setup_percpu.c +15 -3
@@ -169,9 +169,21 @@ static ssize_t __init setup_pcpu_remap(size_t static_size, bool chosen)return -EINVAL ; } -/* on non-NUMA, embedding is better */ -if (!chosen && !pcpu_need_numa ()) -return -EINVAL ; +if (!chosen) { +size_t vm_size =VMALLOC_END -VMALLOC_START ; +size_t tot_size =num_possible_cpus () *PMD_SIZE ; + +/* on non-NUMA, embedding is better */ +if (!pcpu_need_numa ()) +return -EINVAL ; + +/* don't consume more than 20% of vmalloc area */ +if (tot_size > vm_size /5 ) { +pr_info ("PERCPU: too large chunk size %zuMB for " +"large page remap\n" , tot_size >>20 ); +return -EINVAL ; + } + }/* * Currently supports only single page. Supporting multiple
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 4/5 x86: implement percpu_alloc kernel parameter
In reply to Tejun HeoAccording to Andi, it isn't clear whether remap allocator is worth the trouble as there are many processors where PMD TLB is far scarcer than PTE TLB. The advantage or disadvantage probably depends on the actual size of percpu area and specific processor. As performance degradation due to TLB pressure tends to be highly workload specific and subtle, it is difficult to decide which way to go without more data.
This patch implements percpu_alloc kernel parameter to allow selecting which first chunk allocator to use to ease debugging and testing.
While at it, make sure all the failure paths report why something failed to help determining why certain allocator isn't working. Also, kill the "Great future plan" comment which had already been realized quite some time ago.
[ Impact: allow explicit percpu first chunk allocator selection ]
--- Documentation/kernel-parameters.txt | 6 +++ arch/x86/kernel/setup_percpu.c | 69 +++++++++++++++++++++++++---------- mm/percpu.c | 13 +++++-- 3 files changed, 65 insertions(+), 23 deletions(-)
Documentation/kernel-parameters.txt +6 -0
@@ -1837,6 +1837,12 @@ and is between 256 and 4096 characters. It is defined in the file Format: { 0 | 1 } See arch/parisc/kernel/pdc_chassis.c + percpu_alloc= [X86] Select which percpu first chunk allocator to use. + Allowed values are one of "remap", "embed" and "4k". + See comments in arch/x86/kernel/setup_percpu.c for + details on each allocator. This parameter is primarily + for debugging and performance comparison. + pf. [PARIDE] See Documentation/blockdev/paride.txt.
arch/x86/kernel/setup_percpu.c +50 -19
131 lines not on this page · raw message
mm/percpu.c +9 -4
@@ -1287,6 +1287,7 @@ static struct page * __init pcpue_get_page(unsigned int cpu, int pageno)ssize_t __init pcpu_embed_first_chunk (size_t static_size,size_t reserved_size,ssize_t dyn_size,ssize_t unit_size) { +size_t chunk_size;unsigned int cpu;/* determine parameters and allocate */
@@ -1301,11 +1302,15 @@ ssize_t __init pcpu_embed_first_chunk(size_t static_size, size_t reserved_size, }else pcpue_unit_size =max_t (size_t, pcpue_size,PCPU_MIN_UNIT_SIZE ); - pcpue_ptr =__alloc_bootmem_nopanic ( -num_possible_cpus () * pcpue_unit_size, -PAGE_SIZE ,__pa (MAX_DMA_ADDRESS )); -if (!pcpue_ptr) + chunk_size = pcpue_unit_size *num_possible_cpus (); + + pcpue_ptr =__alloc_bootmem_nopanic (chunk_size,PAGE_SIZE , +__pa (MAX_DMA_ADDRESS )); +if (!pcpue_ptr) { +pr_warning ("PERCPU: failed to allocate %zu bytes for " +"embedding\n" , chunk_size);return -ENOMEM ; + }/* return the leftover and copy */ for_each_possible_cpu (cpu) {
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 1/1 x86,percpu: fix pageattr handling with remap allocator, take#2
Hello,
Upon ack, please pull from the following git tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git x86-percpu-pageattr
This is the second take of x86-percpu-fix-pageattr patchset. Changes from the last take[L] are...
* 0002 and renamed as per Jan's comment.
* 0003 updated accordingly.
* 0004 updated so that selection logic is left in individual setup functions. This way is easier to add more complex selection logic.
* 0005 added. This patch makes sure remap allocator doesn't consume too much of vmalloc space. The cutoff is currently 20%.
This patchset contains the following five patches.
0001-x86-prepare-setup_pcpu_remap-for-pageattr-fix.patch 0002-x86-reorganize-cpa_process_alias.patch 0003-x86-fix-pageattr-handling-for-remap-percpu-allocato.patch 0004-x86-implement-percpu_alloc-kernel-parameter.patch 0005-x86-ensure-percpu-remap-doesn-t-consume-too-much-vm.patch
and is on top of the current linux-2.6#master (45d44740).
Ingo, 0001-0003 are bug fixes and although 0004-0005 aren't exactly bug fixes, I think it would be better to include them in 2.6.30.
diffstat follows.
Documentation/kernel-parameters.txt | 6 + arch/x86/include/asm/percpu.h | 9 + arch/x86/kernel/setup_percpu.c | 199 +++++++++++++++++++++++++++--------- arch/x86/mm/pageattr.c | 63 +++++++---- mm/percpu.c | 13 +- 5 files changed, 218 insertions(+), 72 deletions(-)
Thanks.
--
tejun
Tejun Heo
maintainerPATCH 2/5 x86: reorganize cpa_process_alias()
Reorganize cpa_process_alias() so that new alias condition can be added easily.
Jan Beulich spotted problem in the original cleanup thread which incorrectly assumed the two existing conditions were mutially exclusive.
[ Impact: code reorganization ]
--- arch/x86/mm/pageattr.c | 44 ++++++++++++++++++++------------------------ 1 files changed, 20 insertions(+), 24 deletions(-)
arch/x86/mm/pageattr.c +20 -24
@@ -686,8 +686,8 @@ static int __change_page_attr_set_clr(struct cpa_data *cpa, int checkalias);static int cpa_process_alias (struct cpa_data *cpa) {struct cpa_data alias_cpa; -int ret =0 ;unsigned long temp_cpa_vaddr, vaddr; +int ret;if (cpa->pfn >= max_pfn_mapped)return 0 ;
@@ -715,38 +715,34 @@ static int cpa_process_alias(struct cpa_data *cpa) alias_cpa.vaddr = &temp_cpa_vaddr; alias_cpa.flags &= ~(CPA_PAGES_ARRAY |CPA_ARRAY ); - ret =__change_page_attr_set_clr (&alias_cpa,0 ); +if (ret) +return ret; }#ifdef CONFIG_X86_64 -if (ret) -return ret;/* -* No need to redo, when the primary call touched the high -* mapping already: -*/ -if (within (vaddr, (unsigned long ) _text, _brk_end)) -return 0 ; - -/* -* If the physical address is inside the kernel map, we need +* If the primary call didn't touch the high mapping already +* and the physical address is inside the kernel map, we need * to touch the high mapped kernel as well: */ -if (!within (cpa->pfn,highmap_start_pfn (),highmap_end_pfn ())) -return 0 ; - - alias_cpa = *cpa; - temp_cpa_vaddr = (cpa->pfn <<PAGE_SHIFT ) + __START_KERNEL_map - phys_base; - alias_cpa.vaddr = &temp_cpa_vaddr; - alias_cpa.flags &= ~(CPA_PAGES_ARRAY |CPA_ARRAY ); +if (!within (vaddr, (unsigned long )_text, _brk_end) && +within (cpa->pfn,highmap_start_pfn (),highmap_end_pfn ())) { + alias_cpa = *cpa; + temp_cpa_vaddr = (cpa->pfn <<PAGE_SHIFT ) + + __START_KERNEL_map - phys_base; + alias_cpa.vaddr = &temp_cpa_vaddr; + alias_cpa.flags &= ~(CPA_PAGES_ARRAY |CPA_ARRAY ); -/* -* The high mapping range is imprecise, so ignore the return value. -*/ -__change_page_attr_set_clr (&alias_cpa,0 ); +/* +* The high mapping range is imprecise, so ignore the +* return value. +*/ +__change_page_attr_set_clr (&alias_cpa,0 ); + }#endif -return ret; + +return 0 ; }static int __change_page_attr_set_clr (struct cpa_data *cpa,int checkalias)
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 1/5 x86: prepare setup_pcpu_remap() for pageattr fix
In reply to Tejun HeoMake the following changes in preparation of coming pageattr updates.
* Define and use array of struct pcpur_ent instead of array of pointers. The only difference is ->cpu field which is set but unused yet.
* Rename variables according to the above change.
* Rename local variable vm to pcpur_vm and move it out of the function.
[ Impact: no functional difference ]
--- arch/x86/kernel/setup_percpu.c | 58 ++++++++++++++++++++++----------------- 1 files changed, 33 insertions(+), 25 deletions(-)
arch/x86/kernel/setup_percpu.c +33 -25
@@ -137,8 +137,14 @@ static void * __init pcpu_alloc_bootmem(unsigned int cpu, unsigned long size,* better than only using 4k mappings while still being NUMA friendly. */ #ifdef CONFIG_NEED_MULTIPLE_NODES +struct pcpur_ent { +unsigned int cpu; +void *ptr; +}; +static size_t pcpur_size__initdata ; -static void **pcpur_ptrs__initdata ; +static struct pcpur_ent *pcpur_map__initdata ; +static struct vm_struct pcpur_vm;static struct page *__init pcpur_get_page (unsigned int cpu,int pageno) {
@@ -147,13 +153,12 @@ static struct page * __init pcpur_get_page(unsigned int cpu, int pageno)if (off >= pcpur_size)return NULL ; -return virt_to_page (pcpur_ptrs[cpu] + off); +return virt_to_page (pcpur_map[cpu].ptr + off); }static ssize_t __init setup_pcpu_remap (size_t static_size) { -static struct vm_struct vm; -size_t ptrs_size, dyn_size; +size_t map_size, dyn_size;unsigned int cpu;ssize_t ret;
@@ -178,12 +183,14 @@ static ssize_t __init setup_pcpu_remap(size_t static_size) dyn_size = pcpur_size - static_size -PERCPU_FIRST_CHUNK_RESERVE ;/* allocate pointer array and alloc large pages */ - ptrs_size =PFN_ALIGN (num_possible_cpus () *sizeof (pcpur_ptrs[0 ])); - pcpur_ptrs =alloc_bootmem (ptrs_size); + map_size =PFN_ALIGN (num_possible_cpus () *sizeof (pcpur_map[0 ])); + pcpur_map =alloc_bootmem (map_size);for_each_possible_cpu (cpu) { - pcpur_ptrs[cpu] =pcpu_alloc_bootmem (cpu,PMD_SIZE ,PMD_SIZE ); -if (!pcpur_ptrs[cpu]) + pcpur_map[cpu].cpu = cpu; + pcpur_map[cpu].ptr =pcpu_alloc_bootmem (cpu,PMD_SIZE , +PMD_SIZE ); +if (!pcpur_map[cpu].ptr) goto enomem;/*
@@ -194,42 +201,43 @@ static ssize_t __init setup_pcpu_remap(size_t static_size)* not well-specified to have a PAT-incompatible area * (unmapped RAM, device memory, etc.) in that hole. */ -free_bootmem (__pa (pcpur_ptrs[cpu] + pcpur_size), +free_bootmem (__pa (pcpur_map[cpu].ptr + pcpur_size),PMD_SIZE - pcpur_size); - memcpy(pcpur_ptrs[cpu], __per_cpu_load, static_size); + memcpy(pcpur_map[cpu].ptr, __per_cpu_load, static_size); }/* allocate address and map */ - vm.flags =VM_ALLOC ; - vm.size =num_possible_cpus () *PMD_SIZE ; -vm_area_register_early (&vm,PMD_SIZE ); + pcpur_vm.flags =VM_ALLOC ; + pcpur_vm.size =num_possible_cpus () *PMD_SIZE ; +vm_area_register_early (&pcpur_vm,PMD_SIZE );for_each_possible_cpu (cpu) { -pmd_t *pmd; +pmd_t *pmd, pmd_v; - pmd =populate_extra_pmd ((unsigned long )vm.addr - + cpu *PMD_SIZE ); -set_pmd (pmd,pfn_pmd (page_to_pfn (virt_to_page (pcpur_ptrs[cpu])), -PAGE_KERNEL_LARGE )); + pmd =populate_extra_pmd ((unsigned long )pcpur_vm.addr + + cpu *PMD_SIZE ); + pmd_v =pfn_pmd (page_to_pfn (virt_to_page (pcpur_map[cpu].ptr)), +PAGE_KERNEL_LARGE ); +set_pmd (pmd, pmd_v); }/* we're ready, commit */ pr_info ("PERCPU: Remapped at %p with large pages, static data " -"%zu bytes\n" , vm.addr, static_size); +"%zu bytes\n" , pcpur_vm.addr, static_size); ret =pcpu_setup_first_chunk (pcpur_get_page, static_size,PERCPU_FIRST_CHUNK_RESERVE , dyn_size, -PMD_SIZE , vm.addr,NULL ); - goto out_free_ar; +PMD_SIZE , pcpur_vm.addr,NULL ); + goto out_free_map; enomem:for_each_possible_cpu (cpu) -if (pcpur_ptrs[cpu]) -free_bootmem (__pa (pcpur_ptrs[cpu]),PMD_SIZE ); +if (pcpur_map[cpu].ptr) +free_bootmem (__pa (pcpur_map[cpu].ptr),PMD_SIZE ); ret = -ENOMEM ; -out_free_ar: -free_bootmem (__pa (pcpur_ptrs), ptrs_size); +out_free_map: +free_bootmem (__pa (pcpur_map), map_size);return ret; }#else
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 3/5 x86: fix pageattr handling for remap percpu allocator
In reply to Tejun HeoRemap allocator aliases a PMD page for each cpu and returns whatever is unused to the page allocator. When the pageattr of the recycled pages are changed, this makes the two aliases point to the overlapping regions with different attributes which isn't allowed and known to cause subtle data corruption in certain cases.
This can be handled in simliar manner to the x86_64 highmap alias. pageattr code should detect if the target pages have PMD alias and split the PMD alias and synchronize the attributes.
pcpur allocator is updated to keep the allocated PMD pages map sorted in ascending address order and provide pcpu_pmd_remapped() function which binary searches the array to determine whether the given address is aliased and if so to which address. pageattr is updated to use pcpu_pmd_remapped() to detect the PMD alias and split it up as necessary from cpa_process_alias().
This problem has been spotted by Jan Beulich.
[ Impact: fix subtle pageattr bug ]
--- arch/x86/include/asm/percpu.h | 9 +++++ arch/x86/kernel/setup_percpu.c | 68 ++++++++++++++++++++++++++++++++++++--- arch/x86/mm/pageattr.c | 23 +++++++++++++ 3 files changed, 94 insertions(+), 6 deletions(-)
arch/x86/include/asm/percpu.h +9 -0
@@ -155,6 +155,15 @@ do { \/* We can use this directly for local CPU (faster). */ DECLARE_PER_CPU(unsigned long, this_cpu_off); +#ifdef CONFIG_NEED_MULTIPLE_NODES +void *pcpu_pmd_remapped (void *kaddr); +#else +static inline void *pcpu_pmd_remapped (void *kaddr) +{ +return NULL ; +} +#endif +#endif /* !__ASSEMBLY__ */ #ifdef CONFIG_SMP
arch/x86/kernel/setup_percpu.c +62 -6
92 lines not on this page · raw message
arch/x86/mm/pageattr.c +23 -0
@@ -11,6 +11,7 @@#include <linux/interrupt.h> #include <linux/seq_file.h> #include <linux/debugfs.h> +#include <linux/pfn.h> #include <asm/e820.h> #include <asm/processor.h>
@@ -687,6 +688,7 @@ static int cpa_process_alias(struct cpa_data *cpa) {struct cpa_data alias_cpa;unsigned long temp_cpa_vaddr, vaddr; +void *remapped;int ret;if (cpa->pfn >= max_pfn_mapped)
@@ -742,6 +744,27 @@ static int cpa_process_alias(struct cpa_data *cpa) }#endif +/* +* If the PMD page was partially used for per-cpu remapping, +* the remapped area needs to be split and modified. Note +* that the partial recycling only happens at the tail of a +* partially used PMD page, so touching single PMD page is +* always enough. +*/ + remapped =pcpu_pmd_remapped ((void *)vaddr); +if (remapped) { +int max_pages =PFN_DOWN (PMD_SIZE - (vaddr & ~PMD_MASK )); + + alias_cpa = *cpa; + temp_cpa_vaddr = (unsigned long )remapped; + alias_cpa.vaddr = &temp_cpa_vaddr; + alias_cpa.numpages =min (alias_cpa.numpages, max_pages); + alias_cpa.flags &= ~(CPA_PAGES_ARRAY |CPA_ARRAY ); + ret =__change_page_attr_set_clr (&alias_cpa,0 ); +if (ret) +return ret; + } +return 0 ; }
Signature · 1 line
1.6.0.2
Re: [PATCH 3/5] x86: fix pageattr handling for remap percpu allocator
arch/x86/mm/pageattr.c static int cpa_process_alias(struct cpa_data *cpa) · line 754
Quoted 13 lines
Quoted 1 line
Quoted 1 line
Tejun Heo <[email protected]> 15.05.09 06:28 >>>
arch/x86/mm/pageattr.c +8
-0@@ -742,6 +744,27 @@ static int cpa_process_alias(struct cpa_data *cpa) }#endif +/* +* If the PMD page was partially used for per-cpu remapping, +* the remapped area needs to be split and modified. Note +* that the partial recycling only happens at the tail of a +* partially used PMD page, so touching single PMD page is +* always enough. +*/ + remapped =pcpu_pmd_remapped ((void *)vaddr);
vaddr here is the passed-in address, but in order for the lookup to be positive it needs to be the canonical address (i.e. the one pointing into the 1:1 mapping), i.e. __va(cpa->pfn << PAGE_SHIFT), just like temp_cpa_vaddr gets calculated earlier in the same function.
Quoted 5 lines
arch/x86/mm/pageattr.c +5
-0+if (remapped) { +int max_pages =PFN_DOWN (PMD_SIZE - (vaddr & ~PMD_MASK )); + + alias_cpa = *cpa; + temp_cpa_vaddr = (unsigned long )remapped;
arch/x86/mm/pageattr.c static int cpa_process_alias(struct cpa_data *cpa) · line 769
Quoted 10 lines
arch/x86/mm/pageattr.c +8
-0+ alias_cpa.vaddr = &temp_cpa_vaddr; + alias_cpa.numpages =min (alias_cpa.numpages, max_pages); + alias_cpa.flags &= ~(CPA_PAGES_ARRAY |CPA_ARRAY ); + ret =__change_page_attr_set_clr (&alias_cpa,0 ); +if (ret) +return ret; + } +return 0 ; }
Sorry for not having spotted this yesterday,
Jan
Re: [PATCH 4/5] x86: implement percpu_alloc kernel parameter
In reply to Tejun HeoQuoted 8 lines
Quoted 1 line
Quoted 1 line
Tejun Heo <[email protected]> 15.05.09 06:28 >>>
arch/x86/kernel/setup_percpu.c +3
-0@@ -395,6 +404,16 @@ out_free_ar:return ret; } +/* for explicit first chunk allocator selection */ +static char pcpu_chosen_alloc[16 ]__initdata ; +
arch/x86/kernel/setup_percpu.c out_free_ar: · line 419
Quoted 10 lines
arch/x86/kernel/setup_percpu.c +7
-0+static int __init percpu_alloc_setup (char *str) +{ +strncpy (pcpu_chosen_alloc, str,sizeof (pcpu_chosen_alloc) -1 ); +return 0 ; +} +early_param ("percpu_alloc" , percpu_alloc_setup); +static inline void setup_percpu_segment (int cpu) {#ifdef CONFIG_X86_32
Does this really need to be early_param()? setup_per_cpu_areas() runs after setup_arch(), which is where the full command line parsing happens.
Jan
Tejun Heo
maintainerRe: [PATCH 3/5] x86: fix pageattr handling for remap percpu allocator
In reply to Jan BeulichHello,
Quoted 5 lines · Jan Beulich
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Tejun Heo <[email protected]> 15.05.09 06:28 >>>
(quoted diff) +0
-0@@ -742,6 +744,27 @@ static int cpa_process_alias(struct cpa_data *cpa) } #endif
Quoted 14 lines
arch/x86/mm/pageattr.c static int cpa_process_alias(struct cpa_data *cpa) · line 754
Quoted 8 lines
arch/x86/mm/pageattr.c +8
-0+/* +* If the PMD page was partially used for per-cpu remapping, +* the remapped area needs to be split and modified. Note +* that the partial recycling only happens at the tail of a +* partially used PMD page, so touching single PMD page is +* always enough. +*/ + remapped =pcpu_pmd_remapped ((void *)vaddr);vaddr here is the passed-in address, but in order for the lookup to be positive it needs to be the canonical address (i.e. the one pointing into the 1:1 mapping), i.e.
__va(cpa->pfn << PAGE_SHIFT), just liketemp_cpa_vaddrgets calculated earlier in the same function.
Hmmm... There's no need to match percpu map -> linear addr. Okay, right, if the recycled part has been aliased to different vmalloc mapping than the pfn should be matched.
Thanks for spotting it. Will posted updated patch.
Signature · 1 line
tejun
Tejun Heo
maintainerRe: [PATCH 4/5] x86: implement percpu_alloc kernel parameter
In reply to Jan BeulichQuoted 14 lines · Jan Beulich
Quoted 14 lines
Quoted 1 line
Quoted 1 line
Tejun Heo <[email protected]> 15.05.09 06:28 >>>
arch/x86/kernel/setup_percpu.c +9
-0@@ -395,6 +404,16 @@ out_free_ar:return ret; } +/* for explicit first chunk allocator selection */ +static char pcpu_chosen_alloc[16 ]__initdata ; + +static int __init percpu_alloc_setup (char *str) +{ +strncpy (pcpu_chosen_alloc, str,sizeof (pcpu_chosen_alloc) -1 ); +return 0 ; +} +early_param ("percpu_alloc" , percpu_alloc_setup);
Quoted 8 lines
arch/x86/kernel/setup_percpu.c out_free_ar: · line 419
Quoted 4 lines
arch/x86/kernel/setup_percpu.c +1
-0+static inline void setup_percpu_segment (int cpu) {#ifdef CONFIG_X86_32 Does this really need to be
early_param()?setup_per_cpu_areas()runs aftersetup_arch(), which is where the full command line parsing happens.
__setup() callbacks ran after PERCPU allocation. Confirmed with printk.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerPATCH 2/5 x86: reorganize cpa_process_alias()
In reply to Tejun HeoReorganize cpa_process_alias() so that new alias condition can be added easily.
Jan Beulich spotted problem in the original cleanup thread which incorrectly assumed the two existing conditions were mutially exclusive.
[ Impact: code reorganization ]
--- arch/x86/mm/pageattr.c | 50 ++++++++++++++++++++++--------------------------- 1 file changed, 23 insertions(+), 27 deletions(-)
arch/x86/mm/pageattr.c +23 -27
@@ -686,8 +686,9 @@ static int __change_page_attr_set_clr(ststatic int cpa_process_alias (struct cpa_data *cpa) {struct cpa_data alias_cpa; -int ret =0 ; -unsigned long temp_cpa_vaddr, vaddr; +unsigned long laddr = (unsigned long )__va (cpa->pfn <<PAGE_SHIFT ); +unsigned long vaddr; +int ret;if (cpa->pfn >= max_pfn_mapped)return 0 ;
@@ -711,42 +712,37 @@ static int cpa_process_alias(struct cpa_ PAGE_OFFSET + (max_pfn_mapped << PAGE_SHIFT)))) { alias_cpa = *cpa; - temp_cpa_vaddr = (unsigned long )__va (cpa->pfn <<PAGE_SHIFT ); - alias_cpa.vaddr = &temp_cpa_vaddr; + alias_cpa.vaddr = &laddr; alias_cpa.flags &= ~(CPA_PAGES_ARRAY |CPA_ARRAY ); - ret =__change_page_attr_set_clr (&alias_cpa,0 ); +if (ret) +return ret; }#ifdef CONFIG_X86_64 -if (ret) -return ret; -/* -* No need to redo, when the primary call touched the high -* mapping already: -*/ -if (within (vaddr, (unsigned long ) _text, _brk_end)) -return 0 ; -/* -* If the physical address is inside the kernel map, we need +* If the primary call didn't touch the high mapping already +* and the physical address is inside the kernel map, we need * to touch the high mapped kernel as well: */ -if (!within (cpa->pfn,highmap_start_pfn (),highmap_end_pfn ())) -return 0 ; - - alias_cpa = *cpa; - temp_cpa_vaddr = (cpa->pfn <<PAGE_SHIFT ) + __START_KERNEL_map - phys_base; - alias_cpa.vaddr = &temp_cpa_vaddr; - alias_cpa.flags &= ~(CPA_PAGES_ARRAY |CPA_ARRAY ); +if (!within (vaddr, (unsigned long )_text, _brk_end) && +within (cpa->pfn,highmap_start_pfn (),highmap_end_pfn ())) { +unsigned long temp_cpa_vaddr = (cpa->pfn <<PAGE_SHIFT ) + + __START_KERNEL_map - phys_base; + alias_cpa = *cpa; + alias_cpa.vaddr = &temp_cpa_vaddr; + alias_cpa.flags &= ~(CPA_PAGES_ARRAY |CPA_ARRAY ); -/* -* The high mapping range is imprecise, so ignore the return value. -*/ -__change_page_attr_set_clr (&alias_cpa,0 ); +/* +* The high mapping range is imprecise, so ignore the +* return value. +*/ +__change_page_attr_set_clr (&alias_cpa,0 ); + }#endif -return ret; + +return 0 ; }static int __change_page_attr_set_clr (struct cpa_data *cpa,int checkalias)
Tejun Heo
maintainerPATCH 3/5 x86: fix pageattr handling for remap percpu allocator
In reply to Tejun HeoRemap allocator aliases a PMD page for each cpu and returns whatever is unused to the page allocator. When the pageattr of the recycled pages are changed, this makes the two aliases point to the overlapping regions with different attributes which isn't allowed and known to cause subtle data corruption in certain cases.
This can be handled in simliar manner to the x86_64 highmap alias. pageattr code should detect if the target pages have PMD alias and split the PMD alias and synchronize the attributes.
pcpur allocator is updated to keep the allocated PMD pages map sorted in ascending address order and provide pcpu_pmd_remapped() function which binary searches the array to determine whether the given address is aliased and if so to which address. pageattr is updated to use pcpu_pmd_remapped() to detect the PMD alias and split it up as necessary from cpa_process_alias().
Jan Beulich spotted the original problem and incorrect usage of vaddr instead of laddr for lookup.
[ Impact: fix subtle pageattr bug ]
--- arch/x86/include/asm/percpu.h | 9 +++++ arch/x86/kernel/setup_percpu.c | 68 +++++++++++++++++++++++++++++++++++++---- arch/x86/mm/pageattr.c | 26 +++++++++++++++ 3 files changed, 96 insertions(+), 7 deletions(-)
arch/x86/include/asm/percpu.h +9 -0
@@ -155,6 +155,15 @@ do { \/* We can use this directly for local CPU (faster). */ DECLARE_PER_CPU(unsigned long, this_cpu_off); +#ifdef CONFIG_NEED_MULTIPLE_NODES +void *pcpu_pmd_remapped (void *kaddr); +#else +static inline void *pcpu_pmd_remapped (void *kaddr) +{ +return NULL ; +} +#endif +#endif /* !__ASSEMBLY__ */ #ifdef CONFIG_SMP
arch/x86/kernel/setup_percpu.c +62 -6
92 lines not on this page · raw message
arch/x86/mm/pageattr.c +25 -1
@@ -11,6 +11,7 @@#include <linux/interrupt.h> #include <linux/seq_file.h> #include <linux/debugfs.h> +#include <linux/pfn.h> #include <asm/e820.h> #include <asm/processor.h>
@@ -687,7 +688,7 @@ static int cpa_process_alias(struct cpa_ {struct cpa_data alias_cpa;unsigned long laddr = (unsigned long )__va (cpa->pfn <<PAGE_SHIFT ); -unsigned long vaddr; +unsigned long vaddr, remapped;int ret;if (cpa->pfn >= max_pfn_mapped)
@@ -742,6 +743,29 @@ static int cpa_process_alias(struct cpa_ }#endif +/* +* If the PMD page was partially used for per-cpu remapping, +* the remapped area needs to be split and modified. Note +* that the partial recycling only happens at the tail of a +* partially used PMD page, so touching single PMD page is +* always enough. +* +* Look up alias using the linear address to detect, for +* example, recycled pages which got vmapped. +*/ + remapped = (unsigned long )pcpu_pmd_remapped ((void *)laddr); +if (remapped) { +int max_pages =PFN_DOWN (PMD_SIZE - (vaddr & ~PMD_MASK )); + + alias_cpa = *cpa; + alias_cpa.vaddr = &remapped; + alias_cpa.numpages =min (alias_cpa.numpages, max_pages); + alias_cpa.flags &= ~(CPA_PAGES_ARRAY |CPA_ARRAY ); + ret =__change_page_attr_set_clr (&alias_cpa,0 ); +if (ret) +return ret; + } +return 0 ; }
Quoted 5 lines · Tejun Heo
Hello,
Upon ack, please pull from the following git tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.gitx86-percpu-pageattr
Patches 0002 and 0003 updated per Jan's comments. New git commit ID is 229f8495873f9f8928c9c99f790f2e0f4b6b0804.
Thanks.
Signature · 1 line
tejun
Jan Beulich
Reviewed-byRe: [GIT PATCH 2.6.30-rc5] x86,percpu: fix pageattr handling withremap allocator, take#2
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Tejun Heo <[email protected]> 15.05.09 11:36 >>>
Quoted 5 lines · Tejun Heo
Hello,
Upon ack, please pull from the following git tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.gitx86-percpu-pageattrPatches 0002 and 0003 updated per Jan's comments. New git commit ID is
229f8495873f9f8928c9c99f790f2e0f4b6b0804.
With this the whole series is
Re: [GIT PATCH 2.6.30-rc5] x86,percpu: fix pageattr handling withremap allocator, take#2
In reply to a message not in the archiveQuoted 6 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Tejun Heo <[email protected]> 15.05.09 11:36 >>>
Quoted 4 lines · Tejun Heo
Hello,
Upon ack, please pull from the following git tree.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.gitx86-percpu-pageattrPatches 0002 and 0003 updated per Jan's comments. New git commit ID is
229f8495873f9f8928c9c99f790f2e0f4b6b0804.With this the whole series is
Reviewed-by Jan Beulich <[email protected]>
Thanks. Much appreciated.
Signature · 1 line
tejun