x86, percpu: implement and use reserved percpu alloc
Subsystems (3)
Tejun Heo
maintainerHello,
Please pull from the following git vector.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git tj-percpu
This patchset implements reserved percpu alloc from the first chunk and uses it for static perpcu variables in modules on x86_64. This is necessary because x86_64 expects symbols to be relocatable using 32bit relocations but dynamic percpu allocation does not guarantee it leading to relocation overflow during module load.
This can happen to any of the x86_64 percpu first chunk allocators but the embedding allocator is most likely to trigger it as the first chunk is located in the normal kernel memory allocation area while all the other chunks will be allocated in the vmalloc area which is very far from the first chunk.
This type of reservation will be necessary for other architectures too where addressing mode used for percpu acess uses offsets with limited range to access symbols (e.g. ia64 w/ the magic 64k percpu page).
This problem was first reported and bisected by Mike Galbraith[2].
This patchset is against the current x86/core/percpu[1] and contains the following patches.
0001-percpu-clean-up-percpu-constants.patch 0002-percpu-cosmetic-renames-in-pcpu_setup_first_chunk.patch 0003-percpu-improve-first-chunk-initial-area-map-handlin.patch 0004-percpu-use-negative-for-auto-for-pcpu_setup_first_c.patch 0005-x86-make-embedding-percpu-allocator-return-excessiv.patch 0006-percpu-add-an-indirection-ptr-for-chunk-page-map-ac.patch 0007-percpu-module-implement-reserved-allocation-and-us.patch 0008-x86-percpu-setup-reserved-percpu-area-for-x86_64.patch
0001-0004 makes misc updates to percpu in preparation of later changes. 0005 fixes a case where excessive memory can be set aside in the first chunk when using the embedding allocator. 0006-0008 implement reserved allocation and use it for x86_64.
Bryan Wu, can you please review #0001? It contains small change in blackfin. It's mostly trivial but just in case.
Diffstat follows.
arch/blackfin/include/asm/percpu.h | 10 - arch/x86/kernel/setup_percpu.c | 73 +++++++--- include/linux/percpu.h | 60 +++----- kernel/module.c | 2 mm/percpu.c | 267 ++++++++++++++++++++++++++----------- 5 files changed, 274 insertions(+), 138 deletions(-)
Thanks.
--
tejun
[1] f254f3909efaf59ca2d0f408de2d044dace60706
[2] http://thread.gmane.org/gmane.linux.kernel/801799
Tejun Heo
maintainerPATCH 1/8 percpu: clean up percpu constants
Impact: cleaup
Make the following cleanups.
* There isn't much arch-specific about PERCPU_MODULE_RESERVE. Always define it whether arch overrides PERCPU_ENOUGH_ROOM or not.
* blackfin overrides PERCPU_ENOUGH_ROOM to align static area size. Do it by default.
* percpu allocation sizes doesn't have much to do with the page size. Don't use PAGE_SHIFT in their definition.
--- arch/blackfin/include/asm/percpu.h | 10 ---------- include/linux/percpu.h | 24 +++++++++++++----------- 2 files changed, 13 insertions(+), 21 deletions(-)
arch/blackfin/include/asm/percpu.h +0 -10
@@ -3,14 +3,4 @@#include <asm-generic/percpu.h> -#ifdef CONFIG_MODULES -#define PERCPU_MODULE_RESERVE 8192 -#else -#define PERCPU_MODULE_RESERVE 0 -#endif - -#define PERCPU_ENOUGH_ROOM \ - (ALIGN(__per_cpu_end - __per_cpu_start, SMP_CACHE_BYTES) + \ - PERCPU_MODULE_RESERVE) -#endif /* __ARCH_BLACKFIN_PERCPU__ */
include/linux/percpu.h +13 -11
@@ -5,6 +5,7 @@#include <linux/slab.h> /* For kmalloc() */ #include <linux/smp.h> #include <linux/cpumask.h> +#include <linux/pfn.h> #include <asm/percpu.h>
@@ -52,17 +53,18 @@#define EXPORT_PER_CPU_SYMBOL (var) EXPORT_SYMBOL(per_cpu__##var)#define EXPORT_PER_CPU_SYMBOL_GPL (var) EXPORT_SYMBOL_GPL(per_cpu__##var) -/* Enough to cover all DEFINE_PER_CPUs in kernel, including modules. */ -#ifndef PERCPU_ENOUGH_ROOM +/* enough to cover all DEFINE_PER_CPUs in modules */ #ifdef CONFIG_MODULES -#define PERCPU_MODULE_RESERVE 8192 +#define PERCPU_MODULE_RESERVE (8 << 10)#else -#define PERCPU_MODULE_RESERVE 0 +#define PERCPU_MODULE_RESERVE 0#endif +#ifndef PERCPU_ENOUGH_ROOM #define PERCPU_ENOUGH_ROOM \ - (__per_cpu_end - __per_cpu_start + PERCPU_MODULE_RESERVE) -#endif /* PERCPU_ENOUGH_ROOM */ + (ALIGN(__per_cpu_end - __per_cpu_start, SMP_CACHE_BYTES) + \ + PERCPU_MODULE_RESERVE) +#endif /* * Must be an lvalue. Since @var must be a simple identifier,
@@ -79,7 +81,7 @@#ifdef CONFIG_HAVE_DYNAMIC_PER_CPU_AREA /* minimum unit size, also is the maximum supported allocation size */ -#define PCPU_MIN_UNIT_SIZE (16UL << PAGE_SHIFT) +#define PCPU_MIN_UNIT_SIZE PFN_ALIGN(64 << 10)/* * PERCPU_DYNAMIC_RESERVE indicates the amount of free area to piggy
@@ -96,15 +98,15 @@#ifndef PERCPU_DYNAMIC_RESERVE # if BITS_PER_LONG >32 # ifdef CONFIG_MODULES -# define PERCPU_DYNAMIC_RESERVE (6 << PAGE_SHIFT) +# define PERCPU_DYNAMIC_RESERVE (24 << 10)# else -# define PERCPU_DYNAMIC_RESERVE (4 << PAGE_SHIFT) +# define PERCPU_DYNAMIC_RESERVE (16 << 10)# endif # else # ifdef CONFIG_MODULES -# define PERCPU_DYNAMIC_RESERVE (4 << PAGE_SHIFT) +# define PERCPU_DYNAMIC_RESERVE (16 << 10)# else -# define PERCPU_DYNAMIC_RESERVE (2 << PAGE_SHIFT) +# define PERCPU_DYNAMIC_RESERVE (8 << 10)# endif # endif #endif /* PERCPU_DYNAMIC_RESERVE */
Signature · 1 line
1.6.0.2
Re: [PATCH 1/8] percpu: clean up percpu constants
Quoted 10 lines · Tejun Heo
Impact: cleaup
Make the following cleanups.
* There isn't much arch-specific about
PERCPU_MODULE_RESERVE. Always define it whether arch overridesPERCPU_ENOUGH_ROOMor not.* blackfin overrides
PERCPU_ENOUGH_ROOMto align static area size. Do it by default.
Thanks
-Bryan
Quoted 99 lines
* percpu allocation sizes doesn't have much to do with the page size. Don't use
PAGE_SHIFTin their definition.Signed-off-by Tejun Heo <[email protected]>Cc Bryan Wu <[email protected]>---
arch/blackfin/include/asm/percpu.h | 10 ----------include/linux/percpu.h| 24 +++++++++++++-----------2 files changed, 13 insertions(+), 21 deletions(-)arch/blackfin/include/asm/percpu.h +0
-10@@ -3,14 +3,4 @@ #include <asm-generic/percpu.h> -#ifdef CONFIG_MODULES -#define PERCPU_MODULE_RESERVE 8192 -#else -#define PERCPU_MODULE_RESERVE 0 -#endif - -#define PERCPU_ENOUGH_ROOM \ - (ALIGN(__per_cpu_end - __per_cpu_start, SMP_CACHE_BYTES) + \ - PERCPU_MODULE_RESERVE) - #endif /* __ARCH_BLACKFIN_PERCPU__ */include/linux/percpu.h +13
-12@@ -5,6 +5,7 @@ #include <linux/slab.h> /* For kmalloc() */ #include <linux/smp.h> #include <linux/cpumask.h> +#include <linux/pfn.h> #include <asm/percpu.h>@@ -52,17 +53,18 @@ #define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var) #define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var) -/* Enough to cover all DEFINE_PER_CPUs in kernel, including modules. */ -#ifndef PERCPU_ENOUGH_ROOM +/* enough to cover all DEFINE_PER_CPUs in modules */ #ifdef CONFIG_MODULES -#define PERCPU_MODULE_RESERVE 8192 +#define PERCPU_MODULE_RESERVE (8 << 10) #else -#define PERCPU_MODULE_RESERVE 0 +#define PERCPU_MODULE_RESERVE 0 #endif +#ifndef PERCPU_ENOUGH_ROOM #define PERCPU_ENOUGH_ROOM \ - (__per_cpu_end - __per_cpu_start + PERCPU_MODULE_RESERVE) -#endif /* PERCPU_ENOUGH_ROOM */ + (ALIGN(__per_cpu_end - __per_cpu_start, SMP_CACHE_BYTES) + \ + PERCPU_MODULE_RESERVE) +#endif /** Must be an lvalue. Since @var must be a simple identifier, @@ -79,7 +81,7 @@ #ifdef CONFIG_HAVE_DYNAMIC_PER_CPU_AREA /* minimum unit size, also is the maximum supported allocation size */ -#define PCPU_MIN_UNIT_SIZE (16UL << PAGE_SHIFT) +#define PCPU_MIN_UNIT_SIZE PFN_ALIGN(64 << 10) /** PERCPU_DYNAMIC_RESERVE indicates the amount of free area to piggy @@ -96,15 +98,15 @@ #ifndef PERCPU_DYNAMIC_RESERVE # if BITS_PER_LONG > 32 # ifdef CONFIG_MODULES -# define PERCPU_DYNAMIC_RESERVE (6 << PAGE_SHIFT) +# define PERCPU_DYNAMIC_RESERVE (24 << 10) # else -# define PERCPU_DYNAMIC_RESERVE (4 << PAGE_SHIFT) +# define PERCPU_DYNAMIC_RESERVE (16 << 10) # endif # else # ifdef CONFIG_MODULES -# define PERCPU_DYNAMIC_RESERVE (4 << PAGE_SHIFT) +# define PERCPU_DYNAMIC_RESERVE (16 << 10) # else -# define PERCPU_DYNAMIC_RESERVE (2 << PAGE_SHIFT) +# define PERCPU_DYNAMIC_RESERVE (8 << 10) # endif # endif #endif /* PERCPU_DYNAMIC_RESERVE */ --1.6.0.2
Tejun Heo
maintainerPATCH 2/8 percpu: cosmetic renames in pcpu_setup_first_chunk()
In reply to Tejun HeoImpact: cosmetic, preparation for future changes
Make the following renames in pcpur_setup_first_chunk() in preparation for future changes.
* s/free_size/dyn_size/
* s/static_vm/first_vm/
* s/static_chunk/schunk/
--- include/linux/percpu.h | 2 +- mm/percpu.c | 58 ++++++++++++++++++++++++------------------------ 2 files changed, 30 insertions(+), 30 deletions(-)
include/linux/percpu.h +1 -1
@@ -118,7 +118,7 @@ typedef void (*pcpu_populate_pte_fn_t)(unsigned long addr);extern size_t __init pcpu_setup_first_chunk (pcpu_get_page_fn_t get_page_fn,size_t static_size,size_t unit_size, -size_t free_size,void *base_addr, +size_t dyn_size,void *base_addr,pcpu_populate_pte_fn_t populate_pte_fn);/*
mm/percpu.c +29 -29
126 lines not on this page · raw message
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 3/8 percpu: improve first chunk initial area map handling
In reply to Tejun HeoImpact: no functional change
When the first chunk is created, its initial area map is not allocated because kmalloc isn't online yet. The map is allocated and initialized on the first allocation request on the chunk. This works fine but the scattering of initialization logic between the init function and allocation path is a bit confusing.
This patch makes the first chunk initialize and use minimal statically allocated map from pcpu_setpu_first_chunk(). The map resizing path still needs to handle this specially but it's more straight-forward and gives more latitude to the init path. This will ease future changes.
--- mm/percpu.c | 53 +++++++++++++++++++++++++++-------------------------- 1 files changed, 27 insertions(+), 26 deletions(-)
mm/percpu.c +27 -26
@@ -93,9 +93,6 @@ static size_t pcpu_chunk_struct_size __read_mostly;void *pcpu_base_addr__read_mostly ;EXPORT_SYMBOL_GPL (pcpu_base_addr); -/* the size of kernel static area */ -static int pcpu_static_size__read_mostly ; -/* * One mutex to rule them all. *
@@ -316,15 +313,28 @@ static int pcpu_split_block(struct pcpu_chunk *chunk, int i, int head, int tail)/* reallocation required? */ if (chunk->map_alloc < target) { -int new_alloc = chunk->map_alloc; +int new_alloc;int *new; + new_alloc =PCPU_DFL_MAP_ALLOC ;while (new_alloc < target) new_alloc *=2 ; - new =pcpu_realloc (chunk->map, - chunk->map_alloc *sizeof (new[0 ]), - new_alloc *sizeof (new[0 ])); +if (chunk->map_alloc <PCPU_DFL_MAP_ALLOC ) { +/* +* map_alloc smaller than the default size +* indicates that the chunk is one of the +* first chunks and still using static map. +* Allocate a dynamic one and copy. +*/ + new =pcpu_realloc (NULL ,0 , new_alloc *sizeof (new[0 ])); +if (new) +memcpy (new, chunk->map, + chunk->map_alloc *sizeof (new[0 ])); + }else + new =pcpu_realloc (chunk->map, + chunk->map_alloc *sizeof (new[0 ]), + new_alloc *sizeof (new[0 ]));if (!new)return -ENOMEM ;
@@ -367,22 +377,6 @@ static int pcpu_alloc_area(struct pcpu_chunk *chunk, int size, int align)int max_contig =0 ;int i, off; -/* -* The static chunk initially doesn't have map attached -* because kmalloc wasn't available during init. Give it one. -*/ -if (unlikely (!chunk->map)) { - chunk->map =pcpu_realloc (NULL ,0 , -PCPU_DFL_MAP_ALLOC *sizeof (chunk->map[0 ])); -if (!chunk->map) -return -ENOMEM ; - - chunk->map_alloc =PCPU_DFL_MAP_ALLOC ; - chunk->map[chunk->map_used++] = -pcpu_static_size; -if (chunk->free_size) - chunk->map[chunk->map_used++] = chunk->free_size; - } -for (i =0 , off =0 ; i < chunk->map_used; off +=abs (chunk->map[i++])) {bool is_last = i +1 == chunk->map_used;int head, tail;
@@ -874,12 +868,14 @@ size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn, pcpu_populate_pte_fn_t populate_pte_fn) {static struct vm_struct first_vm; +static int smap[2 ];struct pcpu_chunk *schunk;unsigned int cpu;int nr_pages;int err, i;/* santiy checks */ +BUILD_BUG_ON (ARRAY_SIZE (smap) >=PCPU_DFL_MAP_ALLOC );BUG_ON (!static_size);BUG_ON (!unit_size && dyn_size);BUG_ON (unit_size && unit_size < static_size + dyn_size);
@@ -893,7 +889,6 @@ size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn, pcpu_unit_pages =max_t (int,PCPU_MIN_UNIT_SIZE >>PAGE_SHIFT ,PFN_UP (static_size)); - pcpu_static_size = static_size; pcpu_unit_size = pcpu_unit_pages <<PAGE_SHIFT ; pcpu_chunk_size =num_possible_cpus () * pcpu_unit_size; pcpu_chunk_struct_size =sizeof (struct pcpu_chunk )
@@ -912,14 +907,20 @@ size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn, schunk =alloc_bootmem (pcpu_chunk_struct_size);INIT_LIST_HEAD (&schunk->list); schunk->vm = &first_vm; + schunk->map = smap; + schunk->map_alloc =ARRAY_SIZE (smap);if (dyn_size) schunk->free_size = dyn_size;else - schunk->free_size = pcpu_unit_size - pcpu_static_size; + schunk->free_size = pcpu_unit_size - static_size; schunk->contig_hint = schunk->free_size; + schunk->map[schunk->map_used++] = -static_size; +if (schunk->free_size) + schunk->map[schunk->map_used++] = schunk->free_size; +/* allocate vm address */ first_vm.flags =VM_ALLOC ; first_vm.size = pcpu_chunk_size;
@@ -948,7 +949,7 @@ size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn, *pcpu_chunk_pagep (schunk, cpu, i) = page; } -BUG_ON (i <PFN_UP (pcpu_static_size)); +BUG_ON (i <PFN_UP (static_size));if (nr_pages <0 ) nr_pages = i;
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 4/8 percpu: use negative for auto for pcpu_setup_first_chunk() arguments
In reply to Tejun HeoImpact: argument semantic cleanup
In pcpu_setup_first_chunk(), zero @unit_size and @dyn_size meant auto-sizing. It's okay for @unit_size as 0 doesn't make sense but 0 dynamic reserve size is valid. Alos, if arch @dyn_size is calculated from other parameters, it might end up passing in 0 @dyn_size and malfunction when the size is automatically adjusted.
This patch makes both @unit_size and @dyn_size ssize_t and use -1 for auto sizing.
--- arch/x86/kernel/setup_percpu.c | 2 +- include/linux/percpu.h | 5 ++- mm/percpu.c | 46 +++++++++++++++++++++------------------ 3 files changed, 29 insertions(+), 24 deletions(-)
arch/x86/kernel/setup_percpu.c +1 -1
@@ -344,7 +344,7 @@ static ssize_t __init setup_pcpu_4k(size_t static_size)pr_info ("PERCPU: Allocated %d 4k pages, static data %zu bytes\n" , pcpu4k_nr_static_pages, static_size); - ret =pcpu_setup_first_chunk (pcpu4k_get_page, static_size,0 ,0 ,NULL , + ret =pcpu_setup_first_chunk (pcpu4k_get_page, static_size,-1 ,-1 ,NULL , pcpu4k_populate_pte); goto out_free_ar;
include/linux/percpu.h +3 -2
@@ -117,8 +117,9 @@ typedef struct page * (*pcpu_get_page_fn_t)(unsigned int cpu, int pageno);typedef void (*pcpu_populate_pte_fn_t )(unsigned long addr);extern size_t __init pcpu_setup_first_chunk (pcpu_get_page_fn_t get_page_fn, -size_t static_size,size_t unit_size, -size_t dyn_size,void *base_addr, +size_t static_size, +ssize_t unit_size,ssize_t dyn_size, +void *base_addr,pcpu_populate_pte_fn_t populate_pte_fn);/*
mm/percpu.c +25 -21
85 lines not on this page · raw message
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 5/8 x86: make embedding percpu allocator return excessive free space
In reply to Tejun HeoImpact: reduce unnecessary memory usage on certain configurations
Embedding percpu allocator allocates unit_size * smp_num_possible_cpus() bytes consecutively and use it for the first chunk. However, if the static area is small, this can result in excessive prellocated free space in the first chunk due to PCPU_MIN_UNIT_SIZE restriction.
This patch makes embedding percpu allocator preallocate only what's necessary as described by PERPCU_DYNAMIC_RESERVE and return the leftover to the bootmem allocator.
--- arch/x86/kernel/setup_percpu.c | 44 +++++++++++++++++++++++++-------------- 1 files changed, 28 insertions(+), 16 deletions(-)
arch/x86/kernel/setup_percpu.c +28 -16
@@ -241,24 +241,31 @@ static ssize_t __init setup_pcpu_remap(size_t static_size)* Embedding allocator * * The first chunk is sized to just contain the static area plus -* PERCPU_DYNAMIC_RESERVE and allocated as a contiguous area using -* bootmem allocator and used as-is without being mapped into vmalloc -* area. This enables the first chunk to piggy back on the linear -* physical PMD mapping and doesn't add any additional pressure to -* TLB. +* module and dynamic reserves, and allocated as a contiguous area +* using bootmem allocator and used as-is without being mapped into +* vmalloc area. This enables the first chunk to piggy back on the +* linear physical PMD mapping and doesn't add any additional pressure +* to TLB. Note that if the needed size is smaller than the minimum +* unit size, the leftover is returned to the bootmem allocator. */ static void *pcpue_ptr__initdata ; +static size_t pcpue_size__initdata ;static size_t pcpue_unit_size__initdata ;static struct page *__init pcpue_get_page (unsigned int cpu,int pageno) { -return virt_to_page (pcpue_ptr + cpu * pcpue_unit_size - + ((size_t )pageno <<PAGE_SHIFT )); +size_t off = (size_t )pageno <<PAGE_SHIFT ; + +if (off >= pcpue_size) +return NULL ; + +return virt_to_page (pcpue_ptr + cpu * pcpue_unit_size + off); }static ssize_t __init setup_pcpu_embed (size_t static_size) {unsigned int cpu; +size_t dyn_size;/* * If large page isn't supported, there's no benefit in doing
@@ -269,25 +276,30 @@ static ssize_t __init setup_pcpu_embed(size_t static_size)return -EINVAL ;/* allocate and copy */ - pcpue_unit_size =PFN_ALIGN (static_size +PERCPU_DYNAMIC_RESERVE ); - pcpue_unit_size =max_t (size_t, pcpue_unit_size,PCPU_MIN_UNIT_SIZE ); + pcpue_size =PFN_ALIGN (static_size +PERCPU_DYNAMIC_RESERVE ); + pcpue_unit_size =max_t (size_t, pcpue_size,PCPU_MIN_UNIT_SIZE ); + dyn_size = pcpue_size - static_size; + pcpue_ptr =pcpu_alloc_bootmem (0 ,num_possible_cpus () * pcpue_unit_size,PAGE_SIZE );if (!pcpue_ptr)return -ENOMEM ; -for_each_possible_cpu (cpu) -memcpy (pcpue_ptr + cpu * pcpue_unit_size, __per_cpu_load, - static_size); +for_each_possible_cpu (cpu) { +void *ptr = pcpue_ptr + cpu * pcpue_unit_size; + +free_bootmem (__pa (ptr + pcpue_size), + pcpue_unit_size - pcpue_size); +memcpy (ptr, __per_cpu_load, static_size); + }/* we're ready, commit */ pr_info ("PERCPU: Embedded %zu pages at %p, static data %zu bytes\n" , - pcpue_unit_size >>PAGE_SHIFT , pcpue_ptr, static_size); + pcpue_size >>PAGE_SHIFT , pcpue_ptr, static_size);return pcpu_setup_first_chunk (pcpue_get_page, static_size, - pcpue_unit_size, - pcpue_unit_size - static_size, pcpue_ptr, -NULL ); + pcpue_unit_size, dyn_size, + pcpue_ptr,NULL ); }/*
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 6/8 percpu: add an indirection ptr for chunk page map access
In reply to Tejun HeoImpact: allow sharing page map, no functional difference yet
Make chunk->page access indirect by adding a pointer and renaming the actual array to page_ar. This will be used by future changes.
--- mm/percpu.c | 5 ++++- 1 files changed, 4 insertions(+), 1 deletions(-)
mm/percpu.c +4 -1
@@ -80,7 +80,8 @@ struct pcpu_chunk {int map_alloc;/* # of map entries allocated */ int *map;/* allocation map */ bool immutable;/* no [de]population allowed */ -struct page *page[];/* #cpus * UNIT_PAGES */ +struct page **page;/* points to page array */ +struct page *page_ar[];/* #cpus * UNIT_PAGES */ };static int pcpu_unit_pages__read_mostly ;
@@ -696,6 +697,7 @@ static struct pcpu_chunk *alloc_pcpu_chunk(void) PCPU_DFL_MAP_ALLOC * sizeof(chunk->map[0])); chunk->map_alloc =PCPU_DFL_MAP_ALLOC ; chunk->map[chunk->map_used++] = pcpu_unit_size; + chunk->page = chunk->page_ar; chunk->vm =get_vm_area (pcpu_chunk_size,GFP_KERNEL );if (!chunk->vm) {
@@ -918,6 +920,7 @@ size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn, schunk->vm = &first_vm; schunk->map = smap; schunk->map_alloc =ARRAY_SIZE (smap); + schunk->page = schunk->page_ar; schunk->free_size = dyn_size; schunk->contig_hint = schunk->free_size;
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 7/8 percpu, module: implement reserved allocation and use it for module percpu variables
In reply to Tejun HeoImpact: add reserved allocation functionality and use it for module
percpu variables
This patch implements reserved allocation from the first chunk. When setting up the first chunk, arch can ask to set aside certain number of bytes right after the core static area which is available only through a separate reserved allocator. This will be used primarily for module static percpu variables on architectures with limited relocation range to ensure that the module perpcu symbols are inside the relocatable range.
If reserved area is requested, the first chunk becomes reserved and isn't available for regular allocation. If the first chunk also includes piggy-back dynamic allocation area, a separate chunk mapping the same region is created to serve dynamic allocation. The first one is called static first chunk and the second dynamic first chunk. Although they share the page map, their different area map initializations guarantee they serve disjoint areas according to their purposes.
If arch doesn't setup reserved area, reserved allocation is handled like any other allocation.
--- arch/x86/kernel/setup_percpu.c | 8 +- include/linux/percpu.h | 10 ++- kernel/module.c | 2 +- mm/percpu.c | 153 ++++++++++++++++++++++++++++++++++----- 4 files changed, 144 insertions(+), 29 deletions(-)
arch/x86/kernel/setup_percpu.c +4 -4
@@ -217,7 +217,7 @@ proceed:pr_info ("PERCPU: Remapped at %p with large pages, static data " "%zu bytes\n" , vm.addr, static_size); - ret =pcpu_setup_first_chunk (pcpur_get_page, static_size,PMD_SIZE , + ret =pcpu_setup_first_chunk (pcpur_get_page, static_size,0 ,PMD_SIZE , pcpur_size - static_size, vm.addr,NULL ); goto out_free_ar;
@@ -297,7 +297,7 @@ static ssize_t __init setup_pcpu_embed(size_t static_size)pr_info ("PERCPU: Embedded %zu pages at %p, static data %zu bytes\n" , pcpue_size >>PAGE_SHIFT , pcpue_ptr, static_size); -return pcpu_setup_first_chunk (pcpue_get_page, static_size, +return pcpu_setup_first_chunk (pcpue_get_page, static_size,0 , pcpue_unit_size, dyn_size, pcpue_ptr,NULL ); }
@@ -356,8 +356,8 @@ static ssize_t __init setup_pcpu_4k(size_t static_size)pr_info ("PERCPU: Allocated %d 4k pages, static data %zu bytes\n" , pcpu4k_nr_static_pages, static_size); - ret =pcpu_setup_first_chunk (pcpu4k_get_page, static_size,-1 ,-1 ,NULL , - pcpu4k_populate_pte); + ret =pcpu_setup_first_chunk (pcpu4k_get_page, static_size,0 ,-1 ,-1 , +NULL , pcpu4k_populate_pte); goto out_free_ar; enomem:
include/linux/percpu.h +6 -4
@@ -117,10 +117,10 @@ typedef struct page * (*pcpu_get_page_fn_t)(unsigned int cpu, int pageno);typedef void (*pcpu_populate_pte_fn_t )(unsigned long addr);extern size_t __init pcpu_setup_first_chunk (pcpu_get_page_fn_t get_page_fn, -size_t static_size, -ssize_t unit_size,ssize_t dyn_size, -void *base_addr, -pcpu_populate_pte_fn_t populate_pte_fn); +size_t static_size,size_t reserved_size, +ssize_t unit_size,ssize_t dyn_size, +void *base_addr, +pcpu_populate_pte_fn_t populate_pte_fn);/* * Use this to get to a cpu's version of the per-cpu object
@@ -129,6 +129,8 @@ extern size_t __init pcpu_setup_first_chunk(pcpu_get_page_fn_t get_page_fn,*/ #define per_cpu_ptr (ptr, cpu) SHIFT_PERCPU_PTR((ptr), per_cpu_offset((cpu))) +extern void *__alloc_reserved_percpu (size_t size,size_t align); +#else /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */ struct percpu_data {
kernel/module.c +1 -1
@@ -381,7 +381,7 @@ static void *percpu_modalloc(unsigned long size, unsigned long align, align =PAGE_SIZE ; } - ptr =__alloc_percpu (size, align); + ptr =__alloc_reserved_percpu (size, align);if (!ptr)printk (KERN_WARNING "Could not allocate %lu bytes percpu data\n" , size);
mm/percpu.c +133 -20
291 lines not on this page · raw message
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 8/8 x86, percpu: setup reserved percpu area for x86_64
In reply to Tejun HeoImpact: fix relocation overflow during module load
x86_64 uses 32bit relocations for symbol access and static percpu symbols whether in core or modules must be inside 2GB of the percpu segement base which the dynamic percpu allocator doesn't guarantee. This patch makes x86_64 reserve PERCPU_MODULE_RESERVE bytes in the first chunk so that module percpu areas are always allocated from the first chunk which is always inside the relocatable range.
This problem exists for any percpu allocator but is easily triggered when using the embedding allocator because the second chunk is located beyond 2GB on it.
This patch also changes the meaning of PERCPU_DYNAMIC_RESERVE such that it only indicates the size of the area to reserve for dynamic allocation as static and dynamic areas can be separate. New PERCPU_DYNAMIC_RESERVED is increased by 4k for both 32 and 64bits as the reserved area separation eats away some allocatable space and having slightly more headroom (currently between 4 and 8k after minimal boot sans module area) makes sense for common case performance.
x86_32 can address anywhere from anywhere and doesn't need reserving.
Mike Galbraith first reported the problem first and bisected it to the embedding percpu allocator commit.
--- arch/x86/kernel/setup_percpu.c | 37 ++++++++++++++++++++++++++++--------- include/linux/percpu.h | 35 ++++++++++++----------------------- 2 files changed, 40 insertions(+), 32 deletions(-)
arch/x86/kernel/setup_percpu.c +28 -9
@@ -42,6 +42,19 @@ unsigned long __per_cpu_offset[NR_CPUS] __read_mostly = { };EXPORT_SYMBOL (__per_cpu_offset); +/* +* On x86_64 symbols referenced from code should be reachable using +* 32bit relocations. Reserve space for static percpu variables in +* modules so that they are always served from the first chunk which +* is located at the percpu segment base. On x86_32, anything can +* address anywhere. No need to reserve space in the first chunk. +*/ +#ifdef CONFIG_X86_64 +#define PERCPU_FIRST_CHUNK_RESERVE PERCPU_MODULE_RESERVE +#else +#define PERCPU_FIRST_CHUNK_RESERVE 0 +#endif +/** * pcpu_need_numa - determine percpu allocation needs to consider NUMA *
@@ -141,7 +154,7 @@ static ssize_t __init setup_pcpu_remap(size_t static_size) {static struct vm_struct vm;pg_data_t *last; -size_t ptrs_size; +size_t ptrs_size, dyn_size;unsigned int cpu;ssize_t ret;
@@ -169,12 +182,14 @@ proceed:* Currently supports only single page. Supporting multiple * pages won't be too difficult if it ever becomes necessary. */ - pcpur_size =PFN_ALIGN (static_size +PERCPU_DYNAMIC_RESERVE ); + pcpur_size =PFN_ALIGN (static_size +PERCPU_MODULE_RESERVE + +PERCPU_DYNAMIC_RESERVE );if (pcpur_size >PMD_SIZE ) {pr_warning ("PERCPU: static data is larger than large page, " "can't use large page\n" );return -EINVAL ; } + dyn_size = pcpur_size - static_size -PERCPU_FIRST_CHUNK_RESERVE ;/* allocate pointer array and alloc large pages */ ptrs_size =PFN_ALIGN (num_possible_cpus () *sizeof (pcpur_ptrs[0 ]));
@@ -217,8 +232,9 @@ proceed:pr_info ("PERCPU: Remapped at %p with large pages, static data " "%zu bytes\n" , vm.addr, static_size); - ret =pcpu_setup_first_chunk (pcpur_get_page, static_size,0 ,PMD_SIZE , - pcpur_size - static_size, vm.addr,NULL ); + ret =pcpu_setup_first_chunk (pcpur_get_page, static_size, +PERCPU_FIRST_CHUNK_RESERVE , +PMD_SIZE , dyn_size, vm.addr,NULL ); goto out_free_ar; enomem:
@@ -276,9 +292,10 @@ static ssize_t __init setup_pcpu_embed(size_t static_size)return -EINVAL ;/* allocate and copy */ - pcpue_size =PFN_ALIGN (static_size +PERCPU_DYNAMIC_RESERVE ); + pcpue_size =PFN_ALIGN (static_size +PERCPU_MODULE_RESERVE + +PERCPU_DYNAMIC_RESERVE ); pcpue_unit_size =max_t (size_t, pcpue_size,PCPU_MIN_UNIT_SIZE ); - dyn_size = pcpue_size - static_size; + dyn_size = pcpue_size - static_size -PERCPU_FIRST_CHUNK_RESERVE ; pcpue_ptr =pcpu_alloc_bootmem (0 ,num_possible_cpus () * pcpue_unit_size,PAGE_SIZE );
@@ -297,7 +314,8 @@ static ssize_t __init setup_pcpu_embed(size_t static_size)pr_info ("PERCPU: Embedded %zu pages at %p, static data %zu bytes\n" , pcpue_size >>PAGE_SHIFT , pcpue_ptr, static_size); -return pcpu_setup_first_chunk (pcpue_get_page, static_size,0 , +return pcpu_setup_first_chunk (pcpue_get_page, static_size, +PERCPU_FIRST_CHUNK_RESERVE , pcpue_unit_size, dyn_size, pcpue_ptr,NULL ); }
@@ -356,8 +374,9 @@ static ssize_t __init setup_pcpu_4k(size_t static_size)pr_info ("PERCPU: Allocated %d 4k pages, static data %zu bytes\n" , pcpu4k_nr_static_pages, static_size); - ret =pcpu_setup_first_chunk (pcpu4k_get_page, static_size,0 ,-1 ,-1 , -NULL , pcpu4k_populate_pte); + ret =pcpu_setup_first_chunk (pcpu4k_get_page, static_size, +PERCPU_FIRST_CHUNK_RESERVE ,-1 ,-1 ,NULL , + pcpu4k_populate_pte); goto out_free_ar; enomem:
include/linux/percpu.h +12 -23
@@ -85,31 +85,20 @@/* * PERCPU_DYNAMIC_RESERVE indicates the amount of free area to piggy -* back on the first chunk if arch is manually allocating and mapping -* it for faster access (as a part of large page mapping for example). -* Note that dynamic percpu allocator covers both static and dynamic -* areas, so these values are bigger than PERCPU_MODULE_RESERVE. +* back on the first chunk for dynamic percpu allocation if arch is +* manually allocating and mapping it for faster access (as a part of +* large page mapping for example). * -* On typical configuration with modules, the following values leave -* about 8k of free space on the first chunk after boot on both x86_32 -* and 64 when module support is enabled. When module support is -* disabled, it's much tighter. +* The following values give between one and two pages of free space +* after typical minimal boot (2-way SMP, single disk and NIC) with +* both defconfig and a distro config on x86_64 and 32. More +* intelligent way to determine this would be nice. */ -#ifndef PERCPU_DYNAMIC_RESERVE -# if BITS_PER_LONG >32 -# ifdef CONFIG_MODULES -# define PERCPU_DYNAMIC_RESERVE (24 << 10) -# else -# define PERCPU_DYNAMIC_RESERVE (16 << 10) -# endif -# else -# ifdef CONFIG_MODULES -# define PERCPU_DYNAMIC_RESERVE (16 << 10) -# else -# define PERCPU_DYNAMIC_RESERVE (8 << 10) -# endif -# endif -#endif /* PERCPU_DYNAMIC_RESERVE */ +#if BITS_PER_LONG >32 +#define PERCPU_DYNAMIC_RESERVE (20 << 10) +#else +#define PERCPU_DYNAMIC_RESERVE (12 << 10) +#endif extern void *pcpu_base_addr;
Signature · 1 line
1.6.0.2
Quoted 11 lines · Tejun Heo
Hello,
Please pull from the following git vector.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.gittj-percpuThis patchset implements reserved percpu alloc from the first chunk and uses it for static perpcu variables in modules on
x86_64. This is necessary becausex86_64expects symbols to be relocatable using 32bit relocations but dynamic percpu allocation does not guarantee it leading to relocation overflow during module load.
Tested, my module loading woes are cured.
-Mike
Quoted 48 lines · Tejun Heo
Hello,
Please pull from the following git vector.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.gittj-percpuThis patchset implements reserved percpu alloc from the first chunk and uses it for static perpcu variables in modules on
x86_64. This is necessary becausex86_64expects symbols to be relocatable using 32bit relocations but dynamic percpu allocation does not guarantee it leading to relocation overflow during module load.This can happen to any of the
x86_64percpu first chunk allocators but the embedding allocator is most likely to trigger it as the first chunk is located in the normal kernel memory allocation area while all the other chunks will be allocated in the vmalloc area which is very far from the first chunk.This type of reservation will be necessary for other architectures too where addressing mode used for percpu acess uses offsets with limited range to access symbols (e.g. ia64 w/ the magic 64k percpu page).
This problem was first reported and bisected by Mike Galbraith[2].
This patchset is against the current x86/core/percpu[1] and contains the following patches.
0001-percpu-clean-up-percpu-constants.patch 0002-percpu-cosmetic-renames-in-pcpu_setup_first_chunk.patch 0003-percpu-improve-first-chunk-initial-area-map-handlin.patch 0004-percpu-use-negative-for-auto-for-pcpu_setup_first_c.patch 0005-x86-make-embedding-percpu-allocator-return-excessiv.patch 0006-percpu-add-an-indirection-ptr-for-chunk-page-map-ac.patch 0007-percpu-module-implement-reserved-allocation-and-us.patch 0008-x86-percpu-setup-reserved-percpu-area-for-x86_64.patch
0001-0004 makes misc updates to percpu in preparation of later changes. 0005 fixes a case where excessive memory can be set aside in the first chunk when using the embedding allocator. 0006-0008 implement reserved allocation and use it for
x86_64.Bryan Wu, can you please review #0001? It contains small change in blackfin. It's mostly trivial but just in case.
Quoted 8 lines
Diffstat follows.
arch/blackfin/include/asm/percpu.h | 10 - arch/x86/kernel/setup_percpu.c | 73 +++++++--- include/linux/percpu.h | 60 +++----- kernel/module.c | 2 mm/percpu.c | 267 ++++++++++++++++++++++++++----------- 5 files changed, 274 insertions(+), 138 deletions(-)
Pulled, thanks Tejun!
Ingo
Quoted 37 lines · Tejun Heo
Hello,
Please pull from the following git vector.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git tj-percpu
This patchset implements reserved percpu alloc from the first chunk and uses it for static perpcu variables in modules on
x86_64. This is necessary becausex86_64expects symbols to be relocatable using 32bit relocations but dynamic percpu allocation does not guarantee it leading to relocation overflow during module load.This can happen to any of the
x86_64percpu first chunk allocators but the embedding allocator is most likely to trigger it as the first chunk is located in the normal kernel memory allocation area while all the other chunks will be allocated in the vmalloc area which is very far from the first chunk.This type of reservation will be necessary for other architectures too where addressing mode used for percpu acess uses offsets with limited range to access symbols (e.g. ia64 w/ the magic 64k percpu page).
This problem was first reported and bisected by Mike Galbraith[2].
This patchset is against the current x86/core/percpu[1] and contains the following patches.
0001-percpu-clean-up-percpu-constants.patch 0002-percpu-cosmetic-renames-in-pcpu_setup_first_chunk.patch 0003-percpu-improve-first-chunk-initial-area-map-handlin.patch 0004-percpu-use-negative-for-auto-for-pcpu_setup_first_c.patch 0005-x86-make-embedding-percpu-allocator-return-excessiv.patch 0006-percpu-add-an-indirection-ptr-for-chunk-page-map-ac.patch 0007-percpu-module-implement-reserved-allocation-and-us.patch 0008-x86-percpu-setup-reserved-percpu-area-for-x86_64.patch
Quoted 8 lines
0001-0004 makes misc updates to percpu in preparation of later changes. 0005 fixes a case where excessive memory can be set aside in the first chunk when using the embedding allocator. 0006-0008 implement reserved allocation and use it for
x86_64.Bryan Wu, can you please review #0001? It contains small change in blackfin. It's mostly trivial but just in case.
Hi Tejun,
Sorry for the delay, it looks fine for me.
-Bryan
Quoted 17 lines
Diffstat follows.
arch/blackfin/include/asm/percpu.h | 10 -
arch/x86/kernel/setup_percpu.c| 73 +++++++---include/linux/percpu.h| 60 +++-----
kernel/module.c | 2
mm/percpu.c | 267 ++++++++++++++++++++++++++-----------5 files changed, 274 insertions(+), 138 deletions(-)Thanks.
--
tejun[1]
f254f3909efaf59ca2d0f408de2d044dace60706
[2] http://thread.gmane.org/gmane.linux.kernel/801799