percpu: convert most archs to dynamic percpu, take#4
Subsystems (30)
Tejun Heo
maintainerPATCH 1/1 percpu: convert most archs to dynamic percpu, take#4
Hello,
As per David Miller's request, I trimmed most arch maintainers from cc list and just added the arch mailing list.
This is the fourth take of percpu-convert-most-archs-to-dynamic-percpu patchset. Changes from the last take[L] are
* Linus nacked enforcing global definitions for all percpu variables. Instead, revive the dummy guard variable tricks but apply it only to the archs which require it (s390 and alpha) and when the corresponding debug option is enabled.
* Rebased on top of percpu#for-tip
Weak definitions use two guard variables. One to enforce scope and the other to guarantee symbol uniqueness, which add the following two restrictions to percpu variables.
1. percpu symbols must be unique whether static or not
2. percpu variables can't be defined inside a function
To ease checking that the above restrictions are followed in generic code, config option - DEBUG_FORCE_WEAK_PER_CPU - is added which forces the use of weak definitions.
For cases where weak definition is unnecessary and if the debug option is not set, percpu variables are declared and defined in the same way as before this patch.
This patchset contains the following ten patches.
0001-percpu-use-dynamic-percpu-allocator-as-the-default.patch 0002-linker-script-throw-away-.discard-section.patch 0003-CRIS-Change-DEFINE_PER_CPU-of-current_pgd-to-be-non.patch 0004-percpu-cleanup-percpu-array-definitions.patch 0005-percpu-use-DEFINE_PER_CPU_SHARED_ALIGNED.patch 0006-percpu-clean-up-percpu-variable-definitions.patch 0007-percpu-implement-optional-weak-percpu-definitions.patch 0008-alpha-kill-unnecessary-__used-attribute-in-PER_CPU_.patch 0009-alpha-switch-to-dynamic-percpu-allocator.patch 0010-s390-switch-to-dynamic-percpu-allocator.patch
0001 converts archs which used to use the default percpu init code to dynamic percpu allocator. Embedding first chunk allocator is used such that the end result is basically the same. Kyle McMartin reported that default allocator change breaks parisc. The problem is still under investigation and he is okay with pushing this patchset forward and fixing parisc later.
0002-0006 prepares for weak percpu variable definition patch. 0007 implements weak percpu variable definitions. 0009-0010 convert s390 and alpha to dynamic percpu allocator using weak percpu variable definitions.
This patchset is on top of the current percpu#for-tip and available in the following git tree.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git convert-most-archs
Unless the whole approach gets nacked. I'll keep the tree incremental from this point on and will publish the tree through linux-next.
This patchset contains the following changes.
Makefile | 2 arch/alpha/include/asm/percpu.h | 100 ++------------------------------- arch/alpha/include/asm/tlbflush.h | 1 arch/alpha/kernel/vmlinux.lds.S | 1 arch/arm/kernel/vmlinux.lds.S | 1 arch/avr32/kernel/vmlinux.lds.S | 1 arch/blackfin/kernel/vmlinux.lds.S | 1 arch/blackfin/mm/sram-alloc.c | 6 - arch/cris/include/asm/mmu_context.h | 3 arch/cris/kernel/vmlinux.lds.S | 1 arch/cris/mm/fault.c | 2 arch/frv/kernel/vmlinux.lds.S | 2 arch/h8300/kernel/vmlinux.lds.S | 1 arch/ia64/Kconfig | 3 arch/ia64/kernel/smp.c | 3 arch/ia64/kernel/vmlinux.lds.S | 1 arch/ia64/sn/kernel/setup.c | 2 arch/m32r/kernel/vmlinux.lds.S | 1 arch/m68k/kernel/vmlinux-std.lds | 1 arch/m68k/kernel/vmlinux-sun3.lds | 1 arch/m68knommu/kernel/vmlinux.lds.S | 1 arch/microblaze/kernel/vmlinux.lds.S | 2 arch/mips/kernel/vmlinux.lds.S | 1 arch/mn10300/kernel/vmlinux.lds.S | 1 arch/parisc/kernel/vmlinux.lds.S | 1 arch/powerpc/Kconfig | 3 arch/powerpc/kernel/vmlinux.lds.S | 1 arch/powerpc/mm/stab.c | 2 arch/powerpc/platforms/ps3/smp.c | 2 arch/s390/include/asm/percpu.h | 32 ++-------- arch/s390/kernel/vmlinux.lds.S | 1 arch/sh/kernel/vmlinux.lds.S | 1 arch/sparc/Kconfig | 3 arch/sparc/kernel/vmlinux.lds.S | 1 arch/um/kernel/dyn.lds.S | 2 arch/um/kernel/uml.lds.S | 2 arch/x86/Kconfig | 3 arch/x86/kernel/cpu/cpu_debug.c | 4 - arch/x86/kernel/cpu/mcheck/mce.c | 8 +- arch/x86/kernel/cpu/mcheck/mce_amd.c | 2 arch/x86/kernel/cpu/perf_counter.c | 14 ++-- arch/xtensa/kernel/vmlinux.lds.S | 1 block/as-iosched.c | 10 +-- block/cfq-iosched.c | 10 +-- drivers/cpufreq/cpufreq_conservative.c | 12 +-- drivers/cpufreq/cpufreq_ondemand.c | 15 ++-- drivers/xen/events.c | 13 ++-- include/asm-generic/vmlinux.lds.h | 8 ++ include/linux/percpu-defs.h | 65 ++++++++++++++++++--- include/linux/percpu.h | 12 ++- init/main.c | 24 ------- kernel/module.c | 6 - kernel/perf_counter.c | 6 - kernel/sched.c | 4 - kernel/trace/trace_events.c | 6 - lib/Kconfig.debug | 15 ++++ mm/Makefile | 2 mm/allocpercpu.c | 28 +++++++++ mm/kmemleak-test.c | 6 - mm/page-writeback.c | 5 - mm/percpu.c | 40 ++++++++++++- mm/quicklist.c | 2 mm/slub.c | 4 - net/ipv4/syncookies.c | 5 - net/ipv6/syncookies.c | 5 - net/rds/ib_stats.c | 2 net/rds/iw_stats.c | 2 net/rds/page.c | 2 scripts/module-common.lds | 8 ++ 69 files changed, 303 insertions(+), 236 deletions(-)
Thanks.
--
tejun
[L] http://thread.gmane.org/gmane.linux.kernel.cross-arch/3818
(some patches missing probably due to too long cc list)
Tejun Heo
maintainerPATCH 1/10 percpu: use dynamic percpu allocator as the default percpu allocator
This patch makes most !CONFIG_HAVE_SETUP_PER_CPU_AREA archs use dynamic percpu allocator. The first chunk is allocated using embedding helper and 8k is reserved for modules. This ensures that the new allocator behaves almost identically to the original allocator as long as static percpu variables are concerned, so it shouldn't introduce much breakage.
s390 and alpha use custom SHIFT_PERCPU_PTR() to work around addressing range limit the addressing model imposes. Unfortunately, this breaks if the address is specified using a variable, so for now, the two archs aren't converted.
The following architectures are affected by this change.
* sh
* arm
* cris
* mips
* sparc(32)
* blackfin
* avr32
* parisc (broken, under investigation)
* m32r
* powerpc(32)
As this change makes the dynamic allocator the default one, CONFIG_HAVE_DYNAMIC_PER_CPU_AREA is replaced with its invert - CONFIG_HAVE_LEGACY_PER_CPU_AREA, which is added to yet-to-be converted archs. These archs implement their own setup_per_cpu_areas() and the conversion is not trivial.
* powerpc(64)
* sparc(64)
* ia64
* alpha
* s390
Boot and batch alloc/free tests on x86_32 with debug code (x86_32 doesn't use default first chunk initialization). Compile tested on sparc(32), powerpc(32), arm and alpha.
Kyle McMartin reported that this change breaks parisc. The problem is still under investigation and he is okay with pushing this patch forward and fixing parisc later.
[ Impact: use dynamic allocator for most archs w/o custom percpu setup ]
--- arch/alpha/Kconfig | 3 +++ arch/ia64/Kconfig | 3 +++ arch/powerpc/Kconfig | 3 +++ arch/s390/Kconfig | 3 +++ arch/sparc/Kconfig | 3 +++ arch/x86/Kconfig | 3 --- include/linux/percpu.h | 12 +++++++++--- init/main.c | 24 ------------------------ kernel/module.c | 6 +++--- mm/Makefile | 2 +- mm/allocpercpu.c | 28 ++++++++++++++++++++++++++++ mm/percpu.c | 40 +++++++++++++++++++++++++++++++++++++++- 12 files changed, 95 insertions(+), 35 deletions(-)
arch/alpha/Kconfig +3 -0
@@ -70,6 +70,9 @@ config AUTO_IRQ_AFFINITY depends on SMP default y +config HAVE_LEGACY_PER_CPU_AREA + def_bool y + source "init/Kconfig" source "kernel/Kconfig.freezer"
arch/ia64/Kconfig +3 -0
@@ -89,6 +89,9 @@ config GENERIC_TIME_VSYSCALL bool default y +config HAVE_LEGACY_PER_CPU_AREA + def_bool y + config HAVE_SETUP_PER_CPU_AREA def_bool y
arch/powerpc/Kconfig +3 -0
@@ -46,6 +46,9 @@ config GENERIC_HARDIRQS_NO__DO_IRQ bool default y +config HAVE_LEGACY_PER_CPU_AREA + def_bool PPC64 + config HAVE_SETUP_PER_CPU_AREA def_bool PPC64
arch/s390/Kconfig +3 -0
@@ -75,6 +75,9 @@ config VIRT_CPU_ACCOUNTING config ARCH_SUPPORTS_DEBUG_PAGEALLOC def_bool y +config HAVE_LEGACY_PER_CPU_AREA + def_bool y + mainmenu "Linux Kernel Configuration" config S390
arch/sparc/Kconfig +3 -0
@@ -92,6 +92,9 @@ config AUDIT_ARCH bool default y +config HAVE_LEGACY_PER_CPU_AREA + def_bool y if SPARC64 + config HAVE_SETUP_PER_CPU_AREA def_bool y if SPARC64
arch/x86/Kconfig +0 -3
@@ -149,9 +149,6 @@ config ARCH_HAS_CACHE_LINE_SIZE config HAVE_SETUP_PER_CPU_AREA def_bool y -config HAVE_DYNAMIC_PER_CPU_AREA - def_bool y - config HAVE_CPUMASK_OF_CPU_MAP def_bool X86_64_SMP
include/linux/percpu.h +9 -3
@@ -34,7 +34,7 @@#ifdef CONFIG_SMP -#ifdef CONFIG_HAVE_DYNAMIC_PER_CPU_AREA +#ifndef CONFIG_HAVE_LEGACY_PER_CPU_AREA /* minimum unit size, also is the maximum supported allocation size */ #define PCPU_MIN_UNIT_SIZE PFN_ALIGN(64 << 10)
@@ -80,7 +80,7 @@ extern ssize_t __init pcpu_embed_first_chunk( extern void *__alloc_reserved_percpu(size_t size, size_t align); -#else /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */ +#else /* CONFIG_HAVE_LEGACY_PER_CPU_AREA */ struct percpu_data { void *ptrs[1];
@@ -99,11 +99,15 @@ struct percpu_data { (__typeof__ (ptr ))__p->ptrs[(cpu)]; \ }) -#endif /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */ +#endif /* CONFIG_HAVE_LEGACY_PER_CPU_AREA */ extern void *__alloc_percpu (size_t size,size_t align);extern void free_percpu (void *__pdata); +#ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA +extern void __init setup_per_cpu_areas (void ); +#endif +#else /* CONFIG_SMP */ #define per_cpu_ptr (ptr, cpu) ({ (void)(cpu); (ptr); })
@@ -124,6 +128,8 @@ static inline void free_percpu(void *p) kfree(p); } +static inline void __init setup_per_cpu_areas (void ) { } +#endif /* CONFIG_SMP */ #define alloc_percpu (type) (type *)__alloc_percpu(sizeof(type), \
init/main.c +0 -24
@@ -357,7 +357,6 @@ static void __init smp_init(void)#define smp_init () do { } while (0)#endif -static inline void setup_per_cpu_areas (void ) { }static inline void setup_nr_cpu_ids (void ) { }static inline void smp_prepare_cpus (unsigned int maxcpus) { }
@@ -378,29 +377,6 @@ static void __init setup_nr_cpu_ids(void) nr_cpu_ids =find_last_bit (cpumask_bits (cpu_possible_mask),NR_CPUS ) +1 ; } -#ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA -unsigned long __per_cpu_offset[NR_CPUS ]__read_mostly ; - -EXPORT_SYMBOL (__per_cpu_offset); - -static void __init setup_per_cpu_areas (void ) -{ -unsigned long size, i; -char *ptr; -unsigned long nr_possible_cpus =num_possible_cpus (); - -/* Copy section for each CPU (we discard the original) */ - size =ALIGN (PERCPU_ENOUGH_ROOM ,PAGE_SIZE ); - ptr =alloc_bootmem_pages (size * nr_possible_cpus); - -for_each_possible_cpu (i) { - __per_cpu_offset[i] = ptr - __per_cpu_start; -memcpy (ptr, __per_cpu_start, __per_cpu_end - __per_cpu_start); - ptr += size; - } -} -#endif /* CONFIG_HAVE_SETUP_PER_CPU_AREA */ -/* Called by boot processor to activate the rest. */ static void __init smp_init (void ) {
kernel/module.c +3 -3
@@ -364,7 +364,7 @@ EXPORT_SYMBOL_GPL(find_module);#ifdef CONFIG_SMP -#ifdef CONFIG_HAVE_DYNAMIC_PER_CPU_AREA +#ifndef CONFIG_HAVE_LEGACY_PER_CPU_AREA static void *percpu_modalloc (unsigned long size,unsigned long align,const char *name)
@@ -389,7 +389,7 @@ static void percpu_modfree(void *freeme) free_percpu(freeme); } -#else /* ... !CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */ +#else /* ... CONFIG_HAVE_LEGACY_PER_CPU_AREA */ /* Number of blocks used and allocated. */ static unsigned int pcpu_num_used, pcpu_num_allocated;
@@ -535,7 +535,7 @@ static int percpu_modinit(void) }__initcall (percpu_modinit); -#endif /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */ +#endif /* CONFIG_HAVE_LEGACY_PER_CPU_AREA */ static unsigned int find_pcpusec(Elf_Ehdr *hdr, Elf_Shdr *sechdrs,
mm/Makefile +1 -1
@@ -33,7 +33,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o obj-$(CONFIG_MEMORY_HOTPLUG) += memory_hotplug.o obj-$(CONFIG_FS_XIP) += filemap_xip.o obj-$(CONFIG_MIGRATION) += migrate.o -ifdef CONFIG_HAVE_DYNAMIC_PER_CPU_AREA +ifndef CONFIG_HAVE_LEGACY_PER_CPU_AREA obj-$(CONFIG_SMP) += percpu.o else obj-$(CONFIG_SMP) += allocpercpu.o
mm/allocpercpu.c +28 -0
@@ -5,6 +5,8 @@*/ #include <linux/mm.h> #include <linux/module.h> +#include <linux/bootmem.h> +#include <asm/sections.h> #ifndef cache_line_size#define cache_line_size () L1_CACHE_BYTES
@@ -147,3 +149,29 @@ void free_percpu(void *__pdata)kfree (__percpu_disguise (__pdata)); }EXPORT_SYMBOL_GPL (free_percpu); + +/* +* Generic percpu area setup. +*/ +#ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA +unsigned long __per_cpu_offset[NR_CPUS ]__read_mostly ; + +EXPORT_SYMBOL (__per_cpu_offset); + +void __init setup_per_cpu_areas (void ) +{ +unsigned long size, i; +char *ptr; +unsigned long nr_possible_cpus =num_possible_cpus (); + +/* Copy section for each CPU (we discard the original) */ + size =ALIGN (PERCPU_ENOUGH_ROOM ,PAGE_SIZE ); + ptr =alloc_bootmem_pages (size * nr_possible_cpus); + +for_each_possible_cpu (i) { + __per_cpu_offset[i] = ptr - __per_cpu_start; +memcpy (ptr, __per_cpu_start, __per_cpu_end - __per_cpu_start); + ptr += size; + } +} +#endif /* CONFIG_HAVE_SETUP_PER_CPU_AREA */
mm/percpu.c +39 -1
@@ -43,7 +43,7 @@* * To use this allocator, arch code should do the followings. * -* - define CONFIG_HAVE_DYNAMIC_PER_CPU_AREA +* - drop CONFIG_HAVE_LEGACY_PER_CPU_AREA * * - define __addr_to_pcpu_ptr() and __pcpu_ptr_to_addr() to translate * regular address to percpu pointer and back if they need to be
@@ -1275,3 +1275,41 @@ ssize_t __init pcpu_embed_first_chunk(size_t static_size, size_t reserved_size, reserved_size, dyn_size, pcpue_unit_size, pcpue_ptr, NULL); } + +/* +* Generic percpu area setup. +* +* The embedding helper is used because its behavior closely resembles +* the original non-dynamic generic percpu area setup. This is +* important because many archs have addressing restrictions and might +* fail if the percpu area is located far away from the previous +* location. As an added bonus, in non-NUMA cases, embedding is +* generally a good idea TLB-wise because percpu area can piggy back +* on the physical linear memory mapping which uses large page +* mappings on applicable archs. +*/ +#ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA +unsigned long __per_cpu_offset[NR_CPUS ]__read_mostly ; +EXPORT_SYMBOL (__per_cpu_offset); + +void __init setup_per_cpu_areas (void ) +{ +size_t static_size = __per_cpu_end - __per_cpu_start; +ssize_t unit_size; +unsigned long delta; +unsigned int cpu; + +/* +* Always reserve area for module percpu variables. That's +* what the legacy allocator did. +*/ + unit_size =pcpu_embed_first_chunk (static_size,PERCPU_MODULE_RESERVE , +PERCPU_DYNAMIC_RESERVE ,-1 ); +if (unit_size <0 ) +panic ("Failed to initialized percpu areas." ); + + delta = (unsigned long )pcpu_base_addr - (unsigned long )__per_cpu_start; +for_each_possible_cpu (cpu) + __per_cpu_offset[cpu] = delta + cpu * unit_size; +} +#endif /* CONFIG_HAVE_SETUP_PER_CPU_AREA */
Signature · 1 line
1.6.0.2
James Bottomley
maintainerRe: [PATCH 01/10] percpu: use dynamic percpu allocator as the default percpu allocator
Quoted 14 lines · Tejun Heo
This patch makes most !CONFIG_HAVE_SETUP_PER_CPU_AREA archs use dynamic percpu allocator. The first chunk is allocated using embedding helper and 8k is reserved for modules. This ensures that the new allocator behaves almost identically to the original allocator as long as static percpu variables are concerned, so it shouldn't introduce much breakage.
s390 and alpha use custom
SHIFT_PERCPU_PTR()to work around addressing range limit the addressing model imposes. Unfortunately, this breaks if the address is specified using a variable, so for now, the two archs aren't converted.The following architectures are affected by this change.
Quoted 8 lines
* sh
* arm
* cris
* mips
* sparc(32)
* blackfin
* avr32
* parisc (broken, under investigation)
OK, I thought I'd investigate this. Unfortunately, your git tree doesn't build when pulled into head ... could you update?
mm/percpu.c: In function 'setup_per_cpu_areas':mm/percpu.c:2007: error: too many arguments to function 'pcpu_embed_first_chunk'
James
Tejun Heo
maintainerRe: [PATCH 01/10] percpu: use dynamic percpu allocator as the default percpu allocator
Quoted 24 lines · James Bottomley
Quoted 14 lines · Tejun Heo
This patch makes most !CONFIG_HAVE_SETUP_PER_CPU_AREA archs use dynamic percpu allocator. The first chunk is allocated using embedding helper and 8k is reserved for modules. This ensures that the new allocator behaves almost identically to the original allocator as long as static percpu variables are concerned, so it shouldn't introduce much breakage.
s390 and alpha use custom
SHIFT_PERCPU_PTR()to work around addressing range limit the addressing model imposes. Unfortunately, this breaks if the address is specified using a variable, so for now, the two archs aren't converted.The following architectures are affected by this change.
Quoted 8 lines
* sh
* arm
* cris
* mips
* sparc(32)
* blackfin
* avr32
* parisc (broken, under investigation)
Quoted 5 lines
OK, I thought I'd investigate this. Unfortunately, your git tree doesn't build when pulled into head ... could you update?
mm/percpu.c: In function 'setup_per_cpu_areas':mm/percpu.c:2007: error: too many arguments to function 'pcpu_embed_first_chunk'
Hmm... strange. I don't see such failures. Can you please try the following git tree?
git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git for-james
The contains all posted percpu patches and pulls in the current linus#master (rc1).
Thanks.
Signature · 1 line
tejun
James Bottomley
maintainerRe: [PATCH 01/10] percpu: use dynamic percpu allocator as the default percpu allocator
Quoted 31 lines · Tejun Heo
Quoted 24 lines · James Bottomley
Quoted 14 lines · Tejun Heo
This patch makes most !CONFIG_HAVE_SETUP_PER_CPU_AREA archs use dynamic percpu allocator. The first chunk is allocated using embedding helper and 8k is reserved for modules. This ensures that the new allocator behaves almost identically to the original allocator as long as static percpu variables are concerned, so it shouldn't introduce much breakage.
s390 and alpha use custom
SHIFT_PERCPU_PTR()to work around addressing range limit the addressing model imposes. Unfortunately, this breaks if the address is specified using a variable, so for now, the two archs aren't converted.The following architectures are affected by this change.
Quoted 8 lines
* sh
* arm
* cris
* mips
* sparc(32)
* blackfin
* avr32
* parisc (broken, under investigation)Quoted 5 lines
OK, I thought I'd investigate this. Unfortunately, your git tree doesn't build when pulled into head ... could you update?
mm/percpu.c: In function 'setup_per_cpu_areas':mm/percpu.c:2007: error: too many arguments to function 'pcpu_embed_first_chunk'
Quoted 7 lines
Hmm... strange. I don't see such failures. Can you please try the following git tree?
git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.gitfor-jamesThe contains all posted percpu patches and pulls in the current linus#master (rc1).
It still fails with the above error.
If you look at your own file:
on line 1456 pcpu_embed_first_chunk is defined with *three* arguments
on line 2006 it's used with *four* arguments
This is what the compiler doesn't like.
James
Tejun Heo
maintainerRe: [PATCH 01/10] percpu: use dynamic percpu allocator as the default percpu allocator
Quoted 11 lines · James Bottomley
It still fails with the above error.
If you look at your own file:
on line 1456
pcpu_embed_first_chunkis defined with *three* argumentson line 2006 it's used with *four* arguments
This is what the compiler doesn't like.
Oops, I was somehow completely forgetting about the default setup_per_cpu_areas(). Fixed. The new commit is a3152acc226ec4859235e521286c9ca45c4b0683.
git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.git for-next
Thanks.
Signature · 1 line
tejun
Re: [PATCH 01/10] percpu: use dynamic percpu allocator as the default percpu allocator
In reply to Tejun HeoQuoted 14 lines · Tejun Heo
This patch makes most !CONFIG_HAVE_SETUP_PER_CPU_AREA archs use dynamic percpu allocator. The first chunk is allocated using embedding helper and 8k is reserved for modules. This ensures that the new allocator behaves almost identically to the original allocator as long as static percpu variables are concerned, so it shouldn't introduce much breakage.
s390 and alpha use custom
SHIFT_PERCPU_PTR()to work around addressing range limit the addressing model imposes. Unfortunately, this breaks if the address is specified using a variable, so for now, the two archs aren't converted.The following architectures are affected by this change.
Quoted 11 lines
* sh
* arm
* cris
* mips
* sparc(32)
* blackfin
* avr32
* parisc (broken, under investigation)
* m32r
* powerpc(32)
No problems on sh.
Tejun Heo
maintainerPATCH 2/10 linker script: throw away .discard section
In reply to Tejun Heox86 throws away .discard section but no other archs do. Also, .discard is not thrown away while linking modules. Make every arch and module linking throw it away. This will be used to define dummy variables for percpu declarations and definitions.
This patch is based on Ivan Kokshaysky's alpha percpu patch.
[ Impact: always throw away everything in .discard ]
--- Makefile | 2 +- arch/alpha/kernel/vmlinux.lds.S | 1 + arch/arm/kernel/vmlinux.lds.S | 1 + arch/avr32/kernel/vmlinux.lds.S | 1 + arch/blackfin/kernel/vmlinux.lds.S | 1 + arch/cris/kernel/vmlinux.lds.S | 1 + arch/frv/kernel/vmlinux.lds.S | 2 ++ arch/h8300/kernel/vmlinux.lds.S | 1 + arch/ia64/kernel/vmlinux.lds.S | 1 + arch/m32r/kernel/vmlinux.lds.S | 1 + arch/m68k/kernel/vmlinux-std.lds | 1 + arch/m68k/kernel/vmlinux-sun3.lds | 1 + arch/m68knommu/kernel/vmlinux.lds.S | 1 + arch/microblaze/kernel/vmlinux.lds.S | 2 ++ arch/mips/kernel/vmlinux.lds.S | 1 + arch/mn10300/kernel/vmlinux.lds.S | 1 + arch/parisc/kernel/vmlinux.lds.S | 1 + arch/powerpc/kernel/vmlinux.lds.S | 1 + arch/s390/kernel/vmlinux.lds.S | 1 + arch/sh/kernel/vmlinux.lds.S | 1 + arch/sparc/kernel/vmlinux.lds.S | 1 + arch/um/kernel/dyn.lds.S | 2 ++ arch/um/kernel/uml.lds.S | 2 ++ arch/xtensa/kernel/vmlinux.lds.S | 1 + include/asm-generic/vmlinux.lds.h | 8 ++++++++ scripts/module-common.lds | 8 ++++++++ 26 files changed, 44 insertions(+), 1 deletions(-) create mode 100644 scripts/module-common.lds
Makefile +1 -1
@@ -327,7 +327,7 @@ CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \ MODFLAGS = -DMODULE CFLAGS_MODULE = $(MODFLAGS) AFLAGS_MODULE = $(MODFLAGS) -LDFLAGS_MODULE = +LDFLAGS_MODULE = -T $(srctree)/scripts/module-common.lds CFLAGS_KERNEL = AFLAGS_KERNEL = CFLAGS_GCOV = -fprofile-arcs -ftest-coverage
arch/alpha/kernel/vmlinux.lds.S +1 -0
@@ -139,6 +139,7 @@ SECTIONS EXIT_TEXT EXIT_DATA *(.exitcall.exit) + *(.discard) } .mdebug 0 : {
arch/arm/kernel/vmlinux.lds.S +1 -0
@@ -82,6 +82,7 @@ SECTIONS EXIT_TEXT EXIT_DATA *(.exitcall.exit) + *(.discard) *(.ARM.exidx.exit.text) *(.ARM.extab.exit.text) #ifndef CONFIG_MMU
arch/avr32/kernel/vmlinux.lds.S +1 -0
@@ -131,6 +131,7 @@ SECTIONS /DISCARD/ : { EXIT_DATA *(.exitcall.exit) + *(.discard) } DWARF_DEBUG
arch/blackfin/kernel/vmlinux.lds.S +1 -0
@@ -280,5 +280,6 @@ SECTIONS /DISCARD/ : { *(.exitcall.exit) + *(.discard) } }
arch/cris/kernel/vmlinux.lds.S +1 -0
@@ -145,6 +145,7 @@ SECTIONS EXIT_TEXT EXIT_DATA *(.exitcall.exit) + *(.discard) } dram_end = dram_start + (CONFIG_ETRAX_DRAM_SIZE - __CONFIG_ETRAX_VMEM_SIZE)*1024*1024;
arch/frv/kernel/vmlinux.lds.S +2 -0
@@ -177,6 +177,8 @@ SECTIONS .debug_ranges 0 : { *(.debug_ranges) } .comment 0 : { *(.comment) } + + /DISCARD/ : { *(.discard) } } __kernel_image_size_no_bss = __bss_start - __kernel_image_start;
arch/h8300/kernel/vmlinux.lds.S +1 -0
@@ -154,6 +154,7 @@ SECTIONS } /DISCARD/ : { *(.exitcall.exit) + *(.discard) } .romfs : {
arch/ia64/kernel/vmlinux.lds.S +1 -0
@@ -29,6 +29,7 @@ SECTIONS EXIT_TEXT EXIT_DATA *(.exitcall.exit) + *(.discard) *(.IA_64.unwind.exit.text) *(.IA_64.unwind_info.exit.text) }
arch/m32r/kernel/vmlinux.lds.S +1 -0
@@ -125,6 +125,7 @@ SECTIONS EXIT_TEXT EXIT_DATA *(.exitcall.exit) + *(.discard) } /* Stabs debugging sections. */
arch/m68k/kernel/vmlinux-std.lds +1 -0
@@ -87,6 +87,7 @@ SECTIONS EXIT_TEXT EXIT_DATA *(.exitcall.exit) + *(.discard) } /* Stabs debugging sections. */
arch/m68k/kernel/vmlinux-sun3.lds +1 -0
@@ -82,6 +82,7 @@ __init_begin = .; EXIT_TEXT EXIT_DATA *(.exitcall.exit) + *(.discard) } .crap : {
arch/m68knommu/kernel/vmlinux.lds.S +1 -0
@@ -188,6 +188,7 @@ SECTIONS { EXIT_TEXT EXIT_DATA *(.exitcall.exit) + *(.discard) } .bss : {
arch/microblaze/kernel/vmlinux.lds.S +2 -0
@@ -162,4 +162,6 @@ SECTIONS { } . = ALIGN(4096); _end = .; + + /DISCARD/ : { *(.discard) } }
arch/mips/kernel/vmlinux.lds.S +1 -0
@@ -179,6 +179,7 @@ SECTIONS /* Sections to be discarded */ /DISCARD/ : { *(.exitcall.exit) + *(.discard) /* ABI crap starts here */ *(.MIPS.options)
arch/mn10300/kernel/vmlinux.lds.S +1 -0
@@ -146,6 +146,7 @@ SECTIONS /* Sections to be discarded */ /DISCARD/ : { *(.exitcall.exit) + *(.discard) } STABS_DEBUG
arch/parisc/kernel/vmlinux.lds.S +1 -0
@@ -240,6 +240,7 @@ SECTIONS /* Sections to be discarded */ /DISCARD/ : { *(.exitcall.exit) + *(.discard) #ifdef CONFIG_64BIT /* temporary hack until binutils is fixed to not emit these * for static binaries
arch/powerpc/kernel/vmlinux.lds.S +1 -0
@@ -40,6 +40,7 @@ SECTIONS /* Sections to be discarded. */ /DISCARD/ : { *(.exitcall.exit) + *(.discard) EXIT_DATA }
arch/s390/kernel/vmlinux.lds.S +1 -0
@@ -161,6 +161,7 @@ SECTIONS /DISCARD/ : { EXIT_DATA *(.exitcall.exit) + *(.discard) } /* Debugging sections. */
arch/sh/kernel/vmlinux.lds.S +1 -0
@@ -171,6 +171,7 @@ SECTIONS */ /DISCARD/ : { *(.exitcall.exit) + *(.discard) } STABS_DEBUG
arch/sparc/kernel/vmlinux.lds.S +1 -0
@@ -175,6 +175,7 @@ SECTIONS EXIT_TEXT EXIT_DATA *(.exitcall.exit) + *(.discard) } STABS_DEBUG
arch/um/kernel/dyn.lds.S +2 -0
@@ -156,4 +156,6 @@ SECTIONS STABS_DEBUG DWARF_DEBUG + + /DISCARD/ : { *(.discard) } }
arch/um/kernel/uml.lds.S +2 -0
@@ -100,4 +100,6 @@ SECTIONS STABS_DEBUG DWARF_DEBUG + + /DISCARD/ : { *(.discard) } }
arch/xtensa/kernel/vmlinux.lds.S +1 -0
@@ -287,6 +287,7 @@ SECTIONS EXIT_TEXT EXIT_DATA *(.exitcall.exit) + *(.discard) } .xt.lit : { *(.xt.lit) }
include/asm-generic/vmlinux.lds.h +8 -0
@@ -628,6 +628,14 @@#define INITRAMFS #endif +#define DISCARDS \ + /DISCARD/ : { \ + EXIT_TEXT \ + EXIT_DATA \ + *(.exitcall.exit) \ + *(.discard) \ + } +/** * PERCPU_VADDR - define output section for percpu area * @vaddr: explicit base address (optional)
scripts/module-common.lds new file +8 -0
@@ -0,0 +1,8 @@ +/* + * Common module linker script, always used when linking a module. + * Archs are free to supply their own linker scripts. ld will + * combine them automatically. + */ +SECTIONS { + /DISCARD/ : { *(.discard) } +}
Signature · 1 line
1.6.0.2
RE: [PATCH 02/10] linker script: throw away .discard section
Quoted 4 lines
x86 throws away .discard section but no other archs do. Also, .discard is not thrown away while linking modules. Make every arch and module linking throw it away. This will be used to define dummy variables for percpu declarations and definitions.
Seems harmless on ia64. Kernel still builds, boots and loads modules.
-Tony
Tejun Heo
maintainerPATCH 3/10 CRIS: Change DEFINE_PER_CPU of current_pgd to be non volatile.
In reply to Tejun HeoFrom: Jesper Nilsson <[email protected]>
The DEFINE_PER_CPU of current_pgd was on CRIS defined using volatile, which is not needed. Remove volatile.
Tested on an ARTPEC-3 (CRISv32) board.
tj: extern DEFINE_PER_CPU() replaced with DECLARE_PER_CPU()
[ Impact: code cleanup ]
--- arch/cris/include/asm/mmu_context.h | 3 ++- arch/cris/mm/fault.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
arch/cris/include/asm/mmu_context.h +2 -1
@@ -17,7 +17,8 @@ extern void switch_mm(struct mm_struct *prev, struct mm_struct *next,* registers like cr3 on the i386 */ -extern volatile DEFINE_PER_CPU(pgd_t *,current_pgd); /* defined in arch/cris/mm/fault.c */ +/* defined in arch/cris/mm/fault.c */ +DECLARE_PER_CPU(pgd_t *, current_pgd);static inline void enter_lazy_tlb (struct mm_struct *mm,struct task_struct *tsk) {
arch/cris/mm/fault.c +1 -1
@@ -29,7 +29,7 @@ extern void die_if_kernel(const char *, struct pt_regs *, long);/* current active page directory */ -volatile DEFINE_PER_CPU(pgd_t *,current_pgd); +DEFINE_PER_CPU(pgd_t *, current_pgd);unsigned long cris_signal_return_page;/*
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 4/10 percpu: cleanup percpu array definitions
In reply to Tejun HeoCurrently, the following three different ways to define percpu arrays are in use.
1. DEFINE_PER_CPU(elem_type[array_len], array_name);
2. DEFINE_PER_CPU(elem_type, array_name[array_len]);
3. DEFINE_PER_CPU(elem_type, array_name)[array_len];
Unify to #1 which correctly separates the roles of the two parameters and thus allows more flexibility in the way percpu variables are defined.
[ Impact: cleanup ]
--- arch/ia64/kernel/smp.c | 2 +- arch/ia64/sn/kernel/setup.c | 2 +- arch/powerpc/mm/stab.c | 2 +- arch/powerpc/platforms/ps3/smp.c | 2 +- arch/x86/kernel/cpu/cpu_debug.c | 4 ++-- arch/x86/kernel/cpu/mcheck/mce_amd.c | 2 +- arch/x86/kernel/cpu/perf_counter.c | 2 +- drivers/xen/events.c | 4 ++-- mm/quicklist.c | 2 +- mm/slub.c | 4 ++-- net/ipv4/syncookies.c | 2 +- net/ipv6/syncookies.c | 2 +- 12 files changed, 15 insertions(+), 15 deletions(-)
arch/ia64/kernel/smp.c +1 -1
@@ -58,7 +58,7 @@ static struct local_tlb_flush_counts {unsigned int count; } __attribute__((__aligned__(32))) local_tlb_flush_counts[NR_CPUS]; -static DEFINE_PER_CPU(unsigned short, shadow_flush_counts[NR_CPUS]) ____cacheline_aligned; +static DEFINE_PER_CPU(unsigned short [NR_CPUS], shadow_flush_counts) ____cacheline_aligned;#define IPI_CALL_FUNC 0#define IPI_CPU_STOP 1
arch/ia64/sn/kernel/setup.c +1 -1
@@ -71,7 +71,7 @@ EXPORT_SYMBOL(sn_rtc_cycles_per_second); DEFINE_PER_CPU(struct sn_hub_info_s, __sn_hub_info);EXPORT_PER_CPU_SYMBOL (__sn_hub_info); -DEFINE_PER_CPU (short, __sn_cnodeid_to_nasid[MAX_COMPACT_NODES ]); +DEFINE_PER_CPU (short [MAX_COMPACT_NODES ], __sn_cnodeid_to_nasid);EXPORT_PER_CPU_SYMBOL (__sn_cnodeid_to_nasid); DEFINE_PER_CPU(struct nodepda_s *, __sn_nodepda);
arch/powerpc/mm/stab.c +1 -1
@@ -31,7 +31,7 @@ struct stab_entry {#define NR_STAB_CACHE_ENTRIES 8 static DEFINE_PER_CPU(long, stab_cache_ptr); -static DEFINE_PER_CPU(long, stab_cache[NR_STAB_CACHE_ENTRIES]); +static DEFINE_PER_CPU(long [NR_STAB_CACHE_ENTRIES], stab_cache);/* * Create a segment table entry for the given esid/vsid pair.
arch/powerpc/platforms/ps3/smp.c +1 -1
@@ -37,7 +37,7 @@*/ #define MSG_COUNT 4 -static DEFINE_PER_CPU(unsigned int, ps3_ipi_virqs[MSG_COUNT]); +static DEFINE_PER_CPU(unsigned int [MSG_COUNT], ps3_ipi_virqs);static void do_message_pass (int target,int msg) {
arch/x86/kernel/cpu/cpu_debug.c +2 -2
@@ -30,8 +30,8 @@#include <asm/apic.h> #include <asm/desc.h> -static DEFINE_PER_CPU(struct cpu_cpuX_base, cpu_arr[CPU_REG_ALL_BIT]); -static DEFINE_PER_CPU(struct cpu_private *, priv_arr[MAX_CPU_FILES]); +static DEFINE_PER_CPU(struct cpu_cpuX_base [CPU_REG_ALL_BIT], cpu_arr); +static DEFINE_PER_CPU(struct cpu_private * [MAX_CPU_FILES], priv_arr); static DEFINE_PER_CPU(int, cpu_priv_count);static DEFINE_MUTEX (cpu_debug_lock);
arch/x86/kernel/cpu/mcheck/mce_amd.c +1 -1
@@ -69,7 +69,7 @@ struct threshold_bank {struct threshold_block *blocks;cpumask_var_t cpus; }; -static DEFINE_PER_CPU(struct threshold_bank *, threshold_banks[NR_BANKS]); +static DEFINE_PER_CPU(struct threshold_bank * [NR_BANKS], threshold_banks);#ifdef CONFIG_SMP static unsigned char shared_bank[NR_BANKS] = {
arch/x86/kernel/cpu/perf_counter.c +1 -1
@@ -862,7 +862,7 @@ amd_pmu_disable_counter(struct hw_perf_counter *hwc, int idx)x86_pmu_disable_counter (hwc, idx); } -static DEFINE_PER_CPU(u64, prev_left[X86_PMC_IDX_MAX]); +static DEFINE_PER_CPU(u64 [X86_PMC_IDX_MAX], prev_left);/* * Set the next IRQ period, based on the hwc->period_left value.
drivers/xen/events.c +2 -2
@@ -47,10 +47,10 @@static DEFINE_SPINLOCK (irq_mapping_update_lock);/* IRQ <-> VIRQ mapping. */ -static DEFINE_PER_CPU(int, virq_to_irq[NR_VIRQS]) = {[0 ... NR_VIRQS-1] = -1}; +static DEFINE_PER_CPU(int [NR_VIRQS], virq_to_irq) = {[0 ... NR_VIRQS-1] = -1};/* IRQ <-> IPI mapping */ -static DEFINE_PER_CPU(int, ipi_to_irq[XEN_NR_IPIS]) = {[0 ... XEN_NR_IPIS-1] = -1}; +static DEFINE_PER_CPU(int [XEN_NR_IPIS], ipi_to_irq) = {[0 ... XEN_NR_IPIS-1] = -1};/* Interrupt types. */ enum xen_irq_type {
mm/quicklist.c +1 -1
@@ -19,7 +19,7 @@#include <linux/module.h> #include <linux/quicklist.h> -DEFINE_PER_CPU(struct quicklist, quicklist)[CONFIG_NR_QUICK]; +DEFINE_PER_CPU(struct quicklist [CONFIG_NR_QUICK], quicklist);#define FRACTION_OF_NODE_MEM 16
mm/slub.c +2 -2
@@ -2086,8 +2086,8 @@ init_kmem_cache_node(struct kmem_cache_node *n, struct kmem_cache *s)*/ #define NR_KMEM_CACHE_CPU 100 -static DEFINE_PER_CPU(struct kmem_cache_cpu, - kmem_cache_cpu)[NR_KMEM_CACHE_CPU]; +static DEFINE_PER_CPU(struct kmem_cache_cpu [NR_KMEM_CACHE_CPU], + kmem_cache_cpu); static DEFINE_PER_CPU(struct kmem_cache_cpu *, kmem_cache_cpu_free); static DECLARE_BITMAP(kmem_cach_cpu_free_init_once, CONFIG_NR_CPUS);
net/ipv4/syncookies.c +1 -1
@@ -37,7 +37,7 @@ __initcall(init_syncookies);#define COOKIEBITS 24/* Upper bits store count */ #define COOKIEMASK (((__u32)1 << COOKIEBITS) - 1) -static DEFINE_PER_CPU(__u32, cookie_scratch)[16 + 5 + SHA_WORKSPACE_WORDS]; +static DEFINE_PER_CPU(__u32 [16 + 5 + SHA_WORKSPACE_WORDS], cookie_scratch);static u32 cookie_hash (__be32 saddr,__be32 daddr,__be16 sport,__be16 dport,u32 count,int c)
net/ipv6/syncookies.c +1 -1
@@ -74,7 +74,7 @@ static inline struct sock *get_cookie_sock(struct sock *sk, struct sk_buff *skb,return child; } -static DEFINE_PER_CPU(__u32, cookie_scratch)[16 + 5 + SHA_WORKSPACE_WORDS]; +static DEFINE_PER_CPU(__u32 [16 + 5 + SHA_WORKSPACE_WORDS], cookie_scratch);static u32 cookie_hash (struct in6_addr *saddr,struct in6_addr *daddr,__be16 sport,__be16 dport,u32 count,int c)
Signature · 7 lines
1.6.0.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Tejun Heo
maintainerPATCH 5/10 percpu: use DEFINE_PER_CPU_SHARED_ALIGNED()
In reply to Tejun HeoThere are a few places where ___cacheline_aligned* is used with DEFINE_PER_CPU(). Use DEFINE_PER_CPU_SHARED_ALIGNED() instead.
DEFINE_PER_CPU_SHARED_ALIGNED() applies alignment only on SMPs. While all other converted places used _in_smp variant or only get compiled for SMP, net/rds used unconditional ____cacheline_aligned. I don't see any reason these data structures should be aligned on UP and thus converted together.
--- arch/blackfin/mm/sram-alloc.c | 6 +++--- arch/ia64/kernel/smp.c | 3 ++- kernel/sched.c | 4 ++-- net/rds/ib_stats.c | 2 +- net/rds/iw_stats.c | 2 +- net/rds/page.c | 2 +- 6 files changed, 10 insertions(+), 9 deletions(-)
arch/blackfin/mm/sram-alloc.c +3 -3
@@ -42,9 +42,9 @@#include <asm/mem_map.h> #include "blackfin_sram.h" -static DEFINE_PER_CPU(spinlock_t, l1sram_lock) ____cacheline_aligned_in_smp; -static DEFINE_PER_CPU(spinlock_t, l1_data_sram_lock) ____cacheline_aligned_in_smp; -static DEFINE_PER_CPU(spinlock_t, l1_inst_sram_lock) ____cacheline_aligned_in_smp; +static DEFINE_PER_CPU_SHARED_ALIGNED(spinlock_t, l1sram_lock); +static DEFINE_PER_CPU_SHARED_ALIGNED(spinlock_t, l1_data_sram_lock); +static DEFINE_PER_CPU_SHARED_ALIGNED(spinlock_t, l1_inst_sram_lock);static spinlock_t l2_sram_lock____cacheline_aligned_in_smp ;/* the data structure for L1 scratchpad and DATA SRAM */
arch/ia64/kernel/smp.c +2 -1
@@ -58,7 +58,8 @@ static struct local_tlb_flush_counts {unsigned int count; } __attribute__((__aligned__(32))) local_tlb_flush_counts[NR_CPUS]; -static DEFINE_PER_CPU(unsigned short [NR_CPUS], shadow_flush_counts) ____cacheline_aligned; +static DEFINE_PER_CPU_SHARED_ALIGNED(unsigned short [NR_CPUS], + shadow_flush_counts);#define IPI_CALL_FUNC 0#define IPI_CPU_STOP 1
kernel/sched.c +2 -2
@@ -318,12 +318,12 @@ struct task_group root_task_group;/* Default task group's sched entity on each cpu */ static DEFINE_PER_CPU(struct sched_entity, init_sched_entity);/* Default task group's cfs_rq on each cpu */ -static DEFINE_PER_CPU(struct cfs_rq, init_cfs_rq) ____cacheline_aligned_in_smp; +static DEFINE_PER_CPU_SHARED_ALIGNED(struct cfs_rq, init_cfs_rq);#endif /* CONFIG_FAIR_GROUP_SCHED */ #ifdef CONFIG_RT_GROUP_SCHED static DEFINE_PER_CPU(struct sched_rt_entity, init_sched_rt_entity); -static DEFINE_PER_CPU(struct rt_rq, init_rt_rq) ____cacheline_aligned_in_smp; +static DEFINE_PER_CPU_SHARED_ALIGNED(struct rt_rq, init_rt_rq);#endif /* CONFIG_RT_GROUP_SCHED */ #else /* !CONFIG_USER_SCHED */ #define root_task_group init_task_group
net/rds/ib_stats.c +1 -1
@@ -37,7 +37,7 @@#include "rds.h" #include "ib.h" -DEFINE_PER_CPU(struct rds_ib_statistics, rds_ib_stats) ____cacheline_aligned; +DEFINE_PER_CPU_SHARED_ALIGNED(struct rds_ib_statistics, rds_ib_stats);static char *rds_ib_stat_names[] = {"ib_connect_raced" ,
net/rds/iw_stats.c +1 -1
@@ -37,7 +37,7 @@#include "rds.h" #include "iw.h" -DEFINE_PER_CPU(struct rds_iw_statistics, rds_iw_stats) ____cacheline_aligned; +DEFINE_PER_CPU_SHARED_ALIGNED(struct rds_iw_statistics, rds_iw_stats);static char *rds_iw_stat_names[] = {"iw_connect_raced" ,
net/rds/page.c +1 -1
@@ -39,7 +39,7 @@ struct rds_page_remainder {unsigned long r_offset; }; -DEFINE_PER_CPU(struct rds_page_remainder, rds_page_remainders) ____cacheline_aligned; +DEFINE_PER_CPU_SHARED_ALIGNED(struct rds_page_remainder, rds_page_remainders);/* * returns 0 on success or -errno on failure.
Signature · 1 line
1.6.0.2
Re: [PATCH 05/10] percpu: use DEFINE_PER_CPU_SHARED_ALIGNED()
Tejun Heo
maintainerPATCH 6/10 percpu: clean up percpu variable definitions
In reply to Tejun HeoPercpu variable definition is about to be updated such that all percpu symbols including the static ones must be unique. Update percpu variable definitions accordingly.
* as,cfq: rename ioc_count uniquely
* cpufreq: rename cpu_dbs_info uniquely
* xen: move nesting_count out of xen_evtchn_do_upcall() and rename it
* mm: move ratelimits out of balance_dirty_pages_ratelimited_nr() and rename it
* ipv4,6: rename cookie_scratch uniquely
* x86 perf_counter: rename prev_left to pmc_prev_left, irq_entry to pmc_irq_entry and nmi_entry to pmc_nmi_entry
* perf_counter: rename disable_count to perf_disable_count
* ftrace: rename test_event_disable to ftrace_test_event_disable
* kmemleak: rename test_pointer to kmemleak_test_pointer
* mce: rename next_interval to mce_next_interval
[ Impact: percpu usage cleanups, no duplicate static percpu var names ]
--- arch/x86/kernel/cpu/mcheck/mce.c | 8 ++++---- arch/x86/kernel/cpu/perf_counter.c | 14 +++++++------- block/as-iosched.c | 10 +++++----- block/cfq-iosched.c | 10 +++++----- drivers/cpufreq/cpufreq_conservative.c | 12 ++++++------ drivers/cpufreq/cpufreq_ondemand.c | 15 ++++++++------- drivers/xen/events.c | 9 +++++---- kernel/perf_counter.c | 6 +++--- kernel/trace/trace_events.c | 6 +++--- mm/kmemleak-test.c | 6 +++--- mm/page-writeback.c | 5 +++-- net/ipv4/syncookies.c | 5 +++-- net/ipv6/syncookies.c | 5 +++-- 13 files changed, 58 insertions(+), 53 deletions(-)
arch/x86/kernel/cpu/mcheck/mce.c +4 -4
@@ -1091,7 +1091,7 @@ void mce_log_therm_throt_event(__u64 status)*/ static int check_interval =5 *60 ;/* 5 minutes */ -static DEFINE_PER_CPU(int, next_interval); /* in jiffies */ +static DEFINE_PER_CPU(int, mce_next_interval); /* in jiffies */ static DEFINE_PER_CPU(struct timer_list, mce_timer);static void mcheck_timer (unsigned long data)
@@ -1110,7 +1110,7 @@ static void mcheck_timer(unsigned long data)* Alert userspace if needed. If we logged an MCE, reduce the * polling interval, otherwise increase the polling interval. */ - n = &__get_cpu_var (next_interval); + n = &__get_cpu_var (mce_next_interval);if (mce_notify_irq ()) *n =max (*n/2 ,HZ /100 );else
@@ -1311,7 +1311,7 @@ static void mce_cpu_features(struct cpuinfo_x86 *c)static void mce_init_timer (void ) {struct timer_list *t = &__get_cpu_var (mce_timer); -int *n = &__get_cpu_var (next_interval); +int *n = &__get_cpu_var (mce_next_interval);if (mce_ignore_ce)return ;
@@ -1914,7 +1914,7 @@ mce_cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)case CPU_DOWN_FAILED :case CPU_DOWN_FAILED_FROZEN : t->expires =round_jiffies (jiffies + -__get_cpu_var (next_interval)); +__get_cpu_var (mce_next_interval));add_timer_on (t, cpu);smp_call_function_single (cpu, mce_reenable_cpu, &action,1 );break ;
arch/x86/kernel/cpu/perf_counter.c +7 -7
@@ -862,7 +862,7 @@ amd_pmu_disable_counter(struct hw_perf_counter *hwc, int idx)x86_pmu_disable_counter (hwc, idx); } -static DEFINE_PER_CPU(u64 [X86_PMC_IDX_MAX], prev_left); +static DEFINE_PER_CPU(u64 [X86_PMC_IDX_MAX], pmc_prev_left);/* * Set the next IRQ period, based on the hwc->period_left value.
@@ -901,7 +901,7 @@ x86_perf_counter_set_period(struct perf_counter *counter,if (left > x86_pmu.max_period) left = x86_pmu.max_period; -per_cpu (prev_left[idx],smp_processor_id ()) = left; +per_cpu (pmc_prev_left[idx],smp_processor_id ()) = left;/* * The hw counter starts counting from this counter offset,
@@ -1089,7 +1089,7 @@ void perf_counter_print_debug(void)rdmsrl (x86_pmu.eventsel + idx, pmc_ctrl);rdmsrl (x86_pmu.perfctr + idx, pmc_count); - prev_left =per_cpu (prev_left[idx], cpu); + prev_left =per_cpu (pmc_prev_left[idx], cpu);pr_info ("CPU#%d: gen-PMC%d ctrl: %016llx\n" , cpu, idx, pmc_ctrl);
@@ -1561,8 +1561,8 @@ void callchain_store(struct perf_callchain_entry *entry, u64 ip) entry->ip[entry->nr++] = ip; } -static DEFINE_PER_CPU(struct perf_callchain_entry, irq_entry); -static DEFINE_PER_CPU(struct perf_callchain_entry, nmi_entry); +static DEFINE_PER_CPU(struct perf_callchain_entry, pmc_irq_entry); +static DEFINE_PER_CPU(struct perf_callchain_entry, pmc_nmi_entry); static void
@@ -1709,9 +1709,9 @@ struct perf_callchain_entry *perf_callchain(struct pt_regs *regs)struct perf_callchain_entry *entry;if (in_nmi ()) - entry = &__get_cpu_var (nmi_entry); + entry = &__get_cpu_var (pmc_nmi_entry);else - entry = &__get_cpu_var (irq_entry); + entry = &__get_cpu_var (pmc_irq_entry); entry->nr =0 ;
block/as-iosched.c +5 -5
@@ -146,7 +146,7 @@ enum arq_state {#define RQ_STATE (rq) ((enum arq_state)(rq)->elevator_private2)#define RQ_SET_STATE (rq, state) ((rq)->elevator_private2 = (void *) state) -static DEFINE_PER_CPU(unsigned long, ioc_count); +static DEFINE_PER_CPU(unsigned long, as_ioc_count);static struct completion *ioc_gone;static DEFINE_SPINLOCK (ioc_gone_lock);
@@ -161,7 +161,7 @@ static void as_antic_stop(struct as_data *ad);static void free_as_io_context (struct as_io_context *aic) {kfree (aic); -elv_ioc_count_dec (ioc_count); +elv_ioc_count_dec (as_ioc_count);if (ioc_gone) {/* * AS scheduler is exiting, grab exit lock and check
@@ -169,7 +169,7 @@ static void free_as_io_context(struct as_io_context *aic)* complete ioc_gone and set it back to NULL. */ spin_lock (&ioc_gone_lock); -if (ioc_gone && !elv_ioc_count_read (ioc_count)) { +if (ioc_gone && !elv_ioc_count_read (as_ioc_count)) {complete (ioc_gone); ioc_gone =NULL ; }
@@ -211,7 +211,7 @@ static struct as_io_context *alloc_as_io_context(void) ret->seek_total =0 ; ret->seek_samples =0 ; ret->seek_mean =0 ; -elv_ioc_count_inc (ioc_count); +elv_ioc_count_inc (as_ioc_count); }return ret;
@@ -1507,7 +1507,7 @@ static void __exit as_exit(void) ioc_gone = &all_gone;/* ioc_gone's update must be visible before reading ioc_count */ smp_wmb (); -if (elv_ioc_count_read (ioc_count)) +if (elv_ioc_count_read (as_ioc_count))wait_for_completion (&all_gone);synchronize_rcu (); }
block/cfq-iosched.c +5 -5
@@ -48,7 +48,7 @@ static int cfq_slice_idle = HZ / 125;static struct kmem_cache *cfq_pool;static struct kmem_cache *cfq_ioc_pool; -static DEFINE_PER_CPU(unsigned long, ioc_count); +static DEFINE_PER_CPU(unsigned long, cfq_ioc_count);static struct completion *ioc_gone;static DEFINE_SPINLOCK (ioc_gone_lock);
@@ -1422,7 +1422,7 @@ static void cfq_cic_free_rcu(struct rcu_head *head) cic = container_of(head, struct cfq_io_context, rcu_head);kmem_cache_free (cfq_ioc_pool, cic); -elv_ioc_count_dec (ioc_count); +elv_ioc_count_dec (cfq_ioc_count);if (ioc_gone) {/*
@@ -1431,7 +1431,7 @@ static void cfq_cic_free_rcu(struct rcu_head *head)* complete ioc_gone and set it back to NULL */ spin_lock (&ioc_gone_lock); -if (ioc_gone && !elv_ioc_count_read (ioc_count)) { +if (ioc_gone && !elv_ioc_count_read (cfq_ioc_count)) {complete (ioc_gone); ioc_gone =NULL ; }
@@ -1557,7 +1557,7 @@ cfq_alloc_io_context(struct cfq_data *cfqd, gfp_t gfp_mask)INIT_HLIST_NODE (&cic->cic_list); cic->dtor = cfq_free_io_context; cic->exit = cfq_exit_io_context; -elv_ioc_count_inc (ioc_count); +elv_ioc_count_inc (cfq_ioc_count); }return cic;
@@ -2658,7 +2658,7 @@ static void __exit cfq_exit(void)* this also protects us from entering cfq_slab_kill() with * pending RCU callbacks */ -if (elv_ioc_count_read (ioc_count)) +if (elv_ioc_count_read (cfq_ioc_count))wait_for_completion (&all_gone);cfq_slab_kill (); }
drivers/cpufreq/cpufreq_conservative.c +6 -6
@@ -65,7 +65,7 @@ struct cpu_dbs_info_s {int cpu; unsigned int enable:1; }; -static DEFINE_PER_CPU(struct cpu_dbs_info_s, cpu_dbs_info); +static DEFINE_PER_CPU(struct cpu_dbs_info_s, cs_cpu_dbs_info);static unsigned int dbs_enable;/* number of CPUs using this policy */
@@ -138,7 +138,7 @@ dbs_cpufreq_notifier(struct notifier_block *nb, unsigned long val, void *data) {struct cpufreq_freqs *freq = data; -struct cpu_dbs_info_s *this_dbs_info = &per_cpu (cpu_dbs_info, +struct cpu_dbs_info_s *this_dbs_info = &per_cpu (cs_cpu_dbs_info, freq->cpu);struct cpufreq_policy *policy;
@@ -298,7 +298,7 @@ static ssize_t store_ignore_nice_load(struct cpufreq_policy *policy,/* we need to re-evaluate prev_cpu_idle */ for_each_online_cpu (j) {struct cpu_dbs_info_s *dbs_info; - dbs_info = &per_cpu (cpu_dbs_info, j); + dbs_info = &per_cpu (cs_cpu_dbs_info, j); dbs_info->prev_cpu_idle =get_cpu_idle_time (j, &dbs_info->prev_cpu_wall);if (dbs_tuners_ins.ignore_nice)
@@ -388,7 +388,7 @@ static void dbs_check_cpu(struct cpu_dbs_info_s *this_dbs_info)cputime64_t cur_wall_time, cur_idle_time;unsigned int idle_time, wall_time; - j_dbs_info = &per_cpu (cpu_dbs_info, j); + j_dbs_info = &per_cpu (cs_cpu_dbs_info, j); cur_idle_time =get_cpu_idle_time (j, &cur_wall_time);
@@ -528,7 +528,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,unsigned int j;int rc; - this_dbs_info = &per_cpu (cpu_dbs_info, cpu); + this_dbs_info = &per_cpu (cs_cpu_dbs_info, cpu);switch (event) {case CPUFREQ_GOV_START :
@@ -548,7 +548,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,for_each_cpu (j, policy->cpus) {struct cpu_dbs_info_s *j_dbs_info; - j_dbs_info = &per_cpu (cpu_dbs_info, j); + j_dbs_info = &per_cpu (cs_cpu_dbs_info, j); j_dbs_info->cur_policy = policy; j_dbs_info->prev_cpu_idle = get_cpu_idle_time(j,
drivers/cpufreq/cpufreq_ondemand.c +8 -7
@@ -73,7 +73,7 @@ struct cpu_dbs_info_s { unsigned int enable:1, sample_type:1; }; -static DEFINE_PER_CPU(struct cpu_dbs_info_s, cpu_dbs_info); +static DEFINE_PER_CPU(struct cpu_dbs_info_s, od_cpu_dbs_info);static unsigned int dbs_enable;/* number of CPUs using this policy */
@@ -151,7 +151,8 @@ static unsigned int powersave_bias_target(struct cpufreq_policy *policy,unsigned int freq_hi, freq_lo;unsigned int index =0 ;unsigned int jiffies_total, jiffies_hi, jiffies_lo; -struct cpu_dbs_info_s *dbs_info = &per_cpu (cpu_dbs_info, policy->cpu); +struct cpu_dbs_info_s *dbs_info = &per_cpu (od_cpu_dbs_info, + policy->cpu);if (!dbs_info->freq_table) { dbs_info->freq_lo =0 ;
@@ -196,7 +197,7 @@ static void ondemand_powersave_bias_init(void) {int i;for_each_online_cpu (i) { -struct cpu_dbs_info_s *dbs_info = &per_cpu (cpu_dbs_info, i); +struct cpu_dbs_info_s *dbs_info = &per_cpu (od_cpu_dbs_info, i); dbs_info->freq_table =cpufreq_frequency_get_table (i); dbs_info->freq_lo =0 ; }
@@ -297,7 +298,7 @@ static ssize_t store_ignore_nice_load(struct cpufreq_policy *policy,/* we need to re-evaluate prev_cpu_idle */ for_each_online_cpu (j) {struct cpu_dbs_info_s *dbs_info; - dbs_info = &per_cpu (cpu_dbs_info, j); + dbs_info = &per_cpu (od_cpu_dbs_info, j); dbs_info->prev_cpu_idle =get_cpu_idle_time (j, &dbs_info->prev_cpu_wall);if (dbs_tuners_ins.ignore_nice)
@@ -391,7 +392,7 @@ static void dbs_check_cpu(struct cpu_dbs_info_s *this_dbs_info)unsigned int load, load_freq;int freq_avg; - j_dbs_info = &per_cpu (cpu_dbs_info, j); + j_dbs_info = &per_cpu (od_cpu_dbs_info, j); cur_idle_time =get_cpu_idle_time (j, &cur_wall_time);
@@ -548,7 +549,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,unsigned int j;int rc; - this_dbs_info = &per_cpu (cpu_dbs_info, cpu); + this_dbs_info = &per_cpu (od_cpu_dbs_info, cpu);switch (event) {case CPUFREQ_GOV_START :
@@ -570,7 +571,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,for_each_cpu (j, policy->cpus) {struct cpu_dbs_info_s *j_dbs_info; - j_dbs_info = &per_cpu (cpu_dbs_info, j); + j_dbs_info = &per_cpu (od_cpu_dbs_info, j); j_dbs_info->cur_policy = policy; j_dbs_info->prev_cpu_idle = get_cpu_idle_time(j,
drivers/xen/events.c +5 -4
@@ -602,6 +602,8 @@ irqreturn_t xen_debug_interrupt(int irq, void *dev_id)return IRQ_HANDLED ; } +static DEFINE_PER_CPU(unsigned, xed_nesting_count); +/* * Search the CPUs pending events bitmasks. For each one found, map * the event number to an irq, and feed it into do_IRQ() for
@@ -617,7 +619,6 @@ void xen_evtchn_do_upcall(struct pt_regs *regs)struct pt_regs *old_regs =set_irq_regs (regs);struct shared_info *s = HYPERVISOR_shared_info;struct vcpu_info *vcpu_info =__get_cpu_var (xen_vcpu); - static DEFINE_PER_CPU(unsigned, nesting_count);unsigned count;exit_idle ();
@@ -628,7 +629,7 @@ void xen_evtchn_do_upcall(struct pt_regs *regs) vcpu_info->evtchn_upcall_pending =0 ; -if (__get_cpu_var (nesting_count)++) +if (__get_cpu_var (xed_nesting_count)++) goto out;#ifndef CONFIG_X86 /* No need for a barrier -- XCHG is a barrier on x86. */
@@ -653,8 +654,8 @@ void xen_evtchn_do_upcall(struct pt_regs *regs)BUG_ON (!irqs_disabled ()); - count =__get_cpu_var (nesting_count); -__get_cpu_var (nesting_count) =0 ; + count =__get_cpu_var (xed_nesting_count); +__get_cpu_var (xed_nesting_count) =0 ; }while (count !=1 ); out:
kernel/perf_counter.c +3 -3
@@ -98,16 +98,16 @@ hw_perf_group_sched_in(struct perf_counter *group_leader,void __weak perf_counter_print_debug (void ) { } -static DEFINE_PER_CPU(int, disable_count); +static DEFINE_PER_CPU(int, perf_disable_count);void __perf_disable (void ) { -__get_cpu_var (disable_count)++; +__get_cpu_var (perf_disable_count)++; }bool __perf_enable (void ) { -return !--__get_cpu_var (disable_count); +return !--__get_cpu_var (perf_disable_count); }void perf_disable (void )
kernel/trace/trace_events.c +3 -3
@@ -1318,7 +1318,7 @@ static __init void event_trace_self_tests(void)#ifdef CONFIG_FUNCTION_TRACER -static DEFINE_PER_CPU(atomic_t, test_event_disable); +static DEFINE_PER_CPU(atomic_t, ftrace_test_event_disable);static void function_test_events_call (unsigned long ip,unsigned long parent_ip)
@@ -1334,7 +1334,7 @@ function_test_events_call(unsigned long ip, unsigned long parent_ip) pc =preempt_count (); resched =ftrace_preempt_disable (); cpu =raw_smp_processor_id (); - disabled =atomic_inc_return (&per_cpu (test_event_disable, cpu)); + disabled =atomic_inc_return (&per_cpu (ftrace_test_event_disable, cpu));if (disabled !=1 ) goto out;
@@ -1352,7 +1352,7 @@ function_test_events_call(unsigned long ip, unsigned long parent_ip)trace_nowake_buffer_unlock_commit (event, flags, pc); out: -atomic_dec (&per_cpu (test_event_disable, cpu)); +atomic_dec (&per_cpu (ftrace_test_event_disable, cpu));ftrace_preempt_enable (resched); }
mm/kmemleak-test.c +3 -3
@@ -36,7 +36,7 @@ struct test_node { };static LIST_HEAD (test_list); -static DEFINE_PER_CPU(void *, test_pointer); +static DEFINE_PER_CPU(void *, kmemleak_test_pointer);/* * Some very simple testing. This function needs to be extended for
@@ -86,9 +86,9 @@ static int __init kmemleak_test_init(void) }for_each_possible_cpu (i) { -per_cpu (test_pointer, i) =kmalloc (129 ,GFP_KERNEL ); +per_cpu (kmemleak_test_pointer, i) =kmalloc (129 ,GFP_KERNEL );pr_info ("kmemleak: kmalloc(129) = %p\n" , -per_cpu (test_pointer, i)); +per_cpu (kmemleak_test_pointer, i)); }return 0 ;
mm/page-writeback.c +3 -2
@@ -607,6 +607,8 @@ void set_page_dirty_balance(struct page *page, int page_mkwrite) } } +static DEFINE_PER_CPU(unsigned long, bdp_ratelimits) = 0; +/** * balance_dirty_pages_ratelimited_nr - balance dirty memory state * @mapping: address_space which was dirtied
@@ -624,7 +626,6 @@ void set_page_dirty_balance(struct page *page, int page_mkwrite)void balance_dirty_pages_ratelimited_nr (struct address_space *mapping,unsigned long nr_pages_dirtied) { - static DEFINE_PER_CPU(unsigned long, ratelimits) = 0;unsigned long ratelimit;unsigned long *p;
@@ -637,7 +638,7 @@ void balance_dirty_pages_ratelimited_nr(struct address_space *mapping,* tasks in balance_dirty_pages(). Period. */ preempt_disable (); - p = &__get_cpu_var (ratelimits); + p = &__get_cpu_var (bdp_ratelimits); *p += nr_pages_dirtied;if (unlikely (*p >= ratelimit)) { *p =0 ;
net/ipv4/syncookies.c +3 -2
@@ -37,12 +37,13 @@ __initcall(init_syncookies);#define COOKIEBITS 24/* Upper bits store count */ #define COOKIEMASK (((__u32)1 << COOKIEBITS) - 1) -static DEFINE_PER_CPU(__u32 [16 + 5 + SHA_WORKSPACE_WORDS], cookie_scratch); +static DEFINE_PER_CPU(__u32 [16 + 5 + SHA_WORKSPACE_WORDS], + ipv4_cookie_scratch);static u32 cookie_hash (__be32 saddr,__be32 daddr,__be16 sport,__be16 dport,u32 count,int c) { -__u32 *tmp =__get_cpu_var (cookie_scratch); +__u32 *tmp =__get_cpu_var (ipv4_cookie_scratch);memcpy (tmp +4 , syncookie_secret[c],sizeof (syncookie_secret[c])); tmp[0 ] = (__force u32 )saddr;
net/ipv6/syncookies.c +3 -2
@@ -74,12 +74,13 @@ static inline struct sock *get_cookie_sock(struct sock *sk, struct sk_buff *skb,return child; } -static DEFINE_PER_CPU(__u32 [16 + 5 + SHA_WORKSPACE_WORDS], cookie_scratch); +static DEFINE_PER_CPU(__u32 [16 + 5 + SHA_WORKSPACE_WORDS], + ipv6_cookie_scratch);static u32 cookie_hash (struct in6_addr *saddr,struct in6_addr *daddr,__be16 sport,__be16 dport,u32 count,int c) { -__u32 *tmp =__get_cpu_var (cookie_scratch); +__u32 *tmp =__get_cpu_var (ipv6_cookie_scratch);/* * we have 320 bits of information to hash, copy in the remaining
Signature · 7 lines
1.6.0.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Tejun Heo
maintainerPATCH 7/10 percpu: implement optional weak percpu definitions
In reply to Tejun HeoSome archs (alpha and s390) need to use weak definitions for percpu variables in modules so that the compiler generates external references for them.
This patch implements weak percpu definitions which arch can enable by defining ARCH_NEEDS_WEAK_PER_CPU in arch percpu header file. This weak definition adds the following two restrictions on percpu variable definitions.
1. percpu symbols must be unique whether static or not
2. percpu variables can't be defined inside a function
To ensure that these restrictions are observed in generic code, config option DEBUG_FORCE_WEAK_PER_CPU enables weak percpu definitions for all cases.
This patch is inspired by Ivan Kokshaysky's alpha percpu patch.
[ Impact: stricter rules for percpu variables, one more debug config option ]
--- include/linux/percpu-defs.h | 65 +++++++++++++++++++++++++++++++++++++------ lib/Kconfig.debug | 15 ++++++++++ 2 files changed, 71 insertions(+), 9 deletions(-)
include/linux/percpu-defs.h +56 -9
@@ -10,21 +10,68 @@/* * Base implementations of per-CPU variable declarations and definitions, where * the section in which the variable is to be placed is provided by the -* 'section' argument. This may be used to affect the parameters governing the +* 'sec' argument. This may be used to affect the parameters governing the * variable's storage. * * NOTE! The sections for the DECLARE and for the DEFINE must match, lest * linkage errors occur due the compiler generating the wrong code to access * that section. */ -#define DECLARE_PER_CPU_SECTION (type, name, section) \ - extern \ - __attribute__((__section__(PER_CPU_BASE_SECTION section))) \ - PER_CPU_ATTRIBUTES __typeof__(type) per_cpu__##name - -#define DEFINE_PER_CPU_SECTION (type, name, section) \ - __attribute__((__section__(PER_CPU_BASE_SECTION section))) \ - PER_CPU_ATTRIBUTES __typeof__(type) per_cpu__##name +#define __PCPU_ATTRS (sec) \ + __attribute__((section(PER_CPU_BASE_SECTION sec))) \ + PER_CPU_ATTRIBUTES + +#define __PCPU_DUMMY_ATTRS \ + __attribute__((section(".discard"), unused)) + +/* +* s390 and alpha modules require percpu variables to be defined as +* weak to force the compiler to generate GOT based external +* references for them. This is necessary because percpu sections +* will be located outside of the usually addressable area. +* +* This definition puts the following two extra restrictions when +* defining percpu variables. +* +* 1. The symbol must be globally unique, even the static ones. +* 2. Static percpu variables cannot be defined inside a function. +* +* Archs which need weak percpu definitions should define +* ARCH_NEEDS_WEAK_PER_CPU in asm/percpu.h when necessary. +* +* To ensure that the generic code observes the above two +* restrictions, if CONFIG_DEBUG_FORCE_WEAK_PER_CPU is set weak +* definition is used for all cases. +*/ +#if defined(ARCH_NEEDS_WEAK_PER_CPU ) || defined(CONFIG_DEBUG_FORCE_WEAK_PER_CPU ) +/* +* __pcpu_scope_* dummy variable is used to enforce scope. It +* receives the static modifier when it's used in front of +* DEFINE_PER_CPU() and will trigger build failure if +* DECLARE_PER_CPU() is used for the same variable. +* +* __pcpu_unique_* dummy variable is used to enforce symbol uniqueness +* such that hidden weak symbol collision, which will cause unrelated +* variables to share the same address, can be detected during build. +*/ +#define DECLARE_PER_CPU_SECTION (type, name, sec) \ + extern __PCPU_DUMMY_ATTRS char __pcpu_scope_##name; \ + extern __PCPU_ATTRS(sec) __weak __typeof__(type) per_cpu__##name + +#define DEFINE_PER_CPU_SECTION (type, name, sec) \ + __PCPU_DUMMY_ATTRS char __pcpu_scope_##name; \ + __PCPU_DUMMY_ATTRS char __pcpu_unique_##name; \ + __PCPU_ATTRS(sec) __weak __typeof__(type) per_cpu__##name +#else +/* +* Normal declaration and definition macros. +*/ +#define DECLARE_PER_CPU_SECTION (type, name, sec) \ + extern __PCPU_ATTRS(sec) __typeof__(type) per_cpu__##name + +#define DEFINE_PER_CPU_SECTION (type, name, sec) \ + __PCPU_ATTRS(sec) __typeof__(type) per_cpu__##name +#endif /* * Variant on the per-CPU variable declaration/definition theme used for
lib/Kconfig.debug +15 -0
@@ -777,6 +777,21 @@ config DEBUG_BLOCK_EXT_DEVT Say N if you are unsure. +config DEBUG_FORCE_WEAK_PER_CPU + bool "Force weak per-cpu definitions" + depends on DEBUG_KERNEL + help + s390 and alpha require percpu variables in modules to be + defined weak to work around addressing range issue which + puts the following two restrictions on percpu variable + definitions. + + 1. percpu symbols must be unique whether static or not + 2. percpu variables can't be defined inside a function + + To ensure that generic code follows the above rules, this + option forces all percpu variables to be defined as weak. + config LKDTM tristate "Linux Kernel Dump Test Tool Module" depends on DEBUG_KERNEL
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 8/10 alpha: kill unnecessary __used attribute in PER_CPU_ATTRIBUTES
In reply to Tejun HeoWith the previous percpu variable definition change, all percpu variables are global and there's no need to specify __used, which only triggers on recent compilers anyway. Kill it.
[ Impact: remove unnecessary percpu attribute ]
--- arch/alpha/include/asm/percpu.h | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-)
arch/alpha/include/asm/percpu.h +0 -5
@@ -30,7 +30,6 @@ extern unsigned long __per_cpu_offset[NR_CPUS];#ifndef MODULE #define SHIFT_PERCPU_PTR (var, offset) RELOC_HIDE(&per_cpu_var(var), (offset)) -#define PER_CPU_ATTRIBUTES #else /* * To calculate addresses of locally defined variables, GCC uses 32-bit
@@ -49,8 +48,6 @@ extern unsigned long __per_cpu_offset[NR_CPUS]; : "=&r"(__ptr), "=&r"(tmp_gp)); \ (typeof(&per_cpu_var(var)))(__ptr + (offset)); }) -#define PER_CPU_ATTRIBUTES __used -#endif /* MODULE */ /*
@@ -71,8 +68,6 @@ extern unsigned long __per_cpu_offset[NR_CPUS];#define __get_cpu_var (var) per_cpu_var(var)#define __raw_get_cpu_var (var) per_cpu_var(var) -#define PER_CPU_ATTRIBUTES -#endif /* SMP */ #ifdef CONFIG_SMP
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 9/10 alpha: switch to dynamic percpu allocator
In reply to Tejun HeoAlpha implements custom SHIFT_PERCPU_PTR for modules because percpu area can be located far away from the 4G area where the module text is located. The custom SHIFT_PERCPU_PTR forces GOT usage using ldq instruction with literal relocation; however, the relocation can't be used with dynamically allocated percpu variables. Fortunately, similar result can be achieved by using weak percpu variable definitions.
This patch makes alpha use weak definitions and switch to dynamic percpu allocator.
asm/tlbflush.h was getting linux/sched.h via asm/percpu.h which no longer needs it. Include linux/sched.h directly in asm/tlbflush.h.
Compile tested. Generation of litereal relocation verified.
This patch is based on Ivan Kokshaysky's alpha percpu patch.
[ Impact: use dynamic percpu allocator ]
--- arch/alpha/Kconfig | 3 - arch/alpha/include/asm/percpu.h | 95 +++--------------------------------- arch/alpha/include/asm/tlbflush.h | 1 + 3 files changed, 9 insertions(+), 90 deletions(-)
arch/alpha/Kconfig +0 -3
@@ -70,9 +70,6 @@ config AUTO_IRQ_AFFINITY depends on SMP default y -config HAVE_LEGACY_PER_CPU_AREA - def_bool y - source "init/Kconfig" source "kernel/Kconfig.freezer"
arch/alpha/include/asm/percpu.h +8 -87
105 lines not on this page · raw message
arch/alpha/include/asm/tlbflush.h +1 -0
@@ -2,6 +2,7 @@#define _ALPHA_TLBFLUSH_H#include <linux/mm.h> +#include <linux/sched.h> #include <asm/compiler.h> #include <asm/pgalloc.h>
Signature · 1 line
1.6.0.2
Tejun Heo
maintainerPATCH 10/10 s390: switch to dynamic percpu allocator
In reply to Tejun Heo64bit s390 shares the same problem with alpha regarding percpu symbol addressing from modules. It needs assembly magic to force GOTENT reference when building module as the percpu address will be outside the usual 4G range from the module text. This can be solved by using weak percpu variable definitions.
This patch makes s390 use weak definitions and switch to dynamic percpu allocator. Please note that weak attribute is not added if !SMP as percpu variables behave exactly the same as normal variables on UP.
Compile tested. Generation of GOTENT reference verified.
This patch is based on Ivan Kokshaysky's alpha percpu patch.
[ Impact: use dynamic percpu allocator ]
--- arch/s390/Kconfig | 3 --- arch/s390/include/asm/percpu.h | 32 ++++++++------------------------ 2 files changed, 8 insertions(+), 27 deletions(-)
arch/s390/Kconfig +0 -3
@@ -75,9 +75,6 @@ config VIRT_CPU_ACCOUNTING config ARCH_SUPPORTS_DEBUG_PAGEALLOC def_bool y -config HAVE_LEGACY_PER_CPU_AREA - def_bool y - mainmenu "Linux Kernel Configuration" config S390
arch/s390/include/asm/percpu.h +8 -24
@@ -1,37 +1,21 @@#ifndef __ARCH_S390_PERCPU__#define __ARCH_S390_PERCPU__ -#include <linux/compiler.h> -#include <asm/lowcore.h> -/* * s390 uses its own implementation for per cpu data, the offset of * the cpu local data area is cached in the cpu's lowcore memory. -* For 64 bit module code s390 forces the use of a GOT slot for the -* address of the per cpu variable. This is needed because the module -* may be more than 4G above the per cpu area. */ -#if defined(__s390x__) && defined(MODULE ) - -#define SHIFT_PERCPU_PTR (ptr,offset) (({ \ - extern int simple_identifier_##var(void); \ - unsigned long *__ptr; \ - asm ( "larl %0, %1@GOTENT" \ - : "=a" (__ptr) : "X" (ptr) ); \ - (typeof(ptr))((*__ptr) + (offset)); })) - -#else - -#define SHIFT_PERCPU_PTR (ptr, offset) (({ \ - extern int simple_identifier_##var(void); \ - unsigned long __ptr; \ - asm ( "" : "=a" (__ptr) : "0" (ptr) ); \ - (typeof(ptr)) (__ptr + (offset)); })) +#define __my_cpu_offset S390_lowcore.percpu_offset +/* +* For 64 bit module code, the module may be more than 4G above the +* per cpu area, use weak definitions to force the compiler to +* generate external references. +*/ +#if defined(CONFIG_SMP ) && defined(__s390x__) && defined(MODULE ) +#define ARCH_NEEDS_WEAK_PER_CPU #endif -#define __my_cpu_offset S390_lowcore.percpu_offset -#include <asm-generic/percpu.h> #endif /* __ARCH_S390_PERCPU__ */
Signature · 1 line
1.6.0.2
Tejun Heo
maintainer[PATCH 11/10] sparc64: fix build breakage introduced by percpu-convert-most patchset
In reply to Tejun HeoCommit e74e396204bfcb67570ba4517b08f5918e69afea incorrectly added added HAVE_LEGACY_PER_CPU_AREA to sparc64 although it already has been converted to dynamic percpu allocator. Drop both HAVE_{LEGACY|DYNAMIC}_PER_CPU_AREA.
[ Impact: fix build breakage ]
---
This patch fixes build breakage caused by changed patch apply order. Thanks.
arch/sparc/Kconfig | 6 ------ 1 files changed, 0 insertions(+), 6 deletions(-)
arch/sparc/Kconfig +0 -6
@@ -92,15 +92,9 @@ config AUDIT_ARCH bool default y -config HAVE_LEGACY_PER_CPU_AREA - def_bool y if SPARC64 - config HAVE_SETUP_PER_CPU_AREA def_bool y if SPARC64 -config HAVE_DYNAMIC_PER_CPU_AREA - def_bool y if SPARC64 - config GENERIC_HARDIRQS_NO__DO_IRQ bool def_bool y if SPARC64
Signature · 1 line
1.6.0.2
Re: [PATCH 11/10] sparc64: fix build breakage introduced by percpu-convert-most patchset
From: Tejun Heo <[email protected]>
Date: Wed, 24 Jun 2009 17:03:50 +0900
Quoted 8 lines
Commit
e74e396204bfcb67570ba4517b08f5918e69afeaincorrectly added addedHAVE_LEGACY_PER_CPU_AREAto sparc64 although it already has been converted to dynamic percpu allocator. Drop both HAVE_{LEGACY|DYNAMIC}_PER_CPU_AREA.[ Impact: fix build breakage ]
Signed-off-by Tejun Heo <[email protected]>
Tejun Heo
maintainerRe: [PATCH 11/10] sparc64: fix build breakage introduced by percpu-convert-most patchset
Quoted 8 lines · David Miller
From: Tejun Heo <[email protected]>
Date: Wed, 24 Jun 2009 17:03:50 +0900Quoted 5 lines
Commit
e74e396204bfcb67570ba4517b08f5918e69afeaincorrectly added addedHAVE_LEGACY_PER_CPU_AREAto sparc64 although it already has been converted to dynamic percpu allocator. Drop both HAVE_{LEGACY|DYNAMIC}_PER_CPU_AREA.
Quoted 5 lines
Quoted 3 lines
[ Impact: fix build breakage ]
Signed-off-by Tejun Heo <[email protected]>Acked-by David S. Miller <[email protected]>
Acked-by added, patch description slightly updated as suggested by Ingo and patch now available in percpu#for-next.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerQuoted 1 line · Tejun Heo
git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.gitconvert-most-archs
Rebased tree w/ the build breakage James spotted fixed pulled into #for-next. It won't be rebased from now on unless something drastic happens.
Thanks.
Signature · 1 line
tejun
Tejun Heo
maintainerQuoted 6 lines · Tejun Heo
Quoted 1 line · Tejun Heo
git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu.gitconvert-most-archsRebased tree w/ the build breakage James spotted fixed pulled into #for-next. It won't be rebased from now on unless something drastic happens.
Eh... after hitting send I realized alpha was still broken. It'll go through another revision.
Thanks.
Signature · 1 line
tejun