add new macros to make percpu readmostly section correctly align
Subsystems (2)
PATCH 2/3 add new macros to make percpu readmostly section correctly align
percpu readmostly section should start and end at address cachline aligned. Idealy we should change PERCPU_VADDR/PERCPU, but I can't change all arch code, so I add new macros for x86.
--- include/asm-generic/vmlinux.lds.h | 66 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+)
include/asm-generic/vmlinux.lds.h +66 -0
@@ -726,6 +726,72 @@ VMLINUX_SYMBOL(__per_cpu_end) = .; \ } +/** +* PERCPU_VADDR_CACHEALIGNED - define output section for percpu area +* @vaddr: explicit base address (optional) +* @phdr: destination PHDR (optional) +* @cacheline: cachline size required by readmostly percpu data +* +* Macro which expands to output section for percpu area. If @vaddr +* is not blank, it specifies explicit base address and all percpu +* symbols will be offset from the given address. If blank, @vaddr +* always equals @laddr + LOAD_OFFSET. +* +* @phdr defines the output PHDR to use if not blank. Be warned that +* output PHDR is sticky. If @phdr is specified, the next output +* section in the linker script will go there too. @phdr should have +* a leading colon. +* +* Note that this macros defines __per_cpu_load as an absolute symbol. +* If there is no need to put the percpu section at a predetermined +* address, use PERCPU_CACHEALIGNED(). +*/ +#define PERCPU_VADDR_CACHEALIGNED (vaddr, phdr, cacheline) \ + VMLINUX_SYMBOL(__per_cpu_load) = .; \ + .data..percpu vaddr : AT(VMLINUX_SYMBOL(__per_cpu_load) \ + - LOAD_OFFSET) { \ + VMLINUX_SYMBOL(__per_cpu_start) = .; \ + *(.data..percpu..first) \ + . = ALIGN(PAGE_SIZE); \ + *(.data..percpu..page_aligned) \ + . = ALIGN(cacheline); \ + *(.data..percpu..readmostly) \ + . = ALIGN(cacheline); \ + *(.data..percpu) \ + *(.data..percpu..shared_aligned) \ + VMLINUX_SYMBOL(__per_cpu_end) = .; \ + } phdr \ + . = VMLINUX_SYMBOL(__per_cpu_load) + SIZEOF(.data..percpu); + +/** +* PERCPU_CACHEALIGNED - define output section for percpu area, simple version +* @align: required alignment +* @cacheline: cachline size required by readmostly percpu data +* +* Align to @align and outputs output section for percpu area. This +* macro doesn't maniuplate @vaddr or @phdr and __per_cpu_load and +* __per_cpu_start will be identical. +* +* This macro is equivalent to ALIGN(align); PERCPU_VADDR_CACHEALIGNED( , ) except +* that __per_cpu_load is defined as a relative symbol against +* .data..percpu which is required for relocatable x86_32 +* configuration. +*/ +#define PERCPU_CACHEALIGNED (align, cacheline) \ + . = ALIGN(align); \ + .data..percpu : AT(ADDR(.data..percpu) - LOAD_OFFSET) { \ + VMLINUX_SYMBOL(__per_cpu_load) = .; \ + VMLINUX_SYMBOL(__per_cpu_start) = .; \ + *(.data..percpu..first) \ + . = ALIGN(PAGE_SIZE); \ + *(.data..percpu..page_aligned) \ + . = ALIGN(cacheline); \ + *(.data..percpu..readmostly) \ + . = ALIGN(cacheline); \ + *(.data..percpu) \ + *(.data..percpu..shared_aligned) \ + VMLINUX_SYMBOL(__per_cpu_end) = .; \ + }/* * Definition of the high level *_SECTION macros
Hello,
Quoted 5 lines · Shaohua Li
percpu readmostly section should start and end at address cachline aligned. Idealy we should change PERCPU_VADDR/PERCPU, but I can't change all arch code, so I add new macros for x86.
Signed-off-by Shaohua Li <[email protected]>
Why add another set of macros? Can't you just make the default one aligned?
Signature · 1 line
tejun
Quoted 11 lines · Tejun Heo
Hello,
Quoted 5 lines · Shaohua Li
percpu readmostly section should start and end at address cachline aligned. Idealy we should change PERCPU_VADDR/PERCPU, but I can't change all arch code, so I add new macros for x86.
Signed-off-by Shaohua Li <[email protected]>Why add another set of macros? Can't you just make the default one aligned?
how to do it without changing all arch code? There isn't cache line size macro for all arch which could be used in vmlinux.lds.h, IIRC.
Thanks,
Shaohua
Hello,
Quoted 4 lines · Shaohua Li
Quoted 2 lines
Why add another set of macros? Can't you just make the default one aligned?
how to do it without changing all arch code? There isn't cache line size macro for all arch which could be used in
vmlinux.lds.h, IIRC.
Can't we get around that with some ifdefferies? And even if we'll have to change lds on every arch, I think we better do it that way. It's not like the problem exists only on x86, right?
Thanks.
Signature · 1 line
tejun
Quoted 10 lines · Tejun Heo
Hello,
Quoted 4 lines · Shaohua Li
Quoted 2 lines
Why add another set of macros? Can't you just make the default one aligned?
how to do it without changing all arch code? There isn't cache line size macro for all arch which could be used in
vmlinux.lds.h, IIRC.Can't we get around that with some ifdefferies? And even if we'll have to change lds on every arch, I think we better do it that way.
I don't understand what you mean. defining a cachine line size macro for all archs? There is such macro, but using it in vmlinux.ld.h always report error. There is some other defines which can't be included in a link script.
Quoted 1 line
It's not like the problem exists only on x86, right?
yes, though currently only x86 has it.
Thanks,
Shaohua
Hello,
Quoted 4 lines · Shaohua Li
I don't understand what you mean. defining a cachine line size macro for all archs? There is such macro, but using it in
vmlinux.ld.halways report error. There is some other defines which can't be included in a link script.
I haven't really looked through it but wouldn't it be possible to ifdef it. ie. if cacheline macro is available, align it to it; otherwise, don't. And, ultimately, the correct thing to do is making it cacheline aligned on all archs. There can be several different ways to get there but it might just as well be making cacheline size available in all archs first and then update the PERCPU macro.
Thank you.
Signature · 1 line
tejun
Quoted 8 lines · Tejun Heo
Hello,
Quoted 4 lines · Shaohua Li
I don't understand what you mean. defining a cachine line size macro for all archs? There is such macro, but using it in
vmlinux.ld.halways report error. There is some other defines which can't be included in a link script.
Quoted 6 lines
I haven't really looked through it but wouldn't it be possible to ifdef it. ie. if cacheline macro is available, align it to it; otherwise, don't. And, ultimately, the correct thing to do is making it cacheline aligned on all archs. There can be several different ways to get there but it might just as well be making cacheline size available in all archs first and then update the PERCPU macro.
How about this one?
Subject: Make x86 percpu readmostly section correctly aligned
percpu readmostly section should start and end at address cachline aligned to avoid cache false sharing. For ARCHs care about the cache false sharing, they should define INTERNODE_CACHE_BYTES. We use it to do the alignment. Currently only x86 has percpu readmostly section, so only changed it so far.
--- arch/x86/kernel/vmlinux.lds.S | 2 +- include/asm-generic/vmlinux.lds.h | 11 +++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-)
include/asm-generic/vmlinux.lds.h +11 -0
@@ -665,6 +665,13 @@ *(.discard.*) \ } +#ifdef INTERNODE_CACHE_BYTES +#define INTERNODE_CACHEALIGNED \ + . = ALIGN(INTERNODE_CACHE_BYTES); +#else +#define INTERNODE_CACHEALIGNED +#endif +/** * PERCPU_VADDR - define output section for percpu area * @vaddr: explicit base address (optional)
@@ -692,7 +699,9 @@ *(.data..percpu..first) \ . = ALIGN(PAGE_SIZE); \ *(.data..percpu..page_aligned) \ +INTERNODE_CACHEALIGNED \ *(.data..percpu..readmostly) \ +INTERNODE_CACHEALIGNED \ *(.data..percpu) \ *(.data..percpu..shared_aligned) \ VMLINUX_SYMBOL(__per_cpu_end) = .; \
@@ -720,7 +729,9 @@ *(.data..percpu..first) \ . = ALIGN(PAGE_SIZE); \ *(.data..percpu..page_aligned) \ +INTERNODE_CACHEALIGNED \ *(.data..percpu..readmostly) \ +INTERNODE_CACHEALIGNED \ *(.data..percpu) \ *(.data..percpu..shared_aligned) \ VMLINUX_SYMBOL(__per_cpu_end) = .; \
arch/x86/kernel/vmlinux.lds.S +1 -1
@@ -20,11 +20,11 @@ #define LOAD_OFFSET __START_KERNEL_map #endif -#include <asm-generic/vmlinux.lds.h> #include <asm/asm-offsets.h> #include <asm/thread_info.h> #include <asm/page_types.h> #include <asm/cache.h> +#include <asm-generic/vmlinux.lds.h> #include <asm/boot.h> #undef i386 /* in case the preprocessor is a 32bit one */
Hello,
Quoted 1 line · Shaohua Li
How about this one?
Much better. :-)
include/asm-generic/vmlinux.lds.h line 673
Quoted 6 lines
include/asm-generic/vmlinux.lds.h +6
-0+#ifdef INTERNODE_CACHE_BYTES +#define INTERNODE_CACHEALIGNED \ + . = ALIGN(INTERNODE_CACHE_BYTES); +#else +#define INTERNODE_CACHEALIGNED +#endif
Yeah, this looks good.
Quoted 5 lines
Index:
linux/arch/x86/kernel/vmlinux.lds.S
===================================================================
arch/x86/kernel/vmlinux.lds.S line 28
Quoted 10 lines
arch/x86/kernel/vmlinux.lds.S +1
-1@@ -20,11 +20,11 @@ #define LOAD_OFFSET __START_KERNEL_map #endif -#include <asm-generic/vmlinux.lds.h> #include <asm/asm-offsets.h> #include <asm/thread_info.h> #include <asm/page_types.h> #include <asm/cache.h> +#include <asm-generic/vmlinux.lds.h> #include <asm/boot.h>
Why do we need this chunk?
Thanks.
Signature · 1 line
tejun
Hello,
Quoted 8 lines · Tejun Heo
include/asm-generic/vmlinux.lds.h line 673
Quoted 6 lines
include/asm-generic/vmlinux.lds.h +6
-0+#ifdef INTERNODE_CACHE_BYTES +#define INTERNODE_CACHEALIGNED \ + . = ALIGN(INTERNODE_CACHE_BYTES); +#else +#define INTERNODE_CACHEALIGNED +#endif Yeah, this looks good.
Just one more thing. If you look at various arch linker scripts, cache line size is always present. After all, RW_DATA_SECTION() needs it. They're different define's or sometimes hardcoded values but it would be nice if we can take this chance to unify them and use it for this too. Would you be interested in doing that?
Thanks.
Signature · 1 line
tejun
Quoted 12 lines · Tejun Heo
Hello,
Quoted 8 lines · Tejun Heo
include/asm-generic/vmlinux.lds.h line 673
Quoted 6 lines
include/asm-generic/vmlinux.lds.h +6
-0+#ifdef INTERNODE_CACHE_BYTES +#define INTERNODE_CACHEALIGNED \ + . = ALIGN(INTERNODE_CACHE_BYTES); +#else +#define INTERNODE_CACHEALIGNED +#endif Yeah, this looks good.
Quoted 5 lines
Just one more thing. If you look at various arch linker scripts, cache line size is always present. After all,
RW_DATA_SECTION()needs it. They're different define's or sometimes hardcoded values but it would be nice if we can take this chance to unify them and use it for this too. Would you be interested in doing that?
might not, sorry.
Quoted 24 lines · Tejun Heo
Hello,
Quoted 1 line · Shaohua Li
How about this one?
Much better. :-)
include/asm-generic/vmlinux.lds.h line 673
Quoted 6 lines
include/asm-generic/vmlinux.lds.h +6
-0+#ifdef INTERNODE_CACHE_BYTES +#define INTERNODE_CACHEALIGNED \ + . = ALIGN(INTERNODE_CACHE_BYTES); +#else +#define INTERNODE_CACHEALIGNED +#endif Yeah, this looks good.
Quoted 8 lines
Index:
linux/arch/x86/kernel/vmlinux.lds.S
===================================================================arch/x86/kernel/vmlinux.lds.S +0
-0@@ -20,11 +20,11 @@ #define LOAD_OFFSET __START_KERNEL_map #endif
Quoted 9 lines
arch/x86/kernel/vmlinux.lds.S line 28
Quoted 7 lines
arch/x86/kernel/vmlinux.lds.S +1
-1-#include <asm-generic/vmlinux.lds.h> #include <asm/asm-offsets.h> #include <asm/thread_info.h> #include <asm/page_types.h> #include <asm/cache.h> +#include <asm-generic/vmlinux.lds.h> #include <asm/boot.h>Why do we need this chunk?
the cache size is defined in cache.h, so I need move vmlinux.lds.h after cache.h
Thanks,
Shaohua
Quoted 34 lines · Shaohua Li
Quoted 24 lines · Tejun Heo
Hello,
Quoted 1 line · Shaohua Li
How about this one?
Much better. :-)
include/asm-generic/vmlinux.lds.h line 673
Quoted 6 lines
include/asm-generic/vmlinux.lds.h +6
-0+#ifdef INTERNODE_CACHE_BYTES +#define INTERNODE_CACHEALIGNED \ + . = ALIGN(INTERNODE_CACHE_BYTES); +#else +#define INTERNODE_CACHEALIGNED +#endif Yeah, this looks good.
Quoted 8 lines
Index:
linux/arch/x86/kernel/vmlinux.lds.S
===================================================================arch/x86/kernel/vmlinux.lds.S +0
-0@@ -20,11 +20,11 @@ #define LOAD_OFFSET __START_KERNEL_map #endifQuoted 9 lines
arch/x86/kernel/vmlinux.lds.S line 28
Quoted 7 lines
arch/x86/kernel/vmlinux.lds.S +1
-1-#include <asm-generic/vmlinux.lds.h> #include <asm/asm-offsets.h> #include <asm/thread_info.h> #include <asm/page_types.h> #include <asm/cache.h> +#include <asm-generic/vmlinux.lds.h> #include <asm/boot.h>Why do we need this chunk?
Quoted 2 lines
the cache size is defined in
cache.h, so I need movevmlinux.lds.haftercache.h
The right fix is to move the inclusion of cache.h to asm-generic/vmlinux.lds.h. A quick audit only found sparc that failed to guard non assembler stuff.
Sam
Quoted 35 lines · Sam Ravnborg
Quoted 34 lines · Shaohua Li
Quoted 33 lines · Tejun Heo
Hello,
Quoted 1 line · Shaohua Li
How about this one?
Much better. :-)
include/asm-generic/vmlinux.lds.h line 673
Quoted 6 lines
include/asm-generic/vmlinux.lds.h +6
-0+#ifdef INTERNODE_CACHE_BYTES +#define INTERNODE_CACHEALIGNED \ + . = ALIGN(INTERNODE_CACHE_BYTES); +#else +#define INTERNODE_CACHEALIGNED +#endif Yeah, this looks good.
Quoted 5 lines
Index:
linux/arch/x86/kernel/vmlinux.lds.S
===================================================================arch/x86/kernel/vmlinux.lds.S line 28
Quoted 10 lines
arch/x86/kernel/vmlinux.lds.S +1
-1@@ -20,11 +20,11 @@ #define LOAD_OFFSET __START_KERNEL_map #endif -#include <asm-generic/vmlinux.lds.h> #include <asm/asm-offsets.h> #include <asm/thread_info.h> #include <asm/page_types.h> #include <asm/cache.h> +#include <asm-generic/vmlinux.lds.h> #include <asm/boot.h>Why do we need this chunk?
Quoted 5 lines
Quoted 2 lines
the cache size is defined in
cache.h, so I need movevmlinux.lds.haftercache.hThe right fix is to move the inclusion of
cache.htoasm-generic/vmlinux.lds.h. A quick audit only found sparc that failed to guard non assembler stuff.
with this, we need check every arch, at least doing a compile. I'm afraid I can't, sorry.
Hello, Shaohua.
Quoted 12 lines · Shaohua Li
Quoted 12 lines
Quoted 11 lines
Quoted 9 lines
arch/x86/kernel/vmlinux.lds.S line 28
Quoted 7 lines
arch/x86/kernel/vmlinux.lds.S +1
-1-#include <asm-generic/vmlinux.lds.h> #include <asm/asm-offsets.h> #include <asm/thread_info.h> #include <asm/page_types.h> #include <asm/cache.h> +#include <asm-generic/vmlinux.lds.h> #include <asm/boot.h>Why do we need this chunk?
the cache size is defined in
cache.h, so I need movevmlinux.lds.haftercache.h
Quoted 6 lines
Quoted 3 lines
The right fix is to move the inclusion of
cache.htoasm-generic/vmlinux.lds.h. A quick audit only found sparc that failed to guard non assembler stuff.with this, we need check every arch, at least doing a compile. I'm afraid I can't, sorry.
Not being able to cross build every arch is okay but you at least need to make an effort to make things easily applicable to other archs and avoid adding subtle ugliness like the above. Please at least try to look at other arch codes and see how things can be made to work across different archs. Setting up cross compilers for the major archs, for example, sparc, power and ia64 isn't that difficult either.
Thanks.
Signature · 1 line
tejun
Quoted 22 lines · Tejun Heo
Hello, Shaohua.
Quoted 12 lines · Shaohua Li
Quoted 12 lines
Quoted 11 lines
Quoted 9 lines
arch/x86/kernel/vmlinux.lds.S line 28
Quoted 7 lines
arch/x86/kernel/vmlinux.lds.S +1
-1-#include <asm-generic/vmlinux.lds.h> #include <asm/asm-offsets.h> #include <asm/thread_info.h> #include <asm/page_types.h> #include <asm/cache.h> +#include <asm-generic/vmlinux.lds.h> #include <asm/boot.h>Why do we need this chunk?
the cache size is defined in
cache.h, so I need movevmlinux.lds.haftercache.hQuoted 6 lines
Quoted 3 lines
The right fix is to move the inclusion of
cache.htoasm-generic/vmlinux.lds.h. A quick audit only found sparc that failed to guard non assembler stuff.with this, we need check every arch, at least doing a compile. I'm afraid I can't, sorry.
Quoted 6 lines
Not being able to cross build every arch is okay but you at least need to make an effort to make things easily applicable to other archs and avoid adding subtle ugliness like the above. Please at least try to look at other arch codes and see how things can be made to work across different archs. Setting up cross compilers for the major archs, for example, sparc, power and ia64 isn't that difficult either.
This still needs I fix every arch, for example, as Sam pointed out, spark build will fail. I really have the bandwidth and capability to do this. Increment changes are always preferred. My original patch is trying to follow increment changes way.
Hello,
Quoted 11 lines · Shaohua Li
Quoted 6 lines
Not being able to cross build every arch is okay but you at least need to make an effort to make things easily applicable to other archs and avoid adding subtle ugliness like the above. Please at least try to look at other arch codes and see how things can be made to work across different archs. Setting up cross compilers for the major archs, for example, sparc, power and ia64 isn't that difficult either.
This still needs I fix every arch, for example, as Sam pointed out, spark build will fail. I really have the bandwidth and capability to do this. Increment changes are always preferred. My original patch is trying to follow increment changes way.
Yeah, well, in my experience, those approaches usually just end up piling partial conversions on top of different set of partial conversions. It's not like we're talking about a major arch dependent feature it's just a matter of looking through different arch codes, maybe test building some, pushing things to linux-next and dealing with fallouts if there is any. I'll try to do it myself.
Thanks.
Signature · 1 line
tejun
Quoted 15 lines · Tejun Heo
Hello,
Quoted 11 lines · Shaohua Li
Quoted 6 lines
Not being able to cross build every arch is okay but you at least need to make an effort to make things easily applicable to other archs and avoid adding subtle ugliness like the above. Please at least try to look at other arch codes and see how things can be made to work across different archs. Setting up cross compilers for the major archs, for example, sparc, power and ia64 isn't that difficult either.
This still needs I fix every arch, for example, as Sam pointed out, spark build will fail. I really have the bandwidth and capability to do this. Increment changes are always preferred. My original patch is trying to follow increment changes way.
Quoted 6 lines
Yeah, well, in my experience, those approaches usually just end up piling partial conversions on top of different set of partial conversions. It's not like we're talking about a major arch dependent feature it's just a matter of looking through different arch codes, maybe test building some, pushing things to linux-next and dealing with fallouts if there is any. I'll try to do it myself.
Thanks for doing it and sorry I didn't do it.
The first patch in the series isn't related to this issue, please consider merge it.
Thanks,
Shaohua
Quoted 2 lines · Shaohua Li
The first patch in the series isn't related to this issue, please consider merge it.
Can you please re-post it? Thanks.
Signature · 1 line
tejun
Quoted 5 lines · Tejun Heo
Quoted 2 lines · Shaohua Li
The first patch in the series isn't related to this issue, please consider merge it.
Can you please re-post it? Thanks.
here it is.
Subject: make readmostly section correctly align
readmostly section should end at cache line aligned address, otherwise the last several data might share cachline with other data and make the readmostly data still have cache bounce.
For example, in ia64, secpath_cachep is the last readmostly data, and it shares cacheline with init_uts_ns.a000000100e80480 d secpath_cachepa000000100e80488 D init_uts_ns
--- include/asm-generic/vmlinux.lds.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/asm-generic/vmlinux.lds.h +2 -1
@@ -192,7 +192,8 @@#define READ_MOSTLY_DATA (align) \ . = ALIGN(align); \ - *(.data..read_mostly) + *(.data..read_mostly) \ + . = ALIGN(align);#define CACHELINE_ALIGNED_DATA (align) \ . = ALIGN(align); \
Tejun Heo
Acked-byHello,
Quoted 11 lines · Shaohua Li
Subject: make readmostly section correctly align
readmostly section should end at cache line aligned address, otherwise the last several data might share cachline with other data and make the readmostly data still have cache bounce.
For example, in ia64,secpath_cachepis the last readmostly data, and it shares cacheline withinit_uts_ns.a000000100e80480dsecpath_cachepa000000100e80488Dinit_uts_nsSigned-off-by Shaohua Li <[email protected]>
Yeap, this looks good to me.
I can route this through percpu tree but think -mm is the better fit. Andrew, can you please take this one?
Thank you.
Signature · 1 line
tejun