percpu: introduce read mostly percpu API
Subsystems (2)
PATCH 1/2 percpu: introduce read mostly percpu API
Add a new readmostly percpu section and api, next patch will use it.
--- include/asm-generic/vmlinux.lds.h | 4 ++++ include/linux/percpu-defs.h | 9 +++++++++ 2 files changed, 13 insertions(+)
include/asm-generic/vmlinux.lds.h +4 -0
@@ -677,6 +677,8 @@ - LOAD_OFFSET) { \ VMLINUX_SYMBOL(__per_cpu_start) = .; \ *(.data..percpu..first) \ + . = ALIGN(PAGE_SIZE); \ + *(.data..percpu..readmostly) \ *(.data..percpu..page_aligned) \ *(.data..percpu) \ *(.data..percpu..shared_aligned) \
@@ -703,6 +705,8 @@ VMLINUX_SYMBOL(__per_cpu_load) = .; \ VMLINUX_SYMBOL(__per_cpu_start) = .; \ *(.data..percpu..first) \ + . = ALIGN(PAGE_SIZE); \ + *(.data..percpu..readmostly) \ *(.data..percpu..page_aligned) \ *(.data..percpu) \ *(.data..percpu..shared_aligned) \
include/linux/percpu-defs.h +9 -0
@@ -139,6 +139,15 @@__aligned(PAGE_SIZE) /* +* Declaration/definition used for per-CPU variables that must be read mostly. +*/ +#define DECLARE_PER_CPU_READ_MOSTLY (type, name) \ + DECLARE_PER_CPU_SECTION(type, name, "..readmostly") + +#define DEFINE_PER_CPU_READ_MOSTLY (type, name) \ + DEFINE_PER_CPU_SECTION(type, name, "..readmostly") + +/* * Intermodule exports for per-CPU variables. sparse forgets about * address space across EXPORT_SYMBOL(), change EXPORT_SYMBOL() to * noop if __CHECKER__.
Quoted 4 lines · Shaohua Li
Add a new readmostly percpu section and api, next patch will use it.
Signed-off-by Shaohua Li <[email protected]>---
Could you precisely describe why grouping together read mostly percpu variables is a win ? Especially when you add in your next patch a single variable ?
Quoted 9 lines
include/asm-generic/vmlinux.lds.h | 4 ++++ include/linux/percpu-defs.h | 9 +++++++++ 2 files changed, 13 insertions(+)Index:
linux/include/asm-generic/vmlinux.lds.h
===================================================================
include/asm-generic/vmlinux.lds.h line 684
Quoted 8 lines
include/asm-generic/vmlinux.lds.h +2
-0@@ -677,6 +677,8 @@ - LOAD_OFFSET) { \ VMLINUX_SYMBOL(__per_cpu_start) = .; \ *(.data..percpu..first) \ + . = ALIGN(PAGE_SIZE); \ + *(.data..percpu..readmostly) \ *(.data..percpu..page_aligned) \ *(.data..percpu) \ *(.data..percpu..shared_aligned) \
So percpu..page_aligned is not any more aligned to a PAGE ? or we have a big hole before it ? Hmm....
Maybe you should put first data..percpu..page_aligned, then align to one cache line (L1_CACHE_BYTES), then data..percpu..readmostly, so that hole is small.
We should take care of not introducing too much holes in percpu zone. Maybe using a new subzone ".data..percpu..small_objects" to put small objects in it.
nm -v vmlinux | grep -10 sockets_in_use # select one random part
0000000000011e0c d cpu_min_freq0000000000011e10 d cpu_cur_freq0000000000011e14 d cpu_set_freq0000000000011e18 d cpu_is_managed0000000000011e20 d od_cpu_dbs_info0000000000011fa0 d cs_cpu_dbs_info00000000000120f0 d cpufreq_show_table
0000000000012100 D cpuidle_devices
0000000000012120 d ladder_devices
0000000000012200 d menu_devices00000000000122c0 d sockets_in_use # object_size = 4 , hole = 28bytes00000000000122e0 d prot_inuse00000000000123e0 D nf_conntrack_untracked
0000000000012560 d rt_cache_stat00000000000125a0 d ipv4_cookie_scratch
0000000000012740 D init_tss0000000000014a00 D irq_stat0000000000014a40 D cpu_info0000000000014b00 d hv_clock0000000000014b40 D cpu_tlbstate0000000000014b80 d runqueues
We can see many holes because of 2^5 alignments of individual .o .data..percpu sections.
find . -name "*.o"|xargs objdump -h|grep percpu
Linker promotes a section alignment from natural alignment to 2^5 as soon as the size reaches 2^5
For example in net/ipv4/route.o, we have a per_cpu structure (rt_cache_stat), that is an array of 16 integers. The natural alignement should be 4 (alignof(int)), but we get :
# objdump -h net/ipv4/route.o|grep percpu 19 .data..percpu 00000040 0000000000000000 0000000000000000 00007a80 2**5
For a section replicated N times, this really is a concern.
Thanks !
Quoted 5 lines · Eric Dumazet
Could you precisely describe why grouping together read mostly percpu variables is a win ? Especially when you add in your next patch a single variable ?
The logic is that those variables are less likely to end up dirty in the cache. It is, however, much less of an issue for percpu variables, since dirty lines there aren't quite so likely to bounce around.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Quoted 10 lines · Eric Dumazet
Quoted 4 lines · Shaohua Li
Add a new readmostly percpu section and api, next patch will use it.
Signed-off-by Shaohua Li <[email protected]>---
Could you precisely describe why grouping together read mostly percpu variables is a win ? Especially when you add in your next patch a single variable ?
Not Shaohua, but I can explain it:
There is some per cpu data which is read by other CPUs. In this case it's a win to use the separate section because you avoid false sharing and the cache line can be kept in shared mode on all CPUs.
The next patch has an example of such data: data which is read by another CPU to send something to the target CPU.
I think the concept is useful and makes a lot of sense. The alternative would be __read_mostly NR_CPUS arrays, but we all know that is a bad idea because it wastes too much memory on CONFIG_MAX_SMP setups.
-Andi
Quoted 18 lines · Andi Kleen
Quoted 10 lines · Eric Dumazet
Quoted 4 lines · Shaohua Li
Add a new readmostly percpu section and api, next patch will use it.
Signed-off-by Shaohua Li <[email protected]>---
Could you precisely describe why grouping together read mostly percpu variables is a win ? Especially when you add in your next patch a single variable ?
Not Shaohua, but I can explain it:
There is some per cpu data which is read by other CPUs. In this case it's a win to use the separate section because you avoid false sharing and the cache line can be kept in shared mode on all CPUs.
Quoted 8 lines
The next patch has an example of such data: data which is read by another CPU to send something to the target CPU.
I think the concept is useful and makes a lot of sense. The alternative would be
__read_mostlyNR_CPUSarrays, but we all know that is a bad idea because it wastes too much memory onCONFIG_MAX_SMPsetups.
My question was more a rethoric one. I understand for sure the intent.
All this should be explained in changelog, so that people know when/where use this new class of per_cpu variables ;)
Thanks
Quoted 6 lines · Eric Dumazet
My question was more a rethoric one. I understand for sure the intent.
All this should be explained in changelog, so that people know when/where use this new class of
per_cpuvariables ;)
Changelog really isn't the best place for this, a Documentation file would be better.
I'm trying to grok the intended semantic of the shared_aligned section right now... I'm not sure if there is a significant difference between the read mostly and the shared aligned section?
-hpa
Quoted 11 lines · H. Peter Anvin
Quoted 6 lines · Eric Dumazet
My question was more a rethoric one. I understand for sure the intent.
All this should be explained in changelog, so that people know when/where use this new class of
per_cpuvariables ;)Changelog really isn't the best place for this, a Documentation file would be better.
Quoted 6 lines
I'm trying to grok the intended semantic of the
shared_alignedsection right now... I'm not sure if there is a significant difference between the read mostly and the shared aligned section?From the looks of it, it's a manual way to do what the linker would,
quite frankly, do better with SORT_BY_ALIGNMENT() -- although by separating it out it might still be a win to avoid lots of "N+1"-sized sections aligned to N.
-hpa
Quoted 9 lines · Eric Dumazet
We can see many holes because of 2^5 alignments of individual .o .data..percpu sections.
find . -name "*.o"|xargs objdump -h|grep percpu
Linker promotes a section alignment from natural alignment to 2^5 as soon as the size reaches 2^5
Quoted 9 lines
For example in
net/ipv4/route.o, we have aper_cpustructure (rt_cache_stat), that is an array of 16 integers. The natural alignement should be 4 (alignof(int)), but we get :# objdump -h net/ipv4/route.o|grep percpu 19 .data..percpu 00000040 0000000000000000 0000000000000000 00007a80 2**5For a section replicated N times, this really is a concern.
That wouldn't be the linker, that would be the compiler or assembler -- I suspect it's the compiler -- and that needs to be fixed.
To reduce linker-induced padding, we may want to use SORT_BY_ALIGNMENT() in the linker script.
-hpa
[tip:x86/mm] percpu: Introduce a read-mostly percpu API
In reply to Shaohua LiCommit-ID: c957ef2c59e952803766ddc22e89981ab534606f Gitweb: http://git.kernel.org/tip/c957ef2c59e952803766ddc22e89981ab534606f Author: Shaohua Li <[email protected]> AuthorDate: Wed, 20 Oct 2010 11:07:02 +0800 Committer: H. Peter Anvin <[email protected]> CommitDate: Wed, 20 Oct 2010 14:33:58 -0700
percpu: Introduce a read-mostly percpu API
Add a new readmostly percpu section and API. This can be used to avoid dirtying data lines which are generally not written to, which is especially important for data which may be accessed by processors other than the one for which the percpu area belongs to.
[ hpa: moved it *after* the page-aligned section, for obvious
reasons. ]
LKML-Reference: <[email protected]>
--- include/asm-generic/vmlinux.lds.h | 4 ++++ include/linux/percpu-defs.h | 9 +++++++++ 2 files changed, 13 insertions(+), 0 deletions(-)
include/asm-generic/vmlinux.lds.h +4 -0
@@ -677,7 +677,9 @@ - LOAD_OFFSET) { \ VMLINUX_SYMBOL(__per_cpu_start) = .; \ *(.data..percpu..first) \ + . = ALIGN(PAGE_SIZE); \ *(.data..percpu..page_aligned) \ + *(.data..percpu..readmostly) \ *(.data..percpu) \ *(.data..percpu..shared_aligned) \ VMLINUX_SYMBOL(__per_cpu_end) = .; \
@@ -703,6 +705,8 @@ VMLINUX_SYMBOL(__per_cpu_load) = .; \ VMLINUX_SYMBOL(__per_cpu_start) = .; \ *(.data..percpu..first) \ + . = ALIGN(PAGE_SIZE); \ + *(.data..percpu..readmostly) \ *(.data..percpu..page_aligned) \ *(.data..percpu) \ *(.data..percpu..shared_aligned) \
include/linux/percpu-defs.h +9 -0
@@ -139,6 +139,15 @@__aligned(PAGE_SIZE) /* +* Declaration/definition used for per-CPU variables that must be read mostly. +*/ +#define DECLARE_PER_CPU_READ_MOSTLY (type, name) \ + DECLARE_PER_CPU_SECTION(type, name, "..readmostly") + +#define DEFINE_PER_CPU_READ_MOSTLY (type, name) \ + DEFINE_PER_CPU_SECTION(type, name, "..readmostly") + +/* * Intermodule exports for per-CPU variables. sparse forgets about * address space across EXPORT_SYMBOL(), change EXPORT_SYMBOL() to * noop if __CHECKER__.
Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API
Hi hpa,
Quoted 2 lines · tip-bot for Shaohua Li
[ hpa: moved it *after* the page-aligned section, for obvious
reasons. ]
move it after page-aligned section can't guarantee to avoid cache false sharing. we need:
. = ALIGN(CACHE_LINE_SIZE);
*(.data..percpu..readmostly)
. = ALIGN(CACHE_LINE_SIZE);
To make sure before/after readmostly section doesn't share cache line with others. The headcache is some arch need L1_CACHE_SIZE and others need (1 << INTERNODE_CACHE_SHIFT). We need change all ARCHs to make this happen.
Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API
I know, but I can't do that immediately. It is an optimizatiin we can do for .38.
Quoted 15 lines · Shaohua Li
Hi hpa,
Quoted 2 lines · tip-bot for Shaohua Li
[ hpa: moved it *after* the page-aligned section, for obvious
reasons. ]move it after page-aligned section can't guarantee to avoid cache false sharing.
we need:
. =ALIGN(CACHE_LINE_SIZE);
*(.data..percpu..readmostly)
. =ALIGN(CACHE_LINE_SIZE);
To make sure before/after readmostly section doesn't share cache line with
others. The headcache is some arch needL1_CACHE_SIZEand others need (1 <<INTERNODE_CACHE_SHIFT). We need change all ARCHs to make this happen.
Signature · 1 line
Sent from my mobile phone. Please pardon any lack of formatting.
Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API
In reply to Shaohua LiQuoted 4 lines · Shaohua Li
Hi hpa,
Quoted 2 lines · tip-bot for Shaohua Li
[ hpa: moved it *after* the page-aligned section, for obvious
reasons. ]
Quoted 8 lines
move it after page-aligned section can't guarantee to avoid cache false sharing. we need:
. =ALIGN(CACHE_LINE_SIZE);
*(.data..percpu..readmostly)
. =ALIGN(CACHE_LINE_SIZE);
To make sure before/after readmostly section doesn't share cache line with others. The headcache is some arch needL1_CACHE_SIZEand others need (1 <<INTERNODE_CACHE_SHIFT). We need change all ARCHs to make this happen.
Really, readmostly percpu variables dont need INTERNODE_CACHE_SHIFT, as this is read mostly, and per cpu vars ;)
Just use L1_CACHE_BYTES
*(.data..percpu..first) \
*(.data..percpu..page_aligned) \
+ . = ALIGN(L1_CACHE_BYTES \
+ *(.data..percpu..readmostly) \
+ . = ALIGN(L1_CACHE_BYTES \
*(.data..percpu) \Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API
Quoted 6 lines · Eric Dumazet
Really, readmostly percpu variables dont need
INTERNODE_CACHE_SHIFT, as this is read mostly, and per cpu vars ;)Just use
L1_CACHE_BYTES
L1_CACHE_BYTES is completely pointless, since if there is sharing to worry about *at all*, it's probably at the L2 or L3 cache levels.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API
Quoted 3 lines · H. Peter Anvin
L1_CACHE_BYTESis completely pointless, since if there is sharing to worry about *at all*, it's probably at the L2 or L3 cache levels.
I see, and we dont have better way to express this hint/requirement than using PAGE_SIZE or INTERNODE_CACHE_SHIFT ?
In your patch you force a PAGE_SIZE alignement *before* *(.data..percpu..page_aligned)
If this alignment is really needed, this should be part of another patch, since this fixes a previous bug in 2.6.36 ?
Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API
Quoted 6 lines · Eric Dumazet
Quoted 3 lines · H. Peter Anvin
L1_CACHE_BYTESis completely pointless, since if there is sharing to worry about *at all*, it's probably at the L2 or L3 cache levels.
Quoted 9 lines
I see, and we dont have better way to express this hint/requirement than using
PAGE_SIZEorINTERNODE_CACHE_SHIFT?In your patch you force a
PAGE_SIZEalignement *before* *(.data..percpu..page_aligned)If this alignment is really needed, this should be part of another patch, since this fixes a previous bug in 2.6.36 ?
It was part of Shaohua's patch... I don't know if it does anything useful, but it also doesn't hurt.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API
Quoted 13 lines · H. Peter Anvin
Quoted 12 lines · Eric Dumazet
Quoted 3 lines · H. Peter Anvin
L1_CACHE_BYTESis completely pointless, since if there is sharing to worry about *at all*, it's probably at the L2 or L3 cache levels.I see, and we dont have better way to express this hint/requirement than using
PAGE_SIZEorINTERNODE_CACHE_SHIFT?In your patch you force a
PAGE_SIZEalignement *before* *(.data..percpu..page_aligned)
Quoted 6 lines
Quoted 3 lines
If this alignment is really needed, this should be part of another patch, since this fixes a previous bug in 2.6.36 ?
It was part of Shaohua's patch... I don't know if it does anything useful, but it also doesn't hurt.
This isn't a previous bug. see
#define DECLARE_PER_CPU_PAGE_ALIGNED(type, name) \ DECLARE_PER_CPU_SECTION(type, name, "..page_aligned") \ __aligned(PAGE_SIZE)
the ..page_aligned is already page aligned. I add it is because it can make the .readmostly section guarantee to have no cache false sharing, because I add the . = ALIGN(PAGE_SIZE); before .readmostly section, and .page_aligned follows, but for sure this wates some memory.
Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API
Quoted 8 lines · Shaohua Li
This isn't a previous bug. see
#define DECLARE_PER_CPU_PAGE_ALIGNED(type, name) \ DECLARE_PER_CPU_SECTION(type, name, "..page_aligned") \ __aligned(PAGE_SIZE)the ..page_aligned is already page aligned. I add it is because it can make the .readmostly section guarantee to have no cache false sharing, because I add the . =
ALIGN(PAGE_SIZE); before .readmostly section, and .page_aligned follows, but for sure this wates some memory.
A small note:
The __aligned(XXX) makes sure object _starts_ at XXX boundary, not that following one will also use same alignment. Of course, we can argue that we dont try to put in PAGE_ALIGNED section small objects, but still...
Explicit .ALIGN() uses in .lds are more readable IMHO...
Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API
Quoted 10 lines · Eric Dumazet
Quoted 8 lines · Shaohua Li
This isn't a previous bug. see
#define DECLARE_PER_CPU_PAGE_ALIGNED(type, name) \ DECLARE_PER_CPU_SECTION(type, name, "..page_aligned") \ __aligned(PAGE_SIZE)the ..page_aligned is already page aligned. I add it is because it can make the .readmostly section guarantee to have no cache false sharing, because I add the . =
ALIGN(PAGE_SIZE); before .readmostly section, and .page_aligned follows, but for sure this wates some memory.
Quoted 8 lines
A small note:
The
__aligned(XXX)makes sure object _starts_ at XXX boundary, not that following one will also use same alignment. Of course, we can argue that we dont try to put inPAGE_ALIGNEDsection small objects, but still...Explicit .ALIGN() uses in .lds are more readable IMHO...
There is. However, see again my previous note about SORT_BY_ALIGNMENT.
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API
In reply to tip-bot for Shaohua LiQuoted 16 lines · tip-bot for Shaohua Li
include/asm-generic/vmlinux.lds.h +2
-0@@ -677,7 +677,9 @@ - LOAD_OFFSET) { \ VMLINUX_SYMBOL(__per_cpu_start) = .; \ *(.data..percpu..first) \ + . = ALIGN(PAGE_SIZE); \ *(.data..percpu..page_aligned) \ + *(.data..percpu..readmostly) \ *(.data..percpu) \ *(.data..percpu..shared_aligned) \ VMLINUX_SYMBOL(__per_cpu_end) = .; \
include/asm-generic/vmlinux.lds.h line 712
Quoted 8 lines
include/asm-generic/vmlinux.lds.h +2
-0@@ -703,6 +705,8 @@ VMLINUX_SYMBOL(__per_cpu_load) = .; \ VMLINUX_SYMBOL(__per_cpu_start) = .; \ *(.data..percpu..first) \ + . = ALIGN(PAGE_SIZE); \ + *(.data..percpu..readmostly) \ *(.data..percpu..page_aligned) \ *(.data..percpu) \ *(.data..percpu..shared_aligned) \
ARGH... looks like I didn't catch it everywhere...
-hpa
Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.
[tip:x86/mm] x86-32, percpu: Correct the ordering of the percpu readmostly section
In reply to Shaohua LiCommit-ID: 2aeb66d3036dbafc297ac553a257a40283dadb3e Gitweb: http://git.kernel.org/tip/2aeb66d3036dbafc297ac553a257a40283dadb3e Author: H. Peter Anvin <[email protected]> AuthorDate: Thu, 21 Oct 2010 00:15:00 -0700 Committer: H. Peter Anvin <[email protected]> CommitDate: Thu, 21 Oct 2010 00:15:00 -0700
x86-32, percpu: Correct the ordering of the percpu readmostly section
Checkin c957ef2c59e952803766ddc22e89981ab534606f had inconsistent ordering of .data..percpu..page_aligned and .data..percpu..readmostly; the still-broken version affected x86-32 at least.
The page aligned version really must be page aligned...
LKML-Reference: <[email protected]>
Cc: Shaohua Li <[email protected]>
Cc: Eric Dumazet <[email protected]>
--- include/asm-generic/vmlinux.lds.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
include/asm-generic/vmlinux.lds.h +1 -1
@@ -706,8 +706,8 @@ VMLINUX_SYMBOL(__per_cpu_start) = .; \ *(.data..percpu..first) \ . = ALIGN(PAGE_SIZE); \ - *(.data..percpu..readmostly) \ *(.data..percpu..page_aligned) \ + *(.data..percpu..readmostly) \ *(.data..percpu) \ *(.data..percpu..shared_aligned) \ VMLINUX_SYMBOL(__per_cpu_end) = .; \