Skip to contentSkip to conversation

percpu: introduce read mostly percpu API

PATCH In review Shaohua Li 4 participants 20 messages Oct 20, 2010 – Oct 21, 2010

Subsystems (2)
lore
Share & tools
lore

Shaohua Li

author
#0 lore raw

PATCH 1/2 percpu: introduce read mostly percpu API

Add a new readmostly percpu section and api, next patch will use it.

---
 include/asm-generic/vmlinux.lds.h |    4 ++++
 include/linux/percpu-defs.h       |    9 +++++++++
 2 files changed, 13 insertions(+)
include/asm-generic/vmlinux.lds.h +4 -0
Index: linux/include/asm-generic/vmlinux.lds.h
===================================================================
@@ -677,6 +677,8 @@
 				- LOAD_OFFSET) {			\
 		VMLINUX_SYMBOL(__per_cpu_start) = .;			\
 		*(.data..percpu..first)					\
+		. = ALIGN(PAGE_SIZE);					\
+		*(.data..percpu..readmostly)				\
 		*(.data..percpu..page_aligned)				\
 		*(.data..percpu)					\
 		*(.data..percpu..shared_aligned)			\
@@ -703,6 +705,8 @@
 		VMLINUX_SYMBOL(__per_cpu_load) = .;			\
 		VMLINUX_SYMBOL(__per_cpu_start) = .;			\
 		*(.data..percpu..first)					\
+		. = ALIGN(PAGE_SIZE);					\
+		*(.data..percpu..readmostly)				\
 		*(.data..percpu..page_aligned)				\
 		*(.data..percpu)					\
 		*(.data..percpu..shared_aligned)			\
include/linux/percpu-defs.h +9 -0
Index: linux/include/linux/percpu-defs.h
===================================================================
@@ -139,6 +139,15 @@
 	__aligned(PAGE_SIZE)
 
 /*
+ * Declaration/definition used for per-CPU variables that must be read mostly.
+ */
+#define DECLARE_PER_CPU_READ_MOSTLY(type, name)			\
+	DECLARE_PER_CPU_SECTION(type, name, "..readmostly")
+
+#define DEFINE_PER_CPU_READ_MOSTLY(type, name)				\
+	DEFINE_PER_CPU_SECTION(type, name, "..readmostly")
+
+/*
  * Intermodule exports for per-CPU variables.  sparse forgets about
  * address space across EXPORT_SYMBOL(), change EXPORT_SYMBOL() to
  * noop if __CHECKER__.
#1 lore raw
Quoted 4 lines · Shaohua Li

Add a new readmostly percpu section and api, next patch will use it.

---

Could you precisely describe why grouping together read mostly percpu variables is a win ? Especially when you add in your next patch a single variable ?

Quoted 9 lines
 include/asm-generic/vmlinux.lds.h |    4 ++++
 include/linux/percpu-defs.h       |    9 +++++++++
 2 files changed, 13 insertions(+)

Index: linux/include/asm-generic/vmlinux.lds.h
===================================================================

include/asm-generic/vmlinux.lds.h line 684

Quoted 8 lines
include/asm-generic/vmlinux.lds.h +2 -0
@@ -677,6 +677,8 @@
 				- LOAD_OFFSET) {			\
 		VMLINUX_SYMBOL(__per_cpu_start) = .;			\
 		*(.data..percpu..first)					\
+		. = ALIGN(PAGE_SIZE);					\
+		*(.data..percpu..readmostly)				\
 		*(.data..percpu..page_aligned)				\
 		*(.data..percpu)					\
 		*(.data..percpu..shared_aligned)			\

So percpu..page_aligned is not any more aligned to a PAGE ? or we have a big hole before it ? Hmm....

Maybe you should put first data..percpu..page_aligned, then align to one cache line (L1_CACHE_BYTES), then data..percpu..readmostly, so that hole is small.

We should take care of not introducing too much holes in percpu zone. Maybe using a new subzone ".data..percpu..small_objects" to put small objects in it.

nm -v vmlinux | grep -10 sockets_in_use # select one random part

0000000000011e0c d cpu_min_freq
0000000000011e10 d cpu_cur_freq
0000000000011e14 d cpu_set_freq
0000000000011e18 d cpu_is_managed
0000000000011e20 d od_cpu_dbs_info
0000000000011fa0 d cs_cpu_dbs_info
00000000000120f0 d cpufreq_show_table
0000000000012100 D cpuidle_devices
0000000000012120 d ladder_devices
0000000000012200 d menu_devices
00000000000122c0 d sockets_in_use # object_size = 4 , hole = 28bytes
00000000000122e0 d prot_inuse
00000000000123e0 D nf_conntrack_untracked
0000000000012560 d rt_cache_stat
00000000000125a0 d ipv4_cookie_scratch
0000000000012740 D init_tss
0000000000014a00 D irq_stat
0000000000014a40 D cpu_info
0000000000014b00 d hv_clock
0000000000014b40 D cpu_tlbstate
0000000000014b80 d runqueues

We can see many holes because of 2^5 alignments of individual .o .data..percpu sections.

find . -name "*.o"|xargs objdump -h|grep percpu

Linker promotes a section alignment from natural alignment to 2^5 as soon as the size reaches 2^5

For example in net/ipv4/route.o, we have a per_cpu structure (rt_cache_stat), that is an array of 16 integers. The natural alignement should be 4 (alignof(int)), but we get :

# objdump -h net/ipv4/route.o|grep percpu
 19 .data..percpu 00000040  0000000000000000  0000000000000000  00007a80  2**5

For a section replicated N times, this really is a concern.

Thanks !

#2 lore raw
Quoted 5 lines · Eric Dumazet

Could you precisely describe why grouping together read mostly percpu variables is a win ? Especially when you add in your next patch a single variable ?

The logic is that those variables are less likely to end up dirty in the cache. It is, however, much less of an issue for percpu variables, since dirty lines there aren't quite so likely to bounce around.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#3 lore raw
In reply to Eric Dumazet
Quoted 10 lines · Eric Dumazet
Quoted 4 lines · Shaohua Li

Add a new readmostly percpu section and api, next patch will use it.

---

Could you precisely describe why grouping together read mostly percpu variables is a win ? Especially when you add in your next patch a single variable ?

Not Shaohua, but I can explain it:

There is some per cpu data which is read by other CPUs. In this case it's a win to use the separate section because you avoid false sharing and the cache line can be kept in shared mode on all CPUs.

The next patch has an example of such data: data which is read by another CPU to send something to the target CPU.

I think the concept is useful and makes a lot of sense. The alternative would be __read_mostly NR_CPUS arrays, but we all know that is a bad idea because it wastes too much memory on CONFIG_MAX_SMP setups.

-Andi

#4 lore raw
Quoted 18 lines · Andi Kleen
Quoted 10 lines · Eric Dumazet
Quoted 4 lines · Shaohua Li

Add a new readmostly percpu section and api, next patch will use it.

---

Could you precisely describe why grouping together read mostly percpu variables is a win ? Especially when you add in your next patch a single variable ?

Not Shaohua, but I can explain it:

There is some per cpu data which is read by other CPUs. In this case it's a win to use the separate section because you avoid false sharing and the cache line can be kept in shared mode on all CPUs.

Quoted 8 lines

The next patch has an example of such data: data which is read by another CPU to send something to the target CPU.

I think the concept is useful and makes a lot of sense. The alternative would be __read_mostly NR_CPUS arrays, but we all know that is a bad idea because it wastes too much memory on CONFIG_MAX_SMP setups.

My question was more a rethoric one. I understand for sure the intent.

All this should be explained in changelog, so that people know when/where use this new class of per_cpu variables ;)

Thanks

#5 lore raw
Quoted 6 lines · Eric Dumazet

My question was more a rethoric one. I understand for sure the intent.

All this should be explained in changelog, so that people know when/where use this new class of per_cpu variables ;)

Changelog really isn't the best place for this, a Documentation file would be better.

I'm trying to grok the intended semantic of the shared_aligned section right now... I'm not sure if there is a significant difference between the read mostly and the shared aligned section?

-hpa

#6 lore raw
Quoted 11 lines · H. Peter Anvin
Quoted 6 lines · Eric Dumazet

My question was more a rethoric one. I understand for sure the intent.

All this should be explained in changelog, so that people know when/where use this new class of per_cpu variables ;)

Changelog really isn't the best place for this, a Documentation file would be better.

Quoted 6 lines

I'm trying to grok the intended semantic of the shared_aligned section right now... I'm not sure if there is a significant difference between the read mostly and the shared aligned section?

From the looks of it, it's a manual way to do what the linker would,

quite frankly, do better with SORT_BY_ALIGNMENT() -- although by separating it out it might still be a win to avoid lots of "N+1"-sized sections aligned to N.

-hpa

#7 lore raw
In reply to Eric Dumazet
Quoted 9 lines · Eric Dumazet

We can see many holes because of 2^5 alignments of individual .o .data..percpu sections.

find . -name "*.o"|xargs objdump -h|grep percpu

Linker promotes a section alignment from natural alignment to 2^5 as soon as the size reaches 2^5

Quoted 9 lines

For example in net/ipv4/route.o, we have a per_cpu structure (rt_cache_stat), that is an array of 16 integers. The natural alignement should be 4 (alignof(int)), but we get :

# objdump -h net/ipv4/route.o|grep percpu
 19 .data..percpu 00000040  0000000000000000  0000000000000000  00007a80  2**5

For a section replicated N times, this really is a concern.

That wouldn't be the linker, that would be the compiler or assembler -- I suspect it's the compiler -- and that needs to be fixed.

To reduce linker-induced padding, we may want to use SORT_BY_ALIGNMENT() in the linker script.

-hpa

#8 lore raw

[tip:x86/mm] percpu: Introduce a read-mostly percpu API

In reply to Shaohua Li
Commit-ID:  c957ef2c59e952803766ddc22e89981ab534606f
Gitweb:     http://git.kernel.org/tip/c957ef2c59e952803766ddc22e89981ab534606f
Author:     Shaohua Li <[email protected]>
AuthorDate: Wed, 20 Oct 2010 11:07:02 +0800
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Wed, 20 Oct 2010 14:33:58 -0700

percpu: Introduce a read-mostly percpu API

Add a new readmostly percpu section and API. This can be used to avoid dirtying data lines which are generally not written to, which is especially important for data which may be accessed by processors other than the one for which the percpu area belongs to.

[ hpa: moved it *after* the page-aligned section, for obvious
reasons. ]

LKML-Reference: <[email protected]>

---
 include/asm-generic/vmlinux.lds.h |    4 ++++
 include/linux/percpu-defs.h       |    9 +++++++++
 2 files changed, 13 insertions(+), 0 deletions(-)
include/asm-generic/vmlinux.lds.h +4 -0
index 8a92a17..d7e7b21 100644
@@ -677,7 +677,9 @@
 				- LOAD_OFFSET) {			\
 		VMLINUX_SYMBOL(__per_cpu_start) = .;			\
 		*(.data..percpu..first)					\
+		. = ALIGN(PAGE_SIZE);					\
 		*(.data..percpu..page_aligned)				\
+		*(.data..percpu..readmostly)				\
 		*(.data..percpu)					\
 		*(.data..percpu..shared_aligned)			\
 		VMLINUX_SYMBOL(__per_cpu_end) = .;			\
@@ -703,6 +705,8 @@
 		VMLINUX_SYMBOL(__per_cpu_load) = .;			\
 		VMLINUX_SYMBOL(__per_cpu_start) = .;			\
 		*(.data..percpu..first)					\
+		. = ALIGN(PAGE_SIZE);					\
+		*(.data..percpu..readmostly)				\
 		*(.data..percpu..page_aligned)				\
 		*(.data..percpu)					\
 		*(.data..percpu..shared_aligned)			\
include/linux/percpu-defs.h +9 -0
index ce2dc65..27ef6b1 100644
@@ -139,6 +139,15 @@
 	__aligned(PAGE_SIZE)
 
 /*
+ * Declaration/definition used for per-CPU variables that must be read mostly.
+ */
+#define DECLARE_PER_CPU_READ_MOSTLY(type, name)			\
+	DECLARE_PER_CPU_SECTION(type, name, "..readmostly")
+
+#define DEFINE_PER_CPU_READ_MOSTLY(type, name)				\
+	DEFINE_PER_CPU_SECTION(type, name, "..readmostly")
+
+/*
  * Intermodule exports for per-CPU variables.  sparse forgets about
  * address space across EXPORT_SYMBOL(), change EXPORT_SYMBOL() to
  * noop if __CHECKER__.
author
#9 lore raw

Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API

Hi hpa,

Quoted 2 lines · tip-bot for Shaohua Li

[ hpa: moved it *after* the page-aligned section, for obvious
reasons. ]

move it after page-aligned section can't guarantee to avoid cache false sharing. we need:
. = ALIGN(CACHE_LINE_SIZE);
*(.data..percpu..readmostly)
. = ALIGN(CACHE_LINE_SIZE);
To make sure before/after readmostly section doesn't share cache line with others. The headcache is some arch need L1_CACHE_SIZE and others need (1 << INTERNODE_CACHE_SHIFT). We need change all ARCHs to make this happen.

#10 lore raw

Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API

I know, but I can't do that immediately. It is an optimizatiin we can do for .38.

Quoted 15 lines · Shaohua Li

Hi hpa,

Quoted 2 lines · tip-bot for Shaohua Li

[ hpa: moved it *after* the page-aligned section, for obvious
reasons. ]

move it after page-aligned section can't guarantee to avoid cache false sharing.
we need:
. = ALIGN(CACHE_LINE_SIZE);
*(.data..percpu..readmostly)
. = ALIGN(CACHE_LINE_SIZE);
To make sure before/after readmostly section doesn't share cache line with
others. The headcache is some arch need L1_CACHE_SIZE and others need (1 << INTERNODE_CACHE_SHIFT). We need change all ARCHs to make this happen.

Signature · 1 line
Sent from my mobile phone.  Please pardon any lack of formatting.
#11 lore raw

Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API

In reply to Shaohua Li
Quoted 4 lines · Shaohua Li

Hi hpa,

Quoted 2 lines · tip-bot for Shaohua Li

[ hpa: moved it *after* the page-aligned section, for obvious
reasons. ]

Quoted 8 lines

move it after page-aligned section can't guarantee to avoid cache false sharing. we need:
. = ALIGN(CACHE_LINE_SIZE);
*(.data..percpu..readmostly)
. = ALIGN(CACHE_LINE_SIZE);
To make sure before/after readmostly section doesn't share cache line with others. The headcache is some arch need L1_CACHE_SIZE and others need (1 << INTERNODE_CACHE_SHIFT). We need change all ARCHs to make this happen.

Really, readmostly percpu variables dont need INTERNODE_CACHE_SHIFT, as this is read mostly, and per cpu vars ;)

Just use L1_CACHE_BYTES

                *(.data..percpu..first)                           \
                *(.data..percpu..page_aligned)                    \
+               . = ALIGN(L1_CACHE_BYTES                          \
+               *(.data..percpu..readmostly)                      \
+               . = ALIGN(L1_CACHE_BYTES                          \
                *(.data..percpu)                                  \
#12 lore raw

Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API

Quoted 6 lines · Eric Dumazet

Really, readmostly percpu variables dont need INTERNODE_CACHE_SHIFT, as this is read mostly, and per cpu vars ;)

Just use L1_CACHE_BYTES

L1_CACHE_BYTES is completely pointless, since if there is sharing to worry about *at all*, it's probably at the L2 or L3 cache levels.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#13 lore raw

Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API

Quoted 3 lines · H. Peter Anvin

L1_CACHE_BYTES is completely pointless, since if there is sharing to worry about *at all*, it's probably at the L2 or L3 cache levels.

I see, and we dont have better way to express this hint/requirement than using PAGE_SIZE or INTERNODE_CACHE_SHIFT ?

In your patch you force a PAGE_SIZE alignement *before* *(.data..percpu..page_aligned)

If this alignment is really needed, this should be part of another patch, since this fixes a previous bug in 2.6.36 ?

#14 lore raw

Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API

Quoted 6 lines · Eric Dumazet
Quoted 3 lines · H. Peter Anvin

L1_CACHE_BYTES is completely pointless, since if there is sharing to worry about *at all*, it's probably at the L2 or L3 cache levels.

Quoted 9 lines

I see, and we dont have better way to express this hint/requirement than using PAGE_SIZE or INTERNODE_CACHE_SHIFT ?

In your patch you force a PAGE_SIZE alignement *before* *(.data..percpu..page_aligned)

If this alignment is really needed, this should be part of another patch, since this fixes a previous bug in 2.6.36 ?

It was part of Shaohua's patch... I don't know if it does anything useful, but it also doesn't hurt.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
author
#15 lore raw

Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API

Quoted 13 lines · H. Peter Anvin
Quoted 12 lines · Eric Dumazet
Quoted 3 lines · H. Peter Anvin

L1_CACHE_BYTES is completely pointless, since if there is sharing to worry about *at all*, it's probably at the L2 or L3 cache levels.

I see, and we dont have better way to express this hint/requirement than using PAGE_SIZE or INTERNODE_CACHE_SHIFT ?

In your patch you force a PAGE_SIZE alignement *before* *(.data..percpu..page_aligned)

Quoted 6 lines
Quoted 3 lines

If this alignment is really needed, this should be part of another patch, since this fixes a previous bug in 2.6.36 ?

It was part of Shaohua's patch... I don't know if it does anything useful, but it also doesn't hurt.

This isn't a previous bug. see

#define DECLARE_PER_CPU_PAGE_ALIGNED(type, name)			\
	DECLARE_PER_CPU_SECTION(type, name, "..page_aligned")		\
	__aligned(PAGE_SIZE)

the ..page_aligned is already page aligned. I add it is because it can make the .readmostly section guarantee to have no cache false sharing, because I add the . = ALIGN(PAGE_SIZE); before .readmostly section, and .page_aligned follows, but for sure this wates some memory.

#16 lore raw

Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API

Quoted 8 lines · Shaohua Li

This isn't a previous bug. see

#define DECLARE_PER_CPU_PAGE_ALIGNED(type, name)			\
	DECLARE_PER_CPU_SECTION(type, name, "..page_aligned")		\
	__aligned(PAGE_SIZE)

the ..page_aligned is already page aligned. I add it is because it can make the .readmostly section guarantee to have no cache false sharing, because I add the . = ALIGN(PAGE_SIZE); before .readmostly section, and .page_aligned follows, but for sure this wates some memory.

A small note:

The __aligned(XXX) makes sure object _starts_ at XXX boundary, not that following one will also use same alignment. Of course, we can argue that we dont try to put in PAGE_ALIGNED section small objects, but still...

Explicit .ALIGN() uses in .lds are more readable IMHO...

#17 lore raw

Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API

Quoted 10 lines · Eric Dumazet
Quoted 8 lines · Shaohua Li

This isn't a previous bug. see

#define DECLARE_PER_CPU_PAGE_ALIGNED(type, name)			\
	DECLARE_PER_CPU_SECTION(type, name, "..page_aligned")		\
	__aligned(PAGE_SIZE)

the ..page_aligned is already page aligned. I add it is because it can make the .readmostly section guarantee to have no cache false sharing, because I add the . = ALIGN(PAGE_SIZE); before .readmostly section, and .page_aligned follows, but for sure this wates some memory.

Quoted 8 lines

A small note:

The __aligned(XXX) makes sure object _starts_ at XXX boundary, not that following one will also use same alignment. Of course, we can argue that we dont try to put in PAGE_ALIGNED section small objects, but still...

Explicit .ALIGN() uses in .lds are more readable IMHO...

There is. However, see again my previous note about SORT_BY_ALIGNMENT.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#18 lore raw

Re: [tip:x86/mm] percpu: Introduce a read-mostly percpu API

In reply to tip-bot for Shaohua Li
Quoted 16 lines · tip-bot for Shaohua Li
include/asm-generic/vmlinux.lds.h +2 -0
index 8a92a17..d7e7b21 100644
@@ -677,7 +677,9 @@
 				- LOAD_OFFSET) {			\
 		VMLINUX_SYMBOL(__per_cpu_start) = .;			\
 		*(.data..percpu..first)					\
+		. = ALIGN(PAGE_SIZE);					\
 		*(.data..percpu..page_aligned)				\
+		*(.data..percpu..readmostly)				\
 		*(.data..percpu)					\
 		*(.data..percpu..shared_aligned)			\
 		VMLINUX_SYMBOL(__per_cpu_end) = .;			\

include/asm-generic/vmlinux.lds.h line 712

Quoted 8 lines
include/asm-generic/vmlinux.lds.h +2 -0
@@ -703,6 +705,8 @@
 		VMLINUX_SYMBOL(__per_cpu_load) = .;			\
 		VMLINUX_SYMBOL(__per_cpu_start) = .;			\
 		*(.data..percpu..first)					\
+		. = ALIGN(PAGE_SIZE);					\
+		*(.data..percpu..readmostly)				\
 		*(.data..percpu..page_aligned)				\
 		*(.data..percpu)					\
 		*(.data..percpu..shared_aligned)			\

ARGH... looks like I didn't catch it everywhere...

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#19 lore raw

[tip:x86/mm] x86-32, percpu: Correct the ordering of the percpu readmostly section

In reply to Shaohua Li
Commit-ID:  2aeb66d3036dbafc297ac553a257a40283dadb3e
Gitweb:     http://git.kernel.org/tip/2aeb66d3036dbafc297ac553a257a40283dadb3e
Author:     H. Peter Anvin <[email protected]>
AuthorDate: Thu, 21 Oct 2010 00:15:00 -0700
Committer:  H. Peter Anvin <[email protected]>
CommitDate: Thu, 21 Oct 2010 00:15:00 -0700

x86-32, percpu: Correct the ordering of the percpu readmostly section

Checkin c957ef2c59e952803766ddc22e89981ab534606f had inconsistent ordering of .data..percpu..page_aligned and .data..percpu..readmostly; the still-broken version affected x86-32 at least.

The page aligned version really must be page aligned...

LKML-Reference: <[email protected]>
Cc: Shaohua Li <[email protected]>
Cc: Eric Dumazet <[email protected]>

---
 include/asm-generic/vmlinux.lds.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
include/asm-generic/vmlinux.lds.h +1 -1
index d7e7b21..1457b81 100644
@@ -706,8 +706,8 @@
 		VMLINUX_SYMBOL(__per_cpu_start) = .;			\
 		*(.data..percpu..first)					\
 		. = ALIGN(PAGE_SIZE);					\
-		*(.data..percpu..readmostly)				\
 		*(.data..percpu..page_aligned)				\
+		*(.data..percpu..readmostly)				\
 		*(.data..percpu)					\
 		*(.data..percpu..shared_aligned)			\
 		VMLINUX_SYMBOL(__per_cpu_end) = .;			\