percpu: Per cpu code simplification V3

PATCH In review [email protected] 4 participants 19 messages Jan 8, 2008

Subsystems (5)

Memory management x86 PowerPC Generic include/asm header files +1 more subsystems

lore

Share & tools

lore

[email protected]

author

2008-01-08 02:11 UTC #0 lore raw

cover percpu: Per cpu code simplification V3

This patchset simplifies the code that arches need to maintain to support per cpu functionality. Most of the code is moved into arch independent code. Only a minimal set of definitions is kept for each arch.

The patch also unifies the x86 arch so that there is only a single asm-x86/percpu.h

V1->V2:
- Add support for specifying attributes for per cpu declarations (preserves IA64 model(small) attribute).
- Drop first patch that removes the model(small) attribute for IA64
- Missing #endif in powerpc generic config / Wrong Kconfig
- Follow Randy's suggestions on how to do the Kconfig settings

V2->V3:
- fix x86_64 non-SMP case
- change SHIFT_PTR to SHIFT_PERCPU_PTR
- fix various percpu_modcopy()'s to reference correct per_cpu_offset()
- s390 has a special way to determine the pointer to a per cpu area

Signed-off-by Christoph Lameter <[email protected]>

Signed-off-by Mike Travis <[email protected]>

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[email protected]

author

2008-01-08 02:11 UTC #1 lore raw

PATCH 1/10 percpu: Use a kconfig variable to signal arch specific percpu setup

- Use def_bool as suggested by Randy.

The use of the __GENERIC_PERCPU is a bit problematic since arches may want to run their own percpu setup while using the generic percpu definitions. Replace it through a kconfig variable.

Cc Rusty Russell <[email protected]>

Cc Andi Kleen <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

Signed-off-by Mike Travis <[email protected]>

---
 arch/ia64/Kconfig            |    3 +++
 arch/powerpc/Kconfig         |    3 +++
 arch/sparc64/Kconfig         |    3 +++
 arch/x86/Kconfig             |    3 +++
 include/asm-generic/percpu.h |    1 -
 include/asm-s390/percpu.h    |    2 --
 include/asm-x86/percpu_32.h  |    2 --
 init/main.c                  |    4 ++--
 8 files changed, 14 insertions(+), 7 deletions(-)

arch/ia64/Kconfig +3 -0

@@ -80,6 +80,9 @@ config GENERIC_TIME_VSYSCALL
 	bool
 	default y
 
+config ARCH_SETS_UP_PER_CPU_AREA
+	def_bool y
+
 config DMI
 	bool
 	default y

arch/powerpc/Kconfig +3 -0

@@ -42,6 +42,9 @@ config GENERIC_HARDIRQS
 	bool
 	default y
 
+config ARCH_SETS_UP_PER_CPU_AREA
+	def_bool PPC64
+
 config IRQ_PER_CPU
 	bool
 	default y

arch/sparc64/Kconfig +3 -0

@@ -66,6 +66,9 @@ config AUDIT_ARCH
 	bool
 	default y
 
+config ARCH_SETS_UP_PER_CPU_AREA
+	def_bool y
+
 config ARCH_NO_VIRT_TO_BUS
 	def_bool y

arch/x86/Kconfig +3 -0

@@ -100,6 +100,9 @@ config GENERIC_TIME_VSYSCALL
 	bool
 	default X86_64
 
+config ARCH_SETS_UP_PER_CPU_AREA
+	def_bool X86_64
+
 config ARCH_SUPPORTS_OPROFILE
 	bool
 	default y

include/asm-generic/percpu.h +0 -1

@@ -3,7 +3,6 @@
 #include <linux/compiler.h>
 #include <linux/threads.h>
 
-#define __GENERIC_PER_CPU
 #ifdef CONFIG_SMP
 
 extern unsigned long __per_cpu_offset[NR_CPUS];

include/asm-s390/percpu.h +0 -2

@@ -4,8 +4,6 @@
 #include <linux/compiler.h>
 #include <asm/lowcore.h>
 
-#define __GENERIC_PER_CPU
-
 /*
  * s390 uses its own implementation for per cpu data, the offset of
  * the cpu local data area is cached in the cpu's lowcore memory.

include/asm-x86/percpu_32.h +0 -2

@@ -41,8 +41,6 @@
  *    PER_CPU(cpu_gdt_descr, %ebx)
  */
 #ifdef CONFIG_SMP
-/* Same as generic implementation except for optimized local access. */
-#define __GENERIC_PER_CPU
 
 /* This is used for other cpus to find our section. */
 extern unsigned long __per_cpu_offset[];

init/main.c +2 -2

@@ -363,7 +363,7 @@ static inline void smp_prepare_cpus(unsi
 
 #else
 
-#ifdef __GENERIC_PER_CPU
+#ifndef CONFIG_ARCH_SETS_UP_PER_CPU_AREA
 unsigned long __per_cpu_offset[NR_CPUS] __read_mostly;
 
 EXPORT_SYMBOL(__per_cpu_offset);

@@ -384,7 +384,7 @@ static void __init setup_per_cpu_areas(v
 		ptr += size;
 	}
 }
-#endif /* !__GENERIC_PER_CPU */
+#endif /* CONFIG_ARCH_SETS_UP_CPU_AREA */
 
 /* Called by boot processor to activate the rest. */
 static void __init smp_init(void)

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sam Ravnborg

2008-01-08 19:24 UTC #2 lore raw

Re: [PATCH 01/10] percpu: Use a kconfig variable to signal arch specific percpu setup

Quoted 40 lines · On Mon, Jan 07, 2008 at 06:11:43PM -0800, [email protected] wrote:

V1->V2:
- Use def_bool as suggested by Randy.
The use of the __GENERIC_PERCPU is a bit problematic since arches may want to run their own percpu setup while using the generic percpu definitions. Replace it through a kconfig variable.
Cc Rusty Russell <[email protected]>
Cc Andi Kleen <[email protected]>
Signed-off-by Christoph Lameter <[email protected]>
Signed-off-by Mike Travis <[email protected]>
---
 arch/ia64/Kconfig            |    3 +++
 arch/powerpc/Kconfig         |    3 +++
 arch/sparc64/Kconfig         |    3 +++
 arch/x86/Kconfig             |    3 +++
 include/asm-generic/percpu.h |    1 -
 include/asm-s390/percpu.h    |    2 --
 include/asm-x86/percpu_32.h  |    2 --
 init/main.c                  |    4 ++--
 8 files changed, 14 insertions(+), 7 deletions(-)
arch/ia64/Kconfig +3 -0
@@ -80,6 +80,9 @@ config GENERIC_TIME_VSYSCALL
 	bool
 	default y
 
+config ARCH_SETS_UP_PER_CPU_AREA
+	def_bool y
+
 config DMI
 	bool
 	default y

arch/powerpc/Kconfig config GENERIC_HARDIRQS · line 50

Quoted 9 lines

arch/powerpc/Kconfig +3 -0

@@ -42,6 +42,9 @@ config GENERIC_HARDIRQS
 	bool
 	default y
 
+config ARCH_SETS_UP_PER_CPU_AREA
+	def_bool PPC64
+
 config IRQ_PER_CPU
 	bool
 	default y

Please do not create one variable per arch to enable this functionality. Define one common variable and name it:
config HAVE_SETUP_PER_CPU_AREA

and then for the arch's that supports it you select this symbol. For X86 it would look like:

config X86
+ select HAVE_SETUP_PER_CPU_AREA

This is the recommended methiond today - albeit not widely used yet.

Sam

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[email protected]

author

2008-01-08 02:11 UTC #3 lore raw

PATCH 2/10 percpu: Move arch XX_PER_CPU_XX definitions into linux/percpu.h

In reply to [email protected]

- Special consideration for IA64: Add the ability to specify arch specific per cpu flags

V2->V3:
- remove .data.percpu attribute from DEFINE_PER_CPU for non-smp case.

The arch definitions are all the same. So move them into linux/percpu.h.

We cannot move DECLARE_PER_CPU since some include files just include asm/percpu.h to avoid include recursion problems.

Cc Rusty Russell <[email protected]>

Cc Andi Kleen <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

Signed-off-by Mike Travis <[email protected]>

---
 include/asm-generic/percpu.h |   18 ------------------
 include/asm-ia64/percpu.h    |   24 ++----------------------
 include/asm-powerpc/percpu.h |   17 -----------------
 include/asm-s390/percpu.h    |   18 ------------------
 include/asm-sparc64/percpu.h |   16 ----------------
 include/asm-x86/percpu_32.h  |   12 ------------
 include/asm-x86/percpu_64.h  |   17 -----------------
 include/linux/percpu.h       |   24 ++++++++++++++++++++++++
 8 files changed, 26 insertions(+), 120 deletions(-)

include/asm-generic/percpu.h +0 ~~-18~~

@@ -9,15 +9,6 @@ extern unsigned long __per_cpu_offset[NR
 
 #define per_cpu_offset(x) (__per_cpu_offset[x])
 
-/* Separate out the type, so (int[3], foo) works. */
-#define DEFINE_PER_CPU(type, name) \
-    __attribute__((__section__(".data.percpu"))) __typeof__(type) per_cpu__##name
-
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)		\
-    __attribute__((__section__(".data.percpu.shared_aligned"))) \
-    __typeof__(type) per_cpu__##name				\
-    ____cacheline_aligned_in_smp
-
 /* var is in discarded region: offset to particular copy we want */
 #define per_cpu(var, cpu) (*({				\
 	extern int simple_identifier_##var(void);	\

@@ -35,12 +26,6 @@ do {								\
 } while (0)
 #else /* ! SMP */
 
-#define DEFINE_PER_CPU(type, name) \
-    __typeof__(type) per_cpu__##name
-
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)	\
-    DEFINE_PER_CPU(type, name)
-
 #define per_cpu(var, cpu)			(*((void)(cpu), &per_cpu__##var))
 #define __get_cpu_var(var)			per_cpu__##var
 #define __raw_get_cpu_var(var)			per_cpu__##var

@@ -49,7 +34,4 @@ do {								\
 
 #define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
 
-#define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
-#define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
-
 #endif /* _ASM_GENERIC_PERCPU_H_ */

include/asm-ia64/percpu.h +2 ~~-22~~

@@ -16,28 +16,11 @@
 #include <linux/threads.h>
 
 #ifdef HAVE_MODEL_SMALL_ATTRIBUTE
-# define __SMALL_ADDR_AREA	__attribute__((__model__ (__small__)))
-#else
-# define __SMALL_ADDR_AREA
+# define PER_CPU_ATTRIBUTES	__attribute__((__model__ (__small__)))
 #endif
 
 #define DECLARE_PER_CPU(type, name)				\
-	extern __SMALL_ADDR_AREA __typeof__(type) per_cpu__##name
-
-/* Separate out the type, so (int[3], foo) works. */
-#define DEFINE_PER_CPU(type, name)				\
-	__attribute__((__section__(".data.percpu")))		\
-	__SMALL_ADDR_AREA __typeof__(type) per_cpu__##name
-
-#ifdef CONFIG_SMP
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)			\
-	__attribute__((__section__(".data.percpu.shared_aligned")))	\
-	__SMALL_ADDR_AREA __typeof__(type) per_cpu__##name		\
-	____cacheline_aligned_in_smp
-#else
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)	\
-	DEFINE_PER_CPU(type, name)
-#endif
+	extern PER_CPU_ATTRIBUTES __typeof__(type) per_cpu__##name
 
 /*
  * Pretty much a literal copy of asm-generic/percpu.h, except that percpu_modcopy() is an

@@ -68,9 +51,6 @@ extern void *per_cpu_init(void);
 
 #endif	/* SMP */
 
-#define EXPORT_PER_CPU_SYMBOL(var)		EXPORT_SYMBOL(per_cpu__##var)
-#define EXPORT_PER_CPU_SYMBOL_GPL(var)		EXPORT_SYMBOL_GPL(per_cpu__##var)
-
 /*
  * Be extremely careful when taking the address of this variable!  Due to virtual
  * remapping, it is different from the canonical address returned by __get_cpu_var(var)!

include/asm-powerpc/percpu.h +0 ~~-17~~

@@ -16,15 +16,6 @@
 #define __my_cpu_offset() get_paca()->data_offset
 #define per_cpu_offset(x) (__per_cpu_offset(x))
 
-/* Separate out the type, so (int[3], foo) works. */
-#define DEFINE_PER_CPU(type, name) \
-    __attribute__((__section__(".data.percpu"))) __typeof__(type) per_cpu__##name
-
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)		\
-    __attribute__((__section__(".data.percpu.shared_aligned"))) \
-    __typeof__(type) per_cpu__##name				\
-    ____cacheline_aligned_in_smp
-
 /* var is in discarded region: offset to particular copy we want */
 #define per_cpu(var, cpu) (*RELOC_HIDE(&per_cpu__##var, __per_cpu_offset(cpu)))
 #define __get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __my_cpu_offset()))

@@ -43,11 +34,6 @@ extern void setup_per_cpu_areas(void);
 
 #else /* ! SMP */
 
-#define DEFINE_PER_CPU(type, name) \
-    __typeof__(type) per_cpu__##name
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)	\
-    DEFINE_PER_CPU(type, name)
-
 #define per_cpu(var, cpu)			(*((void)(cpu), &per_cpu__##var))
 #define __get_cpu_var(var)			per_cpu__##var
 #define __raw_get_cpu_var(var)			per_cpu__##var

@@ -56,9 +42,6 @@ extern void setup_per_cpu_areas(void);
 
 #define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
 
-#define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
-#define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
-
 #else
 #include <asm-generic/percpu.h>
 #endif

include/asm-s390/percpu.h +0 ~~-18~~

@@ -34,16 +34,6 @@
 
 extern unsigned long __per_cpu_offset[NR_CPUS];
 
-/* Separate out the type, so (int[3], foo) works. */
-#define DEFINE_PER_CPU(type, name) \
-    __attribute__((__section__(".data.percpu"))) \
-    __typeof__(type) per_cpu__##name
-
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)		\
-    __attribute__((__section__(".data.percpu.shared_aligned"))) \
-    __typeof__(type) per_cpu__##name				\
-    ____cacheline_aligned_in_smp
-
 #define __get_cpu_var(var) __reloc_hide(var,S390_lowcore.percpu_offset)
 #define __raw_get_cpu_var(var) __reloc_hide(var,S390_lowcore.percpu_offset)
 #define per_cpu(var,cpu) __reloc_hide(var,__per_cpu_offset[cpu])

@@ -60,11 +50,6 @@ do {								\
 
 #else /* ! SMP */
 
-#define DEFINE_PER_CPU(type, name) \
-    __typeof__(type) per_cpu__##name
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)	\
-    DEFINE_PER_CPU(type, name)
-
 #define __get_cpu_var(var) __reloc_hide(var,0)
 #define __raw_get_cpu_var(var) __reloc_hide(var,0)
 #define per_cpu(var,cpu) __reloc_hide(var,0)

@@ -73,7 +58,4 @@ do {								\
 
 #define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
 
-#define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
-#define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
-
 #endif /* __ARCH_S390_PERCPU__ */

include/asm-sparc64/percpu.h +0 ~~-16~~

@@ -16,15 +16,6 @@ extern unsigned long __per_cpu_shift;
 	(__per_cpu_base + ((unsigned long)(__cpu) << __per_cpu_shift))
 #define per_cpu_offset(x) (__per_cpu_offset(x))
 
-/* Separate out the type, so (int[3], foo) works. */
-#define DEFINE_PER_CPU(type, name) \
-    __attribute__((__section__(".data.percpu"))) __typeof__(type) per_cpu__##name
-
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)		\
-    __attribute__((__section__(".data.percpu.shared_aligned"))) \
-    __typeof__(type) per_cpu__##name				\
-    ____cacheline_aligned_in_smp
-
 /* var is in discarded region: offset to particular copy we want */
 #define per_cpu(var, cpu) (*RELOC_HIDE(&per_cpu__##var, __per_cpu_offset(cpu)))
 #define __get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __local_per_cpu_offset))

@@ -41,10 +32,6 @@ do {								\
 #else /* ! SMP */
 
 #define real_setup_per_cpu_areas()		do { } while (0)
-#define DEFINE_PER_CPU(type, name) \
-    __typeof__(type) per_cpu__##name
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)	\
-    DEFINE_PER_CPU(type, name)
 
 #define per_cpu(var, cpu)			(*((void)cpu, &per_cpu__##var))
 #define __get_cpu_var(var)			per_cpu__##var

@@ -54,7 +41,4 @@ do {								\
 
 #define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
 
-#define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
-#define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
-
 #endif /* __ARCH_SPARC64_PERCPU__ */

include/asm-x86/percpu_32.h +0 ~~-12~~

@@ -47,16 +47,7 @@ extern unsigned long __per_cpu_offset[];
 
 #define per_cpu_offset(x) (__per_cpu_offset[x])
 
-/* Separate out the type, so (int[3], foo) works. */
 #define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
-#define DEFINE_PER_CPU(type, name) \
-    __attribute__((__section__(".data.percpu"))) __typeof__(type) per_cpu__##name
-
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)		\
-    __attribute__((__section__(".data.percpu.shared_aligned"))) \
-    __typeof__(type) per_cpu__##name				\
-    ____cacheline_aligned_in_smp
-
 /* We can use this directly for local CPU (faster). */
 DECLARE_PER_CPU(unsigned long, this_cpu_off);

@@ -81,9 +72,6 @@ do {								\
 		       (src), (size));				\
 } while (0)
 
-#define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
-#define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
-
 /* fs segment starts at (positive) offset == __per_cpu_offset[cpu] */
 #define __percpu_seg "%%fs:"
 #else  /* !SMP */

include/asm-x86/percpu_64.h +0 ~~-17~~

@@ -16,15 +16,6 @@
 
 #define per_cpu_offset(x) (__per_cpu_offset(x))
 
-/* Separate out the type, so (int[3], foo) works. */
-#define DEFINE_PER_CPU(type, name) \
-    __attribute__((__section__(".data.percpu"))) __typeof__(type) per_cpu__##name
-
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)		\
-    __attribute__((__section__(".data.percpu.shared_aligned"))) \
-    __typeof__(type) per_cpu__##name				\
-    ____cacheline_internodealigned_in_smp
-
 /* var is in discarded region: offset to particular copy we want */
 #define per_cpu(var, cpu) (*({				\
 	extern int simple_identifier_##var(void);	\

@@ -49,11 +40,6 @@ extern void setup_per_cpu_areas(void);
 
 #else /* ! SMP */
 
-#define DEFINE_PER_CPU(type, name) \
-    __typeof__(type) per_cpu__##name
-#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)	\
-    DEFINE_PER_CPU(type, name)
-
 #define per_cpu(var, cpu)			(*((void)(cpu), &per_cpu__##var))
 #define __get_cpu_var(var)			per_cpu__##var
 #define __raw_get_cpu_var(var)			per_cpu__##var

@@ -62,7 +48,4 @@ extern void setup_per_cpu_areas(void);
 
 #define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
 
-#define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
-#define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
-
 #endif /* _ASM_X8664_PERCPU_H_ */

include/linux/percpu.h +24 -0

@@ -9,6 +9,30 @@
 
 #include <asm/percpu.h>
 
+#ifndef PER_CPU_ATTRIBUTES
+#define PER_CPU_ATTRIBUTES
+#endif
+
+#ifdef CONFIG_SMP
+#define DEFINE_PER_CPU(type, name)					\
+	__attribute__((__section__(".data.percpu")))			\
+	PER_CPU_ATTRIBUTES __typeof__(type) per_cpu__##name
+
+#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)			\
+	__attribute__((__section__(".data.percpu.shared_aligned")))	\
+	PER_CPU_ATTRIBUTES __typeof__(type) per_cpu__##name		\
+	____cacheline_aligned_in_smp
+#else
+#define DEFINE_PER_CPU(type, name)					\
+	PER_CPU_ATTRIBUTES __typeof__(type) per_cpu__##name
+
+#define DEFINE_PER_CPU_SHARED_ALIGNED(type, name)		      \
+	DEFINE_PER_CPU(type, name)
+#endif
+
+#define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
+#define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
+
 /* Enough to cover all DEFINE_PER_CPUs in kernel, including modules. */
 #ifndef PERCPU_ENOUGH_ROOM
 #ifdef CONFIG_MODULES

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[email protected]

author

2008-01-08 02:11 UTC #4 lore raw

PATCH 3/10 percpu: Make the asm-generic/percpu.h more "generic"

In reply to [email protected]

- add support for PER_CPU_ATTRIBUTES

V2->V3:
- fix generic smp percpu_modcopy to use per_cpu_offset() macro.

Add the ability to use generic/percpu even if the arch needs to override several aspects of its operations. This will enable the use of generic percpu.h for all arches.

An arch may define:

__per_cpu_offset Do not use the generic pointer array. Arch must

			define per_cpu_offset(cpu) (used by x86_64, s390).

__my_cpu_offset Can be defined to provide an optimized way to determine

			the offset for variables of the currently executing
			processor. Used by ia64, x86_64, x86_32, sparc64, s/390.

SHIFT_PTR(ptr, offset) If an arch defines it then special handling

			of pointer arithmentic may be implemented. Used
			by s/390.

(Some of these special percpu arch implementations may be later consolidated so that there are less cases to deal with.)

Cc Rusty Russell <[email protected]>

Cc Andi Kleen <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

Signed-off-by Mike Travis <[email protected]>

---
 arch/ia64/kernel/module.c    |    2 -
 include/asm-generic/percpu.h |   74 ++++++++++++++++++++++++++++++++++++-------
 2 files changed, 64 insertions(+), 12 deletions(-)

arch/ia64/kernel/module.c +1 -1

@@ -947,7 +947,7 @@ percpu_modcopy (void *pcpudst, const voi
 {
 	unsigned int i;
 	for_each_possible_cpu(i) {
-		memcpy(pcpudst + __per_cpu_offset[i], src, size);
+		memcpy(pcpudst + per_cpu_offset(i), src, size);
 	}
 }
 #endif /* CONFIG_SMP */

include/asm-generic/percpu.h +63 ~~-11~~

98 lines not on this page · raw message

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[email protected]

author

2008-01-08 02:11 UTC #5 lore raw

PATCH 4/10 x86_32: Use generic percpu.h

In reply to [email protected]

x86_32 only provides a special way to obtain the local per cpu area offset via x86_read_percpu. Otherwise it can fully use the generic handling.

Cc [email protected]

Signed-off-by Christoph Lameter <[email protected]>

Signed-off-by Mike Travis <[email protected]>

---
 include/asm-x86/percpu_32.h |   30 +++++++++---------------------
 1 file changed, 9 insertions(+), 21 deletions(-)

include/asm-x86/percpu_32.h +9 ~~-21~~

@@ -42,26 +42,7 @@
  */
 #ifdef CONFIG_SMP
 
-/* This is used for other cpus to find our section. */
-extern unsigned long __per_cpu_offset[];
-
-#define per_cpu_offset(x) (__per_cpu_offset[x])
-
-#define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
-/* We can use this directly for local CPU (faster). */
-DECLARE_PER_CPU(unsigned long, this_cpu_off);
-
-/* var is in discarded region: offset to particular copy we want */
-#define per_cpu(var, cpu) (*({				\
-	extern int simple_indentifier_##var(void);	\
-	RELOC_HIDE(&per_cpu__##var, __per_cpu_offset[cpu]); }))
-
-#define __raw_get_cpu_var(var) (*({					\
-	extern int simple_indentifier_##var(void);			\
-	RELOC_HIDE(&per_cpu__##var, x86_read_percpu(this_cpu_off));	\
-}))
-
-#define __get_cpu_var(var) __raw_get_cpu_var(var)
+#define __my_cpu_offset x86_read_percpu(this_cpu_off)
 
 /* A macro to avoid #include hell... */
 #define percpu_modcopy(pcpudst, src, size)			\

@@ -74,11 +55,18 @@ do {								\
 
 /* fs segment starts at (positive) offset == __per_cpu_offset[cpu] */
 #define __percpu_seg "%%fs:"
+
 #else  /* !SMP */
-#include <asm-generic/percpu.h>
+
 #define __percpu_seg ""
+
 #endif	/* SMP */
 
+#include <asm-generic/percpu.h>
+
+/* We can use this directly for local CPU (faster). */
+DECLARE_PER_CPU(unsigned long, this_cpu_off);
+
 /* For arch-specific code, we can use direct single-insn ops (they
  * don't give an lvalue though). */
 extern void __bad_percpu_size(void);

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[email protected]

author

2008-01-08 02:11 UTC #6 lore raw

PATCH 5/10 x86_64: Use generic percpu

In reply to [email protected]

x86_64 provides an optimized way to determine the local per cpu area offset through the pda and determines the base by accessing a remote pda.

Cc Rusty Russell <[email protected]>

Cc Andi Kleen <[email protected]>

Cc [email protected]

Signed-off-by Christoph Lameter <[email protected]>

Signed-off-by Mike Travis <[email protected]>

---
 include/asm-x86/percpu_64.h |   23 ++---------------------
 1 file changed, 2 insertions(+), 21 deletions(-)

include/asm-x86/percpu_64.h +2 ~~-21~~

@@ -12,21 +12,10 @@
 #include <asm/pda.h>
 
 #define __per_cpu_offset(cpu) (cpu_pda(cpu)->data_offset)
-#define __my_cpu_offset() read_pda(data_offset)
+#define __my_cpu_offset read_pda(data_offset)
 
 #define per_cpu_offset(x) (__per_cpu_offset(x))
 
-/* var is in discarded region: offset to particular copy we want */
-#define per_cpu(var, cpu) (*({				\
-	extern int simple_identifier_##var(void);	\
-	RELOC_HIDE(&per_cpu__##var, __per_cpu_offset(cpu)); }))
-#define __get_cpu_var(var) (*({				\
-	extern int simple_identifier_##var(void);	\
-	RELOC_HIDE(&per_cpu__##var, __my_cpu_offset()); }))
-#define __raw_get_cpu_var(var) (*({			\
-	extern int simple_identifier_##var(void);	\
-	RELOC_HIDE(&per_cpu__##var, __my_cpu_offset()); }))
-
 /* A macro to avoid #include hell... */
 #define percpu_modcopy(pcpudst, src, size)			\
 do {								\

@@ -36,16 +25,8 @@ do {								\
 		       (src), (size));				\
 } while (0)
 
-extern void setup_per_cpu_areas(void);
-
-#else /* ! SMP */
-
-#define per_cpu(var, cpu)			(*((void)(cpu), &per_cpu__##var))
-#define __get_cpu_var(var)			per_cpu__##var
-#define __raw_get_cpu_var(var)			per_cpu__##var
-
 #endif	/* SMP */
 
-#define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
+#include <asm-generic/percpu.h>
 
 #endif /* _ASM_X8664_PERCPU_H_ */

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[email protected]

author

2008-01-08 02:11 UTC #7 lore raw

PATCH 6/10 s390: Use generic percpu

In reply to [email protected]

Quoted 8 lines · Martin Schwidefsky

Quoted 6 lines · Christoph Lameter
s390 has a special way to determine the pointer to a per cpu area plus there is a way to access the base of the per cpu area of the currently executing processor.
Note: I had to do a minor change to ASM code. Please check that this was done right.

Quoted 6 lines

Hi Christoph,
after fixing the trainwreck with Gregs kset changes I've got rc3-mm2 compiled with your percpu patches. The new s390 percpu code works fine:
Acked-by Martin Schwidefsky <[email protected]>

Cc [email protected]

Signed-off-by Christoph Lameter <[email protected]>

Signed-off-by Mike Travis <[email protected]>

---
 include/asm-s390/percpu.h |   42 +++++++++---------------------------------
 1 file changed, 9 insertions(+), 33 deletions(-)

include/asm-s390/percpu.h +9 ~~-33~~

@@ -13,49 +13,25 @@
  */
 #if defined(__s390x__) && defined(MODULE)
 
-#define __reloc_hide(var,offset) (*({			\
+#define SHIFT_PERCPU_PTR(ptr,offset) (({			\
 	extern int simple_identifier_##var(void);	\
 	unsigned long *__ptr;				\
-	asm ( "larl %0,per_cpu__"#var"@GOTENT"		\
-	    : "=a" (__ptr) : "X" (per_cpu__##var) );	\
-	(typeof(&per_cpu__##var))((*__ptr) + (offset));	}))
+	asm ( "larl %0, %1@GOTENT"		\
+	    : "=a" (__ptr) : "X" (ptr) );		\
+	(typeof(ptr))((*__ptr) + (offset));	}))
 
 #else
 
-#define __reloc_hide(var, offset) (*({				\
+#define SHIFT_PERCPU_PTR(ptr, offset) (({				\
 	extern int simple_identifier_##var(void);		\
 	unsigned long __ptr;					\
-	asm ( "" : "=a" (__ptr) : "0" (&per_cpu__##var) );	\
-	(typeof(&per_cpu__##var)) (__ptr + (offset)); }))
+	asm ( "" : "=a" (__ptr) : "0" (ptr) );			\
+	(typeof(ptr)) (__ptr + (offset)); }))
 
 #endif
 
-#ifdef CONFIG_SMP
+#define __my_cpu_offset S390_lowcore.percpu_offset
 
-extern unsigned long __per_cpu_offset[NR_CPUS];
-
-#define __get_cpu_var(var) __reloc_hide(var,S390_lowcore.percpu_offset)
-#define __raw_get_cpu_var(var) __reloc_hide(var,S390_lowcore.percpu_offset)
-#define per_cpu(var,cpu) __reloc_hide(var,__per_cpu_offset[cpu])
-#define per_cpu_offset(x) (__per_cpu_offset[x])
-
-/* A macro to avoid #include hell... */
-#define percpu_modcopy(pcpudst, src, size)			\
-do {								\
-	unsigned int __i;					\
-	for_each_possible_cpu(__i)				\
-		memcpy((pcpudst)+__per_cpu_offset[__i],		\
-		       (src), (size));				\
-} while (0)
-
-#else /* ! SMP */
-
-#define __get_cpu_var(var) __reloc_hide(var,0)
-#define __raw_get_cpu_var(var) __reloc_hide(var,0)
-#define per_cpu(var,cpu) __reloc_hide(var,0)
-
-#endif /* SMP */
-
-#define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
+#include <asm-generic/percpu.h>
 
 #endif /* __ARCH_S390_PERCPU__ */

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[email protected]

author

2008-01-08 02:11 UTC #8 lore raw

PATCH 7/10 Powerpc: Use generic per cpu

In reply to [email protected]

- add missing #endif

V2->V3:
- use generic percpy_modcopy()

Powerpc has a way to determine the address of the per cpu area of the currently executing processor via the paca and the array of per cpu offsets is avoided by looking up the per cpu area from the remote paca's (copying x86_64).

Cc Paul Mackerras <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

Signed-off-by Mike Travis <[email protected]>

---
 include/asm-powerpc/percpu.h |   29 ++---------------------------
 1 file changed, 2 insertions(+), 27 deletions(-)

include/asm-powerpc/percpu.h +2 ~~-27~~

@@ -16,34 +16,9 @@
 #define __my_cpu_offset() get_paca()->data_offset
 #define per_cpu_offset(x) (__per_cpu_offset(x))
 
-/* var is in discarded region: offset to particular copy we want */
-#define per_cpu(var, cpu) (*RELOC_HIDE(&per_cpu__##var, __per_cpu_offset(cpu)))
-#define __get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __my_cpu_offset()))
-#define __raw_get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, local_paca->data_offset))
+#endif /* CONFIG_SMP */
+#endif /* __powerpc64__ */
 
-/* A macro to avoid #include hell... */
-#define percpu_modcopy(pcpudst, src, size)			\
-do {								\
-	unsigned int __i;					\
-	for_each_possible_cpu(__i)				\
-		memcpy((pcpudst)+__per_cpu_offset(__i),		\
-		       (src), (size));				\
-} while (0)
-
-extern void setup_per_cpu_areas(void);
-
-#else /* ! SMP */
-
-#define per_cpu(var, cpu)			(*((void)(cpu), &per_cpu__##var))
-#define __get_cpu_var(var)			per_cpu__##var
-#define __raw_get_cpu_var(var)			per_cpu__##var
-
-#endif	/* SMP */
-
-#define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
-
-#else
 #include <asm-generic/percpu.h>
-#endif
 
 #endif /* _ASM_POWERPC_PERCPU_H_ */

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[email protected]

author

2008-01-08 02:11 UTC #9 lore raw

PATCH 8/10 Sparc64: Use generic percpu

In reply to [email protected]

- use generic percpy_modcopy()

Sparc64 has a way of providing the base address for the per cpu area of the currently executing processor in a global register.

Sparc64 also provides a way to calculate the address of a per cpu area from a base address instead of performing an array lookup.

Cc David Miller <[email protected]>

Signed-off-by Christoph Lameter <[email protected]>

Signed-off-by Mike Travis <[email protected]>

---
 arch/sparc64/mm/init.c       |    5 +++++
 include/asm-sparc64/percpu.h |   22 +++-------------------
 2 files changed, 8 insertions(+), 19 deletions(-)

arch/sparc64/mm/init.c +5 -0

@@ -1328,6 +1328,11 @@ pgd_t swapper_pg_dir[2048];
 static void sun4u_pgprot_init(void);
 static void sun4v_pgprot_init(void);
 
+/* Dummy function */
+void __init setup_per_cpu_areas(void)
+{
+}
+
 void __init paging_init(void)
 {
 	unsigned long end_pfn, pages_avail, shift, phys_base;

include/asm-sparc64/percpu.h +3 ~~-19~~

@@ -7,7 +7,6 @@ register unsigned long __local_per_cpu_o
 
 #ifdef CONFIG_SMP
 
-#define setup_per_cpu_areas()			do { } while (0)
 extern void real_setup_per_cpu_areas(void);
 
 extern unsigned long __per_cpu_base;

@@ -16,29 +15,14 @@ extern unsigned long __per_cpu_shift;
 	(__per_cpu_base + ((unsigned long)(__cpu) << __per_cpu_shift))
 #define per_cpu_offset(x) (__per_cpu_offset(x))
 
-/* var is in discarded region: offset to particular copy we want */
-#define per_cpu(var, cpu) (*RELOC_HIDE(&per_cpu__##var, __per_cpu_offset(cpu)))
-#define __get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __local_per_cpu_offset))
-#define __raw_get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __local_per_cpu_offset))
-
-/* A macro to avoid #include hell... */
-#define percpu_modcopy(pcpudst, src, size)			\
-do {								\
-	unsigned int __i;					\
-	for_each_possible_cpu(__i)				\
-		memcpy((pcpudst)+__per_cpu_offset(__i),		\
-		       (src), (size));				\
-} while (0)
+#define __my_cpu_offset __local_per_cpu_offset
+
 #else /* ! SMP */
 
 #define real_setup_per_cpu_areas()		do { } while (0)
 
-#define per_cpu(var, cpu)			(*((void)cpu, &per_cpu__##var))
-#define __get_cpu_var(var)			per_cpu__##var
-#define __raw_get_cpu_var(var)			per_cpu__##var
-
 #endif	/* SMP */
 
-#define DECLARE_PER_CPU(type, name) extern __typeof__(type) per_cpu__##name
+#include <asm-generic/percpu.h>
 
 #endif /* __ARCH_SPARC64_PERCPU__ */

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[email protected]

author

2008-01-08 02:11 UTC #10 lore raw

PATCH 9/10 ia64: Use generic percpu

In reply to [email protected]

V1->V2:
- Merge fixes
- Remove transitional check for PER_CPU_ATTRIBUTES from linux/percpu.h

V2-.V3:
- use generic percpy_modcopy()

ia64 has a special processor specific mapping that can be used to locate the offset for the current per cpu area.

Cc [email protected]

Signed-off-by Christoph Lameter <[email protected]>

Signed-off-by Mike Travis <[email protected]>

---
 arch/ia64/kernel/module.c |   11 -----------
 include/asm-ia64/percpu.h |   29 +++++++----------------------
 include/linux/percpu.h    |    4 ----
 3 files changed, 7 insertions(+), 37 deletions(-)

arch/ia64/kernel/module.c +0 ~~-11~~

@@ -940,14 +940,3 @@ module_arch_cleanup (struct module *mod)
 	if (mod->arch.core_unw_table)
 		unw_remove_unwind_table(mod->arch.core_unw_table);
 }
-
-#ifdef CONFIG_SMP
-void
-percpu_modcopy (void *pcpudst, const void *src, unsigned long size)
-{
-	unsigned int i;
-	for_each_possible_cpu(i) {
-		memcpy(pcpudst + per_cpu_offset(i), src, size);
-	}
-}
-#endif /* CONFIG_SMP */

include/asm-ia64/percpu.h +7 ~~-22~~

@@ -19,34 +19,14 @@
 # define PER_CPU_ATTRIBUTES	__attribute__((__model__ (__small__)))
 #endif
 
-#define DECLARE_PER_CPU(type, name)				\
-	extern PER_CPU_ATTRIBUTES __typeof__(type) per_cpu__##name
-
-/*
- * Pretty much a literal copy of asm-generic/percpu.h, except that percpu_modcopy() is an
- * external routine, to avoid include-hell.
- */
 #ifdef CONFIG_SMP
 
-extern unsigned long __per_cpu_offset[NR_CPUS];
-#define per_cpu_offset(x) (__per_cpu_offset[x])
-
-/* Equal to __per_cpu_offset[smp_processor_id()], but faster to access: */
-DECLARE_PER_CPU(unsigned long, local_per_cpu_offset);
+#define __my_cpu_offset	__ia64_per_cpu_var(local_per_cpu_offset)
 
-#define per_cpu(var, cpu)  (*RELOC_HIDE(&per_cpu__##var, __per_cpu_offset[cpu]))
-#define __get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __ia64_per_cpu_var(local_per_cpu_offset)))
-#define __raw_get_cpu_var(var) (*RELOC_HIDE(&per_cpu__##var, __ia64_per_cpu_var(local_per_cpu_offset)))
-
-extern void percpu_modcopy(void *pcpudst, const void *src, unsigned long size);
-extern void setup_per_cpu_areas (void);
 extern void *per_cpu_init(void);
 
 #else /* ! SMP */
 
-#define per_cpu(var, cpu)			(*((void)(cpu), &per_cpu__##var))
-#define __get_cpu_var(var)			per_cpu__##var
-#define __raw_get_cpu_var(var)			per_cpu__##var
 #define per_cpu_init()				(__phys_per_cpu_start)
 
 #endif	/* SMP */

@@ -57,7 +37,12 @@ extern void *per_cpu_init(void);
  * On the positive side, using __ia64_per_cpu_var() instead of __get_cpu_var() is slightly
  * more efficient.
  */
-#define __ia64_per_cpu_var(var)	(per_cpu__##var)
+#define __ia64_per_cpu_var(var)	per_cpu__##var
+
+#include <asm-generic/percpu.h>
+
+/* Equal to __per_cpu_offset[smp_processor_id()], but faster to access: */
+DECLARE_PER_CPU(unsigned long, local_per_cpu_offset);
 
 #endif /* !__ASSEMBLY__ */

include/linux/percpu.h +0 -4

@@ -9,10 +9,6 @@
 
 #include <asm/percpu.h>
 
-#ifndef PER_CPU_ATTRIBUTES
-#define PER_CPU_ATTRIBUTES
-#endif
-
 #ifdef CONFIG_SMP
 #define DEFINE_PER_CPU(type, name)					\
 	__attribute__((__section__(".data.percpu")))			\

[email protected]

author

2008-01-08 02:11 UTC #11 lore raw

PATCH 10/10 x86: Unify percpu.h

In reply to [email protected]

Form a single percpu.h from percpu_32.h and percpu_64.h. Both are now pretty small so this is simply adding them together.

Cc Rusty Russell <[email protected]>

Cc [email protected]

Signed-off-by Christoph Lameter <[email protected]>

Signed-off-by Mike Travis <[email protected]>

---
 include/asm-x86/percpu.h    |  145 ++++++++++++++++++++++++++++++++++++++++++--
 include/asm-x86/percpu_32.h |  128 --------------------------------------
 include/asm-x86/percpu_64.h |   32 ---------
 3 files changed, 141 insertions(+), 164 deletions(-)

include/asm-x86/percpu.h +141 -4

@@ -1,5 +1,142 @@
-#ifdef CONFIG_X86_32
-# include "percpu_32.h"
-#else
-# include "percpu_64.h"
+#ifndef _ASM_X86_PERCPU_H_
+#define _ASM_X86_PERCPU_H_
+
+#ifdef CONFIG_X86_64
+#include <linux/compiler.h>
+
+/* Same as asm-generic/percpu.h, except that we store the per cpu offset
+   in the PDA. Longer term the PDA and every per cpu variable
+   should be just put into a single section and referenced directly
+   from %gs */
+
+#ifdef CONFIG_SMP
+#include <asm/pda.h>
+
+#define __per_cpu_offset(cpu) (cpu_pda(cpu)->data_offset)
+#define __my_cpu_offset read_pda(data_offset)
+
+#define per_cpu_offset(x) (__per_cpu_offset(x))
+
 #endif
+#include <asm-generic/percpu.h>
+
+DECLARE_PER_CPU(struct x8664_pda, pda);
+
+#else /* CONFIG_X86_64 */
+
+#ifdef __ASSEMBLY__
+
+/*
+ * PER_CPU finds an address of a per-cpu variable.
+ *
+ * Args:
+ *    var - variable name
+ *    reg - 32bit register
+ *
+ * The resulting address is stored in the "reg" argument.
+ *
+ * Example:
+ *    PER_CPU(cpu_gdt_descr, %ebx)
+ */
+#ifdef CONFIG_SMP
+#define PER_CPU(var, reg)				\
+	movl %fs:per_cpu__##this_cpu_off, reg;		\
+	lea per_cpu__##var(reg), reg
+#define PER_CPU_VAR(var)	%fs:per_cpu__##var
+#else /* ! SMP */
+#define PER_CPU(var, reg)			\
+	movl $per_cpu__##var, reg
+#define PER_CPU_VAR(var)	per_cpu__##var
+#endif	/* SMP */
+
+#else /* ...!ASSEMBLY */
+
+/*
+ * PER_CPU finds an address of a per-cpu variable.
+ *
+ * Args:
+ *    var - variable name
+ *    cpu - 32bit register containing the current CPU number
+ *
+ * The resulting address is stored in the "cpu" argument.
+ *
+ * Example:
+ *    PER_CPU(cpu_gdt_descr, %ebx)
+ */
+#ifdef CONFIG_SMP
+
+#define __my_cpu_offset x86_read_percpu(this_cpu_off)
+
+/* fs segment starts at (positive) offset == __per_cpu_offset[cpu] */
+#define __percpu_seg "%%fs:"
+
+#else  /* !SMP */
+
+#define __percpu_seg ""
+
+#endif	/* SMP */
+
+#include <asm-generic/percpu.h>
+
+/* We can use this directly for local CPU (faster). */
+DECLARE_PER_CPU(unsigned long, this_cpu_off);
+
+/* For arch-specific code, we can use direct single-insn ops (they
+ * don't give an lvalue though). */
+extern void __bad_percpu_size(void);
+
+#define percpu_to_op(op,var,val)				\
+	do {							\
+		typedef typeof(var) T__;			\
+		if (0) { T__ tmp__; tmp__ = (val); }		\
+		switch (sizeof(var)) {				\
+		case 1:						\
+			asm(op "b %1,"__percpu_seg"%0"		\
+			    : "+m" (var)			\
+			    :"ri" ((T__)val));			\
+			break;					\
+		case 2:						\
+			asm(op "w %1,"__percpu_seg"%0"		\
+			    : "+m" (var)			\
+			    :"ri" ((T__)val));			\
+			break;					\
+		case 4:						\
+			asm(op "l %1,"__percpu_seg"%0"		\
+			    : "+m" (var)			\
+			    :"ri" ((T__)val));			\
+			break;					\
+		default: __bad_percpu_size();			\
+		}						\
+	} while (0)
+
+#define percpu_from_op(op,var)					\
+	({							\
+		typeof(var) ret__;				\
+		switch (sizeof(var)) {				\
+		case 1:						\
+			asm(op "b "__percpu_seg"%1,%0"		\
+			    : "=r" (ret__)			\
+			    : "m" (var));			\
+			break;					\
+		case 2:						\
+			asm(op "w "__percpu_seg"%1,%0"		\
+			    : "=r" (ret__)			\
+			    : "m" (var));			\
+			break;					\
+		case 4:						\
+			asm(op "l "__percpu_seg"%1,%0"		\
+			    : "=r" (ret__)			\
+			    : "m" (var));			\
+			break;					\
+		default: __bad_percpu_size();			\
+		}						\
+		ret__; })
+
+#define x86_read_percpu(var) percpu_from_op("mov", per_cpu__##var)
+#define x86_write_percpu(var,val) percpu_to_op("mov", per_cpu__##var, val)
+#define x86_add_percpu(var,val) percpu_to_op("add", per_cpu__##var, val)
+#define x86_sub_percpu(var,val) percpu_to_op("sub", per_cpu__##var, val)
+#define x86_or_percpu(var,val) percpu_to_op("or", per_cpu__##var, val)
+#endif /* !__ASSEMBLY__ */
+#endif /* !CONFIG_X86_64 */
+#endif /* _ASM_X86_PERCPU_H_ */

include/asm-x86/percpu_32.h deleted +0 ~~-128~~

128 lines not on this page · raw message

include/asm-x86/percpu_64.h deleted +0 ~~-32~~

@@ -1,32 +0,0 @@
-#ifndef _ASM_X8664_PERCPU_H_
-#define _ASM_X8664_PERCPU_H_
-#include <linux/compiler.h>
-
-/* Same as asm-generic/percpu.h, except that we store the per cpu offset
-   in the PDA. Longer term the PDA and every per cpu variable
-   should be just put into a single section and referenced directly
-   from %gs */
-
-#ifdef CONFIG_SMP
-
-#include <asm/pda.h>
-
-#define __per_cpu_offset(cpu) (cpu_pda(cpu)->data_offset)
-#define __my_cpu_offset read_pda(data_offset)
-
-#define per_cpu_offset(x) (__per_cpu_offset(x))
-
-/* A macro to avoid #include hell... */
-#define percpu_modcopy(pcpudst, src, size)			\
-do {								\
-	unsigned int __i;					\
-	for_each_possible_cpu(__i)				\
-		memcpy((pcpudst)+__per_cpu_offset(__i),		\
-		       (src), (size));				\
-} while (0)
-
-#endif	/* SMP */
-
-#include <asm-generic/percpu.h>
-
-#endif /* _ASM_X8664_PERCPU_H_ */

Signature · 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Ingo Molnar

2008-01-08 09:07 UTC #12 lore raw

In reply to [email protected]

Quoted 15 lines · * [email protected] <[email protected]> wrote:

This patchset simplifies the code that arches need to maintain to support per cpu functionality. Most of the code is moved into arch independent code. Only a minimal set of definitions is kept for each arch.
The patch also unifies the x86 arch so that there is only a single asm-x86/percpu.h
V1->V2:
- Add support for specifying attributes for per cpu declarations (preserves IA64 model(small) attribute).
- Drop first patch that removes the model(small) attribute for IA64
- Missing #endif in powerpc generic config / Wrong Kconfig
- Follow Randy's suggestions on how to do the Kconfig settings

Quoted 5 lines

V2->V3:
- fix x86_64 non-SMP case
- change SHIFT_PTR to SHIFT_PERCPU_PTR
- fix various percpu_modcopy()'s to reference correct per_cpu_offset()
- s390 has a special way to determine the pointer to a per cpu area

thanks, i've picked up the x86 and core bits, for testing.

i had the patch below for v2, it's still needed (because i didnt apply the s390/etc. bits), right?

Ingo

------------->
Subject: x86: let other arches build
From: Ingo Molnar <[email protected]>

let architectures which still have the DEFINE_PER_CPU/etc. build properly.

Signed-off-by Ingo Molnar <[email protected]>

---
 include/linux/percpu.h |    2 ++
 1 file changed, 2 insertions(+)

include/linux/percpu.h +2 -0

Index: linux-x86.q/include/linux/percpu.h
===================================================================
@@ -14,6 +14,7 @@
 #endif
 
 #ifdef CONFIG_SMP
+#ifndef DEFINE_PER_CPU
 #define DEFINE_PER_CPU(type, name)					\
 	__attribute__((__section__(".data.percpu")))			\
 	PER_CPU_ATTRIBUTES __typeof__(type) per_cpu__##name

@@ -32,6 +33,7 @@
 
 #define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
 #define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
+#endif
 
 /* Enough to cover all DEFINE_PER_CPUs in kernel, including modules. */
 #ifndef PERCPU_ENOUGH_ROOM

Mike Travis

author

2008-01-08 16:27 UTC #13 lore raw

Quoted 23 lines · Ingo Molnar

Quoted 15 lines · * [email protected] <[email protected]> wrote:
This patchset simplifies the code that arches need to maintain to support per cpu functionality. Most of the code is moved into arch independent code. Only a minimal set of definitions is kept for each arch.
The patch also unifies the x86 arch so that there is only a single asm-x86/percpu.h
V1->V2:
- Add support for specifying attributes for per cpu declarations (preserves IA64 model(small) attribute).
- Drop first patch that removes the model(small) attribute for IA64
- Missing #endif in powerpc generic config / Wrong Kconfig
- Follow Randy's suggestions on how to do the Kconfig settings
Quoted 5 lines
V2->V3:
- fix x86_64 non-SMP case
- change SHIFT_PTR to SHIFT_PERCPU_PTR
- fix various percpu_modcopy()'s to reference correct per_cpu_offset()
- s390 has a special way to determine the pointer to a per cpu area

Quoted 6 lines

thanks, i've picked up the x86 and core bits, for testing.
i had the patch below for v2, it's still needed (because i didnt apply the s390/etc. bits), right?
Ingo

Yes, good point. Thanks.

Mike

Quoted 26 lines

------------->
Subject: x86: let other arches build
From: Ingo Molnar <[email protected]>
let architectures which still have the DEFINE_PER_CPU/etc. build properly.
Signed-off-by Ingo Molnar <[email protected]>
---
 include/linux/percpu.h |    2 ++
 1 file changed, 2 insertions(+)
Index: linux-x86.q/include/linux/percpu.h
===================================================================
include/linux/percpu.h +1 -0
@@ -14,6 +14,7 @@
 #endif
 
 #ifdef CONFIG_SMP
+#ifndef DEFINE_PER_CPU
 #define DEFINE_PER_CPU(type, name)					\
 	__attribute__((__section__(".data.percpu")))			\
 	PER_CPU_ATTRIBUTES __typeof__(type) per_cpu__##name

include/linux/percpu.h line 39

Quoted 7 lines

include/linux/percpu.h +1 -0

@@ -32,6 +33,7 @@
 
 #define EXPORT_PER_CPU_SYMBOL(var) EXPORT_SYMBOL(per_cpu__##var)
 #define EXPORT_PER_CPU_SYMBOL_GPL(var) EXPORT_SYMBOL_GPL(per_cpu__##var)
+#endif
 
 /* Enough to cover all DEFINE_PER_CPUs in kernel, including modules. */
 #ifndef PERCPU_ENOUGH_ROOM

Christoph Lameter

2008-01-08 19:04 UTC #14 lore raw

In reply to Ingo Molnar

Quoted 2 lines · Ingo Molnar

i had the patch below for v2, it's still needed (because i didnt apply the s390/etc. bits), right?

Well the patch really should go through mm because it is a change that covers multiple arches. I think testing with this is fine. I think Mike has diffed this against Linus tree so this works but will now conflict with the modcopy patch already in mm.

Ingo Molnar

2008-01-08 22:16 UTC #15 lore raw

Quoted 9 lines · Christoph Lameter

Quoted 2 lines · Ingo Molnar
i had the patch below for v2, it's still needed (because i didnt apply the s390/etc. bits), right?
Well the patch really should go through mm because it is a change that covers multiple arches. I think testing with this is fine. I think Mike has diffed this against Linus tree so this works but will now conflict with the modcopy patch already in mm.

well we cannot really ack it for x86 inclusion without having tested it through, so it will stay in x86.git for some time. That approach found a few problems with v1 already. In any case, v3 is looking pretty good so far - and it's cool stuff - i'm all for unifying/generalizing arch code.

Ingo

Mike Travis

author

2008-01-08 23:53 UTC #16 lore raw

Quoted 11 lines · Ingo Molnar

Quoted 8 lines · Christoph Lameter
Quoted 2 lines · Ingo Molnar
i had the patch below for v2, it's still needed (because i didnt apply the s390/etc. bits), right?
Well the patch really should go through mm because it is a change that covers multiple arches. I think testing with this is fine. I think Mike has diffed this against Linus tree so this works but will now conflict with the modcopy patch already in mm.

Quoted 6 lines

well we cannot really ack it for x86 inclusion without having tested it through, so it will stay in x86.git for some time. That approach found a few problems with v1 already. In any case, v3 is looking pretty good so far - and it's cool stuff - i'm all for unifying/generalizing arch code.
Ingo

Hi Ingo,

You probably will want to pick up V4 though I didn't add that ifndef patch you mentioned earlier. There are no functional changes, basically only a rebasing on the correct mm version.

Thanks,
Mike

Sam Ravnborg

2008-01-08 19:26 UTC #17 lore raw

In reply to [email protected]

Quoted 15 lines · On Mon, Jan 07, 2008 at 06:11:42PM -0800, [email protected] wrote:

This patchset simplifies the code that arches need to maintain to support per cpu functionality. Most of the code is moved into arch independent code. Only a minimal set of definitions is kept for each arch.
The patch also unifies the x86 arch so that there is only a single asm-x86/percpu.h
V1->V2:
- Add support for specifying attributes for per cpu declarations (preserves IA64 model(small) attribute).
- Drop first patch that removes the model(small) attribute for IA64
- Missing #endif in powerpc generic config / Wrong Kconfig
- Follow Randy's suggestions on how to do the Kconfig settings

Quoted 5 lines

V2->V3:
- fix x86_64 non-SMP case
- change SHIFT_PTR to SHIFT_PERCPU_PTR
- fix various percpu_modcopy()'s to reference correct per_cpu_offset()
- s390 has a special way to determine the pointer to a per cpu area

In your changelog comments you have this:
V1->V2
- ...

V2->V3
- ...

But that really belongs below the "end-of-changelog" comment as this info is relevant only for this submission to lkml and not in whats get committed.

As your submission did not include an RFC I assume this is expected to be the final version.

Sam

Mike Travis

author

2008-01-08 20:29 UTC #18 lore raw

Quoted 3 lines

As your submission did not include an RFC I assume this is expected to be the final version.

I have another version coming that includes the changes requested by you and others.

Thanks,
Mike