Skip to contentSkip to conversation

ARM: optimize copy_page() for systems with cache line != 32

PATCH RESEND Superseded Kirill A. Shutemov 4 participants 10 messages Sep 12, 2009 Arm (32-bit)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover ARM: optimize copy_page() for systems with cache line != 32

I can't find this changes in any git tree, so resend.

Russell, please add these patches into your patch queue for 2.6.32.

Thanks.

#1 lore raw

PATCH RESEND v2 1/2 ARM: Introduce ARM_L1_CACHE_SHIFT to define cache line size

Currently kernel believes that all ARM CPUs have L1_CACHE_SHIFT == 5. It's not true at least for CPUs based on Cortex-A8.

List of CPUs with cache line size != 32 should be expanded later.

---
 arch/arm/include/asm/cache.h |    2 +-
 arch/arm/mm/Kconfig          |    5 +++++
 2 files changed, 6 insertions(+), 1 deletions(-)
arch/arm/include/asm/cache.h +1 -1
index feaa75f..2ee7743 100644
@@ -4,7 +4,7 @@
 #ifndef __ASMARM_CACHE_H
 #define __ASMARM_CACHE_H
 
-#define L1_CACHE_SHIFT		5
+#define L1_CACHE_SHIFT		(CONFIG_ARM_L1_CACHE_SHIFT)
 #define L1_CACHE_BYTES		(1 << L1_CACHE_SHIFT)
 
 /*
arch/arm/mm/Kconfig +5 -0
index 83c025e..3c37d4c 100644
@@ -771,3 +771,8 @@ config CACHE_XSC3L2
 	select OUTER_CACHE
 	help
 	  This option enables the L2 cache on XScale3.
+
+config ARM_L1_CACHE_SHIFT
+	int
+	default 6 if ARCH_OMAP3
+	default 5
Signature · 1 line
1.6.4.2
#2 lore raw

PATCH RESEND v2 1/2 ARM: Introduce ARM_L1_CACHE_SHIFT to define cache line size

Hello.

Quoted 16 lines · Kirill A. Shutemov

Currently kernel believes that all ARM CPUs have L1_CACHE_SHIFT == 5. It's not true at least for CPUs based on Cortex-A8.

List of CPUs with cache line size != 32 should be expanded later.

---
 arch/arm/include/asm/cache.h |    2 +-
 arch/arm/mm/Kconfig          |    5 +++++
 2 files changed, 6 insertions(+), 1 deletions(-)
arch/arm/include/asm/cache.h +0 -0
index feaa75f..2ee7743 100644

arch/arm/include/asm/cache.h line 8

Quoted 6 lines
arch/arm/include/asm/cache.h +1 -1
@@ -4,7 +4,7 @@
 #ifndef __ASMARM_CACHE_H
 #define __ASMARM_CACHE_H
 
-#define L1_CACHE_SHIFT		5
+#define L1_CACHE_SHIFT		(CONFIG_ARM_L1_CACHE_SHIFT)
 

Parens not needed.

WBR, Sergei

#3 lore raw

PATCH RESEND v2 1/2 ARM: Introduce ARM_L1_CACHE_SHIFT to define cache line size

Quoted 15 lines · Sergei Shtylyov

Hello.

Quoted 11 lines · Kirill A. Shutemov

Currently kernel believes that all ARM CPUs have L1_CACHE_SHIFT == 5. It's not true at least for CPUs based on Cortex-A8.

List of CPUs with cache line size != 32 should be expanded later.

---
?arch/arm/include/asm/cache.h | ? ?2 +-
?arch/arm/mm/Kconfig ? ? ? ? ?| ? ?5 +++++
?2 files changed, 6 insertions(+), 1 deletions(-)

Quoted 12 lines
Quoted 10 lines
arch/arm/include/asm/cache.h +0 -0
index feaa75f..2ee7743 100644
@@ -4,7 +4,7 @@

?#ifndef __ASMARM_CACHE_H
?#define __ASMARM_CACHE_H
?-#define L1_CACHE_SHIFT ? ? ? ? 5
+#define L1_CACHE_SHIFT ? ? ? ? (CONFIG_ARM_L1_CACHE_SHIFT)

?Parens not needed.

You can write anything in your .config (like 2+3), so I think parens is needed.

#4 lore raw

PATCH RESEND v2 1/2 ARM: Introduce ARM_L1_CACHE_SHIFT to define cache line size

Hello.

Quoted 25 lines · Kirill A. Shutemov
Quoted 11 lines
Quoted 11 lines

Currently kernel believes that all ARM CPUs have L1_CACHE_SHIFT == 5. It's not true at least for CPUs based on Cortex-A8.

List of CPUs with cache line size != 32 should be expanded later.

---
 arch/arm/include/asm/cache.h |    2 +-
 arch/arm/mm/Kconfig          |    5 +++++
 2 files changed, 6 insertions(+), 1 deletions(-)
Quoted 13 lines

arch/arm/include/asm/cache.h line 9

Quoted 11 lines
arch/arm/include/asm/cache.h +1 -0
index feaa75f..2ee7743 100644
@@ -4,7 +4,7 @@
 #ifndef __ASMARM_CACHE_H
 #define __ASMARM_CACHE_H
 -#define L1_CACHE_SHIFT         5
+#define L1_CACHE_SHIFT         (CONFIG_ARM_L1_CACHE_SHIFT)
 
 

Parens not needed.

Quoted 2 lines

You can write anything in your .config (like 2+3), so I think parens is needed.

You can -- but that won't be an integer option.

WBR, Sergei

#5 lore raw

PATCH RESEND v2 1/2 ARM: Introduce ARM_L1_CACHE_SHIFT to define cache line size

Quoted 30 lines · Sergei Shtylyov

Hello.

Quoted 26 lines · Kirill A. Shutemov
Quoted 25 lines
Quoted 16 lines

Currently kernel believes that all ARM CPUs have L1_CACHE_SHIFT == 5. It's not true at least for CPUs based on Cortex-A8.

List of CPUs with cache line size != 32 should be expanded later.

---
?arch/arm/include/asm/cache.h | ? ?2 +-
?arch/arm/mm/Kconfig ? ? ? ? ?| ? ?5 +++++
?2 files changed, 6 insertions(+), 1 deletions(-)

arch/arm/include/asm/cache.h +0 -0
index feaa75f..2ee7743 100644
@@ -4,7 +4,7 @@
Quoted 6 lines

?#ifndef __ASMARM_CACHE_H
?#define __ASMARM_CACHE_H
?-#define L1_CACHE_SHIFT ? ? ? ? 5
+#define L1_CACHE_SHIFT ? ? ? ? (CONFIG_ARM_L1_CACHE_SHIFT)

?Parens not needed.

Quoted 5 lines
Quoted 3 lines

You can write anything in your .config (like 2+3), so I think parens is needed.

?You can -- but that won't be an integer option.

But kernel will be still compilable, but broken.

I think parens for defines from .config is a good practice.

#6 lore raw

PATCH RESEND v2 1/2 ARM: Introduce ARM_L1_CACHE_SHIFT to define cache line size

Quoted 9 lines · Kirill A. Shutemov
Quoted 7 lines · Sergei Shtylyov

Hello.

Quoted 3 lines · Kirill A. Shutemov

You can write anything in your .config (like 2+3), so I think parens is needed.

Quoted 5 lines
Quoted 1 line

?You can -- but that won't be an integer option.

But kernel will be still compilable, but broken.

I think parens for defines from .config is a good practice.

However, defines are not taken from .config. The .config file is processed by the kernel configurator, which writes out the autoconf.h header. It does this by re-evaluating the contents of .config against all the Kconfig files. It does not merely copy .config to autoconf.h.

In fact, if you try to write '2+3' into your .config, Kconfig will reject it. Eg:

.config:210:warning: symbol value '127+1' invalid for OMAP_32K_TIMER_HZ

and then it asks you to provide a valid value instead.

So yes, as pointed out, the parens are not required since Kconfig will ensure that the symbol is a simple integer.

#7 lore raw

PATCH RESEND v2 2/2 ARM: copy_page.S: take into account the size of the cache line

In reply to Kirill A. Shutemov

Optimized version of copy_page() was written with assumption that cache line size is 32 bytes. On Cortex-A8 cache line size is 64 bytes.

This patch tries to generalize copy_page() to work with any cache line size if cache line size is multiple of 16 and page size is multiple of two cache line size.

After this optimization we've got ~25% speedup on OMAP3(tested in userspace).

There is test for kernelspace which trigger copy-on-write after fork():

#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define BUF_SIZE (10000*4096)
#define NFORK 200

int main(int argc, char **argv)

 {
         char *buf = malloc(BUF_SIZE);
         int i;

         memset(buf, 0, BUF_SIZE);

         for(i = 0; i < NFORK; i++) {
                 if (fork()) {
                         wait(NULL);
                 } else {
                         int j;

                         for(j = 0; j < BUF_SIZE; j+= 4096)
                                 buf[j] = (j & 0xFF) + 1;
                         break;
                 }
         }

         free(buf);
         return 0;

}

Before optimization this test takes ~66 seconds, after optimization takes ~56 seconds.

V2:
- include <asm/cahce.h>
- remove unnecessary parens and fix style

---
 arch/arm/lib/copy_page.S |   16 ++++++++--------
 1 files changed, 8 insertions(+), 8 deletions(-)
arch/arm/lib/copy_page.S +8 -8
index 6ae04db..6ee2f67 100644
@@ -12,8 +12,9 @@
 #include <linux/linkage.h>
 #include <asm/assembler.h>
 #include <asm/asm-offsets.h>
+#include <asm/cache.h>
 
-#define COPY_COUNT (PAGE_SZ/64 PLD( -1 ))
+#define COPY_COUNT (PAGE_SZ / (2 * L1_CACHE_BYTES) PLD( -1 ))
 
 		.text
 		.align	5
@@ -26,17 +27,16 @@
 ENTRY(copy_page)
 		stmfd	sp!, {r4, lr}			@	2
 	PLD(	pld	[r1, #0]		)
-	PLD(	pld	[r1, #32]		)
+	PLD(	pld	[r1, #L1_CACHE_BYTES]		)
 		mov	r2, #COPY_COUNT			@	1
 		ldmia	r1!, {r3, r4, ip, lr}		@	4+1
-1:	PLD(	pld	[r1, #64]		)
-	PLD(	pld	[r1, #96]		)
-2:		stmia	r0!, {r3, r4, ip, lr}		@	4
-		ldmia	r1!, {r3, r4, ip, lr}		@	4+1
-		stmia	r0!, {r3, r4, ip, lr}		@	4
-		ldmia	r1!, {r3, r4, ip, lr}		@	4+1
+1:	PLD(	pld	[r1, #2 * L1_CACHE_BYTES])
+	PLD(	pld	[r1, #3 * L1_CACHE_BYTES])
+2:
+	.rept	(2 * L1_CACHE_BYTES / 16 - 1)
 		stmia	r0!, {r3, r4, ip, lr}		@	4
 		ldmia	r1!, {r3, r4, ip, lr}		@	4
+	.endr
 		subs	r2, r2, #1			@	1
 		stmia	r0!, {r3, r4, ip, lr}		@	4
 		ldmgtia	r1!, {r3, r4, ip, lr}		@	4
Signature · 1 line
1.6.4.2
#8 lore raw

PATCH RESEND v2 2/2 ARM: copy_page.S: take into account the size of the cache line

Hello,

Just two minor nitpicks:

Quoted 2 lines

V2:
- include <asm/cahce.h>

s/cahce/cache/

Quoted 1 line

- remove unnecessary parens and fix style

s/parens/parenthesis/

Note I didn't look at the patch.

Best regards
Uwe

Signature · 2 lines
Pengutronix e.K.                              | Uwe Kleine-K?nig            |
Industrial Linux Solutions                    | http://www.pengutronix.de/  |
#9 lore raw

PATCH RESEND v2 2/2 ARM: copy_page.S: take into account the size of the cache line

Hello.

Quoted 6 lines · Uwe Kleine-K?nig

s/cahce/cache/

Quoted 2 lines

- remove unnecessary parens and fix style

s/parens/parenthesis/

Isn't "parens" a common contraction for "parentheses"?

WBR, Sergei