Optimize copy_page by re-arranging instruction sequence and saving register
RFC PATCH 2/2 Optimize copy_page by re-arranging instruction sequence and saving register
From: Ma Ling <[email protected]>
Load and write operation occupy about 35% and 10% respectively for most industry benchmarks. Fetched 16-aligned bytes code include about 4 instructions, implying 1.34(0.35 * 4) load, 0.4 write. Modern CPU support 2 load and 1 write per cycle, so throughput from write is bottleneck for memcpy or copy_page, and some slight CPU only support one mem operation per cycle. So it is enough to issue one read and write instruction per cycle, and we can save registers.
In this patch we also re-arrange instruction sequence to improve performance The performance on atom is improved about 11%, 9% on hot/cold-cache case respectively.
--- arch/x86/lib/copy_page_64.S | 103 +++++++++++++++++------------------------- 1 files changed, 42 insertions(+), 61 deletions(-)
arch/x86/lib/copy_page_64.S +42 -61
@@ -20,76 +20,57 @@ ENDPROC(copy_page_rep) ENTRY(copy_page) CFI_STARTPROC - subq $2*8, %rsp - CFI_ADJUST_CFA_OFFSET 2*8 - movq %rbx, (%rsp) - CFI_REL_OFFSET rbx, 0 - movq %r12, 1*8(%rsp) - CFI_REL_OFFSET r12, 1*8 + mov $(4096/64)-5, %ecx - movl $(4096/64)-5, %ecx - .p2align 4 .Loop64: - dec %rcx - - movq 0x8*0(%rsi), %rax - movq 0x8*1(%rsi), %rbx - movq 0x8*2(%rsi), %rdx - movq 0x8*3(%rsi), %r8 - movq 0x8*4(%rsi), %r9 - movq 0x8*5(%rsi), %r10 - movq 0x8*6(%rsi), %r11 - movq 0x8*7(%rsi), %r12 - prefetcht0 5*64(%rsi) - - movq %rax, 0x8*0(%rdi) - movq %rbx, 0x8*1(%rdi) - movq %rdx, 0x8*2(%rdi) - movq %r8, 0x8*3(%rdi) - movq %r9, 0x8*4(%rdi) - movq %r10, 0x8*5(%rdi) - movq %r11, 0x8*6(%rdi) - movq %r12, 0x8*7(%rdi) - - leaq 64 (%rsi), %rsi - leaq 64 (%rdi), %rdi - + decb %cl + + movq 0x8*0(%rsi), %r10 + movq 0x8*1(%rsi), %rax + movq 0x8*2(%rsi), %r8 + movq 0x8*3(%rsi), %r9 + movq %r10, 0x8*0(%rdi) + movq %rax, 0x8*1(%rdi) + movq %r8, 0x8*2(%rdi) + movq %r9, 0x8*3(%rdi) + + movq 0x8*4(%rsi), %r10 + movq 0x8*5(%rsi), %rax + movq 0x8*6(%rsi), %r8 + movq 0x8*7(%rsi), %r9 + leaq 64(%rsi), %rsi + movq %r10, 0x8*4(%rdi) + movq %rax, 0x8*5(%rdi) + movq %r8, 0x8*6(%rdi) + movq %r9, 0x8*7(%rdi) + leaq 64(%rdi), %rdi jnz .Loop64 - movl $5, %ecx - .p2align 4 + mov $5, %dl .Loop2: - decl %ecx - - movq 0x8*0(%rsi), %rax - movq 0x8*1(%rsi), %rbx - movq 0x8*2(%rsi), %rdx - movq 0x8*3(%rsi), %r8 - movq 0x8*4(%rsi), %r9 - movq 0x8*5(%rsi), %r10 - movq 0x8*6(%rsi), %r11 - movq 0x8*7(%rsi), %r12 - - movq %rax, 0x8*0(%rdi) - movq %rbx, 0x8*1(%rdi) - movq %rdx, 0x8*2(%rdi) - movq %r8, 0x8*3(%rdi) - movq %r9, 0x8*4(%rdi) - movq %r10, 0x8*5(%rdi) - movq %r11, 0x8*6(%rdi) - movq %r12, 0x8*7(%rdi) - - leaq 64(%rdi), %rdi + decb %dl + movq 0x8*0(%rsi), %r10 + movq 0x8*1(%rsi), %rax + movq 0x8*2(%rsi), %r8 + movq 0x8*3(%rsi), %r9 + movq %r10, 0x8*0(%rdi) + movq %rax, 0x8*1(%rdi) + movq %r8, 0x8*2(%rdi) + movq %r9, 0x8*3(%rdi) + + movq 0x8*4(%rsi), %r10 + movq 0x8*5(%rsi), %rax + movq 0x8*6(%rsi), %r8 + movq 0x8*7(%rsi), %r9 leaq 64(%rsi), %rsi + movq %r10, 0x8*4(%rdi) + movq %rax, 0x8*5(%rdi) + movq %r8, 0x8*6(%rdi) + movq %r9, 0x8*7(%rdi) + leaq 64(%rdi), %rdi jnz .Loop2 - movq (%rsp), %rbx - CFI_RESTORE rbx - movq 1*8(%rsp), %r12 - CFI_RESTORE r12 - addq $2*8, %rsp - CFI_ADJUST_CFA_OFFSET -2*8 ret .Lcopy_page_end: CFI_ENDPROC
Signature · 1 line
1.6.5.2
Quoted 9 lines · [email protected] writes:
From: Ma Ling <[email protected]>
Load and write operation occupy about 35% and 10% respectively for most industry benchmarks. Fetched 16-aligned bytes code include about 4 instructions, implying 1.34(0.35 * 4) load, 0.4 write. Modern CPU support 2 load and 1 write per cycle, so throughput from write is bottleneck for memcpy or
copy_page, and some slight CPU only support one mem operation per cycle. So it is enough to issue one read and write instruction per cycle, and we can save registers.
I don't think "saving registers" is a useful goal here.
Quoted 4 lines
In this patch we also re-arrange instruction sequence to improve performance The performance on atom is improved about 11%, 9% on hot/cold-cache case respectively.
That's great, but the question is what happened to the older CPUs that also this sequence. It may be safer to add a new variant for Atom, unless you can benchmark those too.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Quoted 10 lines
Quoted 6 lines
Load and write operation occupy about 35% and 10% respectively for most industry benchmarks. Fetched 16-aligned bytes code include about 4 instructions, implying 1.34(0.35 * 4) load, 0.4 write. Modern CPU support 2 load and 1 write per cycle, so throughput from write is bottleneck for memcpy or
copy_page, and some slight CPU only support one mem operation per cycle. So it is enough to issue oneread
Quoted 1 line
and write instruction per cycle, and we can save registers.
I don't think "saving registers" is a useful goal here.
Ling: issuing one read and write ops in one cycle is enough for copy_page or memcpy performance, so we could avoid saving and restoring registers operation.
Quoted 8 lines
Quoted 4 lines
In this patch we also re-arrange instruction sequence to improve performance The performance on atom is improved about 11%, 9% on hot/cold-cache case respectively.
That's great, but the question is what happened to the older CPUs that also this sequence. It may be safer to add a new variant for Atom, unless you can benchmark those too.
Ling:
I tested new and original version on core2, the patch improved performance about 9%, Although core2 is out-of-order pipeline and weaken instruction sequence requirement, because of ROB size limitation, new patch issues write operation earlier and get more parallelism possibility for the pair of write and load ops and better result. Attached core2-cpu-info (I have no older machine)
Thanks
Ling
Quoted 1 line
I tested new and original version on core2, the patch improved performance about 9%,
That's not useful because core2 doesn't use this variant, it uses the rep string variant. Primary user is P4.
Quoted 4 lines
Although core2 is out-of-order pipeline and weaken instruction sequence requirement, because of ROB size limitation, new patch issues write operation earlier and get more parallelism possibility for the pair of write and load ops and better result. Attached core2-cpu-info (I have no older machine)
If you can't test the CPUs who run this code I think it's safer if you add a new variant for Atom, not change the existing well tested code. Otherwise you risk performance regressions on these older CPUs.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 3 lines
If you can't test the CPUs who run this code I think it's safer if you add a new variant for Atom, not change the existing well tested code. Otherwise you risk performance regressions on these older CPUs.
I found one older machine, and tested the code on it, the results between them are almost the same as below(attached cpu info).
1 copy_page_org copy_page_new
2 TPT: Len 4096, alignment 0/ 0: 2252 2218
3 TPT: Len 4096, alignment 0/ 0: 2244 2193
4 TPT: Len 4096, alignment 0/ 0: 2261 2227
5 TPT: Len 4096, alignment 0/ 0: 2235 2244
6 TPT: Len 4096, alignment 0/ 0: 2261 2184Thanks
Ling
Quoted 5 lines · Ling
Quoted 3 lines
If you can't test the CPUs who run this code I think it's safer if you add a new variant for Atom, not change the existing well tested code. Otherwise you risk performance regressions on these older CPUs.
I found one older machine, and tested the code on it, the results between them are almost the same as below(attached cpu info).
Was that a P4 (family 15)?
Those were the main users. There were a few others, but they are obscure (early steppings of K8)
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 9 lines · On Thu, Oct 11, 2012 at 08:29:08PM +0800, [email protected] wrote:
From: Ma Ling <[email protected]>
Load and write operation occupy about 35% and 10% respectively for most industry benchmarks. Fetched 16-aligned bytes code include about 4 instructions, implying 1.34(0.35 * 4) load, 0.4 write. Modern CPU support 2 load and 1 write per cycle, so throughput from write is bottleneck for memcpy or
copy_page, and some slight CPU only support one mem operation per cycle. So it is enough to issue one read and write instruction per cycle, and we can save registers.
So is that also true for AMD CPUs?
Quoted 142 lines
In this patch we also re-arrange instruction sequence to improve performance The performance on atom is improved about 11%, 9% on hot/cold-cache case respectively.
Signed-off-by Ma Ling <[email protected]>--- arch/x86/lib/copy_page_64.S | 103 +++++++++++++++++------------------------- 1 files changed, 42 insertions(+), 61 deletions(-)arch/x86/lib/copy_page_64.S +42
-61@@ -20,76 +20,57 @@ ENDPROC(copy_page_rep) ENTRY(copy_page) CFI_STARTPROC - subq $2*8, %rsp - CFI_ADJUST_CFA_OFFSET 2*8 - movq %rbx, (%rsp) - CFI_REL_OFFSET rbx, 0 - movq %r12, 1*8(%rsp) - CFI_REL_OFFSET r12, 1*8 + mov $(4096/64)-5, %ecx - movl $(4096/64)-5, %ecx - .p2align 4 .Loop64: - dec %rcx - - movq 0x8*0(%rsi), %rax - movq 0x8*1(%rsi), %rbx - movq 0x8*2(%rsi), %rdx - movq 0x8*3(%rsi), %r8 - movq 0x8*4(%rsi), %r9 - movq 0x8*5(%rsi), %r10 - movq 0x8*6(%rsi), %r11 - movq 0x8*7(%rsi), %r12 - prefetcht0 5*64(%rsi) - - movq %rax, 0x8*0(%rdi) - movq %rbx, 0x8*1(%rdi) - movq %rdx, 0x8*2(%rdi) - movq %r8, 0x8*3(%rdi) - movq %r9, 0x8*4(%rdi) - movq %r10, 0x8*5(%rdi) - movq %r11, 0x8*6(%rdi) - movq %r12, 0x8*7(%rdi) - - leaq 64 (%rsi), %rsi - leaq 64 (%rdi), %rdi - + decb %cl + + movq 0x8*0(%rsi), %r10 + movq 0x8*1(%rsi), %rax + movq 0x8*2(%rsi), %r8 + movq 0x8*3(%rsi), %r9 + movq %r10, 0x8*0(%rdi) + movq %rax, 0x8*1(%rdi) + movq %r8, 0x8*2(%rdi) + movq %r9, 0x8*3(%rdi) + + movq 0x8*4(%rsi), %r10 + movq 0x8*5(%rsi), %rax + movq 0x8*6(%rsi), %r8 + movq 0x8*7(%rsi), %r9 + leaq 64(%rsi), %rsi + movq %r10, 0x8*4(%rdi) + movq %rax, 0x8*5(%rdi) + movq %r8, 0x8*6(%rdi) + movq %r9, 0x8*7(%rdi) + leaq 64(%rdi), %rdi jnz .Loop64 - movl $5, %ecx - .p2align 4 + mov $5, %dl .Loop2: - decl %ecx - - movq 0x8*0(%rsi), %rax - movq 0x8*1(%rsi), %rbx - movq 0x8*2(%rsi), %rdx - movq 0x8*3(%rsi), %r8 - movq 0x8*4(%rsi), %r9 - movq 0x8*5(%rsi), %r10 - movq 0x8*6(%rsi), %r11 - movq 0x8*7(%rsi), %r12 - - movq %rax, 0x8*0(%rdi) - movq %rbx, 0x8*1(%rdi) - movq %rdx, 0x8*2(%rdi) - movq %r8, 0x8*3(%rdi) - movq %r9, 0x8*4(%rdi) - movq %r10, 0x8*5(%rdi) - movq %r11, 0x8*6(%rdi) - movq %r12, 0x8*7(%rdi) - - leaq 64(%rdi), %rdi + decb %dl + movq 0x8*0(%rsi), %r10 + movq 0x8*1(%rsi), %rax + movq 0x8*2(%rsi), %r8 + movq 0x8*3(%rsi), %r9 + movq %r10, 0x8*0(%rdi) + movq %rax, 0x8*1(%rdi) + movq %r8, 0x8*2(%rdi) + movq %r9, 0x8*3(%rdi) + + movq 0x8*4(%rsi), %r10 + movq 0x8*5(%rsi), %rax + movq 0x8*6(%rsi), %r8 + movq 0x8*7(%rsi), %r9 leaq 64(%rsi), %rsi + movq %r10, 0x8*4(%rdi) + movq %rax, 0x8*5(%rdi) + movq %r8, 0x8*6(%rdi) + movq %r9, 0x8*7(%rdi) + leaq 64(%rdi), %rdi jnz .Loop2 - movq (%rsp), %rbx - CFI_RESTORE rbx - movq 1*8(%rsp), %r12 - CFI_RESTORE r12 - addq $2*8, %rsp - CFI_ADJUST_CFA_OFFSET -2*8 ret .Lcopy_page_end: CFI_ENDPROCSignature · 7 lines
1.6.5.2 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Quoted 10 lines
Quoted 6 lines
Load and write operation occupy about 35% and 10% respectively for most industry benchmarks. Fetched 16-aligned bytes code include about 4 instructions, implying 1.34(0.35 * 4) load, 0.4 write. Modern CPU support 2 load and 1 write per cycle, so throughput from write is bottleneck for memcpy or
copy_page, and some slight CPU only support one mem operation per cycle. So it is enough to issue oneread
Quoted 1 line
and write instruction per cycle, and we can save registers.
So is that also true for AMD CPUs?
Although Bulldozer put 32byte instruction into decoupled 16byte entry buffers, it still decode 4 instructions per cycle, so 4 instructions will be fed into execution unit and 2 loads ,1 write will be issued per cycle.
Thanks
Ling
Borislav Petkov
maintainerQuoted 6 lines · Ling
Quoted 6 lines
Quoted 6 lines
Load and write operation occupy about 35% and 10% respectively for most industry benchmarks. Fetched 16-aligned bytes code include about 4 instructions, implying 1.34(0.35 * 4) load, 0.4 write. Modern CPU support 2 load and 1 write per cycle, so throughput from write is bottleneck for memcpy or
copy_page, and some slight CPU only support one mem operation per cycle. So it is enough to issue one
Quoted 7 lines
Quoted 4 lines
read
Quoted 1 line
and write instruction per cycle, and we can save registers.
So is that also true for AMD CPUs?
Although Bulldozer put 32byte instruction into decoupled 16byte entry buffers, it still decode 4 instructions per cycle, so 4 instructions will be fed into execution unit and 2 loads ,1 write will be issued per cycle.
I'd be very interested with what benchmarks are you seeing that perf improvement on Atom and who knows, maybe I could find time to run them on Bulldozer and see how your patch behaves there :-).
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.Quoted 9 lines
Quoted 5 lines
Quoted 1 line
So is that also true for AMD CPUs?
Although Bulldozer put 32byte instruction into decoupled 16byte entry buffers, it still decode 4 instructions per cycle, so 4 instructions will be fed into execution unit and
2 loads ,1 write will be issued per cycle.I'd be very interested with what benchmarks are you seeing that perf improvement on Atom and who knows, maybe I could find time to run them on Bulldozer and see how your patch behaves there :-).M
I use another benchmark from gcc, there are many code, and extract one simple benchmark, you may use it to test (cc -o copy_page copy_page.c), my initial result shows new copy page version is still better on bulldozer machine, because the machine is first release, please verify result. And CC to Ian.
Thanks
Ling
#include<stdio.h>
#include <stdlib.h>
typedef unsigned long long int hp_timing_t;
#define MAXSAMPLESTPT 1000 #define MAXCOPYSIZE (1024 * 1024) #define ORIG 0 #define NEW 1
static char* buf1 = NULL;
static char* buf2 = NULL;
static int repeat_one_test = 32;
hp_timing_t _dl_hp_timing_overhead;
# define HP_TIMING_NOW(Var) \
({ unsigned long long _hi, _lo; \
asm volatile ("rdtsc" : "=a" (_lo), "=d" (_hi)); \
(Var) = _hi << 32 | _lo; })#define HP_TIMING_DIFF(Diff, Start, End) (Diff) = ((End) - (Start))
#define HP_TIMING_TOTAL(total_time, start, end) \
do \
{ \
hp_timing_t tmptime; \
HP_TIMING_DIFF (tmptime, start + _dl_hp_timing_overhead, end); \
total_time += tmptime; \
} \while (0)
#define HP_TIMING_BEST(best_time, start, end) \
do \
{ \
hp_timing_t tmptime; \
HP_TIMING_DIFF (tmptime, start + _dl_hp_timing_overhead, end); \
if (best_time > tmptime) \
best_time = tmptime; \
} \while (0)
void copy_page_org(char *dst, char *src, int len); void copy_page_new(char *dst, char *src, int len); void memcpy_c(char *dst, char *src, int len); void (*do_memcpy)(char *dst, char *src, int len);
static voiddo_one_test ( char *dst, char *src,
size_t len)
{
hp_timing_t start __attribute ((unused));
hp_timing_t stop __attribute ((unused));
hp_timing_t best_time = ~ (hp_timing_t) 0;
size_t i,j;
for (i = 0; i < repeat_one_test; ++i)
{
HP_TIMING_NOW (start);
do_memcpy ( dst, src, len);
HP_TIMING_NOW (stop);
HP_TIMING_BEST (best_time, start, stop);
}
printf ("\t%zd", (size_t) best_time);
}static voiddo_test (size_t align1, size_t align2, size_t len) {
size_t i, j;
char *s1, *s2;
s1 = (char *) (buf1 + align1); s2 = (char *) (buf2 + align2);
printf ("TPT: Len %4zd, alignment %2zd/%2zd:", len, align1, align2); do_memcpy = copy_page_org;
do_one_test (s2, s1, len);
do_memcpy = copy_page_new;
do_one_test (s2+ (1 << 16), s1 + (1 << 16), len);
putchar ('\n');
}static test_init(void)
{
int i;
buf1 = valloc(MAXCOPYSIZE);
buf2 = valloc(MAXCOPYSIZE);
for (i = 0; i < MAXCOPYSIZE ; i = i + 64) {
buf1[i] = buf2[i] = i & 0xff;
}}
void copy_page_new(char *dst, char *src, int len)
{
__asm__("mov $(4096/64)-5, %ecx");
__asm__("1:");
__asm__("prefetcht0 5*64(%rsi)");
__asm__("decb %cl");
__asm__("movq 0x8*0(%rsi), %r10");
__asm__("movq 0x8*1(%rsi), %rax");
__asm__("movq 0x8*2(%rsi), %r8");
__asm__("movq 0x8*3(%rsi), %r9");
__asm__("movq %r10, 0x8*0(%rdi)");
__asm__("movq %rax, 0x8*1(%rdi)");
__asm__("movq %r8, 0x8*2(%rdi)");
__asm__("movq %r9, 0x8*3(%rdi)");
__asm__("movq 0x8*4(%rsi), %r10");
__asm__("movq 0x8*5(%rsi), %rax");
__asm__("movq 0x8*6(%rsi), %r8");
__asm__("movq 0x8*7(%rsi), %r9");
__asm__("leaq 64(%rsi), %rsi");
__asm__("movq %r10, 0x8*4(%rdi)");
__asm__("movq %rax, 0x8*5(%rdi)");
__asm__("movq %r8, 0x8*6(%rdi)");
__asm__("movq %r9, 0x8*7(%rdi)");
__asm__("leaq 64(%rdi), %rdi");
__asm__("jnz 1b");
__asm__("mov $5, %dl");
__asm__("2:");
__asm__("decb %dl");
__asm__("movq 0x8*0(%rsi), %r10");
__asm__("movq 0x8*1(%rsi), %rax");
__asm__("movq 0x8*2(%rsi), %r8");
__asm__("movq 0x8*3(%rsi), %r9");
__asm__("movq %r10, 0x8*0(%rdi)");
__asm__("movq %rax, 0x8*1(%rdi)");
__asm__("movq %r8, 0x8*2(%rdi)");
__asm__("movq %r9, 0x8*3(%rdi)");
__asm__("movq 0x8*4(%rsi), %r10");
__asm__("movq 0x8*5(%rsi), %rax");
__asm__("movq 0x8*6(%rsi), %r8");
__asm__("movq 0x8*7(%rsi), %r9");
__asm__("leaq 64(%rsi), %rsi");
__asm__("movq %r10, 0x8*4(%rdi)");
__asm__("movq %rax, 0x8*5(%rdi)");
__asm__("movq %r8, 0x8*6(%rdi)");
__asm__("movq %r9, 0x8*7(%rdi)");
__asm__("leaq 64(%rdi), %rdi");
__asm__("jnz 2b");}
void copy_page_org(char *dst, char *src, int len)
{
__asm__("subq $2*8,%rsp");
__asm__("movq %rbx,(%rsp)");
__asm__("movq %r12,1*8(%rsp)");
__asm__("movl $(4096/64)-5,%ecx");
__asm__(".p2align 4");
__asm__("1:");
__asm__("dec %rcx");
__asm__("movq (%rsi), %rax");
__asm__("movq 8 (%rsi), %rbx");
__asm__("movq 16 (%rsi), %rdx");
__asm__("movq 24 (%rsi), %r8");
__asm__("movq 32 (%rsi), %r9");
__asm__("movq 40 (%rsi), %r10");
__asm__("movq 48 (%rsi), %r11");
__asm__("movq 56 (%rsi), %r12");
__asm__("prefetcht0 5*64(%rsi)");
__asm__("movq %rax, (%rdi)");
__asm__("movq %rbx, 8 (%rdi)");
__asm__("movq %rdx, 16 (%rdi)");
__asm__("movq %r8, 24 (%rdi)");
__asm__("movq %r9, 32 (%rdi)");
__asm__("movq %r10, 40 (%rdi)");
__asm__("movq %r11, 48 (%rdi)");
__asm__("movq %r12, 56 (%rdi)");
__asm__("leaq 64 (%rsi), %rsi");
__asm__("leaq 64 (%rdi), %rdi");
__asm__("jnz 1b");
__asm__("movl $5,%ecx");
__asm__(".p2align 4");
__asm__("2:");
__asm__("decl %ecx");
__asm__("movq (%rsi), %rax");
__asm__("movq 8 (%rsi), %rbx");
__asm__("movq 16 (%rsi), %rdx");
__asm__("movq 24 (%rsi), %r8");
__asm__("movq 32 (%rsi), %r9");
__asm__("movq 40 (%rsi), %r10");
__asm__("movq 48 (%rsi), %r11");
__asm__("movq 56 (%rsi), %r12");
__asm__("movq %rax, (%rdi)");
__asm__("movq %rbx, 8 (%rdi)");
__asm__("movq %rdx, 16 (%rdi)");
__asm__("movq %r8, 24 (%rdi)");
__asm__("movq %r9, 32 (%rdi)");
__asm__("movq %r10, 40 (%rdi)");
__asm__("movq %r11, 48 (%rdi)");
__asm__("movq %r12, 56 (%rdi)");
__asm__("leaq 64(%rdi),%rdi");
__asm__("leaq 64(%rsi),%rsi");
__asm__("jnz 2b");
__asm__("movq (%rsp),%rbx");
__asm__("movq 1*8(%rsp),%r12");
__asm__("addq $2*8,%rsp");
}void main(void)
{
int i;
test_init();
printf ("%23s", "");
printf ("\t%s\t%s\t%s\n", "copy_page_org", "copy_page_new");
do_test(0, 0, 4096);
do_test(0, 0, 4096);
do_test(0, 0, 4096);
do_test(0, 0, 4096);
do_test(0, 0, 4096);
return ;
}Borislav Petkov
maintainerQuoted 6 lines · Ling
Quoted 6 lines
Quoted 5 lines
Quoted 1 line
So is that also true for AMD CPUs?
Although Bulldozer put 32byte instruction into decoupled 16byte entry buffers, it still decode 4 instructions per cycle, so 4 instructions will be fed into execution unit and
2 loads ,1 write will be issued per cycle.
Quoted 8 lines
Quoted 3 lines
I'd be very interested with what benchmarks are you seeing that perf improvement on Atom and who knows, maybe I could find time to run them on Bulldozer and see how your patch behaves there :-).M
I use another benchmark from gcc, there are many code, and extract one simple benchmark, you may use it to test (cc -o
copy_pagecopy_page.c), my initial result shows new copy page version is still better on bulldozer machine, because the machine is first release, please verify result. And CC to Ian.
Right, so benchmark shows around 20% speedup on Bulldozer but this is a microbenchmark and before pursue this further, we need to verify whether this brings any palpable speedup with a real benchmark, I don't know, kernbench, netbench, whatever. Even something as boring as kernel build. And probably check for perf regressions on the rest of the uarches.
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.Borislav Petkov
maintainerQuoted 6 lines · Borislav Petkov
Right, so benchmark shows around 20% speedup on Bulldozer but this is a microbenchmark and before pursue this further, we need to verify whether this brings any palpable speedup with a real benchmark, I don't know, kernbench, netbench, whatever. Even something as boring as kernel build. And probably check for perf regressions on the rest of the uarches.
Ok, so to summarize, on AMD we're using REP MOVSQ which is even faster than the unrolled version. I've added the REP MOVSQ version to the µbenchmark. It nicely validates that we're correctly setting X86_FEATURE_REP_GOOD on everything >= F10h and some K8s.
So, to answer Konrad's question: those patches don't concern AMD machines.
Thanks.
Signature · 2 lines
Regards/Gruss,
Boris.#include<stdio.h>
#include <stdlib.h>
typedef unsigned long long int hp_timing_t;
#define MAXSAMPLESTPT 1000 #define MAXCOPYSIZE (1024 * 1024) #define ORIG 0 #define NEW 1
static char* buf1 = NULL;
static char* buf2 = NULL;
static int repeat_one_test = 32;
hp_timing_t _dl_hp_timing_overhead;
# define HP_TIMING_NOW(Var) \
({ unsigned long long _hi, _lo; \
asm volatile ("rdtsc" : "=a" (_lo), "=d" (_hi)); \
(Var) = _hi << 32 | _lo; })#define HP_TIMING_DIFF(Diff, Start, End) (Diff) = ((End) - (Start))
#define HP_TIMING_TOTAL(total_time, start, end) \
do \
{ \
hp_timing_t tmptime; \
HP_TIMING_DIFF (tmptime, start + _dl_hp_timing_overhead, end); \
total_time += tmptime; \
} \while (0)
#define HP_TIMING_BEST(best_time, start, end) \
do \
{ \
hp_timing_t tmptime; \
HP_TIMING_DIFF (tmptime, start + _dl_hp_timing_overhead, end); \
if (best_time > tmptime) \
best_time = tmptime; \
} \while (0)
void copy_page_org(char *dst, char *src, int len); void copy_page_new(char *dst, char *src, int len); void copy_page_rep_movsq(char *dst, char *src, int len); void memcpy_c(char *dst, char *src, int len); void (*do_memcpy)(char *dst, char *src, int len);
static voiddo_one_test ( char *dst, char *src,
size_t len)
{
hp_timing_t start __attribute ((unused));
hp_timing_t stop __attribute ((unused));
hp_timing_t best_time = ~ (hp_timing_t) 0;
size_t i;
for (i = 0; i < repeat_one_test; ++i)
{
HP_TIMING_NOW (start);
do_memcpy ( dst, src, len);
HP_TIMING_NOW (stop);
HP_TIMING_BEST (best_time, start, stop);
}
printf ("\t\t%zd", (size_t) best_time);
}static voiddo_test (size_t align1, size_t align2, size_t len) {
char *s1, *s2;
s1 = (char *) (buf1 + align1); s2 = (char *) (buf2 + align2);
printf ("TPT: Len %4zd, alignment %2zd/%2zd:", len, align1, align2); do_memcpy = copy_page_org;
do_one_test (s2, s1, len);
do_memcpy = copy_page_new;
do_one_test (s2+ (1 << 16), s1 + (1 << 16), len); do_memcpy = copy_page_rep_movsq;
do_one_test(s2, s1, len);
putchar ('\n');
}static void test_init(void)
{
int i;
buf1 = valloc(MAXCOPYSIZE);
buf2 = valloc(MAXCOPYSIZE);
for (i = 0; i < MAXCOPYSIZE ; i = i + 64) {
buf1[i] = buf2[i] = i & 0xff;
}}
void copy_page_new(char *dst, char *src, int len)
{
__asm__("mov $(4096/64)-5, %ecx");
__asm__("1:");
__asm__("prefetcht0 5*64(%rsi)");
__asm__("decb %cl");
__asm__("movq 0x8*0(%rsi), %r10");
__asm__("movq 0x8*1(%rsi), %rax");
__asm__("movq 0x8*2(%rsi), %r8");
__asm__("movq 0x8*3(%rsi), %r9");
__asm__("movq %r10, 0x8*0(%rdi)");
__asm__("movq %rax, 0x8*1(%rdi)");
__asm__("movq %r8, 0x8*2(%rdi)");
__asm__("movq %r9, 0x8*3(%rdi)");
__asm__("movq 0x8*4(%rsi), %r10");
__asm__("movq 0x8*5(%rsi), %rax");
__asm__("movq 0x8*6(%rsi), %r8");
__asm__("movq 0x8*7(%rsi), %r9");
__asm__("leaq 64(%rsi), %rsi");
__asm__("movq %r10, 0x8*4(%rdi)");
__asm__("movq %rax, 0x8*5(%rdi)");
__asm__("movq %r8, 0x8*6(%rdi)");
__asm__("movq %r9, 0x8*7(%rdi)");
__asm__("leaq 64(%rdi), %rdi");
__asm__("jnz 1b");
__asm__("mov $5, %dl");
__asm__("2:");
__asm__("decb %dl");
__asm__("movq 0x8*0(%rsi), %r10");
__asm__("movq 0x8*1(%rsi), %rax");
__asm__("movq 0x8*2(%rsi), %r8");
__asm__("movq 0x8*3(%rsi), %r9");
__asm__("movq %r10, 0x8*0(%rdi)");
__asm__("movq %rax, 0x8*1(%rdi)");
__asm__("movq %r8, 0x8*2(%rdi)");
__asm__("movq %r9, 0x8*3(%rdi)");
__asm__("movq 0x8*4(%rsi), %r10");
__asm__("movq 0x8*5(%rsi), %rax");
__asm__("movq 0x8*6(%rsi), %r8");
__asm__("movq 0x8*7(%rsi), %r9");
__asm__("leaq 64(%rsi), %rsi");
__asm__("movq %r10, 0x8*4(%rdi)");
__asm__("movq %rax, 0x8*5(%rdi)");
__asm__("movq %r8, 0x8*6(%rdi)");
__asm__("movq %r9, 0x8*7(%rdi)");
__asm__("leaq 64(%rdi), %rdi");
__asm__("jnz 2b");}
void copy_page_org(char *dst, char *src, int len)
{
__asm__("subq $2*8,%rsp");
__asm__("movq %rbx,(%rsp)");
__asm__("movq %r12,1*8(%rsp)");
__asm__("movl $(4096/64)-5,%ecx");
__asm__(".p2align 4");
__asm__("1:");
__asm__("dec %rcx");
__asm__("movq (%rsi), %rax");
__asm__("movq 8 (%rsi), %rbx");
__asm__("movq 16 (%rsi), %rdx");
__asm__("movq 24 (%rsi), %r8");
__asm__("movq 32 (%rsi), %r9");
__asm__("movq 40 (%rsi), %r10");
__asm__("movq 48 (%rsi), %r11");
__asm__("movq 56 (%rsi), %r12");
__asm__("prefetcht0 5*64(%rsi)");
__asm__("movq %rax, (%rdi)");
__asm__("movq %rbx, 8 (%rdi)");
__asm__("movq %rdx, 16 (%rdi)");
__asm__("movq %r8, 24 (%rdi)");
__asm__("movq %r9, 32 (%rdi)");
__asm__("movq %r10, 40 (%rdi)");
__asm__("movq %r11, 48 (%rdi)");
__asm__("movq %r12, 56 (%rdi)");
__asm__("leaq 64 (%rsi), %rsi");
__asm__("leaq 64 (%rdi), %rdi");
__asm__("jnz 1b");
__asm__("movl $5,%ecx");
__asm__(".p2align 4");
__asm__("2:");
__asm__("decl %ecx");
__asm__("movq (%rsi), %rax");
__asm__("movq 8 (%rsi), %rbx");
__asm__("movq 16 (%rsi), %rdx");
__asm__("movq 24 (%rsi), %r8");
__asm__("movq 32 (%rsi), %r9");
__asm__("movq 40 (%rsi), %r10");
__asm__("movq 48 (%rsi), %r11");
__asm__("movq 56 (%rsi), %r12");
__asm__("movq %rax, (%rdi)");
__asm__("movq %rbx, 8 (%rdi)");
__asm__("movq %rdx, 16 (%rdi)");
__asm__("movq %r8, 24 (%rdi)");
__asm__("movq %r9, 32 (%rdi)");
__asm__("movq %r10, 40 (%rdi)");
__asm__("movq %r11, 48 (%rdi)");
__asm__("movq %r12, 56 (%rdi)");
__asm__("leaq 64(%rdi),%rdi");
__asm__("leaq 64(%rsi),%rsi");
__asm__("jnz 2b");
__asm__("movq (%rsp),%rbx");
__asm__("movq 1*8(%rsp),%r12");
__asm__("addq $2*8,%rsp");
}void copy_page_rep_movsq(char *dst, char *src, int len)
{
__asm__("movl $4096/8,%ecx");
__asm__("rep movsq");
}int main(void)
{
test_init();
printf ("%35s", "");
printf ("\t%s\t%s\t%s\n", "copy_page_org", "copy_page_new", "REP MOVSQ");
do_test(0, 0, 4096);
do_test(0, 0, 4096);
do_test(0, 0, 4096);
do_test(0, 0, 4096);
do_test(0, 0, 4096);
return 0;
}Thanks Boris!
So the patch is helpful and no impact for other/older machines, I will re-send new version according to comments.
Any further comments are appreciated!
Regards
Ling
Quoted 24 lines
-----Original Message-----
From: Borislav Petkov [mailto:[email protected]]
Sent: Sunday, October 14, 2012 6:58 PM
To: Ma, Ling
Cc: Konrad Rzeszutek Wilk; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; George Spelvin
Subject: Re: [PATCH RFC 2/2] [x86] Optimizecopy_pageby re-arranging instruction sequence and saving registerQuoted 4 lines · Borislav Petkov
Right, so benchmark shows around 20% speedup on Bulldozer but this is a microbenchmark and before pursue this further, we need to verify whether this brings any palpable speedup with a real benchmark, I don't know, kernbench, netbench, whatever. Even something as boring
as
Quoted 2 lines
kernel build. And probably check for perf regressions on the rest of the uarches.
Ok, so to summarize, on AMD we're using REP MOVSQ which is even faster than the unrolled version. I've added the REP MOVSQ version to the µbenchmark. It nicely validates that we're correctly setting
X86_FEATURE_REP_GOODon everything >= F10h and some K8s.
Quoted 8 lines
So, to answer Konrad's question: those patches don't concern AMD machines.
Thanks.
--
Regards/Gruss,
Boris.
���{.n�+�������+%�����ݶ��w��{.n�+����{��G�����{ay� ʇڙ�,j��f���h��������z_�(�階�ݢj"���m������G������?����&���~��iO��z��v�^�m���� ������?�I�
Just for everyone's information, here's the updated benchmark code on the same Phenom. The REP MOVSQ code is indeed much faster.
vendor_id : AuthenticAMD
cpu family : 16
model : 2
model name : AMD Phenom(tm) 9850 Quad-Core Processor
stepping : 3
microcode : 0x1000083
cpu MHz : 2500.210
cache size : 512 KB
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs hw_pstate npt lbrv svm_lock
bogomips : 5000.42
TLB size : 1024 4K pages
clflush size : 64cache_alignment : 64
copy_page_org copy_page_new REP MOVSQ
TPT: Len 4096, alignment 0/ 0: 708 758 611
TPT: Len 4096, alignment 0/ 0: 672 758 611
TPT: Len 4096, alignment 0/ 0: 708 758 611
TPT: Len 4096, alignment 0/ 0: 708 758 611
TPT: Len 4096, alignment 0/ 0: 708 758 611
copy_page_org copy_page_new REP MOVSQ
TPT: Len 4096, alignment 0/ 0: 694 759 611
TPT: Len 4096, alignment 0/ 0: 672 758 611
TPT: Len 4096, alignment 0/ 0: 708 758 611
TPT: Len 4096, alignment 0/ 0: 708 759 611
TPT: Len 4096, alignment 0/ 0: 708 757 611
copy_page_org copy_page_new REP MOVSQ
TPT: Len 4096, alignment 0/ 0: 697 758 611
TPT: Len 4096, alignment 0/ 0: 708 758 611
TPT: Len 4096, alignment 0/ 0: 708 758 611
TPT: Len 4096, alignment 0/ 0: 708 758 611
TPT: Len 4096, alignment 0/ 0: 708 757 611
copy_page_org copy_page_new REP MOVSQ
TPT: Len 4096, alignment 0/ 0: 703 758 612
TPT: Len 4096, alignment 0/ 0: 709 758 611
TPT: Len 4096, alignment 0/ 0: 709 757 611
TPT: Len 4096, alignment 0/ 0: 709 759 613
TPT: Len 4096, alignment 0/ 0: 709 759 611
copy_page_org copy_page_new REP MOVSQ
TPT: Len 4096, alignment 0/ 0: 669 758 613
TPT: Len 4096, alignment 0/ 0: 671 758 611
TPT: Len 4096, alignment 0/ 0: 708 758 611
TPT: Len 4096, alignment 0/ 0: 708 758 611
TPT: Len 4096, alignment 0/ 0: 708 758 613
copy_page_org copy_page_new REP MOVSQ
TPT: Len 4096, alignment 0/ 0: 679 758 612
TPT: Len 4096, alignment 0/ 0: 671 758 612
TPT: Len 4096, alignment 0/ 0: 684 759 612
TPT: Len 4096, alignment 0/ 0: 709 759 613
TPT: Len 4096, alignment 0/ 0: 709 759 611
copy_page_org copy_page_new REP MOVSQ
TPT: Len 4096, alignment 0/ 0: 682 758 612
TPT: Len 4096, alignment 0/ 0: 673 758 613
TPT: Len 4096, alignment 0/ 0: 704 759 613
TPT: Len 4096, alignment 0/ 0: 709 758 613
TPT: Len 4096, alignment 0/ 0: 709 758 611
copy_page_org copy_page_new REP MOVSQ
TPT: Len 4096, alignment 0/ 0: 669 759 611
TPT: Len 4096, alignment 0/ 0: 671 759 611
TPT: Len 4096, alignment 0/ 0: 709 759 613
TPT: Len 4096, alignment 0/ 0: 709 759 613
TPT: Len 4096, alignment 0/ 0: 708 759 613
copy_page_org copy_page_new REP MOVSQ
TPT: Len 4096, alignment 0/ 0: 668 759 612
TPT: Len 4096, alignment 0/ 0: 709 759 612
TPT: Len 4096, alignment 0/ 0: 709 759 612
TPT: Len 4096, alignment 0/ 0: 709 759 612
TPT: Len 4096, alignment 0/ 0: 709 759 612
copy_page_org copy_page_new REP MOVSQ
TPT: Len 4096, alignment 0/ 0: 694 758 611
TPT: Len 4096, alignment 0/ 0: 671 759 611
TPT: Len 4096, alignment 0/ 0: 708 759 611
TPT: Len 4096, alignment 0/ 0: 708 759 611
TPT: Len 4096, alignment 0/ 0: 708 759 613