Skip to contentSkip to conversation

performance improvement for memcpy_64.S by fast string.

RFC PATCH In review [email protected] 6 participants 33 messages Nov 6, 2009 – Nov 13, 2009 x86

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 performance improvement for memcpy_64.S by fast string.

From: Ma Ling <[email protected]>

Hi All

Intel Nehalem improves the performance of REP strings significantly over previous microarchitectures in several ways:

1. Startup overhead have been reduced in most cases.
2. Data transfer throughput are improved.
3. REP string can operate in "fast string" even if address is not aligned to 16bytes.

According to the experiment when copy size is big enough movsq almost can get 16bytes throughput per cycle, which approximate SSE instruction set. The patch intends to utilize the optimization when copy size is over 1024.

Experiment data speedup under Nehalem platform:

  Len    alignment   Speedup
 1024,    0/ 0:      1.04x
 2048,    0/ 0:      1.36x
 3072,    0/ 0:      1.51x
 4096,    0/ 0:      1.60x
 5120,    0/ 0:      1.70x
 6144,    0/ 0:      1.74x
 7168,    0/ 0:      1.77x
 8192,    0/ 0:      1.80x
 9216,    0/ 0:      1.82x
 10240,   0/ 0:      1.83x
 11264,   0/ 0:      1.85x
 12288,   0/ 0:      1.86x
 13312,   0/ 0:      1.92x
 14336,   0/ 0:      1.84x
 15360,   0/ 0:      1.74x

'perf stat --repeat 10 ./static_orig' command get data before patch:

Performance counter stats for './static_orig' (10 runs):

    2835.650105  task-clock-msecs         #      0.999 CPUs    ( +-   0.051% )
              3  context-switches         #      0.000 M/sec   ( +-   6.503% )
              0  CPU-migrations           #      0.000 M/sec   ( +-     nan% )
           4429  page-faults              #      0.002 M/sec   ( +-   0.003% )
     7941098692  cycles                   #   2800.451 M/sec   ( +-   0.051% )
    10848100323  instructions             #      1.366 IPC     ( +-   0.000% )
         322808  cache-references         #      0.114 M/sec   ( +-   1.467% )
         280716  cache-misses             #      0.099 M/sec   ( +-   0.618% )
    2.838006377  seconds time elapsed   ( +-   0.051% )

'perf stat --repeat 10 ./static_new' command get data after patch:

Performance counter stats for './static_new' (10 runs):

    7401.423466  task-clock-msecs         #      0.999 CPUs    ( +-   0.108% )
             10  context-switches         #      0.000 M/sec   ( +-   2.797% )
              0  CPU-migrations           #      0.000 M/sec   ( +-     nan% )
           4428  page-faults              #      0.001 M/sec   ( +-   0.003% )
    20727280183  cycles                   #   2800.445 M/sec   ( +-   0.107% )
     1472673654  instructions             #      0.071 IPC     ( +-   0.013% )
        1092221  cache-references         #      0.148 M/sec   ( +-  12.414% )
         290550  cache-misses             #      0.039 M/sec   ( +-   1.577% )
    7.407006046  seconds time elapsed   ( +-   0.108% )

Appreciate your comments.

Thanks
Ma Ling

---
 arch/x86/lib/memcpy_64.S |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)
arch/x86/lib/memcpy_64.S +17 -0
index ad5441e..2ea3561 100644
@@ -50,6 +50,12 @@ ENTRY(memcpy)
 	movl %edx, %ecx
 	shrl   $6, %ecx
 	jz .Lhandle_tail
+	/*
+	 * If length is more than 1024 we chose optimized MOVSQ,
+	 * which has more throughput.
+	 */
+	cmpl $0x400, %edx 
+	jae .Lmore_0x400
 
 	.p2align 4
 .Lloop_64:
@@ -119,6 +125,17 @@ ENTRY(memcpy)
 
 .Lend:
 	ret
+
+	.p2align 4
+.Lmore_0x400:
+	movq %rdi, %rax
+	movl %edx, %ecx
+	shrl $3, %ecx
+	andl $7, %edx
+	rep movsq
+	movl %edx, %ecx
+	rep movsb
+	ret
 	CFI_ENDPROC
 ENDPROC(memcpy)
 ENDPROC(__memcpy)
Signature · 1 line
1.6.2.5
#1 lore raw
Quoted 2 lines · [email protected] writes:

Intel Nehalem improves the performance of REP strings significantly over previous microarchitectures in several ways:

The problem is that it's not necessarily a win on older CPUs to do it this way.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#2 lore raw
Quoted 7 lines · Andi Kleen
Quoted 2 lines · [email protected] writes:

Intel Nehalem improves the performance of REP strings significantly over previous microarchitectures in several ways:

The problem is that it's not necessarily a win on older CPUs to do it this way.

I'm wondering, why are you writing such obtruse comments to Intel submitted patches? I know it and you know it too which older CPUs have a slow string implementation, and you know the rough order of magnitude and significance as well and you have ideas how to solve it all.

Instead you injected just the minimal amount of information into this thread to derail this patch you can see a problem with, but you didnt at all explain your full opinion openly and honestly and you certainly didnt give enough information to allow Ling Ma to act upon your opinion with maximum efficiency.

I.e. you are not being helpful at all here and you are obstructing Intel folks actively, making their workflow and progress as inefficient as you possibly can. Why are you doing that?

Ingo

#3 lore raw
In reply to [email protected]
Quoted 27 lines · On 11/06/2009 01:41 AM, [email protected] wrote:

Performance counter stats for './static_orig' (10 runs):

    2835.650105  task-clock-msecs         #      0.999 CPUs    ( +-   0.051% )
              3  context-switches         #      0.000 M/sec   ( +-   6.503% )
              0  CPU-migrations           #      0.000 M/sec   ( +-     nan% )
           4429  page-faults              #      0.002 M/sec   ( +-   0.003% )
     7941098692  cycles                   #   2800.451 M/sec   ( +-   0.051% )
    10848100323  instructions             #      1.366 IPC     ( +-   0.000% )
         322808  cache-references         #      0.114 M/sec   ( +-   1.467% )
         280716  cache-misses             #      0.099 M/sec   ( +-   0.618% )
    2.838006377  seconds time elapsed   ( +-   0.051% )

'perf stat --repeat 10 ./static_new' command get data after patch:

Performance counter stats for './static_new' (10 runs):

    7401.423466  task-clock-msecs         #      0.999 CPUs    ( +-   0.108% )
             10  context-switches         #      0.000 M/sec   ( +-   2.797% )
              0  CPU-migrations           #      0.000 M/sec   ( +-     nan% )
           4428  page-faults              #      0.001 M/sec   ( +-   0.003% )
    20727280183  cycles                   #   2800.445 M/sec   ( +-   0.107% )
     1472673654  instructions             #      0.071 IPC     ( +-   0.013% )
        1092221  cache-references         #      0.148 M/sec   ( +-  12.414% )
         290550  cache-misses             #      0.039 M/sec   ( +-   1.577% )
Quoted 2 lines
    7.407006046  seconds time elapsed   ( +-   0.108% )

I assume these are backwards? If so, it's a dramatic performance improvement.

Where did the 1024 byte threshold come from? It seems a bit high to me, and is at the very best a CPU-specific tuning factor.

Andi is of course correct that older CPUs might suffer (sadly enough), which is why we'd at the very least need some idea of what the performance impact on those older CPUs would look like -- at that point we can make a decision to just unconditionally do the rep movs or consider some system where we point at different implementations for different processors -- memcpy is probably one of the very few operations for which something like that would make sense.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#4 lore raw
Quoted 4 lines · H. Peter Anvin

Where did the 1024 byte threshold come from? It seems a bit high to me, and is at the very best a CPU-specific tuning factor.

Quoted 8 lines

Andi is of course correct that older CPUs might suffer (sadly enough), which is why we'd at the very least need some idea of what the performance impact on those older CPUs would look like -- at that point we can make a decision to just unconditionally do the rep movs or consider some system where we point at different implementations for different processors -- memcpy is probably one of the very few operations for which something like that would make sense.

To be expicit: Ling, would you be willing to run some benchmarks across processors to see how this performs on non-Nehalem CPUs?

-hpa

author
#5 lore raw

Hi All

Today we run our benchmark on Core2 and Sandy Bridge:

1. Retrieve result on Core2
Speedup on Core2

   Len        Alignement             Speedup
  1024,       0/ 0:                 0.95x 
  2048,       0/ 0:                 1.03x 
  3072,       0/ 0:                 1.02x 
  4096,       0/ 0:                 1.09x 
  5120,       0/ 0:                 1.13x 
  6144,       0/ 0:                 1.13x 
  7168,       0/ 0:                 1.14x 
  8192,       0/ 0:                 1.13x 
  9216,       0/ 0:                 1.14x 
  10240,      0/ 0:                 0.99x 
  11264,      0/ 0:                 1.14x 
  12288,      0/ 0:                 1.14x 
  13312,      0/ 0:                 1.10x 
  14336,      0/ 0:                 1.10x 
  15360,      0/ 0:                 1.13x
Application run through perf
For (i= 1024; i < 1024 * 16; i = i + 64)
	do_memcpy(0, 0, i);

Run application by 'perf stat --repeat 10 ./static_orig/new' Before the patch:
Performance counter stats for './static_orig' (10 runs):

    3323.041832  task-clock-msecs         #      0.998 CPUs  ( +-   0.016% )
             22  context-switches         #      0.000 M/sec ( +-  31.913% )
              0  CPU-migrations           #      0.000 M/sec ( +-     nan% )
           4428  page-faults              #      0.001 M/sec ( +-   0.003% )
     9921549804  cycles                   #   2985.683 M/sec ( +-   0.016% )
    10863809359  instructions             #      1.095 IPC   ( +-   0.000% )
      972283451  cache-references         #    292.588 M/sec ( +-   0.018% )
          17703  cache-misses             #      0.005 M/sec ( +-   4.304% )
    3.330714469  seconds time elapsed   ( +-   0.021% )
After the patch:
Performance counter stats for './static_new' (10 runs):
    3392.902871  task-clock-msecs         #      0.998 CPUs ( +-   0.226% )
             21  context-switches         #      0.000 M/sec ( +-  30.982% )
              0  CPU-migrations           #      0.000 M/sec ( +-     nan% )
           4428  page-faults              #      0.001 M/sec ( +-   0.003% )
    10130188030  cycles                   #   2985.699 M/sec ( +-   0.227% )
      391981414  instructions             #      0.039 IPC   ( +-   0.013% )
      874161826  cache-references         #    257.644 M/sec ( +-   3.034% )
          17628  cache-misses             #      0.005 M/sec ( +-   4.577% )
    3.400681174  seconds time elapsed   ( +-   0.219% )

2. Retrieve result on Sandy Bridge
Speedup on Sandy Bridge

  Len        Alignement            Speedup
  1024,       0/ 0:                1.08x 
  2048,       0/ 0:                1.42x 
  3072,       0/ 0:                1.51x 
  4096,       0/ 0:                1.63x 
  5120,       0/ 0:                1.67x 
  6144,       0/ 0:                1.72x 
  7168,       0/ 0:                1.75x 
  8192,       0/ 0:                1.77x 
  9216,       0/ 0:                1.80x 
  10240,      0/ 0:                1.80x 
  11264,      0/ 0:                1.82x 
  12288,      0/ 0:                1.85x 
  13312,      0/ 0:                1.85x 
  14336,      0/ 0:                1.88x 
  15360,      0/ 0:                1.88x 

Application run through perf
For (i= 1024; i < 1024 * 16; i = i + 64)

	do_memcpy(0, 0, i);

Run application by 'perf stat --repeat 10 ./static_orig/new' Before the patch:
Performance counter stats for './static_orig' (10 runs):

    3787.441240  task-clock-msecs         #      0.995 CPUs  ( +-   0.140% )
              8  context-switches         #      0.000 M/sec ( +-  22.602% )
              0  CPU-migrations           #      0.000 M/sec ( +-     nan% )
           4428  page-faults              #      0.001 M/sec ( +-   0.003% )
     6053487926  cycles                   #   1598.305 M/sec ( +-   0.140% )
    10861025194  instructions             #      1.794 IPC   ( +-   0.001% )
        2823963  cache-references         #      0.746 M/sec ( +-  69.345% )
         266000  cache-misses             #      0.070 M/sec ( +-   0.980% )
    3.805400837  seconds time elapsed   ( +-   0.139% )

After the patch:
Performance counter stats for './static_new' (10 runs):

    2879.424879  task-clock-msecs         #      0.995 CPUs  ( +-   0.076% )
             10  context-switches         #      0.000 M/sec ( +-  24.761% )
              0  CPU-migrations           #      0.000 M/sec ( +-     nan% )
           4428  page-faults              #      0.002 M/sec ( +-   0.003% )
     4602155158  cycles                   #   1598.290 M/sec ( +-   0.076% )
      386146993  instructions             #      0.084 IPC   ( +-   0.005% )
         520008  cache-references         #      0.181 M/sec ( +-   8.077% )
         267345  cache-misses             #      0.093 M/sec ( +-   0.792% )
    2.893813235  seconds time elapsed   ( +-   0.085% )

Thanks
Ling

Quoted 22 lines

-----Original Message-----
From: H. Peter Anvin [mailto:[email protected]]
Sent: 2009年11月7日 3:26
To: Ma, Ling
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC] [X86] performance improvement for memcpy_64.S by fast string.

Quoted 4 lines · H. Peter Anvin

Where did the 1024 byte threshold come from? It seems a bit high to me, and is at the very best a CPU-specific tuning factor.

Quoted 8 lines

Andi is of course correct that older CPUs might suffer (sadly enough), which is why we'd at the very least need some idea of what the performance impact on those older CPUs would look like -- at that point we can make a decision to just unconditionally do the rep movs or consider some system where we point at different implementations for different processors -- memcpy is probably one of the very few operations for which something like that would make sense.

Quoted 4 lines

To be expicit: Ling, would you be willing to run some benchmarks across processors to see how this performs on non-Nehalem CPUs?

-hpa

�翳簕.n�+壏煯壄�+%娝�遍荻w�簕.n�+壏{炳G珴�妠ay�蕠跈�,jf"穐殢飦�戧鐉_璁(殠娸"濟�m���G珴�⒏?櫒璀�&x忈秈O曟瑉窔v豝�m��� �鹅�⒏?朓�

#6 lore raw
Quoted 4 lines · Ling

Hi All

Today we run our benchmark on Core2 and Sandy Bridge:

Hi Ling,

Thanks for doing that. Do you also have access to any older CPUs? I suspect that the CPUs that Andi are worried about are older CPUs like P4, K8 or Pentium M/Core 1. (Andi: please do clarify if you have additional information.)

My personal opinion is that if we can show no significant slowdown on P4, K8, P-M/Core 1, Core 2, and Nehalem then we can simply use this code unconditionally. If one of them is radically worse than baseline, then we have to do something conditional, which is a lot more complicated.

[Ingo, Thomas: do you agree?]

Thanks,

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#7 lore raw
Quoted 13 lines · H. Peter Anvin
Quoted 4 lines · Ling

Hi All

Today we run our benchmark on Core2 and Sandy Bridge:

Hi Ling,

Thanks for doing that. Do you also have access to any older CPUs? I suspect that the CPUs that Andi are worried about are older CPUs like P4, K8 or Pentium M/Core 1. (Andi: please do clarify if you have additional information.)

Quoted 7 lines

My personal opinion is that if we can show no significant slowdown on P4, K8, P-M/Core 1, Core 2, and Nehalem then we can simply use this code unconditionally. If one of them is radically worse than baseline, then we have to do something conditional, which is a lot more complicated.

[Ingo, Thomas: do you agree?]

Yeah. IIRC the worst-case were the old P2's which had a really slow, microcode based string ops. (Some of them even had erratums in early prototypes although we can certainly ignore those as string ops get relied on quite frequently.)

IIRC the original PPro core came up with some nifty, hardwired string ops, but those had to be dumbed down and emulated in microcode due to SMP bugs - making it an inferior choice in the end.

But that should be ancient history and i'd suggest we ignore the P4 dead-end too, unless it's some really big slowdown (which i doubt). If anyone cares then some optional assembly implementations could be added back.

Ling, if you are interested, could you send a user-space test-app to this thread that everyone could just compile and run on various older boxes, to gather a performance profile of hand-coded versus string ops performance?

( And i think we can make a judgement based on cache-hot performance
alone - if then the strings ops will perform comparatively better in cache-cold scenarios, so the cache-hot numbers would be a conservative estimate. )

Ingo

author
#8 lore raw

Hi All
Please use the memcpy.c(cc -o memcpy memcpy.c -O2) to test more cases, if you have interest. In this program we did simple modification on memcpy_new function.

Thanks
Ling

Quoted 51 lines

-----Original Message-----
From: Ingo Molnar [mailto:[email protected]]
Sent: 2009年11月9日 16:09
To: H. Peter Anvin
Cc: Ma, Ling; Ingo Molnar; Thomas Gleixner; linux-kernel
Subject: Re: [PATCH RFC] [X86] performance improvement for memcpy_64.S by fast string.

Quoted 13 lines · H. Peter Anvin
Quoted 4 lines · Ling

Hi All

Today we run our benchmark on Core2 and Sandy Bridge:

Hi Ling,

Thanks for doing that. Do you also have access to any older CPUs? I suspect that the CPUs that Andi are worried about are older CPUs like P4, K8 or Pentium M/Core 1. (Andi: please do clarify if you have additional information.)

Quoted 7 lines

My personal opinion is that if we can show no significant slowdown on P4, K8, P-M/Core 1, Core 2, and Nehalem then we can simply use this code unconditionally. If one of them is radically worse than baseline, then we have to do something conditional, which is a lot more complicated.

[Ingo, Thomas: do you agree?]

Yeah. IIRC the worst-case were the old P2's which had a really slow, microcode based string ops. (Some of them even had erratums in early prototypes although we can certainly ignore those as string ops get relied on quite frequently.)

IIRC the original PPro core came up with some nifty, hardwired string ops, but those had to be dumbed down and emulated in microcode due to SMP bugs - making it an inferior choice in the end.

But that should be ancient history and i'd suggest we ignore the P4 dead-end too, unless it's some really big slowdown (which i doubt). If anyone cares then some optional assembly implementations could be added back.

Ling, if you are interested, could you send a user-space test-app to this thread that everyone could just compile and run on various older boxes, to gather a performance profile of hand-coded versus string ops performance?

Quoted 6 lines

( And i think we can make a judgement based on cache-hot performance
alone - if then the strings ops will perform comparatively better in cache-cold scenarios, so the cache-hot numbers would be a conservative estimate. )

Ingo

#include<stdio.h>
#include <stdlib.h>

typedef unsigned long long int hp_timing_t;

#define  MAXSAMPLESTPT        100000
#define  MAXCOPYSIZE          (1024 * 32)
#define  ORIG  0
#define  NEW   1

static char* buf1 = NULL;
static char* buf2 = NULL;

hp_timing_t _dl_hp_timing_overhead;

# define HP_TIMING_NOW(Var) \
  ({ unsigned long long _hi, _lo; \
     asm volatile ("rdtsc" : "=a" (_lo), "=d" (_hi)); \
     (Var) = _hi << 32 | _lo; })

#define HP_TIMING_DIFF(Diff, Start, End) (Diff) = ((End) - (Start))
#define HP_TIMING_TOTAL(total_time, start, end) \

  do									\
    {									\
      hp_timing_t tmptime;						\
      HP_TIMING_DIFF (tmptime, start + _dl_hp_timing_overhead, end);	\
	total_time += tmptime;						\
    }									\

while (0)

void memcpy_orig(char *dst, char *src, int len);
void memcpy_new(char *dst, char *src, int len);
void (*do_memcpy)(char *dst, char *src, int len);

static void
do_one_throughput ( char *dst, char *src,
size_t len)

{
      __asm__("cpuid" : : : "eax", "ebx", "ecx", "edx");
      size_t i;
      hp_timing_t start __attribute ((unused));
      hp_timing_t stop __attribute ((unused));
      hp_timing_t total_time =  (hp_timing_t) 0;

      __asm__("cpuid" : : : "eax", "ebx", "ecx", "edx");
      for (i = 0; i < MAXSAMPLESTPT; ++i)  {
          HP_TIMING_NOW (start);
		do_memcpy(buf1, buf2, len);
	  HP_TIMING_NOW (stop);
	  HP_TIMING_TOTAL (total_time, start, stop);
      }

      printf ("\t%zd", (size_t) total_time/MAXSAMPLESTPT);

}

static void
do_tpt_test (size_t align1, size_t align2, size_t len) {
size_t i, j;
char *s1, *s2;

  s1 = (char *) (buf1 + align1);
  s2 = (char *) (buf2 + align2);
   printf ("TPT: Len %4zd, alignment %2zd/%2zd:", len, align1, align2);
   do_memcpy = memcpy_orig;
   do_one_throughput (s2, s1, len);
   do_memcpy = memcpy_new;
   do_one_throughput (s2, s1, len);
    putchar ('\n');
}

static test_init(void)
{
int i;
buf1 = valloc(MAXCOPYSIZE);
buf2 = valloc(MAXCOPYSIZE);

  for (i = 0; i < MAXCOPYSIZE ; i = i + 64) {
        buf1[i] = buf2[i] = i & 0xff;
  }

}

void memcpy_new(char *dst, char *src, int len)
{

	__asm__("movq %rdi, %rax");
	__asm__("movl %edx, %ecx");
	__asm__("shrl   $6, %ecx");
	__asm__("jz 2f");

	__asm__("cmp $0x400, %rdx");
	__asm__("jae 8f");

	__asm__("1:");
	__asm__("decl %ecx");

	__asm__("movq 0*8(%rsi), %r11");
	__asm__("movq 1*8(%rdi), %r8");
	__asm__("movq %r11,	0*8(%rdi)");
	__asm__("movq %r8,	1*8(%rdi)");

	__asm__("movq 2*8(%rsi), %r9");
	__asm__("movq 3*8(%rdi), %r10");
	__asm__("movq %r9,	2*8(%rdi)");
	__asm__("movq %r10,	3*8(%rdi)");

	__asm__("movq 4*8(%rsi), %r11");
	__asm__("movq 5*8(%rdi), %r8");
	__asm__("movq %r11,	4*8(%rdi)");
	__asm__("movq %r8,	5*8(%rdi)");

	__asm__("movq 6*8(%rsi), %r9");
	__asm__("movq 7*8(%rdi), %r10");
	__asm__("movq %r9,	6*8(%rdi)");
	__asm__("movq %r10,	7*8(%rdi)");

	__asm__("leaq 64(%rsi), %rsi");
	__asm__("leaq 64(%rdi), %rdi");

	__asm__("jnz  1b");

	__asm__("2:");
	__asm__("movl %edx, %ecx");
	__asm__("andl $63, %ecx");
	__asm__("shl   $3, %ecx");
	__asm__("jz 5f");

	__asm__("3:");
	__asm__("cmp %edi, %esi");
	__asm__("mov $8, %r9");
	__asm__("jl 4f");
	__asm__("neg %r9");

	__asm__("4:");
	__asm__("decl %ecx");
	__asm__("movq (%rsi),	%r8");
	__asm__("movq %r8,	(%rdi)");
	__asm__("leaq 8(%rdi),	%rdi");
	__asm__("leaq 8(%rsi),	%rsi");
	__asm__("jnz 3b");

	__asm__("5:");
	__asm__("movl %edx,	%ecx");
	__asm__("andl $7,	%ecx");
	__asm__("jz 7f");

	__asm__("6:");
	__asm__("movb (%rsi),	%r8b");
	__asm__("movb %r8b, (%rdi)");
	__asm__("incq %rdi");
	__asm__("incq %rsi");
	__asm__("decl %ecx");
	__asm__("jnz 6b");

	__asm__("7:");
	__asm__("retq");

	__asm__("8:");
	__asm__("movl %edx, %ecx");
	__asm__ ("shr $3, %ecx");
	__asm__ ("andl $7, %edx");
	__asm__("rep movsq ");
	__asm__ ("jz 9f");
	__asm__("movl %edx, %ecx");
	__asm__("rep movsb");

	__asm__("9:");
}
void memcpy_orig(char *dst, char *src, int len)
{
	__asm__("movq %rdi, %rax");
	__asm__("movl %edx, %ecx");
	__asm__("shrl   $6, %ecx");
	__asm__("jz 2f");

	__asm__("mov $0x80, %r8d  ");  /*aligned case for loop 1 */

	__asm__("1:");
	__asm__("decl %ecx");

	__asm__("movq 0*8(%rsi), %r11");
	__asm__("movq 1*8(%rdi), %r8");
	__asm__("movq %r11,	0*8(%rdi)");
	__asm__("movq %r8,	1*8(%rdi)");

	__asm__("movq 2*8(%rsi), %r9");
	__asm__("movq 3*8(%rdi), %r10");
	__asm__("movq %r9,	2*8(%rdi)");
	__asm__("movq %r10,	3*8(%rdi)");

	__asm__("movq 4*8(%rsi), %r11");
	__asm__("movq 5*8(%rdi), %r8");
	__asm__("movq %r11,	4*8(%rdi)");
	__asm__("movq %r8,	5*8(%rdi)");

	__asm__("movq 6*8(%rsi), %r9");
	__asm__("movq 7*8(%rdi), %r10");
	__asm__("movq %r9,	6*8(%rdi)");
	__asm__("movq %r10,	7*8(%rdi)");

	__asm__("leaq 64(%rsi), %rsi");
	__asm__("leaq 64(%rdi), %rdi");

	__asm__("jnz  1b");

	__asm__("2:");
	__asm__("movl %edx, %ecx");
	__asm__("andl $63, %ecx");
	__asm__("shl   $3, %ecx");
	__asm__("jz 5f");

	__asm__("3:");
	__asm__("cmp %edi, %esi");
	__asm__("mov $8, %r9");
	__asm__("jl 4f");
	__asm__("neg %r9");

	__asm__("4:");
	__asm__("decl %ecx");
	__asm__("movq (%rsi),	%r8");
	__asm__("movq %r8,	(%rdi)");
	__asm__("leaq 8(%rdi),	%rdi");
	__asm__("leaq 8(%rsi),	%rsi");
	__asm__("jnz 3b");

	__asm__("5:");
	__asm__("movl %edx,	%ecx");
	__asm__("andl $7,	%ecx");
	__asm__("jz 7f");

	__asm__("6:");
	__asm__("movb (%rsi),	%r8b");
	__asm__("movb %r8b, (%rdi)");
	__asm__("incq %rdi");
	__asm__("incq %rsi");
	__asm__("decl %ecx");
	__asm__("jnz 6b");

	__asm__("7:");
	__asm__("retq");
}

void main(void)
{
int i;
test_init();
printf ("%23s", "");
printf ("\t%s\t%s\n", "memcpy_orig", "memcpy_new");

  for (i = 1024; i < 1024 * 16; i = i + 1024)
     do_tpt_test(8, 0, i);

}

#9 lore raw
Quoted 4 lines · Ma, Ling

Hi All
Please use the memcpy.c(cc -o memcpy memcpy.c -O2) to test more cases, if you have interest. In this program we did simple modification on memcpy_new function.

FYI:

earth4:~/s> cc -o memcpy memcpy.c -O2
memcpy.c: In function 'do_one_throughput':
memcpy.c:45: error: impossible register constraint in 'asm' memcpy.c:53: error: impossible register constraint in 'asm' memcpy.c:47: error: impossible register constraint in 'asm' memcpy.c:53: error: impossible register constraint in 'asm'

Ingo

author
#10 lore raw

Hi Ingo

This program is for 64bit version, so please use 'cc -o memcpy memcpy.c -O2 -m64'

Thanks
Ling

Quoted 18 lines

-----Original Message-----
From: Ingo Molnar [mailto:[email protected]]
Sent: 2009年11月11日 15:19
To: Ma, Ling
Cc: H. Peter Anvin; Ingo Molnar; Thomas Gleixner; linux-kernel
Subject: Re: [PATCH RFC] [X86] performance improvement for memcpy_64.S by fast string.

Quoted 4 lines · Ma, Ling

Hi All
Please use the memcpy.c(cc -o memcpy memcpy.c -O2) to test more cases, if you have interest. In this program we did simple modification on memcpy_new function.

FYI:

Quoted 8 lines

earth4:~/s> cc -o memcpy memcpy.c -O2
memcpy.c: In function 'do_one_throughput':
memcpy.c:45: error: impossible register constraint in 'asm' memcpy.c:53: error: impossible register constraint in 'asm' memcpy.c:47: error: impossible register constraint in 'asm' memcpy.c:53: error: impossible register constraint in 'asm'

Ingo

�翳簕.n�+壏煯壄�+%娝�遍荻w�簕.n�+壏{炳G珴�妠ay�蕠跈�,jf"穐殢飦�戧鐉_璁(殠娸"濟�m���G珴�⒏?櫒璀�&x忈秈O曟瑉窔v豝�m��� �鹅�⒏?朓�

#11 lore raw
Quoted 4 lines · Ling

Hi Ingo

This program is for 64bit version, so please use 'cc -o memcpy memcpy.c -O2 -m64'

I did some measurements with this program; I added power-of-two measurements from 1-512 bytes, plus some different alignments, and found some very interesting results:

Nehalem:
memcpy_new is a win for 1024+ bytes, but *also* a win for 2-32 bytes, where the old code apparently performs appallingly bad.

memcpy_new loses in the 64-512 byte range, so the 1024 threshold is probably justified.

Core2:
memcpy_new is a win for <= 512 bytes, but a lose for larger copies (possibly a win again for 16K+ copies, but those are very rare in the Linux kernel.) Surprise...

However, the difference is very small.

However, I had overlooked something much more fundamental about your patch. On Nehalem, at least *it will never get executed* (except during very early startup), because we replace the memcpy code with a jmp to memcpy_c on any CPU which has X86_FEATURE_REP_GOOD, which includes Nehalem.

So the patch is a no-op on Nehalem, and any other modern CPU.

Am I guessing that the perf numbers you posted originally were all from your user space test program?

-hpa

author
#12 lore raw
Quoted 31 lines

-----Original Message-----
From: H. Peter Anvin [mailto:[email protected]]
Sent: 2009年11月12日 7:21
To: Ma, Ling
Cc: Ingo Molnar; Ingo Molnar; Thomas Gleixner; linux-kernel
Subject: Re: [PATCH RFC] [X86] performance improvement for memcpy_64.S by fast string.

Quoted 3 lines · Ling

Hi Ingo

This program is for 64bit version, so please use 'cc -o memcpy memcpy.c -O2

-m64'

Quoted 1 line

I did some measurements with this program; I added power-of-two measurements from 1-512 bytes, plus some different alignments, and found some very interesting results:

Nehalem:
memcpy_new is a win for 1024+ bytes, but *also* a win for 2-32 bytes, where the old code apparently performs appallingly bad.

memcpy_new loses in the 64-512 byte range, so the 1024 threshold is probably justified.

Core2:
memcpy_new is a win for <= 512 bytes, but a lose for larger copies (possibly a win again for 16K+ copies, but those are very rare in the Linux kernel.) Surprise...

Quoted 8 lines

However, the difference is very small.

However, I had overlooked something much more fundamental about your patch. On Nehalem, at least *it will never get executed* (except during very early startup), because we replace the memcpy code with a jmp to memcpy_c on any CPU which has X86_FEATURE_REP_GOOD, which includes Nehalem.

So the patch is a no-op on Nehalem, and any other modern CPU.

[Ma Ling]
It is good for modern CPU, our original intention is also to introduce movsq for Nehalem, above method is more smart.

Quoted 2 lines

Am I guessing that the perf numbers you posted originally were all from your user space test program?

[Ma Ling]
Yes, they are all from this program, and I'm confused about measurement values will be different for only one case after multiple tests. (3 times at least on my core2 platform).

Thanks
Ling �翳簕.n�+壏煯壄�+%娝�遍荻w�簕.n�+壏{炳G珴�妠ay�蕠跈�,jf"穐殢飦�戧鐉_璁(殠娸"濟�m���G珴�⒏?櫒璀�&x忈秈O曟瑉窔v豝�m��� �鹅�⒏?朓�

#13 lore raw
In reply to Ma, Ling
Quoted 7 lines · Ling

Hi All
Please use the memcpy.c(cc -o memcpy memcpy.c -O2) to test more cases, if you have interest. In this program we did simple modification on memcpy_new function.

Thanks
Ling

Just my 0.2$ :)

-- Cyrill
---
memcpy_orig memcpy_new
TPT: Len 1024, alignment 8/ 0: 490 570
TPT: Len 2048, alignment 8/ 0: 826 329
TPT: Len 3072, alignment 8/ 0: 441 464
TPT: Len 4096, alignment 8/ 0: 579 596
TPT: Len 5120, alignment 8/ 0: 723 729
TPT: Len 6144, alignment 8/ 0: 859 861
TPT: Len 7168, alignment 8/ 0: 996 994
TPT: Len 8192, alignment 8/ 0: 1165 1127
TPT: Len 9216, alignment 8/ 0: 1273 1260
TPT: Len 10240, alignment 8/ 0: 1402 1395
TPT: Len 11264, alignment 8/ 0: 1543 1525
TPT: Len 12288, alignment 8/ 0: 1682 1659
TPT: Len 13312, alignment 8/ 0: 1869 1815
TPT: Len 14336, alignment 8/ 0: 1982 1951
TPT: Len 15360, alignment 8/ 0: 2185 2110
---

I've run this test a few times and results almost the same, with alignment 1024, 3072, 4096, 5120, 6144, new version a bit slowly.

---
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz
stepping : 6
cpu MHz : 800.000
cache size : 3072 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm ida tpr_shadow vnmi flexpriority
bogomips : 4189.60
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz
stepping : 6
cpu MHz : 800.000
cache size : 3072 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 lahf_lm ida tpr_shadow vnmi flexpriority
bogomips : 4189.46
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

#14 lore raw
Quoted 17 lines · Cyrill Gorcunov

memcpy_orig memcpy_new
TPT: Len 1024, alignment 8/ 0: 490 570
TPT: Len 2048, alignment 8/ 0: 826 329
TPT: Len 3072, alignment 8/ 0: 441 464
TPT: Len 4096, alignment 8/ 0: 579 596
TPT: Len 5120, alignment 8/ 0: 723 729
TPT: Len 6144, alignment 8/ 0: 859 861
TPT: Len 7168, alignment 8/ 0: 996 994
TPT: Len 8192, alignment 8/ 0: 1165 1127
TPT: Len 9216, alignment 8/ 0: 1273 1260
TPT: Len 10240, alignment 8/ 0: 1402 1395
TPT: Len 11264, alignment 8/ 0: 1543 1525
TPT: Len 12288, alignment 8/ 0: 1682 1659
TPT: Len 13312, alignment 8/ 0: 1869 1815
TPT: Len 14336, alignment 8/ 0: 1982 1951
TPT: Len 15360, alignment 8/ 0: 2185 2110

Quoted 3 lines

I've run this test a few times and results almost the same, with alignment 1024, 3072, 4096, 5120, 6144, new version a bit slowly.

Was the result for 2048 consistent (it seems odd in the extreme)... the discrepancy between this result and Ling's results bothers me; perhaps the right answer is to leave the current code for Core2 and use new code (with a lower than 1024 threshold?) for NHM and K8?

-hpa

#15 lore raw
Quoted 22 lines · H. Peter Anvin
Quoted 17 lines · Cyrill Gorcunov

                                              memcpy_orig     memcpy_new
TPT: Len 1024, alignment  8/ 0:               490             570
TPT: Len 2048, alignment  8/ 0:               826             329
TPT: Len 3072, alignment  8/ 0:               441             464
TPT: Len 4096, alignment  8/ 0:               579             596
TPT: Len 5120, alignment  8/ 0:               723             729
TPT: Len 6144, alignment  8/ 0:               859             861
TPT: Len 7168, alignment  8/ 0:               996             994
TPT: Len 8192, alignment  8/ 0:               1165            1127
TPT: Len 9216, alignment  8/ 0:               1273            1260
TPT: Len 10240, alignment  8/ 0:      1402            1395
TPT: Len 11264, alignment  8/ 0:      1543            1525
TPT: Len 12288, alignment  8/ 0:      1682            1659
TPT: Len 13312, alignment  8/ 0:      1869            1815
TPT: Len 14336, alignment  8/ 0:      1982            1951
TPT: Len 15360, alignment  8/ 0:      2185            2110

Quoted 3 lines

I've run this test a few times and results almost the same, with alignment 1024, 3072, 4096, 5120, 6144, new version a bit slowly.

Quoted 7 lines

Was the result for 2048 consistent (it seems odd in the extreme)... the discrepancy between this result and Ling's results bothers me; perhaps the right answer is to leave the current code for Core2 and use new code (with a lower than 1024 threshold?) for NHM and K8?

       -hpa

Hi Peter,

no, results for 2048 is not repeatable (that is why I didn't mention this number in a former report).

Test1:
TPT: Len 2048, alignment 8/ 0: 826 329
Test2:
TPT: Len 2048, alignment 8/ 0: 359 329
Test3:
TPT: Len 2048, alignment 8/ 0: 306 331
Test4:
TPT: Len 2048, alignment 8/ 0: 415 329

I guess this was due to cpu frequency change from 800 to 2.1Ghz since I did tests manually
not using any kind of bash cycle to run the test program.

author
#16 lore raw

Hi All
The attachment is latest memcpy.c, please update by "cc -o memcpy memcpy.c -O2 -m64".

Thanks
Ling

Quoted 54 lines

-----Original Message-----
From: Cyrill Gorcunov [mailto:[email protected]]
Sent: 2009年11月12日 12:28
To: H. Peter Anvin
Cc: Ma, Ling; Ingo Molnar; Ingo Molnar; Thomas Gleixner; linux-kernel
Subject: Re: [PATCH RFC] [X86] performance improvement for memcpy_64.S by fast string.

Quoted 2 lines · H. Peter Anvin
Quoted 1 line · Cyrill Gorcunov

                                              memcpy_orig     memc

py_new

Quoted 20 lines
Quoted 16 lines

TPT: Len 1024, alignment  8/ 0:               490             570
TPT: Len 2048, alignment  8/ 0:               826             329
TPT: Len 3072, alignment  8/ 0:               441             464
TPT: Len 4096, alignment  8/ 0:               579             596
TPT: Len 5120, alignment  8/ 0:               723             729
TPT: Len 6144, alignment  8/ 0:               859             861
TPT: Len 7168, alignment  8/ 0:               996             994
TPT: Len 8192, alignment  8/ 0:               1165            1127
TPT: Len 9216, alignment  8/ 0:               1273            1260
TPT: Len 10240, alignment  8/ 0:      1402            1395
TPT: Len 11264, alignment  8/ 0:      1543            1525
TPT: Len 12288, alignment  8/ 0:      1682            1659
TPT: Len 13312, alignment  8/ 0:      1869            1815
TPT: Len 14336, alignment  8/ 0:      1982            1951
TPT: Len 15360, alignment  8/ 0:      2185            2110

Quoted 3 lines

I've run this test a few times and results almost the same, with alignment 1024, 3072, 4096, 5120, 6144, new version a bit slowly.

Quoted 7 lines

Was the result for 2048 consistent (it seems odd in the extreme)... the discrepancy between this result and Ling's results bothers me; perhaps the right answer is to leave the current code for Core2 and use new code (with a lower than 1024 threshold?) for NHM and K8?

       -hpa

Hi Peter,

no, results for 2048 is not repeatable (that is why I didn't mention this number in a former report).

Test1:
TPT: Len 2048, alignment 8/ 0: 826 329
Test2:
TPT: Len 2048, alignment 8/ 0: 359 329
Test3:
TPT: Len 2048, alignment 8/ 0: 306 331
Test4:
TPT: Len 2048, alignment 8/ 0: 415 329

Quoted 3 lines

I guess this was due to cpu frequency change from 800 to 2.1Ghz since I did tests manually
not using any kind of bash cycle to run the test program.

#include<stdio.h>
#include <stdlib.h>

typedef unsigned long long int hp_timing_t;

#define  MAXSAMPLESTPT        100000
#define  MAXCOPYSIZE          (1024 * 32)
#define  ORIG  0
#define  NEW   1

static char* buf1 = NULL;
static char* buf2 = NULL;

hp_timing_t _dl_hp_timing_overhead;

# define HP_TIMING_NOW(Var) \
  ({ unsigned long long _hi, _lo; \
     asm volatile ("rdtsc" : "=a" (_lo), "=d" (_hi)); \
     (Var) = _hi << 32 | _lo; })

#define HP_TIMING_DIFF(Diff, Start, End) (Diff) = ((End) - (Start))
#define HP_TIMING_TOTAL(total_time, start, end) \

  do									\
    {									\
      hp_timing_t tmptime;						\
      HP_TIMING_DIFF (tmptime, start + _dl_hp_timing_overhead, end);	\
	total_time += tmptime;						\
    }									\

while (0)

void memcpy_orig(char *dst, char *src, int len); void memcpy_new(char *dst, char *src, int len); void memcpy_c(char *dst, char *src, int len); void (*do_memcpy)(char *dst, char *src, int len);

static void
do_one_throughput ( char *dst, char *src,
size_t len)

{
      __asm__("cpuid" : : : "eax", "ebx", "ecx", "edx");
      size_t i;
      hp_timing_t start __attribute ((unused));
      hp_timing_t stop __attribute ((unused));
      hp_timing_t total_time =  (hp_timing_t) 0;

      __asm__("cpuid" : : : "eax", "ebx", "ecx", "edx");
      for (i = 0; i < MAXSAMPLESTPT; ++i)  {
          HP_TIMING_NOW (start);
		do_memcpy(buf1, buf2, len);
	  HP_TIMING_NOW (stop);
	  HP_TIMING_TOTAL (total_time, start, stop);
      }

      printf ("\t%zd", (size_t) total_time/MAXSAMPLESTPT);

}

static void
do_tpt_test (size_t align1, size_t align2, size_t len) {
size_t i, j;
char *s1, *s2;

  s1 = (char *) (buf1 + align1);
  s2 = (char *) (buf2 + align2);
   printf ("TPT: Len %4zd, alignment %2zd/%2zd:", len, align1, align2);
   do_memcpy = memcpy_orig;
   do_one_throughput (s2, s1, len);
   do_memcpy = memcpy_new;
   do_one_throughput (s2, s1, len);
    putchar ('\n');
}

static test_init(void)
{
int i;
buf1 = valloc(MAXCOPYSIZE);
buf2 = valloc(MAXCOPYSIZE);

  for (i = 0; i < MAXCOPYSIZE ; i = i + 64) {
        buf1[i] = buf2[i] = i & 0xff;
  }

}

void memcpy_new(char *dst, char *src, int len)

{
	__asm__("movq %rdi, %rax");
	__asm__("movl %edx, %ecx");
	__asm__("shrl   $6, %ecx");
	__asm__("jz 2f");

	__asm__("cmp $0x400, %edx");
	__asm__("jae 7f");

	__asm__("1:");
	__asm__("decl %ecx");

	__asm__("movq 0*8(%rsi), %r11");
	__asm__("movq 1*8(%rdi), %r8");
	__asm__("movq %r11,	0*8(%rdi)");
	__asm__("movq %r8,	1*8(%rdi)");

	__asm__("movq 2*8(%rsi), %r9");
	__asm__("movq 3*8(%rdi), %r10");
	__asm__("movq %r9,	2*8(%rdi)");
	__asm__("movq %r10,	3*8(%rdi)");

	__asm__("movq 4*8(%rsi), %r11");
	__asm__("movq 5*8(%rdi), %r8");
	__asm__("movq %r11,	4*8(%rdi)");
	__asm__("movq %r8,	5*8(%rdi)");

	__asm__("movq 6*8(%rsi), %r9");
	__asm__("movq 7*8(%rdi), %r10");
	__asm__("movq %r9,	6*8(%rdi)");
	__asm__("movq %r10,	7*8(%rdi)");

	__asm__("leaq 64(%rsi), %rsi");
	__asm__("leaq 64(%rdi), %rdi");

	__asm__("jnz  1b");

	__asm__("2:");
	__asm__("movl %edx, %ecx");
	__asm__("andl $63, %ecx");
	__asm__("shl   $3, %ecx");
	__asm__("jz 4f");


	__asm__("3:");
	__asm__("decl %ecx");
	__asm__("movq (%rsi),	%r8");
	__asm__("movq %r8,	(%rdi)");
	__asm__("leaq 8(%rdi),	%rdi");
	__asm__("leaq 8(%rsi),	%rsi");
	__asm__("jnz 3b");

	__asm__("4:");
	__asm__("movl %edx,	%ecx");
	__asm__("andl $7,	%ecx");
	__asm__("jz 6f");

	__asm__("5:");
	__asm__("movb (%rsi),	%r8b");
	__asm__("movb %r8b, (%rdi)");
	__asm__("incq %rdi");
	__asm__("incq %rsi");
	__asm__("decl %ecx");
	__asm__("jnz 5b");

	__asm__("6:");
	__asm__("retq");

	__asm__("7:");
	__asm__("movl %edx, %ecx");
	__asm__ ("shr $3, %ecx");
	__asm__ ("andl $7, %edx");
	__asm__("rep movsq ");
	__asm__ ("jz 8f");
	__asm__("movl %edx, %ecx");
	__asm__("rep movsb");

	__asm__("8:");
}
void memcpy_orig(char *dst, char *src, int len)
{
	__asm__("movq %rdi, %rax");
	__asm__("movl %edx, %ecx");
	__asm__("shrl   $6, %ecx");
	__asm__("jz 2f");

	__asm__("mov $0x80, %r8d  ");  /*aligned case for loop 1 */

	__asm__("1:");
	__asm__("decl %ecx");

	__asm__("movq 0*8(%rsi), %r11");
	__asm__("movq 1*8(%rdi), %r8");
	__asm__("movq %r11,	0*8(%rdi)");
	__asm__("movq %r8,	1*8(%rdi)");

	__asm__("movq 2*8(%rsi), %r9");
	__asm__("movq 3*8(%rdi), %r10");
	__asm__("movq %r9,	2*8(%rdi)");
	__asm__("movq %r10,	3*8(%rdi)");

	__asm__("movq 4*8(%rsi), %r11");
	__asm__("movq 5*8(%rdi), %r8");
	__asm__("movq %r11,	4*8(%rdi)");
	__asm__("movq %r8,	5*8(%rdi)");

	__asm__("movq 6*8(%rsi), %r9");
	__asm__("movq 7*8(%rdi), %r10");
	__asm__("movq %r9,	6*8(%rdi)");
	__asm__("movq %r10,	7*8(%rdi)");

	__asm__("leaq 64(%rsi), %rsi");
	__asm__("leaq 64(%rdi), %rdi");

	__asm__("jnz  1b");

	__asm__("2:");
	__asm__("movl %edx, %ecx");
	__asm__("andl $63, %ecx");
	__asm__("shl   $3, %ecx");
	__asm__("jz 4f");


	__asm__("3:");
	__asm__("decl %ecx");
	__asm__("movq (%rsi),	%r8");
	__asm__("movq %r8,	(%rdi)");
	__asm__("leaq 8(%rdi),	%rdi");
	__asm__("leaq 8(%rsi),	%rsi");
	__asm__("jnz 3b");

	__asm__("4:");
	__asm__("movl %edx,	%ecx");
	__asm__("andl $7,	%ecx");
	__asm__("jz 6f");

	__asm__("5:");
	__asm__("movb (%rsi),	%r8b");
	__asm__("movb %r8b, (%rdi)");
	__asm__("incq %rdi");
	__asm__("incq %rsi");
	__asm__("decl %ecx");
	__asm__("jnz 5b");

	__asm__("6:");
}

void main(void)
{
int i;
test_init();
printf ("%23s", "");
printf ("\t%s\t%s\t%s\n", "memcpy_orig", "memcpy_new");

  for (i = 1024; i < 1024 * 16; i = i+ 1024)
     do_tpt_test(0, 0, i);

}

#17 lore raw
Quoted 3 lines · Ling

Hi All
The attachment is latest memcpy.c, please update by "cc -o memcpy memcpy.c -O2 -m64".

OK... given that there seems to be no point since the actual code we're talking about modifying doesn't ever actually get executed on the real kernel, we can just drop this, right?

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
author
#18 lore raw

Hi H. Peter Anvin

After running the test program in my attachment-memcpy.c on Nehalem platform, when copy size is less 1024 memcopy_c function has very big regression compared with original memcopy function. I think we have to combine original memcopy and memcpy_c for Nehalem and other modern CPUS, so memcpy_new is on the right track.

Thanks
Ling
(

Quoted 17 lines

-----Original Message-----
From: H. Peter Anvin [mailto:[email protected]]
Sent: 2009年11月12日 13:27
To: Ma, Ling
Cc: Cyrill Gorcunov; Ingo Molnar; Ingo Molnar; Thomas Gleixner; linux-kernel
Subject: Re: [PATCH RFC] [X86] performance improvement for memcpy_64.S by fast string.

Quoted 3 lines · Ling

Hi All
The attachment is latest memcpy.c, please update by "cc -o memcpy memcpy.c -O2 -m64".

OK... given that there seems to be no point since the actual code we're talking about modifying doesn't ever actually get executed on the real kernel, we can just drop this, right?

Quoted 5 lines

-hpa

--
H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf.

#include<stdio.h>
#include <stdlib.h>

typedef unsigned long long int hp_timing_t;

#define  MAXSAMPLESTPT        1000
#define  MAXCOPYSIZE          (1024 * 32)
#define  ORIG  0
#define  NEW   1

static char* buf1 = NULL;
static char* buf2 = NULL;

hp_timing_t _dl_hp_timing_overhead;

# define HP_TIMING_NOW(Var) \
  ({ unsigned long long _hi, _lo; \
     asm volatile ("rdtsc" : "=a" (_lo), "=d" (_hi)); \
     (Var) = _hi << 32 | _lo; })

#define HP_TIMING_DIFF(Diff, Start, End) (Diff) = ((End) - (Start))
#define HP_TIMING_TOTAL(total_time, start, end) \

  do									\
    {									\
      hp_timing_t tmptime;						\
      HP_TIMING_DIFF (tmptime, start + _dl_hp_timing_overhead, end);	\
	total_time += tmptime;						\
    }									\

while (0)

#define HP_TIMING_BEST(best_time, start, end) \

  do									\
    {									\
      hp_timing_t tmptime;						\
      HP_TIMING_DIFF (tmptime, start + _dl_hp_timing_overhead, end);	\
      if (best_time > tmptime)						\
	best_time = tmptime;						\
    }									\

while (0)

void memcpy_orig(char *dst, char *src, int len);
void memcpy_new(char *dst, char *src, int len);
void memcpy_c(char *dst, char *src, int len);
void (*do_memcpy)(char *dst, char *src, int len);

static void
do_one_throughput ( char *dst, char *src,
size_t len)
{

     __asm__("cpuid" : : : "eax", "ebx", "ecx", "edx");
      hp_timing_t start __attribute ((unused));
      hp_timing_t stop __attribute ((unused));
      hp_timing_t best_time = ~ (hp_timing_t) 0;
      size_t i;

      __asm__("cpuid" : : : "eax", "ebx", "ecx", "edx");
      HP_TIMING_NOW (start);
      for (i = 0; i < MAXSAMPLESTPT; ++i)
	      do_memcpy ( dst, src, len);
	  HP_TIMING_NOW (stop);
	  HP_TIMING_BEST (best_time, start, stop);

      printf ("\t%zd", (size_t) best_time);

}

static void
do_tpt_test (size_t align1, size_t align2, size_t len) {
size_t i, j;
char *s1, *s2;

  s1 = (char *) (buf1 + align1);
  s2 = (char *) (buf2 + align2);
   printf ("TPT: Len %4zd, alignment %2zd/%2zd:", len, align1, align2);
   do_memcpy = memcpy_orig;
   do_one_throughput (s2, s1, len);
   do_memcpy = memcpy_new;
   do_one_throughput (s2, s1, len);
   do_memcpy = memcpy_c;
   do_one_throughput (s2, s1, len);
    putchar ('\n');
}

static test_init(void)
{
int i;
buf1 = valloc(MAXCOPYSIZE);
buf2 = valloc(MAXCOPYSIZE);

  for (i = 0; i < MAXCOPYSIZE ; i = i + 64) {
        buf1[i] = buf2[i] = i & 0xff;
  }

}

void memcpy_c(char *dst, char *src, int len)
{

	__asm__("movq %rdi, %rax");

	__asm__("movl %edx, %ecx");
	__asm__("shrl $3, %ecx");
	__asm__("andl $7, %edx");
	__asm__("rep movsq");
	__asm__("movl %edx, %ecx");
	__asm__("rep movsb");

}
void memcpy_new(char *dst, char *src, int len)

{
	__asm__("movq %rdi, %rax");
	__asm__("movl %edx, %ecx");
	__asm__("shrl   $6, %ecx");
	__asm__("jz 2f");

	__asm__("cmp $0x400, %edx");
	__asm__("jae 7f");

	__asm__("1:");
	__asm__("decl %ecx");

	__asm__("movq 0*8(%rsi), %r11");
	__asm__("movq 1*8(%rdi), %r8");
	__asm__("movq %r11,	0*8(%rdi)");
	__asm__("movq %r8,	1*8(%rdi)");

	__asm__("movq 2*8(%rsi), %r9");
	__asm__("movq 3*8(%rdi), %r10");
	__asm__("movq %r9,	2*8(%rdi)");
	__asm__("movq %r10,	3*8(%rdi)");

	__asm__("movq 4*8(%rsi), %r11");
	__asm__("movq 5*8(%rdi), %r8");
	__asm__("movq %r11,	4*8(%rdi)");
	__asm__("movq %r8,	5*8(%rdi)");

	__asm__("movq 6*8(%rsi), %r9");
	__asm__("movq 7*8(%rdi), %r10");
	__asm__("movq %r9,	6*8(%rdi)");
	__asm__("movq %r10,	7*8(%rdi)");

	__asm__("leaq 64(%rsi), %rsi");
	__asm__("leaq 64(%rdi), %rdi");

	__asm__("jnz  1b");

	__asm__("2:");
	__asm__("movl %edx, %ecx");
	__asm__("andl $63, %ecx");
	__asm__("shrl   $3, %ecx");
	__asm__("jz 4f");


	__asm__("3:");
	__asm__("decl %ecx");
	__asm__("movq (%rsi),	%r8");
	__asm__("movq %r8,	(%rdi)");
	__asm__("leaq 8(%rdi),	%rdi");
	__asm__("leaq 8(%rsi),	%rsi");
	__asm__("jnz 3b");

	__asm__("4:");
	__asm__("movl %edx,	%ecx");
	__asm__("andl $7,	%ecx");
	__asm__("jz 6f");

	__asm__("5:");
	__asm__("movb (%rsi),	%r8b");
	__asm__("movb %r8b, (%rdi)");
	__asm__("incq %rdi");
	__asm__("incq %rsi");
	__asm__("decl %ecx");
	__asm__("jnz 5b");

	__asm__("6:");
	__asm__("retq");

	__asm__("7:");
	__asm__("movl %edx, %ecx");
	__asm__ ("shr $3, %ecx");
	__asm__ ("andl $7, %edx");
	__asm__("rep movsq ");
	__asm__ ("jz 8f");
	__asm__("movl %edx, %ecx");
	__asm__("rep movsb");

	__asm__("8:");
}
void memcpy_orig(char *dst, char *src, int len)
{
	__asm__("movq %rdi, %rax");
	__asm__("movl %edx, %ecx");
	__asm__("shrl   $6, %ecx");
	__asm__("jz 2f");

	__asm__("mov $0x80, %r8d  ");  /*aligned case for loop 1 */

	__asm__("1:");
	__asm__("decl %ecx");

	__asm__("movq 0*8(%rsi), %r11");
	__asm__("movq 1*8(%rdi), %r8");
	__asm__("movq %r11,	0*8(%rdi)");
	__asm__("movq %r8,	1*8(%rdi)");

	__asm__("movq 2*8(%rsi), %r9");
	__asm__("movq 3*8(%rdi), %r10");
	__asm__("movq %r9,	2*8(%rdi)");
	__asm__("movq %r10,	3*8(%rdi)");

	__asm__("movq 4*8(%rsi), %r11");
	__asm__("movq 5*8(%rdi), %r8");
	__asm__("movq %r11,	4*8(%rdi)");
	__asm__("movq %r8,	5*8(%rdi)");

	__asm__("movq 6*8(%rsi), %r9");
	__asm__("movq 7*8(%rdi), %r10");
	__asm__("movq %r9,	6*8(%rdi)");
	__asm__("movq %r10,	7*8(%rdi)");

	__asm__("leaq 64(%rsi), %rsi");
	__asm__("leaq 64(%rdi), %rdi");

	__asm__("jnz  1b");

	__asm__("2:");
	__asm__("movl %edx, %ecx");
	__asm__("andl $63, %ecx");
	__asm__("shrl   $3, %ecx");
	__asm__("jz 4f");


	__asm__("3:");
	__asm__("decl %ecx");
	__asm__("movq (%rsi),	%r8");
	__asm__("movq %r8,	(%rdi)");
	__asm__("leaq 8(%rdi),	%rdi");
	__asm__("leaq 8(%rsi),	%rsi");
	__asm__("jnz 3b");

	__asm__("4:");
	__asm__("movl %edx,	%ecx");
	__asm__("andl $7,	%ecx");
	__asm__("jz 6f");

	__asm__("5:");
	__asm__("movb (%rsi),	%r8b");
	__asm__("movb %r8b, (%rdi)");
	__asm__("incq %rdi");
	__asm__("incq %rsi");
	__asm__("decl %ecx");
	__asm__("jnz 5b");

	__asm__("6:");
}

void main(void)
{
int i;
test_init();
printf ("%23s", "");
printf ("\t%s\t%s\t%s\n", "memcpy_orig", "memcpy_new", "memcpy_c");

  for (i = 0; i < 64; i = i+ 1)
     do_tpt_test(0, 0, i);
     do_tpt_test(0, 0, 1023);
     do_tpt_test(0, 0, 1024);
     do_tpt_test(0, 0, 2048);

}

#19 lore raw
In reply to Ma, Ling
Quoted 8 lines · Ling

Hi All
The attachment is latest memcpy.c, please update by "cc -o memcpy memcpy.c -O2 -m64".

Thanks
Ling

Here is goes

memcpy_orig memcpy_new memcpy_c
TPT: Len 0, alignment 0/ 0: 34482 31920 123564
TPT: Len 1, alignment 0/ 0: 31815 31710 123564
TPT: Len 2, alignment 0/ 0: 39606 31773 123522
TPT: Len 3, alignment 0/ 0: 175329 37212 123522
TPT: Len 4, alignment 0/ 0: 55440 42357 297129
TPT: Len 5, alignment 0/ 0: 63294 47607 296898
TPT: Len 6, alignment 0/ 0: 71148 52794 296856
TPT: Len 7, alignment 0/ 0: 79023 58044 296877
TPT: Len 8, alignment 0/ 0: 32403 32424 123564
TPT: Len 9, alignment 0/ 0: 31752 31815 123522
TPT: Len 10, alignment 0/ 0: 34482 34545 123522
TPT: Len 11, alignment 0/ 0: 42294 39732 123522
TPT: Len 12, alignment 0/ 0: 50211 42378 296856
TPT: Len 13, alignment 0/ 0: 58107 48279 329007
TPT: Len 14, alignment 0/ 0: 65898 53781 296877
TPT: Len 15, alignment 0/ 0: 73773 58065 296877
TPT: Len 16, alignment 0/ 0: 34482 37107 123522
TPT: Len 17, alignment 0/ 0: 31836 31815 123543
TPT: Len 18, alignment 0/ 0: 39627 37044 123522
TPT: Len 19, alignment 0/ 0: 47565 42294 123522
TPT: Len 20, alignment 0/ 0: 55566 47754 296898
TPT: Len 21, alignment 0/ 0: 63273 52773 296877
TPT: Len 22, alignment 0/ 0: 71148 58149 296856
TPT: Len 23, alignment 0/ 0: 79086 63273 296856
TPT: Len 24, alignment 0/ 0: 39816 45024 123522
TPT: Len 25, alignment 0/ 0: 37086 39753 123522
TPT: Len 26, alignment 0/ 0: 44877 44919 123522
TPT: Len 27, alignment 0/ 0: 52773 50253 123522
TPT: Len 28, alignment 0/ 0: 60690 55545 296898
TPT: Len 29, alignment 0/ 0: 68544 60690 296877
TPT: Len 30, alignment 0/ 0: 76398 65961 296877
TPT: Len 31, alignment 0/ 0: 84273 71211 296856
TPT: Len 32, alignment 0/ 0: 45045 52899 123522
TPT: Len 33, alignment 0/ 0: 42315 47628 123522
TPT: Len 34, alignment 0/ 0: 50127 52773 123522
TPT: Len 35, alignment 0/ 0: 58044 58107 123522
TPT: Len 36, alignment 0/ 0: 129612 63462 297129
TPT: Len 37, alignment 0/ 0: 257607 68733 902034
TPT: Len 38, alignment 0/ 0: 81879 73857 296919
TPT: Len 39, alignment 0/ 0: 89460 79023 296856
TPT: Len 40, alignment 0/ 0: 50253 60753 123543
TPT: Len 41, alignment 0/ 0: 47607 55545 123564
TPT: Len 42, alignment 0/ 0: 55356 60627 123522
TPT: Len 43, alignment 0/ 0: 63357 822843 123585
TPT: Len 44, alignment 0/ 0: 71337 71169 297087
TPT: Len 45, alignment 0/ 0: 79023 353388 297129
TPT: Len 46, alignment 0/ 0: 87024 81690 296856
TPT: Len 47, alignment 0/ 0: 94689 86940 296877
TPT: Len 48, alignment 0/ 0: 55482 68523 123522
TPT: Len 49, alignment 0/ 0: 52857 63336 123564
TPT: Len 50, alignment 0/ 0: 60690 68607 123522
TPT: Len 51, alignment 0/ 0: 68502 73731 123522
TPT: Len 52, alignment 0/ 0: 76419 79086 296856
TPT: Len 53, alignment 0/ 0: 84336 126147 296877
TPT: Len 54, alignment 0/ 0: 92190 89607 296877
TPT: Len 55, alignment 0/ 0: 100023 94668 296856
TPT: Len 56, alignment 0/ 0: 60690 76440 123522
TPT: Len 57, alignment 0/ 0: 58065 71211 123522
TPT: Len 58, alignment 0/ 0: 65877 76356 123522
TPT: Len 59, alignment 0/ 0: 73773 81606 196224
TPT: Len 60, alignment 0/ 0: 81732 86961 297129
TPT: Len 61, alignment 0/ 0: 89523 136689 296877
TPT: Len 62, alignment 0/ 0: 97377 97440 296877
TPT: Len 63, alignment 0/ 0: 105210 102564 296877
TPT: Len 1023, alignment 0/ 0: 457569 457107 719502
TPT: Len 1024, alignment 0/ 0: 422856 542535 575526
TPT: Len 2048, alignment 0/ 0: 819651 8217489 982779

#20 lore raw
In reply to Ingo Molnar
Quoted 10 lines

Ling, if you are interested, could you send a user-space test-app to this thread that everyone could just compile and run on various older boxes, to gather a performance profile of hand-coded versus string ops performance?

( And i think we can make a judgement based on cache-hot performance
alone - if then the strings ops will perform comparatively better in cache-cold scenarios, so the cache-hot numbers would be a conservative estimate. )

Ugh, really? I'd expect cache-cold performance to be not helped at all (memory bandwidth limit) and you'll get slow down from additional i-cache misses...
Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#21 lore raw
Quoted 5 lines · Pavel Machek
Quoted 5 lines

Ling, if you are interested, could you send a user-space test-app to this thread that everyone could just compile and run on various older boxes, to gather a performance profile of hand-coded versus string ops performance?

Quoted 8 lines
Quoted 4 lines

( And i think we can make a judgement based on cache-hot performance
alone - if then the strings ops will perform comparatively better in cache-cold scenarios, so the cache-hot numbers would be a conservative estimate. )

Ugh, really? I'd expect cache-cold performance to be not helped at all (memory bandwidth limit) and you'll get slow down from additional i-cache misses...

That's my point - the new code is shorter, which will run comparatively faster in a cache-cold environment.

Ingo

#22 lore raw
Quoted 13 lines · Ingo Molnar
Quoted 10 lines · Pavel Machek
Quoted 9 lines

Ling, if you are interested, could you send a user-space test-app to this thread that everyone could just compile and run on various older boxes, to gather a performance profile of hand-coded versus string ops performance?

( And i think we can make a judgement based on cache-hot performance
alone - if then the strings ops will perform comparatively better in cache-cold scenarios, so the cache-hot numbers would be a conservative estimate. )

Quoted 7 lines
Quoted 3 lines

Ugh, really? I'd expect cache-cold performance to be not helped at all (memory bandwidth limit) and you'll get slow down from additional i-cache misses...

That's my point - the new code is shorter, which will run comparatively faster in a cache-cold environment.

memcpy_c by itself is by far the shortest variant, of course.

The question is if it makes sense to use the long variants for short (< 1024 bytes) copies.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#23 lore raw
Quoted 18 lines · H. Peter Anvin
Quoted 17 lines · Ingo Molnar
Quoted 13 lines · Pavel Machek
Quoted 9 lines

Ling, if you are interested, could you send a user-space test-app to this thread that everyone could just compile and run on various older boxes, to gather a performance profile of hand-coded versus string ops performance?

( And i think we can make a judgement based on cache-hot performance
alone - if then the strings ops will perform comparatively better in cache-cold scenarios, so the cache-hot numbers would be a conservative estimate. )

Ugh, really? I'd expect cache-cold performance to be not helped at all (memory bandwidth limit) and you'll get slow down from additional i-cache misses...

Quoted 5 lines
Quoted 3 lines

That's my point - the new code is shorter, which will run comparatively faster in a cache-cold environment.

memcpy_c by itself is by far the shortest variant, of course.

yep. The argument i made was when a long function was compared to a short one. As you noted we dont actually enable the long function all that often - which inverts the same argument.

Quoted 2 lines

The question is if it makes sense to use the long variants for short (< 1024 bytes) copies.

I'd say not - the kernel executes in a icache-cold environment most of the time (as user-space is far more cache intense in the majority of workloads and kernel processing starts with a cold icache), so optimizing the kernel for code size is very important. (but numbers done on real workloads can convince me of the opposite.)

Ingo

#24 lore raw
In reply to H. Peter Anvin
Quoted 3 lines · H. Peter Anvin

My personal opinion is that if we can show no significant slowdown on P4, K8, P-M/Core 1, Core 2, and Nehalem then we can simply use this code

The issue is Core 2.

P4 uses a different path, and Core 1 doesn't use the 64bit code.

Quoted 2 lines

unconditionally. If one of them is radically worse than baseline, then we have to do something conditional, which is a lot more complicated.

I have an older patchkit which did this, and some more optimizations to this code.

There was still one open issue, that is why I didn't post it. If there's interest I can post it.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#25 lore raw
Quoted 9 lines · Andi Kleen
Quoted 3 lines · H. Peter Anvin

My personal opinion is that if we can show no significant slowdown on P4, K8, P-M/Core 1, Core 2, and Nehalem then we can simply use this code

The issue is Core 2.

P4 uses a different path, and Core 1 doesn't use the 64bit code.

Ling's numbers didn't seem to show a significant slowdown on Core 2 (it was something like 0.95x baseline in the worst case, and most of the cases were positive) so Core 2 doesn't seem to have a problem.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
#26 lore raw
Quoted 3 lines

Ling's numbers didn't seem to show a significant slowdown on Core 2 (it was something like 0.95x baseline in the worst case, and most of the cases were positive) so Core 2 doesn't seem to have a problem.

I ran quite a lot of micro benchmarks with various alignments and sizes the 'q' variant was not always a win. I haven't checked that particular version though.

There's also K8 of course.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#27 lore raw
Quoted 7 lines · Andi Kleen
Quoted 3 lines

Ling's numbers didn't seem to show a significant slowdown on Core 2 (it was something like 0.95x baseline in the worst case, and most of the cases were positive) so Core 2 doesn't seem to have a problem.

I ran quite a lot of micro benchmarks with various alignments and sizes the 'q' variant was not always a win. I haven't checked that particular version though.

Well, if you have concrete information about what the problem cases are, then please provide it. If you don't, but have a hunch where these potential problems may lie, then please indicate what they might be. Otherwise, there isn't any actionable information here.

-hpa

#28 lore raw
In reply to Ma, Ling
Quoted 9 lines · Ling

Hi All

Today we run our benchmark on Core2 and Sandy Bridge:

1. Retrieve result on Core2
Speedup on Core2

   Len        Alignement             Speedup
  1024,       0/ 0:                 0.95x 
  2048,       0/ 0:                 1.03x 

Well, so you are running cache hot and it is only a win on huge copies... how common are those?

Quoted 3 lines

Application run through perf
For (i= 1024; i < 1024 * 16; i = i + 64)

	do_memcpy(0, 0, i);

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
author
#29 lore raw
Quoted 3 lines

Well, so you are running cache hot and it is only a win on huge copies... how common are those?

Hi Pavel Machek
Yes, we intend to introduce movsq for huge hot size(over 1024bytes) and avoid regression for less 1024bytes. I guess you suggest using prefetch instruction for cold data (if I was wrong please correct me). memcpy don't know whether data has been in cache or not,
so only when copy size is over (first level 1 cache)/2 and lower (last level cache)/2 , prefetch will get benefit. Currently first level cache size of most cpus is around 32KB, so it is useful for prefetch when copy size is over 16KB, but as H. Peter Anvin mentioned in last email, over 16KB copy in kernel is rare.

Thanks
Ling

#30 lore raw
Quoted 14 lines · Ling
Quoted 3 lines

Well, so you are running cache hot and it is only a win on huge copies... how common are those?

Hi Pavel Machek
Yes, we intend to introduce movsq for huge hot size(over 1024bytes) and avoid regression for less 1024bytes. I guess you suggest using prefetch instruction for cold data (if I was wrong please correct me). memcpy don't know whether data has been in cache or not,
so only when copy size is over (first level 1 cache)/2 and lower (last level cache)/2 , prefetch will get benefit. Currently first level cache size of most cpus is around 32KB, so it is useful for prefetch when copy size is over 16KB, but as H. Peter Anvin mentioned in last email, over 16KB copy in kernel is rare.

What it sounds to me is that for Nehalem, we want to use memcpy_c for >= 1024 bytes and the old code for < 1024 bytes; for Core2 it might be the exact opposite.

Either way, whatever we do should use the appropriate static replacement mechanism.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.
author
#31 lore raw

Hi H. Peter Anvin

Quoted 2 lines

What it sounds to me is that for Nehalem, we want to use memcpy_c for >= 1024 bytes and the old code for < 1024 bytes;

Yes, so we modify memcpy_c as memcpy_new for Nehalem, and keep old code for Core2 is acceptable?

Thanks
Ling

#32 lore raw
Quoted 6 lines · Ling

Hi H. Peter Anvin

Quoted 2 lines

What it sounds to me is that for Nehalem, we want to use memcpy_c for >= 1024 bytes and the old code for < 1024 bytes;

Yes, so we modify memcpy_c as memcpy_new for Nehalem, and keep old code for Core2 is acceptable?

No, what I think we should do is to rename the old memcpy to something like memcpy_o, and then have the actual memcpy routine look like:

	cmpq $1024, %rcx
	ja memcpy_c
	jmp memcpy_o

... where the constant as well as the ja opcode can be patched by the alternatives mechanism (to a jb if needed).

memcpy is *definitely* frequent enough that static patching is justified.

-hpa

Signature · 2 lines
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.