Skip to contentSkip to conversation

x86/cpufeatures: Add feature flag for fast short rep movsb

PATCH In review Tony Luck 4 participants 8 messages Dec 12, 2019 – Jan 8, 2020 x86

lore
Share & tools
lore

Tony Luck

author
#0 lore raw

PATCH 1/1 x86/cpufeatures: Add feature flag for fast short rep movsb

From the Intel Optimization Reference Manual:

3.7.6.1 Fast Short REP MOVSB
Beginning with processors based on Ice Lake Client microarchitecture, REP MOVSB performance of short operations is enhanced. The enhancement applies to string lengths between 1 and 128 bytes long. Support for fast-short REP MOVSB is enumerated by the CPUID feature flag: CPUID [EAX=7H, ECX=0H).EDX.FAST_SHORT_REP_MOVSB[bit 4] = 1. There is no change in the REP STOS performance.

Add an X86_FEATURE_FSRM flag for this.

---

Net effect of this patch is just to make "fsrm" appear in the flags section of /proc/cpuinfo. Maybe someone can look into whether we should make copy routines that use "rep movsb" check for this flag to optimize copies on older CPUs that don't have it?

 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index 0652d3eed9bd..ab441b15d582 100644
@@ -356,6 +356,7 @@
 /* Intel-defined CPU features, CPUID level 0x00000007:0 (EDX), word 18 */
 #define X86_FEATURE_AVX512_4VNNIW	(18*32+ 2) /* AVX-512 Neural Network Instructions */
 #define X86_FEATURE_AVX512_4FMAPS	(18*32+ 3) /* AVX-512 Multiply Accumulation Single precision */
+#define X86_FEATURE_FSRM		(18*32+ 4) /* Fast Short Rep Mov */
 #define X86_FEATURE_AVX512_VP2INTERSECT (18*32+ 8) /* AVX-512 Intersect for D/Q */
 #define X86_FEATURE_MD_CLEAR		(18*32+10) /* VERW clears CPU buffers */
 #define X86_FEATURE_TSX_FORCE_ABORT	(18*32+13) /* "" TSX_FORCE_ABORT */
Signature · 1 line
2.20.1
maintainer
#1 lore raw
Quoted 12 lines · Tony Luck

From the Intel Optimization Reference Manual:

3.7.6.1 Fast Short REP MOVSB
Beginning with processors based on Ice Lake Client microarchitecture, REP MOVSB performance of short operations is enhanced. The enhancement applies to string lengths between 1 and 128 bytes long. Support for fast-short REP MOVSB is enumerated by the CPUID feature flag: CPUID [EAX=7H, ECX=0H).EDX.FAST_SHORT_REP_MOVSB[bit 4] = 1. There is no change in the REP STOS performance.

Add an X86_FEATURE_FSRM flag for this.

Quoted 7 lines

---

Net effect of this patch is just to make "fsrm" appear in the flags section of /proc/cpuinfo. Maybe someone can look into whether we should make copy routines that use "rep movsb" check for this flag to optimize copies on older CPUs that don't have it?

We can then add the feature flag too. Just showing it in /proc/cpuinfo without any users is kinda pointless...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author
#2 lore raw

PATCH 1/1 x86/cpufeatures: Add support for fast short rep mov

From the Intel Optimization Reference Manual:

3.7.6.1 Fast Short REP MOVSB
Beginning with processors based on Ice Lake Client microarchitecture, REP MOVSB performance of short operations is enhanced. The enhancement applies to string lengths between 1 and 128 bytes long. Support for fast-short REP MOVSB is enumerated by the CPUID feature flag: CPUID [EAX=7H, ECX=0H).EDX.FAST_SHORT_REP_MOVSB[bit 4] = 1. There is no change in the REP STOS performance.

Add an X86_FEATURE_FSRM flag for this.

memmove() avoids REP MOVSB for short (< 32 byte) copies. Fix it to check FSRM and use REP MOVSB for short copies on systems that support it.

---

Time (cycles) for memmove() sizes 1..31 with neither source nor destination in cache.

1800 +-+-------+--------+---------+---------+---------+--------+-------+-+
+ + + + + + + +
1600 +-+ 'memmove-fsrm' *******-+

       |   ######                                   'memmove-orig' ####### |
  1400 +-+ #     #####################                                   +-+
       |   #                          ############                         |
  1200 +-+#                                       ##################     +-+
       |  #                                                                |
  1000 +-+#                                                              +-+
       |  #                                                                |
       | #                                                                 |
   800 +-#                                                               +-+
       | #                                                                 |
   600 +-***********************                                         +-+
       |                        *****************************              |
   400 +-+                                                   *******     +-+
       |                                                                   |
   200 +-+                                                               +-+
       +         +        +         +         +         +        +         +
     0 +-+-------+--------+---------+---------+---------+--------+-------+-+
       0         5        10        15        20        25       30        35
---
 arch/x86/include/asm/cpufeatures.h | 1 +
 arch/x86/lib/memmove_64.S          | 6 +++---
 2 files changed, 4 insertions(+), 3 deletions(-)
arch/x86/include/asm/cpufeatures.h +1 -0
index e9b62498fe75..98c60fa31ced 100644
@@ -357,6 +357,7 @@
 /* Intel-defined CPU features, CPUID level 0x00000007:0 (EDX), word 18 */
 #define X86_FEATURE_AVX512_4VNNIW	(18*32+ 2) /* AVX-512 Neural Network Instructions */
 #define X86_FEATURE_AVX512_4FMAPS	(18*32+ 3) /* AVX-512 Multiply Accumulation Single precision */
+#define X86_FEATURE_FSRM		(18*32+ 4) /* Fast Short Rep Mov */
 #define X86_FEATURE_AVX512_VP2INTERSECT (18*32+ 8) /* AVX-512 Intersect for D/Q */
 #define X86_FEATURE_MD_CLEAR		(18*32+10) /* VERW clears CPU buffers */
 #define X86_FEATURE_TSX_FORCE_ABORT	(18*32+13) /* "" TSX_FORCE_ABORT */
arch/x86/lib/memmove_64.S +3 -3
index 337830d7a59c..4a23086806e6 100644
@@ -29,10 +29,7 @@
 SYM_FUNC_START_ALIAS(memmove)
 SYM_FUNC_START(__memmove)
 
-	/* Handle more 32 bytes in loop */
 	mov %rdi, %rax
-	cmp $0x20, %rdx
-	jb	1f
 
 	/* Decide forward/backward copy mode */
 	cmp %rdi, %rsi
@@ -43,6 +40,7 @@ SYM_FUNC_START(__memmove)
 	jg 2f
 
 .Lmemmove_begin_forward:
+	ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "", X86_FEATURE_FSRM
 	ALTERNATIVE "", "movq %rdx, %rcx; rep movsb; retq", X86_FEATURE_ERMS
 
 	/*
@@ -114,6 +112,8 @@ SYM_FUNC_START(__memmove)
 	 */
 	.p2align 4
 2:
+	cmp $0x20, %rdx
+	jb	1f
 	cmp $680, %rdx
 	jb 6f
 	cmp %dil, %sil
Signature · 1 line
2.20.1
maintainer
#3 lore raw

Re: [PATCH] x86/cpufeatures: Add support for fast short rep mov

Quoted 42 lines · Tony Luck

From the Intel Optimization Reference Manual:

3.7.6.1 Fast Short REP MOVSB
Beginning with processors based on Ice Lake Client microarchitecture, REP MOVSB performance of short operations is enhanced. The enhancement applies to string lengths between 1 and 128 bytes long. Support for fast-short REP MOVSB is enumerated by the CPUID feature flag: CPUID [EAX=7H, ECX=0H).EDX.FAST_SHORT_REP_MOVSB[bit 4] = 1. There is no change in the REP STOS performance.

Add an X86_FEATURE_FSRM flag for this.

memmove() avoids REP MOVSB for short (< 32 byte) copies. Fix it to check FSRM and use REP MOVSB for short copies on systems that support it.

---

Time (cycles) for memmove() sizes 1..31 with neither source nor destination in cache.

1800 +-+-------+--------+---------+---------+---------+--------+-------+-+
+ + + + + + + +
1600 +-+ 'memmove-fsrm' *******-+

       |   ######                                   'memmove-orig' ####### |
  1400 +-+ #     #####################                                   +-+
       |   #                          ############                         |
  1200 +-+#                                       ##################     +-+
       |  #                                                                |
  1000 +-+#                                                              +-+
       |  #                                                                |
       | #                                                                 |
   800 +-#                                                               +-+
       | #                                                                 |
   600 +-***********************                                         +-+
       |                        *****************************              |
   400 +-+                                                   *******     +-+
       |                                                                   |
   200 +-+                                                               +-+
       +         +        +         +         +         +        +         +
Quoted 2 lines

0 +-+-------+--------+---------+---------+---------+--------+-------+-+
0 5 10 15 20 25 30 35

I don't mind this graph being part of the commit message - it shows nicely the speedup even if with some microbenchmark. Or you're not adding it just because it is a microbenchmark and not something more representative?

Quoted 32 lines
 arch/x86/include/asm/cpufeatures.h | 1 +
 arch/x86/lib/memmove_64.S          | 6 +++---
 2 files changed, 4 insertions(+), 3 deletions(-)
arch/x86/include/asm/cpufeatures.h +1 -0
index e9b62498fe75..98c60fa31ced 100644
@@ -357,6 +357,7 @@
 /* Intel-defined CPU features, CPUID level 0x00000007:0 (EDX), word 18 */
 #define X86_FEATURE_AVX512_4VNNIW	(18*32+ 2) /* AVX-512 Neural Network Instructions */
 #define X86_FEATURE_AVX512_4FMAPS	(18*32+ 3) /* AVX-512 Multiply Accumulation Single precision */
+#define X86_FEATURE_FSRM		(18*32+ 4) /* Fast Short Rep Mov */
 #define X86_FEATURE_AVX512_VP2INTERSECT (18*32+ 8) /* AVX-512 Intersect for D/Q */
 #define X86_FEATURE_MD_CLEAR		(18*32+10) /* VERW clears CPU buffers */
 #define X86_FEATURE_TSX_FORCE_ABORT	(18*32+13) /* "" TSX_FORCE_ABORT */
arch/x86/lib/memmove_64.S +0 -3
index 337830d7a59c..4a23086806e6 100644
@@ -29,10 +29,7 @@
 SYM_FUNC_START_ALIAS(memmove)
 SYM_FUNC_START(__memmove)
 
-	/* Handle more 32 bytes in loop */
 	mov %rdi, %rax
-	cmp $0x20, %rdx
-	jb	1f
 
 	/* Decide forward/backward copy mode */
 	cmp %rdi, %rsi

arch/x86/lib/memmove_64.S SYM_FUNC_START(__memmove) · line 43

Quoted 4 lines
arch/x86/lib/memmove_64.S +1 -0
@@ -43,6 +40,7 @@ SYM_FUNC_START(__memmove)
 	jg 2f
 
 .Lmemmove_begin_forward:
+	ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "", X86_FEATURE_FSRM

So the enhancement is for string lengths up to two cachelines. Why are you limiting this to 32 bytes?

I know, the function handles 32-bytes at a time but what I'd imagine here is having the fastest variant upfront which does REP; MOVSB for all lengths since FSRM means fast short strings and ERMS - and I'm strongly assuming here FSRM *implies* ERMS - means fast "longer" strings, so to speak, so FSRM would mean fast *all length* strings in the end, no?

Also, does the copy direction influence the FSRM's REP; MOVSB variant's performance? If not, you can do something like this:

SYM_FUNC_START_ALIAS(memmove)
SYM_FUNC_START(__memmove)

	mov %rdi, %rax

	/* FSRM handles all possible string lengths and directions optimally. */
	ALTERNATIVE "", "movq %rdx, %rcx; rep movsb; retq", X86_FEATURE_FSRM

	cmp $0x20, %rdx
	jb 1f
	...

Or?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
author
#4 lore raw

Re: [PATCH] x86/cpufeatures: Add support for fast short rep mov

Quoted 4 lines · Borislav Petkov

I don't mind this graph being part of the commit message - it shows nicely the speedup even if with some microbenchmark. Or you're not adding it just because it is a microbenchmark and not something more representative?

I'm not sure it should be archived forever in the commit message. The benchmark was run on A-step silicon, so may not be representative of production results.

Quoted 6 lines

arch/x86/lib/memmove_64.S SYM_FUNC_START(__memmove) · line 43

Quoted 2 lines

.Lmemmove_begin_forward:

arch/x86/lib/memmove_64.S +1 -0
+	ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "", X86_FEATURE_FSRM

So the enhancement is for string lengths up to two cachelines. Why are you limiting this to 32 bytes?

Quoted 8 lines

I know, the function handles 32-bytes at a time but what I'd imagine here is having the fastest variant upfront which does REP; MOVSB for all lengths since FSRM means fast short strings and ERMS - and I'm strongly assuming here FSRM *implies* ERMS - means fast "longer" strings, so to speak, so FSRM would mean fast *all length* strings in the end, no?

Also, does the copy direction influence the FSRM's REP; MOVSB variant's performance? If not, you can do something like this:

Yes FSRM implies ERMS

You can't use REP MOVS for overlapping src/dst strings (not even with the fancy newer, faster, shinier FSRM version). So your suggestion will not work.

The old memmove code looked something like:

if (len < 32)
copy tail (backwards ... 8/4/2/1 bytes. works for both overlap & non-overlap case) return
else if overlap src/dst
copy backwards 32-byte unrolled
copy tail
return
else if (ERMS)
REP MOVS;
return
else
unrolled copy 32-byte
copy tail

The new one with my changes looks something like

	if (! overlap src/dst)
		if (FSRM)
			rep movs
			return
		if (len < 32)
			copy tail
			return
		if (ERMS)
			rep movs
			return
		unrolled copy
	else
		if (len < 32)
			copy tail
			return
		copy backwards 32-byte unrolled
		copy tail

-Tony

maintainer
#5 lore raw

Re: [PATCH] x86/cpufeatures: Add support for fast short rep mov

Quoted 1 line · Tony

Yes FSRM implies ERMS

Ok, I've added this comment ontop so that it is clear what's going on there:

	/* FSRM implies ERMS => no length checks, do the copy directly */
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
merge bot Applied
#6 lore raw

[tip: x86/asm] x86/cpufeatures: Add support for fast short REP; MOVSB

In reply to Tony Luck
The following commit has been merged into the x86/asm branch of tip:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     f444a5ff95dce07cf4353cbb85fc3e785019d430
Gitweb:        https://git.kernel.org/tip/f444a5ff95dce07cf4353cbb85fc3e785019d430
Author:        Tony Luck <[email protected]>
AuthorDate:    Mon, 16 Dec 2019 13:42:54 -08:00
Committer:     Borislav Petkov <[email protected]>
CommitterDate: Wed, 08 Jan 2020 11:29:25 +01:00

x86/cpufeatures: Add support for fast short REP; MOVSB

Quoted 1 line

From the Intel Optimization Reference Manual:

3.7.6.1 Fast Short REP MOVSB
Beginning with processors based on Ice Lake Client microarchitecture, REP MOVSB performance of short operations is enhanced. The enhancement applies to string lengths between 1 and 128 bytes long. Support for fast-short REP MOVSB is enumerated by the CPUID feature flag: CPUID [EAX=7H, ECX=0H).EDX.FAST_SHORT_REP_MOVSB[bit 4] = 1. There is no change in the REP STOS performance.

Add an X86_FEATURE_FSRM flag for this.

memmove() avoids REP MOVSB for short (< 32 byte) copies. Check FSRM and use REP MOVSB for short copies on systems that support it.

[ bp: Massage and add comment. ]

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 arch/x86/lib/memmove_64.S          | 7 ++++---
 2 files changed, 5 insertions(+), 3 deletions(-)
arch/x86/include/asm/cpufeatures.h +1 -0
index e9b6249..98c60fa 100644
@@ -357,6 +357,7 @@
 /* Intel-defined CPU features, CPUID level 0x00000007:0 (EDX), word 18 */
 #define X86_FEATURE_AVX512_4VNNIW	(18*32+ 2) /* AVX-512 Neural Network Instructions */
 #define X86_FEATURE_AVX512_4FMAPS	(18*32+ 3) /* AVX-512 Multiply Accumulation Single precision */
+#define X86_FEATURE_FSRM		(18*32+ 4) /* Fast Short Rep Mov */
 #define X86_FEATURE_AVX512_VP2INTERSECT (18*32+ 8) /* AVX-512 Intersect for D/Q */
 #define X86_FEATURE_MD_CLEAR		(18*32+10) /* VERW clears CPU buffers */
 #define X86_FEATURE_TSX_FORCE_ABORT	(18*32+13) /* "" TSX_FORCE_ABORT */
arch/x86/lib/memmove_64.S +4 -3
index 337830d..7ff00ea 100644
@@ -29,10 +29,7 @@
 SYM_FUNC_START_ALIAS(memmove)
 SYM_FUNC_START(__memmove)
 
-	/* Handle more 32 bytes in loop */
 	mov %rdi, %rax
-	cmp $0x20, %rdx
-	jb	1f
 
 	/* Decide forward/backward copy mode */
 	cmp %rdi, %rsi
@@ -42,7 +39,9 @@ SYM_FUNC_START(__memmove)
 	cmp %rdi, %r8
 	jg 2f
 
+	/* FSRM implies ERMS => no length checks, do the copy directly */
 .Lmemmove_begin_forward:
+	ALTERNATIVE "cmp $0x20, %rdx; jb 1f", "", X86_FEATURE_FSRM
 	ALTERNATIVE "", "movq %rdx, %rcx; rep movsb; retq", X86_FEATURE_ERMS
 
 	/*
@@ -114,6 +113,8 @@ SYM_FUNC_START(__memmove)
 	 */
 	.p2align 4
 2:
+	cmp $0x20, %rdx
+	jb 1f
 	cmp $680, %rdx
 	jb 6f
 	cmp %dil, %sil
#7 lore raw

Re: [tip: x86/asm] x86/cpufeatures: Add support for fast short REP; MOVSB

Quoted 23 lines · * tip-bot2 for Tony Luck <[email protected]> wrote:

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     f444a5ff95dce07cf4353cbb85fc3e785019d430
Gitweb:        https://git.kernel.org/tip/f444a5ff95dce07cf4353cbb85fc3e785019d430
Author:        Tony Luck <[email protected]>
AuthorDate:    Mon, 16 Dec 2019 13:42:54 -08:00
Committer:     Borislav Petkov <[email protected]>
CommitterDate: Wed, 08 Jan 2020 11:29:25 +01:00

x86/cpufeatures: Add support for fast short REP; MOVSB

Quoted 1 line

From the Intel Optimization Reference Manual:

3.7.6.1 Fast Short REP MOVSB
Beginning with processors based on Ice Lake Client microarchitecture, REP MOVSB performance of short operations is enhanced. The enhancement applies to string lengths between 1 and 128 bytes long. Support for fast-short REP MOVSB is enumerated by the CPUID feature flag: CPUID [EAX=7H, ECX=0H).EDX.FAST_SHORT_REP_MOVSB[bit 4] = 1. There is no change in the REP STOS performance.

Add an X86_FEATURE_FSRM flag for this.

Quoted 8 lines

memmove() avoids REP MOVSB for short (< 32 byte) copies. Check FSRM and use REP MOVSB for short copies on systems that support it.

[ bp: Massage and add comment. ]

BTW., just for the record, the 32-bit version of memmove() has a similar cut-off as well, at 680 bytes (!):

                /*
                 * movs instruction have many startup latency
                 * so we handle small size by general register.
                 */
                "cmp  $680, %0\n\t"
                "jb 3f\n\t"

...

                /*
                 * Start to prepare for backward copy.
                 */
                ".p2align 4\n\t"
                "2:\n\t"
                "cmp  $680, %0\n\t"
                "jb 5f\n\t"

This logic was introduced in 2010 via:

3b4b682becdf: ("x86, mem: Optimize memmove for small size and unaligned cases")

However because those patches came without actual performance measurements, I'd be inclined to switch back to the old REP MOVSB version
- which would also automatically improve it should anyone run 32-bit kernels on the very latest CPUs.

Thanks,

Ingo