Skip to contentSkip to conversation

x86/asm changes for v4.18

GIT PULL Ingo Molnar 3 participants 11 messages Jun 4, 2018 – Jun 5, 2018 x86

lore
Share & tools
lore
#0 lore raw

Linus,

Please pull the latest x86-asm-for-linus git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git x86-asm-for-linus

# HEAD: 6469a0ee0a06b2ea1f5afbb1d5a3feed017d4c7a x86/io: Define readq()/writeq() to use 64-bit type

Two smaller changes:

- better support (non-atomic) 64-bit readq()/writeq() variants (Andy Shevchenko)

- __clear_user() micro-optimization (Alexey Dobriyan)

Thanks,

Ingo

------------------>
Alexey Dobriyan (1):
x86/asm/64: Micro-optimize __clear_user() - Use immediate constants

Andy Shevchenko (1):
x86/io: Define readq()/writeq() to use 64-bit type

 arch/x86/include/asm/io.h  | 8 ++++----
 arch/x86/lib/usercopy_64.c | 9 ++++-----
 2 files changed, 8 insertions(+), 9 deletions(-)
arch/x86/include/asm/io.h +4 -4
index f6e5b9375d8c..6de64840dd22 100644
@@ -94,10 +94,10 @@ build_mmio_write(__writel, "l", unsigned int, "r", )
 
 #ifdef CONFIG_X86_64
 
-build_mmio_read(readq, "q", unsigned long, "=r", :"memory")
-build_mmio_read(__readq, "q", unsigned long, "=r", )
-build_mmio_write(writeq, "q", unsigned long, "r", :"memory")
-build_mmio_write(__writeq, "q", unsigned long, "r", )
+build_mmio_read(readq, "q", u64, "=r", :"memory")
+build_mmio_read(__readq, "q", u64, "=r", )
+build_mmio_write(writeq, "q", u64, "r", :"memory")
+build_mmio_write(__writeq, "q", u64, "r", )
 
 #define readq_relaxed(a)	__readq(a)
 #define writeq_relaxed(v, a)	__writeq(v, a)
arch/x86/lib/usercopy_64.c +4 -5
index 75d3776123cc..a624dcc4de10 100644
@@ -23,13 +23,13 @@ unsigned long __clear_user(void __user *addr, unsigned long size)
 	asm volatile(
 		"	testq  %[size8],%[size8]\n"
 		"	jz     4f\n"
-		"0:	movq %[zero],(%[dst])\n"
-		"	addq   %[eight],%[dst]\n"
+		"0:	movq $0,(%[dst])\n"
+		"	addq   $8,%[dst]\n"
 		"	decl %%ecx ; jnz   0b\n"
 		"4:	movq  %[size1],%%rcx\n"
 		"	testl %%ecx,%%ecx\n"
 		"	jz     2f\n"
-		"1:	movb   %b[zero],(%[dst])\n"
+		"1:	movb   $0,(%[dst])\n"
 		"	incq   %[dst]\n"
 		"	decl %%ecx ; jnz  1b\n"
 		"2:\n"
@@ -40,8 +40,7 @@ unsigned long __clear_user(void __user *addr, unsigned long size)
 		_ASM_EXTABLE(0b,3b)
 		_ASM_EXTABLE(1b,2b)
 		: [size8] "=&c"(size), [dst] "=&D" (__d0)
-		: [size1] "r"(size & 7), "[size8]" (size / 8), "[dst]"(addr),
-		  [zero] "r" (0UL), [eight] "r" (8UL));
+		: [size1] "r"(size & 7), "[size8]" (size / 8), "[dst]"(addr));
 	clac();
 	return size;
 }
#1 lore raw
Quoted 2 lines · Ingo Molnar

- __clear_user() micro-optimization (Alexey Dobriyan)

Was this actually tested?

I think one reason people avoided the constant was that on some microarchitecture it ended up being a separate uop just for the constant generation, because it wouldn't fit in a single uop.

I'm pretty sure that used to be the case for P4, for example.

Afaik there have also been issues with decoding instructions that have both an immediate and a memory offset.

I suspect none of this is an issue on modern cores, but there really at least historically were cases where

mov %reg,mem

was better than

mov $imm,mem

if %reg already had the right value, so it's not at all 100% obvious that the micro-optimization really _optimizes_ anything.

Any time people do this, they should add numbers.

Linus

#2 lore raw

Re: x86/asm: __clear_user() micro-optimization (was: "Re: [GIT PULL] x86/asm changes for v4.18")

Quoted 5 lines · Linus Torvalds
Quoted 2 lines · Ingo Molnar

- __clear_user() micro-optimization (Alexey Dobriyan)

Was this actually tested?

I'm not sure - Alexey?

Quoted 14 lines

I think one reason people avoided the constant was that on some microarchitecture it ended up being a separate uop just for the constant generation, because it wouldn't fit in a single uop.

I'm pretty sure that used to be the case for P4, for example.

Afaik there have also been issues with decoding instructions that have both an immediate and a memory offset.

I suspect none of this is an issue on modern cores, but there really at least historically were cases where

mov %reg,mem

Quoted 8 lines

was better than

mov $imm,mem

if %reg already had the right value, so it's not at all 100% obvious that the micro-optimization really _optimizes_ anything.

Any time people do this, they should add numbers.

Ok, fair point and agreed - if Alexey sends some measurements to back the change I'll keep this, otherwise queue up a revert.

Thanks,

Ingo

#3 lore raw

Re: x86/asm: __clear_user() micro-optimization (was: "Re: [GIT PULL] x86/asm changes for v4.18")

Quoted 3 lines · Ingo Molnar

Ok, fair point and agreed - if Alexey sends some measurements to back the change I'll keep this, otherwise queue up a revert.

I don't think it needs to be reverted, it's not like it's likely to hurt on any modern CPU's. The issues I talked about are fairly historical - barely even 64-bit cpus - and I'm not sure an extra uop to carry a constant around even matters in that code sequence.

It was more a generic issue - any micro-optimization should be based on numbers (and there should be some numbers in the commit message), not on "this should be faster". Because while intuitively immediates _should_ be faster than registers, that's simply not always "obviously true". It _may_ be true. But numbers talk.

Linus

#4 lore raw

Re: x86/asm: __clear_user() micro-optimization (was: "Re: [GIT PULL] x86/asm changes for v4.18")

In reply to Ingo Molnar
Quoted 9 lines · Ingo Molnar
Quoted 5 lines · Linus Torvalds
Quoted 2 lines · Ingo Molnar

- __clear_user() micro-optimization (Alexey Dobriyan)

Was this actually tested?

Quoted 8 lines

I'm not sure - Alexey?

Quoted 3 lines

I think one reason people avoided the constant was that on some microarchitecture it ended up being a separate uop just for the constant generation, because it wouldn't fit in a single uop.

Ok, fair point and agreed - if Alexey sends some measurements to back the change I'll keep this, otherwise queue up a revert.

Tested? :^) I had P4 maybe ~15(?) years ago.

godbolt.org earliest compiler is 4.1.2 and it generates "movb [r32], imm8" with "-m32 -O2 -march=pentium4" for simple memset-style loop if it counts for something.

Actually I think __clear_user should be rewritten in C with assembly. It's biggest user is probably ELF loader and those partial page .bss clears should be noticeable.

#5 lore raw

Re: x86/asm: __clear_user() micro-optimization (was: "Re: [GIT PULL] x86/asm changes for v4.18")

Quoted 2 lines · Alexey Dobriyan

Tested? :^) I had P4 maybe ~15(?) years ago.

Did you EVEN test it on what you have today?

Do you have any numbers at all, in other words?

Micro-optimizations need numbers. Otherwise they aren't micro-optimizations, they are just "change code randomly".

Linus

#6 lore raw

Re: x86/asm: __clear_user() micro-optimization (was: "Re: [GIT PULL] x86/asm changes for v4.18")

Quoted 10 lines · Linus Torvalds
Quoted 2 lines · Alexey Dobriyan

Tested? :^) I had P4 maybe ~15(?) years ago.

Did you EVEN test it on what you have today?

Do you have any numbers at all, in other words?

Micro-optimizations need numbers. Otherwise they aren't micro-optimizations, they are just "change code randomly".

On my potato performance increase is 33%, sheesh. And CPU starts doing 3 instructions per cycle vs 2.

benchmark is "clear_user(p + 4096 - 4068, 4068)"
4068 comes from booting Debian 8 with printk.

f0(4068) (old clear_user)
--------

$ taskset -c 15 perf stat -r 16 ./a.out

Performance counter stats for './a.out' (16 runs):

       2033.189084      task-clock (msec)         #    1.000 CPUs utilized            ( +-  0.41% )
                 2      context-switches          #    0.001 K/sec                    ( +- 11.11% )
                 0      cpu-migrations            #    0.000 K/sec
                46      page-faults               #    0.023 K/sec                    ( +-  0.91% )
     4,268,425,486      cycles                    #    2.099 GHz                      ( +-  0.41% )
     8,672,326,256      instructions              #    2.03  insn per cycle           ( +-  0.00% )
     2,169,900,710      branches                  # 1067.240 M/sec                    ( +-  0.00% )
         4,226,258      branch-misses             #    0.19% of all branches          ( +-  0.01% )

2.033700109 seconds time elapsed ( +- 0.41% )

f1(4068) (new clear_user)

$ taskset -c 15 perf stat -r 16 ./a.out

Performance counter stats for './a.out' (16 runs):

       1345.149992      task-clock (msec)         #    1.000 CPUs utilized            ( +-  0.01% )
                 2      context-switches          #    0.002 K/sec                    ( +-  8.35% )
                 0      cpu-migrations            #    0.000 K/sec
                46      page-faults               #    0.034 K/sec                    ( +-  0.82% )
     2,823,965,728      cycles                    #    2.099 GHz                      ( +-  0.01% )
     8,661,733,733      instructions              #    3.07  insn per cycle           ( +-  0.00% )
     2,169,437,410      branches                  # 1612.785 M/sec                    ( +-  0.00% )
         4,216,469      branch-misses             #    0.19% of all branches          ( +-  0.01% )

1.345375114 seconds time elapsed ( +- 0.01% )

-------------------------------------
CFLAGS = -Wall -fno-strict-aliasing -fno-common -fshort-wchar -std=gnu89 -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -funit-at-a-time -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -O2 --param=allow-store-data-races=0 -fno-stack-protector -fomit-frame-pointer -fno-var-tracking-assignments -g -femit-struct-debug-baseonly -fno-var-tracking -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack

0000000000000780 <f0>:

 780:	mov    rax,rsi
 783:	mov    rcx,rsi
 786:	xor    edx,edx
 788:	and    eax,0x7
 78b:	shr    rcx,0x3
 78f:	mov    esi,0x8
 794:	test   rcx,rcx
 797:	je     7a3 <f0+0x23>
 799:	mov    QWORD PTR [rdi],rdx
 79c:	add    rdi,rsi
 79f:	dec    ecx
 7a1:	jne    799 <f0+0x19>
 7a3:	mov    rcx,rax
 7a6:	test   ecx,ecx
 7a8:	je     7b3 <f0+0x33>
 7aa:	mov    BYTE PTR [rdi],dl
 7ac:	inc    rdi
 7af:	dec    ecx
 7b1:	jne    7aa <f0+0x2a>
 7b3:	mov    rax,rcx

7b6: ret

00000000000007c0 <f1>:

 7c0:	mov    rax,rsi
 7c3:	shr    rsi,0x3
 7c7:	and    eax,0x7
 7ca:	mov    rcx,rsi
 7cd:	test   rcx,rcx
 7d0:	je     7e1 <f1+0x21>
 7d2:	mov    QWORD PTR [rdi],0x0
 7d9:	add    rdi,0x8
 7dd:	dec    ecx
 7df:	jne    7d2 <f1+0x12>
 7e1:	mov    rcx,rax
 7e4:	test   ecx,ecx
 7e6:	je     7f2 <f1+0x32>
 7e8:	mov    BYTE PTR [rdi],0x0
 7eb:	inc    rdi
 7ee:	dec    ecx
 7f0:	jne    7e8 <f1+0x28>
 7f2:	mov    rax,rcx

7f5: ret

#7 lore raw

Re: x86/asm: __clear_user() micro-optimization (was: "Re: [GIT PULL] x86/asm changes for v4.18")

Quoted 3 lines · Alexey Dobriyan

On my potato performance increase is 33%, sheesh. And CPU starts doing 3 instructions per cycle vs 2.

Whee. That's a shockingly big difference. On my CPU (i7-6700K) it makes absolutely no difference whether the values are integers or in registers.

Linus

#8 lore raw

Re: x86/asm: __clear_user() micro-optimization (was: "Re: [GIT PULL] x86/asm changes for v4.18")

Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Alexey Dobriyan

On my potato performance increase is 33%, sheesh. And CPU starts doing 3 instructions per cycle vs 2.

Whee. That's a shockingly big difference. On my CPU (i7-6700K) it makes absolutely no difference whether the values are integers or in registers.

In fact, looking at Agner Fog's instruction lists, I don't see any CPU where it would make a difference, except for the P4 (where the immediate looks like it's a bad idea because it's an extra uop, but it might pack fine and not be noticeable).

But maybe I'm missing something subtle. What CPU, out of morbid interest?

Linus

#9 lore raw

Re: x86/asm: __clear_user() micro-optimization (was: "Re: [GIT PULL] x86/asm changes for v4.18")

Quoted 12 lines · Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Alexey Dobriyan

On my potato performance increase is 33%, sheesh. And CPU starts doing 3 instructions per cycle vs 2.

Whee. That's a shockingly big difference. On my CPU (i7-6700K) it makes absolutely no difference whether the values are integers or in registers.

Quoted 6 lines

In fact, looking at Agner Fog's instruction lists, I don't see any CPU where it would make a difference, except for the P4 (where the immediate looks like it's a bad idea because it's an extra uop, but it might pack fine and not be noticeable).

But maybe I'm missing something subtle. What CPU, out of morbid interest?

This is Broadwell Xeon E5-2620 v4.
Which is somewhat strange indeed because it should be modern enough.

#10 lore raw

Re: x86/asm: __clear_user() micro-optimization (was: "Re: [GIT PULL] x86/asm changes for v4.18")

Quoted 3 lines · Alexey Dobriyan

This is Broadwell Xeon E5-2620 v4.
Which is somewhat strange indeed because it should be modern enough.

Yeah, odd.

Here's the benchmark I used:

  #define SIZE 4068

  int main(int argc, char **argv)
  {
    int i;
    unsigned char buffer[SIZE], *p;
    for (i = 0; i < 1000000; i++)
        asm volatile(
            "1: movq %[zero],(%[mem]); addq %[eight],%[mem]; decl
%[count]; jne 1b"
            : [mem] "=r" (p)
            : [zero] "i" (0l), [eight] "i" (8l),
             "0" (buffer), [count] "r" (SIZE/8));

}

where you can change that "i" for [zero] and [eight] to be "r" to get the register version.

I just timed it, because I'm lazy and perf seemed to be overkill.

It might be some very specific loop buffer issue or something.

Or maybe my benchmark above is broken, I didn't really verify that the end result was any good (I just did an objdump to verify the asm code superficially).

Linus