Skip to content

x86: Avoid using INC and DEC instructions on hot paths

Ammar Faizi linux-hardening lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest2 patches 3 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/2x86/include/asm: Avoid using INC and DEC instructions on hot pathsarch/x86/include/asm/xor_32.h+8 -80 replies
2/2x86/lib: Avoid using INC and DEC instructions on hot pathsarch/x86/lib/copy_mc_64.S arch/x86/lib/copy_user_64.S arch/x86/lib/memset_64.S +3 more+41 -410 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH v1 0/2] x86: Avoid using INC and DEC instructions on hot paths

Hi,

In order to take maximum advantage of out-of-order execution, avoid using INC/DEC instructions when appropriate. INC/DEC only writes to part of the flags register, which can cause a partial flag register stall. This series replaces INC/DEC with ADD/SUB.

Agner Fog's optimization manual says [1]:
"""
The INC and DEC instructions are inefficient on some CPUs because they write to only part of the flags register (excluding the carry flag). Use ADD or SUB instead to avoid false dependences or inefficient splitting of the flags register, especially if they are followed by an instruction that reads the flags.
"""

Intel's optimization manual 3.5.1.1 says [2]:
"""
The INC and DEC instructions modify only a subset of the bits in the flag register. This creates a dependence on all previous writes of the flag register. This is especially problematic when these instructions are on the critical path because they are used to change an address for a load on which many other instructions depend.

Assembly/Compiler Coding Rule 33. (M impact, H generality) INC and DEC instructions should be replaced with ADD or SUB instructions, because ADD and SUB overwrite all flags, whereas INC and DEC do not, therefore creating false dependencies on earlier instructions that set the flags.
"""

Newer compilers also do it for generic x86-64 CPU (https://godbolt.org/z/rjsfbdx54).
# C code:

  int fy_inc(int a, int b, int c)
  {
      a++; b++; c++;
      return a * b * c;
  }

# ASM
## GCC 4.1.2 and older use INC (old).
fy_inc:

    incl    %edi
    incl    %esi
    leal    1(%rdx), %eax
    imull   %esi, %edi
    imull   %edi, %eax

ret

## GCC 4.4.7 to GCC 11.2 use ADD (new).
fy_inc:

    addl    $1, %edi
    addl    $1, %esi
    addl    $1, %edx
    imull   %esi, %edi
    movl    %edi, %eax
    imull   %edx, %eax

ret

## Clang 5.0.2 and older use INC (old).
fy_inc:

    incl    %edi
    leal    1(%rsi), %eax
    imull   %edi, %eax
    incl    %edx
    imull   %edx, %eax

retq

## Clang 6.0.0 to Clang 13.0.1 use ADD (new).
fy_inc:

    addl    $1, %edi
    leal    1(%rsi), %eax
    imull   %edi, %eax
    addl    $1, %edx
    imull   %edx, %eax

retq

[1]: https://www.agner.org/optimize/optimizing_assembly.pdf
[2]: https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-optimization-manual.pdf

---
Ammar Faizi (2):
x86/include/asm: Avoid using INC and DEC instructions on hot paths x86/lib: Avoid using INC and DEC instructions on hot paths

 arch/x86/include/asm/xor_32.h | 16 ++++++++--------
 arch/x86/lib/copy_mc_64.S     | 14 +++++++-------
 arch/x86/lib/copy_user_64.S   | 26 +++++++++++++-------------
 arch/x86/lib/memset_64.S      |  6 +++---
 arch/x86/lib/string_32.c      | 20 ++++++++++----------
 arch/x86/lib/strstr_32.c      |  4 ++--
 arch/x86/lib/usercopy_64.c    | 12 ++++++------
 7 files changed, 49 insertions(+), 49 deletions(-)

base-commit: ffb217a13a2eaf6d5bd974fc83036a53ca69f1e2

Signature · 1 line
Ammar Faizi