lib/crypto: x86/sha2: Drop one-off implementations for Sandy/Ivy Bridge
Subsystems (2)
Eric Biggers
maintainerPATCH 1/1 lib/crypto: x86/sha2: Drop one-off implementations for Sandy/Ivy Bridge
[ This is proposed to supersede my patch
"lib/crypto: x86/sha512: Partially roll up sha512_transform_avx()" which was starting to clean up one of the files being dropped. ]
Currently, the kernel has SSSE3, AVX, and AVX2 implementations of each of SHA-256 and SHA-512.
The AVX implementations of SHA-256 and SHA-512 run only on CPUs that have AVX but do *not* have AVX2. These are all the ones I could find:
- Intel Sandy Bridge and Ivy Bridge
- AMD Bulldozer, Piledriver, Steamroller, Jaguar, and Puma
- Zhaoxin KX-5000 and KX-6000 (possibly SHA-512 only, as these CPUs might be able to run the PHE optimized SHA-256 code)
- Maybe some oddly configured VMs
Oddly enough, this "AVX" code doesn't actually have much to do with AVX. The only AVX capability it uses is support for 3-operand VEX-coded instructions, which is unlikely to make a huge difference.
Its actual purpose is mostly unrelated to AVX: it replaces the 'ror' instruction with 'shld'. This is entirely to work around a quirk specific to Sandy Bridge and Ivy Bridge where the 'ror' instruction was very slow for microarchitectural reasons, and 'shld' worked better.
This is documented in the contemporaneous paper "Speeding up SHA-1, SHA-256 and SHA-512 on the 2nd Generation Intel Core Processors"
(https://ieeexplore.ieee.org/document/6209073). There, the 'shld' workaround was claimed to speed up SHA-256 and SHA-512 by about 20% on Sandy Bridge.
However, that was a one-off quirk. And even the paper itself warns about using the 'shld' approach on other microarchitectures.
On AMD CPUs, and on Intel CPUs after Sandy and Ivy Bridge, the exact opposite applies: ror is a fast single-cycle and single µop instruction, while shld is either a slow microcoded or multi-µop instruction.
From what I can find, this is true even on the older AMD CPUs mentioned above that run this code. Thus this code has likely always been harmful on AMD. (I don't have such a CPU available to test, but for what it's worth, on Zen 5 the AVX SHA-512 code is 58% as fast as the SSSE3 code. Clearly, the 'shld' hack can be quite slow on CPUs it wasn't meant for.)
We generally shouldn't be carrying asm files that are one-offs for single microarchitectures, as it's just not worth it. This is especially true when the code has been mistakenly targeted to other CPUs on which it is likely harmful, and when it has other issues that would require maintenance, such as excessive loop unrolling and spilling the entire 640-byte SHA-512 message schedule to the stack. (The SSSE3 code has the same issues, but it's easier to fix it in one place.)
Therefore, let's drop sha256-avx-asm.S and sha512-avx-asm.S.
The CPUs that could have run this code will instead run sha256-ssse3-asm.S and sha512-ssse3-asm.S. SHA-256 and SHA-512 may get around 20% slower on Sandy and Ivy Bridge, but they likely get significantly faster on AMD CPUs from around the same time (2011-2014).
Everything with AVX2 continues to run the newer code and is entirely unaffected by this cleanup.
--- lib/crypto/Makefile | 2 - lib/crypto/x86/sha256-avx-asm.S | 493 -------------------------------- lib/crypto/x86/sha256.h | 12 +- lib/crypto/x86/sha512-avx-asm.S | 421 --------------------------- lib/crypto/x86/sha512.h | 14 +- 5 files changed, 6 insertions(+), 936 deletions(-) delete mode 100644 lib/crypto/x86/sha256-avx-asm.S delete mode 100644 lib/crypto/x86/sha512-avx-asm.S
lib/crypto/Makefile +0 -2
@@ -330,7 +330,6 @@ libsha256-$(CONFIG_PPC) += powerpc/sha256-spe-asm.o libsha256-$(CONFIG_RISCV) += riscv/sha256-riscv64-zvknha_or_zvknhb-zvkb.o libsha256-$(CONFIG_SPARC) += sparc/sha256_asm.o libsha256-$(CONFIG_X86) += x86/sha256-ssse3-asm.o \ - x86/sha256-avx-asm.o \ x86/sha256-avx2-asm.o \ x86/sha256-ni-asm.o endif # CONFIG_CRYPTO_LIB_SHA256_ARCH
@@ -358,7 +357,6 @@ endif libsha512-$(CONFIG_RISCV) += riscv/sha512-riscv64-zvknhb-zvkb.o libsha512-$(CONFIG_SPARC) += sparc/sha512_asm.o libsha512-$(CONFIG_X86) += x86/sha512-ssse3-asm.o \ - x86/sha512-avx-asm.o \ x86/sha512-avx2-asm.o endif # CONFIG_CRYPTO_LIB_SHA512_ARCH
lib/crypto/x86/sha256-avx-asm.S deleted +0 -493
493 lines not on this page · raw message
lib/crypto/x86/sha256.h +3 -9
@@ -27,7 +27,6 @@ DEFINE_STATIC_CALL(sha256_blocks_x86, sha256_blocks_generic); }DEFINE_X86_SHA256_FN (sha256_blocks_ssse3, sha256_transform_ssse3); -DEFINE_X86_SHA256_FN (sha256_blocks_avx, sha256_transform_avx);DEFINE_X86_SHA256_FN (sha256_blocks_avx2, sha256_transform_rorx);DEFINE_X86_SHA256_FN (sha256_blocks_ni, sha256_ni_transform);
@@ -104,14 +103,9 @@ static void sha256_mod_init_arch(void) boot_cpu_has(X86_FEATURE_PHE_EN) && boot_cpu_data.x86 >= 0x07) {static_call_update (sha256_blocks_x86, sha256_blocks_phe); - } else if (boot_cpu_has(X86_FEATURE_AVX)) { -if (boot_cpu_has (X86_FEATURE_AVX2 ) && -boot_cpu_has (X86_FEATURE_BMI2 )) -static_call_update (sha256_blocks_x86, - sha256_blocks_avx2); -else -static_call_update (sha256_blocks_x86, - sha256_blocks_avx); + } else if (boot_cpu_has(X86_FEATURE_AVX2) && +boot_cpu_has (X86_FEATURE_BMI2 )) { +static_call_update (sha256_blocks_x86, sha256_blocks_avx2); } else if (boot_cpu_has(X86_FEATURE_SSSE3)) {static_call_update (sha256_blocks_x86, sha256_blocks_ssse3); }
lib/crypto/x86/sha512-avx-asm.S deleted +0 -421
421 lines not on this page · raw message
lib/crypto/x86/sha512.h +3 -11
@@ -25,7 +25,6 @@ DEFINE_STATIC_CALL(sha512_blocks_x86, sha512_blocks_generic); }DEFINE_X86_SHA512_FN (sha512_blocks_ssse3, sha512_transform_ssse3); -DEFINE_X86_SHA512_FN (sha512_blocks_avx, sha512_transform_avx);DEFINE_X86_SHA512_FN (sha512_blocks_avx2, sha512_transform_rorx); static void sha512_blocks(struct sha512_block_state *state,
@@ -37,15 +36,8 @@ static void sha512_blocks(struct sha512_block_state *state,#define sha512_mod_init_arch sha512_mod_init_archstatic void sha512_mod_init_arch (void ) { -if (boot_cpu_has (X86_FEATURE_AVX )) { -if (boot_cpu_has (X86_FEATURE_AVX2 ) && -boot_cpu_has (X86_FEATURE_BMI2 )) -static_call_update (sha512_blocks_x86, - sha512_blocks_avx2); -else -static_call_update (sha512_blocks_x86, - sha512_blocks_avx); - }else if (boot_cpu_has (X86_FEATURE_SSSE3 )) { +if (boot_cpu_has (X86_FEATURE_AVX2 ) &&boot_cpu_has (X86_FEATURE_BMI2 )) +static_call_update (sha512_blocks_x86, sha512_blocks_avx2); +else if (boot_cpu_has (X86_FEATURE_SSSE3 ))static_call_update (sha512_blocks_x86, sha512_blocks_ssse3); - } }
base-commit: 37b86799c8fd38b2d640d4cac900a2c9de94271a
Signature · 1 line
2.56.1