lib/crypto: blake2b: Roll up BLAKE2b round loop on 32-bit
Subsystems (3)
Eric Biggers
maintainerPATCH v2 1/1 lib/crypto: blake2b: Roll up BLAKE2b round loop on 32-bit
BLAKE2b has a state of 16 64-bit words. Add the message data in and there are 32 64-bit words. With the current code where all the rounds are unrolled to enable constant-folding of the blake2b_sigma values, this results in a very large code size on 32-bit kernels, including a recurring issue where gcc uses a large amount of stack.
There's just not much benefit to this unrolling when the code is already so large. Let's roll up the rounds when !CONFIG_64BIT.
To avoid having to duplicate the code, just write code once using a loop, and conditionally use 'unrolled_full' from <linux/unroll.h>.
Then, fold the now-unneeded ROUND() macro into the loop. Finally, also remove the now-unneeded override of the stack frame size warning.
Code size improvements for blake2b_compress_generic():
Size before (bytes) Size after (bytes)
------------------- ------------------
i386, gcc 27584 3632
i386, clang 18208 3248
arm32, gcc 19912 2860
arm32, clang 21336 3344Running the BLAKE2b benchmark on a !CONFIG_64BIT kernel on an x86_64 processor shows a 16384B throughput change of 351 => 340 MB/s (gcc) or 442 MB/s => 375 MB/s (clang). So clearly not much of a slowdown either. But also that microbenchmark also effectively disregards cache usage, which is important in practice and is far better in the smaller code.
Note: If we rolled up the loop on x86_64 too, the change would be 7024 bytes => 1584 bytes and 1960 MB/s => 1396 MB/s (gcc), or 6848 bytes => 1696 bytes and 1920 MB/s => 1263 MB/s (clang). Maybe still worth it, though not quite as clearly beneficial.
--- lib/crypto/Makefile | 1 - lib/crypto/blake2b.c | 44 ++++++++++++++++++++------------------------ 2 files changed, 20 insertions(+), 25 deletions(-)
lib/crypto/Makefile +0 -1
@@ -31,11 +31,10 @@ obj-$(CONFIG_CRYPTO_LIB_GF128MUL) += gf128mul.o ################################################################################ obj-$(CONFIG_CRYPTO_LIB_BLAKE2B) += libblake2b.o libblake2b-y := blake2b.o -CFLAGS_blake2b.o := -Wframe-larger-than=4096 # https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105930 ifeq ($(CONFIG_CRYPTO_LIB_BLAKE2B_ARCH),y) CFLAGS_blake2b.o += -I$(src)/$(SRCARCH) libblake2b-$(CONFIG_ARM) += arm/blake2b-neon-core.o endif # CONFIG_CRYPTO_LIB_BLAKE2B_ARCH
lib/crypto/blake2b.c +20 -24
65 lines not on this page · raw message
base-commit: 3f9f0252130e7dd60d41be0802bf58f6471c691d
Signature · 1 line
2.52.0
Quoted 30 lines · Eric Biggers
BLAKE2b has a state of 16 64-bit words. Add the message data in and there are 32 64-bit words. With the current code where all the rounds are unrolled to enable constant-folding of the
blake2b_sigmavalues, this results in a very large code size on 32-bit kernels, including a recurring issue where gcc uses a large amount of stack.There's just not much benefit to this unrolling when the code is already so large. Let's roll up the rounds when !CONFIG_64BIT.
To avoid having to duplicate the code, just write code once using a loop, and conditionally use '
unrolled_full' from <linux/unroll.h>.Then, fold the now-unneeded
ROUND()macro into the loop. Finally, also remove the now-unneeded override of the stack frame size warning.Code size improvements for
blake2b_compress_generic():Size before (bytes) Size after (bytes) ------------------- ------------------ i386, gcc 27584 3632 i386, clang 18208 3248 arm32, gcc 19912 2860 arm32, clang 21336 3344Running the BLAKE2b benchmark on a !CONFIG_64BIT kernel on an
x86_64processor shows a 16384B throughput change of 351 => 340 MB/s (gcc) or 442 MB/s => 375 MB/s (clang). So clearly not much of a slowdown either. But also that microbenchmark also effectively disregards cache usage, which is important in practice and is far better in the smaller code.
Quoted 8 lines
Note: If we rolled up the loop on
x86_64too, the change would be 7024 bytes => 1584 bytes and 1960 MB/s => 1396 MB/s (gcc), or 6848 bytes => 1696 bytes and 1920 MB/s => 1263 MB/s (clang). Maybe still worth it, though not quite as clearly beneficial.Fixes 91d689337fe8 ("crypto: blake2b - add blake2b generic implementation")Reviewed-by Ard Biesheuvel <[email protected]>Signed-off-by Eric Biggers <[email protected]>
Applied to https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=libcrypto-fixes
- Eric