crypto: poly1305 improvements
Subsystems (6)
Jason A. Donenfeld
maintainerv5 cover crypto: poly1305 improvements
These are some improvements to the Poly1305 code that I think should be fairly uncontroversial. The first part, the new C implementations, adds cleaner code in two forms that can easily be compared and reviewed, and also results in performance speedups. The second part, the new x86_64 implementation, replaces an slow unvetted implementation with an extremely fast implementation that has received many eyeballs. Finally, we fix up some deadcode.
This v5 improves on v3 with better function signatures for the core implementation, and on v4 with more information about performance and benchmarking in the commit messages.
Jason A. Donenfeld (3):
crypto: poly1305 - add new 32 and 64-bit generic versions
crypto: x86_64/poly1305 - add faster implementations
crypto: arm/arm64/mips/poly1305 - remove redundant non-reduction from
emit
arch/arm/crypto/poly1305-glue.c | 18 +- arch/arm64/crypto/poly1305-glue.c | 18 +- arch/mips/crypto/poly1305-glue.c | 18 +- arch/x86/crypto/Makefile | 11 +- arch/x86/crypto/poly1305-avx2-x86_64.S | 390 --- arch/x86/crypto/poly1305-sse2-x86_64.S | 590 ---- arch/x86/crypto/poly1305-x86_64.pl | 4266 ++++++++++++++++++++++++ arch/x86/crypto/poly1305_glue.c | 308 +- crypto/adiantum.c | 4 +- crypto/nhpoly1305.c | 2 +- crypto/poly1305_generic.c | 27 +- include/crypto/internal/poly1305.h | 50 +- include/crypto/nhpoly1305.h | 4 +- include/crypto/poly1305.h | 16 +- lib/crypto/Kconfig | 4 +- lib/crypto/Makefile | 4 +- lib/crypto/poly1305-donna32.c | 204 ++ lib/crypto/poly1305-donna64.c | 185 + lib/crypto/poly1305.c | 174 +- 19 files changed, 4926 insertions(+), 1367 deletions(-) delete mode 100644 arch/x86/crypto/poly1305-avx2-x86_64.S delete mode 100644 arch/x86/crypto/poly1305-sse2-x86_64.S create mode 100644 arch/x86/crypto/poly1305-x86_64.pl create mode 100644 lib/crypto/poly1305-donna32.c create mode 100644 lib/crypto/poly1305-donna64.c
Signature · 1 line
2.24.1
Jason A. Donenfeld
maintainerPATCH v5 1/3 crypto: poly1305 - add new 32 and 64-bit generic versions
These two C implementations from Zinc -- a 32x32 one and a 64x64 one, depending on the platform -- come from Andrew Moon's public domain poly1305-donna portable code, modified for usage in the kernel. The precomputation in the 32-bit version and the use of 64x64 multiplies in the 64-bit version make these perform better than the code it replaces. Moon's code is also very widespread and has received many eyeballs of scrutiny.
There's a bit of interference between the x86 implementation, which relies on internal details of the old scalar implementation. In the next commit, the x86 implementation will be replaced with a faster one that doesn't rely on this, so none of this matters much. But for now, to keep this passing the tests, we inline the bits of the old implementation that the x86 implementation relied on. Also, since we now support a slightly larger key space, via the union, some offsets had to be fixed up.
Nonce calculation was folded in with the emit function, to take advantage of 64x64 arithmetic. However, Adiantum appeared to rely on no nonce handling in emit, so this path was conditionalized. We also introduced a new struct, poly1305_core_key, to represent the precise amount of space that particular implementation uses.
Testing with kbench9000, depending on the CPU, the update function for the 32x32 version has been improved by 4%-7%, and for the 64x64 by 19%-30%. The 32x32 gains are small, but I think there's great value in having a parallel implementation to the 64x64 one so that the two can be compared side-by-side as nice stand-alone units.
--- arch/x86/crypto/poly1305-avx2-x86_64.S | 20 +-- arch/x86/crypto/poly1305_glue.c | 215 +++++++++++++++++++++++-- crypto/adiantum.c | 4 +- crypto/nhpoly1305.c | 2 +- crypto/poly1305_generic.c | 27 +++- include/crypto/internal/poly1305.h | 50 ++---- include/crypto/nhpoly1305.h | 4 +- include/crypto/poly1305.h | 16 +- lib/crypto/Kconfig | 2 +- lib/crypto/Makefile | 4 +- lib/crypto/poly1305-donna32.c | 204 +++++++++++++++++++++++ lib/crypto/poly1305-donna64.c | 185 +++++++++++++++++++++ lib/crypto/poly1305.c | 174 ++------------------ 13 files changed, 677 insertions(+), 230 deletions(-) create mode 100644 lib/crypto/poly1305-donna32.c create mode 100644 lib/crypto/poly1305-donna64.c
arch/x86/crypto/poly1305-avx2-x86_64.S +10 -10
@@ -34,16 +34,16 @@ ORMASK: .octa 0x00000000010000000000000001000000 #define u2 0x08(%r8) #define u3 0x0c(%r8) #define u4 0x10(%r8) -#define w0 0x14(%r8) -#define w1 0x18(%r8) -#define w2 0x1c(%r8) -#define w3 0x20(%r8) -#define w4 0x24(%r8) -#define y0 0x28(%r8) -#define y1 0x2c(%r8) -#define y2 0x30(%r8) -#define y3 0x34(%r8) -#define y4 0x38(%r8) +#define w0 0x18(%r8) +#define w1 0x1c(%r8) +#define w2 0x20(%r8) +#define w3 0x24(%r8) +#define w4 0x28(%r8) +#define y0 0x30(%r8) +#define y1 0x34(%r8) +#define y2 0x38(%r8) +#define y3 0x3c(%r8) +#define y4 0x40(%r8) #define m %rsi #define hc0 %ymm0 #define hc1 %ymm1
arch/x86/crypto/poly1305_glue.c +204 -11
252 lines not on this page · raw message
crypto/adiantum.c +2 -2
@@ -72,7 +72,7 @@ struct adiantum_tfm_ctx {struct crypto_skcipher *streamcipher;struct crypto_cipher *blockcipher;struct crypto_shash *hash; -struct poly1305_key header_hash_key; +struct poly1305_core_key header_hash_key; };struct adiantum_request_ctx {
@@ -249,7 +249,7 @@ static void adiantum_hash_header(struct skcipher_request *req)poly1305_core_blocks (&state, &tctx->header_hash_key, req->iv,TWEAK_SIZE /POLY1305_BLOCK_SIZE ,1 ); -poly1305_core_emit (&state, &rctx->header_hash); +poly1305_core_emit (&state,NULL , &rctx->header_hash); }/* Hash the left-hand part (the "bulk") of the message using NHPoly1305 */
crypto/nhpoly1305.c +1 -1
@@ -210,7 +210,7 @@ int crypto_nhpoly1305_final_helper(struct shash_desc *desc, u8 *dst, nh_t nh_fn)if (state->nh_remaining)process_nh_hash_value (state, key); -poly1305_core_emit (&state->poly_state, dst); +poly1305_core_emit (&state->poly_state,NULL , dst);return 0 ; }EXPORT_SYMBOL (crypto_nhpoly1305_final_helper);
crypto/poly1305_generic.c +25 -2
@@ -31,6 +31,29 @@ static int crypto_poly1305_init(struct shash_desc *desc)return 0 ; } +static unsigned int crypto_poly1305_setdesckey (struct poly1305_desc_ctx *dctx, +const u8 *src,unsigned int srclen) +{ +if (!dctx->sset) { +if (!dctx->rset && srclen >=POLY1305_BLOCK_SIZE ) { +poly1305_core_setkey ((struct poly1305_core_key *)dctx->r, src); + src +=POLY1305_BLOCK_SIZE ; + srclen -=POLY1305_BLOCK_SIZE ; + dctx->rset =2 ; + } +if (srclen >=POLY1305_BLOCK_SIZE ) { + dctx->s[0 ] =get_unaligned_le32 (src +0 ); + dctx->s[1 ] =get_unaligned_le32 (src +4 ); + dctx->s[2 ] =get_unaligned_le32 (src +8 ); + dctx->s[3 ] =get_unaligned_le32 (src +12 ); + src +=POLY1305_BLOCK_SIZE ; + srclen -=POLY1305_BLOCK_SIZE ; + dctx->sset = true; + } + } +return srclen; +} +static void poly1305_blocks (struct poly1305_desc_ctx *dctx,const u8 *src,unsigned int srclen) {
@@ -42,8 +65,8 @@ static void poly1305_blocks(struct poly1305_desc_ctx *dctx, const u8 *src, srclen = datalen; } -poly1305_core_blocks (&dctx->h, dctx->r, src, - srclen /POLY1305_BLOCK_SIZE ,1 ); +poly1305_core_blocks (&dctx->h, (struct poly1305_core_key *)dctx->r, + src, srclen /POLY1305_BLOCK_SIZE ,1 ); } static int crypto_poly1305_update(struct shash_desc *desc,
include/crypto/internal/poly1305.h +15 -35
63 lines not on this page · raw message
include/crypto/nhpoly1305.h +2 -2
@@ -7,7 +7,7 @@#define _NHPOLY1305_H#include <crypto/hash.h> -#include <crypto/poly1305.h> +#include <crypto/internal/poly1305.h> /* NH parameterization: */
@@ -33,7 +33,7 @@#define NHPOLY1305_KEY_SIZE (POLY1305_BLOCK_SIZE + NH_KEY_BYTES)struct nhpoly1305_key { -struct poly1305_key poly_key; +struct poly1305_core_key poly_key;u32 nh_key[NH_KEY_WORDS ]; };
include/crypto/poly1305.h +14 -2
@@ -13,12 +13,24 @@#define POLY1305_KEY_SIZE 32#define POLY1305_DIGEST_SIZE 16 +/* The poly1305_key and poly1305_state types are mostly opaque and +* implementation-defined. Limbs might be in base 2^64 or base 2^26, or +* different yet. The union type provided keeps these 64-bit aligned for the +* case in which this is implemented using 64x64 multiplies. +*/ +struct poly1305_key { -u32 r[5 ];/* key, base 2^26 */ +union { +u32 r[5 ]; +u64 r64[3 ]; + }; };struct poly1305_state { -u32 h[5 ];/* accumulator, base 2^26 */ +union { +u32 h[5 ]; +u64 h64[3 ]; + }; };struct poly1305_desc_ctx {
lib/crypto/Kconfig +1 -1
@@ -92,7 +92,7 @@ config CRYPTO_LIB_POLY1305_RSIZE default 2 if MIPS default 4 if X86_64 default 9 if ARM || ARM64 - default 1 + default 2 config CRYPTO_ARCH_HAVE_LIB_POLY1305 tristate
lib/crypto/Makefile +3 -1
@@ -28,7 +28,9 @@ obj-$(CONFIG_CRYPTO_LIB_DES) += libdes.o libdes-y := des.o obj-$(CONFIG_CRYPTO_LIB_POLY1305_GENERIC) += libpoly1305.o -libpoly1305-y := poly1305.o +libpoly1305-y := poly1305-donna32.o +libpoly1305-$(CONFIG_ARCH_SUPPORTS_INT128) := poly1305-donna64.o +libpoly1305-y += poly1305.o obj-$(CONFIG_CRYPTO_LIB_SHA256) += libsha256.o libsha256-y := sha256.o
lib/crypto/poly1305-donna32.c new file +204 -0
204 lines not on this page · raw message
lib/crypto/poly1305-donna64.c new file +185 -0
185 lines not on this page · raw message
lib/crypto/poly1305.c +11 -163
211 lines not on this page · raw message
Signature · 1 line
2.24.1
Jason A. Donenfeld
maintainerPATCH v5 2/3 crypto: x86_64/poly1305 - add faster implementations
In reply to Jason A. DonenfeldThese x86_64 vectorized implementations are based on Andy Polyakov's implementation, and support AVX, AVX-2, and AVX512F. The AVX-512F implementation is disabled on Skylake, due to throttling, but it is quite fast on >= Cannonlake.
On the left is cycle counts on a Core i7 6700HQ using the AVX-2 codepath, comparing this implementation ("new") to the implementation in the current crypto api ("old"). On the right are benchmarks on a Xeon Gold 5120 using the AVX-512 codepath. The new implementation is faster on all benchmarks.
AVX-2 AVX-512
--------- -----------
size old new size old new ---- ---- ---- ---- ---- ---- 0 70 68 0 74 70 16 92 90 16 96 92 32 134 104 32 136 106 48 172 120 48 184 124 64 218 136 64 218 138 80 254 158 80 260 160 96 298 174 96 300 176 112 342 192 112 342 194 128 388 212 128 384 212 144 428 228 144 420 226 160 466 246 160 464 248 176 510 264 176 504 264 192 550 282 192 544 282 208 594 302 208 582 300 224 628 316 224 624 318 240 676 334 240 662 338 256 716 354 256 708 358 272 764 374 272 748 372 288 802 352 288 788 358 304 420 366 304 422 370 320 428 360 320 432 364 336 484 378 336 486 380 352 426 384 352 434 390 368 478 400 368 480 408 384 488 394 384 490 398 400 542 408 400 542 412 416 486 416 416 492 426 432 534 430 432 538 436 448 544 422 448 546 432 464 600 438 464 600 448 480 540 448 480 548 456 496 594 464 496 594 476 512 602 456 512 606 470 528 656 476 528 656 480 544 600 480 544 606 498 560 650 494 560 652 512 576 664 490 576 662 508 592 714 508 592 716 522 608 656 514 608 664 538 624 708 532 624 710 552 640 716 524 640 720 516 656 770 536 656 772 526 672 716 548 672 722 544 688 770 562 688 768 556 704 774 552 704 778 556 720 826 568 720 832 568 736 768 574 736 780 584 752 822 592 752 826 600 768 830 584 768 836 560 784 884 602 784 888 572 800 828 610 800 838 588 816 884 628 816 884 604 832 888 618 832 894 598 848 942 632 848 946 612 864 884 644 864 896 628 880 936 660 880 942 644 896 948 652 896 952 608 912 1000 664 912 1004 616 928 942 676 928 954 634 944 994 690 944 1000 646 960 1002 680 960 1008 646 976 1054 694 976 1062 658 992 1002 706 992 1012 674 1008 1052 720 1008 1058 690
While this is CRYPTOGAMS code, the originating code for this happens to be derived from OpenSSL's commit 4dfe4310c31c4483705991d9a798ce9be1ed1c68.
--- arch/x86/crypto/Makefile | 11 +- arch/x86/crypto/poly1305-avx2-x86_64.S | 390 --- arch/x86/crypto/poly1305-sse2-x86_64.S | 590 ---- arch/x86/crypto/poly1305-x86_64.pl | 4266 ++++++++++++++++++++++++ arch/x86/crypto/poly1305_glue.c | 473 +-- lib/crypto/Kconfig | 2 +- 6 files changed, 4443 insertions(+), 1289 deletions(-) delete mode 100644 arch/x86/crypto/poly1305-avx2-x86_64.S delete mode 100644 arch/x86/crypto/poly1305-sse2-x86_64.S create mode 100644 arch/x86/crypto/poly1305-x86_64.pl
arch/x86/crypto/Makefile +9 -2
@@ -73,6 +73,10 @@ aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o nhpoly1305-sse2-y := nh-sse2-x86_64.o nhpoly1305-sse2-glue.o blake2s-x86_64-y := blake2s-core.o blake2s-glue.o +poly1305-x86_64-y := poly1305-x86_64.o poly1305_glue.o +ifneq ($(CONFIG_CRYPTO_POLY1305_X86_64),) +targets += poly1305-x86_64.S +endif ifeq ($(avx_supported),yes) camellia-aesni-avx-x86_64-y := camellia-aesni-avx-asm_64.o \
@@ -101,10 +105,8 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o sha1-ssse3-y := sha1_ssse3_asm.o sha1_ssse3_glue.o -poly1305-x86_64-y := poly1305-sse2-x86_64.o poly1305_glue.o ifeq ($(avx2_supported),yes) sha1-ssse3-y += sha1_avx2_x86_64_asm.o -poly1305-x86_64-y += poly1305-avx2-x86_64.o endif ifeq ($(sha1_ni_supported),yes) sha1-ssse3-y += sha1_ni_asm.o
@@ -118,3 +120,8 @@ sha256-ssse3-y += sha256_ni_asm.o endif sha512-ssse3-y := sha512-ssse3-asm.o sha512-avx-asm.o sha512-avx2-asm.o sha512_ssse3_glue.o crct10dif-pclmul-y := crct10dif-pcl-asm_64.o crct10dif-pclmul_glue.o + +quiet_cmd_perlasm = PERLASM $@ + cmd_perlasm = $(PERL) $< > $@ +$(obj)/%.S: $(src)/%.pl FORCE + $(call if_changed,perlasm)
arch/x86/crypto/poly1305-avx2-x86_64.S deleted +0 -390
390 lines not on this page · raw message
arch/x86/crypto/poly1305-sse2-x86_64.S deleted +0 -590
590 lines not on this page · raw message
arch/x86/crypto/poly1305-x86_64.pl new file +4266 -0
1,992 lines not on this page · raw message
arch/x86/crypto/poly1305_glue.c +167 -306
473 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.24.1
Part of the diff is not shown here. View the raw message.
Eric Biggers
maintainerRe: [PATCH crypto-next v5 2/3] crypto: x86_64/poly1305 - add faster implementations
arch/x86/crypto/Makefile aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o · line 79
Quoted 12 lines · Jason A. Donenfeld
arch/x86/crypto/Makefile +4
-0@@ -73,6 +73,10 @@ aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o nhpoly1305-sse2-y := nh-sse2-x86_64.o nhpoly1305-sse2-glue.o blake2s-x86_64-y := blake2s-core.o blake2s-glue.o +poly1305-x86_64-y := poly1305-x86_64.o poly1305_glue.o +ifneq ($(CONFIG_CRYPTO_POLY1305_X86_64),) +targets += poly1305-x86_64.S +endif
poly1305-x86_64.S is a generated file, so it needs to be listed in a gitignore file arch/x86/crypto/.gitignore.
- Eric
Jason A. Donenfeld
maintainerRe: [PATCH crypto-next v5 2/3] crypto: x86_64/poly1305 - add faster implementations
Quoted 8 lines · Eric Biggers
Quoted 6 lines · Jason A. Donenfeld
arch/x86/crypto/Makefile +0
-0@@ -73,6 +73,10 @@ aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o
Quoted 9 lines
arch/x86/crypto/Makefile aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o · line 79
Quoted 6 lines
arch/x86/crypto/Makefile +4
-0nhpoly1305-sse2-y := nh-sse2-x86_64.o nhpoly1305-sse2-glue.o blake2s-x86_64-y := blake2s-core.o blake2s-glue.o +poly1305-x86_64-y := poly1305-x86_64.o poly1305_glue.o +ifneq ($(CONFIG_CRYPTO_POLY1305_X86_64),) +targets += poly1305-x86_64.S +endif
poly1305-x86_64.Sis a generated file, so it needs to be listed in a gitignore filearch/x86/crypto/.gitignore.
Ack. I made the change and put it in for-cryptodev-2.6:
Jason A. Donenfeld
maintainerPATCH v5 3/3 crypto: arm/arm64/mips/poly1305 - remove redundant non-reduction from emit
In reply to Jason A. DonenfeldThis appears to be some kind of copy and paste error, and is actually dead code.
Pre: f = 0 ⇒ (f >> 32) = 0
f = (f >> 32) + le32_to_cpu(digest[0]);
Post: 0 ≤ f < 2³²
put_unaligned_le32(f, dst);Pre: 0 ≤ f < 2³² ⇒ (f >> 32) = 0
f = (f >> 32) + le32_to_cpu(digest[1]);
Post: 0 ≤ f < 2³²
put_unaligned_le32(f, dst + 4);Pre: 0 ≤ f < 2³² ⇒ (f >> 32) = 0
f = (f >> 32) + le32_to_cpu(digest[2]);
Post: 0 ≤ f < 2³²
put_unaligned_le32(f, dst + 8);Pre: 0 ≤ f < 2³² ⇒ (f >> 32) = 0
f = (f >> 32) + le32_to_cpu(digest[3]);
Post: 0 ≤ f < 2³²
put_unaligned_le32(f, dst + 12);Therefore this sequence is redundant. And Andy's code appears to handle misalignment acceptably.
--- arch/arm/crypto/poly1305-glue.c | 18 ++---------------- arch/arm64/crypto/poly1305-glue.c | 18 ++---------------- arch/mips/crypto/poly1305-glue.c | 18 ++---------------- 3 files changed, 6 insertions(+), 48 deletions(-)
arch/arm/crypto/poly1305-glue.c +2 -16
@@ -20,7 +20,7 @@void poly1305_init_arm (void *state,const u8 *key);void poly1305_blocks_arm (void *state,const u8 *src,u32 len,u32 hibit); -void poly1305_emit_arm (void *state,__le32 *digest,const u32 *nonce); +void poly1305_emit_arm (void *state,u8 *digest,const u32 *nonce);void __weak poly1305_blocks_neon (void *state,const u8 *src,u32 len,u32 hibit) {
@@ -179,9 +179,6 @@ EXPORT_SYMBOL(poly1305_update_arch);void poly1305_final_arch (struct poly1305_desc_ctx *dctx,u8 *dst) { -__le32 digest[4 ]; -u64 f =0 ; -if (unlikely (dctx->buflen)) { dctx->buf[dctx->buflen++] =1 ; memset(dctx->buf + dctx->buflen, 0,
@@ -189,18 +186,7 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst)poly1305_blocks_arm (&dctx->h, dctx->buf,POLY1305_BLOCK_SIZE ,0 ); } -poly1305_emit_arm (&dctx->h, digest, dctx->s); - -/* mac = (h + s) % (2^128) */ - f = (f >>32 ) +le32_to_cpu (digest[0 ]); -put_unaligned_le32 (f, dst); - f = (f >>32 ) +le32_to_cpu (digest[1 ]); -put_unaligned_le32 (f, dst +4 ); - f = (f >>32 ) +le32_to_cpu (digest[2 ]); -put_unaligned_le32 (f, dst +8 ); - f = (f >>32 ) +le32_to_cpu (digest[3 ]); -put_unaligned_le32 (f, dst +12 ); - +poly1305_emit_arm (&dctx->h, dst, dctx->s); *dctx = (struct poly1305_desc_ctx ){}; }EXPORT_SYMBOL (poly1305_final_arch);
arch/arm64/crypto/poly1305-glue.c +2 -16
@@ -21,7 +21,7 @@ asmlinkage void poly1305_init_arm64(void *state, const u8 *key); asmlinkage void poly1305_blocks(void *state, const u8 *src, u32 len, u32 hibit); asmlinkage void poly1305_blocks_neon(void *state, const u8 *src, u32 len, u32 hibit); -asmlinkage void poly1305_emit(void *state, __le32 *digest, const u32 *nonce); +asmlinkage void poly1305_emit(void *state, u8 *digest, const u32 *nonce);static __ro_after_init DEFINE_STATIC_KEY_FALSE (have_neon);
@@ -162,9 +162,6 @@ EXPORT_SYMBOL(poly1305_update_arch);void poly1305_final_arch (struct poly1305_desc_ctx *dctx,u8 *dst) { -__le32 digest[4 ]; -u64 f =0 ; -if (unlikely (dctx->buflen)) { dctx->buf[dctx->buflen++] =1 ; memset(dctx->buf + dctx->buflen, 0,
@@ -172,18 +169,7 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst)poly1305_blocks (&dctx->h, dctx->buf,POLY1305_BLOCK_SIZE ,0 ); } -poly1305_emit (&dctx->h, digest, dctx->s); - -/* mac = (h + s) % (2^128) */ - f = (f >>32 ) +le32_to_cpu (digest[0 ]); -put_unaligned_le32 (f, dst); - f = (f >>32 ) +le32_to_cpu (digest[1 ]); -put_unaligned_le32 (f, dst +4 ); - f = (f >>32 ) +le32_to_cpu (digest[2 ]); -put_unaligned_le32 (f, dst +8 ); - f = (f >>32 ) +le32_to_cpu (digest[3 ]); -put_unaligned_le32 (f, dst +12 ); - +poly1305_emit (&dctx->h, dst, dctx->s); *dctx = (struct poly1305_desc_ctx ){}; }EXPORT_SYMBOL (poly1305_final_arch);
arch/mips/crypto/poly1305-glue.c +2 -16
@@ -15,7 +15,7 @@ asmlinkage void poly1305_init_mips(void *state, const u8 *key); asmlinkage void poly1305_blocks_mips(void *state, const u8 *src, u32 len, u32 hibit); -asmlinkage void poly1305_emit_mips(void *state, __le32 *digest, const u32 *nonce); +asmlinkage void poly1305_emit_mips(void *state, u8 *digest, const u32 *nonce);void poly1305_init_arch (struct poly1305_desc_ctx *dctx,const u8 *key) {
@@ -134,9 +134,6 @@ EXPORT_SYMBOL(poly1305_update_arch);void poly1305_final_arch (struct poly1305_desc_ctx *dctx,u8 *dst) { -__le32 digest[4 ]; -u64 f =0 ; -if (unlikely (dctx->buflen)) { dctx->buf[dctx->buflen++] =1 ; memset(dctx->buf + dctx->buflen, 0,
@@ -144,18 +141,7 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst)poly1305_blocks_mips (&dctx->h, dctx->buf,POLY1305_BLOCK_SIZE ,0 ); } -poly1305_emit_mips (&dctx->h, digest, dctx->s); - -/* mac = (h + s) % (2^128) */ - f = (f >>32 ) +le32_to_cpu (digest[0 ]); -put_unaligned_le32 (f, dst); - f = (f >>32 ) +le32_to_cpu (digest[1 ]); -put_unaligned_le32 (f, dst +4 ); - f = (f >>32 ) +le32_to_cpu (digest[2 ]); -put_unaligned_le32 (f, dst +8 ); - f = (f >>32 ) +le32_to_cpu (digest[3 ]); -put_unaligned_le32 (f, dst +12 ); - +poly1305_emit_mips (&dctx->h, dst, dctx->s); *dctx = (struct poly1305_desc_ctx ){}; }EXPORT_SYMBOL (poly1305_final_arch);
Signature · 1 line
2.24.1