Skip to contentSkip to conversation

crypto: poly1305 improvements

PATCH Superseded Jason A. Donenfeld 2 participants 7 messages Dec 17, 2019 – Dec 18, 2019

Subsystems (6)
Arm (32-bit) x86 MIPS Crypto API +2 more subsystems

v6 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Jason A. Donenfeld

author maintainer
#0 lore raw

v6 cover crypto: poly1305 improvements

With no feedback on v5 beyond the need for a .gitignore in the second commit of this v6, I think this series should now be good to go.

These are some improvements to the Poly1305 code that I think should be fairly uncontroversial. The first part, the new C implementations, adds cleaner code in two forms that can easily be compared and reviewed, and also results in performance speedups. The second part, the new x86_64 implementation, replaces an slow unvetted implementation with an extremely fast implementation that has received many eyeballs. Finally, we fix up some deadcode.

Cc Eric Biggers <[email protected]>
Cc Ard Biesheuvel <[email protected]>

Jason A. Donenfeld (3):
crypto: poly1305 - add new 32 and 64-bit generic versions
crypto: x86_64/poly1305 - add faster implementations
crypto: arm/arm64/mips/poly1305 - remove redundant non-reduction from
emit

 arch/arm/crypto/poly1305-glue.c        |   18 +-
 arch/arm64/crypto/poly1305-glue.c      |   18 +-
 arch/mips/crypto/poly1305-glue.c       |   18 +-
 arch/x86/crypto/.gitignore             |    1 +
 arch/x86/crypto/Makefile               |   11 +-
 arch/x86/crypto/poly1305-avx2-x86_64.S |  390 ---
 arch/x86/crypto/poly1305-sse2-x86_64.S |  590 ----
 arch/x86/crypto/poly1305-x86_64.pl     | 4266 ++++++++++++++++++++++++
 arch/x86/crypto/poly1305_glue.c        |  308 +-
 crypto/adiantum.c                      |    4 +-
 crypto/nhpoly1305.c                    |    2 +-
 crypto/poly1305_generic.c              |   27 +-
 include/crypto/internal/poly1305.h     |   50 +-
 include/crypto/nhpoly1305.h            |    4 +-
 include/crypto/poly1305.h              |   16 +-
 lib/crypto/Kconfig                     |    4 +-
 lib/crypto/Makefile                    |    4 +-
 lib/crypto/poly1305-donna32.c          |  204 ++
 lib/crypto/poly1305-donna64.c          |  185 +
 lib/crypto/poly1305.c                  |  174 +-
 20 files changed, 4927 insertions(+), 1367 deletions(-)
 create mode 100644 arch/x86/crypto/.gitignore
 delete mode 100644 arch/x86/crypto/poly1305-avx2-x86_64.S
 delete mode 100644 arch/x86/crypto/poly1305-sse2-x86_64.S
 create mode 100644 arch/x86/crypto/poly1305-x86_64.pl
 create mode 100644 lib/crypto/poly1305-donna32.c
 create mode 100644 lib/crypto/poly1305-donna64.c
Signature · 1 line
2.24.1
author maintainer
#1 lore raw

PATCH v6 1/3 crypto: poly1305 - add new 32 and 64-bit generic versions

These two C implementations from Zinc -- a 32x32 one and a 64x64 one, depending on the platform -- come from Andrew Moon's public domain poly1305-donna portable code, modified for usage in the kernel. The precomputation in the 32-bit version and the use of 64x64 multiplies in the 64-bit version make these perform better than the code it replaces. Moon's code is also very widespread and has received many eyeballs of scrutiny.

There's a bit of interference between the x86 implementation, which relies on internal details of the old scalar implementation. In the next commit, the x86 implementation will be replaced with a faster one that doesn't rely on this, so none of this matters much. But for now, to keep this passing the tests, we inline the bits of the old implementation that the x86 implementation relied on. Also, since we now support a slightly larger key space, via the union, some offsets had to be fixed up.

Nonce calculation was folded in with the emit function, to take advantage of 64x64 arithmetic. However, Adiantum appeared to rely on no nonce handling in emit, so this path was conditionalized. We also introduced a new struct, poly1305_core_key, to represent the precise amount of space that particular implementation uses.

Testing with kbench9000, depending on the CPU, the update function for the 32x32 version has been improved by 4%-7%, and for the 64x64 by 19%-30%. The 32x32 gains are small, but I think there's great value in having a parallel implementation to the 64x64 one so that the two can be compared side-by-side as nice stand-alone units.

---
 arch/x86/crypto/poly1305-avx2-x86_64.S |  20 +--
 arch/x86/crypto/poly1305_glue.c        | 215 +++++++++++++++++++++++--
 crypto/adiantum.c                      |   4 +-
 crypto/nhpoly1305.c                    |   2 +-
 crypto/poly1305_generic.c              |  27 +++-
 include/crypto/internal/poly1305.h     |  50 ++----
 include/crypto/nhpoly1305.h            |   4 +-
 include/crypto/poly1305.h              |  16 +-
 lib/crypto/Kconfig                     |   2 +-
 lib/crypto/Makefile                    |   4 +-
 lib/crypto/poly1305-donna32.c          | 204 +++++++++++++++++++++++
 lib/crypto/poly1305-donna64.c          | 185 +++++++++++++++++++++
 lib/crypto/poly1305.c                  | 174 ++------------------
 13 files changed, 677 insertions(+), 230 deletions(-)
 create mode 100644 lib/crypto/poly1305-donna32.c
 create mode 100644 lib/crypto/poly1305-donna64.c
arch/x86/crypto/poly1305-avx2-x86_64.S +10 -10
index d6063feda9da..8f56989ea599 100644
@@ -34,16 +34,16 @@ ORMASK:	.octa 0x00000000010000000000000001000000
 #define u2 0x08(%r8)
 #define u3 0x0c(%r8)
 #define u4 0x10(%r8)
-#define w0 0x14(%r8)
-#define w1 0x18(%r8)
-#define w2 0x1c(%r8)
-#define w3 0x20(%r8)
-#define w4 0x24(%r8)
-#define y0 0x28(%r8)
-#define y1 0x2c(%r8)
-#define y2 0x30(%r8)
-#define y3 0x34(%r8)
-#define y4 0x38(%r8)
+#define w0 0x18(%r8)
+#define w1 0x1c(%r8)
+#define w2 0x20(%r8)
+#define w3 0x24(%r8)
+#define w4 0x28(%r8)
+#define y0 0x30(%r8)
+#define y1 0x34(%r8)
+#define y2 0x38(%r8)
+#define y3 0x3c(%r8)
+#define y4 0x40(%r8)
 #define m %rsi
 #define hc0 %ymm0
 #define hc1 %ymm1
arch/x86/crypto/poly1305_glue.c +204 -11

252 lines not on this page · raw message

crypto/adiantum.c +2 -2
index aded26092268..c846a887abe1 100644
@@ -72,7 +72,7 @@ struct adiantum_tfm_ctx {
 	struct crypto_skcipher *streamcipher;
 	struct crypto_cipher *blockcipher;
 	struct crypto_shash *hash;
-	struct poly1305_key header_hash_key;
+	struct poly1305_core_key header_hash_key;
 };
 
 struct adiantum_request_ctx {
@@ -249,7 +249,7 @@ static void adiantum_hash_header(struct skcipher_request *req)
 	poly1305_core_blocks(&state, &tctx->header_hash_key, req->iv,
 			     TWEAK_SIZE / POLY1305_BLOCK_SIZE, 1);
 
-	poly1305_core_emit(&state, &rctx->header_hash);
+	poly1305_core_emit(&state, NULL, &rctx->header_hash);
 }
 
 /* Hash the left-hand part (the "bulk") of the message using NHPoly1305 */
crypto/nhpoly1305.c +1 -1
index f6b6a52092b4..8a3006c3b51b 100644
@@ -210,7 +210,7 @@ int crypto_nhpoly1305_final_helper(struct shash_desc *desc, u8 *dst, nh_t nh_fn)
 	if (state->nh_remaining)
 		process_nh_hash_value(state, key);
 
-	poly1305_core_emit(&state->poly_state, dst);
+	poly1305_core_emit(&state->poly_state, NULL, dst);
 	return 0;
 }
 EXPORT_SYMBOL(crypto_nhpoly1305_final_helper);
crypto/poly1305_generic.c +25 -2
index 21edbd8c99fb..6488e8b8379c 100644
@@ -31,6 +31,29 @@ static int crypto_poly1305_init(struct shash_desc *desc)
 	return 0;
 }
 
+static unsigned int crypto_poly1305_setdesckey(struct poly1305_desc_ctx *dctx,
+					       const u8 *src, unsigned int srclen)
+{
+	if (!dctx->sset) {
+		if (!dctx->rset && srclen >= POLY1305_BLOCK_SIZE) {
+			poly1305_core_setkey((struct poly1305_core_key *)dctx->r, src);
+			src += POLY1305_BLOCK_SIZE;
+			srclen -= POLY1305_BLOCK_SIZE;
+			dctx->rset = 2;
+		}
+		if (srclen >= POLY1305_BLOCK_SIZE) {
+			dctx->s[0] = get_unaligned_le32(src +  0);
+			dctx->s[1] = get_unaligned_le32(src +  4);
+			dctx->s[2] = get_unaligned_le32(src +  8);
+			dctx->s[3] = get_unaligned_le32(src + 12);
+			src += POLY1305_BLOCK_SIZE;
+			srclen -= POLY1305_BLOCK_SIZE;
+			dctx->sset = true;
+		}
+	}
+	return srclen;
+}
+
 static void poly1305_blocks(struct poly1305_desc_ctx *dctx, const u8 *src,
 			    unsigned int srclen)
 {
@@ -42,8 +65,8 @@ static void poly1305_blocks(struct poly1305_desc_ctx *dctx, const u8 *src,
 		srclen = datalen;
 	}
 
-	poly1305_core_blocks(&dctx->h, dctx->r, src,
-			     srclen / POLY1305_BLOCK_SIZE, 1);
+	poly1305_core_blocks(&dctx->h, (struct poly1305_core_key *)dctx->r,
+			     src, srclen / POLY1305_BLOCK_SIZE, 1);
 }
 
 static int crypto_poly1305_update(struct shash_desc *desc,
include/crypto/internal/poly1305.h +15 -35

63 lines not on this page · raw message

include/crypto/nhpoly1305.h +2 -2
index 53c04423c582..306925fea190 100644
@@ -7,7 +7,7 @@
 #define _NHPOLY1305_H
 
 #include <crypto/hash.h>
-#include <crypto/poly1305.h>
+#include <crypto/internal/poly1305.h>
 
 /* NH parameterization: */
 
@@ -33,7 +33,7 @@
 #define NHPOLY1305_KEY_SIZE	(POLY1305_BLOCK_SIZE + NH_KEY_BYTES)
 
 struct nhpoly1305_key {
-	struct poly1305_key poly_key;
+	struct poly1305_core_key poly_key;
 	u32 nh_key[NH_KEY_WORDS];
 };
 
include/crypto/poly1305.h +14 -2
index 74c6e1cd73ee..57f73285b4a2 100644
@@ -13,12 +13,24 @@
 #define POLY1305_KEY_SIZE	32
 #define POLY1305_DIGEST_SIZE	16
 
+/* The poly1305_key and poly1305_state types are mostly opaque and
+ * implementation-defined. Limbs might be in base 2^64 or base 2^26, or
+ * different yet. The union type provided keeps these 64-bit aligned for the
+ * case in which this is implemented using 64x64 multiplies.
+ */
+
 struct poly1305_key {
-	u32 r[5];	/* key, base 2^26 */
+	union {
+		u32 r[5];
+		u64 r64[3];
+	};
 };
 
 struct poly1305_state {
-	u32 h[5];	/* accumulator, base 2^26 */
+	union {
+		u32 h[5];
+		u64 h64[3];
+	};
 };
 
 struct poly1305_desc_ctx {
lib/crypto/Kconfig +1 -1
index 0b2c4fce26d9..2d4a8f385e62 100644
@@ -92,7 +92,7 @@ config CRYPTO_LIB_POLY1305_RSIZE
 	default 2 if MIPS
 	default 4 if X86_64
 	default 9 if ARM || ARM64
-	default 1
+	default 2
 
 config CRYPTO_ARCH_HAVE_LIB_POLY1305
 	tristate
lib/crypto/Makefile +3 -1
index 34a701ab8b92..d19e5195569e 100644
@@ -28,7 +28,9 @@ obj-$(CONFIG_CRYPTO_LIB_DES)			+= libdes.o
 libdes-y					:= des.o
 
 obj-$(CONFIG_CRYPTO_LIB_POLY1305_GENERIC)	+= libpoly1305.o
-libpoly1305-y					:= poly1305.o
+libpoly1305-y					:= poly1305-donna32.o
+libpoly1305-$(CONFIG_ARCH_SUPPORTS_INT128)	:= poly1305-donna64.o
+libpoly1305-y					+= poly1305.o
 
 obj-$(CONFIG_CRYPTO_LIB_SHA256)			+= libsha256.o
 libsha256-y					:= sha256.o
lib/crypto/poly1305-donna32.c new file +204 -0

204 lines not on this page · raw message

lib/crypto/poly1305-donna64.c new file +185 -0

185 lines not on this page · raw message

lib/crypto/poly1305.c +11 -163

211 lines not on this page · raw message

Signature · 1 line
2.24.1
maintainer
#2 lore raw

Re: [PATCH crypto-next v6 1/3] crypto: poly1305 - add new 32 and 64-bit generic versions

Quoted 5 lines · Jason A. Donenfeld
crypto/poly1305_generic.c +0 -0
index 21edbd8c99fb..6488e8b8379c 100644

crypto/poly1305_generic.c static int crypto_poly1305_init(struct shash_desc *desc) · line 39

Quoted 9 lines
crypto/poly1305_generic.c +6 -0
@@ -31,6 +31,29 @@ static int crypto_poly1305_init(struct shash_desc *desc)
 	return 0;
 }
 
+static unsigned int crypto_poly1305_setdesckey(struct poly1305_desc_ctx *dctx,
+					       const u8 *src, unsigned int srclen)
+{
+	if (!dctx->sset) {
+		if (!dctx->rset && srclen >= POLY1305_BLOCK_SIZE) {
+			poly1305_core_setkey((struct poly1305_core_key *)dctx->r, src);

Seems this should be using a union, so that all these casts to a different struct type (which is normally undefined behavior) aren't needed.

- Eric

author maintainer
#3 lore raw

PATCH v6 2/3 crypto: x86_64/poly1305 - add faster implementations

In reply to Jason A. Donenfeld

These x86_64 vectorized implementations are based on Andy Polyakov's implementation, and support AVX, AVX-2, and AVX512F. The AVX-512F implementation is disabled on Skylake, due to throttling, but it is quite fast on >= Cannonlake.

On the left is cycle counts on a Core i7 6700HQ using the AVX-2 codepath, comparing this implementation ("new") to the implementation in the current crypto api ("old"). On the right are benchmarks on a Xeon Gold 5120 using the AVX-512 codepath. The new implementation is faster on all benchmarks.

AVX-2 AVX-512
--------- -----------

size    old     new      size   old     new
----    ----    ----     ----   ----    ----
0       70      68       0      74      70
16      92      90       16     96      92
32      134     104      32     136     106
48      172     120      48     184     124
64      218     136      64     218     138
80      254     158      80     260     160
96      298     174      96     300     176
112     342     192      112    342     194
128     388     212      128    384     212
144     428     228      144    420     226
160     466     246      160    464     248
176     510     264      176    504     264
192     550     282      192    544     282
208     594     302      208    582     300
224     628     316      224    624     318
240     676     334      240    662     338
256     716     354      256    708     358
272     764     374      272    748     372
288     802     352      288    788     358
304     420     366      304    422     370
320     428     360      320    432     364
336     484     378      336    486     380
352     426     384      352    434     390
368     478     400      368    480     408
384     488     394      384    490     398
400     542     408      400    542     412
416     486     416      416    492     426
432     534     430      432    538     436
448     544     422      448    546     432
464     600     438      464    600     448
480     540     448      480    548     456
496     594     464      496    594     476
512     602     456      512    606     470
528     656     476      528    656     480
544     600     480      544    606     498
560     650     494      560    652     512
576     664     490      576    662     508
592     714     508      592    716     522
608     656     514      608    664     538
624     708     532      624    710     552
640     716     524      640    720     516
656     770     536      656    772     526
672     716     548      672    722     544
688     770     562      688    768     556
704     774     552      704    778     556
720     826     568      720    832     568
736     768     574      736    780     584
752     822     592      752    826     600
768     830     584      768    836     560
784     884     602      784    888     572
800     828     610      800    838     588
816     884     628      816    884     604
832     888     618      832    894     598
848     942     632      848    946     612
864     884     644      864    896     628
880     936     660      880    942     644
896     948     652      896    952     608
912     1000    664      912    1004    616
928     942     676      928    954     634
944     994     690      944    1000    646
960     1002    680      960    1008    646
976     1054    694      976    1062    658
992     1002    706      992    1012    674
1008    1052    720      1008   1058    690

While this is CRYPTOGAMS code, the originating code for this happens to be derived from OpenSSL's commit 4dfe4310c31c4483705991d9a798ce9be1ed1c68.

---
 arch/x86/crypto/.gitignore             |    1 +
 arch/x86/crypto/Makefile               |   11 +-
 arch/x86/crypto/poly1305-avx2-x86_64.S |  390 ---
 arch/x86/crypto/poly1305-sse2-x86_64.S |  590 ----
 arch/x86/crypto/poly1305-x86_64.pl     | 4266 ++++++++++++++++++++++++
 arch/x86/crypto/poly1305_glue.c        |  473 +--
 lib/crypto/Kconfig                     |    2 +-
 7 files changed, 4444 insertions(+), 1289 deletions(-)
 create mode 100644 arch/x86/crypto/.gitignore
 delete mode 100644 arch/x86/crypto/poly1305-avx2-x86_64.S
 delete mode 100644 arch/x86/crypto/poly1305-sse2-x86_64.S
 create mode 100644 arch/x86/crypto/poly1305-x86_64.pl
arch/x86/crypto/.gitignore new file +1 -0
new file mode 100644
index 000000000000..c406ea6571fa
@@ -0,0 +1 @@
+poly1305-x86_64.S
arch/x86/crypto/Makefile +9 -2
index 958440eae27e..6982a2f8863f 100644
@@ -73,6 +73,10 @@ aegis128-aesni-y := aegis128-aesni-asm.o aegis128-aesni-glue.o
 
 nhpoly1305-sse2-y := nh-sse2-x86_64.o nhpoly1305-sse2-glue.o
 blake2s-x86_64-y := blake2s-core.o blake2s-glue.o
+poly1305-x86_64-y := poly1305-x86_64.o poly1305_glue.o
+ifneq ($(CONFIG_CRYPTO_POLY1305_X86_64),)
+targets += poly1305-x86_64.S
+endif
 
 ifeq ($(avx_supported),yes)
 	camellia-aesni-avx-x86_64-y := camellia-aesni-avx-asm_64.o \
@@ -101,10 +105,8 @@ aesni-intel-y := aesni-intel_asm.o aesni-intel_glue.o
 aesni-intel-$(CONFIG_64BIT) += aesni-intel_avx-x86_64.o aes_ctrby8_avx-x86_64.o
 ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
 sha1-ssse3-y := sha1_ssse3_asm.o sha1_ssse3_glue.o
-poly1305-x86_64-y := poly1305-sse2-x86_64.o poly1305_glue.o
 ifeq ($(avx2_supported),yes)
 sha1-ssse3-y += sha1_avx2_x86_64_asm.o
-poly1305-x86_64-y += poly1305-avx2-x86_64.o
 endif
 ifeq ($(sha1_ni_supported),yes)
 sha1-ssse3-y += sha1_ni_asm.o
@@ -118,3 +120,8 @@ sha256-ssse3-y += sha256_ni_asm.o
 endif
 sha512-ssse3-y := sha512-ssse3-asm.o sha512-avx-asm.o sha512-avx2-asm.o sha512_ssse3_glue.o
 crct10dif-pclmul-y := crct10dif-pcl-asm_64.o crct10dif-pclmul_glue.o
+
+quiet_cmd_perlasm = PERLASM $@
+      cmd_perlasm = $(PERL) $< > $@
+$(obj)/%.S: $(src)/%.pl FORCE
+	$(call if_changed,perlasm)
arch/x86/crypto/poly1305-avx2-x86_64.S deleted +0 -390

390 lines not on this page · raw message

arch/x86/crypto/poly1305-sse2-x86_64.S deleted +0 -590

590 lines not on this page · raw message

arch/x86/crypto/poly1305-x86_64.pl new file +4266 -0

1,991 lines not on this page · raw message

arch/x86/crypto/poly1305_glue.c +167 -306

473 lines not on this page · raw message

lib/crypto/Kconfig +1 -1
index 2d4a8f385e62..c2f4a2ea41d7 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.24.1

Part of the diff is not shown here. View the raw message.

maintainer
#4 lore raw

Re: [PATCH crypto-next v6 2/3] crypto: x86_64/poly1305 - add faster implementations

Quoted 3 lines · Jason A. Donenfeld

While this is CRYPTOGAMS code, the originating code for this happens to be derived from OpenSSL's commit 4dfe4310c31c4483705991d9a798ce9be1ed1c68.

This doesn't make sense, since there are changes in this patch that are only in the CRYPTOGAMS version, not in the OpenSSL 4dfe4310c31c version.

Quoted 4 lines

Each "Co-developed-by" always needs its own "Signed-off-by". See Documentation/process/submitting-patches.rst

Quoted 7 lines
arch/x86/crypto/poly1305-x86_64.pl new file +1 -0
new file mode 100644
index 000000000000..f994855cdbe2
@@ -0,0 +1,4266 @@
+#!/usr/bin/env perl

arch/x86/crypto/poly1305-x86_64.pl line 11

Quoted 10 lines
arch/x86/crypto/poly1305-x86_64.pl new file +10 -0
+# SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
+#
+# Copyright (C) 2017-2018 Samuel Neves <[email protected]>. All Rights Reserved.
+# Copyright (C) 2017-2019 Jason A. Donenfeld <[email protected]>. All Rights Reserved.
+# Copyright (C) 2006-2017 CRYPTOGAMS by <[email protected]>. All Rights Reserved.
+#
+# This code is taken from the OpenSSL project but the author, Andy Polyakov,
+# has relicensed it under the licenses specified in the SPDX header above.
+# The original headers, including the original license headers, are
+# included below for completeness.

And this says it's taken from the OpenSSL project, not CRYPTOGAMS.

Also how is this copyright 2006? The newest date in either the OpenSSL or CRYPTOGAMS versions is March 2015.

Also, can you elaborate on what changes have been made from version actually being used in production in OpenSSL, and how those changes were vetted? You keep claiming that this implementation is extremely well vetted in comparison to the kernel's existing implementation, so presumably you have something in mind.

- Eric

author maintainer
#5 lore raw

PATCH v6 3/3 crypto: arm/arm64/mips/poly1305 - remove redundant non-reduction from emit

In reply to Jason A. Donenfeld

This appears to be some kind of copy and paste error, and is actually dead code.

Pre: f = 0 ⇒ (f >> 32) = 0

    f = (f >> 32) + le32_to_cpu(digest[0]);
Post: 0 ≤ f < 2³²
    put_unaligned_le32(f, dst);

Pre: 0 ≤ f < 2³² ⇒ (f >> 32) = 0

    f = (f >> 32) + le32_to_cpu(digest[1]);
Post: 0 ≤ f < 2³²
    put_unaligned_le32(f, dst + 4);

Pre: 0 ≤ f < 2³² ⇒ (f >> 32) = 0

    f = (f >> 32) + le32_to_cpu(digest[2]);
Post: 0 ≤ f < 2³²
    put_unaligned_le32(f, dst + 8);

Pre: 0 ≤ f < 2³² ⇒ (f >> 32) = 0

    f = (f >> 32) + le32_to_cpu(digest[3]);
Post: 0 ≤ f < 2³²
    put_unaligned_le32(f, dst + 12);

Therefore this sequence is redundant. And Andy's code appears to handle misalignment acceptably.

---
 arch/arm/crypto/poly1305-glue.c   | 18 ++----------------
 arch/arm64/crypto/poly1305-glue.c | 18 ++----------------
 arch/mips/crypto/poly1305-glue.c  | 18 ++----------------
 3 files changed, 6 insertions(+), 48 deletions(-)
arch/arm/crypto/poly1305-glue.c +2 -16
index abe3f2d587dc..ceec04ec2f40 100644
@@ -20,7 +20,7 @@
 
 void poly1305_init_arm(void *state, const u8 *key);
 void poly1305_blocks_arm(void *state, const u8 *src, u32 len, u32 hibit);
-void poly1305_emit_arm(void *state, __le32 *digest, const u32 *nonce);
+void poly1305_emit_arm(void *state, u8 *digest, const u32 *nonce);
 
 void __weak poly1305_blocks_neon(void *state, const u8 *src, u32 len, u32 hibit)
 {
@@ -179,9 +179,6 @@ EXPORT_SYMBOL(poly1305_update_arch);
 
 void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst)
 {
-	__le32 digest[4];
-	u64 f = 0;
-
 	if (unlikely(dctx->buflen)) {
 		dctx->buf[dctx->buflen++] = 1;
 		memset(dctx->buf + dctx->buflen, 0,
@@ -189,18 +186,7 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst)
 		poly1305_blocks_arm(&dctx->h, dctx->buf, POLY1305_BLOCK_SIZE, 0);
 	}
 
-	poly1305_emit_arm(&dctx->h, digest, dctx->s);
-
-	/* mac = (h + s) % (2^128) */
-	f = (f >> 32) + le32_to_cpu(digest[0]);
-	put_unaligned_le32(f, dst);
-	f = (f >> 32) + le32_to_cpu(digest[1]);
-	put_unaligned_le32(f, dst + 4);
-	f = (f >> 32) + le32_to_cpu(digest[2]);
-	put_unaligned_le32(f, dst + 8);
-	f = (f >> 32) + le32_to_cpu(digest[3]);
-	put_unaligned_le32(f, dst + 12);
-
+	poly1305_emit_arm(&dctx->h, dst, dctx->s);
 	*dctx = (struct poly1305_desc_ctx){};
 }
 EXPORT_SYMBOL(poly1305_final_arch);
arch/arm64/crypto/poly1305-glue.c +2 -16
index 83a2338a8826..e97b092f56b8 100644
@@ -21,7 +21,7 @@
 asmlinkage void poly1305_init_arm64(void *state, const u8 *key);
 asmlinkage void poly1305_blocks(void *state, const u8 *src, u32 len, u32 hibit);
 asmlinkage void poly1305_blocks_neon(void *state, const u8 *src, u32 len, u32 hibit);
-asmlinkage void poly1305_emit(void *state, __le32 *digest, const u32 *nonce);
+asmlinkage void poly1305_emit(void *state, u8 *digest, const u32 *nonce);
 
 static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon);
 
@@ -162,9 +162,6 @@ EXPORT_SYMBOL(poly1305_update_arch);
 
 void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst)
 {
-	__le32 digest[4];
-	u64 f = 0;
-
 	if (unlikely(dctx->buflen)) {
 		dctx->buf[dctx->buflen++] = 1;
 		memset(dctx->buf + dctx->buflen, 0,
@@ -172,18 +169,7 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst)
 		poly1305_blocks(&dctx->h, dctx->buf, POLY1305_BLOCK_SIZE, 0);
 	}
 
-	poly1305_emit(&dctx->h, digest, dctx->s);
-
-	/* mac = (h + s) % (2^128) */
-	f = (f >> 32) + le32_to_cpu(digest[0]);
-	put_unaligned_le32(f, dst);
-	f = (f >> 32) + le32_to_cpu(digest[1]);
-	put_unaligned_le32(f, dst + 4);
-	f = (f >> 32) + le32_to_cpu(digest[2]);
-	put_unaligned_le32(f, dst + 8);
-	f = (f >> 32) + le32_to_cpu(digest[3]);
-	put_unaligned_le32(f, dst + 12);
-
+	poly1305_emit(&dctx->h, dst, dctx->s);
 	*dctx = (struct poly1305_desc_ctx){};
 }
 EXPORT_SYMBOL(poly1305_final_arch);
arch/mips/crypto/poly1305-glue.c +2 -16
index b37d29cf5d0a..fc881b46d911 100644
@@ -15,7 +15,7 @@
 
 asmlinkage void poly1305_init_mips(void *state, const u8 *key);
 asmlinkage void poly1305_blocks_mips(void *state, const u8 *src, u32 len, u32 hibit);
-asmlinkage void poly1305_emit_mips(void *state, __le32 *digest, const u32 *nonce);
+asmlinkage void poly1305_emit_mips(void *state, u8 *digest, const u32 *nonce);
 
 void poly1305_init_arch(struct poly1305_desc_ctx *dctx, const u8 *key)
 {
@@ -134,9 +134,6 @@ EXPORT_SYMBOL(poly1305_update_arch);
 
 void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst)
 {
-	__le32 digest[4];
-	u64 f = 0;
-
 	if (unlikely(dctx->buflen)) {
 		dctx->buf[dctx->buflen++] = 1;
 		memset(dctx->buf + dctx->buflen, 0,
@@ -144,18 +141,7 @@ void poly1305_final_arch(struct poly1305_desc_ctx *dctx, u8 *dst)
 		poly1305_blocks_mips(&dctx->h, dctx->buf, POLY1305_BLOCK_SIZE, 0);
 	}
 
-	poly1305_emit_mips(&dctx->h, digest, dctx->s);
-
-	/* mac = (h + s) % (2^128) */
-	f = (f >> 32) + le32_to_cpu(digest[0]);
-	put_unaligned_le32(f, dst);
-	f = (f >> 32) + le32_to_cpu(digest[1]);
-	put_unaligned_le32(f, dst + 4);
-	f = (f >> 32) + le32_to_cpu(digest[2]);
-	put_unaligned_le32(f, dst + 8);
-	f = (f >> 32) + le32_to_cpu(digest[3]);
-	put_unaligned_le32(f, dst + 12);
-
+	poly1305_emit_mips(&dctx->h, dst, dctx->s);
 	*dctx = (struct poly1305_desc_ctx){};
 }
 EXPORT_SYMBOL(poly1305_final_arch);
Signature · 1 line
2.24.1
maintainer
#6 lore raw
In reply to Jason A. Donenfeld
Quoted 2 lines · Jason A. Donenfeld

With no feedback on v5 beyond the need for a .gitignore in the second commit of this v6, I think this series should now be good to go.

Well, you've also sent 6 versions of this patchset in less than a week, so I'm not sure people have had enough time to review it. I've just left a few comments for things I noticed in a quick read-through.

That being said, I don't think anyone plans to review the 4200 lines of assembly code anyway (which even the other OpenSSL developers don't understand [1]), so maybe it doesn't matter.

[1] https://github.com/openssl/openssl/pull/2056#pullrequestreview-12242604

- Eric